* fix(http): bd export over http through issueops.ReadExport (S21)
bd export read through raw SearchIssues and the bulk label, dependency and
comment reads, which the HTTP backend refuses (B3 #16), so gc's JSONL export
order (gc bd export --all) failed on every remote scope.
issueops.ReadExport composes role reads every deployed server publishes: an
unlimited Reader.List for rows, labels and counts, EdgeReader.ReadEdges for
dependency records, and Reader.Get with comments for rows that have any. The
remote route feeds it into the same runExportFromSource the direct and
proxied routes use. A failed read fails the export. gc-export-all passes
against the reference profile; mc-promote stays pending (S21b) with its exact
reason: it needs a new server capability.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(export): read remote memories from the memory plane, pin unmarked no-history rows (bd-o4qlui)
Review fixes for #7551, attempts 1 to 3.
Attempt 3. No test reached roleExportSource.ExportMemories's error
path. A unit test now drives it with a fake store: a memory role that
cannot be reached, and a listMemories that fails, each end the export
with that error and no map. Rewriting either branch to degrade to an
empty map would bring back a remote `bd export --all` that exits 0 with
no memory in it.
Declined in attempt 3, for finalize to file: an unlimited HTTP listing
that stops on a has_more page with no next_cursor returns the rows it
has with HasMore false, so ReadExport's refusal cannot fire over HTTP.
Only a server that breaks the spec's has_more/next_cursor pairing gets
there; this repo's server always sends a cursor with a non-empty
has_more page. The walker (fetchIssuePages) is unchanged since #7247
and shared with bd list, so the fix needs its own PR that checks bd
list's callers first. The remote route also has no single snapshot: a
comment changed between List and Get can make the written count
disagree with the bodies kept. That belongs in one follow-up with the
bounded retry declined in attempt 1, as a count/body consistency check.
The --include-infra e2e leg stays declined. The wire does carry the
knob: the client sends include_infra and include_templates on
listIssues, and the server decodes both. The GetInfraTypes nil-store
guard stays declined, and auto-export's own GetAllConfig memory read
predates this PR.
Attempt 2. A remote export dropped every memory. Over HTTP, `bd export
--all` and `--include-memories` exited 0 without a memory record, where
the base refused with exit 1. The memory lines came from GetAllConfig,
which over HTTP is listSettings, and the server's settings enumeration
withholds every kv.* row, memories included. gc's JSONL export runs
`gc bd export --all`, so its archive lost them without a word.
The exportSource seam now asks each source for its memories
(ExportMemories), not for the whole config table. The local and proxied
sources still take them from their config table. The remote source reads
the served memory plane, store.Memories().List (listMemories), which the
deployed bd 1.3.0 server advertises as memories.list. httpMemories.List
refuses a page that reports has_more: ListResult has no cursor, so a
truncated plane would pass for the whole one. The memory error now says
"failed to read memories"; nothing in beads or gascity matches the old
text.
TestE2E_ExportOverHTTP seeds two memories with `bd remember` over HTTP.
It requires exactly those two in --all, in --include-memories and in the
local re-export of the imported file, and none in the default export. A
unit case pins the has_more refusal. The CHANGELOG entry says where
remote memories come from, so its "Nothing is dropped" holds. The
WispPlaneIDs doc no longer says the listing refuses an id set, which
stops being true once a server advertises issues.list.ids (#7550).
Attempt 1. The remote route's WispPlaneIDs doc and the CHANGELOG had the
effect of the missing wisp_plane marker backwards. They said import then
routes a no-history row by its flags. A current import does the
opposite: it pins an unmarked no-history row to the durable plane
(applyImportWispPlane). So a genuine no-history wisp in a remote export,
which gc's session, wait and order beads are by default, re-imports onto
the Dolt-versioned issues plane, and gc's JSONL archive keeps it.
The route still stamps nothing. No served read publishes plane
membership: listIssues rows carry no plane field, however the listing is
scoped. Inferring the marker from the flags would re-plane a durable row
that still carries the flag and drop its relations (bd-r9uce), and
refusing the export would re-break gc's remote JSONL export. The doc and
the CHANGELOG now state the real consequence, and serving the membership
is bd-1lbvas. TestE2E_ExportOverHTTP pins it. A no-history row is
exported without the marker by --all and by the default export. A fresh
local workspace imports it onto the durable plane, against a locally
created control row that the local route does mark.
Attempt 1 minors and nits: ReadExport refuses a listing page marked
HasMore, as the uow source does, and unit cases pin that and a failed
comment read. The exportSource interface doc names ExportIssues and its
contract. The export error now says "failed to read the export set"
rather than "failed to search issues"; nothing in beads or gascity
matches the old text. roleExportSource.GetInfraTypes says it is unused
on this route.
Declined, for finalize to file: a bounded retry of ReadExport when a row
vanishes between the reads (a design change; today's fatal behavior is
pinned), and e2e rows for pinned, gate, template and infra-typed issues,
including an --include-infra leg over HTTP. Also declined: a nil-store
guard on roleExportSource.GetInfraTypes. The method is dead on this
route, and the source's other store readers (ExportIssues, GetConfig,
ExportMemories) dereference the store unguarded too.
Validation, attempt 3: git diff --check, the go.mod-pinned gofmt, go
vet, gazelle (no change) and the go_srcs check. The new test passed with
no skips. Three mutants were killed. An empty map with no error on a
failed listing, and the same on an unreachable role, each fail their own
subtest. An empty map returned beside the listing error fails the no-map
assertion; export.go discards the map on error today, so that case pins
the interface's "fail rather than return a partial set" contract, not
visible output.
Validation, attempt 2: git diff --check, gofmt and go vet. The
httpclient memory tests passed, 23 of them: 4 units and 19 served. So
did TestE2E_ExportOverHTTP on embedded Dolt with
BEADS_HTTP_TEST_REQUIRED=1 and no skips. Two mutants were killed: a
remote source that reads the settings listing fails 3 e2e assertions,
and dropping the has_more refusal fails the unit case. Export parity
passed with no skips on the classic (memory determinism, byte
stability), embedded and proxied tiers, including the proxied-versus-
classic byte identity with memories. The PR is now behind main, which
added #7549 and #7550, and conflicts with it textually in CHANGELOG.md,
backend/http/BUILD.bazel and issueops/BUILD.bazel. With those resolved
by union, the merge with main e6a891b900 builds and vets. On that merge,
in CI's pinned-shard layout, the served http tier passed 49 of 49,
including #7550's list-scope e2e. The command corpus ran 128 shapes: 113
pass, 3 allowed, 12 pending and 0 fail. gc-export-all passes without its
pending mark, the one shape that moved. The issueops ReadExport and
ListGates tests passed, as did the embedded and proxied export tiers.
Validation, attempt 1: the issueops ReadExport tests, with two mutants
killed; TestE2E_ExportOverHTTP with a flag-stamping WispPlaneIDs mutant
killed on three assertions; the cmd/bd export tests in a scrubbed env.
Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
203 lines
7.7 KiB
Go
203 lines
7.7 KiB
Go
//go:build cgo
|
|
|
|
package bdhttp_test
|
|
|
|
import (
|
|
"encoding/json"
|
|
"maps"
|
|
"os"
|
|
"path/filepath"
|
|
"strconv"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
type exportLine struct {
|
|
Type string `json:"_type"`
|
|
ID string `json:"id"`
|
|
Ephemeral bool `json:"ephemeral"`
|
|
NoHistory bool `json:"no_history"`
|
|
WispPlane *bool `json:"wisp_plane"`
|
|
Labels []string `json:"labels"`
|
|
CommentCount int `json:"comment_count"`
|
|
DepCount int `json:"dependency_count"`
|
|
Dependencies []struct {
|
|
IssueID string `json:"issue_id"`
|
|
DependsOnID string `json:"depends_on_id"`
|
|
Type string `json:"type"`
|
|
} `json:"dependencies"`
|
|
Comments []struct {
|
|
Text string `json:"text"`
|
|
} `json:"comments"`
|
|
// Key and Value are a memory record's fields.
|
|
Key string `json:"key"`
|
|
Value string `json:"value"`
|
|
}
|
|
|
|
// plane renders the explicit wisps-plane marker: "absent", "true" or "false".
|
|
func (l exportLine) plane() string {
|
|
if l.WispPlane == nil {
|
|
return "absent"
|
|
}
|
|
return strconv.FormatBool(*l.WispPlane)
|
|
}
|
|
|
|
func readExportRecords(t *testing.T, path string) []exportLine {
|
|
t.Helper()
|
|
data, err := os.ReadFile(path)
|
|
if err != nil {
|
|
t.Fatalf("read export: %v", err)
|
|
}
|
|
var out []exportLine
|
|
for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") {
|
|
if line == "" {
|
|
continue
|
|
}
|
|
var rec exportLine
|
|
if err := json.Unmarshal([]byte(line), &rec); err != nil {
|
|
t.Fatalf("parse export line %q: %v", line, err)
|
|
}
|
|
out = append(out, rec)
|
|
}
|
|
return out
|
|
}
|
|
|
|
func readExportLines(t *testing.T, path string) map[string]exportLine {
|
|
t.Helper()
|
|
out := map[string]exportLine{}
|
|
for _, rec := range readExportRecords(t, path) {
|
|
if rec.Type == "issue" {
|
|
out[rec.ID] = rec
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// readExportMemories returns an export's memory records, value by key.
|
|
func readExportMemories(t *testing.T, path string) map[string]string {
|
|
t.Helper()
|
|
out := map[string]string{}
|
|
for _, rec := range readExportRecords(t, path) {
|
|
if rec.Type == "memory" {
|
|
out[rec.Key] = rec.Value
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
func wantExportMemories(t *testing.T, export string, got, want map[string]string) {
|
|
t.Helper()
|
|
if !maps.Equal(got, want) {
|
|
t.Errorf("%s memories = %v, want %v", export, got, want)
|
|
}
|
|
}
|
|
|
|
// TestE2E_ExportOverHTTP is S21. `bd export` read through the raw SearchIssues
|
|
// method and the bulk relation reads, which the http backend refuses, so gc's
|
|
// JSONL-export order failed on every remote scope. It now composes served role
|
|
// reads (issueops.ReadExport): the rows with their labels and counts, every
|
|
// row's dependency records and the comment bodies. Memories come from the
|
|
// served memory plane (listMemories): the settings enumeration withholds every
|
|
// kv.* row, so a remote export that read them there carried none.
|
|
//
|
|
// The no-history row pins the one place this route still differs from the
|
|
// local ones. No served read publishes plane membership, so the row is
|
|
// exported without the wisp_plane marker, and a current import pins it to the
|
|
// durable issues plane: the data-safe side, since a wrong marker would drop a
|
|
// durable row's relations (bd-r9uce). Serving the membership (bd-1lbvas)
|
|
// flips those assertions: the row is then marked and re-imports as a wisp.
|
|
func TestE2E_ExportOverHTTP(t *testing.T) {
|
|
bd := connectE2EWorkspace(t, "worker")
|
|
|
|
blocked := createE2EIssue(t, bd, "blocked work")
|
|
blocker := createE2EIssue(t, bd, "blocker")
|
|
mustRun(t, bd, "dep", "add", blocked, blocker, "--json")
|
|
mustRun(t, bd, "label", "add", blocked, "lane:x", "--json")
|
|
mustRun(t, bd, "comments", "add", blocked, "an export comment")
|
|
closed := createE2EIssue(t, bd, "closed work")
|
|
mustRun(t, bd, "close", closed, "--json")
|
|
wisp := firstJSONField(t, mustRun(t, bd, "create", "a wisp", "--ephemeral", "--json").stdout, "id")
|
|
noHistory := firstJSONField(t, mustRun(t, bd, "create", "a no-history wisp", "--no-history", "--json").stdout, "id")
|
|
memories := map[string]string{
|
|
"export-alpha": "remembered over http",
|
|
"export-beta": "a second memory",
|
|
}
|
|
for key, content := range memories {
|
|
mustRun(t, bd, "remember", content, "--key", key)
|
|
}
|
|
|
|
dir := t.TempDir()
|
|
all := filepath.Join(dir, "all.jsonl")
|
|
mustRun(t, bd, "export", "--all", "-o", all)
|
|
wantExportMemories(t, "bd export --all", readExportMemories(t, all), memories)
|
|
got := readExportLines(t, all)
|
|
for _, id := range []string{blocked, blocker, closed, wisp, noHistory} {
|
|
if _, ok := got[id]; !ok {
|
|
t.Errorf("bd export --all is missing %s; got %d issues", id, len(got))
|
|
}
|
|
}
|
|
rec := got[blocked]
|
|
if len(rec.Dependencies) != 1 || rec.Dependencies[0].DependsOnID != blocker || rec.DepCount != 1 {
|
|
t.Errorf("%s dependencies = %+v (count %d), want one edge onto %s", blocked, rec.Dependencies, rec.DepCount, blocker)
|
|
}
|
|
if len(rec.Labels) != 1 || rec.Labels[0] != "lane:x" {
|
|
t.Errorf("%s labels = %v, want [lane:x]", blocked, rec.Labels)
|
|
}
|
|
if rec.CommentCount != 1 || len(rec.Comments) != 1 || rec.Comments[0].Text != "an export comment" {
|
|
t.Errorf("%s comments = %+v (count %d), want the one comment", blocked, rec.Comments, rec.CommentCount)
|
|
}
|
|
if !got[wisp].Ephemeral {
|
|
t.Errorf("%s exported without ephemeral: %+v", wisp, got[wisp])
|
|
}
|
|
if rec := got[noHistory]; !rec.NoHistory || rec.plane() != "absent" {
|
|
t.Errorf("%s exported as no_history=%v wisp_plane=%s, want no_history with no marker", noHistory, rec.NoHistory, rec.plane())
|
|
}
|
|
|
|
// The default export keeps every status and the no-history row, and drops
|
|
// the true wisp.
|
|
def := filepath.Join(dir, "default.jsonl")
|
|
mustRun(t, bd, "export", "-o", def)
|
|
gotDef := readExportLines(t, def)
|
|
if _, ok := gotDef[wisp]; ok {
|
|
t.Errorf("bd export (default) exported the wisp %s", wisp)
|
|
}
|
|
for _, id := range []string{blocked, blocker, closed, noHistory} {
|
|
if _, ok := gotDef[id]; !ok {
|
|
t.Errorf("bd export (default) is missing %s", id)
|
|
}
|
|
}
|
|
if rec := gotDef[noHistory]; rec.plane() != "absent" {
|
|
t.Errorf("bd export (default) stamped %s wisp_plane=%s, want no marker", noHistory, rec.plane())
|
|
}
|
|
// Memories are opt-in: absent by default, present with --include-memories.
|
|
wantExportMemories(t, "bd export (default)", readExportMemories(t, def), map[string]string{})
|
|
withMemories := filepath.Join(dir, "memories.jsonl")
|
|
mustRun(t, bd, "export", "--include-memories", "-o", withMemories)
|
|
wantExportMemories(t, "bd export --include-memories", readExportMemories(t, withMemories), memories)
|
|
|
|
// Import the remote export into a fresh local workspace and export it
|
|
// again there. The local route stamps wisp_plane from table membership,
|
|
// which the locally created control row proves, so the imported row's
|
|
// missing marker means it sits in the durable issues table.
|
|
bin := buildBD(t)
|
|
localDir := t.TempDir()
|
|
local := func(args ...string) bdResult {
|
|
t.Helper()
|
|
return runBD(t, bin, localDir, []string{"BEADS_ACTOR=worker"}, args...)
|
|
}
|
|
mustRun(t, local, "init", "--prefix", "e2e", "--quiet")
|
|
mustRun(t, local, "import", "-i", all)
|
|
control := firstJSONField(t, mustRun(t, local, "create", "a local no-history wisp", "--no-history", "--json").stdout, "id")
|
|
back := filepath.Join(dir, "reexport.jsonl")
|
|
mustRun(t, local, "export", "--all", "-o", back)
|
|
gotBack := readExportLines(t, back)
|
|
if rec, ok := gotBack[control]; !ok || rec.plane() != "true" {
|
|
t.Fatalf("local control %s exported with wisp_plane=%s (present=%v), want true", control, rec.plane(), ok)
|
|
}
|
|
wantExportMemories(t, "the local re-export", readExportMemories(t, back), memories)
|
|
if rec, ok := gotBack[noHistory]; !ok {
|
|
t.Errorf("the local re-export is missing the imported %s", noHistory)
|
|
} else if !rec.NoHistory || rec.plane() != "absent" {
|
|
t.Errorf("imported %s re-exported as no_history=%v wisp_plane=%s, want no_history on the durable plane (no marker)", noHistory, rec.NoHistory, rec.plane())
|
|
}
|
|
}
|