beads/issueops/export.go
Julian Knutsen e7ad85bcff
fix(http): bd export over http through issueops.ReadExport (S21) (#7551)
* fix(http): bd export over http through issueops.ReadExport (S21)

bd export read through raw SearchIssues and the bulk label, dependency and
comment reads, which the HTTP backend refuses (B3 #16), so gc's JSONL export
order (gc bd export --all) failed on every remote scope.

issueops.ReadExport composes role reads every deployed server publishes: an
unlimited Reader.List for rows, labels and counts, EdgeReader.ReadEdges for
dependency records, and Reader.Get with comments for rows that have any. The
remote route feeds it into the same runExportFromSource the direct and
proxied routes use. A failed read fails the export. gc-export-all passes
against the reference profile; mc-promote stays pending (S21b) with its exact
reason: it needs a new server capability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(export): read remote memories from the memory plane, pin unmarked no-history rows (bd-o4qlui)

Review fixes for #7551, attempts 1 to 3.

Attempt 3. No test reached roleExportSource.ExportMemories's error
path. A unit test now drives it with a fake store: a memory role that
cannot be reached, and a listMemories that fails, each end the export
with that error and no map. Rewriting either branch to degrade to an
empty map would bring back a remote `bd export --all` that exits 0 with
no memory in it.

Declined in attempt 3, for finalize to file: an unlimited HTTP listing
that stops on a has_more page with no next_cursor returns the rows it
has with HasMore false, so ReadExport's refusal cannot fire over HTTP.
Only a server that breaks the spec's has_more/next_cursor pairing gets
there; this repo's server always sends a cursor with a non-empty
has_more page. The walker (fetchIssuePages) is unchanged since #7247
and shared with bd list, so the fix needs its own PR that checks bd
list's callers first. The remote route also has no single snapshot: a
comment changed between List and Get can make the written count
disagree with the bodies kept. That belongs in one follow-up with the
bounded retry declined in attempt 1, as a count/body consistency check.
The --include-infra e2e leg stays declined. The wire does carry the
knob: the client sends include_infra and include_templates on
listIssues, and the server decodes both. The GetInfraTypes nil-store
guard stays declined, and auto-export's own GetAllConfig memory read
predates this PR.

Attempt 2. A remote export dropped every memory. Over HTTP, `bd export
--all` and `--include-memories` exited 0 without a memory record, where
the base refused with exit 1. The memory lines came from GetAllConfig,
which over HTTP is listSettings, and the server's settings enumeration
withholds every kv.* row, memories included. gc's JSONL export runs
`gc bd export --all`, so its archive lost them without a word.

The exportSource seam now asks each source for its memories
(ExportMemories), not for the whole config table. The local and proxied
sources still take them from their config table. The remote source reads
the served memory plane, store.Memories().List (listMemories), which the
deployed bd 1.3.0 server advertises as memories.list. httpMemories.List
refuses a page that reports has_more: ListResult has no cursor, so a
truncated plane would pass for the whole one. The memory error now says
"failed to read memories"; nothing in beads or gascity matches the old
text.

TestE2E_ExportOverHTTP seeds two memories with `bd remember` over HTTP.
It requires exactly those two in --all, in --include-memories and in the
local re-export of the imported file, and none in the default export. A
unit case pins the has_more refusal. The CHANGELOG entry says where
remote memories come from, so its "Nothing is dropped" holds. The
WispPlaneIDs doc no longer says the listing refuses an id set, which
stops being true once a server advertises issues.list.ids (#7550).

Attempt 1. The remote route's WispPlaneIDs doc and the CHANGELOG had the
effect of the missing wisp_plane marker backwards. They said import then
routes a no-history row by its flags. A current import does the
opposite: it pins an unmarked no-history row to the durable plane
(applyImportWispPlane). So a genuine no-history wisp in a remote export,
which gc's session, wait and order beads are by default, re-imports onto
the Dolt-versioned issues plane, and gc's JSONL archive keeps it.

The route still stamps nothing. No served read publishes plane
membership: listIssues rows carry no plane field, however the listing is
scoped. Inferring the marker from the flags would re-plane a durable row
that still carries the flag and drop its relations (bd-r9uce), and
refusing the export would re-break gc's remote JSONL export. The doc and
the CHANGELOG now state the real consequence, and serving the membership
is bd-1lbvas. TestE2E_ExportOverHTTP pins it. A no-history row is
exported without the marker by --all and by the default export. A fresh
local workspace imports it onto the durable plane, against a locally
created control row that the local route does mark.

Attempt 1 minors and nits: ReadExport refuses a listing page marked
HasMore, as the uow source does, and unit cases pin that and a failed
comment read. The exportSource interface doc names ExportIssues and its
contract. The export error now says "failed to read the export set"
rather than "failed to search issues"; nothing in beads or gascity
matches the old text. roleExportSource.GetInfraTypes says it is unused
on this route.

Declined, for finalize to file: a bounded retry of ReadExport when a row
vanishes between the reads (a design change; today's fatal behavior is
pinned), and e2e rows for pinned, gate, template and infra-typed issues,
including an --include-infra leg over HTTP. Also declined: a nil-store
guard on roleExportSource.GetInfraTypes. The method is dead on this
route, and the source's other store readers (ExportIssues, GetConfig,
ExportMemories) dereference the store unguarded too.

Validation, attempt 3: git diff --check, the go.mod-pinned gofmt, go
vet, gazelle (no change) and the go_srcs check. The new test passed with
no skips. Three mutants were killed. An empty map with no error on a
failed listing, and the same on an unreachable role, each fail their own
subtest. An empty map returned beside the listing error fails the no-map
assertion; export.go discards the map on error today, so that case pins
the interface's "fail rather than return a partial set" contract, not
visible output.

Validation, attempt 2: git diff --check, gofmt and go vet. The
httpclient memory tests passed, 23 of them: 4 units and 19 served. So
did TestE2E_ExportOverHTTP on embedded Dolt with
BEADS_HTTP_TEST_REQUIRED=1 and no skips. Two mutants were killed: a
remote source that reads the settings listing fails 3 e2e assertions,
and dropping the has_more refusal fails the unit case. Export parity
passed with no skips on the classic (memory determinism, byte
stability), embedded and proxied tiers, including the proxied-versus-
classic byte identity with memories. The PR is now behind main, which
added #7549 and #7550, and conflicts with it textually in CHANGELOG.md,
backend/http/BUILD.bazel and issueops/BUILD.bazel. With those resolved
by union, the merge with main e6a891b900 builds and vets. On that merge,
in CI's pinned-shard layout, the served http tier passed 49 of 49,
including #7550's list-scope e2e. The command corpus ran 128 shapes: 113
pass, 3 allowed, 12 pending and 0 fail. gc-export-all passes without its
pending mark, the one shape that moved. The issueops ReadExport and
ListGates tests passed, as did the embedded and proxied export tiers.

Validation, attempt 1: the issueops ReadExport tests, with two mutants
killed; TestE2E_ExportOverHTTP with a flag-stamping WispPlaneIDs mutant
killed on three assertions; the cmd/bd export tests in a scrubbed env.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 03:59:35 +00:00

130 lines
4.5 KiB
Go

package issueops
import (
"context"
"errors"
"fmt"
"github.com/steveyegge/beads/internal/types"
)
// ExportRequest describes which issues `bd export` writes.
type ExportRequest struct {
// All exports every row: every status, every type (infra, templates,
// gates), and both planes, wisps included.
All bool
// IncludeInfra admits the workspace's infrastructure types (agents, roles,
// messages by default) without lifting the other default exclusions.
IncludeInfra bool
}
// ExportRow is one exported issue, fully hydrated: its labels, its stored
// outgoing dependency records and its comment bodies, plus the three
// cardinalities the export line carries.
type ExportRow struct {
Issue *types.Issue
DependencyCount int
DependentCount int
CommentCount int
}
// ReadExport answers `bd export`'s read set by composing three role reads that
// every backend serves (a remote one as listIssues, listDependencies and
// getIssue):
//
// - Reader.List, unlimited, for the rows, their labels and their counts;
// - EdgeReader.ReadEdges, for every row's stored dependency records;
// - Reader.Get with IncludeComments, for the comment bodies of each row whose
// comment count is nonzero (a row with no comments costs no read).
//
// THE SELECTION IS EXPORT'S, not the listing's. Every status is exported
// (AllFlag), and gates are always exported. Without All, templates and the
// workspace's infra types are excluded (IncludeInfra admits the latter), and
// TRUE wisps are excluded while no-history rows, which live on the wisps plane
// with ephemeral unset, are kept (GH#3649). A listing cannot say "the wisps
// plane minus its ephemeral rows", so the plane is admitted and the ephemeral
// rows are dropped here. That is exact because the listing is unlimited: no
// row is dropped before a limit could have counted it.
//
// A FAILED READ IS AN ERROR. Export is a data-integrity path, so a refusal or
// a missing relation never degrades to a shorter export. An anchor the edge
// read reports missing (a row deleted between the reads) is an error too, and
// so is a listing page marked HasMore: the listing is unlimited, so a
// truncated page means the reader did not honor that.
func ReadExport(ctx context.Context, rd Reader, edges EdgeReader, req ExportRequest) ([]ExportRow, error) {
if rd == nil || edges == nil {
return nil, &ErrUnsupported{Op: "ReadExport", Backend: "nil"}
}
unlimited := 0
list := ListRequest{
AllFlag: true,
IncludeGates: true,
IncludeEphemeral: true,
Limit: &unlimited,
}
// --all is EXPORT's scope, spelled as export defines it (infra,
// templates, gates and wisps), not ListRequest.IncludeAllTypes: that
// union is the listing's "hide nothing" contract, which a remote reader
// refuses to restate, and export has always named its own four.
list.IncludeInfra = req.All || req.IncludeInfra
list.IncludeTemplates = req.All
page, err := rd.List(ctx, list)
if err != nil {
return nil, fmt.Errorf("export: listing issues: %w", err)
}
if page.HasMore {
return nil, errors.New("export: listing issues: the listing returned a truncated page; export must never truncate")
}
rows := make([]ExportRow, 0, len(page.Items))
ids := make([]string, 0, len(page.Items))
for _, item := range page.Items {
if item == nil || item.Issue == nil {
continue
}
if !req.All && item.Ephemeral {
continue
}
rows = append(rows, ExportRow{
Issue: item.Issue,
DependencyCount: item.DependencyCount,
DependentCount: item.DependentCount,
CommentCount: item.CommentCount,
})
ids = append(ids, item.ID)
}
if len(rows) == 0 {
return rows, nil
}
edgeResult, err := edges.ReadEdges(ctx, EdgeReadRequest{IDs: ids})
if err != nil {
return nil, fmt.Errorf("export: reading dependencies: %w", err)
}
deps := make(map[string][]*types.Dependency, len(edgeResult.Anchors))
for _, anchor := range edgeResult.Anchors {
if anchor.Missing {
return nil, fmt.Errorf("export: reading dependencies: %s is no longer in the store", anchor.ID)
}
if len(anchor.Edges) > 0 {
deps[anchor.ID] = anchor.Edges
}
}
for i := range rows {
issue := rows[i].Issue
issue.Dependencies = deps[issue.ID]
issue.Comments = nil
if rows[i].CommentCount == 0 {
continue
}
detail, err := rd.Get(ctx, GetRequest{ID: issue.ID, IncludeComments: true})
if err != nil {
return nil, fmt.Errorf("export: reading comments of %s: %w", issue.ID, err)
}
if detail != nil {
issue.Comments = detail.Comments
}
}
return rows, nil
}