beads/cmd/bd/export.go
Julian Knutsen e7ad85bcff
fix(http): bd export over http through issueops.ReadExport (S21) (#7551)
* fix(http): bd export over http through issueops.ReadExport (S21)

bd export read through raw SearchIssues and the bulk label, dependency and
comment reads, which the HTTP backend refuses (B3 #16), so gc's JSONL export
order (gc bd export --all) failed on every remote scope.

issueops.ReadExport composes role reads every deployed server publishes: an
unlimited Reader.List for rows, labels and counts, EdgeReader.ReadEdges for
dependency records, and Reader.Get with comments for rows that have any. The
remote route feeds it into the same runExportFromSource the direct and
proxied routes use. A failed read fails the export. gc-export-all passes
against the reference profile; mc-promote stays pending (S21b) with its exact
reason: it needs a new server capability.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(export): read remote memories from the memory plane, pin unmarked no-history rows (bd-o4qlui)

Review fixes for #7551, attempts 1 to 3.

Attempt 3. No test reached roleExportSource.ExportMemories's error
path. A unit test now drives it with a fake store: a memory role that
cannot be reached, and a listMemories that fails, each end the export
with that error and no map. Rewriting either branch to degrade to an
empty map would bring back a remote `bd export --all` that exits 0 with
no memory in it.

Declined in attempt 3, for finalize to file: an unlimited HTTP listing
that stops on a has_more page with no next_cursor returns the rows it
has with HasMore false, so ReadExport's refusal cannot fire over HTTP.
Only a server that breaks the spec's has_more/next_cursor pairing gets
there; this repo's server always sends a cursor with a non-empty
has_more page. The walker (fetchIssuePages) is unchanged since #7247
and shared with bd list, so the fix needs its own PR that checks bd
list's callers first. The remote route also has no single snapshot: a
comment changed between List and Get can make the written count
disagree with the bodies kept. That belongs in one follow-up with the
bounded retry declined in attempt 1, as a count/body consistency check.
The --include-infra e2e leg stays declined. The wire does carry the
knob: the client sends include_infra and include_templates on
listIssues, and the server decodes both. The GetInfraTypes nil-store
guard stays declined, and auto-export's own GetAllConfig memory read
predates this PR.

Attempt 2. A remote export dropped every memory. Over HTTP, `bd export
--all` and `--include-memories` exited 0 without a memory record, where
the base refused with exit 1. The memory lines came from GetAllConfig,
which over HTTP is listSettings, and the server's settings enumeration
withholds every kv.* row, memories included. gc's JSONL export runs
`gc bd export --all`, so its archive lost them without a word.

The exportSource seam now asks each source for its memories
(ExportMemories), not for the whole config table. The local and proxied
sources still take them from their config table. The remote source reads
the served memory plane, store.Memories().List (listMemories), which the
deployed bd 1.3.0 server advertises as memories.list. httpMemories.List
refuses a page that reports has_more: ListResult has no cursor, so a
truncated plane would pass for the whole one. The memory error now says
"failed to read memories"; nothing in beads or gascity matches the old
text.

TestE2E_ExportOverHTTP seeds two memories with `bd remember` over HTTP.
It requires exactly those two in --all, in --include-memories and in the
local re-export of the imported file, and none in the default export. A
unit case pins the has_more refusal. The CHANGELOG entry says where
remote memories come from, so its "Nothing is dropped" holds. The
WispPlaneIDs doc no longer says the listing refuses an id set, which
stops being true once a server advertises issues.list.ids (#7550).

Attempt 1. The remote route's WispPlaneIDs doc and the CHANGELOG had the
effect of the missing wisp_plane marker backwards. They said import then
routes a no-history row by its flags. A current import does the
opposite: it pins an unmarked no-history row to the durable plane
(applyImportWispPlane). So a genuine no-history wisp in a remote export,
which gc's session, wait and order beads are by default, re-imports onto
the Dolt-versioned issues plane, and gc's JSONL archive keeps it.

The route still stamps nothing. No served read publishes plane
membership: listIssues rows carry no plane field, however the listing is
scoped. Inferring the marker from the flags would re-plane a durable row
that still carries the flag and drop its relations (bd-r9uce), and
refusing the export would re-break gc's remote JSONL export. The doc and
the CHANGELOG now state the real consequence, and serving the membership
is bd-1lbvas. TestE2E_ExportOverHTTP pins it. A no-history row is
exported without the marker by --all and by the default export. A fresh
local workspace imports it onto the durable plane, against a locally
created control row that the local route does mark.

Attempt 1 minors and nits: ReadExport refuses a listing page marked
HasMore, as the uow source does, and unit cases pin that and a failed
comment read. The exportSource interface doc names ExportIssues and its
contract. The export error now says "failed to read the export set"
rather than "failed to search issues"; nothing in beads or gascity
matches the old text. roleExportSource.GetInfraTypes says it is unused
on this route.

Declined, for finalize to file: a bounded retry of ReadExport when a row
vanishes between the reads (a design change; today's fatal behavior is
pinned), and e2e rows for pinned, gate, template and infra-typed issues,
including an --include-infra leg over HTTP. Also declined: a nil-store
guard on roleExportSource.GetInfraTypes. The method is dead on this
route, and the source's other store readers (ExportIssues, GetConfig,
ExportMemories) dereference the store unguarded too.

Validation, attempt 3: git diff --check, the go.mod-pinned gofmt, go
vet, gazelle (no change) and the go_srcs check. The new test passed with
no skips. Three mutants were killed. An empty map with no error on a
failed listing, and the same on an unreachable role, each fail their own
subtest. An empty map returned beside the listing error fails the no-map
assertion; export.go discards the map on error today, so that case pins
the interface's "fail rather than return a partial set" contract, not
visible output.

Validation, attempt 2: git diff --check, gofmt and go vet. The
httpclient memory tests passed, 23 of them: 4 units and 19 served. So
did TestE2E_ExportOverHTTP on embedded Dolt with
BEADS_HTTP_TEST_REQUIRED=1 and no skips. Two mutants were killed: a
remote source that reads the settings listing fails 3 e2e assertions,
and dropping the has_more refusal fails the unit case. Export parity
passed with no skips on the classic (memory determinism, byte
stability), embedded and proxied tiers, including the proxied-versus-
classic byte identity with memories. The PR is now behind main, which
added #7549 and #7550, and conflicts with it textually in CHANGELOG.md,
backend/http/BUILD.bazel and issueops/BUILD.bazel. With those resolved
by union, the merge with main e6a891b900 builds and vets. On that merge,
in CI's pinned-shard layout, the served http tier passed 49 of 49,
including #7550's list-scope e2e. The command corpus ran 128 shapes: 113
pass, 3 allowed, 12 pending and 0 fail. gc-export-all passes without its
pending mark, the one shape that moved. The issueops ReadExport and
ListGates tests passed, as did the embedded and proxied export tiers.

Validation, attempt 1: the issueops ReadExport tests, with two mutants
killed; TestE2E_ExportOverHTTP with a flag-stamping WispPlaneIDs mutant
killed on three assertions; the cmd/bd export tests in a scrubbed env.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 03:59:35 +00:00

430 lines
15 KiB
Go

package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"sort"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/atomicfile"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/metrics"
"github.com/steveyegge/beads/internal/storage/domain"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/issueops"
)
var exportCmd = &cobra.Command{
Use: "export",
Short: "Export issues to JSONL format",
Long: `Export all issues to JSONL (newline-delimited JSON) format.
Each line is a complete JSON object representing one issue, including its
labels, dependencies, and comments.
This command is for issue export, migration, and interoperability. It exports
records from the issues table; it is not a full database backup and does not
capture Dolt branches, commit history, working-set state, or non-issue tables.
For supported full backup/restore flows, use 'bd backup init', 'bd backup sync',
and 'bd backup restore'.
By default, exports only regular issues (excluding infrastructure beads
like agents, roles, and messages). Use --all to include everything.
Memories (from 'bd remember') are excluded by default because they may
contain sensitive agent context. Use --include-memories or --all to
include them.
EXAMPLES:
bd export # Export issues to stdout
bd export -o issues.jsonl # Export issues to file
bd export --include-memories # Export issues + memories
bd export --all -o full.jsonl # Include infra + templates + gates + memories
bd export --scrub -o clean.jsonl # Exclude test/pollution records`,
GroupID: "sync",
SilenceUsage: true,
SilenceErrors: true,
RunE: runExport,
}
var (
exportOutput string
exportAll bool
exportIncludeInfra bool
exportScrub bool
exportNoMemories bool
exportIncludeMemories bool
exportExcludeOwners []string
exportVerbose bool
)
func init() {
exportCmd.Flags().StringVarP(&exportOutput, "output", "o", "", "Output file path (default: stdout)")
exportCmd.Flags().BoolVar(&exportAll, "all", false, "Include all records (infra, templates, gates, memories)")
exportCmd.Flags().BoolVar(&exportIncludeInfra, "include-infra", false, "Include infrastructure beads (agents, roles, messages)")
exportCmd.Flags().BoolVar(&exportScrub, "scrub", false, "Exclude test/pollution records")
exportCmd.Flags().BoolVar(&exportIncludeMemories, "include-memories", false, "Include persistent memories (from 'bd remember') in the export")
exportCmd.Flags().BoolVar(&exportNoMemories, "no-memories", false, "Exclude persistent memories (deprecated: now the default)")
_ = exportCmd.Flags().MarkHidden("no-memories")
exportCmd.Flags().StringArrayVar(&exportExcludeOwners, "exclude-owner", nil, "Exclude issues created by this identity (repeatable; also reads export.exclude_owners config)")
exportCmd.Flags().BoolVar(&exportVerbose, "verbose", false, "Print filtered issue count when owners are excluded")
rootCmd.AddCommand(exportCmd)
}
func runExport(cmd *cobra.Command, args []string) error {
evt := metrics.NewCommandEvent("export")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
ctx := rootCtx
if usesProxiedServer() {
if uowProvider == nil {
return HandleErrorRespectJSON("proxied-server UOW provider not initialized")
}
// Run the ENTIRE read set inside one read transaction so the exported
// issues, labels, dependencies, comments, and memories are a single
// consistent snapshot. Export is read-only: RunTxRead never commits
// (the attempt is always rolled back on close).
_, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (struct{}, error) {
return struct{}{}, runExportFromSource(ctx, &uowExportSource{uw: uw})
})
return err
}
if storeIsRemoteBackend(store) {
// A remote backend serves no raw search or bulk relation read; its
// export is composed from served role reads (issueops.ReadExport).
return runExportFromSource(ctx, &roleExportSource{})
}
return runExportFromSource(ctx, storeExportSource{})
}
// runExportFromSource is the whole classic export body with the storage reads
// routed through an exportSource. Everything downstream of the reads —
// filtering, sanitizeZeroTime, record shaping, marshal order, atomic file
// handling, the stderr summary — is shared verbatim between the embedded and
// proxied-server modes, which is what keeps the two outputs byte-identical.
func runExportFromSource(ctx context.Context, src exportSource) error {
// Determine output destination. File output uses atomic writes
// (temp file + rename) so concurrent exports and crashes never
// leave a truncated or interleaved JSONL file.
var w io.Writer
var aw *atomicfile.Writer
if exportOutput != "" {
var err error
aw, err = atomicfile.Create(exportOutput, 0o644)
if err != nil {
return HandleErrorRespectJSON("failed to create output file: %v", err)
}
defer func() {
// Abort is a no-op if Close was already called.
_ = aw.Abort()
}()
w = aw
} else {
w = os.Stdout
}
issues, err := src.ExportIssues(ctx, issueops.ExportRequest{All: exportAll, IncludeInfra: exportIncludeInfra})
if err != nil {
return HandleErrorRespectJSON("failed to read the export set: %v", err)
}
// Scrub test/pollution records if requested
if exportScrub {
issues = filterOutPollution(issues)
}
// Owner-keyed filtering: exclude issues by created_by identity.
// Merges --exclude-owner flag values with export.exclude_owners config.
ownerExcludes := buildOwnerExcludeSet(ctx, src, exportExcludeOwners)
filteredOwnerCount := 0
if len(ownerExcludes) > 0 {
before := len(issues)
issues = filterOutOwners(issues, ownerExcludes)
filteredOwnerCount = before - len(issues)
}
if len(issues) == 0 && exportNoMemories {
if exportOutput != "" {
fmt.Fprintln(os.Stderr, "No issues to export.")
}
return nil
}
// Bulk-load relational data
rel, err := src.LoadExportRelations(ctx, issues)
if err != nil {
return HandleErrorRespectJSON("failed to load relational data: %v", err)
}
// Explicit plane markers (bd-r9uce): a no_history=true row is either an
// unpromoted no-history wisp (wisps table) or a promoted one (durable
// issues-table row still carrying the stray flag) — only table membership
// can tell them apart, and import routes by the "wisp_plane" marker
// stamped here. Ephemeral rows are unambiguous (ephemeral=true only ever lives in
// the wisps table) and are deliberately NOT stamped, keeping their export
// bytes unchanged.
var noHistoryIDs []string
for _, issue := range issues {
if issue.NoHistory && !issue.Ephemeral {
noHistoryIDs = append(noHistoryIDs, issue.ID)
}
}
wispPlane := map[string]bool{}
if len(noHistoryIDs) > 0 {
wispPlane, err = src.WispPlaneIDs(ctx, noHistoryIDs)
if err != nil {
return HandleErrorRespectJSON("failed to classify wisp-plane rows: %v", err)
}
}
// Populate relational data on each issue
for _, issue := range issues {
issue.Labels = rel.labels[issue.ID]
issue.Dependencies = rel.deps[issue.ID]
issue.Comments = rel.comments[issue.ID]
}
// Write JSONL: one JSON object per line
count := 0
for _, issue := range issues {
counts := rel.depCounts[issue.ID]
if counts == nil {
counts = &types.DependencyCounts{}
}
// Sanitize zero-value timestamps that can't be marshaled to JSON.
// NULL datetime columns scanned as time.Time{} (year 0001) cause
// MarshalJSON to fail with "year outside of range [0,9999]". (GH#2488)
sanitizeZeroTime(issue)
record := &exportIssueRecord{
RecordType: "issue",
IssueWithCounts: &types.IssueWithCounts{
Issue: issue,
DependencyCount: counts.DependencyCount,
DependentCount: counts.DependentCount,
CommentCount: rel.commentCounts[issue.ID],
},
WispPlane: wispPlane[issue.ID],
}
data, err := json.Marshal(record)
if err != nil {
return HandleErrorRespectJSON("failed to marshal issue %s: %v", issue.ID, err)
}
if _, err := w.Write(data); err != nil {
return HandleErrorRespectJSON("failed to write: %v", err)
}
if _, err := w.Write([]byte{'\n'}); err != nil {
return HandleErrorRespectJSON("failed to write newline: %v", err)
}
count++
}
// Export memories only when explicitly requested (GH#3650).
// Memories may contain sensitive agent context and are excluded by default.
memoryCount := 0
if (exportIncludeMemories || exportAll) && !exportNoMemories {
memories, err := src.ExportMemories(ctx)
if err != nil {
return HandleErrorRespectJSON("failed to read memories: %v", err)
}
// Sort keys for deterministic output order (GH#3474).
memKeys := make([]string, 0, len(memories))
for userKey := range memories {
memKeys = append(memKeys, userKey)
}
sort.Strings(memKeys)
for _, userKey := range memKeys {
record := map[string]string{
"_type": "memory",
"key": userKey,
"value": memories[userKey],
}
data, err := json.Marshal(record)
if err != nil {
return HandleErrorRespectJSON("failed to marshal memory %s: %v", userKey, err)
}
if _, err := w.Write(data); err != nil {
return HandleErrorRespectJSON("failed to write: %v", err)
}
if _, err := w.Write([]byte{'\n'}); err != nil {
return HandleErrorRespectJSON("failed to write newline: %v", err)
}
memoryCount++
}
}
// Finalize atomic write if writing to file (fsync + rename).
if aw != nil {
if err := aw.Close(); err != nil {
return HandleErrorRespectJSON("failed to finalize export file: %v", err)
}
}
// Print summary to stderr (not stdout, to avoid mixing with JSONL)
if exportOutput != "" {
if memoryCount > 0 {
fmt.Fprintf(os.Stderr, "Exported %d issues and %d memories to %s\n", count, memoryCount, exportOutput)
} else {
fmt.Fprintf(os.Stderr, "Exported %d issues to %s\n", count, exportOutput)
}
if exportVerbose && filteredOwnerCount > 0 {
fmt.Fprintf(os.Stderr, " (%d filtered as personal by owner exclusion)\n", filteredOwnerCount)
}
}
return nil
}
// exportIssueFilter is the storage filter the store-shaped export sources
// search with: every status, infra types excluded unless --all or
// --include-infra, templates and true wisps excluded unless --all. The
// role-backed source asks the same question through issueops.ReadExport.
func exportIssueFilter(ctx context.Context, src exportSource, req issueops.ExportRequest) types.IssueFilter {
// Build filter for issues table. Export all statuses by default.
// Opt out of BEADS_MAX_ROWS (designer §4.1) — export is a data-integrity
// path and must never abort partway through an export run.
filter := types.IssueFilter{
Limit: 0,
MaxRows: 0,
MaxRowsSource: "",
}
// Exclude infra types by default (agents, roles, messages).
if !req.All && !req.IncludeInfra {
var infraTypes []string
infraSet := src.GetInfraTypes(ctx)
if len(infraSet) > 0 {
for t := range infraSet {
infraTypes = append(infraTypes, t)
}
}
if len(infraTypes) == 0 {
infraTypes = domain.DefaultInfraTypes()
}
for _, t := range infraTypes {
filter.ExcludeTypes = append(filter.ExcludeTypes, types.IssueType(t))
}
}
// Exclude templates by default
if !req.All {
isTemplate := false
filter.IsTemplate = &isTemplate
}
// Exclude ephemeral wisps by default — they are private/transient and
// must not reach git history or external integrations (GH#3649).
// --all overrides to include everything.
if !req.All {
persistentOnly := false
filter.Ephemeral = &persistentOnly
}
return filter
}
// exportIssueRecord wraps IssueWithCounts with a _type discriminator so that
// every line in the JSONL export is self-describing. Memory lines already
// carry "_type":"memory"; this gives issue lines "_type":"issue". (GH#3271)
type exportIssueRecord struct {
RecordType string `json:"_type"`
*types.IssueWithCounts
// WispPlane is the explicit wisps-plane marker (bd-r9uce): true when the
// row lives in the WISPS table AND its flags alone cannot prove it (the
// no_history shape; ephemeral rows are self-describing and stay
// unstamped). Import routes by this marker — never by no_history — so a
// promoted no-history wisp (durable issues-table row still carrying the
// stray flag) round-trips to the durable plane instead of being silently
// re-planed. Declared after the embedded struct so it serializes last.
// Deliberately a FRESH key, not the legacy "wisp" alias key: pre-fix
// binaries' alias branch would import a marked no-history wisp as
// ephemeral (purge-eligible, export-excluded), so an unknown-to-them key
// that degrades to flag routing is the data-safe choice (lion, #5368).
WispPlane bool `json:"wisp_plane,omitempty"`
}
// sanitizeZeroTime replaces Go zero-value time.Time fields with Unix epoch.
// NULL datetime columns in Dolt scan as time.Time{} (year 0001-01-01), which
// causes json.Marshal to fail with "year outside of range [0,9999]". (GH#2488)
func sanitizeZeroTime(issue *types.Issue) {
epoch := time.Unix(0, 0).UTC()
if issue.CreatedAt.IsZero() {
issue.CreatedAt = epoch
}
if issue.UpdatedAt.IsZero() {
issue.UpdatedAt = epoch
}
}
// filterOutPollution removes issues that look like test/pollution records.
func filterOutPollution(issues []*types.Issue) []*types.Issue {
var clean []*types.Issue
for _, issue := range issues {
if !isTestIssue(issue.Title) {
clean = append(clean, issue)
}
}
return clean
}
// buildOwnerExcludeSet merges --exclude-owner flag values with the
// export.exclude_owners (and legacy export.exclude_owner) config entries.
// Returns the combined set as a map for O(1) lookup.
func buildOwnerExcludeSet(ctx context.Context, src exportSource, flagOwners []string) map[string]struct{} {
set := make(map[string]struct{})
for _, o := range flagOwners {
if o != "" {
set[o] = struct{}{}
}
}
// export.* keys are YAML-only (config.IsYamlOnlyKey returns true for the
// "export." prefix), so bd config set stores them in config.yaml rather than
// the database. Read from YAML first, then fall back to the database for any
// instance that was written directly to the store.
addOwners := func(val string) {
for _, o := range strings.Split(val, ",") {
if o = strings.TrimSpace(o); o != "" {
set[o] = struct{}{}
}
}
}
if val := config.GetYamlConfig("export.exclude_owners"); val != "" {
addOwners(val)
}
if val := config.GetYamlConfig("export.exclude_owner"); val != "" {
set[strings.TrimSpace(val)] = struct{}{}
}
// Also read from database for any value stored there directly.
if val, err := src.GetConfig(ctx, "export.exclude_owners"); err == nil && val != "" {
addOwners(val)
}
if val, err := src.GetConfig(ctx, "export.exclude_owner"); err == nil && val != "" {
set[strings.TrimSpace(val)] = struct{}{}
}
return set
}
// filterOutOwners removes issues whose created_by identity is in the exclude set.
func filterOutOwners(issues []*types.Issue, exclude map[string]struct{}) []*types.Issue {
var keep []*types.Issue
for _, issue := range issues {
if _, excluded := exclude[issue.CreatedBy]; !excluded {
keep = append(keep, issue)
}
}
return keep
}