* fix(export): promoted no-history wisps survive import round-trip (bd-r9uce) A promoted no-history wisp is a durable issues-table row that still carried no_history=true (PromoteFromEphemeralInTx cleared only Ephemeral), and both import routes inferred the storage plane from row flags (issueops.IsWisp: Ephemeral || NoHistory). Importing an export stream containing such a row silently re-planed it into the wisps table, dropping its cross-plane relations as cross-bucket dependencies on the way — classic-mode data loss across export->import->export. Belt and braces, per triage + the #5351 review endorsement: 1. Promote clears NoHistory too (and normalizes an explicit ephemeral StorageClass to empty, the NormalizePersistenceMode promotion rule): a promoted wisp is fully durable and the flag has no post-promotion meaning. 2. Import routes the plane by an explicit marker, never by no_history. Export stamps "wisp": true on rows that live in the WISPS table and whose flags alone cannot prove it (the no_history shape; ephemeral rows self-describe), classified by table membership via a new PartitionWispIDs store capability (classic: batched wisps-membership query behind an Unwrap walk through the telemetry/hook decorators; proxied: the plane-pinned GetWispsByIDs read). Both parse loops (bd import in both storage modes, bootstrap/init/auto-import) resolve the plane through one shared helper, applyImportWispPlane, which pins marker-less no_history records to the durable plane via a new non-serialized Issue.WispPlaneOverride consulted by IsWisp. The no_history flag itself is preserved on the row, keeping the export->import->export byte-identity for wild data; the legacy v0.35-v0.37 "wisp"-as-ephemeral alias behavior is unchanged. Tests: the promoted shape now lives IN the proxied round-trip fixture (cross_mode_byte_identical covers it, with relations); a classic-mode round-trip test (fails on base: cross-bucket dependency drop + missing plane marker); a proxied import plane-routing subtest; embeddeddolt pins for promote clearing NoHistory and PartitionWispIDs membership classification. * docs(schema): document the export stream's explicit wisp plane marker (bd-r9uce) * fix(export): fresh wisp_plane marker key — legacy-alias reuse was lossy under rollout skew (bd-r9uce) Adopts lion's #5368 review recommendation: pre-fix v0.38+ binaries' alias branch (hasWisp && !Ephemeral => Ephemeral=true) would import a marked genuine no-history wisp as ephemeral — purge-eligible and excluded from that rig's next export — making the common skew case lossy. A fresh key degrades to flag routing on older readers, data-safe in both directions. Legacy 'wisp' read alias preserved verbatim. * fix(storage): clone WispPlaneOverride in cloneIssueForHook (bd-r9uce) The hook snapshot guard (TestCloneIssueForHookCoversReferenceFields) correctly flagged the new *bool field as uncloned: a hook mutating the snapshot could alias the live issue's plane pin. Deep-copy it and add it to the guard's covered-fields manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FZXjihTayFcngYzo6xPd3p --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
421 lines
14 KiB
Go
421 lines
14 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"os"
|
|
"sort"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/spf13/cobra"
|
|
"github.com/steveyegge/beads/internal/atomicfile"
|
|
"github.com/steveyegge/beads/internal/config"
|
|
"github.com/steveyegge/beads/internal/metrics"
|
|
"github.com/steveyegge/beads/internal/storage/domain"
|
|
"github.com/steveyegge/beads/internal/storage/uow"
|
|
"github.com/steveyegge/beads/internal/types"
|
|
)
|
|
|
|
var exportCmd = &cobra.Command{
|
|
Use: "export",
|
|
Short: "Export issues to JSONL format",
|
|
Long: `Export all issues to JSONL (newline-delimited JSON) format.
|
|
|
|
Each line is a complete JSON object representing one issue, including its
|
|
labels, dependencies, and comments.
|
|
|
|
This command is for issue export, migration, and interoperability. It exports
|
|
records from the issues table; it is not a full database backup and does not
|
|
capture Dolt branches, commit history, working-set state, or non-issue tables.
|
|
For supported full backup/restore flows, use 'bd backup init', 'bd backup sync',
|
|
and 'bd backup restore'.
|
|
|
|
By default, exports only regular issues (excluding infrastructure beads
|
|
like agents, roles, and messages). Use --all to include everything.
|
|
|
|
Memories (from 'bd remember') are excluded by default because they may
|
|
contain sensitive agent context. Use --include-memories or --all to
|
|
include them.
|
|
|
|
EXAMPLES:
|
|
bd export # Export issues to stdout
|
|
bd export -o issues.jsonl # Export issues to file
|
|
bd export --include-memories # Export issues + memories
|
|
bd export --all -o full.jsonl # Include infra + templates + gates + memories
|
|
bd export --scrub -o clean.jsonl # Exclude test/pollution records`,
|
|
GroupID: "sync",
|
|
SilenceUsage: true,
|
|
SilenceErrors: true,
|
|
RunE: runExport,
|
|
}
|
|
|
|
var (
|
|
exportOutput string
|
|
exportAll bool
|
|
exportIncludeInfra bool
|
|
exportScrub bool
|
|
exportNoMemories bool
|
|
exportIncludeMemories bool
|
|
exportExcludeOwners []string
|
|
exportVerbose bool
|
|
)
|
|
|
|
func init() {
|
|
exportCmd.Flags().StringVarP(&exportOutput, "output", "o", "", "Output file path (default: stdout)")
|
|
exportCmd.Flags().BoolVar(&exportAll, "all", false, "Include all records (infra, templates, gates, memories)")
|
|
exportCmd.Flags().BoolVar(&exportIncludeInfra, "include-infra", false, "Include infrastructure beads (agents, roles, messages)")
|
|
exportCmd.Flags().BoolVar(&exportScrub, "scrub", false, "Exclude test/pollution records")
|
|
exportCmd.Flags().BoolVar(&exportIncludeMemories, "include-memories", false, "Include persistent memories (from 'bd remember') in the export")
|
|
exportCmd.Flags().BoolVar(&exportNoMemories, "no-memories", false, "Exclude persistent memories (deprecated: now the default)")
|
|
_ = exportCmd.Flags().MarkHidden("no-memories")
|
|
exportCmd.Flags().StringArrayVar(&exportExcludeOwners, "exclude-owner", nil, "Exclude issues created by this identity (repeatable; also reads export.exclude_owners config)")
|
|
exportCmd.Flags().BoolVar(&exportVerbose, "verbose", false, "Print filtered issue count when owners are excluded")
|
|
rootCmd.AddCommand(exportCmd)
|
|
}
|
|
|
|
func runExport(cmd *cobra.Command, args []string) error {
|
|
evt := metrics.NewCommandEvent("export")
|
|
defer func() {
|
|
if c := metrics.Global(); c != nil {
|
|
c.CloseEventAndAdd(evt)
|
|
}
|
|
}()
|
|
|
|
ctx := rootCtx
|
|
|
|
if usesProxiedServer() {
|
|
if uowProvider == nil {
|
|
return HandleErrorRespectJSON("proxied-server UOW provider not initialized")
|
|
}
|
|
// Run the ENTIRE read set inside one read transaction so the exported
|
|
// issues, labels, dependencies, comments, and memories are a single
|
|
// consistent snapshot. Export is read-only: RunTxRead never commits
|
|
// (the attempt is always rolled back on close).
|
|
_, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (struct{}, error) {
|
|
return struct{}{}, runExportFromSource(ctx, &uowExportSource{uw: uw})
|
|
})
|
|
return err
|
|
}
|
|
|
|
return runExportFromSource(ctx, storeExportSource{})
|
|
}
|
|
|
|
// runExportFromSource is the whole classic export body with the storage reads
|
|
// routed through an exportSource. Everything downstream of the reads —
|
|
// filtering, sanitizeZeroTime, record shaping, marshal order, atomic file
|
|
// handling, the stderr summary — is shared verbatim between the embedded and
|
|
// proxied-server modes, which is what keeps the two outputs byte-identical.
|
|
func runExportFromSource(ctx context.Context, src exportSource) error {
|
|
// Determine output destination. File output uses atomic writes
|
|
// (temp file + rename) so concurrent exports and crashes never
|
|
// leave a truncated or interleaved JSONL file.
|
|
var w io.Writer
|
|
var aw *atomicfile.Writer
|
|
if exportOutput != "" {
|
|
var err error
|
|
aw, err = atomicfile.Create(exportOutput, 0o644)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to create output file: %v", err)
|
|
}
|
|
defer func() {
|
|
// Abort is a no-op if Close was already called.
|
|
_ = aw.Abort()
|
|
}()
|
|
w = aw
|
|
} else {
|
|
w = os.Stdout
|
|
}
|
|
|
|
// Build filter for issues table. Export all statuses by default.
|
|
// Opt out of BEADS_MAX_ROWS (designer §4.1) — export is a data-integrity
|
|
// path and must never abort partway through an export run.
|
|
filter := types.IssueFilter{
|
|
Limit: 0,
|
|
MaxRows: 0,
|
|
MaxRowsSource: "",
|
|
}
|
|
|
|
// Exclude infra types by default (agents, roles, messages).
|
|
if !exportAll && !exportIncludeInfra {
|
|
var infraTypes []string
|
|
infraSet := src.GetInfraTypes(ctx)
|
|
if len(infraSet) > 0 {
|
|
for t := range infraSet {
|
|
infraTypes = append(infraTypes, t)
|
|
}
|
|
}
|
|
if len(infraTypes) == 0 {
|
|
infraTypes = domain.DefaultInfraTypes()
|
|
}
|
|
for _, t := range infraTypes {
|
|
filter.ExcludeTypes = append(filter.ExcludeTypes, types.IssueType(t))
|
|
}
|
|
}
|
|
|
|
// Exclude templates by default
|
|
if !exportAll {
|
|
isTemplate := false
|
|
filter.IsTemplate = &isTemplate
|
|
}
|
|
|
|
// Exclude ephemeral wisps by default — they are private/transient and
|
|
// must not reach git history or external integrations (GH#3649).
|
|
// --all overrides to include everything.
|
|
if !exportAll {
|
|
persistentOnly := false
|
|
filter.Ephemeral = &persistentOnly
|
|
}
|
|
|
|
issues, err := src.SearchIssues(ctx, "", filter)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to search issues: %v", err)
|
|
}
|
|
|
|
// Scrub test/pollution records if requested
|
|
if exportScrub {
|
|
issues = filterOutPollution(issues)
|
|
}
|
|
|
|
// Owner-keyed filtering: exclude issues by created_by identity.
|
|
// Merges --exclude-owner flag values with export.exclude_owners config.
|
|
ownerExcludes := buildOwnerExcludeSet(ctx, src, exportExcludeOwners)
|
|
filteredOwnerCount := 0
|
|
if len(ownerExcludes) > 0 {
|
|
before := len(issues)
|
|
issues = filterOutOwners(issues, ownerExcludes)
|
|
filteredOwnerCount = before - len(issues)
|
|
}
|
|
|
|
if len(issues) == 0 && exportNoMemories {
|
|
if exportOutput != "" {
|
|
fmt.Fprintln(os.Stderr, "No issues to export.")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Bulk-load relational data
|
|
rel, err := src.LoadExportRelations(ctx, issues)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to load relational data: %v", err)
|
|
}
|
|
|
|
// Explicit plane markers (bd-r9uce): a no_history=true row is either an
|
|
// unpromoted no-history wisp (wisps table) or a promoted one (durable
|
|
// issues-table row still carrying the stray flag) — only table membership
|
|
// can tell them apart, and import routes by the "wisp_plane" marker
|
|
// stamped here. Ephemeral rows are unambiguous (ephemeral=true only ever lives in
|
|
// the wisps table) and are deliberately NOT stamped, keeping their export
|
|
// bytes unchanged.
|
|
var noHistoryIDs []string
|
|
for _, issue := range issues {
|
|
if issue.NoHistory && !issue.Ephemeral {
|
|
noHistoryIDs = append(noHistoryIDs, issue.ID)
|
|
}
|
|
}
|
|
wispPlane := map[string]bool{}
|
|
if len(noHistoryIDs) > 0 {
|
|
wispPlane, err = src.WispPlaneIDs(ctx, noHistoryIDs)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to classify wisp-plane rows: %v", err)
|
|
}
|
|
}
|
|
|
|
// Populate relational data on each issue
|
|
for _, issue := range issues {
|
|
issue.Labels = rel.labels[issue.ID]
|
|
issue.Dependencies = rel.deps[issue.ID]
|
|
issue.Comments = rel.comments[issue.ID]
|
|
}
|
|
|
|
// Write JSONL: one JSON object per line
|
|
count := 0
|
|
for _, issue := range issues {
|
|
counts := rel.depCounts[issue.ID]
|
|
if counts == nil {
|
|
counts = &types.DependencyCounts{}
|
|
}
|
|
|
|
// Sanitize zero-value timestamps that can't be marshaled to JSON.
|
|
// NULL datetime columns scanned as time.Time{} (year 0001) cause
|
|
// MarshalJSON to fail with "year outside of range [0,9999]". (GH#2488)
|
|
sanitizeZeroTime(issue)
|
|
|
|
record := &exportIssueRecord{
|
|
RecordType: "issue",
|
|
IssueWithCounts: &types.IssueWithCounts{
|
|
Issue: issue,
|
|
DependencyCount: counts.DependencyCount,
|
|
DependentCount: counts.DependentCount,
|
|
CommentCount: rel.commentCounts[issue.ID],
|
|
},
|
|
WispPlane: wispPlane[issue.ID],
|
|
}
|
|
|
|
data, err := json.Marshal(record)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to marshal issue %s: %v", issue.ID, err)
|
|
}
|
|
if _, err := w.Write(data); err != nil {
|
|
return HandleErrorRespectJSON("failed to write: %v", err)
|
|
}
|
|
if _, err := w.Write([]byte{'\n'}); err != nil {
|
|
return HandleErrorRespectJSON("failed to write newline: %v", err)
|
|
}
|
|
count++
|
|
}
|
|
|
|
// Export memories only when explicitly requested (GH#3650).
|
|
// Memories may contain sensitive agent context and are excluded by default.
|
|
memoryCount := 0
|
|
if (exportIncludeMemories || exportAll) && !exportNoMemories {
|
|
allConfig, err := src.GetAllConfig(ctx)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to read config for memories: %v", err)
|
|
}
|
|
fullPrefix := kvPrefix + memoryPrefix
|
|
// Sort keys for deterministic output order (GH#3474).
|
|
var memKeys []string
|
|
for k := range allConfig {
|
|
if strings.HasPrefix(k, fullPrefix) {
|
|
memKeys = append(memKeys, k)
|
|
}
|
|
}
|
|
sort.Strings(memKeys)
|
|
for _, k := range memKeys {
|
|
v := allConfig[k]
|
|
userKey := strings.TrimPrefix(k, fullPrefix)
|
|
record := map[string]string{
|
|
"_type": "memory",
|
|
"key": userKey,
|
|
"value": v,
|
|
}
|
|
data, err := json.Marshal(record)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("failed to marshal memory %s: %v", userKey, err)
|
|
}
|
|
if _, err := w.Write(data); err != nil {
|
|
return HandleErrorRespectJSON("failed to write: %v", err)
|
|
}
|
|
if _, err := w.Write([]byte{'\n'}); err != nil {
|
|
return HandleErrorRespectJSON("failed to write newline: %v", err)
|
|
}
|
|
memoryCount++
|
|
}
|
|
}
|
|
|
|
// Finalize atomic write if writing to file (fsync + rename).
|
|
if aw != nil {
|
|
if err := aw.Close(); err != nil {
|
|
return HandleErrorRespectJSON("failed to finalize export file: %v", err)
|
|
}
|
|
}
|
|
|
|
// Print summary to stderr (not stdout, to avoid mixing with JSONL)
|
|
if exportOutput != "" {
|
|
if memoryCount > 0 {
|
|
fmt.Fprintf(os.Stderr, "Exported %d issues and %d memories to %s\n", count, memoryCount, exportOutput)
|
|
} else {
|
|
fmt.Fprintf(os.Stderr, "Exported %d issues to %s\n", count, exportOutput)
|
|
}
|
|
if exportVerbose && filteredOwnerCount > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d filtered as personal by owner exclusion)\n", filteredOwnerCount)
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// exportIssueRecord wraps IssueWithCounts with a _type discriminator so that
|
|
// every line in the JSONL export is self-describing. Memory lines already
|
|
// carry "_type":"memory"; this gives issue lines "_type":"issue". (GH#3271)
|
|
type exportIssueRecord struct {
|
|
RecordType string `json:"_type"`
|
|
*types.IssueWithCounts
|
|
// WispPlane is the explicit wisps-plane marker (bd-r9uce): true when the
|
|
// row lives in the WISPS table AND its flags alone cannot prove it (the
|
|
// no_history shape; ephemeral rows are self-describing and stay
|
|
// unstamped). Import routes by this marker — never by no_history — so a
|
|
// promoted no-history wisp (durable issues-table row still carrying the
|
|
// stray flag) round-trips to the durable plane instead of being silently
|
|
// re-planed. Declared after the embedded struct so it serializes last.
|
|
// Deliberately a FRESH key, not the legacy "wisp" alias key: pre-fix
|
|
// binaries' alias branch would import a marked no-history wisp as
|
|
// ephemeral (purge-eligible, export-excluded), so an unknown-to-them key
|
|
// that degrades to flag routing is the data-safe choice (lion, #5368).
|
|
WispPlane bool `json:"wisp_plane,omitempty"`
|
|
}
|
|
|
|
// sanitizeZeroTime replaces Go zero-value time.Time fields with Unix epoch.
|
|
// NULL datetime columns in Dolt scan as time.Time{} (year 0001-01-01), which
|
|
// causes json.Marshal to fail with "year outside of range [0,9999]". (GH#2488)
|
|
func sanitizeZeroTime(issue *types.Issue) {
|
|
epoch := time.Unix(0, 0).UTC()
|
|
if issue.CreatedAt.IsZero() {
|
|
issue.CreatedAt = epoch
|
|
}
|
|
if issue.UpdatedAt.IsZero() {
|
|
issue.UpdatedAt = epoch
|
|
}
|
|
}
|
|
|
|
// filterOutPollution removes issues that look like test/pollution records.
|
|
func filterOutPollution(issues []*types.Issue) []*types.Issue {
|
|
var clean []*types.Issue
|
|
for _, issue := range issues {
|
|
if !isTestIssue(issue.Title) {
|
|
clean = append(clean, issue)
|
|
}
|
|
}
|
|
return clean
|
|
}
|
|
|
|
// buildOwnerExcludeSet merges --exclude-owner flag values with the
|
|
// export.exclude_owners (and legacy export.exclude_owner) config entries.
|
|
// Returns the combined set as a map for O(1) lookup.
|
|
func buildOwnerExcludeSet(ctx context.Context, src exportSource, flagOwners []string) map[string]struct{} {
|
|
set := make(map[string]struct{})
|
|
for _, o := range flagOwners {
|
|
if o != "" {
|
|
set[o] = struct{}{}
|
|
}
|
|
}
|
|
// export.* keys are YAML-only (config.IsYamlOnlyKey returns true for the
|
|
// "export." prefix), so bd config set stores them in config.yaml rather than
|
|
// the database. Read from YAML first, then fall back to the database for any
|
|
// instance that was written directly to the store.
|
|
addOwners := func(val string) {
|
|
for _, o := range strings.Split(val, ",") {
|
|
if o = strings.TrimSpace(o); o != "" {
|
|
set[o] = struct{}{}
|
|
}
|
|
}
|
|
}
|
|
if val := config.GetYamlConfig("export.exclude_owners"); val != "" {
|
|
addOwners(val)
|
|
}
|
|
if val := config.GetYamlConfig("export.exclude_owner"); val != "" {
|
|
set[strings.TrimSpace(val)] = struct{}{}
|
|
}
|
|
// Also read from database for any value stored there directly.
|
|
if val, err := src.GetConfig(ctx, "export.exclude_owners"); err == nil && val != "" {
|
|
addOwners(val)
|
|
}
|
|
if val, err := src.GetConfig(ctx, "export.exclude_owner"); err == nil && val != "" {
|
|
set[strings.TrimSpace(val)] = struct{}{}
|
|
}
|
|
return set
|
|
}
|
|
|
|
// filterOutOwners removes issues whose created_by identity is in the exclude set.
|
|
func filterOutOwners(issues []*types.Issue, exclude map[string]struct{}) []*types.Issue {
|
|
var keep []*types.Issue
|
|
for _, issue := range issues {
|
|
if _, excluded := exclude[issue.CreatedBy]; !excluded {
|
|
keep = append(keep, issue)
|
|
}
|
|
}
|
|
return keep
|
|
}
|