beads/cmd/bd/export_source.go
Steve Yegge 80af94c0a7
fix(export): promoted no-history wisps survive import round-trip (bd-r9uce) (#5368)
* fix(export): promoted no-history wisps survive import round-trip (bd-r9uce)

A promoted no-history wisp is a durable issues-table row that still
carried no_history=true (PromoteFromEphemeralInTx cleared only
Ephemeral), and both import routes inferred the storage plane from row
flags (issueops.IsWisp: Ephemeral || NoHistory). Importing an export
stream containing such a row silently re-planed it into the wisps table,
dropping its cross-plane relations as cross-bucket dependencies on the
way — classic-mode data loss across export->import->export.

Belt and braces, per triage + the #5351 review endorsement:

1. Promote clears NoHistory too (and normalizes an explicit ephemeral
   StorageClass to empty, the NormalizePersistenceMode promotion rule):
   a promoted wisp is fully durable and the flag has no post-promotion
   meaning.

2. Import routes the plane by an explicit marker, never by no_history.
   Export stamps "wisp": true on rows that live in the WISPS table and
   whose flags alone cannot prove it (the no_history shape; ephemeral
   rows self-describe), classified by table membership via a new
   PartitionWispIDs store capability (classic: batched wisps-membership
   query behind an Unwrap walk through the telemetry/hook decorators;
   proxied: the plane-pinned GetWispsByIDs read). Both parse loops
   (bd import in both storage modes, bootstrap/init/auto-import) resolve
   the plane through one shared helper, applyImportWispPlane, which pins
   marker-less no_history records to the durable plane via a new
   non-serialized Issue.WispPlaneOverride consulted by IsWisp. The
   no_history flag itself is preserved on the row, keeping the
   export->import->export byte-identity for wild data; the legacy
   v0.35-v0.37 "wisp"-as-ephemeral alias behavior is unchanged.

Tests: the promoted shape now lives IN the proxied round-trip fixture
(cross_mode_byte_identical covers it, with relations); a classic-mode
round-trip test (fails on base: cross-bucket dependency drop + missing
plane marker); a proxied import plane-routing subtest; embeddeddolt pins
for promote clearing NoHistory and PartitionWispIDs membership
classification.

* docs(schema): document the export stream's explicit wisp plane marker (bd-r9uce)

* fix(export): fresh wisp_plane marker key — legacy-alias reuse was lossy under rollout skew (bd-r9uce)

Adopts lion's #5368 review recommendation: pre-fix v0.38+ binaries' alias
branch (hasWisp && !Ephemeral => Ephemeral=true) would import a marked
genuine no-history wisp as ephemeral — purge-eligible and excluded from
that rig's next export — making the common skew case lossy. A fresh key
degrades to flag routing on older readers, data-safe in both directions.
Legacy 'wisp' read alias preserved verbatim.

* fix(storage): clone WispPlaneOverride in cloneIssueForHook (bd-r9uce)

The hook snapshot guard (TestCloneIssueForHookCoversReferenceFields)
correctly flagged the new *bool field as uncloned: a hook mutating the
snapshot could alias the live issue's plane pin. Deep-copy it and add
it to the guard's covered-fields manifest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZXjihTayFcngYzo6xPd3p

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 12:48:44 -07:00

345 lines
13 KiB
Go

package main
import (
"context"
"errors"
"fmt"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/dberrors"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
)
// exportSource abstracts the storage reads `bd export` performs, so the one
// export body (runExportFromSource) serves both storage stacks:
//
// - storeExportSource reads through the classic `store` global
// (embedded / direct / server modes), preserving the exact pre-seam
// call pattern.
// - uowExportSource reads through a proxied-server unit of work, whose
// domain use cases are plane-pinned (issues vs wisps tables) where the
// classic bulk loaders partition internally — the impl queries both
// planes with the full ID set and merges, so both modes read the same
// rows regardless of which table an issue lives in.
//
// Everything downstream of these reads is shared code; the acceptance bar for
// the seam is byte-identical JSONL output across modes.
type exportSource interface {
// GetInfraTypes returns the resolved infra-type set (config, YAML, or
// hardcoded defaults). A nil/empty result makes the caller fall back to
// domain.DefaultInfraTypes(), mirroring the pre-seam behavior.
GetInfraTypes(ctx context.Context) map[string]bool
// SearchIssues returns the full, untruncated result set for the export
// filter (Limit=0, MaxRows=0). Implementations MUST fail rather than
// return a truncated page — export is a data-integrity path.
SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error)
// GetConfig reads one config key from the database (used for the
// export.exclude_owners owner-filter keys).
GetConfig(ctx context.Context, key string) (string, error)
// GetAllConfig reads the whole config table (used to extract kv.memory.*
// rows when memories are exported).
GetAllConfig(ctx context.Context) (map[string]string, error)
// LoadExportRelations bulk-loads labels, dependency records, comments,
// comment counts, and dependency counts for the searched issues.
LoadExportRelations(ctx context.Context, issues []*types.Issue) (exportRelations, error)
// WispPlaneIDs reports which of ids currently live in the WISPS table.
// Export uses it to stamp the explicit "wisp_plane" marker on records
// whose row flags are ambiguous: a no_history=true row is either an
// unpromoted no-history wisp (wisps table) or a promoted one (durable
// issues-table row that may still carry the stray flag), and only table
// membership can tell them apart — import routes by the marker, so
// mis-stamping a durable row would re-plane it and drop its relations
// (bd-r9uce). Implementations must classify by table membership, never
// by flags, and both modes must agree (byte-identity oracle).
WispPlaneIDs(ctx context.Context, ids []string) (map[string]bool, error)
}
// storeWispPartitioner is the optional store capability WispPlaneIDs uses in
// classic mode. Both real stores (DoltStore, EmbeddedDoltStore) implement it;
// a store that does not simply gets no plane markers, which degrades to the
// data-safe side (import then routes bare no_history rows to the durable
// plane, where nothing is ever excluded from export).
type storeWispPartitioner interface {
PartitionWispIDs(ctx context.Context, ids []string) (wispIDs, permIDs []string, err error)
}
// exportRelations carries the bulk-loaded relational data for the export set,
// keyed by issue ID.
type exportRelations struct {
labels map[string][]string
deps map[string][]*types.Dependency
comments map[string][]*types.Comment
commentCounts map[string]int
depCounts map[string]*types.DependencyCounts
}
// errExportNoStore reports a classic-source read attempted with no store
// open. Callers of exportSource.GetConfig treat any error as "key unset",
// which reproduces the pre-seam `if store == nil` early return.
var errExportNoStore = errors.New("no store available")
// storeExportSource is the classic-mode exportSource over the `store` global.
type storeExportSource struct{}
func (storeExportSource) GetInfraTypes(ctx context.Context) map[string]bool {
if store == nil {
return nil
}
return store.GetInfraTypes(ctx)
}
func (storeExportSource) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) {
return store.SearchIssues(ctx, query, filter)
}
func (storeExportSource) GetConfig(ctx context.Context, key string) (string, error) {
if store == nil {
return "", errExportNoStore
}
return store.GetConfig(ctx, key)
}
func (storeExportSource) GetAllConfig(ctx context.Context) (map[string]string, error) {
return store.GetAllConfig(ctx)
}
func (storeExportSource) LoadExportRelations(ctx context.Context, issues []*types.Issue) (exportRelations, error) {
issueIDs := make([]string, len(issues))
for i, issue := range issues {
issueIDs[i] = issue.ID
}
// Individual bulk-load failures deliberately degrade to empty maps rather
// than aborting the export — unchanged from the pre-seam classic behavior.
labelsMap, _ := store.GetLabelsForIssues(ctx, issueIDs)
allDeps, _ := store.GetDependencyRecordsForIssues(ctx, issueIDs)
commentsMap, _ := store.GetCommentsForIssues(ctx, issueIDs)
commentCounts, _ := store.GetCommentCounts(ctx, issueIDs)
depCounts, _ := store.GetDependencyCounts(ctx, issueIDs)
return exportRelations{
labels: labelsMap,
deps: allDeps,
comments: commentsMap,
commentCounts: commentCounts,
depCounts: depCounts,
}, nil
}
func (storeExportSource) WispPlaneIDs(ctx context.Context, ids []string) (map[string]bool, error) {
if len(ids) == 0 || store == nil {
return nil, nil
}
// The store global is decorator-wrapped (telemetry, hook firing); walk
// Unwrap() down to the store that carries the partition capability.
s := store
var p storeWispPartitioner
for {
if partitioner, ok := s.(storeWispPartitioner); ok {
p = partitioner
break
}
u, ok := s.(interface{ Unwrap() storage.DoltStorage })
if !ok {
return nil, nil
}
s = u.Unwrap()
if s == nil {
return nil, nil
}
}
wispIDs, _, err := p.PartitionWispIDs(ctx, ids)
if err != nil {
return nil, err
}
set := make(map[string]bool, len(wispIDs))
for _, id := range wispIDs {
set[id] = true
}
return set, nil
}
// uowExportSource is the proxied-server exportSource over a unit of work. All
// reads run inside the single read transaction runExport opened, so the
// export is one consistent snapshot.
type uowExportSource struct {
uw uow.UnitOfWork
}
func (s *uowExportSource) GetInfraTypes(ctx context.Context) map[string]bool {
infra, err := s.uw.ConfigUseCase().GetInfraTypes(ctx)
if err != nil {
// Match the classic source's error-free signature: an unreadable
// config degrades to the caller's DefaultInfraTypes() fallback.
return nil
}
return infra
}
func (s *uowExportSource) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) {
page, err := s.uw.IssueUseCase().SearchIssues(ctx, query, filter)
if err != nil {
return nil, err
}
// Export builds its filter with Limit=0/MaxRows=0, so the whole result
// arrives as one page and HasMore can only be true if a limit creeps back
// in. Guard anyway: an export must fail loudly, never truncate silently
// (domain.SearchPage carries no continuation cursor to loop with).
if page.HasMore {
return nil, fmt.Errorf("issue search returned a truncated page (limit=%d); export must never truncate", filter.Limit)
}
return page.Items, nil
}
func (s *uowExportSource) GetConfig(ctx context.Context, key string) (string, error) {
return s.uw.ConfigUseCase().GetConfig(ctx, key)
}
func (s *uowExportSource) GetAllConfig(ctx context.Context) (map[string]string, error) {
return s.uw.ConfigUseCase().GetAllConfig(ctx)
}
// LoadExportRelations reproduces the classic bulk loaders over the
// plane-pinned domain use cases:
//
// - Labels, comments, and comment counts: the classic loaders partition IDs
// by wisps-table membership (issueops.PartitionWispIDsInTx). Row flags are
// NOT a substitute for that partition — a promoted no-history wisp is a
// durable issues-table row that (as wild data from the pre-bd-r9uce
// promote) still carries NoHistory=true — so both
// planes are queried with the FULL ID set and merged. The planes are
// ID-disjoint (GH#4455 cross-table guard), so at most one plane returns
// rows for any id and the merge cannot collide.
// - Dependency records: domain GetIssueDependencyRecords already partitions
// internally (same issueops helper), so one call covers both planes.
// - Dependency counts: classic GetDependencyCountsInTx queries BOTH dep
// tables for EVERY id and sums (a wisp blocking a durable issue
// contributes to the durable issue's dependent_count), so both planes are
// queried for the full ID set and summed here too.
//
// Wisp-plane reads tolerate a rig without the wisp tables (IsTableNotExist),
// mirroring the dependency-counts leg.
func (s *uowExportSource) LoadExportRelations(ctx context.Context, issues []*types.Issue) (exportRelations, error) {
rel := exportRelations{
labels: map[string][]string{},
deps: map[string][]*types.Dependency{},
comments: map[string][]*types.Comment{},
commentCounts: map[string]int{},
depCounts: map[string]*types.DependencyCounts{},
}
allIDs := make([]string, 0, len(issues))
for _, issue := range issues {
allIDs = append(allIDs, issue.ID)
}
if len(allIDs) == 0 {
return rel, nil
}
labels, err := s.uw.LabelUseCase().GetLabelsForIssues(ctx, allIDs)
if err != nil {
return exportRelations{}, fmt.Errorf("load labels: %w", err)
}
mergeExportMap(rel.labels, labels)
comments, err := s.uw.CommentUseCase().GetCommentsForIssues(ctx, allIDs)
if err != nil {
return exportRelations{}, fmt.Errorf("load comments: %w", err)
}
mergeExportMap(rel.comments, comments)
counts, err := s.uw.CommentUseCase().GetCommentCounts(ctx, allIDs)
if err != nil {
return exportRelations{}, fmt.Errorf("load comment counts: %w", err)
}
mergeExportMap(rel.commentCounts, counts)
wispLabels, err := s.uw.LabelUseCase().GetLabelsForWisps(ctx, allIDs)
if err != nil {
if !dberrors.IsTableNotExist(err) {
return exportRelations{}, fmt.Errorf("load wisp labels: %w", err)
}
} else {
mergeExportMap(rel.labels, wispLabels)
}
wispComments, err := s.uw.CommentUseCase().GetCommentsForWisps(ctx, allIDs)
if err != nil {
if !dberrors.IsTableNotExist(err) {
return exportRelations{}, fmt.Errorf("load wisp comments: %w", err)
}
} else {
mergeExportMap(rel.comments, wispComments)
}
wispCounts, err := s.uw.CommentUseCase().GetWispCommentCounts(ctx, allIDs)
if err != nil {
if !dberrors.IsTableNotExist(err) {
return exportRelations{}, fmt.Errorf("load wisp comment counts: %w", err)
}
} else {
mergeExportMap(rel.commentCounts, wispCounts)
}
deps, err := s.uw.DependencyUseCase().GetIssueDependencyRecords(ctx, allIDs)
if err != nil {
return exportRelations{}, fmt.Errorf("load dependency records: %w", err)
}
rel.deps = deps
depCounts, err := s.uw.DependencyUseCase().CountsByIssueIDs(ctx, allIDs)
if err != nil {
return exportRelations{}, fmt.Errorf("load dependency counts: %w", err)
}
rel.depCounts = depCounts
wispDepCounts, err := s.uw.DependencyUseCase().CountsByWispIDs(ctx, allIDs)
if err != nil {
// A rig without the wisp tables simply has no wisp-plane edges.
if !dberrors.IsTableNotExist(err) {
return exportRelations{}, fmt.Errorf("load wisp dependency counts: %w", err)
}
} else {
for id, wc := range wispDepCounts {
if wc == nil || (wc.DependencyCount == 0 && wc.DependentCount == 0) {
continue
}
c := rel.depCounts[id]
if c == nil {
c = &types.DependencyCounts{}
rel.depCounts[id] = c
}
c.DependencyCount += wc.DependencyCount
c.DependentCount += wc.DependentCount
}
}
return rel, nil
}
// WispPlaneIDs classifies by wisps-table membership through the plane-pinned
// GetWispsByIDs read (row flags are NOT a substitute — see the exportSource
// interface comment). A rig without the wisp tables has no wisp-plane rows,
// mirroring the tolerance of the wisp-plane legs in LoadExportRelations.
func (s *uowExportSource) WispPlaneIDs(ctx context.Context, ids []string) (map[string]bool, error) {
if len(ids) == 0 {
return nil, nil
}
wisps, err := s.uw.IssueUseCase().GetWispsByIDs(ctx, ids)
if err != nil {
if dberrors.IsTableNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("wisp plane membership: %w", err)
}
set := make(map[string]bool, len(wisps))
for _, w := range wisps {
set[w.ID] = true
}
return set, nil
}
// mergeExportMap copies src entries into dst. The label/comment planes are
// disjoint by ID (an issue lives in exactly one of issues/wisps), so a plain
// overwrite-merge reassembles the classic partitioned loaders' single map.
func mergeExportMap[V any](dst, src map[string]V) {
for k, v := range src {
dst[k] = v
}
}