Memories (from `bd remember`) may contain sensitive agent context — API
patterns, debugging notes, credential hints. They were included in both
`bd export` and auto-export (pre-commit hook) by default, leaking
private data into git history.
Changes:
- auto-export: pass includeMemories=false to exportToFile
- bd export: flip default to exclude memories; add --include-memories
opt-in flag; --all also includes memories
- --no-memories is preserved (hidden) for backward compat but is now
the default behavior
Adds TestExportExcludesMemoriesByDefault to verify default exclusion
and that --include-memories / --all still include them.
Wisps are private/transient and must not reach git history or external
integrations. Both bd export and auto-export (pre-commit hook) included
wisps because the SearchIssues filter had Ephemeral=nil (match all).
Set filter.Ephemeral=false (persistent-only) in both export.go and
export_auto.go. The --all flag overrides to include everything.
Adds TestExportExcludesWispsByDefault to verify default exclusion and
that --all still includes wisps.
* fix(export): sort memory keys for deterministic JSONL output (GH#3474)
Go map iteration is non-deterministic, so _type:"memory" lines were
shuffled on every export. Sort keys before writing to eliminate noisy
diffs in committed issues.jsonl.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(export): sort memory keys in export_auto.go + add determinism test (GH#3474)
export_auto.go:210 had the same unsorted map iteration as export.go.
Apply identical sort fix and upgrade the silent json.Marshal error
to a returned error for parity.
Add TestExportMemoryDeterminism: seeds 5 memories, exports twice,
asserts byte-identical output and alphabetical key order.
Addresses review feedback on PR #3482.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`gitAddFile` at cmd/bd/export_auto.go called cmd.Run(), discarding
stdout + stderr. The caller at line 113 then formatted the warning as
"Warning: auto-export: git add failed: %v" — leaving users with just
"exit status 1" and no diagnostic to act on. Common real-world
triggers (paths-are-ignored, index.lock contention, detached HEAD)
all leave their signal in stderr, which we were dropping.
Switch to cmd.CombinedOutput() and wrap any non-nil error with the
trimmed output via fmt.Errorf("%w: %s", err, trimmed). Function
signature stays `error`, caller's existing %v formatter naturally
prints "exit status 1: <git's stderr>". No change for the success
path, no change for callers.
Add TestGitAddFile_CapturesStderrOnFailure: temp git repo with
.gitignore excluding .beads/, asserts the returned error contains
"ignored" (proxy for git's stderr making it through). Regression
guard so a future refactor doesn't silently drop the captured output
again.
Fixes#3504
* fix: remove duplicate wisp query in bd export (#3352)
SearchIssues with Ephemeral=nil already queries both the issues and
wisps tables with ID-based dedup (issueops/search.go). The separate
Ephemeral=true query followed by append caused every wisp to appear
twice in the exported JSONL.
Remove the redundant wisp query from both export.go (manual export)
and export_auto.go (auto-export/exportToFile). The SearchIssues call
with Ephemeral=nil continues to include all wisps correctly.
* fix: suppress gosec G304 false positive in hookWorkTreeRoot
The path is filepath.Join(os.Getenv("GIT_DIR"), "gitdir") — reading a
well-known git internal file from a trusted environment variable set by
git itself during hook execution. Not a user-controlled file inclusion.
This false positive causes the Lint CI check to fail on every PR in
the repo, masking real lint issues.
* test(export): add regression test for duplicate wisp entries (GH#3352)
Addresses review feedback requesting test coverage for the duplicate-wisp
export fix. Creates 3 regular issues and 3 ephemeral wisps, runs export,
and asserts: no duplicate IDs in output, wisp count matches source count,
and total exported count equals expected.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add BD_JSON_ENVELOPE=1 opt-in for uniform JSON wrapping
Follow-up to #3368 as discussed with @maphew. Adds the envelope format
that wraps all --json output as {"schema_version": N, "data": <payload>},
gated behind BD_JSON_ENVELOPE=1 to avoid breaking existing consumers.
Legacy mode (default): unchanged behavior from #3368 — objects get
schema_version injected inline, arrays pass through raw.
Envelope mode (BD_JSON_ENVELOPE=1): uniform wrapping for all types.
The original payload is untouched inside .data — no type corruption,
no field injection. A one-time deprecation notice is emitted to stderr
when --json is used without the env var.
Timeline: envelope becomes default in v2.0, with BD_JSON_ENVELOPE=0
as a temporary escape hatch for one release cycle.
Includes:
- wrapWithSchemaVersion dual-mode logic
- outputJSONError and buildJSONError envelope support
- Deprecation notice on stderr (once per process)
- Tests for both legacy and envelope modes
- Migration guide in docs/JSON_SCHEMA.md
* fix: suppress deprecation notice when stderr is not a TTY
Gate emitEnvelopeDeprecation() on ui.IsStderrTerminal() so the
"NOTE: bd --json output format will change in v2.0" message only
appears in interactive terminals. Test harnesses and pipelines
capture stderr via CombinedOutput(), and the trailing notice after
the JSON closing brace caused json.Unmarshal to fail with
"invalid character 'N' after top-level value" in 15 CI tests.
Follows the same TTY-gating pattern used by ShouldUseColor and
ShouldUseEmoji.
* fix: suppress gosec G304 false positive in hookWorkTreeRoot
The path is filepath.Join(os.Getenv("GIT_DIR"), "gitdir") — reading a
well-known git internal file from a trusted environment variable set by
git itself during hook execution. Not a user-controlled file inclusion.
This false positive causes the Lint CI check to fail on every PR in
the repo, masking real lint issues.
When export.git-add=true (the v1.0.1+ default), the pre-commit hook in
a worktree stages a spurious issues.jsonl at the repo root instead of
(or in addition to) .beads/issues.jsonl. The ghost file lands in the
commit's diff but not on disk — easy to miss and painful to clean up.
Root cause (not what the issue body guessed): the absolute path passed
to `git add` was always correct. The bug is env inheritance. Git
invokes the pre-commit hook with GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/
GIT_COMMON_DIR/GIT_PREFIX set in the environment, and bd's git-add
subprocesses inherit them.
Both call sites set `cmd.Dir = filepath.Dir(fullPath)` (= `.beads/`).
With GIT_DIR set in the environment and GIT_WORK_TREE unset, git
defaults the work-tree to cwd. So the subprocess's git sees `.beads/`
as the work-tree root and records `.beads/issues.jsonl` at the root of
that broken view — which ends up in the worktree's actual index as a
bare `issues.jsonl`. Verified empirically: with GIT_DIR set and
cwd=`.beads/`, `git rev-parse --show-toplevel` returns the `.beads/`
directory itself.
Fix, in two parts:
1. scrubGitHookEnv() strips the vars that poison repo/worktree
auto-discovery, index routing, object routing, and config
injection:
- discovery: GIT_DIR, GIT_WORK_TREE, GIT_COMMON_DIR, GIT_PREFIX,
GIT_CEILING_DIRECTORIES, GIT_DISCOVERY_ACROSS_FILESYSTEM
- index: GIT_INDEX_FILE
- objects: GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES
- config: the whole GIT_CONFIG* family (covers the case where the
parent invoked `git -c core.worktree=… commit`, which sets
GIT_CONFIG_PARAMETERS / GIT_CONFIG_COUNT in the child env)
With these cleared, git rediscovers the repo/worktree from cwd.
2. hookWorkTreeRoot() reads the inherited GIT_DIR to determine the
worktree whose hook is currently firing. If the target path is
outside that worktree, gitAddFile returns without staging. This
prevents a narrower but real regression that would otherwise land
after part 1 alone: in the .beads/redirect configuration, fullPath
points into the main repo (not the worktree), and post-scrub git
would silently stage the file into main's index from a worktree
hook. The existing preCommitHookBody template (init_git_hooks.go)
documents the same intent: "For worktrees: .beads is in the main
repo's working tree... we skip it."
The guard uses pathInsideDir which handles the macOS symlinked-parent
case: on /tmp → /private/tmp, a fresh-file target expressed as
/tmp/.../new.txt compared against a worktree root resolved to
/private/tmp/... would otherwise misclassify as outside the worktree.
pathInsideDir resolves the parent of each side via EvalSymlinks and
reattaches the basename — sufficient for the real caller shapes, since
gitAddFile's parent directory (beadsDir or the export output dir)
always exists at call time.
Also fixes the variant reported in a follow-up comment where the
.beads/redirect workaround did not resolve the root ghost: the env
pollution broke git-add inside the `bd export -o` subprocess spawned
from exportJSONLForCommit (that subprocess has BD_GIT_HOOK stripped,
but git env vars preserved, so its PostRun auto-export hits the same
bug via the shared gitAddFile).
Deduplicates the parallel git-add invocation in exportJSONLForCommit
by delegating to gitAddFile, so both call sites share the env scrub
and the cross-worktree guard.
Adds:
- TestGitAddFile_InWorktreeHook_StagesCorrectPath — end-to-end
worktree regression test; fails on main, passes with the fix.
- TestGitAddFile_RedirectCase_DoesNotStageInMainRepo — verifies
the cross-worktree skip does not pollute main's index when a
worktree has .beads/redirect pointing into main.
- TestGitAddFile_NonHookContext_GuardDoesNotFire — regression guard
confirming the cross-worktree skip is a no-op outside git hooks.
- TestScrubGitHookEnv — unit test for the env scrub helper, covering
the discovery/index/object vars and the GIT_CONFIG* family, and
verifying non-discovery vars (author/committer/editor/pager) pass
through untouched.
- TestPathInsideDir — covers structural cases plus the macOS
fresh-file + symlinked-parent regression.
- TestHookWorkTreeRoot — covers linked-worktree gitdir file,
plain-repo .git, bare/unrecognized, and not-a-hook contexts.
Workaround for users on older releases remains `BD_EXPORT_GIT_ADD=false`.
Closes GH#3311
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When no-git-ops=true is set, bd still attempted to git add the
auto-export file on every write, producing a warning on every
invocation. Users had to set both no-git-ops=true and
export.git-add=false separately.
Check the no-git-ops flag in both the auto-export path
(export_auto.go) and the pre-commit hook export path (hooks.go).
When no-git-ops is true, skip the git add without warning.
Co-authored-by: kev <kglynn@pryoninc.com>
* fix(export): add _type discriminator to JSONL issue lines (GH#3271)
bd export produces JSONL with mixed schemas: issue lines and memory lines.
Memory lines carry "_type":"memory" but issue lines had no _type field,
causing downstream consumers to crash when they encounter an unexpected
schema (e.g. json.loads(line)["status"] throws KeyError on memory lines).
Add "_type":"issue" to all exported issue records via an exportIssueRecord
wrapper struct. This makes every JSONL line self-describing without changing
the Issue or IssueWithCounts types used elsewhere.
Both export paths updated: manual (runExport) and auto (exportToFile).
Import already handles this gracefully - unknown fields are ignored during
json.Unmarshal into types.Issue.
* test(export): add _type discriminator assertion to embedded export tests
Verifies that every JSONL line from bd export carries a _type field
("issue" or "memory") and that each type has its expected key fields.
Addresses codecov 0% patch coverage flag on PR #3272.
Adds a project-agnostic beads_global database to shared-server mode so
clients can track issues globally across all projects on the server.
- Add EnsureGlobalDatabase() to create the DB during shared server init
- Add --global persistent flag to switch CLI commands to the global DB
- Add global_dolt_database field to metadata.json for connection info
- Seed global DB with issue_prefix="global" and sentinel project ID
- Export/import default to global-issues.jsonl with --global flag
- --global errors when not in shared-server mode (additive, no breakage)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: enable auto-export by default with canonical issues.jsonl name (GH#2973)
Users expected JSONL export to refresh automatically but export.auto
defaulted to false, requiring explicit opt-in. Also the default filename
(export.jsonl) didn't match the canonical issues.jsonl used in docs and
git hooks. Change defaults: export.auto=true, export.path=issues.jsonl,
export.git-add=true so viewers (bv) and git workflows see fresh data
without extra configuration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(init): add auto-export prompt and document export.* config (GH#2973)
Interactive bd init now asks whether to enable auto-export (default: yes).
Users can opt out during setup instead of discovering the setting later.
Also documents the export.* config namespace in bd config --help with
all four keys (export.auto, export.path, export.interval, export.git-add)
and adds export examples.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: document export.* auto-export config keys (GH#2973)
Document the export.auto/path/interval/git-add keys in docs/CONFIG.md
and note the default-on switch in CHANGELOG.md [Unreleased].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(export): don't prime throttle on empty init and skip git-add without git (GH#2973)
Review feedback on #3204 surfaced two regressions in the default-on
auto-export path:
1. bd init's PersistentPostRun fired maybeAutoExport with zero issues,
writing an empty .beads/issues.jsonl and saving export-state.json with
the current dolt commit + timestamp. The next `bd create` was then
throttled for up to export.interval (default 60s) because the
timestamp check in maybeAutoExport returned before considering the new
commit, leaving issues.jsonl stale on fresh repos.
Skip saving state and remove the empty file when the export produced
no issues and no memories. The next write runs a real export
immediately.
2. export.git-add defaulting to true emitted
`Warning: auto-export: git add failed: exit status 128` on every
write in the supported standalone BEADS_DIR flow where init
intentionally skips `git init`. Gate the git add call on isGitRepo()
so standalone users get silent, working auto-export.
Adds a scripttest regression guard for the init -> immediate create
path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(memory): disable auto-export in concurrent test (GH#2973)
TestEmbeddedMemoryConcurrent started failing on this branch after
auto-export became the default: every concurrent 'bd remember'/'bd
forget' call now fires maybeAutoExport, which reads store state after
the write. Under 8-worker contention this intermittently leaves a
forgotten key visible to 'bd memories'.
The test exercises memory concurrency, not export behavior; opt it
out of auto-export so the signal stays focused. Followup: investigate
the race between write commit and post-write auto-export read in
embedded Dolt concurrent paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: make auto-export and auto-backup errors visible on stderr
Previously, failures in auto-export (JSONL flush) and auto-backup were
logged only via debug.Logf, invisible to normal users. One user lost 46
issues because JSONL export silently stopped working with no warnings.
Upgrade critical error paths from debug.Logf to fmt.Fprintf(os.Stderr)
so users see warnings when their backup/export pipeline breaks. Keep
debug.Logf for non-critical diagnostics (throttle, no-change skip).
Closes#2489
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: remove stale sync references
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bd export populated comment_count but never called GetCommentsForIssues()
to populate the Comments field on each issue. This meant bd export | bd import
silently lost all comment history. Now both export.go and export_auto.go
bulk-load comments alongside labels and dependencies.
Fixes#3007
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The HookFiringStore decorator wraps DoltStorage for automatic hook
firing, but hides the concrete store's optional interfaces
(StoreLocator, BackupStore, Flattener, Compactor, etc.) from type
assertions. This caused "storage backend does not support X" errors
for flatten, backup, compact, dolt remote, and other operations.
Add storage.UnwrapStore() helper that peels the decorator before type
assertions, and apply it at all 24 callsites in cmd/bd/.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When `export.auto: true` is set in config.yaml, beads automatically
writes a git-tracked JSONL file (issues + memories) after mutations.
This provides full portability across machines via git alone, without
requiring a Dolt remote.
New config keys:
export.auto: false # opt-in to auto-export
export.interval: 60s # throttle interval
export.path: export.jsonl # output file relative to .beads/
export.git-add: false # auto-stage the file
The auto-export:
- Runs in PersistentPostRun after auto-backup
- Uses Dolt commit hash for change detection (no-op if nothing changed)
- Throttles by interval (default 60s)
- Skips during git hooks (BD_GIT_HOOK=1)
- Includes both issues and memories in the JSONL output
Also extracts exportToFile() as a shared helper usable by both
`bd export -o` and the auto-export system.
Files:
- cmd/bd/export_auto.go: Core auto-export logic
- cmd/bd/main.go: Wire maybeAutoExport into PersistentPostRun
- internal/config/config.go: New export.* config defaults
- internal/config/yaml_config.go: Add "export." to yaml-only prefix list
- cmd/bd/doctor/gitignore.go: Add export-state.json to gitignore
- cmd/bd/doctor/tracked_runtime.go: Flag export-state.json if tracked
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>