Commit graph

17 commits

Author SHA1 Message Date
maphew
ea9fa3efca Merge remote-tracking branch 'origin/main' into fix/export-exclude-memories
# Conflicts:
#	cmd/bd/export_test.go
2026-05-02 17:44:16 -07:00
kev
61e85bc319 fix(export): exclude memories from bd export by default (GH#3650)
Memories (from `bd remember`) may contain sensitive agent context — API
patterns, debugging notes, credential hints. They were included in both
`bd export` and auto-export (pre-commit hook) by default, leaking
private data into git history.

Changes:
- auto-export: pass includeMemories=false to exportToFile
- bd export: flip default to exclude memories; add --include-memories
  opt-in flag; --all also includes memories
- --no-memories is preserved (hidden) for backward compat but is now
  the default behavior

Adds TestExportExcludesMemoriesByDefault to verify default exclusion
and that --include-memories / --all still include them.
2026-05-02 01:28:12 -04:00
kev
7978fae1ad fix(export): exclude ephemeral wisps from bd export by default (GH#3649)
Wisps are private/transient and must not reach git history or external
integrations. Both bd export and auto-export (pre-commit hook) included
wisps because the SearchIssues filter had Ephemeral=nil (match all).

Set filter.Ephemeral=false (persistent-only) in both export.go and
export_auto.go. The --all flag overrides to include everything.

Adds TestExportExcludesWispsByDefault to verify default exclusion and
that --all still includes wisps.
2026-05-02 01:25:27 -04:00
coffeegoddd☕️✨
2f1cd4f818 /cmd/{cmd, internal}: make exports atomic writes 2026-05-01 16:42:47 -07:00
Kevin Glynn
651a52afed
fix(export): sort memory keys for deterministic JSONL output (GH#3474) (#3482)
* fix(export): sort memory keys for deterministic JSONL output (GH#3474)

Go map iteration is non-deterministic, so _type:"memory" lines were
shuffled on every export. Sort keys before writing to eliminate noisy
diffs in committed issues.jsonl.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(export): sort memory keys in export_auto.go + add determinism test (GH#3474)

export_auto.go:210 had the same unsorted map iteration as export.go.
Apply identical sort fix and upgrade the silent json.Marshal error
to a returned error for parity.

Add TestExportMemoryDeterminism: seeds 5 memories, exports twice,
asserts byte-identical output and alphabetical key order.

Addresses review feedback on PR #3482.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 06:34:17 -07:00
sms
4fcad21cee fix(auto-export): surface git add stderr in failure warning
`gitAddFile` at cmd/bd/export_auto.go called cmd.Run(), discarding
stdout + stderr. The caller at line 113 then formatted the warning as
"Warning: auto-export: git add failed: %v" — leaving users with just
"exit status 1" and no diagnostic to act on. Common real-world
triggers (paths-are-ignored, index.lock contention, detached HEAD)
all leave their signal in stderr, which we were dropping.

Switch to cmd.CombinedOutput() and wrap any non-nil error with the
trimmed output via fmt.Errorf("%w: %s", err, trimmed). Function
signature stays `error`, caller's existing %v formatter naturally
prints "exit status 1: <git's stderr>". No change for the success
path, no change for callers.

Add TestGitAddFile_CapturesStderrOnFailure: temp git repo with
.gitignore excluding .beads/, asserts the returned error contains
"ignored" (proxy for git's stderr making it through). Regression
guard so a future refactor doesn't silently drop the captured output
again.

Fixes #3504
2026-04-25 22:10:56 -04:00
Kevin Glynn
eef21dec2d
fix: remove duplicate wisp query in bd export (#3352) (#3372)
* fix: remove duplicate wisp query in bd export (#3352)

SearchIssues with Ephemeral=nil already queries both the issues and
wisps tables with ID-based dedup (issueops/search.go). The separate
Ephemeral=true query followed by append caused every wisp to appear
twice in the exported JSONL.

Remove the redundant wisp query from both export.go (manual export)
and export_auto.go (auto-export/exportToFile). The SearchIssues call
with Ephemeral=nil continues to include all wisps correctly.

* fix: suppress gosec G304 false positive in hookWorkTreeRoot

The path is filepath.Join(os.Getenv("GIT_DIR"), "gitdir") — reading a
well-known git internal file from a trusted environment variable set by
git itself during hook execution. Not a user-controlled file inclusion.

This false positive causes the Lint CI check to fail on every PR in
the repo, masking real lint issues.

* test(export): add regression test for duplicate wisp entries (GH#3352)

Addresses review feedback requesting test coverage for the duplicate-wisp
export fix. Creates 3 regular issues and 3 ephemeral wisps, runs export,
and asserts: no duplicate IDs in output, wisp count matches source count,
and total exported count equals expected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-25 11:41:45 -07:00
Kevin Glynn
8518e71d66
feat: BD_JSON_ENVELOPE=1 opt-in for uniform JSON wrapping (#3393)
* feat: add BD_JSON_ENVELOPE=1 opt-in for uniform JSON wrapping

Follow-up to #3368 as discussed with @maphew. Adds the envelope format
that wraps all --json output as {"schema_version": N, "data": <payload>},
gated behind BD_JSON_ENVELOPE=1 to avoid breaking existing consumers.

Legacy mode (default): unchanged behavior from #3368 — objects get
schema_version injected inline, arrays pass through raw.

Envelope mode (BD_JSON_ENVELOPE=1): uniform wrapping for all types.
The original payload is untouched inside .data — no type corruption,
no field injection. A one-time deprecation notice is emitted to stderr
when --json is used without the env var.

Timeline: envelope becomes default in v2.0, with BD_JSON_ENVELOPE=0
as a temporary escape hatch for one release cycle.

Includes:
- wrapWithSchemaVersion dual-mode logic
- outputJSONError and buildJSONError envelope support
- Deprecation notice on stderr (once per process)
- Tests for both legacy and envelope modes
- Migration guide in docs/JSON_SCHEMA.md

* fix: suppress deprecation notice when stderr is not a TTY

Gate emitEnvelopeDeprecation() on ui.IsStderrTerminal() so the
"NOTE: bd --json output format will change in v2.0" message only
appears in interactive terminals. Test harnesses and pipelines
capture stderr via CombinedOutput(), and the trailing notice after
the JSON closing brace caused json.Unmarshal to fail with
"invalid character 'N' after top-level value" in 15 CI tests.

Follows the same TTY-gating pattern used by ShouldUseColor and
ShouldUseEmoji.

* fix: suppress gosec G304 false positive in hookWorkTreeRoot

The path is filepath.Join(os.Getenv("GIT_DIR"), "gitdir") — reading a
well-known git internal file from a trusted environment variable set by
git itself during hook execution. Not a user-controlled file inclusion.

This false positive causes the Lint CI check to fail on every PR in
the repo, masking real lint issues.
2026-04-21 13:24:17 -07:00
Michael Percy
d0f0ad6f22
fix(export): scrub git hook env and skip cross-worktree git-add (GH#3311) (#3347)
When export.git-add=true (the v1.0.1+ default), the pre-commit hook in
a worktree stages a spurious issues.jsonl at the repo root instead of
(or in addition to) .beads/issues.jsonl. The ghost file lands in the
commit's diff but not on disk — easy to miss and painful to clean up.

Root cause (not what the issue body guessed): the absolute path passed
to `git add` was always correct. The bug is env inheritance. Git
invokes the pre-commit hook with GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/
GIT_COMMON_DIR/GIT_PREFIX set in the environment, and bd's git-add
subprocesses inherit them.

Both call sites set `cmd.Dir = filepath.Dir(fullPath)` (= `.beads/`).
With GIT_DIR set in the environment and GIT_WORK_TREE unset, git
defaults the work-tree to cwd. So the subprocess's git sees `.beads/`
as the work-tree root and records `.beads/issues.jsonl` at the root of
that broken view — which ends up in the worktree's actual index as a
bare `issues.jsonl`. Verified empirically: with GIT_DIR set and
cwd=`.beads/`, `git rev-parse --show-toplevel` returns the `.beads/`
directory itself.

Fix, in two parts:

 1. scrubGitHookEnv() strips the vars that poison repo/worktree
    auto-discovery, index routing, object routing, and config
    injection:
      - discovery: GIT_DIR, GIT_WORK_TREE, GIT_COMMON_DIR, GIT_PREFIX,
        GIT_CEILING_DIRECTORIES, GIT_DISCOVERY_ACROSS_FILESYSTEM
      - index: GIT_INDEX_FILE
      - objects: GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES
      - config: the whole GIT_CONFIG* family (covers the case where the
        parent invoked `git -c core.worktree=… commit`, which sets
        GIT_CONFIG_PARAMETERS / GIT_CONFIG_COUNT in the child env)
    With these cleared, git rediscovers the repo/worktree from cwd.

 2. hookWorkTreeRoot() reads the inherited GIT_DIR to determine the
    worktree whose hook is currently firing. If the target path is
    outside that worktree, gitAddFile returns without staging. This
    prevents a narrower but real regression that would otherwise land
    after part 1 alone: in the .beads/redirect configuration, fullPath
    points into the main repo (not the worktree), and post-scrub git
    would silently stage the file into main's index from a worktree
    hook. The existing preCommitHookBody template (init_git_hooks.go)
    documents the same intent: "For worktrees: .beads is in the main
    repo's working tree... we skip it."

The guard uses pathInsideDir which handles the macOS symlinked-parent
case: on /tmp → /private/tmp, a fresh-file target expressed as
/tmp/.../new.txt compared against a worktree root resolved to
/private/tmp/... would otherwise misclassify as outside the worktree.
pathInsideDir resolves the parent of each side via EvalSymlinks and
reattaches the basename — sufficient for the real caller shapes, since
gitAddFile's parent directory (beadsDir or the export output dir)
always exists at call time.

Also fixes the variant reported in a follow-up comment where the
.beads/redirect workaround did not resolve the root ghost: the env
pollution broke git-add inside the `bd export -o` subprocess spawned
from exportJSONLForCommit (that subprocess has BD_GIT_HOOK stripped,
but git env vars preserved, so its PostRun auto-export hits the same
bug via the shared gitAddFile).

Deduplicates the parallel git-add invocation in exportJSONLForCommit
by delegating to gitAddFile, so both call sites share the env scrub
and the cross-worktree guard.

Adds:
 - TestGitAddFile_InWorktreeHook_StagesCorrectPath — end-to-end
   worktree regression test; fails on main, passes with the fix.
 - TestGitAddFile_RedirectCase_DoesNotStageInMainRepo — verifies
   the cross-worktree skip does not pollute main's index when a
   worktree has .beads/redirect pointing into main.
 - TestGitAddFile_NonHookContext_GuardDoesNotFire — regression guard
   confirming the cross-worktree skip is a no-op outside git hooks.
 - TestScrubGitHookEnv — unit test for the env scrub helper, covering
   the discovery/index/object vars and the GIT_CONFIG* family, and
   verifying non-discovery vars (author/committer/editor/pager) pass
   through untouched.
 - TestPathInsideDir — covers structural cases plus the macOS
   fresh-file + symlinked-parent regression.
 - TestHookWorkTreeRoot — covers linked-worktree gitdir file,
   plain-repo .git, bare/unrecognized, and not-a-hook contexts.

Workaround for users on older releases remains `BD_EXPORT_GIT_ADD=false`.

Closes GH#3311

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 21:08:13 -07:00
Kevin Glynn
27e9a85e48
fix: no-git-ops=true now implies export.git-add=false (#3314) (#3376)
When no-git-ops=true is set, bd still attempted to git add the
auto-export file on every write, producing a warning on every
invocation. Users had to set both no-git-ops=true and
export.git-add=false separately.

Check the no-git-ops flag in both the auto-export path
(export_auto.go) and the pre-commit hook export path (hooks.go).
When no-git-ops is true, skip the git add without warning.

Co-authored-by: kev <kglynn@pryoninc.com>
2026-04-20 17:42:42 -07:00
seanmartinsmith
f990135081
fix(export): add _type discriminator to JSONL issue lines (#3272)
* fix(export): add _type discriminator to JSONL issue lines (GH#3271)

bd export produces JSONL with mixed schemas: issue lines and memory lines.
Memory lines carry "_type":"memory" but issue lines had no _type field,
causing downstream consumers to crash when they encounter an unexpected
schema (e.g. json.loads(line)["status"] throws KeyError on memory lines).

Add "_type":"issue" to all exported issue records via an exportIssueRecord
wrapper struct. This makes every JSONL line self-describing without changing
the Issue or IssueWithCounts types used elsewhere.

Both export paths updated: manual (runExport) and auto (exportToFile).
Import already handles this gracefully - unknown fields are ignored during
json.Unmarshal into types.Issue.

* test(export): add _type discriminator assertion to embedded export tests

Verifies that every JSONL line from bd export carries a _type field
("issue" or "memory") and that each type has its expected key fields.
Addresses codecov 0% patch coverage flag on PR #3272.
2026-04-20 17:40:34 -07:00
coffeegoddd☕️✨
fa87dd2bdd feat: add global beads_global database for shared-server mode
Adds a project-agnostic beads_global database to shared-server mode so
clients can track issues globally across all projects on the server.

- Add EnsureGlobalDatabase() to create the DB during shared server init
- Add --global persistent flag to switch CLI commands to the global DB
- Add global_dolt_database field to metadata.json for connection info
- Seed global DB with issue_prefix="global" and sentinel project ID
- Export/import default to global-issues.jsonl with --global flag
- --global errors when not in shared-server mode (additive, no breakage)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 09:18:43 -07:00
matt wilkie
64e504d541
fix: enable auto-export by default and add init prompt (GH#2973) (#3204)
* fix: enable auto-export by default with canonical issues.jsonl name (GH#2973)

Users expected JSONL export to refresh automatically but export.auto
defaulted to false, requiring explicit opt-in. Also the default filename
(export.jsonl) didn't match the canonical issues.jsonl used in docs and
git hooks. Change defaults: export.auto=true, export.path=issues.jsonl,
export.git-add=true so viewers (bv) and git workflows see fresh data
without extra configuration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(init): add auto-export prompt and document export.* config (GH#2973)

Interactive bd init now asks whether to enable auto-export (default: yes).
Users can opt out during setup instead of discovering the setting later.

Also documents the export.* config namespace in bd config --help with
all four keys (export.auto, export.path, export.interval, export.git-add)
and adds export examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: document export.* auto-export config keys (GH#2973)

Document the export.auto/path/interval/git-add keys in docs/CONFIG.md
and note the default-on switch in CHANGELOG.md [Unreleased].

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(export): don't prime throttle on empty init and skip git-add without git (GH#2973)

Review feedback on #3204 surfaced two regressions in the default-on
auto-export path:

1. bd init's PersistentPostRun fired maybeAutoExport with zero issues,
   writing an empty .beads/issues.jsonl and saving export-state.json with
   the current dolt commit + timestamp. The next `bd create` was then
   throttled for up to export.interval (default 60s) because the
   timestamp check in maybeAutoExport returned before considering the new
   commit, leaving issues.jsonl stale on fresh repos.

   Skip saving state and remove the empty file when the export produced
   no issues and no memories. The next write runs a real export
   immediately.

2. export.git-add defaulting to true emitted
   `Warning: auto-export: git add failed: exit status 128` on every
   write in the supported standalone BEADS_DIR flow where init
   intentionally skips `git init`. Gate the git add call on isGitRepo()
   so standalone users get silent, working auto-export.

Adds a scripttest regression guard for the init -> immediate create
path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(memory): disable auto-export in concurrent test (GH#2973)

TestEmbeddedMemoryConcurrent started failing on this branch after
auto-export became the default: every concurrent 'bd remember'/'bd
forget' call now fires maybeAutoExport, which reads store state after
the write. Under 8-worker contention this intermittently leaves a
forgotten key visible to 'bd memories'.

The test exercises memory concurrency, not export behavior; opt it
out of auto-export so the signal stays focused. Followup: investigate
the race between write commit and post-write auto-export read in
embedded Dolt concurrent paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 16:02:58 -07:00
matt wilkie
b8ec14d29c
fix: surface auto-export and auto-backup errors to stderr (#3122)
* fix: make auto-export and auto-backup errors visible on stderr

Previously, failures in auto-export (JSONL flush) and auto-backup were
logged only via debug.Logf, invisible to normal users. One user lost 46
issues because JSONL export silently stopped working with no warnings.

Upgrade critical error paths from debug.Logf to fmt.Fprintf(os.Stderr)
so users see warnings when their backup/export pipeline breaks. Keep
debug.Logf for non-critical diagnostics (throttle, no-change skip).

Closes #2489

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: remove stale sync references

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 20:37:13 -07:00
Jing-a-ling
05a90697fb fix: include comment bodies in bd export JSONL output
bd export populated comment_count but never called GetCommentsForIssues()
to populate the Comments field on each issue. This meant bd export | bd import
silently lost all comment history. Now both export.go and export_auto.go
bulk-load comments alongside labels and dependencies.

Fixes #3007

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 07:07:04 +10:00
Julian Knutsen
9d9d0e53c2
fix: unwrap HookFiringStore decorator before optional interface type assertions (#2937)
The HookFiringStore decorator wraps DoltStorage for automatic hook
firing, but hides the concrete store's optional interfaces
(StoreLocator, BackupStore, Flattener, Compactor, etc.) from type
assertions. This caused "storage backend does not support X" errors
for flatten, backup, compact, dolt remote, and other operations.

Add storage.UnwrapStore() helper that peels the decorator before type
assertions, and apply it at all 24 callsites in cmd/bd/.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 15:12:59 -07:00
paf0186
0fdb6a76bd
feat: auto-export git-tracked JSONL for portability (fix-merge PR #2808)
When `export.auto: true` is set in config.yaml, beads automatically
writes a git-tracked JSONL file (issues + memories) after mutations.
This provides full portability across machines via git alone, without
requiring a Dolt remote.

New config keys:
  export.auto: false       # opt-in to auto-export
  export.interval: 60s     # throttle interval
  export.path: export.jsonl  # output file relative to .beads/
  export.git-add: false    # auto-stage the file

The auto-export:
- Runs in PersistentPostRun after auto-backup
- Uses Dolt commit hash for change detection (no-op if nothing changed)
- Throttles by interval (default 60s)
- Skips during git hooks (BD_GIT_HOOK=1)
- Includes both issues and memories in the JSONL output

Also extracts exportToFile() as a shared helper usable by both
`bd export -o` and the auto-export system.

Files:
- cmd/bd/export_auto.go: Core auto-export logic
- cmd/bd/main.go: Wire maybeAutoExport into PersistentPostRun
- internal/config/config.go: New export.* config defaults
- internal/config/yaml_config.go: Add "export." to yaml-only prefix list
- cmd/bd/doctor/gitignore.go: Add export-state.json to gitignore
- cmd/bd/doctor/tracked_runtime.go: Flag export-state.json if tracked

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 19:36:48 -07:00