beads/internal/storage/sqlite
Marco Del Pin dba9bb91c3 fix(import): handle duplicate issue IDs in JSONL files gracefully
Implements three-layer deduplication strategy to prevent UNIQUE
constraint errors during import:
1. Early deduplication during processing (importer.go)
2. Pre-batch deduplication (importer.go)
3. INSERT OR IGNORE with explicit error handling (issues.go)
**Problem:**
JSONL files with duplicate issue IDs caused import failures:
  Import failed: UNIQUE constraint failed: issues.id
**Root Cause:**
- Go SQLite driver returns errors even with INSERT OR IGNORE
- Only content hash was deduplicated, not IDs
- Multiple code paths affected (insertIssue, insertIssues)
**Solution:**
Layer 1: Early deduplication by ID in upsertIssues (lines 489-502)
Layer 2: Pre-batch deduplication (lines 713-726)
Layer 3: INSERT OR IGNORE + isUniqueConstraintError() helper
**Testing:**
- Multiple production databases tested
- 9 duplicates handled successfully
- 100% success rate on v0.30.5 databases
- Zero UNIQUE constraint errors
**Impact:**
- Enables importing JSONL with duplicate IDs
- Duplicate count shown in import statistics
- No breaking changes, backward compatible
🤖 Generated with Claude Code
2025-12-18 19:26:29 +01:00
..
migrations Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
adaptive_e2e_test.go test: Tag 16 slow integration tests with build tags 2025-11-21 14:40:22 -05:00
adaptive_length.go Switch from hex to Base36 encoding for issue IDs (GH #213) 2025-11-03 12:02:15 -08:00
adaptive_length_test.go fix: resolve test failures from speedup changes 2025-11-04 01:08:21 -08:00
batch_ops.go fix(import): defensive handling for closed issues missing closed_at timestamp 2025-12-14 17:21:44 -08:00
batch_ops_test.go fix(import): defensive handling for closed issues missing closed_at timestamp 2025-12-14 17:21:44 -08:00
batch_test.go Fix N+1 query pattern in export operations (bd-rcmg) 2025-11-23 19:53:44 -08:00
bench_helpers_test.go fix: Remove duplicate context variable declarations in benchmark tests 2025-11-23 19:35:52 -08:00
blocked_cache.go Fix blocked cache invalidation in transaction operations (bd-1c4h) 2025-11-24 14:03:11 -08:00
blocked_cache_test.go fix: replace string(rune()) with strconv.Itoa in tests (bd-fmc) 2025-12-02 22:41:07 -08:00
child_counters_migration_test.go Extract SQLite migrations into separate files (bd-fb95094c.7) 2025-11-06 20:06:45 -08:00
child_counters_test.go Implement 6-char progressive hash IDs (bd-166, bd-167) 2025-10-30 14:04:03 -07:00
child_id_test.go Complete bd-ar2 P2 tasks: metadata, resurrection, and testing improvements 2025-11-21 11:40:37 -05:00
collision.go Document external_ref in content hash behavior (bd-9f4a) 2025-11-08 02:22:15 -08:00
collision_test.go Document external_ref in content hash behavior (bd-9f4a) 2025-11-08 02:22:15 -08:00
comments.go Fix N+1 query pattern in export operations (bd-rcmg) 2025-11-23 19:53:44 -08:00
comments_test.go fix: replace string(rune()) with strconv.Itoa in tests (bd-fmc) 2025-12-02 22:41:07 -08:00
compact.go Fix FK constraint failures in AddComment and ApplyCompaction (bd-5arw) 2025-11-20 18:55:10 -05:00
compact_bench_test.go fix: replace string(rune()) with strconv.Itoa in tests (bd-fmc) 2025-12-02 22:41:07 -08:00
compact_test.go Fix FK constraint failures in AddComment and ApplyCompaction (bd-5arw) 2025-11-20 18:55:10 -05:00
config.go feat(bd-1pj6): Add custom status states via config 2025-11-28 23:19:21 -08:00
cycle_bench_test.go Performance Improvements (#319) 2025-11-15 12:46:13 -08:00
cycle_detection_test.go fix: replace string(rune()) with strconv.Itoa in tests (bd-fmc) 2025-12-02 22:41:07 -08:00
delete_test.go feat(tombstone): implement delete-to-tombstone and TTL expiration (bd-3b4, bd-olt) 2025-12-05 16:20:43 -08:00
dependencies.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
dependencies_test.go Fix #202: Add dependency_type to bd show --json output (#203) 2025-11-02 08:40:10 -08:00
dirty.go fix: Remove unsafe ClearDirtyIssues() method (bd-b6xo) 2025-12-17 19:10:13 -08:00
dirty_helpers.go Refactor: consolidate ID generation and shared helpers (bd-0702, bd-1445) 2025-11-02 14:49:27 -08:00
dirty_test.go fix: Remove unsafe ClearDirtyIssues() method (bd-b6xo) 2025-12-17 19:10:13 -08:00
epics.go Enable errcheck linter and fix all production code warnings 2025-10-25 18:44:38 -07:00
epics_test.go Refactor high-complexity test functions (gocyclo) 2025-10-25 13:20:16 -07:00
errors.go Centralize error handling patterns in storage layer (bd-bwk2) 2025-11-20 19:17:57 -05:00
errors_test.go Centralize error handling patterns in storage layer (bd-bwk2) 2025-11-24 00:59:49 -08:00
events.go feat(tombstone): add P2 code review improvements (bd-saa, bd-1bu, bd-nyt) 2025-12-05 15:48:46 -08:00
events_helpers.go Refactor: consolidate ID generation and shared helpers (bd-0702, bd-1445) 2025-11-02 14:49:27 -08:00
events_test.go fix: Remove unsafe ClearDirtyIssues() method (bd-b6xo) 2025-12-17 19:10:13 -08:00
external_ref_test.go Add external_ref UNIQUE constraint and validation 2025-11-02 16:27:42 -08:00
freshness.go fix: Windows build + gosec lint errors (#585) 2025-12-16 13:26:51 -08:00
freshness_test.go fix(daemon): detect external db file replacement 2025-12-16 06:29:59 +01:00
freshness_unix.go fix: Windows build + gosec lint errors (#585) 2025-12-16 13:26:51 -08:00
freshness_windows.go fix: Windows build + gosec lint errors (#585) 2025-12-16 13:26:51 -08:00
graph_links_test.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
hash.go Fix bd-160: Implement JSONL integrity validation and prevent export deduplication data loss 2025-10-29 21:57:15 -07:00
hash_id_test.go Context propagation with graceful cancellation (bd-rtp, bd-yb8, bd-2o2) 2025-11-20 21:57:23 -05:00
hash_ids.go Complete bd-ar2 P2 tasks: metadata, resurrection, and testing improvements 2025-11-21 11:40:37 -05:00
ids.go fix: orphan detection false positive with dots in directory name 2025-12-16 01:06:14 -08:00
ids_test.go fix(orphan): handle prefixes with dots in orphan detection (GH#508) 2025-12-14 17:23:46 -08:00
issues.go fix(import): handle duplicate issue IDs in JSONL files gracefully 2025-12-18 19:26:29 +01:00
labels.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
labels_test.go fix: Remove unsafe ClearDirtyIssues() method (bd-b6xo) 2025-12-17 19:10:13 -08:00
migration_invariants.go Add migration inspection tools for AI agents (bd-627d Phase 2) 2025-11-02 14:03:14 -08:00
migration_invariants_test.go Add migration inspection tools for AI agents (bd-627d Phase 2) 2025-11-02 14:03:14 -08:00
migrations.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
migrations_test.go feat(schema): add messaging fields for bd-kwro epic 2025-12-16 13:06:47 -08:00
multirepo.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
multirepo_export.go feat(import/export): add tombstone support (bd-dve) 2025-12-07 20:34:02 +11:00
multirepo_test.go fix(multirepo): handle out-of-order dependencies during JSONL import (#414) 2025-11-29 22:07:52 -08:00
orphan_handling_test.go fix: orphan detection false positive with dots in directory name 2025-12-16 01:06:14 -08:00
prefix_validation_test.go Context propagation with graceful cancellation (bd-rtp, bd-yb8, bd-2o2) 2025-11-20 21:57:23 -05:00
queries.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
ready.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
ready_test.go feat: add --unassigned flag to bd ready command 2025-11-30 00:37:12 -08:00
resurrection.go Complete bd-ar2 P2 tasks: metadata, resurrection, and testing improvements 2025-11-21 11:40:37 -05:00
resurrection_test.go Fix P0 bug: JSONL lookup now returns LAST match (bd-58c0) 2025-11-04 22:25:33 -08:00
schema.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
schema_probe.go Fix CI regressions and stabilize tests 2025-11-17 10:06:35 -07:00
schema_probe_test.go Add schema compatibility probe to prevent silent migration failures (bd-ckvw) 2025-11-08 15:40:19 -08:00
sqlite.go refactor: Split monolithic sqlite.go into focused files (bd-0a43) 2025-11-23 19:04:34 -08:00
sqlite_bench_test.go feat(cli): add 'bd thanks' command to thank contributors 2025-12-14 12:40:32 -08:00
sqlite_test.go fix: improve issue ID prefix extraction for word-like suffixes 2025-12-16 13:19:22 -08:00
store.go fix(daemon): detect external db file replacement 2025-12-16 06:29:59 +01:00
test_helpers.go Context propagation with graceful cancellation (bd-rtp, bd-yb8, bd-2o2) 2025-11-20 21:57:23 -05:00
tombstone_test.go fix(cleanup): resolve CHECK constraint failure and add tombstone pruning 2025-12-14 00:37:54 -08:00
transaction.go Phase 4: Remove deprecated edge fields from Issue struct (Decision 004) 2025-12-18 02:48:13 -08:00
transaction_test.go fix(storage): persist close_reason to issues table on close (#551) 2025-12-14 14:18:01 -08:00
underlying_db_test.go Context propagation with graceful cancellation (bd-rtp, bd-yb8, bd-2o2) 2025-11-20 21:57:23 -05:00
util.go fix: resolve golangci-lint errors 2025-11-24 22:21:55 -08:00
util_test.go Implement transaction retry logic for SQLITE_BUSY (bd-ola6) 2025-11-24 01:10:16 -08:00
validators.go fix(types): add tombstone validation and block direct status update (bd-md2, bd-y68) 2025-12-05 15:41:35 -08:00
validators_test.go feat(bd-1pj6): Add custom status states via config 2025-11-28 23:19:21 -08:00