cleanup: delete deprecated SQLite-era docs (EXTENDING.md, MULTI_REPO_HYDRATION.md)

These documents described SQLite-specific features that no longer exist
after the Dolt migration. Preserved in git history. Updated references
in ARCHITECTURE.md and FAQ.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Executed-By: beads/crew/emma
Rig: beads
Role: crew
This commit is contained in:
emma 2026-02-16 01:16:44 -08:00 • committed by Steve Yegge
commit 5083873779
4 changed files with 2 additions and 1051 deletions

View file

@ -379,7 +379,7 @@ The `bd mol squash` command uses hard delete intentionally - tombstones would be
- [MOLECULES.md](MOLECULES.md) - Molecular chemistry metaphor (protos, pour, bond, squash, burn)
- [INTERNALS.md](INTERNALS.md) - FlushManager, Blocked Cache implementation details
- [DAEMON.md](DAEMON.md) - Daemon management and configuration
- [EXTENDING.md](EXTENDING.md) - Extending bd (deprecated, SQLite-era)
- [ADVANCED.md](ADVANCED.md) - Advanced features and configuration
- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Recovery procedures and common issues
- [FAQ.md](FAQ.md) - Common questions about the architecture
- [COLLISION_MATH.md](COLLISION_MATH.md) - Hash collision probability analysis

View file

@ -1,731 +0,0 @@
# Extending bd with Custom Tables
> **⚠️ DEPRECATED**: This document describes extending bd via custom SQLite tables.
> This approach is being deprecated as bd migrates from SQLite to Dolt as the
> storage backend. Dolt does not support hosting custom tables in the same database.
>
> **For external integrations** (syncing with Jira, Shortcut, Linear, etc.), build
> a standalone sync tool following the pattern of [jira-beads-sync](https://github.com/conallob/jira-beads-sync).
> These tools use bd's CLI with `--json` flags or read/write the `.beads/` JSONL files directly.
>
> **For orchestration layers**, consider using a separate database that references
> bd issue IDs, or wait for the forthcoming plugin architecture.
For SQLite-backend users, bd can be extended by applications that need more than basic issue tracking. The pattern below adds custom tables to the same SQLite database that bd uses. **This does not apply to the Dolt backend.**
## Philosophy
**bd is focused** - It tracks issues, dependencies, and ready work. That's it.
**Your application adds orchestration** - Execution state, agent assignments, retry logic, etc.
**Shared database = simple queries** - Join `issues` with your tables for powerful queries.
This is the same pattern used by tools like Temporal (workflow + activity tables) and Metabase (core + plugin tables).
## Quick Example
```sql
-- Create your application's tables in the same database
CREATE TABLE myapp_executions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
issue_id TEXT NOT NULL,
status TEXT NOT NULL, -- pending, running, failed, completed
agent_id TEXT,
started_at DATETIME,
completed_at DATETIME,
error TEXT,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
CREATE INDEX idx_executions_issue ON myapp_executions(issue_id);
CREATE INDEX idx_executions_status ON myapp_executions(status);
-- Query across layers
SELECT
i.id,
i.title,
i.priority,
e.status as execution_status,
e.agent_id,
e.started_at
FROM issues i
LEFT JOIN myapp_executions e ON i.id = e.issue_id
WHERE i.status = 'in_progress'
ORDER BY i.priority ASC;
```
## Integration Pattern
### 1. Initialize Your Database Schema
```go
package main
import (
"database/sql"
_ "modernc.org/sqlite"
)
const myAppSchema = `
-- Your application's tables
CREATE TABLE IF NOT EXISTS myapp_executions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
issue_id TEXT NOT NULL,
status TEXT NOT NULL,
agent_id TEXT,
started_at DATETIME,
completed_at DATETIME,
error TEXT,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS myapp_checkpoints (
id INTEGER PRIMARY KEY AUTOINCREMENT,
execution_id INTEGER NOT NULL,
step_name TEXT NOT NULL,
step_data TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (execution_id) REFERENCES myapp_executions(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_executions_issue ON myapp_executions(issue_id);
CREATE INDEX IF NOT EXISTS idx_executions_status ON myapp_executions(status);
CREATE INDEX IF NOT EXISTS idx_checkpoints_execution ON myapp_checkpoints(execution_id);
`
func InitializeMyAppSchema(dbPath string) error {
db, err := sql.Open("sqlite3", dbPath)
if err != nil {
return err
}
defer db.Close()
_, err = db.Exec(myAppSchema)
return err
}
```
### 2. Use bd for Issue Management
```go
import (
"github.com/steveyegge/beads"
)
// Open bd's storage
ctx := context.Background()
store, err := beads.Open(ctx, dbPath)
if err != nil {
log.Fatal(err)
}
// Initialize your schema
if err := InitializeMyAppSchema(dbPath); err != nil {
log.Fatal(err)
}
// Use bd to find ready work
readyIssues, err := store.GetReadyWork(ctx, beads.WorkFilter{Limit: 10})
if err != nil {
log.Fatal(err)
}
// Use your tables for orchestration
for _, issue := range readyIssues {
execution := &Execution{
IssueID: issue.ID,
Status: "pending",
AgentID: selectAgent(),
StartedAt: time.Now(),
}
if err := createExecution(db, execution); err != nil {
log.Printf("Failed to create execution: %v", err)
}
}
```
### 3. Query Across Layers
```go
// Complex query joining bd's issues with your execution data
query := `
SELECT
i.id,
i.title,
i.priority,
i.status as issue_status,
e.id as execution_id,
e.status as execution_status,
e.agent_id,
e.error,
COUNT(c.id) as checkpoint_count
FROM issues i
INNER JOIN myapp_executions e ON i.id = e.issue_id
LEFT JOIN myapp_checkpoints c ON e.id = c.execution_id
WHERE e.status = 'running'
GROUP BY i.id, e.id
ORDER BY i.priority ASC, e.started_at ASC
`
rows, err := db.Query(query)
// Process results...
```
## Real-World Example: VC Orchestrator
Here's how the VC (VibeCoder) orchestrator extends bd using `UnderlyingDB()`:
```go
package vc
import (
"database/sql"
"github.com/steveyegge/beads"
_ "modernc.org/sqlite"
)
type VCStorage struct {
beads.Storage // Embed bd's storage
db *sql.DB // Cache the underlying DB
}
func NewVCStorage(dbPath string) (*VCStorage, error) {
// Open bd's storage
ctx := context.Background()
store, err := beads.Open(ctx, dbPath)
if err != nil {
return nil, err
}
vc := &VCStorage{
Storage: store,
db: store.UnderlyingDB(),
}
// Create VC-specific tables
if err := vc.initSchema(); err != nil {
return nil, err
}
return vc, nil
}
func (vc *VCStorage) initSchema() error {
schema := `
-- VC's orchestration layer
CREATE TABLE IF NOT EXISTS vc_executor_instances (
id TEXT PRIMARY KEY,
issue_id TEXT NOT NULL,
executor_type TEXT NOT NULL,
status TEXT NOT NULL, -- pending, assessing, executing, analyzing, completed, failed
agent_name TEXT,
created_at DATETIME NOT NULL,
claimed_at DATETIME,
completed_at DATETIME,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS vc_execution_state (
id INTEGER PRIMARY KEY AUTOINCREMENT,
executor_id TEXT NOT NULL,
phase TEXT NOT NULL, -- assessment, execution, analysis
state_data TEXT NOT NULL, -- JSON checkpoint data
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (executor_id) REFERENCES vc_executor_instances(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_vc_executor_issue ON vc_executor_instances(issue_id);
CREATE INDEX IF NOT EXISTS idx_vc_executor_status ON vc_executor_instances(status);
CREATE INDEX IF NOT EXISTS idx_vc_execution_executor ON vc_execution_state(executor_id);
`
_, err := vc.db.Exec(schema)
return err
}
// ClaimReadyWork atomically claims the highest priority ready work
func (vc *VCStorage) ClaimReadyWork(agentName string) (*ExecutorInstance, error) {
query := `
UPDATE vc_executor_instances
SET status = 'executing', claimed_at = CURRENT_TIMESTAMP, agent_name = ?
WHERE id = (
SELECT ei.id
FROM vc_executor_instances ei
JOIN issues i ON ei.issue_id = i.id
WHERE ei.status = 'pending'
AND NOT EXISTS (
SELECT 1 FROM dependencies d
JOIN issues blocked ON d.depends_on_id = blocked.id
WHERE d.issue_id = i.id
AND d.type = 'blocks'
AND blocked.status IN ('open', 'in_progress', 'blocked')
)
ORDER BY i.priority ASC
LIMIT 1
)
RETURNING id, issue_id, executor_type, status, agent_name, claimed_at
`
var ei ExecutorInstance
err := vc.db.QueryRow(query, agentName).Scan(
&ei.ID, &ei.IssueID, &ei.ExecutorType,
&ei.Status, &ei.AgentName, &ei.ClaimedAt,
)
return &ei, err
}
```
**Key benefits of this approach:**
- ✅ VC extends bd without forking or modifying it
- ✅ Single database = simple JOINs across layers
- ✅ Foreign keys ensure referential integrity
- ✅ bd handles issue tracking, VC handles orchestration
- ✅ Can use bd's CLI alongside VC's custom operations
## Best Practices
### 1. Namespace Your Tables
Prefix your tables with your application name to avoid conflicts:
```sql
-- Good
CREATE TABLE vc_executions (...);
CREATE TABLE myapp_checkpoints (...);
-- Bad
CREATE TABLE executions (...); -- Could conflict with other apps
CREATE TABLE state (...); -- Too generic
```
### 2. Use Foreign Keys
Always link your tables to `issues` with foreign keys:
```sql
CREATE TABLE myapp_executions (
issue_id TEXT NOT NULL,
-- ...
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
```
This ensures:
- Referential integrity
- Automatic cleanup when issues are deleted
- Ability to join with `issues` table
### 3. Index Your Query Patterns
Add indexes for common queries:
```sql
-- If you query by status frequently
CREATE INDEX idx_executions_status ON myapp_executions(status);
-- If you join on issue_id
CREATE INDEX idx_executions_issue ON myapp_executions(issue_id);
-- Composite index for complex queries
CREATE INDEX idx_executions_status_priority
ON myapp_executions(status, issue_id);
```
### 4. Don't Duplicate bd's Data
Don't copy fields from `issues` into your tables. Instead, join:
```sql
-- Bad: Duplicating data
CREATE TABLE myapp_executions (
issue_id TEXT NOT NULL,
issue_title TEXT, -- Don't do this!
issue_priority INTEGER, -- Don't do this!
-- ...
);
-- Good: Join when querying
SELECT i.title, i.priority, e.status
FROM myapp_executions e
JOIN issues i ON e.issue_id = i.id;
```
### 5. Use JSON for Flexible State
SQLite supports JSON functions, great for checkpoint data:
```sql
CREATE TABLE myapp_checkpoints (
id INTEGER PRIMARY KEY,
execution_id INTEGER NOT NULL,
step_name TEXT NOT NULL,
step_data TEXT, -- Store as JSON
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
-- Query JSON fields
SELECT
id,
json_extract(step_data, '$.completed') as completed,
json_extract(step_data, '$.error') as error
FROM myapp_checkpoints
WHERE step_name = 'assessment';
```
## Common Patterns
### Pattern 1: Execution Tracking
Track which agent is working on which issue:
```sql
CREATE TABLE myapp_executions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
issue_id TEXT NOT NULL UNIQUE, -- One execution per issue
agent_id TEXT NOT NULL,
status TEXT NOT NULL,
started_at DATETIME NOT NULL,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
-- Claim an issue for execution
INSERT INTO myapp_executions (issue_id, agent_id, status, started_at)
VALUES (?, ?, 'running', CURRENT_TIMESTAMP)
ON CONFLICT (issue_id) DO UPDATE
SET agent_id = excluded.agent_id, started_at = CURRENT_TIMESTAMP;
```
### Pattern 2: Checkpoint/Resume
Store execution checkpoints for crash recovery:
```sql
CREATE TABLE myapp_checkpoints (
id INTEGER PRIMARY KEY AUTOINCREMENT,
execution_id INTEGER NOT NULL,
phase TEXT NOT NULL,
checkpoint_data TEXT NOT NULL, -- JSON
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (execution_id) REFERENCES myapp_executions(id) ON DELETE CASCADE
);
-- Latest checkpoint for an execution
SELECT checkpoint_data
FROM myapp_checkpoints
WHERE execution_id = ?
ORDER BY created_at DESC
LIMIT 1;
```
### Pattern 3: Result Storage
Store execution results linked to issues:
```sql
CREATE TABLE myapp_results (
id INTEGER PRIMARY KEY AUTOINCREMENT,
issue_id TEXT NOT NULL,
result_type TEXT NOT NULL, -- success, partial, failed
output_data TEXT, -- JSON: files changed, tests run, etc.
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
-- Get all results for an issue
SELECT result_type, output_data, created_at
FROM myapp_results
WHERE issue_id = ?
ORDER BY created_at DESC;
```
## Programmatic Access
Use bd's `--json` flags for scripting:
```bash
#!/bin/bash
# Find ready work
READY=$(bd ready --limit 1 --json)
ISSUE_ID=$(echo $READY | jq -r '.[0].id')
if [ "$ISSUE_ID" = "null" ]; then
echo "No ready work"
exit 0
fi
# Create execution record in your table
sqlite3 .beads/myapp.db <<SQL
INSERT INTO myapp_executions (issue_id, agent_id, status, started_at)
VALUES ('$ISSUE_ID', 'agent-1', 'running', datetime('now'));
SQL
# Claim issue in bd
bd update $ISSUE_ID --status in_progress
# Execute work...
echo "Working on $ISSUE_ID"
# Mark complete
bd close $ISSUE_ID --reason "Completed by agent-1"
sqlite3 .beads/myapp.db <<SQL
UPDATE myapp_executions
SET status = 'completed', completed_at = datetime('now')
WHERE issue_id = '$ISSUE_ID';
SQL
```
## Direct Database Access
### Using UnderlyingDB() (Recommended)
The recommended way to extend bd is using the `UnderlyingDB()` method on the storage instance. This gives you access to the same database connection that bd uses, ensuring consistency and avoiding connection overhead:
```go
import (
"database/sql"
"github.com/steveyegge/beads"
_ "modernc.org/sqlite"
)
// Open bd's storage
ctx := context.Background()
store, err := beads.Open(ctx, ".beads/dolt")
if err != nil {
log.Fatal(err)
}
defer store.Close()
// Get the underlying database connection
db := store.UnderlyingDB()
// Create your extension tables using the same connection
schema := `
CREATE TABLE IF NOT EXISTS myapp_executions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
issue_id TEXT NOT NULL,
status TEXT NOT NULL,
agent_id TEXT,
started_at DATETIME,
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
);
`
if _, err := db.Exec(schema); err != nil {
log.Fatal(err)
}
// Query bd's tables
var title string
var priority int
err = db.QueryRow(`
SELECT title, priority FROM issues WHERE id = ?
`, issueID).Scan(&title, &priority)
// Update your tables
_, err = db.Exec(`
INSERT INTO myapp_executions (issue_id, status, agent_id, started_at)
VALUES (?, ?, ?, CURRENT_TIMESTAMP)
`, issueID, "running", "agent-1")
// Join across layers
rows, err := db.Query(`
SELECT i.id, i.title, e.status, e.agent_id
FROM issues i
JOIN myapp_executions e ON i.id = e.issue_id
WHERE e.status = 'running'
`)
```
**Safety warnings when using UnderlyingDB():**
⚠️ **NEVER** close the database connection returned by `UnderlyingDB()`. The storage instance owns this connection.
⚠️ **DO NOT** modify database pool settings (SetMaxOpenConns, SetConnMaxIdleTime) or SQLite PRAGMAs (WAL mode, journal settings) as this affects bd's core operations.
⚠️ **Keep transactions short** - Long write transactions will block bd's core operations. Use read transactions when possible.
⚠️ **Expect errors after Close()** - Once you call `store.Close()`, operations on the underlying DB will fail. Use context cancellation to coordinate shutdown.
✅ **DO** use foreign keys to reference bd's tables for referential integrity.
✅ **DO** namespace your tables with your app name (e.g., `myapp_executions`).
✅ **DO** create indexes for your query patterns.
### When to use UnderlyingDB() vs sql.Open()
**Use `UnderlyingDB()`:**
- ✅ When you want to share the storage connection
- ✅ When you need tables in the same database as bd
- ✅ When you want automatic lifecycle management
- ✅ For most extension use cases (like VC)
**Use `sql.Open()` separately:**
- When you need independent connection pool settings
- When you need different timeout/retry behavior
- When you're managing multiple databases
- When you need fine-grained connection control
### Alternative: Independent Connection
If you need independent connection management, you can still open the database directly:
```go
import (
"database/sql"
_ "modernc.org/sqlite"
"github.com/steveyegge/beads"
)
// Auto-discover bd's database path
dbPath := beads.FindDatabasePath()
if dbPath == "" {
log.Fatal("No bd database found. Run 'bd init' first.")
}
// Open your own connection to the same database
db, err := sql.Open("sqlite3", dbPath)
if err != nil {
log.Fatal(err)
}
defer db.Close()
// Configure your connection independently
db.SetMaxOpenConns(10)
db.SetConnMaxIdleTime(time.Minute)
// Query bd's tables
var title string
var priority int
err = db.QueryRow(`
SELECT title, priority FROM issues WHERE id = ?
`, issueID).Scan(&title, &priority)
// Find corresponding JSONL path (for git hooks, monitoring, etc.)
jsonlPath := beads.FindJSONLPath(dbPath)
fmt.Printf("BD exports to: %s\n", jsonlPath)
```
## Batch Operations for Performance
When creating many issues at once (e.g., bulk imports, batch processing), use `CreateIssues` for significantly better performance:
```go
import (
"context"
"github.com/steveyegge/beads/internal/storage/sqlite"
"github.com/steveyegge/beads/internal/types"
)
// Open bd's storage
store, err := sqlite.New(".beads/issues.db")
if err != nil {
log.Fatal(err)
}
ctx := context.Background()
// Prepare batch of issues to create
issues := make([]*types.Issue, 0, 1000)
for _, item := range externalData {
issue := &types.Issue{
Title: item.Title,
Description: item.Description,
Status: types.StatusOpen,
Priority: item.Priority,
IssueType: types.TypeTask,
}
issues = append(issues, issue)
}
// Create all issues in a single atomic transaction (5-15x faster!)
if err := store.CreateIssues(ctx, issues, "import"); err != nil {
log.Fatal(err)
}
// REMOVED (bd-c7af): SyncAllCounters - no longer needed with hash IDs
```
### Performance Comparison
| Operation | CreateIssue Loop | CreateIssues Batch | Speedup |
|-----------|------------------|---------------------|---------|
| 100 issues | ~900ms | ~30ms | 30x |
| 500 issues | ~4.5s | ~800ms | 5.6x |
| 1000 issues | ~9s | ~950ms | 9.5x |
### When to Use Each Method
**Use `CreateIssue` (single issue):**
- Interactive CLI commands (`bd create`)
- Single issue creation in your app
- User-facing operations
**Use `CreateIssues` (batch):**
- Bulk imports from external systems
- Batch processing workflows
- Creating multiple related issues at once
- Agent workflows that generate many issues
### Batch Import Pattern
```go
// Example: Import from external issue tracker
func ImportFromExternal(externalIssues []ExternalIssue) error {
store, err := sqlite.New(".beads/issues.db")
if err != nil {
return err
}
ctx := context.Background()
// Convert external format to bd format
issues := make([]*types.Issue, 0, len(externalIssues))
for _, ext := range externalIssues {
issue := &types.Issue{
ID: fmt.Sprintf("import-%d", ext.ID), // Explicit IDs
Title: ext.Title,
Description: ext.Description,
Status: convertStatus(ext.Status),
Priority: ext.Priority,
IssueType: convertType(ext.Type),
}
// Normalize closed_at for closed issues
if issue.Status == types.StatusClosed {
closedAt := ext.ClosedAt
issue.ClosedAt = &closedAt
}
issues = append(issues, issue)
}
// Atomic batch create
if err := store.CreateIssues(ctx, issues, "external-import"); err != nil {
return fmt.Errorf("batch create failed: %w", err)
}
// REMOVED (bd-c7af): SyncAllCounters - no longer needed with hash IDs
return nil
}
```
## Summary
The key insight: **bd is a focused issue tracker, not a framework**.
By extending the database:
- You get powerful issue tracking for free
- Your app adds orchestration logic
- Simple SQL joins give you full visibility
- No tight coupling or version conflicts
This pattern scales from simple scripts to complex orchestrators like VC.
## See Also
- [README.md](../README.md) - Complete bd documentation
- [QUICKSTART.md](QUICKSTART.md) - Quick start tutorial
- Check out VC's implementation at `github.com/steveyegge/vc` for a real-world example

View file

@ -398,7 +398,7 @@ That's it! No PostgreSQL, no Redis, no Docker, no node_modules.
### Can I extend bd's database?
See [EXTENDING.md](EXTENDING.md) for the legacy SQLite extension approach (deprecated). With the Dolt backend, use `bd query` for direct SQL access or build integrations using `bd --json` CLI output.
With the Dolt backend, use `bd query` for direct SQL access or build integrations using `bd --json` CLI output.
### Does bd support Windows?

View file

@ -1,318 +0,0 @@
# Multi-Repo Hydration Layer
> **⚠️ DEPRECATED**: This document describes the SQLite-era multi-repo hydration layer.
> The `internal/storage/sqlite/` backend has been removed. The Dolt backend does not
> currently implement multi-repo hydration. This document is preserved for historical reference.
This document describes the implementation of Task 3 from the multi-repo support feature (bd-307): the hydration layer that loaded issues from multiple JSONL files into a unified SQLite database.
## Overview
The hydration layer enables beads to aggregate issues from multiple repositories into a single database for unified querying and analysis. It uses file modification time (mtime) caching to optimize performance by only reimporting files that have changed.
## Architecture
### 1. Database Schema
**Table: `repo_mtimes`**
```sql
CREATE TABLE repo_mtimes (
repo_path TEXT PRIMARY KEY, -- Absolute path to repository root
jsonl_path TEXT NOT NULL, -- Absolute path to .beads/issues.jsonl
mtime_ns INTEGER NOT NULL, -- Modification time in nanoseconds
last_checked DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
```
This table tracks the last known modification time of each repository's JSONL file to enable intelligent skip logic during hydration.
### 2. Configuration
Multi-repo mode is configured via `internal/config/config.go`:
```yaml
# .beads/config.yaml
repos:
primary: /path/to/primary/repo # Canonical source (optional)
additional: # Additional repos to hydrate from
- ~/projects/repo1
- ~/projects/repo2
```
- **Primary repo** (`.`): Issues from this repo are marked with `source_repo = "."`
- **Additional repos**: Issues marked with their relative path as `source_repo`
### 3. Implementation Files
**New Files:**
- `internal/storage/sqlite/multirepo.go` - Core hydration logic
- `internal/storage/sqlite/multirepo_test.go` - Test coverage
- `docs/MULTI_REPO_HYDRATION.md` - This document
**Modified Files:**
- `internal/storage/sqlite/schema.go` - Added `repo_mtimes` table
- `internal/storage/sqlite/migrations/013_repo_mtimes_table.go` - Migration for `repo_mtimes` table
- `internal/storage/sqlite/sqlite.go` - Integrated hydration into storage initialization
- `internal/storage/sqlite/ready.go` - Added `source_repo` to all SELECT queries
- `internal/storage/sqlite/labels.go` - Added `source_repo` to SELECT query
- `internal/storage/sqlite/migrations_test.go` - Added migration tests
## Key Functions
### `HydrateFromMultiRepo(ctx context.Context) (map[string]int, error)`
Main entry point for multi-repo hydration. Called automatically during `sqlite.New()`.
**Behavior:**
- Returns `nil, nil` if not in multi-repo mode (single-repo operation)
- Processes primary repo first (if configured)
- Then processes each additional repo
- Returns a map of `source_repo -> issue count` for imported issues
### `hydrateFromRepo(ctx, repoPath, sourceRepo string) (int, error)`
Handles hydration for a single repository.
**Steps:**
1. Resolves absolute path to repo and JSONL file
2. Checks file existence (skips if missing)
3. Compares current mtime with cached mtime
4. Skips import if mtime unchanged (optimization)
5. Imports issues if file changed or no cache exists
6. Updates mtime cache after successful import
### `importJSONLFile(ctx, jsonlPath, sourceRepo string) (int, error)`
Parses a JSONL file and imports all issues into the database.
**Features:**
- Handles large files (10MB max line size)
- Skips empty lines and comments (`#`)
- Sets `source_repo` field on all imported issues
- Computes `content_hash` if missing
- Uses transactions for atomicity
- Imports dependencies, labels, and comments
### `upsertIssueInTx(ctx, tx, issue *types.Issue) error`
Inserts or updates an issue within a transaction.
**Smart Update Logic:**
- Checks if issue exists by ID
- If new: inserts issue
- If exists: compares `content_hash` and only updates if changed
- Imports associated dependencies, labels, and comments
- Uses `INSERT OR IGNORE` for dependencies/labels to avoid duplicates
### `expandTilde(path string) (string, error)`
Utility function to expand `~` and `~/` paths to absolute home directory paths.
## Mtime Caching
The hydration layer uses file modification time (mtime) as a cache key to avoid unnecessary reimports.
**Cache Logic:**
1. First hydration: No cache exists → import file
2. Subsequent hydrations: Compare mtimes
- If `mtime_current == mtime_cached` → skip import (fast path)
- If `mtime_current != mtime_cached` → reimport (file changed)
3. After successful import: Update cache with new mtime
**Benefits:**
- **Performance**: Avoids parsing/importing unchanged JSONL files
- **Correctness**: Detects external changes via filesystem metadata
- **Simplicity**: No need for content hashing or git integration
**Limitations:**
- Relies on filesystem mtime accuracy
- Won't detect changes if mtime is manually reset
- Cross-platform mtime precision varies (nanosecond on Unix, ~100ns on Windows)
## Source Repo Tracking
Each issue has a `source_repo` field that identifies which repository it came from:
- **Primary repo**: `source_repo = "."`
- **Additional repos**: `source_repo = <relative_path>` (e.g., `~/projects/repo1`)
This enables:
- Filtering issues by source repository
- Understanding issue provenance in multi-repo setups
- Future features like repo-specific permissions or workflows
**Database Schema:**
```sql
ALTER TABLE issues ADD COLUMN source_repo TEXT DEFAULT '.';
CREATE INDEX idx_issues_source_repo ON issues(source_repo);
```
## Testing
Comprehensive test coverage in `internal/storage/sqlite/multirepo_test.go`:
### Test Cases
1. **`TestExpandTilde`**
- Verifies tilde expansion for various path formats
2. **`TestHydrateFromMultiRepo/single-repo_mode_returns_nil`**
- Confirms nil return when not in multi-repo mode
3. **`TestHydrateFromMultiRepo/hydrates_from_primary_repo`**
- Validates primary repo import
- Checks `source_repo = "."` is set correctly
4. **`TestHydrateFromMultiRepo/uses_mtime_caching_to_skip_unchanged_files`**
- First hydration: imports 1 issue
- Second hydration: imports 0 issues (cached)
- Proves mtime cache optimization works
5. **`TestHydrateFromMultiRepo/imports_additional_repos`**
- Creates primary + additional repo
- Verifies both are imported
- Checks source_repo fields are distinct
6. **`TestImportJSONLFile/imports_issues_with_dependencies_and_labels`**
- Tests JSONL parsing with complex data
- Validates dependencies and labels are imported
- Confirms relational data integrity
7. **`TestMigrateRepoMtimesTable`**
- Verifies migration creates table correctly
- Confirms migration is idempotent
### Running Tests
```bash
# Run all multirepo tests
go test -v ./internal/storage/sqlite -run TestHydrateFromMultiRepo
# Run specific test
go test -v ./internal/storage/sqlite -run TestExpandTilde
# Run all sqlite tests
go test ./internal/storage/sqlite
```
## Integration
### Automatic Hydration
Hydration happens automatically during storage initialization:
```go
// internal/storage/sqlite/sqlite.go
func New(path string) (*SQLiteStorage, error) {
// ... schema initialization ...
storage := &SQLiteStorage{db: db, dbPath: absPath}
// Skip for in-memory databases (used in tests)
if path != ":memory:" {
_, err := storage.HydrateFromMultiRepo(ctx)
if err != nil {
return nil, fmt.Errorf("failed to hydrate from multi-repo: %w", err)
}
}
return storage, nil
}
```
### Configuration Example
**`.beads/config.yaml`:**
```yaml
repos:
primary: /Users/alice/work/main-project
additional:
- ~/work/library-a
- ~/work/library-b
- /opt/shared/common-issues
```
**Resulting database:**
- Issues from `/Users/alice/work/main-project` → `source_repo = "."`
- Issues from `~/work/library-a` → `source_repo = "~/work/library-a"`
- Issues from `~/work/library-b` → `source_repo = "~/work/library-b"`
- Issues from `/opt/shared/common-issues` → `source_repo = "/opt/shared/common-issues"`
## Migration
The `repo_mtimes` table is created via standard migration system:
```go
// internal/storage/sqlite/migrations/013_repo_mtimes_table.go
func MigrateRepoMtimesTable(db *sql.DB) error {
// Check if table exists
var tableName string
err := db.QueryRow(`
SELECT name FROM sqlite_master
WHERE type='table' AND name='repo_mtimes'
`).Scan(&tableName)
if err == sql.ErrNoRows {
// Create table + index
_, err := db.Exec(`
CREATE TABLE repo_mtimes (...);
CREATE INDEX idx_repo_mtimes_checked ON repo_mtimes(last_checked);
`)
return err
}
return nil // Already exists
}
```
**Migration is idempotent**: Safe to run multiple times, won't error on existing table.
## Future Enhancements
1. **Incremental Sync**: Instead of full reimport, use git hashes or checksums to sync only changed issues
2. **Conflict Resolution**: Handle cases where same issue ID exists in multiple repos with different content
3. **Selective Hydration**: Allow users to specify which repos to hydrate (CLI flag or config)
4. **Background Refresh**: Periodically check for JSONL changes without blocking CLI operations
5. **Repository Metadata**: Track repo URL, branch, last commit hash for better provenance
## Performance Considerations
**Mtime Cache Hit (fast path):**
- 1 SQL query per repo (check cached mtime)
- No file I/O if mtime matches
- **Typical latency**: <1ms per repo
**Mtime Cache Miss (import path):**
- 1 SQL query (check cache)
- 1 file read (parse JSONL)
- N SQL inserts/updates (where N = issue count)
- 1 SQL update (cache mtime)
- **Typical latency**: 10-100ms for 100 issues
**Optimization Tips:**
- Place frequently-changing repos in primary position
- Use `.beads/config.yaml` instead of env vars (faster viper access)
- Limit `additional` repos to ~10 for reasonable startup time
## Troubleshooting
**Hydration not working?**
1. Check config: `bd config list` should show `repos.primary` or `repos.additional`
2. Verify JSONL exists: `ls -la /path/to/repo/.beads/issues.jsonl`
3. Check logs: Set `BD_DEBUG=1` to see hydration debug output
**Issues not updating?**
- Mtime cache might be stale
- Force refresh by deleting cache: `DELETE FROM repo_mtimes WHERE repo_path = '/path/to/repo'`
- Or touch the JSONL file: `touch /path/to/repo/.beads/issues.jsonl`
**Performance issues?**
- Check repo count: `SELECT COUNT(*) FROM repo_mtimes`
- Measure hydration time with `BD_DEBUG=1`
- Consider reducing `additional` repos if startup is slow
## See Also
- [CONFIG.md](CONFIG.md) - Configuration system documentation
- [EXTENDING.md](EXTENDING.md) - Database schema extension guide
- [bd-307](https://github.com/steveyegge/beads/issues/307) - Original multi-repo feature request