cleanup: delete deprecated SQLite-era docs (EXTENDING.md, MULTI_REPO_HYDRATION.md)
These documents described SQLite-specific features that no longer exist after the Dolt migration. Preserved in git history. Updated references in ARCHITECTURE.md and FAQ.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Executed-By: beads/crew/emma Rig: beads Role: crew
This commit is contained in:
parent
4103785ef3
commit
5083873779
4 changed files with 2 additions and 1051 deletions
|
|
@ -379,7 +379,7 @@ The `bd mol squash` command uses hard delete intentionally - tombstones would be
|
|||
- [MOLECULES.md](MOLECULES.md) - Molecular chemistry metaphor (protos, pour, bond, squash, burn)
|
||||
- [INTERNALS.md](INTERNALS.md) - FlushManager, Blocked Cache implementation details
|
||||
- [DAEMON.md](DAEMON.md) - Daemon management and configuration
|
||||
- [EXTENDING.md](EXTENDING.md) - Extending bd (deprecated, SQLite-era)
|
||||
- [ADVANCED.md](ADVANCED.md) - Advanced features and configuration
|
||||
- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Recovery procedures and common issues
|
||||
- [FAQ.md](FAQ.md) - Common questions about the architecture
|
||||
- [COLLISION_MATH.md](COLLISION_MATH.md) - Hash collision probability analysis
|
||||
|
|
|
|||
|
|
@ -1,731 +0,0 @@
|
|||
# Extending bd with Custom Tables
|
||||
|
||||
> **⚠️ DEPRECATED**: This document describes extending bd via custom SQLite tables.
|
||||
> This approach is being deprecated as bd migrates from SQLite to Dolt as the
|
||||
> storage backend. Dolt does not support hosting custom tables in the same database.
|
||||
>
|
||||
> **For external integrations** (syncing with Jira, Shortcut, Linear, etc.), build
|
||||
> a standalone sync tool following the pattern of [jira-beads-sync](https://github.com/conallob/jira-beads-sync).
|
||||
> These tools use bd's CLI with `--json` flags or read/write the `.beads/` JSONL files directly.
|
||||
>
|
||||
> **For orchestration layers**, consider using a separate database that references
|
||||
> bd issue IDs, or wait for the forthcoming plugin architecture.
|
||||
|
||||
For SQLite-backend users, bd can be extended by applications that need more than basic issue tracking. The pattern below adds custom tables to the same SQLite database that bd uses. **This does not apply to the Dolt backend.**
|
||||
|
||||
## Philosophy
|
||||
|
||||
**bd is focused** - It tracks issues, dependencies, and ready work. That's it.
|
||||
|
||||
**Your application adds orchestration** - Execution state, agent assignments, retry logic, etc.
|
||||
|
||||
**Shared database = simple queries** - Join `issues` with your tables for powerful queries.
|
||||
|
||||
This is the same pattern used by tools like Temporal (workflow + activity tables) and Metabase (core + plugin tables).
|
||||
|
||||
## Quick Example
|
||||
|
||||
```sql
|
||||
-- Create your application's tables in the same database
|
||||
CREATE TABLE myapp_executions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL, -- pending, running, failed, completed
|
||||
agent_id TEXT,
|
||||
started_at DATETIME,
|
||||
completed_at DATETIME,
|
||||
error TEXT,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_executions_issue ON myapp_executions(issue_id);
|
||||
CREATE INDEX idx_executions_status ON myapp_executions(status);
|
||||
|
||||
-- Query across layers
|
||||
SELECT
|
||||
i.id,
|
||||
i.title,
|
||||
i.priority,
|
||||
e.status as execution_status,
|
||||
e.agent_id,
|
||||
e.started_at
|
||||
FROM issues i
|
||||
LEFT JOIN myapp_executions e ON i.id = e.issue_id
|
||||
WHERE i.status = 'in_progress'
|
||||
ORDER BY i.priority ASC;
|
||||
```
|
||||
|
||||
## Integration Pattern
|
||||
|
||||
### 1. Initialize Your Database Schema
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
const myAppSchema = `
|
||||
-- Your application's tables
|
||||
CREATE TABLE IF NOT EXISTS myapp_executions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
agent_id TEXT,
|
||||
started_at DATETIME,
|
||||
completed_at DATETIME,
|
||||
error TEXT,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS myapp_checkpoints (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
execution_id INTEGER NOT NULL,
|
||||
step_name TEXT NOT NULL,
|
||||
step_data TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (execution_id) REFERENCES myapp_executions(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_issue ON myapp_executions(issue_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_status ON myapp_executions(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_checkpoints_execution ON myapp_checkpoints(execution_id);
|
||||
`
|
||||
|
||||
func InitializeMyAppSchema(dbPath string) error {
|
||||
db, err := sql.Open("sqlite3", dbPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
_, err = db.Exec(myAppSchema)
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Use bd for Issue Management
|
||||
|
||||
```go
|
||||
import (
|
||||
"github.com/steveyegge/beads"
|
||||
)
|
||||
|
||||
// Open bd's storage
|
||||
ctx := context.Background()
|
||||
store, err := beads.Open(ctx, dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Initialize your schema
|
||||
if err := InitializeMyAppSchema(dbPath); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Use bd to find ready work
|
||||
readyIssues, err := store.GetReadyWork(ctx, beads.WorkFilter{Limit: 10})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Use your tables for orchestration
|
||||
for _, issue := range readyIssues {
|
||||
execution := &Execution{
|
||||
IssueID: issue.ID,
|
||||
Status: "pending",
|
||||
AgentID: selectAgent(),
|
||||
StartedAt: time.Now(),
|
||||
}
|
||||
if err := createExecution(db, execution); err != nil {
|
||||
log.Printf("Failed to create execution: %v", err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Query Across Layers
|
||||
|
||||
```go
|
||||
// Complex query joining bd's issues with your execution data
|
||||
query := `
|
||||
SELECT
|
||||
i.id,
|
||||
i.title,
|
||||
i.priority,
|
||||
i.status as issue_status,
|
||||
e.id as execution_id,
|
||||
e.status as execution_status,
|
||||
e.agent_id,
|
||||
e.error,
|
||||
COUNT(c.id) as checkpoint_count
|
||||
FROM issues i
|
||||
INNER JOIN myapp_executions e ON i.id = e.issue_id
|
||||
LEFT JOIN myapp_checkpoints c ON e.id = c.execution_id
|
||||
WHERE e.status = 'running'
|
||||
GROUP BY i.id, e.id
|
||||
ORDER BY i.priority ASC, e.started_at ASC
|
||||
`
|
||||
|
||||
rows, err := db.Query(query)
|
||||
// Process results...
|
||||
```
|
||||
|
||||
## Real-World Example: VC Orchestrator
|
||||
|
||||
Here's how the VC (VibeCoder) orchestrator extends bd using `UnderlyingDB()`:
|
||||
|
||||
```go
|
||||
package vc
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"github.com/steveyegge/beads"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
type VCStorage struct {
|
||||
beads.Storage // Embed bd's storage
|
||||
db *sql.DB // Cache the underlying DB
|
||||
}
|
||||
|
||||
func NewVCStorage(dbPath string) (*VCStorage, error) {
|
||||
// Open bd's storage
|
||||
ctx := context.Background()
|
||||
store, err := beads.Open(ctx, dbPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
vc := &VCStorage{
|
||||
Storage: store,
|
||||
db: store.UnderlyingDB(),
|
||||
}
|
||||
|
||||
// Create VC-specific tables
|
||||
if err := vc.initSchema(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return vc, nil
|
||||
}
|
||||
|
||||
func (vc *VCStorage) initSchema() error {
|
||||
schema := `
|
||||
-- VC's orchestration layer
|
||||
CREATE TABLE IF NOT EXISTS vc_executor_instances (
|
||||
id TEXT PRIMARY KEY,
|
||||
issue_id TEXT NOT NULL,
|
||||
executor_type TEXT NOT NULL,
|
||||
status TEXT NOT NULL, -- pending, assessing, executing, analyzing, completed, failed
|
||||
agent_name TEXT,
|
||||
created_at DATETIME NOT NULL,
|
||||
claimed_at DATETIME,
|
||||
completed_at DATETIME,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS vc_execution_state (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
executor_id TEXT NOT NULL,
|
||||
phase TEXT NOT NULL, -- assessment, execution, analysis
|
||||
state_data TEXT NOT NULL, -- JSON checkpoint data
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (executor_id) REFERENCES vc_executor_instances(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_vc_executor_issue ON vc_executor_instances(issue_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_vc_executor_status ON vc_executor_instances(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_vc_execution_executor ON vc_execution_state(executor_id);
|
||||
`
|
||||
|
||||
_, err := vc.db.Exec(schema)
|
||||
return err
|
||||
}
|
||||
|
||||
// ClaimReadyWork atomically claims the highest priority ready work
|
||||
func (vc *VCStorage) ClaimReadyWork(agentName string) (*ExecutorInstance, error) {
|
||||
query := `
|
||||
UPDATE vc_executor_instances
|
||||
SET status = 'executing', claimed_at = CURRENT_TIMESTAMP, agent_name = ?
|
||||
WHERE id = (
|
||||
SELECT ei.id
|
||||
FROM vc_executor_instances ei
|
||||
JOIN issues i ON ei.issue_id = i.id
|
||||
WHERE ei.status = 'pending'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM dependencies d
|
||||
JOIN issues blocked ON d.depends_on_id = blocked.id
|
||||
WHERE d.issue_id = i.id
|
||||
AND d.type = 'blocks'
|
||||
AND blocked.status IN ('open', 'in_progress', 'blocked')
|
||||
)
|
||||
ORDER BY i.priority ASC
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING id, issue_id, executor_type, status, agent_name, claimed_at
|
||||
`
|
||||
|
||||
var ei ExecutorInstance
|
||||
err := vc.db.QueryRow(query, agentName).Scan(
|
||||
&ei.ID, &ei.IssueID, &ei.ExecutorType,
|
||||
&ei.Status, &ei.AgentName, &ei.ClaimedAt,
|
||||
)
|
||||
return &ei, err
|
||||
}
|
||||
```
|
||||
|
||||
**Key benefits of this approach:**
|
||||
- ✅ VC extends bd without forking or modifying it
|
||||
- ✅ Single database = simple JOINs across layers
|
||||
- ✅ Foreign keys ensure referential integrity
|
||||
- ✅ bd handles issue tracking, VC handles orchestration
|
||||
- ✅ Can use bd's CLI alongside VC's custom operations
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Namespace Your Tables
|
||||
|
||||
Prefix your tables with your application name to avoid conflicts:
|
||||
|
||||
```sql
|
||||
-- Good
|
||||
CREATE TABLE vc_executions (...);
|
||||
CREATE TABLE myapp_checkpoints (...);
|
||||
|
||||
-- Bad
|
||||
CREATE TABLE executions (...); -- Could conflict with other apps
|
||||
CREATE TABLE state (...); -- Too generic
|
||||
```
|
||||
|
||||
### 2. Use Foreign Keys
|
||||
|
||||
Always link your tables to `issues` with foreign keys:
|
||||
|
||||
```sql
|
||||
CREATE TABLE myapp_executions (
|
||||
issue_id TEXT NOT NULL,
|
||||
-- ...
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
```
|
||||
|
||||
This ensures:
|
||||
- Referential integrity
|
||||
- Automatic cleanup when issues are deleted
|
||||
- Ability to join with `issues` table
|
||||
|
||||
### 3. Index Your Query Patterns
|
||||
|
||||
Add indexes for common queries:
|
||||
|
||||
```sql
|
||||
-- If you query by status frequently
|
||||
CREATE INDEX idx_executions_status ON myapp_executions(status);
|
||||
|
||||
-- If you join on issue_id
|
||||
CREATE INDEX idx_executions_issue ON myapp_executions(issue_id);
|
||||
|
||||
-- Composite index for complex queries
|
||||
CREATE INDEX idx_executions_status_priority
|
||||
ON myapp_executions(status, issue_id);
|
||||
```
|
||||
|
||||
### 4. Don't Duplicate bd's Data
|
||||
|
||||
Don't copy fields from `issues` into your tables. Instead, join:
|
||||
|
||||
```sql
|
||||
-- Bad: Duplicating data
|
||||
CREATE TABLE myapp_executions (
|
||||
issue_id TEXT NOT NULL,
|
||||
issue_title TEXT, -- Don't do this!
|
||||
issue_priority INTEGER, -- Don't do this!
|
||||
-- ...
|
||||
);
|
||||
|
||||
-- Good: Join when querying
|
||||
SELECT i.title, i.priority, e.status
|
||||
FROM myapp_executions e
|
||||
JOIN issues i ON e.issue_id = i.id;
|
||||
```
|
||||
|
||||
### 5. Use JSON for Flexible State
|
||||
|
||||
SQLite supports JSON functions, great for checkpoint data:
|
||||
|
||||
```sql
|
||||
CREATE TABLE myapp_checkpoints (
|
||||
id INTEGER PRIMARY KEY,
|
||||
execution_id INTEGER NOT NULL,
|
||||
step_name TEXT NOT NULL,
|
||||
step_data TEXT, -- Store as JSON
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Query JSON fields
|
||||
SELECT
|
||||
id,
|
||||
json_extract(step_data, '$.completed') as completed,
|
||||
json_extract(step_data, '$.error') as error
|
||||
FROM myapp_checkpoints
|
||||
WHERE step_name = 'assessment';
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Execution Tracking
|
||||
|
||||
Track which agent is working on which issue:
|
||||
|
||||
```sql
|
||||
CREATE TABLE myapp_executions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL UNIQUE, -- One execution per issue
|
||||
agent_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
started_at DATETIME NOT NULL,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Claim an issue for execution
|
||||
INSERT INTO myapp_executions (issue_id, agent_id, status, started_at)
|
||||
VALUES (?, ?, 'running', CURRENT_TIMESTAMP)
|
||||
ON CONFLICT (issue_id) DO UPDATE
|
||||
SET agent_id = excluded.agent_id, started_at = CURRENT_TIMESTAMP;
|
||||
```
|
||||
|
||||
### Pattern 2: Checkpoint/Resume
|
||||
|
||||
Store execution checkpoints for crash recovery:
|
||||
|
||||
```sql
|
||||
CREATE TABLE myapp_checkpoints (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
execution_id INTEGER NOT NULL,
|
||||
phase TEXT NOT NULL,
|
||||
checkpoint_data TEXT NOT NULL, -- JSON
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (execution_id) REFERENCES myapp_executions(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Latest checkpoint for an execution
|
||||
SELECT checkpoint_data
|
||||
FROM myapp_checkpoints
|
||||
WHERE execution_id = ?
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
### Pattern 3: Result Storage
|
||||
|
||||
Store execution results linked to issues:
|
||||
|
||||
```sql
|
||||
CREATE TABLE myapp_results (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
result_type TEXT NOT NULL, -- success, partial, failed
|
||||
output_data TEXT, -- JSON: files changed, tests run, etc.
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Get all results for an issue
|
||||
SELECT result_type, output_data, created_at
|
||||
FROM myapp_results
|
||||
WHERE issue_id = ?
|
||||
ORDER BY created_at DESC;
|
||||
```
|
||||
|
||||
## Programmatic Access
|
||||
|
||||
Use bd's `--json` flags for scripting:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Find ready work
|
||||
READY=$(bd ready --limit 1 --json)
|
||||
ISSUE_ID=$(echo $READY | jq -r '.[0].id')
|
||||
|
||||
if [ "$ISSUE_ID" = "null" ]; then
|
||||
echo "No ready work"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Create execution record in your table
|
||||
sqlite3 .beads/myapp.db <<SQL
|
||||
INSERT INTO myapp_executions (issue_id, agent_id, status, started_at)
|
||||
VALUES ('$ISSUE_ID', 'agent-1', 'running', datetime('now'));
|
||||
SQL
|
||||
|
||||
# Claim issue in bd
|
||||
bd update $ISSUE_ID --status in_progress
|
||||
|
||||
# Execute work...
|
||||
echo "Working on $ISSUE_ID"
|
||||
|
||||
# Mark complete
|
||||
bd close $ISSUE_ID --reason "Completed by agent-1"
|
||||
sqlite3 .beads/myapp.db <<SQL
|
||||
UPDATE myapp_executions
|
||||
SET status = 'completed', completed_at = datetime('now')
|
||||
WHERE issue_id = '$ISSUE_ID';
|
||||
SQL
|
||||
```
|
||||
|
||||
## Direct Database Access
|
||||
|
||||
### Using UnderlyingDB() (Recommended)
|
||||
|
||||
The recommended way to extend bd is using the `UnderlyingDB()` method on the storage instance. This gives you access to the same database connection that bd uses, ensuring consistency and avoiding connection overhead:
|
||||
|
||||
```go
|
||||
import (
|
||||
"database/sql"
|
||||
"github.com/steveyegge/beads"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// Open bd's storage
|
||||
ctx := context.Background()
|
||||
store, err := beads.Open(ctx, ".beads/dolt")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
// Get the underlying database connection
|
||||
db := store.UnderlyingDB()
|
||||
|
||||
// Create your extension tables using the same connection
|
||||
schema := `
|
||||
CREATE TABLE IF NOT EXISTS myapp_executions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
issue_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
agent_id TEXT,
|
||||
started_at DATETIME,
|
||||
FOREIGN KEY (issue_id) REFERENCES issues(id) ON DELETE CASCADE
|
||||
);
|
||||
`
|
||||
|
||||
if _, err := db.Exec(schema); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Query bd's tables
|
||||
var title string
|
||||
var priority int
|
||||
err = db.QueryRow(`
|
||||
SELECT title, priority FROM issues WHERE id = ?
|
||||
`, issueID).Scan(&title, &priority)
|
||||
|
||||
// Update your tables
|
||||
_, err = db.Exec(`
|
||||
INSERT INTO myapp_executions (issue_id, status, agent_id, started_at)
|
||||
VALUES (?, ?, ?, CURRENT_TIMESTAMP)
|
||||
`, issueID, "running", "agent-1")
|
||||
|
||||
// Join across layers
|
||||
rows, err := db.Query(`
|
||||
SELECT i.id, i.title, e.status, e.agent_id
|
||||
FROM issues i
|
||||
JOIN myapp_executions e ON i.id = e.issue_id
|
||||
WHERE e.status = 'running'
|
||||
`)
|
||||
```
|
||||
|
||||
**Safety warnings when using UnderlyingDB():**
|
||||
|
||||
⚠️ **NEVER** close the database connection returned by `UnderlyingDB()`. The storage instance owns this connection.
|
||||
|
||||
⚠️ **DO NOT** modify database pool settings (SetMaxOpenConns, SetConnMaxIdleTime) or SQLite PRAGMAs (WAL mode, journal settings) as this affects bd's core operations.
|
||||
|
||||
⚠️ **Keep transactions short** - Long write transactions will block bd's core operations. Use read transactions when possible.
|
||||
|
||||
⚠️ **Expect errors after Close()** - Once you call `store.Close()`, operations on the underlying DB will fail. Use context cancellation to coordinate shutdown.
|
||||
|
||||
✅ **DO** use foreign keys to reference bd's tables for referential integrity.
|
||||
|
||||
✅ **DO** namespace your tables with your app name (e.g., `myapp_executions`).
|
||||
|
||||
✅ **DO** create indexes for your query patterns.
|
||||
|
||||
### When to use UnderlyingDB() vs sql.Open()
|
||||
|
||||
**Use `UnderlyingDB()`:**
|
||||
- ✅ When you want to share the storage connection
|
||||
- ✅ When you need tables in the same database as bd
|
||||
- ✅ When you want automatic lifecycle management
|
||||
- ✅ For most extension use cases (like VC)
|
||||
|
||||
**Use `sql.Open()` separately:**
|
||||
- When you need independent connection pool settings
|
||||
- When you need different timeout/retry behavior
|
||||
- When you're managing multiple databases
|
||||
- When you need fine-grained connection control
|
||||
|
||||
### Alternative: Independent Connection
|
||||
|
||||
If you need independent connection management, you can still open the database directly:
|
||||
|
||||
```go
|
||||
import (
|
||||
"database/sql"
|
||||
_ "modernc.org/sqlite"
|
||||
"github.com/steveyegge/beads"
|
||||
)
|
||||
|
||||
// Auto-discover bd's database path
|
||||
dbPath := beads.FindDatabasePath()
|
||||
if dbPath == "" {
|
||||
log.Fatal("No bd database found. Run 'bd init' first.")
|
||||
}
|
||||
|
||||
// Open your own connection to the same database
|
||||
db, err := sql.Open("sqlite3", dbPath)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// Configure your connection independently
|
||||
db.SetMaxOpenConns(10)
|
||||
db.SetConnMaxIdleTime(time.Minute)
|
||||
|
||||
// Query bd's tables
|
||||
var title string
|
||||
var priority int
|
||||
err = db.QueryRow(`
|
||||
SELECT title, priority FROM issues WHERE id = ?
|
||||
`, issueID).Scan(&title, &priority)
|
||||
|
||||
// Find corresponding JSONL path (for git hooks, monitoring, etc.)
|
||||
jsonlPath := beads.FindJSONLPath(dbPath)
|
||||
fmt.Printf("BD exports to: %s\n", jsonlPath)
|
||||
```
|
||||
|
||||
## Batch Operations for Performance
|
||||
|
||||
When creating many issues at once (e.g., bulk imports, batch processing), use `CreateIssues` for significantly better performance:
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"github.com/steveyegge/beads/internal/storage/sqlite"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// Open bd's storage
|
||||
store, err := sqlite.New(".beads/issues.db")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Prepare batch of issues to create
|
||||
issues := make([]*types.Issue, 0, 1000)
|
||||
for _, item := range externalData {
|
||||
issue := &types.Issue{
|
||||
Title: item.Title,
|
||||
Description: item.Description,
|
||||
Status: types.StatusOpen,
|
||||
Priority: item.Priority,
|
||||
IssueType: types.TypeTask,
|
||||
}
|
||||
issues = append(issues, issue)
|
||||
}
|
||||
|
||||
// Create all issues in a single atomic transaction (5-15x faster!)
|
||||
if err := store.CreateIssues(ctx, issues, "import"); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// REMOVED (bd-c7af): SyncAllCounters - no longer needed with hash IDs
|
||||
```
|
||||
|
||||
### Performance Comparison
|
||||
|
||||
| Operation | CreateIssue Loop | CreateIssues Batch | Speedup |
|
||||
|-----------|------------------|---------------------|---------|
|
||||
| 100 issues | ~900ms | ~30ms | 30x |
|
||||
| 500 issues | ~4.5s | ~800ms | 5.6x |
|
||||
| 1000 issues | ~9s | ~950ms | 9.5x |
|
||||
|
||||
### When to Use Each Method
|
||||
|
||||
**Use `CreateIssue` (single issue):**
|
||||
- Interactive CLI commands (`bd create`)
|
||||
- Single issue creation in your app
|
||||
- User-facing operations
|
||||
|
||||
**Use `CreateIssues` (batch):**
|
||||
- Bulk imports from external systems
|
||||
- Batch processing workflows
|
||||
- Creating multiple related issues at once
|
||||
- Agent workflows that generate many issues
|
||||
|
||||
### Batch Import Pattern
|
||||
|
||||
```go
|
||||
// Example: Import from external issue tracker
|
||||
func ImportFromExternal(externalIssues []ExternalIssue) error {
|
||||
store, err := sqlite.New(".beads/issues.db")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
// Convert external format to bd format
|
||||
issues := make([]*types.Issue, 0, len(externalIssues))
|
||||
for _, ext := range externalIssues {
|
||||
issue := &types.Issue{
|
||||
ID: fmt.Sprintf("import-%d", ext.ID), // Explicit IDs
|
||||
Title: ext.Title,
|
||||
Description: ext.Description,
|
||||
Status: convertStatus(ext.Status),
|
||||
Priority: ext.Priority,
|
||||
IssueType: convertType(ext.Type),
|
||||
}
|
||||
|
||||
// Normalize closed_at for closed issues
|
||||
if issue.Status == types.StatusClosed {
|
||||
closedAt := ext.ClosedAt
|
||||
issue.ClosedAt = &closedAt
|
||||
}
|
||||
|
||||
issues = append(issues, issue)
|
||||
}
|
||||
|
||||
// Atomic batch create
|
||||
if err := store.CreateIssues(ctx, issues, "external-import"); err != nil {
|
||||
return fmt.Errorf("batch create failed: %w", err)
|
||||
}
|
||||
|
||||
// REMOVED (bd-c7af): SyncAllCounters - no longer needed with hash IDs
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The key insight: **bd is a focused issue tracker, not a framework**.
|
||||
|
||||
By extending the database:
|
||||
- You get powerful issue tracking for free
|
||||
- Your app adds orchestration logic
|
||||
- Simple SQL joins give you full visibility
|
||||
- No tight coupling or version conflicts
|
||||
|
||||
This pattern scales from simple scripts to complex orchestrators like VC.
|
||||
|
||||
## See Also
|
||||
|
||||
- [README.md](../README.md) - Complete bd documentation
|
||||
- [QUICKSTART.md](QUICKSTART.md) - Quick start tutorial
|
||||
- Check out VC's implementation at `github.com/steveyegge/vc` for a real-world example
|
||||
|
|
@ -398,7 +398,7 @@ That's it! No PostgreSQL, no Redis, no Docker, no node_modules.
|
|||
|
||||
### Can I extend bd's database?
|
||||
|
||||
See [EXTENDING.md](EXTENDING.md) for the legacy SQLite extension approach (deprecated). With the Dolt backend, use `bd query` for direct SQL access or build integrations using `bd --json` CLI output.
|
||||
With the Dolt backend, use `bd query` for direct SQL access or build integrations using `bd --json` CLI output.
|
||||
|
||||
### Does bd support Windows?
|
||||
|
||||
|
|
|
|||
|
|
@ -1,318 +0,0 @@
|
|||
# Multi-Repo Hydration Layer
|
||||
|
||||
> **⚠️ DEPRECATED**: This document describes the SQLite-era multi-repo hydration layer.
|
||||
> The `internal/storage/sqlite/` backend has been removed. The Dolt backend does not
|
||||
> currently implement multi-repo hydration. This document is preserved for historical reference.
|
||||
|
||||
This document describes the implementation of Task 3 from the multi-repo support feature (bd-307): the hydration layer that loaded issues from multiple JSONL files into a unified SQLite database.
|
||||
|
||||
## Overview
|
||||
|
||||
The hydration layer enables beads to aggregate issues from multiple repositories into a single database for unified querying and analysis. It uses file modification time (mtime) caching to optimize performance by only reimporting files that have changed.
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Database Schema
|
||||
|
||||
**Table: `repo_mtimes`**
|
||||
```sql
|
||||
CREATE TABLE repo_mtimes (
|
||||
repo_path TEXT PRIMARY KEY, -- Absolute path to repository root
|
||||
jsonl_path TEXT NOT NULL, -- Absolute path to .beads/issues.jsonl
|
||||
mtime_ns INTEGER NOT NULL, -- Modification time in nanoseconds
|
||||
last_checked DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
This table tracks the last known modification time of each repository's JSONL file to enable intelligent skip logic during hydration.
|
||||
|
||||
### 2. Configuration
|
||||
|
||||
Multi-repo mode is configured via `internal/config/config.go`:
|
||||
|
||||
```yaml
|
||||
# .beads/config.yaml
|
||||
repos:
|
||||
primary: /path/to/primary/repo # Canonical source (optional)
|
||||
additional: # Additional repos to hydrate from
|
||||
- ~/projects/repo1
|
||||
- ~/projects/repo2
|
||||
```
|
||||
|
||||
- **Primary repo** (`.`): Issues from this repo are marked with `source_repo = "."`
|
||||
- **Additional repos**: Issues marked with their relative path as `source_repo`
|
||||
|
||||
### 3. Implementation Files
|
||||
|
||||
**New Files:**
|
||||
- `internal/storage/sqlite/multirepo.go` - Core hydration logic
|
||||
- `internal/storage/sqlite/multirepo_test.go` - Test coverage
|
||||
- `docs/MULTI_REPO_HYDRATION.md` - This document
|
||||
|
||||
**Modified Files:**
|
||||
- `internal/storage/sqlite/schema.go` - Added `repo_mtimes` table
|
||||
- `internal/storage/sqlite/migrations/013_repo_mtimes_table.go` - Migration for `repo_mtimes` table
|
||||
- `internal/storage/sqlite/sqlite.go` - Integrated hydration into storage initialization
|
||||
- `internal/storage/sqlite/ready.go` - Added `source_repo` to all SELECT queries
|
||||
- `internal/storage/sqlite/labels.go` - Added `source_repo` to SELECT query
|
||||
- `internal/storage/sqlite/migrations_test.go` - Added migration tests
|
||||
|
||||
## Key Functions
|
||||
|
||||
### `HydrateFromMultiRepo(ctx context.Context) (map[string]int, error)`
|
||||
|
||||
Main entry point for multi-repo hydration. Called automatically during `sqlite.New()`.
|
||||
|
||||
**Behavior:**
|
||||
- Returns `nil, nil` if not in multi-repo mode (single-repo operation)
|
||||
- Processes primary repo first (if configured)
|
||||
- Then processes each additional repo
|
||||
- Returns a map of `source_repo -> issue count` for imported issues
|
||||
|
||||
### `hydrateFromRepo(ctx, repoPath, sourceRepo string) (int, error)`
|
||||
|
||||
Handles hydration for a single repository.
|
||||
|
||||
**Steps:**
|
||||
1. Resolves absolute path to repo and JSONL file
|
||||
2. Checks file existence (skips if missing)
|
||||
3. Compares current mtime with cached mtime
|
||||
4. Skips import if mtime unchanged (optimization)
|
||||
5. Imports issues if file changed or no cache exists
|
||||
6. Updates mtime cache after successful import
|
||||
|
||||
### `importJSONLFile(ctx, jsonlPath, sourceRepo string) (int, error)`
|
||||
|
||||
Parses a JSONL file and imports all issues into the database.
|
||||
|
||||
**Features:**
|
||||
- Handles large files (10MB max line size)
|
||||
- Skips empty lines and comments (`#`)
|
||||
- Sets `source_repo` field on all imported issues
|
||||
- Computes `content_hash` if missing
|
||||
- Uses transactions for atomicity
|
||||
- Imports dependencies, labels, and comments
|
||||
|
||||
### `upsertIssueInTx(ctx, tx, issue *types.Issue) error`
|
||||
|
||||
Inserts or updates an issue within a transaction.
|
||||
|
||||
**Smart Update Logic:**
|
||||
- Checks if issue exists by ID
|
||||
- If new: inserts issue
|
||||
- If exists: compares `content_hash` and only updates if changed
|
||||
- Imports associated dependencies, labels, and comments
|
||||
- Uses `INSERT OR IGNORE` for dependencies/labels to avoid duplicates
|
||||
|
||||
### `expandTilde(path string) (string, error)`
|
||||
|
||||
Utility function to expand `~` and `~/` paths to absolute home directory paths.
|
||||
|
||||
## Mtime Caching
|
||||
|
||||
The hydration layer uses file modification time (mtime) as a cache key to avoid unnecessary reimports.
|
||||
|
||||
**Cache Logic:**
|
||||
1. First hydration: No cache exists → import file
|
||||
2. Subsequent hydrations: Compare mtimes
|
||||
- If `mtime_current == mtime_cached` → skip import (fast path)
|
||||
- If `mtime_current != mtime_cached` → reimport (file changed)
|
||||
3. After successful import: Update cache with new mtime
|
||||
|
||||
**Benefits:**
|
||||
- **Performance**: Avoids parsing/importing unchanged JSONL files
|
||||
- **Correctness**: Detects external changes via filesystem metadata
|
||||
- **Simplicity**: No need for content hashing or git integration
|
||||
|
||||
**Limitations:**
|
||||
- Relies on filesystem mtime accuracy
|
||||
- Won't detect changes if mtime is manually reset
|
||||
- Cross-platform mtime precision varies (nanosecond on Unix, ~100ns on Windows)
|
||||
|
||||
## Source Repo Tracking
|
||||
|
||||
Each issue has a `source_repo` field that identifies which repository it came from:
|
||||
|
||||
- **Primary repo**: `source_repo = "."`
|
||||
- **Additional repos**: `source_repo = <relative_path>` (e.g., `~/projects/repo1`)
|
||||
|
||||
This enables:
|
||||
- Filtering issues by source repository
|
||||
- Understanding issue provenance in multi-repo setups
|
||||
- Future features like repo-specific permissions or workflows
|
||||
|
||||
**Database Schema:**
|
||||
```sql
|
||||
ALTER TABLE issues ADD COLUMN source_repo TEXT DEFAULT '.';
|
||||
CREATE INDEX idx_issues_source_repo ON issues(source_repo);
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Comprehensive test coverage in `internal/storage/sqlite/multirepo_test.go`:
|
||||
|
||||
### Test Cases
|
||||
|
||||
1. **`TestExpandTilde`**
|
||||
- Verifies tilde expansion for various path formats
|
||||
|
||||
2. **`TestHydrateFromMultiRepo/single-repo_mode_returns_nil`**
|
||||
- Confirms nil return when not in multi-repo mode
|
||||
|
||||
3. **`TestHydrateFromMultiRepo/hydrates_from_primary_repo`**
|
||||
- Validates primary repo import
|
||||
- Checks `source_repo = "."` is set correctly
|
||||
|
||||
4. **`TestHydrateFromMultiRepo/uses_mtime_caching_to_skip_unchanged_files`**
|
||||
- First hydration: imports 1 issue
|
||||
- Second hydration: imports 0 issues (cached)
|
||||
- Proves mtime cache optimization works
|
||||
|
||||
5. **`TestHydrateFromMultiRepo/imports_additional_repos`**
|
||||
- Creates primary + additional repo
|
||||
- Verifies both are imported
|
||||
- Checks source_repo fields are distinct
|
||||
|
||||
6. **`TestImportJSONLFile/imports_issues_with_dependencies_and_labels`**
|
||||
- Tests JSONL parsing with complex data
|
||||
- Validates dependencies and labels are imported
|
||||
- Confirms relational data integrity
|
||||
|
||||
7. **`TestMigrateRepoMtimesTable`**
|
||||
- Verifies migration creates table correctly
|
||||
- Confirms migration is idempotent
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all multirepo tests
|
||||
go test -v ./internal/storage/sqlite -run TestHydrateFromMultiRepo
|
||||
|
||||
# Run specific test
|
||||
go test -v ./internal/storage/sqlite -run TestExpandTilde
|
||||
|
||||
# Run all sqlite tests
|
||||
go test ./internal/storage/sqlite
|
||||
```
|
||||
|
||||
## Integration
|
||||
|
||||
### Automatic Hydration
|
||||
|
||||
Hydration happens automatically during storage initialization:
|
||||
|
||||
```go
|
||||
// internal/storage/sqlite/sqlite.go
|
||||
func New(path string) (*SQLiteStorage, error) {
|
||||
// ... schema initialization ...
|
||||
|
||||
storage := &SQLiteStorage{db: db, dbPath: absPath}
|
||||
|
||||
// Skip for in-memory databases (used in tests)
|
||||
if path != ":memory:" {
|
||||
_, err := storage.HydrateFromMultiRepo(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to hydrate from multi-repo: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return storage, nil
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Example
|
||||
|
||||
**`.beads/config.yaml`:**
|
||||
```yaml
|
||||
repos:
|
||||
primary: /Users/alice/work/main-project
|
||||
additional:
|
||||
- ~/work/library-a
|
||||
- ~/work/library-b
|
||||
- /opt/shared/common-issues
|
||||
```
|
||||
|
||||
**Resulting database:**
|
||||
- Issues from `/Users/alice/work/main-project` → `source_repo = "."`
|
||||
- Issues from `~/work/library-a` → `source_repo = "~/work/library-a"`
|
||||
- Issues from `~/work/library-b` → `source_repo = "~/work/library-b"`
|
||||
- Issues from `/opt/shared/common-issues` → `source_repo = "/opt/shared/common-issues"`
|
||||
|
||||
## Migration
|
||||
|
||||
The `repo_mtimes` table is created via standard migration system:
|
||||
|
||||
```go
|
||||
// internal/storage/sqlite/migrations/013_repo_mtimes_table.go
|
||||
func MigrateRepoMtimesTable(db *sql.DB) error {
|
||||
// Check if table exists
|
||||
var tableName string
|
||||
err := db.QueryRow(`
|
||||
SELECT name FROM sqlite_master
|
||||
WHERE type='table' AND name='repo_mtimes'
|
||||
`).Scan(&tableName)
|
||||
|
||||
if err == sql.ErrNoRows {
|
||||
// Create table + index
|
||||
_, err := db.Exec(`
|
||||
CREATE TABLE repo_mtimes (...);
|
||||
CREATE INDEX idx_repo_mtimes_checked ON repo_mtimes(last_checked);
|
||||
`)
|
||||
return err
|
||||
}
|
||||
|
||||
return nil // Already exists
|
||||
}
|
||||
```
|
||||
|
||||
**Migration is idempotent**: Safe to run multiple times, won't error on existing table.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Incremental Sync**: Instead of full reimport, use git hashes or checksums to sync only changed issues
|
||||
2. **Conflict Resolution**: Handle cases where same issue ID exists in multiple repos with different content
|
||||
3. **Selective Hydration**: Allow users to specify which repos to hydrate (CLI flag or config)
|
||||
4. **Background Refresh**: Periodically check for JSONL changes without blocking CLI operations
|
||||
5. **Repository Metadata**: Track repo URL, branch, last commit hash for better provenance
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
**Mtime Cache Hit (fast path):**
|
||||
- 1 SQL query per repo (check cached mtime)
|
||||
- No file I/O if mtime matches
|
||||
- **Typical latency**: <1ms per repo
|
||||
|
||||
**Mtime Cache Miss (import path):**
|
||||
- 1 SQL query (check cache)
|
||||
- 1 file read (parse JSONL)
|
||||
- N SQL inserts/updates (where N = issue count)
|
||||
- 1 SQL update (cache mtime)
|
||||
- **Typical latency**: 10-100ms for 100 issues
|
||||
|
||||
**Optimization Tips:**
|
||||
- Place frequently-changing repos in primary position
|
||||
- Use `.beads/config.yaml` instead of env vars (faster viper access)
|
||||
- Limit `additional` repos to ~10 for reasonable startup time
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Hydration not working?**
|
||||
1. Check config: `bd config list` should show `repos.primary` or `repos.additional`
|
||||
2. Verify JSONL exists: `ls -la /path/to/repo/.beads/issues.jsonl`
|
||||
3. Check logs: Set `BD_DEBUG=1` to see hydration debug output
|
||||
|
||||
**Issues not updating?**
|
||||
- Mtime cache might be stale
|
||||
- Force refresh by deleting cache: `DELETE FROM repo_mtimes WHERE repo_path = '/path/to/repo'`
|
||||
- Or touch the JSONL file: `touch /path/to/repo/.beads/issues.jsonl`
|
||||
|
||||
**Performance issues?**
|
||||
- Check repo count: `SELECT COUNT(*) FROM repo_mtimes`
|
||||
- Measure hydration time with `BD_DEBUG=1`
|
||||
- Consider reducing `additional` repos if startup is slow
|
||||
|
||||
## See Also
|
||||
|
||||
- [CONFIG.md](CONFIG.md) - Configuration system documentation
|
||||
- [EXTENDING.md](EXTENDING.md) - Database schema extension guide
|
||||
- [bd-307](https://github.com/steveyegge/beads/issues/307) - Original multi-repo feature request
|
||||
Loading…
Add table
Add a link
Reference in a new issue