362 lines
18 KiB
Markdown
362 lines
18 KiB
Markdown
# Architecture
|
|
|
|
This document describes bd's overall architecture - the data model, sync mechanism, and how components fit together. For internal implementation details (FlushManager, Blocked Cache), see [INTERNALS.md](INTERNALS.md).
|
|
|
|
## The Two-Layer Data Model
|
|
|
|
bd's core design enables a distributed, Dolt-powered issue tracker that feels like a centralized database. The architecture has two synchronized layers:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ CLI Layer │
|
|
│ │
|
|
│ bd create, list, update, close, ready, show, dep, sync, ... │
|
|
│ - Cobra commands in cmd/bd/ │
|
|
│ - All commands support --json for programmatic use │
|
|
│ - Direct DB access (server mode via dolt sql-server) │
|
|
└──────────────────────────────┬──────────────────────────────────┘
|
|
│
|
|
v
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Dolt Database │
|
|
│ (.beads/dolt/) │
|
|
│ │
|
|
│ - Version-controlled SQL database with cell-level merge │
|
|
│ - Server mode via dolt sql-server (multi-writer capable) │
|
|
│ - Fast queries, indexes, foreign keys │
|
|
│ - Issues, dependencies, labels, comments, events │
|
|
│ - Automatic Dolt commits on every write │
|
|
│ - Native push/pull to Dolt remotes │
|
|
└──────────────────────────────┬──────────────────────────────────┘
|
|
│
|
|
Dolt push/pull
|
|
(or federation peer sync)
|
|
│
|
|
v
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Remote (Dolt or Git) │
|
|
│ │
|
|
│ - Dolt remotes (DoltHub, S3, GCS, filesystem) │
|
|
│ - All collaborators share the same issue database │
|
|
│ - Cell-level merge for conflict resolution │
|
|
│ - Protected branch support via separate sync branch │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Why This Design?
|
|
|
|
**Dolt for versioned SQL:** Queries complete in milliseconds with full SQL support. Dolt adds native version control — every write is automatically committed to Dolt history, providing a complete audit trail. Cell-level merge resolves conflicts automatically.
|
|
|
|
**Dolt for distribution:** Native push/pull to Dolt remotes (DoltHub, S3, GCS). No special sync server needed. Issues travel with your code. Offline work just works.
|
|
|
|
**Export and backup:** `bd export` outputs issue JSONL for data migration and interoperability. Use `bd backup init` / `bd backup sync` to push Dolt-native backups (preserving full commit history) to a filesystem path or DoltHub, and `bd backup restore` to restore from them.
|
|
|
|
## Write Path
|
|
|
|
When you create or modify an issue:
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ CLI Command │───▶│ Dolt Write │───▶│ Dolt Commit │
|
|
│ (bd create) │ │ (immediate) │ │ (automatic) │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
1. **Command executes:** `bd create "New feature"` writes to Dolt immediately
|
|
2. **Dolt commit:** Every write is automatically committed to Dolt history
|
|
3. **Sync:** Use `bd dolt push` to share changes with Dolt remotes
|
|
|
|
Key implementation:
|
|
- Dolt storage: `internal/storage/dolt/`
|
|
- Export (for portability): `cmd/bd/export.go`
|
|
|
|
## Read Path
|
|
|
|
All queries run directly against the local Dolt database:
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐
|
|
│ CLI Query │───▶│ Dolt Query │
|
|
│ (bd ready) │ │ (SQL) │
|
|
└─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
1. **Query:** Commands read from fast local Dolt database via SQL
|
|
2. **Sync:** Use `bd dolt pull` to fetch updates from Dolt remotes
|
|
|
|
Key implementation:
|
|
- Backup restore: `cmd/bd/backup_restore.go`
|
|
- Issue bootstrap/migration: `cmd/bd/init.go`
|
|
- Dolt storage: `internal/storage/dolt/`
|
|
|
|
## Hash-Based Collision Prevention
|
|
|
|
The key insight that enables distributed operation: **content-based hashing for deduplication**.
|
|
|
|
### The Problem
|
|
|
|
Sequential IDs (bd-1, bd-2, bd-3) cause collisions when multiple agents create issues concurrently:
|
|
|
|
```
|
|
Branch A: bd create "Add OAuth" → bd-10
|
|
Branch B: bd create "Add Stripe" → bd-10 (collision!)
|
|
```
|
|
|
|
### The Solution
|
|
|
|
Hash-based IDs derived from random UUIDs ensure uniqueness:
|
|
|
|
```
|
|
Branch A: bd create "Add OAuth" → bd-a1b2
|
|
Branch B: bd create "Add Stripe" → bd-f14c (no collision)
|
|
```
|
|
|
|
### How It Works
|
|
|
|
1. **Issue creation:** Generate random UUID, derive short hash as ID
|
|
2. **Progressive scaling:** IDs start at 4 chars, grow to 5-6 chars as database grows
|
|
3. **Content hashing:** Each issue has a content hash for change detection
|
|
4. **Merge logic:** Same ID + different content = update, same ID + same content = skip
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Merge Logic │
|
|
│ (used by Dolt pull and init --from-jsonl) │
|
|
│ │
|
|
│ For each issue in incoming data: │
|
|
│ 1. Compute content hash │
|
|
│ 2. Look up existing issue by ID │
|
|
│ 3. Compare hashes: │
|
|
│ - Same hash → skip (already present) │
|
|
│ - Different hash → update (newer version) │
|
|
│ - No match → create (new issue) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
This eliminates the need for central coordination while ensuring all machines converge to the same state.
|
|
|
|
See [COLLISION_MATH.md](COLLISION_MATH.md) for birthday paradox calculations on hash length vs collision probability.
|
|
|
|
## Server Architecture
|
|
|
|
Each workspace can run its own Dolt server for multi-writer access:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Dolt Server Mode │
|
|
│ │
|
|
│ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ RPC Server │ │ dolt │ │
|
|
│ │ │ │ sql-server │ │
|
|
│ └─────────────┘ └─────────────┘ │
|
|
│ │ │ │
|
|
│ └──────────────────┘ │
|
|
│ │ │
|
|
│ v │
|
|
│ ┌─────────────┐ │
|
|
│ │ Dolt │ │
|
|
│ │ Database │ │
|
|
│ └─────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
|
|
CLI commands ───SQL───▶ dolt sql-server ───▶ Database
|
|
or
|
|
CLI commands ───SQL───▶ Database (embedded mode)
|
|
```
|
|
|
|
**Server mode:**
|
|
- Connects to `dolt sql-server` (multi-writer, high-concurrency)
|
|
- PID file at `.beads/dolt-server.pid`
|
|
- Logs at `.beads/dolt-server.log`
|
|
- **Shared server mode** (opt-in): all projects share a single Dolt server at
|
|
`~/.beads/shared-server/` instead of per-project servers. Enable via
|
|
`dolt.shared-server: true` in config.yaml or `BEADS_DOLT_SHARED_SERVER=1`.
|
|
|
|
**Embedded mode:**
|
|
- Direct database access (single-writer, no server process)
|
|
|
|
**Communication:**
|
|
- Protocol defined in `internal/rpc/protocol.go`
|
|
- Used by Dolt server mode for multi-writer access
|
|
|
|
## Data Types
|
|
|
|
Core types in `internal/types/types.go`:
|
|
|
|
| Type | Description | Key Fields |
|
|
|------|-------------|------------|
|
|
| **Issue** | Work item | ID, Title, Description, Status, Priority, Type |
|
|
| **Dependency** | Relationship | FromID, ToID, Type (blocks/related/parent-child/discovered-from) |
|
|
| **Label** | Tag | Name, Color, Description |
|
|
| **Comment** | Discussion | IssueID, Author, Content, Timestamp |
|
|
| **Event** | Audit trail | IssueID, Type, Data, Timestamp |
|
|
|
|
### Dependency Types
|
|
|
|
| Type | Semantic | Affects `bd ready`? |
|
|
|------|----------|---------------------|
|
|
| `blocks` | Issue X must close before Y starts | Yes |
|
|
| `parent-child` | Hierarchical (epic/subtask) | Yes (children blocked if parent blocked) |
|
|
| `related` | Soft link for reference | No |
|
|
| `discovered-from` | Found during work on parent | No |
|
|
|
|
### Status Flow
|
|
|
|
```
|
|
open ──▶ in_progress ──▶ closed
|
|
│ │
|
|
└────────────────────────┘
|
|
(reopen)
|
|
```
|
|
|
|
### Issue Schema
|
|
|
|
Each issue in the Dolt database (and in JSONL exports via `bd export`) has the following fields. Fields marked with `(optional)` use `omitempty` and are excluded when empty/zero.
|
|
|
|
**Core Identification:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `id` | string | Unique identifier (e.g., `bd-a1b2`) |
|
|
|
|
**Issue Content:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `title` | string | Issue title (required) |
|
|
| `description` | string | Detailed description (optional) |
|
|
| `design` | string | Design notes (optional) |
|
|
| `acceptance_criteria` | string | Acceptance criteria (optional) |
|
|
| `notes` | string | Additional notes (optional) |
|
|
|
|
**Status & Workflow:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `status` | string | Current status: `open`, `in_progress`, `blocked`, `deferred`, `closed`, `tombstone`, `pinned`, `hooked` (optional, defaults to `open`) |
|
|
| `priority` | int | Priority 0-4 where 0=critical, 4=backlog |
|
|
| `issue_type` | string | Type: `bug`, `feature`, `task`, `epic`, `chore`, `message`, `merge-request`, `molecule`, `gate`, `agent`, `role`, `convoy` (optional, defaults to `task`) |
|
|
|
|
**Assignment:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `assignee` | string | Assigned user/agent (optional) |
|
|
| `estimated_minutes` | int | Time estimate in minutes (optional) |
|
|
|
|
**Timestamps:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `created_at` | RFC3339 | When issue was created |
|
|
| `created_by` | string | Who created the issue (optional) |
|
|
| `updated_at` | RFC3339 | Last modification time |
|
|
| `closed_at` | RFC3339 | When issue was closed (optional, set when status=closed) |
|
|
| `close_reason` | string | Reason provided when closing (optional) |
|
|
|
|
**External Integration:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `external_ref` | string | External reference (e.g., `gh-9`, `jira-ABC`) (optional) |
|
|
|
|
**Relational Data:**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `labels` | []string | Tags attached to the issue (optional) |
|
|
| `dependencies` | []Dependency | Relationships to other issues (optional) |
|
|
| `comments` | []Comment | Discussion comments (optional) |
|
|
|
|
**Tombstone Fields (soft-delete):**
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `deleted_at` | RFC3339 | When deleted (optional, set when status=tombstone) |
|
|
| `deleted_by` | string | Who deleted (optional) |
|
|
| `delete_reason` | string | Why deleted (optional) |
|
|
| `original_type` | string | Issue type before deletion (optional) |
|
|
|
|
**Note:** Fields with `json:"-"` tags (like `content_hash`, `source_repo`, `id_prefix`) are internal and not included in exports.
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
.beads/
|
|
├── dolt/ # Dolt database, sql-server.pid, sql-server.log (gitignored)
|
|
├── metadata.json # Backend config (local, gitignored)
|
|
└── config.yaml # Project config (optional)
|
|
```
|
|
|
|
## Key Code Paths
|
|
|
|
| Area | Files |
|
|
|------|-------|
|
|
| CLI entry | `cmd/bd/main.go` |
|
|
| Storage interface | `internal/storage/storage.go` |
|
|
| Dolt implementation | `internal/storage/dolt/` |
|
|
| RPC protocol | `internal/rpc/protocol.go`, `server_*.go` |
|
|
| Export logic (portability) | `cmd/bd/export.go` |
|
|
| Backup restore | `cmd/bd/backup_restore.go` |
|
|
| Issue bootstrap/migration | `cmd/bd/init.go` |
|
|
|
|
## Wisps and Molecules
|
|
|
|
**Molecules** are template work items that define structured workflows. When spawned, they create **wisps** - ephemeral child issues that track execution steps.
|
|
|
|
> **For full documentation** on the molecular chemistry metaphor (protos, pour, bond, squash, burn), see [MOLECULES.md](MOLECULES.md).
|
|
|
|
### Wisp Lifecycle
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ bd mol wisp │───▶│ Wisp Issues │───▶│ bd mol squash │
|
|
│ (from template) │ │ (local-only) │ │ (→ digest) │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
1. **Create:** Create wisps from a molecule template
|
|
2. **Execute:** Agent works through wisp steps (local database only)
|
|
3. **Squash:** Compress wisps into a permanent digest issue
|
|
|
|
### Why Wisps Don't Sync
|
|
|
|
Wisps are intentionally **local-only**:
|
|
|
|
- They exist only in the spawning agent's local database
|
|
- They are **never exported or synced**
|
|
- They cannot resurrect from other clones (they were never there)
|
|
- They are **hard-deleted** when squashed (no tombstones needed)
|
|
|
|
This design enables:
|
|
|
|
- **Fast local iteration:** No sync overhead during execution
|
|
- **Clean history:** Only the digest (outcome) enters git
|
|
- **Agent isolation:** Each agent's execution trace is private
|
|
- **Bounded storage:** Wisps don't accumulate across clones
|
|
|
|
### Wisp vs Regular Issue Deletion
|
|
|
|
| Aspect | Regular Issues | Wisps |
|
|
|--------|---------------|-------|
|
|
| Synced to remotes | Yes | No |
|
|
| Tombstone on delete | Yes | No |
|
|
| Can resurrect | Yes (without tombstone) | No (never synced) |
|
|
| Deletion method | `CreateTombstone()` | `DeleteIssue()` (hard delete) |
|
|
|
|
The `bd mol squash` command uses hard delete intentionally - tombstones would be wasted overhead for data that never leaves the local database.
|
|
|
|
### Future Directions
|
|
|
|
- **Separate wisp repo:** Keep wisps in a dedicated ephemeral git repo
|
|
- **Digest migration:** Explicit step to promote digests to main repo
|
|
- **Wisp retention:** Option to preserve wisps in local git history
|
|
|
|
## Related Documentation
|
|
|
|
- [MOLECULES.md](MOLECULES.md) - Molecular chemistry metaphor (protos, pour, bond, squash, burn)
|
|
- [INTERNALS.md](INTERNALS.md) - FlushManager, Blocked Cache implementation details
|
|
- [ADVANCED.md](ADVANCED.md) - Advanced features and configuration
|
|
- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Recovery procedures and common issues
|
|
- [FAQ.md](FAQ.md) - Common questions about the architecture
|
|
- [COLLISION_MATH.md](COLLISION_MATH.md) - Hash collision probability analysis
|