From d0743f89d71f802ed4586c540b965df0ec74cb70 Mon Sep 17 00:00:00 2001 From: Harry Miller Date: Wed, 30 Sep 2026 18:08:44 -0700 Subject: [PATCH] fix(formula): substitute vars in step assignee, labels, and metadata (#5758) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(formula): substitute vars in step assignee, labels, and metadata Cook and pour substituted title/description/notes/gate but copied assignee, labels, and metadata through raw, so a poured bead carried literal `{{var}}` text in all three. - labels: every label-based query, filter, and automation silently stops matching, and nothing errors (GH#5110). - metadata: general keys came through literal; only a top-level "repo" key on a gh:* gate was substituted (GH#5110). - assignee: worse than cosmetic - close refuses when the actor doesn't match the assignee, so every step of a poured molecule is unclosable and the molecule can never advance (GH#5754). internal/formula/expand.go already substituted assignee and labels on the inline-expansion path, so the two paths disagreed on the same fields; Step.Assignee is documented as "supports substitution" and ExtractVariables already scanned it, meaning pour hard-failed demanding a variable it then refused to resolve. Both call sites are fixed together: runtime-mode cook feeds the proto that pour clones, so a field left literal in either path ships a placeholder onto the bead. Fixing cloneSubgraphInto covers pour, wisp, bond, and the proxied-server paths, which all funnel through it. substituteMetadataRepo is generalized to substituteMetadataVars, which resolves every string leaf of the metadata JSON at any depth. This drops the earlier gh:*-gate-only restriction (SF2/SF4): reading a `repo` key AS a GitHub selector is only correct on a gh:* gate, but resolving a {{var}} in it interprets the key not at all. The walk still decodes into json.RawMessage, so untouched values stay byte-identical, and object keys are never rewritten (a rewritten key could collide with a sibling). The variable scan is widened to match: extractAllVariables (pour) and ExtractVariables (cook) now cover assignee, labels, metadata strings, notes, and await ID, so pour demands exactly the variables it resolves. Undeclared handlebars are still filtered out by VarDefs. Regression coverage asserts on a poured bead, and checks the assignee via validation.AssigneeMatches rather than a string compare - a substitution-only assertion would pass while the unclosable symptom persisted. Co-Authored-By: Claude Opus 5 (1M context) * chore(formula): regenerate schema_gen.go after the types.go doc edits TestSchemaGenIsCurrent holds schema_gen.go to a byte-for-byte rerun of the generator over types.go, and the field docs are part of the generated output — so the Labels/Metadata comment updates made it stale. Co-Authored-By: Claude Opus 5 (1M context) * fix(mol bond): resolve vars like pour instead of demanding every handlebar buildAttachCloneOpts demanded every variable extractAllVariables found in the proto subgraph and applied no declared defaults, unlike pour (cmd/bd/pour.go) and wisp (cmd/bd/wisp.go). Widening the scanned field set to assignee, labels and metadata therefore widened bond's hard refusal along with it: a proto whose label or metadata carried an undeclared documentation handlebar, or a declared var carrying only a default, made `bd mol bond` fail with "missing required variables" where it previously succeeded. Run applyVariableDefaults + extractRequiredVariables there, matching the sibling call sites. extractRequiredVariables returns the unfiltered list when VarDefs is nil, so a legacy template still requires every var it mentions; the change is strictly a reduction in false failures. Applying the defaults also puts them in opts.Vars, which is the map cloneSubgraphInto actually substitutes from. Also from review: - docs/workflows/gates.md no longer enumerates a stale three-field list for where `{{var}}` substitution applies. - maxMetadataSubstitutionDepth's doc names walkJSONStrings (not the long-gone substituteJSONVars) and says plainly that hitting the bound deliberately ships a literal placeholder rather than erroring. Reported-by: bee-ghosttrack Agent-Signature: claude-code-opus-5-unknown-reasoning on behalf of Harry Miller Co-Authored-By: Claude Opus 5 (1M context) * build(bazel): add the validation dep to the bd_test go_test (make bazel-sync) mol_embedded_test.go and template_test.go now import internal/validation directly (validation.AssigneeMatches, to check the substituted step assignee), so gazelle emits //internal/validation on bd_test. bd_lib already carries the dep, so bd_test compiles through embed, but gazelle lists every direct test import on the go_test itself. cmd/bd/BUILD.bazel landed on main after this PR branched, so the PR could not carry the line; without it, bazel.yml's "BUILD files in sync" step fails in the required Bazel / test lane. Generated by `make bazel-sync`. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Eddie the Engineer --- cmd/bd/BUILD.bazel | 1 + cmd/bd/cook.go | 42 +++ cmd/bd/cook_test.go | 63 +++++ cmd/bd/mol_bond.go | 12 +- cmd/bd/mol_embedded_test.go | 102 +++++++ cmd/bd/template.go | 228 +++++++++++---- cmd/bd/template_test.go | 474 +++++++++++++++++++++++++++----- docs/workflows/gates.md | 4 +- internal/formula/parser.go | 44 ++- internal/formula/parser_test.go | 20 ++ internal/formula/schema_gen.go | 5 +- internal/formula/types.go | 5 +- 12 files changed, 868 insertions(+), 132 deletions(-) diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 69f59d1ab..af9217085 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1248,6 +1248,7 @@ go_test( "//internal/ui", "//internal/uimd", "//internal/utils", + "//internal/validation", "//internal/workapi", "//internal/workspacegate", "//internal/worktreeremove", diff --git a/cmd/bd/cook.go b/cmd/bd/cook.go index 1a90e0754..f278c4c4f 100644 --- a/cmd/bd/cook.go +++ b/cmd/bd/cook.go @@ -1119,11 +1119,23 @@ func substituteFormulaVars(f *formula.Formula, vars map[string]string) { } // substituteStepVars recursively substitutes variables in step fields. +// +// The fields covered here must stay in sync with cloneSubgraphInto (pour): +// runtime-mode cook feeds the proto that pour later clones, so a field left +// literal in one path is a literal placeholder on the created bead either way +// (GH#5110 labels/metadata, GH#5754 assignee). func substituteStepVars(steps []*formula.Step, vars map[string]string) { for _, step := range steps { step.Title = substituteVariables(step.Title, vars) step.Description = substituteVariables(step.Description, vars) step.Notes = substituteVariables(step.Notes, vars) + step.Assignee = substituteVariables(step.Assignee, vars) + for i, label := range step.Labels { + step.Labels[i] = substituteVariables(label, vars) + } + for k, v := range step.Metadata { + step.Metadata[k] = substituteMetadataValue(v, vars) + } if step.Gate != nil { step.Gate.Type = substituteVariables(step.Gate.Type, vars) step.Gate.ID = substituteVariables(step.Gate.ID, vars) @@ -1137,6 +1149,36 @@ func substituteStepVars(steps []*formula.Step, vars map[string]string) { } } +// substituteMetadataValue substitutes variables in the string leaves of a +// decoded step metadata value (`[steps.metadata]` from TOML/JSON), at any +// nesting depth. Non-string scalars are returned untouched, and map keys are +// left alone so a rewritten key can never collide with a sibling. +func substituteMetadataValue(v interface{}, vars map[string]string) interface{} { + return substituteMetadataValueDepth(v, vars, 0) +} + +func substituteMetadataValueDepth(v interface{}, vars map[string]string, depth int) interface{} { + if depth >= maxMetadataSubstitutionDepth { + return v + } + switch val := v.(type) { + case string: + return substituteVariables(val, vars) + case map[string]interface{}: + for k, elem := range val { + val[k] = substituteMetadataValueDepth(elem, vars, depth+1) + } + return val + case []interface{}: + for i, elem := range val { + val[i] = substituteMetadataValueDepth(elem, vars, depth+1) + } + return val + default: + return v + } +} + func init() { cookCmd.Flags().Bool("dry-run", false, "Preview what would be created") cookCmd.Flags().Bool("persist", false, "Persist proto to database (legacy behavior)") diff --git a/cmd/bd/cook_test.go b/cmd/bd/cook_test.go index 92366fda6..75db50fb8 100644 --- a/cmd/bd/cook_test.go +++ b/cmd/bd/cook_test.go @@ -302,6 +302,69 @@ func TestSubstituteFormulaVars_GateFields(t *testing.T) { } } +// TestSubstituteStepVars_LabelsAssigneeMetadata covers GH#5110 and GH#5754: +// runtime-mode cook substituted Title/Description/Notes/Gate but left labels, +// assignee, and metadata literal, so `bd cook --mode runtime` emitted (and +// `--persist` stored) placeholders that no later step ever resolved. +func TestSubstituteStepVars_LabelsAssigneeMetadata(t *testing.T) { + steps := []*formula.Step{ + { + ID: "qa-contract", + Title: "API-contract QA: {{widget_id}}", + Assignee: "{{agent}}", + Labels: []string{"qa-run", "widget:{{widget_id}}", "domain:{{domain}}"}, + Metadata: map[string]interface{}{ + "ado_id": "{{ado_id}}", + "count": int64(3), + "nested": map[string]interface{}{"k": "{{domain}}"}, + "list": []interface{}{"{{domain}}", "static"}, + }, + Children: []*formula.Step{ + {ID: "child", Assignee: "{{agent}}", Labels: []string{"widget:{{widget_id}}"}}, + }, + }, + } + + vars := map[string]string{ + "widget_id": "my_widget", + "domain": "payroll", + "agent": "design-agent", + "ado_id": "731878", + } + substituteStepVars(steps, vars) + + step := steps[0] + if step.Assignee != "design-agent" { + t.Errorf("Assignee = %q, want %q", step.Assignee, "design-agent") + } + wantLabels := []string{"qa-run", "widget:my_widget", "domain:payroll"} + for i, want := range wantLabels { + if step.Labels[i] != want { + t.Errorf("Labels[%d] = %q, want %q", i, step.Labels[i], want) + } + } + if got := step.Metadata["ado_id"]; got != "731878" { + t.Errorf("Metadata[ado_id] = %v, want %q", got, "731878") + } + if got := step.Metadata["count"]; got != int64(3) { + t.Errorf("Metadata[count] = %v (%T), want int64(3) - non-string scalars must be untouched", got, got) + } + if got := step.Metadata["nested"].(map[string]interface{})["k"]; got != "payroll" { + t.Errorf("Metadata[nested][k] = %v, want %q", got, "payroll") + } + if got := step.Metadata["list"].([]interface{})[0]; got != "payroll" { + t.Errorf("Metadata[list][0] = %v, want %q", got, "payroll") + } + + child := step.Children[0] + if child.Assignee != "design-agent" { + t.Errorf("child Assignee = %q, want %q", child.Assignee, "design-agent") + } + if child.Labels[0] != "widget:my_widget" { + t.Errorf("child Labels[0] = %q, want %q", child.Labels[0], "widget:my_widget") + } +} + // TestSubstituteStepVarsRecursive tests deep nesting works correctly func TestSubstituteStepVarsRecursive(t *testing.T) { steps := []*formula.Step{ diff --git a/cmd/bd/mol_bond.go b/cmd/bd/mol_bond.go index 2a61f0908..3ea600de9 100644 --- a/cmd/bd/mol_bond.go +++ b/cmd/bd/mol_bond.go @@ -490,9 +490,17 @@ func buildAttachCloneOpts(subgraph *TemplateSubgraph, mol *types.Issue, bondType if err := refuseConditionalRefArm(bondType, childRef); err != nil { return CloneOptions{}, err } - requiredVars := extractAllVariables(subgraph) + // Resolve variables the way pour (cmd/bd/pour.go) and wisp (cmd/bd/wisp.go) + // do: fill declared defaults first, then demand only the declared vars that + // have none. Demanding every handlebar the subgraph mentions would fail the + // bond on a proto's documentation placeholders and on declared vars that + // carry only a default - and widening the scanned field set (assignee, + // labels, metadata) would have widened that refusal too. + // extractRequiredVariables returns the unfiltered list when VarDefs is nil, + // so a legacy template still requires every var it mentions. + vars = applyVariableDefaults(vars, subgraph) var missingVars []string - for _, v := range requiredVars { + for _, v := range extractRequiredVariables(subgraph) { if _, ok := vars[v]; !ok { missingVars = append(missingVars, v) } diff --git a/cmd/bd/mol_embedded_test.go b/cmd/bd/mol_embedded_test.go index 4ba004377..64553fb3d 100644 --- a/cmd/bd/mol_embedded_test.go +++ b/cmd/bd/mol_embedded_test.go @@ -3,12 +3,14 @@ package main import ( + "encoding/json" "os" "testing" "github.com/steveyegge/beads/internal/formula" "github.com/steveyegge/beads/internal/storage/embeddeddolt" "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/validation" ) // TestSpawnMolecule_PreservesStepLabels verifies that a cooked formula with @@ -71,3 +73,103 @@ func TestSpawnMolecule_PreservesStepLabels(t *testing.T) { } } } + +// TestSpawnMolecule_SubstitutesTemplatedStepFields is the cook -> pour +// regression test for GH#5110 (labels, metadata) and GH#5754 (assignee). +// TestSpawnMolecule_PreservesStepLabels above fixed labels being dropped +// entirely, but its labels carry no {{vars}}, so the missing substitution went +// unnoticed: the labels ARE carried through - just literal, which silently +// stops matching every label query downstream. +// +// The assignee case is the severe one. An unsubstituted assignee makes the +// spawned bead unclosable by the actor that poured it, so the molecule can +// never advance - which is what the AssigneeMatches assertion pins, not just +// the string value. +func TestSpawnMolecule_SubstitutesTemplatedStepFields(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + + ctx := t.Context() + s, err := embeddeddolt.Open(ctx, t.TempDir(), "beads", "main") + if err != nil { + t.Fatalf("embeddeddolt.Open failed: %v", err) + } + defer s.Close() + if err := s.SetConfig(ctx, "issue_prefix", "test"); err != nil { + t.Fatalf("SetConfig failed: %v", err) + } + + const actor = "design-agent" + + f := &formula.Formula{ + Formula: "subst-test", + Version: 1, + Type: formula.TypeWorkflow, + Steps: []*formula.Step{ + { + ID: "qa", + Title: "API-contract QA: {{widget_id}}", + Assignee: "{{agent}}", + Labels: []string{"qa-run", "widget:{{widget_id}}", "domain:{{domain}}"}, + Metadata: map[string]interface{}{"ado_id": "{{ado_id}}"}, + }, + }, + } + + subgraph, err := cookFormulaToSubgraph(f, "subst-test") + if err != nil { + t.Fatalf("cookFormulaToSubgraph failed: %v", err) + } + + vars := map[string]string{ + "widget_id": "my_widget", + "domain": "payroll", + "agent": actor, + "ado_id": "731878", + } + result, err := spawnMolecule(ctx, s, subgraph, vars, "", actor, false, types.IDPrefixMol) + if err != nil { + t.Fatalf("spawnMolecule failed: %v", err) + } + + newQAID, ok := result.IDMapping["subst-test.qa"] + if !ok { + t.Fatalf("result.IDMapping missing entry for subst-test.qa; got %v", result.IDMapping) + } + issue, err := s.GetIssue(ctx, newQAID) + if err != nil { + t.Fatalf("GetIssue(%s) failed: %v", newQAID, err) + } + + labels, err := s.GetLabels(ctx, newQAID) + if err != nil { + t.Fatalf("GetLabels(%s) failed: %v", newQAID, err) + } + gotLabels := make(map[string]bool, len(labels)) + for _, l := range labels { + gotLabels[l] = true + } + for _, want := range []string{"qa-run", "widget:my_widget", "domain:payroll"} { + if !gotLabels[want] { + t.Errorf("spawned issue %s missing substituted label %q; got %v", newQAID, want, labels) + } + } + + var metadata struct { + ADOID string `json:"ado_id"` + } + if err := json.Unmarshal(issue.Metadata, &metadata); err != nil { + t.Fatalf("spawned issue metadata = %s, not valid JSON: %v", issue.Metadata, err) + } + if metadata.ADOID != "731878" { + t.Errorf("spawned issue metadata.ado_id = %q, want %q", metadata.ADOID, "731878") + } + + if issue.Assignee != actor { + t.Errorf("spawned issue assignee = %q, want %q", issue.Assignee, actor) + } + if err := validation.AssigneeMatches(actor, false)(newQAID, issue); err != nil { + t.Errorf("spawned issue is not closable by the pouring actor %q: %v", actor, err) + } +} diff --git a/cmd/bd/template.go b/cmd/bd/template.go index 2d6ec96bf..b411ed499 100644 --- a/cmd/bd/template.go +++ b/cmd/bd/template.go @@ -340,14 +340,31 @@ func isHandlebarsKeyword(name string) bool { } } -// extractAllVariables finds all variables across the entire subgraph +// extractAllVariables finds all variables across the entire subgraph. +// +// The fields scanned here must stay in sync with the fields cloneSubgraphInto +// substitutes - a var that pour resolves but never demands leaves a silent +// literal placeholder in the poured bead, and a var it demands but never +// resolves is a closed loop that fails the pour for nothing (GH#5110, +// GH#5754). func extractAllVariables(subgraph *TemplateSubgraph) []string { - allText := "" - for _, issue := range subgraph.Issues { - allText += issue.Title + " " + issue.Description + " " - allText += issue.Design + " " + issue.AcceptanceCriteria + " " + issue.Notes + " " + var sb strings.Builder + write := func(parts ...string) { + for _, p := range parts { + if p == "" { + continue + } + sb.WriteString(p) + sb.WriteByte(' ') + } } - return extractVariables(allText) + for _, issue := range subgraph.Issues { + write(issue.Title, issue.Description, issue.Design, issue.AcceptanceCriteria, issue.Notes) + write(issue.Assignee, issue.AwaitID) + write(issue.Labels...) + write(metadataVarStrings(issue.Metadata)...) + } + return extractVariables(sb.String()) } // extractRequiredVariables returns only variables that don't have defaults. @@ -415,64 +432,160 @@ func substituteVariables(text string, vars map[string]string) string { }) } -// substituteMetadataRepo substitutes {{variable}} placeholders in an issue's -// metadata.repo value (SF2 follow-up). A formula gate step's `repo` selector -// (e.g. repo = "{{gate_repo}}") is stored literally on the persisted proto's -// metadata by createGateIssue/persistCookFormula - `bd cook --persist` keeps -// the proto reusable across pours rather than substituting at compile time. -// Substitution instead needs to happen at the same point as every other -// var-bearing issue field (Title, Description, AwaitID, ...): here, in +// maxMetadataSubstitutionDepth bounds the recursion in walkJSONStrings and in +// cook.go's substituteMetadataValueDepth. Metadata is arbitrary JSON that can +// arrive from an untrusted proto, and a deeply nested value must not blow the +// stack. Reaching the bound deliberately stops substituting and returns the +// value as-is rather than erroring: a `{{var}}` nested deeper than this ships +// as a literal placeholder. That is the intended trade - a fence against a +// hostile proto, not a limit any real formula is expected to meet. +const maxMetadataSubstitutionDepth = 32 + +// substituteMetadataVars substitutes {{variable}} placeholders in every string +// value of an issue's metadata, at any nesting depth. +// +// Formula step metadata (`[steps.metadata]`) and a gate step's `repo` selector +// (repo = "{{gate_repo}}") are stored literally on the persisted proto by +// processStepToIssue/createGateIssue - `bd cook --persist` keeps the proto +// reusable across pours rather than substituting at compile time. Substitution +// instead happens at the same point as every other var-bearing issue field +// (Title, Description, Assignee, Labels, AwaitID, ...): here, in // cloneSubgraphInto, when a proto is poured/spawned into real issues. // -// Restricted to gh:* gate types (SF4), matching createGateIssue's write-side -// rule: `repo` on a human/timer/bead gate is unrelated, ordinary metadata, -// not a GitHub repo selector, so it must not be touched here either. +// This supersedes the earlier gh:*-gate-only, top-level-"repo"-only rule +// (SF2/SF4). That restriction existed because interpreting a `repo` key as a +// GitHub selector is only correct on a gh:* gate - but substituting a +// {{var}} placeholder interprets nothing about the key, and general metadata +// carrying literal placeholders was its own bug (GH#5110). A value with no +// placeholder is unaffected either way. // -// Metadata is arbitrary JSON on any issue, so this only touches a top-level -// string-valued "repo" key; anything else (missing key, non-object, non- -// string value) is left untouched for githubRepoFromIssue to validate at -// check time. The round-trip unmarshals into map[string]json.RawMessage -// rather than map[string]interface{} and replaces only the "repo" entry, so -// every OTHER key's value survives byte-identical - interface{} would -// mangle numbers to float64, and a full re-marshal of decoded values can -// reshuffle nested object keys and HTML-escape strings that were never -// touched. -func substituteMetadataRepo(metadata json.RawMessage, awaitType string, vars map[string]string) json.RawMessage { - if len(metadata) == 0 || !isGitHubGateType(awaitType) { +// The walk decodes into json.RawMessage rather than interface{} and rebuilds +// only the containers along a changed path, so every untouched value survives +// byte-identical - interface{} would mangle numbers to float64, and a full +// re-marshal of decoded values can HTML-escape strings that were never +// touched. Metadata with no substitutable placeholder is returned as-is. +func substituteMetadataVars(metadata json.RawMessage, vars map[string]string) json.RawMessage { + if len(metadata) == 0 { return metadata } - - var raw map[string]json.RawMessage - if err := json.Unmarshal(metadata, &raw); err != nil { + out, changed := walkJSONStrings(metadata, 0, func(s string) string { + return substituteVariables(s, vars) + }) + if !changed { return metadata } + return out +} - repoRaw, hasRepo := raw["repo"] - if !hasRepo { - return metadata +// metadataVarStrings returns every string leaf in an issue's metadata, for +// variable extraction. Object keys are excluded because substitution does not +// touch them - scanning them would make pour demand a variable it then refuses +// to resolve. +func metadataVarStrings(metadata json.RawMessage) []string { + if len(metadata) == 0 { + return nil + } + var found []string + walkJSONStrings(metadata, 0, func(s string) string { + found = append(found, s) + return s + }) + return found +} + +// walkJSONStrings applies fn to every string leaf of a JSON value, at any +// nesting depth. It reports whether fn changed anything; when nothing did, the +// input bytes are returned untouched. +// +// Object keys are deliberately left alone: rewriting a key could collide with +// a sibling key and silently drop a value. +func walkJSONStrings(raw json.RawMessage, depth int, fn func(string) string) (json.RawMessage, bool) { + trimmed := bytes.TrimLeft(raw, " \t\r\n") + if len(trimmed) == 0 { + return raw, false } - var repoStr string - if err := json.Unmarshal(repoRaw, &repoStr); err != nil { - // Non-string (e.g. null) repo value: leave untouched for - // githubRepoFromIssue to reject at check time. - return metadata + switch trimmed[0] { + case '"': + var s string + if err := json.Unmarshal(raw, &s); err != nil { + return raw, false + } + replaced := fn(s) + if replaced == s { + return raw, false + } + encoded, err := marshalNoHTMLEscape(replaced) + if err != nil { + return raw, false + } + return encoded, true + + case '{': + if depth >= maxMetadataSubstitutionDepth { + return raw, false + } + var obj map[string]json.RawMessage + if err := json.Unmarshal(raw, &obj); err != nil { + return raw, false + } + changed := false + for k, v := range obj { + if newV, c := walkJSONStrings(v, depth+1, fn); c { + obj[k] = newV + changed = true + } + } + if !changed { + return raw, false + } + out, err := marshalNoHTMLEscape(obj) + if err != nil { + return raw, false + } + return out, true + + case '[': + if depth >= maxMetadataSubstitutionDepth { + return raw, false + } + var arr []json.RawMessage + if err := json.Unmarshal(raw, &arr); err != nil { + return raw, false + } + changed := false + for i, v := range arr { + if newV, c := walkJSONStrings(v, depth+1, fn); c { + arr[i] = newV + changed = true + } + } + if !changed { + return raw, false + } + out, err := marshalNoHTMLEscape(arr) + if err != nil { + return raw, false + } + return out, true } - substituted := substituteVariables(repoStr, vars) - if substituted == repoStr { - return metadata - } + // Number, bool, null: no string leaf here. + return raw, false +} - substitutedJSON, err := marshalNoHTMLEscape(substituted) - if err != nil { - return metadata +// substituteLabels returns labels with {{variable}} placeholders substituted. +// A formula step's labels are carried onto the proto literally by +// processStepToIssue, so - like Title and Description - they resolve here, at +// pour time (GH#5110). Returns nil for an empty input so an issue with no +// labels keeps a nil slice. +func substituteLabels(labels []string, vars map[string]string) []string { + if len(labels) == 0 { + return nil } - raw["repo"] = substitutedJSON - - out, err := marshalNoHTMLEscape(raw) - if err != nil { - return metadata + out := make([]string, len(labels)) + for i, l := range labels { + out[i] = substituteVariables(l, vars) } return out } @@ -480,7 +593,7 @@ func substituteMetadataRepo(metadata json.RawMessage, awaitType string, vars map // marshalNoHTMLEscape is json.Marshal without HTML-escaping '<', '>', and // '&' - the stdlib's json.Marshal escapes them by default (aimed at // embedding JSON in HTML), which would silently corrupt an unrelated -// metadata value round-tripped through substituteMetadataRepo. +// metadata value round-tripped through substituteMetadataVars. func marshalNoHTMLEscape(v interface{}) (json.RawMessage, error) { var buf bytes.Buffer enc := json.NewEncoder(&buf) @@ -678,8 +791,13 @@ func cloneSubgraphInto(ctx context.Context, w molWriter, subgraph *TemplateSubgr if opts.RootOnly && oldIssue.ID != subgraph.Root.ID { continue } - // Determine assignee: use override for root epic, otherwise keep template's - issueAssignee := oldIssue.Assignee + // Determine assignee: use override for root epic, otherwise substitute + // the template's. Step.assignee is documented as supporting + // substitution, and an unsubstituted one is worse than cosmetic - it + // makes the poured bead unclosable, because close refuses when the + // actor doesn't match the assignee (GH#5754). The --assignee override + // is a literal value supplied on the command line, so it wins as-is. + issueAssignee := substituteVariables(oldIssue.Assignee, opts.Vars) if oldIssue.ID == subgraph.Root.ID && opts.Assignee != "" { issueAssignee = opts.Assignee } @@ -702,8 +820,8 @@ func cloneSubgraphInto(ctx context.Context, w molWriter, subgraph *TemplateSubgr AwaitType: oldIssue.AwaitType, AwaitID: substituteVariables(oldIssue.AwaitID, opts.Vars), Timeout: oldIssue.Timeout, - Labels: oldIssue.Labels, - Metadata: substituteMetadataRepo(oldIssue.Metadata, oldIssue.AwaitType, opts.Vars), + Labels: substituteLabels(oldIssue.Labels, opts.Vars), + Metadata: substituteMetadataVars(oldIssue.Metadata, opts.Vars), CreatedAt: time.Now(), UpdatedAt: time.Now(), } diff --git a/cmd/bd/template_test.go b/cmd/bd/template_test.go index bcdcb8c8a..354917557 100644 --- a/cmd/bd/template_test.go +++ b/cmd/bd/template_test.go @@ -17,6 +17,7 @@ import ( "github.com/steveyegge/beads/internal/storage/dolt" "github.com/steveyegge/beads/internal/testutil" "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/validation" ) // ============================================================================= @@ -86,80 +87,86 @@ func TestSubstituteVariables(t *testing.T) { } } -// TestSubstituteMetadataRepo covers the SF2 follow-up: a gate step's `repo` -// selector ("repo": "{{gate_repo}}") is stored literally in the persisted -// proto's metadata by cook --persist (compile-time mode never substitutes), -// so substitution must happen at the same point other var-bearing issue -// fields (Title, Description, AwaitID, ...) are substituted: here, when a -// proto is cloned/poured into real issues. -func TestSubstituteMetadataRepo(t *testing.T) { +// TestSubstituteMetadataVars covers the SF2 follow-up and GH#5110: step +// metadata (`[steps.metadata]`) and a gate step's `repo` selector +// ("repo": "{{gate_repo}}") are stored literally in the persisted proto's +// metadata by cook --persist (compile-time mode never substitutes), so +// substitution must happen at the same point other var-bearing issue fields +// (Title, Description, Assignee, Labels, AwaitID, ...) are substituted: here, +// when a proto is cloned/poured into real issues. +func TestSubstituteMetadataVars(t *testing.T) { t.Parallel() tests := []struct { - name string - metadata string - awaitType string - vars map[string]string - want string + name string + metadata string + vars map[string]string + want string }{ { - name: "substitutes repo variable", - metadata: `{"repo":"{{gate_repo}}"}`, - awaitType: "gh:run", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"repo":"srobroek/agentic-packages"}`, + name: "substitutes repo variable", + metadata: `{"repo":"{{gate_repo}}"}`, + vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, + want: `{"repo":"srobroek/agentic-packages"}`, }, { - name: "gh:pr gate type also substitutes", - metadata: `{"repo":"{{gate_repo}}"}`, - awaitType: "gh:pr", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"repo":"srobroek/agentic-packages"}`, + // GH#5110: general step metadata, not a gate repo selector. + name: "substitutes arbitrary metadata key", + metadata: `{"ado_id":"{{ado_id}}","domain":"team/{{domain}}"}`, + vars: map[string]string{"ado_id": "731878", "domain": "payroll"}, + want: `{"ado_id":"731878","domain":"team/payroll"}`, }, { - name: "missing var leaves placeholder", - metadata: `{"repo":"{{gate_repo}}"}`, - awaitType: "gh:run", - vars: map[string]string{}, - want: `{"repo":"{{gate_repo}}"}`, + name: "substitutes nested and array values", + metadata: `{"outer":{"inner":"{{v}}"},"list":["{{v}}","static"]}`, + vars: map[string]string{"v": "resolved"}, + want: `{"outer":{"inner":"resolved"},"list":["resolved","static"]}`, }, { - name: "no repo key untouched", - metadata: `{"other":"value"}`, - awaitType: "gh:run", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"other":"value"}`, + name: "missing var leaves placeholder", + metadata: `{"repo":"{{gate_repo}}"}`, + vars: map[string]string{}, + want: `{"repo":"{{gate_repo}}"}`, }, { - name: "empty metadata untouched", - metadata: "", - awaitType: "gh:run", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: "", + name: "no placeholder untouched", + metadata: `{"other":"value"}`, + vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, + want: `{"other":"value"}`, }, { - name: "null repo left for check-time validation", - metadata: `{"repo":null}`, - awaitType: "gh:run", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"repo":null}`, + name: "empty metadata untouched", + metadata: "", + vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, + want: "", }, { - // SF4 consistency: a repo field on a non-gh:* gate (or any other - // issue with an unrelated "repo" metadata key) is ordinary - // metadata, not a GitHub repo selector - matching createGateIssue's - // write-side isGitHubGateType restriction. - name: "non-github gate type leaves repo untouched", - metadata: `{"repo":"{{gate_repo}}"}`, - awaitType: "human", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"repo":"{{gate_repo}}"}`, + name: "null repo left for check-time validation", + metadata: `{"repo":null}`, + vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, + want: `{"repo":null}`, }, { - name: "empty await type leaves repo untouched", - metadata: `{"repo":"{{gate_repo}}"}`, - awaitType: "", - vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, - want: `{"repo":"{{gate_repo}}"}`, + // Supersedes the earlier SF4 gh:*-only rule. That restriction + // existed because reading a `repo` key AS a GitHub selector is + // only correct on a gh:* gate - but resolving a {{var}} in it + // interprets the key not at all, and a bead poured with a literal + // `{{gate_repo}}` in its metadata is broken whatever the gate type. + name: "non-github gate metadata also substitutes", + metadata: `{"repo":"{{gate_repo}}"}`, + vars: map[string]string{"gate_repo": "srobroek/agentic-packages"}, + want: `{"repo":"srobroek/agentic-packages"}`, + }, + { + name: "object keys are never rewritten", + metadata: `{"{{key}}":"{{val}}"}`, + vars: map[string]string{"key": "k", "val": "v"}, + want: `{"{{key}}":"v"}`, + }, + { + name: "malformed metadata returned untouched", + metadata: `{not json`, + vars: map[string]string{"v": "x"}, + want: `{not json`, }, } @@ -169,10 +176,17 @@ func TestSubstituteMetadataRepo(t *testing.T) { if tt.metadata != "" { metadata = json.RawMessage(tt.metadata) } - got := substituteMetadataRepo(metadata, tt.awaitType, tt.vars) + got := substituteMetadataVars(metadata, tt.vars) if tt.want == "" { if len(got) != 0 { - t.Errorf("substituteMetadataRepo(%q, %q, %v) = %q, want empty", tt.metadata, tt.awaitType, tt.vars, got) + t.Errorf("substituteMetadataVars(%q, %v) = %q, want empty", tt.metadata, tt.vars, got) + } + return + } + if !json.Valid([]byte(tt.want)) { + // Malformed input: assert the bytes came back verbatim. + if string(got) != tt.want { + t.Errorf("substituteMetadataVars(%q, %v) = %s, want %s", tt.metadata, tt.vars, got, tt.want) } return } @@ -187,25 +201,60 @@ func TestSubstituteMetadataRepo(t *testing.T) { gotJSON, _ := json.Marshal(gotObj) wantJSON, _ := json.Marshal(wantObj) if string(gotJSON) != string(wantJSON) { - t.Errorf("substituteMetadataRepo(%q, %q, %v) = %s, want %s", tt.metadata, tt.awaitType, tt.vars, got, tt.want) + t.Errorf("substituteMetadataVars(%q, %v) = %s, want %s", tt.metadata, tt.vars, got, tt.want) } }) } } -// TestSubstituteMetadataRepo_PreservesUnrelatedValuesByteIdentical covers the -// round-trip-fidelity half of the SF2/SF4 follow-up: substituteMetadataRepo -// must decode into map[string]json.RawMessage, not map[string]interface{}, -// so a sibling key's value survives untouched when "repo" is substituted - -// a decode into interface{} would mangle a JSON number to float64 (losing -// its original formatting/precision on re-encode), reorder a nested object's -// keys, and HTML-escape '<'/'>'/'&' inside string values that were never -// touched by the substitution. -func TestSubstituteMetadataRepo_PreservesUnrelatedValuesByteIdentical(t *testing.T) { +// TestSubstituteMetadataVars_DepthLimit verifies the recursion bound holds and +// that a value nested past the limit is returned untouched rather than +// panicking the pour. +func TestSubstituteMetadataVars_DepthLimit(t *testing.T) { + t.Parallel() + deep := `"{{v}}"` + for i := 0; i < maxMetadataSubstitutionDepth+5; i++ { + deep = `{"a":` + deep + `}` + } + got := substituteMetadataVars(json.RawMessage(deep), map[string]string{"v": "resolved"}) + if !json.Valid(got) { + t.Fatalf("result is not valid JSON: %s", got) + } + if !strings.Contains(string(got), "{{v}}") { + t.Errorf("expected the past-limit placeholder to survive untouched, got %s", got) + } +} + +// TestSubstituteLabels covers GH#5110: a step's labels are carried onto the +// proto literally by processStepToIssue, so they must resolve at pour time +// like Title and Description do. An unsubstituted label doesn't error - it +// just silently stops matching every label query downstream. +func TestSubstituteLabels(t *testing.T) { + t.Parallel() + vars := map[string]string{"widget_id": "my_widget", "domain": "payroll"} + got := substituteLabels([]string{"qa-run", "widget:{{widget_id}}", "domain:{{domain}}", "{{unset}}"}, vars) + want := []string{"qa-run", "widget:my_widget", "domain:payroll", "{{unset}}"} + if !slices.Equal(got, want) { + t.Errorf("substituteLabels() = %v, want %v", got, want) + } + if got := substituteLabels(nil, vars); got != nil { + t.Errorf("substituteLabels(nil) = %v, want nil", got) + } +} + +// TestSubstituteMetadataVars_PreservesUnrelatedValuesByteIdentical covers the +// round-trip-fidelity half of the SF2/SF4 follow-up: the walk must decode into +// map[string]json.RawMessage, not map[string]interface{}, so a sibling key's +// value survives untouched when another value is substituted - a decode into +// interface{} would mangle a JSON number to float64 (losing its original +// formatting/precision on re-encode), reorder a nested object's keys, and +// HTML-escape '<'/'>'/'&' inside string values that were never touched by the +// substitution. +func TestSubstituteMetadataVars_PreservesUnrelatedValuesByteIdentical(t *testing.T) { t.Parallel() metadata := json.RawMessage(`{"repo":"{{gate_repo}}","count":123456789012345,"nested":{"z":1,"a":2,"m":3},"note":"a & c"}`) - got := substituteMetadataRepo(metadata, "gh:run", map[string]string{"gate_repo": "srobroek/agentic-packages"}) + got := substituteMetadataVars(metadata, map[string]string{"gate_repo": "srobroek/agentic-packages"}) var raw map[string]json.RawMessage if err := json.Unmarshal(got, &raw); err != nil { @@ -293,6 +342,192 @@ func TestCloneSubgraph_SubstitutesGateRepoMetadata(t *testing.T) { } } +// TestCloneSubgraph_SubstitutesLabelsAssigneeAndMetadata is the end-to-end +// regression test for GH#5110 and GH#5754: a proto step persisted with +// templated labels, assignee, and metadata (as cook writes them - compile-time +// mode keeps placeholders so the proto stays reusable across pours) must have +// all three resolved when the proto is poured, exactly like Title and +// Description are. +// +// The assignee assertion goes through validation.AssigneeMatches rather than +// just comparing strings: an unsubstituted assignee is not a cosmetic defect, +// it makes the poured bead unclosable by the actor that poured it, so every +// step of the molecule is dead on arrival. +func TestCloneSubgraph_SubstitutesLabelsAssigneeAndMetadata(t *testing.T) { + t.Parallel() + tmpDir := t.TempDir() + testDB := filepath.Join(tmpDir, ".beads", "beads.db") + s := newTestStore(t, testDB) + ctx := context.Background() + h := &templateTestHelper{s: s, ctx: ctx, t: t} + + const actor = "design-agent" + + epic := h.createIssue("QA {{widget_id}}", "", types.TypeEpic, 1) + h.addLabel(epic.ID, BeadsTemplateLabel) + + step := &types.Issue{ + Title: "API-contract QA: {{widget_id}}", + IssueType: types.TypeTask, + Status: types.StatusOpen, + Assignee: "{{agent}}", + Labels: []string{"qa-run", "widget:{{widget_id}}", "domain:{{domain}}"}, + Metadata: json.RawMessage(`{"ado_id":"{{ado_id}}"}`), + } + if err := s.CreateIssue(ctx, step, "test-user"); err != nil { + t.Fatalf("Failed to create step issue: %v", err) + } + h.addParentChild(step.ID, epic.ID) + + subgraph, err := loadTemplateSubgraph(ctx, s, epic.ID) + if err != nil { + t.Fatalf("loadTemplateSubgraph failed: %v", err) + } + + vars := map[string]string{ + "widget_id": "my_widget", + "domain": "payroll", + "agent": actor, + "ado_id": "731878", + } + result, err := cloneSubgraph(ctx, s, subgraph, CloneOptions{Vars: vars, Actor: actor}) + if err != nil { + t.Fatalf("cloneSubgraph failed: %v", err) + } + + newStepID, ok := result.IDMapping[step.ID] + if !ok { + t.Fatalf("IDMapping missing entry for step %s: %+v", step.ID, result.IDMapping) + } + newStep, err := s.GetIssue(ctx, newStepID) + if err != nil { + t.Fatalf("Failed to get poured step: %v", err) + } + + // GH#5110: labels. An unsubstituted label doesn't error - it just stops + // matching every label-based query and automation downstream. + gotLabels := slices.Clone(newStep.Labels) + sort.Strings(gotLabels) + wantLabels := []string{"domain:payroll", "qa-run", "widget:my_widget"} + if !slices.Equal(gotLabels, wantLabels) { + t.Errorf("poured step labels = %v, want %v", gotLabels, wantLabels) + } + + // GH#5110: general step metadata, not just a gh:* gate's repo selector. + var metadata struct { + ADOID string `json:"ado_id"` + } + if err := json.Unmarshal(newStep.Metadata, &metadata); err != nil { + t.Fatalf("poured step metadata = %s, not valid JSON: %v", newStep.Metadata, err) + } + if metadata.ADOID != "731878" { + t.Errorf("poured step metadata.ado_id = %q, want %q", metadata.ADOID, "731878") + } + + // GH#5754: assignee, and the symptom that makes it severe. + if newStep.Assignee != actor { + t.Errorf("poured step assignee = %q, want %q", newStep.Assignee, actor) + } + if err := validation.AssigneeMatches(actor, false)(newStepID, newStep); err != nil { + t.Errorf("poured step is not closable by the pouring actor %q: %v", actor, err) + } +} + +// TestCloneSubgraph_AssigneeOverrideBeatsSubstitution pins the precedence +// between an explicit `bd mol pour --assignee` and a templated Step.assignee: +// the flag is a literal value supplied on the command line and wins on the +// root, while the steps still resolve their own placeholder (GH#5754). +func TestCloneSubgraph_AssigneeOverrideBeatsSubstitution(t *testing.T) { + t.Parallel() + tmpDir := t.TempDir() + testDB := filepath.Join(tmpDir, ".beads", "beads.db") + s := newTestStore(t, testDB) + ctx := context.Background() + h := &templateTestHelper{s: s, ctx: ctx, t: t} + + epic := &types.Issue{ + Title: "Root", + IssueType: types.TypeEpic, + Status: types.StatusOpen, + Assignee: "{{agent}}", + } + if err := s.CreateIssue(ctx, epic, "test-user"); err != nil { + t.Fatalf("Failed to create epic: %v", err) + } + h.addLabel(epic.ID, BeadsTemplateLabel) + + step := &types.Issue{ + Title: "Step", + IssueType: types.TypeTask, + Status: types.StatusOpen, + Assignee: "{{agent}}", + } + if err := s.CreateIssue(ctx, step, "test-user"); err != nil { + t.Fatalf("Failed to create step: %v", err) + } + h.addParentChild(step.ID, epic.ID) + + subgraph, err := loadTemplateSubgraph(ctx, s, epic.ID) + if err != nil { + t.Fatalf("loadTemplateSubgraph failed: %v", err) + } + + opts := CloneOptions{ + Vars: map[string]string{"agent": "from-var"}, + Assignee: "from-flag", + Actor: "test-user", + } + result, err := cloneSubgraph(ctx, s, subgraph, opts) + if err != nil { + t.Fatalf("cloneSubgraph failed: %v", err) + } + + newEpic, err := s.GetIssue(ctx, result.IDMapping[epic.ID]) + if err != nil { + t.Fatalf("Failed to get poured epic: %v", err) + } + if newEpic.Assignee != "from-flag" { + t.Errorf("poured root assignee = %q, want %q (--assignee overrides the template)", newEpic.Assignee, "from-flag") + } + + newStep, err := s.GetIssue(ctx, result.IDMapping[step.ID]) + if err != nil { + t.Fatalf("Failed to get poured step: %v", err) + } + if newStep.Assignee != "from-var" { + t.Errorf("poured step assignee = %q, want %q (override applies to the root only)", newStep.Assignee, "from-var") + } +} + +// TestExtractAllVariables_CoversSubstitutedFields pins the scan side of +// GH#5110/GH#5754: pour must demand every variable it is going to resolve. A +// var appearing only in an assignee, label, metadata value, or await ID used +// to go undemanded, so a declared-but-unsupplied var silently shipped a +// literal placeholder onto the poured bead. +func TestExtractAllVariables_CoversSubstitutedFields(t *testing.T) { + t.Parallel() + subgraph := &TemplateSubgraph{ + Issues: []*types.Issue{ + { + Title: "Step {{in_title}}", + Assignee: "{{in_assignee}}", + Labels: []string{"static", "widget:{{in_label}}"}, + Metadata: json.RawMessage(`{"ado_id":"{{in_metadata}}","nested":{"k":"{{in_nested_metadata}}"}}`), + AwaitID: "{{in_await_id}}", + }, + }, + } + + got := extractAllVariables(subgraph) + for _, want := range []string{ + "in_title", "in_assignee", "in_label", "in_metadata", "in_nested_metadata", "in_await_id", + } { + if !slices.Contains(got, want) { + t.Errorf("extractAllVariables() = %v, missing %q", got, want) + } + } +} + // templateTestHelper provides helpers for Beads template tests type templateTestHelper struct { s *dolt.DoltStore @@ -1170,3 +1405,104 @@ func TestFlattenUnregisteredIssueTypes(t *testing.T) { } } } + +// TestBuildAttachCloneOpts_VarResolutionMatchesPour pins that `bd mol bond` +// resolves variables exactly the way pour and wisp do, rather than demanding +// every handlebar the subgraph mentions. +// +// buildAttachCloneOpts used the unfiltered extractAllVariables and applied no +// defaults, so widening the scanned field set to assignee/labels/metadata +// (GH#5110, GH#5754) would have widened bond's hard refusal along with it: a +// proto whose label or metadata carried an undeclared documentation handlebar +// would newly fail with "missing required variables". It now runs +// applyVariableDefaults + extractRequiredVariables, matching cmd/bd/pour.go +// and cmd/bd/wisp.go. +func TestBuildAttachCloneOpts_VarResolutionMatchesPour(t *testing.T) { + t.Parallel() + + mol := &types.Issue{ID: "mol-host", Title: "Host molecule"} + + tests := []struct { + name string + issues []*types.Issue + varDefs map[string]formula.VarDef + vars map[string]string + wantErr string + wantVarVals map[string]string + }{ + { + name: "undeclared handlebars in the widened fields do not block the bond", + issues: []*types.Issue{{ + Title: "Review {{component}}", + Assignee: "{{reviewer_placeholder}}", + Labels: []string{"owner:{{team_placeholder}}"}, + Metadata: json.RawMessage(`{"runbook":"see {{runbook_placeholder}}"}`), + }}, + varDefs: map[string]formula.VarDef{"component": {Required: true}}, + vars: map[string]string{"component": "api"}, + wantVarVals: map[string]string{"component": "api"}, + }, + { + name: "a declared var that only has a default is filled, not demanded", + issues: []*types.Issue{{ + Title: "Deploy {{component}} to {{env}}", + Assignee: "{{agent}}", + }}, + varDefs: map[string]formula.VarDef{ + "component": {Required: true}, + "env": {Default: formula.StringPtr("prod")}, + "agent": {Default: formula.StringPtr("deploy-agent")}, + }, + vars: map[string]string{"component": "api"}, + wantVarVals: map[string]string{ + "component": "api", + "env": "prod", + "agent": "deploy-agent", + }, + }, + { + name: "a declared var with no default is still demanded", + issues: []*types.Issue{{Title: "Deploy {{component}}"}}, + varDefs: map[string]formula.VarDef{"component": {Required: true}}, + vars: map[string]string{}, + wantErr: "missing required variables: component", + wantVarVals: nil, + }, + { + name: "a legacy template with no VarDefs still requires every var it mentions", + issues: []*types.Issue{{Title: "Deploy {{component}}", Assignee: "{{agent}}"}}, + varDefs: nil, + vars: map[string]string{"component": "api"}, + wantErr: "missing required variables: agent", + wantVarVals: nil, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + subgraph := &TemplateSubgraph{Issues: tt.issues, VarDefs: tt.varDefs} + opts, err := buildAttachCloneOpts(subgraph, mol, types.BondTypeSequential, tt.vars, "", "actor", false, false) + + if tt.wantErr != "" { + if err == nil { + t.Fatalf("expected error %q, got nil", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("error = %q, want it to contain %q", err.Error(), tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("buildAttachCloneOpts: %v", err) + } + // The defaults have to land in opts.Vars too, not just in the + // missing-var check - cloneSubgraphInto substitutes from this map. + for name, want := range tt.wantVarVals { + if got := opts.Vars[name]; got != want { + t.Errorf("opts.Vars[%q] = %q, want %q", name, got, want) + } + } + }) + } +} diff --git a/docs/workflows/gates.md b/docs/workflows/gates.md index 6387814c9..36c032a04 100644 --- a/docs/workflows/gates.md +++ b/docs/workflows/gates.md @@ -110,7 +110,9 @@ repo = "org/downstream-repo" # check gh:run against this repo, not the current `repo` accepts a `{{var}}` placeholder (e.g. `repo = "{{gate_repo}}"`); for a formula persisted with `bd cook --persist`, the placeholder is substituted when the proto is later poured with `bd mol pour --var gate_repo=...`, the -same as `title`, `description`, and `await_id`. +same as every other var-bearing step field (`title`, `description`, `design`, +`acceptance_criteria`, `notes`, `assignee`, `labels`, `await_id`, and the +string values inside `metadata`). `bd gate discover` (auto-discovery of a `gh:run` gate's run ID) requires a workflow name hint (`await_id`/`id`, not left blank) for a gate targeting diff --git a/internal/formula/parser.go b/internal/formula/parser.go index 74c12d21f..32adfa548 100644 --- a/internal/formula/parser.go +++ b/internal/formula/parser.go @@ -400,13 +400,24 @@ func ExtractVariables(formula *Formula) []string { // Extract from formula fields extract(formula.Description) - // Extract from steps + // Extract from steps. The fields scanned here must stay in sync with the + // fields substituteStepVars (cook) and cloneSubgraphInto (pour) resolve - + // a scanned-but-unresolved field demands a variable it then ignores, and a + // resolved-but-unscanned field ships a literal placeholder (GH#5110, + // GH#5754). var extractFromStep func(*Step) extractFromStep = func(step *Step) { extract(step.Title) extract(step.Description) + extract(step.Notes) extract(step.Assignee) extract(step.Condition) + for _, label := range step.Labels { + extract(label) + } + for _, s := range metadataStrings(step.Metadata, 0) { + extract(s) + } if step.Gate != nil { extract(step.Gate.Type) extract(step.Gate.ID) @@ -426,6 +437,37 @@ func ExtractVariables(formula *Formula) []string { return vars } +// maxMetadataScanDepth bounds metadataStrings' recursion into decoded step +// metadata, which is arbitrary user-supplied TOML/JSON. +const maxMetadataScanDepth = 32 + +// metadataStrings returns every string leaf of a decoded step metadata value, +// at any nesting depth. Map keys are excluded: substitution does not rewrite +// them, so scanning them would surface a variable that never resolves. +func metadataStrings(v interface{}, depth int) []string { + if depth >= maxMetadataScanDepth { + return nil + } + switch val := v.(type) { + case string: + return []string{val} + case map[string]interface{}: + var out []string + for _, elem := range val { + out = append(out, metadataStrings(elem, depth+1)...) + } + return out + case []interface{}: + var out []string + for _, elem := range val { + out = append(out, metadataStrings(elem, depth+1)...) + } + return out + default: + return nil + } +} + // Substitute replaces {{variable}} placeholders with values. func Substitute(s string, vars map[string]string) string { return varPattern.ReplaceAllStringFunc(s, func(match string) string { diff --git a/internal/formula/parser_test.go b/internal/formula/parser_test.go index 6770ff933..28db99269 100644 --- a/internal/formula/parser_test.go +++ b/internal/formula/parser_test.go @@ -520,6 +520,21 @@ func TestExtractVariables(t *testing.T) { {ID: "s1", Title: "Deploy {{project}} to {{env}}"}, {ID: "s2", Title: "Notify {{owner}}"}, {ID: "s3", Gate: &Gate{Type: "gh:{{gate_kind}}", AwaitID: "{{pr_url}}", Timeout: "{{gate_timeout}}", Repo: "{{gate_repo}}"}}, + // The fields below are substituted by cook/pour, so they must be + // scanned here too - otherwise a var used only in one of them is + // never demanded and ships as a literal placeholder (GH#5110, + // GH#5754). + { + ID: "s4", + Notes: "see {{runbook}}", + Assignee: "{{agent}}", + Labels: []string{"static", "widget:{{widget_id}}"}, + Metadata: map[string]interface{}{ + "ado_id": "{{ado_id}}", + "count": 3, + "nested": map[string]interface{}{"k": "{{nested_var}}"}, + }, + }, }, } @@ -532,6 +547,11 @@ func TestExtractVariables(t *testing.T) { "pr_url": true, "gate_timeout": true, "gate_repo": true, + "runbook": true, + "agent": true, + "widget_id": true, + "ado_id": true, + "nested_var": true, } if len(vars) != len(want) { diff --git a/internal/formula/schema_gen.go b/internal/formula/schema_gen.go index 08a9fe3b6..b92fbc286 100644 --- a/internal/formula/schema_gen.go +++ b/internal/formula/schema_gen.go @@ -676,7 +676,7 @@ or poured.`, Name: "Labels", Type: "[]string", JSONName: "labels", - Doc: `Labels are applied to the created issue.`, + Doc: `Labels are applied to the created issue (supports substitution).`, }, { Name: "Metadata", @@ -684,7 +684,8 @@ or poured.`, JSONName: "metadata", Doc: `Metadata is carried through to the created issue's Metadata field as JSON. Lets formulas pre-declare keys that downstream tooling can project -without a post-pour compose step.`, +without a post-pour compose step. String values support substitution at +any nesting depth; keys are never rewritten.`, }, { Name: "DependsOn", diff --git a/internal/formula/types.go b/internal/formula/types.go index e5d25969e..e0321ed96 100644 --- a/internal/formula/types.go +++ b/internal/formula/types.go @@ -223,12 +223,13 @@ type Step struct { // Priority is the issue priority (0-4). Priority *int `json:"priority,omitempty"` - // Labels are applied to the created issue. + // Labels are applied to the created issue (supports substitution). Labels []string `json:"labels,omitempty"` // Metadata is carried through to the created issue's Metadata field as // JSON. Lets formulas pre-declare keys that downstream tooling can project - // without a post-pour compose step. + // without a post-pour compose step. String values support substitution at + // any nesting depth; keys are never rewritten. Metadata map[string]interface{} `json:"metadata,omitempty" toml:"metadata,omitempty"` // DependsOn lists step IDs this step blocks on (within the formula).