beads/cmd/bd/pour_unknown_vars_pipeline_test.go
Harry Miller 35bac24ecc
fix(pour): reject --var names the proto cannot consume (#5762)
* fix(pour): reject --var names the proto cannot consume

An unrecognized --var name was dropped without comment. The failure that
surfaces is not a missing substitution but a missing step: a typo in an
optional var leaves its condition evaluating against an unset name, so
the conditional step is quietly never poured and the molecule looks like
it simply does not have that step.

checkPourVars now rejects names that no proto in the pour declares in
VarDefs or references as a {{handlebar}}, listing what is available.
Anything that could still affect the pour is accepted: a documentation
handlebar the formula never declared, and a var belonging to an --attach
proto rather than the root, both pass. Missing required vars are reported
first, unchanged, since the caller's hint is derived from them.

That hint is now only attached when a required var really is missing.
It interpolated missingVarHint unconditionally, so any other failure from
checkPourVars would have advised "--var =<value>".

* fix(wisp): reject --var names the proto cannot consume

bd mol wisp validates vars through checkRequiredVars rather than
checkPourVars, so the unknown-name rejection added for pour did not reach
it and a typo was still dropped without comment on the wisp path.

Route it through the same check. A wisp instantiates one proto with no
attachments, so the known set is that proto's declared vars plus the
handlebars it references.

The missing-var hint is now only attached when a var really is missing
here too, matching pour: otherwise any other failure from this check
advised "--var =<value>".

* fix(pour): count a var used only by a step condition as consumable

checkUnknownVars built its known set from the cooked subgraph, so it could
not see step conditions: formula.FilterStepsByCondition consumes them and
drops the steps carrying them BEFORE the cook, and nothing requires a
condition var to be declared in [vars]. An undeclared var that decides
which steps get poured therefore left no trace anywhere the check looked,
and a hard error turned a working pour into a failing one - exactly the
case the rule ("a name is rejected only when nothing in the pour could
consume it") means to accept.

TemplateSubgraph gains ConditionVars, recorded by
resolveAndCookFormulaWithVars immediately before the condition filter runs
- the last point at which those names are still visible. Collected
unconditionally, since they are a property of the formula rather than of
this pour's --var values, which is what makes the falsey case work: the
step is gone, and the var that removed it still counts. knownVarsAcross
adds them. A persisted proto loaded from the database leaves the field
empty, correctly: it has no formula behind it and its conditions were
resolved at cook time.

The reviewer's reproduction, against the pre-fix code:

    Error: unknown variables: has_spike (available: story)

Regression coverage answers the fifth review point as well - that every
existing test called checkPourVars directly with a hand-built subgraph,
which is precisely why this was invisible.
TestBdMolPour_AcceptsAVarUsedOnlyByAStepCondition runs the CLI through the
real cook pipeline with has_spike both true and false (the falsey case
being the one that regresses if the collection ever moves after the
filter), plus a typo case proving the widened set does not disable the
check. Two unit cases cover the same pair.

The check does not disable itself: a typo in a condition var is still
reported, now with the condition var among the available names.

Non-blocking review points 3 and 4: pour's and wisp's --var flag help says
an unusable name is an error, and the CHANGELOG entry states the new
behavior and says plainly that `bd mol bond --var` and `bd mol seed --var`
remain out of scope - both go through formula.ValidateProvidedVars, which
only iterates declared vars and so never examines a provided name with no
VarDef.

On review point 2 (the interaction with #5758): no action needed either
way round. #5758 widens extractAllVariables itself, and knownVarsAcross
calls that same function, so a var referenced only from a step's
assignee/labels/metadata joins this known set automatically when #5758
lands, in whichever order the two merge.

Reported-by: bee-ghosttrack
Agent-Signature: claude-code-opus-5-unknown-reasoning on behalf of Harry Miller

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(pour): exercise the real cook pipeline without spawning a binary

The end-to-end case spawned an embedded bd and ran three CLI invocations,
the most expensive shape of test in cmd/bd - and cmd/bd already runs 477s
of its 600s budget on the macos runner, so the addition timed the package
out.

The process boundary was not the point. What the unit tests could not see
was the ORDERING inside the cook pipeline, and
resolveAndCookFormulaWithVars takes its search paths as an argument, so
the real pipeline - control flow, expansions, FilterStepsByCondition, the
cook - runs directly against a formula in a temp dir. Same coverage in
0.01s instead of 32s.

It is also a sharper test than the CLI version was. It now asserts the
PREMISE of the falsey case (that the conditional step really is absent
from the cooked subgraph, so the case is testing the ordering rather than
passing for an unrelated reason), and that the typo error offers the
condition var among the available names, which is the actual user-visible
point of carrying those names into the known set.

Both reds confirmed, including the near-miss: with the known-set change
reverted all three cases fail, and with the collection moved to AFTER the
condition filter - the plausible fix, and the one the reviewer's suggested
call site would have produced - has_spike=true passes while has_spike=false
still fails:

    --- FAIL: .../has_spike_false
        pour rejected a var its own step condition consumes (has_spike=false):
        unknown variables: has_spike (available: story)

Agent-Signature: claude-code-opus-5-unknown-reasoning on behalf of Harry Miller

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(changelog): move this PR's entry under [Unreleased]

The 1.3.0 cut renamed the old [Unreleased] block, so the entry this branch
added there now merges cleanly into the released [1.3.0] section (#6643).

Agent-Signature: claude-code-opus-5-5-unknown-reasoning on behalf of Harry Miller
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore(bazel): enumerate the PR's new cmd/bd test files in go_test srcs

gazelle-generated. cmd/bd/BUILD.bazel lists go_test srcs explicitly, so the
two untagged test files this PR adds would not run under the advisory Bazel
lane added in #6881. Base parity at 6da2a4347 is exhaustive for untagged
tests (the only two unenumerated files are //go:build cgo && integration),
so the omission is PR-introduced rather than a pre-existing pattern.

Produced by make bazel-sync; no deps line was needed because go_test embeds
:bd_lib, which already depends on //internal/formula and //internal/types.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pour): read every substituted field into the unknown-var known set

Maintainer review fixup for the unknown-`--var` refusal. The refusal failed
CLOSED on a known set narrower than what the pour actually substitutes, so it
rejected names the clone demonstrably consumes.

- knownVarsAcross read the five prose fields only, missing a gate's await_id
  and a gh:* gate's metadata.repo selector; a formula's `repo = "{{gate_repo}}"`
  is stored literally on the proto and filled in at clone time, and nothing
  mirrors that name into a scanned field. substitutedIssueFields now mirrors
  cloneSubgraphInto's substitution set, and gateRepoSelector is factored out of
  substituteMetadataRepo so the read and write sides of the selector cannot
  disagree about which issues carry one.
- a proto loaded from the database has no formula behind it, so its [vars]
  declarations are gone and a declared-but-unreferenced name cannot be told
  apart from a typo. `VarDefs == nil` does not identify that case - a formula
  declaring no [vars] is also nil, and that proto really does take none - so the
  cook stamps DeclaredVarsKnown and the refusal stands down without it. That
  covers every `bd cook --persist` proto and every --attach proto, which is
  always DB-loaded.
- a handlebar living only inside a step the condition filter dropped left the
  known set, so `--var deploy=false --var deploy_target=prod` failed while
  deploy=true succeeded. ConditionVars becomes FormulaVarRefs and is collected
  from the unfiltered steps' text and gate fields as well as their conditions,
  making a name's validity a property of the formula rather than of another
  var's value.
- a standalone `type = "expansion"` formula has no [[steps]]:
  MaterializeExpansion builds them from its [[template]] before the cook,
  substituting --var values into single-brace `{name}` placeholders, so a name
  used only there was consumed and then refused as unknown.
  formula.ExpansionVarRefs reports those names by running that same expansion
  - expandStep now wraps expandStepWith - with a substitution that records
  instead of replacing, so it covers exactly the fields the expansion fills in,
  including a name that `{target.description}` pulls in from the formula's own
  description. cook adds them to FormulaVarRefs ahead of the filter, and
  without MaterializeExpansion's has-Steps skip: the filter can remove every
  step, and then the template is materialized after all.

The hint-suppression branch was hand-copied to three sites and tested at none;
it is now handleVarErrorWithHint, pinned by a stderr assertion that an
unknown-var error prints no `Provide them with: --var =<value>`.

Tests: table cases for both gate fields and for `repo` on a non-gh gate staying
unsubstituted (the polarity that keeps the read side honest), for the DB-loaded
stand-down directly and via an --attach proto, and real-pipeline cases for the
dropped-step var, for vars only a dropped step's gate uses (await_id and a
gh:* repo), and for a template-only placeholder var, whose typo is still
refused with the real name offered. ExpansionVarRefs is pinned per substituted
field, per field the expansion drops (notes, condition), and for an expansion
formula that also has steps. Mutating each arm reds exactly the matching cases
and nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Julian Knutsen <agent@gascity.ai>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 05:09:35 -07:00

327 lines
12 KiB
Go

package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// has_spike appears ONLY in a step condition: it is not declared in [vars] and
// no issue field references it. spike_area appears ONLY in the text of the step
// that same condition can remove, so it is undeclared too and survives into the
// cooked subgraph only while has_spike is truthy.
const conditionVarFormula = `formula = "condvar-pour"
version = 1
type = "workflow"
[vars.story]
required = true
[[steps]]
id = "design"
title = "Design {{story}}"
type = "task"
[[steps]]
id = "spike"
title = "Spike {{spike_area}} first"
type = "task"
condition = "{{has_spike}}"
`
// writePipelineFormula puts a formula where the cook pipeline will find it.
func writePipelineFormula(t *testing.T, name, body string) string {
t.Helper()
dir := filepath.Join(t.TempDir(), "formulas")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatalf("mkdir formulas dir: %v", err)
}
if err := os.WriteFile(filepath.Join(dir, name+".formula.toml"), []byte(body), 0o644); err != nil {
t.Fatalf("write formula: %v", err)
}
return dir
}
// The unknown-var check runs against a subgraph produced by the REAL cook
// pipeline, which is where its known set is actually assembled.
//
// The unit tests above hand-build subgraphs, so they cannot see the ordering
// that matters most here: formula.FilterStepsByCondition consumes step
// conditions and drops the steps carrying them BEFORE the cook, so a var used
// only in a condition leaves no trace in the cooked subgraph. cook records
// those names ahead of the filter for exactly this reason.
//
// Pouring with the condition var set to a FALSEY value is the case that
// regresses if the collection ever moves after the filter: the step is gone,
// and the var that removed it must still count as consumable. Reviewing this
// fix, that was the case a plausible-looking fix one call later would have
// missed.
func TestCookedSubgraphAcceptsAVarUsedOnlyByAStepCondition(t *testing.T) {
searchPaths := []string{writePipelineFormula(t, "condvar-pour", conditionVarFormula)}
for _, spike := range []string{"true", "false"} {
t.Run("has_spike_"+spike, func(t *testing.T) {
vars := map[string]string{"story": "s1", "has_spike": spike}
subgraph, err := resolveAndCookFormulaWithVars("condvar-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
// The premise of the falsey case: the step really is gone, so
// nothing in the subgraph mentions has_spike any more.
if spike == "false" {
for _, issue := range subgraph.Issues {
if strings.Contains(issue.Title, "Spike") {
t.Fatalf("the conditional step survived a falsey condition, so this case is not testing the ordering: %q", issue.Title)
}
}
}
if err := checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph)); err != nil {
t.Errorf("pour rejected a var its own step condition consumes (has_spike=%s): %v", spike, err)
}
})
}
// A var referenced only inside the step the condition removes must not lose
// its validity when that step goes: otherwise `--var has_spike=false --var
// spike_area=parser` fails while `has_spike=true` succeeds, and whether a
// name is accepted depends on another name's VALUE. The falsey case is the
// one that regresses if collection ever moves after the filter.
for _, spike := range []string{"true", "false"} {
t.Run("dropped_step_text_var_has_spike_"+spike, func(t *testing.T) {
vars := map[string]string{"story": "s1", "has_spike": spike, "spike_area": "parser"}
subgraph, err := resolveAndCookFormulaWithVars("condvar-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
if spike == "false" {
for _, issue := range subgraph.Issues {
if strings.Contains(issue.Title, "spike_area") {
t.Fatalf("the conditional step survived, so this case is not testing the dropped-step path: %q", issue.Title)
}
}
}
if err := checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph)); err != nil {
t.Errorf("pour rejected a var referenced only by a step it dropped (has_spike=%s): %v", spike, err)
}
})
}
// The condition var widens the known set; it does not disable the check.
t.Run("typo_in_the_condition_var_is_still_rejected", func(t *testing.T) {
vars := map[string]string{"story": "s1", "has_spke": "true"}
subgraph, err := resolveAndCookFormulaWithVars("condvar-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
err = checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph))
if err == nil {
t.Fatal("pour accepted a typo'd var name")
}
if !strings.Contains(err.Error(), "has_spke") {
t.Errorf("error does not name the unusable var: %v", err)
}
// The real name has to be offered, which is the whole point of
// carrying condition vars into the known set.
if !strings.Contains(err.Error(), "has_spike") {
t.Errorf("error does not offer the condition var among the available names: %v", err)
}
})
}
// run_id and gate_repo appear ONLY in the gate of a step that deploy can
// remove, and neither is declared in [vars]. createGateIssue turns that gate
// into its own issue - await_id mirrored into its title and AwaitID, and a
// gh:* gate's repo selector into metadata.repo - which goes with the step.
const gateVarFormula = `formula = "gatevar-pour"
version = 1
type = "workflow"
[[steps]]
id = "build"
title = "Build"
type = "task"
[[steps]]
id = "await-ci"
title = "Wait for CI"
type = "task"
condition = "{{deploy}}"
[steps.gate]
type = "gh:run"
await_id = "{{run_id}}"
repo = "{{gate_repo}}"
`
// A var referenced only by the GATE of a step the condition filter drops stays
// consumable, for the same reason dropped step text does: otherwise whether
// run_id is accepted would depend on deploy's value. cook reads a step's gate
// fields ahead of the filter along with its text, and the falsey case is the
// one that regresses if it ever stops - the gate issue is gone, so nothing
// else in the subgraph names either var.
func TestCookedSubgraphAcceptsAVarUsedOnlyByADroppedStepsGate(t *testing.T) {
searchPaths := []string{writePipelineFormula(t, "gatevar-pour", gateVarFormula)}
for _, deploy := range []string{"true", "false"} {
t.Run("deploy_"+deploy, func(t *testing.T) {
vars := map[string]string{"deploy": deploy, "run_id": "ci.yml", "gate_repo": "octo/app"}
subgraph, err := resolveAndCookFormulaWithVars("gatevar-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
// The premise, both ways: the gate issue exists exactly while
// its step does.
gates := 0
for _, issue := range subgraph.Issues {
if issue.AwaitType != "" {
gates++
}
}
if want := map[string]int{"true": 1, "false": 0}[deploy]; gates != want {
t.Fatalf("deploy=%s cooked %d gate issues, want %d, so this case is not testing the dropped-gate path", deploy, gates, want)
}
if err := checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph)); err != nil {
t.Errorf("pour rejected a var referenced only by a step's gate (deploy=%s): %v", deploy, err)
}
})
}
}
// release_owner, release_train and change_ticket appear ONLY in the assignee, a
// label and a nested metadata value of the step deploy can remove, and none is
// declared in [vars]. The pour substitutes all three fields (GH#5110,
// GH#5754), but no prose field mirrors them.
const fieldVarFormula = `formula = "fieldvar-pour"
version = 1
type = "workflow"
[[steps]]
id = "build"
title = "Build"
type = "task"
[[steps]]
id = "release"
title = "Release"
type = "task"
condition = "{{deploy}}"
assignee = "{{release_owner}}"
labels = ["train:{{release_train}}"]
metadata = { change = { ticket = "{{change_ticket}}" } }
`
// A var referenced only by the assignee, a label or a metadata value of a step
// the condition filter drops stays consumable too: cook reads those fields
// ahead of the filter along with the step's text, and the falsey case is the
// one that regresses if it ever stops - the step is gone, so nothing else in
// the subgraph names any of the three.
func TestCookedSubgraphAcceptsAVarUsedOnlyByADroppedStepsNonProseFields(t *testing.T) {
searchPaths := []string{writePipelineFormula(t, "fieldvar-pour", fieldVarFormula)}
for _, deploy := range []string{"true", "false"} {
t.Run("deploy_"+deploy, func(t *testing.T) {
vars := map[string]string{"deploy": deploy, "release_owner": "alice", "release_train": "r42", "change_ticket": "CHG-7"}
subgraph, err := resolveAndCookFormulaWithVars("fieldvar-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
// The premise, both ways: the release step - and with it every
// field naming these vars - exists exactly while deploy is truthy.
released := false
for _, issue := range subgraph.Issues {
if issue.Title != "Release" {
continue
}
released = true
if issue.Assignee == "" || len(issue.Labels) == 0 || len(issue.Metadata) == 0 {
t.Fatalf("the release step lost its assignee, labels or metadata in the cook, so this case is not testing them: %+v", issue)
}
}
if released != (deploy == "true") {
t.Fatalf("deploy=%s cooked the release step = %v, so this case is not testing the dropped-step path", deploy, released)
}
if err := checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph)); err != nil {
t.Errorf("pour rejected a var referenced only by a step's assignee, label or metadata (deploy=%s): %v", deploy, err)
}
})
}
}
// A standalone expansion formula has no [[steps]]: formula.MaterializeExpansion
// builds them from its [[template]] before the cook, substituting --var values
// into the template's single-brace {name} placeholders as it goes. component
// is not declared in [vars], so once it has been substituted nothing in the
// cooked subgraph - no issue field, not VarDefs - names it any more.
const expansionVarFormula = `formula = "expansion-pour"
version = 1
type = "expansion"
[[template]]
id = "{target}.build"
title = "Build {component}"
type = "task"
`
// A var that a standalone expansion formula's template consumes is accepted,
// even though consuming it is what erased every trace of it: like a step
// condition, the placeholder is used up before the cook, so cook has to record
// the name ahead of time. That widens the known set without disabling the
// check - a typo is still refused, and offered the real name.
func TestCookedSubgraphAcceptsAVarAStandaloneExpansionTemplateConsumes(t *testing.T) {
searchPaths := []string{writePipelineFormula(t, "expansion-pour", expansionVarFormula)}
t.Run("template_placeholder_var", func(t *testing.T) {
vars := map[string]string{"component": "api"}
subgraph, err := resolveAndCookFormulaWithVars("expansion-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
// The premise: the template really did consume the var before the
// cook, so the subgraph no longer mentions it.
consumed := false
for _, issue := range subgraph.Issues {
if issue.Title == "Build api" {
consumed = true
}
}
if !consumed {
t.Fatal("the template did not substitute component, so this case is not testing a consumed var")
}
if err := checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph)); err != nil {
t.Errorf("pour rejected a var the expansion template consumed: %v", err)
}
})
t.Run("typo_is_still_rejected", func(t *testing.T) {
vars := map[string]string{"componnet": "api"}
subgraph, err := resolveAndCookFormulaWithVars("expansion-pour", searchPaths, vars)
if err != nil {
t.Fatalf("cook: %v", err)
}
err = checkPourVars(subgraph, nil, applyVariableDefaults(vars, subgraph))
if err == nil {
t.Fatal("pour accepted a typo'd var name")
}
if !strings.Contains(err.Error(), "componnet") {
t.Errorf("error does not name the unusable var: %v", err)
}
if !strings.Contains(err.Error(), "available: component") {
t.Errorf("error does not offer the template's placeholder among the available names: %v", err)
}
})
}