beads/internal/httpclient/ca.go
Julian Knutsen 6d7fecaa3e
feat(httpclient): OSS HTTP client wire/encode layers (contributed from bd-enterprise under MIT) (#7210)
* feat(httpapi): wire-shape signal — wire_revision, floor header, golden digest, param tokens

S0 of the HTTP role-parity program: give a client a single place to learn
whether this build's wire shape has moved under it, and give CI a gate that
catches an undocumented move.

- GET /v0/beads/context reports wire_revision (bumps only on a non-additive
  response-shape change) and min_client_wire_revision. An optional
  Bd-Wire-Revision request header lets a client declare the revision it was
  built for; checkWireRevision refuses a declared revision below the floor
  with 400 invalid_argument/reason=wire_revision_unsupported and a
  min_wire_revision field, on every route but /healthz (wireRevisionExempt) —
  including the identity handshake, where catching the mismatch is cheapest.
  Absent header is today's behavior; malformed value is invalid_value, never
  wire_revision_unsupported. The floor lives in Server.minClientWireRevision
  (defaulting to the MinClientWireRevision constant) so a test can raise it.
- internal/httpapi/wireshape: a golden digest over every response's every
  content-type schema (including application/problem+json) across the whole
  spec, keyed by schema+member. TestWireShapeDigest fails closed when a
  member's JSON name or type changes without wire_revision bumping to match;
  regenerate with `go run ./internal/httpapi/wireshape/cmd/gendigest`.
- GET /v0/beads/issues' `sort` parameter now carries the issues.listSort
  capability token, the same convention `capabilities` entries use.
  TestNewParameterOnExistingOperationHasABehaviorToken requires every new
  parameter on an existing operation to carry one, checked against a
  write-once grandfather baseline (testdata/pretoken_parameters.json) of
  parameters that predate the rule; TestUntokenedParameterRuleFires proves
  the checker itself against a synthetic fixture.
- Docs: engdocs/design/bd-serve-v0.md gets a "The wire-shape signal" section;
  CHANGELOG.md gets an Added entry.

Live-mutated and reverted three RED demonstrations during development:
retyping api_version in the spec failed TestWireShapeDigest; short-circuiting
checkWireRevision's floor comparison failed
TestWireRevisionBelowFloorRefusesEveryEnforcedRoute; adding an untokened
query parameter to listIssues failed
TestNewParameterOnExistingOperationHasABehaviorToken. All three now pass
against the unmutated tree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpapi): rename issues.listSort to issues.list.sort, harden token rule

Opus review of 12c5144ca (HIGH-1, MEDIUM-5):

- Rename the capability token issues.listSort -> issues.list.sort to match
  bd-enterprise's client (httpstore/wire/handshake.go:121, DESIGN.txt S4e),
  and widen capabilityToken to accept an optional third segment.
- untokenedParameters() now requires the extracted token to be one
  Capabilities() actually serves, not just backtick-shaped text. This
  catches real false positives that existed in the spec already
  (workapi.DefaultReadyLimit, status.custom) which only shape-matched the
  old regex; they're added to the pretoken baseline as genuinely
  pre-existing untokened parameters.
- Extend the rule to request body members (new
  testdata/pretoken_request_body_members.json baseline) and exempt
  brand-new operations via a frozen testdata/pretoken_operations.json
  snapshot.
- Add TestPretokenBaselinesNeverGrow pinning all three baseline sizes so
  a future change can't hand-add an exemption instead of a token.
- Update spec prose for the widened rule scope.

go test ./internal/httpapi/... and the reviewer's pmut.py all pass as
expected: plain/bogus modes go RED, reuse (real-but-unrelated token) stays
green, matching the review's literal ask ("a token must be one advertised
by Capabilities()").

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(wireshape): close four digest-coverage gaps, add gendigest write guard

Opus review of 12c5144ca (HIGH-2, MEDIUM-4). The digest only tracked
Type/Format/Enum/Required on direct properties, missing:

- nullable: true (never read at all)
- array item shape (Issue.labels items string->integer was invisible;
  only object-shaped items got recursed into)
- additionalProperties value shape (IssueCount.groups)
- request bodies entirely (Compute() only walked responses, so
  DeleteIssuesRequest.expected_version string->integer, the other half
  of #6053, was invisible)

Also added defensive oneOf flattening alongside the existing allOf merge
(dormant today -- grep confirms this spec uses oneOf nowhere -- but no
longer silently invisible if that changes).

Extracted the golden-vs-computed comparison into wireshape.Compare(),
shared by TestWireShapeDigest and the new wireshape.SafeToWrite(), which
gendigest now calls before writing: it refuses to overwrite golden.json
with changed/removed entries unless the candidate's wire_revision is
strictly greater than the existing golden's. Pure additions always write.
TestSafeToWrite exercises both guard directions.

golden.json is regenerated. This single regen is a one-time exception to
the new guard, done by writing the digest directly rather than through
gendigest: confirmed via a throwaway diff (not checked in) that every one
of the 57 "changed" entries differs only in the NEW fields the wider walk
now populates (ItemType/ItemFormat/Nullable/AdditionalProps*) -- every
pre-existing Type/Format/Required/Enum value is byte-identical. Nothing on
the wire changed; the measurement got more precise. Bumping
CurrentWireRevision for this would have written a false "the protocol
changed here" entry into the revision history table. Going forward, any
REAL change the guard now reports uses the normal bump-then-regenerate
path, same as before.

Verified against the reviewer's mut.py: nested_type, nullable,
nullable_issue, array_items, addlprops, request_body and format all go RED
on TestWireShapeDigest, each reverted after confirming.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpapi): make bd_version wire_revision inference advisory

Opus review of 12c5144ca (HIGH-3). The spec told a client inferring
wire_revision from bd_version to use the release cutoff >= 1.3.0, and said
nothing about what happens when that inference is wrong -- which it
demonstrably is for real builds already in the field:

- main builds up to commit b3ef65c85 report bd_version 1.2.2 (below any
  1.3.0 cutoff) while already sending the string-typed revision/
  expected_version shape (#6053).
- 1.3.0-rc.N pre-release builds sort BELOW the release cutoff 1.3.0 under
  plain string comparison despite already carrying the string shape.
- The bd-enterprise compatibility line at d3ab32773462 reports bd_version
  1.1.0 while also sending strings.

Moved the cutoff to >= 1.3.0-rc.1 (compared as semver-with-prerelease,
which at least captures the rc builds), documented both named exceptions
inline, and made the whole inference explicitly ADVISORY: a client must
accept revision/expected_version as either a string or a number whenever
wire_revision is absent (or probe directly) rather than let the inferred
value choose a decoder, and must never refuse solely because the inferred
revision is 0. Mirrored into engdocs/design/bd-serve-v0.md.

This is spec/doc prose only -- there is no server-side inference code to
change, since this server always sends wire_revision directly; the
inference is guidance for clients reading a pre-signal server's response
(this server, past builds, or an unrelated implementation).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpapi): LOW-7/8/9/10 review follow-ups for wire-revision signal

LOW-7: WireRevisionUnsupported now also sets Problem.wire_revision (this
server's CurrentWireRevision) and Problem.bd_version on the
wire_revision_unsupported refusal, alongside the existing
min_wire_revision, so a client logging the refusal has this server's
full self-description without a second /v0/beads/context call.

LOW-8: gazelle + make bazel-sync add the BUILD.bazel entries the prior
commits were missing: wire_revision.go/wire_revision_test.go in
internal/httpapi's srcs, new BUILD.bazel files for wireshape and
wireshape/cmd/gendigest, and a testdata glob on httpapi's go_test so
the pretoken_*.json baselines are visible under `bazel test`. Verified
with `bazel build //internal/httpapi/...` and
`bazel test //internal/httpapi/... --test_env=BEADS_TEST_SKIP=dolt`
(both green) and `python3 tools/bazel/go_srcs.py --check` (clean).

LOW-9: narrowed CHANGELOG.md and engdocs/design/bd-serve-v0.md to
describe the digest's actual final coverage (format/enum/required/
nullable, array items, additionalProperties, request bodies, $ref/
allOf/oneOf recursion, the SafeToWrite revision-bump guard) and the
token rule's actual final shape (three pinned baselines, request-body
members, Capabilities()-membership validation, new-operation
exemption) instead of the original, narrower claims.

LOW-10: Bd-Wire-Revision now parses via a strict decimal-only
parseStrictNonNegativeInt (no leading '+', no leading zero except a
bare "0") instead of strconv.Atoi alone, so two different header
bytes can no longer name the same revision. New malformed-header test
cases: "+1", "01", "007".

Regenerated apigen/types.gen.go (make api-gen) for the new Problem
fields and wireshape/testdata/golden.json (go run
.../wireshape/cmd/gendigest) for the two new additive Problem members;
the latter wrote cleanly through SafeToWrite itself, since both are
pure additions.

go test ./... and bazel test //internal/httpapi/... both green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpapi): re-review LOW follow-ups — retirement test, gendigest -init, docs

1. TestRetiredRevisionsStayBelowTheFloor pins the PRESENCE guarantee the
   wire_revision property doc relies on: CurrentWireRevision >= 2 (so 0
   and 1 stay permanently retired) and MinClientWireRevision <=
   CurrentWireRevision (so this build's own floor never refuses every
   request against itself).

2. gendigest now refuses to create a golden.json that does not exist yet
   unless run with -init. Previously a missing file (moved testdata,
   wrong path, accidental delete) was indistinguishable from a real first
   run and silently wrote a fresh golden with no revision history checked
   against anything. writeGolden() is extracted out of main() so this is
   covered directly: TestWriteGoldenRequiresInitWhenMissing,
   TestWriteGoldenInitCreatesWhenMissing,
   TestWriteGoldenIgnoresInitWhenGoldenExists (the existing-golden path is
   still governed only by SafeToWrite, -init or not).

3. Narrowed the two remaining stale claims that still described the
   digest's original, narrower coverage: the wire_revision property's
   prose in openapi.v0.yaml (regenerated into apigen/types.gen.go via
   make api-gen) and wireshape/doc.go now both list nullable, array item
   shape, additionalProperties value shape, request bodies, and
   $ref/allOf/oneOf recursion, plus the new -init guard.

go test ./internal/httpapi/... and make api-check both green;
bazel test //internal/httpapi/... (httpapi, wireshape, and the new
gendigest test target) green; python3 tools/bazel/go_srcs.py --check
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(wireshape): cover operation parameters in the wire-shape digest

The digest previously walked only response and request-body schemas, so a
type, enum, required, or removal change on one of the spec's 138 operation
parameters (123 query, 14 path, 1 header) passed the gate silently — the
open item from PR review on #7182.

Each parameter is now recorded as its own Entry, keyed by
"param:"+operationId / location+":"+name (operation-scoped, not
schema-scoped, since two operations' same-named parameter are independent
wire contracts), capturing type, array item shape, enum, required, style,
explode and default. Adding a parameter stays additive; removing,
retyping, narrowing its enum, or changing required/style/explode/default
now trips SafeToWrite exactly like a response or request-body change.

Verified against the live spec: retyping listIssues' `limit` query
parameter from integer to string now fails TestWireShapeDigest (confirmed
red, then reverted — not part of this commit). golden.json regenerated via
gendigest; the new parameter entries are a pure addition, no wire_revision
bump needed. Added TestParameterMutationsAreCaught (remove/retype/narrow-
enum against real golden entries) and updated wireshape's doc.go, the
`wire_revision` spec prose, and the CHANGELOG to describe the new
coverage.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(wireshape): review fixes — stale-golden and lowered-revision arms, effective param serialization, digest boundary

Maintainer fixup from the adopt-pr review loop (iterations 1 and 2), on
top of e17ba924f.

- goldenDrift (A, I): TestWireShapeDigest's comparison moves into a
  goldenDrift helper. An addition alongside a moved wire_revision no
  longer passes with a stale golden: it asks for a regenerate, as a
  changed or removed entry after a bump already did. Every message now
  names response/request-body members AND parameters, and TestGoldenDrift
  pins each arm. This takes only the test half of the stranded fixup
  (adopt-pr/bd-47kdge/stranded-fixup-528cd6095); its doc half, which says
  parameters are outside the digest, is false since e17ba924f and is not
  replayed (J).
- Lowered wire_revision (D): a CurrentWireRevision LOWER than the
  golden's is a wrong constant, not a stale golden, so goldenDrift says to
  restore it instead of regenerating, and SafeToWrite (so gendigest)
  refuses any write at a lower revision, whatever the entries.
- Effective style/explode (B): walkParam records style and explode with
  OpenAPI 3.0's defaults filled in (form for query/cookie, simple for
  path/header; explode defaults to style == form). Adding `explode: false`
  to a query array that relied on form's implicit explode is now a
  changed entry, while spelling out a default changes nothing.
  testdata/golden.json is regenerated with -init: 111 parameter entries
  change only in style/explode (96 to form/true, 15 to simple), 828
  entries before and after, wire_revision stays 2. The golden is new in
  this PR and unreleased, so this needs no bump.
- TestWalkParam (H): a package-internal table test
  (wireshape_internal_test.go) pins every field walkParam extracts,
  through a computeFrom(document, revision) seam that Compute now wraps:
  per-location defaults, $ref'd parameters and items, format, nullable and
  default, content parameters, and the nil returns.
  TestEffectiveSerializationIsCompared pins B end to end through Compare
  and SafeToWrite. BUILD.bazel embeds the library in the go_test, which is
  gazelle's own output.
- Required inputs and the digest boundary (C, E, F; docs only): all five
  prose surfaces (openapi.v0.yaml's wire_revision description and the
  regenerated apigen/types.gen.go, wireshape/doc.go, CHANGELOG.md,
  engdocs/design/bd-serve-v0.md) now say that a new REQUIRED request
  input is recorded as an ordinary addition and needs its own review,
  that value constraints and member-level composition are outside the
  digest, and that object and content parameters are recorded only as
  their container. bd-serve-v0.md also gains the parameter coverage it
  was missing.
- G: a brand-new operation is permanently exempt from the
  capability-token rule; its own review decides its capability story.
- Iteration 2 (nit): walkParam's comment now says that the location
  default it records as a `content` parameter's style and explode is a
  placeholder, since OpenAPI 3.0 applies style and explode only to a
  parameter described by `schema`. All 138 operation parameters in the
  spec use `schema`, so the golden does not change.

Validated: go test ./internal/httpapi/... (gms_pure_go), make api-check,
go vet for linux/windows/darwin, gofmt, git diff --check, gazelle on the
package (no drift), make bazel-sync-check, and bazel test
//internal/httpapi/wireshape/... in the local processwrapper sandbox
(2 of 2 pass). Overlay mutation probes: dropping the added-after-bump
arm, the lowered-revision arm, SafeToWrite's lowered check, or the
effective-default fill each turns the new tests red.
Iteration 2 changes only that comment; go test
./internal/httpapi/wireshape/... (gms_pure_go), go vet, gofmt and git
diff --check pass again.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(httpclient): lift the HTTP client wire/encode layers from bd-enterprise (MIT contribution)

Contributed by gascity from bd-enterprise (github.com/gascity/bd-enterprise,
internal/enterprise/httpstore @ 49d1df2f6) to OSS beads under the MIT
license, with the owner's explicit approval. Every lifted file carries a
provenance header naming its source path.

Lifted into internal/httpclient:
- wire/    the HTTP transport, lazy handshake, project-identity stamping,
           problem-code decoding, retry behaviour and the capability vocabulary
- encode/  request/response encoders for the role operations
- credential*.go, ca*.go, target*.go   the bearer credential ladder,
           host-scoped CA trust and the http_target.json sidecar

Not lifted (stays enterprise-only): the hosted gateway (httpgw/gwauth),
box-login, storage profiles and gasworks.

Deferred to later slices (kept aside, not in this tree): the store transport
seam (WireClient/WireDialer/process-wide dialer, which goes to S3 with
dial.go), the enterprise-only BatchCloseRequest.claim_next encoder
(ClaimNextBody; the OSS server has no claim_next on batch close), and the
tests that need the store/dial layer.

This commit is the mechanical lift: import paths rewritten, package renamed,
two compile gaps removed. The production code builds. Known follow-up in the
next commit: reconcile the client's vocabularies and encoder tables with the
OSS spec (e.g. the OSS problem code notes_overwrite_refused, policy/sort
differences), restore the test helpers the lifted tests need, and wire in the
S0 wire_revision handling and S1 OpenOptions credentials.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(httpclient): compile the lifted client and reconcile with the OSS wire

The lift in ab11cb1f8 carried relative test paths and vocabulary tables
from bd-enterprise's internal/enterprise/httpstore (4 directories below
repo root) into internal/httpclient (3 below), and several tables
described enterprise-only server behavior OSS does not serve. Reconcile:

- Add testTarget test helper (written fresh) so internal/httpclient
  compiles and its tests pass.
- Fix four relative paths off by one "../" from the package relocation
  (skew_test.go, ledger_render_test.go, bijection_test.go x2).
- skew_test.go: importsHTTPClient now matches internal/httpclient instead
  of the retired internal/enterprise/httpstore path; the cmd/bd sweep no
  longer hard-fails when it legitimately finds zero matching files (cmd/bd
  does not import this client yet - that's S3's wiring).
- encode/table.go, encode/ledger_fields.go: removed a Reserved "reverse"
  entry and a refused ExcludeIDs field claimed for listIssues/ReadyRequest;
  OSS's listIssues never accepts "reverse" (confirmed against
  internal/httpapi/reads.go and routes.go's CapIssuesListSort, which gates
  `sort` only) and issueops.ReadyRequest carries no ExcludeIDs member in
  OSS. Each removal is documented in place.
- encode/table.go, encode/encode.go, encode/roundtrip_test.go: countIssues
  publishes has_metadata_key and include_ephemeral that CountRequest
  carries and the table/encoder/round-trip corpus all silently dropped;
  mapped and wired through.
- encode/ledger.go: ~70 ledger rows cited enterprise store-conformance and
  cmd/bd subprocess-parity tests (dual-run reader, claim races, flag-mode
  refusals) that don't exist in this package and belong to S3's store/dial
  seam and cmd/bd wiring. Re-pinned to a documented TODO(S3) sentinel
  (pinnedByS3Conformance) rather than deleted, per the ledger's own
  well-formedness discipline.
- wire/handshake.go, wire/vocabulary_test.go: added CapBatchApplyLarge
  (issues.batchApplyLarge) alongside the server's behaviorCapabilities.
- wire/problem.go: mapped the notes_overwrite_refused problem code to
  issueops.ErrNotesOverwrite.

internal/httpclient/... is fully green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(httpclient): wire-revision handshake gate and batch cap from capability

S0 integration (task #2): Client.Handshake now reads ContextResponse's
wire_revision/min_client_wire_revision and refuses with a typed
*WireRevisionSkewError before any post-handshake call when the server's wire
shape falls outside [ClientMinWireRevision, ClientWireRevision] -- mirroring
DESIGN.txt sec 4's "Client rule". A decoded 0 is treated as "the server
omitted the member" per the permanently-retired-values rule, never as a value
needing refusal. The client now also stamps Bd-Wire-Revision on every request
(client.go's stampRequest), and problem.go gained the per-request mirror:
*WireRevisionUnsupportedError, decoded from the reason:"wire_revision_unsupported"
extension members, for the case where the SERVER refuses this client's
declared revision as too old. Both error types are deliberately distinct --
one is this client declining after reading a 200, the other is the server
refusing a request outright -- and neither branches on bd_version (only
assigns it into error text), so TestNoFeatureBranchesOnBdVersionOrSchemaVersion
stays green.

Also added legacyRevisionFields/decodeRevisionToken to problem.go: a
precondition_failed refusal's expected_version/actual_version, typed *string
on the generated Problem, now tolerates a pre-#6053 server sending either as a
bare JSON integer rather than the decimal string every server since has used,
recovering the field the primary decode silently drops rather than losing it.

Batch cap (task #3): ApplyBatch reads the handshake snapshot on every call and
enforces 100 items unless the server advertises issues.batchApplyLarge, in
which case the ceiling is issueops.MaxApplyBatchItems (1000) -- never a bare
compiled constant. A plan over the applicable cap refuses before the dial: via
the existing *CapabilityError machinery when advertising the capability would
help (D7 case 2), or via a new *BatchTooLargeError (unwraps to
issueops.ErrValidation) when it would not. BatchTooLargeError lives in
handshake.go rather than beside ApplyBatch because every exported method in
writes.go is swept by TestEveryOperationMethodRoutesThroughTheSharedDispatch as
an operation-dispatch method.

All three internal/httpclient/... packages build and `go test` passes with
TMPDIR=/data/projects/.scratch/s2t BEADS_TEST_SKIP=dolt.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(httpclient): REQUIRE-mode credential resolution bridge to backends.Credential

Task #4: adds the knob internal/storage/backends/backends.go's OpenOptions.Credential
doc explicitly deferred ("S1 does not add a RequireCredential knob to enforce
the multi-tenant rule mechanically -- that is deferred to S2").

ProvidedCredential wraps an embedder-supplied wire.CredentialProvider (via its
httpclient alias) as a backends.Credential, so a multi-tenant embedder (for
example gc, Gas City: one process serving many workspaces) can carry a
per-workspace credential through backends.OpenOptions without the backends
package needing to know this package's CredentialProvider shape.

ResolveCredential(opts, base, require) is the bridge itself:
  - nil Credential, require=false: NewBearerProvider(base), the existing
    single-tenant ambient bearer ladder (unchanged).
  - nil Credential, require=true: ErrCredentialRequired -- the multi-tenant
    fail-closed case backends.go's doc names: a missing per-workspace
    credential must be a loud refusal, not a silent fall-through to
    process-global ambient env that cannot tell one tenant's workspace from
    another's.
  - a ProvidedCredential with a non-nil Provider: returned directly,
    regardless of require.
  - a ProvidedCredential with a nil Provider, or any other
    backends.Credential this package does not recognise: refused -- the
    former as ErrCredentialRequired (a present-but-empty wrapper must not
    look like "a credential was supplied"), the latter wrapping
    backends.ErrUnsupportedCredential per the OpenWith contract
    internal/storage/backends/open_with_test.go already pins
    (TestBackendOpenWithRefusesUnsupportedCredentialType).

No bd-enterprise source was copied for this file; it is new code written
against the OSS backends.go contract and the existing BearerProvider/
CredentialProvider types already lifted into this package. Live-mutated all
three guards (the require branch, the ProvidedCredential type assertion, and
the nil-Provider check) and confirmed each produces a RED test before
reverting.

go build/vet/gofmt clean; `go test ./internal/httpclient/...` passes with
TMPDIR=/data/projects/.scratch/s2t BEADS_TEST_SKIP=dolt.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): S2 task #6 wire-revision/legacy-decode coverage, fix legacyRevisionFields bug

Add httptest coverage for the two wire-revision mechanisms task #2 added
(commit 7ff68b29d8):

- handshake_wire_revision_test.go: the client-initiated skew gate
  (checkWireRevision/WireRevisionSkewError) and the per-request
  Bd-Wire-Revision header stamp. Live-mutated both switch arms in
  checkWireRevision and the stamped header value; confirmed RED, reverted.

- problem_wire_revision_test.go: the server-initiated
  wire_revision_unsupported refusal (reason-before-code dispatch priority)
  and the legacy-integer revision-token fallback.

Writing the legacy-decode tests surfaced a real bug in legacyRevisionFields:
its presence guard treated any non-nil *string field as "already decoded,"
but encoding/json's indirect() pre-allocates a pointer to "" for a *string
field before discovering a type mismatch (e.g. a bare JSON integer). A
type-mismatched expected_version/actual_version therefore arrived as a
non-nil pointer to the empty string, not nil -- exactly the one case the
fallback exists to recover, and exactly the case the old guard skipped.
Fixed by adding revisionNeedsFallback(p) (nil-or-empty), used in place of
the bare nil checks. No legitimate server response uses a literal
empty-string revision token, so this changes no other behavior.
Live-mutated the fixed guard back to the old nil-only checks; confirmed RED
on both recovery tests, reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): S2 task #6 ApplyBatch item-cap coverage

Add writes_batch_cap_test.go: httptest coverage for ApplyBatch's
capability-gated item ceiling (task #3), separate from the generic
two-speed "is issues.batchApply routed at all" gate writes_twospeed_test.go
already covers.

Cases: at-or-under the 100-item floor succeeds without the capability;
over the floor without issues.batchApplyLarge refuses with a
*CapabilityError before any dial past the handshake; the capability raises
the ceiling to issueops.MaxApplyBatchItems (1000), both at the new ceiling
and strictly between the old floor and it; over the absolute ceiling
refuses with *BatchTooLargeError regardless of the capability, also
without dialing the batch-apply endpoint.

Live-mutated all three guards in ApplyBatch (the limit computation, the
n > limit check, and the capability-vs-absolute-ceiling branch); confirmed
each produces a distinct RED, reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): S2 task #6 live TLS coverage for wrong-CA refusal

Add ca_wrong_ca_test.go: an end-to-end httptest TLS dial proving
TransportFor's configured CA pool is actually enforced at the wire, not
just recorded on the transport struct. ca_test.go already proves the
latter (TestTransportForReplacesRootPoolNotExtends, via
x509.CertPool.Equal) and proves the RIGHT CA dials successfully
(TestTransportForFileRotationClosesSupersededIdleConnections's live GETs);
neither exercises a server whose leaf a DIFFERENT CA signed.

TestTransportForWrongCARefusesTheTLSHandshake dials a server through a
transport built from an unrelated CA file and asserts the handshake fails
with x509.UnknownAuthorityError. TestTransportForRightCASucceedsOverTLS is
the positive control, using the same (newly-exercised) ca.startServer
helper that was defined but had no caller before this test -- without it,
a mutation that drops RootCAs entirely (falling back to the system pool)
would still pass the wrong-CA test for the wrong reason; the positive
control catches that case. Live-mutated transportForFile to drop RootCAs;
confirmed the positive control goes RED, reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): gazelle BUILD files for the lifted httpclient package tree

Task #7: bazel run //:gazelle, now that internal/httpclient and its
encode/wire subpackages exist in the OSS tree (ab11cb1f8) and have gained
this S2 slice's new files. No existing BUILD.bazel files changed; these
three packages previously had none.

Note for S3: bazel test //internal/httpclient/... fails on tests that read
sibling source trees by relative path at runtime
(TestLedgerRowsAreWellFormed, TestEveryEncoderTableParameterIsPublished,
TestDivergenceLedgerDocMatchesLedger in encode, and
TestNoFeatureBranchesOnBdVersionOrSchemaVersion,
TestEveryOperationMethodRoutesThroughTheSharedDispatch,
TestEveryOperationMethodIsCoveredByTheTwoSpeedTable in wire) -- paths like
../../../cmd/bd and ../../httpapi/spec/openapi.v0.yaml resolve under `go
test`'s working directory but not Bazel's sandbox, which gazelle cannot
infer `data` attributes for. Pre-existing in code from task #1/#2, not
introduced by this commit; `go test ./internal/httpclient/...`,
`go vet ./...`, `gofmt -l .`, `golangci-lint run ./internal/httpclient/...`,
and `make api-check` are all clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpclient/wire): tolerate bare-integer revision on success-path responses

Review HIGH #2: against a fake pre-#6053 server (one whose ContextResponse
omits wire_revision), CloseIssue and its revision-bearing siblings
(ApplyBatch, ReleaseIssue, ReopenIssue, UpdateIssue) failed with an untyped
`json: cannot unmarshal number ... revision of type string`. S0's
string-or-number tolerance (problem.go's legacyRevisionFields) only covered
the problem+json error path, never the success path.

Adds revision_tolerance.go: a pre-unmarshal rewrite that walks a response
body and turns a bare JSON number at a "revision" key into its exact decimal
string, scoped to the five apigen response types that carry the member and
gated on the handshake having seen no wire_revision at all. Left
apigen/types.gen.go and the OpenAPI spec untouched (both are
regenerate-and-diff gated by `make api-check`), rather than introducing a
custom Revision JSON type through codegen, to keep the fix's blast radius to
the wire package's own reader.

Updates ClientMinWireRevision's doc comment (handshake.go) to describe
tolerance on both halves of the wire instead of overclaiming the error-path
fix alone covered it. Adds revision_tolerance_test.go: a read (ListReadyWork)
and a guarded write (CloseIssue with ExpectedVersion) against a fake
pre-#6053 server, plus a modern-server control proving the rewrite never
fires when wire_revision is present. Mutation-tested: disabling the rewrite
reproduces the exact untyped UnmarshalTypeError the review reported.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): sandbox internal/httpclient's source-reading tests (Bazel gate green)

Review HIGH #1: the Bazel lane was red because three tests under
internal/httpclient read sibling/cross-package files by relative OS path at
runtime, which bazel test's sandbox does not materialize without an explicit
`data` dependency.

- internal/httpclient/encode/BUILD.bazel: encode_test gets
  data = ["//cmd/bd:go_srcs", "//internal/httpapi/spec:openapi.v0.yaml",
  "//:engdocs/design/http-divergence-ledger.md"] + glob(["*.go"]), covering
  bijection_test.go's cmd/bd scan and ledger_render_test.go's golden-doc read.
- internal/httpclient/wire/BUILD.bazel: wire_test gets
  data = ["//cmd/bd:go_srcs", "//internal/httpclient:go_srcs"] + glob(["*.go"]),
  covering skew_test.go's AST sweep of both trees.
- internal/httpclient/BUILD.bazel gains a go_srcs filegroup via `make
  bazel-sync` (tools/bazel/go_srcs.py): added "internal/httpclient" to the
  script's managed PACKAGES list rather than hand-writing the block.
- internal/httpapi/spec/BUILD.bazel and the root BUILD.bazel gain
  exports_files for openapi.v0.yaml and engdocs/design/http-divergence-ledger.md,
  scoped to //internal/httpclient/encode:__pkg__.
- bijection_test.go's loadPublishedParameters now reads the wire contract
  through spec.OpenAPIV0() (the package's existing go:embed) instead of a
  relative os.ReadFile path, per review suggestion -- the embed makes the
  parameter set reachable without a `data` dependency on the YAML at all; the
  openapi.v0.yaml data entry above stays as defense in depth for any other
  reader the package gains.

Ran via `bazel run //:gazelle` then `python3 tools/bazel/go_srcs.py`
(bazel-sync-check passes). `bazel test //internal/httpclient/...` is fully
green: 3/3 test targets pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(httpclient): scrub private/enterprise infra references from code and docs

Review MEDIUM #3. Provenance headers ("Contributed by gascity from
bd-enterprise...") stay by design -- this removes references to private
infrastructure and unported internals from comment bodies and generated docs:

- wire/doc.go: dropped the internal/enterprise/hostedbeads and
  internal/enterprise/storageprofile build-tag comparison (neither exists in
  this OSS tree) and the dangling citation of the never-written
  engdocs/design/http-client-backend.md, replacing both with an accurate
  description of this package's own build-tag-free status and a pointer to
  the divergence ledger that does exist.
- credential.go: CredentialProvider's doc no longer names gwauth/DPoP/gateway
  auto-detection by their private spelling; describes the interface generically.
- ca.go: dropped "cherry-to-AWS RTT" (generalized to "a typical cross-region
  RTT") and the cmd/bd/enterprise_connect.go file reference (generalized to
  "a `bd connect --ca-file X`-style command").
- ca_permissions_unix.go / ca_permissions_unix_test.go: ~/.config/gascity and
  /etc/gascity example paths replaced with ~/.config/bd and /etc/bd.
- encode/skew_test.go: beads.corp replaced with bd.example.invalid.
- encode/ledger_render.go + ledger_render_test.go: the generated-doc preamble
  and regenerate-hint strings now name internal/httpclient/encode (this
  package) instead of internal/enterprise/httpstore/encode; regenerated
  engdocs/design/http-divergence-ledger.md via `go test
  ./internal/httpclient/encode -run TestDivergenceLedgerDocMatchesLedger
  -update-ledger-doc` rather than hand-editing it.

Final grep for enterprise/cherry/gascity/corp/gwauth under internal/httpclient
and engdocs turns up only: provenance headers (kept deliberately), this
session's own "no bd-enterprise source copied" disclaimers, and a handful of
generic "enterprise-tagged"/"enterprise server" references describing a
build-tag or commercial-tier server concept in the abstract -- none name a
private host, path, or credential.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): review LOW items -- S3 TODO, env-driven CA test, lockstep doc

Review LOW #4-#6:

- wire/skew_test.go: TODO(S3) on the branch where the cmd/bd AST sweep is
  vacuous (no store/dial wiring yet), naming the slice that closes it.
- ca_wrong_ca_test.go: TestTransportForWrongCAViaEnvRefusesTheTLSHandshake,
  the BEADS_HTTP_CA_FILE env-driven sibling of the existing direct-CAFile
  wrong-CA test -- same live-TLS refusal, resolved through resolveCAFile's
  host-scoped env syntax instead of Target.CAFile.
- vocabulary_test.go: expanded the ClientWireRevision/CurrentWireRevision
  comment to explain why this repo holds them in strict lockstep (client and
  server ship from the same commit) rather than merely <=, distinguishing it
  from the real cross-version compatibility ClientMinWireRevision exists for.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpclient/wire): re-review HIGH-a/b -- cover IssueDetails, stop recursing into metadata

HIGH-a: the legacy-revision set was missing apigen.IssueDetails (GET
/v0/beads/issues/{id}'s body, an alias for types.IssueDetails that a grep for
"type ... struct" never sees). getIssue is a baseline op that can dispatch
with no cached handshake at all, so serverPredatesRevisionStrings now treats
"no snapshot" the same as "WireRevision == 0" instead of only the latter.
revisionBearingResponse's set is now pinned exhaustively against
internal/httpapi/wireshape/testdata/golden.json -- the same committed digest
TestWireShapeDigest guards -- by a new test
(TestRevisionBearingResponseCoversEveryWireShapeSchemaWithARevisionMember), so
a future schema gaining a `revision` member can't silently miss coverage
again. New coverage: both "with" and "without" a prior handshake for getIssue,
and an explicit items[].revision test for ApplyBatchResponse.

HIGH-b: the previous fix recursed into every nested object looking for a
bare-number "revision" key, which reached caller-owned blobs like
CloseIssueResponse.issue.metadata -- real user data shaped like
{"revision":3} came back silently coerced to a string, on every server the
broadened gate above now also covers. tolerateLegacyRevisionNumbers no longer
recurses at all: it rewrites only the body's own top-level "revision", and,
for *apigen.ApplyBatchResponse specifically, each item's top-level
"revision" in the top-level "items" array. New
TestCloseIssueLegacyRewriteNeverTouchesIssueMetadata proves a metadata
document with "revision" keys at three nesting depths round-trips
byte-for-byte unchanged.

Mutation-tested both guards by hand (not committed): dropping the
IssueDetails case turns TestRevisionBearingResponseCoversEveryWireShapeSchemaWithARevisionMember
and both getIssue tests red with the exact missing-coverage/decode-failure
messages; reinstating a fully recursive rewrite turns
TestCloseIssueLegacyRewriteNeverTouchesIssueMetadata red, showing the
metadata's revision keys silently rewritten at all three depths. Both
reverted clean before this commit.

internal/httpapi/wireshape/BUILD.bazel and internal/httpclient/wire/BUILD.bazel
export/declare testdata/golden.json as data for wire_test's new exhaustiveness
test.

Verified: go test ./internal/httpclient/... (ok, one pre-existing unrelated
flake in ca_wrong_ca_test.go confirmed not reproducible across 8 reruns),
golangci-lint run ./internal/httpclient/... (0 issues), make api-check (no
apigen drift), bazel test //internal/httpclient/... //internal/httpapi/wireshape/...
(5/5 PASSED).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpclient): stop the wrong-CA TLS tests racing Happy Eyeballs

Root cause: startServer rewrites its httptest TLS server's URL to the
hostname "localhost" so SNI is actually sent, but the server itself only
binds 127.0.0.1. net/http's default dialer does RFC 6555 Happy Eyeballs
for a hostname with more than one address family, racing a tcp6 dial to
::1:<port> against the real tcp4 dial. On this host, many unrelated
processes (JVM tooling) are bound to high ::1 ports, so the tcp6 race
occasionally "won" against a listener that had nothing to do with the
test, and the client spoke TLS to it instead -- surfacing as "tls: first
record does not look like a TLS handshake" rather than the expected
x509.UnknownAuthorityError. Confirmed via ss -tlnp captured at the moment
of failure (a java process holding the exact port raced) and via an
immediate manual redial to 127.0.0.1 on that port succeeding against the
real, still-live test server.

Fix: forceIPv4Loopback (new shared helper in ca_test.go) rewrites the
returned *http.Transport's DialContext to dial tcp4 only. SNI and the
request Host are unaffected, since crypto/tls and net/http derive both
from the URL, never from which address family carried the bytes. Wired
into every TLS fixture test in the package (ca_wrong_ca_test.go's three
tests, and ca_test.go's rotation test) -- grepped the rest of the repo
for the same "rewrite to localhost for SNI" pattern and found no other
occurrences.

Also fixed a latent bug this surfaced: TestTransportForWrongCAViaEnvRefusesTheTLSHandshake
called TransportFor(Target{}) with a nil BaseURL, so caHostMatches always
returned false and the CAFileEnv path was never actually exercised --
the nil RoundTripper was silently masked by http.Client's fallback to
http.DefaultTransport, which also failed to verify (via system roots)
for an unrelated reason. Now passes Target{BaseURL: u} so the test
exercises the host-scoped env-resolution path it documents.

go test -race -count=50 -run WrongCA: 8/8 clean. Whole package
go test -race -count=20: clean, twice. bazel test, golangci-lint,
make api-check: all green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpclient): repair CI-red tests and apply review fixes (adopt-pr #7210)

Three defects on the contributor head, none of them present on the base:

- test/docsync's TestEngdocsAndRootMarkdownLinks fails: the generated
  divergence-ledger doc links http-client-backend.md, an architecture
  spec this repository does not carry. Name it in a code span instead,
  in ledgerDocPreamble, and regenerate the doc (-update-ledger-doc).

- TestCAFileHygieneRefusesGroupWritableFile fails under umask 022:
  os.WriteFile's 0o660 passes through the umask, so the file lands 0640,
  is never group-writable, and TransportForFile accepts it. Chmod it
  explicitly. Its acceptance twin had the same gap on its file (only the
  explicitly chmodded directory reached the user-private-group
  exception), so chmod that file too: a mutant that drops the exception
  for files alone passed the old test under umask 022 and fails the new
  one.

- That acceptance twin calls the unix-only isUserPrivateGroup from
  ca_test.go, which carries no build constraint, so the package's tests
  do not compile for GOOS=windows. Move it, otherwise verbatim, into
  ca_permissions_unix_test.go beside the other isUserPrivateGroup tests.

Review fixes (iteration 1):

- Encode dispatched through nine case arms of op/shape conjunctions
  (cyclomatic 24). Switch on an (op, shape) struct key instead, and move
  the getIssue and search-bridge arms into encodeGetIssue and
  encodeSearch. The arms are disjoint, so dispatch and every error string
  are unchanged. Cyclomatic 24 -> 10.

- EventStream.Next kept four accumulators and the per-field switch in
  one loop (cognitive 27). Gather the accumulators into sseEvent and
  fold each field line, per-event cap included, in applyField. Next
  cognitive 27 -> 12; applyField is 11/11.

- The CA hygiene walk joined a relative symlink target onto the textual
  containing directory, so with a/link -> ../b/c and b/c/x -> ../y it
  resolved a/y where the kernel opens b/y: a valid CA was refused, or a
  hygienic file at a/y was checked and read instead. readLinkTarget joins
  onto the directory's EvalSymlinks path, taken after that directory's
  own ancestors passed the hop-by-hop check. The regression test plants
  a decoy at a/y; a lexical-join mutant reads it and fails.

- SaveTarget wrote a relative CAFile that every later LoadTarget
  refuses. Both now go through checkCAFileAbsolute, so SaveTarget refuses
  first, with LoadTarget's own message, and writes nothing.

- Document the cached transport TransportFor and TransportForFile return
  as shared and read-only.

Review fixes (iteration 2):

- The walk still dropped a ".." inside a symlink target textually:
  filepath.Join cleans a relative target, and an absolute target's
  ancestors come from filepath.Dir, while the kernel takes that ".."
  from wherever the component before it really leads. Both reviews
  rated the relative leaf as a wrong-file read, and it is one: with
  x/d -> ../e/s, x/L -> d/../y read a decoy at x/y where the kernel
  opens e/y. The other three shapes are hygiene bypasses: an absolute
  leaf target x/d/../y, and an ancestor link to d/../dir or x/d/../dir
  with the CA at x/L/y, each read a file in a world-writable e that the
  walk never checked. readLinkTarget now refuses a target with a ".."
  after any other component, relative or absolute; a leading run of
  ".." is still taken from the directory's physical path. The test pins
  all four shapes behind a control that refuses e/y named directly, and
  a table pins hasInteriorDotDot's boundary. Mutants that drop the
  refusal, limit it to relative targets, refuse every "..", or count
  "." as a component each fail.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <adopt-pr@gascity.com>
2026-10-05 02:58:34 -07:00

797 lines
36 KiB
Go

// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/ca.go@49d1df2f6)
// to OSS beads under the MIT license.
package httpclient
import (
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/pem"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"golang.org/x/net/http/httpproxy"
"golang.org/x/net/idna"
)
// CAFileEnv names a PEM file that becomes the ENTIRE trusted root pool for the
// ONE target it names — not an addition to the system store, and not applied
// to any other target this process happens to dial.
//
// It exists for a private CA with no name constraints (the Gas City Beads
// Serve CA): a CA like that can mint a certificate for ANY hostname, so
// trusting it system-wide, or via SSL_CERT_FILE, would let it vouch for every
// TLS connection the process makes, not just the one bd serve it was issued
// for. TransportFor instead builds a pool that contains ONLY this CA and scopes
// it to one target's *http.Client.
//
// # Syntax is host-scoped: host[:port]=path
//
// The value is "host[:port]=path", e.g. "serve.example.com=/etc/bd/ca.pem" or
// "serve.example.com:8443=/etc/bd/ca.pem". A pattern with no port matches a
// target's host regardless of its own port; a pattern with a port matches only
// that exact host:port. A malformed value (no "=", or nothing on either side
// of it) is refused outright — including the old bare-path form this
// superseded, which applied to every target the process dialed regardless of
// host, the impersonation risk this syntax exists to close. A well-formed
// value whose host does not match a given target is silently not applied to
// that target: it falls through to the sidecar (or system roots), never to
// the env's CA.
//
// # Precedence: env beats sidecar, but never silently
//
// When the env var's host pattern matches a target, it wins over that
// target's http_target.json "ca_file" field (Target.CAFile, written by
// `bd connect --ca-file <path>`) — the same rung order the bearer ladder
// already uses (BEADS_HTTP_TOKEN beats the credentials file, see
// credential.go): the environment is the operator's per-invocation override,
// and the sidecar is per-user state persisted once at connect time. But
// unlike the bearer ladder, an env value that matches a target AND disagrees
// with that target's own sidecar ca_file is a hard refusal, not a silent
// override: the two rungs naming different CAs for the same target is very
// likely a stale sidecar or a misdirected env var, and picking one silently
// would trust whichever file the operator did NOT intend.
//
// # The embedder door does not read this at all
//
// This env var is the CLI's own per-invocation override and is consulted only
// by the automatic dial path (DialWith, when the caller leaves
// DialOptions.HTTPClient nil) — the path `bd connect`, `bd ready`, and every
// other cmd/bd invocation rides. backend/http's public TransportFor (the
// embedder door gc calls explicitly to compose a transport into its own
// *http.Client) never reads it: an embedder juggling many targets in one
// process must not have any of them silently renarrowed by an ambient
// variable a completely unrelated invocation exported, so it always builds
// from Target.CAFile alone. See TransportForFile.
//
// #nosec G101 -- the NAME of an environment variable, not a credential.
const CAFileEnv = "BEADS_HTTP_CA_FILE"
// maxIdleConnsPerHost raises TransportFor's per-host idle pool past
// http.DefaultTransport's default of 2. A CA-scoped target is, by
// construction, one host — that's the whole shape this feature exists for —
// so a concurrency-8 caller (gc's ready-veto fan-out) hammering it legitimately
// wants more than 2 warm connections; below that ceiling, every request past
// the second pays a fresh TLS handshake instead of reusing one already paid
// for. See TransportFor's "Idle connection reuse" doc. It applies equally to
// every http-backend target, CA-configured or not (see baselineTransport in
// dial.go): the RTT cost of an under-pooled host is the same either way.
const maxIdleConnsPerHost = 16
// maxCAFileBytes caps how large a configured CA file may be. A CA bundle is
// operator-owned configuration, not arbitrary input, and legitimate ones are a
// handful of certificates — a few KiB. A file past this ceiling is refused
// rather than parsed: it is almost certainly the wrong file (a full trust
// store, a log, something else entirely pointed at by mistake), and PEM
// decoding an unbounded file is needless work for something that is never a
// deliberate CA bundle.
const maxCAFileBytes = 1 << 20 // 1 MiB
// resolvedCA is what resolveCAFile found: the file to trust for a target, and
// which knob it came from, so a refusal names the thing to fix rather than an
// unqualified "ca file".
type resolvedCA struct {
path string
label string
}
// configured reports whether a CA applies at all; the zero value means
// "system roots", the same signal TransportFor's nil, nil return carries.
func (r resolvedCA) configured() bool { return r.path != "" }
// parseCAFileEnv parses CAFileEnv's host-scoped syntax, "host[:port]=path".
// Anything else — no "=", an empty host, an empty path, or a path that is
// not absolute — is malformed and refused: there is no reasonable default
// scope for a value this package cannot parse, and falling back to "applies
// everywhere" is exactly the vulnerability this syntax replaces. The split
// is on the FIRST "=", not the last: a path may itself contain "=" (rare,
// but legal in a POSIX filename), and splitting on the last one would silently
// mis-parse the host pattern in that case rather than refusing loudly.
// Requiring the path be absolute closes a smaller but real ambiguity: a
// relative path's meaning would depend on the process's current directory at
// resolution time, which the sidecar and --ca-file paths never do.
func parseCAFileEnv(raw string) (hostPattern, path string, err error) {
eq := strings.Index(raw, "=")
if eq <= 0 || eq == len(raw)-1 {
return "", "", fmt.Errorf(
"%s %q: want host[:port]=path (e.g. serve.example.com=/etc/bd/ca.pem, or serve.example.com:8443=/etc/bd/ca.pem); a bare path is no longer accepted because it applied to every target this process dials",
CAFileEnv, raw)
}
hostPattern = strings.TrimSpace(raw[:eq])
path = strings.TrimSpace(raw[eq+1:])
if hostPattern == "" || path == "" {
return "", "", fmt.Errorf("%s %q: want host[:port]=path", CAFileEnv, raw)
}
if !filepath.IsAbs(path) {
return "", "", fmt.Errorf("%s %q: path %q must be absolute", CAFileEnv, raw, path)
}
if strings.HasSuffix(hostPattern, ":") {
// "h.example:" (or "[::1]:") names a colon with nothing after it — a
// stray trailing colon, almost certainly a dropped port rather than
// a deliberate pattern, and NEITHER of the two meanings a colon can
// carry here (no colon at all: any port; host:port: exactly one):
// silently picking either would be a guess, not a parse.
return "", "", fmt.Errorf("%s %q: host pattern %q ends with \":\" and no port; want host, or host:port", CAFileEnv, raw, hostPattern)
}
return hostPattern, path, nil
}
// caHostMatches reports whether pattern (the host half of a parsed
// CAFileEnv value) names target. Both sides are IDNA/punycode-normalized to
// ASCII (so "café.example" and "xn--caf-dma.example" agree, whichever form
// an operator typed) and lowercased, with a trailing dot stripped, before
// comparison, and ports are normalized against target's own URL scheme
// (https defaults to 443, http to 80) before being compared, so "host" and
// "host:443" agree for an https target exactly as they would if the
// target's URL had spelled the port out. A pattern with no port matches
// target's host regardless of target's own port — so "example.com=..."
// applies whether the target dials :443 or :8443 — but a pattern WITH a
// port matches only that exact, normalized host:port. IPv6 literals ("[::1]"
// or "[::1]:8443") are handled the same way net/url and net.SplitHostPort
// do. A host that fails IDNA normalization on either side is reported as an
// error rather than silently compared as raw text: an unnormalizable
// pattern could otherwise disagree with a target that a browser or the
// server's own cert treats as the same name.
func caHostMatches(pattern string, target Target) (bool, error) {
pattern = strings.TrimSpace(pattern)
if pattern == "" || target.BaseURL == nil {
return false, nil
}
patternHost, patternPort, err := splitCAHostPort(pattern, "")
if err != nil {
return false, fmt.Errorf("host pattern %q: %w", pattern, err)
}
targetHost, targetPort, err := splitCAHostPort(target.BaseURL.Host, target.BaseURL.Scheme)
if err != nil {
return false, fmt.Errorf("target host %q: %w", target.BaseURL.Host, err)
}
if patternHost == "" || targetHost == "" || patternHost != targetHost {
return false, nil
}
if patternPort == "" {
return true, nil
}
return patternPort == targetPort, nil
}
// caIDNAProfile normalizes a hostname label to lowercase ASCII/punycode
// without the stricter registration-validity checks idna.Lookup would apply
// (rejecting labels a real, already-issued certificate's SAN can legally
// contain, e.g. a bare underscore label, would turn a working pattern into a
// refusal for no security reason) — this comparison only needs both sides
// folded to the SAME ASCII form, not a verdict on whether the name could be
// newly registered.
var caIDNAProfile = idna.New(idna.MapForLookup(), idna.Transitional(true))
// splitCAHostPort splits hostport into an IDNA/punycode-normalized,
// lowercased, trailing-dot-stripped host and its port, handling a bracketed
// IPv6 literal with or without a port. When hostport carries no port and
// scheme names a well-known one (http, https), the returned port defaults to
// that scheme's own — the normalization caHostMatches uses so an explicit
// ":443" pattern still matches an https target whose URL never spells the
// default port out. An IPv6 literal (bracketed, or one that fails ASCII
// hostname normalization for any other reason) is returned lowercased but
// otherwise unnormalized: idna.ToASCII on a literal address is a no-op or an
// error depending on form, and this package already compares IPv6 literals
// as plain text.
func splitCAHostPort(hostport, scheme string) (host, port string, err error) {
hostport = strings.TrimSpace(hostport)
if hostport == "" {
return "", "", nil
}
isIPv6Literal := false
if h, p, splitErr := net.SplitHostPort(hostport); splitErr == nil {
host, port = h, p
} else if strings.HasPrefix(hostport, "[") && strings.HasSuffix(hostport, "]") {
host = strings.TrimSuffix(strings.TrimPrefix(hostport, "["), "]")
} else {
host = hostport
}
host = strings.TrimSuffix(host, ".")
if strings.HasPrefix(host, "[") && strings.HasSuffix(host, "]") {
host = strings.TrimSuffix(strings.TrimPrefix(host, "["), "]")
}
if net.ParseIP(host) != nil {
isIPv6Literal = true
}
if isIPv6Literal {
host = strings.ToLower(host)
} else {
ascii, asciiErr := caIDNAProfile.ToASCII(host)
if asciiErr != nil {
return "", "", fmt.Errorf("host %q is not valid ASCII or a valid internationalized hostname: %w", host, asciiErr)
}
host = strings.ToLower(ascii)
}
if port == "" {
switch strings.ToLower(scheme) {
case "https":
port = "443"
case "http":
port = "80"
}
}
return host, port, nil
}
// sameCAFile reports whether a and b name the same file: identical after
// filepath.Abs+Clean, or — when both exist — the same file by os.SameFile
// (a hardlink or bind-mount alias). This replaces a raw string compare in
// resolveCAFile's env-vs-sidecar disagreement check, which would otherwise
// refuse two paths that plainly name the same file (one relative, one
// absolute; one with a trailing "/./"; a symlink alias) purely because their
// text differs.
func sameCAFile(a, b string) bool {
a = strings.TrimSpace(a)
b = strings.TrimSpace(b)
if a == "" || b == "" {
return a == b
}
absA, errA := filepath.Abs(a)
absB, errB := filepath.Abs(b)
if errA != nil || errB != nil {
return a == b
}
absA, absB = filepath.Clean(absA), filepath.Clean(absB)
if absA == absB {
return true
}
infoA, errA2 := os.Stat(absA)
infoB, errB2 := os.Stat(absB)
if errA2 == nil && errB2 == nil {
return os.SameFile(infoA, infoB)
}
return false
}
// resolveCAFile applies the env-over-sidecar precedence documented on
// CAFileEnv, now host-scoped: a "" path in the result means system roots
// apply, unchanged from today, for a target that matches no configured CA.
func resolveCAFile(target Target) (resolvedCA, error) {
sidecar := strings.TrimSpace(target.CAFile)
raw := strings.TrimSpace(os.Getenv(CAFileEnv))
if raw == "" {
if sidecar == "" {
return resolvedCA{}, nil
}
return resolvedCA{path: sidecar, label: "ca_file"}, nil
}
pattern, envPath, err := parseCAFileEnv(raw)
if err != nil {
return resolvedCA{}, err
}
matches, err := caHostMatches(pattern, target)
if err != nil {
return resolvedCA{}, fmt.Errorf("%s %q: %w", CAFileEnv, raw, err)
}
if !matches {
if sidecar == "" {
return resolvedCA{}, nil
}
return resolvedCA{path: sidecar, label: "ca_file"}, nil
}
if sidecar != "" && !sameCAFile(sidecar, envPath) {
return resolvedCA{}, fmt.Errorf(
"%s (%s=...) names %q for %s, but its sidecar ca_file names %q for the same target; remove one so the CA used here is unambiguous",
CAFileEnv, pattern, envPath, target, sidecar)
}
return resolvedCA{path: envPath, label: CAFileEnv}, nil
}
// TransportFor builds the http.RoundTripper that trusts ONLY the CA named by
// target's resolved, host-scoped setting (CAFileEnv when its host pattern
// matches target, else Target.CAFile), REPLACING rather than extending the
// system root pool. It returns nil, nil when neither is configured — the
// caller's signal to keep using the default, system-roots transport, which is
// what preserves today's behavior for loopback dev and every target that has
// never set a CA.
//
// This is exported so a caller that builds its own *http.Client for other
// reasons — a tuned timeout, a proxy — can still pick up the automatic,
// env-aware resolution DialWith itself uses: compose the returned
// RoundTripper into that client's Transport field. DialWith calls this itself
// when the caller left DialOptions.HTTPClient nil, so cmd/bd's own dial path
// (the default dialer, the gateway dialer, and `bd connect`'s own verifying
// handshake) needs no further wiring.
//
// An embedder outside this module that wants a transport from an EXPLICIT
// file, ignoring CAFileEnv entirely, should call TransportForFile(target.CAFile)
// instead — see that function and backend/http's public door.
//
// The returned transport never sets ServerName: TLS's SNI and the Host header
// both continue to come from the request's URL, exactly as they do for the
// default transport, so this only narrows which root certificates are
// trusted — it never changes which host a request addresses.
//
// Errors refuse loudly rather than fall back to system roots: a missing file,
// an unreadable one, a malformed CAFileEnv value, a hygiene violation (file
// permissions, size), or one with no valid PEM-encoded certificate is a hard
// error, never a silent downgrade to public trust or to no trust check at
// all.
//
// # Idle connection reuse
//
// A CA-scoped target is still a single host, so it gets the same
// keep-alive budget raised elsewhere for this reason: http.DefaultTransport's
// MaxIdleConnsPerHost is 2, which is too small for a concurrency-8 burst
// against one target — anything past the first 2 idle connections pays a
// fresh TLS handshake, tens of milliseconds at a typical cross-region RTT.
// maxIdleConnsPerHost raises that ceiling so a warm burst reuses connections
// instead of re-handshaking per request.
//
// # Caching and root rotation
//
// The transport returned for a given absolute path is cached and reused for
// the life of the process, keyed by (absolute path, file content hash), so a
// long-lived process dialing the same target repeatedly reuses one transport
// and its warm connection pool rather than paying a fresh TLS handshake and
// leaking the previous transport's idle connections on every dial. A CHANGED
// file at the same path — a different content hash — produces a fresh
// transport, and the previous one's idle connections are closed. Ship a root
// rotation as a bundle containing BOTH the old and the new CA (a
// multi-certificate PEM, one CERTIFICATE block per certificate): every
// process, however long-lived, keeps verifying through the rotation, and
// converges on trusting only the new CA once the bundle is trimmed back down
// to it — with no window where some processes trust only the old certificate
// and others only the new one.
//
// Because it is cached, the returned transport is SHARED: every caller in
// this process asking for the same file holds the same *http.Transport. Treat
// it as read-only — wrap it rather than type-asserting and reconfiguring it —
// since setting its TLSClientConfig, Proxy or DialContext would silently
// change trust for every other holder.
//
// # HTTPS proxies
//
// When a CA is configured, the returned transport refuses to reach a target
// through an https:// proxy — see caAwareProxy. A plain http:// proxy is
// unaffected: the destination's TLS is still verified against the configured
// CA end to end through it.
func TransportFor(target Target) (http.RoundTripper, error) {
resolved, err := resolveCAFile(target)
if err != nil {
return nil, err
}
if !resolved.configured() {
return nil, nil
}
return transportForFile(resolved.path, resolved.label)
}
// TransportForFile builds the http.RoundTripper that trusts ONLY the CA at
// path, REPLACING rather than extending the system root pool — the same kind
// of transport TransportFor builds, but for a path the caller already
// resolved itself, with NO CAFileEnv involvement at all.
//
// It exists for two callers that must not go through env resolution:
//
// - A `bd connect --ca-file X`-style command verifies the server with X
// BEFORE writing anything; if CAFileEnv happened to be set to a
// different, valid CA for the same host, resolving through the normal
// env-aware path would verify against the WRONG file relative to what
// --ca-file names, then write X unchecked.
// - backend/http's public TransportFor (the embedder door) builds from
// Target.CAFile alone, because CAFileEnv is this process's ambient,
// per-invocation override for the CLI's own dial path, and an embedder
// juggling several targets in one process must not have any of them
// silently renarrowed by a variable it may not even know exists.
//
// Like TransportFor, this participates in the (absolute path, content hash)
// transport cache — so what it returns is the same shared, read-only
// transport — and refuses loudly (missing file, hygiene, malformed PEM)
// rather than falling back to system roots.
func TransportForFile(path string) (http.RoundTripper, error) {
return transportForFile(path, "ca_file")
}
// cachedTransport pairs a built transport with the content hash of the file
// it was built from and the sequence number of the read that produced it, so
// a later call can tell a cache hit from a rotation, and a stale concurrent
// read from a genuine one.
//
// seq, not the file's mtime, orders reads: a rotation landed by cp -p,
// rsync -t, or tar can preserve (or coincidentally produce) the SAME mtime
// as the file it replaces, down to whatever resolution the filesystem
// keeps — a same-second write is common, not exotic. Comparing mtimes then
// treats a genuine rotation as "no newer than what's cached" and keeps
// serving the superseded (possibly revoked) transport forever, since the
// file's mtime never again exceeds the stale cached one. seq instead counts
// which read of the file started first, a purely in-process fact
// transportReadSeq's own monotonic counter can always order correctly,
// whatever the filesystem clock says.
type cachedTransport struct {
hash [32]byte
seq uint64
transport *http.Transport
}
// maxCachedTransports bounds the process-wide transport cache. Each entry
// pins one *http.Transport and its idle connection pool for the life of the
// process; an unbounded cache keyed by absolute path would let a caller that
// churns through many distinct CA file paths over a long-lived process's
// life (many short-lived tenants, an ephemeral-workspace-heavy CI runner)
// grow it without limit. 64 comfortably covers every legitimate shape this
// package expects — one process rarely dials more than a handful of distinct
// CA-scoped hosts.
const maxCachedTransports = 64
var (
transportCacheMu sync.Mutex
transportCache = map[string]*cachedTransport{}
// transportCacheOrder tracks least-recently-used first; touched under
// transportCacheMu alongside transportCache itself.
transportCacheOrder []string
// transportReadSeq is the monotonic counter behind cachedTransport.seq,
// incremented under transportCacheMu immediately BEFORE the file read it
// tags begins — never after, and never derived from the file's own
// mtime; see cachedTransport's doc.
transportReadSeq uint64
)
// transportRecord is what transportProvenance remembers about one
// *http.Transport this package built: the absolute path and content hash it
// was built from, at build time.
type transportRecord struct {
path string
hash [32]byte
}
// transportProvenance maps a *http.Transport this package built to the
// transportRecord it was built from, keyed by the transport's own pointer
// identity. Unlike transportCache (bounded, LRU-evicted, keyed by path),
// this is NEVER evicted and never keyed by path: a caller that obtained a
// transport from TransportFor/TransportForFile and held onto it is entitled
// to keep using it for as long as the file it names is unchanged, even
// after churn through many other CA paths has pushed its cache entry (or a
// later rotation's entry for the same path) out of transportCache entirely.
// matchesCachedTransport consults this, then re-reads and re-hashes the
// file's CURRENT content, so eviction alone never causes a false refusal
// and a rotation is never silently trusted just because the pointer once
// belonged here.
//
// This does grow by one entry per transport ever built, for the life of the
// process — an intentional, deliberately unbounded trade against the
// alternative (a bounded/evicted provenance table would reintroduce exactly
// the false-refusal-on-eviction bug this exists to close). A process that
// churns through many thousands of distinct CA files over its lifetime
// leaks a few dozen bytes per rotation; there is no plausible deployment of
// this feature where that outweighs the correctness this buys.
var transportProvenance sync.Map // map[*http.Transport]transportRecord
// errCAFileChangedSinceTransportBuilt is matchesCachedTransport's sentinel
// for "this IS a transport we built, but the file has changed since" — kept
// distinct from "never heard of this transport at all" so DialWith can give
// a caller a clear, specific instruction (rebuild it) instead of the generic
// unrecognized-transport refusal.
var errCAFileChangedSinceTransportBuilt = errors.New("CA file changed since this transport was built; rebuild via TransportFor or TransportForFile")
// matchesCachedTransport reports whether rt is a *http.Transport this
// package built for path, AND that the file at path still has the exact
// content it was built from — the same instance TransportFor or
// TransportForFile would return for it right now, or would have returned
// for it at some point before path's cache entry was evicted or replaced by
// a later rotation. DialWith uses this to recognize the documented embedder
// recipe (backend/http.TransportFor(target) composed into the caller's own
// *http.Client) as trustworthy rather than refusing it outright: the caller
// built its transport through this package's own CA-scoped constructor, so
// there is nothing left to verify beyond "has the file since changed". A
// transport built any other way — even one that happens to trust the right
// CA, even one this package built for a DIFFERENT path — does not match,
// because this package cannot know what else the caller may have layered
// onto an unrelated transport.
//
// A non-nil error (other than errCAFileChangedSinceTransportBuilt, checked
// via errors.Is) means the current file could not be read at all — a
// distinct failure from "does not match", which DialWith surfaces rather
// than treating as a silent refusal indistinguishable from an unrelated
// transport.
func matchesCachedTransport(rt http.RoundTripper, path string) (bool, error) {
if rt == nil || path == "" {
return false, nil
}
t, ok := rt.(*http.Transport)
if !ok {
return false, nil
}
abs, err := filepath.Abs(path)
if err != nil {
return false, nil
}
recVal, ok := transportProvenance.Load(t)
if !ok {
return false, nil
}
rec, ok := recVal.(transportRecord)
if !ok || rec.path != abs {
return false, nil
}
// Re-read and re-hash the file's CURRENT content: recognizing the
// pointer alone would accept a transport built from a CA that has since
// been rotated (or revoked) out from under it, whether or not this
// path's cache entry still names this same transport.
data, err := readCAFile(abs, "ca_file")
if err != nil {
return false, err
}
if sha256.Sum256(data) != rec.hash {
return false, errCAFileChangedSinceTransportBuilt
}
return true, nil
}
// transportForFile is the shared build behind TransportFor and
// TransportForFile: resolve path to an absolute one, read and hygiene-check
// it, and either reuse the cached transport for that (path, content) pair or
// build and cache a fresh one. label names which setting (CAFileEnv or
// "ca_file") a refusal should point at.
func transportForFile(path, label string) (http.RoundTripper, error) {
abs, err := filepath.Abs(path)
if err != nil {
return nil, fmt.Errorf("%s %q: resolve absolute path: %w", label, path, err)
}
// Claim this read's sequence number BEFORE doing any file I/O, under the
// same lock the cache itself is guarded by: this is what lets two
// concurrent reads be ordered correctly regardless of what the
// filesystem's clock says about either one's mtime. See
// cachedTransport's doc.
transportCacheMu.Lock()
transportReadSeq++
mySeq := transportReadSeq
transportCacheMu.Unlock()
data, err := readCAFile(abs, label)
if err != nil {
return nil, err
}
hash := sha256.Sum256(data)
transportCacheMu.Lock()
if cached, ok := transportCache[abs]; ok && cached.hash == hash {
touchTransportCacheLocked(abs)
t := cached.transport
transportCacheMu.Unlock()
return t, nil
}
transportCacheMu.Unlock()
pool, err := parseCAPool(data, abs, label)
if err != nil {
return nil, err
}
transport := http.DefaultTransport.(*http.Transport).Clone() //nolint:errcheck // http.DefaultTransport is always *http.Transport
transport.TLSClientConfig = &tls.Config{MinVersion: tls.VersionTLS12, RootCAs: pool}
transport.MaxIdleConnsPerHost = maxIdleConnsPerHost
transport.Proxy = caAwareProxy
transportCacheMu.Lock()
defer transportCacheMu.Unlock()
if cached, ok := transportCache[abs]; ok {
if cached.hash == hash {
// Another goroutine built and cached this exact content while we
// were building ours; reuse theirs. Ours was never assigned to
// any connection, so there is nothing on it to close.
touchTransportCacheLocked(abs)
return cached.transport, nil
}
if cached.seq > mySeq {
// A read that claimed its sequence number AFTER ours already
// landed a different (necessarily newer) hash. Never let a read
// that started earlier clobber a cache entry from one that
// started later just because its content hash differs — the
// file may have been rewritten again between this read's open
// and this lock, and ordering by seq (not mtime) is exactly
// what stays correct under a same-second rewrite.
touchTransportCacheLocked(abs)
return cached.transport, nil
}
// Genuine rotation: close the outgoing transport's idle connections
// so they do not leak, then replace it.
cached.transport.CloseIdleConnections()
}
transportCache[abs] = &cachedTransport{hash: hash, seq: mySeq, transport: transport}
transportProvenance.Store(transport, transportRecord{path: abs, hash: hash})
touchTransportCacheLocked(abs)
evictOverCapacityLocked()
return transport, nil
}
// touchTransportCacheLocked moves abs to the most-recently-used end of
// transportCacheOrder. Callers must hold transportCacheMu.
func touchTransportCacheLocked(abs string) {
for i, k := range transportCacheOrder {
if k == abs {
transportCacheOrder = append(transportCacheOrder[:i], transportCacheOrder[i+1:]...)
break
}
}
transportCacheOrder = append(transportCacheOrder, abs)
}
// evictOverCapacityLocked closes and drops the least-recently-used entries
// until the cache is back within maxCachedTransports. Callers must hold
// transportCacheMu.
func evictOverCapacityLocked() {
for len(transportCacheOrder) > maxCachedTransports {
oldest := transportCacheOrder[0]
transportCacheOrder = transportCacheOrder[1:]
if cached, ok := transportCache[oldest]; ok {
cached.transport.CloseIdleConnections()
delete(transportCache, oldest)
}
}
}
// caFileReadHookForTest, when non-nil, runs immediately after
// checkCAFilePermissions succeeds and BEFORE the file is opened for reading
// — the exact TOCTOU window the O_NOFOLLOW-open-then-fstat/SameFile guard in
// readCAFile below exists to close. Production never sets it; a test sets it
// to swap the file (or replace it with a symlink) at exactly that point and
// confirms the read is refused rather than silently trusting whatever
// replaced it.
var caFileReadHookForTest func(real string)
// readCAFile reads path after checking its hygiene (permissions, ownership,
// every symlink hop's containing directory) and confirming that the file it
// actually reads is provably the same one that hygiene check examined,
// returning its content. path must already be absolute; label names the
// setting a refusal should point at.
//
// checkCAFilePermissions resolves path to a real, non-symlink file (checking
// every symlink hop's own containing directory along the way, not just the
// final target's) and returns both that real path and the os.FileInfo it
// stat'd for it. Between that check and this function's own open, an
// attacker who can still write somewhere in the resolved path could swap the
// leaf out from under it; this function's own open uses O_NOFOLLOW — so a
// swap for a symlink is refused outright — and then fstats the opened
// descriptor and compares it against the checked FileInfo with os.SameFile,
// so a swap for a DIFFERENT regular file (same name, same directory) is also
// caught before a single byte is trusted: reading is always from the fd just
// opened, never from a second, independent path lookup.
func readCAFile(path, label string) ([]byte, error) {
real, checked, err := checkCAFilePermissions(path)
if err != nil {
return nil, fmt.Errorf("%s %q: %w", label, path, err)
}
if caFileReadHookForTest != nil {
caFileReadHookForTest(real)
}
// #nosec G304 -- path is operator-provided configuration
// (BEADS_HTTP_CA_FILE, or the per-user http_target.json sidecar written by
// `bd connect --ca-file`), not attacker-controlled input; its permissions
// were just checked above, and openCAFileNoFollow additionally refuses to
// open real at all if it has become a symlink since that check (unix).
f, err := openCAFileNoFollow(real)
if err != nil {
return nil, fmt.Errorf("%s %q: %w", label, path, err)
}
defer f.Close() //nolint:errcheck // read-only fd; nothing to flush on close
opened, err := f.Stat()
if err != nil {
return nil, fmt.Errorf("%s %q: %w", label, path, err)
}
if !os.SameFile(checked, opened) {
return nil, fmt.Errorf("%s %q: changed between the permission check and the read; refusing rather than trust whatever replaced it", label, path)
}
data, err := io.ReadAll(io.LimitReader(f, maxCAFileBytes+1))
if err != nil {
return nil, fmt.Errorf("%s %q: %w", label, path, err)
}
if len(data) > maxCAFileBytes {
return nil, fmt.Errorf("%s %q: larger than %d bytes; a CA bundle this size is almost certainly the wrong file", label, path, maxCAFileBytes)
}
return data, nil
}
// parseCAPool decodes data as a sequence of PEM blocks and requires EVERY
// block to be a well-formed CERTIFICATE. A block of any other type, or one
// that fails to parse, refuses the WHOLE file rather than silently keeping
// whatever did parse — crypto/x509.CertPool.AppendCertsFromPEM's behavior,
// which this deliberately does not use. A CA file is small, operator-owned
// configuration, so a block it cannot vouch for (a private key pasted in by
// accident, a corrupted certificate) is far more likely a mistake than an
// intentional block meant to be skipped, and accepting the file anyway would
// trust a pool the operator never actually reviewed.
//
// A single LEAF certificate — no CA:true, no key-cert-sign usage — still
// works here: TLS root-of-trust verification only requires the chain to
// terminate at a certificate in RootCAs, so a file holding just the server's
// own leaf is a valid way to pin that one certificate rather than a CA. It is
// operationally brittle (it must be replaced on every renewal), but this
// package does not refuse it: pinning is a legitimate, if narrower, use of
// this same mechanism.
func parseCAPool(data []byte, path, label string) (*x509.CertPool, error) {
pool := x509.NewCertPool()
rest := data
n := 0
for {
var block *pem.Block
block, rest = pem.Decode(rest)
if block == nil {
break
}
n++
if block.Type != "CERTIFICATE" {
return nil, fmt.Errorf("%s %q: PEM block %d is %q, not CERTIFICATE; only certificates (a CA, or a single leaf to pin) belong in this file", label, path, n, block.Type)
}
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, fmt.Errorf("%s %q: PEM block %d does not parse as a certificate: %w", label, path, n, err)
}
pool.AddCert(cert)
}
if n == 0 {
return nil, fmt.Errorf("%s %q: no PEM certificates found", label, path)
}
return pool, nil
}
// caAwareProxy is the CA-scoped transport's Proxy func. For a plain http://
// proxy it behaves exactly like http.ProxyFromEnvironment: the destination
// host's own TLS is still verified end to end against this transport's
// narrow RootCAs, so routing through a plaintext-to-the-proxy CONNECT tunnel
// changes nothing about what gets trusted.
//
// It refuses an https:// proxy instead of dialing it. The CONNECT to the
// proxy ITSELF would be a TLS handshake made with this same TLSClientConfig,
// so the proxy's own certificate — issued by whatever CA actually vouches for
// it, almost never the single private CA this transport was scoped to — would
// be checked against that narrow pool and (correctly) fail, or, worse, if it
// happened to validate, Proxy-Authorization would be sent across a connection
// verified by the wrong trust anchor. Refusing up front is clearer than
// either outcome.
//
// This reads the proxy configuration via golang.org/x/net/http/httpproxy
// directly rather than http.ProxyFromEnvironment: the latter caches the
// environment the first time ANY transport in the process resolves a proxy,
// behind a package-level sync.Once, which would make this refusal depend on
// unrelated dial or test ordering. httpproxy.FromEnvironment reads the
// environment fresh on every call.
func caAwareProxy(req *http.Request) (*url.URL, error) {
proxyURL, err := httpproxy.FromEnvironment().ProxyFunc()(req.URL)
if err != nil {
return nil, err
}
if proxyURL != nil && proxyURL.Scheme == "https" {
return nil, fmt.Errorf(
"refusing to reach %s through an https:// proxy (%s) while a CA is configured for this target: the proxy's own TLS would be checked against that CA instead of the system roots that actually vouch for it; use an http:// proxy instead (the destination's TLS is still verified against the configured CA through it), or exclude this host via NO_PROXY",
req.URL.Host, proxyURL.Redacted())
}
return proxyURL, nil
}