beads/third_party/cc_toolchain/deb_sysroot.bzl
Julian Knutsen a945eca145
ci(bazel): fetch pinned Ubuntu debs in parallel from a gascity mirror first (#7434)
* ci(bazel): fetch pinned Ubuntu debs in parallel from a gascity mirror first

cc_sysroot_noble_amd64 downloaded its 7 pinned .debs one at a time from
snapshot.ubuntu.com, with only archive.ubuntu.com behind it. On
2026-10-09 the snapshot returned 503, and Bazel's retries against it
cost ~95 s per cold fetch before falling back (and archive.ubuntu.com
drops superseded versions, which would fail the fetch outright).

Same change as gastownhall/gascity (ga-vnycm2.11): deb_sysroot takes URL
templates ({path}, {file}, {sha256}) and starts every package download at
once (rctx.download block = False). NOBLE_DEB_URLS tries gascity's
content-addressed release mirror (toolchain-noble-debs-20261001, which
holds these exact pins; beads already takes its LLVM slice from a gascity
release), the snapshot, Launchpad's librarian, then archive.ubuntu.com.
The sha256 pins are unchanged and decide what is accepted; the unpacked
sysroot is byte-identical. The hermetic-cc guard now checks every
deb_sysroot: placeholders, the snapshot kept, two or more hosts, no
moving mirror first, no ICU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37966817176
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-09 17:47:17 +00:00

93 lines
3.3 KiB
Python

"""deb_sysroot: a C/C++ sysroot assembled from pinned Debian/Ubuntu packages.
Every package is fetched by sha256, so the sysroot is byte-identical on every
host whichever source served it; the sources are only an availability chain.
All packages download concurrently (Bazel bounds it by
--http_max_parallel_downloads), each trying its URLs in order. It gives the hermetic LLVM toolchain (MODULE.bazel) the glibc,
libstdc++ and ICU headers and link libraries that cgo targets need, instead
of whatever the host has under /usr.
"""
# Trims what no compile or link needs, then makes the sysroot self-contained:
# package symlinks may be absolute (outside a chroot they would resolve
# against the HOST), and glibc's linker scripts name the pre-usr-merge /lib
# and /lib64, which the sysroot only has under /usr.
_POSTPROCESS = """
set -euo pipefail
shopt -s globstar nullglob
rm -rf -- $REMOVE
find . -type l -lname '/*' -print0 | while IFS= read -r -d '' l; do
ln -sfn "$(realpath -m --relative-to="$(dirname "$l")" "$PWD$(readlink "$l")")" "$l"
done
find . -xtype l -delete
grep -rlI --include='*.so' 'GROUP' usr/lib | while IFS= read -r s; do
sed -i -e 's@ /lib/@ /usr/lib/@g' -e 's@ /lib64/@ /usr/lib64/@g' "$s"
done
"""
def _package_urls(templates, path, sha256):
return [
t.format(path = path, file = path.rsplit("/", 1)[-1], sha256 = sha256)
for t in templates
]
def _deb_sysroot_impl(rctx):
pending = []
for path, sha256 in rctx.attr.packages.items():
deb = "_debs/" + path.rsplit("/", 1)[-1]
pending.append((deb, rctx.download(
url = _package_urls(rctx.attr.urls, path, sha256),
output = deb,
sha256 = sha256,
block = False,
)))
for _, download in pending:
download.wait()
for deb, _ in pending:
unpacked = deb + ".d"
rctx.extract(deb, output = unpacked)
data = [f for f in rctx.path(unpacked).readdir() if f.basename.startswith("data.tar")]
if len(data) != 1:
fail("%s: expected one data.tar.* member, found %s" % (deb, data))
rctx.extract(data[0], output = ".")
rctx.delete("_debs")
res = rctx.execute(
["bash", "-c", _POSTPROCESS],
environment = {"REMOVE": " ".join(rctx.attr.remove)},
)
if res.return_code != 0:
fail("deb_sysroot: post-processing failed:\n" + res.stderr)
rctx.file("BUILD.bazel", """\
filegroup(
name = "sysroot",
srcs = glob(
["**"],
exclude = [
"BUILD.bazel",
"REPO.bazel",
],
),
visibility = ["//visibility:public"],
)
""")
return rctx.repo_metadata(reproducible = True)
deb_sysroot = repository_rule(
implementation = _deb_sysroot_impl,
attrs = {
"packages": attr.string_dict(
mandatory = True,
doc = "Pool path (pool/<component>/...) to the .deb's sha256.",
),
"urls": attr.string_list(
mandatory = True,
doc = "URL templates tried in order for every package: {path} is the pool " +
"path, {file} its basename and {sha256} its pinned digest.",
),
"remove": attr.string_list(
doc = "Bash globs (globstar) dropped after unpacking: docs, unused runtimes and archives.",
),
},
)