Skip to content

verify-release-binaries

Proves that every release binary published to the estate's object store is still there, and still the bytes it claimed to be.

stages: [verify]

include:
  - component: gitlab.com/phpboyscout/cicd/verify-release-binaries@main
    inputs:
      stage: verify

Runs only on a pipeline schedule carrying VERIFY_TASK=binaries. It is an estate-wide chore, so it wants one schedule in one place rather than a copy per project, and that place is the private project phpboyscout-private/renovate, beside the group Renovate bot.

It must not run in a project whose pipelines are public, cicd included. It enumerates the group and logs every project with releases, private ones too (spec 0112 D2). The engine is embedded in the component, so the job needs no checkout of cicd and runs wherever it is included.

Why it exists

Release binaries live on pkg.phpboyscout.uk and carry a public guarantee: a published file is never changed and never removed. Nothing enforces that.

The object store's bucket lock, which did enforce it absolutely, was dropped deliberately — under it an accidentally published secret would have been unremovable, which is a worse position than the tampering it defended against. The credential that writes releases holds DELETE, and the store has no object versioning. So enforcement was replaced with detection, and this is the detection. See colophon spec 0025 D13 and infra spec 0019 D4.

It depends on nothing

The engine imports only Python's standard library. No pip, no install step, nothing fetched at run time. This job is the only thing standing behind the release store's guarantee and it runs unattended, so it must not be able to fail because a package was missing from an image — which it did, once, on ci-base, which carries python3 and no pip.

It holds no bucket credential

It reads the public URL like any other consumer. That is deliberate twice over:

  • it verifies what a user actually receives, rather than what the bucket believes it holds;
  • a verifier that cannot write can never be the cause of the damage it reports.

The only credential is a GitLab token with api scope, to enumerate releases across the group and to raise an issue when something is wrong. CI_JOB_TOKEN will not do — it cannot read other projects' releases.

Three tiers

The estate does not publish uniformly, so a verifier written to one shape would fail on a third of it.

tier what it proves where it runs
reachability every published link resolves everywhere, every run
integrity the bytes match the published digest where checksums.txt exists
authenticity the digest list is signed where a signature exists
project publishes
colophon, go-tool-base, ffmpeg-wasi checksums.txt + a PGP .sig
keryx, skillup checksums.txt, no signature
rust-tool-base per-file .minisig, no checksums.txt

A tier with nothing to run against is counted, not failed. Reporting rust-tool-base as broken every night because it signs differently would train everyone to ignore this job, which is the only failure mode that really matters for a scheduled check.

Integrity rotates; reachability does not

Verifying every digest means downloading everything the estate has ever released — about 23 GiB, growing roughly 1 GiB a month. Nightly, that is hours of runner1 spent re-reading bytes that were correct yesterday.

So each run digest-verifies one share, and integrity_parts runs cover everything. Shares are sliced on a hash of the release's identity rather than its position, so adding a release does not reshuffle every other release into a different share.

Reachability still covers everything, every run, because it is a HEAD. That is the tier that catches deletion, which is the failure this job exists for.

It is allowed to fail

allow_failure defaults to false, unlike most scheduled chores. A missing or altered release binary is a broken public guarantee and possibly a compromised credential. A check that cannot fail is not a check.

Issues are filed on the affected project and are idempotent: a fault that persists does not file a new issue every night.

Inputs

Input Type Default Description
stage string test Stage to run in.
image string ci-base:v0.3.4 Any image with python3; the engine is stdlib-only.
if string '$CI_PIPELINE_SOURCE == "schedule" && $VERIFY_TASK == "binaries"' The gate.
group string phpboyscout Group to enumerate.
projects string "" Space-separated project paths to check instead of the group.
base_url string https://pkg.phpboyscout.uk Only release links under this are checked.
integrity_parts number 7 Cover every release's digests over this many runs.
skip_integrity boolean false Reachability only, the cheap pass.
open_issues boolean true File an issue on each affected project.
token_variable string RELEASE_VERIFY_TOKEN Name of the CI/CD variable holding the token.
allow_failure boolean false Whether a finding fails the pipeline.

token_variable names a purpose-named, hidden variable holding a service account's token, per spec 0108. It was the group GITLAB_TOKEN until that variable was retired on 2026-10-07. CI_JOB_TOKEN will not do: it cannot enumerate the group or file issues.

Scheduling it

Off-peak. runner1 is concurrent = 2 and is the whole Linux estate while the AWS fleet is off; the nightly Renovate scan holds 00:00 UTC.

The schedule must set VERIFY_TASK=binaries. That variable is load-bearing: schedules all run against the default branch, so without it every schedule on the project would fire every scheduled job. In phpboyscout-private/renovate it runs daily at 03:30 UTC, clear of the Renovate scan at 00:00 and the hourly merge sweep.