Signed Container Images: Sigstore + Admission Gates

Q: Why is signing container images in CI not enough to secure the supply chain?

Because a signature only proves that a holder of a key or identity produced the artefact; it proves nothing about whether the artefact was built from reviewed source by a trusted pipeline, and it proves nothing at all if nothing at deploy time refuses an unsigned or wrongly-signed image. Most teams that say they sign everything have a cosign sign step in CI and no admission control in the cluster, so a kubectl apply from a laptop runs whatever image it references. Worse, every recent supply-chain compromise shipped through a pipeline that would have signed the malicious output too, because the build was poisoned upstream of the signature. The control is the combination of a signature bound to a specific CI identity, a SLSA provenance attestation naming the builder and source, and an admission gate that fails closed when either is missing or wrong.

Q: What did the August 2026 cosign advisory GHSA-fx35-mq7g-6g98 actually allow?

It was a verification bypass in cosign verify-blob and cosign verify-blob-attestation when a legacy JSON bundle was supplied. If the bundle's cert field failed to parse as an X.509 certificate, cosign silently fell back to loading it as a raw public key, and once a raw key was set the certificate-chain validation and the certificate-identity and certificate-oidc-issuer policy checks were skipped. Verification then succeeded for any signature that was cryptographically valid for the embedded key, which the attacker chose, and an embedded key could even overwrite an explicitly trusted key passed with the key flag. OCI container-image verification and the standardised bundle format that cosign v3 uses by default were not affected. It was fixed in v3.1.3 and backported to v2.6.5 on 6 August 2026. The architectural lesson is that the identity policy is the entire value of keyless signing, and a verifier must be tested with inputs it is supposed to reject.

Q: What is the difference between verifying a signing key and verifying a signing identity?

Verifying against a static public key answers whether the signer held the corresponding private key, which is a long-lived secret that must be stored in CI, rotated, and can be stolen without any signal that theft occurred; any job that can read the secret can produce an admissible image. Keyless signing with Sigstore instead ties the signature to a short-lived certificate issued by Fulcio for an OIDC identity, typically the CI workflow itself, with the signing event recorded in the Rekor transparency log. The verification policy then becomes a statement about identity: the certificate's OIDC issuer must be your CI provider and the subject must match a specific repository and workflow path. That constraint is what stops a stolen token being used from a different workflow, provided the subject match is tight rather than a wildcard. It is also the check the August 2026 advisory bypassed for blob verification, which is why it deserves explicit negative tests.

Q: What does a SLSA provenance attestation add that a signature does not?

A signature says who signed the artefact. A SLSA provenance attestation says how it was built: which build platform, from which source repository and commit, on which branch or tag, with which parameters. SLSA v1.0 defines Build L1 as provenance existing, L2 as provenance generated on a hosted build platform and signed, and L3 as hardened builds in which the platform isolates runs from one another and protects the signing material from the build steps. A gate that requires L2 or L3 provenance from a named builder with the source repository pinned is the only control that catches an attacker who has push access to the registry, or a stolen CI credential, and produces a perfectly-signed image from unreviewed source. GitHub's artifact attestations and similar features generate this provenance through Sigstore for any repository on a current plan, so the cost is a workflow step plus a policy that insists on the builder and source.

Q: How should an admission gate handle a registry or transparency-log outage without taking the cluster down?

By bounding where fail-closed applies and caching what has already been verified. Kubernetes validating webhooks have a default timeout of 10 seconds, configurable from 1 to 30, and a failurePolicy that defaults to Fail, so a gate applied to every namespace including kube-system turns a registry or Rekor outage into a cluster that cannot reschedule its own control-plane pods. The correct design enforces with failurePolicy Fail in workload namespaces, exempts system namespaces explicitly with the exemption logged and alerting, and gives the controller a verdict cache keyed on image digest so that node restarts and rolling deployments do not re-verify every pod against a struggling registry. It also pins the Sigstore trust root through TUF rather than fetching it live, treats "log unreachable" as a distinct, alertable outcome from "signature invalid", and provides an expiring, audited break-glass annotation rather than a global bypass label.

Q: Why must images be admitted by digest rather than by tag?

Because a tag is a mutable pointer and a digest is the content. If the admission controller verifies the signature on the image a tag currently resolves to and then admits the pod spec with the tag still in it, there is a window in which the registry can serve different bytes to the kubelet than the controller verified, either through a race, a malicious push, or a mirror inconsistency. Verifying a name is not verifying an image. Policy engines address this by rewriting the tag to the verified digest before the object is persisted; Kyverno's verifyImages rule does this by default through its mutateDigest setting. The stricter alternative for production namespaces is to reject any non-digest image reference outright so that manifests are immutable by construction and the gate has nothing to race against.

Q: How do you roll out image-verification enforcement across an existing fleet without blocking legitimate deploys?

Stage it per namespace and gate each stage on evidence. First, build an inventory of running images by registry of origin and namespace, because the policy needs a rule for first-party images, mirrored third-party base images, vendor images and the unknown remainder. Second, make every first-party pipeline sign keylessly and emit SLSA provenance, and route third-party images through a mirror pipeline that re-signs after scanning. Third, run the policy controller in audit mode everywhere, logging verdicts without denying, until the would-deny count in a namespace has been zero for longer than that namespace's slowest release cadence. Fourth, flip that namespace to enforce with an expiring break-glass path that pages when used. Teams that switch to enforce on a calendar date rather than on a quiet audit log produce the outage that gets the programme cancelled; teams that never leave audit mode have a report, not a control.

Q: How do you know the image verifier is actually working rather than silently approving everything?

You prove it with negative tests, continuously, the same way you would test a detection rule or a firewall. The policy CI should contain images that must be denied: one signed by the right identity but the wrong workflow, one with a valid signature and no provenance, one whose provenance names a self-hosted builder when policy requires a hosted one, and one using a legacy bundle format where the policy expects the standardised one. Every one must produce a denial in a test cluster on every policy change and every controller or cosign upgrade. In production, a canary image that must be denied should be submitted daily in every cluster with an alert if it is ever admitted. The August 2026 cosign advisory is the reason: the bug went undetected because verification kept returning success, and no dashboard distinguishes a verifier that checks identity from one that has stopped.