The modern software supply chain is a complex patchwork of dependencies and libraries from first and third party vendors and open source projects. With no industry-wide standard to secure these software supply chains, attacks have exploded – there has been a 742% average annual increase in supply chain attacks in the past three years.
Recognizing the threat to national security, and after incidents like SolarWinds and the Colonial Pipeline attack made national headlines, in May of 2021, U.S. President Joe Biden released an Executive Order on Improving the Nation’s Cybersecurity. The goal for this order was to improve software supply chain security, by creating requirements for securing development environments; and establish trusted supply chains by ensuring code integrity. At Virtru, our migration to Kubernetes forced us to confront the realities of an ecosystem that doesn’t have a standard way of verifying the origin or integrity of the many pieces of software that now comprise our cloud infrastructure. Our solution was to adopt the software and standards of a quickly growing project that is taking the Kubernetes world by storm – the Sigstore project.
Cosign: Cryptographic Signing for Stronger Security
The Sigstore project, introduced in 2021, is an open source collaboration between Red Hat, Google, Linux Foundation, and many other software vendors. Their mission is aimed at “improving software supply chain integrity and verification.” Part of this initiative is a suite of software tools that includes Cosign, Rekor, and Fulcio.
Cosign is a utility developed to support a new standard for signing and verifying container images. It is much simpler to start using and more flexible than alternatives like Docker Notary, which is a competing standard developed by Docker. It simplifies the process of signing and verification by allowing teams to reuse existing development infrastructure, like cloud-managed KMS and container registries. With Cosign, the container image signatures are stored in the same registry as the container, and linked to the associated image via the container’s digest. Cosign is very flexible in terms of the signing key. It supports customer-managed keys, as well as keys generated by cloud key management service (KMS). The cloud KMS integration is seamless and doesn’t require any extra coding. There is also a way to opt out of managing the keys entirely, and go with the keyless signing. It is a common practice to integrate the Cosign signing with code scanning in a CI pipeline. This way, the image can be analyzed first to ensure there are no known vulnerabilities, or compliance violations. The analyzed image can then be signed.
These advantages, plus the many other features of Cosign, have seen the project explode in popularity across the Kubernetes community. The official Kubernetes images are signed using cosign format, and the open source community has been incorporating it into other projects in the Kubernetes ecosystem, including Kyverno, another essential piece in Virtru’s Kubernetes infrastructure.
Kyverno: Verifying Signatures for Kubernetes Image Containers
With container images, ensuring that the version running is the version intended to run is crucial. If an image stored in a registry is replaced with a different one, either by accident or a threat actor, the container runtime may run a vulnerable container version, which can lead to a number of exploits. The default protection against these exploits is provided by the registry's security controls. For example, using immutable tags helps to ensure the image content cannot be tampered with. However, immutable tag policy is not a strong guarantee. Any attacker with access to the registry can disable the policy and push a new image to the same tag. Using cryptographic signing can provide a strong guarantee that the container image hasn’t been tampered with and is coming from the trusted source – but container image signing is only useful as long as the signatures are verified. That is where Kyverno comes in.
In Kubernetes clusters, the verification step is handled by admission controllers. There are a number of open source admission controllers supporting the Cosign signature format, including the official Sigstore admission controller, called policy-controller. The admission controller choice is often based on the needs for various features, like keyless verification or flexible enforcement policies written as code. Virtru chose Kyverno for our signature verification admission controller. Our team found the project mature and stable with strong community support on the CNCF Slack. Kyverno supports most of the functionality provided by Cosign, including keyless signing. It allows us to enforce signing not only for internal container images, but also for a number of upstream applications that have begun signing their images with Cosign. To take it one step further, Virtru recently made a decision to sign all the images in the OpenTDF stack to provide an extra layer of security to consumers of our open source products.
The Result: Streamlined Security for the Kubernetes Ecosystem
With the combination of Cosign and Kyverno, we’ve streamlined the process of signing and verifying the container images that we allow on our Kubernetes clusters, hardening our infrastructure from both developer mishaps and threat actors alike. Stay tuned for our next post, where we'll lay out some of the code we used to deploy both Cosign and Kyverno into our Kubernetes clusters!
