Admission controllers are one of the built-in security features that Kubernetes has to help govern and enforce how clusters are used. From The Linux Foundation, “An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.”
Although there are several admission controllers, two are unique in that they are dynamic, webhook driven, and can be developed as custom extensions. They are the MutatingAdmissionWebhook and ValidatingAdmissionWebhook, of which only the latter is in scope for this blog.
Our new Kubernetes Admission Controller is a Validating Admission Controller, which features Checkov as the core validator for Kubernetes manifests.
Watch the video walkthrough below or keep reading to learn the what, why, and how of Whorf.
In keeping with our Star Trek-themed open source project names like Checkov, Yor, and AirIAM, we have nicknamed it Whorf, combining the namesake attributes of an oceanside wharf ready to take you to sea with a Klingon security chief called Worf. Whorf is your last line of defence against deploying vulnerable or misconfigured Kubernetes objects by scanning them against Checkov policies.
Image Concept Credit: Kristijan Mitevski
Why a Validating Admission Controller
As suggested by DevSecOps, it’s far from best practice to wait until the very last minute to validate that Kubernetes manifests are secure. It slows down our pipeline drastically as the feedback loop between a blocked deployment and the correct details reaching a developer for a fix and redeploy is far from a recipe for DevSecOps. It also contributes to the perception of security being the department of “NO!”. In a perfect world, all of our developers would be scanning their code locally with Checkov CLI or, better yet, in the IDE using the VS Code or Jetbrains plugins. The alternate—and perhaps more realistic and efficient—way to apply security checkpoints is in CI pipelines via integrations like GitHub Actions.
Our perfect world is, however, far from reality. Different segments of our software pipeline are controlled by different personas, and each persona has some control over security governance. In fact, some compliance standards mandate this defence-in-depth.
The Validating Admission Controller is security applied late in the game, directly from within the Kubernetes cluster serving as the bouncer or final guard. Each manifest applied to the cluster, be it in its original form as YAML, JSON, Helm, or Kustomize, will be sent to our controller. At that point, it is checked against a user-defined subset of Checkov policies before being allowed to become persistent in the cluster.
By default, the entire spectrum of policies is not applied. Only the worst of the worst are being checked as, at this point in the pipeline, we don’t have the luxury of context. Thus, we have stripped the checks back to only a small but configurable subset that exhibits clear and present danger, such as deploying privileged containers (CKV_K8S_16) or allowing excessive privileges like CAP_SYS_ADMIN (CKV_K8S_39). Check out our recent blog about CVE-2022-0185 for some details on why that is an essential rule. The default Kubernetes configmap with the complete admission controller ruleset can be found in the GitHub repository.
Properly configured and secure manifests will flow seamlessly through our admission controller unaltered, and the checks will be invisible. It will appear as if there is no Validating Admission Controller at all (with this exception of logging, of course). Manifests that fail will be blocked with output looking similar to this:
You’ll also see admission controller scan results in the Projects screen in the Bridgecrew platform:
You might ask yourself: why bother adding an admission controller if I’m already scanning my manifests in CI with even more strict rulesets? There are multiple reasons for this:
- Trust but verify. If you’re scanning all manifests in CI, the admission controller is simply an invisible defence-in-depth measure to ensure nothing bypasses your existing security checks.
- In some organisations where applying CI security controls is not as straightforward or simple, the size of the organisation means that many pipelines can produce assets for a single cluster. Thus, it is in the interest of central security teams to ensure that teams use CI controls. A failure of the Admission Controller is a key indicator that a CI workflow was missing crucial security checks.
- Admission Controllers do not just check the deployment of new resources. They also check resources after they have been changed to ensure security requirements are still being met.
This last point might strike a chord for security enthusiasts and GitOps practitioners who believe in the immutability of runtime cloud and Kubernetes resources. Kubernetes objects in a production runtime environment should not be changing via imperative commands or direct calls to the Kubernetes API. At best, this is considered reckless debugging. At worst, it may indicate malicious intent to elevate privilege after persistence has been established in the kill chain. (). An admission controller also validates changes is often one of the only ways to both detect and prevent such modifications at runtime.
Getting started with the Checkov Validating Runtime Controller
1. The admission controller requires the use of a Bridgecrew API key, so you’ll need to create a free account. Once your account is verified, head to the Integrations screen and select the Kubernetes Admission Controller integration:
2. You’ll first be prompted to name and create your new API key:
3. Next, define a cluster name for unique identification both for your organisation and within our platform UI:
4. You’ll then need to run a short series of commands to create and install the manifests for the Kubernetes Admission Controller within the scope of your Kubernetes cluster:
Note: Ensure your KUBECONFIG and context are set correctly first.
If you already have a Bridgecrew API key, instructions for how to deploy can also be found in the Whorf README.
Everything gets installed into the bridgecrew namespace and the manifests for the installation are created and stored into a unique subdirectory below where the installation began called bridgecrewYYYYMMDDHHMMSS (e.g., bridgecrew20220118145756).
To undo the installation, you can simply run:
kubectl delete -f bridgecrew20220118145756 kubectl delete ns bridgecrew
Note: Only run the second step if you are not running the Kubernetes Checkov cronjob as this will delete all objects in the namespace bridgecrew.
···
Admission controllers provide essential defence-in-depth for new manifests entering Kubernetes clusters. They also provide guardrails to prevent runtime modifications, be that from manual or malicious changes via command-line or API or even risks introduced by well-meaning Mutating Admission Webhooks.
For security practitioners looking to embrace their own “trust but verify” approach to Kubernetes security, an admission controller that leverages the same scanning technology as DevOps teams is an ideal backstop to ensure the security of Kubernetes clusters now and in the future.
