For as long as there have been deployment tools, there have been efforts to modularize software components for reuse.
VMs have golden images and templates. Configuration management tools like Puppet and Terraform have modules. Containers are the epitome of reuse, and Kubernetes has taken that to a whole new level, helping developers describe and deploy complex Docker container sets and their associated services, sidecars, and connectivity.
Although undoubtedly powerful, Kubernetes manifests can become cumbersome to manage, with directly embedded variables changing per user, deployment, or environment.
That’s where Helm comes in, making Kubernetes manifest management easier and continuing the tradition of reusable components.
Helm, the package manager for Kubernetes, separates commonly changed variables—often with defaults—to generate the necessary Kubernetes manifests. Helm also does kubectl apply, tracks which templates (called charts) you have deployed into your cluster, and neatly packages charts for easy consumption.
Helm reduces the need for manually-written and managed Kubernetes YAML in the name of efficiency. But just like pulling in your favorite NodeJS, Python, or Golang modules, care needs to be taken when using someone else’s code. Infrastructure as code (IaC) is no exception.
Helm security considerations
Just like every other infrastructure language, Kubernetes comes with its own set of industry security guidelines, including those from the Center for Internet Security (CIS). The CIS Kubernetes Benchmarks highlight numerous ways an object deployed to Kubernetes can be given too many permissions if specific configuration is omitted from the Kubernetes YAML defining the object. It also makes recommendations against configuration options that may provide a larger surface area for attack.
The following example Kubernetes snippet taken from the “vulnerable-by-design Kubernetes” project, KubernetesGoat, highlights misconfigurations with privileged: true, and exposure of the underlying host filesystem via a VolumeMount:
apiVersion: apps/v1
kind: Deployment
metadata:
name: system-monitor-deployment
spec:
selector:
matchLabels:
app: system-monitor
template:
metadata:
labels:
app: system-monitor
spec:
hostPID: true
hostIPC: true
hostNetwork: true
volumes:
- name: host-filesystem
hostPath:
path: /
containers:
- name: system-monitor
image: madhuakula/k8s-goat-system-monitor
resources:
limits:
memory: "50Mi"
cpu: "20m"
securityContext:
allowPrivilegeEscalation: true
privileged: true
ports:
- containerPort: 8080
volumeMounts:
- name: host-filesystem
mountPath: /host-system
Helm chart misconfiguration
Because Helm makes the reuse of templated Kubernetes manifests much more consumable, it’s important to check these packaged templates for resulting manifests that do not follow CIS Kubernetes guidelines.
Helm chart misconfigurations will look exactly the same as that of a Kubernetes YAML manifest misconfiguration. But because the default values and boilerplate YAML is still in unprocessed template form and have not yet been brought together, misconfigurations are far easier to miss.
Finding issues at build time with Checkov
As you may already know, Bridgecrew’s open-source IaC scanner, Checkov, already supports finding security issues in Kubernetes YAML with 150+ out-of-the-box checks for Kubernetes deployments.
To include Helm templates in our scans, we’ve used the helm template command to output the resultant Kubernetes manifests and scan those against the Kubernetes specific policies in Checkov with checkov --framework kubernetes -d <template files>.
Finding security misconfigs in Helm charts
It was awesome to see one of our community members take this approach and implement a pre-commit hook to scan all of his Helm charts for issues automatically.
While useful for the users specific directory layout, we can extend this to find any Helm directory containing a Chart.yaml below our current path, render the charts, and scan them. The rendered YAML will be named after the chart to make it easier to parse in the Checkov output when the scan finds multiple misconfigured charts.
bash -c 'find -iname chart.yaml' | xargs -n1 -I% bash -c " dirname %" | xargs -n1 -I% bash -c "helm template % > %.yaml && checkov -f %.yaml --framework kubernetes || true" --
Running over the Prometheus Helm chart repository, as an example, we see Checkov scan the twenty-three charts in the repo.
Assessing scan results
For each policy violation found, Checkov will provide the section of rendered Kubernetes manifest involved as well as a description of the issue and link to further guidance.
In the example above, we can see the violation relates to the prometheus-node-exporter chart, specifically, the PodSecurityPolicy isn’t preventing containers from running as root.
The Guide link provides more context on the issue, as well as information for the developer on common remediations:
In this example, the violation may be necessary to allow the node-exporter to access low-level system metrics from the host. However, it’s important to understand, validate, and where necessary, document suppressed violations within your infrastructure.
Checkov supports inline documentation of suppressed Kubernetes checks using annotations. For example, consider the following diff of the prometheus-node-exporter chart:
We now have version controlled history of security-related exceptions which will allow Checkov to safely skip a check for this given resource when rendered with our Helm scanning command:
Automating Helm scanning with GitHub Actions
Now let’s turn our working command into automated Helm chart security scanning in our CI/CD pipeline using GitHub Actions.
Leaving the pipeline to return ||true will allow you to see the entire output, but if you want to fail your build when errors are identified, just remove that argument.
By scanning Helm charts continuously and automatically, we can identify and address security issues before they’re packaged and deployed.
Get started with Checkov here and join the conversation in our #CodifiedSecurity Slack community.
