6 delivery pipeline security best practices for secure software supply chains

Modern software supply chains are made up of multiple components that take different forms across each stage of the development process. During the development stage, third-party software components such as open source packages, container images, and IaC modules are integrated into codebases. The build and deploy stages rely on third-party pipelines such as VCS providers, container image registries, and CI/CD pipelines that facilitate integrating, building, packaging, testing, and deploying artifacts. And in runtime, third-party software components become real running containers, resources, workloads, and services.

One weakness in any of those software supply chain components can be an entry point or pivot used within a software supply chain attack. And when you consider just how connected and complex our software supply chains have become, it should come as no surprise that 2021 saw a 51% increase in software supply chain attacks.

Until recently, security vendors and practitioners were focused mostly on locking down software components. From SAST, DAST, and SCA to container image and infrastructure as code (IaC) scanning, eliminating vulnerabilities and misconfigurations has become a multi-billion dollar industry. We’ve only recently come to realize the importance of securing the underlying pipelines, such as the version control systems (VCS)—sometimes referred to as source control management (SCM) tools—and continuous integration/continuous delivery (CI/CD) pipelines.

Delivery pipelines and software supply chain security

To support fast, iterative, and high-quality deployments, hosted VCSs and CI/CD pipelines have become the lifeblood of cloud-native organizations. But as organizations adopt more and more tools and processes to keep up with the growing complexity of the development lifecycle, it becomes more challenging to maintain visibility across the entire software supply chain. For that very reason, VCS repositories and CI/CD pipelines are increasingly becoming targets for attacks.

VCS or CI/CD pipeline flaws can expose sensitive information that can then be used to gain privileged access and leak sensitive data. Without the right controls in place, malicious code injections or poisoning can corrupt entire delivery pipelines. Adopting VCS and CI/CD security best practices will help safeguard the components, actions, and procedures involved in the development and deployment of software.

In this post, we’ll look at some of the most common security weaknesses within VCS and CI/CD pipelines that can expose supply chains to attacks. Then we’ll walk through several best practices that security professionals can implement to mitigate supply chain attacks.

Weak VCS organization configuration

Version control systems like GitHub, GitLab, and Bitbucket are a vital part of software supply chains. As the central repositories that store, version, and manage access to all infrastructure and application code, it’s clear what could happen if a bad actor were to gain unauthorized access to them.

The good news is that these vendors provide mechanisms to restrict access. VCS security best practices include enforcing two-factor authentication and configuring single sign-on (SSO). Automating scanning of your VCS organization settings to ensure that they’re complying with VCS security best practices is a good way to get continuous guardrails if any settings change.

Lax code integration policies

Your next line of VCS defense is at the repository level. If a bad actor—whether internal or external—is able to get access to repositories and could attempt to inject malicious code, enforcing policies around how and who can merge code is crucial.

The best way to enforce that is through branch protection rules. Branch protection rules enable you to tightly control who can delete or force pushes to the branch, and they can also be used to require that pushes meet certain criteria before merging. To ensure malicious code doesn’t make its way into your build pipeline or production environment, it’s also best practice to require multiple approvers for code reviews and to enforce signed commits. Requiring multiple approvers makes it harder for a bad actor to merge their own code and also ensures that multiple people are aware of what is being merged. Verifying that commits have come from a trusted source and not from an impersonating identity can be done using a GPG key and by using branch protection rules.

Overly privileged CI/CD access

To harden your CI/CD pipeline, it’s important to minimize overly privileged access, as a rogue actor or misplaced credential can lead to the wrong person gaining access to your pipeline. To enforce the principle of least privilege in your CI/CD pipeline, you need to not only restrict access to only the right users, but you also need to only allow access at the right time. This is a pervasive industry challenge of balancing security with the operational ease of pushing code fast.

Auditing your IAM policies for unused permissions for individuals accessing CI/CD host environments is a good place to start with right-sizing. There are also several ways to restrict when specific individuals can access and deploy to environments using CI provider policies or policy-as-code solutions. However you approach this challenge, automation is the only way to minimize risk while allowing CI/CD host environment access to the people that need it. That said, it’s generally a best practice to prevent individuals from deploying directly and instead use CD systems to programmatically validate and deploy secure code without human intervention.

Missing protection against code injection and poisoning

Your next line of CI pipeline defense is protection against malicious code or command injection. There are several ways malicious code or command injection can happen, and the more complex your pipeline is, the more opportunities there may be for a bad actor to get creative.

Here are some common weaknesses that can lead to malicious injections:

  • Allowing unsecure commands
  • Allowing risky references to variables that are controlled by third parties
  • Allowing the use of deprecated commands/beta features
  • Allowing network call commands

To protect against these suspicious behaviors, start by configuring your CI configuration file to prevent them. And to continuously ensure that your configuration files do have these protections, make sure to enable automated scanning against CI/CD policies.

Test environments vulnerable to takeover

Automated testing is one of the core use cases for CI/CD pipelines. By adding functional and security testing checks as build steps, you can validate your code to ensure it meets your standards and is ready to be deployed. For many types of testing, you may need to spin up test environments or pods that, without the proper controls, can be hijacked for nefarious purposes. Additionally, using arbitrary and vulnerable images to execute builds and testing opens them up to poisoning and attacks.
Even if an attacker were to gain access to a build system isolated to a container, it may be possible to pivot via network takeover or work upstream to compromise the code within the repo. To prevent these types of takeovers, you should isolate test environments from hosts and ensure pods used for testing are not privileged.

Credentials exposure and exfiltration

Lastly, one of the most common CI/CD weaknesses is the exposure of secrets. As Unit 42’s Cloud Threat Report found, having hardcoded credentials enables lateral movement and CI/CD pipeline poisoning. Allowing specific combinations of commands with secrets in pipelines allows bad actors to exfiltrate those secrets to domains they own. Blocking suspicious use of commands such as curl will prevent that exfiltration.

Similarly, IP addresses may be leveraged to establish a connection to an external computer or server that can then be used as a backdoor. Blocking suspicious netcat with IP addresses is a great way to block that connection for data exfiltration.

···

Leveraging VCSs and CI/CD pipelines are industry standards by now. They help engineering teams build, update, and fix modular application components quickly. Their default configurations are not, however, built with security in mind. That, plus the fact that they are at the center of the software supply chain, makes them an easy target for hackers. To safeguard their software supply chains, organizations should take a preventative, defense-in-depth approach to following VCS and CI/CD security best practices and leveraging policy-as-code to enforce best practices over time.

Interested in learning more best practices to secure your software supply chain? Check out our Supply Chain Security Checklist!