Intellyx BrainBlog: Don’t break the software supply chain. Secure it.

An Intellyx BrainBlog by Jason English, for Bridgecrew

Supply chain seems to be on the tip of everyone’s tongue these days.

We’re experiencing massive volatility in global supply chains, the multiparty networks that produce the commodities and manufactured products every society depends upon. Inconsistent supplies of raw materials and parts, logistical nightmares, and constrained industrial capacity are interrupting practically everything. Customers and businesses are bearing the brunt of inventory shortages and higher prices.

Just like the physical supply chains of factories, container ships, and trains behind the delivery of goods to customers, there is a parallel software supply chain (or SSC) to deliver cloud-native applications. Most companies didn’t worry enough about the SSC until the recent Solarwinds hack pointed out just how vulnerable our software can be throughout its lifecycle.

Although it was certainly not the first software supply chain attack, the world sat up and took notice because of the breadth and depth of this attack. And this problem isn’t going away anytime soon. The European Union predicted a 4X increase in SSC attacks in 2020, and analysts predict that nearly half of all companies will experience such an exploit by the year 2025. So what can be done to prevent it?

Every layer that comprises a cloud-native application, from root-level operating systems, to application code and the repositories it’s stored in, to the access and control planes used to manage a cloud-hosted application estate, has potential weaknesses that introduce risk within a highly interconnected software supply chain.

What comprises the software supply chain?

Much like the many parts and suppliers that contribute to physical supply chains, software supply chains are made up of the combination of several platforms and components – each of which arrived through their own open-source or commercial production pipelines.

  • Version control systems (VCS) such as GitHub, GitLab, or Bitbucket can govern multiple project repositories to store and manage all of the assets within a given company’s SSC. Unfettered access to these systems can not only allow exploits to enter company systems at a code level, but it can also result in a loss of intellectual property and operational integrity.
  • Infrastructure as code (or IaC) tools such as Terraform, CloudFormation, Chef or Puppet scripts, Kubernetes manifests or Helm charts provide the instructions for configuring all elements of an application’s infrastructure. Misconfigurations of IaC can get persisted again and again across deployments if gone un-remediated pre-deployment.
  • Application code encompasses actual developer work in Python, Go, Node.js, etc. that includes open source libraries and defines functional logic that is executed when the app runs. Any source code in the repo that cannot be tracked back to a known author may be considered unsupported, or worse, injected.
  • Open-source packages and images from GitHub, Docker Hub, Artifact Hub, and many other project sites can include hundreds or thousands of unique OSS downloads and patches that developers have downloaded at some time – but which ones are correct and free from vulnerabilities and malware for use with the application? And when do they need to be updated?
  • CI/CD pipelines such as CircleCI, Jenkins, and GitHub Actions contain orchestration instructions for testing and provisioning infrastructure, staging environments, and deploying software to production. Any tampering or poisoning of automation code could rapidly propagate through several environments with each run.

As many as 40,000 or more open source releases appear in repositories every day, and the average enterprise can download well over 300,000 packages a day, according to a recent software supply chain study. And that’s only the tip of the supply chain security iceberg. Simply eliminating known CVEs, and protecting applications code and interfaces from SQL injection, and/or setting network perimeter access controls are no longer sufficient for securing SSC protection in highly distributed software supply chains.

Each and every element and delivery pipeline that contributes to cloud application delivery can represent a potential supply chain weakness. Further, the complex interdependencies and data handoffs that occur when multiple components are chained together provide leverage points for attackers to cleverly weave in new supply chain threats.

Organizations need to get their arms around the specific ways in which all of these elements are configured and deployed throughout their process of moving software from code to cloud.

To mitigate the risk, de-risk the process

Traditional supply chain optimization (SCO) focuses on improving the visibility of supply, demand, and work-in-process signals in order to better plan out each stage of a production pipeline that is both efficient and more resilient to disruptive changes.

SSC security practices share some similarities with SCO. To prevent costly production interruptions and breaches, they must be integrated within the developer’s flow at every stage to de-risk the cloud software delivery process.

Figure 1. The software supply chain offers a broad threat surface for attacks at the component and configuration level beyond the coding of application logic. Source: Bridgecrew

  • Code: Developers can download insecure software packages and components, and specify misconfigured IaC and container images. Automated and audited scanning and checks of each component or configuration code should be conducted before they progress to later stages.
  • Build: While linting and quality checks are often included as part of build pipelines, more attention should be paid to securing the permissions of version control systems and branch protection rules as well as CI/CD workflow configuration. Without proper rules and enforcement in place, an unauthorized user or irresponsible actor could allow malicious code to be introduced.
  • Deploy: Admission controls and governance policies should prevent insecure images from becoming part of the deployment manifest. Exploits such as data exfiltration, ransomware, and crypto-jacking can be introduced here and move laterally through the organization’s environment.
  • Runtime: This is where the observability aspect of security comes into play, as continuous scanning and threat detection can hopefully detect new vulnerabilities and zero-day attacks before they can be used against production systems.

Are developers becoming the new attack surface?

With the plethora of security tools in the market, SecOps teams can scan application components, cloud environments, and networks for vulnerabilities, misconfigurations, and suspicious activities. But as the pace of development and deployment speeds up, and supply chains get more complex and distributed, is that enough? Who’s looking at the underlying code that built up that whole environment?

It seems obvious that developers would be best equipped to tackle vulnerabilities in the application code they write. Input fields and API calls should be validated to prevent SQL injections or buffer overflow attacks, and secure secrets storage and encryption should be used to prevent data leakage.

As we’ve discussed, there are way more components than application code within SSCs, which presents an ownership challenge. Different product and engineering teams may own VCS repositories and configuration code. Other dev and ops teams own CI/CD pipelines and the integrations and environments coded within them.

We have plenty of useful linting and security research tools available to point out the coding mistakes a developer might make before that next pull request. Development security provider Bridgecrew offers a platform that scrutinizes code and configurations within an end-to-end DevOps process, in order to mitigate risks and automate remediation steps before exploits can be propagated.

Let’s not forget how little code in the supply chain is actually coded from scratch by a developer. Unless they are in the tool-building business, it would be financially impractical for a dev to spend time writing their own Javascript libraries, reinventing containers, or designing custom cloud-native networking protocols and environments, when reusable code and components can be checked out.

As it turns out, one developer’s download is another hacker’s attack vector!

The Intellyx Take

Developers aspire to build nothing but the securest products, but they are often incentivized to deliver new, high-value application functionality first. That’s finally starting to change.

Once, security was mostly about protecting external-facing functionality and network perimeters from unwanted hackers. Now developers are taking a direct interest in securing the software supply chain as a primary consideration, and SecOps teams are gaining deeper insight into how to help developers take stock of the organization’s risks and attack surfaces within every deployment.

The complexity of today’s distributed cloud architectures means that humans will need to lean on security automation and controls at each stage to deliver software that is truly resilient enough to survive constant variability and volatility.

©2022 Intellyx LLC. Intellyx retains editorial control over the content of this document. At the time of writing, Bridgecrew is an Intellyx customer.