Buildkite, meet Bridgecrew!

Making infrastructure security simple and accessible is a top priority for us at Bridgecrew and supporting as many developer tools as possible is a massive part of that.

As a dev advocate, I love trying out new integrations, so jumped at the opportunity to spend some time getting to grips with Buildkite, the “bring your own workers” CI/CD solution and “best-kept secret in the dev tool space.”

In this tutorial, I’ll show how easy it is to set up a continuous infrastructure as code scanning workflow using CloudFormation, Buildkite, and Bridgecrew.

First things first

You’ll need a Bridgecrew account and a Buildkite account. (They both begin with a “B,” and neither of them requires a credit card to sign up!)

Bridgecrew is free to use for up to to 100 scanned cloud resources across any supported cloud or code repository. Buildkite also offers a free plan for open source projects, a two-week trial for private repos, and $15/mo per user after that.

For this project, I’m using a fork of our CfnGoat project, our vulnerable-by-design training project for CloudFormation. This will allow us to see some misconfigurations right away and explore mitigations.

Create your new Buildkite project as follows:

Follow the Buildkite instructions to enable webhooks from your chosen GitHub repository and authorize GitHub access if you’ve chosen a private repo.

Running the Buildkite agent

Next, retrieve and configure the Bridgecrew API key which will allow our Buildkite pipeline to interact with Bridgecrew. In the Bridgecrew platform, select Integrations > API Token.

To keep your key secret, add it to your Buildkite agent as an environment variable rather than storing it in the pipeline or on the Buildkite site. This way, it never leaves the machine you’re running the Buildkite agent(s) on. A more production-ready approach would be a secret store, such as a vault for all secrets used with Buildkite, but a local environment variable passed to the agent will be fine for our purposes.

I’m running the docker agent for simplicity on my local Mac. I’ve taken the “quickstart” docker command from the Buildkite site and added our Bridgecrew API key, alongside the existing BUILDKITE_AGENT_TOKEN:

docker run -v /var/run/docker.sock:/var/run/docker.sock -e BUILDKITE_AGENT_TOKEN="your-buildkite-api-token" -e BC_API_KEY="your-bridgecrew-api-token" buildkite/agent

Now for the fun part!

We configured our simple Buildkite pipeline above to read its config from the Git repo (with the buildkite-agent pipeline upload command), which is where we want our pipeline config, version-controlled, and auditable. It’s worth noting that you could take our sample pipeline below and provide it directly in the pipeline config on the Buildkite site, replacing buildkite-agent pipeline upload command, or even do something like this to pull in the pipeline config from somewhere else entirely:

steps:
- label: "Pipeline upload"
command: curl https://raw.githubusercontent.com/metahertz/terragoat/buildkite-bridgecrew/buildkite.yaml | buildkite-agent pipeline upload

Adding our pipeline config to the repo

In your Git repo, add a buildkite.yaml file in the base of the repo. This is what the buildkite-agent pipeline upload command will import to run our CI pipeline.

The contents should be as follows:

steps:
  - label: "Setup Bridgecrew.cloud agent"
    command: 
      - apk add python3
      - git clone https://github.com/bridgecrewio/bridgecrew-action.git
      - python3 -m pip install -r bridgecrew-action/requirements.txt
      - cp bridgecrew-action/bridgecrew-problem-matcher.json /usr/local/lib/bridgecrew-problem-matcher.json

  - label: "Extract repo details"
    command:
        - echo ${BUILDKITE_REPO} | sed -nr  's/^(https|git)(:\/\/|@)([^\/:]+)[\/:]([^\/:]+)\/(.+).git$$/\4\/\5/p' | buildkite-agent meta-data set "repoDetails"

  - label: "Bridgecrew.cloud IaC Scan" 
    command:
        - export GIT_REPO=`buildkite-agent meta-data get "repoDetails"`
        - echo "Git Repository $${GIT_REPO}. Remote; ${BUILDKITE_REPO}"
        - if [[ ! -v BC_API_KEY ]]; then echo "FAIL No Bridgecrew API key, cannot run IaC scan! on ${GIT_REPO}"; fi
        - if [[ -v BC_API_KEY ]]; then bridgecrew --quiet --bc-api-key ${BC_API_KEY} --branch ${BUILDKITE_BRANCH} --repo-id $${GIT_REPO} -d ${BUILDKITE_BUILD_CHECKOUT_PATH} ; fi

Go ahead and git add, git commit, and git push to trigger your first Buildkite build.

The pipeline is very simple:

• We download dependencies for the bridgecrew-agent, such as Python and Python modules, followed by the agent itself.
• We then extracting the orgname/reponame using some regex for the Bridgecrew agent as the built-in environment variables in Buildkite only provide us with the full git clone string as $BUILDKITE_REPO.
• Next, we check if we have a Bridgecrew API key and run the agent to start scanning for IaC misconfigurations via Bridgecrew.

Success! 🚀

We should now see a new failing build in Buildkite since our CfnGoat repository is full of IaC misconfigurations (by design). Let’s explore the identified misconfigs in the Bridgecrew platform!

Reviewing, filtering, and prioritizing errors

On the Incidents page, you’ll see all the errors from your scan listed with the latest date stamp and the cloud resources affected.

You can filter incidents by repository, severity, type, and a number of other options.

Filtering down to just critical errors, we see these high-priority policies that aren’t compliant in our CfnGoat project:

• Ensure the bucket ACL does not grant ‘Everyone’ READ permission (list S3 objects)
• Ensure rotation for customer created CMKs is enabled
• Ensure IAM policies are attached only to groups or roles
• Ensure no security groups allow ingress from 0.0.0.0.0 to port 22

Under the Errors tab in the right panel, you can drill into the impacted resources to see which file and lines of code need attention:

The Guidelines tab provides further context and helpful information for correcting misconfigurations:

Using this guidance, let’s implement some fixes.

Fixing misconfigurations

Starting with the flagged critical errors, let’s update our CloudFormation manifests and commit those changes to trigger a new Buildkite build.

Bridgecrew will scan our repo again, and our new build will fail until there are zero identified violations. This allows for misconfigurations to block further parts of your CI/CD pipeline if you wish.

Heading back to the platform, we can see that we have no outstanding critical issues left.

When we update the filter to show all errors, we can see we have one issue left—we’re missing secrets encryption in an EKS cluster configuration:

Let’s remediate this simple misconfiguration together!

Update your CloudFormation manifest to enable encryption of secrets within the EKS cluster and commit the changes.

git add eks.yaml
git commit -m "Fix Policy ID: BC_AWS_KUBERNETES_3 Violation"
git push

Because there are no further violations in the repository, our new build goes green.

A happy pipeline is a green pipeline. ✅

Conclusion

Running Bridgecrew as part of your CI/CD pipeline enables you to get continuous security and compliance scanning. In addition to Buildkite, Bridgecrew supports many popular CI/CD providers and version control systems. To see our GitHub integration in more detail, including PR checks, check out our super simple CfNGoat demo video.

Want to discuss anything in this post? Come and join our CodifiedSecurity community on Slack or tweet at me.