Automate AWS Config compliance fixes with Yor + Slack

Note: Yor is an open-source tool that automatically tags infrastructure as code (IaC) templates with GitOps friendly attribution and ownership details. These unique IDs get carried across to cloud resources.

We recently talked about the impact of consistent and strategic tagging when using Bridgecrew’s open-source project Yor. With Yor, you can leverage these tags to trace back cloud misconfigurations found both with AWS Config and the Bridgecrew platform to the original commit in git. You can then resolve who owned the code and the best fix location to reduce your mean time to remediate (MTTR).

What if you could automate this process using a combination of native AWS controls and a popular notification system like Slack?

In this blog, we’ll show you how to use Yor’s tagging in tandem with AWS Config rules, IAM roles, Cloudwatch Events, and AWS Lambda to create rich, actionable Slack notifications when resource misconfigurations arise in runtime.

Here is a diagram of what we’ll have when we are done:

flow chart of Yor with AWS

This blog assumes Yor is being used to tag IaC resources. Please check out our Yor GitHub Action to automate the application of Yor tags to your IaC repository.

Setting up the workflow

We begin with a newly checked-in IaC Terraform S3 bucket which now has the Yor tags as seen below. We’ll use this template to deploy an S3 bucket in AWS.

screenshot of s3 bucket

With our Terraform resources in our repository successfully tagged and subsequently deployed to our cloud, we can start setting up our workflow.

Below are the steps required:

  1. Create and configure an incoming webhook in your Slack workspace.
  2. Set up rules in AWS Config to monitor resources for compliance violations. In this case, we will use built-in rules for Amazon S3 bucket resources.
  3. Create an IAM role and policy that grants a Lambda function permissions to use the resource group API.
  4. Create a Lambda function that uses the IAM role to review S3 bucket policies, determine attribution.
  5. Use a Slack incoming webhook to notify a team of out-of-compliance policies.
  6. Create and configure a CloudWatch Rule to trigger our Lambda function when our AWS Config rule detects the misconfiguration.
  7. Test our setup by forcing AWS to re-evaluate our rule.

Ready? Let’s start by configuring Slack for an incoming webhook.

 

1. Configuring Slack for an incoming webhook

In your Slack workspace, select the top-left dropdown menu for the workspace configuration and select Manage apps.

screenshot of slack manage apps

Select Custom integrations. If you already have Incoming Webhooks added as an App, we’re already on our way. If you don’t, you can use the search above to Search App Directory for “WebHook” and select Add to Slack.

screenshot of slack app directory

Once added, you’re provided with instructions on how to test the webhook with simple curl commands to ensure it’s working properly. What we need to know for later when setting up our Lambda is:

  • The channel that will receive our notifications
  • The webhook URL

Now, provide a description for internal use.

screenshot of description for internal use

Choose a custom name for those who will see the notification (pick a sensible one as this will appear as the author for the message).

screenshot of slack custom name

Now, save your settings and let’s head over to AWS.

2. Creating a Rule in AWS Config

First, sign into the AWS Management Console and open the AWS Config console. In this tutorial, we are assuming you already have some experience with AWS Config and have an S3 bucket already chosen to store configuration history, etc. If not, please start with this guide.

From the menu on the left, select Rules and then select the Add rule button on the right.

screenshot of aws config

Specify a rule type of Add AWS managed rule, as there are plenty to get started with. For this tutorial, we will type s3-default into the search, which should reveal the rule we’re going to use. This rule checks for default S3 bucket encryption using the AWS Key Management Service. You might notice that our example S3 bucket Terraform does not comply with this rule.

screenshot of aws config specify rule type

Now, select that rule and click Next. The default settings for the Trigger type will be fine and should be set to trigger when any changes are made to the resource. The resource scope of AWS S3 Bucket should already be set.

Select Next, review the settings, and then click Add Rule.

3. Create an IAM Role and Policy

We’ll be using a specific API (the resource group’s tagging api) to get the details (more specifically, our Yor tags) of the non-compliant resource. We will need to create an IAM role to allow us to have access to resource tags and permissions generally associated with Lambda functions.

From the AWS console services, select IAM and from the left menu select Roles. Then click on the blue Create Role button. You’ll be presented with a screen with the AWS service already selected. You should see Lambda as an option under “Common use cases.” Select this and then click Next: Permissions.

screenshot of aws config create role

From the list of available permission policies, add AWSLambdaBasicExecutionRole and AWSResourceGroupsReadOnlyAccess and then click Next: Tags. You can optionally add tags or continue to Next: Review and then Create Role.

4. Create a Lambda function

In our GitHub repository for this blog, you can find the Python code for a Lambda function you can already use and modify.

From the AWS console select the Lambda service and then click Create function in the top-right corner.

  1. Choose Author from scratch
  2. Give your function a name like `YorSlack-Noncompliance`
  3. Select the Python 3.8 runtime

screenshot of aws config

  1. Expand Change default execution role
  2. Select Use an existing role
  3. Search for the Role you just created in the dropdown
  4. Click the Create function in the bottom right corner

screenshot of aws config

In the Code source window presented, double click on the `lamba_function.py` and replace the default “Hello World” code with the code sample from the GitHub repository. This function will import the Yor tags from compliance violating resources, such as our S3 bucket, using the AWS API and formats a message with those tags to be sent as a webhook to the Slack endpoint. The editor saves changes to the code automatically, however, you’ll need to click Deploy for it to be productionised.

screenshot of aws

5. Use a Slack incoming webhook to notify a team of out-of-compliance policies

Almost there! You’ll need to add three environment variables specific to the Slack Incoming Webhook we created in our first step. To do this, select Configuration, which is the fourth item in the menu about the Code source window. The fifth item down on the left menu will be Environment Variables. Choose this and click Edit to add:

Key: SLACK_WEBHOOK_URL  Value:  <your webhook URL>
Key: SLACK_USER  Value:  Bridgecrew
Key: SLACK_CHANNEL Value: <the webhook slack channel>

Ok great! We just need to create our CloudWatch trigger and we can test it out.

6. Create and configure a CloudWatch Trigger

From the AWS console select the CloudWatch service, select Event / Rules from the left menu and then click Create rule.

screenshot of aws

      1. Create rule under Event Source, select the dropdown list, and click Build custom event pattern
      2. Copy the following pattern and paste it into the text box
    {
      "source": [
        "aws.config"
      ],
      "detail": {
        "requestParameters": {
          "evaluations": {
            "complianceType": [
              "NON_COMPLIANT"
            ]
          }
        },
        "additionalEventData": {
          "managedRuleIdentifier": [
            "S3_DEFAULT_ENCRYPTION_KMS",
          ]
        }
      }

    Note: The reason for creating a custom pattern is to match the CloudWatch event data pattern which creates a chicken or egg scenario. Without seeing the schema in advance it can be difficult to create a rule. We’ve saved you the time above. To expand this to other AWS Config rules is simply a case of adding to the managedRuleIdentifier section, for example EC2_INSTANCE_DETAILED_MONITORING_ENABLED

        1. Click on the Add target in the Targets area to the right of the Event Source
        2. Pick Lambda function from the main dropdown menu, which should present a list of functions
        3. Pick our new Lambda function from the list
        4. Expand Configure input and make sure Matched event is selected

    screenshot of aws

        1. Scroll to the bottom of the page and click the blue button Configure details
        2. Give the Rule a name and description and click Create Rule
        3. Go back to our Lambda and verify that the Trigger section is now configured

    screenshot of aws

    7. Test the setup

    We’ve done it! An AWS Config rule to detect noncompliance is now connected via CloudWatch to our Lambda. Let’s test it out.

    Go to AWS Config and click on Rules again from the left menu. Select our new rule, and from the Action menu dropdown, select Re-evaluate. This will force AWS Config to recheck out resources that may already be noncompliant to trigger our new notification process.

    Let’s check our Slack channel for messages:

    screenshot of slack message

    Huzzah! The notifications have started coming in. From our Slack message, we can see which resource is non-compliant by type and the specific resource id. We can see who modified it last and see that Slack automatically tagged that individual, as the email address used for the git commit was the same as the one used for Slack.

    From this message, we can click on Trace on GitHub to jump directly to the commit of that resource. We also have the option to view the entire commit and jump directly to that resource within AWS Config. Now the operations team’s investigation time of a misconfiguration has been significantly reduced to a click of a button.

    video of slack notification setup

    Looking into our  Lambda function code, see how we have enriched our Slack message using Yor tags to provide the developer who last modified that resource from the git_last_modified_by tag, a last modified date from the git_last_modified_at tag. Not to mention, we have leveraged the yor_trace tag to create a Trace on GitHub button for us to jump directly to the specific git commit and resource.

    Conclusion

    In this blog, we saw how we can make Yor tag and trace a part of our compliance remediation workflow through a few simple connecting of our AWS dots. Using AWS Config to trigger a CloudWatch Event to trigger an AWS Lambda function, we were able to create an enriched triaging message (thanks to Yor tags) in Slack which highlights the Slack user, the git commit, and the resource in a single stroke of DevSecOps automation.

    If you found this post helpful, you’ll probably like these other Yor blog posts: