What’s the best way to track costs when multiple teams are managing services across dev, stage, and test environments co-located in a single AWS environment? It can be a lot to manage, but there are some really interesting ways to simplify the task when you combine Yor with AWS Cost Explorer. That’s right, the topic for the next section of our Yor use case blog series is cost allocation.
A bit of background on Yor, Bridgecrew’s latest open-source tool. Yor simplifies and automates tagging of cloud resources, in infrastructure as code, enabling a whole host of “code to cloud” tracking, auditing, and security features, such as:
- Easily trace which block of code created which resource.
- Easily trace where in a codebase a cloud resource came from.
- At a glance information for owners and previous editors of a cloud resource.
- Custom logic for adding team, department, and security permissions tags via simple YAML configuration.
Why you can’t rely on manual cost allocation
Most businesses leveraging the cloud at scale will depend on some form of cost analytics, ranging from solutions built into the cloud provider’s platform itself, to third-party solutions offering automated cost reduction and sizing suggestions. To ensure these platforms provide useful data, businesses typically tag resources with information to help group by project, team, business unit, or cost center.
Doing this manually can lead to several issues. First, different teams may use slightly different tagging and naming conventions, preventing the cost analysis from correctly grouping resources. Manual discrepancies in tags like this could be solved by rejecting all but a business-defined set of tags in the CI pipeline. However, instead of putting this burden on the development teams, Yor allows us to automate the addition and maintenance of these tags, sitting exactly where a “check for tags” item would otherwise be inside our CI pipeline.
By default, Yor will automatically add a number of tags to each infrastructure as code resource (such as a Terraform or CloudFormation object) it detects, including information pulled from git, such as the last user to modify the resource, the commit where the last change to the resource occurred, and so on.
However, the magic happens when using Yor with a custom `tag_groups.yaml` file. This allows us to easily implement our own custom tagging strategy, with Yor handling the heavy lifting!
Custom tags YAML schema explained
The Yor configuration file consists of a list of tag groups, where each tag group can contain tags from simple, static strings, or, more interestingly, conditional values based on environment variables or even other tags Yor has already created! In the following example, we’ve created a single tag group for mapping resources to a business owner.
tag_groups: - name: cost # map infrastructure cost center tags: - name: cost_center value: default: external # default cost center matches: - bridgecrew: tags: git_org: # map cost centers by github orgs - bridgecrewio
Here we are defining a cost center of “bridgecrew” if the GitHub organization containing our cloud resource is “bridgecrewio” and defaulting to a cost center of “external” if not.
Running Yor with this config against a sample Terraform resource adds our new tags, one per resource.
We can see they report an external cost center, as the repo is within my own personal GitHub org.
We could build on this using existing tags Yor populated to add extra cost center information for teams or groups of developers:
tag_groups: - name: cost_team tags: - name: cost_center_team value: default: bridgecrew matches: - bridgecrew-devrel: tags: git_modifiers: - matt - steve - name: cost_org tags: - name: cost_center_org value: default: external matches: - bridgecrew: tags: git_org: - bridgecrewio
We can use these tags in combination with cost-management systems, such as AWS Cost Explorer, to monitor cloud spend.
AWS Cost Explorer with Yor tags
We can use these tags in combination with cost-management systems, such as AWS Cost Explorer, to monitor cloud spend.
Imagine a common scenario, where multiple teams, managing multiple services across multiple environments (dev, stage, test) are co-located in a single AWS environment.
Suddenly there is a spike in costs. Having these cost allocation tags allows us to easily narrow down costs per team, project, or environment. This gives us a more useful cross-section of searches in our cost management system of choice.
For example, let’s say we have deployed six new Terraform resources to our AWS account, three from a `bridgecrewio` GitHub repository, and three from a personal repository.
Using AWS Cost Explorer, we can filter by the tags Yor generated to easily allocate costs across departments, teams, or environments and find any anomalies.
Here we see cost explorer in a default configuration, showing all resources for the last 6 months. We have expanded the Tag filter to the right hand side, and can see one of our tags, cost_center_team, selected for filtering.
It is worth noting that tags for use with cost explorer need enabling (once) in the cost explorer settings, in this way they will then be available in the “Tag” filter payne shown.
By clicking apply, we can drill down just to the costs incurred by that team, which in this environment are minimal:
The power of Yor’s default tagging can be extended to enable consistent, customized tags for your own organizational needs. Let’s see how this looks inside our cloud environment, and how these tags will help us identify and control costs.
Extensible API integrations
Yor has also been written with extensibility in mind, to support tagging from other programmatic information sources. GoLang interfaces `ITag` and `ITagGroup` can be implemented within the Yor codebase to expand the default set of “taggers.” This can be used to pull data from internal asset management systems, public cloud object cost API’s (watch this space!), or other business or compliance systems.
For more information on extending Yor in this way, see our documentation and join the conversation in our #CodifiedSecurity Slack channel!
Happy tagging!
If you found this post helpful, you’ll probably like these other Yor blog posts:
