Last week we sat down (virtually) with some of our friends who happen to be among the industry’s leading experts in security. Hosted by Bridgecrew Co-founder and CEO Idan Tendler, the discussion centered around the strategies and tips for building better relationships between engineering and security teams. The panelists included:
- Ely Kahn, Principal Product Manager @ AWS
- Caleb Sima, VP of Security @ Databricks
- Karthik Rangarajan, Director of Engineering, Security, and Privacy @ Robinhood
Our goal in holding this discussion was to give security professionals insight into ways to make security more efficient and effective by working with engineering and DevOps teams. Watch the full recording below, or keep reading for some highlights from our panelists.
What is the biggest challenge facing security and engineering DevOps?
Caleb: Ultimately, I think there are two different priorities, and being aligned on one priority is difficult. Engineering’s number one focus is building and delivering product in order to enable sales. Although security also wants to support business growth, its top priority is managing and reducing risk, which has been predominantly more conservative, less risk-taking. Enabling and delivering product is the opposite, so you have a fundamental pull between goals. Fundamentally, that’s what creates that tension.
Ely: I think the biggest challenge in getting alignment is having support from leadership at the top level, coming all the way down from the CEO. You need to be able to deliver security in an agile, flexible, fast manner so that you don’t slow down your development teams. But there needs to be a very clear agreement that security is a top priority. Once you have top-level leadership buy-in, the next challenge is building out the mechanisms to operationalize efficient and scalable security, because it’s not going to happen on its own.
Karthik: Friction generally comes from conflicting interests—DevOps has pressure to get features out while security often ends up blocking. Security doesn’t typically say, ‘Here’s how we can do it better.’ It says, ‘Don’t do it that way’ and that isn’t always helpful. Not providing actionable alternatives only gives rise to further friction.
How do you convince your organization that security is a priority?
Karthik: At Robinhood, we care so deeply for the customer of our customers and company. But I know that that isn’t always the case. To present security as a priority, you often have to present it as a business risk. By asking, “By not doing X, these are the risks you’re accepting for your business. Is that something you’re okay with?” Talking through the best case, worst case, and in-between options is the best way to get on the same page.
Caleb: Practically speaking, security’s level of priority depends on what kind of company you are, the stage of company you are, and the market you’re in. For example, companies in financial sectors naturally have a lot to lose, and security is naturally going to be a higher level of priority. It will be easier to spend security budget. In others, compliance is the only forcing function.
If there is no driver, you may have to convince leadership that security can be a business enabler or a revenue generator. As a security operator, you have to know what kind of company you’re in and go from there.
Ely: The first step is to work with your leadership team to understand the top risks facing your business more broadly and understand where security fits in.
How do you work with engineers to make them engaged with security?
Ely: This is where AWS does some absolutely amazing things. We’ve built out lots of mechanisms and automated processes to ensure that security remains a top priority when developing and delivering new services and features.
For example, if an engineering point of contact believes a feature doesn’t need a security review, that person must file a ticket and get approval from the security team. So, in reality, every feature that AWS launches gets a security review. Those guardrails eliminate choice.
Caleb: Having a fully-resourced security team and executive buy-in, declaring that no feature gets deployed without a security review, is the vision. But in a fast-growing company, the security team doesn’t usually have these resources, and they’re definitely not as mature in the organization. There is no edict from the top, so you have to figure out how to manage top risks with what you have.
The reality is, our small security team doesn’t have the bandwidth to review every single new feature at an architecture level, code level, and pen-testing level. This forces us to choose the important features that touch things like customer data or auth. We also have to find ways to automate security and focus on what progress we can feasibly make. Some ways we’ve done that is by prioritizing secure design review of new features, threat modeling, and creating security champions internally.
Karthik: We deal with a similar situation where there’s no way you can review every single thing that shipped. So one of the moves I made was building an engineering team inside security that handles anything with security risk. Doing that has proved that we know what we’re doing and has enabled us to work closely with engineering. We still run into challenges related to who owns what, but it helps alleviate much of the friction.
One of my engineers calls this the umbrella versus the storm. Most times, you’re going to be the umbrella. But a lot of times, you’ve got to be the storm—speaking up and saying “no.” The key to successfully being the storm is to be the umbrella more often than not.
How do you balance who owns what between engineering and security teams?
Ely: I think there is a huge trend happening right now around the decentralization of security from centralized security teams down to DevOps engineers that are actually responsible for the cloud infrastructure that they’re building on. At AWS, we like to limit cross-team dependencies and not depend on other teams’ infrastructure. That can cause friction in terms of our agile release processes.
Inside AWS, every AWS account owner is ultimately responsible for the security inside their accounts, and that’s operationalized in a lot of interesting ways. In addition to our continuous automated security checks and robust threat modeling tooling, we try to foster transparency and create a culture of making sure security is a priority.
Caleb: The struggle is convincing the top that security standards and controls need to be enforceable. Everything is driven by revenue. Even though we have the right processes in place, we need to get to the right maturity where they’re fully enforced. In the meantime, you patch the right level of priority and focus on things like detection and response, so we don’t slow engineering down.
Is shifting security left going to be the trend?
Karthik: Eventually, we want to get to a point where engineering teams own their vulnerabilities and security issues. But to do that successfully in a fast-growing company, you have to be successful the first time you do it. If you don’t, you may not lose credibility, but you will lose capital. And as a security person, there’s only so much capital you can lose.
Security should be a check and balance where on the one hand, we constantly assess risk and get buy-in to address those areas, and on the other, we get into the trenches with the engineers to create trust.
Caleb: The “paved road process” is a great way to not slow deployments out, but in a quickly changing environment, it’s hard to set the models in the first place.
Karthik: The way I think about it is the paved road is the path of least resistance. If you go through the jungle, you’ll be met with more control-based security measures that will end up slowing you down.
Ely: With the paved road process, we try to make it so that engaging with security earlier, you can actually get products out fastest. But you have to prioritize security from the top down for the paved road to be upheld.
What’s the KPI to know how security and engineering are working together?
Caleb: There are several hard metrics we track for security, but gauging the relationship is challenging. One way we did it was through an NPS survey. That gives us a unique data point to track and improve the relationship over time.
Ultimately, having the right people is everything. Building trust from a technical capability and getting leadership to hold engineering accountable is also absolutely critical. Even if you have an amazing engineering team, you’re still always fighting upstream until you can get the people at the top to agree that there is a watering line that all teams must appeal to.
Ely: As a security leader, I think you’re there to raise the level of security in a customer-obsessed way. I would veer more toward time-to-live metrics for highly critical vulnerabilities, etc. Then get leadership buy-in and ownership that these are a priority and drive performance against them.
Karthik: Build a strong engineering team. If you are at an early stage company, if you have a high-growth company, you’re going to have more mileage bringing in people that are strong engineers, that can roll up their sleeves and get things done.
Idan: My biggest takeaway from this discussion is the need for trust and how to create it. We will always have this tension between engineering and security. It’s a huge dilemma, and it will not be solved today. Security leaders must strive to empower engineering to work with security more smoothly. It’s the secret sauce for how companies can walk faster, move faster, and build faster.
We heard a lot of different perspectives and insights during this panel, more than a write-up can capture. To drill into all the highlights, watch the full recording and feel free to reach out to us with your thoughts.
Thanks to our panelists and to those who joined in on the discussion! Be sure to follow us on Twitter, LinkedIn, and Twitch to be invited to future virtual events.