Understanding the alert budget: A security team’s second ledger

What is an alert budget?

An alert budget is a site reliability engineering (SRE) concept used to measure the volume of alerts (measured by combining the number of alerts and the complexity of their handling) that an organization can effectively handle. If you know what your budget is, you can make better decisions about whether you can effectively handle adding new types of alerts or detecting issues with additional systems, as well as what tools to invest in to help you manage your alerts. 

We need tools that effectively manage alert budgets for two reasons. First, security teams have limited time. Second, there are more alerts than we can look at manually. Once a budget is properly managed it’s possible to build efficiency into teams, processes and tools. 

Time is money and money is time 

Security teams have a limited amount of time to handle alerts, so we can think of seconds and minutes as their money. For example, a foobar alert might eat up 5 minutes to handle, but a bazbop alert might take 30 seconds; handling 10 foobar alerts and 60 bazbop alerts eats up 80 minutes. Adding up this workload can be tricky, because the time in question comes from both the security and system owner’s budget. How busy those teams are varies from company to company and can be taken into account in a more complex version of this budget detailing these considerations. 

Given these guidelines, security teams apply alert budgets like financial budgets to buy a specific resource: incidents. “Buying” incidents means that your security team is finding incidents, something you want to do as quickly as possible to minimize use of the limited time. 

Not every team has to track their alert budget explicitly in time. Some can track it through the number of alerts or another proxy metric, but the idea is the same: you need to have enough budget to cover your alerts. As a CISO, you want to keep alert budgets as low as possible and efficiently buy the incidents and risk reduction, so your team can either spend that remaining time or reduce the cost of the security team.

Tools to help you optimize your alert budget

You can improve your alert budget in two key ways:

Better signal

If a team needs to handle a smaller number of alerts to “buy” the same outcome, it allows for better budget use. High-signal, low-noise — attempts to send meaningful alerts should thus be a priority. 

Make alerts easier

Making alerts faster to handle increases efficiency. There are a few ways to do this, including:

• Include the relevant context in the alert. If the security person needs to take the alert and look up the necessary context to decide if it’s a problem, they’ve used some of their time budget. Ideally the security person should be able to immediately differentiate between high- and low-priority alerts directly within the notification. From there, the path to more information should be clear.
• Make the alert easy to read. Clarity is always a challenge within the context of security, but it’s key to providing teams with the tools they need to do their jobs. This doesn’t necessarily mean a longer alert — but it does mean a more effective one.
• Build effective relationships between security analysts and systems teams. When teams that manage a system know that security analysts quickly need help it’s possible to cut down on extra communication cycles. Teams should use whatever collaboration tools are already in their arsenal — do not change ticketing systems — and that they’re investing in building interpersonal relationships. Organizations can do this by supporting security team outings, sending swag and more. 

Managing your alert budget is supporting your security team

I once was on a team that wanted to hire more security people than the volume of experts that literally existed in the world at the time. The field of cybersecurity is severely understaffed, so I know from experience that time limitations are a very real issue. Proactively managing an alert budget — a robust system where time and alerts are tracked — can make or break success for security teams (which are often made up of scrappy experts forced to get creative with their problem-solving skills no matter how well-resourced their companies are). We need to support those teams to avoid burnout. Luckily, managing your alert budget with better signal and easier alerts can ensure that your team is still operating at full capacity — and not losing their mind in the process.