In the film Moneyball, based on the bookof the same name by Michael Lewis, the Oakland A’s Major League baseball team had a tough task: assemble a competitive team with one of the lowest budgets in the league. Desperate times call for desperate measures, and in this case, the measures were a new set of metrics. General Manager Billy Beane focused not on the typical player performance statistics but on the less-flashy metrics like slugging and on-base percentages that exert greater influence on how many runs are scored – because that is what wins games.

Many were reluctant to believe that this new data set could accurately predict performance, but the data proved to be correct. The A’s spent that season challenging the American League record for consecutive wins. By keeping their eyes squarely focused on the real problem – protecting and safeguarding their franchise’s future – the A’s used simple, meaningful metrics to manage risk, guide their operating and decision-making practices, and strengthen their brand.

Metrics Hit a Home Run

This example from baseball provides many insights into the usefulness of metrics. Good metrics have three primary attributes: consistency, cost-effectiveness and significance. A different kind of team altogether, the Cisco® Information Security (Infosec) team, applies similar fundamentals to protect Cisco IT infrastructure against attacks. One of its key governance programs, Unified Security Metrics (USM), is part of a broader CIO initiative called the Pervasive Security Accelerator (PSA).

USM’s mission includes measuring the security posture of an IT service over time, promoting continuous improvement, and providing a quarterly two-way reporting, feedback mechanism to IT service owners and leaders. Increased visibility of these security indicators provides critical system vulnerability intelligence, which can be used for preventative or prescriptive remediation; risk management and security posture assessment; improved security hygiene; and operational/business decision-making activities. More importantly, the introduction of USM represents a paradigm shift at Cisco. Security issues are now handled much more strategically than reactively, and they give organizations like IT expanded operational control and flexibility in managing their security investments, actions and processes.

As Moneyball demonstrates, meaningful metrics have the ability to solve real business problems and transform an organization. In addition, our baseball example proves that metrics do not need to be sophisticated to be meaningful. But they do need to be measured properly. The policies we use for ensuring hygiene – patching systems, building security in and managing vulnerabilities – have existed for many years. However, when we first started measuring these existing activities, very few teams were doing it well. Today, with enhanced measurement and reporting activities through USM, we’ve improved our own vulnerability on-time closure rate by 70 percent, which shows that expanded visibility motivates people to do their part.

Higher-value actionable business metrics and decision-making capabilities: that is what USM creates by combining multiple sources of individual data. These outcomes protect Cisco’s business processes, data, operational integrity and brand from attacks. For us, that’s a home run.

Measuring for Improvement

Metrics are a link in the chain of better business processes. H. James Harrington, noted author of Business Process Improvement, wrote, “Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.” Good idea, but where do you start? How do you mine data through the use of metrics in order to provide greater insight into your organization’s security posture, while simultaneously using it as a vehicle to protect your most critical assets?

There are myriad statistical data sources, including IT system logs and dashboards, for Infosec’s USM team to mine information from. In fact, early research conducted by the team identified 30 different types of meaningful data to track. Comprehensive, yes, but not feasible or sustainable to implement long-term across Cisco. The USM team’s solution centered on the primary outcomes they were trying to achieve – namely, driving security process improvement behaviors and actions within IT. Subsequently, the list was narrowed down to five key measurements:

• Anti-malware compliance:quantifies whether malware protection software has been properly installed and is up-to-date
• Stack compliance:measures vulnerabilities found on the TCP/IP stack (i.e., network devices, operating systems, application servers, middleware, etc.)
• Design exceptions:measures the total number of open security exceptions, based on deviations from established security standards and best practices
• Baseline application vulnerability assessment:computes whether automatic vulnerability system scans have been performed in accordance with Cisco policy and, if post-scan, any open security weaknesses remain
• Deep application vulnerability assessment:computes whether penetration testing has been performed on our most business-critical applications in accordance with Cisco policy and, if post-testing, any open security weaknesses remain

Numerous benefits came from using these measurements. All were readily available, provided good quality data, and could be easily collected and correlated to existing IT service delivery success factors. A great starting point, yet how do you translate these measurements into meaningful security metrics? For USM, the data output from these baseline measurements were used to calculate two critical security metrics: (1) vulnerability, which reveals how many vulnerabilities exist in my service, and how many are infrastructure versus application related; and (2) on-time closure, which answers the question, “Are vulnerabilities closed and compliant with the team’s given Service Level Agreement?”

IT service owners were not fully convinced that these security metrics would yield quantifiable information during the early rollout phase of this program. However, when USM discovered that only 15 percent of vulnerabilities were actually closed on time, leaving Cisco exposed, IT service owners stepped up and managed to raise the rate to 85 percent within a year.

The program overcame some initial difficulties to provide real value to IT service owners. One year later, these same individuals now routinely use these metrics as part of their executive review process. Before the USM program launch, there was not much visibility into InfoSec’s security posture. IT service owners and executives incorrectly assumed that their IT systems were secure. However, USM now gives them more confidence, understanding and insight about what is actually going on within the enterprise. This enables quick diagnosis and remediation of current security issues and those to come.

(This is the first installment in a two-part series on the benefits of leveraging unified security metrics to improve responsiveness and reduce vulnerabilities across the enterprise. The second installment, on putting the metrics to use, can be read next month in the October 14 SecurityE-Newsletter).