The signs are all around that Big Data is the first major post-cloud technology that can change our useof data as much as the cloud has changed the way we deploy it.

Industry analyst IDC, for example, predicts that Big Data will grow from only $3.2B in 2010 to $16.9B in 2015, for a compound annual growth rate of 40 percent. While servers and software account for significant portions of this growth, storage alone will grow 61 percent per year over this period — a familiar benchmark for anyone who has been following the explosive growth of data generated by IP surveillance video.

Using a more technical measure another analyst, CSC, predicts that by 2020, we will see a 4,300 percent increase in the rate of annual data generation. The Web itself is a huge data generator, with every click literally generating more information about who used the Internet and how. But the “Internet of Things” (IOT) is also now poised to become an equally large data contributor, as more and more smart devices become part of our lives and report back to one database or another on interactions in the physical world.

With respect to IOT, electronic security has for once been out in front of the IT industry, as we have been measuring and monitoring interactions with the physical world for many years before IOT came into vogue. Now that Big Data is beginning to unlock the value in all that IOT data, the question is whether the security industry will also move toward exploiting this data with the latest and most powerful tools.

In this three-article series, we will look at the underlying technologies that make up the Big Data revolution, and we’ll examine which of these technologies hold the most promise for improving the way we deliver security to our customers. And by security I mean real security – not just the parts and pieces and wires and cables and servers – real security that improves our customers’ safety and their bottom line.

One Big Trend, Many Technologies

The first thing to recognize about Big Data is that it’s not just one technology, or one type of technology, or even technologies that are all aimed at the same target. It’s really a collection of many confluent technologies that have all made the macroscopic phenomenon of Big Data possible at this point in history. We’ve seen similar technology confluence in the security industry before, for example, once widely available inexpensive CCDs and worldwide Internet connectivity suddenly made IP surveillance far more valuable and effective than its predecessor, and far more valuable than either of those technologies alone.

Now in its second year on the Gartner hype cycle, Big Data is seen to be a multiplicity of technologies that are all either climbing toward the peak of hype, or just slightly past it as they move inexorably through the trough of disillusionment and on to the plateau of productive applications. The core technologies often associated with Big Data that have some relevance in security are summarized in the table. This table is by no means exhaustive, as the tool sets in this space evolve rapidly, and the criteria for inclusion are somewhat in the eye of the beholder.

Big Data and the Cloud

It is difficult to think about Big Data technologies without cloud computing. Due to the scale of Big Data problems, the massive IaaS and SaaS cloud platforms now commercially available by the hour are almost always going to be the most cost-efficient way to try out solutions and run them over the long term. As Raul Chong of IBM states, “The cloud computing model is a perfect match for big data since cloud computing provides unlimited resources on demand.”

            Many of the technologies described above are now available “as a service” through a growing number of startups which can help companies deploy Big Data solutions without having to master all of the underlying technology or build out a massive data center. GoodData, for example, is a new company that provides the entire analytical software stack, servers and storage. You just bring your data. Similarly, Amazon’s new Redshift service promises a petabyte-scale data warehouse at one-tenth the cost of traditional systems.

            These types of services put Big Data within reach of many organizations who would not otherwise consider it. Can they do the same for the security industry?

Cost-Benefit and ROI

            One of the toughest things to understand at this point in the evolution of Big Data is what the ROI might be for any particular application. What makes sense for a financial services institution or a large oil company may not make sense for a physical security application. A million dollar project can be justified when there is $10 million in near-term ROI, but can that same expense be supported by a security system? What if that million dollar expenditure would save ten lives in the next year?

What's in it for the
Security Industry?

            The question of what’s in it for the security industry will force us to look at security systems differently. The small business owner with three doors and a few cameras probably has little use for Big Data. On the other hand, the property manager or retail chain with a thousand locations would clearly benefit from viewing security holistically across the entire enterprise. But if the security systems in those thousand locations are all managed locally, the data can’t be analyzed in the aggregate, and its value is severely limited. Cloud-based systems that can easily span the entire enterprise therefore take on a new significance as Big Data platforms in addition to the many other cost-benefit advantages they offer.

            In the remainder of this series, we’ll take on three central aspects of how Big Data technology can enable a much richer use of the data that our systems are already collecting:

•  Big Data support for real-time decision making

•  Big Data and advanced analytics of security event data

•  Industry norms and benchmarks made possible by Big Data

            What will we find?  If, as the Wall Street Journalsuggests, only 0.5 percent of the digital universe stable is actually being analyzed, then I’m pretty sure there’s a pony somewhere in that pile of the other 99.5 percent.