A first-hand account of ransomware: To pay or not to pay

A first-hand account of one CISO's response to ransomware

Digital transformation has led to the deployment of a greater number of innovations and applications that generate more and more data. While the world’s collective knowledge and advancement depends on the ongoing aggregation, analysis and distribution of vast amounts of data, the preservation of these digital assets, especially during the pandemic, is at risk from cyberattacks.

Data undergirds the operations of the vast majority of organizations, and is considered a precious commodity that delivers a competitive advantage. Protecting these valuable and irreplaceable digital assets is a key mission of every security and IT department, including ours.

Like many companies, we developed our own best-of-breed business continuity and disaster recovery plans, but never imagined they would be put to the test so dramatically as they were this year. Due to the pandemic, we pivoted in a single day from supporting a large on-premise workforce to sustaining a workforce that was almost 100% remote. This type of unanticipated phenomenon required us to further strengthen our security measures, including closing security gaps, maximizing scalability and future-proofing our systems.

In early May 2020, many of our servers and associated applications suddenly stopped functioning. It took my team mere minutes to realize that unrelated applications were starting to glitch – causing concern because we have automatic recovery. As we raced around checking servers, we saw ransom notes that demanded as much as $3.6 million USD (requested in bitcoin) or lose our corporate data.

My team and I immediately cut the cord between all of our servers and shut them down to mitigate the damage. The next several hours were spent checking the state of our systems and the extent of the damage. Our email server was not compromised and we turned it back on to notify the entire workforce of the breach and advise them about further actions as well as system disruptions.

We placed a call to the FBI cybersecurity team and contacted our insurance company (we have ransomware insurance) who set us up with a security consulting firm. They helped us stop the bleeding and assess where we were.

Our initial assessment, revealed that 90% of the data on our servers was encrypted. In addition, the ransomware encrypted critical system files that rendered some servers completely inoperable.

To our dismay we realized we were 100% down. Any data on production disk was encrypted at the speed of disk. On the other hand, we also rely on tape for backup and disaster recovery which provided an “air gap” from the ransomware. This is also augmented by disk snapshots, many of which escaped being encrypted. We realized that we could have survived an even worse attack or physical disaster (fire, flood), due to the unique attributes of tape storage being removable and off of the network stream. As part of our data recovery strategy, we regularly store copies of all our data on tape, and this gave us the confidence to decide that we would not pay the ransom. We had our data; now it was time to reconstruct.

Our tape copies gave us the peace of mind to advance without second guessing our decisions. In fact, we had tape backups that were 99.9% current from the week before. A huge advantage in our recovery was that nothing was accessed through our VPN. It took us a total of four days, 24 hours a day, to stop the virus from spreading. It took five days to get the company back up and another week to get all of our systems online and another two weeks to fix everything else. In total, out of 600 total servers, we had 150 servers, including virtual machines, that were compromised.

In the end, we overcame the attack, with no data stolen (confirmed by a third-party security audit). While many organizations that fall victim to ransomware attacks opt to pay the ransom, we were able to rely on our cybersecurity defense systems that proved to be successful.

Effective IT security is a balance of culture and strategy. No matter what the level of security that is in place, there is always room for more. At some point, however, those tight measures will start to impact the user experience and possibly derail the goals of the company that are accomplished through sound IT systems.

If ransomware makes it into your organization, and based on recent statistics, there’s a good chance it will, there’s no easy way out. Downtime and added expenses are guaranteed, even in the best scenario. Without a doubt, one of the best lines of defense to protect data from cybercrime is to store copies on an electronically disconnected device. The role of tape-based offline storage offers unique advantages and cannot be underestimated in this scenario due to tape air gap. The air gap prevents cyberattacks from accessing your data since the data stored offline cannot be hacked.

Evaluating and testing your business continuity and disaster recovery plans regularly, ensuring there are multiple copies of data on multiple mediums stored in various locations, and making sure to have cybersecurity experts onboard or close at hand (as we did thanks to our ransomware insurance) will help organizations prevail over a ransomware attack in the future.

As Senior Director of Enterprise Business Solutions at Spectra Logic, Tony Mendoza is responsible for the entire IT infrastructure at Spectra Logic. He oversees hardware, DevOps, software and cloud system integration at the company. Mendoza also oversees the business intelligence department, which is dedicated to collecting and analyzing data to provide decisions makers within Spectra Logic with the right dashboards and tools necessary to improve processes and accelerate growth. He provides overall strategy designed to proactively deliver improved systems and services in partnership with the business. An employee with Spectra Logic for more than two decades, Mendoza’s IT expertise is often tapped for corporate webinars, product viability discussions and customer presentations. Tony has a B.S. in Business Information Systems from Regis University.