One of the main contributors to the weak security posture of development environments is the complexity and knowledge gap created by the number of tools and services taking part in this process. With more than a hundred continuous integration and continuous deployment (CI/CD) tools to choose from and hundreds of plugins and services connected to those tools, no wonder security teams have a hard time grasping the amount of information and security requirements of these environments.

It is not rare to see a CI/CD pipeline that includes 10 to 20 different tools and services; some of these are cloud-based, others open-source tools with various plugins installed on them. It is impossible to keep track of this complexity manually, and it often results in an exposure of your environment, code, secrets, and network through those tools and plugins’ vulnerabilities.

The DevOps tools’ sprawl continues as more and more companies introduce their DevOps products and services. Development teams take advantage of these new CI/CD tools and services to build their pipelines and enhance the process, but by doing that, they are also increasing their exposure to risks. Add to it the limited collaboration between development and security teams and the lack of visibility and control over these services, and there is no surprise that CISOs and application security managers look puzzled when asked about the security of their CI/CD pipelines.

Codecov hackers continue to inflict damage on major enterprises

The Japanese e-commerce giant Mercari, and project-management tool provider Monday.com, are the latest victim of the Codecov attack, which has already affected hundreds of major companies, including HashiCorp, Confluent, Twilio and Rapid7, and others.

Mercari has stated that the compromised records include financial and personal information from customers and partners, such as:

• 17,085 records related to customer sales occurred between August 5, 2014, and January 20, 2014, containing bank codes, branch codes, account number, account holder (kana), and transfer amount.
• 7,966 records on business partners of “Mercari,” including names, date of birth, affiliation, e-mail address, etc.
• 2,615 employee records, including those working for a Mercari subsidiary. Names of some employees current as of April 2021, company email address, employee ID, telephone number, date of birth, etc.

Just a few days before, Monday.com had disclosed that it was also impacted by the Codecov supply-chain attack and that after their investigation into the Codecov breach, they had found that unauthorized actors had gained access to a read-only copy of their source code.

Minimize risk and complexity, and avoid the pitfalls of supply chain attacks

The recent series of supply chain attacks affected tens of thousands of companies. Nowadays, CI/CD pipelines form the backbone of modern-day DevOps operations, and as we see this trend continue, we cannot ignore the urgency in protecting customer’s development environments from these pervasive attacks.

The complexity and collaborative nature of these environments provide an easy target for attackers, who can take advantage of vulnerabilities and misconfigurations within pipeline plugins and services. By gaining access to the CI/CD pipeline, attackers can hijack your updates, inject malicious code and get a backdoor to your and your customers’ environments.

The latest Codecov attack taught us two alarming facts:

1. Attackers can gain easy access to your most valuable process and data through your pipeline’s many services and plugins which are usually not monitored at all.
2. Those attacks can go unnoticed for months, impacting thousands of companies and inflicting massive damage in the process.

Organizations must take proactive action to secure their software supply chain from such attacks and prevent attackers from using these backdoors in their environment. This requires considering the complexity of the development environments, the various 3rd party plugins and services connected, and the sophisticated nature of today’s supply chain attacks.

Building a strong CI/CD pipeline security posture

Security and DevOps teams need to watch their pipeline dependencies closely to identify and respond to vulnerabilities and attacks against those addons services and tools.

Whenever a new service is connected to your pipeline, these services must be checked and monitored constantly for any vulnerability or suspicious activity. Any suspicion should automatically trigger an alert to the appropriate stakeholders that need to verify the integrity of the service and ensure there is no risk associated with it.