Add 'prompt' to the long list of injection attacks
Image via Unsplash
Injection attacks have been around a long time and are still one of the most dangerous forms of attack vectors used by cybercriminals. Injection attacks refer to when threat actors “inject” or provide a non-validated input to a program. This initial input gets processed as part of a command or query which in turn manipulates, changes or overrides the execution of a program. Injection attacks are known to cause data loss, data corruption, security breaches, leakage of information and loss of control. A ‘successful’ injection can empower adversaries with administrator privileges, allowing them to access or manipulate database information without authorization.
Considered a top web security risk, injection attacks usually aim at web applications. Although injection attacks come in a variety of flavors, certain attack types are more common than others. A 2022 study by Radware reported the most common types of injection attacks include predictable resource location, code injection and SQL injection attacks, while overall attacks on web applications grew by 128%.
Experts warn of a new type of injection attack aimed at AI
Generative AI bots such as ChatGPT and Google Bard are designed to give human-like responses and narratives and follow instructions when “prompted” with questions. However, studies show these tools can be manipulated to accomplish malicious tasks, respond in undesirable ways, reveal sensitive information or ignore their safety filters if prompts are carefully tailored or engineered to overcome AI guardrails — a.k.a. prompt injection attacks.
Prompt injection attacks draw parallels with code injection attacks where attackers insert malicious code via an input to a system. The key difference between the two is that with AI, the “input” is the prompt itself. Prompt injection attacks may grow to become more common than standard injection attacks because the barriers to entry are extremely low. Even if someone has no coding or technical skills, they can still trick AI into following their instructions as long as they are clever and creative with their ill-intentioned prompts.
AI-enhanced tools can be turned into sophisticated phishers
Businesses and startups have already started integrating ChatGPT plugins to develop AI-enhanced virtual assistants that help with appointment settings and bookings, customer service, social media and other applications. Even Cornell University computer science labs is studying this phenomenon. These chatbots can be exploited, manipulated or hijacked to retrieve sensitive information using a new technique called indirect prompt injection.
AI-enhanced chatbots operate by scraping information off web pages and therefore they can be triggered to follow a malicious instruction without requiring further input from a user. Imagine a situation where a hacker poisons a webpage and hides malicious prompts by adding comments or using zero-point fonts on the webpage. A researcher recently demonstrated that he was able to successfully leverage Microsoft Bing Chat and generate phishing messages that looked like they came from a Microsoft employee. The chatbot even requested the user’s credit card information.
The scary part about indirect prompt injection is that attackers do not need to take over or control the entire website that the user visits. All they need to do is simply inject regular text or a comment in the webpage that is invisible to the user simply by changing the font color to white. When the chatbot ingests this content, it reprograms its goals based on the prompt provided. As long as the poisoned web page remains open, the injection will continue to remain active.
How can organizations protect against prompt injection attacks?
To mitigate risks related to prompt injection attacks, businesses need a multi-pronged approach that builds secure behavior in employees as well as safeguards AI technology against malicious attacks. From an employee standpoint, users should be comprehensively trained to recognize scams and social engineering attacks that are delivered using AI. At the AI-level, organizations should consider building stronger filters and rules that prevent AI from behaving unexpectedly. For instance, experimenting with methods such as reinforcement learning from user feedback so that AI models better align with business expectations. Also, introducing bug bounty programs can incentivize users to research and report vulnerabilities and weaknesses in AI.
Don’t forget that security is a cat and mouse game. Every time AI becomes stronger and more secure, threat actors will discover new ways to work around it. This is why it’s crucial for organizations to not limit mitigation activity strictly to technical controls, but to support those efforts with investments in training people to be aware of these various AI prompts and injection attack types.
This article originally ran in Today’s Cybersecurity Leader, a monthly cybersecurity-focused eNewsletter for security end users, brought to you by Security magazine. Subscribe here.
Looking for a reprint of this article?
From high-res PDFs to custom plaques, order your copy today!
