Google's Gemini workplace agents vulnerable to exploitation through a single invitation, researchers reveal worrisome security loophole
In the ever-evolving world of technology, a new threat has emerged that poses a critical risk to Language Model (LLM)-based applications. Researchers from SafeBreach Labs have developed a novel attack method called "Targeted Promptware Attacks", which can compromise AI-powered assistants such as Gemini.
Promptware attacks utilize an input, in the form of text, images, or audio samples, designed to exploit an LLM interface at the time of inference to trigger malicious activities. One such technique, known as context poisoning, manipulates the assistant by inserting unwanted instructions into the context space - invisible to the user, but readable and actionable by the assistant.
The most alarming aspect of these attacks is that they can be initiated through two methods: Direct Prompt Injection and Indirect Prompt Injection. In the case of Indirect Prompt Injection, when a user asks Gemini about their emails or calendar, the indirect prompt is triggered, compromising the Gemini contact area.
The implications of these attacks are far-reaching, with 73% of the identified threats being classified as high to critical risk by the Threat Analysis and Risk Assessment Framework (TARA). This evaluation methodology was used to assess the risks posed by these attacks on LLM-based assistants.
Companies must reassess the risk that promptware poses to their LLM-based systems through a threat analysis and risk assessment (TARA) and prioritize the provision of necessary mitigations. Google, acknowledging the research on indirect prompt injection techniques, has launched a targeted initiative to address the issues.
Their response includes implementing improved user confirmations for sensitive actions, robust URL processing with cleanup and policies for trust levels, and enhanced prompt injection detection by content classifiers.
Unfortunately, most security experts are either unfamiliar with promptware or do not consider it a critical risk due to several misconceptions related to attacks on AI-based systems. However, tests have shown that a wide range of attacks can be performed via the compromised agents, including sending spam and phishing messages, generating harmful content, deleting calendar entries, remote control of connected home devices, determining the victim's location, accessing Zoom video streams, and exfiltrating private emails.
As the annual Virus Bulletin Conference returns to Europe to discuss the latest developments in cybersecurity, it is crucial that the industry remains vigilant against emerging threats like promptware. New variants are in preparation, including 0-click variants targeting automatic LLM inferences and non-targeted variants sending promptware to all users (e.g., via YouTube, Google Maps).
Images for this article are sourced from Depositphotos.com. It is essential that companies prioritize the security of their LLM-based systems to protect their users and maintain trust in AI-powered assistants.
Read also:
- Dazzling Accomplishment: Constructing a £5m Venture by Age 25
- Transportation via roads plays a critical role in India's shift towards clean energy.
- EU Commission President von der Leyen emphasizes continuity, affirming that Europe's automotive future will be powered by electricity
- BYD aspires to emulate the success of Apple within the automobile sector.
