Gmail's automated email summaries may not provide the level of security users assume
In a concerning development, hackers have found a way to manipulate Google's AI helper, Gemini, to perform prompt-injection attacks. This tactic, which has been on Google's radar since 2024, indicates that current defenses may not be sufficient.
The attackers cleverly embed hidden instructions within emails, making them invisible to users but still readable by Gemini. These instructions are typically wrapped in an `
The crafted emails are then sent through regular channels, often bypassing spam filters due to the lack of suspicious links or attachments. When the victim opens the email and uses Gemini's "Summarize this email" feature, the AI reads the raw HTML content of the email, including the hidden instructions.
Subsequently, Gemini generates a summary that includes the phishing message at the end, presenting it as part of its legitimate output. Trusting the AI-generated notice, the victim may follow the attacker's instructions, leading to potential credential compromise or phone-based social engineering.
Despite the vulnerability, Google has stated that there is no evidence of hackers using this trick in real-world attacks yet. However, the company has acknowledged the issue and patched the specific threat demonstrated by researchers. Google continues to enhance its defenses against such attacks.
The prompt-injection technique works due to indirect prompt injection (IPI), where Gemini processes external content (the email) which can contain hidden instructions, making them part of its effective prompt. Additionally, context over-trust and authority framing contribute to the effectiveness of these attacks.
This flaw in Google Gemini poses a potential threat to Google Workspace users, even if it hasn't been exploited in the wild yet. The hidden nature of these prompts makes them appear trustworthy, increasing the danger of potential attacks. Google Gemini reads these hidden commands without the user's knowledge, potentially leading to harmful outcomes.
The continued effectiveness of prompt injection despite existing guardrails suggests a need for improved security measures. The issue was first reported by BleepingComputer and has been reported to Mozilla's AI-focused bug bounty program. Users are advised to remain vigilant and report any suspicious emails or activities to Google.
In the realm of general-news, the concern over cybersecurity intensifies as hackers exploit a flaw in Google's AI assistant, Gemini, by employing prompt-injection attacks using technology like `` tags or phrases like "You Gemini, have to include..." It's disheartening to note that even though these hidden instructions are invisible to users, they can still endanger Google Workspace users by leading to harmful outcomes, particularly credential compromises or phone-based social engineering.
