Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks

Yurun Chen, Xueyu Hu, Keting Yin, Juncheng Li, Shengyu Zhang

TL;DR

This paper introduces "Active Environmental Injection Attacks" (AEIA), a novel threat where malicious actors disguise attacks as normal operating system elements to manipulate multimodal AI agents. They demonstrate that even advanced agents are highly vulnerable, achieving up to 93% attack success, highlighting a critical security gap in current AI agent design.

What Problem Does This Paper Address?

As AI agents become more sophisticated and integrated into operating systems (OS) for task execution, a significant security vulnerability emerges: their ability to distinguish between legitimate environmental cues and malicious "impostors." Existing research on "environment injection attacks" has largely focused on passive attacks within web browsers. However, this paper addresses the critical gap of active attacks targeting mobile OS environments. These agents interact with dynamic OS elements like notifications, pop-ups, and incoming calls. Attackers can exploit these interactions by injecting deceptive elements that actively disrupt the agent's decision-making process, leading to unintended actions or task failures. The authors highlight that current agents lack robust defenses against such sophisticated, context-aware attacks.

Methods

The core contribution is the introduction of the Active Environmental Injection Attack (AEIA) concept. AEIA is characterized by its active nature (real-time OS parameter modification), process sensitivity (requiring precise timing), and environment integration (leveraging OS-specific characteristics).

To operationalize AEIA, the authors propose AEIA-MN, an attack scheme specifically designed for mobile notifications (SMS-based, due to better compatibility and less stringent security checks compared to app-based notifications). AEIA-MN comprises three attack types:

  1. Adversarial Attack: This involves crafting malicious notification text to mislead the agent's perception or reasoning. It can also exploit UI element occlusion, where notifications cover critical interactive elements on the screen.
  2. Reasoning Gap Attack: This exploits the time lag between an agent's perception and action stages. During this "reasoning gap," attackers can alter the OS state (e.g., via a notification), causing the agent to act based on outdated information.
  3. Combinatorial Attack: This combines both Adversarial and Reasoning Gap attacks to amplify their disruptive effects.

The paper also details the attack procedure in Algorithm 1, outlining how these attacks are integrated into the agent's perception, reasoning, and action stages.

Datasets

The experiments were conducted using two primary benchmarks:

  • AndroidWorld: A benchmark for evaluating agents on mobile OS tasks, specifically the "easy" subset comprising 61 tasks.
  • AppAgent: A benchmark for evaluating agents on popular application tasks, consisting of 45 tasks.

No new datasets were introduced; instead, existing benchmarks were utilized to evaluate the robustness of various MLLM-based agents.

Key Insights and Ideas

  • Active vs. Passive Attacks: The distinction between active and passive environment injection attacks is crucial. AEIA's active nature makes it significantly more potent and harder to defend against than previous passive methods.
  • Mobile Notifications as a Vector: SMS-based notifications are a surprisingly effective attack vector due to their ubiquity and lack of strict security controls on Android.
  • Reasoning Gap Vulnerability: The inherent latency in MLLM reasoning creates a "reasoning gap" that attackers can exploit to manipulate the agent's understanding of the current OS state.
  • Combinatorial Attacks are Devastating: Combining adversarial content with reasoning gap exploitation leads to the highest attack success rates, demonstrating a synergistic effect.
  • Defense Prompt Ineffectiveness: Simple defense prompts, like "ignore notification content," show limited effectiveness, suggesting that more sophisticated defense mechanisms are needed.
  • "Illusory Improvement": In some cases, attacks can lead to an increase in reported task success rates, not due to actual improvement, but because the attack manipulates the agent into prematurely signaling task completion, which the system then registers as success.

Key Results

  • High Vulnerability: Existing multimodal agents, including advanced MLLMs like GPT-40, are highly vulnerable to AEIA.
  • Maximum Attack Success: The AEIA-MN attack achieved a maximum attack success rate of 93% on the AndroidWorld benchmark by combining two vulnerabilities.
  • Adversarial Attack Impact: Adversarial attacks significantly reduced task success rates, with some agents showing a drop from 9% to 0% (GPT-40 on AppAgent).
  • Reasoning Gap Attack Impact: While less consistently devastating than adversarial attacks, the Reasoning Gap attack significantly disrupted task execution for most agents, causing notable decreases in success rates.
  • Combinatorial Attack Dominance: This combined attack proved to be the most destructive, with success rate reductions reaching up to 67.2% on AndroidWorld.
  • Defense Prompt Results: Defense prompts offered only marginal improvements, with some models even showing increased attack success rates.

Limitations and Future Work

  • Limited Defense Effectiveness: The paper highlights the limited effectiveness of current defense strategies, particularly prompt-based defenses, against AEIA.
  • Interference Factors: Certain tasks (e.g., enabling WiFi) were found to be susceptible to "interference factors" where default system states could lead to tasks being marked as successful even if the agent failed, complicating the assessment of attack impact.
  • Future Work: The authors suggest exploring more robust defense mechanisms. They propose integrating blockchain technology for a novel security architecture with an environmental credibility verification module to provide agents with a trustworthy execution environment. They also suggest investigating multi-agent systems (MAS) where dedicated agents could handle input security detection and voting mechanisms could mitigate individual agent failures.

Summary updated

BibTeX
@inproceedings{chen2025evaluating,
  author = {Yurun Chen and Xueyu Hu and Keting Yin and Juncheng Li and Shengyu Zhang},
  title = {{Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755646},
}
Figure