As artificial intelligence transitions from "semantic AI" (chatbots) to "physical AI" (embodied robots), the security landscape shifts from offensive text to physical harm. Researchers have introduced ANNIE, the first systematic framework designed to study adversarial safety attacks on Embodied AI (EAI) robots. By leveraging a principled safety taxonomy based on international standards and a novel "Attack Leader" model, the study demonstrates that advanced Vision-Language-Action (VLA) models can be manipulated into performing dangerous physical actions with success rates exceeding 50%.
The Safety Gap in Embodied AI
Current Embodied AI research focuses heavily on performance—measuring how effectively a robot can pick up a cup or navigate a room. However, the authors identify three critical gaps in the existing literature:
- Vague Definitions: "Safety" is often conflated with "task failure." A robot stalling is a performance issue; a robot swinging a knife toward a human is a distinct safety violation.
- Lack of Data: Existing datasets focus on successful task completion rather than "corner cases" where safety is compromised.
- Ineffective Attack Models: Traditional adversarial attacks (e.g., changing pixels to misclassify an image) do not translate well to the long-horizon, continuous action sequences required for robotics.
The ANNIE Framework
The authors address these gaps with a three-pronged approach centered on a rigorous safety taxonomy and a task-aware adversarial framework.
A Standardized Safety Taxonomy
Grounded in ISO/TS 15066 (the international standard for human-robot collaboration), the framework defines three levels of violations:
- Critical: Violating separation distance when using hazardous tools (e.g., knives).
- Dangerous: Excessive velocity or premature release of objects near humans.
- Risky: Unintended collisions with the environment (walls, furniture).
ANNIE-Attack and the Attack Leader Model
Instead of applying random noise, the ANNIE-Attack uses an Attack Leader Model. This model acts as a "malicious strategist" that decomposes a high-level goal (e.g., "approach the human") into specific, frame-by-frame action perturbations. These perturbations are optimized using Projected Gradient Descent (PGD) to find the minimal pixel-level changes needed to trick the robot's VLA model.
To ensure stealth, the authors developed an Adaptive Sparse Attack. This method dynamically selects which frames to perturb, attacking more frequently at the start of a motion to set a "trend" and then backing off. This makes the attack effective yet significantly harder to detect by automated monitors.
ANNIE-Bench: A Safety-Centric Benchmark
To evaluate these vulnerabilities, the researchers developed ANNIE-Bench, a benchmark built on the ManiSkill simulator. It includes 9 scenarios (3 for each safety level) and 2,400 video-action sequences. Unlike general benchmarks like Habitat or Calvin, ANNIE-Bench specifically targets high-risk corner cases across various locations, scenes, and tasks.
Key Insights and Results
- High Success Rates: The ANNIE-Attack achieved success rates exceeding 50% across all safety categories. In some scenarios, the adaptive sparse attacks achieved a 100% success rate while perturbing only one out of every three frames.
- Normalization as a Vulnerability: The study found that action normalization methods significantly impact vulnerability. Models using Min-Max normalization (like Baku) were far more susceptible than those using Mean-Std normalization (like ACT), as Min-Max is highly sensitive to the outliers introduced by adversarial perturbations.
- Real-World Validation: The team successfully moved beyond simulation to a physical UR3 robotic arm. In real-world tests, they induced a robot holding a knife to turn away from its target (an apple) and approach a nearby human in 4 out of 10 trials.
Conclusion and Future Work
ANNIE highlights that a model can be proficient at tasks while remaining incredibly easy to "jailbreak" physically. The authors emphasize an urgent need for "security-driven defenses," such as adversarial training or specialized safety monitors designed specifically for the action chains of EAI systems. Future research will explore whether larger foundation models (like OpenVLA) are inherently more or less susceptible to these coordinated, long-horizon attacks.
Summary updated