Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation

Ning Wang, Zihan Yan, Weiyang Li, Chuan Ma, He Henry Chen, Tao Xiang

Introduction

As Large Language Models (LLMs) increasingly serve as the cognitive engines for autonomous systems like drones and robots, a critical security gap has emerged: behavioral risk. Unlike traditional LLM safety research, which focuses on "generative risks" such as offensive text or data leakage, embodied agents face physical risks. A malicious instruction like "Fly into that crowd" or "Disable the safety camera" can lead to real-world harm.

This research addresses two primary hurdles in securing these agents:

  1. Lack of Benchmarks: The absence of standardized datasets mapping unique physical risks for embodied agents.
  2. Prompt Interference: Embodied agents rely on long "functional prompts" (system instructions defining roles and environments). Existing safety filters often fail to distinguish between these necessary system instructions and malicious user commands embedded within them.

To bridge this gap, the authors introduce EAsafetyBench, a comprehensive safety benchmark, and Pinpoint, a high-speed, internal input moderation framework.

EAsafetyBench: A Taxonomy of Risk

The researchers established a seven-category taxonomy of behavioral risks specific to embodied agents: Violent Crimes, Non-Violent Crimes, Privacy Violations, Malfunction, Discrimination/Hate, Harassment, and Illegal Assistance.

To support this taxonomy, they developed EAsafetyBench-Drone, a dataset containing 9,435 instructions (including 4,875 safety threats) generated via GPT-4 and refined for semantic diversity. This was augmented by SafeAgentBench (focused on household robots) and EAsafetyBench-Prompt, a collection of 100 operational contexts used to test moderation robustness.

The Pinpoint Framework

Pinpoint is an internal moderation method that operates during the LLM's inference process. It decouples user instructions from system prompts to prevent "functional noise" from masking malicious intent.

The Pinpoint framework architecture, illustrating the instruction localization, masked attention mechanism, and internal MLP classifier.

As shown in the architecture above, Pinpoint's workflow involves:

  • Instruction Localization: Using special markers (<|begin_of_instruction|>) to define the boundaries of the user's command.
  • Masked Attention Mechanism: Instead of analyzing the entire noisy prompt, Pinpoint applies a mask at a specific layer of the LLM. This forces the moderation classifier to focus exclusively on the features of the user instruction.
  • Lightweight Classifier: A 3-layer Multi-Layer Perceptron (MLP) with only 4 million parameters processes these features, ensuring minimal latency.

Key Insights and Results

Experimental results across 10 open-source LLMs (including Llama 3, Mistral, and Falcon) demonstrate that Pinpoint achieves an average 94.58% detection accuracy.

The "Middle Layer" Sweet Spot

A critical finding of this study is that extracting features from the middle layers of an LLM is optimal for safety classification. Shallow layers lack sufficient semantic processing, while the deepest layers become overly "contaminated" by the specific functional prompts of the agent.

F1 scores across different model layers, showing that peak performance typically occurs in the middle layers and that the masked attention mechanism is vital for maintaining accuracy.

As illustrated in the performance plots, the masked attention mechanism is vital; without it, F1 scores drop significantly as models struggle to ignore complex system prompts. Furthermore, Pinpoint demonstrates superior generalization on "in-the-wild" prompts compared to baselines like GradSafe.

Efficiency for Real-Time Robotics

For embodied agents like drones, speed is a safety requirement. Unlike external moderation APIs which introduce significant latency, Pinpoint’s internal approach adds a negligible 0.002 seconds per instance, making it ideal for real-time deployment.

Conclusion and Future Work

Pinpoint and EAsafetyBench provide a robust foundation for securing the physical actions of AI-driven agents. While the current framework excels on causal decoder models, future work will focus on expanding the taxonomy to humanoid and industrial robots and integrating multi-modal inputs (such as vision) to detect situational hazards in the agent's physical surroundings.

Summary updated

BibTeX
@inproceedings{wang2025advancing,
  author = {Ning Wang and Zihan Yan and Weiyang Li and Chuan Ma and He Henry Chen and Tao Xiang},
  title = {{Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/867},
}
Figure