EAST: Embodied AI Safety Testing via Systematic Fault Injection in Digital Twins

Yuhao Tian, Tao Li, Ling Zhou, Dave Towey, Zhaoxin Fan, Wenjun Wu, Rubing Huang, Tsong Yueh ChenarXiv 2026 · arXiv

TL;DR

EAST (Embodied AI Safety Testing) is a novel framework that uses digital twins to systematically stress-test the safety of robotic agents. By injecting literature-grounded faults—ranging from garbled instructions to delayed motor commands—EAST reveals a critical "success-safety gap" where robots successfully complete tasks while simultaneously performing dangerous actions.

What Problem Does This Paper Address?

As Embodied AI (EAI) moves from research labs to the physical world, evaluating these systems solely on task performance (e.g., "did the robot pick up the cup?") is no longer enough. A robot might succeed in its goal but do so by colliding with a human, dropping a fragile object, or ignoring safety constraints.

Current evaluation methods often focus on robustness (handling noise or new environments) rather than safety (avoiding hazards). Furthermore, testing safety on physical hardware is risky and expensive. There is a lack of systematic methods to translate theoretical safety knowledge into executable, reproducible tests that can be run safely in simulation.

Methods

The authors introduce the EAST framework, which operates in three distinct stages:

  1. Systematic Safety-Test Generation: The researchers built a safety-fault model grounded in existing literature, identifying six "safety phenotypes" (like unsafe contact or grasp failure). They mapped these to eight executable fault operators (Op1–Op8) that intervene at different points in the control loop:
  2. Perception: Masking visual targets.
  3. Instruction: Omitting or swapping roles in language commands (e.g., "put the plate on the container" vs. "put the container on the plate").
  4. Environment: Shifting object poses or overriding friction.
  5. Action/Timing: Delaying arm commands or triggering premature releases during handovers.
  6. Fault-Injected Digital-Twin Execution: These tests are run in a high-fidelity simulator (the "Digital Twin"). The framework verifies that the requested fault actually occurred at runtime (e.g., confirming a command was indeed delayed).
  7. Simulator-State Safety Assessment: Crucially, EAST uses independent "oracles" to judge task success and safety violations separately. This allows the system to detect "unsafe success" cases.
The three-stage EAST pipeline: Generation, Execution, and Assessment.

Datasets

The evaluation was conducted using the RoboTwin 2.0 simulation platform. The authors tested two prominent Vision-Language-Action (VLA) policy architectures:

  • ACT (Action Chunking with Transformers)
  • RDT (Robotics Diffusion Transformer)

They utilized six diverse manipulation tasks, including block handovers, hammering, and microwave door opening. The study compared EAST against several baselines: Random-Legal (random valid configurations), Rule-Based (fixed safety rules), and RoboTwin-DR (standard domain randomization).

Key Insights and Ideas

  • The Success-Safety Gap: The most striking finding is that task success is a poor proxy for safety. In many cases, the robot's policy was "clever" enough to finish the job despite a fault, but it violated safety constraints to do so.
  • Systematic vs. Random: Unlike standard domain randomization, which applies broad noise, EAST’s targeted fault injection is more efficient at uncovering specific failure modes.
  • Digital Twins as Safety Sandboxes: Digital twins allow for "dangerous" testing—like deliberately causing a robot to drop a heavy object—that would be impossible or irresponsible to perform on physical hardware.
  • Traceability: Because every test is generated from a specific fault operator and recorded with a "behavior signature," researchers can trace a safety failure back to a specific weakness in the policy's perception or timing.
Examples of "unsafe success": The robot completes the task but exhibits wrong placement, early release, or wrong positioning.

Key Results

  • Broad Coverage: EAST successfully targeted all 74 admissible task-policy-operator combinations, providing a much more comprehensive safety profile than baseline methods.
  • Unsafe Success Detection: The framework exposed 82 valid executions where the robot satisfied the task objective but violated a safety requirement.
  • High Validity: EAST achieved a Valid Safety-Test Yield (VSTY) of 85.83%, significantly outperforming unconstrained random generation (69.33%), meaning it wastes less computational budget on "meaningless" tests.
  • Reproducibility: Through controlled replay experiments, the authors achieved 100% observable-semantic replay consistency, proving that the safety failures discovered are not just "flukes" of the simulator but reproducible bugs in the policy.
Heatmap showing the many-to-many relationship between fault operators and safety failure categories.

Limitations and Future Work

  • Sim-to-Real Gap: While digital twins are powerful, the authors acknowledge that forces and sensing in simulation don't always perfectly match the physical world. Future work involves validating these digital-twin results on physical robot deployments.
  • Scope of Tasks: The current study focuses on six manipulation tasks. The authors plan to extend the taxonomy to non-manipulation tasks (like navigation) and more diverse robot embodiments.
  • Complex Safety Logic: Future iterations could include richer temporal predicates to capture hazards that emerge over longer execution histories or involve human proximity.

Summary updated

Abstract

Task success is not a sufficient measure of safety for embodied artificial intelligence (AI), because robots may complete objectives while violating safety requirements. Existing evaluations often emphasize performance and robustness, but lack a systematic route from documented safety hazards to executable, verifiable tests. We present EAST (Embodied AI Safety Testing), a digital-twin framework that converts literature-grounded safety evidence into fault-injection tests across perception, instructions, environments, actions, and timing. EAST combines an expert-reviewed safety-fault taxonomy and eight executable operators with applicability and validity constraints. Its three-stage workflow generates test specifications, verifies fault realization during simulated execution, and independently assesses safety challenges, violations, task outcomes, and behavioral signatures using simulator-state evidence. The framework preserves execution traces and replayable records, supporting attribution and reproducibility across experiments. We evaluate EAST on six RoboTwin manipulation tasks and two policy architectures, ACT and RDT, against alternative testing strategies under matched budgets. Across five comparative campaigns, EAST targets all 74 admissible task-policy-operator combinations and obtains valid evidence for 63, while achieving safety-challenge exposure rates close to Random-Legal and Rule-Based. Moreover, 82 valid EAST executions complete their tasks despite safety violations, revealing risks concealed by success-oriented evaluation. These findings support systematic, validity-aware, reproducible safety testing beyond task-completion metrics.
BibTeX
@misc{tian2026east,
  author = {Yuhao Tian and Tao Li and Ling Zhou and Dave Towey and Zhaoxin Fan and Wenjun Wu and Rubing Huang and Tsong Yueh Chen},
  title = {{EAST: Embodied AI Safety Testing via Systematic Fault Injection in Digital Twins}},
  year = {2026},
  eprint = {2610.34358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34358},
}
Figure