An Empirical Study on Embodied Artificial Intelligence Robot (EAIR) Software Bugs

Zeqin Liao, Zibin Zheng, Peifan Reng, Henglong Liang, Zixu Gao, Zhixiang Chen, Wei Li, Yuhong NanarXiv 2025 · DOI · arXiv

Abstract

Embodied Artificial Intelligence Robots (EAIRs) are increasingly integrated into physical environments, yet their software reliability remains under-explored. This paper presents the first large-scale empirical study of EAIR software bugs, analyzing 885 manually verified bugs from 80 open-source projects. We characterize these bugs through a multi-dimensional taxonomy of symptoms, root causes, and affected modules, revealing that EAIRs exhibit unique failure modes tied to AI reasoning and physical simulation. Our findings provide a foundation for developing specialized testing and debugging tools for the next generation of embodied AI.

Introduction

Embodied AI Robots (EAIRs), such as humanoid robots and advanced autonomous agents, represent a paradigm shift from traditional robotics. Unlike legacy systems governed by rigid rules, EAIRs utilize complex AI models—including Large Language Models (LLMs) and Reinforcement Learning (RL)—to perceive and interact with dynamic physical spaces.

However, the fusion of high-level AI logic with low-level physical control introduces novel software risks. A bug in an EAIR system isn't just a software crash; it can manifest as a "violation of physical reality" or an "embodied control anomaly," potentially leading to hardware damage or human injury. Despite these stakes, the community lacks a systematic understanding of EAIR failure patterns. This study addresses this gap by characterizing 885 real-world bugs to identify where these systems are most vulnerable.

Methodology

To ensure a rigorous analysis, we followed a structured four-phase pipeline to collect and categorize EAIR bugs.

The comprehensive workflow for EAIR bug collection, manual investigation, and taxonomy building.
  1. Project Selection: We identified 80 popular, deployable EAIR projects from GitHub, academic literature, and industry sources, including frameworks like Nvidia Isaac-GR00T and Facebook Habitat.
  2. Bug Mining: We extracted over 6,000 issues and 17,000 commits from these repositories.
  3. Manual Verification: Given the difficulty of reproducing EAIR bugs (which often require specific simulation environments), we spent six person-months manually verifying bugs, resulting in a dataset of 885 confirmed instances.
  4. Taxonomy Construction: Six domain experts used open card sorting to classify bugs by symptoms, underlying causes, and modules, achieving a high inter-rater agreement (Cohen’s Kappa = 0.769).

The EAIR Bug Dataset

We introduce the first comprehensive EAIR bug dataset. Each entry includes the vulnerable version, reproduction steps (often with visual evidence), and the repair patch. This dataset serves as a benchmark for training automated bug detection and repair models specifically tailored for embodied systems.

Empirical Results and Key Insights

Our analysis reveals that EAIRs fail in ways fundamentally different from traditional software.

Symptom Analysis

We identified 15 distinct symptoms, categorized into EAIR-specific and general system symptoms. Over one-third of the bugs are EAIR-specific, highlighting the unique challenges of these systems.

Taxonomy of EAIR bug symptoms, distinguishing between EAIR-specific failures and general system issues.

Key findings include:

  • AI Agent Vulnerability: Issues stemming from the AI agent (e.g., model errors, output integrity) account for 56.7% of EAIR-specific root causes.
  • Simulation Bottlenecks: The simulation module is the most bug-prone component. Modeling complex physical interactions (e.g., soft bodies) for AI training is inherently fragile.
  • Compound Bugs: 67% of EAIR-specific causes are "compound," where a single root cause triggers multiple symptoms or multiple causes converge into one failure, significantly complicating debugging.

Root Causes and Module Distribution

While "Incorrect Assignment" remains a common general cause, EAIR-specific failures are driven by deeper architectural issues. Approximately 74.24% of all bugs are tied to the four core EAIR mechanisms: perception, interaction, control, and simulation. Furthermore, API misuse is rampant across modules due to the complex interplay between AI agents and hardware interfaces.

Conclusion and Future Work

This study underscores the need for specialized debugging and testing frameworks for EAIRs. Future research should focus on the automated detection of API misuse and targeted debugging using module-symptom mappings to "prune" search spaces during diagnostics. By providing the first systematic map of the EAIR bug landscape, we hope to accelerate the development of safer and more reliable embodied AI systems.

Summary updated

BibTeX
@misc{liao2025empirical,
  author = {Zeqin Liao and Zibin Zheng and Peifan Reng and Henglong Liang and Zixu Gao and Zhixiang Chen and Wei Li and Yuhong Nan},
  title = {{An Empirical Study on Embodied Artificial Intelligence Robot (EAIR) Software Bugs}},
  year = {2025},
  eprint = {2507.18267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.18267},
}
Figure