ROS-LLM: A Framework for Embodied AI

Christopher E. Mower, Yuhui Wan, Hongzhan Yu, Antoine Grosnit, Jonas Gonzalez-Billandon, Matthieu Zimmer, Puze Liu, Daniel Palenicek, Davide Tateo, Jan Peters, Kaixian Qu, Mike Zhang, Ganqiu Lan, César Cadena, Marco Hutter, Guangjian Tian, Yingping Zhuang, Kun Shao, Xiaoqing Zhuang, Jianye Hao, Jun Wang, Haitham Bou AmmarResearch Square 2025 · DOI

Abstract

<title>Abstract</title> In this work, we present a framework for embodied artificial intelligence (AI). Our proposed system integrates a Large language model (LLM) based AI agent with the popular Robot Operating System (ROS). Key features of the framework include: integration of ROS with an AI agent connected to many open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for several behavior execution modes (e.g. code, behavior tree), imitation learning for adding new actions to the atomic action library, automated atomic action optimization, and LLM reflection via human and environmental feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios and embodiments, including long-horizon tasks, tabletop rearrangements, dynamic task optimization, and remote supervisory control. Moreover, all the results presented in this work were achieved by utilizing open-source models.

Introduction

Traditionally, deploying robots in unstructured environments like homes or laboratories has required extensive expert programming of specific behaviors, state machines, or behavior trees. This "expert-driven" model is often rigid and expensive, struggling when robots encounter unanticipated scenarios. While Large Language Models (LLMs) offer sophisticated reasoning, "grounding" this reasoning into physical actions remains a significant challenge. Most existing solutions rely on closed-source models and operate in "open-loop" modes, lacking the ability to recover from errors or acquire new physical skills autonomously.

ROS-LLM addresses these gaps by providing a modular, feedback-driven bridge between the LLM "brain" and the Robot Operating System (ROS) "body." As shown in the comparison below, ROS-LLM integrates a wider array of features—including human/environmental feedback and imitation learning—than previous state-of-the-art frameworks.

Comparison of ROS-LLM with existing embodied AI frameworks.

Methodology

The ROS-LLM framework is designed for adaptability and intelligence through several interconnected components that allow non-experts to command robots using natural language.

System architecture of the ROS-LLM framework, illustrating the interaction between the AI Agent, Atomic Action Library, and Deployment.

AI Agent and ROS Interface

The core of the system is an AI Agent that processes natural language prompts and environmental observations. It selects actions from an Atomic Action Library. To ensure flexibility, the framework supports multiple execution modes: single-action execution, sequential planning, Python code generation, and hierarchical Behavior Trees.

Learning and Optimization

To expand its capabilities, ROS-LLM employs Imitation Learning via kinesthetic teaching. Non-experts can physically guide the robot to teach new skills, which are then saved as Dynamic Movement Primitives (DMP) and added to the library. Furthermore, an Atomic Action Optimizer uses an LLM to identify critical hyperparameters (e.g., gains or velocities) and fine-tunes them using Bayesian Optimization (HEBO) in simulation before real-world deployment.

Feedback and Reflection

The system incorporates a "reflection" loop. It processes environmental feedback (such as failure flags) and human corrections. If a robot performs an incorrect action, a human can provide text-based guidance, allowing the LLM to adjust its plan dynamically and recover from errors.

Experimental Evaluation

The framework was evaluated across diverse real-world scenarios, demonstrating that high-tier performance can be achieved using open-source models like Deepseek-7B and Llama-70B.

  • Long-Horizon Success: In a kitchen mock-up, the system successfully decomposed the command "Can you make me a coffee?" into a 12-step sequence involving cabinet manipulation and coffee machine operation without human intervention.
  • Dynamic Task Optimization: In a robot air hockey challenge, the automated optimizer increased the success rate from 30% (manual expert tuning) to 52%.
  • Scientific Automation: A Viper X300 robot successfully automated pH testing experiments in a chemistry lab, with larger models showing near-perfect interpretation of complex protocols.
  • Remote Supervisory Control: The framework proved robust to network latency during cross-continental tests, where natural language commands mitigated the difficulties of traditional joystick teleoperation.

Conclusion and Future Work

ROS-LLM demonstrates that open-source LLMs can effectively manage complex robotic tasks when integrated into a feedback-rich framework. However, challenges remain regarding prompt sensitivity—where minor wording changes can alter outcomes—and the inherent latency of LLM inference, which limits use in millisecond-level reactive control. Future research will focus on improving semantic parsing and contextual tagging to further distinguish between illustrative examples and direct commands.

Summary updated

Abstract

<title>Abstract</title> In this work, we present a framework for embodied artificial intelligence (AI). Our proposed system integrates a Large language model (LLM) based AI agent with the popular Robot Operating System (ROS). Key features of the framework include: integration of ROS with an AI agent connected to many open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for several behavior execution modes (e.g. code, behavior tree), imitation learning for adding new actions to the atomic action library, automated atomic action optimization, and LLM reflection via human and environmental feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios and embodiments, including long-horizon tasks, tabletop rearrangements, dynamic task optimization, and remote supervisory control. Moreover, all the results presented in this work were achieved by utilizing open-source models.
BibTeX
@inproceedings{mower2025ros,
  author = {Christopher E. Mower and Yuhui Wan and Hongzhan Yu and Antoine Grosnit and Jonas Gonzalez-Billandon and Matthieu Zimmer and Puze Liu and Daniel Palenicek and Davide Tateo and Jan Peters and Kaixian Qu and Mike Zhang and Ganqiu Lan and César Cadena and Marco Hutter and Guangjian Tian and Yingping Zhuang and Kun Shao and Xiaoqing Zhuang and Jianye Hao and Jun Wang and Haitham Bou Ammar},
  title = {{ROS-LLM: A Framework for Embodied AI}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-6046846/v1},
}
Figure