Abstract
<title>Abstract</title> In this work, we present a framework for embodied artificial intelligence (AI). Our proposed system integrates a Large language model (LLM) based AI agent with the popular Robot Operating System (ROS). Key features of the framework include: integration of ROS with an AI agent connected to many open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for several behavior execution modes (e.g. code, behavior tree), imitation learning for adding new actions to the atomic action library, automated atomic action optimization, and LLM reflection via human and environmental feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios and embodiments, including long-horizon tasks, tabletop rearrangements, dynamic task optimization, and remote supervisory control. Moreover, all the results presented in this work were achieved by utilizing open-source models.
Introduction
Traditionally, deploying robots in unstructured environments like homes or laboratories has required extensive expert programming of specific behaviors, state machines, or behavior trees. This "expert-driven" model is often rigid and expensive, struggling when robots encounter unanticipated scenarios. While Large Language Models (LLMs) offer sophisticated reasoning, "grounding" this reasoning into physical actions remains a significant challenge. Most existing solutions rely on closed-source models and operate in "open-loop" modes, lacking the ability to recover from errors or acquire new physical skills autonomously.
ROS-LLM addresses these gaps by providing a modular, feedback-driven bridge between the LLM "brain" and the Robot Operating System (ROS) "body." As shown in the comparison below, ROS-LLM integrates a wider array of features—including human/environmental feedback and imitation learning—than previous state-of-the-art frameworks.
Methodology
The ROS-LLM framework is designed for adaptability and intelligence through several interconnected components that allow non-experts to command robots using natural language.
AI Agent and ROS Interface
The core of the system is an AI Agent that processes natural language prompts and environmental observations. It selects actions from an Atomic Action Library. To ensure flexibility, the framework supports multiple execution modes: single-action execution, sequential planning, Python code generation, and hierarchical Behavior Trees.
Learning and Optimization
To expand its capabilities, ROS-LLM employs Imitation Learning via kinesthetic teaching. Non-experts can physically guide the robot to teach new skills, which are then saved as Dynamic Movement Primitives (DMP) and added to the library. Furthermore, an Atomic Action Optimizer uses an LLM to identify critical hyperparameters (e.g., gains or velocities) and fine-tunes them using Bayesian Optimization (HEBO) in simulation before real-world deployment.
Feedback and Reflection
The system incorporates a "reflection" loop. It processes environmental feedback (such as failure flags) and human corrections. If a robot performs an incorrect action, a human can provide text-based guidance, allowing the LLM to adjust its plan dynamically and recover from errors.
Experimental Evaluation
The framework was evaluated across diverse real-world scenarios, demonstrating that high-tier performance can be achieved using open-source models like Deepseek-7B and Llama-70B.
- Long-Horizon Success: In a kitchen mock-up, the system successfully decomposed the command "Can you make me a coffee?" into a 12-step sequence involving cabinet manipulation and coffee machine operation without human intervention.
- Dynamic Task Optimization: In a robot air hockey challenge, the automated optimizer increased the success rate from 30% (manual expert tuning) to 52%.
- Scientific Automation: A Viper X300 robot successfully automated pH testing experiments in a chemistry lab, with larger models showing near-perfect interpretation of complex protocols.
- Remote Supervisory Control: The framework proved robust to network latency during cross-continental tests, where natural language commands mitigated the difficulties of traditional joystick teleoperation.
Conclusion and Future Work
ROS-LLM demonstrates that open-source LLMs can effectively manage complex robotic tasks when integrated into a feedback-rich framework. However, challenges remain regarding prompt sensitivity—where minor wording changes can alter outcomes—and the inherent latency of LLM inference, which limits use in millisecond-level reactive control. Future research will focus on improving semantic parsing and contextual tagging to further distinguish between illustrative examples and direct commands.
Summary updated