Abstract
Large Language Models (LLMs) have shown promise in generating robot control code, yet ensuring the reliability of such code often requires expensive dynamic simulations or risky real-world trials. This paper introduces a framework that utilizes LLMs as "static" simulators to predict robot trajectories and provide semantic feedback for iterative code correction. By bypassing traditional physics engines, the system achieves a 90% success rate on complex navigation tasks. Our results demonstrate that modern LLMs possess sufficient spatial reasoning capabilities to serve as effective world models for robotic verification and refinement.
Introduction
Generating error-free robot control code from natural language instructions remains a significant challenge. While LLMs can produce syntactically correct code, logical errors—such as coordinate system mismatches or incorrect API usage—are common. Traditional solutions rely on dynamic simulators (e.g., Gazebo, AirSim) to provide feedback. However, these "heavy" simulators require significant setup time and computational resources. This work proposes a "static" text-based simulation approach, where an LLM predicts the outcome of code execution and provides corrective feedback in a closed-loop system.
Methodology
The core of the framework is a Generation-Simulation-Evaluation loop that relies on three distinct LLM roles:
- Code Generator: Translates human instructions into initial Python control scripts.
- LLM-Simulator: Interprets the generated code to predict the robot's state transitions and produces a "Semantic Observation"—a textual description of the resulting trajectory.
- Evaluator: Compares the semantic observation with the original task to identify discrepancies and generate feedback for refinement.
The LLM-Simulator is guided by structured prompts containing API definitions, coordinate policies, and few-shot examples, allowing it to function as a text-based world model that reasons about spatial transitions without a physics engine.
Experimental Setup
The framework was evaluated using an "Advanced Task Set" comprising 20 complex UAV missions involving 6–19 sequential actions, such as flying geometric patterns. Testing was conducted using OpenAI's o3-mini and o4-mini models. Furthermore, the system's robustness was verified through real-world deployments on a Holybro X500 V2 quadcopter (running PX4) and a ROSMASTER X3 ground vehicle (running ROS).
Results and Discussion
The LLM-Simulator demonstrated remarkable accuracy, correctly describing intended trajectories with 97.5% to 100% precision compared to ground-truth simulations. In terms of task execution, the framework achieved a Success Rate (SR) of 90% using the o4-mini model, matching or exceeding state-of-the-art methods that require heavy dynamic simulators.
Ablation studies highlighted that semantic feedback is more effective for LLM debugging than raw numerical data, as it allows the model to reason about logical mismatches in natural language. The system also achieved over 96.9% completeness across various robot types, proving its adaptability to different hardware platforms.
Conclusion
This research validates the use of LLMs as static simulators for robotic code refinement. By leveraging semantic feedback, the framework achieves high success rates without the overhead of traditional physics engines. While limited in predicting complex physical interactions like wind or collisions, the approach provides a scalable method for deploying robots in new environments quickly. Future work will explore the use of fine-tuned local models to enhance privacy and reduce reliance on cloud APIs.
Summary updated