TL;DR
ROSTrans is a new execution-grounded benchmark designed to evaluate how well Large Language Models (LLMs) handle the complex task of migrating robotics code from ROS1 to ROS2. The study reveals that while frontier LLMs can often generate code that compiles, they struggle significantly with the behavioral and communication semantics of robotics, with over half of all generated samples failing only at runtime.
What Problem Does This Paper Address?
Robotics software development, particularly within the Robot Operating System (ROS) ecosystem, presents unique challenges for code generation. Unlike standard programming tasks, robotics code relies heavily on complex inter-node coordination, middleware configuration (like Quality of Service settings), and specific lifecycle semantics.
The ongoing industry transition from the legacy ROS1 to the modern ROS2 is not a simple matter of API renaming; it requires a fundamental rethinking of communication middleware and system organization. Existing code generation benchmarks typically focus on isolated functions or static analysis. However, in robotics, the most critical correctness properties—such as whether two nodes can actually talk to each other—only become observable once the code is launched and running. There has been a lack of standardized, execution-based benchmarks to see if LLMs can actually handle this "repository-scale" architectural reasoning.
Methods
The authors developed ROSTrans, a pipeline for creating and verifying migration tasks. The process involves:
- Scaffolded Hollowing: They selected 108 tasks from over 70 open-source ROS1 repositories. Key logic blocks were "hollowed out" and replaced with TODO markers, while preserving the surrounding architectural context (imports, class structures, etc.) to focus the LLM on migration logic rather than boilerplate.
- Hybrid Injection Protocol: To evaluate a model, the benchmark takes a verified ROS2 reference package and injects only the LLM's generated files into it. This ensures the model cannot "cheat" by modifying the test suite or the build configuration.
- Containerized Execution: Every task is built and run inside a dedicated Docker container. This ensures reproducibility and allows for integration-level runtime testing using
pytestand ROS2 launch testing. - Execution-Guided Repair: The authors also tested a multi-round "agentic" workflow where models receive build and test error logs to iteratively fix their code.
Datasets
The ROSTrans benchmark consists of 108 tasks derived from diverse real-world sources like the official ROS tutorials, navigation stacks, and manipulation frameworks (e.g., MoveIt).
The tasks are categorized into a hierarchical taxonomy:
- Interface-level (44 tasks): Basic patterns like publishers, subscribers, and parameters.
- Behavior-level (29 tasks): Coordinating multiple components, like perception-control pipelines.
- System-level (35 tasks): Architectural reasoning across multiple files and nodes.
The benchmark includes 976 runtime test functions and spans both C++ (61 tasks) and Python (41 tasks).
Key Insights and Ideas
- The "Silent" Runtime Failure: A staggering 56.5% of all samples build successfully but fail at runtime. This highlights a major gap: models can mimic the syntax of ROS2, but they often fail to preserve the semantics of the original ROS1 program.
- Dependency Blindness: A primary source of failure is the models' inability to reconstruct the ROS2 package and dependency structure. They often try to carry forward legacy ROS1 dependencies (like
#include <ros/ros.h>) that no longer exist or have been renamed in ROS2. - Service-Client Bottleneck: Models performed worst on service-client interactions. In fact, in the initial zero-shot evaluation, no model passed a single service-client task.
- Feedback is Essential: The jump in performance during the execution-guided repair phase suggests that LLMs are currently much better at debugging robotics code with specific error logs than they are at generating it correctly from scratch.
Key Results
The evaluation of six frontier LLMs (including Claude Opus 4.6 and GPT-5.5) showed that the task is far from solved:
- Low Initial Success: The
pass@1rate (success on the first try) was below 10.9% for all models. - Failure Breakdown: As shown in Figure 5, only 7.8% of samples passed. The majority of failures occurred during runtime tests (56.5%) rather than the build stage (26.1%).
- Repair Gains: Execution-guided repair significantly boosted performance. After three rounds of feedback, Claude Opus 4.6 reached a 49.1% success rate, and GPT-5.5 reached 47.2%.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The current benchmark focuses on middleware and communication correctness. It does not yet evaluate physical robot control quality, perception accuracy, or trajectory performance.
- Repair Strategy: The execution-guided repair study was preliminary, using a basic feedback loop. More sophisticated agentic strategies (e.g., multi-model debates or specialized repair prompts) remain to be explored.
- Environment: While Docker provides reproducibility, it doesn't fully simulate the hardware-level timing issues that can plague real-world robotics deployments.
Future work aims to extend ROSTrans to include more complex "embodied" tasks and to refine the feedback mechanisms used for automated code repair.
Summary updated