ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration

Yichen Zhao, Minghao Shao, Zhuoyan Yu, Yijun Shen, Boyuan Chen, Yik-Cheung Tam, Muhammad ShafiquearXiv 2026 · arXiv

TL;DR

ROSTrans is a new execution-grounded benchmark designed to evaluate how well Large Language Models (LLMs) handle the complex task of migrating robotics code from ROS1 to ROS2. The study reveals that while frontier LLMs can often generate code that compiles, they struggle significantly with the behavioral and communication semantics of robotics, with over half of all generated samples failing only at runtime.

What Problem Does This Paper Address?

Robotics software development, particularly within the Robot Operating System (ROS) ecosystem, presents unique challenges for code generation. Unlike standard programming tasks, robotics code relies heavily on complex inter-node coordination, middleware configuration (like Quality of Service settings), and specific lifecycle semantics.

The ongoing industry transition from the legacy ROS1 to the modern ROS2 is not a simple matter of API renaming; it requires a fundamental rethinking of communication middleware and system organization. Existing code generation benchmarks typically focus on isolated functions or static analysis. However, in robotics, the most critical correctness properties—such as whether two nodes can actually talk to each other—only become observable once the code is launched and running. There has been a lack of standardized, execution-based benchmarks to see if LLMs can actually handle this "repository-scale" architectural reasoning.

Methods

The authors developed ROSTrans, a pipeline for creating and verifying migration tasks. The process involves:

  1. Scaffolded Hollowing: They selected 108 tasks from over 70 open-source ROS1 repositories. Key logic blocks were "hollowed out" and replaced with TODO markers, while preserving the surrounding architectural context (imports, class structures, etc.) to focus the LLM on migration logic rather than boilerplate.
  2. Hybrid Injection Protocol: To evaluate a model, the benchmark takes a verified ROS2 reference package and injects only the LLM's generated files into it. This ensures the model cannot "cheat" by modifying the test suite or the build configuration.
  3. Containerized Execution: Every task is built and run inside a dedicated Docker container. This ensures reproducibility and allows for integration-level runtime testing using pytest and ROS2 launch testing.
  4. Execution-Guided Repair: The authors also tested a multi-round "agentic" workflow where models receive build and test error logs to iteratively fix their code.
The ROSTrans pipeline, from hollowing ROS1 code to containerized evaluation.

Datasets

The ROSTrans benchmark consists of 108 tasks derived from diverse real-world sources like the official ROS tutorials, navigation stacks, and manipulation frameworks (e.g., MoveIt).

The tasks are categorized into a hierarchical taxonomy:

  • Interface-level (44 tasks): Basic patterns like publishers, subscribers, and parameters.
  • Behavior-level (29 tasks): Coordinating multiple components, like perception-control pipelines.
  • System-level (35 tasks): Architectural reasoning across multiple files and nodes.

The benchmark includes 976 runtime test functions and spans both C++ (61 tasks) and Python (41 tasks).

Key Insights and Ideas

  • The "Silent" Runtime Failure: A staggering 56.5% of all samples build successfully but fail at runtime. This highlights a major gap: models can mimic the syntax of ROS2, but they often fail to preserve the semantics of the original ROS1 program.
  • Dependency Blindness: A primary source of failure is the models' inability to reconstruct the ROS2 package and dependency structure. They often try to carry forward legacy ROS1 dependencies (like #include <ros/ros.h>) that no longer exist or have been renamed in ROS2.
  • Service-Client Bottleneck: Models performed worst on service-client interactions. In fact, in the initial zero-shot evaluation, no model passed a single service-client task.
  • Feedback is Essential: The jump in performance during the execution-guided repair phase suggests that LLMs are currently much better at debugging robotics code with specific error logs than they are at generating it correctly from scratch.

Key Results

The evaluation of six frontier LLMs (including Claude Opus 4.6 and GPT-5.5) showed that the task is far from solved:

  • Low Initial Success: The pass@1 rate (success on the first try) was below 10.9% for all models.
  • Failure Breakdown: As shown in Figure 5, only 7.8% of samples passed. The majority of failures occurred during runtime tests (56.5%) rather than the build stage (26.1%).
  • Repair Gains: Execution-guided repair significantly boosted performance. After three rounds of feedback, Claude Opus 4.6 reached a 49.1% success rate, and GPT-5.5 reached 47.2%.
Breakdown of outcomes for all evaluated samples, showing the prevalence of runtime failures.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The current benchmark focuses on middleware and communication correctness. It does not yet evaluate physical robot control quality, perception accuracy, or trajectory performance.
  • Repair Strategy: The execution-guided repair study was preliminary, using a basic feedback loop. More sophisticated agentic strategies (e.g., multi-model debates or specialized repair prompts) remain to be explored.
  • Environment: While Docker provides reproducibility, it doesn't fully simulate the hardware-level timing issues that can plague real-world robotics deployments.

Future work aims to extend ROSTrans to include more complex "embodied" tasks and to refine the feedback mechanisms used for automated code repair.

Summary updated

Abstract

Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.
BibTeX
@misc{zhao2026rostrans,
  author = {Yichen Zhao and Minghao Shao and Zhuoyan Yu and Yijun Shen and Boyuan Chen and Yik-Cheung Tam and Muhammad Shafique},
  title = {{ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration}},
  year = {2026},
  eprint = {2610.41168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41168},
}
Figure