A tutorial note on collecting simulated data for vision-language-action models

Heran Wu, Zirun Zhou, Jingfeng ZhangarXiv 2025 · DOI · arXiv

Abstract

Vision-Language-Action (VLA) models are revolutionizing robotics by mapping multimodal inputs directly to actions. However, their effectiveness is heavily dependent on the availability of massive, high-quality datasets. This tutorial provides a practical guide to collecting and generating VLA training data, exploring three distinct approaches: automated simulation via PyBullet, human-teleoperated data via LIBERO, and large-scale cross-embodiment learning via the RT-X dataset.

Introduction: The VLA Data Challenge

Traditional robotic systems rely on the "sense-plan-act" paradigm, which decomposes intelligence into separate perception, planning, and control modules. Recently, Vision-Language-Action (VLA) models (such as RT-1, RT-2, and OpenVLA) have emerged to replace this pipeline with a single, end-to-end neural network. These models process camera feeds and text instructions to directly output robot actions.

The success of VLA models is currently bottlenecked by data. Unlike standard computer vision or natural language processing, robotic data collection is expensive, time-consuming, and complex, requiring synchronized streams of images, language, and physical action states. This paper addresses the practical knowledge gap by providing a concrete tutorial on how to build, organize, and scale these multimodal datasets.

Methodologies for Dataset Construction

The authors detail three distinct methodologies for VLA dataset construction, highlighting the trade-offs of each:

  1. Automated Simulation (PyBullet + Ravens): An automated pipeline using PyBullet for physics simulation and the Ravens framework for scripted "oracle" (expert) policies. This allows for rapid, deterministic collection of perfectly annotated vision-language-action triplets without human intervention.
  2. Human Teleoperation (LIBERO): To capture natural, human-like manipulation, the authors demonstrate a pipeline using the LIBERO benchmark. They record smooth 6-degree-of-freedom trajectories at 20Hz using a SpaceMouse. This method allows for the modification of Behavioral Domain Definition Language (BDDL) files to inject custom "distractor" objects into scenes to test model robustness.
  3. Cross-Embodiment Real Data (RT-X): This paradigm shifts away from single-robot data collection by standardizing actions from vastly different robot kinematics into a unified 7-dimensional format (x, y, z, roll, pitch, yaw, gripper state).
The Ravens framework provides a diverse set of manipulation tasks for automated data generation.

Key Insights and Technical Considerations

  • The "Distractor" Challenge: When modifying scenes, careful placement of distractor objects is critical. Because standard simulation cameras generate 2D projective images without inherent depth separation, objects placed too close together might be perceived as a single connected component.
  • Robust Success Validation: In human teleoperation data, brief accidental object contacts can trigger false positives for task completion. The authors implemented a 10-step (0.5 seconds at 20Hz) continuous validation mechanism to ensure genuine task success.
  • Suboptimal Data as a Feature: While multi-institutional datasets like RT-X suffer from quality variations, this variance improves model robustness. Exposure to suboptimal human demonstrations helps the model learn recovery behaviors.

Results and Data Organization

The PyBullet scripted oracle pipeline achieved a 95% success rate across diverse manipulation tasks, proving to be a highly reliable source of stable data. In the RT-X dataset, 60% of demonstrations are pick-and-place operations, with the Franka Panda robot accounting for 40% of all trajectories.

To manage this data, the authors demonstrated a hierarchical storage format that synchronizes 7-timestep sequences of RGB (480x640), depth, actions, and rewards. This structure enables efficient random access during model training.

Detailed breakdown of the multimodal data structures used for training VLA models.

Conclusion

Simulation-based approaches provide perfect annotations but lack real-world noise, while human-guided methods are realistic but costly. The authors suggest that researchers should use massive datasets like RT-X as a pre-training foundation and extend them with smaller, task-specific data collections for specialized applications such as tool use or complex assembly.

Summary updated

Abstract

Traditional robotic systems typically decompose intelligence into independent modules for computer vision, natural language processing, and motion control. Vision-Language-Action (VLA) models fundamentally transform this approach by employing a single neural network that can simultaneously process visual observations, understand human instructions, and directly output robot actions -- all within a unified framework. However, these systems are highly dependent on high-quality training datasets that can capture the complex relationships between visual observations, language instructions, and robotic actions. This tutorial reviews three representative systems: the PyBullet simulation framework for flexible customized data generation, the LIBERO benchmark suite for standardized task definition and evaluation, and the RT-X dataset collection for large-scale multi-robot data acquisition. We demonstrated dataset generation approaches in PyBullet simulation and customized data collection within LIBERO, and provide an overview of the characteristics and roles of the RT-X dataset for large-scale multi-robot data acquisition.
BibTeX
@misc{wu2025tutorial,
  author = {Heran Wu and Zirun Zhou and Jingfeng Zhang},
  title = {{A tutorial note on collecting simulated data for vision-language-action models}},
  year = {2025},
  eprint = {2508.06547},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.06547},
}
Figure