Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models

Huaping Liu, Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Han‐Bo ZhangResearch Square 2025 · DOI

Abstract Foundation Vision Language Models (VLMs) exhibit strong capabilities in multi-modal representation learning, comprehension, and reasoning. By injecting action components into the VLMs, Vision-Language-Action models (VLAs) can be naturally formed and also show promising performance. Existing work has demonstrated the effectiveness and generalization of VLAs in multiple scenarios and tasks. Nevertheless, the transfer from VLMs to VLAs is not trivial since existing VLAs differ in their backbones, action-prediction formulations, data distributions, and training recipes. This leads to a missing piece for a systematic understanding of the design choices of VLAs. In this work, we disclose the key factors that significantly influence the performance of VLA and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research.

The Challenge of Generalist Robot Policies

While Foundation Vision-Language Models (VLMs) like LLaVA or Flamingo have revolutionized multi-modal reasoning, their adaptation into Vision-Language-Action (VLA) models for robotics remains a complex challenge. The current landscape of VLA research is fragmented, with various models employing different backbones, action representation methods (discrete vs. continuous), and training protocols. This lack of standardization makes it difficult to determine which design choices truly drive performance and generalization in robotic tasks.

The central goal of this research is to move beyond ad-hoc model building and provide a systematic "guidebook" for VLA design. By evaluating over 600 experimental configurations, the authors identify the critical factors that enable a VLM to effectively control a robot across diverse environments and embodiments.

The RoboVLMs Framework and Design Choices

To facilitate this large-scale study, the authors introduced RoboVLMs, a flexible and open-source framework designed for the rapid integration and evaluation of various VLA components. The investigation focuses on three primary pillars of design: backbone selection, architectural formulation, and data recipes.

Key design choices for building effective Vision-Language-Action (VLA) models.

The framework allows researchers to systematically explore:

  1. Backbone Selection: Testing 8 distinct VLM backbones (e.g., KosMos-2, PaliGemma, Flamingo) to assess the impact of pre-training data and model architecture.
  2. Architecture Formulation: Comparing discrete action tokens against continuous action spaces, and evaluating different methods for history aggregation (interleaved tokens vs. dedicated policy heads).
  3. Data Strategies: Determining the most effective way to leverage cross-embodiment data from the Open X-Embodiment (OXE) dataset, comparing pre-training, co-training, and post-training approaches.

Systematic Evaluation of VLA Components

The RoboVLMs framework serves as a unified testbed where language instructions and multi-view visual observations are processed by a VLM backbone to generate robot actions. This modularity is crucial for understanding how high-level reasoning from web-scale pre-training translates to low-level motor control.

The RoboVLMs framework supports various VLM backbones and multi-modal inputs for cross-embodiment tasks.

The study utilized a comprehensive suite of benchmarks, including the CALVIN simulation for multi-task manipulation, SimplerEnv for real-to-sim evaluation, and a custom real-world dataset collected on a Kinova Gen3 arm. This diverse set of environments ensures that the findings are robust across both simulated and physical platforms.

Key Findings: Continuous Actions and Policy Heads

The extensive ablation studies yielded several counter-intuitive and impactful insights:

  • Continuous Action Spaces are Superior: While some prominent models use discrete tokenization for actions, this study found that continuous action spaces consistently outperform discrete ones. Discrete bins often lead to compounding errors, especially in long-horizon tasks where precision is vital.
  • The Advantage of Dedicated Policy Heads: For integrating temporal history, adding a separate Policy Head (such as a small Transformer or RNN) after the VLM backbone proved more effective than interleaving history tokens directly into the VLM prompt. This modular approach allows the VLM to focus on spatial reasoning while the policy head handles temporal dynamics.
  • Backbone Quality is Paramount: The choice of VLM backbone is the single most important factor. Models pre-trained on massive, high-quality vision-language datasets (like KosMos-2 and PaliGemma) provide a significantly better foundation for robotic control than smaller or less diverse models.
  • Post-training for Cross-Embodiment Data: When incorporating data from different robot types, the "Post-training" strategy—pre-training on broad datasets like OXE followed by fine-tuning on target robot data—yielded the best results, providing a superior initialization for specific tasks.

Results and Generalization

The optimized RoboVLM configurations achieved state-of-the-art (SOTA) results. On the CALVIN benchmark, the KosMos-based RoboVLM achieved an average task completion rate of 4.49 out of 5, surpassing previous leaders like GR-1.

Beyond benchmarks, the models demonstrated impressive zero-shot generalization in real-world settings. They remained robust when faced with novel distractors, backgrounds, and objects that were not present in the training set. A notable emergent behavior was "self-correction": if a robot missed an initial grasp, it would re-evaluate the scene and adjust its movement for a successful second attempt, showcasing the reasoning power inherited from the VLM backbones.

Conclusion and Future Directions

This work provides a rigorous foundation for the next generation of generalist robot policies. By identifying that continuous actions, dedicated policy heads, and high-quality VLM backbones are the "golden path" for VLAs, the authors have cleared much of the uncertainty in the field.

Future work will focus on reducing inference latency for higher-frequency control and exploring the internal mechanisms of VLMs to further tighten the integration between reasoning and action. The open-sourcing of the RoboVLMs framework, models, and datasets at robovlms.github.io invites the community to build upon these findings and push the boundaries of robotic intelligence.

Summary updated

Abstract

<title>Abstract</title> Foundation Vision Language Models (VLMs) exhibit strong capabilities in multi-modal representation learning, comprehension, and reasoning. By injecting action components into the VLMs, Vision-Language-Action models (VLAs) can be naturally formed and also show promising performance. Existing work has demonstrated the effectiveness and generalization of VLAs in multiple scenarios and tasks. Nevertheless, the transfer from VLMs to VLAs is not trivial since existing VLAs differ in their backbones, action-prediction formulations, data distributions, and training recipes. This leads to a missing piece for a systematic understanding of the design choices of VLAs. In this work, we disclose the key factors that significantly influence the performance of VLA and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io.
BibTeX
@inproceedings{liu2025towards,
  author = {Huaping Liu and Xinghang Li and Peiyan Li and Minghuan Liu and Dong Wang and Jirong Liu and Bingyi Kang and Xiao Ma and Tao Kong and Han‐Bo Zhang},
  title = {{Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-5770637/v1},
}
Figure