Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

Moo Hyun Kim, Chelsea Finn, Percy Liang2025 · DOI

TL;DR

Researchers from Stanford have introduced Optimized Fine-Tuning (OFT), a recipe that transforms slow, "chatty" Vision-Language-Action models (VLAs) into high-speed, high-success robotic controllers. By swapping autoregressive decoding for parallel prediction and continuous actions, they achieved a 26x speedup in action generation and set a new state-of-the-art success rate of 97.1% on the LIBERO benchmark.

What Problem Does This Paper Address?

General-purpose Vision-Language-Action models (like OpenVLA) are trained on massive datasets to understand instructions and perceive scenes. However, when it’s time to put them on a specific robot in a new lab, they usually need fine-tuning.

The standard way to do this—treating actions like words in a sentence (autoregressive decoding)—is painfully slow, often managing only 3-5 actions per second. This is too sluggish for complex, high-frequency tasks like bimanual manipulation (using two arms at once). Furthermore, there hasn't been a clear consensus on the "best" way to fine-tune these models: Should we use discrete tokens or continuous numbers? Should we use diffusion or simple regression? This paper systematically explores these design choices to find the optimal "recipe."

Methods

The authors use OpenVLA (a 7B parameter model) as their base and introduce the OFT recipe, which consists of three main shifts:

  1. Parallel Decoding & Action Chunking: Instead of predicting one action "token" at a time, the model predicts a "chunk" of future actions (e.g., the next 8 or 25 timesteps) all at once. They replace the standard causal mask with a bidirectional one, allowing the model to generate the entire sequence in a single forward pass.
  2. Continuous Action Representation: Rather than discretizing robot movements into "bins" (like vocabulary words), the model outputs raw continuous values.
  3. L1 Regression Objective: Instead of predicting the "next token," the model is trained to minimize the simple L1 distance between its predicted actions and the ground truth. This is computationally much lighter than the popular diffusion-based alternatives.
  4. FiLM (Feature-wise Linear Modulation): For real-world tasks, they add FiLM layers to better "infuse" language instructions into the visual features, preventing the model from ignoring the user's command in favor of visual distractions.

Datasets

The study evaluates the OFT recipe across several rigorous environments:

  • LIBERO Simulation Benchmark: Includes four task suites (Spatial, Object, Goal, and Long-horizon) involving a Franka Emika Panda arm.
  • Real-World ALOHA Robot: A bimanual (two-armed) setup used for dexterous tasks like folding clothes and scooping ingredients.
  • BridgeData V2: Used to test how well the fine-tuning recipe scales to larger, more diverse datasets.

Key Insights and Ideas

  • Speed is a Feature, Not Just a Metric: Parallel decoding doesn't just make the robot faster; it makes it smarter. By predicting chunks of actions, the model better captures temporal dependencies, leading to smoother and more reliable movements.
  • Simplicity Wins: While many researchers are moving toward complex Diffusion Policies for action generation, this paper shows that a high-capacity VLA fine-tuned with simple L1 regression can match or beat diffusion performance while being significantly faster to train and run.
  • The "Language Grounding" Trap: In real-world bimanual tasks with multiple cameras, VLAs often "overfit" to visual cues and ignore language instructions (e.g., picking up a red pepper when told to pick up a green one). The authors found that FiLM is essential for forcing the model to pay attention to the text prompt.

Key Results

  • State-of-the-Art Performance: On the LIBERO benchmark, OpenVLA-OFT improved the average success rate from 76.5% to 97.1%.
  • Massive Throughput Boost: In simulation, action generation throughput increased by 26x (from ~4 Hz to ~110 Hz).
  • Real-World Superiority: On the ALOHA robot, the OFT recipe outperformed other leading VLAs (like $\pi_0$ and RDT-1B) and strong from-scratch policies (like Diffusion Policy and ACT) by up to 15% in absolute success rate.
  • Efficiency: OpenVLA-OFT achieved 77.9 Hz throughput on a real bimanual robot, enabling real-time control that was previously impossible for a 7B parameter model.

Limitations and Future Work

  • Multimodality: L1 regression assumes there is one "correct" average action. In datasets where there are multiple valid ways to complete a task (multimodal distributions), L1 might struggle compared to diffusion-based methods.
  • Language Consistency: While FiLM helped, the authors noted that language grounding remains a challenge in some bimanual scenarios, suggesting a need for even better vision-language integration.
  • Pretraining vs. Fine-tuning: The study focused strictly on fine-tuning. It remains to be seen if these parallel, continuous-action architectures would also be superior for the initial large-scale pretraining of VLAs.

Summary updated

Abstract

Recent vision-language-action models (VLAs) build upon pretrained vision-language models and leverage diverse robot datasets to demonstrate strong task execution, language following ability, and semantic generalization.Despite these successes, VLAs struggle with novel robot setups and require finetuning to achieve good performance, yet how to most effectively fine-tune them is unclear given many possible strategies.In this work, we study key VLA adaptation design choices such as different action decoding schemes, action representations, and learning objectives for fine-tuning, using OpenVLA as our representative base model.Our empirical analysis informs an Optimized Fine-Tuning (OFT) recipe that integrates parallel decoding, action chunking, a continuous action representation, and a simple L1 regression-based learning objective to altogether improve inference efficiency, policy performance, and flexibility in the model's input-output specifications.We propose OpenVLA-OFT, an instantiation of this recipe, which sets a new state of the art on the LIBERO simulation benchmark, significantly boosting OpenVLA's average success rate across four task suites from 76.5% to 97.1% while increasing action generation throughput by 26.In real-world evaluations, our fine-tuning recipe enables OpenVLA to successfully execute dexterous, high-frequency control tasks on a bimanual ALOHA robot and outperform other VLAs (0 and RDT-1B) fine-tuned using their default recipes, as well as strong imitation learning policies trained from scratch (Diffusion Policy and ACT) by up to 15% (absolute) in average success rate.We release code for OFT and pretrained model checkpoints at https://openvla-oft.github.io.
BibTeX
@misc{kim2025fine,
  author = {Moo Hyun Kim and Chelsea Finn and Percy Liang},
  title = {{Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success}},
  year = {2025},
  doi = {10.15607/rss.2025.xxi.017},
}
Figure