MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent

Yuxia Fu, Zhizhen Zhang, Yuqi Zhang, Zijian Wang, Zi Huang, Yadan Luo

Abstract

Recent Vision-Language-Action (VLA) models reformulate vision-language models by tuning them with millions of robotic demonstrations. While they perform well when fine-tuned for a single embodiment or task family, extending them to multi-skill settings remains challenging: directly merging VLA experts trained on different tasks results in near-zero success rates. This raises a fundamental question: what prevents VLAs from mastering multiple skills within one model? With an empirical decomposition of learnable parameters during VLA fine-tuning, we identify two key sources of non-mergeability: (1) Finetuning drives LoRA adapters in the VLM backbone toward divergent, task-specific directions beyond the capacity of existing merging methods to unify. (2) Action experts develop inter-block dependencies through self-attention feedback, causing task information to spread across layers and preventing modular recombination. To address these challenges, we present MergeVLA, a merging-oriented VLA architecture that preserves mergeability by design. MergeVLA introduces sparsely activated LoRA adapters via task masks to retain consistent parameters and reduce irreconcilable conflicts in the VLM. Its action expert replaces self-attention with cross-attention-only blocks to keep specialization localized and composable. When the task is unknown, it uses a test-time task router to adaptively select the appropriate task mask and expert head from the initial observation, enabling unsupervised task inference. Across LIBERO, LIBERO-Plus, RoboTwin, and multi-task experiments on the real SO101 robotic arm, MergeVLA achieves performance comparable to or even exceeding individually finetuned experts, demonstrating robust generalization across tasks, embodiments, and environments.

Introduction

Vision-Language-Action (VLA) models represent the state-of-the-art in robotic control, translating visual inputs and language instructions into motor commands. However, most current VLAs are "specialists," optimized for single tasks or specific environments. A natural path toward a "generalist" robot would involve merging several of these specialized experts into a single model. Unfortunately, standard weight-merging techniques that succeed for Large Language Models (LLMs) fail catastrophically in robotics, often resulting in near-zero success rates. MergeVLA addresses this "non-mergeability" problem by identifying and mitigating the root causes of destructive interference during model fusion.

The Challenge of Non-Mergeability

Empirical analysis reveals two primary barriers to merging VLA models. First, fine-tuning drives the model's adapters (LoRA) in divergent, task-specific directions, creating "selfish" parameters that prioritize individual task performance at the expense of global compatibility. Second, standard action decoders utilize self-attention, which causes task-specific information to propagate globally across all layers. This creates structural dependencies that make layers impossible to swap or combine without corrupting the model's logic.

Empirical analysis showing the high ratio of "selfish" parameters and the divergence of weights in deeper blocks during VLA fine-tuning.

The MergeVLA Architecture

MergeVLA introduces a "merging-oriented" architecture designed to maintain modularity and parameter consistency:

  • Sparsely Activated LoRA Adapters: Instead of merging all weights into a single average, MergeVLA employs task-specific masks. These masks act as filters, activating only the parameters essential for a given task while suppressing those that cause interference.
  • Cross-Attention-Only Action Expert: By replacing traditional self-attention layers with cross-attention-only blocks, MergeVLA localizes task specialization. This prevents the "global spread" of task-specific noise, ensuring that the action expert remains modular and mergeable.
  • Test-time Task Router: To handle scenarios where the specific task is unknown, MergeVLA includes a training-free router. It utilizes Singular Value Decomposition (SVD) on initial visual observations to adaptively select the correct task mask and expert head at runtime.
Comparison of MergeVLA with standard VLA architectures, highlighting the transition from non-mergeable to mergeable modules.

Experimental Results

MergeVLA was rigorously tested across multiple simulated and real-world environments, including LIBERO, LIBERO-Plus (distribution shifts), and RoboTwin (cross-embodiment).

Key results include:

  • Superior Generalization: MergeVLA achieved success rates of 90.2% on LIBERO and 70.7% on RoboTwin, significantly outperforming existing merging baselines.
  • Robustness to Distribution Shifts: On the LIBERO-Plus benchmark, MergeVLA demonstrated an 18.7% higher success rate in out-of-distribution scenarios (e.g., changes in lighting or camera angles) compared to previous VLA architectures.
  • Real-World Validation: In physical experiments with an SO101 robotic arm, the merged model maintained a 90.0% success rate, matching the performance of individually fine-tuned experts across tasks like cube picking and stacking.

Conclusion

MergeVLA provides a scalable framework for building generalist robotic agents by enabling the effective merging of specialized skills. By addressing the fundamental issues of parameter divergence and structural dependency, MergeVLA allows researchers to fuse independent robotic capabilities into a single, robust model that maintains high performance across diverse tasks and embodiments.

Summary updated

Abstract

Recent Vision-Language-Action (VLA) models reformulate vision-language models by tuning them with millions of robotic demonstrations. While they perform well when fine-tuned for a single embodiment or task family, extending them to multi-skill settings remains challenging: directly merging VLA experts trained on different tasks results in near-zero success rates. This raises a fundamental question: what prevents VLAs from mastering multiple skills within one model? With an empirical decomposition of learnable parameters during VLA fine-tuning, we identify two key sources of non-mergeability: (1) Finetuning drives LoRA adapters in the VLM backbone toward divergent, task-specific directions beyond the capacity of existing merging methods to unify. (2) Action experts develop inter-block dependencies through self-attention feedback, causing task information to spread across layers and preventing modular recombination. To address these challenges, we present MergeVLA, a merging-oriented VLA architecture that preserves mergeability by design. MergeVLA introduces sparsely activated LoRA adapters via task masks to retain consistent parameters and reduce irreconcilable conflicts in the VLM. Its action expert replaces self-attention with cross-attention-only blocks to keep specialization localized and composable. When the task is unknown, it uses a test-time task router to adaptively select the appropriate task mask and expert head from the initial observation, enabling unsupervised task inference. Across LIBERO, LIBERO-Plus, RoboTwin, and multi-task experiments on the real SO101 robotic arm, MergeVLA achieves performance comparable to or even exceeding individually finetuned experts, demonstrating robust generalization across tasks, embodiments, and environments.
BibTeX
@inproceedings{fu2026mergevla,
  author = {Yuxia Fu and Zhizhen Zhang and Yuqi Zhang and Zijian Wang and Zi Huang and Yadan Luo},
  title = {{MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=wpsBJIdw3u},
}
Figure