Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging

Mohamed Salim Aissi, Olivier Sigaud, Laure Soulier, Nicolas ThomearXiv 2026 · arXiv

TL;DR

Fine-tuning Vision-Language Models (VLMs) for robotics often causes them to "forget" how to reason and describe scenes. This paper introduces a post-hoc parameter merging technique that uses Singular Value Decomposition (SVD) to surgically blend a VLM with its robotic derivative, successfully restoring high-level reasoning without sacrificing low-level motor control.

What Problem Does This Paper Address?

In the quest for generalist robotic agents, researchers typically take a pre-trained Vision-Language Model (VLM)—which is great at describing images and reasoning—and fine-tune it on robot demonstrations to create a Vision-Language-Action (VLA) model.

The catch? This process suffers from catastrophic forgetting. As the model learns the "low-level" language of motor commands (joint angles, velocities), it overwrites the "high-level" semantic weights that allowed it to understand complex instructions or describe a scene. You end up with a robot that can move precisely but can no longer explain what it is doing or why, making it brittle in the face of complex, multi-step tasks. Existing solutions usually require expensive co-training on massive mixed datasets; this paper seeks a cheaper, "post-hoc" way to fix models that have already been fine-tuned.

Methods

The authors propose a lightweight merging method that doesn't require retraining the full 3B+ parameter models. Their approach follows three main steps:

  1. Task Vector Extraction: They calculate the "task vector" ($\Delta$), which is simply the difference between the weights of the fine-tuned VLA and the original VLM. This vector represents everything the model learned during its robotics training.
  2. SVD Decomposition: They break this task vector down using Singular Value Decomposition (SVD) for every layer. This decomposes the adaptation into individual orthogonal directions (singular components), ranked by their importance.
  3. Learned Merging: Instead of a uniform blend, they assign a unique scalar coefficient ($\alpha$) to each singular component. They then optimize these coefficients using a composite objective function that penalizes both poor action prediction and poor language generation.

By keeping the base models frozen and only learning these small scalar coefficients, the authors can "dial in" exactly which parts of the robotic adaptation to keep and which parts of the original VLM reasoning to restore.

Datasets

The researchers evaluated their method across several diverse benchmarks:

  • Robotics (Action): LIBERO (Spatial, Goal, and Object suites) and RoboCasa (a challenging kitchen simulation environment).
  • Vision-Language (Reasoning): COCO for image captioning and VQAv2 for visual question answering.
  • Embodied Reasoning: An EQA (Embodied Question Answering) benchmark that asks the model questions specifically grounded in robotic settings (e.g., "What is the state of the object?").

Key Insights and Ideas

  • The Rank Asymmetry: Perhaps the most striking finding is that language and action capabilities live at different "scales" in weight space. The authors found that language recovery is low-rank, requiring only a few dominant singular directions to be restored. In contrast, action preservation is diffuse (high-rank), meaning the robotic skills are spread across many weaker components.
  • Why Naive Merging Fails: Because of this asymmetry, standard merging techniques (like "Model Soup" or simple averaging) fail. If you blend the models enough to save the actions, you haven't restored enough language. If you restore the language, you've likely wiped out the diffuse action weights.
  • Surgical Precision: By using SVD, the method can keep the "tail" of the singular spectrum (where the actions live) while reverting the "head" of the spectrum back toward the VLM (where the language reasoning lives).

Key Results

The merged model, dubbed Ours, achieved the "best of both worlds" across all tests:

  • Action Success: On the LIBERO benchmark, the merged model maintained a success rate of ~89-93%, nearly matching the pure VLA's performance, while baselines like DARE or SLERP collapsed to 0%.
  • Language Recovery: The merged model reached 29.5 ROUGE-L on captioning tasks, nearly recovering the original VLM's score (33.6), whereas the pure VLA scored a dismal 1.25.
  • Qualitative Improvement: Qualitative examples show that while a pure VLA might output garbled text like "Robot Bot Bot..." when asked to describe a scene, the merged model provides coherent descriptions like "A kitchenware set containing various items such as pots, pans, and utensils," all while still being able to pick up the mug.

Limitations and Future Work

  • Backbone Dependency: The method assumes the VLM and VLA share the same architecture and tokenizer. It cannot currently merge models with different "DNA."
  • Optimization Loop: While much cheaper than full fine-tuning, the method still requires a short training loop to optimize the $\alpha$ coefficients.
  • Scale: The experiments were conducted on 3B-parameter models (PaliGemma). Whether these findings on rank asymmetry hold true for 70B+ parameter models remains an open question.
  • Future Directions: The authors suggest that the restored reasoning capabilities could be used to help robots perform "long-horizon" tasks, where the robot needs to plan several steps ahead rather than just reacting to immediate commands.

Summary updated

Abstract

Vision-Language-Action models (VLAs) are obtained by fine-tuning pre-trained Vision-Language Models (VLMs) on robot demonstration data, enabling action prediction from visual observations and language instructions. However, this fine-tuning induces catastrophic forgetting, substantially degrading the semantic and visual grounding capabilities of the original VLM. We address this by merging the parameters of a VLM and its VLA derivative without retraining either model. Concretely, we extract the task vector (the difference between VLA and VLM weights) and decompose it via Singular Value Decomposition. For each component, we learn a scalar mixing coefficient controlling how much of the VLA adaptation is retained, optimised to jointly preserve action prediction and vision-language capabilities. Experiments show that our merged model performs well on both axes simultaneously, consistently outperforming established merging baselines.
BibTeX
@misc{aissi2026bring,
  author = {Mohamed Salim Aissi and Olivier Sigaud and Laure Soulier and Nicolas Thome},
  title = {{Bring Back Language to Actions: Restoring VLM capabilities in VLA Models via Parameter Merging}},
  year = {2026},
  eprint = {2611.18891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18891},
}
Figure