Transferring Textual Preferences to Vision-Language Understanding through Model Merging

Abstract: Large vision-language models (LVLMs) perform outstandingly across various multimodal tasks. However, their ability to evaluate generated content remains limited, and training vision-language reward models (VLRMs) with preference data is computationally expensive. This paper explores a training-free alternative by merging text-based reward models (RMs) with LVLMs to create VLRMs. Our approach shows that integrating these models leads to improved performance over LVLMs’ scoring and text-based RMs, offering an efficient method for incorporating textual preferences into LVLMs.

Introduction

Large Vision-Language Models (LVLMs) are proficient at generating content but often struggle with evaluating it. Improving these models typically involves Reinforcement Learning from Human Feedback (RLHF), which necessitates a Reward Model (RM) to serve as a judge. However, developing a multimodal reward model (a VLRM) is a significant challenge due to the high cost of collecting human preference data and the computational resources required for training.

This research addresses this bottleneck by proposing a training-free method: merging existing high-quality text-only reward models with LVLMs. The core hypothesis is that the "wisdom" regarding human preferences stored in text-only RMs can be injected into vision-language models through model merging, bypassing the need for expensive multimodal training.

Methods: Merging for Multimodality

The proposed approach leverages the shared architecture of many modern LVLMs and text-only RMs. For instance, Llama-3.2-Vision uses a standard Large Language Model (LLM) as its backbone, which is often the same base model used for fine-tuning text-only RMs. This compatibility allows for direct parameter merging.

Expand

Source, p. 1

The study evaluates four primary merging strategies to combine the LVLM and the text-based RM:

1. Weighted Averaging: A simple linear combination of model weights.
2. Task Arithmetic: Adding "task vectors" (the difference between fine-tuned and base weights) to the base model.
3. TIES (Trimming, Isolate, and Elect Signed): Reduces parameter interference by pruning small values and resolving sign conflicts.
4. DARE (Drop and Rescale): Randomly drops redundant delta parameters to further streamline the merge.

Crucially, only the language modeling components (transformer blocks) are merged, while the vision encoder and adapters remain unchanged.

Experimental Setup

The merged models were evaluated across several key benchmarks:

• VL-RewardBench: A comprehensive benchmark covering general instructions, hallucination detection, and multimodal reasoning.
• TextVQA: Used for "Best-of-N" sampling to pick the correct answer from multiple generated candidates.
• MMMU-Pro: A multi-discipline reasoning benchmark used to test the model's ability to rank complex responses.
• RLAIF-V: A subset of 400 instances was used as a validation set to tune merging hyperparameters like the weight $\lambda$.

Key Results and Insights

The results demonstrate that textual preferences are highly compatible with multimodal tasks. Preferences learned in a text-only environment—such as conciseness and truthfulness—successfully transfer to visual tasks, as the model applies textual logic to visual tokens.

Expand

Source, p. 3

Key findings include:

• Superior Performance: The merged VLRM (using DARE + Task Arithmetic) achieved an overall score of 59.7 on VL-RewardBench, significantly higher than the base Llama-3.2-11B-Vision score of 42.9.
• Efficiency: Merging is a low-cost alternative to training, taking minutes on a CPU compared to days on a GPU cluster.
• Hallucination Mitigation: The most substantial gains occurred in hallucination detection, where the text RM's strictness helped the model identify discrepancies between descriptions and images.
• Merging vs. Cascading: The merged model consistently outperformed "Cascading" (where an LVLM describes an image and a text RM judges the text), indicating a deeper integration of visual and textual cues.

Limitations and Future Work

While effective, the method currently requires the LVLM and RM to share the same base architecture (e.g., both derived from Llama-3.1-8B). Future research could explore cross-architecture merging. Additionally, while this study focuses on creating the "judge" (the VLRM), the next step is using these merged models to fine-tune LVLMs via RLHF to improve generation quality.

@inproceedings{li2025transferring,
  author = {Chen-An Li and Tzu-Han Lin and Yun-Nung Chen and Hung-yi Lee},
  title = {{Transferring Textual Preferences to Vision-Language Understanding through Model Merging}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=0M9ktDyj1Y},
}