Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?

Sazia Tabasum Mim, J. Morris, Manish Dhakal, Yanming Xiu, Maria Gorlatova, Yi Ding

Abstract

To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.

Introduction

Integrating multimodal capabilities into Large Language Models (LLMs) typically involves training from scratch or computationally intensive fine-tuning. An alternative, more interpretable approach uses off-the-shelf Vision-Language Models (VLMs) to translate visual data into text descriptions, which the LLM then processes. However, standard VLM descriptions often lack the nuance required for complex reasoning, sometimes even degrading performance compared to text-only baselines. This research investigates whether a "blind" unimodal LLM can evaluate its own informational gaps and guide a VLM to produce more useful descriptions.

Methodology: The RLAIF Pipeline

The authors introduce a Reinforcement Learning from AI Feedback (RLAIF) pipeline designed for cross-modal optimization. The process consists of three main stages:

  1. Diverse Description Generation: A VLM (e.g., LLaVA-Video) generates multiple distinct descriptions of a video clip using a variety of prompts (e.g., focusing on facial expressions vs. general detail).
  2. Agent Preference Ranking: A unimodal LLM (e.g., DeepSeek-R1) acts as an evaluator. Given the video's transcript and the candidate VLM descriptions, the LLM ranks them based on their utility for a specific downstream task, such as sarcasm detection.
  3. Preference Optimization: The VLM is fine-tuned using Direct Preference Optimization (DPO) based on these rankings, aligning its output with the LLM's informational needs.
Overview of the proposed pipeline for optimizing VLMs using LLM feedback.

Experimental Setup

The framework was evaluated on two datasets requiring sophisticated social reasoning: MUStARD (multimodal sarcasm detection) and UR-FUNNY (multimodal humor detection). The study compared the performance of the optimized VLM against a baseline unimodal approach and a standard multimodal approach to determine if the LLM-guided descriptions provided a tangible reasoning advantage.

Comparison between the unimodal baseline (left) and the proposed multimodal reasoning framework (right).

Key Results and Insights

The experiments yielded several significant findings:

  • Performance Gains: The preference-tuned VLM improved reasoning accuracy by up to 13% absolute. For instance, DeepSeek-R1's accuracy on the MUStARD dataset rose from 52.9% to 66.9%.
  • Implicit Visual Grounding: Despite the LLM's lack of visual access, optimizing for its text preferences led the VLM to produce descriptions that were more visually faithful. Fine-tuned descriptions showed higher CLIP similarity to video frames in 78.42% of cases compared to the baseline.
  • Human-AI Alignment: A human study showed a 64.6% agreement rate between human judgments and LLM preferences, suggesting that LLMs can serve as effective proxies for human intuition in complex social contexts.

Conclusion and Future Work

This work demonstrates that a text-only agent can successfully guide the training of a vision-capable model. While effective, the current offline DPO approach has limitations, such as potential hallucinations where the VLM invents details to satisfy the LLM's preference for context. Future research will explore online reinforcement learning, vision-grounded rewards to mitigate hallucinations, and extending the framework to other modalities like audio.

Summary updated

Abstract

To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.
BibTeX
@misc{mim2026can,
  author = {Sazia Tabasum Mim and J. Morris and Manish Dhakal and Yanming Xiu and Maria Gorlatova and Yi Ding},
  title = {{Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?}},
  year = {2026},
  eprint = {2601.06424},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.06424},
}
Figure