Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?

Sazia Tabasum Mim, J. Morris, Manish Dhakal, Yanming Xiu, Maria Gorlatova, Yi Ding2025 · DOI

TL;DR

This paper explores whether a text-only "unimodal" LLM can effectively coach a Vision-Language Model (VLM) to be a better observer. By using Direct Preference Optimization (DPO) based on an LLM's feedback, the authors fine-tune a VLM to generate video descriptions that are significantly more useful for complex social reasoning tasks, like detecting sarcasm and humor, leading to accuracy gains of up to 13%.

What Problem Does This Paper Address?

Integrating visual and acoustic data into Large Language Models (LLMs) is notoriously difficult and expensive, often requiring massive retraining from scratch. A popular "shortcut" is to use a VLM to translate images or videos into text descriptions that a standard LLM can then process.

However, there is a major catch: generic VLM descriptions are often "noisy." They might focus on irrelevant background details while missing the subtle facial expressions or tonal shifts essential for understanding human social cues. In fact, the authors found that naively adding these unoptimized descriptions can actually degrade an LLM's performance compared to just giving it the text transcript alone. The challenge is: how do we make a VLM "see" what an LLM actually needs to know?

Methods

The authors propose a novel framework that turns a unimodal LLM into a "preference agent" to tune a multimodal VLM. The process follows three main steps:

  1. Diverse Description Generation: A VLM (LLaVA-Video) generates five different descriptions for a single video clip using various prompts (e.g., focusing on general content, facial expressions, or emotional cues).
  2. Agent Preference Ranking: A unimodal LLM (like DeepSeek R1) reviews these five text descriptions alongside the conversation transcript. It ranks them based on how helpful they are for a specific task (like sarcasm detection). Crucially, the LLM never "sees" the video; it reasons purely about which text description provides the most coherent context.
  3. DPO Fine-Tuning: The VLM is fine-tuned using Direct Preference Optimization (DPO). It learns to favor the types of descriptions the LLM ranked highly and avoid those it rejected.

This creates a feedback loop where the text-only agent "teaches" the vision model how to describe the world in a way that is most useful for downstream reasoning.

Datasets

The framework was evaluated on two challenging social reasoning benchmarks:

  • MUStARD: A dataset for multimodal sarcasm detection featuring 690 video clips from popular TV shows.
  • UR Funny: A large-scale dataset for humor detection, containing over 16,000 instances of punchlines from TED Talks.

These datasets were chosen specifically because they require understanding subtle, non-verbal cues that are often lost in generic captions.

Key Insights and Ideas

  • The "Latent" Modality: Text descriptions act as an interpretable "latent" modality. Unlike raw visual embeddings, we can read exactly what the VLM thinks it sees, making the model's decision-making process transparent.
  • Interpretive vs. Factual: Fine-tuning shifts the VLM from being a "passive recorder" to an "active interpreter." Instead of just saying "a man is sitting," the tuned model might say "the man's tone is sarcastic and dismissive," which is far more useful for the reasoning agent.
  • LLMs as Effective Critics: Surprisingly, a unimodal LLM is a robust judge of visual descriptions. A human study showed a 64.6% alignment rate between the LLM’s preferences and human judgments.
  • Hallucination with a Purpose: The authors noted that tuned models were more prone to certain hallucinations (like describing a non-existent "voiceover"). However, these hallucinations often captured the emotional truth or tone of the scene so well that they still improved the LLM's final reasoning accuracy.

Key Results

  • Significant Accuracy Boost: On the MUStARD dataset, the preference-tuned VLM helped the DeepSeek-r1 agent achieve 66.9% accuracy, a massive jump from the 51.8% achieved by the zero-shot VLM baseline.
  • Consistency Across Models: The method showed improvements across six different LLM architectures (including Qwen, Llama, and Gemma), proving that the "preference signal" is generalizable.
  • Visual Faithfulness: Using CLIP similarity scores, the authors found that the fine-tuned descriptions were actually more faithful to the source video frames in 78.42% of cases compared to the baseline.
  • Surpassing Text-Only: In many experiments, the multimodal approach finally beat the "Utterance Only" baseline, proving that the optimized visual context was providing genuine value rather than just noise.

Limitations and Future Work

  • Offline Training: The current DPO process happens in batches (offline). The authors suggest that moving toward Online RL fine-tuning would allow the model to adapt to new data patterns in real-time.
  • Hallucination Risks: While some hallucinations were "helpful" for the task, they are still factually incorrect. Future work needs to balance "task-relevance" with "factual grounding" to prevent the model from inventing visual details.
  • Task Scope: The study focused on high-level social reasoning. It remains to be seen if this method works as well for low-level perceptual tasks like object counting or spatial navigation, where factual precision is more critical than emotional interpretation.

Summary updated

BibTeX
@misc{mim2025can,
  author = {Sazia Tabasum Mim and J. Morris and Manish Dhakal and Yanming Xiu and Maria Gorlatova and Yi Ding},
  title = {{Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-ijcnlp.133},
}
Figure