Debating for Better Reasoning in Vision-Language Models

Ashutosh Adhikari, Mirella Lapata2025 · DOI

TL;DR

This paper introduces a multimodal debate framework where two "sighted" vision-language models (VLMs) argue over visual questions to convince a "blind" text-only judge. The researchers found that this adversarial process not only identifies correct answers more accurately than individual models but also generates high-quality reasoning traces that can be used to fine-tune and significantly improve the reasoning capabilities of other VLMs.

What Problem Does This Paper Address?

As AI models become increasingly sophisticated, we face the "scalable oversight" problem: how do we supervise and evaluate models that might eventually surpass human expertise? In the realm of Vision-Language Models (VLMs), this is particularly tricky.

Current alignment relies heavily on expensive human-labeled data. The authors explore whether a "weaker" model—specifically a text-only Large Language Model (LLM) that cannot "see" the image—can effectively supervise "stronger" sighted models. The challenge is to move beyond simple consensus and create a mechanism where models must provide evidence-based, logical arguments that a non-expert (the blind judge) can verify for consistency and relevance.

Methods

The authors propose a Multimodal Debate protocol focused on Visual Question Answering (VQA):

  1. Disagreement Sets: Instead of debating every question, the system identifies instances where two expert VLMs provide different answers.
  2. The Debate: Two "sighted" experts (e.g., Molmo or Qwen2.5-VL) engage in two rounds of debate. Crucially, they only defend answers they actually "believe" in, avoiding the pitfalls of forced role-playing. They provide premises grounded in the image and logical arguments to support their conclusion.
  3. The Blind Judge: A text-only model (Qwen32B) acts as the adjudicator. It does not see the image but receives the debate transcript and image descriptions generated by the experts. It evaluates arguments based on consistency, relevance, and logical sufficiency.
  4. Reasoning Trace Extraction: The rationale generated by the judge to support its verdict is extracted as a "reasoning trace."
  5. Fine-tuning: These traces are used to fine-tune models (using LoRA) to instill better reasoning habits, even in models that didn't participate in the original debate.

The authors also compared this to a Consultancy protocol, where a single expert tries to convince the judge of its answer without an opponent.

Datasets

The framework was evaluated on three major multimodal benchmarks:

  • MME: Focuses on perception and cognition (Yes/No questions).
  • MMMU: A massive multi-discipline benchmark requiring college-level subject knowledge.
  • MathVista: Evaluates mathematical reasoning and visual understanding.

Key Insights and Ideas

  • Blind Judges are Surprisingly Effective: A strong text-only model can often spot logical inconsistencies or "evasive" arguments from a sighted model, even without visual access. If an expert's description of an image contradicts its own argument, the judge can penalize it.
  • Debate > Consultancy: In a consultancy setting, experts tend to be "deceptive"—they can easily convince a judge even when they are wrong. In a debate, the presence of an adversary makes it much harder to get away with logical leaps or hallucinations.
  • Truthfulness as a Competitive Advantage: The results showed that for debate, win rates are roughly proportional to model accuracy. This suggests that the "truth" is generally easier to defend logically than a falsehood.
  • Reasoning is Transferable: One of the most significant findings is that reasoning traces from these debates can "teach" other models. Fine-tuning a weaker model (like LLaVA-1.5) on these traces led to substantial performance gains on unseen tasks.

Key Results

  • Superior Accuracy: The debate framework consistently outperformed individual expert models. For example, on the disagreement sets, the judge's accuracy using the debate protocol was significantly higher than the baseline performance of the individual models (e.g., Molmo-O's baseline of 30.7% jumped to 59.3% under debate adjudication).
  • Fine-tuning Gains: Fine-tuning LLaVA-1.5 on debate traces improved its aggregate score across benchmarks from 56.0 to 62.3.
  • Out-of-Domain Success: Models fine-tuned on traces from two datasets (e.g., MMMU and MathVista) showed improved performance on the third (e.g., MME), proving that the reasoning capabilities gained were not just memorization but generalized logic.

Limitations and Future Work

  • Dependency on Instruction Following: The framework relies on the experts' ability to follow complex debate guidelines and provide detailed, honest descriptions.
  • Risk of "Eloquent Hallucination": A very articulate model might still lead a judge to a false conclusion if it generates highly convincing but hallucinatory image descriptions.
  • Inference Cost: Running multiple rounds of debate between several large models is computationally expensive compared to a single model inference.
  • Future Directions: The authors suggest exploring more complex tasks beyond VQA, involving larger numbers of experts, and incorporating reinforcement learning to further refine the debate strategies. They also suggest giving the judge "partial" visual access to further improve oversight.

Summary updated

Abstract

As Large Language Models (LLMs) gain expertise across diverse domains and modalities, scalable oversight becomes increasingly challenging, particularly when their capabilities may surpass human evaluators.Debate has emerged as a promising mechanism for enabling such oversight.We extend the debate paradigm to a multimodal setting, exploring its potential for blind models to supervise and enhance the performance of sighted ones.We focus on visual question answering (VQA), where two "sighted" expert vision-language models debate an answer, while a "blind" (textonly) judge adjudicates based solely on the quality of the arguments.In our framework, the experts only defend answers aligned with their beliefs, thereby obviating the need for explicit role-playing and concentrating the debate on instances of expert disagreement.Experiments on several multimodal tasks demonstrate that the debate framework consistently outperforms individual expert models.Moreover, judgments from blind LLMs can be used to instil reasoning capabilities in visionlanguage models through fine-tuning. 1
BibTeX
@misc{adhikari2025debating,
  author = {Ashutosh Adhikari and Mirella Lapata},
  title = {{Debating for Better Reasoning in Vision-Language Models}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.853},
}
Figure