Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces

John Gkountouras, Ivan TitovSubmission to ICLR 2026 · Rejected

TL;DR

Pairing lightweight vision-language models with powerful, frozen text reasoners is a practical way to solve visual reasoning tasks, but standard captions often omit the granular details that reasoners need. Adaptive-Clarification Reinforcement Learning (AC-RL) uses an interactive training scaffold where downstream clarification requests provide partial credit ($R=0.7$) versus single-pass success ($R=1.0$). By penalizing clarification-dependent success during training, the vision model learns to proactively "front-load" critical visual details, boosting single-pass test-time accuracy by an average of 4.4 points across seven benchmarks without human caption annotations.

What Problem Does This Paper Address?

Decoupled "vision-to-text + text reasoner" pipelines offer compelling modularity: they let developers reuse state-of-the-art text reasoners (like DeepSeek-R1) via frozen APIs while using small, cheap vision-language models (VLMs) for perception.

However, this setup suffers from an interface mismatch. Pretrained VLMs are optimized for human readability, so their captions frequently gloss over exact measurements, coordinates, and structural nuances. When the text reasoner fails, it is often not because it cannot reason, but because the caption withheld critical information. Training the captioner via reinforcement learning (RL) using binary task rewards ($1$ for correct, $0$ for incorrect) typically fails due to severe reward sparsity: producing a completely irrelevant caption receives the exact same zero reward as producing a caption missing just one vital number.

Methods

AC-RL introduces an interactive training scaffold that treats clarification as implicit supervision:

  1. Interactive Training Scaffold:
  2. The trainable captioner $V_\theta$ generates an initial description $c_0$.
  3. A frozen reasoner $\mathcal{R}$ evaluates the description. It either solves the problem directly or asks a specific follow-up question $q_1$.
  4. If clarification is requested, a frozen reference captioner $\pi_{\text{ref}}$ answers ($c_1$). The reasoner then predicts a final answer using the full dialogue.
  5. Gradients flow only through the initial caption generation $c_0$, ensuring the captioner cannot rely on the clarifier to do the work.
  6. Tiered Clarification-Aware Reward: The reward function densifies the learning signal into three tiers: $$R(\tau) = \begin{cases} 1.0 & \text{if correct without clarification} \\ \alpha & \text{if correct with clarification } (\alpha = 0.7) \\ 0 & \text{if incorrect} \end{cases}$$ The partial reward $\alpha$ acknowledges that $c_0$ provided useful context, while the $(1 - \alpha)$ penalty forces the model to discover self-sufficient descriptions.

  7. Optimization & Inference:

  8. Optimization uses Beta-Normalization Policy Optimization (BNPO) (a GRPO variant fitted with Beta distributions for bounded rewards) combined with LoRA and KL regularization.
  9. At inference time, clarification is completely disabled: the captioner operates strictly in a single pass.
[Image + Question]
        │
        ▼
 Trainable Captioner (V_θ) ──► Initial Caption (c_0)
                                      │
                                      ▼
                            Frozen Reasoner (R)
                               /              \
     [Direct Answer Path]     /                \    [Clarification Path (Train Only)]
                             ▼                  ▼
                       R = 1 / R = 0       Asks follow-up question (q_1)
                                                │
                                                ▼
                                    Frozen Reference VLM (π_ref)
                                                │
                                                ▼
                                    Provides clarification (c_1)
                                                │
                                                ▼
                                       Final Answer by R
                                                │
                                                ▼
                                         R = α / R = 0

Datasets

  • Training:
  • ViRL-39K: A visual instruction dataset tailored for mathematical reasoning.
  • Evaluation (7 Benchmarks):
  • MathVista (testmini), MathVision, MathVerse (and MathVerse_MINI), MMMU (dev/val), WeMath (testmini, strict), DynaMath (worst-case), and LogicVista.
  • No new datasets were introduced; the authors used established visual math and logic benchmarks where ground-truth answers enable clean credit assignment.

Key Insights and Ideas

  • Clarification as a Learning Scaffold: Needing clarification indicates an information gap, not total failure. Converting these episodes into intermediate rewards creates a graded feedback landscape that drastically improves RL exploration and prevents generation diversity collapse (Figure 4).
  • Information "Front-Loading": The penalty $(1 - \alpha)$ successfully trains the captioner to anticipate what the reasoner will need. During training, clarification requests dropped from ~65% to ~20%.
  • Selective Remaining Requests: When tested in clarification-enabled settings, AC-RL reduced clarification requests by 29% on MathVision and 39% on MathVerse. Denying clarification on AC-RL's remaining requests caused a steeper drop ($\Delta_{\text{deny}} = 14.21$ vs. $11.43$), proving the model only asked for help when information was genuinely missing from the context.
  • Reasoner Generalization: A captioner trained against DeepSeek-R1-Qwen-32B generalized directly to an unseen 8B reasoner (+8.7 on LogicVista, +6.5 on MathVision), proving it learned objective information-completeness rather than idiosyncratic communication tricks.

Key Results

  • Overall Benchmark Improvements:
  • Using a Qwen2.5-VL-3B captioner with a frozen DeepSeek-R1-32B reasoner, AC-RL achieved 43.4% average accuracy (+4.4 points over the 39.0% zero-shot pretrained baseline, and +3.1 over binary-reward RL at 40.3%).
  • Using InternVL-2B, performance reached 42.6% (+3.3 points over baseline).
  • Massive Extraction Gains:
  • Largest leaps occurred on diagram- and visual-heavy tasks: DynaMath (+10.6 points over baseline; +7.2 over binary RL) and MathVerse (+5.2 points). Per-subject analysis showed the biggest spikes in metric/solid geometry and algebra.
  • Ablation on Penalty $\alpha$: Intermediate penalties ($\alpha = 0.5$ to $0.7$, corresponding to $1-\alpha \in [0.3, 0.5]$) performed best; setting the penalty too low (0.1) provided inadequate optimization pressure, while setting it too high (0.7) collapsed back to sparse binary RL.

Limitations and Future Work

  • Reasoner Knowledge Deficits: AC-RL fixes visual extraction bottlenecks, but cannot fix internal knowledge gaps inside the reasoner. On benchmarks like WeMath (which test missing mathematical formulas/theorems), AC-RL showed minor regressions ($-0.7$ to $-1.5$ points).
  • Unilateral Adaptation: The framework currently assumes a completely frozen reasoner. The authors cite bidirectional co-evolution (training both reasoner and captioner) as an open direction.
  • Narrow Domain Scope: Empirically validated on visual mathematical and logic reasoning. Validating the framework in other specialized decoupled settings—such as medical imaging or CAD/engineering diagrams—remains future work.

Summary updated

Abstract

Recent text-only models demonstrate remarkable mathematical reasoning capabilities. Extending these to visual domains requires vision-language models to translate images into text descriptions. However, current models, trained to produce captions for human readers, often omit the precise details that reasoning systems require. This creates an interface mismatch: reasoners often fail not due to reasoning limitations but because they lack access to critical visual information. We propose Adaptive-Clarification Reinforcement Learning (AC-RL), which teaches vision models what information reasoners need through interaction. Our key insight is that clarification requests during training reveal information gaps; by penalizing success that requires clarification, we create pressure for comprehensive initial captions that enable the reasoner to solve the problem in a single pass. AC-RL improves average accuracy by $4.4$ points over pretrained baselines across seven visual mathematical reasoning benchmarks, and analysis shows it would cut clarification requests by up to $39$\% if those were allowed. By treating clarification as a form of implicit supervision, AC-RL demonstrates that vision-language interfaces can be effectively learned through interaction alone, without requiring explicit annotations.
BibTeX
@misc{gkountouras2025clarification,
  author = {John Gkountouras and Ivan Titov},
  title = {{Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces}},
  year = {2025},
  eprint = {2509.26594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.26594},
}
Figure