Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces

John Gkountouras, Ivan Titov

Introduction

Modern AI systems often tackle complex visual tasks using a decoupled architecture: a vision model generates a text description (caption), which a text-only Large Language Model (LLM) then uses to solve the problem. While this approach is modular and interpretable, it frequently suffers from an interface mismatch. Vision models, typically trained on human-centric captions, often omit the granular, quantitative details—such as exact coordinates or specific angle measurements—that a mathematical reasoner requires. Manually annotating "perfect" captions for every possible task is an unscalable challenge.

Adaptive-Clarification Reinforcement Learning (AC-RL)

To bridge this gap, this paper introduces Adaptive-Clarification Reinforcement Learning (AC-RL). This framework treats the interaction between the vision model (the "captioner") and the reasoner as a reinforcement learning problem, where the reasoner's needs provide implicit supervision.

The Training Scaffold and Tiered Rewards

During training, the system is allowed a multi-turn dialogue. If the reasoner finds the initial caption insufficient, it can issue a clarification request (e.g., "What is the value of angle BCA?"). A frozen reference model provides the missing information, allowing the reasoner to attempt the problem again. To encourage the captioner to be comprehensive from the start, the authors implement a tiered reward structure:

  • $R = 1.0$: Correct answer achieved in a single pass (no clarification).
  • $R = 0.7$: Correct answer achieved after clarification.
  • $R = 0$: Incorrect answer.
The AC-RL framework: Direct Answer Path vs. Clarification Path with tiered rewards.

By imposing a "clarification tax" (the 0.3 reward penalty), the model is pressured to front-load all necessary information into its initial output. At inference, the clarification mechanism is removed, and the model must rely on the more detailed captioning strategy learned during training.

Experimental Results

The AC-RL framework was evaluated using a Qwen-3B captioner across seven diverse benchmarks, including MathVista, MathVision, and DynaMath. The model was trained on ViRL-39K, a visual instruction dataset focused on mathematical reasoning.

Performance Gains

AC-RL demonstrated significant improvements over standalone VLMs and decoupled baselines. On average, the framework improved accuracy by 4.4 points (from 39.0% to 43.4%). The most substantial gains were observed in vision-centric and quantitatively-intensive domains, such as DynaMath (+10.6 points) and MathVerse (+7.9 points).

Performance comparison of AC-RL against proprietary and open-weights models across seven mathematical reasoning benchmarks.

Efficiency and Diversity

Beyond raw accuracy, AC-RL proved highly effective at reducing information gaps. Analysis showed that if clarification were allowed at test time, the AC-RL model would have required 39% fewer follow-up questions on the MathVerse dataset compared to standard training methods. Furthermore, the tiered reward system helped maintain higher generation diversity, preventing the model from collapsing into repetitive, low-information captioning styles.

Discussion and Future Work

The core insight of this work is that clarification acts as implicit supervision. Instead of requiring explicit human feedback, the reasoner’s own confusion identifies exactly where the information gap lies. This "densifies" sparse rewards, providing a smoother learning gradient than binary success/failure signals.

However, the authors noted that AC-RL is primarily effective at improving perception (the "what") rather than reasoning steps (the "why"). Future work may explore bidirectional adaptation, where both the vision model and the reasoner co-evolve, or multi-turn dialogues with decaying rewards for each additional hint required.

Summary updated

Abstract

Recent text-only models demonstrate remarkable mathematical reasoning capabilities. Extending these to visual domains requires vision-language models to translate images into text descriptions. However, current models, trained to produce captions for human readers, often omit the precise details that reasoning systems require. This creates an interface mismatch: reasoners often fail not due to reasoning limitations but because they lack access to critical visual information. We propose Adaptive-Clarification Reinforcement Learning (AC-RL), which teaches vision models what information reasoners need through interaction. Our key insight is that clarification requests during training reveal information gaps; by penalizing success that requires clarification, we create pressure for comprehensive initial captions that enable the reasoner to solve the problem in a single pass. AC-RL improves average accuracy by $4.4$ points over pretrained baselines across seven visual mathematical reasoning benchmarks, and analysis shows it would cut clarification requests by up to $39$\% if those were allowed. By treating clarification as a form of implicit supervision, AC-RL demonstrates that vision-language interfaces can be effectively learned through interaction alone, without requiring explicit annotations.
BibTeX
@misc{gkountouras2025clarification,
  author = {John Gkountouras and Ivan Titov},
  title = {{Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces}},
  year = {2025},
  eprint = {2509.26594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.26594},
}
Figure