Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning

Hongbin Lin, Sizhe Zou, Juangui Xu, Xinyue Xu, Zhuoli Ouyang, Stanislav Kolchin, Chengwei Qin, Zhongxiang Dai, Yao ShuarXiv 2026 · arXiv

Vision-Language Models (VLMs) are increasingly capable of complex reasoning, often by generating "Chain-of-Thought" (CoT) traces before arriving at an answer. However, forcing a model to "think out loud" for every single query—like identifying a cat in a clear photo—is a waste of compute and can even lead to "overthinking" errors. The ideal VLM should be adaptive: answering directly when possible and reasoning step-by-step only when necessary.

TL;DR

The authors introduce DR-BOLT (Dual-Reference BOLT), an offline optimization framework that trains VLMs to switch adaptively between "fast" direct answers and "slow" reasoning traces. By correcting for the statistical mismatch between different behavior policies (fast vs. slow), DR-BOLT achieves superior accuracy-to-length trade-offs and a 2.9x training speedup compared to online reinforcement learning methods like GRPO.

What Problem Does This Paper Address?

The core challenge is adaptive thinking. While we want models to use both fast and slow modes, training them to do so is tricky.

Existing offline alignment methods, such as BOLT, typically assume that all training samples come from a single "reference" policy. In the adaptive setting, however, "fast" answers and "slow" reasoning traces are essentially generated by two different behaviors. If you try to force these into a single-reference objective, the math breaks: the model often collapses into one mode, either becoming "always-fast" (losing reasoning power) or "always-slow" (becoming inefficient).

The authors identify this as a dual-reference optimization problem. They aim to create a model that internalizes "when to think" without needing a separate, external router or expensive online reinforcement learning (RL).

Methods

The researchers propose DUAL-REFERENCE BOLT (DR-BOLT). Here are the key technical components:

  1. Dual-Reference Objective: Instead of a single reference policy, they define a desired target as a mixture of a "fast" policy ($\pi_f$) and a "slow" policy ($\pi_s$).
  2. Behavior-Reference Correction: Because the actual data-sampling ratio (e.g., 70% fast, 30% slow) might differ from the desired target mixture (e.g., 50/50), they derive a correction factor ($\kappa$) to reweight the offline training samples. This prevents the model from being biased toward whichever mode was more prevalent in the training bank.
  3. Partition Competition: The paper proves that the model's "routing" behavior emerges naturally from the competition between the reward-weighted partition functions of the two modes. If the "slow" mode provides a significantly higher correctness reward that outweighs its length penalty, the model will naturally choose to reason.
  4. Mode-Wise Approximation: To avoid expensive likelihood evaluations of every sample under both policies, the authors use "mode-format separation." Since slow responses usually start with a specific prefix (like <think>), they can efficiently approximate the correction factor based on the response format.

Datasets

The model was trained and evaluated on several rigorous multimodal reasoning benchmarks:

  • ScienceQA: Used for in-distribution (IID) evaluation.
  • ViRL39K: A large-scale reasoning dataset used for training the transfer models.
  • MathVision, DynaMath, and MMMU-Pro: Used as held-out transfer benchmarks to test the model's robustness across different types of mathematical and scientific reasoning.

Key Insights and Ideas

  • The Mismatch Diagnosis: The most significant "aha moment" is the realization that single-reference offline objectives systematically downweight one response family. If you use a "fast" reference to evaluate a "slow" trace, the likelihood is near zero, effectively deleting that sample's influence on the gradient.
  • Router-Free Intelligence: Adaptive behavior doesn't require a separate "is this hard?" classifier. By optimizing against a mixture, the model's internal policy learns to route based on the expected reward of the output format.
  • Efficiency via Offline Training: Unlike online RL (like GRPO), which requires constant, expensive model rollouts during training, DR-BOLT is offline. You generate a bank of candidates once, score them, and then perform weighted supervised fine-tuning.

Key Results

DR-BOLT consistently outperformed both online RL baselines and other offline alternatives.

Table 1: Comparison of DR-BOLT against online RL baselines (GRPO, FAST-GRPO, TON, DualMindVLM).
  • Performance vs. Efficiency: As shown in the table above, DR-BOLT achieved the highest scores across all benchmarks while maintaining significantly shorter response lengths than standard GRPO. For example, on MathVision, it scored 0.606 with a length of 1058, whereas GRPO scored 0.463 with a massive length of 2498.
  • Training Speed: DR-BOLT achieved a 2.90x end-to-end training speedup over reproduced GRPO, largely because it avoids the overhead of online rollouts.
  • Adaptive Routing: The model achieved a cross-benchmark "slow ratio" of 24.1%. This means it chose to reason for about a quarter of the problems (presumably the harder ones) and answered directly for the rest.
Table 2: Matched internal comparisons. Cells report Score/Length in words; ScienceQA and cross- benchmark results use separately trained checkpoints. Slow ratio (%) aggregates only MathVision, DynaMath, and MMMU-Pro.

Limitations and Future Work

While DR-BOLT is a major step forward for efficient, adaptive VLMs, the authors note several areas for further exploration:

  • Two-Mode Constraint: The current work focuses on a binary "fast vs. slow" choice. Real-world applications might benefit from multiple levels of reasoning depth or "thinking budgets."
  • Bank Dependence: As an offline method, the model's ultimate performance is capped by the quality and coverage of the initial candidate response bank. It cannot "discover" entirely new reasoning strategies that weren't present in the sampling phase.
  • Format Ambiguity: The efficient approximation relies on clear format separation (e.g., the <think> tag). If models start blending styles or using inconsistent formatting, the simpler correction method may falter.

Summary updated

Abstract

Reasoning traces improve vision-language models on complex multimodal tasks, but forcing every input to follow a long reasoning process leads to overthinking and unnecessary inference cost. We study adaptive fast/slow thinking for VLMs, where one model answers directly for simple inputs and reasons step by step when needed. Fast answers and slow traces often come from distinct behavior policies, making a single-reference offline objective misweight one mode. We propose DUAL-REFERENCE BOLT (DR-BOLT), which distinguishes the desired fast/slow mixture reference from the sampling distribution and corrects their mismatch in reward-weighted training. Fast/slow partition competition explains routing in the optimal target; an efficient source-wise correction approximates the exact objective when response formats are well separated. Matched Qwen3-VL comparisons use identical candidate banks to isolate the objective from slow-teacher quality. DR-BOLT improves score-length trade-offs over the evaluated offline and online baselines on ScienceQA and three held-out transfer benchmarks, with a 24.1% cross-benchmark slow ratio and a 2.90-fold end-to-end training speedup over reproduced GRPO. These results support adaptive reasoning in the studied two-reference setting, while broader robustness remains open. Our code is available at https://github.com/HongbinLin3589/DR-BOLT.
BibTeX
@misc{lin2026learning,
  author = {Hongbin Lin and Sizhe Zou and Juangui Xu and Xinyue Xu and Zhuoli Ouyang and Stanislav Kolchin and Chengwei Qin and Zhongxiang Dai and Yao Shu},
  title = {{Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning}},
  year = {2026},
  eprint = {2611.09154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09154},
}
Figure