Model Guides You How to Draw: Adaptive Visual Gating for Unified Multimodal Reasoning

Wenxi Gao, Guanxi Lu, Didi Zhu, Hao Mark Chen, Quan Deng, Zhican Wang, Jiankang Deng, Hongxiang FanarXiv 2026 · arXiv

TL;DR

This paper introduces AdaViG (Adaptive Visual Gating), a training-free, plug-and-play method that prevents Unified Multimodal Models (UMMs) from hallucinating harmful intermediate drawings during complex math reasoning. By analyzing internal attention signals just five steps into the image diffusion process, AdaViG dynamically aborts low-utility visual generations, boosting reasoning accuracy by up to 5.7% while slashing wall-clock latency by up to 45.6%.

What Problem Does This Paper Address?

Recent advances in multimodal large language models have popularized Multimodal Chain-of-Thought (MCoT) reasoning, where models generate interleaved text and images to solve complex problems. For geometry or calculus tasks, a model might natively "draw" an auxiliary line or transform a diagram to help itself reason.

However, the authors uncover a major pitfall in this paradigm: self-generated visual steps are frequently incorrect, redundant, or misleading. When a model generates a flawed diagram, it suffers from "attention drift"—subsequent text reasoning steps focus heavily on the erroneous self-generated image rather than the original correct input image, leading to a wrong final answer. Furthermore, constantly triggering full image diffusion processes introduces massive computational overhead, consuming up to 50% of end-to-end inference latency and 80% of extra KV-cache memory.

Figure 1: A representative harmful generation case from MathVision under the fixed MathCanvas paradigm exhibiting two failure modes (left), and how AdaViG adaptively aborts it (right).

Methods

To solve these accuracy and efficiency challenges without expensive retraining, the authors propose AdaViG. The core idea is simple: evaluate the utility of a visual generation early in the drawing process, and kill it if it looks like a hallucination.

During the early stages of the shared Transformer's denoising diffusion process (specifically at step $t^\star = 5$ out of $T = 50$), AdaViG extracts two internal introspective signals from the VAE query attention weights:

  1. Generation Intent ($\mathcal{I}$): Measures whether the emerging image is heavily conditioned on the model's textual reasoning plan. A low $\mathcal{I}$ indicates "Intent Vacancy"—the model is drawing without a clear plan.
  2. Visual Fidelity ($\mathcal{F}$): Measures whether the generation remains grounded in the original input image. A low $\mathcal{F}$ indicates "Perception Drift"—the drawing is deviating dangerously from the problem geometry.

If both signals fall below conservative thresholds ($\mathcal{I} < \tau_I \wedge \mathcal{F} < \tau_F$), AdaViG immediately aborts the diffusion process, discards the partial VAE latents, and forces the model to fall back to a text-only reasoning path.

How early diffusion signals separate helpful and harmful visual traces

Datasets

The authors evaluated AdaViG across five challenging mathematical reasoning benchmarks:

  • MathVista (GPS category)
  • MathVerse (testmini subset)
  • OlympiadBench (Math subset)
  • WeMath
  • DynaMath

Hyperparameter tuning for the thresholds ($\tau_I, \tau_F$) was conducted on the MathVision testmini dataset.

Key Insights and Ideas

  • The Generation Utility Gap: Intrinsic image generation is a double-edged sword. While helpful in roughly 28% of cases, it is actively harmful in about 26% of cases, meaning its net accuracy benefit is often close to zero without a gate.
  • Trace-Dependent Harmfulness: The harmfulness of an image step isn't determined by the problem itself, but rather emerges dynamically based on how the reasoning trace evolves at runtime.
  • Attention Drift Failure Mode: Once a flawed image is generated, the model's subsequent text tokens allocate higher attention ratios to the bad generation than the original input diagram (in up to 92% of post-image tokens), locking in the error.
  • Predictive Early Attention: The internal cross-modal attention maps within the first 10% of diffusion steps contain enough signal to accurately forecast whether the final image will be high-fidelity or a hallucinated mess.

Key Results

AdaViG was integrated into two state-of-the-art UMM frameworks: Bagel-Zebra-CoT and MathCanvas.

  • Accuracy Boosts: AdaViG consistently improved accuracy across all benchmarks, achieving a peak improvement of +5.7% on DynaMath using the Bagel-Canvas model.
  • Massive Compute Savings: By stopping bad drawings early, AdaViG reduced image-generation FLOPs by 25.0% to 91.0%.
  • Latency & Throughput Gains: End-to-end wall-clock latency dropped by 15.4%–45.6%, resulting in up to a 1.4x faster overall sample throughput.
Figure 2: Accuracy and efficiency im- provements over MathCanvas.

Limitations and Future Work

While AdaViG successfully filters out harmful visual steps, the authors note a few limitations:

  • No Quality Enhancement: AdaViG acts strictly as a gate; it decides whether to draw, but it does not improve the rendering quality or capabilities of the underlying diffusion model when it does draw.
  • Dependency on Trigger Frequency: On reasoning chains where intermediate visual generation is rarely triggered by the base model, the efficiency benefits are naturally minimized.

Future Direction: The authors suggest that instead of using Generation Intent and Visual Fidelity purely as inference-time gates, these signals could be integrated directly into the supervised fine-tuning (SFT) or reinforcement learning (RL) pipelines as training-time rewards to teach models how to draw better from the start.

Summary updated

Abstract

Unified multimodal models (UMMs) with interleaved reasoning, which generate both textual and visual steps as part of intermediate reasoning traces, have demonstrated great potential for visual mathematical reasoning tasks. However, we identify a key insight in this paradigm: generating intermediate visual reasoning steps is not always beneficial and can even be harmful, as self-generated visual steps may introduce erroneous visual evidence that misleads subsequent reasoning. Moreover, frequently triggering visual steps during reasoning incurs substantial computational and memory overhead, degrading inference efficiency. To address these accuracy and efficiency challenges, we observe that the model's internal signals can indicate whether a visual step will benefit reasoning before the entire visual generation is completed. Specifically, this work identifies two internal signals: 1) Generation Intent, which reflects whether the model has a concrete textual plan for what to draw, and 2) Visual Fidelity, which measures whether the visual generation remains grounded in the original input image. Leveraging these internal signals, we propose AdaViG, a training-free adaptive visual gating method for unified multimodal reasoning. AdaViG dynamically evaluates each triggered visual step at an early visual generation stage and aborts it when both signals are weak, thereby preventing misleading visual evidence from entering the reasoning trace while avoiding unnecessary computation. Comprehensive experiments demonstrate that AdaViG improves accuracy by up to 5.7% while reducing visual generation FLOPs by 25.0%-91.0% and wall-clock latency by 15.4%-45.6%.
BibTeX
@misc{gao2026model,
  author = {Wenxi Gao and Guanxi Lu and Didi Zhu and Hao Mark Chen and Quan Deng and Zhican Wang and Jiankang Deng and Hongxiang Fan},
  title = {{Model Guides You How to Draw: Adaptive Visual Gating for Unified Multimodal Reasoning}},
  year = {2026},
  eprint = {2607.10004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.10004},
}
Figure