TL;DR
This paper introduces AdaViG (Adaptive Visual Gating), a training-free, plug-and-play method that prevents Unified Multimodal Models (UMMs) from hallucinating harmful intermediate drawings during complex math reasoning. By analyzing internal attention signals just five steps into the image diffusion process, AdaViG dynamically aborts low-utility visual generations, boosting reasoning accuracy by up to 5.7% while slashing wall-clock latency by up to 45.6%.
What Problem Does This Paper Address?
Recent advances in multimodal large language models have popularized Multimodal Chain-of-Thought (MCoT) reasoning, where models generate interleaved text and images to solve complex problems. For geometry or calculus tasks, a model might natively "draw" an auxiliary line or transform a diagram to help itself reason.
However, the authors uncover a major pitfall in this paradigm: self-generated visual steps are frequently incorrect, redundant, or misleading. When a model generates a flawed diagram, it suffers from "attention drift"—subsequent text reasoning steps focus heavily on the erroneous self-generated image rather than the original correct input image, leading to a wrong final answer. Furthermore, constantly triggering full image diffusion processes introduces massive computational overhead, consuming up to 50% of end-to-end inference latency and 80% of extra KV-cache memory.
Methods
To solve these accuracy and efficiency challenges without expensive retraining, the authors propose AdaViG. The core idea is simple: evaluate the utility of a visual generation early in the drawing process, and kill it if it looks like a hallucination.
During the early stages of the shared Transformer's denoising diffusion process (specifically at step $t^\star = 5$ out of $T = 50$), AdaViG extracts two internal introspective signals from the VAE query attention weights:
- Generation Intent ($\mathcal{I}$): Measures whether the emerging image is heavily conditioned on the model's textual reasoning plan. A low $\mathcal{I}$ indicates "Intent Vacancy"—the model is drawing without a clear plan.
- Visual Fidelity ($\mathcal{F}$): Measures whether the generation remains grounded in the original input image. A low $\mathcal{F}$ indicates "Perception Drift"—the drawing is deviating dangerously from the problem geometry.
If both signals fall below conservative thresholds ($\mathcal{I} < \tau_I \wedge \mathcal{F} < \tau_F$), AdaViG immediately aborts the diffusion process, discards the partial VAE latents, and forces the model to fall back to a text-only reasoning path.
Datasets
The authors evaluated AdaViG across five challenging mathematical reasoning benchmarks:
- MathVista (GPS category)
- MathVerse (testmini subset)
- OlympiadBench (Math subset)
- WeMath
- DynaMath
Hyperparameter tuning for the thresholds ($\tau_I, \tau_F$) was conducted on the MathVision testmini dataset.
Key Insights and Ideas
- The Generation Utility Gap: Intrinsic image generation is a double-edged sword. While helpful in roughly 28% of cases, it is actively harmful in about 26% of cases, meaning its net accuracy benefit is often close to zero without a gate.
- Trace-Dependent Harmfulness: The harmfulness of an image step isn't determined by the problem itself, but rather emerges dynamically based on how the reasoning trace evolves at runtime.
- Attention Drift Failure Mode: Once a flawed image is generated, the model's subsequent text tokens allocate higher attention ratios to the bad generation than the original input diagram (in up to 92% of post-image tokens), locking in the error.
- Predictive Early Attention: The internal cross-modal attention maps within the first 10% of diffusion steps contain enough signal to accurately forecast whether the final image will be high-fidelity or a hallucinated mess.
Key Results
AdaViG was integrated into two state-of-the-art UMM frameworks: Bagel-Zebra-CoT and MathCanvas.
- Accuracy Boosts: AdaViG consistently improved accuracy across all benchmarks, achieving a peak improvement of +5.7% on DynaMath using the Bagel-Canvas model.
- Massive Compute Savings: By stopping bad drawings early, AdaViG reduced image-generation FLOPs by 25.0% to 91.0%.
- Latency & Throughput Gains: End-to-end wall-clock latency dropped by 15.4%–45.6%, resulting in up to a 1.4x faster overall sample throughput.
Limitations and Future Work
While AdaViG successfully filters out harmful visual steps, the authors note a few limitations:
- No Quality Enhancement: AdaViG acts strictly as a gate; it decides whether to draw, but it does not improve the rendering quality or capabilities of the underlying diffusion model when it does draw.
- Dependency on Trigger Frequency: On reasoning chains where intermediate visual generation is rarely triggered by the base model, the efficiency benefits are naturally minimized.
Future Direction: The authors suggest that instead of using Generation Intent and Visual Fidelity purely as inference-time gates, these signals could be integrated directly into the supervised fine-tuning (SFT) or reinforcement learning (RL) pipelines as training-time rewards to teach models how to draw better from the start.
Summary updated