TL;DR
When reasoning through complex problems, humans constantly look back at diagrams, but Multimodal Large Language Models (MLLMs) typically look at an image only once. v1 introduces a lightweight "point-and-copy" mechanism that allows an MLLM to dynamically re-reference input image patches and inject their embeddings back into its reasoning chain. This approach combats "visual grounding decay" and allows a 7B model to substantially outperform comparable baselines and rival models nearly ten times its size on multimodal math benchmarks.
What Problem Does This Paper Address?
Current MLLMs process image features up front and generate multi-step reasoning autoregressively in pure text. Due to causal attention masking, the model reasons primarily over the frozen key–value cache of earlier visual tokens.
The authors empirically show that this leads to visual grounding decay:
- Overall attention to image tokens steadily declines as reasoning sequences lengthen.
- The relative attention paid to task-relevant visual regions drops even faster, with salient regions eventually receiving less attention than background pixels (falling to an attention ratio of ~0.8).
As a result, during extended chain-of-thought (CoT) reasoning, models detach from visual evidence and hallucinate.
Methods
To let models actively revisit visual features, the authors introduce v1, an architecture extending standard MLLM backbones (instantiated on Qwen2.5-VL-7B):
- Point-and-Copy Mechanism: Inspired by Pointer-Generator Networks, v1 expands the output space $\bar{V} = V \cup C$ to include pointers to continuous input image patch embeddings $C = \{c_1, \dots, c_K\}$.
- Lightweight Dual Heads: Alongside the standard vocabulary prediction head, the model adds two linear projection layers—a query head $L_q$ and a key head $L_k$. Pointing logits over visual tokens are computed via scaled dot-product attention: $$\text{logit}_{\text{ptr}}^{(k)} = \frac{L_q(h_t) \cdot L_k(c_k)^\top}{\sqrt{D}}$$
- Dynamic Re-Injection: When a pointer token $\langle\text{ptr}: c_k\rangle$ is selected, its original continuous patch embedding $c_k$ is directly injected as the input embedding for the next decoding step.
- Stable Optimization: To manage the enlarged mixed output space, training incorporates an efficient top-40 approximate $z$-loss regularization term to stabilize the softmax partition function.
Datasets
- v1g (New Dataset): A dataset of 300K multimodal reasoning trajectories with step-by-step visual grounding annotations. Starting from raw CoT traces in the TVC dataset (generated by QvQ-72B), the authors used Gemini-2.0-Flash to rewrite visual references into executable
detect()calls. Regions were localized using relative cross-attention contrast maps ($R = A / A'$) extracted directly from Qwen2.5-VL. - Evaluation Benchmarks: Evaluated on three multimodal mathematical reasoning benchmarks: MathVista (mini), MathVision (mini and full), and MathVerse (mini). RefCOCO testA was used for the initial attention-decay probing experiments.
Key Insights and Ideas
- Perceptual Evidence in Native Representation: Unlike coordinate-based methods that force the model to output text coordinates (e.g.,
[x1, y1, x2, y2]), v1 directly retrieves the high-dimensional visual embedding into the context window, bypassing coordinate parsing errors and resolution mismatches. - Structured Internal Reasoning Dynamics: Tracking attention across layers revealed a three-phase cognitive loop:
- Localization: Attention to original image tokens spikes immediately before a copy action.
- Focused Processing: Intermediate layers (e.g., layers 2 and 14) shift their attention overwhelmingly to the copied token.
- Integration: Deeper layers (layer 27) balance attention across copied tokens and the broader original context to plan the next reasoning steps.
Key Results
- Benchmark Performance: Across all math benchmarks, v1-7B achieved top-tier performance for its size class:
- MathVision (mini / full): Reached 34.5% / 28.1%, outperforming the 7B text-only CoT baseline (22.7% full) and the base Qwen2.5-VL-7B (23.6% mini).
- MathVista (mini): Scored 68.6%, exceeding TVC-7B (68.1%) and LLaVA-CoT-11B (54.8%).
- MathVerse (mini): Scored 48.6%, beating InternVL2.5-8B (39.5%) and approaching TVC-72B (48.8%).
- Ablations on MathVision Mini:
- Base Qwen2.5-VL-7B: 23.6%
- Supervised with coordinate bounding boxes only: 31.9%
- v1 without pointing during inference: 25.3%
- v1 with active point-and-copy: 34.5%
- Grounding Quality: In human evaluation of localized regions, v1's attention-based annotation pipeline achieved 83.3% correctness, compared to just 29.3% for GroundingDINO on complex geometric and chart entities.
Limitations and Future Work
- Domain Breadth: The dataset and evaluation center predominantly on structured mathematical and geometric problems; generalizing to unstructured domains like general video or medical scans remains unexplored.
- Supervision Dependency: Training currently relies on a multi-stage automated distillation pipeline (Gemini parsing + Qwen attention maps). The authors highlight reinforcement learning or weak supervision as promising avenues to learn when and where to look without explicit ground-truth bounding boxes.
- Bounding Geometries: The current patch selection assumes rectangular bounding boxes, which may be suboptimal for irregular, non-convex visual objects.
Summary updated