TL;DR
Vision-Language Models (VLMs) often get the right answer for the wrong reasons in text-rich tasks—a phenomenon called Correct-but-Visually-Misgrounded Reasoning (CVMR). GRIT is a new training framework that identifies these "lucky guesses" by analyzing attention patterns and uses gradient masking to prevent the model from reinforcing its own hallucinations.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Rewards (RLVR) has been a game-changer for Large Language Models (like DeepSeek-R1), teaching them to "think" by rewarding correct final answers. However, when we apply this to Vision-Language Models (VLMs) dealing with text-rich images—like financial reports, charts, or dense documents—performance often plateaus or even degrades.
The authors identify a specific culprit: CVMR. In complex visual reasoning, a model might misread every number on a chart but, through sheer luck or linguistic priors, still output the correct final answer.
Because standard RLVR only cares about the final answer, it rewards these misgrounded intermediate steps. This effectively trains the model to be a "better hallucinator" rather than a better reader.
Methods
The authors propose GRIT (Gradient Masking for Visually Misgrounded Tokens), which introduces three clever components to the RLVR pipeline:
- Visual Tracker Heads: Not all attention heads are created equal. The authors automatically identify a subset of "tracker heads" that specifically shift their focus to different image regions as the model generates different tokens.
- Gradient Masking: By analyzing the attention patterns of these tracker heads, GRIT detects tokens that are likely misgrounded. Specifically, tokens with low attention mass (not looking at the image much) and high entropy (looking everywhere at once/diffuse focus) are flagged. During training, the gradients for these tokens are masked—meaning the model doesn't "learn" from those specific steps, even if the final answer is correct.
- Attention Reward: To further encourage good behavior, the authors add a reward signal that favors responses with high-quality visual grounding (high attention mass and low entropy).
Datasets
- Training: The models were trained on the ViRL 39K dataset, a standard collection for multimodal RL.
- Evaluation: The authors tested on existing benchmarks like OCR-Reasoning, ChartQA-PRO, and CharXiv.
- GRIT-Bench: A major contribution of this paper is a new benchmark created by rendering text-based QA datasets (like tables and financial docs) into images. Because the original source text is known, an LLM judge can precisely verify if the VLM's reasoning steps match the actual data in the image, allowing for a metric called Reasoning Fidelity (RF).
Key Insights and Ideas
- The CVMR Epidemic: In text-rich reasoning tasks, a staggering 35.3% of correct answers involve misgrounded reasoning. This is significantly higher than in simpler visual tasks.
- Attention is a "Tell": Well-grounded tokens exhibit "sharp" attention (high mass, low entropy), while misgrounded tokens look "blurry" to the model's internal mechanisms.
- Correctness is a Noisy Signal: In the multimodal world, a "correct" label is a weak supervisor. Without grounding-level intervention, RL can actually impair a model's ability to handle long-form text by reinforcing "lucky" shortcuts.
Key Results
GRIT shows that "less is more" when it comes to reinforcement. By refusing to learn from misgrounded tokens, the model becomes much more reliable.
- Substantial Gains: On the OCR-Reasoning benchmark, while other RL methods showed marginal or negative progress, GRIT-7B achieved a +4.96 point improvement over the base model.
- GRIT-Bench Performance: On the authors' new benchmark, GRIT-7B saw a massive +11.7 point jump in accuracy and a +5.9% increase in Reasoning Fidelity.
- Generalization: The method isn't tied to one architecture; the authors successfully transferred the technique to the newer Qwen3-VL family with similar success.
Limitations and Future Work
- Attention as a Proxy: While effective, attention statistics are an indirect measure of grounding. The authors acknowledge that attention doesn't perfectly determine if a token is grounded.
- Training Overhead: Materializing attention weights for masking adds some computational cost (about 1.29x the time of standard GRPO), though they mitigated this by only extracting weights from the five "tracker" layers.
- Future Directions: The authors suggest moving from fixed percentile-based masking to more sophisticated, learned "grounding-confidence" estimators. They also hope to extend the Reasoning Fidelity evaluation to real-world documents beyond their rendered benchmark.
Summary updated