GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning

Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin RoarXiv 2026 · arXiv

TL;DR

Vision-Language Models (VLMs) often get the right answer for the wrong reasons in text-rich tasks—a phenomenon called Correct-but-Visually-Misgrounded Reasoning (CVMR). GRIT is a new training framework that identifies these "lucky guesses" by analyzing attention patterns and uses gradient masking to prevent the model from reinforcing its own hallucinations.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) has been a game-changer for Large Language Models (like DeepSeek-R1), teaching them to "think" by rewarding correct final answers. However, when we apply this to Vision-Language Models (VLMs) dealing with text-rich images—like financial reports, charts, or dense documents—performance often plateaus or even degrades.

The authors identify a specific culprit: CVMR. In complex visual reasoning, a model might misread every number on a chart but, through sheer luck or linguistic priors, still output the correct final answer.

Figure 2. An example of Correct-but-Visually-Misgrounded Reasoning (CVMR). The model correctly identifies the relative ranking, noting that Health Care has the highest representation in the infographic, and arrives at the correct answer. However, despite multiple industries having meaningful percentages, the model misreads all values including Health Care itself. Red-highlighted text indicates visually misgrounded reasoning.

Because standard RLVR only cares about the final answer, it rewards these misgrounded intermediate steps. This effectively trains the model to be a "better hallucinator" rather than a better reader.

Methods

The authors propose GRIT (Gradient Masking for Visually Misgrounded Tokens), which introduces three clever components to the RLVR pipeline:

  1. Visual Tracker Heads: Not all attention heads are created equal. The authors automatically identify a subset of "tracker heads" that specifically shift their focus to different image regions as the model generates different tokens.
  2. Gradient Masking: By analyzing the attention patterns of these tracker heads, GRIT detects tokens that are likely misgrounded. Specifically, tokens with low attention mass (not looking at the image much) and high entropy (looking everywhere at once/diffuse focus) are flagged. During training, the gradients for these tokens are masked—meaning the model doesn't "learn" from those specific steps, even if the final answer is correct.
  3. Attention Reward: To further encourage good behavior, the authors add a reward signal that favors responses with high-quality visual grounding (high attention mass and low entropy).

Datasets

  • Training: The models were trained on the ViRL 39K dataset, a standard collection for multimodal RL.
  • Evaluation: The authors tested on existing benchmarks like OCR-Reasoning, ChartQA-PRO, and CharXiv.
  • GRIT-Bench: A major contribution of this paper is a new benchmark created by rendering text-based QA datasets (like tables and financial docs) into images. Because the original source text is known, an LLM judge can precisely verify if the VLM's reasoning steps match the actual data in the image, allowing for a metric called Reasoning Fidelity (RF).

Key Insights and Ideas

  • The CVMR Epidemic: In text-rich reasoning tasks, a staggering 35.3% of correct answers involve misgrounded reasoning. This is significantly higher than in simpler visual tasks.
Figure 3. CVMR rate among correct responses by task type. Text- rich reasoning exhibits a substantially higher CVMR rate com- pared to other task types.
  • Attention is a "Tell": Well-grounded tokens exhibit "sharp" attention (high mass, low entropy), while misgrounded tokens look "blurry" to the model's internal mechanisms.
  • Correctness is a Noisy Signal: In the multimodal world, a "correct" label is a weak supervisor. Without grounding-level intervention, RL can actually impair a model's ability to handle long-form text by reinforcing "lucky" shortcuts.

Key Results

GRIT shows that "less is more" when it comes to reinforcement. By refusing to learn from misgrounded tokens, the model becomes much more reliable.

  • Substantial Gains: On the OCR-Reasoning benchmark, while other RL methods showed marginal or negative progress, GRIT-7B achieved a +4.96 point improvement over the base model.
  • GRIT-Bench Performance: On the authors' new benchmark, GRIT-7B saw a massive +11.7 point jump in accuracy and a +5.9% increase in Reasoning Fidelity.
  • Generalization: The method isn't tied to one architecture; the authors successfully transferred the technique to the newer Qwen3-VL family with similar success.
Figure 1. Performance change over the base model on text-rich reasoning benchmarks. Existing RLVR methods mostly degrade or show marginal improvement, while GRIT achieves substantial gains.

Limitations and Future Work

  • Attention as a Proxy: While effective, attention statistics are an indirect measure of grounding. The authors acknowledge that attention doesn't perfectly determine if a token is grounded.
  • Training Overhead: Materializing attention weights for masking adds some computational cost (about 1.29x the time of standard GRPO), though they mitigated this by only extracting weights from the five "tracker" layers.
  • Future Directions: The authors suggest moving from fixed percentile-based masking to more sophisticated, learned "grounding-confidence" estimators. They also hope to extend the Reasoning Fidelity evaluation to real-world documents beyond their rendered benchmark.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently been extended to vision-language models (VLMs), yet its effectiveness remains limited for text-rich reasoning tasks. We highlight a key factor contributing to this limitation: models frequently arrive at correct final answers despite misperceiving visual information during intermediate steps. We define this phenomenon as Correct-but-Visually-Misgrounded Reasoning (CVMR). Because standard RLVR rewards only final-answer correctness, it fails to differentiate between properly grounded and misgrounded reasoning, reinforcing both equally. Through token-level analysis, we reveal that CVMR tokens exhibit distinct attention patterns that differ systematically from well-grounded ones. Based on this observation, we propose GRIT, a training framework that incorporates visual grounding into RLVR to mitigate the impact of CVMR. Furthermore, to address the lack of text-rich reasoning benchmarks that allow for systematic evaluation across text length and reasoning depth, we introduce GRIT-Bench. Experiments demonstrate that GRIT significantly improves both reasoning accuracy and grounding fidelity over existing RLVR methods.
BibTeX
@misc{cho2026grit,
  author = {Chanseul Cho and Noori Bae and Dongheon Lee and Youngmin Ro},
  title = {{GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning}},
  year = {2026},
  eprint = {2610.41629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41629},
}
Figure