Grounding Multimodal Reasoning with Evidence-Ablated Negatives

Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Heng HuangarXiv 2026 · arXiv

TL;DR

EVAN (EVidence-Ablated Negatives) is a reinforcement learning framework designed to stop Large Multimodal Models (LMMs) from "cheating" by relying on text-based shortcuts instead of visual evidence. By identifying and masking the most critical parts of an image to elicit plausible but ungrounded responses, EVAN creates high-quality negative samples that train the model to genuinely ground its reasoning in the visual input.

What Problem Does This Paper Address?

Large Multimodal Models (LMMs) are often just powerful Large Language Models (LLMs) with a vision encoder attached. A recurring issue in the field is modality imbalance: the language backbone is so strong that the model often ignores the image entirely, relying on linguistic priors or dataset shortcuts to guess the correct answer.

Current Reinforcement Learning with Verifiable Rewards (RLVR) techniques—like the popular Group Relative Policy Optimization (GRPO)—typically only reward the final answer's correctness. This creates a "credit assignment ambiguity." If a model gets a math problem right because it recognized a common pattern in the text rather than looking at the diagram, it still receives a positive reward. This leads to brittle models that hallucinate when the visual evidence doesn't match their internal text-based expectations.

Methods

The authors propose a two-stage framework called EVAN:

  1. Evidence-Ablated Data Mining:
    • Scaled Image Information Gain (scaled-IIG): The framework first measures how much a model's response actually depends on the image by comparing the likelihood of the correct answer with the original image versus a blank one.
  2. RankMix Selection: To avoid focusing only on a few easy categories, EVAN uses a mixed-ranking strategy to select a diverse subset of visually dependent training examples.
  3. Adversarial Masking: For these selected examples, EVAN calculates the gradient of the scaled-IIG score with respect to the image patches. It then masks (blacks out) the top 50% most influential patches, effectively "blinding" the model to the critical evidence needed for the task.
  4. Stable Policy Optimization:
    • During training, the model generates responses for both the clean image and the "evidence-ablated" image.
  5. Responses generated from the ablated image that still look plausible (pass a likelihood filter) are treated as hard negatives.
  6. These negatives are optimized under the clean image context with a bounded negative advantage. This teaches the model: "Even if this answer sounds plausible based on the text, you must suppress it because the visual evidence (now visible) doesn't support it."

Datasets

The models were trained on the ViRL39K dataset. Evaluation was conducted across a comprehensive suite of nine multimodal reasoning benchmarks:

  • Mathematics & Geometry: Geometry3K, MathVista, MathVerse, We-Math, DynaMath, MathVision, and MMK12.
  • Multi-disciplinary Reasoning: MMMU-Pro.
  • Logical Reasoning: LogicVista.
  • Transfer Tasks: ChartQA, GQA, and Visual CounterFact.

Key Insights and Ideas

  • Plausible Negatives > Random Failures: The most effective way to improve a model is to penalize it when it is "confidently wrong" or "right for the wrong reasons." Random gibberish doesn't provide a useful learning signal; a fluent, logical-sounding response that ignores the image does.
  • Targeted Intervention: Masking images randomly is inefficient. By using gradients to find "query-critical" patches, EVAN specifically targets the visual evidence the model should have been using.
  • Stability is Crucial: Directly training on negatives can be unstable. EVAN uses "bounded negative advantages" to ensure that these auxiliary signals don't overwhelm the standard RL updates, preventing the model's policy from collapsing.

Key Results

EVAN consistently outperformed standard RLVR baselines across different model scales and families. As shown in the comparison table below, EVAN achieved the best average performance in every size group tested.

Table 1: Comparison with state-of-the-art open-source reasoning LMMs on multimodal reasoning benchmarks. All methods are evaluated under the same evaluation configuration. The best perfor- mance is marked in bold.
  • Qwen2.5-VL-3B: Improved from 50.3% (DAPO baseline) to 51.6% average accuracy.
  • Qwen2.5-VL-7B: Improved from 56.5% to 57.9%, outperforming strong competitors like VPPO-7B.
  • Qwen3-VL-8B: Reached 73.0% average accuracy, a significant jump over the 71.9% baseline.
  • Qualitative Grounding: In case studies involving train schedules and counting, EVAN-trained models were less likely to fall for "textual templates" (like simply subtracting two nearby numbers) and more likely to follow the row-and-column logic required by the visual table.

Limitations and Future Work

The authors acknowledge that while EVAN improves grounding, it is not a complete solution to the general limitations of LMMs. The framework is inherently tied to the quality of the reward design and the specific benchmarks used.

Future work could explore:

  • Applying EVAN to more diverse, open-ended settings beyond structured reasoning.
  • Refining the reward functions to better capture faithfulness and robustness.
  • Investigating how these techniques might scale to even larger models or different modalities (like video).

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful tool for improving multimodal reasoning. However, standard outcome supervision suffers from credit assignment ambiguity, as a model may receive positive rewards by exploiting language priors or dataset shortcuts rather than grounding its reasoning in the requisite visual evidence. This makes high-quality negative responses critical, since effective negatives should remain plausible under the current policy while exposing the shortcut that should be suppressed. In this paper, we propose EVAN, a multimodal reinforcement learning framework utilizing EVidence-Ablated Negatives to train LMMs against their own language-prior failures. EVAN first identifies highly visually dependent examples using a scaled image information gain metric and preserves data diversity with a mixed-ranking strategy. It then constructs evidence-ablated images by masking the most discriminative image patches, eliciting plausible but visually ungrounded responses from the current policy. During policy optimization, these negative responses are incorporated as bounded auxiliary signals under the clean image, penalizing shortcut learning while maintaining RL stability. Experiments across challenging multimodal reasoning benchmarks demonstrate that EVAN mitigates modality imbalance and improves over strong RLVR baselines. Our code is available at https://github.com/DawnLIU35/EVAN.git.
BibTeX
@misc{liu2026grounding,
  author = {Chenxi Liu and Tianyi Xiong and Yanshuo Chen and Heng Huang},
  title = {{Grounding Multimodal Reasoning with Evidence-Ablated Negatives}},
  year = {2026},
  eprint = {2610.39237},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39237},
}
Figure