Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang, Zhicai Wang, Beier Zhu, Hanwang Zhang

TL;DR

The paper introduces AIR (Adaptive vIsual Reinforcement), a training-free, inference-time framework designed to mitigate hallucinations in Multimodal Large Language Models (MLLMs). By combining prototype-based token reduction with Optimal Transport (OT)-guided patch reinforcement, AIR selectively amplifies salient visual cues while filtering out distracting background noise. This targeted approach significantly reduces object hallucinations across multiple benchmarks with minimal computational overhead and no degradation in general model capabilities.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at vision-language reasoning, they are notoriously prone to hallucinations—generating text that describes non-existent objects or contradicts the visual input.

Existing fixes generally fall into two camps: training-time fine-tuning (which is annotation-heavy and computationally expensive) or inference-time interventions like contrastive decoding (which add significant latency). Recently, some training-free methods have tried re-injecting visual tokens directly into the model's decoder layers during generation. However, these methods typically re-inject all visual tokens indiscriminately. The authors observe that this introduces substantial background noise and redundant information, which actually dilutes the model's attention and causes it to lose track of critical visual cues.

The core challenge addressed here is: How can we dynamically reinforce only the most relevant visual information during decoding without needing extra training or causing massive inference delays?

Figure 1: Analysis of existing hallucination mitigation strategies in multimodal large language mod- els (MLLMs). (a) existing strategies hallucinate non-existent objects, while AIR produces faithful, image-grounded answers. (b) Similarity across decoder layers between hidden states and differ- ent visual tokens, showing that salient patches consistently achieve higher alignment than irrelevant ones. (c) Attention heatmaps comparing existing re-injection with AIR: prior methods spread at- tention to irrelevant regions, whereas AIR focuses on semantically critical areas. (d) AIR reduces hallucination across benchmarks with minimal impact on general multimodal performance.

Methods

The authors propose the Adaptive vIsual Reinforcement (AIR) framework, which operates directly within the feed-forward networks (FFNs) of the transformer decoder layers. It consists of two primary stages:

  1. Prototype-Based Token Reduction: To eliminate global redundancy, AIR first calculates a "global prototype" ($\mathbf{h}_p$) by averaging the hidden states of the visual tokens. It then computes the Euclidean distance of each token from this average. Tokens that are furthest from the prototype represent the most distinctive, non-redundant visual features. AIR retains only the top $Q$ distinctive tokens ($\mathbf{H}'$), effectively filtering out repetitive background signals.

  2. OT-Guided Patch Reinforcement: Even within a condensed token set, different regions vary in importance. AIR crops the input image into multiple local patches and models the original hidden states and patch tokens as discrete probability distributions. It leverages Entropically Regularized Optimal Transport (OT) via the Sinkhorn-Knopp algorithm to measure the global semantic alignment between the hidden states and these patch embeddings. Patches with the lowest transport distance (highest alignment) are selected and fused back into the FFN layers.

Figure 2: Overview of our proposed AIR framework. Given a multimodal input (image and ques- tion), visual features are extracted by the visual encoder and aligned with textual embeddings via the projector. Inside the Transformer layers, visual tokens are first compressed through prototype- based selection to remove redundancy, and then reinforced by patch-level alignment using optimal transport. These selective reinforcement strategies enrich the hidden states with salient visual cues, enabling AIR to produce safer, more faithful, and visually grounded responses.

Datasets

The authors evaluated AIR across an extensive suite of standard vision-language benchmarks:

  • Hallucination Evaluation: CHAIR (evaluated on 500 randomly sampled MSCOCO images) and POPE (evaluated on MSCOCO, A-OKVQA, and GQA under random, popular, and adversarial settings).
  • General Capabilities: MME, MMBench, and LLaVA-Bench (In-the-Wild).
  • Comprehensive Hallucination Suites: HallusionBench, V* Bench, MMHal-Bench, and MM-Vet.

No new datasets were introduced; the focus was entirely on validating robustness across diverse, pre-existing evaluation metrics.

Key Insights and Ideas

  • Less is More in Visual Re-injection: Simply flooding the decoder with all original image tokens introduces background clutter that misleads the model. Selectively reinforcing distinct and well-aligned patches yields much cleaner attention maps.
  • The Power of Optimal Transport over Cosine Distance: Pointwise metrics like cosine similarity average out pairwise costs, diluting discriminative features. Optimal Transport captures the global geometric structure of token distributions. The authors mathematically prove that the OT-based metric provides strictly higher sensitivity in distinguishing safe patches from hallucination-prone ones.
  • Mid-to-Deep Layer Criticality: Ablation studies confirmed that applying visual reinforcement in the mid-to-deep layers (specifically layers 24–32 in a 32-layer model) provides the best trade-off, as these layers are where the model typically shifts from visual processing to text-heavy generation bias.

Key Results

AIR was validated across three prominent MLLM backbones: LLaVA-1.5-7B, Qwen-VL-Chat, and GLM-4V-9B.

  • Significant Hallucination Reduction: On the CHAIR benchmark using LLaVA-1.5-7B, AIR reduced the sentence-level hallucination rate ($\text{CHAIR}_S$) from 22.0% to 18.4% and the object-level rate ($\text{CHAIR}_I$) from 6.7% to 5.7%, outperforming state-of-the-art baselines like VCD and MemVR.
  • Robustness to Adversarial Prompts: On the POPE adversarial setup for the GQA dataset, AIR achieved an accuracy of 80.4%, compared to the vanilla model's 75.1%.
  • Preserved General Capabilities: On MME and MMBench, AIR maintained or slightly improved performance scores, proving that mitigating hallucinations did not come at the cost of general reasoning or language fluency.
  • High Efficiency: Compared to the baseline LLaVA-1.5-7B, AIR only introduced a minor latency overhead (average latency went from 1.68s to 2.07s per generation) and negligible GPU memory increase (13.5G to 13.7G), making it highly practical for deployment.

Limitations and Future Work

While AIR delivers robust performance on standard perception and description tasks, the authors note that its application to complex reasoning-heavy multimodal models and autonomous agents remains unexplored. Future work will focus on extending AIR to these agentic settings, where multi-step reasoning might introduce more complex drift. Additionally, the authors see a strong potential for applying OT-based distribution alignment to broader cross-modal grounding problems beyond inference-time patch selection.

Summary updated

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.
BibTeX
@inproceedings{zhu2026look,
  author = {Xingyu Zhu and Kesen Zhao and Liang Yi and Shuo Wang and Zhicai Wang and Beier Zhu and Hanwang Zhang},
  title = {{Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=KhEjFE1QvQ},
}
Figure