MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs

Haonan Ge, Yiwei Wang, Ming-Hsuan Yang, Yujun CaiarXiv 2025 · OpenReview

Abstract

Large Vision-Language Models (LVLMs) have shown strong performance across multimodal tasks. However, they often produce hallucinations—text that is inconsistent with visual input—due to the limited ability to verify information in different regions of the image. To address this, we propose Multi-Region Fusion Decoding (MRFD), a training-free decoding method that improves factual grounding by modeling inter-region consistency. MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence (JSD) among the responses. These weights guide a consistency-aware fusion of per-region predictions, using region-aware prompts inspired by Chain-of-Thought reasoning. Experiments across multiple LVLMs and benchmarks show that MRFD significantly reduces hallucinations and improves response factuality without requiring model updates.

Introduction

Large Vision-Language Models (LVLMs) like LLaVA or InstructBLIP are impressive, but they suffer from a persistent issue known as hallucination. They often describe objects that are absent or misattribute attributes like color and position. This typically occurs because these models process images holistically. When a model views a complex scene all at once, its attention can become "diffused," leading it to rely on linguistic priors—guessing based on what usually appears in a scene—rather than actual visual evidence. MRFD addresses this by forcing the model to "double-check" its work across different parts of an image, ensuring that local evidence supports the final output.

The MRFD Framework

Multi-Region Fusion Decoding (MRFD) is a "plug-and-play" decoding strategy that operates without the need for expensive fine-tuning or model updates. The process follows three main steps:

  1. Attention-Guided Region Selection: The model utilizes its internal cross-attention maps to identify "salient" regions—the parts of the image most relevant to the user's question.
  2. Multi-Region Analysis & JSD Weighting: The model generates a preliminary "mini-caption" for each selected region. It then calculates the Jensen-Shannon Divergence (JSD) between these regional responses. If a region's prediction aligns with the consensus of other regions, it is considered reliable; outliers are weighted less.
  3. Consistency-Aware Fusion: Finally, the model performs a final decoding pass using "region-aware prompts" (similar to Chain-of-Thought) to fuse the predictions from all regions, weighted by their reliability scores, to produce a single, factually grounded response.
The MRFD pipeline: identifying salient regions, weighting them by consistency (JSD), and fusing them into a factually grounded response.

Experimental Evaluation

The authors evaluated MRFD using LLaVA-1.5 and InstructBLIP across several rigorous benchmarks, including POPE (object existence), CHAIR (caption relevance), and MME-Hallucination.

On the CHAIR benchmark, MRFD reduced the hallucination score on LLaVA-1.5 from 26.2 to 14.1, a relative reduction of over 46% compared to the baseline. It also achieved the highest F1 scores across all POPE settings, showing particular strength in adversarial scenarios where models are intentionally tricked into hallucinating.

Comparison between global image processing (leading to hallucination) and local region-aware decoding (correctly identifying the laptop).

A key insight from these experiments is that local evidence often trumps global guessing. By cropping the image into salient regions, the model is forced to look at high-resolution details, preventing its attention from being diffused by unrelated background clutter.

Discussion and Conclusion

MRFD demonstrates that inter-region consistency is a strong proxy for factual truth. While the method involves more computation than simple greedy decoding (roughly 2.8x to 3.3x latency increase), it remains significantly faster and more memory-efficient than feedback-heavy methods that require multiple full passes or auxiliary models.

Future work may focus on reducing dependency on the base model's initial attention maps and extending this multi-perspective consistency check to video data to ensure temporal accuracy. Overall, MRFD provides a robust, training-free path toward more reliable and factually grounded multimodal AI.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) have shown strong performance across multimodal tasks. However, they often produce hallucinations -- text that is inconsistent with visual input, due to the limited ability to verify information in different regions of the image. To address this, we propose Multi-Region Fusion Decoding (MRFD), a training-free decoding method that improves factual grounding by modeling inter-region consistency. MRFD identifies salient regions using cross-attention, generates initial responses for each, and computes reliability weights based on Jensen-Shannon Divergence (JSD) among the responses. These weights guide a consistency-aware fusion of per-region predictions, using region-aware prompts inspired by Chain-of-Thought reasoning. Experiments across multiple LVLMs and benchmarks show that MRFD significantly reduces hallucinations and improves response factuality without requiring model updates.
BibTeX
@misc{ge2025mrfd,
  author = {Haonan Ge and Yiwei Wang and Ming-Hsuan Yang and Yujun Cai},
  title = {{MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMs}},
  year = {2025},
  eprint = {2508.10264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10264},
}
Figure