Introduction
Large Vision-Language Models (LVLMs), such as LLaVA, have demonstrated remarkable capabilities in generating coherent and contextually relevant text based on visual inputs. However, these models are frequently plagued by hallucinations—instances where the model generates descriptions of objects or relationships not present in the image.
Existing solutions typically involve extensive data-centric fine-tuning or complex decoding strategies. HalluRNN proposes a different approach: an architecture-level solution that addresses hallucinations as they emerge during the model's internal reasoning process. The authors identify a phenomenon called representational drift, where visual grounding is gradually lost as information propagates through the Transformer layers, eventually leading to hallucinated outputs.
Methodology: Cross-Layer Recurrent Reasoning
The core innovation of HalluRNN is treating the stack of Transformer layers not as a static feed-forward pipeline, but as a temporal sequence. By integrating a recurrent mechanism across the model's depth, HalluRNN allows the system to "remember" and "refine" its internal state, ensuring that visual evidence remains a primary constraint throughout the reasoning process.
The Dual-Gated Depth Propagation Unit (DG-DPU)
At the heart of HalluRNN is the Dual-Gated Depth Propagation Unit (DG-DPU), a lightweight recurrent module shared across all layers. The DG-DPU utilizes two specialized gates to maintain representational integrity:
- Constraint Gate: Performs a coarse-grained check by comparing the current layer's output with the previous recurrent state to detect potential drift.
- Correction Gate: Executes a fine-grained blending of the current layer's evidence with the stabilized historical context from preceding layers.
This approach is highly parameter-efficient; the massive pre-trained LVLM backbone remains frozen, while only the compact DG-DPU module is fine-tuned.
Experimental Evaluation
HalluRNN was evaluated across a diverse set of benchmarks to test its robustness in mitigating object hallucinations and maintaining general reasoning capabilities. The training utilized a 17K subset of the LLaVA-Instruct-150K dataset.
Key Benchmarks:
- POPE (Poll Object Probing Evaluation): Evaluates object hallucination across MSCOCO, A-OKVQA, and GQA datasets using Random, Popular, and Adversarial settings.
- MME: Assesses perception and cognition capabilities.
- CHAIR: Measures object hallucination rates in image captioning.
- MM-Vet & MMMU: Tests complex reasoning and expert-level task performance.
Key Results and Analysis
HalluRNN consistently outperformed vanilla models and state-of-the-art decoding strategies across all major benchmarks.
As shown in the MME benchmark results, HalluRNN variants significantly exceed the performance of the vanilla baseline in both perception and recognition tasks. Similarly, in the POPE benchmark, HalluRNN achieved superior accuracy and F1 scores, particularly in the challenging "Adversarial" and "Popular" settings.
Beyond raw performance, the researchers highlighted several critical insights:
- Depth as Time: Treating model depth as a temporal flow allows for effective self-correction.
- Information Persistence: Carrying forward visual grounding from early layers prevents the language component from dominating the output with false narratives.
- Plug-and-Play Transferability: A DG-DPU module trained on one model (e.g., LLaVA-1.5) can be transferred to others (e.g., Qwen-VL) to reduce hallucinations without additional training, suggesting the learned drift patterns are universal.
Conclusion and Future Work
HalluRNN offers a robust, efficient, and transferable method for mitigating hallucinations in LVLMs. By addressing representational drift at the architectural level, it provides a more stable alternative to hyperparameter-sensitive decoding tweaks.
Future work will focus on improving robustness against corrupted early-layer signals—such as those caused by visual noise or adversarial attacks—and scaling the architecture to even larger models (70B+ parameters) to further explore the benefits of cross-layer recurrent reasoning.
Summary updated