When we ask a Large Vision-Language Model (LVLM) to describe an image, we want two things: we want it to be detailed (adequacy) and we want it to be correct (fidelity). However, current research reveals a persistent tension between these two goals. As models become more descriptive, they tend to hallucinate more; as they become more factually grounded, they often become vague or omit key details.
This fundamental conflict, often called the fidelity-adequacy trade-off, is the central challenge in moving from simple image tagging to sophisticated, human-like visual reasoning.
The Pareto Frontier of Detail and Truth
In the context of image captioning, "adequacy" refers to the coverage of visual information—how many objects, attributes, and relationships the model describes. "Fidelity" refers to how much of that description is actually supported by the pixels.
Assaf Ben-Kish and colleagues in Mitigating Open-Vocabulary Caption Hallucinations describe this as a Pareto frontier. If you optimize a model solely for truthfulness, it learns a "safe" strategy: it produces generic, low-detail outputs like "a room with furniture" to avoid the risk of being wrong. Conversely, if you push a model to be highly descriptive, it often falls into the "Likelihood Trap" described by Qingmei Tang, where it generates plausible-sounding but fabricated details to satisfy the demand for verbosity.
This trade-off is measurable. Haoyi Qiu introduced VALOR-EVAL to benchmark this specifically, finding that even top-tier models like GPT-4V struggle to maintain high faithfulness as their descriptive coverage increases.
Why Models Lie: Linguistic Priors vs. Visual Evidence
To understand why this trade-off exists, we must look at the mechanics of how these models "see." Hallucinations typically stem from two distinct failures, as categorized by Guangzong Si in Two Causes, Not One:
- Fabrication: The model "sees" something that isn't there because its internal language model expects it. For example, if a model sees a kitchen, its linguistic training might suggest there is a "toaster" on the counter, even if the pixels don't show one. Zhe Cheng notes that models often rely on these linguistic priors (e.g., "bananas are usually yellow") rather than the actual visual input.
- Omission: The model fails to mention a real object because it is being too conservative. This is often a byproduct of "hallucination mitigation" techniques. When we train models to be more precise, they often become "cautious," leading to what Shujun Liu calls "implicit hallucination"—avoiding errors by simply saying less.
The mechanical root of this is often found in the attention mechanism. Liu Yu 0001 argues in Causally-Grounded Dual-Path Attention Intervention that hallucinations occur when the ratio of visual-to-textual attention is imbalanced. If the model pays too much attention to its own generated text history, it loses its "grip" on the image.
Navigating the Trade-off
Researchers are developing several strategies to push the Pareto frontier outward, allowing for more detail without sacrificing truth.
The Hallucination Knob Chenfeng Xu introduced HallE-Switch, which provides a "hallucination knob." This allows users to explicitly dial between strict visual grounding (for high-stakes tasks) and creative inference (for storytelling). This acknowledges that the "correct" balance depends on the application.
Attention Recalibration To prevent the model from "forgetting" the image during long, detailed captions, Mingi Jung proposed SPARC. This method selectively amplifies visual tokens that the model is starting to ignore, ensuring that the descriptive richness remains grounded in the actual image throughout the entire generation process.
Dual-Reward Alignment Since optimizing for a single metric (like accuracy) often leads to vague outputs, Qinghao Ye suggests in Painting with Words that models should be aligned using a dual-reward system. By simultaneously rewarding information density and factual precision, models can be trained to provide "helpful and honest" outputs rather than choosing between the two.
The Path Forward
The goal of this research area is to move beyond the binary of "hallucinating" vs. "not hallucinating." As Marie-Francine Moens and others have shown, even structured data like scene graphs aren't a silver bullet if the underlying perception is noisy. The future of vision-language modeling lies in sophisticated alignment techniques—like the vision-guided preference optimization (V-DPO) proposed by Yuxi Xie—that force models to justify every descriptive detail with specific visual evidence.