Distorted or Fabricated? A Survey on Hallucination in Video LLMs

Yiyang Huang, Yitian Zhang, Yizhou Wang, Xiaojuan Liu, Liang Shi, Huimin Zeng, Yun Fu2026 · DOI

TL;DR

This paper surveys the growing problem of "hallucinations" in Video Large Language Models (Vid-LLMs), where models generate plausible but incorrect descriptions of video content. The authors propose a novel taxonomy to categorize these hallucinations into "dynamic distortion" and "content fabrication," providing a structured framework for understanding and addressing this critical challenge in building reliable video-AI systems.

What Problem Does This Paper Address?

As Video Large Language Models (Vid-LLMs) become more sophisticated, they are increasingly capable of understanding and generating text based on video input. However, a significant hurdle remains: hallucinations. These are outputs that sound perfectly reasonable and coherent but fundamentally misrepresent or invent information not present in the video. This is particularly problematic in safety-critical applications like autonomous driving or embodied AI, where factual accuracy is paramount. While hallucinations in image-based models have been studied, the temporal, motion-rich, and multimodal nature of video introduces unique challenges that require a dedicated approach. This paper aims to provide a comprehensive overview and structured understanding of these video-specific hallucinations.

Methods

The core contribution of this paper is a novel, mechanism-driven taxonomy for video hallucinations. They categorize them into two primary types:

  1. Dynamic Distortion: This occurs when the model sees entities but misinterprets their temporal evolution or identity. It's further broken down into:
  2. Spatiotemporal Dynamics: Errors in event ordering, duration, or frequency.
  3. Referential Inconsistency: Confusing characters or scenes across different parts of the video.
  4. Content Fabrication: This happens when outputs lack clear visual grounding and are instead driven by learned priors or other modalities. It includes:
  5. Context-Driven Fabrication: Relying on statistical associations (e.g., "cooking" often happens in a "kitchen") rather than direct visual evidence.
  6. Audio-Visual Conflict: When strong audio cues override or contradict visual information.

The paper then reviews existing benchmarks, metrics, and mitigation strategies through the lens of this taxonomy, analyzing their root causes (e.g., limited temporal representation, poor visual grounding) and suggesting future research directions.

Datasets

The paper references numerous datasets used in existing benchmarks and for training/evaluating mitigation strategies. Some notable examples include:

  • ActivityNet: A large-scale dataset for activity recognition.
  • YouCook2: A dataset for cooking videos.
  • Ego4D: Egocentric video dataset.
  • YouTube: Various datasets derived from YouTube videos.
  • WebVid: A large-scale dataset of short videos.

The paper also implicitly discusses the need for new benchmarks that specifically target the identified hallucination types, especially for longer videos and audio-visual conflict scenarios.

Key Insights and Ideas

  • Video Hallucinations are Fundamentally Different: The temporal and multimodal nature of video introduces unique hallucination types (dynamic distortion, audio-visual conflict) that go beyond static image hallucinations.
  • A Unified Taxonomy is Crucial: The proposed "dynamic distortion" vs. "content fabrication" framework provides a much-needed structured way to analyze and compare different hallucination phenomena and their underlying causes.
  • Root Causes are Key: Hallucinations often stem from fundamental limitations in current Vid-LLMs, such as insufficient temporal reasoning capabilities and weak grounding of predictions in actual visual evidence.
  • Long Videos Exacerbate Problems: As videos get longer, issues like referential inconsistency and temporal errors become significantly more pronounced.
  • Audio is a Powerful, Yet Deceptive, Cue: Audio-visual conflict highlights how dominant audio can lead models astray, even when visual evidence suggests otherwise.

Key Results

The paper doesn't present new experimental results but rather synthesizes findings from existing research. Key takeaways from their analysis include:

  • Benchmark Imbalance: There's a heavy concentration of benchmarks on "Spatiotemporal Dynamics" (8 benchmarks) and "Context-Driven Fabrication," while "Referential Inconsistency" and "Audio-Visual Conflict" are significantly underexplored, with only three benchmarks each.
  • Performance Gaps: While state-of-the-art models achieve high scores on some tasks (e.g., VidHalluc, EventHallusion >80%), they struggle with fine-grained temporal reasoning (e.g., VideoHallucer at 37.8%) and long-context consistency (e.g., ELV-Halluc at 53.1%).
  • Mitigation Trade-offs: Training-based mitigation methods show larger gains (up to 30-36%) but require significant computational resources. Training-free methods are more accessible but offer more limited improvements.

Limitations and Future Work

  • Underexplored Areas: Referential Inconsistency and Audio-Visual Conflict are identified as areas needing more research and dedicated benchmarks, especially for long-form videos.
  • Inference-Time vs. Training-Time: Most current mitigation strategies are applied at inference time or as post-training adjustments, suggesting a need for more scalable, training-time alignment methods.
  • Future Directions: The authors propose focusing on:
  • Video-Native Encoders: Developing architectures that better preserve motion cues and temporal structure.
  • Memory Mechanisms: Incorporating explicit memory to support long-term temporal grounding.
  • Counterfactual Learning: Using techniques to disentangle model reasoning from potentially misleading prior knowledge.
  • Improved Benchmarks: Creating more comprehensive benchmarks for under-addressed areas like referential inconsistency and audio-visual conflict in long videos.

Summary updated

Abstract

Despite significant progress in video-language modeling, hallucinations remain a persistent challenge in Video Large Language Models (Vid-LLMs), referring to outputs that appear plausible yet contradict the content of the input video. This survey presents a comprehensive analysis of hallucinations in Vid-LLMs and introduces a systematic taxonomy that categorizes them into two core types: dynamic distortion and content fabrication, each comprising two subtypes with representative cases. Building on this taxonomy, we review recent advances in the evaluation and mitigation of hallucinations, covering key benchmarks, metrics, and intervention strategies. We further analyze the root causes of dynamic distortion and content fabrication, which often result from limited capacity for temporal representation and insufficient visual grounding. These insights inform several promising directions for future work, including the development of motionaware visual encoders and the integration of counterfactual learning techniques. This survey consolidates scattered progress to foster a systematic understanding of hallucinations in Vid-LLMs, laying the groundwork for building robust and reliable video-language systems. For more information, readers are encouraged to visit the repository at https://github.com/ hukcc/Awesome-Video-Hallucination.
BibTeX
@misc{huang2026distorted,
  author = {Yiyang Huang and Yitian Zhang and Yizhou Wang and Xiaojuan Liu and Liang Shi and Huimin Zeng and Yun Fu},
  title = {{Distorted or Fabricated? A Survey on Hallucination in Video LLMs}},
  year = {2026},
  doi = {10.36227/techrxiv.177138968.82704736/v1},
}
Figure