TL;DR
VidHal is a new benchmark designed to expose the "temporal hallucinations" of Vision Large Language Models (VLLMs)—errors where models misinterpret the dynamic flow of video. By introducing a novel caption ordering task, the researchers demonstrate that even state-of-the-art models like GPT-4o and Gemini 2.5 struggle to distinguish between subtle and severe temporal errors, often relying on static image shortcuts rather than true video understanding.
What Problem Does This Paper Address?
While we know Vision LLMs are prone to "hallucinations" (generating plausible-sounding but factually incorrect descriptions), most research has focused on static images. Videos introduce a much harder challenge: spatiotemporal dynamics. A model might correctly identify a "ball" and a "pool" (static objects) but hallucinate that the ball is moving out of the pool instead of falling into it.
Existing benchmarks often rely on simple binary (Yes/No) questions, which are too coarse to capture nuanced errors. There is a significant gap in our ability to measure how well these models actually "see" time, sequence, and direction.
Methods
The authors developed a pipeline to create VidHal, a benchmark of 1,000 video instances. Their approach involves:
- Temporal Aspect Selection: Focusing on five key areas: Action, Attribute, Direction, Object (interactions), and Event Order.
- Automated Caption Generation: Using GPT-4o to generate an "Anchor" (accurate) caption based on ground-truth metadata, followed by two levels of hallucinated captions: Moderate (plausible but wrong) and High (clearly incorrect).
- The Caption Ordering Task: Instead of just asking "is this caption right?", they require models to rank a set of captions from most accurate to most hallucinatory. This tests fine-grained discrimination.
- Evaluation Metrics: They use standard Multiple-Choice Question Answering (MCQA) accuracy and a ranking-based metric called NDCG (Normalized Discounted Cumulative Gain) to reward models for getting the relative order of hallucinations correct.
Datasets
The researchers didn't record new videos; instead, they "bootstrapped" from four high-quality existing video understanding datasets:
- TempCompass
- Perception Test
- MVBench
- AutoEval-Video
The resulting VidHal benchmark is unique because it provides a "hallucination spectrum" for every video, allowing for much more granular testing than previous datasets.
Key Insights and Ideas
- The "Image Prior" Trap: One of the most striking findings is that VLLMs heavily rely on "image priors." When the researchers tested models using only a single salient frame instead of the full video, the models often gave the same (sometimes correct, often incorrect) answers. This suggests they aren't actually "watching" the video but are making educated guesses based on a snapshot.
- Temporal Blindness: Models are relatively good at identifying objects and general actions (which can be inferred from a few frames) but fail miserably at Direction and Event Order. These are the "pure" video tasks that require understanding the arrow of time.
- Scale Doesn't Guarantee Sanity: Interestingly, increasing the number of parameters doesn't always help. In some model families (like InternVL2.5), the 8B version outperformed the 26B version on hallucination tasks, suggesting that architectural design and specific temporal tuning are more important than raw size.
- Hallucination is a Gradient: By forcing models to rank captions, the authors showed that VLLMs struggle to distinguish between "Moderate" and "High" hallucinations. They can often tell what is perfectly right, but they get confused when choosing between two different flavors of "wrong."
Key Results
The authors benchmarked 23 different models, including proprietary giants and open-source contenders:
- Top Performers: Proprietary models like GPT-4.1 and Gemini 2.5 Pro generally lead the pack. Among open-source models, LongVU and Qwen2.5-VL showed impressive competitive performance.
- The Performance Gap: On the "Relative Caption Ordering" task, most models saw a significant drop in performance when dealing with Direction and Order compared to Object recognition.
- Human Validation: To ensure the benchmark was fair, the authors ran human trials. Humans agreed with the benchmark's hallucination rankings 87% of the time, confirming that the "Moderate" vs. "High" distinction is meaningful to people, even if it confuses the AI.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Synthetic Bias: The benchmark relies on GPT-4o to generate the initial captions. While they used human verification and other LLMs (like LLaMA3 and Gemini) to filter these, there is always a risk of "LLM-induced bias" where models are better at ranking text written by other models.
- Static Shortcuts: The study confirms that current models are "lazy" and rely on single-frame information.
Future Work should focus on "temporal-centric" pretraining—teaching models to prioritize motion and sequence over static object recognition—and developing architectures that explicitly mitigate the reliance on single-frame image priors.
Summary updated