VidHal: Benchmarking Hallucinations in Vision LLMs
Lists
Only you see this
Abstract
Vision Large Language Models (VLLMs) are widely acknowledged to be prone to hallucinations. Existing research addressing this problem has primarily been confined to image inputs, with sparse exploration of their video-based counterparts. Furthermore, current evaluation methods fail to capture nuanced errors in generated responses, which are often exacerbated by the rich spatiotemporal dynamics of videos. To address these two limitations, we introduce \textsc{VidHal}, a benchmark specially designed to evaluate video-based hallucinations in VLLMs. \textsc{VidHal} is constructed by bootstrapping video instances across a wide range of common temporal aspects. A defining feature of our benchmark lies in the careful creation of captions which represent varying levels of hallucination associated with each video. To enable fine-grained evaluation, we propose a novel caption ordering task requiring VLLMs to rank captions by hallucinatory extent. We conduct extensive experiments on \textsc{VidHal} and comprehensively evaluated a broad selection of models, including both open-source and proprietary ones such as GPT-4o. Our results uncover significant limitations in existing VLLMs with respect to video-based hallucination generation. Through our benchmark, we aim to inspire further research on i) holistic understanding of VLLM capabilities, particularly regarding hallucination, and ii) advancing VLLMs to alleviate this problem.
BibTeX
@misc{choong2026vidhal,
author = {Wey Yeh Choong and Yangyang Guo and Mohan Kankanhalli},
title = {{VidHal: Benchmarking Hallucinations in Vision LLMs}},
year = {2026},
url = {https://openreview.net/forum?id=OALhVHiRba},
howpublished = {OpenReview},
note = {Submitted to ICLR 2026},
}Loading related content...