WildVideo: Benchmarking LMMs for Understanding Video-Language Interaction

Songyuan Yang, Weijiang Yu, Wenjing Yang, Xinwang Liu, Huibin Tan, Long Lan, Nong XiaoIEEE TPAMI 2025 · OpenReview

Abstract

We introduce WildVideo, an open-world benchmark dataset designed to address how to assess hallucination of Large Multi-modal Models (LMMs) for understanding video-language interaction in the wild. Our WildVideo comprehensively tests the perceptual, cognitive, and contextual comprehension hallucination of LMMs through both single-turn and multi-turn open-ended question-answering (QA) tasks on videos captured from two human perspectives (i.e. first-person view and third-person view). We define 9 distinct tasks that challenge LMMs across multi-level perceptual tasks (e.g., static and dynamic perception), multi-aspect cognitive tasks (e.g., commonsense, world knowledge), and multi-faceted contextual comprehension tasks (e.g., contextual ellipsis, cross-turn retrieval). The benchmark consists of 1,318 meticulously curated videos, supplemented with 13,704 single-turn QA pairs and 1,585 multi-turn dialogues (up to 5 turns). We evaluated 14 commonly-used LMMs on WildVideo, revealing significant hallucination issues of current LMMs, highlighting substantial gaps in their current capabilities.
BibTeX
@article{yang2025wildvideo,
  author = {Songyuan Yang and Weijiang Yu and Wenjing Yang and Xinwang Liu and Huibin Tan and Long Lan and Nong Xiao},
  title = {{WildVideo: Benchmarking LMMs for Understanding Video-Language Interaction}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=X0WwHzom9f},
}

Loading related content...

Figure