Abstract
A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem, involving various reasoning types on both visual and language inputs. Existing benchmarks do not have enough annotations to thoroughly analyze dialogue systems and understand their capabilities and limitations in isolation. These benchmarks are also not explicitly designed to minimise biases that models can exploit without actual reasoning. To address these limitations, in this paper, we present DVD, a Diagnostic Dataset for Video-grounded Dialogues. The dataset is designed to contain minimal biases and has detailed annotations for the different types of reasoning over the spatio-temporal space of video. Dialogues are synthesized over multiple question turns, each of which is injected with a set of cross-turn semantic relationships. We use DVD to analyze existing approaches, providing interesting insights into their abilities and limitations. In total, DVD is built from 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video, resulting in more than 100k dialogues and 1M question-answer pairs. Our code and dataset are publicly available at https://github.com/facebookresearch/DVDialogues.
Introduction
Building a system capable of conversing about video content—Video-Grounded Dialogue (VGD)—is a formidable challenge. It requires the simultaneous tracking of visual cues, such as object movement and containment, alongside the maintenance of conversational context, such as resolving pronominal references across multiple turns. Current benchmarks often suffer from language biases that allow models to "guess" answers without true visual or linguistic reasoning. Furthermore, they lack the granularity needed to diagnose specific failure modes. To solve this, we introduce DVD (Diagnostic Dataset for Video-grounded Dialogues), a benchmark designed to pinpoint exactly where reasoning breaks down in spatial, temporal, and dialogue-memory tasks.
Methodology: Functional Programs and Semantic Relations
DVD is constructed using the CATER synthetic environment, featuring geometric objects in a 3D space. The core innovation is the use of functional programs to generate questions. By using 17 distinct functional modules (e.g., Filter Action, Relate Spatial), every question follows a verifiable logical path.
To simulate natural conversation, we inject three types of cross-turn semantic dependencies:
- Video Temporal Relation (TR): Questions referencing specific time windows (e.g., "before," "after") relative to previous turns.
- Dialogue Object Reference (OR): The use of pronouns or descriptive phrases to refer back to previously discussed objects.
- Topic Transfer (TT): Shifting the focus of the dialogue while maintaining context (e.g., "how about to the left of it?").
The DVD Dataset
The resulting dataset is massive in scale, comprising 11,000 videos, 100,000 dialogues, and over 1 million question-answer pairs. Each dialogue consists of exactly 10 turns. Unlike standard VQA or VideoQA, DVD necessitates simultaneous "Video Interval Tracking" and "Dialogue Object Tracking," making it a significantly more complex reasoning task than its predecessors.
Key Experimental Insights
Our analysis of state-of-the-art models on DVD revealed several critical findings:
- The Human-Machine Gap: There is a stark performance disparity. Humans achieve 89.3% accuracy, while the best Transformer-based models reach only 51.1%.
- The "Blind" Model Trap: Models provided with only dialogue history (no video) can reach ~45% accuracy by exploiting linguistic patterns, underscoring the need for DVD's diagnostic rigor.
- Reasoning Failures: Models struggle most with Action Count (~38%) and Object Count (~40%), while performing better on simple Object Existence (~69%).
- Object Permanence: Current architectures fail significantly on "object containment" tasks, where one object hides another, suggesting a lack of robust 3D-spatial understanding.
- Temporal Decay: Performance degrades as the "turn distance" of an object reference increases, indicating limitations in long-term dialogue memory.
Conclusion and Future Work
DVD exposes significant limitations in current VGD systems, particularly in spatial localization and temporal tracking. While the synthetic nature of CATER limits real-world visual complexity, the logical depth of DVD provides a necessary testbed for reasoning. Future work should focus on Dialogue Object Tracking (DOT) and Video Interval Tracking (VIT) to bridge the gap toward human-level video understanding.
Summary updated