DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Hung Le, Chinnadhurai Sankar, Seungwhan Moon, Ahmad Beirami, Alborz Geramifard, Satwik Kottur

TL;DR

This paper introduces DVD, a novel diagnostic dataset for video-grounded dialogue systems. DVD aims to address the limitations of existing benchmarks by providing detailed annotations for multi-step reasoning over spatio-temporal video content, with minimal biases, to better understand model capabilities and limitations.

What Problem Does This Paper Address?

Building intelligent systems that can understand and reason about both spoken language and dynamic visual information (like videos) is a significant challenge. Video-grounded dialogue systems, which aim to answer questions about a video through a conversational interface, are particularly complex. Existing datasets often lack the detailed annotations needed to pinpoint why these systems succeed or fail, especially concerning their reasoning abilities. Furthermore, many current benchmarks can be "gamed" by models that exploit biases rather than performing genuine reasoning. This makes it difficult to assess true progress and identify areas for improvement.

Methods

The authors introduce DVD, a diagnostic dataset built upon the CATER synthetic video dataset. They generate dialogues by simulating questions that require various reasoning types, including action recognition, temporal reasoning, spatial reasoning, and object tracking across dialogue turns. A key innovation is the use of 17 novel functional modules, extending existing ones to handle video and dialogue context. These modules are used to generate questions that incorporate three types of cross-turn semantic relationships:

  • Temporal Relation (TR): Understanding relationships between different time segments in the video.
  • Object Reference (OR): Resolving pronouns or descriptive phrases that refer to objects mentioned in previous turns.
  • Topic Transfer (TT): Maintaining context and adapting to shifts in focus across dialogue turns.

The dataset is structured to minimize biases and includes detailed annotations for these reasoning types.

Datasets

The primary dataset introduced in this paper is DVD (Diagnostic Dataset for Video-grounded Dialogues). It is built using 11k CATER synthetic videos, resulting in over 100,000 dialogues and 1 million question-answer pairs. The authors also compare their work to existing benchmarks such as VQA, Visual7W, TGIF-QA, TV-QA, VisDial, Guess What, AVSD, CVDN, SHAPE, CLEVR, SVQA, CLEVRER, bAbI, MNIST Dialog, and CLEVR-Dialog.

Key Insights and Ideas

  • The Need for Diagnostic Datasets: Existing benchmarks are insufficient for truly understanding the reasoning capabilities of video-grounded dialogue systems. DVD's detailed annotations are crucial for this diagnostic purpose.
  • Multi-faceted Reasoning is Key: Success in video-grounded dialogue requires a combination of understanding temporal dynamics, spatial relationships, and maintaining coherent dialogue context (object and topic tracking).
  • Current Models Struggle with Long-Term Dependencies: Models often fail to effectively track objects or relevant video segments over extended dialogue turns, indicating a weakness in long-term memory and reasoning.
  • Bias Mitigation is Crucial: Designing datasets that minimize exploitable biases is essential for ensuring models learn genuine reasoning skills rather than superficial patterns.

Key Results

The paper evaluates several baseline models on DVD. Key findings include:

  • "Blind" systems (relying only on answers or questions) perform poorly (up to 39% accuracy).
  • Systems incorporating dialogue history show significant improvement (up to 45% accuracy), highlighting the importance of context.
  • Even with visual information, the best models achieve around 51% accuracy, still far below human performance (89.3% for human evaluation).
  • Models struggle with questions requiring spatial localization and long-term reasoning, particularly when dealing with object containment.
  • Performance degrades as video interval lengths increase, indicating challenges with understanding extended temporal segments.
  • Models show instability in resolving object references, especially with longer turn distances.

Limitations and Future Work

The authors acknowledge that DVD is a synthetic dataset and suggest it should supplement, not replace, real-world dialogue data. They propose two key research directions:

  • Dialogue Object Tracking (DOT): Developing metrics and tasks to specifically evaluate a system's ability to track objects and their attributes throughout a dialogue.
  • Video Interval Tracking (VIT): Creating learning tasks to assess a system's capability to localize relevant video segments across multiple dialogue turns, similar to text-to-clip tasks.

Summary updated

BibTeX
@inproceedings{le2021dvd,
  author = {Hung Le and Chinnadhurai Sankar and Seungwhan Moon and Ahmad Beirami and Alborz Geramifard and Satwik Kottur},
  title = {{DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=M648mFDoV1l},
}

Loading related content...

Figure