Event-Content-Oriented Dialogue Generation in Short Video

Fenghua Cheng, Xue Li, Zi Huang, Jinxiang Wang, Sen Wang

Introduction

Traditional video-grounded dialogue systems often focus on simple object movements or fact-based question answering. However, real-world social media interactions are driven by complex events and subjective opinions. To address this, researchers have introduced SportsVD, a novel dataset, and VCD (Video Commentary Dialogue), a framework designed to generate human-like, opinionated responses by integrating visual features with external knowledge.

The SportsVD Dataset

The Sports domain Video-dialogue Dataset (SportsVD) is the first of its kind to prioritize "Event Contents"—complicated activities involving multiple entities—over simple "Movement Contents." Collected from YouTube, it consists of 5,114 sports highlight videos (basketball and football) and 39,097 multi-turn dialogues derived from actual user comments.

Example of a multi-turn dialogue from the SportsVD dataset, showcasing the shift from factual QA to opinion-based commentary.

Unlike previous datasets like AVSD, which focus on descriptive facts, SportsVD captures the emotional and contextual nuances of how fans discuss sports. This dataset allows models to learn how to engage in commentary that reflects team loyalties and historical context.

The VCD Framework

The proposed Video Commentary Dialogue (VCD) model bridges the "heterogeneity gap" between visual and textual data through a unified Transformer architecture. Key components include:

  • Advanced Video Encoding: Utilizing InternVideo to capture high-level temporal features of complex events rather than frame-by-frame analysis.
  • Knowledge Integration: VCD pulls commonsense knowledge (via COMET-ATOMIC) to predict emotional reactions and domain-specific knowledge (via Google Knowledge Graph) to identify specific athletes and teams.
  • Multi-Task Training: The model is trained on Masked Caption, Knowledge, and Response Prediction tasks, ensuring a deep understanding of the interplay between video, facts, and conversation.
A basketball game frame from the SportsVD dataset, illustrating the complex multi-entity events the model must process.

Key Results and Insights

The integration of domain knowledge (VCD+Domain) significantly boosts performance. In evaluations, it outperformed state-of-the-art baselines like Blip-2 and DialoGPT, achieving a BLEU_4 of 0.20 and a CIDEr score of 2.80. Human evaluators also rated it highest for overall quality and event relevance.

A critical finding of the research is the "ChatGPT blind spot": while ChatGPT is linguistically proficient, it struggles with video-grounded dialogue because it cannot directly perceive the visual evidence, often leading to generic responses that merely echo the user's input.

Conclusion and Future Directions

While VCD excels at relevance, it occasionally faces challenges with linguistic diversity due to the constraints of external knowledge. Future work aims to expand this event-oriented approach to other complex domains such as news, cooking, and DIY tutorials, further refining the ability of AI to engage in meaningful, context-aware conversations about the world around us.

Summary updated

Abstract

Understanding complex events from different modalities, associating to external knowledge and generating response in a clear point of view are still unexplored in today’s multi-modal dialogue research. The great challenges include 1) lack of event-based multi-modal dialogue dataset; 2) understanding of complex events and 3) heterogeneity gap between different modalities. To overcome these challenges, we firstly introduce a novel event-oriented video-dialogue dataset called SportsVD (Sports-domain Video-dialogue Dataset). To our best knowledge, SportsVD is the first dataset that consists of complex events videos and opinion-based conversations with regards to contents in these events. Meanwhile, we present multi-modal dialogue generation method VCD (Video Commentary Dialogue) to generate human-like response according to event contents in the video and related external knowledge. In contrast to previous video-based dialogue generation, we focus on opinion-based response and the understanding of longer and more complex event contents. We evaluate VCD’s performance on SportsVD and other baselines under several automatic metrics. Experiments demonstrate VCD can outperform among other state-of-the-art baselines. Our work is available at https://github.com/Cheng-Fenghua/SportsVD.
BibTeX
@inproceedings{cheng2024event,
  author = {Fenghua Cheng and Xue Li and Zi Huang and Jinxiang Wang and Sen Wang},
  title = {{Event-Content-Oriented Dialogue Generation in Short Video}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.229},
}
Figure