Knowledge Enhanced Model for Live Video Comment Generation

Jieting Chen, Junkai Ding, Wenping Chen, Qin Jin

TL;DR

This paper introduces a novel "Knowledge Enhanced Model for Live Video Comment Generation" (KLVCG) that leverages external knowledge to generate more informative and diverse comments for long-form videos like movies. They also introduce the "MovieLC" dataset, a new benchmark for this task, and demonstrate that their model significantly outperforms existing methods in both objective metrics and human evaluations.

What Problem Does This Paper Address?

Live video commenting is a popular feature that enhances user experience by creating a shared atmosphere and providing supplementary information. Automatically generating these comments can improve user engagement and enable more human-like bot interactions. While existing work has focused on short, user-generated videos, this paper highlights the gap in research for long-form videos like movies. These longer videos present unique challenges due to their extended duration and the potential for richer, more diverse comments that often incorporate external knowledge beyond the immediate visual content. The authors argue that current models, often adapted from video captioning, fail to capture this crucial aspect of live video comments.

Methods

The core of the paper is the Knowledge Enhanced Model for Live Video Comment Generation (KLVCG), built upon an encoder-decoder framework. Key components include:

  • Modality-Specific Encoders: Separate encoders for visual context (using ResNet101 features), comment context (using a transformer to capture temporal comment sequences), and external knowledge.
  • Knowledge Encoder: This novel component is designed to integrate external knowledge. It first uses self-attention on the recalled knowledge (from knowledge graphs or other video comments) and then employs cross-attention with the comment context to refine the knowledge representation.
  • Cross Encoder: This module fuses the outputs from the visual, comment, and knowledge encoders to capture inter-modal interactions.
  • Decoder: A standard transformer decoder generates the comment sequence autoregressively.

A crucial aspect of KLVCG is its pre-training strategy. The model is pre-trained using a masked language prediction task, where external knowledge is incorporated by masking words in the target comment and predicting them based on the context. This allows the model to learn how to effectively utilize the recalled knowledge.

Datasets

  • MovieLC Dataset: The authors introduce this new dataset, containing 1,406,219 live comments from 85 movies, totaling 175 hours. It's designed to address the lack of benchmarks for long-form video comment generation and includes movie scripts and metadata for future exploration.
  • Existing Datasets: The model is also evaluated on two existing datasets: Livebot and VideoIC, which primarily consist of short, user-generated videos.

Key Insights and Ideas

  • Divergent Nature of Live Comments: Live comments are not just objective descriptions but often include subjective opinions, external references, and personal experiences, making them distinct from video captions.
  • Importance of External Knowledge: For long-form videos like movies, comments frequently draw upon external knowledge (e.g., actor's other works, awards, historical context) to enrich the discussion. KLVCG's success hinges on effectively integrating this knowledge.
  • Temporal Dynamics in Comments: The paper notes that comment patterns can vary based on the viewing period (beginning, middle, end of a video), suggesting the utility of period-specific embeddings.
  • Parameter Efficiency: KLVCG achieves strong performance with fewer parameters than its competitors, indicating that its architectural design and knowledge integration are key drivers of improvement.

Key Results

  • State-of-the-Art Performance: KLVCG significantly outperforms baseline models (Unified Transformer and MML-CG) on all three datasets across various metrics like Recall@k (R@k), Mean Rank (MR), and Mean Reciprocal Rank (MRR). For instance, on the MovieLC dataset, KLVCG achieved an R@1 of 7.36, a notable improvement.
  • Robustness: The model demonstrates strong robustness in sparse context scenarios, maintaining performance even when comment density is reduced.
  • Human Evaluation: In a human evaluation, KLVCG-generated comments were chosen as the best 44% of the time, significantly outperforming MML-CG (15%) and Unified Transformer (11%).
  • Coverage: KLVCG can generate non-null comments for 97% of test samples, compared to 32% for MML-CG and 62% for UT on the MovieLC dataset, highlighting its ability to produce meaningful content more consistently.

Limitations and Future Work

  • Limitations: The paper acknowledges that performance on the MovieLC dataset is lower than on shorter video datasets, indicating the inherent challenges of long-form video understanding and comment generation.
  • Future Work: The authors plan to further explore long video understanding and leverage richer external knowledge sources. They also intend to release the MovieLC dataset and their code to facilitate future research.

Summary updated

Abstract

Live video commenting is popular on video media platforms, as it can create a chatting atmosphere and provide supplementary information for users while watching videos. Automatically generating live video comments can improve user experience and enable human-like generation for bot chatting. Existing works mostly focus on short video datasets while ignoring other important video types such as long videos like movies. In this work, we collect a new Movie Live Comments (MovieLC) dataset to support research on live video comment generation for long videos. We also propose a knowledge enhanced generation model inspired by the divergent and informative nature of live video comments. Our model adopts a pre-training encoder-decoder framework and incorporates external knowledge. Extensive experiments show that both objective metrics and human evaluation demonstrate the effectiveness of our proposed model. The MovieLC dataset and our code will be released.
BibTeX
@inproceedings{chen2023knowledge,
  author = {Jieting Chen and Junkai Ding and Wenping Chen and Qin Jin},
  title = {{Knowledge Enhanced Model for Live Video Comment Generation}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00387},
}
Figure