Cold Start Problem For Automated Live Video Comments

Hao Wu, François Pitie, Gareth JonesNAACL 2021

TL;DR

This paper tackles the "cold start" problem for automated live video comments (danmu), aiming to generate engaging comments for videos with few or no existing comments. They propose a multi-modal Transformer network that leverages video, audio, and subtitle data, trained across various comment densities to improve performance on new or less popular content.

What Problem Does This Paper Address?

Live video comments, or "danmu," are a popular feature on Asian video platforms, enriching user experience and influencing a video's popularity. However, a significant challenge, analogous to the "cold start problem" in recommender systems, is that videos with few or no danmu comments struggle to gain initial attention. This paper addresses the problem of automatically generating danmu comments for these "cold start" videos, where existing methods often rely on a wealth of existing comments, making them ineffective for new content.

Methods

The core of the proposed solution is a Multi-density Cold Video Transformer (MCVT). This model takes a multi-modal approach, integrating:

  • Video Frames: Processed using a pre-trained ResNet encoder.
  • Audio Signals: Encoded using MFCCs and a GRU.
  • Subtitles/Text: Extracted via OCR from video subtitles and combined with existing comments.

These modalities are fused within a Transformer architecture. A key innovation is the multi-density learning strategy. Instead of training solely on densely commented videos, the model is trained across various "comment densities" (represented by p, the percentage of earliest comments used as context). This allows the model to learn how to generate comments even when very few or no prior comments are available (p=0%). The training regime randomly samples these p values for each mini-batch, effectively training for both cold-start and more established scenarios.

Datasets

The authors constructed a large-scale dataset by combining:

  • A portion of the publicly available Response to Livebot dataset (2,322 videos, 857,993 comments).
  • An additional 2,350 videos scraped from the same danmu video website (bilibili.com), specifically focusing on "Daily Life" category videos to maintain thematic consistency.

This resulted in a dataset with 4,672 videos and 2,789,360 danmu comments. The dataset was split into training, development, and test sets with a larger scale than previous work (4272/200/200 videos). A crucial aspect of their dataset construction is the use of comment publication timestamps to reconstruct different phases of a video's commenting lifetime.

Key Insights and Ideas

  • Timestamp Matters: Exploiting the publication timestamps of danmu comments is crucial for simulating and training for different comment densities, a factor overlooked in prior work.
  • Multi-modal Fusion is Key: Combining visual, audio, and textual (subtitle/comment) information significantly improves comment generation quality, especially in challenging cold-start scenarios.
  • Multi-density Training Bridges the Gap: Training a single model across various comment densities (from 0% to 100%) is more effective for the cold-start problem than training models exclusively on densely commented data.
  • OCR for Subtitles is Viable: Using OCR for subtitle extraction proved to be a robust method for capturing textual context, even outperforming speech recognition in some cases for this specific platform.

Key Results

  • Improved Performance: The proposed MCVT model significantly outperforms the baseline Livebot model across various metrics (R@k, P@k, MRR) in retrieval-based evaluation, particularly in cold-start scenarios (lower p values).
  • Human Evaluation: Human evaluators found the generated comments to be almost indistinguishable from real danmu comments, with relevancy and engagement scores even higher than human comments for p > 50%.
  • Cold Start Resilience: The MCVT model shows strong performance even in the complete cold-start scenario (p=0%), demonstrating its effectiveness for new videos.
  • Ablation Study: Training Livebot with the multi-density strategy (Livebot-t) already shows improvement over the original Livebot, highlighting the benefit of the training regime. The full MCVT model, incorporating audio and subtitles, further enhances performance.

Limitations and Future Work

  • Limitations: The paper acknowledges that while performance degrades slightly in the complete cold-start scenario (p=0%), it remains competitive. The authors also note that for p=100%, the original Livebot model trained solely on dense data slightly outperforms Livebot-t, suggesting that multi-density training might introduce some noise when the goal is solely to generate comments for popular videos.
  • Future Work: The authors suggest leveraging highlight detection methods to identify likely user interest points in a video timeline. This could further refine the automated creation of danmu comments by focusing on more relevant moments.

Summary updated

BibTeX
@inproceedings{wu2021cold,
  author = {Hao Wu and François Pitie and Gareth Jones},
  title = {{Cold Start Problem For Automated Live Video Comments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure