Enhancing Multimodal Affective Analysis with Learned Live Comment Features

Zhaoyuan Deng, Amith Ananthram, Kathleen McKeownAAAI 2025 · DOI

TL;DR

This paper tackles the challenge of limited live comment data for affective analysis by proposing a novel method to generate synthetic live comment features from video content. By training a multimodal encoder on a new, diverse dataset of live comments and videos, they achieve state-of-the-art results in sentiment analysis, emotion recognition, and sarcasm detection, even for videos without existing live comments.

What Problem Does This Paper Address?

Affective analysis, which aims to understand the emotional content of videos, often benefits from user-generated live comments (like Danmaku) that provide real-time emotional reactions. However, the practical application of these comments is hindered by their limited availability across different video platforms and genres. Many platforms don't support live comments, and existing datasets often focus on specific content types (e.g., gaming, sports) that may not elicit a wide range of emotions. This scarcity makes it difficult to train robust models for affective analysis, especially for videos lacking these valuable annotations.

Methods

The core of this work lies in a two-stage approach:

  1. Contrastive Pre-training (V2LC Encoder): The authors introduce a novel Video-to-Live Comment (V2LC) encoder. This encoder is trained using a contrastive learning framework, similar to CLIP. It learns to project video segments into a representation space that aligns with live comment embeddings. The key innovation here is training the encoder to predict correct pairings between video segments and their corresponding live comments, effectively learning to generate "synthetic" live comment features from video content alone. This allows the model to infer emotional context even when real live comments are absent. They also adapt the contrastive learning objective to handle the many-to-many relationships between video segments and comments, and to mitigate confusion from similar comments appearing in the same batch.
  2. Downstream Multimodal Fusion Encoder: For actual affective analysis tasks, they employ a multimodal fusion encoder. This encoder integrates features from text, audio, and visual modalities using pre-trained encoders. Crucially, they augment this fusion encoder with the synthetic live comment features generated by the V2LC encoder. This augmentation aims to enrich the multimodal representation with the nuanced emotional insights learned from live comments.

Datasets

  • LCAffect Dataset: The authors introduce a new, large-scale dataset called LCAffect. It comprises over 11 million live comments aligned with 829 hours of video. This dataset is notable for its bilingual (Chinese and English) content, spanning diverse genres including user-generated content, TV shows, documentaries, and movies, aiming to capture a broad spectrum of emotions.
  • Evaluation Datasets: The proposed method is evaluated on six established multimodal affective analysis datasets:
  • CH-SIMS and CH-SIMS v2: For Chinese sentiment analysis.
  • MOSI and MOSEI: For English sentiment analysis.
  • MELD: For emotion recognition.
  • MUSTARD: For sarcasm detection.

Key Insights and Ideas

  • Synthetic Live Comments as a Powerful Augmentation: The most significant insight is that synthetic live comment features, learned from video content, can significantly boost performance in multimodal affective analysis, even outperforming models that rely solely on traditional modalities. This suggests that live comments capture a unique and valuable dimension of emotional expression.
  • Bridging the Data Gap: The V2LC encoder effectively addresses the scarcity of live comment data by enabling the generation of these features for any video, making affective analysis more broadly applicable.
  • Contrastive Learning for Cross-Modal Alignment: The success of the V2LC encoder highlights the power of contrastive learning for aligning representations from different modalities (video and text comments), even when the target modality (live comments) is scarce.
  • Diversity in Pre-training Data is Key: The construction of the LCAffect dataset, with its diverse genres and languages, is crucial for training a V2LC encoder that can generalize well to various types of videos and emotional expressions.

Key Results

The paper reports significant improvements across multiple affective analysis tasks:

  • New State-of-the-Art (SOTA): The proposed method, augmented with synthetic live comment features ("+ LC"), achieves new SOTA performance on several benchmarks.
  • CH-SIMS v2 (Sentiment Analysis): A 3.18-point increase in accuracy.
  • MELD (Emotion Recognition): A 2.89-point increase in weighted F1-score.
  • MUSTARD (Sarcasm Detection): A 3.0-point increase in weighted F1-score.
  • Vanilla Model Performance: Even the authors' "vanilla" Multimodal Fusion Encoder (without synthetic live comment augmentation) outperforms many existing baselines, demonstrating the strength of their multimodal fusion approach.
  • Feature Comparison: The synthetic live comment features, when used with a simple logistic regression, outperform text-based sentiment features trained on large datasets, and are competitive with other specialized features when used in a more expressive multimodal model.

Limitations and Future Work

  • Genre Shift for English Datasets: The authors note that while their method shows improvements on English sentiment analysis datasets (MOSI, MOSEI), the gains are more modest compared to Chinese datasets. They attribute this to a genre shift, as MOSI/MOSEI primarily feature single-speaker monologues, differing from the conversational content in their pre-training corpus.
  • Future Work: The success of this approach opens avenues for further research into:
  • Exploring more sophisticated multimodal fusion techniques.
  • Investigating the transferability of learned live comment features to other affective analysis tasks or even non-affective tasks.
  • Expanding the LCAffect dataset with more diverse content and languages.
  • Developing more fine-grained control over the generation of synthetic live comment features.

Summary updated

BibTeX
@inproceedings{deng2025enhancing,
  author = {Zhaoyuan Deng and Amith Ananthram and Kathleen McKeown},
  title = {{Enhancing Multimodal Affective Analysis with Learned Live Comment Features}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33785},
}
Figure