TL;DR
This paper investigates whether sentiment analysis models trained on video transcripts perform better when the sentiment labels are derived solely from text or from a combination of text, audio, and visual cues. The study finds that text-only annotations lead to more robust sentiment models, highlighting challenges in cross-modal generalization.
What Problem Does This Paper Address?
The proliferation of user-generated content (UGC) on platforms like YouTube and TikTok has made video a rich source of public opinion. However, automatically analyzing sentiment in videos is complex, often relying on transcriptions. A key question arises: when annotating these transcripts for sentiment, should annotators consider only the text, or should they incorporate information from the video's audio and visual elements? While multimodal sentiment analysis aims to leverage all cues, it's unclear if this richer annotation translates to better performance when the ultimate goal is to analyze text-based transcripts. This paper addresses this gap by comparing sentiment models trained on text-only versus multimodal annotations of video transcripts.
Methods
The researchers conducted a two-pronged approach:
- Statistical Analysis: They analyzed the distribution of sentiment labels (positive, negative, neutral) across textual and multimodal annotations for two datasets. They also used Chi-Square tests to assess the statistical significance of the relationship between annotation types and Cohen's kappa to measure inter-annotator agreement. Confusion matrices were generated to visualize how sentiment labels shift between modalities.
- Sentiment Modelling: They fine-tuned pre-trained language models (RoBERTa-base for English and a Chinese RoBERTa for Chinese) on the video transcript data. Models were trained using both text-only and multimodal sentiment annotations and then evaluated on both text-only and multimodal test sets. This allowed them to assess performance and generalization capabilities.
Datasets
The study utilized two datasets:
- UniC: An English audio-visual emotion dataset collected from YouTube, focusing on reviews. It provides independent annotations for text, audio, and silent video, as well as overall video sentiment. This is considered a User-Generated Content (UGC) dataset.
- CH-SIMS: A Chinese multimodal sentiment analysis dataset featuring over 2,000 video segments from movies, TV series, and variety shows. It includes both multimodal and independent unimodal annotations. This is considered a non-UGC dataset.
Key Insights and Ideas
- Textual Annotations Lead to Better Text-Based Models: The most significant finding is that sentiment models fine-tuned with text-only annotations consistently outperform those trained with multimodal annotations, even when evaluated on multimodal data. This suggests that text-based models are more adept at learning from and generalizing textual sentiment cues.
- Multimodal Annotations Don't Always Align with Text: The study highlights that multimodal sentiment labels don't always reflect the sentiment expressed purely in the text. A simple example is the text "I love this weather" which might be positive, but if accompanied by a sarcastic tone or a frown, the multimodal sentiment could be negative.
- Cross-Modality Generalization is Challenging: There's a notable performance drop when models trained on one modality (e.g., text) are evaluated on another (e.g., multimodal), and vice-versa. This indicates that models struggle to generalize sentiment understanding across different types of information.
- Dataset Differences Matter: The study observed differences in annotation agreement and sentiment shifts between the UniC (UGC) and CH-SIMS (non-UGC) datasets, suggesting that the nature of the content and annotator expertise can influence multimodal sentiment.
Key Results
- Agreement: While statistically significant relationships exist between textual and multimodal annotations, the agreement is not perfect. In UniC, 63.69% of instances had the same sentiment across modalities, and in CH-SIMS, this was 69.09%. Cohen's kappa scores were moderate (0.4964 for UniC, 0.5494 for CH-SIMS).
- Sentiment Modelling Performance:
- Models fine-tuned with textual annotations and evaluated on textual data achieved higher F1 scores (e.g., UniC-text: 74.62, CH-SIMS-text: 68.33).
- When these text-trained models were evaluated on multimodal data, their performance dropped significantly (e.g., UniC: F1 from 74.62 to 53.03).
- Models fine-tuned with multimodal annotations showed moderate performance (e.g., UniC: F1 43.38, CH-SIMS: F1 45.13), indicating limitations in text-based models generalizing to multimodal inputs.
- Cross-Modality Gap: For UniC, the macro F1 score dropped from 74.62 (textual evaluation) to 53.03 (multimodal evaluation) when models were fine-tuned on textual data, underscoring a significant generalization gap.
Limitations and Future Work
- Linguistic Differences: A key limitation is the use of datasets in different languages (English UniC vs. Chinese CH-SIMS). This linguistic difference could influence the comparison between UGC and non-UGC content, and future work should ideally use datasets from the same linguistic and cultural context.
- Future Directions: The authors suggest incorporating additional modalities (audio, facial expressions) and exploring advanced multimodal fusion models for a more comprehensive analysis. They also aim to investigate the integration of more sophisticated models to bridge the cross-modality generalization gap.
Summary updated