TL;DR
This paper introduces a novel method for automatically labeling humorous video segments using time-aligned user comments, achieving high agreement with human annotations. They then leverage these labels to train deep learning models that predict perceived humor from audio and text features, reaching an AUC of 0.751 on a manually annotated test set. This work pioneers large-scale humor prediction from audio data.
What Problem Does This Paper Address?
Understanding and identifying humor is a complex but crucial aspect of human communication. While previous research has explored humor detection, it has largely focused on text-based humor (like jokes or tweets) or relied on artificial cues like canned laughter in sitcoms. A significant challenge has been the lack of large-scale multimedia datasets annotated for humor, which is highly subjective and individualistic. This paper aims to bridge this gap by developing an unsupervised approach to label humor in videos using readily available user comments and then building predictive models that can generalize across different video genres.
Methods
The core innovation lies in their unsupervised humor labeling strategy. They collected videos from Bilibili, a Chinese video-sharing platform, where user comments are time-aligned and displayed on the video itself. Recognizing that laughter is a strong indicator of perceived humor, they identified comments containing Chinese laughter onomatopoeia ("233" and "哈哈"). To account for user reaction and typing time, they established a response window and then applied contextual smoothing to identify peaks of laughter comments within specific time segments. These smoothed peaks were used to generate unsupervised humor labels for 1-second video units.
For humor prediction, they trained machine learning models (Random Forest and CNN) using both acoustic-prosodic features (like pitch, energy, and speaking rate) and text features (TF-IDF and word vectors) derived from the audio and transcriptions of the video segments.
Datasets
The authors created a new corpus called the "Bilibili corpus," consisting of 241 videos (8 comedy movies and 233 gameplay videos) totaling over 77 hours. This corpus includes over 558,000 user comments, with a significant portion containing laughter indicators. They also manually annotated a subset of this corpus to serve as a gold standard for evaluating their unsupervised labels and the performance of their predictive models.
Key Insights and Ideas
- Leveraging User Comments as Implicit Annotations: The paper demonstrates that time-aligned user comments, particularly those expressing laughter, can serve as a surprisingly accurate proxy for human perception of humor, even without explicit human annotation.
- Unsupervised Labeling for Scale: The unsupervised labeling approach is key to creating large datasets for humor prediction, overcoming the bottleneck of manual annotation.
- Generalizability Across Genres: The models trained on one genre (comedy movies) showed promising generalization capabilities when tested on another (gameplay videos), suggesting that some humor cues are universal.
- Multimodal Cues are Crucial: While text alone was insufficient, combining audio features with text significantly improved humor prediction, highlighting the multimodal nature of humor.
Key Results
- The unsupervised humor labels generated from user comments showed high agreement with human annotations (accuracy of 0.881 for comedy movies and 0.942 for gameplay videos).
- The CNN model trained on speech and text features achieved an AUC of 0.751 on the manually annotated test set, which is a significant improvement over chance (AUC of 0.5).
- Acoustic-prosodic features, particularly energy-related ones, were found to be strong indicators of humor, especially in gameplay videos where exaggeration is more pronounced.
- Cross-category experiments showed that models trained on one genre could predict humor in another with AUCs around 0.65, indicating some degree of genre-independent humor detection.
Limitations and Future Work
The authors acknowledge that the humor expressed in comedy movies is often more subtle and relies on a complex interplay of speech, content, facial expressions, and body gestures, making it harder to capture than the more overt humor in gameplay videos. They also note that the accuracy of Automatic Speech Recognition (ASR) for Chinese can be a limitation for text-based analysis.
Future work includes incorporating visual features (facial expressions, body gestures) into their models to achieve a more comprehensive multimodal understanding of humor. They also plan to extend their framework to other live streaming platforms with real-time user comments, further expanding the potential for collecting unsupervised labels.
Summary updated