The Laughing Machine: Predicting Humor in Video

Yuta Kayatani, Zekun Yang, Mayu Otani, Noa Garcia, Chenhui Chu, Yuta Nakashima, Haruo Takemura

Introduction

Humor is a cornerstone of human interaction, yet it remains a formidable challenge for artificial intelligence. Unlike simple sentiment analysis, humor relies on a sophisticated interplay of linguistic setup, timing, and non-verbal cues. While previous research has largely focused on text-based humor detection, real-world comedy is inherently multimodal. This paper explores "The Laughing Machine," a system designed to predict humor in video by leveraging a massive dataset from the sitcom The Big Bang Theory.

Methodology

The researchers proposed a multimodal neural network architecture that integrates four distinct information streams:

  1. Punchline and Setup Modeling: Utilizing BERT for semantic understanding and LSTMs to capture the temporal progression of dialogue. The model analyzes a sequence of subtitles, treating the final line as the potential punchline.
  2. In-story Laughter Modeling: Using the OpenFace toolkit, the system extracts Facial Action Units (FACS) to identify "funny" expressions or physical reactions that might signal humor independently of speech.
  3. Character-based Context: Given that comedic timing and style are often character-specific, the model incorporates face recognition to identify which of the 17 main characters are present in a scene.
  4. Modality Attention: A dynamic weighting mechanism that prioritizes different modalities (text, face, character) based on the specific context of the scene.
Temporal alignment of video frames, humor labels derived from laugh tracks, and spoken utterances.

The Big Bang Theory Dataset

To train and evaluate the model, the authors introduced a substantial new benchmark based on 10 seasons (228 episodes) of The Big Bang Theory. The dataset comprises over 77 hours of video and 31,852 humor labels.

A key innovation in this work was the automated extraction of "ground truth" labels. By subtracting left and right audio channels to cancel out centered dialogue, the researchers isolated the laugh track. A Hilbert transform was then applied to these signals to pinpoint the exact timestamps of audience laughter, which were then mapped back to the corresponding dialogue and video segments.

Distribution of humorous vs. non-humorous utterances across main characters in the dataset.

Key Insights and Results

The study yielded several critical findings regarding the nature of automated humor detection:

  • Verbal Dominance: Surprisingly, the "verbal modality" (the text of the subtitles) proved to be the most powerful predictor. Even when provided with rich visual data, the attention mechanism consistently assigned the highest weights to the punchline modeling.
  • The Importance of Context: The model's performance improved significantly when it considered the four preceding subtitles ("the setup") rather than just the final line. This confirms that humor is a cumulative process.
  • Visual Noise: While facial expressions are vital in specific comedic moments (e.g., deadpan delivery), they were found to be inconsistent signals across the entire series, often introducing noise that could slightly degrade performance in certain configurations.
  • Performance: The best-performing model achieved a baseline accuracy of 70.50%. Interestingly, a text-only model using the punchline and context reached this same threshold, highlighting the current difficulty of effectively integrating visual features.

Conclusion and Future Work

"The Laughing Machine" demonstrates that while text remains the primary signal for humor in scripted sitcoms, multimodal context provides a necessary foundation for a more holistic understanding. Future research will aim to move beyond facial expressions to incorporate body gestures, physical "slapstick" comedy, and the detection of "silent humor" where laughter occurs in the absence of dialogue.

Summary updated

Abstract

Humor is a very important communication tool; yet, it is an open problem for machines to understand humor. In this paper, we build a new multimodal dataset for humor prediction that includes subtitles and video frames, as well as humor labels associated with video’s timestamps. On top of it, we present a model to predict whether a subtitle causes laughter. Our model uses the visual modality through facial expression and character name recognition, together with the verbal modality, to explore how the visual modality helps. In addition, we use an attention mechanism to adjust the weight for each modality to facilitate humor prediction. Interestingly, our experimental results show that the performance boost by combinations of different modalities, and the attention mechanism and the model mostly relies on the verbal modality.
BibTeX
@inproceedings{kayatani2021laughing,
  author = {Yuta Kayatani and Zekun Yang and Mayu Otani and Noa Garcia and Chenhui Chu and Yuta Nakashima and Haruo Takemura},
  title = {{The Laughing Machine: Predicting Humor in Video}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00212},
}
Figure