Multimodal Humor Dataset: Predicting Laughter tracks for Sitcoms

Badri N. Patro, Mayank Lunayach, Deepankar Srivastava, Sarvesh, Hunar Singh, Vinay P. Namboodiri

Introduction

Humor is a quintessentially human trait that remains a "holy grail" for Artificial Intelligence. While AI has made significant strides in sentiment analysis, humor detection is notoriously difficult because it relies heavily on context, sarcasm, and subtle non-verbal cues. Existing research has often focused on "punchline detection" in isolated text or single images, but real-world humor—especially in conversational settings like sitcoms—is a temporal process involving multiple speakers and visual interactions.

Researchers from IIT Kanpur and the University of Bath have addressed this gap by introducing the Multimodal Humor Dataset (MHD) and a novel Multimodal Self-Attention Model (MSAM) designed to predict where laughter tracks should occur in situational comedies.

The Multimodal Humor Dataset (MHD)

The primary contribution of this work is the MHD, a large-scale collection of over 13,000 dialogues derived from the first five seasons of The Big Bang Theory. The dataset features 210 speakers and a vocabulary of over 17,000 words.

Examples from the Multimodal Humor Dataset (MHD) illustrating the integration of scene descriptions, visual frames, and dialogue metadata.

Each entry includes the scene description, speaker and listener identities, dialogue text, and precise start/end timestamps for both the speech and the laughter. The authors utilized the show's actual laughter tracks (recorded from a live audience) as a natural proxy for humor, providing a concrete, measurable task for AI to learn the mechanics of situational comedy.

Methodology: The Multimodal Self-Attention Model (MSAM)

The proposed MSAM architecture is designed to handle the sequential and multimodal nature of conversation through several key stages:

  1. Feature Extraction: The model uses BERT to extract features from dialogue text and C3D networks to extract features from video frames.
  2. Co-Attention: Instead of simple concatenation, a co-attention mechanism learns which visual cues (like a facial expression) correlate with specific words in the dialogue.
  3. Self-Attention and Temporal Modeling: A self-attention block captures positional information across multiple dialogue turns, helping the model understand how a joke is "set up." Finally, an LSTM network processes the sequence to make a binary prediction: Humorous or Non-Humorous.

Key Insights and the "Speaker Effect"

The researchers found that model performance peaks when looking at 5 to 6 dialogue turns; fewer turns lack context, while too many dilute the signal. They also confirmed that multimodality is essential, as text-only models often fail to detect sarcasm where the literal words seem serious.

Distribution of humorous dialogue turns by character, highlighting the significant role of speaker identity in humor prediction.

Interestingly, the dataset reveals a strong "Speaker Effect." In The Big Bang Theory, humor is not distributed equally; characters like Sheldon and Leonard account for nearly half of the humorous dialogue turns. This suggests that speaker identity is a powerful feature for humor prediction in scripted media.

Results and Generalization

The MSAM model achieved an accuracy of 72.37% and an F1 score of 81.32%, significantly outperforming text-only and video-only baselines. When tested on other sitcoms like Friends without fine-tuning, the model saw only a minor performance drop of 2-3%, indicating it learned general patterns of humor rather than show-specific tropes. In randomized tests, the model approached human-level performance, which sits at approximately 81% accuracy.

Limitations and Future Work

While promising, the authors acknowledge that current models lack the external "world knowledge" required for some jokes (e.g., cultural references). Future iterations could incorporate audio features like tone and pitch, which are critical for punchline delivery, and explore more fine-grained humor types such as slapstick versus wordplay.

Summary updated

Abstract

A great number of situational comedies (sitcoms) are being regularly made and the task of adding laughter tracks to these is a critical task. Providing an ability to be able to predict whether something will be humorous to the audience is also crucial. In this project, we aim to automate this task. Towards doing so, we annotate an existing sitcom (`Big Bang Theory') and use the laughter cues present to obtain a manual annotation for this show. We provide detailed analysis for the dataset design and further evaluate various state of the art baselines for solving this task. We observe that existing LSTM and BERT based networks on the text alone do not perform as well as joint text and video or only video-based networks. Moreover, it is challenging to ascertain that the words attended to while predicting laughter are indeed humorous. Our dataset and analysis provided through this paper is a valuable resource towards solving this interesting semantic and practical task. As an additional contribution, we have developed a novel model for solving this task that is a multi-modal self-attention based model that outperforms currently prevalent models for solving this task. The project page for our paper is https://delta-lab-iitk.github.io/Multimodal-Humor-Dataset/.
BibTeX
@inproceedings{patro2021multimodal,
  author = {Badri N. Patro and Mayank Lunayach and Deepankar Srivastava and Sarvesh and Hunar Singh and Vinay P. Namboodiri},
  title = {{Multimodal Humor Dataset: Predicting Laughter tracks for Sitcoms}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00062},
}
Figure