Introduction
Humor is a quintessentially human trait that remains a "holy grail" for Artificial Intelligence. While AI has made significant strides in sentiment analysis, humor detection is notoriously difficult because it relies heavily on context, sarcasm, and subtle non-verbal cues. Existing research has often focused on "punchline detection" in isolated text or single images, but real-world humor—especially in conversational settings like sitcoms—is a temporal process involving multiple speakers and visual interactions.
Researchers from IIT Kanpur and the University of Bath have addressed this gap by introducing the Multimodal Humor Dataset (MHD) and a novel Multimodal Self-Attention Model (MSAM) designed to predict where laughter tracks should occur in situational comedies.
The Multimodal Humor Dataset (MHD)
The primary contribution of this work is the MHD, a large-scale collection of over 13,000 dialogues derived from the first five seasons of The Big Bang Theory. The dataset features 210 speakers and a vocabulary of over 17,000 words.
Each entry includes the scene description, speaker and listener identities, dialogue text, and precise start/end timestamps for both the speech and the laughter. The authors utilized the show's actual laughter tracks (recorded from a live audience) as a natural proxy for humor, providing a concrete, measurable task for AI to learn the mechanics of situational comedy.
Methodology: The Multimodal Self-Attention Model (MSAM)
The proposed MSAM architecture is designed to handle the sequential and multimodal nature of conversation through several key stages:
- Feature Extraction: The model uses BERT to extract features from dialogue text and C3D networks to extract features from video frames.
- Co-Attention: Instead of simple concatenation, a co-attention mechanism learns which visual cues (like a facial expression) correlate with specific words in the dialogue.
- Self-Attention and Temporal Modeling: A self-attention block captures positional information across multiple dialogue turns, helping the model understand how a joke is "set up." Finally, an LSTM network processes the sequence to make a binary prediction: Humorous or Non-Humorous.
Key Insights and the "Speaker Effect"
The researchers found that model performance peaks when looking at 5 to 6 dialogue turns; fewer turns lack context, while too many dilute the signal. They also confirmed that multimodality is essential, as text-only models often fail to detect sarcasm where the literal words seem serious.
Interestingly, the dataset reveals a strong "Speaker Effect." In The Big Bang Theory, humor is not distributed equally; characters like Sheldon and Leonard account for nearly half of the humorous dialogue turns. This suggests that speaker identity is a powerful feature for humor prediction in scripted media.
Results and Generalization
The MSAM model achieved an accuracy of 72.37% and an F1 score of 81.32%, significantly outperforming text-only and video-only baselines. When tested on other sitcoms like Friends without fine-tuning, the model saw only a minor performance drop of 2-3%, indicating it learned general patterns of humor rather than show-specific tropes. In randomized tests, the model approached human-level performance, which sits at approximately 81% accuracy.
Limitations and Future Work
While promising, the authors acknowledge that current models lack the external "world knowledge" required for some jokes (e.g., cultural references). Future iterations could incorporate audio features like tone and pitch, which are critical for punchline delivery, and explore more fine-grained humor types such as slapstick versus wordplay.
Summary updated