Can Language Models Laugh at YouTube Short-form Videos?

Dayoon Ko, Sangho Lee, Gunhee Kim

Abstract

Humor is a sophisticated human behavior that relies on the intricate interplay of timing, cultural context, and the subversion of expectations. While Large Language Models (LLMs) have demonstrated proficiency in explaining text-based jokes, they frequently struggle with the chaotic, visually-dependent nature of internet "shorts." This paper explores the development of ExFunTube, a large-scale dataset of over 10,000 multimodal YouTube videos annotated with humor explanations. By employing a modular "video-to-text" framework, researchers demonstrate that translating visual and auditory cues into structured chronological prompts significantly enhances an LLM's ability to reason about and explain complex multimodal humor.

1. Introduction

Humor research in Artificial Intelligence has historically been "deaf and blind," focusing almost exclusively on linguistic puns or verbal irony. Even existing video humor datasets are often limited to narrow domains like sitcoms or scripted talks, where the humor remains primarily verbal. However, the modern digital landscape is dominated by short-form multimodal content where the "punchline" is often purely visual or auditory—such as a well-timed pratfall or an absurd sound effect.

Bridging this gap requires models to move beyond simple text processing toward high-level multimodal reasoning. This study addresses whether current LLMs can be taught to understand why a video is funny by providing them with a structured linguistic representation of visual and auditory events.

2. The ExFunTube Dataset

The core contribution of this work is ExFunTube, a dataset comprising 10,136 user-generated funny videos sourced from the "r/youtubehaiku" subreddit. This source was chosen for its focus on short, punchy, and often surreal humor that characterizes modern internet culture.

2.1 Multimodal Filtering

Unlike previous datasets such as MUStARD or UR-FUNNY, ExFunTube specifically targets videos where the humor is inherently multimodal. To ensure this, the researchers used a GPT-3.5-based pipeline to filter out videos where the joke could be understood through the transcript alone. If a model could explain the humor without visual context, the video was discarded.

2.2 Human Annotations

Each video in the dataset includes precise timestamps for the humorous climax and detailed human-written explanations. These annotations describe the specific mechanism of the joke (e.g., "The humor arises when the dog unexpectedly eats the dandelion while the owner is attempting a sentimental gesture"), providing a ground-truth "Gold" standard for model evaluation.

3. Methodology: Video-to-Text Reasoning

Rather than using an end-to-end multimodal architecture—which often struggles with high-level reasoning—the authors adopted a modular "video-to-text" approach. This method translates raw video data into a rich, chronological "script" that leverages the pre-trained reasoning capabilities of LLMs.

  1. Visual Translation: Videos are segmented using PySceneDetect. For each segment, BLIP-2 generates frame captions, and InternVideo selects the most representative action descriptions.
  2. Speech and Audio: Whisper provides speech-to-text transcripts, while ChatGPT performs speaker diarization. To capture non-verbal humor, EfficientAT tags environmental sounds (e.g., "glass shattering," "awkward silence").
  3. Structured Prompting: These elements are synthesized into a chronological narrative: Scene 1: [Visual Action] + [Speaker Dialogue] + (Sound Tag).

This linguistic summary allows the LLM to process a multimodal event as a structured story, applying its internal world knowledge to decode the subversion of expectations.

4. Experimental Results

The researchers evaluated this prompting method against text-only baselines and existing multimodal models using both automatic metrics (SentBERT, ROSCOE) and human evaluation.

4.1 Performance Gains

GPT-3.5, when provided with the proposed multimodal prompts, achieved a SentBERT score of 0.602, a significant improvement over the text-only baseline (0.529). Human evaluators also showed a strong preference for the explanations generated via the video-to-text method, particularly in categories like "Slapstick," "Visual Gags," and "Clownish" humor, where visual context is paramount.

4.2 Modularity vs. End-to-End

Interestingly, the modular approach outperformed several end-to-end multimodal models. This suggests that for complex tasks like humor explanation, the sophisticated reasoning and world knowledge embedded in large-scale language models currently outweigh the benefits of raw visual feature extraction used in integrated architectures.

5. Discussion and Limitations

While the study marks a significant step forward, several challenges remain:

  • The Human-AI Gap: Even the highest-performing models scored lower than human "Gold" explanations in qualitative ratings (0.523 vs 0.792), indicating that true humor "comprehension" is still elusive.
  • Temporal Precision: While sound tags are included, the current pipeline does not perfectly align the micro-timing of sounds with visual frames—a critical component of comedic timing.
  • Subjectivity: Humor is inherently subjective. Future iterations of this work may need to account for individual user preferences and cultural nuances that define what is considered "funny."

6. Conclusion

The ExFunTube dataset and the associated video-to-text prompting framework demonstrate that LLMs can indeed "get the joke" if the visual and auditory world is translated into a language they can process. By structuring multimodal data into a chronological narrative, we can unlock the reasoning power of language models to interpret one of the most complex aspects of human communication: humor.

Summary updated

Abstract

As short-form funny videos on social networks are gaining popularity, it becomes demanding for AI models to understand them for better communication with humans. Unfortunately, previous video humor datasets target specific domains such as speeches or sitcoms, and mostly focus on verbal cues. We curate a user-generated dataset of 10K multimodal funny videos from YouTube, called ExFunTube. Using a video filtering pipeline with GPT-3.5, we verify both verbal and visual elements contributing to humor. After filtering, we annotate each video with timestamps and text explanations for funny moments. Our ExFunTube is unique over existing datasets in that our videos cover a wide range of domains with various types of humor that necessitate a multimodal understanding of the content. Also, we develop a zero-shot video-to-text prompting to maximize video humor understanding of large language models (LLMs). With three different evaluation methods using automatic scores, rationale quality experiments, and human evaluations, we show that our prompting significantly improves LLMs’ ability for humor explanation.
BibTeX
@inproceedings{ko2023can,
  author = {Dayoon Ko and Sangho Lee and Gunhee Kim},
  title = {{Can Language Models Laugh at YouTube Short-form Videos?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.176},
}
Figure