Abstract This paper presents ManzaiSet, a large-scale multimodal dataset designed to capture viewer responses to Japanese manzai comedy. Comprising 191.8 hours of synchronized facial and audio data from 241 participants, the dataset addresses the Western-centric bias prevalent in current affective computing models. By providing high-quality, stimulus-controlled data from a non-Western cultural context, ManzaiSet enables a deeper understanding of how humor is expressed and perceived. Our analysis reveals a "momentum effect"—where viewer appreciation increases over the course of a session—and identifies distinct clusters of viewer response patterns, challenging common assumptions about audience fatigue.
Introduction
Current "Emotion AI" or affective computing models are predominantly trained on datasets originating from Western cultures. This leads to a significant cultural bias, as emotional expressions—particularly complex ones like humor—are deeply embedded in cultural norms and social hierarchies. For instance, Japanese display rules often encourage more regulated expressivity in public compared to Western contexts.
Furthermore, existing datasets often force a trade-off: they are either small and highly controlled, which limits their real-world applicability, or massive and "in-the-wild," which makes it difficult to link specific reactions to specific stimuli. ManzaiSet fills this "missing quadrant" by being both large-scale and stimulus-controlled within a non-Western context, capturing how a diverse group of individuals reacts to the same professional comedy performances in a naturalistic home setting.
Methods
The researchers developed a custom web-based system to record participants as they watched 10 professional manzai performances in a randomized order. This setup allowed for spontaneous reactions while maintaining control over the visual and auditory stimuli.
Data processing involved several advanced tools:
- Facial Analysis: OpenFace 2.0 was used to extract facial action units (AUs) and head pose data from the 640x360, 25 fps video.
- Audio Analysis: Audio (48 kHz) was transcribed using Whisper and analyzed for laughter events.
- Humor Classification: To categorize the comedic stimuli, the authors utilized GPT-5-mini to classify 77 specific humor instances into nine types, such as dajare (wordplay), oogesa (exaggeration), and igai-sei (unexpected twists).
- Statistical Modeling: K-means clustering was employed to identify viewer "types," and Generalized Estimating Equations (GEE) were used to model the relationship between humor types and response intensity.
Dataset Characteristics
ManzaiSet stands out due to its scale and cultural specificity. The dataset includes 2,301 viewing sessions from 241 Japanese viewers (with 228 completing at least 8 videos). The content consists of 10 professional manzai performances from the "Legendary Day" event by Yoshimoto Kogyo.
Key Insights and Results
The analysis of ManzaiSet yielded several significant findings that challenge traditional assumptions in affective computing:
- The Momentum Effect: Contrary to the "fatigue hypothesis," which suggests that audience engagement wanes over time, the data showed a significant positive viewing-order effect. Mean subjective "funniness" ratings rose from 77.95 for the first video to 83.15 for the tenth, a 6.7% increase. This suggests that comedy builds momentum as viewers become more immersed in the experience.
- Diversity of Viewer Responses: Clustering revealed three distinct viewer types. While the majority (72.8%) were "High & Stable Appreciators," nearly 27% fell into "Variable Improver" or "Decliner" categories. This diversity indicates that standard emotion models trained on "average" responses may fail to accurately represent over a quarter of the population.
- Execution vs. Category: Statistical modeling found no reliable difference in laughter intensity between different types of humor (e.g., physical gags vs. puns). This suggests that in manzai, the quality of execution, timing, and the "boke-tsukkomi" (funny man/straight man) dynamic are more critical to the audience's response than the specific comedic mechanism employed.
- Response Stability: Individual rating styles were remarkably consistent (split-half reliability r = 0.866), suggesting that humor appreciation is a stable, trait-like characteristic within individuals.
Conclusion and Future Work
ManzaiSet provides a robust baseline for Japanese affective responses and a valuable resource for developing culturally-aware AI. However, the authors acknowledge that the current dataset is limited to Japanese viewers. To achieve truly universal emotion recognition, future work must include cross-cultural comparisons using identical stimuli.
The authors envision this data powering "laugh-aware" interactive agents capable of detecting amusement and adjusting their behavior in real-time. Future research could also involve finer-grained human annotation of "micro-responses" and the study of the "contagion" effect—how laughter builds within a session—to further refine our understanding of the social and cultural dynamics of humor.
Summary updated