A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

Kazuki Kawamura, Kengo Nakai, Jun Rekimoto

Abstract This paper presents ManzaiSet, a large-scale multimodal dataset designed to capture viewer responses to Japanese manzai comedy. Comprising 191.8 hours of synchronized facial and audio data from 241 participants, the dataset addresses the Western-centric bias prevalent in current affective computing models. By providing high-quality, stimulus-controlled data from a non-Western cultural context, ManzaiSet enables a deeper understanding of how humor is expressed and perceived. Our analysis reveals a "momentum effect"—where viewer appreciation increases over the course of a session—and identifies distinct clusters of viewer response patterns, challenging common assumptions about audience fatigue.

Introduction

Current "Emotion AI" or affective computing models are predominantly trained on datasets originating from Western cultures. This leads to a significant cultural bias, as emotional expressions—particularly complex ones like humor—are deeply embedded in cultural norms and social hierarchies. For instance, Japanese display rules often encourage more regulated expressivity in public compared to Western contexts.

Furthermore, existing datasets often force a trade-off: they are either small and highly controlled, which limits their real-world applicability, or massive and "in-the-wild," which makes it difficult to link specific reactions to specific stimuli. ManzaiSet fills this "missing quadrant" by being both large-scale and stimulus-controlled within a non-Western context, capturing how a diverse group of individuals reacts to the same professional comedy performances in a naturalistic home setting.

Methods

The researchers developed a custom web-based system to record participants as they watched 10 professional manzai performances in a randomized order. This setup allowed for spontaneous reactions while maintaining control over the visual and auditory stimuli.

The data collection setup: participants watched manzai performances at home while their spontaneous reactions were recorded via a web browser.

Data processing involved several advanced tools:

  • Facial Analysis: OpenFace 2.0 was used to extract facial action units (AUs) and head pose data from the 640x360, 25 fps video.
  • Audio Analysis: Audio (48 kHz) was transcribed using Whisper and analyzed for laughter events.
  • Humor Classification: To categorize the comedic stimuli, the authors utilized GPT-5-mini to classify 77 specific humor instances into nine types, such as dajare (wordplay), oogesa (exaggeration), and igai-sei (unexpected twists).
  • Statistical Modeling: K-means clustering was employed to identify viewer "types," and Generalized Estimating Equations (GEE) were used to model the relationship between humor types and response intensity.

Dataset Characteristics

ManzaiSet stands out due to its scale and cultural specificity. The dataset includes 2,301 viewing sessions from 241 Japanese viewers (with 228 completing at least 8 videos). The content consists of 10 professional manzai performances from the "Legendary Day" event by Yoshimoto Kogyo.

Summary statistics of the ManzaiSet dataset, highlighting the scale of the recording duration and participant pool.

Key Insights and Results

The analysis of ManzaiSet yielded several significant findings that challenge traditional assumptions in affective computing:

  • The Momentum Effect: Contrary to the "fatigue hypothesis," which suggests that audience engagement wanes over time, the data showed a significant positive viewing-order effect. Mean subjective "funniness" ratings rose from 77.95 for the first video to 83.15 for the tenth, a 6.7% increase. This suggests that comedy builds momentum as viewers become more immersed in the experience.
  • Diversity of Viewer Responses: Clustering revealed three distinct viewer types. While the majority (72.8%) were "High & Stable Appreciators," nearly 27% fell into "Variable Improver" or "Decliner" categories. This diversity indicates that standard emotion models trained on "average" responses may fail to accurately represent over a quarter of the population.
  • Execution vs. Category: Statistical modeling found no reliable difference in laughter intensity between different types of humor (e.g., physical gags vs. puns). This suggests that in manzai, the quality of execution, timing, and the "boke-tsukkomi" (funny man/straight man) dynamic are more critical to the audience's response than the specific comedic mechanism employed.
  • Response Stability: Individual rating styles were remarkably consistent (split-half reliability r = 0.866), suggesting that humor appreciation is a stable, trait-like characteristic within individuals.

Conclusion and Future Work

ManzaiSet provides a robust baseline for Japanese affective responses and a valuable resource for developing culturally-aware AI. However, the authors acknowledge that the current dataset is limited to Japanese viewers. To achieve truly universal emotion recognition, future work must include cross-cultural comparisons using identical stimuli.

The authors envision this data powering "laugh-aware" interactive agents capable of detecting amusement and adjusting their behavior in real-time. Future research could also involve finer-grained human annotation of "micro-responses" and the study of the "contagion" effect—how laughter builds within a session—to further refine our understanding of the social and cultural dynamics of humor.

Summary updated

Abstract

We present the first large-scale multimodal dataset of viewer responses to Japanese manzai comedy. We recorded synchronized facial video and audio from 241 native speakers who watched up to ten professional performances in randomized order at home (94.6% watched ≥8; analyses use <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$n=228$</tex>), yielding ≈192 hours of data. Our findings are simple. (1) Viewers cluster into two stable styles of appreciation: about three quarters rate most acts consistently high, whereas about one quarter are more selective and variable; each person's style is stable across videos. (2) Ratings tend to rise, not fall, as a session progresses—a positive “momentum” effect rather than fatigue. (3) Across 77 annotated humor instances spanning nine categories, no single category clearly dominated; delivery and context appear to matter more than labels. This dataset provides a culturally grounded benchmark for affective computing and supports personalization and cross-cultural evaluation of emotion-aware systems.
BibTeX
@misc{kawamura2025multimodal,
  author = {Kazuki Kawamura and Kengo Nakai and Jun Rekimoto},
  title = {{A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy}},
  year = {2025},
  eprint = {2510.18014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.18014},
}

Loading related content...

Figure