Laugh-aware virtual agent and its impact on user amusement

Radosław Niewiadomski, Jennifer Hofmann, Jérôme Urbain, Tracey Platt, Johannes Wagner, Bilal Piot, Hüseyin Çakmak, Sathish Pammi, Tobias Baur, Stéphane Dupont, Matthieu Geist, Florian Lingenfelser, Gary McKeown, Olivier Pietquin, Willibald Ruch

Abstract

In this paper, we present a complete interactive system enabled to detect human laughs and respond appropriately by integrating information from human behavior and context. Furthermore, the impact of our autonomous laughter-aware agent on the humor experience of the user and the interaction between user and agent is evaluated by subjective and objective means. Preliminary results show that the laughter-aware agent increases the humor experience (i.e., felt amusement of the user and the funniness rating of the film clip) and creates the notion of a shared social experience, indicating that the agent is useful to elicit positive humor-related affect and emotional contagion.

Introduction: The Challenge of Social Laughter

Laughter is a cornerstone of human social interaction, yet most virtual agents and AI tutors remain "blind" and "deaf" to it. While some systems can detect laughter, few can respond in a way that feels natural or socially appropriate. This research addresses the challenge of creating a closed-loop interaction where an agent doesn't just react to a script, but to the user's behavior and the context of a funny situation. The goal is to move beyond "canned" laughter toward an autonomous companion that can enhance a user's humor experience through shared affect.

System Architecture and Methods

The researchers developed a modular, real-time system architecture consisting of three main stages: Input, Decision, and Output components.

The modular system architecture for the laughter-aware agent, showing input, decision, and output components.
  1. Input Components: Using a Microsoft Kinect and microphones, the system extracts audio features such as MFCCs and intensity. A Support Vector Machine (SVM) distinguishes laughter from speech or noise, while a custom module estimates laughter intensity in 200ms windows.
  2. Decision Components: This "brain" of the agent includes a Dialog Manager that uses a hybrid supervised/unsupervised approach (K-means and Gaussian Mixture Models) to decide when the agent should laugh based on user input and the "funniness" of the video context. A Laughter Planner then selects a specific laughter episode from a library that matches the intended duration and intensity.
  3. Output Components: The agent generates both acoustic and visual laughter. Acoustic output uses Hidden Markov Model (HMM) based speech synthesis (HTS), while visual output is driven by a BML Realizer, animating expressions via manual FACS coding or automatic face tracking.

The study compared three conditions: Fixed Speech (agent says "that's funny"), Fixed Laughter (agent laughs at pre-set times), and Interactive Laughter (agent reacts dynamically to the user).

Datasets and Contextual Ground Truth

To train and validate the system, several datasets were utilized:

  • SEMAINE Corpus: Used for training the laughter detection framework.
  • AVLC Database: A collection of ~1,000 audiovisual spontaneous laughter episodes used for intensity estimation and the synthesis library.
  • Custom Annotations: 14 human labelers provided continuous "funniness" ratings for stimulus videos (candid camera pranks) to establish ground-truth context for the Dialog Manager.

Key Insights: The Power of Interaction

The most critical finding is that interactivity matters more than the laughter itself. Users felt more amused when the agent responded to them, rather than just laughing at the video. By laughing along with the user, the agent acts as a "social catalyst," facilitating a feedback loop that makes the experience feel like a "shared social experience." A successful agent must understand both the "punch line" (context) and the user's reaction (behavior) to feel responsive rather than robotic.

Results and Evaluation

The evaluation focused on naturalness, amusement, contagion, and social presence. The interactive agent significantly outperformed fixed conditions in eliciting positive affect.

Comparison of user experience metrics between Fixed and Interactive conditions.
  • Increased Amusement: The Interactive Laughter condition led to significantly higher "felt amusement" compared to fixed conditions ($F(1, 19) = 3.10$).
  • Higher Contagion: Users rated the agent's laughter as significantly more "contagious" in the interactive mode ($F(1, 19) = 6.35$).
  • Naturalness: Perceived naturalness was rated similarly across conditions (averaging around 3.5 on a 7-point scale), suggesting that while synthesis can be improved, the timing of the interaction is the primary driver of positive experience.
  • Social Presence: While the interactive agent trended toward higher social presence scores, the result was not statistically significant, likely due to the small sample size ($N=21$).

Limitations and Future Directions

While promising, the study identified several areas for growth. The small sample size limits the statistical power for complex metrics like social presence. Additionally, the stimulus was limited to soundless "candid camera" pranks; different types of humor (verbal or dry) might require more nuanced agent behaviors. Future iterations could incorporate visual laughter detection (face tracking) to improve accuracy and account for user personality traits, such as "gelotophobia" (the fear of being laughed at), which may significantly alter perception of the agent.

Summary updated

Abstract

In this paper we present a complete interactive system enabled to detect human laughs and respond appropriately, by integrating the information of the human behavior and the context. Furthermore, the impact of our autonomous laughter-aware agent on the humor experience of the user and interaction between user and agent is evaluated by subjective and objective means. Preliminary results show that the laughter-aware agent increases the humor experience (i.e., felt amusement of the user and the funniness rating of the film clip), and creates the notion of a shared social experience, indicating that the agent is useful to elicit positive humor-related affect and emotional contagion.
BibTeX
@inproceedings{niewiadomski2013laugh,
  author = {Radosław Niewiadomski and Jennifer Hofmann and Jérôme Urbain and Tracey Platt and Johannes Wagner and Bilal Piot and Hüseyin Çakmak and Sathish Pammi and Tobias Baur and Stéphane Dupont and Matthieu Geist and Florian Lingenfelser and Gary McKeown and Olivier Pietquin and Willibald Ruch},
  title = {{Laugh-aware virtual agent and its impact on user amusement}},
  booktitle = {AAMAS},
  year = {2013},
  url = {https://openreview.net/forum?id=QL3UeOHIoj6v},
}
Figure