Laugh-aware virtual agent and its impact on user amusement

Radoslaw Niewiadomski, Jennifer Hofmann, Jérôme Urbain, Tracey Platt, Johannes Wagner, Bilal Piot, Hüseyin Çakmak, Sathish Pammi, Tobias Baur, Stéphane Dupont, Matthieu Geist, Florian Lingenfelser, Gary McKeown, Olivier Pietquin, Willibald Ruch

TL;DR

Researchers have developed a "laugh-aware" virtual agent capable of detecting human laughter in real-time and responding with its own synthesized laughter. The study demonstrates that an interactive agent—one that "shares" the joke with the user—significantly increases the user's felt amusement and creates a sense of social connection through emotional contagion.

What Problem Does This Paper Address?

Laughter is a cornerstone of human social interaction, yet most virtual agents and AI tutors remain "blind" and "deaf" to it. While some systems can detect laughter, few can respond in a way that feels natural or socially appropriate.

The authors address the challenge of creating a closed-loop interaction where an agent doesn't just react to a script, but to the user's behavior and the context of a funny situation. The goal is to move beyond "canned" laughter toward an autonomous companion that can enhance a user's humor experience through shared affect.

Methods

The authors built a modular, real-time system architecture consisting of three main stages:

  1. Input Components: Using a Microsoft Kinect and microphones, the system extracts audio features (like MFCCs and intensity). It uses a Support Vector Machine (SVM) to distinguish laughter from speech/noise and a custom module to estimate laughter intensity in 200ms windows.
  2. Decision Components: This is the "brain" of the agent.
  3. Dialog Manager: Uses a hybrid supervised/unsupervised approach (K-means and Gaussian Mixture Models) to decide when the agent should laugh based on user input and the "funniness" of the video context.
  4. Laughter Planner: Selects the specific laughter episode from a library that best matches the intended duration and intensity.
  5. Output Components: The agent generates both acoustic and visual laughter.
  6. Acoustic: Uses Hidden Markov Model (HMM) based speech synthesis (HTS) to create natural-sounding laughs.
  7. Visual: Driven by a BML Realizer, which uses either manual FACS (Facial Action Coding System) coding or automatic face tracking to animate the agent's expressions.

The researchers tested three conditions: Fixed Speech (agent says "that's funny"), Fixed Laughter (agent laughs at pre-set times), and Interactive Laughter (agent reacts to the user).

Datasets

  • SEMAINE Corpus: Used to train the laughter detection framework.
  • AVLC Database: A collection of ~1,000 audiovisual spontaneous laughter episodes used to train the intensity estimation and provide the basis for the synthesis library.
  • Custom Annotations: 14 human labelers provided continuous "funniness" ratings for the stimulus videos (candid camera pranks) to establish the ground-truth context.

Key Insights and Ideas

  • The Power of Interaction: The most critical finding is that interactivity matters more than the laughter itself. Users felt more amused when the agent responded to them, rather than just laughing at the video.
  • Emotional Contagion: The agent acts as a "social catalyst." By laughing along with the user, the agent facilitates a feedback loop that makes the experience feel like a "shared social experience," even though the agent is virtual.
  • Context + Behavior: A successful agent needs to understand both the "punch line" (context) and the user's reaction (behavior). The Dialog Manager's ability to weigh both inputs is what makes the interaction feel responsive rather than robotic.

Key Results

  • Increased Amusement: The Interactive Laughter (ILC) condition led to significantly higher "felt amusement" compared to fixed conditions ($F(1, 20) = 3.10, p < .05$).
  • Higher Contagion: Users rated the agent's laughter as significantly more "contagious" in the interactive mode ($F(1, 19) = 6.35, p < .05$).
  • Naturalness: Interestingly, the perceived naturalness of the laughter was rated similarly across all conditions (averaging around 3.5 on a 7-point scale). This suggests that while the synthesis isn't perfect yet, the timing of the interaction is what drives the positive user experience.
  • Social Presence: While the interactive agent trended toward higher "social presence" scores, the result was not statistically significant, likely due to the small sample size.

Limitations and Future Work

  • Sample Size: The study was conducted with only 21 participants, which limits the statistical power for some metrics like social presence.
  • Humor Variety: The stimulus was limited to soundless "candid camera" pranks. Different types of humor (verbal, dry, etc.) might require different agent behaviors.
  • Unimodal User Input: Currently, the system only "hears" the user. The authors suggest that adding visual laughter detection (watching the user's face) would improve accuracy.
  • Personality Factors: The authors note that "gelotophobia" (the fear of being laughed at) could drastically change how a user perceives a laughing agent, suggesting future work should account for user personality traits.

Summary updated

Abstract

In this paper we present a complete interactive system enabled to detect human laughs and respond appropriately, by integrating the information of the human behavior and the context. Furthermore, the impact of our autonomous laughter-aware agent on the humor experience of the user and interaction between user and agent is evaluated by subjective and objective means. Preliminary results show that the laughter-aware agent increases the humor experience (i.e., felt amusement of the user and the funniness rating of the film clip), and creates the notion of a shared social experience, indicating that the agent is useful to elicit positive humor-related affect and emotional contagion.
BibTeX
@inproceedings{niewiadomski2013laugh,
  author = {Radoslaw Niewiadomski and Jennifer Hofmann and Jérôme Urbain and Tracey Platt and Johannes Wagner and Bilal Piot and Hüseyin Çakmak and Sathish Pammi and Tobias Baur and Stéphane Dupont and Matthieu Geist and Florian Lingenfelser and Gary McKeown and Olivier Pietquin and Willibald Ruch},
  title = {{Laugh-aware virtual agent and its impact on user amusement}},
  booktitle = {AAMAS},
  year = {2013},
  url = {https://openreview.net/forum?id=QL3UeOHIoj6v},
}

Loading related content...

Figure