Laugh When You're Winning

Maurizio Mancini, Laurent Ach, Emeline Bantegnie, Tobias Baur, Nadia Berthouze, Debajyoti Datta, Yu Ding, Stéphane Dupont, Harry J. Griffin, Florian Lingenfelser, Radoslaw Niewiadomski, Catherine Pelachaud, Olivier Pietquin, Bilal Piot, Jérôme Urbain, Gualtiero Volpe, Johannes Wagner

Abstract

This paper introduces a socially aware virtual avatar designed to participate in the "Yes/No" game by detecting and responding to human laughter in real-time. By integrating audio, facial tracking, and body movement data through a multimodal architecture, the system enables the avatar to mimic the intensity and rhythm of human laughter. Experimental results indicate that while laughter increases the perceived naturalness of the avatar, it can paradoxically decrease perceived competence if not contextually appropriate.

Introduction

In Human-Computer Interaction (HCI), virtual characters often lack the emotional nuance required for natural social engagement, frequently appearing "stiff" or emotionally tone-deaf. While speech recognition has advanced significantly, laughter—a critical and complex social signal—remains under-investigated.

This research addresses the need for "emotional competence" in virtual companions, enabling them to detect subtle emotional displays and respond with appropriate non-verbal cues. The study focuses on the "Yes/No" game, a scenario ripe for spontaneous laughter, requiring a closed-loop system capable of distinguishing between speech, silence, and various laughter types while providing a naturalistic, synthesized response.

Methods and System Architecture

The project, titled Laugh When You’re Winning, utilizes a sophisticated multimodal architecture built upon the Social Signal Interpretation (SSI) framework. This framework synchronizes data from microphones, Kinects, and webcams to provide a holistic view of user behavior.

The multimodal system architecture for real-time laughter detection and response.

A "vector fusion" approach is employed, where inputs such as facial Action Units (smiles) and shoulder shaking (detected via the EyesWeb platform) contribute to a "laughter probability" score. This score decays over time to ensure the system remains responsive to the current interaction state. By requiring "agreement" between multiple modalities, the system significantly reduces false positives, such as mistaking a cough for a laugh.

Synthesis and Mimicry

To enhance social presence, a novel mimicry module analyzes the user's laughter properties, such as the periodicity of shoulder movements, to influence the avatar’s response. If a user laughs with high intensity, the avatar attempts to match that energy. The synthesis of the avatar's reaction is handled across three domains:

  • Acoustic: Uses Hidden Markov Models (HMMs) trained on the Audiovisual Laughter Corpus (AVLC) to generate laughter sounds.
  • Facial: Employs Gaussian Mixture Models (GMMs) to map phonetic laughter sounds to realistic lip shapes and facial expressions.
  • Body: Uses a graph-based animation system to trigger movements like leaning back or shoulder rotations based on detected intensity.
Examples of the avatar's facial expressions generated via GMM-based mapping.

Key Results and Insights

A study involving 18 participants compared three conditions: No-Laughter, Laughter, and Mimicry. The Laughter and Mimicry conditions were rated significantly higher for "overall naturalness" compared to the control. Participants specifically noted that the avatar's body movements felt more matching and natural in the Mimicry condition.

However, an interesting trade-off was observed: while laughter increased naturalness, it decreased the perceived competence of the avatar as a game player. If the avatar laughed too frequently or without clear context, users perceived it as distracted or random. Additionally, the researchers identified a "laughter loop" technical hurdle, where the avatar's own synthesized laughter would trigger its detection system, leading to an endless feedback loop of mirth.

Limitations and Future Work

The current system faces several challenges, including a 1-2 second detection latency that can disrupt interaction flow. Future work will focus on improving echo cancellation to prevent self-triggering and moving beyond rule-based dialogue managers toward context-aware systems that understand why a user is laughing. Enhancing the quality of open-source speech synthesis is also a priority to ensure the avatar's questions are clearly understood by human participants.

Summary updated

Abstract

Developing virtual characters with naturalistic game playing capabilities is an increasingly researched topic in Human-Computer Interaction. Possible roles for such characters include virtual teachers, personal care assistants, and companions for children. Laughter is an under-investigated emotional expression both in Human-Human and Human-Computer Interaction. The EU Project ILHAIRE, aims to study this phenomena and endow machines with laughter detection and synthesis capabilities. The Laugh when you’re winning project, developed during the eNTERFACE 2013 Workshop in Lisbon, Portugal, aimed to set up and test a game scenario involving two human participants and one such virtual character. The game chosen, the yes/no game, induces natural verbal and non-verbal interaction between participants, includ-ing frequent hilarious events, e.g., one of the participants saying “yes ” or “no ” and so losing the game. The setup includes software platforms, devel-oped by the ILHAIRE partners, allowing automatic analysis and fusion of
BibTeX
@inproceedings{mancini2013laugh,
  author = {Maurizio Mancini and Laurent Ach and Emeline Bantegnie and Tobias Baur and Nadia Berthouze and Debajyoti Datta and Yu Ding and Stéphane Dupont and Harry J. Griffin and Florian Lingenfelser and Radoslaw Niewiadomski and Catherine Pelachaud and Olivier Pietquin and Bilal Piot and Jérôme Urbain and Gualtiero Volpe and Johannes Wagner},
  title = {{Laugh When You're Winning}},
  booktitle = {eNTERFACE},
  year = {2013},
  doi = {10.1007/978-3-642-55143-7\_3},
}
Figure