Abstract
This paper introduces a socially aware virtual avatar designed to participate in the "Yes/No" game by detecting and responding to human laughter in real-time. By integrating audio, facial tracking, and body movement data through a multimodal architecture, the system enables the avatar to mimic the intensity and rhythm of human laughter. Experimental results indicate that while laughter increases the perceived naturalness of the avatar, it can paradoxically decrease perceived competence if not contextually appropriate.
Introduction
In Human-Computer Interaction (HCI), virtual characters often lack the emotional nuance required for natural social engagement, frequently appearing "stiff" or emotionally tone-deaf. While speech recognition has advanced significantly, laughter—a critical and complex social signal—remains under-investigated.
This research addresses the need for "emotional competence" in virtual companions, enabling them to detect subtle emotional displays and respond with appropriate non-verbal cues. The study focuses on the "Yes/No" game, a scenario ripe for spontaneous laughter, requiring a closed-loop system capable of distinguishing between speech, silence, and various laughter types while providing a naturalistic, synthesized response.
Methods and System Architecture
The project, titled Laugh When You’re Winning, utilizes a sophisticated multimodal architecture built upon the Social Signal Interpretation (SSI) framework. This framework synchronizes data from microphones, Kinects, and webcams to provide a holistic view of user behavior.
A "vector fusion" approach is employed, where inputs such as facial Action Units (smiles) and shoulder shaking (detected via the EyesWeb platform) contribute to a "laughter probability" score. This score decays over time to ensure the system remains responsive to the current interaction state. By requiring "agreement" between multiple modalities, the system significantly reduces false positives, such as mistaking a cough for a laugh.
Synthesis and Mimicry
To enhance social presence, a novel mimicry module analyzes the user's laughter properties, such as the periodicity of shoulder movements, to influence the avatar’s response. If a user laughs with high intensity, the avatar attempts to match that energy. The synthesis of the avatar's reaction is handled across three domains:
- Acoustic: Uses Hidden Markov Models (HMMs) trained on the Audiovisual Laughter Corpus (AVLC) to generate laughter sounds.
- Facial: Employs Gaussian Mixture Models (GMMs) to map phonetic laughter sounds to realistic lip shapes and facial expressions.
- Body: Uses a graph-based animation system to trigger movements like leaning back or shoulder rotations based on detected intensity.
Key Results and Insights
A study involving 18 participants compared three conditions: No-Laughter, Laughter, and Mimicry. The Laughter and Mimicry conditions were rated significantly higher for "overall naturalness" compared to the control. Participants specifically noted that the avatar's body movements felt more matching and natural in the Mimicry condition.
However, an interesting trade-off was observed: while laughter increased naturalness, it decreased the perceived competence of the avatar as a game player. If the avatar laughed too frequently or without clear context, users perceived it as distracted or random. Additionally, the researchers identified a "laughter loop" technical hurdle, where the avatar's own synthesized laughter would trigger its detection system, leading to an endless feedback loop of mirth.
Limitations and Future Work
The current system faces several challenges, including a 1-2 second detection latency that can disrupt interaction flow. Future work will focus on improving echo cancellation to prevent self-triggering and moving beyond rule-based dialogue managers toward context-aware systems that understand why a user is laughing. Enhancing the quality of open-source speech synthesis is also a priority to ensure the avatar's questions are clearly understood by human participants.
Summary updated