Abstract
In this paper, we present a complete interactive system enabled to detect human laughs and respond appropriately by integrating information from human behavior and context. Furthermore, the impact of our autonomous laughter-aware agent on the humor experience of the user and the interaction between user and agent is evaluated by subjective and objective means. Preliminary results show that the laughter-aware agent increases the humor experience (i.e., felt amusement of the user and the funniness rating of the film clip) and creates the notion of a shared social experience, indicating that the agent is useful to elicit positive humor-related affect and emotional contagion.
Introduction: The Challenge of Social Laughter
Laughter is a cornerstone of human social interaction, yet most virtual agents and AI tutors remain "blind" and "deaf" to it. While some systems can detect laughter, few can respond in a way that feels natural or socially appropriate. This research addresses the challenge of creating a closed-loop interaction where an agent doesn't just react to a script, but to the user's behavior and the context of a funny situation. The goal is to move beyond "canned" laughter toward an autonomous companion that can enhance a user's humor experience through shared affect.
System Architecture and Methods
The researchers developed a modular, real-time system architecture consisting of three main stages: Input, Decision, and Output components.
- Input Components: Using a Microsoft Kinect and microphones, the system extracts audio features such as MFCCs and intensity. A Support Vector Machine (SVM) distinguishes laughter from speech or noise, while a custom module estimates laughter intensity in 200ms windows.
- Decision Components: This "brain" of the agent includes a Dialog Manager that uses a hybrid supervised/unsupervised approach (K-means and Gaussian Mixture Models) to decide when the agent should laugh based on user input and the "funniness" of the video context. A Laughter Planner then selects a specific laughter episode from a library that matches the intended duration and intensity.
- Output Components: The agent generates both acoustic and visual laughter. Acoustic output uses Hidden Markov Model (HMM) based speech synthesis (HTS), while visual output is driven by a BML Realizer, animating expressions via manual FACS coding or automatic face tracking.
The study compared three conditions: Fixed Speech (agent says "that's funny"), Fixed Laughter (agent laughs at pre-set times), and Interactive Laughter (agent reacts dynamically to the user).
Datasets and Contextual Ground Truth
To train and validate the system, several datasets were utilized:
- SEMAINE Corpus: Used for training the laughter detection framework.
- AVLC Database: A collection of ~1,000 audiovisual spontaneous laughter episodes used for intensity estimation and the synthesis library.
- Custom Annotations: 14 human labelers provided continuous "funniness" ratings for stimulus videos (candid camera pranks) to establish ground-truth context for the Dialog Manager.
Key Insights: The Power of Interaction
The most critical finding is that interactivity matters more than the laughter itself. Users felt more amused when the agent responded to them, rather than just laughing at the video. By laughing along with the user, the agent acts as a "social catalyst," facilitating a feedback loop that makes the experience feel like a "shared social experience." A successful agent must understand both the "punch line" (context) and the user's reaction (behavior) to feel responsive rather than robotic.
Results and Evaluation
The evaluation focused on naturalness, amusement, contagion, and social presence. The interactive agent significantly outperformed fixed conditions in eliciting positive affect.
- Increased Amusement: The Interactive Laughter condition led to significantly higher "felt amusement" compared to fixed conditions ($F(1, 19) = 3.10$).
- Higher Contagion: Users rated the agent's laughter as significantly more "contagious" in the interactive mode ($F(1, 19) = 6.35$).
- Naturalness: Perceived naturalness was rated similarly across conditions (averaging around 3.5 on a 7-point scale), suggesting that while synthesis can be improved, the timing of the interaction is the primary driver of positive experience.
- Social Presence: While the interactive agent trended toward higher social presence scores, the result was not statistically significant, likely due to the small sample size ($N=21$).
Limitations and Future Directions
While promising, the study identified several areas for growth. The small sample size limits the statistical power for complex metrics like social presence. Additionally, the stimulus was limited to soundless "candid camera" pranks; different types of humor (verbal or dry) might require more nuanced agent behaviors. Future iterations could incorporate visual laughter detection (face tracking) to improve accuracy and account for user personality traits, such as "gelotophobia" (the fear of being laughed at), which may significantly alter perception of the agent.
Summary updated