Abstract
Children with autism spectrum disorder (ASD) often struggle to produce facial expressions appropriate for social contexts. While many tools exist for emotion recognition, few address production. JEMImE is a new serious game designed to help players learn to produce happiness, anger, and sadness within a 3D virtual environment. Using real-time facial expression recognition and visual feedback gauges, the game guides children through social scenarios. A feasibility study with 23 children with ASD demonstrated significant progression in expression quality and high levels of player engagement, suggesting that JEMImE is a promising tool for clinical validation.
Introduction
Social adaptation for children with ASD is frequently hindered by difficulties in generating context-appropriate facial expressions (FE). While a child might recognize a "happy" face, producing one naturally in response to a social cue is a distinct and often more challenging skill. JEMImE (Jeux Éducatifs pour le Mimétisme des Émotions) addresses this by providing a platform for "production on request" within a gamified 3D world.
The JEMImE System: Real-Time Feedback and Gameplay
Developed in Unity, JEMImE utilizes a sophisticated technical architecture to ensure a seamless user experience. The core of the game is a real-time feedback loop powered by a Pairwise Conditional Random Forests (PCRF) classifier. This algorithm analyzes webcam input to evaluate the child's facial expressions against a trained model.
To make this feedback "transparent" and engaging, the game uses color-coded gauges. As the child attempts an expression, these gauges rise or fall based on the algorithm's confidence level. This allows for immediate self-correction without interrupting the flow of the game. The system architecture uses C++ for the classifier and ZeroMQ middleware to communicate with Unity, ensuring zero-latency feedback—a critical requirement for maintaining immersion.
Study Insights and Results
The feasibility of JEMImE was tested with 23 children with ASD (ages 6–12). The results highlighted several key findings:
- Learning Curve: Children showed a significant improvement in expression accuracy from the beginning to the end of the session ($p < 0.001$), indicating that the algorithmic feedback was effective in guiding their performance.
- Emotion Specificity: Happiness and anger were produced with relatively high accuracy, but sadness proved significantly more difficult for the participants to mimic effectively.
- Motivation through Personalization: A vital component of the game's success was its high "playability." 91% of participants engaged deeply with the avatar customization features, which helped maintain therapeutic engagement.
Conclusion and Future Directions
The beta version of JEMImE demonstrates that automated, real-time feedback can successfully teach facial expression production to children with ASD. Future iterations will focus on retraining the classifier with child-specific datasets to account for differences in facial morphology and expanding the 3D environment with more interactive NPCs and scenarios. Long-term clinical trials are planned to determine if these in-game improvements translate to better social outcomes in the real world.
Summary updated