Teaching Facial Expression Production in Autism: The Serious Game JEMImE

Charline Grossard, Stéphanie Hun, Arnaud Dapogny, Estelle Juillet, Fanny Hamel, Heidy Jean-Marie, Jérémy Bourgeois, Hugues Pellerin, Pierre Foulon, Sylvie Serret, Ouriel Grynszpan, Kévin Bailly, David CohenCreative 2019 · DOI

Abstract

Children with autism spectrum disorder (ASD) often struggle to produce facial expressions appropriate for social contexts. While many tools exist for emotion recognition, few address production. JEMImE is a new serious game designed to help players learn to produce happiness, anger, and sadness within a 3D virtual environment. Using real-time facial expression recognition and visual feedback gauges, the game guides children through social scenarios. A feasibility study with 23 children with ASD demonstrated significant progression in expression quality and high levels of player engagement, suggesting that JEMImE is a promising tool for clinical validation.

Introduction

Social adaptation for children with ASD is frequently hindered by difficulties in generating context-appropriate facial expressions (FE). While a child might recognize a "happy" face, producing one naturally in response to a social cue is a distinct and often more challenging skill. JEMImE (Jeux Éducatifs pour le Mimétisme des Émotions) addresses this by providing a platform for "production on request" within a gamified 3D world.

The JEMImE System: Real-Time Feedback and Gameplay

Developed in Unity, JEMImE utilizes a sophisticated technical architecture to ensure a seamless user experience. The core of the game is a real-time feedback loop powered by a Pairwise Conditional Random Forests (PCRF) classifier. This algorithm analyzes webcam input to evaluate the child's facial expressions against a trained model.

The JEMImE interface features real-time color gauges that provide immediate feedback on the quality of the player's facial expressions during social scenarios.

To make this feedback "transparent" and engaging, the game uses color-coded gauges. As the child attempts an expression, these gauges rise or fall based on the algorithm's confidence level. This allows for immediate self-correction without interrupting the flow of the game. The system architecture uses C++ for the classifier and ZeroMQ middleware to communicate with Unity, ensuring zero-latency feedback—a critical requirement for maintaining immersion.

Study Insights and Results

The feasibility of JEMImE was tested with 23 children with ASD (ages 6–12). The results highlighted several key findings:

  • Learning Curve: Children showed a significant improvement in expression accuracy from the beginning to the end of the session ($p < 0.001$), indicating that the algorithmic feedback was effective in guiding their performance.
  • Emotion Specificity: Happiness and anger were produced with relatively high accuracy, but sadness proved significantly more difficult for the participants to mimic effectively.
  • Motivation through Personalization: A vital component of the game's success was its high "playability." 91% of participants engaged deeply with the avatar customization features, which helped maintain therapeutic engagement.
Players can personalize their experience through avatar customization, a feature that significantly enhanced engagement and motivation.

Conclusion and Future Directions

The beta version of JEMImE demonstrates that automated, real-time feedback can successfully teach facial expression production to children with ASD. Future iterations will focus on retraining the classifier with child-specific datasets to account for differences in facial morphology and expanding the 3D environment with more interactive NPCs and scenarios. Long-term clinical trials are planned to determine if these in-game improvements translate to better social outcomes in the real world.

Summary updated

Abstract

Background: Children with autism spectrum disorder (ASD) show impairment in producing facial expressions adapted to social contexts. Several serious games have been computed to help them dealing with facial expression recognition but very few focused on facial expression production adapted to a given social context. Method: JEMImE is a new serious game which aims to help the player to learn how to produce happiness, anger and sadness in a 3D virtual environment with social situations that should be resolved by producing the correct facial expression. He is guided in the game thanks to facial expression feedback and gauges that help him evaluating the quality of his/her production in real time. The feedbacks on the children productions are timely given by a facial expression recognition algorithm integrated in JEMImE architecture. Specific attention was paid to the visual and motivational aspects of the game. Using a brief feasibility study with children with ASD (N = 23), we evaluated the impression of the players on the game aspect and the possibility to insert algorithmic feedbacks in real time inside JEMImE. Results: During the training phase, children with ASD showed a significant progression during training for facial expression production after algorithmic autonomous feedbacks. This means children understand the challenge and that the algorithmic feedbacks are transparent enough to allow gaming. They expressed an overall good subjective experience with JEMImE in terms of ergonomics, playability, visual aspect and motivation. Conclusion: We conclude that the beta-version of JEMImE shows promising potential and that research should proceed on computing more games and scenarios to offer a longer game exposure to children to allow adequate clinical validation.
BibTeX
@inproceedings{grossard2019teaching,
  author = {Charline Grossard and Stéphanie Hun and Arnaud Dapogny and Estelle Juillet and Fanny Hamel and Heidy Jean-Marie and Jérémy Bourgeois and Hugues Pellerin and Pierre Foulon and Sylvie Serret and Ouriel Grynszpan and Kévin Bailly and David Cohen},
  title = {{Teaching Facial Expression Production in Autism: The Serious Game JEMImE}},
  booktitle = {Creative},
  year = {2019},
  doi = {10.4236/ce.2019.1011167},
}
Figure