Introduction
Individuals with Autism Spectrum Disorder (ASD) often face significant challenges in interpreting social and emotional cues. This difficulty in "social motivation"—the innate drive to read others' emotional states—can lead to a profound sense of isolation, increasing the risk of loneliness and depression. While clinical interventions like social skills training are effective in controlled environments, there remains a substantial "translation gap" when applying these skills to the fast-paced, unpredictable nature of daily life.
To address this, researchers have developed FEELS (Full-spectrum Enhanced Emotion Learning System). FEELS is designed as a real-time assistive tool that bridges the gap between clinical learning and real-world social interaction by providing immediate, augmented reality-style emotional feedback.
The FEELS Framework
The FEELS system operates through a four-stage pipeline optimized for low-latency, real-time performance:
- Data Acquisition: The system captures a live video stream of the user's immediate social environment.
- Facial Landmark Detection: Rather than processing raw video frames, which is computationally expensive and noisy, FEELS utilizes a deep Convolutional Neural Network (CNN) to extract 135 facial keypoints. This focuses the model on critical expressive regions like the eyes, nose, and mouth.
- Emotional State Classification: This stage employs a custom hybrid neural network. A sequence of facial landmarks is processed through time-distributed feed-forward layers to capture spatial features, followed by a Long Short-Term Memory (LSTM) network to interpret temporal dynamics.
- Visualization: The identified emotion is overlaid directly onto the live video feed, offering the user a clear, real-time social cue.
Key Innovations
The core strength of FEELS lies in its approach to temporal context and feature engineering. Unlike traditional systems that classify emotions based on static, single frames, FEELS recognizes that human expression is a dynamic process. By using an LSTM, the system can interpret how a smile or a frown unfolds over time, mimicking neurotypical social perception.
Furthermore, the system enhances landmark data with 26 custom geometric features, such as L2 and cosine pair distances relative to the nose tip. This normalization makes the model more robust to variations in head tilt and distance from the camera, ensuring consistent performance in "in the wild" scenarios.
Results and Future Directions
The proof-of-concept FEELS system successfully identifies five primary emotional states: neutral, sad, happy, surprise, and anger. Initial testing demonstrates that the system can accurately label these expressions in real-time, providing a legible user interface for individuals with ASD.
While promising, the current system is limited to a basic emotional vocabulary. Future work aims to expand the system to detect more complex social states—such as embarrassment or empathy—and to conduct rigorous validation studies with individuals in the ASD community to measure its impact on long-term social communication and well-being.
Summary updated