FEELS: a full-spectrum enhanced emotion learning system for assisting individuals with autism spectrum disorder

Amir h. Karimi, Ameneh Boroomand, Kaylen J. Pfisterer, Alexander WongJournal of Computational Vision and Imaging Systems 2018

Introduction

Individuals with Autism Spectrum Disorder (ASD) often face significant challenges in interpreting social and emotional cues. This difficulty in "social motivation"—the innate drive to read others' emotional states—can lead to a profound sense of isolation, increasing the risk of loneliness and depression. While clinical interventions like social skills training are effective in controlled environments, there remains a substantial "translation gap" when applying these skills to the fast-paced, unpredictable nature of daily life.

To address this, researchers have developed FEELS (Full-spectrum Enhanced Emotion Learning System). FEELS is designed as a real-time assistive tool that bridges the gap between clinical learning and real-world social interaction by providing immediate, augmented reality-style emotional feedback.

The FEELS Framework

The FEELS system operates through a four-stage pipeline optimized for low-latency, real-time performance:

  1. Data Acquisition: The system captures a live video stream of the user's immediate social environment.
  2. Facial Landmark Detection: Rather than processing raw video frames, which is computationally expensive and noisy, FEELS utilizes a deep Convolutional Neural Network (CNN) to extract 135 facial keypoints. This focuses the model on critical expressive regions like the eyes, nose, and mouth.
  3. Emotional State Classification: This stage employs a custom hybrid neural network. A sequence of facial landmarks is processed through time-distributed feed-forward layers to capture spatial features, followed by a Long Short-Term Memory (LSTM) network to interpret temporal dynamics.
  4. Visualization: The identified emotion is overlaid directly onto the live video feed, offering the user a clear, real-time social cue.
The FEELS architecture combines time-distributed layers with an LSTM classifier to process sequences of facial landmarks.

Key Innovations

The core strength of FEELS lies in its approach to temporal context and feature engineering. Unlike traditional systems that classify emotions based on static, single frames, FEELS recognizes that human expression is a dynamic process. By using an LSTM, the system can interpret how a smile or a frown unfolds over time, mimicking neurotypical social perception.

Furthermore, the system enhances landmark data with 26 custom geometric features, such as L2 and cosine pair distances relative to the nose tip. This normalization makes the model more robust to variations in head tilt and distance from the camera, ensuring consistent performance in "in the wild" scenarios.

Results and Future Directions

The proof-of-concept FEELS system successfully identifies five primary emotional states: neutral, sad, happy, surprise, and anger. Initial testing demonstrates that the system can accurately label these expressions in real-time, providing a legible user interface for individuals with ASD.

Examples of the FEELS system successfully classifying and labeling neutral, happy, and sad emotional states in real-time.

While promising, the current system is limited to a basic emotional vocabulary. Future work aims to expand the system to detect more complex social states—such as embarrassment or empathy—and to conduct rigorous validation studies with individuals in the ASD community to measure its impact on long-term social communication and well-being.

Summary updated

Abstract

Autism Spectrum Disorder (ASD) is a developmental disorder thatcan lead to a variety of social and communication challenges, andindividuals with ASD are at a higher risk of loneliness and depres-sion as a result of the disconnect and isolation they may feel fromthe rest of society as a result of their ASD. Interventions targetingimproved emotional detection has been clinically shown to be quitepromising; however, there are considerable barriers that make itchallenging to incorporate emotion detection within daily life sce-narios. Motivated by the need to fill this gap, we introduce theconcept of FEELS, a full-spectrum enhanced emotion learning sys-tem which could be useful as a tool to assist individuals with ASD.FEELS facilitates enhanced emotion detection by capturing a livevideo stream of individuals in real-time, then leveraging deep con-volutional neural networks to detect facial landmarks and a customhybrid neural network consisting of a time distributed feed-forwardneural network and a LTSM neural network to determine the emo-tional state of the individuals based on a sequence of facial land-marks over time. The feasibility of such an approach was exploredthrough the construction of a proof-of-concept FEELS system thatcan detect between five different basic emotional states: neutral,sad, happy, surprise, and anger. Future work will include extend-ing the proof-of-concept FEELS system to detect more emotionalstates and evaluate the system in more natural settings.
BibTeX
@article{karimi2018feels,
  author = {Amir h. Karimi and Ameneh Boroomand and Kaylen J. Pfisterer and Alexander Wong},
  title = {{FEELS: a full-spectrum enhanced emotion learning system for assisting individuals with autism spectrum disorder}},
  journal = {Journal of Computational Vision and Imaging Systems},
  year = {2018},
}
Figure