Abstract
A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people's emotions based on their facial expressions. This study explores a novel deep time windowed convolutional neural network design (TimeConvNets) for real-time video facial expression recognition. We explore an efficient convolutional deep neural network design for spatiotemporal encoding of time windowed video frame sub-sequences. To evaluate the design, we introduce BigFaceX, a dataset composed of a modified aggregation of the CK+, BAUM-1, and eNTERFACE datasets. Experimental results demonstrate that TimeConvNets can better capture transient nuances of facial expressions and boost classification accuracy while maintaining low inference time.
Introduction
For many individuals with Autism Spectrum Disorder (ASD), interpreting subtle, fleeting changes in facial expressions is a significant challenge. While computer vision has advanced Facial Emotion Classification (FEC), existing solutions often struggle with a trade-off between speed and temporal awareness. Static classification analyzes single "peak" images, missing the nuances of how emotions form. Conversely, dynamic video classification using 3D convolutions or LSTMs captures temporal data but is often too computationally heavy for real-time interaction. TimeConvNets aims to bridge this gap by providing a system fast enough for live interaction yet capable of understanding the transition from neutral to emotional states.
Methods: The TimeConvNet Architecture
The core innovation of TimeConvNets is its ability to strike a balance between 2D and 3D processing through a specialized approach:
- Time Windowing and Channel Stacking: Instead of processing single frames or entire clips, the system analyzes a "t-channel sub-sequence stack," typically consisting of 5 consecutive frames. These frames are stacked along the color channel dimension, effectively treating a grayscale sequence as a single multi-channel image.
- Spatiotemporal Encoding: The initial layer utilizes 2D convolutions to learn filters that capture both spatial features (e.g., eye and mouth shapes) and temporal changes (movement across the 5-frame window) simultaneously.
- Efficient Backbones: These encoded features are then processed by lightweight architectures such as mini-Xception, ResNet20, or MobileNetV2.
This design allows the model to leverage temporal information without the prohibitive computational cost of 3D convolutions.
Datasets: BigFaceX
To evaluate TimeConvNets, the authors introduced BigFaceX, a more challenging benchmark for video-based facial expression recognition (FER).
BigFaceX is a modified aggregation of three public datasets: CK+ (posed lab settings), BAUM-1 (spontaneous expressions), and eNTERFACE (scripted emotional speech). It contains 68,363 data points across seven classes. Crucially, BigFaceX includes the onset and offset phases of expressions, forcing models to learn the evolution of an emotion rather than just its peak state.
Results and Discussion
TimeConvNets were compared against standard 2D, 3D, and (2+1)D models. Key findings include:
- High Accuracy: The ResNet20-based TimeConvNet achieved 97.9% accuracy on the BigFaceX test set.
- Low Latency: The model maintained an inference time of just 6.14 ms, approximately 6x faster than standard 3D ConvNets.
- Real-time Viability: The total pipeline speed is estimated at 25 FPS, meeting the standard for smooth video processing.
The results suggest that 2D convolutions on stacked frames are sufficient to capture the transient nature of emotions—the onset, peak, and offset—which is critical for "in-the-wild" scenarios where expressions are not static.
Conclusion and Future Work
TimeConvNets offer a promising path toward real-time assistive technology for the ASD community. Future work will address class imbalances in datasets, improve robustness to environmental factors like occlusion and lighting, and optimize models for mobile and wearable deployment (e.g., AR glasses). Additionally, expanding the dataset to include more diverse cultural and linguistic backgrounds will be essential for global applicability.
Summary updated