TimeConvNets: A Deep Time Windowed Convolution Neural Network Design for Real-time Video Facial Expression Recognition

James Ren Hou Lee, Alexander Wong

Abstract

A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people's emotions based on their facial expressions. This study explores a novel deep time windowed convolutional neural network design (TimeConvNets) for real-time video facial expression recognition. We explore an efficient convolutional deep neural network design for spatiotemporal encoding of time windowed video frame sub-sequences. To evaluate the design, we introduce BigFaceX, a dataset composed of a modified aggregation of the CK+, BAUM-1, and eNTERFACE datasets. Experimental results demonstrate that TimeConvNets can better capture transient nuances of facial expressions and boost classification accuracy while maintaining low inference time.

Introduction

For many individuals with Autism Spectrum Disorder (ASD), interpreting subtle, fleeting changes in facial expressions is a significant challenge. While computer vision has advanced Facial Emotion Classification (FEC), existing solutions often struggle with a trade-off between speed and temporal awareness. Static classification analyzes single "peak" images, missing the nuances of how emotions form. Conversely, dynamic video classification using 3D convolutions or LSTMs captures temporal data but is often too computationally heavy for real-time interaction. TimeConvNets aims to bridge this gap by providing a system fast enough for live interaction yet capable of understanding the transition from neutral to emotional states.

Methods: The TimeConvNet Architecture

The core innovation of TimeConvNets is its ability to strike a balance between 2D and 3D processing through a specialized approach:

  1. Time Windowing and Channel Stacking: Instead of processing single frames or entire clips, the system analyzes a "t-channel sub-sequence stack," typically consisting of 5 consecutive frames. These frames are stacked along the color channel dimension, effectively treating a grayscale sequence as a single multi-channel image.
  2. Spatiotemporal Encoding: The initial layer utilizes 2D convolutions to learn filters that capture both spatial features (e.g., eye and mouth shapes) and temporal changes (movement across the 5-frame window) simultaneously.
  3. Efficient Backbones: These encoded features are then processed by lightweight architectures such as mini-Xception, ResNet20, or MobileNetV2.

This design allows the model to leverage temporal information without the prohibitive computational cost of 3D convolutions.

Datasets: BigFaceX

To evaluate TimeConvNets, the authors introduced BigFaceX, a more challenging benchmark for video-based facial expression recognition (FER).

Example sequences from the BigFaceX dataset showing the temporal evolution of facial expressions.

BigFaceX is a modified aggregation of three public datasets: CK+ (posed lab settings), BAUM-1 (spontaneous expressions), and eNTERFACE (scripted emotional speech). It contains 68,363 data points across seven classes. Crucially, BigFaceX includes the onset and offset phases of expressions, forcing models to learn the evolution of an emotion rather than just its peak state.

Results and Discussion

TimeConvNets were compared against standard 2D, 3D, and (2+1)D models. Key findings include:

  • High Accuracy: The ResNet20-based TimeConvNet achieved 97.9% accuracy on the BigFaceX test set.
  • Low Latency: The model maintained an inference time of just 6.14 ms, approximately 6x faster than standard 3D ConvNets.
  • Real-time Viability: The total pipeline speed is estimated at 25 FPS, meeting the standard for smooth video processing.

The results suggest that 2D convolutions on stacked frames are sufficient to capture the transient nature of emotions—the onset, peak, and offset—which is critical for "in-the-wild" scenarios where expressions are not static.

Conclusion and Future Work

TimeConvNets offer a promising path toward real-time assistive technology for the ASD community. Future work will address class imbalances in datasets, improve robustness to environmental factors like occlusion and lighting, and optimize models for mobile and wearable deployment (e.g., AR glasses). Additionally, expanding the dataset to include more diverse cultural and linguistic backgrounds will be essential for global applicability.

Summary updated

Abstract

A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people's emotions based on their facial expressions. With significant recent advances in machine learning, one potential approach to leveraging technology to assist such individuals to better recognize facial expressions and reduce the risk of possible loneliness and depression due to social isolation is the design of computer vision-driven facial expression recognition systems. Motivated by this social need as well as the low latency requirement of such systems, this study explores a novel deep time windowed convolutional neural network design (TimeConvNets) for the purpose of real-time video facial expression recognition. More specifically, we explore an efficient convolutional deep neural network design for spatiotemporal encoding of time windowed video frame sub-sequences and study the respective balance between speed and accuracy. Furthermore, to evaluate the proposed TimeConvNet design, we introduce a more difficult dataset called BigFaceX, composed of a modified aggregation of the extended Cohn-Kanade (CK+), BAUM-1, and the eNTERFACE public datasets. Different variants of the proposed TimeConvNet design with different backbone network architectures were evaluated using BigFaceX alongside other network designs for capturing spatiotemporal information, and experimental results demonstrate that TimeConvNets can better capture the transient nuances of facial expressions and boost classification accuracy while maintaining a low inference time.
BibTeX
@inproceedings{lee2020timeconvnets,
  author = {James Ren Hou Lee and Alexander Wong},
  title = {{TimeConvNets: A Deep Time Windowed Convolution Neural Network Design for Real-time Video Facial Expression Recognition}},
  booktitle = {CRV},
  year = {2020},
  url = {https://openreview.net/forum?id=Tqo-pMPsp4},
}
Figure