TimeConvNets: A Deep Time Windowed Convolution Neural Network Design for Real-time Video Facial Expression Recognition

James Ren Hou Lee, Alexander Wong

This paper presents TimeConvNets, a specialized neural network architecture designed for real-time facial expression recognition (FER) in video. By cleverly using 2D convolutions on "stacks" of video frames, the researchers achieved high accuracy and low latency, specifically aiming to create assistive technology for individuals with Autism Spectrum Disorder (ASD).

TL;DR

TimeConvNets achieve state-of-the-art facial expression recognition by treating short sequences of video frames as multi-channel images. This approach captures the "movement" of an emotion with 97.9% accuracy while maintaining a lightning-fast 6ms inference time, making it viable for real-time social assistance tools.

What Problem Does This Paper Address?

For many individuals with Autism Spectrum Disorder (ASD), interpreting the subtle, fleeting changes in a person's face is a significant challenge. While computer vision has made strides in Facial Emotion Classification (FEC), most existing solutions fall into two problematic categories:

  1. Static Classification: Analyzes a single "peak" image. It’s fast but misses the temporal "nuance" (how a smile forms or fades), leading to lower accuracy in natural settings.
  2. Dynamic Video Classification: Uses 3D convolutions or LSTMs to analyze entire video clips. While accurate, these are computationally heavy and slow—too laggy for a real-time conversation.

The authors sought a "Goldilocks" solution: a system fast enough for live interaction but smart enough to understand the transition from a neutral face to an emotional one.

Methods

The core innovation is the TimeConvNet architecture, which strikes a balance between 2D and 3D processing.

  • Time Windowing: Instead of looking at one frame or a whole video, the system looks at a "t-channel sub-sequence stack" (usually 5 consecutive frames).
  • Channel Stacking: These frames are stacked along the color channel dimension. To the network, a 5-frame grayscale sequence looks like a single image with 5 "colors."
  • Spatiotemporal Encoding Layer: The first layer uses 2D convolutions to learn filters that capture both spatial features (eyes, mouth) and temporal changes (movement over those 5 frames) simultaneously.
  • Efficient Backbones: The encoded features are passed into lightweight, proven architectures like mini-Xception, ResNet20, or MobileNetV2 to produce the final emotion label.

This design allows the model to reap the benefits of temporal data without the massive computational cost of 3D convolutions.

Datasets

The authors introduced a new, more challenging benchmark called BigFaceX.

  • Composition: It is a "modified aggregation" of three public datasets: CK+ (posed lab settings), BAUM-1 (spontaneous expressions in Turkish), and eNTERFACE (scripted emotional speech).
  • Scale: It contains 68,363 data points across seven classes (Angry, Disgust, Fear, Happy, Sad, Surprise, and Neutral).
  • Why it matters: Unlike older datasets that only show the "peak" of an emotion, BigFaceX includes the onset and offset phases, forcing the model to learn how expressions actually evolve.

Key Insights and Ideas

  • 2D is enough for 3D problems: You don't necessarily need expensive 3D convolutions to capture time. Stacking frames as channels allows standard 2D kernels to "see" motion across the stack.
  • The "Transient" Nature of Emotion: Expressions aren't static masks; they are events with an onset, peak, and offset. Capturing these transitions is the key to boosting accuracy in "in-the-wild" scenarios where people aren't just holding a pose.
  • Latency is UX: For assistive tech, a 50ms delay can ruin the flow of a conversation. By keeping inference under 10ms, the authors leave plenty of "headroom" for other processing tasks like face detection and tracking.

Key Results

The researchers compared TimeConvNets against standard 2D, 3D, and (2+1)D models. The results were decisive:

  • Accuracy: The TimeConvNet variant using a ResNet20 backbone achieved 97.9% accuracy on the BigFaceX test set.
  • Speed: That same model had an inference time of just 6.14 ms.
  • Efficiency: Compared to a standard 3D ConvNet, TimeConvNet was roughly 6x faster and significantly more accurate.
  • Real-time Performance: When accounting for preprocessing overhead, the authors estimate a total pipeline speed of 25 FPS, which is the standard for smooth video.

Limitations and Future Work

While the results are impressive, the authors note a few areas for improvement:

  • Class Imbalance: Like most FER datasets, BigFaceX has more "Happy" and "Sad" samples than "Fear" or "Surprise," which can bias the model.
  • Environmental Factors: While the dataset includes some variety, real-world issues like heavy facial occlusions (hands, masks) or extreme lighting weren't the primary focus.
  • Language and Culture: The inclusion of BAUM-1 (Turkish) helps, but a truly global system would need even more diverse data to account for how different cultures express emotion and how mouth movements during different languages might confuse the model.
  • Mobile Deployment: The tests were run on a high-end GPU (RTX 2080 Ti). The next step is optimizing these models for mobile or wearable devices (like AR glasses) to actually reach the ASD community.

Summary updated

Abstract

A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people’s emotions based on their facial expressions. With significant recent advances in machine learning, one potential approach to leveraging technology to assist such individuals to better recognize facial expressions and reduce the risk of possible loneliness and depression due to social isolation is the design of computer vision-driven facial expression recognition systems. Motivated by this social need as well as the low latency requirement of such systems, this study explores a novel deep time windowed convolutional neural network design (TimeConvNets) for the purpose of realtime video facial expression recognition. More specifically, we explore an efficient convolutional deep neural network design for spatiotemporal encoding of time windowed video frame subsequences and study the respective balance between speed and accuracy. Furthermore, to evaluate the proposed TimeConvNet design, we introduce a more difficult dataset called BigFaceX, composed of a modified aggregation of the extended Cohn- Kanade (CK+), BAUM-1, and the eNTERFACE public datasets. Different variants of the proposed TimeConvNet design with different backbone network architectures were evaluated using BigFaceX alongside other network designs for capturing spatiotemporal information, and experimental results demonstrate that TimeConvNets can better capture the transient nuances of facial expressions and boost classification accuracy while maintaining a low inference time.
BibTeX
@inproceedings{lee2020timeconvnets,
  author = {James Ren Hou Lee and Alexander Wong},
  title = {{TimeConvNets: A Deep Time Windowed Convolution Neural Network Design for Real-time Video Facial Expression Recognition}},
  booktitle = {CRV},
  year = {2020},
  url = {https://openreview.net/forum?id=Tqo-pMPsp4},
}

Loading related content...

Figure