Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition

Théo Ayral, Marco Pedersoli, Simon Bacon, Eric Granger

Abstract

Training 3D Convolutional Neural Networks (CNNs) for video-based tasks is computationally demanding, typically relying on uniform random sampling of clips. This approach is often suboptimal for Facial Expression Recognition (FER), where informative frames (the "apex") are sparse. We introduce Temporal Stochastic Softmax, a lightweight attention mechanism integrated into the data-loading pipeline. By learning a probability distribution over video segments based on their classification scores, the model prioritizes informative clips during training. Experimental results on AFEW and BioVid datasets demonstrate that this method improves accuracy, enhances robustness to noisy labels, and accelerates convergence by approximately 20%.

Introduction

In video-based Facial Expression Recognition, the most critical information is often concentrated in short temporal windows. A typical video might transition from a neutral state to a peak expression and back. Standard training protocols utilize uniform sampling, treating all clips as equally representative of the global label. This leads to two primary inefficiencies: computational waste on uninformative "neutral" frames and reduced accuracy due to the inclusion of noisy or irrelevant data in the learning process.

The challenge lies in identifying these "important" segments without the prohibitive cost of processing the entire video at every training iteration.

Methods: Temporal Stochastic Softmax

The proposed Temporal Stochastic Softmax strategy replaces uniform sampling with a learned, weighted approach. Instead of a standard attention layer within the network, this method operates at the data-sampling level.

Overview of the Temporal Stochastic Softmax training loop.

The process involves several key steps:

  1. Weighted Sampling: The system maintains a running estimate of "informativeness" scores ($w$) for different temporal segments of each training video.
  2. Softmax Distribution: These scores are converted into a probability distribution using a softmax function. A single clip $x_t$ is sampled based on this distribution.
  3. Iterative Updates: After processing the sampled clip, its classification score for the correct class is used to update the running estimate $w$ for that segment.
  4. Temperature Control: An inverse temperature parameter $\gamma$ allows the model to transition from exploration (uniform sampling) to exploitation (focusing on high-scoring clips) as training progresses.

This stochastic approximation avoids the high variance associated with Reinforcement Learning methods like REINFORCE while remaining computationally efficient.

Key Insights and Results

The method was evaluated on three benchmarks: AFEW (Acted Facial Expressions in the Wild), UNBC-McMaster Shoulder Pain Archive, and the BioVid Heat Pain Database.

Visualization of sampling behavior over epochs. The model learns to focus on informative frames (high logits) as training progresses.

Performance Gains

On the AFEW dataset, the proposed method achieved 47.35% accuracy, a significant improvement over the 45.66% baseline. Notably, the model reached its peak performance roughly 20% faster than uniform sampling. This acceleration occurs because the network spends more time learning from relevant features rather than "neutral" noise.

Robustness to Coarse Annotations

The strength of the stochastic softmax is most evident in datasets like BioVid, where long videos (5.5s) contain only brief moments of the target expression (pain). By automatically ignoring segments where the expression is absent or occluded, the model achieves a 1.5% accuracy boost over uniform baselines. Unlike max-pooling, which can lead to overfitting on a single "easy" clip, the stochastic nature of this approach maintains diversity in the training data, improving generalization.

Discussion and Future Work

While effective, the authors note that the method requires a "warm-up" phase of uniform sampling to avoid premature convergence on suboptimal segments. Additionally, the temperature parameter $\gamma$ requires careful tuning to balance exploration and exploitation. Future research could extend this sampling strategy to multi-modal inputs or broader video tasks such as action recognition, where temporal relevance is equally critical.

Conclusion

Temporal Stochastic Softmax provides a robust and efficient alternative to uniform sampling for 3D CNNs. By focusing training on the most informative temporal regions, it delivers higher accuracy and faster convergence, making it particularly well-suited for real-world video datasets with sparse or noisy labels.

Summary updated

Abstract

Training deep learning models for accurate spatiotemporal recognition of facial expressions in videos requires significant computational resources. For practical reasons, 3D Convolutional Neural Networks (3D CNNs) are usually trained with relatively short clips randomly extracted from videos. However, such uniform sampling is generally sub-optimal because equal importance is assigned to each temporal clip. In this paper, we present a strategy for efficient video-based training of 3D CNNs. It relies on softmax temporal pooling and a weighted sampling mechanism to select the most relevant training clips. The proposed softmax strategy provides several advantages: a reduced computational complexity due to efficient clip sampling, and an improved accuracy since temporal weighting focuses on more relevant clips during both training and inference. Experimental results obtained with the proposed method on several facial expression recognition benchmarks show the benefits of focusing on more informative clips in training videos. In particular, our approach improves performance and computational cost by reducing the impact of inaccurate trimming and coarse annotation of videos, and heterogeneous distribution of visual information across time.
BibTeX
@inproceedings{ayral2021temporal,
  author = {Théo Ayral and Marco Pedersoli and Simon Bacon and Eric Granger},
  title = {{Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00307},
}
Figure