Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition

Théo Ayral, Marco Pedersoli, Simon Bacon, Eric Granger

Introduction

Training 3D Convolutional Neural Networks (CNNs) on long video sequences presents a significant computational challenge. Due to high memory requirements, standard practice involves extracting a single, randomly chosen short clip to represent an entire video. However, uniform random sampling is often suboptimal. In facial expression recognition (FER), for instance, the most informative "peak" of an expression might only occupy a small fraction of the total video length. Training on uninformative neutral frames labeled with a specific emotion leads to slow convergence and reduced accuracy.

This paper introduces a Temporal Stochastic Softmax strategy designed to identify and prioritize these informative segments during training. By learning a sampling distribution that evolves with the model's performance, this approach achieves higher accuracy and faster convergence in complex video recognition tasks.

Methodology: Temporal Stochastic Softmax

The proposed method avoids the overhead of complex attention modules by integrating a dynamic sampling mechanism directly into the training loop. This is achieved through two primary components:

1. Softmax Temporal Pooling

The model employs a softmax-weighted average of clip scores rather than simple average or max pooling. A temperature parameter ($\gamma$) controls the focus: at $\gamma=0$, it behaves like average pooling; as $\gamma$ increases, it approaches max pooling. This flexibility allows the model to adapt to different video characteristics.

2. Stochastic Sampling and the Training Loop

To avoid the cost of evaluating every possible clip in every epoch, the authors maintain a running estimate of importance scores for all potential clips across the training set.

The Temporal Stochastic Softmax training loop: sampling clips based on learned importance, computing scores, and updating the sampling distribution.

As illustrated in the training loop above, the process involves:

  • Sampling: A clip $x_t$ is selected based on the current estimated distribution $\sigma(w)$.
  • Score Computation: The model computes a classification score $w_t$ for the sampled clip.
  • Distribution Update: The "scoreboard" for that video is updated based on the new score.
  • Backpropagation: The model parameters $\theta$ are updated using the sampled clip.

Training progresses through three phases: warm-up (uniform sampling), exploration (deterministic sampling to populate the scoreboard), and exploitation (stochastic softmax sampling).

Experimental Evaluation

The researchers evaluated the method on three major benchmarks for Facial Expression Recognition (FER) and pain detection:

  • AFEW (Acted Facial Expressions in the Wild): Real-world movie clips featuring six universal emotions.
  • UNBC-McMaster Shoulder Pain: Videos of patients experiencing spontaneous shoulder pain.
  • BioVid Heat Pain: A large-scale dataset of subjects reacting to controlled heat stimuli.

Results and Discussion

The Temporal Stochastic Softmax approach demonstrated significant improvements across all metrics:

  • Enhanced Accuracy: On the AFEW dataset, the method achieved 47.35% accuracy, outperforming the uniform sampling baseline (45.66%). For pain detection in the BioVid dataset, the ROC-AUC improved to 84.39% from a baseline of 82.67%.
  • Efficiency and Convergence: The model reached peak performance with approximately 20% fewer training iterations. In typical runs, convergence was achieved in ~20 epochs compared to ~25 for uniform sampling.
  • Robustness to Noise: The system effectively "auto-trims" videos by ignoring uninformative or occluded frames, focusing instead on high-signal segments.
Visualization of sampling behavior over epochs. The model learns to concentrate samples (dots) on the frames with the highest logit scores, corresponding to the apex of the facial expression.

As shown in the qualitative analysis above, the sampling distribution (represented by the dots across epochs) naturally gravitates toward the frames where the facial expression is most pronounced (the "apex"), which correlates with the highest logit scores.

Conclusion and Future Work

Temporal Stochastic Softmax provides a computationally efficient way to implement "attention" in 3D CNNs. By using the model’s own feedback to guide data selection, it maximizes the information gain from each training step. While the method requires careful tuning of the temperature hyperparameter and a robust warm-up phase, its ability to handle messy, "in-the-wild" video data makes it a powerful tool for video analysis. Future research may explore decoupled temperatures for training and inference or apply this logic to broader action recognition tasks.

Summary updated

Abstract

Training deep learning models for accurate spatiotemporal recognition of facial expressions in videos requires significant computational resources. For practical reasons, 3D Convolutional Neural Networks (3D CNNs) are usually trained with relatively short clips randomly extracted from videos. However, such uniform sampling is generally sub-optimal because equal importance is assigned to each temporal clip. In this paper, we present a strategy for efficient video-based training of 3D CNNs. It relies on softmax temporal pooling and a weighted sampling mechanism to select the most relevant training clips. The proposed softmax strategy provides several advantages – a reduced computational complexity due to efficient clip sampling, and an improved accuracy since temporal weighting focuses on more relevant clips during both training and inference. Experimental results obtained with the proposed method on several facial expression recognition benchmarks show the benefits of focusing on more informative clips in training videos. In particular, our approach improves performance and computational cost by reducing the impact of inaccurate trimming and coarse annotation of videos, and heterogeneous distribution of visual information across time.
BibTeX
@inproceedings{ayral2021temporal,
  author = {Théo Ayral and Marco Pedersoli and Simon Bacon and Eric Granger},
  title = {{Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00307},
}
Figure