Abstract
Training 3D Convolutional Neural Networks (CNNs) for video-based tasks is computationally demanding, typically relying on uniform random sampling of clips. This approach is often suboptimal for Facial Expression Recognition (FER), where informative frames (the "apex") are sparse. We introduce Temporal Stochastic Softmax, a lightweight attention mechanism integrated into the data-loading pipeline. By learning a probability distribution over video segments based on their classification scores, the model prioritizes informative clips during training. Experimental results on AFEW and BioVid datasets demonstrate that this method improves accuracy, enhances robustness to noisy labels, and accelerates convergence by approximately 20%.
Introduction
In video-based Facial Expression Recognition, the most critical information is often concentrated in short temporal windows. A typical video might transition from a neutral state to a peak expression and back. Standard training protocols utilize uniform sampling, treating all clips as equally representative of the global label. This leads to two primary inefficiencies: computational waste on uninformative "neutral" frames and reduced accuracy due to the inclusion of noisy or irrelevant data in the learning process.
The challenge lies in identifying these "important" segments without the prohibitive cost of processing the entire video at every training iteration.
Methods: Temporal Stochastic Softmax
The proposed Temporal Stochastic Softmax strategy replaces uniform sampling with a learned, weighted approach. Instead of a standard attention layer within the network, this method operates at the data-sampling level.
The process involves several key steps:
- Weighted Sampling: The system maintains a running estimate of "informativeness" scores ($w$) for different temporal segments of each training video.
- Softmax Distribution: These scores are converted into a probability distribution using a softmax function. A single clip $x_t$ is sampled based on this distribution.
- Iterative Updates: After processing the sampled clip, its classification score for the correct class is used to update the running estimate $w$ for that segment.
- Temperature Control: An inverse temperature parameter $\gamma$ allows the model to transition from exploration (uniform sampling) to exploitation (focusing on high-scoring clips) as training progresses.
This stochastic approximation avoids the high variance associated with Reinforcement Learning methods like REINFORCE while remaining computationally efficient.
Key Insights and Results
The method was evaluated on three benchmarks: AFEW (Acted Facial Expressions in the Wild), UNBC-McMaster Shoulder Pain Archive, and the BioVid Heat Pain Database.
Performance Gains
On the AFEW dataset, the proposed method achieved 47.35% accuracy, a significant improvement over the 45.66% baseline. Notably, the model reached its peak performance roughly 20% faster than uniform sampling. This acceleration occurs because the network spends more time learning from relevant features rather than "neutral" noise.
Robustness to Coarse Annotations
The strength of the stochastic softmax is most evident in datasets like BioVid, where long videos (5.5s) contain only brief moments of the target expression (pain). By automatically ignoring segments where the expression is absent or occluded, the model achieves a 1.5% accuracy boost over uniform baselines. Unlike max-pooling, which can lead to overfitting on a single "easy" clip, the stochastic nature of this approach maintains diversity in the training data, improving generalization.
Discussion and Future Work
While effective, the authors note that the method requires a "warm-up" phase of uniform sampling to avoid premature convergence on suboptimal segments. Additionally, the temperature parameter $\gamma$ requires careful tuning to balance exploration and exploitation. Future research could extend this sampling strategy to multi-modal inputs or broader video tasks such as action recognition, where temporal relevance is equally critical.
Conclusion
Temporal Stochastic Softmax provides a robust and efficient alternative to uniform sampling for 3D CNNs. By focusing training on the most informative temporal regions, it delivers higher accuracy and faster convergence, making it particularly well-suited for real-world video datasets with sparse or noisy labels.
Summary updated