Abstract
Deep learning has significantly advanced facial analysis, yet audiovisual emotion recognition in unconstrained environments remains challenging due to data scarcity. This paper demonstrates that for small datasets like the Acted Facial Expressions in the Wild (AFEW), a "philosophy of simplicity" outperforms complex models. By utilizing pre-trained ResNet-18 architectures, simple temporal pooling, and late fusion, the authors achieved a state-of-the-art 60.64% accuracy on the AFEW dataset, ranking 4th in the 2018 Emotion in the Wild challenge while maintaining extreme computational efficiency.
Introduction
The primary hurdle in "in-the-wild" emotion recognition is the scarcity of labeled data. Standard benchmarks, such as AFEW, contain fewer than 800 training clips. Feeding such limited data into massive deep neural networks often leads to overfitting, where the model memorizes noise instead of learning emotional patterns. Furthermore, integrating temporal dynamics and fusing audio-visual modalities without an explosion of learnable parameters remains a critical research question.
Methodology
The authors adopt an "Occam’s Razor" approach, systematically selecting the simplest methods that yield high performance.
Visual Modality and Temporal Aggregation
Instead of training from scratch, the pipeline uses a ResNet-18 pre-trained on AffectNet (300,000+ images). This model is fine-tuned on intermediate datasets (RAF and SFEW) before targeting AFEW.
To handle the temporal dimension of video without the overfitting risks of LSTMs or 3D convolutions, a simple frame selection mechanism is employed. Videos are divided into $n$ chunks; the frame with the highest emotion score from each chunk is selected, and their features are averaged.
Audio Modality and Late Fusion
Audio features are extracted using the OpenSmile toolbox and classified using simple models like Random Forests or small Multi-Layer Perceptrons (MLPs). To combine modalities, the authors utilize late fusion—taking a weighted mean of the prediction scores. This avoids the parameter overhead associated with early fusion techniques.
Experimental Setup
The model was evaluated primarily on the AFEW dataset (773 training, 383 validation, and 653 test clips). Pre-training leveraged AffectNet, while intermediate fine-tuning used RAF (12,271 images) and SFEW. Audio MLPs were pre-trained on the IEMOCAP dataset.
Results and Discussion
The proposed approach achieved a state-of-the-art 60.64% accuracy on the AFEW test set, securing 4th place in the 2018 EmotiW challenge.
Efficiency and Robustness
A key finding is the extreme efficiency of the model. As shown in the comparison below, the visual backbone uses only 2 million parameters—significantly fewer than competing ensembles—while achieving superior or competitive accuracy on benchmarks like AFEW.
The LSTM Paradox
Surprisingly, complex temporal models like LSTMs offered no improvement over simple average pooling. In small-data regimes, the additional parameters of an LSTM act as a liability, increasing the risk of overfitting. The authors also noted that validation set performance (with only 383 samples) was often an unreliable predictor of test set success, further justifying the choice of robust, simple models.
Conclusion
This work demonstrates that "less is more" in the context of small-scale audiovisual emotion recognition. By prioritizing transfer learning and simple aggregation, researchers can build models that are both highly accurate and computationally efficient (180M FLOPs), suitable for real-time applications. Future work may involve better exploitation of dataset biases or the adoption of complex temporal models as larger datasets become available.
Summary updated