An Occam's Razor View on Learning Audiovisual Emotion Recognition with Small Training Sets

Valentin Vielzeuf, Corentin Kervadec, Stéphane Pateux, Alexis Lechervy, Frédéric Jurie

Abstract

Deep learning has significantly advanced facial analysis, yet audiovisual emotion recognition in unconstrained environments remains challenging due to data scarcity. This paper demonstrates that for small datasets like the Acted Facial Expressions in the Wild (AFEW), a "philosophy of simplicity" outperforms complex models. By utilizing pre-trained ResNet-18 architectures, simple temporal pooling, and late fusion, the authors achieved a state-of-the-art 60.64% accuracy on the AFEW dataset, ranking 4th in the 2018 Emotion in the Wild challenge while maintaining extreme computational efficiency.

Introduction

The primary hurdle in "in-the-wild" emotion recognition is the scarcity of labeled data. Standard benchmarks, such as AFEW, contain fewer than 800 training clips. Feeding such limited data into massive deep neural networks often leads to overfitting, where the model memorizes noise instead of learning emotional patterns. Furthermore, integrating temporal dynamics and fusing audio-visual modalities without an explosion of learnable parameters remains a critical research question.

Methodology

The authors adopt an "Occam’s Razor" approach, systematically selecting the simplest methods that yield high performance.

Visual Modality and Temporal Aggregation

Instead of training from scratch, the pipeline uses a ResNet-18 pre-trained on AffectNet (300,000+ images). This model is fine-tuned on intermediate datasets (RAF and SFEW) before targeting AFEW.

To handle the temporal dimension of video without the overfitting risks of LSTMs or 3D convolutions, a simple frame selection mechanism is employed. Videos are divided into $n$ chunks; the frame with the highest emotion score from each chunk is selected, and their features are averaged.

Overview of the proposed audiovisual emotion recognition architecture, showing the visual and audio branches fused at the score level.

Audio Modality and Late Fusion

Audio features are extracted using the OpenSmile toolbox and classified using simple models like Random Forests or small Multi-Layer Perceptrons (MLPs). To combine modalities, the authors utilize late fusion—taking a weighted mean of the prediction scores. This avoids the parameter overhead associated with early fusion techniques.

Experimental Setup

The model was evaluated primarily on the AFEW dataset (773 training, 383 validation, and 653 test clips). Pre-training leveraged AffectNet, while intermediate fine-tuning used RAF (12,271 images) and SFEW. Audio MLPs were pre-trained on the IEMOCAP dataset.

Results and Discussion

The proposed approach achieved a state-of-the-art 60.64% accuracy on the AFEW test set, securing 4th place in the 2018 EmotiW challenge.

Efficiency and Robustness

A key finding is the extreme efficiency of the model. As shown in the comparison below, the visual backbone uses only 2 million parameters—significantly fewer than competing ensembles—while achieving superior or competitive accuracy on benchmarks like AFEW.

Comparison of the proposed visual model against state-of-the-art methods, highlighting its superior efficiency in terms of parameters and FLOPs.

The LSTM Paradox

Surprisingly, complex temporal models like LSTMs offered no improvement over simple average pooling. In small-data regimes, the additional parameters of an LSTM act as a liability, increasing the risk of overfitting. The authors also noted that validation set performance (with only 383 samples) was often an unreliable predictor of test set success, further justifying the choice of robust, simple models.

Conclusion

This work demonstrates that "less is more" in the context of small-scale audiovisual emotion recognition. By prioritizing transfer learning and simple aggregation, researchers can build models that are both highly accurate and computationally efficient (180M FLOPs), suitable for real-time applications. Future work may involve better exploitation of dataset biases or the adoption of complex temporal models as larger datasets become available.

Summary updated

Abstract

This paper presents a light-weight and accurate deep neural model for audiovisual emotion recognition. To design this model, the authors followed a philosophy of simplicity, drastically limiting the number of parameters to learn from the target datasets, always choosing the simplest learning methods: i) transfer learning and low-dimensional space embedding allows to reduce the dimensionality of the representations, ii) visual temporal information handled by a simple score-per-frame selection process averaged across time, iii) simple frame selection mechanism for weighting images within sequences, iv) fusion of the different modalities at prediction level (late fusion). The paper also highlights the inherent challenges of the AFEW dataset and the difficulty of model selection with as few as 383 validation sequences. The proposed real-time emotion classifier achieved a state-of-the-art accuracy of 60.64 % on the test set of AFEW, and ranked 4th at the Emotion in the Wild 2018 challenge.
BibTeX
@inproceedings{vielzeuf2018occam,
  author = {Valentin Vielzeuf and Corentin Kervadec and Stéphane Pateux and Alexis Lechervy and Frédéric Jurie},
  title = {{An Occam's Razor View on Learning Audiovisual Emotion Recognition with Small Training Sets}},
  booktitle = {ICMI},
  year = {2018},
  url = {https://openreview.net/forum?id=9fI1rXcVZt},
}
Figure