Discriminatively Trained Recurrent Neural Networks for Continuous Dimensional Emotion Recognition from Audio

Felix Weninger, Fabien Ringeval, Erik Marchi, Björn W. SchullerIJCAI 2016

Introduction

Affective computing aims to enable machines to recognize and respond to human emotional states. In continuous emotion recognition, the goal is to predict emotional dimensions—typically Arousal (intensity) and Valence (positivity/negativity)—as they evolve over time. While Deep Recurrent Neural Networks (DRNNs) have become a standard tool for this task, their performance is often limited by the choice of the objective function used during training.

The Limitations of Mean Squared Error

Traditionally, neural networks for regression tasks are trained to minimize the Mean Squared Error (MSE). However, in the context of emotion recognition, MSE has significant drawbacks. Because MSE is a point-wise metric, it focuses on the absolute distance between predicted and target values at each time step, often ignoring the overall temporal "shape" or trend of the emotion.

A common consequence of MSE-based training is that models "play it safe" by predicting values close to the dataset mean. This results in "flat" predictions that may have a low MSE but fail to capture the emotional dynamics (low correlation). Conversely, while Pearson’s Correlation Coefficient (CC) captures the "wiggle" or trend, it is scale-invariant. A model could predict values with the correct shape but incorrect magnitude, and CC would still yield a high score.

Proposed Method: Concordance Correlation Coefficient (CCC)

To address these issues, this paper proposes training DRNNs directly using the Concordance Correlation Coefficient (CCC) as the loss function. CCC is a more comprehensive metric that reaches its maximum value of 1.0 only when the prediction perfectly matches the target in both shape and scale (mean and variance).

Illustration of how shifts and scaling affect signal alignment. While correlation remains high across these variations, the CCC is sensitive to these discrepancies, making it a superior training objective for matching human ratings.

The authors derived the necessary gradients to make CCC differentiable, allowing it to be used in standard backpropagation. They specifically implemented a $∑$CCC objective, which calculates the CCC for each individual audio sequence and sums them. This approach ensures the model learns the unique emotional dynamics of every clip rather than just global statistics.

Experimental Results

The approach was evaluated on two datasets: MediaEval 2013 (music) and RECOLA/AV+EC 2015 (spontaneous speech). The results demonstrate that switching the loss function from MSE (referred to as SSE in the results) to CCC provides a dramatic performance boost.

Performance comparison on the RECOLA dataset. Training with the $∑$CCC objective leads to a substantial increase in CCC scores compared to standard MSE (SSE) training, particularly for arousal prediction.

As shown in the RECOLA results, the CCC score for arousal prediction on the test set improved from 0.097 using standard MSE training to 0.350 with the proposed $∑$CCC training. Similar improvements were observed in the music domain, where the model achieved better results on the official challenge metric (Kendall’s $τ$).

Discussion and Conclusion

The shift to a discriminative training objective like CCC forces the network to match the variance and contours of human emotional ratings. While this can sometimes lead to a slightly higher absolute error (MSE), the resulting models are far more robust and capable of tracking emotional trends across unseen speakers and songs. This technique highlights the importance of aligning the training objective with the desired evaluation metric in sequence-based regression tasks.

Summary updated

BibTeX
@inproceedings{weninger2016discriminatively,
  author = {Felix Weninger and Fabien Ringeval and Erik Marchi and Björn W. Schuller},
  title = {{Discriminatively Trained Recurrent Neural Networks for Continuous Dimensional Emotion Recognition from Audio}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure