How deep neural networks can improve emotion recognition on video data

Pooya Khorrami, Tom Le Paine, Kevin Brady, Charlie K. Dagli, Thomas S. HuangICIP 2016 · DOI

TL;DR

This paper introduces a deep learning framework combining Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to improve dimensional emotion recognition from video. By leveraging CNNs for feature extraction and RNNs for temporal modeling, the proposed CNN+RNN model significantly outperforms previous methods, demonstrating the power of deep learning for capturing nuanced emotional states in video.

What Problem Does This Paper Address?

Recognizing human emotions from video is a long-standing challenge in human-computer interaction. While many approaches have focused on categorizing emotions into discrete labels (like anger, happiness, sadness), this paper tackles the more nuanced task of dimensional emotion recognition. This approach models emotions along continuous scales, typically arousal (engagement) and valence (positivity/negativity), which can capture a broader spectrum of human emotional expression. Existing methods often rely on hand-crafted features or simpler neural network architectures, leaving room for improvement by exploring more powerful deep learning models that can learn richer representations directly from raw video data and account for the temporal dynamics of emotions.

Methods

The authors propose two deep learning frameworks:

  • Single-Frame CNN: This model uses a CNN to extract features from individual video frames. The CNN architecture consists of three convolutional layers with increasing filter counts, followed by max pooling and quadrant pooling. The output is then fed into a fully-connected layer and a linear regression layer to predict the valence score for that frame. This approach treats each frame independently.
  • CNN+RNN: This is the core contribution. It builds upon the single-frame CNN by treating it as a feature extractor. For a given time point, a window of W frames is passed through the pre-trained CNN, generating a sequence of feature vectors. These feature vectors are then fed, one by one, into an RNN. The RNN learns to propagate temporal information across these features, ultimately predicting the valence score at the current time point. This allows the model to capture the dynamic evolution of emotions over time.

Datasets

The primary dataset used for training and evaluation is the AV+EC 2015 corpus. This dataset is derived from the RECOLA dataset and includes audio, video, ECG, and EDA modalities from 27 French-speaking subjects performing a collaborative task. The paper specifically focuses on the video modality and uses the dimensional labels for arousal and valence, which were annotated by six individuals. The dataset is partitioned into train, development, and test sets. The authors evaluate their models on the development set due to the unavailability of test set labels.

Key Insights and Ideas

  • Temporal Dynamics Matter: The most significant insight is that incorporating temporal information via RNNs dramatically improves emotion recognition performance compared to processing frames independently. Emotions are not static; their evolution over time is crucial for accurate recognition.
  • CNNs as Powerful Feature Extractors: The paper demonstrates that CNNs, when trained for emotion recognition, learn valuable features that can be effectively leveraged by subsequent temporal models like RNNs.
  • Dimensionality is Key: The dimensional approach to emotion recognition, as opposed to categorical, allows for a more nuanced and continuous representation of emotions, which is better suited for modeling dynamic video data.
  • Hyperparameter Sensitivity: The performance of the CNN+RNN model is sensitive to hyperparameters like the number of hidden units in the RNN and the temporal window size. Careful tuning is necessary to achieve optimal results.

Key Results

  • The CNN+RNN model significantly outperforms the single-frame CNN and baseline methods.
  • The best CNN+RNN model achieved a CCC score of 0.506, surpassing the baseline method's CCC of 0.273.
  • Adding regularization (dropout and data augmentation) to the single-frame CNN also improved its performance, with the CNN+D model achieving a CCC of 0.326.
  • The CNN+RNN model with a temporal window of 100 frames (approximately 4 seconds) and 100 hidden units in the RNN yielded the best performance.
  • Qualitatively, the CNN+RNN model produced smoother predictions that more accurately approximated the ground truth, especially capturing peaks in valence scores, compared to the single-frame CNN.

Limitations and Future Work

  • Dataset Size: While the AV+EC 2015 dataset is valuable, the authors note that their CNN+RNN model's performance is not as strong as some other CNN+LSTM models that were trained on larger, external datasets. This suggests that more extensive training data could further boost performance.
  • Hyperparameter Optimization: While extensive hyperparameter analysis was conducted, further exploration might yield even better results. The paper suggests that increasing the number of hidden layers in the RNN provided only a small improvement, but optimal configurations might still exist.
  • Future Work: The authors implicitly suggest that exploring larger CNN architectures, different RNN variants (like LSTMs or GRUs), and potentially combining modalities (audio, physiological signals) alongside video could lead to further advancements in emotion recognition.

Summary updated

BibTeX
@inproceedings{khorrami2016how,
  author = {Pooya Khorrami and Tom Le Paine and Kevin Brady and Charlie K. Dagli and Thomas S. Huang},
  title = {{How deep neural networks can improve emotion recognition on video data}},
  booktitle = {ICIP},
  year = {2016},
  doi = {10.1109/icip.2016.7532431},
}
Figure