A Lightweight Model Based on Separable Convolution for Speech Emotion Recognition

Ying Zhong, Ying Hu, Hao Huang, Wushour Silamu2020 · DOI

TL;DR

This paper introduces a novel, lightweight model for Speech Emotion Recognition (SER) that significantly reduces parameter count while maintaining high accuracy. By combining depthwise separable convolutions with inverted residuals and employing a focal loss function, the model effectively handles imbalanced and difficult-to-annotate emotional speech data.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) aims to identify human emotions from speech, a crucial capability for more intuitive human-computer interaction. However, a major hurdle in SER is the need for models that are both accurate and computationally efficient, especially given that training data for emotional speech is often limited and can be inconsistently annotated by multiple human raters. Existing deep learning models, while powerful, tend to be parameter-heavy, making them unsuitable for real-time applications or resource-constrained environments. Furthermore, the inherent ambiguity in emotional expression leads to "hard samples" where even human annotators disagree, posing a challenge for model training.

Methods

The proposed model leverages two key architectural innovations:

  • Depthwise Separable Convolutions: Inspired by models like Xception and MobileNetV2, the authors employ depthwise separable convolutions. This technique factorizes a standard convolution into two steps: a depthwise convolution that applies filters to each input channel independently, and a pointwise convolution that combines the outputs. This drastically reduces the number of parameters and computations compared to standard convolutions, leading to a more lightweight model.
  • Inverted Residuals: Building on MobileNetV2, the model incorporates inverted residual blocks. These blocks first expand the low-dimensional input features to a higher dimension using a 1x1 convolution, then apply a depthwise convolution for feature extraction, and finally project the features back to a low-dimensional representation using a linear bottleneck. This design is crucial for maintaining representational power while keeping the model efficient.

Additionally, the paper introduces an attention mechanism in the exit flow, applied after a Bidirectional RNN (Bi-RNN). This mechanism allows the model to focus on the most emotion-salient parts of the speech utterance, enhancing the extraction of discriminative features.

For training, the authors address class imbalance and hard samples by using Focal Loss. This loss function modifies the standard cross-entropy loss by down-weighting easy-to-classify samples and giving more importance to hard-to-classify ones, thereby improving the model's ability to learn from challenging data.

Datasets

The proposed model was evaluated on two well-known SER corpora:

  • IEMOCAP: A 12-hour English conversation database containing 9 emotion classes. For evaluation, it was reduced to 4 classes ({happy, angry, sad, neutral}) by combining certain emotions. This dataset is characterized by multiple annotators per utterance, leading to potential inconsistencies.
  • Emo-DB: A German emotional speech database with 535 utterances across 7 emotions. Unlike IEMOCAP, Emo-DB has only one annotation per utterance.

Key Insights and Ideas

  • Lightweight is Achievable and Effective: The paper demonstrates that significant parameter reduction through separable convolutions and inverted residuals does not necessarily come at the cost of accuracy in SER.
  • Focal Loss for Hard Samples: The use of focal loss proves crucial for improving performance, especially on datasets with imbalanced classes and ambiguous emotional content (hard samples). This highlights the importance of tailored loss functions for specific challenges in SER.
  • Attention for Salient Features: The attention mechanism effectively guides the model to focus on relevant emotional cues within an utterance, leading to better feature representations.
  • Human Difficulty Maps to Model Difficulty: The authors hypothesize that samples difficult for humans to annotate are also difficult for models, justifying the use of focal loss to address these challenging cases.

Key Results

The proposed lightweight model achieved state-of-the-art performance on both IEMOCAP and Emo-DB datasets.

  • On IEMOCAP, the model achieved 71.72% Unweighted Accuracy (UA), 70.37% Weighted Accuracy (WA), and 70.37% F1-score.
  • On Emo-DB, the model achieved 90.1% UA, 91.81% WA, and 90.67% F1-score.

Crucially, the model's parameter count is remarkably low. Compared to existing models like BCRNN, the proposed model's size is approximately 5 times smaller, while still outperforming it.

Limitations and Future Work

  • Limitations: The paper notes that while the model is lightweight, further exploration into optimizing the number of inverted residual blocks for different datasets could be beneficial. The performance on Emo-DB, while high, is achieved with a single annotator per utterance, which is a simpler scenario than IEMOCAP's multi-annotator setup.
  • Future Work: The authors suggest that their lightweight architecture and training strategy can be extended to other SER tasks and potentially deployed on resource-constrained devices for real-time applications. Further research could explore different attention mechanisms or more advanced regularization techniques.

Summary updated

BibTeX
@misc{zhong2020lightweight,
  author = {Ying Zhong and Ying Hu and Hao Huang and Wushour Silamu},
  title = {{A Lightweight Model Based on Separable Convolution for Speech Emotion Recognition}},
  year = {2020},
  doi = {10.21437/interspeech.2020-2408},
}
Figure