Mingling or Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-trained Representations

Siyuan Shen, Feng Liu, Aimin Zhou

TL;DR

This paper introduces a novel "temporal shift module" that enhances speech emotion recognition (SER) by intelligently mixing channel-wise information in pre-trained speech representations. By proposing strategies to balance this "mingling" with potential "misalignment," the authors achieve state-of-the-art results on the IEMOCAP dataset, demonstrating the effectiveness of their approach across various network architectures.

What Problem Does This Paper Address?

Speech emotion recognition (SER) aims to identify human emotions from audio. While pre-trained models like wav2vec 2.0 and HuBERT have shown great promise for SER, most existing methods simply add a linear layer on top of these powerful representations. This approach often overlooks the crucial task of designing specialized downstream networks that can effectively leverage these pre-trained features. The paper argues that simply using pre-trained features with a generic head might not be optimal and that the downstream architecture itself plays a significant role. Furthermore, the paper highlights a potential trade-off: while mixing information across time (temporal shift) can improve performance, it might also introduce misalignment between the original and shifted features, potentially hindering accuracy.

Methods

The core contribution is the temporal shift module. This module is designed to introduce channel-wise information mingling without adding any parameters or computational cost (FLOPs). The module works by shifting a portion of the feature channels along the temporal dimension by one frame. This allows information from adjacent time steps to be incorporated into the current frame's representation.

To address the potential misalignment issue, the authors propose two key strategies:

  • Proportion of Shift ($\alpha$): This hyperparameter controls the ratio of channels that are shifted. A smaller proportion aims to minimize misalignment, while a larger proportion promotes more mingling. The paper explores proportions like 1/2, 1/4, 1/8, and 1/16.
  • Placement of Shift: Two ways of integrating the shift module are proposed:
  • In-place Shift: The shifted data directly replaces the original data, forcing the model to adapt to the new representation.
  • Residual Shift: The shifted features are added to the original features (via a residual connection). This allows the model to benefit from both aligned and mingled information, acting somewhat like an ensemble.

These temporal shift modules are integrated into three common building blocks for SER: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs, specifically LSTMs), and Transformers. This results in new variants named ShiftCNN, ShiftLSTM, and Shiftformer.

Datasets

  • IEMOCAP (Interactive Emotional Dyadic Motion Capture database): This is the primary dataset used for training and evaluation. It's a widely-used benchmark for SER, containing approximately 12 hours of audio across five sessions, with four emotion categories (happy, sad, angry, neutral). The authors use a leave-one-session-out 5-fold cross-validation strategy.

Key Insights and Ideas

  • Temporal Shift as a Parameter-Free, FLOP-Free Module: The ability to improve performance without increasing computational burden is a significant advantage.
  • Balancing Mingling and Misalignment is Crucial: The paper effectively demonstrates that simply shifting information isn't enough; controlling how much and where the shift occurs is key to maximizing benefits while mitigating negative effects.
  • Residual Shift is Generally Beneficial for CNNs: For CNNs, the residual shift strategy proved more effective than in-place shift, especially with smaller kernel sizes. This is attributed to the implicit ensemble effect, combining original and shifted information.
  • Shiftformer Offers a Promising Alternative to Self-Attention: By replacing the self-attention mechanism with temporal shift in a Transformer-like architecture, the Shiftformer achieves strong results with fewer parameters, suggesting a more efficient path for SER.
  • SER Task's Tolerance for Misalignment: The paper notes that the SER task, which focuses on invariant and discriminative representations rather than strict sequence alignment, might be more amenable to temporal shifts than other speech tasks.

Key Results

  • State-of-the-Art Performance: The proposed ShiftCNN, ShiftLSTM, and Shiftformer models consistently outperform existing state-of-the-art methods on the IEMOCAP dataset in both finetuning and feature extraction settings.
  • Improved Accuracy: For example, in feature extraction using wav2vec 2.0 Base features, Shiftformer achieved a higher accuracy than the advanced WavLM Large model by 1.5%.
  • Parameter Efficiency: The Shiftformer, which replaces multi-head self-attention with temporal shift, achieved superior performance with 3.6M fewer parameters compared to a standard Transformer.
  • Ablation Studies: The paper shows that while standard Transformer with residual shift didn't improve, their Shiftformer architecture (which uses temporal shift as a token mixer) was highly effective.

Limitations and Future Work

  • Residual Shift for LSTM: The paper notes that for LSTMs, the projection shortcut added for residual shift introduced extra parameters and led to a performance degradation compared to in-place shift.
  • Future Work: The authors express hope that their temporal shift module will inspire further research into downstream model design for SER and look forward to exploring its applications in other areas. The paper also suggests that the temporal shift can be applied as a data augmentation technique, which showed competitive performance.

Summary updated

BibTeX
@inproceedings{shen2023mingling,
  author = {Siyuan Shen and Feng Liu and Aimin Zhou},
  title = {{Mingling or Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-trained Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095193},
}
Figure