Unsupervised Low-Rank Representations for Speech Emotion Recognition

Georgios Paraskevopoulos, Efthymios Tzinis, Nikolaos Ellinas, Theodoros Giannakopoulos, Alexandros Potamianos

Abstract: We examine the use of linear and non-linear dimensionality reduction algorithms for extracting low-rank feature representations for speech emotion recognition. Two feature sets are used, one based on low-level descriptors and their aggregations (IS10) and one modeling recurrence dynamics of speech (RQA), as well as their fusion. We report speech emotion recognition (SER) results for learned representations on two databases using different classification methods. Classification with low-dimensional representations yields performance improvement in a variety of settings. This indicates that dimensionality reduction is an effective way to combat the curse of dimensionality for SER. Visualization of features in two dimensions provides insight into discriminatory abilities of reduced feature sets.

Introduction

Speech Emotion Recognition (SER) faces a significant challenge known as the Curse of Dimensionality (CoD). Modern feature extraction toolkits, such as openSMILE, generate thousands of descriptors for each audio segment. However, annotated SER datasets are typically small, containing only a few thousand utterances from a limited number of speakers. This disparity leads to sparse data in high-dimensional spaces, making it difficult for machine learning models to generalize without overfitting to specific speakers. This research investigates whether unsupervised dimensionality reduction (DR)—finding a simpler "latent" representation—can mitigate these effects and improve model performance.

Methodology

The study evaluates a variety of linear and non-linear dimensionality reduction algorithms:

  • Linear Methods: Principal Component Analysis (PCA) and classical Multidimensional Scaling (cMDS).
  • Non-linear (Manifold Learning): Metric and Non-metric MDS (SMACOF, Pattern Search), ISOMAP, Locally Linear Embedding (LLE), and Spectral Embedding.
  • Neural Networks: Undercomplete Autoencoders with ReLU activations.

These techniques were applied to three feature configurations: IS10 (1,582 features including MFCCs and statistical aggregations), RQA (432 features modeling non-linear speech dynamics), and a Fused set (2,014 dimensions). Evaluation was conducted on the Emo-DB (German, 535 utterances) and IEMOCAP (English, ~12 hours) datasets using a Speaker-Independent (Leave-One-Speaker-Out) protocol.

Results

The experiments demonstrate that reducing feature dimensions often leads to superior classification accuracy compared to using the full feature set. For instance, on the IEMOCAP dataset with IS10 features, reducing 1,582 dimensions to just 25–100 dimensions improved Unweighted Accuracy (UA). A linear SVM achieved 56.0% UA with Pattern Search MDS, surpassing the 54.7% UA achieved with the original high-dimensional data.

Performance of various DR methods across different feature sets and datasets.

Across both datasets, distance-preserving transformations like MDS yielded the most consistent improvements, effectively "cleaning" the feature space and allowing classifiers to operate more efficiently on as little as 1.5% of the original dimensions.

Discussion and Insights

Visualizing the reduced feature space in two dimensions provided several key insights:

  • Arousal Encoding: In 2D MDS projections, the primary axis (x-axis) naturally aligned with emotional arousal. High-energy emotions like anger clustered on one side, while low-energy emotions like sadness were positioned on the opposite side.
  • Linearity vs. Complexity: Simple linear methods like PCA often outperformed complex non-linear manifold learning, suggesting the speech feature space is relatively flat.
  • Speaker vs. Emotion: While non-linear methods like ISOMAP were less effective for emotion classification, they showed a strong ability to separate individual speakers, indicating potential utility for speaker diarization.
2D visualization of features showing clustering by emotion and speaker session.

Furthermore, the performance of Autoencoders scaled with data size, performing significantly better on the larger IEMOCAP dataset than on Emo-DB, highlighting the data requirements of deep learning approaches compared to traditional geometric DR methods.

Conclusion

Unsupervised dimensionality reduction is an effective strategy for enhancing SER robustness. By projecting high-dimensional features into a low-rank space, models can overcome the curse of dimensionality and achieve better generalization. Future work will explore end-to-end autoencoders with integrated distance-preserving regularization and the application of geodesic distances for improved speaker separation.

Summary updated

Abstract

We examine the use of linear and non-linear dimensionality reduction algorithms for extracting low-rank feature representations for speech emotion recognition. Two feature sets are used, one based on low-level descriptors and their aggregations (IS10) and one modeling recurrence dynamics of speech (RQA), as well as their fusion. We report speech emotion recognition (SER) results for learned representations on two databases using different classification methods. Classification with low-dimensional representations yields performance improvement in a variety of settings. This indicates that dimensionality reduction is an effective way to combat the curse of dimensionality for SER. Visualization of features in two dimensions provides insight into discriminatory abilities of reduced feature sets.
BibTeX
@inproceedings{paraskevopoulos2019unsupervised,
  author = {Georgios Paraskevopoulos and Efthymios Tzinis and Nikolaos Ellinas and Theodoros Giannakopoulos and Alexandros Potamianos},
  title = {{Unsupervised Low-Rank Representations for Speech Emotion Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=iZmtm3CddC},
}
Figure