Abstract
We examine the use of linear and non-linear dimensionality reduction algorithms for extracting low-rank feature representations for speech emotion recognition. Two feature sets are used, one based on low-level descriptors and their aggregations (IS10) and one modeling recurrence dynamics of speech (RQA), as well as their fusion. We report speech emotion recognition (SER) results for learned representations on two databases using different classification methods. Classification with low-dimensional representations yields performance improvement in a variety of settings. This indicates that dimensionality reduction is an effective way to combat the curse of dimensionality for SER. Visualization of features in two dimensions provides insight into discriminatory abilities of reduced feature sets.
Introduction
Speech Emotion Recognition (SER) faces a significant hurdle: data scarcity. While modern audio feature sets, such as IS10, contain thousands of dimensions, annotated emotional speech datasets are relatively small, often featuring only a few dozen speakers. This mismatch leads to the Curse of Dimensionality (CoD). In a high-dimensional space with few samples, the data becomes sparse, making it nearly impossible for a classifier to learn a generalized boundary without overfitting. This research investigates whether unsupervised dimensionality reduction (DR)—compressing these features into a "low-rank" representation—can act as a form of regularization to improve model robustness and performance.
Methodology
The researchers evaluated a wide array of dimensionality reduction algorithms, categorized into linear methods (Principal Component Analysis (PCA) and classical Multidimensional Scaling (cMDS)) and non-linear methods (Metric and Non-metric MDS, ISOMAP, Locally Linear Embedding (LLE), Spectral Embedding, and Deep Autoencoders). These techniques were applied to three feature configurations:
- IS10: 1,582 features representing statistical functionals of low-level descriptors (pitch, energy, etc.).
- RQA (Recurrence Quantification Analysis): 432 features modeling the non-linear dynamics of speech.
- Fused Set: A combination of both (2,014 dimensions).
The study utilized two cornerstone datasets: Emo-DB (535 utterances, 7 emotions) and IEMOCAP (5,531 utterances, 4 emotions). The resulting low-rank features (ranging from 5 to 100 dimensions) were then evaluated using standard classifiers like k-Nearest Neighbors (kNN), Support Vector Machines (SVM), and Logistic Regression.
Results and Discussion
Across almost all tests, reducing dimensions to a rank of 20–60 provided a "sweet spot" where performance peaked before declining. As illustrated in Figure 1, many unsupervised DR methods achieved accuracies that met or exceeded the performance of the original high-dimensional data (represented by the dashed line).
Key Insights
- Linearity in High Dimensions: Surprisingly, linear methods like PCA often outperformed complex non-linear methods on the IS10 feature set, suggesting these high-dimensional spaces behave more like flat hyperplanes than twisted manifolds.
- DR as a Regularizer: Reducing the feature space significantly often increased classification accuracy, confirming that many features in standard sets are redundant or introduce noise.
- Latent "Arousal" Axis: 2D visualizations (Figure 2) revealed that the primary axis of the reduced space often naturally aligns with "arousal," effectively separating high-energy emotions like anger from low-energy ones like sadness.
- The Speaker Identity Trap: Non-linear methods like ISOMAP were found to be better at separating speakers than emotions. While this makes them less ideal for emotion recognition, it suggests utility for speaker diarization tasks.
Conclusion
This study demonstrates that "less is more" in SER. By applying unsupervised DR techniques, researchers can combat the curse of dimensionality, resulting in simpler models that often outperform their high-dimensional counterparts. Future work will explore end-to-end learning with distance-preserving regularization and more sophisticated fusion strategies to capture both global structure and local dynamics. While powerful, unsupervised DR remains sensitive to domain shifts, necessitating further investigation into robust cross-domain representations.
Summary updated