TL;DR
Current objective evaluations for emotional speech synthesis rely heavily on Emotion Similarity (EMO-SIM) metrics derived from models like emotion2vec. This paper reveals that EMO-SIM is fundamentally flawed for zero-shot evaluation because it is easily disrupted by speaker and linguistic variations, ultimately rewarding acoustic mimicry over genuine emotional expression. The authors demonstrate that these metrics fail to align with human perception and struggle to track fine-grained emotional dimensions.
What Problem Does This Paper Address?
Evaluating how well generative speech models (like expressive Text-to-Speech or Emotional Voice Conversion) capture emotion is notoriously difficult. While human listening tests—specifically Mean Opinion Scores (MOS)—are the gold standard, they are slow, subjective, and expensive.
To automate this process, the speech community widely uses "Emotion Similarity" (EMO-SIM). This metric calculates the cosine similarity between embeddings from state-of-the-art Speech Emotion Recognition (SER) models, most notably emotion2vec. The underlying assumption is that these self-supervised models isolate emotional features from speaker identity and spoken text. This paper challenges that assumption, exposing a critical "false resonance" where the metric measures acoustic similarity rather than actual emotional prosody.
Methods
The authors design a rigorous, systematic pipeline to test EMO-SIM against three core criteria:
- Categorical Emotion Robustness: Using a triplet task $(x_{ref}, x_{pos}, x_{neg})$, they test if the metric can correctly identify matching emotions under adversarial constraints (e.g., introducing speaker or linguistic distractors).
- Dimensional Emotion Sensitivity: They evaluate if the metric correlates with continuous emotional attributes like valence (positivity/negativity) and arousal (energy) using trend monotonicity (Spearman's $\rho$) and shift discriminability.
- Human Perception Alignment: They compare EMO-SIM choices against a dataset of human preference judgments on synthetic speech generated by modern TTS and voice conversion models (e.g., CosyVoice, F5-TTS, MaskGCT).
Additionally, they introduce a Mean Centering Calibration step. They discovered the embedding space is highly anisotropic (compressed into a narrow cone), which artificially inflates uncalibrated similarity scores (consistently ranging between 0.92 and 0.98) and compresses the metric's resolution.
Datasets
The evaluation spans six diverse speech corpora across three languages:
- English: CREMA-D, MSP-Improv, MSP-Podcast
- Chinese: BIIC-Podcast, NNIME
- Russian: Dusha (crowd test set only)
To ensure a true zero-shot evaluation setting, the authors carefully filter out any datasets that were part of the encoders' original pre-training data. They map all dataset-specific labels into four core categories: neutral, happy, sad, and angry.
Key Insights and Ideas
- High Classification Accuracy $\neq$ Good Similarity Metric: Even though models like
emotion2vecexcel at classifying discrete emotions, their raw latent spaces are poorly structured for zero-shot distance or similarity calculations. - Acoustic Mimicry Over Emotion: EMO-SIM is easily fooled by non-affective acoustic features. If a positive sample has a different speaker or text than the reference, the metric often ranks the negative sample (with matching speaker/text but wrong emotion) higher.
- The Deep Layer Curse: A layer-wise analysis of
emotion2vecreveals that deeper layers actually suppress affective features, causing alignment with human judgments to degrade as you go deeper into the network. - Inherited Biases: The authors hypothesize that
emotion2vecdirectly inherits acoustic representations from its foundational model,data2vec. While supervised SER utilizes linear classification layers as filters to suppress these non-affective properties, zero-shot EMO-SIM relies exclusively on cosine similarity, leaving spatial distances dominated by speaker and text features.
Key Results
- Categorical Vulnerability: Under controlled adversarial settings, performance frequently drops to or below random chance (50%). For instance, introducing a linguistic distractor caused base
emotion2vecaccuracy to plummet to a shocking 3.38% on the CREMA-D dataset.
- Dimensional Blindness: The metric completely fails to track continuous emotional magnitude. Spearman's rank correlation ($\rho$) for valence and arousal hovered near zero across all datasets, and shift discriminability accuracy remained near random chance.
- Human Misalignment: When tested against human preference annotations, fine-tuned
emotion2vec+variants only achieved between 52.25% and 65.00% accuracy, barely clearing the random baseline.
- Layer Trajectory: As shown in Figure 2, human perception alignment drops from 58.0% at layer L0 to a sub-random 45.0% by layer L7, proving that deeper layers actively degrade the metric's utility for human alignment.
Limitations and Future Work
The primary limitation highlighted is that current zero-shot EMO-SIM applications treat embeddings as black boxes, leaving them vulnerable to representation crowding and non-affective acoustic features.
To establish valid automated metrics, the authors suggest exploring better representation calibration techniques. Specifically, introducing contrastive learning objectives during encoder training could help suppress acoustic variations (speaker identity and linguistic content) and encourage distinct, well-separated emotional clusters that truly reflect genuine affective prosody.
Summary updated