The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation

Yun-Shao Tsai, Yi-Cheng Lin, Huang-Cheng Chou, Tzu-Wen Hsu, Yun-Man Hsu, Chun Wei Chen, Shrikanth Narayanan, Hung-yi LeearXiv 2026 · arXiv

TL;DR

Current objective evaluations for emotional speech synthesis rely heavily on Emotion Similarity (EMO-SIM) metrics derived from models like emotion2vec. This paper reveals that EMO-SIM is fundamentally flawed for zero-shot evaluation because it is easily disrupted by speaker and linguistic variations, ultimately rewarding acoustic mimicry over genuine emotional expression. The authors demonstrate that these metrics fail to align with human perception and struggle to track fine-grained emotional dimensions.

What Problem Does This Paper Address?

Evaluating how well generative speech models (like expressive Text-to-Speech or Emotional Voice Conversion) capture emotion is notoriously difficult. While human listening tests—specifically Mean Opinion Scores (MOS)—are the gold standard, they are slow, subjective, and expensive.

To automate this process, the speech community widely uses "Emotion Similarity" (EMO-SIM). This metric calculates the cosine similarity between embeddings from state-of-the-art Speech Emotion Recognition (SER) models, most notably emotion2vec. The underlying assumption is that these self-supervised models isolate emotional features from speaker identity and spoken text. This paper challenges that assumption, exposing a critical "false resonance" where the metric measures acoustic similarity rather than actual emotional prosody.

Methods

The authors design a rigorous, systematic pipeline to test EMO-SIM against three core criteria:

  1. Categorical Emotion Robustness: Using a triplet task $(x_{ref}, x_{pos}, x_{neg})$, they test if the metric can correctly identify matching emotions under adversarial constraints (e.g., introducing speaker or linguistic distractors).
  2. Dimensional Emotion Sensitivity: They evaluate if the metric correlates with continuous emotional attributes like valence (positivity/negativity) and arousal (energy) using trend monotonicity (Spearman's $\rho$) and shift discriminability.
  3. Human Perception Alignment: They compare EMO-SIM choices against a dataset of human preference judgments on synthetic speech generated by modern TTS and voice conversion models (e.g., CosyVoice, F5-TTS, MaskGCT).

Additionally, they introduce a Mean Centering Calibration step. They discovered the embedding space is highly anisotropic (compressed into a narrow cone), which artificially inflates uncalibrated similarity scores (consistently ranging between 0.92 and 0.98) and compresses the metric's resolution.

Datasets

The evaluation spans six diverse speech corpora across three languages:

  • English: CREMA-D, MSP-Improv, MSP-Podcast
  • Chinese: BIIC-Podcast, NNIME
  • Russian: Dusha (crowd test set only)

To ensure a true zero-shot evaluation setting, the authors carefully filter out any datasets that were part of the encoders' original pre-training data. They map all dataset-specific labels into four core categories: neutral, happy, sad, and angry.

Key Insights and Ideas

  • High Classification Accuracy $\neq$ Good Similarity Metric: Even though models like emotion2vec excel at classifying discrete emotions, their raw latent spaces are poorly structured for zero-shot distance or similarity calculations.
  • Acoustic Mimicry Over Emotion: EMO-SIM is easily fooled by non-affective acoustic features. If a positive sample has a different speaker or text than the reference, the metric often ranks the negative sample (with matching speaker/text but wrong emotion) higher.
  • The Deep Layer Curse: A layer-wise analysis of emotion2vec reveals that deeper layers actually suppress affective features, causing alignment with human judgments to degrade as you go deeper into the network.
  • Inherited Biases: The authors hypothesize that emotion2vec directly inherits acoustic representations from its foundational model, data2vec. While supervised SER utilizes linear classification layers as filters to suppress these non-affective properties, zero-shot EMO-SIM relies exclusively on cosine similarity, leaving spatial distances dominated by speaker and text features.

Key Results

  • Categorical Vulnerability: Under controlled adversarial settings, performance frequently drops to or below random chance (50%). For instance, introducing a linguistic distractor caused base emotion2vec accuracy to plummet to a shocking 3.38% on the CREMA-D dataset.
Table 1: Categorical Emotion Evaluation. Triplet accuracy (%) under controlled adversarial settings. Values are Mean ± ±SD (dashes denote exclusion due to presence in pre-training corpora for zero-shot evaluation). Performance frequently falls to or below random chance (50 %), demonstrating a fundamental vulnerability to acoustic distractors.
  • Dimensional Blindness: The metric completely fails to track continuous emotional magnitude. Spearman's rank correlation ($\rho$) for valence and arousal hovered near zero across all datasets, and shift discriminability accuracy remained near random chance.
Table 2: Shift Discriminability (Accuracy). Performance comparison across four datasets. Results are reported in Mean ± ±SD (%) (dashes denote exclusion due to presence in pre-training corpora for zero-shot evaluation). Accuracies remain near random chance (50 %), exposing the metric’s inability to track continuous emotion attributes.
  • Human Misalignment: When tested against human preference annotations, fine-tuned emotion2vec+ variants only achieved between 52.25% and 65.00% accuracy, barely clearing the random baseline.
Figure 1: Human Perception Alignment. Accuracy (%) of various models evaluated on human perception annotations. ∗ indicates statistical significance (p < 0.05) compared to the 50% random baseline via binomial test.
  • Layer Trajectory: As shown in Figure 2, human perception alignment drops from 58.0% at layer L0 to a sub-random 45.0% by layer L7, proving that deeper layers actively degrade the metric's utility for human alignment.
Figure 2: Categorical Shift Tracking. The trajectory of accuracy across transformer layers for the base emotion2vec on MSP- Improv. Shaded regions denote standard deviation.

Limitations and Future Work

The primary limitation highlighted is that current zero-shot EMO-SIM applications treat embeddings as black boxes, leaving them vulnerable to representation crowding and non-affective acoustic features.

To establish valid automated metrics, the authors suggest exploring better representation calibration techniques. Specifically, introducing contrastive learning objectives during encoder training could help suppress acoustic variations (speaker identity and linguistic content) and encourage distinct, well-separated emotional clusters that truly reflect genuine affective prosody.

Summary updated

Abstract

Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring emotional prosody transfer. To quantify this, the field widely relies on emotion similarity between reference and generated samples. This approach computes cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations. We challenge this assumption through controlled adversarial tasks and human alignment tests. Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation. Representational limitations cause linguistic and speaker interference to overshadow emotional features, degrading discriminative ability. Consequently, the metric misaligns with human perception. This acoustic vulnerability reveals it rewards acoustic mimicry over genuine emotional synthesis.
BibTeX
@misc{tsai2026false,
  author = {Yun-Shao Tsai and Yi-Cheng Lin and Huang-Cheng Chou and Tzu-Wen Hsu and Yun-Man Hsu and Chun Wei Chen and Shrikanth Narayanan and Hung-yi Lee},
  title = {{The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation}},
  year = {2026},
  eprint = {2604.26347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.26347},
}
Figure