EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations

Wenze Ren, Yi-Cheng Lin, Huang-Cheng Chou, Haibin Wu, Yi-Chiao Wu, Chi-Chun Lee, Hung-yi Lee, Yu Tsao

Abstract

The neural codec model reduces speech data transmission delay and serves as the foundational tokenizer for speech language models (speech LMs). Preserving emotional information in codecs is crucial for effective communication and context understanding. However, there is a lack of studies on emotion loss in existing codecs. This paper evaluates neural and legacy codecs using subjective and objective methods on emotion datasets like IEMOCAP. Our study identifies which codecs best preserve emotional information under various bitrate scenarios. We found that training codec models with both English and Chinese data had limited success in retaining emotional information in Chinese. Additionally, resynthesizing speech through these codecs degrades the performance of speech emotion recognition (SER), particularly for emotions like sadness, depression, fear, and disgust. Human listening tests confirmed these findings. This work guides future speech technology developments to ensure new codecs maintain the integrity of emotional information in speech.

Introduction

As Speech Language Models (Speech LMs) evolve, audio codecs are increasingly utilized as "tokenizers," converting continuous sound into discrete units for AI processing. Traditionally, codecs are optimized for signal-level fidelity or linguistic intelligibility, often neglecting paralinguistic features such as emotion. If a codec discards subtle emotional cues—like the tremors of fear or the resonance of sadness—to save bits, AI assistants may fail to perceive a user's emotional state, leading to inappropriate interactions. This study addresses this "emotion gap" by providing a comprehensive benchmark for emotion preservation in modern speech technologies.

Methodology

The researchers established the Emo-Codec evaluation pipeline, which combines objective and subjective assessments to measure emotional integrity across 17 different codecs (14 neural and 3 legacy) and 15 emotion recognition models.

The EMO-Codec evaluation pipeline, illustrating the parallel objective (machine-based) and subjective (human-based) assessment paths.
  1. Objective Evaluation: 15 Self-Supervised Learning (SSL) models (e.g., Wav2Vec 2.0, HuBERT) were trained on high-quality original audio and then tested on audio resynthesized by various codecs. Emotion loss was quantified by the drop in Macro-F1 scores.
  2. Subjective Evaluation: Human listening tests were conducted via the Prolific platform. Annotators rated Mean Opinion Score (MOS) for quality and identified emotions in resynthesized clips to validate machine findings.
  3. Cross-Lingual Analysis: The study examined how codecs trained on English data performed on Chinese emotional datasets (BIIC-PODCAST, NNIME) to identify language-specific nuances.

Key Insights and Results

The benchmarking process revealed several critical findings regarding how different architectures and bitrates affect emotional retention.

The Superiority of Neural Codecs

At low bitrates (e.g., 6 kbps), neural codecs such as SpeechTokenizer and AcademiCodec significantly outperform legacy standards like MP3 or Opus in preserving emotional information. Legacy codecs only begin to achieve comparable performance at much higher bitrates. Among neural models, the Descript Audio Codec (DAC) consistently emerged as the top performer. Its success is attributed to the use of a "snake" activation function and balanced data sampling during training.

Macro F1 scores across different bitrates for various codecs. The red dashed line represents the original audio baseline. DAC consistently maintains high performance near the baseline.

Vulnerability of Negative Emotions

Emotions characterized by negative valence—specifically sadness, depression, fear, and disgust—showed the most significant performance degradation after compression. These emotions rely on subtle acoustic cues that are frequently smoothed over by compression algorithms, particularly at lower bitrates.

Language and Bitrate Scaling

While increasing bitrate generally improves emotion retention, the rate of improvement is much faster for neural codecs. Interestingly, training codecs on a mixture of English and Chinese data yielded only marginal gains for Chinese emotion preservation, suggesting that emotional expression contains deep phonetic and tonal characteristics that current general-purpose codecs fail to capture fully across language families.

Conclusion and Future Work

This research highlights the necessity of "emotion-aware" codecs. While DAC sets a high bar for current technology, several challenges remain:

  • Language Diversity: Expanding testing to a broader range of languages with unique emotional prosody.
  • Demographic Representation: Addressing potential biases in human perception by diversifying annotator backgrounds.
  • Contextual Emotion: Moving beyond isolated clips to evaluate codecs in the context of interactive, multi-turn dialogues.

Future developments should prioritize the preservation of paralinguistic cues to ensure that the next generation of empathetic speech technologies remains emotionally resonant.

Summary updated

Abstract

The neural codec model reduces speech data transmission delay and serves as the foundational tokenizer for speech language models (speech LMs). Preserving emotional information in codecs is crucial for effective communication and context understanding. However, there is a lack of studies on emotion loss in existing codecs. This paper evaluates neural and legacy codecs using subjective and objective methods on emotion datasets like IEMOCAP. Our study identifies which codecs best preserve emotional information under various bitrate scenarios. We found that training codec models with both English and Chinese data had limited success in retaining emotional information in Chinese. Additionally, resynthesizing speech through these codecs degrades the performance of speech emotion recognition (SER), particularly for emotions like sadness, depression, fear, and disgust. Human listening tests confirmed these findings. This work guides future speech technology developments to ensure new codecs maintain the integrity of emotional information in speech.
BibTeX
@inproceedings{ren2024emo,
  author = {Wenze Ren and Yi-Cheng Lin and Huang-Cheng Chou and Haibin Wu and Yi-Chiao Wu and Chi-Chun Lee and Hung-yi Lee and Yu Tsao},
  title = {{EMO-Codec: An In-Depth Look at Emotion Preservation capacity of Legacy and Neural Codec Models With Subjective and Objective Evaluations}},
  booktitle = {APSIPA},
  year = {2024},
  doi = {10.1109/apsipaasc63619.2025.10849259},
}
Figure