EMO-Codec: An In-Depth Look at Emotion Preservation Capacity of Legacy and Neural Codec Models with Subjective and Objective Evaluations

Wenze Ren, Yi-Cheng Lin, Huang-Cheng Chou, Haibin Wu, Yi-Chiao Wu, Chi-Chun Lee, Hung-Yi Lee, Hsin-Min Wang, Yu Tsao

TL;DR

EMO-Codec is a comprehensive benchmark evaluating how well modern neural and legacy audio codecs preserve emotional information in speech. By testing 17 different codecs across six datasets and 15 emotion recognition models, the researchers found that while higher bitrates generally help, the Descript Audio Codec (DAC) stands out as the superior choice for maintaining emotional integrity, especially compared to legacy standards like MP3 or Opus at low bitrates.

What Problem Does This Paper Address?

As we move toward more sophisticated Speech Language Models (Speech LMs), we are increasingly using audio codecs not just for compression, but as "tokenizers"—turning continuous sound into discrete units that AI can process. However, most codecs are designed to optimize for signal-level fidelity (how close the waveform looks to the original) or intelligibility (can you understand the words?), often ignoring paralinguistic features like emotion.

If a codec discards the subtle tremors of fear or the low resonance of sadness to save bits, an AI assistant might fail to perceive a user's emotional state, leading to robotic or inappropriate interactions. This paper addresses the "emotion gap" in codec evaluation, providing a much-needed roadmap for developers building the next generation of empathetic speech technologies.

Methods

The authors established a rigorous evaluation pipeline called Emo-Codec:

  1. Objective Evaluation: They trained 15 different Self-Supervised Learning (SSL) models (like Wav2Vec 2.0 and HuBERT) for Speech Emotion Recognition (SER) using original, high-quality audio. They then ran these models on audio that had been compressed and resynthesized by 14 neural codecs (e.g., Encodec, DAC, Funcodec) and 3 legacy codecs (MP3, Opus, AAC). The "emotion loss" was measured by the drop in the Macro-F1 score.
  2. Subjective Evaluation: They conducted human listening tests via the Prolific platform. Annotators rated the speech quality (MOS) and attempted to identify the emotions in the resynthesized clips to see if human perception aligned with the machine models.
  3. Cross-Lingual Analysis: They specifically looked at how codecs trained on English data performed on Chinese emotional datasets to test for language-specific nuances.

Datasets

The study utilized six diverse datasets to ensure the results weren't a fluke of one specific recording environment:

  • English: IEMOCAP (acted/scripted), CREAM-D (acted), IMPROV (improvised), and MSP-PODCAST (real-world).
  • Chinese: BIIC-PODCAST (real-world) and NNIME (acted/improvised).

These datasets cover a range of scenarios, from professional actors in studios to spontaneous real-world conversations.

Key Insights and Ideas

  • The "Snake" Advantage: The Descript Audio Codec (DAC) consistently outperformed all others. The authors attribute this to two specific design choices: the use of a "snake" activation function and balanced data sampling during training.
  • Negative Emotions are Fragile: Emotions with "negative valence"—specifically sadness, depression, fear, and disgust—suffered the most significant performance drops after compression. These emotions often rely on subtle acoustic cues that are easily smoothed over by compression algorithms.
  • Neural vs. Legacy: At low bitrates (e.g., 6 kbps), neural codecs like SpeechTokenizer and AcademiCodec are far better at preserving emotion than legacy codecs like MP3 or Opus. Legacy codecs only start to catch up as the bitrate increases significantly.
  • Language Matters: Interestingly, training a codec on a mix of English and Chinese data provided only marginal improvements for Chinese emotion preservation. This suggests that emotional expression has deep phonetic and tonal characteristics that current general-purpose codecs aren't fully capturing across different language families.

Key Results

  • DAC Dominance: In the IEMOCAP dataset, DAC at 24kbps (D2) maintained an F1 score very close to the original audio baseline (the red dashed line in their charts), while Encodec at the same bitrate (E5) showed a noticeable dip.
  • Bitrate Scaling: There is a clear, consistent trend: as bitrate increases, emotion retention improves. However, the rate of improvement varies; neural codecs reach a high level of "emotional fidelity" much faster than legacy ones.
  • Human-Machine Alignment: Subjective tests confirmed the objective findings. Humans also found it harder to identify emotions in audio processed by lower-performing codecs, and DAC 24k was rated highest for both speech quality and emotional retention.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Language Diversity: While the study included English and Chinese, there are thousands of other languages with unique emotional prosody that remain untested.
  • Demographic Bias: The human annotators were from a specific demographic (US-based), which might not reflect how emotions are perceived globally.
  • Contextual Interaction: The study focused on isolated clips. In the real world, emotion is often understood through the context of a conversation. Future work should explore how codecs perform in interactive, multi-turn dialogue systems.
  • Optimization: The ultimate goal is to develop "emotion-aware" codecs that prioritize the preservation of paralinguistic cues even at ultra-low bitrates.

Summary updated

Abstract

The neural codec model reduces speech data transmission delay and serves as the foundational tokenizer for speech language models (speech LMs). Preserving emotional information in codecs is crucial for effective communication and context understanding. However, there is a lack of studies on emotion loss in existing codecs. This paper evaluates neural and legacy codecs using subjective and objective methods on emotion datasets like IEMOCAP. Our study identifies which codecs best preserve emotional information under various bitrate scenarios. We found that training codec models with both English and Chinese data had limited success in retaining emotional information in Chinese. Additionally, resynthesizing speech through these codecs degrades the performance of speech emotion recognition (SER), particularly for emotions like sadness, depression, fear, and disgust. Human listening tests confirmed these findings. This work guides future speech technology developments to ensure new codecs maintain the integrity of emotional information in speech.
BibTeX
@inproceedings{ren2024emo,
  author = {Wenze Ren and Yi-Cheng Lin and Huang-Cheng Chou and Haibin Wu and Yi-Chiao Wu and Chi-Chun Lee and Hung-Yi Lee and Hsin-Min Wang and Yu Tsao},
  title = {{EMO-Codec: An In-Depth Look at Emotion Preservation Capacity of Legacy and Neural Codec Models with Subjective and Objective Evaluations}},
  booktitle = {APSIPA},
  year = {2024},
  doi = {10.1109/apsipaasc63619.2025.10849259},
}

Loading related content...

Figure