Speech Separation using Neural Audio Codecs with Embedding Loss

Jia Qi Yip, Kwok Chin Yuen, Bin Ma, Engsiong Chng

Abstract

Neural audio codecs have revolutionized audio processing by enabling speech tasks to be performed on highly compressed representations. Recent work has shown that speech separation can be achieved within these compressed domains, offering faster training and reduced inference costs. However, current approaches still rely on waveform-based loss functions, necessitating unnecessary decoding steps during training. We propose a novel embedding loss for neural audio codec-based speech separation that operates directly on compressed audio representations, eliminating the need for decoding during training. To validate our approach, we conduct comprehensive evaluations using both objective metrics and perceptual assessment techniques, including intrusive and non-intrusive methods. Our results demonstrate that embedding loss can be used to train codec-based speech separation models with a 2x improvement in training speed and computational cost while achieving better DNSMOS and STOI performance on the WSJ0-2mix dataset across 3 different pre-trained codecs.

Introduction: The Decoding Tax in Speech Separation

Speech separation—the "cocktail party problem" of isolating individual voices from a noisy mixture—is notoriously resource-hungry. Traditional models struggle with high-resolution audio waveforms, which require massive memory and compute. Recent "codec-based" models address this by using neural audio codecs (like EnCodec or DAC) to compress audio by approximately 100x.

However, even these models typically use waveform-based loss functions (like SI-SDR) during training. This forces the model to run a "decoder" step every single iteration just to calculate the error, wasting time and energy on a component that isn't needed for the separation logic itself. Furthermore, waveform-matching metrics often penalize models for tiny phase shifts imperceptible to the human ear, leading to a mismatch between mathematical optimization and human perception.

Methods: Training in the Latent Space

We introduce Codecformer-EL (Embedding Loss), which eliminates the need for a decoder during the training loop. Instead of comparing the separated audio to the ground truth in the waveform domain, the model compares them directly in the latent embedding space of a pre-trained neural audio codec.

Comparison of the proposed Embedding Loss training paradigm (top) versus traditional Waveform Loss (bottom).

The architecture utilizes a series of Transformer blocks and a "mask gating" mechanism. It takes the compressed embedding of a mixture and predicts masks to isolate the embeddings of individual speakers. The core innovation is applying Mean Squared Error (MSE) directly to these latent representations. Because the encoder of the codec is frozen, the model learns to produce embeddings that the codec’s decoder can later transform into clean speech. To optimize performance, the model's internal activation functions (e.g., ELU or Snake) are matched to the specific codec being used.

Key Results: Efficiency and Perceptual Quality

Codecformer-EL demonstrates significant advantages in both computational efficiency and the quality of separated speech.

Computational Efficiency

By removing the "Decoding Tax," Codecformer-EL achieves a 2.5x speedup in training time per epoch compared to its waveform-loss predecessor. It also requires 1.9x fewer GMACs (Giga Multiply-Accumulate operations), making it nearly 100x more efficient than traditional models like Sepformer.

Computational efficiency comparison of Codecformer-EL against baseline models.

Perceptual Performance

While the model scores lower on "objective" metrics like SI-SDR (which demands sample-by-sample alignment), it consistently outperforms waveform-loss models on perceptual metrics like DNSMOS (quality) and STOI (intelligibility). For example, using the SoundStream codec, embedding loss improved the DNSMOS score from 1.73 to 1.80 and STOI from 0.74 to 0.83. This suggests that training in embedding space encourages the model to focus on perceptually relevant features rather than invisible waveform alignments.

Conclusion

Codecformer-EL proves that speech separation is more efficient and perceptually effective when performed on the structured "map" of a codec's latent space rather than raw waveforms. By drastically reducing training costs while maintaining or improving quality, this approach enables more scalable development of speech separation technologies. Future research will explore fine-tuning codec encoders specifically for separation and the impact of varying embedding resolutions.

Summary updated

BibTeX
@inproceedings{yip2024speech,
  author = {Jia Qi Yip and Kwok Chin Yuen and Bin Ma and Engsiong Chng},
  title = {{Speech Separation using Neural Audio Codecs with Embedding Loss}},
  booktitle = {APSIPA},
  year = {2024},
  url = {https://openreview.net/forum?id=POnnGUQD0E},
}
Figure