DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners

Xiaoxue Luo, Jinwei Huang, Runyan Yang, Yingying Gao, Junlan Feng, Chao Deng, Shilei Zhang

Abstract

Universal audio codecs typically learn entangled representations across audio types, whereas specialized codecs offering decoupled representations are often limited to speech. Real-world audio, however, frequently contains mixed speech and background sounds, requiring selective access to these components for various downstream tasks. We introduce DeCodec, a novel neural codec designed as a universal disentangled representation learner. DeCodec decouples audio into orthogonal subspaces for speech and background sound (BGS), further decomposing speech into semantic and paralinguistic components. This hierarchical disentanglement enables flexible feature selection, positioning DeCodec as a universal front-end for applications including speech recognition, voice conversion, and controllable text-to-speech.

Introduction

In the real world, audio is rarely "clean"; it is usually a mixture of speech and background sounds. Traditionally, tasks like Automatic Speech Recognition (ASR) on noisy audio required cascaded pipelines: first running a Speech Separation (SS) model to clean the audio, then passing the result to an ASR model. This approach suffers from error propagation, signal distortion from the cleaning process, and significant computational redundancy.

DeCodec addresses these issues by performing disentanglement at the representation level. By mathematically decoupling these components, it provides a "menu" of features that downstream tasks can selectively utilize, eliminating the need for separate, specialized pre-processors.

Methodology

DeCodec is built upon a neural codec framework but introduces three key architectural innovations to achieve hierarchical disentanglement:

  1. Subspace Orthogonal Projection (SOP): This module projects initial audio embeddings into two mathematically orthogonal subspaces—one for speech and one for background sound. This ensures that information from one domain does not leak into the other.
  2. Representation Swap Training (RST): To enforce the separation, the model uses a "swap" procedure during training. It combines the speech representation from one audio sample with the background representation from another, forcing the decoder to reconstruct a hybrid signal. This teaches the model exactly which features belong in each subspace.
  3. Hierarchical Speech Decomposition: Within the speech subspace, the model further separates semantic information (the words) from paralinguistic information (tone and timbre). This is achieved via Semantic Guidance (SG), where the first layer of the speech quantizer is supervised by Hubert-L9 features.
The DeCodec architecture featuring Subspace Orthogonal Projection (SOP), Representation Swap Training (RST), and Semantic Guidance (SG).

Experimental Results

DeCodec was trained on a massive 700-hour dataset of speech and background noise. Evaluation on the LibriTTS and DNS Challenge sets demonstrates its superiority as both a codec and a feature extractor.

Reconstruction and Robustness

As shown in the comparative analysis below, DeCodec achieves a Signal-to-Distortion Ratio (SDR) of 7.61, significantly outperforming established codecs like EnCodec (6.86) and DAC (0.60). Beyond compression, its disentangled features provide exceptional robustness for ASR. On noisy speech, DeCodec-based ASR achieved a Word Error Rate (WER) of 23.6%, while the previous state-of-the-art disentangled codec, SpeechTokenizer, collapsed to 55.1%.

Comparison of DeCodec against baseline codecs across clean and noisy audio conditions.

Downstream Applications

The hierarchical disentanglement enables several novel capabilities through simple feature selection:

  • One-Shot Voice Conversion: By swapping paralinguistic tokens while keeping semantic and background tokens intact, DeCodec performs high-quality voice conversion even in noisy environments.
  • Controllable TTS: Users can explicitly control whether generated speech should include the background environment of a prompt or provide a perfectly clean output.
  • Efficient Speech Enhancement: Instead of complex filtering, speech can be enhanced by simply ignoring the background quantizers during reconstruction, preserving the integrity of the speech signal.

Conclusion

DeCodec moves beyond simple audio compression to act as a universal front-end for audio processing. By enforcing mathematical orthogonality between subspaces, it prevents background noise from corrupting semantic tokens, making the representation inherently robust. While future work will focus on narrowing the performance gap for causal, real-time applications, DeCodec currently offers a powerful, unified solution for multi-task audio representation learning.

Summary updated

Abstract

Universal audio codecs learn entangled representations across audio types, whereas some specific codecs offer decoupled representations but are limited to speech. Real-world audio, however, often contains mixed speech and background sounds, and downstream tasks require selective access to these components. Therefore, we rethink the audio codec as a universal disentangled representation learner to enable controllable feature selection across different audio tasks. To this end, we introduce DeCodec, a novel neural codec that learns to decouple audio representations into orthogonal subspaces dedicated to speech and background sound, and within speech, representations are further decomposed into semantic and paralinguistic components. This hierarchical disentanglement allows flexible feature selection, making DeCodec a universal front-end for multiple audio applications. Technically, built upon a codec framework, DeCodec incorporates two key innovations: a subspace orthogonal projection module that factorizes the input into two decoupled orthogonal subspaces, and a representation swap training procedure that ensures these two subspaces are correlate to the speech and background sound, respectively. These allows parallel RVQs to quantize speech and background sound components independently. Furthermore, we employ semantic guidance to the speech RVQ to achieve semantic and paralinguistic decomposition. Experimental results show that DeCodec maintains advanced signal reconstruction while enabling new capabilities: superior speech enhancement and effective one-shot voice conversion on noisy speech via representation recombination, improved ASR robustness through clean semantic features, and controllable background sound preservation/suppression in TTS. Demo Page: https://luo404.github.io/DeCodecV2/
BibTeX
@misc{luo2025decodec,
  author = {Xiaoxue Luo and Jinwei Huang and Runyan Yang and Yingying Gao and Junlan Feng and Chao Deng and Shilei Zhang},
  title = {{DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners}},
  year = {2025},
  eprint = {2509.09201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.09201},
}
Figure