Abstract
Universal audio codecs typically learn entangled representations across audio types, whereas specialized codecs offering decoupled representations are often limited to speech. Real-world audio, however, frequently contains mixed speech and background sounds, requiring selective access to these components for various downstream tasks. We introduce DeCodec, a novel neural codec designed as a universal disentangled representation learner. DeCodec decouples audio into orthogonal subspaces for speech and background sound (BGS), further decomposing speech into semantic and paralinguistic components. This hierarchical disentanglement enables flexible feature selection, positioning DeCodec as a universal front-end for applications including speech recognition, voice conversion, and controllable text-to-speech.
Introduction
In the real world, audio is rarely "clean"; it is usually a mixture of speech and background sounds. Traditionally, tasks like Automatic Speech Recognition (ASR) on noisy audio required cascaded pipelines: first running a Speech Separation (SS) model to clean the audio, then passing the result to an ASR model. This approach suffers from error propagation, signal distortion from the cleaning process, and significant computational redundancy.
DeCodec addresses these issues by performing disentanglement at the representation level. By mathematically decoupling these components, it provides a "menu" of features that downstream tasks can selectively utilize, eliminating the need for separate, specialized pre-processors.
Methodology
DeCodec is built upon a neural codec framework but introduces three key architectural innovations to achieve hierarchical disentanglement:
- Subspace Orthogonal Projection (SOP): This module projects initial audio embeddings into two mathematically orthogonal subspaces—one for speech and one for background sound. This ensures that information from one domain does not leak into the other.
- Representation Swap Training (RST): To enforce the separation, the model uses a "swap" procedure during training. It combines the speech representation from one audio sample with the background representation from another, forcing the decoder to reconstruct a hybrid signal. This teaches the model exactly which features belong in each subspace.
- Hierarchical Speech Decomposition: Within the speech subspace, the model further separates semantic information (the words) from paralinguistic information (tone and timbre). This is achieved via Semantic Guidance (SG), where the first layer of the speech quantizer is supervised by Hubert-L9 features.
Experimental Results
DeCodec was trained on a massive 700-hour dataset of speech and background noise. Evaluation on the LibriTTS and DNS Challenge sets demonstrates its superiority as both a codec and a feature extractor.
Reconstruction and Robustness
As shown in the comparative analysis below, DeCodec achieves a Signal-to-Distortion Ratio (SDR) of 7.61, significantly outperforming established codecs like EnCodec (6.86) and DAC (0.60). Beyond compression, its disentangled features provide exceptional robustness for ASR. On noisy speech, DeCodec-based ASR achieved a Word Error Rate (WER) of 23.6%, while the previous state-of-the-art disentangled codec, SpeechTokenizer, collapsed to 55.1%.
Downstream Applications
The hierarchical disentanglement enables several novel capabilities through simple feature selection:
- One-Shot Voice Conversion: By swapping paralinguistic tokens while keeping semantic and background tokens intact, DeCodec performs high-quality voice conversion even in noisy environments.
- Controllable TTS: Users can explicitly control whether generated speech should include the background environment of a prompt or provide a perfectly clean output.
- Efficient Speech Enhancement: Instead of complex filtering, speech can be enhanced by simply ignoring the background quantizers during reconstruction, preserving the integrity of the speech signal.
Conclusion
DeCodec moves beyond simple audio compression to act as a universal front-end for audio processing. By enforcing mathematical orthogonality between subspaces, it prevents background noise from corrupting semantic tokens, making the representation inherently robust. While future work will focus on narrowing the performance gap for causal, real-time applications, DeCodec currently offers a powerful, unified solution for multi-task audio representation learning.
Summary updated