TL;DR
DeCodec is a neural audio codec that explicitly factorizes mixed audio representations into separate, orthogonal subspaces for speech and background sound, while further decomposing speech into semantic and paralinguistic tokens. By decoupling these audio streams directly within the latent space, DeCodec acts as a universal front-end that performs speech enhancement, voice conversion, noise-robust speech recognition, and controllable speech synthesis without requiring cascaded separation models.
What Problem Does This Paper Address?
Real-world audio rarely occurs in isolation—it blends speech with complex background noise or environmental sound. Standard universal audio codecs (like EnCodec or DAC) preserve high acoustic fidelity by compressing signals as single entangled streams, making it impossible to separate speech from background ambiance in the token space. Conversely, specialized speech tokenizers (like SpeechTokenizer) decompose speech into semantic and acoustic components, but collapse when exposed to real-world noise.
Traditionally, practitioners tackle this with a cascaded pipeline: a speech separation front-end filters audio before passing it to downstream tasks. However, cascaded systems suffer from error accumulation, introduce processing artifacts, and require redundant, task-specific feature extraction pipelines.
Methods
DeCodec extends a DAC-style convolutional encoder-decoder framework with three key mechanisms to achieve hierarchical disentanglement:
Input Audio Mixture (y = s + n)
│
[ Encoder ]
│ Latent Y
[ Subspace Orthogonal Projection ]
┌──────┴──────┐
Speech S Background N
│ │
[ SRVQ ] [ NRVQ ]
┌────┴────┐ │
Semantic Acoustic │
(Zc) (Zr) │
└────┬────┘ │
Speech Zs Background Zn
└──────┬──────┘
[ Decoder ]
│
Reconstructed Audio
- Subspace Orthogonal Projection (SOP): Inspired by the human auditory cortex, continuous encoder embeddings are projected into two orthogonal subspaces via learned projection operators ($P_S$ and $P_N$). An orthogonality loss ($\mathcal{L}_\perp$) forces the dot product of the speech and noise representations to zero.
- Representation Swap Training (RST): To ensure each subspace captures the correct audio stream, the training swaps representations across two independent mixtures ($y_{11} = s_1 + n_1$ and $y_{22} = s_2 + n_2$). The decoder reconstructs a hybrid audio signal ($\hat{y}_{12} \approx s_1 + n_2$) using the speech tokens of the first and background tokens of the second. The authors mathematically prove via the mean value theorem that this swap loss forces mutual independence between speech and background components.
- Semantic Guidance (SG): The first layer of the speech quantizer ($SRVQ_1$) is trained via cosine similarity distillation against HuBERT layer-9 representations of clean speech. This isolates clean linguistic content into $Z_c$, leaving the remaining speech quantizers ($SRVQ_{2:8}$) to encode paralinguistic features (speaker timbre, acoustic details) as $Z_r$.
Datasets
- Training: ~700 hours of multi-speaker speech from Aishell-3, LibriTTS (clean subsets), VCTK, WSJ0, and WSJ1, mixed with environmental audio from ESC-50 and DNS-Noise at SNRs ranging from -5 dB to 40 dB.
- Evaluation: LibriSpeech
test-clean, a synthetic noisy test set (LibriSpeech test-clean mixed with DNS-Noise at -5 to 20 dB SNR), and the official 2nd DNS Challenge blind test set for speech enhancement. Downstream ASR and zero-shot TTS evaluations were conducted using LibriSpeech models.
Key Insights and Ideas
- Decoupling in latent space beats time-domain separation: Recombining latent representations directly—such as substituting background tokens with silence—avoids the phase distortion and artifacts typical of time-domain speech separation filters.
- Noise robustness flows downstream: Downstream models trained purely on clean speech (like standard ASR or VALL-E) maintain strong performance on noisy audio when fed DeCodec's clean semantic or speech tokens, eliminating the need to fine-tune downstream models on noisy data.
- Feature-level controllability: Because background sound is isolated into distinct codebooks, downstream generators can either strip background noise completely or explicitly preserve it to maintain ambient environmental realism.
Key Results
- Audio Reconstruction: Achieved speech reconstruction SDR of 7.61 dB on clean audio and 5.21 dB on noisy audio, outperforming DAC, EnCodec, and SpeechTokenizer in preserving speech signals.
- Speech Enhancement (SE): On the DNS Challenge blind test set, non-causal DeCodec achieved top background noise suppression with a BAK score of 4.13 (without reverb) and 3.99 (real recordings), surpassing dedicated enhancement baselines such as StoRM and SELM.
- One-Shot Voice Conversion: On noisy inputs, DeCodec achieved an intelligibility WER of 50.46% with 0.83 speaker similarity, outperforming a cascaded StoRM + SpeechTokenizer pipeline (52.73% WER) and dramatically improving over standalone SpeechTokenizer (74.18% WER).
- Downstream Robustness:
- ASR: Feeding DeCodec's speech representations to an ASR model on noisy inputs yielded a 23.6% WER, outperforming the cascaded StoRM-SpeechTokenizer baseline (32.1%).
- Zero-shot TTS: Human subjective evaluations showed DeCodec achieved a background removal score (BRMOS) of 4.74/5.0, while retaining the option to synthesize speech with the original background sound at a naturalness MOS of 4.09/5.0.
Limitations and Future Work
- Semantic Guidance via Distillation: DeCodec currently relies on an external pre-trained model (HuBERT) for semantic guidance rather than an intrinsic, self-supervised subspace decomposition for speech components.
- Codebook Efficiency: The model uses parallel 8-stage RVQs (16 codebooks in total: 8 for speech, 8 for noise), resulting in higher token overhead compared to recent single-layer speech codecs.
- Voice Conversion Distortions: Voice conversion on unaligned, variable-length noisy samples still yields relatively high error rates (~50% WER) when voiced/unvoiced phonetic boundaries misalign between input and reference audio.
Summary updated