Abstract
Neural audio codecs have significantly advanced audio compression by efficiently converting continuous audio signals into discrete tokens. These codecs preserve high-quality sound and enable sophisticated sound generation through generative models trained on these tokens. However, existing neural codec models are typically trained on large, undifferentiated audio datasets, neglecting the essential discrepancies between sound domains like speech, music, and environmental sound effects. This oversight complicates data modeling and poses additional challenges to the controllability of sound generation. To tackle these issues, we introduce the Source-Disentangled Neural Audio Codec (SD-Codec), a novel approach that combines audio coding and source separation. By jointly learning audio resynthesis and separation, SD-Codec explicitly assigns audio signals from different domains to distinct codebooks, sets of discrete representations. Experimental results indicate that SD-Codec not only maintains competitive resynthesis quality but also, supported by the separation results, demonstrates successful disentanglement of different sources in the latent space, thereby enhancing interpretability in audio codec and providing potential finer control over the audio generation process.
Introduction
Modern Neural Audio Codecs (NACs), such as EnCodec and Descript Audio Codec (DAC), have revolutionized audio storage and generative modeling by compressing complex waveforms into compact discrete tokens. Despite their efficiency, these models generally treat audio as a monolithic signal. Whether the input is a human voice, a musical instrument, or environmental noise, it is mapped into a single, undifferentiated latent space.
This "one-size-fits-all" approach presents two significant challenges. First, it limits controllability; once audio is encoded, it is difficult to isolate or manipulate specific components, such as removing background noise from a vocal track. Second, it introduces modeling inefficiencies. Different audio domains—speech, music, and sound effects (SFX)—possess distinct acoustic properties. Forcing these diverse signals into a shared representation ignores their fundamental discrepancies, which can hinder downstream tasks like controllable audio generation. To address these issues, we propose SD-Codec, a framework that integrates source separation directly into the quantization process.
The SD-Codec Architecture
SD-Codec builds upon the foundation of the DAC architecture but introduces a specialized quantization strategy designed for disentanglement.
Domain-Specific Residual Vector Quantizers (RVQs)
Instead of a single RVQ, SD-Codec employs three parallel branches: $RVQ_{speech}$, $RVQ_{music}$, and $RVQ_{sfx}$. During training, the model is encouraged to route information from specific domains into their corresponding branches. This explicit assignment creates a structured latent space where the identity of the sound is tied to the codebook it occupies.
Latent Additivity and Shared Codebooks
A key innovation is the principle of Latent Additivity. The model assumes that the latent representation of an audio mixture is the sum of the quantized features from each domain. This allows the decoder to reconstruct either the full mixture or individual tracks by simply selecting which quantized features to sum.
Furthermore, the researchers explored a "Shared Codebook" variant. In this setup, deeper layers of the RVQ are shared across all domains. This design is based on the insight that while shallow layers identify the broad domain (e.g., "this is speech"), deeper layers capture fine-grained acoustic details common to all sounds, such as high-frequency transients.
Training and Datasets
To ensure robust performance, SD-Codec was trained on a massive scale, utilizing approximately 6,000 hours of diverse audio data. The training set included:
- DNS 5: Providing clean speech and various noise profiles.
- MTG-Jamendo: A source for high-quality music recordings.
- MUSAN & WHAM!: Contributing additional speech, music, and environmental sound effects.
The model was trained using a combination of multi-scale mel-spectrogram reconstruction loss and adversarial loss to ensure high-fidelity resynthesis. Evaluation was conducted on the Divide and Remaster (DnR) dataset, serving as a "zero-shot" test to measure the model's ability to generalize to unseen, complex audio environments.
Results and Key Insights
SD-Codec demonstrates that source separation can be effectively achieved within the quantizer itself. By forcing specific features into specific codebooks, the model naturally learns to "sort" audio components during encoding.
Superior Reconstruction and Separation
Experimental results show that SD-Codec outperforms the baseline DAC in audio resynthesis. For audio mixtures, SD-Codec achieved an SI-SDR of ~7.0 dB, significantly higher than DAC's ~4.5 dB. Remarkably, despite being primarily a codec, SD-Codec's source separation performance is competitive with TDANet, a state-of-the-art model dedicated solely to separation tasks.
As shown in the figure above, each RVQ branch specializes in its target domain. For instance, the $RVQ_{speech}$ branch achieves high SI-SDR on speech signals while maintaining very low values for music and SFX, confirming successful disentanglement in the latent space.
Conclusion and Future Work
SD-Codec represents a significant step toward interpretable and controllable neural audio coding. By disentangling sources at the quantization level, it provides a clear mapping between discrete tokens and audio domains.
Future research will focus on reducing the bitrate overhead, as the current model requires separate streams for each domain. Additionally, expanding the framework to include more granular categories (e.g., individual musical instruments) and integrating these disentangled tokens into Large Language Models (LLMs) for prompt-based audio editing are promising directions for future development.
Summary updated