Abstract
Neural audio codecs (NACs), which use neural networks to generate compact audio representations, have garnered interest for their applicability to many downstream tasks—especially quantized codecs due to their compatibility with large language models. However, unlike text, speech conveys not only linguistic content but also rich paralinguistic features. Encoding these elements in an entangled fashion may be suboptimal, as it limits flexibility. For instance, voice conversion (VC) aims to convert speaker characteristics while preserving the original linguistic content, which requires a disentangled representation. Inspired by VC methods utilizing $k$-means quantization with self-supervised features to disentangle phonetic information, we develop a discrete NAC capable of structured disentanglement. Experimental evaluations show that our approach achieves reconstruction performance on par with conventional NACs that do not explicitly perform disentanglement, while also matching the effectiveness of conventional VC techniques.
Introduction
In the evolving landscape of AI audio, Neural Audio Codecs (NACs) serve as critical bridges, transforming complex sound waves into compact digital tokens suitable for Large Language Models (LLMs). However, a significant hurdle remains: most current codecs are "entangled." A single token often encapsulates a mixture of speaker identity, prosody, and linguistic content.
This entanglement limits the flexibility of audio manipulation. For tasks like Voice Conversion (VC), where the goal is to alter a speaker's voice while maintaining the original emotion and words, an entangled representation makes it nearly impossible to isolate and swap specific elements. Furthermore, many existing disentanglement methods rely on supervised labels (like phonemes) or external tools (like F0 estimation), which hinders scalability. This research proposes a discrete NAC designed for structured disentanglement using self-supervised learning (SSL) features to separate speech into three distinct streams: content, prosody, and speaker identity.
Methodology
The proposed architecture utilizes a frozen, pre-trained WavLM-Large model as its backbone, specifically extracting features from the 6th layer to balance phonetic and speaker information.
The Disentangling Pipeline
- Phonetic Stream (Content): WavLM features are mapped to discrete "content" tokens (C) using $k$-means clustering.
- The Residual Trick: By subtracting the quantized content from the original features, the model isolates a residual (R) containing paralinguistic information (speaker identity and prosody).
- Prosody vs. Speaker: The system applies Instance Normalization (IN) to the residual. The time-invariant components (mean and standard deviation) represent the Speaker (S), while the time-variant normalized fluctuations represent the Prosody (P).
- Quantization: All three streams are converted into discrete tokens using Vector Quantization (RVQ and GRVQ), ensuring compatibility with LLMs.
- Decoder: A GAN-based decoder recombines these streams using Feature-wise Linear Modulation (FiLM) to reconstruct the original waveform.
Experimental Setup
The model was trained on the LibriSpeech-train-clean-100 dataset at 16 kHz. Evaluation was conducted on the LibriSpeech-test-clean subset. Centroids for $k$-means clustering were pre-calculated using a separate large-scale speech dataset to ensure robust phonetic categorization without manual labels.
Results and Discussion
Reconstruction and Voice Conversion
The disentangled codec achieved sound quality (measured by ViSQOL and STFT distance) nearly identical to state-of-the-art non-disentangled codecs like DAC and SNAC. In one-shot voice conversion tests—changing a voice using only a single 3-second reference clip—the model matched or outperformed specialized VC methods. It maintained a low Word Error Rate (WER) of ~2.6%, indicating that linguistic content remains highly intelligible even after identity substitution.
Interpretability and Prosody Control
A key finding is the model's ability to capture prosody without explicit pitch tracking. Visualizations show that the prosody tokens naturally align with frequency deviations (F0), allowing for seamless prosody substitution.
Discrete Speaker Identity
While most VC models use continuous embeddings, this work demonstrates that speaker identity can be effectively discretized. However, a bitrate threshold exists: excessive compression of the speaker representation can lead to a loss of the unique vocal "fingerprint." Increasing the number of groups in the Group-Residual Vector Quantization (GRVQ) was found to significantly improve Speaker Verification (SV) scores.
Conclusion
This research successfully demonstrates a discrete neural speech codec capable of high-quality reconstruction and structured disentanglement without human supervision. By explicitly modeling the residual signal after phonetic quantization, the codec captures subtle paralinguistic features essential for realistic speech synthesis. Future work will focus on integrating this codec into full-scale Speech LMs to enable more natural, consistent, and emotionally aware dialogue systems.
Summary updated