Abstract
Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results using only a single GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. As a result, our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost.
Introduction
Neural audio codecs like EnCodec and DAC have become essential "tokenizers" for modern speech AI, enabling Large Language Models to process raw waveforms. However, a persistent gap exists between acoustic fidelity (how natural the audio sounds) and intelligibility (how well the words are preserved). Most existing codecs prioritize the former, often leading to "mushy" speech where semantic nuances are lost during quantization.
Traditional solutions utilize Semantic Encoder Distillation (SED), which aligns encoder tokens with self-supervised models like WavLM. While helpful, SED only constrains the encoder; it provides no guarantee that the decoder will faithfully reconstruct those semantic features in the final waveform. Furthermore, high-quality codecs often rely on significant "lookahead" context, introducing latencies that hinder real-time, interactive applications.
Methodology: The SSRR Philosophy
JHCodec addresses these challenges through a "Reconstruct! Don't Encode" philosophy. The core innovation is the Self-Supervised Representation Reconstruction (SSRR) loss.
As illustrated in the architecture above, JHCodec utilizes a causal Transformer with Residual Vector Quantization (RVQ). The SSRR mechanism works as follows:
- Semantic Teacher: A distilled, causal version of W2V-BERT 2.0 (SW2V) acts as the teacher.
- Reconstruction Loss: Instead of merely distilling the encoder, the reconstructed audio from the decoder is passed back through the SW2V teacher. The model is penalized if the semantic features of the output deviate from the original input.
- Zero-Lookahead Design: By using a fully causal architecture with FlashAttention, JHCodec operates with zero lookahead, processing audio frames as they arrive without waiting for future context.
Key Results and Efficiency
SSRR acts as a powerful regularizer, shifting the optimization objective from low-level pixels to linguistically meaningful representations. This shift allows JHCodec to converge remarkably fast, achieving competitive results on a single GPU—a significant reduction in compute compared to the multi-node clusters required by many baselines.
As shown in the comparative analysis, JHCodec-M-8 achieves a record-low latency of 26.8ms, the lowest among all evaluated streaming models. Despite its zero-lookahead constraint, it maintains high intelligibility with a Word Error Rate (WER) of 3.19% on LibriSpeech test-clean, outperforming models like Mimi-32 which utilize higher training budgets. Additionally, the model shows strong cross-lingual generalization, maintaining high performance on non-English datasets despite being trained primarily on English corpora.
Conclusion
By shifting the focus from encoder distillation to decoder reconstruction accountability, JHCodec demonstrates that high-level semantic preservation and low-latency streaming are not mutually exclusive. The SSRR loss provides a new blueprint for building efficient, high-intelligibility codecs suitable for the next generation of real-time speech AI applications.
Summary updated