Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec

Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang, Shrikanth Narayanan, Thomas Thebaud, Laureano Moro-Velázquez, Jesús Villalba, Najim Dehak

Abstract

Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results using only a single GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. As a result, our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost.

Introduction

Neural audio codecs like EnCodec and DAC have become essential "tokenizers" for modern speech AI, enabling Large Language Models to process raw waveforms. However, a persistent gap exists between acoustic fidelity (how natural the audio sounds) and intelligibility (how well the words are preserved). Most existing codecs prioritize the former, often leading to "mushy" speech where semantic nuances are lost during quantization.

Traditional solutions utilize Semantic Encoder Distillation (SED), which aligns encoder tokens with self-supervised models like WavLM. While helpful, SED only constrains the encoder; it provides no guarantee that the decoder will faithfully reconstruct those semantic features in the final waveform. Furthermore, high-quality codecs often rely on significant "lookahead" context, introducing latencies that hinder real-time, interactive applications.

Methodology: The SSRR Philosophy

JHCodec addresses these challenges through a "Reconstruct! Don't Encode" philosophy. The core innovation is the Self-Supervised Representation Reconstruction (SSRR) loss.

The JHCodec architecture highlighting the SSRR loss path, which feeds reconstructed audio back through a semantic teacher (SW2V) to ensure content preservation.

As illustrated in the architecture above, JHCodec utilizes a causal Transformer with Residual Vector Quantization (RVQ). The SSRR mechanism works as follows:

  1. Semantic Teacher: A distilled, causal version of W2V-BERT 2.0 (SW2V) acts as the teacher.
  2. Reconstruction Loss: Instead of merely distilling the encoder, the reconstructed audio from the decoder is passed back through the SW2V teacher. The model is penalized if the semantic features of the output deviate from the original input.
  3. Zero-Lookahead Design: By using a fully causal architecture with FlashAttention, JHCodec operates with zero lookahead, processing audio frames as they arrive without waiting for future context.

Key Results and Efficiency

SSRR acts as a powerful regularizer, shifting the optimization objective from low-level pixels to linguistically meaningful representations. This shift allows JHCodec to converge remarkably fast, achieving competitive results on a single GPU—a significant reduction in compute compared to the multi-node clusters required by many baselines.

Performance comparison of JHCodec-M-8 against state-of-the-art streaming and non-streaming codecs, demonstrating superior latency and training efficiency.

As shown in the comparative analysis, JHCodec-M-8 achieves a record-low latency of 26.8ms, the lowest among all evaluated streaming models. Despite its zero-lookahead constraint, it maintains high intelligibility with a Word Error Rate (WER) of 3.19% on LibriSpeech test-clean, outperforming models like Mimi-32 which utilize higher training budgets. Additionally, the model shows strong cross-lingual generalization, maintaining high performance on non-English datasets despite being trained primarily on English corpora.

Conclusion

By shifting the focus from encoder distillation to decoder reconstruction accountability, JHCodec demonstrates that high-level semantic preservation and low-latency streaming are not mutually exclusive. The SSRR loss provides a new blueprint for building efficient, high-intelligibility codecs suitable for the next generation of real-time speech AI applications.

Summary updated

Abstract

Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results using only a single GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. As a result, our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost. We open-source the full implementation, training pipeline, and demo on Github https://github.com/jhcodec843/jhcodec.
BibTeX
@misc{lee2026reconstruct,
  author = {Junhyeok Lee and Xiluo He and Jihwan Lee and Helin Wang and Shrikanth Narayanan and Thomas Thebaud and Laureano Moro-Velázquez and Jesús Villalba and Najim Dehak},
  title = {{Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec}},
  year = {2026},
  eprint = {2603.05887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05887},
}
Figure