SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization

Wenxi Chen, Xinsheng Wang, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Hanlin Wen, Shunshun Yin, Ming Tao, Xie Chen

Abstract

Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised embeddings. When used as a tokenizer for LLM-based text-to-speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable speech generation.

Introduction

In the evolving landscape of Speech Language Models (SLMs), speech tokenizers serve as the bridge between continuous audio waves and discrete tokens. However, a persistent "Semantic-Acoustic Conflict" exists: tokens optimized for linguistic understanding (semantic tokens) often fail at high-fidelity reconstruction, while those optimized for audio quality (acoustic tokens) lack the semantic structure required for efficient LLM learning. SAC (Semantic-Acoustic Codec) is designed to resolve this tension by explicitly disentangling these two streams.

Methodology

SAC's core innovation is its dual-stream architecture, which separates linguistic content from acoustic nuances such as timbre, prosody, and emotion.

The SAC architecture disentangles speech into parallel semantic and acoustic encoding paths, followed by dual-stream quantization.

Dual-Stream Design

  • Semantic Stream: This path utilizes a pre-trained, frozen semantic tokenizer. By keeping this component frozen, the model ensures that linguistic "meaning" remains uncorrupted by the reconstruction objective.
  • Acoustic Stream: This learnable stream acts as a residual encoder, capturing everything the semantic stream misses—specifically speaker identity and environmental characteristics.
  • Dual-Stream Quantization: Both streams are quantized into discrete tokens ($S_q$ and $A_q$) before being fused for decoding.

Auxiliary Supervision

To sharpen the disentanglement, SAC employs two specialized loss functions: $\mathcal{L}_{sem}$, which ensures the fused representation can predict original semantic features, and $\mathcal{L}_{spk}$, which forces the preservation of unique speaker characteristics using a pre-trained speaker encoder.

Experimental Setup

The model was trained on a massive 20,000-hour bilingual (Chinese and English) corpus, including datasets such as Emilia, WenetSpeech4TTS, and Libriheavy. Evaluation was conducted using the ARCH benchmark for semantic richness and Seed-TTS-eval for downstream generative performance.

Key Results

SAC consistently outperforms existing neural codecs across various bitrates, demonstrating a superior balance of naturalness and intelligibility.

Performance comparison: SAC (stars) achieves higher naturalness (UTMOS) and lower Word Error Rate (WER) than competing codecs across different bitrates.

Reconstruction and Intelligibility

At a bitrate of only 525 BPS, SAC achieves a UTMOS of 4.27 and a WER of 2.53%, significantly outperforming models like Encodec and SpeechTokenizer that operate at much higher bitrates. In noisy conditions, the SAC decoder acts as a generative enhancer, often producing UTMOS scores that exceed the ground truth.

Semantic Representation

On the ARCH benchmark, SAC's representations outperformed all other codec models and rivaled specialized self-supervised models like wav2vec 2.0. This semantic richness allows SAC to function as a highly effective tokenizer for single-stage autoregressive (AR) TTS models, which outperform more complex multi-stage systems.

Conclusion and Future Work

By forcing the acoustic stream to be a residual of the semantic stream, SAC achieves a clean separation that benefits both speech understanding and generation. While currently optimized for speech, future research will explore extending this dual-stream framework to universal audio encoders capable of handling music and general environmental sounds.

Summary updated

Abstract

Speech codecs that convert continuous speech signals into discrete tokens have become essential for speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised embeddings. When used as a tokenizer for LLM-based text-to-speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable speech generation.
BibTeX
@inproceedings{chen2026sac,
  author = {Wenxi Chen and Xinsheng Wang and Ruiqi Yan and Yushen Chen and Zhikang Niu and Ziyang Ma and Xiquan Li and Yuzhe Liang and Hanlin Wen and Shunshun Yin and Ming Tao and Xie Chen},
  title = {{SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.138},
}
Figure