SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models

Linqin Wang, Yaping Liu, Zhengtao Yu, Shengxiang Gao, Cunli Mao, Yuxin Huang, Wenjun Wang, Ling Dong

Abstract

Integrating speech into Large Language Models (LLMs) requires efficient discretization of continuous audio into finite tokens. Traditional methods often rely on fixed-size codebooks and Euclidean distance metrics, which can lead to information loss and audible distortion. This paper introduces SECodec, a novel framework that leverages Structural Entropy (SE) to rethink speech discretization. By modeling speech features as a graph and minimizing 2D structural entropy, SECodec automatically determines optimal codebook sizes and preserves complex structural information through a "node game" quantization strategy. Experimental results demonstrate that SECodec significantly outperforms existing codecs like EnCodec in reconstruction quality and zero-shot text-to-speech tasks.

1. Introduction

The success of Large Language Models (LLMs) in natural language processing has spurred interest in extending their capabilities to the speech domain. A critical bottleneck is the discretization process, where continuous audio signals are converted into discrete tokens. Current state-of-the-art codecs, such as EnCodec, typically utilize $k$-means clustering and Euclidean distance for quantization. However, these approaches suffer from two primary limitations: the need for manually tuned, fixed codebook sizes and a failure to capture the non-Euclidean structural complexity of speech data, resulting in quantization noise.

2. Methodology

SECodec addresses these limitations by treating speech discretization as an information-theoretic optimization problem on graphs.

Overview of the SECodec framework, illustrating graph construction, hierarchical codebook construction via SE minimization, and the node game-based quantization process.

2.1 Hierarchical and Disentangled Codebook Construction

Instead of pre-defining the number of clusters, SECodec constructs a speech feature graph $G = (V, E, W)$. It employs a hierarchical algorithm to minimize 2D structural entropy, which naturally reveals the underlying cluster structure of the data. To ensure that the resulting codebook vectors are distinct, a variational contrastive log-ratio upper bound (vCLUB) is integrated to disentangle features across different clusters. This process results in a codebook that adaptively fits the data distribution, achieving high fidelity with fewer tokens (e.g., 468 vs. EnCodec's 1024).

2.2 Node Game-based Quantization

Rather than assigning frames to the nearest codebook vector in Euclidean space, SECodec introduces a "node game" heuristic. When a new speech frame is processed, it is treated as a node being added to the existing graph. The frame is assigned to the cluster that minimizes the total increase in the graph's structural entropy. This approach preserves the topological relationships between speech features, leading to higher fidelity reconstruction compared to simple spatial distance metrics.

3. Experimental Evaluation

The authors evaluated SECodec and its corresponding language model, SESLM, across several benchmarks including LibriSpeech and VCTK.

3.1 Speech Reconstruction Performance

On the LibriSpeech dataset, SECodec achieved a Word Error Rate (WER) of 4.63%, a substantial improvement over EnCodec's 8.87% at similar bitrates. Subjective evaluations using MUSHRA scores also favored SECodec (90.79) over EnCodec (71.32), indicating superior perceptual quality.

3.2 Zero-Shot Text-to-Speech (TTS)

The SESLM model, trained on 44,000 hours of speech, was tested on the VCTK multi-speaker benchmark. In zero-shot TTS tasks, SESLM achieved a WER of 4.97%, outperforming established models like VALL-E (7.09%) and USLM (5.79%).

Spectrogram and F0 contour comparisons between reference speech and outputs from SECodec/SESLM versus baselines. Green checkmarks indicate successful preservation of acoustic features, while red 'X's highlight distortions in baseline models.

4. Conclusion

SECodec demonstrates that structural entropy is a powerful tool for speech discretization. By moving beyond Euclidean metrics and fixed codebook sizes, the framework achieves state-of-the-art performance in both audio reconstruction and generative speech tasks. While graph-based methods introduce computational complexity, the hierarchical approach mitigates these costs, paving the way for more robust speech-language integration.

Summary updated

Abstract

With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the speech features nodes within the graph and extracting the corresponding codebook by hierarchically and disentangledly minimizing 2D SE. Then, to address the issue of audio distortion, we propose a new quantization method. This method still adheres to the 2D SE minimization principle, adaptively selecting the most suitable token corresponding to the cluster for each incoming original speech node. Furthermore, we develop a Structural Entropy-based Speech Language Model (SESLM) that leverages SECodec. Experimental results demonstrate that SECodec performs comparably to EnCodec in speech reconstruction, and SESLM surpasses VALL-E in zero-shot text-to-speech tasks.
BibTeX
@inproceedings{wang2025secodec,
  author = {Linqin Wang and Yaping Liu and Zhengtao Yu and Shengxiang Gao and Cunli Mao and Yuxin Huang and Wenjun Wang and Ling Dong},
  title = {{SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34726},
}
Figure