Abstract
Integrating speech into Large Language Models (LLMs) requires efficient discretization of continuous audio into finite tokens. Traditional methods often rely on fixed-size codebooks and Euclidean distance metrics, which can lead to information loss and audible distortion. This paper introduces SECodec, a novel framework that leverages Structural Entropy (SE) to rethink speech discretization. By modeling speech features as a graph and minimizing 2D structural entropy, SECodec automatically determines optimal codebook sizes and preserves complex structural information through a "node game" quantization strategy. Experimental results demonstrate that SECodec significantly outperforms existing codecs like EnCodec in reconstruction quality and zero-shot text-to-speech tasks.
1. Introduction
The success of Large Language Models (LLMs) in natural language processing has spurred interest in extending their capabilities to the speech domain. A critical bottleneck is the discretization process, where continuous audio signals are converted into discrete tokens. Current state-of-the-art codecs, such as EnCodec, typically utilize $k$-means clustering and Euclidean distance for quantization. However, these approaches suffer from two primary limitations: the need for manually tuned, fixed codebook sizes and a failure to capture the non-Euclidean structural complexity of speech data, resulting in quantization noise.
2. Methodology
SECodec addresses these limitations by treating speech discretization as an information-theoretic optimization problem on graphs.
2.1 Hierarchical and Disentangled Codebook Construction
Instead of pre-defining the number of clusters, SECodec constructs a speech feature graph $G = (V, E, W)$. It employs a hierarchical algorithm to minimize 2D structural entropy, which naturally reveals the underlying cluster structure of the data. To ensure that the resulting codebook vectors are distinct, a variational contrastive log-ratio upper bound (vCLUB) is integrated to disentangle features across different clusters. This process results in a codebook that adaptively fits the data distribution, achieving high fidelity with fewer tokens (e.g., 468 vs. EnCodec's 1024).
2.2 Node Game-based Quantization
Rather than assigning frames to the nearest codebook vector in Euclidean space, SECodec introduces a "node game" heuristic. When a new speech frame is processed, it is treated as a node being added to the existing graph. The frame is assigned to the cluster that minimizes the total increase in the graph's structural entropy. This approach preserves the topological relationships between speech features, leading to higher fidelity reconstruction compared to simple spatial distance metrics.
3. Experimental Evaluation
The authors evaluated SECodec and its corresponding language model, SESLM, across several benchmarks including LibriSpeech and VCTK.
3.1 Speech Reconstruction Performance
On the LibriSpeech dataset, SECodec achieved a Word Error Rate (WER) of 4.63%, a substantial improvement over EnCodec's 8.87% at similar bitrates. Subjective evaluations using MUSHRA scores also favored SECodec (90.79) over EnCodec (71.32), indicating superior perceptual quality.
3.2 Zero-Shot Text-to-Speech (TTS)
The SESLM model, trained on 44,000 hours of speech, was tested on the VCTK multi-speaker benchmark. In zero-shot TTS tasks, SESLM achieved a WER of 4.97%, outperforming established models like VALL-E (7.09%) and USLM (5.79%).
4. Conclusion
SECodec demonstrates that structural entropy is a powerful tool for speech discretization. By moving beyond Euclidean metrics and fixed codebook sizes, the framework achieves state-of-the-art performance in both audio reconstruction and generative speech tasks. While graph-based methods introduce computational complexity, the hierarchical approach mitigates these costs, paving the way for more robust speech-language integration.
Summary updated