TL;DR
While tokenization (like BPE) is a cornerstone of modern NLP, it has been largely ignored for discrete audio representations. This paper demonstrates that applying Byte Pair Encoding to phonemes and Discrete Acoustic Units (DAUs) significantly improves model accuracy, training efficiency, and inference speed across tasks like text-to-speech and unsupervised speech generation.
What Problem Does This Paper Address?
In the world of Generative AI for speech, researchers often convert continuous audio into Discrete Acoustic Units (DAUs)—essentially "audio words" that a Transformer can process. However, these units are typically extracted at high bitrates (e.g., 50Hz), leading to sequences that are incredibly long and highly redundant (e.g., a single vowel might be represented by the same unit repeated ten times: 52, 52, 52...).
These long sequences pose two major problems:
- Computational Cost: Transformers have a quadratic complexity relative to sequence length. Long sequences make training and inference slow and memory-intensive.
- Error Accumulation: In autoregressive models, a small error in one step can snowball over a long sequence. Furthermore, the high redundancy in raw DAUs can lead to "exposure bias," where a model learns the lazy heuristic of simply repeating the previous token rather than learning the underlying structure of speech.
Methods
The authors propose applying the Byte Pair Encoding (BPE) algorithm—the same technique used by GPT-4 to handle text—to phonetic and acoustic units.
BPE works by iteratively merging the most frequent adjacent pairs of units into a single new "token." For example, if the units for "th" and "at" frequently appear together, BPE creates a new "that" token. This process:
- Reduces sequence length (compression).
- Increases vocabulary size (from ~1000 base units to up to 16,000+ tokens).
- Balances the distribution of tokens in the training data.
The researchers tested this approach on three architectures:
- G2P (Grapheme-to-Phoneme): A T5-small model predicting phonetic transcriptions from text.
- G2DAU (Grapheme-to-DAU): A T5-base model predicting acoustic units from text.
- SpeechLM: A LLaMA-style decoder-only model for unsupervised speech generation.
Datasets
The study utilized several large-scale datasets to ensure robust results:
- Common Crawl: Used for the G2P experiments (3M paragraphs).
- LibriSpeech (English subset): Used for G2DAU tasks (10M transcribed utterances).
- People’s Speech, VoxPopuli, and Common Voice: Combined to provide a massive corpus for training the Speech Language Models (SpeechLM).
Key Insights and Ideas
- The "Edge Case" Logic: The authors provide a theoretical insight into why shorter sequences are better even if the vocabulary is harder to learn. If a model has a 0.1% error rate per token, the probability of getting a 900-token sequence perfectly right is much lower than getting a 300-token sequence right. BPE shifts the burden from "sequence length" to "vocabulary complexity," which Transformers handle much better.
- Mitigating Exposure Bias: Raw DAUs often contain long runs of identical tokens. During training, the model learns to "copy" the previous token. At inference, this often leads to "nonsense" loops. BPE breaks these runs into unique tokens, forcing the model to learn actual transitions.
- Data Balancing: Raw acoustic units often follow a Zipfian distribution (a few units are everywhere, most are rare). The authors use Normalized Entropy to show that BPE creates a more "balanced" vocabulary, making the cross-entropy loss function more effective during training.
Key Results
The improvements across the board were substantial:
- G2DAU (Text-to-Audio Units): This saw the most dramatic gain. The Character Error Rate (CER) dropped from a failing 87.96% (without BPE) to 3.19% (with BPE).
- Speedups: Models using BPE were significantly faster. The G2DAU task saw a 2.55x speedup in training and inference.
- G2P (Text-to-Phonemes): Even for phonemes, which are already somewhat compressed, BPE reduced the Word Error Rate (WER) from 0.83% to 0.40%.
- Speech Generation: In "LLM-as-a-judge" evaluations, the BPE-based SpeechLM was preferred 69.52% of the time over the non-BPE baseline.
Limitations and Future Work
- Diminishing Returns on Vocab Size: The authors noted that you can't increase the vocabulary indefinitely. At a vocabulary size of 16,384 for G2DAU, the speedup actually began to decrease because the computational cost of the final "softmax" layer (the matrix multiplication for the large vocabulary) started to outweigh the gains from shorter sequences.
- Integration with TTS: While the paper proves BPE helps predict units, these units still need to be converted back to waveforms. Future work is needed to integrate these variable-rate BPE tokens directly into end-to-end Text-to-Speech (TTS) architectures that typically expect constant-frame-rate inputs.
- Evaluation Metrics: Evaluating unsupervised speech is notoriously difficult. While the authors used STT (Speech-to-Text) and LLM-judges, these are proxies for human perception.
Summary updated