TL;DR
This paper investigates how the Mimi neural audio codec represents speech sounds, finding that its first codebook functions as a context-sensitive "allophonic" encoder rather than a simple phoneme inventory. By moving beyond standard discrimination tasks, the authors demonstrate that while Mimi tokens are strongly tied to specific phonemes, phonetic information is actually distributed across the entire quantization hierarchy.
What Problem Does This Paper Address?
Neural audio codecs (NACs) like Mimi are the "tokenizers" of the spoken language world, turning continuous audio into discrete units that large language models can process. However, we don't fully understand what these units represent. Do they correspond to phonemes (the smallest units of sound that distinguish meaning), or something else?
A major challenge in probing Mimi is its 80ms tokenization window. Most phonetic evaluation tools (like ABX discrimination tasks) assume a token represents a single, short segment. Because 80ms is often longer than a single phoneme, a Mimi token usually captures multiple phonetic events and their transitions. This paper seeks a more robust, distributional way to "peek inside" these codebooks to understand their linguistic organization.
Methods
The authors move away from pairwise comparisons and instead adopt a distributional probing framework. Their approach includes:
- Forced Alignment: Using the Montreal Forced Aligner (MFA) to map Mimi tokens to the phonemes they most likely represent based on temporal overlap.
- Information Theory Metrics: They use Phone-Normalized Mutual Information (PNMI) to measure the global correspondence between tokens and phonemes, and Entropy ($H$) to measure the "purity" of these associations.
- Permutation Testing: To see if token usage is stable, they compared token distributions for the same phoneme across different sentences using Jensen-Shannon Divergence (JSD).
- Controlled Interventions: They modified the codebook sequences (e.g., replacing tokens with a constant value) and measured the impact on intelligibility using Word Error Rate (WER) from a pretrained speech recognition model.
Datasets
The study utilizes the TIMIT corpus, a classic benchmark for phonetic research. To ensure strict comparability across speakers, the authors focused on the "SA" sentences (SA1 and SA2), which are read by all 630 speakers in the dataset. They specifically filtered for "modal-length" utterances to ensure that tokens at the same position across different speakers represented roughly the same acoustic window.
Key Insights and Ideas
- The Asymmetry of Representation: There is a "many-to-one" mapping. While a specific token is almost always associated with one phoneme (high "purity"), a single phoneme is realized by a "family" of different tokens.
- Allophonic, Not Just Phonemic: The authors found that token distributions for the same phoneme change significantly depending on the sentence context. This suggests Mimi isn't just identifying a "category" (like the letter /t/); it's capturing the specific way that /t/ sounds in a particular environment (coarticulation).
- Phonetic Info is Everywhere: Contrary to the design assumption that the first codebook (CB0) holds the "semantic/phonetic" info while others hold "residual" acoustic detail, PNMI analysis showed that phonetic information is distributed throughout all eight codebooks.
- Robustness to Local Noise: Replacing a single token in the first codebook didn't hurt speech recognition performance. The model's intelligibility is distributed across frames, making it resilient to isolated errors.
Key Results
- High Correspondence: Mimi's first codebook achieved a PNMI of 0.84, significantly outperforming other codecs like EnCodec (0.28) and even SpeechTokenizer (0.71) in phonetic alignment.
- Token "Purity": The mean dominant phoneme probability for tokens was 0.89, meaning if you see a specific token, you can be very certain which phoneme it belongs to.
- Context Sensitivity: The mean JSD across sentences was 0.61 (compared to a null baseline of 0.14), proving that "token families" are highly sensitive to sentence-level context.
- Impact of Disruption: Globally replacing all tokens in the first codebook (CB0) increased the Word Error Rate (WER) from 0.493 to 0.701, confirming that CB0 carries essential linguistic structure.
Limitations and Future Work
The primary limitation is the narrow scope of the data: the study relied on only two specific sentences from the TIMIT corpus. While this allowed for high-precision cross-speaker comparisons, it may not capture the full diversity of English.
The authors suggest that future work should test these findings on more diverse datasets, such as the Edinburgh corpus of English accents. Additionally, they note that the 80ms window might be capturing speaker-specific acoustic details (like timbre or pitch) alongside phonetic content, a possibility that warrants further investigation.
Summary updated