Probing the phon* representations of the Mimi codebooks

Artem Saloev, Nicolas Ballier, Erin PacquetetarXiv 2026 · arXiv

TL;DR

This paper investigates how the Mimi neural audio codec represents speech sounds, finding that its first codebook functions as a context-sensitive "allophonic" encoder rather than a simple phoneme inventory. By moving beyond standard discrimination tasks, the authors demonstrate that while Mimi tokens are strongly tied to specific phonemes, phonetic information is actually distributed across the entire quantization hierarchy.

What Problem Does This Paper Address?

Neural audio codecs (NACs) like Mimi are the "tokenizers" of the spoken language world, turning continuous audio into discrete units that large language models can process. However, we don't fully understand what these units represent. Do they correspond to phonemes (the smallest units of sound that distinguish meaning), or something else?

A major challenge in probing Mimi is its 80ms tokenization window. Most phonetic evaluation tools (like ABX discrimination tasks) assume a token represents a single, short segment. Because 80ms is often longer than a single phoneme, a Mimi token usually captures multiple phonetic events and their transitions. This paper seeks a more robust, distributional way to "peek inside" these codebooks to understand their linguistic organization.

Figure 1: A schematic representation of the Mimi encoding-decoding pipeline

Methods

The authors move away from pairwise comparisons and instead adopt a distributional probing framework. Their approach includes:

  1. Forced Alignment: Using the Montreal Forced Aligner (MFA) to map Mimi tokens to the phonemes they most likely represent based on temporal overlap.
  2. Information Theory Metrics: They use Phone-Normalized Mutual Information (PNMI) to measure the global correspondence between tokens and phonemes, and Entropy ($H$) to measure the "purity" of these associations.
  3. Permutation Testing: To see if token usage is stable, they compared token distributions for the same phoneme across different sentences using Jensen-Shannon Divergence (JSD).
  4. Controlled Interventions: They modified the codebook sequences (e.g., replacing tokens with a constant value) and measured the impact on intelligibility using Word Error Rate (WER) from a pretrained speech recognition model.

Datasets

The study utilizes the TIMIT corpus, a classic benchmark for phonetic research. To ensure strict comparability across speakers, the authors focused on the "SA" sentences (SA1 and SA2), which are read by all 630 speakers in the dataset. They specifically filtered for "modal-length" utterances to ensure that tokens at the same position across different speakers represented roughly the same acoustic window.

Figure 2: Distribution of Mimi token frames (80 ms each) in TIMIT SA1/SA2 across speakers. The modal counts (42 for SA1, 35 for SA2) are indicated by the tallest bar in each histogram.

Key Insights and Ideas

  • The Asymmetry of Representation: There is a "many-to-one" mapping. While a specific token is almost always associated with one phoneme (high "purity"), a single phoneme is realized by a "family" of different tokens.
  • Allophonic, Not Just Phonemic: The authors found that token distributions for the same phoneme change significantly depending on the sentence context. This suggests Mimi isn't just identifying a "category" (like the letter /t/); it's capturing the specific way that /t/ sounds in a particular environment (coarticulation).
  • Phonetic Info is Everywhere: Contrary to the design assumption that the first codebook (CB0) holds the "semantic/phonetic" info while others hold "residual" acoustic detail, PNMI analysis showed that phonetic information is distributed throughout all eight codebooks.
  • Robustness to Local Noise: Replacing a single token in the first codebook didn't hurt speech recognition performance. The model's intelligibility is distributed across frames, making it resilient to isolated errors.

Key Results

  • High Correspondence: Mimi's first codebook achieved a PNMI of 0.84, significantly outperforming other codecs like EnCodec (0.28) and even SpeechTokenizer (0.71) in phonetic alignment.
  • Token "Purity": The mean dominant phoneme probability for tokens was 0.89, meaning if you see a specific token, you can be very certain which phoneme it belongs to.
  • Context Sensitivity: The mean JSD across sentences was 0.61 (compared to a null baseline of 0.14), proving that "token families" are highly sensitive to sentence-level context.
  • Impact of Disruption: Globally replacing all tokens in the first codebook (CB0) increased the Word Error Rate (WER) from 0.493 to 0.701, confirming that CB0 carries essential linguistic structure.
Figure 3: Token variability in Mimi’s first codebook across TIMIT SA1 and SA2

Limitations and Future Work

The primary limitation is the narrow scope of the data: the study relied on only two specific sentences from the TIMIT corpus. While this allowed for high-precision cross-speaker comparisons, it may not capture the full diversity of English.

The authors suggest that future work should test these findings on more diverse datasets, such as the Edinburgh corpus of English accents. Additionally, they note that the 80ms window might be capturing speaker-specific acoustic details (like timbre or pitch) alongside phonetic content, a possibility that warrants further investigation.

Summary updated

Abstract

Neural audio codecs are increasingly used to encode speech for integration with large language models. This paper investigates the phonetic representations in the first codebook (CB0) of the Mimi codec. ABX style evaluation is poorly suited to Mimi fixed 80 ms tokenization window, which often spans multiple phonemes; we instead use forced alignment, entropy analysis, and Phone-Normalized Mutual Information (PNMI). Using TIMIT speech (58/55 speakers, SA1/SA2), individual tokens are strongly associated with specific phonemes (PNMI = 0.84, above random), yet phonemes are realized by multiple competing tokens rather than a one-to-one mapping.
BibTeX
@misc{saloev2026probing,
  author = {Artem Saloev and Nicolas Ballier and Erin Pacquetet},
  title = {{Probing the phon* representations of the Mimi codebooks}},
  year = {2026},
  eprint = {2610.35432},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35432},
}
Figure