Analysing the Language of Neural Audio Codecs

Joonyong Park, Shinnosuke Takamichi, David M. Chan, Shunsuke Kando, Yuki Saito, Hiroshi SaruwatariarXiv 2025 · DOI · OpenReview

TL;DR

This paper investigates whether the discrete tokens generated by Neural Audio Codecs (NACs) exhibit linguistic properties similar to natural language. The authors found that NAC tokens, especially at the 3-gram level, follow statistical laws like Zipf's and Heaps' laws, and this linguistic regularity correlates with improved speech intelligibility and naturalness. This suggests that incorporating linguistic principles into NAC design could lead to more effective generative speech models.

What Problem Does This Paper Address?

Neural Audio Codecs (NACs) are powerful tools for compressing and reconstructing speech by converting audio into sequences of discrete tokens. These tokens serve as intermediate representations for various speech processing tasks, including speech synthesis and recognition. However, the linguistic and statistical properties of these token sequences have remained largely unexplored. Understanding whether these tokens mimic natural language structures is crucial for developing more effective NLP-inspired approaches for generative speech modeling. This paper aims to bridge this gap by systematically analyzing the linguistic characteristics of NAC tokens and their relationship to speech quality.

Methods

The researchers analyzed token sequences generated by 15 different NAC model configurations (detailed in Table I) using a standardized extraction and preprocessing pipeline. They focused on three key linguistic statistical laws:

  • Zipf's Law: This law describes the frequency distribution of words or n-grams, where a few elements are very frequent, and many are infrequent. The paper analyzed the rank-frequency distribution of NAC tokens to see if it followed a power-law relationship.
  • Heaps' Law: This law describes the relationship between the number of unique words (vocabulary size) and the total number of words in a text. The study examined how the vocabulary of NAC tokens grows with the sequence length.
  • Entropy and Redundancy: Shannon's entropy was used to measure the diversity and unpredictability of information in token sequences. Redundancy was quantified by comparing the actual average code length to the theoretical entropy.

To assess the impact of these linguistic properties on speech quality, the researchers used the Codec-SUPERB framework to decode NAC tokens back into speech. They then evaluated speech intelligibility using Automatic Speech Recognition (ASR) error rates (WER for English, CER for Chinese) and speech naturalness using the UTMOS score.

Datasets

The study utilized existing speech datasets for training and evaluation:

  • Training Data for NAC Models: A variety of datasets were used to train the different NAC models, including LibriSpeech, LibriTTS, VCTK, AISHELL, Common Voice, DNSC, Jamendo, AudioSet, FSD50K, and custom datasets.
  • Evaluation Data: 10 hours of single-speaker speech data for English (LJSpeech) and Mandarin (CSS10) were used for the analysis.
  • Ground Truth (GT) Transcripts: Manual transcriptions annotated to the corpora were used for comparison with ASR outputs.

The paper does not introduce new datasets but rather leverages existing ones to analyze the behavior of various NAC models.

Key Insights and Ideas

  • NAC Tokens Mimic Natural Language Statistics: The most significant finding is that NAC tokens, particularly at the 3-gram level, exhibit statistical patterns remarkably similar to natural language. This suggests that the tokenization process, even when focused on acoustic fidelity, inherently captures some linguistic structure.
  • 3-grams are the Sweet Spot: The 3-gram level appears to be a critical granularity for NAC tokens. At this level, the tokens show the strongest adherence to Zipf's law and Heaps' law, and this linguistic regularity is most strongly correlated with improved speech quality metrics.
  • Linguistic Regularity Boosts Performance: There's a clear correlation: NAC tokens with more "language-like" statistical properties (e.g., closer to Zipf's law exponent of 2, higher Heaps' law exponent β, lower redundancy) lead to better speech intelligibility (lower WER/CER) and perceived naturalness (higher UTMOS).
  • Trade-off Between Fidelity and Redundancy: While NACs are designed for high acoustic fidelity, the study suggests that prioritizing statistical redundancy (which aids compressibility and linguistic structure) can enhance downstream performance. This is evidenced by the observation that higher n-gram levels in NAC tokens show decreased compressibility compared to natural language, indicating a shift towards more unique acoustic patterns.
  • Dimension Size Matters: Models with larger dimension sizes generally showed better performance, aligning with the trend of more linguistically regular token sequences.

Key Results

  • Zipf's Law: 3-gram NAC token sequences showed the most consistent fit to Zipf's law, with their exponent (α) values often approaching the theoretical value of 2. Lower α values (closer to 2) correlated with lower error rates and higher UTMOS scores.
  • Heaps' Law: Higher values of the Heaps' law exponent β (closer to 1, indicating more linear vocabulary growth) were associated with improved speech recognition accuracy and perceived naturalness.
  • Entropy and Redundancy: Lower bit reduction rates (indicating higher redundancy and thus better compressibility) were observed for NAC tokens at the 2-gram and 3-gram levels, correlating with improved speech quality. However, this trend reversed at higher n-gram levels.
  • Performance Correlation: The study found consistent trends across various n-gram configurations, showing that token sequences exhibiting higher linguistic regularity and lower redundancy generally correlated with improved speech recognition accuracy and perceived naturalness. For instance, a decrease in the 3-gram Zipf's law exponent (α) led to decreased error rates and increased UTMOS scores.

Limitations and Future Work

  • Language Dependency: While the study analyzed English and Chinese, the authors suggest exploring NACs designed for multilingual domains to understand language dependency more deeply.
  • Real-World Scenarios: The study used single-speaker data. Future work could investigate multi-speaker audio in noisy environments.
  • Optimal Granularity: The paper highlights the importance of the 3-gram level but notes that 2-grams might be too short and 4-grams too sparse. Further research could explore optimal n-gram granularities for different tasks.
  • Design Implications: The findings offer insights for designing more effective generative speech models by balancing acoustic fidelity with linguistic structure. The authors hope these explorations will lead to better token representations that support high-quality speech generation and understanding.

Summary updated

Abstract

This study presents a comparative analysis of the statistical and linguistic properties of neural audio codecs (NACs). We investigate discrete speech tokens produced by various NAC models, examining their adherence to linguistic statistical laws such as Zipf's law and Heaps' law, as well as their entropy and redundancy. To assess how these token-level properties relate to semantic and acoustic preservation in synthesized speech, we evaluate intelligibility using error rates of automatic speech recognition, and quality using the UTMOS score. Our results reveal that NAC tokens, particularly 3-grams, exhibit language-like statistical patterns. Moreover, these properties, together with measures of information content, are found to correlate with improved performances in speech recognition and resynthesis tasks. These findings offer insights into the structure of NAC token sequences and inform the design of more effective generative speech models.
BibTeX
@misc{park2025analysing,
  author = {Joonyong Park and Shinnosuke Takamichi and David M. Chan and Shunsuke Kando and Yuki Saito and Hiroshi Saruwatari},
  title = {{Analysing the Language of Neural Audio Codecs}},
  year = {2025},
  eprint = {2509.01390},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01390},
}
Figure