Zipf’s and Heaps’ Laws for Tokens and LLM-generated Texts

Nikolay MikhaylovskiyEMNLP 2025

TL;DR

This paper investigates whether Large Language Models (LLMs) generate text that adheres to Zipf's and Heaps' laws, fundamental statistical properties of natural language. They discover that these laws only hold within a narrow, model-dependent temperature range, suggesting a "critical state" for LLM generation. This finding has implications for understanding and controlling LLM output quality.

What Problem Does This Paper Address?

Natural language, whether spoken or written, exhibits predictable statistical patterns. Zipf's law describes the relationship between a word's rank and its frequency (more frequent words are ranked higher), while Heaps' law describes how vocabulary size grows with the size of a text. These laws are well-established for human-generated text. However, with the rise of LLMs, a crucial question emerges: do these models, which generate text algorithmically, also produce outputs that follow these natural language statistical regularities? This paper aims to answer this by empirically testing the applicability of Zipf's and Heaps' laws to text generated by various LLMs across different generation parameters, particularly the sampling temperature.

Methods

The authors leverage existing statistical frameworks for analyzing text. They verify Zipf's law by calculating cumulative histograms of token frequencies and fitting them with power laws in log-log coordinates. For Heaps' law, they analyze the growth of unique tokens (vocabulary size) as a function of text length. A novel metric, $R$, is introduced to quantify the ratio of tokens appearing for the first time in the first versus the second half of a text, serving as a proxy for Heaps' law adherence. They systematically vary the sampling temperature ($t$) for LLM generation and analyze how these statistical properties change. To quantify deviations from power laws, they use the Mean Average Percentage Error (MAPE).

Datasets

The study utilizes a dataset of six long literary and philosophical works in five languages from Mikhaylovskiy and Churilov (2023). This dataset was tokenized using both a simple space/punctuation splitting method and the Mistral AI v3 tokenizer (Byte-Pair Encoding).

Key Insights and Ideas

  • LLMs don't naturally follow Zipf's/Heaps' laws: Unlike human text, LLM-generated text generally deviates from these laws, especially at higher temperatures.
  • Temperature is key: A narrow, model-specific temperature range is critical for LLM outputs to exhibit Zipfian and Heapsian properties. This suggests a "phase transition" in LLM generation.
  • Critical State: The paper posits that LLM generation transitions from an "ordered" state (low temperature, degenerate text) through a "critical state" (adhering to power laws) to a "disordered" state (high temperature, gibberish).
  • Instruction Fine-tuning Matters: Fine-tuning LLMs for instructions shifts the optimal temperature for Heaps' and Zipf's law adherence, often to higher temperatures, and can make the phenomenon less pronounced.
  • Model Architecture is Secondary: Factors like pretraining corpus, training algorithm, and model architecture (within the transformer paradigm) have a less significant impact on Zipfian properties compared to temperature.
  • Prompting has Minimal Impact: The use of long prompts versus single-token generation does not significantly alter the adherence to Zipf's law, though it can influence Heaps' law behavior by affecting the emergence of new tokens.

Key Results

  • For natural texts, Zipf's law exponents ($α$) for words and tokens are close to 1 (around 0.981 for words and 1.006 for tokens), and Heaps' law exponents ($β$) are around 0.855 for words and 0.801 for tokens.
  • LLM-generated texts show a pronounced maximum in the $R$ metric (related to Heaps' law) at $t \approx 1$ for many models, indicating human-like text generation at this temperature.
  • Large Llama models exhibit a similar maximum but at a higher temperature ($t \approx 2$).
  • The MAPE for Zipf's law shows a minimum (best adherence) at specific temperatures, typically around $t=1.0-1.2$ for many models, and $t=1.3-1.5$ for larger Llama models.
  • Instruction fine-tuning shifts these optimal temperatures, generally upwards.
  • Prompting had a negligible effect on Zipf's law adherence but influenced Heaps' law behavior by affecting new token generation.

Limitations and Future Work

  • Tokenizer Dependence: The study acknowledges that using a different tokenizer for analysis than the one used for generation can introduce minor artifacts, especially at text boundaries.
  • Language and Script Bias: The dataset is primarily limited to European languages using Latin and Cyrillic scripts, potentially limiting the generalizability of findings to other language groups or scripts (e.g., Chinese, Arabic).
  • Decoding Parameters: The research did not explore the influence of other decoding parameters like top-k, top-p, or no-repeat, which could affect statistical properties.
  • Future Work: The authors suggest further research into the impact of different tokenizers, exploring other language groups, and investigating the role of advanced decoding strategies. They also highlight the interesting phenomenon of finetuning affecting Heaps' and Zipf's laws differently as a topic for future exploration.

Summary updated

BibTeX
@inproceedings{mikhaylovskiy2025zipfs,
  author = {Nikolay Mikhaylovskiy},
  title = {{Zipf’s and Heaps’ Laws for Tokens and LLM-generated Texts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure