Understanding Zipf's law of word frequencies through sample-space\n collapse in sentence formation

Stefan Thurner, Rudolf Hanel, Bo Liu, Bernat Corominas‐Murtra

Abstract

The formation of sentences is a highly structured and history-dependent process. The probability of using a specific word in a sentence strongly depends on the 'history' of word-usage earlier in that sentence. We study a simple history-dependent model of text generation assuming that the sample-space of word usage reduces along sentence formation, on average. We first show that the model explains the approximate Zipf law found in word frequencies as a direct consequence of sample-space reduction. We then empirically quantify the amount of sample-space reduction in the sentences of ten famous English books, by analysis of corresponding word-transition tables that capture which words can follow any given word in a text. We find a highly nested structure in these transition tables and show that this 'nestedness' is tightly related to the power law exponents of the observed word frequency distributions. With the proposed model it is possible to understand that the nestedness of a text can be the origin of the actual scaling exponent, and that deviations from the exact Zipf law can be understood by variations of the degree of nestedness on a book-by-book basis. On a theoretical level we are able to show that in case of weak nesting, Zipf's law breaks down in a fast transition. Unlike previous attempts to understand Zipf's law in language the sample-space reducing model is not based on assumptions of multiplicative, preferential, or self-organised critical mechanisms behind language formation, but simply used the empirically quantifiable parameter 'nestedness' to understand the statistics of word frequencies.

Introduction: The Puzzle of Zipf’s Law

For nearly a century, quantitative linguistics has been defined by Zipf’s Law: the observation that the frequency of a word is inversely proportional to its rank in a frequency table ($f(r) \sim r^{-1}$). While numerous models—ranging from "preferential attachment" (the rich get richer) to "random typing"—have attempted to explain this phenomenon, they often treat language as a collection of independent tokens rather than a structured, history-dependent process. This paper proposes a new mechanism rooted in the actual structure of discourse: Sample-Space Reducing (SSR).

The Mechanism: Language as a Funnel

The core idea of the SSR model is that sentence formation is a process of progressive restriction. When a speaker chooses a word, they are not picking from the entire dictionary; they are picking from a subset of words that make sense given the preceding context.

As illustrated in Figure 4, choosing a subject like "wolf" immediately collapses the sample space of available verbs to those that are semantically and grammatically compatible (e.g., "howls," "runs"). This "nested" structure means that as a sentence progresses, the number of valid next steps shrinks, on average. This history-dependent collapse naturally produces the power-law distributions observed in human language.

Methods and Empirical Analysis

To test this theory, the authors analyzed ten classic English novels sourced from Project Gutenberg, including works by Charles Darwin, Charles Dickens, William Shakespeare, and James Joyce. These texts provided a wide range of vocabulary sizes and stylistic variations.

The researchers constructed Word-Transition Matrices, mapping which words ($j$) can follow any given word ($i$). They then quantified "nestedness"—a metric measuring how much the available sample space shrinks during sentence formation. The SSR model simulates text generation by jumping through these empirical matrices; if the model is at word $i$, it randomly selects the next word from the set of words that actually followed $i$ in the source text.

Key Results: Predicting the Scaling Exponent

The model successfully replicates the power-law distribution of word frequencies found in real-world texts. Crucially, the degree of nestedness in a specific book predicts its unique Zipf exponent ($\alpha$).

As shown in Figure 1, the model's prediction for Darwin's The Origin of Species closely tracks the actual data. Across the entire dataset, the model’s predicted exponents matched the actual exponents with a high correlation coefficient of $\rho = 0.95$.

To prove that the underlying structure was the primary driver, the authors performed randomization tests. When the transition matrices were randomized to destroy their nested structure, the power-law distribution vanished (the exponent dropped to $\sim 0.19$). This confirms that the hierarchical, forward-moving collapse of possibilities—rather than simple word popularity—is the engine driving the distribution.

Conclusion

By framing language as a sample-space reducing process, this research moves beyond statistical abstractions to a linguistically grounded explanation for Zipf's Law. The findings suggest that Zipf’s law is not a universal constant of language itself, but a statistical byproduct of how we use local hierarchical structures to convey meaning. This "nestedness" is a fundamental property that dictates the statistical signature of a text, allowing for book-by-book variations in frequency distributions based on the degree of structural constraint.

Summary updated

Abstract

The formation of sentences is a highly structured and history-dependent\nprocess. The probability of using a specific word in a sentence strongly\ndepends on the 'history' of word-usage earlier in that sentence. We study a\nsimple history-dependent model of text generation assuming that the\nsample-space of word usage reduces along sentence formation, on average. We\nfirst show that the model explains the approximate Zipf law found in word\nfrequencies as a direct consequence of sample-space reduction. We then\nempirically quantify the amount of sample-space reduction in the sentences of\nten famous English books, by analysis of corresponding word-transition tables\nthat capture which words can follow any given word in a text. We find a highly\nnested structure in these transition tables and show that this `nestedness' is\ntightly related to the power law exponents of the observed word frequency\ndistributions. With the proposed model it is possible to understand that the\nnestedness of a text can be the origin of the actual scaling exponent, and that\ndeviations from the exact Zipf law can be understood by variations of the\ndegree of nestedness on a book-by-book basis. On a theoretical level we are\nable to show that in case of weak nesting, Zipf's law breaks down in a fast\ntransition. Unlike previous attempts to understand Zipf's law in language the\nsample-space reducing model is not based on assumptions of multiplicative,\npreferential, or self-organised critical mechanisms behind language formation,\nbut simply used the empirically quantifiable parameter 'nestedness' to\nunderstand the statistics of word frequencies.\n
BibTeX
@article{thurner2015understanding,
  author = {Stefan Thurner and Rudolf Hanel and Bo Liu and Bernat Corominas‐Murtra},
  title = {{Understanding Zipf's law of word frequencies through sample-space\textbackslash\{\}n collapse in sentence formation}},
  journal = {Journal of The Royal Society Interface},
  year = {2015},
  doi = {10.1098/rsif.2015.0330},
}

Loading related content...

Figure