Abstract
Zipf's law, which states that the probability of an observation is inversely proportional to its rank, has been observed in many domains. While there are models that explain Zipf's law in each of them, those explanations are typically domain specific. Recently, methods from statistical physics were used to show that a fairly broad class of models does provide a general explanation of Zipf's law. This explanation rests on the observation that real world data is often generated from underlying causes, known as latent variables. Those latent variables mix together multiple models that do not obey Zipf's law, giving a model that does. Here we extend that work both theoretically and empirically. Theoretically, we provide a far simpler and more intuitive explanation of Zipf's law, which at the same time considerably extends the class of models to which this explanation can apply. Furthermore, we also give methods for verifying whether this explanation applies to a particular dataset. Empirically, these advances allowed us extend this explanation to important classes of data, including word frequencies (the first domain in which Zipf's law was discovered), data with variable sequence length, and multi-neuron spiking activity.
Introduction
For nearly a century, Zipf’s law has intrigued scientists across disciplines. From word frequencies in literature to the sizes of cities and the firing patterns of neurons, a consistent pattern emerges: a few events occur with high frequency, while the vast majority are rare, following a precise mathematical power law. Historically, explanations for this phenomenon have been domain-specific or relied on "fine-tuning" systems to a critical state. This research proposes a more universal and robust explanation: Zipf’s law arises naturally from the mixing of data influenced by underlying latent variables.
The Mechanism: Mixing Latent Variables
The core of this theory rests on the concept of "energy" ($ \mathcal{E} $), defined as the negative log-probability of an observation. Zipf’s law manifests when the distribution of this energy is broad and relatively flat. The authors demonstrate that even if individual "sub-models" (conditioned on a specific latent variable $z$) have narrow, non-Zipfian energy distributions, their mixture can produce a broad distribution if the latent variable significantly shifts the mean energy.
To quantify this effect, the researchers introduced the Proportion of Explained Energy Variance (PEEV). This metric measures how much of the total energy variance—the "broadness" required for Zipf's law—is attributable to a specific latent variable. As shown in the figure above, a higher PEEV indicates that the latent variable is a primary driver of the broad distribution.
Empirical Evidence: From Language to Neurons
The researchers validated this framework across diverse datasets, most notably in the first domain where Zipf's law was discovered: linguistics.
Word Frequencies
Using the British National Corpus, the study identified "part-of-speech" (e.g., nouns, verbs, adjectives) as a key latent variable. While individual grammatical categories have narrow energy distributions, their combination results in the classic Zipfian rank-frequency curve. In this context, the part-of-speech latent variable explains a significant portion of the Zipfian behavior (PEEV = 0.58).
Neural Activity
The theory also extends to high-dimensional biological systems. In retinal ganglion cells, the "time since stimulus" acts as a latent variable. When conditioned on a specific time point, the neural firing patterns do not follow Zipf's law; however, the law emerges beautifully when the entire dataset is considered, as the stimulus modulates the overall "volume" of activity across the population.
Conclusion
This work shifts the perspective on Zipf’s law from a rare "critical" phenomenon to a generic consequence of structured data. By showing that latent variables naturally broaden energy distributions, the authors provide a robust explanation that avoids the need for fine-tuning. While latent variables are not the sole cause in every domain—such as in antibody sequences where other stochastic processes dominate—this framework provides a powerful tool for identifying the hidden structures that drive universal statistical patterns in complex systems.
Summary updated