Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words

Eduardo G. Altmann, Janet B. Pierrehumbert, Adilson E. Motter

Introduction: The "Burstiness" Problem

Most statistical models of language, such as the "bag-of-words" model, assume words appear at random intervals, following a Poisson process. However, natural language is inherently "bursty": if a specific term like "tectonics" appears once, it is highly likely to reappear soon, followed by a long "lull" before it shows up in a different context. This study seeks to establish a universal mathematical scaling law for these recurrence intervals and explain why some words are significantly burstier than others.

Methodology: The Stretched Exponential Model

The researchers analyzed the "recurrence time" ($\tau$), defined as the number of words between successive appearances of the same word. They tested the fit of empirical data against a stretched exponential (Weibull) distribution: $F(\tau) = e^{-a\tau^\beta}$.

In this model, the parameter $\beta$ quantifies burstiness. A value of $\beta = 1$ corresponds to a random Poisson process, whereas lower values indicate higher burstiness—clusters of occurrences separated by long gaps. To explain variations in $\beta$, words were categorized into a "ladder" of four semantic classes based on formal semantics (logicality and permutability). Class 1 includes specific entities (e.g., "Darwin", "Africa"), while Class 4 consists of higher-level operators (e.g., "hence", "supposedly").

Recurrence time distributions and the stretched exponential model.

Key Results: Universality and Semantic Drivers

The study utilized diverse datasets, including USENET archives (200 million words), political debate transcripts, classic literature (War and Peace), and technical texts (Newton’s Principia). Across all datasets, the stretched exponential provided a near-perfect fit ($R^2 \approx 0.99$), significantly outperforming Poisson models.

Empirical validation of burstiness across semantic classes and word frequencies.

Key findings include:

  • Meaning Drives Dynamics: Burstiness is not merely a byproduct of frequency. Semantic class was found to be a stronger predictor of $\beta$ (accounting for 32% of variance) than log-frequency (26%).
  • Logicality vs. Context: Words with high "logicality" (Class 4) appear more regularly (higher $\beta$) because they are useful across almost any context. Conversely, "contextually anchored" words (Class 1) are essential for specific sub-topics but irrelevant once the conversation moves on, leading to extreme burstiness (lower $\beta$).
  • Universality: The scaling laws held true across different languages (English and Portuguese) and varying levels of formality, from spontaneous USENET posts to formal Latin.

Conclusion and Future Directions

While the model treats words as independent renewal processes, future work should address inter-word correlations (e.g., "natural" and "selection" often burst together). Additionally, the model focuses on long-term recurrence rather than short-term syntactic constraints. These patterns may mirror other collective human behaviors, such as the timing of emails or library book returns, suggesting a deep mathematical link between language and general human activity patterns.

Summary updated

BibTeX
@inproceedings{altmann2009beyond,
  author = {Eduardo G. Altmann and Janet B. Pierrehumbert and Adilson E. Motter},
  title = {{Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words}},
  booktitle = {PLoS},
  year = {2009},
  doi = {10.1371/journal.pone.0007678},
}
Figure