Power Law Models of Vocabulary Growth

24 concepts14 papers

When reading a text, you encounter new words rapidly at first. Over time, however, the rate at which you discover brand-new words slows down because a small set of highly frequent words (like the, and, or of) dominates the text. If you plot the number of unique words (vocabulary size) against the total number of words read (text length), you get a curve that flattens out.

Understanding the exact mathematical shape of this curve is the goal of vocabulary growth modeling. It allows researchers to solve a fundamental problem in linguistics: how to measure and compare the lexical richness of texts when the texts themselves are of completely different lengths.

The Problem with Naive Ratios

Imagine trying to determine whether Shakespeare or a modern novelist has a richer vocabulary. A simple approach might be to calculate the Type-Token Ratio (TTR), which divides the number of unique words (types) by the total number of words (tokens).

However, as Evert et al. point out, measures of lexical richness like TTR are inherently dependent on sample size. Because the rate of discovering new words naturally decays as a text gets longer, a shorter text will almost always yield a higher TTR than a longer text, even if they are pulled from the exact same source. To make fair comparisons, researchers require statistical methods to interpolate or extrapolate vocabulary growth curves to a common token count.

Without a robust growth model, it is impossible to distinguish between a genuinely diverse vocabulary and one that merely appears large because the text is long. As Sunehag argues, traditional word frequency models often fail if they do not explicitly incorporate a component that accounts for how the total number of unique word types varies with document length.

The Mechanics of Heaps' Law

To capture this non-linear relationship, researchers rely on power laws, most notably Heaps' Law. Heaps' Law states that vocabulary size scales as a power function of text length.

Meletti demonstrated this relationship empirically by analyzing massive textual datasets, formalizing the connection between word count ($W$) and unique form count ($F$) via the power law:

$$F = A \cdot \left(\frac{W}{10,000}\right)^B$$

When plotted on a bi-logarithmic (log-log) scale, this power-law relationship becomes a straight line. The slope of this line provides a universal principle for how textual complexity scales with length. As Mikhaylovskiy explains, assessing adherence to Heaps' Law allows us to measure exactly how vocabulary diversity scales with corpus size, a property that even modern Large Language Models (LLMs) only replicate under specific generation conditions.

Vocabulary Size (Log)
  ^
  |       /  <- Straight line on a log-log plot
  |      /
  |     /
  |    /
  |   /
  +---------------------------> Total Tokens (Log)

Why does this power-law behavior hold? Ellis et al. describe a fundamental statistical constraint: as a text grows, common words are eventually exhausted. To maintain linguistic variety and prevent a composition from becoming entirely repetitive, longer texts inherently require a more diverse and rare vocabulary.

Reading the Exponent: Morphological Complexity

The parameters of these power laws are not just arbitrary constants; they reveal deep structural properties of a language or domain. Specifically, the growth exponent (the parameter $B$ or $a$ in Heaps' Law) serves as a direct metric for vocabulary richness.

  • High Exponent (Steeper Curve): Indicates a morphologically rich or highly productive system where new words are encountered frequently. Esmaili et al. used power-law exponents to compare dialects of the Kurdish language, demonstrating that a higher growth exponent quantitatively signals a more morphologically complex language system.
  • Low Exponent (Flatter Curve): Indicates a highly specialized, repetitive, or controlled lexicon. Chen applied this framework to multilingual legal texts, showing that "legalese" features a lower exponent because domain-specific drafting prioritizes extreme precision and repetition over linguistic variety.

This scaling behavior can even track human language development. For example, Abdelali et al. tracked vocabulary growth across primary school curricula, showing a structured progression where the unique lemma count systematically expands from grade to grade.

Extrapolation and Lexical Closure

If a text follows a predictable mathematical law, we can use a small sample of text to predict the vocabulary properties of a much larger corpus. This process of vocabulary growth extrapolation is vital for both theoretical linguistics and practical software engineering.

Tuldava proposed a probabilistic model based on a "restriction of variety" principle that accurately predicts vocabulary sizes for texts up to $10^7$ words, matching empirical data within a few percentage points. By fitting observed data to these curves, frameworks like Large-Number-of-Rare-Events (LNRE) modeling allow researchers to normalize sample volumes and compare generative productivity across unequal datasets (Evert et al.).

This predictive capability is highly practical. Caverlee et al. applied Heaps' Law to sample hidden, query-only web databases. By analyzing a tiny initial sample, their system could estimate the total unique vocabulary and information density of a database before committing significant network resources to data extraction.

Vocabulary Size
  ^
  |          / - - - Extrapolated Curve (Predicted)
  |         /
  |   ____/ <- Observed Data (Small Sample)
  |  /
  +---------------------------> Total Tokens

Extrapolation also tells us when a text corpus is "complete" or representative of a language. Kovarik introduced the concept of lexical closure—the point at which a corpus ceases to introduce new unique words at a significant rate. By tracking the rate of new vocabulary discovery per fixed number of additional tokens, researchers can identify when a corpus has captured the essential features of a domain. This concept can be extended to syntactic closure, which measures when a corpus has largely exhausted novel grammatical structures and word combinations.

When building or evaluating annotated corpora, tracking this growth curve helps researchers like Silberztein determine the minimum corpus size required to achieve stable vocabulary coverage. Similarly, Dunn et al. quantified grammar and lexicon growth by incrementally combining sub-corpora, using the Heaps' Law growth parameter to evaluate how individual language production differs from collective language perception.

Beyond Words: Application to Music

Because vocabulary growth models capture the fundamental mathematics of unique symbol discovery in a sequence, their application extends far beyond natural language tokens.

Serrà applied Heaps' Law to six centuries of classical music harmony. By treating unique chordal combinations as "codewords" and musical compositions as texts, the author developed a robust metric for harmonic richness. This model successfully isolated true historical shifts in musical complexity from variations caused merely by the differing lengths of the musical pieces, proving that the mathematics of vocabulary growth is a universal tool for analyzing sequential structure.

Key Foundational Work

Proportion Between Number of Words and Number of Forms in the Works of the “Index Thomisticus” (1973) — contributed 4 concept elements to this direction.
A Mathematical Model of the Vocabulary-Text Relation (1980) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vocabulary_growth_modeling_0db8c744,
  title = {vocabulary growth modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vocabulary-growth-modeling/rdr_0db8c7448b734e649dba3070949797f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_vocabulary_growth_modeling_0db8c744_v1,
  title = {vocabulary growth modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vocabulary-growth-modeling/rdr_0db8c7448b734e649dba3070949797f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake