Quantifying Textual Richness through Lexical Diversity

This direction is split into other directions: lexical diversity analysis. Its last version stays readable and citable here.

Version 1 of 153 papers56 conceptsfrozen 2026-10-04

At the heart of computational linguistics lies a deceptively simple question: how do we measure the "richness" of a text? While a human reader can intuitively sense the difference between a professional poem, a repetitive SEO-optimized blog post, and the fragmented speech of someone facing cognitive decline, capturing this quality mathematically requires moving beyond simple word counts.

Researchers have converged on lexical diversity—the variety and sophistication of vocabulary within a corpus—as a powerful proxy for everything from professional expertise to neurological health.

The Foundation of Type Token Ratios

The most fundamental tool for measuring richness is the Type-Token Ratio (TTR). In this framework, "tokens" are the total number of words, while "types" are the number of unique words. A high TTR suggests a diverse vocabulary, while a low TTR indicates repetition.

However, TTR has a fatal flaw: it is highly sensitive to document length. As a text gets longer, the probability of introducing a brand-new word naturally decreases because common "function words" (like the, and, of) must be repeated. To solve this, researchers like Samuel Cahyawijaya and Jeanne McClure employ more stable metrics:

  • Moving Average TTR (MATTR): This calculates the TTR over a sliding window of fixed size, preventing longer documents from appearing artificially "simpler."
  • Measure of Textual Lexical Diversity (MTLD): This measures the average length of word sequences that can maintain a specific TTR threshold, providing a robust score regardless of total text length.

Lexical Diversity as a Cognitive Fingerprint

Why does the variety of our words matter? Across dozens of studies, lexical diversity acts as a "canary in the coal mine" for cognitive and emotional states.

In mental health, Jason Wei and colleagues identified a "complexity loss paradox," where high anxiety in online therapy clients correlates with significantly reduced lexical diversity. Similarly, in medical diagnostics, Akshay Valsaraj and Yael Dahari have shown that "vocabulary shrinkage"—measured through metrics like Brunet’s Index and Honore’s Statistic—can predict the onset of Alzheimer’s or even an impending stroke weeks before physical symptoms appear.

The intuition is that complex thought requires a wide array of linguistic tools; when the brain is under stress or in decline, it reverts to a "constrained language" profile, relying on high-frequency, "safe" words.

Distinguishing Expertise from Complexity

A common misconception is that "rich" writing simply means using "difficult" words. However, Justine Kao and Dan Jurafsky found that professional poets are distinguished not by obscure vocabulary, but by high lexical diversity—the ability to use a wide variety of standard words in creative ways.

This distinction between lexical sophistication (using rare words) and lexical density (the proportion of content words like nouns and verbs) is crucial. For instance:

  • Professional Critics: Ming Jiang demonstrated that expert film reviewers use higher information entropy, packing more technical and descriptive variety into their reviews than laypeople.
  • Academic Success: Jia Liu found that the primary differentiator in high-scoring student essays is not grammatical perfection, but the breadth of unique terms used to bridge ideas.

Detecting Algorithmic and Constrained Writing

Lexical metrics also allow us to identify text that is "unnatural" or constrained by external forces. I. V. Kononova notes that SEO-optimized text exhibits a tell-tale drop in lexical diversity because it is written for a "technogenic addressee"—a search engine that rewards the repetition of specific keywords.

Similarly, Ella Rabinovich and Shuly Wintner discovered that translated texts and non-native speech share a "constrained language" profile. Because the speaker is managing the cognitive load of two languages, they tend to use more cohesive markers and a narrower, more repetitive vocabulary than native speakers.

From Counts to Information Theory

Modern research is moving toward information-theoretic measures of richness. Instead of just counting unique words, researchers like Xiguang Han use one-dimensional entropy to measure the "uncertainty" or informativeness of a text.

If a text is highly predictable (low entropy), it likely contains little new information. If it is unpredictable (high entropy), it suggests a dense, high-value contribution. This approach allows platforms to automatically filter "noisy" or "routine" content in favor of "knowledge-driven" contributions, as seen in the work of Yeongbin Seo on filtering massive datasets for AI training.

By treating vocabulary not just as a list of words, but as a statistical distribution, these metrics provide a window into the human—or algorithmic—mind behind the text.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quantifying_textual_richness_through_lex_f1adf99f,
  title = {Quantifying Textual Richness through Lexical Diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-textual-richness-through-lexical-diversity/rdr_f1adf99f97944b4e8d9690a2fab640ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_quantifying_textual_richness_through_lex_f1adf99f_v1,
  title = {Quantifying Textual Richness through Lexical Diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-textual-richness-through-lexical-diversity/rdr_f1adf99f97944b4e8d9690a2fab640ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}