When we read a piece of text, we intuitively sense its richness. A repetitive, predictable passage feels shallow, while a vibrant essay or poem feels sophisticated. In computational linguistics, this intuition is formalized through lexical diversity—the measurement of how variety and vocabulary renewal manifest within a text.
Rather than simply counting obscure words, lexical diversity analyzes the mathematical relationship between the unique words used (types) and the total words generated (tokens). By tracking how quickly a speaker or writer introduces new vocabulary, researchers can uncover deep insights into human cognition, literary quality, and the behavioral differences between human and machine intelligence.
The Core Mechanism: From TTR to Length Independence
The simplest way to measure vocabulary richness is the Type-Token Ratio (TTR):
$$\text{TTR} = \frac{\text{Total Unique Words (Types)}}{\text{Total Words (Tokens)}}$$
If you write a 10-word sentence and every word is unique, your TTR is $1.0$. However, TTR suffers from a fundamental mathematical flaw rooted in the nature of language: as a text grows longer, the TTR naturally decays. This occurs because structural words ("the", "and", "of") must repeat continuously, and even the most expressive authors eventually exhaust their core vocabulary. Consequently, raw TTR cannot be used to fairly compare texts of different lengths.
To overcome this limitation, researchers employ length-independent variants:
- Moving Average Type-Token Ratio (MATTR): This approach uses a fixed-size sliding window (e.g., 100 words) across the text, calculating the TTR for each window and averaging the scores. This ensures that the metric reflects local vocabulary renewal rather than total document length.
- Corrected Type-Token Ratio (CTTR): This variant adjusts the denominator—typically scaling it by $\sqrt{2 \times \text{Tokens}}$—to counteract the natural sub-linear growth of vocabulary size.
- Measure of Textual Lexical Diversity (MTLD): This algorithm calculates the average number of words a text can sustain before its TTR drops below a predefined threshold.
As shown by Dimosthenis Antypas et al., combining standard TTR with robust metrics like MTLD allows models to capture the underlying richness of a text, making it possible to distinguish between distinct communication styles, such as separating factual discourse from online misinformation.
Cognitive Phenotyping and Health Tracking
Why does lexical diversity work as a diagnostic tool? Language generation is a complex cognitive task that requires retrieving concepts from semantic memory and structuring them in real time. When cognitive health is compromised, this retrieval mechanism degrades, leading to a narrower selection of words and increased repetition.
This relationship enables computational linguistic phenotyping—using text analysis to detect underlying neurological changes. For instance, Yael Dahari et al. demonstrated that tracking subtle declines in MATTR within social media posts can reveal early signs of an impending stroke up to 20 weeks before medical symptoms manifest. By decoupling vocabulary richness from writing errors, their model isolates genuine cognitive decline from superficial typos.
[Healthy Cognitive State] ---> Broad Semantic Retrieval ---> High MATTR (Varied Vocabulary)
[Cognitive Impairment/Decline] ---> Narrow Semantic Retrieval ---> Low MATTR (Repetitive Vocabulary)
Similarly, lexical diversity serves as a practical proxy for identifying vulnerability in real-world interactions. Xia Cui et al. showed that combining TTR with Part-of-Speech (POS) distributions from short speech segments allows automated systems to detect cognitive impairments or language barriers in telemarketing calls, providing a non-invasive way to protect vulnerable consumers.
Exposing the AI Creativity Gap
Large Language Models (LLMs) are highly proficient at maintaining grammatical fluency, yet they often struggle with the stylistic variance native to human expression. When generating text, optimization objectives can cause models to favor high-probability tokens, resulting in formulaic and repetitive outputs.
Lexical diversity metrics are highly effective at exposing this "creativity gap":
- Style and Nuance: In an evaluation of news commentary, Rayden Tseng et al. utilized TTR and CTTR to demonstrate that fine-tuned classifiers can easily distinguish AI-generated opinions from human ones, largely because LLMs struggle to replicate human expressive richness.
- Creative Writing: When applied to poetry, where repetition must be deliberate rather than accidental, Bingru Li et al. used MATTR to show that human poets consistently outperform LLMs in emotional resonance and stylistic variation, avoiding the formulaic vocabulary loops typical of synthetic text.
Despite these creative limitations, the predictability of LLM outputs makes lexical diversity an excellent evaluation metric. Iuri Macocco et al. discovered that during LLM conditioning and steering, simple TTR serves as a computationally inexpensive yet highly accurate proxy for text fluency, correlating strongly with much heavier evaluation architectures.
Evaluating Literary Craft and Expertise
In human-authored text, lexical diversity is a reliable indicator of domain expertise, writing quality, and structural intent.
In creative writing, professional mastery is defined more by vocabulary variety than by the use of obscure words. Justine Kao demonstrated this by comparing professional and amateur poetry, finding that high lexical diversity—rather than word difficulty—is the primary differentiator of professional craft. In long-form narratives like screenplays, the authors of Ming-Chang Chiu et al. calculated character-specific TTRs to measure the distinctiveness of individual voices, proving that deeper characterization correlates directly with higher overall script quality.
This principle extends to academic and educational writing as well:
- Essay Assessment: Jia Liu showed that lexical variety is a primary differentiator between high- and low-scoring argumentative essays in standardized language tests.
- Formulaic Language: When analyzing academic abstracts, Danfeng Huang applied token-type ratios specifically to "lexical chunks" (formulaic phrases like in terms of). The author found that non-native writers often rely on a much narrower, more repetitive set of standardized expressions compared to native speakers, who deploy these phrases with greater flexibility.
Mapping Corpora and Prompt Evolution
Beyond individual texts, lexical diversity serves as a macro-level diagnostic tool for analyzing entire datasets, tracking user behavior, and understanding model training dynamics.
Pretraining Data Complexity
The richness of a pretraining corpus sets a ceiling on a language model's downstream capabilities. Ameeta Agrawal et al. demonstrated that pretraining on corpora with high lexical complexity (measured via TTR and unique token counts) yields significant performance gains on general language understanding tasks, proving that data quality and variety can be more impactful than raw dataset size.
Prompt Homogenization
As generative models integrate into society, the language users employ to interact with them is shifting. Maria-Teresa De Rosa Palmini et al. analyzed text-to-image prompts over time using TTR and Self-Repetition Scores, uncovering a trend toward linguistic homogenization. As users converge on a narrow set of optimized keywords, prompt diversity drops, which directly limits the visual variety of the generated images.
Behavioral Complexity Bounds
Lexical diversity can also map the boundaries of human adversarial behavior. In an analysis of over two million real-world interactions, Aldan Creo et al. evaluated the lexical diversity and discourse coherence of LLM jailbreak attempts. They discovered a strict "complexity ceiling": despite the evolving nature of safety filters, the linguistic sophistication of human jailbreak prompts remains no higher than that of standard, everyday conversation.
Media and Institutional Profiling
Finally, vocabulary profiles allow researchers to map how information is framed across different sectors. By analyzing specialized terminology with TTR, the authors of Eleonora Cappuccio et al. and Nicola Cirillo et al. demonstrated how media outlets and institutional documents establish distinct lexical fingerprints. These profiles reveal varying levels of technical specialization and accessibility based on the target audience.
Key Foundational Work
A Case Study of Text and Discourse Based on Juku, Coh-Metrix, and Linguistic Inquiry Word Count (LIWC) (2015) — contributed 1 concept elements to this direction.
A Computational Analysis of Style, Affect, and Imagery in Contemporary Poetry (2012) — contributed 1 concept elements to this direction.