How do we determine if a word is difficult? While human language processing involves complex cognitive mechanics, a remarkably simple heuristic has dominated natural language processing for decades: counting how often a word appears in text.
The intuition behind this approach is rooted in cognitive exposure. Words that a reader encounters daily require less cognitive load to retrieve from memory. Heuiseok Lim demonstrated this connection by using corpus-based usage frequency to model the cognitive load of lexical retrieval in language rehabilitation games. Because our brains are statistical engines shaped by exposure, raw statistical counts from massive text collections serve as an incredibly robust, language-independent proxy for linguistic difficulty.
Beyond individual words, this statistical property scales up to capture broader structural and semantic features. For instance, Sharon A. Caraballo et al. showed that word frequency acts as a statistical proxy for conceptual generality, as broader, more foundational terms naturally appear more frequently across a text collection than highly specific ones. Similarly, Aurélie Herbelot et al. demonstrated that simple frequency-based metrics often perform as well as complex distributional measures in identifying word specificity and semantic informativeness.
Absolute Frequency as an Unbeatable Baseline
In tasks like Complex Word Identification (CWI) and Lexical Simplification (LS), the first step is identifying which words will trip up a reader. While one might expect deep semantic models to be necessary here, simple frequency thresholding remains an incredibly robust baseline.
In the SemEval-2012 Task 1: English Lexical Simplification shared task, Sunil Kumar Jauhar et al. observed that distributional frequency from massive web-scale corpora provides a context-agnostic baseline that complex, context-aware models frequently struggle to beat. This finding was echoed over a decade later in The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline, where Matthew Shardlow et al. confirmed that rare words are consistently perceived as more complex by human annotators across diverse language families.
[Raw Text] ---> Count Frequencies in Corpus ---> Low Frequency ===> Complex Word ("pulchritudinous")
---> High Frequency ===> Simple Word ("beautiful")
Even when building systems to flag complex words, a naive strategy of checking if a more frequent synonym exists in a database yields remarkably high recall, as shown by Matthew Shardlow in A Comparison of Techniques to Automatically Identify Complex Words. Shubham Pateria further validated this in Garuda & Bhasha at SemEval-2016 Task 11: Complex Word Identification Using Aggregated Learning Models by demonstrating that simpler models leveraging word frequency can outperform complex ensembles in identifying complex words.
Relative Frequency and Cross-Corpus Contrasts
Raw frequency counts tell us how common a word is in general language, but they fail to capture domain-specific jargon or stylistic shifts. To solve this, researchers look at relative frequency—the ratio of a word's occurrence across two distinct corpora.
Consider the challenge of simplifying scientific or medical text. A word like "pathogen" might be common in medical journals but rare in everyday speech. In Putting it Simply: a Context-Aware Approach to Lexical Simplification, Or Biran et al. calculated a complexity metric using the ratio of a word's frequency in a standard corpus versus a simplified corpus (such as Simple Wikipedia). Words with a high standard-to-simple ratio are prime candidates for replacement.
$$\text{Complexity Score} = \text{Word Length} \times \frac{\text{Frequency}_{\text{Standard}}}{\text{Frequency}_{\text{Simplified}}}$$
This cross-corpus contrast is highly effective for technical translation. In SimpleScience: Lexical Simplification of Scientific Terminology, Jessica Hullman et al. developed a complexity metric by calculating the frequency ratio between a specialized technical corpus and a general-purpose corpus. By ensuring a candidate replacement word passes a minimum frequency threshold in the general corpus, the system guarantees the substitute is genuinely accessible to lay audiences. Similarly, Ellie Pavlick et al. utilized this contrasting corpus methodology in Inducing Lexical Style Properties for Paraphrase and Genre Differentiation to induce lexical style scores for formality and complexity.
Tailoring the Frequency Source to the Audience
Not all large text collections are created equal. A word's absolute frequency in a massive web crawl might not reflect how familiar it is to a specific target audience, such as language learners, patients, or general readers.
For non-native speakers, conversational familiarity matters more than formal text frequency. To capture this, Gustavo H. Paetzold et al. ranked word simplicity using dialogue-heavy corpora like movie subtitles in Unsupervised Lexical Simplification for Non-Native Speakers, finding that conversational frequency serves as a better proxy for the language encountered by English learners. Recently, Hongyu Sun advanced this idea in Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?, showing that word frequencies derived from unscripted YouTube subtitles outperform even massive large language models (LLMs) like GPT-4 in predicting human word familiarity.
When specialized psycholinguistic databases are missing for a language, web search frequency can step in. In the development of tools for Brazilian Portuguese, the authors of Fostering Digital Inclusion and Accessibility: The PorSimples project for Simplification of Portuguese Texts and SIMPLIFICA: a tool for authoring simplified texts in Brazilian Portuguese guided by readability assessments demonstrated that ranking candidate synonyms by their web search popularity provides an excellent proxy for ease of understanding, allowing automated tools to suggest accessible replacements based on common usage patterns.
Frequency as a Guardrail in Neural Architectures
With the advent of word embeddings and transformer-based language models, one might assume that raw frequency metrics have become obsolete. In reality, frequency metrics serve as essential anchors and guardrails for neural systems.
In katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English, Katja Voskoboinik demonstrated that traditional word frequency features remain highly predictive of lexical complexity even when combined with contextual embeddings like BERT. Neural models excel at generating semantically valid substitutes, but they often struggle to judge which substitute is simpler. To address this, Jipeng Qiang et al. combined BERT's prediction probabilities with explicit word frequency metrics in Lexical Simplification with Pretrained Encoders, ensuring that the model prioritizes common words from general literature.
Furthermore, frequency information is embedded deep within the geometry of neural representations. Xuebo Liu et al. designed a curriculum learning strategy for machine translation in Norm-Based Curriculum Learning for Neural Machine Translation by summing the L2 norms of word embeddings.
High-Frequency Word (e.g., "the") ---> Small L2 Norm ---> Low Information / Simple
Low-Frequency Word (e.g., "anomaly") ---> Large L2 Norm ---> High Information / Complex
Because embedding norms naturally correlate with frequency—where frequent, low-information words have smaller norms and rare, high-information words have larger norms—the authors were able to quantify sentence difficulty and accelerate model training without needing an auxiliary language model.
Limitations and Cognitive Nuances
While frequency is a powerful proxy, researchers have noted its boundaries. Lucia Specia argued in UOW-SHEF: SimpLex - Lexical Simplicity Ranking based on Contextual and Psycholinguistic Features that human simplicity perception is driven more strongly by intrinsic cognitive factors, such as the age of acquisition and concreteness, than by raw statistical occurrence alone.
Context also dictates when frequency matters most. In metaphor simplification, Yulia Clausen et al. found in Metaphors in Text Simplification: To change or not to change, that is the question that while age of acquisition is a strong general predictor for word difficulty, word frequency becomes the critical deciding feature specifically when determining whether a metaphorical expression should be altered or retained. Frequency metrics may not capture every psychological nuance, but their computational efficiency—as highlighted by Adam Kilgarriff et al. in their work on corpus similarity—ensures they remain foundational to text complexity modeling.
Key Foundational Work
Simplifying Text for Language-Impaired Readers (1999) — contributed 1 concept elements to this direction.
Determining the specificity of nouns from text (1999) — contributed 1 concept elements to this direction.
Measures for Corpus Similarity and Homogeneity (1998) — contributed 1 concept elements to this direction.