Modeling Word Difficulty in Context

99 concepts63 papers1 proposal

When we think of a "difficult" word, we often think of rare, multi-syllable academic terms like anachronism or obfuscate. For decades, natural language processing systems relied on this intuition, treating lexical complexity as a static, intrinsic property of a word. If a word appeared rarely in a massive text corpus, it was flagged as complex; if it was common, it was assumed to be simple.

However, this static view fails to capture how humans actually process language. A common word like bank can suddenly become difficult when used in a specialized financial or ecological context. Conversely, a text dense with complex vocabulary can increase the cognitive load of understanding even its simpler words.

To build systems that truly understand readability—whether for text simplification, educational technology, or language modeling—researchers have shifted toward contextualized lexical complexity. This paradigm treats word difficulty not as a fixed attribute in a dictionary, but as a dynamic property shaped by a word's immediate syntactic and semantic environment.

The Failure of Static Frequency

Early lexical simplification pipelines operated under a straightforward assumption: identify complex words in isolation and replace them with higher-frequency synonyms. As Matthew Shardlow demonstrates in A Comparison of Techniques to Automatically Identify Complex Words, while frequency thresholds offer a robust baseline, evaluating words in isolation inherently ignores how specific semantic usages dictate difficulty.

This limitation became highly visible during early shared tasks. In their analysis of the SemEval-2012 Task 1: English Lexical Simplification, Sunil Kumar Jauhar and Lucia Specia noted a persistent tension: while global word frequency is a powerful proxy, human perceptions of simplicity are deeply context-dependent. A synonym that fits perfectly in one sentence can become obscure or nonsensical in another. Lucia Specia later demonstrated in UOW-SHEF: SimpLex - Lexical Simplicity Ranking based on Contextual and Psycholinguistic Features that context-blind frequency models suffer from significant noise because they fail to perform word sense disambiguation, treating all instances of a polysemous word identically regardless of its situational difficulty.

How Context Alters Cognitive Load

Why does context matter so much? Empirical evidence from eye-tracking and language learning reveals that the surrounding text directly modulates a reader's cognitive processing.

In a study tracking eye movements, Victoria Yaneva et al. evaluated how lexical properties affect reading speed in Effects of Lexical Properties on Viewing Time per Word in Autistic and Neurotypical Readers. They found that "Time Viewed" per word fluctuates significantly based on the word's syntactic and semantic environment. An identical word imposes varying cognitive loads depending on the structural complexity of the sentence around it.

[Static View]     Word Type ("interest") ──> Fixed Difficulty Score

[Contextual View] Sentence Context ──┐
                                     ├─> Dynamic Token Difficulty
                  Word Token       ──┘

This environmental influence operates in two distinct directions:

  1. Linguistic Bleeding: The complexity of neighboring words spills over. Reno Kriz et al. showed in Simplification Using Paraphrases and Context-Based Lexical Substitution that incorporating features like the average syllable count of surrounding words improves complex word identification. A reader's ability to parse a target word is actively hindered if the immediate textual neighborhood is dense and complex.
  2. Contextual Scaffolding: Conversely, a supportive context can make an otherwise difficult word easy to comprehend. When evaluating language acquisition, Chak Yan Yeung and John Lee discovered a stark discrepancy between isolated and contextual understanding in Personalized Text Retrieval for Learners of Chinese as a Foreign Language. Words that language learners flagged as "complex" when presented in a list were frequently understood without issue when encountered inside a well-structured sentence, proving that surrounding context provides critical semantic clues that aid real-time comprehension.

From Word-Type to Token-Level Modeling

To capture this dynamic behavior, modern NLP architectures have transitioned from predicting difficulty at the word-type level (the dictionary entry) to the token level (the word instance in a specific sentence).

Instead of classifying words individually, Sian Gooding and Ekaterina Kochmar reframed the problem in Complex Word Identification as a Sequence Labelling Task. By using a sequence-based architecture (such as a BiLSTM), the model evaluates the entire sentence at once. This allows the system to assign different complexity labels to the exact same word depending on its position and usage, successfully capturing polysemy and syntactic variations.

The current state of the art relies on fusing intrinsic word properties with deep contextualized representations. During the SemEval-2021 Task 1: Lexical Complexity Prediction shared task organized by Matthew Shardlow et al., the community moved from binary classification to continuous complexity scoring using contextualized datasets.

The most successful systems solve this by looking at the problem through two lenses simultaneously:

  • Token-level signals capture the specific semantic meaning of the word in that exact spot.
  • Sentence-level signals capture the overarching structural and conceptual difficulty of the passage.

For instance, Tuqa Bani Yaseen et al. leveraged this dual perspective in JUST-BLUE at SemEval-2021 Task 1: Predicting Lexical Complexity using BERT and RoBERTa Pre-trained Language Models by training separate model components for token and sentence representations, combining them to modulate the final difficulty score. Similarly, Xiaobing Zhou and Yang Bai demonstrated in hub at SemEval-2021 Task 1: Fusion of Sentence and Word Frequency to Predict Lexical Complexity that fusing global statistical features (like TF-IDF) with deep contextual embeddings from RoBERTa yields significantly higher accuracy than relying on either feature alone.

Why Contextual Modeling Works

The reason contextualized models outperform static frequency lists comes down to how information is distributed in natural language. High-entropy tokens—such as specific nouns and verbs that carry dense semantic meaning—are inherently difficult to predict or understand in a vacuum.

As Hai Wang and David McAllester show in On-The-Fly Information Retrieval Augmentation for Language Models, language models struggle with these content-heavy words when the broader factual context is missing. When relevant background context is provided, the perplexity of these complex tokens drops dramatically.

Contextualized language models work because their internal attention mechanisms mimic this exact process. When a model processes a sentence, the embedding for a target word is updated by attending to its neighbors. If a word is used in a rare, technical sense, the surrounding technical terms alter its contextual embedding.

This behavior is what makes reference-free evaluation metrics possible. For example, Xinran Zhao et al. introduced Towards Reference-free Text Simplification Evaluation with a BERT Siamese Network Architecture, which uses a Siamese BERT architecture to assess text complexity. Because the embeddings are contextualized, the model can directly evaluate whether a substitution simplifies a word based on its exact situational meaning, rather than relying on rigid, pre-computed frequency dictionaries.

Ultimately, modeling word difficulty in context works because it aligns machine representations with human cognitive realities. It recognizes that language processing is an integrative act: we do not read words as isolated units, but as interconnected pieces of a larger mental model.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_context_dependent_word_difficulty_68917ae0,
  title = {context-dependent word difficulty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-word-difficulty/rdr_68917ae0dc49403981b00c92d89cc91e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_context_dependent_word_difficulty_68917ae0_v1,
  title = {context-dependent word difficulty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-word-difficulty/rdr_68917ae0dc49403981b00c92d89cc91e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake