Informativeness through the Lens of Predictability

This direction is merged into another direction: surprisal-based cognitive modeling. Its last version stays readable and citable here.

Version 1 of 110 papers16 conceptsfrozen 2026-10-04

In any given sentence, some words are almost entirely redundant. If you read "The cat sat on the...", your brain immediately fills in "mat" or "floor." These highly predictable words provide the structural backbone of language but carry relatively little new information. Conversely, words that are difficult to guess—like a specific name, a date, or a technical term—are where the actual "news" of the sentence resides.

A growing body of research suggests that this inverse relationship between predictability and informativeness is not just a linguistic curiosity, but a powerful signal that can be used to extract knowledge, understand human communication, and make AI models more truthful.

The Inverse Relationship Principle

The core intuition is that a token’s informational value is inversely proportional to its predictability from the surrounding context. If a language model can easily reconstruct a word from its neighbors, that word is likely a functional or grammatical filler. If the model struggles to guess the word, that word is likely a high-value information carrier.

Zhuoer Wang, James Caverlee, and their colleagues operationalized this in their Differentiable Masker-Reconstructor (DMR) model. By training a system to identify tokens that are difficult for a language model to guess, they created an unsupervised method for extracting candidate answers from text. Their system prioritizes "surprise" over structure, identifying the specific facts that a reader would need to know to answer a question.

Filtering for Meaningful Entities

This predictability signal is particularly effective for identifying specific "slots" or "values" in dialogue, such as a hotel name or a price range. In conversational AI, distinguishing between a user's intent and the specific details they provide is a major challenge.

Renato Vukovic, Milica Gasic, and their team demonstrated that Masked Language Modeling (MLM) scores can serve as a zero-shot filter for these entities. In their SIGDIAL 2022 paper, they showed that words with high "prediction surprise" often correspond to domain-specific slots. By focusing on these low-probability tokens, they achieved significant improvements in transferring dialogue systems to entirely new domains without additional training.

Human Language as an Efficient Code

This relationship isn't just a trick for machine learning; it appears to be a fundamental property of how humans use language. Tiago Pimentel, Ryan Cotterell, and their co-authors explored this across 18 diverse languages in their 2020 study. They found a universal negative correlation between a word’s inherent ambiguity and the informativeness of its context.

Essentially, when speakers use a vague or ambiguous word, they systematically "fill the gaps" by providing a more descriptive, less predictable context. This suggests that human language maintains a constant balance between efficiency (using short, easy words) and clarity (ensuring the context is informative enough to resolve any ambiguity).

Anchoring Truthfulness in Surprise

In the era of Large Language Models (LLMs), this predictability signal has found a new application: preventing hallucinations. LLMs often "drift" into falsehoods because they treat every token with similar structural importance during generation.

The Self-Highlighted Hesitation (SH2) method, developed by Jushi Kai, Zhouhan Lin, and others, uses the model's own internal uncertainty to improve accuracy. By identifying the lowest-probability tokens in a prompt—which almost always correspond to factual content like proper nouns—the system forces the model to "pause and reflect" on these critical constraints. This contrastive decoding approach ensures the model remains anchored to the most informative parts of the input, significantly boosting the truthfulness of its outputs.

The Role of Attention

While predictability is a powerful signal, it is not entirely automatic. Research by Pratik Bhandari and Vera Demberg on degraded speech comprehension suggests that the cognitive benefits of predictability depend on active attention. Listeners use context to "fill in the blanks" of noisy audio only when they are attending to the sentence as a whole. This highlights that while predictability (or "Cloze probability") is an inherent property of the text, its utility as an information-processing shortcut requires the system—whether human or machine—to actively integrate the surrounding context.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_informativeness_through_the_lens_of_pred_8ebf3cf6,
  title = {Informativeness through the Lens of Predictability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informativeness-through-the-lens-of-predictability/rdr_8ebf3cf643b04c398e9952749876a75f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_informativeness_through_the_lens_of_pred_8ebf3cf6_v1,
  title = {Informativeness through the Lens of Predictability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informativeness-through-the-lens-of-predictability/rdr_8ebf3cf643b04c398e9952749876a75f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}