Entropy Rate Constancy in Natural Language Discourse

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 264 papers118 conceptsfrozen 2026-10-04

When we speak or write, we are essentially managing a data stream. If we pack too much information into a single second, our listener’s "processor" might crash, unable to keep up with the density of new ideas. Conversely, if we speak too slowly or use too many predictable words, we waste time and bandwidth.

Research across linguistics, cognitive science, and artificial intelligence suggests that humans have evolved a sophisticated solution to this problem: we strive to maintain a Uniform Information Density (UID). We instinctively adjust our language—from the speed of our vocal cords to the complexity of our grammar—to keep the flow of information at a steady, manageable rate.

The Constant Rate Principle

The foundation of this idea, often called the Constancy Rate Principle or Entropy Rate Constancy, was pioneered by Dmitriy Genzel and Eugene Charniak. In their seminal work, Entropy Rate Constancy in Text, they observed a curious phenomenon: as a story or essay progresses, individual sentences actually become more complex and "surprising" when viewed in isolation.

Why would this be? As a discourse unfolds, the sender and receiver accumulate shared context. This context makes future words more predictable. To prevent the information rate from dropping (which would be inefficient), writers increase the "local" complexity of their sentences. By balancing the growing predictability of the context with the increasing complexity of the words, the overall information flow remains near-constant.

Smoothing the Spikes with Surprisal

The primary metric for information density is surprisal—a measure of how unexpected a word is given its context. The UID hypothesis, as detailed by Matthew W. Crocker, Vera Demberg, and Elke Teich in the IDeaL project, posits that speakers avoid "information spikes" that exceed a listener's cognitive capacity.

We use several "smoothing" mechanisms to achieve this:

  1. Acoustic Duration: When a word is grammatically or semantically surprising, we physically slow down. Vera Demberg and colleagues showed that syntactic surprisal affects spoken word duration, while Asad B. Sayeed demonstrated a similar effect for semantic surprisal. This gives the listener more time to process dense content.
  2. Syntactic Redundancy: If a subject is hard to predict, we might add a redundant pronoun to "smooth" the transition. For example, Yiming Liang found that subject doubling in French (e.g., "Marie, she wants...") acts as a cognitive buffer for high-surprisal subjects.
  3. Omission of Markers: Conversely, when a relationship is highly predictable, we omit words to avoid an information "trough." Fatemeh Torabi Asr and Vera Demberg found that writers omit discourse connectives like "because" or "but" when the logical link is already obvious, maintaining a steady information rate.

A Universal Speed Limit

Is this just a quirk of English or French? Evidence suggests it is a fundamental biological or cognitive constraint. Tiago Pimentel and Clara Meister analyzed over 600 languages and found a universal surprisal-duration trade-off. Languages that pack more information into individual sounds are spoken more slowly, while "information-light" languages are spoken faster.

This principle even extends beyond speech. Yang Xu and Yang Cheng have shown that hand gestures are used rationally to maintain entropy rate constancy. When our words become more predictable toward the end of a sentence, our gestures often become more complex and informative to compensate.

The Probability-Quality Paradox in AI

Understanding UID has profound implications for how we build AI. Currently, many Large Language Models (LLMs) suffer from a "probability-quality paradox": if you always pick the most probable next word (the "mode"), the model becomes repetitive and dull. If you pick low-probability words, it becomes incoherent.

Clara Meister, Tiago Pimentel, and Ryan Cotterell argue that human-like text isn't about maximum probability, but about hitting a "sweet spot" of information. Their Expected Information Hypothesis suggests that high-quality text maintains an information content that clusters around the model's average entropy.

This has led to new decoding strategies like Locally Typical Sampling, which forces the model to choose words that maintain a consistent information rate. Similarly, the Stable Entropy Hypothesis proposed by Timothy J. O'Donnell and colleagues uses this principle to prevent model degeneration, ensuring that AI-generated stories feel as balanced and "natural" as human prose.

Papers in this version

Showing 60 of 64 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 67 papers
  2. Version 1 · 2026-10-04 · 64 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_uniform_information_density_f44489bb,
  title = {uniform information density},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uniform-information-density/rdr_f44489bb46bd4daea3a6632cf5c9e6e2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_entropy_rate_constancy_in_natural_langua_f44489bb_v1,
  title = {Entropy Rate Constancy in Natural Language Discourse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entropy-rate-constancy-in-natural-language-discourse/rdr_f44489bb46bd4daea3a6632cf5c9e6e2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}