Interpreting Classical Readability Formulae as Surprisal

Yo EharaarXiv 2026 · arXiv

This paper provides a long-overdue theoretical bridge between classical readability formulas (like Flesch-Kincaid) and modern cognitive science by interpreting them through the lens of surprisal. It demonstrates that the "grade level" output of these formulas is essentially a measure of information content, revealing that one school grade corresponds almost exactly to one year of empirical vocabulary growth.

TL;DR

By applying information theory to decades-old readability metrics, this research shows that a "grade level" is a measurable unit of information (about 0.18 nats per word). The study reveals that classical formulas like Flesch-Kincaid implicitly track one year of children's vocabulary growth, though they significantly underweight the difficulty contributed by sentence length.

What Problem Does This Paper Address?

Readability formulas have been around for over 70 years, used in everything from school curriculum design to legal requirements for "plain language." However, they have long been criticized as "black boxes"—their coefficients are just unitless regression constants derived from old studies, lacking a grounding in how humans actually process language.

Meanwhile, the cognitive science of reading has converged on surprisal (the negative log probability of a word in context) as the standard metric for processing difficulty. Higher surprisal means a word is less predictable and takes longer to read. This paper tackles the gap between these two traditions, asking: can we explain these old formulas using modern information theory?

Table 1: Classical formulae. “polyk” is the fraction of words with at least k syllables (Gunning’s “complex words” further exclude proper nouns, compounds and some suffixes, which we ignore); “notDC” the fraction of words not on the Dale-Chall familiar list; “long6” the fraction of words longer than six letters; “mono” the number of one-syllable words in a 150-word sample. Coefficients are those of the original publications; the Navy recalculations are from Kincaid et al. (1975). All but SMOG and Coleman-Liau have the additive form (1).

Methods

The author, Yo Ehara, uses a two-step theoretical approach:

  1. The Representation Theorem: He proves that any additive readability formula (one that sums up features like syllables per word and words per sentence) is mathematically equivalent to a cross-entropy under a specific probability distribution.
  2. Zipf Identification: He uses Zipf’s law of abbreviation—the observation that more frequent words tend to be shorter—to link surface features (like syllable count) to surprisal.

By linearizing a "memoryless" surprisal model (where words are independent) at a reference text point, he derives identities that relate a formula's coefficients to measurable linguistic constants like the Zipf slope (how much surprisal increases per syllable) and unigram entropy (the average information per word).

Datasets

  • wordfreq (v3.1.1): A large-scale frequency list used to estimate the Zipf slope for English and Japanese.
  • Brown Corpus: A classic balanced corpus of English used to establish baseline sentence length distributions and text-level surprisal slopes.
  • OneStopEnglish: A corpus of news articles rewritten at three difficulty levels (elementary, intermediate, advanced), used to validate how surprisal changes across human-leveled texts.
  • UniDic: Used for part-of-speech and word-origin tagging in the Japanese analysis.

Key Insights and Ideas

  • Grades as Information Units: The paper finds that one US school grade corresponds to roughly 0.15 to 0.19 nats of lexical surprisal per word. (A "nat" is the natural unit of information; 1 nat $\approx$ 1.44 bits).
  • The Vocabulary Growth Match: Strikingly, this "information step" per grade matches empirical measurements of children's vocabulary growth. Normative data suggests children's vocabulary grows by about 0.16 nats per year. This implies that the Flesch-Kincaid formula is literally tracking one year of vocabulary expansion.
  • The "Light" Length Term: The analysis reveals a major inconsistency. While the word-level terms align well with surprisal, the sentence-length terms in these formulas are 4 to 8 times smaller than they should be if they were truly measuring the surprisal of sentence boundaries.
  • Diagnostic Power: This framework can "stress-test" formulas. For the Japanese jReadability formula, the analysis showed that some coefficients have the "wrong sign" to be surprisal, likely because the formula was picking up on genre-specific textbook conventions rather than fundamental processing difficulty.

Key Results

The study provides concrete information-theoretic values for the constants that drive these formulas:

Table 2: English constants (token-weighted unless marked). H: unigram cross-entropy per word; λ, µ, R2: token- weighted least squares of −log π(w) on the feature; δC: surprisal difference between class C and its complement; τ ∗: the surprisal threshold that best reproduces class membership, with its token-weighted balanced accuracy. The Brown regression uses types with at least two tokens, for which the mean syllable count is 1.520.
  • FKGL Conversion: For the Flesch-Kincaid Grade Level, the word coefficient (11.8) converts to 0.18 nats per word per grade.
  • Zipf Slopes: The token-weighted Zipf slope for English was estimated at 2.13 nats per syllable and 0.90 nats per letter.
  • Ratio Mismatch: A memoryless surprisal model predicts a ratio of word-to-length coefficients of ~5.1. However, the observed ratio for FKGL is 30.3, proving that the formula treats sentence length as much less difficult than information theory would suggest.

Limitations and Future Work

  • Memoryless Assumption: The model assumes words are independent. In reality, context makes words more predictable. Integrating "contextual surprisal" from modern Large Language Models (LLMs) is a suggested next step.
  • Genre Sensitivity: The relationship between word length and surprisal can vary by genre (e.g., technical prose uses long but frequent words), which classical formulas might misjudge.
  • Historical Calibration: The original formulas were calibrated on readers from the mid-20th century; modern language usage and reading habits may have shifted the underlying constants.

Summary updated

Abstract

Surprisal correlates closely with reading time and is widely used in the cognitive science of reading as a measure of processing difficulty. Classical readability formulae such as the Flesch-Kincaid Grade Level, which scores a text by the US school grade expected to read it, are the most widely used practical measures of text difficulty, but their coefficients are unitless regression constants and have never been related to surprisal. We theoretically show that these coefficients can be interpreted as surprisal: every formula of the classical additive shape is a sum of per-word and per-sentence cross-entropies under a memoryless model, and once that model is tied to the language through Zipf's law and the formula is read as its linearisation at a reference text, each coefficient factorises into a scale in grades per nat, a Zipf slope and a reference sentence length. A grade thereby becomes a quantity in surprisal. Because the formula outputs grades and children's vocabulary has been measured by grade, the interpretation recovers the vocabulary growth the formula implicitly assumes and lets it be checked against development: holding sentence length fixed, the word coefficient of the Flesch-Kincaid formula converts to 0.18 nat of unigram surprisal per word per grade (0.15 to 0.19 across the regression-derived formulae), and normative root-word vocabulary growth (Biemiller and Slonim, 2001), converted through Zipf's law, gives 0.16 nat per year at the frontier of the known vocabulary. The word coefficient is one year of vocabulary growth, and the grade is a vocabulary scale. The identification also yields a diagnostic (the sentence-length terms are not those of a memoryless surprisal) and extends to a Japanese formula.
BibTeX
@misc{ehara2026interpreting,
  author = {Yo Ehara},
  title = {{Interpreting Classical Readability Formulae as Surprisal}},
  year = {2026},
  eprint = {2611.01359},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01359},
}
Figure