This paper provides a long-overdue theoretical bridge between classical readability formulas (like Flesch-Kincaid) and modern cognitive science by interpreting them through the lens of surprisal. It demonstrates that the "grade level" output of these formulas is essentially a measure of information content, revealing that one school grade corresponds almost exactly to one year of empirical vocabulary growth.
TL;DR
By applying information theory to decades-old readability metrics, this research shows that a "grade level" is a measurable unit of information (about 0.18 nats per word). The study reveals that classical formulas like Flesch-Kincaid implicitly track one year of children's vocabulary growth, though they significantly underweight the difficulty contributed by sentence length.
What Problem Does This Paper Address?
Readability formulas have been around for over 70 years, used in everything from school curriculum design to legal requirements for "plain language." However, they have long been criticized as "black boxes"—their coefficients are just unitless regression constants derived from old studies, lacking a grounding in how humans actually process language.
Meanwhile, the cognitive science of reading has converged on surprisal (the negative log probability of a word in context) as the standard metric for processing difficulty. Higher surprisal means a word is less predictable and takes longer to read. This paper tackles the gap between these two traditions, asking: can we explain these old formulas using modern information theory?
Methods
The author, Yo Ehara, uses a two-step theoretical approach:
- The Representation Theorem: He proves that any additive readability formula (one that sums up features like syllables per word and words per sentence) is mathematically equivalent to a cross-entropy under a specific probability distribution.
- Zipf Identification: He uses Zipf’s law of abbreviation—the observation that more frequent words tend to be shorter—to link surface features (like syllable count) to surprisal.
By linearizing a "memoryless" surprisal model (where words are independent) at a reference text point, he derives identities that relate a formula's coefficients to measurable linguistic constants like the Zipf slope (how much surprisal increases per syllable) and unigram entropy (the average information per word).
Datasets
- wordfreq (v3.1.1): A large-scale frequency list used to estimate the Zipf slope for English and Japanese.
- Brown Corpus: A classic balanced corpus of English used to establish baseline sentence length distributions and text-level surprisal slopes.
- OneStopEnglish: A corpus of news articles rewritten at three difficulty levels (elementary, intermediate, advanced), used to validate how surprisal changes across human-leveled texts.
- UniDic: Used for part-of-speech and word-origin tagging in the Japanese analysis.
Key Insights and Ideas
- Grades as Information Units: The paper finds that one US school grade corresponds to roughly 0.15 to 0.19 nats of lexical surprisal per word. (A "nat" is the natural unit of information; 1 nat $\approx$ 1.44 bits).
- The Vocabulary Growth Match: Strikingly, this "information step" per grade matches empirical measurements of children's vocabulary growth. Normative data suggests children's vocabulary grows by about 0.16 nats per year. This implies that the Flesch-Kincaid formula is literally tracking one year of vocabulary expansion.
- The "Light" Length Term: The analysis reveals a major inconsistency. While the word-level terms align well with surprisal, the sentence-length terms in these formulas are 4 to 8 times smaller than they should be if they were truly measuring the surprisal of sentence boundaries.
- Diagnostic Power: This framework can "stress-test" formulas. For the Japanese jReadability formula, the analysis showed that some coefficients have the "wrong sign" to be surprisal, likely because the formula was picking up on genre-specific textbook conventions rather than fundamental processing difficulty.
Key Results
The study provides concrete information-theoretic values for the constants that drive these formulas:
- FKGL Conversion: For the Flesch-Kincaid Grade Level, the word coefficient (11.8) converts to 0.18 nats per word per grade.
- Zipf Slopes: The token-weighted Zipf slope for English was estimated at 2.13 nats per syllable and 0.90 nats per letter.
- Ratio Mismatch: A memoryless surprisal model predicts a ratio of word-to-length coefficients of ~5.1. However, the observed ratio for FKGL is 30.3, proving that the formula treats sentence length as much less difficult than information theory would suggest.
Limitations and Future Work
- Memoryless Assumption: The model assumes words are independent. In reality, context makes words more predictable. Integrating "contextual surprisal" from modern Large Language Models (LLMs) is a suggested next step.
- Genre Sensitivity: The relationship between word length and surprisal can vary by genre (e.g., technical prose uses long but frequent words), which classical formulas might misjudge.
- Historical Calibration: The original formulas were calibrated on readers from the mid-20th century; modern language usage and reading habits may have shifted the underlying constants.
Summary updated