Revisiting Readability: A Unified Framework for Predicting Text Quality

Emily Pitler, Ani NenkovaEMNLP 2008 · OpenReview

Abstract

We combine lexical, syntactic, and discourse features to produce a highly predictive model of human readers' judgments of text readability. This is the first study to take into account such a variety of linguistic factors and the first to empirically demonstrate that discourse relations are strongly associated with the perceived quality of text. We show that various surface metrics generally expected to be related to readability are not very good predictors of readability judgments in our Wall Street Journal corpus. We also establish that readability predictors behave differently depending on the task: predicting text readability or ranking the readability. Our experiments indicate that discourse relations are the one class of features that exhibits robustness across these two tasks.

Introduction

For decades, readability has been measured by crude proxies. Formulas like Flesch-Kincaid or the Gunning Fog Index rely on surface-level metrics such as syllable counts and sentence length. While these work for distinguishing a children's book from a technical manual, they fail to capture what makes a text "well-written" for a competent adult audience. This research addresses a major gap in traditional models: the failure to account for coherence—the logical "flow" of a text. A sequence of simple sentences can remain unreadable if the connections between them are missing. This paper proposes a unified computational framework that integrates vocabulary, syntax, and high-level discourse structure to define text quality.

Methodology

The study utilized 30 articles from the Wall Street Journal (WSJ) corpus, chosen for their inclusion in the Penn Treebank (syntactic parses) and the Penn Discourse Treebank (human-annotated discourse relations). Ground-truth quality ratings were obtained from college students who rated each article on a 1–5 scale based on clarity and cohesion.

The framework evaluates several feature classes:

  • Discourse Relations: A multinomial model measuring the likelihood of logical connections (e.g., Comparison, Contingency).
  • Entity Coherence: Tracking subject/object transitions between sentences (the "Entity Grid").
  • Lexical Features: Using unigram language models to compare vocabulary probability against broader news corpora.
  • Syntactic Features: Measuring parse tree height and the frequency of noun phrases, verb phrases, and subordinate clauses.
  • Lexical Cohesion: Calculating word overlap and cosine similarity between adjacent sentences.

Performance was evaluated using Linear Regression for absolute scoring and Support Vector Machines (SVM) for pairwise ranking.

Results and Discussion

The Inadequacy of Surface Metrics

Traditional metrics like "characters per word" or "words per sentence" showed almost zero correlation with human ratings in this dataset. For an educated audience, sentence length is not the primary barrier to readability; rather, it is a lack of clarity and logical structure. As shown in Figure 1, standard surface metrics fail to achieve statistical significance, with the exception of overall text length.

Correlations of surface metrics with human readability judgments.

The Primacy of Discourse Relations

Discourse relations emerged as the most robust features. Even when transitioning from score prediction to pairwise ranking, discourse features remained highly predictive. The unified model achieved 88.88% accuracy in ranking, with discourse features alone contributing 77.36% accuracy. Figure 6 illustrates the strong correlation between discourse likelihood and perceived quality.

Correlations of discourse features with human readability judgments.

Syntactic and Coherence Insights

The study revealed a "Verb Phrase surprise": the average number of verb phrases per sentence was a stronger predictor of quality than sentence length. This suggests that sophisticated adult text often utilizes complex, multi-clause sentences to group related ideas. Conversely, the "Centering Theory Paradox" was observed: a negative correlation between subject-to-subject transitions and readability suggests that excessive repetition of the same subject can make professional prose feel stagnant.

Conclusion

This research demonstrates that discourse relations are the truest predictors of text quality. While entity coherence and syntax assist in ranking tasks, discourse relations are the only feature class consistent across both ranking and scoring. A significant limitation remains the reliance on human-annotated "Oracle" discourse features, as automated discourse parsers were not sufficiently mature at the time of the study. Future work should expand this framework to larger datasets and diverse genres to further validate these findings.

Summary updated

Abstract

We combine lexical, syntactic, and discourse features to produce a highly predictive model of human readers' judgments of text readability. This is the first study to take into account such a variety of linguistic factors and the first to empirically demonstrate that discourse relations are strongly associated with the perceived quality of text. We show that various surface metrics generally expected to be related to readability are not very good predictors of readability judgments in our Wall Street Journal corpus. We also establish that readability predictors behave differently depending on the task: predicting text readability or ranking the readability. Our experiments indicate that discourse relations are the one class of features that exhibits robustness across these two tasks.
BibTeX
@inproceedings{pitler2008revisiting,
  author = {Emily Pitler and Ani Nenkova},
  title = {{Revisiting Readability: A Unified Framework for Predicting Text Quality}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://openreview.net/forum?id=BybohbMdZS},
}
Figure