Measuring text readability with machine comprehension: a pilot study

Marc Benzahra, François YvonACL 2019

TL;DR

This paper investigates whether current machine comprehension systems can effectively measure text readability. The authors hypothesized that simpler texts would be better understood by these systems, but found a surprisingly weak correlation, suggesting that current models are not yet capable of reliably inferring text difficulty.

What Problem Does This Paper Address?

Evaluating the readability of text is crucial for ensuring information is accessible to its intended audience. Traditional readability measures rely on simplistic metrics like sentence and word length. While more sophisticated methods have emerged using machine learning to capture a wider range of linguistic features, they often require manually annotated data, which is costly and time-consuming to produce. This paper explores an alternative: using the performance of automatic text comprehension systems (like those that can fill in missing words in a text) as a proxy for human readability. The core question is whether these systems inherently perform better on simpler texts, thus providing an unsupervised way to gauge readability.

Methods

The researchers employed a range of neural language models (LMs) based on recurrent structures (RNNs) with LSTM cells, varying in their training data and architecture. They specifically used models trained on WikiText-2, WikiText-103, and a simplified version called Wiki-Simple. To test comprehension, they used a simplified cloze test generation strategy: randomly selecting words to be blanked out from texts. The performance metric was the accuracy of predicting the correct word (p@1), along with measures of how often the correct word appeared in the top 5, 25, or 50 predictions. They also experimented with a Transformer-based model (GPT-2) for comparison.

Datasets

  • Training Corpora:
    • WikiText-2 and WikiText-103: Standard English Wikipedia article datasets, differing in size and vocabulary.
  • Wiki-Simple: A simplified version of Wikipedia, intended to represent easier text.
  • Test Corpora:
    • Weebit: Educational news articles categorized into 5 complexity levels, targeting learners aged 7-16.
  • OneStopEnglish (OSE): Journalistic texts from The Guardian, rewritten to three distinct readability levels.

Key Insights and Ideas

  • Weak Correlation: The most striking finding is the surprisingly small correlation between machine comprehension performance (on cloze tests) and human-defined readability levels. This suggests that current systems, despite advancements, don't inherently "understand" text difficulty in the way humans do.
  • Model Diversity: While the researchers aimed for diverse model strengths, their RNN models showed less variation in performance across different training corpora than expected. This hints at potential limitations in their experimental setup or the models themselves.
  • GPT-2 Outperforms: The Transformer-based GPT-2 model significantly outperformed the RNN models, achieving much higher completion rates. However, even this advanced model did not show a strong correlation with readability.
  • POS Tag Bias: The random selection of words for cloze tests introduced a bias. Words like determiners and punctuation, which are easier for LMs to predict, artificially inflated performance, especially on simpler texts. This suggests that a more sophisticated cloze test generation strategy is needed.

Key Results

  • Low Correlation Scores: The Kendall's Tau correlation scores between completion rates and readability levels were consistently small, generally in the range of [0.1, 0.2] for OSE and Weebit. This is significantly lower than the correlation achieved by traditional readability scores like Flesch-Kincaid Grade Level (0.48 for OSE, 0.61 for Weebit).
  • GPT-2 Performance: GPT-2 completion rates (p@1/p@5) were roughly double those of the RNN models.
  • OSE vs. Weebit: Results were generally more consistent on the OSE dataset, with simpler texts showing slightly better completion rates. The Weebit dataset yielded more complex and sometimes contradictory results.

Limitations and Future Work

  • Naive Cloze Test Generation: The random selection of words for cloze tests is a significant limitation, as it doesn't account for the varying predictability of different word types (e.g., nouns vs. determiners). Future work should explore more sophisticated methods for generating cloze tests, potentially focusing on specific word types or using rightward context.
  • Model Strength and Diversity: The researchers acknowledge that their set of models might not have been diverse enough in strength, and suggest exploring more advanced LM architectures or full-fledged machine comprehension systems.
  • Dataset Limitations: The Weebit dataset's unbalanced distribution of complexity levels and the overlapping spans of readability indices were noted.
  • Future Directions: The authors propose several avenues for future research:
  • Exploring comprehension systems of controllable strength.
  • Developing alternative procedures for generating and filling cloze tests (e.g., restricting to specific gap types, increasing context).
  • Conducting direct experiments to see if automatic systems can consistently reproduce human subject performance, particularly with language learners.

Summary updated

BibTeX
@inproceedings{benzahra2019measuring,
  author = {Marc Benzahra and François Yvon},
  title = {{Measuring text readability with machine comprehension: a pilot study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure