A Comparison of Features for Automatic Readability Assessment

Lijun Feng, Martin Jansche, Matt Huenerfauth, Noémie ElhadadCOLING 2010

TL;DR

This paper systematically compares a wide array of linguistic features for automatically predicting the reading grade level of primary school texts. They find that features derived from in-domain language models, particularly those related to nouns and entity density, offer the highest predictive power, leading to a significant improvement over the state-of-the-art in readability assessment.

What Problem Does This Paper Address?

Readability assessment aims to quantify how easy a text is to understand. Automatic readability assessment is crucial for selecting appropriate reading materials for diverse audiences and can also be leveraged for automatic text simplification. However, identifying which text properties best correlate with reading difficulty has been an ongoing challenge. Previous work has explored various features, from simple word and sentence statistics (shallow features) to more complex linguistic analyses like part-of-speech tagging and syntactic parsing. This paper seeks to provide a comprehensive comparison of these feature sets, including novel discourse-based features, to determine their individual and combined effectiveness in predicting reading grade levels for young learners.

Methods

The authors employ a classification approach, treating readability assessment as a task of assigning texts to specific grade levels (grades 2-5). They experiment with various machine learning classifiers, including Logistic Regression and Support Vector Machines (SVMs), using both standard and ordinal regression/classification methods.

Their core contribution lies in the systematic evaluation of a broad spectrum of features, categorized as:

  • Discourse Features: This includes entity-density features (tracking named entities and general nouns), lexical chain features (identifying semantic relations between entities), coreference inference features (resolving pronoun references), and entity grid features (representing entity distribution across sentences). Novelty is introduced in the coreference inference features.
  • Language Modeling Features: They train language models (unigrams to trigrams) directly on their corpus using different text sequence types (word-only, POS-only, word/POS pairs) and an information-gain approach. This is a key departure from prior work that used models trained on unrelated corpora.
  • Parsed Syntactic Features: Building on previous work, they expand the set of syntactic features derived from parsed sentence structures, focusing on noun phrases (NP), verb phrases (VP), prepositional phrases (PP), and subordinate clauses (SBAR).
  • POS-based Features: They analyze the predictive power of features derived from different parts of speech (nouns, verbs, adjectives, adverbs, prepositions), as well as broader categories like content and function words.
  • Shallow Features: These are traditional readability features like average sentence length, syllables per word, and the Flesch-Kincaid score.

Datasets

The primary dataset used is the Weekly Reader Corpus. This corpus consists of articles specifically aimed at elementary school students, labeled with grade levels from 2 to 5. The authors selected 1629 articles, discarding those that were primarily puzzles or quizzes, resulting in a final corpus of 1433 full articles. This corpus is a subset of articles previously used by Schwarm and Ostendorf (2005), but differs in size and content.

Key Insights and Ideas

  • In-Domain Language Models Reign Supreme: Language models trained directly on the target corpus (Weekly Reader) significantly outperform those trained on general or unrelated corpora, highlighting the importance of domain-specific linguistic patterns for readability.
  • Nouns are Powerhouses for Readability: Features based on nouns (both general and named entities) demonstrate the highest predictive power among all feature sets, particularly entity-density features. This suggests that the introduction and tracking of conceptual information through nouns is a strong indicator of text complexity for young readers.
  • Shallow Features Still Matter, Especially Sentence Length: While more complex features show promise, average sentence length remains a surprisingly strong predictor among shallow features, often outperforming more complex syllable-based metrics.
  • Discourse Features Show Limited Impact (for this corpus): For the primary school texts studied, discourse features (like entity grids and coreference) did not significantly improve accuracy beyond entity-density features. The authors hypothesize this might be due to the relatively low complexity of the target texts.
  • A Judicious Combination is Key: While individual feature sets have varying strengths, a well-chosen combination of features, particularly those identified as strong predictors, leads to the most substantial improvements in accuracy.

Key Results

  • The best performing models achieved 74% classification accuracy using a group-wise add-one-best feature selection method.
  • This represents a significant improvement over the state-of-the-art, which was around 63% accuracy.
  • Entity-density features alone achieved 59.63% accuracy with LIBSVM, a substantial gain over previous entity-density features.
  • Noun-based POS features achieved 58.15% accuracy with LIBSVM, demonstrating their strong discriminative power.
  • Language models trained on the Weekly Reader corpus achieved 68.38% accuracy (LIBSVM), outperforming Schwarm and Ostendorf's replicated features (52.21%).
  • Parsed syntactic features, when expanded, improved accuracy to 57.79% (LIBSVM).
  • Shallow features, when combined, reached 52.34% accuracy (Logistic Regression), with average sentence length being the most influential individual shallow feature.

Limitations and Future Work

  • Corpus Specificity: The authors acknowledge that the limited complexity of the Weekly Reader corpus might explain why discourse features showed less impact. They suggest future work should investigate these features on texts with higher grade levels.
  • SBAR Feature Performance: The study found SBAR features to be less useful than expected for predicting grade levels, a result they plan to examine further.
  • Future Work: The paper suggests investigating the discriminative power of discourse features on higher-level texts and further exploring the utility of syntactic features like SBARs. The success of in-domain language models also points to further exploration in this area.

Summary updated

BibTeX
@inproceedings{feng2010comparison,
  author = {Lijun Feng and Martin Jansche and Matt Huenerfauth and Noémie Elhadad},
  title = {{A Comparison of Features for Automatic Readability Assessment}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure