Promoting Pre-trained LM with Linguistic Features on Automatic Readability Assessment

Shudi Hou, Simin Rao, Yu Xia, Sujian LiAACL/IJCNLP 2022 · DOI

TL;DR

This paper investigates how incorporating linguistic features can boost the performance of pre-trained language models (PLMs) for automatic readability assessment (ARA). They found that these features are most beneficial for longer texts and when training datasets are smaller, suggesting that PLMs alone struggle with the nuances of extended content.

What Problem Does This Paper Address?

Automatic Readability Assessment (ARA) aims to objectively classify text difficulty, which is crucial for language learners and educators. Traditionally, this involved hand-crafted linguistic features. While modern approaches leverage powerful pre-trained language models (PLMs) like BERT, these models often struggle with the specific characteristics of longer texts used in ARA. PLMs have a limited input window (typically 510 tokens), meaning they might miss crucial information in longer passages. Furthermore, the sequential nature of leveled reading materials, where authors intentionally vary sentence structure and vocabulary to guide learners, can be difficult for PLMs to capture solely from raw text. This paper addresses the gap in understanding how linguistic features can complement PLMs to overcome these limitations, particularly concerning passage length and dataset size.

Methods

The core of this research involves integrating manually extracted linguistic features with PLMs. Specifically, they use RoBERTa-base as their PLM. They explore two ways of incorporating features:

  1. f_partial: Features extracted from the first 510 tokens of a passage, aligning with the PLM's input window. This aims to provide the PLM with the most relevant linguistic information within its processing limit.
  2. f_full: Features extracted from the entire passage. This approach aims to capture holistic textual characteristics that might be missed by f_partial or the PLM's limited window.

They compare the performance of PLMs with and without these features, as well as with traditional statistical models. They also analyze the impact of passage length (short vs. long, defined by a 510-token threshold) and dataset size on the effectiveness of these features.

Datasets

The study utilizes four datasets for evaluation:

  • Newsela: A text simplification dataset with 5 levels.
  • Weebit: An ARA dataset with texts focusing on different topics across readability levels.
  • OneStopEnglish: A smaller text simplification dataset.
  • RAZ: A newly constructed ARA dataset from RAZ-Kid's leveled books, noted for its longer average passage length and diverse text types.

Key Insights and Ideas

  • Long Passages are Key: Linguistic features significantly boost PLM performance on long passages (over 510 tokens), but have minimal impact on short ones. This suggests PLMs inherently struggle with the extended context and nuanced linguistic cues present in longer texts.
  • Dataset Size Matters: The benefit of linguistic features diminishes as the dataset size grows beyond approximately 750 passages. This implies that with larger datasets, PLMs can learn to infer more of the necessary information themselves, or that the features become less discriminative in a data-rich environment.
  • Newsela's Suitability Questioned: The Newsela dataset, while seemingly useful, shows less improvement with linguistic features. The authors suggest this might be because Newsela is primarily a text simplification dataset, where levels are relative and may not perfectly align with absolute readability, leading to class overlap and confusion.
  • Holistic vs. Partial Features: f_full (features from the entire passage) generally provides a greater performance boost than f_partial (features from the first 510 tokens), reinforcing the idea that information beyond the PLM's immediate window is valuable.

Key Results

  • On smaller datasets like RAZ and OneStop, incorporating linguistic features (f_partial and f_full) led to substantial accuracy improvements for PLMs, reaching up to 9% and 22% respectively.
  • Error analysis confirmed that these improvements were almost exclusively on passages longer than 510 tokens.
  • For larger datasets like Weebit, the impact of linguistic features was less pronounced, with performance gains being marginal or non-existent, likely due to the dataset's size and existing high performance.
  • The study found that for datasets exceeding ~750 passages, the promotion of features on PLMs becomes less significant.

Limitations and Future Work

  • Feature Simplicity: The authors acknowledge that the linguistic features used are relatively simple. They suggest that more sophisticated features could lead to even greater improvements, especially on larger datasets where simpler features might be less discriminative.
  • Dataset Suitability: The findings regarding Newsela raise questions about its suitability for pure ARA tasks, suggesting that datasets designed specifically for readability assessment (like RAZ) might be more appropriate for certain research questions.
  • Integration Methods: While they explored basic feature concatenation, future work could investigate more advanced integration methods between linguistic features and PLMs.
  • Text Simplification vs. ARA: The paper hints that while linguistic features help PLMs in ARA, their effectiveness might be limited in text simplification tasks due to the different underlying objectives and dataset construction methods.

Summary updated

BibTeX
@inproceedings{hou2022promoting,
  author = {Shudi Hou and Simin Rao and Yu Xia and Sujian Li},
  title = {{Promoting Pre-trained LM with Linguistic Features on Automatic Readability Assessment}},
  booktitle = {AACL/IJCNLP},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-short.54},
}
Figure