LXPER Index 2.0: Improving Text Readability Assessment for L2 English Learners in South Korea

Bruce W. Lee, Jason Hyung-Jong LeearXiv 2020 · OpenReview

TL;DR

This paper introduces LXPER Index 2.0, a significantly improved text readability assessment model specifically designed for L2 English learners in South Korea. By expanding a curriculum-specific corpus and employing a logistic regression model with novel features, they achieve a lower average error than previous methods, making it more reliable for educational applications.

What Problem Does This Paper Address?

Assessing the readability of English texts for non-native speakers, particularly within structured English Language Training (ELT) curricula, has been a persistent challenge. Existing readability models, often developed for native English speakers, exhibit low accuracy when applied to L2 learners' texts. This is especially problematic in countries like South Korea, where English education plays a crucial role, and students follow specific national curricula. The lack of accurate readability assessment tools hinders the selection of appropriate learning materials and can impede educational progress. This paper aims to bridge this gap by developing a more accurate and curriculum-aware readability assessment model for L2 English learners.

Methods

The authors propose LXPER Index 2.0, an enhanced readability assessment model. Their approach involves:

  • Corpus Expansion: They expanded the existing CoKEC-text corpus, which contains texts from the Korean ELT curriculum, by adding approximately 900 more texts from MOE-approved middle school textbooks. This expansion aimed to address a skew towards higher grade levels in the original corpus and ensure better representation of lower grades (K7-K9). The final training corpus consists of 3700 texts.
  • Feature Engineering: They explored a comprehensive set of 35 linguistic features, categorized into:
  • Traditional Features: Standard features like average words per sentence and syllables per word.
  • POS-based Features: Features related to the density and types of Part-of-Speech tags (e.g., noun phrases, verb phrases).
  • Entity Density Features: Features capturing the density of named entities and general nouns, hypothesized to relate to working memory load.
  • Lexical Chain Features: Features that track how entities and semantic relationships are connected throughout a text.
  • Word Difficulty Features: Features based on a vocabulary corpus specifically curated from the Korean ELT curriculum, classifying words into 6 difficulty levels.
  • Model Selection: Crucially, they moved from traditional linear regression models to a logistic regression model. They hypothesize that text difficulty might not have a linear correlation with grade levels, making logistic regression a more suitable choice for classification.
  • Parsing and Counting: They utilized a combination of NLP tools, including spaCy, NLTK, and Gensim, along with the Berkeley Neural Parser, for feature extraction.

Datasets

  • CoKEC-text (Expanded Version): This is the primary dataset used for training the LXPER Index 2.0 model. It comprises 3700 unique, grade-labeled texts from the Korean ELT curriculum, including official mock tests and middle school textbooks. The texts are labeled with target grade levels from K7 to K12.5.
  • Test Corpus: A separate test corpus was prepared to evaluate the model's effectiveness. This corpus consists of 270 texts from official mock tests (K10-K12) and 270 texts from government-approved middle school textbooks (K7-K9).

Key Insights and Ideas

  • Curriculum-Specific is Key: The paper strongly reinforces the idea that readability assessment for L2 learners must be tailored to the specific curriculum they follow. Generic features are insufficient.
  • Logistic Regression Outperforms Linear: The shift to logistic regression for grade classification proved to be a significant improvement, suggesting that the relationship between text features and grade level is not strictly linear.
  • Corpus Quality Matters: Expanding and meticulously cleaning the CoKEC-text corpus, especially by addressing the underrepresentation of lower grades and removing noisy data, was critical for improving accuracy.
  • Novel Features Show Promise: While traditional features are still relevant, the exploration of entity density and curriculum-specific word difficulty features contributes to the model's enhanced performance.
  • Small, Well-Labeled Data Can Be Powerful: The research suggests that a well-optimized model trained on a relatively small, but highly relevant and accurately labeled, curriculum-specific corpus can achieve high accuracy.

Key Results

The LXPER Index 2.0 model achieved a target average assessment error of less than 0.5 grade levels. This is a significant improvement compared to previous models, which often showed very low accuracy for L2 English texts. The paper presents a table (Table 4) showing the average error for LXPER Index 2.0 across different grade levels (K7-K12), consistently outperforming traditional models like Flesch-Kincaid and Dale-Chall, as well as the previous LXPER Index 1.0.

Limitations and Future Work

  • Geographic Limitation: The current model is specifically tuned for the Korean ELT curriculum. Applying it to other East Asian L2 ELT curricula (e.g., Japan, China) is not directly possible due to the lack of comparable openly available corpora.
  • Corpus Dependency: The model's effectiveness is highly dependent on the availability of pre-processed and labeled corpora like CoKEC. Constructing such corpora is a prerequisite for applying this methodology elsewhere.
  • Future Work: The authors suggest that future research could focus on constructing similar corpora for other countries to extend the applicability of their model. They also note that while the logistic regression model is effective, further thought could be given to the "grades" classification task.

Summary updated

Abstract

Developing a text readability assessment model specifically for texts in a foreign English Language Training (ELT) curriculum has never had much attention in the field of Natural Language Processing. Hence, most developed models show extremely low accuracy for L2 English texts, up to the point where not many even serve as a fair comparison. In this paper, we investigate a text readability assessment model for L2 English learners in Korea. In accordance, we improve and expand the Text Corpus of the Korean ELT curriculum (CoKEC-text). Each text is labeled with its target grade level. We train our model with CoKEC-text and significantly improve the accuracy of readability assessment for texts in the Korean ELT curriculum.
BibTeX
@misc{lee2020lxper,
  author = {Bruce W. Lee and Jason Hyung-Jong Lee},
  title = {{LXPER Index 2.0: Improving Text Readability Assessment for L2 English Learners in South Korea}},
  year = {2020},
  eprint = {2010.13374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.13374},
}
Figure