TL;DR
This paper introduces the OneStopEnglish corpus, a novel, freely available dataset of English texts rewritten at three distinct reading levels (beginner, intermediate, advanced) for ESL learners. This corpus aims to facilitate research in automatic readability assessment and text simplification by providing comparable texts across levels, addressing a key limitation in existing resources.
What Problem Does This Paper Address?
Assessing and simplifying text for different reading levels are crucial tasks in natural language processing, particularly for educational applications and making information accessible. However, existing corpora for these tasks often lack parallel texts—meaning texts on the same topic but rewritten for different difficulty levels. This makes it hard to develop fine-grained models that can pinpoint why a text is difficult or to train text simplification systems that produce truly comparable outputs. Current approaches often rely on generic texts or are not publicly available, limiting research progress and the generalizability of developed models.
Methods
The authors created the OneStopEnglish corpus by leveraging content from the onestopenglish.com website, which features news articles adapted by teachers for adult ESL learners at elementary, intermediate, and advanced levels. The core methodological contribution is the careful compilation and cleaning of these texts, ensuring comparability across the three levels.
For readability assessment, they experimented with two sets of features:
- Generic text classification features: These included word n-grams, POS n-grams, character n-grams, syntactic production rules, and dependency relations.
- Specific readability features: These encompassed traditional readability formulas (like Flesch-Kincaid), lexical variation, psycholinguistic features, syntactic features, and discourse features.
For text simplification, they treated the task as a phrase-based machine translation (PBMT) problem, using the aligned sentence pairs from the corpus to train a model.
Datasets
The primary dataset introduced is the OneStopEnglish corpus. It consists of 189 original texts, each rewritten into three versions, totaling 567 texts. The corpus is aligned at both text and sentence levels across beginner, intermediate, and advanced reading levels, specifically targeting ESL learners. This corpus is made freely available under a CC BY-SA 4.0 license.
The authors also mention using the Wikipedia-Simple Wikipedia corpus as a benchmark for text simplification, noting its limitations for evaluating generalizability.
Key Insights and Ideas
- The importance of comparable corpora: The paper strongly advocates for the necessity of parallel texts across reading levels to build robust readability assessment and text simplification models. The OneStopEnglish corpus directly addresses this gap.
- Character n-grams show promise for readability: Surprisingly, character n-grams emerged as the best-performing generic feature for readability classification, outperforming more complex linguistic features like dependency relations. The authors speculate this might be due to their ability to capture sub-word patterns like suffixes and prefixes, which could be indicative of simplification.
- Generic features can be competitive: While specialized readability features are important, the study shows that well-chosen generic features (like character n-grams) can achieve competitive accuracy in readability classification.
- PBMT for text simplification: Treating text simplification as a phrase-based machine translation task, even with a relatively small dataset, yielded promising results, demonstrating the utility of the corpus for this application.
Key Results
Readability Assessment: * Character n-grams achieved the highest accuracy among generic features at 77.25%. * Combining all features (generic and specific) yielded the highest overall accuracy at 78.13%. * Dependency relations performed poorly, even below the random baseline.
Text Simplification: * Using a phrase-based machine translation approach (Moses) on the advanced-to-elementary aligned sentences, the model achieved a BLEU score of 54.45 and a METEOR score of 46. While not directly comparable to state-of-the-art neural models trained on much larger datasets, these scores suggest understandable translations according to general guidelines.
Limitations and Future Work
Limitations: * The authors acknowledge that the dataset, while valuable, might not be sufficient on its own to train robust text simplification models, especially compared to much larger datasets used in prior work. * The PBMT model used for simplification was not a neural model, which is the current state-of-the-art.
Future Work: * Exploring readability assessment as a sentence/paragraph level pair-wise ranking problem, potentially leading to more nuanced readability models. * Further investigating the effectiveness of character n-grams for readability assessment. * Using the corpus to test the generalizability of state-of-the-art text simplification approaches. * Combining the OneStopEnglish corpus with larger datasets for training more powerful simplification models. * Conducting user studies to explore the relationship between text simplification and reader comprehension.
Summary updated