TL;DR
This paper investigates whether expert-assigned text readability scores and reader proficiency can predict actual reader comprehension. They found that neither factor alone, nor in combination, reliably predicts comprehension, suggesting that current readability assessment methods might miss crucial linguistic nuances. The study also introduces a new dataset of reader responses to benchmark readability assessment systems.
What Problem Does This Paper Address?
Ensuring that readers can comprehend texts is a fundamental goal in education and information dissemination. Readability assessment aims to predict how easy a text is to understand. However, most computational readability models are trained on texts annotated by experts (like teachers) based on their perception of difficulty, rather than on actual reader comprehension data. This disconnect raises questions about the validity of these expert annotations for predicting how real readers, especially those learning a new language, will actually understand a text. While there's research on creating comprehension questions, few studies have explicitly linked expert readability scores, reader proficiency, and comprehension outcomes, especially for adult English as a Second Language (ESL) learners. This paper aims to bridge this gap by exploring this relationship and creating a valuable dataset for future research.
Methods
The researchers conducted a web-based study involving 112 non-native English speakers. Participants read six newspaper articles, each presented in three versions tailored for beginner, intermediate, and advanced ESL learners. After reading each text, they answered six comprehension questions designed to test three levels: literal (direct information), re-organization (combining information), and inferential (drawing conclusions). These questions were carefully crafted by an experienced language instructor. After completing the reading and comprehension tasks, participants took a free English language proficiency test.
The analysis involved two main approaches:
- Regression Analysis: Initial models explored how well reader proficiency and text reading level (as assigned by experts) could predict overall comprehension scores.
- Facet Rasch Measurement (MFRM) Models: These psychometric models were used to analyze the data from multiple perspectives (facets). They examined:
- The difficulty of different comprehension question types (literal, re-organization, inferential).
- The difficulty of different question forms (True/False vs. short answer).
- The interaction between question form and comprehension type.
Datasets
The study utilized 15 texts from the OneStopEnglish corpus, which consists of manually simplified news articles from The Guardian, adapted for different ESL proficiency levels.
A key contribution of this paper is the release of a new dataset containing reader responses from their user study. This dataset includes 45 reader-benchmarked texts and associated comprehension scores, which the authors state is the first freely available dataset of its kind for evaluating readability assessment systems.
Key Insights and Ideas
- Inferential Questions are the Real Hurdle: The study found that the type of comprehension question asked had a much larger impact on reader responses than either the expert-assigned readability level of the text or the reader's proficiency. Inferential questions proved significantly more difficult for participants across all reading levels.
- Expert Annotations vs. Reader Reality: The initial regression models showed that neither expert-assigned reading level nor reader proficiency alone could reliably predict comprehension scores. This suggests a potential disconnect between how experts label text difficulty and how actual readers experience that difficulty.
- Comprehension Types Matter More Than Reading Levels: The MFRM analysis indicated that the distinction between factual comprehension (literal and re-organization) and inferential comprehension was more significant than differences between the expert-assigned reading levels of the texts.
- Question Format Influences Difficulty: Short-answer questions were generally found to be more difficult than True/False questions, regardless of the comprehension level being tested.
Key Results
- Low Predictive Power of Readability/Proficiency: Regression models explained only 5.3% of the variance in participants' total comprehension scores, with proficiency being a statistically significant but weak predictor. This suggests these factors are insufficient for building practical predictive models for text recommendation.
- Inferential Questions are Hardest: MFRM analysis showed inferential comprehension questions were significantly harder than literal and re-organization questions.
- Short Answer Questions are More Difficult: MFRM also revealed that short-answer questions were more difficult than True/False questions.
- Dataset Release: The paper makes available a new dataset of reader responses to benchmark readability assessment systems.
Limitations and Future Work
The authors acknowledge several limitations:
- Controlled Environment: The web-based study, while more natural than a lab setting, couldn't definitively confirm if participants actually read the texts.
- Background Knowledge Ignored: The study did not account for how readers' prior background knowledge might have influenced their comprehension.
- Text Variation: The study did not specifically analyze the variation within texts of the same readability level.
- Generalizability: The results might be specific to the texts, proficiency test, and question types used.
Future work suggested includes:
- Linguistic Feature Analysis: Investigating specific linguistic properties (e.g., lexical richness, syntactic complexity) that might correlate with comprehension, rather than relying solely on broad readability labels.
- Broader Comprehension Measures: Exploring more than one form of proficiency and comprehension.
- Content Simplification Focus: Conducting similar studies in scenarios specifically focused on content simplification, not just readability adjustments.
- More Datasets: Encouraging more studies to build diverse reader-response datasets to better validate readability assessment models.
- Automated Question Generation: Developing methods to automatically generate questions that target multiple comprehension levels, as this is crucial for educational applications.
Summary updated