TL;DR
This paper introduces a novel unsupervised method for estimating sentence readability by leveraging parallel corpora from text simplification tasks. By training a model to predict which of two sentences is simpler, they achieve performance comparable to supervised methods, even without relying on costly absolute readability annotations. This approach opens doors for more accessible and multilingual readability assessment tools.
What Problem Does This Paper Address?
Estimating text readability is crucial for applications like text simplification and recommendation, aiming to tailor content to readers of varying abilities. However, creating datasets with absolute sentence readability scores (e.g., assigning a specific difficulty level to each sentence) is expensive and time-consuming, limiting the availability of such data, especially for languages other than English. This scarcity hinders the development of high-quality supervised readability estimation models. While unsupervised methods exist, they often focus on documents rather than individual sentences or rely on traditional readability formulas that may not capture nuanced linguistic complexity. This paper tackles the challenge of building effective sentence readability estimators in a low-resource setting by utilizing readily available parallel corpora from text simplification.
Methods
The core of the proposed method is a relative sentence readability estimator. This estimator is trained using fine-tuned pre-trained masked language models (like BERT). The training data consists of sentence pairs from parallel corpora used for text simplification. For each pair, the model is presented with a "complex sentence" and a "simple sentence" (with their positions sometimes swapped to ensure robustness). The model learns to predict which of the two sentences is more readable.
Once trained, this estimator is used for readability ranking. For a given set of sentences, the model performs pairwise comparisons between all possible sentence pairs. The relative readability of each pair is estimated, and this information is aggregated to derive a final ranking of the sentences from most to least readable. This pairwise comparison approach is key to their unsupervised strategy.
Datasets
- Newsela: A parallel corpus for text simplification, used for training and validation. It consists of English news articles simplified into multiple levels. For this study, 385,270 sentence pairs were used for training and 42,323 for validation. A set of 43,171 pairs was used for evaluation.
- CEFR-SP: An English corpus with absolute sentence readability labels (CEFR levels). This dataset is non-parallel and contains non-synonymous sentences. It was used to evaluate the ranking performance on a different type of data. A set of 17,676 sentences was used, with one sentence randomly selected from each of the six CEFR levels.
Key Insights and Ideas
- Leveraging Text Simplification Corpora: The paper demonstrates that parallel corpora designed for text simplification, which inherently contain pairs of complex and simple sentences, are a valuable resource for training readability estimators, even without explicit readability labels.
- Relative vs. Absolute Readability: The success of their relative approach highlights that understanding the difference in readability between two sentences can be more achievable and data-efficient than assigning an absolute score to each sentence individually.
- Pairwise Comparison Power: The pairwise comparison method for ranking is effective because it breaks down the complex task of ranking a set into a series of simpler binary decisions, which the trained estimator can handle well.
- Impact of Readability Difference: The analysis shows that the larger the difference in readability between sentence pairs, the easier it is for the model to correctly identify the simpler sentence, leading to higher accuracy.
Key Results
- Superior Unsupervised Performance: The proposed unsupervised method consistently outperformed existing unsupervised methods (RSRS and LLM-based) on both synonymous (Newsela) and non-synonymous (CEFR-SP) sentence sets, achieving the highest NDCG scores.
- Comparable to Supervised Methods: The unsupervised method achieved performance comparable to supervised methods, particularly outperforming the supervised pointwise method and being on par with the supervised pairwise method.
- Effectiveness of Pairwise Supervised Methods: The results suggest that supervised methods that also use pairwise comparisons (like their "Pairwise" baseline) tend to perform better than those that estimate individual sentence scores (like "Pointwise").
- Training Data Size: The study indicates that around 5,000 sentence pairs from text simplification corpora are sufficient to achieve performance comparable to supervised methods.
Limitations and Future Work
- English-Centric Experiments: The current experiments are limited to English. The authors acknowledge that performance in other languages is not guaranteed and that the scarcity of annotated corpora for non-English languages remains a significant challenge.
- Multilingual Potential: Despite the current limitations, the method is designed with multilingual applications in mind, and the availability of text simplification parallel corpora in multiple languages makes this a promising direction.
- Cost of Annotation: The paper reiterates that annotating corpora with absolute sentence readability is expensive, reinforcing the value of their unsupervised approach.
Summary updated