TL;DR
This paper introduces a novel "AI readability" formula for French as a Foreign Language (FFL) that significantly outperforms previous methods. By leveraging a large corpus and a diverse set of 406 linguistic features, including novel ones, their Support Vector Machine (SVM) model achieves state-of-the-art performance in predicting text difficulty according to the CEFR scale.
What Problem Does This Paper Address?
Assessing the readability of texts is crucial for effective language learning, ensuring that learners are presented with material appropriate for their skill level. While numerous readability formulas exist for English, the field for French, especially for foreign language learners (FFL), has been less developed. Existing French readability models often rely on simpler, classic features or are tailored to specific learner groups. This paper aims to bridge this gap by developing a more robust and generalizable FFL readability formula using advanced machine learning techniques and a comprehensive set of linguistic features.
Methods
The authors propose an "AI readability" approach, which involves three main steps:
- Corpus Creation: A gold-standard corpus of 1,852 French texts was compiled, labeled according to the Common European Framework of Reference for Languages (CEFR) levels (A1-C2). This corpus was derived from 28 FFL textbooks designed for adults and adolescents.
- Feature Engineering: A comprehensive set of 406 linguistic features was extracted from the texts, categorized into four families: lexical, syntactic, semantic, and specific to FFL. This included traditional features like word and sentence length, as well as novel features such as normalized Type-Token Ratio (TTR) and predictors based on orthographic neighborhoods.
- Model Development: Various machine learning algorithms, including logistic regression and Support Vector Machines (SVM), were employed. Feature selection was performed both manually (expert-driven) and automatically (stepwise selection and regularization). The best-performing models were then compared.
Datasets
The primary dataset used is a newly compiled corpus of 1,852 French texts extracted from 28 FFL textbooks. These texts were labeled with CEFR levels (A1-C2). This corpus serves as the gold-standard for training and evaluating the readability formula.
Key Insights and Ideas
- Semantic Features Underperform: Contrary to some findings in English readability studies, semantic features did not perform as well in this FFL context. Their contribution was limited, and dropping them did not significantly impact the performance of the best model.
- Lexical Dominance: Lexical features, particularly those related to word frequency and lists of easy words (like the PA-Alterego feature), proved to be the most predictive of text difficulty.
- "AI Readability" Advantage: The "AI readability" paradigm, utilizing a large number of texts, NLP-enabled features, and machine learning, significantly outperforms traditional methods.
- Novel Features Show Promise: Features like normalized TTR and orthographic neighborhood predictors, while not always the top performers individually, contributed to the overall model's robustness.
- Generic Model Effectiveness: The developed model, designed to be generic and not tied to a specific L1 background, shows strong performance, suggesting that a broad approach is viable for FFL readability.
Key Results
The paper reports several key quantitative findings:
- The best performing model, an SVM classifier using 46 features selected automatically (Auto-SVM), achieved an accuracy of 49.1% and an adjacent accuracy of 79.6%.
- This performance represents an 8% accuracy gain over the previous state-of-the-art FFL model (François, 2009), which achieved 41% accuracy and 72.7% adjacent accuracy.
- The SVM model also significantly outperformed a classic baseline (based on Flesch-like features) by 15% in accuracy.
- The lexical family of features was the most accurate, contributing 40.5% accuracy and 75.6% adjacent accuracy when used alone.
- The model demonstrated higher accuracy for texts at the extreme ends of the difficulty scale (A1, C1, C2), with 100% adjacent accuracy for A1 texts.
Limitations and Future Work
- Semantic Feature Concerns: The authors express reservations about the efficiency of semantic features, suggesting that their information might be correlated with lexico-syntactic features, especially in an L2 context where learners initially struggle more with vocabulary and syntax. Further research is needed to clarify this.
- L1 Specificity: The paper acknowledges that developing specific-to-FFL predictors that account for the impact of a learner's native language (L1) could be a future avenue. It remains to be seen if such specialized models would offer a significant performance boost over a generic model.
- Corpus Heterogeneity: While efforts were made to address heterogeneity in publisher labeling, the authors note that differences in publishers' conceptions of difficulty can still lead to variations.
- Future Work: The authors suggest exploring the impact of L1-specific features and further investigating the relationship between semantic and lexico-syntactic features.
Summary updated