TL;DR
This paper explores the feasibility of a "universal CEFR classifier" for automated essay scoring (AES) across different languages. By leveraging language-agnostic features derived from Universal Dependencies, the authors demonstrate that multilingual and cross-lingual models can achieve performance comparable to monolingual models, suggesting a common underlying structure to language proficiency.
What Problem Does This Paper Address?
Automated Essay Scoring (AES) systems are valuable tools for grading student essays, but existing approaches are largely language-specific, primarily focusing on English. Developing AES systems for other languages often requires significant effort and language-specific feature engineering. The Common European Framework of Reference (CEFR) provides a language-independent scale for describing language proficiency. This paper addresses the gap by investigating whether a single, "universal" AES model can effectively classify essay proficiency across multiple languages, thereby simplifying the development of AES systems for languages with limited resources.
Methods
The authors propose a universal CEFR classification approach by developing a common feature representation that can generalize across languages. They utilize features derived from the Universal Dependencies (UD) project, specifically focusing on part-of-speech tags and dependency relations. This approach is combined with traditional AES features, including:
- N-grams: Word and POS n-grams, commonly used in AES.
- Dependency N-grams: A novel feature type where each unigram is a triplet of dependency relation, POS tag of the dependent, and POS tag of the head.
- Linguistic Features: Document length, lexical richness (density, variation, diversity), and error features (total errors, spelling errors) for German and Italian.
- Embeddings: Task-specific word and character embeddings.
They experiment with three main classification setups:
- Monolingual Classification: Training and testing models on data from a single language.
- Multilingual Classification: Training a single model on a combined dataset of all three languages.
- Cross-lingual Classification: Training a model on one language and testing it on another.
For classification, they compare logistic regression, random forests, multi-layer perceptrons, and support vector machines, with random forests often performing best.
Datasets
The primary dataset used is the MERLIN corpus, a freely available, multi-lingual corpus containing 2286 manually graded texts written by second language learners of German (DE), Italian (IT), and Czech (CZ). These texts are graded according to the CEFR scale (A1-C2). The authors filtered the corpus to remove language-CEFR category combinations with fewer than 10 examples and unrated texts, resulting in a final corpus of 2266 documents.
Key Insights and Ideas
- Evidence for a Universal Notion of Proficiency: The comparable performance of multilingual and cross-lingual models to monolingual models suggests that there might be a common underlying structure to language proficiency that can be captured by language-agnostic features.
- UD Features as Language-Agnostic Powerhouses: Features derived from Universal Dependencies (POS tags and dependency relations) prove to be effective for cross-lingual and multilingual classification, highlighting their potential for building universal NLP models.
- Domain Features Show Promise in Cross-Lingual Settings: While domain-specific features (lexical richness, error features) alone don't perform well in monolingual settings, they yield comparable results to monolingual models in cross-lingual settings for Italian, suggesting some universal patterns in language use related to proficiency.
- Multilingual Models Offer a Viable Alternative: Building a single multilingual classifier is a promising direction, achieving performance close to monolingual models for several feature groups.
Key Results
- Monolingual Performance: All tested feature sets generally outperform a simple document length baseline, with improvements of up to 25% in macro F1 score. Word n-grams and POS n-grams are strong predictors. Domain features alone are less effective but improve results when combined with n-grams. Word embeddings show varied performance across languages.
- Multilingual Performance: The multilingual classifier, trained on combined data, achieves performance comparable to monolingual models for POS n-grams, dependency n-grams, and embeddings. Document length baseline performs worse in this setup.
- Cross-lingual Performance: Training on German and testing on Italian and Czech shows a performance drop compared to monolingual models, as expected. However, the drop is less than 10%, indicating that the models still capture proficiency meaningfully. For Italian, domain features perform comparably to monolingual results. Confusion matrices reveal that most misclassifications occur between adjacent CEFR levels.
Limitations and Future Work
- Limitations:
- The cross-lingual classification performance drops, indicating that feature weights tuned for one language may not be optimal for another.
- German, with a larger dataset and more CEFR levels, performs poorer than Czech and Italian, possibly due to the increased complexity of the classification task.
- The lack of a Czech grammar checker prevented the extraction of error features for this language.
- The paper acknowledges that the current feature representations might not be sufficient to fully model German proficiency.
- Future Work:
- Systematic exploration of feature representations that are meaningful for AES and portable across languages.
- Modeling proficiency classification as a domain adaptation problem.
- Multi-task learning incorporating other annotation dimensions.
- Extending the approach to other languages with available CEFR-graded corpora (e.g., Estonian).
- Investigating the relationship between genetic/typological language similarities and cross/multi-lingual CEFR classification performance.
- Addressing concerns about model validity, bias, and fairness in real-world applications.
Summary updated