TL;DR
This paper introduces a unified training objective for automatic CEFR level assessment that combines Focal Loss and Supervised Contrastive Learning. By specifically addressing class imbalance and the fuzzy boundaries between adjacent proficiency levels, the authors achieve new state-of-the-art results across multiple languages and text granularities (sentences and documents).
What Problem Does This Paper Address?
The Common European Framework of Reference for Languages (CEFR) defines six levels of language proficiency, from A1 (beginner) to C2 (proficient). Automating this assessment is crucial for educational tech, but two major hurdles remain:
- Data Imbalance: Real-world datasets are skewed. There are far more intermediate-level texts (B1/B2) than beginner (A1) or expert (C2) examples. Standard models tend to ignore these "rare" classes to maximize overall accuracy.
- Ambiguous Boundaries: CEFR levels are ordinal and subjective. The difference between a B2 and a C1 sentence is often subtle, leading to "fuzzy" boundaries that make it difficult for models to distinguish between neighboring levels consistently.
Methods
The authors propose a two-stage training pipeline to tackle these issues:
- Stage 1: Supervised Contrastive Learning (SupCon): The model (a pretrained masked language model like BERT or XLM-R) is trained to pull embeddings of the same CEFR level closer together while pushing different levels apart. To handle the lack of data, they use a "dropout-as-augmentation" trick—encoding the same text twice with different dropout masks to create positive pairs.
- Stage 2: Focal Loss with Class Weighting: The model's classification head is trained using Focal Loss, which down-weights "easy" examples the model already understands and forces it to focus on "hard" examples (like rare C2 texts). They further refine this with a class-level weighting scheme to ensure the model doesn't become biased toward the majority classes.
Datasets
The researchers evaluated their approach on two primary benchmarks:
- CEFR-SP: An English sentence-level corpus where each sentence is labeled by two different annotators.
- UniversalCEFR: A large multilingual collection covering 13 languages, featuring both sentence-level and document-level granularities.
Key Insights and Ideas
- Emergent Ordinality: Even though the model wasn't explicitly told that A1 < A2 < B1, the Supervised Contrastive Learning objective naturally arranged the levels in an ordinal progression in the embedding space. You can see this "rainbow" effect in the t-SNE projections below, where levels flow logically from one to the next.
- The "One-Off" Error Pattern: The model's errors are almost exclusively between adjacent levels. Analysis showed that over 97% of predictions were within one level of the target. This suggests the model has a strong grasp of general difficulty but hits a ceiling created by the inherent subjectivity of human annotation.
- Feature Separation: The combination of losses led to a much more "class-separable" embedding space compared to standard cross-entropy, as measured by silhouette scores.
Key Results
- SOTA Performance: The proposed method consistently outperformed strong baselines (including previous SOTA "Prototype" models) across all settings.
- Macro-F1 Gains: On the CEFR-SP dataset, the method improved Macro-F1 by more than 12 points over standard cross-entropy training, largely due to its superior performance on the scarce A1 and C2 classes.
- Robustness: The model proved effective for both short sentences and long documents, and worked across 13 different languages using a single multilingual backbone (XLM-RoBERTa).
Limitations and Future Work
- Implicit vs. Explicit Ordinality: While the model learned an ordinal structure, the training objective itself still treats the task as a flat classification problem. The authors suggest that explicitly modeling the ordinal relationships (e.g., through ranking losses) could further improve results.
- Domain Generalization: The study focused on general language datasets. How these models perform in highly specialized domains, such as legal or medical texts where "difficulty" is defined by technical jargon rather than just linguistic complexity, remains an open question.
Summary updated