CEFR Level Assessment across Languages and Text Granularities

Eugene Matsumura, Yuki ArasearXiv 2026 · arXiv

TL;DR

This paper introduces a unified training objective for automatic CEFR level assessment that combines Focal Loss and Supervised Contrastive Learning. By specifically addressing class imbalance and the fuzzy boundaries between adjacent proficiency levels, the authors achieve new state-of-the-art results across multiple languages and text granularities (sentences and documents).

What Problem Does This Paper Address?

The Common European Framework of Reference for Languages (CEFR) defines six levels of language proficiency, from A1 (beginner) to C2 (proficient). Automating this assessment is crucial for educational tech, but two major hurdles remain:

  1. Data Imbalance: Real-world datasets are skewed. There are far more intermediate-level texts (B1/B2) than beginner (A1) or expert (C2) examples. Standard models tend to ignore these "rare" classes to maximize overall accuracy.
  2. Ambiguous Boundaries: CEFR levels are ordinal and subjective. The difference between a B2 and a C1 sentence is often subtle, leading to "fuzzy" boundaries that make it difficult for models to distinguish between neighboring levels consistently.

Methods

The authors propose a two-stage training pipeline to tackle these issues:

  • Stage 1: Supervised Contrastive Learning (SupCon): The model (a pretrained masked language model like BERT or XLM-R) is trained to pull embeddings of the same CEFR level closer together while pushing different levels apart. To handle the lack of data, they use a "dropout-as-augmentation" trick—encoding the same text twice with different dropout masks to create positive pairs.
  • Stage 2: Focal Loss with Class Weighting: The model's classification head is trained using Focal Loss, which down-weights "easy" examples the model already understands and forces it to focus on "hard" examples (like rare C2 texts). They further refine this with a class-level weighting scheme to ensure the model doesn't become biased toward the majority classes.

Datasets

The researchers evaluated their approach on two primary benchmarks:

  • CEFR-SP: An English sentence-level corpus where each sentence is labeled by two different annotators.
  • UniversalCEFR: A large multilingual collection covering 13 languages, featuring both sentence-level and document-level granularities.
Dataset statistics showing the number of samples across splits

Key Insights and Ideas

  • Emergent Ordinality: Even though the model wasn't explicitly told that A1 < A2 < B1, the Supervised Contrastive Learning objective naturally arranged the levels in an ordinal progression in the embedding space. You can see this "rainbow" effect in the t-SNE projections below, where levels flow logically from one to the next.
  • The "One-Off" Error Pattern: The model's errors are almost exclusively between adjacent levels. Analysis showed that over 97% of predictions were within one level of the target. This suggests the model has a strong grasp of general difficulty but hits a ceiling created by the inherent subjectivity of human annotation.
  • Feature Separation: The combination of losses led to a much more "class-separable" embedding space compared to standard cross-entropy, as measured by silhouette scores.
Figure 2: t-SNE projections of the test-set embeddings for each method (columns), colored by gold CEFR level. Each panel shows a single training seed: the 6th-best of 12 seeds by test macro-F1 for CEFR-SP and the 2nd-best of 3 for UniversalCEFR. Panels are independent projections, so distances are not comparable across panels.

Key Results

  • SOTA Performance: The proposed method consistently outperformed strong baselines (including previous SOTA "Prototype" models) across all settings.
  • Macro-F1 Gains: On the CEFR-SP dataset, the method improved Macro-F1 by more than 12 points over standard cross-entropy training, largely due to its superior performance on the scarce A1 and C2 classes.
  • Robustness: The model proved effective for both short sentences and long documents, and worked across 13 different languages using a single multilingual backbone (XLM-RoBERTa).
Figure 1: Confusion matrices of our model on the CEFR-SP test set against each annotator’s labels. Cells show row-normalised recall, averaged over 12 training seeds discarding the best- and worst-performing seed by macro-F1.

Limitations and Future Work

  • Implicit vs. Explicit Ordinality: While the model learned an ordinal structure, the training objective itself still treats the task as a flat classification problem. The authors suggest that explicitly modeling the ordinal relationships (e.g., through ranking losses) could further improve results.
  • Domain Generalization: The study focused on general language datasets. How these models perform in highly specialized domains, such as legal or medical texts where "difficulty" is defined by technical jargon rather than just linguistic complexity, remains an open question.

Summary updated

Abstract

Automatic assessment of textual CEFR levels is an important task in educational natural language processing. However, accurate CEFR level prediction remains challenging due to the naturally imbalanced distribution of CEFR levels and the ambiguous boundaries between adjacent levels. To address these challenges, we propose a unified training objective that combines focal loss with supervised contrastive learning. Focal loss mitigates class imbalance by emphasizing underrepresented and difficult examples, while supervised contrastive learning learns more discriminative representations by encouraging intra-class compactness and inter-class separation. We evaluate the proposed method on sentence- and document-level CEFR assessment in both monolingual and multilingual settings. Experimental results demonstrate that our method consistently outperforms strong baselines across all settings, establishing new state-of-the-art performance on multiple CEFR assessment benchmarks.
BibTeX
@misc{matsumura2026cefr,
  author = {Eugene Matsumura and Yuki Arase},
  title = {{CEFR Level Assessment across Languages and Text Granularities}},
  year = {2026},
  eprint = {2610.41553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41553},
}
Figure