TL;DR
This paper tackles the challenge of accurately classifying the Common European Framework of Reference for Languages (CEFR) levels in second language (L2) writing by addressing "learnerese" – language influenced by a writer's native tongue (L1). They propose and evaluate a debiasing technique to remove L1-specific features from text representations, leading to a marginal improvement in CEFR classification accuracy for linear models.
What Problem Does This Paper Address?
Automated assessment of L2 writing, particularly for CEFR level classification, is crucial for providing feedback to language learners. However, a significant hurdle is the heterogeneity introduced by learners' native languages (L1s). Texts written by learners from different L1 backgrounds can exhibit distinct patterns, often referred to as "learnerese," which can confuse classification models trained on mixed data. This L1 influence can lead to models performing poorly when generalizing across different L1 groups. The authors aim to mitigate this by creating a more homogeneous representation of L2 text, devoid of L1-specific signals, to improve the robustness and accuracy of CEFR classifiers.
Methods
The core of the paper's methodology revolves around iterative null-space projection (INLP), a debiasing technique adapted from prior work on mitigating gender bias in word embeddings. The authors apply INLP to BERT document-level representations. The process involves training a series of probing classifiers to predict the author's country of origin (as a proxy for L1 influence). The INLP algorithm then iteratively removes the directions in the embedding space that these probing classifiers exploit, effectively "debiasing" the representations. This debiased representation is then used to train CEFR classifiers. They experiment with both linear (Logistic Regression) and non-linear (MLP) classifiers.
Datasets
The authors utilize two primary datasets:
- EF-Cambridge Open Language Database (EFCAMDAT): A large dataset of L2 English texts from learners originating from China, Japan, and Korea. This dataset was chosen for its size and relative balance across L1 groups.
- International Corpus Network of Asian Learners of English (ICNALE): Used for initial experiments and to obtain native English (EN) texts for debiasing directions.
- MERLIN corpus (MERLIN_DE subset): A smaller dataset specifically for German L2 learners, used to test the debiasing approach on a different L1 group and to highlight potential limitations.
Key Insights and Ideas
- L1 influence is a significant factor in CEFR classification: The paper strongly suggests that L1 background introduces systematic biases ("learnerese") that negatively impact the generalization of CEFR classifiers.
- Debiasing can improve linear models: The INLP approach successfully removes L1-specific signals, leading to a measurable performance gain for linear CEFR classifiers.
- Non-linear models are less affected by debiasing: The MLP classifier showed no improvement after debiasing, suggesting that non-linear models might be more robust to L1 influence or that the debiasing method is primarily effective at removing linearly separable biases.
- Data size and balance are critical: The debiasing approach showed poorer results on the smaller, more imbalanced MERLIN_DE dataset, highlighting the need for sufficient data to effectively learn and remove L1-specific directions.
- Text length as a confounding factor: The paper acknowledges that text length correlates strongly with CEFR level, which could be a significant signal for classifiers that the debiasing method doesn't directly address.
Key Results
- For the EFCAMDAT_NATIVE_EN dataset, debiasing resulted in a 1-point performance improvement for the Linear Regression (LR) classifier in CEFR classification. The MLP classifier's performance remained unchanged.
- For the MERLIN_DE dataset, both LR and MLP classifiers showed a drop in performance after debiasing.
- Country of origin classification accuracy dropped significantly after debiasing, from near-perfect to random performance, indicating the effectiveness of INLP in removing L1-specific signals.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Limited improvement for non-linear models: The lack of improvement for MLP classifiers suggests that INLP might not fully capture complex L1 influences or that these influences manifest in ways not easily removed by linear projection.
- Data scarcity: The effectiveness of INLP appears to be highly dependent on the amount of training data. Smaller datasets, like MERLIN_DE, do not yield performance gains and can even lead to degradation.
- Text length correlation: The strong correlation between text length and CEFR level is identified as a potential confounding factor that was not explicitly addressed.
- Future work:
- Experimenting with other debiasing approaches.
- Curating data for language families rather than individual languages for debiasing.
- Investigating combined debiasing approaches for language families.
- Developing more sophisticated techniques to compensate for differences in text length.
Summary updated