Abstract
Tracking the linguistic development of students is a complex task that requires identifying subtle changes in writing style and competence over time. This paper introduces BERT_4EVER, a model designed for the relative chronological ordering of essays. By implementing a novel Sequential Information Attention Mechanism (SIAM) on top of a BERT backbone, the model effectively captures the "dynamic evolution" of a learner's language skills. Our results on the EVALITA 2023 challenge datasets demonstrate that SIAM significantly improves performance, particularly in second-language (L2) acquisition contexts, outperforming standard transformer models.
1. Introduction
The assessment of language development, whether in a first (L1) or second (L2) language, has traditionally relied on manual evaluation by linguistic experts. The "Language Learning Development" task at EVALITA 2023 formalizes this as a relative chronological ordering problem: given two essays by the same student, can a system determine which was written first? This task necessitates models that can look beyond static grammatical correctness to identify the trajectory of linguistic growth.
2. Methodology
The BERT_4EVER system utilizes the bert-base-italian-uncased model as its foundation. The core contribution is the Sequential Information Attention Mechanism (SIAM), which facilitates interaction between different time points in a learner's history.
2.1 Sequential Information Attention Mechanism (SIAM)
The model processes a pair of essays—a "historical" text ($T^a$) and a "current" text ($T^b$)—by concatenating them into a single sequence. While BERT provides initial semantic embeddings, SIAM calculates attention weights between the two texts. This allows the model to identify specific linguistic differences and progress markers by comparing the current state against the historical baseline.
As illustrated in Figure 1, the final prediction is based on a combination of the standard [CLS] token, the updated SIAM representations ($H^a_{new}$), and a global representation ($h^g$). This ensemble of features is passed to a linear classification layer to predict the chronological order.
2.2 Data Augmentation
To address data scarcity, the authors employed transitive data augmentation. By leveraging known orderings (e.g., if A < B and B < C, then A < C), the training set was expanded to provide the model with more diverse comparison pairs.
3. Experimental Setup
The model was evaluated on two distinct corpora:
- CItA: L1 Italian learners.
- COWS-L2H: L2 Spanish learners.
The researchers used an Italian-pretrained BERT for both tasks, finding that the linguistic similarities between Italian and Spanish allowed for effective cross-lingual transfer.
4. Results and Discussion
The experimental results highlight the efficacy of the SIAM module, especially in the more challenging L2 context.
As shown in Figure 2, the BERT with SIAM model (Run 3) achieved an accuracy of 0.6406 on the COWS (L2) dataset, a significant improvement over the 0.6094 achieved by the baseline BERT (Run 2). This corresponds to a 4.27 percentage point improvement in binary F1-score, demonstrating the value of modeling the relationship between essays.
4.1 Joint Training Trade-offs
The study observed that training on both L1 and L2 datasets simultaneously improved performance on the harder COWS dataset but led to a slight decrease in accuracy on the CItA (L1) dataset (from 0.9316 to 0.9251). This suggests that while joint training increases general robustness, it may introduce noise that affects specialized performance in high-accuracy domains.
5. Conclusion
BERT_4EVER demonstrates that a dedicated attention mechanism for sequential information is more effective for tracking language development than standard model fusion or isolated text analysis. Future research will explore the integration of neural language features—explicit linguistic markers of competence—to enhance the interpretability and accuracy of the model's assessments.
Summary updated