BERT_4EVER at LangLearn: Language Development Assessment Model based on Sequential Information Attention Mechanism

Hongyan Wu, Nankai Lin, Shengyi Jiang, Lixian XiaoEVALITA 2023

Abstract

Tracking the linguistic development of students is a complex task that requires identifying subtle changes in writing style and competence over time. This paper introduces BERT_4EVER, a model designed for the relative chronological ordering of essays. By implementing a novel Sequential Information Attention Mechanism (SIAM) on top of a BERT backbone, the model effectively captures the "dynamic evolution" of a learner's language skills. Our results on the EVALITA 2023 challenge datasets demonstrate that SIAM significantly improves performance, particularly in second-language (L2) acquisition contexts, outperforming standard transformer models.

1. Introduction

The assessment of language development, whether in a first (L1) or second (L2) language, has traditionally relied on manual evaluation by linguistic experts. The "Language Learning Development" task at EVALITA 2023 formalizes this as a relative chronological ordering problem: given two essays by the same student, can a system determine which was written first? This task necessitates models that can look beyond static grammatical correctness to identify the trajectory of linguistic growth.

2. Methodology

The BERT_4EVER system utilizes the bert-base-italian-uncased model as its foundation. The core contribution is the Sequential Information Attention Mechanism (SIAM), which facilitates interaction between different time points in a learner's history.

2.1 Sequential Information Attention Mechanism (SIAM)

The model processes a pair of essays—a "historical" text ($T^a$) and a "current" text ($T^b$)—by concatenating them into a single sequence. While BERT provides initial semantic embeddings, SIAM calculates attention weights between the two texts. This allows the model to identify specific linguistic differences and progress markers by comparing the current state against the historical baseline.

Architecture of the BERT_4EVER model featuring the Sequential Information Attention Mechanism (SIAM).

As illustrated in Figure 1, the final prediction is based on a combination of the standard [CLS] token, the updated SIAM representations ($H^a_{new}$), and a global representation ($h^g$). This ensemble of features is passed to a linear classification layer to predict the chronological order.

2.2 Data Augmentation

To address data scarcity, the authors employed transitive data augmentation. By leveraging known orderings (e.g., if A < B and B < C, then A < C), the training set was expanded to provide the model with more diverse comparison pairs.

3. Experimental Setup

The model was evaluated on two distinct corpora:

  • CItA: L1 Italian learners.
  • COWS-L2H: L2 Spanish learners.

The researchers used an Italian-pretrained BERT for both tasks, finding that the linguistic similarities between Italian and Spanish allowed for effective cross-lingual transfer.

4. Results and Discussion

The experimental results highlight the efficacy of the SIAM module, especially in the more challenging L2 context.

Performance comparison of BERT baseline vs. BERT with SIAM across different datasets.

As shown in Figure 2, the BERT with SIAM model (Run 3) achieved an accuracy of 0.6406 on the COWS (L2) dataset, a significant improvement over the 0.6094 achieved by the baseline BERT (Run 2). This corresponds to a 4.27 percentage point improvement in binary F1-score, demonstrating the value of modeling the relationship between essays.

4.1 Joint Training Trade-offs

The study observed that training on both L1 and L2 datasets simultaneously improved performance on the harder COWS dataset but led to a slight decrease in accuracy on the CItA (L1) dataset (from 0.9316 to 0.9251). This suggests that while joint training increases general robustness, it may introduce noise that affects specialized performance in high-accuracy domains.

5. Conclusion

BERT_4EVER demonstrates that a dedicated attention mechanism for sequential information is more effective for tracking language development than standard model fusion or isolated text analysis. Future research will explore the integration of neural language features—explicit linguistic markers of competence—to enhance the interpretability and accuracy of the model's assessments.

Summary updated

BibTeX
@inproceedings{wu2023bert,
  author = {Hongyan Wu and Nankai Lin and Shengyi Jiang and Lixian Xiao},
  title = {{BERT\_4EVER at LangLearn: Language Development Assessment Model based on Sequential Information Attention Mechanism}},
  booktitle = {EVALITA},
  year = {2023},
}
Figure