Transfer Learning of Argument Mining in Student Essays

Yuning Ding, Julian Lohmann, Nils-Jonathan Schaller, Thorben Jansen, Andrea HorbachNAACL 2024

TL;DR

This paper investigates how well argument mining models, trained on native English student essays, can be transferred to essays written by English as a foreign language learners and native German speakers. The study finds that while transfer learning is effective even without target domain data (zero-shot), distinct writing styles across educational contexts pose a greater challenge than language differences.

What Problem Does This Paper Address?

Automated feedback systems for student writing, particularly for argumentative essays, are valuable for education. However, training a dedicated model for every new task, language, or educational context is costly and time-consuming. This paper addresses the challenge of transfer learning in argument mining for student essays. Specifically, it asks whether a model trained on one type of student essay (native English speakers, independent writing task) can be effectively applied to other types of student essays (English as a foreign language learners, native German speakers, integrated writing tasks) without requiring extensive retraining. This is crucial for developing scalable and accessible educational tools.

Methods

The researchers leverage a sequence tagging architecture based on the Longformer model, a transformer designed for long documents. This model is trained using Inside-Outside-Beginning (IOB) tags to identify argumentative elements. For transfer learning, they employ a zero-shot approach, where a model trained on one dataset is directly tested on another without any further training on the target data. They also explore fine-tuning the model with small amounts of target domain data to assess how much data is needed to improve performance. To disentangle language effects from educational context effects, they also experiment with a machine-translated German dataset.

Datasets

The study utilizes three primary datasets:

  • PERSUADE (EN-L1): A large corpus (26,000 essays) of argumentative essays written by native English speakers in the US, focusing on independent writing tasks.
  • MEWS (EN-L2): A corpus of essays written by English as a foreign language learners in Switzerland and Germany. A subset of 210 essays was annotated for this study, focusing on independent writing tasks.
  • DARIUS (DE): A corpus of 5,038 essays written by native German speakers, collected from integrated writing tasks.
  • DE-MT: A machine-translated version of the DARIUS dataset into English, created using DeepL Pro.

Key Insights and Ideas

  • Context over Language: The study strongly suggests that distinct writing styles and structures across different educational contexts (e.g., independent vs. integrated writing tasks) have a more significant impact on argument mining model performance than language differences alone.
  • Zero-Shot is Promising: Even without any target domain training data, a model trained on native English essays shows satisfactory performance when applied to essays from English as a foreign language learners and native German speakers. This highlights the generalizability of argument mining models.
  • Machine Translation's Limited Benefit: Using machine-translated data for transfer learning did not significantly improve performance compared to using the original German data. This reinforces the idea that structural and stylistic differences are the primary hurdles.
  • Writing Style Differences: The research provides detailed insights into how argumentative elements are structured and used differently across the datasets. For instance, German essays tend to be shorter and more direct, while English as a foreign language learners might use more elaborate claims.

Key Results

  • EN-L1 to EN-L2 Transfer: The zero-shot transfer from EN-L1 to EN-L2 achieved an F1 score of 0.56 and 0.42, showing a slight drop from the performance on the source EN-L1 dataset. Fine-tuning with limited EN-L2 data did not significantly boost performance, indicating the need for substantial target domain data for substantial gains.
  • EN-L1 to DE Transfer: The zero-shot transfer to the German dataset (DE) yielded lower F1 scores across most labels (overall F1 of 0.36). This performance was similar even when using the machine-translated DE-MT dataset, reinforcing the impact of educational context.
  • Label-Specific Performance: The model performed best on detecting "lead" elements, likely due to their consistent placement at the beginning of essays. Conversely, "claim" and "evidence" detection showed lower performance, especially in cross-lingual and cross-context transfers, with frequent misclassifications between these two.

Limitations and Future Work

  • Language Pair Specificity: The study focused on the English-German language pair. The transferability to languages that are phylogenetically further from English, or less well-covered by multilingual transformer models, remains an open question.
  • Data Requirements for Fine-Tuning: While fine-tuning can improve performance, the study suggests that a significant amount of labeled target domain data is needed to match the performance achieved through transfer learning. Future work could explore more efficient few-shot or zero-shot learning techniques that require less labeled data.
  • Educational Context Nuances: The paper acknowledges that educational contexts are complex. Future research could delve deeper into specific pedagogical approaches and their influence on student writing and argument structures.

Summary updated

BibTeX
@inproceedings{ding2024transfer,
  author = {Yuning Ding and Julian Lohmann and Nils-Jonathan Schaller and Thorben Jansen and Andrea Horbach},
  title = {{Transfer Learning of Argument Mining in Student Essays}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure