Towards L2-friendly pipelines for learner corpora: A case of written production by L2-Korean learners

Hakyung Sung, Gyu-Ho ShinACL 2023

TL;DR

This paper introduces the Korean-Learner-Morpheme (KLM) corpus, a manually annotated dataset of L2 Korean learner writing, to address the limitations of existing NLP tools when applied to non-native language data. They demonstrate that standard Korean NLP analyzers perform significantly worse on L2 Korean text and show that fine-tuning these models with L2 data can improve performance.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has made tremendous strides in analyzing and processing text. However, most of these advancements and the tools built upon them are trained and evaluated on "native" or "L1" data. This creates a significant challenge when applying these tools to "learner" or "L2" data, such as text written by individuals learning a second language. Learner language often contains unique characteristics like spelling and spacing errors, novel word combinations, and grammatical deviations that can confuse L1-trained models. This paper specifically addresses this gap for Korean, a language with complex morphology and no clear word spacing, by investigating the performance of existing Korean NLP tools on L2 Korean learner writing and proposing a solution.

Methods

The core of this research involves creating and utilizing a new dataset, the Korean-Learner-Morpheme (KLM) corpus. This corpus is derived from an existing L2 Korean learner corpus and has been manually annotated for morpheme tokenization and part-of-speech (POS) tagging. The authors then evaluated four Korean morphological analyzers: two general-purpose deep-learning models (Stanza, Trankit) and two Korean-specific statistical models (Kkma, Komoran). They compared the performance of these analyzers on both the newly annotated L2 Korean corpus and a reference L1 Korean corpus. Additionally, they explored the impact of fine-tuning one of the models (Stanza) using the L2 KLM corpus to see if incorporating L2 data improves performance.

Datasets

  • Korean-Learner-Morpheme (KLM) Corpus: A newly created, manually annotated dataset of L2 Korean learner writing. It contains 129,784 morphemes from 7,527 sentences, with detailed morpheme tokenization and POS tagging based on the Sejong tag set. This corpus was derived from the Kyung Hee Korean learner written corpus.
  • Reference L1 Corpus: Google Korean Universal Dependency Treebank (UD Korean GSD) was used as a baseline for L1 performance. Specifically, 989 sentences from its test set were utilized.

Key Insights and Ideas

  • L2 Data is Different: The study strongly confirms that standard NLP tools, even sophisticated deep-learning ones, struggle with L2 learner data. The performance drop on L2 Korean compared to L1 Korean is a significant finding, highlighting the need for specialized approaches.
  • Morphological Complexity is a Hurdle: Korean's agglutinative nature, where morphemes are heavily attached, makes tokenization and POS tagging particularly challenging, especially for L2 learners who may not adhere to standard patterns.
  • Fine-tuning Matters: Even a relatively small L2 dataset can significantly improve the performance of pre-trained models. This suggests that incorporating L2-specific data into the training pipeline is a promising direction for building more robust NLP tools for language learners.
  • Tag-Specific Challenges: Not all parts of speech are equally problematic. The study identified specific tags (e.g., predicate-related tags like verbs and adjectives, and some suffix/particle tags) that consistently showed lower accuracy, pointing to areas for future research and model improvement.

Key Results

  • All four tested morphological analyzers showed reduced performance on the L2 Korean corpus compared to the L1 Korean corpus.
  • Stanza and Komoran achieved the highest F1 scores (tied) for both tokenization and POS tagging on the L2 data, with overall F1 scores around 0.89 for tokenization and 0.86 for POS tagging.
  • Fine-tuning the Stanza model with the L2 KLM corpus resulted in improved F1 scores on the L2 test set (0.93 for tokenization, 0.91 for POS tagging) compared to the L1-trained Stanza model. However, this fine-tuned model showed a performance drop on the L1 dataset, suggesting a potential "forgetting" effect.
  • By-tag analysis revealed that predicate-related tags (VV, VA, VX) and some error-related tags had particularly low accuracy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: The KLM corpus, while substantial, is still relatively small for training deep-learning models, which could limit the generalizability of the fine-tuning results.
  • Proficiency Measurement: The original corpus's proficiency levels were based on class levels, which may not be a reliable proxy for actual proficiency. Future work should involve more robust proficiency assessment.
  • Computational Resources: Applying cutting-edge deep-learning algorithms, especially with larger datasets, requires significant computational power.

Future work suggested includes:

  • Expanding L2 Corpora: Creating larger and more diverse L2 Korean corpora with refined annotation schemes and richer metadata (like learner proficiency).
  • Incorporating Syntactic Information: Building L2 Korean syntactic treebanks to complement the morphological data.
  • Investigating Learner Language Characteristics: Further research into how specific L2 learner language properties interact with NLP models.
  • Addressing "Forgetting": Investigating methods to mitigate the performance drop on L1 data after L2 fine-tuning.

Summary updated

BibTeX
@inproceedings{sung2023towards,
  author = {Hakyung Sung and Gyu-Ho Shin},
  title = {{Towards L2-friendly pipelines for learner corpora: A case of written production by L2-Korean learners}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure