Introduction
Natural Language Processing (NLP) tools are predominantly trained on standard first-language (L1) corpora, such as news articles or Wikipedia. However, second-language (L2) learners produce language with distinct characteristics, including unique grammatical errors, simplified structures, and non-prototypical morphological usage. In morphologically rich languages like Korean, where words (eojeol) consist of multiple meaningful units (morphemes), these differences pose significant challenges for existing analyzers. This study investigates the performance gap of L1-trained models on L2 Korean and explores strategies for effective fine-tuning.
Methodology and Datasets
The researchers utilized the Stanza NLP framework, specifically the Korean-GSD package pre-trained on L1 data, as a baseline. Two primary experiments were conducted:
- Volume-based Fine-tuning: Fine-tuning the L1 model on the full, relatively noisy L2-NIKL dataset.
- Quality-based Fine-tuning: Fine-tuning on a "cleaned" version of the dataset (L2-NIKL-C), where sentences without punctuation, one-token responses, and misaligned tags were removed.
The study leveraged several key datasets to evaluate performance across different domains and data qualities.
The L2-NIKL corpus provides a large volume of learner data but contains significant noise due to format inconsistencies. In contrast, L2-KLM is a smaller, high-quality dataset manually annotated by experts. The L1-GSD serves as the standard reference for first-language Korean.
Results and Analysis
The experiments revealed that "more data" is not always better. While fine-tuning on L2 data consistently improved performance over the L1 baseline, the quality of that data proved paramount.
The "Quality Over Quantity" Finding
In Experiment 2, the researchers found that the cleaned dataset (L2-NIKL-C)—despite being roughly half the size of the original L2-NIKL-O—yielded superior results in morpheme tokenization. The Perfect Score (PS) for tokenization improved from 94.32% to 97.67% on the L2-NIKL test sets after cleaning.
Tag-Specific Performance and the Punctuation Pitfall
A critical finding was the impact of sentence-final punctuation. Learners frequently omit periods, leading models to misclassify sentence-closing endings (EF) as connecting endings (EC). This error significantly impacted overall accuracy. Furthermore, models fine-tuned on L2 data showed a marked advantage in identifying basic vocabulary common in learner speech but rare in formal L1 datasets, confirming that learner language constitutes a distinct domain.
Conclusion
This research demonstrates that diversifying NLP tools for L2 contexts requires more than just large-scale data collection. Rigorous data cleaning and high-quality annotations are essential for overcoming the "non-prototypical" hurdles of learner language. Future work should extend these findings to dependency parsing to provide a more comprehensive understanding of how language models can better serve diverse language-usage contexts.
Reference: Sung, H., & Shin, G.-H. (2023). Diversifying Language Models for Lesser-studied Languages and Language-usage Contexts: A Case of Second Language Korean. Findings of the Association for Computational Linguistics: EMNLP 2023.
Summary updated