UD-KSL Treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags

Hakyung Sung, Gyu-Ho Shin, Chanyoung Lee, You Kyung Sung, Boo Kyung JungACL 2025

TL;DR

This paper introduces a semi-automated framework to align fine-grained Korean XPOS tags with Universal Dependencies (UD) UPOS tags, improving consistency and accuracy in L2 Korean NLP tasks. By augmenting an existing L2 Korean UD treebank with new data and applying this alignment, the authors demonstrate significant improvements in morphosyntactic tagging and dependency parsing, especially in low-resource settings.

What Problem Does This Paper Address?

Developing high-quality linguistic annotations for learner corpora is crucial for understanding language acquisition and improving NLP tools. While Universal Dependencies (UD) provides a standardized framework for cross-linguistic consistency, Korean's morphologically rich nature presents a challenge. Korean often packs multiple grammatical elements into a single "eojeol" (whitespace-separated unit). Existing L2 Korean UD treebanks often rely on expert-curated, fine-grained Korean-specific XPOS tags but automatically generated, coarser UPOS tags. This disparity can lead to inconsistencies and hinder downstream NLP model performance. This paper addresses this gap by proposing a method to bridge these annotation layers, ensuring that the detailed information from XPOS is better leveraged by UPOS-based models.

Methods

The core of this work is a semi-automated framework for aligning XPOS tags with UPOS categories. The process begins with the gold-standard XPOS annotations from the UD-KSL v1.2 treebank. They extract all unique eojeol-level constructions and their XPOS tag sequences. To manage the manual effort, they prioritize constructions that appear frequently (more than five times), which constitute the vast majority of the corpus. These frequent constructions are then manually aligned with corresponding UPOS tags by two linguists, with disagreements resolved by a third. This alignment process aims to preserve the syntactic abstraction of UPOS while retaining key morphological details from XPOS.

Datasets

The study utilizes two datasets:

  • UD-KSL v1.2: The existing L2-Korean UD treebank, containing 12,984 manually annotated sentences.
  • UD-KSL working set: A newly annotated dataset created for this study, comprising 2,998 sentences from argumentative essays. This new dataset expands the corpus into a different genre and provides additional data for evaluation.

Key Insights and Ideas

  • Bridging the Annotation Gap: The most significant insight is that aligning fine-grained XPOS tags with coarser UPOS tags can significantly improve NLP model performance, especially for morphologically rich languages like Korean. This alignment acts as a bridge, transferring valuable linguistic information.
  • Low-Resource Advantage: The alignment framework shows a disproportionately larger benefit in low-resource settings (the smaller UD-KSL working set). This suggests that alignment can act as a compensatory strategy, enhancing label consistency when training data is scarce.
  • Toolkit Differences Matter: The choice of NLP toolkit significantly impacts the benefits of alignment. Trankit, a transformer-based model, consistently showed greater improvements from alignment compared to spaCy, suggesting its architecture is better equipped to leverage the enhanced annotation.
  • Lemmatization Challenges Persist: While the focus was on UPOS alignment, the study also noted that Trankit's lemmatization accuracy remained lower than spaCy's, even with alignment. This highlights that architectural improvements are still needed for robust lemmatization in L2 Korean.

Key Results

The study reports several key findings:

  • UPOS Tagging Improvement: Aligning UPOS tags consistently improved UPOS tagging accuracy across both datasets and toolkits. For spaCy, improvements ranged from +0.23 to +6.31 (on UD-KSL v1.2). Trankit saw gains of +0.47 to +4.04 (on the UD-KSL working set).
  • XPOS Tagging Improvement: Similar positive trends were observed for XPOS tagging, with Trankit showing notable gains (+3.51) on the UD-KSL working set.
  • Dependency Parsing: The impact on dependency parsing was more varied. Trankit showed significant gains in LAS and UAS on the UD-KSL working set (+4.92 and +4.67 respectively), while spaCy saw slight declines.
  • Dataset Size Effect: The smaller UD-KSL working set benefited more substantially from alignment, particularly with Trankit, suggesting alignment's value in low-resource scenarios.

Limitations and Future Work

The authors acknowledge several limitations:

  • Granularity: The current alignment strategy might not fully capture the complexity of Korean morphosyntax, and some constructions that don't lend themselves to straightforward mapping remain underexplored.
  • Human Annotator Reliance: Despite automation, the process still relies on human annotators, introducing potential variability in consistency and accuracy.
  • Lemmatization: The study reinforces previous findings about Trankit's lower lemmatization accuracy, suggesting a need for architectural refinements or lexicon-based integration.

Future work could explore more nuanced or hierarchical alignment frameworks and investigate strategies to improve transformer-based lemmatization for L2 Korean. Further research is also needed to understand the specific architectural and data characteristics that influence the asymmetric impact of alignment on dependency parsing.

Summary updated

BibTeX
@inproceedings{sung2025ud,
  author = {Hakyung Sung and Gyu-Ho Shin and Chanyoung Lee and You Kyung Sung and Boo Kyung Jung},
  title = {{UD-KSL Treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure