UA-KO at SemEval-2022 Task 11: Data Augmentation and Ensembles for Korean Named Entity Recognition

Hyunju Song, Steven BethardSemEval @ NAACL 2022 · DOI

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, typically focusing on well-defined categories such as persons, locations, and organizations. However, "Complex NER" presents a significant challenge by requiring the identification of nontraditional entities like movie titles, book names, or software in low-context environments. This paper details the UA-KO team's participation in the SemEval-2022 Task 11 (MultiCoNER) for the Korean language, where they addressed these difficulties using transformer-based ensembles and data augmentation.

Korean NER is particularly challenging due to two primary factors. First, short texts such as search queries often lack sufficient semantic context to distinguish titles from common nouns. Second, Korean is an agglutinative language where morphemes are attached to roots, resulting in high linguistic complexity and a large number of out-of-vocabulary (OOV) terms for standard models.

Methodology

The researchers explored several strategies to optimize performance for the Korean language:

Transformer-Based Models

The team fine-tuned a variety of state-of-the-art models, including monolingual Korean models (KoBERT, KR-BERT, Ko-ELECTRA, and KLUE-RoBERTa) and the multilingual XLM-RoBERTa baseline.

Task-Adaptive Pretraining (TAPT) and Augmentation

To specialize the models, task-adaptive pretraining (TAPT) was conducted on unlabeled task data. Additionally, the team implemented data augmentation by replacing person and location entities in the training set with entries from external gazetteers, such as GeoNames and the Encyclopedia of Korean Culture.

Soft-Voting Ensembles

A "soft-voting" ensemble strategy was employed, where final predictions were determined by averaging the probability scores across multiple fine-tuned models.

Experimental Setup

The study utilized the MultiCoNER Korean dataset, consisting of 15,300 training sentences, 800 development sentences, and a large-scale test set ranging from 150,000 to 500,000 sentences. External knowledge bases provided over 18,000 historical figures and 170 country names for augmentation purposes.

Results and Discussion

The experimental results highlighted the effectiveness of ensembling and the nuances of data augmentation in the Korean context.

Performance comparison across various models and modifications on the development set.

Model Performance and Insights

As shown in Figure 5, ensembling consistently provided the most reliable performance gains on the development set. The best-performing system was an ensemble of all five models (KoBERT, KR-BERT, Ko-ELECTRA, KLUE, and XLM-R), which ultimately achieved a final test F1 score of 0.675.

However, the researchers noted that certain techniques were risky. Task-adaptive pretraining (TAPT) yielded little benefit and, in the case of Ko-ELECTRA, caused performance to plummet from 0.827 to 0.767 F1. This was likely due to the small size of the task-specific data. Furthermore, while data augmentation helped on the development set, it negatively impacted performance on the test set, suggesting that the external names used did not align well with the unique entities in the test data.

Class-Specific Analysis

The model demonstrated varying levels of success across different entity types.

Class-specific performance metrics on the test set.

According to Figure 2, the system performed best on standard entities like Persons (0.761 F1) and Locations (0.735 F1). However, it struggled significantly with Creative Works (CW), achieving only a 0.544 F1 score. This disparity underscores the difficulty of identifying complex, nontraditional entities in short, low-context strings.

Error Analysis and Challenges

A major hurdle identified was annotation inconsistency within the dataset, particularly regarding punctuation like brackets (《 》) used for titles. Furthermore, many errors were attributed to "token boundary issues," where models incorrectly included grammatical particles (common in agglutinative languages) within the entity boundaries.

Conclusion

The UA-KO team's approach demonstrated that ensembling diverse monolingual and multilingual models is a robust strategy for complex NER in Korean. While data augmentation via gazetteers provided some benefits, its failure to generalize to the test set suggests a need for more sophisticated external knowledge integration. Future work will focus on enhancing morphological awareness and developing more robust gazetteer-based strategies to better capture the unique nature of complex entities.

Summary updated

BibTeX
@inproceedings{song2022ua,
  author = {Hyunju Song and Steven Bethard},
  title = {{UA-KO at SemEval-2022 Task 11: Data Augmentation and Ensembles for Korean Named Entity Recognition}},
  booktitle = {SemEval @ NAACL},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.222},
}
Figure