HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea

Haneul Yoo, Jiho Jin, Juhee Son, JinYeong Bak, Kyunghyun Cho, Alice Oh

Introduction

For over five centuries, the Joseon Dynasty (1392–1910) maintained one of the most extensive historical records in human history. These documents, primarily written in Hanja, constitute a massive digital archive of approximately 9 billion characters managed by the Institute for the Translation of Korean Classics (ITKC). However, Hanja presents a unique linguistic challenge: while it utilizes Chinese characters, its syntax and lexicon evolved distinctly within the Korean peninsula, making it largely inaccessible to modern speakers of Korean or Chinese.

The sheer volume of these records creates a significant bottleneck for historians. Relying solely on manual translation and categorization would take decades. While Natural Language Processing (NLP) has advanced for modern languages, existing models trained on Ancient Chinese (such as AnchiBERT) often fail to capture the specific nuances of Korean Hanja. To address this, researchers have introduced HUE (Hanja Understanding Evaluation), a comprehensive benchmark and a suite of pretrained models specifically optimized for ancient Korean records.

Methodology: Continued Pretraining

The HUE framework utilizes a "continued pretraining" strategy. Rather than training a model from scratch, the researchers leveraged existing weights from mBERT (multilingual BERT) and AnchiBERT (pretrained on ancient Chinese). These models were then further trained on two primary Hanja corpora:

  1. Annals of the Joseon Dynasty (AJD): The chronological records of 27 Joseon kings.
  2. Diaries of the Royal Secretariats (DRS): Detailed daily accounts of court events.

By exposing the models to these massive, domain-specific datasets, the models learned the specific vocabulary, grammar, and stylistic shifts inherent to the Joseon era.

The HUE Dataset and Benchmark Tasks

The HUE dataset serves as a benchmark for evaluating Hanja proficiency across four critical downstream tasks:

  • Chronological Attribution (CA): Identifying the reigning king at the time of a document's creation.
  • Topic Classification (TC): Categorizing texts into 106 distinct sub-topics, ranging from diplomacy to agriculture.
  • Named Entity Recognition (NER): Identifying and extracting specific entities such as people and locations.
  • Summary Retrieval (SR): Matching detailed historical records with their corresponding summaries.

To test the models' generalization capabilities, the researchers also utilized the Daily Records of the Royal Court and Important Officials (DRRI) for zero-shot evaluation. This dataset represents "unseen" history that has not been systematically analyzed by the NLP community.

Key Insights

The Role of Named Entities

A significant finding in the Chronological Attribution task was the "Name Trap." Models often achieved high performance by memorizing which historical figures were active under specific kings. To ensure the models were learning linguistic style rather than just entity associations, the researchers experimented with masking named entities. Interestingly, masking names occasionally improved performance, suggesting that the underlying linguistic evolution of Hanja provides a robust signal for dating documents.

Temporal Context as a Feature

The research demonstrated that providing the "document age" (the era of the reigning king) as an input feature significantly improved performance in Topic Classification and NER. This confirms that Hanja was not a static language; its evolution over 500 years was significant enough that knowing the era helps the model disambiguate meanings and context.

Hanja vs. Ancient Chinese

The specialized Hanja-BERT models consistently outperformed AnchiBERT across all tasks. This result underscores the fact that "Ancient Chinese" is not a monolithic entity. Regional variations in the Sinosphere require domain-specific tuning to achieve high accuracy in historical document analysis.

Results and Impact

The pretrained HUE models demonstrated substantial improvements over standard baselines. In Chronological Attribution, the best-performing model (AnchiBERT + AJD/DRS) achieved an F1 score of 79.33, a significant leap from the 54.26 achieved by a non-pretrained BERT baseline. In NER tasks, the models reached high precision, with F1 scores around 93.0 for identifying people and locations. Even in zero-shot scenarios on the DRRI dataset, the models maintained a respectable F1 score (~35.85), indicating their potential to assist historians in categorizing previously unstudied archives.

Conclusion

HUE represents a major step forward in the digital humanities for East Asian history. While the current models focus on structured official court records, they provide a foundation for future work involving more diverse genres such as poetry, novels, and personal correspondence. By integrating these models into translation assistance tools, the research community can significantly accelerate the preservation and understanding of Korea's rich historical legacy.

Summary updated

BibTeX
@inproceedings{yoo2022hue,
  author = {Haneul Yoo and Jiho Jin and Juhee Son and JinYeong Bak and Kyunghyun Cho and Alice Oh},
  title = {{HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.140},
}
Figure