Enhancing Sumerian Lemmatization by Unsupervised Named-Entity Recognition

Yudong Liu, Clinton Burkhart, James Hearne, Liang LuoNAACL 2015

TL;DR

This paper tackles the challenge of accurately identifying personal names in ancient Sumerian texts, a crucial step for improving lemmatization (assigning base forms to words). By developing an unsupervised Named Entity Recognition (NER) system, they significantly improve the identification of personal names, especially in damaged texts, reducing the burden on human experts and offering a more robust lemmatization for the Cuneiform Digital Library Initiative (CDLI).

What Problem Does This Paper Address?

Lemmatization, the process of reducing words to their base or dictionary form, is a fundamental task in natural language processing. However, for ancient, dead languages like Sumerian, this process is exceptionally difficult. The language has few living experts, and the available texts are often fragmented and damaged due to age and historical circumstances. The Cuneiform Digital Library Initiative (CDLI) aims to make these texts accessible, but its current lemmatization process is overly conservative, particularly with names that have missing or damaged signs. This leads to a significant number of personal names being missed, hindering deeper analysis of Sumerian social history and economic transactions. The paper addresses the need for a more automated and accurate way to identify personal names within these challenging texts.

Methods

The core of the paper's contribution is an unsupervised Sumerian Named Entity Recognition (NER) system. This system is built upon the Decision List Co-Train (DL-CoTrain) method, a semi-supervised learning approach. The system has three main components:

  1. Preprocessing: This stage cleans the input text by removing metacharacters used to denote damage or uncertainty (e.g., [...], {#}) and applies a set of 13 predefined tags to identify various entity types (like geographical names, titles, units, etc.) based on existing knowledge. This provides initial contextual clues.
  2. DL-CoTrain Component: This is the heart of the system. It iteratively learns a decision list of rules to classify words as personal names. It starts with a few "seed rules" (contextual rules derived from prior work) and then alternates between learning new spelling rules (based on observed name patterns) and new contextual rules (based on the newly identified names). This process refines the decision list, ranking rules by their strength.
  3. Post-processing: Two simple rules are applied to filter out common false positives: words starting with numbers and words immediately following month indicators.

A key aspect of their approach is its unsupervised nature, meaning it doesn't require a large, pre-labeled dataset of personal names to start. Instead, it leverages minimal seed annotation and the existing, albeit conservative, lemmatization in CDLI.

Datasets

The primary dataset used is the Cuneiform Digital Library Initiative (CDLI) repository. Specifically, the authors focused on texts from the Ur III period (2112-2004 BCE). They utilized approximately 53,146 tablets that had in-line annotations (lemmatization) for their NER system. The existing lemmatization in CDLI served as the "gold standard" for evaluation, though the paper highlights its limitations.

Key Insights and Ideas

  • Conservative Lemmatization is a Bottleneck: The existing CDLI lemmatization is overly cautious, especially with damaged signs. This conservatism, while preventing incorrect labeling, leads to a significant undercounting of personal names, missing valuable social and historical information.
  • Unsupervised NER for Damaged Texts: An unsupervised NER system can effectively recover personal names even when signs are missing or damaged, by leveraging contextual and spelling features. This is crucial for ancient texts where damage is common.
  • High Recall, Moderate Precision is Useful: The system achieves high recall (identifying most actual names) but a moderate precision (some identified names are incorrect). This is presented as a valuable tool for human annotators, who can then focus on verifying the system's suggestions, significantly reducing their workload.
  • Minimal Seed Annotation is Sufficient: The DL-CoTrain method, initialized with just a few seed rules, can learn to identify personal names effectively, demonstrating the power of semi-supervised learning in low-resource scenarios.

Key Results

  • The system achieved an average of 91.4% recall and 39.6% precision on the CDLI dataset when using the existing lemmatization as the gold standard.
  • Error analysis revealed that nearly 50% of false positives contained "missing" or "damaged" signs.
  • When a Sumerologist verified these "false positives" with damaged signs, over 40% were confirmed as personal names, suggesting the existing lemmatization was too conservative.
  • Incorporating this expert verification elevated the precision to 55.8% without sacrificing recall.
  • The post-processing rules improved performance by 0.5%.

Limitations and Future Work

  • Limitations:
    • The current evaluation relies on the existing CDLI lemmatization, which the authors argue is flawed due to its conservatism.
  • Due to time constraints and the sheer volume of "false positives," the Assyriologist informant could not verify all system-reported names.
  • The system's precision, while improved, is still moderate, meaning human verification remains necessary.
  • Future Work:
    • Apply supervised learning or combine the current method with gazetteer-based approaches for potentially higher precision.
  • Extend the system to recognize other types of named entities beyond personal names.
  • Further refine the DL-CoTrain method and explore additional features.
  • The authors suggest that their system can be used as an automated annotation tool, making the lemmatization process more efficient.

Summary updated

BibTeX
@inproceedings{liu2015enhancing,
  author = {Yudong Liu and Clinton Burkhart and James Hearne and Liang Luo},
  title = {{Enhancing Sumerian Lemmatization by Unsupervised Named-Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure