HieroLM: Egyptian Hieroglyph Recovery with Next Word Prediction Language Model

Xuheng Cai, Erica ZhangNAACL 2025

TL;DR

This paper introduces HieroLM, a novel approach to recovering damaged or missing Egyptian hieroglyphs by framing it as a next-word prediction task using language models. By leveraging the sequential nature of hieroglyphic texts, HieroLM achieves competitive accuracy and demonstrates robustness, offering a practical tool for Egyptologists.

What Problem Does This Paper Address?

Egyptian hieroglyphs, crucial for understanding ancient Egyptian culture, are often found on artifacts in a degraded state – blurry or incomplete due to erosion. While computer vision (CV) techniques have been applied to recognize these damaged glyphs, they face significant limitations. Primarily, CV models struggle with severely damaged or entirely missing glyphs as they rely on visual features. Furthermore, they treat each glyph in isolation, ignoring the rich contextual and grammatical information inherent in hieroglyphic sentences, which could significantly aid in disambiguation and prediction. This paper aims to bridge this gap by proposing a language modeling approach that captures these sequential dependencies.

Methods

The core innovation of this paper is reframing hieroglyph recovery as a next-word prediction task, a domain where language models excel. Instead of treating glyphs as independent images to be classified, HieroLM models them as a sequence of "words." The authors experimented with different language model architectures, ultimately selecting Long Short-Term Memory (LSTM) networks. This choice is motivated by the observed "strong local affinity of semantics" in Egyptian hieroglyphic texts, suggesting that LSTMs are well-suited to capture these short-range dependencies. The HieroLM architecture processes input words (hieroglyphs) through an embedding layer, followed by LSTM layers that maintain a hidden state and cell state to capture sequential information. The final output is a prediction of the next word in the sequence.

Datasets

The authors evaluated their model on three real-world datasets:

  • Ancient Egyptian Sentences (AES): A collection of over 100,000 ancient Egyptian sentences from various dynasties.
  • Ramses Transliteration Corpus: Contains transliterations from a large corpus of Late Egyptian sentences.
  • Mixed: A synthesized dataset combining AES and Ramses to evaluate cross-distribution modeling ability, accounting for language evolution.

The MdC transliteration format was used for processing, which standardizes certain characters for easier computation. Datasets were split into training, validation, and test sets with an 8:1:1 ratio.

Key Insights and Ideas

  • Language Models for Hieroglyphs: The most significant insight is that treating hieroglyph recovery as a language modeling problem, specifically next-word prediction, is a viable and effective alternative to traditional CV approaches. This highlights the inherent sequential and linguistic nature of hieroglyphic texts.
  • LSTM's Suitability: The paper strongly suggests that LSTMs are particularly well-suited for this task due to the local semantic affinity in Egyptian hieroglyphic sentences, outperforming simpler RNNs and even Transformers on limited data.
  • Context is King (Even for Ancient Languages): The success of HieroLM underscores the importance of contextual information in deciphering damaged texts. Unlike CV models that look at isolated glyphs, HieroLM leverages surrounding glyphs to infer missing ones.
  • Constrained Vocabulary: The relatively low perplexity achieved by HieroLM, even with limited data, suggests that the hieroglyphic vocabulary and sentence structures are more constrained than modern languages, making them amenable to language modeling.

Key Results

  • Accuracy: HieroLM achieved over 44% accuracy on the task of inferring missing words.
  • Perplexity: The model achieved a perplexity of approximately 26, which is notably low for a language with less than a million words in its training corpus, indicating a highly constrained vocabulary.
  • Robustness: HieroLM demonstrated robust performance on multi-shot predictions (predicting multiple consecutive missing words) and in scenarios with scarce context, maintaining reasonable accuracy even when predicting up to 4 words in a row.
  • Comparison to Baselines: HieroLM consistently outperformed NPLM, RNN, and Transformer baselines in terms of accuracy and perplexity, especially on smaller datasets. The Transformer, while powerful, underperformed due to its data requirements.

Limitations and Future Work

  • Transformer Limitations: The authors acknowledge that due to limited data availability, they had "little success in leveraging the power of the state-of-the-art Transformer models." They suggest that future work could explore adapting Transformer architectures with sophisticated training techniques for this domain.
  • Integrating CV and LMs: A promising future direction is to explore ways of integrating computer vision models with language models to create a unified and more effective hieroglyph recovery system. This could combine the visual recognition capabilities of CV with the contextual understanding of LMs.
  • Data Scarcity: While HieroLM shows resilience against data scarcity, it remains a fundamental challenge in the field. Further research into data augmentation or transfer learning techniques could be beneficial.

Summary updated

BibTeX
@inproceedings{cai2025hierolm,
  author = {Xuheng Cai and Erica Zhang},
  title = {{HieroLM: Egyptian Hieroglyph Recovery with Next Word Prediction Language Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure