Lacuna Language Learning: Leveraging RNNs for Ranked Text Completion in Digitized Coptic Manuscripts

Lauren Levine, Cindy Li, Lydia Bremer-McCollum, Nicholas Wagner, Amir ZeldesACL 2024

TL;DR

This paper introduces a bidirectional RNN model for reconstructing missing text (lacunae) in digitized Coptic manuscripts. While not yet capable of definitive reconstruction, the model effectively ranks potential completions, offering scholars a valuable quantitative tool to augment traditional methods.

What Problem Does This Paper Address?

Ancient manuscripts, particularly those written on perishable materials like papyrus, are often damaged, leaving gaps in the text known as lacunae. Restoring these lacunae is crucial for understanding historical texts, but it's a challenging task. Scholars traditionally rely on qualitative methods, such as studying the manuscript's context and comparing it with other texts. However, these methods can be subjective and prone to errors, especially when no other complete manuscripts exist for comparison. While neural language models have shown promise in various linguistic tasks, their application to manuscript reconstruction, especially for less-resourced languages like Coptic, has been limited. This paper aims to bridge this gap by exploring how RNNs can assist in the reconstruction of Coptic lacunae.

Methods

The authors developed a character-based bidirectional Recurrent Neural Network (RNN) model. The core of their approach is a masked language modeling (MLM) task, inspired by models like BERT. In this task, characters within the input text are randomly masked, and the RNN is trained to predict these missing characters based on their surrounding context. They specifically chose an RNN architecture over transformer-based models due to the relatively moderate size of their Coptic dataset and the potential for RNNs to handle long-distance dependencies effectively in such scenarios.

The model was trained using a character-level vocabulary generated by SentencePiece. They explored different masking strategies during training:

  • Masking Distribution: "Random masking" (15% of characters masked, with specific probabilities for replacing with a mask token, a random character, or keeping the original) and "Smart masking" (mimicking the distribution of lacunae found in their gold-standard test data).
  • Re-masking Frequency: "Once masking" (data masked only once at the start) and "Dynamic masking" (data re-masked at each training epoch).

This resulted in four model configurations: random-once, random-dynamic, smart-once, and smart-dynamic.

Datasets

The primary dataset used for training and evaluation is the Coptic SCRIPTORIUM Corpora. This corpus contains approximately 1.22 million tokens of Sahidic Coptic, derived from various manuscript sources and processed into normalized, machine-readable formats.

For evaluation, they created three test sets:

  1. Test Random: Auto-generated masked data from complete sentences (no lacunae).
  2. Test Smart: Auto-generated masked data from complete sentences, mimicking lacunae distribution.
  3. Test Reconstructed Lacunae: A gold-standard test set consisting of sentences from the Coptic SCRIPTORIUM Corpora that contain lacunae which have been manually reconstructed by Coptic language scholars. This set is crucial as it represents a more realistic scenario.

Key Insights and Ideas

  • RNNs as a Ranking Tool: The most significant insight is that while the RNN model's direct prediction accuracy for lacunae is limited (especially for longer gaps), its ability to assign probabilities to potential reconstructions makes it a powerful tool for ranking candidate completions. This shifts the focus from definitive reconstruction to providing scholars with quantitative evidence to support their qualitative judgments.
  • "Smart Masking" vs. "Random Masking": Surprisingly, the "random-dynamic" masking strategy often outperformed "smart-dynamic" on the reconstructed lacunae test set. The authors hypothesize this might be due to the gold-standard test set containing only fully reconstructed lacunae, potentially biasing it towards shorter gaps than might be encountered in real-world scenarios where "smart masking" might be more beneficial.
  • Lacuna Length Matters: Model accuracy significantly degrades as the length of the lacuna increases. This highlights a key limitation but also points towards future research directions.
  • Normalization's Limited Impact: The authors found that normalizing dialectal variations to Sahidic orthography had a surprisingly small impact on model performance, suggesting the model is robust to some degree of dialectal variation.

Key Results

  • Single Character Reconstruction: The best models achieved around 72% accuracy on single character reconstruction.
  • Longer Lacunae Reconstruction: Accuracy dropped significantly for longer lacunae, falling to below 40% for gaps of 6+ characters.
  • Outperforming Baselines: All tested RNN model configurations substantially outperformed rudimentary baselines (random character prediction, mode character, and tri-gram language models) across all test sets.
  • Case Study Findings: In the Isaiah 37:24 case study, the model ranked three potential reconstructions, with the top-ranked one being the most plausible based on linguistic context and auxiliary verbs. In the Gospel of Philip case study, the model provided rankings for candidate reconstructions proposed by editors, highlighting "cheek" and "forehead" as more probable than "mouth" based on its learned probabilities.

Limitations and Future Work

  • Accuracy on Longer Lacunae: The primary limitation is the low accuracy of the model on reconstructing lacunae longer than a few characters.
  • Single Model Architecture: The study focused solely on RNNs. Future work should explore other architectures, such as transformers, and different training tasks.
  • Input Data: The model does not currently account for Coptic diacritics or document-level context (e.g., page layout). Incorporating lexicographic information (like digital dictionaries) and linguistic annotations (like morphosyntactic information) is suggested for future improvements.
  • Ranking by Length: The current ranking mechanism requires candidates to be of the same sequence length. Future work should enable ranking of candidates with varying lengths.

Summary updated

BibTeX
@inproceedings{levine2024lacuna,
  author = {Lauren Levine and Cindy Li and Lydia Bremer-McCollum and Nicholas Wagner and Amir Zeldes},
  title = {{Lacuna Language Learning: Leveraging RNNs for Ranked Text Completion in Digitized Coptic Manuscripts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure