TL;DR
This paper explores using large language models (LLMs) to restore missing words in ancient cuneiform texts, a challenging task due to the script's complexity. The authors fine-tuned three LLMs using different prompting strategies and found that a simple "fill-in-the-blank" approach yielded the best results, achieving up to 26.9% accuracy when ensembling multiple models.
What Problem Does This Paper Address?
Interpreting and translating ancient cuneiform texts, like Akkadian and Sumerian, is a significant challenge. A major hurdle is the polyvalence of cuneiform signs, where a single symbol can represent a whole word (logogram), a syllable, or a grammatical marker (determinative). This ambiguity, coupled with the fact that Akkadian texts can contain Sumerian words, makes automated processing difficult. This paper focuses on a specific sub-task: predicting missing words in transliterated cuneiform texts, aiming to provide a baseline for future research in this area.
Methods
The authors fine-tuned three autoregressive LLMs: Aya Expanse 8B, Command-R v0.1 34B, and Mistral Small 3 24B. They employed a simple engineering approach, using the provided task data without task-specific adjustments, preprocessing, or filtering, given their limited knowledge of the subject matter and languages. The core of their method involved framing the token prediction task in three distinct ways, each with a specific prompt:
- All: The model is prompted to fill in all masked words in a document, outputting a comma-separated list of the predicted words.
- One by one: All but one masked token are replaced with an
[UNK]token, and the model is asked to predict the single masked word. This is repeated for each masked token. - Restore: The model is asked to output the full restored text of the masked document.
They used QLoRA for efficient fine-tuning and experimented with various LoRA rank sizes. The models were trained using the AdamW optimizer with a specific learning rate and batch sizes. For evaluation, they sampled predictions with a temperature of 0.2 and measured accuracy. They also explored ensembling the best checkpoints through majority voting.
Datasets
The primary dataset used was provided by the organizers of the EvaCun 2025 token prediction shared task. This dataset contains transliterated Akkadian and Sumerian texts with masked tokens. The authors used 913,252 tokens from 22,777 documents for training. For evaluation, they set aside 1% of the data (227 documents) as a development set, further filtering it to create a held-out evaluation subset of 135 documents containing 1500 unique masked examples.
Key Insights and Ideas
- Simplicity as a Baseline: The paper demonstrates that even without deep linguistic expertise, fine-tuning existing LLMs with straightforward prompting can yield a reasonable baseline for complex NLP tasks on ancient languages.
- Prompting Matters: The "All" prompt, which asks the model to predict all masked words at once, surprisingly outperformed the "One by one" and "Restore" methods. This suggests that the autoregressive nature of LLMs might be more robust to predicting multiple tokens simultaneously in this context than anticipated.
- LLM Overestimation of Popular Words: The study observed that LLMs tend to overestimate the frequency of the most popular words, a common issue in text-generation models. This leads to an underestimation of less common words, impacting the diversity of generated tokens compared to the reference data.
- Ensembling Boosts Performance: Combining predictions from multiple fine-tuned models through majority voting significantly improved accuracy, highlighting the benefit of ensemble methods in mitigating individual model weaknesses.
Key Results
The best single models achieved an accuracy of up to 22.1% on the held-out development set. By ensembling the top-performing checkpoints using majority voting, the authors reached an accuracy of 26.9%. The "All" prompt method generally yielded the highest accuracies across the tested LLMs. The paper also notes that the "Restore" method, while aiming for full text generation, was negatively impacted by the potential generation of [MASK] tokens and a mechanism that could force suboptimal predictions to maintain word length.
Limitations and Future Work
The authors acknowledge that their approach is a baseline and that biases, such as repetitiveness in the dataset, might lead to overestimating the true capabilities of their method. They also note that the "Restore" method was negatively affected by the possibility of generating [MASK] tokens.
Future work planned by the authors includes tackling the more challenging task of direct translation of cuneiform script into English, potentially by using Unicode transcriptions or visual LLMs to process tablet images directly.
Summary updated