TL;DR
This paper introduces PahGen, a novel framework for translating English text into the endangered ancient language Pahlavi. By combining grammar-guided term extraction with zero-shot translation using Large Language Models (LLMs), PahGen generates syntactically and semantically accurate Pahlavi sentences, addressing a critical gap in digital resources for this historical language.
What Problem Does This Paper Address?
Pahlavi, or Middle Persian, is a crucial linguistic bridge between Old Persian and Modern Persian, holding significant cultural and historical value. However, it faces extinction due to a severe lack of digital resources and comprehensive linguistic tools. While efforts exist to digitize its scripts, translating modern concepts into Pahlavi and generating new, accurate Pahlavi text remains a significant challenge. Existing LLMs struggle with such low-resource languages due to insufficient training data, often leading to nonsensical outputs. This paper aims to preserve Pahlavi by creating a method for generating new texts and a foundational dataset for future research and revitalization efforts.
Methods
PahGen employs a four-step pipeline:
- Lexical Segmentation: Input English sentences are broken down into individual terms using grammatical structure. This step leverages the Stanza library for syntactic features (POS tags, lemmas, named entities) and GPT-4o in a few-shot setting to extract richer morphosyntactic features, enriching each segment for better translation.
- Translation: This is a two-stage process. First, individual terms are translated using a Pahlavi-English dictionary. For terms not found, a rule-based morphological generation system is used, incorporating predefined Pahlavi rules. If still untranslatable, BERT embeddings are used to find the closest semantic match. Second, these translated terms are reconstructed into a coherent Pahlavi sentence, considering word order, kasreh-ezafeh construction, and historical punctuation. GPT-4o is used in a zero-shot setting for this reconstruction.
- Phoneme to Transliteration: The phonemic representation of the Pahlavi sentence is converted into transliteration using a dictionary mapping phonemes to their transliterations and applying specific linguistic rules.
- Script Conversion: The final step converts the transliterated output into the authentic Book Pahlavi script. This involves addressing complexities in the Ham-dibrih typeface, including an improved version 2 of the font developed by the authors.
Datasets
The paper introduces a novel, expert-validated parallel English-Pahlavi dataset. This dataset consists of 200 sentence pairs, including excerpts from historical texts and synthetically generated sentences. The synthetic sentences were carefully filtered to exclude terms that would not have existed in the Sassanian era. Additionally, the authors created a larger dataset of 360 sentence pairs using their framework, aiming to facilitate further research.
Key Insights and Ideas
- LLMs struggle with low-resource languages without guidance: Direct zero-shot translation with LLMs for Pahlavi resulted in significant hallucination and nonsensical outputs, highlighting the need for structured approaches.
- Grammar-guided extraction is crucial: Combining linguistic rules and LLMs (e.g., GPT-4o for feature extraction and sentence reconstruction) significantly improves translation accuracy for endangered languages.
- Pahlavi's unique linguistic features pose challenges: The language's specific syntax (e.g., ergative alignment in transitive verbs) and lack of direct equivalents for modern concepts require specialized handling.
- A new dataset is vital for progress: The creation of an expert-validated parallel corpus is a critical step towards enabling further research and development in Pahlavi NLP.
Key Results
- Automatic Evaluation: PahGen significantly outperformed zero-shot and zero-shot with vocabulary baselines across BLEU and ChrF metrics. Our model achieved a BLEU-1 score of 83.1, BLEU-2 of 65.3, and ChrF of 51.2, demonstrating its effectiveness.
- Human Evaluation: Human evaluators rated PahGen's output highly across fluency, relatedness, and correctness, with an average Likert scale score of 4 out of 5 for all metrics, compared to scores of 1-3 for the baselines.
- LLM Hallucination Analysis: Analysis of token entropy revealed that LLMs are more prone to hallucination when translating into Pahlavi compared to Persian, indicating a higher degree of uncertainty in generating Pahlavi text.
Limitations and Future Work
The authors acknowledge the limited size and diversity of their manually verified dataset due to the scarcity of Pahlavi experts. They suggest future work could involve generating more translations using their model and manually verifying them, or incorporating a wider range of historical texts to enhance generalization. A more in-depth analysis of low-scoring sentences is also recommended to identify common failure patterns and refine the framework. Expanding the dataset and exploring more advanced LLM fine-tuning techniques are also promising directions.
Summary updated