TL;DR
This paper introduces ParsiPy, a novel Python-based NLP toolkit specifically designed for processing historical Middle Persian (Pārsig) texts. It addresses the significant lack of computational resources for this language by offering modules for tokenization, lemmatization, part-of-speech tagging, and phoneme-to-transliteration, paving the way for broader computational philology.
What Problem Does This Paper Address?
Historical languages present unique challenges for Natural Language Processing (NLP) due to complex orthography, fragmented texts, and the absence of standardized digital representations. While NLP has revolutionized modern language analysis, ancient languages, particularly those with limited digital corpora and complex writing systems like Middle Persian (Pārsig), remain largely underexplored. Pārsig, a crucial link between ancient Iranian languages and modern Persian, has received minimal computational attention. This gap hinders the study of its rich cultural heritage and linguistic evolution. Existing NLP tools are often not directly applicable due to Pārsig's specific linguistic features, such as its unique script variations, the use of Aramaeograms, and a limited digital corpus.
Methods
ParsiPy is built on three core components: an embedding module, NLP task modules, and a Parsig character generator.
- Embedding Module: Integrates state-of-the-art word embedding techniques like FastText, GloVe, and Word2Vec, which are well-suited for low-resource scenarios. These embeddings capture semantic relationships, aiding downstream tasks.
- NLP Task Modules:
- Tokenizer: Employs a SentencePiece unigram language model, chosen for its ability to handle raw text without predefined word boundaries, making it ideal for Pārsig's inconsistent orthography. Its subword approach helps manage out-of-vocabulary words and rare morphological variations.
- Lemmatizer: Uses a rule-based approach with a comprehensive table of lemmas and linguistic rules to reduce words to their canonical form, accounting for inflectional variations and compound words.
- POS Tagger: Supports three models: Hidden Markov Model (HMM) with Viterbi decoding, Logistic Regression, and Random Forest. It incorporates hand-crafted linguistic features and morphological features specific to Pārsig.
- Phoneme-to-Transliteration (P2T) Module: Addresses the common representation of Pārsig in digital resources via phonetic transcriptions. It uses a rule-based system, leveraging linguistic rules specific to Pārsig, to bridge the gap between phonemic representation and standardized transliteration.
- Parsig Character Generator: A module to convert transliterated text back into the original Parsig script using a refined font.
Datasets
The primary dataset used for training and evaluation is the Pārsig Database. This meticulously curated resource contains 120 documents, totaling 93,518 words, 8,839 unique tokens, and 4,641 distinct lemmas. It includes annotations for lemmatization, POS tagging, and transliterations, along with English and Persian translations. The database adheres to strict annotation standards, including transcriptions, transliterations preserving original spellings, and Huzvāreš annotations.
Key Insights and Ideas
- Tailored NLP for Ancient Languages: The paper strongly advocates for the necessity of specialized NLP toolkits for historical languages, rather than attempting to adapt general-purpose tools. ParsiPy's design explicitly addresses Pārsig's unique linguistic challenges.
- Bridging the Phonetic-Script Gap: The P2T module is a crucial innovation, providing a standardized transliteration that acts as an intermediary between phonetic representations and the original Pārsig script, a significant hurdle in digital processing.
- Rule-Based Approaches for Low-Resource Scenarios: The authors highlight the effectiveness of rule-based methods, particularly for the P2T module, when dealing with limited annotated data, a common issue in historical linguistics.
- Modular Design for Extensibility: ParsiPy's modular architecture is designed to facilitate the integration of new tasks and models, serving as a foundational scaffolding for future research in Pārsig and other low-resource ancient languages.
Key Results
- P2T Module: Achieved a Word Error Rate (WER) of 29.764% and a Character Error Rate (CER) of 13.525% using a rule-based model, outperforming an LSTM-based approach.
- Lemmatizer: Demonstrated an accuracy of 0.894, correctly reducing 89.4% of words to their base forms.
- POS Tagger: The Random Forest Classifier achieved the highest F1-score of 0.84832, with macro-averaged accuracy of 0.98874. The HMM & Viterbi model achieved a macro F1 of 0.74465, and Logistic Regression achieved a macro F1 of 0.8213.
Limitations and Future Work
- Limited Scope: The current ParsiPy toolkit focuses on fundamental NLP tasks. More advanced tasks like Named Entity Recognition (NER) were not included in this initial phase.
- Data Scarcity: The limited availability of high-quality annotated data prevented the full leverage of state-of-the-art transformer-based models.
- Future Directions:
- Expand ParsiPy to include more NLP tasks.
- Incorporate cutting-edge deep learning techniques.
- Conduct more comprehensive error analysis.
- Develop a standardized Unicode representation for Pārsig.
- Create more linguistic resources, such as annotated corpora and lexicons.
- Enhance transliteration accuracy and expand deep learning models for P2T.
Summary updated