TL;DR
This paper introduces a novel transformer-based pipeline for automatically annotating Byzantine Greek marginal texts, which are akin to medieval social media. By developing a new language model and a meticulously annotated gold standard, the authors achieve significantly improved part-of-speech tagging accuracy compared to existing methods, paving the way for deeper linguistic and literary research into this challenging corpus.
What Problem Does This Paper Address?
The Byzantine Book Epigrams (DBBE) database contains a wealth of marginal writings from medieval manuscripts. These texts, while offering insights into the lives and thoughts of scribes, present significant challenges for natural language processing (NLP) due to their lack of normalization, frequent orthographic inconsistencies (often phonetic), and unique linguistic features. Existing NLP tools, typically trained on normalized Ancient Greek, perform poorly on this data. Manual annotation of the entire DBBE corpus is infeasible due to its size. Therefore, the paper addresses the need for an effective automatic annotation system to enable large-scale linguistic and literary analysis of this valuable historical data.
Methods
The core of the paper's methodology involves developing a transformer-based part-of-speech (POS) tagger. This process begins with training a language model, named DBBErt, specifically for Greek. To maximize performance, this model is trained on a comprehensive dataset including pre-Byzantine, Byzantine, and post-Byzantine Greek texts, as well as Modern Greek Wikipedia data. This enriched dataset helps the model capture the linguistic evolution and idiosyncrasies of the language.
Following language model training, the model is fine-tuned for POS tagging and full morphological analysis. This fine-tuning utilizes existing treebanks of Ancient Greek and a newly created gold standard evaluation set derived from the DBBE occurrences. The authors also employ a bootstrapping approach, using an existing POS tagger to pre-annotate tokens, aiding human annotators.
Datasets
- DBBE (Database of Byzantine Book Epigrams): The primary corpus for this research, containing approximately 12,000 epigrams.
- LM data set: A large dataset for language model training, comprising:
- Pre-Byzantine, Byzantine, and post-Byzantine Greek texts (from various corpora mentioned in Section 2, excluding TLG).
- Modern Greek Wikipedia data.
- Training set for POS tagger: A combination of existing treebanks (Ancient Greek Dependency Treebank, PROIEL, Gorman treebank, Pedalion Trees) totaling 1,132,120 Ancient Greek tokens.
- DBBE Evaluation Set: A newly compiled test set of 10,000 tokens from DBBE occurrences, manually annotated following AGDT guidelines. This serves as a novel benchmark for Byzantine Greek NLP.
Key Insights and Ideas
- Medieval Greek as "Social Media": The paper draws a compelling parallel between Byzantine book epigrams and modern social media posts (like Twitter) due to their brevity, personal nature, and phonetic writing styles. This analogy highlights the relevance of modern NLP techniques to historical texts.
- The Challenge of Unnormalized Data: The research underscores the significant difficulties posed by unnormalized, phonetically transcribed historical texts for NLP, a problem not adequately addressed by tools trained on standardized corpora.
- Importance of Contextual Embeddings: The success of the DBBErt language model, trained on a broad spectrum of Greek data, demonstrates the power of contextual embeddings in capturing the nuances of historical language variations.
- Phonetic Spelling and Word Formation Issues: The error analysis reveals that common issues in Byzantine texts, such as phonetic spelling (itacism) and erroneous word splitting/joining, mirror challenges found in contemporary social media language, suggesting a continuity in certain linguistic phenomena.
- Need for a Dedicated Byzantine Greek Gold Standard: The creation and validation of the DBBE evaluation set fill a critical gap, providing a much-needed resource for evaluating NLP models on this specific historical language variety.
Key Results
- The fine-tuned DBBErt POS tagger achieved an accuracy score of 69.89% on the challenging, unnormalized Byzantine Greek evaluation set.
- This accuracy is over 6 percentage points higher than the state-of-the-art RNN Tagger for Ancient Greek (63% accuracy).
- The developed POS tagger significantly outperforms the widely used rule-based system Morpheus, which achieved only a 19% accuracy on the same test set.
- An inter-annotator agreement study on the newly created gold standard showed very high agreement scores (92.72% for POS tagging and 89.83% for full morphological analysis), indicating the reliability and consistency of the annotation.
Limitations and Future Work
- Data Scarcity: Transformer models are data-hungry, and the limited availability of normalized Byzantine Greek data necessitated the inclusion of Modern Greek data, which might introduce some bias.
- Orthographic Irregularities and Word Formation: Non-existing words resulting from incorrect word splitting or joining significantly impact the morphological analysis performance, as sub-word tokenization can be misled.
- Future Work:
- Development of a lemmatizer to complete the annotation process.
- Research into measuring similarity between hemistichs, verses, and epigrams to link texts copied or altered by different scribes.
Summary updated