TL;DR
This paper investigates how well dependency parsers, trained on modern Italian, perform on Old Italian texts like Dante's Divine Comedy. They found that using in-domain training data (sections of the Divine Comedy itself) significantly boosts parsing accuracy compared to using general modern Italian datasets. This highlights the importance of domain-specific training for historical language processing.
What Problem Does This Paper Address?
Dependency parsing is a crucial task in Natural Language Processing (NLP) that analyzes the grammatical structure of sentences by identifying relationships between words. While significant progress has been made in parsing modern languages, applying these tools to historical languages, like Old Italian, presents a challenge. Old Italian, exemplified by Dante Alighieri's Divine Comedy, has distinct linguistic features that can cause modern Italian parsers to perform poorly. The authors aim to quantify this performance gap and explore whether training parsers on the Divine Comedy itself can improve accuracy for this specific historical text and potentially similar texts. They also investigate the internal linguistic consistency across the different parts (Cantiche) of the Divine Comedy.
Methods
The researchers employed two popular dependency parsing toolkits: UDPipe1 and Stanza. They conducted experiments in two main phases:
- Modern Italian Training: They first trained parsers using existing Universal Dependencies (UD) treebanks for Modern Italian. These models were then used to parse sections of the Divine Comedy.
- In-Domain Training: They then created a training set from portions of the Divine Comedy itself (specifically, Canti IV-XXXIV of Inferno) and retrained the parsers. These in-domain models were then tested on different sections of the Divine Comedy.
To evaluate the linguistic similarity within the Divine Comedy, they also performed correlation analyses using:
- Part of Speech (PoS) Tri-grams: Analyzing sequences of three consecutive PoS tags to see how similar the linguistic patterns are across different sections.
- Sub-tree Label Attachment: Examining the dependency relations between words to assess structural similarity.
Datasets
- Training Data:
- Modern Italian UD treebanks (ISDT, VIT, Par-TUT).
- Sections of Dante Alighieri's Divine Comedy (specifically Canti IV-XXXIV of Inferno for in-domain training).
- Test Data:
- Sections of Dante Alighieri's Divine Comedy (Inferno I-III, Purgatorio I-III, Paradiso I-III).
- A poem by Guido Cavalcanti ("Voi che per li occhi mi passaste il core") for out-of-domain testing.
The paper leverages the DanteSearch corpus for initial linguistic annotation and the Universal Dependencies (UD) framework for syntactic annotation. The Divine Comedy itself serves as the sole Old Italian treebank in the UD collection.
Key Insights and Ideas
- Domain Specificity is King: The most significant finding is that training parsers on data from the same domain (i.e., the Divine Comedy itself) dramatically improves parsing accuracy compared to using general modern Italian data. This underscores the challenge of applying NLP tools across significant linguistic shifts like those between modern and Old Italian.
- Internal Consistency: The analysis of PoS tri-grams and sub-tree labels suggests that the first three Canti of Paradiso are linguistically closer to Inferno than they are to Purgatorio. This hints at potential structural or stylistic differences within the Divine Comedy itself.
- Old Italian is Different: The performance drop when using modern Italian parsers on the Divine Comedy is substantial, indicating that Old Italian syntax and vocabulary are sufficiently distinct to warrant specialized models.
- Beyond the Divine Comedy: Testing an in-domain trained model on a contemporary Old Italian text (Guido Cavalcanti's poem) also showed superior performance, suggesting that Divine Comedy-trained models might be beneficial for other Old Italian texts of a similar style and period.
Key Results
- UAS/LAS Boost: Training parsers on the Divine Comedy data resulted in an approximate +17% increase in Unlabeled Attachment Score (UAS) and +25-30% increase in Labeled Attachment Score (LAS) compared to models trained on Modern Italian data.
- Specific Improvements: For the Stanza model, the boost was around 19.81% for UAS and 29.21% for LAS when trained on the Divine Comedy versus Modern Italian. UDPipe1 saw increases of 14.35% for UAS and 16.56% for LAS.
- Modern Italian Baseline: Parsers trained on Modern Italian data achieved UAS scores around 65.28 and LAS scores around 56.98 when applied to Inferno.
- Cavalcanti Poem: The Stanza model trained on the Divine Comedy achieved significantly higher scores (UAS: 86.49, LAS: 75.68) on Cavalcanti's poem compared to a model trained on Modern Italian data (UAS: 66.37, LAS: 48.65), showing a boost of over 20% in UAS and 27% in LAS.
- Cantiche Comparison: In-domain models showed higher scores for Inferno compared to Purgatorio and Paradiso, with Paradiso generally outperforming Purgatorio.
Limitations and Future Work
- Limited Old Italian Data: The primary limitation is the scarcity of annotated Old Italian data. The Divine Comedy itself is the only Old Italian treebank available in UD, and it only covers the Inferno cantica.
- Representativeness of Subsets: While the first three Canti were shown to be representative of their respective Cantiche through correlation analysis, the authors acknowledge that this doesn't necessarily extend to the entire Cantica or the poem as a whole.
- Genre Diversity: The authors note that the Divine Comedy is a poetic text, and its genre might influence parsing performance. They suggest that future work should investigate how to incorporate data from different genres (e.g., non-poetic Old Italian literature, Modern Italian poetry) to create more robust models.
- Merging Datasets: A promising avenue for future research is to explore augmenting training data by merging Old and Modern Italian datasets, despite genre diversity, to potentially enhance parsing accuracy.
- Broader Textual Domain: Further studies are needed to test the representativeness of the Divine Comedy model on a wider range of authors and texts from the same period.
Summary updated