Abstract
There is a large volume of late antique and medieval Hebrew texts. They represent a crucial linguistic and cultural bridge between Biblical and modern Hebrew. Poetry is prominent in these texts and one of its main characteristics is the frequent use of metaphor. Distinguishing figurative and literal language use is a major task for scholars of the Humanities, especially in the fields of literature, linguistics, and hermeneutics. This paper presents a new, challenging dataset of late antique and medieval Hebrew poetry with expert annotations of metaphor, as well as some baseline results, which we hope will facilitate further research in this area.
Introduction
Metaphor detection is a cornerstone of natural language understanding, yet it remains a "low-resource" challenge for many languages—especially those with long, complex histories like Hebrew. While modern Hebrew has some NLP tools, ancient and medieval variants are often left behind.
The authors target Piyyut, a genre of Hebrew liturgical poetry from the 5th to 8th centuries CE. These texts are a vital linguistic bridge between Biblical and Modern Hebrew, but they are incredibly dense with figurative language. For scholars in the Digital Humanities, identifying these metaphors manually is a grueling, expert-level task. This paper addresses the lack of gold-standard datasets and automated tools for detecting metaphors in this specific, culturally significant corpus.
The Piyyut Dataset
The study introduces a new, expert-annotated dataset consisting of two units recovered from the Cairo Genizah:
- Pre-Classical Piyyut: 172 poems from the 5th century CE (Galilee).
- Pinchas: 137 poems by the 8th-century poet Pinchas Ha-Cohen (Tiberias).
Totaling 73,179 words across 309 poems, the dataset was annotated by a Hebrew literature professor using the CATMA tool. This is currently the only metaphor-annotated corpus available for any stage of the Hebrew language. Metaphors are relatively frequent in this corpus, accounting for approximately 16-21% of the words.
Methodology
The authors framed metaphor detection as a sequence labeling task using a BIO (Beginning, Inside, Outside) scheme to handle multi-word metaphorical expressions. They evaluated several transformer-based approaches:
- AlephBERT: Pre-trained on 1.9 billion words of modern Hebrew.
- BEREL: Pre-trained on a smaller corpus (220 million words) of ancient Jewish texts (Sefaria and Dicta libraries).
- Domain Adaptation: Further training AlephBERT on unlabeled ancient texts from Project Ben-Yehuda and the Piyyut corpus.
- Weighted Cross-Entropy (WCE): Used to penalize missing metaphors more heavily, boosting recall in an imbalanced dataset where metaphors are relatively rare.
Key Results and Insights
The best-performing model was BEREL + WCE, which achieved F1 scores of 48.7 on the Pre-Classical corpus and 49.4 on the Pinchas corpus.
Several key insights emerged from the experiments:
- Domain Trumps Scale: Despite AlephBERT's larger training set, BEREL performed better, suggesting that for specialized tasks like medieval poetry, the style and vocabulary of pre-training data are more critical than sheer volume.
- Generalization: Over 50% of metaphorical expressions in the test sets were novel, indicating the models learned to identify figurative context rather than just memorizing a list.
- The Subjectivity Barrier: Inter-annotator agreement (Cohen’s kappa ~0.62) highlights the inherent ambiguity of poetic language, where expressions like "river of fire" can be literal in myth or metaphorical in poetry.
- WCE is Essential: Weighting the loss function was the key to achieving usable F1 scores, preventing the models from defaulting to "literal" predictions for almost everything.
Conclusion and Future Work
While the F1 scores are lower than standard NLP tasks like Part-of-Speech tagging, they represent a significant leap over baselines and are comparable to results in other languages for similar complex genres. Future work aims to expand the dataset, explore "soft labels" for poetic ambiguity, and increase granularity to include subtypes like metonymy, synecdoche, and personification.
Summary updated