Abstract: There is a large volume of late antique and medieval Hebrew texts. They represent a crucial linguistic and cultural bridge between Biblical and modern Hebrew. Poetry is prominent in these texts and one of its main characteristics is the frequent use of metaphor. Distinguishing figurative and literal language use is a major task for scholars of the Humanities, especially in the fields of literature, linguistics, and hermeneutics. This paper presents a new, challenging dataset of late antique and medieval Hebrew poetry with expert annotations of metaphor, as well as some baseline results, which we hope will facilitate further research in this area.
Introduction
Metaphor is a fundamental element of poetic expression, yet its computational detection remains a complex challenge, particularly for pre-modern languages. This paper addresses a significant gap in Digital Humanities for medieval Hebrew by introducing a novel dataset of liturgical poetry (Piyyut) from the 5th to 8th centuries. These texts are crucial for understanding the linguistic evolution from Biblical to Modern Hebrew but are characterized by dense, figurative language that has traditionally required grueling manual expert annotation. By automating this process, the authors aim to provide a benchmark for machine learning models in low-resource, ancient linguistic contexts.
Dataset and Annotation
The researchers developed an expert-annotated dataset comprising two distinct corpora: Pre-Classical Piyyut (anonymous 5th-century poems from Galilee) and poems by the 8th-century author Pinchas Ha-Cohen. The dataset totals over 73,000 words across 309 poems. Annotation was performed by a professor of Hebrew literature to ensure a "gold standard" that accounts for the complex nuances of medieval poetic tradition.
An inter-annotator agreement study yielded a Cohen’s kappa of ~0.62, which is considered "substantial" in NLP but highlights the inherent subjectivity and "fuzziness" of identifying metaphors in complex poetry.
Methodology
Metaphor detection is framed as a sequence labeling task using the BIO (Beginning, Inside, Outside) scheme. The study compares two primary transformer-based architectures:
- AlephBERT: Pre-trained on 1.9 billion words of modern Hebrew.
- BEREL: Pre-trained on 220 million words of ancient Jewish texts.
To address the class imbalance—where metaphors constitute only about 16–21% of the tokens—the authors utilized Weighted Cross-Entropy (WCE) loss. This approach prioritizes recall by penalizing the omission of metaphorical tags more severely than false positives. They also experimented with domain adaptation, further training AlephBERT on unlabeled ancient texts to bridge the stylistic gap.
Experimental Results
The experimental results demonstrate that domain-specific pre-training is more critical than raw data volume. BEREL, despite being trained on a corpus nearly ten times smaller than AlephBERT, consistently achieved superior performance due to its exposure to the specific vocabulary and syntax of ancient Rabbinic texts.
The best-performing configuration, BEREL with WCE, reached F1 scores of approximately 48.7 and 49.4. These results represent a massive leap over naïve baselines, such as the "most frequent tag" baseline (F1 of 24). Analysis suggests the models are learning contextual figurative patterns rather than simply memorizing vocabulary; over 50% of the metaphorical expressions in the test set were unseen during training.
Discussion and Future Work
While the current models provide a significant baseline, the inherent ambiguity of poetic language suggests a performance ceiling for current architectures. The authors acknowledge that the dataset is still relatively small by modern deep-learning standards.
Future research directions include expanding the dataset to include later medieval periods and different genres, as well as moving beyond binary detection to classify specific types of figurative language, such as metonymy or personification. Ultimately, these tools could assist scholars in processing the vast remaining treasury of the Cairo Genizah.
Summary updated