TL;DR
This paper investigates the feasibility of automatically detecting "allusive text reuse" – subtle literary connections between texts that lack significant lexical overlap. The authors propose an Information Retrieval (IR) framework and explore the impact of lexical semantics and annotation strategies, finding that while inter-annotator agreement is low, semantic information can moderately improve detection.
What Problem Does This Paper Address?
Literary scholars have long been interested in "intertextuality," the phenomenon of texts referencing or alluding to other texts. While direct quotations or paraphrases are relatively easy to detect computationally, "allusive text reuse" presents a significant challenge. These allusions often rely on very few shared words, making traditional text matching methods (like n-grams or fingerprinting) ineffective. Furthermore, the subjective nature of identifying allusions makes it difficult to create standardized benchmark datasets for evaluation. This paper aims to bridge this gap by exploring the feasibility of automated detection for these subtle forms of text reuse.
Methods
The authors frame the problem as an Information Retrieval task, where referencing texts (queries) are used to find referenced texts (documents). They explore several approaches:
- Lexical Approaches: Standard methods like TF-IDF (Term Frequency-Inverse Document Frequency) and a hand-crafted scoring function (Tesserae) are used as baselines.
- Semantic Approaches: This includes using word embeddings (FastText) to represent words and sentences, and calculating similarity using methods like Word Mover's Distance (WMD).
- Hybrid Approaches: The core innovation lies in combining lexical and semantic information. They introduce "soft cosine similarity," a flexible method that can incorporate external linguistic resources (like WordNet or word embeddings) to define similarity between words. This allows for a more nuanced understanding of semantic relationships beyond simple word overlap.
A key methodological contribution is the development of an adaptation of Fleiss's kappa to measure inter-annotator agreement for the task of segmenting the "query" text that best captures the allusion.
Datasets
The primary dataset used in this study is a subset of manually identified biblical references from the works of Bernard of Clairvaux. This dataset was derived from the BiblIndex project. The references are categorized into quotations, mentions, and allusions, with allusions being the focus due to their low lexical overlap. The authors also created a new benchmark dataset by having annotators delineate the precise textual span of the allusion, which was crucial for their IR experiments.
Key Insights and Ideas
- The Challenge of Subjectivity: Inter-annotator agreement for identifying the precise span of an allusion is surprisingly low (kappa = 0.22). This highlights the inherent interpretative nature of allusive text reuse and the difficulty in creating perfectly objective annotations.
- Manual Segmentation Matters: Despite low agreement, manually segmented queries significantly outperform automatic window-based segmentation for retrieval. This suggests that precisely identifying the relevant textual fragment is critical.
- Lexical Semantics Boosts Performance: While purely lexical methods (like TF-IDF) perform surprisingly well, incorporating semantic information, particularly through soft cosine similarity with word embeddings, offers a notable improvement.
- Soft Cosine's Flexibility: The soft cosine similarity function proves to be a powerful tool for combining lexical and semantic information, demonstrating its potential for capturing nuanced relationships.
- Word Embeddings are Key: FastText embeddings, which capture morphological relations, outperform word2vec embeddings, suggesting that richer semantic representations are beneficial.
Key Results
- The best performing model, SCemb (soft cosine with FastText embeddings), achieved a Mean Reciprocal Rank (MRR) of 21.95 and a Precision@20 (P@20) of 47.60.
- A simple TfIdf baseline on lemmatized input performed strongly, outperforming most other purely lexical and purely semantic models.
- Hybrid approaches, particularly those using soft cosine, consistently outperformed purely lexical or purely semantic models.
- Manual segmentation of queries resulted in significantly better retrieval performance than automatic window-based segmentation. For example, SCemb with manual segmentation achieved MRR of 21.95, while with a 10-word window it dropped to 13.98.
Limitations and Future Work
- Low Inter-Annotator Agreement: The authors acknowledge that the low agreement score raises doubts about the overall feasibility of the task, though they also note that some instances showed perfect agreement, suggesting variability at the instance level.
- Query Construction: The paper highlights the significant challenge of automatic query construction for allusive reuse detection, suggesting this is a crucial area for future research.
- Unrestricted Data: The evaluation was performed on a dataset where queries are guaranteed to have a match. The authors note that performance on unrestricted data (where most text is not allusive) would be different.
- Future Directions: The authors suggest exploring cross-lingual allusive text reuse detection using multilingual word embeddings and expanding the use of semantic similarity measures beyond synonymy. They also emphasize the need for further research into automatic query construction.
Summary updated