Abstract
This paper introduces Plot Retrieval, a novel information retrieval (IR) task requiring models to identify specific book passages based on abstract, human-authored descriptions rather than direct quotes or lexical matches. The authors present PLOTRETRIEVAL, a large-scale dataset characterized by minimal word overlap between queries and targets. Benchmarking reveals that while state-of-the-art models excel at traditional IR tasks, they struggle significantly with Plot Retrieval, highlighting a substantial performance gap between current AI and human cognitive intuition.
The Challenge of Abstract Semantic Association
Traditional Information Retrieval (IR) systems are highly effective at finding documents when a clear lexical (word-to-word) or direct semantic link exists. For instance, a query for "bit depth of a pixel" allows a model to easily locate passages containing those specific terms. However, human memory often functions through abstract summaries, emotional descriptions, or character-driven speculations that share little to no vocabulary with the source text.
Existing benchmarks like MS MARCO or Quora do not capture this abstract semantic association. Consequently, current models remain "lexically bound," struggling with the high-level reasoning that humans perform naturally when recalling a scene from a book.
Methodology and Dataset Construction
The authors formalize the task as retrieving a three-sentence "plot chunk" from a book given a query. To evaluate performance accurately, they introduced N-RODCG (Normalized Relative Offset Distance Discounted Cumulative Gain), a metric that rewards models for retrieving plots physically near the ground truth, reflecting a reader's actual "field of vision."
The PLOTRETRIEVAL dataset was constructed through a multi-stage process:
- Data Collection: Over 1 million public comments were scraped from online reading platforms (Kindle, iReader, Douban) where users commented on specific passages.
- Filtering: NLTK and ChatGPT were utilized to remove comments with high word overlap or purely emotional content.
- Annotation: Professional annotators refined comments into high-quality queries, and a BERT-based classifier was used to scale the training set to 400,000 pairs.
- Bilingual Alignment: The benchmark includes aligned Chinese and English versions of classic literature.
PLOTRETRIEVAL is unique for its low word overlap (19.62%), significantly lower than MS MARCO (46.29%) or FEVER (61.57%), making it a rigorous test of semantic understanding.
Results and the Human-Machine Gap
The authors benchmarked a variety of architectures, including lexical (BM25), sparse (DeepCT), dense (DPR, RetroMAE), and cross-encoders. While the Cross-Encoder emerged as the top performer, its MRR@1 remained low at 0.123.
A critical finding is the "Lexical Trap": even dense models rely heavily on surface-level similarities. When word overlap drops below 20%, model performance plummets. In contrast, human performance remains robust.
As shown in the figure above, while models are nearly as proficient as humans on traditional QA tasks, they fall behind by over 40% in Plot Retrieval. This suggests that current IR requires a shift toward deeper cognitive modeling to bridge the gap with human intuition.
Conclusion and Future Directions
The PLOTRETRIEVAL benchmark demonstrates that current IR models fail to capture the multi-layered understanding—encompassing summarization, emotion, and visual environment—that humans use for retrieval. Future research must move beyond single-vector embeddings toward multi-view modeling to explicitly represent these diverse abstract associations.
Summary updated