Plot Retrieval as an Assessment of Abstract Semantic Association

Shicheng Xu, Liang Pang, Jiangnan Li, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, Jie Zhou

Abstract

This paper introduces Plot Retrieval, a novel information retrieval (IR) task requiring models to identify specific book passages based on abstract, human-authored descriptions rather than direct quotes or lexical matches. The authors present PLOTRETRIEVAL, a large-scale dataset characterized by minimal word overlap between queries and targets. Benchmarking reveals that while state-of-the-art models excel at traditional IR tasks, they struggle significantly with Plot Retrieval, highlighting a substantial performance gap between current AI and human cognitive intuition.

The Challenge of Abstract Semantic Association

Traditional Information Retrieval (IR) systems are highly effective at finding documents when a clear lexical (word-to-word) or direct semantic link exists. For instance, a query for "bit depth of a pixel" allows a model to easily locate passages containing those specific terms. However, human memory often functions through abstract summaries, emotional descriptions, or character-driven speculations that share little to no vocabulary with the source text.

Existing benchmarks like MS MARCO or Quora do not capture this abstract semantic association. Consequently, current models remain "lexically bound," struggling with the high-level reasoning that humans perform naturally when recalling a scene from a book.

The conceptual gap between traditional lexical/semantic matching and the abstract associations required for Plot Retrieval.

Methodology and Dataset Construction

The authors formalize the task as retrieving a three-sentence "plot chunk" from a book given a query. To evaluate performance accurately, they introduced N-RODCG (Normalized Relative Offset Distance Discounted Cumulative Gain), a metric that rewards models for retrieving plots physically near the ground truth, reflecting a reader's actual "field of vision."

The PLOTRETRIEVAL dataset was constructed through a multi-stage process:

  1. Data Collection: Over 1 million public comments were scraped from online reading platforms (Kindle, iReader, Douban) where users commented on specific passages.
  2. Filtering: NLTK and ChatGPT were utilized to remove comments with high word overlap or purely emotional content.
  3. Annotation: Professional annotators refined comments into high-quality queries, and a BERT-based classifier was used to scale the training set to 400,000 pairs.
  4. Bilingual Alignment: The benchmark includes aligned Chinese and English versions of classic literature.

PLOTRETRIEVAL is unique for its low word overlap (19.62%), significantly lower than MS MARCO (46.29%) or FEVER (61.57%), making it a rigorous test of semantic understanding.

Results and the Human-Machine Gap

The authors benchmarked a variety of architectures, including lexical (BM25), sparse (DeepCT), dense (DPR, RetroMAE), and cross-encoders. While the Cross-Encoder emerged as the top performer, its MRR@1 remained low at 0.123.

A critical finding is the "Lexical Trap": even dense models rely heavily on surface-level similarities. When word overlap drops below 20%, model performance plummets. In contrast, human performance remains robust.

Accuracy comparison between IR models and humans across MS MARCO, Open-domain QA, and Plot Retrieval.

As shown in the figure above, while models are nearly as proficient as humans on traditional QA tasks, they fall behind by over 40% in Plot Retrieval. This suggests that current IR requires a shift toward deeper cognitive modeling to bridge the gap with human intuition.

Conclusion and Future Directions

The PLOTRETRIEVAL benchmark demonstrates that current IR models fail to capture the multi-layered understanding—encompassing summarization, emotion, and visual environment—that humans use for retrieval. Future research must move beyond single-vector embeddings toward multi-view modeling to explicitly represent these diverse abstract associations.

Summary updated

Abstract

Retrieving relevant plots from the book for a query is a critical task, which can improve the reading experience and efficiency of readers. Readers usually only give an abstract and vague description as the query based on their own understanding, summaries, or speculations of the plot, which requires the retrieval model to have a strong ability to estimate the abstract semantic associations between the query and candidate plots. However, existing information retrieval (IR) datasets cannot reflect this ability well. In this paper, we propose PlotRetrieval, a labeled dataset to train and evaluate the performance of IR models on the novel task Plot Retrieval. Text pairs in PlotRetrieval have less word overlap and more abstract semantic association, which can reflect the ability of the IR models to estimate the abstract semantic association, rather than just traditional lexical or semantic matching. Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and cross-encoder methods compared with human studies on PlotRetrieval show current IR models still struggle in capturing abstract semantic association between texts. PlotRetrieval can be the benchmark for further research on the semantic association modeling ability of IR models.
BibTeX
@inproceedings{xu2024plot,
  author = {Shicheng Xu and Liang Pang and Jiangnan Li and Mo Yu and Fandong Meng and Huawei Shen and Xueqi Cheng and Jie Zhou},
  title = {{Plot Retrieval as an Assessment of Abstract Semantic Association}},
  booktitle = {Student Research Workshop @ ACL},
  year = {2024},
  url = {https://openreview.net/forum?id=IfKWt8X1vu},
}
Figure