LitSearch: A Retrieval Benchmark for Scientific Literature Search

Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, Tianyu Gao

TL;DR

LitSearch is a new retrieval benchmark designed to reflect how researchers actually search for scientific literature, moving beyond noisy citation snippets to realistic, complex questions. The study reveals that while modern dense retrievers significantly outperform traditional keyword search, even top-tier commercial tools like Google Search struggle with the deep reasoning required to navigate the nuances of scientific papers.

What Problem Does This Paper Address?

When researchers look for papers, they don't usually search using a random snippet of text from another article. They ask specific questions: "Which method involves training additional prompt tokens for every layer?" or "Where can I find research on evaluating consistency in generated summaries?"

However, most existing "citation recommendation" benchmarks are built by taking a sentence from a paper, removing the citation, and asking a system to find the missing link. These queries are often noisy, overly broad, or heavily dependent on the surrounding context, making them poor proxies for real-world search behavior. Furthermore, existing benchmarks often fail to distinguish between the performance of top-tier embedding models, as many models achieve similar, high scores on older datasets like MS MARCO.

Methods

The authors developed LitSearch, a benchmark of 597 realistic literature search queries. To ensure the questions were both high-quality and challenging, they used a two-pronged construction strategy:

  1. GPT-4 Rewriting: They sampled paragraphs containing inline citations from the Semantic Scholar Open Research Corpus (S2ORC). GPT-4 was then prompted to transform these snippets into standalone, natural language questions that the cited paper would answer.
  2. Author-Written Queries: They invited authors from major AI conferences (ACL 2023 and ICLR 2024) to write search queries that specifically targeted their own recently published papers. This ensured the "ground truth" was accurate and reflected the core contributions of the work.
LitSearch query generation methods: GPT-4 rewriting of inline citations and author-written questions.

To prevent the task from being too easy, the team applied word-overlap filtering, removing questions that shared too many words with the target paper's title. Finally, every question was manually inspected and categorized by experts as either "Broad" (fitting up to 20 papers) or "Specific" (targeting 5 or fewer).

Datasets

  • The Benchmark: LitSearch consists of 597 queries (351 from inline citations, 246 from authors).
  • The Retrieval Corpus: A collection of 64,183 papers from the ACL Anthology and ICLR.
  • Evaluation Metrics: The primary metric used is Recall@K (specifically R@5 and R@20), measuring whether the correct paper appears in the top results.

Key Insights and Ideas

  • The "Full-Text" Paradox: Surprisingly, providing the retrieval models with the full text of papers (averaging ~6,000 words) often performed worse than just using the title and abstract (~134 words). Current embedding models seem to struggle with "signal-to-noise" issues in long-form scientific text.
  • Commercial Search Gap: In a human study, commercial tools like Google Search and Elicit lagged behind the best dense retrievers by up to 32 recall points on specific queries. This suggests that general-purpose search engines aren't yet optimized for the deep, conceptual reasoning required for specialized scientific search.
  • A Better Ruler: LitSearch proved to be more "sensitive" than older benchmarks. While models like E5 and GritLM might look similar on MS MARCO, LitSearch revealed a 15-point performance gap between them, making it a more effective tool for evaluating the next generation of retrieval systems.
  • Reranking Wins: Using a Large Language Model (like GPT-4o) to "re-rank" the top 100 results from a standard retriever consistently improved performance, especially when the LLM was allowed to look at citations ("one-hop" reranking).

Key Results

  • Dense vs. Sparse: Modern dense retrievers (like GritLM) crushed the traditional BM25 keyword search algorithm, showing a 24.8% absolute improvement in Recall@5.
  • Top Performer: GritLM-7B achieved the best standalone performance with a 74.8% average Recall@5 on specific questions.
  • LLM Boost: Applying GPT-4o reranking on top of GritLM results pushed the performance even higher, reaching 79.2% Recall@5.
  • Specificity Matters: Systems generally performed better on "Specific" queries than "Broad" ones, likely because specific queries have fewer "distractor" papers that look superficially relevant.

Limitations and Future Work

The authors acknowledge several constraints:

  • Domain Specificity: The current version of LitSearch is focused entirely on the fields of Machine Learning (ML) and Natural Language Processing (NLP).
  • Language: The benchmark is currently limited to English-language papers and queries.
  • Model Context: Most existing embedding models have limited context windows, which may explain why full-text retrieval underperformed. Future work could explore models specifically designed for long-document processing.
  • Scale: While 64k papers is a solid testbed, it is a fraction of the millions of papers available globally; scaling the corpus remains a challenge for future iterations.

Summary updated

Abstract

Literature search questions, such as "where can I find research on the evaluation of consistency in generated summaries?" pose significant challenges for modern search engines and retrieval systems. These questions often require a deep understanding of research concepts and the ability to reason over entire articles. In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers. LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions about recently published papers, manually written by their authors. All LitSearch questions were manually examined or edited by experts to ensure high quality. We extensively benchmark state-of-the-art retrieval models and also evaluate two LLM-based reranking pipelines. We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% difference in absolute recall@5. The LLM-based reranking strategies further improve the best-performing dense retriever by 4.4%. Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by 32 points. Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.
BibTeX
@inproceedings{ajith2024litsearch,
  author = {Anirudh Ajith and Mengzhou Xia and Alexis Chevalier and Tanya Goyal and Danqi Chen and Tianyu Gao},
  title = {{LitSearch: A Retrieval Benchmark for Scientific Literature Search}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.840},
}
Figure