TL;DR
The authors introduce LitSearch, a new retrieval benchmark consisting of 597 realistic, high-quality literature search queries designed to mimic how researchers actually look for scientific papers. The evaluation reveals that while modern dense retrievers significantly outperform traditional keyword matching (BM25), commercial search tools like Google Scholar and Elicit struggle with complex, concept-driven queries.
What Problem Does This Paper Address?
When researchers look for related work or try to recall a paper they previously saw, they don't just search for exact keywords. They ask complex, conceptual questions like, "Which method involves training additional prompt tokens for every layer during the fine-tuning of language models?"
Prior benchmarks for citation recommendation usually simplified this task by using raw text surrounding an inline citation (e.g., "We build on [citation]") as the query. However, these snippets are often noisy, overly broad, or highly context-dependent. To bridge this gap, the authors address the need for a realistic, natural language search benchmark that requires deep domain expertise and reasoning across entire scientific articles.
Methods
The authors constructed LitSearch using a two-pronged data collection strategy to ensure both realism and difficulty:
- Inline-Citation Transformation: They sampled paragraphs containing inline citations from academic papers and used GPT-4 to rewrite them into standalone literature search questions. To prevent the task from being trivial, they filtered out questions with high word overlap with the target paper's title.
- Author-Written Queries: They invited authors of recently published papers at top AI conferences (ACL 2023 and ICLR 2024) to manually write challenging search queries that their own papers uniquely answer.
Every generated and collected question underwent rigorous manual inspection and filtering by experts to ensure high quality, factual accuracy, and appropriate specificity.
For evaluation, the authors benchmarked standard lexical retrieval (BM25), state-of-the-art dense retrievers (GTR, Instructor, E5, GritLM), and two LLM-based reranking strategies (Vanilla and One-hop reranking using GPT-4o).
Datasets
The benchmark introduces LitSearch, which contains:
- 597 questions in total (351 derived from inline citations, 246 written by authors).
- Questions are categorized by specificity: Specific (matched with $\le$ 5 target papers) and Broad (matched with $\le$ 20 target papers).
- Retrieval Corpus: A collection of 64,183 computer science papers (59,383 from the ACL Anthology and 4,807 from ICLR) extracted from the Semantic Scholar Open Research Corpus (S2ORC).
Key Insights and Ideas
- The Full-Text Paradox: Surprisingly, encoding the full text of papers (averaging ~6,041 words) did not consistently improve retrieval performance compared to using just titles and abstracts (~134 words). This is likely because modern embedding models are predominantly trained on short-text datasets (like MSMARCO), making them less effective at processing long-form scientific documents.
- Annotator Bias in Sourcing: Author-written questions were found to have a higher word overlap with target papers (0.43) than inline-citation questions (0.33). Authors naturally reuse their own terminology and focus on main abstract findings, making author-sourced questions slightly easier for retrievers.
- High Benchmark Sensitivity: LitSearch is highly effective at differentiating the capabilities of top-tier embedding models. For instance, while GritLM and E5 perform almost identically on the standard MSMARCO benchmark, LitSearch exposes a massive 15-point performance gap between them.
Key Results
- Dense Retrievers Dominate BM25: The best dense retriever, GritLM-7B, achieved an average recall@5 of 74.8% on specific questions, outperforming traditional BM25 by an absolute 24.8%.
- Reranking Provides a Boost: Applying GPT-4o to rerank the top retrieved results further improved GritLM's recall@5 by 4.4%.
- Commercial Search Engines Struggle: In a human study evaluating popular tools on a subset of 80 specific questions, Google Search, Google Scholar, and Elicit performed poorly on inline-citation questions, with Google Scholar scoring a low 20.5% recall@5 compared to GritLM's 67.7% on the same subset.
Limitations and Future Work
- Dataset Scope: LitSearch is currently limited to English-language questions and research papers primarily within the ML and NLP domains.
- Query Difficulty: Despite expert filtering, some author-written questions turned out to be easier than expected due to terminology overlap.
- Evaluation Boundaries: The authors note that their evaluation was not completely exhaustive, leaving room for future work to test more sophisticated multi-vector retrieval or advanced agentic search pipelines on this benchmark.
Summary updated