TL;DR
Researchers have introduced LINEAGE, a massive new benchmark designed to evaluate how well AI agents can retrieve the "empirical lineage" of a scientific paper—specifically, the web of prior methods used as experimental baselines and the later works that compared against them. By moving beyond simple topical relevance to structural, comparison-based relationships, LINEAGE reveals that even our best current retrieval systems struggle to capture the full experimental context of a research contribution.
What Problem Does This Paper Address?
When you're writing a paper or planning an experiment, "topical relevance" isn't enough. You don't just need papers about the same topic; you need to know exactly which prior methods are the standard baselines you must beat.
Existing benchmarks for scientific literature retrieval usually rely on two signals:
- Topical Relevance: "Are these two papers about LLMs?"
- Citation Links: "Does Paper A cite Paper B?"
The problem is that citations are "noisy." A paper might cite another for background info, a specific dataset, or a software tool, without ever actually comparing against its core method. This makes it incredibly difficult for automated research agents to synthesize literature or plan experiments effectively. The authors propose empirical lineage—a directed relationship where one paper explicitly uses another's method as an empirical comparison baseline—as a much higher-signal ground truth for retrieval.
Methods
To build this benchmark at scale, the authors developed LINEAGE-Constructor, an automated pipeline that turns a messy sea of PDFs into a clean, calibrated graph of research relationships.
The pipeline consists of three main stages:
- Unified Literature Collector: It integrates 11 different open-access sources (like DBLP, arXiv, and OpenReview) to build a consistent corpus of 121,048 papers from 17 top AI venues (2018–2025).
- Evidence-Grounded Lineage Extraction: This is the "brain" of the system. It uses a dual-source approach, parsing PDFs with both GROBID and MinerU. An LLM (Qwen3.5-27B) then extracts lineage relations by looking for explicit evidence in tables and experimental sections.
- Graph Calibration: To ensure the resulting graph is useful for evaluation, the system enforces temporal consistency (a paper can't use a future paper as a baseline) and structural consistency (pruning "hubs" or overly broad connections that would make retrieval trivial or meaningless).
Datasets
The resulting LINEAGE benchmark is substantial:
- Corpus: 121,048 papers across 17 top-tier AI venues.
- Graph: 67,502 paper nodes connected by 164,948 lineage edges.
- Queries: 80,000 multi-facet queries. Instead of just using a paper's title, they generate eight different query types (e.g., "topical semantic," "methodological facet," "context keyword") to simulate different ways a researcher might search.
- Validation: They also created LINEAGE-HA, a human-annotated set of 230 papers to verify the accuracy of their automated extraction.
Key Insights and Ideas
- Lineage is Harder than Topicality: Traditional search engines might find "similar" papers, but they often fail to find the specific "ancestors" (prior baselines) or "successors" (later comparisons) that define a paper's place in the field.
- The "Neighborhood" Matters: The authors argue that retrieval shouldn't just be measured by "hits." They introduce structural metrics to see if a system can recover the entire neighborhood of related experimental comparisons.
- Distance Decays Performance: Retrieval accuracy drops off a cliff as you move further away in the lineage graph. Finding a direct baseline (1-hop) is much easier than finding the "grandfather" of a method (3-hops).
- Directional Bias: Current models are significantly better at finding ancestors (the papers a query paper compared against) than successors (later papers that used the query paper as a baseline). This suggests that "forward-looking" discovery is a major weakness in current AI research agents.
Key Results
The authors tested 19 representative retrieval systems, including sparse retrievers (BM25), dense retrievers (like BGE-M3 and Qwen-Embedding), and agentic systems (like Tongyi-DeepResearch).
- Low Ceiling: Even the strongest agentic retriever achieved only 31.87% Recall@20. This means that for a typical paper, the best AI tools today miss roughly two-thirds of the relevant empirical comparison context.
- Rapid Decay: For 1-hop targets (direct baselines), the average Recall@20 was 28.42%. By 3-hops, this plummeted to just 6.68%.
- Structural Failure: While some systems (like Google Search) get occasional "hits," their structural scores are very low, meaning they fail to recover the coherent clusters of papers that actually form a research lineage.
Limitations and Future Work
The current version of LINEAGE is restricted to AI conference papers from 2018 to 2025. The authors acknowledge that expanding this to journals and other scientific domains (like medicine or physics) is a necessary next step, though it faces challenges with PDF availability and licensing.
Future work aims to move beyond simple retrieval to interactive lineage exploration, where an agent could help a researcher navigate the "evolutionary tree" of a specific method to find the most robust and relevant baselines for a new project.
Summary updated