TL;DR
The paper introduces CiteME, a high-quality, human-curated benchmark designed to evaluate whether language models can accurately identify the specific scientific paper behind a given text claim. While standard frontier LMs struggle significantly on this open-ended task, the authors introduce CiteAgent—an autonomous agent that searches and reads full-text papers—narrowing the gap toward human-level citation verification.
What Problem Does This Paper Address?
With tens of thousands of papers uploaded to repositories like arXiv every month, researchers face severe information overload. Verifying scientific claims and finding the exact source paper is a tedious, time-consuming task. Naturally, we might want to use Large Language Models (LMs) as automated research assistants to handle this claim attribution.
However, evaluating LMs on this task has been nearly impossible with existing citation benchmarks. Prior datasets were automatically generated, making them highly flawed: they are riddled with formatting errors, contain "trivial" examples (where the author's name or paper acronym is explicitly left in the text), or are highly ambiguous (where multiple papers could fit the description). Furthermore, older benchmarks frame citation as a simple retrieval task from a small, pre-selected pool of abstracts, rather than an open-ended search across the entire global scientific corpus.
Methods
To mimic how a human researcher tracks down a source, the authors developed CiteAgent, an autonomous system built on top of frontier LMs (like GPT-4o).
Instead of relying purely on its internal weights, CiteAgent uses a ReAct-style (Reasoning + Acting) loop to interact with the real-world internet via the Semantic Scholar search engine. It operates using three custom commands:
search(query, sort): Queries the search engine and returns a list of the top 10 papers, including their IDs, titles, abstracts, and citation counts.read(ID): Fetches and parses the full text of open-access PDFs, allowing the model to look beyond just the abstract.select(ID): Finalizes the choice, records the answer, and ends the session.
The agent can perform up to 15 consecutive actions per sample, iteratively refining its search queries based on what it reads.
Datasets
The core contribution of this work is CiteME (Citation for Model Evaluation), a manually curated benchmark consisting of text excerpts from recent machine learning publications.
To build CiteME, a team of ML graduate students selected excerpts that unambiguously reference exactly one target paper. Crucially, the authors implemented a "Google-Proof" filtering process: they ran every candidate excerpt through GPT-4o five times without internet access. If the model could correctly guess the paper purely from memory, the instance was discarded. This left a robust dataset of 130 highly challenging, tool-dependent questions.
The authors also analyzed 50-sample subsets of four popular existing datasets (FullTextPeerRead, ACL-200, RefSeer, and arXiv) to quantify their limitations against CiteME.
Key Insights and Ideas
- Prior Benchmarks are Noisy: The authors' analysis revealed that older, automatically curated datasets contain an average of 36.5% ambiguous and 28.5% completely unattributable excerpts, making them poor metrics for true citation capability.
- Retrieval Models Fail at Scale: State-of-the-art embedding-based retrieval models like SPECTER2 achieved 0% accuracy on CiteME. Because CiteME requires searching across a massive corpus of 218 million papers using only a short text snippet, standard vector-similarity matching on titles and abstracts completely breaks down.
- LMs Need to Read the Full Text: LMs perform poorly when restricted to searching abstracts alone. Giving an agent the ability to "read" the full body of a paper allows it to cross-reference fine-grained technical details mentioned in the source excerpt.
- The "Citing Paper" Trap: A common failure mode for LMs is stopping one step too early. The agent will frequently find a paper that discusses or cites the target paper, mistake it for the source, and select it instead of continuing the search for the true origin.
Key Results
The paper evaluates several frontier models on the open-ended CiteME dataset:
- Human Experts: Achieved 69.7% accuracy, taking an average of just 38.2 seconds per instance.
- Standard LMs (No Tools): Achieved between 4.2% and 18.5% accuracy (and 0% on the strictly filtered GPT-4o subset), proving that internal knowledge bases are insufficient for precise scientific attribution.
- CiteAgent (GPT-4o + Search + Read): Achieved 35.3% accuracy, significantly closing the human-machine gap.
- Next-Gen Reasoning Models: OpenAI's
o1-previewreached 61.3% accuracy using CiteAgent commands. However, it also scored 38.7% without any tools, indicating that it likely memorized a significant portion of the evaluation papers during its training phase.
Limitations and Future Work
- Context Window Bottlenecks: Full-text papers contain thousands of tokens. Models with smaller context windows, such as LLaMA-3-70B (limited to 8k tokens), were entirely incompatible with the
readcommand because the text history quickly overwhelmed their memory. - Comprehension Errors: Half of CiteAgent's errors were due to the model misunderstanding the excerpt—either focusing on irrelevant text or confusing adjacent citations.
- API and Technical Fragility: Around 5.8% of the agent's failures were caused by technical issues, such as Semantic Scholar failing to provide open-access PDF links or the LM outputting improperly formatted JSON that broke the parser.
- Future Horizons: The authors hope to extend CiteAgent's architecture beyond machine learning to broader domains, paving the way for automated fact-checking systems that can dynamically verify any claim an LLM makes against the wider web.
Summary updated