CiteLLM: An Agentic Platform for Trustworthy Scientific Reference Discovery

Mengze Hong, Di Jiang, Chen Jason Zhang, Zichang Guo, Yawen Li, Jun Chen, Shaobo Cui, Zhiyang Su

TL;DR

CiteLLM is a local, privacy-preserving agentic platform embedded directly within a LaTeX editor to automate scientific reference discovery. By leveraging LLMs strictly to query and rank papers from trusted academic repositories rather than generating text from scratch, it completely eliminates citation hallucinations while keeping unpublished manuscripts secure.

What Problem Does This Paper Address?

Writing a research paper requires grounding every claim in reliable literature. Traditionally, this means repeatedly jumping between your editor and Google Scholar, guessing keywords, and skimming dozens of papers to find the perfect citation.

While large language models (LLMs) can accelerate writing, using them for reference discovery introduces two major roadblocks:

  1. Hallucinations: LLMs notoriously fabricate plausible-sounding but entirely fake references.
  2. Data Privacy: Uploading an unpublished manuscript to a third-party LLM cloud service risks leaking sensitive pre-publication intellectual property.

CiteLLM tackles both problems by creating a local, hallucination-free pipeline that finds real, verifiable references without your data ever leaving your machine.

Methods

CiteLLM introduces a novel interaction paradigm by embedding three LLM-based utility agents directly into a local LaTeX editing environment:

Overview of CiteLLM: trustworthy reference discovery powered by three LLM-based utility agents.

The workflow operates in four distinct steps:

  1. Select Sentence Statement: The author highlights a specific claim inside the LaTeX editor.
  2. One-Click Citation Discovery: The system automatically extracts the sentence along with its surrounding paragraph and document-level context.
  3. Reference Search Process:
  4. Analyzer Agent: Uses a lightweight LLM classifier to infer the paper's discipline and dynamically routes the request to trusted open-access repositories (e.g., arXiv, bioRxiv, medRxiv). It then constructs context-aware search queries.
  5. Reranker Agent: Downloads the candidate PDFs, parses them into structured text using a tool called GROBID, and performs fine-grained semantic matching to rank them.
  6. Informer Agent: Presents the user with the top papers, complete with relevance scores, matching rationales, and the exact supporting paragraphs.
  7. Context-Aware Chatbot: Authors can chat with an integrated bot to discuss how the retrieved reference relates back to their manuscript.

With a single click, the system inserts the \cite{} marker and appends the correct BibTeX entry to the project's .bib file.

Demonstration of user interface: a full LaTeX editor with embedded reference discovery in four steps.

Datasets

The authors evaluated CiteLLM using a validation dataset consisting of 40 sentence statements sampled from public research papers across multiple disciplines. Each statement was paired with human-annotated search queries and at least one verified ground-truth reference from the original paper.

Key Insights and Ideas

  • The "Why" Behind Hallucinations: When standard LLMs (like ChatGPT with internet access) are asked to find citations, they prioritize tight semantic alignment with the prompt over reality. If a perfect paper doesn't exist to support your specific claim, the LLM will simply invent one.
  • Separation of Concerns: CiteLLM avoids hallucinations by completely decoupling the generation of citations from the LLM. The LLM is used strictly as an interface engine—for query formulation, parsing, and ranking—while the actual data is pulled from immutable, trusted academic databases.
  • Context is King: Traditional keyword searches often fail because they lack domain context. By feeding the summary schema of the entire manuscript into the query generator, the system creates highly specific, human-aligned search queries.

Key Results

The researchers compared CiteLLM against Google Scholar (raw sentence query) and ChatGPT (with internet access enabled).

  • Query Quality: CiteLLM's context-aware query generation consistently outperformed raw sentence and keyword-only baselines across clarity, specificity, and consistency metrics.
Comparison of query construction approaches.
  • Reference Validity & Usability: As shown in Table 1, CiteLLM achieved 100% validity (all references were real and accessible) and an impressive 87.5% usability score from human evaluators. Meanwhile, ChatGPT suffered from a 74% validity rate, meaning over a quarter of its suggested citations were completely fabricated.
Table 1: Performance comparison of citation discovery methods.

Limitations and Future Work

Currently, CiteLLM relies exclusively on open-access preprint repositories (arXiv, bioRxiv, and medRxiv). It does not yet index closed-source, paywalled conference or journal databases, which limits its coverage in certain fields.

The authors note that future iterations will focus on expanding database access via secure APIs, refining the user interface, and optimizing the local LLM utilities to reduce processing latency and computational costs.

Summary updated

Abstract

Large language models (LLMs) have created new opportunities to enhance the efficiency of scholarly activities; however, challenges persist in the ethical deployment of AI assistance, including (1) the trustworthiness of AI-generated content, (2) preservation of academic integrity and intellectual property, and (3) protection of information privacy. In this work, we present CiteLLM, a specialized agentic platform designed to enable trustworthy reference discovery for grounding author-drafted claims and statements. The system introduces a novel interaction paradigm by embedding LLM utilities directly within the LaTeX editor environment, ensuring a seamless user experience and no data transmission outside the local system. To guarantee hallucination-free references, we employ dynamic discipline-aware routing to retrieve candidates exclusively from trusted web-based academic repositories, while leveraging LLMs solely for generating context-aware search queries, ranking candidates by relevance, and validating and explaining support through paragraph-level semantic matching and an integrated chatbot. Evaluation results demonstrate the superior performance of the proposed system in returning valid and highly usable references.
BibTeX
@inproceedings{hong2026citellm,
  author = {Mengze Hong and Di Jiang and Chen Jason Zhang and Zichang Guo and Yawen Li and Jun Chen and Shaobo Cui and Zhiyang Su},
  title = {{CiteLLM: An Agentic Platform for Trustworthy Scientific Reference Discovery}},
  booktitle = {The Web Conference},
  year = {2026},
  doi = {10.1145/3774905.3793119},
}
Figure