TL;DR
CiteLLM is a local, privacy-preserving agentic platform embedded directly within a LaTeX editor to automate scientific reference discovery. By leveraging LLMs strictly to query and rank papers from trusted academic repositories rather than generating text from scratch, it completely eliminates citation hallucinations while keeping unpublished manuscripts secure.
What Problem Does This Paper Address?
Writing a research paper requires grounding every claim in reliable literature. Traditionally, this means repeatedly jumping between your editor and Google Scholar, guessing keywords, and skimming dozens of papers to find the perfect citation.
While large language models (LLMs) can accelerate writing, using them for reference discovery introduces two major roadblocks:
- Hallucinations: LLMs notoriously fabricate plausible-sounding but entirely fake references.
- Data Privacy: Uploading an unpublished manuscript to a third-party LLM cloud service risks leaking sensitive pre-publication intellectual property.
CiteLLM tackles both problems by creating a local, hallucination-free pipeline that finds real, verifiable references without your data ever leaving your machine.
Methods
CiteLLM introduces a novel interaction paradigm by embedding three LLM-based utility agents directly into a local LaTeX editing environment:
The workflow operates in four distinct steps:
- Select Sentence Statement: The author highlights a specific claim inside the LaTeX editor.
- One-Click Citation Discovery: The system automatically extracts the sentence along with its surrounding paragraph and document-level context.
- Reference Search Process:
- Analyzer Agent: Uses a lightweight LLM classifier to infer the paper's discipline and dynamically routes the request to trusted open-access repositories (e.g., arXiv, bioRxiv, medRxiv). It then constructs context-aware search queries.
- Reranker Agent: Downloads the candidate PDFs, parses them into structured text using a tool called GROBID, and performs fine-grained semantic matching to rank them.
- Informer Agent: Presents the user with the top papers, complete with relevance scores, matching rationales, and the exact supporting paragraphs.
- Context-Aware Chatbot: Authors can chat with an integrated bot to discuss how the retrieved reference relates back to their manuscript.
With a single click, the system inserts the \cite{} marker and appends the correct BibTeX entry to the project's .bib file.
Datasets
The authors evaluated CiteLLM using a validation dataset consisting of 40 sentence statements sampled from public research papers across multiple disciplines. Each statement was paired with human-annotated search queries and at least one verified ground-truth reference from the original paper.
Key Insights and Ideas
- The "Why" Behind Hallucinations: When standard LLMs (like ChatGPT with internet access) are asked to find citations, they prioritize tight semantic alignment with the prompt over reality. If a perfect paper doesn't exist to support your specific claim, the LLM will simply invent one.
- Separation of Concerns: CiteLLM avoids hallucinations by completely decoupling the generation of citations from the LLM. The LLM is used strictly as an interface engine—for query formulation, parsing, and ranking—while the actual data is pulled from immutable, trusted academic databases.
- Context is King: Traditional keyword searches often fail because they lack domain context. By feeding the summary schema of the entire manuscript into the query generator, the system creates highly specific, human-aligned search queries.
Key Results
The researchers compared CiteLLM against Google Scholar (raw sentence query) and ChatGPT (with internet access enabled).
- Query Quality: CiteLLM's context-aware query generation consistently outperformed raw sentence and keyword-only baselines across clarity, specificity, and consistency metrics.
- Reference Validity & Usability: As shown in Table 1, CiteLLM achieved 100% validity (all references were real and accessible) and an impressive 87.5% usability score from human evaluators. Meanwhile, ChatGPT suffered from a 74% validity rate, meaning over a quarter of its suggested citations were completely fabricated.
Limitations and Future Work
Currently, CiteLLM relies exclusively on open-access preprint repositories (arXiv, bioRxiv, and medRxiv). It does not yet index closed-source, paywalled conference or journal databases, which limits its coverage in certain fields.
The authors note that future iterations will focus on expanding database access via secure APIs, refining the user interface, and optimizing the local LLM utilities to reduce processing latency and computational costs.
Summary updated