TL;DR
CiteLab is an open-source toolkit designed to standardize and simplify how we build, evaluate, and debug citation generation in Large Language Models (LLMs). By combining a modular backend (Citeflow) with an interactive visual frontend (Citefix), it allows researchers to move beyond "black-box" generation and iteratively improve citation accuracy through a collaborative Human-LLM workflow.
What Problem Does This Paper Address?
While LLMs are increasingly used for Question-Answering (QA), they are notorious for "hallucinations"—generating facts that sound plausible but are entirely made up. Retrieval-Augmented Generation (RAG) and citation generation (where the model cites its sources) are the primary defenses against this.
However, the field currently lacks a unified framework. Every research paper uses a different pipeline, making it nearly impossible to perform "apples-to-apples" comparisons or reproduce results. Furthermore, diagnosing why a model failed to cite a source correctly is a labor-intensive process. Researchers often have to manually sift through logs to find the point of failure: was it a bad retrieval, a poor prompt, or a failure in the model's reasoning?
Methods
The authors introduce CiteLab, which consists of two integrated parts:
- Citeflow (The Modular Engine): Instead of a monolithic script, Citeflow breaks citation generation into four primary modules:
- Input: Handles data loading and prompt creation.
- Generator: The LLM core (supporting GPT-4, Llama, etc.) using either direct or iterative generation.
- Enhancing Module: The "toolbox" containing 14 components like retrievers, planners (to organize thoughts), feedbackers (to critique drafts), and editors (to refine the final output).
- Evaluator: Standardizes metrics like ROUGE (fluency) and citation precision/recall.
- Citefix (The Visual Interface): A GUI that visualizes the entire data stream. It allows users to see exactly what documents were retrieved and how the model attributed specific sentences or even individual tokens to those sources.
- Human-LLM Interaction: A novel feature where an "AI Assistant" within the interface analyzes failure cases and suggests specific workflow modifications (e.g., "Your prompt needs more diversity" or "Add an extraction module here").
Datasets
The researchers evaluated their framework using the ASQA (Ambiguous Question Answering) dataset. ASQA is a popular benchmark for long-form QA that requires models to synthesize information from multiple documents to answer complex, multi-faceted questions.
Key Insights and Ideas
- Granularity is King: The paper emphasizes that document-level attribution isn't enough for debugging. CiteLab supports span-level and token-level attribution, allowing developers to see exactly which word triggered a specific citation.
- The "Assistant" as a Debugger: One of the most interesting ideas is using a high-performing LLM (like GPT-4o) to act as a meta-critic for the workflow itself. In their case study, the assistant identified that the model was repeating the same facts because the retrieval prompt was too narrow—a nuance a human might miss after hours of looking at logs.
- Mix-and-Match Innovation: By modularizing the process, researchers can take a "Planner" from one paper and an "Editor" from another to create a hybrid pipeline without rewriting the entire codebase.
Key Results
The authors compared CiteLab against 11 baseline methods (including ALCE, RAG, and self-RAG).
- SOTA Performance: Through three rounds of human-LLM iterative improvement, they developed a new workflow called self-planning-RAG.
- Quantitative Gains: On the ASQA dataset using Llama3-8B, their optimized workflow achieved a Citation Recall of 90.4 and Citation Precision of 90.0, significantly outperforming vanilla RAG (which scored 47.2 and 53.8, respectively).
- Efficiency: The visual interface allowed them to identify and fix systemic prompt issues in a fraction of the time it would take using standard text-based debugging.
Limitations and Future Work
- Model Scope: The current experiments were primarily conducted on two models (GPT-4o and Llama3-8B). The authors acknowledge that the toolkit's effectiveness should be validated across a wider variety of model architectures.
- Assistant Dependency: The diagnostic process relies heavily on the "Assistant LLM." If the assistant isn't powerful enough to interpret the failure cases correctly, the human-in-the-loop process breaks down.
- Retrieval Tech: The authors suggest that future versions could integrate more advanced, "state-of-the-art" retrieval technologies to further reduce the initial bottleneck of finding relevant documents.
Check out the project: The toolkit is open-source and available on GitHub at https://github.com/SjJ1017/CiteLab.
Summary updated