TL;DR
Large language models (LLMs) frequently hallucinate across diverse domains, yet traditional fact-checking pipelines depend on pre-extracted claims and pre-retrieved evidence. FacTool bridges this gap by introducing a multi-domain, tool-augmented framework that decomposes long-form generations into fine-grained claims, queries task-specific external tools (search engines, Python interpreters, Google Scholar), and verifies agreement. Powered by GPT-4, FacTool substantially outperforms self-checking baselines across knowledge QA, code synthesis, math reasoning, and scientific literature review.
What Problem Does This Paper Address?
Factuality detection in NLP has historically focused on narrow, structured tasks (e.g., verifying a short, isolated claim against a given Wikipedia paragraph in FEVER). In the modern LLM era, this paradigm breaks down:
- Scope: LLMs make errors not just in general knowledge, but also in code execution, multi-step math calculations, and academic citations.
- Granularity: LLM outputs are lengthy, containing implicit claims woven together without standardized boundaries.
- Evidence Scarcity: Real-world queries do not come pre-packaged with ground-truth reference texts.
Current LLMs struggle to catch their own hallucinations internally ("self-check"), leading to dangerous false positives in high-stakes settings.
Methods
FacTool frames factuality detection as an end-to-end, 5-stage pipeline adapted across four distinct tasks:
[Prompt + LLM Output]
➔ 1. Claim Extraction
➔ 2. Query Generation
➔ 3. Tool Querying
➔ 4. Evidence Collection
➔ 5. Agreement Verification
➔ [Factuality Decision]
- Claim Extraction: LLMs (via few-shot prompting) decompose long-form outputs into atomic verifiable units:
- KB-QA: Atomic Content Units (ACUs) $\le 15$ words with pronouns resolved.
- Code: Enclosed code blocks/snippets.
- Math: Arithmetic sub-expressions involving real numbers (excluding variables).
- Scientific: Paper citation tuples:
(title, author(s), year). - Query Generation: The system transforms extracted claims into executable tool inputs (e.g., skeptical search queries for KB-QA, simulated unit test inputs and candidate solutions for code, or Python equality assertions like
print(round(30/3, 7) == 10)for math). - Tool Querying & Evidence Collection:
- KB-QA: Uses Google Search API (via Serper) to extract snippets, knowledge graphs, and answer boxes.
- Code: Synthesizes unit tests, executes multiple sampled solutions to generate consensus "pseudo-golden" outputs, and evaluates candidate code.
- Math: Executes Python snippets to evaluate mathematical correctness.
- Scientific Review: Uses Google Scholar API (via Scholarly) to fetch true metadata.
- Agreement Verification: An LLM reasons over the retrieved tool outputs and assigns binary truth labels:
- For KB-QA, it uses zero-shot Chain-of-Thought (CoT) to detect factual contradictions.
- For code and math, execution status determines validity.
- For citations, fuzzy string matching and LLM author-subset checking verify paper existence.
Datasets
The authors evaluate FacTool on four benchmark tasks:
- KB-QA: 100 reference summaries from RoSE (evaluated for claim extraction quality) and 50 responses (23 positive, 27 negative) from FactPrompts (derived from Quora, TruthfulQA, etc.).
- Code Generation: 164 samples (109 positive, 55 negative) using instruction-only prompts from HumanEval.
- Math Reasoning: 100 prompts (47 positive, 53 negative) from GSM-Hard (GSM8k instances modified with larger numeric values).
- Scientific Literature Review: 100 self-instruct generated prompts (10 positive, 90 negative) across CS, physics, medicine, business, and law.
Key Insights and Ideas
- Self-reflection without tools is prone to severe confirmation bias: Self-check baselines (zero-shot and 3-shot CoT without tools) suffer from high false-positive rates (low precision, often classifying hallucinations as "True"). For example, on scientific citation checking, GPT-4 self-check achieves only a $12.73\%$ response-level precision, while FacTool achieves $100.0\%$.
- Execution as factuality: Generalizing "factuality" from textual consistency to operational validity (e.g., code executing against unit tests or Python running math expressions) unifies multi-domain verification under one umbrella.
- Underlying model matters for tool orchestration: When using ChatGPT (GPT-3.5) as the reasoning backbone, it often reaches self-contradictory conclusions even when provided with flawless evidence from Google Search (e.g., claiming fortune cookies were not invented in the US while citing evidence showing they originated in San Francisco). Upgrading to GPT-4 fixes most of this reasoning breakdown.
Key Results
FacTool (powered by GPT-4) consistently outperforms all self-check baselines across every scenario:
| Task | Metric | Self-Check (3-shot CoT, GPT-4) | FacTool (GPT-4) |
|---|---|---|---|
| KB-QA | Response-Level F1 | 57.14 | 71.79 |
| Code | Response-Level F1 | 85.26 | 92.11 |
| Math | Response-Level F1 | 70.59 | 80.36 |
| Scientific | Response-Level F1 | 21.54 | 94.74 |
Chatbot Factuality Auditing
Using FacTool as an automatic benchmark for popular models (weighted across tasks), the authors found:
- GPT-4 ranked highest ($75.60\%$ weighted claim accuracy; $43.33\%$ response accuracy).
- ChatGPT followed at $68.63\%$ claim accuracy.
- Claude-v1 ($63.95\%$) and Google Bard ($61.15\%$) showed intermediate performance.
- Vicuna-13B achieved $50.35\%$ claim accuracy, performing acceptably on general KB-QA but failing sharply on code, math, and scientific citations.
Limitations and Future Work
- Narrow Math Verification: FacTool currently checks individual arithmetic operations rather than high-level semantic reasoning or formula derivation logic. If a model derives the wrong equation, FacTool may flag each sub-calculation as true.
- Synthetic Test Quality: For code generation, synthetic test inputs generated by LLMs occasionally lack diversity (e.g., missing uppercase inputs or edge cases like
0), potentially validating buggy code. - Citation Nuance: The scientific tool only checks if a cited paper exists with matching authors and publication years; it does not evaluate whether the paper actually supports the specific scientific claim made in the text.
- Search and Parsing Brittleness: Conflicting search snippets, rounding variations, and title abbreviations can lead to false rejections during verification.
Summary updated