Introduction
As Large Language Models (LLMs) are increasingly tasked with generating long-form content, such as biographies and technical reports, the methods used to evaluate their truthfulness have come under scrutiny. Researchers from National Taiwan University have identified a subtle but critical flaw in current evaluation frameworks: a paragraph can be composed entirely of verifiable facts and yet remain fundamentally deceptive. This phenomenon occurs when an LLM stitches together true facts about different individuals who happen to share the same name, creating a "Frankenstein" narrative that is factually dense but contextually false.
The Problem: The Aggregation Assumption
Standard evaluation metrics, such as FActScore, typically operate on the "Aggregation Assumption." This approach breaks a generated text into "atomic facts"—short, independent statements—and verifies each one against a knowledge source like Wikipedia. If every individual statement is found to be true somewhere in the knowledge base, the entire paragraph is deemed factual.
However, this method fails to account for entity ambiguity. For example, if an LLM is asked to write a biography of "John Smith," it might combine the birth date of a famous chemist with the career achievements of a professional athlete of the same name. Because each individual claim is technically true for some John Smith, standard evaluators mark the generation as 100% accurate. To a human reader, however, the text describes a person who never existed, rendering the generation a fallacy.
Methodology: Introducing D-FActScore
To address this gap, the authors propose D-FActScore (Disambig-FActScore), a context-aware metric that moves beyond independent fact-checking. The D-FActScore pipeline involves four key steps:
- Atomic Fact Decomposition: The text is broken down into simple, testable claims.
- Fact Grouping: The metric identifies and groups facts that the narrative implies belong to the same specific entity.
- Entity Linking: For each group, the system searches the knowledge base to identify the single most likely entity that matches the collective claims.
- Restricted Verification: Crucially, all facts within a group must be verified using only the information from that specific entity's page. If a fact is true for a different person with the same name but not for the linked entity, it is marked as non-factual within that context.
The researchers implemented this pipeline using an automated GPT-3.5/ChatGPT framework, which was subsequently validated against human annotations to ensure reliability.
The AmbigBio Benchmark
To test this new metric, the study introduced AmbigBio, a benchmark dataset designed to trigger entity confusion. The dataset consists of 500 names sampled from Wikipedia disambiguation pages, where each name corresponds to an average of 4.6 distinct entities. The researchers used these names to prompt several LLMs—including Llama-2, Tulu, Vicuna, and ChatGPT—using Retrieval-Augmented Generation (RAG) to produce biographies.
Key Insights and Results
The study revealed a significant "Factuality Inflation" in standard metrics, particularly for open-source models.
The "Frankenstein" Effect
Open-source models like Llama-13b were found to be highly prone to merging facts from different retrieved documents without recognizing they referred to different people. These models often prioritize local factual density (making sure every sentence looks true) over global narrative consistency.
Metric Discrepancy
The gap between FActScore and D-FActScore is stark:
- Factuality Inflation: For open-source models, FActScore overestimates truthfulness by 8% to 16%.
- Model Performance: Llama-13b-chat achieved a high FActScore of 94.8%, but its score plummeted to 78.4% under D-FActScore evaluation. In contrast, ChatGPT remained robust, scoring 98.3% on FActScore and 92.1% on D-FActScore.
- Scaling Limits: Interestingly, increasing model size (e.g., from Llama-13b to 70b) yielded less than a 2% improvement in D-FActScore. This suggests that entity disambiguation is a reasoning challenge that cannot be solved by scale alone.
Conclusion and Future Work
D-FActScore exposes a critical vulnerability in how we measure LLM accuracy. While ChatGPT shows a superior ability to distinguish between entities—often generating multiple distinct biographies when prompted with an ambiguous name—open-source models still struggle with global consistency.
While this research focuses on human biographies and Wikipedia, the implications extend to any domain where entity ambiguity exists. Future work will need to address noisier knowledge sources beyond Wikipedia and tackle other forms of "true-but-false" generations, such as causal fallacies, where individual facts are correct but the logical connections between them are non-existent.
Summary updated