Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations

Cheng-Han Chiang, Hung-yi Lee

Introduction

As Large Language Models (LLMs) are increasingly tasked with generating long-form content, such as biographies and technical reports, the methods used to evaluate their truthfulness have come under scrutiny. Researchers from National Taiwan University have identified a subtle but critical flaw in current evaluation frameworks: a paragraph can be composed entirely of verifiable facts and yet remain fundamentally deceptive. This phenomenon occurs when an LLM stitches together true facts about different individuals who happen to share the same name, creating a "Frankenstein" narrative that is factually dense but contextually false.

The Problem: The Aggregation Assumption

Standard evaluation metrics, such as FActScore, typically operate on the "Aggregation Assumption." This approach breaks a generated text into "atomic facts"—short, independent statements—and verifies each one against a knowledge source like Wikipedia. If every individual statement is found to be true somewhere in the knowledge base, the entire paragraph is deemed factual.

However, this method fails to account for entity ambiguity. For example, if an LLM is asked to write a biography of "John Smith," it might combine the birth date of a famous chemist with the career achievements of a professional athlete of the same name. Because each individual claim is technically true for some John Smith, standard evaluators mark the generation as 100% accurate. To a human reader, however, the text describes a person who never existed, rendering the generation a fallacy.

Methodology: Introducing D-FActScore

To address this gap, the authors propose D-FActScore (Disambig-FActScore), a context-aware metric that moves beyond independent fact-checking. The D-FActScore pipeline involves four key steps:

  1. Atomic Fact Decomposition: The text is broken down into simple, testable claims.
  2. Fact Grouping: The metric identifies and groups facts that the narrative implies belong to the same specific entity.
  3. Entity Linking: For each group, the system searches the knowledge base to identify the single most likely entity that matches the collective claims.
  4. Restricted Verification: Crucially, all facts within a group must be verified using only the information from that specific entity's page. If a fact is true for a different person with the same name but not for the linked entity, it is marked as non-factual within that context.

The researchers implemented this pipeline using an automated GPT-3.5/ChatGPT framework, which was subsequently validated against human annotations to ensure reliability.

The AmbigBio Benchmark

To test this new metric, the study introduced AmbigBio, a benchmark dataset designed to trigger entity confusion. The dataset consists of 500 names sampled from Wikipedia disambiguation pages, where each name corresponds to an average of 4.6 distinct entities. The researchers used these names to prompt several LLMs—including Llama-2, Tulu, Vicuna, and ChatGPT—using Retrieval-Augmented Generation (RAG) to produce biographies.

Key Insights and Results

The study revealed a significant "Factuality Inflation" in standard metrics, particularly for open-source models.

The "Frankenstein" Effect

Open-source models like Llama-13b were found to be highly prone to merging facts from different retrieved documents without recognizing they referred to different people. These models often prioritize local factual density (making sure every sentence looks true) over global narrative consistency.

Metric Discrepancy

The gap between FActScore and D-FActScore is stark:

  • Factuality Inflation: For open-source models, FActScore overestimates truthfulness by 8% to 16%.
  • Model Performance: Llama-13b-chat achieved a high FActScore of 94.8%, but its score plummeted to 78.4% under D-FActScore evaluation. In contrast, ChatGPT remained robust, scoring 98.3% on FActScore and 92.1% on D-FActScore.
  • Scaling Limits: Interestingly, increasing model size (e.g., from Llama-13b to 70b) yielded less than a 2% improvement in D-FActScore. This suggests that entity disambiguation is a reasoning challenge that cannot be solved by scale alone.

Conclusion and Future Work

D-FActScore exposes a critical vulnerability in how we measure LLM accuracy. While ChatGPT shows a superior ability to distinguish between entities—often generating multiple distinct biographies when prompted with an ambiguous name—open-source models still struggle with global consistency.

While this research focuses on human biographies and Wikipedia, the implications extend to any domain where entity ambiguity exists. Future work will need to address noisier knowledge sources beyond Wikipedia and tackle other forms of "true-but-false" generations, such as causal fallacies, where individual facts are correct but the logical connections between them are non-existent.

Summary updated

Abstract

Long-form generations from large language models (LLMs) contain a mix of factual and non-factual claims, making evaluating factuality difficult. Prior works evaluate the factuality of a long paragraph by decomposing it into multiple facts, verifying those facts independently, and aggregating the results. Such methods assume that combining factual claims forms a factual paragraph. The above assumption can be violated: we show that strong open-source models like Llama-chat can generate paragraphs that contain verifiable facts, but the facts are combined into a non-factual paragraph due to entity ambiguity. We further reveal that existing factuality metrics, including FActScore and citation recall, cannot properly evaluate these non-factual paragraphs and overestimate their factuality. To address this, we introduce an enhanced metric, D-FActScore, specifically designed for content with ambiguous entities. We evaluate the D-FActScores of people biographies generated by retrieval-augmented LLMs. We show that D-FActScore can better assess the factuality of paragraphs with entity ambiguity than FActScore. We also find that four widely used open-source LLMs tend to mix information of distinct entities to form non-factual paragraphs, making their D-FActScore much lower than FActScore by over 10%.
BibTeX
@inproceedings{chiang2024merging,
  author = {Cheng-Han Chiang and Hung-yi Lee},
  title = {{Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.160},
}
Figure