Abstract
Automatic factuality verification of large language model (LLM) generations is becoming more and more widely used to combat hallucinations. A major point of tension in the literature is the granularity of this fact-checking: larger chunks of text are hard to fact-check, but more atomic facts like propositions may lack context to interpret correctly. In this work, we assess the role of context in these atomic facts. We argue that fully atomic facts are not the right representation, and define two criteria for molecular facts: decontextuality, or how well they can stand alone, and minimality, or how little extra information is added to achieve decontexuality. We quantify the impact of decontextualization on minimality, then present a baseline methodology for generating molecular facts automatically, aiming to add the right amount of information. We compare against various methods of decontextualization and find that molecular facts balance minimality with fact verification accuracy in ambiguous settings.
The Problem: Ambiguity vs. Over-specification
The current pipeline for automatic fact-checking involves decomposing a long response into individual propositions. However, this process faces a "Goldilocks" problem regarding the level of context included in each claim.
- The Ambiguity Trap: A claim like "She won a medal" is too atomic. If the text is about Ann Jansson, and there are two famous individuals with that name (a footballer and a runner), a fact-checker might accidentally validate the claim using evidence for the wrong person.
- The Over-specification Trap: To fix ambiguity, standard methods "decontextualize" claims by adding numerous details (e.g., "Ann Jansson, the Swedish footballer born in 1957..."). However, if the evidence document doesn't mention her birth year, the entire claim might be marked "unsupported" even if the core fact is true. This represents a failure of minimality.
Methodology: Generating Molecular Facts
The authors propose a two-stage baseline methodology for generating Molecular Facts—claims that are decontextualized enough to stand alone but minimal enough to remain easy to verify.
- Stage 1: Identifying Ambiguity: Using an LLM's parametric knowledge, the system identifies the primary subject of a claim and checks if multiple entities share that name. It then determines a "disambiguation criterion" such as profession, birth year, or location.
- Stage 2: Molecular Fact Generation: The system rewrites the atomic claim using the identified criterion and the original context. The goal is to add just enough information to distinguish the entity (e.g., "Ann Jansson, the footballer") without over-specifying.
To evaluate this, the researchers utilized a controlled evidence generation framework to see how often over-specification leads to false negatives in fact-checking.
Key Results and Insights
The study utilized the Factcheck-Bench and a new Ambiguous Biographies dataset to test the impact of non-minimality on error localization. Key findings include:
- The Minimality Tax: Standard decontextualization methods (like
SAFE-DECONTEXT) often add so much ancillary information that between 1.7% and 9.6% of claims become impossible to verify against standard evidence, even when the core claim is correct. - Accuracy Improvements: On the Ambiguous Biographies task, Molecular-Decontext achieved 74.7% overall accuracy, outperforming the baseline
ATOMIC(68.7%) and the more restrictiveSAFE-DECONTEXT(73.4%). - Specificity vs. Verifiability: There is a Pareto frontier between the length of a claim and its accuracy. Molecular facts sit at a sweet spot, providing higher accuracy than atomic facts while being more robust than heavy-handed decontextualization.
- Human Evaluation: In a head-to-head analysis, humans rated 52% of Molecular Facts as "Minimal," compared to only 16% for
SIMPLE-DECONTEXT.
Limitations and Future Work
The current methodology relies on the LLM's parametric knowledge to identify ambiguous entities; if an entity is obscure or very recent, the system may fail to flag the ambiguity. Furthermore, the work is currently subject-centric, focusing on entity ambiguity. Future research could extend these principles to ambiguous events or predicates and explore performance in non-biographical domains like scientific or legal texts.
Summary updated