When a language model checks a claim against a knowledge graph (a database of facts stored as subject–relation–object links), it first has to decide which real-world entity each phrase refers to. "Jordan" could be a country, a basketball player, or a politician. Current pipelines settle this automatically and then let a large model reason over whatever facts they retrieved. When the final verdict is wrong, the error is usually blamed on the reasoning step. This proposal argues that some of those "reasoning" errors actually begin as quiet mistakes about which entity is meant. It also argues that a small amount of human help does the most good when it goes to exactly those entity choices whose resolution would change the answer.
The method is simple to state. For each uncertain entity choice, we re-run the verifier once for every plausible candidate entity and check whether the verdict changes. If the verdict holds across all plausible candidates, the ambiguity is harmless and we leave it alone. If the verdict flips, we show a person a small picture of the competing candidates and their graph neighborhoods and ask one quick multiple-choice question. The claim to test is this: under a fixed budget of human seconds, these "verdict-pivotal" entity questions reduce fact-checking errors more than the obvious alternative, which is having people adjudicate the claims the model is least sure about.
Core Claim and Why It Is Not Already Done
The closest prior systems fall into two groups. The first group verifies claims over structured graphs. ClaimPKG has a small fine-tuned model write a "pseudo-subgraph" of triples, with placeholders such as unknown_i for entities the claim mentions only indirectly. It restricts entity names to real DBpedia entries through constrained decoding, scores candidate entities by graph structure to fill the placeholders, and then has a large model reason over the result. GraphFC also represents claims as triples with placeholders. It verifies triples anchored on known entities first, so that those anchors pin down the unknowns before dependent triples are checked. Both systems are fully automatic, and neither has any recourse when several candidate entities fit the graph structure about equally well.
The second group puts humans in the loop, but without this structure. CleanGraph lets experts repair a knowledge graph interactively, but it never measures the effect on a downstream model's outputs. FaxPlainAC collects human corrections as token highlights over Wikipedia text. Dis2Dis has humans iteratively rewrite ambiguous claims as unambiguous text. None of these three routes a specific entity decision to a person.
The obvious combination would be "send low-confidence placeholders to a human." That combination runs into an inconvenient fact reported by ClaimPKG: 71.5% of its errors come from the general reasoning model, not from retrieval or entity resolution. Routing on entity confidence alone would therefore aim at a minority of errors. It would also spend human effort on ambiguities that make no difference to the verdict.
Our idea differs from that obvious combination in two ways. First, we select questions by their counterfactual effect on the verdict, not by how uncertain the entity choice looks. Second, we treat the 71.5% figure as a hypothesis to test. By re-running the reasoner under alternative bindings, we can measure how many apparent reasoning failures disappear once the correct entity is supplied. That diagnostic is a contribution on its own, independent of the human-in-the-loop result.
We define pivotality as follows. For a placeholder $u$, let $p_u(e)$ be the pipeline's normalized structural score for candidate entity $e$ among its top-$k$ candidates. Let $y(e)$ be the verdict the reasoning model produces when $u$ is bound to $e$ and all other bindings stay fixed. The pivotality of $u$ is the probability mass of candidates that would disagree with the majority verdict:
$$\pi(u) = 1 - \max_{v} \sum_{e} p_u(e)\, \mathbb{1}[y(e) = v]$$
Given a budget of human seconds, we ask about placeholders in decreasing order of $\pi(u)/c_u$. Here $c_u$ is the measured human time to answer a question with $k$ candidates, which we fit from the user study described below. Because GraphFC-style planning makes later triples depend on earlier anchors, a single human answer can settle several triples at once. This propagation is the mechanism through which entity-level questions could beat claim-level review.
Research Program
We will build the pipeline on ClaimPKG's released components: the pseudo-subgraph generator with constrained decoding, structural entity scoring, and a large reasoning model. We add three pieces. The first is a rebinding module that enumerates the top-$k$ candidates for each placeholder and computes $\pi(u)$. This costs $k$ extra reasoning calls per placeholder, and we will report that cost. The second is a question interface that shows each candidate with its one-hop neighborhood around the claim's anchor entities. This follows CleanGraph's subgraph views, but the task is reduced to a single choice plus a "none of these" option, which catches entities missing from the knowledge base. The third is an error-attribution protocol. For every wrong verdict we check whether substituting the correct binding fixes it, which splits ClaimPKG-style "reasoning" errors into binding-induced and genuinely logical ones.
Human costs will come from a real study, not assumptions. On a subset of 500 claims, trained annotators will answer both kinds of question: entity-choice questions, and full-claim verdicts with the retrieved subgraph shown. We record time and accuracy for each. ABCD-LINK found that accepting or rejecting machine suggestions is far faster than producing labels from scratch, with a 73% approval rate at roughly 15 minutes per document pair. We expect a similar gap between choosing an entity and judging a whole claim, but we will measure it rather than assume it. The measured costs then drive large-scale simulation on the full benchmark, with an oracle answering from gold labels.
Baselines
The naive reference is the fully automatic ClaimPKG pipeline with zero human budget. For the strongest reported numbers on FactKG, the FactKG paper reports 77.65% accuracy for the graph-augmented GEAR model and 65.20% for text-only BERT (FactKG). ClaimPKG reports beating GEAR and KG-GPT by 9 to 12 accuracy points across reasoning types. The file gives no single absolute FactKG accuracy for ClaimPKG, so we will reproduce it ourselves and report our reproduced number as the zero-budget anchor.
The strongest competitor on our specific question is verdict-confidence routing. At the same budget in human seconds, annotators adjudicate whole claims in order of lowest model confidence in the verdict, and their verdict replaces the model's. This is the standard selective-prediction approach. It aims directly at the 71.5% of errors attributed to reasoning, so it is the baseline most likely to win.
Three ablations isolate the key ingredient. The first, binding-entropy routing, asks the same entity questions but ranks them by the entropy of $p_u$ instead of $\pi(u)$. It tests whether counterfactual rebinding matters or plain uncertainty suffices. The second, betweenness-centrality routing, ranks placeholders by the centrality of their candidates in the graph. It follows the finding in Active Learning for Entity Alignment that this cheap, precomputable score matched model-driven active learning. The third is random routing.
Every condition uses the same claims, the same reasoning model and prompts, the same top-$k$, and the same budget in human seconds, with per-question costs taken from the user study. Hyperparameters such as $k$ are tuned once on a development split and shared across all routing methods.
We would count the proposal as successful if pivotal-binding routing beats verdict-confidence routing in accuracy at matched budgets across a meaningful range, for example budgets covering 5 to 20% of claims. It must also beat binding-entropy routing, since otherwise the counterfactual step contributes nothing.
Why not just…?
Why not just use a bigger reasoning model? ClaimPKG's own error analysis shows that a 70B-parameter or GPT-4o-class reasoner still produces most of the remaining errors. Its ablations show that structural constraints, not scale, keep entity choices correct: entity correctness drops from 100% to 87.5% without them. A larger model will not know which "Jordan" a claim means when the claim does not say. We will nevertheless run the top reasoner we can afford as a zero-budget reference, so that the gain from human questions can be compared with the gain from scale.
Why not just resolve the ambiguity automatically by averaging over bindings? This is the real threat. The rebinding step already produces a verdict under each candidate, so one could take the probability-weighted majority verdict and never ask anyone. If that matches human-resolved accuracy, the human loop is unnecessary, and we will report that result plainly. We include this marginalization as an explicit zero-human condition. We expect it to fail precisely on high-$\pi$ placeholders. On those, the structural scores that weight the average are, by construction, split between candidates that lead to different verdicts. Averaging then favors the more popular entity, which is the failure mode the AIDA work documents for globally coherent but wrong clusters (Robust Disambiguation of Named Entities in Text).
Why not just have people check the least-confident verdicts? This is our main baseline, and it could win. That would happen if most errors are genuinely logical, or if claim-level review turns out to cost little more than an entity choice. The attribution protocol and the timing study decide the question. If fewer than roughly a quarter of wrong verdicts are fixed by correct bindings, or if the time per question is similar for both kinds, we expect verdict routing to dominate and will say so. We will also test a hybrid that spends the budget first on high-$\pi$ entity questions and then on verdict review.
Why not just use graph centrality to pick questions, as passive heuristics often suffice? Centrality measures how important a node is in the graph. It does not measure whether a node matters for this particular claim's verdict. Strong Heuristics for Named Entity Linking also shows that such heuristics degrade on organizations and locations. Still, the entity-alignment result is strong enough that we run centrality routing at every budget rather than dismiss it.
Proposed Experiments
The main experiment uses FactKG, which has 108,674 claims over DBpedia across one-hop, conjunction, existence, multi-hop, and negation types, in both written and colloquial styles. We plot accuracy against human seconds for every routing condition and break results down by reasoning type and style. We expect colloquial and multi-hop claims, which lean most on indirect mentions, to benefit most.
The first decisive check is the attribution result. We will report what fraction of ClaimPKG-style reasoning errors are corrected by gold bindings alone. A substantial fraction would change how that 71.5% figure should be read.
The second check stresses the setting where entity heuristics are known to be weak. We build a FactKG subset whose ambiguous placeholders are organizations, locations, or metonyms, together with claims whose correct entity is absent from DBpedia, where the correct human answer is "none of these." A secondary transfer test runs the same routing on GraphFC's placeholder pipeline over HOVER 3- and 4-hop claims. There, HoVer reports that simple retrieval recall collapses to 15% at four hops, and GraphFC reports an 8.31% gain on 4-hop claims in the open-book setting.
The proposal fails cleanly if either of two things happens: marginalization over bindings matches human-resolved accuracy, or verdict-confidence routing dominates at every budget. Either outcome is informative about where human effort belongs in graph-grounded fact checking.
References
- ClaimPKG: Enhancing Claim Verification via Pseudo-Subgraph Generation with Lightweight Specialized LLM
- A Graph-based Verification Framework for Fact-Checking
- CleanGraph: Human-in-the-loop Knowledge Graph Refinement and Completion
- FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn Correction in the Loop
- Dis2Dis: Explaining Ambiguity in Fact-Checking
- ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links
- FactKG: Fact Verification via Reasoning on Knowledge Graphs
- Active Learning for Entity Alignment
- Robust Disambiguation of Named Entities in Text
- Strong Heuristics for Named Entity Linking
- HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification