Querying the Witness: Topic-Specific Source Probes for Web Fact-Checking

When an automated fact-checker finds two web pages that disagree, it has two ways to make progress. It can search for more evidence about the claim, or it can find out which of the two pages deserves belief. Systems that search repeatedly, reformulating queries over several rounds, only ever do the first. Systems that estimate how reliable each source is only ever do the second, and only over a fixed pile of documents. We propose a fact-checker that does both inside one loop. It keeps a running estimate of each source's reliability per topic, so an outlet can be trusted on finance and distrusted on health. When a verdict hinges on a source whose reliability in the claim's topic is uncertain, the system issues a probe query: a search for what that same source said about other, already-verified claims in the same topic. The answers update the source's topic-level track record, and the claim verdict is recomputed.

In two sentences: today's iterative retrieval asks "what else is out there about this claim?", while we add the question "how has this particular witness done on this particular kind of question before?" A single uncertainty measure from a joint claim-and-source model decides which of the two queries is worth its cost. If this works, it explains why a conflict was resolved, by pointing to a source's measured record in a topic. That is a different kind of decision from following whichever answer appears most often, or trusting a static media-rating label.

Core Claim

Our central claim is narrow and testable. On conflicting-evidence fact-checking, spending part of a fixed retrieval budget on source-probe queries yields better veracity accuracy than spending all of it on claim-evidence queries. This holds when the conflicting sources have track records that differ by topic, and the gain should be near zero when they do not. If the gain does not concentrate on those topic-skewed cases, the idea is wrong and the improvement, if any, comes from something else.

What Exists and What Is Missing

Three bodies of work each own one piece. The first is truth discovery, the "chicken-and-egg" problem in which claim truth depends on source reliability and vice versa, usually solved by alternating estimation (A Survey on Truth Discovery). Evidence-based Trustworthiness (JELTA) runs expectation-maximization over latent claim truth and latent source trust, using textual entailment to read stances from articles. ClaimEval shows that a small set of verified claims can seed source credibility. We reuse this machinery: our probe anchors are exactly such verified seeds. JELTA, however, gives each source one scalar trust value and works on a fixed evidence set, and we change both.

The second is source-aware retrieval-augmented generation, where a language model answers from retrieved documents. RA-RAG estimates per-source reliability by cross-checking answers without labels, then retrieves only from the most reliable sources. It still uses one global score per source and a single retrieval pass. The CONFACT study finds that putting source background (publisher record, bias ratings) into the prompt beats filtering or reranking sources. It also notes two weaknesses. Its profiles are static, so they never update from verification outcomes, and long background text causes "lost in the middle" degradation. It explicitly raises "dynamic credibility" across domains without modeling it. That observation motivates this proposal.

The third is context-dependent trust. CrowdQM learns per-topic user reliability in closed forums and reports that this breaks down in sparse communities such as /r/AskDocs. That negative result constrains our design, and it is why we shrink topic estimates toward a global one rather than fitting them freely. Reliable Truth Discovery for Dynamic and Dependent Sources shows that copying between sources must be modeled, and we use its insight to discount syndicated agreement. In Agents We Trust shows that language models already carry domain-sensitive source preferences that instructions cannot switch off. That warns us against letting the language model itself be the arbiter of source trust.

Finally, iterative query refinement supplies the outer loop. CoV-RAG shows that revising queries after a first draft beats refining them beforehand. Think&Cite runs tree search with an entailment-based reward that is blind to who published the evidence. WKGFC navigates a knowledge graph and the web, but never assigns trust to publishers. All three decide when and what to re-query from semantic gaps alone. When Search Goes Wrong shows why that is dangerous: unconstrained iterative search reliably surfaces low-credibility citations.

No paper in this file issues queries about sources in order to resolve a claim. That is the difference we stake out.

Research Program

The system has three parts.

The first part is a joint model. Each claim $c$ has a latent truth value $y_c$, and each claim is assigned a topic $t(c)$ from a fixed set of $K$ topics, obtained by clustering claim embeddings. Each source $s$ has a global reliability $r_s$ and a topic reliability $r_{s,t}$. We tie these together hierarchically:

$$r_{s,t} \sim \mathrm{Beta}\big(\lambda\, r_s,\ \lambda\,(1-r_s)\big),$$

where $\lambda$ controls how strongly a source's topic reliability is pulled toward its overall reliability. With little topic data the estimate stays near $r_s$, which addresses CrowdQM's sparsity failure. With abundant data it can deviate freely. Stances of documents toward claims come from an entailment model, as in JELTA. Agreement among documents flagged as near-duplicates or syndicated copies is counted once, a lightweight version of the dependence modeling in DepenHMM. Inference alternates between computing posteriors over $y_c$ and updating the reliabilities, as in JELTA.

The second part is a query generator with two actions. A claim query is an ordinary reformulation aimed at new evidence on $c$, generated after a draft as in CoV-RAG. A source probe for source $s$ and topic $t$ searches for that source's content on anchor claims in topic $t$. Anchors are claims whose labels are known: training-split labels, or high-confidence posteriors from earlier rounds in the style of ClaimEval. Probes are restricted to anchors published before the test claim, to prevent leakage.

The third part is a value-of-information controller, which decides which query is worth issuing. At each round it picks the action $a$ that maximizes

$$\frac{H(y_c) - \mathbb{E}_a\big[H(y_c \mid \text{result of } a)\big]}{\mathrm{cost}(a)},$$

where $H(y_c)$ is the entropy of the current truth posterior and the expectation is estimated by simulating plausible outcomes of the query under the current model. It stops when the best ratio falls below a threshold or the token budget is exhausted. Concretely, a probe is chosen when $y_c$ is sensitive to an $r_{s,t}$ whose posterior is wide. A claim query is chosen when the evidence itself is thin.

The final verdict is produced by a language model prompted with the retrieved evidence. For each source it also receives a single line giving the learned topic-reliability estimate and its uncertainty. This is a much shorter background than CONFACT's full profiles, which keeps the prompt short.

Baselines

The naive reference point is single-shot retrieval followed by a language-model verdict over the top documents. We also run a variant that takes a majority vote over document stances. This measures how often frequency alone gets the answer right, which CONFACT and RA-RAG both show collapses when misinformation is over-represented.

The strongest competitor on source credibility is Source Background Augmentation from the CONFACT study. It adds Media Bias/Fact Check or search-derived publisher summaries to the generation prompt. Its reported gain is a "10% absolute improvement in F1 score" for LLaMA-3.1 on the ModC split, and a "5% accuracy boost" for Mistral. The file gives these as improvements over vanilla prompting, not as absolute scores, so we will re-run it ourselves. We will use the same backbone, the same retrieved pool, and Top-5 and Top-10 settings. Beating it means higher macro-F1 on both CONFACT splits at a matched token budget.

Because SBA is single-shot, we also build the composite a reviewer would propose: SBA plus CoV-RAG-style iterative claim queries, given the same number of search calls and tokens as our system. This is the real bar.

On joint modeling, we run JELTA (Evidence-based Trustworthiness) over the final retrieved pool. It reports beating majority-vote baselines by "4% on the Emergent dataset" and "approximately 5% on the FEVER dataset". We also run RA-RAG's global reliability estimation and weighted voting (RA-RAG); for its accuracy under poisoning, no reported numeric result was found on Lacuna, only the qualitative claim that it stays robust where majority vote drops. On AVeriTeC we cite the shared-task winner, TUDA_MAI, at an "AVeriTeC score of 63%" (AVeriTeC shared task), as an external ceiling reference rather than a matched comparison. Its pipeline and retrieval differ from ours, so we report our numbers alongside it without claiming a head-to-head win.

All matched baselines share the backbone language model, the entailment model, a cached search backend, and a hyperparameter search of equal size.

Three ablations isolate the key ingredients. Setting $\lambda \to \infty$ collapses topic reliability into global reliability, testing whether topic structure matters. Forbidding source probes, so the controller can only issue claim queries, tests whether asking about witnesses matters. Replacing the value-of-information controller with "re-query whenever the verdict confidence is low" tests whether the controller matters.

Why not just…?

Why not just give the language model source backgrounds and let it search iteratively? This is the most serious threat, because SBA already works well and CoV-RAG shows iterative revision helps. We think it falls short for two reasons. Static profiles cannot reflect a source's record in the specific topic, which the CONFACT authors themselves flag. And In Agents We Trust shows that language models apply ingrained source preferences that prompts cannot override, so letting the model reason about trust imports those preferences. The matched SBA-plus-iteration baseline settles the question directly. If it ties us on topic-skewed cases, our contribution reduces to cost savings, and we will report it that way.

Why not just use one global reliability score per source, as RA-RAG does? Global scores may suffice when sources are uniformly good or bad. CrowdQM also warns that per-topic estimates are noisy when data is sparse. The $\lambda \to \infty$ ablation answers this. We additionally stratify results by how much the conflicting sources' track records actually vary across topics. If topic modeling helps nowhere, including on synthetic data built to be topic-skewed, we drop it.

Why not just re-query whenever the model is unsure? Chain-of-Rewrite (Chain-of-Rewrite) and CoV-RAG do exactly this, and it may capture most of the gain. However, such triggers only generate more claim evidence. AVeriTeC's audit, in which Google returned useful evidence in "9 out of 20 cases" (AVeriTeC), suggests that more claim evidence often does not exist to be found. Source probes can still pay off in that situation. The no-probe and confidence-trigger ablations measure this.

Why not just follow the majority of retrieved documents? CONFACT and RA-RAG both report that majority following fails when misinformation is syndicated, and JELTA beats majority-vote baselines on Emergent and FEVER. We keep majority vote as the floor.

Proposed Experiments

The controlled test reuses JELTA's FEVER setup, which adds 200 simulated sources with varying accuracies. We assign each source accuracy that depends on topic, for example high on two topics and low on the rest, and expose a probeable history of each source's assertions on anchor claims. Varying the skew from zero to strong should produce a gain over the global-reliability ablation that grows with skew, and is roughly zero when there is no skew. Seeing this curve is the first decisive check.

The real-data tests use CONFACT (both the model-conflict and human-conflict splits), Emergent, and AVeriTeC. For CONFACT and Emergent we report macro-F1 and accuracy. For AVeriTeC we report the official AVeriTeC score with a fixed cached search backend, so every system sees identical results for identical queries. On every dataset we plot accuracy against search calls and tokens, to make the cost comparison explicit.

The second decisive check is stratified. We split test claims by whether the conflicting sources' estimated topic reliability differs from their global reliability. The gain over SBA-plus-iteration should be concentrated in the divergent stratum. If it is spread evenly, the mechanism is not what we claim.

As a secondary check on harm, we run the system on WebSearch-Harm queries (When Search Goes Wrong). We measure whether credibility-aware refinement lowers the share of low-credibility citations relative to a CoV-RAG-style refiner. That paper found that "74.7% of the risks uncovered by CREST-Search involved only citation risks", which makes citation-level risk the natural quantity to measure.

The main failure mode we anticipate is that probes for niche sources return nothing. We will report the probe hit rate. The hierarchical prior should make such misses harmless, leaving the estimate at the global score, but not useful. We will also report the false-anchor rate, meaning how often a self-verified anchor used in a probe later turns out wrong, since errors there would propagate into the reliability estimates.