Verdict Maps: Exact Enumeration of How a Fact-Check Changes With How Much You Trust Each Source

Retrieval-augmented fact-checkers work in three steps. They fetch documents, decide which ones to keep using some mix of topical relevance and source credibility, and ask a language model for a verdict. The middle step is controlled by knobs that are set by hand: how much weight credibility gets relative to relevance, a credibility floor below which sources are dropped, and how many documents to keep. Small changes to these knobs can flip the verdict. Ge et al. found that moving from five to ten retrieved paragraphs often lowered accuracy (Resolving Conflicting Evidence in Automated Fact-Checking). The RAGuard study found that retrieval of misleading Reddit posts makes nearly every model worse than answering with no retrieval at all (Worse than Zero-shot?). Today, a single knob setting quietly decides which sources the model is allowed to hear, and nothing reports whether the answer would survive a slightly different setting.

We propose to stop picking one setting and compute the whole picture. This borrows a standard move from computational geometry. A continuous parameter does not need to be searched point by point if the combinatorial structure it controls changes only at finitely many critical values. Here, the selected evidence set changes only where two documents swap rank or one crosses the credibility floor. Within each region between those boundaries, the evidence set is fixed, so the language model needs to be queried only once per region. The result is a "verdict map" over all trust settings. We read it in two ways: it gives a calibrated decision about whether to answer or abstain, and it shows the specific "hinge" sources whose inclusion or exclusion flips the verdict.

Core Claim

The claim has three parts.

  1. The maps are small. For claims with conflicting evidence, the verdict as a function of the trust knobs is piecewise constant, with few distinct pieces. Exact enumeration is therefore cheap: tens of language-model calls per claim, not hundreds.
  2. The maps are informative. A claim whose map is dominated by one verdict is reliably answered. A claim whose map splits into regions with opposing verdicts is exactly the claim where misleading or low-credibility evidence is steering the model.
  3. Exactness buys something sampling cannot. Enumerating the regions exactly allows a certified statement: the verdict is constant for every trust setting inside a stated box. Random sampling of settings can never certify this, and it systematically misses thin regions where a single source tips the outcome.

The novelty is not a better credibility score. It is treating credibility-aware evidence selection as a geometric arrangement and reading the verdict's structure across all of it.

Research Program

Each retrieved document $i$ comes with three attributes: a relevance score $r_i$ from the retriever, a credibility score $c_i$, and a stance. Credibility comes from Media Bias/Fact Check (MBFC) ratings where available, or else from an estimator. The documents are scored and selected as

$$s_i(\alpha) = (1-\alpha)\, r_i + \alpha\, c_i, \qquad S(\alpha,\tau) = \text{top-}k \text{ of } \{\, i : c_i \ge \tau \,\} \text{ ranked by } s_i(\alpha),$$

where $\alpha \in [0,1]$ trades relevance against credibility, $\tau$ is the credibility floor, and $k$ is the budget.

Two documents swap order exactly at $\alpha_{ij} = (r_j - r_i)/\big((r_j - r_i) - (c_j - c_i)\big)$. The plane of $(\alpha, \tau)$ settings is therefore cut by at most $\binom{n}{2}$ vertical lines and $n$ horizontal lines. Inside each resulting cell the selected set $S$ is constant. This mirrors the role of critical scaling values in Polygon Placement Revisited. That work shows that a continuous scale parameter for placing one polygon inside another induces only $O((pq)^2)$ combinatorially distinct configurations, and exploits this for an exact algorithm. We reuse the technique, not the problem.

Only swaps at the $k$-th boundary actually change the selected set. We therefore expect the number of distinct sets per claim to be far below the worst-case quartic bound in $n$, and we will measure it. For $n \approx 30$ retrieved documents, enumeration takes milliseconds.

The method has three components.

Exact map construction. We sweep the arrangement, deduplicate the selected sets, and run the language-model reader once per distinct set. The reader can use plain prompting or source-background prompting. Each cell is labeled with the resulting verdict.

Map readout. - Stability is the mass of the majority verdict under a prior over $(\alpha, \tau)$. The prior is uniform by default; we also test a version fitted on development data. - The certificate is the largest axis-aligned box around the development-tuned setting inside which the verdict is constant. - Hinge sources are the documents whose entry into the selected set across a cell boundary flips the verdict.

The system answers with the majority verdict when stability exceeds a threshold and abstains otherwise. Hinge sources are surfaced as the explanation.

Optional third knob. We add a uniform "trust strength" that shrinks all $c_i$ toward their mean. This tests whether the approach still pays off in three dimensions, where cell counts grow.

Baselines

Do nothing. The naive reference point is zero-shot verification with no retrieval. RAGuard shows this reference is not trivial: Mistral falls from about 64% zero-shot accuracy to about 59% with standard retrieval, and GPT-4o loses 31.7% accuracy under misleading conditions (Worse than Zero-shot?). We also run plain relevance-only retrieval, which is the setting $\alpha = 0$, $\tau = 0$.

Strongest existing generation-stage method. This is Source Background Augmentation (SBA): publisher history and bias information is appended to each document in the prompt. Ge et al. found it beat both source filtering and credibility-weighted ranking on CONFACT. They report "10% absolute improvement in F1 score for LLaMA-3.1" and a "5% accuracy boost for Mistral" on the ModC split (Resolving Conflicting Evidence). The file gives only these relative gains, not absolute scores, so we will reproduce SBA ourselves with the same readers, retrieval pool, and MBFC metadata. Because our method is compatible with SBA, the fair comparison is SBA at its single best development-tuned setting versus SBA read through the verdict map. Beating it means higher accuracy at full coverage, or higher selective accuracy at matched coverage, on the human-verified HumC split.

Strongest statistical competitor. This is RA-RAG (retrieval augmented generation with estimation of source reliability). It estimates per-source reliability without labels by cross-checking sources on shared queries, selects the top-$\kappa$ reliable and relevant sources, and combines answers by weighted majority vote. It is the strongest competitor because it already folds credibility into selection and aggregation, and it cuts tokens by 99.1% at 1,000 sources. The file reports no absolute accuracy figure; we found no reported result on Lacuna for RA-RAG accuracy on CONFACT or RAGuard. We run it on the same pools with its reliability estimator also feeding our $c_i$, so the two methods differ only in how selection is aggregated.

Ablations. Three ablations isolate the key ingredient.

  • Best single cell uses the same map but reports only the development-tuned cell. This tests whether reading the map matters, not just enumerating it.
  • Budget-matched sampling draws random $(\alpha, \tau)$ settings and deduplicates the resulting sets, under the same language-model call budget. This tests whether exactness matters.
  • Random-stance map keeps the cell structure but shuffles credibility scores. This tests whether credibility carries real signal.

Why not just…?

Why not just tune $k$ and the credibility floor on a development set? This is the best-single-cell ablation, and it is the most likely way the proposal could fail on accuracy at full coverage. The evidence against it is the instability Ge et al. observed between top-5 and top-10, which suggests the best setting varies from claim to claim. The experiment settles it directly. If the tuned single setting matches the map at full coverage, the map's remaining value is calibration and abstention, and we will report it that way.

Why not just sample trust settings at random and take a vote? This is a real threat. With deduplication, sampling may recover most of the map. Sampling finds each region with probability proportional to its area, so it misses thin regions, which are exactly where a single hinge source flips the verdict. Sampling also cannot issue a certificate. We compare the two at matched call budgets and report how often sampling misses a verdict-changing cell. If that rate is negligible on real data, we will say that exact enumeration is a cost saving rather than an accuracy gain.

Why not just put source metadata in the prompt and let the model reason (SBA)? SBA is our strongest baseline and composes with our method. Ge et al. themselves report that the longer prompts trigger "lost in the middle" forgetting. RAGuard's finding that models latch onto specific numbers and names in misleading text, even when given general instructions, suggests that metadata alone does not stop coherent falsehoods. The verdict map tests whether SBA's verdict is robust to which sources it was shown; it does not replace SBA.

Why not just use reliability-weighted voting (RA-RAG)? RA-RAG assumes that lies are idiosyncratic across sources. RAGuard's misleading Reddit narratives are coordinated, which breaks consensus-based reliability estimation. The verdict map does not assume consensus. It exposes when a verdict depends on admitting a coherent cluster of low-credibility documents. The synthetic Adversary-Hammer experiment below, with coordinated adversaries, tests this head to head.

Proposed Experiments

Datasets. The primary benchmark is CONFACT: 3,180 claims with stance-labeled documents and source credibility ratings. We use both the model-identified conflicts split (ModC) and the human-verified conflicts split (HumC), with LLaMA-3.1 and Mistral readers to match the reported SBA results.

The second benchmark is RAGuard: 2,648 PolitiFact claims and 16,331 Reddit documents labeled supporting, misleading, or unrelated. Reddit posts lack MBFC ratings, so credibility comes from RA-RAG-style cross-checking. The "misleading" labels serve purely as evaluation ground truth for hinge detection.

The third setting is controlled. We use the multi-source NQ, TriviaQA, and HotpotQA setups with the Beta and Adversary-Hammer reliability priors from RA-RAG. We add a coordinated-adversary variant in which adversaries agree with one another, so that true source reliability is known.

Metrics. - Accuracy and macro-F1 at full coverage. - The selective-accuracy curve, measured as area under risk–coverage. - Expected calibration error of stability as a confidence score. - Distinct sets and language-model calls per claim. - Precision and recall of hinge sources against RAGuard's misleading labels.

Decisive checks. The proposal stands or falls on four results.

  1. Map size. The median number of distinct selected sets per claim should be small enough, roughly under 30, that exact maps cost less than budget-matched sampling with equal coverage of verdict-changing cells.
  2. Selective accuracy. At matched coverage, map-based abstention should beat both SBA's model-confidence abstention and RA-RAG's vote margin on HumC.
  3. Hinge detection. On RAGuard, flagged hinge sources should be enriched for documents labeled misleading well above their base rate of 11.1%.
  4. Coordinated adversaries. In the controlled experiment, map fragmentation should rise with adversary coordination while RA-RAG's vote margin does not.

If the maps turn out to be large and fragmented for most claims, the geometric structure is not doing work, and the proposal fails cleanly.