Few, Independent, Two-Sided Witnesses: Evidence Selection and Verdicts for Fact-Checking

Retrieval-based fact-checkers and question-answering systems usually treat every retrieved passage as true. Under that assumption, a false story syndicated across thirty websites looks like thirty independent confirmations. We propose to fix this at two points. The first is the choice of which few passages to show the reasoner. The second is how the verdict is formed from them. For selection, we pick a small evidence set by greedily maximizing a score that rewards covering each part of the claim, counts copies of the same story only once, discounts low-credibility origins, and stops rewarding one side of the dispute once it is well covered, so the other side gets room. Because this score has diminishing returns in the formal sense (it is submodular), simple greedy selection is guaranteed to capture at least 63% of the best achievable score. For the verdict, we use bilateral gradual argumentation: a way of scoring a claim in a debate graph with two separate numbers, how well it is supported and how strongly it is opposed, instead of one accept/reject bit. Keeping two numbers lets "nobody has evidence" (both low) be told apart from "credible sources genuinely disagree" (both high).

This matters for practical reasons. Resolving Conflicting Evidence in Automated Fact-Checking (CONFACT) finds that moving from 5 to 10 retrieved paragraphs lowers accuracy, and that adding source background to the prompt helps but makes prompts longer. The budget is therefore a handful of passages, and which handful you choose decides the outcome. AVeriTeC also has separate "Not Enough Evidence" and "Conflicting Evidence/Cherry-picking" classes, which is exactly the distinction that a single acceptability score cannot express.

Core Claim

Under a fixed budget of about five evidence passages, two things together should outperform relevance or credibility-weighted top-$k$ retrieval followed by an LLM judge. The first is selection that is credibility-discounted, collapses syndicated copies, and balances stances. The second is two-score argumentative adjudication. The advantage should be largest in two regimes: when misinformation is duplicated across many URLs, and when the system must separate missing evidence from conflicting evidence.

The selection objective is the following. Let $Q$ be the facets of a claim (sub-questions, as in Generating Fact Checking Briefs), with importance weights $w_q$. Let $s \in \{+,-\}$ be a stance. Candidate passages are grouped into provenance clusters $c$, meaning near-duplicates and wire reprints of one origin. Each cluster has a topic-conditioned credibility $r_{c,q} \in [0,1]$, and $a_s(d,q) \in [0,1]$ is how well passage $d$ answers facet $q$ with stance $s$. The objective is

$$F(S) = \sum_{q \in Q} \sum_{s \in \{+,-\}} w_q \min\Big(\tau,\ \sum_{c} r_{c,q} \max_{d \in S \cap c} a_s(d,q)\Big),$$

where $\tau$ is a saturation level.

Each design choice has a purpose. The within-cluster max means a second copy of the same story adds nothing. The sum over clusters rewards genuinely independent witnesses. The cap $\tau$ means that once one stance on a facet is well covered, further same-side evidence is worth nothing, so the remaining budget flows to uncovered facets and to the opposing stance.

The inner sum is a facility-location-style function over disjoint clusters, which is monotone submodular. Truncating a monotone submodular function at a constant keeps it monotone submodular. Nonnegative sums preserve both properties. The cardinality-constrained greedy rule therefore inherits the $(1-1/e)$ guarantee established in Maximizing the spread of influence through a social network.

Relation to Prior Work

The method draws on three lines of work.

A Dense Subset Index for Collective Query Coverage (DISCo) shows that multi-passage evidence retrieval can be cast as monotone submodular coverage and accelerated by roughly 100×. It assumes every passage is true and has no notion of origin or stance. We reuse its framing and acceleration ideas but change what counts as coverage.

Retrieval augmented generation with estimation of source reliability (RA-RAG) estimates per-source reliability without labels by cross-checking answers across sources. It then takes the top-$\kappa$ sources and aggregates their answers by weighted majority vote. Its reliability estimates are a natural way to fill in $r_{c,q}$, and we run them within topic clusters rather than globally. Its selection and aggregation have no notion of redundancy, which is the weakness our objective targets.

CONFACT motivates the work in two ways. Its evidence that source background helps tells us credibility should enter the process. Its finding that longer contexts hurt tells us the budget is small. CONFACT also leaves adjudication entirely to the LLM's prompted reasoning.

On the argumentation side, Bilateral Gradual Semantics for Weighted Argumentation supplies the verdict mechanism. It defines separate acceptability ($\sigma^+$) and rejectability ($\sigma^-$) scores that converge to a unique pair even on cyclic graphs, and its hybrid variant satisfies nearly all of 23 axioms. It has been tested only on synthetic graphs with hand-set weights; we ground those weights in real credibility and stance estimates. On the Graded Acceptability of Arguments and Counter-Transitivity in Argument Ranking Semantics provide unilateral graded alternatives, which we use as ablations. On the Accrual of Arguments in Defeasible Logic Programming explains why binary semantics fail here: an attack on a minor sub-point should weaken a claim, not erase it.

Neuro-Argumentative Learning with Case-Based Reasoning is a constraint on design. Training gradual argumentation end-to-end by gradient descent is very sensitive to initialization, so we do not train the graph end-to-end. Base weights come from fixed estimators, and only verdict thresholds are fit on development data.

Research Program

The pipeline has four stages.

Facets and stances. Given a claim, an LLM generates facet questions. A shared retriever builds a candidate pool of about 100 passages. An entailment model scores $a_s(d,q)$ for each passage, facet, and stance.

Provenance clusters. Passages are grouped by near-duplicate text detection, shared outbound links, and wire-service attribution strings.

Credibility. Cluster credibility starts from the Media Bias/Fact Check ratings shipped with CONFACT. It is then adjusted per topic using RA-RAG-style cross-checking restricted to claims in the same topic cluster. This addresses the static-credibility weakness that CONFACT identifies.

Selection and verdict. Greedy selection with lazy evaluation picks $k$ passages. The selected passages form a weighted argumentation graph with two claim arguments, $C^+$ (the claim holds) and $C^-$ (it does not). Each selected passage is an argument with base weight $r_{c,q}\cdot a_s(d,q)$, and it attacks the claim argument of the opposite polarity. $C^+$ and $C^-$ attack each other. The bilateral hybrid semantics yields $(\sigma^+,\sigma^-)$ for both claim arguments. A small decision rule, fit on development data, maps these four numbers to a verdict: Supported, Refuted, Not Enough Evidence, or Conflicting.

Whether the bilateral pair actually separates the last two classes better than a single score is tested, not assumed. A secondary analysis treats prompt tokens as a cost and uses ParetoGreedy from Computing Approximate Pareto Frontiers for Submodular Utility and Cost Tradeoffs to trace coverage against context length, with its $(1-1/e,1)$ guarantee. This turns CONFACT's "5 beats 10" observation into a measured curve rather than a fixed rule.

Baselines

Naive reference. Relevance-only top-5 dense retrieval, followed by the same LLM reading the passages and outputting a verdict, with no source information. This is the CONFACT setting before any credibility method is added.

Strongest existing method for conflicting evidence. Source Background Augmentation from CONFACT: the same top-5 passages, each prefixed with its publisher's background, adjudicated by the LLM. CONFACT reports a 10% absolute F1 improvement for LLaMA-3.1 and a 5% accuracy boost for Mistral on the ModC split. The file gives only these gains, not absolute scores, so we re-run Source Background Augmentation ourselves. To beat it, we must match or exceed its macro-F1 on HumC and ModC at equal $k$ and equal LLM. We must also exceed it clearly on the syndication stress test and on the confusion between Not Enough Evidence and Conflicting.

Reliability-aware selection. RA-RAG's $\kappa$-reliable-and-relevant selection plus weighted majority voting. The file reports a 99.1% token reduction with negligible accuracy loss but gives no benchmark accuracy numbers (no reported result found on Lacuna for RA-RAG accuracy on NQ, TriviaQA, or HotpotQA). We run it on the same candidate pools.

Open-web AVeriTeC reference points. The dataset paper's baseline reaches 0.23 macro-F1 and GPT-3.5-turbo reaches 0.39 F1 on veracity, per AVeriTeC. HerO reaches a 0.57 overall shared-task score and 75.2% development-set veracity accuracy with a fine-tuned Llama-3.1-70b. We will not claim to beat HerO's full system. We will insert our selector and adjudicator into its retrieval output and compare like-for-like.

Fairness controls. All systems share the same candidate pool, the same entailment/stance model, the same verdict LLM where an LLM is used, the same $k$, and equal development-set tuning budgets for thresholds and prompts.

Ablations that isolate the key ingredients. The first ablation removes provenance clustering, so every URL is its own cluster. The second removes the cap $\tau$, eliminating stance balancing. The third replaces bilateral semantics with a unilateral gradual semantics in which rejectability is one minus acceptability. The fourth replaces argumentative adjudication with the LLM judge reading our selected set.

Why not just…?

Why not just give the LLM source backgrounds and let it decide? This is the most serious competitor, because CONFACT shows it beats pre-filtering and reweighting. Its weakness is structural. If the top five passages are five reprints of one false story, no amount of reasoning recovers the missing opposing evidence, and longer prompts carrying backgrounds degrade attention. Our stress test settles this directly: it injects increasing numbers of syndicated false copies into CONFACT pools. The fourth ablation (LLM judge reading our selection) also tells us whether a gain comes from selection alone. If the LLM judge on our selection equals the argumentative verdict, we will report that the argumentation component is unnecessary.

Why not just multiply relevance by credibility and take the top five? CONFACT already finds credibility weighting worse than full-context reasoning. Top-$k$ also cannot express redundancy: as DISCo argues, it returns several passages covering the same facet while missing complementary ones. A credibility-weighted coverage objective without provenance clustering or the stance cap is a stronger version of this idea. It is exactly our first and second ablations, so the experiments measure how much those two ingredients add.

Why not just reweight majority voting by estimated reliability? RA-RAG shows plain majority voting collapses when adversarial sources outnumber reliable ones. Reliability-weighted voting still counts syndicated copies separately whenever their origin's reliability is estimated as moderate. Coordinated campaigns can also inflate estimated reliability through mutual agreement, which is CONFACT's majority-vote trap. Collapsing provenance clusters before counting is the specific fix. If RA-RAG's estimator already down-weights such clusters enough, the stress test will show parity, and we will report it.

Why not just use binary argumentation? Under binary semantics, one credible attack on a peripheral facet rejects the entire claim, and "no evidence" is indistinguishable from "opposed." The third ablation tests the weaker, graded version of this objection.

Proposed Experiments

The primary benchmarks are CONFACT (3,180 claims; ModC and the human-verified HumC split; with publisher credibility ratings) and AVeriTeC (4,568 real-world claims, four veracity classes, temporal splits). The metric is macro-F1 throughout. On AVeriTeC we additionally report the per-class F1 for Not Enough Evidence and Conflicting, and the confusion rate between them.

Four checks would decide whether the idea is worth pursuing.

Syndication stress test. Following RA-RAG's adversary-majority setup, we inject $m \in \{0, 5, 10, 20\}$ near-duplicate false passages from new URLs into each CONFACT pool. The prediction is that top-$k$, Source Background Augmentation, and RA-RAG degrade as $m$ grows while our method stays roughly flat. Flatness is predicted because duplicates within a cluster contribute zero marginal gain.

Selection versus adjudication. A 2×2 factorial crosses selector {top-5 with source backgrounds, ours} with judge {LLM, bilateral argumentation}. This attributes any gain to the right stage.

Two-score verdicts. On AVeriTeC, we compare the separability of Not Enough Evidence and Conflicting under bilateral versus unilateral scores.

Budget sweep. We sweep $k$ from 3 to 10 and plot the ParetoGreedy coverage–token frontier. The prediction is that our selector does not show CONFACT's degradation beyond five passages, because the cap $\tau$ stops redundant same-side evidence from entering the set.

The proposal fails if, after matched tuning, Source Background Augmentation with top-5 retrieval equals our full system on HumC, AVeriTeC, and the stress test. In that case the formal machinery would have bought nothing, and we would report it as such.