Platforms increasingly want many independent automated verifiers, each run by a different provider, to mark which phrases in a news article are false. That raises two problems at once. The first is money: someone has to pay providers for honest, effortful reports. The second is ground truth: professional fact-checkers can audit only a small fraction of posts, and they mostly audit the ones going viral. We propose a verification market that is strictly self-funding. Providers stake money on their phrase-level reports. When a post is audited, the stakes are redistributed so that more accurate providers gain exactly what less accurate ones lose. Each audited post's payout is scaled by the inverse of the probability that it was going to be audited. The second step is the key ingredient. Without it, providers learn that only viral posts are ever scored and stop working on everything else. With it, every post is worth the same to a provider in expectation, so effort is spread evenly even though audits are concentrated.
In two sentences, the idea is this. Pay-on-audit verification markets combined with targeted auditing create an incentive to work only where the auditor looks. Inverse-audit-probability scoring with a random-audit floor, plus zero-sum stake redistribution, removes that incentive without any outside subsidy. The open question is how much aggregation accuracy and payout risk this costs relative to sponsored mechanisms.
Where the gap is
Three strands of prior work each solve part of this problem.
Wagering, which supplies the method and the main contrast. Decentralized Aggregation of LLM Predictions via Wagering Mechanisms introduces WALLA. In WALLA, language models submit a probability together with a learned wager, and each model is scored against a leave-one-out aggregate of the others. This makes honest reporting a dominant strategy, and WALLA matches centralized stacking on QA benchmarks. It leaves two things open. First, it runs a deficit that is bounded by a constant, which a sponsor must cover. Second, it assumes every question eventually resolves and works only on single labels. We give up the leave-one-out construction in exchange for exact budget balance, and we move the setting to phrase-level reports scored only on a sparse, non-random subset of items.
Scarce, targeted audits, which supply the regime. Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation introduces Curb. Curb decides when to send a post to professional fact-checkers by modeling exposures and crowd flags as stochastic event streams, and its optimal checking intensity is proportional to current exposure. Curb's crowd has no incentives, and its fact-checking budget comes from outside. We reuse Curb's audit rule as the source of audit probabilities. What we add is the observation that such a rule is predictable, and that predictable audits distort paid verifiers.
Phrase-level misinformation, which supplies the task. MisSpans shows that false and true details co-occur within the same sentences. It reports that the F1 of 15 LLMs on identifying and typing false spans was "generally falling below 0.5". No single model is reliable here, so combining several models' span reports is a natural place for an aggregation mechanism.
Two further papers constrain the design. Incentive-Compatible Forecasting Competitions proves that its lottery-based selection loses truthfulness when events are correlated. Spans inside one article are correlated, so we avoid rank- or lottery-based rewards and use additive proper scores, whose truthfulness for a risk-neutral agent does not depend on independence. Stochastically Dominant Peer Prediction shows that risk attitudes can break linear scoring. Our inverse-probability weights increase payout variance, so risk is a live concern that we measure rather than assume away. LOPT lists exact balance between taxes and allowances as unsolved for learned tax planners. That is the same gap in a different setting, and it motivates making balance hold by construction rather than by learning.
Core Claim
For post $d$, each provider $i$ reports a falsity probability for every token and a stake $m_{i,d} \ge 0$. The post is audited with probability $q_d$. This probability is computed only from signals the providers do not control, namely exposure and crowd flags (Curb's intensity), and it is floored at $q_{\min}$. Let $A_d \in \{0,1\}$ indicate whether $d$ was audited, and let $s_{i,d} \in [0,1]$ be a bounded proper score of $i$'s token report against the audited span labels; a token-averaged Brier score is one choice. Provider $i$'s net transfer is
$$t_i = \sum_d \frac{A_d}{q_d}\, m_{i,d}\,\big(s_{i,d} - \bar s_d\big), \qquad \bar s_d = \frac{\sum_j m_{j,d}\, s_{j,d}}{\sum_j m_{j,d}}.$$
Because $\sum_i m_{i,d}(s_{i,d} - \bar s_d) = 0$ for every post, the transfers sum to exactly zero on every realization. No sponsor is needed and no money is burned.
Because $q_d$ does not depend on any provider's report, $\mathbb{E}[A_d/q_d] = 1$. Each post therefore contributes to a provider's expected payoff as if it were always audited. That is what removes the incentive to concentrate effort on likely-audited viral posts. The floor $q_{\min}$ bounds a provider's per-post loss at $m_{i,d}/q_{\min}$, which sets the collateral each provider must post.
One cost must be faced honestly. Because $\bar s_d$ includes provider $i$'s own report, truthfulness holds only approximately for large stakes. We quantify that distortion and compare it with a leave-one-out variant, which restores exact truthfulness but reintroduces a deficit.
Research Program
The work has three parts.
The first part is analysis. We will characterize how far the self-inclusive weighted score departs from truthfulness as a function of provider $i$'s share of the stake pool. We will bound the variance of $t_i$ in terms of $q_{\min}$. We will also show that when payouts are not reweighted, a provider who can choose effort per post, and who knows $q_d$ roughly, optimally invests effort in proportion to $q_d$.
The second part is construction. We will build a pool of verifier agents with deliberately varied quality: small and large open models, models with and without retrieval in the style of RAAR, and uninformed noise agents. Each agent has a stake policy that is a small network over its own hidden states, following WALLA's wager networks. Each also has an effort choice: it can answer cheaply from the model alone, or pay a compute cost to run retrieval. The platform's aggregate span prediction is the stake-weighted average of the token probabilities.
The third part is a semi-synthetic test bed. MisSpans articles have span labels but no spread histories, and Curb's Twitter and Weibo stories have spread histories but no span labels. We therefore attach exposure and flag streams sampled from the Curb cascades to MisSpans items, and run Curb's rule to set $q_d$. This pairing is artificial, and we report results separately for whether the true falsity of an item is correlated with its assigned virality or independent of it.
Baselines
The naive reference is uniform averaging of the agents' token probabilities, with no stakes and no payments. WALLA reports that uniform averaging degrades sharply as the pool grows from 4 to 12 models. On the same pool, we will measure span F1 on MisSpans for pool sizes from 4 to 12.
The strongest existing mechanism is WALLA itself (Decentralized Aggregation of LLM Predictions via Wagering Mechanisms). We will adapt it to token reports, give it the same agents, the same audited items, and the same stake-network architecture and tuning budget, and record its accuracy alongside the sponsor deficit it actually incurs. The file reports that WALLA matches Stacked Generalization on PubMedQA. No reported result was found on Lacuna for WALLA's scalar accuracy or deficit, and none for any span-level use of it. We count our mechanism as beating WALLA if it reaches span F1 within one standard error of WALLA at zero subsidy. If our F1 is lower, the price of exact balance is that gap.
The centralized upper reference is a stacked aggregator trained with full access to all agents' outputs on all labeled items. It shows how much information decentralization leaves on the table. The best single verifier is another reference. MisSpans reports F1 "generally below 0.5" across 15 LLMs, and the file gives no number for the top individual model.
Two ablations isolate the key ingredient. The first removes the inverse-probability weights and scores only the audited posts, unweighted. This variant is still truthful per post but invites selective effort. The second replaces Curb's targeted audits with uniform random audits at the same audit budget. It tests whether targeting matters once reweighting is in place.
Why not just…?
Why not just weight models by their past accuracy on audited posts, with exponential weights and no money at all? In a pool of fixed, non-strategic models this is a real threat, and it may match our aggregation accuracy. We will report that comparison directly. Its weakness appears when providers choose effort or decide whether to participate. Reputation weighting pays nothing for the retrieval compute that improves accuracy. Like unweighted pay-on-audit scoring, it also judges providers only on viral posts. Our effort-choice experiment measures whether agents under reputation weighting, unweighted scoring, and our scheme keep doing costly retrieval on low-exposure posts. If reputation weighting sustains effort equally well, the money adds nothing and we will say so.
Why not just run WALLA and accept a small sponsored deficit? A bounded deficit may be cheap. But WALLA's guarantee assumes outcomes resolve. Under sparse, targeted audits, the question of where its scores come from is exactly the selective-effort problem, and a deficit does not fix that. We will run WALLA with and without our inverse-probability weights. This separates the contribution of exact balance from the contribution of reweighting, and shows whether the reweighting idea alone is the real result.
Why not just fine-tune one strong span detector on the audited labels? Pooling labels into a single centralized model is the obvious move, and the stacked reference measures its ceiling. It assumes one party owns all the models and labels. The decentralized setting exists because providers keep their weights private, as WALLA argues. If a single fine-tuned detector clearly dominates the market's aggregate, that is a valid negative finding about the value of decentralized verification on this task.
Why not just audit uniformly at random so no reweighting is needed? Uniform audits remove the shirking incentive trivially. But Curb shows that exposure-targeted checking reduces misinformation exposure compared with simpler rules, and a platform will not give up that targeting. The uniform-audit ablation measures what exposure-weighted error the platform would sacrifice to avoid reweighting.
Proposed Experiments
The first decisive check is selective shirking. With effort-choosing agents and Curb-targeted audits, we measure two things for each scoring rule: the correlation between each agent's retrieval rate and the post's exposure, and the span F1 on never-audited posts. Our prediction is that unweighted scoring produces a clear positive correlation and worse F1 on low-exposure posts, and that reweighting removes both. If the correlation does not appear without reweighting, the core motivation is wrong.
The second check is the cost of balance. We sweep the floor $q_{\min}$ and plot three quantities: aggregate span F1 on MisSpans, the payout variance for an honest high-quality agent, and realized subsidy. Subsidy should be exactly zero for our rule and positive for WALLA. A small $q_{\min}$ is expected to make honest agents' payouts noisy, and with risk-averse agents that noise could erode truthful reporting as Stochastically Dominant Peer Prediction warns. We will test that by adding agents with concave utilities and checking whether they shade their reports toward the stake-weighted consensus.
The third check is transfer. We repeat the document-level version of the task on AMTCele, PHEME and COCO, the cross-domain benchmarks used by RAAR. The file reports that RAAR-14b achieved the highest average F1 across these three, but gives no scalar value, so no reported result was found on Lacuna for that number. On these sets we test whether stakes learned in one domain put uninformed agents near zero in another.
Across all three checks, the metrics are span-level and document-level F1, sponsor subsidy, payout variance, effort allocation against exposure, and exposure-weighted error under the Curb cascades.
References
- Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
- Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation
- MisSpans: Fine-Grained False Span Identification in Cross-Domain Fake News
- Incentive-Compatible Forecasting Competitions
- Stochastically Dominant Peer Prediction
- LOPT: Learning Optimal Pigovian Tax in Sequential Social Dilemmas
- RAAR: Retrieval Augmented Agentic Reasoning for Cross-Domain Misinformation Detection