What do people want to fact-check?

Bijean Ghafouri, Dorsaf Sallami, Luca Luceri, Taylor Lynn Curtis, Jean-Francois Godbout, Emilio Ferrara, Reihaneh RabbanyarXiv 2026 · arXiv

TL;DR

While most misinformation research focuses on the supply of fake news, this paper investigates the demand for fact-checking by analyzing what ordinary people actually choose to verify when given an open-ended AI tool. The authors reveal a critical mismatch: over 21% of user queries are fundamentally unverifiable (e.g., moral or subjective claims), and standard evaluation benchmarks like FEVER fail to reflect real-world user behavior, exposing a severe alignment gap for AI verification systems.

What Problem Does This Paper Address?

Existing research on misinformation heavily favors the "supply side"—analyzing what falsehoods circulate, who spreads them, and how to correct them. However, we know very little about the "demand side." When everyday users encounter uncertainty, what do they actually want to verify?

Currently, fact-checking pipelines are curated by professional editors, journalists, or trending algorithms. AI models trained on these expert-curated datasets inherit this specific framing. If real-world user queries look fundamentally different from these datasets, current AI fact-checking tools will generalize poorly when deployed in the wild. This paper maps out public verification demand to help researchers design AI systems that match actual human cognitive needs.

Methods

The authors built an interactive AI fact-checking application called Veracity. They recruited 1,025 participants via Prolific to submit statements they were personally uncertain about. After filtering out inattentive participants using an instructional manipulation check, they were left with a final sample of 457 participants who contributed 2,473 unique English-language claims.

To analyze these claims, the researchers used an LLM (GPT-OSS-20B) to automatically classify each statement along five semantic dimensions:

  1. Domain: The topic area (e.g., Politics, Science, Lifestyle).
  2. Epistemic Type: The reasoning structure (Descriptive, Causal, Statistical, Normative).
  3. Verifiability Level: How checkable the claim is (Direct, Indirect, Unverifiable).
  4. Target Entity: What the claim is about (Individual, Organization, Phenomenon, etc.).
  5. Temporal Orientation: The time frame (Past, Present, Future, General).

They also calculated a Shannon entropy score for each user to evaluate behavioral diversity and compared their real-world user dataset against a random sample of 2,500 claims from FEVER, a standard fact-checking benchmark.

Datasets

The paper introduces and analyzes a novel dataset of 2,473 user-generated fact-checking requests.

For evaluation and comparison, the authors utilized the FEVER (Fact Extraction and VERification) dataset, a widely used benchmark derived primarily from Wikipedia hyperlinks and text.

Key Insights and Ideas

  • Cognitive Anchoring: Users exhibit broad topical curiosity but narrow epistemic habits. Even though they ask about a massive variety of subjects, they overwhelmingly default to simple, descriptive claims about present-day observables. They use fact-checking tools to fill discrete information gaps rather than to parse complex causal arguments.
  • The Adjudication Mismatch: A substantial portion of public demand falls entirely outside the scope of empirical science. Users frequently attempt to "outsource" subjective, moral, or predictive uncertainties (e.g., "Is AI going to destroy human creativity?") to systems built strictly for factual verification.
  • The Synthetic Benchmark Trap: Standard evaluation datasets like FEVER do not reflect real-world demand. FEVER is heavily skewed toward past-tense, entertainment-related trivia about specific individuals. Real users, by contrast, care about present-day phenomena, science, health, and politics.
  • The "Grey Area" Reality: FEVER claims are sharply polarized (clearly true or clearly false). Real user queries cluster heavily in the ambiguous middle (veracity scores around 0.58), meaning people seek help with complex, nuanced, and grey-area statements.

Key Results

The semantic mapping of user requests yielded distinct distributions across all five dimensions:

  • Domain: Science and Technology was the most popular category at 22.3%, followed closely by Politics and Government at 18.6%.
Table 1: Distribution of domain classifications
  • Epistemic Type & Verifiability: An overwhelming 72.1% of claims were purely descriptive. Crucially, 21.4% of all submitted claims were classified as completely unverifiable empirical statements (such as moral judgments or speculative predictions).
Table 2: Distribution of epistemic classifications
Table 3: Distribution of verifiability classifications
  • Target Entity: Half of all user queries (50.5%) targeted abstract or systemic phenomena rather than specific individuals.
Table 4: Distribution of target classifications
  • The FEVER Divergence: When compared to FEVER, the structural differences were stark. For example, 63.2% of FEVER claims fall under "Lifestyle/Misc" (mostly pop culture trivia), compared to just 16.1% of real user queries. Conversely, FEVER contains only 3.1% Science/Technology claims, while real users dedicate 22.3% of their queries to this domain.

Limitations and Future Work

The authors note several limitations to their study:

  1. Sample Bias: The participant pool was restricted to English-speaking Prolific users interacting with a single experimental application, limiting cultural and linguistic generalizability.
  2. Automated Labeling: The taxonomy classification relied entirely on a single LLM without human annotation validation.
  3. Model-Generated Veracity: The veracity scores used to determine claim ambiguity were themselves generated by an LLM, which could introduce systematic biases.

Future Work: The authors emphasize the urgent need to build "demand-driven" evaluation corpora that mirror actual public uncertainty. They also suggest that next-generation verification tools must be designed to perform "epistemic triage"—meaning they shouldn't just look for evidence, but actively detect and communicate to the user when a query is inherently unverifiable.

Summary updated

Abstract

Research on misinformation has focused almost exclusively on supply, asking what falsehoods circulate, who produces them, and whether corrections work. A basic demand-side question remains unanswered. When ordinary people can fact-check anything they want, what do they actually ask about? We provide the first large-scale evidence on this question by analyzing close to 2{,}500 statements submitted by 457 participants to an open-ended AI fact-checking system. Each claim is classified along five semantic dimensions (domain, epistemic form, verifiability, target entity, and temporal reference), producing a behavioral map of public verification demand. Three findings stand out. First, users range widely across topics but default to a narrow epistemic repertoire, overwhelmingly submitting simple descriptive claims about present-day observables. Second, roughly one in four requests concerns statements that cannot be empirically resolved, including moral judgments, speculative predictions, and subjective evaluations, revealing a systematic mismatch between what users seek from fact-checking tools and what such tools can deliver. Third, comparison with the FEVER benchmark dataset exposes sharp structural divergences across all five dimensions, indicating that standard evaluation corpora encode a synthetic claim environment that does not resemble real-world verification needs. These results reframe fact-checking as a demand-driven problem and identify where current AI systems and benchmarks are misaligned with the uncertainty people actually experience.
BibTeX
@misc{ghafouri2026what,
  author = {Bijean Ghafouri and Dorsaf Sallami and Luca Luceri and Taylor Lynn Curtis and Jean-Francois Godbout and Emilio Ferrara and Reihaneh Rabbany},
  title = {{What do people want to fact-check?}},
  year = {2026},
  eprint = {2602.10935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.10935},
}
Figure