Social scientists and public-health researchers now routinely ask large language models to read thousands of social media posts and report what people are saying. A typical output reads: "most users describe financial dependence as the reason they stayed," or "a small minority report withdrawal cramps." These sentences make claims about how common something is across a corpus, not about any single post. Current hallucination checks cannot catch them. Those checks ask whether some piece of evidence supports a claim. One post saying "I stayed because of money" is enough to pass "most users say they stayed because of money." Nobody verifies the word "most."
We propose to treat these frequency claims ("most," "many," "rarely," "about a third") as a distinct, measurable kind of hallucination, which we call prevalence hallucination. We also propose an audit that checks such claims by counting instead of by retrieving a supporting example. The audit labels every post with a cheap automatic annotator and corrects that annotator's bias using a few hundred human-labeled posts. This yields a confidence interval for the true share of posts expressing each theme. A summary sentence is flagged when the share it implies falls outside that interval. The work contributes three things: a benchmark with known ground truth, the audit method, and a direct test of whether today's claim-level verifiers are blind to this error.
Core Claim
Our central claim is that faithfulness to a corpus has two levels, and current tools check only the lower one. The lower level asks whether a statement is grounded in some post. The higher level asks whether the statement's implied frequency matches the corpus distribution. We expect LLM summaries of user posts to pass the first test while failing the second, in two predictable directions. The first is inflation of vivid or stereotyped themes. The hallucination literature calls a closely related pattern epistemic overconfidence: tentative evidence gets rewritten as a definitive statement, as catalogued by Liyan Tang et al.. The second is erasure of minority themes. The key claim is falsifiable. If an existence-based verifier, meaning one that asks only whether a supporting passage exists, already rejects inflated prevalence claims at a rate close to a counting-based audit, then prevalence hallucination is not a separate problem, and the proposal fails.
Why the existing intersection does not cover this
The closest work falls into three groups, and each checks a different target than ours.
The first group builds generation directly over user posts and scores hallucination against the source text. The two-layer retrieval system of Two-layer retrieval augmented generation framework for low-resource medical question-answering: proof of concept using Reddit data summarizes retrieved Reddit posts on xylazine and ketamine. Both GPT-4 and a 7B model receive a median hallucination score of 0 across 20 clinician queries. That score asks only whether the content is ungrounded. Because retrieval selects the posts most relevant to the query, the retrieved set is by construction a biased sample of the corpus. Any frequency language in the answer is therefore unverifiable under that protocol. This paper motivates our work: it shows a real deployment where frequency claims are made and never checked.
The second group gives statistical guarantees for claim-level factuality. CiteGuard: Conformal False-Discovery Control for Faithful Retrieval-Augmented Generation splits outputs into single-fact claims and scores each against retrieved passages with a model that judges whether one text supports another. It then caps the fraction of false claims that get through, a quantity called the false discovery rate. CoFact: Conformal Factuality Guarantees for Language Models under Covariate Shift extends such guarantees to drifting prompt streams. Mitigating LLM Hallucinations via Conformal Abstention bounds hallucination rates by letting the model decline to answer. We reuse their statistical spirit, calibrating against a small human-labeled set, but not their verification target. All three decide claim by claim whether a claim is supported. None estimates a population quantity. CiteGuard itself notes that its guarantee is only as good as its verifier, and our argument is that for prevalence claims the verifier is asking the wrong question.
The third group uses LLMs for qualitative coding of user text. LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis reaches a Cohen's κ of 0.81–0.87 with human coders, but only on at most 280 survey responses with a human reviewing every step. Can Large Language Models Transform Computational Social Science? finds zero-shot LLM labels reach only fair-to-good agreement, κ between 0.40 and 0.65 in 47% of tasks. That finding constrains our design: raw LLM labels cannot be counted naively, because systematic label bias becomes systematic prevalence bias. Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models catalogues the lack of evaluation for LLM-induced themes. It proposes matching LLM themes to human themes but never implements it, and never addresses how common each theme is.
Attributable and Scalable Opinion Summarization (HERCULES) is the one system that builds counting into generation. It maps review sentences onto a discrete tree and decodes summaries from the most frequent paths. It supplies part of our intuition, that aggregation should be counting. It differs in three ways. It is a generator, not an auditor of arbitrary LLM summaries. It fails on long-tail and contradictory opinions. It was tested only on hotel and product reviews.
Research Program
The work has three parts.
Benchmark with known ground truth. We build it by controlled resampling of already-labeled user-post corpora. The primary source is the #WhyIStayed/#WhyILeft corpus from #WhyIStayed, #WhyILeft: Microblogging to Make Sense of Domestic Abuse. It has 24,861 and 8,767 hashtag-labeled tweets, a 1,000-tweet subset coded by four annotators (free-marginal κ of 0.72), and a known natural mix: 36% reasons for staying, 44% reasons for leaving, 12% meta-comments. We add finer reason codes (financial, fear, children, self-blame, and so on) to a few thousand tweets. We then draw many sub-corpora of 500–5,000 posts in which the share of each theme is set by design, from about 1% to about 70%. The design deliberately includes cases where a salient theme is rare and a mundane one dominates.
Secondary sources are labeled tasks from the 25-benchmark suite used in Can Large Language Models Transform Computational Social Science?, resampled the same way. These test whether findings hold beyond one sensitive topic. We then prompt several LLMs of different sizes to write thematic summaries of each sub-corpus. Summaries are produced both by whole-corpus chunked summarization and by retrieval-based pipelines modeled on the two-layer Reddit system.
Prevalence-claim extraction and a quantifier scale. We split each summary into claims of the form theme plus frequency expression. We map each frequency expression to an interval of shares. For example, "most" is read as roughly above one half, and "a few" as roughly below one fifth. These intervals are fixed from human judgments collected once, before any system is evaluated, so the audit cannot tune them. Each theme is mapped to a label the annotator can apply to individual posts. We verify this mapping by hand on a sample, because an error in the mapping would be mistaken for an error in the summary.
Bias-corrected counting audit. For a theme $t$, an automatic annotator (a zero-shot LLM or a fine-tuned classifier) gives each of the $N$ posts a label $\hat{y}_i \in \{0,1\}$. A human labels a random subset $S$ of $n$ posts, giving $y_j$. The estimated share is
$$ \hat{\theta}_t = \frac{1}{N}\sum_{i=1}^{N}\hat{y}_i \;+\; \frac{1}{n}\sum_{j\in S}\left(y_j-\hat{y}_j\right). $$
The first term is the annotator's raw count. The second term corrects the annotator's average error, measured where human labels exist. Because $S$ is a random sample, the correction is unbiased regardless of how the annotator errs. By the central limit theorem, the interval width shrinks like $1/\sqrt{n}$, scaled by how often the annotator disagrees with humans. A claim is flagged as a prevalence hallucination when its quantifier interval does not overlap the confidence interval for $\hat{\theta}_t$. A theme whose interval lies above 10% but which the summary never mentions is flagged as an omission. We target human budgets of 200–500 posts per corpus. That is roughly the calibration size CiteGuard reports needing per domain, and it is small enough to be affordable for a working social scientist.
Baselines
The naive reference point is to accept the LLM summary as written. It sets the raw error rate we are trying to expose: the fraction of prevalence claims whose implied share is wrong against known ground truth. If that rate is small, below about 5% of claims across models and sub-corpora, the problem is not worth a method, and we will report that.
The strongest existing competitor is claim-level verification with false-discovery control in the style of CiteGuard: Conformal False-Discovery Control for Faithful Retrieval-Augmented Generation. It is the most rigorous published tool for deciding which generated claims to trust, and a reviewer would naturally expect it to handle this case. CiteGuard reports cutting the false discovery rate on Natural Questions from 30.6% for plain retrieval-augmented generation to 9.4% at a target of 0.10, while keeping 86–92% of supported claims. We run it on the same summaries with the user posts as the retrieval corpus. It gets the same number of human-labeled calibration examples as our audit (claims instead of posts, matched in annotation hours) and the same support-judging model. Beating it means catching substantially more of the inflated and deflated prevalence claims at matched precision. We also check whether its false discovery rate, measured against prevalence ground truth rather than existence, stays near its nominal target. We predict it will not.
The generation-side competitor is HERCULES from Attributable and Scalable Opinion Summarization. Its summaries are built from frequency counts, so it should rarely inflate. We run it on the same sub-corpora to test whether counting-based generation alone avoids prevalence errors, especially for rare themes. No numeric ROUGE or SummaC result was found on Lacuna for HERCULES; the file reports only that it beat InstructGPT and COOP qualitatively. We therefore treat it as a behavioral comparison, not a number to beat.
We also include LLM self-checking, where the summarizer is asked to verify its own frequency claims. The reference point for self-checking is weak: HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models reports ChatGPT self-detection accuracy of 58.53% on summarization.
Two ablations isolate the key ingredients. The first drops the human correction term and counts raw annotator labels. This isolates whether bias correction matters, and we measure interval coverage with and without it. The second keeps human correction but replaces counting with retrieval of the top-$k$ matching posts, as a retrieval pipeline would. This isolates whether counting over the full corpus, rather than sampling by relevance, is what catches the error.
Why not just…?
Why not just label every post with an LLM and report the proportions? This is the most serious threat, because it is our method minus the correction term, and it may well be most of the gain. The evidence against using it unaltered is that zero-shot LLM agreement with humans sits at κ 0.40–0.65 in about half of social-science tasks (Can Large Language Models Transform Computational Social Science?). Disagreement that leans in one direction translates directly into biased shares. The first ablation settles this. If raw counts already give intervals with close to 95% coverage across themes, we will say so plainly and present the contribution as the benchmark plus the finding that counting, not retrieval, is what matters. If coverage collapses for particular themes, such as stigmatized or sarcastic ones, the correction earns its place.
Why not just run an existing claim verifier like CiteGuard? Existing verifiers ask whether support exists, and one supporting post exists for almost any theme that occurs at all. We expect them to pass "most users say X" whenever a single post says X. Their guarantees also degrade under domain shift: CiteGuard reports a 4–6% loss of false-discovery control when calibration moves from Wikipedia to medicine. The head-to-head comparison measures this directly. If the verifier catches inflated claims about as well as the audit, our core claim is false.
Why not just generate summaries by counting, as HERCULES does? For customer reviews with strong agreement, this may be enough. It needs roughly 8–10 or more reviews behind a tree path to work, it drops long-tail and contradictory views, and it cannot audit the free-form LLM summaries researchers actually use. We test it on rare-theme sub-corpora. If it matches our audit there, we recommend counting-based generation for those settings.
Why not just prompt the model to hedge or to double-check its frequency words? Self-detection is weak even for simpler errors, at 58.53% on summarization in HaluEval. Hedging prompts may also simply shift all quantifiers toward vagueness, which lowers the measured error without making the summaries more informative. We include a hedging-prompt condition and report both the error rate and how specific the quantifiers are, so that vague summaries cannot win by saying nothing.
Proposed Experiments
The first decisive check is whether prevalence hallucination exists at a meaningful rate. On the controlled #WhyIStayed and social-science sub-corpora, we measure the fraction of frequency claims whose interval misses the true designed share, split by model size, summarization pipeline, and theme rarity. Bigger models are not expected to fix this. TruthfulQA: Measuring How Models Mimic Human Falsehoods and FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation both found larger models no more factual, and sometimes less. We test the specific prediction that larger models inflate stereotyped themes more.
The second decisive check is the comparison with the existing claim verifier. We report detection recall and precision for inflated, deflated, and omitted themes, and the verifier's effective false discovery rate measured against prevalence ground truth.
The third is statistical: the empirical coverage of the audit's intervals across human budgets of 100, 200, and 500 posts, against the raw-count ablation, and whether interval width shrinks at the expected $1/\sqrt{n}$ rate.
Finally, an in-the-wild audit applies the method to the Reddit xylazine and ketamine corpora from Two-layer retrieval augmented generation framework for low-resource medical question-answering: proof of concept using Reddit data. There, answers scored as hallucination-free will be re-examined for frequency claims. Posts are sampled by stratified random sampling from the full corpus, not from the retrieved set. This asks whether the "zero hallucination" verdict survives once "commonly reported" must match what is actually common. The Reddit corpora have no gold theme labels, so this last study reports flagged claims and their human-checked shares as a case study rather than as a benchmark number.
References
- Evaluating large language models on medical evidence summarization
- Two-layer retrieval augmented generation framework for low-resource medical question-answering: proof of concept using Reddit data
- CiteGuard: Conformal False-Discovery Control for Faithful Retrieval-Augmented Generation
- CoFact: Conformal Factuality Guarantees for Language Models under Covariate Shift
- Mitigating LLM Hallucinations via Conformal Abstention
- LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
- Can Large Language Models Transform Computational Social Science?
- Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models
- Attributable and Scalable Opinion Summarization
- #WhyIStayed, #WhyILeft: Microblogging to Make Sense of Domestic Abuse
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation