Retrieval as the Fact-Checker for Vaccine-Hesitancy Explanations

People who hesitate about a vaccine usually hold a specific belief underneath their question, and that belief is often never stated. One example: "my daughter isn't sexually active, so she doesn't need the HPV shot." The unstated premise is that the vaccine only matters after exposure, and a good reply has to find that premise and correct it with evidence. Large language models can write fluent replies of this kind with no task-specific training, but health communication has two gaps that math and code do not. There is almost no data of worked explanations to learn from. There is also no automatic checker, like a compiler or a numeric answer, that says whether an explanation is correct.

This note proposes to use retrieval to fill the checker gap. Retrieval here means looking up passages from a fixed library of authoritative documents such as public-health guidance, vaccine labels and systematic reviews. The system works in four steps. First, a prompted model with no examples (zero-shot) names the hidden premise behind a user's concern. Second, it retrieves evidence about that premise rather than about the literal wording of the question. Third, it writes its explanation as a chain of short, individual claims, each tied to a retrieved passage. Fourth, it keeps only the claims that a separate model judges the passage actually supports. The explanations that pass become the scarce training resource: they are reused as worked examples in the prompt (few-shot) for smaller models.

Core Claim

In a domain without a programmatic verifier, checking each generated claim against retrieved authoritative text is a usable substitute verifier. Its value comes from two things together: retrieving on the inferred premise, and filtering at the level of single claims.

We expect the gain over ordinary retrieval-augmented generation (RAG, prompting a model with passages retrieved for the user's literal question) to be concentrated on concerns whose premise is implicit. These are exactly the concerns that keyword classifiers such as CoNTACT are reported to miss. For concerns that are stated explicitly, we expect little or no gain. If the gain turns out to be uniform or absent, the premise-inference step is not earning its place.

Positioning Against Prior Work

The motivating evidence comes from several directions. Generated Knowledge Prompting has a model write background statements before answering. It improves commonsense accuracy, but 17% of those statements were non-factual, and non-factual statements caused 86% of the cases where the model was misled. That is the failure a claim-level retrieval check targets. We keep its "generate intermediate knowledge first" idea and replace trust in the model's memory with checking against documents.

A systematic review of language models in chronic disease care, Unveiling the Potential of LLMs in Chronic Disease Management, found that grounding in guidelines roughly tripled the odds of an accurate recommendation (odds ratio 2.89, pooled accuracy 71%). It also reports fabricated citations in over half of evaluated instances in one colorectal review. So retrieval alone does not make outputs faithful, which motivates checking after generation.

Work that constrains the design comes from methods that depend on verifiers. Template-based data generation, LaTRO and Mixed SFT followed by RLVR all synthesize or optimize reasoning traces, but only where correctness can be checked by code execution or answer matching. We borrow their logic of generate-then-filter as a way around missing rationale data. We do not assume their verifier exists; we ask whether evidence entailment can stand in for one.

GPS turns if-then rules in retrieved documents into a graph and asks clarifying questions. It is explicitly limited to documents with Boolean conditional structure, which belief-driven misconceptions lack. We infer premises from the user's text rather than from rules in the documents.

The vaccine-specific systems contribute data and a comparison point. VIRADialogs provides more than 8,000 real chatbot conversations with a 181-intent taxonomy. It shows that intent-based pipelines cover only part of what users say: unsupervised discovery reached 45% intent coverage. The South Asian chatbot trial used scripted 5C-model messages, the 5C model being five drivers of hesitancy: confidence, complacency, constraints, calculation and collective responsibility. The trial moved influenza uptake but not COVID-19 uptake, a warning that fixed scripts age badly as guidance changes. The rural HPV survey documents premise-type misconceptions and the absence of explanation corpora for such communities, but it builds no system.

Research Program

The system has four prompted stages, all running on the same frozen model.

Premise extraction. A zero-shot prompt reads the user's utterance and outputs one to three hidden premises in plain words. Each premise is tagged with a 5C dimension and with whether it is empirically checkable, such as a safety or timing belief, or value-laden, such as a liberty or institutional-trust concern.

Retrieval. For each checkable premise, the system writes a query and retrieves passages from a frozen corpus of authoritative documents with a hybrid of keyword and dense retrieval. The corpus is frozen so that every system sees identical evidence.

Explanation writing. The generator writes the explanation as numbered atomic claims $c_1, \dots, c_N$, each pointing to a passage $e_i$.

Checking. A separate entailment judge, a natural-language-inference model cross-checked against an LLM judge, scores each claim. The support rate of an explanation is

$$S = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}[\text{judge}(e_i \Rightarrow c_i)],$$

where the indicator is 1 when the passage entails the claim. Unsupported claims are regenerated once and then dropped. For value-laden premises that no document can settle, the system is instructed to acknowledge the value and restate only the verifiable facts, rather than fabricate "evidence."

Explanations with $S = 1$ that also pass a length and coherence filter enter a growing bank of verified examples. At inference time, a smaller model (7–8B parameters) receives the $k$ bank entries most similar to the query as few-shot demonstrations. As a secondary arm, we fine-tune on the bank to test whether retrieval-verified traces work as training data the way code-verified traces do in math.

Baselines

Do-nothing reference. The naive reference point is the same frozen model answering the user's concern zero-shot with no retrieval. It measures how often parametric memory alone produces supported claims. No reported result was found on Lacuna for claim support rates on vaccine-hesitancy replies, so this must be measured.

Strongest existing method: plain RAG. The strongest competitor is ordinary RAG over the identical corpus, retrieving on the user's literal utterance and producing a cited answer. It is the design the chronic-disease review found most effective (odds ratio 2.89 for accuracy, pooled accuracy 71%). The review also reports over 50% fabricated citations in one setting, which gives a concrete failure rate to beat. For fairness, plain RAG gets the same retriever, the same number of retrieved passages, the same generator model, and the same budget for prompt tuning (an equal number of prompt variants tried on a shared development split). It also gets one regeneration pass, so that our system's extra call is matched in compute.

We count ourselves as beating plain RAG only if two things hold on held-out VIRADialogs utterances. The support rate $S$ must be higher by at least 10 points on the implicit-premise subset. And there must be no loss in clinician-rated relevance on the explicit subset.

Intent retrieval. The second baseline is VIRA-style intent retrieval: classify the utterance into one of the 181 intents and return a response template for that intent. The file reports its classifier at weighted-F1 of about 0.795. This baseline is near-perfect in factuality whenever the intent exists, so it tests whether generation adds anything. Its natural weakness is utterances outside the taxonomy.

Generated Knowledge Prompting. The third baseline is Generated Knowledge Prompting with self-generated background statements and no retrieval. Its published non-factual statement rate of 17% on commonsense tasks serves as a reference, not as a directly comparable number.

Ablations. Two ablations isolate our ingredients. The first swaps premise-targeted retrieval for literal-query retrieval while keeping claim filtering. The second keeps premise retrieval but removes claim filtering. If either ablation matches the full system, the removed component is unnecessary.

Why not just…?

Why not just use plain RAG with a good system prompt? This is the real threat, and we treat it as the null hypothesis. For explicit concerns, such as whether a vaccine raises clotting markers in pregnancy, retrieving on the literal question likely finds the right passage, and we expect parity. Our claim is narrower: for utterances like "she isn't sexually active," literal retrieval tends to fetch passages about sexual activity rather than about vaccination timing. The first ablation measures this directly. If plain RAG matches us on the implicit subset, we will report that premise inference is unnecessary.

Why not just classify the intent and return an expert-written answer? For covered intents this is safer than any generator. But the VIRADialogs work shows that intent taxonomies leave much of real conversation uncovered, with only 45% coverage found by discovery. The South Asian trial shows that scripted content lost effect as COVID-19 policy shifted. We test this by splitting evaluation according to the intent classifier's confidence. We expect intent retrieval to win when confidence is high and to fail when it is low. A system that routes between the two is a legitimate outcome of this study.

Why not just use a bigger model, or self-consistency, without retrieval? Generated Knowledge Prompting shows that the model's own knowledge helps but carries a 17% error rate that drives most failures. Nothing in the file suggests scale removes this error in the health domain. We include a larger frozen model without retrieval as an additional reference to measure how much scale alone buys.

Why not just fine-tune on synthetic explanations, as in math? Math pipelines such as Mixed SFT followed by RLVR rely on an answer checker to filter traces, and no such checker exists here. Our secondary arm tests exactly this: fine-tuning on unfiltered versus retrieval-verified traces. If the unfiltered traces work just as well, the verifier adds nothing for training.

Proposed Experiments

Evaluation sets. The main test set is 1,000 held-out user utterances from VIRADialogs. Annotators mark each utterance as having an explicit or implicit premise, and as checkable or value-laden. Before any system is run, 400 utterances are double-annotated by public-health reviewers for premise, relevance, accuracy, and whether the reply avoids condescension.

Premise extraction check. To test premise extraction separately, we use the Dutch argument corpus from CoNTACT, about 8,800 tweets and 5,200 Facebook comments across eight argument types. We measure whether the zero-shot premise tags recover the argument label, focusing on posts CoNTACT misclassifies. This gives an independent check on the implicit cases.

Low-resource probe. For the low-resource setting that motivates the work, we write a small probe set of rural HPV concerns, built from the refusal reasons reported in the rural HPV survey. We use it only for qualitative error analysis, since we have no raw survey text.

Metrics. The metrics are claim support rate $S$ under two independent judges, with judge agreement reported. We also measure the fabricated-citation rate, meaning citations to passages that do not exist or do not contain the claim. Finally, we collect clinician ratings of accuracy and of whether the reply addresses the user's actual concern.

Decisive checks. Four results would settle the proposal. First, the full system beats plain RAG on implicit-premise utterances while tying on explicit ones. Second, the premise-retrieval ablation accounts for most of that gap. Third, on low-confidence intents the system outperforms intent retrieval in clinician-rated relevance without losing accuracy. Fourth, an 8B model prompted with bank-verified examples approaches the large model's support rate, while the same model given unverified examples does not.

Persuasion. We deliberately do not claim effects on persuasion or vaccine uptake. Simulated agent populations such as VACSIM showed instability across demographic subgroups, so they would not be a trustworthy stand-in for people. Any persuasion claim would need a follow-up randomized trial.