Large language models answer specialist questions in medicine, pharmacy, and clinical research with the same fluency they bring to trivia, and they are wrong much more often. The usual fix is to give the model reference material at answer time, but in expert domains this often fails. Appending formal theorem definitions to prompts did not significantly help models on university STEM problems (TheoremQA). Domain knowledge stores also have large holes: 44% of a medical reader's errors came from facts missing from both its textbook and its knowledge graph (KMQA). This note proposes using external knowledge to check answers rather than to write them. The method breaks a model's answer into small claims and tests each claim separately against several differently shaped knowledge sources: a knowledge graph of (subject, relation, object) facts, an ontology or code hierarchy, and free text or tables. It then answers only when the sources agree, with a statistical guarantee on how often an answered question is wrong.
The central distinction is between a source that contradicts a claim and a source that is silent about it. Open-domain hallucination detectors merge the two. In expert domains, silence is how the domain knowledge gap shows up, and it calls for a different action, namely abstaining or deferring, than contradiction does, which calls for rejecting the claim. If silence and disagreement across heterogeneous sources are scored correctly, the result should be a better confidence signal than the model's own self-consistency. Self-consistency is a weak signal in domains where models are consistently wrong rather than randomly wrong. That signal can then be fed into conformal risk control, a distribution-free calibration method that sets an answer/abstain threshold so that the error rate stays below a target.
Core Claim
On expert-domain question answering, a confidence score built from per-claim, per-source verdicts (support, contradict, or silent) will answer more questions at a fixed guaranteed error rate than four alternatives: a score from the model's own sampling consistency, a score from a single text-retrieval source, a score from a judge that reads all evidence pooled into one prompt, and the text-plus-triples detector of KnowHalu. The guarantee itself takes the standard conformal form. A threshold $\hat\lambda$ is chosen on $n$ held-out calibration questions so that, for a new question, $P(\text{answered} \wedge \text{wrong}) \le \alpha$, where $\alpha$ is the user's tolerated error. Because the guarantee holds for any score, the scientific question is entirely about which score answers the most questions. We predict the advantage will be larger in expert domains than on open-domain HaluEval, because silence is rarer there and self-consistency is better calibrated.
Relation to Prior Work
Three lines of work motivate this proposal, and each stops short of it. KnowHalu breaks answers into one-hop queries and checks them against both Wikipedia passages and Wikipedia triples, and it also catches "off-target" answers that are true but do not answer the question. It is the closest prior work. It differs from this proposal in three ways: it is evaluated only on open-domain HaluEval, it uses one encyclopedia in two formats rather than genuinely different domain sources, and it outputs a detection judgment with no calibrated abstention guarantee. MultiHal shows that two-hop Wikidata paths in the prompt reduce hallucination scores by 0.29 to 0.42. It injects knowledge for generation and never examines what happens when the graph has nothing to say. TRAC supplies the statistical machinery we reuse, conformal prediction sets over retrieval and generation. It also supplies a constraint: on Natural Questions it needs about 20.4 unique answers per set to reach 0.90 coverage at $\alpha=0.1$. That "set bloat" is what a sharper score should shrink, and our selective-answering formulation replaces large sets with abstention.
Two papers constrain the design. TheoremQA finds that supplying definitions does not help models apply them. This is why we use knowledge only to verify, which only requires matching a claim against a fact. HaluEval finds that free-form chain-of-thought hurts hallucination recognition in QA. It also finds that retrieval raises ChatGPT's QA recognition accuracy from 62.59% to 76.83%. Together these push us toward short, structured claim checks instead of long deliberation.
Two papers supply domain resources and heterogeneous sources. KMQA pairs pharmacist-exam questions with textbook passages and Chinese Medical Knowledge Graph triples. GKI-ICD combines ICD code descriptions, UMLS synonyms, and the ICD code tree. Both bake knowledge into a discriminative classifier's weights and never measure generative hallucination. We reuse their sources as independent checkers. Multi-source work on commonsense QA (Chen et al.) found that dictionary definitions added 5.5 points against 2.3 for ConceptNet. This supports the premise that differently shaped sources carry complementary signal rather than redundant copies.
Research Program
The pipeline has four stages. In the first, a base model answers the question with a short rationale. In the second, the rationale is decomposed into atomic claims, and one extra claim is added: "the answer addresses the question as asked," which imports KnowHalu's off-target check. In the third, each claim is checked against each source by a source-appropriate routine. Triples are checked by entity linking followed by relation lookup, hierarchy and code existence by exact lookup, and passages and tables by a natural-language-inference judge restricted to retrieved spans. Each check returns one of three verdicts with a confidence. In the fourth stage, a small aggregator turns the verdict matrix into a single score. The aggregator is logistic regression on features: the counts of support, contradiction, and silence per source type, the minimum claim support, and whether any source contradicts.
The aggregator is trained on a split kept separate from the conformal calibration split, so the guarantee stays valid. Conformal risk control then sets the answer threshold.
Beyond the method, the analysis asks three questions. Does silence predict error differently from contradiction? Which source type contributes most per domain? Does the score degrade when the base model shrinks from roughly 70B to 8B parameters, the scale at which AtomR reports that planning-based decomposition breaks down? Our claim checks are deliberately shallow, so the method should survive at small scale. If it does not, that is a reportable limit.
Baselines
The naive reference point is to answer every question. Its error rate is simply the base model's accuracy gap, and on the pharmacist exam the bar to clear is the 61.8% accuracy reported for RoBERTa-KMQA (KMQA). A second reference is conformal abstention driven by the model's own token-level confidence.
The self-knowledge baseline scores each answer by agreement across sampled answers and indirect re-questions, following evidence that fabricating models vary across related questions (Agrawal et al.). It is the strongest competitor that needs no knowledge base, so it is the one the proposal must beat to justify any external sources.
The strongest existing method is KnowHalu-style text-plus-triples checking (KnowHalu), re-implemented over the same domain sources. Its output probability is used as the conformal score. Its reported result is more than 15% accuracy improvement over prior baselines on HaluEval QA and more than 6% on summarization. No reported result was found on Lacuna for KnowHalu on any expert-domain benchmark, so we will produce those numbers ourselves. We also include TRAC-style conformal sets over single-source retrieval and generation (TRAC), measured by average set size at matched coverage.
Two ablations isolate the key ingredient. The first, "silence-as-contradiction," collapses the three verdicts to two. The second, "pooled judge," gives one LLM judge all evidence from all sources in a single prompt instead of using per-source verdicts.
Fairness rules apply to every comparison. All methods use the same base model at two scales, the same retrieved evidence, the same calibration and test splits, the same budget of model calls per question (self-consistency gets as many samples as our method makes verification calls), and the same hyperparameter search budget for any aggregator. Beating a baseline means a higher fraction of questions answered at $\alpha \in \{0.05, 0.1, 0.2\}$, with the realized test error at or below $\alpha$, over five random calibration splits with non-overlapping bootstrap intervals.
Why not just…?
Why not just use self-consistency, which needs no knowledge sources? This is the most serious threat, and the experiment is built to settle it. Our prediction is domain-dependent. On open-domain HaluEval, self-consistency may match us. On exam and clinical questions, models that lack the knowledge tend to repeat the same wrong answer, as the uniform sub-50% F1 failure of 20 models on AppealCase and the below-15% open-model scores on TheoremQA suggest. If self-consistency is not beaten in the expert domains, the central claim is false and we will say so.
Why not just put all the evidence in one prompt and ask a strong judge? This is the pooled-judge ablation. Prior evidence predicts it will underperform. HaluEval shows long deliberation hurts QA recognition, KnowHalu documents "lazy" judges that stop verifying early, and G-SAP finds that adding more graph context beyond a short prompt degrades performance. The pooled judge also cannot distinguish "no source mentions this" from "a source denies this." If pooling matches per-source verdicts, the three-state design is unnecessary, and we will drop it in favor of the simpler system.
Why not just fine-tune the model on the domain sources? Fine-tuning could raise base accuracy, but it does not produce an error guarantee. It also has a mixed record: fine-tuned GPT-3.5 underperformed prompting on guideline questions (Chen et al.), whereas LoRA tuning helped in cloud incidents (Ahmed et al.). The two approaches are complementary, so we will run our filter on top of a LoRA-tuned base. If the filter still adds answered coverage at fixed $\alpha$, the methods stack. If tuning alone closes the gap, that is a useful negative result.
Proposed Experiments
The primary testbed is the Chinese pharmacist exam (NLPEC) with CMeKG triples and textbook passages from KMQA, posed generatively, with answer correctness taken from the answer key. The second testbed is NLI4CT clinical trial inference. There the sources are the report's eligibility, intervention, and results sections, treated as separate text and table sources, plus UMLS. Reference numbers from Saama Technologies are F1 0.659 for the submitted configuration and best faithfulness 0.781. The third testbed is MIMIC-III code assignment with justifications, using ICD descriptions, UMLS synonyms, and the code tree from GKI-ICD. Fabricated or wrong codes count as errors, and rare codes (where GKI-ICD's F1 is only 0.132) are reported separately. HaluEval QA serves as the open-domain control. The 34-question AAOS guideline set is too small for calibration and is used only for qualitative error analysis.
The headline metric is coverage, the fraction of questions answered, at guaranteed $\alpha$. Secondary metrics are realized error, area under the risk–coverage curve, and the share of abstentions triggered by silence versus contradiction. Two hundred answers per domain will be annotated at the claim level to check verdict accuracy directly.
Three checks are decisive. The first is whether the coverage gain over self-consistency is larger in the three expert domains than on HaluEval. The second is whether the silence-as-contradiction ablation loses coverage specifically on the questions where KMQA-style source gaps occur. The third is whether the advantage holds at 8B scale. If all three hold, the result supports the thesis that, in expert domains, heterogeneous knowledge is more reliable for deciding when to stay silent than for generating answers.
References
- TheoremQA: A Theorem-driven Question Answering Dataset
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
- KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checking
- MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations
- TRAC: Trustworthy Retrieval Augmented Chatbot
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- A General Knowledge Injection Framework for ICD Coding
- Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
- AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning
- Do Language Models Know When They're Hallucinating References?
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
- G-SAP: Graph-based Structure-Aware Prompt Learning over Heterogeneous Knowledge for Commonsense Reasoning
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
- Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
- Saama Technologies at SemEval-2024 Task 2: Three-module System for NLI4CT Enhanced by LLM-generated Intermediate Labels