Ask the Expert Only About the Premise: Calibrated Premise Confirmation for Domain LLMs

When a language model reasons through an expert task, such as deciding whether an appeals court will overturn a civil ruling, which theorem solves a physics problem, or what attack is unfolding on a network, it usually commits early to one hidden intermediate fact. That fact might be the procedural error in the ruling, the governing theorem, or the attack class. If the commitment is wrong, everything downstream is a fluent, confident fabrication built on a false premise. This is how hallucination compounds during generation (CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools). In expert domains, the wrong premise is usually exactly the piece of latent domain knowledge the model lacks. The human expert sitting next to the system usually does have that knowledge and can confirm it in seconds. The expert cannot cheaply audit a page of generated reasoning.

We propose a system that does three things. It builds a calibrated shortlist of candidate premises with a statistical coverage guarantee. It proceeds automatically when the shortlist has one entry. Otherwise it asks the expert one multiple-choice question ("which of these is it?") before generating anything downstream. The approach treats the expert as the source of the missing domain knowledge and treats the hallucination as a premise error. The claim to test is that a small number of targeted, calibrated questions recovers most of the accuracy of a fully human-guided pipeline at a fraction of the expert's time. It should also beat showing experts uncertainty over the final answer.

Core Claim

Prior work already uses conformal prediction to put uncertainty on LLM outputs. Conformal prediction is a method that turns any confidence score into a set of candidates guaranteed to contain the truth with a chosen probability, such as 90%, assuming test cases resemble calibration cases. Existing methods place these sets on the final answer. TRAC wraps open-domain retrieval-augmented question answering in conformal sets. Prune 'n Predict prunes multiple-choice options to a conformal set and re-asks the model. Conformal Prediction Sets Improve Human Decision Making shows lay crowdworkers such sets for classification.

We instead place the set on the intermediate premise $z$ and use the expert to collapse it. Write $C_\alpha(x)$ for the conformal premise set for case $x$, calibrated so that $P(z^\star \in C_\alpha(x)) \ge 1-\alpha$, where $z^\star$ is the true premise. If the expert reliably picks $z^\star$ whenever it is in the set, the final error rate obeys

$$ P(\text{final answer wrong}) \;\le\; \alpha + \varepsilon_{\text{human}} + \varepsilon_{\text{down}}, $$

where $\varepsilon_{\text{human}}$ is the expert's selection error and $\varepsilon_{\text{down}}$ is the model's error rate given the correct premise. The bound follows from a union bound over the three ways the pipeline can fail.

The inequality says the method is only worth anything when $\varepsilon_{\text{down}}$ is small, meaning the premise really is the bottleneck. That is directly testable, and one piece of prior work gives reason to doubt it in some domains. TheoremQA found that supplying theorem definitions in-context barely helped, because knowing a theorem is not the same as applying it. We treat this as a constraint. The first experiment measures the "oracle premise gain" in each domain. The method is pursued only where that gain is large, and a domain with near-zero gain is reported as a negative result rather than tuned around.

This design also addresses the "more is less" tension in the literature. Evaluating large language models on medical evidence summarization found that feeding models fuller context increased contradictions and certainty illusions. In contrast, the structured injection in KMQA was needed to pass a pharmacist exam. A single expert-confirmed premise is the most targeted context possible: one confirmed fact rather than more retrieved text.

Research Program

The work has three parts.

The first part defines and labels premises in three public benchmarks where an intermediate variable plausibly drives the answer. On AppealCase, the premise is the type of first-instance error (procedural, fact-finding, legal application, or none). We use its procedural-error annotations where available and expert-annotate a calibration subset otherwise. On TheoremQA, the premise is the governing theorem among its 350 theorems. On the cyber incident datasets used by Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination, the premise is the attack class or compromised asset that conditions the recovery plan. That paper names conformal abstention as unaddressed future work.

The second part builds the premise-set machinery. Premise scores come from the model's own probability over a discrete premise vocabulary. We also test a consistency-based score adapted from the indirect-query idea in Do Language Models Know When They're Hallucinating References?. That paper found that repeated indirect probing exposes fabrications better than asking the model directly. We calibrate split-conformal thresholds per domain. We check coverage under realistic shift, such as new case categories or new incident types, because exchangeability is the assumption most likely to break in deployment. The downstream generator is conditioned on the confirmed premise only. Full reasoning is generated after confirmation.

The third part is a human study. It moves the evidence from Conformal Prediction Sets Improve Human Decision Making, which was explicitly limited to laypeople, to practicing experts. Following the multivariate view of Human-machine symbiosis, we measure decision accuracy alongside time per case, workload, and sense of agency. That framework argues these dimensions trade off and are rarely measured together. Human-AI interaction in safety-critical network infrastructures argues the same for over-reliance under time pressure. The cloud incident study motivates the whole design. In that study, mitigation recommendations improved sharply when an engineer supplied the true root cause, and hallucinated root causes misdirected investigations. Our contribution is to make that confirmation cheap, selective, and statistically bounded.

Baselines

The naive reference point is the model answering zero-shot with no human involvement. The literature already establishes how poor this is. On AppealCase, all 20 mainstream models scored below 50% F1 on reversal prediction (AppealCase). On TheoremQA, GPT-4 with Program-of-Thought prompting reached 51.0% accuracy while open models scored below 15% (TheoremQA). We rerun this baseline with the same base models we use for our method rather than relying only on reported numbers.

The strongest automated competitor in the incident domain is the fine-tuned 14B planner with retrieval and lookahead from Incident Response Planning. It reports an 8.4% ineffective-action rate, a 3.0% failed-recovery rate, and up to 22% shorter recovery than frontier models. We run our premise confirmation on top of that same planner with the same data and model. Our method beats it only if it lowers failed or ineffective actions while asking the human on a minority of incidents.

The strongest uncertainty-based competitor is answer-level conformal assistance, in the style of TRAC and Prune 'n Predict. Here the expert is shown a calibrated set of final answers or plans at the same nominal coverage. TRAC reports that at 90% target coverage on Natural Questions its sets averaged 20.4 unique answers. That is the kind of set an expert would have to read. Prune 'n Predict reports accuracy gains of up to 7.24% over one-shot inference. We run both with identical calibration splits, base models, and $\alpha$. Our method wins if it matches their accuracy at lower expert time, or exceeds it at equal time.

The ablation that isolates the key ingredient is always-ask: the expert confirms the premise on every case, with no calibration. This is the ceiling on accuracy and the floor on efficiency. Our method should approach its accuracy while asking on far fewer cases. A second ablation is uncalibrated top-k premises, which shows the expert a fixed number of candidates. This separates the value of calibrated, variable-size sets from the value of asking at all, mirroring the comparison the lay-user trial made at the answer level.

Why not just…?

Why not just show the expert conformal sets over the final answer? This is the most serious alternative, because it already has a coverage guarantee and human evidence behind it. The weakness is that for generative outputs the candidates are long and numerous; TRAC's 20.4 unique answers at 90% coverage illustrate the burden. Premises live in a small discrete space, such as a few error types or a few attack classes. If answer-level sets on AppealCase reversal, which is binary, turn out as useful as premise sets, our advantage there disappears. For that reason the main comparison is on the incident-response plans and the TheoremQA solutions, where answers are free-form.

Why not just defer the whole case to the human when the model is uncertain? Whole-case deferral throws away the model's value on exactly the hard cases, and it gives the expert a full task rather than a one-click choice. We implement it as a selective-prediction baseline with an equal deferral budget. Our method should be more accurate at matched expert time. If it is not, premise confirmation adds no value beyond triage.

Why not just detect hallucinations automatically with self-consistency? Indirect-query consistency does flag fabrications (Do Language Models Know When They're Hallucinating References?), but detection does not supply the missing fact. A flagged case still needs someone who knows the answer, and the 44% of medical errors traced to facts absent from both text and pre-training (KMQA) cannot be repaired by the model re-querying itself. We use consistency as a candidate scoring function inside our sets, so the experiment directly tests whether it suffices alone.

Why not just fine-tune on domain data? Fine-tuning helps, as AFDBench's GRPO model and the cloud LoRA results show, but it does not remove residual premise errors. Prompt engineering in consistency and reliability found that fine-tuned GPT-3.5 sometimes did worse than prompting. Our method is applied on top of the strongest fine-tuned model in each domain, so the two are complementary rather than competing.

Proposed Experiments

The first decisive check is the oracle premise gain: downstream accuracy when the model is given the true premise versus none, in each of the three domains. TheoremQA's weak theorem-augmentation result predicts a small gain there. If that holds, TheoremQA becomes the negative control that shows the method's scope. AppealCase and incident response are predicted to show large gains. If neither does, the core claim fails.

The second check is offline coverage and efficiency with simulated experts. We treat the expert as a selector that picks the true premise when it is in the set, with injected selection-error rates. We report empirical premise coverage against the nominal $1-\alpha$, the fraction of cases requiring a question, the mean set size, and final accuracy, F1, or failed-recovery rate against all baselines above. We repeat this under category or incident-type shift to test where coverage breaks.

The third check is a within-subject study with domain professionals: legal practitioners on AppealCase cases and security analysts on incident scenarios. They are randomized across no assistance, answer-level sets, uncalibrated top-k premises, always-ask, and calibrated premise confirmation. The primary outcome is accuracy at matched time. Secondary outcomes are time per case, self-reported workload, sense of agency, and how often experts accept a wrong singleton. That last measure directly tests the over-reliance concern from the safety-critical literature.

The proposal succeeds if premise confirmation reaches within a few points of always-ask accuracy while asking on a minority of cases, and beats answer-level sets on free-form domains at equal expert time. It fails, informatively, if oracle premise gains are small or if experts rubber-stamp singletons.