Catching Additive Guesses: Factorial Probing for Black-Box Uncertainty on Gene-Environment Risk

Language models behind commercial APIs are increasingly asked clinical questions of the form "what is this patient's risk?" Many of these questions involve gene-environment synergy. That means a genetic variant and an exposure, such as smoking or hypertension, raise risk together far more than the sum of their separate effects. The usual black-box ways of checking a model's confidence ask the same question many times and measure how much the answers disagree. They have a known blind spot: a model that is consistently wrong looks confident. We think gene-environment synergy is where this blind spot does the most damage. A model that has learned to add risk factors will add them every time, and its answers will agree with each other perfectly.

We propose to take uncertainty from a structured family of related questions rather than from repeated samples of one question. The four questions cover the combined profile, the gene alone, the exposure alone, and neither. From the model's four answers we compute the interaction it implicitly believes in. We then score how stable and how self-consistent that implied interaction is. The claim is that this "factorial" score detects confident additive errors that sample-consistency methods miss, at a matched query budget. It also gives prediction sets whose coverage holds inside the high-risk synergy strata, not just on average.

Core Claim

Suppose a model is asked for four risks: $p_{11}$ for carrier plus exposed, $p_{10}$ for carrier unexposed, $p_{01}$ for non-carrier exposed, and $p_{00}$ for neither. Its implied additive interaction is the relative excess risk due to interaction, the standard epidemiological contrast for departure from additivity:

$$\widehat{\mathrm{RERI}} = \frac{p_{11} - p_{10} - p_{01} + p_{00}}{p_{00}}.$$

A model that reasons additively produces $\widehat{\mathrm{RERI}} \approx 0$ reliably. Its semantic entropy on the combined question alone is then near zero, even when the true interaction is large.

Our uncertainty score has three parts. The first is the dispersion of $\widehat{\mathrm{RERI}}$ across a few samples of each cell. The second is disagreement between the model's direct answer for $p_{11}$ and the answer it reconstructs when asked for relative risks. The third is mismatch between what the model says ("these factors act synergistically") and what its numbers imply. Each part tests whether the model's answers hang together across the factorial structure.

We predict two things. First, large errors in the combined-profile risk will concentrate where this score is high. Second, this holds in cases where discrete semantic entropy stays low.

Why This Gap, and What It Builds On

Three prior results motivate the proposal. Hallucination filtering in radiology vision-language models using discrete semantic entropy clusters 15 sampled answers by meaning and filters on entropy. It raised GPT-4o accuracy from 51.7% to 76.3% while keeping only 47.3% of questions, and it states plainly that the method measures consistency, not truth. On the gene-environment side, Associations of risk factor burden and genetic predisposition with the 10-year risk of atrial fibrillation reports significant additive-scale synergy between a 165-variant polygenic score and lifestyle burden, alongside suboptimal calibration. It also publishes stratum risks, such as 11.10% in the highest-risk group at age 65. MetaGeno reports a nearly five-fold stroke-risk increase when high genetic risk combines with hypertension and atrial fibrillation.

Together these supply the target, stratum risks with large interactions, and the suspected failure: consistent errors that sample-agreement checks cannot see.

Other work supplies the machinery we reuse or compare against. Kernel Language Entropy replaces hard clusters with a continuous similarity kernel and von Neumann entropy. API Is Enough builds logit-free conformal prediction sets from response frequency and semantic similarity. Predicting the Performance of Black-box Language Models with Follow-up Queries trains a linear probe on answers to follow-up questions. All three operate on a single prompt and its samples or follow-ups. None exploits the counterfactual structure that the question itself carries, and that structure is what we add.

The closest clinical precedent is Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models. It serializes electronic health records into text and measures class entropy for GPT-3.5 and GPT-4. It finds that asking for several related outcomes at once reduces uncertainty. We ask related counterfactual profiles instead of related outcomes. We also treat their joint coherence as the signal, rather than as a regularizer. That paper contains no genetic data.

Research Program

The first component is a benchmark with known interactions. It has three tiers, each serving a different purpose.

The literature tier turns published stratum-specific risks into factorial question sets. Its sources are the atrial fibrillation polygenic-score-by-lifestyle strata, the MetaGeno stroke strata, and the gestational diabetes genotype-by-BMI and fasting-glucose nomogram in Causal inference and risk prediction of gestational diabetes mellitus. That paper reports an rs6127416 AA odds ratio of 2.35 and a nomogram AUC of 0.794 in validation. The tier also includes case-control synergy reports, such as VEGF alleles with smoking in Interaction Between Vascular Endothelial Growth Factor Gene Polymorphism and Smoking on Gastric Cancer Risk and 15q25.1 with tobacco in The SNP rs931794 in 15q25.1 Is Associated with Lung Cancer Risk.

The cohort tier, if we obtain UK Biobank access, fits interaction models directly. This yields thousands of patient-level profiles with reference risks. It is the scale test.

The synthetic tier uses fictitious variant names with planted interactions described in an in-prompt evidence summary. The planted interactions range from strongly sub-additive to strongly super-additive. This tier separates memorized epidemiology from reasoning: an additive default cannot be excused as recall.

The second component is the factorial probe and its scoring. For every profile we issue the four cell prompts, with a few samples per cell, plus one relative-risk framing and one verbal synergy question. We extract numeric risks and form the three-part score defined above. We combine the parts with a small logistic model trained on a held-out calibration split. The combination model deliberately stays low-capacity, so that it cannot memorize dataset identity.

The third component is stratum-conditional conformal prediction. We build intervals around the model's combined-profile risk using the factorial score as the nonconformity measure, which is a measure of how unusual an answer is relative to the calibration data. We calibrate separately within each gene-by-exposure stratum. We then test whether coverage in the high-risk synergy cell matches the target level. Gap 3 of the research file raises the possibility that synergy breaks exchangeability, the assumption that calibration and test cases are interchangeable; if it does, this stratified check is where the failure should show.

Baselines

The naive reference point is the model's unfiltered combined-profile answer, with the model's own stated confidence as the ranking score. This shows what an unaudited deployment gets.

The first sample-consistency baseline is discrete semantic entropy as run in the radiology paper: 15 samples at temperature 1.0 with entailment clustering. That paper reports 76.3% accuracy at 47.3% coverage for GPT-4o on VQA-Med 2019 plus RadDataset. We found no reported result on Lacuna for any gene-environment risk task. The second is Kernel Language Entropy over the same 15 samples. The research file reports its gains over semantic entropy only in figures, with no scalar quoted. Class entropy follows the clinical-outcomes paper.

The strongest competitor on selective prediction is the follow-up-query probe. It reports AUROC 0.9596 on Natural Questions for LLaMA3-3B, and it is the most flexible black-box learned detector. We will train it on exactly our calibration split with the same logistic-model capacity. We will also cut it to a subset of queries that matches our budget. For intervals, the strongest competitor is LofreeCP, which reports a 7.7% reduction in average set size over logit-based conformal prediction on TriviaQA at error rate 0.25. We run it both marginally and with the same per-stratum calibration we use, so that stratification alone cannot explain any gain.

Every method gets the same models (GPT-4o-class and one open model), the same profiles, and the same total budget of about 16 generations per profile. For our method that is four cells times three samples plus the framing and verbal queries. The key ablation keeps that budget but replaces the four factorial cells with four paraphrases of the combined question. This isolates counterfactual structure from simply spending more queries. A second ablation drops each of the three score parts in turn.

Beating the baselines means two things. The first is higher AUROC for flagging answers whose combined-profile risk error exceeds a clinically meaningful threshold, specifically in the super-additive synergy strata. The second is target-level coverage in the high-risk cell at no larger interval width than LofreeCP.

Why not just…?

Why not just fit a logistic regression with a gene-by-environment term and skip the language model? When structured cohort data exist, that regression is the right risk model. We use it as the source of reference risks in the cohort tier. The proposal does not try to out-predict it. It audits language models that clinicians and patients already query in free text, often without any cohort. The regression cannot tell you when GPT-4o's narrative answer about a smoking NOTCH3 carrier is wrong. The synthetic tier makes this concrete: there is no data to fit, only a model's reasoning to check.

Why not just use semantic entropy or Kernel Language Entropy? The radiology paper itself concedes that consistent errors pass its filter. Kernel entropy refines similarity between samples but inherits the same dependence on disagreement. Our experiment settles this directly. If semantic entropy's AUROC in the super-additive strata matches the factorial score's, the core claim fails and we will report that.

Why not just ask the model whether the factors interact, or train a follow-up-query probe? This is the real threat. The follow-up-query probe beat white-box probing on Natural Questions, and a verbal synergy question is already one of our score parts. The budget-matched comparison against the probe, together with dropping the verbal part, decides whether the numeric factorial contrast adds anything. If the probe alone matches us, the contribution shrinks to the benchmark and the finding about the failure mode.

Decisive Checks

The first check is whether additive defaulting exists at all. Across literature and synthetic profiles with large true interaction, does the implied $\widehat{\mathrm{RERI}}$ cluster near zero while semantic entropy on the combined prompt stays below the 0.3 filter threshold? If models do not default to additivity, the motivation collapses early and cheaply.

The second check is the budget-matched paraphrase ablation. If factorial cells beat paraphrases at equal query cost, the structure is doing the work.

The third check is stratum-conditional coverage in the high-risk cell under the cohort tier. We run it with and without per-stratum calibration, which tests whether synergy breaks the exchangeability that logit-free conformal prediction relies on.

We report all results separately for super-additive, additive, and sub-additive strata, since averaging across them would hide exactly the effect under study.