When the Data Disagree with the Textbook: Testing Whether LLM Analysts Can See Gene-Environment Interactions

Language models are now being asked to act as junior data analysts in genomics. They read cohort tables, choose statistical models, run code, and write conclusions. Much of cancer genetics rests on one kind of finding: an inherited variant changes how much harm an exposure does. An alcohol-metabolism allele, for example, makes drinking and smoking far more dangerous for the esophagus. These findings are called gene-environment interactions. Detecting one requires asking whether an effect differs across subgroups, not just whether an effect exists. Some of the field's most important results also run against intuition. In breast cancer, higher inherited polygenic risk goes with longer survival. This proposal asks a pointed question. When the numbers in front of a model contradict what it absorbed from biology papers during pretraining, does it report the numbers or the prior?

We hypothesize a specific failure, which we call prior anchoring. It resembles the known tendency of models to copy flawed reasoning already sitting in their context. The difference is that the anchor here is the model's own background knowledge of the named genes and exposures. We predict three things. First, models will under-detect or reverse-sign interactions that contradict textbook direction, compared with statistically identical interactions that match it. Second, giving the model a Python interpreter will not remove the gap, because the error enters when the model decides which model to fit and how to read the result, not in the arithmetic. Third, a simple split of the analysis will close most of the gap. The model specifies and estimates the model on anonymized variables, and real names are revealed only for the final interpretation.

Core Claim

The claim is narrow and falsifiable. On cohorts where the true interaction structure is known, an LLM analysis agent's accuracy at recovering interaction direction and size will depend on whether that structure agrees with biological priors, even at matched statistical power. Blinding the agent to variable meaning during model specification and estimation will remove most of this dependence. Blinding should cost little on prior-congruent cases.

We do not claim that LLMs estimate interactions better than regression. They should not, and our design treats a fixed regression pipeline as the reference that sets the ceiling. The point is to measure, and then reduce, a failure that arises because these agents are being deployed with names attached.

How This Relates to Prior Work

Four lines of prior work bear on the claim, in different roles.

Two papers motivate the claim by showing that context steers reasoning. Contextual Drag shows that placing incorrect draft solutions in context causes 10–20% accuracy drops on AIME, GPQA and Game of 24, with smaller models losing almost half their accuracy. Models keep replicating the flawed reasoning structure even when told the draft is wrong. Its evidence comes entirely from math, QA and puzzles, and the anchor is explicit text. We test whether named biological variables act as an implicit anchor of the same kind in a scientific task. ConsisEval shows that models solving hard problems fail strictly easier versions, often because of redundant information. We borrow its paired-difficulty design to build matched easy and hard interaction tasks.

The genomics agent literature defines the application regime. GenoMAS is a six-agent, code-executing system evaluated on GenoTEX. It reaches 60.48% F1 for gene identification and 98.78% executable code. It also reports a collapse to about 32% accuracy on clinical metadata, and GenoTEX notes that extra review rounds became unstable. Both systems address bulk gene-trait association with code. Neither tests effect modification or separates arithmetic errors from specification and interpretation errors, which is exactly the separation we make. GeneGPT shows that in-context demonstrations teach biomedical API use better than documentation. That work covers lookup rather than inference.

Gene-environment epidemiology supplies the ground truth and two constraints. The effect structures we plant come from published findings. These include alcohol-metabolism alleles combined with smoking for esophageal cancer (ADH1B/ALDH2) and a 15q25.1 deletion that roughly quadruples nasopharyngeal carcinoma risk under smoking (15q25.1). They also include the counterintuitive protective association of breast-cancer polygenic risk with survival, HR 0.89 (PRS and survival). The last is the germline/somatic "see-saw", in which high inherited burden goes with earlier onset and fewer somatic mutations (germline burden). The constraints come from Isgut et al.. They found that a neural network on UK Biobank breast cancer data matched logistic regression at about 0.63 AUC overall, while recalling 10–18% of cases at a high-risk threshold of 0.8 versus under 1% for regression. That result rules out promising better global discrimination. It also tells us interactions show up in tails and subgroups, so our metrics target those.

Context is Environment supplies a lens rather than a method. It treats in-context examples as a description of the environment a model is operating in, reaching 92.0% versus 68.6% for standard training on Camelyon17. Its authors warn that the model can latch onto spurious environment cues instead. Our blinding intervention is a direct response to that warning in a language setting. Variable names are a cue that can override the rows.

Research Program

The core artifact is a benchmark of synthetic and semi-synthetic cohorts with planted interactions. Each cohort is generated from a structural template taken from the findings above. Templates include a variant multiplying exposure risk, a variant protecting under exposure, a risk score with an inverted survival association, and a burden variable that shifts age of onset. Each template is rendered in two "worlds" with identical statistics. In the congruent world, variable names and effect directions match the literature. In the incongruent world, the named variant's effect is reversed. Examples are a deletion that protects smokers, or an ALDH2 allele that lowers risk among drinkers.

Sample size, effect size, censoring and number of distractor covariates are varied on a grid. This lets us compare congruent and incongruent accuracy at matched power, computed from the generating model. Each item also has a ConsisEval-style easy twin, such as a stratified main effect, so we can measure whether a model that finds a three-way interaction misses the simpler fact underneath it.

Tasks require the full analysis loop. The model receives a cohort description and either inline rows (small cohorts) or a file plus an interpreter (large cohorts). It must decide whether an interaction exists, report its direction and approximate size, and write a one-paragraph conclusion. We log the fitted model specification separately from the final claim. This lets us attribute each error to one of three stages: specification (wrong or missing interaction term, inappropriate stratification), estimation (bad code or arithmetic), or interpretation (correct numbers, wrong conclusion).

The intervention is a blind-then-name split, in the spirit of divide-and-conquer prompting, which bounds error accumulation by isolating sub-tasks in separate contexts. In the first context, variables are renamed (G1, E1, Z1…) and the model specifies and estimates the model. A second, fresh context receives the estimates and the real names and writes the interpretation. It is instructed to report the estimate even when it conflicts with known biology, and to flag the conflict rather than resolve it.

A secondary mechanistic probe asks whether "test for effect modification" behaves like a transferable task vector. A task vector is the compressed task representation that forms at the end of a few-shot prompt and can be patched into another forward pass. The original paper reports that patched task vectors recover 80–90% of in-context accuracy on single-token tasks. It leaves multi-step reasoning open. We would patch vectors from anonymized runs into named runs on open-weight models and measure whether this moves specification choices. A null result here is informative and does not affect the main claim.

Baselines

The naive reference point is a fixed statistical pipeline with no language model. It fits a logistic or Cox regression with all pairwise gene-by-exposure terms and false discovery rate control. It sees exactly the same data and cannot be influenced by names. It is the ceiling for estimation and the floor any agent should not fall below. For context, the analogous linear model in Isgut et al. reached about 0.63 AUC on UK Biobank breast cancer. An agent "beats" this baseline only on dimensions it does not cover. These include correct specification when distractors make all-pairs testing underpowered, and conclusions that correctly describe the effect.

The strongest existing competitor is a GenoMAS-style multi-agent code-executing system, run from the released GenoMAS codebase with its agents and review loop intact. On its native benchmark it reports 60.48% gene-identification F1 on GenoTEX, and GenoTEX reports AUROC 0.74 for the earlier GenoAgent with o1 against 0.89 for human experts. We found no reported result on Lacuna for any LLM agent on gene-environment interaction detection, so all numbers on our benchmark will be new. For fairness, GenoMAS and our method use the same base model and the same interpreter. They also share a matched token budget, with blinding counting its two contexts against the budget, and prompts tuned on the same held-out development templates.

A plain chain-of-thought baseline and a single-agent code baseline isolate the value of tools. Our intervention beats the strongest baseline if it cuts the congruent-incongruent accuracy gap by at least half at matched budget, while losing no more than a few points of congruent accuracy. Two ablations isolate the key ingredient. Blinding without context separation renames variables but interprets in the same context. Separation without blinding uses two contexts with names visible throughout. If the gain comes only from separation, the mechanism is context hygiene rather than prior anchoring, and we will say so.

Why not just…?

Why not just run the regression and skip the LLM? For estimation alone this is correct, and our naive baseline is exactly this pipeline. But labs are already handing whole analyses to agents like GenoMAS, whose job includes choosing the model and writing the conclusion. The benchmark measures whether that handoff is safe, and the stage-wise error attribution shows whether failures sit in steps a fixed pipeline does not automate.

Why not just tell the model to trust the data over its priors? We run this as a baseline. Contextual Drag is direct evidence against it: models stayed anchored to flawed reasoning even when explicitly told it was wrong. If instruction alone closes the gap, the blinding step is unnecessary. That outcome would still be useful, and the experiment settles it directly.

Why not just give the model a code interpreter? Tools fix arithmetic, as Multi-TAG and ChatCoT show on math. Our hypothesis is that anchoring acts on which interaction terms get fitted and how estimates get described. The single-agent code baseline tests this. If the gap disappears with tools, our central hypothesis is wrong, and we will report it as such.

Why not just fine-tune on oncology data? Domain fine-tuning such as Woollie, at 0.90 AUROC on progression from radiology notes, strengthens exactly the priors we suspect cause the failure. It is a genuine threat, though, if fine-tuned models also read data more carefully. Where open weights allow, we will include one fine-tuned biomedical model to test whether domain training widens or narrows the gap.

Proposed Experiments

The decisive experiment has two stages. We first compute the prior-congruence gap, congruent minus incongruent accuracy on interaction sign and on detection at matched power, for each base model under chain-of-thought, code, and multi-agent baselines. We then rerun with blinding.

The claim survives only if three things hold together. The gap is clearly positive under tool use. Stage-wise attribution places most incongruent errors in specification or interpretation. Blinding shrinks the gap substantially while the fixed regression pipeline shows no gap, which confirms the worlds are statistically matched. A secondary analysis uses the easy twins to test whether incongruent items also produce the hard-to-easy inconsistency ConsisEval describes.

We will repeat the core comparison on the inverted polygenic-risk survival template, under right censoring. This is where intuition and data disagree in the real literature, not just in our counterfactual worlds. We will also report how often each agent's written conclusion contradicts its own fitted estimate. This measures the failure that matters most if these systems are ever trusted with real cohorts.