Biobank-Labeled Fragility of Published Gene–Environment Claims

Thousands of papers report that a genetic score matters more in some environments than others. Examples include a depression risk score that explains more variance after childhood trauma, or a dyslexia score whose effect on reading depends on family income. Many of these environments, such as income, deprivation, and trauma, are laid out geographically. Genetic ancestry is too, even inside populations that look homogeneous. A reported "gene × environment" interaction can therefore be partly an artifact of where participants were born. No literature tool can tell a reader which published claims are exposed to this problem. We propose to train one. The labels come not from a curated database but from a biobank. For every published interaction claim that can be re-estimated in UK Biobank, we compute how much the interaction shrinks when fine-scale geographic ancestry is properly adjusted. That shrinkage becomes an automatically generated training label. We then learn to predict it from the paper's text: what was measured, in whom, and how ancestry was controlled.

The approach is distant supervision, meaning training labels produced by aligning text with an external source rather than by human annotation. Here the external source is data, not facts. The resulting model reads a gene–environment paper and flags whether its headline interaction is likely to survive stratification-robust re-analysis. The flag is most useful for the many claims whose exposures or phenotypes a biobank cannot re-test. Applied across the literature, this produces an evidence map of gene–environment findings. Each claim carries a stratification-fragility score and the cohort context it was drawn from.

Core Claim

The claim is that stratification fragility of a gene–environment interaction is partly predictable from text. Fragility depends on two things. One is properties of the triad (which genetic score, which exposure, which outcome), such as how geographically patterned the exposure is. The other is properties of the study (ancestry composition, number and source of principal components, use of local versus external reference projections). Both are stated in papers. A biobank can supply ground truth for a large subset of claims. That makes it possible to learn this mapping and to evaluate it inductively on triads never seen in training.

This differs sharply from existing distantly supervised biomedical extraction. ChemDisGene aligns the Comparative Toxicogenomics Database with PubMed abstracts. In doing so it explicitly breaks nested statements into binary chemical–gene, gene–disease, and chemical–disease pairs, which destroys the conditional structure an interaction claim has. MedDistant19 aligns MEDLINE with SNOMED CT and is likewise pairwise. Both take their labels from a curated knowledge base, so a label means "the database asserts this." In our design a label means "this claim holds up, or does not, when re-estimated with better ancestry control." The closest literature-synthesis system, the EpiCausalX agent, builds causal diagrams from retrieved papers and computes adjustment sets. It never checks a claim against individual-level data, and it does not reason about genetic population structure.

Why this is the right intersection

The motivating evidence comes from population genetics. Apparent latent structure within the UK Biobank sample showed that polygenic scores for BMI and educational attainment stay associated with birth location even after adjusting for 40 principal components and recruitment center. Principal components here are the top axes of genome-wide variation, used as ancestry covariates. The same paper showed that the association between a BMI polygenic score and household income attenuates by over 30% once non-linear geographic terms are added. Income is exactly the kind of variable that serves as the "environment" in interaction studies.

Population stratification in GWAS meta-analysis should be standardized showed that cohort-local principal components in ALSPAC explain 1% of educational-attainment variance, against 7% for components projected from UK Biobank. Local correction under-adjusts. Together these results imply that the adjustment choices written into a paper's methods section carry real information about bias.

Meanwhile, headline interaction findings use precisely these geographically patterned exposures. Examples are the socioeconomic moderation of a dyslexia polygenic index and the more-than-threefold increase in depression variance explained by genetic liability under trauma or deprivation in a UK Biobank phenome-wide study. We do not claim these findings are artifacts. We claim nobody currently has a scalable way to say which ones are at risk.

Research Program

The first component is a schema and extractor for triadic claims. Each record holds the genetic term (variant, gene, or polygenic score and its trait of origin), the exposure, the outcome, the reported interaction estimate and its scale, and the stratification context. That context covers the ancestry composition of the cohort, the number of principal components, whether they were local or externally projected, use of mixed models, and recruitment geography.

Extraction follows the decoupled design that Affinage found necessary for faithfulness. It reached 97.95% literature-supported claims with a 0.029% contradiction rate by separating evidence extraction from synthesis. We depart from Affinage in two ways. First, it deliberately discarded phenotypic and environmental associations, and those are precisely our target. Second, we must read full-text methods, because adjustment details rarely appear in abstracts. Affinage reports that full text added 16% more findings at 15× the cost. We accept that cost only for the methods and results sections.

The second component is the biobank labeler. For each extracted claim whose genetic score, exposure, and outcome can be constructed in UK Biobank, we fit the standard interaction model. In that model the outcome depends on the genetic score, the exposure, and their product, and the product's coefficient $\beta_3$ is the interaction. We fit it along a ladder of increasingly thorough adjustment. The rungs are no ancestry control, 10 local principal components, 40 components plus center, 200 externally referenced components, and finally 40 components plus a smooth function of birth coordinates. The external components follow the pcapred approach, and the birth-coordinate term follows the non-linear geographic adjustment of the latent-structure paper. The label is the relative attenuation

$$\phi = 1 - \frac{\hat\beta_3^{\text{geo}}}{\hat\beta_3^{\text{40PC}}},$$

where $\hat\beta_3^{\text{40PC}}$ is the interaction estimated under conventional 40-component adjustment and $\hat\beta_3^{\text{geo}}$ is the estimate after adding the geographic smooth. A claim counts as fragile when $\phi$ exceeds a pre-registered threshold and its bootstrap interval excludes zero attenuation.

Because the label compares two adjustments within the same biobank, it isolates stratification sensitivity. It is separate from the original study's power or measurement choices. We also record, as a secondary label, whether the paper's own adjustment rung reproduces its reported sign.

The third component is the predictor. It is a domain encoder (PubMedBERT, as used in both MedDistant19 and ChemDisGene) over the claim's methods and results passages, combined with the structured stratification fields. It is trained to predict $\phi$ and the fragile/robust label.

Following the leakage lesson of MedDistant19, all headline results use inductive splits by exposure family and outcome family. In that paper, filtering overlapping triples dropped a model's AUC from 82.6 to 11.8. Under our splits, a model cannot succeed by memorizing that "income triads are fragile."

Baselines

The naive reference point treats every published claim as robust, or equivalently predicts the training-set base rate of fragility. Any learned model must beat this on area under the ROC curve (AUC) and on calibration. If fragility turns out to be rare, this baseline will look deceptively strong on accuracy, which is why we report AUC and precision at a fixed recall.

The strongest simple competitor is a methods-checklist rule applied to the same extracted fields. It flags a claim as fragile if the cohort used no principal components, used fewer than 10, used only local components, or pooled ancestries without stratified analysis. The prior evidence on local versus global components suggests this rule captures real signal. It shares our extractor, so the comparison isolates whether learning adds anything beyond encoding current best-practice guidance.

The strongest existing system is an EpiCausalX-style agent (An AI Agent for Automated Causal Inference in Epidemiology) run on the same papers with the same underlying LLM and a matched token budget. It is prompted to build the causal diagram for each claim and to report whether ancestry or geography belongs in the minimal adjustment set but was omitted. That omission is taken as its fragility prediction. It is the strongest competitor because it already reasons explicitly about confounders from text. No reported result found on Lacuna for any system predicting stratification fragility of gene–environment claims. For the extraction stage the nearest reported figures are pairwise. PubMedBERT with BRAN reaches micro F1 of 43.8 on ChemDisGene's curated test set, and PubMedBERT reaches AUC 0.80 and F1 0.76 on MedDistant19. We use these only to confirm our extractor is competitive on binary relations before trusting it on triads.

Two ablations isolate the key ingredient. One trains the same predictor on labels computed only up to the 40-component rung, which removes the fine-scale geographic signal. The other drops the triad-level exposure text and keeps only study-level adjustment fields. Beating the checklist rule and the agent by a clear AUC margin on inductive splits, with both ablations degrading, would count as success. We would set the margin in advance, for example 0.05 with non-overlapping bootstrap intervals.

Why not just…?

Why not just re-estimate every claim in the biobank? Where possible we do; that is the label source. The model is for the rest. These are claims whose exposures (specific drugs, parenting measures, social-media behavior as in the serotonin transporter–maternal care study) or non-European cohorts cannot be reconstructed in UK Biobank. This alternative is a real threat. If most claims of interest turn out to be re-testable, a predictor adds little. Our first decisive check therefore measures the re-testable fraction of the extracted corpus. We will report the result even if it undercuts the transfer motivation.

Why not just prompt a frontier LLM to judge stratification risk from the paper? Such a model plausibly "knows" that income is geographically patterned. MedDistant19 found that pretrained models act as soft knowledge bases that degrade on unseen entity pairs, and Affinage found that end-to-end generation without decoupled extraction hallucinated substantially. Neither result addresses this exact task, so we run the zero-shot judge as an additional baseline on identical inductive splits. If it matches the trained predictor, the honest conclusion is that the biobank labels are best used as an evaluation set for LLM judges, not as training data.

Why not just require 40 principal components and move on? The latent-structure paper shows associations with birth location survive 40 components and center, and that non-linear geographic terms change a polygenic-score–income association by over 30%. Our label ladder measures directly how often this happens for interaction terms specifically. The resulting distribution of $\phi$ at the 40-component rung is itself a finding, whichever way it comes out.

Proposed Experiments

The corpus is gene–environment papers from PubMed with polygenic-score, candidate-variant, and copy-number genetic terms. Labels come from UK Biobank. A transfer check uses ALSPAC, recomputing the ladder with local versus pcapred-projected components, where the reference paper documents a 1% versus 7% gap in explained educational-attainment variance.

Three checks decide the proposal. First, label validity. Bootstrap stability of $\phi$ must be high. A negative control, in which the exposure is permuted within birth-region strata, must yield near-zero fragility. Well-replicated interactions with non-geographic exposures, such as clinical stage in the kind of survival analysis done in structured gene–environment interaction analysis, should score as robust. Second, predictive value. On held-out exposure and outcome families, the trained model must beat the checklist rule, the causal-diagram agent, and the zero-shot judge on AUC. Third, the random-versus-inductive split gap. If performance collapses under inductive splits as it did on MedDistant19, the model has only memorized triad identities, and we would report the approach as a labeling resource rather than a predictor.

The final deliverable is the scored evidence map. For every extracted claim it records its stratification context, its biobank fragility where measurable, and its predicted fragility otherwise. Living-review systems such as the social-science meta-review interface could weight evidence by this map instead of pooling it additively.