Much of what clinicians know about a patient's disease never becomes a structured field. It stays in the free text of notes: the cup-to-disc ratio an ophthalmologist wrote down, the highest eye pressure ever recorded, which eye progressed first. Genetic studies that rely on billing codes or crude averages lose this detail. That loss is one reason twin studies find a large inherited component that genome-wide association studies (GWAS, which test millions of common DNA variants one at a time against a trait) cannot account for, the so-called missing heritability. Large language models (LLMs) can now pull these values out of notes at scale. But a single automated extractor makes errors nobody checks. Systematic reviewers solved the same problem long ago with independent dual extraction: two people extract the same data without seeing each other's work, and an adjudicator resolves the disagreements. Their mistakes are assumed to be uncorrelated, so the errors that survive are rare. That assumption is what makes dual review worth its cost, and it is exactly what fails silently when both "reviewers" are LLMs trained on overlapping data.
This proposal turns the genetics into the test. If two extractors read the same notes independently, their disagreement on each patient is made only of their errors. If those errors are random noise, the disagreement should have no heritability: genotype should not predict it. If it does, the errors are systematic, for example tied to documentation habits that differ by clinic, ancestry, or socioeconomic group, and they will bias any genetic discovery built on them. This yields a check on extraction quality that needs no gold labels and can run on every patient in a genotyped cohort. It also answers the practical question of whether automated dual extraction actually recovers heritability that single-pass extraction loses.
Core Claim
Write extractor A's value for a trait as $A = T + e_A$ and extractor B's as $B = T + e_B$, where $T$ is the true value and $e_A, e_B$ are extraction errors. Their difference is
$$D = A - B = e_A - e_B,$$
which contains no true signal. The SNP-heritability of $D$ (the share of its variance explained by common genotyped variants), written $h^2_{\mathrm{SNP}}(D)$, is therefore a direct measure of how much genetic or ancestry-linked structure the extraction errors carry. We claim three things. First, $h^2_{\mathrm{SNP}}(D)$ near zero, together with a genetic correlation between $A$ and $B$ near one, is a label-free certificate that the two extractors' errors behave like independent noise. Second, extractors from different model families, run blind to each other, pass this test where two samples from the same model do not. Third, when the test passes, the reconciled phenotype has higher observed heritability and recovers more known loci than single-pass extraction at the same sample size. This is because random error dilutes heritability roughly in proportion to the trait's reliability, and dual review raises reliability.
None of the work in the research file does this. Auto Review built an automated second-stage check for extraction from phone calls. It compared a verifier that judges the first-stage value (high recall) with an independent re-extractor (high precision), and reported a 5.5% average F1 gain for the re-extraction route after speech-error correction. It evaluated against human gold labels in an administrative domain and never asked whether its two stages' errors were correlated. The human dual-review protocols in the file, such as Acupoint herbal patching for postherpetic neuralgia with third-party arbitration and Diagnosis for Chinese patients with Light Chain Amyloidosis with blinded screening, supply the method we automate: blinding, a fixed extraction template, and adjudication. Gap analyses in the file propose "dual-agent extraction of EHR endophenotypes" in general terms. The idea here differs because it makes error independence the object of measurement, with genotype as the instrument.
Why this matters for translation
The motivating evidence is that better phenotypes recover hidden heritability. Deep-learning-derived glaucoma-related endophenotypes replaced case/control labels and mean nerve-fiber thickness, against a background where known variants explained "only about 9%" of glaucoma variance. Using imaging-derived representations in 47,908 UK Biobank participants, it found 43 independent loci, 21 of them novel. That paper explicitly ignored the text of ophthalmic records. Note-derived traits are not a weaker substitute for imaging. They are written in the vocabulary clinicians act on: a pressure threshold triggers treatment, and a cup-to-disc ratio triggers referral. Variants found on those traits attach directly to clinical decisions. That matters for the final step of the data-to-information-to-knowledge-to-action hierarchy laid out in The Training of Next Generation Data Scientists in Biomedicine. It also bears on the vocabulary mismatch between hospital records and research ontologies described in Towards an interoperable information infrastructure and The Ontology of Biological Attributes. We map each extracted trait to an Ontology of Biological Attributes term so the resulting associations are computable on both sides.
The ancestry concern is a constraint, not an afterthought. The Mexican familial hypercholesterolemia cohort study shows that genetic architecture invisible in European-derived pipelines appears in other populations. If an extractor misreads notes differently for patients seen in different clinics, which often differ in ancestry, it can create spurious associations or erase real ones. The $h^2_{\mathrm{SNP}}(D)$ test, together with regressing $D$ on ancestry principal components, is designed to catch exactly this.
Research Program
The anchor application is glaucoma. It has an external reference set (the 43 imaging-derived loci), and its note-recorded traits include the hard cases. Cup-to-disc ratios and pressure readings are short numeric strings where a one-character slip changes meaning, the same vulnerability Auto Review found for alphanumeric fields. The traits are maximum recorded pressure, worst-eye vertical cup-to-disc ratio, visual-field defect pattern, laterality of first progression, and age at first treatment.
We build a fixed extraction template with operational definitions, in the style of the registered human protocols. Two extractors from different model families each receive the same notes and template and never see each other's output. Where they disagree beyond a field-specific tolerance, a third model sees both values and the source passages and adjudicates; a fallback rule sends the case to a human. The program requires a genotyped cohort with linked clinical notes. The file's resources include UK Biobank, which has linked records, but none that clearly offers rich ophthalmology free text. Securing a hospital biobank with notes and genotypes is therefore the first dependency, and we flag it as the main practical risk.
A human gold subset of about 1,000 patients is extracted by two blinded clinicians with senior arbitration, following the protocols above. It serves only to calibrate the label-free test: does $h^2_{\mathrm{SNP}}(D)$ track the known error correlation between extractors? It is not the main evaluation. Heritability and genetic correlations are estimated with standard genotype-based variance-component methods. A second trait class, lipid values recorded in notes but missing from structured lab feeds, provides a replication domain where the familial hypercholesterolemia study gives known genes (LDLR, APOB, PCSK9) as positive controls.
Baselines
The naive reference point is the structured record alone: billing codes for glaucoma, plus whatever pressure values exist in structured fields. This is what most biobank GWAS use. Its heritability and locus count set the floor. The file reports the roughly 9% variance explained by known glaucoma variants from the glaucoma endophenotype paper, though that figure is not computed on our cohort.
The strongest single-extractor competitor is one frontier LLM run once per patient with the same template. Practitioners would actually deploy this, and if modern models are accurate enough, dual review adds nothing. To make the comparison fair on cost, we also run that same model with $k$ sampled extractions and majority vote, with $k$ chosen so its total token budget equals the dual-plus-arbiter pipeline. All extractors get the same notes, template, and prompt-tuning budget, measured in development iterations on a held-out 200-patient set, and the same downstream genetic analysis.
The strongest existing two-stage method is the Auto Review design, implemented in both of its forms on our notes. In the verification form, the second model sees the first value and judges it. In the re-extraction form, the second model extracts from scratch and the two are compared. Its reported result is a 5.5% average F1 gain for re-extraction in its own domain. No reported result was found on Lacuna for heritability recovered by any NLP-extracted clinical phenotype, so on the genetic metrics there is no published number to beat.
Two ablations isolate the key ingredient, independence. The first is dual extraction with two samples from the same model family, which removes diversity of model origin. The second is the verification-form pipeline, which removes blinding and lets anchoring occur. Our pipeline beats these baselines if three things hold together. Its reconciled phenotype must reach higher observed heritability than the compute-matched majority-vote baseline, with non-overlapping bootstrap intervals. It must replicate more of the 43 reference loci at nominal significance. Its $h^2_{\mathrm{SNP}}(D)$ must be indistinguishable from zero while the same-family ablation's is not.
Why not just…?
Why not just run one strong LLM once? This is the real threat. If single-pass extraction is already accurate on these fields, random error dilutes heritability only slightly, and the dual pipeline's gain disappears. We do not argue this away. The compute-matched single-pass and majority-vote arms measure it directly. If their heritability and locus replication match the dual pipeline, the honest conclusion is that dual review is unnecessary for these traits. The $h^2_{\mathrm{SNP}}(D)$ test computed between two independent single-pass runs remains useful as a cheap audit even then.
Why not just sample the same model several times and vote? Voting removes sampling noise but not shared blind spots. If a model consistently misreads one clinic's abbreviation for cup-to-disc ratio, every sample repeats the error. The same-family ablation tests whether such shared errors exist. They would show up as nonzero $h^2_{\mathrm{SNP}}(D)$ across clinics in cross-family comparisons but zero disagreement within the family, which is precisely why within-family agreement is not evidence of correctness.
Why not just correct heritability analytically for measurement error? Standard attenuation corrections divide observed heritability by an estimated reliability. They work when errors are random and the error rate is known. They cannot repair errors correlated with ancestry or genotype, which bias effect estimates in a direction no reliability factor fixes. The gold subset tells us whether the simple correction applied to single-pass extraction recovers the dual pipeline's heritability. If it does for some traits, we report those traits as not needing dual review.
Why not just use imaging endophenotypes as Chen et al. did? Where imaging exists, it is likely the better phenotype, and we use it as the external reference. But many clinically decisive traits, such as treatment history, maximum pressure before treatment, and progression laterality, are recorded only in text. Many cohorts also lack archived scans. The genetic correlation between note-derived and imaging-derived traits shows how much the text adds beyond imaging.
Proposed Experiments and Decisive Checks
The first check calibrates the instrument. On the human gold subset, we compute the true error correlation between each extractor pair and test whether $h^2_{\mathrm{SNP}}(D)$ in the full cohort orders the pairs the same way. If it does not, genotype is not a usable referee and the core claim fails. A complementary synthetic test injects controlled, ancestry-correlated errors into one extractor's output and checks that $h^2_{\mathrm{SNP}}(D)$ and the regression of $D$ on ancestry components detect them at realistic sample sizes.
The second check is the head-to-head comparison above, run on all five glaucoma traits and replicated on note-recorded lipids. It reports observed heritability, the genetic correlation between extractors $A$ and $B$, $h^2_{\mathrm{SNP}}(D)$, replication of the 43 reference glaucoma loci or the known lipid genes, and field-level accuracy on the gold subset.
The third check tests the translational payoff in a narrow, falsifiable way. Loci found only on note-derived treatment-threshold traits are mapped to genes. We then ask whether those genes are enriched among the effector genes prioritized in the glaucoma endophenotype paper, relative to loci from the billing-code phenotype.
The decisive result is a clean split between conditions. Cross-family blinded dual extraction shows near-zero disagreement heritability and higher trait heritability than compute-matched single-model voting. The same-family and unblinded ablations show measurable genetic structure in their disagreements. If instead every pipeline shows zero disagreement heritability and equal heritability, the conclusion is that LLM errors on clinical notes are already independent enough that single-pass extraction suffices. That would also be worth knowing before biobanks spend money on dual review.
References
- Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversations
- Acupoint herbal patching for postherpetic neuralgia: A systematic review and meta-analysis
- Diagnosis for Chinese patients with Light Chain Amyloidosis: A scoping review
- Deep-learning-derived glaucoma-related endophenotypes enable novel genome-wide genetic and functional discovery
- The Training of Next Generation Data Scientists in Biomedicine
- Towards an Interoperable Information Infrastructure Providing Decision Support for Genomic Medicine
- The Ontology of Biological Attributes (OBA)—computational traits for the life sciences
- Analysis of a deeply-phenotyped familial hypercholesterolemia cohort from Mexico shows a role for both rare and common alleles across known dyslipidemia genes and reveals structural variation in a novel locus