When a clinical exome pipeline names a variant as the cause of a family's disease, the clinician needs to know how often that call is right. This proposal argues that we currently cannot measure that properly, and that the number we do measure is too optimistic in a specific, predictable way. Variant prioritizers and language-model agents are scored on solved historical cases. The knowledge sources they consult were often updated because those cases were solved: the disease databases, the gene–disease annotations, and the published case reports that end up in language-model training text. Each later carries the answer. A retrospective benchmark therefore rewards recall of the answer, and confidence estimates fit on it inherit the same flaw. The family pedigree is different. Whether a candidate allele tracks with affected status across relatives (co-segregation) is evidence that comes from the family itself, and it does not change when databases are updated.
The core idea is to borrow the "time as a firewall" discipline used for contamination-free language-model benchmarks and apply it to rare-disease diagnosis. We freeze every knowledge source at a snapshot date. We evaluate only on cases solved after that date. We then build a confidence model that anchors on an explicit, penetrance-aware co-segregation likelihood and learns how much to trust database-derived evidence only from cases the databases could not yet have absorbed. Put in two sentences: retrospective evaluation makes diagnostic confidence look calibrated when it is not, especially for newly discovered gene–disease links. Pedigree evidence is the time-invariant signal that can keep confidence honest after the knowledge cutoff.
Core Claim
We claim three testable things. First, the calibration of top-ranked candidates degrades as the gap between knowledge snapshot and case-solution date grows. Calibration here means the agreement between stated confidence and the empirical rate of being correct. The degradation affects classic prioritizers and language-model agents alike, and it is concentrated in cases whose causal gene had no established disease association at the snapshot. Second, a calibrated posterior can stay close to calibrated across that gap. It combines a co-segregation likelihood ratio computed by explicit Mendelian calculation, with penetrance treated as uncertain rather than fixed, with a knowledge-based score whose weight depends on the gene's annotation status at snapshot time. Third, language-model agents show the same pattern in stronger form, because their pre-training cutoff is an additional, hidden snapshot.
The motivating evidence comes from three places. Refined preferences of prioritizers improve intelligent diagnosis for Mendelian diseases benchmarked Exomiser, PhenIX, AMELIE and LIRICAL on DDD and KGD trios and found that adding parents ("trio mode") sharply improves ranking. That study reports ranks rather than calibrated probabilities. It evaluates retrospectively against current databases. It also notes that rigid frequency filters discarded true causal variants with low penetrance. Clinical and genetic landscape of optic atrophy in 826 families documents incomplete penetrance and variable expressivity in OPA1 and WFS1 within a manual co-segregation workflow, with no algorithmic treatment of uncertainty. On the temporal side, LiveCodeBench indexes problems by release date so evaluation can post-date a model's cutoff, and Is Your LLM Outdated? shows performance decays on post-cutoff data. Nobody in the file applies either idea to calibration in genetic diagnosis.
Research Program
The first component is a temporally partitioned diagnostic benchmark. For each solved family we record the date the causal variant was established. We also record whether the causal gene–disease association existed in archived releases of the disease and phenotype-annotation databases at each candidate snapshot date. Every tool is then run against snapshot-frozen copies of its knowledge sources. This is the genomic analogue of the history-pruned repository initialization in Beyond Isolated Tasks. For language-model agents, it is the analogue of the search truncation used in ARAC and the multi-layer date-leakage filter of Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs. That paper still reports about 1.5% residual leakage, a warning that we must audit leakage rather than assume it away.
The second component is the confidence model. For candidate variant $v$ in family $f$, we write the posterior log-odds of causality as
$$\text{logit}\, P(v \text{ causal}) = \alpha_{s} + \beta \log \mathrm{LR}_{\text{seg}}(v, f) + \gamma_{s}\, k(v),$$
where $\mathrm{LR}_{\text{seg}}$ is the co-segregation likelihood ratio. It compares the probability of the observed genotypes and phenotypes in the pedigree if $v$ is causal against the probability if it is unlinked, with penetrance integrated over a prior instead of set to one. The term $k(v)$ is the knowledge-based prioritizer score (for example Exomiser's), and $s$ indexes the gene's annotation status at snapshot time: established disease gene, candidate gene, or no annotation. The intercepts $\alpha_s$ and slopes $\gamma_s$ are fit with a rolling origin. Parameters used to score cases solved in window $t+1$ are fit only on cases solved in window $t$ against the snapshot preceding $t$, so the calibrator never sees knowledge newer than its test cases. The structure borrows the per-source intercepts of hierarchical Platt scaling from the forecasting paper above. The difference is that we stratify by the knowledge status of the evidence and feed in an explicit pedigree likelihood, not a language-model belief summary.
The third component treats language-model agents as an evaluated system rather than the method. We give an open-weights model with a documented cutoff the same snapshot-frozen tools. We then compare its confidence, read out either from logits or from a hidden-state probe in the style of Uncertainty Calibration for Tool-Using Language Agents, on cases solved before versus after its pre-training cutoff. Many families in the file's ocular-genetics literature are published case reports, so a model may simply recall that a gene explains a phenotype. That recall should inflate both accuracy and confidence on pre-cutoff cases, which is exactly what the split is designed to expose.
The data are the 305 DDD and 152 KGD trios used by the prioritizer benchmark and the 826-family optic atrophy cohort. The KGD and optic atrophy cohorts are in-house or controlled-access, so this work requires collaboration with their holders. Trios carry limited segregation information beyond de novo and compound-heterozygous configurations, so we expect most of the benefit of $\mathrm{LR}_{\text{seg}}$ on the multi-generation pedigrees. We will report the two kinds of families separately.
Baselines
The naive reference point is Exomiser in trio mode with current databases, its rank-1 candidate treated as "confident." Its retrospective top-1 rate is the strongest reported number in the file: 61.6% of DDD cases, with 83.3% to 91.8% of causal genes in the top ten under refined settings (Refined preferences of prioritizers). No reported calibration result (ECE or Brier score) for any variant prioritizer was found on Lacuna, so calibration baselines must be computed here. We rerun Exomiser and LIRICAL on the same families against snapshot-frozen databases. Comparing the retrospective and snapshot-frozen top-1 rates directly measures leakage.
The strongest calibration competitor is Platt or isotonic calibration of the prioritizer score, fit on a random split of the same cases. This is what a careful practitioner would do today. We give it the same features, the same families and the same tuning budget, and evaluate it on the temporally held-out windows. A second, stronger version fits the same calibrator with a rolling origin but without the segregation term. This version isolates whether temporal honesty alone fixes the problem.
For language-model agents, the strongest existing method is the hidden-state probe of Uncertainty Calibration for Tool-Using Language Agents, which reduced expected calibration error from 0.690 to 0.035 on TabMWP with 500 training questions. Our version trains the probe on pre-cutoff families and tests it on post-cutoff families with matched training-set size. We also report verbal confidence and five-sample agreement, given the finding in Assessing the possibility of using large language models in ocular surface diseases that 28% of GPT-4's consistent answers were wrong.
The key ablation removes the penetrance prior, replacing $\mathrm{LR}_{\text{seg}}$ with a full-penetrance Mendelian filter. It also separately drops the status-dependent slopes $\gamma_s$. Beating the baselines means lower expected calibration error and Brier score on post-snapshot windows, with top-1 accuracy no worse, and the largest gains on the no-annotation stratum and on incompletely penetrant families.
Why not just…?
Why not just calibrate Exomiser's score with isotonic regression? This is the real threat. If a rolling-origin isotonic calibrator stratified by annotation status matches our model, the pedigree likelihood adds nothing, and the contribution reduces to the benchmark plus that finding. We would report it that way. Our reason to expect a gap is that, for genes with no annotation at the snapshot, the knowledge score is nearly uninformative. In that stratum only family evidence can separate a good candidate from a bad one, and a one-dimensional calibrator on a weak score cannot create resolution it does not have. The ablation without the segregation term settles this directly.
Why not just evaluate on a random split of historical cases? Random splits leave the leak in place. Test cases' answers are already in the frozen databases and, for language models, in the training text. A Comprehensive Survey of Contamination Detection Methods and Unveiling Causal Reasoning in Large Language Models describe this "illusion of progress" in general settings. We measure it here by computing every metric under both random and temporal splits on identical families.
Why not just probe a language-model agent's hidden states, which already calibrate well? The probe's strong result on TabMWP comes from a static benchmark without temporal isolation. In diagnosis, a probe trained on pre-cutoff families can learn "this case looks familiar" as a proxy for "this answer is right." That proxy fails exactly on new gene discoveries. If the probe stays calibrated on post-cutoff families, we would conclude that language-model confidence is more robust than we expect, and that is worth knowing either way.
Why not just reanalyze unsolved cases periodically with updated databases? Reanalysis improves eventual yield, but it does not tell the clinician at decision time how much to trust today's top candidate. Calibration under a frozen snapshot is precisely that decision-time quantity.
Proposed Experiments
The first decisive check tests the premise. On the snapshot-frozen benchmark, we plot top-1 accuracy and expected calibration error of each baseline against the gap between snapshot and solution date, stratified by annotation status. If retrospective and temporally isolated calibration agree within bootstrap error, the premise is false and we stop the methods work. The benchmark result would still stand as a negative finding.
The second check compares our posterior with the rolling-origin calibrators on held-out windows, reporting reliability diagrams, Brier score, and precision at fixed coverage. Precision at fixed coverage means the fraction of families that could be reported automatically at 90% empirical precision. We break results out by trio versus extended pedigree and by penetrant versus incompletely penetrant families. The incompletely penetrant group is drawn from the optic atrophy cohort's OPA1 and WFS1 families, where the prioritizer benchmark reports that filters failed.
The third check runs the language-model agent across its own pre-training cutoff, holding tools and snapshot fixed. Any accuracy or confidence drop at the cutoff, beyond what the database snapshot alone explains, is attributable to memorized literature. We audit residual leakage by searching agent traces for the post-snapshot gene–disease names, following the second-pass filtering idea of the forecasting work.
References
- Refined preferences of prioritizers improve intelligent diagnosis for Mendelian diseases
- Clinical and genetic landscape of optic atrophy in 826 families: insights from 50 nuclear genes
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Is Your LLM Outdated? A Deep Look at Temporal Generalization
- Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
- ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs
- Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
- Uncertainty Calibration for Tool-Using Language Agents
- Assessing the possibility of using large language models in ocular surface diseases
- A Comprehensive Survey of Contamination Detection Methods in Large Language Models
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?