Targeted sequencing finds small DNA changes. Clinicians then have to decide whether each change causes disease. Under the ACMG/AMP clinical guidelines, a key input to that decision is published laboratory evidence that the variant breaks, or fails to break, the protein's function. That evidence is also graded by strength: "Supporting," "Moderate," or "Strong." Current language-model curators handle the first question well. VarLitBench and VarLitAgent reports that o4-mini separates damaging (PS3) from benign (BS3) functional evidence with an F1 of 0.979. The same work states that models fail at the second question, grading strength, because the facts that decide strength are mostly in supplementary files the pipeline never reads. Those facts are how many control variants the assay used, how many replicates it ran, and how wide its readout range was.
This note proposes a model-based judge for evidence strength. It reads supplements as well as main texts, extracts those facts explicitly, and turns them into a grade by a fixed rule. When it is not confident enough, it hands the case to a human curator. It does so under a statistical guarantee that limits how often it overstates strength among the cases it does grade. Overstatement is the error that can push a variant of uncertain significance into a "likely pathogenic" report. The central bet is that strength grading fails for two reasons. First, the judge is missing assay-validation facts. Second, it is asked for one holistic verdict that cannot be checked. Fixing both, and wrapping the result in a one-sided risk guarantee, should make automated strength grading usable for a substantial fraction of cases.
Core Claim
LLM-as-a-judge means using a large language model to grade outputs or evidence in place of a human reviewer. We claim that a judge built in three stages will grade ACMG functional-evidence strength more accurately than a judge asked for the grade directly, even when both see identical documents including supplements. The first stage extracts assay-validation facts from the full text and supplements. The second applies a fixed mapping from those facts to a strength level. The third accepts its own grade only when an uncertainty score clears a calibrated threshold.
The threshold is chosen so that, with high probability, the overstatement rate among accepted cases stays below a target $\alpha$ the user picks. The guarantee depends only on exchangeability between calibration and test cases, not on the model being correct. It also works with closed models that do not expose token probabilities.
Here is the precise sense of "overstatement." Let $y \in \{0,1,2,3\}$ be the curator's strength level (none, Supporting, Moderate, Strong). Let $\hat y$ be the judge's grade, and let $s$ be its uncertainty score. The judge accepts a case when $s \le t$. The quantity we bound is
$$ R(t) = \Pr\left(\hat y > y \mid s \le t\right), $$
the probability that an accepted grade is stronger than the curator's.
We pick the most permissive threshold $t$ satisfying one condition. On a held-out calibration set, the one-sided Clopper–Pearson upper confidence bound on $R(t)$ must be at most $\alpha$ at confidence $1-\delta$. To keep the guarantee valid while scanning over thresholds, $\delta$ is split across the candidates. Understating strength is costed separately and reported, not bounded, because it sends cases to human review rather than into reports.
Relation to Prior Work
The proposal reuses its statistical machinery from Judge, Retrieve, or Abstain. That paper routes a factual-QA judge through three options: answer from its own knowledge, answer after retrieving evidence, or abstain. It sets the thresholds with Clopper–Pearson bounds so the false discovery rate stays below $\alpha$. With a Qwen3-8B judge on Natural Questions at $\alpha = 0.20$, retrieval raised coverage from 7% to 82%.
We differ from it in three ways. First, the label is ordinal and the controlled error is one-sided. That paper lists graded labels as an open problem; we do not claim to solve graded guarantees in general, only that one-sided ordinal risk reduces cleanly to a binary event. Second, its uncertainty score is predictive entropy over tokens, which needs access to model probabilities. We replace this with agreement across repeated samples and across the extracted facts. Third, its "retrieve" step is web search. Ours is reading the paper's supplements.
AgentEval supplies the decomposition idea and constrains how we evaluate it. It grades each step of an agent's execution trace against that step's inputs rather than grading only the final answer. That approach raised failure-detection recall from 0.41 to 0.89, and it found that 63% of step failures were inherited from earlier steps. We apply step-level judging to a short, fixed chain: variant match, then fact extraction, then strength mapping. This lets us report which stage causes each overstatement.
Three genomics papers motivate the gap without filling it. VarLitBench documents the failure at strength grading and the missing supplements. It also supplies its benchmark, its safety gate (return not_clear when the variant tested cannot be matched to the target), and its cost reference points: about $0.02 per sample in its single-pass mode versus $0.06 in its multi-page agentic mode.
AIVARI Agent shows the practical cost of uncontrolled automated adjudication. It reaches a group sensitivity of 0.956 but a row precision of only 0.351, so clinicians must review many false candidates. Our design trades coverage for a stated error bound rather than trading precision for sensitivity.
CAGI maps computational pathogenicity scores onto ACMG strength tiers with a Bayesian calibration. In its NAGLU challenge, 6% of predictions reached "Strong" and 31% reached "Supporting." That work grades predictors, not published wet-lab evidence, but it shows strength levels can be tied to explicit numbers. This is the premise of our fixed mapping stage.
The sequencing direction sets the application regime. The variants at stake are the single-base changes and small insertions or deletions found by targeted amplification and Sanger or amplicon sequencing. Examples in the file include the PAX3, RS1 and STK11 studies (PAX3 in Waardenburg syndrome, RS1 in retinoschisis, STK11 in Peutz-Jeghers). In these studies, finding the variant is the easy step and interpreting it is the bottleneck. We deliberately do not attempt upstream primer judging. Primer3 plus BLAST and thermodynamic checks already cover most of what a language model would read in that setting. And the file contains no public wet-lab primer-outcome benchmark beyond PrimerZ's aggregate success rate of over 70%.
Research Program
The work has four parts.
Data. We extend VarLitBench's 463 ClinGen-curated functional-evidence variants with the curator's strength level for each PS3/BS3 assertion. Supplementary files are retrieved for each source paper. Where ClinGen records do not state strength unambiguously, two curators annotate it and we report their agreement as the human ceiling. Because 463 cases give wide confidence bounds, we add further ClinGen functional-evidence assertions from the same repository until the calibration and test splits each have several hundred items. Splits are made by paper, so no paper's figures appear on both sides.
Fact extraction. For each variant and paper, the judge extracts a fixed schema from the main text, figures and supplements:
- whether the assay was run on the exact variant (the VarLitBench safety gate);
- the number of known pathogenic and known benign control variants;
- the number of biological replicates;
- whether a statistical test or threshold separated normal from abnormal readouts;
- the assay's readout range relative to its controls.
Each extracted field is judged step-wise against its quoted source span, in the AgentEval style. This produces a confidence for every field.
Mapping. A fixed, human-readable rule turns the facts into a strength level. The rule's cut-points (for example, how many controls are needed for "Moderate") are tuned only on the training split.
Abstention. The uncertainty score combines two signals: disagreement in the final grade across $k$ independent sampled extractions, and the lowest field-level confidence. The threshold is calibrated as above, for $\alpha \in \{0.05, 0.10, 0.20\}$ and $\delta = 0.1$.
Baselines
The naive reference point is a majority-class rule: assign every PS3/BS3 assertion its most common strength level in the training split. A slightly stronger version assigns the most common level for each assay type. This baseline is free and shows how much signal any judge must add. If it already achieves a low overstatement rate (for example, because most labels are "Supporting"), that is itself a finding, and we report it.
The strongest existing method is VarLitAgent in its agentic multi-page mode from VarLitBench. It is prompted to output a strength level using the ACMG rubric, and we give it the same supplements our system reads, which the original pipeline did not ingest. It is the right competitor because it already reads figures and multi-column tables, and its claimed limitation is exactly the missing supplements. Adding them is the most direct test of whether supplements alone close the gap. The file reports its direction F1 of 0.979 (o4-mini). There is no reported result found on Lacuna for strength-grading accuracy on VarLitBench or any other benchmark, so the bar is set by running it.
For fairness, both systems use the same base model, the same documents, and the same sampling budget, measured in model calls per case. The baseline gets as many prompt-tuning iterations on the training split as our mapping rule gets cut-point tuning. Both systems receive the identical Clopper–Pearson abstention wrapper, with self-consistency of the holistic grade as the uncertainty score. We beat it if, at the same certified $\alpha$, our coverage is higher by more than the bootstrap confidence interval. That is, we must grade more cases automatically while the realized overstatement rate on the test split stays below $\alpha$.
Two ablations isolate the key ingredients. The first keeps fact extraction but replaces the fixed mapping with the model's own grade given the extracted facts. This tests whether the explicit rule matters or only the extraction does. The second removes supplements from our pipeline, which tests the supplement-gap claim directly. A third run swaps self-consistency for token entropy on an open-weights model, to check that the probability-free score does not lose much coverage.
Why not just…?
Why not just give the existing agent the supplements and ask for a grade? This is the most serious threat, and the main experiment exists to settle it. If the supplement-augmented VarLitAgent matches our coverage at equal $\alpha$, the decomposition adds nothing. The contribution would then shrink to "supplements plus risk control," and we would report it that way. Our reason to expect a gap is AgentEval's finding that holistic, end-to-end judging hid most step failures (recall 0.41 versus 0.89). Strength is a function of several countable facts, so one misread replicate count silently changes a holistic grade but appears as a low-confidence field in ours.
Why not just turn the problem into a binary one and reuse the existing guarantee? In part we do: one-sided overstatement is a binary event, and the Clopper–Pearson machinery of Judge, Retrieve, or Abstain applies unchanged. What that work does not provide is an uncertainty score for closed models, or a reason why abstention should rise on the cases that are actually hard. We do not claim a new theorem. The open question is empirical: can probability-free scores that know which facts are missing reach useful coverage on a few hundred calibration cases? A plain "is this grade correct" bound is reported as a baseline wrapper. It controls a different and less clinically relevant error, since it treats an understated grade the same as an overstated one.
Why not just use a calibrated computational predictor, as in CAGI, instead of reading papers? Predictors such as REVEL and VEST reach ROC-AUCs of 0.85–0.92 for pathogenicity in CAGI. However, ACMG treats computational evidence and functional evidence as separate criteria. A predictor score cannot certify that a published assay had adequate controls, which is the question curators must answer under PS3/BS3. We include predictor scores only as an optional extra feature, to check whether they reduce the judge's abstention.
Proposed Experiments and Decisive Checks
The main result is a coverage–risk curve for every system at $\alpha \in \{0.05, 0.10, 0.20\}$. It reports the realized test overstatement rate (which must stay at or below $\alpha$ across repeated random calibration/test splits), coverage, exact-level accuracy and quadratic-weighted kappa against curators on accepted cases, and median cost and latency per case. Cost and latency are compared against VarLitBench's $0.02 and $0.06 reference points.
The first decisive check is the supplement ablation. If removing supplements does not lower coverage, the stated cause of the strength failure is wrong.
The second is the comparison between fixed mapping and the model's own grade given the same extracted facts. If they tie, the explicit rule is unnecessary.
The third is stage attribution. For each accepted overstatement, we record whether it came from variant mismatch, a misread fact, or the mapping, following AgentEval's tracing. If most errors come from variant mismatch, the next investment belongs in the safety gate, not the judge.
A transfer check calibrates on papers published before a date cutoff and tests on later ones. Because the guarantee assumes calibration and test cases are exchangeable, this shows whether it degrades under publication drift, which a deployed curator will face.
References
- VarLitBench and VarLitAgent for Benchmarking and Automating LLM-Assisted Functional Evidence Curation in Genomic Variant Interpretation
- Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- AIVARI Agent: An Evidence-Grounded Agentic LLM for Variant Reportability and Interpretation
- CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods
- PAX3 mutations and clinical characteristics in Chinese patients with Waardenburg syndrome type 1.
- Novel RS1 mutations associated with X-linked juvenile retinoschisis
- Germline mutations of the STK11 gene in Korean Peutz-Jeghers syndrome patients.
- PrimerZ: streamlined primer design for promoters, exons and human SNPs