Grading Somatic Mutation Calls Without an Answer Key

When a cancer genomics lab runs a pipeline on a new tumor sample, it cannot tell how many of the reported mutations are real. Truth sets exist only for a handful of benchmark genomes. Those benchmarks were built from very pure tumors and balanced coverage, while clinical samples are often low-purity biopsies, contaminated normals, or blood draws in which tumor DNA is a fraction of a percent. As a result, every new assay, purity level, or library protocol is a setting where the pipeline's precision is assumed rather than measured.

This note proposes a way to measure it anyway. A somatic variant caller is a program that compares tumor and normal DNA reads and flags mutations specific to the tumor. Several such callers, each with a different statistical model, can be treated as a panel of noisy annotators. Their pattern of agreement and disagreement on a single new sample can be used to estimate that sample's precision and recall, broken down by genomic context. The estimate is then used to pick the decision threshold for that sample, instead of applying one fixed cutoff everywhere. Ensembling is usually used to make better calls. Here it is used to evaluate the calls when no answer key exists.

Core Claim

A latent-class model over multiple callers can recover per-sample, per-context precision to within a few points of the truth, on benchmark conditions it was never fitted to. A latent-class model treats the true status of each site as hidden and infers it jointly with each caller's error rates. The key requirement is that it represents correlated caller errors as separate "artifact families" (for example, homopolymer slippage, low-mappability misalignment, or germline leakage through a contaminated normal), rather than assuming callers err independently.

If the claim holds, two things follow. First, label-free estimates should rank pipelines and thresholds in the same order that held-out truth does. Second, choosing each sample's threshold from its own estimated precision–recall curve should lose less F1 than SomaticSeq's fixed probability cutoff of 0.7 when samples shift away from the training distribution. The closest existing work is supervised: it improves the calls but never estimates their accuracy on unlabeled samples.

Why This Gap, and Why Now

The motivating evidence is the scale of disagreement among callers. The ICGC benchmarking studies (A comprehensive assessment of somatic mutation detection in cancer using whole-genome sequencing; A Comprehensive Assessment of Somatic Mutation Calling in Cancer Genomes) ran 18 pipelines on the same tumors. Reported mutation counts ranged from 467 to 3,159 for chronic lymphocytic leukemia and from 801 to 8,876 for medulloblastoma. Those benchmarks used tumors of 95–98% purity, so even this spread understates clinical conditions. Systematic comparison of somatic variant calling performance among different sequencing depth and mutation frequency mixed two genomes digitally to create known mutations. It found recall below 35% at 1% allele frequency at every depth from 100X to 800X, and it found that the better-performing caller changes with allele frequency. Performance therefore depends on operating conditions, which is exactly what a fixed validation figure hides.

The method builds on supervised ensembles but departs from their main assumption. SomaticSeq trains boosted trees on five caller outputs plus more than 70 sequencing features. It reaches F1 90.5% against MuTect's 62.4% in the low-purity DREAM Setting D. Its own authors note that it needs platform-specific training data. VEF does the same for single-sample germline filtering with random forests. Both need labels from the deployment distribution, which the clinical setting lacks. AIVariant avoids real labels by building synthetic tumor–normal dilutions for training. That approach supplies our evaluation design but does not address how to validate a model once it is deployed. Evaluating Bioinformatics Processing of Somatic Variant Detection in cfDNA Using Targeted Sequencing with UMIs evaluates four callers on cell-free DNA (tumor DNA fragments circulating in blood plasma). It shows that the best caller differs between tumor-informed and tumor-agnostic use, and that the sequencing depth of the matched blood-cell normal shifts false-positive rates. That last finding is a shift a fixed validation figure would miss.

The methodological stance comes from Design and Analysis of the NIPS 2016 Review Process. When ground truth is unavailable, one should build statistical proxies and then check them wherever truth can be obtained. We adopt that discipline for variant calling.

The Bayesian Nonparametric Ensemble (Accurate Uncertainty Estimation and Decomposition in Ensemble Learning) decomposes ensemble uncertainty into noise and model misspecification. It is the obvious tool at this intersection, but it constrains rather than supplies our design. It still needs labeled targets to fit its residual process. Its Gaussian-process cost grows cubically with the number of sites, which is prohibitive across millions of candidates. Its output is an uncertainty score, not a precision estimate a lab can act on.

Research Program

Each candidate site $i$ in a sample carries a hidden label $z_i$. The label is either "somatic" or one of $M$ artifact or germline families. The site also has an observed context $c_i$: a stratum defined by allele-frequency bin, depth bin, homopolymer or tandem-repeat membership, and mappability. Each caller $k$ emits a discretized score $s_{ik}$. We model

$$ p(s_{i1},\dots,s_{iK}\mid c_i)=\sum_{z}\pi_{z\mid c_i}\prod_{k=1}^{K}\theta_{k}(s_{ik}\mid z, c_i), $$

where $\pi_{z\mid c}$ is the share of each class in context $c$, and $\theta_k$ is caller $k$'s score distribution given the class and context.

Callers are independent only given the family. Their tendency to fail together in a homopolymer is absorbed by a homopolymer-artifact family in which all of them fire. Classical latent-class results are the source of the expectation that the model can be fitted. They imply the parameters are recoverable when at least three views are conditionally independent given the latent class. Our design choice is to supply such views that do not share the callers' alignment. These include a panel-of-normals frequency, strand and read-position statistics, and, in cell-free DNA, the matched blood-cell allele count.

Priors on $\theta$ come from public labeled benchmarks, and the model is refitted on each new sample by expectation–maximization. The estimated precision at any score threshold is the posterior expected fraction of somatic sites among the calls that pass it. Recall is estimated the same way from the posterior mass of somatic sites. As a secondary, independent check on recall, we inject synthetic variants into the sample's own reads at chosen allele frequencies.

The work has three parts. We build the estimator over Mutect2, Strelka2, VarScan2, and VarDict, with shearwater and DREAMS-vc added for cell-free DNA. We characterize where the model cannot be fitted, and report "no estimate" for strata where it is underdetermined. Finally, we release the shift-stratified evaluation suite as an artifact in its own right.

Baselines

The naive reference point is "trust the validation number." This takes SomaticSeq's precision at P ≥ 0.7 on DREAM training conditions and reports it for every target sample. It is what labs effectively do today, and beating it means smaller absolute precision error on shifted conditions.

The strongest competitor is SomaticSeq retrained on the same source benchmarks and features, with its probabilities recalibrated by isotonic regression on source data. Its mean posterior over passing calls then serves as the precision estimate. It is the strongest because it uses supervision the latent model only takes as priors, and because its reported F1 is 90.5% in DREAM Setting D (SomaticSeq). For label-free precision estimation itself, no reported result was found on Lacuna for any method.

Both systems receive identical callers, features, source data, and hyperparameter search budget. Beating SomaticSeq requires lower mean absolute precision error across held-out shift conditions. It also requires lower F1 regret from per-sample threshold choice.

Two ablations isolate the key ingredient. The first is classic Dawid–Skene, the standard model for combining noisy annotators: a single artifact class with caller independence. The second keeps multiple families but removes context stratification. If either matches the full model, the artifact-family idea is not doing the work.

We also report majority-vote agreement rate as a crude proxy. As an upper bound, we report a supervised oracle trained on target-condition labels.

Why not just…?

Why not just use in silico spike-ins in each sample? Spike-ins measure recall well, and we use them for that. However, false positives come from real artifacts already present in the sample's reads, such as GC-biased PCR libraries (ICGC WGS assessment) or contaminated normals (MSK-IMPACT Heme). Injecting known true variants says nothing about which observed calls are fake. Precision is the missing quantity.

Why not just trust a calibrated supervised ensemble? This is the real threat. If calibration transfers across purity, depth, and assay, the latent model adds nothing. The SomaticSeq authors' own note on platform dependence, and the cfDNA finding that normal-sample depth confounds callers (cfDNA UMI evaluation), suggest it will not transfer. The leave-one-condition-out experiment below settles the question directly.

Why not just use agreement rates or independent-annotator models? Callers share alignments and therefore share blind spots. Both the ICGC studies and SomaticSeq argue that voting rewards correlated errors. Under independence, a homopolymer artifact that every caller reports looks maximally confident. The Dawid–Skene ablation quantifies how much precision is overstated in repeat strata.

Why not just apply the Bayesian Nonparametric Ensemble? It yields uncertainty flags rather than precision. It needs target labels, and it scales cubically with the number of sites. We include a sparse-approximation version on the ICGC gold-set candidate sites. If its uncertainty correlates with true error as well as our estimates do, we will report that.

Proposed Experiments

The core test is leave-one-condition-out. Truth comes from three sources. The first is the 30 depth-by-allele-frequency digital mixtures of NA12878 and YH-1 (Chen et al.). The second is DREAM Setting D. The third is the ICGC CLL and medulloblastoma gold sets, downsampled and diluted with normal reads to 20–40% purity following the AIVariant dilution recipe.

For each held-out condition, we fit priors on the remaining conditions and estimate precision and F1 label-free. We then compare against truth by absolute error, by Spearman correlation across conditions and pipelines, and by threshold-selection F1 regret.

The decisive checks are three. First, precision error must stay low in homopolymer and low-mappability strata, where independence models should fail. Second, estimated precision must fall as purity drops, tracking the collapse that AIVariant reports for single callers, whose precision–recall AUC ranges from 0.048 to 0.524 at 40% purity and 15x depth. Third, on the colorectal cell-free DNA cohort, estimated per-patient tumor-variant burden must separate the 111 patients from the 37 healthy controls. That separation is checked against the reported tumor-agnostic ROC-AUC of 0.808 for DREAMS-vc (cfDNA UMI evaluation), with tumor-exome-informed sites serving as partial truth.

The estimator fails if the calibrated supervised baseline matches it on shifted conditions, or if its errors in repeat strata exceed those of Dawid–Skene.