Two fields that never cite each other are stuck on the same problem. Machine learning cannot evaluate how much a model should doubt itself, because no dataset contains a label saying "the correct amount of uncertainty for this input is 0.3" — the doubt a model should express is a latent variable we can never observe directly. Epidemiology hit an isomorphic problem when mass vaccination began masking natural infection in antibody tests: a vaccinated animal and an infected animal both test positive, so the true infection state became invisible. The biological field solved it not with better inference but by engineering the system — the DIVA strategy deletes a marker gene (like wbkA in Brucella abortus) from the vaccine, so reactivity against that missing marker becomes a verifiable, almost-certified signal of wild infection (Development of an Inactivated H7N9 Subtype Avian Influenza Serological DIVA Vaccine Using the Chimeric HA Epitope Approach, A highly efficient indirect ELISA and monoclonal antibody established against African swine fever virus pK205R). This proposal imports that design move into machine learning: if we cannot find ground truth for uncertainty, we can manufacture it, by deliberately withholding a known piece of information from the model's training set and thereby guaranteeing, by construction, exactly how ignorant the model must be about it.
The exchange runs both ways. Machine learning's uncertainty tooling is exactly what DIVA-based surveillance lacks: current serological pipelines classify "infected versus vaccinated" with fixed optical-density cutoffs, ignoring assay noise and the documented decay of the infection marker, anti-N IgG, over an 18-month horizon (Persistence of SARS-CoV-2 antibodies over 18 months following infection: UK Biobank COVID-19 Serology Study), while cohort studies like KoCo19 openly operate under unmodeled uncertainty about who was actually infected (From first to second wave: follow-up of the prospective COVID-19 cohort (KoCo19) in Munich (Germany)). So the proposal has one mechanism at its center — the engineered blind spot — used as an evaluation instrument on the machine learning side and as a deployment target on the serological side.
Core Claim
A negative marker is a designated subspace of the data — a class subset, a feature channel, a sensor parameter range, an annotation source — that is verifiably excluded from the model's training set while a matched control model sees it. Because the experimenter controls the exclusion, the epistemic uncertainty of the blind model on marker queries is known by construction to be large, and on non-marker queries known to be reducible. This gives the first per-instance, affordable reference for epistemic uncertainty, without the exhaustive data-resampling that Position: Epistemic uncertainty estimation methods are fundamentally incomplete identifies as the only rigorous alternative — and which its authors show is computationally out of reach (their NYC Taxi reference distribution required partitioning 1.2M records and retraining many models, and no reported result found on Lacuna for that method at production scale).
With this reference in hand, two long-open questions become testable. First, the bias-absorption problem: Navarro et al. prove via Bregman divergences that a model's approximation failure is silently absorbed into its aleatoric (data-noise) estimate, but nobody could measure this contamination in practice. A negative marker makes it measurable: if a method's predicted aleatoric uncertainty inflates on marker queries — where the model's failure is provably epistemic, not noisy data — the contamination is directly observed. We quantify it as the contamination gap
$$C = \mathbb{E}[\hat{u}_{\text{alea}}(x) \mid x \in \mathcal{M}] - \mathbb{E}[\hat{u}_{\text{alea}}(x) \mid x \notin \mathcal{M}],$$
where $\hat{u}_{\text{alea}}(x)$ is the method's predicted aleatoric uncertainty for input $x$ and $\mathcal{M}$ is the marker subspace; a clean method should show $C \approx 0$. Second, the decay problem: the ground-truth-free metrics UQ-AUC and UQ-C-index, which Legitimate ground-truth-free metrics for deep uncertainty classification scoring proves are maximized exactly when the score matches the true misclassification probability, assume time-invariant, unconfounded labels. In serology the label itself (anti-N seropositivity) decays. We extend these metrics to markers whose validity holds only within a time window, using the proper scoring rules for interval-censored data developed in A Strictly Proper Scoring Rule and a Calibration Metric for Interval-Censored Data Analysis — the exact statistical machinery needed when an event horizon is only partially observed, as with a past infection whose antibody signal is fading.
Research Program
The program has three components. First, we build the marker-construction toolkit: formal definitions of marker subspaces for classification, segmentation, and longitudinal tabular data; training regimes that enforce the exclusion; and matched control models that differ only in marker visibility, so any behavioral difference is attributable to the withheld information alone. This is the direct analogue of the chimeric-HA epitope swap: the engineer decides what the learner cannot know, and that decision becomes the reference standard. Second, we run the audits: marker-stratified UQ-AUC (the probability that a correct prediction outside the marker subspace receives lower uncertainty than a marker query — the stratified version of Pignet et al.'s ranking criterion, which achieved 94% to 99% correlation with hidden ground truth on synthetic 2D mixtures in their validation), plus the contamination gap above, applied to the full zoo of estimators: softmax confidence, MC Dropout, Deep Ensembles, and post-hoc temperature scaling. Third, we deploy the same machinery in the DIVA direction: a joint model of anti-S (the stable vaccine marker) and anti-N (the decaying infection marker) that outputs a calibrated posterior probability of "past wild infection versus vaccination-only," replacing static cutoffs, with the decay-parameterized interval-censored scoring rule as its evaluation — and validated first in simulation, because per-individual infection ground truth in real cohorts like KoCo19 is precisely what is missing.
What we do not do is equally important. We do not propose a new uncertainty estimator; we propose the reference standard that existing estimators have never been held to. And unlike Benchmarking uncertainty quantification for protein engineering, which benchmarks UQ methods on downstream protein fitness regression but constructs no markers and never touches immunological ground truth, our markers are the object of study, not a side condition.
Baselines
The "do nothing" reference point is raw softmax confidence from a single deterministic model — the default confidence signal in deployed systems, documented as uncalibrated and dangerously overconfident on inputs the model has never seen (I Find Your Lack of Uncertainty in Computer Vision Disturbing) — and, on the serological side, the current practice of a fixed optical-density cutoff, which is exactly what the cohort studies in Ethiopia and Munich use today (Long-term monitoring of SARS-CoV-2 seroprevalence and variants in Ethiopia provides prediction for immunity and cross-immunity). The strongest existing competitor on the machine learning side is Deep Ensembles, the field's de facto gold standard: an average of independently initialized networks whose predictive spread is treated as epistemic uncertainty. Navarro et al. prove it captures only procedural variance (sensitivity to initialization and batch order) and systematically underestimates total uncertainty, but that proof rests on a resampled reference distribution; our marker audit is the first independent check of it. For fairness, every method gets the same total parameter count and forward-pass budget — an ensemble of five members is compared against a single marker-audited model granted five passes' worth of compute — and the same tuning budget on the same marker and non-marker splits. A method counts as passing the audit only if its marker-stratified UQ-AUC stays high (uncertainty on marker queries genuinely outranks uncertainty on ordinary mistakes) while its contamination gap stays near zero. On the aleatoric side, the strongest competitor is the supervised approach of Supervised Uncertainty Quantification for Segmentation with Multiple Annotations, which trains predicted variance to match inter-radiologist disagreement and reports Normalized Cross-Correlation between predicted uncertainty and grader disagreement of 0.669 on LIDC-IDRI, beating the Kendall & Gal baseline of 0.597; this is the right baseline for the aleatoric half of the audit, and we run it unchanged. The key ablation is the matched control: the identical architecture trained with the marker information included. If the blind and control models show the same uncertainty profile, the marker construction — our only new ingredient — is doing no work, and the proposal's premise fails. As reference points for how bad current calibration can get, ModelNet40-E reports that PointNet++ holds 91.61% accuracy on clean point clouds but collapses to 16.21% under heavy LiDAR noise, while SimpleView holds 63.01% under the same noise (ModelNet40-E: An Uncertainty-Aware Benchmark for Point Cloud Classification); no reported result found on Lacuna for Deep Ensembles on that benchmark.
Why not just…?
Why not just use existing corruption or unanswerability benchmarks? This is the closest prior work and the most serious objection. ModelNet40-E injects physically calibrated sensor noise with exact parameters ($\sigma, \mu$) embedded as metadata, and CertainlyUncertain removes visual evidence via inpainting to create provably unanswerable questions. Both alter the input; neither alters what the learner was allowed to know during training. But epistemic uncertainty is by definition about the training information, not the test input — a model can fail on a clean input precisely because a class was withheld from its training set, and no input-side perturbation can produce that condition or the bias-absorption measurement that depends on it. There is also a real convergence to concede: CertainlyUncertain's inpainting is a data-side negative marker in spirit. The distinguishing experiment is the control pair — the same input, two learners differing only in training-time marker visibility — which no input-perturbation benchmark can run.
Why not just collect multi-rater labels? The supervised aleatoric route of Supervised UQ for Segmentation with Multiple Annotations already converts inter-annotator variance into an empirical uncertainty signal, and it works (NCC 0.669 on LIDC-IDRI). But disagreement between radiologists measures ambiguity in the data — aleatoric uncertainty only — and says nothing about what the model is ignorant of; it is also expensive at scale, as A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework notes. Our markers are complementary: they supply the epistemic half that no amount of extra raters can, and on LIDC-IDRI we can hold out one radiologist's contour family as a marker to test whether methods that pass the aleatoric audit (NCC near 0.669) still fail the epistemic one.
Why not just use Deep Ensembles plus a resampled reference distribution? Navarro et al.'s own remedy is to build the Reference Distribution by repeated resampling from the data-generating process. On small tabular data this works; on the benchmarks here it means retraining hundreds of models per evaluation, which is exactly the cost their paper does not escape. More importantly, the resampling route tests ensemble variance against data variance but still needs the ground-truth-generating process in hand. The marker route gets a per-instance reference from a single controlled training run plus one matched control. The decisive comparison is to run both on the smallest benchmark (a two-model budget on a synthetic set): if the marker audit's verdicts agree with the resampled reference's, the cheap instrument is validated; if they disagree, the markers are measuring something else and we report that honestly.
Why not just fit a Bayesian longitudinal decay model to the serology? In the DIVA application, a principled Bayesian model of anti-N waning with a probabilistic cutoff would already be an improvement over static thresholds, and we treat it as a serious baseline rather than a strawman — it is, in effect, the serological analogue of the binomial-process confidence priors of Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling. Where it falls short is twofold, both documented in the file: waning makes the infection label interval-censored, so a point cutoff on a decaying quantity is miscalibrated by construction and needs the proper scoring rule of A Strictly Proper Scoring Rule and a Calibration Metric for Interval-Censored Data Analysis to even be evaluated; and antigenic drift means novel variants can silently break assay binding, as documented across vaccine platforms in Ghana (Functional antibody responses to SARS-CoV-2 variants before and after booster vaccination among adults in Ghana) — a pure decay model has no mechanism for flagging that epistemic blind spot, whereas a marker-aware model trained with variant-withholding markers is audited for exactly this failure.
Proposed Experiments
Three experiments, escalating in realism. The first, on LIDC-IDRI (15,096 CT patches, each contoured by four radiologists), withholds one radiologist's annotation family or a lesion-type subset as the marker. Methods under audit: softmax confidence, MC Dropout, Deep Ensembles, temperature scaling, and the supervised aleatoric model. Success means preserving the reported aleatoric calibration (NCC near the 0.669 reference) while ranking marker queries as high-uncertainty — the two-axis audit that no existing evaluation can perform. The second, on ModelNet40-E, withholds a shape class or a range of the benchmark's embedded sensor-noise parameters from training; because the noise parameters are known per point, we can cleanly separate "the input is noisy" (aleatoric) from "the model never saw this regime" (epistemic, our marker), and test Navarro et al.'s prediction that ensembles misattribute the second to the first. Reference calibration levels to beat or contextualize: Point Transformer v3's clean-data ECE of 0.0437 and SimpleView's heavy-noise ECE of 0.2392, both reported in the benchmark paper. The third, on the serological side, is a simulation study parameterized by the documented facts — anti-N waning over 18 months, anti-S persistence, and cohort prevalence on the order of KoCo19's below-5% second-wave seroprevalence — followed by re-analysis of published dual-antigen cohort data, evaluating whether the interval-censored calibrated model recovers infection attack rates that static-cutoff pipelines miss, and how detection degrades as marker half-life shortens.
Decisive Checks
Two outcomes could falsify the core claim. If Deep Ensembles and MC Dropout pass the marker audit — near-zero contamination gap and high marker-stratified UQ-AUC — then the widely used estimators are better than current theory predicts, the bias-absorption problem is less severe than Navarro et al.'s proofs suggest, and the marker apparatus is an evaluation of modest value; we would report that as the finding. If, conversely, the audits confirm contamination across all popular estimators while the matched control models show it vanishing once the marker is restored, then engineered blind spots are doing real work, and the DIVA fields gain, in return, the calibrated surveillance machinery they currently lack. Either way, the deepest deliverable is the demonstration that ground truth for doubt can be built rather than found — the move biology made twenty years ago when it deleted a gene so that a missing antibody could certify an infection.
References
- Development of an Inactivated H7N9 Subtype Avian Influenza Serological DIVA Vaccine Using the Chimeric HA Epitope Approach
- A highly efficient indirect ELISA and monoclonal antibody established against African swine fever virus pK205R
- Persistence of SARS-CoV-2 antibodies over 18 months following infection: UK Biobank COVID-19 Serology Study
- From first to second wave: follow-up of the prospective COVID-19 cohort (KoCo19) in Munich (Germany)
- Position: Epistemic uncertainty estimation methods are fundamentally incomplete
- Legitimate ground-truth-free metrics for deep uncertainty classification scoring
- A Strictly Proper Scoring Rule and a Calibration Metric for Interval-Censored Data Analysis
- Benchmarking uncertainty quantification for protein engineering
- I Find Your Lack of Uncertainty in Computer Vision Disturbing
- Long-term monitoring of SARS-CoV-2 seroprevalence and variants in Ethiopia provides prediction for immunity and cross-immunity
- Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
- ModelNet40-E: An Uncertainty-Aware Benchmark for Point Cloud Classification
- CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness
- A Bayesian Approach to Quantify the Uncertainty of Human Ratings in a Single-Instance Multimodal Framework
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling
- Functional antibody responses to SARS-CoV-2 variants before and after booster vaccination among adults in Ghana