Specification Uncertainty: Measuring How Much a Prediction Depends on Undocumented Preprocessing

When a hospital adopts a published medical imaging model, it has to rebuild the data pipeline from the paper's description. That pipeline covers how slides are color-normalized, how tissue is separated from background, and how image patches are cut and labeled. Papers routinely leave these steps out. A careful replication of three top Camelyon pathology methods found that the omissions alone could push slide accuracy from 92% to 54.7% and lesion-localization score from 0.981 to 0.499 (Reproducibility of deep learning in digital pathology whole slide image analysis). Generative uncertainty methods can tell the adopter where the model is unsure. These are models that draw many plausible outputs for one input and report how much the outputs disagree. But they cannot say whether the answer would change if the slide had been prepared the way the original authors prepared it. The reason is that their disagreement is computed after preprocessing, conditional on the adopter's choices.

We propose to measure that missing quantity directly. We treat every preprocessing decision the paper left unstated as a random variable. We sample plausible pipelines, run the generative model under each, and split the total spread of predictions into two parts. One part is the model's own doubt given a fixed pipeline. The other is the spread caused by the pipeline choice itself, which we call specification uncertainty. The claim is that the second part is large, spatially localized, and poorly predicted by existing uncertainty estimates, including methods built to separate model doubt from data noise. The claim also has a human side. Telling a non-specialist operator "this region depends on how you prepared the slide" gives them something they can act on. A generic confidence map mostly makes them anxious, and it does not tell them what to do.

Core Claim

Let $y$ be a prediction (a segmentation mask or a patch label). Let $\pi$ be a preprocessing pipeline drawn from the set of choices consistent with a paper's written description, and let $z$ be the generative model's random seed. By the law of total variance,

$$ \mathrm{Var}[y] = \underbrace{\mathbb{E}_{\pi}\big[\mathrm{Var}_{z}(y \mid \pi)\big]}_{\text{model doubt under a fixed pipeline}} + \underbrace{\mathrm{Var}_{\pi}\big(\mathbb{E}_{z}[y \mid \pi]\big)}_{\text{specification uncertainty}}, $$

and both terms can be computed per pixel.

Existing generative uncertainty work estimates only the first term, because it evaluates each sample under one fixed pipeline. That holds for implicit diffusion ensembles (Diffusion Models for Implicit Image Segmentation Ensembles), flow-matching standard-deviation maps (Conditional Generative Models for Contrast-Enhanced Synthesis of T1w and T1 Maps in Brain MRI), and methods that separate epistemic from aleatoric uncertainty such as Hyper-Diffusion and FLARE (Quantifying Epistemic Uncertainty in Diffusion Models). Here "epistemic" means doubt from missing training data, and "aleatoric" means irreducible noise in the data. We predict that the second term explains a substantial share of replication error that the first term misses.

How This Relates to Prior Work

Three papers motivate the gap. Bhatt et al. argue conceptually that uncertainty is a form of transparency, and they leave open how users interpret different kinds of uncertainty (Uncertainty as a Form of Transparency). We make the argument concrete for one kind of opacity, missing data documentation, and we test it empirically. The mammography subgroup study used Monte Carlo dropout to show that rejecting uncertain cases with naive confidence scores can widen demographic and scanner disparities, a "triage trap." In that study, predictive variance lowered race disparity at 10% rejection from 0.051 to 0.034 (Evaluating subgroup disparity using epistemic uncertainty in mammography). That study covers classification only and never varies preprocessing. We ask whether rejecting by specification uncertainty avoids the trap in dense prediction. Fell et al. supply the list of omitted choices that becomes our pipeline space, and they propose a 12-point checklist. We treat the checklist as the remedy for future papers and our method as the measurement tool for the large body of existing ones.

Two further papers constrain the design. WILDS shows that robustness-oriented training methods fail to beat plain training in 8 of 10 real-world shifts. It also shows that hospital-level metadata is what makes worst-group analysis possible (WILDS). We therefore do not propose a new training method, and we use the WILDS hospital splits as the natural shift axis. The diffusion segmentation paper reports 48 seconds per sample over 1,000 steps. That latency limits how many pipeline-by-seed combinations we can afford, and it shapes the sampling budget below.

The literacy literature shapes the evaluation. Fan et al. find that combining experience with external data had no significant effect on teaching management (path coefficient 0.034, p = 0.704) unless teachers had the mediating disposition they call data consciousness (How data knowledge transformation affects teachers' precision teaching ability). TPACK surveys show technological knowledge lagging pedagogical knowledge (3.12 versus 3.98) even among young professionals (The Investigation and Analysis of Pre-Service Teachers TOWARD TPACK Competencies). Breakable Machine teaches AI literacy with deterministic saliency maps and training-data browsing, but it conveys no uncertainty (Breakable Machine). Together these suggest that showing uncertainty is not enough by itself. It should be tied to an action the operator can take. They also show that most literacy measures are self-reported. We therefore measure literacy and decision quality with objective tasks.

Research Program

The first component builds the pipeline space. For Camelyon16 and Camelyon17-WILDS, we take the omitted steps Fell et al. identified: stain normalization template (including none), color-space conversion, tissue-background threshold (Otsu versus several fixed cutoffs), and patch edge-sampling and labeling rules. We enumerate a small grid of configurations, each consistent with the published text of the replicated methods. To test whether the finding holds outside pathology, we build an analogous space for BRATS2020 brain MRI. That space covers intensity normalization scheme, skull-stripping tool, and resampling resolution.

The second component is the model and the sampling estimator. For segmentation, we train a conditional diffusion segmenter in the style of Wolleb et al. under one hidden reference pipeline. For Camelyon17-WILDS patch classification, we additionally use a deep ensemble. At test time, we draw $K$ pipelines and $S$ seeds per pipeline and compute both variance terms. Because each diffusion sample is costly, we use shortened sampling schedules. We also report how the estimate of each term stabilizes as $K$ and $S$ grow. Wolleb et al. report that Dice saturates near 20 seeds, which bounds $S$.

The third component is an objective human study. Participants are pathology trainees and laboratory technicians. They first complete a short performance-based technical-knowledge task rather than a Likert survey. They then triage model outputs under three display conditions: seed-variance map only, total-variance map, and the decomposed maps with plain-language labels ("model unsure" versus "depends on slide preparation"). For each case they choose among accepting the output, re-preparing the slide, or escalating to an expert.

Baselines

The naive reference point is the model with no uncertainty, where each case is accepted or rejected by maximum softmax probability or mean foreground probability. This is the "naive" metric that caused the triage trap in the mammography study.

The strongest existing uncertainty baseline is the diffusion implicit ensemble's seed variance under the adopter's single pipeline (Diffusion Models for Implicit Image Segmentation Ensembles). It reached Dice 0.881 on BRATS2020 with five samples, against 0.866 for one sample and 0.891 for nnU-Net. The closest comparable number on how well uncertainty tracks error is a Pearson correlation of up to 0.68 between generative standard-deviation maps and voxel-wise error in MRI synthesis (Conditional Generative Models for Contrast-Enhanced Synthesis of T1w and T1 Maps in Brain MRI). We found no reported result on Lacuna for diffusion uncertainty-to-error correlation on Camelyon.

The strongest disentangling competitor is Hyper-Diffusion. Its epistemic term might, in principle, already flag inputs prepared unlike the training data. That paper reports ERA5 PSNR 33.15 dB and SSIM 0.95 at 3% training overhead. We found no reported result on Lacuna for HyperDM on histopathology or on preprocessing shift.

The ablation that isolates our key ingredient is generic test-time augmentation. It uses random stain and color jitter of matched magnitude in place of the structured discrete pipeline choices. This separates the value of sampling "choices the paper could have made" from the value of sampling perturbations in general.

The upper reference is an oracle that knows the reference pipeline, which approximates perfect documentation.

All methods use the same trained weights where the method allows it. HyperDM and the ensemble necessarily have their own training runs, which we match in total GPU-hours. All methods get the same test slides, the same total number of forward samples per case, and the same hyperparameter search budget on a validation hospital. Our method beats the strongest baseline if two conditions hold. First, its specification term must predict per-pixel replication error with a rank correlation at least 0.1 higher than seed variance or HyperDM's epistemic term. Second, rejecting by it must yield a lower hospital-level disparity (the mammography paper's pairwise disparity metric δ, applied to hospitals) at 10% and 25% rejection.

Why not just…?

Why not just enforce the reproducibility checklist and release code? For new papers this is the correct fix, and we say so. It does nothing for already-published models, whose authors are often unreachable. It also does not help a deployment site that cannot reproduce the original pipeline, for example because its slide scanner differs. The oracle condition measures exactly how much documentation would recover. The gap between the oracle and the adopter's single-pipeline performance is the quantity our method tries to expose when documentation is absent.

Why not just use epistemic uncertainty from HyperDM or FLARE? These are the real threat. A pipeline mismatch shifts the input, and epistemic uncertainty is meant to rise on shifted inputs. There are two reasons to doubt it will suffice. First, FLARE shows that cheap epistemic approximations badly underestimate doubt and were validated only on low-dimensional data. Second, a mismatched pipeline can produce inputs that look in-distribution but carry different labels, for example a tissue threshold that silently drops tumor edges. The head-to-head correlation test settles this. If HyperDM's epistemic term matches ours within 0.05 rank correlation, we will report that result and fold our method into an evaluation protocol for epistemic methods.

Why not just use test-time augmentation? Color jitter covers stain variation but not discrete decisions such as tissue thresholds or patch labeling rules. Those decisions drove the largest drops in Fell et al. The matched-magnitude augmentation ablation tests this directly. If generic augmentation performs as well, the contribution shrinks to the empirical finding that such augmentation measures specification sensitivity, and we would say so.

Why not just train with preprocessing augmentation so the model becomes invariant? WILDS suggests that robustness training rarely beats plain training under real shifts. More importantly, our method measures sensitivity rather than competing with robustness training. We will apply the audit to an augmentation-trained model as well. If its specification term collapses, that confirms the measure tracks real invariance.

Proposed Experiments and Decisive Checks

The central experiment holds out the reference pipeline. For each of about ten adopter pipelines, we compute ground-truth replication error, defined as the difference between the prediction under the adopter pipeline and under the reference pipeline. We then ask which uncertainty signal, computed without knowing the reference, best predicts that error per pixel and per slide. We run this on Camelyon16 metastasis segmentation, Camelyon17-WILDS patch classification across hospitals, and BRATS2020 segmentation.

The first decisive check is falsifiable. The idea fails if the specification term is small relative to seed variance, or if it adds no predictive power beyond seed variance and HyperDM epistemic uncertainty.

The second check concerns triage. On Camelyon17-WILDS, where plain training already falls from about 93% in-distribution to about 70% out-of-distribution accuracy, we test whether rejecting by the specification term narrows the between-hospital gap. We compare it against rejecting by total variance and by naive confidence. This is the triage-trap test carried over to preprocessing.

The third check is the human study. Its preregistered hypothesis is that decomposed, action-labeled maps raise the rate of correct "re-prepare" decisions more for participants with low objective technical knowledge than for those with high knowledge. It also predicts that total-variance maps alone raise escalation without improving accuracy. That second prediction is what Fan et al.'s null direct path would lead one to expect.

Across all three checks we report the sampling cost in GPU-minutes per slide, so that adopters can judge whether the audit is affordable.