Suppose two hospitals train the same diagnostic model, with the same code and the same random seed, each on its own patients. They will usually end up with different classifiers. Some patients will receive opposite calls depending on which hospital's model reads their slide. Learning theory has a precise remedy for this. An algorithm is replicable if, given a random seed that both parties share, it returns the exact same hypothesis on two independent datasets with high probability. The standard way to get there is to round the learned parameters onto a grid whose position is set by the shared seed. The cost is known: roughly $1/\rho^2$ more samples to reach failure probability $\rho$. That cost was derived for the case where both datasets come from one distribution. Hospitals do not satisfy that assumption. Different scanners and staining labs give each hospital a different distribution of images.
This note argues that under such domain shift the cost of replicability changes kind. Within one distribution, it is paid in samples. Across scanners, it is paid in accuracy, and more data does not buy that accuracy back. We propose to test this prediction directly. The test uses a dataset in which the same tissue was digitized on five different scanners, which lets us separate disagreement caused by sampling from disagreement caused by hardware. We also release a fixed evaluation harness that measures how often independently trained pathology models agree, rather than only how accurate each one is.
Core Claim
Take a learned head that sits on top of a frozen image encoder. Round its $d$ parameters onto a grid of width $w$, with the grid's offset drawn uniformly from the shared seed. Two runs then disagree only if a grid line falls between their unrounded estimates $\hat\theta_1$ and $\hat\theta_2$. For each coordinate, the probability of that is $|\hat\theta_{1,j}-\hat\theta_{2,j}|/w$, which gives
$$\Pr[h_1 \neq h_2] \;\le\; \frac{1}{w}\sum_{j=1}^{d} \mathbb{E}\,|\hat\theta_{1,j}-\hat\theta_{2,j}| \;\approx\; \frac{d}{w}\left(\frac{c\,\sigma}{\sqrt{n}} + \bar\Delta\right).$$
The symbols are as follows. $n$ is the number of independent tissue samples per run. $\sigma$ is the per-sample spread of the estimator, and $c$ is a constant. $\bar\Delta$ is the average per-coordinate gap between the best head for scanner A and the best head for scanner B. The sampling term $\sigma/\sqrt{n}$ comes from ordinary concentration of an empirical estimate.
When $\bar\Delta = 0$, we can shrink $w$ to protect accuracy and pay with $n$. This is the familiar $1/\rho^2$ tax analyzed in Reproducibility in Learning and shown to be essentially unavoidable in Stability is Stable. When $\bar\Delta > 0$, reaching failure probability $\rho$ requires $w \gtrsim d\bar\Delta/\rho$ no matter how large $n$ is. Coarse rounding costs accuracy, so cross-site replicability has an accuracy floor whose height is set by the size of the shift.
This bound is simple, but it makes testable predictions that no existing pathology benchmark checks:
- Disagreement between models trained on the same scanner should fall toward zero as cohorts grow.
- Disagreement between models trained on different scanners should level off.
- The level-off point should track a measured $\bar\Delta$.
- Anything that shrinks $\bar\Delta$, such as stain augmentation, should lower the accuracy floor in proportion.
Research Program
The method is deliberately modest, because the open problem noted throughout the replicability literature is that rounding does not transfer to overparameterized networks. We therefore never round network weights. We keep a frozen ImageNet-pretrained ConvNeXt encoder, chosen because From Modern CNNs to Vision Transformers found modern CNNs the most robust family under histopathology shift. Because the encoder never sees the training cohort, it contributes no sample-dependent variance.
On top of the encoder we fit two small heads. The first is a tile-level logistic head. The second is a slide-level head over mean-pooled tile features. Both work in a low-dimensional space produced by a projection drawn from the shared seed. We borrow this projection-then-round recipe from Replicable Learning of Large-Margin Halfspaces, which showed it avoids dependence on dimension. We also borrow from Replicable Clustering, which used data-independent projections before building its rounded summaries but validated them only on two-dimensional synthetic data.
We train each head with plain gradient descent, not accelerated methods. Reproducibility in Optimization shows that acceleration amplifies early sampling noise. We then round the head's weights on the seed-shifted grid. The clinical operating threshold is selected with the replicable median and quantile procedure of Reproducibility in Learning. The threshold is where two otherwise identical models most often split patients, so it needs the same treatment as the weights.
The second deliverable is an evaluation harness. It follows the approach of NerfBaselines, which found that most published methods in its field no longer installed and that small preprocessing differences reordered leaderboards. The harness fixes the container, tile extraction, the encoder checkpoint, the splits, and the protocol for sampling run pairs. Its main outputs are three numbers:
- Exact-hypothesis agreement ($1-\hat\rho$) across pairs of independently trained runs.
- Patient-level prediction disagreement on a fixed held-out set.
- Discrimination, reported as AUC and balanced accuracy.
All three are reported as functions of $n$ and $w$.
The main dataset is the Multi-Scanner Canine Cutaneous SCC dataset. It contains 44 tissue samples, each scanned on five systems, with registered annotations. Its authors used it only to show that a segmentation UNet loses up to 0.38 class-averaged IoU across scanners. We use it for something its design uniquely allows, namely three kinds of run pairs:
- Same scanner, disjoint tissue. Disagreement here is sampling only.
- Different scanners, identical tissue. Disagreement here is shift only, with the $\sigma/\sqrt n$ term removed by construction.
- Different scanners, disjoint tissue. Both sources act together.
Tumor-versus-non-tumor tile labels come from the transferred polygons. Resampling is done at the level of tissue samples, not tiles, because tiles from one sample are correlated. With only 44 samples, $n$ is small. We treat that as the realistic regime and report bootstrap intervals.
Two further settings complement it. Camelyon17 supplies natural multi-center shift across five hospitals. The synthetic bags from Reproducibility in Multiple Instance Learning let us dial the shift continuously by moving the mean of the poisoned negative features. That gives a clean dose-response curve for $\bar\Delta$.
Baselines
The naive reference point is a standard logistic head or attention-based multiple-instance model, trained independently at each site with fixed software seeds and deterministic GPU kernels. This is current practice. It fixes the execution path but not the sample, so it shows how much disagreement exists before we intervene. It uses the same encoder, splits, and tuning budget as everything else.
The strongest competitor is variance reduction without rounding. Each site trains a bagged ensemble of heads on bootstrap resamples of its tissue samples, with heavy L2 regularization and early-stopped gradient descent. Ensemble size is matched to the proposal's total compute, and hyperparameters are tuned on a site-internal validation split with the same number of configurations. This competitor is dangerous because averaging shrinks $\sigma$ directly and may push prediction disagreement low enough that exact identity is never needed.
To beat it, rounding must reach lower patient-level disagreement at equal AUC on cross-scanner pairs, or equal disagreement at higher AUC. On published numbers, there is no reported result found on Lacuna for cross-site prediction agreement of any pathology model. The nearest anchors are accuracy figures: in-domain mIoU of 0.82–0.86 and a cross-scanner drop of up to 0.38 for a UNet (Wilm et al.), and test AUC of 0.000 for TransMIL, Attention-MIL and GNN-MIL on the poisoned unit test (Reproducibility in MIL). These are context, not targets for our metric.
The ablation that isolates the key ingredient is deterministic rounding. It uses the same grid width with a fixed, unshifted grid. The boundary-problem argument predicts that this behaves no better than no rounding whenever estimates sit near a grid line, so the gap between shifted and fixed grids at matched $w$ measures what the shared seed contributes.
The final baseline is HistAuGAN augmentation (Structure-Preserving Multi-Domain Stain Color Augmentation), applied before feature extraction, both alone and combined with rounding. It is reported to raise mean local diversity on Camelyon17 from 0.20 to 0.85 and to improve PR-AUC across centers, though per-center values appear only in a figure. We treat it as a way to shrink $\bar\Delta$, not only as a rival.
Why not just…?
Why not just fix the seed? Seeding makes one run repeatable on one dataset. It does nothing when the two hospitals' data differ, which is exactly what Reproducibility in Learning and Stability is Stable separate formally. The naive baseline measures this gap directly. If seeded independent training already showed near-zero disagreement on same-tissue cross-scanner pairs, the problem would not exist, and we would report that.
Why not just ensemble and regularize? This is the real threat. Within one distribution, bagging may match rounding on prediction agreement at lower accuracy cost, because it attacks $\sigma$ without discretizing anything. Under shift, however, both sites' ensembles converge to their own population heads, which still differ by $\bar\Delta$. Ensembling should therefore also level off on cross-scanner pairs. If it levels off at the same height as rounding with better AUC, then exact identity offers no practical advantage beyond auditability, meaning two sites can verify they hold the same model by comparing parameters. We would state the contribution that way, and the measured floor would remain the main finding.
Why not just remove the shift with stain normalization or generative augmentation? Shrinking $\bar\Delta$ is exactly what lowers the floor, so we test these methods inside the proposal rather than against it. Two cautions apply. CycleGAN recoloring cost up to 18% even within one distribution in Springenberg et al.. And A Sober Look at Disentanglement found that generative disentanglement depends heavily on the random seed. A GAN trained separately at each site may therefore add a disagreement source of its own, which the harness will measure. Simpler schemes such as inter-slide MixUp hurt accuracy in MixUp-MIL, so we include only intra-slide mixing as a cheap way to reduce shift.
Proposed Experiments and Decisive Checks
Floor check. The first check sweeps $n$ over the canine tissue samples at fixed $w$ and plots exact-hypothesis disagreement for the three pair types. The core claim is confirmed if same-scanner disagreement keeps falling while same-tissue cross-scanner disagreement levels off. It is refuted if both decay together. We estimate $\bar\Delta$ independently, from heads trained on all tissue on each scanner, and check whether the observed floor matches $d\bar\Delta/w$ within bootstrap error.
Shared-offset check. The second check compares the shifted grid with the fixed grid across $w$. If the fixed grid matches the shifted one, the replicability machinery is doing nothing beyond plain quantization.
Dose-response check. The third check uses the synthetic unit-test bags, moving the poisoned-feature mean in steps. Agreement between a model trained on clean bags and one trained on shifted bags should fall linearly in the shift at fixed $w$, as the bound predicts. The same test reports whether rounding changes the unit-test pass/fail outcomes. We expect it does not, since rounding addresses agreement between runs, not causal correctness.
Shift-reduction check. The fourth check repeats the floor experiment with HistAuGAN and with intra-slide mixing. Each should lower $\bar\Delta$ and the floor together. Camelyon17 leave-one-center-out pairs provide the multi-center version.
Outputs. Throughout, we report the trade-off curve of AUC against cross-site disagreement for every method at matched compute. Even if the rounding method itself turns out unremarkable, this frontier is what the field currently lacks.
References
- Reproducibility in learning
- Stability Is Stable: Connections between Replicability, Privacy, and Adaptive Generalization
- From modern CNNs to vision transformers: Assessing the performance, robustness, and classification strategies of deep learning models in histopathology
- Replicable Learning of Large-Margin Halfspaces
- Replicable Clustering
- Reproducibility in Optimization: Theoretical Framework and Limits
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- Multi-scanner Canine Cutaneous Squamous Cell Carcinoma Histopathology Dataset
- Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests
- Structure-Preserving Multi-domain Stain Color Augmentation Using Style-Transfer with Disentangled Representations
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- MixUp-MIL: Novel Data Augmentation for Multiple Instance Learning and a Study on Thyroid Cancer Diagnosis