Predicting a Frozen Classifier's Calibration at a New Hospital Before Seeing Any Labels

When a hospital deploys a certified diagnostic model, the feature extractor usually cannot be changed, and often only the small decision layer on top of it can be refit. When the model moves to a new site, with a different scanner, stain protocol or patient mix, the model's accuracy drops. Worse, its stated confidence stops meaning anything, and the new site typically has no labeled data with which to measure either problem. We propose to treat the refittable decision layer as what it mathematically is: a regularized logistic regression on high-dimensional features. For that object, statistical physics offers exact formulas that predict test error and calibration from a few summary numbers. We extend those formulas to the case where the test features come from a different distribution than the training features. We then compute the needed summary numbers from the source hospital's labeled data and the new hospital's unlabeled features. The result is a label-free forecast of out-of-hospital accuracy and calibration error, plus a principled choice of regularization strength for each site.

The core idea fits in two sentences. A logistic "probe" (a linear classifier trained on frozen features) has an output distribution on new data that is, in high dimensions, governed by three overlap numbers. One of them, the spread of its own scores on the new site, is directly observable without labels. The other two can be predicted by an exact asymptotic theory once that theory is extended from same-distribution testing to covariate shift. If the prediction holds on real pathology features across hospitals, a locked clinical model gets a calibration audit and a regularization knob for each site without any target labels and without touching the backbone.

Core Claim

The relevant theory is state evolution: a low-dimensional deterministic recursion that tracks the exact behavior of certain iterative estimators on large random problems, and by extension the regularized estimators they converge to. The canonical algorithm is approximate message passing, whose correction term makes the iteration's noise effectively Gaussian and independent of the data (Montanari et al.). Using the generalized version of this algorithm, Clarté et al. derived the joint distribution of the true class probability, the Bayes-optimal prediction and the prediction of $\ell_2$-regularized logistic regression. They showed that the latter is systematically overconfident when the sample-to-dimension ratio $\alpha = n/d$ is of order one, and that the regularization strength $\lambda$ acts as a scalar knob moving the classifier between over- and under-confidence. Their analysis assumes training and test inputs are identically distributed. Tripuraneni et al. handle the complementary case. They give exact out-of-distribution error for random-feature ridge regression under covariate shift and show that shift "hardness" is set by how the test covariance loads on directions that were weak in training. Their analysis covers squared loss only and does not address calibration. Neither paper addresses classification calibration under shift, and neither is tested on medical data.

Our claim is concrete. Take source features $x \sim \mathcal{N}(0,\Sigma_s)$ and target features $x \sim \mathcal{N}(0,\Sigma_t)$, with labels generated from the same conditional rule $P(y=1\mid x)=\Phi(w_*^\top x/\sqrt{d})$ at both sites. A logistic probe $\hat w$ is trained on $n$ source samples with penalty $\lambda$. In the proportional limit, the probe's target-site behavior is fully determined by the target overlaps

$$ q_t = \tfrac{1}{d}\hat w^\top \Sigma_t \hat w,\qquad m_t = \tfrac{1}{d}\hat w^\top \Sigma_t w_*,\qquad \rho_t = \tfrac{1}{d} w_*^\top \Sigma_t w_* . $$

This holds because the pair (true logit, predicted logit) is jointly Gaussian with these second moments. Here $\hat w$ is the trained probe and $w_*$ the unknown true direction. Target accuracy and target expected calibration error (ECE) are explicit two-dimensional Gaussian integrals in $(q_t, m_t, \rho_t)$.

The quantity $q_t$ can be read directly from unlabeled target features. The quantities $m_t$ and $\rho_t$ involve $w_*$ and so cannot be read off directly. A state-evolution analysis of the probe expresses $\hat w$ as a deterministic, resolvent-type transform of $\Sigma_s w_*$ plus an effective Gaussian noise whose variance is the state-evolution fixed point. In the source-only setting this is the structure Clarté et al. exploit. Substituting that representation turns $m_t$ and $\rho_t$ into trace functionals of $\Sigma_s$ and $\Sigma_t$ applied to the source-estimable vector $\Sigma_s w_* \propto \mathbb{E}_s[y\,x]$. The new work is deriving these functionals and their high-dimensional debiasing. The practical payoff is that $\lambda$, and optionally a scalar temperature, can be chosen per target site to minimize predicted target ECE, with no target labels.

This proposal also takes a position on a tension in the literature. Tripuraneni et al. find that overparameterization monotonically helps under covariate shift for ridge regression, while Clarté et al. find catastrophic overconfidence for weakly regularized classification below the separability threshold. We predict that under shift, the out-of-hospital ECE-versus-$\alpha$ curve is non-monotone. We also predict that the ECE-optimal $\lambda$ at the target is larger than at the source whenever the shift is "hard" in Tripuraneni's spectral sense. Both predictions are falsifiable.

Role of Each Direction

State evolution is the mechanism. Decoupled two-stage learning is the structure that makes the mechanism applicable, because a frozen backbone reduces the problem to a linear readout. Medical domain shift is the regime and the stakes. The two-stage motif is long established in clinical imaging. Havaei et al. trained filters on balanced patches and then refit only the last layer to the true lesion prevalence. Freezing features before fitting a head is also known to preserve representation geometry (Yu et al.; Vaze et al.). These works motivate the setup but offer no way to predict how the refit head behaves at an unseen site.

The regulatory motivation comes from Norcliffe et al., who keep a certified ICU predictor locked and adapt its inputs instead. Their method needs labeled target data and gives no performance forecast. Ours needs no target labels and produces a forecast, but it adapts only the head and does not fix representation failure.

Ouyang et al. recalibrate a frozen MRI segmenter with a learned spatial temperature map built from test-time perturbations and a shape autoencoder. They report ECE falling from 14.48% to 12.78% under synthetic artifacts. That work is the heuristic counterpart of what we propose analytically, and it constrains us: if spatial anatomical context is truly necessary for calibration, a scalar theory will underperform on segmentation. We therefore make patch-level classification our primary testbed and treat segmentation as a stress test.

Research Program

The theoretical component extends the Gaussian-covariate probit analysis of Clarté et al. to separate train and test covariances. It derives the fixed-point equations for the effective regularization and noise variance of the logistic probe under anisotropic $\Sigma_s$, and then the target overlaps $m_t$ and $\rho_t$ as functionals of $\Sigma_s$, $\Sigma_t$ and $\Sigma_s w_*$. Where the i.i.d.-Gaussian assumption is too strong, we follow the rotationally invariant generalizations of Gerbelot et al. and vector approximate message passing. These tolerate the ill-conditioned spectra that real deep features have. We also derive a class-conditional Gaussian-mixture variant, since pathology features may look more like two clusters than like a single Gaussian with a probit label. The model whose predictions better match in-distribution behavior will be used downstream.

The estimation component turns the formulas into a procedure. From labeled source features we estimate $\Sigma_s$ and $\mathbb{E}_s[y x]$. From unlabeled target features we estimate $\Sigma_t$ and $q_t$ for any candidate $\lambda$. We then plug in debiased trace estimates to get $m_t$ and $\rho_t$. The main technical risk sits here. The naive plug-in $\Sigma_t\Sigma_s^{-1}\mathbb{E}_s[yx]$ is badly biased when $d$ is comparable to $n$, and the random-matrix corrections that state evolution supplies are exactly what we expect to fix it.

The empirical component uses frozen backbones. The first is an ImageNet-pretrained ResNet-50, as a stand-in for a generic locked backbone. The second is DINOv2, as a self-supervised foundation backbone. The third is a backbone trained end-to-end on the source hospitals and then locked, mimicking a certified device whose head is refit. On each backbone we train logistic probes while subsampling the source labels to sweep $\alpha$ across the separability regime and beyond. Before touching real features, we validate the theory on synthetic Gaussian data with controlled covariance shifts, where predictions should match to finite-size error.

Baselines

The naive reference is a probe trained on source hospitals with $\lambda$ and temperature chosen by source-validation ECE, then deployed unchanged. This is what a locked device does today. Its forecast of target calibration is simply its source-validation ECE. Beating it means lower realized target ECE after our label-free $\lambda$ selection, and a more accurate forecast of target ECE and accuracy.

The strongest label-free competitor is confidence-based forecasting. Here the probe's average maximum softmax confidence on unlabeled target features is used as the predicted target accuracy, and a temperature is chosen to minimize target prediction entropy under a prevalence constraint. It uses exactly the same inputs as our method: source labels, target features, and no target labels. It is therefore the fair comparison for whether theory adds anything beyond the observable $q_t$. We run it with the same probes, the same $\lambda$ grid, and the same compute. The file contains no reported result for this approach on Camelyon17, and no result found on Lacuna for label-free calibration forecasting on Camelyon17. We will report our own runs.

A second label-free competitor follows the accuracy-on-the-line idea formalized by Tripuraneni et al.. We fit a linear relation between in-distribution and out-of-distribution accuracy across held-out hospitals in leave-one-hospital-out fashion and use it to forecast the remaining hospital. This baseline is empirical and needs labels from other target sites, which gives it an advantage; beating it would be strong evidence for our method.

The oracle upper bound is temperature scaling and $\lambda$ selection on a labeled split of the target hospital. We count success as closing at least half of the gap between the naive reference and this oracle in realized target ECE.

For accuracy context, fully fine-tuned models on Camelyon17-WILDS drop from about 93% in-distribution to about 70% out-of-distribution under ERM according to WILDS. Zhang et al. report 90.5% OOD accuracy with targeted stain augmentation, against a 75.3% standard baseline. We do not expect a head-only method to match 90.5% accuracy. That number marks what representation-level intervention buys, and we report it so readers can see the scope of our claim, which is about calibration and forecasting rather than raw accuracy.

Two ablations isolate the key ingredient. The first replaces the state-evolution-corrected overlaps with naive plug-in estimates, testing whether the high-dimensional correction matters. The second sets $\Sigma_t=\Sigma_s$ in the formulas, which reduces our method to Clarté et al.'s in-distribution theory applied off the shelf, testing whether modeling the shift matters.

Why not just…?

Why not just temperature-scale on the target site? If a labeled target split exists, this is the right answer, and it is our oracle. The proposal exists for the common case where the new hospital has no labels at deployment time. In that case one also wants a forecast of degradation before trusting the model. Clarté et al. further show that regularization, not just a post-hoc temperature, controls high-dimensional overconfidence, so the oracle is run over both $\lambda$ and temperature.

Why not just use average confidence on the unlabeled target data as the forecast? This is the real threat. Average confidence already uses $q_t$, the one overlap that is directly observable. Our theory says it is blind to $m_t$, the alignment between the probe and the true direction under the target geometry. That quantity is precisely what separates a confidently wrong model from a confidently right one under hard shifts. If across the Camelyon17 hospital pairs the confidence baseline forecasts ECE as well as our method, the extra machinery is not justified. The planned comparison is designed to decide this, and we will report it either way.

Why not just retrain the backbone with targeted augmentations? On accuracy this is clearly better: 90.5% versus 75.3% on Camelyon17 (Zhang et al.). However, it changes a certified model and still gives no label-free calibration forecast. The two approaches are complementary. We will apply our forecast to probes on a backbone trained with targeted augmentations, to check that the theory still holds when shift is milder.

Why not just use a learned post-hoc calibrator like Ouyang et al.'s? Their module is trained on synthetic perturbations and relies on a shape prior that the authors say fails on variable anatomy. It does not apply to patch classification at all. On the segmentation stress test we compare directly. A scalar theory losing there would confirm that spatial context matters for dense prediction, and that result would be worth reporting.

Proposed Experiments

The primary benchmark is Camelyon17-WILDS (WILDS). We use leave-one-hospital-out splits over its five hospitals, and all ordered source/target pairings for finer-grained analysis. On each split we train probes on each of the three backbones at several $\alpha$ values and a grid of $\lambda$ values.

For every configuration we record realized target accuracy and ECE. We compare them against three forecasts: from our theory, from average confidence, and from accuracy-on-the-line. Forecast quality is measured by absolute error and rank correlation across configurations. We then measure realized target ECE after label-free $\lambda$ and temperature selection, relative to the naive reference and to the oracle.

The secondary benchmark is AMOS (AMOS). There, models reportedly lose over 20% of performance on MRI from unseen scanners. We probe frozen features for voxel-patch organ classification across scanner groups, as a harder, more non-linear shift.

The decisive checks come in order. First, on synthetic data the predicted target ECE must match simulation up to finite-size effects; otherwise the derivation is wrong. Second, in distribution on real features, the predicted overlaps must match measured source behavior. If Gaussian equivalence fails for deep pathology features at this stage, we stop and report that negative result, which is itself informative for the high-dimensional theory community. Third, on held-out hospitals, our forecast must beat average confidence in ECE forecast error. Without that, the method has no reason to exist.

Two further predictions test the theory specifically. The non-monotone dependence of target ECE on $\alpha$, and the upward shift of the target-optimal $\lambda$ for spectrally harder hospitals, must both appear in real data. They are predictions no baseline makes, so observing them would be the clearest evidence that the theory captures something true about locked clinical classifiers under shift.