Refresh the Map or Ask the Expert? Out-of-Sample Manifold Checks for Label-Scarce Pathology

When a model trained at one hospital is deployed at another, a pathologist can afford to label only a handful of new tissue patches. Two different things can be wrong at the new site. The new patches may lie in regions the old data already describes but never labeled. Or they may come from a different stain or scanner and lie off the old data's geometry altogether. The first problem needs expert labels. The second can largely be fixed for free, by folding the unlabeled new patches into the model of the data's shape. Current active learning methods do not separate these cases, so they spend scarce expert time on both.

We propose to separate them with a tool from classical manifold learning. A manifold method summarizes a dataset by a few smooth coordinate functions, the eigenfunctions of a neighborhood graph. The out-of-sample extension (the Nyström formula) evaluates those coordinates at a new point without recomputing anything. The core claim is that the disagreement between this cheap extension and what the graph would become if the point were added is a label-free signal of "the map is wrong here." When the signal is high, the system rebuilds the graph using unlabeled target data at no expert cost. Only after that rebuild does it spend expert labels where a manifold-aware Gaussian process is uncertain. We test this on the hospital-shift split of Camelyon17-WILDS and on the colorectal histology dataset NCT-CRC-HE-100K, using frozen foundation-model features.

What exists and what is missing

The method rests on two earlier works. Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering showed that spectral embeddings are eigenfunctions of a data-dependent kernel, so a new point $x$ can be placed on embedding dimension $l$ with

$$f_l(x) = \frac{1}{\lambda_l}\sum_{j=1}^{N} P(x, x_j)\, v_l(j),$$

where $x_j$ are the $N$ pool points, $P(x,x_j)$ is the row-normalized affinity between $x$ and its graph neighbors, and $(\lambda_l, v_l)$ is the $l$-th eigenpair of the pool's normalized graph. That paper reports that, for in-distribution points, the extension's error is comparable to the variability caused by swapping 1–4% of the training points. It also warns that the extension assumes new points do not alter the graph. A point falling between two folds of the data violates this assumption. That warning, read as a measurable quantity rather than a caveat, is our starting point.

Implicit Manifold Gaussian Process Regression (IMGP) supplies the predictive model. It builds a sparse nearest-neighbor graph over labeled and unlabeled points and uses its Laplacian to approximate the manifold's natural smoothness operator. It extends the eigenvectors with Nyström to obtain a Gaussian process (GP) whose uncertainty follows the data's shape. A "bump" term makes the GP revert to an ordinary Euclidean prior far from the data. IMGP reports a negative log-likelihood of −2.48 versus −0.80 for a Euclidean GP on UCI CT slices. It is evaluated only as a passive regressor on fixed splits.

The closest active-learning work treats manifold structure statically. MEAL fits UMAP on the whole pool, filters by entropy, and clusters in two dimensions. It reports 0.596 mIoU versus 0.528 for entropy on CamVid, but it has no out-of-sample operator and no notion of new-site data. Xu et al.'s Gershgorin landmarking and its follow-up pick landmarks that keep a fixed alignment matrix well conditioned. These are closed-pool methods on raw pixels. Manifold-preserving Trajectory Sampling motivates our concern: uncertainty sampling drifts the labeled set away from the data distribution. It corrects this drift with a global MMD penalty, a statistical distance between the labeled and unlabeled feature distributions, rather than with explicit geometry. MedCAL-Bench constrains our design choices. It finds DINOv2 features the best backbone for cold-start selection, and it finds density-aware selectors (ALPS, RepDiv) beating Coreset. We therefore use DINOv2 features, and the density-aware selectors become required baselines.

The obvious combination is "use IMGP's variance as an acquisition function," and it is not new in a way that matters. For fixed hyperparameters, GP variance does not depend on labels. Variance-based querying is therefore a geometric coverage rule, close in spirit to the graph-coverage selectors MedCAL-Bench already benchmarks. Our contribution is the out-of-sample consistency check that decides when that geometry can be trusted under site shift. The variance rule is the second stage, not the novelty.

Core Claim

Insert a candidate target point $x$ into the pool graph. Let $\Delta W$ be the small change this makes to the graph weights: the $k$ new edges plus the renormalization of the affected neighbors. First-order eigenvector perturbation gives

$$r(x) = \Big(\sum_{l=1}^{L}\Big\|\sum_{m\neq l}\frac{v_m^\top \Delta W\, v_l}{\lambda_l-\lambda_m}\, v_m\Big\|^2\Big)^{1/2},$$

where $L$ is the number of retained eigenfunctions. This quantity measures how much the coordinate system itself would move if $x$ were admitted. It costs roughly $O(kL^2)$ per point and needs no labels. In-distribution, the Bengio et al. perturbation result predicts that $r(x)$ stays at the scale of resampling noise. For a patch from a new stain, or one bridging two tissue types, $r(x)$ should be large.

The policy is two-stage. First, accumulate high-residual unlabeled target points and refresh the graph and its eigenfunctions, using sparse Lanczos, only when their mass crosses a threshold. Second, spend each expert label on the point of maximal IMGP variance under the current graph. Hyperparameters, including the threshold, are tuned only on source hospitals. This matters because the file identifies unlabeled-target tuning as an open problem.

The prediction is specific. At matched expert budgets, residual-triggered refreshing should beat both never refreshing and querying directly on residual. It should also roughly match always refreshing at a fraction of the eigensolver calls.

Research Program

We freeze a DINOv2 encoder and extract patch features for Camelyon17-WILDS and NCT-CRC-HE-100K. On the source hospitals we build a $k$-nearest-neighbor graph with the random-walk normalization from IMGP and compute $L$ eigenfunctions. IMGP's regression likelihood is applied to the binary tumor label for Camelyon17, and to one-vs-rest targets for the nine NCT classes. The predicted scores are then calibrated with a logistic map fit on source data. Target-hospital patches arrive as a stream in randomized order. The system observes each patch's features, scores $r(x)$, buffers it, and periodically refreshes. When an expert query is due, it labels the point of highest variance.

The analysis has three parts. The first is a shift-detection study: does $r(x)$ separate target-hospital patches from held-out source patches better than simple distance-to-nearest-neighbor novelty? The second is a decomposition of accuracy gains into the part due to refreshing with unlabeled data and the part due to labels. The third is a failure analysis on patches where $r(x)$ is high but refreshing does not reduce it. These are the cases where we expect the short-circuiting geometry Bengio et al. describe. The CT-slices regression task serves as a sanity check that our IMGP implementation reproduces the reported behavior before any active learning is layered on.

Baselines

The naive reference is random expert labeling of target patches on the same frozen features and the same IMGP model, with no refresh. It shows how much any selection policy buys over chance.

Two baselines bracket our design. "Never refresh" uses pure Nyström extension with variance querying, which is the obvious IMGP-for-active-learning idea. "Always refresh" recomputes the graph after every arrival, which is the transductive upper bound. Both run with identical models and budgets. They isolate whether the residual trigger matters at all.

The strongest selection baselines come from MedCAL-Bench's top performers, ALPS, RepDiv and ProbCover, plus TypiClust and Coreset. We rerun all of them on the same DINOv2 features, recomputed on the pooled source and target features each round, and feed their selections to the same IMGP classifier and to a linear probe. Each method gets the same expert budgets (10, 25, 50 and 100 target labels) and the same tuning budget on source-only validation. Entropy sampling and the MEAL-style UMAP-plus-K-Means++ pipeline are included as uncertainty and manifold-heuristic references.

The strongest existing number on this split comes from a different regime. Extending the WILDS Benchmark for Unsupervised Adaptation reports Noisy Student self-training reaching 86.7% out-of-distribution accuracy on Camelyon17-WILDS, against 70.8% for ERM, using unlabeled target data and no target labels. That system trains a full network. We will also run a frozen-feature self-training baseline on the same DINOv2 features so that the comparison is not confounded by the backbone.

We count our method as beating the field under two conditions. It must exceed the best MedCAL-Bench selector by a margin outside seed variability at budgets of 25 labels or fewer. It must also exceed 86.7% target accuracy with at most 50 expert labels.

On NCT we do not have a matched hospital split. There, the reference is MyriadAL, which reports 87.1% accuracy and 88.0% macro F1 at 10 labels per class, and 95.9% accuracy at a 5% budget against 96.16% with full labels. We regard matching MyriadAL at 10 labels per class with a frozen encoder as success. The NCT experiment tests the variance stage in a cold-start, in-distribution setting where refreshing should rarely trigger, which is itself a falsifiable prediction.

Why not just…?

Why not just recompute the graph every round? For a few hundred thousand patch features, recomputing a sparse kNN graph and a few dozen eigenvectors is affordable offline. This is the real threat. If "always refresh" is cheap and equals our method, the residual contributes only speed. The "always refresh" baseline settles this directly. Our claim survives only if residual-triggered refreshing is statistically indistinguishable in accuracy at a much smaller number of eigensolver calls, in a streaming setting where labeling decisions must be made per item, or if the residual additionally improves which points are buffered. We will report both cost and accuracy and will not claim more than the data supports.

Why not just run ProbCover or TypiClust on DINOv2 features? These coverage and typicality rules already select diverse, dense-region points. MedCAL-Bench shows density-aware selectors beat Coreset, and IMGP variance with fixed hyperparameters is itself a coverage criterion. The honest difference is that the manifold variance measures coverage along the data's geometry rather than within Euclidean balls, and our pipeline knows when that geometry is stale. If ProbCover on pooled features matches us, then the Laplacian geometry is not earning its place on foundation-model features. The matched-budget comparison will show this.

Why not just use distance to the nearest labeled point as the novelty signal? Distance novelty is cheaper and catches far-off-manifold points. IMGP's bump term already makes variance large there. What distance cannot capture is a point that is close in feature space but bridges two tissue clusters, the short-circuit case. Such a point changes the eigenfunctions without being far from anything. The shift-detection study compares $r(x)$ against nearest-neighbor distance, using area under the curve for separating target from source patches. A tie there would mean $r(x)$ adds nothing beyond novelty, and we would drop it.

Why not just self-train with unlabeled target data and skip experts? Noisy Student's 86.7% shows unlabeled target data alone recovers most of the gap. Our refresh stage is in fact the geometric analogue of exploiting that data. The question is whether a few expert labels, placed by geometry, add beyond it. The frozen-feature self-training baseline and the decomposition analysis answer that question.

Proposed Experiments

The decisive checks are four. First, on held-out source patches, $r(x)$ should sit at the 1–4% resampling-noise scale predicted by Bengio et al., and it should rise on target-hospital patches. If it does not separate them better than nearest-neighbor distance, the central mechanism fails. Second, plotted as target accuracy against expert labels on Camelyon17-WILDS over five seeds and three target-hospital orders, the residual-triggered curve should dominate never-refresh, should track always-refresh at lower cost, and should exceed the MedCAL-Bench selectors at budgets of 25 labels or fewer. Third, at 50 labels the method should pass the 86.7% Noisy Student reference with a frozen backbone. Fourth, on NCT-CRC-HE-100K, refresh should almost never trigger, and the variance stage alone should approach MyriadAL's 87.1% at 10 labels per class. Frequent triggering there would indicate the residual is responding to noise rather than to shift.