AI models can now read a routine tissue slide, stained with hematoxylin and eosin (H&E, the cheap stain every pathology lab already uses), and flag patients who are unlikely to carry a treatable molecular alteration. Those patients can then skip the expensive, slow, tissue-consuming genetic test. Validated tools include MSIntuit for colorectal microsatellite instability (Validation of MSIntuit) and EAGLE for lung EGFR (Real-world deployment of a fine-tuned pathology foundation model). Their value rests on one number: the operating threshold below which a slide counts as "ruled out." That threshold is set once, on a small number of positive cases, and is then expected to stay safe for years while scanners, stains and patient mix change.
This proposal argues that rule-out triage creates a monitoring problem that general drift methods do not address. Ruled-out patients are never tested, so the false negatives that signal falling sensitivity are never observed. Because positives are rare, the few that are observed come almost entirely from slides that scored above the threshold. A pre-screener can therefore lose sensitivity silently, and naive re-estimation from deployment data will report that everything is fine. We propose a threshold-maintenance procedure for H&E pre-screeners with three parts. It sends a small randomized audit sample of ruled-out cases to confirmatory testing. It sets the threshold only from the score distribution of positive cases, which by construction does not change when prevalence changes. It decides how far back in time it may pool those scarce positive scores using a test computed on the fully observed, unlabeled slide stream. The result is a rule-out threshold that carries a finite-sample sensitivity guarantee over calendar time, together with an explicit accounting of how many molecular tests the guarantee costs.
Core Claim
Temporal drift and class imbalance interact in triage through label censoring. Three consequences follow, and each can be tested.
First, in rule-out triage the sensitivity of the deployed tool is not identifiable from routine deployment data. Positives that the tool misses below its threshold are never labeled. Re-fitting the threshold on observed positives is therefore biased toward optimism, and the bias grows exactly when drift pushes positive scores downward.
Second, the correct object to track is the lower tail of the positive-class score distribution, not calibrated probabilities. If $s(x)$ is the model's score and $t$ the threshold, sensitivity is $P(s(x) \ge t \mid Y=1)$. This quantity is unaffected by changes in prevalence and is moved only by changes in how positive slides look: scanners, stains, sampling, case mix within the positive class. This runs against the frequency-adjusted thresholds of Addressing Domain Shift via Imbalance-Aware Domain Adaptation, which shift the cutoff with class ratios. That approach suits accuracy, but for a sensitivity target it moves the threshold for the wrong reason.
Third, rarity makes the guarantee statistically expensive. Suppose the threshold is placed at the lowest of $n$ positive calibration scores, with no drift at all. The probability that true sensitivity is still below a target $\beta$ is $\beta^n$. This follows because the fraction of positives scoring below the minimum of $n$ exchangeable draws is Beta(1, $n$)-distributed. For $\beta = 0.95$ and the 30 MSI cases MSIntuit used, our calculation gives $0.95^{30} \approx 0.21$. Roughly 45 positives are needed before that risk falls to 10%. Pooling positives over many months is therefore unavoidable, and pooling across a drift event is what breaks the guarantee. Deciding when pooling is safe is the central technical question.
Research Program
The procedure wraps any slide-level model. Here we use attention-based multiple-instance learning over frozen pathology foundation-model tile features: the slide is cut into thousands of small tiles, each tile is embedded, and a learned weighted average produces one slide score. Retraining is treated as a separate decision. The program has three components.
The first component is audited labeling. Each slide scoring below $t$ is sent to confirmatory testing with a small fixed probability $a$, for example 2–5%. Every slide above $t$ is tested anyway as part of normal care. Positives found by audit receive weight $1/a$. This turns the censored stream into an unbiased sample of the positive-class score distribution. Clinically, the audit converts a fraction of "saved" tests back into tests, and we report that cost explicitly rather than hiding it.
The second component is an order-statistic threshold. Within a pooled window holding an effective number $n$ of positive scores, the threshold is the $k$-th smallest weighted score. Here $k$ is the largest integer satisfying
$$ P\big(\mathrm{Bin}(n, 1-\beta) \le k-1\big) \le \delta , $$
where $\beta$ is the target sensitivity and $\delta$ the tolerated probability of missing it. The rule comes directly from the Beta distribution of order statistics. We will derive and empirically check the adjustment to effective sample size that the inverse-probability weights require. When no admissible $k$ exists, the procedure does something the current tools never do: it declares that it cannot certify $\beta$ and stops ruling out.
The third component is shift-gated pooling. Two kinds of change must be told apart. In a pure prevalence change, positive slides look the same and are simply more or less common, so all past positive scores remain valid. In an appearance change, such as a new scanner or stain lot, past scores become stale. The unlabeled stream is fully observed and abundant, so we test whether the change in its score distribution from one period to the next can be explained by a change in the positive/negative mixing weight alone, holding the class-conditional score distributions fixed. We pair this with a two-sample test on the slide embeddings. Pooling extends backward until either test rejects. After a rejection, the window restarts and the procedure leans on audit positives, temporarily raising $a$, until enough post-shift positives accumulate.
This turns the finding of Evaluating Model Performance in Medical Datasets Over Time into a rule that switches automatically. That paper found that sliding windows win under fast drift and all-history training wins when data are scarce. Rare positives make pathology biomarkers the data-scarce extreme, so knowing when pooling is safe matters most there.
Baselines
The naive reference point is the current deployment practice: a threshold fixed once at time zero on a small set of positives. This is the MSIntuit protocol, which on its 600-patient blind cohort reported sensitivity of 0.96–0.98, NPV of 0.99 and 46–47% of MSS patients ruled out (Validation of MSIntuit). EAGLE reported a silent-trial AUC of 0.890 and up to 43% fewer rapid PCR tests (Real-world deployment). These are cross-sectional figures. No reported result was found on Lacuna for the longitudinal sensitivity of any H&E biomarker pre-screener, so the frozen-threshold baseline's behavior over time is itself a first measurement of this study. We run it with the same model and the same initial calibration positives as every other arm.
The strongest existing competitor combines periodic retraining on all accumulated data with recalibration on the most recent window. Pooled empirical risk minimization was the method no domain-generalization or adaptation technique beat in Evaluation of domain generalization and adaptation. Platt or isotonic recalibration is the standard repair for calibration rot noted by Guo et al.. We give it the same labels the clinic would actually have, meaning only above-threshold tested cases plus the same audit budget. It also gets the same retraining schedule and the same hyperparameter budget, and it chooses its threshold as the score achieving target sensitivity on its recalibrated outputs.
A third baseline is a static conservative margin. The threshold is lowered once so that it hits a stricter sensitivity target, for example 0.99, at time zero, and is never touched again.
Ablations isolate each ingredient. Removing the audit, so thresholds are fit on truncated positives, measures the censoring bias. Fixed sliding windows and all-history pooling, each without the gate, measure the value of shift-gated pooling. A prevalence-adjusted threshold in the style of IADA tests the claim that class-conditional thresholds are the right object. Our procedure beats the baselines if, at an equal total number of confirmatory tests including audits, it keeps the fraction of time periods with realized sensitivity below $\beta$ at or under $\delta$ while ruling out more patients than the conservative margin. The frozen and recalibrated baselines should violate the target after drift events.
Why not just…?
Why not just retrain on all history and recalibrate? Retraining improves the ranking but does not tell you where to put the threshold. Recalibration under censoring is fit on positives that already cleared the old threshold, so it cannot see the low-scoring positives that define sensitivity. Guo et al. also note that recalibration cannot restore lost discrimination. We still treat this as a genuine threat. If drift in pathology turns out to be mild enough that recalibrated thresholds rarely violate the target, the audit is wasted cost. The censored-versus-audited comparison will show this directly.
Why not just set a conservative threshold once and leave it? This is the most dangerous alternative, because it is simple and may be enough. A fixed margin buys protection against some unknown amount of drift by giving up rule-out rate permanently, and it offers no protection against a large stain or scanner change. The decisive comparison is the curve of rule-out rate against violation frequency. If the static margin matches our procedure on that curve across the drift scenarios, we will report that adaptive maintenance is unnecessary for pathology pre-screeners.
Why not just harmonize the inputs with stain normalization or ComBat? Towards Robust Foundation Models for Digital Pathology found that ComBat improved its robustness index by an average of 27.4%, but even the best models remained well below the ideal, and models still identified source hospitals with 88–98% accuracy. Harmonization reduces drift. It does not measure what drift remains, and it cannot certify sensitivity. We include ComBat-harmonized features as an upstream variant for all arms, so the two can be compared directly or combined.
Why not continual learning with imbalance correction? The methods in AIR and continual learning for toxicologic pathology improve how the model is updated, not how its threshold is checked. AIR rebalances class contributions in closed-form updates; the toxicologic-pathology work replays stored features to prevent forgetting. Both still need labels for the positives they miss. They complement this proposal rather than compete with it, and one AIR-style update arm tests whether better updating reduces how often our gate fires.
Proposed Experiments
We build two kinds of temporal stream. The first is semi-synthetic, which gives controlled ground truth. TCGA-COAD MSI and TCGA-BRCA ER/HER2 slides are ordered by tissue source site and by year of diagnosis where the clinical metadata provides it. Into these orderings we inject scheduled events: abrupt changes in stain and scanner appearance, prevalence swings between 5% and 30%, and changes in case mix within the positive class, such as a shift toward biopsies, which MSIntuit identifies as harder. These are crossed with PathoROB-style multi-center patches to calibrate realistic feature-space shifts. Labels are complete, so true sensitivity per period is known exactly.
The second is a real longitudinal archive: multiple years of consecutive colorectal cases with routine MSI immunohistochemistry from a partner pathology department, backtested in the rolling manner of EMDOT. Here complete labels exist because universal testing was performed. We replay deployment by hiding the labels of below-threshold cases except for the audit sample.
The metrics are realized per-period sensitivity, the fraction of periods violating $\beta$, the rule-out rate, the total confirmatory tests including audits, and NPV.
Three checks decide the proposal. The first asks whether censoring bias actually appears. On real archives, the no-audit ablation should report sensitivity above the truth after documented scanner or stain changes. If it does not, the core motivation fails. The second asks whether the shift gate fires on appearance changes and stays quiet on pure prevalence swings in the semi-synthetic streams. If it cannot tell them apart, pooling offers nothing over a fixed window. The third asks whether, at matched test budgets, the procedure holds violation frequency near $\delta$ while ruling out more patients than the static conservative margin. If it does, it shows a pre-screener can hold its sensitivity target over years while still sparing patients molecular tests.
References
- Validation of MSIntuit as an AI-based pre-screening tool for MSI detection from colorectal cancer histology slides
- Real-world deployment of a fine-tuned pathology foundation model for lung cancer biomarker detection
- Addressing Domain Shift via Imbalance-Aware Domain Adaptation in Embryo Development Assessment
- Evaluating Model Performance in Medical Datasets Over Time
- Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine
- Towards Robust Foundation Models for Digital Pathology
- AIR: Analytic Imbalance Rectifier for Continual Learning
- Towards exploring continual learning for toxicologic pathology in pharmaceutical drug discovery