A pathology model deployed at a new hospital will see two kinds of "new" images. Some come from a different lab: the same tissue, but with different dye batches, section thickness or scanner. Others show a lesion the model was never trained on. An open-set anomaly detector is a model trained on normal tissue plus a few labeled abnormal examples, and it is asked to flag anything abnormal, including kinds it has never seen. Such a detector should raise an alarm for the second kind of image and stay quiet for the first. Current open-set detectors have only been tested on pre-registered MRI and CT slices, where the first problem hardly exists. We expect that, once moved into histology, they will mostly detect "this slide came from somewhere else."
A second complication compounds the first. In grading tasks such as Barrett's esophagus, expert pathologists disagree with one another. The "normal" class the detector learns therefore has blurry edges. Borderline cases, where no pathologist is confident, are currently thrown out of training rather than modeled. We propose a detector that produces two separate outputs. The first is a novelty score, computed in a representation from which the acquisition site has been deliberately removed. The second is an ambiguity score, derived from how much a panel of per-pathologist models disagrees. We test whether this separation cuts false alarms on new sites without losing sensitivity to lesions the model has never seen. A smaller transfer study checks the same idea on needle-delivered muscle recordings, where each new electrode insertion plays the role of a new stain.
Core Claim
In biomedical open-set detection, a single anomaly score mixes three sources:
- acquisition shift, meaning stain and scanner in histology, or electrode insertion placement in intramuscular recording;
- label ambiguity, which pathologists do not agree on;
- genuine biological novelty.
Only the third should trigger the "unseen anomaly" alarm. Our claim is that factoring out acquisition shift explicitly, and routing ambiguity to its own output, lowers the false-alarm rate on normal tissue from unseen sites at a fixed sensitivity. It should do so without lowering the area under the ROC curve (AUROC) on held-out lesion types.
The new element is not a better density model. It is the evaluation regime, in which site shift, rater disagreement and unseen lesions are all present at once, together with a decomposition designed for that regime. We expect that simple stain augmentation might close much of the gap. That outcome would itself be informative, so we treat it as a primary competitor rather than a strawman.
Where This Sits in Prior Work
Three open-set detectors supply the machinery.
- DRA splits abnormality into four heads: seen anomalies, pseudo-anomalies, latent residuals and normality.
- DPDL fits Gaussian prototypes of normality through a diffusion bridge.
- MPFM maps normal data onto a multi-component Gaussian mixture with flow matching. It argues that normal biological data has sub-modes a single center cannot capture, a point that constrains us against the single-hypersphere design of Deep SAD.
All three evaluate on Brain-MRI, HeadCT and industrial images, never on stained slides. MPFM also reports that performance is sensitive to the number of mixture components. We reuse the mixture-prototype idea but change what the components are allowed to encode.
Two papers motivate the problem. Ciompi et al. show a colorectal classifier falling from about 94% internal accuracy to 50.96% on an external center. Normalization raises this to 79.66%, but a 14-point gap remains, which they attribute to thickness, blur and scanner artifacts rather than color. This is why we model acquisition shift in a learned representation rather than by color correction alone.
StainGAN translates stain styles between two labs and improves downstream AUC. It works only pairwise, however, and generative translation can hallucinate structure. An anomaly detector could mistake such hallucinated structure for a real outlier.
Grove et al. supply our method for ambiguity. They train one attention-based multiple-instance model per pathologist on the Dutch Esophageal Pathology Panel and ensemble them. This raises F1 from 72.9% (consensus-trained) to 78.2% and lowers expected calibration error from 0.183 to 0.050. They are also the source of a gap we target: they excluded the 145 "indefinite for dysplasia" cases. We keep those cases, as the natural test of whether a detector confuses ambiguity with novelty. The evaluation is ours; they were closed-set and did not model site shift.
AMIGO avoids stain sensitivity by building cell graphs. It is a closed-set survival model. We treat its cell-graph input as an alternative backbone rather than a competitor.
For time series, IMPACT handles open-set detection on scalp EEG and ECG. The needle-delivered Myomatrix arrays and thin-film intramuscular electrodes produce recordings in which each insertion lands at a slightly different depth and orientation. These papers define the electrode regime. None of them studies anomaly detection.
Research Program
The detector has three parts.
The first is an encoder whose output splits into two parts: a "biology" part $z_b$ and an "acquisition" part $z_a$.
- $z_a$ is trained to predict the site or scanner label. Where no site label exists, it predicts stain optical-density statistics.
- $z_b$ is trained so that a site classifier on it performs at chance, using an adversarial head.
- A reconstruction or contrastive term keeps $z_b$ from collapsing.
- The same DRA-style heads for seen and pseudo-anomalies operate on $z_b$.
The second part fits normality as a mixture over $z_b$. Each normal training example is weighted by how many pathologists agreed it was normal. The novelty score is
$$s(x) = -\log \sum_{k=1}^{K} \pi_k \,\mathcal{N}\big(z_b(x);\,\mu_k,\Sigma_k\big),$$
where the $\pi_k,\mu_k,\Sigma_k$ are mixture weights, means and covariances fitted with agreement-weighted maximum likelihood. Because site information has been pushed out of $z_b$, the components should capture tissue sub-types rather than lab styles. If so, the result should be less sensitive to the number of components $K$ than MPFM reports. We test this directly.
The third part is an ambiguity score. This is the entropy of the virtual-panel predictions from a Grove-style ensemble. Since training dozens of full models is the scaling problem Grove et al. flag, we compress the panel into a shared backbone with per-rater output heads. The detector reports $s(x)$ and the ambiguity score as two separate outputs rather than blending them. The clinical reading is "novel" versus "contested."
The electrode transfer uses the same design on intramuscular EMG, with "site" replaced by "insertion session." Here $z_a$ predicts session identity. Normality is modeled from motor-unit waveform windows.
Proposed Experiments
Cross-center colorectal open-set split. This uses Ciompi's setup.
- Training data comes from the Radboud rectal cancer slides and testing from the external colorectal patch set.
- Stroma, muscle, healthy mucosa, fat and lymphocytes are "normal."
- Tumor is the seen anomaly. Necrosis and mucus are held out as unseen anomalies.
- The key metric is the false-positive rate on external-center normal patches at 95% recall of unseen anomalies. We also report AUROC on unseen anomalies.
Camelyon16 site-shift stress test. A detector is trained on normal lymph node tissue from Radboud and tested on normal and metastatic tissue from Utrecht. This isolates the site-shift false alarm.
DEPP, the main multi-rater test. Non-dysplastic Barrett's is treated as normal, low-grade dysplasia as the seen anomaly, and high-grade dysplasia as held out. The 145 indefinite cases are returned as a probe set. A detector that confuses ambiguity with novelty will assign indefinite cases novelty scores as high as high-grade dysplasia. Ours should rank them high on ambiguity and intermediate on novelty. We measure this with rank correlation between each output and observed panel disagreement. We also report calibration error on the closed-set grades, so it can be compared against Grove's 0.050. If slide-level scanner or site metadata are unavailable, we cluster slides by stain vectors and use cluster identity as the site label.
Electrode transfer. This uses the Myomatrix archive of mouse and macaque recordings.
- Sessions are held out by insertion.
- Unseen anomalies are boundary perturbations generated IMPACT-style, together with real contact-degradation segments where they can be identified.
- IMPACT is the baseline. On scalp EEG it reports 82.91% AUROC on TUSZ.
- No reported result was found on Lacuna for open-set detection on needle-delivered intramuscular recordings, so this is exploratory.
Baselines
The naive reference point is a closed-set classifier on the seen classes that flags low maximum softmax probability. It is trained on the same data and backbone. Grove et al.'s consensus model, with a calibration error of 0.183, suggests these confidences are unreliable. We report this baseline to anchor how much any open-set machinery helps.
The strongest existing method is MPFM, with DRA and DPDL also run. On histology we found no reported result on Lacuna for any of them, so all numbers will be ours. The published figures are relative, for example MPFM exceeding DPDL by 3.4 AUROC points on Mastcam, and are not directly comparable.
All three baselines are run in two configurations:
- raw;
- after StainGAN or Macenko normalization. For the classification pipeline this gave 79.66% external accuracy in Ciompi et al.
Every method shares the same backbone, the same pretraining, the same number of labeled anomalies (one and ten shots), and an equal hyperparameter search budget. MPFM's $K$ is tuned on a validation site that is disjoint from the test site. Beating MPFM means two things: a lower external-site false-positive rate at matched unseen-anomaly recall, and AUROC no worse by more than one point, each averaged over seeds with confidence intervals.
Two ablations isolate our ingredients:
- Removing the site-adversarial split tests whether acquisition factoring does the work.
- Replacing agreement-weighted normality with majority-vote labels tests whether rater information matters.
Why not just…?
Why not just stain-normalize and run MPFM? Normalization corrects color only. Ciompi's persisting 14-point gap suggests that non-color site artifacts would still read as anomalies. StainGAN is also pairwise, so it cannot handle a site absent from training. The "normalized MPFM" baseline tests this directly. If it matches our false-positive rate, the decomposition adds nothing for histology.
Why not just train with aggressive stain augmentation and treat augmented images as normal? This is the real threat. It is cheap, and it could make a detector invariant to color. Our bet is that augmentation covers synthetic color changes but not thickness or blur. We therefore run augmentation-as-normal with the same budget as our method. We also run a variant with blur and thickness augmentations. If either matches our method on the Camelyon and colorectal external-site tests, we will report that augmentation is sufficient. We will then retain only the ambiguity-routing contribution.
Why not just use virtual-panel disagreement as the anomaly score? Disagreement flags contested cases, but it does not flag lesions no rater was trained on. A novel pattern could produce confident, unanimous misclassification. The DEPP probe set settles this. Panel entropy alone should score indefinite cases highly but miss held-out high-grade dysplasia at matched recall. If it does not miss them, the novelty branch is redundant.
Decisive Checks
The proposal is supported if both of the following hold on at least two of the three histology splits:
- external-site false positives fall relative to normalized and augmented MPFM at matched recall;
- the indefinite cases separate cleanly on ambiguity rather than novelty.
It is refuted if augmentation-as-normal matches the false-positive result, or if removing the site-adversarial term leaves performance unchanged. In either case the finding worth publishing is the benchmark itself, which shows how badly published open-set detectors behave under stain shift.
References
- Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection
- Distribution Prototype Diffusion Learning for Open-set Supervised Anomaly Detection
- Mixture Prototype Flow Matching for Open-Set Supervised Anomaly Detection
- Deep Semi-Supervised Anomaly Detection
- The importance of stain normalization in colorectal tissue classification with convolutional networks
- Staingan: Stain Style Transfer for Digital Histological Images
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images
- IMPACT: Influence Modeling for Open-Set Time Series Anomaly Detection
- Myomatrix arrays for high-definition muscle recording
- Multichannel thin-film electrode for intramuscular electromyographic recordings