Pathology models usually fail on slides from a new hospital, and the usual fix is to repaint the slide's colors before inference. This proposal locates a more specific point of failure. Most image networks shrink their feature maps several times, keeping only a weighted summary of each small neighborhood. For a cell nucleus or a dividing (mitotic) cell only a few pixels across, that shrinking step decides whether the object survives into the deeper layers at all. If the network learned to decide "what to keep" from the particular purple of one lab's hematoxylin dye, a different scanner or stain will quietly erase the very objects the network was built to find. Nothing downstream can recover them.
We propose to make that keep-or-discard decision depend on whether a small, closed-boundary object is present, which is a property that holds across categories, rather than on stain hue. Concretely, we replace the network's downsampling layers with a learnable pooling layer. Its importance weights are trained to predict class-agnostic objectness, meaning "there is some object here" without saying which kind, and are explicitly held constant when the same tissue is recolored, blurred, or contrast-reduced. We then test whether this targeted constraint beats the obvious alternative of augmenting the whole network with the same perturbations. The test runs on benchmarks where the shift is chromatic (stain), optical (scanner), and across tissue types.
Core Claim
Under scanner and stain shift, detection and segmentation errors on tiny histology targets arise disproportionately at the downsampling stages. We claim that constraining only the small module that sets pooling weights, so that it tracks stain- and blur-invariant objectness, closes more of the shift gap than spending the same augmentation budget on the entire network.
The claim is falsifiable in two ways. A probe of how well each object "survives" each stage could show that pooling is not where information is lost. Alternatively, whole-network augmentation could match the targeted version. Either outcome would itself be a useful finding for a field that currently treats shift as a color problem.
Where This Sits
The pooling mechanism comes from LIP: Local Importance-Based Pooling. LIP replaces fixed pooling with a weighted sum whose weights come from a small learned subnetwork. It raised small-object COCO detection (AP_s) for Faster R-CNN from 21.9% to 24.0%. LIP was only ever evaluated on natural images. Its importance subnetwork receives no objectness supervision, so nothing prevents it from keying on dye color. We found no reported result on Lacuna for LIP on histology or under any domain shift.
Two papers motivate the claim that color-only fixes are insufficient. The MIDOG challenge (Mitosis domain generalization in histopathology images - The MIDOG challenge) benchmarked mitosis detection on unseen scanners. It found that a low-contrast scanner (Leica GT 450) produced false positives through contrast and focus differences, not color. The importance of stain normalization in colorectal tissue classification with convolutional networks found that normalization lifted external accuracy from 50.96% to 79.66% but left a gap of roughly 14 points to internal validation. Both results argue that the invariance must cover optics as well as hue. Both also argue that pre-processors such as Staingan, an unpaired CycleGAN color translator, cannot restore detail that blur removed.
The objectness signal follows two earlier results. SFCN-OPI uses a class-agnostic nucleus map to gate the classification loss, and removing that gate dropped classification F1 from 0.742 to 0.496. We move the same kind of gate from the loss into the pooling operator itself. BING showed that gradient patterns in small windows encode a category-independent closed-boundary cue. We use locally contrast-normalized gradients of the hematoxylin optical-density channel as an optional color-free input to the importance module.
CSDS is the closest work on disentanglement. It separates color and structure into two students for semi-supervised gland segmentation. It operates on the whole representation, on large glands, and without instance separation. Our proposal instead isolates one operator and targets objects a few pixels wide.
Receptive field enters as a constraint. LKCell shows that large depthwise kernels, convolutions that filter each channel separately over a wide window of up to 13×13, give useful local context for nuclei. The file also notes the open concern that an overly wide view dilutes tiny targets. We therefore treat the importance module's receptive field as a variable to sweep rather than a fixed design choice.
Research Program
We first build a survival probe. At each downsampling stage of a trained network, we train a linear classifier to decode "object present" at annotated nucleus or mitosis locations. We also measure the ratio of feature energy at object locations to a surrounding background ring. Both are compared between source-scanner and unseen-scanner images. If decodability drops sharply at an early pooling stage under shift, and only modestly elsewhere, pooling is a bottleneck worth fixing. If the drop instead accumulates in the deep layers, the core claim is weakened before any method is built.
Second, we build the objectness-guided pooling layer. It keeps LIP's formulation, in which pooled output is a local average weighted by exponentiated importance logits. The logits come from a depthwise-plus-pointwise module that receives a cheap structural channel (locally normalized optical-density gradients) alongside the backbone features. The module is trained with two auxiliary losses. The first is a class-agnostic objectness loss, which matches the importance map to a blurred map of instance centroids; these centroids come for free from the existing instance labels. The second is a consistency loss that penalizes changes in the importance map when the input is perturbed. Perturbations include stain-space jitter in hematoxylin and eosin optical density, Gaussian and defocus blur, and contrast compression. These are drawn from one fixed perturbation suite that every baseline also receives. The perturbations act on the importance map only. The rest of the network sees ordinary training.
Third, we sweep the importance module's kernel size from 3 to 13. This tests whether more context helps it reject stromal flecks and apoptotic look-alikes, or whether it dilutes tiny targets as the receptive-field literature warns. A single layer, without stacked dilation, avoids gridding artifacts.
Baselines
The naive reference is the same detector or segmenter with ordinary strided downsampling, trained with standard flips and rotations. It establishes the raw shift gap.
The strongest controlled competitor is plain LIP trained with the full perturbation suite applied to the whole network. This is what a careful engineer would do, and it gives the network every chance to learn invariance on its own. Beating it means our pooling must win on unseen scanners or tissues. That win must hold over three seeds, with the same epochs, data, and a matched hyperparameter search budget of twenty trials per method. We report FLOPs alongside each result because our module adds a small branch.
The strongest published reference points are external. On MIDOG unseen scanners, the winning single model reported F1 0.748 and a five-model ensemble 0.773 (MIDOG). On PanNuke, LKCell reports mPQ 0.5080 and FCIS (The Four Color Theorem for Cell Instance Segmentation) reports PQ 0.610. On MoNuSeg unseen organs, the zero-parameter HUNIS (HUNIS) reports AJI 0.6548, above supervised CIA-Net at 0.6306. On KIRC→TNBC, NuSegDA (NuSegDA) reports 60.51% IoU against 52.66% for a source-only U-Net. Training pipelines differ from ours, so these numbers anchor plausibility rather than serve as the controlled test. A single model approaching 0.748 on MIDOG without ensembling would be practically meaningful. We include the LKCell backbone, our pooling inserted into it, and the released FCIS code (zhangye-zoe/FCIS) as instance-segmentation heads.
Three ablations isolate the key ingredient. The first removes the objectness loss but keeps consistency. The second removes consistency but keeps objectness. The third removes the structural gradient channel. If only the consistency term matters, the "objectness" part of the story is wrong.
Why not just…?
Why not just normalize stains or use StainGAN before inference? Normalization addresses color, but MIDOG attributes much of the unseen-scanner failure to contrast and focus, which color mapping cannot undo. The Ciompi study's residual 14-point gap points the same way. We will still run Macenko normalization and a StainGAN pre-processor in front of the naive baseline. Our method should help most on the low-contrast and blurred conditions, and we report results per scanner so this pattern is visible.
Why not just augment the whole network with the same color, blur, and contrast perturbations? This is the real threat, and it is the main controlled baseline rather than a straw man. Our argument is mechanistic: once pooling discards a three-pixel nucleus, later layers cannot regain it, so invariance must be enforced at the point of loss. The survival probe tests this directly. If whole-network augmentation yields equal survival curves and equal unseen-scanner F1, we will report that the targeted constraint is unnecessary.
Why not just avoid downsampling and keep a high-resolution stream, as PointNu-Net's HRNet backbone does? PointNu-Net keeps fine resolution and performs well on PanNuke. Resolution-preserving streams cost memory, however, and the file names exactly this cost as the bottleneck for gigapixel slides. We will compare against an HRNet-style backbone at matched FLOPs. If HRNet wins at equal compute, our method's value narrows to compute-limited settings, and we will say so.
Proposed Experiments
Mitosis detection uses MIDOG 2021. We train on the training scanners and test on unseen scanners E and F. A leave-one-scanner-out run holds out the Leica scanner to stress optical shift. We report F1 and the false-positive rate on apoptotic look-alikes. Nucleus instance segmentation uses PanNuke with held-out tissue types, MoNuSeg Test-2 (unseen organs, AJI), and KIRC→TNBC (IoU), measured with PQ and mPQ.
Two decisive checks follow. The first is the survival probe described above, comparing strided, LIP, LIP-plus-augmentation, and our pooling. The second is a counterfactual recoloring test. We correlate each model's importance maps with hematoxylin optical density versus objectness masks, and measure how much the maps move when the same patch is restained or blurred. The method succeeds if three conditions hold together. Plain LIP's importance maps track dye density and shift across scanners. Ours track objects and stay stable. The unseen-domain gains concentrate in the smallest objects and lowest-contrast scanners.
References
- LIP: Local Importance-Based Pooling
- Mitosis domain generalization in histopathology images - The MIDOG challenge
- The importance of stain normalization in colorectal tissue classification with convolutional networks
- Staingan: Stain Style Transfer for Digital Histological Images
- SFCN-OPI: Detection and Fine-Grained Classification of Nuclei Using Sibling FCN With Objectness Prior Interaction
- BING: Binarized Normed Gradients for Objectness Estimation at 300fps
- Learning Disentangled Stain and Structural Representations for Semi-Supervised Histopathology Segmentation
- LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernels
- The Four Color Theorem for Cell Instance Segmentation
- HUNIS: High-Performance Unsupervised Nuclei Instance Segmentation
- NuSegDA: Domain adaptation for nuclei segmentation
- PointNu-Net: Keypoint-Assisted Convolutional Neural Network for Simultaneous Multi-Tissue Histology Nuclei Segmentation and Classification