Do Object-Based Pathology Models Keep Their Edge Over Pathologists When the Stain Changes?

Some breast biopsies sit on a continuum between benign and malignant, and pathologists often disagree about them. Object-based models address this by first cutting the image into biological units, such as nuclei and tissue regions, and then reasoning over how those units are shaped and arranged. On the BRACS benchmark, one such model is reported to beat the average agreement of three board-certified pathologists on seven-way breast lesion subtyping. The catch is that the model was trained and tested under one laboratory's staining and scanning conditions. Pathologists read slides from other labs without much trouble. A pipeline that starts by detecting nuclei from local color and texture may not. If its advantage disappears under a new stain, the "better than pathologists" claim describes one lab's slides, not the model's diagnostic ability.

This note proposes to find where that advantage breaks and to repair it at that point. The central hypothesis is that most of the performance lost when an object-based pathology model moves to a new site enters through the segmentation stage and not through the classifier. Under a new stain, the nucleus detector misses cells, splits or merges them, and redraws region boundaries. This changes the graph the classifier sees before any texture feature is computed. We will build a protocol that measures how much of the drop comes from the segmentation stage and how much from texture measured inside each object. We will then test a targeted repair: train the segmenter so its masks do not change across stain variants, and keep only the object-level texture features that are reproducible across stain variants. The benchmark for success is whether the model's reported margin over pathologists survives the shift.

Core Claim

We make two claims. The first is diagnostic. For hierarchical cell-and-tissue graph models, shift-induced error splits into two parts that can be measured separately. One part comes from segmentation drift: changes in which objects exist and where their boundaries lie. The other comes from appearance drift: changes in the numbers computed inside a fixed object. We expect the first part to dominate. Let $X_s$ and $X_t$ be the same tissue under source and shifted staining, let $M(\cdot)$ be the segmenter's output masks, and let $F(M, X)$ be the classifier's accuracy when features are computed from image $X$ using masks $M$. The total drop then decomposes exactly as

$$ F(M(X_s),X_s) - F(M(X_t),X_t) = \underbrace{\big[F(M(X_s),X_s) - F(M(X_s),X_t)\big]}_{\text{appearance drift}} + \underbrace{\big[F(M(X_s),X_t) - F(M(X_t),X_t)\big]}_{\text{segmentation drift}}. $$

The middle term needs the same tissue under two stains, so it is only computable where paired images exist. We get them from synthetic restaining of BRACS.

The second claim is corrective. If segmentation drift dominates, then making the segmenter's masks consistent across stain perturbations should recover more out-of-distribution accuracy than perturbing the classifier's inputs alone. It should also preserve the auditable, object-level evidence that motivates object-based pathology in the first place.

Positioning Against Prior Work

The work that motivates this proposal is Hierarchical Cell-to-Tissue Graph Representations for Breast Cancer Subtyping in Digital Pathology. Its model, HACT-Net, builds a cell graph from HoVer-Net nuclei and a tissue graph from SLIC superpixels, which are small compact regions that follow natural image boundaries. A binary assignment matrix links each cell to its enclosing region. The paper reports 61.53% weighted F1 on BRACS against 56.36% for pathologists. It did not test multi-center stain shift, and its segmenters depend on exactly the local color gradients that stain shift disturbs. We take its model as our object of study and ask whether its central comparison holds up.

Two papers supply the robustness methods we reuse, and both work on pixel tiles. Out-of-Distribution Robustness via Targeted Augmentations argues that with few training domains a model latches onto hospital-specific shortcuts. Its remedy is to randomize only the spurious features with color jitter confined to hematoxylin-and-eosin (H&E) color space, which raised Camelyon17-WILDS out-of-distribution accuracy from 75.3% to 90.5%. Stain-invariant self supervised learning for histopathology image analysis pulls each image's embedding toward that of its stain-normalized version. It holds 59.0 ± 1.6% accuracy across 70 synthetic stain variants of BRACS, where baselines fall to 46.7 ± 7.5%. We borrow its synthetic-variant protocol as our main controlled testbed. We differ from both by applying invariance at the segmentation stage and by measuring it on masks rather than on class scores.

A third paper supplies the feature-screening idea. Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation showed that CT texture features become more reproducible across human observers when boundaries come from a consistent algorithm: the intraclass correlation coefficient (ICC) rose to 0.85 from 0.77 for manual contours. We apply the same logic with stain variants playing the role of observers. An object-level texture feature is kept only if its value for the same object is stable across variants. The stability is measured under masks we have made consistent, so boundary noise does not contaminate the screen.

One paper constrains how much we should expect. Clinical utility of receptor status prediction in breast cancer and misdiagnosis identification using deep learning on hematoxylin and eosin-stained slides shows that a plain tile-based multiple-instance model can generalize across six cohorts and two scanner brands. Objects are therefore not necessary for multi-center robustness. Our question is narrower: whether an object pipeline can be made robust without giving up its interpretable structure. Quantitative immunohistochemistry analysis of breast Ki67 based on artificial intelligence is a three-stage object pipeline whose authors name cross-lab stain shift as their open barrier. It illustrates how common the failure mode we target is.

Research Program

The first stage is attribution. We will train HACT-Net as published on BRACS. We will then generate stain-shifted versions of the BRACS test set using the variant procedure of the stain-invariant SSL paper, and compute the two drift terms above for every variant. We will also track direct measures of segmentation drift: change in nucleus count per region of interest, matched-nucleus overlap between source and shifted masks, change in the number of superpixels, and graph edit statistics. These tell us whether accuracy losses follow object-count changes, especially in the atypical classes (FEA and ADH), where pathologists disagree most and where a few missed atypical nuclei could flip a label.

The second stage is repair, built from two ingredients that can be removed independently.

  • Consistency-trained segmentation. We fine-tune the nucleus segmenter and adapt the superpixel stage so that masks predicted on a stain-jittered image agree with masks predicted on the original. The jitter is the targeted H&E jitter, and the agreement loss is overlap-based. This needs no new annotations because the target is the model's own output on clean input, anchored by the original supervised loss.
  • Reproducibility gating. For every handcrafted or learned within-object feature, such as nuclear texture statistics and region co-occurrence texture, we compute an ICC across stain variants under the now-consistent masks. Features below a threshold, chosen on validation data only, are dropped before graph learning.

The third stage is the clinical test. On clean and shifted data, we will ask whether the repaired model's weighted F1 stays above the 56.36% pathologist reference, and whether its remaining errors concentrate in the atypical classes, where disagreement between observers is intrinsic rather than a sign of model failure.

Baselines

The naive reference point is HACT-Net exactly as published, trained on BRACS with no stain handling and evaluated on the same stain variants. Its reported clean score is 61.53% weighted F1. The first result we need is how far this falls. If it barely falls, the premise is wrong and the proposal stops there.

The most obvious competitor is HACT-Net with targeted H&E jitter applied to its input images during training. This is the method of the targeted-augmentations paper transplanted into the object pipeline, and a skeptical reviewer would propose exactly this. It gets the same augmentation distribution, the same epochs, and the same hyperparameter search budget as our method. The only difference is that our method adds the mask-consistency loss and the feature gate. Beating it means a statistically reliable gain, across seeds and variants, in mean and worst-variant weighted F1 on shifted BRACS.

The strongest pixel-based competitors are two tile classifiers trained on the same BRACS training split with matched compute:

  • a ResNet trained with targeted stain jitter, following the paper that reports 75.3% to 90.5% on Camelyon17-WILDS;
  • the stain-invariant SSL ResNet-18, which reports 59.0 ± 1.6% accuracy across 70 BRACS variants.

That BRACS number is accuracy, not weighted F1. We will report both metrics for every model rather than compare across metrics. For efficient slide-level aggregation, Domain Adaptation Without the Compute Burden for Efficient Whole Slide Image Analysis reports beating other multiple-instance aggregators on three-way BRACS. No reported BRACS number was found on Lacuna for it, so we will rerun it rather than quote it.

Two ablations isolate the key ingredient. One uses consistency-trained segmentation without gating. The other uses gating with the original segmenter. If gating alone matches the full method, segmentation was not the bottleneck.

Why not just…?

Why not just add stain jitter to HACT-Net's training images? This is the real threat. Jittering the input also exposes the segmenter's supervised loss to shifted colors, which may already make its masks stable. The question is settled by the attribution protocol applied to the jitter-only model. If its segmentation-drift term is already near zero, our consistency loss adds nothing, and we will report that input jitter suffices for object pipelines. If a large segmentation-drift term remains, the gap is what our loss targets.

Why not normalize stains at test time and skip all of this? Normalization maps one lab's colors onto another's. It can therefore change exactly the within-nucleus textures that carry the sub-visual signal. The stain-invariant SSL paper also reports generalizing without test-time normalization. We will still include a normalize-then-segment baseline with the same backbone, because it is cheap. It loses if its worst-variant F1 is lower or its feature ICCs are worse.

Why not drop objects and use a tile model, which already generalizes across centers? The receptor-status work shows tile models can be robust across six cohorts, so object structure is not required for robustness. If the tile baselines match our repaired model on shifted data, the honest conclusion is that objects buy interpretability and nothing more. What objects buy is still worth having: per-nucleus and per-region evidence a pathologist can audit on borderline calls. Our accuracy claim rests only on the head-to-head numbers.

Why not just screen features by ICC, as in the radiomics literature? Screening without consistent masks measures stability of features computed on unstable objects. It will throw away good features whose variance came from boundaries. The gating-only ablation tests this directly.

Proposed Experiments

The controlled experiment is the attribution and repair study on BRACS stain variants described above. It reports:

  • weighted F1 and accuracy, as mean, standard deviation, and worst variant;
  • per-class F1 for the atypical classes;
  • the two drift terms;
  • nucleus-count drift.

Synthetic restaining is not a real second lab, so two real-shift checks follow. First, the external BACH set, where HACT-Net reports 91% accuracy, is used as a genuinely different institution. Second, a cell-graph variant is built on Camelyon17-WILDS patches, with training on four hospitals and testing on the held-out fifth. There we compare against the 90.5% targeted-augmentation tile result. The patches are small, so the tissue graph is weak; we expect this to test the cell-graph arm mainly and will say so.

The decisive checks are three.

  1. On shifted BRACS, the segmentation-drift term must account for most of the drop in the unrepaired model. If appearance drift dominates, the hypothesis fails.
  2. The full method must beat HACT-Net with input jitter on worst-variant weighted F1, and the ablations must show the gain comes mainly from mask consistency.
  3. The repaired model's shifted-data weighted F1 must remain above 56.36%. If it does not, the reported advantage over pathologists should be understood as specific to one lab's conditions.