Virtual staining trains a network to turn a cheap, routine tissue image (an H&E slide) into an expensive diagnostic stain (an immunohistochemistry, or IHC, slide that marks a specific protein). The training pairs come from two adjacent slices of the same tissue block. These slices are never pixel-aligned: each one stretches and tears independently when it is cut. The diagnostic signal for HER2 breast cancer scoring is a thin, often faint brown ring on cell membranes, which is exactly the kind of low-contrast boundary that misalignment destroys. When the target ring sits a few pixels away from where the network draws it, a pixel-wise loss is minimized by averaging over the possible positions. The result is a weak, smeared halo, and current models do show the predicted symptom. Pyramid Pix2pix on the BCI benchmark reaches only 37.5–40% pathologist HER2 grading accuracy on synthesized images (BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix). The BCI challenge review names the collapse toward generic low-intensity staining the "alignment trap" (Breast Cancer Immunohistochemical Image Generation: a Benchmark Dataset and Challenge Review).
We propose supervising membranes and image content separately. The overall color and expression content gets a loss that ignores pixel position. The membranes get a loss that asks only that a predicted membrane of the right strength exist somewhere within a small tolerance radius of each target membrane, and vice versa. The tolerance radius is set per region from how uncertain the slice registration is there. The image is never warped. The loss uses a local maximum over shifts instead of an average, so its minimum is a single sharp membrane at full intensity rather than a blur. This targets the specific failure the literature reports, losing HER2 3+ membranes.
Core Claim
The claim is that boundary blur in misaligned virtual staining comes from the loss averaging over the unknown shift, and that replacing the average with a local maximum restores sharp, correctly graded membranes. Two conditions apply. First, the comparison is made on a thin membrane map extracted by the same differentiable operator from both prediction and target. Second, the tolerance stays below half the spacing between neighboring cell membranes. Beyond that radius, the two slices no longer show the same cells, and only regional supervision is meaningful.
Concretely, let $p$ and $t$ be membrane-strength maps of the generated and real IHC images. Let $M_{\tau}[\cdot]$ be a max-pool over a disk of radius $\tau(x)$ centered at pixel $x$. The boundary loss is
$$ L_{\text{mem}} = \sum_x \max\big(0,\; t(x) - M_{\tau}[p](x)\big) \;+\; \sum_x \max\big(0,\; p(x) - M_{\tau}[t](x)\big). $$
The first term is a recall term. It is paid when strong target membrane has no equally strong predicted membrane nearby. The second is a precision term, which penalizes membrane drawn where no real membrane is nearby. Because both terms compare intensities, a faint ring cannot stand in for a 3+ ring.
Research Program
The membrane map is built without any learned branch. IHC images are first separated into a brown (DAB, the chromogen that marks HER2) channel and a blue (hematoxylin) channel by color deconvolution, a fixed linear unmixing that is differentiable. The DAB channel then passes through a Hessian ridge filter tuned to membrane width. This filter responds strongly to thin lines and weakly to thick blobs. A network that "cheats" by painting a broad band around the target therefore earns little ridge response, and the precision term discourages the band. A local mass-matching term, which compares summed ridge strength per patch, guards against the remaining degenerate solution of many thin spurious lines.
This operator replaces the hard Otsu-plus-morphology masks of ATST-Net (Advancing Stain Transfer for Multi-Biomarkers: A Human Annotation-Free Method Based on Auxiliary Task Supervision), which in that work yield only region-level expression masks. Global thresholds drop exactly the faint 1+ and equivocal 2+ membranes that distinguish grades.
The tolerance $\tau(x)$ uses the fact that both slices carry the hematoxylin counterstain, so nuclei are visible in both. For each patch we compute local normalized cross-correlation between the H&E-derived nuclear channel and the IHC hematoxylin channel after standard global-plus-Elastix registration. We set $\tau$ from the spread of the correlation peak. Where the peak spread exceeds half the local nuclear spacing, $L_{\text{mem}}$ is switched off for that patch and only regional losses apply.
Regional content is supervised by Frequency Distribution Loss, which compares the amplitude and phase spectra of deep features via sliced Wasserstein distance (Misalignment-Robust Frequency Distribution Loss for Image Transformation). We adopt it as the strongest available shift-tolerant content loss. Our hypothesis is that its global phase comparison does not localize individual membranes.
The generator backbone and adversarial setup follow the strongest available stain-transfer pipeline so that only the loss changes. Training and evaluation use BCI (BCI) and MIST-HER2 (ATST-Net), with MIST to BCI as a cross-dataset check.
The work connects to prior methods as follows. STEAL lets noisy label contours snap toward network predictions with a level set (Devil Is in the Edges: Learning Semantic Boundaries From Noisy Annotations.). It is the closest boundary-under-misalignment method, but it needs strong normal gradients, a warm-up phase, and a label-to-image setting. Our loss has no contour evolution and works image-to-image. Learning to Align warps the prediction with a learned deformation field before computing the loss (Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data). That gives a reported ~2 dB PSNR gain on misaligned pairs, but its own limitations note that unconstrained warps can bend real boundaries onto non-corresponding structures. We never warp, so predicted membrane shape is never edited to fit a different slice. The segmentation side motivates the problem: boundary-attention networks such as MISNet (Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation) and Polyper (Polyper: Boundary Sensitive Polyp Segmentation) show that low-contrast margins need dedicated supervision. Both assume co-registered masks, which is the assumption we drop.
Baselines
The naive reference point is Pyramid Pix2pix trained on the pre-registered BCI pairs with its multi-scale pixel loss. It reports PSNR 21.160, SSIM 0.477, and 37.5–40% pathologist HER2 grading accuracy (BCI). We retrain it from its released configuration on our splits rather than relying on the quoted numbers.
The strongest existing method for this exact task is ATST-Net, whose auxiliary location, intensity, and positive-nucleus-count tasks are the closest thing to region-level shift-tolerant supervision (ATST-Net). On MIST-HER2 it reports gains over its predecessors of 0.019 CSS, 0.027 average PHV, 11.2 FID, and 4.6 KID. No absolute reported result was found on Lacuna for ATST-Net on BCI or MIST-HER2, so we will reproduce it. To be fair, we use the same pretrained encoder, the same patch sampling, the same training iterations, and an equal hyperparameter search budget of about twenty trials per method on a held-out validation split. Beating ATST-Net means a statistically significant gain in HER2 grade agreement on held-out patients, with FID no worse.
We also run two image-to-image misalignment competitors under identical backbones. One is Learning to Align, our reimplementation of the deformation sub-network (Digital Staining with Knowledge Distillation). The other is FDL alone. For reference, FDL reports FID 37.50 versus 74.55 for Contextual Loss on DPED (FDL).
Two ablations isolate the key ingredient. The first replaces the local max-pool with a local average-pool of the same radius. This keeps the membrane operator but restores averaging, and should reintroduce faint halos if our diagnosis is right. The second replaces the adaptive $\tau(x)$ with a single fixed radius chosen by validation. This is the "just dilate the target" alternative, and it tells us whether registration-derived tolerance earns its complexity.
Why not just…?
Why not just use Frequency Distribution Loss or Contextual Loss on the whole image? These are the most serious threats, because FDL's phase term is designed to carry structure. Contextual Loss treats features as an unordered bag, and FDL itself shows it underperforms (PSNR 25.13 versus 26.70 dB on misaligned Urban100) (FDL; The Contextual Loss for Image Transformation with Non-aligned Data). FDL's phase comparison is still pooled over the whole feature map, and thin membranes are a small fraction of that spectrum. The FDL-alone baseline settles this directly. If FDL matches our membrane recall on the synthetic-shift test below, the proposal reduces to "use FDL," and we will report that.
Why not just warp the prediction with an in-the-loop registration network? Learning to Align does improve PSNR on misaligned pairs (Digital Staining with Knowledge Distillation). A dense warp, however, is free to deform a predicted membrane toward the neighboring cell's membrane in the other slice. The BCI review attributes the alignment trap to exactly these true slice-to-slice differences (Challenge Review). We test this by measuring membrane shape distortion, specifically curvature and continuity of rings relative to the input nuclei, under both methods.
Why not just dilate the target membrane mask and use a tolerant cross-entropy? This is the fixed-band ablation. A dilated mask with an averaging loss still rewards thick, faint bands. A dilated mask with a max-pool loss is our method with constant $\tau$. If the fixed version matches the adaptive one, we drop the registration-uncertainty machinery and keep the simpler loss.
Why not just treat the data as unpaired (CycleGAN or CUT)? OT-StainNet and the digital-staining work both report that unpaired translation hallucinates or loses fine pathology (OT-StainNet: Optimal Transport Driven Semantic Matching for Weakly Paired H&E-to-IHC Stain Transfer; Digital Staining with Knowledge Distillation). Pyramid Pix2pix outperforms CycleGAN on BCI. We include CUT only as a lower bound.
Proposed Experiments
The decisive check is controlled, because pixel metrics on real adjacent slices cannot tell blur from a correct but shifted membrane. We take real BCI and MIST IHC tiles and pair each with its own H&E. We then apply known elastic deformations of increasing magnitude to the IHC target only, as FDL did with synthetically misaligned DIV2K. Because the undeformed IHC is retained, true membrane positions are known. For each method we plot membrane recall and precision against displacement magnitude, using boundary F-score with a 2-pixel tolerance against the undeformed target, together with mean predicted DAB intensity on true 3+ membranes. The prediction is specific. Pixel-loss and average-pool models should lose 3+ intensity roughly in proportion to displacement. The max-pool loss should hold intensity flat until displacement approaches half the cell spacing, and degrade only beyond that point.
On real data, a fixed HER2 grading classifier trained only on real IHC scores every synthesized held-out tile, and agreement with the real-slide grade is the primary metric. FID, KID, and the CSS and PHV measures used by ATST-Net are secondary, and PSNR and SSIM are reported but not trusted. A blinded pathologist reading of 40 cases per method, matching the BCI protocol, gives a grading accuracy directly comparable to the 37.5–40% reference. We report per-grade confusion so that gains on 3+ can be told apart from gains on 0/1+.
The proposal fails cleanly in either of two cases. If FDL alone matches our 3+ membrane intensity under synthetic shift, the boundary loss is unnecessary. If the average-pool ablation matches max-pool, our diagnosis of the alignment trap is wrong.
References
- BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix
- Breast Cancer Immunohistochemical Image Generation: a Benchmark Dataset and Challenge Review
- Advancing Stain Transfer for Multi-Biomarkers: A Human Annotation-Free Method Based on Auxiliary Task Supervision
- Misalignment-Robust Frequency Distribution Loss for Image Transformation
- Devil Is in the Edges: Learning Semantic Boundaries From Noisy Annotations.
- Digital Staining With Knowledge Distillation: A Unified Framework for Unpaired and Paired-but-Misaligned Data
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Polyper: Boundary Sensitive Polyp Segmentation
- The Contextual Loss for Image Transformation with Non-aligned Data
- OT-StainNet: Optimal Transport Driven Semantic Matching for Weakly Paired H&E-to-IHC Stain Transfer