Pathology AI is increasingly asked to write the diagnostic report itself: given a digitized whole-slide image (a gigapixel scan of a stained tissue section), produce the text a pathologist would sign. When such a model is moved to a hospital in another country, its reports get worse. Nobody can currently say why. Two very different causes are mixed together in that drop. The first is technical: each lab's dyes, protocols, and scanners give its slides a characteristic tint. The second is real: patient populations differ in disease presentation and case mix, and their pathologists phrase reports differently. The usual fix, stain normalization (recoloring every slide to match a reference color profile), assumes all color differences are technical noise. If part of the gap is biological, normalization either does nothing useful or erases real signal, and we would not know which.
We propose a measurement method that separates the two. We take test slides from one site and digitally re-render them in another site's stain colors while leaving tissue structure untouched. We then score the generated reports field by field, by converting them into structured fields (diagnosis, grade, tumor percentage, margins) with a hierarchical language-model extractor. The output is, for each site, the share of report error caused by color alone and the share that remains when color is held fixed. That residual is the part no stain normalization method can fix. We then use the same decomposition to judge normalization methods by what they remove and what they damage, not by a single averaged score.
Core Claim
Cross-population degradation in pathology report generation can be split into a stain-attributable component and a residual component. The split is done with counterfactual stain swaps, meaning test-time recoloring of a slide into another site's measured stain vectors while morphology is held fixed. Error is measured per structured report field, not with text-overlap scores like ROUGE, because overlap scores conflate wording differences between countries with diagnostic error. Let $E(s, c)$ be the per-field error on test slides from site $s$ rendered in color profile $c$, let $c_s$ be site $s$'s native profile, and let $c_0$ be the training sites' pooled profile. The split is then
$$ \underbrace{E(s,c_s) - E(0,c_0)}_{\text{total gap}} \;=\; \underbrace{E(s,c_s) - E(s,c_0)}_{\text{stain-attributable}} \;+\; \underbrace{E(s,c_0) - E(0,c_0)}_{\text{residual: biology, case mix, scanner texture}}, $$
where $E(0,c_0)$ is error on held-out slides from the training sites. We predict two things. First, the large "numeric hallucinations" on tumor burden reported in the REG 2025 challenge will turn out to be mostly stain-attributable. Second, categorical diagnosis errors on the external cohort will be mostly residual. If both hold, normalization is the right tool for the first problem and the wrong tool for the second.
Why this gap, and what the prior work does
The motivating evidence comes from three places. Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation benchmarked 24 teams on about 10,500 slide–report pairs from five Asian institutions, with a hidden German test cohort. It found that structured slot filling beat flat captioning and that models produced numeric hallucinations, such as predicting 80% tumor volume when the true value was 5%. It also left stain handling unstandardized, so it cannot say whether those errors are color shortcuts. EXAONEPath 1.0 showed that self-supervised pathology encoders trained without normalization cluster patches by slide tint, which it calls "WSI-specific feature collapse," and that normalizing all 285M pretraining patches fixes this. That makes a color shortcut in downstream report generators plausible. However, EXAONEPath measures representation quality on classification, not report errors or population effects. Critical insights on statistical consistency, generalizability, and feature optimization in AI-driven CcRCC prognostic modeling argues that an Asian-trained renal cancer model degraded on multi-ethnic TCGA because pathology itself differs across ancestries. That is exactly the residual term a normalization-only view ignores.
Other work constrains the design. The importance of stain normalization in colorectal tissue classification with convolutional networks raised external accuracy from 50.96% to 79.66% with normalization, but about 94% internal accuracy left an unexplained 14-point gap. Its authors also note that Macenko normalization can over-normalize red blood cells and fat. Our decomposition is designed to explain that kind of leftover gap rather than shrink it blindly. Multi-target stain normalization for histology slides showed that single-reference normalization is fragile: averaging stain matrices from 8–10 references ("Avg-post") reached 0.44 mIoU on Lizard, versus 0.43 for single-target and 0.37 for stochastic targets. We therefore build target profiles $c_0$ from multiple references, not one slide.
Two papers supply methods we reuse. The Macenko and Vahadane decompositions, which split each pixel's color into hematoxylin and eosin concentrations and stain-color vectors (Stain Normalization for Digital Pathology), give us the swap operation: keep a slide's concentrations and replace its stain vectors with another site's. The two-phase ontology parser of mCODEGPT supplies the extractor. It first asks whether each field is present and only then extracts its value, which reached 93.9% accuracy with 1.1% hallucination. That evaluation used only synthetic notes, so we must validate the extractor on real reports ourselves. We treat Stain-invariant self supervised learning for histopathology image analysis, which trains encoders to map raw and normalized versions of a patch together (59.0 ± 1.6 vs 46.7 ± 7.5 accuracy across 70 BRACS stain variants), as a competing remedy to evaluate, not as part of our method.
Research Program
The first component is the counterfactual stain swap. For each site we estimate a stain profile, the average hematoxylin and eosin vectors and concentration range, from 8–10 reference slides using Avg-post. To render a test slide in profile $c$, we decompose it into concentrations, keep them, and rebuild the color with $c$'s vectors. Structure is unchanged by construction, but we verify this empirically rather than assume it. A swap that also changed morphology would contaminate the residual term. We also run the swap in reverse, rendering training-site slides in external profiles. That gives a second estimate of the stain-attributable term that does not depend on the external site's case mix.
The second component is field-level report scoring. We define a schema per organ following the CAP and WHO terms the REG data already uses: diagnosis category, histologic grade, tumor percentage, margin and invasion status, and organ-specific fields. A two-phase extractor converts both reference and generated reports into this schema. Errors are computed per field: categorical mismatch for diagnosis, and absolute error for tumor percentage. The extractor is a possible source of population bias in its own right, since reports written in different countries in translated or idiosyncratic English could be parsed unevenly. Pathologists will therefore hand-label roughly 300 reference reports stratified by site, and we report extractor accuracy per site. Any site where the extractor is clearly worse is flagged, and its decomposition is reported with that caveat.
The third component is the audit of remedies. We train report generators under five stain regimes: no normalization, single-target Macenko, Avg-post multi-target, normalization to each test site's own profile, and the stain-invariant encoder. We compute the decomposition for each. A good remedy should drive the stain-attributable term toward zero without raising the residual term. A rise in the residual term is the signature of over-normalization erasing biology. To find where such damage concentrates, we break the residual down by field and by tissue type, including blood-rich and adipose-rich regions.
We are honest about population labels. REG gives country of institution, not patient ancestry, so "population" here means site, which bundles lab practice, ancestry, and referral patterns. The decomposition cannot separate ancestry from case mix inside the residual. It can only establish that the residual is not color. We treat that limited claim as the contribution and avoid stronger language about ethnicity.
Baselines
The naive reference point is the total gap with no intervention: a report generator trained on raw slides from the Asian sites, scored per field on held-out Asian slides and on the German cohort. It defines how much there is to explain.
The strongest existing system is the best structured-slot pipeline from the REG 2025 challenge (Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation). Top teams reported composite ranking scores ">0.80", combining ROUGE, BLEU, keyword Jaccard, and embedding similarity. Public captioning baselines such as HistGen trailed by up to 30%. No reported field-level result was found on Lacuna for REG, so we will rescore the strongest reproducible structured-slot architecture under our field-level metric. That architecture uses the same frozen foundation-model features teams used (UNI or CONCH). The comparison is fair because every stain regime shares that architecture, the same training split, the same number of training steps, and the same hyperparameter search budget, with search done on Asian validation data only and the German cohort touched once.
The strongest stain-specific competitor is the stain-invariant encoder of Stain-invariant self supervised learning for histopathology image analysis, pretrained on the same slides and compute budget. A second is Avg-post normalization, whose best reported figure is 0.44 ± 0.01 mIoU on Lizard nuclei segmentation (Multi-target stain normalization for histology slides).
The ablation that isolates our key ingredient replaces field-level scoring with the REG composite text score while keeping everything else identical. If the decomposition looks the same under text overlap, field-level extraction added nothing. We expect the composite to inflate the residual on the German cohort because wording differs. A second ablation replaces the concentration-preserving swap with global color-histogram matching, which changes structure-relevant contrast. This tests whether the swap's structure preservation matters.
"Beating" the baselines here has a specific meaning, because we are not proposing a new generator. Success means our decomposition predicts which remedy helps which field: a remedy that shrinks the measured stain-attributable term should cut external tumor-percentage error by a comparable amount. A normalization method chosen by our audit should match or beat the best regime picked by composite score on external field-level error, while not degrading any field by more than its across-seed standard deviation.
Why not just…?
Why not just normalize everything and report the external score? That is what EXAONEPath 1.0 and the colorectal normalization study effectively do, and it works on average. The latter still left a 14-point gap unexplained and documented over-normalization of blood and fat. A single averaged score cannot show whether normalization removed a shortcut on one field while erasing signal on another. The audit measures exactly that. If normalization lowers both terms on every field, we will say so, and the practical recommendation becomes simply "normalize."
Why not just rely on foundation-model encoders or color-jitter augmentation, which should already be stain-robust? This is a real possibility, given that REG teams used UNI, CONCH, and Virchow features. EXAONEPath's finding that jitter-trained DINO encoders still cluster by slide argues against it, but that finding concerns patch representations, not reports. Our swap test answers it directly. If swapping German slides into Asian colors leaves field errors unchanged for these encoders, the stain-attributable term is near zero, and the gap is residual. That is a useful negative result: it would tell the field to stop tuning normalization for cross-site report generation.
Why not just train a site classifier on the features to measure how much site information they carry? Probing shows that site is decodable, not that it changes the report. Tint can be linearly decodable yet ignored by the report head. The swap measures causal effect on output fields, which is the quantity clinicians care about. We will report probe accuracy alongside the swap results to show where the two disagree.
Why not just stratify ROUGE or the REG composite by site? Text overlap penalizes a German pathologist's phrasing as if it were a diagnostic error, which inflates apparent population bias. The first ablation above tests this. The real risk runs the other way: our extractor could be biased by site. That is why we validate it per site on hand-labeled reports before trusting any decomposition.
Proposed Experiments
The primary experiment computes the decomposition on REG 2025, with all five Asian institutions as training sites and the German cohort plus each held-out Asian site as test sites, per organ and per field. Three seeds are run per regime, with bootstrap confidence intervals over cases. The decisive check concerns tumor percentage. If rendering German slides in the training profile removes most of the external tumor-percentage error, numeric hallucination is a color shortcut. If the reverse swap, Asian slides rendered in German colors, induces a comparable error, the attribution is confirmed from both directions. If neither swap moves the error, the hallucination stems from something else, such as the language model's priors, and normalization is ruled out as its fix.
Two supporting experiments check that the swap preserves morphology. On Lizard, which pools six colorectal sources, nuclei segmentation under swapped versus native colors should agree to within the 0.01 mIoU spread reported for Avg-post. On the Radboud and external CRC colorectal data, nine-class tissue predictions should be stable under swaps for blood and fat, the classes where over-normalization was reported. A failure on either means the residual term is contaminated, and we would restrict claims to tissue types where the swap is shown to be safe.
The final check targets over-normalization. For each regime we compare the residual term before and after normalization on the German cohort. A statistically reliable increase on any field, especially in blood-rich or adipose-rich cases, is direct evidence that normalization destroys real signal. That would be the first measurement in this literature that turns the over-normalization worry from an anecdote into a number.
References
- Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
- EXAONEPath 1.0 Patch-level Foundation Model for Pathology
- Critical insights on statistical consistency, generalizability, and feature optimization in AI-driven CcRCC prognostic modeling
- The importance of stain normalization in colorectal tissue classification with convolutional networks
- Multi-target Stain Normalization for Histology Slides
- Deep learning can predict survival directly from histology in clear cell renal cell carcinoma
- Introducing mCODEGPT as a zero-shot information extraction from clinical free text data tool for cancer research
- Stain-invariant self supervised learning for histopathology image analysis