TL;DR
This paper tackles the challenge of identifying specific cell subtypes within colon tissue using standard Hematoxylin and Eosin (H&E) stains, which are traditionally insufficient for such detailed classification. By leveraging multiplexed immunofluorescence (MxIF) data and advanced style transfer techniques, they create "virtual H&E" images that allow for supervised learning of cell subtypes, enabling the identification of previously un-labelable cells on H&E.
What Problem Does This Paper Address?
Histological analysis of tissue samples, particularly in clinical settings, heavily relies on Hematoxylin and Eosin (H&E) staining. While H&E is excellent for visualizing tissue microanatomy, it lacks the specificity to differentiate between various cell subtypes. This limitation hinders a deeper understanding of cellular functions in both health and disease. Specialized stains like multiplexed immunofluorescence (MxIF) offer this detailed subclassification but are more complex, costly, and less ubiquitous than H&E. The challenge, therefore, is to bridge this gap: how can we achieve fine-grained cell subclassification on readily available H&E images? Previous efforts, like the CoNIC Challenge, focused on broader cell type classification on H&E but struggled with epithelial and immune cell subtypes. This paper aims to overcome this by enabling the identification of specific cell types, such as helper T cells and epithelial progenitors, directly from H&E images.
Methods
The core of this research lies in inter-modality learning and virtual staining.
- Leveraging MxIF for Ground Truth: The authors started with multiplexed immunofluorescence (MxIF) data, which provides rich, multi-channel information allowing for precise cell subclassification. They used this data to generate detailed cell-level annotations for 14 distinct subclasses.
- Style Transfer to Virtual H&E: To apply this detailed knowledge to H&E, they employed a CycleGAN (a type of Generative Adversarial Network) for style transfer. This process transforms the MxIF images into realistic-looking "virtual H&E" images. The goal is to make the virtual H&E visually resemble real H&E while retaining the rich subclass information from the MxIF.
- Supervised Learning on Virtual H&E: With the virtual H&E images and their corresponding 14 subclass labels, they trained a supervised learning model (specifically, a ResNet-18 classifier). This model learns to associate visual features in the virtual H&E with specific cell subtypes.
- Evaluation on Real H&E: Crucially, the trained model was then evaluated on actual, real H&E images. To address the domain shift between virtual and real H&E, and the difference in label granularity, they used techniques like bounded metrics and domain-specific CycleGANs for stain harmonization.
Datasets
- In-house MxIF Data: A dataset of 28 whole slides of colon tissue stained with 27 markers was used. This data was crucial for generating the ground truth labels for 14 cell/nucleus subclasses. Samples were from patients with varying degrees of Crohn's disease activity and control tissues.
- Public Real H&E Data (CoNIC Challenge): This dataset, comprising 4981 image patches, was used for external validation. It provided real H&E images of colon tissue with labels for six broader cell types (epithelial, lymphocyte, plasma, eosinophil, neutrophil, and connective). This dataset differs from the in-house data in its label granularity and source diversity.
Key Insights and Ideas
- Virtual H&E as a Bridge: The concept of creating "virtual H&E" from more informative but less common stains (like MxIF) is a powerful strategy to leverage existing, abundant H&E data for tasks requiring higher specificity.
- Style Transfer for Domain Adaptation: CycleGANs are effective not only for generating realistic virtual H&E but also for mitigating domain shift issues when evaluating models on real-world data from different sources or with varying staining characteristics.
- MxIF's Granularity is Key: The richness of MxIF data, even when reduced to a few key stains, provides sufficient information to train models that can then infer finer distinctions on H&E.
- Helper T and Epithelial Progenitors are Detectable: The study demonstrates that even challenging subtypes like helper T cells and epithelial progenitors, which are difficult to distinguish with standard H&E alone, can be identified with this approach.
Key Results
- Virtual H&E Classification: On virtual H&E, the model showed learning behavior for several classes, including helper T, macrophage, enterocyte, epithelial progenitor, enteroendocrine, and fibroblast nuclei, as well as goblet cells. Classification accuracy was stable whether using ground truth or predicted centroids.
- Real H&E Performance: When evaluated on real H&E, the model demonstrated reasonable performance for helper T, epithelial progenitor, enterocyte, fibroblast, and stromal (undetermined) nuclei.
- Upper bound positive predictive values (PPVs) for helper T and epithelial progenitor nuclei on real H&E were 0.43±0.03 and 0.94±0.02, respectively.
- The study is the first to automatically identify helper T and epithelial progenitor nuclei on H&E.
- Instance Segmentation: Instance segmentation performance was better on recall than precision, with a noticeable drop on real H&E compared to virtual H&E, highlighting the domain shift.
Limitations and Future Work
- Domain Shift: A significant challenge remains the domain shift between virtual and real H&E, which impacts performance. While style transfer helps, inter-site variability in real H&E data still causes performance instability.
- Label Granularity Mismatch: The real H&E dataset (CoNIC) lacked the fine-grained labels present in the virtual H&E, necessitating the use of bounded metrics and parent class matching, which limits direct quantitative comparison.
- Goblet Cell Identification: Goblet cells were identified via MUC2 staining in MxIF, which highlights regions rather than nuclei. This difference in definition made direct comparison on real H&E (which only labels nuclei) challenging.
- Future Work: The authors suggest that having paired H&E and MxIF on the exact same tissue cuts would eliminate the need for virtual H&E and allow for more direct comparisons. Further research into more sophisticated harmonization approaches for inter-site variability is also needed. The paper also notes that while their model shows limitations compared to real H&E-trained models on coarser labels, it represents a significant step forward in H&E nucleus subtyping.
Summary updated