TL;DR
This paper introduces DeepLIIF, a novel deep learning framework that transforms standard, low-cost IHC images into high-quality, multiplex immunofluorescence (mpIF) representations. This allows for single-step stain deconvolution, cell segmentation, and quantitative protein expression analysis, significantly improving the interpretability and accuracy of IHC diagnostics.
What Problem Does This Paper Address?
Currently, immunohistochemistry (IHC) is a cornerstone of diagnostic pathology, but its reporting is largely qualitative or semi-quantitative. This is due to inherent challenges in standard brightfield IHC, such as superimposed staining channels and high chromogen overlap, which make accurate deconvolution and quantification difficult. While multiplex immunofluorescence (mpIF) offers superior clarity and quantitative potential, its high cost and limited availability hinder widespread adoption. This paper addresses the need for a method that can bridge this gap, leveraging the accessibility of IHC to achieve the quantitative insights typically only available with expensive mpIF.
Methods
DeepLIIF employs a multitask deep learning approach, trained on a unique dataset of co-registered IHC and mpIF images. The core innovation lies in its ability to perform several tasks simultaneously:
- Stain Deconvolution/Separation: It translates low-cost IHC images into cleaner, separated mpIF channels (including Hematoxylin, DAPI, and specific protein markers).
- Cell Segmentation: It accurately delineates individual cells, enhanced by the novel use of a nuclear envelope stain, LAP2beta, which improves the segmentation of touching or overlapping nuclei.
- Quantitative Single-Cell IHC Scoring: It classifies segmented cells as positive or negative for protein expression, enabling precise quantification.
The framework utilizes a Generative Adversarial Network (GAN) architecture with both translation and segmentation components. Notably, it's trained to perform these tasks in a unified process, avoiding error-prone manual deconvolution or thresholding steps.
Datasets
- Internal Training Dataset: A de novo dataset of co-registered IHC and mpIF staining from lung and bladder tissues, including Hematoxylin, mpIF DAPI, mpIF Lap2, and mpIF Ki67.
- BCDataset: A publicly available dataset for Ki67 staining with manual centroid annotations from breast carcinoma.
- NuClick Dataset: A dataset for CD3/CD8 IHC staining.
- LYON19 Challenge Dataset: A dataset for lymphocyte detection in IHC stained specimens (breast, colon, prostate).
- Human Protein Atlas: Used for IHC Ki67 and TP53 images.
- DLBCL-Morph Dataset: Contains IHC stained tissue microarrays for BCL2, BCL6, CD10, MYC, and MUM1 markers.
- MonuSeg Dataset: Used for H&E images to test out-of-the-box generalization.
The authors also generated synthetic IHC images to augment their training data, particularly for regions with dense positive cell clusters.
Key Insights and Ideas
- Bridging the IHC-mpIF Gap: The core idea of translating low-cost IHC to high-cost mpIF representations is a significant step towards democratizing advanced quantitative pathology.
- Multitask Learning for Robustness: By training a single model to perform deconvolution, segmentation, and quantification simultaneously, DeepLIIF achieves a more robust and generally useful solution.
- LAP2beta as a Segmentation Enhancer: The introduction of LAP2beta as a nuclear envelope stain demonstrably improves cell delineation, especially in challenging regions with overlapping nuclei.
- Generalization Across Markers and Datasets: DeepLIIF, trained primarily on Ki67, shows remarkable generalization to a wide array of nuclear and non-nuclear markers, as well as across different datasets and imaging conditions.
- Synthetic Data for Improved Performance: Generating synthetic IHC images with increased positive cell density significantly boosted the model's performance, particularly in challenging regions.
Key Results
DeepLIIF consistently outperformed state-of-the-art models across various metrics on benchmark datasets. For instance, on the BC Dataset, DeepLIIF achieved a pixel accuracy of 94.18% and a Dice score of 68.15%, surpassing other methods like Mask_RCNN and nnU-Net. It also demonstrated strong performance on the NuClick dataset for CD3/CD8 segmentation. The model's ability to generalize was highlighted by its successful application to various IHC markers (CD3, CD8, BCL2, BCL6, MYC, MUM1, CD10, TP53, PDL1) and its ability to infer modalities from H&E images. The generated mpIF images were shown to be highly realistic, with low MSE and high SSIM scores.
Limitations and Future Work
- Non-Nuclear Marker Accuracy: While DeepLIIF shows surprising success with non-nuclear markers, its accuracy is slightly lower compared to nuclear markers. The authors anticipate that further training with non-nuclear markers will improve performance.
- Computational Cost: While the inference time for a 512x512 patch is 4 seconds, processing whole slide images (WSIs) can take 10-25 minutes, depending on image size and magnification.
- Future Directions: The authors plan to extend the work to handle more challenging membranous/cytoplasmic markers, incorporate additional mpIF tumor and immune markers for more precise phenotypic quantification, and explore its use in assessing Ki67 quantification in tumors with unusual morphologic features. They also envision using the generated clean mpIF DAPI images for whole slide image registration.
Summary updated