Single patch super-resolution of histopathology whole slide images: a comparative study

Mehdi Afshari, Saba Yasir, Gary L. Keeney, Rafael E. Jiménez, Joaquín J. García, Hamid R. TizhooshJournal of Medical Imaging 2023 · DOI

TL;DR

This paper explores the use of deep learning-based super-resolution (SR) techniques to reconstruct high-resolution (HR) histopathology images from lower-resolution (LR) versions. By comparing six different SR networks on a large dataset of TCGA cancer images, the study demonstrates that deep learning can effectively recover fine details, making it a promising approach to reduce storage and computational burdens in digital pathology.

What Problem Does This Paper Address?

Digital pathology, which involves digitizing entire histopathology glass slides into gigapixel Whole Slide Images (WSIs), offers significant advantages for research and clinical analysis. However, the sheer size of these HR images (often exceeding 100,000 x 100,000 pixels) leads to severe storage and processing bottlenecks. This makes it challenging to access, analyze, and share these valuable diagnostic resources. The paper addresses the need for techniques that can generate HR-quality images from LR versions, thereby enabling efficient storage and on-demand reconstruction without information loss. This would allow for the use of less expensive scanning equipment and reduce the overall computational demands of digital pathology workflows.

Methods

The study focuses on single-image super-resolution (SISR) using deep neural networks. They compare six different SR network architectures: Deep Back-Projection Networks (DBPNs), Residual Dense Networks (RDNs), Residual Channel Attention Networks (RCANs), SR Feedback-based networks (SRFBNs), Enhanced Deep Residual Networks (EDSRs), and Enhanced SR Generative Adversarial Networks (ESRGANs).

The core idea is to train these networks to learn the mapping from LR to HR images. The degradation model is assumed to be a downsampling process, and the SR networks aim to invert this process. The paper discusses various loss functions commonly used in SR, including reconstruction loss (L1 and L2 norms), perceptual loss (comparing feature representations from pre-trained networks like VGG16), and adversarial loss (used in GANs to make generated images more realistic). Evaluation metrics include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) for quantitative assessment, and qualitative visual inspection by pathologists.

Datasets

The primary dataset used for training and evaluation is derived from The Cancer Genome Atlas (TCGA) repository. This dataset comprises over 2.5 petabytes of genomic, epigenomic, transcriptomic, and proteomic data, including 30,072 SVS files from 11,007 cases across 32 cancer subtypes. The authors created a specific dataset for histopathology SR by extracting patches from diagnostic TCGA slides, ensuring they were from the same domain. They removed background patches and cropped images into various tile sizes (640x640 down to 16x16 pixels) at different magnification levels (40x to 1x).

For evaluating diagnostic accuracy, the BreaKHis dataset was utilized. This dataset contains 9109 microscopic images of breast tumor tissues from 82 individuals, with images available at four magnification levels. It categorizes images into benign and malignant tumors, with further subtypes.

Key Insights and Ideas

  • Pixel-level information is learnable: The study strongly suggests that deep networks can effectively learn the intricate pixel-level details, structures, and textures present in histopathology images, making SR a viable solution.
  • Network performance varies by cancer type: Different SR networks exhibit varying strengths and weaknesses across different cancer sites and subtypes. This implies that the optimal SR approach might be context-dependent.
  • ESRGAN excels in visual realism, while RCAN/DBPN offer better estimation: ESRGAN-generated images appear visually sharper and more realistic, while RCAN and DBPN provide closer estimations of unknown pixel values, leading to more accurate cellular shapes.
  • SR images are diagnostically comparable to HR images: Pathologist assessments indicate that SR images can achieve diagnostic accuracy levels comparable to original HR images, with no significant difference in classification accuracy for benign vs. malignant tumors.
  • Pathologists often prefer SR images: In a blind evaluation, pathologists frequently preferred the visually enhanced SR images over the original HR images, highlighting the perceptual improvements offered by SR.

Key Results

  • Quantitative Performance: Table 2 shows the PSNR and SSIM scores for six SR networks across various TCGA data categories. DBPN and RCAN consistently achieved the highest scores in many subtypes, with DBPN outperforming in 15 subtypes and RCAN in another 15. The endocrine/thyroid carcinoma (THCA) showed the highest SSIM, while brain/glioblastoma multiforme (GBM) had the lowest.
  • Qualitative Results: Figures 3, 4, and 5 visually demonstrate the effectiveness of the SR networks. ESRGAN generally produced the sharpest images, while RCAN and DBPN offered more accurate cellular shapes.
  • Diagnostic Assessment: Table 3 summarizes the pathologist assessment. The average accuracy for classifying tumor types was 76.0% for both SR and HR images, with a kappa score of 0.52. For subtypes, the average accuracy was 24.0% for SR and 25.0% for HR, with a kappa score of 0.14. This indicates comparable diagnostic performance.
  • Quality Preference: In terms of quality preference, an average of 4.7% of images were strongly preferred as SR, 40.0% as slightly SR, 36.7% were indistinguishable, 18.0% were slightly preferred as HR, and 2.0% were strongly preferred as HR.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that while ESRGAN produces visually appealing images, its generative nature might slightly manipulate cell formation. RCAN and DBPN, while more accurate, produce less sharp edges.
  • The diagnostic assessment, while promising, involved a limited number of pathologists and a specific dataset (BreaKHis). Observer variability is also a factor.
  • The paper mentions that some slides in the TCGA dataset were removed due to missing information or having fewer than 20 WSIs, which could introduce bias.
  • Future Work:
    • The authors suggest that different SR networks may perform best for various cancer sites and subtypes, implying the need for further investigation into domain-specific SR models.
  • The study highlights the potential for SR to be integrated into computerized processing workflows and for visual inspection, suggesting further research into its practical applications in digital pathology.
  • The findings indicate that SR images are not only diagnostically comparable but often visually preferred, opening avenues for improving the user experience in digital pathology.

Summary updated

Abstract

Pixel-level information, including structures and textures in histopathology images, are learnable by deep networks; hence improving the resolution quantity of scanned slides is possible by training appropriate networks. Different SR networks may perform best for various cancer sites and subtypes.
BibTeX
@article{afshari2023single,
  author = {Mehdi Afshari and Saba Yasir and Gary L. Keeney and Rafael E. Jiménez and Joaquín J. García and Hamid R. Tizhoosh},
  title = {{Single patch super-resolution of histopathology whole slide images: a comparative study}},
  journal = {Journal of Medical Imaging},
  year = {2023},
  doi = {10.1117/1.jmi.10.1.017501},
}
Figure