Scale-Aware Transformers for Diagnosing Melanocytic Lesions

Wenjun Wu, Sachin Mehta, Shima Nofallah, Stevan Knezevich, Caitlin J. May, Oliver Chang, Joann G. Elmore, Linda G. ShapiroIEEE Access 2021 · OpenReview

TL;DR

This paper introduces SCATNet, a novel "scale-aware transformer" network designed to improve the accuracy of diagnosing melanocytic skin lesions from whole slide images (WSIs). By considering information from multiple magnifications (scales) and employing a soft labeling strategy for tissue slices, SCATNet achieves performance comparable to practicing pathologists.

What Problem Does This Paper Address?

Diagnosing melanocytic lesions, such as melanoma, from histopathological whole slide images (WSIs) is a complex task for pathologists. A major challenge is the significant intra- and inter-observer variability, meaning different pathologists might interpret the same image differently, leading to diagnostic errors and potential patient harm. While digitized WSIs offer opportunities for AI assistance, effectively extracting relevant diagnostic information from these massive images, which can be orders of magnitude larger than standard images, is difficult. Existing methods often struggle to capture both local details and global context across different magnifications, and the presence of multiple tissue slices within a single WSI, some of which may be less informative, adds further complexity.

Methods

The core of SCATNet is a novel Scale-Aware Transformer Network (SCATNet). It addresses the challenges by:

  • Multi-Scale Input Processing: SCATNet processes WSIs at multiple magnifications (e.g., 7.5x, 10x, 12.5x). For each scale, it first uses a Convolutional Neural Network (CNN, specifically MobileNetv2) to extract local "patch embeddings."
  • Contextualized Patch Embeddings: These local embeddings are then fed into transformer layers to learn "contextualized patch embeddings," capturing relationships between patches within a single scale.
  • Scale-Aware Embeddings: Crucially, the contextualized patch embeddings from all scales are concatenated. Another transformer network then processes these to learn "scale-aware embeddings," effectively weighing information from different magnifications. This allows the model to adaptively focus on the most informative scales for diagnosis.
  • Soft Label Assignment: To handle the ambiguity of multiple tissue slices in a WSI, where only a few might be truly diagnostic, SCATNet employs a "soft labeling" method. Instead of assigning a hard label to all slices, it uses singular value decomposition (SVD) to derive soft labels that better represent the diagnostic uncertainty and reduce false tissue-label pairs during training. This method also incorporates a diagnostic constraint, ensuring slices without a clear region of interest (ROI) are assigned labels consistent with lower diagnostic categories.

Datasets

The study utilizes a dataset of 240 skin biopsy WSIs from the MPATH study (protocol number STUDY00008506). These images were stained with hematoxylin and eosin (H&E). The dataset was split into training (102 WSIs), validation (23 WSIs), and an independent test set (115 WSIs). The ground truth diagnoses were established by a consensus panel of three experienced dermatopathologists. The paper also references a prior clinical study involving 187 practicing U.S. pathologists who interpreted the same test set cases, providing a benchmark for human performance.

Key Insights and Ideas

  • Scale Matters: The study strongly suggests that diagnostic information for melanocytic lesions is distributed across different magnifications. SCATNet's ability to learn from and weigh these multiple scales is key to its improved performance.
  • Context is Crucial: Beyond local patch features, understanding the relationships between patches (context) and across different scales is vital for accurate WSI classification. Transformers excel at capturing these complex dependencies.
  • Soft Labels for Ambiguity: The soft labeling approach is a significant contribution, effectively addressing the inherent ambiguity in WSIs with multiple, potentially non-diagnostic, tissue slices. This leads to more robust model training.
  • Human-Level Performance: Achieving performance comparable to practicing pathologists is a major milestone, indicating the potential of AI to serve as a valuable "second reader" in clinical practice.

Key Results

  • SCATNet significantly outperforms five other state-of-the-art WSI classification methods by a "significant margin" across various metrics (accuracy, F1 score, sensitivity, specificity, AUC).
  • Specifically, SCATNet achieved 8% higher accuracy than the method proposed by Chikontwe et al. [7] and 6% higher accuracy than the method proposed by Hashimoto et al. [8].
  • When compared to 187 practicing U.S. pathologists on the same test set, SCATNet achieved comparable overall accuracy (0.64 vs. 0.65).
  • The use of multiple input scales (specifically, a combination of 7.5x and 12.5x) showed improvements, particularly in the more aggressive cancer categories (pT1a and pT1b).
  • The soft labeling method demonstrated better performance than hard labeling, label smoothing, and constrained label smoothing methods, achieving an accuracy of 0.60 compared to 0.50-0.56 for others.

Limitations and Future Work

  • Limited Scope: The current study is focused solely on melanocytic lesions. Future work could extend SCATNet to other types of biopsy images (e.g., breast, lung).
  • Dataset Size: While the dataset is substantial for WSI analysis, it is still relatively small for deep learning models. Testing SCATNet on larger, independent test sets is crucial for demonstrating broader clinical applicability.
  • Patch Size Optimization: The paper notes that scaling patch size alone might not be optimal, suggesting that future studies could explore compound model scaling techniques (e.g., in EfficientNet) to further improve performance.
  • Patch Sampling: The authors suggest that complementing SCATNet with advanced patch sampling methods (like those from Chikontwe et al.) could further enhance performance, particularly for specific diagnostic categories.

Summary updated

Abstract

Diagnosing melanocytic lesions is one of the most challenging areas of pathology with extensive intra- and inter-observer variability. The gold standard for a diagnosis of invasive melanoma is the examination of histopathological whole slide skin biopsy images by an experienced dermatopathologist. Digitized whole slide images offer novel opportunities for computer programs to improve the diagnostic performance of pathologists. In order to automatically classify such images, representations that reflect the content and context of the input images are needed. In this paper, we introduce a novel self-attention-based network to learn representations from digital whole slide images of melanocytic skin lesions at multiple scales. Our model softly weighs representations from multiple scales, allowing it to discriminate between diagnosis-relevant and -irrelevant information automatically. Our experiments show that our method outperforms five other state-of-the-art whole slide image classification methods by a significant margin. Our method also achieves comparable performance to 187 practicing U.S. pathologists who interpreted the same cases in an independent study. To facilitate relevant research, full training and inference code is made publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/meredith-wenjunwu/ScATNet</uri> .
BibTeX
@inproceedings{wu2021scale,
  author = {Wenjun Wu and Sachin Mehta and Shima Nofallah and Stevan Knezevich and Caitlin J. May and Oliver Chang and Joann G. Elmore and Linda G. Shapiro},
  title = {{Scale-Aware Transformers for Diagnosing Melanocytic Lesions}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=r_7bzmwXR5},
}
Figure