Deep Multi-Scale Resemblance Network for the Sub-class Differentiation of Adrenal Masses on Computed Tomography Images

Lei Bi, Jinman Kim, Tingwei Su, Michael Fulham, David Dagan Feng, Guang NingarXiv 2020 · arXiv

TL;DR

Differentiating sub-classes of adrenal masses on CT scans is notoriously difficult due to high visual similarity across different tumor types and severe data imbalance. This paper introduces the Deep Multi-Scale Resemblance Network (DMRN), a paired-CNN architecture that leverages multi-scale feature embedding and similarity learning to achieve state-of-the-art classification accuracy (89.52%).

What Problem Does This Paper Address?

Adrenal masses detected via computed tomography (CT) can be benign or malignant, spanning several distinct sub-classes such as adrenocortical carcinomas (ACAs), adenomas (NAAs), and pheochromocytomas (PCCs). Correctly identifying the specific sub-class is vital for patient diagnosis and treatment planning. However, automated classification using standard Convolutional Neural Networks (CNNs) faces two major hurdles:

  1. High Intra-Class Variation and Inter-Class Similarity: Different sub-classes often look remarkably alike in density and shape, while tumors within the same sub-class can vary wildly due to differences in size, fat content, or calcification.
  2. Data Imbalance: Some benign masses are highly prevalent, while malignant tumors like ACAs are rare. Standard models trained on such data tend to overfit to the dominant classes and perform poorly on rarer ones.
Figure 1. Paired transaxial contrast-enhanced CT images of different adrenal masses (red arrows).

Methods

The authors propose the Deep Multi-Scale Resemblance Network (DMRN), which shifts the problem from standard single-image classification to a relative resemblance paradigm using a Siamese-like paired network architecture.

Figure 2. Flow diagram of our deep multi-scale resemblance network (DMRN). [∙] indicates the output size.

Key components include:

  • Paired CNN Architecture with Weight Sharing: The network processes pairs of images simultaneously, allowing it to directly evaluate feature distances between samples.
  • Multi-Scale Feature Embedding: Instead of relying only on the final layer's features, DMRN extracts feature maps from four distinct stages of a ResNet backbone. At each stage, a custom Residual Pooling Unit (RPU) downsamples the 2D maps into structured 1D feature descriptors.
  • Similarity Feature Learning (SFL): The network is trained using a contrastive-style loss function. It minimizes the distance between paired images of the same sub-class (similarity loss) and maximizes the distance between different sub-classes (dissimilarity loss) up to a predefined margin.
  • Balanced Random Sampling: To mitigate data imbalance, training pairs are generated by randomly sampling across all sub-classes, ensuring that rare classes contribute equally to model optimization.
  • SVM Classifier: During inference, one branch of the trained DMRN extracts features from the input image, which are then classified using a linear Support Vector Machine (SVM).

Datasets

The model was evaluated on two datasets:

  1. Adrenal Mass Dataset: A primary dataset consisting of 229 contrast-enhanced abdominal CT scans (4,065 transaxial slices) from Ruijin Hospital, Shanghai. It includes five sub-classes: ACAs (54), NAAs (35), Ganglioneuromas (GN, 58), Adrenal Myelolipomas (AM, 33), and PCCs (49).
  2. ImageCLEF 2016: A benchmark dataset used for generalizability analysis, containing 6,776 training images and 4,166 test images across 30 diverse medical subfigure categories.

Key Insights and Ideas

  • Multi-Level Supervision Wins: By enforcing similarity learning at multiple intermediate resolutions, the network captures both low-level texture details (essential for identifying fat/soft tissue mixtures) and high-level semantic structures.
  • Pairing as a Regularizer: Transforming an imbalanced multi-class problem into a balanced pair-matching problem naturally prevents the network from favoring dominant classes.
  • Curriculum-like Adaptation: By modeling distances rather than hard labels, the network easily adapts to straightforward training pairs first, and then gradually learns to separate highly similar cross-class pairs.

Key Results

  • Superior Accuracy: In a five-fold cross-validation on the adrenal dataset, DMRN achieved a lesion-level classification accuracy of 89.52%, outperforming the second-best state-of-the-art method (Synergic Deep Learning) by more than 4%.
  • Perfect Class Separation for Specific Tumors: The model achieved 100% sensitivity and specificity when identifying Adrenal Myelolipomas (AMs), which typically present with distinct mixed fat densities.
  • Strong Generalizability: On the ImageCLEF 2016 dataset, DMRN achieved 85.90% accuracy, outperforming standard ResNet-152 and remaining highly competitive with massive ensemble models that required additional training data.

Limitations and Future Work

  • Manual Pre-processing Dependency: The current pipeline requires an experienced reader to manually place a 3D bounding box around the adrenal mass before it can be processed. The authors plan to integrate an automatic lesion detection module to streamline the clinical workflow.
  • Backbone Evolution: The architecture currently relies on a standard ResNet-101 encoder. The authors note that extending the framework to leverage Vision Transformers (ViTs) could yield better performance, though it would increase computational and GPU memory overhead.

Summary updated

Abstract

The accurate classification of mass lesions in the adrenal glands (adrenal masses), detected with computed tomography (CT), is important for diagnosis and patient management. Adrenal masses can be benign or malignant and benign masses have varying prevalence. Classification methods based on convolutional neural networks (CNNs) are the state-of-the-art in maximizing inter-class differences in large medical imaging training datasets. The application of CNNs, to adrenal masses is challenging due to large intra-class variations, large inter-class similarities and imbalanced training data due to the size of the mass lesions. We developed a deep multi-scale resemblance network (DMRN) to overcome these limitations and leveraged paired CNNs to evaluate the intra-class similarities. We used multi-scale feature embedding to improve the inter-class separability by iteratively combining complementary information produced at different scales of the input to create structured feature descriptors. We augmented the training data with randomly sampled paired adrenal masses to reduce the influence of imbalanced training data. We used 229 CT scans of patients with adrenal masses for evaluation. In a five-fold cross-validation, our method had the best results (89.52% in accuracy) when compared to the state-of-the-art methods (p<0.05). We conducted a generalizability analysis of our method on the ImageCLEF 2016 competition dataset for medical subfigure classification, which consists of a training set of 6,776 images and a test set of 4,166 images across 30 classes. Our method achieved better classification performance (85.90% in accuracy) when compared to the existing methods and was competitive when compared with methods that require additional training data (1.47% lower in accuracy). Our DMRN sub-classified adrenal masses on CT and was superior to state-of-the-art approaches.
BibTeX
@misc{bi2020deep,
  author = {Lei Bi and Jinman Kim and Tingwei Su and Michael Fulham and David Dagan Feng and Guang Ning},
  title = {{Deep Multi-Scale Resemblance Network for the Sub-class Differentiation of Adrenal Masses on Computed Tomography Images}},
  year = {2020},
  eprint = {2007.14625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.14625},
}
Figure