Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications

Lijun Gong, Kai Ma, Yefeng Zheng

Introduction

Medical image classification presents a unique challenge compared to natural image classification. While a standard Convolutional Neural Network (CNN) can easily distinguish between a cat and a car, differentiating between benign and malignant skin lesions or grading diabetic retinopathy is significantly harder. This difficulty arises from small inter-class distance, where images from different categories appear nearly identical to the human eye and standard algorithms alike.

In these scenarios, CNNs are often misled by "distractors"—features within an image that correlate with an incorrect class, leading to misdiagnosis. This paper proposes Distractor-Aware Neuron Intrinsic Learning (DANIL), a novel training framework designed to identify these internal distractors and explicitly teach the model to ignore them.

The DANIL Framework

The core innovation of DANIL is its "neuron intrinsic" approach. Rather than simply selecting difficult training examples, DANIL analyzes how the network's internal activations respond to specific inputs to identify the features causing confusion.

The DANIL framework architecture, illustrating the two-pass training strategy to identify and neutralize distractors.

The framework employs a two-pass training strategy during each iteration:

  1. Identification of Distractors: The network performs an initial forward pass. It generates two "intrinsic response maps" by calculating the gradient of the loss with respect to the input image:
  2. $A^+$ (The Positive Map): Derived from the ground truth label, highlighting pixels that contribute to a correct classification.
  3. $A^-$ (The Distractor Map): Derived from a "pseudo label" (the most probable incorrect class), highlighting pixels that trick the network into a wrong answer.
  4. Distraction Loss ($L_d$): A novel loss function is introduced to maximize the distance between $A^+$ and $A^-$ in the feature space. By minimizing this loss, the network is forced to differentiate between features that lead to correct vs. incorrect predictions.

Visualizing Neuron Intrinsic Responses

One of the most compelling aspects of DANIL is the ability to visualize what the network considers a "distractor." As shown in Figure 3, the $A^+$ map typically focuses on the primary pathology (e.g., the lesion), while the $A^-$ map often highlights irrelevant background noise or peripheral skin features that might be shared across different classes.

Visualizations of intrinsic response maps $A^+$ (correct features) and $A^-$ (distractor features) for dermatoscopic and fundus images.

By explicitly separating these representations, the model becomes more robust to background variations and focuses more accurately on the diagnostic features.

Experimental Results

The researchers validated DANIL across three major 2D medical imaging benchmarks:

  • HAM10000: Dermatoscopic skin images (7 categories).
  • APTOS2019: Fundus images for diabetic retinopathy (5 categories).
  • CRC-MSI: Histology patches for gastrointestinal cancer (binary classification).

Across all datasets and multiple CNN backbones (including ResNet50, VGGNet11, Inception V4, and EfficientNet-b0), DANIL consistently outperformed standard baselines and existing techniques like Online Hard Example Mining (OHEM). For instance, on the HAM10000 dataset, ResNet50 accuracy improved from 77.9% to 82.5% when trained with the DANIL scheme. Unlike OHEM, which can sometimes degrade performance on certain architectures, DANIL provided a stable and robust performance boost in every tested scenario.

Conclusion and Future Work

DANIL represents a significant step forward in making medical AI more reliable by addressing the root cause of feature-level confusion. By synthesizing distractors directly in the neuron intrinsic space, the method provides a "plug-and-play" strategy that enhances various CNN architectures without requiring structural changes.

Future research will look into reducing the computational overhead of the two-pass training and extending the framework to 3D medical imaging data, such as CT and MRI scans, where inter-class similarities are equally prevalent.

Summary updated

Abstract

Medical image analysis benefits Computer Aided Diagnosis (CADx). A fundamental analyzing approach is the classification of medical images, which serves for skin lesion diagnosis, diabetic retinopathy grading, and cancer classification on histological images. When learning these discriminative classifiers, we observe that the convolutional neural networks (CNNs) are vulnerable to distractor interference. This is due to the similar sample appearances from different categories (i.e., small inter-class distance). Existing attempts select distractors from input images by empirically estimating their potential effects to the classifier. The essences of how these distractors affect CNN classification are not known. In this paper, we explore distractors from the CNN feature space via proposing a neuron intrinsic learning method. We formulate a novel distractor-aware loss that encourages large distance between the original image and its distractor in the feature space. The novel loss is combined with the original classification loss to update network parameters by back-propagation. Neuron intrinsic learning first explores distractors crucial to the deep classifier and then uses them to robustify CNN inherently. Extensive experiments on medical image benchmark datasets indicate that the proposed method performs favorably against the state-of-the-art approaches.
BibTeX
@inproceedings{gong2020distractor,
  author = {Lijun Gong and Kai Ma and Yefeng Zheng},
  title = {{Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  doi = {10.1007/978-3-030-59713-9\_57},
}
Figure