Revisiting the Shape-Bias of Deep Learning for Dermoscopic Skin Lesion Classification

Adriano Lucieri, Fabian Schmeisser, Christoph Peter Balada, Shoaib Ahmed Siddiqui, Andreas Dengel, Sheraz Ahmed

Introduction

In the field of General Computer Vision (GCV), a significant body of research has highlighted the "texture-bias" of Convolutional Neural Networks (CNNs). While humans typically identify objects based on their global shape and structure, CNNs often rely on local textures—identifying a cat, for instance, by the pattern of its fur rather than its silhouette. Consequently, many researchers advocate for making models more "shape-biased" to improve robustness and mimic human-like generalization.

However, medical imaging presents a different set of challenges. In dermoscopy, clinical diagnosis relies on a complex interplay of subtle pigment networks, specific hues, and vascular patterns. This paper investigates whether the drive for shape-bias is appropriate for skin lesion classification or if it overlooks the fundamental nature of medical data, where color and texture are often as diagnostic as the lesion's border.

Methodology

To dissect how deep learning models perceive skin lesions, the researchers employed three distinct analytical strategies:

1. Feature Ablations

The authors generated seven unique versions of skin lesion images to isolate specific visual components:

  • Color Isolation: Pixels were randomly scrambled to preserve the overall color distribution while destroying all spatial structure.
  • Texture + Shape: Images were transformed into "sketches" to remove color information while retaining edges and internal textures.
  • Shape Isolation: Segmentation masks were used to present only the lesion's outline, removing all internal information.

2. Spectral Analysis

Using Fourier Transforms, the researchers separated images into two components:

  • Phase: Encodes spatial structure, edges, and shape.
  • Amplitude: Encodes the magnitude of frequency components, which in this context relates closely to color and lighting intensity.

3. Deep Feature Reweighting (DFR)

The authors utilized DFR to determine if models trained on standard images actually encoded information about isolated features (like shape) internally, even if those features were not prioritized for the final classification decision.

Dataset Context

The study utilized two primary benchmarks to represent different scales of medical data:

  • ISIC (International Skin Imaging Collaboration): A large-scale archive (69,000+ images) used for both multi-class and binary (Melanoma vs. Nevus) classification.
  • D7P (Seven-Point Checklist Dataset): A smaller, clinically-annotated dataset consisting of 1,011 lesions.
  • Baselines: Imagenette and Imagewoof (subsets of ImageNet) were used to provide a comparison against natural image classification.

Key Insights

The Role of Dataset Scale

The researchers found that bias is highly dependent on the dataset. The smaller D7P dataset showed a stronger bias toward Shape, whereas the larger ISIC dataset was more sensitive to Texture. This suggests that as datasets grow, models learn to exploit more nuanced, fine-grained features that go beyond simple outlines.

The Primacy of Color

Across nearly all experiments, color emerged as the most critical feature. The removal of color information resulted in the most significant performance drops, confirming that spectral information is a primary driver of diagnostic accuracy in dermoscopy.

Feature Entanglement

While models trained on standard images performed poorly when tested on ablated (e.g., shape-only) images, the DFR analysis revealed that the models had actually encoded these features in their internal representations. The "knowledge" of the lesion's shape existed within the network, but it was "tangled" with other features and ignored during the final decision-making process.

Dermoscopy vs. Natural Images

In GCV tasks like identifying a dog, the Phase (shape) of an image is often sufficient for high accuracy. In dermoscopy, however, models require both Phase and Amplitude. Skin lesions are not merely "objects" with defined outlines; they are complex spectral signals where the intensity and distribution of color are inseparable from the diagnosis.

Results and Proposed Solutions

Failure of GCV Robustness Techniques

Techniques designed to improve robustness in natural images by focusing on the phase spectrum (such as Amplitude Phase Recombination, or APR) were found to actually decrease accuracy in skin classification tasks. This highlights the danger of blindly applying GCV heuristics to the medical domain.

Amplitude-Focused Augmentation (AF-APR-P)

To address this, the authors proposed a new approach: "Amplitude-Focused" augmentation. By forcing the network to focus more on the amplitude (color and intensity) rather than just the phase (shape), they achieved a statistically significant improvement in binary classification for the ISIC dataset, increasing the F1-score from 88.45% to 89.23%.

Conclusion

The study concludes that skin lesion classification requires specifically tailored solutions rather than a one-size-fits-all approach borrowed from general computer vision. While shape is a component of the diagnostic puzzle, the "spectral signal"—the entanglement of color and texture—is the true driver of performance. Future work may involve using contrastive loss or newer architectures like Vision Transformers (ViTs) to better disentangle these features, but the core takeaway remains: in dermoscopy, the "object" is more than just its shape.

Summary updated

BibTeX
@inproceedings{lucieri2022revisiting,
  author = {Adriano Lucieri and Fabian Schmeisser and Christoph Peter Balada and Shoaib Ahmed Siddiqui and Andreas Dengel and Sheraz Ahmed},
  title = {{Revisiting the Shape-Bias of Deep Learning for Dermoscopic Skin Lesion Classification}},
  booktitle = {MIUA},
  year = {2022},
  doi = {10.1007/978-3-031-12053-4\_4},
}
Figure