Introduction
In the field of General Computer Vision (GCV), a significant body of research has highlighted the "texture-bias" of Convolutional Neural Networks (CNNs). While humans typically identify objects based on their global shape and structure, CNNs often rely on local textures—identifying a cat, for instance, by the pattern of its fur rather than its silhouette. Consequently, many researchers advocate for making models more "shape-biased" to improve robustness and mimic human-like generalization.
However, medical imaging presents a different set of challenges. In dermoscopy, clinical diagnosis relies on a complex interplay of subtle pigment networks, specific hues, and vascular patterns. This paper investigates whether the drive for shape-bias is appropriate for skin lesion classification or if it overlooks the fundamental nature of medical data, where color and texture are often as diagnostic as the lesion's border.
Methodology
To dissect how deep learning models perceive skin lesions, the researchers employed three distinct analytical strategies:
1. Feature Ablations
The authors generated seven unique versions of skin lesion images to isolate specific visual components:
- Color Isolation: Pixels were randomly scrambled to preserve the overall color distribution while destroying all spatial structure.
- Texture + Shape: Images were transformed into "sketches" to remove color information while retaining edges and internal textures.
- Shape Isolation: Segmentation masks were used to present only the lesion's outline, removing all internal information.
2. Spectral Analysis
Using Fourier Transforms, the researchers separated images into two components:
- Phase: Encodes spatial structure, edges, and shape.
- Amplitude: Encodes the magnitude of frequency components, which in this context relates closely to color and lighting intensity.
3. Deep Feature Reweighting (DFR)
The authors utilized DFR to determine if models trained on standard images actually encoded information about isolated features (like shape) internally, even if those features were not prioritized for the final classification decision.
Dataset Context
The study utilized two primary benchmarks to represent different scales of medical data:
- ISIC (International Skin Imaging Collaboration): A large-scale archive (69,000+ images) used for both multi-class and binary (Melanoma vs. Nevus) classification.
- D7P (Seven-Point Checklist Dataset): A smaller, clinically-annotated dataset consisting of 1,011 lesions.
- Baselines: Imagenette and Imagewoof (subsets of ImageNet) were used to provide a comparison against natural image classification.
Key Insights
The Role of Dataset Scale
The researchers found that bias is highly dependent on the dataset. The smaller D7P dataset showed a stronger bias toward Shape, whereas the larger ISIC dataset was more sensitive to Texture. This suggests that as datasets grow, models learn to exploit more nuanced, fine-grained features that go beyond simple outlines.
The Primacy of Color
Across nearly all experiments, color emerged as the most critical feature. The removal of color information resulted in the most significant performance drops, confirming that spectral information is a primary driver of diagnostic accuracy in dermoscopy.
Feature Entanglement
While models trained on standard images performed poorly when tested on ablated (e.g., shape-only) images, the DFR analysis revealed that the models had actually encoded these features in their internal representations. The "knowledge" of the lesion's shape existed within the network, but it was "tangled" with other features and ignored during the final decision-making process.
Dermoscopy vs. Natural Images
In GCV tasks like identifying a dog, the Phase (shape) of an image is often sufficient for high accuracy. In dermoscopy, however, models require both Phase and Amplitude. Skin lesions are not merely "objects" with defined outlines; they are complex spectral signals where the intensity and distribution of color are inseparable from the diagnosis.
Results and Proposed Solutions
Failure of GCV Robustness Techniques
Techniques designed to improve robustness in natural images by focusing on the phase spectrum (such as Amplitude Phase Recombination, or APR) were found to actually decrease accuracy in skin classification tasks. This highlights the danger of blindly applying GCV heuristics to the medical domain.
Amplitude-Focused Augmentation (AF-APR-P)
To address this, the authors proposed a new approach: "Amplitude-Focused" augmentation. By forcing the network to focus more on the amplitude (color and intensity) rather than just the phase (shape), they achieved a statistically significant improvement in binary classification for the ISIC dataset, increasing the F1-score from 88.45% to 89.23%.
Conclusion
The study concludes that skin lesion classification requires specifically tailored solutions rather than a one-size-fits-all approach borrowed from general computer vision. While shape is a component of the diagnostic puzzle, the "spectral signal"—the entanglement of color and texture—is the true driver of performance. Future work may involve using contrastive loss or newer architectures like Vision Transformers (ViTs) to better disentangle these features, but the core takeaway remains: in dermoscopy, the "object" is more than just its shape.
Summary updated