Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions

Rafael Pedro, Arlindo L. Oliveira

In the world of medical imaging, the stakes for computer vision are incredibly high. A missed pixel could mean a missed diagnosis. While Convolutional Neural Networks (CNNs) have been the workhorse of this field for a decade, a new contender—the Transformer—is challenging the status quo.

This paper, "Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions," provides a rigorous head-to-head comparison between traditional CNNs augmented with "attention modules" and modern, self-attention-based Transformer architectures.

TL;DR

The researchers compared various attention-enhanced ResNets against Vision Transformers (ViTs) for skin lesion classification. Their findings: while "plug-and-play" attention modules provide inconsistent boosts to CNNs, pure self-attention architectures (Transformers) consistently deliver superior accuracy, often with significantly fewer parameters.

What Problem Does This Paper Address?

CNNs are inherently "local." They process images through small filters (kernels) that only see a tiny neighborhood of pixels at a time. To understand the "big picture," a CNN relies on stacking many layers to gradually increase its receptive field. This makes it difficult for the network to relate distant parts of an image—for example, comparing the border of a lesion on one side to the center on the other.

"Attention" was designed to fix this. It allows a model to selectively focus on the most informative parts of an input while ignoring noise (like hair or skin texture). The authors wanted to know: Is it better to simply add attention "modules" to existing CNNs, or should we move to architectures built entirely on self-attention?

Methods

The authors divided their investigation into two main categories:

  1. Attention Modules (The "Add-ons"): These are blocks inserted into standard ResNet architectures to reweight feature maps.
  2. Channel Attention: (SE, ECA) Focuses on which feature maps are important.
  3. Spatial & Channel Attention: (CBAM, TA) Focuses on both where the important features are and what they are.
  4. Attention Augmented Convolutions (AA): Replaces some convolutions with a mix of convolution and self-attention.
  5. Self-Attention Architectures (The "Ground-Up" approach): These models treat an image like a sequence of patches, similar to how a language model treats words in a sentence.
  6. Vision Transformer (ViT): The baseline transformer for vision.
  7. Data-Efficient Image Transformer (DeiT): A version optimized to require less data.
  8. Convolutional Vision Transformer (ConViT): A hybrid that introduces "gated positional self-attention" to mimic some of the beneficial biases of CNNs.

Datasets

The study utilized the Skin Cancer MNIST (HAM10000) dataset.

  • Composition: 10,013 dermatoscopic images.
  • Challenge: It includes seven classes of skin lesions (e.g., melanoma, basal cell carcinoma). The images are notoriously noisy, containing hair, bubbles, and varying skin tones, with the actual lesion often occupying only a small portion of the frame.

Key Insights and Ideas

  • The "Aha" Moment on Shape vs. Texture: The authors suggest that Transformers perform better because they focus more on the global shape of a lesion rather than just the local texture. In dermatology, the symmetry and border of a lesion are critical diagnostic markers that global self-attention captures more naturally than local convolutions.
  • The Inconsistency of Plug-ins: Adding attention modules (like SE or CBAM) to a ResNet that was already pre-trained on ImageNet often degraded performance. This suggests that these modules might disrupt the finely-tuned features the network learned during pre-training.
  • Efficiency Wins: One of the most surprising findings was that ViT-Tiny, with only 5.5 million parameters, achieved higher accuracy than a ResNet-50, which has over 23 million parameters. This challenges the "bigger is better" mantra in deep learning.

Key Results

  • Top Performer: The ViT-Base model achieved the highest balanced accuracy of 0.737, outperforming the best ResNet variant (ResNet-152 at 0.682).
  • Transformer Dominance: Even the smallest Transformer models (ViT-Tiny and ConViT-Small) consistently beat or matched much larger ResNet models.
  • Pre-training Paradox: While pre-training on the massive ImageNet-21k dataset is usually a "cheat code" for performance, it provided no significant benefit for this specific skin lesion task compared to standard ImageNet-1k pre-training.

Limitations and Future Work

  • Data Hunger: While Transformers performed well here, they generally require massive amounts of data to beat CNNs. The authors noted that "sample efficiency" remains a hurdle for Transformers in niche medical domains with limited data.
  • Computational Cost: Training these models from scratch on large datasets is prohibitively expensive for many researchers, making them dependent on publicly available pre-trained weights.
  • Future Direction: The authors suggest that future research should focus on refining self-attention specifically for medical imaging, perhaps by finding better ways to incorporate "inductive biases" (like the locality of CNNs) without losing the global perspective of the Transformer.

Summary updated

Abstract

Attention mechanisms have raised significant interest in the research community, since they promise relevant improvements in the performance of neural network architectures. However, in any specific problem, we still lack a principled way to choose specific mechanisms and hyper-parameters that lead to guaranteed improvements. More recently, self-attention has been proposed and widely used in transformer-like architectures, leading to significant breakthroughs in some applications. In this work we focus on two forms of attention mechanisms, attention modules and self-attention. Attention modules are used to reweigh the features of each layer input tensor. Different modules have different ways to perform this reweighting in fully connected or convolutional layers. The attention models studied are completely modular and in this work they will be used with the popular ResNet architecture. Self-attention, originally proposed in the area of natural language processing makes it possible to relate all the items in an input sequence. Self-attention is becoming increasingly popular in computer vision, where it is sometimes combined with convolutional layers, although some recent architectures do away entirely with convolutions. In this work, we study and perform an objective comparison of a number of different attention mechanisms in a specific computer vision task, the classification of samples in the widely used Skin Cancer MNIST dataset. The results show that attention modules do sometimes improve the performance of convolutional neural network architectures, but also that this improvement, although noticeable and statistically significant, is not consistent in different settings. The results obtained with self-attention mechanisms, on the other hand, show consistent and significant improvements, leading to the best results even in architectures with a reduced number of parameters.
BibTeX
@inproceedings{pedro2022assessing,
  author = {Rafael Pedro and Arlindo L. Oliveira},
  title = {{Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9892274},
}

Loading related content...

Figure