Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions

Rafael Pedro, Arlindo L. Oliveira

Abstract

Attention mechanisms have raised significant interest in the research community, since they promise significant improvements in the performance of neural network architectures. However, in any specific problem, we still lack a principled way to choose specific mechanisms and hyper-parameters that lead to guaranteed improvements. More recently, self-attention has been proposed and widely used in transformer-like architectures, leading to significant breakthroughs in some applications. In this work we focus on two forms of attention mechanisms: attention modules and self-attention. Attention modules are used to reweight the features of each layer input tensor. Different modules have different ways to perform this reweighting in fully connected or convolutional layers. The attention models studied are completely modular and in this work they will be used with the popular ResNet architecture. Self-Attention, originally proposed in the area of Natural Language Processing makes it possible to relate all the items in an input sequence. Self-Attention is becoming increasingly popular in Computer Vision, where it is sometimes combined with convolutional layers, although some recent architectures do away entirely with convolutions. In this work, we study and perform an objective comparison of a number of different attention mechanisms in a specific computer vision task, the classification of samples in the widely used Skin Cancer MNIST dataset. The results show that attention modules do sometimes improve the performance of convolutional neural network architectures, but also that this improvement, although noticeable and statistically significant, is not consistent in different settings. The results obtained with self-attention mechanisms, on the other hand, show consistent and significant improvements, leading to the best results even in architectures with a reduced number of parameters.

Introduction

In medical imaging, the precision of computer vision models is paramount. While Convolutional Neural Networks (CNNs) have served as the industry standard for a decade, they possess an inherent limitation: locality. CNNs process images through small kernels that only observe local pixel neighborhoods. To capture global context—such as the relationship between distant borders of a skin lesion—CNNs must stack numerous layers to expand their receptive field.

Attention mechanisms were introduced to mitigate this by allowing models to selectively focus on informative regions while suppressing noise like hair or skin texture. This study evaluates whether it is more effective to augment existing CNNs with modular attention "add-ons" or to adopt architectures built entirely on self-attention, such as Vision Transformers (ViTs).

Methodology

Attention Modules (The "Add-on" Approach)

These modules are inserted into standard ResNet architectures to reweight feature maps based on importance. The study examined:

  • Channel Attention: (e.g., SE, ECA) focusing on which feature maps are most relevant.
  • Spatial & Channel Attention: (e.g., CBAM, TA) focusing on both where and what the important features are.
The CBAM module architecture, which combines channel and spatial attention to reweight CNN feature maps.

Self-Attention Architectures (The "Ground-Up" Approach)

Unlike CNNs, these models treat an image as a sequence of patches. This allows the model to relate any two parts of an image regardless of distance from the very first layer. The study evaluated the baseline Vision Transformer (ViT), the Data-Efficient Image Transformer (DeiT), and the hybrid Convolutional Vision Transformer (ConViT).

The Vision Transformer (ViT) architecture, which processes images as sequences of patches using self-attention.

Dataset and Experimental Setup

The models were tested on the Skin Cancer MNIST (HAM10000) dataset, containing 10,013 dermatoscopic images across seven lesion classes. This dataset is challenging due to noise (hair, bubbles) and the fact that lesions often occupy only a small portion of the image.

Results

The experimental results demonstrated a clear advantage for self-attention architectures:

  • Top Performance: The ViT-Base model achieved the highest balanced accuracy of 0.737, significantly outperforming the best ResNet variant (ResNet-152 at 0.682).
  • Efficiency: Remarkably, ViT-Tiny (5.5M parameters) outperformed ResNet-50 (23M parameters), suggesting that self-attention is a more parameter-efficient way to model complex medical images.
  • Inconsistency of Modules: Adding attention modules like SE or CBAM to pre-trained ResNets often resulted in performance degradation, likely because these modules disrupted features already learned during ImageNet pre-training.

Discussion

The superiority of Transformers in this task may stem from their ability to prioritize global shape over local texture. In dermatology, diagnostic markers like symmetry and border regularity are global properties that self-attention captures more naturally than the local filters of a CNN.

However, Transformers face challenges regarding "data hunger." While they performed well here, they typically require larger datasets to reach peak performance compared to CNNs. Additionally, the study found a "pre-training paradox": while ImageNet-21k pre-training is generally superior, it offered no significant advantage over ImageNet-1k for this specific skin lesion classification task.

Conclusion

While modular attention can provide statistically significant boosts to CNNs, these improvements are inconsistent across different settings. In contrast, pure self-attention architectures like Vision Transformers provide consistent and significant performance gains. For skin lesion classification, the global perspective offered by Transformers appears to be a critical advantage, even when using models with fewer parameters than traditional CNNs.

Summary updated

Abstract

Attention mechanisms have raised significant interest in the research community, since they promise significant improvements in the performance of neural network architectures. However, in any specific problem, we still lack a principled way to choose specific mechanisms and hyper-parameters that lead to guaranteed improvements. More recently, self-attention has been proposed and widely used in transformer-like architectures, leading to significant breakthroughs in some applications. In this work we focus on two forms of attention mechanisms: attention modules and self-attention. Attention modules are used to reweight the features of each layer input tensor. Different modules have different ways to perform this reweighting in fully connected or convolutional layers. The attention models studied are completely modular and in this work they will be used with the popular ResNet architecture. Self-Attention, originally proposed in the area of Natural Language Processing makes it possible to relate all the items in an input sequence. Self-Attention is becoming increasingly popular in Computer Vision, where it is sometimes combined with convolutional layers, although some recent architectures do away entirely with convolutions. In this work, we study and perform an objective comparison of a number of different attention mechanisms in a specific computer vision task, the classification of samples in the widely used Skin Cancer MNIST dataset. The results show that attention modules do sometimes improve the performance of convolutional neural network architectures, but also that this improvement, although noticeable and statistically significant, is not consistent in different settings. The results obtained with self-attention mechanisms, on the other hand, show consistent and significant improvements, leading to the best results even in architectures with a reduced number of parameters.
BibTeX
@inproceedings{pedro2022assessing,
  author = {Rafael Pedro and Arlindo L. Oliveira},
  title = {{Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9892274},
}
Figure