A Rotation Meanout Network with Invariance for Dermoscopy Image Classification and Retrieval

Yilan Zhang, Fengying Xie, Xuedong Song, Hangning Zhou, Yiguang Yang, Haopeng Zhang, Jie Liu

This paper introduces the Rotation Meanout (RM) network, a novel architectural module designed to grant standard Convolutional Neural Networks (CNNs) rotation invariance. By processing rotated versions of feature maps through shared weights and fusing them, the authors achieve significant performance gains in skin lesion classification and retrieval without increasing the model's parameter count.

TL;DR

The RM module provides a plug-and-play solution for rotation invariance in dermoscopy images by rotating internal feature maps and fusing them via a "meanout" strategy. It outperforms traditional data augmentation and specialized equivariant architectures while remaining compatible with standard pre-trained CNNs.

What Problem Does This Paper Address?

In dermoscopy—the microscopic imaging of skin—there is no "natural" orientation. A lesion can be captured from any angle, meaning a dataset contains the same pathological features in countless rotated versions.

Standard CNNs are not naturally rotation-invariant; they treat a 90-degree rotated image as a distinct entity. While common workarounds like Data Augmentation (DA) help, they increase training time and the risk of overfitting. Other solutions, like Group Equivariant CNNs (GCNNs), require specialized layers that add parameters and break compatibility with popular pre-trained models. The authors seek a "general operation" that provides invariance without these overheads.

Methods

The proposed RM module consists of four sequential stages, as illustrated in the flow chart below:

Figure 1: RM network. In expansion stage, feature maps are expanded k times by rotating them at same interval angles θ. In realignment stage, the k groups of rotated feature maps are input to a shared convolutional structure to extract features. Then, inversely rotate the feature map back to its original angle, so that the outputs can be realigned to keep their spatial position relation. Finally, these k sets of realigned feature maps are fused to obtain one set of final feature maps by using the mean pooling operation.
  1. Expansion: Input feature maps are rotated $k$ times at equal intervals (e.g., 0°, 90°, 180°, 270°).
  2. Feature Extraction: These $k$ sets of maps are fed into a shared convolutional structure (like a ResNet block). Because weights are shared, the network learns features that are consistent across orientations.
  3. Realignment: The output maps are rotated back to their original orientation to maintain spatial alignment.
  4. Fusion (Meanout): The realigned maps are averaged (mean pooling across the rotation dimension).

The authors mathematically prove that this process is rotation-equivariant. When this module is followed by a Global Average Pooling (GAP) layer—standard in most modern classifiers—the entire network becomes rotation-invariant, meaning the final feature vector is identical regardless of the input image's orientation.

Datasets

The authors evaluated their method using the ISIC 2019 dataset (International Skin Imaging Collaboration).

  • Size: 25,331 dermoscopy images.
  • Categories: 8 diagnostic classes, including Melanoma (MEL), Melanocytic nevus (NV), and Basal cell carcinoma (BCC).
  • Split: 8:1:1 ratio for training, validation, and testing.

Key Insights and Ideas

  • Zero-Parameter Overhead: Unlike many anti-rotation methods, RM does not add any learnable parameters. It simply changes how existing layers process data, making it highly efficient and easy to integrate into architectures like ResNet, Inception, or EfficientNet.
  • Strategic Embedding: The authors found that embedding the RM module in the middle stages of a CNN yields the best results. Shallow layers lack semantic depth, while the feature maps in very deep layers are often too small, leading to significant interpolation errors during rotation.
  • Meanout vs. Maxout: While "maxout" (taking the maximum value across rotations) is common, the authors found that "meanout" (averaging) better preserves feature continuity and leads to higher accuracy in dermoscopy tasks.
  • Strict vs. Relaxed Invariance: The paper distinguishes between a "Strict RM-GAP" structure (where RM is immediately followed by GAP) and a "Relaxed" version. Strict structures provide true invariance and generally perform better.
Figure 2: Embedding an RM in ResNet18.

Key Results

The RM module consistently improved performance across multiple baseline architectures:

  • Classification: On ResNet18, RM increased Average Precision (AP) from 55.41% to 64.66%. It outperformed other state-of-the-art methods like TI-pooling (59.33%) and GCNN (53.19%).
  • Retrieval: For image retrieval tasks, RM achieved a mean Average Precision (mAP@10) of 60.20%, significantly higher than the baseline (53.75%) and other hashing-based methods.
  • Feature Separability: t-SNE visualizations show that RM produces much tighter and more distinct clusters for different skin disease categories compared to standard ResNet18.

Limitations and Future Work

  • Interpolation Noise: Rotating feature maps requires interpolation. When feature maps are small (e.g., $7 \times 7$), rotating by non-90-degree angles (like 45°) introduces errors that can degrade performance. Consequently, the authors recommend 90-degree intervals.
  • Segmentation: The authors noted that RM did not significantly improve skin lesion segmentation. They hypothesize that for segmentation, spatial equivariance is more important than the global invariance provided by the RM-GAP structure. Future work may explore how to better adapt RM for pixel-level tasks.

Summary updated

Abstract

The computer-aided diagnosis (CAD) system can provide a reference basis for the clinical diagnosis of skin diseases. Convolutional neural networks (CNNs) can not only extract visual elements such as colors and shapes but also semantic features. As such they have made great improvements in many tasks of dermoscopy images. The imaging of dermoscopy has no principal orientation, indicating that there are a large number of skin lesion rotations in the datasets. However, CNNs lack rotation invariance, which is bound to affect the robustness of CNNs against rotations. To tackle this issue, we propose a rotation meanout (RM) network to extract rotation-invariant features from dermoscopy images. In RM, each set of rotated feature maps corresponds to a set of outputs of the weight-sharing convolutions and they are fused using meanout strategy to obtain the final feature maps. Through theoretical derivation, the proposed RM network is rotation-equivariant and can extract rotation-invariant features when followed by the global average pooling (GAP) operation. The extracted rotation-invariant features can better represent the original data in classification and retrieval tasks for dermoscopy images. The RM is a general operation, which does not change the network structure or increase any parameter, and can be flexibly embedded in any part of CNNs. Extensive experiments are conducted on a dermoscopy image dataset. The results show our method outperforms other anti-rotation methods and achieves great improvements in dermoscopy image classification and retrieval tasks, indicating the potential of rotation invariance in the field of dermoscopy images.
BibTeX
@article{zhang2022rotation,
  author = {Yilan Zhang and Fengying Xie and Xuedong Song and Hangning Zhou and Yiguang Yang and Haopeng Zhang and Jie Liu},
  title = {{A Rotation Meanout Network with Invariance for Dermoscopy Image Classification and Retrieval}},
  journal = {Computers in Biology and Medicine},
  year = {2022},
}
Figure