This paper introduces the Rotation Meanout (RM) network, a novel architectural module designed to grant standard Convolutional Neural Networks (CNNs) rotation invariance. By processing rotated versions of feature maps through shared weights and fusing them, the authors achieve significant performance gains in skin lesion classification and retrieval without increasing the model's parameter count.
TL;DR
The RM module provides a plug-and-play solution for rotation invariance in dermoscopy images by rotating internal feature maps and fusing them via a "meanout" strategy. It outperforms traditional data augmentation and specialized equivariant architectures while remaining compatible with standard pre-trained CNNs.
What Problem Does This Paper Address?
In dermoscopy—the microscopic imaging of skin—there is no "natural" orientation. A lesion can be captured from any angle, meaning a dataset contains the same pathological features in countless rotated versions.
Standard CNNs are not naturally rotation-invariant; they treat a 90-degree rotated image as a distinct entity. While common workarounds like Data Augmentation (DA) help, they increase training time and the risk of overfitting. Other solutions, like Group Equivariant CNNs (GCNNs), require specialized layers that add parameters and break compatibility with popular pre-trained models. The authors seek a "general operation" that provides invariance without these overheads.
Methods
The proposed RM module consists of four sequential stages, as illustrated in the flow chart below:
- Expansion: Input feature maps are rotated $k$ times at equal intervals (e.g., 0°, 90°, 180°, 270°).
- Feature Extraction: These $k$ sets of maps are fed into a shared convolutional structure (like a ResNet block). Because weights are shared, the network learns features that are consistent across orientations.
- Realignment: The output maps are rotated back to their original orientation to maintain spatial alignment.
- Fusion (Meanout): The realigned maps are averaged (mean pooling across the rotation dimension).
The authors mathematically prove that this process is rotation-equivariant. When this module is followed by a Global Average Pooling (GAP) layer—standard in most modern classifiers—the entire network becomes rotation-invariant, meaning the final feature vector is identical regardless of the input image's orientation.
Datasets
The authors evaluated their method using the ISIC 2019 dataset (International Skin Imaging Collaboration).
- Size: 25,331 dermoscopy images.
- Categories: 8 diagnostic classes, including Melanoma (MEL), Melanocytic nevus (NV), and Basal cell carcinoma (BCC).
- Split: 8:1:1 ratio for training, validation, and testing.
Key Insights and Ideas
- Zero-Parameter Overhead: Unlike many anti-rotation methods, RM does not add any learnable parameters. It simply changes how existing layers process data, making it highly efficient and easy to integrate into architectures like ResNet, Inception, or EfficientNet.
- Strategic Embedding: The authors found that embedding the RM module in the middle stages of a CNN yields the best results. Shallow layers lack semantic depth, while the feature maps in very deep layers are often too small, leading to significant interpolation errors during rotation.
- Meanout vs. Maxout: While "maxout" (taking the maximum value across rotations) is common, the authors found that "meanout" (averaging) better preserves feature continuity and leads to higher accuracy in dermoscopy tasks.
- Strict vs. Relaxed Invariance: The paper distinguishes between a "Strict RM-GAP" structure (where RM is immediately followed by GAP) and a "Relaxed" version. Strict structures provide true invariance and generally perform better.
Key Results
The RM module consistently improved performance across multiple baseline architectures:
- Classification: On ResNet18, RM increased Average Precision (AP) from 55.41% to 64.66%. It outperformed other state-of-the-art methods like TI-pooling (59.33%) and GCNN (53.19%).
- Retrieval: For image retrieval tasks, RM achieved a mean Average Precision (mAP@10) of 60.20%, significantly higher than the baseline (53.75%) and other hashing-based methods.
- Feature Separability: t-SNE visualizations show that RM produces much tighter and more distinct clusters for different skin disease categories compared to standard ResNet18.
Limitations and Future Work
- Interpolation Noise: Rotating feature maps requires interpolation. When feature maps are small (e.g., $7 \times 7$), rotating by non-90-degree angles (like 45°) introduces errors that can degrade performance. Consequently, the authors recommend 90-degree intervals.
- Segmentation: The authors noted that RM did not significantly improve skin lesion segmentation. They hypothesize that for segmentation, spatial equivariance is more important than the global invariance provided by the RM-GAP structure. Future work may explore how to better adapt RM for pixel-level tasks.
Summary updated