Geometric Data Augmentation Based on Feature Map Ensemble

Takashi Shibata, Masayuki Tanaka, Masatoshi Okutomi

Standard Convolutional Neural Networks (CNNs) are notoriously sensitive to geometric transformations. A model trained on upright images often fails when presented with rotated or shifted versions of the same objects. To address this, researchers from NTT and Tokyo Tech have introduced a novel "plug-and-play" CNN wrapper that enhances robustness by ensembling aligned feature maps at intermediate layers.

The Limitations of Test-Time Augmentation

The current industry standards for geometric robustness are Data Augmentation (DA) during training and Test-Time Augmentation (TTA) during inference. TTA involves averaging the final prediction scores of several transformed versions of an input image. However, TTA suffers from two primary drawbacks:

  1. Confidence Bias: If a model is confidently incorrect about a single transformed view, it can disproportionately skew the final average.
  2. Information Loss: By ensembling only at the final output layer, TTA discards the rich spatial information contained within intermediate feature maps.

Proposed Methodology: Feature Map Ensemble

The authors propose a "sandwich" architecture that wraps an existing CNN backbone. This framework processes multiple geometric variants of an image and merges their features before final classification.

Proposed architecture for feature map ensembling across geometric transformations.

The process follows five distinct steps:

  1. Forward Transformation: The input image $x$ is transformed into $N$ variants (e.g., rotations of 0°, 90°, 180°, and 270°).
  2. Feature Extraction: These variants pass through the initial portion of the backbone, $f_F$.
  3. Reverse Transformation: The resulting feature maps $\tilde{z}_N$ are geometrically "un-rotated" to ensure they are spatially aligned.
  4. Feature Map Ensemble ($S$): The aligned maps are merged using a Maximum ($S_{max}$) or Mean ($S_{mean}$) operation. The Max operation effectively selects the orientation where specific feature detectors were most certain.
  5. Final Processing: The ensembled map $\hat{z}$ passes through the remainder of the backbone ($f_R$) and the classification head ($g$).

This approach is inspired by Maxout networks but applies the selection mechanism across geometric orientations rather than just activation functions.

Key Results and Robustness

The method was evaluated on standard benchmarks including CIFAR-10/100, SVHN, and CUB-200. A critical test involved the Mnist-rot-12k dataset, where the proposed method achieved 98.96% accuracy, outperforming specialized rotation-invariant architectures like RotEqNet.

Comparison of accuracy across different rotation angles for various methods.

As shown in the figure above, the "Ours (max)" approach maintains high accuracy regardless of the input rotation angle, whereas the baseline (BS) performance drops significantly at 90°, 180°, and 270°. Even standard TTA and DA do not reach the same level of consistent robustness.

On CIFAR-10, using a ResNet18 backbone, the proposed method reached 94.66% accuracy, significantly higher than standard TTA (83.26%) and standard data augmentation (91.28%). Furthermore, the method shows strong synergy with other techniques; when combined with WideResNet and RandAugment, it achieved 93.59% on CIFAR-10, surpassing previous state-of-the-art TTA methods.

Conclusion

By preserving spatial context through intermediate feature map ensembling, this method provides a robust alternative to standard TTA. Its "plug-and-play" nature allows it to be used with pre-trained backbones, offering a massive boost in geometric invariance with minimal retraining. Future research will explore extending this framework to photometric transformations, such as changes in lighting and contrast, and applying it to dense prediction tasks like semantic segmentation.

Summary updated

Abstract

Deep convolutional networks have become the mainstream in computer vision applications. Although CNNs have been successful in many computer vision tasks, it is not free from drawbacks. The performance of CNN is dramatically degraded by geometric transformation, such as large rotations. In this paper, we propose a novel CNN architecture that can improve the robustness against geometric transformations without modifying the existing backbones of their CNNs. The key is to enclose the existing backbone with a geometric transformation (and the corresponding reverse transformation) and a feature map ensemble. The proposed method can inherit the strengths of existing CNNs that have been presented so far. Furthermore, the proposed method can be employed in combination with state-of-the-art data augmentation algorithms to improve their performance. We demonstrate the effectiveness of the proposed method using standard datasets such as CIFAR, CUB-200, and Mnist-rot-12k.
BibTeX
@inproceedings{shibata2021geometric,
  author = {Takashi Shibata and Masayuki Tanaka and Masatoshi Okutomi},
  title = {{Geometric Data Augmentation Based on Feature Map Ensemble}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506654},
}
Figure